what i actually learned building alphascanner

Everyone talks about backtesting like it's the finish line. I thought so too. Then I watched a strategy with a Sharpe of 1.4 in backtest go flat for 6 weeks live and had to decide whether to trust it.

that's the real problem. not the math; the epistemics.

What I learned:

Walk-forward validation is the minimum bar, not the gold standard. You can still overfit to regimes if you're not careful. I eventually built a market regime filter that just turns the whole system off when the S&P 500 is below its 200-day MA. Simple. Works.

the 29 survivors out of however many strategy/asset combinations i tested, i had to get comfortable with the fact that some of those are probably noise. The ones with OOS Sharpe > 0.79 across 9 regime windows feel real. But "feel real" is not a great epistemic foundation.

Live trading is a different sport. Latency, slippage, the way your hands feel when you're down 3% in a week. None of that is in the backtest.

The thing I'm still not sure about:

Whether it's actually better to run a systematic strategy you don't fully trust, or to just buy and hold. The numbers say systematic. My gut says "you don't know what you don't know."

maybe that's the whole game: operating under uncertainty without letting that uncertainty paralyze you.