Common Backtesting Mistakes: 6 Pitfalls That Turn Good Data into Bad Decisions

A backtest shows 200% annual returns. You mortgage your house and go all-in. Three months later, you’re broke. What went wrong? The backtest was real — the problem was what the backtest didn’t tell you. Here are the most common traps that turn backtesting from a useful tool into a dangerous illusion.

1. Overfitting: The #1 Killer

Overfitting happens when your strategy is too perfectly tuned to historical data. “Buy when RSI(13.7) crosses above 48.3 on Tuesdays after a full moon” might produce amazing backtested returns — because you’ve essentially memorized the past, not discovered a pattern that will repeat.

How to spot it:

• Too many parameters (more than 2-3 is a red flag)
• Unusual, specific values (why 13.7? Why not 14?)
• Strategy breaks down when you change any parameter slightly
• Works amazingly on one asset/timeframe but fails on everything else

How we avoid it at Boring Edge: All our backtests use standard, widely-accepted parameter values (RSI-14, SMA-200, MACD 12/26/9). We don’t optimize. The 200-day moving average isn’t special because we tested 50 different periods and 200 worked best — it’s the industry standard that millions of traders watch.

2. Survivorship Bias

You test your strategy on “the top 10 cryptocurrencies” and get great results. But you selected those coins because they’re currently successful. What about the hundreds of coins that were once in the top 10 but crashed to zero? (Remember Luna? FTT? BitConnect?)

If you only backtest on winners, every strategy looks good. The same applies to stocks — backtesting on current S&P 500 members ignores companies that were removed after failing.

Our approach: We primarily backtest on BTC/USDT, which has existed continuously since 2017 on Binance. It hasn’t been “selected for success” — it’s the entire market reference asset.

3. Look-Ahead Bias

This is subtle and deadly. It happens when your strategy uses information that wouldn’t have been available at the time of the decision. Common examples:

• Using the day’s closing price to make a trading decision at market open
• Calculating an indicator using today’s data to generate today’s signal
• Adjusting parameters after seeing the full dataset

How we avoid it: All our signals use the previous day’s closing data to make today’s decision. When we say “buy when RSI crosses above 50,” we mean yesterday’s RSI crossed above 50, so you buy at today’s open. The signal is always delayed by one day.

4. Ignoring Transaction Costs

A strategy that trades 368 times in 8 years (like our Heikin Ashi test) pays fees 368 times. At 0.1% per trade, that’s 36.8% of your capital eaten by fees over the period. At 0.2% per trade (many retail exchanges), it’s 73.6%.

Always check: Does the backtest include fees? What fee rate? Is the fee rate realistic for your exchange tier? Some backtests show amazing returns with 0% fees that evaporate when you add real costs.

Our approach: Every Boring Edge backtest includes 0.1% transaction fees on both buy and sell. This is conservative for major exchanges (Binance VIP tiers go as low as 0.02%), but we’d rather underestimate returns than overestimate them.

5. Slippage Blindness

Backtests assume you can buy and sell at exactly the prices shown. In reality, large orders move the market. If you’re trading $1M and the strategy says “buy at $68,000,” you might actually get filled at $68,050-$68,200 depending on liquidity. This is slippage, and it compounds over many trades.

For most retail traders with normal position sizes, slippage is minimal on BTC/USDT. But for strategies with very frequent trading, even small slippage adds up significantly.

6. Regime Change

This is the hardest one to protect against. Markets change. Bitcoin in 2018 behaved differently from Bitcoin in 2021 which behaves differently from Bitcoin in 2025. A strategy that worked perfectly in one regime may fail in the next.

This is why we test over the longest period possible (8+ years for BTC, covering multiple bull and bear cycles). A strategy that works across 2018’s crash, 2020’s COVID, 2021’s bull run, and 2022’s bear market is more likely to continue working than one only tested on 6 months of data.

The Bottom Line

Backtesting is the best tool we have for evaluating trading strategies. But it’s not a crystal ball. Every backtest tells you what DID happen, not what WILL happen. Use backtests to eliminate bad ideas, validate good ones, and build conviction — but never assume past performance guarantees future results.

A good backtest should make you cautiously optimistic, not recklessly confident.

Understand the metrics in our backtests: Backtest Metrics Decoded. See all our results: Strategy Backtests.