Texas sharpshooter fallacy
The Texas sharpshooter fallacy is related to hypothesis testing. It occurs when you rely on the same data to both come up with a hypothesis and test it.
It is named after a parable where a (Texan) gunman fires his rifle at the side of a barn, paints a target around the bullet holes and then claims to be a sharpshooter. Clearly, the gunman is not taking chance into account.
Rather than following the example of the Texan, we should first formulate a hypothesis. Only after having formulated the hypothesis, can we use data to try and test our hypothesis. If we rely on data to construct a hypothesis, then we should use an alternative data set to test whether it contiues to hold. This way we can avoid that the patterns we observe are simply due to chance.
The Texas Sharpshooter fallacy can occur when designing trading systems and. Traders are often tempted to test various alternative trading systems until they find a system that proves profitable when applied to the available data. Such a ‘backtest’ is called an ‘in-sample analysis’.
However, without testing the trading system ‘out-of-sample’, i.e. using novel data, a trader cannot be sure that the trading system will also work in real life. Only by applying it to new data can we verify whether our results are meaningful, or whether they are simply the result of luck.