Data Science Mistake 6 – Leaks from the Future

We’re continuing our look at the “Deadly Dozen” data science mistakes—the costly errors that derail otherwise promising analytics projects. Last time, in Mistake 5: Listen Only to the Data, we explored why information from outside the data is often needed to tell the full story. This time we’ll examine scenarios where the answer seems clear, but the model is invalid due to future information inadvertently slipping into model training.

For years when I taught workshops on data science mistakes, I surveyed attendees about which of the dozen they had ever encountered. After hundreds of responses, that distribution was surprisingly level, but this “leaks from the future” mistake edged out the others as having been seen the least. However, in my own experience, it is instead the most common, so it must often go undiagnosed. This mistake may also be the deadliest one as it leads to over-confident use of a model that is much worse than one thinks!

When assembling a dataset to predict or classify output Y using inputs X at time T, it is essential that all the X’s are knowable by or before T. If some information beyond T infects the dataset, the model will cheat and look good on paper but fail in the real world. Two real-world examples of this were described in Mistake #3: Use Only One Technique. In one, a neural network was 95% accurate at predicting future interest rates, but the output itself turned out to be one of the inputs.^[1] In the second, a strange set of codes was associated 100% with the sale of insurance.^[2] The codes turned out to indicate how someone canceled their insurance, which is information from after the outcome being predicted!^[3] In both cases, the first clue was a result that was much better than plausible. Always aggressively examine a result that seems too good to be true.^[4]

Speaking of Too Good to Be True

I was called in to examine a quantitative hedge fund strategy that was 70% accurate in forecasting the direction an index would move the next day. Its developers were quite secretive (befitting such an astonishing result), and believed their triumph came from their expert use of expensive “fourth generation” software. After intense work, I duplicated the decisions of their system exactly with a simple moving average of three days of prices. Worst of all, those three days were centered on today. That is, tomorrow’s price was one of the inputs!^[5] The discovery led to the demise of their start-up, but had they not learned that and proceeded blindly the end would have been worse, with lawsuits and accusations of fraud.

Leaks Can Be Indirect, Affecting Cases or Features

A leak can be indirect too, affecting features or cases. For example, I was working on developing a stock-picking system in the days before there were sophisticated price and return datasets. Historic prices could be gathered but they needed to be adjusted for dividends and stock splits so that the true return could be calculated.^[6] I locked down the most recent price for each stock then backward adjusted to find the effective prices to use in my dataset by calculating what the previous price needed to be to reflect the true total return. This handily made return calculations simple and accurate, but I unknowingly infected the dataset with a leak from the future. Models showed some surprisingly good results using well-known features such as price-to-earnings ratios. When I noticed that raw prices alone were a strong indicator (low prices leading to great long-term returns), I felt the dread of something being fatally wrong. In retrospect, it was obvious: a stock that does very well will have more splits and dividends,^[7] so the backward-adjusting of prices I performed will greatly lower the early effective prices. Thus, a very low effective price is a clear sign that the future returns will be very positive. So, the effective price value and any feature employing it (other than the simple returns for which it was derived) were not valid to use.

In general, data warehouses are built to hold the best information known to date on each item (e.g., customer, client, case); they are not easily able to pull out only what was known at the time that you wish to study a decision. So, when storing data for future analysis, it’s important to date-stamp records and to archive the full collection at regular intervals. Otherwise, recreating realistic information states will be extremely difficult, and will lead to wrong conclusions.

For example, the recent enormous run-up in AI company values reminds one of the dot-com mania at the turn of the millennium. There was a crash then, but the prior rise was long and rapid; were those firms a bad bet or not? Using a price-quoting service, you can pull down the histories of all such existing companies and study their returns. Your study might conclude that they would have been a fantastic bet, despite the horrible shakeout in that market sector that started in March 2000. Why? Were their early gains so great as to absorb later massive losses? No; you would have made a type of leak from the future known as “survivor bias” – by looking back from current companies, which is all most data services carry. A re-creation of the set of all dot-com companies that existed at the earlier time, including those that later failed, would be needed to find the true returns of the sector, and the result would be much more negative.

In the example above on low stock prices some features were infected by future information; in the dot-com example it was the selection of cases that was biased by future knowledge. A final example (of the features sort) shows how subtle this issue can be.

We were called in to help a start-up firm in the medical diagnostic field. Their founder had innovated a way to shine infrared light through skin to gain information about blood chemistry without having to draw blood. Their backers invested a lot to build the new hardware and software required and they recruited many volunteers diverse in age, sex, skin coloration, sun exposure, health conditions, etc. to assemble a database for modeling. They had written a paper accepted by a well-regarded peer-reviewed journal showing diagnostic results more accurate than existing techniques. Still, they believed higher accuracy would be required to overtake the entrenched advantage of familiar methods.

Our initial finding was not received as good news. We showed them that they had inadvertently leaked information from the future and their true accuracy was significantly worse.^[8] We then steadily improved the models until they actually exceeded the accuracy the team had initially believed they’d reached. But start-ups are always in a race with their bank account, and that pressure seemed to keep their otherwise brilliant chief technical officer from ever understanding the concept of leaks from the future. (And their project required even more care than usual in handling the dataset as the time and cost to gather completely new data for true out-of-sample (OOS) testing was prohibitive.) One example of their many experiments: They augmented available features by calculating principal components (PCs). Then they split the data into training and evaluation subsets and built a powerful model on the training data. It did well on the OOS data! But they had done a key step wrong. The PC calculations used all the data; thus, for instance, if there was an outlier in (what would later be designated) the OOS data, it was already seen and factored in. The PC vectors were already “bent towards” the OOS data, so the data was not completely new. When they went back and calculated the PCs using only the training data, the resulting evaluation accuracy was hurt more than any of us would have guessed.

Conclusion

To avoid these problems, use only information that would have been available at the time of the prediction, and never let out-of-sample (OOS) data influence training. OOS data often needs to be put in a lock box, with absolutely nothing about it known to the modeling team, so that it can properly do its role of representing reality. Ask yourself, “Is it possible that I will be truly caught off guard by something in the OOS data?” If not, then it’s not OOS enough.

[1] With that leak removed the model was honest but worthless.

[2] In the earlier blog this insurance sales example is just referred to as an upselling problem.

[3] These two examples of leaks from the future were discussed in the section on using multiple modeling techniques as different models best discovered their diagnoses: regression (for the interest rate problem) and decision trees (for the insurance sales problem).

[4] A judgement that often requires domain knowledge.

[5] They’d have been right 100% of the time if they’d just dropped the oldest input variable.

[6] If a stock is $100 on Monday, pays a $3 dividend and the price ends at $98 on Tuesday, it looks like it went down $2, but the value of the stock actually went up $1. Even more dramatically, if it instead split two-for-one on Monday the flat price Tuesday would be $50. So, naïve historical prices can’t be used in simulation experiments.

[7] In theory, stock prices are arbitrary; only returns matter. But practice is different. Most companies use splits and, to a lesser extent dividends, to tamp down their stock prices to near the $100 range, which they believe to be psychologically satisfying to potential clients. Berkshire-Hathaway’s Class A shares have done very well over six decades and have not split or paid a cash dividend since 1967. It has the highest share price of any stock (> $784K as of October 2026). On the lower end, if a stock’s average price is below $1 for a month on the NY Stock Exchange, it falls out of compliance with listing requirements. To avoid being de-listed they can do reverse-splits to raise their price.

[8] Even the journal’s peer reviewers missed this leak, illustrating how difficult these errors can be to detect. We’ll describe how to do research more reliability in Mistake #12: Cherry Picking.