09.07.2026
The “Deadly Dozen” Data Science Mistakes
Mistake 2 – Focus on Training
The Deadly Dozen Data Science Mistakes
We’re continuing our look at the “Deadly Dozen” data science mistakes—the costly errors my colleagues and I have seen derail analytics projects. Last time we explored the foundational problem of lacking crucial data (Mistake 1). Now we turn to another common hazard: the urge to squeeze every drop of performance out of the data in front of you, inadvertently crippling the model for cases the future will throw at it.
It’s natural to judge a model by how well it handles the information it was built on, but to prove it works, it must be tested against data it has never seen before. Those out-of-sample (OOS) results are all that matter. If you instead seek to best fit the training data, then a trivial lookup table (which memorizes the data rather than finding patterns) will always win. Your goal should be to build a model that performs well on similar but never-before-seen cases.
Three decades ago, I was invited to meet with researchers at the UT MD Anderson hospital in Houston. They were trying out the then-hot method of neural networks to detect cancer. Their out-of-sample results were fairly accurate, though worse than training (as is the norm). They had been training networks for one full day and believed that longer training would improve results – after all, that’s the way it works with doctors.[1] Instead, they were astonished to find that training for a full week led to only slightly lower training error and much worse evaluation error. This was a classic case of overfit. Obsessing on training accuracy focused the model so much on the details of the data sample that it failed to learn general lessons for the real world. When a model overfits, it memorizes the historical noise and exact details of the past instead of discovering the underlying rules needed to predict the future.
A common temptation for new practitioners is to refine a model until it achieves an exact fit on known data. That is virtually guaranteed to lead to overfit. Look at Figure 1 below as an example. It shows a one-dimensional function (the red line) sampled at 10 points (the dark blue diamonds), and the polynomial equations (with increasing powers of x) fit by regression. The most complex model fits the training data most closely but oscillates wildly, poorly estimating values everywhere else.[2]
Model Selection
It has been known for centuries that making a model more complex improves training error but eventually worsens evaluation error[3]. Model-selection metrics that penalize a weighted combination of error and complexity can be useful to obtain good models. I used these heavily in my early career and found that the metrics with the harshest complexity penalties were the most useful – for example, the Bayesian Information Criterion and the elegant Minimum Description Length measure based on information theory[4]. In practical terms, these are mathematical rules that force the model to stay simple, automatically rejecting complex equations unless they significantly improve accuracy. But I eventually learned to rely solely on direct OOS measurements using resampling—a technique that repeatedly shuffles and re-tests the data you already have.
A single split of the data into training and evaluation sets is the most popular evaluation approach. However, it’s surprising how different the error measure can be between two ways of splitting. Cross-validation (and other resampling methods like bootstrap, jackknife and leave-one-out) provides reliable OOS error estimates by splitting, fitting, and scoring the data multiple times to obtain a distribution of OOS results. Instead of relying on one single test, this method repeatedly divides your historical data, building the model on one section and verifying it on the hidden section to guarantee consistent performance. The mean (or median) of that distribution becomes your error estimate, and the standard deviation (or inter-quartile distance) represents your confidence in that estimate. The mean tells you the average accuracy you can expect, while the standard deviation reveals your operational risk—how wildly that accuracy might fluctuate when deployed.
For instance, 5-fold cross-validation would:
- Randomly assign each case to one of five sets.[5]
- Hold out set 1 and fit a model using the other 80% of the data.
- Score the model by its fit on the OOS data in set 1.
- Repeat steps 2–3 for each of the four remaining sets.
- Use the mean and sigma of the distribution of five OOS values as the quality of the modeling procedure.
Footnotes
- Arguably, learning works best—with both humans and machines—when we spend our time budget on encountering more cases (stimulus X + response Y) rather than diving deeply into the details of a few cases.
- Unless two cases have identical inputs and different outputs, a polynomial with K terms (degrees of freedom) can exactly fit a dataset with K cases—even if the data is random. The same is true for a decision tree having K -1 splits. If the data has any noise in it, those overfit models are almost surely worthless.
- Occam’s Razor, from the 14th century, urges you to “not multiply complexity beyond necessity.” The concept of simplicity improving accuracy even goes back millennia to Aristotle and Ptolemy.
- My first journal article, published in 1991, explained how to apply Minimum Description Length to market forecasting.
- Don’t instead divide the data by using the top 20% for subset 1, the next 20% for 2, etc. Datasets usually arrive sorted in some way, so that approach would define subsets distinct from one another in a structured way, which leads to a worse OOS fit than natural.
- To be more concrete: If you run a decision tree on its default settings with cross-validation, then you are evaluating not a single model but a modeling procedure (a decision tree on its default settings).
- Ensembles will appear later in this series on mistakes. Explore my book on Ensembles.
