The “Deadly Dozen” Data Science Mistakes:
Mistake 1 - Lack of Crucial Data

In my introduction to this series, I shared a saying that resonated with me: “Good judgment comes from experience, and experience comes from bad judgment.” My goal is to help you succeed by revealing the key mistakes my colleagues and I have made over many years—the “deadly dozen” culprits that derail projects, starting with Mistake 1: Lack of Crucial Data.

The best analyses require labeled cases, i.e., an output or target variable. If you only have input variables, all you can do is look for subsets with similar characteristics (cluster) or find the dimensions which best capture the data variation (principal components). Those unsupervised techniques often provide useful insight but are much less powerful than a good (supervised) prediction or classification model.

Given a target variable, the most interesting class or type of observation is usually the rarest by orders of magnitude. For instance, roughly 1/10 of “risky” individuals given credit will default within two years, 1/100 people mailed a catalog will respond with a purchase, and 1/1,000 CT scans actually cause cancer.[1] The less probable the interesting events, the more overall data it takes to obtain enough to generalize a model to unseen cases.[2] Some projects probably should not proceed until enough critical data is gathered to make them worthwhile.

For example, on a project to discover fraud in government contracting, known fraud cases were so rare that strenuous effort could only initially reduce the size of the haystack in which the needles were hiding.[3] The model predictions did enable auditors to focus their effort on the highest-scoring cases. But more known fraud cases—good for data scientists but bad for taxpayers—could have provided the modeling the traction needed to automatically flag suspicious new cases more quickly and accurately.

This was certainly the situation on another project of ours, focused on discovering tax fraud collusion[4]. Unfortunately (for taxpayers), there were plenty of training examples, but that did enable stronger analytic results. Excellent models were created, enthusiastically implemented by the IRS, and ultimately credited with saving taxpayers over $20 billion in just their first three years.

One can’t uncover insights without data, but not just any data will work. Many data science projects must make do with “found” data, not the results of an experiment designed to illuminate the question studied. It’s like making a salad out of what can be foraged in the yard.

One sophisticated credit-issuing company realized this when seeking to determine if there was a market for their products in the class of applicants once routinely dismissed as being too risky. Perhaps a low-limit card would be profitable and even help a deserving subset of applicants pull themselves up in their credit rating? But the company had no data on such applicants by which to distinguish the truly risky from those worth a try; their traditional filters excluded such individuals from even initial consideration. So, they essentially gave (small amounts of) credit almost randomly to thousands of risky applicants and monitored their repayments for two years. Then, they built models to forecast defaulters (those significantly late on payments) trained only on initial application information. The models revealed a profitable subset of formerly ignored prospects. Their large investment in creating relevant data paid off in revealing a way to profitably expand their customer base.

So, make sure that the data you’re working with is relevant to the problem to be solved!

I’ll briefly summarize the other major ways that lacking data can be hazardous:

Few degree programs prepare graduates for these last two problems, but they are very common in the real world. And few projects consider the value of (or budget for) following up on the problem cases. It is very helpful to carefully examine a case that the model had great trouble getting “right.” It’s likely mislabeled, but if not, it can reveal even more information.

I’ll close with a heartening story—at least for modelers. We worked hard on a challenging biometric classification task. Our client was very skeptical that analytics would help. They agreed to re-examine the top 200 cases our model got wrong (likely to prove their point and be freed of such nerds). Instead, they found that 75% of those initial labels were wrong, and they became enthusiastic backers of our further assistance.

In my next blog, we’ll explore the second deadly data science mistake: focusing on training. We’ll explore why a model can look great in the lab but completely fail in the wild and how you can ensure your models actually work when it matters most. Stay tuned!