DataTaunew | comments | leaders | submitlogin
3 points by Tomrod 3530 days ago | link | parent

> How is a "story" compelling when it is less accurate?

I have some reasoning to this.

Business leaders will generally use heuristics. If a model isn't, or can't, be communicated in such a way as to match the generally accepted heuristic, that model will have a short shelf life if it ever gets approved at all.

Consider these heuristics as part of a strong, partially or fully unobservable (to the econometrician or data scientist) prior that should be communicated. Since the prior is unobservable, models are rejected until the prior is accounted for.

Backwards? A bit. But there are models that exist that just don't make sense--and often that is because the causality is wrong but the evaluator of such point has difficulty communicating it.

A good idea before building a model is to gather the downstream clients, and ask what they've observed in the relationship the model being built will explain. Pay attention to the variables they name, the events that occurred in the past. Discuss why they think coefficients will be of certain size, or certain direction, or why another variable wouldn't work better.

Then build their model, and build your model.

As both models are evaluated over time (under good model governance!) compare the best model to the model that was built on consensus. If the model outperforms the model build on consensus, you have a compelling case to your model. You can then begin to persuade and invite others to modify their heuristics to include your model.

Many industries have been as burned by over-reliance on models as under-reliance. Consider, for example, Long-Term Capital Management, who crashed because of a model, or the London Whale, which was also caused by a model (though the model was flawed and untested, business folks may not believe it as so).




RSS | Announcements