Assumptions, Explanation, and Prediction in Marketing Science: “It’s the Findings, Stupid, Not the Assumptions"

Back to the Peer Reviewed Journals List

Executive Summary

Tsang references a previous debate in Marketing Science on whether analytic models need to have realistic assumptions and stakes out a position that modifies Shugan’s “It’s the Findings, Stupid, Not the Assumptions” point of view in 2007.

Tsang reasons that the realism of an assumption is a continuous variable – not binary. That is to say, an assumption isn’t necessary unrealistic or realistic – the realism of assumptions is more like a thermometer than it is a light-switch.

Tsang suggests that: “Authors should discuss the probable impacts of the assumptions on their findings, predictions, or implications, and if possible, test the sensitivity by varying the assumptions and compare results...The burden of proof should lie on those who use the assumption.” He goes onto suggest: “As the field advances, efforts should be directed toward making assumptions more realistic…More realistic assumptions result in better theories.”

Tsang concludes: “although Shugan (2007) rightly stresses that it is inappropriate to dismiss a model or theory based only on the realism of its assumptions, realism does matter, and it matters a great deal for model building and theory development.”

This is an important conclusion. Instead of building theories and models based on unrealistic assumptions, like a house of cards that falls down when a door slams, we ought to be building a brick house. Building on a solid foundation of assumptions ensures the long-term predictive value of a discipline.

Review

Implications for Practitioners

I’d have to retype the article in its entirety to do it justice. The summary above highlights the most relevant bits for web analysts.

Web analysts are frequently asked to make assumptions. Making unrealistic assumptions has gotten us into deep trouble in the past. For instance, the assumption that a unique visitor really represented a unique person was intensely flawed. And yet, for reasons that have been well documented, we as an industry stretched the assumption for the sake of precision. We as practitioners are frequently pressured to bend on assumptions – one way or another. We ought to become self-aware of why that happens and use sensitivity models to communicate the consequences.

Web analysts make predictions about the future. Optimization is not possible without making a range of predictions about the future and selecting the best scenario to guide action (Even A/B testing falls into this definition). Any prediction is fraught with unknowns and uncertainty. We use assumptions to simplify those unknowns, and to a large extent, quantify them.

For instance, a common question is how many more sales would we get if we employed [generic tactic X]. That kind of analysis requires reasoning and preferably accurate evidence for previous circumstances. We lack a common a pool of findings in our industry. Many analysts lack a common pool of evidence at best, and at worst, a sanitized version of what was a success and what was not. Regardless of how much data we have, we still make assumptions for the purposes of making predictions.

Tsang makes reference to the fact that not all sciences are predictive in nature. For instance, Earth Scientists are great at explaining earthquakes after they’ve happened, but are seldom predictive. Indeed, (tongue-in-cheek), many web analysts are indeed deployed like Earth Scientists. Like Earth Scientists, in certain circumstances, it’s worth pushing assumptions to the limit of believability so as to understand the range of what’s possible.

We should take advantage of those types of models to communicate with non-analysts. We can treat assumptions like a continuous variable and push the limits if it helps us make better predictions in the future, so long as those assumptions return to Earth in the final analysis and recommendation. The next generation of visualization software could assist us in communicating the bounds of what’s likely when given a set of assumptions – going beyond the spreadsheet.

Understanding and acknowledging that assumptions exist in our own analytical world will free us to challenge each other on those assumptions more often. We will all be better off for it.

I recommend that Web Analytics Practitioners read this article.