Marketing Measurement: Is Econometrics the Answer?

I recently chatted to the excellent Philip Gaudoin of marketing insights agency Piquant about the role econometrics can play in marketing measurement.


I learnt so much from our chat that I invited him to participate in a quick Q&A on the Navigator Marketing blog.


His view is that many see econometric analysis is only valid for huge brands with millions of transactions, whereas in fact it can be a useful way for much smaller enterprises to assess the effectiveness of their activity and determine the optimum marketing mix.


Ben: What marketing measurement questions can econometrics answer?


Philip: There are literally hundreds from the big, strategic questions (e.g. ‘How should we allocate our budget across our markets, brands and media?’) to the smaller, tactical ones (e.g. ‘What is the optimum number of TV ratings to deploy each week?’). It’s not all about above-the-line media, either. Econometrics can be used to evaluate and optimise your pricing strategy, promotional deployment, and your digital and direct media.


Ben:  OK, there are plenty of travel & tourism marketers that would like to know the answers to those. So how does econometrics work?


Philip:  The USP of econometrics is that it is the only technique capable of disentangling the effects of simultaneous drivers on, for example, sales. The key word is ‘simultaneous’. We don’t have the luxury of a lab in which we can perform endless test and control experiments to establish the exact impact of each marketing lever.  Econometrics gets around this by measuring the correlations among the variables in recent history. Think of it as a very precise eye. So, where you and I could eyeball two data series and see any obvious correlation, econometrics can, in effect, do this for multiple series at once, and much more accurately.


Ben: That sound almost too good to be true. Be honest, what are its limitations?


Philip: Econometrics is great at measuring immediate uplifts, but it’s less good at measuring subtler, long-term effects. For instance, we all know that the right media can build your brand equity. Eventually, this should translate into more sales. You would struggle to tie this back to the original media using econometrics (or, in fact, any other technique).
It doesn’t furnish you with explanations – it tells you the ‘what’ not the ‘why’. Sometimes the ‘why’ is obvious, but at other times more interpretation is needed. This is when the experience and calibre of your consultants comes to the fore.
Thirdly, of course, it isn’t a crystal ball. It’s worth stating the obvious here, as a lot of confusion is caused by talk of ‘predictive analytics’. An econometric model is a great basis for making this sort of prediction, but prediction always assumes that the future behaves like the past.


Ben:  Don’t you need masses of data, though? Surely it’s only for really big companies.


Philip:  The key principle is that you need to collate data for anything you believe (based on experience, common sense, insight from other sources) will have significantly affected the thing you are trying to model, e.g. sales. Sometimes this can be a lot of data, but at other times, surprisingly little is required to build a robust model. You definitely need data, there’s no escaping that. However, the amount you need varies, not all of it needs to come from you (your agencies and public bodies provide a lot) and you probably have more than you realise. On more than one occasion we’ve readily turned up data from the IT or systems department that the marketers didn’t know was available – econometrics also helps break down silos!


Ben: Can it really work for companies of that scale? Aren’t there too many variables and not enough transactions?


Philip: This is a common confusion about econometrics and aggregation. If we were to attempt to model each and every purchase of a product, even something purchased every week, we would quickly run short of suitable variables in the available data.
Econometrics works in the aggregate. By the time you sum every purchase made in a given week, you usually have enough transactions to work with. And working at the aggregate level means we can use aggregate driver variables (e.g. total media deployment) which are the ones we have plenty of.  Of course, there’s a limit to everything. It would not work for Lamborghinis, where the sales are about 4 a week globally, but it will work for, say, Toyotas, or Volkswagons or Fords.


Ben:  You hear about econometrics modelling failing in the economic world. I heard about one recent example where an IMF model was undermined by the exclusion of a relevant data input. Isn’t there the same risk when you apply econometrics to marketing?


Philip: Yes. Whenever an econometrics model fails it is because of misspecification. Sometimes you start with a correctly specified model, but the world changes around you and your predictions don’t come true. Many macroeconomic models are being revised, now, since the financial crash and the onset of ‘quantitative easing’ changed the rules.  You can’t do anything about the world changing in unpredictable ways. You can, however, ensure that your model is correctly specified for the current world. Again, the calibre of your modellers matters. Expert modellers who understand what they are doing have a good feel for when something is right. That experience, combined with the relevant statistics, is a powerful combination.


Ben: Isn’t it very expensive though? You could argue that the money would be better invested in media or creative rather than econometric marketing measurement, couldn’t you?


Philip:  But, how do you know whether it’s the right media or the right creative? If you know, 100%, in your heart that it’s right, then don’t build an econometric model – the chances are that you won’t be persuaded even if it says something completely different!  However, most of us aren’t nearly so sure. We’re also conscientious and want to know that we’re spending often very large amounts of money in the right way. Added to that, there’s the pressure of persuading your Finance Director, who may not have the same confidence in your gut-feel that you do! Suppose a project costs £40,000? That’s 2% of a marketing budget of £2m. £40,000 buys you an extra week in the national newspapers. Isn’t it worth being off-air one single week to know how the other £1.96m works? Now what if, as a result of the project, you find that newspapers don’t work, and you move £1m of your budget on, say radio, instead producing a conservative 20% improvement in ROI? The project has likely paid for itself and the missing week of magazines in one go.


Ben: Have you got any good examples of businesses that have benefited from econometrics?


Philip:  Every project we do is intended to solve business problems; it’s rare we do a project that is a ‘nice-to-know’ only. Thus we always kick off a project with an immersion session to understand the underlying business issues. The modelling approach is tailored to answer these questions.  So, for one client we used our models to change their media strategy towards their core product range, producing an improvement in ROI. For another, we used econometrics to measure the best level of media deployment, and avoid wasting money on unnecessary exposures. We’ve used models to define promotional strategy, optimise pricing, ranging and quantify distribution gaps. We’ve also used them to optimise investment across markets, brands and media channels. All of these cases have produced some amount of tangible business benefit – more sales and/or profit.


Ben: There must be occasions when it hasn’t worked, though?


Philip: I haven’t known an out-and-out failure. Sometimes a model isn’t able to answer all of a client’s questions definitively. This has been obvious from the outset (we can usually tell from our experience and understanding of your category) and we have managed the client’s expectations accordingly from the start. If everyone’s clear on what they will/wont/might get from a project, then that makes the risk minimal. Some plain speaking upfront is strongly recommended!


Ben: The final word is yours.


Philip:  Get good people. The recipe for econometric success is expert modellers working on a well-scoped, well-run project. That takes a certain amount of experience. Beware of paying too much (day rates of £1,500 probably mean you are subsidising corporate overheads, rather than paying for actual working time). Beware of paying too little (projects that cost £1,000 per model are probably of dubious quality and cross-subsidised by other purchases, so you are really paying for it elsewhere). Shop around, don’t buy the first thing offered you and always meet the people and make sure they fill you with confidence before you start.