The guide to attribution and Marketing Mix Modelling (MMM) in 2024

Key Takeaways with Resources:

1. Challenges with Attribution:
   • Attribution models like last-click often over-credit certain channels (e.g., paid search) and under-credit others (e.g., offline media).
   • Google’s data-driven attribution is an improvement but still limited to digital channels.
2. Understanding MMM:
   • MMM uses econometrics to quantify how different marketing activities affect sales.
   • It includes all influences throughout the customer journey, such as offline and online advertising, weather, seasonality, and more.
3. Practical Tips for Getting Started:
   • Use metrics wisely: Focus on simple, convincing metrics that clearly show campaign effectiveness.
   • Build a balanced measurement framework: Combine econometrics with other methods to get a comprehensive view of your marketing impact.
   • Use models that make it easy: Utilize tools like the recovery budget planner from Magic Numbers to simulate different scenarios and optimize your marketing strategy.
4. Using Metrics Wisely:
   • Identify key metrics at different stages of the customer journey (e.g., awareness, consideration, intent, purchase).
   • Use clear, visual data to demonstrate the impact of marketing campaigns on these metrics.
5. Building a Balanced Measurement Framework:
   • Combine different measurement approaches to capture all aspects of marketing performance.
   • Recommended methods include year-on-year comparisons, share of search, pre- and post-campaign tracking, and regional A/B testing.
6. Using MMM for Different Business Types:
   • For B2B businesses with long sales cycles, use long-tailed decay rates to account for the extended customer journey.
   • For smaller businesses, start with simpler models and focus on optimizing budget allocation and proving the value of marketing.
7. Adapting to Changes in the Industry:
   • With the deprecation of third-party cookies, rely on first-party data and other measurement techniques.
   • MMM does not depend on tracking data, making it a robust option even as digital tracking becomes more challenging.

Practical Tips:

• Empathy and Communication: Foster open communication with stakeholders to understand their data needs and ensure buy-in for MMM initiatives.
• Collaboration: Work closely with different teams to gather comprehensive data and insights.
• Adaptation: Regularly revisit and update your models to reflect changes in the business environment and marketing strategies.

Final Thoughts:

• Continuous Learning: Stay updated with the latest tools and techniques in econometrics and marketing measurement.
• Human Element: Ensure a human touch in building and validating models to capture the nuances of your business and market.

Additional Resource Mentioned:

• Magic Numbers Recovery Budget Planner: A tool to input business data and simulate different economic scenarios to optimize marketing spend. Available on the Magic Numbers website.
• Magicworks Courses: For more in-depth learning on MMM, data analysis, and marketing strategies. Check out the courses starting in autumn 2024.

Transcript

Speaker 1: Hello everyone. It’s so lovely to see you all. Thank you so much for taking the time. It’s a real pleasure. I just saw Peter say greetings from brutal Birmingham in the chat which is a hell of a way to describe your city. It’s a real pleasure. If you haven’t already do drop in the chat where you’re watching from and say hello to everyone. I can see that Kim says in the chat, I haven’t joined a marketing meetup for ages, but I love the new credits. It is fabulous. Kim, this is for you. Your messages came into hosts and panelists only. If your messages are presently in the chat feature and say to and then a colon hosts and panelists, then make sure to switch that to everyone so everyone can see your messages just like Eric has in Barcelona, Annette has in Burnley, Lydia has in Devon, and we’ve got folks coming in from Columbia as well, which is really quite fabulous. It’s a real pleasure and thank you for taking the time. Today, we have Joy Talbot who I think her parents knew what they were doing when they named her because over the duration of every interaction I’ve had with Joy over these past few weeks in curating today’s session, she’s just been exactly that, a complete joy. One demonstration of this is that you will see right now in the QR code that Joy has provided the slides for today’s event right there for you. If you want the slides from today’s event, they’re already there. You don’t need to take the notes, which is fabulous. Do take the time to get those slides. Also while you’re there, check out DataWorks, which is Magic Numbers’ opportunity for you to upskill on your data. If you’re part of this webinar series that we’re doing on data and you’re like, hey, I could be doing more here than like Magic Numbers course here is the place to go after this series of webinars. Absolutely check it out. A big thank you to Joy for taking the time today. Today’s session is important because we all spend a whole lot of time as marketers having to justify our actions. Attribution has been seen as the sort of solution to this problem, but like I was speaking with Joy about this before we went live, like people go, no, it’s a rubbish solution, but nobody seems to suggest anything else. What we’re going to do today is explore attribution and MMM, which is seen as an alternative. It’s not a topic I know a whole lot about, so I’m going to be learning with you too. I think it’s just going to be a really great hour. We’ll have a talk first and then we’ll have Q&A afterwards. Before we get going, I just want to say a big thank you to this week’s featured sponsor who are Exclaimer. Now Exclaimer are, A, lovely folks, but also B, a company who solve a problem, a simple problem in a very lovely way. Exclaimer help you manage your email signatures, which sounds tiny, but if you’ve ever tried to get your whole company to change their email signature, God, you would have felt the pain. Then they’ve also laid in a bunch of other features, which makes your email signature work for you as a marketing channel as well. It starts to become a really powerful tool. As I say, every interaction that we’ve had with Exclaimer folks as well has been really nice. In the QR code on your screen right now, if you start using Exclaimer, you can get 20% off by mentioning the marketing meetup as part of the onboarding process, which is lovely. Also a big thank you to our other sponsors. We’ve got Frontify, Sticky Beak, Plannable, Cambridge Marketing College and Redgate. We’ll speak about each of those in a little bit more turn over the course of the next few weeks, but you’ll also see them in the follow-up email. Be sure to say thank you to those people as well, please. As a quick aside, Plannable, one of the other sponsors, do have a webinar today, which is what should you expect from your social media agency? Starts at five o’clock tonight. If you’re up for another webinar today, then head to that QR code and you can sign up for that event as well. As you can see Amy in the chat saying Exclaimer has been wonderful for you and your business. Thank you, Amy. Great endorsement. With all that said, that’s the introduction done. Joy, thank you for taking the time and it’s over to you.

Speaker 2: Amazing. Thank you so much, Joe, for the lovely intro. Right, I’m going to do the fun bit, which is always to share my screen. Hopefully everyone can see that.

Speaker 1: You’ve nailed it. Seconds.

Speaker 2: That’s always a great start. Okay, fab. Right, off we go. Thank you so much for having me. I’m really excited to be here and meet you all. As Joe said, today the plan is to chat through a guide to attribution and MLM in 2024. I am Joy. I work as a principal economist at Magic Numbers and we are a independent consultancy, mainly doing econometrics projects for lots of different clients. Also lots of different data analyses and incrementality testing and various bits of measurement to help with marketing, also one of the drivers of people’s businesses. Really excited to chat to you about this today. As we go through, if you have any questions, please do put them in the chat. Be brilliant and we’ll make sure we come back to them at the end as well. Without further ado, the plan for today is to chat through challenges with attribution to start with, how MLM is different and then hopefully give you some really practical tips for getting started. The plan is that you leave the session and know exactly what you can do to start using some of these things. Let’s kick off with challenges with attribution. Now I want to tell you a happy and true story that will help explain two different roles for the very same digital advertising. In this case, I’m going to specifically focus on paid search. Now this story is about me and my journey to buying some new trainers. Here’s me in the photo and I decided that I wanted to buy some trainers. I needed to update my wardrobe. I was open to pretty much any pair and I also didn’t have a budget in mind because I didn’t know which pair I wanted to get. I went onto ASOS to have a look what’s available and saw some that I liked from Adidas, but I hadn’t quite decided exactly which ones I wanted. I then saw a TV ad with my son and as a Liverpool fan, I thought, amazing, that’s perfect. Adidas seems pretty cool. Helped drive my perception of the brand. Then I kept googling for trainers that I liked and saw that the same pair kept coming up on sponsored listings. It was the same pair of trainers that I’d seen on ASOS. I still wasn’t quite ready to buy. Maybe I was waiting for payday or I hadn’t quite made my mind up. I hadn’t properly engaged with it, but the ad was really persistent. I’m sure you’ve all had this, where the ad follows you around and you think, Okay, I’ll check it out. When I got to the site, I could see that I could get them delivered next year, which was amazing. I thought, Okay, I’ll take it. Now, in this situation, if we were using digital attribution to measure marketing, it would have given credit to ads that I’ve been exposed to throughout that purchase journey that could be tracked. One commonly used digital attribution, which is last click, would give all of the credit to that paid search ad at the end of the journey. It’s not totally wrong because it did help me to push me over the line. It was an effective ad. Now, imagine a different universe where I’m about to go into the summer. There’s a bit of seasonality that makes me think, oh, it might be time to refresh my wardrobe. My friend Izzy tells me exactly which trainers I should get. I carry on with the rest of the purchase journey as before, doing research and watching that same TV ad and Googling for trainers. Now I’m much more receptive to the Adidas brand, thanks to that recommendation from my friend Izzy, so that when I see the ad, rather than thinking, oh, I might as well have a look, I’m thinking, oh, those Izzy’s trainers are next day delivery. What a bonus. I’m going to buy them now. Now, in the first example, that genuinely helped to push me over the line. In the second, it was more like signposting. Actually, I knew what I wanted at that point. I may well have ended up clicking on an organic listing for exactly the same trainers. With just that little bit of change in the consumer journey, the same ad could either be making a big difference to someone buying or not really have any impact at all. Last click would only attribute credit at the end of the journey. Any other type of digital attribution to some of the steps in the middle. We need to think about alternatives. Now, this example is just in the case where we look at a really simplified journey with a few touch points to buying trainers. In reality, a lot can happen between that first trigger and the last click. I would have seen thousands of different marketing messages and other nudges that got me to the stage where I was happy to make that purchase. The whole time, I was moving backwards and forwards between mentalities of exploration and evaluation of all the different options. We’re not alone thinking about this. Google tried to model the purchase journey with a report a few years ago. Their conclusion was, it’s messy. Modeling purchase journeys is really difficult. It’s something that really big consultancies and lots of other people have grappled with for decades, coming up with their own simplified pictures. I do really like this one that literally just goes right in the middle. It really helps to show that we’re basically talking about millions of consumers for each different brand. Everyone’s different. Coming up with one model for all of the shoppers is going to be tricky. It’s also not surprising that Google focuses on the middle, because that’s where they have the data. It’s no surprise that simple rules-based attributions models aren’t going to give us the right answer for what’s driving our sales all of the time. Google have made some improvements to their digital attribution methodology. They now have something called data-driven attribution. We thought we could start here with some definitions. When different types of what they mean, we try to visualize them. Last click would give all of the credit to the last point, and first click to the first point. Linear attribution says, Okay, everything’s going to be equivalent, and just splits up across the journey. Time decay will give more to what was more recent. It will give a little bit to every other part of the journey as well. Then you’ve got position-based, which will basically split it up depending on where you are in that journey. Then finally, we’ve got data-driven, which is the new Google version, which basically says, depending on different tracking and data that’s available, how can we do that more accurately to give the credit to the right place? Now, if it’s Google or Meta’s own attribution modeling, then it’s only going to be for the channels within that ecosystem. We’re talking about credit for a sale potentially being given to the wrong thing, which can lead to incorrect budget decisions when you’re trying to decide what to do next with your marketing. In reality, it’s an oversimplification of something that can be really complicated. Now, I really like this little picture that comes when we think about looking for your keys under the streetlight. It’s basically making decisions based on the data that you have easily available, rather than looking at a big holistic picture of everything that’s going on. The thing is, this actually works for a lot of small businesses. When you’ve got smaller budgets, you’re starting with digital media where actually everything can be tracked. You’re actually finding your keys. It does work. The important thing to know is that as you grow, the picture that’s missing will become more and more important, and you’ll be unable to see more of that picture. The other thing to note is that particularly on massive attribution, it can miss the mark on payback. Over a lot of studies from Lesbonet, we could see that when we look at the attribution efficiency error versus econometrics, that brand TV, DRTV, press, all these offline channels will be undercredited, and things like paid search and other online channels will be overcredited. In this study, we could see that PPC got three times the credit it should, as far as econometrics was concerned. We’re thinking about actually, how do you come up with a measurement framework that can do this accurately? Also, if you’ve got digital attribution and you’re running other things, how do you connect the dots so that you can make one clear story? In terms of some definitions, digital attribution uses third-party cookies. That’s a cookie that’s tracked by websites other than the one that you’re visiting. They can be accessed by other websites to identify visitors and see where you’ve been. Essentially, they’re tracking you as you move around the web. Now, the scary thing is that we all know about cookies being deprecated. Google have delayed this for a third time. It now is supposedly going to be in early 2025 instead of the end of this year. I thought it was worth checking in on this something to be aware of that will happen at some point. This is specifically in Chrome, where the majority of the share of search is happening. Actually, in other browsers, that’s already happened. The reason it’s delayed is because Google have done a report on this and cited it themselves. The industry developers, regulators, including the CMA, have basically said that they need more time to review evidence from tests and make sure that it’s fully independent and is going to work accurately. There is an alternative. Google have proposed something called the privacy sandbox. That is an alternative to third-party cookies. It has raised quite a lot of criticism because it will favor Google’s ad products. It also is currently lacking the industry accreditation that we really need it to have to make sure it’s secure. Also, they have third-party audits as well. Also, that raises concerns of things around data quality, accuracy, and advertiser risks. What do you do if you’re currently relying on attribution and these cookies are about to disappear? The truth is that friends can still collect and use your first-party cookies. That means cookies that you’ve collected on your own site. It will make accurate digital attribution almost impossible because you can’t see those touch points as people have moved around. Understanding the buyer journey will require other data and measurement techniques. We need to find something else that we can use to accurately represent our digital marketing as well as one of the drivers. Now, as an econometrician, it is no surprise that I’m going to focus on that today. I thought we would start with how M&M is different. We’ve touched on this already, but athletic attribution essentially gives too much credit to the things that happened last and no credit at all to things that happened offline. If you imagine you’ve got a journey here with various offline touch points, they won’t be in the results. Part of our online touch points will be. When we’re looking at our sales, it might be a little bit skewed to what actually happened. What we can see is actually there’s an alternative in econometrics. This includes all influences throughout the journey. It can entangle incremental sales from your online ads versus sales caused by other things. As an example, what impact does the weather have? What impact does our TV, print, out of home advertising have? Through COVID, obviously, that had a massive impact on people’s shopping behaviours. It’s really important to capture that, as well as things like competitors, seasonality, underlying demand in the industry. There’s also online touch points. Not only your marketing, so various social displays, search, but also perhaps if someone’s received an email nudging them to go onto the website, or if they’re doing various other searching online. All these things will deliver your sale. We need to be able to get to that effect. What I thought I would show you is how actually that works. First up, I thought it would be helpful to clear up some terminology. Marketing people in the UK have traditionally used the term MMM and the term econometrics interchangeably. You might often hear the two and wonder what the difference is between them. To many people, they mean the same thing. To be really precise, market mixed modelling just means using analytics to quantify how marketing affects sales. It often, but not always, uses a typical toolkit called econometrics. That’s something that economists working with all sorts of places, so finance, climate, government, lots of other sectors will use. It’s really well established and has a very clear list of checks and balances that have been taught in universities for decades. MMM can be carried out using different toolkits, many that will come under the heading of data science. They still develop a quantified description of how marketing drives sales. They don’t always report back on other drivers. They can use a range of techniques, some with more transparency and here due diligence checks than others. In what follows, I’ll be talking about the version of MMM that uses econometrics. Okay, so this is the fun bit where I’m going to show you basically what I do in my job as an econometrician. We are going to use the example of an ice cream business. This is a much simplified example, but it really does illustrate how we can untangle what different sales drivers are doing. If you imagine that we’ve got weekly sales of ice cream in Tesco over a given period of time, and you can see that the sales go up and down. We’ve got these spikes and dips. What we’re trying to do is understand why the line goes up and why it goes down. Now, the first step is to chart our sales and some of our explanatory variables change over time. These are the things that we think are driving our sales. We’re going to start off by thinking about availability. You can see here we’ve still got our sales in pink and then we’ve got a green line which shows the number of shops where those ice creams are being sold. At some point during our time period that we’re looking at, the number of shops starts to increase. We’ve got an increasing trend. Now what we can do is put that data set into the model and start to explain our sales. These slides are going to build and essentially what we have is that the top pink line is our sales over time that we saw previously. Then the blue line shows what our model thinks is happening to sales. We call that fitted. Essentially what we want to do is get the pink line and the blue line as close together as possible so that we’re explaining as much of the sales as we can. We start to build out an equation of how important each of these different drives are. Underneath it in the pink we’ve got what we call a residual which is basically the gap between the pink line and the blue line. We want that to be as small as possible so that we’re explaining lots and lots of those sales. We also want it to be random so it goes up and down at different points. That means that we can be certain we’re not missing anything and everything’s been captured in the model. We’re going to keep adding to this. This next chart shows how another key driver changes over time and that’s temperature. Obviously we know for ice cream when it’s hot there are a lot more sales, a lot more demand. We can see that there’s a real seasonal pattern to temperature. It goes up in the summer and it’s a little bit cooler in the winter. Actually when we add this in it really closely follows the shape of our sales and helps to improve the model fit a lot. We can see that the pink line and the blue line are much closer together and our residuals have got a lot smaller as well. The next thing that we would want to add in is price. Obviously we know if you’re in the supermarket and you’re looking at the shelves one of the things that you’re going to have a look at is what promotions might be on and how much each of those ice creams are going to cost. It’s super important when we think about this in the business. Again we can see that’s really helped to improve the model fit and our residuals are getting smaller and smaller. Finally the model confirms that advertising is working. By adding in marketing we can see that actually we’ve got even closer to our sales figure and we’re now expanding most of those wiggles, trends and blips in sales. Our residuals are going up and down randomly so statistically that’s a big tick and our two lines are really close together. Now obviously this is a massively simplified example. In reality you would have tens and tens of variables in your model to explain what’s happening and lots of granular detail but it shows you how we go through this iterative process of adding more and more things in and making sure that they fit to get to total sales as best we can. How it works in summary is that we’re looking for those relationships between a given KPI and the drivers of that KPI. The chart on the left is a scatter chart between sales and availability to say the number of dealers and shops. What we’ve done is drawn a straight line through those scatter plots and produce an estimate for the slope of that line. How much does that line go up and down? If you imagine we’re doing this in lots of dimensions at one time what we can do is start to pull all those results together. In this example one additional dealer brings on average nine more sales each week and everything else in the model would remain equal. This rule of thumb is really useful for decision making so that we can go back to a business and say if you change a driver by one unit then we forecast that this thing will happen to sales. Now there’s lots of different reasons for why different people might choose to use LMS and econometrics and it depends who you are in the team and in the business that you’re in. If we start in the top there of the diagram people in the C-suite will often choose it for three reasons. Firstly it can carry the effect of the economy and market trends on the business. This is often really helpful in knowing what the future might look like and building up those big picture strategies. It can also reveal optimal pricing because it gives an accurate read on price elasticity of demand and from that a picture of whether it’s possible to increase prices or not. Finally for the C-suite it resolves debates and conflicts over which departments will get resources. For example it can resolve whether it’s best to spend on advertising or on price promotions. Now if we go to the next one down in the blue we can see for marketing directors they often choose LMS because they need to make the business case for marketing and econometrics can really deliver those findings on optimal budget size and also the danger of spending too little if there are cuts coming. It also shows how to get the most from the budget by choosing media channels that have the highest return on investment. LMS is the only way to accurately compare paybacks from different channels because we can include that offline and offline mix. Finally it does all of this in a way that adds up to what finance sees so it’s credible and I can’t over estimate how important that is when we actually are trying to get finance on board. Then the bottom row in orange is for the day-to-day teams who really benefit from LMS because it provides that complete picture of the effect of advertising with all the different nuances that we know we need to capture. It will include the effect of advertising across a portfolio of products so we can include halo effects say if you’ve got an ad that features one product but it’s also going to benefit another and cannibalization effects so where the ad benefits one product but it’s costly and so it might be stealing from a sister product. It’s uniquely able to quantify longer term effects on advertising because we can see sales that come from people remembering ads after the campaign is over and I’m going to come on to explain a little bit more about how we do this. Finally it can entangle how those online and offline ads are working together so we know that TV ads will often make someone search for your product and so search and TV actually work really harmoniously together and become more efficient when they’re both on it. Now if you think this sounds good and you might be thinking about commissioning LMM have a think about who you might want to pitch this to and the sorts of arguments that you could put them to or where you think it would be most useful for the teams to focus on and we’ll start to think about actually how you might go about setting it up. Now this comes back to the point that I just made about the decay rates from different channels and this is a question that we get a lot around actually how do what different media channels are doing when they all have such different jobs. Some media channels are there to drive demand and build a brand and some channels are there to harvest demand and drive conversion and so we need to test those different media channels in different ways. One way is to look for media choices that do the targeting, content and call to action criteria differently but also what we can do is then look at the effects in the model. On the left hand side we’ve got the sales effect of spending on building your brand by the number of weeks since airing. Say we have media in weeks two and weeks five what we can see is that the decay from the advertising lasts several weeks afterwards where people will remember that they’ve seen your ad but actually the sale won’t arrive for a fair few weeks after it’s aired. This is super important for building demand over a long time, building up your base sales and making sure that there’s enough demand that’s been built ready for your harvesting channels to go and capture. On the right hand side we’ve then got some of those activating or harvesting channels where you spend a week two and five and it causes a huge spike in sales in those weeks but actually it decays very quickly and so these two things are equally as important but they’re doing very different jobs. Now we know from lots of different analysis including R which is a database with econometric studies from six different providers. We’ve collected lots of hundreds of studies to see actually what do those results look like and you can see which media channels are doing which job. Typically things like paid search, Facebook, Instagram, DITV and radio are really good at that activation job where they are harvesting the demand that’s been built and then things like Brand TV, Sponsorship, YouTube, online video, Press Upvote, all of these channels are really good at brand building and they will decay over a much longer period of time and so if you’re thinking about which channels to use in terms of which job you’re trying to achieve and hopefully it gives you some ideas in terms of where the benchmarks are. Okay so we’ve gone through a lot of theory, let’s have a think about practical tips for getting started. I thought that we could focus on three really key things that apply for a range of budgets and categories so hopefully depending on who you are you can find something in this that you can take away. Number one to use metrics wisely, number two to build a balanced measurement framework and number three using models that make it really easy. Let’s jump into number one using metrics wisely. There’s a quote here from lovely Rachel Chapman who’s the ex-head of marketing at Santander and she said that when we launched the value of Ant & Dec we did some very simple stuff which demonstrated the uplift it had on our most profitable product and elsewhere and what she means by this is that actually the most simple things are the most convincing. If a bump in your sales when your campaign is on it’s really easy to understand them by Intu. On this chart we can see online sales from a really profitable product in the pink and you can see that there are really clear spikes when that campaign spend is on and then you can see the decay actually as it bleeps out as well. This would be really easy to show someone in the team and go actually look clearly it’s working and some of the best metrics are three. We’ve seen all of these work really well to indicate a campaign is working well so if you imagine that you’ve got your brand advertising and you’ve got a funnel through to sales, we love a funnel, awareness, in-market awareness, consideration, intent, there are various different things that you can look at each stage to be able to say actually I think this is working at driving this metric. For example for in-market awareness, paid search click-through rate, new customer visits, page views or your share of brand search, all really easy to track. Through to consideration if people are putting your products in wish lists or they’re making inspiration pages, they’re visiting specific product pages or your dwell time’s increased then actually it’s likely that you’re increasing in consideration for your brand and then for intent things like people adding things to their cart, checking out visits, right through to sale where obviously we can see conversions, revenue and also sometimes cancellations and refunds is an interesting thing to have a look at here as well. Now all of this could be data if you’re an econ brand we have online sales data that you’d be able to get access to and track when your campaign is live. The other thing to mention here is that there are a few things that you can do to rule out other drivers that will help to make your argument more concrete and so the first one is to rule out seasonality. If you can look at your sales this year versus sales over the last several years and say during this period there’s an underlying seasonality that we can strip out then you’ll know that the increase during the campaign isn’t because it’s a particularly seasonal moment and you’ve just timed it really well but actually it’s because of the campaign itself. That’s number one. Number two is to look at your market share, rule out other category drivers. For example if you can see that the month before you’ve got a certain share and during your activity increases and then afterwards it stays a little bit higher it’s likely that that’s been driven by your campaign whereas if your market share remains stable then that’s a bit harder to prove. Then our third one, rule out other things that are because by because. For example you want to be able to say it couldn’t be because our competitors have gone quiet, we’ve reduced the price, maybe the website’s improved, you’ve launched a new product, maybe more availability or you have new packaging. The point here is if you can and if you need to run these A B tests then try not to change other things at the same time. Okay on to number two, building a balanced measurement framework. Now what’s really important here is that although I think monometrics is absolutely brilliant I appreciate that it’s not always the first thing that you might start with and there are lots of other things that you can do and include in your measurement framework to make sure that you can capture everything effectively and also for the budget that you have as well. Now some of these approaches are better than others and what’s really important is that they work really nicely together. For example if you can see that media effect bump in sales that we’ve just seen, big easy tick, looking at year-on-year comparisons, have your sales increased year-on-year, looking at movement and share search, looking at what founder investor VC thinks, automated econometrics around them, what your in-house data scientist is doing, media agency econometrics or loss of contribution. Now these are all good things but we would say do them with care. The ones in pink are the things I think are really good for a recommended approach. Go to a reputable provider for econometrics, have a look at pre- and post-campaign tracking dips, look at one region or one city blitz versus control. This is a real classic one, AB regional testing works really nicely for much smaller budgets and make a really clear statistical case about how something’s worked. Then the other thing is to identify the right metrics to watch And then the other thing is to identify the right metrics to watch when you’re on air. As we’ve just seen some of those things that you can monitor for free can really help. The important thing to know for now is that it’s a way of getting an estimate of what your ads are doing for incremental sales and that means for driving growth which is super important. Once you’ve done that I want you to think about what some tips could be to get the most out of your analytics. If you do decide that you wanted to go down the m-man route and actually how would you start and how would you go about that? The first one here is to make the case. You’ve got to go to the team and say actually I need some budget for this, how are you going to make the case that this is important for the business and think about the questions that you might start to answer. Write a really good brief so that when you’re sending this to different providers they know exactly what you’re looking for, what questions you want to answer, what objectives are, what metrics you look at, what data might be available. You also probably want to have a think about whether to in-house it or outsource it and depending on the team that you have available to you and your budget might give you different answers. Select really great suppliers, ask them loads of really difficult questions, make sure they’re up for the job and this will make your life so much easier if you choose the right team to work with and then with them co-create a set of questions to answer. Go to the team and say these are my exam questions, these are the things I need help with and if they’re good they should be able to come back to you and say look these are our ideas as well, what about this thing and this thing. The thing with this is that you’ll probably go around circles here where you add something and then you tweak it and change it and you can do that as much as you like until you feel comfortable to proceed. Part of this also will be thinking about whether you want it to be people-based or automated and again you will get very different experiences depending which one of those that you choose. If you want to work with a people-focused business you will chat to everyone in your team and make sure that you get results that people are fully brought into and usable then that people-based approach is for you. If you want something that’s automated, you’ve got all your data set up in an API, you don’t need the people part and you’re confident that’s okay then perhaps a more automated approach would be. Then the last one is to manage the process and outputs. What’s really important is that the results get used and don’t just get put in a drawer and never seen again. Think about actually when you get your results back what are you going to do with them and how are you going to integrate those results into a day-to-day planning making sure that it’s as useful and valuable as possible. The other tip here is only use attributed sales for tactical tweaks within media channels and not for allocating budget across channels. The thing here is that it is safe to use digital attribution in some cases. Questions around within display which website should I use, within social which objective, within a search perhaps which keyword, making really quick decisions and moving small amounts of spend between those channels can be really helpful but don’t use it for choosing which channels to put in the plan because remember only some are going to be captured, how much in each of those channels, your paybacks and things like return on investment or CPA and then anything for finance or the board or your investors. Have a think about if you are using attribution what are you going to use it for to make sure it’s really safe. Okay and then number three using models that make it easy. I’m going to leave you with a gift which hopefully is always fun. There’s a link here to our recovery budget planner and it’s a tool that we built where you can wall gain versus quite frankly shape your economy for free. You can go in, input some numbers about your business, so things like your revenue, your margin, the category that you’re in and then you can play with different scenarios of the economy and say if this happens what would I expect happen to my market share and then how do I think my marketing will perform as well. Go and have a play with it, it’s super fun and it’s all built on lots and lots of data from real businesses and statistically sound sources. There’s lots of information there about where it comes from. It will also give you an estimate of what you’ll get back for your spend which can be super helpful. When you’re thinking about scenario planning and actually versus your competitors what your share of voice is going to be if you don’t have really complicated measurement scopes set up but you need to make decisions then this can be a really nice starting point to hopefully get going. Okay the last thing for me is that if you’re interested in this and you think you’d like to learn more we offer courses as part of Magicworks. Scaling up works is really helpful if you sell online and you’re trying to get your brand versus performance mix right. Unlocking that new phase of growth and getting out that performance plateau that we often see and then Dataworks if you want to run down a marketing data and how to use it to diagnose marketing and sell those really good strategies in. There’s a lot more around MMM, data analysis, measurement in general but also telling stories of data and how you can get buy-in from the teams as well. There’s a link on here, I’ll also pop it in the chat and there’s a QR code there as well from the beginning. If you go on here you can download slides from today and there’s also lots more informed courses which are starting in autumn 2024. Please do get in touch if you’re interested we would love to have you. Okay that’s it for me, if you would like to connect then my LinkedIn is there, it would be lovely to connect with you and I think now I’m going to pass back to Joy

Speaker 1: for some questions. Absolutely that was fabulous Joy, oh my goodness like we’ve got Kim here in the chat saying such a clear presentation style and so helpful. Thank you Joy and I actually took a photo of one because I didn’t want to lose this comment from Agnes here who said I’ve seen many MMM studies for the brands I’ve worked on but this is the best explanation I’ve seen. Done. Thank you very much, I’ve really enjoyed it. Thank you very much, super useful and like I say this as a lay person so this specific topic actually like really instructive as well for to be able to follow through so thank you, like endlessly useful. I’ve actually also been writing down questions as I’ve gone because I’m like I need to find this out so I can see that we’ve got 10 open questions presently but folks if you have any more do pop them in the Q&A feature not the chat feature please because we’ll lose them in the chat. Let’s get going with some questions Joy if you don’t mind them being fired your way. One of the first things that you said in the model for you had the different stages of commissioning MMM and sort of as a thing and then the first stage was make the case and so I was curious like what are some of the example cases that you’ve heard from folks when they’ve been like okay we need to make the case or they come to you or the most regular things that people would ask when they look to start on an MMM journey?

Speaker 2: Yes it’s a really good question. I would say most people come to us with marketing as the core objective so they want to be able to prove the incremental sales that their marketing is driving and have a number so that you can go back to the business and say look I’m spending this much money but look how much it’s returning and it’s statistically robust and it actually gives you a bit of evidence to say it is profitable and is paying back. That tends to be the starting point and then from there it can get more granular so media channels and campaigns and right down to keyword level if you want to. What is nice is that to get the marketing bit right you have to capture everything else correctly so things like the weather, seasonality, competitors, price and so all these other business learnings come as part of it. What people tend to do is say I can prove that marketing and then look at all this bonus stuff because it’s easier to sell that into the team rather than go or look at all this stuff that comes out of the model.

Speaker 1: Okay cool and so you speak about those variables and I’m fascinated in that because that requires it seems like it requires real thinking work to sort of sit down and go okay these are the variables say for example ice cream sales it’s like oh the weather might impact it oh and the building work outside our shop might impact it. How do you practically start to find out what those variables are because they’re presumably you’re layering things on things on

Speaker 2: things for quite a long time? Yes to be honest that’s the fun bit that’s why it’s good fun. Depending on what category you’re in there are probably things that will apply to most brands in that category. If you’re an FMCG then it tends to be as you say price distribution and a product’s vulnerability versus if you’re a b2b client then actually thinking about the events that you’ve got on in the emails that you’re sending out and so there are some things that will be really obvious straight away. The way that we try and understand that is by doing something called stakeholder interviews. We go to the brands and say actually can we chat to people in your team not just in marketing but in all areas of business and get them to tell us all the little nuggets of information that are in their heads about I know these things are happening because I work in the business and it means that we can collect all the data that we need to. Also it means their questions are fed into the analysis so they’re fully bought into that journey and it means that when you get the results out everyone’s bought into it and everything’s captured because you’re right there’s so many different like little nuanced things depending on who you’re working with. Yes definitely the front bit. Nice but that makes perfect

Speaker 1: sense. How often do you subsequently revisit those things because it strikes me that this could appear as a report that sits on a desk or it could be something that is ongoing for the duration of time. How often do you sort of go back to those variables and say is that still relevant and indeed how often do you sort of repeat the process to make sure that it’s I appreciate there’s not a definitive answer you do it 10 times and then you’re done but like what regularity do people typically most revisit these things? It’s a bit of a complex answer but

Speaker 2: it really varies. Some clients will do it once a year and that’s okay and some clients will do it more regularly so perhaps every quarter. There are a few things so statistically the relationship between your sales and any driver will be quite robust because these models are built over at least three years worth of data so the strength of the relationship between those two things doesn’t change very much but what you’re doing as a business might be changing so you might start a different marketing strategy or you might change your price and so what we tend to say is that the models will be robust but if you make big changes then that helps to show what cadence you should run it in. Then also things like forecasting tools can be super helpful so we can take your results and build you a tool that says if you run different scenarios then this is what will happen to sales and so people can put different marketing laydowns in or if the economy changes or if my share of voice changes and actually sometimes that’s sufficient for people to say without needing to update the whole model that scenario planning is really what people are trying to get out of it. It’s quite unique depending on who you are and we would

Speaker 1: choose the right solution for you. Nice that makes perfect sense, thank you. Your answer there feeds in two more questions around types of businesses as you started to speak through it there. A couple of questions in the Q&A and I’m trying to find the name so I can credit them. We’ve got someone like Victoria who asks how can we make this work over a long cycle for B2B when our deals take approximately two years to get signed off? There were other folks who asked about B2B specifically. Is there any nuance involved in B2B versus B2C and so on?

Speaker 2: Yes no you’re absolutely right it is different and it’s much easier to see the spike in sales if you are working with the products that people will buy day to day. Whereas you’re right B2B obviously the purchase cycle is much longer. It’s the same with things like cars and big furniture items people don’t buy very often but it’s still econometrics that works and the way that we do that is by putting the really long tails on things. I touched on with media we know that there’ll be a decay rate from advertising and the way that we test for how long that decay rate should be is statistically actually which one is the best fit and so if we were looking at your data we could say you’ve run these different campaigns even as a B2B business but your decay rate might be months rather than weeks for ice cream sales. Then also working with the team again to say actually how long do you think that lag is between something airing and that lead coming in. It’s a case of almost if you imagine stretching it out so you get that long tail and for the several years it takes a lead to come in and then also if there are specific things that about your business we can move things to match that. Nice that makes perfect

Speaker 1: sense and I guess that speaks to an observation which I think is implied by a question that actually comes in from anonymous who sort of speaks about the differences between large and small businesses and how they may approach MMM. The question says I’m getting the feeling MMM is more for larger businesses or agencies with massive spends. The practical tips are great but how applicable is this for smaller businesses or agencies? Is it just the case of scaling it down? I guess the question is like how applicable is this for smaller businesses with potentially smaller budgets? If I layer in an extra question which may be a bit cheeky but like how would you begin to approach it if you were a smaller business to sort of start exploring this?

Speaker 2: Yes cool so we actually work with loads of businesses who are doing their first MMM and what I would say is if you are spending anywhere from about a million pounds a year on advertising so that’s not a small number but you don’t need to be spending tens of millions then it can work really well and what we can do is do things like data audits and say statistically are we going to be able to pick up an effect before you totally agree to the full project. There are things that you can do to go yes I feel really confident that I’m spending enough and then also we would tailor the questions to you. Often for smaller businesses things like I may only be using one or two media channels where should I go next or if I’m running the two together how are they working effectively so that you can make the case to the business to say can I have a bit more money or probably for smaller businesses making that money work as hard as possible because every pound that you spend is really going to be effective and so we would make sure that the questions that we’re answering are right for you and then also things like the scope and the cadence as well so you don’t need to be doing huge scopes with tens of models and really complicated stuff you can start with one model of sales and make sure that is still going to be really impactful but it’s still fairly straightforward to begin with and then building up from there. I would say it’s definitely not just big clients we work lots of smaller clients but still find it probably as helpful because it’s that first

Speaker 1: ever case of going look my marketing is working. Yes 100% and there’s some broader principles that I think you can take from your presentation as well so I love that slide that you had which showed the different data sources that you may choose to include or not and even if you’re in that sort of sub 1 million category there was a lesson there for me which I really took which was like just thinking about those data sources and trusting more than one so I would be guilty of this and maybe this is just me but I would be as the solo marketing manager sort of sitting in one of those smaller companies and I’ll go okay Google Analytics says xyz and so that’s the thing that I’m going to report and so one of the lessons that I took from your presentation was like okay yes let’s start layering these nuance and this interesting bit of data so even though it’s a bit of homespun sort of hacky mmm like I found that really useful actually for my own analysis and how we may choose to report things because it’s dead easy to get a Google Analytics sort of print out or whatever and say this is the truth and the broader principle which you illustrated for us I think is endlessly useful actually in terms of sort of providing more in-depth nuanced reporting to folks who may or may not need it so I really appreciated that as well just as a as a point to raise so I was curious about so one of my questions that I’ve written down is how your role has changed with technology so we’re seeing stuff like chap GPT come in and Mark Ritson wrote a article recently about synthetic data which is basically for those who haven’t heard about and I’m probably butchering this but data that is generated by AI taking in all bunches of sort of variables and plotting it out in certain ways that sort of presents in a very similar way to what you would see elsewhere and I was just interested as we’re sort of like now progressing into that sort of AI era and stuff like that how your role and how you interact with MMM sort of has changed over these sort of past 6, 12, 18 months or so? Yes that’s a really good

Speaker 2: question and it has changed a lot in some areas I would say so if we think about what we’re doing in two parts so one of it is collecting data and processing it and getting ready for modelling and then the other part is building the model itself so that data part has become a lot easier to automate it’s a lot faster and we can write scripts and process things a lot more quickly we can automate how the data’s being downloaded so it can fit into that really nicely and all of that helps to speed things up which is great and the bit that I would say hasn’t changed for me and I hope doesn’t change is the building the model because exactly as you said earlier Jo there are so many nuances that you need to build into it to make it a really good model where it doesn’t just it’s not just statistically robust but it makes intuitive sense so that if you show someone in a business they’ll go yes that feels right and it makes sense and so we still build all of our models ourselves like people are literally building them because you have to otherwise what you get out is nonsense and there are some platforms who are doing it in a much more automated way Meta and Google obviously have automated versions now and the problem with those is that actually you don’t have that person there sense checking it and chatting to the people in that team going oh have you got some data to feed into this and what do you think of that and so that would be the warning I guess to say it can be brilliant but just make sure that you’ve got some one sense checking it and doing that people bit that I think is super important. Spot on and that’s just really interesting

Speaker 1: as well which I guess is a wider point around the presentation but when you reference specific companies and specific ecosystems that they are a source of the truth and they’re not the truth and they may or may not be depending on your activity but just to sort of have that sense check along the way seems really sensible but again sample set one isn’t necessarily the type of thing I would do and so I’m grateful for you illuminating that because I think that’s really useful. There’s a question here about historical data and you’ve touched upon this already and mentioned forecasting but it did strike me that feels like quite an important point as part of the MMM so it’s probably worth revisiting. Based on what you presented today a lot of MMM activity is based on the last three years. How do you start to think about say for example Covid which is a fairly extraordinary event and sort of modelling it into this historical data for companies who may or may not be doing their first reports right now for example? Yes Covid was fun.

Speaker 2: The good thing is that Google has some really brilliant mobility data which basically said in different clubs so people going to the shops, going to parks, going out, staying at home, how much are people moving around which is great and that was time series so it would change over time and then also we would make what we call dummy variables so they’re either on or they’re off for when lockdowns happened or when eat out to help happened or easing of restrictions and so you have some data that you can use to explain Covid and then for clients what we would say to them is that we have to go back further than three years because if you do three years of just your Covid period it’s not really like any sort of future that hopefully we’ll get back to and it also doesn’t really reflect anything before Covid so what we would then try and do is ask people for four, five, six years of data if they had it available and then by looking at that longer time period you can control for Covid with the data that we’ve collected and then say actually this is what pre-Covid and post-Covid look like. Sitting here in 2024 it’s quite nice because we’re now far enough out of it that it’s just about the model periods it’s basically you have to control for it with data and then make your the time period that you’re looking at a bit

Speaker 1: longer. I love that and I think again it speaks to that human touch as part of all of this which is really sensible but not as again coming as a lay person not necessarily the most intuitive thing straight off the bat so thank you for speaking through that. You mentioned third-party cookies so this is a question from Sean in the Q&A and you mentioned that this is going to be quite an important thing that’s going to happen so Sean’s asking how are you going to deal with third-party cookies going away in MMM? It’s a very direct way of doing it but I think maybe that’s the simplest way to ask the question how are you beginning to think about this internally at Magic Numbers or as an economist industry in general?

Speaker 2: Yes the nice thing with econometrics is that we don’t use any tracking data so the way that it works is we basically look at those spikes and dips over time and then look at the strength of the relationship between any driver and the sales so we’re not looking at this person did this thing and then bought an item or anything that’s tracked it’s specifically what’s the probability that thing has driven sales and so luckily for us it actually is okay that third party cookies are going because we can carry on as we are but we work with loads of clients who do have attribution as part of their frameworks and it won’t it just won’t work as well anymore because you can’t track people through that journey and so I think generally helping our clients with what their frameworks look like and attribution might not be able to do that same job and helping them with those quick decisions that they were perhaps using it for before so things like we were saying those scenario planning tools or making sure that they have the model updates that they need to we’re lucky because it’s okay for us.

Speaker 1: Two final questions to finish off I think. The first one I’m going to ask and then I’m going to ask the second one because it’s always unfair when people ask for favorite examples of things because it puts you on the spot. I’d love your favorite example of how an MMM sort of analysis that you’ve done has actually changed how a company does things because I’d be really interested to sort of see about the impact of these pieces of research but one of the questions in the Q&A who has again come from anonymous so anonymous is like properly dominating the charts today in terms of the Q&A is asking for further reading or resources that folks could explore. In fact at this point I can put the QR code up for your course again. If we take it as given that is one of them what are the other places for further reading and resources that folks may be able to check out if they’re interested in finding out more? Yes that’s a

Speaker 2: fab question. We do lots and lots of articles and speaking things and so if you have a look on Magic Numbers on LinkedIn and Magicworks as well we post all those articles and bits of research and our founder Dr Grace Pye does lots and lots of these sorts of pieces of analysis and talks at various conferences so that’s always good have a look on there. Again all the articles get published on our website things like marketing week as well goes on there and then aside from that as you say the course if you’re interested in data and you’re in a business where you’re trying to scale up and go from that performance to brand marketing then get involved we would love to have you.

Speaker 1: Nice perfect cool and so just to loop back to the first question which is your example of a behavior change in a company do you have one just to finish us off? Yes you’re right it is a tough

Speaker 2: question isn’t it? I am currently working with a TV network and they have lots of different TV channels and we have worked really closely with them to basically evaluate marketing for each of the channels and then help them to optimize their budget across them so across a network in a way that without fundamentals it’s really tricky to say this is the halo across different parts of the business and this is how different channels and campaigns and messaging is working and so going to that point earlier around then being able to take you back to the c-suite and go this is the value of marketing this is how you can optimize your budget make it work really hard and also then give people the evidence that they need to be able to put that behind the decisions that they’re making. I think it’s really helped them to put the numbers that they needed to behind that so yes it’s a really fun one and nice as you say to help try

Speaker 1: help make a difference to it to what they do. Fabulous I love it. Joy it’s been such a pleasure thank you for taking the time today like as I say I’ve said numerous times over today it’s not a topic I know an awful lot about and so to have this explained to us and explored and just presented as an option and even to take out some principles even if you’re below that million pound mark I think you can take a bunch of principles away from today but if you’re above that then I think you’ve had a really good option presented to you as well so thank you for speaking through that it was really fabulous and just really interesting to explore so thank you for taking the time today and thank you to everyone for your great questions as well today we got through a bunch of them and not all of them so we’re going to take those away and try and get those answered as well but in the meantime I just want to say a big thank you we’re back next week with Thierry Nguettiere who’s going to be speaking about how to tell the story of your data to the rest of your team and senior management so you’ve got these beautiful reports how do you then present this to the rest of your organization so folks buy in understand it and Thierry is a force of nature is a really good guy and I think you’ll really enjoy that if you come back next week all that said joy it was a real pleasure a big thank you for all of our sponsors today but especially exclaimer our featured sponsor for this week for sponsoring today’s session we will be back next week so we’ll see you very soon thank you joy and thank you everyone for

Speaker 2: taking the time thanks so much Thierry love to see you all see you bye