ForthRight Women.
work with us

By submitting this form, you are consenting to receive marketing emails from: . You can revoke your consent to receive emails at any time by using the SafeUnsubscribe® link, found at the bottom of every email. Emails are serviced by Constant Contact
work with us

Classics: Remaining Resilient – The Art of “Test-and-Learn”: Show Notes & Transcript

Post | Sep 26, 2023

Welcome back to Marketing Smarts! From brand-building and marketing veterans Anne Candido and April Martini (that’s us) comes a podcast committed to cutting through all the confusing marketing BS so you can actually understand how to take action and change your business today. We deep-dive into topics most would gloss-over, infusing real-world examples from our combined 35+ years of corporate and agency experience. We tell it how it is so whether you are just starting out or have been in business awhile, you have the Marketing Smarts to immediately impact your business.

In this episode, we’re talking the art of test-and-learn. Listen to the episode on Apple Podcasts, Spotify, Google Podcasts, and your other favorite podcast spots – follow and leave a 5-star review if you’re exercising your Marketing Smarts!

  • Episode Summary & Player
  • Show Notes
  • Marketing Smarts Summary
  • Transcript

Marketing Smarts: Remaining Resilient – The Art of “Test-and-Learn”

Many are struggling to figure out how to make decisions about their business amidst an uncertain economy. In this episode, we discuss how you can stay resilient by adopting “Test-and-Learn” practices. This is the cycle of doing, learning, refining, and repeating. It allows for fast-cycle iteration, saving money and time, and it can be applied to all aspects of your business. We talk about specific practices and examples based on our experience so you can immediately integrate Testing-and-Learning into your business today. This episode covers everything from iteration to refining. Here’s a small sample of what you will hear in this episode:

  • How do you use the art of test-and-learn?
  • Why should you leverage an expert panel?
  • When should you make small bets?
  • How do you A/B test?
  • What are some examples of how to effectively leverage testing-and-learning?
  • Do you have to run testing-and-learning sequentially or can you do it in parallel?
  • How do you know when you’re done testing-and-learning?
  • Are there times when it does not make sense to test-and-learn?

And as always, if you need help in building your Marketing Smarts, don’t hesitate to reach out to us at: ForthRight-People.com.

Check out the episode, show notes, and transcript below:

Show Notes

What is Marketing Smarts?

From brand-building and marketing veterans Anne Candido and April Martini comes a podcast committed to cutting through all the confusing marketing BS so you can actually understand how to take action and change your business today. They deep-dive into topics most would gloss-over, infusing real-world examples from their combined 35+ years of corporate and agency experience. They tell it how it is so whether you are just starting out or have been in business awhile, you have the Marketing Smarts to immediately impact your business.

How do I exercise my Marketing Smarts?

Thanks for listening to Marketing Smarts. Get in touch here to become a savvier marketer. 

Transcript

Please note: this transcript is not 100% accurate.

Anne Candido 0:02
This is Marketing Smarts – a podcast committed to helping you become a savvier marketing leader, no matter your level. In each episode, we will dive into a relevant topic or challenge that marketing leaders are currently facing. We will also give you practical tools and applications that will help you put what you learn into practice today. And if you miss anything, don’t worry, we put worksheets on our website that summarize the key points. Now, let’s get to it. Welcome to Marketing Smarts. I’m Anne Candido, and I am April martini, and today is another Marketing Smarts Classics: Remaining Resilient – The Art of Test-and-Learn. When we originally aired this episode, we were just emerging from COVID. And still facing an uncertain economy when many businesses especially small and mid sized ones, were afraid to invest in their branding and marketing efforts without a high level of certainty. Well, guess what, little more than two years later, and we are facing yet another uncertain economy and business again are struggling to decide how to right size these efforts. What we know is that branding and marketing are the lifeblood of growth and scale. So we need to do something in order for our businesses to thrive. By adopting the practice of testing and learning you can action with confidence and pivot more nimbly and cost effectively. So enjoy this episode, and continue persevering. And today, we’re gonna talk about how to stay resilient in this uncertain economy. By adopting test and learn practices, you hear April, and I talk about this a lot as a way to test the waters without fully committing. But we’re finding that it still feels a bit nebulous to some, so we want to put some structure behind it.

April Martini 1:38
And if that’s not enough, we also are seeing an unhealthy pattern of behavior with some of our small and midsize businesses, which has really been exacerbated with the uncertainty of the economy. And what we’re seeing very specifically, is when you’re doing well, you don’t feel like you have to invest in things like marketing and branding, because you’re doing well. And then when you’re not doing well, you don’t feel like you have the money to spend to do any branding and marketing. And this can really have a disastrous effect on top line growth, both in the short term. And in the long term. Yeah, in a goal really is to flatten out the peaks and valleys with a more strategic approach of this process of doing learning, refining and repeating. And this is the practice of testing and learning really, and it can be applied to all aspects of your business to fast cycle decisions

Anne Candido 2:31
and a more economical way. But since this is a marketing and branding podcast, we’re going to focus there, but we’ll highlight other places you can practice testing and learning as well. And yeah, just as a little aside to this, again, is big brand knowledge that we’re really breaking down for you. So this is a lot of the conversations and a lot of the processes and a lot of the execution that we that we have done in the big brand world in our previous lives. That’s April and I. So without further ado, let’s get into the four approaches for testing and learning. So the first one is leveraging an expert panel for quick iterative learning. April, I’ll let you take this one.

April Martini 3:10
Yes. So I will jump in here. And you guys have heard me say it before on the show. And you know, if you know us as as your partners, I am not a fan of huge quant studies. And that is not what we are talking about here. So the panel we’re talking about is really just a small number of folks that represent your target consumer, both psychographically and demographically. And what we mean by that is you may have defined a certain age group, or you know, a segment of the population like moms with kids two to four, you know, that kind of thing. But then along with that, you need to have psychographic knowledge of your consumer so that you understand how to emotionally connect with them. So what things do they like? What things do they love? How do they feel about certain products? How do they feel when they’re disappointed by certain products, you know, all of these emotional things play into it. But if you get it right, with just a handful of consumers, that is really all you need in order to use this test and learn strategy. So you can source these people really from anywhere again, when we used to do the big heavy quant studies, like Dan mentioned in the intro with all the big brands and needing to prove things out with numbers. Now that we have especially things like social media channels or sites like Survey Monkey, I mean, you can pop together a quick 10 question survey posted on your social or recruit for people before and then send them the survey. And you can get a vast number of answers but also insightful information from those people if you craft the survey right. And you can really leverage this panel however you want to. There are additional ways outside of just you know, serving you can hop on the phone with them for 30 Minutes, if it’s packaging at shelf you want to look at, you can pop into the store with them and look, if you want to understand their product usage at home, not right now during COVID. But generally speaking, you can pop in for a visit and kind of observe. I mean, again, we used to do these big long studies where it would be hours and days in fields at, you know, multiple homes, multiple cities, multiple consumers per day. And that’s not what we’re talking about here. Because we what we believe is that if you are a true tester and learner, you build up your craft of being able to listen really well for the type of information you need. And if you’re not asking your consumers to solve things for you, but rather, you’re looking for information from them to help guide you, that’s really the purpose of this. And therefore that panel, however, again, you define, it can just be that handful of people. And the other part about is it can rotate too, right. So you can pick up a group of people and stick with them for a period of time, if that makes sense. You can pick a few different groups of people if you have different segments, or you can just like I said, with the survey, monkey approach, put it out there, see which people match up to your psychographics and your demographics of your target, and then use whoever pops up in whatever order and go about it that way. So tons and tons of options. You know, not to belabor the point, but you really just need a few

Anne Candido 6:18
array. And I think what you’re really articulating here is, even though there’s lots of options is to keep it very simple, absolute right? So that it’s easy to do this iterative learning because if if you select these people well, and you screen for them, well, you can go back and iterate with them, in order to optimize something in a very quick timeframe, without having to go through all of the screening. And in all of that whole process. Again, we’re waiting a long time in order to feel to study and wait for other results to come back. So keep that in mind. In just other some other ways to that we’ve seen other people do this. And actually we do it frankly, ourselves. We do it with our own clients. So we’ll test and learn with our expert plan, oh clients and say, Hey, let us try this on you. Let’s try this, like this, whatever approach or want to do it, whether it’s a new coaching style, or whether or not it’s a new positioning, or anything like that. So we’ll use a even a expert panel of our clients as ways to kind of test and learn. So this can be defined in in multiple different ways. But the key is to keep it simple. Know what you’re asking for and be really good listener.

April Martini 7:31
Yeah. And people are really apt to help you if they’re interested and fans of whatever you’re talking about. There used to be the belief that you had an incentivized with a whole bunch of money. And that’s not the case, either. We feel like sometimes you can just send some product, or a $10 Starbucks card, if you feel inclined, a lot of people will do it for free, or a very nominal fee, or item, you know, such as your product, and you don’t have to worry then too much about all those incentives.

Anne Candido 7:57
Exactly. All right. Another approach for testing and learning is to make small bets first. And this is especially true if you’re considering any kind of strategic partnership or sponsorship to boost your marketing. And really anything that’s going to require a big and I say big and air quotes spend or investment. And we might put big and air quotes because big is variable for everybody. So it’s whatever big looks like for you. So for example, you might find that the NFL would might be a good place for you to position your product or your service. Instead of jumping into a full NFL sponsorship, you may want to try like contacting a local player who your consumers may appreciate even if they’re not as popular nationally. So way more economical approach and you can see if they work as an ambassador first for your brand, can you extend your message can you talk authentically about your message to your consumers react to this in a in a positive way. And then if they do, then you can scale from there. As an example and we live in Cincinnati. Anthony Muñoz is a Hall of Famer to this day, that man could still command around. That’s true, and it doesn’t. It doesn’t require a lot of money in order to get him to speak. And he still is very popular. So um, that is like one example of somebody who is like a local celebrity, if you will, that can really help potentially drive a meaningful brand connection that it can help you sense out whether or not an NFL partnership may be a good one for you going forward. Another example, if you’re in a in a b-2-b or even if you’re somewhat in a b-2-c and and you rely on industry events or conventions as a way to connect in and share what’s up with you or business, you might want to try like a smaller presence to begin with, instead of like going all in on the big presence. I know it’s really tempting to go on and on the big presence because it’s pretty magnetic. send to you know, have all that real estate dedicated to you and, and it’s very impressive it for your, your competition to you’re making the same with your competition, but it can be very expensive. So you know, you want to like feel your way into these and see if actually your consumer, your customers actually there is this a good way to network with them even to the point where you might want to just like attend one of the sponsor dinners, for example, and just, you know, network and see if they’re even there. And this is, this is a good venue for engaging them. Another example is, you know, before you jump into a brick and mortar store, try selling your goods at your local farmers market. I’ve had several of my friends who are looking to start restaurants and businesses do this. And you can really try out to see if like people are attracted to your brand, are they attracted to your product that they continue to come back in this is, like I said, good for like food or goods in general. And then just round us out with a little bit of a marketing example is you might want to leverage influencers, as an example, to be storytellers for your brand message. But instead of leaning into a full influencer campaign, you may want to start like a few well sourced influencers, and just track their call to actions to see if you get traction. And if you do, you can then extend the type of influence or you’re using to to get to even more consumers. And if it doesn’t work, then you have to determine is influencers the right strategy for your marketing. Or, you know, maybe it’s just those types of influencers. And this is the same approach for any marketing channel, like from billboards to advertising. So those are just some examples of how to make small bets. But you can kind of see how you take like the bigger piece and you kind of scale it down. But some opportunities may not be able to be scaled down, you need to really look for this too. So for example, NASCAR and when I was working on NASCAR, for Tide, this was one thing that was very, very clear. NASCAR fans will support anybody or any brand that keeps their driver on the track. But it’s a visibility play, if you’re not showing up on a regular basis, then you’re not going to get the traction with them. And you’re not going to get the loyalty that is anticipated with that kind of a relationship. So one time or two times is may not actually be a small bet that actually scales to whether or not this will be a bigger bet for you. So you may have to try a different test and learn approach for this.

April Martini 12:28
Yeah. And I think the point that we’re making here, right is, these items that Anne just outlined, minus the NASCAR example, are things that you you hear and you think that’s going to cost too much money. And that’s what we hear from our clients a lot of times is, you know, you’re crazy, there’s no way we can afford these things. And so the point of this is yes, you know, definitely make your small bets first, but it’s also shedding some light on the fact that actually you can do these things in a more economical way that lines up with the spend that you have available for your business, to then yes, test and learn, but also just be there in general. And I think you know, the NASCAR is a good contrast, because that highlights one word just definitely will not work. So in addition to knowing your consumer and your target, like we talked about in the first point here, this is about knowing enough about the approach you’re taking to kind of inherently understand whether or not that would work for you. So NASCAR is an example right and just said, you have to be there all the time. That’s a big ticket item. If you’re not there, then that’s an easy way to opt out of that one. But it kind of shows you the balance of some of these big things really are possible because they’re not as big as you think they’re really big things. Don’t you dare try to place a small bet on those because it just won’t work.

Anne Candido 13:53
All right, the third approach for testing-and-learning, you can A/B test within a defined space.

April Martini 13:58
And I will tell you, when the creative world got a hold of this one, we had a fantastic time because it meant we could test our ideas and some of the clients ideas a B

Anne Candido 14:08
test forever. You see the f test.

April Martini 14:12
For those of you that don’t know what this is, first of all, this is the idea of picking two ads or sometimes you do three or four depending on what you’re talking about. And putting them into market and seeing which ones resonate the most. And then that becomes your lead your next a from which to optimize and then do the same thing over again, when you have were out with whatever that ad is. This is popular with like Google Ads or pre roll, you can put your money and divide it a little bit and then not spend as much right. So you might take your total amount, divide it in half for a period of time. Give it some time in market, see how it’s working. Pick the one that’s working well run that till it has were out, start again, pick a couple more and just keep iterating that way. This works with social posts too. So we’ve talked a lot about use channel and use of message and how those have to work together for your brand to attract the right people, you can test here too. So it might be different imagery or a different message or a different call to action. For example, if you want to, you know, get people to respond or user generated content, you can ask for those types of things. And then again, see which one performs the best. And in this case, you know, it might give you an inkling of future posts. So if you see that really rich, gorgeous, beautiful imagery works versus black and white, you might say, oh, that’s something that really entices my consumer, my target to want to engage with me. So you know, we need to have that richness throughout, or that just works better to feature my product, because it really brings it to life versus the black and white toning it down. And then similarly, you know, with the calls to action, asking people to do something and seeing what happens, and then measuring, of course, all of that against your KPIs. So we’ve talked a ton and all of our episodes about having KPIs, you identify those first and then just see which one tracks better. You can also do this, before you’ve inputted anything out into the marketplace, and this was the traditional practice, in advertising before the digital space came out, quite frankly. So we would design different ads or billboards or you know, like pages of a magazine. And we would put them in front of consumers. And we’d be you know, in the back room with the two way mirror watching what happens. But that was the way to test out what were their reactions to those types of things. I think the important thing here, if you’re going to test and learn in total is make sure that you’re controlling the variables that could sway someone one way or another. So for example, what the what I just said about the print ads, you know, we would never show a billboard, and then a print ad, and then a digital ad. With separate concepts, you want to show the same type of ad for each, because that then limits people sway toward like, Oh, I’m more used to seeing billboards. So I like that one better, or I love my fancy magazines. And that looks like a chair out of there. So I like that one better. So you want to make sure that you control those types of variables. But also, it’s really important to make sure that they’re in context when you are testing and learning so that people can give you an authentic reaction that’s going to be meaningful, and react to kind of like in the space it’s going to be in. So with that example, again, if you’re going to put a billboard out, you want to show people in Billboard context on a sheet of paper perhaps or on the screen, what it’s going to look like, so they can internalize, okay, when I see that among the other billboards, as I’m driving down the highway, will I internalize this message versus just putting black and white print on a piece of eight and a half by 11 paper and saying, Do you like this message without the context, it can be really hard to test and learn. But this ABX approach in total is one we really, really like, we’ve told you how big of fans we are of testing and learning. But I think this lets you get stuff into market, especially with how digitally inclined we are these days and get some really fast answers about what sways people one way or another and then just keep pumping stuff into the marketplace. And the beauty of it really is that it only exists there for a couple of seconds. Couple of minutes. Right? So it’s not like people now expect that you’re, you know, digital ad that pops up next to their, you know, shopping cart on Amazon is the most beautifully, perfectly, you know, exquisitely written ad they’ve ever seen. They’re just like, oh, okay, that’s an ad from these guys. And then you kind of move on. So all of that to say another way to test and learn, really a fan of this from the creative space I come from and more of the agency world, and a place where you can get quick answers.

Anne Candido 18:44
Yeah, and I love this one when we were arguing about design, right? So this was one where we’re just like, let’s just put them both out and test them, then you know, because it’s what I said, and we love it. It’s the best way of getting over those internal discussions and those internal like, just the do loops that could go on forever. It’s just to say, let’s just put them both out and take a look and see what does better. And actually, it should be part of any Google based ad strategy, any SEO SEM strategy, which is this optimization of dollars based on what’s working the best and that all can be done almost automatically now. So it should always be part of that strategy to optimize your ads, your ad spend based on what ads are performing the best. So great technique for avoiding internal arguments about who subjectively or objectively thinks it’s better or not better. So also good

April Martini 19:37
for fueling some fun internal competition among creative strategists and account people just

Anne Candido 19:42
as an aside, oh, yeah, I can imagine that would be the case. All right. Our fourth approach for testing and learning is what we call limited exposure testing. All right, in the olden days, we call these tests market olden days, the audit trail, it is kind of like the olden days. But now It can take many forms, but you can still think about it as test marketing a proposition, even if you’re not actually doing like a actual test market. So for example, here, you can test your new idea by offering it for a limited time. So this could be like a new menu item, a new bundle of product offerings, a new service is whatever you want to do in order just to get a reaction to your idea, but you only do it for a limited time just to kind of limit the exposure there and not mix it, make sure that you’re setting the expectation that may not be there all the time. And this is kind of like an opportunity for people to give you feedback. You can also do it through a defined location. So you can limit your exposure through a defined location. So this is, like more typical test market strategy. Like if you do a new product, or a new service launch, or even a new marketing or advertising approach, you might want to do it in a like smaller representative market or a single store or a focused industry before scaling bigger.

April Martini 20:56
And one of the things we used to love to do is, you know, follow the curve of early adopters versus people that come along at a later date. So it wasn’t, you know, uncommon for us to play stuff like in California, if it was going to be a new super healthy trend that was coming out to see if it would be adopted and then indicate whether that curve would continue or if it was heavy fashion, placing it in New York City and seeing if it took off. So that way, you catch those people who are the trend setters or like we said the early adopters, and then you can kind of gauge whether that will continue as you roll it out in other markets.

Anne Candido 21:32
Yeah, the only thing you just have to be a little bit careful here is making sure you don’t have location bias, rare enough. So and that’s what you really have to consider from your scalability standpoint, is that if you’re designing something in you intentionally, they believe it’ll do well in LA, and you put it out in LA and it does well in LA, but then you try to launch it in like a Cincinnati, because it’s a different consumer, a different culture, a different expectation, a different vibe, a may not work so well. So just be conscientious of what you’re designing for to make sure you don’t get that location bias, that may give you a false positive or actually a false negative as well. Yep, fair enough. And then a third way that you could do this limited exposure is through a group size, okay, so it’s a defined group size. And this feels a little bit like expert panel, but you’re probably not recruiting these folks. So for example, we have colleagues who test new webinar coaching strategies with a group of college students before launching in primetime. So again, not anybody that the recruiting but just a small defined group with which that they are testing this just to get the feedback and refine it and optimize it before scaling it into a bigger format or into a more people. And what’s really important here is that you have to set your KPIs in order to monitor results, and so that a broken record for us. And we’re gonna continue to say that, initially, you may have like, no benchmarks, or very little benchmarks, but you can always define what success looks like, even if it’s within the confines of what you understand about your business, and about your resourcing and your staffing right now. Just make sure you have some metric to align success or failure against.

April Martini 23:13
Yeah, exactly. I think, you know, we talk about metrics all the time, like ansat. And we’ll keep talking about them. But if you don’t have anything to work against, it makes it really gray and really hard to continue along a linear path towards some kind of goal. And therefore you’re you kind of are questioning like, why are we doing this again, what are we doing? What are we working toward? And without those goals and metrics, no matter how soft or kind of inconsequential they are to the bigger picture, it’s great to have that is kind of your guiding force to have gates that you’re trying to get to as you move along.

Anne Candido 23:46
Yeah, exactly. Unless, if you don’t do that everything feels very arbitrary. And decisions aren’t objective. And we’re actually going to get to that in a in the trenches question, which is our next section. So this is where we give real world examples. And we’ll talk about specific industries and situations, but they really do have broad application for anyone. So we will hit some of those where we feel specifically that the broader application applies, but but listen for the insights, and we think you’ll be able to apply them to your own businesses. And this first one is going to be fun. All right, it’s our first and utterances question. Can you give examples of how you’ve effectively leveraged testing and learning? So we’ll just hit it right on the sweet spot from the very beginning here. Okay. So if one or both could provide two examples here, we’re going to provide one that actually scaled and did well, and one that actually didn’t scale and got killed. So just so you could kind of see hillsides get killed. I mean, you got to just be clear, I mean, indecisive. And we’re gonna get to that later, too. You learn a lot through failure. Yes, yes, you do. And it might be a little bit more generally reference but I think you guys are gonna get the, the insights you’re still so So, when I was in fabricare, we had an expert panel that we incentivize to give us real quick data on anything that we happen to be doing at the moment. So it was more of like our gut check. And we did this before we really like took it down into any further development or design, it was really just to understand, hey, is this going to resonate one with our loyal users, let’s call martide users. So we don’t actually make them mad. Because once you have an established brand, you’ve got to really consider your loyal users and make sure that whatever you plan to do isn’t going to alienate them to an excessive extent. But also, we had some other folks in there as well, that we could then kind of segment out and just see if something that we were testing maybe from like, you were targeting young people, or maybe we were targeting people who didn’t use it all the time, and only use it occasionally, that we could kind of sense if we were going to if our new positioning or ad copy or product may hit them as well. So a lot of this was done. During surveys, we used to use a lot of surveys, because like I said, was like really quick and rapid turnaround. Some of it was giving them the product, letting them use a product for a week, and then getting that, that feedback back with a diary, if you will. But again, what it allowed us to do was to really figure out quickly, are these things resonating? are they hitting? Is there any like big like O’s that we should know, right up from the very front? And again, it was anything from like, ad copy, to messaging to product ideas, to do taglines, I mean, you name it, we would put it in there. So that’s one way that we were able to leverage an extra panel on fabric care to get really quick by cycle responses that we were able to iterate upon on a regular basis in order to get that knowledge really upfront. Before we even like took anything further from that. APR, do you have a example of one that worked well, for you?

April Martini 27:01
Yeah, absolutely. So I probably shouldn’t say who it is. But I worked for one of the local brewers in town, and they wanted to put on an online survey. And the inclination, like I have mentioned before, was to go a bit more traditional and recruit people and kind of get that, you know, proof of 100 people, so it was viable, and all of that kind of stuff from a representative standpoint. And so we recommended actually to do one of the examples that we referenced before, which is to have people answer a short survey online, list their name, and then we would select the people that fit the target and reach back out to them, and then have them fill out the survey. And there was a concern on whether or not we would get enough responses. So we said, Okay, well, let’s go ahead and offer an incentive. What do we want to offer? You know, and it was like, what do we have to pay each person? And we’re like, no, no, no, no, no, you know, we’re talking about the beer category, you know, let Don’t you have like packages or something that you get out. So we’re like, Okay, well put together, you know, a pack of, you know, I think it was like six beers and then swag from this brewery. And you would not have believed I mean, it worked perfectly for a few reasons. One, it’s beer. Two, it was COVID. And it COVID had just hit. So everyone was home and on their computers, people were drinking a lot more than normal. And so it was very top of mind. And so we just got like, I mean, an overwhelming amount of responses. And then not only that, but I said can we at least put on the bottom, if any of these people would be willing to talk to us one on one, because you never know, we might get some hits. And you know, the real enthusiast are the ones that we want to hear from. So they should self select themselves by saying, Yeah, I’d be more than happy to talk to you. So not only did we get plenty of survey results, not only did we hit the target, we were able to get a handful of people that were willing to have like a 1520 minute conversation, which added really nice contexts to the survey responses we got because we could probe answers and frame the questions through the Okay, the survey Got it 70%. But there’s a few more things we want to dig into. And the whole thing went really well. And the the prize, the incentive was a single package, I think we had almost 200 responses. And so all we had to do was put out that one gift basket which, you know, retails at, I don’t know, 120 bucks, and essentially cost nothing to the company that’s putting it out there. And so that was one where, you know, Pat, myself on the back, I felt like that did really, really well. It was super fun that people were very engaged, that does not work for every category. But in this instance, it was kind of like perfect time perfect place perfect category.

Anne Candido 29:44
So what did they do with that information? And once they got it, how are they able to scale based on that?

April Martini 29:50
Yeah, so you know, for all of you that are beer fans out there, you know that the category especially craft beer have exploded, and it was already a really, you know, fragmented category For that even happened, I think the status something like it’s gone from 2000 to over 10,000 breweries in the country in a matter of six or eight years or something like that. So this Brewer was kind of trying to figure out where to innovate, where to pull back and also kind of redefine and reposition themselves in the category because they were finding that they were failing against some of the competition. And so what we were able to do was actually identify that they were not positioned correctly, and that their target was not actually their target. So chasing those, you know, true craft beer, like on the edge, like I was saying earlier adopters, before, those people that are willing to try anything and they’re, you know, they’re out there just to get the latest thing and all of that that was not this brand, this brand, appreciate it innovation wanted to feel cool drinking the beer, but really wasn’t that far out there. So the SKU proliferation that had happened with this brand was not necessary. The necessary thing was like things like the seasonal that everyone looked forward to, or a new seasonal coming out, because seasons queued that something was coming, you know, so we were able to pull back, really cleaned up distribution significantly streamline the portfolio, and redefine the meaning of the brand, to where those loyalists that reached out to us were like this, this is all we want. And now the brands out there again, and it’s been redefined. And there’s quite a lot of fewer, fewer skews than before. And they’re back on track.

Anne Candido 31:34
Yeah, that’s interesting. So there’s two things I wanted to point out on that one is, it’s a blend rate of what’s what people would call consumer research and testing and learning rate. So yes, consumer research is an element of testing and learning because you’re really trying to understand where your consumer is coming from in order to address what option you should go forward with. But what I thought was so beautiful about what you did was Yes, you did you start with your consumer research and did indeed your survey, you might have been testing a couple of different concepts, maybe in aibee, aibee format, or just understanding if we did or didn’t do, which is another AB format. But then really then selecting the people who are extremely interested to be on an expert panel potentially going forward, right? Yep. So is a way of kind of cascading into, or actually screening for people who could potentially be experts that are going to help you iterate going forward, until you could have gone back to that group and then say, hey, is this better? You know, is

April Martini 32:32
this this positioning better? So I thought, that’s a really good example of testing and learning. Yeah, expert consumer research. Yeah. And I mean, that I should have said that you’re exactly right, that some of those folks did continue. So since the information was out there, and we had talked to them before, and the response was, yeah, call us anytime. Then there was and still is, but was that ability to go in and say, here’s some packaging concepts, for example, or, you know, we’re thinking and leaning into this space, does that make sense to you, and all that kind of stuff. So yeah, that really good point that you can continue to leverage and test and learn with them as you go through the process and even beyond that particular project?

Anne Candido 33:08
Yep. I think that’s a great example. All right, so let’s talk about some examples that got killed. All right. So I’ll start. So if you guys are out there, and you any of your communications or PR efforts, you’re going to feel my pain because the bane of our industry is that it is hard to prove ROI on communications and public relations, and marketing in general, and marketing in general, but specifically everybody picks on communications and public relations, okay? And the reason why is because it’s very hard to quantify and in to localize and to define, where it’s a single variable piece, because a lot of times communications and PR is part of a broader program. And that that all when that works together, it lifts the brand all together. So, um, so, what we were trying to do is we were trying to quantify an ROI for communications and PR. So yes, we were trying to solve all the industries problems by doing one little like localized test that always works. Yeah, it always works. Okay, so um, yeah, I’m trying to set yourselves up for failure. I’m trying to keep a little bit of resentment out of my voice if you guys can’t tell but anyway um, we were we we decided to do this like, like a localized test. So a little bit like a test market where we employed a lot of communication tactics from influencers to social but specifically we’re like looking to see if the influencers are going to raise the bar with regards to sales. Now, a couple problems with this is one is when you’re especially even if you’re in a localized market, and you’re trying to analyze sales, especially on a big brand, it takes a lot to lift the big brand, and you guys are probably out there all going to die and yeah, I don’t We disagree. But um, it what happened was that we got a false negative actually, and saying that, hey, when you do this activity, the influencers didn’t actually lift the sails. But when we actually dug deeper, what we realized was that there wasn’t enough magnitude in order to drive that. And the problem is in Wow, he’s no problem as much as this the reality of influencers specifically is that unless you’re going very ultra local, like local publications, or something like that influencers still tend to be a national based influencing mechanism, right. So to say, hey, we’re gonna concentrate all this onto one like little city or, you know, one little area and hope that it raises the sales in that area is it’s a little, yeah, it’s a little, just not really like, that’s yeah, anyway. Well, not to

April Martini 35:53
mention the spend, right, so you’re spending tons and tons of money on this national brand in total, and then you’re expecting this, like you said, tiny city tiny piece. Because even if it does go up, it becomes a rounding error, depending on the actual or against the actual spent.

Anne Candido 36:09
And that’s what and that’s what’s really hard to tease out. So I think that’s exactly what we were we were seeing here is that we all knew that influencers work. I mean, they do they work, I mean, there’s just no doubt about it, but trying to actually quantify it is very, very difficult. Like we said before, when you can measure call to actions and some other things, but actually to see a sales lift with respect to that it’s very, very difficult, especially if you have a big brand. So that’s an example of something that didn’t quite go so well from a test and learn it was largely due to like localized biases there when we we talked about being careful if you can scale it, but also just to the fact that that’s just not how that channel works and operates to its best. And we were not we were trying to make it do something that it was not naturally designed to do. So you gotta be careful about creating arbitrary or like, non scalable situations, and really trying to, in trying to make something work that’s going against the grain, if you will. Yeah,

April Martini 37:09
exactly. Well, and so mine, I actually took the killed very literally, because this campaign was killed. It just went away after, after this. But um, so for mine, we were testing campaigns with small groups. So like we’ve been talking about throughout this, you know, three concepts, we did exactly what I talked about here, we used, you know, a network of enthusiast that were tapped on a regular basis to come in and chat with us. And we put some creative concepts in front of them. And the first round, just went miserably. I mean, I can’t I can’t even put it any other way. Like, it just did not go over well, not with any of these groups. And so then, what often happens, and unfortunately happened in this situation is, you’re like, well lit, which one did the least bad of the three, which is a terrible place to start from? Right.

Anne Candido 38:05
You had to try to salvage something.

April Martini 38:06
Yes. salvage something, you know. And I mean, the clients right there with us, right? So it was we couldn’t do anything other than to just say, Yeah, this just completely missed the mark. And so instead of going back to the drawing board, and I mean, all the way back to the drawing board, like starting over with the brief, which we have an upcoming episode that will focus on briefing, that would have tremendously helped the situation number one, but that also framed a lot of that episode number two. But in any case, we should have started over because hindsight being 2020, the campaign we were building, and the offering that we were trying to put forth, was not going to be something beneficial to the groups of moms age, I think it was 35 to 55. We were really trying to target more college age kids. And so fundamentally, the research was broken, because I have this team of designers who are the target. And I have this offering, but we have this panel who we didn’t want to alienate the consumers, and we really wanted them to buy it for their teenage sons. And so we ended up with this whole convoluted situation. And we only made it worse, because instead of going back and aligning all that stuff, we just kept trying and trying and trying. And after the third abysmal round of research with these moms, I was like, Alright, guys, like time out, this campaign is dead if we don’t go back and realign and whatever. And actually, we ended up realizing there were things that were better served to go back to the moms and really use them because that was the proven audience and that that lifecycle part in the lifecycle of the brand that was the thing to do, because it really wasn’t as mature as it should have been all of those types of things. But I will tell you a lot of pain, a lot of heartache for those creatives, a lot of anger from the call wasn’t, but it was just like we were iterating for the sake of iterating. And following this test and learn process that wasn’t working, because the whole idea was broken from the beginning.

Anne Candido 40:08
Yeah. So there’s another example of when you get your results, believe them.

April Martini 40:13
Yeah. I couldn’t believe it. Like, I will never forget sitting in my chair, and everyone’s kind of looking around. And you know, to me, it was so obvious, like, we gotta go back to the drawing board, right? It was like, Okay, well, which one did the least bad? And I was like, to be kidding me.

Anne Candido 40:30
I’ve been in that position before, too. That’s real. That’s definitely real. So yeah, if you if you moral, that story is, when you’re testing and learning, you get bad results, it’s real. So believe them and make the right adjustments before continuing to waste your time and money. Yes. Awesome.

April Martini 40:52
I feel like I’m having some PTSD. Yeah.

Anne Candido 40:57
All right, moving on. Do you have to run tests and learning sequentially? Or can they be done in parallel? April? What do you think?

April Martini 41:06
Yeah. So science would say that you want to do things with a single variable in order to keep control and really be able to have pure results that then align? And, you know, the answer is very, very clear and non objectionable. Unfortunately, that’s not how marketing works. And that’s not how the life we live in works. We’re not in a lab, right. So there’s all kinds of things that are influencing all of us all of the time. And no matter the size of your brand, or your company, you should hopefully have multiple touch points and things going on anyway. And this is why it’s so hard to to point to ROI when you’re talking, talking marketing and branding and total, because you can’t really say this one thing, move the needle when there’s so many other things that are going on. So really, we say no harm, no foul and running things parallel as long as you’re doing it intentionally. And that you are not allowing things to kind of come into the evaluation that really aren’t affecting it one way or another. And what I mean by that is, so let’s say you want to get into social media ads, and you also want to test a new package for your product. Those are two things that can be tested at the same time, because on the ad side, you’re more in the campaign space. And that stuff, you can put right out there right away and get some immediate, immediate responses. With packaging, that is a longer term thing, honestly, and you go through several iterations. And then it takes a really long time honestly, to get the product on shelf from when you start with that redesign to when you actually see it in store. So you can run those things you can do your testing and learning, you can have separate projects going on. But the thing that I’ve seen happen is you don’t want some great thing that went really well and social to suddenly find its way into your package. And then completely changed the direction of where you’re going. So if a piece of creative gets, you know, hysterical reaction, and lots of people are engaging, and you have all kinds of comments and whatever, that doesn’t mean that you go then and run, change, and change something that’s so fundamental to your brand, like your package, because one little thing that was out there for a few seconds did really, really well. And I’ve seen brands get really excited and distracted about things like that. And then instead of maintaining their brand essence, and that part of them that really should live for a long time they buy into something fun and engaging that happened over here. And we don’t want to see that happen. So just make sure that the intention of what you’re doing with each of the different projects remains within itself and fulfills or doesn’t those objectives, and then keep going along those kind of more linear paths not to say that you can’t learn things or gather insights or any of that, but just allow each of the channels or items or whatever they are to do what they’re supposed to do in the space they’re in.

Anne Candido 44:13
Yeah, I’ve seen that happen as well. And I think it’s about discipline wrapped salutely it because it can inform one can inform the other lacks hint, and maybe it is something that you learned that you want to apply. But it’s it shouldn’t be a knee jerk reaction, which I think is what you’re getting at, right, which is what we see a lot, which is like, oh, that works. So well. Let’s put it everywhere. And yeah, wait a second, wait a second. Wait a second. Not, you know, just because something worked well on one channel doesn’t mean it’s gonna work well everywhere. And it doesn’t mean we should disrupt everything that we’re doing in order to apply it. And maybe we should, but we need to do the proper due diligence in order to understand that. And I think that what you’re saying is exactly right, is that sometimes those paths cross inadvertently without the discipline, and that causes a lot of turmoil. I also think on the other side, that there is an element of cause Compound effects that can happen to sure that also can lift together so that one plus one equals three factor versus the one plus one equals two. And so when two things come together, and they work together, that creates a more exponential effect. And when those things happen in isolation, that happens a lot in the marketing, going back to my original example, and communications, when you have PR out there, and then you have social that’s supporting and you have influencers, all saying the same thing, the compound effect of that raises the brand, a whole lot more than any one of those things individually, you just have to be very clear about what you’re putting into market so that you know if it’s all things working together, or if it’s one thing that’s kind of caring to to, and sometimes it’s a little hard to tell. So sometimes we you might want to stagger them out, layer them on and just kind of see how they build upon each other. But you also have to be strategic on what leads. So if that’s all very confusing, reach out to us, we can more than happy to help you like work through a marketing strategy, or test and learn strategy for how to do things together in parallel and things to consider. And it’s an art more than a science. So I’ll say that as well.

April Martini 46:04
Yeah, this is one of the ones where I like to say it depends. It depends. Yeah.

Anne Candido 46:09
All right, our next in the trenches question, how do you you know, you are done testing and learning. And this kind of goes back to what we were talking about before, when we’re talking about what works and what doesn’t work is you are done when you have enough confidence to scale it or to kill it. And that’s really your two options. Now, it doesn’t necessarily mean that everything is 100%, perfect in order to have the competence to skill, but you’ve gotten to a point where you know that the risk is worth the reward. Okay. And sometimes you can’t pull the trigger just on your test to learn, you have to do maybe a risk and reward assessment that helps to provide some perspective because it’s not as easy to say, yes, we’re going to do it, and no, we’re not going to do it. There’s resources involved, there’s money involved, there’s time involved, there’s other things going on in your pipeline. So there’s a lot of extenuating circumstances. So sometimes you do need to do that. But you should have at least enough confidence in that risk versus reward assessment should help to push you either in the right direction to go forward or the right direction to kill it. And this is where we’ll say that your KPIs are so critical. And you want to make sure, as we said before, that you actually believe your data, because that is very, very important. And as we’ve said already, too, it takes a subjectivity out of it when people will tend to be passionate one way or the other, and their emotions get involved. When you’re data driven, it helps to make that decision a little bit more clear. Now, where we see people kind of get stuck in this test and learn loop is when you don’t feel like you have enough information to go forward and you don’t have enough information to kill it. Here, we’ll say set a timeframe. So there should be a reasonable amount of timeframe that you would expect to see some level of results. If it’s still kind of vanilla at that timeframe, you probably want to kill it because the chances of it getting to like a chocolate chip ice cream versus just a vanilla ice cream are probably a lot lower, because you just haven’t at that point in time figured out what that level is going to be. So you have to continue to kind of see hey, how much time and how much money and how much we’re going to invest in this in order to kind of keep this going. Versus Okay, let’s kill this. Let’s move on to something else. So if it’s still at that period of time, or it’s still kind of just vanilla, and you can’t decide either way, you probably are going to want to kill it.

April Martini 48:31
Yeah, and this is a bit of an aside. But as Anne was talking, I think it’s definitely something worth noting, even if you end up killing things, or in some cases, especially if you end up killing things, make sure you learn from those things, the doc Word document that was gonna steal my thunder, that was another point. Because, you know, people change positions, people get promoted, people leave the company bosses change, change, objectives, change, the marketplace changes. And it’s so often we’ve seen iterations of didn’t we try this before? And then it’s like, wait a minute, what do you mean, we tried this before, or that doesn’t even happen, that recognition is completely gone. And then the same mistakes are made over and over again, which costs money and time and resources. And there are many things that are just never going to work for your brand. So keep track and make sure that that is somewhere where as people change, there is a record of what worked well and what didn’t and why.

Anne Candido 49:30
Yeah, and just for the record, too. Sometimes, if something didn’t work well in the past, it can work again in the future, that is fair, that will be one of those just sits there and says, Oh, I didn’t work, you know, 10 years ago, that’s not going to work now. Maybe it was before it’s time. So it’s okay to bring these things out, dust them off, reevaluate them. Alright, so that’s my little plug for that.

April Martini 49:53
Yeah, exactly. And, you know, it’s just more that the knowledge is power, you know, it was done before, and you can figure out whether to do it. Can or not? All right,

Anne Candido 50:00
our final in the trenches question Are there times when it does not make sense to test and learn?

April Martini 50:07
Yes, and we have mentioned some of these. So on some, I’ll just be making, you know, points to reiterate, and some are kind of new things. But this first one, when a scaled down version isn’t representative of the full scale. So, you know, I mentioned the different locations and early adopters, and and countered that point with, you know, making sure that you’re not then skewed the other direction, where it’s going to work in one or two markets, but not in general, and you’re trying to create something that’s going to go across the country, for example. So that’s what we’re talking about there, you know, and that idea of false positive or false negative, really making sure that you’re giving it a valid chance to succeed, even though you’re doing that scaled down version. So just be really clear on that. And if it isn’t, then we would say testing and learning really isn’t gonna teach you anything about that, just like the influencer example, where it’s like, everyone knows influencers work. We didn’t set it up, right, you know, so now we’re going against this like generalization that everybody knows right and, and shaking your head. And I feel like I’m bringing up more pain. So but like, sorry, I won’t say anything more about that one. And then the second example, we talked about NASCAR, and this is, you know, that example of you have to be there all the time in order to be rewarded with people’s loyalty. And if you can’t afford that, or that’s not where you want to put all your eggs, or whatever the case might be, just don’t do that. Because one or two times isn’t going to gain you any kind of lift when it is that type of situation. So just opt out of that example, or totally lean all into it. Oh, yeah. I mean, it’s you. Yeah, you did. But if you have a small budget, going all in on NASCAR is not an option, because you’re all in is only a couple points of visibility, as well. And then this next one, which is one that drives me, completely crazy as a runner, is where you show up, but you show up in such a tiny way with so many others in a place where it really doesn’t matter to people. And then you wonder why you don’t get results. So race T shirts are one of those for me, I mean, I have a drawer, really probably three drawers full of these T shirts, with all different kinds of logos on the back. I mean, I guess somebody at one point thought, Oh, it’s a good idea, because you’re looking at the runners in front of you. So you’re seeing the backs and then they’re wearing their shirts all day and our names on there. And so why wouldn’t they want to connect, while the people selling the ad space for those T shirts, we’ll pack on as many logos first of all, as they possibly can. So some of them get so small, you can’t even read them. But also, when I’m running a race, I’m concentrating on not dying on the, on the dhania, racetrack, right, or when’s the next water thing or you know what, at my pace, whatever the case might be, and then the people on the sidelines are like on the sidelines, right? So they’re not really paying attention to those advertisements, either. And the same goes for things like banners out events and sponsorships that are, you know, you can pay 100 bucks, and you’ll show up with 85 of your closest friends on this banner, all of those types of things. If you’re doing it because you believe in the cause we have our cause marketing episode, or you’re doing it because you want to support the efforts that are going on. That is all amazing. And kudos to everyone that does that, especially in the racing community. But that’s not going to be a place where you’re going to test and learn effectively, because you’re probably not going to have much of an effect when it comes to selling new business or more product or whatever the case might be. And then the final thing we’ll say is an ad kind of alluded to this already, if you do decide to go all in, make sure you go all in and part of that which and loves to talk about is a genius you with is asking a genius set a genius, wow, I will tell you, I’ve seen this woman work. And it’s asking for everything you want, instead of just accepting whatever the other party is willing to give you. So back to the t shirt example, right? I just said, Don’t do that, that doesn’t make any sense. But if you could negotiate that you get to be, you know, the biggest logo on the finish line above everybody else, and you’re lined up with, you know, whatever your brand is, and the flying pig marathon and you know, 100,000 people or whatever are going to see that and they’re going to announce your company over the loudspeaker. And they’re going to hand out tchotchkes and you’re going to sponsor the bar down the street and everyone gets a free beer. And, you know, that’s the kind of stuff we’re talking about. And those are the types of things that people do in fact, remember, and so there’s a lot of limitations when you go to negotiate this type of stuff. It’s like you get no 12 times before you get Yes. And because everyone’s trying to be fair and you know, not do too much work and stay within the parameters of what everybody knows and all those kinds of things. But ask you know, the worst you can receive as a no but if you’re really persistent and you are willing to go all layin. Usually you can meet somewhere in the middle and get

Anne Candido 55:03
your way. Yeah, I think that’s really good advice and genius. Advice. Scholler genius in our head just inflated. Yeah, sure, it certainly did. And I think what you said is exactly right on it is all about leveraging your investment to its fullest is that, you know, if you’re going to invest, make sure that it’s right for your brand, not just what people are telling you, they’re willing to give. Because you’re right, like I say, maybe 50% of the time, if I made the request, I would get the yes, just because it adds value. I mean, so that’s the whole key of this is that you had to add something of value to. And all those things that April said are different ways of adding value. So really think about that. If you’re going to go all in it’s it’s all the different ways you can leverage your investment, both on the platform and then off the platform. So great points there. All right. And our third and final segment is generally a real world example of a brand who’s doing as well or not well, but it’s actually super hard to decipher, like, what is it Tesla and what is it from the outside? So it’s hard to pick a brand on this, which is why we really spend a lot of time on that first in the trenches question about what worked and what are actually what scaled and what got killed. And all the examples that we provide you along the way. So we suggest you rewind this episode back to that and really listen to those and internalize those. And like I said, if you guys are still struggling with this, and you need more strategies, or you need more customized strategies for how to test and learn, reach out to us. I mean, it’s definitely a practice that we appreciate and we actually use on a regular basis ourselves. And also don’t forget that we have worksheets to help you get started on all of our topics for all of our episodes. And with that, go exercise your Marketing Smarts!

April Martini 56:51
Still need help in growing your Marketing Smarts? Contact us through our website: ForthRight-People.com. We can help you become a savvier marketer through coaching or training you and your team or doing the work on your behalf. Please also help us grow the podcast by rating and reviewing on your player of choice and sharing with at least one person. Now, go show off your Marketing Smarts!