Subscribe on

Enjoy this episode? Leave us a review.

All Episodes

Episode 177

Does AI Have Taste? with Catrina McAuliffe

AI and human evaluators rated the same 20 ad concepts within 0.2 points of each other, across nearly every dimension. So can a machine really judge great creative?

This week, Elena and Rob are joined by SVP Brand Strategy Catrina McAuliffe to unpack Marketing Architects' new seven-criteria framework for creative quality. Catrina shares how the framework was built, how it held up against 500 marketing and creative pros, and where AI still falls short. Marketers will get a clearer sense of when to trust AI's creative feedback, and when human debate still wins.

This video is hosted on YouTube and requires cookie consent to display.

Topics Covered

• [01:00] The seven-criteria framework for creative quality

• [09:00] Testing the framework against 500 marketing and creative pros

• [11:00] AI and human scores landed within 0.2 of each other

• [13:00] Does AI push ideas toward safe, average creative?

• [15:00] How the framework differs from ScriptSooth pretesting

• [16:00] Using AI to sharpen writers, not replace them

• [22:00] Where pretesting and the framework each fit in

Resources:

2026 Marketing Architects Research Press Release

Today's Hosts

Elena Jasper

CMO

Rob DeMars

Chief Misfit

Catrina McAuliffe

SVP Brand Strategy

Transcript

Elena: I'm Elena Jasper. I'm on the marketing team here at Marketing Architects, and I'm joined by my co-host, Rob DeMars, the chief product architect at Misfits & Machines.

Rob: Hello, Elena.

Elena: Hello, and we're joined by Catrina McAuliffe, our SVP of brand strategy at Marketing Architects.

Catrina: Hey, Elena, and thanks for having me on.

Elena: Thank you for coming back.

Rob: Oh yeah, Catrina in the hizzy.

Elena: We're back with our thoughts on some recent marketing news, always trying to root our opinions in data, research, and what drives business results.

And today, we are talking about one of the toughest things to pin down in advertising, what actually makes creative quality. Um, but I'm going to kick us off, as I always do, with some research, and I'm going to use one of our own press releases today, which went out this week. It's titled "Marketing Architects Research Gives Humans and AI a Shared Language for Creative Quality." Traditionally, marketers decide whether an ad concept is any good by, you know, maybe gut feel, internal debate, which rarely produces a consistent standard. So we set out to try to create one. We developed a framework that breaks creative quality in TV advertising into seven criteria: concept originality, narrative surprise and emotional punch, visual freshness, dialogue and voice distinctiveness, craft cohesion, brand fit, risk and reward, and cultural resonance.

Then we commissioned a national study of 500 marketing and creative professionals to test a bigger question. With the right prompts, could AI use this framework to evaluate creative the way that a human might? And the results were surprisingly positive. We're going to talk about that a bit today with Catrina, who led a lot of this thinking. She's joining us to talk about it.

But before we get into it, Rob, I wanted to kick us off with sort of a broader question for you. Why do you think creative quality is so difficult for marketers to define?

Rob: I think you have to go one step further down the bunny hole of that question and go, you know, what is creative and what is quality? These are two words alone that are about as old as David Ogilvy's pipe. When you look at, like, okay, let's just talk about creative for a minute on a broader level.

I think Moulin Rouge is the greatest movie ever made, and I think Monty Python is not funny. And that can cause a lot of debate and a lot of crazy, and that's just —

Catrina: That is debatable, Rob.

Rob: I've literally been uninvited to dinner because of that belief in —

Catrina: I am not surprised. Monty Python is hilarious.

Rob: Right, and that's just creative. And then you talk about quality, you know, right? And now what are we talking, production value? Are we talking about the strategic insight? So I am just really glad that God invented smart people like Catrina, who can help us discern this at a deeper level. So I'm excited to hear more about this research that they've been doing.

Elena: Perfect. Well, let's get into it. Catrina, thanks for joining us.

I wanted to start with the idea of creating a creative evaluation framework in general. Why do you think it's important that brands have a consistent way to evaluate their creative quality?

Catrina: Well, I'm super excited to be here too, Elena, and actually this is a subject very close to my heart. First of all, I'm kind of with Rob in a lot of ways, is that I doubt we'll ever have a totally consistent way to evaluate creative, cause it's like clothes and we all have a subjective opinion. Rob, I probably wouldn't wear his T-shirt. He probably wouldn't wear mine. It's like, whatever. Tastes are different and they always will be, and I don't think we can ignore that at all. But I do think we can have more constructive discussions about creative rather than I hate it, I love it, when we understand the elements that make creative more effective at building brands in addition to driving short-term performance.

And this framework, I think importantly, what a framework does is actually gives us a common language to discuss it. It's particularly true when we're working on brands where we're way outside the target audience, but it's also true for where we think we are in the target audience, but we're not necessarily representative of that target audience.

So if we're only judging work based on taste, beliefs, experience, rather than more constructive feedback, it's really difficult. Like how do we know where we're going to make it better? And so that's what I think the joys of having a framework and having a common language is all about.

Elena: Well, let's talk about the framework. Can you walk us through those seven criteria and how you got to them?

Catrina: We actually used a lot of work from Binet and Field, System1, a whole ton of other research organizations, academic research, WARC, and a whole lot of other people to try and inform and develop this framework. It's based on our beliefs about what makes advertising effective and builds performance both long term and short term, and also what makes an idea strong.

Clearly, and I think we all feel this more than we know it, is that the best work does make us feel something. It's emotional at its core. But importantly, how it's executed, that idea has to be relevant, and there has to be some craft to it. I mean, Rob was talking at the beginning about what are we talking about? Quality? What... So there's different things that go into it, you know? And then it has to be fresh, and it has to be brand distinct and well-executed. I think Adam Morgan's recent work on the dangers of dull has opened our eyes to a whole new importance to fresh breaking through.

We've all read the stats on how many ads break through, so it's got to really lean into a cultural truth as well. So, a lot of things to take into consideration. It's based on our beliefs about what makes an idea strong. Talked about it has to feel something, and it's got to be emotional at its core. And then the executional idea has to be relevant to the target, built on an insight which ideally leans into a cultural truth. It has to be fresh and brand distinct and well executed.

Adam Morgan's recent work on the dangers of dull, I think, opened our eyes to a huge new dimension on why we have to be fresh. The other thing is we wanted something that was usable, so we collaborated really closely with our creative teams and Steve, our CCO, to base the framework not only on academic research, but also the day-to-day practicalities of judging work, whether or not it's right to present to a client and appropriate for their brand and on strategy.

So there's a lot of different layers to do it. I think in terms of the framework itself, it actually builds on seven elements, all of which have pretty detailed descriptors in terms of training how people use this, what we meant by it, and then how we wanted respondents in this research, both AI and human, to be able to evaluate spots by it.

So as you mentioned, it was concept originality, which was clarified in terms of not familiar, hasn't been used in a previous genre. It has some caveats around it. Second one is narrative surprise and emotional punch, like to try and see if it's that break through. Is it actually getting there? And the descriptor there talks about memorability and interest throughout the spot. It doesn't go where you expected it to, basically. Visual freshness in terms of like a different way of looking at things. Dialogue and voice distinctiveness, and that's not just the actual voiceover, but it's the voice with which the brand is speaking.

Craft cohesion, which is how much it actually all fits together and tells a complete story, and whether or not the pieces all belong in that place. Then brand fit, and the descriptor here was very much about risk and reward, like thinking about how big a risk is it worth for the bold step you might be taking. And then the last one was cultural reach, which is very topical for today. Does it actually fit in? And those elements and those detailed narratives were all run through our creative teams and some outside creative before we actually ever exposed them to anybody.

Elena: That's probably why a lot of people don't judge things by a framework. Your gut feels a lot faster than that, to be honest.

Catrina: Yeah.

Elena: But it's important that all that stuff is based on research, and that's amazing that you got there.

So then, as you mentioned, you got this framework, then you decided to commission a study to see if AI could take this framework and use it to judge creative quality. What surprised you the most when you compared those AI ratings to human ratings?

Catrina: Yeah. Well, for the cynics in the audience like me, just to backtrack a little, the study was actually conducted amongst 500 experienced advertising professionals, half drawn from the marketing community and half from the advertising creative community. All had at least five years' experience, and most had significantly more than that and were —

Rob: These all synthetic humans?

Catrina: No, no, no, they were real humans. We did for the AI one, we did some synthetic, but whatever.

Rob: Trying to be ironic.

Catrina: I know. Um, and they've all had experience being involved in the development of TV ideas. So this was a robust sample, so we can't question that. And then, how we did this: we asked them to evaluate 20 spots across the seven framework elements. We used a sliding scale. They didn't necessarily even have to mumble and jumble with words. They could kind of go up strongly or not, as it were, along the scale. Yeah, and then we asked them for an overall scale.

So what surprised me? Three main surprises, and first one after that little buildup is, because you know I was cynical about the results, I was shocked at how similar — like, literally within .2 of a difference across virtually all dimensions averaged out across the board. I was just a little blown away. And in fact, and I think these are differences, but this one's less surprising to me, there were bigger differences between how men and women judged the ideas versus how men and AI judged ideas.

Elena: Wow.

Catrina: Not saying anything about that, but women were the harsher critics, let's put it that way.

Elena: That's super interesting. So basically you found that they were evaluating the creative in a similar way against all those elements of the framework.

Catrina: Absolutely. Second surprise, the greatest spread of scores happened as one might expect with outlier ideas. Out of these 20 scripts, there were a couple of, two or three, way out there outlier ideas. But there was no consistency between AI and humans in how well or badly these outliers were judged.

So AI did not appear to be more conservative or more creatively supportive of the outlier ideas, which I thought was super interesting bearing in mind what everybody says about AI. And then the spots that generated the greatest spread of discussions did tend to be like bolder, and I think one of the most interesting things coming out of this research is that I think it really demonstrates a need for human discussion in terms of the ability to talk about it.

When you get a spread that big, let's talk about it in terms of risk and reward. So that I thought was a super learning. And then the third one was AI and humans tend to be very consistent in their judgment of spots based on key metrics like idea originality, narrative surprise, and emotional punch.

Those scores matched up on those metrics. But overall, the agency-based creative community was much more concerned about the craft, cohesion, and storytelling within an execution than AI was, and more concerned than their marketing peers. So the story on the differences was in the craft, the voice, the kind of softer stuff. You can probably understand that, Rob, as a creative, right?

Rob: Absolutely.

Elena: Yeah. That's super interesting.

You mentioned this a bit there, but when I hear people talking about AI and creativity, one thing that gets brought up a lot as an objection is that AI is too rational to judge something like creativity and that it will bias us towards the consensus since AI is built off of all of this data, like it's gonna just pick the most average ideas. After doing this research, I'm guessing you'd disagree with that, or what would you say to someone who really holds that as their main objection?

Catrina: I think it's interesting really because this research would actually suggest that AI is not too rational to judge creativity in terms of originality or relevance. But, and my big caveat to this is the more I thought about it, but remember, we gave it a framework to judge things by.

Elena: Right.

Catrina: The framework was grounded in our belief systems and decades of research from the IPA and many others. So left to its own devices, AI may err on the side of safe consensus creative. We don't necessarily have that evidence, you know, and I think particularly after the last couple of years, we're all aware of the amount of trial and error you need to work with AI to get there and in order to get the appropriate solution.

So I'd say with the guardrails we gave it and the instruction, AI can get there.

Rob: I think you really hit the nail on the head, Catrina. I would definitely echo that I think some people have this perspective that AI is this magic catch-all that has all of the answers. But as marketers, the real opportunity is how are we creating AI moats based on the frameworks and the knowledge and the expertise —

Catrina: I totally agree.

Rob: We build AI products? It's not like you just throw something into Claude and it magically can evaluate your creative. You have to build and be a builder, be a thinker, and create truly innovative AI products, which is what you guys just did.

Elena: No, that makes a lot of sense that if the takeaway from this is not I should just go and plug my TV commercial script into ChatGPT and see what it says. Definitely not, not the takeaway. So —

Any time that kind of AI creativity is brought up, there's this tension, which is understandable, around how will something like this impact the human experience of creating the work, kind of impact the craft? Catrina, we've been using this internally. How have you seen this tool kind of empower our writers and change their day-to-day work?

Catrina: I think it's amazing. Rob's giving me a look that's like, "Oh my gosh, she didn't really say that." But I do because I actually think this research project suggests AI needs writers and strategists working in partnership. It, you know, used with appropriate prompts, AI can be a tool to stretch creative thinking and develop ideas.

It does not seem capable at this point to replace human discussion or debate, but you can debate with AI, and you can actually make it stronger. I think that's particularly true when evaluating bolder and potentially more polarizing ideas. There will always be subjectivity. It's not going to go away, and I happen to believe Monty Python is hilarious.

But, you know, I mean, I think as we've learnt, as we've been playing more and more with this tool, asking it to suggest optimizations, extensions of campaigns, we've learnt the more you challenge and the more you critique the outcomes, the more illuminating the results become. And I think strategically and creatively, we're learning how to use AI as a tool to empower and not limit creative thinking.

But I think one of the things people forget, and Rob touched on this a little bit, is the burden's on us, too. We cannot accept anything at face value and become lazy. It's too easy just to plug it in and say, "Oh, Claude told me." You're going to get a wrong answer. Your experience, your thinking can help it get better along the way.

Elena: Well, Rob, as someone who is a creative and you were once a copywriter yourself, how do you feel about using tools like this to produce creative work?

Rob: I have a bit of a different take on this than maybe I even had, you know, a couple years ago. There's two camps. One side sees AI as creative Armageddon for humans, that is basically eating or stealing our lunch money, and we're getting our lunch money stolen by the robots.

The other side sees this creative nirvana, that we have tools that are elevating our ideas beyond what we could have ever done before. And honestly, I think both camps are right, and I think neither of them are useful. I think the reality is technology is fundamentally changing how we work as creatives, as marketers, as strategists in advertising.

And at the end of the day, we aren't creating art. We are creating communication that helps marketers sell their soap. If AI can help us do that better and help us do that faster, then we have a responsibility to our clients to use every bit of that power in their best interest. But all these tools, we should be learning them, we should be stress testing them, we should be doing what Catrina and her teams are doing.

We should be helping to see how can we serve our clients better with this technology. And I know that's hard to hear because there is so much passion in the craft of advertising. But it is just the reality of where the universe is going. We have to be agile. We have to look at this and remember we are not in the entertainment business. We are in the business of helping our clients reach their customers in compelling ways, and AI is doing that, in ways that are challenging all of us.

Catrina: But I also think it's exciting though too, and I think that's the bit that gets forgotten because I do think it can stretch our thinking. Like, it can actually... It's like, oh my gosh, I hadn't thought of that angle. Where did that suddenly come from? And it's like, I think used right, and I mean, Rob just said we weren't entertainers. We are entertainers if it serves our brands to develop entertaining work. And it's, I think, we have to think about how can we use AI to better serve and make us better? I think instead there's an abdication of responsibility rather than saying, "This thing is giving me all this information. What do I do with this to make myself smarter and get to the next level?"

Rob: Absolutely.

Elena: No, it's like, you're both making great points. Rob, when you said that, you know, there's — we're on a podcast talking about AI for advertising — there's a whole separate conversation happening right now with AI for creative work, like purely creative work, and it reminded me of, I heard that in The Odyssey, Christopher Nolan, when he created the whirlpool, he made that out of jet skis.

He didn't want to use CGI. It's like, that's a cool fact for Christopher Nolan, but if you did that for your brand, for a commercial, there might be a question of could we not just CGI the whirlpool? Like, it's probably going to look the same.

Rob: And then you have the flip side of Elon Musk. I'm sure you guys all heard that. We'll see how it happens. But he's making The Odyssey right now using 100% AI, and he's gonna do it in, like, three months. And we'll see how it goes. But, that is... Yeah, isn't that interesting?

He's like, "Okay, I'm gonna make the same movie." You know, obviously it's a classic tale. We'll see how it shapes up. But yes, it does challenge us all to go, are we rooted in our old ways just because that's the way that it was done, or is there a new way to reimagine all of it? And we, I do think, have a greater responsibility as being stewards of our clients' resources and energies and time to be making sure we're using the best and the brightest tech to get them there.

Elena: Yeah, agreed.

One thing that I wanted to make sure we touched on in this conversation is a lot of marketers are still getting familiar with the idea of pre-testing in general and pre-testing creative. I think that it's become more popular and people like System1 have been promoting it. So I wanted to make sure we talk about that a little bit because Catrina, you've been a big part of, as was Rob, of developing our creative pre-testing platform called ScriptSooth, and we've talked about that on the podcast.

How is this framework and how you're gonna use the framework different from what a pre-testing platform like ScriptSooth already does? Cause some brands might be invested in that and curious, like where does a framework like this fit in?

Catrina: You know, I think it's actually about a blend of both, but I think this framework and its likely application is a little different at the moment from how we've been using ScriptSooth. So we developed ScriptSooth as a tool to evaluate, refine, and enhance scripts, to drive short-term performance primarily, and it was validated using 25 years of our clients' performance data and also informed by 10 years of proprietary copy testing results.

So a lot of substance driving it to that set of results. And it does use a little bit of what we have been using in the new framework. And I think this new framework contains some of that learning, but it also contains that whole breadth of other learning that I talked about, more academic research, our belief about how TV advertising works at a brand level over the long term rather than necessarily contributing to that short-term growth, and improving the efficiency of lower channel funnels, which is what we want that mid-funnel to actually do.

And I think the role of this framework is going to help us structure internal conversations, train strategic growth and creative to have those conversations using a shared common language, which is gonna allow us for much more constructive debate rather than the, "I don't like it. Go find me a different one," which doesn't help at all. And so I'm really excited about where we take this framework in the future. I'm increasingly thinking that it can help challenge us to the next level, both strategically and creatively with much richer diagnostics and ideation potential. Whether or not we do use it as a broader pre-testing tool, I'd rather like to think it's a little higher up in our development process than that.

But who knows? It's kind of too early to say. We're having fun with it anyway.

Elena: Yeah, so it's not like an either/or thing with pre-testing. This is being used in —

Catrina: No. I actually think we can use them both. And actually, funnily enough, we are still using ScriptSooth before we go on air, particularly for those clients who are actually much more performance-driven in the short term. So it's kind of a blend of both at the moment.

Elena: That makes sense.

Well, to wrap us up here with something kind of fun, I wanted to ask both of you — maybe Catrina can start. What is something that you have surprisingly strong, you might even say snobby taste about?

Catrina: Oh, well, not surprisingly, coming from me, it's going to be tea. So I won't drink Lipton's. I love Twinings English breakfast tea. I love PG Tips. I love Fortnum & Mason's breakfast tea too, with milk, of course. All those hot black teas with milk. And then actually Celestial Seasonings herb and fruit teas — they're one of the first brands I ever worked on once I came to the States, and they were still independent at that time, and I loved their philosophy, and I've stuck with them ever since.

But yeah, I'm very snobby and picky about my tea.

Rob: I need to try the milk in the tea. I haven't done that. It sounds delicious, and I keep... So, it's black tea only?

Catrina: Black tea only. Breakfast tea. Don't put milk in Earl Grey.

Rob: Okay. Do you eat little sandwiches with it when you're drinking the —

Catrina: No, cause I know I have it at breakfast.

Rob: Okay. Got it. Got it. Got it.

Catrina: I don't have a cup of coffee in the morning. I have a cup of tea.

Rob: All right. Well, I'm pretty snobby about something. I believe that all underwear should be made out of bamboo. Bamboo is fan-freaking-tastic for underwear, and so I highly recommend it if you haven't tried it out.

Elena: Okay. I don't think I wanna take that one any further than that, but I'm sure it's lovely.

Catrina: Come on, Elena, what about you?

Elena: So mine is that I'm kinda snobby about flying. I really like to fly Delta, and I am a Comfort Plus type of person. And my reasoning is I don't fly often, but I'm six feet tall, my husband's 6'6", we simply do not fit in economy.

So we will shell out a little more to sit a little bit better. I think those economy seats are just horrible, and I'm not about a middle seat either. But I get a little snobby.

Rob: I'm gonna support you on that. As a hub city for Delta, I'm such a snob. Like, I got to fly Delta, and there's no such thing as a connecting flight.

Elena: Yes, I'm the same way about that too.

Rob: That's just not happening.

Catrina: And definitely not middle seats. Never.

Rob: No middle seats. I — what's a middle seat?

Catrina: I have no idea.

Elena: Yeah, you're right though. We're spoiled living in Minneapolis because we've got a really nice airport and we got lots of Delta flights.

Rob: Poor people that live in like Oregon, where you have to fly the wrong direction in order to start to head into the right direction, that would make me angry.

Catrina: Oh, I always had this rule, like I had to live no more than an hour from the nearest international airport. I think it's being a Brit. Somehow you got to be able to make your escape if you need to.

Elena: That seems like a good rule.

Well, thank you so much, Catrina, for joining us. That was really fun and informative, and thanks for coming back on the pod.

Catrina: Loved it. I'll see you again.