I was recently reminded of Goodhart’s Law:
“When a measure becomes a target, it ceases to be a good measure”
This is particularly relevant for publishers struggling with recent changes, so I thought it deserved a re-visit. …
As the top of the funnel (Google search) collapses, publishers are under pressure to get more from the audience and the traffic they already have. That often means more engagement, which often translates into metrics.
If you want more (or less) of something, you have to measure it. Right?
When I want to lose weight, I put a graph on the bathroom mirror where I can track my daily progress against my goal. And when a marketer wants more engagement, they have to convert that abstract concept into measurable things, like pageviews, time on site, opens, clicks, etc.
In that context, gamification (and similar strategies) seems like a godsend.
Tricking human psychology
When you first learn that sales and marketing are largely a matter of hacking our biases, it can feel a little dirty.
People are more activated by fear or loss than hope of gain, so you can get a better response to your pitch if you incorporate that. “This deal is only good for today.”
How about this one? People are motivated by group or team membership even when the group or team is ridiculous – like “everybody who’s stayed on the 7th floor of the hotel.” Still, appealing to a group score can work. E.g., “Save water! People who stayed on the 7th floor were 10 percent more likely to reuse their towels than other guests at the hotel.”
Gamification is another way to hack the brain. Add badges, streaks, points, rewards for comments, and so on, and you give people a reason to come back.
But is their reason related to your mission, your brand, or how you’re meeting their needs, or are you just playing with them?
More than this — there’s a deeper problem than manipulation. It’s what happens once you start managing to the number instead of the mission.
Measurement is great, but dangerous
It’s not just that gamification can feel cheesy or manipulative. The deeper danger is that it can cause an organization to confuse measurable activity with meaningful engagement, and then to optimize for the wrong thing.
That’s the problem philosopher C. Thi Nguyen has been exploring in his recent work on scores, metrics, and games. (See The Score: How to Stop Playing Somebody Else’s Game.)
Games are great fun – within the constrained world of the game. They create a bounded world with a simplified goal.
Real life is more complicated.
If you rank a law school by its rejection rate (“Gee, you must be exclusive!”) it creates false motives – like soliciting applications from students with no chance of getting in.
Metrics can lead you to forget doing things based on your values and focus on what moves the numbers.
Reducing values to metrics
This isn’t some crazy idea. It’s a logical process. You have a value, like “provide the world with a better banking experience.”
Then you create a goal. “Get more accounts.”
Then you create a metric. “How many accounts are my sales reps creating?”
Then you incentivize around that metric.
Congrats. Now you’re Wells Fargo, and you have a million phony accounts on file.
You could call this “value capture,” where the numbers are more important than the value they were supposed to serve.
Engagement vs. measurement
“Engagement” plays along the edges of this same problem. It sounds as if “engaged” is an obvious good – until you ask what it actually means, and how you’re measuring it.
Engagement is often measured by clicks, opens, comments, return visits, and so on. That’s great, right? Those are “objective” indications of engagement.
Well … no, they aren’t. Engagement doesn’t really mean “people are doing measurable things on our site.” That’s a proxy. Engagement means something more like “we’re building a relationship with the right people around something that genuinely improves their lives.”
There’s the challenge. You can’t measure how well you’re improving people’s lives, so you create a proxy you can measure that gets close.
For example, a car company can’t measure customer satisfaction, but it can measure complaints, repeat purchases, and positive reviews.
The mistake is to allow the proxy to become the goal, because that shifts your values and simplifies something that’s multi-layered and horribly complicated.
That’s the trap. The moment the proxy becomes the goal, you stop asking what’s true and start asking what can be measured, and how to do better on that measurement.
Imagine the “data-driven” suitor who wants to know if his romantic prospect really likes him, so he creates a series of metrics.
- She said hi to me today.
- She smiled when I told a joke.
- She liked my post on Facebook.
The drama writes itself. The awkward, introverted guy has a whole spreadsheet proving that Sally loves him, but Sally’s just being polite.
How to interrogate your metrics
You have to have metrics. You can’t fly by the seat of the pants and “intuit” that your business is doing well.
The challenge is what and how to measure so that the measurement doesn’t distort incentives or replace the goal.
Here’s a simple model to follow.
- Create the measurable proxy.
- Ask what else that proxy might be showing.
- Catastrophize. Imagine how that proxy might distort incentives or actions.
- Create a different proxy to measure that.
Let’s take Wells Fargo as an example.
1. The measurable proxy was new accounts, which isn’t a terrible idea. If people think Wells Fargo provides a better banking experience, they’re more likely to move their business to Wells Fargo.
2. Or … maybe there’s some other reason. (I’m making this up.) Some promotion allowed you to get ½ off Spotify if you created a Wells Fargo account. People didn’t want Wells Fargo. They wanted Spotify.
3. If we incentivize more accounts, will people create fake accounts to get the incentive?
4. How can we measure if an account is real? Maybe by activity. Maybe by checking names and addresses against a database.
If Wells Fargo had thought about the false incentives their measurement was creating, they could have avoided a huge scandal.
Let’s apply this to gamification
Take something simple: a daily streak. “You’ve studied Spanish 14 days in a row!”
The measurable proxy is consecutive-day activity. Reasonable enough — if people are showing up every day, presumably they’re learning the language.
Or maybe not. Maybe they’re showing up because they don’t want to lose the streak. The streak has become its own reward irrespective of the goal of learning the language.
How can this go wrong? People start doing the bare minimum to keep the number alive — tapping through one flashcard, translating “the cat is on the table” for the four-hundredth time, just enough to register as a visit.
Worse, you may be training your most serious learners, the ones who used to sit down for a real 30-minute lesson, to instead do a 20-second review just to protect the streak before bed. You’ve converted language acquisition into streak maintenance. The number goes up. The Spanish doesn’t.
How do you know if you’re measuring what you think you’re measuring? Check whether long-streak users are actually progressing — vocabulary retained, grammar concepts mastered, conversations attempted — or just logging in. If your 200-day-streak users can’t hold a basic conversation, the streak isn’t measuring learning. It’s measuring an aversion to losing a game.
That’s the pattern with every gamification mechanic — badges, points, leaderboards. Each one starts as a proxy for something real (in this case, fluency) and can quietly become the thing people are actually optimizing for. At that point you haven’t attracted a language learner. You’ve attracted someone who’s very good at not breaking a streak.
Let’s apply this to engagement
How about something simple like “engaged time on site”?
1. We all know that “time on site” isn’t the best because somebody might be talking on the phone while their browser is stuck on one page. “Engaged time on site” would check to see if the user is scrolling, clicking, or otherwise interacting with the page.
It’s a decent metric. How can it go wrong?
2. Scrolling and clicking might mean the person is engaged with the content, or it might mean the person is confused and keeps moving around on the page to find what he’s really looking for.
3. How can this metric go wrong?
Let’s say you reward editors for content with lots of “engaged time on site,” and they discover that if they write articles that require the user to scroll around (to find definitions, charts, etc.) they get a better score and get the attaboys at the monthly editorial meeting.
4. How do we know if we’re measuring what we think we’re measuring?
Maybe you try a satisfaction survey on the page and compare “engaged time on site” with satisfaction.
Conclusion
The point of engagement is not to get people to click more, scroll more, comment more, or maintain a streak. Those things are just proxies for the real goal, which is to attract the right audience, serve them well enough that they want to come back, and build enough trust over time that the relationship deepens.
Metrics are essential because – if they’re correctly configured – they help you see whether you’re moving in the right direction. But the moment the metric becomes the strategy, you’re in danger of serving the score instead of the customer.
That’s why gamification and engagement metrics need to be treated with caution. They can be useful tools. They can help reinforce good habits, highlight value, and make a product easier to use. But they can also distort incentives, attract the wrong behavior, and train an organization to chase measurable activity rather than meaningful relationships.
