Statistics fail 2: The UN Women ads
The UN Women ads tell us nothing about how widespread inequality is, except in a very self-selected group. We should keep to examples that actually show sexism.
A collection of ads has been making the rounds lately. The ads, run by UN Women, show the most common Google searches beginning with phrases like “women should”, “women cannot” and so on. The results look pretty dire. Example (credit Adweek):
As we know, Google tries to be helpful by guessing what you want to search, based on the most common searches with what you’ve typed so far. The conclusion drawn is that sexism is so widespread, that searches like “women need to be put in their place” make it to the top. Which seems logical, until you think about it for more than two seconds.
The problem is that the searches shown aren’t representative of searches generally. By definition, we’re only examining searches that start with “women shouldn’t”, and so on. Ordinary people don’t search for phrases starting with “women should”, “men should” or even “people should”. Have you ever run a search even remotely similar?
Moreover, when people do search for them, they’re looking to search for something specific. Erase the ads from your brain for a second and ask, what could they be? Here’s the thing: any phrase beginning with “women shouldn’t”, “women need to”, or anything with some sort of imperative-like verb after “women”, is always going to be sexist. It’s always going imply some sort of normative statement about how women should be—otherwise you wouldn’t have started your search like that!
Of course, it’s true that normative statements could go the other way. Like, we could run “women should be equal to men”. (This is the first suggestion after “women should be e”.) But how often do the more enlightened actually search for such things? You’re probably one of them. Think through your own search history, I bet it won’t feature. People who understand equality have better things to search for than a generic search for the proposition. Even if they did, it would most likely just be a phrase like “gender equality” or something like that.
So not only does the test consider only a subset of searches, but it’s a very self-selected subset of searches. How does this cripple our conclusion?
It cripples it because it tells us nothing about how sexist the general trends in searching are. All we’re doing is we’re taking a group of searches where the criterion for consideration implies sexism, and we’re then asking ourselves how sexist they are. Rephrased more bluntly: Of sexist searches, how many are sexist? The results aren’t surprising.
Even if you don’t buy that the phrase “women should” implies sexism, it’s still not something people go about searching as a matter of course. It’s still unlikely to be representative of the whole set of searches at large. In statistical jargon, we’re assessing a conditional probability, that of a search being sexist given that it starts with “women should”.
Importantly, the test doesn’t even tell us how often our criterion is fulfilled, i.e. how often people start searches with “women should”. My guess is it’s a very small subset, or at least not a big one. Why do I say this? Because it takes a while to get there:
UN Women also conveniently overlooked some other examples. To be clear, these are not examples demonstrating the contrary, because they are subject to the same flaws as the ones I described above. And there are still some really troubling issues in there. But the picture’s not as universally negative:
You can see the temptation to use the examples to show that, even in modern tech-savvy society, sexism is still rife. You can see an inclination to wave away statistical theory as a pesky inconvenience, a technical detail immaterial to a wider point. That would be a mistake. Let’s not kid ourselves: gender inequality is still too widespread, in too many places, and too much of it under the radar. But we should use examples that actually demonstrate it. Pay gaps, dowries, glass ceilings, genital mutilation, predefined gender roles, stereotypes in certain industries, I mean, hell, stereotypes just generally. All things that we really need to combat. The last thing we should do now is divert attention from real issues by drawing attention to a flawed metric.