My Case Against Review Metrics

I don’t understand the thought process that leads to using established metrics in game reviews. That sounds pompous but I’m being earnest. I’m open to being convinced that I’m wrong here but I can’t yet see why you’d limit yourself in that way as a critic. Here’s my case against metrics.

I’ve got to explain what I’m talking about before I get into the rant. I’ve seen two applications of metrics that bother me. The first is when someone sets up a mathematical system for rating games. They have a number of different categories for different aspects of games, and they give ratings in each of those categories, add it all up, and arrive at a final score. The second is the same sort of thing but without the explicit math. Every review goes through the same categories, evaluating every game through the same lens. My befuddlement and criticisms are certainly more severe for the former, but all of my arguments apply to both, just to different degrees. I’m also not calling out anyone specifically. I rarely read or watch other people’s game reviews.

Fundamentally, I think my issue with this approach is that it reduces games criticism to what you’d see in a product review for a blender. A blender needs to blend things well, so you evaluate it on that metric. A game needs to…what? I’ve got games that make me laugh, games that make me cheer, games that make me angry, games that give me a headache, games that make me think about my own psychology, and they’re all great. Each game defines its own purpose. How can I bring a pre-selected set of metrics to that? I don’t think people actually believe that they’ve figured out all of the necessary criteria for a great board game. Indeed, I don’t think there is a set of necessary criteria.

Perhaps some people only enjoy games that fit within a narrow band, and they’ve figured out what those parameters are. Maybe they hate randomness and love emergent player interactions. Reviewing a game, for them, is simply measuring it up against those two criteria, and the benefit of reading their review is to see how it measures up on that narrow scale. I don’t think most people have such narrow preferences, and while everyone has their broad likes and dislikes most people in the hobby are willing to approach a game on its own terms. Most people also won’t automatically enjoy every single game that fits within their favorite mechanisms. That kind of thing is usually on a sliding scale.

(I’m reminded of perhaps my biggest pet peeve in game reviewing: when someone says something like “X is an auction game, so if you like auctions you’ll have a blast”. You don’t know that. I love auction games but I don’t automatically like all of them. Who operates that way? I’m willing to bet a very small number of people do.)

Anyways, setting aside the question of if board games are art (I couldn’t even begin to tell you how I define “art” and that’s the only way to arrive at an answer), they’re certainly not a purely functional device like a blender. Or, if they are, their function is so broad as to be useless for analysis–”attention” or “interest”, perhaps. Furthermore, even if we were to pinpoint the sole function of a game (maybe “simulation” for the most rigorous of wargames), that doesn’t apply to all games. I’ll allow that a reviewer who was solely analyzing the simulational fidelity of games might find a static set of metrics useful. But if you’re measuring the quality of a game generally, that’s a harder beast to pin down.

Games are often more (or less) than the sum or their parts, and using metrics is literally summing the parts. It erases the possibility of an experience beyond the obvious. It treats individual aspects of the game as separate from each other, when in reality they interact in new and novel ways all the time. 

Pre-set metrics also don’t understand emphasis. Some of my reviews contain no mention of the art, because the art is fine. It doesn’t noticeably add or subtract from my experience. It’s not worth mentioning. If art was necessarily part of my calculation this would frequently create a conflict between my experience of the game and the score my metrics spat out. Castles of Burgundy is an exceptional game with perfectly forgettable art. Does that make it less than exceptional? No, I still have that brilliant experience when I play. Metrics run counter to my actual experience, and it’s pretty clear which I should trust.

In response to this point, someone might retort “that just demonstrates that you don’t value art highly, so if you were to create a set of metrics art would be weighted insignificantly”. But that doesn’t work either, because in a number of games the art is core to the experience. Tokaido and Above and Below come to mind. 

The presentation of metrics also lacks grace, as it either puts a quantitative number on each metric or simply lists them with the same implied weight. This is the only area in which the quantitative approach can be more accurate because at least you can weigh the most important things more. But I cringe when I see something like “rulebook quality” next to “enjoyability”, implying that both are equal partners in the evaluation. Sure, you can talk/write your way past that, but why set up the disconnect in the first place?

Finally, I think using pre-established metrics closes the reviewer’s mind, even if only subtly. We should be open to having our preconceptions blasted away by something truly new and innovative. If you organize your reviews into the same handful of talking points you start thinking of games only in terms of those talking points. 18xx games would die to most review metrics. But they capture an almost wholly unique game experience that render the negatives obsolete.

Fog of Love is my review white whale, a game that gave me emotions and thoughts I’d never felt while playing a game before or since. But it’s also almost impossibly fragile, relying on a level of buy-in and shared commitment that’s hard to generate or maintain. Every once in a while I’ll dive back in and try to finish my review. But I couldn’t even approach the review responsibly if I used metrics. Fog of Love operates outside of such constraints.

As a reminder, I’m only arguing against the use of pre-established metrics that you use to evaluate every game. I’m not against bullet-point summaries highlighting the highs and lows (if done well), or a behind the scenes checklist you might use to make sure you don’t miss anything important. My case is against constructing an evaluative cage that you crudely smash every game into. Let’s push board game reviewing forward.