If you design even just one game, it doesn't take long for you to realise that a game is mostly a maths exercise in disguise. You have a random selection of cards, and you compare numbers to see who wins. You roll dice, and the number you roll changes the state of the game. You count spaces on a board. You collect some number of imaginary resources, and multiply them over time. So why do some games feel more fun than others? And why do you play some games even though they're not the most fun game you own? It's probably partly due to a game's theme, but what's interesting about a theme is that it can be expressed in both the game mechanics (the rules and maths) and in the "flavour" (the narrative and artwork).
It can be tempting to think of a theme expressed through narrative elements as a "skin" or a costume. Theoretically, you can take any game, give it a new name themed after anything you like, assign artwork to its cards or tokens or board to reflect what you've named it after, and you end up with a themed game. Alice in Wonderland Poker. Star Trek Monopoly. Star Wars Pandemic. Godzilla Yahtzee. The possibilities are embarrassingly endless, but often not actually effective in conveying the theme.
That doesn't mean narrative themes don't work, though. There are lots of games that have primarily narrative themes that actually feel like its theme.
Sometimes it's a matter of synchronicity. Alice in Wonderland Poker doesn't make much sense, but Firefly Poker could work. Star Wars Pandemic turns out to be a stretch, but Cthulhu Pandemic works brilliantly.
In many games, a theme sparks the design or else emerges from the design as the game is developed. Mansions of Madness is set in the Lovecraft universe. It has many telltale Innsmouth tropes (cultists, Deep Ones, suspicious locals). The cards included in the box are thematic to the superstitions and occult of the 1900s. The game is structured around exploration, investigation, and in the end the primary goal is often not to defeat the predictably horrific enemy, but to just get away from it alive. It feels like a game about the fictional world of Lovecraft. Mechanically, the game would probably work just as well with a theme built around Sherlock Holmes or the Alien franchise or Friday the 13th, but not quite as well for Star Wars or Conan the Barbarian or Mad Max.
The original Pandemic game demonstrates this pretty well. It's about the outbreak of a disease, so the problem you have to solve during the game is the rapid spread of a virus. No matter how much you try to quarantine people who have been infected, it seems you can never quite stop the spread.
It's a good game and luckily became popular, and then the licensing deals started. The game designers (or maybe the publisher, I don't know) realised that the Pandemic "engine" could be just as fun when you swap "infection" with literally any other problem.
Pandemic: Reign of Cthulhu trades infected patients spreading a disease for cultists attempting to summon evil Shoggoths. You frantically travel from classic Lovecraft location to classic Lovecraft location, gathering cards that you can later spend to close the summoning circles initiated by the cultists. It works so well that, to me, it's the canonical version of Pandemic.
And then there's Pandemic: Star Wars, which tries to trade infection for Imperial occupation. You fly from planet to planet (unless one planet doesn't connect...through space...with another planet...?) to eject the Imperial troops using your special Jedi powers. All the while, you're pursued by Darth Maul and his variable level of wrath. It's a baffling implementation of Pandemic, straying the farthest from the core engine than any other adaptation I've seen. There's a lot about the game that doesn't work for me, but the most visual element is the board, a bland matrix of indistinct planets. I'm told that the theme is Star Wars, and I can see that there's space stuff and cards with popular Star Wars characters on them, and yet the game doesn't feel like it. Star Wars has never been about sitting down to look at a star chart, carefully plotting a course, moving numbers from one column to another. That could work for something famous for its hard sci fi, or something that's just more cerebral in general. Arguably, it could have worked for the Dune universe, swapping Imperial troops for House occupation, and Jedi and Darth Maul for Bene Gesserit influence. But as a Star Wars game, the theme truly is just a fancy costume.
A narrative theme is best, I think, when it's the process of the game.
The other expression of a theme comes through the mechanics of the game, the way the numbers and move options and strategies interact. Theoretically, a theme expressed through mechanics ought to be evident in a game before it even has a name or artwork, playing with generic gaming tokens and functional, text-only cards.
A game about Roman infantry ought to reward players for moving units as modular groups, while a game about Greek infantry ought to reward players for using a monolithic unit to express the difference between legion warfare and phalanx warfare. A game might force players to bid for advantages in a game about high society auctions or corporate monopolies, or players to gamble in a game about the Old West. These themes come through because of what a player can do in a game, and arguably the best theme for a game grows organically from how the game has been designed to function (or you can fake it by emphasizing specific mechanics in consideration of the theme you're trying to broadcast.)
The card game Coup is about sci fi politics, and one of its core mechanics is that you can lie about what card you have in your hand. It's legal unless someone accuses you of lying, in which case you must reveal your card. If you were lying, then you're punished, but if you were telling the truth then your accuser is punished. Lying and etiquette perfectly expressed in just one interaction. It's perfectly aligned to its theme.
The game Dark Cults is a game about an unsettling late night walk, with one player trying to heap misfortune on the imaginary character and the other player trying to save the character. It's an spooky battle of fate, and it plays like a tarot reading, with each player drawing cards and placing them on the table in an arrangement that gradually forms the story of the character's harrowing misadventures.
My favourite example, though, is that many wargames feature defence rolls. This intrigues me, because it adds a whole new interaction into what's often an already complex game. You don't just roll dice to avoid being hit. You generally have to compute how many dice to roll, what number you're hoping to see on each dice for success, and then your opponent has to do the math to figure out which of several "successful" hits (except some were actually blocked by your defence) deals damage, and then you apply the damage.
It takes the glory of the moment away from the active player and places it in the hands of the opposing player, and demands several new mental cycles. It stops the game in its tracks.
Or does it? Try playing a wargame without the defence roll. Admittedly, I prefer wargames without a defence roll, but even I have to admit that a defence roll provides an illusion of a reaction. When you're fired upon and you're hit, your soldiers feel a little less powerful than they seemed when YOU were the active player. Why didn't their armour help them? Why didn't they dodge the spear thrust? Why didn't they parry the sword strike? What's wrong with them?
Add a defence roll, and the same thoughts occur when you fail, but even in total failure you're able to blame yourself instead of your troops. They were doing their best, but you rolled poorly. They're not the problem, you're just a bad General. Next time, you'll do better.
I think the theme of a game is a fascinating thing to observe. It's easy to think that a game's theme is just artwork, and it can be re-skinned as anything. I don't think that's true, nor do I believe there's any one right way to express a theme. Gaming is a unique form of art, with lots of ways to influence the player's thoughts, imagination, and feelings. Playtesting reveals much, of course, so do it often and optimize your theme in all the different ways you're expressing it.