This is the second post in my series about participating in the Pokémon TCG Kaggle Competition. Here's my previous entry, which is where you should probably start.
Cold Reading the Competition
Creativity during deck-building is one of those things that keeps so many players returning to their beloved TCGs. Let's consider the goal of the competition: to build an agent that beats all the other agents. What makes PTCG more challenging 1 than a game like Chess that has had super-human models for decades? (1) hidden information — what your opponent has in their hand, what card you will draw next turn —, (2) randomness, and (3) diverse and non-overlapping strategies — Crustle decks and Alakazam decks fundamentally have different capabilities and costs.
As someone in the field of ML Research 2, I hoped I'd have a substantial advantage over a competition whose average participant who would be forced to rely on more tutorials and agentic guidance 3. This competition was the 4th largest competition (by teams) ever hosted on Kaggle: so it obviously attracted many first-time Kaggle competitors who were engaged enough to join because it was Pokémon. So, in order to win — or at least place top 50 —, you need to do something different than the masses.
In order to be successful in a tournament like this you must have…
- a deck with sufficiently high power level,
- a deck well positioned in the meta 4,
- an agent able to learn how to play its deck competently,
- a deck resistant to straightforward counterplay 5
Dragapult presence was single-digit when I first investigated the competition. Since the best deck in the format was practically unused by the leaderboard, I thought it was my best bet for a high-placed finish.
Dragapult certainly checks positions #1 and #2 in paper: it defines the meta. #4 seems to be mostly true 6. However, it is a difficult deck to pilot, especially in the mirror. If you're not familiar with PTCG and comfortable adding another 2k words to your read count, I strongly recommend clicking that link. All-in-all, it meant that I needed to overcome #3, which was certainly an unknown; no one else had been able to construct a competent Dragapult Agent so far.
Reinforcement Learning Algorithms
For a modern overview of RL I turned to The MARL Book which is an academic overview starting from classic tabular RL learning, progressing to single-agent Deep Learning strategies using modern neural network architectures, and then progressing to its namesake: distributed agent learning. I devoured the book, spending about a week or so of my precious competition time digesting this rather than jumping right into coding.
Policy Learning
The agent's actual decision making kernel is called a "policy"; the act of training a model is incrementally improving that model's policy so that it makes better decisions more frequently: whether by cleverly deducing the marginally better option of two similar actions, simply avoiding making catastrophic mistakes, or identifying a single good option out of a sea of mediocre ones. In a complicated game like PTCG, we expect all of these to apply.
In a simple game like Tic-Tac-Toe it is possible to enumerate every single game state and optimal actions. That isn't possible once you get to a level of complexity of Chess or PTCG. A "reward" is a measurement of a single action's relative utility; in a game like Tetris, eliminating a row block grants points, which is a fantastic measurement of the actual goal of the game. In Pokémon, taking a prize card is an easy to model reward 7.
On-Policy vs Off-Policy
It's quite computationally expensive to only analyze moves as you perform them: "on-policy". Being able to review previous moves you made with a slightly different model is much more sample efficient, which ultimately means that your model requires less data in order to generalize and play at a given desired skill level. Off-policy learning also includes reviewing public games. One of the big facets of the competition is that all games were open and 20 GB of top-100 leaderboard data was released every day. A critical part of my strategy heavily relied on attempting to learn the average policies of top-performing decks, so that my local agents were not biased against their own playstyles.
Self-Play
Ultimately, a successful RL model must be able to generalize from its current state to estimate the possible reward for each legal action it could take in the current step. During live play, it "exploits" the environment: perform the action that has the highest discounted cumulative reward 8. However, during training, models need to be able to "explore" their environment by taking actions that they think might be bad and then adjusting their hypothesis position with the actual observed result ("oh yeah that really was bad"). Otherwise, a model may get stuck in a local maximum: executing an "alright" move over and over that it found early in its training, never deducing which better options could be available.
Self-Play is when models play many games against itself. There's many variations of self-play, and it works best in symmetric games; you get twice as many samples per game for your policy (one from each perspective), and, in theory, it naturally learns to execute dominant linear strategies, counter strategies, and counter-counter strategies, even in imperfect-information games like PTCG. The core reasoning is that in an imperfect information game 9, you can always pretend that you have access to a strategy that you do not actually have.
This is a critical form of skill expression for games like Poker; optimized play in information imperfect games requires utilizing a mixture of strategies: (1) using the strongest move considering an "average response" of your opponent, (2) playing your strongest move without considering your opponent's response, (3) bluffing a stronger move than you actually have. Which of those three modes you should be activate at any game state is part of what an effective information-imperfect agent needs to figure out on its own.
Next Step
Next time I'll talk extensively about some actual deep learning algorithms. Here's a spoiler:
-
Messy use of this word. but, for purposes of this discussion, I am stipulating that TCGs like PTCG and Magic are a priori more challenging for an ML model to learn how to play simply by observing that there isn't a super human agent rather than making a positive or de facto statement about whatever the heck "challenging" means. This competition hasn't resulted in super-human agents: while working on this blog I idly watched the top-8 leaderboard to see the quality of their play. The at-the-time #2 deck during setup played a Meowth EX to their bench. This is a game-losing unforced error obvious to any human with half a dozen games of Dragapult. ↩
-
Why would The Pokémon Company decide to host a $350k prize tournament to crowd-source a ML PTCG agent rather than fund it internally? While frontier models are seeing a lot of business success in news, Partially Observable Markov Decision Processes 10 are an area still under active research, and even the best frontier models are still below PhD-level reasoning in research applications. My #1 piece of evidence is how many successful teams publicly reported using Behavior Cloning and PPO despite both of those algorithms being unsuitable for imperfect information environments. For the 100 teams using the strongest Claude models, they simply aren't going to break out top-8 without actually doing the hard work of literature review and creating novel research code. ↩
-
I want to stress that many other teams saw a lot more success than I did, most-probably-in-part to being able to iterate faster, earlier, and more extensively using an LLM-first approach despite lacking the ML foundation that I possess; so I don't want to portray myself as better than them. Spending a week doing literature review is great for my own personal growth, but it wasn't a winning strategy in a competition with such a tight timeline. I bet it is, however, a winning strategy in a competition with a longer horizon. ↩
-
No matter how powerful the core of a deck built around, say, Lucario-EX might be… if Crustle makes up half of the meta, you are literally incapable of doing damage to the opponent. ↩
-
And if the move is, for whatever reason, extremely difficult to counter, then wouldn’t I be a fool for not using that move? The first step in becoming a top player is the realization that playing to win means doing whatever most increases your chances of winning. That is true by definition of playing to win. The game knows no rules of “honor” or of “cheapness.” The game only knows winning and losing. ↩
-
N's Zoroark is the closest thing to an innate counter to Dragapult. It features a 3-5% paper presence and with a > 5% top-16 rate… but it rarely top 1s. So: N might be a great counter against average Dragapult players and thus qualifies as a #4 caveat, but the counter isn't unbeatable by the very best Dragapult pilots (#2 caveat). Plus, N's was virtually unseen this entire tournament, so, with the benefit of hindsight, Dragapult absolutely passes this requirement. ↩
-
This is actually a very bad reward to actually use for a zero-sum game like PTCG. Why? You are eliminated if you have no Pokémon in play; a reward mechanism that tracks you taking 5 prize cards, then losing to an opponent taking their very first prize by knocking out your only Pokémon in play is unable to track this lose condition. And that's only one practical reason; from the theoretical basis, MARL book is clear that the only reward scheme that is guaranteed to converge for any zero-sum game is whether or not you ultimately win or not. Which, unfortunately, is very sparse, and adds yet another difficulty in training the model. Still, Prize Cards make an intuitive reward mechanism so I'll be referring to it often in this blog, even if it isn't what I actually used for my models. ↩
-
Discounted cumulative reward distributes reward to moves that happen before a very good move; suppose when you attack you get a prize card! You could assign all that reward just to the attack action, or you could "discount" it to previous steps and assign only a partial reward to each of those steps, giving the attack action itself the largest share. This is essential because PTCG plays are "lines": 3- or 4- step plan has a delayed payoff, and each step is essential. (1) Ultra-Ball, (2) discard 2 bad cards, (3) fetch Dragapult EX, (4) play Dragapult EX on my active Pokémon, (5) attach a Fire Energy, (6) attack with Phantom Dive, (7) assign 6 damage tokens to an enemy with 60 hp. That's a lot of steps for ultimately 1 prize card! But each step was essential! You can't Phantom Dive if you're not evolved, you can't evolve without Dragapult EX in your hand, you can't play Ultra Ball to fetch Dragapult EX without first discarding two cards. Even then you could slightly rearrange the order: while you must play the Fire Energy somewhere in this sequence you could play Fire Energy at any time before attacking. Figuring out the rewards here is really challenging. ↩
-
assuming that there is not a single prevailing ultimate strategy; something so powerful that it has no counterplay and will always win and win with a maximal score. your opponent should always play as if you don't have it, since there's no point in trying to counterplay it. ↩
-
this will be explained more thoroughly later; but this is the formal name for "choosing the best action in a situation where there is hidden information and unpredictable outcomes". y'know, the competition. ↩
