Beating Gemini 3 Flash with a 4B model on Dune
Trained a GRPO RL policy to play a 90 turn complex strategy game
Earlier this year, I was visiting my friends in the beautiful city of San Francisco and I got hooked to playing board games quite a bit. From heat, parks, catan, we kept increasing difficulty, and finally ended up with my favorite game during the trip, Dune.
While on my flight back home, I kept wondering, what I'd miss about spending time with friends in SF the most and I realized, game nights were one of the things that i'd miss most. Because at home I didn't grow up playing board games, and was more into Counter Strike or Fifa style of games.
So on my flight back, thanks to Starlink and Qatar Airways, I hooked up claude and tried to build a version of dune for me to play with bots. But then I thought why not turn it into an RL environment.
What my agent thinks we did lol
Trained a GRPO RL policy to play Dune: Imperium – Uprising, a complex strategy board game where early sacrifices compound into late-game advantages. A typical 2-player game runs ~80-90 turns with a massive action space — agent & spy placement, deck-building, combat, and intrigue all interleaved.
Approach
Built a full game engine from scratch that enforces all board game rules, then trained a Qwen3 4B Instruct 2507 model in two stages — first SFT from games played between Minimax M2.5 and Grok 4.1 Fast, then RL against self-play using GRPO. The model learns to evaluate board states and select actions through natural language reasoning.
The key insights were a compact game state representation paired with available actions, and an RL rubric that rewards long-term strategic thinking over locally perfect moves.
Results
The 4B model consistently beats Gemini 3 Flash when both are given the same game state and asked to choose moves. A model with 1000x fewer parameters outplays a frontier model on a task that requires genuine long-term strategic planning.
More details and codebase coming soon.