Teaching agents to play Paper Mario

I was playing Paper Mario and wondering how much of it we could get running at 60 FPS. The game has decompiled source available, so it seemed worth trying, even if some animations had to stay at their original rate. I was already giving coding agents difficult programming problems, and I wanted to see how far we could get with this one.

The agent would need to play the modified game to check its work, which seemed straightforward enough. The opening of Paper Mario mostly asks you to create a save file, get through some dialogue, and walk into a castle, so I expected the difficult questions to be about timing, rendering, and what we had broken.

I ended up coaching it through file select.

Getting to the door

This was early in 2026, with GPT-5.3-Codex. It could write useful software around the emulator and talk about Paper Mario, but actually operating the game was a different matter.

I told it to look up walkthroughs and use whatever information would help. A walkthrough would tell it to go through the door with a star on it, and there was a giant door with a giant star right there in the screenshot. To me, someone who has played games my whole life, the game was practically pointing at the destination. The agent still struggled to connect what it had read with where it needed to go.

I could watch the problem developing on screen. It would run a script that pressed A repeatedly while seeing only occasional screenshots. I was watching the actions happen; it was trying to reconstruct them afterward. It couldn't reliably tell whether a button press had worked, whether it was still in dialogue, or whether it had moved closer to its goal.

Savestates should have helped. I told it to use them, and then it lost track of which state held useful progress and what it was safe to reload. Even basic navigation became a long exercise in explaining things I normally do without thinking. I needed it to go up the stairs.

By then I was using agents for much more than programming. Given ways to find information and check their answers, they were useful for research, everyday questions, and thinking through problems. I had become used to finding another thing they could help with, so I hadn't expected walking through a door in Paper Mario to be where that stopped.

It could explain what I needed to do in the game, but I couldn't get it to actually do it. GPT-5.4 got farther, but I kept running into the same problems with navigation and checking whether an action had worked. GPT-5.5 was the first model where I started seeing a way through.

The renderer I wanted

I eventually put the 60 FPS work aside because I couldn't get the agents to make enough progress. I still wanted hi-res textures and better scaling in ParaLLEl, so I turned my attention there.

I liked playing with ParaLLEl and wanted to keep its rendering behavior and Vulkan path, but I also wanted the community texture packs I used with GLideN64. Switching renderers to get one feature meant giving up something else I liked, so my goal was fairly specific: make a fork that could handle the hi-res textures and scaling while preserving the rest of the emulator's behavior.

MasterKillua and other pack authors had already spent years redrawing Paper Mario's textures at high resolution, by hand. I wanted to make that work usable in the renderer I preferred.

To check the renderer changes, I needed the agents to reach more of the game. A title screen that looked right wasn't enough when every new room could reveal a rendering problem we hadn't seen. To feel confident in the changes, I wanted coverage across a whole playthrough, and eventually across other games. Building up a library of useful savestates takes time, and someone still has to inspect what renders. I wanted agents to help with both parts.

When I play a game, I want to enjoy it. I don't want every room to become an assignment to capture debug data. If the agents could reach a known location and inspect it after each change, I wouldn't have to keep making those trips myself. That was why I kept trying to get them to play.

Making the project workable

The codebase was dense, the relevant tests were missing, and the agents had trouble finding their way through it. I started adding N64 documentation and asking them to build tests, separate responsibilities, and refactor while preserving existing behavior.

GPT-5.3-Codex did useful work here, and parts of those refactors and tests survived into the version I use today. Getting the replacement textures to render correctly was still a struggle.

From March through May, the renderer work went through two serious attempts. The plans sounded plausible, and parts of the feature would work, but then another change would break something else or the agent would spend days chasing the wrong difference between two images.

In the first attempt, we compared our output with GLideN64, where the texture packs already worked. The renderers didn't draw everything the same way, though, and the agents confused some of those differences with texture problems. An agent trying to match the images could work on the shape or placement of something instead of fixing which texture was being loaded. I kept directing it back toward the texture pipeline while watching it pursue the wrong target.

In the next attempt, we converted texture packs ahead of time into our own PHRB format and served them through a common runtime. We kept that approach, but the tests could still report a texture hit while the picture was wrong. Finding a replacement didn't tell us whether it made it through the rest of the pipeline and appeared correctly on screen. It might look wrong or fail to show up at all. We needed to inspect the final image.

I spent a lot of time with the emulator on screen, watching the agents work. Getting comparable screenshots was itself a struggle. They could load the wrong state, misread a visual difference, or fix one artifact while introducing another. Asking them to play farther into the game to find more evidence usually got us stuck on gameplay again.

By the time I put the renderer work on hold, I knew the pattern. I could keep the agents busy, but I wasn't getting a renderer I wanted to play with.

When Fable arrived

I remember following the rumors before Fable 5 arrived, wondering whether this would finally be enough of a jump to make another attempt worthwhile. After spending so much time watching the earlier models struggle, I wanted to be ready if something came along that could get this project working.

I cleaned up the project so the next model would start with a manageable codebase, clear requirements, and a record of what had already failed.

The difference was visible within hours of trying it. Problems that had survived months of work began to resolve. Scaling worked. The hi-res textures looked like the textures they were supposed to be. After watching so many partial fixes, it felt almost like a one-shot result, even though the following days still involved testing, corrections, and my feedback.

Fable also investigated the problems differently. When a screenshot wasn't enough to explain an artifact, it wrote probes to find out more. It separated baseline rendering problems from hi-res, scaling, and tooling problems, and used other agents to review findings and challenge diagnoses.

One useful example was a vertical seam between strips of a scrolling backdrop. I had hoped getting the hi-res textures working might also resolve the scaling artifacts. It didn't. We had to separate the two problems: the seam could appear with no texture pack loaded, when those backdrop textures were allowed to scale up. Keeping them at native resolution avoided the gap but limited their detail. The investigation traced the gap to how the renderer assigned pixel coverage at an upscaled edge. Fixing that coverage let us keep the detail without the seam.

I still needed to watch the screen and push back on some of its conclusions. In one investigation, it decided that the texture pack contained gaps, but I had seen those textures render correctly in the reference renderer. Pointing that out sent it back to our code.

In another, it concluded that a pack texture was corrupt and prepared to remove it. I pushed back because I thought we might be hiding another bug, and the problem turned out to be in our renderer. The pack entry stayed.

We used code review throughout this work, but agents could read the code carefully and still give a confident, incorrect explanation for what was wrong on screen. We needed to test those explanations against what the renderer actually produced.

The agents reviewed frames too. Paired screenshots, independent reviewers, and deliberately challenging their conclusions became part of the process. That gave us more ways to catch mistakes, but my own observation could still overturn a diagnosis that looked settled.

The refactors, tests, documentation, and texture format from the preceding months all contributed, but I had reached the limit of what I could get the earlier models to do. I know how to program, and I know games and the N64, but I didn't know this area of renderer development. I was still learning how to work with agents, too.

Someone with more expertise might have been able to steer those models through it. I couldn't. With Fable, I could finally get the textures and scaling working well enough to use, even without knowing how to solve those renderer problems myself.

A computer for the playthrough

Once the renderer looked promising, I wanted evidence from farther into the game. I was hoping Codex computer use would help GPT-5.5 get there, and at the time it was only available on macOS. I bought a dedicated Mac mini so I could try it and give the agent a machine of its own.

The move between machines produced one of my favorite small scenes in the project. I pointed the agents at each other, and the agent on the Linux machine copied the project and wrote a briefing for the one on the Mac. There were platform problems to work through, but the receiving agent kept investigating and fixing them until it got the game running.

With a screenshot tool and a virtual controller, it could already see the game and send inputs. Trying Codex computer use didn't help as much as I had hoped, because it still struggled to tell what its inputs had done.

The dedicated machine did let me leave the task running, come back to see how it was doing, and steer it over time without needing that computer for something else. That gave us room for an extremely long playthrough, with the controls and my guidance developing as we went.

Giving the agent control of time

I had been telling the agents about tool-assisted speedruns, or TAS: exact inputs, counted frames, and savestates. Watching the agent overshoot or lose track of an action made me keep asking it to slow down. Advance a few frames, take a screenshot, and check what happened.

What helped was turning that advice into code it could use. I had been thinking about Claude Code's Dynamic Workflows, where Claude writes an orchestration script that can coordinate many agents in parallel or through successive stages, with conditions determining what happens next. If Claude had been trained to work this way, I wondered whether I could give it a similar structure for gameplay. The idea was to put the sequence in code: send an input, advance a known number of frames, inspect the result, and decide what to do next.

We developed a TypeScript TAS runtime with frame-counted inputs, neutral frames, save and load points, screenshots, and monitoring. That let the game wait while the agent thought, and a sequence could include checks and recovery points instead of being one long burst of button presses.

The controller replay shows the inputs from the script as button presses and stick movements, making it easier to follow what the code is asking the game to do.

The agent also had the decompiled game source and memory reads, which let it research a room's logic and inspect the state that controlled it. That was useful when a screenshot showed where Mario was without explaining why a door wouldn't open.

At that point, I felt we had given it every useful advantage we could think of. It could inspect the game's logic and state, control exactly when inputs happened, and go back to try again. I wanted to see how far it could get with all of that available.

Even with better controls, I still had to coach it through decisions like what to do after winning a difficult battle. I wanted it to go back, heal, and save so it could preserve that progress. Given a goal like finishing Chapter 1, it would keep throwing itself forward when a small retreat would have left it in a much better position.

My role became watching, describing what I saw, and helping it break the task into pieces it could manage. As the controls improved, it could work for longer stretches between those conversations.

The nine-day run

The playthrough reached from the game's opening to the Chapter 2 title screen over nine calendar days, including breaks and work on the tools. That distinction matters when reading the progress chart: a long flat stretch means there was no new recorded milestone. It doesn't tell us whether the agent was struggling or the run had paused.

The strip under the chart shows when Codex produced log events, including research and work on the tools. It helps show the breaks in the run, but it isn't a measure of continuous gameplay or model thinking time.

The fortress became a long routing struggle. It reached the jail from the wrong side, worked through other rooms and battles, and eventually used research in the decompiled source to find the trap block that led into the cell. Getting there also meant recovering from an earlier checkpoint.

Getting through the second Jr. Troopa encounter took more than two hours, including work on button timing. The recorded win used ordinary battle inputs. Other parts of the run needed help: the Koopa Bros boss used a disclosed memory-write assist after timed action commands proved unreliable, and the Bill Blaster fight was abandoned and bypassed from a save.

When I say it reached Chapter 2, that includes my coaching, the tools we developed during the run, and the assisted obstacles. With that help, it was reaching places in the game that I had previously been unable to get it to reach.

At a smaller scale, the action timeline follows a half-hour stretch where a stall led to research into a hidden Koopa ambush. The game's source suggested what was happening, and a memory read helped confirm it.

As I watched the playthrough, I was seeing very few graphical bugs, and I liked what was on screen. Earlier versions had been unusable for me, even when it was exciting that they rendered anything at all. Now, with some issues still there, this looked like the way I wanted to play Paper Mario.

We still needed broader coverage, but the run gave me much more confidence in the renderer, along with more locations to inspect and a set of controls we could use again.

From a playthrough to a benchmark

When I started writing about the project, I wanted to know how other models would handle the same task. The tools developed during GPT-5.5's run became the basis for all the subsequent evals and marathons.

Paper Mario Bench began with a smaller, fixed-budget route through the opening. That made repeated comparisons more affordable and put a stopping point on runs that could otherwise spend hours repeating unsuccessful actions. The site contains the results, the scoring rules, and the behavior analysis.

Running those comparisons also meant expanding the hardware beyond the dedicated Mac. I added more RAM and an Intel GPU to my Proxmox server so I could run more virtual machines with graphics support for the emulator.

A route that was a serious obstacle during development is now something many models can complete. In September, Astra finished all 15 opening attempts and Fable 5.1 finished 12 of 15. That gets them to the first Bowser battle, with Chapter 1 still ahead.

For models that now finish that route consistently, the longer marathons are more interesting. Both September marathons used medium effort and a six-hour budget. Fable 5.1 found the hammer, defeated the first Jr. Troopa, and left the playground. It kept working on combat and recovery after that, but recorded no later story milestone. Astra reached Koopa Bros. Fortress and lowered its first staircase. Neither completed Chapter 1, but the difference in how far they got gives us more to investigate than finishing the opening alone.

Fable 5.1 wasn't the strongest Claude marathon result. An earlier six-hour Opus 5 run at medium effort reached Toad Town, farther than Fable 5.1 managed here. Astra went farther still, reaching Koopa Bros. Fortress. These are individual runs under different harness versions, but they show why I want to test what newer models actually do rather than assume each release will improve on the last.

GPT-5.5's original run still reached farther because it had much more time, along with my coaching and the disclosed assists. Looking at Astra's results, I thought it had been on track to pass that endpoint if the run had continued, though that remains an impression rather than a result. Cost is why I keep the marathon budget to six hours. The later models also had the tools available from the start, while GPT-5.5 helped build them during its run.

What I found encouraging in the results was how far Astra had progressed with so little intervention. After all that time explaining how to get through file select, I was now considering how much more of a run like this I could afford.

A low token price doesn't make an attempt cheap if the model keeps spending without getting through the task. On difficult work, I care about whether it can finish as well as what it costs to try. Long runs also make cache-read pricing more consequential, so I want to compare progress with the cost of each attempt.

For the two September marathons, the recorded completed-token usage gives API-equivalent estimates of about $193 for Astra and $68.05 for Fable 5.1. Astra’s estimate prices all non-cached input as cache writes because Codex did not reliably report the split. Both estimates exclude unreported usage and are not subscription bills. The gap caught my attention, but Astra also got much farther, so this isn't a comparison of the price of doing the same work.

What I still want to build

parallel-ish is open source. I want to keep testing it, clean up the benchmark and its tools for an open-source release, and let the experiments evolve as we find more useful things to measure. How much I can run will depend on the cost and the work involved.

The Paper Mario Bench introduction goes into the directions I'm considering for the benchmark: limiting the tools and guidance, changing the challenge, and deciding what makes a fair budget for comparing models. Those questions will matter as more models finish the opening consistently.

I still want to return to the 60 FPS idea. More immediately, I want agents to reach a scene, check a change, and repeat that check without needing me to guide the whole trip. After reviewing Astra's results, I feel much closer to being able to build that loop.

I'm also exploring whether agents could help translate saves between the Super Nintendo and DS versions of Chrono Trigger. The differences between the games make it more complicated than copying a file, but an agent could help investigate them by playing through scenarios, inspecting memory and saves, and comparing what happens in each version. It would still take time, testing, and investment, but I can see a way to approach it now that doesn't depend on me manually playing through every comparison.

When I next sit down to play, I want to see the improvements and enjoy the game. The agents can help make sure it works before I get there.