PS-Env

What PS-Env Is

PS-Env is an OpenEnv-style reinforcement-learning environment for PlayStation 1 games. At the bottom sits the game itself, a Crash Bandicoot: Warped disc image and a PS1 BIOS, neither of which ever enters the repository. The SwanStation libretro core runs them headlessly; it is downloaded as a binary and loaded at runtime with dlopen. A small C++ host exposes exactly the four things we need from the core: press buttons, grab the frame, read memory, and save or restore state. pybind11 lifts that host into Python, where a FastAPI server offers reset() and step(action) over a WebSocket JSON protocol. The agent, a Nature DQN, sits at the far end of that socket and only ever sees stacked frames and rewards.

The Loop

reset() restores a save state, so every episode begins on the same frame of Toad Village with the same four lives. step(a) turns one of 13 discrete actions into a button bitmask, holds it for eight emulator frames (frame skip 8; the recovered design used 2, and the training story below explains the change), and returns the last of those frames as an 84 by 84 grayscale image together with a reward and a done flag. The actor keeps the last four frames, so the network's input is 4 by 84 by 84.

What the Agent Sees

the emulator's frame

t − 3

t − 2

t − 1

t

What the Agent Can Do

Thirteen discrete actions, each a PS1 button bitmask whose bit index is the libretro joypad id (Cross is bit 0, Square bit 1, the d-pad bits 4 to 7). Cross jumps, Square spins, and Up is forward, because Toad Village runs away from the camera.

• 0NOOP0x0000
• 1UP0x0010
• 2DOWN0x0020
• 3LEFT0x0040
• 4RIGHT0x0080
• 5JUMP0x0001
• 6SPIN0x0002
• 7JUMP_LEFT0x0041
• 8JUMP_RIGHT0x0081
• 9JUMP_UP0x0011
• 10SPIN_LEFT0x0042
• 11SPIN_RIGHT0x0082
• 12SPIN_UP0x0012

Rewards Live in RAM

Nothing the reward needs is on screen in a usable form. Fruit collected, boxes broken and lives left live somewhere in the console's 2 MB of RAM, at addresses nobody documented in 1998. The team found six of them with a memory scanner (search for a known value, change the game state, search again, until one address survives), and the reward function reads them as single bytes every step with no image processing at all. The 2026 reproduction added Crash's position (CRASH_Z, a 32-bit word) for dense progress shaping and for the level-exit check, since the recovered "78 = level done" value was never observed on the real disc.

Two more charges exist only in the trainer, and the training story explains why they were needed: a stall cut (an episode ends after a run of steps without new progress) that costs a lost life, and a cost of 1 on every step that makes no new progress.

The Training Architecture

The talk's architecture diagram shows the distributed design: actors on an epsilon ladder from 1.0 down to 0.01, each with its own emulator, push transitions to Redis; a GPU learner samples batches of 32, trains the policy net (three convolutions, a 512-unit layer, 13 outputs), hard-copies it to the target net and publishes weights; TensorBoard reads its logs. The reproduction below ran the single-process trainer (agent.train) against one emulator server instead, because the emulator, not the GPU, was the bottleneck: about 50 steps per second at frame skip 8 on a g5.xlarge, with the A10G mostly idle.

The Training Story

The hackathon team reported the first level completed after roughly 500K steps with the distributed trainer. The 2026-09 reproduction on one g5.xlarge started from that recovered design, single-process, and found that it did not finish Toad Village at all. From scratch at frame skip 2 the agent learned to run forward and spin and never cleared the first pit: 0 completions in 186 episodes and 805K steps. A reverse curriculum at frame skip 2 wedged at the same pit (about 10% pit success for 200K steps with uniform one-step replay, and about 10% for another 200K with prioritized replay and three-step returns), and frame skip 12 never left fine stage 142.

Everything that moved the needle after that is below, in the order it was found. The numbers come from the training logs and from section 15 of the design reference.

One earlier completion from the level start exists outside this chain: at step 820,822 of a frame-skip-2 curriculum run, a lone probe episode from the level start at stage 133 finished the level in 3,251 steps (reward 6,887). It was 1 of 39 such probes in that log and was never followed up.

Results

Run 14

Run 14 resumed the frame-skip-8 chain from step 1,525,322 at epsilon 0.05 with every lever above in place, 24,269 demonstration transitions loaded, and ran to 2,000,000 steps: 475K steps in about 2 h 25 min, at 61 falling to 54 steps per second.

The Plateau

The from-start completion rate did not trend over the run. A logistic fit of completion against step has a slope of +0.05 per 100K steps with p = 0.79, and the quadratic term is negative, an inverted U. The last completion came at step 1,933,153; the final 66K steps (151 episodes) produced none, and the run ended with three consecutive from-start game overs. Stalls stayed at about 54% of from-start episodes in every bucket, while the game-over share went 47%, 32%, 41%.

Evaluating the Checkpoints

Two checkpoints were evaluated afterwards from the level start with agent.evaluate: no stall cut, a 1,600-step cap, 40 seeds at epsilon 0.05.

Greedy Play Is One Episode

Epsilon 0 is deterministic: select_action draws rng.random() < 0, which is never true, so with a deterministic emulator every seed replays the same episode. Filmed with tools.record_episode, the greedy dqn_final.pt reaches the crate beside the first tent, 36,594 units into a 164.5K-unit level (22%), and jumps in place for the remaining 1,300 or so steps until the cap. The recorder's return for that episode is 1,638.64 against the evaluation server's 527.62, because the recorder steps the emulator one frame at a time and the server eight, but both hit the cap without completing. Training at epsilon 0.05 was measuring a policy plus noise; the policy on its own is stuck at one crate.

The Films

Two completions from run 14 were replayed frame for frame by feeding the trainer's recorded actions back through the deterministic emulator (tools.record_episode --actions): success_01929825.npz (154.1 s, 9,245 frames, 1,156 steps) and success_01933153.npz (157.5 s, 9,452 frames, 1,182 steps). They are the first from-start completions on film. The earlier completion at step 1,479,301 was recorded before the success file kept every step, and diverged on replay.

Charts

Five charts rendered with matplotlib from the trainer's per-episode log lines across the 27 training logs, by tools/plot_training.py (the originals live in docs/charts/). Buckets are by the step at which an episode ended.

Completion Leaderboard

Every completion the trainer recorded is a replayable action list, so the level's counters can be read back exactly for each one: fruit collected, boxes broken, lives lost and steps taken. The 30 completions of runs 13 and 14 (17 from the level start, 13 from curriculum start states) were replayed one emulator frame at a time with tools/replay_stats.py; every replay matched the trainer's stored observations frame for frame and ended with the level complete after exactly the number of steps the log reports. The records below cover the 17 from the level start, the only ones that played the whole level with all their lives. Steps are agent steps of eight emulator frames; seconds are frames divided by 60.

The 13 completions from curriculum start states are kept out of the records: those episodes are cut at the first lost life, and most began part-way into the level. Among them, step 1,742,347 (from fine state 0, which is at the level start) collected 31 Wumpa in 1,476 steps, step 1,777,001 (state 53) broke 9 boxes, and step 1,658,605 (state 76) reached the exit in 472 steps (62.9 s). All 30 records, each with a progress curve sampled every 10 steps, are in assets/completions.json (63 KB), a copy of docs/data/completions.json.

Honest Status and Next Steps

Run 14 ended on a plateau. The from-start completion rate showed no trend over its 475K steps, the tail degraded (no completion after step 1,933,153, and 0 of the last 39 from-start episodes), and stalls stayed at about 54% of from-start episodes in every bucket. The greedy policy by itself does not finish the level from the start; the 4.88% is a policy plus epsilon 0.05 noise. Training longer at these settings is not the next step. The stall segment is.

What the evidence weakly favours:

• Ship a checkpoint near 1.93M steps rather than dqn_final.pt. The last completions came at 1.92M to 1.93M, dqn_final.pt loops in place instead of dying, and over 40 seeds at epsilon 0.05 it completed 0 where dqn_01925000.pt completed 1.
• Attack the 54% stall share directly: find where the from-start stalls happen, and put curriculum states, demonstrations or a stronger idle charge there, before spending more steps on the whole level.
• A second level. Bone Yard is playable: its save state is scripted from the Warp Room, it runs toward the camera so progress carries the opposite sign, its in-level WARP_STATE is 38 and its counters sit 0x27C above Toad Village's. The solver covered 40,275 units in 38 macros, and a 1,000,000-step run finished 2,138 episodes with zero completions, most likely because that segment mapped only the first 40,275 units, so the reverse curriculum never had a mapped path to an exit to walk back from. The next attempt should map the level through to its exit before training on it.

Cost and Provenance

Everything above ran on one on-demand g5.xlarge in us-east-1 (4 vCPU, one A10G, a 100 GB gp3 root volume) at $1.006 per hour, over four sessions. The first ran 59.17 h ($59.52), of which about 42 h were idle: run 14 had finished at 08:51 UTC on 2026-09-06 and ssh from a new network was blocked by the security group until the instance was stopped on 2026-09-08. A second session of 1.17 h ($1.17) read the results, filmed the completions and ran the evaluations; a third of 0.33 h ($0.34) came up impaired and was force stopped; a fourth of 8.75 h ($8.80) ported Bone Yard and ran its solver and its training run. The running time is measured from the 833 CloudWatch CPU datapoints the instance publishes only while it runs. Total: 69.42 h, about $69.8, plus about $8 a month of EBS while the instance sits stopped.

The original source code of PS-Env is gone. It lived in two repositories on the hackathon laptop, which is no longer accessible. The code on GitHub is a reconstruction from what survived: the team's writeup, a blog draft and its figures, the recording of the April 2026 talk, the architecture diagram shown during it, two whiteboard photos from the design session, and the pre-hackathon proposal and emulator notes. Every address, protocol field and hyperparameter in the code is traced to one of those sources, and the design reference marks what had to be reconstructed. The reconstruction was then run end to end on the real core and disc, which is where the numbers on this page come from.

The repository on GitHub, and docs/DESIGN_REFERENCE.md, the single source of truth for the recovered facts and the record of the 2026-09 training.