From Q-learning to LLM agents

The first agent I trained lived in a grid. One cell paid +1, one cost −1, one was a wall, and the rest was empty floor. It knew nothing. It learned from rewards alone, with the Bellman equation doing the bookkeeping:

• at episode 0 it wandered;
• at episode 199 it knew more, and still failed;
• at episode 599 it was getting there;
• by episode 799 it solved the maze every time.

Today “agent” almost always means a language model in a loop: read the state, pick a tool, look at the result, repeat. The vocabulary survived intact. Agent, environment, reward, policy. What did not survive is the training. Nobody gives you 800 episodes on real users before the agent has to behave.

Fig. 1 — An agent learning a mazeQ-learning · 3 × 4 · γ 0.9

Loading interactive figure…

Static version: my first agent was still failing at episode 199 and solved this kind of maze by episode 799.

Train one episode at a time and watch the arrows spread backwards from the +1, like a stain. Set the step penalty to zero and the agent gets lazy: it still arrives, in no hurry at all. Set exploration to zero and it may settle for the first path that happened to work. The maze makes something obvious that is easy to forget with LLMs: the reward function is the design. Penalties are what make the agent finish quickly.

With an LLM agent the policy arrives pre-trained, the model plus your prompt. The reward is still yours to write. It becomes the eval that scores the whole trajectory instead of only the final answer. The step penalty becomes the limit on steps, tokens and time. And one property of Q-learning carries over exactly: the values only stay right while the environment stays the same. Change a tool, an API or a schema and the old policy is stale until you run the evals again.

799 episodes for a three-by-four grid. Your agent in production gets zero. Write the reward first.