An RL Environment Is a Programmable Simulator of Your Business

Goku Mohandas defines the most misunderstood piece of a learning loop concretely: “An RL environment is a programmable simulator of your business.” It has four parts.

1. State — the situation the model is looking at: a support ticket, a code repo, a financial transaction, a camera frame, a medical record.
2. Action space — what the model can do: write a reply, call a tool, edit a file, place a trade, send a motor command, order a lab test.
3. Transition — how the world updates after the action: a real tool execution, a sandboxed environment, or a simulator.
4. Reward — something that scores the action against the outcome that actually matters: “Did the ticket get resolved? Did the tests pass? Did the trade make money? Did the bot grasp the burrito?”

The model runs this loop millions of times (with heavy synthetic-data generation), and an RL algorithm (PPO, GRPO, DAPO, DPO-on-rollouts) updates the policy weights to make high-reward actions more likely over time. Mohandas stresses that the sandbox in the transition step is the proprietary part: “The sandbox is yours. No frontier API gives you this and this is your IP.” Building the simulator of your own business is what no rented model can hand you.

Related

• RL Learns Outcomes While Fine-Tuning Imitates Labels — the companion distinction: this note defines the environment, that one explains why training in it beats supervised fine-tuning
• The Moat Is the Learning Loop Not the Model — the RL environment is the load-bearing box in the loop that becomes the firm’s IP
• Loops Make Sense Only With a Fixed Feedback Signal — the reward function here is the enterprise-scale version of the “fixed feedback signal” that makes any loop work
• Using a Frontier Model Leaks Your Institutional Knowledge — labs embed engineers specifically to build this environment from your context; owning it keeps the signal inside
• Benchmarks Measure the Frame, Not the Framer — the reward function is a frame you design; deciding what counts as a resolved ticket smuggles the same human judgment a benchmark rubric does, and that framing is the part that stays yours