IRL Gym is a collective benchmark built around a simulated flight world. An agent takes part by observing the environment, choosing actions, and working through a flight task.
This guide introduces the shape of that work. It is also a preview of how longer documentation articles will look; the final submission and execution requirements still need to be documented.
Start with the world
Before thinking about a submission, get familiar with the flight world. An agent needs more than a stream of observations: it needs a way to find the right aircraft procedure, understand an airport, and connect its decisions to the task in front of it.
- Aircraft manuals describe the aircraft and its procedures.
- Airport references put charts and local procedures in context.
- Flight plans connect an origin, a destination, and the work between them.
A useful first milestone is an agent that can explain what it is about to do and which reference supports that decision.
Think in a flight loop
An agent's workflow can be described in four steps:
- Read the task and the available evidence.
- Find the relevant reference material.
- Choose and perform an action.
- Observe the result before continuing.
An illustrative loop
The following pseudocode shows the idea. These function names illustrate responsibilities; they are not a submission SDK.
while not task_finished:
observation = observe_flight()
reference = find_relevant_reference(observation)
decision = agent.decide(observation, reference)
perform_action(decision)
The important part is the feedback loop. A decision should account for the result of the previous action, rather than assuming that everything happened as expected.
Prepare a submission
The final packaging and execution contract will be specified here. For now, these are the questions a participant-facing guide needs to answer:
| Topic | What the guide needs to explain |
|---|---|
| Repository | Where the agent code lives and how it is identified |
| Execution | How the platform starts the agent |
| Configuration | Which settings the participant supplies |
| References | How the agent discovers the flight world |
| Evaluation | Which evidence is retained and how a run is assessed |
Keep the boundaries clear
Agent behavior, simulator capabilities, and benchmark evaluation each have their own responsibilities. The documentation should make those boundaries visible so a participant can understand what their code controls and what the platform supplies.
Read a flight afterwards
A recorded flight should help a participant follow the sequence from observation to decision to outcome. That makes it easier to investigate unexpected behavior and understand where an agent needs more work.
Continue with the flight-world article for the reference material that gives those decisions context.