Microsoft Research releases Orchard framework for scalable AI agent training

AIResearch

5 Aug

The open-source project covers software engineering, web navigation and personal assistant tasks, with training data and evaluation methods available to researchers

A hand interacts with a virtual dashboard displaying charts, graphs and global data. The image represents AI agent training, evaluation and digital research infrastructure. — ***Microsoft Research has released Orchard, an open-source framework for training and evaluating AI agents across multiple task types***

Microsoft Research has released Orchard, an open-source framework designed to train and evaluate AI agents across different tasks without requiring researchers to rebuild the underlying infrastructure for each project.

Orchard centers on Orchard Env, a lightweight, Kubernetes-native service that creates and manages isolated components at scale. It can support data collection, reinforcement learning rollouts and evaluation across software engineering, web navigation and personal-assistant tasks.

The release also includes three domain-specific training recipes, Orchard-SWE, Orchard-GUI and Orchard-Claw, alongside the training data and evaluation methods used to develop them.

The practical proposition is reuse. Microsoft Research says the same environment service can accommodate different benchmarks, training algorithms and agent systems, while allowing models to be trained within deployment harnesses including Codex, OpenClaw and ZeroClaw.

That addresses a mismatch identified by the researchers between simplified training environments and the more complex harnesses through which agents operate in practice. Orchard records a harness’s model calls as training data, with each rollout running in a separate container.

Smaller models tested on software engineering tasks

Orchard-SWE applies the framework to software engineering using Mini-SWE-Agent. It was evaluated on SWE-bench Verified, which tests whether models can navigate, diagnose and repair real-world codebases.

The researchers distilled 107,000 agent interactions from the MiniMax-M2.5 and Qwen3.5-397B open-weight models, covering a range of GitHub Issues. The workflow uses sections of unsuccessful attempts where the agent still completed productive work, rather than discarding those interactions entirely.

Microsoft Research reports that Orchard-SWE improved from a 61.4% baseline on SWE-bench Verified to 69.1% following Balanced Adaptive Rollout and 69.7% after dense-reward techniques were added. Performance reached 73% when candidate solutions were reranked using a four-billion-parameter value model trained on previous rollouts.

The system uses approximately three billion active parameters. Microsoft positions its results as approaching those of frontier systems more than 10 times larger, although the benchmark comparisons and performance claims presented with the release come from Microsoft Research.

Browser and personal-assistant agents

Orchard-GUI uses a four-billion-parameter vision-language model trained with 400 distilled demonstrations and 2,200 open-ended tasks.

It recorded scores of 74.1% on WebVoyager, 67% on Online-Mind2Web and 64% on DeepShop, producing a 68.4% average across the three web-navigation benchmarks.

For personal-assistant workflows, Orchard-Claw was trained on 200 synthetic tasks covering activities such as working with email, calendars, information searches and other tools. It completed 59.6% of tasks on Claw-Eval when allowed up to three attempts, according to the published results. That figure increased to 73.9% when the model was paired with ZeroClaw.

Training was conducted across ReACT, ZeroClaw, OpenClaw and Codex rather than through a single agent loop. Within the Codex harness, Microsoft Research reports that the model’s success rate increased from 18.6% before Orchard training to 51.5% afterward.

Microsoft Research has released the Orchard environment service, training pipelines, datasets and evaluation methods for use by the research community.