Research2026

Make Code as Policy Great Again Frontier Agents Write, Call, and Evolve Robot Tools

URAI · Universal Robot–Agent Interface

Shijia Ge¹
Alex Zhou¹
Jianshu Zeng²
Yexing Wan
Di Wu³
Zelin Zheng¹
Yazhe Wang³
Zhiqi Jia¹
Xuan Shangguan
Jay Zhu¹
Yijun Liu⁴
Lingyu He¹
Sihang Wu¹
Xiao He^1,†
Hongcheng Gao^1,4,†

¹Hexafuture Inc. ²Peking University ³Beijing Institute of Technology ⁴Tsinghua University

^†Corresponding authors

Paper coming soon arXiv coming soon Code

Models build executable tools, code handles multi-phase motions, and models decide what to do next.

An AgileX robot arm places a block on a tic-tac-toe grid while a person places a green block; fading outlines trace the arm's earlier poses.

Two AgileX robot arms: the left holds a bottle while the right grips its cap; fading outlines trace the right arm's approach.

Frames from the demonstration video on our AgileX dual-arm robot. Fading outlines indicate earlier arm poses within one tool call: the reusable pick-and-place tool (blue) and a task-specific tool written by the programming agent (dashed).

Overview

Abstract

Frontier models can control robots, but reasoning through every reach, grasp, and retreat makes manipulation slow and token-intensive. We revisit code as policy with a different division of labor: models build executable tools, code handles multi-phase motions, and models decide what to do next. We introduce URAI (Universal Robot–Agent Interface), which couples a programming agent that constructs robot tools with an execution agent that uses them in a feedback loop. The programming agent writes reusable and task-specific tools from task intent and refines them through execution feedback and human guidance. The execution agent selects and parameterizes these tools from current observations; each call runs a complete motion locally before returning control to the agent. Unlike delegating subsequent decisions to a generated program, this design retains model-level decision-making between tool executions. Validated tool revisions persist across episodes without updating foundation-model weights, and a shared GUI and API make the same tools available to humans and agents. Across five RoboDojo tasks and four frozen execution agents, URAI raises aggregate success from 18.0% to 53.0% relative to direct fingertip control, with the largest gain on Swap Blocks; with the same tools, a program written in advance reaches only 24% against 56% for two agents deciding after each call. Three of the four agents also finish episodes 1.3–1.5× faster with 1.5–1.7× fewer execution-agent output tokens; DeepSeek-V4-Flash’s cost barely changes. We further evaluate URAI on seven real-world AgileX dual-arm tasks, spanning object manipulation, cloth folding, and human-interactive tic-tac-toe. URAI connects the coding and decision-making capabilities of frontier agents, organizing robot control around reusable tools that agents can both invoke and revise.

01Method

Agents write the tools, the tools run the motions, the agent decides

A programming agent writes and validates robot tools. An execution agent selects a tool and its arguments from the current observation; one call runs a complete multi-phase motion locally, and the agent decides the next step from the feedback. Adaptation happens in the tools, never in the model weights.

(a)Closed-loop execution

Within an episode: one tool per model call, run locally.

Execution agent → tool call (arm, pixels, …) → URAI: Ground (pixels → 3D), Plan (timed path), Execute (local, 50 Hz) → feedback + new observation → next decision.

Adapted from the paper’s Figures 1(a) and 2(b). The execution agent, a frontier model or a person using the GUI, selects and parameterizes tools from observations and feedback; the shared GUI/API backend grounds, checks and executes each multi-phase motion locally. Photos are demonstration-video frames; fading outlines indicate earlier arm poses within one tool call. After the outlines, each photo plays on as the recording of the same run, at 3× and 4× speed; waits while the agent decides are cut.

(b)Tool programming and persistent evolution

Between episodes: the programming agent revises φ = (L, C, P); foundation-model weights θ stay fixed.

Failures and human guidance → programming agent → candidate φ = (L, C, P) → validate (contracts, regression, cost) → freeze → accepted tools in URAI for later episodes. Foundation-model weights θ stay fixed.

Adapted from the paper’s Figure 2(a). φ = (L, C, P) is the external system: the tool collection L, the executable tool code C, and the execution strategy P, meaning the task guidance and call conventions given to the execution agent. Each accepted version is tested and frozen before evaluation. In our experiments the programming agent is Claude Fable 5.1 in Claude Code (Claude Opus 5.5 for one task-specific pouring tool). The failed check is drawn for illustration.

(c)What one model call controls

One schematic episode under each way of driving the robot with a frontier model.

Direct end-effector control

one call per end-effector command
the model decides every step

many calls: slow and token-heavy

model calls 9

Code as Policies

the model writes the program first
the code decides what runs next

handles only what the code checks

model calls 1

Code as Policy with URAI

one call per complete, timed motion
the agent decides each next call

agent in the loop between motions

model calls 4

Adapted from the paper’s Figure 1(b); schematic, not to scale. Direct control queries the model for each end-effector command; Code as Policies delegates subsequent decisions to generated code, which may query perception; URAI returns control to the agent between tool executions. Tool executions differ in length because complete motions do not all take the same time: on RoboDojo, URAI used 3.5–10.0 motion calls per episode lasting 29–72 s each, against 21.0–25.6 native calls of 19–20 s.

02Results

More success from fewer, longer calls

Five RoboDojo tasks with four frozen execution agents, seven real-robot tasks on an AgileX dual-arm platform, and one case study of tool evolution. All numbers are from the paper.

Simulation: five RoboDojo tasks, four execution agents

Claude Fable 5.1 and Claude Opus 5.5 in Claude Code, GPT-6 Astra in the Codex CLI, and DeepSeek-V4-Flash in an open-source coding agent control the same simulator either through URAI tool calls or natively, by commanding fingertip targets and gripper values. Each task has one episode per seed 0–4, each in a fresh agent session; only RoboDojo’s own terminal state counts as success.

53.0% vs 18.0% Aggregate success

Through URAI versus direct fingertip control, over four agents and five tasks. Every agent succeeds more often through URAI.

56% vs 24% Deciding after each call

Same tools, two agents (GPT-6 Astra, Claude Opus 5.5): deciding after each call versus one program written in advance and run without the model.

1.3–1.5× faster 1.5–1.7× fewer output tokens

For three of the four agents. DeepSeek-V4-Flash’s cost barely changes: about 5% less time and 9% fewer tokens.

Average success rate over the five tasks, per execution agent

Five episodes per task (seeds 0–4). The program control was run for GPT-6 Astra and Claude Opus 5.5.

DeepSeek-V4-Flashopen-source coding agent

8.0%

48.0%
GPT-6 AstraCodex CLI

16.0%

24.0%

52.0%
Claude Fable 5.1Claude Code

16.0%

52.0%
Claude Opus 5.5Claude Code

32.0%

24.0%

60.0%

Success, time and tokens per task

Execution agent, condition	Pour liquid into cup	Pour balls into vase	Fold clothes	Swap blocks	Stack blocks	Avg. SR (%) ↑	Time (min) ↓	Tokens (k) ↓
DeepSeek-V4-Flash, native	0.0	0.0	0.0	0.0	40.0	8.0	21.4	44.3
DeepSeek-V4-Flash + URAI	40.0	60.0	0.0	100.0	40.0	48.0	20.2	40.3
GPT-6 Astra, native	0.0	0.0	20.0	0.0	60.0	16.0	13.8	6.9
GPT-6 Astra, program	40.0	40.0	0.0	0.0	40.0	24.0	7.0	4.6
GPT-6 Astra + URAI	40.0	60.0	0.0	100.0	60.0	52.0	9.4	4.7
Claude Fable 5.1, native	0.0	0.0	20.0	0.0	60.0	16.0	25.0	52.8
Claude Fable 5.1 + URAI	20.0	40.0	20.0	100.0	80.0	52.0	17.4	30.4
Claude Opus 5.5, native	40.0	0.0	40.0	0.0	80.0	32.0	17.4	32.3
Claude Opus 5.5, program	0.0	40.0	0.0	20.0	60.0	24.0	15.1	38.0
Claude Opus 5.5 + URAI	40.0	60.0	40.0	80.0	80.0	60.0	13.4	20.0

Success is the official terminal state, five episodes per task (seeds 0–4). Avg. SR is the mean of the five per-task rates. Time and tokens average all episodes, failures included, and count execution-agent output tokens with reasoning. Program: one program over the same tools, written in advance and run without the model; its time includes both phases. Bold: within each agent, the best Avg. SR and the lowest time and tokens. Swap blocks: its tools bundle a task-level place-on-pad procedure with button pressing, so URAI and native control differ in tool semantics as well as call granularity; the program rows share these tools and give the cleaner contrast. Without swap blocks, URAI still leads for every agent, 42.5% against 22.5%. A person driving the same tools through the GUI succeeds in 23 of 25 simulated episodes.

Where the time goes. URAI replaces many short motion calls with fewer, longer ones: 3.5–10.0 calls per episode lasting 29–72 s each, versus 21.0–25.6 calls of 19–20 s natively. For Claude Fable 5.1, Claude Opus 5.5 and GPT-6 Astra, time outside robot and observation commands falls by about 45%, 38% and 32%. DeepSeek-V4-Flash shows the opposite pattern: its motion calls fall from 25.6 to 10.0, but time outside commands rises from 441 to 509 s. Coarser motion calls alone do not ensure lower total cost; the execution agent’s observation and decision behavior also matters.

Real robot: seven AgileX dual-arm tasks with GPT-6 Astra

Our AgileX PiPER-X dual-arm system with four RealSense RGB-D cameras runs seven tabletop tasks, all with GPT-6 Astra as the execution agent calling URAI tools. Three tasks have a published result of the same model on other hardware; for these, each URAI number is the mean over three trials, failures included.

Unscrew bottle cap

3.5× faster

Time per trial (min)

URAI5.1

GPT-Policy, published (R3)17.9

Success 3/3 with URAI and 3/3 for the reference.

Tic-tac-toe against a human

2.7× faster

Time per game (min)

URAI5.0

GPT-Policy, published (R8)13.6

Success 3/3 for both; the robot lost none of three games.

Block into bowl

2.4× faster

4.7× fewer output tokens

Time per trial (min)

URAI1.0

Robocurve, published2.5

Output tokens per trial

URAI443

Robocurve, published2.1k

Success 3/3 with URAI; 19/20 for the reference.

The references are published runs of GPT-6 Astra on other hardware: GPT-Policy’s dual-arm tasks R3 and R8, and Robocurve’s block placement under direct end-effector control on YAM arms with a 25% speed cap and a 20-call budget. There is no same-robot step-by-step baseline, so these ratios are not causal estimates of a speedup. Block-into-bowl time is 61.3 s per trial (shown as 1.0 min); the first bottle-cap trial excludes unmetered diagnosis and repair.

10/10Hot-dog serving

7/10Toss blocks into bowl · 91.7% mean progress

10/10Pour blocks

8/10Fold clothes · 90.0% mean progress

Four tasks have no published same-task reference and are reported for URAI alone, ten trials each; success requires the whole task.

Tic-tac-toe, key frame 1 of 6 — Key frames of the seven AgileX tasks under URAI with GPT-6 Astra, from the initial scene (t1) to completion (t6); chips name the tool called, blue for reusable tools and dashed for task-specific ones. Fold clothes is the demonstration-video run; the other rows are evaluated trials.

Tic-tac-toe, key frame 2 of 6 — Key frames of the seven AgileX tasks under URAI with GPT-6 Astra, from the initial scene (t1) to completion (t6); chips name the tool called, blue for reusable tools and dashed for task-specific ones. Fold clothes is the demonstration-video run; the other rows are evaluated trials.

Initial scenes of five AgileX dual-arm tasks seen from the front camera: unscrew bottle cap, hot-dog serving, toss blocks into bowl, fold clothes, and pour blocks. — Initial scenes of five AgileX dual-arm tasks, seen from the front scene camera and cropped to a common scale (paper appendix). Tic-tac-toe and block into bowl are not pictured. Backgrounds are blurred for clarity; the table, arms, and objects are unmodified.

Tool evolution: revising one tool for fold clothes

Steered by the author in natural language, the programming agent revised the existing two-point PnP tool seven times instead of writing a folding routine. The initial version rejected every click on the shirt as a large rigid object; the revisions add sheet pinching with local thickness checks, synchronized dual-arm transport, outward wrist tilt while carrying, vertical release, and a drop height and clearance taken from depth. Each version keeps the call signature, ships with unit tests, and has no branch on seed, garment, or task name.

The frozen final version is tested by the four frozen execution agents in fresh sessions, choosing pixels themselves. With this tool, choosing pinch and drop points is the agents’ main difficulty: in most failed GPT-6 Astra and Claude Opus 5.5 episodes, the first sleeve pinch sits at the sleeve tip, where the fingers close on nothing.

8 versionsthe initial PnP tool and seven revisions, no weight updates

3 of 20 · 2 of 20episodes solved by the four agents on seeds 0–4 and on seeds 5–9

4 of 5a person choosing pixels through the GUI, with the same tool

Scope. The measurements cover five simulated and seven real-robot tasks with few episodes per cell; five episodes per cell do not establish per-task differences. Tool programming and the author’s steering of it cost unmetered time and tokens that may offset the execution-phase savings, and the interface is validated on one simulator and one position-controlled dual-arm platform without force regulation.

03Citation

BibTeX

@misc{ge2026urai,  title = {Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools},  author = {Shijia Ge and Alex Zhou and Jianshu Zeng and Yexing Wan and Di Wu and Zelin Zheng and Yazhe Wang and Zhiqi Jia and Xuan Shangguan and Jay Zhu and Yijun Liu and Lingyu He and Sihang Wu and Xiao He and Hongcheng Gao},  year = {2026}}