vLLM Agentic API
Responses and Messages protocols, state, streaming, and gateway-owned tool execution.
vLLM+
llm-d+
Agentic API
arrows navigate · n notes · o overview
llm-d community demo
Open agents.
Distributed inference.
One stack.
Claude Code and Codex running on vLLM through llm-d and the vLLM Agentic API.
COREWEAVE · KUBERNETES
DGX SPARK · KIND
● WORKING DEMO / Responses + Messages / server-side web search
Francisco Javier ArceoSenior Principal Software Engineer · Red Hat AI
Three projects · one agentic path
llm-d
Kubernetes-native routing and distributed inference orchestration above model servers.
vLLM
High-throughput inference and the model-facing OpenAI-compatible contract.
The seam stays inference-provider neutral: harness behavior and tools do not need to move with the model backend. One API—from a single local node to distributed Kubernetes.
PROJECT GOALWhy Agentic API?
Agentic APIs are becoming server-side runtimes.
01
Stateful by default
Conversations, reasoning, and tool state persist across turns.
02
Tools execute above inference
Web search, MCP, and other tools re-enter the same response loop.
03
Decouple responsibilities
State CRUD and tool execution belong in the Agentic API; vLLM remains focused on token generation.
Demo architecture
One agent loop.
Direct inference.
Both harnesses called Agentic API. On CoreWeave, Agentic API called one vLLM pod through a direct in-cluster Service.
Harnessesoutside cluster
Claude CodeANTHROPIC MESSAGES
CodexOPENAI RESPONSES
Agent execution planeAgentic API
Agentic API
Protocol translation
response orchestration
streaming + continuation
State storeSQLITE · DEMO
Gateway-owned toolsWEB SEARCH · SERVER-SIDE
Provider-neutral inference seam
CoreWeave Kubernetes1 NODE · 1 GPU
Direct Kubernetes ServiceCLUSTERIP · vLLM:8000
Single vLLM pod
Qwen/Qwen3-8B1 × H100
32,768 TOKENS
32,768 TOKENS
02 · TOOL LOOPWeb search result resumes the same response → Agentic API calls the same vLLM Service again.
Live demo
Watch the loop,
not just the answer.
01
Ask for a coding task that needs current web research.
02
The model selects the web search tool mid-response.
03
Agentic API executes it and continues inference through llm-d.
04
Switch from Claude Code to Codex. The stack stays put.
What we actually ran
One open stack.
Two hardware targets.
KinD on DGX Spark
- 1 × NVIDIA DGX Spark
- NVIDIA GB10 Grace Blackwell Superchip
- 128 GB unified memory
- KinD local Kubernetes control plane
CoreWeave Kubernetes
- 1 × H100 NVLink · 80 GB (1 of 8 on node)
- Qwen/Qwen3-8B · 32,768-token context
- CPU · 4 requested / 8 limit
- Memory · 24 GiB requested / 48 GiB limit
- Node g7269d8 · driver 580.126.09
- Hermes tool parser · tool calling enabled
DEMO: SQLite for a simple, single-instance deployment.
SUPPORTED: PostgreSQL for shared, multi-worker deployments.
Next: disaggregated Responses
The response lifecycle becomes its own scalable system.
INFERENCE-PROVIDER NEUTRALINDEPENDENT SCALINGOPEN PROTOCOLS
Response execution plane
State, memory, streaming, tool loops, approvals, continuation.
Inference plane
llm-d routes across vLLM today, with a neutral seam for other providers.
Learning plane
Replayable trajectories, evaluations, and eventually RL environments.
Come build the missing surface
Help build the
open agent runtime.
Keep the harness, agent loop, and tools inference-provider neutral. Make open models a first-class target for real agentic workloads.
01Memory and long-running response state
02MCP enhancements, resources, approvals, and richer execution
03More harnesses beyond Claude Code and Codex
04RL environments built from open, replayable agent trajectories
START ANYWHERE: an issue, a protocol fixture, a model integration, a deployment recipe, or a benchmark.
github.com/vllm-project/agentic-api
github.com/vllm-project/agentic-api
Presenter notes
01 / 07