Building a local-first AI ops platform on a prize GPU (ROCm 10, 30B coder, multi-agent)

I stopped treating local LLMs as a side project and started routing real product work through them.

I'm building InnerOS / InnerOps, a local-first control plane for running development and operational work through AI agents without sending every task to cloud APIs.

The basic idea is simple:

• portal + commercial/workforce modules
• MCP-based orchestration
• Cursor, Codex and chat agents as interchangeable workers
• automatic routing to the machine/model that best fits the task
• cloud models only when they add something the local stack cannot

Why I pushed this now

I received a Radeon AI PRO R9700 (gfx1201, 32 GB VRAM) after finishing 10th in the AMD Dev Quest at AMD Advancing AI in San Francisco on July 22–23.

That was the push I needed to stop treating “local inference” as a weekend experiment and make it part of the development loop.

The stack I actually use

AMD node

• Radeon AI PRO R9700
• official ROCm 10 vLLM container
• PyTorch 2.12.0+rocm10.0.0
• vLLM 0.27.x
• Qwen3-Coder-30B-A3B-Instruct-AWQ on port 8000
• systemd-managed so it survives reboots

Intel node

• Ollama for lighter chat/classification work
• smaller Qwen/Phi-class models

Routing

• coding / refactor / review → 30B model on the AMD GPU
• lighter ops / classification / chat → smaller local models

Supporting services

• Lemonade Server for local TTS/STT on CPU so voice workloads do not eat the GPU’s VRAM
• MCP server + agent inbox
• Dev Swarm for heavier implementation work
• IDE task bridge for Cursor/Codex-style workers

What changed this month

The biggest improvement was not raw tokens/sec.

My previous vLLM container was named as if it were ROCm 10. When I inspected the actual runtime, it was still ROCm 7.14, PyTorch 2.11 and vLLM 0.23.x.

That was a useful lesson:

Container names are not evidence. torch.__version__ is.

I cut over to the official ROCm 10 stack and made the serving layer persistent.

Since then I have also:

• documented rollback
• wired heavy coding tasks to the local AMD endpoint
• verified the Codex adapter path
• expanded the A2A agent registry to 55 functional agents plus control/execution cards
• added MCP-based ops updates through Discord
• started measuring whether agent automation is actually returning human time instead of merely generating activity

Before vs now

| | Old path | Current |
| ---------------- | ------------------------------------ | --------------------------------------------------- |
| Stack | ROCm 7.14 / vLLM 0.23 / PyTorch 2.11 | ROCm 10 / vLLM 0.27 / PyTorch 2.12 |
| Operations | manual Docker | systemd-managed |
| Smoke throughput | ~25–40 tok/s mixed | ~19–41 tok/s prompt-dependent |
| Main gain | local inference worked | local inference is part of the development workflow |

The throughput did not magically double.

The win is reliability, routing and utilization.

The GPU is now doing useful work for hours instead of appearing in benchmark screenshots once a week.

What is still hard

32 GB means one large model at a time.

VRAM budgeting is part of the architecture.

The infrastructure is also ahead of the visible product. The browser portal and the real Workforce product integration still need work.

And multi-IDE coordination is messy. Durable inboxes, ops tasks and evidence help, but this is still early software rather than a polished “autonomous company” fantasy.

Business angle

The initial market is SMB operations in Ecuador/LATAM: workforce, quotes, WhatsApp operations and internal company workflows.

The bet is that local-first AI can reduce recurring model spend and keep more sensitive operational work on owned infrastructure, while still using Gemini or other cloud models for specialized lanes when there is a real reason to pay for them.

One thing I’m increasingly convinced of:

The interesting benchmark is not tokens per second. It is how many hours per day the hardware becomes useful enough that you trust it with real work.

The InnerOS repository is currently private, so I’m not using the GitHub link as a public proof point yet.

I’m building in public from the server rack upward and will keep sharing the parts that are actually verified.

If you’re running vLLM on consumer AMD hardware at roughly 30B scale, I’m especially interested in comparing notes on VRAM budgeting, AWQ/MoE behavior and long-running agent workloads.