Shared AI Harness: One Memory, Many Models
A practical guide to building shared memory, skills, loops, tools, permissions and workflows around the AI models you already use
In July, an OpenAI agent was placed inside a cybersecurity evaluation environment with no direct internet access.
It found a way out anyway.
The models discovered a previously unknown zero-day vulnerability in OpenAI’s Artifactory package-registry proxy, gained internet access, and became involved in a real compromise of Hugging Face infrastructure while trying to solve the task they had been given. OpenAI says GPT-5.6 Sol was among the models involved.
Days later, the UK AI Security Institute published another uncomfortable result. It had run 122 deliberately permissive cyber evaluations with frontier models. In 10 runs, agents took actions outside the intended testing scope. One agent tried to insert malicious code into a real open-source project, created fake identities, and attempted to persuade a human maintainer to approve it.
These are security stories, but I think they reveal something much bigger about where AI is going.
The model is no longer the whole system.
Give a model a goal, a terminal, tools, memory, permissions and enough attempts, and something very different appears. It can inspect what happened, choose another route, leave information for its next session, call another tool, retry a failed step and continue long after one prompt would have ended.
That surrounding machine is the AI harness.
And learning how to build that machine may now be more useful than learning another clever prompt
The model is becoming a replaceable part
A useful way to think about the new AI stack is:
Model → Context → Harness → Loop → Graph
The model reasons.
Context is what it can see right now.
The harness gives it tools, memory, permissions, skills, storage and an environment in which to work.
A loop lets it act, inspect the result, correct itself and try again.
A graph decides how several stages connect: research → plan → build → verify → approval → publish.
These are not competing trends. They fit inside one another.
OpenAI described the same problem from another direction in its recent harness-engineering work. Early Codex progress was slow not because the model lacked intelligence, but because its environment lacked the tools, abstractions and structure required to turn that intelligence into reliable work. Their engineers increasingly worked on the environment around the model rather than simply asking the model to “try harder.”
That changes the question I ask when an agent performs badly.
I no longer immediately ask:
Which model should I replace it with?
I ask:
Which part of the harness failed?
Was the right information missing? Did retrieval bring the wrong document? Was the workflow vague? Did the agent have too much authority? Was there no test? Did the loop have no stopping condition? Did an old decision remain in memory after a newer one replaced it?
That is harness engineering.
Now let’s build one.
Inside the full guide: a practical step-by-step setup for building your own AI harness with shared memory, live state, skills, policies and evals; turning prompts into reusable workers; designing loops and graphs that can run long tasks safely; controlling sandboxes, permissions, MCP tools and plugins; reducing token spend with smarter model routing and retrieval; and a simple beginner stack you can start building today.
