If you access a large language model (LLM) at home in your web browser or app, then you appear to be directly interacting with the model. In actual fact there is extra software surrounding this conversation. Before your query is passed to the LLM, there are multiple software components that handle your prompt, as shown in the following diagram. These layers are collectively called a “harness”. A harness is a control and orchestration layer around LLMs that manages inputs, outputs, tools, safety and evaluation.
Conversation management keeps track of the dialogue so the LLM understands what “it” or “that” refers to. A memory layer may recall longer-term preferences or projects to make responses more relevant. There is a tool orchestration layer that decides whether to use a web search, Python, image generation, or other capabilities. Additionally, a safety and policy layer checks requests before they’re answered. A context management layer decides which parts of the conversation to include when generating the next response. Response assembly formats the final answer, including images, citations, or interactive elements when appropriate.
In an enterprise environment there will also be a harness, though probably with more layers. For example, in an enterprise there may be a layer to access company documents, to allow the LLM to incorporate enterprise data at inference time. This pattern is known as retrieval-augmented generation (RAG).
An enterprise-grade harness may contain many elements, such as:
- Prompt and dataset management
- Evaluation and testing
- Automated regression testing
- Monitoring and observability
- Governance and safety
- Operations.
Enterprises may build their own harnesses, but there are commercial ones available. Examples are shown in the table below, such as LangSmith, Humanloop, Arize and Galileo.
There are also specialist security harnesses. Examples include Harness AI Security, Protect AI, HiddenLayer, Lakera and Robust Intelligence. These products help detect and mitigate:
- prompt injection
- jailbreaks
- data leakage
- unsafe outputs
- OWASP LLM Top 10 vulnerability categories.
Over time, it is likely that “AI Harness Platforms” may become recognised as a distinct market. This would be a layer in the enterprise AI stack, sitting between foundation models and business applications. This layer would provide governance, observability, evaluation, security, orchestration and operational management.
One reason why this is important is that LLMs themselves offer little in the way of a competitive “moat”. Frontier labs are releasing new models all the time, leapfrogging each other in terms of capabilities against benchmarks. They may compete in various aspects, such as coding skill or context window or cost, but they are all based on the same underlying transformer-based architecture. There are dozens of models, and vendors have struggled to find a way to make money from them. Certainly, there are subscription models as we see with Anthropic’s Claude and OpenAI’s ChatGPT, paid either by individuals or enterprises. However, these have to compete with open-weight and open-source models such as Llama and Mistral, and particularly the Chinese models like Qwen, DeepSeek, and others. These are increasingly capable, barely lagging the premium models from the leading US vendors.
Building an effective harness around these LLMs may be an effective way to compete and to charge a fee for what is certainly a useful software layer. For example, LLMs have considerable security weaknesses and are susceptible to prompt injection attacks. Security harnesses that prove to be effective in reducing this risk are likely to command significant enterprise spend.
The software industry has often developed in this way. Originally every application had to deal with hardware directly, but then operating systems appeared and took over tasks like memory management, disk retrieval and networking. Later, databases emerged as an additional layer, removing the need for applications to deal with locking, indexing and file recovery. This process continued, with layers such as middleware, virtualisation and cloud computing all providing higher levels of abstraction. AI, or at least the subset of AI called LLMs, is a relatively new player in the enterprise, and it would not be surprising to see abstraction layers like harnesses continue to develop and spawn new vendors competing for control of that software layer.
