The Seven Parts of a Working Local AI System

A local language model becomes useful business infrastructure only when it is connected to the right information, bounded by suitable instructions and permissions, supported by human review and operated with a usable record of its work.

Selecting the model is one decision. Designing the service around it is the larger job.

Why model tests give an incomplete answer

Most early AI tests compare outputs. A prompt and a set of documents are sent to two or more models, and a reviewer decides which response is best.

That can identify a promising model, but it leaves out much of the work required for routine use. The final service needs to obtain current information, control access, perform reliable calculations, route exceptions, retain evidence, recover from failure and let a responsible person approve the result.

These needs apply to hosted and local AI. Local deployment makes some of them more visible because the organisation controls more of the stack.

1. Model and inference runtime

The model must meet the required quality for the task, but the inference configuration is part of that result.

Record at least:

• Model family and exact version
• Weight format and quantisation
• Inference runtime and version
• Context limit used in the test
• Thinking or reasoning setting
• Sampling and generation settings
• Tool-call and structured-output settings
• Hardware and operating system

The purpose is not documentation for its own sake. It allows the team to reproduce a result, compare changes and return to a known version if an update causes a problem.

Local models can be compressed to use less memory, but compression may affect quality. A long advertised context window may also require more memory and may not provide reliable use of every document placed inside it. Test the configuration the organisation could run, not the largest specification on the model card.

2. Information and provenance

Define the sources the AI service is allowed to use and how their status is known.

A business workflow often includes structured records, documents, correspondence and local knowledge. Some sources are current, some are historic and some conflict. The system should preserve enough provenance for a reviewer to understand which evidence supported the output.

For each source, record:

• Owner
• Authoritative location
• Version or effective date
• Access classification
• Expected availability
• Retrieval method
• What happens if it is missing or contradictory

Do not assume that adding every available document improves the answer. A larger context can include more obsolete, irrelevant or hostile content. Selection rules and source priority still matter.

3. Instructions, policy and decision boundaries

Separate model guidance from rules that must be enforced.

The model may receive instructions about the purpose, output format, evidence standard and referral conditions. Approval limits, customer access rights and prohibited actions may belong in ordinary software or permissions where enforcement does not depend on generated text.

For each decision, ask:

• Is the model extracting, recommending or deciding?
• Which evidence is required?
• Can the result be checked?
• Does it create a financial, legal, safety or customer commitment?
• Who has authority to approve it?
• What condition requires refusal or referral?

This prevents a writing task from quietly acquiring decision authority.

4. Tools and business systems

Tools let a model act beyond its text response. A tool may retrieve a document, query a system, perform a calculation, create a draft record or send a communication.

Treat each tool as a controlled system connection. Define:

• Permitted operations
• Required inputs and validation
• Read and write access
• Transaction or approval limits
• Logging
• Timeouts and failure behaviour
• Whether a person must confirm the action

Prefer deterministic software for calculations and fixed policy checks. The language model can identify the inputs and explain the result without being asked to perform every arithmetic or policy function itself.

5. Identity, permissions and boundaries

The AI service should have a distinct identity and access appropriate to its task.

The NCSC's secure deployment guidance recommends access controls for models, data, APIs and processing infrastructure, together with segregation for sensitive code and data. It also calls for transparency about where data is used, accessed and stored.

For a local service, inspect the full data path. Local inference may still use an external embedding service, plug-in, telemetry endpoint, document store or web tool. Logs and temporary files may contain the same sensitive information as the original prompt.

The boundary should be demonstrated through configuration and testing, not inferred from the location of the model file.

6. Human review and correction

Human oversight needs a defined job.

State who reviews the output, what they must check, which evidence they receive, how much time is allowed and what authority they hold. A reviewer who must reopen every source and reconstruct the work may protect quality while removing most of the intended benefit.

A good review view distinguishes:

• Sourced facts
• Calculated values
• Model inferences
• Missing or contradictory evidence
• Decisions requiring approval
• Changes made after challenge

Correction should take place inside the recorded process where possible. The person needs to be able to question a result, obtain the basis for it and publish a revised version without losing the original exchange.

7. Record, monitoring and feedback

Keep enough information to explain and operate the service:

• Input and source references
• Model and system versions
• Instructions and tool calls
• Generated output
• Refusals, errors and exceptions
• Human changes and approval
• End-to-end and review time
• Final outcome where it becomes known

The record should support incident investigation and routine improvement. It should also have a retention rule. Keeping every prompt and document forever creates its own data and access problem.

The NIST AI Risk Management Framework asks organisations to define human oversight, identify controls for third-party components and evaluate systems under conditions similar to deployment. Those practices apply whether the model is hosted by a supplier or operated internally.

What a shared workspace can add

Buzz is an early example of a self-hostable workspace in which people and agents share rooms. The project describes messages, workflow steps, reviews and other actions as signed events in a common log, with agents receiving identities and channel membership rather than appearing as an unaccountable background process.

In a practical quotation test, that design allowed a reviewer to challenge an AI-produced line in the thread. The model returned to the rate card, accepted the correction and republished the quote in view of the channel.

The tested release still had issues with document handling and model configuration, so it should not be presented as settled infrastructure. It demonstrates why the interaction model matters. AI work needs a place for evidence, challenge, correction and approval, rather than a private conversation whose output is copied into a business process.

One-page local AI system checklist

Before moving beyond a test, produce a one-page view containing:

Work

• Trigger, output and business owner
• Users and people affected
• Normal cases, exceptions and exclusions
• Success measure and stop condition

Intelligence

• Exact model and runtime
• Configuration and hardware
• Approved information sources
• Instructions, rules and decision boundaries

Action

• Tools and system connections
• Read, write and approval permissions
• Actions requiring confirmation
• Manual route and recovery

Control

• Reviewer and approval method
• Evidence visible to the reviewer
• Log, retention and access rules
• Monitoring, feedback and incident owner

If this page cannot be completed, the project has found a system-design question rather than a reason to choose a larger model.