When Is Local AI Good Enough for Business Work?

Local AI has moved from a technical experiment to a credible option for some business workflows. Open-weight models can now work across documents, use tools, draft commercial material and take part in multi-step processes on hardware that a smaller organisation could own or rent.

That does not make local deployment the default. The decision depends on the work, the quality required, the data boundary, the hardware and the organisation's ability to operate the whole system.

What local AI means

Local AI usually means that model inference takes place on infrastructure controlled by the organisation or user rather than through a shared public model service. The model may run on a workstation, an internal server, dedicated data-centre equipment or isolated cloud infrastructure.

The term can be misleading because the security and operating arrangements differ between these options. A model on a laptop, a centrally managed internal service and a dedicated private cloud deployment may all be described as local, but they do not have the same access controls, availability, audit record or support.

A useful definition should state:

• Where the model runs
• Who administers it
• Where prompts, documents, outputs and logs are stored
• Which external services it can contact
• Who can use it and what permissions they receive
• How updates, incidents and recovery are handled

Without those details, "local" describes a location rather than an operating model.

Why the capability boundary has moved

Model architecture and inference software have reduced the hardware needed for useful local work.

Qwen3.6-35B-A3B, released in April 2026, is a mixture-of-experts model. The complete model contains 35 billion learned values, called parameters, but it activates a smaller group of around 3 billion while processing each piece of text. This reduces the computation required for an answer, although the complete model still has to be held in memory. It should not be confused with a model containing only 3 billion parameters.

Its model files are openly available, and the Qwen project documents support through local inference tools including llama.cpp and MLX. An organisation can therefore download and operate the model on infrastructure it controls, subject to having enough suitable memory and processing capacity.

DeepSeek-V4-Flash shows the same direction at a different scale. It has 284 billion total parameters and 13 billion active, a one-million-token context window and open weights. DeepSeek documents local serving through tools including vLLM, SGLang, Docker Model Runner and compatible quantisations.

DeepSeek-V4-Flash still needs substantial memory, and the ability to load a compressed model does not prove useful speed or output quality. The important point is that model families with stronger reasoning, long-context and tool-use capabilities are available outside a hosted provider's API.

Vendor benchmark results should be treated as evidence for what to test. They should not be treated as proof that the model is fit for a business process.

Machine-readable marking becomes part of the system

Article 50 of the EU AI Act applies from 2 August 2026. It requires providers of generative AI systems to mark AI-generated or manipulated audio, images, video and text in a machine-readable form so the content can be detected.

The legal obligation is mandatory. The EU Code of Practice on Transparency of AI-generated Content is voluntary, but gives signatories a recognised route for demonstrating how they meet it. OpenAI, Anthropic and Google have signed the provider section, along with companies including Meta, Microsoft and Mistral.

The mark is not necessarily a visible notice in the generated prose. The Commission says it must be effective, reliable and interoperable as far as technically feasible, taking account of the type of content and the state of the art. The public material does not give an ordinary user a dependable way to inspect a copied passage from ChatGPT, Claude or Gemini and identify which marking method was applied or whether it survived later editing.

Running an open-weight model locally changes this relationship. The downloaded model does not automatically acquire the hosted provider's system for marking its output. This should not be confused with an exemption from Article 50. The obligation applies at AI-system level, and an organisation that develops or has a system developed and puts it into service under its own name may become the provider responsible for implementing the mark.

There is also a separate obligation for deployers to display a clear label on AI-generated text published to inform the public on matters of public interest. The Commission's guidance provides an exception where the publication has received substantive human review or editorial control and a person or organisation takes editorial responsibility. Superficial spelling or grammar checks do not meet that test.

For a local AI design, provenance therefore joins access, retention and review:

• Who is the provider of the finished AI system?
• Which outputs must receive a machine-readable mark?
• Where is the mark added?
• Does it survive copying, conversion and editing?
• How can another party detect it?
• Who reviews published text and accepts editorial responsibility?

These questions are still developing with the technical standards. They belong in the system decision now because local operation can transfer the implementation duty from a large platform provider to the organisation running the service.

The business test is different from the model test

A model benchmark asks whether the model can solve a defined set of problems under stated conditions. A business test asks whether the resulting system improves real work without creating unacceptable error, review or operating cost.

That includes questions a general benchmark cannot answer:

• Can the system find the current source rather than a superseded document?
• Does it identify contradictory customer information?
• Can a reviewer see the evidence behind a calculation?
• Does it refuse a decision outside its authority?
• How much correction is required before the output can be used?
• What happens when an input or service is missing?

The business unit being assessed is the workflow, not the model response.

Evidence from a quotation test

In one practical test, OpenAI's hosted ChatGPT 5.6 model and Qwen3.6-35B-A3B running locally were asked to prepare an annual HVAC maintenance quotation for a fictional factory.

Both received the same instruction and the same five documents:

• Client enquiry
• Scrappy site survey notes
• Current rate card
• Callout history
• Superseded 2023 quote

The documents included three deliberate problems: an incorrect equipment count, a discount that should not be applied and a historic fee that no longer existed.

The assessment looked beyond drafting quality. The quote needed to select the current sources, reconcile its calculations, apply the commercial rules, identify assumptions and avoid pricing anything that could not be supported by the documents.

This was a workflow test rather than a formal model benchmark. ChatGPT supplied the hosted comparison. The local Qwen model was then run with reasoning disabled and at different reasoning levels to examine how much the configuration affected the result.

The local model failed when reasoning was disabled. It invented a breakdown count and produced arithmetic that did not reconcile. With a small amount of reasoning, the same model produced the strongest commercial result in that test. It used the callout history to estimate reactive labour and applied the discount rule correctly.

When a line was challenged in the shared workspace, the model returned to the rate card, corrected its interpretation and republished the quote.

This was one controlled example using a new collaboration tool. It did not prove general superiority. It showed that useful local AI can now be assessed through the quality of completed work rather than dismissed because of model size alone.

The case for local deployment

Local deployment deserves examination when one or more of the following conditions apply:

• The work uses data that should remain inside a defined technical boundary
• The organisation needs more control over retention, logging or model versions
• External connectivity is unreliable or unavailable
• Usage volumes make owned infrastructure economically credible
• The task is narrow enough for a local model to meet the required quality
• Supplier concentration or service changes create material continuity risk
• Response time is important and the workload can be served close to the user

None of these conditions settles the decision. Local infrastructure may reduce one exposure while adding patching, access control, monitoring, model provenance and support duties.

The case for hosted deployment

A hosted service may be the better choice when:

• The work needs the strongest available capability
• Usage is low or unpredictable
• The organisation lacks the skills to run model infrastructure
• Rapid model improvements matter more than version stability
• The provider's contractual, security and data terms satisfy the requirement
• Availability and support would cost more to reproduce internally

A hybrid design can keep sensitive extraction or classification inside the organisation while using a hosted model for work that does not expose the underlying data. It can also provide a hosted fallback for cases the local model cannot handle, provided the routing and data rules are explicit.

How to compare local and hosted AI

Choose one bounded task and test three routes: the existing human process, a hosted model and a local model that could be operated in practice.

Use representative work rather than demonstration examples. Include incomplete inputs, historic errors, policy boundaries and cases that should be refused. Measure:

• Final quality after review
• Error type and consequence
• Review and correction time
• Coverage and refusal
• End-to-end time
• Cost at expected volume
• Data movement and retention
• Availability and recovery
• Ongoing support and update work

Record model, quantisation, reasoning setting, prompt, context limit and system version for each run. A comparison that changes several of these at once cannot show what caused the difference.

The threshold is not parity on every task. A local model may earn a bounded role because it meets the required quality while satisfying a data or continuity constraint. A hosted model may earn its role because the quality difference outweighs the additional dependency.

Local AI is now credible enough that organisations should test rather than assume. The answer should still be allowed to be local, hosted, hybrid or no AI at all.