Local AI models: when they are worth it
Local AI is an operations decision, not an ideology. Own hardware solves data protection and sometimes cost. It does not solve quality, updates or who is responsible at three in the morning.
When local AI is right
Local AI means the model runs on hardware you control. It is right when data may not leave the building, when the network is unreliable or when volume is so high that an API becomes expensive.
It is not automatically cheaper. A server costs money, someone has to run it and models age quickly. Do the maths over three years, not three months.
- Sensitive personal data or trade secrets that may not be sent out.
- Work that must function without the internet.
- High, steady volume where API cost becomes significant.
- A requirement to change model without changing vendor.
Hardware sets the ceiling
Memory and bandwidth decide which models you can run, not just the graphics processor. A model that does not fit must be quantised or split, and that changes quality.
For a first installation, a machine with 64–128 GB of shared memory is often enough for smaller models and embeddings. Large models with long context need more. Measure on your own data before buying.
Choose the model for the task
A large model is not always best. For classification, extraction and short answers, a smaller model can do the job faster and cheaper. For reasoning and long documents you often need more capacity.
Think about the whole stack: language model, embeddings for search and a reranker. A good embedding model can do more for answer quality than switching to a larger language model.
Hybrid is often the pragmatic choice
You do not have to choose all or nothing. A common setup runs sensitive parts locally and sends harder questions to an external model, but only after sensitive fields are masked.
Routing can be based on document sensitivity, task type or a simple quality threshold. What matters is that the rule is visible and that nobody accidentally sends the wrong data to the wrong place.
Operations and updates
A local model is a service like any other. It needs monitoring, logging, spare parts and a plan for what happens when a disk fails. It also needs updating when a better model arrives.
Set a fallback to an external model if the local one goes down, and test that the fallback works. Otherwise you find out when someone is left without an answer.
Do the full maths
Do not compare only price per token. Include hardware, electricity, space, backup, updates and the time of someone keeping it running. Compare with API cost for the same volume and the same quality requirement.
If volume is low and requirements are moderate, the API usually wins. If data may not leave the building, local wins even when it costs more, but then you should know you are paying for control.
Related services
Read next
Questions and answers
Can we run AI completely without the internet?
Yes, with a local model on your own hardware. You are then also responsible for updates, security and operations.
Is it cheaper than ChatGPT?
Sometimes, at high and steady volume. At low volume an API is usually cheaper once hardware and operations are included.
How sensitive data can we send to an external model?
It depends on the contract, the data and the purpose. Sensitive fields can often be masked and only necessary context sent. That should be documented before the solution goes live.