When people think of an artificial intelligence assistant, they usually picture an online service: you open the browser, type, and the answer arrives from a distant data centre. There is, however, an increasingly practical alternative: running a model directly on your own computer or on a company server. The two routes have very different advantages and limits, and the choice depends on what you need to do and with what data.
What "local AI" means
A language model is essentially a large file containing its parameters, plus a program that runs it. Some developers publish their models' parameters: these are known as open weight models. Anyone can download and run them on their own hardware, in compliance with the accompanying licence, which should always be read because it may restrict commercial use.
There are programs that make the process relatively simple: you install the application, choose a model, wait for the download and start using it, even without an internet connection.
The advantages of local AI
- Confidentiality. Data never leaves the device. For sensitive documents, such as professional case files or health information, this is a decisive advantage.
- Offline operation. Useful in places with poor connectivity or in environments isolated for security reasons.
- Predictable costs. After the hardware investment there are no usage charges.
- Control. The model does not change without notice: the version you tested is the one you use.
- Customisation. You can fine-tune the model or integrate it freely into your own systems.
The limits of local AI
- Lower capabilities. Models that can run on an ordinary computer are smaller than those offered by the main cloud services and generally less capable at complex tasks.
- Hardware requirements. You need a lot of memory, ideally on a suitable graphics card.
- Speed. On modest hardware, answers can arrive slowly.
- Maintenance. Updates, security and configuration are your responsibility.
- Extra features. Web search, image analysis or ready-made integrations require additional work.
How much memory you need: quantisation
The amount of memory required depends mainly on the number of parameters in the model and the precision with which each one is stored. Originally each parameter takes 16 bits, or 2 bytes: a 7-billion-parameter model would need about 14 GB for the weights alone.
Quantisation reduces the precision of the parameters, for example to 8 or 4 bits, greatly reducing the memory used with an often modest loss of quality. The same 7-billion-parameter model, quantised to 4 bits, drops to about 4 GB for the weights, plus memory for the conversation context.
| Model size | 16-bit weights (approx.) | 4-bit weights (approx.) | Indicative hardware |
|---|---|---|---|
| 3 billion parameters | 6 GB | 2 GB | Recent laptop |
| 7-8 billion | 14-16 GB | 4-5 GB | Computer with a good GPU or plenty of RAM |
| About 30 billion | 60 GB | 16-18 GB | Dedicated workstation |
| About 70 billion | 140 GB | 35-40 GB | Server or multiple GPUs |
These are rough estimates, useful for getting your bearings: actual values vary depending on the format and the program used.
The advantages of the cloud
- More powerful models, kept up to date by the provider.
- No hardware to buy or maintain.
- Built-in features: search, file analysis, image generation, collaboration.
- Scalability: you can go from one user to a hundred without changing infrastructure.
The limits of the cloud
- Data leaves your perimeter. Contracts, processing locations and retention policies need to be assessed.
- Dependence on the provider: prices, terms and models can change.
- Variable costs that grow with use.
- A connection is required.
How to choose
A few questions help you decide.
- How sensitive is the data? If it cannot leave the organisation, running locally or on your own servers becomes almost mandatory.
- How complex is the task? Classifying emails or summarising short documents is within reach of small models; long, elaborate reasoning benefits from larger ones.
- How many users and what volume? For a few occasional users the cloud is cheaper; for high, constant loads your own hardware can pay for itself.
- Who will manage the system? Without in-house skills, a local solution risks staying out of date and insecure.
The middle way
Many organisations adopt a hybrid approach: a local model for tasks involving confidential data and a cloud service, with an appropriate contract, for general tasks. Another solution is a private cloud, where the model runs on dedicated infrastructure with stronger contractual guarantees.
Security applies locally too
"Local" does not automatically mean "secure". Download models only from reliable sources, update the programs that run them, restrict access to the service if it is reachable over the network and protect saved conversations like any other sensitive data.
In short
- Local AI offers confidentiality, control and offline operation, with less powerful models.
- The cloud offers more capable models and ready-made features, but data leaves your perimeter.
- Quantisation greatly reduces the memory needed to run a model locally.
- The choice depends on data sensitivity, task complexity, volumes and skills.
- A hybrid approach often combines the advantages of both solutions.
