Why this is a question at all
With a cloud API the provider updates the model for you, and you often do not notice the moment it changes — one day the answers are simply a little different. With a self-hosted server it is the opposite: you control exactly which version runs and when it changes. That brings independence and control, but it needs a process — otherwise you install a new version one day and get a regression straight into production, where the model answers differently from what users, integrations or scripts expect.
The absence of a process is why teams either sit on an obsolete model for years out of fear, or update blind and deal with the consequences afterwards. Both are solved by the same thing: a predictable verification routine before the switch.
When to update and when to leave it alone
| Situation | What to do |
|---|---|
| A new version is out but current tasks are handled fine | leave it — updating without a reason adds risk for no gain |
| The new model is measurably better on tasks like yours | test on your own examples; update only on a confirmed win |
| The current model makes mistakes on real queries | find the cause first — it is often the knowledge base or the prompt, not the model |
| A security fix or critical bugfix is released | update with priority, but still through a test run rather than straight to production |
| The task itself changes (new language, new document types) | check whether the current model copes; switch if it does not |
| A cheaper model appears at comparable quality | test economics and quality together, not the inference price alone |
How to test a candidate before replacing anything
- •Collect a set of real queries from production logs — not invented examples, but what users actually send,
- •Run that set through the new model alongside the old one, on the same server or in a separate test environment,
- •Compare the answers manually on a representative sample — quality, tone, factual accuracy and output format,
- •Check the edge cases separately — very short and very long queries, unusual phrasing, off-topic questions,
- •Only then move live traffic to the new version, and preferably gradually rather than for everyone at once.
On which model to move to in the first place, the open options and their strengths are in the best open-source LLMs to self-host. Hype around a name is a poor selection criterion — go by measured quality on your own tasks rather than general benchmarks.
Fallback: why the old model should not be deleted immediately
Even after a successful test run, production can surface a case that was not in the sample — users phrase things more variously than anyone can anticipate. The right practice is to keep the previous version on disk and switchable for an observation period after the update, typically a few weeks of active use. It costs almost nothing in hardware: both models sit on the server and only one is active at a time.
Rolling back a regression
If quality visibly drops after an update — more user complaints, factual errors, output that no longer matches the expected format — the rollback should take minutes, not hours of approvals. For that, the routing to the model belongs in configuration (which model is active, where traffic goes) rather than hard-coded into every service that calls it. Then rolling back is one variable change instead of redeploying the whole system.
The same staged principle applies when moving off a cloud API onto your own server — see migrating from the ChatGPT API.
Frequently asked questions
How often should a self-hosted model be updated?
There is no single schedule and there should not be. Change the model when there is a clear reason — a new version is measurably better on your tasks, the old one has stopped coping, or a critical security fix has landed. “Quarterly by default” is a poor guide because it is not tied to any actual benefit.
What if the new model turns out worse on our data?
That is exactly why you do a test run on real examples before switching production traffic, and keep the old model available as a fallback. If the new one loses the comparison you simply do not switch; if you already switched and noticed a regression, you roll back.
Can we test a new model without stopping the old one?
Yes, and it is standard practice — keep both on the server or in the inference environment, route test traffic or a set of real logged queries to the new one, and compare the answers before moving the live flow.