S
Stitex
Infrastructure

Updating a Self-Hosted LLM Without Breaking Production

Short answer: update a self-hosted model for a reason, not on a schedule — because a new version is measurably better on your tasks, or the current one has stopped coping. Test the candidate on real examples before switching, and keep the old one available as a fallback. Here is the procedure step by step.

July 23, 202610 min readStitex Technologies

Why this is a question at all

With a cloud API the provider updates the model for you, and you often do not notice the moment it changes — one day the answers are simply a little different. With a self-hosted server it is the opposite: you control exactly which version runs and when it changes. That brings independence and control, but it needs a process — otherwise you install a new version one day and get a regression straight into production, where the model answers differently from what users, integrations or scripts expect.

The absence of a process is why teams either sit on an obsolete model for years out of fear, or update blind and deal with the consequences afterwards. Both are solved by the same thing: a predictable verification routine before the switch.

When to update and when to leave it alone

SituationWhat to do
A new version is out but current tasks are handled fineleave it — updating without a reason adds risk for no gain
The new model is measurably better on tasks like yourstest on your own examples; update only on a confirmed win
The current model makes mistakes on real queriesfind the cause first — it is often the knowledge base or the prompt, not the model
A security fix or critical bugfix is releasedupdate with priority, but still through a test run rather than straight to production
The task itself changes (new language, new document types)check whether the current model copes; switch if it does not
A cheaper model appears at comparable qualitytest economics and quality together, not the inference price alone
If it works, do not touch it without a reason
A model running reliably is an asset, not “an old version due for an upgrade”. Every update risks changing behaviour where users and integrations have already adapted to the current answers, formats and tone. Change it when there is a measurable reason — better quality, lower cost, or a specific problem solved.

How to test a candidate before replacing anything

  • Collect a set of real queries from production logs — not invented examples, but what users actually send,
  • Run that set through the new model alongside the old one, on the same server or in a separate test environment,
  • Compare the answers manually on a representative sample — quality, tone, factual accuracy and output format,
  • Check the edge cases separately — very short and very long queries, unusual phrasing, off-topic questions,
  • Only then move live traffic to the new version, and preferably gradually rather than for everyone at once.

On which model to move to in the first place, the open options and their strengths are in the best open-source LLMs to self-host. Hype around a name is a poor selection criterion — go by measured quality on your own tasks rather than general benchmarks.

Fallback: why the old model should not be deleted immediately

Even after a successful test run, production can surface a case that was not in the sample — users phrase things more variously than anyone can anticipate. The right practice is to keep the previous version on disk and switchable for an observation period after the update, typically a few weeks of active use. It costs almost nothing in hardware: both models sit on the server and only one is active at a time.

Rolling back a regression

If quality visibly drops after an update — more user complaints, factual errors, output that no longer matches the expected format — the rollback should take minutes, not hours of approvals. For that, the routing to the model belongs in configuration (which model is active, where traffic goes) rather than hard-coded into every service that calls it. Then rolling back is one variable change instead of redeploying the whole system.

The same staged principle applies when moving off a cloud API onto your own server — see migrating from the ChatGPT API.

Frequently asked questions

How often should a self-hosted model be updated?

There is no single schedule and there should not be. Change the model when there is a clear reason — a new version is measurably better on your tasks, the old one has stopped coping, or a critical security fix has landed. “Quarterly by default” is a poor guide because it is not tied to any actual benefit.

What if the new model turns out worse on our data?

That is exactly why you do a test run on real examples before switching production traffic, and keep the old model available as a fallback. If the new one loses the comparison you simply do not switch; if you already switched and noticed a regression, you roll back.

Can we test a new model without stopping the old one?

Yes, and it is standard practice — keep both on the server or in the inference environment, route test traffic or a set of real logged queries to the new one, and compare the answers before moving the live flow.

We will set up model updates without risking production

We test the new model on your data before the switch and configure a fast rollback in case of regression.