A firmware upgrade on one small component stopped three manufacturing plants for two hours at an automotive manufacturer. The devices involved existed as configuration items (CIs) in the configuration management database (CMDB), but they carried no populated attributes and no owner, so when the plants needed someone to coordinate a rollback, there was nobody to call. Apex documented the machinery in each plant and then worked with the client to run change management through ServiceNow and its CMDB.
| At a glance | |
|---|
| Sector | Automotive manufacturing |
| Estate | Millions of CIs |
| What went wrong | A firmware upgrade was incompatible with another device, and no one owned the CIs involved |
| What it cost | Production stopped at three plants for two hours |
| What fixed it | The machinery documented in each plant, and change management run through ServiceNow and the CMDB |
Why the machinery was left out of scope
The reasoning followed how the CMDB was used. ServiceNow did not control or manage the production line, so the machinery was treated as out of scope for the CMDB and for vulnerability management, and it therefore did not need an owner.
The devices did exist as CIs, and that made things worse. They counted in the totals and told nobody anything. A CI with no populated attributes cannot answer a question about its firmware, its model or who looks after it. It helps to know what a CMDB holds and why before deciding what can safely be left out.
How a firmware upgrade stopped three plants
A small but critical component received a firmware upgrade that was incompatible with another device. There was no end-to-end view of the production line, so the incompatibility was not picked up until it was too late.
Three manufacturing plants became unable to function. They were offline for two hours.
The rollback that needed one person
Undoing the upgrade meant coordinating a firmware rollback across all three plants. That needs a single point of contact.
The CIs had no owner, so there was no one to fill that role. Production stopped, and no one was named as responsible for the devices. This is the same gap we describe in what happens when CI owners leave, except that here no owner had been recorded in the first place.
What we changed
We documented the machinery in each plant. We then worked with the client to implement change management using ServiceNow and its CMDB, to prevent a repeat.
The point of running a firmware change through the CMDB is that the change is assessed against the CIs it touches. The same idea drives change impact analysis, which is only as good as the relationships behind it.
Four checks for any production estate
- Scope by consequence. If a device's failure or change can stop production, it belongs in the CMDB, whether or not ServiceNow operates the line.
- Report attribute completeness by class. Owner, support group, model and firmware version are the first ones to look at. A report of CIs where they are empty is the fastest way to find CIs that count but cannot answer questions. The configuration item classes your estate uses decide which attributes apply.
- Name an owner and a deputy for every plant and line. Certify them on a schedule so the names stay current.
- Route firmware changes through change management. The change record should show the CIs touched and the devices that depend on them.
How Apex helps
Our two-week CMDB health baseline scores which of your CIs are trustworthy and which are guesses. It shows where attributes and owners are missing, and the output is a report you can take to your change advisory board.
Two related pieces of work sit alongside it. A data quality assessment scores your estate against weighted critical success factors, key performance indicators (KPIs) and metrics. A discovery and integration coverage review establishes what is missing from your estate view and what that absence costs you during an incident.
Book a CMDB diagnostic call or arrange a meeting with one of our consultants.
Frequently asked questions
Should production machinery be in the CMDB?
If its failure or a change to it can stop production, yes. Scope by consequence rather than by who operates the device, because a change to an out-of-scope device can still take in-scope services down.
Which attributes matter most on CIs that have no owner?
Owner and support group come first, then model, firmware version and location. Those are the values someone needs in the first few minutes of an incident.
Who should own a device on a production line?
A named person with a deputy, at plant or line level. Certify the names on a schedule, so that a departure does not leave a CI pointing at nobody.
Is a CI with no attributes still useful?
It counts in your totals but cannot answer a question about the device. A CI earns its place when someone can use it to find out what the device is, who owns it and what depends on it.
