Three Plants Down For Two Hours And No Owner To Call
A firmware upgrade on one small component stopped three manufacturing plants for two hours at an automotive manufacturer. The devices involved existed as configuration items (CIs) in the configuration management database (CMDB), but they carried no populated attributes and no owner, so when the plants needed someone to coordinate a rollback, there was nobody to call. Apex documented the machinery in each plant and then worked with the client to run change management through ServiceNow and its CMDB.
Key Takeaways
- A firmware upgrade on one small component, incompatible with another device, stopped three manufacturing plants for two hours at an automotive manufacturer.
- The devices existed as CIs with no populated attributes and no owner, so nobody could be called to coordinate the rollback.
- The machinery had been treated as out of scope because ServiceNow did not control the production line. A CI with no attributes counts in the totals but can't answer a question about its firmware, its model or who looks after it.
- Apex documented the machinery in each plant and worked with the client to run change management through ServiceNow and the CMDB.
- Four checks apply to any production estate: scope by consequence, report attribute completeness by class, name an owner and a deputy for every plant and line, and route firmware changes through change management.
| At a glance | |
|---|---|
| Sector | Automotive manufacturing |
| Estate | Millions of CIs |
| What went wrong | A firmware upgrade was incompatible with another device, and no one owned the CIs involved |
| What it cost | Production stopped at three plants for two hours |
| What fixed it | The machinery documented in each plant, and change management run through ServiceNow and the CMDB |
Why the machinery was left out of scope
The reasoning followed how the CMDB was used. ServiceNow did not control or manage the production line, so the machinery was treated as out of scope for the CMDB and for vulnerability management, and it therefore did not need an owner.
The devices did exist as CIs, and that made things worse. They counted in the totals and told nobody anything. A CI with no populated attributes cannot answer a question about its firmware, its model or who looks after it. It helps to know what a CMDB holds and why before deciding what can safely be left out.
How a firmware upgrade stopped three plants
A small but critical component received a firmware upgrade that was incompatible with another device. There was no end-to-end view of the production line, so the incompatibility was not picked up until it was too late.
Three manufacturing plants became unable to function. They were offline for two hours.
The rollback that needed one person
Undoing the upgrade meant coordinating a firmware rollback across all three plants. That needs a single point of contact.
The CIs had no owner, so there was no one to fill that role. Production stopped, and no one was named as responsible for the devices. This is the same gap we describe in what happens when CI owners leave, except that here no owner had been recorded in the first place.
What we changed
We documented the machinery in each plant. We then worked with the client to implement change management using ServiceNow and its CMDB, to prevent a repeat.
The point of running a firmware change through the CMDB is that the change is assessed against the CIs it touches. The same idea drives change impact analysis, which is only as good as the relationships behind it.
Four checks for any production estate
- Scope by consequence. If a device's failure or change can stop production, it belongs in the CMDB, whether or not ServiceNow operates the line.
- Report attribute completeness by class. Owner, support group, model and firmware version are the first ones to look at. A report of CIs where they are empty is the fastest way to find CIs that count but cannot answer questions. The configuration item classes your estate uses decide which attributes apply.
- Name an owner and a deputy for every plant and line. Certify them on a schedule so the names stay current.
- Route firmware changes through change management. The change record should show the CIs touched and the devices that depend on them.
How Apex helps
Our two-week CMDB health baseline scores which of your CIs are trustworthy and which are guesses. It shows where attributes and owners are missing, and the output is a report you can take to your change advisory board.
Two related pieces of work sit alongside it. A data quality assessment scores your estate against weighted critical success factors, key performance indicators (KPIs) and metrics. A discovery and integration coverage review establishes what is missing from your estate view and what that absence costs you during an incident.
Book a CMDB diagnostic call or arrange a meeting with one of our consultants.
Frequently asked questions
Should production machinery be in the CMDB?
If its failure or a change to it can stop production, yes. Scope by consequence rather than by who operates the device, because a change to an out-of-scope device can still take in-scope services down.
Which attributes matter most on CIs that have no owner?
Owner and support group come first, then model, firmware version and location. Those are the values someone needs in the first few minutes of an incident.
Who should own a device on a production line?
A named person with a deputy, at plant or line level. Certify the names on a schedule, so that a departure does not leave a CI pointing at nobody.
Is a CI with no attributes still useful?
It counts in your totals but cannot answer a question about the device. A CI earns its place when someone can use it to find out what the device is, who owns it and what depends on it.
Written by Kinga Staniszewska, Apex Configuration Group.
Book a free call to discuss how Apex can help you on your journey to a better CMDB
Recent
Topics
- ServiceNow (34)
- CMDB (26)
- CMDB Data Quality (11)
- Case Studies (5)
- Change Management (5)
- Duplicate CIs (5)
- CI Ownership (4)
- Compliance & Audit (4)
- Service Mapping (4)
- Vulnerability Management (4)
- Agentic AI (3)
- Data Model Design (3)
- Enterprise Risk Management (3)
- IRE (3)
- IT Governance (3)
- Incident Management (3)
- Service Graph Connectors (3)
- ServiceNow Discovery (3)
- Cyber Security (2)
- Dependency Views (2)
- IT Cost Optimisation (2)
- Manufacturing (2)
- Problem Management (2)
- CI Reconciliation (1)
- CMDB Remediation (1)
- CSDM (1)
- Configuration Management (1)
- IT Asset Management (1)
- ITOM (1)
- Operational Efficiency (1)
- ServiceNow Advisory (1)
Subscribe by email

Configuration Management Consultant at Apex Configuration Group, based in the Warsaw Metropolitan Area. Kinga specialises in operating the day-to-day processes of configuration management and guiding…
Full profile & credentials →You May Also Like
These Related Stories

Wiz Integration Doubled Cloud CIs With No De-duplication Tasks

CI Reclassification in ServiceNow: Managing Class Changes Safely

