A change request went out for a single piece of infrastructure. The service map showed one application sitting on top of it. The change went ahead. When the device came offline, several business applications failed at once, not one.
Key Takeaways
- A change went ahead against a piece of infrastructure whose service map showed one application. When the device came offline, several business applications failed, including the advertising schedule application, at a direct cost of $600,000 in advertising revenue.
- The map came from a tagging exercise in which application managers tagged the infrastructure their applications used. Shared switches, storage arrays and load balancers were tagged for only the application each manager considered most important.
- Tag-based mapping fails quietly. Every application has a map, so the picture looks complete, and nothing checks whether a component appears in more than one manager's inventory.
- The audit found the same single-application bias on every shared component it checked. The work changed from correcting one map to rebuilding the mapping approach across the estate.
- Top-down, pattern-based service mapping starts from the business application, captures many-to-one relationships as they exist, and reads real dependencies from configuration and connectivity data.
- A change against a shared component now carries a list of every dependent application, generated from the service map. A shared component showing several applications is a flag in its own right.
This is what happens when service mapping is built from a tagging exercise instead of from the estate itself, and it's worth setting out exactly how it happens. For a fuller picture of what a CMDB needs to hold to avoid this, see our guide to how an enterprise configuration management database is structured. The fix runs deeper than more careful tagging. It means mapping the estate differently from the start, which is what the rest of this piece sets out.
How a tagging exercise misses shared components
The client's estate had gone through a tag-based mapping project. Application managers were asked to tag the infrastructure their applications used, then those tags were used to build a service map in the CMDB. On paper, every application had a map.
The gap was in the assumption behind the exercise: that each piece of infrastructure serves one application, and the person responsible for tagging it knows every application it serves. Neither held. Underpinning components in this estate, like shared storage arrays, network switches and load balancers, routinely served more than one business application at once.
When application managers tagged their infrastructure, most tagged only the application they considered most important. A shared switch serving four business applications got tagged for one. The other three had no record of depending on it at all.
What broke when the change went ahead
A change was raised against a critical piece of infrastructure. The change advisory board checked the service map, saw one application listed, and assessed the risk on that basis. There was no indication that anything else depended on it.
When the device was taken offline for the change, multiple business applications and services failed. The advertising schedule application was one of them. For a broadcaster, that's the application that puts commercials into the transmission. When it went offline, the commercial breaks ran with no commercials in them, at a direct cost to advertising revenue of $600,000 and a 4% drop in the company's share price on the day.
The connection between the change and the outage wasn't obvious at first. Problem management linked the incidents back to the change only after the fact, once someone traced the failed applications back to the same piece of shared infrastructure. That lag matters. In a live broadcast environment, the gap between an outage starting and its cause being understood is the gap during which nobody can say with confidence what else might be at risk.
It also meant the first response treated the failures as unrelated incidents rather than one root cause. Three separate incident tickets were opened before anyone connected them, each one worked by a different team without visibility of what the others were seeing.
Why the gap wasn't caught sooner
A subsequent audit of the estate's applications found something bigger than one missing link. The gap was structural, present across every shared component in the environment. Any application whose underpinning infrastructure was shared, and hadn't been marked as the primary user, was invisible to change impact analysis.
Tag-based mapping fails quietly. It produces a service map that looks complete, because every application has one, right up until a change touches infrastructure the map didn't know was shared.
The tagging exercise itself wasn't badly run. Application managers did what they were asked: identify the infrastructure they used and tag it. The instruction didn't account for shared ownership, and there was no validation step afterwards to check whether a piece of infrastructure appeared in more than one application manager's inventory. A component tagged by two people would have been an early warning sign. Nothing in the process was set up to surface that.
What the audit found across the rest of the estate
Once the pattern was understood, Apex checked whether it was isolated to the infrastructure involved in the outage or systemic across the estate. It was systemic. Every shared switch, storage array and load balancer checked showed the same single-application bias in its tagging, to varying degrees, because the same instruction had produced the same blind spot everywhere it was applied.
That's the finding that changed the scope of the work, from correcting one service map to rebuilding the mapping approach across the estate. A targeted fix on the infrastructure involved in the outage would have left the same risk sitting on every other shared component, waiting for the next change to find it.
What top-down, pattern-based mapping does differently
Apex rebuilt the service maps using top-down, pattern-based ServiceNow service mapping instead of tag inheritance. The difference is in where the relationship comes from:
- Start from the business application, not the infrastructure. Pattern-based mapping traces down from each business service through its technical services to the actual infrastructure it depends on, rather than relying on someone tagging the infrastructure from the other direction.
- Capture many-to-one relationships as they exist. A shared switch, storage array or load balancer gets linked to every application that depends on it, not just the one a person judged most important.
- Validate against real traffic and configuration, not intent. Pattern-based discovery reads the actual dependency, through configuration and connectivity data, instead of relying on a person's memory of what uses what.
Once the maps were rebuilt this way, change impact analysis had something to work from. A change against a shared component now shows every application it could affect, before the change advisory board approves it, not after an incident links them together.
The rebuild also changed how the change advisory board reviews risk, in a way that connects directly to how good change impact analysis depends on the CMDB behind it. Before, a service map showing one application was treated as evidence of low impact. Now, a shared component showing multiple dependent applications is a flag in its own right, regardless of which single application first raised the change. The board's question has shifted from what this application needs to what else sits on the same infrastructure.
That question changes what a change record actually needs to contain before it reaches the board. A change against a shared switch now carries a list of every dependent application, generated from the service map rather than typed in by whoever raised the request. The board isn't relying on the requester to know, or to remember to mention, that three other applications sit on the same hardware. The map tells them, which is the difference between impact analysis that depends on one person's memory and impact analysis that depends on data.
What this means for your own change advisory board
If your service maps were built from a tagging exercise, ask a narrower question than "do we have service maps." Ask whether a piece of shared infrastructure, like a switch, a storage array or a load balancer, could be linked to only one of the applications that actually depend on it. If nobody can answer that with confidence, your change advisory board is assessing risk on an incomplete picture, the same way this one was.
How Apex helps
We run a discovery and integration coverage review that establishes what's missing from your estate view, where the gaps sit, and what they would cost you during an incident. For an estate mapped by tagging exercise or spreadsheet, this is usually where the shared-component gap shows up first, before a change finds it for you.
Book a CMDB diagnostic call to talk through what your current service maps do and don't cover, or go straight to booking a meeting with the team.
Frequently asked questions
How is pattern-based service mapping different from tag-based mapping?
Tag-based mapping relies on a person marking which infrastructure their application uses, from the application side. Pattern-based mapping starts from the business application and traces down through its technical services to the infrastructure automatically, using configuration and connectivity data, so it can capture infrastructure shared across several applications, not just the one a person tagged.
How would we know if our service maps have this gap?
Check a handful of shared components, such as core switches, shared storage or load balancers, and see how many applications they're linked to in the CMDB against how many actually depend on them. If the CMDB shows one application where you know there are several, the maps were likely built from tagging rather than from the estate itself.
Does this only affect broadcast environments?
No. It's most visible in environments with a lot of shared infrastructure underneath many applications, which includes broadcast, but the same tagging-exercise gap turns up in any estate where service maps were built by asking application owners what they use, rather than by tracing dependencies from the infrastructure up.
Can this be fixed without a full re-platform of the CMDB?
Yes. Rebuilding service maps with pattern-based discovery works against the existing CMDB and ServiceNow instance. It's a mapping exercise, not a data model change, though it does need the underlying discovery and pattern extensions configured correctly for the estate.
Written by Warren O'Neill, Apex Configuration Group.
Book a free call to discuss how Apex can help you on your journey to a better CMDB
