Companies in highly regulated industries face a host of issues when they experience data loss and corruption, from the time lost dealing with incidents to the financial cost of recovery. Here are some real stories of how data loss occurs in ServiceNow—and how Own could have augmented the native ServiceNow backup and recovery capabilities to help prevent these issues.
The cost of data corruption in the world of banking and finance
For banking, financial services, and financial technology companies, data is gold. Companies such as these use the ServiceNow platform for a range of activities, including managing IT services (ITSM) and operations (ITOM), customer service management (CSM), and HR service delivery (HRSD), among others.
Banks may also use custom ServiceNow apps for investigations: working with agencies to prevent fraudulent activity.
Big bank bust
One multinational bank suffered data loss during a data normalization cleanup activity. All users, branches, incidents, and assets in their configuration management database (CMDB) defaulted to a single location in London.
It took several team members more than a week to restore one field alone. They had no way to know if they actually got all of their data back because they knew one field had been changed, but had no idea if associated fields had been corrupted as well.
That represented hundreds of resource hours and delays to other projects. The restoration also delayed technical support to the wider business, because the team had no way to locate critical assets that needed fixing.
Own’s Smart Alerts control data loss by alerting users before it goes too far. The bank's team could have used Own's self-service restore tools to know exactly which fields had been changed and fix them at a granular level. Preventing issues is a lot cheaper than fixing them.
Disintegration overnight
A US fintech company learned that overnight integrations can cause havoc: In their case, a data corruption issue locked over 2,000 employees out of both physical buildings and ServiceNow.
Because the customer service team got locked out, they couldn't work—meaning the company couldn't meet its customers' SLAs.
Meanwhile, the IT department—who would’ve been responsible for fixing the issue—was also locked out and couldn’t help. This only compounded the problem, as IT wasn’t able to use ServiceNow’s native capabilities to roll the entire instance back. And because the company was unable to risk losing new data, it took over a day of manual updating and restoration. In some cases, they even had to create new user records—costing a lot of time and money.
Own would have notified IT about the login issues before employees started calling about it. Own also restores affected users smoothly and painlessly: Corrupted fields could’ve been restored within hours at most without impacting other business lines.
Upgrade in name, downgrade in effect
Finally, a multinational financial services company based in the US found its ServiceNow business tables compromised by auto-management during an instance upgrade. When the upgrade re-enabled the auto-management feature, it caused widespread data corruption.
Turning the feature back on queued all configuration items into a stack for processing. The company does 80% of its business—with 21 terabytes of data—via ServiceNow. It took over a month to restore the company’s services and apps—and they still aren't sure that they restored everything, even after spending millions of dollars.
Own would’ve enabled the company to trigger an on-demand backup as soon as the company noticed the issue. This would have precisely restored only the impacted data at a table and column level.
The company could have also anticipated the impact with a comparison: either between two points in time, or between their subproduction and their production.
Telecommunication pipeline breakdown
Data corruption doesn’t just hit banking and finance companies. An EMEA-based telecommunications company learned about a ServiceNow-GitHub integration bug the hard way.
This particular company primarily used ServiceNow’s DevOps, source control, and continuous integration and continuous delivery (CI/CD) products. The bug not only caused data loss, but also prevented the company from recovering code from the GitHub repository.
The broken CI/CD pipeline halted development work and stopped new project production. A ServiceNow support ticket remained unresolved for four weeks, creating a two-and-a-half-month backlog and costing millions of dollars to fix.
Own would have recovered the project data and restored the integration—possibly within hours. The company would have been able to identify exactly what was happening very quickly and restore from a previous backup with Own’s Recover solution, meaning little to no financial fallout.
Biotech: a wipeout and a save
Another global industry that increasingly uses vast amounts of customer data intersects at healthcare, life sciences, and business services.
In addition to ITSM and ITOM, companies focused on pharmaceuticals, biotech, and diagnostics often use the ServiceNow platform for integrated risk management (IRM) and governance, risk, and compliance (GRC).
Failure to back up
Developers at one biotech company based in the EMEA region cloned data from production to sub-production. There was just one problem: They forgot to back up their data in their development and test environments.
This is common with many teams working on sub-production instances who feel protected by the cloud. But here, the developers overwrote their data with each cloned instance. Trying to figure out where to recover the data from—and if it was even possible—caused unnecessary business-supporting project delays. It took three days to recover the data, and idle developers also lost a week of productivity due to the rework.
Own schedules backups and takes on-demand backups of sub-production instances before cloning from production. Running backups significantly mitigates cloning risks, and these instances enable business continuity and reduce team idleness while issues are resolved.
Own mitigates downstream impact
A U.S.-based biotech and diagnostics company fared much better: When they hit a snag, they already had Own’s support.
Simple human error caused the deletion of several update sets and hundreds of records by selecting the wrong export option. The company had ServiceNow’s Delete Recovery module, but when they tried to use it, it failed. This meant that production instance changes couldn’t go live until the data was recovered.
Luckily, the company already had Own and was able to easily follow the restore process. The first time they ran the restore process with Own, they identified an issue with the access control list (ACL), which dictates which users can push integrations back into ServiceNow. (This is why ServiceNow's Delete Recovery module didn't work.) After reconfiguring the ACL to fix the issue Own identified, they successfully restored the deleted data and pushed production instance changes live. Ultimately, they were up and running in under two days with no further negative impact.
Own Recover for ServiceNow: An extra layer of data protection
Multinational corporations can’t afford to leave themselves open to easily avoidable data loss and corruption. Millions of customers depend on their services and products every day.
ServiceNow enables companies to provide top-notch services to customers and colleagues alike. But such complex software is easy to misconfigure. When businesses take the right precautions to avoid data loss, they reap the benefits of ServiceNow without the risks.
If you want to prevent costly, time-consuming data loss and corruption issues, schedule a demo of Own Recover for ServiceNow.
