On October 20, the Bank of England’s Real-Time Gross Settlement (RTGS) system was halted for roughly 10 hours. From 6 am to 3:30 pm London time, absolutely no interbank or large business-to-business transactions could be processed throughout the United Kingdom. The RTGS is an interbank payments system designed to shuffle large payments one at a time on an immediate basis, meaning there is typically no waiting period for settlement and the payments aren’t batched or netted before being sent through the payment network.
The shutdown of the Bank of England’s RTGS system effectively froze £277 billion ($447 billion) worth of payments, which was everything running through the Clearing House Automated Payments System (CHAPS) that day. What lesson can we take from this 10-hour halting of UK bank operations? A simple one when it comes to financial risk management: it pays to be prepared.
What Happened to Halt the RTGS System?
Prior to its opening at 6 am London time on October 20, the technicians at the Bank of England, the U.K.’s central bank, took part of the RTGS system offline following maintenance over the preceding weekend. IT personnel had, according to Bloomberg News, updated the list of participating banks in the underlying database that runs the section of the RTGS system which governs CHAPS. This caused a fault in the software which forced the Bank of England to take the RTGS offline while they fixed it, a fact they didn’t reveal to the public until five hours after the RTGS system was supposed to be open.
CHAPS is a critical part of the UK’s financial infrastructure. Consumer home purchases flow through that system, as do high-value interbank transactions on behalf of the business community. Between its 140,000 average daily transactions totaling roughly £277 billion, the CHAPS system turns over the equivalent of Britain’s GDP every five working days.
Adam Horner, CTO of Satago, a UK-based payments technology provider, was amazed at the shutdown when we spoke with him about the situation, saying that “it is quite clear that best practices are not being followed at the BoE RTGS system maintenance level.”
Horner said that making database changes by hand, as it appears the Bank of England did, can cause significant issues downstream without a robust process to test changes each and every time. Because of this possibility, Horner said that “routine changes should never be manual, they should be automated, they should be reversible, and reversible doesn’t mean a process that takes 10 hours to resolve and restore service!”
What Impact Did the Halt Have?
CHAPS is used extensively for large business-to-business and interbank transactions, including home purchases and other big-ticket payments that require same-day settlement. With a daily average transaction total of £277 billion ($447 billion), which figures out to £70 trillion ($113 trillion) per calendar year, CHAPS is an integral part of the UK financial world.
The shutdown in the underlying RTGS system at the Bank of England resulted in CHAPS and the RTGS system remaining open until roughly 8 pm local time to clear all the transactions that got held up by the shutdown. Earlier in the day, several large “critical” transactions were conducted manually so the process could move through for at least a few accounts, according to The Guardian, but for the most part transactions couldn’t be completed.
News reports said several banks had to dust off manual contingency plans to allow business to operate as usual, sending paper contracts to show that money had in fact been sent into CHAPS. Mark Carney, the governor of the Bank of England, later in the day announced a thorough independent investigation to understand what went wrong.
Horner, of Satago, said that an automated process for database changes such as updating the list of member banks would simplify matters tremendously and allow errors to be noticed soon enough to avoid a massive 10-hour shutdown. Despite this, Horner said that “The investment required to move an established organization, like the BoE, to a highly automated regression testing environment is high, with costs in obvious areas like technical systems and infrastructure, but also with high costs in non-obvious areas like cultural change, outage responsibility and ownership assignment.”
Horner also said that startups like Satago and other midsized companies learn early on that automation is the way to avoid a lot of problems. In his view, this is one of the key lessons to take away from the RTGS system shutdown—critical systems should never be updated manually.
What Lesson can Enterprises Learn?
At its core, the Bank of England’s RTGS system is a technology process and, like most technology, can fail for completely benign reasons that have nothing to do with cybercrime. Despite the Bank’s claims of RTGS being “risk-free,” there is in fact a very real risk—if the system is shut down to avoid a wider fault, payments infrastructure basically grinds to a halt.
In the case of CHAPS, there aren’t very many alternatives for large payments made in real-time. Horner said the Faster Payments Service and BACS are alternatives for businesses needing to make payments, but the Faster Payments Service is limited to transactions of £100,000 ($159,988) or less and BACS transactions take much longer to clear than CHAPS, in addition to having the ability to be recalled (something which CHAPS doesn’t allow).
So what’s an enterprise to do when their payment system goes down? There are a few options, such as switching to a different payment portal or manually contacting creditors and assuring them of payment; this assumes of course that the enterprise is aware in a timely fashion of problems in their payment system. . Enterprises should have access to multiple methods of payment in case of something similarly catastrophic; moreover, the contingency strategy needs to be current, viable, and well-understood by all the relevant players. This isn’t something to develop and then leave on the shelf in the CFO’s office. Much like a fire evacuation plan, the relevant participants should periodically drill what to do in the event of a payments-system shutdown.
For U.S.-based enterprises, consider what to do if either Fedwire or the Clearing House Interbank Payments System (CHIPS) were to experience a shutdown. The CHAPS shutdown left the banks and enterprises that depend on it in the lurch, forcing those banks and enterprises to scramble to assure whomever was waiting for payment that money was sent into the system—it just couldn’t get back out yet. In terms of business continuity, the ability to communicate with suppliers in this manner is critical; an average transaction through CHAPS was £2.1 million ($3.4 million), so having a communication or alternative payment strategy is absolutely critical for business continuity.
Horner also said changes are on the horizon for this landscape. “Guaranteed same day payments of significant value are just one of the areas where new FinTech startups are beginning to invent innovative alternative solutions,” he said, “The more problems that are exposed with CHAPS, the more enticing these startup services will look as more and more companies consider the risk of the unknown to be lower than the risk of lower reliability from a system as old as CHAPS.”
Final Thoughts
Financial technology like the UK’s RTGS system is, at its core, nothing more than a collection of computer code designed to perform a process. As the world becomes more interconnected, the enterprises that possess—and test—strategies to mitigate B2B payment risk factors such as technology failure are the ones that have a higher possibility of avoiding negative consequences in the long term. When it comes to the financial function, after all, there is no such thing as too careful.
Interested in more news like this? Sign up for the Ardent Partners email newsletter today and get it sent direct to your inbox!
