Preventing the $9.3 Million/Hour Loss to Voice Trading Downtime

Automation can stop revenue loss, improve voice trading service 

Missing an important call can be especially costly for financial services firms, particularly capital markets companies that engage in voice trading. Downtime can mean a significant loss — averaging $9.3 million per hour — not to mention the risk of long-term reputational damage. Factoring in compliance and regulatory pressures makes uptime for voice trading systems a top priority.

Given what is at stake both financially and from a brand image perspective, investment firms are looking for better ways to manage their voice trading systems, boost reliability and mitigate risk. The process is largely manual and even reactive today. Issues are often discovered first when the voice trading system is down, or a turret is not performing. This often translates into longer downtime, as firms need to first diagnose what is causing the outage before actively addressing it.

Automated solutions are coming to the market that can validate performance and foresee issues, allowing you to address them preemptively. They have the ability to maximize communications uptime and identify potential issues as they arise – in real-time. 

Why Voice Trading Matters

While most securities (stocks, bonds and derivatives) are currently traded electronically, fixed income, commodities and currency (FICC) trading is largely based on voice trading utilizing a phone or Unified Communications (UC) platform including voice trading systems (Dealerboard or Turret systems) and Private Branch Exchange (PBX) platforms. Voice trading is not only important, it’s mission critical and heavily dependent on the functionality and the reliability of the Dealerboard/Turret Systems.

Since FICC trading is conducted globally 23 hours each market day, special solutions are required that include voice trading software applications and turrets as endpoints. FICC trading is subject to tighter regulations due to the financial instruments traded and the risk of speculation. To meet compliance requirements, the solutions include recording systems for email, text and voice and a plethora of compliance related applications.

Effective asset management, trading and brokering are relevant factors in the performance and sustainability of financial services companies and bring more opportunities for expansion and re-investment. These companies have a key role in this value creation chain. UC technologies as well as other Information Systems support this fulfillment. For traders and investment portfolio managers, it is crucial to have control over their operations and know that:

  • that transactions can be completed
  • the timing for the transactions is correct and meets compliance
  • the UC systems are reliable to meet the demands of mission-critical trading

Value creation is a constant endeavor in this industry and optimizing tech resources to improve and accelerate connected business can significantly increase profitability. This value will eventually turn into equity growth and lead to resource expansion for future operations. Technologies such as voice trading systems must be managed efficiently to fulfill the company’s objectives and any tool or service that helps manage resources is an asset that every organization should strive to obtain to improve efficiency. That is the core of the value proposition for UC technologies that support almost every operation on different levels.

The Compliance Challenge

In addition to maximizing profitably with reliable voice trading networks, financial service firms must also meet regulatory obligations. Trading must be halted if compliance is not possible, for example, due to a malfunction of the recording system, which translates to loss of revenues until the functionality is restored. Large players in FICC trading have trading volumes in the billions of dollars per fiscal year, meaning every minute a system is shutdown translates to substantial losses. It goes without saying that the reliability of the voice trading solutions is critical to the financial success of the business.

 Six (or more) nines are better than five. An ITIC survey from 2016 found that reliability requirements for the Banking and Finance sector were above the average of other industries, and 86% of companies in this sector require a minimum of 99.999% (five nines) of uptime from their IT systems. Moreover, 27% of companies in this sector require more than ‘five nines’ or less than one hour per year of downtime. This requirement can be translated into a financial impact when companies from this sector report that they can lose an average of $9.3 Million during one hour of downtime.

Mitigating Risk

Risk management is core to financial operations and systems testing is a process that must be included to mitigate risk in trading environments. UC testing is required to prevent operational losses – estimated in the millions of dollars per hour for many firms. Different actions can be taken during the risk management assessment loss of transaction traceability such as voice recording of calls, can result in large fines compounding the and investment stages, and their implementation may have different costs.

Automation Preempts the Disconnect

Manual testing of systems is a good start but a more cost-effective way is to automate the test processes. Automation is more efficient because testing can be continuous and could also be applied on a regular basis to validate that systems are working properly and are compliant.

Automate, automate, automate! Manual testing could be your first option to prevent downtime of your system to avoid significant losses. Manual testing costs can be as high as 6% of your UC investment forcing your engineers to work on a time consuming and repetitive tasks. In the era of technology acceleration and after investing millions of dollars in your voice trading solution, it is prudent to invest in automation that improves your workflows, further reduce risks and enables your engineers to focus on more innovative activities.

Probability of failures increases with the number of turrets and operational testing workflows should have at least three layers of application of test automation:

  • Functional testing using established testing methodologies in a testing lab, running targeted tests.
  • Risk mitigation of onboarding and software upgrades of your new and/or existing systems.
  • Proactive testing of system functionality and compliance via scheduled tests.

If no testing is included in the operational workflow, potential losses will likely to happen and are unknown. Manual testing is the first alternative to prevent those losses and mitigate that risk. However, this requires engineers to either follow a schedule to test all turrets/lines or run random tests, translating to a high operational fixed cost. Automation accelerates testing by an order of magnitude which enables more frequent test cycles, around the clock testing and a larger test coverage at a fraction of the cost. Automation in average can cost a third of what manual testing costs under the same conditions and scale.

The Hard Numbers 

UC systems that are not tested have a base reliability of no more than 99.5% guaranteed uptime under industry standards. That means at the very highest performance level, an investment firm will lose 29 hours. At an average loss of $9.3 million per hour, that’s a costly $269.7 million thrown away.

Manual testing increases system uptime to 99.8% on average, a 0.3 percent improvement in reliability that cuts downtime to 12 hours. That’s still $111.6 million in losses that could be prevented by keeping voice trading systems up. Plus manual testing adds operational costs to deployments and regular upgrades.

Automation can increase system reliability up to minimum of 99.99%, which reduces downtime per year to 36 minutes or less. That brings voice system outage losses down to about $5.6 million. That’s still a hefty sum, but a mere fraction of what is lost through manual test or no testing.

A more proactive automated program that increased frequency and includes after-hours testing can get that reliability to ‘five nines,’ driving annual downtime to five minutes. That brings the downtime loss below a million to $775,000.

An automated solution costs about two percent of the overall UC and voice trading system and represents a 60 to 70 percent annual savings compared to manual testing. That’s rather modest compared to the savings in preventable loss, in the hundreds of million dollars.

The tekVizion automation platform can help your company increase the reliability of your voice trading systems by automating the testing of your software upgrades. With tekVizion automation you can now introduce continuous nightly testing to ensure that the production systems are ready for trading when the market opens. The automation increases your uptime and mitigates the risks in your production environment. tekVizion’s automation platform accelerates the delivery of the latest features and functionality to your traders and increases their productivity and value creation for your business.

To learn more about tekVizion automation and other test automation solutions, go to:

Share This Story: