Methods To Safely Modernize Critical Infrastructure

As a key provider of critical networking infrastructure to industry and government, Verizon routinely faces a set of complex challenges in maintaining operational availability while keeping pace with technology advancements. Unlike traditional IT system operations, critical infrastructure doesn't allow for planned downtime, with expected availability exceeding "six nines" (99.9999%)—that's less than 32 seconds per year of unplanned system-wide outages. Yet systems must still be updated, technologies refreshed, and new security threats countered. How does Verizon do it?

Controlling Change

According to Verizon Business Senior Vice President of 5G & Enterprise Solutions Debika Bhattacharya, the essential element is rigorous change management. "The challenge is changing a system that cannot be down. Change should be transparent to users. Verizon uses a multilayer process for every change: first, lab testing, followed by a pilot project, and finally backward and forward compatibility testing. At every step all stakeholders are involved in evaluating and approving results, and this is before anything is touched in the live network." A multi-department organization called the Change Advisory Board mediates communications between customers and Verizon stakeholders.

Every change also includes a "service wrapper" for end users, what Verizon calls "people management": quantifying the current customer service experience and ensuring that any change improves, or at least doesn't degrade, that service experience. "What does a change from TDM to IP mean for the user? Certainly, more features. But we can't degrade the reliability and clarity of existing voice and data services," says Bhattacharya.

Verizon's Chief Systems Engineer for Critical Infrastructure, Randy Hahn, expands on the service wrapper concept. "We understand that service quality is key, which we quantify as reliability, availability, survivability, and security (RASS). It's in Verizon's DNA. Yes, we have a packetized internal core running IP, including voice traffic. But we isolate TDM [Time Division Multiplexing] at the edge to bridge to new technologies with no reduction in RASS."

After a change has been designed, tested, and vetted for compatibility, with service wrapper requirements met, changes can be scheduled. They're classified as either routine or emergency. Routine changes take one path, while emergencies follow another, to ensure that emergency changes have the additional resources on hand to complete the work seamlessly. For both processes, a single control document, called a Methods of Procedures (MOP) in the telecom industry, specifies a detailed, step-by-step procedure for a particular job. The MOP incorporates test procedures before, during and after the maintenance event. "The MOP also includes a back-out plan for each step, should an implementation test fail," Bhattacharya notes, "so that the operational network is never put at risk." Verizon's architecture is built to ensure highly available networks to mitigate possible impacts of planned maintenance.

Identifying and Selecting New Technologies

System changes often involve deploying completely new technologies, such as Machine Learning (ML) in network security, or Software-Defined WAN (SD WAN) for communications path diversity. Any technology upgrade introduces risk. "We identify the risks and architect around them by, for example, using multiple layers of security or segregating administrative from commercial traffic," Bhattacharya says. "Whatever it takes to match or exceed legacy performance throughout the deployment. The risks of new technologies must be balanced against the risks of staying with older technologies: end of support, aging hardware, and software weaknesses."

In his hands-on role as Chief Systems Engineer, Hahn is directly involved in technology selection. "Verizon has separate product development teams, for forecasting needs and designing new service profile rollouts. We create a generic RASS model and tailor that to the customer's outcome expectations. We improve survivability wherever possible—to meet critical infrastructure requirements."

Users comfortable with legacy services sometimes resist new technologies, so Verizon helps mitigate their concerns through proven methods and procedures, in-depth regression testing and robust fall-back plans to realize the benefits of upgrades. Bhattacharya elaborates: "It's hard when technology is still working for the time being—customers often resist change, even though there are risks that aging infrastructure may fail. We explain how cost optimization, new features and capabilities, and improved RASS can be helpful to end users. We'll start with proof-of-concept lab demonstrations, to show customers the viability of the new technology. We then quantify the risk of not changing, complete technology validation testing and thorough transition processes with ongoing operational testing to measure performance."

The Future is Happening Now

One major technology deployment—going on right now—is 5G, the wireless network upgrade from 4G LTE, which could bring up to 100 times existing 4G LTE maximum speeds, new or expanded business models such as IoT and Multi-access Edge Computing (MEC), and a fresh security architecture. Hahn explains the new 5G service landscape: "The concept of the edge of the network is getting closer to our customers. Users at an airport could get access directly to their core business network from anywhere on the 5G edge, without traversing the entire infrastructure. Verizon moves the security concept of ‘zero trust' right to the edge." Instead of sending all data to a customer data center or a regional cloud for processing, a MEC platform located at the 5G network edge can securely analyze, process and store the data locally.

"We think of 5G on two layers: the public network, and the onsite networks," Bhattacharya explains. "It's all 5G, but we localize onsite as a private service for IoT and MEC to keep data local. Over the next three years, we'll spend $120 billion, and have acquired a lot of spectrum to deploy this architecture. 5G will eventually handle millions of IoT and other devices and process it in the network, while maintaining security, which is a huge game changer."

Ready to Go

Through tight change control, extensive technology review processes, commitment to revolutionary technologies such as 5G, and keeping the customer's operational environment front and center, Verizon successfully modernizes aging critical infrastructure: transitioning to new technologies, ensuring rigorous change management and hardening security. All while keeping RASS intact.