Source: Compiled and translated from IDTechEx

With the continuous advancements in optical transceivers in terms of bandwidth, energy efficiency and integration, their scope of influence has expanded from data center networks to the very architecture of artificial intelligence systems themselves. Today, the transition to optical interconnection is driven not only by bandwidth demands but also by the growing contradictions between the scalability of electrical SerDes, system power budgets and physical architecture constraints.

To understand how this shift is unfolding, it is useful to first distinguish between scaling architectures and connectivity within AI systems.

Vertical Scaling vs. Horizontal Scaling, and the SerDes Challenge

Vertical scaling refers to maximizing performance within tightly coupled systems (e.g., a single server or accelerator domain). Its goal is to aggregate more computing power, memory and bandwidth while maintaining ultra-low latency and high synchronization.

From a physical perspective, scalable network architectures for vertical scaling feature short transmission distances, typically within a server or a single rack, often well under ten meters. In this domain, high-speed copper interconnects still dominate, supported by mature electrical serializer/deserializer (SerDes) and protocols (e.g., NVLink), as well as emerging open alternatives.

In contrast, horizontal scaling distributes workloads across multiple servers to boost total system throughput. Once communication extends beyond a rack or row of racks, optical interconnection becomes critical. As such, Ethernet and InfiniBand form the backbone of today’s large-scale AI clusters, enabling high-bandwidth, energy-efficient communication over distances ranging from tens to hundreds of meters.

A simplified AI accelerator architecture illustrates how these two domains coexist. At the compute layer, accelerators connect upward to L1 compute switches via high-bandwidth copper links. These are classic vertical scaling connections: short-reach, high-density, and optimized for moving massive volumes of data with minimal latency. L1 switches are also interconnected via copper, forming a tightly coupled network fabric that allows multiple accelerators to operate almost as a single large device at the software level.

As traffic moves up the hierarchy, it converges on Layer 2 network switches that connect to the broader data center network. At this level, optical pluggables predominate, as the system must support longer transmission distances, a higher number of ports, and scalable bandwidth growth.

A mounting challenge for both domains is that while electrical SerDes continue to evolve, their system-level limitations are multiplying. On silicon, SerDes capacity is steadily scaling from 112G to 224G PAM4 and beyond. However, as data rates rise, the electrical channel—including packaging, substrates, PCB traces, connectors and cables—gradually becomes a bottleneck. Maintaining signal integrity over longer reaches requires increasingly powerful equalization and digital signal processing (DSP) capabilities, which drive up power consumption per bit and add thermal load.

For large AI switch and accelerator architectures with thousands of SerDes channels, even a small increase in energy per bit translates to hundreds of watts of power at the rack level. SerDes have thus evolved from a mere circuit-level concern to a primary architectural constraint.

Co-packaged Optics (CPO) – What Role Does It Play?

It is here that optical components, particularly co-packaged optics (CPO), are beginning to reshape system design.

According to a report published by IDTechEx, the short-to-medium term evolution of AI system architectures will be evolutionary rather than revolutionary. Copper remains highly effective in the vertical scaling domain as long as links stay short, controlled and energy-efficient. Leading platforms, especially companies like NVIDIA, continue to heavily promote copper for vertical scaling architectures, citing its low latency, cost advantages and, crucially, scalable reliability. From this perspective, optical components are not yet a full replacement for copper in tightly coupled GPU architectures.

Instead, the most immediate pressure point lies at the network switching layer.

Today’s horizontal scaling connections rely on pluggable optical modules installed on switch front panels. However, as switch ASIC bandwidth scales from tens of terabits per second to hundreds of terabits per second, this pluggable model faces mounting challenges in terms of power consumption, signal integrity and front-panel density. Once links must traverse long electrical paths to the front panel, simply scaling SerDes to higher data rates becomes increasingly inefficient.

CPO addresses this issue by placing the optical engine much closer to the switch ASIC, typically within the same package. By drastically shortening the electrical path, CPO reduces I/O power consumption, improves signal integrity, and enables higher total bandwidth scaling without relying on increasingly complex electrical channels. In effect, the optical module solves the transmission distance problem, while the electrical SerDes is confined to its most efficient operating range.

This selective deployment strategy aligns with the views of different industry players on optical link adoption. While NVIDIA remains copper-first for vertical scaling, it is clear that optical links—including CPO—play a critical role in scaling network architectures. Other vendors, such as Marvell and Broadcom, appear more open to introducing optical links in vertical scaling architectures.

In the long term, the line between vertical and horizontal scaling itself may become less distinct. As the number of accelerators per logical node increases and system physical dimensions grow, even copper-based vertical scaling architectures, while still feasible in terms of electrical performance, will face mounting pressure from power density, airflow and wiring complexity. In such scenarios, optical I/O may also begin to play a role in vertical scaling, particularly in inference-optimized architectures where throughput per watt is more important than ultra-low latency.

Key Takeaways

The crux is that optical components will not fully replace copper overnight. Instead, AI system architectures are evolving through a pragmatic division of labor: copper remains dominant in scenarios where latency and reliability are critical and transmission distances are short, while optics scale in scenarios where electrical SerDes scaling conflicts with power, reach and density constraints. Co-packaging of optics represents a key inflection point in this evolution—it does not replace SerDes, but rather confines it to its optimal physical and economic range.

*Disclaimer: This article is created by the original author. The content reflects the author’s personal opinions only. Our reprint is solely for the purpose of conveying a different perspective and does not imply our endorsement or support of such opinions. For any objections, please contact our backend team.