What Is the AI Memory Wall? How Optical Interconnects Ease Inference Bottlenecks

Slow AI inference can result from memory failing to supply data quickly enough, even when GPU compute is sufficient. Optical interconnects do more than extend transmission distance: they allow new compute and memory layouts to move beyond the limits of short electrical connections, creating opportunities to increase capacity and bandwidth together.

The AI Memory Wall: When Data Supply Cannot Keep Up with Compute

The memory wall is the gap between a processor's computing capability and the speed of memory access, which limits performance as the system waits for data. In AI systems, a GPU may have ample compute capability yet remain underused because data cannot reach it fast enough.

For AI deployment, memory constraints can be separated into three areas:

Insufficient capacity and the memory wall need to be assessed separately. More capacity holds more data; transmission paths and bandwidth determine how quickly that data can be used for computation.

Why AI Inference Faces Memory Constraints

AI inference repeatedly reads model weights and context data. Longer contexts and more concurrent requests increase memory requirements, while generating tokens step by step can be limited by memory bandwidth.

High Bandwidth Memory (HBM) is a stacked memory technology widely used in AI accelerators. It supplies data to GPUs through wide, short electrical connections. HBM typically sits close to the compute chip within the same package, using short connections to achieve high bandwidth and energy efficiency.

Large language model inference has two stages: prefill, which processes the input, and decode, which generates new tokens one at a time. The decode stage is particularly susceptible to memory bandwidth constraints, although the actual bottleneck depends on the model, batch size and hardware configuration.

The key-value cache (KV cache) stores keys and values that attention calculations reuse. This reduces repeated computation but consumes additional memory. For a simplified full-attention model:

KV cache size in bytes ≈ 2 × number of layers × number of KV heads × head dimension × cached token count × bytes per element × number of sequences.

Assume 32 layers, 8 KV heads, a head dimension of 128 and 2 bytes per element, with 32,768 cached tokens per sequence. One sequence requires about 4 GiB; 8 independent sequences require about 32 GiB. This illustrative estimate excludes model weights, working memory and management overhead. Cache strategies, shared prefixes and sliding windows can also change the result.

How Optical Interconnects Connect Compute and Memory

An optical interconnect transmits data using optical signals between components or systems. It can help address the distance, density and energy constraints of high-bandwidth transmission.

As electrical connections become longer, maintaining high bandwidth while controlling signal degradation and transmission energy becomes more difficult. HBM therefore usually sits close to the compute chip, and the amount of memory that can be placed around it is constrained by chip edges and package space. Optical interconnects relax the distance constraints of high-bandwidth links, allowing new architectures to place more memory chips across a larger area and increase capacity and aggregate bandwidth together. This requires redesigning memory interfaces, controllers and packaging; existing HBM cannot simply be moved farther away by attaching optical fiber. Actual bandwidth, latency and energy use still depend on the complete architecture.

Intel's optical I/O demonstration illustrates the direction of bringing optical interfaces closer to compute chips. Optical connections can also support resource disaggregation, but memory pooling requires controllers, access mechanisms and software to manage multiple memory components as usable resources.

The value of adoption depends on the full data path. Energy saved on optical transmission must be assessed alongside conversion, control and memory power consumption.

How CPO, Optical I/O and Optical Interconnects Differ

Optical interconnect describes the transmission method, optical I/O describes the input/output interface, and co-packaged optics (CPO) describes how optical components are integrated with electronic chips. All three can exist in the same system. Optical I/O can implement an optical interconnect and can be integrated into a package through CPO; these are not mutually exclusive technologies.

Deployment location determines which bottleneck a technology addresses. NVIDIA's CPO network switches, for example, focus on cluster networking. Optical connections between compute chips and memory must meet different access requirements. Upgrading a switch does not directly increase the memory capacity of an individual GPU.

Three Startup Approaches: From Chip I/O to Memory Pools

Ayar Labs, Lightmatter and Volantis approach the problem through chip interfaces, interconnects within a package and memory architecture, respectively. A useful comparison starts with what each technology connects, then identifies which part of data movement it improves.

Progress reflects each company's public disclosures in 2026. Refer to the companies' announcements for the latest status.

Ayar Labs: Optical I/O Chiplets Designed for Package Integration

Ayar Labs' TeraPHY connects to compute chip systems through a UCIe electrical interface, while SuperNova provides an external light source. It makes optical connectivity a component that can be integrated into a package, extending high-bandwidth connections between accelerators to more distant nodes.

This approach focuses on how chips exchange data. Evaluation should cover whether the compute chip can connect to the optical interface and whether link distance, latency and light-source placement meet system requirements. The memory capacity of an individual accelerator still depends on its underlying architecture.

Lightmatter: Increasing Interconnect Density Through 3D Integration

Lightmatter's Passage L200 vertically integrates photonic and electronic chips, extending optical interfaces from the chip edge across its area to increase the number of data channels a package can support. It addresses the difficulty of providing enough peripheral connections as chip compute capability grows.

While Ayar Labs emphasizes integrating optical I/O chiplets and external light sources, Lightmatter's L200 places greater emphasis on the vertical arrangement of optical and electronic components within the package. Their applications may overlap, but interface compatibility, packaging and bandwidth density provide useful points of comparison.

Volantis: Redesigning the Connection Between Compute and Memory Pools

The commercial maturity of optical interconnects varies by deployment location. Data centers already use optical communications, while bringing optical links deeper into the connection between compute chips and memory still requires product-by-product verification.

In October 2026, Volantis announced plans for its A-1 inference system. It uses optical connections to form pools of memory chips, aiming to increase aggregate bandwidth as capacity grows. The company plans to deliver its first integrated inference engines to customers in 2027.

According to Volantis' technical explanation, the design uses optical waveguides on an interposer to connect more memory chips into a shared pool. It is not an external remote-HBM add-on for existing GPUs. These chip-to-memory connections address a different bottleneck from the CPO switches used for cluster networking.

The plan illustrates how interconnects can change memory layouts. Commercial value still depends on supported models, sustained operation and total system cost. The company's speed and capacity targets should be distinguished from results in customer deployments.

Three questions clarify the approaches: how does Ayar Labs connect chips to optical links? How does Lightmatter fit more interconnect channels into a package? How does Volantis allow more memory to feed inference compute? The first two focus on interconnect components for integration into other chip systems; the third incorporates interconnects and memory layout into an entire inference system design.

Memory Vendors, Chipmakers and Packaging Platforms Move Together

Deployable optical interconnect systems require coordination across memory architecture, chip products and package manufacturing. Work by SK hynix, Marvell and TSMC reflects progress in these three areas.

SK hynix: From HBM to Optical Interconnects at the System Level

SK hynix and academic researchers jointly published a CPO research roadmap proposing optical connections between compute and memory resources to address data bottlenecks within racks and larger clusters. This extends the discussion to a memory vendor's perspective on system design. The paper describes a development direction rather than production qualification for a specific product.

Marvell: Bringing Startup Technology into a Larger Chipmaker

Marvell completed its acquisition of Celestial AI in February 2026, bringing the company's Photonic Fabric optical interconnect technology and team into its data center business. This allows optical interconnects to develop alongside Marvell's existing connectivity products and adds resources for customer integration. Completion of the acquisition does not establish that all solutions are shipping at volume.

TSMC COUPE: Bringing Optical Engines into Package Manufacturing

TSMC's Compact Universal Photonic Engine (COUPE) integrates optical engines into packaging to address the manufacturing and integration of optical and electronic components. At its 2026 Technology Symposium, TSMC announced plans for COUPE on substrate to begin production that year. This provides a path toward packaging implementation; validation and shipments of individual customer systems still need to be confirmed separately.

Optical interconnects are advancing along several parallel paths, from startup interfaces and architectures to vendor research, larger-company integration and packaging platforms. Assessing progress requires separating component, package and complete-system results, then validating them against actual models, latency, power consumption and deployment cost.

How to Assess Memory and Interconnects for a GPU Cluster

Enterprises deploying GPU clusters should first define their model and service requirements, then validate memory capacity, data transfer and facility readiness separately.

1. Capacity: how much memory is needed for weights, the KV cache and working data combined?

2. Service performance: do time to first token and sustained generation speed meet targets at the intended context length and concurrency?

3. Data paths: is the bottleneck in GPU memory, between GPUs, across the network or in storage?

4. Deployment conditions: can power, cooling and expansion space support the target configuration?

Use the same model, precision and service conditions for testing. Identify capacity and data-transfer bottlenecks before deciding whether to adjust caching, the single-server configuration or deployment across nodes. If the model and service scale require multiple GPU servers to work together, plan node connectivity, networking and storage, and confirm the associated power and cooling requirements.

FAQ

Are Optical Interconnects the Same as Optical Computing?

Optical interconnects use light to transmit data; optical computing uses optical mechanisms to perform computation. A system with optical interconnects can still use electronic processors and electronic memory.

Can Optical Interconnects Replace HBM?

Optical interconnects and High Bandwidth Memory (HBM) operate at different levels: one is a connection method and the other is a memory technology. New systems may use different memory arrangements, but they require a complete architectural redesign and validation. Optical interconnects cannot be treated as a direct replacement component for HBM.

Does the Memory Wall Affect Only Training, or Inference Too?

Both can be affected. Training involves weights, gradients and intermediate results, while inference involves weights and cached data. Bottlenecks vary with the model, batch size and hardware configuration at each stage.