The Digital Signal Processor (DSP) represents the intellectual core of modern coherent optical transmission systems. Since the commercial introduction of coherent detection around 2008, DSP technology has evolved from enabling basic 40G transmission to becoming the central determinant of system performance at 400G, 800G, and now 1.6T wavelengths. The DSP performs critical functions including chromatic dispersion compensation, polarization mode dispersion mitigation, nonlinear effect equalization, forward error correction (FEC), carrier recovery, and adaptive modulation.

The architectural choice between in-house (proprietary) and merchant (third-party) DSP development carries profound implications across technical, economic, and strategic dimensions. In-house development enables tight co-optimization between the DSP, optical front-end, and system software, delivering maximum performance for demanding applications such as submarine and ultra-long-haul terrestrial networks. Merchant DSP solutions provide standardized building blocks that enable faster market entry, multi-vendor interoperability through specifications like OIF 400ZR and 800ZR, and volume economics that benefit from aggregated demand.

The Strategic Bifurcation

The coherent optical networking industry has split decisively along two development paths. Vertically-integrated vendors with in-house DSPs continue to dominate high-performance applications, achieving spectral efficiencies within 1-2 dB of Shannon limits. Meanwhile, merchant DSPs have democratized 400G-800G pluggable coherent for data center interconnect (DCI) with compound annual growth rates exceeding 100% since 2022.

The evolution of coherent DSP technology traces a remarkable trajectory from the first commercial deployments in 2008 to today's 1.6T wavelengths. Each generation has delivered approximately 2× capacity at similar power levels by combining silicon node advances with algorithmic innovations in forward error correction, probabilistic constellation shaping, and nonlinear compensation.

Silicon Node Progression

Silicon process node advancement has followed a predictable cadence that directly enables coherent DSP capability improvements. The 28nm CMOS node enabled 100G coherent circa 2012. The 16nm generation pushed systems to 200-400G by 2016, while 7nm unlocked 600-800G starting in 2019. Current 5nm technology enables 800G-1.2T commercial deployments. Most significantly, Ciena made the strategic decision to skip 5nm entirely, jumping directly from 7nm to 3nm for WaveLogic 6 Extreme, delivering 200 GBaud symbol rates and 1.6 Tb/s single-carrier wavelengths by late 2024.

The Baud Rate Wars

Symbol rate (baud rate) has emerged as the primary competitive metric defining coherent DSP generations. Increasing baud rate allows higher data transmission without increasing modulation complexity, which is critical because higher-order formats require exponentially higher OSNR.

Ciena's 200 GBaud capability positions them a full generation ahead of competitors. The gap between 200 GBaud and 130 GBaud (54% higher) to 148 GBaud (35% higher) translates directly into increased spectral efficiency or extended reach at equivalent data rates.

Understanding the fundamental differences between in-house and merchant DSP architectures requires examining the core technical concepts that define coherent optical transmission. This section provides the foundational knowledge essential for evaluating DSP solutions across different network applications.

In-House vs Merchant: Architectural Philosophy

In-house DSPs are custom ASICs developed by system vendors for exclusive use in their own transponders and transceiver modules. The key advantage is tight co-optimization between the DSP and the accompanying optics. Merchant DSPs follow a standardization path, implementing industry-defined interface standards for multi-vendor interoperability.

Forward Error Correction Fundamentals

Forward error correction provides the coding gain essential for practical coherent transmission. FEC represents one of the most significant differentiators between in-house and merchant architectures.

Key Architectural Differences

The technical architecture of coherent DSPs encompasses multiple functional blocks working in concert to compensate fiber impairments and maximize transmission performance. Understanding these architectural elements is essential for evaluating the trade-offs between in-house and merchant solutions.

DSP Functional Block Diagram

A modern coherent DSP contains several key functional blocks that process the received signal through a complex pipeline. The ADC/DAC subsystem converts between analog optical signals and digital processing domains. Chromatic dispersion (CD) compensation uses frequency-domain equalization to remove accumulated dispersion. Adaptive equalizers track and compensate for polarization mode dispersion (PMD) and other time-varying impairments. Carrier recovery extracts the carrier phase from the modulated signal. Finally, FEC decoding corrects residual errors to achieve the target bit error rate.

In-House DSP: Ciena WaveLogic 6 Extreme Architecture

Ciena's WaveLogic 6 Extreme represents the state-of-the-art in in-house DSP development. Built on a 3nm CMOS process, it achieves 200 GBaud operation with single-carrier capacity of 1.6 Tb/s for metro applications and 800 Gb/s for ultra-long-haul reaches exceeding 10,000 km. Key architectural features include:

• 224G SerDes technology integrated directly into the coherent ASIC
• Ultra-high-bandwidth ADCs/DACs sampling at rates significantly higher than Nyquist
• Multi-dimensional constellation shaping for maximum spectral efficiency
• 50% power-per-bit reduction compared to previous WL5 generation

Merchant DSP: Acacia CIM-8 (Jannu DSP) Architecture

Acacia's Coherent Interconnect Module 8 (CIM-8), powered by the 8th generation Jannu DSP fabricated in 5nm CMOS, represents the industry's first single-carrier 1.2 Tb/s pluggable module. Operating at 140 GBaud, it delivers 1.2 Tb/s for metro/DCI, 800 Gb/s full network coverage across long-haul distances, and 400 Gb/s for ultra-long-haul and submarine transmission. Key architectural innovations include:

• Second Generation 3D-Shaping: Enhanced probabilistic constellation shaping (PCS) algorithms providing continuous modulation, baud rate, and spectral efficiency adjustment with 20% higher spectral efficiency over the previous generation
• 3D Siliconization: Advanced integration technique combining high-speed opto-electronic functions with reduced electrical interconnects for superior signal integrity
• Adaptive Baud Rate: Continuous baud rate adjustment up to 140 GBaud to optimize spectrum utilization across single-span or cascaded ROADM paths
• Advanced SD-FEC: Innovative soft-decision error correction with enhanced nonlinear impairment compensation and 2× faster state-of-polarization (SOP) tracking
• More than 65% power-per-bit savings compared to previous generation, with C and L Band support

Optical Front-End Integration

The interface between the DSP and optical components represents a critical differentiation point. In-house designs enable 3D Siliconization (pioneered by Cisco/Acacia), where high-speed electronics and DSP are mounted directly atop the silicon photonics PIC using vertical copper pillars rather than wire bonds. This minimizes RF path length, reducing signal loss and impedance mismatches. Merchant DSPs typically connect via standardized SerDes interfaces that must accommodate various optical front-ends from different suppliers.

These interactive simulators demonstrate the technical trade-offs between in-house and merchant DSP architectures. Adjust the parameters to explore how different configurations affect system performance.

Deployment Scenario Decision Matrix

Recent Industry Deployments (2024-2025)

Ciena WaveLogic 6 Deployments

Ciena has secured over 20 WaveLogic 6 Extreme customers in Q1 2025 alone. Key deployments include Telstra's 1.6 Tb/s demonstration over 736 km of installed fiber in Australia, and Southern Cross Networks' modeled 1 Tb/s capacity over 13,500 km trans-Pacific submarine routes. WL6 technology enables "800G Anywhere" capability, closing 800 Gb/s links using robust QPSK modulation over transoceanic distances where merchant solutions would require 16QAM with dramatically reduced reach.

Nokia/Infinera Merger Integration

Following the February 2025 merger completion, Nokia-Infinera has combined PSE-6s and ICE7 portfolios. The merged entity targets €200M net synergies by 2027 with 10%+ EPS accretion. Infinera's design wins, valued at hundreds of millions in 800ZR/ZR+ pluggable commitments, complement Nokia's strong carrier relationships. The combined optical networking powerhouse now controls both leading in-house DSP platforms and significant merchant-compatible pluggable volume.

Hyperscaler 800ZR Adoption and Vertical Integration Trends

The "big four" hyperscalers—Google, Microsoft, Meta, and AWS—have adopted distinctly different coherent optical strategies that collectively shape the merchant DSP market. According to Cignal AI, these cloud providers now drive 60% YoY growth in DWDM purchasing (versus 5% for total market in H1 2025), with pluggable coherent accounting for over 70% of coherent bandwidth deployed in 2024.

Microsoft—The 400ZR Pioneer, Waiting for 1600ZR: As the largest 400ZR user globally, Microsoft pioneered the hyperscaler adoption of pluggable coherent optics. Microsoft was a principal backer of the original OIF 400ZR work alongside Google starting in 2016, targeting regional data center interconnects up to 80-120 km. However, Microsoft has announced it will not adopt 800ZR, instead following a 1:4:16 upgrade path (400G→1600G) and waiting for 1600ZR technology expected circa 2027. This strategic patience reflects Microsoft's assessment that changing switching infrastructure for a 2× upgrade is not worth the effort when 4× (1.6T) is only a few years away. Microsoft presented at OFC 2025 on the transition from 400G to 1600G deployment, emphasizing AI/ML-driven bandwidth demands and intelligent network operations.

Google—The Standards Driver: Google was the primary promoter of the OIF 800ZR specification, which kicked off in November 2020. Unlike Microsoft, Google and Meta have adopted more flexible upgrade paths, planning to utilize 800ZR in some network segments and 1600ZR in others based on specific application requirements. Google has defined specific "Long-Haul Coherent Pluggable ZR+" requirements and is actively promoting pluggable firmware upgrade standardization within OIF to enable universal standards across platforms and module vendors. At OFC 2025, Google's Jason Wang participated in OIF panels on coherent optics advancement, emphasizing the importance of backward compatibility for 400G-to-800G upgrades and standardized PCS modes.

Meta—AI Infrastructure at Scale: Meta has invested aggressively in optical infrastructure to support AI training clusters, committing $66-72 billion in 2025 CapEx (up 70% from 2024) with "similarly significant" growth projected for 2026. Meta's "10× Backbone" initiative has transformed their network through IP/optical integration using ZR technology, eliminating standalone transponders and recovering significant space and power. Key optical initiatives include:

• Deployed optics: 2×400G FR4 BASE (3-km) optics supporting 51T platforms across backend/frontend networks
• New variants: 2×400G FR4 LITE (500-m) optimized for majority of intra-datacenter use cases
• AI cluster connectivity: 400G DR4 OSFP-RHS for host-side NIC and 2×400G DR4 for switch-side connections
• Prometheus cluster: 1-gigawatt AI cluster spanning multiple data center buildings requiring distributed training across geographically distributed sites

Meta identifies "advanced optical solutions" as the only viable path to increasing shoreline beyond 3.2T and moving beyond backplane constraints. The company is actively engaging in research on higher performance, more reliable optical interconnects and has been a founding participant in the ESUN (Ethernet for Scale-Up Networking) initiative within OCP at the 2025 Global Summit.

AWS In-House Optical Development: In a significant industry shift, AWS has developed proprietary DWDM transponder technology for both metro and long-haul routes, representing the first significant whitebox optical system deployment by a hyperscaler. Development began in 2020 targeting metro networks, with updated architecture in January 2024 supporting unified data center, metro, and long-haul connections. Key achievements include:

• Metro deployment: Upgraded iteration deployed on metro networks by February 2025
• Long-haul milestone: First 1,500-km connection went live in July 2025
• Performance gains: 73% bandwidth increase and 35% power reduction versus vendor-sourced equipment
• Flexibility: Supports multiple technology generations and any fiber type with flexible bandwidth options
• Strategic rationale: Full software/hardware control enabling security verification of every byte, rapid updates, deep network management integration, and supply chain continuity

AWS has explicitly stated it will not sell its proprietary optical technology externally, focusing instead on "providing the best network for AWS customers" while acknowledging that "existing DWDM vendors do a great job supporting the diverse needs of the market." This positions AWS's in-house development as a strategic capability differentiator rather than a competitive threat to optical equipment vendors.

Hyperscaler Strategy Summary: These divergent approaches illustrate the market segmentation: Microsoft prioritizes standardization and long upgrade cycles; Google drives standards development while maintaining flexibility; Meta invests heavily in AI-optimized optical infrastructure; and AWS pursues vertical integration for maximum control. The OIF 800ZR Implementation Agreement, released October 2024, has already driven multi-vendor interoperability demonstrations with 35+ participating companies at OFC 2025, validating the ecosystem's readiness for 800G DCI deployment at scale.