Universal Engineering Reference

Coherent Optical Transmission
Physics, Formulas & Scaling Laws
400G through 1.6T+

A first-principles engineering reference extracting every physics relationship, scaling law, and design rule governing single-carrier coherent transmission with PCS-QAM modulation. Grounded in Shannon theory and fiber physics — valid across all coherent DSP implementations, current and future generations.

Contents

01

Shannon Limit & Spectral Efficiency Gap

Physics

How Close Can Coherent DSP Get to Shannon?

Shannon's theorem defines the maximum spectral efficiency at any SNR. Measuring the gap to this limit across line rates reveals the maturity of PCS + SD-FEC implementation and predicts future headroom.

// Shannon capacity (dual-polarization AWGN):
SE_Shannon = 2 x log2(1 + SNR_lin)    [b/s/Hz]

// OSNR (0.1nm) to SNR conversion:
SNR_lin = OSNR_lin x (B_ref / Rs)    [B_ref = 12.5 GHz = 0.1nm @ 1550nm]
Rateb/sOSNR dBSE ActualSE ShannonGapGen.
400G2.00~163.610.4~4.6 dBGen1-2
600G2.72~194.412.4~4.5 dBGen1-2
800G3.62~225.814.3~3.9 dBGen2
1.0T4.53~267.317.0~3.7 dBGen2
1.2T5.43~308.719.7~3.5 dBGen2
1.6T5.7-6.0~32-349.1-9.6~21~3.3 dBGen3 (3nm)
Key Deductions
  • Shannon gap decreases from ~5.0 dB to ~3.3 dB at higher rates — PCS + SD-FEC becomes more efficient at higher modulation orders
  • Modern coherent DSPs achieve 35-44% of Shannon SE, improving with modulation order
  • Gap breakdown: FEC OH (~1.0 dB) + implementation (~1.0-1.5 dB) + aging margin (~0.5 dB) + coding gap (~1.0-1.5 dB)
  • At 1.6T with ~200 GBd, the gap is projected to reach ~3.3 dB with next-gen SD-FEC and 3nm DSP
02

OSNR Scaling Laws — Fundamental Constants

Formula

OSNR Requirement vs. Bits/Symbol

Across all PCS-capable coherent DSPs, required OSNR follows a near-linear dB relationship with bits/symbol. The slope and intercept depend on FEC type and implementation quality.

// Universal empirical fit (PCS-QAM, 15% SD-FEC):
OSNR_req = A x b/s + B    [dB]

// Typical values (state-of-art, as of 2025):
A = 4.0 - 4.2 dB per bit/sym
B = 7.0 - 8.0 dB (intercept)

// Next-gen (3nm DSP, improved FEC):
A = 3.8 - 4.0    (slope improves ~0.2 dB)
B = 6.5 - 7.5    (floor drops with better Rx)
A = OSNR slope (dB per bit/symbol) — physics-limited
B = OSNR floor — implementation-limited (improves with DSP node)
Relationship

OSNR Penalty per 100G Step — Accelerates

StepdB/100GCharacter
400G → 500G~1.6Linear zone
600G → 700G~1.4Linear zone
800G → 900G~2.0Acceleration begins
1.0T → 1.1T~1.8High penalty zone
1.1T → 1.2T~2.6PCS gain exhaustion
1.2T → 1.6T~1.0**Only with 200 GBd
1.6T projection: Moving from 1.2T to 1.6T at 140 GBd would need ~6.5 b/s (beyond 64QAM). But at 200 GBd, 1.6T needs only ~5.0 b/s (32QAM territory) — similar OSNR to today's 1.0T. Higher baud rate is the primary path to 1.6T.
Behavior

Same Rate, Different Baud Rate = Different OSNR (up to 5+ dB range)

When a coherent DSP achieves the same line rate at multiple baud rates, lower baud means higher modulation and worse OSNR. This is a universal trade-off independent of DSP implementation.

// Example: 800G at multiple baud rates (PCS-QAM, 15% FEC)
800G @  98 GBd → ~5.1 b/s → OSNR ~ 27 dB  (32QAM region)
800G @ 128 GBd → ~3.9 b/s → OSNR ~ 23 dB  (16QAM)
800G @ 140 GBd → ~3.6 b/s → OSNR ~ 22 dB  (16QAM)

// Future: 800G @ 200 GBd → ~2.5 b/s → OSNR ~ 18 dB  (QPSK!)

// Universal rule: each 10 GBd increase saves ~1.0-1.3 dB OSNR
// This holds across all coherent DSP generations
Design Rules
  • Long-haul (OSNR-limited): Always use the highest available baud rate — minimize modulation order, maximize reach
  • Metro (bandwidth-limited): Lower baud rate + tighter spacing = more channels per fiber
  • 5+ dB OSNR flexibility means the difference between 1500 km and 600 km on the same plant
  • 200 GBd generation will make 800G long-haul QPSK-class — transformative for submarine/ULH
03

PCS Shaping Gain — Physics & Measured Values

PCS

Maxwell-Boltzmann Distribution & Fractional Bits/Symbol

PCS replaces uniform constellation probability with a Maxwell-Boltzmann distribution. Inner (low-amplitude) points get higher probability, reducing average signal power while maintaining entropy. This creates any b/s value between integer QAM steps.

// Maxwell-Boltzmann distribution:
P(xi) = exp(-v x |xi|^2) / Z

// v = shaping parameter (optimized per SNR)
// Z = normalization (partition function)
// Entropy: H = -sum(P(xi) x log2(P(xi)))

// Key: v=0 → uniform QAM, v→inf → only center point
// Optimal v maximizes mutual information for given SNR
PCS

Measured PCS Gain: 0.8 to 1.5+ dB

PCS gain increases with modulation order because higher-order constellations have more high-amplitude points to de-prioritize. This is fundamental — true for all PCS implementations.

BaseUniform b/sPCS RangeGain
DP-QPSK2.02.0-2.3~0.8 dB
DP-8QAM3.02.7-3.2~1.0 dB
DP-16QAM4.03.6-4.1~1.1 dB
DP-32QAM5.04.5-5.0~1.3 dB
DP-64QAM6.05.4-5.8~1.5 dB
DP-256QAM8.07.0-7.5~1.8 dB
PCS approaches the theoretical Gaussian-input limit as M grows. At 64QAM, PCS closes ~1.5 dB of the ~3.5 dB gap to Shannon. Future 256QAM systems could gain ~1.8 dB.
04

Bits/Symbol vs. OSNR — The Exponential Penalty

Physics

Each Additional Bit/Symbol Doubles Constellation Points

Adding 1 bit/symbol doubles the number of constellation points. Minimum Euclidean distance between points shrinks, requiring exponentially more SNR. This is the fundamental physics that limits all coherent systems.

// Minimum distance for square M-QAM:
d_min = 2 x sqrt(3 x Es / (M-1))

// Required SNR (for target BER):
SNR_req ~ (2^b/s - 1) / 3     [linear, theoretical]

// In dB (PCS-QAM practical, 15% SD-FEC):
OSNR_req ~ 4.1 x b/s + 7.5    [dB, current gen]

// Prediction for any b/s (2.0 to 8.0):
b/s=2.0 → ~16 dB   b/s=4.0 → ~24 dB   b/s=6.0 → ~32 dB
b/s=3.0 → ~20 dB   b/s=5.0 → ~28 dB   b/s=7.0 → ~36 dB

OSNR vs. Bits/Symbol (400G to 1.6T+)

OSNR Penalty Acceleration per 100G

05

Chromatic Dispersion Tolerance

Physics

CD Tolerance Scales as 1/Rs^2

CD-induced pulse broadening is proportional to signal bandwidth. For a given penalty, the tolerable accumulated CD is inversely proportional to the square of the symbol rate. This is physics — not DSP-dependent.

CD_tol ~ K / Rs^2    [ps/nm]

// K depends on modulation order & penalty level
// Higher modulation = lower K (denser constellation
// is more sensitive to phase rotation from CD)

// DSP CD equalizer is limited by FIR tap count
// Max CD = function of DSP complexity budget
Relationship

CD Tolerance Across Generations

RateTypical RsCD @ 0.5dBMax km*
400G68 GBd~280,00016,471
600G88 GBd~180,00010,588
800G140 GBd~80,0004,706
1.0T140 GBd~20,0001,176
1.2T140 GBd~7,500441
1.6T200 GBd~3,500206
*G.652 fiber, D=17 ps/nm/km. At 200 GBd / 1.6T, the DSP CD equalizer covers only ~200 km of uncompensated dispersion. This is a DSP complexity constraint — future DSPs with larger FIR taps can extend this.
06

Baud Rate, Spectral Width & Channel Planning

Formula

BW = Rs x (1 + Roll-off)

BW_20dB = Rs x (1 + alpha)    [GHz]

// Typical alpha = 0.1 (RRC pulse shaping)
 68 GBd → 74.8 GHz  → 75 GHz slot
100 GBd → 110 GHz   → 112.5 GHz slot
140 GBd → 154 GHz   → 150 GHz slot (exceeds!)
200 GBd → 220 GHz   → 225 GHz slot (future)
Design Rule

Spectral Occupancy Ratio — Critical at High Baud

ConfigRsBWCSOcc.C-Band Ch
400G/75GHz6874.87599.7%65
800G/150GHz140154150102.7%32
1.6T/225GHz20022022597.8%21
At 140 GBd / 150 GHz, spectral occupancy exceeds 100%. ROADM filtering penalties (0.5-1.5 dB per cascade of >5 nodes) must be included. At 200 GBd, new slot sizes (225 GHz) or S-band extension become essential.
07

Reach-Capacity Fundamental Trade-off

Trade-off

3 dB OSNR Increase = Half the Reach (Linear Regime)

// EDFA chain OSNR (simplified):
OSNR_rx = 58 - NF - Span_Loss - 10log10(N) + Pch    [dB]

// Reach estimates (80km EDFA spans, G.652, NF=5.5dB, Pch=0dBm):
400G (~16 dB) → ~4,000 km (50 spans)
600G (~19 dB) → ~2,000 km (25 spans)
800G (~22 dB) → ~900 km  (11 spans)
1.0T (~26 dB) → ~350 km  (4 spans)
1.2T (~30 dB) → ~80 km   (1 span)
1.6T @140GBd (~34 dB) → ~20 km   (DCI back-to-back)
1.6T @200GBd (~28 dB) → ~200 km  (metro viable!)

Capacity vs. Reach — Current & Future Generations

Scaling Laws
  • 3 dB OSNR increase ~ halving of reach (linear ASE-limited regime)
  • Sweet spot: 600G-800G — 1.5-2x capacity of 400G with manageable reach reduction
  • 200 GBd transforms 1.6T from back-to-back only (~20 km at 140 GBd) to metro-viable (~200 km)
  • Every baud rate doubling (70→140→200+) effectively shifts the reach curve one full generation forward
08

GOSNR — ASE, NLI & Transceiver Noise

Physics

Three Independent Noise Sources Add Inversely

1/GOSNR = 1/OSNR_ASE + 1/OSNR_NLI + 1/OSNR_TX

// OSNR_ASE: from amplifier chain (span-count dependent)
// OSNR_NLI: Kerr nonlinear interference (power dependent)
// OSNR_TX: transceiver implementation (TX IB-OSNR)

// Nonlinear OSNR (GN model):
P_NLI ~ eta x Pch^3    [per span, Gaussian noise model]

// Optimal launch power (derivative = 0):
P_opt = (P_ASE / (2 x eta))^(1/3)
Design Rule

TX IB-OSNR Sets a Hard System Ceiling

Typical coherent modules: TX IB-OSNR = 33-37 dB (C-band), 1-2 dB worse in L-band. This floor directly limits achievable GOSNR for high-rate modes.

// TX noise impact at different OSNR requirements:
Rate needing 16 dB: penalty < 0.1 dB  (irrelevant)
Rate needing 22 dB: penalty ~0.3 dB   (small)
Rate needing 30 dB: penalty ~1.5-2 dB (CRITICAL)
Rate needing 34 dB: penalty ~3+ dB    (system-limiting)

// This is why >1.2T at 140 GBd is DCI-only
// At 200 GBd, 1.6T needs ~28 dB — manageable
09

FEC Overhead & Net Coding Gain

Formula

SD-FEC Overhead, Throughput & NCG

// Standard FEC overhead options:
15% OH: R_FEC = 1/1.15 = 0.8696  NCG ~ 11.5 dB  Pre-FEC BER ~ 1.5e-2
20% OH: R_FEC = 1/1.20 = 0.8333  NCG ~ 12.0 dB  Pre-FEC BER ~ 2.4e-2
25% OH: R_FEC = 1/1.25 = 0.8000  NCG ~ 12.5 dB  Pre-FEC BER ~ 3.0e-2

// Net data rate:
Net_Rate = 2 x Rs x b/s x R_FEC

// Examples:
 400G: 2 x  68 x 2.00 x 0.87 = ~237G gross, 400G incl framing
 800G: 2 x 140 x 3.62 x 0.87 = ~881G gross
1200G: 2 x 140 x 5.43 x 0.87 = ~1321G gross
1600G: 2 x 200 x 5.00 x 0.87 = ~1739G gross
10

Rx Power Budget & Dynamic Range

Behavior

Operating Power Window Shrinks with Rate — Universal Pattern

// Typical coherent Rx dynamic range (all vendors):
400G:  Rx Low ~ -14 dBm, Rx High ~ +3 dBm → 17 dB range
600G:  Rx Low ~ -11 dBm, Rx High ~ +3 dBm → 14 dB range
800G:  Rx Low ~ -7 dBm,  Rx High ~ +3 dBm → 10 dB range
1.0T:  Rx Low ~ -4 dBm,  Rx High ~ +3 dBm → 7 dB range
1.2T:  Rx Low ~ -3 dBm,  Rx High ~ +3 dBm → 6 dB range
1.6T:  Rx Low ~ -1 dBm,  Rx High ~ +3 dBm → 4 dB range (est)

// Physics: Higher modulation needs higher OSNR_rx, so
// minimum Rx power rises while max stays ADC-limited
Implications
  • At 1.2T+ with 4-6 dB dynamic range, received power must be precisely controlled
  • Per-channel ROADM power equalization becomes mandatory, not optional
  • Protection switching and channel add/drop transient management must converge within 100 ms
  • At 1.6T, the 4 dB window means essentially zero tolerance for power excursions — closed-loop control is essential
11

Multi-Rate Flexibility & Design Space

Design Rule

Flexibility Decreases with Line Rate — Universal Pattern

// Baud rate options per line rate (typical flex-rate coherent):
 400G: 5-7 baud rates (68-140 GBd)   → Maximum flexibility
 600G: 5-7 baud rates (78-140 GBd)
 800G: 3-5 baud rates (98-140 GBd)
1.0T:  2-3 baud rates (118-140 GBd)  → Limited
1.2T:  1 baud rate   (140 GBd only)  → No choice
1.6T:  1 baud rate   (200 GBd only)  → No choice (next gen)

// Pattern: highest rates are ALWAYS single-baud-rate
// Only lower rates offer baud/modulation trade-off
// This is fundamental: max rate = max baud x max b/s
ScenarioOptimal StrategyReasoning
Long-haul 400GLowest baud, 75 GHz slotMax channels, QPSK-class OSNR
Long-haul 800GHighest baud availableLowest modulation → best reach
Metro 800GMid baud, tighter spacingMore channels, OSNR margin available
DCI 1.2T-1.6TMax baud (only option)Short link, clean path required
Submarine 400GMin baud, flex-gridMaximum reach, minimum OSNR
12

Fiber Capacity — C, L & S Band Planning

Formula

Band Capacity Calculation

Fiber_Cap = N_ch x Rate_per_ch
N_ch = floor(BW_band / CS)

// Band widths (ITU-T standard):
C-band: ~4.8 THz (1530-1565nm)
L-band: ~4.8 THz (1570-1610nm)
S-band: ~4.0 THz (1460-1530nm) [emerging]
C+L:    ~9.6 THz
C+L+S:  ~13.6 THz  [future]
Trade-off

Capacity Scenarios Across Generations

ConfigC-BandC+LC+L+S
400G@75GHz26 Tb/s5273
800G@150GHz25.6 Tb/s51.272
1.2T@150GHz38.4 Tb/s76.8108
1.6T@225GHz34 Tb/s6895
Key insight: 1.6T@225 GHz gives fewer channels but higher per-channel rate. Total fiber capacity may not increase vs 1.2T@150 GHz unless S-band is added. The real gain is cost per bit — fewer transceivers for same capacity.
13

Future Projections: 1.6T, 200 GBd & Beyond

Future

The Path to 1.6T — Two Routes

// Route 1: 1.6T at current baud rate (140 GBd)
b/s needed = 1600 / (2 x 140 x 0.87) = 6.57 b/s
// This exceeds 64QAM (6.0) → needs 128QAM or 256QAM base
// OSNR ~ 34-36 dB → back-to-back only, not deployable

// Route 2: 1.6T at next-gen baud rate (200 GBd)
b/s needed = 1600 / (2 x 200 x 0.87) = 4.60 b/s
// This is 32QAM territory → OSNR ~ 26-28 dB
// Metro-viable (~200 km), regional with Raman

// Route 3: 1.6T at future baud rate (260 GBd, ~2028+)
b/s needed = 1600 / (2 x 260 x 0.87) = 3.54 b/s
// 16QAM territory → OSNR ~ 22 dB → long-haul viable!
ParameterCurrent Gen (7nm)Next Gen (3nm)Future (~2028+)
Max Baud Rate~140 GBd~200 GBd~260 GBd
Max Single-Carrier1.2T1.6T2.0T+
DAC/ADC ENOB6-7 bits7-8 bits8+ bits
FEC NCG~11.5 dB~12.0 dB~12.5 dB
PCS Efficiency~1.5 dB gain~1.7 dB~1.8 dB
TX IB-OSNR~35 dB~37 dB~39 dB
Power (typ)~60-80W~40-60W~35-50W
Key Future Insights
  • Baud rate increase is the primary path to 1.6T — not higher modulation order
  • At 200 GBd, 800G becomes a QPSK-class signal — viable for submarine/ULH
  • At 200 GBd, 1.6T needs only 32QAM-class — viable for metro/regional
  • S-band amplification adds ~30% more fiber capacity beyond C+L
  • The trend is clear: each baud rate doubling shifts the entire capacity-reach curve one generation forward
14

Complete Formula Reference Sheet

Master Reference

All Formulas — Vendor-Neutral, Scalable 400G to 2T+

===== CAPACITY & SPECTRAL EFFICIENCY =====
SE_gross  = 2 x b/s                                [b/s/Hz, dual-pol]
SE_net    = Line_Rate / Ch_Spacing                  [b/s/Hz]
Capacity  = 2 x Rs x b/s x R_FEC                   [Gb/s net per carrier]
Fiber_Cap = N_ch x Capacity                         [Gb/s per fiber]
N_ch      = floor(BW_band / CS)                     [channels]

===== OSNR & SNR =====
OSNR_req  ~ A x b/s + B                            [dB, A~4.0-4.2, B~7.0-8.0]
SNR       = OSNR x (B_ref / Rs)                     [linear, B_ref=12.5GHz]
1/GOSNR   = 1/OSNR_ASE + 1/OSNR_NLI + 1/OSNR_TX   [linear]
SE_Shannon= 2 x log2(1 + SNR_lin)                  [b/s/Hz, theoretical max]

===== REACH & LINK BUDGET =====
OSNR_rx   = 58 - NF - Span_Loss - 10log10(N) + Pch [dB, per 0.1nm]
Span_Loss = alpha x L_span + splices + connectors   [dB]
Margin    = OSNR_rx - OSNR_req                      [dB, target >= 3]

===== BANDWIDTH & SPECTRAL =====
BW_20dB   = Rs x (1 + alpha)                       [GHz, alpha~0.1 RRC]
Occ_ratio = BW_20dB / CS                            [target < 1.0]

===== IMPAIRMENT TOLERANCES =====
CD_tol    ~ K / Rs^2                                [ps/nm, K=f(mod,penalty)]
PMD_tol   ~ 20-30 ps                               [typical coherent DSP]
PDL_tol   ~ 1-3 dB                                  [function of modulation]

===== PCS & MODULATION =====
P(xi) = exp(-v|xi|^2) / Z                          [Maxwell-Boltzmann]
H     = -sum(P(xi) log2 P(xi))                     [entropy, bits/symbol]
Line_Rate = 2 x Rs x b/s                           [gross, before FEC]
Net_Rate  = Line_Rate x R_FEC                       [client payload]

===== FIBER PHYSICS =====
CD    = D x L                                       [ps/nm; D~17 G.652]
PMD   = PMD_coeff x sqrt(L)                         [ps; coeff~0.04-0.1]
gamma = 2pi/lambda x n2/Aeff                        [rad/W/km; ~1.3 SMF]
P_NLI ~ eta x Pch^3                                [GN model, per span]
P_opt = (P_ASE/(2 x eta))^(1/3)                    [optimal launch power]

===== GENERATION CONSTANTS =====
Current (7nm):  Rs_max~140 GBd, Max~1.2T, IB-OSNR~35dB
Next (3nm):     Rs_max~200 GBd, Max~1.6T, IB-OSNR~37dB
Future:         Rs_max~260 GBd, Max~2.0T+, IB-OSNR~39dB

===== BAND DEFINITIONS =====
C-band: 1530-1565nm (~4.8 THz)  Grid: 6.25 GHz (ITU G.694.1)
L-band: 1570-1610nm (~4.8 THz)  Slots: 75/100/112.5/150/225 GHz
S-band: 1460-1530nm (~4.0 THz)  Total C+L+S: ~13.6 THz

Developed by MapYourTech Team

For educational purposes in Optical Networking Communications Technologies