Introduction

Forward error correction has almost nothing left to give. A decade of soft-decision FEC work took coherent links from hard-decision Reed-Solomon to modern low-density parity-check and staircase codes with net coding gains in the 11-13 dB range, and that harvest is largely spent: the practical codes now sit within a fraction of a decibel of the coding bound, so each new generation returns tenths of a decibel rather than whole decibels. The gap that remains between a real coherent transceiver and the Shannon capacity of its channel is no longer mostly a coding gap. It is a signalling gap, and the tool that closes it is constellation shaping.

Uniform quadrature amplitude modulation (QAM) spends the same energy budget on a rarely useful outer symbol as it does on a frequently useful inner one. Information theory has known since the 1980s that a Gaussian-like input, not a uniform one, maximises the mutual information of an additive white Gaussian noise (AWGN) channel, and that the penalty for using uniform QAM instead approaches 1.53 dB at high spectral efficiency. That figure is a theoretical limit, and probabilistic constellation shaping (PCS) is the engineering that reaches for it by transmitting inner constellation points more often than outer ones while leaving the constellation grid untouched.

PCS is not a laboratory curiosity. It has been carried in commercial coherent digital signal processors (DSPs) since Nokia shipped it in the PSE-3 in 2018 (vendor timeline), and at OFC 2026 the industry marked roughly ten years of the technique with a dedicated session on where it goes next (industry event). Every serious terrestrial and submarine coherent engine now shipping applies some form of shaping, and the OpenROADM Multi-Source Agreement has defined an interoperable PCS mode so that shaped 800G links can run between equipment from different suppliers. If you build, plan, or operate coherent networks, PCS is now part of the working vocabulary alongside OSNR, generalised OSNR (GOSNR), and baud rate.

This article works through PCS end to end: the information-theoretic reason it works, the probabilistic amplitude shaping (PAS) architecture that made it practical, the distribution matching that sits at its core, the mathematics of near-continuous rate adaptation, the way its clean AWGN gain erodes on the nonlinear fibre channel, the engines and standards that ship it in 2026, the gains measured on real link classes, and the operational consequences of being able to dial a wavelength's data rate almost continuously. Where a number appears, its evidence class appears with it, because the difference between a theoretical limit, a measured value, and a vendor claim is exactly the difference between engineering and marketing.

1. The Shannon gap and where shaping gain comes from

Shannon's capacity for a band-limited AWGN channel sets the ceiling every coherent design works against. The capacity-achieving input is a continuous Gaussian, and no discrete constellation with uniform probabilities can reach it. The shortfall has a precise asymptotic value that any optical engineer should carry in their head.

For a fixed constellation cardinality and high spectral efficiency, the distance between the best information rate achievable with uniform M-QAM and the Gaussian capacity tends to a constant. That constant is the ultimate shaping gain.

The physical reading is simple. A uniform QAM source assigns equal probability to inner points near the origin, which sit at low energy, and to corner points, which sit at high energy and are the first to be corrupted as noise grows. Spending scarce launch energy equally across both is wasteful. A Gaussian-like source concentrates probability where energy is low, lowering the mean symbol energy for the same minimum Euclidean distance, which is the same as buying back SNR. The mutual information of a uniform format saturates at log₂M bit/symbol and pulls away from the capacity curve well before it, and that vertical separation at a given SNR is the room shaping has to work in.

Two boundaries keep this honest. First, the 1.53 dB is asymptotic: it is the limit as spectral efficiency grows and the constellation becomes large, so a link running QPSK at 2 bit/symbol sees only a small fraction of it, while one running shaped 64-QAM at 5-6 bit/symbol sees most of it. Second, the AWGN gain is the clean-channel gain; the fibre is not an AWGN channel once launch power rises, and a later section shows how much of the 1.53 dB survives. For the geometry underneath all of this — symbol energy, minimum distance, and how they set the OSNR threshold — the constellation-diagram fundamentals are the right companion read, and the broader Shannon-limit treatment for fibre puts the number in system context.

2. Probabilistic versus geometric shaping

There are two ways to make a discrete constellation look Gaussian. Geometric shaping moves the points: it keeps every symbol equally likely but places them at non-uniform locations, denser near the origin, as in amplitude-and-phase-shift keying (APSK), iterative polar modulation, or rate-optimised constellations. Probabilistic shaping keeps the points on the regular QAM grid and varies how often each is used. Both approach the Gaussian input; they differ sharply in where they put the implementation cost.

Geometric shaping adds no redundancy, which is attractive, but unequal point spacing raises the effective-number-of-bits (ENOB) demand on the digital-to-analog converter (DAC) and complicates pairing with an off-the-shelf FEC code, because the bit-to-symbol mapping is no longer a clean memoryless Gray map. Probabilistic shaping leaves the grid, the Gray labelling, and the equaliser untouched and instead moves the work into a distribution matcher at the transmitter and its inverse at the receiver. For a fixed number of points, probabilistic shaping also tends to deliver more gain than circular geometric shaping: the shaping literature reports probabilistically shaped 64-QAM roughly 0.5 dB ahead of 64-APSK at matched cardinality (measured, AWGN simulation), while a ring constellation buys about 0.57 dB over uniform 64-QAM at 4.1 bit/symbol (measured, AWGN simulation).

There is a second reason the optical community leaned toward probabilistic shaping, and it lives in the fibre nonlinearity. The higher-order moments of a constellation — its kurtosis and sixth-order moment — drive the nonlinear interference (NLI) generated along the span, so a shaped constellation changes not only its linear shaping gain but its nonlinear tolerance. That coupling cuts both ways and is the subject of Section 6; the point here is that a probabilistic shaper can tune the distribution to trade linear gain against nonlinear penalty in a way a fixed geometry cannot. For a wider map of the levers that raise spectral efficiency, shaping is one entry among Nyquist pulse shaping, polarisation multiplexing, and dense channel spacing.

Sanjay Yadav

Optical Communications & Network Automation Expert | Author of 3 Books for Optical Engineers | Founder, MapYourTech

Optical networking engineer with nearly two decades of experience across DWDM, OTN, coherent optics, submarine systems, and cloud infrastructure. Founder of MapYourTech. Read full bio →

Follow on LinkedIn

Related Articles on MapYourTech

Free · Jul 11, 2026

Connector Types and Their Loss Budgets: SC, LC, MPO

Automation · Jul 11, 2026

Cooled vs Uncooled Lasers: The Pluggable Power Trade-off

Automation · Jul 11, 2026

Automation Blast Radius: Scoping What a Bad Intent Can Touch

Analysis · Jul 11, 2026

ASE Channel Loading: Running a Line System Full From Day One