Skip to main content
Celebrating 1,000+ LinkedIn posts: a free 3-hour Premium Pro pass for every optical professional. Start your pass
Generic selectors
Exact matches only
Search in title
Search in content
Post Type Selectors
Articles
lp_course
lp_lesson
Back
HomePremiumUnderstanding Probabilistic Constellation Shaping: A Comprehensive Guide for Optical Engineers
Last Updated: August 31, 2026
42 min read
199
Probabilistic Constellation Shaping in Coherent Optical Transmission
MapYourTech | InDepth Series

Probabilistic Constellation Shaping in Coherent Optical Transmission

The Maxwell-Boltzmann distribution, the probabilistic amplitude shaping architecture, the distribution matcher families, the standardized 800G shaped modes, and the published long-haul and submarine results — with the entropy, shaping gain and OSNR arithmetic carried through to numbers.

Coherent Transmission

Higher-order modulation adds capacity and subtracts reach.

1. Introduction

A coherent transponder provisioned for 800 Gb/s in a 150 GHz slot has one degree of freedom left once the symbol rate, the constellation and the FEC code are fixed: how often each constellation point is transmitted. Uniform quadrature amplitude modulation spends that degree of freedom on nothing. Every point is equiprobable, the four outermost points of a 64QAM grid are sent as often as the four points nearest the origin, and the average symbol energy is higher than the delivered information rate requires. That surplus energy is charged to the receiver as additional required signal-to-noise ratio, and to the line system as launch power and nonlinear interference.

Probabilistic constellation shaping (PCS) removes that waste by transmitting inner points more often than outer ones. The constellation grid does not move; only the probability assigned to each point changes. Because the average energy falls while the grid spacing stays the same, the same information rate is delivered at lower signal-to-noise ratio, and the achievable rate of the shaped constellation tracks the Shannon bound far more closely than the uniform one does. The bound on that improvement is exactly 1.53 dB, the ratio πe/6 expressed in decibels (theoretical limit), and practical shaped modes in deployed coherent systems recover roughly half to three quarters of it.

The second consequence matters as much as the first. Entropy is a continuous quantity, so a shaped transmitter delivers fractional bits per symbol (3.61, 4.40, 5.35) where a uniform format offers only the integer steps 2, 3, 4, 5 and 6. A route that has 1.4 dB more optical signal-to-noise ratio than 16QAM needs but 5 dB less than 64QAM needs is stranded with uniform signalling; it runs 16QAM and leaves the margin unused. With shaping it runs at the entropy the margin supports. That is why shaping sits alongside baud rate, modulation order and FEC as one of the four levers a modem generation can pull.

This article covers the shaped distribution itself, the transmitter and receiver architecture that produces and undoes it, the distribution matcher families that differ in rate loss and complexity, the behaviour of shaped signals on the nonlinear fiber channel, the interoperable shaped modes now specified for 800G coherent interfaces, and the published results from long-haul and submarine plant. Vendor-proprietary shaping schemes are named only where a public specification describes them.

2. Shaped Constellation Definition and Component Terms

A probabilistically shaped constellation is a fixed, uniformly spaced grid of M quadrature amplitude modulation points transmitted with unequal probabilities, chosen so that low-energy inner points occur more often than high-energy outer points. The information rate such a constellation carries is the entropy of that probability distribution, expressed in bits per symbol per polarisation, and it is always lower than log2 M.

Anatomy of a probabilistically shaped 64QAM constellation Left panel shows the 64QAM grid with marker area proportional to symbol probability under a Maxwell-Boltzmann distribution at entropy 5.00 bits per symbol per polarisation. Right panel shows the one-dimensional amplitude marginal. The lower panel states the entropy and net-rate relationships. Shaped 64QAM Constellation, Entropy 5.00 bits/symbol/pol marker area proportional to symbol probability I Q amplitude shells (dashed) group points of equal energy One-Dimensional Amplitude Marginal -7 0.012 -5 0.058 -3 0.161 -1 0.269 +1 0.269 +3 0.161 +5 0.058 +7 0.012 PAM amplitude level on one quadrature (unit-spacing grid) Constellation Order, Entropy and Rate Constellation order M 64 points — the grid the DSP addresses Entropy H(X) 5.00 bits/symbol/pol — the information carried Uniform bits log₂M 6.00 bits/symbol/pol — the ceiling, not the rate Shaping parameter λ 0.0641 on this unit-spacing grid Defining Relationships H(X) = − Σ p(xᵢ) log₂ p(xᵢ) [bits per symbol per polarisation] R_net = 2 · Rₛ · H(X) / (1 + OH) [bits per second, dual polarisation] Rₛ = symbol rate in GBd · OH = FEC overhead as a fraction · factor 2 = two orthogonal polarisations Uniform signalling is the special case p(xᵢ) = 1/M, for which H(X) = log₂ M exactly.
Fig. 1: Anatomy of a probabilistically shaped 64QAM constellation at 5.00 bits per symbol per polarisation. Marker area is proportional to symbol probability; the dashed rings group points of equal energy. The right-hand panels give the one-dimensional amplitude marginal and the three quantities that must be kept apart.

2.1 Distinctions From the Adjacent Quantities

Three quantities are conflated more often than any others in shaping discussions, and Fig. 1 separates them on purpose. Constellation order M is the number of points the digital-to-analogue converter and the demapper must address; it sets hardware complexity and nothing else. Entropy H(X) is the information the symbol stream carries, and it is the number that enters every rate and capacity calculation. The uniform bit count log2 M is the ceiling that entropy approaches when shaping is switched off, not a rate the shaped mode delivers. A mode described as PCS-64QAM at 5.00 bits per symbol per polarisation addresses 64 points and carries five bits, and quoting it as a 64QAM mode overstates its rate by 20%.

Premium Article — Free 12% Preview

Read the Full Analysis with Premium

The remaining 88% of this article — the design numbers, trade-offs and field guidance — is part of MapYourTech Premium, along with the full premium library, courses and professional tools.

1038+Technical Articles
82+Professional Courses
19+Engineering Tools
400K+Professionals
View Membership Plans Already a member? Sign In
Instant access Cancel anytime 48-hour trial available
Share:

You May Also Like

What multi-vendor pluggable testing reveals about required optical signal-to-noise ratio (OSNR), margin allocation and …
  • Free
  • September 4, 2026
What the ratified coherent line-interface specifications fix, what …
  • Premium
  • September 4, 2026
56 min read 25 0 Like ROADM Add/Drop Architecture: Colorless, Directionless, Contentionless ROADM Add/Drop Architecture Choices: Colorless, Directionless, Contentionless Design...
  • Premium
  • September 4, 2026

Course Title

Course description and key highlights

Course Content

Course Details