What is Open Access?

Open Access is an initiative that aims to make scientific research freely available to all. To date our community has made over 100 million downloads. It’s based on principles of collaboration, unobstructed discovery, and, most importantly, scientific progression. As PhD students, we found it difficult to access the research we needed, so we decided to create a new Open Access publisher that levels the playing field for scientists across the world. How? By making research easy to access, and puts the academic needs of the researchers before the business interests of publishers.

Our Authors and Editors

We are a community of more than 103,000 authors and editors from 3,291 institutions spanning 160 countries, including Nobel Prize winners and some of the world’s most-cited researchers. Publishing on IntechOpen allows authors to earn citations and find new collaborators, meaning more people see your work not only from your own field of study, but from other related fields too.

Content Alerts

Brief introduction to this section that descibes Open Access especially from an IntechOpen perspective

How it Works Manage preferences

Contact

Want to get in touch? Contact our London head office or media team here

Careers

Our team is growing all the time, so we’re always on the lookout for smart people who want to help us reshape the world of scientific publishing.

Home > Books > Fiber Optics - Technology and Applications

Open access peer-reviewed chapter

Written By

Zhipei Li, Dong Guo and Ran Gao

Submitted: 30 May 2021 Reviewed: 10 June 2021 Published: 06 July 2021

DOI: 10.5772/intechopen.98811

From the Edited Volume

## Fiber Optics - Technology and Applications

Edited by Guillermo Huerta-Cuellar

Book Details Order Print

To purchase hard copies of this book, please contact the representative in India:

CBS Publishers & Distributors Pvt. Ltd. www.cbspd.com | customercare@cbspd.com

## Abstract

This chapter deals with coded modulation and impairment compensation techniques in optical fiber communication. Probabilistic shaping is a new coded modulation technology, which can reduce transmission power by precoding, reduce bit error rate and improve communication rate. We proposed a probabilistic shaping 16QAM modulation scheme based on trellis coded modulation. Experimental results show that this scheme can achieve better optical SNR gain and BER performance. On the other hand, in order to meet the demand of transmission rate of next generation high speed optical communication systems, multi-dimensional modulation and coherent detection are sufficiently applied. The imperfect characteristics of optoelectronic devices and fiber link bring serious impairments to the high baud-rate and high order modulation format signal, causes of performance impairment are analyzed, pre-compensation and receiver side’s DSP techniques designed for coherent systems are introduced.

### Keywords

- Coherent Optical Communication
- Coded Modulation
- Probabilistic Shaping
- Digital Signal Processing
- Pre-Compensation
- Quadrature Amplitude Modulation

## Author Information

#### Zhipei Li

- School of Information and Electronics, Beijing Institute of Technology, Beijing, China

#### Dong Guo

- School of Information and Electronics, Beijing Institute of Technology, Beijing, China

#### Ran Gao*

- School of Information and Electronics, Beijing Institute of Technology, Beijing, China

*Address all correspondence to: 6120190142@bit.edu.cn

## 1. Introduction

With the rapid development of Internet services, higher requirements are put forward for the transmission rate, system capacity and stability of communication. Optical fiber communication has become one of the main communication methods in the world because of its large transmission bandwidth, long-distance transmission and strong anti-interference ability.

The optical fiber communication systems are mainly divided into two types: intensity modulation-direct detection (IM-DD) and coherent optical communication systems. The IM-DD system is mainly used in access networks, passive optical networks (PON) and data centers, and its transmission distance is usually less than 80km. Specified in the standard, the highest transmission rate with single-wavelength of IM-DD system is 100Gb/s, transmitting 56GBaud 4-level pulse amplitude modulation (PAM4) signal. The coherent optical transmission system, which uses multi-dimensional modulation to improve spectrum efficiency, local oscillator lasers to increase sensitivity, and digital signal processing (DSP) technology for impairment compensation, thereby can greatly improve transmission performance, increase transmission rate and distance, and is usually used in in large-capacity long-distance backbone networks and metropolitan area networks.

Quadrature amplitude modulation (QAM) is commonly used in coherent optical fiber transmission system. In this method, evenly distributed constellation points are arranged in two-dimensional space to form a constellation diagram. The performance loss of the modulation scheme tends to $\mathit{\pi e}/6$ (1.53dB) from the Shannon limit. To further approach Shannon limit capacity and improve system performance, probabilistic shaping technology can be used [1]. Probabilistic shaping is a new coded modulation technology. It can reduce transmission power by precoding, reduce bit error rate and improve communication rate without changing the original system. And with the change of channel environment, the channel can be matched by changing the size of the shaping, thus improving the flexibility of the system.

The future research direction of high-speed optical fiber communication is digital coherent optical communication technology [2, 3, 4]. With the development of high-bandwidth optoelectronic devices, digital-to-analog converter (DAC), analog-to-digital converter (ADC) and application specific integrated circuit (ASIC) chips, beyond 800Gb/s transmission with single wavelength above becomes possible. In order to achieve such high-speed transmission, DSP technology plays an important role in dealing with chromatic dispersion (CD), polarization mode dispersion (PMD), frequency offset and phase noise, by compensating the signal at the transmitter and receiver in the electrical domain. With highly integrated and flexible digital coherent optical detection technology, high-speed, large-capacity and long-distance optical communication can be effectively realized.

The most important technologies in coherent optical communication system are introduced in this chapter, a novel coded modulation technology based on probabilistic shaping and DSP-based Impairment compensation techniques.

## 2. Novel coded modulation technology based on probabilistic shaping

### 2.1 Principle of probabilistic shaping

The main idea of probabilistic shaping is to reduce the probability of occurrence of constellation points in outer ring, increase ones in the inner ring, and change the constellation of uniform distribution into non-uniform distribution. The common distribution matcher is constant composition distribution matcher (CCDM), which makes the occurrence probability of each constellation point conform to Maxwell Boltzmann distribution:

${P}_{X}\left({x}_{i}\right)=\frac{{e}^{-v{\left|{x}_{i}\right|}^{2}}}{{\sum}_{j=1}^{m}{e}^{-v{\left|{x}_{j}\right|}^{2}}}$E1

Where, $X=\left\{{x}_{1},{x}_{2},\dots \dots ,{x}_{m}\right\}$ is the constellation symbol set, $v$ is the probability distribution factor, and the value range is 0∼1. The larger value of $v$, the higher the degree of constellation shaping [5].

Taking 16QAM modulation format as an example, the constellation probability distribution diagram is shown in Figure 1, and the constellation diagram after recovery at the receiving end is shown in Figure 2.

### 2.2 Experiment

To evaluate the performance of the proposed scheme, an experiment is carried out by employing the coherent optical communication system setup illustrated in Figure 3. The system parameters are shown in Table 1. At the transmitter, a 1550nm lightwave with power of 10dBm and line-width of 100kHz is employed as the laser source, followed by a polarization beam splitter to divide the output light into orthogonal polarized pair. Then two I/Q modulator are applied to modulate two orthogonal light waves, respectively. Trellis-Coded Modulation (TCM)-PS-16QAM signals are generated offline by MATLAB program. And each modulator is driven by two 10Gbaud amplified electrical TCM-PS-16QAM signals with a frame length of 33,336 symbols, and 10 patterns (333,360 symbols) are collected for bit error ratio (BER) calculation. The peak-peak voltage of the two amplified electrical signals are both set as 2.0V. And a polarized beam combiner is applied to combine two orthogonal modulated lightwaves, which is amplified by an erbium-doped fiber amplifier (EDFA) in the standard single mode fiber (SSMF) for transmission. The gain and the noise figure of the EDFA are 20dB and 4dB, respectively. The input power of the fiber is 0dBm, and the fiber is a SSMF with attenuation of α=0.2dB/km, nonlinear coefficient of γ=1.3 (W·km)^{−1}, and dispersion of * D*=17ps/nm/km. At the receiver, the ASE noise is added to the received optical signal to adjust the received optical signal noise ratio (OSNR) with the resolution of 0.1nm. The optical signal added with noise is bandpass-filtered and converted into an electrical signal by a coherent receiver. The laser diode (LD) generates a local oscillating light with the power of 5dBm and linewidth of 100kHz. In the offline DSP module, timing recovery is implemented by Gardner algorithm and chromatic dispersion is compensated digitally. To process different shaped signals, a pre-convergence constant modulus algorithm with step of 2e-6 and taps of 9 followed by 1% pilot aided multi-modulus algorithm are used for polarization demultiplexing and compensation of polarization mode dispersion without training sequence. In addition, fast Fourier transform based frequency offset estimation (FFT-FOE) algorithm and blind phase search algorithm are applied to realize frequency offset estimation and compensate the laser phase noise, respectively. Finally, TCM-PS-16QAM decoding and statistics of BER are implemented.

Parameter | Specification |
---|---|

Center wavelength | 1550nm |

Laser linewidth | 100kHz |

Input power of fiber | 0dBm |

Amplification | EDFA |

EDFA gain | 20dB |

EDFA noise figure | 4dB |

Fiber | SSMF |

Attenuation | 0.2dB/km |

Nonlinear coefficient | 1.3(W·km)^{−1} |

Chromatic dispersion | 17ps/(nm·km) |

Span length | 25km |

### Table 1.

System parameters.

To verify our proposed scheme, we compare the performance of 8QAM, TCM-16QAM-4state, TCM-16QAM-8state, TCM-16QAM-16state, and TCM-PS-16QAM-4state respectively. The convolutional encoders and information entropy parameters are displayed in Table 2. Here, some modulation formats have different entropy, and lower entropy means a higher baud-rate, which makes their bit-rates consistent.

Scheme | Entropy (bits/symbol) | Baud rate (GBaud) |
---|---|---|

8QAM | 3 | 10 |

TCM-16QAM-4state | 3 | 10 |

TCM-16QAM-8state | 3 | 10 |

TCM-16QAM-16state | 3 | 10 |

TCM- PS-16QAM-4state | 2.9 | 10.4 |

2.8 | 10.7 | |

2.7 | 11.1 |

### Table 2.

Modulation format parameters.

In the experiment, the TCM-PS-16QAM (* H*=2.8 bits/symbol) with OSNR=15dB is firstly transmitted. After the optical transmission system, the received signal of the subset S0, S1, S2, S3 and the overall constellation at the receiver is shown in Figure 4. The successful transmission of the TCM-PS-16QAM signal means that our proposed novel scheme is reasonable and realizable. Meantime, it can be seen that the probability distribution of the constellation points achieves the goal of low energy point density and high energy point sparseness.

Figure 5 depicts the BERs of 8QAM, TCM-16QAM-4state, TCM-16QAM-8state, TCM-16QAM-16state, and TCM-PS-16QAM-4state (* H*=2.9 bits/symbol) with the same data rate of 60 Gb/s after 25-km SSMF transmission. It can be seen that when the OSNR is above 5dB, the performance of TCM-16QAM-nstate (n=4, 8, 16) is better than 8QAM. And with the increase of OSNR, the BERs of TCM-16QAM-nstate (n=4, 8, 16) fall faster than 8QAM. At the BER of 1×10

^{−3}, TCM-16QAM-4state obtains the gain of 3.4dB compared to 8QAM. However, TCM-16QAM-8state slightly outperforms TCM-16QAM-4state and the OSNR gain is just 0.4dB. Meantime, compared to 8 state, the gain of TCM-16QAM-16state grows only 0.4dB too. And the decoding complexity double as the number of state double. So, increasing the number of states is not a suitable way to obtain coding gain, especially when the coding gain is close to the limit. The required OSNR for BER of 1×10

^{−3}is about 7.7dB for TCM-16QAM-4state, and 6.8dB for TCM-PS-16QAM-4state (

*=2.9 bits/symbol), The OSNR improvement is increased by 0.9dB with a little more complexity. Compared to TCM-16QAM-8/16state, TCM-PS-16QAM-4state (*H

*=2.9 bits/symbol) has lower decoding complexity and better performance. And the gain of TCM-PS-16QAM-4state (*H

*=2.9 bits/symbol) grows 0.5dB and 0.1dB, respectively. This is mainly due to the shaping gain brought by PS. Under the condition that the minimum Euclidean distance in the constellation remains unchanged, the convergence of most constellation points greatly reduces the average power of the constellation, improving the performance of the entire modulation system.*H

The BER curves of TCM-PS-16QAM-4state with different information entropy after 25-km transmission are also measured, and the measured results are shown in Figure 6. In our experiments, according to their different information entropy, the baud rate is set differently, their information entropy is 2.9, 2.8, 2.7 bits/symbol, so the baud rate is 20.7, 21.4, 22.2Gbaud, namely, the bit rate is 60 Gb/s. It can be seen that the required OSNRs are 6.8dB, 6.5dB, and 5.9dB at the BER of 1×10^{−3}. In other words, TCM-PS-16QAM-4state (* H*=2.7 bits/symbol) obtains OSNR gain of 0.6 and 0.9dB compared with TCM-PS-16QAM-4state (

*=2.8 and 2.9bits/symbol), respectively. This advantage proves that as the information entropy decreases, more shaping gains can be obtained. Obviously, at the same OSNR, the lower the information entropy, the better the BER performance and the more flexible information entropy and gain compared to the traditional TCM-16QAM.*H

## 3. Impairment compensation techniques in coherent communication

Coherent optical communication systems are implemented in the form of optical modules in commercial communication equipment [6]. As shown in Figure 7, The transmitted information is encoded by forward error correction (FEC) through the electrical interface and then transmitted to transmitter-side (TX) DSP. The TX DSP includes constellation mapping, pulse shaping, pre-equalization, skew compensation, dispersion pre-compensation, etc. The main purpose of implementing DSP-based pre-compensation techniques at the transmitter is to increase the signal-to-noise ratio (SNR) of the transmitted signal while the transmission power is limited, reduce the bandwidth requirement for signal transmission, avoid inter-symbol interference (ISI), and reduce the impairment caused by the optoelectronic devices.

After single mode fiber channel, the coherent optical signal is transmitted into the coherent receiver and ADC to obtain four electrical signals, which consist of dual polarizations X/Y and in-phase/quadrature (IQ) components XI, XQ, YI, and YQ. The receiver-side (RX) DSP contains a series of impairment compensation and equalization algorithms for the optoelectronic devices and fiber link, including IQ signal orthogonalization, normalization, dispersion compensation, clock recovery, polarization demultiplexing, frequency offset estimation, and phase noise recovery. DSP compensation technology at the receiver side is the core of coherent optical communication. It can recover multi-dimensional modulated signals from distorted constellation, to realized large-capacity transmission.

This chapter will introduce the impairment compensation technology based on DSP in two parts, all common compensation techniques are described in detail.

### 3.1 Impairment compensation techniques at transmitter side

Pre-equalization and skew compensation are basic DSP techniques at transmitter side, pre-equalization technique is used to compensate the filtering effect caused by the bandwidth limitation of transmitting devices, and skew compensation is to compensate the delay of XI, XQ, YI and YQ signals while passing through the electrical and optical paths [7]. At the same time, in order to cope with more and more high-speed transmission, such as 400Gb/s, 800Gb/s and 1.2Tb/s transmission, high-order modulation format and high baud rate signal is generated and transmitted, such as 96Gbaud 32QAM and 80Gbaud 64QAM, which is extremely sensitive to linear and nonlinear impairment of optoelectronic devices, therefore, other pre-compensation algorithms such as look-up table (LUT) and digital predistortion (DPD) are usually added to the next generation coherent optical module’s DSP algorithm. In this section, basic principles of the impairment compensation techniques at transmitter side are described for the development of next generation coherent optical transmission systems.

#### 3.1.1 Bandwidth limitation and pre-equalization

Restricted by material and technical level, the frequency response of the optoelectronic device is not flat in the range of the bandwidth needed to transmit the signal, as shown in Figure 8, the transmitted signal will fade at high frequency, which leads to inter symbol interference, the smaller the bandwidth is, the more serious the ISI is and the worse the BER performance.

To suppress the performance degradation of high-speed signal caused by bandwidth limited system, the pre-equalization technology based on DSP can alleviate the bandwidth shortage of the transmitter device, which is an effective bandwidth compensation method. To implement pre-equalization, it is necessary to obtain the frequency response of the transmitter, including DAC, electric driver, modulator and so on. First, we send specific training sequence $X$ without any compensation, then receive the signal $Y$ with a high bandwidth digital sampling oscilloscope, we can estimate the frequency response * H* of the transmitter by comparing the transmitted and received signals with least square (LS) algorithm

${H}_{\mathit{LS}}={\left({X}^{H}X\right)}^{-1}{X}^{H}Y$E2

Then we multiply the inverse of the estimated frequency response with the transmitted data in the frequency domain

${X}_{\mathit{TX}}=X\xb7{H}_{\mathit{LS}}^{-1}$E3

Thus, the high frequency component of the signal can be raised at the transmitter to resist the low-pass filtering effect of the device. The spectrum of the transmitted signal is flat to reduce the inter symbol crosstalk.

#### 3.1.2 Look-up table algorithm

LUT is a pre-compensation method, which is used to compensate the memory effect of the amplifier. From the time domain point of view, the memory effect means that the current output symbol of the amplifier is not only dependent on the current input symbol, but also related to the past input symbol value. From the perspective of frequency domain, memory effect can be defined as the phenomenon that the amplitude and phase characteristics of intermodulation distortion term of amplifier change with the variation of envelope frequency of input signal. Considering the influence of $2N+1\phantom{\rule{0ex}{0ex}}$symbols before and after the symbols at the intermediate time, all possible transmitted sequences are $X\left(k-N:k+N\right)$, and the corresponding received sequence is $Y\left(k-N:k+N\right)$. All data in lookup table are set to 0 in the initial state, and the sliding window selects $2N+1$ symbols in the transmission sequence each time, after looking up the index and finding the address of this pattern, the error $E\left(k\right)$ is obtained by subtracting the central symbol of the sending sequence and the receiving sequence. As the sliding window moves forward, error values of all transmission modes can be traversed. Suppose the lookup table index is $i$, the number of data stored in index is $M\left(i\right)$, and the lookup table is updated as follows

$\mathit{LUT}\left(i\right)=\mathit{LUT}\left(i\right)+E\left(k\right)$E4

$M\left(i\right)=M\left(i\right)+1$E5

$\mathit{LUT}\left(i\right)=\frac{\mathit{LUT}\left(i\right)}{M\left(i\right)}$E6

#### 3.1.3 Summary

In terms of modulation and impairment suppression technology in transmitter of optical transmission system, researchers have carried out a lot of research, such as peak to average power ratio (PAPR) reduction technology, DAC resolution enhancement [8], joint pre-equalization in electrical and optical domain, etc. In recent years, artificial intelligence techniques have been recently proposed as a promising tool to address various challenges in optical communication, and machine learning technology based on indirect learning [9] and neural network [10] has also been used for impairment compensation of transmitter devices.

### 3.2 Impairment compensation techniques at receiver side

In the long haul and large capacity optical transmission system, the optical link will introduce chromatic dispersion [11], polarization mode dispersion (PMD) [12], fiber nonlinearity, etc., the laser linewidth and frequency jitter will bring frequency offset and phase noise [13, 14], and the ADC sampling frequency and phase cannot be synchronized with the DAC at the transmitter. All these problems can be solved by using mature DSP technology, so as to avoid the use of a series of complex devices such as phase locked loop. With the development of ASIC chip manufacturing technology, highly integrated and flexible digital signal processing technology can meet the needs of high-speed optical transmission system in the future.

As shown in the Figure 9, the basic DSP algorithm flow of a typical coherent optical communication receiver, including IQ imbalance compensation, CD compensation, timing recovery, polarization demultiplexing, frequency offset estimation, and carrier phase recovery. According to the algorithm design, different algorithms may have nested or parallel processing, these algorithms are closely linked and indispensable, which is the basis of coherent optical communication with high baud rate and high order modulation format.

#### 3.2.1 I/Q imbalance compensation

Ideally, the I and Q components in the received signal are completely orthogonal, but in the actual system, for the extinction ratio of the two arms of the IQ modulator is not completely consistent, the division ratio of the 3dB coupler in the receiver is asymmetric, and the response of the balance detector is inconsistent, the amplitude and phase of the two IQ components will be imbalanced. So, it is necessary to implement IQ imbalance compensation and normalization in the first step of digital signal processing. In general, Gram-Schmidt orthogonalization procedure (GSOP) algorithm is used to map one set of non-orthogonal vectors as reference variables and the other set as orthogonal variables. Suppose that ${I}_{\mathit{\text{in}}}\left(k\right)$ and ${Q}_{\mathit{\text{in}}}\left(k\right)$ are non-orthogonal vectors, and ${I}_{\mathit{\text{out}}}\left(k\right)$ and ${Q}_{\mathit{\text{out}}}\left(k\right)$ are vectors processed by GSOP, as

${I}_{\mathit{\text{out}}}\left(k\right)=\frac{{I}_{\mathit{\text{in}}}\left(k\right)}{\sqrt{{P}_{I}}}$E7

${Q}^{\prime}\left(k\right)={Q}_{\mathit{\text{in}}}\left(k\right)-\frac{\rho {I}_{\mathit{\text{in}}}\left(k\right)}{{P}_{I}}$E8

${Q}_{\mathit{\text{out}}}\left(k\right)=\frac{{Q}^{\prime}\left(k\right)}{\sqrt{{P}_{Q}}}$E9

where $\rho =E\left\{{I}_{\mathit{\text{in}}}\left(k\right)\xb7{Q}_{\mathit{\text{in}}}\left(k\right)\right\},{P}_{I}=E\left\{{{I}_{\mathit{\text{in}}}}^{2}\left(k\right)\right\},{P}_{Q}=E\left\{{{Q}^{\prime}}^{2}\left(k\right)\right\}$, and $E\left\{\xb7\right\}$ represents expectation.

#### 3.2.2 Chromatic dispersion compensation

Chromatic dispersion is a static impairment for optical signal in fiber transmission. The main factor of CD is that the characteristics of optical fiber material lead to different propagation group velocity of different frequency components of optical signal, which is similar to the multipath effect in wireless communication, resulting in time-domain pulse broadening. For the early optical fiber communication system, chromatic dispersion is mainly compensated by negative dispersion coefficient media such as dispersion compensation fiber, fiber Bragg grating and other dispersion compensation modules. With the development of DSP technology, digital signal processing technology can completely replace the function of optical dispersion compensation module, it is easy to realize dispersion compensation based on DSP.

Generally, dispersion coefficient $D$ is used to quantify the pulse broadening caused by fiber dispersion, the unit is $\mathit{ps}/\mathit{nm}/\mathit{km}$. The partial differential equation of the influence of fiber dispersion on signal envelope is derived, and the frequency domain transmission equation can be obtained by Fourier transform

$G\left(z,w\right)=\mathit{exp}\left(-j\frac{D{\lambda}^{2}}{4\mathit{\pi c}}{\omega}^{2}\right)$E10

where $\lambda $ is the wavelength of light wave, $c$ represents light speed, and $\omega $ is arbitrary frequency component. The frequency-domain transfer function of the dispersion compensation filter is obtained by inversing the dispersion coefficient of the transfer function as

$G\left(z,w\right)=\mathit{exp}\left(j\frac{D{\lambda}^{2}}{4\mathit{\pi c}}{\omega}^{2}\right)$E11

In the long-distance optical communication system, the signal sub block must be large enough to compensate for the dispersion effect in the transmission. Therefore, an overlapped frequency domain equalization structure is proposed to improve the transmission and DSP efficiency by forming overlaps between sub blocks as shown in Figure 10.

#### 3.2.3 Clock recovery

After photodetection, the electrical signal is sampled and digitized by the ADC. However, in the actual system, because the local sampling clock is not synchronized with the transmitter signal clock, the sampling point of the ADC is not the best sampling point of the signal in most cases. On the other hand, due to the instability of the local clock source itself, it may also cause the sampling error of the system. This sampling error includes both the sampling phase error and the sampling frequency error. The clock error of the sampling signal, on the one hand, is due to the imperfection of the sampling point, causing interference between sampling symbols; on the other hand, the jitter of the sampling clock will also cause fluctuations in signal performance. Therefore, in order to achieve optimal digital signal recovery, a clock recovery module is needed in the actual system to eliminate the impact of clock sampling errors. Considering that the dispersion will cause the disappearance of the clock component, usually, the clock recovery module is placed after the dispersion compensation or works with the dispersion compensation module to form a unified balanced feedback module.

The feedback time-domain clock recovery algorithm is proposed by Gardner [15]. This algorithm uses a feedback clock synchronization structure to estimate the phase of the retiming digital clock source feedback by calculating the timing error. The estimation of the timing error can track the frequency jitter of the signal, and the application of this algorithm can achieve dynamic clock recovery. On the other hand, the Gardner clock recovery algorithm only needs two samples per symbol and the algorithm complexity is low. It is widely used in the digital signal processing module of the coherent optical communication system.

#### 3.2.4 Adaptive equalization and polarization demultiplexing

As CD compensation technology has been well promoted in optical fiber communication systems, PMD becomes the primary impairment which limits the information capacity and transmission distance for over 40Gbit/s systems [16, 17, 18]. Besides, PMD is a stochastic impairment that converts with time, temperature, wavelength and fiber conditions which makes PMD hard to estimate and compensate. In recent years, the research on how to overcome the performance deterioration of optical communication system caused by PMD effect has been a hot topic, most of which focuses on PMD compensation [19, 20].

The adaptive equalizer is generally used to polarization demultiplex and PMD compensation of channels. A two-by-two multiple-input multiple-output (MIMO) structured finite impulse response (FIR) filters as shown in Figure 11 is used to estimate the inverse Jones matrix of the dynamic channel [17].

The input sequence of the filter is symbol-spaced with index $n$, while the $N$ tap FIR filters, ${h}_{\mathit{xx}}$,$\phantom{\rule{0ex}{0ex}}{h}_{\mathit{xy}}$, ${h}_{\mathit{yx}}$ and ${h}_{\mathit{yy}}$ are the column vector of length $N$. Tap weights are updated every two samples as the input sequence is two-fold sampled. Therefore, ${x}_{i}$ and ${y}_{i}$ represent a sliding block of $N$ samples such that

${x}_{i}\left(n\right)=\left[{x}_{i}\left(n\right),{x}_{i}\left(n-1\right)\dots {x}_{i}\left(n-N\right)\right]$E12

${y}_{i}\left(n\right)=\left[{y}_{i}\left(n\right),{y}_{i}\left(n-1\right)\dots {y}_{i}\left(n-N\right)\right]$E13

We consider that ${u}_{i}\left(\mathrm{n}\right)=\left[{x}_{i}\left(n\right),{y}_{i}\left(n\right)\right]$, ${h}_{x}\left(\mathrm{n}\right)=\left[{h}_{\mathit{xx}}\left(n\right),{h}_{\mathit{xy}}\left(n\right)\right]$, ${h}_{y}\left(\mathrm{n}\right)=\left[{h}_{\mathit{yx}}\left(n\right),{h}_{\mathit{yy}}\left(n\right)\right]$. And the filters outputs form as

${x}_{o}\left(n\right)={h}_{x}^{H}\left(n\right){u}_{i}\left(n\right)$E14

${y}_{o}\left(n\right)={h}_{y}^{H}\left(n\right){u}_{i}\left(n\right)$E15

where superscript${\left(.\right)}^{H}$ means the conjugate transpose.

For fast adaptive equalizer, we generally use stochastic gradient descent (SGD) optimizer to update the tap weights. Meanwhile, we need to choose a cost function to describe the error degree of the output samples so that the equalizer can get the response and update tap weights. For constant modulus algorithm (CMA), its cost functions are given as

${\epsilon}_{x}\left(n\right)={R}_{2}-{\left|{x}_{o}\left(n\right)\right|}^{2}$E16

${\epsilon}_{y}\left(n\right)={R}_{2}-{\left|{y}_{o}\left(n\right)\right|}^{2}$E17

where ${R}_{2}$ is the real-valued constant and given by ${R}_{2}=E{\left|{X}_{\mathit{sym}.}\right|}^{4}/E{\left|{X}_{\mathit{sym}.}\right|}^{2}$. The update equations of FIR filters are given as

${h}_{x}\left(n+1\right)={h}_{x}\left(n\right)+\mu {\epsilon}_{x}\left(n\right){x}_{o}^{\ast}\left(n\right){u}_{i}\left(n\right)$E18

${h}_{y}\left(n+1\right)={h}_{y}\left(n\right)+\mu {\epsilon}_{y}\left(n\right){y}_{o}^{\ast}\left(n\right){u}_{i}\left(n\right)$E19

where $\mu $ is the step size parameter and the superscript${\left(.\right)}^{\ast}$ means the complex conjugate operation.

The cost functions of CMA describe error degree between the amplitude of output symbols and the proposed convergence radius [17, 18]. By updating the tap weights of FIR filters with cost functions, the error degree can be minimized in a certain extent and the output symbols can converge on a circle with the proposed radius that equals to $\sqrt{{R}_{2}}$. The value of the constant ${R}_{2}$ and step size can significantly affect the convergence degree of CMA. Appropriate ${R}_{2}$ helps CMA to converge in less steps. Shorter step size can help CMA to get better convergence, but the computation of the algorithm is also increased. CMA needs enough steps to update and optimize its FIR filters. The output performance of CMA will be insufficient if the proposed convergence length is not long enough.

All tap weights are initialized to zero except the central tap of ${h}_{\mathit{xx}}$ and ${h}_{\mathit{yy}}$, which are initialized to unity. The filter taps of ${h}_{\mathit{xx}}$, ${h}_{\mathit{xy}}$, ${h}_{\mathit{yx}}$ and ${h}_{\mathit{yy}}$ estimate the components for data sequences. At the transmitter, data sequences for X and Y polarization are independent and only contain their own information. Thus, the central tap of ${h}_{\mathit{xx}}$ and ${h}_{\mathit{yy}}$ are set to unity. After fiber transmission, the received signal is contaminated by channel impairments and noise. With the contamination of noise and the channel impairments caused by linear effects such as CD, PMD etc. or fiber nonlinear effects, the received data sequences for X and Y polarization are no longer independent. One or more symbols in the received sequences can interfere other symbols, while the symbols for one polarization can interfere other symbols for another polarization. The interference between polarizations can be estimated by filter taps of ${h}_{\mathit{xy}}$ and ${h}_{\mathit{yx}}$. Though updating filter tap weights, the 2×2 MIMO structured FIR filters can gradually identify the components of each data sequence, and the dynamic impairments of channel can be compensated. This process can also achieve polarization demultiplexing.

It is worth noting that the CMA can be implemented in a full-blind mode, but it set no constrain with its outputs. Therefore, it is possible for the equalizer to converge on the same output, corresponding to the Jones matrix becoming singular. In practice, we need to check if ${h}_{x}$ and ${h}_{y}$ become singular after the algorithm running for certain steps, and if so, a mathematic process is necessary to make ${h}_{x}$ and ${h}_{y}$ nonsingular and the whole algorithm should be restarted.

CMA is especially suited to the modulation format with constant amplitude such as quadrature phase-shift keying (QPSK) and M-ary phase-shift keying (MPSK). For the formats with inconstant amplitudes such as quadrature amplitude modulation, the CMA error cannot converge to zero as extra noise is introduced during equalizing. To improve the SNR performance for the formats with inconstant amplitudes, several multi-modulus algorithms such as radius-directed algorithm (RDA) and cascaded multi-modulus algorithm (CMMA) are established [21, 22].

As CMA set at real-valued constant ${R}_{2}$, CMMA set several constants which depends on the ideal constellations. According to the distance between the input symbol and the constellation origin, the algorithm estimates which circle the symbol belongs to. Then CMMA calculates the error using the estimated radius. The rest of CMMA algorithm steps are same as CMA.

Compared with CMA, the CMMA significantly improves the SNR performance for high order QAM. But it reduces the robustness of the filter converging process. This is because multi-modulus algorithm depends on correct decision on symbol radius. For high order QAM format, the distance between different circles is less than the minimum symbol interval. Therefore, the algorithm may make massive mistake over the circle decision if the signal is severely contaminated by channel impairments and noise.

#### 3.2.5 Frequency offset estimation

In the coherent optical communication system, the transmitting laser and the local oscillator laser work independently, so the central wavelength cannot be exactly the same, so there is a certain frequency deviation $\u2206f$. It will introduce a continuous phase variation along with time to the received signal, resulting in constellation rotation, so it is necessary to estimate and compensate the frequency offset by DSP. Through FFT operation, the spectrum of received symbol’s fourth power value is obtained and analyzed, it can be found that there is a peak component at the frequency of $4\u2206f$, therefore, $\u2206f$ can be obtained by searching for the maximum spectral component of the fourth-power value of received symbol. Usually, due to the lack of spectral resolution and other reasons, there will be residual frequency offset after estimation, which can be looked at as additional phase noise and recovered by carrier phase recovery.

#### 3.2.6 Carrier phase recovery

Carrier phase recovery (CPR) is an essential DSP unit in coherent systems and has been extensively investigated for QPSK and QAM signals. Like frequency offset estimation algorithms, carrier recovery algorithms can be classified as either blind or data-aided estimation techniques. And the algorithms can be implemented in feedforward manner or in feedback structure [23, 24]. Compared to the constellation for QPSK, the constellation points for QAM vary in both phase and amplitude. Moreover, the modulated signal phase is with multiple values. Viterbi-Viterbi phase estimation (VVPE) algorithm using a fourth-power operation which is suitable for QPSK is hard to completely remove the signal phase and estimate the carrier phase. At present, the algorithms for QAM CPR mainly include Blind phase searching (BPS), improved BPS (BPS/maximum likelihood), decision aided maximum likelihood (DA-ML), etc. [25, 26, 27]

BPS is recognized as a favorable solution due to its high performance and suitability for parallel processing and can be used in feedforward manner.

Figure 12 shows the block diagram of the BPS algorithm in pure feedforward manner. The input signal ${x}_{i}$ is sampled at the symbol rate. The received signal ${x}_{i}$ is rotated by * B* test carrier angles ${\phi}_{b}$ with

${\phi}_{b}=\frac{b}{B}\bullet \frac{\pi}{2},b\in \left\{0,1,\dots ,B-1\right\}$E20

then all rotated symbols are fed into a decision circuit, which output the ideal constellation points with the minimum Euclidean distance to the input symbols. The squared distance ${\left|{d}_{k,b}\right|}^{2}$ to the closest constellation point is calculated in the complex plane

${\left|{d}_{k,b}\right|}^{2}={\left|{x}_{k}exp\left\{j{\phi}_{b}\right\}-\u230a{x}_{k}exp{\left\{j{\phi}_{b}\right\}}_{D}\u230b\right|}^{2}$E21

in order to remove the other initial noise distortions from the receiver, the distance of 2N+1 consecutive test symbols rotated by the same carrier phase angle ${\phi}_{b}$ are summed up

${s}_{k,b}=\sum _{n=-N}^{N}{\left|{d}_{k-n,b}\right|}^{2}$E22

the optimum value of the filter half width N depends on the laser linewidth times symbol rate product. N=6, …,10 is generally a good choice.

After filtering the optimum phase angle $\u2206{\phi}_{\mathit{est}.}$ by searching the minimum sum of distance values, then the output symbols ${y}_{i}$, which is the input symbols ${x}_{i}$ rotated by $\u2206{\phi}_{\mathit{est}.}$, is outputted for the following DSP in coherent systems.

Due to the 4-fold ambiguity of the recovered phase in the square M-QAM, the blind algorithms may cause incorrect phase estimation by a multiple of $\mathrm{\pi}/2$ causing cycle slip. This problem can be solved by using framing information or by applying differential coding [23, 24, 25].

Though BPS shows a good tolerance to laser phase noise and can be flexibly applied to higher order QAM, with an increasing modulation order a larger number of test phases are required and the computation complexity increases. Therefore, an improved BPS algorithm with a two-stage diagram has been established [24]. The first stage of improved BPS just requires rough estimation and the required number of test phase ${\phi}_{b}$ can be reduced. A maximum likelihood phase estimator is introduced in the second phase to improve the accuracy. This two-stage BPS/ML algorithm effectively improves the performance with the computation complexity and availability of BPS algorithm remained.

#### 3.2.7 Summary

This section introduces a series of impairment compensation algorithms for coherent optical communication system. Here, the constellation diagrams of 60GBaud 16QAM signal in DSP process is given in the Figure 13. Through the algorithms in this section, we can realize the signal orthogonalization and normalization, compensate the fiber dispersion, eliminate the clock sampling error, depolarize the multiplexing and equalize channel response at the same time, and finally recover the carrier phase to get the constellation of the original transmission signal. A series of impairment compensation algorithms based on DSP at the receiver end lay the foundation for high-speed, large-capacity and long-haul optical transmission.

## 4. Conclusion

In this chapter, first we propose a probabilistic shaping 16QAM modulation scheme based on trellis coded modulation. Through non-uniform probability mapping of TCM-16QAM subset, an effective and good overall probability distribution of 16QAM constellation is obtained. The scheme is successfully demonstrated in a 25km single-mode fiber transmission system, and better OSNR gain and BER performance are obtained. Then impairment compensation techniques in coherent optical communication are introduced from two aspects, transmitter side and receiver side, pre-emphasis and look-up table technology have been widely used in the impairment compensation of the transmitting devices, while the DSP process at the receiver side is more complicated, including GSOP, clock recovery, dispersion compensation, dynamic equalization, and carrier phase recovery. Using DSP technology can effectively mitigate the impairment of optoelectronic devices and optical fiber links, and it is an effective method to realize ultra-high speed, large-capacity and long-haul transmission.

## Acknowledgments

This work is supported by National Natural Science Foundation of China No.62022016, No.61835002, No.61727817, National Key R&D Program of China from Ministry of Science and Technology, China No.2019YFA0706300, Open Fund of IPOC (BUPT) No.IPOC2020A007, No.IPOC2020A006, and China Postdoctoral Science Foundation No.2020M680384, No.2020M680385.

## References

- 1.
Yankov M P, Zibar D, Larsen K J, Christensen L, Forchhammer S. Constellation Shaping for Fiber-Optic Channels with QAM and High Spectral Efficiency. IEEE Photonics Technology Letters, 2014; 26(23):2407-2410. DOI: 10.1109/LPT.2014.2358274 - 2.
Gui T, Wang X, Tang M, Yu Y, Li L. Real-Time Demonstration of 600-Gb/s DP-64QAM Self-Homodyne Coherent Bi-Direction Transmission with Un-Cooled DFB Laser. Optical Fiber Communication Conference. 2020 - 3.
Napoli A, Sun H, Maher R, Torbatian M, & Zhang J, 800G DSP ASIC Design Using Probabilistic Shaping and Digital Sub-Carrier Multiplexing. Journal of Lightwave Technology, 2020; 38(17): 4744-4756. DOI: 10.1109/JLT.2020.2996188 - 4.
Zhou X, Urata R, Liu H. Beyond 1Tb/s Datacenter Interconnect Technology: Challenges and Solutions. Optical Fiber Communication Conference. 2019. - 5.
Raphaeli D, Gurevitz A, Constellation shaping for pragmatic turbo-coded modulation with high spectral efficiency. IEEE Transactions on Communications, 2004; 52, 341–345. DOI: 10.1109/tcomm.2004.823564 - 6.
Zhang Z, Li C, Chen J, Ding T, Wang Y, Xiang H, Xiao Z, Li L, Si M, Cui X. Coherent transceiver operating at 61-Gbaud/s. Optics Express, 2015; 23(15):18988-95. DOI: 10.1364/OE.23.018988 - 7.
Napoli A, Berenguer P W, Rahman T, Khanna G, Mezghanni M M, Gardian L, Riccardi E, Piat A C, Calabrò S, Dris S. Digital pre-compensation techniques enabling high-capacity bandwidth variable transponders. Optics Communications, 2017; S0030401817308465. DOI: 10.1016/j.optcom.2017.09.062 - 8.
Sadot D, Yoffe Y, H Faig, Wohlgemuth E. Digital Pre-Compensation Techniques Enabling Cost-Effective High-Order Modulation Formats Transmission. Journal of Lightwave Technology, 2018; 37(2): 441-450. DOI: 10.1109/JLT.2018.2888941 - 9.
Paryanti G, Faig H, Rokach S L, Dan S. A Direct Learning Approach for Neural Network Based Pre-Distortion for Coherent Nonlinear Optical Transmitter. Journal of Lightwave Technology, 2020; 38(15): 3883-3896. DOI: 10.1109/JLT.2020.2983229 - 10.
Freire P J, Neskorniuk V, Napoli A, Spinnler B, Turitsyn S. Complex-Valued Neural Network Design for Mitigation of Signal Distortions in Optical Links. Journal of Lightwave Technology, 2021; 39(6):1696-1705. DOI: 10.1109/JLT.2020.3042414 - 11.
Prince M. Performance Limitations of an Optical TDM Transmission Link Caused by Fiber Chromatic Dispersion. 2020 FORTEI-International Conference on Electrical Engineering (FORTEI-ICEE); 2020. DOI: 10.1109/FORTEI-ICEE50915.2020.9249807 - 12.
Ip E, Kahn J M. Digital Equalization of Chromatic Dispersion and Polarization Mode Dispersion. Journal of Lightwave Technology, 2007; 25(8):2033-2043. DOI: 10.1109/JLT.2007.900889 - 13.
Huerta-Cuellar G, Pisarchik A N, Barmenkov Y O. Experimental characterization of hopping dynamics in a multistable fiber laser. Physical Review E, 2008; 78(3):035202. DOI: 10.1103/PhysRevE.78.035202 - 14.
Fatemi F K, Lou J W, Carruthers T F. Frequency comb linewidth of an actively mode-locked fiber laser. Optics Letters, 2004; 29(9):944-946. DOI: 10.1364/OL.29.000944 - 15.
Gardner F M. A BPSK/QPSK timing-error detector for sampled receivers. IEEE Transactions on Communications, 1986; 34(5):423-429. DOI: 10.1109/TCOM.1986.1096561 - 16.
Kikuchi K, Fundamentals of Coherent Optical Fiber Communications. Journal of Lightwave Technology, 2016; 34(1):157-179. DOI: 10.1109/JLT.2015.2463719. - 17.
Faruk M S, Savory S J. Digital Signal Processing for Coherent Transceivers Employing Multilevel Formats. Journal of Lightwave Technology, 2017, 35(5):1125-1141. DOI: 10.1109/JLT.2017.2662319 - 18.
Lagha M K, Gerzaguet R, Bramerie L, Gay M, Scalart P, Blind Joint Polarization Demultiplexing and IQ Imbalance Compensation for M -QAM Coherent Optical Communications. Journal of Lightwave Technology. 2020, 38, 4213-4220. DOI: 10.1109/JLT.2020.2986601 - 19.
El-Fiky E, Chagnon M, Sowailem M, Samani A, Morsy-Osman M, Plant D V. 168-Gb/s Single Carrier PAM4 Transmission for Intra-Data Center Optical Interconnects. IEEE Photonics Technology Letters, 2017; 29(3):314-317 DOI: 10.1109/LPT.2016.2647232. - 20.
Yu Z, Zhao Y, Hu S, Wan Z, Xu K. PDL and CD insensitive low complexity equalizer for short reach coherent systems. Optics Express, 2021; 29(5), 6657-6667. DOI: 10.1364/OE.418456 - 21.
Zhang T, Xiang Q, Zhang S, Liu L, & Zuo T. Cost-effective digital coherent short-reach transmission system with D8QAM and low-complexity DSP. Optics Express, 2021; 29, 11892-11902. DOI: 10.1364/OE.422456 - 22.
Zhang Q, Li X, Zhang N, Hu L, Xi L, A Low Complexity Frequency Domain Adaptive Equalizer for Coherent Optical Receivers. Asia Communications and Photonics Conference/International Conference on Information Photonics and Optical Communications 2020 (ACP/IPOC), 24-27 October 2020; Beijing, China. M4A.287. - 23.
Pfau T, Hoffmann S, R. Noe. Hardware-Efficient Coherent Digital Receiver Concept with Feedforward Carrier Recovery for MM -QAM Constellations. Journal of Lightwave Technology, 2009; 27(8): 989-999, DOI: 10.1109/JLT.2008.2010511 - 24.
X. Zhou. An Improved Feed-Forward Carrier Recovery Algorithm for Coherent Receivers With M-QAM Modulation Format. IEEE Photonics Technology Letters, 2010; 22(14):1051-1053. DOI: 10.1109/LPT.2010.2049644. - 25.
Zhao J, Chen L K. Carrier Phase Recovery Based on KL Divergence in Probabilistically Shaped Coherent Systems. Journal of Lightwave Technology. 2021; 39, 2684-2695. DOI: 10.1109/JLT.2021.3054949 - 26.
Borjeson E, Fougstedt C, Larsson-Edefors P. VLSI Implementations of Carrier Phase Recovery Algorithms for M-QAM Fiber-Optic Systems. Journal of Lightwave Technology. 2020: 38, 3616-3623. DOI: 10.1109/JLT.2020.2976166 - 27.
Lu J, Fu S, Hu Z, D Lei, Tang M, D Liu. Carrier Phase Recovery for Set-Partitioning QAM Formats. Journal of Lightwave Technology, 2018; 36(18):4129-4137. DOI: 10.1109/JLT.2018.2859238

Written By

Zhipei Li, Dong Guo and Ran Gao

Submitted: 30 May 2021 Reviewed: 10 June 2021 Published: 06 July 2021

© 2021 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution 3.0 License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### Continue reading from the same book

View All

#### Fiber Optics

Edited by Guillermo Huerta Cuellar

Published: 24 November 2021

Chapter 1 Multi-core Fiber Technology

By Muhammad Irfan Anis and Hamdan Ali

667 downloads

Chapter 2 Functional Tapered Fiber Devices Using Polymeric C...

By Oscar González-Cortez, Rodolfo A. Carrillo-Betanco...

370 downloads

Chapter 3 Optical Inhouse Networks

By Ulrich H.P. Fischer, Matthias Haupt and Peter Kußm...

314 downloads