The implemented existing high capacity optical networks are based on 50 GHz wavelength-division multiplexing (WDM) channel spacing and carry up to 50 Gb/s data rate per channel. In recent years, there is increasing interest in using dual-polarization (DP) technique to enable future systems with at least 100 Gb/s rate per channel to operate over existing optical networks. The performance of these advanced systems can be enhanced further by using coherent optical orthogonal frequency division multiplexing (CO-OFDM) technique which offers high spectral efficiency and outstanding tolerance of fiber dispersion. This thesis addresses the effect of fiber nonlinear optics on the performance of dual polarization CO-OFDM/WDM system operating with 100 Gb/s per channel. Different modulation formats, namely BPSK, QPSK and 16- QAM, are used. First, the performance of a single-channel system is investigated in the absence and presence of fiber nonlinearity. The results are compared with those of a conventional (single-polarization) system to identify the key role played by the DP technique. The investigation is then extended to WDM systems incorporating DP-OFDM technique. The results reveal that the effect of fiber nonlinearity can be reduced or cancelled by using optimum transmitter laser power Popt. Further, the value of Popt is a function of transmission distance, number of multiplexed channels, and modulation formats. Popt of -11, -12, and -5 dBm is needed for a ten-channel DP system operating with BPSK, QPSK, and 16-QAM formats, respectively. Simulation results presented in this thesis are obtained using OptiSystem (version 11.1) which is a commercial software package.