This paper presents the significant meaning of choosing the suitable modulation formats in DWDM-PON transmission system, and selects four kinds of formats: RZ, NRZ, MD-RZ and ODB according to references. After introducing the principal of these four formats, high efficient simulation software- OpticSystem has been used to analyse the 16×2.5Gbps experimental links. As the conclusion, the ODB format exhibits the greatest preponderance for multi-path and high-speed DWDM-PON transmission system.