Chromatin conformation capture (3C) and high throughput 3C (Hi-C) assays allow for studies of three-dimensional (3D) genome structures in bulk samples through proximity ligation of DNA. However, the difference between cells can only be observed by single-cell measurements that avoid ensemble averaging. To study 3D chromatin organization and dynamics before and after fertilization in flowering plants, we analyzed the 3D genomes of rice egg, sperm, unicellular zygote, and shoot mesophyll cells. We show that chromatin architectures of rice egg and sperm are comparable to that of mesophyll cells and are reorganized after fertilization. The rice single cell 3D genomes display specific features of chromosome compartments and configuration of telomeres/centromeres compared to those shown in mammalian single cells. Active and silent chromatin domains gather to form multiple foci in the nuclear space. Notably, the 3D genomes of egg and unicellular zygote contain a compact silent center (CSC), which is absent in sperm. CSC appears to be reorganized after fertilization, which may be involved in gene regulation related to zygotic genome activation (ZGA). Our results reveal specific 3D genome features of plant gametes and unicellular zygote and provide a spatial chromatin basis for ZGA and gene expression in plant.