The aim of this study was to investigate whether livestock and poultry farms affect fungal communities in the surrounding soils, and how the fungal communities respond to manure application. Fresh manure samples were collected from 10 beef cattle farms and 12 egg-laying chicken farms in Ningxia, China, during five different rearing stages. Surface soil samples were collected at different distances from the farms (20–300 m) and after manure. Real-time quantitative PCR and high-throughput sequencing were used to explore fungal communities in the manure and soil samples. The fungal α-diversity of cattle manure was relatively high in the mid-rearing stages, and the fungal α-diversity of chicken manure was highest during the initial laying stage. Ascomycota (71.9% of total sequences), Neocallimastigomycota (8.1%), and Basidiomycota (3.9%) were the dominant phyla in cattle manure. Ascomycota (62.3%) also predominated in chicken manure, followed by Basidiomycota (9.1%) and unclassified_Fungi (9.0%). The presence of cattle and chicken farms had no significant effects on fungal diversity or the abundance of dominant taxa in the surrounding soils (within the 300 m range). Compared with the control soils (barren wasteland), the fungal diversity was higher in the manure-treated vegetable soils; the abundance of Ascomycota was decreased, while unclassified_Fungi and Basidiomycota were enriched. In cattle manure, organic matter and Cr contents were strongly linked to fungal diversity and the abundance of specific taxa. Meanwhile, in chicken manure, total nitrogen, total phosphorus, and Cu contents were major factors controlling the fungal diversity and community composition. Major soil nutrients, together with Cu and Zn contents, had pronounced effects on fungal communities around the farms. These results will be useful for the selection of microbial inoculants to facilitate the composting of cattle and chicken manures, and for the scientific application of animal manure as organic fertilizer.