So, fungal spores are literally everywhere, and the requirements for fungus to thrive seem to be trivially low; give it a moderately humid environment and it’ll grow on a bare concrete wall ffs eating god only knows what; the dust from the air maybe?
Well, and the great outdoors is full of slightly damp places, many of them downright soggy most of the time - and absolutely rife with organic material to snack on.
Where’s the bottleneck? Why isn’t the world a choking fungal hellscape?
In addition to other reasons listed in replies, there’s a fair bit of bacteria that excrete antifungal compounds into their environment so that they can colonize it more effectively than the mold.
There are viruses that attack fungi, too.
Any environment that hasn’t been sterilized by some means is a constant microbial warzone between bacteria, fungi, algae, archea, viruses, etc. with the current winners being determined by how favorable the conditions (moisture, pH, toxin concentration, etc.) are to each species.
Springtails and certain mites also love to eat fungus. I use them to keep mold under control in my terrariums