Understanding the structure and dynamics of highly diverse tropical forests is challenging. Here we investigate the factors that drive the spatio-temporal variation of local tree numbers and species richness in a tropical forest (including 1250 plots of 20 × 20 m2). To this end, we use a series of dynamic models that are built around the local spatial variation of mortality and recruitment rates, and ask which combination of processes can explain the observed spatial and temporal variation in tree and species numbers. We find that processes not included in classical neutral theory are needed to explain these fundamental patterns of the observed local forest dynamics. We identified a large spatio-temporal variability in the local number of recruits as the main missing mechanism, whereas variability of mortality rates contributed to a lesser extent. We also found that local tree numbers stabilize at typical values which can be explained by a simple analytical model. Our study emphasized the importance of spatio-temporal variability in recruitment beyond demographic stochasticity for explaining the local heterogeneity of tropical forests.