We used the nestedness temperature calculator to examine patterns of nestedness in two datasets on birds in fragmented landscapes in southeastern Australia. During our initial analyses, we found that the nestedness calculator was susceptible to detect nestedness as an artifact of passive sampling. To examine this problem in more detail, we created random matrices that simulated a situation where nestedness did not occur, and we re-analysed two previously published datasets. Our results showed that the nestedness calculator may (1) overestimate the degree of nestedness and its statistical significance, and (2) consider some datasets which include both ubiquitous and rare species to be significantly nested although they are not. Our results highlight the danger of blindly relying on the p-values generated by analytical packages such as the nestedness calculator. We suggest that users of analytical packages (often field ecologists) will need to be more critical in future and familiarise themselves in more detail with the packages they use. In addition, the developers of analytical packages (often theoretical ecologists) will need to communicate more clearly the limitations and assumptions underlying analytical tools such as the nestedness calculator.