Spatial regions that do not mix effectively with their surroundings and, thus, contribute less to the heat transport in fully turbulent three-dimensional Rayleigh-Bénard flows are identified by Lagrangian trajectories that stay together for a longer time. These trajectories probe Lagrangian coherent sets (CSs) which we investigate here in direct numerical simulations in convection cells with a square cross section of aspect ratio Γ=16, Rayleigh number Ra=105, and Prandtl numbers Pr=0.1,0.7, and 7. The analysis is based on N=524288 Lagrangian tracer particles which are advected in the time-dependent flow. Clusters of trajectories are identified by a graph Laplacian with a diffusion kernel, which quantifies the connectivity of trajectory segments, and a subsequent sparse eigenbasis approximation (SEBA) for cluster detection. The combination of graph Laplacian and SEBA leads to a significantly improved cluster identification that is compared with the large-scale patterns in the Eulerian frame of reference. We show that the detected CSs contribute by a third less to the global turbulent heat transport for all investigated Prandtl numbers compared to the trajectories in the spatial complement. This is realized by monitoring Nusselt numbers along the tracer trajectory ensembles, a dimensionless local measure of heat transfer.