The analysis of the turbulent heat transfer in thermal convection flow is mostly done in the Eulerian frame of reference. Here, we use an ensemble of more than half a million individual Lagrangian tracer trajectories, which are advected together with the evolving turbulent flow, to identify subsets that contribute least to the heat transfer. These trajectories probe Lagrangian Coherent Sets — sets that remain spatially connected for a finite time and evolve in time. The analysis is based on three-dimensional direct numerical simulations at three different Prandtl numbers. Our Lagrangian analysis results provide a complementary view on the transport and are found to be consistent with the standard Eulerian results.