The ability to reliably capture performance parameters must be considered as crucially important to produce valid study results. The ICC and the inclusion of the calculation of the standard error of measurement and the minimal detectable change became the most common way to justify subsequent testing procedures to be reliable. However, early studies around the new millennium identified weaknesses of the ICC and proposed the implementation of more elaborate procedures, including the quantification of the systematic bias and the quantification of the random error via the mean absolute error or mean absolute percentage error. According to the law of large number and earlier research indicating that relative indices such as correlation coefficients necessitate a minimum sample size to stabilize, it was hypothesized that reliability indices follow an optimal sample size trend. In accordance with previous studies in correlation coefficients, this study highlights the importance of including high numbers of participants to receive stable reliability measures. The random error was not significantly affected by increased samples while providing important information about the performed standardization success in the testing, the study also underlines the relevance of reporting not only ICC-based reliability statistics but also the quantification of random errors.