layout: page title: Reliability section_id: methods —

Many details about reliability, its role, the indices that measure it, etc., are known in the scientific community. Still, many other details are hardly known or applied. We try to advance the relevant use of reliability with several of our research projects.

See a presentation about some lesser-known properties of reliability from the viewpoint of cognitive science.

The relation between reliability and the power of the hypothesis tests

The relation between reliability and statistical power is complex. On the one hand, it is well known that reliability attenuates the measured correlation and, consequently, the power of the related hypothesis test. On the other hand, some robust cognitive (and other) phenomena are not reliable, termed the reliability paradox (e.g., Hedge, Powell, Sumner, 2018). This issue has been discussed and investigated in the literature. Here, I attempt to provide a more extensive, still accessible summary and explanation for the key relations between reliability and statistical power. Understanding the relation helps researchers to plan and evaluate research designs better.

Krajcsi, A. (2025, May 5). Reliability and statistical power: Conceptual background and practical implications. https://doi.org/10.31234/osf.io/mu896_v1

See a presentation about this work.

Appropriate bootstrapping method for calculating reliability

A bootstrapping method recommended in the methodological literature relies on sampling with replacement. However, this method may overestimate the reliability. We evaluate critical properties using a Monte Carlo simulation and provide recommendations for more accurate reliability calculations.

The manuscript is in preparation. Check back later for more details.

The simulation code is in preparation. Check back later for it.

Theoretical maximum of reliability

We often think about reliability as the result of some noise in the environment or in the participant that comes from occasional events or impulses. Still, according to some models, random noise is an inherent part of the processing. This latter description means an unavoidable noise in the measurement and, consequently, a theoretical maximum in the reliability. In the present project, it is investigated which factors influence the reliability and how. In addition, a software solution is provided to quantify the expected maximum possible reliability after some key parameters are estimated.

The manuscript is in preparation. Check back later for it.

The script is in preparation. Check back later for it.

Calculating reliability in CogStat

CogStat is a data and statistical analysis software that runs the analyses automatically and compiles an output that is easier to understand and interpret. In version 2.4, we added some initial support for reliability analyses and plan to improve them in later releases.

The example of the Alternating Serial Reaction Time task

We investigated some fundamental properties of the ASRT task’s reliability.

Farkas, B. C., Krajcsi, A., Janacsek, K., & Nemeth, D. (2023). The complexity of measuring reliability in learning tasks: An illustration using the Alternating Serial Reaction Time Task. Behavior Research Methods, 1–17. https://doi.org/10.3758/s13428-022-02038-5. Or read the preprint.

The example of the Mnemonic Similarity task

We investigated some fundamental properties of the MST’s reliability.

Nemecz, Z., Szűcs, T., Keresztes, A., & Krajcsi, A. (2025). Studying brain-behavior correlations and individual differences using the Mnemonic Similarity Task – a note of caution on reliability. Check back later for the manuscript.

Find the analysis code and other related materials at osf.io.

The example of the Approximate Number System acuity

Work in progress. Come back later for more details.