New Research Suggests that Leadership Assessments May Be Unreliable.

An international group of researchers at the University of Groningen in the Netherlands and West Texas A&M University in the United States have shown that assessments of leadership are not as reliable as once believed, providing evidence that common tools used to asses leadership may need to be taken with a grain of salt.

Assessments of leadership behaviors are common in organizations, often used in 360 degree feedback sessions, leadership training programs, and even as criteria for promotions and advancement for managers. Scholars who study leadership also use these types of assessments in scholarly publications. Many popular books have been written about the value of different types of leadership based on these assessments. Most well-known leadership measures—measures like transformational leadership, servant leadership, ethical leadership—have well-established technical manuals and reputable studies demonstrating that these assessments are reliable (i.e., are measuring what they say they are measuring), valid (i.e., that higher scores on these measures translate into results), and distinct (i.e., Leadership Style A is clearly something different than Leadership Style B).

Details from a study recently published in Organizational Research Methods raises questions about the reliability and distinctness of some of these assessments. Based on a study of nearly 400 individuals conducted over several weeks, researchers Jonathan Shaffer, David de Geest and Andrew Li assessed the reliability of 13 common leadership assessments. Their study focused on three types of error that can reduce reliability in leadership assessments: random response error, specific factor error, and transient error. Random response error is the kind of error that occurs when someone makes a simple error like writing a 3 when they meant to write a 2, or accidentally skipping question. Specific factor error occurs when an item or set of items in an assessment is too different from the other items. Transient error is random, meaningless change that occurs over multiple administrations of the same assessment. Most research on leadership assessments focuses on reducing or even eliminating the first two types of error, and in general, most leadership assessments have little random response or specific factor error.

Shaffer and colleagues’ work shows that for transient error, there is a different story. “Almost all of the thirteen types of leadership measured showed much, much more transient error than we expected,” said de Geest. When an assessment has a lot of transient error, its reliability is over-estimated. The study shows that for a number of assessments, the reliability of these assessments is over-estimated by as much as twenty-eight percent.” When transient error in a measure is high, researchers cannot use these assessments to measure changes in individuals over time, and it creates a bias in these assessments that makes it harder to determine their meaning.

High transient error also reduces the distinctness of different assessments. All types of measurement error bias observed relationships downwards, reducing the correlation between different assessments. In order to assess the true relationships between assessments, scholars need to evaluate and correct assessments for measurement error. Taking into account the new estimates of transient error, Shaffer and colleagues’ work showed that many leadership assessments have remarkably high correlations and may be less distinct than once believed. For example, scholars consider transformational leadership a unique and powerful form of leadership. The results of this study showed that, after correcting for transient error, followers cannot distinguish transformational leadership from several other types of leadership.

Despite this somewhat pessimistic view on leadership assessments, the study’s authors have an optimistic view. “A lot very smart people have made incredible strides in understanding leadership with these assessments,” said de Geest. “Our study just makes clearer the limits of these assessments and suggests that future assessments of leadership take into account new, nonconventional ways to assess leadership. We believe that doing so can lead to even better insights for future leaders.”