Take Your Text Analysis up a Level
Organizational research is complicated and the levels of analysis we are interested in is no small contributor to this complexity. In one room at our annual Academy of Management meeting you could find one scholar looking at the within-individual cognitive processes of individuals at work and another comparing the influence of institutional-level processes on firm strategy.
This diversity of interests enriches our field and the human/social origins of many of our constructs creates myriad opportunities for cross-pollination across levels. However, application of constructs across can introduce both theoretical and methodological challenges. For instance, management researchers famously debated the appropriate level of measurement and analysis of the organizational climate construct (i.e., Glick, 1985; James, Joyce, & Slocum, 1988; Glick, 1988).
There are well-established guidelines regarding how to use survey methods to elevate constructs’ level of measurement (i.e., Chen, Mathieu, & Bliese, 2005). However, survey response rates in organizational research can make rigorous applications of these methods difficult, causing researchers to turn to other techniques for organizational-level measurement. Computer-aided text analysis (CATA) is particularly attractive for measuring organizational-level constructs given the regular publication and availability of organizational texts. However, research in this area lacked guidelines for how to elevate constructs’ level of measurement.
Elevate Level of Analysis Using CATA
In our 2013 article, Using Computer-Aided Text Analysis to Elevate Constructs: An Illustration Using Psychological Capital, Jeremy Short, G. Tyge Payne, and I set out to adapt best practices from survey research to demonstrate how to elevate level of analysis using CATA while guarding against common theoretical and methodological concerns arising from doing so. We identify 19 steps in 5 phases that CATA researchers should follow.
Five phases:
- Definition of construct and development of deductive word lists
- Specification of the theoretical nature of the elevated construct
- Selection of appropriate texts and finalization of word lists
- Assessment of psychometric properties
- Examination of construct relationships
Key Considerations
In the first phase, you need to specify the definition and dimensionality of the construct at both the original and destination levels of analysis. These provide the basis for the development of initial deductive word lists for the CATA measures.
In the second phase, you need to provide the theoretical backing for the nature and existence of the higher-level construct. This involves not only accounting for the processes by which the higher-level construct emerges, but specifying the ways in which the elevated construct is similar to/differs from the original construct. This is key, because the conceptualization of the construct drives methodological decisions you will make in your content analysis.
In the third phase, you will use the theory developed in the first two phases to identify and collect a sample of appropriate texts with which to finalize and validate the measure. For instance, if measuring a construct at the organizational level, you will ideally collect a sample of texts reflecting organizational-level phenomena and contributed to by multiple individuals in the organization (e.g., Shareholder letters, 10-K filings, etc).
In the final two phases, you should put your measure through a battery of validation and psychometric assessments to ascertain the measurement qualities of the elevated construct. Many of these tests should be done to validate all new measures, regardless of their origin. However, a distinguishing characteristic here is that you want to compare your findings to the theory you specified regarding the elevation of the construct. Should there be close ties to findings of the lower-level construct, or is the elevated construct likely to be fairly different?
Organizational Psychological Capital
To demonstrate this process, our study elevated positive psychological capital to the organizational level. At the individual level, organizational behavior research defines psychological capital as being “an individual’s positive psychological state of development” (Luthans, Youssef, & Avolio, 2007: 3) and reflects the shared variance among an individuals optimism, hope, confidence, and resilience. Scholars in this literature have posited that this construct may also manifest at the organizational level, reflecting the positive psychological resources of the organization and may be a potential source of competitive advantage (e.g., Luthans, Luthans, & Luthans, 2004).
Our study created CATA measures for Organizational Psychological Capital. Using a sample of shareholder letters from S&P 500 companies, we examined how CEOs use language associated with organizational psychological capital in communications with shareholders. In-line with Luthans, Luthans, and Luthans’ (2004) assertion, we found preliminary evidence that organizational psychological capital may be an organizational resource leading to improved firm performance.
The primary contribution of this study is methodological. However, given the well-documented benefits of positive psychology for individuals (e.g., Avey et al., 2011) and our finding that positive psychological firm resources may influence firm-level outcomes, there may be a valuable managerial implication as well. Firms that find ways to instill in their employees the optimism, hope, resilience, and confidence that engender psychological capital may benefit not only from a healthier, happier, and more productive employee base, but also (and likely by consequence) may find positive performance outcomes for the firm as well.