RESOURCES

Methods Column

The Open Science Movement is for All of Us: Three Things You Can Start Doing Right Now
By Moin Syed, University of Minnesota, moin@umn.edu
SSEA Newsletter – April 2019

“Open Science” refers to a broad movement across the social, physical, and natural sciences. Open science is sometimes viewed as synonymous with the “replicability crisis” in psychology and related fields, and thus seen as relevant to only narrow subfields of psychology (e.g., social psychology). In reality the open science movement concerns a much broader and widely applicable set of reforms; issues that should be attended to by all producers and consumers of research regardless of discipline, theoretical orientation, or methodological preference. In other words, all members and affiliates of the SSEA community should be concerned with open science.

There is no strict definition of “open science,” but for the purposes of this post I will provide one here: Open science consists of principles and behaviors that promote transparent, credible, reproducible, and accessible science. Let’s take a closer look:

Transparency pertains to researchers being honest about theoretical, methodological, and analytic decisions made throughout the research cycle. Transparency means rejecting incomplete or opaque explanations that can mislead others. It is about reducing, as much as possible, the knowledge asymmetry that is inherent to the process (see Vazire, 2017). Transparency is not only about fraud or otherwise intentional misreporting of the details of research. To paraphrase Feynman (1974), it is very easy to fool oneself without intending to. Thankfully, there are safeguards that can be put in place to reduce the frequency and severity of sub-optimal behaviors due to self-deception (e.g., preregistration; more on that below).

Credibility is closely related to transparency: it pertains to how others will view your work and the work of a field in general. Credibility is the degree of trustworthiness and believability of the research reported in the literature. For a variety of inter-related reasons associated with suboptimal research practices, large swaths of the current body of research is not credible. The credibility of research is enhanced when it is transparently reported, well-powered, subject to replications, and relies on samples that reflect human diversity.

Reproducibility pertains to how well we keep records of what we do, at all phases of the research cycle, so that everything can be reproduced when needed (not if needed, because there will always be a need for reproducibility). Reproducibility is not only about enabling other people to reproduce your work. Indeed, in a saying attributed to Mark Holder, “Your primary collaborator is yourself six months from now, and your past self doesn’t answer e-mails.” Using clear and consistent documentation in your workflow allows you to better remember what you did, why you did it, and importantly, allows you to do it again.

Accessibility pertains to making all aspects of the research cycle open and available for those who are interested. This includes not only openly sharing data and materials, but sharing them in a format that is understandable and useable for outside researchers. Accessibility also includes making research products freely available to researchers and the public around the world. For example, posting non-copyrighted post-prints of articles and chapters on a server such as PsyArXiv (more on that below) increases accessibility. Similarly, posting psychological tests and measures online in an accessible format, free of cost to the user, can contributes to their broader use and thus broader range of samples from which we gain knowledge.

Open science clearly covers a lot of ground, and those who are new to these issues can be quickly overwhelmed with the myriad tools and behaviors that can be used to advance open science in their own research. Honestly, even those who are very well-acquainted with the issues at hand and have been engaging in open practices for years can sometimes feel overwhelmed (e.g., me). Additionally, open science is not an “all or nothing” enterprise and open science is not an identity group that people belong to or do not (Corker, 2018). Rather, open science is a varied array of principles and behaviors you can use in your research. Here I will outline three practices that are relatively easy to enact and are good starting points to moving toward a more open approach to your research:

Examine your workflow. How much have you ever even considered your research workflow? When you are planning a new study, do you formally write down your research questions and hypotheses? Do you develop a corresponding analysis plan? Do you have a clear plan for how and when you clean and prepare newly collected data sets? Do you clean your data manually or using automatic scripts? Are clear and detailed codebooks available for all of your data? Do you have multiple versions of the same dataset? Do you have naming conventions for your variables? Are all of your files stored in a single location and clearly labeled? Do you have 17 versions of the same manuscript or do you practice version control? How do you handle collaborative writing and revisions of manuscripts? If you are anything like I was (and still am in many ways), you have not spent sufficient time thinking about these issues. Challenge to you: just take a step back and examine your workflow. My guess is that if you do so you will rather quickly identify some poor practices that could be changed relatively easily. At the end of this post are links to resources that contain concrete suggestions for improving your workflow. Consult some of these resources and then try to identify 1-2 practices that you can change right now. Do those, then identify 1-2 more practices to change. Rinse and repeat.

Upload pre-prints and post-prints to PsyArXiv. PsyArXiv.com is a “pre-print server” for psychological research. A pre-print is a broad term used to refer to a version of a manuscript that has not yet been accepted for publication in a journal. This can include manuscripts that have not yet been submitted or those that are currently under review. Posting pre-preprints can facilitate receiving valuable feedback from colleagues and can speed up dissemination. PsyArXiv also accepts post-prints, which are final versions of manuscripts that have been accepted for publication. In most cases you cannot post the final journal-formatted version of the paper, but you can post your own Word doc (or equivalent). You can look up the journal in which you article appears on SHERPA/RoMEO to determine if uploading post-prints is permitted. Uploading post-prints facilitates dissemination and increases access to scholars who may not have subscriptions to certain journals. This is especially the case for book chapters, which are notoriously difficult to locate. Uploading your chapters can greatly increase their reach. Challenge to you: go to PsyArXiv right now and upload one of your published papers. It only takes a few minutes! After that, think about how you can incorporate the posting of preprints into your research process so that it is embedded in your workflow.

Preregister all of your studies. Preregistration involves clearly writing out all of your research questions, hypotheses, methods, and the analyses that you will use to test your hypotheses before you collect the data and/or do the analysis. Preregistration can enhance the credibility of your research because, when done well, can reduce bias due to data-dependent decision-making (e.g., p-hacking, Wicherts et al., 2016; HARKing, Kerr, 1998). Preregistration takes a bit more work than examining your workflow and posting a pre/post-print, but it is well worth it. Preregistering your study does not mean that you cannot also conduct exploratory analyses. You can! It simply helps you make a clear distinction between confirmatory/planned analyses and exploratory analyses. Preregistering your study also does not mean you cannot make any changes to your analysis plan. You can! You will just need to provide justification for why you made the changes you did, and still report the preregistered analyses (potentially as a supplement). For example, in a recent study with Ummul Kathawalla we were examining predictors of university students’ GPAs. It was not until after we finalized the preregistration plan that we remembered that we needed to standardize GPA within college due to different norms, variations, and entry requirements. The primary analyses were conducted using the non-registered standardized GPA measure with the registered raw GPA analyses reported in a supplement. Along the same lines, reviewers and editors often request different and/or new analyses through the review process. You can still include these suggested analyses (if you deem them relevant), clearly indicating that they were non-registered analysis. And, again, you will want to be sure to report all registered analyses in text or in a supplement. Finally, preregistration is not only suitable for hypothesis-driven experimental research. You can preregister exploratory work, longitudinal studies, secondary data analysis, meta-analyses, and qualitative/mixed methods studies. Preregistration is about developing a plan in advance, whatever that plan may be. We have found that doing so leads us to be much more thoughtful about what we are doing and why. Challenge to you: complete a preregistration for your next study. If you will be analyzing an existing data set, preregister your analysis plan before working with the data. Give it a try. It is entirely doable. I guarantee that you will make some mistakes and forget important details, but you will also learn a great deal and enhance the credibility of your findings. Preregistration is not about being able to predict the future in full, it is about developing a plan and making a clear distinction between confirmatory and exploratory analyses (or decision independence vs. dependence – see Srivastava, 2019). Any journal that expressly supports open science initiatives will understands this, will understand what you are doing, and will value your efforts.

In sum, if you are not yet on the open science bus, it is well past time to hop aboard. Taking the movement and associated practices in full can be overwhelming. Don’t do that. Start small, with the practices that are most accessible for your particular work. I have suggested three places to start here, but if these do not work for you, think about what will. Open science is about good science. It is about being more thoughtful about your research, and that is always a good thing.

This article has been excerpted and modified from a longer preprint titled, The Open Science Movement is For All of Us. The full preprint can be downloaded at https://psyarxiv.com/cteyb/.
Moin Syed is an Associate Professor at the University of Minnesota (USA) and is the Editor-in-Chief of Emerging Adulthood, the official journal of the SSEA. You can learn more about how he is fostering open science at the journal here. Feel free to email him (moin@umn) or follow him on Twitter, @syeducation.

References

Corker, K. (2018, September 12). Open science is a behavior [Blogpost]. Retrieved from:
https://cos.io/blog/open-science-is-a-behavior/

Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196-217.

Feynman, R. P. (1974). Cargo cult science. Engineering and Science, 37(7), 10-13.

Srivastava, S. (2018). Sound inference in complicated research: A multi-strategy approach. PsyArXiv.
Open access: https://psyarxiv.com/bwr48/

Vazire, S. (2017). Quality uncertainty erodes trust in science. Collabra: Psychology, 3(1).

Wicherts, J. M., Veldkamp, C. L., Augusteijn, H. E., Bakker, M., Van Aert, R., & Van Assen, M. A. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832.

Selected Resources

History and Overview of Open Science in Psychology

Gelman, A. (2016, September 21). What has happened down here is the winds have changed. [Blog post]. Retried from: http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/

Spellman, B. A. (2015). A short (personal) future history of Revolution 2.0. Perspectives on Psychological Science, 10(6), 886-899.
Open access: https://journals.sagepub.com/doi/full/10.1177/1745691615609918

Reproducible workflow:

Crüwell, S., van Doorn, J., Etz, A., Makel, M. C., Moshontz, H., Niebaum, J., … Schulte-Mecklenbeck, M. (2018). 8 easy steps to open science: An annotated reading list. PsyArXiv.
Open access: https://psyarxiv.com/cfzyx/

Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., ... & Frank, M. C. (2018). A practical guide for transparency in psychological science. Collabra: Psychology, 4(1).
Open access: http://www.collabra.org/article/10.1525/collabra.158

Rouder, J. N., Haaf, J. M., & Snyder, H. K. (2018). Minimizing mistakes in psychological science. Advances in Methods and Practices in Psychological Science.
Paywall: https://journals.sagepub.com/doi/abs/10.1177/2515245918801915
Open access: https://psyarxiv.com/gxcy5/

Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013). Ten simple rules for reproducible computational research. PLoS Computational Biology, 9(10), e1003285.
Open access: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285

Preprints:

Tennant, J., Bauin, S., James, S., & Kant, J. (2018). The evolving preprint landscape: Introductory report for the Knowledge Exchange working group on preprints. MetaArXiv.
Open access: https://osf.io/preprints/metaarxiv/796tu/

Preregistration:

Haven, T., & Van Grootel, L. (2019). Preregistering qualitative research. Accountability in Research.
Paywall: https://www.tandfonline.com/doi/abs/10.1080/08989621.2019.1580147

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606.
Paywall: https://www.pnas.org/content/115/11/2600.short
Open access: https://osf.io/2dxu5

Weston, S. J., Ritchie, S. J., Rohrer, J. M., & Przybylski, A. K. (2018). Recommendations for increasing the transparency of analysis of pre-existing datasets. PsyArXiv.
Open access: https://psyarxiv.com/zmt3q/

A collection of preregistration templates for different types of research: https://osf.io/zab38/

Within-Person Methods
Theo A. Klimstra, Tilburg University, The Netherlands, 2017

A vast majority of researchers examining developmental processes in young adults (as well as in other age groups) mainly report on the “average” person and use a relatively limited number of measurement occasions to do so. Often, this suffices to answer the research questions we are interested in. However, for certain types of research questions we need a different kind of data. I’ll provide some examples of how I work with this different kind of data. It should be noted that my expertise is as an applied researcher, not as a statistician.

It is important to note the limitations of typical regression or correlation analysis. Indeed, we cannot conclude from these types of analysis that two variables are associated with each other within a particular person. A positive Pearson (or Spearman) correlation between identity exploration and fearfulness does NOT indicate that when your adolescent child starts exploring, they’d also become more fearful. Because the correlation and regression analyses we typically employ produce between-person associations, they only only tell us that adolescents who explore more than their peers in the sample (i.e., are higher up in the rank order of exploration) tend to be more fearful when compared to other people (i.e., are higher up in the rank order on fearfulness). Often, this will be the kind of information we are looking for, as it is an important task of psychologists to identify individuals who are at risk for developing some kind of problem.

However, to examine whether exploration induces fear in a particular adolescent child, we need a different kind of analysis and a different kind of data. That is, we need frequent measurements of both identity exploration and mood, and examine within-person associations. It has long been quite difficult to collect such data with frequent measurements of the same constructs within the same individuals, but the internet and especially the widespread use of smartphones has made it a lot easier to collect such data. Once the data is in, it can be analyzed with relatively complex multilevel analyses (see e.g., Klimstra et al., 2016, Journal of Research on Adolescence). In fact, in many cases, these will be the preferred method. In my own research, I have also used the somewhat easier-to-apply q-correlation. This q-correlation can be very appealing for examining some research questions.

The q-correlation is a within-person association and is therefore calculated for every single individual within a particular sample. So, after you run the syntax (there are syntaxes available for SPSS, but likely also for R and other software), a new variable appears in your dataset. This variable indicates, for every single individual in the sample, how the two constructs that you are interested in (e.g., identity exploration and some mood variable like fear) are associated. Values might range from –1.00 and 1.00. So, for some individuals this value might be positive (e.g., .70), meaning that more exploration comes together with more fear in these individuals. In others, values might be negative (e.g., -.30), which would indicate that they tend to become less fearful whenever they are exploring. In individuals with values close to zero, exploration and fear would be (almost) unrelated. Because the q-correlations appear as variables in the dataset, they can be used for any kind of follow-up analysis. For example, they can be correlated with other variables to answer research questions like “are fear and exploration more strongly associated in individuals with higher levels of neuroticism”? One could even identify profiles based on q-correlations, to distinguish individuals in whom these variables are positively related, from those in whom these same variables are unrelated or negatively related. Some year ago, I for example used this method to study individual differences in the extent to which mood is related to the weather (Klimstra et al., 2011, Emotion).

The possibilities of data with frequent measurements of the same construct are not limited to calculating within-person associations. Another possibility is to track the stability of certain constructs and examine individual differences in this stability. An easy way to do this is to calculate within-person standard deviations (for an application, see e.g., Klimstra et al., 2010, Journal of Personality and Social Psychology). However, these within-person standard deviations do often not produce the best estimates of within-person variability (e.g., Maciejewski et al., 2015, Child Development). Other measures, such as summing up absolute differences for each individual between all consecutive measurements for a particular construct might be better representation of variability. In addition to just studying variability, it is also possible to examine particular patterns in, for example, mood. For example, Kuppens and colleagues have studied mood inertia (resistance of certain mood states to change; e.g., Kuppens, Allen, & Sheeber, 2010, Psychological Science). Similar phenomena can obviously be studied in other variables that may be of interest to researchers studying young adults.

Many interesting phenomena can be examined if frequent measurements are collected among the same individuals and then analyzed with within-person analyses. The current review was meant to provide an illustration of some of these possibilities, however it is by no means an exhaustive list. When diving in to this kind of research, it is very important to read your way into the literature and consult methodologists if you do not fully understand certain analyses. Also, it is important to reflect on whether it makes conceptual sense to examine the construct you’re interested in on a daily level. For example, it doesn’t make much conceptual sense to examine short-term fluctuations in a supposedly stable construct like intelligence or some personality trait. Finally, I would not recommend examining within-person associations in datasets with just a few (e.g., 3 to 5) measurement occasions. Like with any type of method, only use within-person analyses if it matches your research questions AND you’ve got the right kind of data.


 

The Self-Perception Profile for Emerging Adults
Susan Harter, University of Denver, 2016, sharter@du.edu

Susan Harter is a Professor of Psychology at the University of Denver. She obtained her Ph.D. from Yale University where she joined their faculty before moving to Denver. She has been listed in two separate international surveys as one of the 50 top developmentalists in the field. She is well known for her life-span battery of Self-Perception Profiles as well as two developmental books (1999, 2012) on the Construction of the Self (Guilford Press).11:45 AM 7/24/2016


Introduction and Rationale

Why do we need another new multidimensional instrument, the Self-Perception Profile for Emerging Adults, described here, to tap self-perceptions during the period between adolescence and adulthood? Research reveals that most 18 to 25-year-olds do not yet consider themselves to be adults (Arnett, 1997). They realize that they are no longer adolescents but have yet to embrace adult roles and responsibilities in US culture. Thus, Arnett (2000, 2007, 2010) introduced the term “emerging adulthood,” bringing credibility to a new and distinct transitional period of development. This stage occurs between the age of 18, when most individuals graduate from high school, and around the age of 25-plus, when young people are presumed to enter “true” adulthood. During emerging adulthood, youth are typically no longer as dependent upon parents as they were during childhood and adolescence. However, most have not yet taken on the full and more enduring responsibilities of adulthood, for example, an occupational or career commitment, financial independence, marriage, home ownership or parenting.

Individuals during the period of emerging adulthood undergo many transitions (see summary in Harter, 2012). A major developmental task for emerging adults involves occupational exploration, often leading to education beyond high school, for example, college, vocational school, graduate school, or further professional training and degrees. There is often experimentation in the realm of romantic relationships. During this period, romantic liaisons typically last longer than during adolescence and many emerging adults engage in an exploration of a more serious intimate relationship and commitment that, for some, leads to the consideration of cohabitation or marriage.

Peer social relationships also undergo change. There are no longer the ready-made high school niches that comfortably provided friendship opportunities. The emerging adult must put forth more individual effort to make friends and garner acceptance in the workplace, on the college campus, or in other wider societal contexts.
Relationships with parents also undergo challenging transitions. During the high school years, the norm is still dependence on parents who meet many of the adolescent’s needs. For emerging adults, there is a developmentally-appropriate need to establish increasing independence from parents, and to become a more autonomous individual. However, the healthy emerging adult must also remain more maturely connected to parents (see Harter, 1999; 2012).

Global concepts of self may also change or be in flux during emerging adulthood. For example, a person’s level of self-esteem may undergo alterations. Another self construct, the perception that one is displaying one’s true self versus donning a false self, can also be subject to alteration (see Harter, 2012, for a more detailed description). Other new domains also arise during emerging adulthood or take on new developmentally-appropriate manifestations.

Domains of the Instrument

Thirteen different domains, each defined by their own subscale, were identified. Given that the very definition of emerging adulthood implies a sense of “becoming,” self-perceptions in each domain reflect the fact that these judgments are evolving. The emerging adult is a psychological “work in progress.” Thus, the actual wording of these thirteen domains, as well as of the actual items themselves, should reflect this normative developmental process.

• Intelligence. This subscale assesses how intelligent or cognitively competent the young adult perceives the self to be at this transitional period of life.
• Job/occupational competence. This subscale taps the extent to which young adults feel that they are successful at exploring job or occupational options where they can perform competently.
• Athletic/physical competence. Given that high school provides a structure for playing sports that will vanish upon graduation, the items on this subscale tap the young adult’s success in finding new opportunities within which to pursue their athletic interests and perceived physical skills post-graduation.
• Physical appearance. This subscale assesses the young adult’s perceptions of his or her current physical appearance, meeting personal or societal standards of attractiveness.
• Peer friendships/social acceptance. This subscale taps whether or not, after high school, the emerging adult has been able to develop new and meaningful friendships and to garner acceptance in his or her current social environment.
• Intimate relationships. This subscale assesses whether the emerging adult feels that he/she has the capacity to develop an intimate relationship, one that is more serious and reflects commitment, in contrast to casual romantic encounters.
• Relationships with parents. This subscale taps the challenge of moving toward increasing autonomy or independence from parents while continuing to maintain a more mature sense of psychological connectedness to parents.
• Morality. This subscale addresses the goal of developing more internalized moral or ethical standards that the young adult has personally come to construct and to own.
• Sense of humor. This subscale is specific to young adults’ ability to laugh at themselves in current life situations that may be unplanned, awkward, or potentially embarrassing.
• Daily life management. This subscale taps a major challenge for emerging adults, namely the practical ability to meet the new demands of managing one’s daily life responsibilities.
• Optimism. This subscale addresses emerging adults’ feelings of optimism or hopefulness about their futures versus pessimism and hopelessness.
• True-self/false-self behavior. This subscale taps the young adult’s ability to be his or her true self in social situations versus the tendency to don a false self in order to cope socially.
• Global self-esteem. This subscale taps the young adult’s current perception of his or her global self-esteem, that is, how much he or she likes himself or herself as a person, in contrast to disappointment with his or her overall self.

The actual instrument, formatted for use, with an explanation of the question format, administration instructions, scoring procedures, etc., are all described on my website: http://portfolio.du.edu/SusanHarter

Data Collection

We are currently planning our own data-gathering strategies. However, given advanced knowledge about this instrument, there has been considerable interest on the part of investigators studying this period of emerging adulthood, including requests to be allowed to administer this measure. I have been ambivalent, given that I think it first needs to have documented psychometric properties. I have discussed this with Jeffrey Arnett who has seen the measure and with his blessing, and a few tweaks of items, we have agreed on the following: For me to release the instrument to thoughtful researchers who will examine its psychometric properties including its validity in studies on this population, and report back to both of us.

I am particularly concerned that it be employed in the service of thoughtful research questions and hypotheses, to which I am told is what the members of this society are devoted. I feel it is essential that researchers include appropriate non-college samples. Too many studies have used university students as “samples of convenience”. However, the majority of emerging adults are not college students. So whom does one select? There are many considerations. One is a person’s socio-economic level. Another is the particular occupation. For example, are we talking about those who are still employed as wait staff or in service jobs vs. those who have become more skilled laborers such as electricians, plumbers, or those employed in the technological world of computers, who do not have college degrees? Are we talking about those who have moved along the path to independence from their families versus those who, for various reasons, are still living in their parents’ home, not uncommon in today’s world?

As Arnett has thoughtfully observed, an emerging adult is not a generic term for anyone in our culture who has graduated from high school and is of a certain age. Nor should researchers race off to other cultures, though cross-cultural issues are critical in our global, expanding or perhaps constricting, world today. I would not advise this. This instrument, like many others that I have constructed, was designed for American youth. I have the utmost respect for many colleagues who are doing thoughtful cross-cultural research, who know and understand the culture that they are investigating. But too many well-meaning researchers take American measures off to other countries or cultures, with little cultural sensitivity or knowledge of the culture, returning with meaningless results. People have taken my measures to cultures where there is not even a term for “self-esteem” in their language. I have written about cross-cultural issues in a chapter in my 2012 book, The Construction of the Self: Developmental and sociocultural foundations.

Jeffrey and I propose that, if people want to use this instrument toward examining thoughtful hypotheses, and if the instrument proves to be reliable and valid, such investigators submit their research, in an article (APA style) to both of us. We will review such articles and select a few to be the basis for a symposium at the October 2017 convention. If there is interest, we could schedule a follow-up session for discussion of the use of this instrument, for those interested.

This is an exciting new venture. I am a believer in emerging adulthood. I raised one of these creatures! We joked about how we would just “freeze-dry” her for a few years, and have her emerge as a fully-developed adult! Such is obviously not to understand development. Of course, I imagine myself to have been fully-evolved, skipping this tortuous period of development. Witnesses to the contrary, please do not come forth.
Please join us in this endeavor.