[ad_1]
Summary box
-
Academic peer review is a foundation on which scientific knowledge is built, however, this type of training is not widely available.
-
High-quality peer-reviewer training open to researchers across the globe has the potential to improve the published literature. Our paper offers a model for other programmes and points to lessons learnt and considerations for future work.
-
Given the comprehensive instruction on peer review as well as potential spillover effects of learning writing skills, this peer-reviewer programme could have the potential to strengthen research capacity.
Academic peer review is a foundation on which scientific knowledge is built; peer reviews shape knowledge by optimising manuscript quality and acting as a gatekeeper for publication.1 Peer review is a key professional skill that helps researchers participate in the academic community, build their professional reputations and advance their careers.2 3 In addition, giving writing feedback improves the writing skills of a reviewer.4 5 In fact, providing feedback to another writer has been shown as more helpful in improving a reviewer’s writing skills than receiving and incorporating feedback on one’s own writing.4 6 Most early-career researchers do not receive formal training on peer review,7 8 and this unmet need is particularly relevant for early-stage researchers without available mentorship.8 9 These disparities mirror publication disparities across the globe, wherein those with less access to resources and mentors are not as positioned to succeed compared with those with more privilege.10 11
Improving the quality of peer reviews, however, has been an ongoing challenge. Peer-reviewer training programmes have met with limited success in improving the quality and rigour of reviews.12–14 The best way to carry out such a training is also unclear.15
Another challenge in academic publishing is the unprofessional tone and disrespectful way in which some authors are treated by peer reviewers. Journal reviews with disrespectful language and even personal attacks are not uncommon in the peer-reviewed literature,16 and authors writing in English as a secondary language may experience this mistreatment more frequently than those for whom English is primary.3 Maintaining a supportive stance in peer review is considered a professional obligation17 and is more helpful in encouraging writers to productively revise their work compared with disrespectful and/or harsh treatment.18
PREPSS (Pre-Publication Support Service) is an organisation, which provides manuscript development and writing support to assist authors and address authorship disparities in the published literature.10 One of the ways PREPSS approaches this goal is by training researchers on how to write a publishable paper.19 A team of volunteer peer reviewers helps the PREPSS organisation offset its limited funding and staff to support authors in ‘getting in the door’ at a peer-reviewed journal. PREPSS peer reviewers have experience publishing papers in academic journals, but they do not necessarily have peer-review experience. The peer-review training programme was established to ensure all are equally prepared for the role.
Since its launch in 2017, a major goal of PREPSS has been to increase the diversity of our peer review team in terms of geographical representation. Prior to the COVID-19 pandemic, we held our peer-reviewer training workshops in person at the University of Michigan. During the COVID-19 pandemic, PREPSS converted to virtual training and we realised that this strategy allowed us to improve the geographical diversity of our peer review team. We offered our first virtual training workshop in 2020 and have continued with this format since.
Our peer-reviewer training programme is designed to help reviewers learn to create a peer review that is rigorous, thorough and that motivates a revision that will be accepted for peer review by an academic journal. A major focus of our programme is helping participants (particularly those from the Global North) understand the forces behind disparities in the published literature. We emphasise engaging with a respectful and supportive stance in a global community of scholars.
This Practice paper describes the current efforts PREPSS has undertaken to develop a virtual peer-reviewer training programme with a diverse participant group and the first steps in evaluating the programme and identifying opportunities for growth. Central questions include the feasibility of developing an online peer-reviewer training programme to fill existing needs, and in what ways the intervention influenced participant confidence in their peer-review skills. We also evaluated participants’ self-reported impact on their writing skills. In this Practice paper, we describe the programme, highlight its effectiveness in building peer review and writing skills, and discuss strengths, challenges, and lessons learnt.
Workshop overview and goals
The objective of the peer-reviewer training is to acquaint participants with inequities in publishing and educate them about writing effective peer reviews. A focal point is how to provide specific and respectful feedback to help authors get accepted for peer review at an academic journal.
There is no cost to peer reviewers to attend the training programme, but those who complete the training are asked to join the PREPSS volunteer peer review team to review up to two papers over the course of 1 year. Peer reviewers are featured on the PREPSS website, awarded badges for service and are invited to participate in our annual peer-reviewer appreciation week that includes a free scientific communication training. Furthermore, peer reviewers receive access to the other reviews written for the papers they are evaluating as this is a learning opportunity identified in the literature.8 Finally, reviewers are notified when the paper they have reviewed is published, providing them an opportunity to see the culmination of their supportive efforts.
Our workshops are designed to be collaborative and highly participatory. Figure 1 provides an overview of the training time frame and activities. After participants complete the peer-reviewer training programme, they are considered ‘Provisional’ within the PREPSS system. Reviews written by ‘graduates’ of our training programme are assessed by the PREPSS editor-in-chief for quality and completeness. After a provisional reviewer’s initial review is approved as meeting the quality standards laid out in the training, the reviewer moves to active status.
Stages and timing of the Fall 2020 and Spring 2021 PREPSS virtual peer-reviewer training programs. The process consisted of an application period and 1.5-hour orientation (participants could attend synchronously or watch the recording afterward), time for attendees to complete a preworkshop activity on their own and attendance at a 4-hour synchronous virtual workshop (offered at two different times to accommodate various time zones).
Programme delivery and logistics
Applicants to this free online programme are accepted if they meet our criteria of having a minimum educational background of PhD candidate status, being a postdoctoral research fellow in a health-related field, or a medical resident with a research focus, and having first-authored a published paper in a peer-reviewed journal. We do allow for two exceptions to these criteria, as follows. First, we accept participants into the training programme who do not have a first-authored published paper if they have a paper under review at a peer-reviewed journal. These individuals are considered as ‘reserve’ status (they are part of our team, but on hold for reviewing PREPSS papers) until their first-authored paper is published, after which they are moved to ‘provisional’ status. Second, in rare circumstances, participants without the above minimum educational background (eg, someone with a master’s degree but no PhD) are accepted if they have an extensive publication record. Individuals with a terminal degree and at least one first-authored paper in a peer-reviewed journal are not required to complete the training and may join the peer review team with a simple request. These requirements are intended to ensure that authors will be reviewed by researchers with applicable academic and publishing experience. Ella August, a public health faculty member with expertise in writing and health sciences research (and the editor-in-chief of PREPSS), developed the training and led the main training workshop. August also led most of the orientation, however, Anam Khan (PREPSS’s then associate editor) guided orientation participants on how to complete the take home activity. To apply, individuals completed an online application posted on the PREPSS website, Twitter and University of Michigan newsletters.
The training programme consisted of a 1.5-hour orientation (participants could attend synchronously or watch the recorded orientation afterwards), a ‘take-home’ activity to help participants prepare for the main training workshop, and synchronous attendance at the virtual main training workshop (the components and timing are summarised in figure 1 and topics are highlighted in table 1).
Description of topics included in the Fall 2020 and Spring 2021 PREPSS virtual peer-reviewer trainings
The orientation covered disparities in global health research, challenges to publishing research, an overview of PREPSS and discussion of training activities. Synchronous attendees participated in Zoom breakout rooms to discuss barriers to publishing. The PREPSS staff created geographically diverse breakout rooms to foster sharing of differing perspectives.
Between the orientation and the main training, participants had 2 weeks to review an early unpublished draft of an example paper (anonymised and shared with the author’s permission) and complete a preliminary assessment of the paper. We identified the target journal for which the paper was written so reviewers could assess how well the paper was tailored to that readership. The assessment comprised five worksheets corresponding to different parts of the manuscript (see online supplemental appendix for worksheets). This assessment did not take the form of a review.
Participants were given approval to attend the final training based on the completeness of their forms. PREPSS editorial staff looked for engagement with the text and thoughtfulness of responses as an indication of future participation in the synchronous training and likelihood of providing valuable feedback to authors. Participants attended the 4-hour main training synchronously, choosing from two time options to accommodate different time zones (the topics are described in table 1).
In the first part of the main training, the workshop leader (EA) covered writing with clarity and a respectful tone, reviewed PREPSS peer-review guidelines and emphasised leveraging your strengths when writing a peer review, as well as writing across cultures (eg, avoiding idioms). Participants analysed and discussed excerpts from open (ie, publicly available) peer reviews.
In the second part of the main training, participants collaborated with their breakout group to write a review of the example paper based on the analysis in their worksheets and their collaborative discussion. Again, we created diverse breakout rooms, intentionally mixing people from different geographical regions with the understanding that this would improve the quality of their review.20
A link to an anonymous cross-sectional programme evaluation was emailed to participants who completed the programme. The evaluation queried trainee demographics, prior exposure to concepts covered in the training and their assessment of the programme. Short answer spaces were provided for open-ended feedback. Our programme evaluation was reviewed by the University of Michigan Institutional Review Board HUM00192169 and given the status of ‘not regulated.’
Programme participants
A total of 118 qualified participants applied to the programme (48 and 70 in 2020 and 2021, respectively), and 42% (49/118 across the 2 years) completed the training, including the 1.5-hour orientation, the at-home activity and the 4-hour synchronous virtual workshop (table 2). Sixty-five per cent (n=23 and n=9 in 2020 and 2021, respectively) of the 49 who finished the training completed an anonymous evaluation. We present combined results across the 2020 and 2021 cohorts here due to the small number of participants and because participants in the two cohorts are not meaningfully different with respect to data we collected.
The number of participants who met the eligibility requirements for, attended the orientation and finished the peer-reviewer training programme in the Fall 2020 and Spring 2021 PREPSS peer-reviewer training workshops
Just over half (53%; 17/32) of participants who completed the programme and evaluation originated from North America or Europe, a quarter (25%; 8/32) originated from Asia, 19% (6/32) originated from Africa and 1 originated from South America (table 3). Two-thirds of participants who completed the programme and evaluation (66%; 21/32) resided in North America or Europe.
Total eligible applicants (n=118) versus those who completed the programme and postprogramme evaluation (n=32), by continent (nobody resided in or was from Antarctica) in combined 2020 and 2021 training events
Most of the 49 participants in our 2020 and 2021 cohorts were advanced students or early-career academics. The majority were doctoral candidates (n=22), 12 were postdoctoral research fellows, 6 were MDs, 6 were PhDs and 3 had master’s degrees with extensive publications.
Participants’ peer review and publishing experience
For both cohorts combined, 53% (17/32) of evaluation respondents had not previously/were not sure if they received peer reviewer training compared with the 47% (n=15/32) who had received training. In terms of writing for a peer-reviewed journal, 91% (29/32) received some training. Meanwhile, 63% (20/32) served as a peer reviewer for an academic journal. Participants reported authoring (in any author position) between 1 and 75 articles (median: 2). Six participants reported having 0 publications in peer-reviewed journals, and this was acceptable for participants at the training stage (participants with a paper under review are accepted to the training programme, remaining as ‘reserve’ reviewers until their first-authored paper is published).
Programme evaluation
Orientation evaluation
All evaluation respondents agreed that the orientation was helpful in providing them a better understanding of their role as a peer reviewer at PREPSS, and nearly all respondents found the content helpful in gaining insight on peer reviewing and preparing for the main training (table 4). Most respondents valued the opportunity to engage with attendees from diverse geographical locations (table 4). Participants learnt about barriers to publishing in the Global South in both years; in 2020, this was referred to as ‘colonial science’. In the postassessment from the earlier training, 57% (13/23) said their understanding of ‘colonial science’ increased. In 2021, participants’ mean understanding of similar content, called ‘barriers to publishing in the Global South’ was 5.2 and 8.8 before and after the orientation (on a scale of 1–10 where 10 was the highest level of understanding).
Evaluations of the orientation and main training programme, in 2020 (n=22/30) and 2021 (n=8/30) training events
Evaluations of the take-home activity worksheets
It took most (67%; 20/30) participants between 1 and 3 hours to review the example paper and complete the worksheets, and the others (33%; 10/30) reported spending four or more hours on this activity. Nearly all participants who responded to the evaluations reported that completing the worksheets prior to the training was useful for structuring their evaluation of the example manuscript (table 4).
Evaluations of the main programme
Over-three quarters of participants (77%; 23/30) agreed that the training was helpful in improving their peer-review skills (table 4). Likewise, 77% (23/30) agreed that the training was helpful in improving their own writing skills. A majority (90%; 26/29) would recommend the workshop to their peers. Participants appreciated learning about comprehensive peer-reviewing skills, the impact of tone on a writer, and the power of providing encouraging comments.
Lesson learnt
In this section, we share several lessons learnt, areas for improvement, and considerations for future training. Some participants wanted the opportunity to get to know other participants beyond their interactions during the 4-hour training. We subsequently initiated a peer-reviewer appreciation week that provides opportunities for volunteers to network. Furthermore, several participants shared that they enjoyed and benefited from collaborating on peer reviews and we subsequently changed our policy to allow for this as an option if the reviewers wish to do so for their post-training peer reviews. Some participants commented that the activity took longer than expected and suggested that we let applicants know about the commitment ahead of time (we implemented this suggestion for the 2021 training).
Internet connectivity was unstable for certain participants, which made some of the group discussions feel frustrating. We recognise now how technology can reduce barriers but also introduce new challenges. Online-only is more geographically inclusive than in-person training, but consideration should still be given for different internet capacities. For example, we could have helped participants with more stable connectivity be more sensitive to this frustration by discussing it at the beginning of the programme and explaining the reasons it might occur (ie, limited resources).
In our training, we included those with a range of publishing and peer-reviewing experience. Teaching writing skills to a diverse audience is quite challenging,21 and this range in experience and skill levels could help explain why some people benefited more from our training and some benefited less. Demographic factors like gender10 22 and age23 have been shown to impact participation and success in academic activities and we will capture such variables in future iterations of our programme evaluation. In addition, though most of our participants were quantitative researchers, others were qualitative researchers and shared a desire to incorporate a qualitatively researched example paper. This is something we plan to add to a future training. In addition, adding a qualitative component to future evaluations of our own programme will add value.
It is also likely that some participants in our programme, perhaps those with less writing experience, learnt something new from completing the worksheets and from creating the collaborative peer review in the main training. Each worksheet contained specific prompts; for example, the introduction worksheet asked reviewers to assess whether the standard introduction arguments were made such as why the research is important (see online supplemental appendix). Reviewers were also asked to consider whether the text was tailored to the target audience and journal.
One novel and important focus of the programme is creating an awareness of the harsh and disrespectful treatment that second-language writers sometimes receive. This type of treatment should be avoided.3 16 Participants discussed examples and identified ways in which peer reviewers could be supportive and orient comments toward productive revision. This professional development skill will help reviewers in their future collaborations and even in their teaching, if they choose to pursue it.
Our programming accommodated participants in different time zones. We offered two options for the orientation (synchronous or recorded) and two time options for the main training. This is important for any training seeking to include geographically diverse participants.
Given the comprehensive instruction on peer review as well as potential spillover effects of learning writing skills, this peer-reviewer programme could have the potential to strengthen research capacity, though that was not the primary intention. Research capacity is of particular importance in settings where training on academic discourse is limited.
Conclusions
We have described a novel online peer-reviewer training programme that included participants from or residing in six continents across the globe. The programme featured education about publication barriers and author disparities, training on peer review, and an opportunity to build community with others.
Peer review is a powerful tool that can build a reviewer’s own writing skills4 24 and our participants felt their own writing would benefit from the programme. Peer review is thought to help a reviewer improve their own writing because they can see problems more clearly in another person’s work, likely because they are seeing it with fresh eyes.25 Learning to recognise and reflect on different writing problems in another writer’s work can translate into detecting them in a person’s own writing.4 25
PREPSS is among the organisations26–28 committed to supporting authors in publishing their work in academic journals, and we have a policy that prohibits our peer reviewers from participating as authors on the papers they review. PREPSS is a small organisation which partners with and is supported by institutions such as non-profit organisations and universities.29 Individual members of these partnering entities (eg, researchers employed by universities) receive PREPSS services. A future larger funding source would allow PREPPS to achieve its goal of being an open and free resource.
PREPSS is a unique but small organisation striving to serve as a scalable model to build a community of scholars and strengthen capacity across the globe. Our peer-reviewer training programme is innovative with the potential rippling effect of improved writing skills. We hope that others in a variety of geographical settings implement peer-reviewer training programmes to support their own affiliated researchers and to build community with other researchers.
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study.
Ethics statements
Patient consent for publication
Acknowledgments
We thank Dr. Anam M. Khan, the former Associate Editor at PREPSS, for her outstanding work in coordinating the peer reviewer training activities. We also thank our PREPSS peer reviewer team.
[ad_2]
Source link