How will study sections meet in the future? NIH peer review depends on robust meetings where groups of scientists, through vigorous discussion, identify the applications of highest merit. For the last 75 years, until last March, nearly all chartered review committee meetings were held in-person. Today, in response to the pandemic, 90% of all CSR review meetings are run as video (“Zoom”) meetings. CSR is taking steps now so that when all options are back on the table, we can make informed choices about how best to convene review meetings.
Last round we obtained survey responses from 3,000 NIH reviewers, ratings by scientific review officers (SRO) of 230 review meetings, compiled quantitative data comparing in-person versus Zoom instances of over 275 meetings, analyzed rosters from those meetings, and also surveyed our support staff.
The data give no indication that the forced switch to Zoom has introduced major problems. Quality of review is our number one priority; 60% of reviewers and 85% of SROs say that overall quality has remained the same. By a large margin, reviewers report feeling equally able to speak and be heard in Zoom meetings compared to in-person meetings (see Figure). Reviewers, SROs, and support staff generally report the platform is easy to use, meetings are easy to manage, and technical problems are no more common or difficult to resolve than for in-person meetings.
However, some of the survey results give us pause. Despite favorable ratings of Zoom meetings, reviewers and SROs prefer in-person meetings by very similar margins (43% to 31% for reviewers; 44% to 36% for SROs). Zoom meetings tend to run longer and the increased duration may explain some of that preference. Comparing Zoom to in-person, almost half of reviewers reported diminished attention, 51% perceived lower engagement, 30% contributed less, and 36% rated discussions as worse. In addition, many reviewers commented that they missed the social aspects of in-person meetings, the chances to network, to build collaborations, and to enjoy the comradery that in-person meetings foster.
The highly uneven impacts of COVID on scientists’ daily lives make it difficult to discern how meeting format affects their ability/willingness to participate as reviewers. For many, family care responsibilities have increased during the pandemic and women are bearing a disproportionate load. We examined roster composition carefully to see if reviewer demographics have changed during the pandemic. So far, we see little change. Interpretation is complicated by the fact that reviewers are dealing with two large changes at once – video format, which may make it easier to participate, and COVID, which likely makes it harder.
We are all still learning. “How to run better Zoom meetings” is a frequent topic of workshops, meetings, and staff conversations. Reviewers have reached out, for example, reminding us of the challenging home/work realities they are contending with. We understand that many reviewers must work from home and cannot completely absent themselves from family and household responsibilities. SROs generally are implementing more frequent breaks to accommodate this new reality. Even so, cameos by pets and children are not “unprofessional”, they’re OK, perhaps even fun. We understand you might want to eat your breakfast with the video off. However, please come back, and mostly stay on screen because video seems to improve engagement. Hopefully, more breaks will help with Zoom fatigue, but one size does not fit all, and good communication between SROs and reviewers is vital.
Going forward CSR and OER plan to gather additional data from the February and March 2021 review meetings. We will consider those data, your comments, and input from our NIH colleagues when setting post-pandemic practices for study section meetings. There are many considerations: Travel is time-consuming, has environmental impacts, and is expensive; review should be an engaging and rewarding experience; meeting format may systematically make it easier, or harder for some scientists to participate; meetings vary considerably in size and so one solution may not fit all. And, hybrid solutions are on the table —for example, with some reviewers in the room and others joining by Zoom, or holding two meetings per year entirely by Zoom and one in person. The paramount consideration is quality of review. CSR is committed to maintaining the quality of review and improving it when possible.
We invite you to look at the details of our analyses thus far and tell us your ideas for how to improve Zoom review meetings. Please send your ideas to the Director of CSR’s Office of Communications and Outreach, Kristin Kramer, or submit a comment.
Acknowledgments: Dr. Hope Cummings for survey work; Lia Fleming and Aditi Jain for quantitative analyses.
I would love to see at least one zoom meeting a year, preferably in the winter when travel is not optimal.
I’ve done 2 Zoom review panels since March. Both have gone very well and I much prefer this to in person meetings. Not needing to travel saves all of us time and not paying for travel and hotel rooms saves precious NIH dollars.
I’ve been on study sections for 19 years and have chaired both a standing study section and many special emphasis panels. It is great that CSR is examining the impact of this natural Zoom review experiment. I agree that reviews have generally going well. I think it will be very interesting to see the results of the scoring analysis. I’d particularly like to see a comparison of the change from initial to final scores between in-person and Zoom meetings. My impression is that assigned reviewers are as engaged in Zoom reviews as in-person reviews. However, it seems that there are fewer questions and less discussion from the unassigned reviewers, which may be difficult to quantify. Even as an East-coaster, I like the hybrid idea. We’d get many of the benefits of Zoom meetings but also retain a greater ability to build relationships and impart the culture of a study section to new members.
I am happy to see NIH evaluating the process of Peer review, and the opportunity the Zoom vs no-Zoom brings to the analysis. I agree that reviewing by zoom is an efficient way to go in the future (barring technical issues, that are readily resolved). Once we know the impact on scoring, that will help to discern whether that aspect (the final outcome) is shifted. If both formats produce the same spread in scoring over a year-two year evaluation period, my instincts are to go for alternate in person meetings for all the reasons pointed out already — namely better interpretation of the reviewer’s enthusiasm for a proposal, better able to judge whether it is time to amplify or reduce discussion, and of course the important comradery that makes the process collaborative, fun and engaging. It would be interesting to see how the distribution of responses varies with prior service on study sections, and academic rank/funding history of the reviewer.
I agree with the above posts highlighting the benefits of each format. Perhaps a hybrid approach would maximize cost/time savings while maintaining relationships among reviewers. Perhaps once a year face-to-face and twice a year Zoom?
I agree with the previous comments that although I miss the face-to-face interaction I don’t miss the 2 days of travel and jet lag from the west coast. I think as we all get more familiar with Zoom and other online interaction these will be even better in the future. The future is now!
I have been on NIH grant review committees for 25 years. As part of my professional work, I have been using Zoom and its predecessors for many years prior to the pandemic. I can definitely see the advantages to continuing on Zoom, or other platforms, in terms of money savings and also time savings (travel time to DC). Nonetheless, I do find the in-person meetings preferable, for primarily two reasons. First, I do think that the reviews are more in-depth when we are in-person, the online meetings become a little mechanical, less depth and discussion, in my perception. Second, for me individually, I appreciate the opportunity to meet with other reviewers, talk about things informally (I don’t particularly mean proposals),network, and that is partly what motivates me to be involved in the reviews, which are obviously quite time consuming.
Re the hybrid options, I would quite strongly avoid any meetings where some are present in person and others join via zoom. I’ve been participating in such hybrid meetings for years, and it is *very* difficult for the persons not present to have equal participation/contribution/engagement. Just the basic logistics of reading body language, hearing the diverse speakers or the one on the far end away from the mic, catching the banter, nods of agreement, murmurs of disagreement/agreement are very difficult in those half-present-half zooming-in meetings. If all are on zoom, it’s much more equal; everyone gets facial expressions roughly equally, everyone can be heard, there is regular effort and a social norm to better ensure everyone can speak, etc. Alternating meetings, some by zoom, others in person is highly recommended. The in-person meetings build social capital that then facilitates good zoom meetings.
I agree with the analysis that indicates that virtual Zoom reviews seem to provide equivalent review outcomes. Although relative to at-table reviews, virtual reviews require some adjustments to expectations for SROs, reviewers, and program staff, there are also notable advantages. One sees the speaker from several feet away, not from across the room. Physiology breaks are less disruptive, just mute and turn off video. Irregular communication during breaks is less feasible. Travel and lodging is more feasible for busy reviewers. In-room connection issues are less dependent on particular hotel specialists. Costs are about 1/10’th. The one omission that I will miss is the pre-evening or first-evening dinner with Chair and reviewers, although virtual variants for these also seem feasible.
I recently did a panel and thought it went quite well. I agree that people should be asked to keep the video on, though.
Yes, I miss the in person contact, but as a West-Coaster, I greatly appreciated that I did NOT have to budget 2 days of travel to get to and from DC. That travel wears on me more the older I get–and those 8am EST starts for a jet-lagged Californian are really tough. I am more likely to be willing to participate in panels if I can do it without the travel burden.
Also, reducing travel is more responsible in a time of climate challenge.
I would like to thank CSR for this comprehensive survey and discussion of these critical issues. As someone who has chaired multiple study sections using different formats, I would like to emphasize the substantial downside of losing in-person contact with other reviewers. While everyone is doing the best they can with our new reality, we no longer have the opportunity for meeting other reviewers and learning about shared interests that benefit science as a whole, and we lose the general sense (previously clear through visual contact) of whether there is need for more discussion to optimally inform the scoring. I suspect the longer review time is more about technical difficulties and less about greater discussion; my experience has been that discussion on scientific issues is more brief and less substantive than with in-person discussion. While I do recognize that virtual meetings eliminate travel and associated costs, I feel strongly that the upside of that is outweighed by the downsides. I would strongly recommend that (once we eventually return to in-person travel) that at least half of the reviews are in-person. Perhaps alternating between in-person and web formats would be an acceptable middle ground.
I recently chaired a Zoom meeting and I personally think that this is the way to go in the future. Yes I missed seeing old friends and colleagues at the meeting, but I did not miss the Air Travel, the ground connections to the meeting whether it be by Metro or taxi, and the issues associated with being away for 3 days for a one day job. I agree with TG that the chair needs to make certain that they have a good internet connection, I do at home, but could have gone into the office if I did not, but that is an easy fix. I do not care where the money saved goes, as it will all go back into the total NIH pot, and it can be used for better things than travel expenses.
There’s a relatively simple way to measure relative engagement: How much do final scores change from initial in phone vs. Zoom vs. in-person meetings? one might imagine that the more engagement, the smaller the standard deviation in final scores.
Thank you for your comment. We plan to analyze scoring data (e.g. out-of-range scoring; movement from preliminary scores to final scores) next.
One thing to keep in mind is that after the immediate switch to zoom, many standing study sections had already been established in-person suggesting that the in-person relationships and synergy may have carried forward to zoom. These strong interations might be expected to diminish with time if all-zoom review groups were the norm. Simply put, one can maintain relationships through zoom that were previously established in person, at least for a while. However, it is much more difficult to establish the complex and trusted relationships among study section members wholly through e-formats that are needed for top functioning review groups.
If zoom were to be used in the future, the Study Section chair should be at the NIH headquarters with the SRO. More efficient and less dependent on the Chair’s home wifi (which was horrible on the review this week!). Also, where will all the cost savings go? To CSR or the Institutes?