Embargoed until 11am EST,  October 22, 2013

PubMed Commons:  A system for commenting on  articles in PubMed

The Need for a Comments System

We all read a lot of papers and often have useful things to say about them, but there is no systematic way to do this - lots of journals have commenting systems, but they're clunky, and, most importantly, they're scattered across thousands of sites. Journals don’t encourage critical

comments from readers, and letters to the editor are difficult to publish and given too little space. If we're ever going to develop a culture of commenting on the literature, we need to have a simple and centralized way of doing it.

Pat Brown, Mike Eisen and David Lipman

Last year, I approached my Stanford colleague Pat Brown, a founder of PLOS, with the idea of creating a site where scientists could comment on ANY published research article - something like comments on movies at Internet Movie Data Base (IMDB) or comments on books and other products at Amazon. Pat said that he been discussing similar ideas with his PLOS co-founder Michael Eisen, and that they felt strongly that a standalone site would be unlikely to work because it would not get enough traffic. They felt that the best way to develop a successful culture of commenting on science papers would be to make this an option at PubMed.  Pat introduced me to David Lipman, the Director of the NCBI (the home of PubMed), who said that the idea has been raised many times in the past, and that he was open to implementing such a system if I could demonstrate broad support in the community.

So I organized a group of 34 team leaders,  representing diverse scientific fields. They recruited teams of prominent researchers in their fields - 250 in all, who were committed to the idea. David took the idea to the NIH leadership,  who approved the development of a pilot commenting system called PubMed Commons. The team of scientists I assembled agreed to beta test the system during development and to provide feedback on its design and operation.  

Who should be able to post comments?

A central issue for PubMed Commons was  the question of who should be able to post comments.  One would like the system to be inclusive as possible but many scientists would not be interested in posting comments in a system with a high proportion of irrelevant or uninformed comments.  NIH also needed a rule for who could post that would be pretty clear cut and not based on e.g. some judgment of the experience or knowledge of the participants.  The decision was made that comments could only be posted by authors of papers in PubMed. This would make the situation symmetric in that all people who comment can have their own work commented on.  It would also include a large number of potential participants and would meet NIH’s need for something unambiguous.  Unfortunately it would leave out many people who could add valuable input, including many graduate students, patient advocates, and science journalists.  I'm a little worried about this restriction, as I want to make the system open to as many users as possible. But hopefully that is a pretty wide net, and it may be widened further in the future.  And a group commenting feature to be described below could help improve inclusiveness.

Anonymous comments allowed?

One big issue that we have faced was the question of whether anonymous comments should be allowed. After much discussion, the group remained deeply split on this issue.  Those wanting anonymous posts were concerned that many scientists, especially junior researchers, would be reluctant to make critical comments.  But those opposed to anonymous comments believed that the quality of interchange would be higher if commenters were required to identify themselves.  In the end, these differences weren’t really resolved and the decision was to start without anonymous comments and re-evaluate after the system had been fully public for a while.  While debating this issue various proposals were put on the table for ways to allow participants to review and essentially sponsor the anonymous post of another participant.  

Group comments

Gary Ward, an active member of the lead user group, was very keen on using PubMed Commons to post comments from a journal club for a class he participates in  at  the University of Vermont. He proposed that there should be some way for PubMed Commons to accommodate comments posted by a group.  David Lipman noted that group comments would also be a way to allow participation by a wider range of commenters:  A group could be initiated by a regular PubMed Commons participant (i.e. was an author of a paper indexed in PubMed), giving it a title, short description, and list of participants and then posting comments on their behalf.  While a group comment could be submitted by a particular group member, in many cases, they would reflect the consensus of the group and such collective comments  could  be quite valuable.

PubMed Commons is here!

The NCBI team developed a working version of PubMed Commons earlier this summer and I posted the first comment in the closed pilot on June 17.  Since then the user group has noted bugs and made a number of requests for modifications.  Jonathan Dugan of PLOS labs pulled together members of the publishing world for strategic advice, and has provided many valuable suggestions about the design of the system.  Hilda Bastian,  the editor of PubMed Health and a blogger at Scientific American rallied the community’s science bloggers to help get the word out. The current system is pretty simple - after registering you’ll see the PubMed Commons landing page which has all the most recent comments and links for information on how to use the system.  When you’re signed in you’ll see below each PubMed record  a box for posting comments or replies to existing comments as well as a place to indicate that an existing comment or reply was useful.  There are instructions for how to specify simply formatting of a comment and if you cite another PubMed record in your comment, there are links back from that cited paper to your comment.

We believe the system is now ready for a wider range of participants.  If you’ve been funded by an NIH Extramural grant (or in the NIH Intramural program), NIH has the information it needs to get you into PubMed Commons automatically.  Once you’re a registered participant, you can invite other published scientists to join.  NCBI is investigating ways to open Commons up directly and automatically to more groups of published scientists but if new participants invite their colleagues, the network effect could broaden membership and expand participation dramatically.

The system will still be in a closed pilot mode where only registered participants can see the posted comments but NIH leadership will be evaluating the closed pilot with the hope of making all comments visible to all users of PubMed. All comments are covered by Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/ ) and if the decision is to make the system fully public, NCBI will provide an API so that other groups (e.g. publishers or other information resources) can make these comments useful to the community.

PubMed Commons was released for broad use on October 22, 2013

 

See the PubMed Commons Landing page 

I am very excited by this initiative, and hope it can improve the quality of scientific

interchange the community.

Rob Tibshirani, Stanford University

tibs@stanford.edu

http://www-stat.stanford.edu/~tibs/