DR Update is published monthly by dr2 Consulting for its clients and potential clients.
Editor and principal consultant: Matthew D. Mitchell, Ph.D.
©2005 Diagnostic Research Design and Reporting
One last check of the macros, a pass through the graphs to make sure all the labeling is consistent, and out the door it goes. The first copies of dr-ROC, have been shipped out. A few additional features have been added, particularly for data checking and additional graph elements, but the program is pretty much as originally announced.
One improvement I was able to make during development was to get more real-time updates. You can switch studies in or out of the analysis without having to re-run the analysis macro (though you may need to redraw graphs, depending on what options you are graphing). This will make it faster and easier to do sensitivity analysis.
I've also added more options for graphing, particularly to show or hide the excluded studies. When you show the excluded studies in the ROC or logit plots, they are marked with an X instead of an O, so you can see at a glance how the two sets of studies compare. On the forest plot, the excluded studies (if you choose to show them) are in a group at the top of the graph, with the included studies below them.
When I got to final testing, I was really pleased to see how fast the program runs, particularly in Windows. Excel was originally written for the Macintosh (I've used it since the very first version), but in the 90s it was completely rewritten using a custom coding environment of Microsoft's, which tends to favor Windows in execution speed. So I was really concerned about speed when I started work on the program (the research tool on which dr-ROC is based was quite slow). But as it became clear the program would be reasonably fast, I could make some changes to its architecture and make the program more convenient to use, which I think is a good trade-off. Even the longest macros take just a few seconds to complete.
Speaking of time, I'm quite confident that dr-ROC will save you time and money compared to other tools for calculating summary ROC curves or diagnostic confidence intervalsăeven the free (but old) software that's in wide use today. The actual calculation is only a small part of the jobădr-ROC does much more for you, from checking data entries to keep you from making silly errors (nothing like learning from experience...) to preparing manuscript-ready tables with just one click. And think about the impression polished and professional graphics will make on peer-reviewers and grant review committees. Investing in dr-ROC will pay dividends quickly.
I've already started on a list of improvements to make in version 1.1, which of course will be sent free of charge to all licensed dr-ROC users. Some additional tutorial material will be added to the manual, I'll streamline a few internal calculations, and I'll add a few more graph options.
Finally, I'd like to thank Madukhar Pai of the University of California Berkeley School of Public Health, and Rebecca Smith-Bindman of UCSF for their suggestions about features to include in the program.
Visit www.dr2consulting.com for full specifications and ordering information.
No role for quality scores in systematic reviews of diagnostic accuracy studies.
Penny Whiting, Roger Harbord, Jos Kleijnen.
BMC Medical Research Methodology 2005, 5:19
http://www.biomedcentral.com/1471-2288/5/19
This article from the MRC at Bristol (UK) ought to spur debate over the standardized checklists that are gaining traction with journal editors and technology assessment specialists. Quite rightly dismayed with the quality of peer-reviewed published research articles, they've come up with systems like STARD and QUADAS to try and assign a numeric score to article quality.
But those scales are based on arbitrary decisions about what is and isn't relevant to "quality" in research. At ECRI, we grappled with this question a lot, usually in the context of a debate over whether we should be using quality scores to weight studies in a meta-analysis or to select studies to include or exclude. Some studies may be good in one respect and bad in another, while others are good in the second and bad in the first. So if your scoring system weights one aspect of quality more highly than another aspect, one of those studies will score high and the other will score low.
Whiting and colleagues have done just thatăto demonstrate this effect using real data. Then they followed through to show that use of quality scores in meta-analysis means that the bottom line result is quite dependent on how you weight different aspects of quality.
Does that mean we should ignore quality? Not necessarilyăthere definitely still are errors in research design and shortcomings in reporting that pose real threats to the validity of research results. That is: some research findings may be artifacts of how a study was conducted and not an accurate reflection of how the test truly performs. Including those studies in our systematic reviews that are the basis of decisions on which test to use in routine practice risks making wrong decisions that worsen care and add unnecessary costs to the health care system.
Ultimately, as with research in quality of treatment studies, it comes down to knowing the truth. If we could somehow by magic know the truth about how a test or treatment performs, then we could measure how the different study quality aspects relate to the accuracy of the study in portraying the truth. Unfortunately, that truth is going to be very hard to come by. That leads us to our funding opportunity of the month...
NIH and AHRQ have reissued their RFA (Request for Applications) on research integrity research. The object is to get empirical data on the relationships between professional standards and the actual conduct of research, with "integrity" defined as "the use of honest and verifiable methods in proposing, performing, and evaluating research and reporting research results with particular attention to adherence to rules, regulations, guidelines, and commonly accepted professional codes or norms." The sponsors are also interested in learning the true costs of misconduct such as duplicate publication and conflict of interest.
This is an R01 grantmaking program, and is open to for-profit as well as academic and non-profit applicants. Direct costs are limited to $175,000 per year for a two year period. The sponsors expect to make 5 to 10 grants under this program. Applicants should submit a letter of intent by August 16, 2005.
In a field like ours, we hardly ever leave the job entirely at the office at the end of the work day. We take home questions, the weighty and the trivial, so we can mull them over at times when there might be less intellectual stimulation, in hopes that a flash of insight comes over us.
It doesn't have to be while mowing the lawn--this column could just as well have been called "Thoughts while hanging out the laundry"--but the point here is to share with you some interesting and stimulating ideas.
One of my consulting clients recently sent me copies of the other external reviews of a recent technology assessment report I had reviewed. Some of the reviews were detailed and insightful while others were cursory at best. Assuming the other reviewers were being paid as I was, the client didn't get her money's worth from some of those reviewers.
This is pretty typical of my experience too. When I was writing technology assessment reports, I got good reviews and bad reviews; and the difference between the two had little to do with whether or not the reviewer agreed with me. Having read scores of external reviews of my technology assessment reports, I saw a wide variation in what the different reviewers thought was a satisfactory job. I think it's because those reviewers didn't always understand the purpose of their review, and treated the draft TA report like a manuscript submitted for publication in a journal.
In other cases, it could have been that the reviewer really didn't have enough time to thoroughly review the report, but either didn't want to admit it and disappoint us. That's consistent with the finding that younger reviewers tend to be more thorough in their reviews than older ones (Kliewer MA et al. AJR June 2005; 184: 1731-1735. http://www.ajronline.org/cgi/content/abstract/184/6/1731), and with the fact that sometimes we had to remind the reviewer several times that we were waiting for his comments.
A referee's review of a journal article is different from an external review of a draft reportănot just in the length of the document being reviewed, but also in the purpose of the review and how it will be used. The referee assists the journal editor in deciding which articles are most worthy of publication. It is not the referee's job to substitute his interpretations for the author's interpretationsăit's simply to judge whether the research methods are valid, whether the results are reported clearly, and whether the subject matter is of sufficient interest to readers of the journal.
The external reviewer has a broader task, since he or she is working for the author rather than the journal. The author is under no obligation to heed the reviewer's advice, but a wise author will take care to respond fully to the reviewer's points. Other readers are likely to have the same concerns as the reviewer.
The external reviewer also has more leeway to constructively criticize the draft, since the stakes aren't as highăthe review isn't going to determine whether or not the report will be published. Of course, if one is working on an internal review for a colleague, and nobody else is going to see the results, then no holds are barred. Constructive criticism often is the most useful kind of external review, because it helps the author anticipate the opposition that will arise once the report is published, and try to respond to those points in advance.
So if you're asked to review a manuscript, be realistic about your schedule, and only accept the task if you can complete it promptly. Then consider the purpose of your review before you start reading and writing. If you're working for the author rather than the journal, don't be afraid to report your disagreements as well as your agreements with the draft.
If you're on the soliciting end, the best thing you can do is be clear about what your expectations are, both in terms of the scope and depth of the review, and your interest in hearing the negative findings as well as the positive ones. Also, respect the reviewer's time. If you're asking for anything more than a brief review, you should offer the reviewer suitable compensation for his or her efforts.
Most bad reviews are the result of miscommunication: either the publisher not being clear about what kind of review is being sought, or the reviewer not being candid about the amount of time he or she can devote to the job.--MDM