Double Sampling with Multiple Imputation to Answer Large Sample Meta-Research Questions: Introduction and Illustration by Evaluating Adherence to Two Simple CONSORT Guidelines
Patrice L Capers, Andrew William Brown, John Dawson et al. · Frontiers in Nutrition · 2015
BACKGROUND: Meta-research can involve manual retrieval and evaluation of research, which is resource intensive. Creation of high throughput methods (e.g., search heuristics, crowdsourcing) has improved feasibility of large meta-research questions, but possibly at the cost of accuracy. OBJECTIVE: To evaluate the use of double sampling combined with multiple imputation (DS + MI) to address meta-research questions, using as an example adherence of PubMed entries to two simple consolidated standards of reporting trials guidelines for titles and abstracts. METHODS: For the DS large sample, we retrieved all PubMed entries satisfying the filters: RCT, human, abstract available, and English language (n = 322, 107). For the DS subsample, we randomly sampled 500 entries from the large sample. The large sample was evaluated with a lower rigor, higher throughput (RLOTHI) method using search heuristics, while the subsample was evaluated using a higher rigor, lower throughput (RHITLO) human rating method. Multiple imputation of the missing-completely at-random RHITLO data for the large sample was informed by: RHITLO data from the subsample; RLOTHI data from the large sample; whether a study was an RCT; and country and year of publication. RESULTS: The RHITLO and RLOTHI methods in the subsample largely agreed (phi coefficients: title = 1.00, abstract = 0.92). Compliance with abstract and title criteria has increased over time, with non-US countries improving more rapidly. DS + MI logistic regression estimates were more precise than subsample estimates (e.g., 95% CI for change in title and abstract compliance by year: subsample RHITLO 1.050-1.174 vs. DS + MI 1.082-1.151). As evidence of improved accuracy, DS + MI coefficient estimates were closer to RHITLO than the large sample RLOTHI. CONCLUSION: Our results support our hypothesis that DS + MI would result in improved precision and accuracy. This method is flexible and may provide a practical way to examine large corpora of literature.
Cited by 3 works. doi:10.3389/fnut.2015.00006 Open-access PDF
Authors
References
- Improving the quality of reporting of randomized controlled trials. The CONSORT statement
- Improving the Quality of Reporting of Randomized Controlled Trials
- CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials
- More Informative Abstracts Revisited
- An introduction to modern missing data analyses
- More Informative Abstracts Revisited
- Planned missing data designs in psychological research.
- Power and money: Designing statistically powerful studies while minimizing financial costs.
- Adoption of structured abstracts by general medical journals and format for a structured abstract.
- Using Crowdsourcing to Evaluate Published Scientific Literature: Methods and Example