An influencer cites a study. A supplement brand quotes a paper. A wellness podcast mentions research. Almost none of us click through, read the actual study, and evaluate whether the results support the claim being made. This is not a personal failing — papers are dense, jargon-heavy, and time-consuming.
But you don't need to read every study end-to-end. You need to read the abstract, and you need to know what to look for. A trained clinician can evaluate a study's likely credibility in about 90 seconds using a specific set of criteria. Most of those criteria live in the abstract. This post is the field guide.
By the end you should be able to spot a suspect study within 60 seconds of opening the abstract, and know within another 60 whether the results actually support the claim you saw quoted.
What an abstract is
An abstract is a 200-400 word summary at the top of a scientific paper. It exists so that clinicians and researchers can quickly triage whether a paper is worth reading in full. Modern journals use a structured abstract format with specific sections: Background, Methods, Results, Conclusions.
You can find the abstract of almost any biomedical paper on PubMed at pubmed.ncbi.nlm.nih.gov. Search by the study title or authors and the abstract will appear. Even if the full paper is behind a paywall, the abstract is virtually always free.
The five things to check
1. Sample size
How many subjects were in the study? This is the single most important criterion for interpreting the results. Studies with very small samples can produce results by chance that don't replicate in larger studies.
Rough guidelines by study type:
- Pilot studies: 20-50 subjects. Useful for feasibility and effect-size estimation, not for confirming benefit.
- Small RCTs: 50-200 subjects. Can detect large effects but underpowered for subtle ones.
- Adequate RCTs: 200-1000 subjects. Standard for most clinical trials.
- Large trials: 1000+ subjects. Standard for pharmaceutical approval trials.
If a supplement brand cites a study with 32 participants as proof their product works, treat that as preliminary at best. Small studies are hypothesis-generating, not confirmatory.
2. Control group
What did the treatment group get compared against?
- Placebo controlled: The control group received an inactive substitute (sugar pill, saline injection, sham device). This is the strongest control because it isolates the specific effect of the intervention.
- Active controlled: The control group received a different, standard-of-care treatment. Useful for showing whether the new treatment is better than what's already available.
- Open label / no control: Everyone got the treatment; results are compared to their pre-treatment baseline. This is the weakest design because you cannot separate the treatment effect from placebo, regression to the mean, or natural improvement over time.
Any study without a control group cannot establish that the intervention caused the observed changes. Nearly all supplement marketing that cites "studies" is citing open-label studies, which cannot demonstrate causation.
3. Blinding
Did the participants and researchers know which group got the treatment?
- Double-blind: Neither participants nor the researchers evaluating outcomes knew who got what. The strongest design.
- Single-blind: Participants didn't know, but researchers did. Vulnerable to observer bias.
- Open-label: Everyone knew who got what. Highly vulnerable to placebo and expectancy effects.
Blinding matters because participants who know they got the "real" treatment consistently report better outcomes on subjective measures, even when the objective outcome hasn't changed. This is not lying; it's how human perception works. Blinding removes the effect.
4. Effect size and statistical significance
Not the same thing. Both matter.
Statistical significance (p-value) tells you whether the observed difference is unlikely to be due to chance. The conventional threshold is p<0.05, meaning less than 5% probability the result was random. A study with p=0.03 is "statistically significant" — the result is unlikely to be pure chance.
Effect size tells you how big the difference actually was in real terms. This is separate from statistical significance. In a large enough study, a tiny effect can be highly statistically significant while being clinically meaningless.
Example: a trial of a memory supplement in 5,000 people might find a 0.3-point improvement on a 100-point cognitive test with p=0.001. Highly statistically significant. Also clinically meaningless — nobody notices a 0.3-point change on any real cognitive measure.
The abstract usually gives you both. Look for phrases like "a 15% reduction" or "a 3-point improvement" alongside the p-value. If the abstract emphasizes statistical significance without giving you the actual effect size, be suspicious.
5. Funding disclosure
Who paid for the study? Almost every abstract or the associated paper will disclose this.
- Government / foundation funded. NIH, NIHR, MRC, foundation grants. Lowest conflict-of-interest risk.
- Independent nonprofit funded. Cochrane, various disease foundations. Low COI risk.
- Industry funded (transparent). Pharmaceutical companies fund their own drug trials by necessity. The design should be pre-registered on ClinicalTrials.gov, the analysis should be independent, and the disclosure should be clear.
- Industry funded (opaque). Supplement industry-funded studies with the study authors also being industry consultants. High COI risk. Results systematically more favorable than independent studies.
Industry funding does not automatically mean the study is wrong. But you should give industry-funded studies with subjective endpoints and small samples less weight than large independent trials with objective endpoints.
The specific tells to watch for
1. Pre-registration on ClinicalTrials.gov
Legitimate clinical trials register their design, sample size, and primary outcome measure on ClinicalTrials.gov before enrolling patients. This prevents p-hacking — the practice of running many analyses and reporting only the significant ones. Look for a NCT identifier (e.g., NCT01234567) in the abstract or paper. If none is present, the study was not pre-registered and the results are less reliable.
2. Intent-to-treat analysis
An intent-to-treat (ITT) analysis includes all randomized participants in the final analysis, regardless of whether they completed the treatment. This is the standard for rigorous trials because it accounts for dropouts and non-compliance. Studies that only report results for "completers" or "per-protocol" analyses systematically overestimate effects.
3. Primary versus secondary outcomes
A trial should specify a primary outcome measure in advance. That's the one it was designed to detect. Secondary outcomes are exploratory. If a trial's primary outcome fails but a secondary one shows significance, the marketing will feature the secondary outcome. This is technically legitimate reporting but the interpretation is much weaker.
4. Confidence intervals
A 95% confidence interval gives you the range of plausible values for the effect. "A 15% reduction (95% CI: 12-18%)" tells you the true effect is very likely between 12% and 18%. "A 15% reduction (95% CI: -2 to 30%)" tells you the effect might be anywhere from slightly harmful to very beneficial. Wide confidence intervals mean uncertainty.
5. Number needed to treat
For clinical outcomes (heart attacks, deaths, hospitalizations), the Number Needed to Treat (NNT) tells you how many patients need to receive the treatment for one to benefit. NNT of 10 means 10 people treated for 1 to benefit. NNT of 500 means the treatment is helping a very small fraction. NNT is a much better way to think about clinical relevance than percentage changes.
Common tricks in cited-study marketing
1. The pilot study cited as proof
A supplement brand cites a 30-person open-label pilot study as evidence of efficacy. The study was designed to test feasibility and estimate effect sizes for a future confirmatory trial. It cannot demonstrate that the supplement works. But the marketing treats it as if it does.
2. The rat study extrapolated
Animal studies in rodents or other species are hypothesis-generating. Effect sizes and even mechanisms often don't translate to humans. If a supplement's evidence base is entirely rodent studies, that's preliminary work, not proof of human effect.
3. The in vitro study
"Shown to affect cancer cells in vitro" means "the compound killed cells in a dish." This is at the beginning of the drug discovery pipeline, not the end. Bleach kills cancer cells in vitro. That doesn't mean you should drink bleach.
4. The wrong dose
A supplement contains 100 mg of an ingredient. The cited study used 500 mg per day. The dose in the supplement is subtherapeutic even if the ingredient does what the study showed. This is one of the most common marketing tricks — citing evidence at a dose the product does not deliver.
5. The composite endpoint
A trial defines its primary outcome as "a composite of heart attacks, strokes, and hospitalizations." The composite comes out significant, driven mainly by hospitalizations. The marketing headlines say "reduced heart attacks and strokes," which is not quite what the data showed.
6. Relative versus absolute risk reduction
"A 50% reduction in risk" sounds dramatic. If the underlying event rate is 2% versus 4%, that's a 50% relative reduction and a 2 percentage point absolute reduction. In an event that occurs rarely, absolute reduction matters more than relative. Marketing headlines almost always feature the relative number.
A worked example
Suppose a supplement brand says: "A clinical study showed our product reduced anxiety by 47% (Reference: Smith et al., 2021)."
Open PubMed. Search for the study. Read the abstract.
- Sample size: 24 subjects.
- Design: Open-label, no control group.
- Duration: 4 weeks.
- Outcome measure: Self-reported anxiety on a 10-point scale.
- Baseline mean anxiety: 6.4. Post-treatment mean: 3.4.
- Funding: The supplement manufacturer.
- Pre-registration: None cited.
Verdict: this is a small, open-label, industry-funded study with subjective self-report as the outcome measure. The "47% reduction" is real in the data but likely reflects placebo response, regression to the mean, and expectancy effects in a group that knew they were getting active treatment. It is not evidence that the supplement causes anxiety reduction. It is a hypothesis worth testing in a real RCT.
This whole evaluation took 90 seconds.
Where to find actual high-quality evidence
If you want to evaluate a specific ingredient or intervention seriously, look for:
- Cochrane Reviews at cochranelibrary.com. Rigorous systematic reviews and meta-analyses across most clinical topics.
- Examine.com. Independent evidence summaries for supplements and interventions, with grade ratings for each ingredient-outcome combination.
- PubMed with filters. Filter by "Randomized Controlled Trial," "Meta-Analysis," or "Systematic Review" and by recent publication dates.
- ClinicalTrials.gov. Look up ongoing and completed trials by condition or intervention.
- The USPSTF (US Preventive Services Task Force). Independent evidence-based recommendations for preventive interventions.
The bigger picture
Clinical evidence is not intuitive. Effects that look real in small studies often disappear in larger ones. Interventions that look intuitive often fail in trials. Interventions that look pointless (like taking a small dose of a common vitamin) sometimes turn out to be dramatically effective.
The way to navigate this is not to trust individual studies. It is to look at the aggregate evidence base — multiple large RCTs, systematic reviews, meta-analyses — and give more weight to the aggregate than to any single dramatic result. Cherry-picking one study to support a marketing claim is the industry's favorite tactic. Looking at the aggregate is the corrective.
Vyvata's methodology page lays out how we weigh evidence — systematic reviews and meta-analyses count for more than individual RCTs, pre-registered trials count for more than post-hoc analyses, independent funding counts for more than industry-funded work. The Evidence dimension (25% of the total score) rewards products whose claims align with the aggregate literature and penalizes ones that cite selectively.
The next time you see a supplement or device cite a study, take the 90 seconds. Open PubMed. Read the abstract. Check the five criteria. You will find that most cited studies do not support the claims made about them, and a small number do. The small number is where your money should go.
