From Paper to Tag: How Claims Are Checked

Last updated 2026-11-06

Every research claim on Feedback & Forgetting goes through the same steps: find the original paper, record how much of it could be read, check for retractions and corrections, look for independent replications, then tag the specific claim with what that evidence supports. These are the rules, so you can judge a tag yourself.

Reading a Study File

Articles about a single experiment open with a Study File, a card whose fields always come in this order:

• Experiment: the study's usual name.
• Researcher and year: who ran it and when it was published.
• Subjects: the number and kind of participants.
• Materials and delays: what was learned, and when the test came.
• Design: the comparison in one sentence.
• Main finding: the effect's direction, in the original study's units.
• Replication status: the tag, its scope and its check date.
• Overall support and Consistency across studies: a meta-analysis's bias-corrected direction and size, and whether studies of equal quality agree.
• Validity: any retraction, correction or expression of concern.
• Meaning for learners, within these conditions: what the result suggests for practice, limited to what was actually tested.

The card is built from the same data file you can download, so the two cannot drift apart.

How much of each source was read

Each source in an article is marked with one of three reading levels.

• Full text: the whole paper or book was read. Numbers, quotations, tags and CSV rows can rest on it.
• Abstract: only the published summary was read. The article uses only numbers that appear in the abstract, softens its wording to match, and labels any tag that depends on it “Based on abstracts”.
• Citation only: only the bibliographic record was read. The article can say the study exists, when it appeared and what its title says, but gives no results, numbers or tag.

The short versions of an article (title, short answer, card text, image descriptions, table headings) are never more certain than the body: if the body says “in this sample”, so does the summary.

The five tags

A tag goes on a specific claim, never on the bare name of an effect: “retrieval practice beat rereading on a one-week test, for college students reading prose” can be tagged, “the testing effect works” cannot. Every tag states its scope (population, material, test delay) and the date it was last checked.

[replicated] At least one independent replication, completed and published after peer review, with a matching design and enough precision to detect the original effect, found the same direction, and no replication of equal or higher quality found the opposite or close to nothing. A meta-analysis that still supports the direction after correction for publication bias also qualifies, if no equally good replication points the other way.

[mixed] Replications or meta-analyses of equal quality disagree on direction or size, or the effect appears only under some conditions. The article names those conditions.

[not supported] Suitable replications consistently failed to find the original effect, or a bias-corrected meta-analysis puts it near zero, and no study of equal or higher quality supports the original direction. “Near zero” needs a confidence interval narrow enough to rule out an effect large enough to matter. A result that merely missed statistical significance is not enough.

[single study] No suitable independent replication exists yet: there is only the original study, or only follow-ups whose design does not match.

[retracted] The original paper has been retracted. It is not used as evidence and appears only where the retraction itself is being explained.

Which tag wins

The tags are tried in a fixed order and the first that fits is used: retracted, single study, mixed, not supported, replicated. When evidence of equal quality conflicts, the claim is tagged mixed; not supported is never used to hide a real disagreement. If a bias-corrected meta-analysis is positive but an equally strong replication points the other way, the tag is mixed, and the Study File records overall support and consistency separately.

If a cited paper carries a correction or an expression of concern, the tag gets a note, (correction) or (EoC), linked to the notice, and the article says whether it changed any number quoted.

When no tag is given

When only abstracts could be read and they do not settle the question, the claim is marked “status not assigned: abstract-level evidence” rather than forced into a tag. Claims backed only by a citation never get a tag. When an original study can be known only through another paper’s account of it, the claim gets no tag either: it is marked “status not assigned”, and the article names the paper that gave the account. A paper read in full that tabulates the original results condition by condition counts as evidence in its own right.

Articles that trace where a famous quotation or cautionary tale came from use a Source status field instead of a tag. It records the earliest source that could be read, how much of it was read, and the search date, with values such as “contemporary record read”, “later retelling only” and “not found”. “Not found” means not found as of that date, not proof that no record exists.

A quotation whose original could not be read gets quotation marks only when two or more independent, readable scholarly sources give it the same way; it is then marked “as quoted in” with those sources. Otherwise it is paraphrased.

Weighing studies that disagree

When sources disagree, study types are not ranked mechanically. Each is weighed on four things: whether it tested the same claim (population, material, test delay, outcome); design quality and risk of bias, including random assignment and correction for publication bias; precision, meaning sample size and confidence interval width; and independence from the original authors. One mismatched or imprecise replication does not overturn a meta-analysis of many matching studies, or the reverse. The reasoning goes into the Study File. Textbooks and secondhand summaries are leads, never the only support for a claim.

Preprints and preregistrations

Each cited study is recorded on two counts: finished or not, and through peer review or not. A preregistered plan or an unfinished Registered Report has no results and is never described as if it did. A preprint's results can be described, with a note that it has not been through peer review, but a preprint alone does not change a tag.

Retractions, corrections and expressions of concern

Before an article goes up, every cited DOI is checked for retractions, corrections and expressions of concern in the notices registered with Crossref, which include Retraction Watch records. Study Files and myth reviews are rechecked every 12 months; scheduling advice is compared against new meta-analyses every 18 months. A retraction found later is tagged the same day and logged as a correction.

Charts, figures and data files

Figures from journal articles are never reposted, even when openly licensed; every chart is drawn fresh and does not copy the original's layout. When a paper's tables are readable and openly licensed or public domain, the chart is recomputed from them; when only the abstract was read, it uses only the abstract's figures. Charts built from data carry a Data line naming the paper and DOI.

Data files follow six rules:

1. Figures come only from what could be read; an abstract supplies only its own figures.
2. A copyrighted paper supplies only the few figures a claim needs. Whole tables come only from CC BY or public-domain sources.
3. Columns and row order are this site's own; each row gives the DOI and the table or page.
4. Figures from other publications are marked as quoted values. This site's CC BY license does not cover them.
5. Sources under non-commercial licenses stay out of data files and are only described in the text.
6. Values this site calculated can be included, with the formula and the source of each input.

Each row stands on its own, with the claim, its source, the kind of evidence and the reading level together. Reuse terms are on Handouts.

Quotations and old texts

A work is reproduced at length only when it is in the public domain in both the United States and the European Union, which means checking the death dates of author, translator and illustrator. Works in copyright are quoted briefly (about one sentence, and no more than about 50 words from one work per article) with a page number. Named test materials, such as questionnaire items or puzzle drawings, are not reproduced.

Illustrations

Scene illustrations, such as a desk or a doorway, are labeled Illustration and credited “Generated schematic illustration — not a photograph or historical record.” They carry no numerical meaning; any comparison of size or effect is a chart. They never depict a real person, experiment or historical event; portraits and historical images come only from public-domain or openly licensed sources.

What this site does not do

No medical or clinical advice, and no sleep, supplement or drug recommendations for learning. Persuasion techniques are explained so you can recognize them, never scripted for use. No education-policy commentary and no app reviews. A single unreplicated study appears only inside a wider review of the evidence.

If a tag looks wrong

Send the page, the claim and a DOI to contact@feedbackandforgetting.com (Raise a hand). A changed tag is logged on Erased and rewritten, and the old tag stays on record.