How to Fact-Check an AI Claim About a Chinese Herb, Step by Step

How to Fact-Check an AI Claim About a Chinese Herb, Step by Step

The gap between "sounds right" and "checks out"

Ask an AI assistant about a Chinese herb and you will get back something fluent, organised, and plausible. Some of it will be correct. Some of it will be correct about a later period and wrong about the text it names. And some of it will be an error that has been copied between sources for so long that it now reads as common knowledge.

The problem is that all three arrive in the same confident tone. You cannot tell them apart by reading. You can only tell them apart by checking — and checking is a procedure, not an instinct.

We have written before about why traceability matters in the AI era. This piece is the other half: not the argument, but the walkthrough. Below are three claims of exactly the kind an AI assistant produces about herbs — plausible, specific, the sort of sentence that would pass unchallenged. Each one gets run through the same six steps. They come out with three different verdicts.

Two things this article is not. It is not about any particular AI product, and it does not quote one — the claims below are written as generic examples of the shape of answer these tools produce, not as the output of any named system. And it is not about whether any herb works: every check here is about provenance and attribution — who said this, in which book, in which century — never about effects.

The procedure

Six steps, the same every time:

1. Isolate the claim. Reduce it to one checkable assertion. "Ancient texts say X" is not checkable; "the Mingyi Bielu says X" is.
2. Find the entry. Which herb, in which source.
3. Find the section. Classical entries are structured — 释名 (names), 集解 (collected explanations), 气味 (nature and flavour), 主治 (indications). A claim about a name is not verified from the indications section.
4. Name the attributing voice. Almost nothing in the Bencao Gangmu is anonymous. Every line carries a 〔…曰〕 tag telling you who said it.
5. Date the layer. Is this from the base classic, or from a commentator writing a thousand years later? Both are "classical." They are not the same claim.
6. Look for a correction. The classics argue with each other. If the claim was disputed, the dispute is usually still on the page.

If you would rather see the procedure applied than described, here it is three times.

Claim 1: "Licorice was known as guo lao, the 'state elder'."

Step 1–3. The claim is about a name, so the section to open is 【释名】 in the gan cao (甘草) entry — not the indications.

Step 4. There it is, and it is attributed: 国老, tagged to the Mingyi Bielu. So the name is real and it is old. Tao Hongjing then explains the metaphor:

國老即帝師之稱，雖非君而為君所宗，是以能安和草石而解諸毒也。

"Guo lao is the title of an emperor's teacher: not himself the sovereign, yet the one the sovereign defers to." Zhen Quan (甄权) gives a parallel account, saying that among all drugs gan cao serves as sovereign, and that because it 「調和眾藥有功」 — "has merit in harmonising the many drugs" — it carries the title of guo lao.

Step 5. The name is in the Bielu layer; the explanations are commentary from Tao Hongjing (5th–6th century) and Zhen Quan (Tang). Different layers, different weight.

Step 6. No correction — the commentators agree here.

Verdict: confirmed. But notice what the check added. The original claim said "was known as." The record says something more precise: the name appears in a specific text, and the reasons given for it are the reasoning of two named later commentators, offered as their explanation of a metaphor — not as a property of the plant that anyone measured. An AI answer that renders this as "licorice harmonises other herbs" has quietly converted an attributed 6th-century gloss into a flat statement of fact. The name checks out. The upgrade from "Tao Hongjing explained the title this way" to "this is what it does" is the part that does not.

Claim 2: "E jiao is a gelatin made from donkey hide."

This one is interesting because almost every modern source will tell you the same thing, and it is true today.

Step 1–3. Open the e jiao (阿胶) entry, 【集解】.

Step 4–5. The oldest layer says something else entirely:

阿膠出東平郡東阿縣，煮牛皮作之。

The Mingyi Bielu: e jiao "comes from Dong'e county in Dongping commandery, made by boiling ox hide." Not donkey. Su Song, in the Song dynasty, does prefer donkey hide — 「其膠以烏驢皮得阿井水煎成乃佳尔。」, the gelatin is best "made from black donkey hide decocted with Dong'e well water" — but in the same passage he notes that both were considered usable:

《本經》阿膠，亦用牛皮，是二皮可通用。

"The Ben Jing's e jiao also used ox hide; the two hides are interchangeable." He adds that the ox-hide gelatin of his own day, huang ming jiao (黄明胶), was no longer well made and so was not fit for medicinal use — a comment about manufacturing quality, not about the original definition. Then Li Shizhen states the whole shift in one line:

大抵古方所用多是牛皮，後世乃貴驢皮。

"On the whole, what the ancient formulas used was mostly ox hide; it was later ages that prized donkey hide."

Step 6. No correction is needed, because nothing was hidden. The texts recorded the change as it happened.

Verdict: true of the later record, false of the base classic. The claim is not a hallucination — it describes the material accurately as of roughly the Song dynasty onward. What it loses is the date. "E jiao is donkey-hide gelatin" presented as a timeless fact silently overwrites the Bielu's ox hide, erases Su Song's note that both were used, and drops Li Shizhen's explicit 古方 / 后世 contrast. This is the most common failure mode in AI answers about the classics, and the hardest to spot: not a false statement, but a true statement with its century removed. The only defence is step 5.

Claim 3: "The Erya identifies ling, 'the great bitter', as licorice."

This one has the strongest pedigree of the three. It has a named ancient source, a named commentator, and it circulated in the materia medica literature for centuries.

Step 1–3. Back to gan cao, 【集解】. Su Song is the one who brings the Erya in: he cites 「蘦，大苦」 and Guo Pu's (郭璞) gloss that ling "resembles rehmannia," works through the Book of Songs line 「采苓采苓，首陽之巔」, and observes that Shouyang mountain is near where gan cao grows. But even Su Song is uneasy — he ends by noting that what earlier scholars described for leaf and stem is 「與今全別」, completely different from today's plant, and asks whether the species might not be the same.

Step 6 — and this is where it breaks. Li Shizhen relays Shen Kuo's (沈括) Mengxi Bitan, which rejects the identification outright:

郭璞之注，乃黃藥也，其味極苦，故謂之大苦，非甘草也。

"Guo Pu's gloss is huang yao (黄药); its taste is extremely bitter, which is why it is called 'the great bitter' — it is not gan cao." Li Shizhen then weighs the two and rules:

郭說形狀殊不相類，沈說近之。

"Guo's description of the form does not resemble it at all; Shen's account is closer." He notes too that Kou Zongshi's Yanyi had repeated the identification without flagging the problem — an early record of an error being propagated by citation.

Verdict: no basis, and the classical record already caught it. The claim traces to a real gloss by a real commentator on a real ancient text — which is exactly why it survived. It is also wrong, and it was refuted in the eleventh century, and the refutation is printed alongside it. A source that only preserved "the Erya says ling is the great bitter, glossed by Guo Pu" would be reproducing an error with an impeccable-looking citation attached. Only a record that keeps Shen Kuo's correction and Li Shizhen's adjudication and the note about Kou Zongshi repeating it lets you see the whole shape of the mistake.

Notice the deeper point: the classical record is not a pile of assertions to be trusted or distrusted wholesale. It is an argument, conducted over centuries, with corrections attached. Fact-checking against it is not "look up the answer" — it is reading the argument.

What the three verdicts have in common

Three claims, all fluent, all specific, all the sort of thing that reads as authoritative. One survives intact. One survives with a date attached that changes its meaning. One does not survive at all — and would not have been catchable without the correction that a good record keeps next to the error.

None of those verdicts came from knowing more about herbs. They came from steps 4, 5 and 6: name the voice, date the layer, look for the correction. That is the whole method.

Worth stating plainly: this article is about verifying the provenance and attribution of historical claims. It is not advice on using, sourcing, or dosing any herb, and it makes no claim about any substance's effects. MateriaTrace records what sources say and makes no treatment recommendation; any question about a herb belongs with a qualified professional.

Why this is the thing worth building

An AI assistant can produce a paragraph about any herb in a second. What it cannot do, unprompted, is tell you which century its sentence belongs to, which named person said it, or whether someone refuted it eight hundred years ago. Those are precisely the three things that decided all three cases above.

So the useful artefact is not a better-written summary. It is a record where every line still carries its 〔…曰〕 tag, its section, and its source — so that steps 4, 5 and 6 are a click rather than a research project. That is what MateriaTrace is for: not to tell you the answer, but to let you check the claim.

Try it on the entries used here: gan cao and the huang yao it was confused with; e jiao and the oxhide huang ming jiao beside it. For more on why a herb's origin needs the same treatment, read why the classics record where a herb came from; for the same procedure applied to which plant is which, verifying real ginseng against the classics and how to read a classical Chinese medicine text.

If this "every line clicks back to its own source" approach is useful to you, see what else is in MateriaTrace on the homepage.

MateriaTrace is a source-traceable, bilingual knowledge graph of Chinese medicine. It records how classical and modern sources describe each entry — each field traceable to its primary source, contradictions kept side by side — and makes no treatment or usage recommendations. Every line quoted in this article can be traced to its classical source.