Tutorial 1: Getting started¶
Run it in your browser with nothing to install, or on your own computer: how to run the tutorials.
pubmed-stream searches PubMed Central (PMC) and saves the full text of each matching article as one JSON file, with its metadata, reference list and plain text. This tutorial downloads a small search and looks at everything it produced:
- Download a search and read the summary.
- See what was saved, and what one article contains.
- Turn the downloads into a table and chart them.
- Choose an output format.
- Re-run a search, which only fetches what is new.
- Download one article you already know.
- Rescue a search that finds nothing.
- Read what you downloaded, compactly.
The notebook queries NCBI and takes about a minute.
Your settings¶
This is the only cell you need to edit. Change the values, keeping text between the quotes, then run the whole notebook: Run > Run All Cells in Jupyter, or Runtime > Run all in Colab.
QUERY = '"gut microbiota"[Title] AND frailty[Title/Abstract]' # what to search for; tutorial 2 explains the syntax
MAX_RESULTS = 12 # how many articles to download
ARTICLE = "PMC6683074" # one article to download by its PMC ID, shown on its PubMed page
NCBI_API_KEY = "" # optional: paste your free NCBI key between the quotes to run about three times faster
# Setup: run this cell as it is. In Google Colab it first fetches the tutorial files.
import os
import subprocess
import sys
if "google.colab" in sys.modules:
if not os.path.isdir("/content/pubmed-stream"):
subprocess.run(["git", "clone", "-q", "--depth", "1",
"https://github.com/melhzy/pubmed-stream.git", "/content/pubmed-stream"], check=True)
sys.path.insert(0, "/content/pubmed-stream/tutorials")
import json
from pathlib import Path
import matplotlib.pyplot as plt
import pandas as pd
import tutorial_utils as tu
tu.setup(api_key=NCBI_API_KEY) # work in tutorials/workspace/
tu.style()
import pubmed_stream
from pubmed_stream import search_and_download
print("pubmed-stream", pubmed_stream.__version__)
pubmed-stream 0.2.0
1. Download a search¶
search_and_download searches PMC, then fetches each article's full-text XML in parallel threads, within NCBI's rate limits. Field tags such as [Title] keep a search focused; tutorial 2 covers search design.
Unavailable articles are not errors: their publishers do not allow the full text to be downloaded as XML, so they are counted and skipped.
stats = search_and_download(QUERY, max_results=MAX_RESULTS)
Found 12 PMC IDs for '"gut microbiota"[Title] AND frailty[Title/Abstract]' (total PubMed results: 66) Output directory: publications\gut_microbiota_Title_AND_frailty_Title_Abstract
Progress: 10/12 processed (OK:8 | FAIL:2 | SKIP:0)
2 articles were not available in PMC full-text
Progress: 12/12 processed (OK:10 | FAIL:2 | SKIP:0) ============================================================ Download Summary ============================================================ Keyword: "gut microbiota"[Title] AND frailty[Title/Abstract] Total found: 66 Requested: 12 [OK] Successful: 10 [FAIL] Failed: 2 - Unavailable: 2 - Errors: 0 [SKIP] Skipped: 0 Duration: 4.6s Output directory: publications\gut_microbiota_Title_AND_frailty_Title_Abstract ============================================================
The summary is also returned as a DownloadStats object, for use in code:
print(f"PMC has {stats.total_found} matches; {stats.requested} were requested.")
print(f"Saved {stats.successful}, unavailable {stats.unavailable}, errors {stats.errors}.")
print(f"Success rate {stats.success_rate:.0f}% in {stats.duration_seconds:.1f} s, saved to {stats.output_dir}")
PMC has 66 matches; 12 were requested. Saved 10, unavailable 2, errors 0. Success rate 83% in 4.6 s, saved to publications\gut_microbiota_Title_AND_frailty_Title_Abstract
2. What was saved¶
Each search gets a folder named after the query. It holds one PMC<id>.json file per article and query.json, which records the exact search so tools like the knowledge graph can name it.
folder = stats.output_dir
for path in sorted(folder.iterdir())[:6]:
print(f"{path.name:22} {path.stat().st_size / 1024:6.1f} KB")
print("...")
print(json.loads((folder / "query.json").read_text(encoding="utf-8")))
PMC12797849.json 41.4 KB
PMC12832458.json 53.8 KB
PMC12888429.json 78.0 KB
PMC12899124.json 62.7 KB
PMC13002377.json 47.1 KB
PMC13346421.json 80.1 KB
...
{'query': '"gut microbiota"[Title] AND frailty[Title/Abstract]', 'retrieved': '2026-10-18T17:38:03'}
One article file has four parts: identifiers, the download date, a metadata block, and the plain text of the article.
first = sorted(folder.glob("PMC*.json"))[0]
article = json.loads(first.read_text(encoding="utf-8"))
meta = article["metadata"]
def preview(value):
text = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False)
return tu.short(text, 90)
pd.DataFrame([(key, preview(value)) for key, value in meta.items()], columns=["metadata field", "value"])
| metadata field | value | |
|---|---|---|
| 0 | journal_title | Journal of Health, Population, and Nutrition |
| 1 | journal_nlm_ta | J Health Popul Nutr |
| 2 | journal_iso_abbrev | J Health Popul Nutr |
| 3 | journal | Journal of Health, Population, and Nutrition |
| 4 | pmcid | PMC12797849 |
| 5 | pmid | 41361501 |
| 6 | doi | 10.1186/s41043-025-01184-8 |
| 7 | linked_discoveries_url | https://linkeddiscoveries.ncbi.nlm.nih.gov/413... |
| 8 | title | Association between a dietary index for gut mi... |
| 9 | year | 2025 |
| 10 | month | 12 |
| 11 | day | 8 |
| 12 | pub_date | {"year": "2025", "month": "12", "day": "8"} |
| 13 | authors | ["Luo Liang", "He Kaiying", "Zhang Min", "Wang... |
| 14 | abstract | Background Frailty is an increasing public hea... |
| 15 | keywords | ["Diet", "Gut microbiota", "Frailty", "Sex dif... |
| 16 | references | [{"doi": "10.1016/s0140-6736(19)31786-6", "pmi... |
references lists the cited works that have a PubMed ID or DOI. These become citation links in a knowledge graph. The text field has the title, abstract and body, one paragraph per line, ready for search or NLP:
print(f"{len(meta.get('references', []))} references with a PMID or DOI, for example:")
for ref in meta.get("references", [])[:3]:
print(" ", ref)
print()
print(article["text"][:700], "...")
38 references with a PMID or DOI, for example:
{'doi': '10.1016/s0140-6736(19)31786-6', 'pmid': '31609228', 'title': 'Frailty: implications for clinical practice and public health', 'year': '2019'}
{'doi': '10.1016/s2468-2667(18)30091-4', 'pmid': '29908859', 'title': 'Frailty and pre-frailty in middle-aged and older adults and its association with multimorbidity and mortality: a prospective analysis of 493 737 UK biobank participants', 'year': '2018'}
{'doi': '10.1016/j.ejim.2016.03.007', 'pmid': '27039014', 'title': 'Frailty measurement in research and clinical practice: a review', 'year': '2016'}
Association between a dietary index for gut microbiota and frailty in U.S. adults aged 40 and older: evidence from NHANES 1999–2018
Background
Frailty is an increasing public health concern, and diet is a key modifiable factor in its management. However, it remains unclear whether the dietary index for gut microbiota (DI-GM), which reflects gut-supporting dietary patterns, is associated with frailty. We therefore examined this association in U.S. adults.
Methods
In this population-based cross-sectional study, we used data from the National Health and Nutrition Examination Survey 1999–2018, including adults aged ≥ 40 years. Frailty was measured using a 49-item frailty index. We conducted weig ...
3. All downloads as a table¶
Reading every file into a pandas table makes the corpus easy to sort, filter and chart.
rows = []
for path in sorted(folder.glob("PMC*.json")):
a = json.loads(path.read_text(encoding="utf-8"))
m = a["metadata"]
rows.append({
"pmcid": a["pmcid"],
"year": int(m["year"]) if m.get("year", "").isdigit() else None,
"title": m.get("title", ""),
"journal": m.get("journal", ""),
"authors": len(m.get("authors", [])),
"references": len(m.get("references", [])),
"words": len(a.get("text", "").split()),
})
corpus = pd.DataFrame(rows).sort_values("words", ascending=False)
corpus.assign(title=corpus.title.map(lambda t: tu.short(t, 60)))
| pmcid | year | title | journal | authors | references | words | |
|---|---|---|---|---|---|---|---|
| 6 | PMC13411117 | 2026 | Gut Microbiota and Ageing: Mechanisms, Age-Rel... | Healthcare | 3 | 58 | 11085 |
| 5 | PMC13346421 | 2026 | Gut microbiota associates with frailty in olde... | Nature Communications | 13 | 54 | 9492 |
| 2 | PMC12888429 | 2026 | Exploring gut microbiota mechanisms in frailty... | Nutrition & Metabolism | 11 | 68 | 8607 |
| 7 | PMC13468338 | 2026 | Gut Microbiota-Targeted Nutrition for Healthy ... | Nutrients | 11 | 288 | 7815 |
| 1 | PMC12832458 | 2026 | Correlation between the gut microbiota composi... | Frontiers in Nutrition | 5 | 47 | 5906 |
| 3 | PMC12899124 | 2026 | Composition of the Gut Microbiota in Older Adu... | Nutrients | 10 | 76 | 5823 |
| 9 | PMC13512376 | 2026 | How Gut Microbiota Influence Healthy Aging: Ov... | Geriatrics | 5 | 34 | 5159 |
| 4 | PMC13002377 | 2026 | Dietary index for gut microbiota is inversely ... | Frontiers in Nutrition | 5 | 40 | 5093 |
| 0 | PMC12797849 | 2025 | Association between a dietary index for gut mi... | Journal of Health, Population, and Nutrition | 6 | 38 | 4324 |
| 8 | PMC13472779 | 2026 | Commentary: Correlation between the gut microb... | Frontiers in Nutrition | 4 | 11 | 921 |
Full-text length and reference lists vary a lot between articles. A short text often means a brief report or an article whose body is mostly tables and figures.
fig, (left, right) = plt.subplots(1, 2, figsize=(12, 4), sharey=True)
labels = corpus.pmcid
bars = left.barh(labels, corpus.words, color=tu.BLUE, height=0.62)
tu.bar_labels(left, bars, fmt="{:,.0f}", horizontal=True)
tu.gridlines(left, "x")
left.invert_yaxis()
left.set_xlabel("Words of full text")
left.set_title("Article length")
bars = right.barh(labels, corpus.references, color=tu.BLUE, height=0.62)
tu.bar_labels(right, bars, horizontal=True, zero=True)
tu.gridlines(right, "x")
right.set_xlabel("References with a PMID or DOI")
right.set_title("Linkable references")
plt.tight_layout()
plt.show()
4. Choose an output format¶
Every format saves one JSON file with the same metadata. They differ in what else they keep:
fmt |
Adds | Use it for |
|---|---|---|
"text" (default) |
text: plain text, one paragraph per line |
Reading, search, embeddings, most NLP |
"xml" |
xml: the raw JATS XML |
Section-level parsing, tables, figures |
"both" |
text and xml |
Keeping every option open |
include_text=False drops the text field to save space. Here are the same three articles in each format:
import contextlib
import io
sizes = {}
for fmt in ["text", "xml", "both"]:
with contextlib.redirect_stdout(io.StringIO()): # hide the three download reports
s = search_and_download(QUERY, max_results=3, fmt=fmt, output_dir=f"formats/{fmt}")
files = list(s.output_dir.glob("PMC*.json"))
sizes[fmt] = sum(f.stat().st_size for f in files) / len(files) / 1024
fig, ax = plt.subplots(figsize=(7, 2.6))
bars = ax.barh(list(sizes)[::-1], list(sizes.values())[::-1], color=tu.BLUE, height=0.6)
tu.bar_labels(ax, bars, fmt="{:,.0f} KB", horizontal=True)
tu.gridlines(ax, "x")
ax.set_xlabel("Average file size, KB")
ax.set_title("File size by format")
plt.show()
5. Re-run a search¶
Files that already exist are skipped, so running a search again only downloads new articles. This is how a study stays current, and how an interrupted download resumes. Downloads run several at a time; on an unreliable network, use_concurrent=False fetches them one by one.
again = search_and_download(QUERY, max_results=MAX_RESULTS)
print(f"New: {again.successful}, already downloaded: {again.skipped}")
Found 12 PMC IDs for '"gut microbiota"[Title] AND frailty[Title/Abstract]' (total PubMed results: 66) Output directory: publications\gut_microbiota_Title_AND_frailty_Title_Abstract Progress: 10/12 processed (OK:0 | FAIL:0 | SKIP:10)
2 articles were not available in PMC full-text
Progress: 12/12 processed (OK:0 | FAIL:2 | SKIP:10) ============================================================ Download Summary ============================================================ Keyword: "gut microbiota"[Title] AND frailty[Title/Abstract] Total found: 66 Requested: 12 [OK] Successful: 0 [FAIL] Failed: 2 - Unavailable: 2 - Errors: 0 [SKIP] Skipped: 10 Duration: 2.9s Output directory: publications\gut_microbiota_Title_AND_frailty_Title_Abstract ============================================================ New: 0, already downloaded: 10
6. Download one article you already know¶
A colleague recommends a paper, or a reference list cites one. If it is in PMC, its PubMed page shows a PMC ID such as PMC6683074, and efetch_pmc downloads just that article, into the same kind of JSON file. Its status is success, exists when it was downloaded before, or unavailable when the publisher does not allow downloading.
from pubmed_stream import RateLimiter, create_session, efetch_pmc
api_key = os.environ.get("NCBI_API_KEY")
session = create_session("pubmed-stream-tutorial")
limiter = RateLimiter(0.1 if api_key else 0.34)
ok, status = efetch_pmc(ARTICLE, Path("single_articles"), "text", api_key, session, limiter)
print("Status:", status)
if ok:
one = json.loads((Path("single_articles") / f"PMC{ARTICLE.removeprefix('PMC')}.json").read_text(encoding="utf-8"))
print(f"{one['metadata']['title']} ({one['metadata'].get('year')}), {len(one['text'].split()):,} words")
Status: success Gut Microbiota, Muscle Mass and Function in Aging: A Focus on Physical Frailty and Sarcopenia (2019), 6,067 words
7. When a search finds nothing¶
A search with too many restrictions can find nothing. The command line then exits with code 1, and total_found is 0. Relax one restriction at a time. search_pmc checks each version without downloading anything: it counts the matches and lists the newest titles, so you can see whether they are on topic.
from pubmed_stream import search_pmc
strict = '"gut microbiota"[Title] AND frailty[Title] AND exercise[Title]'
with tu.quiet():
empty = search_and_download(strict, max_results=5)
print(f"{empty.total_found:>6,} articles: {strict}")
session.close()
for looser in ['"gut microbiota"[Title] AND frailty[Title/Abstract] AND exercise[Title/Abstract]',
'"gut microbiota" AND frailty AND exercise']:
print()
print(search_pmc(looser, max_results=3))
0 articles: "gut microbiota"[Title] AND frailty[Title] AND exercise[Title]
5 articles in PMC for: "gut microbiota"[Title] AND frailty[Title/Abstract] AND exercise[Title/Abstract] Newest 3: 2026 PMC13411117 PMID 42512634 Gut Microbiota and Ageing: Mechanisms, Age-Related Diseases, and Therapeutic Perspectives. (Healthcare (Basel, Switzerland)) 2025 PMC12726137 no PMID Blautia genus enrichment following a multidomain intervention and its association with gut microbiota and psychosocial parameters in cognitive frail older adults. (Alzheimer's & Dementia) 2025 PMC12348868 PMID 40805993 The Oral-Gut Microbiota Axis as a Mediator of Frailty and Sarcopenia. (Nutrients)
4,450 articles in PMC for: "gut microbiota" AND frailty AND exercise Newest 3: 2025 PMC12535578 PMID 41116875 The complicated relationship between inflammation and metabolic dysfunction. (Journal of diabetes and metabolic disorders) 2026 PMC13674395 PMID 42890801 Bile acid metabolism and signaling in human and animal health: A One Health perspective. (iMetaOmics) 2026 PMC13672029 PMID 42890639 Exploring family support in relation to depression and nutritional status among older adults. (Frontiers in public health)
Letting exercise appear in the abstract as well as the title finds a few articles, few enough to check each title, and they are about the gut microbiota in older age. The loosest version finds thousands, because it matches the words anywhere in the text, including reference lists; its newest titles are not about the question at all. Tutorial 2 measures this trade-off.
8. Read what you downloaded, compactly¶
Each article file holds a full text, tens of kilobytes. To scan a collection, list_articles returns what you need to choose what to read: identifiers, year, journal, title, abstract and length, newest first, and each article once however many searches found it. most_cited lists the works your downloads cite most, the field's foundations.
from pubmed_stream import list_articles, most_cited
catalogue = pd.DataFrame(list_articles("publications"))
print(f"{len(catalogue)} articles")
catalogue[["year", "pmid", "title", "journal", "words"]].head(8).assign(title=lambda d: d.title.map(lambda t: tu.short(t, 70)))
10 articles
| year | pmid | title | journal | words | |
|---|---|---|---|---|---|
| 0 | 2026 | 42645011 | How Gut Microbiota Influence Healthy Aging: Ov... | Geriatrics | 5159 |
| 1 | 2026 | 42601860 | Commentary: Correlation between the gut microb... | Frontiers in Nutrition | 921 |
| 2 | 2026 | 42588100 | Gut Microbiota-Targeted Nutrition for Healthy ... | Nutrients | 7815 |
| 3 | 2026 | 42512634 | Gut Microbiota and Ageing: Mechanisms, Age-Rel... | Healthcare | 11085 |
| 4 | 2026 | 42420265 | Gut microbiota associates with frailty in olde... | Nature Communications | 9492 |
| 5 | 2026 | 41867676 | Dietary index for gut microbiota is inversely ... | Frontiers in Nutrition | 5093 |
| 6 | 2026 | 41683327 | Composition of the Gut Microbiota in Older Adu... | Nutrients | 5823 |
| 7 | 2026 | 41530755 | Exploring gut microbiota mechanisms in frailty... | Nutrition & Metabolism | 8607 |
pd.DataFrame(most_cited("publications", 5))[["cited_by", "pmid", "year", "title"]].assign(title=lambda d: d.title.map(lambda t: tu.short(t or "", 80)))
| cited_by | pmid | year | title | |
|---|---|---|---|---|
| 0 | 6 | 35468952 | 2022 | The gut microbiome as a modulator of healthy a... |
| 1 | 4 | 32066625 | 2020 | Mediterranean diet intervention alters the gut... |
| 2 | 4 | 26785481 | 2015 | Gut microbiota and aging |
| 3 | 4 | 28407483 | 2017 | Age-Associated Microbial Dysbiosis Promotes In... |
| 4 | 4 | 22797518 | 2012 | Gut microbiota composition correlates with die... |
The same from the command line¶
pubmed-stream search '"gut microbiota"[Title] AND frailty[Title/Abstract]' # size and newest titles
pubmed-stream download '"gut microbiota"[Title] AND frailty[Title/Abstract]' --max-results 12
pubmed-stream download '"gut microbiota"[Title] AND frailty[Title/Abstract]' --format both -o formats
pubmed-stream list publications --abstracts # read compactly
pubmed-stream list publications --cited # the works cited most
The command exits with 0 when articles were saved or already present, 1 when the search found nothing, and 2 when every download failed.
Next: tutorial 2 shows how to design searches that find the right papers.