paper-reader/docs/plans/2026-02-25-unpaywall-and-pdftotext-design.md

Unpaywall + pdftotext — Design

Two improvements to the paper CLI: try free open-access sources before piracy, and try simple PDF text extraction before heavy ML OCR.

Download Pipeline

Priority chain for fetching the PDF:

1. Unpaywall (new) — hit https://api.unpaywall.org/v2/{doi}?email={email}, check best_oa_location.url_for_pdf, download if present
2. LibGen (existing)
3. Anna's Archive (existing)

Unpaywall requires an email address (no API key). Read from UNPAYWALL_EMAIL env var. If unset, skip silently — same pattern as ANNAS_ARCHIVE_KEY.

Conversion Pipeline

Priority chain for PDF-to-markdown:

1. pdftotext (new) — shell out to pdftotext -layout <pdf> -, check output quality
2. marker-pdf (existing) — heavy ML OCR fallback

Quality heuristic for pdftotext output

• Length > 500 characters

• 80% printable ASCII / common unicode (letters, digits, punctuation, whitespace)

If either check fails, or pdftotext isn't on PATH, fall back to marker.

Nix changes

Add poppler_utils to the flake for pdftotext. Include in both the dev shell and the wrapped binary PATH.

Dependencies

No new Rust crate dependencies — reqwest and serde_json already handle the Unpaywall API call. pdftotext is an external binary like marker_single.