← Về thư mục
📄 / / academic-research-skills / skills / deep-research / references / arxiv_api_protocol.md

arXiv API Verification Protocol

Status: v3.11 (#182 Delta 1); #495 ToU-alignment refresh (2026-07) Used by: bibliography_agent, scripts/contamination_signals.py (resolve_arxiv_unmatched) API base: http://export.arxiv.org/api/query Rate limit: arXiv's API Terms of Use (https://info.arxiv.org/help/api/tou.html): no more than one request every three seconds, a single connection at a time. The limits apply to all machines under the caller's control as a whole — multi-machine / multi-client fan-out to overcome them is explicitly prohibited. No polite-pool / higher-tier mechanism. Polite email env var: none (arXiv has no such convention)

Purpose

Adds a fourth bibliographic-index lookup to the cross-index triangulation surface (S2 + OpenAlex + Crossref + arXiv) per spec v3.11 #182 Delta 1. arXiv is the preprint registry of record — strongest coverage for CS / physics / math preprints carrying an arXiv ID. It complements the three published-literature indexes: a citation with a bogus arXiv ID that fails to resolve is positive non-existence evidence, while a published paper without an arXiv ID is legitimately skipped by this resolver (arXiv applicability is ID-gated, not a coverage gap).

Mirrors the structure of crossref_api_protocol.md and openalex_api_protocol.md.

Response format

Unlike the JSON siblings, the arXiv query API returns an Atom 1.0 XML feed. The client parses it with xml.etree.ElementTree; the Atom namespace is {http://www.w3.org/2005/Atom}. A match yields one or more <entry> elements; a miss yields a feed with zero <entry> elements (not a 404). Per-entry fields the client reads:

Query Patterns

Pattern 1: arXiv ID Lookup with Title Cross-Check (primary when an arXiv ID is available)

GET ?id_list={arxiv_id}

Matching rule (mirrors the Crossref/S2 DOI_MISMATCH pattern): ID lookup hits are gated by a 0.70 title cross-check (same SequenceMatcher threshold as the sibling clients). If the resolved entry's title is below threshold → ID_MISMATCH, return None, fall through to title search. An empty feed (non-existent ID) is a miss → None.

Pattern 2: Title Search (fallback on ID-miss / ID_MISMATCH)

GET ?search_query=ti:"{title}"&max_results=5

Matching rule: similarity >= 0.70. When multiple candidates pass, prefer the matching-year tiebreaker via a +0.05 score bonus (year read from <published>).

arxiv_unmatched derivation

true if and only if the citation has an arXiv ID AND the ID lookup either returns an empty feed (miss) or misses the title cross-check, AND the title-search fallback returns no match meeting threshold.

A citation with no arXiv ID is skipped (not unmatched) — the resolver does not run and the caller omits arxiv_unmatched (absent ≠ false, #331). arXiv applicability is ID-gated: a title-only miss against arXiv is a coverage gap for a non-arXiv work, not non-existence evidence, so it must never emit a triangulation signal. The check fires only when obtained_via != 'manual' AND an arXiv ID is present.

Note: the unified lookup_verified summary (#182 Delta 4, a later batch) narrows the existence-gate false to ID-keyed unmatched — a title-only arxiv_unmatched with no resolvable ID is a coverage-gap signal, not fabrication evidence (C-V6(a)). This protocol's arxiv_unmatched boolean is the raw triangulation signal; the narrowing happens at the Delta 4 reducer, not here.

Degradation handling

Condition Action
Empty feed (zero <entry>) Treat as miss — caller falls through to title search / reports unmatched. NOT a degradation.
HTTP 429 (rate limit) Back off 3 seconds (the ToU pacing floor — a sub-3s retry would itself violate the one-request-per-three-seconds pacing), retry up to 3 times. After exhaustion, raise ArxivUnavailable. Throttle anchor refreshed after each backoff.
HTTP 5xx Raise ArxivUnavailable immediately (no retry).
Network timeout (30s default) / URLError Raise ArxivUnavailable.
Malformed XML body (truncated / unclosed mid-stream) Raise ArxivUnavailable (the read/parse narrow-except translates ET.ParseError, OSError, http.client.IncompleteRead). A complete HTML error page is well-formed XML and parses to zero entries — a miss, not a degradation.
ArxivUnavailable raised Caller MUST omit arxiv_unmatched from the entry (absent != false). Other indexes proceed independently.

arXiv-specific notes

Client implementation

See scripts/arxiv_client.py. Class ArxivClient exposes arxiv_id_lookup(arxiv_id, expected_title) and title_search(title, year=None). Both return dict | None (the dict being a projected {title, year} view of one Atom <entry>). Both raise ArxivUnavailable on degradation per the table above.

Cross-references