name: public-records-osint description: Investigate public records across OSINT sources. version: 0.1.0 author: Hermes platforms: [linux, macos, windows] metadata: hermes: tags: [Osint, PublicRecords, Investigation, DueDiligence]

Public-Records OSINT Investigation

Use this skill to investigate people, companies, contracts, sanctions, property, courts, archives, and news using public records. It focuses on entity resolution, cross-link analysis, timing correlation, and evidence chains. It does not do generic web research or private-data collection, and it stays Python stdlib only.

When to Use

• "follow the money"
• corporate due diligence
• sanctions screening
• pay-to-play investigation
• property ownership lookup
• litigation history
• entity resolution across naming variants
• evidence-chain construction
• "what's been said about X"

Prerequisites

• Python stdlib only; no third-party install required.
• Optional source credentials / headers:
• SEC_USER_AGENT
• SENATE_LDA_TOKEN
• OPENCORPORATES_API_TOKEN
• COURTLISTENER_TOKEN
• HERMES_OSINT_UA
• Optional cache root:
• HERMES_OSINT_CACHE

How to Run

Invoke through the terminal tool.

1. Read the source wiki entries under SKILL_DIR/references/sources/ and the VPS fallback reference under SKILL_DIR/references/vps-fallback-sources.md.
2. Fetch the relevant datasets with the stdlib scripts in SKILL_DIR/scripts/.
3. Resolve entities across CSVs with entity_resolution.py.
4. Optionally run timing correlation with timing_analysis.py.
5. Build the final evidence chain with build_findings.py.

Quick Reference

• python3 SKILL_DIR/scripts/fetch_sec_edgar.py --cik 0000320193 --types 10-K,10-Q --out data/edgar_filings.csv
• python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" --fy 2024 --out data/contracts.csv
• python3 SKILL_DIR/scripts/fetch_senate_ld.py --client "EXAMPLE CORP" --year 2024 --out data/lobbying.csv
• python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --out data/ofac_sdn.csv
• python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" --out data/icij.csv
• python3 SKILL_DIR/scripts/fetch_nyc_acris.py --name "SMITH, JOHN" --out data/acris.csv
• python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" --jurisdiction us_ny --out data/opencorporates.csv
• python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Smith v. Example Corp" --type opinions --out data/courts.csv
• python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" --match host --collapse digest --out data/wayback.csv
• python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" --out data/wp.csv
• python3 SKILL_DIR/scripts/fetch_gdelt.py --query '"Example Corp"' --timespan 1y --out data/gdelt.csv
• python3 SKILL_DIR/scripts/entity_resolution.py --left data/lobbying.csv --left-name-col client_name --right data/contracts.csv --right-name-col recipient_name --out data/cross_links.csv
• python3 SKILL_DIR/scripts/timing_analysis.py --donations data/lobbying.csv --donation-date-col filing_date --donation-amount-col income --donation-donor-col client_name --donation-recipient-col registrant_name --contracts data/contracts.csv --contract-date-col award_date --contract-vendor-col recipient_name --cross-links data/cross_links.csv --permutations 1000 --out data/timing.json
• python3 SKILL_DIR/scripts/build_findings.py --cross-links data/cross_links.csv --timing data/timing.json --out data/findings.json

Procedure

1. Identify which sources apply.
2. SEC EDGAR for corporate filings.
3. USAspending for federal contracts.
4. Senate LD-1 / LD-2 for lobbying disclosures.
5. OFAC SDN for sanctions.
6. ICIJ Offshore Leaks for offshore entities.
7. NYC ACRIS for deeds and mortgages.
8. OpenCorporates for corporate registries.
9. CourtListener for opinions and dockets.
10. Wayback for historical captures.
11. Wikipedia + Wikidata for narrative and structured facts.

12. GDELT for global news monitoring.

13. Acquire data.

14. Invoke the matching fetch_*.py script through the terminal tool.
15. Keep outputs normalized as CSV with a header row.

16. Treat 0-row results with warnings as meaningful when the target simply is not covered by that source.

17. Resolve entities across sources.

18. Normalize names before matching.
19. Treat exact matches as high confidence.

20. Treat fuzzy and token-overlap matches as candidates, not conclusions.

21. Optionally test timing correlation.

22. Use timing_analysis.py when event timing near awards or filings matters.

23. Treat statistical significance as a lead, not proof.

24. Build the findings JSON.

25. Use build_findings.py to emit findings with evidence[] and sources[].
26. Every claim should trace back to a source row.

Pitfalls

• Anti-Hallucination (CRITICAL): If a source is unreachable (429, 403, timeout), report the access failure honestly in the findings. Do NOT synthesize, guess, or "try to create" a summary from LLM knowledge to fill the gap. State what you could not access, why, and what data is therefore missing. The user explicitly forbids fabrication.
• VPS/Contabo IP blocks: Wikimedia (403), Google Scholar (captcha), GDELT (429), Wayback (429), Startpage, DuckDuckGo all aggressively block datacenter IPs. When blocked:
• Add time.sleep(2) between requests.
• Split multi-entity queries into sequential single-entity calls.
• Fall back to Google News RSS (most resilient source from VPS IPs).
• Use Crossref API (https://api.crossref.org/works/{doi}) and Unpaywall API (https://api.unpaywall.org/v2/{doi}?email=...) for academic papers — these have no bot detection.
• Google News RSS links are redirects: Links from Google News RSS go through news.google.com/rss/articles/... and must be resolved to direct URLs. If redirect resolution is blocked, declare this limitation in the report rather than presenting redirect URLs as direct sources.
• Mandatory Workspace Assignment: When claiming a Kanban task, ALWAYS assign it to a workspace folder using hermes kanban claim <task_id> --workspace <path> or ensure the worker environment is mapped to the task workspace immediately. NEVER claim a task without an explicit workspace assignment.
• Tailscale Funnel URLs: Deliver all user-facing deliverables (PDFs, Markdown reports) using the Tailscale Funnel URL format https://vmi3427693.tail8c1aaf.ts.net/files/<absolute_path>. Do not use local IP or port-based links.
• Entity name noise: Common acronyms (e.g. "FUV") may collide with unrelated entities (Arcimoto FUV electric vehicle, Ferrari FUV). Always filter results by context keywords (country, domain) before including in findings.
• Do not use this skill for general web research; use web_search or web_extract instead.
• Do not use it for domain / infrastructure OSINT; use the domain-intel skill instead.
• Do not use it for academic literature behind paywalls; use Unpaywall/Crossref first, then arxiv skill for preprints.
• Fuzzy matches are leads, not facts.
• Timing significance does not imply wrongdoing.
• Public records can still be stale, redacted, or incomplete.
• Some sources rate-limit aggressively; honor their quotas and headers.
• US federal campaign finance is intentionally out of scope here.

Verification

Run python3 SKILL_DIR/scripts/build_findings.py --cross-links data/cross_links.csv --timing data/timing.json --out data/findings.json and confirm each finding contains an evidence[] trail.