Query: work kanban task t_6bd19b51 Initializing agent... ──────────────────────────────────────── ┊ 📋 preparing kanban_show… ┊ ⚡ kanban_sh 0.0s ┊ 📚 preparing skill_view… ┊ 📚 preparing skills_list… ┊ 📚 preparing skills_list… ┊ 📚 skill software-development/hermes-agent-skill-authoring 0.3s ┊ 📚 skills list software-development 0.2s ┊ 📚 preparing skill_view… ┊ 📚 skill research-pipeline 0.1s ┊ 📚 preparing skill_view… ┊ 📚 preparing skill_view… ┊ 📚 preparing skill_view… ┊ 🔎 preparing search_files… ┊ 🔎 preparing search_files… ┊ 🔎 preparing search_files… ┊ 🔎 find *crawl* 0.2s ┊ 🔎 preparing search_files… ┊ 💻 preparing terminal… ┊ 💻 $ ls -la /root/.hermes/profiles/it-ai/skills/ 0.1s ┊ 📖 preparing read_file… ┊ 📖 preparing read_file… ┊ 📖 preparing read_file… ┊ 📖 read SKILL.md L1-220 0.1s ┊ 🔎 preparing search_files… ┊ 🔎 find * 0.2s ┊ 📝 preparing skill_manage… ┊ ⚡ skill_man web-crawl-specialist 0.0s ┊ review diff a//root/.hermes/profiles/it-ai/skills/research/web-crawl-specialist/SKILL.md → b//root/.hermes/profiles/it-ai/skills/research/web-crawl-specialist/SKILL.md @@ -0,0 +1,183 @@ +--- +name: web-crawl-specialist +description: Use when building or tuning web crawlers for source-specific data collection. Choose requests, Playwright, or Selenium, and validate extraction, throttling, robots, and anti-bot handling. +version: 1.0.0 +author: Hermes Agent +license: MIT +metadata: + hermes: + tags: [web, crawling, scraping, requests, playwright, selenium, monitoring, brand-monitor] + related_skills: [blogwatcher, arxiv, ocr-and-documents] +--- + +# Web Crawl Specialist + +## Overview + +Use this skill when you need a crawler that reliably pulls structured data from one or more known sources: news sites, social feeds, public portals, or government pages. + +The goal is not "scrape the page" in the abstract. The goal is to pick the least expensive extraction path that is robust for the specific source, then harden it with rate limits, retry policy, and verification. + +## When to Use + +Use for: +- Building a new source-specific crawler +- Tuning an existing scraper that breaks on layout changes +- Choosing between `requests`, `Playwright`, and `Selenium` +- Monitoring news, brand mentions, press releases, public tenders, or government notices +- Turning messy web pages into a clean dataset with deduping and provenance + +Do not use for: +- Generic browsing or one-off page reading +- Long-form research synthesis without extraction +- Tasks that already have a stable RSS/API feed + +## Core Decision Rule + +Choose the lightest tool that can finish the job: + +| Source shape | Preferred approach | Why | +|---|---|---| +| Static HTML, predictable markup | `requests` + parser | Fast, cheap, easiest to test | +| HTML rendered by JS | Playwright | Full browser, deterministic waits | +| Heavy interaction or anti-bot | Playwright first, Selenium only if needed | Better control and modern automation | +| Public JSON/XHR endpoint behind the page | Direct HTTP call to the endpoint | Less brittle than DOM scraping | +| Multi-page archival crawl | `requests` with retry + pagination loop | Scales better than a browser | + +Rule of thumb: if you can get the data without rendering a browser, do that first. + +## Workflow + +1. Identify the source contract. + - Find the page template, pagination pattern, and whether content comes from HTML, JSON, or XHR. + - Record canonical URLs and stable selectors before coding. + +2. Inspect the actual response path. + - Prefer page source and network calls over visible DOM when possible. + - If the data exists in an XHR response, target that endpoint directly. + +3. Build a thin extractor. + - Parse only the fields you need. + - Keep extraction functions small and source-specific. + - Preserve raw source URL, fetched timestamp, and source type for every record. + +4. Add operational safeguards. + - Respect robots and site terms when applicable. + - Use bounded concurrency. + - Add backoff, jitter, timeout, and retry caps. + - Make crawl windows explicit so the job can resume safely. + +5. Verify on real samples. + - Test against at least 3 representative pages. + - Confirm field counts, date parsing, dedupe keys, and link normalization. + - Compare extracted records to the visible page. + +6. Harden for change. + - Prefer stable attributes, URLs, and semantic containers over brittle class chains. + - Store selector fallbacks in priority order. + - If layout drift is common, add a small regression sample set. … omitted 105 diff line(s) across 1 additional file(s)/section(s) ┊ 📝 preparing skill_manage… ┊ 📝 preparing skill_manage… ┊ 📝 preparing skill_manage… ┊ review diff a//root/.hermes/profiles/it-ai/skills/research/web-crawl-specialist/templates/requests_crawler.py → b//root/.hermes/profiles/it-ai/skills/research/web-crawl-specialist/templates/requests_crawler.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""Requests-based crawler template for static pages or JSON/XHR endpoints.""" + +from __future__ import annotations + +import argparse +import dataclasses +import datetime as dt +import json +import random +import time +from urllib.parse import urljoin, urlparse + +import requests +from bs4 import BeautifulSoup + + +USER_AGENT = "HermesBrandMonitor/1.0 (+contact@example.com)" +TIMEOUT = 20 +MAX_RETRIES = 3 +BACKOFF_BASE = 1.5 + + +@dataclasses.dataclass +class Record: + source: str + url: str + title: str + published_at: str | None + body: str | None + fetched_at: str + + +def fetch(session: requests.Session, url: str) -> str: + last_error: Exception | None = None + for attempt in range(1, MAX_RETRIES + 1): + try: + response = session.get(url, timeout=TIMEOUT) + if response.status_code in {429, 500, 502, 503, 504}: + raise requests.HTTPError(f"retryable status {response.status_code}", response=response) + response.raise_for_status() + return response.text + except Exception as exc: # noqa: BLE001 - template keeps retry logic compact + last_error = exc + sleep_s = (BACKOFF_BASE ** attempt) + random.uniform(0, 0.5) + time.sleep(sleep_s) + raise RuntimeError(f"failed to fetch {url}: {last_error}") + + +def parse_listing(html: str, base_url: str) -> list[str]: + soup = BeautifulSoup(html, "html.parser") + links: list[str] = [] + for a in soup.select("article a[href], h2 a[href], h3 a[href]"): + href = a.get("href") + if href: + links.append(urljoin(base_url, href)) + return sorted(set(links)) + + +def parse_article(html: str, url: str) -> Record: + soup = BeautifulSoup(html, "html.parser") + title_el = soup.select_one("h1") or soup.select_one("title") + title = title_el.get_text(" ", strip=True) if title_el else "" + time_el = soup.select_one("time[datetime]") + published_at = time_el.get("datetime") if time_el else None + body_el = soup.select_one("article") or soup.select_one("main") or soup.body + body = body_el.get_text("\n", strip=True) if body_el else None + if not title: + raise ValueError(f"missing title for {url}") + return Record( + source=urlparse(url).netloc, + url=url, + title=title, + published_at=published_at, + body=body, + fetched_at=dt.datetime.now(dt.UTC).isoformat(), + ) + … omitted 20 diff line(s) across 1 additional file(s)/section(s) ┊ review diff a//root/.hermes/profiles/it-ai/skills/research/web-crawl-specialist/templates/playwright_crawler.py → b//root/.hermes/profiles/it-ai/skills/research/web-crawl-specialist/templates/playwright_crawler.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +"""Playwright crawler template for JavaScript-rendered pages.""" + +from __future__ import annotations + +import argparse +import asyncio +import dataclasses +import datetime as dt +import json + +from playwright.async_api import async_playwright, TimeoutError as PlaywrightTimeoutError + + +@dataclasses.dataclass +class Record: + source: str + url: str + title: str + text: str + fetched_at: str + + +async def fetch_rendered_text(url: str, selector: str, timeout_ms: int = 30000) -> Record: + async with async_playwright() as p: + browser = await p.chromium.launch(headless=True) + context = await browser.new_context(user_agent="HermesBrandMonitor/1.0") + page = await context.new_page() + try: + await page.goto(url, wait_until="domcontentloaded", timeout=timeout_ms) + await page.wait_for_selector(selector, timeout=timeout_ms) + title = await page.title() + text = await page.locator(selector).inner_text(timeout=timeout_ms) + return Record( + source=page.url.split("/")[2], + url=page.url, + title=title, + text=text, + fetched_at=dt.datetime.now(dt.UTC).isoformat(), + ) + except PlaywrightTimeoutError as exc: + raise RuntimeError(f"render timeout for {url}; selector={selector}") from exc + finally: + await context.close() + await browser.close() + + +async def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("url") + parser.add_argument("--selector", default="main") + args = parser.parse_args() + record = await fetch_rendered_text(args.url, args.selector) + print(json.dumps(dataclasses.asdict(record), ensure_ascii=False)) + + +if __name__ == "__main__": + asyncio.run(main()) ┊ review diff a//root/.hermes/profiles/it-ai/skills/research/web-crawl-specialist/references/source-playbook.md → b//root/.hermes/profiles/it-ai/skills/research/web-crawl-specialist/references/source-playbook.md @@ -0,0 +1,54 @@ +# Source Playbook Notes + +Use this reference when converting a new site into a durable crawler. + +## Minimum source profile + +For each source, write down: + +- Source name and owner +- Base URL +- Target record type: article, post, tender, press release, document, comment +- Access method: RSS/API/static HTML/XHR/Playwright/Selenium +- Pagination pattern +- Dedupe key +- Critical fields +- Optional fields +- Known failure modes +- Crawl frequency and freshness requirement + +## Field contract for R&D / Brand Monitor output + +Recommended JSONL fields: + +```json +{ + "source": "example-news", + "source_type": "news", + "url": "https://example.com/article", + "canonical_url": "https://example.com/article", + "title": "Article title", + "published_at": "2026-07-14T08:30:00+07:00", + "author": "Reporter", + "section": "Business", + "body": "Full text or cleaned excerpt", + "language": "vi", + "tags": ["brand", "policy"], + "attachments": [], + "fetched_at": "2026-07-14T09:00:00Z", + "crawl_run_id": "20260714-example-news", + "extraction_version": "example-news-v1" +} +``` + +## Quality gates + +Before shipping a crawler: + +1. Run on 3-10 representative URLs. +2. Save raw samples for regression debugging. +3. Confirm output is valid JSONL or CSV. +4. Verify no duplicate dedupe keys in one run. +5. Check that missing critical fields fail loudly. +6. Confirm retry and timeout behavior with one bad URL. +7. Document the source profile next to the crawler. ┊ 💻 preparing terminal… ┊ 💻 $ git status 0.1s ┊ 💻 preparing terminal… ┊ 💻 preparing terminal… ┊ 💻 preparing terminal… ┊ 💻 preparing terminal… ┊ 💻 preparing terminal… ┊ ✔ preparing kanban_complete… ┊ ⚡ kanban_co 0.0s ╭─ ⚕ Hermes ───────────────────────────────────────────────────────────────────╮ Done. I created the new skill at: - /opt/ai-os/products/ceo/profiles/it-ai/skills/research/web-crawl-specialis t/SKILL.md - /opt/ai-os/products/ceo/profiles/it-ai/skills/research/web-crawl-specialis t/templates/requests_crawler.py - /opt/ai-os/products/ceo/profiles/it-ai/skills/research/web-crawl-specialis t/templates/playwright_crawler.py - /opt/ai-os/products/ceo/profiles/it-ai/skills/research/web-crawl-specialis t/references/source-playbook.md What’s in it: - source-specific crawl strategy selection - requests vs Playwright decision rule - source playbook for news/social/government/listing pages - anti-bot, retry, timeout, dedupe, and verification guidance - runnable starter templates for static and browser-rendered crawlers Verification: - Python syntax check passed for both template scripts - Git commit created: b8896842fffc8db15a927c9f1a647b8322730148 If you want, I can next tighten the skill to match your exact brand-monitoring workflow or add a Selenium fallback template too. ╰──────────────────────────────────────────────────────────────────────────────╯ Resume this session with: hermes --resume 20260714_163507_91d4fa -p it-ai Session: 20260714_163507_91d4fa Duration: 4m 39s Messages: 48 (1 user, 46 tool calls)