← Về thư mục

---

name: public-data-brand-monitoring description: "Build crisis-first, public-data-only brand monitoring systems. Covers WAF by-pass (Google News RSS proxy), source feasibility testing, 4-layer pipeline architecture, storage decisions, and delivery cadence." version: 1.0.0 author: Hermes + CEO Nhi metadata: hermes: tags: [Brand-Monitoring, RSS, Pipeline, WAF, Crisis-Management] related_skills: [research-pipeline, hermes-kanban]

---

Public Data Brand Monitoring

Trigger: User wants to monitor brand health / reputation using only publicly available data and no private API access.

Principles

1. Public data only — never depend on private page API, logged-in content, or shareable credentials.
2. Crisis first — the primary goal is detecting reputation risk before it escalates.
3. Multi-brand by design — folder layout and keyword config must support adding brands without structural changes.
4. Graceful degradation — if a source is temporarily unavailable the pipeline must not crash.

Architecture (4-Layer Pipeline)

[Source Intake] → [Normalize & Dedupe] → [Score & Crisis Gate] → [Store & Deliver]

Layer 1: Source Intake

Each source belongs to one of three fetch strategies: | Strategy | Method | Example | |----------|--------|---------| | Public RSS | feedparser on RSS/Atom feed | Google News RSS, VnExpress RSS | | Public page scrape | requests + HTML parse on tag/search page | VietnamPlus tag, Tuổi Trẻ search | | Search only | Query via search engine (no direct scrape) | site:fulbright.edu.vn via Google News |

Layer 2: Normalize & Dedupe

• Canonicalize URL (strip tracking params)
• Normalize date to ISO8601 (VN timezone)
• Dedupe by URL, then by title+snippet similarity (>85%)
• Tag brand_id via keyword matching

Layer 3: Relevance Score & Crisis Gate

Component	Points
Core entity in title	+30
Crisis trigger keyword	+25
Brand variant match	+15
Core entity in body	+15
Policy context match	+10
Source tier A/B	+5

Crisis Gate — if Negative sentiment + Relevance ≥ 70: HOT ALERT (deliver immediately). All other entries go into the periodic digest queue.

Layer 4: Store & Deliver

• Storage V1: SQLite (primary) + Google Sheet sync (CEO view)
• Archive: Daily compressed .jsonl.gz (raw items) with 90-day DB retention
• Deliver: Telegram thread for hot alerts + Telegram topic for periodic digests + Markdown file archive

WAF Bypass Technique for Blocked Sources

When a target domain returns HTTP 403 (e.g. Azure WAF):

1. Google News RSS proxy — always try first: https://news.google.com/rss/search?q=site:<blocked_domain>&hl=vi&gl=VN&ceid=VN:vi Google's crawler passes most WAFs. Data includes: title, source, published date, link. Verified working (tested against fulbright.edu.vn, fsppm.fulbright.edu.vn).

2. Sitemap.xml — check /sitemap.xml, /sitemap_index.xml, /post-sitemap.xml. WAFs sometimes bypass these. (Did NOT work in test against Azure WAF.)

3. RSS feed guessing — try /feed/, /rss/, /news/feed/, /en/feed/. (Did NOT work against Azure WAF.)

4. Bing News API — alternative search engine with site: query. Free tier 1K queries/month.

5. Do NOT use cloudscraper, cheap rotating proxies, or default-headless Selenium — these are easily detected by Azure WAF and waste time.

Frequency & Cron (Recommended for a Single Brand)

Job	Cadence	Notes
RSS collect	Every 6 hours	feedparser on news sources
Sentinel crisis check	Every 1 hour	Narrow Google News query with crisis keywords
Report digest	Mon/Wed/Fri 09:00 VN	Full sentiment breakdown + alerts
Archive	Daily at 23:00 VN	Compress raw → .jsonl.gz
Cleanup raw	Weekly	Remove raw > 90 days (already archived)

Feasibility Testing Procedure

Before building the full pipeline, test each candidate source:

# 1. HTTP status check
urllib.request.urlopen(url, timeout=8)  # expect 200

# 2. RSS parseability
import feedparser
feed = feedparser.parse(url)          # expect > 0 entries, parsed title/source/date

# 3. Content relevance (tag/search pages)
import re
body = response.read().decode('utf-8', errors='replace')
matches = len(re.findall(r'brand_keyword', body, re.IGNORECASE))  # expect > 0

Save results as JSON for IT to reference. See references/ for a full feasibility test transcript.

Reporting & Notification Standards

Telegram Digest format (Hot Alert / Periodic update)

Avoid sending single-article alerts. Use an aggregated digest format with the following structure: 1. Header: 📢 FULBRIGHT BRAND MONITORING UPDATE + 🕒 Date period: [dd/mm/yyyy] → [dd/mm/yyyy]. 2. Summary: 🧾 Alerts: [count] pending. 3. Item details (grouped by Source): - 📰 Source: [Source Name] (e.g. VnExpress) - 📅 Air date: [dd/mm/yyyy] - 🔗 Article: [Hyperlink Title](URL) (always use hyperlink to minimize text block width) - ✨ Fulbright highlight: Extract a clean 1-2 sentence context snippet containing the keyword. - 📊 Sentiment / Brand health: [sentiment] / [crisis_label] | score [relevance_score] - ⚠️ Discussion: Include brief explanation of the crisis reason (e.g. crisis_trigger:học phí) if triggered.

Output quality pitfalls found in this session

• Google News RSS can surface historical articles from years outside the active monitoring window. Always apply a hard date gate before scoring or alerting.
• Do not trust relevance score alone. Add hard rejects for search pages, tag hubs, and other non-article pages.
• Digest/report rows should carry an explicit air date and a readable highlight (excerpt containing the keyword from the article snippet), not just title and score.
• If a source title is a hub page like #Fulbright or Tin tức, hình ảnh, video clip mới nhất về ..., reject it even if the URL looks parseable.
• Google News Redirect Block: Google News RSS article URLs redirect to consent gates or tracking redirects (e.g. consent.google.com or news.google.com/rss/articles/...). Do not fetch or scrape these URLs directly with simple HTTP libraries. Instead, parse original source URLs safely or switch to native RSS feeds from target domains (like VnExpress or Tuổi Trẻ RSS) to retrieve readable body texts/snippets.
• RSS Snippet Quality: RSS summaries (snippet) might contain HTML junk or simple titles instead of actual content. Always fetch and sanitise the summary/description fields from entries, extract brand keywords dynamically, and fallback to title substrings rather than leaving highlights empty or using static placeholders.

Receiver Channel logic

• For automated reporting threads, do NOT create a new Hermes profile or department if no human assignee exists.
• Use a direct thread mapping inside the project config (report.yaml) for receiver topics.
• Keep the operational work on existing departments like IT or R&D; the Telegram topic is only the delivery endpoint.

Support files

• See references/fulbright-feasibility-test-20260714.md for the source feasibility transcript and failure modes.

Channel Logic

• Receiver Channels: For automated reporting threads, do NOT create new Hermes profiles/departments. Use "Receiver Channels" by mapping the thread_id directly in the project config (report.yaml). This prevents "ghost profiles" from cluttering the system.

Data Filtering (Date Gate)

• Active Window Gate: The pipeline MUST enforce a date filter. Skip/discard any raw items or mentions published before the target operating window (e.g. July 1, 2026).
• Format Sanitization: Parse dates safely (e.g. ISO8601 matching the current year/month). If date parsing fails and the title contains past years (e.g., 2016, 2019, 2025 when checking 07/2026), discard it to prevent historical noise.