name: kb-data-pipeline description: Workflow for extracting, translating, and integrating AI-generated summaries (e.g., from NotebookLM) into structured master markdown files for a knowledge base.

KB Data Ingestion Pipeline

This skill governs the process of moving data from external AI sources into the company knowledge base, specifically when dealing with large volumes of summaries that require translation and strict formatting.

Workflow

1. Authentication Check: Ensure the source tool (e.g., NotebookLM) is authenticated. Use nlm login or a revive script.
2. Candidate Identification: Scan the master file for entries needing repair:
3. Missing English summary.
4. Missing Vietnamese summary.
5. "Authentication Error" or "Authentication expired" markers.
6. Corrupted text (mojibake) in Vietnamese (e.g., characters like Ã, Ä).
7. Strict Batch Processing:
8. Process sources in small batches (3-5 items).
9. Retrieve: Get the latest English summary from the source.
10. Translate: Translate to Vietnamese using a stable LLM provider.
11. Internal Validation: Verify that BOTH English and Vietnamese summaries are present and valid before writing the batch to disk.
12. Atomic Write: Only write the batch to the master file if every item in that batch passed validation. This prevents "half-fixed" states and file corruption.
13. Final Review: Do not mark the task as done after individual batches. Run the entire pipeline, then leave the task in blocked status for a final human audit of the full file.

Technical Implementation Details

LLM Translation (Local Providers)

• Avoid Streaming: When using local providers (e.g., 9router on VPS), set stream=False. Synchronous requests are significantly more stable and avoid parsing errors.
• Retry Logic: Implement exponential backoff retries for translation calls to handle transient Read timed out or HTTP 500 errors.
• Prompting: Explicitly instruct the model to avoid dashes (-) or hyphens in narrative prose (often called the "no-dash rule") to comply with user style guides. This applies specifically to the values/prose fields, not the formatting labels (e.g., - **Tóm tắt (VN):** is correct, but the paragraph following it must not contain dashes).

Data Extraction

• Robust Regex: When extracting IDs from markdown, use patterns that account for bold labels (e.g., \*\*ID:\*\*\s+([^]+)``) to avoidNone` matches.

Support Files

• scripts/verify_master.py: Script to scan for auth errors and mojibake.
• references/sibling-agent-file-conflicts.md: Recovery pattern when another Hermes agent modifies the same master file concurrently.

Pitfalls & Guardrails

• Task Sprawl: Never create fragmented sub-tasks like _FIX or _RV for minor errors. Use a single AUDIT or FINAL task to track the overall repair progress.
• False Positives: Do not rely on grep counts of "Tóm tắt (VN):" to verify completion. A line can exist but contain "Authentication Error" or be empty.
• Mojibake: Watch for encoding errors in Vietnamese text. If Ã or Ä appear in the translation, discard the result and retry.
• Human-in-the-Loop: Never auto-complete an ingestion task. The final deliverable is a file that must be visually verified by the CEO.