📄 / / root / ceo-project / company_kb / notebooklm_auth_guide.md

NotebookLM Auth Guide — SOP & Deep Analysis

NotebookLM Auth Guide — SOP & Deep Analysis

File: company_kb/notebooklm_auth_guide.md
Target: notebooklm-mcp-cli integration at /opt/ai-os/products/ceo/integrations/notebooklm-mcp-cli/
Last updated: 2026-07-12

1. Auth Architecture (How It Works)

1.1 Data Flow

User runs `nlm notebook_list`
       │
       ▼
AuthManager.load_profile("default")
       │
       ├── reads ~/.notebooklm-mcp-cli/profiles/default/cookies.json
       ├── reads ~/.notebooklm-mcp-cli/profiles/default/metadata.json
       │
       ▼
BaseClient.__init__(cookies, csrf_token, session_id, build_label)
       │
       ├── _refresh_auth_tokens()
       │     └── fetch https://notebooklm.google.com/ with cookies
       │     └── if redirect to accounts.google.com → AUTH EXPIRED
       │     └── else → extract SNlM0e (CSRF), FdrFJe (session_id), cfb2h (build_label)
       │
       ▼
RPC call (batchexecute) with:
  - Cookie header (SID, HSID, SSID, APISID, SAPISID, etc.)
  - X-Goog-Csrf-Token header
  - ?f.sid=<session_id>&bl=<build_label> in URL

1.2 Storage Layout

~/.notebooklm-mcp-cli/
├── auth.json                         ← Legacy flat cache (auto-migrated to profile)
├── config.toml                       ← CLI config (auth.browser, default_profile)
├── chrome-port-map.json              ← Maps CDP ports → profile names
├── profiles/
│   └── default/
│       ├── cookies.json              ← Cookie dict/list (can be 50+ cookies)
│       └── metadata.json             ← csrf_token, session_id, email, build_label, last_validated
└── chrome-profiles/
    └── default/                      ← Chrome user-data-dir (real browser profile)
        ├── Default/Cookies           ← SQLite cookie DB (36KB+)
        ├── Default/Login Data        ← Saved logins
        └── ...

1.3 Key Components

Component	File	Role
`AuthManager`	`core/auth.py`	Profile CRUD + cookie serialization
`BaseClient`	`core/base.py`	HTTP client, batchexecute, CSRF auto-refresh
`check_auth()`	`core/auth.py`	Live auth validity check (homepage fetch + RPC probe)
`rotate_google_cookies()`	`core/cookie_rotation.py`	POST to `accounts.google.com/RotateCookies` to refresh short-lived `*PSIDTS` cookies
`run_headless_auth()`	`utils/cdp.py`	Launch headless Chrome, navigate to NotebookLM, extract cookies from persisted profile
`extract_cookies_via_cdp()`	`utils/cdp.py`	Full interactive CDP extraction (launches Chrome, waits for login)
`extract_cookies_via_existing_cdp()`	`utils/cdp.py`	Cookie extraction from already-running Chrome endpoint

Google auth uses layered cookies. The long-lived ones (SID, HSID, SSID, APISID, SAPISID) persist for weeks. The short-lived ones (__Secure-1PSIDTS, __Secure-3PSIDTS, SIDCC) rotate hourly/daily.

The cookie_rotation.py module calls the Google RotateCookies endpoint to refresh short-lived cookies before checking if auth is really expired. This is a non-fatal recovery step embedded in _refresh_auth_tokens().

2. Root Cause Analysis: Current Auth Failure

2.1 Observed State

Chrome headless alive on port 9223 with --user-data-dir=~/.notebooklm-mcp-cli/chrome-profiles/default/
52 cookies cached in auth.json and profiles/default/cookies.json — all required SID/HSID/SSID/APISID/SAPISID present
Profile has metadata with valid csrf_token, session_id, email, build_label
check_auth() returns: valid=False, reason='network_error: ClientAuthenticationError'
Chrome pages all show "Sign in - Google Accounts" login page

2.2 Why It Happens

The Chrome profile (Default/Cookies SQLite DB) has expired cookies that Google no longer accepts.
When BaseClient._refresh_auth_tokens() fetches https://notebooklm.google.com/, the cookies redirect to accounts.google.com login page.
The short-lived *PSIDTS cookies have rotated and the cached ones are stale.
rotate_google_cookies() was called but failed because the underlying session (SID/HSID) is also stale — rotation only works when long-lived tokens are still valid.
Since headless can't render the interactive login page, and the persisted session is expired, recovery loops fail.

2.3 Triggers for Auth Death

Chrome process restart without proper session hydration (most common)
VPS reboot — Chrome process dies, profile persists but session tokens are stale
Google session expiry — Google's ~2 week session timeout
Password change / 2FA re-verification on the Google account
Cookie rotation failure — RotateCookies endpoint returns 403 or 401

3. Optimal Strategy for Stable VPS Auth

The project supports two auth methods: 1. Browser/profile-based (recommended): Use nlm login to authenticate via Chrome → persisted profile → auto-refresh via headless CDP. 2. Manual cookie import (nlm login --manual): Copy-paste cookie headers. Fragile.

We use method 1 with a persistent Chrome headless daemon.

3.2 Architecture

┌─────────────────────────────────────────────────────────┐
│                    VPS (Headless)                        │
│                                                          │
│  Chrome headless ───port 9223─── CDP WebSocket           │
│  (always running)                 │                      │
│       │                            │                      │
│       ▼                            ▼                      │
│  Chrome profile              extract_cookies_via_cdp()    │
│  ~/.notebooklm-mcp-cli/      → profile cookies.json      │
│    chrome-profiles/default/  → profile metadata.json     │
│       │                                                   │
│       ▼                                                   │
│  nlm / MCP server reads disk profile                      │
│                                                          │
│  Auth Refresh Cron:                                      │
│  ┌─────────────────────────────────────┐                 │
│  │ Every 6h: check_auth(live=True)     │                 │
│  │ If expired:                         │                 │
│  │   1. Kill → relaunch Chrome headless│                 │
│  │   2. Run headless auth extraction   │                 │
│  │   3. If still dead → notify admin   │                 │
│  └─────────────────────────────────────┘                 │
└─────────────────────────────────────────────────────────┘

Action	Frequency	Who	Notes
RotateCookies POST	Every request	`BaseClient._refresh_auth_tokens()`	Non-fatal, best-effort
Auth validity check	Every 6 hours	Cron: `auth_refresh.sh`	Homepage fetch + RPC probe via `check_auth()`
Full re-auth (headless)	On auth failure	Cron → `revive_notebooklm_auth.sh`	Requires working Chrome profile
Interactive re-auth	When headless fails	User (via SSH tunnel)	See SOP Section 5

3.4 Why This Beats Alternative Approaches

Approach	Problem	Verdict
Only cookie injection	Cookies expire in hours-days, no rotation	❌ Fragile
nlm login every time	Interactive, blocks automation	❌ Not automated
Persistent headless Chrome	Chrome memory ~250MB but session persists via profile	✅ Best for VPS
Environment variables	Can't rotate cookies, env vars lost on restart	❌ Not persistent
Chrome + CDP + profile	Combines persistence with automation	✅ WINNER

4. SOP: Setting Up NotebookLM Auth on VPS

4.1 Initial Authentication (One-Time Setup)

Step 1: Verify Chrome is Running

# Check Chrome headless on port 9223
curl -s http://127.0.0.1:9223/json/version | python3 -m json.tool
# Expected: returns JSON with "Browser" and "webSocketDebuggerUrl"

If Chrome is not running, start it:

/opt/ai-os/products/ceo/scripts/revive_notebooklm_auth.sh

Step 2: Tunnel Chrome to Your Local Machine

On your local machine:

ssh -N -L 9223:127.0.0.1:9223 root@<VPS_IP>

Step 3: Authenticate via Local Browser

Open Chrome on your local machine
Go to chrome://inspect
Click "Configure..." and add localhost:9223
Find the NotebookLM tab (or open one in the remote VPS Chrome if needed)
If you see the Google login page, log in with your Google account
Complete any 2FA / verification challenges
Verify you reach https://notebooklm.google.com/ in the remote tab

Important: The browser session is running on the VPS. What you see in chrome://inspect is a mirror. The login happens on the VPS, and cookies are saved to the VPS Chrome profile.

Step 4: Extract and Save Auth Tokens

After successful login, run on the VPS:

cd /opt/ai-os/products/ceo
python3 scripts/notebooklm_refresh.py

This script: 1. Connects to Chrome headless CDP on port 9223 2. Extracts all cookies from notebooklm.google.com domain 3. Extracts CSRF token, session ID, build_label from the page 4. Saves to both profiles/default/ and auth.json 5. Runs nlm login --check to verify

Expected output:

✓ NotebookLM page loaded and authenticated
✓ Cookies extracted: 52 cookies
✓ CSRF token: AABr7s... (present)
✓ Session ID: 898100228... (present)
✓ Build label: boq_labs... (present)
✓ Tokens saved to profile
✓ Auth check: VALID

4.2 Verification

# Fast check (disk heuristic)
nlm login --check --no-live

# Live check (actual HTTP request)
nlm login --check

# Test an API call
nlm notebook list

4.3 Routine Auth Refresh (Cron)

The cron job at /etc/cron.d/notebooklm-auth runs every 6 hours:

0 */6 * * * root /opt/ai-os/products/ceo/scripts/auth_refresh.sh >> /var/log/notebooklm-auth.log 2>&1

This script: 1. Calls check_auth(live=True) via Python 2. If valid, runs rotate_google_cookies() as proactive maintenance 3. If invalid, runs the revive script + re-extraction 4. Logs all results

5. SOP: Auth Recovery (When It Breaks)

5.1 Symptom Detection

nlm login --check
# Output: "Authentication expired" or "No cached tokens found"

Check for detailed state:

# Is Chrome running?
curl -s http://127.0.0.1:9223/json/version > /dev/null && echo "Chrome alive" || echo "Chrome DEAD"

# Are cached cookies present?
ls -la ~/.notebooklm-mcp-cli/profiles/default/cookies.json

# What email is configured?
python3 -c "import json; d=json.load(open('/root/.notebooklm-mcp-cli/profiles/default/metadata.json')); print('Email:', d.get('email','?'))"

5.2 Automated Recovery (First Try)

# One-command recovery
/opt/ai-os/products/ceo/scripts/revive_notebooklm_auth.sh

What the script does: 1. Kills any existing Chrome on port 9223 2. Relaunches Chrome headless with the existing profile 3. Waits 10s for Chrome to initialize 4. Navigates to https://notebooklm.google.com/ 5. Checks login status by examining page URL 6. If logged in → extracts cookies via CDP → saves to profile 7. If not logged in → returns with error (needs interactive login)

5.3 Interactive Recovery (When Automated Fails)

If automated recovery says "not logged in", you need interactive authentication:

# Step 1: Make sure Chrome is running
/opt/ai-os/products/ceo/scripts/revive_notebooklm_auth.sh

# Step 2: On your LOCAL machine, tunnel to VPS
ssh -N -L 9223:127.0.0.1:9223 root@<VPS_IP>

# Step 3: Open chrome://inspect on local machine
# Add localhost:9223
# Find the NotebookLM page, check URL

# Step 4: If on login page → log in interactively
# If on notebooklm.google.com → tokens should work

# Step 5: After login, re-extract tokens
python3 /opt/ai-os/products/ceo/scripts/notebooklm_refresh.py

5.4 Emergency Recovery (Cookies from Another Browser)

If headless Chrome is completely broken, extract cookies from your desktop browser:

Open Chrome on your desktop
Go to https://notebooklm.google.com/ and ensure you're logged in
Open DevTools (F12) → Application → Cookies → notebooklm.google.com
Export cookies as Netscape format, or use an extension like "Get cookies.txt"
Copy the cookie file to the VPS
Run: nlm login --manual /path/to/cookies.txt
Verify: nlm login --check

6. Technical Deep Dive: Auth Code Flow

6.1 `check_auth()` Flow (core/auth.py:579-680)

check_auth(profile="default", live=True)
│
├── [fast path] profile_exists()? → no → return (valid=False, reason="no_tokens")
├── load_profile() → get cookies dict
│
├── [non-live] heuristic based on last_validated timestamp (7 day window)
│
└── [live] {the authoritative path}
    │
    ├── _fetch_notebooklm_homepage(cookies)
    │     └── httpx GET https://notebooklm.google.com/
    │     └── with browser-like headers (Sec-Fetch-*, User-Agent)
    │     └── follow_redirects=True
    │     │
    │     ├── status=200 AND not accounts.google.com in URL
    │     │   └── ✓ Authenticated! Extract CSRF, save profile, return valid
    │     │
    │     ├── status!=200 AND not accounts.google.com
    │     │   └── ✗ HTTP error, return valid=False
    │     │
    │     └── accounts.google.com in URL (redirect to login)
    │         └── Not definitive! Some valid sessions also bounce.
    │             Need secondary check via RPC...
    │
    └── [secondary check] NotebookLMClient.list_notebooks()
          │
          ├── raises AuthenticationError → ✗ expired
          ├── raises other exception → network error
          └── success → ✓ recovered! Save refreshed CSRF/session_id

6.2 `_refresh_auth_tokens()` Flow (base.py:1044-1122)

_refresh_auth_tokens()
│
├── rotate_google_cookies(client)  ← non-fatal, best effort
│     └── POST to https://accounts.google.com/RotateCookies
│     └── With Content-Type: application/json
│     └── Body: '[000,"-0000000000000000000"]'
│     └── Rate-limited: 60s between attempts per storage path
│     └── Disabled via NOTEBOOKLM_DISABLE_ROTATE_COOKIES=1
│
├── GET https://notebooklm.google.com/
│     ├── Redirect to accounts.google.com → raise ValueError (expired)
│     └── status=200 → extract from HTML:
│         ├── SNlM0e → csrf_token
│         ├── FdrFJe → session_id
│         └── cfb2h  → build_label
│
└── snapshot_cookie_input() → update cookies with rotated values
└── _update_cached_tokens() → save to auth.json + profile

6.3 `run_headless_auth()` Flow (cdp.py:1547-1634)

run_headless_auth(port=9223, timeout=30, profile_name="default")
│
├── has_chrome_profile("default")?
│     └── Check Default/Cookies or Default/Network/Cookies exists
│     └── If no profile → return None (can't auth without saved login)
│
├── Try find_existing_nlm_chrome() → reuse existing instance
│     └── If not found → launch_chrome_process(port, headless=True)
│
├── find_or_create_notebooklm_page(port)
│     └── Navigate to https://notebooklm.google.com/
│
├── Poll for login (timeout=30s)
│     └── Check current_url for accounts.google.com redirect
│     └── If logged_in = False after timeout → return None
│
├── _wait_for_page_ready() → poll for DOM tokens (session_id/build_label)
│
├── get_page_cookies(ws_url) → extract cookies via CDP Network.getAllCookies
├── validate_cookies() → check SID, HSID, SSID, APISID, SAPISID present
├── extract_csrf_token(html), extract_session_id(html)
│
└── save_tokens_to_cache(tokens)
└── cleanup_chrome_profile_cache() → remove Cache, Code Cache, etc.

6.4 CDP Communication Pattern (utils/cdp.py)

Connection via Chrome DevTools Protocol WebSocket:
  ws://127.0.0.1:{port}/devtools/browser/{browser_id}

Commands:
  Network.getAllCookies → list[{name, value, domain, path, ...}]
  Runtime.evaluate({expression: "document.documentElement.outerHTML"}) → HTML
  Runtime.evaluate({expression: "window.location.href"}) → current URL
  Page.navigate({url}) → navigate to URL
  DOM.getDocument → get DOM root node

The CDP WebSocket is cached (_cached_ws) per process for reuse.
Proxy env vars are explicitly disabled for CDP connections (httpx_client with trust_env=False).

7. Troubleshooting Reference

7.1 Common Errors

Error	Cause	Fix
`Authentication expired`	Cookies no longer valid	Run revive script → interactive tunnel re-auth
`ProfileNotFoundError`	No auth profile	Run `nlm login` for initial setup
`ClientAuthenticationError`	Backend auth check failed	Check Chrome alive + cookies not expired
`No supported browser found`	Chrome/chromium not installed	`apt install google-chrome-stable`
Chrome "Restore Pages" warning	Unclean shutdown of headless Chrome	Add `--disable-extensions` to launch args
`WebSocket 403`	CDP origin blocked	Add `--remote-allow-origins=*` to Chrome args
`no_tokens` heuristic pass but live check fails	Profile metadata stale, actual cookies dead	Re-run interactive auth

7.2 Chrome Profile Health Check

# Check if Chrome profile has cookies
ls -la ~/.notebooklm-mcp-cli/chrome-profiles/default/Default/Cookies
# If missing or 0 bytes → profile is empty, need re-auth

# Check if profile is locked by another Chrome instance
ls -la ~/.notebooklm-mcp-cli/chrome-profiles/default/SingletonLock
# If present and Chrome not running → delete it: rm -f SingletonLock

# Check profile size
du -sh ~/.notebooklm-mcp-cli/chrome-profiles/default/
# Should be ~50-100MB for a working profile

7.3 Forcing Clean Slate (Last Resort)

If all else fails, force a clean auth profile:

# Stop Chrome
pkill -f "remote-debugging-port=9223"

# Backup existing profile
mv ~/.notebooklm-mcp-cli ~/.notebooklm-mcp-cli.bak.$(date +%Y%m%d)

# Start fresh Chrome
google-chrome --headless=new --no-sandbox --remote-debugging-port=9223 \
  --user-data-dir=~/.notebooklm-mcp-cli/chrome-profiles/default/ \
  --no-first-run --disable-extensions --disable-gpu &

# Now authenticate via SSH tunnel (see Section 5.3)

8. Monitoring & Alerting

8.1 Recommended Cron Schedule

# /etc/cron.d/notebooklm-auth
# Auth check every 6 hours
0 */6 * * * root /opt/ai-os/products/ceo/scripts/auth_refresh.sh >> /var/log/notebooklm-auth.log 2>&1

# Docker restart if Chrome OOM (every hour check)
*/15 * * * * root /opt/ai-os/products/ceo/scripts/check_chrome_alive.sh >> /var/log/notebooklm-chrome.log 2>&1

8.2 Expected Log Output

2026-07-12 04:00:01 [AUTH CHECK] Starting
2026-07-12 04:00:02 [AUTH CHECK] Chrome on 9223: ALIVE
2026-07-12 04:00:03 [AUTH CHECK] Profile: nhivo2504@gmail.com
2026-07-12 04:00:04 [AUTH CHECK] Result: VALID
2026-07-12 04:00:04 [AUTH CHECK] RotateCookies: SUCCESS (200)
2026-07-12 04:00:04 [AUTH CHECK] Completed OK

8.3 Alerting

If auth check fails twice in a row, the system should: 1. Log the error to auth_failures.log 2. Attempt revive automatically 3. If revive also fails → flag for admin intervention

9. Script Reference

Script	Path	Purpose
`revive_notebooklm_auth.sh`	`/opt/ai-os/products/ceo/scripts/revive_notebooklm_auth.sh`	Kill Chrome → restart → attempt headless auth extraction
`notebooklm_refresh.py`	`/opt/ai-os/products/ceo/scripts/notebooklm_refresh.py`	Extract cookies from running Chrome headless → save to CLI profile
`auth_refresh.sh`	`/opt/ai-os/products/ceo/scripts/auth_refresh.sh`	Cron-friendly: check auth → revive if dead → log results
`check_chrome_alive.sh`	`/opt/ai-os/products/ceo/scripts/check_chrome_alive.sh`	Verify Chrome process + CDP endpoint alive