mirror of
https://github.com/pewdiepie-archdaemon/odysseus.git
synced 2026-06-16 17:55:26 -04:00
refactor(search): centralize the web-scraping User-Agent into one constant (#4325)
The outbound UA for web_fetch / web_search was inlined in four places with two different values and nothing keeping them current: content.py pinned a mid-2021 Chrome 91 build, and providers.py sent a bare Mozilla/5.0 in three spots. Some sites serve a degraded or blocked page to a UA that old. Add WEB_FETCH_USER_AGENT to src/constants.py (env-overridable, matching the existing Copilot/Kimi UA-constant pattern) and import it in content.py and providers.py. Default to a current, common desktop UA so pages return their normal HTML: the market-leading desktop OS (Windows; NT 10.0 covers Windows 10 and 11) and browser (Chrome) on a current stable build. The version is now bumped in one place. Service-specific self-identifying agents (Copilot, Kimi, webhooks, cookbook) are intentionally left separate. Adds a regression pinning the constant shape, the env override, and a guard against a new inline Mozilla literal in the search sources. Closes #4324
This commit is contained in:
committed by
GitHub
parent
b58af4267b
commit
fafaf089c5
@@ -0,0 +1,18 @@
|
||||
"""The web scraping path routes its User-Agent through one constant.
|
||||
|
||||
Guards the dedup: web_fetch / web_search outbound UAs go through
|
||||
WEB_FETCH_USER_AGENT, so a stale or bare Mozilla string cannot be re-inlined in
|
||||
the search sources.
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
_SEARCH = Path(__file__).resolve().parent.parent / "services" / "search"
|
||||
|
||||
|
||||
def test_search_sources_have_no_inline_mozilla_ua():
|
||||
offenders = [
|
||||
str(py.relative_to(_SEARCH.parent.parent))
|
||||
for py in _SEARCH.rglob("*.py")
|
||||
if "Mozilla/" in py.read_text(encoding="utf-8")
|
||||
]
|
||||
assert not offenders, f"inline Mozilla UA found; use WEB_FETCH_USER_AGENT: {offenders}"
|
||||
Reference in New Issue
Block a user