refactor(search): centralize the web-scraping User-Agent into one constant (#4325)

The outbound UA for web_fetch / web_search was inlined in four places with
two different values and nothing keeping them current: content.py pinned a
mid-2021 Chrome 91 build, and providers.py sent a bare Mozilla/5.0 in three
spots. Some sites serve a degraded or blocked page to a UA that old.

Add WEB_FETCH_USER_AGENT to src/constants.py (env-overridable, matching the
existing Copilot/Kimi UA-constant pattern) and import it in content.py and
providers.py. Default to a current, common desktop UA so pages return their
normal HTML: the market-leading desktop OS (Windows; NT 10.0 covers Windows
10 and 11) and browser (Chrome) on a current stable build. The version is now
bumped in one place.

Service-specific self-identifying agents (Copilot, Kimi, webhooks, cookbook)
are intentionally left separate. Adds a regression pinning the constant shape,
the env override, and a guard against a new inline Mozilla literal in the
search sources.

Closes #4324
This commit is contained in:
Kenny Van de Maele
2026-06-16 03:33:47 +02:00
committed by GitHub
parent b58af4267b
commit fafaf089c5
4 changed files with 31 additions and 6 deletions
+18
View File
@@ -0,0 +1,18 @@
"""The web scraping path routes its User-Agent through one constant.
Guards the dedup: web_fetch / web_search outbound UAs go through
WEB_FETCH_USER_AGENT, so a stale or bare Mozilla string cannot be re-inlined in
the search sources.
"""
from pathlib import Path
_SEARCH = Path(__file__).resolve().parent.parent / "services" / "search"
def test_search_sources_have_no_inline_mozilla_ua():
offenders = [
str(py.relative_to(_SEARCH.parent.parent))
for py in _SEARCH.rglob("*.py")
if "Mozilla/" in py.read_text(encoding="utf-8")
]
assert not offenders, f"inline Mozilla UA found; use WEB_FETCH_USER_AGENT: {offenders}"