fix(compare): stop blind mode leaking model identities via session names (#1318)

Blind Compare anonymized the pane headers, but each pane still created a helper chat session named "[CMP] <real-model>" and GET /api/sessions returned the session's model field. So the sidebar and the session-list API let a user map "Model A" back to its real model before voting, defeating the blind test.

- Frontend (static/js/compare/index.js, panes.js): in blind mode, name helper sessions by their neutral slot ("[CMP] Model A") instead of the model, matching the existing blind pane labels.
- Backend GET /api/sessions (routes/session_routes.py): blank the model field for [CMP]-prefixed helper sessions via a new _public_model helper.
- Backend /api/compare/start (routes/compare_routes.py): name blind sessions by slot and withhold model_left/model_right/mapping from the blind response (revealed at /vote).
- Tests: tests/test_blind_compare_redaction.py.

Fixes #1285.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rudy Wolf
2026-06-04 06:39:01 +03:00
committed by GitHub
parent 3d8c364689
commit 1c43daa564
5 changed files with 153 additions and 22 deletions
+17 -1
View File
@@ -21,6 +21,22 @@ def _sanitize_export_filename(name: str) -> str:
return name[:128]
# Blind-compare helper sessions are created with this name prefix. Their real
# model must never surface in the session list / sidebar — otherwise a blind
# comparison can be de-anonymized before the user votes (issue #1285).
COMPARE_SESSION_PREFIX = "[CMP] "
def _public_model(name: str, model: str) -> str:
"""Blank out the real model of blind-compare helper sessions so the
session list can't be used to map a neutral pane label ("Model A") back
to its model. The Compare UI tracks models client-side, so hiding it here
costs the sidebar nothing. See issue #1285."""
if (name or "").startswith(COMPARE_SESSION_PREFIX):
return ""
return model
def _verify_session_owner(request: Request, session_id: str, session_manager=None):
"""Verify the current user owns the session. Raises 404 if not.
@@ -215,7 +231,7 @@ def setup_session_routes(session_manager: SessionManager, config: dict, webhook_
finally:
db.close()
sessions = [{"id": s.id, "name": s.name, "model": s.model,
sessions = [{"id": s.id, "name": s.name, "model": _public_model(s.name, s.model),
"endpoint_url": s.endpoint_url, "rag": s.rag,
"archived": s.archived, "folder": folder_map.get(s.id),
"total_tokens": token_map.get(s.id, 0),