Salastil/odysseus
1
0
Fork 0
mirror of https://github.com/pewdiepie-archdaemon/odysseus.git synced 2026-06-17 18:25:26 -04:00
Code Issues Packages Projects Releases Wiki Activity

test: report cwd in order-sensitivity runner

Browse Source
This commit is contained in:
Alexandre Teixeira
2026-06-11 19:47:03 +01:00
parent ecb9e7f2cf
commit 10f7c8e458
3 changed files with 49 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/README.md
+35 -5
View File
@@ -97,11 +97,41 @@ python3 tests/run_order_report.py --seed 123 -- tests/cli/ -q
python3 tests/run_order_report.py -- tests/cli/ -q # generates and prints a seed
```
The same seed reproduces the same order, so to reproduce a failing run, re-run
with the seed it printed (the runner prints an exact reproduction command at
the top of each run). Failures discovered this way are real isolation bugs:
fix them in separate scoped PRs - do not silence them with `skip`/`xfail`, and
do not "fix" them by depending on a particular order.
The same seed reproduces the same order when the reported working directory,
pytest target arguments, and test environment are also the same. The runner
prints all command arguments with shell-safe POSIX quoting and uses the
invoking Python interpreter.
A generated-seed run starts with output like:
```text
[order-report] working directory: /path/to/odysseus
[order-report] shuffling test order with seed 284734921
[order-report] reproduce from this working directory with the same test environment:
[order-report] reproduce with: /path/to/odysseus/.venv/bin/python /path/to/odysseus/tests/run_order_report.py --seed 284734921 -- tests/cli/ -q
```
Run the printed command from the reported working directory to reproduce the
same fixed-seed order:
```text
[order-report] working directory: /path/to/odysseus
[order-report] shuffling test order with seed 284734921
[order-report] reproduce from this working directory with the same test environment:
[order-report] reproduce with: /path/to/odysseus/.venv/bin/python /path/to/odysseus/tests/run_order_report.py --seed 284734921 -- tests/cli/ -q
```
Pytest output remains visible between the report header and footer. A failing
run ends with pytest's normal failure report followed by:
```text
FAILED tests/example_test.py::test_example - AssertionError
[order-report] seed 284734921: pytest exit code 1 (report-only; fix order-sensitive failures in separate scoped PRs)
```
Failures discovered this way are real isolation bugs: fix them in separate
scoped PRs - do not silence them with `skip`/`xfail`, and do not "fix" them by
depending on a particular order.
The runner propagates pytest's exit code, so it composes with normal local
workflows; "report-only" means it is not a CI gate, not that failures are