Repository navigation
feat: run each benchmark in its own process - #140
Draft
not-matthias wants to merge 3 commits into
Draft
not-matthias wants to merge 3 commits into
not-matthias wants to merge 3 commits into
Conversation
All benchmarks of a pytest session share one process, so the allocator state, caches and threads that one benchmark leaves behind change the measurement of every benchmark after it. Adding or changing one benchmark can then shift unrelated results. Add `Isolation`, read from the environment: - `CODSPEED_PROCESS_ISOLATION` is on by default (unset, `1`, `true`) and off with `0` or `false`. Any other value is an error. - `CODSPEED_ISOLATED_BENCHMARK` is set by the parent on each child and holds the URI of the only benchmark that child runs. `run_in_child_process` re-runs pytest as `python -m pytest <same args> -q` in the original directory. `sys.orig_argv` is not used because it needs Python 3.10 and the package supports 3.9; pytest's own invocation args also work for in-process `pytest.main` runs. Stdout and stderr are flushed first so the parent's output stays before the child's. A child killed by signal N reports exit code 128 + N, as shells do. Re-exec instead of fork(): fork only copies the calling thread.
Wire process isolation into the plugin for the simulation and memory modes, and only when the instrument hooks are active. Walltime and runs outside CodSpeed keep sharing one process. - Parent: a `pytest_pyfunc_call` hook spawns a child for each benchmark item and returns, so the parent never runs the test body. Fixture setup and teardown still run in the parent. `pytest_runtest_protocol` does not wrap marker-only benchmarks in the parent, so no measurement markers surround the child spawn. - Child: `pytest_collection_modifyitems` deselects every item whose URI is not `CODSPEED_ISOLATED_BENCHMARK`, which overrides `-k` and node id selection. The child measures and reports the benchmark as before. - A failing child fails only its test item with its exit code in the message, so the run continues and ends with TESTS_FAILED, like a raising benchmark in a shared process. - An invalid `CODSPEED_PROCESS_ISOLATION` value is a `pytest.UsageError`.
Run real pytest subprocesses in simulation mode and check that: - each benchmark runs in its own process by default, and all share one process with `CODSPEED_PROCESS_ISOLATION` set to `0` or `false`; - `benchmark(...)` returns the target's value with isolation on; - a raising benchmark fails only its test, the other benchmarks still run, and the run exits with TESTS_FAILED; - an invalid `CODSPEED_PROCESS_ISOLATION` value fails with a clear error. An autouse fixture sets `CODSPEED_PROCESS_ISOLATION=false` for the other tests. They run pytest in-process with `pytester` and check results that only exist where the benchmark body runs, which isolation moves into a child. The isolation tests set the variable themselves.
Merging this PR will degrade performance by 19.24%ℹ️ 1 benchmark has no flamegraph
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Memory | test_multiprocessing_map[1000] |
2.9 MB | 40.2 MB | -92.91% |
| ❌ | Walltime | test_sum_of_squares[sum_of_squares_sum_comprehension_product] |
195.4 µs | 223.9 µs | -12.73% |
| ❌ | Walltime | test_sum_of_squares[sum_of_squares_sum_comprehension_power] |
226.1 µs | 249.6 µs | -9.43% |
| ❌ | Walltime | test_tcp_connection[1.1.1.1-53] |
886.4 µs | 955.9 µs | -7.28% |
| ❌ | Walltime | test_noop_lambda_decorated |
1.2 µs | 1.3 µs | -6.95% |
| ❌ | Walltime | test_sum_of_squares[sum_of_squares_sum_labmda_product] |
278.4 µs | 298.4 µs | -6.7% |
| ❌ | Walltime | test_generate_all_combinations[5-4] |
10.4 µs | 11 µs | -5.39% |
| ⚡ | Walltime | test_multiprocessing_map[100000] |
438.7 ms | 307.4 ms | +42.72% |
| ⚡ | Walltime | test_multiprocessing_map[10000] |
71.6 ms | 52.7 ms | +35.86% |
| ⚡ | Walltime | test_is_complete[board0] |
1.9 µs | 1.8 µs | +7.6% |
| ⚡ | Walltime | test_noop_pass |
504 ns | 471 ns | +7.02% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing cod-3961-implement-process-isolation-in-pytest-codspeed (c5f9e54) with master (5af4238)
Footnotes
-
192 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR: Benchmarks in one pytest process share allocator, cache and thread state, so results depend on order and on unrelated benchmarks; now each benchmark runs in a fresh pytest process (each child re-collects the suite).
CODSPEED_PROCESS_ISOLATION: on by default (unset/1/true), off with0/false, any other value is a usage error.CODSPEED_ISOLATED_BENCHMARK=<uri>, sequentially; the child measures and reports it.-kand node ids).TESTS_FAILED.Review notes
benchmark(...)run in the child only.python -m pytest <args> -q(nosys.orig_argv, Python 3.9 is supported).-n(xdist) is passed through to children unchanged.Mirrors the codspeed-rust implementation (CodSpeedHQ/codspeed-rust, process isolation).
Closes COD-3961