Skip to content

Parallelize docs builds with sphinx-build -j - #10760

Open
GarrettBeatty wants to merge 1 commit into
feature/build-perffrom
build-perf-sphinx-parallel
Open

GarrettBeatty wants to merge 1 commit into
feature/build-perffrom
build-perf-sphinx-parallel

Conversation

@GarrettBeatty

Copy link
Copy Markdown

Description of changes:

Part 1 of 3 (stacked) of the CI build-perf changes from #10759, targeting feature/build-perf. Speeds up the docs build (the Catapult BuildDocs step).

sphinx-build -j N barely helped before because the guzzle theme was not marked parallel-safe and because Sphinx does most of the parallel work in its main process for a doc set this size (~20k pages):

  • guzzle_sphinx_theme: declare the theme parallel-safe, and build sitemap.xml from the environment (plus pages written by the main process) so it is complete and ordered like a serial build when pages are written in worker processes.
  • parallel_chunks (new extension): while reading, use one interleaved chunk per process. Sphinx otherwise creates ~180 small chunks and the main process unpickles each worker's entire environment, which grows with every chunk. Also restores the serial document order after reading, since it leaks into searchindex.js.
  • parallel_write (new extension): load, resolve, post-transform and index doctrees in the worker processes instead of the main process, and merge the search index fragments and images back in chunk order. This is the approach from Parallel post transform including write_doc_serialized sphinx-doc/sphinx#11746, specialized for the HTML builder.

Both extensions override private Sphinx methods, so they need to be re-validated when upgrading Sphinx (6.2.0 is pinned in requirements-docs-lock.txt).

Testing:

Measured on a c5.9xlarge (same vCPU/memory as CodeBuild BUILD_GENERAL1_XLARGE): 56.6 min (serial) → 9.4 min with -j 16.

  • Output is identical to a serial build: all 40,916 files are byte-identical except .buildinfo (which hashes the config), including searchindex.js and sitemap.xml; same warning count.
  • End-to-end dry run in a Catapult dev stack: BuildDocs 49.5 → 9.8 min compared with prod medians.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

* Declare the guzzle theme parallel-safe and build sitemap.xml from the
  environment so it is complete and ordered like a serial build when
  writing in parallel.
* Add the parallel_chunks extension, which reads with one interleaved
  chunk per process so the main process does not repeatedly unpickle
  large environments, and restores the serial document order.
* Add the parallel_write extension, which loads, resolves and indexes
  doctrees in the worker processes and merges search index entries
  back in chunk order.

With sphinx-build -j 16 this builds the docs in ~9.5 minutes instead of
~57 minutes, with output identical to a serial build.
@GarrettBeatty
GarrettBeatty added this pull request to stack #10763 October 15, 2026 14:29
@GarrettBeatty
GarrettBeatty marked this pull request as ready for review October 15, 2026 14:44
@GarrettBeatty
GarrettBeatty requested a review from a team as a code owner October 15, 2026 14:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant