Run fits in parallel with n_jobs

This guide shows how to spread Starfold’s fits over several CPU processes with n_jobs, and where the option does nothing.

Prerequisites

  • A CPU run (engine="cpu", or "auto" on a machine without a usable GPU).

  • A machine with more cores than you use now. n_jobs counts worker processes, so it should not exceed your physical cores.

Steps

  1. Set n_jobs in the call.

    import starfold

result = starfold.recover(X, engine="cpu", n_jobs=3, random_state=0)

  2. Read what it used.

    prov = result.provenance
print(prov["n_jobs"], prov["n_jobs_forced_to_one_on_cuml"])

What n_jobs spreads

Stage

Parallel over

Limit

search

the fits of one trial

a complete trial has n_resamples fits, 3 by default, so more than 3 workers do not speed up a trial

refits

the n_worlds refits (refit)

n_worlds

calibration test

null copies

only up to 20,000 rows per resample

Each worker is a separate single-threaded process, so throughput comes from processes and not from threads. Memory grows with the number of workers. The peak memory of multi-process runs is not measured here, and multi-process CPU runs at scale are not validated.

Expected result

The output equals that of the serial run for the refits, and for the calibration copies up to 20,000 rows per resample. The docstrings state both, and the test suite checks the calibration case. Above 20,000 rows per resample, the neighbour graph comes from an approximate search whose result depends on the thread count, so a run reproduces only with the same n_jobs. Give the same n_jobs to Run the calibration test in that case.

No recorded run uses n_jobs above 1, so the speed-up is not measured here.

On cuML

On cuML, n_jobs above 1 is forced to 1, and result.flags contains n_jobs_forced_to_one_on_cuml. The GPU runs one fit at a time. To use several CPU cores, pass engine="cpu".

Variations