cytoscape.js v4 HEAD · 47a5c66d · built 2026-09-16 14:48Z

Benchmarking: making a row measure the thing it is named for

Read this before adding a benchmark row or acting on one. A row that measures nothing looks exactly like a row that measures something, so the rule that runs through all of it is: ask what would make this number move, then run it that way once.

The suites, the runner and the published archive

  • benchmark/: v4's benchmark suites — 25 as of 2026-08-10 (round 33 took them to 22; style-bundle.mjs joined in round 36, arrows.mjs in round 55, algorithms-gpu-bench.mjs in round 65), each headed by a comment saying what it prices and how its rows avoid measuring nothing. Most measure against v3, imported from v3/src/. One measures through the built bundle rather than through src/, which is where a hot-path figure has to come from — see "What the instrument does to the measurement" below. npm run benchmark is the core/collection micro sweep, benchmark:report renders the HTML report (quick profile), benchmark:all adds every standalone sweep, benchmark:renderer drives the browser scenes on a real adapter, and benchmark:algorithms-gpu (round 65) prices executor 'cpu' vs 'gpu' for the async algorithm tier in a served page — it reports the adapter identity and refuses SwiftShader, since a software-rasterizer "GPU" number lies about the crossover. Since 65.9 it writes the standard results shape under its own algorithms-gpu profile (benches cpu/gpu/gpu first call; the report renders cpu-baseline pairs beside the v3 ones), so benchmark:publish promotes its runs onto the status site like any other profile. node scripts/bench-coverage.mjs --verbose says which public members no benchmark calls. benchmark/bench-size.mjs holds the run size (N, MIDNUM, MID) and imports nothing; graph.mjs re-exports it. Take constants from there — importing anything from graph.mjs evaluates both v3 and v4, which is how a spec about HTML rendering came to need v3's heap (2026-08-06). Publish with --repeat 3 (round 65.12): the runner runs each job three times and publishes the per-row median plus the band the three spanned, because this harness's run-to-run spread on the v4 side is wider than the ±10% the comparison flags at — measured, single runs make a change table that is 11% noise and medians of three make one that is 0%. Each job also carries a harness fingerprint (benchmark/harness-id.mjs): a hash of the suite file, its ./ import closure and the shared inputs, but never src/, which is the subject. A change across two fingerprints renders as a break rather than a percentage — the machine fingerprint's rule applied to the instrument, after round 65.11 found that 62.5c's pre-warm had moved v4 rows 12–35% and every page since had shown that step as a regression. Cosmetic harness changes are absorbed by normalising comments and formatting out of the hash, and anything left over can be declared in EQUIVALENT_HARNESSES with a reason (audited, so a stale entry fails). node scripts/benchmark-backfill-harness.mjs stamped the pre-65.12 archive from git. --jobs N runs the jobs concurrently (round 68, benchmark/schedule.mjs), and the default is still 1 — a serial run spawns exactly the sequence it always has, and stamps the same harness hash the archive carries. --jobs auto takes one worker per physical core less one, capped at 6, pinned with taskset (a bench process is ~1.1 cores over 7 threads, so one-per-core already oversubscribes). Use it for the iteration loop; publish serial. One scheduling rule is worth knowing because it shows up in the output: a job's repeats are kept apart (they are meant to be independent samples), except for whichever job's remaining chain is longer than the work left over the workers — that job's repeats overlap, its repeatSpread is correlated as a result, and the runner prints which jobs it did that to. Without the exception the run is bounded by the longest job's chain (measured: 11.0 min against a work bound of 8.4). A concurrent run measures every row against N-1 neighbours — all-core turbo instead of single-core, a share of the L3 instead of all of it — so concurrentHash folds the worker count into the harness hash and the comparison refuses a serial-to-parallel line the way it refuses a cross-machine one. The reason to keep publishing serial until the validation run happens is not that the page would be wrong but that it would be blind: the mover screen compares each change against that row's own repeatSpread, contention widens the band, and a widened band files real 10–20% wins as noise. Two related notes: the GPU profiles (renderer, algorithms-gpu) are exclusive and never overlap anything — one adapter, one queue, and frame time is the measurement; and splitting a long suite into BENCH_OP chunks to pack better is a measurement change the harness hash cannot see (report.mjs is in NOT_INSTRUMENT), so it needs more thought than a job-table edit, and below ~13 workers it buys nothing anyway. benchmark/published/ is tracked, unlike benchmark/results/: benchmarks are machine-dependent and slow, so no run happens on the status site's builder, and a run reaches the site only by npm run benchmark:publish on the machine that measured it. Runs are grouped by a machine fingerprint and never compared across machines; see benchmark/published/README.md for the retention rule. The status site renders a cross-commit comparison page per (machine, profile) with two or more published runs (benchmark/report-compare.mjs): per-row p50 across runs, the movers beyond ±10% with the frozen-v3 twin as a per-row noise control, and a whole-run drift figure — so a progression or regression is read against the machine's own noise, not assumed from one number.

Before the run: the box, the engine and the id

  • Run the sweep under the pinned Node (round 113.1). report.mjs refuses any other major than .nvmrc's unless --any-node is passed, because the machine fingerprint ignores the Node version by design (an upgrade must not split a box's history) and so nothing downstream can tell a V8 change from a library change. A shell that has not activated mise runs the system Node: one --all --repeat 3 sweep ran that way on Node 22 against a Node 24 archive and read +8% drift with every frozen v3 control moving too — an hour that compared two engines. mise exec -- npm run benchmark:report -- --all --repeat 3 is the form that cannot go wrong; the index carries each run's node so a step that coincides with an upgrade is visible on the page, and benchmark:publish warns when it changes.
  • The fingerprint rounds RAM to the GiB (round 113.1). os.totalmem() reports what the kernel leaves usable, and a kernel upgrade (6.19 → 7.1) moved it by 1.1 MB, which made the same i9-9900K a second machine (0e0e967f beside 2d2ea233) and left the comparison refusing to draw any line. The archive was re-stamped from each run's stored meta.machine block (scripts/benchmark-backfill-fingerprint.mjs) to the one id, 5cf3f79c. A fitted-RAM change still splits the history; a reservation does not.
  • A one-shot or a single-repeat suite cannot be the row that fails a run. Not new, but the sweep that opened round 113 makes the point in a new shape: layouts.mjs threw on every pass because its flow fixture was two components (a column permutation's orbits) and flow packs components as stacked tiles — the assertion "ranks form rows" was right and had never been run at the table's N. A row's assertion has to be exercised once at the size the job table runs it.

Rows that measured nothing

  • Before pricing anything GPU, run npm run gpu. The GPU profiles refuse SwiftShader by design, and an ad-hoc browser launch can land on SwiftShader on a box that has real silicon (wrong flags, about:blank — see the adapter note in rendering.md). A "no hardware here" deferral is only valid when npm run gpu says SOFTWARE-ONLY.
  • A benchmark row is guilty until it is shown to discriminate. Round 33 wrote six rows that measured nothing and caught them only because it checked: preset read 2388× because v4's preset does no work without a positions map while v3 walks every node; a compound style row read 3.55× faster than flat because one side was built without edges; a custom-polygon pick row read 1500× faster than its siblings because the pick point was inside the shape, so the walk stopped at the first node; and two box-selection rows measured an empty result because cy.elementsInBox takes four numbers and silently answers the empty collection when handed a box object — a bug benchmark/curves.mjs had carried since round 29.4, in a premium the README published. Before believing a row, ask what would make it move, and run it that way once. Round 39.1 adds a variant worth naming — a fixture can be styled into a mode it never enters: a benchmark row priced curved-edge box selection under curve-style: bezier, which bundles multi-edges only, so on a fixture with no parallel pairs every edge rendered straight and the row read identical to the straight row above it. unbundled-bezier curves per edge; the row now prints how many of its edges are actually curved, which is the general fix — have the row assert the property it is named for.
  • A comparative row can measure the harness instead of the library, in two ways round 62 pinned. (1) A cmp()-style row sharing one op closure between its v3 and gpu benches samples the first-declared side against monomorphic inline caches and the second against polymorphic ones — a systematic ~0.5–1 ns bias that decided the sign of every near-parity row until 62.5c added pre-warm alternations through both sides before either bench samples (round 55's measurement-order rule in IC form; check any new cmp helper carries them — data.mjs's was missed by the first sweep). (2) Below ~10 ns a row sits at the harness floor, where group order and sampling artifacts own the sign outright: pan() get read 0.83–0.87× across eight independent --all runs while replicating at 1.01× outside the suite and reading 4× v4-faster in per-process monomorphic loops. A row whose subject is cheaper than its instrument must amplify (32 reads per op, renamed (x32)) or it is a coin-flip wearing a ratio. Diagnose with three probes before touching the code: replicate the row faithfully outside the suite; swap the benches' declaration order; and time both sides in separate processes.

What the instrument does to the measurement

  • A large switch over strings is a linear scan, so position costs. V8 does not hash one: a generated 145-case string switch measures 48.7 ns at its first case and 552.9 ns at its last, and in src/style.mts moving one case (body untouched) from sixth to last cost it 56 → 90 ns. Two consequences. First, benchmark a dispatch at more than one position — rounds 33 and 34 both measured background-color, which happened to be the 4th case, and so understated every other property. Second, the fix is a Map of small readers (round 35), which flattens rather than uniformly lowers: the worst case improved 2.6×, the earliest few got ~15 ns slower. Judge such a change on the aggregate — a whole-object style() — not on one property. And note that size alone is not the trigger: the write path's applyProp is the same shape and stays a switch, because it runs per sheet compile rather than per read.
  • For a hot path, check a finding against the built bundle before acting on it. The benchmark/ suites import src/ through tsx, which injects esbuild's __name wrapper — an Object.defineProperty on every closure creation. Round 33 published StyleEngine.readProp at 13–21× v3 from those suites; profiling it for round 34 found 23% of samples in __name alone, and through build/cytoscape.esm.mjs (where grep -c __name is 0) the same getter measures 292 ns against v3's 50 ns — 5.8×. Allocation-light rows are unaffected, so this is not a reason to distrust the suites generally; it is a reason to re-measure any hot path that builds closures per call before you rewrite it. This is round 30's transpiled-coverage lesson in a second guise.
  • A v3 side needs styleEnabled: true and an explicit layout. Both halves bias, in opposite directions: a headless v3 defaults styleEnabled to false, so it does less work than v4 (which always applies its sheet), and v3's default layout is grid, so a plain cytoscape( { elements } ) runs a whole layout inside the measured region. Round 33.4's init comparison moved from 1.89× to 5.47× when both were fixed. benchmark/graph.mjs's helpers take the options; use them.