cytoscape.js v4 HEAD · 47a5c66d · built 2026-09-16 14:48Z

Testing: what the suites are, and what a spec has to do to count

Read this before writing or changing a spec. Nearly every rule below is a defect that was paid for once — most of them specs that passed while testing nothing — so the recurring theme is the control: run the spec once with the thing it tests deliberately broken, and watch it fail.

The suites and how CI runs them

  • npm test runs the same work CI does, but not in the same shape. Round 53 split the workflow into ci-node (npm run build, then npm run test:node — typecheck, test:js, test:modules, test:soak, test:throws, lint — then test:types:all) and one ci-browser job per Playwright project, each installing only the browser it drives. npm test is run-s test:node test:playwright, so the chain has one definition; the split exists because the browser projects dominate the wall clock and share no work with the Node tier.
  • The Node tier needs no v3 install, and ci-node does not do one. It used to need one for a single integer — see AGENTS.md's command list — which is the shape to watch for generally: benchmark/graph.mjs imports both libraries to build its factories, so importing anything from it evaluates v3 and v4. Take run-size constants from benchmark/bench-size.mjs instead. If a spec ever does need v3, ci-node will say so loudly rather than pass on an accident.
  • A fresh checkout is a configuration nothing here tests, and it is what CI runs. Both of round 53's chronic failures were that: v3/tsconfig.json naming a type package only the root install provides (green everywhere a root node_modules exists to walk up into, red in the v3-only job), and three status-site.mjs specs needing build/cytoscape.umd.js, which is gitignored — Playwright's webServer built one, but later in the run-s chain than test:modules. test:modules builds for itself now, so the scripts no longer depend on the order; the lesson stands. Before blaming CI, reproduce it: git worktree add --detach <dir> HEAD, install there, and run the job's commands. It reproduced both exactly.
  • Almost nothing in the Node suite runs against the built bundle. v3 has a test:build script whose TEST_BUILD env var v3/src/test.mjs reads; v4 has no such switch, and the root copy of that script was dead from the round-42 split until round 43 removed it. What does exercise the bundle: the Playwright projects (which load build/cytoscape.umd.js), benchmark/style-bundle.mjs, and the round-46.5 wire-fixture specs in test/modules/status-site.mjs (which encode through build/cytoscape.umd.js, the bundle the page loads — round 65.13) — which is why test:modules and test:playwright each run build first (2026-08-06; the builds are sub-second, and a stale bundle had made the wire spec fail with a swallowed TypeError). That matters more here than it sounds — see the __name note in docs/agents/benchmarking.md, where a hot-path figure measured through tsx was 7x off what the bundle does.
  • Playwright setup depends on a built UMD bundle and a local HTTP server. Use the existing scripts rather than inventing a parallel harness.
  • The cross-runtime smoke tier runs the built bundles under Node, Bun and Deno (round 98). test/runtimes/smoke.mjs is one framework-free file — plain asserts, imports nothing beyond node:module/node:url — that loads the ESM, minified-ESM and CJS bundles and asserts values and ordering, never "it didn't throw", because a compat layer can pass a completion check while handing back a subtly wrong TextDecoder (the round-46.5 no-labels graph). Run it with npm run -s test:runtimes:node / test:runtimes:bun / test:runtimes:deno, each run-s build … so a stale bundle cannot pass for a fresh one. It must never soft-skip; test/modules/runtime-smoke.mjs enforces the import list and the never-skip property by running the controls as specs (a missing bundle path, and the dict-as-array degraded reader). CI runs ci-bun/ci-deno at a pinned floor plus latest stable.
  • Run one spec file with npm run -s test:js:one -- test/<file>.mjs (round 109). Specs get describe/it/beforeEach from test/node-test-setup.mjs, which the tier supplies as a preload (--import) and no spec file imports for itself — so a hand-rolled node --import tsx --test test/foo.mjs dies with ReferenceError: describe is not defined before a single assertion runs. That script is the tier's own invocation with the glob left off; append --test-name-pattern '<part of the name>' to narrow it further.

Playwright: the page, the frames and the workers

  • playwright-page/ runs a frame driver, and the specs depend on it (round 53). Chrome issues a BeginFrame only when something invalidated, and requestAnimationFrame runs only on a BeginFrame — so on the SwiftShader adapter CI pins, an animation started while the page is idle does not begin for ~1 s. Measured: a 1500 ms tween had run zero frames 800 ms after animate() returned. Two specs that sample a tween mid-flight failed deterministically because of it. frame-driver.js ticks a 1 px element at z-index: -1, behind the opaque full-viewport container, so frames keep coming without touching a pixel any spec samples. Do not remove it without re-running the mid-animation specs under CI=true on Linux, and do not give it a visible probe.

  • A mid-flight assertion polls; it does not sleep to an offset. How far a tween has run at a given wall-clock offset depends on when frames landed, so waitForTimeout( 900 ) + one pixel read is a race — and against a software rasterizer it is one the suite loses. untilMidFlight in renderer.spec.js waits for the state instead, which is faster too (the ten animation specs: 29.5 s -> 12.7 s). Its timeouts are deliberately generous and the timeout is not what makes a spec honest — the state has to be unobservable at rest, either in the predicate itself or in the assertion right after the poll. Read that helper's doc before adding one.

  • A tween's compute pipelines compile on the first animate() of a page, because Dawn defers compilation to first use. On the software adapter under load that stalls the first animation by up to ~1.8 s (measured: a screenshot 1779 ms after animate() returned still showed the node at its start, animated() true). Any spec that reasons about when a tween should be somewhere has to allow for it.

  • Half the cores, not one worker per core (playwright.config.js). SwiftShader is multi-threaded, so a browser per core oversubscribes the box and specs start failing — one-per-core is the ratio the runner was using, and the one that fails. The table in that file has the measurements; do not raise it without re-running them.

  • On a non-Debian distro Playwright's host check is a false negative, so the config skips it there (never on CI). Fedora additionally needs WebKit's libjpeg.so.8 — the jpeg8 ABI Fedora's libjpeg-turbo does not build. Drop one in beside the bundle's own private libs, which is where Playwright already ships libjxl and libbacktrace:

    # from Ubuntu's libjpeg-turbo8 .deb, extracted anywhere
cp libjpeg.so.8.2.2 ~/.cache/ms-playwright/webkit-*/minibrowser-*/sys/lib/
ln -s libjpeg.so.8.2.2 ~/.cache/ms-playwright/webkit-*/minibrowser-*/sys/lib/libjpeg.so.8

    It has to be redone after playwright install refreshes the browser. To simulate CI on such a host, set PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1 yourself — the config's skip is disabled when CI is set, deliberately.

  • Rebuild the bundle before trusting a Playwright run. playwright.config.js sets reuseExistingServer: !CI, so when an http-server is already listening on 3333 (a leftover npm run watch, or an earlier run's server), Playwright attaches to it and the test:playwright:build half of test:playwright:setup never runs. (Round 42 gave v3's project port 3334 for exactly this reason — before that both served 3333, so a stray v3 server could feed v4's specs v3's pages.) The suite then silently exercises a stale bundle, and a green run proves nothing about the source you just changed. Since 2026-08-06 npm run test:playwright runs test:playwright:build unconditionally before the suite, so the scripted path is safe; the trap remains when invoking npx playwright test directly — run npm run test:playwright:build yourself first, or kill the listener on 3333. This is invisible when it happens — the only symptom is a pass you did not earn.

  • Visual regression coverage for the GPU prototype lives in the visual Playwright project (playwright-tests/visual.spec.js): golden-image diffs against PNGs checked into playwright-tests/goldens/ plus live v3-vs-v4 parity diffs. After an intended visual change, regenerate goldens with UPDATE_GOLDENS=1 npx playwright test --project=visual and commit the updated PNGs; never edit goldens by hand.

  • playwright-tests/ and playwright.config.js: Browser-level regression coverage — the renderer project (and renderer-webkit) — which since round 65 also carries algorithms-gpu.spec.js, the CPU-vs-GPU executor parity suite for the async algorithm tier (headless instances, adapter soft-skip) — and visual (goldens + live v3-vs-v4 parity diffs — including round 56's close-up tier at zoom 3-4 — plus round 55's numeric routing parity in routing.spec.js).

    • routing.spec.js compares geometry, not pixels — both libraries' controlPoints()/segmentPoints()/endpoints/midpoint() through one symmetric probe on playwright-page/parity.html, reporting which field on which edge diverged and by how many model px. It exists because a 6 px arrow gap is 0.005% of a 400x300 canvas, 400x under the pixel bound. It needs no WebGPU adapter and draws no frame, so it carries no hasAdapter skip — do not add one. It runs in the browser only because v3 computes routing inside its renderer; importing v3's entry from test/ would break the Node tier's no-v3-install invariant.

Goldens and v3 parity

  • An opaque filled shape hides its own overlap, so a scene built around one measures nothing. Round 55's first arrow-gap scene used a huge head (width 20, arrow-scale 3) reasoning that a 120 px gap on a 300 px chord would dominate the frame; it read 0.495% and passed, because the head is opaque and 137 px long, so both renderers paint the same pixels over nearly all of the difference. The visible residue is only the wedge near the tip where the head is narrower than the line (~`width²/2` per end). Tuning the other way — thick line, small head, and more ends rather than bigger ones — took it to 3.537%. When a scene is named for a difference, ask what is painted over it.
  • Count the ends, not the elements. Several of this suite's differences scale with the number of arrow ends or corners, not with element size, so the way to push a scene over a bound is usually more instances rather than bigger ones.
  • Zoom in. The parity suite has a close-up tier (round 56, parity-closeup-*): short edges rendered at zoom 3-4, carrying bounds 4-20x tighter than the zoom-1 scenes. It works because anti-aliasing is a boundary effect that does not scale — magnifying grows the ink while the fringe stays a pixel wide, so AA's share of the mismatch falls and geometry has nowhere to hide. A change to edge or arrow geometry should gain a close-up scene, not another 400x300 one; and note the second requirement, short edges, or the ends fall outside the frame and an arrow test quietly becomes a test of the middle of a line.
  • A golden only sees what is not painted over. Round 56's arrow trim moved eleven of 43 goldens by at most 0.178%, under the 0.5% bound goldens carried then — not bad tuning, but the nature of the thing: v3 sizes its gap so the line stops under the head, and every arrow golden used opaque filled heads, which hide the whole difference. When adding golden coverage for something, build the scene out of the configuration that exposes it (here: hollow and translucent heads) and prove it with a control — the arrow-gap golden moves 1.4% when the fix is degraded and 5.3% when it is removed.
  • A golden exports the viewport, not the graph. Six of 43 were silently cropping their own scene, the worst losing 109 px of a 300 px canvas — and the cropped rows were covered by nothing. expectGraphFits now runs before every golden diff; when a scene outgrows its canvas, give it a bigger one with useViewport() rather than letting it clip. Uncropping that one exposed four arrowheads listed in the scene since round 27.6 and never drawn, because their mapper clause was missing — two defects concealing each other.
  • Goldens are exact — zero differing pixels — since round 57.1e, and widening that is not the fix for a red one. The default was 0.5%, with eleven label scenes granted 2% on top, and the price was measured: six goldens' committed PNGs no longer matched what the code rendered, one by 1.597%, drift accumulated across five label modules and the shaders over a week of green runs. Exact is affordable because every input is pinned — SwiftShader by the visual project, the browser by Playwright, and the label font vendored through @fontsource/open-sans in node_modules rather than taken from the system — and was verified at zero over all 45, repeatedly and at different worker counts. A browser bump that legitimately moves antialiasing will therefore fail: look at the diff, and if that is what it is, regenerate and commit. A tolerance does not make a golden robust, it makes it quiet.
  • If a change claims v3 parity, verify it with a live parity diff, not a golden. Goldens compare against v4's own previous output, so they answer "did this change?" and not "is this right" — in round 27 the arrow goldens passed both before and after v4's arrow sizing was corrected to v3's formula, because the difference sat under the tolerance goldens carried then. The parity tests render the same scene through both renderers in one run and diff them, which is what actually caught it.
  • Check that a new parity test can fail. Run it once with the feature deliberately disabled (or the shape swapped for its unstyled counterpart) and confirm the mismatch jumps. Round 27 shipped one test whose first version passed at 0.514% with the feature on and 0.672% with it off — it was measuring nothing. Make the scene dominated by whatever the change affects.
  • The same control applies to plain unit specs, and a spec's name is not evidence that it tests what it says. Round 27 shipped three specs named 'picks by its slanted outline, not its bounding box', 'picks inside the body and outside the cut corners' and 'picks inside the body ...' whose only assertions were on boundingBox() — which is the node box for every shape keyword, so all three passed with the shape under test swapped for ellipse. Each comment described the pick it meant to check and then never called a pick path. When a spec is named for behaviour X, assert X, and run the file once with X's implementation swapped out to confirm the failure lands.
  • A spec that accepts either outcome discriminates on neither. Round 48.5 lost the GPU device mid-export and asserted only that the promise settled — resolved or rejected are both correct, and which you get depends on how far the readback had progressed. With the loss hook neutered the export simply resolved and the spec passed. When the outcome is legitimately either/or, assert the precondition as well (here: that devicelost actually fired), or the spec is testing that promises settle. Its two siblings, which assert a specific end state, failed the control as they should.
  • A fixture must be written in the shape the tool actually parses. Round 36's audit fixtures declared members as one-liners with the comment inline (/** a */ a(): void {}), a shape the scanner does not match and the real sources never use — so two specs passed with the behaviour they tested deliberately broken. The control caught it, which is the argument for running controls on tool specs too: a fixture that the tool silently skips is a spec that can never fail.

Leaks, fuzzing and the throw gate

  • A leak gate measures reachability, not bytes. Round 48.1 measured 1000 create/destroy cycles growing heapUsed by a steady ~2.2 KB per cycle — linear across five 200-cycle blocks, which is what a leak looks like and is not what this was. A WeakRef per destroyed instance showed every one collected; the growth is V8's own bookkeeping, and a byte bound would have pinned the engine's behaviour as the library's contract. Assert that the thing is collected (--expose-gc, hence the separate npm run test:soak), and keep a byte bound only as a wide backstop. The first spec in such a file must be the probe's own control — held instances stay reachable, unheld ones do not — or every spec after it passes by doing nothing.

  • Validate a payload-derived count where something already walks it. Round 48.3 fuzzed the wire format and found three defects of one shape: a count or index read out of the buffer, unvalidated, driving allocation (a dictionary index of 2,566,914,049 made a load never return; a corrupt offset total made it take 25.9 s). The fix's placement is the lesson. Fusing the dictionary check into DataStore.ingestColumn, whose two branches already walk every index, costs nothing; the same check in deserializeElements measured 4× on that function (0.106 → 0.46 ms per 200k indices) against a reader whose headline property is being O(1) per column. Prefer an O(1) invariant where one exists (a declared total cannot exceed the blob carrying it), then a fused walk, and only then a new pass.

  • A guard reachable only from a fuzzer is invisible to the throw gate. scripts/throw-coverage.mjs runs test/!(types-*|node-test-setup).mjs — not test/modules/, not test/soak/. A new throw new exercised solely by a soak or fuzz spec reads as Node-reachable-and-never-run and fails the build, correctly. Pin each guard deterministically in test/ as well; round 48.3 hand-built a 28-byte buffer for one of them rather than relying on the mutation that found it.

  • Reformatting the tree is a free control on every source-scanning tool, and it found four defects when round 57.2 ran it. These audits (jsdoc-coverage, throw-coverage, the docs generator, the status build's golden captions) read the sources as text, so each carries an unstated assumption about layout, and a formatter falsifies all of them at once. What it turned up: five public members whose parameters wrapped were skipped by the @param gate — reading 232/232 while the true surface was 239 and five had no tag at all; memberBody stopped at the first wrapped parameter line (it matches the member pattern) and again at a multi-line return type's } {, so @throws detection silently fell; an export const f = that broke after the = vanished from every audit; and a throw inside a module-level arrow const had been reading as covered by misattribution — the throw gate's one documented blind spot — so a guard no spec had ever fired read as tested. None of it was caused by the formatter; all of it was hidden by the previous layout. Two lessons: join the signature before you parse it (signatureOf/argListOf exist for this), and when a tool's numbers move under a purely cosmetic change, the old numbers were the wrong ones.

  • A source-scanning tool that skips a file reads identical to a clean file, so count what it touched (round 127). The round's replacement walker was comment- and string-aware and not regex-aware; src/style.mts holds /^url\s*\(\s*['"]?…/ on one line, the walker read the ' as opening a string, and the remaining 8,300 lines were silently skipped — reported as "zero literals to replace", the same answer a finished file gives. The tell was 24 of 25 files touched. test/modules/string-keys.mjs carries that exact line as a control — the literal planted after the regex must be found — beside the ones every scanner needs (a comment, a template, a double-quoted string are not code; a division is not a regex), and pins the walk's size (file count, and the declaration table's own member count) so a walk that stops early cannot pass as a tree with nothing to find.

  • A guard nothing has ever triggered is not tested. v4 fails loudly by design, so its throws are public contract; node scripts/throw-coverage.mjs [--verbose] reports which of the throw new sites in src the Node suite actually runs (round 30 took that from 34 never-executed to 0 Node-reachable). Since round 37.1 it gates, at zero tolerance — npm run test:throws, part of npm test: a Node-reachable site with no spec fails the build, and so does an UNREACHABLE/MISATTRIBUTED entry that no longer names a throw new line or carries no reason (the lists are keyed by file:line, so an insertion above an exempted site silently hands its exemption to a different throw). When adding a guard, add the spec that fires it — and prefer a message assertion when several guards share a method, since "it threw" does not say which one.

  • Coverage of transpiled sources needs source maps, or it lies. Reading raw NODE_V8_COVERAGE offsets against the .mts files is fiction: tsx transpiles before V8 sees them, and round 30's first measurement "found" 47 dead throw sites including two with specs since round 13. Collect through the test runner instead (--enable-source-maps --experimental-test-coverage --test-reporter=lcov), and use line-level (DA:) data only — the function-level records misattribute one-line arrow functions, and even DA attributes the body of a module-level arrow const to the module-evaluation count.

Specs that test what they say they test

  • A headless spec for browser-page behaviour inherits the headless defaults, not the page's. The 2026-08-05 review pass pinned that debug/'s compound fixture lays out into disjoint parent boxes, and the control came back BAD: deleting the fixture's cols: 3 — one of the two things that had made it unreadable — left the spec green, because grid derives its column count from the container's aspect ratio and a headless instance is 800 × 600 where the page is 930 × 900, so headless picked 3 anyway. Any spec asserting a layout, a fit or anything else that reads cy.width()/cy.height() must set headlessWidth/headlessHeight to the dimensions it is standing in for, or it is testing a different graph than the one that broke.

  • A layout change goes through test/layout-quality.mjs before its own file (round 114.8). The suite runs every built-in, and the spiral example through the contract, over six fixtures and asserts the properties a user sees first — nothing overlaps (bodies, and labels under nodeDimensionsIncludeLabels) and nothing is over-separated (round 115: a crammed run leaves some pair at exactly the padding, force's separation grows the settle by less than 2x — overlap-free was only half the property, and the first version of every overlap rule passed the half it had while spreading graphs several times wider than v3), the viewport fits with the padding exactly on the binding axis, a locked node stays, an animated run ends where the sync run ends, components sit apart — and every overlap row carries the control this file requires, asserted red. Its controls found what the per-layout files' green rows had not: concentric letting two squares meet corner-on, and a ring fixture that never exercised flow's extents (cycle removal makes a ring a chain, one node per rank). Run it once with the layout's overlap code stubbed before believing a new row; a control that stays green is a finding.

  • A control that fails to fail is a finding, not a wasted control. Round 46.5 ran eleven and two of them landed nowhere. One deleted an existsSync guard and every spec stayed green — the catch beneath already handled a missing file, so the guard was dead code and was removed. The other removed a fixture-minifying branch to break a "nothing exceeds the 25 MiB cap" loop, and that loop stayed green too, because both oversized fixtures load from a remote so nothing planned was near the cap: the loop was not discriminating, and a spec that measures the minified size on disk was written in its place. When a control does not fail, do not shrug and move on — either the code is redundant or the spec is.

  • An ignore pattern without a leading slash matches at every depth. .gitignore gained status for the round-46.5 build output, and it also matched scripts/status/ — the ten modules that are the build. git add -A staged nothing from them and the commit would have been unrunnable. Anchor a build-output pattern to the root (/status), and read what git status --short actually lists before committing a new directory.

  • An asset check that reads HTML attributes does not see URLs fetched from JS. Round 42 verified every src/href in all six harness pages resolved, and four of the debug page's seven networks were 404ing the whole time: their URLs live in debug/networks.js and are handed to fetch(). The page rendered nothing, said nothing (the Promise.all had no .catch), and no test noticed, because debug/ had no tests. Round 43 fixed all three halves — the paths, the missing .catch, and the absence of coverage — but the general lesson is the one to keep: enumerate what the runtime asks for, not what the markup declares.

  • A restructure is behaviour-neutral only if you check every file, not every test. Round 42 moved ~1100 files and rewrote imports across all of them; a green suite says the paths resolve, not that nothing else changed. What actually proved it was comparing blobs: every file now under v3/ against its pre-move blob (6 differed, all intended), and every file under src/ against its src/gpu/ original, with the diff filtered to the two changes the round was allowed to make — an import-depth fix or the factory rename. Anything printed by that filter was a bug. Do this before trusting the suite, not instead of it.

  • A vendored copy joins the audits. Round 42 copied five utility modules out of v3 so src/ imports nothing outside itself, and the JSDoc gate failed immediately: 19 undocumented exports, the internal tier at 96.8%. That is the gate working — a file in src/ is v4's regardless of where it came from. The fix is to document the copy, not to exempt it; and it is a reason to copy lean, since v3's math.mts would have dropped 1500 mostly-dead lines into scope.

  • Do not record a GPU measurement as "blocked, no adapter here" without checking. That conclusion has been reached and corrected twice (round 18.5 and round 27.9 — see PLAN.md). requestAdapter() returns null on about:blank, so a bare-page probe reads as "no GPU" on a box that has one; probe from a served page, which is what benchmark:renderer already does. Only the visual goldens pin SwiftShader, deliberately, and only for the WebGPU adapter — that pin says nothing about what hardware is present.

The quiet twins, in full

  • Run the quiet twins (round 101): every verification script has a :quiet twin — test:js:quiet, test:modules:quiet, test:soak:quiet, test:throws:quiet, test:runtimes:node:quiet (and its bun/deno siblings), lint:quiet, typecheck:quiet, build:quiet, and the composites test:node:quiet, test:playwright:quiet, test:quiet — that prints only actual failures: a green run is zero bytes (the exit code is the contract) and a red run prints the failing tests' blocks and nothing else. Invoke them as npm run -s <script> — measured 2026-08-24, -s drops npm's banner on the outer call and on every nested run-s child (loglevel inherits through the environment), so a green composite is zero bytes end to end. A green test:node was ~3,800 lines of context; its quiet twin is 0. The loud originals remain for humans watching progress, for debugging (a quiet run shows nothing until it finishes — diagnose a hang by rerunning the loud twin) and for CI, which stays loud deliberately because its logs are the record. The interactive scripts (test:js:debug and kin) have no quiet twins on purpose: quiet is for verification, not debugging. test/modules/quiet-scripts.mjs enforces that each twin is the loud command modulo the reporter flag or quiet-run prefix, so the pairs cannot drift.