Controlled isolation and evidence
Controlled isolation answers one question: what smallest changed boundary explains the different result? It requires a known-good or known-failing baseline, one intentional delta, the same eligible inputs, and retained evidence for both sides.
Freeze the baseline
Before creating a candidate, record:
| Input | Frozen value |
|---|---|
| Game/framework stack | Exact versions and storefront/build |
| Candidate files | Names, sizes, SHA-256 hashes, and install paths |
| Archive contents | Sorted depot-path listing and extracted payload hashes |
| Unrelated mods | Explicit inventory or verified absence |
| Starting save | Capture name, slot directory, pre-install assertion, and hash |
| Route | Exact player actions and activation boundary |
| Oracle | Visible/log result that distinguishes pass from fail |
| Logs | Complete file paths and hashes |
“Same setup” is not evidence unless these values are retained.
State one falsifiable hypothesis
Write the hypothesis before editing:
If actor slot 1 is the out-of-range lookup, then changing only the scene's
addressable lipsync table/consumer indexes will move the launch result from the
same crash to successful scene entry under the same route and save class.
A useful hypothesis names:
- one suspected boundary;
- one controlled change;
- one expected observation if correct;
- one observation that would reject it.
“Make the scene more vanilla” is not falsifiable. “Change actor readiness, trigger radius, lipsync rows, VO paths, and graph timing” cannot identify a cause.
Build a one-delta candidate
- Copy the working baseline project.
- Make one semantic change. Generated binary noise does not count as another semantic change, but it must still be inspected.
- Round-trip every changed CR2W and compare decisive properties.
- Build in a clean staging directory.
- List/extract the archive and prove the expected payload set.
- Compare extracted payloads with the baseline and name every changed depot path.
- Stage one canonical candidate; record that earlier labs/copies and unrelated mods are absent.
- Hash installed files after staging.
If more payloads changed than planned, stop. Explain or eliminate the drift before runtime testing.
Run paired cases
Use independent full-slot clones from the same eligible capture:
| Run | Installed bytes | Save clone | Purpose |
|---|---|---|---|
| Baseline reproduction | Baseline hash | Clone A | Prove the original symptom still occurs |
| Candidate test | Candidate hash | Clone B | Test the one-delta hypothesis |
| Baseline return, when practical | Baseline hash restored | Clone C | Reject an unrelated one-time environment change |
| Removal isolation, when defined | Candidate pair deliberately absent | Clone D | Prove mounted candidate dependency without mutating save state |
Close the game before changing files or cloning slots. Run the same route and collect a fresh full log bundle each time.
Record observations before explanations
Separate these fields:
| Field | Example |
|---|---|
| Expected | Scene reaches its first section without process termination |
| Observed | Crash at setup boundary; no dialogue visualizer; report ID retained |
| Difference | Candidate moved past setup and returned through named exit |
| Interpretation | Result is consistent with the indexed table diagnosis |
| Rejected alternatives | World/trigger path was byte-identical; subtitle/VO resources unchanged |
| Limitations | Shared slot is diagnostic; facial quality and reload not tested |
Do not rewrite “consistent with” into a universal engine law. Runtime evidence is bounded by the exact bytes, versions, save class, and route.
Use negative controls
Negative controls prevent false attribution:
- bypass a suspect scene while leaving world/community setup intact;
- invoke a Start-to-End shell before restoring dialogue;
- remove only the candidate pair while keeping framework/unrelated-mod inventory frozen;
- restore the baseline hash after a passing candidate;
- test a clean original and an old save with identical installed bytes;
- move an activation boundary without changing the suspected resource, then record whether the failure follows launch or world position.
Each control answers a narrow question. A scene bypass proves the preceding path can continue; it does not prove the full scene is correct.
Keep a reproducible test
Save the baseline project, installed files, starting save, versions, and logs. Note the single change, the route you followed, and what happened. Keep failed runs too, so later changes can be compared with the same baseline.
If several changes were made together, split them into separate tests before assigning a cause.
Handoff checklist
Before another tester receives the candidate, they should be able to answer:
- Which exact bytes should be installed?
- Which files and mods must be absent?
- Which save capture and clone should be used?
- What actions reach the tested boundary?
- What observation passes or fails the case?
- Which logs and evidence must be retained?
- Which neighboring claims are explicitly outside scope?
If any answer is missing, the next run may be informative, but it is not yet a reproducible acceptance case.
Previous: Save state and clean retests. Back to: Troubleshooting.