Docs
Upstream parity
The same P model, schedules and observable results.
Parity means running the same project and test case with the same schedule count and seed, on one worker, then getting the same exit code, separate stdout and stderr, and the same output files after the fixed normalizations below. This measures snappea-compat; the native snappea CLI has its own presentation and defaults.
The committed check corpus contains 465 cases, of which 459 must match completely after normalization. The vendored observe corpus contains 96 runs, of which 94 must match completely after normalization. The remaining cases and runs are listed with reasons below.
What is measured
| Level | Checks |
|---|---|
| Exact | Exit codes, file lists and history rotation; schedules, coverage and scenario coverage contents. |
| Exact after normalization | Console output, parse and type errors, compiler warnings, readable, JSON and XML traces, summaries, and PObserve output files. Large check files are compared by SHA-256 of normalized contents. |
| Written only | .sci files must exist at the correct paths. Their contents are a reduced form and are not compared. |
| Not compared | Generated C#/Java and binaries, version strings, multiworker runs, and the exclusions below. |
The check corpus was recorded from upstream P at 260da928f67569ca7a8c0039a02d0bad7cde6096, pinned in UPSTREAM. CI runs Snap Pea against those frozen outputs; it does not need to run P.
Results by area
These inventories are computed at site build time from the committed manifests, output lists, file paths and known-difference lists. “Match” is the assertion enforced by passing compatibility CI, not a new run performed by the docs build. A run here is one recorded command, including usage errors and test listings; it is not one explored schedule.
| Area | Cases | Match | Runs | Files | Contents compared | Written only | Bug trace files |
|---|---|---|---|---|---|---|---|
| regression | 423 | 423 | 425 | 1195 | 922 | 273 | 250 |
| tutorial | 8 | 8 | 82 | 255 | 181 | 74 | 22 |
| cli | 14 | 8 | 18 | 47 | 39 | 8 | 15 |
| options | 20 | 20 | 20 | 36 | 26 | 10 | 4 |
Regression cases cover static errors and bounded checks; tutorials list and check each test case. CLI and option cases cover help, errors, replay, output paths, rotation, coverage merging, strategies and step limits. Bug trace files count readable, JSON and XML traces, including history copies. Schedules are also included in the file-content comparisons. Files are counted at their final paths per case, not once per command that writes them. All content assertions must pass outside the known-difference cases; individual matching-file counts inside those cases are not recorded.
| Area | Runs | Match | Files recorded | Files compared | Replay logs compared |
|---|---|---|---|---|---|
| happy-logs | 48 | 46 | 65 | 60 | 14 |
| error-logs | 48 | 48 | 120 | 120 | 72 |
Observe file counts exclude the recorded console and exit-code metadata. Each log is run with and without --assumeInputsAreSorted. For the known observe differences, CI checks only the exit code; console and files are skipped entirely.
The separate, manually checked full upstream log suite has 192 runs: all exit codes match, 186 outputs match after normalization, and six differ in the null-dereference message. That historical measurement comes from compare-pobserve.py, not the vendored CI inventory above. The vendor script selects logs smaller than 8 KB.
Every normalization
The recorder and Rust check test read the same normalize.tsv. A “drop” removes matching console lines only; a “mask” replaces matches in both console and files. The following table is read directly from that file.
| Rule | Pattern | Replacement | Reason |
|---|---|---|---|
| drop | ^\.\. Searching for a P project file | whole line removed | Upstream discovers compiled output and logs compilation progress; Snap Pea reads sources directly. |
| drop | ^\.\. No P project file | whole line removed | Upstream discovers compiled output and logs compilation progress; Snap Pea reads sources directly. |
| drop | ^\.\. Searching for a P compiled file | whole line removed | Upstream discovers compiled output and logs compilation progress; Snap Pea reads sources directly. |
| drop | ^\.\. Found a P compiled file | whole line removed | Upstream discovers compiled output and logs compilation progress; Snap Pea reads sources directly. |
| drop | ^\.\. Checking | whole line removed | Upstream discovers compiled output and logs compilation progress; Snap Pea reads sources directly. |
| drop | ^==== Loading project file | whole line removed | Upstream discovers compiled output and logs compilation progress; Snap Pea reads sources directly. |
| drop | ^\.+ includes (p|foreign) file | whole line removed | Upstream discovers compiled output and logs compilation progress; Snap Pea reads sources directly. |
| drop | ^-{40}$ | whole line removed | Upstream discovers compiled output and logs compilation progress; Snap Pea reads sources directly. |
| drop | ^Parsing \.\.\.$ | whole line removed | Upstream discovers compiled output and logs compilation progress; Snap Pea reads sources directly. |
| drop | ^Type checking \.\.\.$ | whole line removed | Upstream discovers compiled output and logs compilation progress; Snap Pea reads sources directly. |
| mask | Elapsed [0-9.E-]+ sec( and used [0-9.]+ GB)?\. | Elapsed <t>. | Elapsed time and memory depend on the runtime and machine. |
| mask | time_seconds:[0-9.E-]+ | time_seconds:<t> | Elapsed time and memory depend on the runtime and machine. |
| mask | memory_max_mb:[0-9.E-]+ | memory_max_mb:<m> | Elapsed time and memory depend on the runtime and machine. |
| mask | \n at [^\n]*(?:\n at [^\n]*)*(</Exception(?:Thrown|Handled)>) | ${1} | Stack frames describe upstream's .NET implementation, not the P model. |
| mask | (?m)^ at [^\n]*(\n at [^\n]*)* | empty | Stack frames describe upstream's .NET implementation, not the P model. |
| mask | at (?:[^"\\]|\\[^n])*(?:\\n at (?:[^"\\]|\\[^n])*)* | empty | Stack frames describe upstream's .NET implementation, not the P model. |
| mask | task '[0-9]+' | task '<n>' | Task IDs come from .NET's thread pool. |
| mask | task \\u0027[0-9]+\\u0027 | task \u0027<n>\u0027 | Task IDs come from .NET's thread pool. |
In addition, both sides remove a leading UTF-8 byte order mark and replace the working directory with <dir>: encoding markers and temporary paths do not describe model behavior. The recorder moves compile warnings to the beginning of check output, because Snap Pea compiles when checking. No other check output is ignored.
PObserve normalization
- Strip ANSI colors and replace the output directory with
<out>to remove terminal styling and temporary paths. - Remove configuration lines for
outputDir,supplierJars,pprojandpfiles, since the tools load specs differently. - Mask total runtime and per-event average and maximum times in metrics. Verified event counts remain exact.
- Remove Java stack frames from parser and unknown-error files, since they describe generated Java rather than the P spec.
- Sort console error blocks and mask their numbered filenames; sort error-file contents and renumber files. Upstream checks keys in parallel, so error order and numbering vary.
The executable definitions are in compat_observe_upstream.rs and compare-pobserve.py.
Known differences
Check cases still compare every recorded field, but are expected to differ. The test fails if a listed case starts matching, prompting removal of its exception.
| Check case | Reason |
|---|---|
cli/check-help | check usage lists --pproj, --pfiles and --projname (check reads P sources; upstream: options of p compile) |
cli/check-bad-option | check usage lists --pproj, --pfiles and --projname |
cli/xml-trace-bad-value | check usage lists --pproj, --pfiles and --projname |
cli/no-command | there is no compile command; the tool lists check, observe and merge-scenario-coverage |
cli/unknown-command | there is no compile command; the tool lists check, observe and merge-scenario-coverage |
cli/illegal-project-name | the error does not point to `p compile --help`, since there is no compile command |
These observe runs compare only exit codes. The null-dereference message refers to upstream's generated Java variables.
| Observe run | Reason |
|---|---|
test_case_7_log_11_1_happy_unsorted.sorted | the NullPointerException message names a variable of the generated Java code ("TMP_tmp2_12" is null) |
test_case_7_log_12_2_happy_unsorted.sorted | the NullPointerException message names a variable of the generated Java code ("TMP_tmp2_12" is null) |
What this does not cover
Finite schedule counts and step bounds measure the recorded executions, not every possible execution of every model. Strategies are sampled by the option cases. PEx mode, the PVerifier tutorials, the PriorityQueue tutorial with C# foreign code, and Liveness/DynamicError/InfiniteLoopInAtomicBlock are not recorded; upstream hangs on the last one. Multiworker schedule ordering, generated code, C#/Java foreign code, and .NET/Java stack frames are outside this comparison. Timings and memory are not performance assertions.
See the compatibility reference for supported features and the validation guide for broader compiler, checker and tutorial tests.
Reproduce the comparison
These commands run Snap Pea against the committed upstream outputs, without rebuilding upstream:
cargo test -p snappea --locked --test compat_check_upstream --test compat_observe_upstream
# Optional: restrict the check corpus by case ID prefix
UPSTREAM_CASES=cli/ cargo test -p snappea --locked --test compat_check_upstreamRead each case.json under goldens/cases for the exact upstream and Snap Pea arguments, exit codes, source directory and command order. Adjacent files contain separate stdout/stderr, output paths and normalized file contents.
To independently reproduce the upstream recordings, follow the recording instructions or manually run Upstream goldens at the pinned commit. Its weekly run compares upstream master with the pin. Observe recording uses vendor-pobserve-goldens.py; the full manual comparison uses the environment and command documented in compare-pobserve.py.