Context: Credify/postgresql-extensions runs its per-PR CI against a real, shared RDS instance
(no local Postgres available there). To avoid collisions between concurrent PR builds sharing
that one instance, test-harness role names get a per-build prefix/suffix -- which changes
pgTAP assertion DESCRIPTION TEXT (it embeds the role name), so the committed, unsuffixed
test/expected/*.out can never byte-match via ordinary pg_regress/verify-results diffing,
even when every assertion genuinely passes.
Rather than every extension writing its own ad hoc script to scan test/results/*.out for
"not ok" lines / plan-line presence / plan mismatches while ignoring description-text diffs
(which is what we ended up doing -- see the rds-verify-results-equivalent script in that
repo), it'd be better for pgxntool itself to offer a pgTAP-aware results validator as an
alternative to base pg_regress diffing: something that understands pgTAP's own TAP output
(plan lines, ok/not ok, TODO/SKIP directives) well enough to verify correctness independent of
description text, rather than relying on regression.diffs' literal byte-for-byte comparison.
Any extension using pgTAP against a non-fixed-identity test environment (per-build suffixed
role/object names, or similar) would benefit from this rather than reinventing it.
Not scoped yet -- flagging the need; happy to discuss what the validator's actual pass/fail
contract should look like.
Context: Credify/postgresql-extensions runs its per-PR CI against a real, shared RDS instance
(no local Postgres available there). To avoid collisions between concurrent PR builds sharing
that one instance, test-harness role names get a per-build prefix/suffix -- which changes
pgTAP assertion DESCRIPTION TEXT (it embeds the role name), so the committed, unsuffixed
test/expected/*.outcan never byte-match via ordinarypg_regress/verify-resultsdiffing,even when every assertion genuinely passes.
Rather than every extension writing its own ad hoc script to scan
test/results/*.outfor"not ok" lines / plan-line presence / plan mismatches while ignoring description-text diffs
(which is what we ended up doing -- see the
rds-verify-results-equivalent script in thatrepo), it'd be better for pgxntool itself to offer a pgTAP-aware results validator as an
alternative to base
pg_regressdiffing: something that understands pgTAP's own TAP output(plan lines, ok/not ok, TODO/SKIP directives) well enough to verify correctness independent of
description text, rather than relying on regression.diffs' literal byte-for-byte comparison.
Any extension using pgTAP against a non-fixed-identity test environment (per-build suffixed
role/object names, or similar) would benefit from this rather than reinventing it.
Not scoped yet -- flagging the need; happy to discuss what the validator's actual pass/fail
contract should look like.