Skip to content

pgTAP-aware results validator, tolerant of suffixed/prefixed role names in assertion descriptions #106

Description

@jnasbyupgrade

Context: Credify/postgresql-extensions runs its per-PR CI against a real, shared RDS instance
(no local Postgres available there). To avoid collisions between concurrent PR builds sharing
that one instance, test-harness role names get a per-build prefix/suffix -- which changes
pgTAP assertion DESCRIPTION TEXT (it embeds the role name), so the committed, unsuffixed
test/expected/*.out can never byte-match via ordinary pg_regress/verify-results diffing,
even when every assertion genuinely passes.

Rather than every extension writing its own ad hoc script to scan test/results/*.out for
"not ok" lines / plan-line presence / plan mismatches while ignoring description-text diffs
(which is what we ended up doing -- see the rds-verify-results-equivalent script in that
repo), it'd be better for pgxntool itself to offer a pgTAP-aware results validator as an
alternative to base pg_regress diffing: something that understands pgTAP's own TAP output
(plan lines, ok/not ok, TODO/SKIP directives) well enough to verify correctness independent of
description text, rather than relying on regression.diffs' literal byte-for-byte comparison.

Any extension using pgTAP against a non-fixed-identity test environment (per-build suffixed
role/object names, or similar) would benefit from this rather than reinventing it.

Not scoped yet -- flagging the need; happy to discuss what the validator's actual pass/fail
contract should look like.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions