Skip to content

fix: show real values without float noise and keep row matching - #144

Open
verbaux wants to merge 4 commits into
TabularisDB:mainfrom
verbaux:fix/float4-json-precision
Open

verbaux wants to merge 4 commits into
TabularisDB:mainfrom
verbaux:fix/float4-json-precision

Conversation

@verbaux

@verbaux verbaux commented Oct 9, 2026

Copy link
Copy Markdown

Refs TabularisDB/tabularis#906

Summary

PostgreSQL real (float4) values reached the host widened to f64, so 89.9 showed as 89.9000015258789.

  • Display: scalars, arrays and composite fields share one FLOAT4 conversion that serializes the shortest f32 decimal. For the two midpoint values (±7.038531e-26), the exact widened value is used instead, so the JSON number converts back to the stored f32. FLOAT8, NULL and NaN/Infinity (still null) are unchanged.
  • Row matching: when the declared column type is REAL/FLOAT4, the numeric WHERE value is cast to real (CAST(CAST($n AS double precision) AS real)). UPDATE, DELETE and BLOB lookup keep finding the row after the shorter value comes back, including composite keys and tables without a primary key.
  • Diagnostics: the CRUD and BLOB handlers log a warning when the column type lookup fails, since the row may then not match. The plugin does not install a logger yet, so these warnings stay silent until it does, like the existing log::warn! calls.
  • Tests: the FLOAT4 identity live test drops its table before creating it, so a failed run does not break the next one.

Companion application PR: TabularisDB/tabularis#950 (also covers foreign-key navigation).

Verification

  • cargo test --lib: 402 passed, 3 ignored.
  • Live tests against PostgreSQL 18 with the release binary: 32 passed, 2 ignored.
  • Cross-repository parity against the application branch: 84 passed.
  • Coverage: scalars, arrays, composites, finite extremes, both midpoint values, single and composite real keys through JSON.
  • Release build, cargo clippy --all-targets -- -D warnings and cargo fmt --check pass.

Use the shortest float4 decimal representation for scalars, arrays, and composite fields. Compare REAL row identities at column precision so normalized values still support updates and deletes.
Avoid double rounding at f32 midpoints, report missing column metadata, and cover REAL aliases and primary keys through JSON-RPC.
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown

Version suggestion

Based on this PR's title (fix) and the prerelease:rc label:

Current 1.0.0-rc.7
Suggested next tag v1.0.0-rc.8

This is informational only — no tag or release is created automatically yet.

@aesslinger

Copy link
Copy Markdown
Collaborator

@verbaux — really appreciate you taking the initiative on this one. The float noise issue (89.9 → 89.9000015258789) is a real user-facing pain point and this is a thorough, well-researched fix — the midpoint handling in float4_to_json is a particularly nice piece of work, and the CRUD row-matching coverage across keyless / single-PK / composite-PK tables is excellent test design.

We ran two independent review passes (an 11-dimension ensemble pre-handoff review and a focused code-review) and aggregated the results. Here's the summary:

Blocking (must address before merge)

1. Sequencing: must merge after companion PR #950

The float4_to_json change produces 89.9 where the builtin driver (on current main) still produces 89.9000015258789. The parity suite (parity_execute_query_all_types) compares these byte-for-byte, so this PR would go RED against current main. Per the repo's CLAUDE.md: "behavioral differences are regressions here, not fixes."

Once tabularis#950 lands, the builtin also produces short decimals and parity is restored. So the merge order is: #950 first, then #144 (or simultaneously). We'll hold this PR until #950 is on main, then re-run the parity suite to confirm green.

2. Value::String PK path doesn't cast to real

The new Value::Number arm in bind_pk_value wraps the bind in CAST(... AS real) for REAL/FLOAT4 columns, but the Value::String arm (used for keyless tables where all columns serialize as strings) routes through bind_pg_numeric_string which casts to double precision — no CAST(... AS real) wrapper. A keyless table with a REAL column where the row-identity value arrives as a JSON string would fail to match (double precision 89.9 ≠ 89.9000015258789 widened real). The number path was fixed; the string path was not.

Nice-to-haves (low severity, not blocking)

  • Stale comments (src/extract.rs:56, 194): extract_simple_kind and extract_array_kind doc comments say "unchanged from the pre-restructure flat match" — now stale since the FLOAT4 arms changed to float4_to_json.
  • Vacuous unit test (src/utils/values_tests.rs): finite_float4_extremes_round_trip_without_losing_precision only asserts round-trip (which both the old and new code satisfy), not the short-decimal output that distinguishes the fix. A non-vacuous assertion would check e.g. float4_to_json(89.9) == json!(89.9).
  • Midpoint guard untested: the else { f64::from(value) } branch in float4_to_json is exercised by the test but not distinguished from the if branch — removing the guard wouldn't fail any test.
  • Logging branches untested: the three unwrap_or_else(|error| { log::warn!(...) }) sites in crud.rs and blob.rs have no test triggering a get_column_types_map failure.
  • Test table cleanup (tests/live_db.rs): if DROP TABLE fails between the three primary_key loop iterations, stale rows could accumulate. Low risk since DROP IF EXISTS is used, but a defensive TRUNCATE or DELETE before each insert would harden it.

What's clean

The rest of the PR is solid — security (no injection via CAST, no data leakage in logs), concurrency (no resource cleanup issues), error handling (NaN/Infinity → Null is correct, pre-existing swallow pattern unchanged), backward compatibility (documented intentional fix), test structure (all tests properly attributed), boundary cases (all f32 edge values verified correct), and validation (REAL/FLOAT4 case-insensitivity is correct).

@TabularisDB TabularisDB deleted a comment from github-actions Bot Oct 9, 2026
Add short-decimal output coverage and remove stale extraction comments.
@verbaux

verbaux commented Oct 10, 2026

Copy link
Copy Markdown
Author

@aesslinger thanks for the review.

  • Sequencing: agreed, this waits for fix(postgres): show real values without float noise and keep row matching tabularis#950.
  • String PK path: fixed in 05f4830, with a unit test and a live test for a two-column keyless update.
  • Stale comments: removed in src/extract.rs.
  • Short-decimal unit test: added (float4_preserves_short_decimal_values).
  • Midpoint guard: it is covered. finite_float4_extremes_round_trip_without_losing_precision fails without the guard for ±7.038531e-26. I checked every finite f32, and these are the only two values that need it.
  • Logging branches: the plugin does not install a logger yet, so there is nothing observable to assert.
  • Test cleanup: each iteration runs DROP TABLE IF EXISTS before CREATE, so leftover rows from a failed run cannot carry over.

@aesslinger

Copy link
Copy Markdown
Collaborator

@verbaux — thanks for the follow-up! The row_value closure approach is a clean way to handle this — wrapping both the Value::Number and Value::String paths in a single place so they can't drift apart again. The keyless-table live test (float4_keyless_string_identity_updates_real_value) is exactly the right coverage for the gap we flagged.

Our blocking concern (the string PK path) is resolved. This PR is ready for merge once tabularis#950 lands — that's the sequencing dependency we noted earlier (the builtin must produce short decimals first so the parity suite stays green).

If you'd like to address any of the non-blocking nice-to-haves, we'd welcome them but they're not required for merge:

  • The stale "unchanged from the pre-restructure flat match" comments in extract.rs (FLOAT4 arms changed)
  • The vacuous unit test in values_tests.rs (only checks round-trip, not the short-decimal output)
  • The log::warn! branches in crud.rs / blob.rs have no test coverage (no test triggers a get_column_types_map failure)

Appreciate the effort on this one — it's a quality fix for a real user-facing issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working prerelease:rc Version suggestion targets a release candidate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants