Add skipDuplicates parameter on @entity directive#6458
Open
Conversation
Add kw::SKIP_DUPLICATES constant, skip_duplicates bool field to ObjectType, and parsing logic in ObjectType::new() defaulting to false when absent.
Added SkipDuplicatesRequiresImmutable error variant, bool_arg validation for skipDuplicates in validate_entity_directives(), and three test functions covering non-boolean value, mutable entity, and timeseries+skipDuplicates.
Add TypeInfo::skip_duplicates(), InputSchema::skip_duplicates(), and EntityType::skip_duplicates() following the is_immutable() three-layer delegation pattern. Object types return immutable && skip_duplicates; Interface and Aggregation types return false.
Add skip_duplicates: bool field to RowGroup struct alongside immutable, update RowGroup::new() to accept the parameter, and wire it from entity_type.skip_duplicates() in RowGroups::group_entry(). All other call sites (RowGroupForPerfTest, test helpers, example) pass false.
Modify RowGroup::append_row() so that when immutable=true and skip_duplicates=true, cross-block duplicate inserts and Overwrite/Remove operations log warnings and return Ok(()) instead of failing. Same-block duplicates remain allowed. Default behavior (skip_duplicates=false) is preserved exactly. Added Logger field to RowGroup/RowGroups with CacheWeight impl, threaded through all construction sites. 5 unit tests covering all scenarios.
Added skip_duplicates: bool field to Table struct in relational.rs, wired from EntityType::skip_duplicates() in Table::new(), copied in Table::new_like(), and defaulted to false in make_poi_table().
Added logger parameter to Layout::update() and Layout::delete() in relational.rs. When table.immutable && table.skip_duplicates, these methods now log a warning and return Ok(0) instead of returning an error. Default immutable behavior (skip_duplicates=false) is preserved. Updated all callers including deployment_store.rs and test files. Added 4 unit tests with SkipDupMink entity type to verify both skip_duplicates and default immutable behavior.
Add conditional ON CONFLICT (primary_key) DO NOTHING clause to InsertQuery::walk_ast() when table.immutable && table.skip_duplicates. This handles cross-batch duplicates where an entity committed in a previous batch is re-inserted due to the immutable entity cache skipping store lookups. Two unit tests verify: skip_duplicates inserts include ON CONFLICT, default immutable inserts do not.
Restructured Layout::insert() from if-let-Err to match to capture affected_rows from InsertQuery::execute(). Tracks expected vs actual row counts across both the main batch path and the row-by-row fallback path. Logs a warning when affected_rows < expected for skip_duplicates immutable tables.
Add end-to-end runner test exercising @entity(immutable: true, skipDuplicates: true) with duplicate entity inserts across blocks. - tests/runner-tests/skip-duplicates/: subgraph with Ping entity using skipDuplicates, block handler that saves same ID every block - tests/tests/runner_tests.rs: skip_duplicates() test syncs 4 blocks and verifies entity is queryable (indexing did not fail)
isum
force-pushed
the
ion/skip-duplicates
branch
from
3a426e4 to
78c451c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
skipDuplicatesparameter to the@entitydirective for immutable entities.@entity(immutable: true, skipDuplicates: true)skips duplicate inserts with warnings instead of failing indexing. This unblocks subgraph composition on Amp-powered subgraphs where SQL queries can produce the same entities across block ranges.Changes
skipDuplicatesfrom@entitydirective (requiresimmutable: true)ON CONFLICT (id) DO NOTHINGfor skip-duplicates INSERT queries, with affected-rows loggingDefault behavior (
skipDuplicates: false) is unchanged - duplicates still fail indexing.