Repository navigation
[fix](nereids) select common queryable backend for colocate buckets - #69589
zhaixiangli wants to merge 1 commit into
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
There was a problem hiding this comment.
Reviewed the full authoritative diff at head a9a902522323df1d4d610dc7c83b296b28cffd67. No substantiated new defect was found. There are no existing inline findings to carry forward and no additional user focus was supplied.
The common-backend intersection enforces the existing downstream requirement that every tablet assigned to a bucket contain the same backend. It fixes selection when the first tablet offers an unsuitable alternative; an empty intersection already made the old assignment fail. Query/load backend requirements, preferred tiers, workload accounting and each tablet's original scan range remain preserved.
Critical checkpoint conclusions:
- Goal and evidence: the implementation selects a queryable backend common to every scanned OLAP tablet in a bucket. Three added JUnit tests cover overlapping replicas, disjoint replicas and a query-disabled common replica. Tests were inspected, not executed.
- Scope and reuse: the change is limited to the selector and its backend-selection tests; the extracted replica helper reuses existing ordering, availability and workload logic.
- Concurrency and locking: the selector is statement-local and fragment assignments run sequentially. No new threads, locks, lock ordering or blocking work under locks is introduced. Backend availability is rechecked during selection as before.
- Lifecycle and initialization: candidate lists are local and short-lived. No new static initialization, ownership cycle or resource lifecycle is introduced.
- Configuration: no configuration is added. Existing ordered-scan and backend-selection settings retain their use sites.
- Compatibility: no public interface, persisted metadata, storage format or wire format changes.
- Thrift/protobuf: no definitions are changed; a release-branch field-ID audit is inapplicable.
- Parallel paths: colocate and single-scan bucket-shuffle assignments satisfy the same final per-tablet constraint. Ordinary scans retain the shared helper; the non-OLAP fallback is preserved. Cloud backend placement is resolved upstream; stable-colocate and single-partition distribution gates were checked.
- Conditions and errors: pruned scan-node buckets are skipped, all real tablets constrain the intersection, and invalid assignments fail explicitly. The catalog comparison uses primitive value equality; supported colocate paths preserve catalog identity. Required query candidates are filtered upstream, and required load candidates are filtered before the singleton shortcut, so intersection does not bypass those policies.
- Test coverage: the main positive and negative cases have focused unit tests. Added tests do not cover multiple tablets/partitions, multiple buckets, pruning, ordered mode, cloud or policy-after-intersection; those paths were inspected through their producers and consumers. No end-to-end or runtime validation was performed.
- Test results: no regression output files are changed. The new JUnit assertions agree with the intended assignment and error behavior; no passing build/test result is claimed.
- Observability: the new failure identifies the bucket with no common backend. Existing per-tablet backend-selection profiling remains; no new distributed operation requiring additional metrics was found.
- Transactions, persistence and failover: no transaction or EditLog mutation is added. Existing scan-range versions are forwarded unchanged.
- Data writes and atomicity: this is read scheduling; no write atomicity or persistent crash-recovery mechanism changes.
- FE/BE propagation: no transmitted variable is added. Final scan parameters still use each tablet's original range.
- Performance: the shrinking candidate list uses O(R) additional state and O(T * R^2) worst-case membership work for T tablets and R replicas, with early termination and no per-tablet map/set. No material performance defect was substantiated.
- Other invariants: visibility, delete-bitmap and BE memory-accounting mechanisms are unchanged. No optimizer rule registration/stage, slot/ExprId, mark-join or physical-property derivation is modified. No additional issue was substantiated in the final missed-case recheck.
Coverage limits: this was a static-only review, as required. No builds, tests or runtime experiments were run. Both changed files and the relevant scheduling, placement, policy and assignment paths were covered; no unresolved candidate or unreviewed changed scope remains.
Review converged after 2 round(s).
What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
For colocate joins, Nereids selected the worker for a bucket from the first tablet of the first non-pruned ScanNode. It did not verify that the selected backend had a queryable replica for every tablet in the same bucket across all participating Olap ScanNodes. When replica layouts differed temporarily, assignment could fail later with
Can not find tablet ... in the bucketeven though another backend was common to all tablets.This PR intersects the queryable replica backends of all tablets in the current bucket across all participating Olap ScanNodes, then applies the existing backend preference and minimum-workload selection to that common candidate list. If no common queryable backend exists, it reports an explicit analysis error. The intersection keeps a single candidate list bounded by the replica count and does not allocate a Map or Set per tablet.
Release note
Fix Nereids colocate bucket scheduling to select a queryable backend shared by all scanned tablets.
Check List (For Author)
Test
Added unit tests for mismatched replica layouts, no common queryable backend, and query-disabled common replicas. The tests were not run locally; CI will verify them.
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)