MDEV-30073 MDEV-35673 MDEV-40465 outer reference and subquery merge fixes for prepared statements#5441
Open
mariadb-RexJohnston wants to merge 3 commits into
Open
MDEV-30073 MDEV-35673 MDEV-40465 outer reference and subquery merge fixes for prepared statements#5441mariadb-RexJohnston wants to merge 3 commits into
mariadb-RexJohnston wants to merge 3 commits into
Conversation
Summary: Items of type Item_direct_view_ref which are reverted with the
change_item_tree mechanism are involved in permanent optimizer
transformations. This commit ensures that items involved in these
permanent transformations are created during the first execution
and re-used for subsequent executions.
Queries affected by this bug are numerous, but will always involve
1) 2nd execution of a prepared statement or procedure
2) a permanent transformation, such as a semi-join optimization
Detail:
Consider this run as a prepared statement
SELECT * FROM t1
WHERE EXISTS
(
SELECT dt.a FROM
(
SELECT t2a as a, t2b as b FROM t2
) dt
WHERE dt.b = t1a
)
During name resolution of field dt.b (in the where clause) we end
up calling find_field_in_view()/.../create_view_field().
This is responsible for creating a wrapper around the found Item
(Item_field*)`test`.`t2`.`t2b`
While this Item_direct_view_ref representing 'dt.b' is allocated on
Statement (permanent) memory the change is registered to be reversed
at the end of statement execution. This is odd and contrary to the
permanent nature of this transformation.
Item::exists2in_processor() is called during the preparation in the
first execution.
We transform the query from
select * from t1 where
exists
(
select `test`.`t2`.`t2b` from
(
select `test`.`t2`.`t2a` AS `a`,`test`.`t2`.`t2b` AS `b` from `test`.`t2`
) `dt`
where `test`.`t2`.`t2b` = `test`.`t1`.`t1a`
limit 1
)
select * from t1 where
`test`.`t1`.`t1a` in
(
select `test`.`t2`.`t2b` from
(
select `test`.`t2`.`t2a` AS `a`,`test`.`t2`.`t2b` AS `b` from `test`.`t2`
) `dt`
where 1
)
later, the optimizer merges the derived table dt into it's parent
select * from t1 where
`test`.`t1`.`t1a` in
(
select `test`.`t2`.`t2b` from t2 where 1
)
then this is transformed into a semi-join
select t1.* from t1 semi join t2 on t1a = t2b
At the end of the first execution, the item t2b above is reverted to
dt.b. During the subsequent name resolution of dt.b, it is resolved
t2a, and the semi-join executed corresponds to
select t1.* from t1 semi join t2 on t1a = t2a
causing a different result set.
Initial Author: Igor Babaev
Reformatted and refactored by: Rex Johnston (rex.johnston@mariadb.com)
Add assert to ensure Item_direct_view_refs are not allocated on
the 2nd execution.
…ver crash Split from MDEV-32294, discovered while inspecting how Item_subselect::used_tables_cache is recalculated across the 1st and 2nd executions of a prepared statement. Core name-resolution / used_tables rework: - Maintain SELECT_LEX::outer_references_resolved_here, a statement-memory list of the outer references resolved in each select_lex (relies on MDEV-30073 so these are not freed at end of PS execution), and rewrite Item_subselect::recalc_used_tables() to compute used_tables_cache from it (Item_belongs_to + Field_fixer). - Preserve Item_field::depended_from across executions and use it in fix_fields/fix_outer_field instead of re-running fix_outer_field, so 2nd-execution resolution is stable. - Maintain nest_level/nest_level_base and merged_into during derived and semi-join merges, and update outer_references_resolved_here when a subquery is merged into its parent. find_field_in_tables: wrap a HAVING outer reference in an Item_ref during name resolution (previously done only in Item_field::fix_outer_field), fixing a marked_for_read() assertion on queries such as SELECT 1 FROM (SELECT a FROM t1) b HAVING (SELECT b.a)=1 create_view_field: resolve view-field substitutions against current_select and allocate them on statement memory. Ban EXPLAIN EXTENDED under the mtr --ps-protocol (warning output differs because some select_transformers do not run during PREPARE). Tests: introduce main.outer_reference with ~90 labelled cases and the execute_various_ways.inc harness, which runs each query six ways (direct, prepare+execute twice, derived table, view, CTE, stored procedure) and, via include/evw_capture.inc, cross-checks that all six return identical rows.
In setup_fields() we call item->update_used_tables() before split_sum_func so that used_tables() is not consulted before the caches are set up on the 2nd execution of a prepared statement (fixes a 1st/2nd execution result mismatch in main.subselect_nulls under --ps-protocol). That recalculation exposes items whose caches are read while still being (re)built, so add the guards it now depends on: - Item_field::used_tables(): return 0 when field / field->table is not yet set, instead of dereferencing a null pointer. - Item_direct_view_ref::used_tables(): return 0 when the item is not fixed yet, instead of asserting. - Item_func::fix_fields(): reset used_tables_cache/const_item_cache at entry (assignment) instead of asserting they are already clear, so a re-fix is idempotent. Remove select,ps.rdiff / select_jcl6,ps.rdiff / select_pkeycache,ps.rdiff: the extra "resolved in SELECT #1" notes they recorded no longer appear under --ps-protocol, so the select tests now match without an rdiff.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MDEV-30073 Wrong result on 2nd execution of PS for query with NOT EXISTS
Summary: Items of type Item_direct_view_ref which are reverted with the
change_item_tree mechanism are involved in permanent optimizer transformations.
This commit ensures that items involved in these permanent transformations are
created during the first execution and re-used for subsequent executions.
MDEV-35673 Correlated subquery problems causing wrong results and server crash
Core name-resolution / used_tables rework:
list of the outer references resolved in each select_lex (relies on
MDEV-30073 so these are not freed at end of PS execution), and rewrite
Item_subselect::recalc_used_tables() to compute used_tables_cache from
it (Item_belongs_to + Field_fixer).
fix_fields/fix_outer_field instead of re-running fix_outer_field, so
2nd-execution resolution is stable.
semi-join merges, and update outer_references_resolved_here when a
subquery is merged into its parent.
MDEV-40465 table map consulted during setup fields on unfixed items