Repository navigation
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
https://github.com/quickwit-oss/quickwit/blob/4b0b29c442503c18e6a4ef6124529f81a2089d95/quickwit-query/src/tokenizers/datadog_v2.rs#L81-L82
When datadog_v2 analyzes a dotted prefix, this emits the whole token first and queues its parts, so the final emitted token is only the last part. PhrasePrefixQuery::get_terms forwards that flat order and build_tantivy_ast_impl makes the last term the prefix (quickwit-query/src/query_ast/phrase_prefix_query.rs:66-71,148-150), while dropping position_length. Thus, on a datadog_v2 field containing foo.bar, an ES match_phrase_prefix or quoted prefix query for foo.ba requires exact foo.ba and foo terms and only expands ba, so it cannot match foo.bar even though the preserved whole token has the requested prefix. Prefix-query construction needs to account for this token graph rather than treating every emitted token as one phrase path.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
883f29c to
438a121
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 438a121f73
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| return Ok(self.zero_terms_query.into()); | ||
| } | ||
| if terms.len() == 1 { | ||
| if terms.len() == 1 && !matches!(self.mode, FullTextMode::BoolPrefix { .. }) { |
There was a problem hiding this comment.
For a datadog_v2 field, a dotted prefix such as foo.ba is reduced to one term by collect_terms, so this new condition routes it into the branch that calls set_max_expansions(max_expansions). The ES match_bool_prefix conversion supplies 50 (elastic_query_dsl/match_bool_prefix.rs:38), even though FullTextMode documents that value as a fuzzy limit rather than a prefix-range limit; consequently, when more than 50 indexed dotted terms share the prefix, Tantivy expands only 50 and silently omits matching documents. Lift the cap for this one-term case just as PhrasePrefixQuery::get_terms does.
Useful? React with 👍 / 👎.
| | `chinese_compatible` | Chop between each CJK character in addition to what `default` does. Should be used with `record: position` to be able to properly search | | ||
| | `lowercase` | Applies a lowercase transformation on the text. It does not tokenize the text. | | ||
| | `datadog`, `unicode_segmenter` | Splits at Unicode word boundaries, lowercases tokens, and applies the 255-byte token-length limit. | | ||
| | `datadog_v2` | Like `datadog`, plus dot-separated parts: `Foo.Bar` → `foo.bar`, `foo`, `bar`. Numeric dotted tokens such as `192.168.1.1` and `32.3` stay whole. | |
There was a problem hiding this comment.
datadog_v2 is also accepted for JSON string values because validate_fields_tokenizers looks up JsonObject tokenizers in the same manager (quickwit/quickwit-doc-mapper/src/doc_mapper/doc_mapper_impl.rs:366-372), but the JSON-field table later on this page still says the only choices are raw, default, en_stem, and chinese_compatible (line 379). This leaves the public configuration documentation internally contradictory and tells users that the new tokenizer cannot be used for JSON or dynamic mappings; update that explicit list or point it to the shared tokenizer table.
AGENTS.md reference: AGENTS.md:L23-L24
Useful? React with 👍 / 👎.
dayaffe
added
area:doc-mapping
This branch has not been deployed
2 participants
Summary
Add
datadog_v2so dotted names can be searched by their individual parts as well as their complete token. Numeric dotted values, such as IP addresses and decimals, stay whole.Changes
datadog_v2, using Unicode word boundaries and emitting each original token followed by its dot-separated parts.Examples:
Foo.Bar.Bazfoo.bar.baz,foo,bar,bazv1.2.3v1.2.3,v1,2,3192.168.1.1192.168.1.132.332.3Adoption
Opt in with
tokenizer: datadog_v2. Existingdatadogbehavior stays unchanged. Existing documents need reindexing to gain the additional tokens. Dotted nonnumeric tokens produce additional postings, increasing index size.Verification
make fmt— passed.cargo nextest run -p quickwit-query— all 188 tests passed.git diff --check— passed.