SOLR-18362: remove org.apache.solr.spelling.Token

[serhiy-bzhezytskyy]

https://issues.apache.org/jira/browse/SOLR-18362

Removes the deprecated org.apache.solr.spelling.Token -- a class whose own TODO called it out: "Refactor the spellchecker API to use TokenStreams properly, rather than this hack" (it worked by extending Lucene's internal PackedTokenAttributeImpl, never meant as a general-purpose value holder).

• SpellingQueryConverter/SuggestQueryConverter/SimpleQueryConverter now build a real TokenStream (new package-private QueryWordsTokenStream, analyzing each parsed query word lazily) instead of materializing a Collection<Token> up front
• Every SolrSpellChecker implementation (DirectSolrSpellChecker, AbstractLuceneSpellChecker, Suggester, WordBreakSolrSpellChecker) consumes via incrementToken() and Lucene's own attribute interfaces (CharTermAttribute, OffsetAttribute, FlagsAttribute, etc.), not a pre-built collection
• One plain value type remains, SpellCheckToken (not a Lucene AttributeImpl subclass) -- it's structurally forced by SolrSpellChecker#mergeSuggestions, which correlates suggestions from remote-shard responses by (text, offset) pairs deserialized off the wire, where there's no TokenStream to read from at all
• SpellingOptions.tokens (a Collection<Token>) is now tokenStreamSupplier (a Supplier<TokenStream>): a TokenStream is single-use (reset/incrementToken-loop/end/close), so multiple consumers of the same query's terms (e.g. ConjunctionSolrSpellChecker's sub-checkers, or SpellCheckComponent's own emptiness-check + real use + originalTerms echo) each get a fresh instance rather than sharing/resetting one
• Breaking change for anyone with a custom QueryConverter or SolrSpellChecker registered via solrconfig.xml: convert()'s return type, SpellingOptions's field, and SpellingResult's map key all changed
• Trade-off: the query is now re-analyzed up to 3x per request (peek-for-non-empty, the real suggestion pass, and -- for shard requests -- the originalTerms echo) instead of once. Cheap for short queries; flagging it as a deliberate, known cost rather than leaving it implicit

118 tests, 0 failures (full spelling/suggest package coverage plus SpellCheckComponentTest and the suggest-component tests).

AI-assisted (Claude Sonnet 5)

[epugh]

is this the kind of thing that leveraging solr-benchmark for to confirm the minimal but real perf impact is acceptable would be worth doing?

[serhiy-bzhezytskyy]

Measured it directly against the real SpellingQueryConverter.convert() path (not the full solr-benchmark JMH harness -- just a tight timing loop calling the actual production code):

• documemt (1 word): ~5.3us per analysis pass
• the quick reb fox jumped over the lazy brown dogs (10 words): ~20.7us per pass
• field:value AND another_field:(term1 term2) OR +required -prohibited term (10 words, boolean/field syntax): ~33.4us per pass

Going from 1 pass to 3 (the worst case -- shard requests hit the peek, the real suggestion pass, and the originalTerms echo) adds up to ~67us in the worst case measured. Against a typical query's millisecond-scale round-trip (index search, scoring, network), that's noise.

I don't think the full solr-benchmark harness is needed to answer this specific question, but happy to add a proper benchmark class if you'd like this tracked over time rather than a one-off measurement.

[serhiy-bzhezytskyy]

@epugh does this need a changelog entry, or can you add no-changelog?