feat(StreamSearch): add stream search on stores by forshev · Pull Request #496 · ozontech/seq-db

forshev · 2026-08-13T14:05:44Z

Description

Add StreamSearch method on stores
Add Executors
Serve StreamSearch with executors

Fixes #483

I have read and followed all requirements in CONTRIBUTING.md;
I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

github-actions · 2026-08-13T14:06:10Z

❌ PR Title Validation Failed
Please refer to CONTRIBUTING.md

codecov-commenter · 2026-08-13T14:09:30Z

Codecov Report

❌ Patch coverage is 68.20941% with 419 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.39%. Comparing base (58f9de1) to head (3c42f9d).

Files with missing lines	Patch %	Lines
proxy/search/stream_search.go	51.66%	94 Missing and 22 partials ⚠️
storeapi/grpc_stream_search.go	61.23%	83 Missing and 24 partials ⚠️
storeapi/client.go	0.00%	71 Missing ⚠️
proxyapi/grpc_stream_search.go	53.65%	50 Missing and 7 partials ⚠️
query/exec/datasource.go	85.71%	12 Missing and 7 partials ⚠️
query/exec/merger.go	88.81%	12 Missing and 5 partials ⚠️
parser/seqql_pipes.go	77.77%	4 Missing and 4 partials ⚠️
query/record.go	84.00%	6 Missing and 2 partials ⚠️
query/exec/aggregator.go	92.94%	4 Missing and 2 partials ⚠️
query/exec/filter.go	92.30%	2 Missing and 2 partials ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #496      +/-   ##
==========================================
- Coverage   71.55%   71.39%   -0.16%     
==========================================
  Files         239      249      +10     
  Lines       20145    21359    +1214     
==========================================
+ Hits        14415    15250     +835     
- Misses       4649     4961     +312     
- Partials     1081     1148      +67

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-08-13T14:14:59Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Indexer-4`	`a73114`	`bedad6`
	`678752766.00 B/op`	`758284750.00 B/op`	`1.12`	`🔴`
`MutexListAppend-4`	`a73114`	`bedad6`
	`194.53 MB/s`	`167.62 MB/s`	`0.86`	`🔴`
	`83148539.00 ns/op`	`95451518.00 ns/op`	`1.15`	`🔴`

github-actions · 2026-08-14T10:25:35Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Indexer-4`	`a73114`	`d1db91`
	`678752766.00 B/op`	`780674889.00 B/op`	`1.15`	`🔴`

github-actions · 2026-08-14T19:46:06Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Indexer-4`	`a73114`	`a92f9b`
	`678752766.00 B/op`	`762860470.00 B/op`	`1.12`	`🔴`
`MutexListAppend-4`	`a73114`	`a92f9b`
	`194.53 MB/s`	`164.33 MB/s`	`0.84`	`🔴`
	`83148539.00 ns/op`	`97365690.00 ns/op`	`1.17`	`🔴`

github-actions · 2026-08-18T09:40:39Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`MutexListAppend-4`	`a73114`	`f83852`
	`194.53 MB/s`	`173.83 MB/s`	`0.89`	`🔴`

dkharms

In overall I like the idea. Would be interesting to see how it will evolve :)

dkharms · 2026-09-01T11:02:57Z

+}
+
+message ResponseHeader {
+  repeated Typing typing = 1;


Do I understand correctly that ResponseHeader (and therefore type information) describes all values that are returned from store as Record?

So, for example, if result of aggregation, like (I am using PRQL syntax here):

filter foo == "bar" group {service, handler} ( aggregate {avg_duration = average duration} )

header will contain [(avg_duration, FLOAT64); (service, STRING); (handler, STRING)]? And each record must satisfy this constraint, right?

this is correct

dkharms · 2026-09-01T11:05:16Z

+  ControlAction action = 1;
+}
+
+enum ControlAction {


IIRC, we were thinking about pull-based model where seqproxy will pull data from store in blocks so it could have more control on memory usage. Should we add PULL control action here as well? Or we rely on gRPC protocol and stream.Recv?

P.S. Sorry, I am just starting reviewing this PR so I may ask stupid questions.

yes, stream.Recv acts as a pull command

dkharms · 2026-09-01T11:10:53Z

+  repeated Typing typing = 1;
+}
+
+message Typing {


Also I believe we should add (maybe even here) encoding we've used for this type (or block of records). What do you think? Of course, it is out of scope of this PR.

we will need it if there will be multiple ways to encode the same data, which is possible

dkharms · 2026-09-01T11:36:35Z

+}
+
+message Record {
+  repeated bytes raw_data = 1;


Why Record consists of array of byte arrays? Why not just single byte array?

each inner array represents single "field" of a "table". it's easier to pack and unpack this way
and we get partial unpacking of a record out-of-the-box

dkharms · 2026-09-01T13:02:29Z

+	done bool
+}
+
+func NewMerger(


I do not think that this should be part of public API -- NewNMergedProducers is enough.

dkharms · 2026-09-01T13:02:55Z

+	}
+}
+
+func NewNMergedProducers(


Why not NewDistributedMerger?

We might want to move each individual fraction to a producer (when we implement timestamp binning) and merge them into a single stream at the store using the same executor.

dkharms · 2026-09-01T13:13:17Z

+import (
+	"cmp"
+
+	insaneJSON "github.com/ozontech/insane-json"


I think usage of insane-json for now is just a overcomplication -- it brings to the table a lot of unnecessary memory-management.

maybe we'll replace insane-json with something else

dkharms · 2026-09-01T13:24:11Z

+	DataTypeUint32
+	DataTypeUint64
+	DataTypeInt32
+	DataTypeInt64


Maybe we can reduce to UINT, INT and FLOAT? And make them 64 bits by default.

it's possible

forshev changed the title ~~StreamSearch stores~~ feat(StreamSearch): add stream search on stores Aug 13, 2026

forshev marked this pull request as draft August 13, 2026 14:07

forshev force-pushed the 483-stream-search-stores branch from d37021e to cc4aa61 Compare August 14, 2026 10:16

forshev force-pushed the 483-stream-search-stores branch from cc4aa61 to c407209 Compare August 14, 2026 14:27

forshev marked this pull request as ready for review August 14, 2026 14:27

forshev force-pushed the 483-stream-search-stores branch 2 times, most recently from 0225e10 to b64b851 Compare August 14, 2026 19:30

forshev force-pushed the 483-stream-search-stores branch 3 times, most recently from ddeba26 to 71187db Compare August 15, 2026 20:25

eguguchkin modified the milestone: v0.78.0 Aug 17, 2026

eguguchkin requested review from cheb0 and eguguchkin August 17, 2026 11:48

forshev force-pushed the 483-stream-search-stores branch 3 times, most recently from 2d709f3 to ea640c6 Compare August 17, 2026 13:38

eguguchkin requested review from dkharms and removed request for cheb0 August 24, 2026 11:16

forshev added 2 commits August 27, 2026 16:50

fix: fix stream search response proto

4b56f76

feat(StreamSearch): add stream search method on stores

0d42fed

forshev force-pushed the 483-stream-search-stores branch from ea640c6 to 0d42fed Compare August 27, 2026 11:58

eguguchkin reviewed Aug 28, 2026

View reviewed changes

eguguchkin approved these changes Aug 28, 2026

View reviewed changes

review fixes

3c42f9d

dkharms approved these changes Sep 1, 2026

View reviewed changes

Uh oh!

Conversation

forshev commented Aug 13, 2026

Description

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

codecov-commenter commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Aug 13, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Aug 14, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Aug 14, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Aug 18, 2026

🔴 Performance Degradation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkharms left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

forshev Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

forshev Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Aug 13, 2026 •

edited

Loading

forshev Sep 1, 2026 •

edited

Loading

forshev Sep 1, 2026 •

edited

Loading