Windows x86_64 support, rustls TLS migration, and vector-index perf - #6
Merged
Conversation
The graph explorer crashed ("Index out of bounds: 1 at 215:38") and rendered
a disconnected pile of nodes on graphs whose keys contain colons — e.g. the
edifice_rag code graph with ids like "external:Some.Ns" and
"file:api:...:Foo.cs".
_plain_id stripped the "<db>:" cursor prefix by splitting on every ":" and
keeping the last segment, which mangled colon-bearing keys down to a slashless
id. That id crashed _node_entry (parts[1] out of bounds) and never matched an
edge's _from/_to, so every real node was orphaned. Strip only the leading
"<db>:" prefix instead, and harden _node_entry / _fetch_vertex against
slashless ids so a malformed endpoint degrades gracefully instead of throwing.
Client: freeze vis-network physics once the layout stabilizes — re-running the
solver only when new nodes are merged (double-click expand) — so large graphs
(500 nodes, depth 4) settle and hold still instead of drifting forever.
Adds a regression spec covering colon-in-key ids.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
persist_vector_indexes() re-serializes the entire index (all vectors plus the HNSW graph) into one blob, so calling it after every write batch made a bulk load O(batches x index size). Two changes cut that: - Persist at most once per 5s behind a dirty flag, with a shutdown flush via flush_all_stats() so the trailing window survives a graceful restart. Same throttle-on-write + flush-on-shutdown model already used for collection stats. A hard crash can lose at most one window, rebuildable from the documents' embedding fields. - On UPDATE, compare the extracted f32 vectors and skip the delete+reinsert when every index's embedding is unchanged. An incremental graph sync that only rewrites metadata now pays no HNSW churn and doesn't dirty the index into a full re-serialize. Adds coverage for batch-write durability across reopen and for the skip-reindex path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Publishes solidb-windows-amd64.zip alongside the existing Linux/macOS tarballs. The server core needed no porting - TCP listeners, protocol multiplexing, RocksDB paths and the test suite were already portable, and the Unix-only pieces (daemonize, signals, 0600 modes) were already cfg-gated. The blockers were native C dependencies and distribution. TLS moves from native-tls to rustls, which removes OpenSSL from the graph entirely (cargo tree -i openssl-sys now matches nothing) and with it the Perl/NASM/openssl-src build requirement on Windows: - Drop the vendored openssl dep. It was referenced by no Rust code; it only existed to force vendoring onto the openssl-sys that reqwest and tokio-tungstenite pulled in. - reqwest 0.13 and tokio-tungstenite move to rustls. clients/rust-client is a dev-dependency of solidb, so it moves too - otherwise openssl stays in cargo test. - The queue's dev HTTP client drops danger_accept_invalid_hostnames, which only exists on the native-tls backend. Behavior is preserved: under rustls danger_accept_invalid_certs already skips hostname verification. - jsonwebtoken keeps the aws_lc_rs backend (7320477 chose it to drop the vulnerable rsa crate, which the rust_crypto alternative reintroduces). A windows-gated aws-lc-rs with prebuilt-nasm removes the NASM requirement; CMake and MSVC are already on the runner image. - Docker no longer installs libssl3. CI gains the x86_64-pc-windows-msvc release target with LLVM (librocksdb-sys bindgen needs libclang) and zip packaging, plus a windows-check compile job so portability breakage surfaces on PRs rather than at tag time. Also switches sdbql_join_tests off hardcoded /tmp paths to TempDir; they never cleaned up, so they leaked a RocksDB directory per run. Known Windows gaps, documented in the README: --daemon is Unix-only and exits with an error, and .admin_password is written without the owner-only ACL used on Unix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The doc comment said "at most once per second" while VEC_PERSIST_THROTTLE_SECS is 5, contradicting the comment a few lines below that explains why a 1s window would defeat the throttle. Reference the constant instead of restating a number that can drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windows x86_64 builds, the rustls TLS migration, and the vector-index persistence throttle. Keeps the docs-site version pill in sync with Cargo.toml per CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
install-llvm-action with env:true exported CC/CXX=clang++. Because the target is x86_64-pc-windows-msvc, cc-rs emits MSVC-style flags, which plain clang++ rejects: clang++: error: unknown argument: '-EHsc' clang++: error: unknown argument: '-std:c++20' librocksdb-sys only needs libclang for bindgen, not a compiler override, and LLVM is already preinstalled on windows-latest. Set LIBCLANG_PATH and leave CC/CXX unset so RocksDB is compiled by cl.exe. Also drops the LLVM download from both Windows jobs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both flags are documented as overrides ("use instead of name in dump"),
but all three restore paths resolved the target as
record.get("_database").or_else(|| args.database.clone())
so the name embedded in the dump always won and the flag was only
consulted when the dump carried no name. Every dump emits `_database`
and `_collection`, so the fallback was unreachable and both flags were
dead: `solidb-restore -d staging --input prod.dump` silently restored
into `prod`.
Flip the precedence in process_index_record, process_blob_chunk and
process_doc so the CLI flag wins and the dump name is the fallback.
Verified end to end against a throwaway instance: a dump carrying
`_database: fixture_src` restored with `-d fixture_dst` creates only
fixture_dst (documents, collection and index records all follow the
override); fixture_src is never created.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Patch release for the solidb-restore --database/--collection override fix. Keeps the docs-site version pill in sync with Cargo.toml per CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts v0.32.0. Adds Windows x86_64 builds, moves TLS off OpenSSL to rustls, and throttles vector-index persistence.
Windows support
Publishes
solidb-windows-amd64.zipalongside the Linux/macOS tarballs. The server core needed no porting — TCP listeners, protocol multiplexing, RocksDB paths and the test suite were already portable, and the Unix-only pieces (daemonize, signals,0600modes) were alreadycfg-gated. The blockers were native C dependencies and distribution.Known gaps, documented in the README:
--daemonis Unix-only and exits with an error,.admin_passwordis written without the owner-only ACL used on Unix, and FUSE /solidb updateremain Unix-only.TLS: OpenSSL → rustls
OpenSSL is no longer in the dependency graph at all (
cargo tree -i openssl-sysmatches nothing), which removes the Perl/NASM/openssl-srcbuild requirement on Windows.openssldep — it was referenced by no Rust code and only existed to force vendoring onto theopenssl-systhat reqwest and tokio-tungstenite pulled in.clients/rust-clientmoved too: it's a dev-dependency ofsolidb, so otherwise OpenSSL stayed incargo test.danger_accept_invalid_hostnames(native-tls only). Behavior preserved — under rustls,danger_accept_invalid_certsalready skips hostname verification.jsonwebtokenkeeps theaws_lc_rsbackend: 7320477 chose it specifically to drop the vulnerablersacrate, which therust_cryptoalternative reintroduces (RUSTSEC-2023-0071, no fixed version). A windows-gatedaws-lc-rswithprebuilt-nasmremoves the NASM requirement instead.libssl3; building from source no longer needslibssl-dev.Operator-visible: certificate validation now goes through rustls and the platform trust store rather than OpenSSL.
ca-certificatesis still required in the container image.Vector-index performance
persist_vector_indexes()re-serializes the entire index (all vectors + HNSW graph) into one blob, so calling it after every write batch made bulk loads O(batches × index size). Now throttled to at most once per 5s behind a dirty flag with a shutdown flush, and document updates that leave every embedding unchanged skip the delete+reinsert entirely.Verification
clippy --all-targets -D warningsandfmt --checkclean.lddconfirms no libssl linked.windows-checkin this PR is for. A first attempt failed becauseinstall-llvm-actionexportedCC/CXX=clang++while the msvc target makes cc-rs emit MSVC flags (-EHsc,-std:c++20); fixed by setting onlyLIBCLANG_PATHand lettingcl.execompile RocksDB.🤖 Generated with Claude Code