feat: Enforce ClickStack schemas by default

[wrn14897]

• Introduce a new flag HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA (default to false) to otel collector
• Custom ClickStack schemas should be enforced by default
• ClickHouse tables migration logs should be stored in clickstack_db_version_xxx tables
• The collector will run the migration at startup and retry if it fails to connect to the database (using exponential backoff).
• Fully backward compatible

Ref: HDX-3301

[changeset-bot]

🦋 Changeset detected

Latest commit: 1492367

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/otel-collector	Minor
@hyperdx/api	Minor
@hyperdx/app	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-v2-oss-app	Ready	Preview, Comment	Jan 30, 2026 10:16pm

[claude]

Code Review

Critical Issues

❌ Schema breaking change: Removed materialized columns from hyperdx_sessions table (docker/otel-collector/schema/sessions/00001_hyperdx_sessions.sql:2-35) → Existing queries and indexes referencing __hdx_materialized_rum.sessionId and __hdx_materialized_type will fail. Add migration script to drop dependent indexes and columns if they exist.

❌ Index breaking change: Removed idx_rum_session_id index and changed idx_body to idx_lower_body with different function → Queries using these indexes will fail or perform poorly. Need migration to handle existing tables.

⚠️ Migration ordering risk: Directory iteration order may not be guaranteed lexicographic on all systems (entrypoint.sh:37) → Explicitly sort directories or ensure _init runs first before other migrations.

⚠️ Password in connection string: Database password passed in plain URL (entrypoint.sh:20-21) → While this is internal Docker networking, consider if logs/process listings could expose credentials.

Suggestions

🔍 Missing goose down migrations: Only goose Up defined, no rollback path → Consider adding down migrations for future schema changes.

📝 Type inconsistency: TypeScript interface defines create_schema: string but uses boolean-like values (opampController.ts:204-205, 220-221) → Should be create_schema: boolean or clarify string representation.

Note: The TODO comment in docker-compose.yml:35 suggests this is a known migration path, but ensure coordination with existing deployments.

[github-actions]

E2E Test Results

✅ All tests passed • 61 passed • 4 skipped • 763s

Status	Count
✅ Passed	61
❌ Failed	0
⚠️ Flaky	0
⏭️ Skipped	4

Tests ran across 4 shards in parallel.

View full report →

[wrn14897]

Not related to this PR. But I think the TTL should be configurable. I will address it in later PRs

[wrn14897]

Will flip this flag in a later PR