diff --git a/.gitignore b/.gitignore index d674b55e15..e27322b73a 100644 --- a/.gitignore +++ b/.gitignore @@ -92,7 +92,10 @@ hugegraph-server/hugegraph-dist/docker/data/ # AI-IDE prompt files (We only keep AGENTS.md, other files could soft-linked it when needed) # Serena MCP memories -.serena/ +.serena/* +!.serena/project.yml +!.serena/memories/ +.mcp.json # Claude Projects CLAUDE.md CLAUDE_*.md @@ -114,3 +117,4 @@ codeium-instructions.md .ai-instructions.md *.ai-prompt.md WARP.md +.mcp.json diff --git a/.serena/memories/architecture_and_modules.md b/.serena/memories/architecture_and_modules.md index e4f34cbdad..8d6c85888e 100644 --- a/.serena/memories/architecture_and_modules.md +++ b/.serena/memories/architecture_and_modules.md @@ -3,97 +3,33 @@ ## Three-Tier Architecture ### 1. Client Layer -- Gremlin/Cypher query interfaces -- REST API endpoints -- Multiple client language bindings +- Gremlin/Cypher queries, REST APIs, Swagger UI -### 2. Server Layer (hugegraph-server) -- **REST API Layer** (hugegraph-api): GraphAPI, SchemaAPI, GremlinAPI, CypherAPI, AuthAPI -- **Graph Engine Layer** (hugegraph-core): Schema management, traversal optimization, task scheduling -- **Backend Interface**: Abstraction over storage backends +### 2. Server Layer (hugegraph-server, 13 submodules) +- **REST API** (hugegraph-api): GraphAPI, SchemaAPI, GremlinAPI, CypherAPI, AuthAPI, GraphSpaceAPI (distributed only), ManagerAPI (distributed only) +- **Graph Engine** (hugegraph-core): Schema (with TTL update), traversal, task scheduling, GraphSpace multi-tenancy +- **Backend Interface**: Pluggable via `BackendStore` ### 3. Storage Layer -- Pluggable backend implementations -- Each backend extends `hugegraph-core` abstractions -- Implements `BackendStore` interface +- RocksDB (default/embedded), HStore (distributed/production) +- Legacy (≤1.5.0, deprecated, excluded from context): MySQL, PostgreSQL, Cassandra, ScyllaDB, HBase, Palo -## Multi-Module Structure +## Module Structure (7 top-level modules) -The project consists of 7 main modules: +### hugegraph-server (13 submodules) +`hugegraph-core`, `hugegraph-api` (includes `opencypher/`, `space/`), `hugegraph-dist`, `hugegraph-test`, `hugegraph-example`, plus backends: `hugegraph-rocksdb`, `hugegraph-hstore`, `hugegraph-hbase`, `hugegraph-mysql`, `hugegraph-postgresql`, `hugegraph-cassandra`, `hugegraph-scylladb`, `hugegraph-palo` -### 1. hugegraph-server (13 submodules) -Core graph engine, REST APIs, and backend implementations: -- `hugegraph-core` - Core graph engine and abstractions -- `hugegraph-api` - REST API implementations (includes OpenCypher in `opencypher/`) -- `hugegraph-dist` - Distribution packaging and scripts -- `hugegraph-test` - Test suites (unit, core, API, TinkerPop) -- `hugegraph-example` - Example code -- Backend implementations: - - `hugegraph-rocksdb` (default) - - `hugegraph-hstore` (distributed) - - `hugegraph-hbase` - - `hugegraph-mysql` - - `hugegraph-postgresql` - - `hugegraph-cassandra` - - `hugegraph-scylladb` - - `hugegraph-palo` +### hugegraph-pd (8 submodules) +Placement Driver: `hg-pd-core`, `hg-pd-service`, `hg-pd-client`, `hg-pd-common`, `hg-pd-grpc`, `hg-pd-cli`, `hg-pd-dist`, `hg-pd-test` -### 2. hugegraph-pd (8 submodules) -Placement Driver for distributed deployments (meta server): -- `hg-pd-core` - Core PD logic -- `hg-pd-service` - PD service implementation -- `hg-pd-client` - Client library -- `hg-pd-common` - Shared utilities -- `hg-pd-grpc` - gRPC protocol definitions (auto-generated) -- `hg-pd-cli` - Command line interface -- `hg-pd-dist` - Distribution packaging -- `hg-pd-test` - Test suite +### hugegraph-store (9 submodules) +Distributed storage + Raft: `hg-store-core`, `hg-store-node`, `hg-store-client`, `hg-store-common`, `hg-store-grpc`, `hg-store-rocksdb`, `hg-store-cli`, `hg-store-dist`, `hg-store-test` -### 3. hugegraph-store (9 submodules) -Distributed storage backend with RocksDB and Raft: -- `hg-store-core` - Core storage logic -- `hg-store-node` - Storage node implementation -- `hg-store-client` - Client library -- `hg-store-common` - Shared utilities -- `hg-store-grpc` - gRPC protocol definitions (auto-generated) -- `hg-store-rocksdb` - RocksDB integration -- `hg-store-cli` - Command line interface -- `hg-store-dist` - Distribution packaging -- `hg-store-test` - Test suite +### Others +- **hugegraph-commons**: Shared utilities, RPC framework +- **hugegraph-struct**: Data structures (must build before PD/Store) +- **install-dist**: Distribution packaging, license files +- **hugegraph-cluster-test**: Cluster integration tests -### 4. hugegraph-commons -Shared utilities across modules: -- Locks and concurrency utilities -- Configuration management -- RPC framework components - -### 5. hugegraph-struct -Data structure definitions shared between modules. -**Important**: Must be built before PD and Store modules. - -### 6. install-dist -Distribution packaging and release management: -- License and NOTICE files -- Dependency management scripts -- Release documentation - -### 7. hugegraph-cluster-test -Cluster integration tests for distributed deployments - -## Cross-Module Dependencies - -``` -hugegraph-commons → (shared by all modules) -hugegraph-struct → hugegraph-pd + hugegraph-store -hugegraph-core → (extended by all backend implementations) -``` - -## Distributed Architecture (Optional) - -For production distributed deployments: -- **hugegraph-pd**: Service discovery, partition management, metadata -- **hugegraph-store**: Distributed storage with Raft (3+ nodes) -- **hugegraph-server**: Multiple server instances (3+) -- Communication: All use gRPC with Protocol Buffers - -**Status**: Distributed components (PD + Store) are in BETA +## Distributed Deployment (BETA) +PD + Store + Server (3+ nodes each), all gRPC. Docker compose configs in `docker/` directory, using bridge networking (migrated from host mode). diff --git a/.serena/memories/code_style_and_conventions.md b/.serena/memories/code_style_and_conventions.md index b4e344ebae..159920cd3b 100644 --- a/.serena/memories/code_style_and_conventions.md +++ b/.serena/memories/code_style_and_conventions.md @@ -1,91 +1,28 @@ # Code Style and Conventions -## Code Style Configuration -- **EditorConfig**: `.editorconfig` file defines style rules -- **Checkstyle**: `style/checkstyle.xml` defines additional rules and enforcement - -## Core Style Rules (from .editorconfig) - -### General -- Charset: UTF-8 -- End of line: LF (Unix-style) -- Insert final newline: true -- Max line length: 100 characters (120 for XML) -- Visual guides at column 100 - -### Java Files -- Indent: 4 spaces (not tabs) -- Continuation indent: 8 spaces -- Wrap on typing: true -- Wrap long lines: true - -### Import Organization -``` -$* -| -java.** -| -javax.** -| -org.** -| -com.** -| -* -``` -- Class count to use import on demand: 100 -- Names count to use import on demand: 100 - -### Formatting Rules -- Line comments not at first column -- Align multiline: chained methods, parameters in calls, binary operations, assignments, ternary, throws, extends, array initializers -- Wrapping: normal (wrap if necessary) -- Brace forcing: - - if: if_multiline - - do-while: always - - while: if_multiline - - for: if_multiline -- Enum constants: split_into_lines - -### Blank Lines -- Max blank lines in declarations: 1 -- Max blank lines in code: 1 -- Blank lines between package declaration and header: 1 -- Blank lines before right brace: 1 -- Blank lines around class: 1 -- Blank lines after class header: 1 - -### Documentation -- Add `

` tag on empty lines: true -- Do not wrap if one line: true -- Align multiline annotation parameters: true - -### XML Files -- Indent: 4 spaces -- Max line length: 120 -- Text wrap: off -- Space inside empty tag: true - -### Maven -- Compiler source/target: Java 11 -- Max compiler errors: 500 -- Compiler args: `-Xlint:unchecked` -- Source encoding: UTF-8 - -## Lombok Usage -- Version: 1.18.30 -- Scope: provided -- Optional: true - -## License Headers -- All source files MUST include Apache Software License header -- Validated by apache-rat-plugin and skywalking-eyes -- Exclusions defined in pom.xml (line 171-221) -- gRPC generated code excluded from license check - -## Naming Conventions -- Package names: lowercase, dot-separated (e.g., org.apache.hugegraph) -- Class names: PascalCase -- Method names: camelCase -- Constants: UPPER_SNAKE_CASE -- Variables: camelCase +## Style Tools +- `.editorconfig` — Primary style definition +- `style/checkstyle.xml` — Enforcement (note: `hugegraph-style.xml` was removed) +- `.licenserc.yaml` + apache-rat-plugin + skywalking-eyes — License header validation + +## Core Rules +- **Line length**: 100 chars (120 for XML) +- **Indent**: 4 spaces, continuation 8 spaces +- **Charset**: UTF-8, LF line endings, final newline +- **Imports**: Sorted `$*` → `java` → `javax` → `org` → `com` → `*`, no star imports (threshold 100) +- **Braces**: `if`/`while`/`for` forced if multiline, `do-while` always +- **Blank lines**: Max 1 in code/declarations +- **JavaDoc**: `

` on empty lines, no wrap if one line + +## Naming +- Packages: `org.apache.hugegraph.*` (lowercase dot-separated) +- Classes: PascalCase, Methods/Variables: camelCase, Constants: UPPER_SNAKE_CASE + +## License +- All source files require Apache 2.0 header +- gRPC generated code excluded from checks +- Validate: `mvn apache-rat:check -ntp` + `mvn editorconfig:check` + +## Build +- Java 11 target, `-Xlint:unchecked`, Lombok 1.18.30 (provided/optional) +- Swagger: `io.swagger.core.v3:swagger-jaxrs2-jakarta` for REST API docs diff --git a/.serena/memories/ecosystem_and_related_projects.md b/.serena/memories/ecosystem_and_related_projects.md index 4ec094235c..6bb1344d66 100644 --- a/.serena/memories/ecosystem_and_related_projects.md +++ b/.serena/memories/ecosystem_and_related_projects.md @@ -1,63 +1,23 @@ # HugeGraph Ecosystem and Related Projects -## Core Repository (This Project) -**Repository**: apache/hugegraph (server) -**Purpose**: Core graph database engine (OLTP) +## This Repo: apache/hugegraph (server, OLTP) -## Related Repositories +## Ecosystem +| Repo | Purpose | +|------|---------| +| hugegraph-toolchain | Loader, Hubble (visualization), Tools CLI, Java Client | +| hugegraph-computer | OLAP: PageRank, Connected Components, Shortest Path | +| incubator-hugegraph-ai | Graph RAG, KG construction, NL→Gremlin/Cypher | +| hugegraph-doc | Docs & website (hugegraph.apache.org) | -### 1. hugegraph-toolchain -**Repository**: https://github.com/apache/hugegraph-toolchain -**Components**: -- **hugegraph-loader**: Bulk data loading tool -- **hugegraph-hubble**: Web-based visualization dashboard -- **hugegraph-tools**: Command-line utilities -- **hugegraph-client**: Java client SDK - -### 2. hugegraph-computer -**Repository**: https://github.com/apache/hugegraph-computer -**Purpose**: Distributed graph computing framework (OLAP) -**Features**: PageRank, Connected Components, Shortest Path, Community Detection - -### 3. hugegraph-ai -**Repository**: https://github.com/apache/incubator-hugegraph-ai -**Purpose**: Graph AI, LLM, and Knowledge Graph integration -**Features**: Graph-enhanced LLM, KG construction, Graph RAG, NL to Gremlin/Cypher - -### 4. hugegraph-website -**Repository**: https://github.com/apache/hugegraph-doc -**Purpose**: Official documentation and website -**URL**: https://hugegraph.apache.org/ - -## Integration Points - -### Data Pipeline +## Data Flow ``` -Data Sources → hugegraph-loader → hugegraph-server - ↓ - ┌───────────────────┼───────────────────┐ - ↓ ↓ ↓ - hugegraph-hubble hugegraph-computer hugegraph-ai - (Visualization) (Analytics) (AI/ML) +Sources → hugegraph-loader → hugegraph-server → Hubble / Computer / AI ``` -## External Integrations - -### Big Data Platforms -- Apache Flink, Apache Spark, HDFS - -### Storage Backends -- RocksDB (default), HBase, Cassandra, ScyllaDB, MySQL, PostgreSQL - -### Query Languages -- Gremlin (Apache TinkerPop), Cypher (OpenCypher), REST API - -## Version Compatibility -- Server: 1.7.0 -- TinkerPop: 3.5.1 -- Java: 11+ required +## Integrations +- Big Data: Flink, Spark, HDFS +- Queries: Gremlin (TinkerPop 3.5.1), OpenCypher, REST API + Swagger UI +- Storage: RocksDB (default), HStore (distributed) -## Use Cases -- Social networks, Fraud detection, Recommendation systems -- Knowledge graphs, Network analysis, Supply chain management -- IT operations, Bioinformatics +## Version: Server 1.7.0, TinkerPop 3.5.1, Java 11+ diff --git a/.serena/memories/implementation_patterns_and_guidelines.md b/.serena/memories/implementation_patterns_and_guidelines.md index 91f3145f90..c2b98313e2 100644 --- a/.serena/memories/implementation_patterns_and_guidelines.md +++ b/.serena/memories/implementation_patterns_and_guidelines.md @@ -1,104 +1,50 @@ # Implementation Patterns and Guidelines -## Backend Development - -### Backend Architecture Pattern -- All backends extend abstractions from `hugegraph-server/hugegraph-core` -- Implement the `BackendStore` interface -- Each backend is a separate Maven module under `hugegraph-server/` -- Backend selection configured in `hugegraph.properties` via `backend` property - -### Available Backends -- **RocksDB** (default, embedded): `hugegraph-rocksdb` -- **HStore** (distributed, production): `hugegraph-hstore` -- **Legacy** (≤1.5.0): MySQL, PostgreSQL, Cassandra, ScyllaDB, HBase, Palo - -### Backend Testing Profiles -- `memory`: In-memory backend for fast unit tests -- `rocksdb`: RocksDB for realistic local tests -- `hbase`: HBase for distributed scenarios -- `hstore`: HStore for production-like distributed tests - -## gRPC Protocol Development - -### Protocol Buffer Definitions +## Backend Architecture +- Backends implement `BackendStore` interface from `hugegraph-core` +- Each backend = separate Maven module under `hugegraph-server/` +- Configured via `hugegraph.properties` → `backend` property +- **Active backends (focus here)**: RocksDB (default/embedded), HStore (distributed) +- **Legacy backends** (deprecated, excluded from Serena context): MySQL, PostgreSQL, Cassandra, ScyllaDB, HBase, Palo + +## GraphSpace Multi-Tenancy +- Core: `hugegraph-core/.../space/` (GraphSpace, SchemaTemplate, Service, register/) +- API: `hugegraph-api/.../api/space/GraphSpaceAPI.java` (includes GS profile endpoints) +- **Standalone mode**: GraphSpaceAPI and ManagerAPI are disabled + +## Auth System +- Disabled by default, enable via `bin/enable-auth.sh` +- ConfigAuthenticator was removed, use standard auth +- Multi-level: Users, Groups, Projects, Targets, Access control +- Location: `hugegraph-api/.../api/auth/` + +## gRPC Protocol - PD protos: `hugegraph-pd/hg-pd-grpc/src/main/proto/` - Store protos: `hugegraph-store/hg-store-grpc/src/main/proto/` - -### Code Generation -When modifying `.proto` files: -1. Run `mvn clean compile` to regenerate gRPC stubs -2. Generated Java code goes to `*/grpc/` packages -3. Output location: `target/generated-sources/protobuf/` -4. Generated files excluded from Apache RAT checks -5. All inter-service communication uses gRPC - -## Authentication System - -### Default State -- Authentication **disabled by default** -- Enable via `bin/enable-auth.sh` or configuration -- **Required for production deployments** - -### Implementation Location -`hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/auth/` - -### Multi-Level Security Model -- Users, Groups, Projects, Targets, Access control - -## TinkerPop Integration - -### Compliance -- Full Apache TinkerPop 3 implementation -- Custom optimization strategies -- Supports both Gremlin and OpenCypher query languages - -### Query Language Support -- **Gremlin**: Native via TinkerPop integration -- **OpenCypher**: Implementation in `hugegraph-api/opencypher/` - -## Testing Patterns - -### Test Suite Organization -- **UnitTestSuite**: Pure unit tests, no external dependencies -- **CoreTestSuite**: Core functionality tests with backend -- **ApiTestSuite**: REST API integration tests -- **StructureStandardTest**: TinkerPop structure compliance -- **ProcessStandardTest**: TinkerPop process compliance - -### Backend Selection in Tests -Use Maven profiles: -```bash --P core-test,memory # Fast in-memory --P core-test,rocksdb # Persistent local --P api-test,rocksdb # API with persistent backend -``` - -## Distribution and Packaging - -### Creating Distribution -```bash -mvn clean package -DskipTests -``` -Output: `install-dist/target/hugegraph-.tar.gz` - -## Code Organization - -### Package Structure -``` -org.apache.hugegraph -├── backend/ # Backend implementations -├── api/ # REST API endpoints -├── core/ # Core graph engine -├── schema/ # Schema definitions -├── traversal/ # Traversal and query processing -├── task/ # Background tasks -├── auth/ # Authentication/authorization -└── util/ # Utilities -``` - -### Module Dependencies -- Commons is shared by all modules -- Struct must be built before PD and Store -- Backend modules depend on core -- Test module depends on all server modules +- After `.proto` changes: `mvn clean compile` → `target/generated-sources/protobuf/` + +## Query Languages +- **Gremlin**: Native TinkerPop 3.5.1 +- **OpenCypher**: `hugegraph-api/opencypher/` +- TinkerPop exceptions are passed through in Gremlin responses + +## Schema +- Labels support TTL with runtime update +- Edge label conflicting conditions are handled safely + +## Testing +- **Profiles**: `unit-test`, `core-test`, `api-test`, `tinkerpop-structure-test`, `tinkerpop-process-test` +- **Backends in CI**: memory, rocksdb, hbase (matrix) +- **Single test class**: `mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,memory -Dtest=ClassName` +- TinkerPop tests: only on `release-*`/`test-*` branches +- Raft tests: only on `test*`/`raft*` branches + +## Docker +- Single-node: `docker/docker-compose.yml` (bridge network, pd+store+server) +- Cluster: `docker/docker-compose-3pd-3store-3server.yml` +- Container logs: stdout-based + +## CI Pipelines +- `server-ci.yml`: compile + unit/core/API tests (memory/rocksdb/hbase × Java 11) +- `rerun-ci.yml`: auto-rerun flaky failures (max 2 reruns, 180s delay) +- `auto-pr-review.yml`: auto-comment on new PRs diff --git a/.serena/memories/key_file_locations.md b/.serena/memories/key_file_locations.md index fc9c62ec3d..3f2a60dee0 100644 --- a/.serena/memories/key_file_locations.md +++ b/.serena/memories/key_file_locations.md @@ -1,87 +1,44 @@ # Key File and Directory Locations -## Project Root -The project root contains the multi-module Maven structure. - -## Configuration Files - -### Build Configuration -- `pom.xml` - Root Maven POM (multi-module) -- `.editorconfig` - Code style rules -- `style/checkstyle.xml` - Checkstyle rules -- `.licenserc.yaml` - License checker config - -### Documentation -- `README.md` - Project overview -- `BUILDING.md` - Build instructions -- `CONTRIBUTING.md` - Contribution guide -- `AGENTS.md` - AI agent development guide -- `LICENSE` - Apache License 2.0 -- `NOTICE` - Copyright notices - -## Server Module (hugegraph-server) - -### Core Implementation -- `hugegraph-core/src/main/java/org/apache/hugegraph/` - Core engine - - `backend/` - Backend interface - - `schema/` - Schema management - - `traversal/` - Query processing - - `task/` - Background tasks - -### API Layer -- `hugegraph-api/src/main/java/org/apache/hugegraph/api/` - REST APIs - - `graph/` - GraphAPI - - `schema/` - SchemaAPI - - `gremlin/` - GremlinAPI - - `cypher/` - CypherAPI - - `auth/` - AuthAPI - - `opencypher/` - OpenCypher implementation - -### Backend Implementations -- `hugegraph-rocksdb/` - RocksDB backend (default) -- `hugegraph-hstore/` - HStore distributed backend -- `hugegraph-hbase/` - HBase backend -- `hugegraph-mysql/` - MySQL backend -- `hugegraph-postgresql/` - PostgreSQL backend -- `hugegraph-cassandra/` - Cassandra backend -- `hugegraph-scylladb/` - ScyllaDB backend -- `hugegraph-palo/` - Palo backend - -### Distribution and Scripts -- `hugegraph-dist/src/assembly/static/` - Distribution files - - `bin/` - Shell scripts (init-store.sh, start-hugegraph.sh, stop-hugegraph.sh, etc.) - - `conf/` - Configuration files (hugegraph.properties, rest-server.properties, gremlin-server.yaml, log4j2.xml) - - `lib/` - JAR dependencies - - `logs/` - Log files - -### Testing -- `hugegraph-test/src/main/java/org/apache/hugegraph/` - Test suites - - `unit/` - Unit tests - - `core/` - Core functionality tests - - `api/` - API integration tests - - `tinkerpop/` - TinkerPop compliance tests - -## PD Module (hugegraph-pd) -- `hg-pd-core/` - Core PD logic -- `hg-pd-service/` - Service implementation -- `hg-pd-client/` - Client library -- `hg-pd-grpc/src/main/proto/` - Protocol definitions -- `hg-pd-dist/src/assembly/static/` - Distribution files - -## Store Module (hugegraph-store) -- `hg-store-core/` - Core storage logic -- `hg-store-node/` - Storage node -- `hg-store-client/` - Client library -- `hg-store-grpc/src/main/proto/` - Protocol definitions -- `hg-store-dist/src/assembly/static/` - Distribution files - -## Commons Module (hugegraph-commons) -- Shared utilities, RPC framework - -## Struct Module (hugegraph-struct) -- Data structure definitions (must be built before PD and Store) - -## Distribution Module (install-dist) -- `release-docs/` - LICENSE, NOTICE, licenses/ -- `scripts/dependency/` - Dependency management scripts -- `target/` - Build output (hugegraph-.tar.gz) +## Build & Config +- `pom.xml` — Root multi-module POM +- `.editorconfig` — Code style rules +- `style/checkstyle.xml` — Checkstyle enforcement (hugegraph-style.xml was removed) +- `.licenserc.yaml` — License checker config + +## Server (hugegraph-server) +- Core engine: `hugegraph-core/src/main/java/org/apache/hugegraph/` → `backend/`, `schema/`, `traversal/`, `task/` +- GraphSpace: `hugegraph-core/.../space/` → `GraphSpace.java`, `SchemaTemplate.java`, `Service.java`, `register/` +- REST APIs: `hugegraph-api/src/main/java/org/apache/hugegraph/api/` → `graph/`, `schema/`, `gremlin/`, `cypher/`, `auth/`, `space/` (GraphSpaceAPI), `metrics/`, `arthas/` +- OpenCypher: `hugegraph-api/.../opencypher/` +- Backend interface: `hugegraph-core/.../backend/store/BackendStore.java` +- Distribution: `hugegraph-dist/src/assembly/static/` → `bin/`, `conf/`, `lib/`, `logs/` +- Tests: `hugegraph-test/src/main/java/.../` → `unit/`, `core/`, `api/`, `tinkerpop/` + +## Docker +- `docker/docker-compose.yml` — Single-node (bridge network, pd+store+server) +- `docker/docker-compose-3pd-3store-3server.yml` — 3-node cluster +- `docker/docker-compose.dev.yml` — Dev mode + +## PD Module +- Proto: `hugegraph-pd/hg-pd-grpc/src/main/proto/` +- Dist: `hugegraph-pd/hg-pd-dist/src/assembly/static/` + +## Store Module +- Proto: `hugegraph-store/hg-store-grpc/src/main/proto/` +- Dist: `hugegraph-store/hg-store-dist/src/assembly/static/` + +## CI Workflows (.github/workflows/) +- `server-ci.yml` — Server tests (matrix: memory/rocksdb/hbase × Java 11) +- `pd-store-ci.yml` — PD, Store & HStore tests +- `commons-ci.yml` — Commons tests +- `cluster-test-ci.yml` — Cluster integration +- `licence-checker.yml` — License headers +- `rerun-ci.yml` — Auto-rerun for flaky workflows +- `auto-pr-review.yml` — Auto-comment on new PRs +- `check-dependencies.yml` — Dependency checks +- `codeql-analysis.yml` — CodeQL security scanning +- `stale.yml` — Stale issue/PR cleanup + +## Docs +- `README.md`, `BUILDING.md`, `CONTRIBUTING.md`, `AGENTS.md`, `CLAUDE.md` diff --git a/.serena/memories/project_overview.md b/.serena/memories/project_overview.md index 34f402243b..c494aaed44 100644 --- a/.serena/memories/project_overview.md +++ b/.serena/memories/project_overview.md @@ -1,35 +1,27 @@ # Apache HugeGraph Project Overview ## Project Purpose -Apache HugeGraph is a fast-speed and highly-scalable graph database that supports billions of vertices and edges (10+ billion scale). It is designed for OLTP workloads with excellent performance and scalability. +Apache HugeGraph is a fast-speed, highly-scalable graph database supporting 10+ billion vertices/edges for OLTP workloads. Graduated from Apache Incubator (incubating branding removed). ## Key Capabilities -- Graph database compliant with Apache TinkerPop 3 framework -- Supports both Gremlin and Cypher query languages -- Schema metadata management (VertexLabel, EdgeLabel, PropertyKey, IndexLabel) +- Apache TinkerPop 3 compliant graph database +- Gremlin + OpenCypher query languages +- Schema metadata management (VertexLabel, EdgeLabel, PropertyKey, IndexLabel) with TTL update support - Multi-type indexes (exact, range, complex conditions) -- Pluggable backend storage architecture -- Integration with big data platforms (Flink/Spark/HDFS) -- Complete graph ecosystem (computing, visualization, AI/ML) +- Pluggable backend storage (RocksDB default, HStore distributed) +- GraphSpace multi-tenancy (standalone mode disables GraphSpaceAPI/ManagerAPI) +- Swagger UI for REST API documentation +- Integration with Flink/Spark/HDFS ## Technology Stack - **Language**: Java 11+ (required) -- **Build Tool**: Apache Maven 3.5+ (required) +- **Build**: Maven 3.5+ - **Graph Framework**: Apache TinkerPop 3.5.1 -- **RPC**: gRPC with Protocol Buffers -- **Storage Backends**: - - RocksDB (default, embedded) - - HStore (distributed, production) - - Legacy (≤1.5.0): MySQL, PostgreSQL, Cassandra, ScyllaDB, HBase, Palo +- **RPC**: gRPC + Protocol Buffers +- **API Docs**: Swagger (io.swagger.core.v3) +- **Storage**: RocksDB (default/embedded), HStore (distributed/production) +- **Legacy backends** (≤1.5.0): MySQL, PostgreSQL, Cassandra, ScyllaDB, HBase, Palo -## Project Version -- Current version: 1.7.0 (managed via `${revision}` property) -- Version management uses Maven flatten plugin for CI-friendly versioning - -## License -- Apache License 2.0 -- All code must include Apache license headers -- Third-party dependencies require proper license documentation - -## Repository Structure -This is a multi-module Maven project +## Version +- Current: 1.7.0 (`${revision}` property, Maven flatten plugin) +- License: Apache License 2.0 diff --git a/.serena/memories/suggested_commands.md b/.serena/memories/suggested_commands.md index 25b5972b05..346304432f 100644 --- a/.serena/memories/suggested_commands.md +++ b/.serena/memories/suggested_commands.md @@ -1,131 +1,58 @@ # Suggested Development Commands -## Quick Reference - -### Prerequisites Check +## Build ```bash -java -version # Must be 11+ -mvn -version # Must be 3.5+ +mvn clean install -DskipTests # Full build +mvn clean install -pl hugegraph-server -am -DskipTests # Server only +mvn clean compile -U -Dmaven.javadoc.skip=true -ntp # Compile only +mvn clean package -DskipTests # Distribution → install-dist/target/ ``` -### Build Commands +## Test ```bash -# Full build without tests (fastest) -mvn clean install -DskipTests - -# Full build with all tests -mvn clean install - -# Build specific module (e.g., server) -mvn clean install -pl hugegraph-server -am -DskipTests - -# Compile only -mvn clean compile -U -Dmaven.javadoc.skip=true -ntp - -# Build distribution package -mvn clean package -DskipTests -# Output: install-dist/target/hugegraph-.tar.gz -``` - -### Testing Commands -```bash -# Unit tests (memory backend) -mvn test -pl hugegraph-server/hugegraph-test -am -P unit-test,memory - -# Core tests with specific backend +# Server tests (memory/rocksdb/hbase backends) +mvn test -pl hugegraph-server/hugegraph-test -am -P unit-test mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,memory mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,rocksdb -mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,hbase - -# API tests mvn test -pl hugegraph-server/hugegraph-test -am -P api-test,rocksdb -# TinkerPop compliance tests (release branches) -mvn test -pl hugegraph-server/hugegraph-test -am -P tinkerpop-structure-test,memory -mvn test -pl hugegraph-server/hugegraph-test -am -P tinkerpop-process-test,memory - -# Run single test class +# Single test class mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,memory -Dtest=YourTestClass -# PD module tests (build struct first) -mvn install -pl hugegraph-struct -am -DskipTests -mvn test -pl hugegraph-pd/hg-pd-test -am +# TinkerPop compliance (release/test branches only) +mvn test -pl hugegraph-server/hugegraph-test -am -P tinkerpop-structure-test,memory -# Store module tests (build struct first) +# PD/Store (build struct first) mvn install -pl hugegraph-struct -am -DskipTests +mvn test -pl hugegraph-pd/hg-pd-test -am mvn test -pl hugegraph-store/hg-store-test -am ``` -### Code Quality & Validation +## Validation ```bash -# License header check (Apache RAT) -mvn apache-rat:check -ntp - -# Code style check (EditorConfig) -mvn editorconfig:check - -# Compile with warnings -mvn clean compile -Dmaven.javadoc.skip=true +mvn apache-rat:check -ntp # License headers +mvn editorconfig:check # Code style (.editorconfig) +mvn checkstyle:check # Code style (style/checkstyle.xml) +mvn clean compile -Dmaven.javadoc.skip=true # Compile warnings ``` -### Server Operations +## Server Ops +Scripts in `hugegraph-server/hugegraph-dist/src/assembly/static/bin/` (or extracted distribution `bin/`): ```bash -# Scripts location: hugegraph-server/hugegraph-dist/src/assembly/static/bin/ - -# Initialize storage backend -bin/init-store.sh - -# Start HugeGraph server -bin/start-hugegraph.sh - -# Stop HugeGraph server -bin/stop-hugegraph.sh - -# Start Gremlin console -bin/gremlin-console.sh - -# Enable authentication -bin/enable-auth.sh - -# Dump effective configuration -bin/dump-conf.sh - -# Monitor server -bin/monitor-hugegraph.sh +bin/init-store.sh && bin/start-hugegraph.sh # Init + start +bin/stop-hugegraph.sh # Stop +bin/enable-auth.sh # Enable auth ``` -### Git Operations (macOS/Darwin) +## Docker ```bash -# View git log (avoid pager) -git --no-pager log -n 20 --oneline - -# View git diff (avoid pager) -git --no-pager diff - -# Check git status -git status +cd docker && docker compose up -d # Single-node (bridge network) +cd docker && docker compose -f docker-compose-3pd-3store-3server.yml up -d # Cluster ``` -### Docker Commands (Test/Dev) +## Distributed Build (BETA) ```bash -# Start HugeGraph in Docker (RocksDB backend) -docker run -itd --name=graph -p 8080:8080 hugegraph/hugegraph:1.5.0 - -# Start with preloaded sample graph -docker run -itd --name=graph -e PRELOAD=true -p 8080:8080 hugegraph/hugegraph:1.5.0 -``` - -### Distributed Components Build (BETA) -```bash -# 1. Build hugegraph-struct (required dependency) -mvn install -pl hugegraph-struct -am -DskipTests - -# 2. Build hugegraph-pd (Placement Driver) -mvn clean package -pl hugegraph-pd -am -DskipTests - -# 3. Build hugegraph-store (distributed storage) -mvn clean package -pl hugegraph-store -am -DskipTests - -# 4. Build hugegraph-server with HStore backend -mvn clean package -pl hugegraph-server -am -DskipTests +mvn install -pl hugegraph-struct -am -DskipTests # 1. Struct first +mvn clean package -pl hugegraph-pd -am -DskipTests # 2. PD +mvn clean package -pl hugegraph-store -am -DskipTests # 3. Store ``` diff --git a/.serena/memories/task_completion_checklist.md b/.serena/memories/task_completion_checklist.md index c2bffcac99..5e8ec4d82e 100644 --- a/.serena/memories/task_completion_checklist.md +++ b/.serena/memories/task_completion_checklist.md @@ -1,139 +1,32 @@ # Task Completion Checklist -When completing a coding task, follow these steps to ensure quality and compliance: - -## 1. Code Quality Checks (MANDATORY) - -### License Header Check -Run Apache RAT to verify all files have proper license headers: -```bash -mvn apache-rat:check -ntp -``` -Fix any violations by adding the Apache license header. - -### Code Style Check -Run EditorConfig validation: -```bash -mvn editorconfig:check -``` -Fix violations according to `.editorconfig` rules. - -### Compilation Check -Compile with warnings enabled: -```bash -mvn clean compile -Dmaven.javadoc.skip=true -``` -Resolve all compiler warnings, especially unchecked operations. - -## 2. Testing (REQUIRED) - -### Determine Test Scope -Check project README or ask user for test commands. Common patterns: - -#### Server Module Tests -- Unit tests: -```bash -mvn test -pl hugegraph-server/hugegraph-test -am -P unit-test,memory -``` -- Core tests (choose backend): -```bash -mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,memory -mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,rocksdb -``` -- API tests: -```bash -mvn test -pl hugegraph-server/hugegraph-test -am -P api-test,rocksdb -``` - -#### PD Module Tests -```bash -# Build struct dependency first -mvn install -pl hugegraph-struct -am -DskipTests -# Run PD tests -mvn test -pl hugegraph-pd/hg-pd-test -am -``` - -#### Store Module Tests -```bash -# Build struct dependency first -mvn install -pl hugegraph-struct -am -DskipTests -# Run Store tests -mvn test -pl hugegraph-store/hg-store-test -am -``` - -### Run Appropriate Tests -Execute tests relevant to your changes: -- For bug fixes: run existing tests to verify fix -- For new features: write and run new tests -- For refactoring: run all affected module tests - -## 3. Dependencies Management - -If adding new third-party dependencies: - -1. Add license file to `install-dist/release-docs/licenses/` -2. Declare dependency in `install-dist/release-docs/LICENSE` -3. Append NOTICE (if exists) to `install-dist/release-docs/NOTICE` -4. Update dependency list: -```bash -./install-dist/scripts/dependency/regenerate_known_dependencies.sh -``` -Or manually update `install-dist/scripts/dependency/known-dependencies.txt` - -## 4. Build Verification - -Build the affected module(s) with tests: -```bash -mvn clean install -pl -am -``` - -## 5. Documentation (if applicable) - -- Update JavaDoc for public APIs -- Update README if adding user-facing features -- Update AGENTS.md if adding dev-facing information - -## 6. Commit Preparation - -### NEVER Commit Unless Explicitly Asked -- Do NOT auto-commit changes -- Only commit when user explicitly requests it -- This is CRITICAL to avoid surprising users - -### When Asked to Commit -- Write clear commit messages: -``` -Fix bug: - -fix #ISSUE_ID -``` +## 1. Code Quality (MANDATORY) +```bash +mvn apache-rat:check -ntp # License headers +mvn editorconfig:check # Style (.editorconfig) +mvn checkstyle:check # Style (style/checkstyle.xml) +mvn clean compile -Dmaven.javadoc.skip=true # Compile warnings +``` + +## 2. Testing +- Choose backend: `memory` (fast), `rocksdb` (realistic), `hbase` (distributed) +- Single test: `-Dtest=ClassName` works with all profiles +- Bug fix → existing tests; New feature → write tests; Refactor → affected module tests + +## 3. Dependencies (if adding new) +1. License file → `install-dist/release-docs/licenses/` +2. Declare in `install-dist/release-docs/LICENSE` +3. Append NOTICE → `install-dist/release-docs/NOTICE` +4. Run `./install-dist/scripts/dependency/regenerate_known_dependencies.sh` + +## 4. CI Awareness +- `server-ci.yml`: memory/rocksdb/hbase × Java 11 +- `rerun-ci.yml`: auto-retries flaky failures +- `licence-checker.yml`: header validation +- Raft tests: only `test*`/`raft*` branches +- TinkerPop tests: only `release-*`/`test-*` branches + +## 5. Commit +- NEVER commit unless explicitly asked +- Format: `feat|fix|refactor(module): msg` - Include issue ID if available -- Describe what and how the change works - -## 7. Pre-PR Checklist - -Before creating a Pull Request, ensure: -- [ ] All license checks pass -- [ ] All code style checks pass -- [ ] All relevant tests pass -- [ ] Code compiles without warnings -- [ ] Dependencies properly documented (if added) -- [ ] Changes tested locally -- [ ] Commit message is clear and references issue - -## Common CI Workflows - -Your changes will be validated by: -- `server-ci.yml`: Compiles + unit/core/API tests (memory, rocksdb, hbase) -- `licence-checker.yml`: License header validation -- `pd-store-ci.yml`: PD and Store module tests -- `commons-ci.yml`: Commons module tests -- `cluster-test-ci.yml`: Distributed cluster tests - -## Notes - -- **Test Backend Selection**: Use `memory` for quick tests, `rocksdb` for realistic tests, `hbase` for distributed scenarios -- **TinkerPop Tests**: Only run on release branches (release-*/test-*) -- **Raft Tests**: Only run when branch name starts with `test` or `raft` -- **Build Time**: Full build can take 5-15 minutes depending on hardware -- **Test Time**: Test suites can take 10-30 minutes depending on backend diff --git a/.serena/project.yml b/.serena/project.yml index d4fd762ad4..3329e5b5af 100644 --- a/.serena/project.yml +++ b/.serena/project.yml @@ -1,15 +1,23 @@ # list of languages for which language servers are started; choose from: -# al bash clojure cpp csharp csharp_omnisharp -# dart elixir elm erlang fortran go -# haskell java julia kotlin lua markdown -# nix perl php python python_jedi r -# rego ruby ruby_solargraph rust scala swift -# terraform typescript typescript_vts zig +# al bash clojure cpp csharp +# csharp_omnisharp dart elixir elm erlang +# fortran fsharp go groovy haskell +# java julia kotlin lua markdown +# matlab nix pascal perl php +# php_phpactor powershell python python_jedi r +# rego ruby ruby_solargraph rust scala +# swift terraform toml typescript typescript_vts +# vue yaml zig +# (This list may be outdated. For the current list, see values of Language enum here: +# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py +# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.) # Note: # - For C, use cpp # - For JavaScript, use typescript +# - For Free Pascal/Lazarus, use pascal # Special requirements: -# - csharp: Requires the presence of a .sln file in the project folder. +# Some languages require additional setup/installations. +# See here for details: https://oraios.github.io/serena/01-about/020_programming-languages.html#language-servers # When using multiple languages, the first language server that supports a given file will be used for that file. # The first language is the default language and the respective language server will be used as a fallback. # Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored. @@ -20,22 +28,40 @@ languages: # For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings encoding: "utf-8" -# whether to use the project's gitignore file to ignore files -# Added on 2025-04-07 +# whether to use project's .gitignore files to ignore files ignore_all_files_in_gitignore: true -# list of additional paths to ignore -# same syntax as gitignore, so you can use * and ** -# Was previously called `ignored_dirs`, please update your config if you are using that. -# Added (renamed) on 2025-04-07 -ignored_paths: [] +# list of additional paths to ignore in this project. +# Same syntax as gitignore, so you can use * and **. +# Note: global ignored_paths from serena_config.yml are also applied additively. +ignored_paths: + # --- Deprecated backends (focus on RocksDB/HStore only) --- + - "hugegraph-server/hugegraph-cassandra/**" + - "hugegraph-server/hugegraph-scylladb/**" + - "hugegraph-server/hugegraph-mysql/**" + - "hugegraph-server/hugegraph-postgresql/**" + - "hugegraph-server/hugegraph-palo/**" + - "hugegraph-server/hugegraph-hbase/**" + # --- gRPC generated Java (235k lines, never hand-edited, regenerated by mvn compile) --- + - "hugegraph-pd/hg-pd-grpc/src/main/java/**" + - "hugegraph-store/hg-store-grpc/src/main/java/**" + # --- License/legal files (585 txt files, only touched when adding dependencies) --- + - "install-dist/release-docs/licenses/**" + - "install-dist/scripts/dependency/known-dependencies.txt" + # --- Rarely modified tests/examples --- + - "hugegraph-server/hugegraph-test/**/tinkerpop/**" + - "hugegraph-server/hugegraph-example/**" + - "hugegraph-cluster-test/**" + # --- Note: target/, .flattened-pom.xml, .idea/*, apache-hugegraph-*/ already covered by .gitignore --- # whether the project is in read-only mode # If set to true, all editing tools will be disabled and attempts to use them will result in an error # Added on 2025-04-18 read_only: false -# list of tool names to exclude. We recommend not excluding any tools, see the readme for more details. +# list of tool names to exclude. +# This extends the existing exclusions (e.g. from the global configuration) +# # Below is the complete list of tools for convenience. # To make sure you have the latest list of tools, and to view their descriptions, # execute `uv run scripts/print_tool_overview.py`. @@ -82,7 +108,8 @@ initial_prompt: "" # the name by which the project can be referenced within Serena project_name: "server" -# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default) +# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default). +# This extends the existing inclusions (e.g. from the global configuration). included_optional_tools: [] # list of mode names to that are always to be included in the set of active modes @@ -104,8 +131,10 @@ default_modes: # This cannot be combined with non-empty excluded_tools or included_optional_tools. fixed_tools: [] -# override of the corresponding setting in serena_config.yml, see the documentation there. -# If null or missing, the value from the global config is used. +# time budget (seconds) per tool call for the retrieval of additional symbol information +# such as docstrings or parameter information. +# This overrides the corresponding setting in the global configuration; see the documentation there. +# If null or missing, use the setting from the global configuration. symbol_info_budget: # The language backend to use for this project. @@ -123,3 +152,17 @@ read_only_memory_patterns: [] # Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default) # This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings. line_ending: + +# list of regex patterns for memories to completely ignore. +# Matching memories will not appear in list_memories or activate_project output +# and cannot be accessed via read_memory or write_memory. +# To access ignored memory files, use the read_file tool on the raw file path. +# Extends the list from the global configuration, merging the two lists. +# Example: ["_archive/.*", "_episodes/.*"] +ignored_memory_patterns: [] + +# advanced configuration option allowing to configure language server-specific options. +# Maps the language key to the options. +# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available. +# No documentation on options means no options are available. +ls_specific_settings: {} diff --git a/AGENTS.md b/AGENTS.md index ce6ea35c57..faa33094ec 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,261 +1,110 @@ # AGENTS.md -This file provides guidance to an AI coding tool when working with code in this repository. +Development reference for AI coding agents. See [README.md](README.md) if the task really needs project overview or architecture context. -## Project Overview +## Key Architectural Patterns -Apache HugeGraph is a fast-speed and highly-scalable graph database that supports billions of vertices and edges. It is compliant with Apache TinkerPop 3 and supports both Gremlin and Cypher query languages. +1. **Pluggable Backend Architecture**: Storage backends implement the `BackendStore` interface — new backends require no core changes. Active backends: RocksDB (default/embedded), HStore (distributed) +2. **gRPC Communication**: All distributed components (PD, Store, Server) communicate via gRPC. Proto definitions in `*/grpc/` directories +3. **Multi-Language Queries**: Native Gremlin support(Main) + OpenCypher translation(Backup) -**Technology Stack**: -- Java 11+ (required) -- Apache Maven 3.5+ -- Apache TinkerPop 3.5.1 -- gRPC for distributed communication -- RocksDB as default storage backend +## Build Commands -## Architecture +Requires Java 11 for compilation and testing. -### Multi-Module Structure - -This is a Maven multi-module project with 7 main modules: - -1. **hugegraph-server**: Core graph engine, REST APIs, and backend implementations (13 submodules) -2. **hugegraph-pd**: Placement Driver (meta server) for distributed deployments (8 submodules) -3. **hugegraph-store**: Distributed storage backend with RocksDB and Raft (9 submodules) -4. **hugegraph-commons**: Shared utilities (locks, configs, RPC framework) -5. **hugegraph-struct**: Data structure definitions -6. **install-dist**: Distribution packaging -7. **hugegraph-cluster-test**: Cluster integration tests - -### Three-Tier Architecture - -```bash -Client Layer (Gremlin/Cypher queries, REST APIs) - ↓ -Server Layer (hugegraph-server) - ├─ REST API Layer (hugegraph-api): GraphAPI, SchemaAPI, GremlinAPI, CypherAPI, AuthAPI - ├─ Graph Engine Layer (hugegraph-core): Schema management, traversal optimization, task scheduling - └─ Backend Interface: Abstraction over storage backends - ↓ -Storage Layer (pluggable backends) - ├─ RocksDB (default, embedded) - ├─ HStore (distributed, production) - └─ Legacy: MySQL, PostgreSQL, Cassandra, ScyllaDB, HBase, Palo -``` - -### Distributed Components (Optional) - -For production distributed deployments: -- **hugegraph-pd**: Service discovery, partition management, metadata coordination -- **hugegraph-store**: Distributed storage with Raft consensus (typically 3+ nodes) -- **hugegraph-server**: Multiple server instances (typically 3+) - -All inter-service communication uses gRPC with Protocol Buffers. - -### Key Architectural Patterns - -1. **Pluggable Backend Architecture**: Storage backends implement the `BackendStore` interface, allowing new backends without modifying core code -2. **TinkerPop Compliance**: Full Apache TinkerPop 3 implementation with custom optimization strategies -3. **gRPC Communication**: All distributed components communicate via gRPC (proto definitions in `*/grpc/` directories) -4. **Multi-Language Queries**: Native Gremlin support + OpenCypher implementation in `hugegraph-api/opencypher` - -## Build & Development Commands - -### Prerequisites Check -```bash -# Verify Java version (11+ required) -java -version - -# Verify Maven version (3.5+ required) -mvn -version -``` - -### Full Build ```bash -# Clean build with all modules +# Full build (all modules) mvn clean install -DskipTests -# Build with tests -mvn clean install - -# Build specific module (e.g., server only) +# Single module (e.g., server only) mvn clean install -pl hugegraph-server -am -DskipTests ``` -### Testing +## Testing -#### Server Module Tests -```bash -# Unit tests (memory backend) -mvn test -pl hugegraph-server/hugegraph-test -am -P unit-test +All test commands target `hugegraph-server/hugegraph-test` with `-am` flag: -# Core tests with specific backend -mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,memory -mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,rocksdb -mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,hbase +| Profile | Command | +|---------|---------| +| Unit tests | `mvn test -P unit-test` | +| Core tests | `mvn test -P core-test,rocksdb` | +| API tests | `mvn test -P api-test,rocksdb` | +| TinkerPop structure | `mvn test -P tinkerpop-structure-test,memory` | +| TinkerPop process | `mvn test -P tinkerpop-process-test,memory` | +| Single test class | `mvn test -P core-test,rocksdb -Dtest=YourTestClass` | -# API tests with backend -mvn test -pl hugegraph-server/hugegraph-test -am -P api-test,rocksdb +All commands above implicitly start with `mvn test -pl hugegraph-server/hugegraph-test -am`. -# TinkerPop compliance tests (for release branches) -mvn test -pl hugegraph-server/hugegraph-test -am -P tinkerpop-structure-test,memory -mvn test -pl hugegraph-server/hugegraph-test -am -P tinkerpop-process-test,memory -``` +#### PD & Store Tests -#### PD & Store Module Tests ```bash -# Build and test hugegraph-struct first (dependency) +# Build dependency first mvn install -pl hugegraph-struct -am -DskipTests -# Test PD module mvn test -pl hugegraph-pd/hg-pd-test -am - -# Test Store module mvn test -pl hugegraph-store/hg-store-test -am ``` -### Code Quality & Validation - -```bash -# License header check (Apache RAT) -mvn apache-rat:check - -# Code style check (EditorConfig) -mvn editorconfig:check +## Development Conventions -# Compile with warnings -mvn clean compile -Dmaven.javadoc.skip=true -``` +- Any code change (bug fix or feature) must have sufficient test coverage for the affected logic +- Check existing suites in `hugegraph-server/hugegraph-test` before writing new tests -### Running the Server +## Code Quality -Scripts are located in `hugegraph-server/hugegraph-dist/src/assembly/static/bin/`: +Run before every commit: ```bash -# Initialize storage backend -bin/init-store.sh - -# Start server -bin/start-hugegraph.sh - -# Stop server -bin/stop-hugegraph.sh - -# Gremlin console -bin/gremlin-console.sh - -# Enable authentication -bin/enable-auth.sh +mvn editorconfig:format # Apply code style (root .editorconfig) +mvn clean compile -Dmaven.javadoc.skip=true # Compile with warnings ``` -### Creating Distribution Package +Key rules from `.editorconfig`: 100-char line limit, 4-space indent, LF endings, UTF-8. -```bash -# Build distribution tarball (auto-enabled by default) -mvn clean package -DskipTests +## Running the Server -# Skip assembly creation (if needed) -mvn clean package -DskipTests -Dskip-assembly-hugegraph +Scripts in `hugegraph-server/hugegraph-dist/src/assembly/static/bin/`: -# Output: install-dist/target/hugegraph-.tar.gz +```bash +bin/init-store.sh # Initialize storage backend +bin/start-hugegraph.sh # Start server +bin/stop-hugegraph.sh # Stop server ``` -## Important File Locations - -### Configuration Files -- Server configs: `hugegraph-server/hugegraph-dist/src/assembly/static/conf/` - - `hugegraph.properties` - Main server configuration - - `rest-server.properties` - REST API settings - - `gremlin-server.yaml` - Gremlin server configuration -- PD configs: `hugegraph-pd/hg-pd-dist/src/assembly/static/conf/` -- Store configs: `hugegraph-store/hg-store-dist/src/assembly/static/conf/` - -### Proto Definitions (gRPC) -- PD protos: `hugegraph-pd/hg-pd-grpc/src/main/proto/` -- Store protos: `hugegraph-store/hg-store-grpc/src/main/proto/` +## Configuration Files -### Core Implementation Paths -- Graph engine: `hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/` -- REST APIs: `hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/` -- Backend implementations: `hugegraph-server/hugegraph-{backend}/` (e.g., `hugegraph-rocksdb`) +| Component | Path | Key Files | +|-----------|------|-----------| +| Server | `hugegraph-server/hugegraph-dist/src/assembly/static/conf/` | `hugegraph.properties`, `rest-server.properties`, `gremlin-server.yaml` | +| PD | `hugegraph-pd/hg-pd-dist/src/assembly/static/conf/` | `application.yml` | +| Store | `hugegraph-store/hg-store-dist/src/assembly/static/conf/` | `application.yml` | -## Development Workflow +## Development Workflows -### Code Style -Configure your IDE to use `.editorconfig` for code style and `style/checkstyle.xml` for Checkstyle rules +### Adding Third-Party Dependencies -### Adding Dependencies - -When adding third-party dependencies: -1. Add license files to `install-dist/release-docs/licenses/` -2. Declare dependency in `install-dist/release-docs/LICENSE` -3. Append NOTICE info to `install-dist/release-docs/NOTICE` (if upstream has NOTICE) -4. Update `install-dist/scripts/dependency/known-dependencies.txt` (run `regenerate_known_dependencies.sh`) - -### Backend Development - -When working on storage backends: -- All backends extend `hugegraph-server/hugegraph-core` abstractions -- Implement the `BackendStore` interface -- Each backend is a separate Maven module in `hugegraph-server/` -- Backend selection is configured in `hugegraph.properties` via the `backend` property +Follow ASF compliance: update `install-dist/release-docs/` (LICENSE, NOTICE, licenses/) and `install-dist/scripts/dependency/known-dependencies.txt`. ### gRPC Protocol Changes -When modifying `.proto` files: -- Generated Java code goes to `*/grpc/` packages (excluded from Apache RAT checks) -- Run `mvn clean compile` to regenerate gRPC stubs -- Generated files are in `target/generated-sources/protobuf/` +When modifying `.proto` files: Run `mvn clean compile` to regenerate gRPC stubs -### Authentication System +### Build Order & Cross-Module Dependencies -Authentication is optional and disabled by default: -- Enable via `bin/enable-auth.sh` or configuration -- Auth implementation: `hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/auth/` -- Multi-level: Users, Groups, Projects, Targets, Access control -- Required for production deployments +For distributed development, build in this order: -## Common Workflows - -### Running a Single Test Class ```bash -# Use Maven's -Dtest parameter -mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,memory -Dtest=YourTestClass +mvn install -pl hugegraph-struct -am -DskipTests # 1. Shared data structures +mvn clean package -pl hugegraph-pd -am -DskipTests # 2. Placement Driver +mvn clean package -pl hugegraph-store -am -DskipTests # 3. Distributed storage +mvn clean package -pl hugegraph-server -am -DskipTests # 4. Server ``` -### Working with Distributed Components - -For distributed development: -1. Build struct module first: `mvn install -pl hugegraph-struct -am -DskipTests` -2. Build PD: `mvn clean package -pl hugegraph-pd -am -DskipTests` -3. Build Store: `mvn clean package -pl hugegraph-store -am -DskipTests` -4. Build Server with HStore backend: `mvn clean package -pl hugegraph-server -am -DskipTests` - -See Docker Compose examples: `docker/` directory. Single-node quickstart (pre-built images): `docker/docker-compose.yml`. Single-node dev build (from source): `docker/docker-compose.dev.yml`. 3-node cluster: `docker/docker-compose-3pd-3store-3server.yml`. See `docker/README.md` for full setup guide. - -### Debugging Tips - -- Enable detailed logging in `hugegraph-server/hugegraph-dist/src/assembly/static/conf/log4j2.xml` -- Use `bin/dump-conf.sh` to view effective configuration -- Arthas diagnostics tool is included (version 3.7.1) -- Monitor with `bin/monitor-hugegraph.sh` - -## CI/CD Profiles - -The project uses multiple GitHub Actions workflows: -- `server-ci.yml`: Server module tests (memory, rocksdb, hbase backends) -- `pd-store-ci.yml`: PD and Store module tests -- `commons-ci.yml`: Commons module tests -- `cluster-test-ci.yml`: Distributed cluster integration tests -- `licence-checker.yml`: Apache RAT license validation - -## Special Notes +Key dependencies: `hugegraph-commons` is shared by all modules. `hugegraph-struct` must be built before PD and Store. Server backends depend on `hugegraph-core`. -### Cross-Module Dependencies -- `hugegraph-commons` is a shared dependency for all modules -- `hugegraph-struct` must be built before PD and Store -- Server backends depend on `hugegraph-core` +## Reference Documents -### Version Management -- Version is managed via `${revision}` property (currently `1.7.0`) -- Flatten Maven plugin used for CI-friendly versioning +| Resource | When to consult | +|----------|-----------------| +| `README.md` | Project overview, deployment topology, contribution guide | +| `.serena/memories/` | Project agent memory; key files: `suggested_commands.md` (commands), `task_completion_checklist.md` (pre-commit checks) | diff --git a/README.md b/README.md index 9b92fd2bea..610c29fc91 100644 --- a/README.md +++ b/README.md @@ -7,8 +7,8 @@

[![License](https://img.shields.io/badge/license-Apache%202-0E78BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html) -[![HugeGraph-CI](https://github.com/apache/hugegraph/actions/workflows/ci.yml/badge.svg)](https://github.com/apache/hugegraph/actions/workflows/ci.yml) -[![License checker](https://github.com/apache/hugegraph/actions/workflows/licence-checker.yml/badge.svg)](https://github.com/apache/hugegraph/actions/workflows/licence-checker.yml) +[![HugeGraph-Server CI](https://github.com/apache/hugegraph/actions/workflows/server-ci.yml/badge.svg)](https://github.com/apache/hugegraph/actions/workflows/server-ci.yml) +[![HugeGraph-PD & Store CI](https://github.com/apache/hugegraph/actions/workflows/pd-store-ci.yml/badge.svg)](https://github.com/apache/hugegraph/actions/workflows/pd-store-ci.yml) [![GitHub Releases Downloads](https://img.shields.io/github/downloads/apache/hugegraph/total.svg)](https://github.com/apache/hugegraph/releases) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/apache/hugegraph) @@ -110,8 +110,8 @@ HugeGraph supports both **standalone** and **distributed** deployments: | Module | Description | |--------|-------------| | [hugegraph-server](hugegraph-server) | Core graph engine with REST API, Gremlin/Cypher support, and pluggable backends (RocksDB default) | -| [hugegraph-pd](hugegraph-pd) | Placement Driver for distributed mode - handles meta storage, partition management and cluster scheduling | -| [hugegraph-store](hugegraph-store) | Distributed storage with Raft consensus for high availability and horizontal scaling | +| [hugegraph-pd](hugegraph-pd/README.md) | Placement Driver for distributed mode - handles meta storage, partition management and cluster scheduling | +| [hugegraph-store](hugegraph-store/README.md) | Distributed storage with Raft consensus for high availability and horizontal scaling | | [hugegraph-commons](hugegraph-commons) | Shared utilities, RPC framework and common components |