CSV import: map CSV data onto an existing metamodel#23
Open
amirrza777 wants to merge 33 commits into
Open
Conversation
…n plugin with import CSV syntax
…teCsvNodes instead
The POST /api/projects/{id}/csv-import endpoint materialized a CSV into
a standalone .metamodel/.model file pair using metamodel inference.
That workflow is superseded by the import CSV {} syntax, which reads
CSV data directly at render/execution time against an existing,
manually authored metamodel. Removes CsvImportService, CsvImportRoutes,
the now-unused CsvModelInference engine and its tests, the backend's
:metamodel dependency (no longer needed), and the unused
CSV_IMPORT_FAILED error code.
The import CSV {} syntax was hardcoded directly into the base model
grammar. Model now exposes a generic extension point (imports:
[BaseModelImport], mirroring Config's sections: [BaseConfigSection]),
and CSV is implemented as a real contribution plugin using the same
SerializedGrammar-based merging approach as config/config-mdeo:
- ModelContributionPlugin now carries a serialized grammar and import
metadata instead of raw parser rule objects, so it can cross service
boundaries as plain JSON
- resolveModelPlugins deserializes and wraps each contributed import in
a BaseModelImport-extending rule, and createModelRule builds the root
Model rule from whatever imports are actually resolved
- language-model-csv now builds a real contribution plugin (external
references for Class/ID/STRING/NEWLINE so its serialized grammar
doesn't duplicate definitions already provided by the host)
- New service-model-csv microservice mirrors service-config-mdeo:
registers the CSV contribution with the model language and hosts its
standalone generated language service
- Deleted the dead mutable-registry scaffolding (registerModelContributionPlugin)
that caused the original ESM dual-instance bug, and the orphaned files
left over from the previous abandoned attempt
- modelScopeProvider's class-reference resolution is now plugin-agnostic
(dispatches on the "class" property generically instead of hardcoding
CsvClassImport by type)
- Fixed a pre-existing bug in modelDataHandler where the CSV file path
was passed as a full URI to an API expecting a plain relative path
Verified end to end: grammar parses with no errors, live diagram
rendering from CSV still works, and the model-data execution pipeline
correctly resolves CSV-backed instances.
There was a problem hiding this comment.
Pull request overview
This PR introduces a plugin-contribution mechanism for the Model language to support import CSV { ClassName from "file.csv" } blocks, and wires up new CSV-related language/services so CSV files become first-class in the workbench and can drive live diagram rendering.
Changes:
- Adds a generic “model contribution plugin” framework that lets external plugins contribute
import <KEYWORD> { ... }blocks into the Model language grammar. - Adds CSV language + services (
language-csv,service-csv) and the Model CSV contribution (language-model-csv,service-model-csv) plus docker/nginx/compose wiring. - Updates model file-data computation and diagram rendering to recognize CSV imports and synthesize instances/nodes from CSV-backed data.
Reviewed changes
Copilot reviewed 43 out of 49 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| platform/common/src/main/kotlin/com/mdeo/common/model/ApiResult.kt | Whitespace-only change in error code list area. |
| platform/backend/build.gradle.kts | Whitespace-only change in dependencies block. |
| infra/docker/workbench/nginx.conf | Adds reverse-proxy routes for /plugin/csv/ and /plugin/model-csv/. |
| infra/docker/workbench/Dockerfile | Increases Node build memory for workbench build step. |
| infra/docker/service-model-csv/Dockerfile | New container build for service-model-csv. |
| infra/docker/service-csv/Dockerfile | New container build for service-csv. |
| infra/docker-compose-dev.yaml | Registers CSV plugins in defaults and adds new CSV services/env vars. |
| app/tsconfig.build.json | Includes new CSV-related packages in the build references. |
| app/packages/service-model/tsconfig.json | Adds reference to language-model-csv. |
| app/packages/service-model/src/index.ts | Updates Model service plugin registration (now only .m). |
| app/packages/service-model/src/handler/modelDataHandler.ts | Adds CSV-import-aware ModelData computation path. |
| app/packages/service-model/package.json | Adds dependency on @mdeo/language-model-csv. |
| app/packages/service-model-csv/vite.config.ts | New Vite build for serving the model-csv language module. |
| app/packages/service-model-csv/tsconfig.json | New TS config for service-model-csv. |
| app/packages/service-model-csv/src/served/language.ts | Re-exports model-csv language plugin provider for server import. |
| app/packages/service-model-csv/src/index.ts | New service registering the model-csv contribution for the model language. |
| app/packages/service-model-csv/package.json | New package definition for @mdeo/service-model-csv. |
| app/packages/service-csv/vite.config.ts | New Vite build for serving the CSV language module. |
| app/packages/service-csv/tsconfig.json | New TS config for service-csv. |
| app/packages/service-csv/src/served/language.ts | Re-exports CSV language plugin provider for server import. |
| app/packages/service-csv/src/index.ts | New service registering .csv as a first-class language plugin. |
| app/packages/service-csv/package.json | New package definition for @mdeo/service-csv. |
| app/packages/language-model/src/plugin/resolvePlugins.ts | New resolver that merges contributed import grammars into the Model language. |
| app/packages/language-model/src/plugin/modelContributionPlugin.ts | New contribution plugin contract for model import formats. |
| app/packages/language-model/src/modelPlugin.ts | Updates Model plugin creation to resolve and include contribution plugins. |
| app/packages/language-model/src/index.ts | Exports contribution plugin types/resolution helpers. |
| app/packages/language-model/src/grammar/modelTypes.ts | Adds BaseModelImport + Model.imports to host contributed imports. |
| app/packages/language-model/src/grammar/modelRules.ts | Adds createModelRule() to include contributed import wrapper rules. |
| app/packages/language-model/src/features/modelScopeProvider.ts | Broadens class-reference scoping to work inside contributed import blocks. |
| app/packages/language-model/src/features/diagram-server/modelMetadataManager.ts | Persists metadata for synthetic CSV nodes across re-renders. |
| app/packages/language-model/src/features/diagram-server/modelGModelFactory.ts | Creates synthetic diagram nodes based on CSV import blocks. |
| app/packages/language-model-csv/tsconfig.json | New TS config for language-model-csv. |
| app/packages/language-model-csv/src/plugin/modelCsvContributionPlugin.ts | Defines the CSV model-contribution plugin and its serialized grammar. |
| app/packages/language-model-csv/src/index.ts | Exports CSV import grammar + contribution plugin + import feature helpers. |
| app/packages/language-model-csv/src/grammar/csvImportTypes.ts | Defines AST types for import CSV { ... } content. |
| app/packages/language-model-csv/src/grammar/csvImportRules.ts | Defines grammar rules for CSV import content block. |
| app/packages/language-model-csv/src/features/csvImport.ts | Implements CSV-to-ModelData instance/link mapping. |
| app/packages/language-model-csv/src/csvPlugin.ts | Standalone “model-csv” generated language plugin for parsing CSV import blocks. |
| app/packages/language-model-csv/package.json | New package definition for @mdeo/language-model-csv. |
| app/packages/language-csv/tsconfig.json | New TS config for language-csv. |
| app/packages/language-csv/src/index.ts | Exports CSV language grammar/plugin. |
| app/packages/language-csv/src/grammar/csvTypes.ts | Defines a minimal AST for a CSV file. |
| app/packages/language-csv/src/grammar/csvRules.ts | Defines a permissive CSV grammar (captures full content). |
| app/packages/language-csv/src/csvPlugin.ts | Defines the CSV language plugin provider. |
| app/packages/language-csv/package.json | New package definition for @mdeo/language-csv. |
| app/package-lock.json | Adds workspace links/deps for new packages. |
| .gitignore | Ignores .DS_Store. |
Files not reviewed (1)
- app/package-lock.json: Generated file
Comments suppressed due to low confidence (1)
infra/docker-compose-dev.yaml:118
workbenchproxies/plugin/csv/toservice-csv, butworkbench.depends_ondoes not wait forservice-csvto become healthy. This can cause startup-time 502s / missing plugin manifests while the CSV service is still starting.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+55
to
+59
| location ^~ /plugin/csv/ { | ||
| proxy_pass ${PLUGIN_CSV_SERVICE}/; | ||
| proxy_set_header Host $host; | ||
| proxy_set_header X-Real-IP $remote_addr; | ||
| } |
Comment on lines
+52
to
+56
| id: "model-service", | ||
| name: "Model", | ||
| description: "Language support for model definitions (.m files)", | ||
| icon: convertIcon(BookOpen), | ||
| languagePlugins: [modelLanguagePlugin], |
Comment on lines
+94
to
+100
| const result = importCsvEntries(csvEntries, metamodelClasses); | ||
|
|
||
| const modelData: ModelData = { | ||
| metamodelPath, | ||
| instances: result.instances, | ||
| links: result.links | ||
| }; |
Comment on lines
+497
to
+500
| const uri = resolveRelativePath(doc, entry.file ?? ""); | ||
| const csvContent = await this.modelState.languageServices.shared.workspace.FileSystemProvider.readFile(uri); | ||
| const rows = csvContent.split(/\r?\n/).filter((line: string) => line.trim() !== "").map((line: string) => line.split(",")); | ||
| if (rows.length < 2) continue; |
amirrza777
marked this pull request as ready for review
July 14, 2026 17:01
npm ci in CI failed because these two new packages, added as part of the contribution-plugin refactor, were never registered in the committed lock file.
workbench proxies /plugin/csv/ to service-csv but was missing it from depends_on, unlike every other plugin service. Could cause startup-time 502s or a missing plugin manifest while service-csv was still starting.
Contributor
Author
|
Two fixes pushed since this went up:
Everything else in the PR description still stands. Ready for another look whenever you get a chance. |
Member
|
@amirrza777 what about the other Copilot review comments? |
Member
|
What would help me massively with this (and #26) is a short demo. Perhaps you could put together a quick show-and-tell video and share it either in this PR or via email (link rather than attachment of course)? |
- nginx: /plugin/csv/ location was missing the proxy_http_version,
forwarded headers, and cross-origin headers every other plugin
location has, breaking ESM module loading and cross-origin isolation
for the CSV plugin specifically.
- service-model: register the generated model language (.m_gen) that
was dropped when this service was rewritten for the CSV contribution
plugin refactor, even though the platform still produces .m_gen
files (e.g. optimizer solutions) and language-model still has full
support for it.
- modelDataHandler: a CSV import present in a .m file was replacing
all hand-authored objects/links instead of merging with them, so
mixing manually-authored data with CSV-imported data in the same
file silently lost the manual entries.
- Moved the quote-aware CSV row/field parser out of
language-model-csv (used by the backend import) into language-shared
so language-model's live diagram rendering (modelGModelFactory) can
use the same correct parser instead of a naive split(",")/split("\n")
that miscounts rows when a field contains a quoted comma or newline.
Verified live: mixed manual + CSV objects now render together in the
diagram (confirmed with a hand-authored Company instance alongside
CSV-imported Person rows), .m_gen is now registered in the plugin
manifest, and the /plugin/csv/ headers match the other plugin
locations.
Contributor
Author
|
Addressed all four @szschaler :
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relates to #12
Approach
Metamodels stay manually authored. A
.modelfile can import instance data from a CSV file for an existing class, matching CSV columns to metamodel properties by name:What's included
.csvregistered as a first-class file type in the workbenchimport CSV { ClassName from "file.csv" }syntax in.modelfiles, validated against the metamodelPOST /api/projects/{id}/csv-import) and the metamodel-inference engine behind it, superseded by the aboveKnown gaps, tracked separately
import CSV {}grammar is currently hardcoded into the base model language rather than implemented as a proper contribution plugin (likeconfig/config-mdeo). Working on this refactor next, in this PR.Still a draft while the contribution-plugin refactor is in progress.