Score 2774 traceability by FScholPer · Pull Request #484 · eclipse-score/docs-as-code

FScholPer · 2026-04-13T13:44:39Z

Frank Scholter Peres frank.scholter_peres@mercedes-benz.com, Mercedes-Benz Tech Innovation GmbH
Provider Information

github-actions · 2026-04-13T13:45:14Z

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run --lockfile_mode=error //src:license-check

Status: ⚠️ Needs Review

Click to expand output

[License Check Output]
Extracting Bazel installation...
Starting local Bazel server (8.4.2) and connecting to it...
INFO: Invocation ID: 542302cb-aa12-4d8e-9587-9e3c8ba57f4f
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
WARNING: Target pattern parsing failed.
ERROR: Skipping '//src:license-check': no such target '//src:license-check': target 'license-check' not declared in package 'src' defined by /home/runner/work/docs-as-code/docs-as-code/src/BUILD
ERROR: no such target '//src:license-check': target 'license-check' not declared in package 'src' defined by /home/runner/work/docs-as-code/docs-as-code/src/BUILD
INFO: Elapsed time: 5.062s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target

github-actions · 2026-04-13T13:49:05Z

The created documentation from the pull request is available at: docu-html

AlexanderLanin

see discussion in #485

FScholPer · 2026-04-16T08:42:49Z

closed #485 @AlexanderLanin can you please check again?

MaximilianSoerenPollak

Some comments for things I noticed.

Will take another look after lunch.

Overall looks like genuine good work that we can build upon.

MaximilianSoerenPollak · 2026-04-16T10:31:39Z

+            if isinstance(item, str) and item.strip():
+                out.append(item.strip())
+        return out
+    return []


Since you are already inside sphinx (due to it being inside docs-as-code extensions) you could also just get the all sphinx-needs from sphinx itself.

You can do so via:

Needs_Data = SphinxNeedsData(app.env) needs_data = set(x["id"] in Needs_Data.get_needs_view().values())

If you only wanted to have the ID's in the end.

Though you then need to have this as a full extension. If you want I can convert this in a future PR to a full extension to enable this & other things.

MaximilianSoerenPollak · 2026-04-16T10:33:13Z

+            implemented = str(need.get("implemented", "")).upper().strip()
+            if implemented not in {"YES", "PARTIAL"}:


Implemented is tool_req specific. Unsure if this should be hidden / hardcoded in the code?

MaximilianSoerenPollak · 2026-04-16T10:34:01Z

+def filter_requirements(
+    all_needs: Sequence[Any],
+    requirement_types: set[str],
+    include_not_implemented: bool,
+) -> list[Any]:
+    """Extract requirements by type and implementation state."""
+    requirements: list[dict[str, Any]] = []
+    for need in all_needs:
+        need_type = str(need.get("type", "")).strip()
+        if need_type not in requirement_types:
+            continue
+        if not include_not_implemented:
+            implemented = str(need.get("implemented", "")).upper().strip()
+            if implemented not in {"YES", "PARTIAL"}:
+                continue
+        requirements.append(need)
+    return requirements


If this was a sphinx-extension could also here make use of sphinx-needs internal functions etc. to get the needs list / objects for filtering etc.
As mentioned above I can take this PR (after merge) and convert it in a future one.

MaximilianSoerenPollak · 2026-04-16T10:38:38Z

+def calculate_requirement_metrics(
+    requirements: Sequence[Any],
+) -> dict[str, Any]:
+    """Calculate requirement traceability statistics for links and completeness."""
+    total = len(requirements)
+    with_code = sum(
+        1 for need in requirements if is_non_empty(need.get("source_code_link"))
+    )
+    with_test = sum(1 for need in requirements if is_non_empty(need.get("testlink")))
+    fully_linked = sum(
+        1
+        for need in requirements
+        if is_non_empty(need.get("source_code_link"))
+        and is_non_empty(need.get("testlink"))
+    )
+
+    missing_code_ids = [
+        str(need.get("id", ""))
+        for need in requirements
+        if not is_non_empty(need.get("source_code_link")) and need.get("id")
+    ]
+    missing_test_ids = [
+        str(need.get("id", ""))
+        for need in requirements
+        if not is_non_empty(need.get("testlink")) and need.get("id")
+    ]
+    not_fully_linked_ids = [
+        str(need.get("id", ""))
+        for need in requirements
+        if (
+            (
+                not is_non_empty(need.get("source_code_link"))
+                or not is_non_empty(need.get("testlink"))
+            )
+            and need.get("id")
+        )
+    ]


Can be done in future PR:

I would like to have this at least a tiny bit more efficient.
Loops are one of the least efficient things in python, even though this already greatly improves due to use of set comprehension I think it should be possible with just one itteration through all requirements instead of 6.

But as said, future PR improvement, for me currently not a block as we don't have 10k+ reqs to look through.

MaximilianSoerenPollak · 2026-04-16T10:48:46Z

+  --needs-json bazel-bin/needs_json/_build/needs/needs.json \
+       --test-types unit-test
+
+Use lower thresholds during rollout and tighten towards 100% over time.


We can also achieve this '100%' thing with two further changes, at least in some repos.

Make both source_code_link & testlink mandatory option for each requirement (par some exceptions).

Make the pytest plugin or sphinx extension throw an error if a test does not have properties => therefore isn''t linked

MaximilianSoerenPollak · 2026-04-16T10:49:48Z


 .. needpie:: Requirements Status
-   :labels: not implemented, implemented but not tested, implemented and tested
+   :labels: not implemented, implemented but incomplete docs, fully documented


If I read this correctly, this seams like it does something different than before?
statistics wise

MaximilianSoerenPollak · 2026-04-16T10:51:46Z

+def _write_needs_json(tmp_path: Path) -> Path:
+    needs_json = tmp_path / "needs.json"
+    payload = {
+        "current_version": "main",


I don't think current_version can be a non ID (in a realistic scenario).
Though don't think it matters.

MaximilianSoerenPollak · 2026-04-16T10:53:06Z

+                        "partially_verifies": "",
+                        "fully_verifies": "",


This would not be allowed.

If a test does not have the right prperties (either fully or partially verifies & other things) the testcase need will not be generated.

MaximilianSoerenPollak · 2026-04-16T11:01:24Z

+To enforce traceability in CI:
+
+1. Run tests.
+2. Generate ``needs.json``.


This is a bit flawed, as currently testcase needs are generated as external needs and those are excluded from the needs.json.

They can be included if we change the needs.json build filter which is possible so also external needs will be in there.
It's like this in conf.py or probably in the score_metamodel extension:

needs_builder_filter = '(is_external==True and need_type==testcase)''

The exact syntax we would have to figure out but something like this.

a-zw · 2026-04-16T10:56:21Z

 | `bazel run //:docs`                            | Builds documentation                                                                              |
 | `bazel run //:docs_check`                      | Verifies documentation correctness                                                                |
 | `bazel run //:docs_combo`                      | Builds combined documentation with all external dependencies included                             |
+| `bazel run //scripts_bazel:traceability_coverage -- --needs-json bazel-bin/needs_json/needs.json --min-req-code 100 --min-req-test 100 --min-req-fully-linked 100 --min-tests-linked 100 --fail-on-broken-test-refs` | Calculates requirement/test traceability percentages and fails if thresholds are not met |


These commands are intended to work in any score repo. So it should be:

Suggested change

| `bazel run //scripts_bazel:traceability_coverage -- --needs-json bazel-bin/needs_json/needs.json --min-req-code 100 --min-req-test 100 --min-req-fully-linked 100 --min-tests-linked 100 --fail-on-broken-test-refs` | Calculates requirement/test traceability percentages and fails if thresholds are not met |

| `bazel run @score_docs_as_code//scripts_bazel:traceability_coverage -- --needs-json bazel-bin/needs_json/needs.json --min-req-code 100 --min-req-test 100 --min-req-fully-linked 100 --min-tests-linked 100 --fail-on-broken-test-refs` | Calculates requirement/test traceability percentages and fails if thresholds are not met |

a-zw · 2026-04-16T10:57:34Z

+.. code-block:: bash
+
+    bazel test //...
+    bazel build //:needs_json


Makes no sense to me to run this build manually. The coverage check can simply depend on it and Bazel takes care of the execution.

a-zw · 2026-04-16T10:58:24Z

+
+.. code-block:: bash
+
+    bazel test //...


We need a way to "build" a test report and then a coverage check can depend on that.

a-zw · 2026-04-16T10:59:02Z

+    bazel test //...
+    bazel build //:needs_json
+    bazel run //scripts_bazel:traceability_coverage -- \
+       --needs-json bazel-bin/needs_json/_build/needs/needs.json \


Manually using files in bazel-bin is a red flag. Use Bazel dependencies instead.

a-zw · 2026-04-16T11:01:45Z

+# SPDX-License-Identifier: Apache-2.0
+# *******************************************************************************
+
+"""Compute requirement and test traceability coverage from sphinx-needs output."""


I believe this script, a CI quality gate, (and related files) should actually be in tooling and not in docs-as-code.

a-zw · 2026-04-16T11:02:01Z


 # bug: This file is created in repo root on test discovery.
 /consumer_test.log
+.clwb


What are clwb files?

MaximilianSoerenPollak · 2026-04-16T11:13:28Z

Also a general thing.

If there is any 'substantial' or partial generation from AI please add the disclaimer in the header.
According to 'eclipse' something to the lines of this should be fine:

# ╓                                                          ╖
# ║ Some portions generated by <AI Model Name>               ║
# ╙                                                          ╜

Model name meaning like 'Gpt' or 'Copilot' or 'Claude' etc.

Co-authored-by: Maximilian Sören Pollak <maximilian.pollak@qorix.com> Signed-off-by: Frank Scholter Peres(MBTI) <145544737+FScholPer@users.noreply.github.com>

FScholPer · 2026-04-16T13:44:09Z

Also a general thing.

If there is any 'substantial' or partial generation from AI please add the disclaimer in the header. According to 'eclipse' something to the lines of this should be fine:
# ╓                                                          ╖
# ║ Some portions generated by <AI Model Name>               ║
# ╙                                                          ╜
Model name meaning like 'Gpt' or 'Copilot' or 'Claude' etc.

I will add. But we should think about using tools like https://github.com/git-ai-project/git-ai in general

MaximilianSoerenPollak · 2026-04-16T14:18:17Z

I will add. But we should think about using tools like https://github.com/git-ai-project/git-ai in general

Probably might be worth a look to see if we could use that. I can bring it up in infra.

MaximilianSoerenPollak · 2026-04-16T14:25:44Z

@FScholPer I would propose that I can take these Ideas & Proposals that you have made in this PR and re-implement them into Docs-As-Code so they fit a better architecturally and makes sense with how testcases etc. actually behave & are created inside the DaC frameworks that deal with it.

I fear it would be a lot more work to merge this PR and then rebuild it / change it quiet heavily later when we want to integrate it.

Do you think this would make sense and is a good way to move forward with this issue?

@AlexanderLanin @a-zw please also advice here in your opinion what would be a good approach.

FScholPer added 2 commits April 13, 2026 13:29

feat(traceability): add coverage checker and reporting docs

fd1f3dd

add coverage check

59b1ced

FScholPer requested review from AlexanderLanin, MaximilianSoerenPollak, dcalavrezo-qorix and nradakovic as code owners April 13, 2026 13:44

FScholPer linked an issue Apr 13, 2026 that may be closed by this pull request

Improvement: Introduce traceability coverage checker and CI reporting mode eclipse-score/score#2774

Open

github-project-automation bot added this to INF - Infrastructure Community Apr 13, 2026

github-project-automation bot moved this to Backlog in INF - Infrastructure Community Apr 13, 2026

fix lint

9250faa

a-zw suggested changes Apr 13, 2026

View reviewed changes

Comment thread scripts_bazel/traceability_coverage.py

Comment thread docs/how-to/test_to_doc_links.rst

Comment thread scripts_bazel/traceability_coverage.py

Comment thread scripts_bazel/traceability_coverage.py

AlexanderLanin requested changes Apr 14, 2026

View reviewed changes

FScholPer and others added 3 commits April 14, 2026 12:54

Merge branch 'main' into score-2774-traceability

1677f66

refactoring the coverage, metrics and dashboard

499d02f

add generic filters

e1dbfe7

Merge branch 'main' into score-2774-traceability

ee7b148

MaximilianSoerenPollak reviewed Apr 16, 2026

View reviewed changes

a-zw suggested changes Apr 16, 2026

View reviewed changes

Update src/extensions/score_metamodel/traceability_metrics.py

b98669f

Co-authored-by: Maximilian Sören Pollak <maximilian.pollak@qorix.com> Signed-off-by: Frank Scholter Peres(MBTI) <145544737+FScholPer@users.noreply.github.com>

		implemented = str(need.get("implemented", "")).upper().strip()
		if implemented not in {"YES", "PARTIAL"}:

	\| `bazel run //scripts_bazel:traceability_coverage -- --needs-json bazel-bin/needs_json/needs.json --min-req-code 100 --min-req-test 100 --min-req-fully-linked 100 --min-tests-linked 100 --fail-on-broken-test-refs` \| Calculates requirement/test traceability percentages and fails if thresholds are not met \|
	\| `bazel run @score_docs_as_code//scripts_bazel:traceability_coverage -- --needs-json bazel-bin/needs_json/needs.json --min-req-code 100 --min-req-test 100 --min-req-fully-linked 100 --min-tests-linked 100 --fail-on-broken-test-refs` \| Calculates requirement/test traceability percentages and fails if thresholds are not met \|

Conversation

FScholPer commented Apr 13, 2026

Uh oh!

github-actions bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

License Check Results

Uh oh!

github-actions bot commented Apr 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexanderLanin left a comment

Choose a reason for hiding this comment

Uh oh!

FScholPer commented Apr 16, 2026

Uh oh!

MaximilianSoerenPollak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

a-zw Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaximilianSoerenPollak commented Apr 16, 2026

Uh oh!

FScholPer commented Apr 16, 2026

Uh oh!

MaximilianSoerenPollak commented Apr 16, 2026

Uh oh!

MaximilianSoerenPollak commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Apr 13, 2026 •

edited

Loading

a-zw Apr 16, 2026 •

edited

Loading