Optimize bundle compilation workflow

[nicolaswill]

This pull request introduces resource tuning options and refactors the CodeQL compilation cache workflow.

Resource management improvements:

• Added CLI options and internal support for tuning resource usage: --threads for parallelism, --ram for memory limits, and --compression-level for gzip archive compression. These are now configurable via CLI options and repository variables, with sensible defaults and job-specific overrides. [1] [2] [3] [4] [5] [6] [7] [8]

Workflow refactoring for C++ support:

• Split C++ build and test steps into dedicated jobs (build-cpp and consumer-test) in .github/workflows/codeql-compilation-caches.yml, ensuring C++ targets are handled separately from other languages and can be assigned different runners and resources. [1] [2] [3] [4] [5] [6]

Code quality and maintainability:

• Improved exclusion logic for generated files and error handling in bundle creation, including more robust file operations and tarball generation with customizable compression levels. [1] [2] [3] [4] [5]
• Removed redundant or misplaced test logic from the main workflow, moving it to the new consumer-test job for better separation of concerns.

These changes make the build and cache process more efficient/configurable for large or resource-constrained runners.

[Copilot]

Pull request overview

This PR optimizes CodeQL bundle compilation and cache workflows by introducing configurable resource limits (threads/RAM) and faster archive generation via tunable gzip compression, plus workflow adjustments to better isolate C++ cache building/validation.

Changes:

• Add CLI and internal plumbing for --threads, --ram, and --compression-level to tune compilation and bundle archive creation.
• Speed up bundle creation by excluding generated pack artifacts when copying standard packs, making deletions more resilient, and limiting archive worker concurrency.
• Split the GitHub Actions cache workflow so C++ cache building and consumer validation run in dedicated jobs.

Show a summary per file

File	Description
tests/test_codeql.py	Extends tests to assert --threads defaulting and --ram propagation into CodeQL pack commands.
tests/test_bundle_archive.py	Adds tests for gzip compression level handling and platform-specific tool exclusions in bundle archives.
README.md	Documents new resource-tuning flags and new workflow variables for runner/RAM selection.
pyproject.toml	Bumps package version to 0.5.1.
codeql_bundle/helpers/codeql.py	Adds ram configuration and passes --threads/--ram into pack create/pack bundle.
codeql_bundle/helpers/bundle.py	Adds default compression level, uses compresslevel= for gzip archives, improves pack copying/exclusion, and caps archive concurrency.
codeql_bundle/cli.py	Exposes --ram and --compression-level CLI flags and wires them into bundling.
.github/workflows/codeql-compilation-caches.yml	Refactors cache build workflow to separate C++ build and add a consumer validation job; adds optional RAM injection.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Suppressed comments (1)

tests/test_bundle_archive.py:37

• Same as above: the archive being opened is a .tar.gz; open it with an explicit gzip-capable read mode to avoid failures under tarfile's default mode.

            with tarfile.open(output / "codeql-bundle-linux64.tar.gz") as archive:
                names = set(archive.getnames())

• Files reviewed: 8/8 changed files
• Comments generated: 3
• Review effort level: Lite

[nicolaswill]

Updated both archive reads to use explicit r:gz mode.

[nicolaswill]

Updated this to use the module logger and describe the pack create operation as creating the pack.

[nicolaswill]

Updated the documentation to refer explicitly to version 0.5.1.

[Copilot]

Review details

Suppressed comments (7)

codeql_bundle/helpers/codeql.py:251

• logging.warn is deprecated and bypasses the module logger configuration. Use logger.warning for consistency with the rest of this module.

        if disable_precompilation:
            args.append("--no-precompile")
            logging.warn(
                f"NOTE: Precompilation is disabled for {pack.config.name}! This may result in slower query execution."
            )

.github/workflows/codeql-compilation-caches.yml:249

• For the same reason as build-cpp, this job should be skipped when plan.outputs.cpp falls back to {language: "skip"}; otherwise it will fail later when trying to download/use a C++ cache artifact.

  consumer-test:
    needs:
      - plan
      - build-cpp
    if: needs.plan.outputs.skip != 'true'
    runs-on: ${{ vars.CODEQL_CACHE_CONSUMER_RUNNER || 'ubuntu-latest' }}
    env:

.github/workflows/codeql-compilation-caches.yml:268

• This artifact name is hard-coded to cache-cpp, but build-cpp uploads cache-${{ fromJSON(needs.plan.outputs.cpp).language }}. Using the same expression here avoids a mismatch (especially if the cpp output falls back to a non-cpp sentinel).

      - uses: actions/download-artifact@v4
        with:
          name: cache-cpp
          path: dist

codeql_bundle/helpers/codeql.py:206

• logging.warn is deprecated and bypasses the module logger configuration. Use logger.warning here (and fix the mismatched indentation/paren).

This issue also appears on line 247 of the same file.

        if disable_precompilation:
            args.append("--no-precompile")
            logging.warn(
                f"NOTE: Precompilation is disabled for {pack.config.name}! This may result in slower query execution."
             )

codeql_bundle/helpers/bundle.py:341

• Likely accidental missing whitespace around =; this reduces readability and may trip linters.

    @threads.setter
    def threads(self, value: int):
        self.codeql.threads= value

.github/workflows/codeql-compilation-caches.yml:197

• plan.outputs.cpp has a fallback {"language":"skip","target":"skip"}, but this job will still run and try to build --target skip when no C++ target is present. Gate the job on the parsed language so it cleanly skips in that scenario.

This issue also appears in the following locations of the same file:

• line 243
• line 264

  build-cpp:
    needs: plan
    if: needs.plan.outputs.skip != 'true'
    runs-on: ubuntu-latest
    env:

codeql_bundle/helpers/codeql.py:240

• When --threads=0 is provided explicitly, CodeQL interprets it as “use all available cores”, but this log message will read as “Using 0 threads”. Consider special-casing 0 so the log remains accurate.

        if self.threads is not None:
            logger.info(
                f"Using {self.threads} threads for creating {pack.config.name}."
            )
        args.append(f"--threads={threads}")

• Files reviewed: 8/8 changed files
• Comments generated: 0 new
• Review effort level: Lite