refactor: use Dependency Groups (PEP 735) instead of misusing package extras for dependencies that are not direct package dependencies

[jenstroeger]

For background, see PEP 735 and Dependency Groups. The purpose here is to simplify the package’s API wrt. the package’s deps: previously the dev packages were part of the package’s API via extras, but now we can separate cleanly the package’s API and extra dependencies from the dev packages.

I hope that Dependabot understands these dependency groups 🤓

@behnazh I think we can now simplify the CI build process here

python-package-template/.github/workflows/_build.yaml

Lines 80 to 111 in 17c1d75

	- name: Install dependencies
	run: make setup
	# Audit all currently installed packages for security vulnerabilities. This step can
	# be disabled by setting the repository variable DISABLE_PIP_AUDIT to 'true'.
	- name: Audit installed packages
	if: ${{ !inputs.disable-pip-audit }}
	run: make audit
	# Build the sdist and wheel distribution of the package and docs as a zip file.
	# We don't need to check and test the package separately because `make dist` runs
	# those targets first and only builds the package if they succeed. The Hypothesis
	# profile picks the Hypothesis settings (see tests/conftest.py) which, for Github,
	# disable randomized testing to avoid breaking CI.
	- name: Build the package
	run: make dist
	env:
	HYPOTHESIS_PROFILE: github
	# Generate the requirements.txt that contains the hash digests of the dependencies and
	# generate the SBOM using CyclonDX SBOM generator.
	- name: Generate requirements.txt and SBOM
	if: matrix.os == env.ARTIFACT_OS && matrix.python == env.ARTIFACT_PYTHON
	run: make requirements sbom
	# Remove the old requirements.txt file (which includes _all_ packages) and generate a
	# new one for the package and its actual and required dependencies only.
	- name: Prune packages and generate required requirements.txt
	if: matrix.os == env.ARTIFACT_OS && matrix.python == env.ARTIFACT_PYTHON
	run: |
	rm requirements.txt
	make prune requirements

like so

make upgrade-pkg  # Install only package direct deps (and all extras).
make requirements  # Generate the requirements.txt file.
make sbom  # Generate the SBOM from the environment, instead of the requirements.
make upgrade-dev  # Install dev packages via dependency group.
make audit
make dist

Note that we don’t make prune here which should save some time. I also think it’s safe to install package deps and then dev deps in two steps, because if the dev deps introduce a package that’s already installed by the package itself and

• the required version is in range then pip will keep the existing package (we’re not using --ignore-installed); or
• the required version is out of range then pip will simply fail.

What do you think?