Skip to content

Add commitment discount eligibility open data set#2056

Merged
RolandKrummenacher merged 7 commits intodevfrom
feature/commitment-discount-eligibility-opendata
Mar 25, 2026
Merged

Add commitment discount eligibility open data set#2056
RolandKrummenacher merged 7 commits intodevfrom
feature/commitment-discount-eligibility-opendata

Conversation

@RolandKrummenacher
Copy link
Copy Markdown
Collaborator

@RolandKrummenacher RolandKrummenacher commented Mar 15, 2026

Summary

Adds a new open data CSV (CommitmentDiscountEligibility.csv) listing Azure meters eligible for Reserved Instances and/or Savings Plans, sourced from the Azure Retail Prices API.

  • src/scripts/Update-CommitmentDiscountEligibility.ps1 — PowerShell script that pages through the Retail Prices API to collect RI-eligible meters (priceType eq 'Reservation') and SP-eligible meters (Consumption items with savingsPlan arrays), then merges them into a single CSV.
  • .github/workflows/opendata-commitment-eligibility.yml — GitHub Action that runs monthly (1st of each month) or on manual dispatch, executes the script, and creates a PR if data changed.
  • src/open-data/CommitmentDiscountEligibility.csv — Initial dataset with ~84K meters.

Motivation

FinOps Hub Prices_v1_2() / Prices_v1_0() functions currently compute commitment discount eligibility via cross-row lookups at query time, scanning the full prices table on every call. This open data set provides a pre-computed lookup that can replace those per-query scans, and also solves the per-batch ingestion issue described in #1625.

Schema

Column Description
MeterId Azure meter GUID (region-specific)
ReservationEligible Whether the meter has RI pricing
SavingsPlanEligible Whether the meter has SP pricing

Scope

  • Covers Reserved Instances and Savings Plans only
  • Pre-Purchase Plans (e.g., Databricks, Sentinel SCUs) show as Reservation in the API and are included
  • Commitment tiers (e.g., Sentinel GB/day, Log Analytics) are out of scope for now — they use a different pricing model

Next steps

  • Integrate into Build-OpenData.ps1 to generate PowerShell functions and KQL lookup tables
  • Use in Hub ingestion/view functions to replace cross-row eligibility computation

Test plan

  • Full test run completed — script fetched ~142 pages of RI data and ~609 pages of Consumption data
  • Rate limit retry logic validated (exponential backoff with up to 5 retries)
  • Output CSV verified: 84,612 unique meters, no duplicates
  • Review GitHub Action workflow triggers and PR creation logic
  • Validate a sample of meter IDs against the Azure Pricing Calculator

🤖 Generated with Claude Code

Add a new open data CSV listing Azure meters eligible for Reserved
Instances and/or Savings Plans, sourced from the Azure Retail Prices API.

- PowerShell script to fetch RI/SP eligibility from the public API
- GitHub Action to refresh the data monthly and create a PR
- Initial CSV with ~84K meters

This data can replace the per-query cross-row eligibility lookups in
FinOps Hub Prices functions, addressing the performance concern in #1625.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
…ty data

These columns are already present in the cost/prices data that this CSV
will be joined against. Removing them reduces file size and keeps the
dataset focused on what it uniquely provides: the eligibility flags.

Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Change schedule from monthly to weekly (every Monday)
- Add [Open Data] prefix to auto-PR title
- Rename columns to FOCUS-aligned x_CommitmentDiscountSpendEligibility
  and x_CommitmentDiscountUsageEligibility with Eligible/Not Eligible values
- Update PR body text from monthly to weekly

Co-Authored-By: Claude Opus 4.6 <[email protected]>
Copy link
Copy Markdown
Collaborator

@flanakin flanakin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [AI][Claude] PR Review

Summary: Solid addition of a pre-computed commitment discount eligibility dataset that will improve Hub query performance. The script and workflow are well-structured, but there are a few issues to address around workflow formatting, error handling, documentation, and scalability.

🚫 Blockers (1)

  1. Workflow PR body will have leading whitespace due to heredoc indentation

⚠️ Should fix (2)

  1. Retry logic doesn't differentiate error types or respect Retry-After headers
  2. src/open-data/README.md not updated with the new dataset — please add a section for CommitmentDiscountEligibility following the same format as the existing datasets (PricingUnits, Regions, ResourceTypes, Services)

💡 Suggestions (2)

  1. Workflow should handle existing branches/PRs to avoid conflicts on repeated runs
  2. Build-OpenData.ps1 auto-generates PowerShell functions from all CSVs in src/open-data/. With 84K rows, the generated file will be ~4MB. See #2069 for tracking this concern.

@flanakin flanakin self-requested a review March 24, 2026 08:46
@microsoft-github-policy-service microsoft-github-policy-service bot added Needs: Attention 👋 Issue or PR needs to be reviewed by the author or it will be closed due to no activity and removed Needs: Review 👀 PR that is ready to be reviewed labels Mar 24, 2026
- Fix heredoc indentation: move PR body to env var with YAML block scalar
- Include day in branch name for weekly runs (YYYYmmdd)
- Improve retry logic: fail fast on 4xx (except 429), respect Retry-After
  header, show accurate error reason in log messages
- Regenerate CSV with FOCUS-aligned column names

Co-Authored-By: Claude Opus 4.6 <[email protected]>
@microsoft-github-policy-service microsoft-github-policy-service bot added Needs: Review 👀 PR that is ready to be reviewed and removed Needs: Attention 👋 Issue or PR needs to be reviewed by the author or it will be closed due to no activity labels Mar 24, 2026
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Open Data CI task failed: https://github.com/microsoft/finops-toolkit/actions/runs/23549054843

@flanakin flanakin added Status: ▶️ Ready Issue is ready for a dev to start work and removed Needs: Review 👀 PR that is ready to be reviewed labels Mar 25, 2026
@RolandKrummenacher RolandKrummenacher merged commit 7fbec83 into dev Mar 25, 2026
6 checks passed
@RolandKrummenacher RolandKrummenacher deleted the feature/commitment-discount-eligibility-opendata branch March 25, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Status: ▶️ Ready Issue is ready for a dev to start work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants