Skip to content

Incident repair: migrate 1 feature flag check(s) off raising helpers to resilient default-value variants or graduate #8390

Description

@mar-liu

Context

On 2026-06-08, the feature-flag-data service experienced a 5-minute outage caused by istio-proxy CPU saturation. During this window, any feature flag check using the enabled_or_raise (a.k.a. *_or_raise) variant raised an exception instead of gracefully falling back to a default value.

This caused cascading 5xx errors across Web, REST API, GraphQL API, and Webhooks — impacting external customers for up to 12 minutes.

~430 feature flags across the monolith were identified as using the raising variant without a safe default. This list was defined based on monolith-side metrics — evaluation of these feature flags was unsuccessful during the time of the incident. Your service owns the flag(s) listed below.

ℹ️ If feature flags listed below were already deleted or the calling methods already use default values, you don't need to do anything — just close this issue.

📓 Default values for feature flag checks guide
🚨 Incident review


What you need to do

For each feature flag listed below, do one of the following:

  1. Add a default value — Switch the check from enabled_or_raise / *_or_raise to the default-value variant (enabled? / enabled_with_details?) with an appropriate fallback (typically false). This ensures that when the feature-flag service is unavailable, your code path degrades gracefully instead of raising an exception.

  2. Graduate (remove) the flag — If the flag is fully shipped and no longer needed, remove the flag check entirely and hard-code the enabled behavior. Then mark the flag as graduated in DevPortal.

⚠️ Do not leave the raising variant in place. If you cannot add a default value or graduate the flag, please reach out to the #feature-management channel for guidance.

🔎 Can't find a static callsite? Some flags are checked via dynamically-generated names or indirect helpers, so a grep for the literal flag name may not locate the check. Trace where the flag is evaluated (e.g. a name built at runtime, a shared wrapper, or a metaprogrammed helper) and apply a default there.

Code change examples

Before (raising variant — causes errors during outage):

# This raises Vexi::Errors::CircuitBreakerOpenError when feature-flag-data is down
enabled_or_raise(:my_flag, actor: current_user)

After (default-value variant — graceful degradation):

# Returns false (or your chosen default) when feature-flag-data is unavailable
enabled?(:my_flag, actor: current_user, default: false)

⚠️ Choose your default carefully. For most flags, default: false (feature off) is safest. For flags protecting critical paths that should stay enabled, use default: true.


Why this matters

  • The or_raise variant converts a transient infrastructure issue into a customer-facing error.
  • During the 2026-06-08 incident, ~75.5M feature flag evaluations raised exceptions, causing ~494K web 5xx errors and ~804K REST API 5xx errors in just 5 minutes.
  • Using the default-value variant provides circuit-breaker resilience — your feature gracefully degrades instead of failing catastrophically.

Feature flags to migrate

  • primer_react_select_panel_fullscreen_on_narrow

Acceptance criteria

  • All flags above are either migrated to the default-value variant or graduated/removed
  • No remaining uses of enabled_or_raise / *_or_raise for the listed flags
  • Changes are deployed to production

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions