Identity overrides don't work in offline local evaluation mode

[Nikolay-Andreychuk]

I'm using version 3.8.0 and have identity overrides enabled for my environment.

I'm using a copy of a config for local development and initialize Flagsmith as follows:

local_file_handler = LocalFileHandler(
    environment_document_path=f"./feature_flags_configuration.local.json"
)
flagsmith = Flagsmith(
    environment_key=settings.FLAGSMITH_API_KEY,
    enable_local_evaluation=True,
    offline_handler=local_file_handler,
    offline_mode=True,
)

I have following settings on my config:

  "feature_states": [
    {
      "feature": {
        "id": 103972,
        "name": "ipqs_geo_ip",
        "type": "STANDARD"
      },
      "enabled": true,
      "django_id": 576867,
      "feature_segment": null,
      "featurestate_uuid": "04dcc7d5-b387-4668-8264-d30a2da2e111",
      "feature_state_value": null,
      "multivariate_feature_state_values": []
    }
  ],
  "identity_overrides": [
    {
      "identifier": "146433",
      "environment_api_key": ...,
      "created_date": "2024-08-26T17:45:31.959128Z",
      "identity_features": [
        {
          "feature": {
            "id": 103972,
            "name": "ipqs_geo_ip",
            "type": "STANDARD"
          },
          "enabled": false,
          "django_id": null,
          "feature_segment": null,
          "featurestate_uuid": "964dd1e9-b34a-4bf7-9984-937abbba3f0e",
          "feature_state_value": null,
          "multivariate_feature_state_values": []
        }
      ],
      "identity_traits": [],
      "identity_uuid": "8fd6de03-b834-41d2-b5ec-7076d00e6cd3",
      "django_id": null,
      "composite_key": "cfWNJQwEQkcAiKxSPFL2rv_146433"
    }
  ],

However, identity overrides are ignored unless I disable offline mode.

The root cause must be that it is updated only here when API request is made and are not loaded from offline handler here.

In production, I'm also using the local file as a fallback for the case when Flagsmith is unavailable during app start, so it might become a problem in production in rare cases.

[matthewelwell]

Hi @Nikolay-Andreychuk , thanks for raising this issue. This is an interesting problem and one that we made a conscious decision to exclude from offline mode as we didn't see the use case for it since identity overrides should (in most cases) be fairly temporary. Are you able to elaborate a little bit on your use case here?

[Nikolay-Andreychuk]

Due to performance issues, I have a killswitch that disables some functionality for specific big customers. I have a feature flag enabled by default and disabled for a few big customers by identity. So we will have a problem if it is enabled in case of Flagsmith downtime.

Another, but more critical, case I had before was using the feature flag to remove some fields from external API. In such cases, we return the old field only for a few selected customers until they stop using it. By default, we don't return this to other customers to ensure we can eventually remove it fully. It may take months for all old customers to stop using the field. And if, in case of downtime, this FF flag is switched to the default state, it will lead to errors on the old customers' side because their code expects that this field will always be returned.
Similar cases can be with removing a field from a request or adding a required field to a request.

[matthewelwell]

Thanks for the details here, that's super useful.

I would say that the first use case would be a more appropriate use case for segments vs identity overrides.

With regards to your second use case, this is something that we actually advocate not using feature flags for in the first place. API schema changes are a dangerous beast and we would not recommend using feature flags for this.

That being said, I have discussed further with the team and we do think that it would be beneficial to add identity overrides for our offline mode. I cannot give any time frames around this, however.

In the meantime, I would recommend using segments to mimic the identity overrides where needed.