RFC: Canonical Environment Configuration for Salesforce Projects

[mitchspano]

Problem Statement

Today, managing the lifecycle of a Salesforce project across multiple environments (QA, UAT, Prod) requires a significant amount of bespoke code and configuration. Developers and DevOps engineers often spend hours writing custom bash scripts, configuring environment variables and secrets in their DevOps pipeline runners, and managing .forceignore to ensure that each of the environments acts properly.

Because there is no canonical way to define an environment's identity and behavior in source, the IDE (VS Code) and the CLI often don't "know" which rules apply to the org you are currently authorized against.

The lack of environment-aware tooling can create a real operational risk. As mentioned in Discussion #2189, the unification of commands like sf project deploy start means that a command intended for a Scratch Org can easily be run against Production if a developer forgets they switched their default org. We’ve lost some of the natural guardrails that existed when "push" and "deploy" were separate concepts.

Vision

We are exploring a standardized way to define environment configurations within your Salesforce DX projects. The goal is to have teams define their environment configurations in source and have it drive dynamic environment-specific behavior in both the IDE and CI/CD pipelines.

What could this include? We are considering including the following capabilities in this standard representation:

• Branch-to-Environment Mapping: Defining a specified git branch -> Salesforce environment mapping as a configuration file. Perhaps this will support regex to identify source branches and target environments.
• Injected Environment Variables and Secrets: Standardize references for variables such as userName, instanceUrl, and injected secrets (ex: DEV_KEY_SECRET) to simplify headless auth for CI and CD jobs and anything else that's helpful.
• Environment-Specific Test Levels: ex: During CI, "Integration" always runs RunRelevantTests, while "Production" requires RunLocalTests.
• IDE Behavior: Customizing "Deploy on Save" or "SObject Read/Write" permissions based on the authenticated environment.
• Environment Variables: A more robust way to inject variables into your metadata than the currently supported with the metadata string replacements.
• Augmented .forceignore: Ability to ignore certain metadata types specifically for one environment but not others.
• Environment Safeguards: Inhibit the use of certain commands or interactions like querying data, initiating deployments, etc. for specific environments. Perhaps with a CLI flag option to override in case of emergency.

We want your feedback!

Before we lock in a specification, we want to hear from the community:

• What’s missing? Are there other environment-specific settings you find yourself constantly re-coding in every project?
• What have you built? If you’ve already created a custom solution for this (ex: a config/environments.json file or custom CLI plugins), what worked? What didn't?
• Format Preferences - How would you prefer to manage this? (ex: a single sfdx-environments.json, a /config/environments/ directory with YAML files, TextProto representation, etc.)
• Security - How are you currently handling the hand-off between source-controlled config and sensitive secrets?

[j-schreiber]

Great initiative! One of the most common use cases is not really mentioned, though: Different "unpackaged" source-components per environment, as part of a modularized deployment.

Think about 10 packages and then environment specific overrides (named credentials, custom metadata settings, this stuff).

I believe, a simple env name / src path mapping is enough. Let CI handle the authentication.

Before I write too much, take a look at my open source project "sf-plugin" and the "manifest" implementation: https://github.com/j-schreiber/js-sf-cli-plugin

It works incredibly well and powers my deployment pipelines for more than 2 years now.

[aheber]

I like the idea of environment specific context. Unfortunately I would create a more complicated layer of indirection (if you're familiar with CumulusCI's environments, very similar) where a given project could have a "prod" org but that is mapped locally to an alias or username in the CLI. It is too common for someone to build a required set of "use this alias and magic will happen" but what if I work on two separate projects that use the same environment naming and now I'm creating conflicts or juggling aliases per-project. My machine's aliases shouldn't have to align with a given projects preferences. Things like community projects and open source work would be much harder if you had to align with their context aside from your day-job. (hopefully they can all be creative enough to avoid this but not always)

If you were to do env-specific files such as .forceignore then I would prefer the ability to have a primary file and augment with an additional file. Rather than making the file hold all the context I'd prefer to maintain different files, mapped wherever you're tracking the org's purpose and it's individual contexts. Maybe the option for "merge" vs. "use instead" depending on how different the files might be. When I've built this in the past I just had CI steps that overwrote the files. I used to have merging but it got too messy and had weird failure modes (disk write was broken but the script didn't notice, pushed things that should be ignored). I think merge/append in the code would be more successful as it wouldn't need to write to the file first.

I'd also like to recommend that for the command behaviors, introduce the concept of a "protected" org and the CLI will crash if a destructive command is executed against that org (maybe with some way of registering custom commands on that list). Maybe I work somewhere that I need a login to the QA or UAT sandboxes but we still need them protected from changes from any source except CI/CD, now prod, QA, and UAT can all be protected.

I personally definitely don't want a per-branch-to-env mapping, I'm sure some do but I don't want the CLI to have to be aware of my source tracking preferences and patterns. I can also say that my branch checkout doesn't usually influence which org I want to work with at any given moment.

[volodiag]

When setting LOGIN IP RANGE by profile, one usually wants to configure systems with keeping environments contained. Prod <-> external Prod, UAT <-> external UAT, etc.

However this is only possible manually (Not even BULK import). It means that values are not stored in source, and refreshing the org would require to manually re-write everything post-refresh.

This shortcoming is getting worse because with integration with serverless microservices there is a plethora of potential IPs to enter manually.

See this link
https://salesforce.stackexchange.com/questions/324010/automatically-update-profile-ip-restrictions-that-depend-on-the-environment

Thank you for the consideration

[markgarg]

Thanks for the initiative, love it!

The important thing we do with different environments is the change in named credentials, and sometimes custom metadata (like different cache times) across production and stage/integration/nft environments. We have a small script that copies these from a different directory, like config/stage/namedCredentials or config/nft/customMetadata, to the force-app or custom directory name. It also injects passwords from environment variables before deployment.

We don't store prod named credentials at all, so it won't be deployed when code is pushed to production. Scratch orgs/dev sandboxes need to point to mocks, so that's a different named credential too at config/dev/namedCredentials.

Another small thing we do is change the CORS whitelisting using a similar script (that copies it from another directory before deployment) as we don't want to add a localhost whitelist to production 😄

Format Preferences - If it's possible, then a config/environments directory with yaml/json config in it would be preferable only because a single sfdx-environments.json might get huge.

Security - Currently, we inject them via environment variables in our local script and secrets in our Github actions. We find they are alright.

[dschach]

Reiterating the namedCredential part:
I have a QA named credential and a prod named credential. It would be good to specify which to use in Apex classes based on the environment (or branch name?). I don't know if both go into the org and we update the Apex or if we update the Named Credential record. Open to suggestions.

[john-hsu-bah]

Would environmental configurations allow for additional licenses in scratch orgs in the future? For example, only having one customer community plus license per scratch-org made it difficult to test certain scenarios for multiple external users.

[dschach]

I'd like to see different governance for branches. This isn't specifically your wheelhouse, but running different Code Analyzer rules based on the branch name would be good.

[dschach]

To the format question: It's easier to work with JSON/JS than YAML. Like I prefer Prettier config to be in JSON-style... and this would be better in the same style. YAML is just a pain with arrays, spaces, etc. The line breaks matter, whereas JSON ignores line breaks... and so on.

[cole-bradley]

What’s missing? Are there other environment-specific settings you find yourself constantly re-coding in every project?
I think named credentials is the big one. Custom settings. Any place teams have used as a means to control behaviors, features, connections, etc. for an org. I'd really like to see integration to key management systems, such as Hashicorp Vault or AWS Keys. Systems like those have version control for things like credentials, so then it is just a matter of ensuring your connection and path to the right details is deployed to the org. I think what is also missing is there's no way to easily tell all the orgs present for the devhub or that have been made sandboxes from prod. It would be good if there was a way to query for it (have to be granted the permission to maintain security).

What have you built? If you’ve already created a custom solution for this (ex: a config/environments.json file or custom CLI plugins), what worked? What didn't?
Built a custom deployment script once. That has now moved to a GitHub action that uses the CLI to deploy different project folders to orgs. GitHub has credential management too, so we keep environment details there. I prefer not to keep anything about environments written in the clear or committed to source control. I think environments should be kept fluid to a degree. As in, I should be able to deploy wherever, whenever, as long as I pass any quality gates and have the permissions to.

Format Preferences - How would you prefer to manage this? (ex: a single sfdx-environments.json, a /config/environments/ directory with YAML files, TextProto representation, etc.)
For anything that gets to source control, I'm a fan of JSON or YAML. XML is also fine, it just needs to be structured data with a defined schema. Prettier just needs to be able to format it, like David said earlier. I also mention XML because I think the approach to this should originate from how this is handled in the UI via Setup. Once it is handled in Setup, then the CLI is built to interact. I think this would also help promote the pipeline tooling Salesforce has made for teams still getting past change sets.

I'd imagine a place in Setup where, when granted the permissions, a person can view all of the orgs related to the production org the sandbox or scratch org is part of. From there, orgs can be designated certain properties (e.g., QA, UAT, prod data for full copy sandboxes, etc.). Then people can build pipelines around fetching those properties for org connection details when doing deployments and what not. This would also help other vendors that are building deployment pipelines (e.g., Harness, Gearset, Copado, etc.). What different commands do with orgs that have certain properties could be defined in source control (e.g., run all org tests, deploy this extra thing, etc.).

I think if a solution is designed without the non-dx community in mind then it could cause more annoyances down the road 😄

Security - How are you currently handling the hand-off between source-controlled config and sensitive secrets?
Usually kept in a secret management system that people have access to. Minimizing passing things through direct messages. Looking at building out tooling to access those systems directly to minimize having a human in the middle as well.