Introducing a Multi-Org Property Orchestrator for SFDX Projects
This blog post is the second in a series of blog posts about Picnic’s Salesforce Makeover Project.
So you’ve decided to go for a multi-org strategy in Salesforce for your business.
Well, maybe you haven’t actually done that. Maybe you’re just researching your options. Maybe you have no idea what that means. Maybe you don’t even know anything about Salesforce! Don’t worry, neither did I a year ago. In fact, like a third of our Customer Success tech team, I am not even a Salesforce developer — I am a Java developer. Since we adopted Salesforce as our platform of choice for our Customer Success agents last year, I and the other Java developers on the team have been steadily learning more and more about it. As with any time a developer starts working in a new programming language, a new framework, a new platform — there are things you have been taking for granted that are simply not there in the new environments.
One of those things, and the topic of this blog post, is the possibility of having a single code base while still deploying slightly adjusted versions of the same package for each country/environment we operate in. In Java, Python, Node.js and .NET we can already do this, utilizing properties or settings that are automatically applied by the respective framework at compile-time or runtime. This feature (or something like it) unfortunately does not exist yet for SFDX packages, and as the usual Picnic mindset goes — if it doesn’t exist, we will create it! And if it does exist…we will still see if we can do it better ourselves! This blog post will thus detail how we went about developing our own feature which supports the use of properties in sfdx package development while keeping our development and release processes lean and efficient.
Why do we need this?
The multi-org strategy
Picnic currently operates on a multi-org strategy in three countries: The Netherlands, Germany and since a few months ago, France. Due to various reasons, the code and metadata we deploy to each org can’t be exactly the same. Some variations are due to differences in localized text and labels and others are due to slightly adjusted business rules due to different national laws and regulations. In addition, each org needs to connect to a different ecosystem of Picnic services. If you want to know more about how this works for us, make sure to check out the first blog in this series about the Salesforce Makeover Project.
Unlocked packages
At Picnic we work with unlocked packages which contain our code and metadata, which can then be installed in multiple orgs. Since each org’s metadata and code needs to be slightly different, this means that for each release we create three separate package versions of the same package, each containing the code and metadata that is uniform over all countries, plus whatever code and metadata is unique to each country.
Existing options
Before embarking on this journey, we of course did our research into what options were already available from Salesforce to achieve parameterization.
Custom labels/Translation workbench. For localization of some text, custom labels can be used. Another option is to utilize the Translation workbench tool. However, this localization is dependent on the language settings each user has. We needed to enforce that certain labels and text were in Dutch in our NL org, and so on, regardless of what language setting agents have.
Custom settings/metadata. For org-specific business logic, an approach could be to use custom metadata and settings to control which logical paths are taken for each org. However, that would require every org to have all code paths and metadata, as well as complicate the code with all the added conditional clauses. Additionally, some changes can’t be done this way — for example, a use case might be that the same field needs to be slightly differently configured per org. Maybe the length of a text field needs to differ, or whether it’s unique or not, or even have a different type altogether! That can’t really be controlled through any custom setting.
Metadata API and postInstall scripts. One option would be to maintain the different version of our metadata in separate folders, and utilize postInstall scripts to update the metadata appropriately for each org. However, at Picnic we work with unlocked packages, and post-install scripts are not supported for unlocked packages. That means the metadata would need to be unpackaged, along with any other metadata that depends on it, and so on. In addition, we would need to maintain one version per org of all metadata, causing us to store a lot of duplicated code and metadata. Finally, auth providers cannot be updated via the metadata API, which is one of the things that needs to be different per org (since we have different auth URLs per country).
Our first approach
When we started out with Salesforce a year ago, we unfortunately didn’t have the foresight (nor the time or resources) to find a good solution for this challenge. The naive approach we went for was something that should immediately ring some warning bells for experienced git users: treating the Dutch code as our master branch and having a separate branch and an always-open PR for any differences between the Dutch and the German code. Whenever a developer needed to make a change that involved org-specific code or metadata, they would
- Open a PR targeting develop with the NL-specific version
- Once that has been reviewed, approved and merged, rebase the German branch
- Push a commit to that branch with the DE-specific changes
This meant that the initial PR would not contain all the changes actually needed for the ticket, and that the German-specific changes were not properly reviewed. Of course, one option would be to replace step 3. with “Open another PR targeting the German branch” but that would still mean that for every ticket of this type, two PRs would need to be opened, sequentially (since the German branch would always need to be rebased to include the rest of the changes). This would in turn require more reviewing time and effort.
All in all, this process was very error-prone. It was easy to make mistakes or forget a detail when doing the German-specific changes and the lack of review meant that these mistakes would not be caught by others. In addition, we were of course abusing the PR feature of Github; PRs are intended to be eventually merged or closed!
Furthermore, our approach made releasing very tedious. Our Salesforce package releases and installations needed to be done locally, which meant that for each org, the developer tasked with releasing that week had to not only switch between branches but also do some extra work on checking the branch, making sure it was up to date with develop and making sure it didn’t contain any NL-specific code it shouldn’t. This made the release process error-prone, time-consuming and honestly, very stressful. Coupled with the fact that we were, as a company, new to Salesforce packages, meant that the release process in May 2020 took 5 hours from start to finish. That’s more than half a day of valuable developer time spent being stressed doing something that ideally should not take more than 30 minutes.
When we got the news later in 2020 that Picnic would be opening in France in 2021, we knew immediately that something needed to change: we had tentatively accepted always having one unclosable PR, but two? And when Picnic goes even more global in the future, how many more of these PRs would we need? This approach was obviously not scalable, and a solution needed to be found.
Our solution
The solution needed to fulfill a few requirements. It should allow us to specify these org-specific differences for each possible environment that our package would be installed on, while also allowing us to have a single source of truth for our code. It should be easy to use, and allow our developers to work on their features, deploy them and test them in any environment without having to deal with any org-specific differences. Importantly, we would also need to make sure our solution would not interfere with whatever implementation changes they were doing. Finally, we wanted to be able to quickly prototype the solution to evaluate its success with minimal time and effort.
After reviewing the existing options and determining that none of them were perfectly suited for our use case, we turned to what Picnic does best — we decided to make our own! We came up with a bash-based solution which would look for property placeholders in our code and metadata and replace them with the appropriate property value. These replacements would then be reversible, without affecting any other changes or diffs. The last part turned out to be one of the trickiest to pull off.
To start with, we couldn’t really rely on git keeping track of the diffs. Whether we use git diff, git stash or git patch, we would always run into the issue that changes made by the script would be mixed in with changes made by the developer, and there’s not really an easy way to differentiate between the two. Thus, the way we go to keep track of the property replacements would need to be separate from whatever tracking git does. In the end we found our perfect candidates, diff and patch: Unix commands that, respectively, compare two files line by line and create a file containing the diff, and apply the diffs to a file.
Our solution consists of four main scripts, which each accept two parameters: country (in our case, nl/de/fr/global) and environment (prod/uat/scratch)
- Compiler: Works on one file. Looks for any placeholder of the form
{{some.property.name}}, finds the corresponding property value in the right country/environment properties file and creates a temporary copy of the file with all placeholders replaced. Then, it compares this file with the original using diff and generates a file containing the diff. - Orchestrator: Handles running the compiler for all the applicable files in the repository, and saves the overall results in an output file.
- Applier: Uses the diffs created by the compiler and applies them with patch, replacing each placeholder with the right property. Also creates a file detailing which country/environment combo is currently applied.
- Reverser: Uses the diff files to reverse the process, and revert each instance back to the placeholder.
All of these scripts are maintained in a git submodule, allowing all of our Salesforce package repositories to have access to the same features.
The properties are stored in .properties files, one for each country/environment combo, under the following folder structure:
root/config/resources/
├── nl/
│ ├── prod/
│ │ ├── templates/
│ │ └── app.properties
│ └── dev/
├── de
│ ├── prod/
│ └── dev/
├── fr
│ ├── prod/
│ └── dev/
└── global
└── scratch/Each .properties file then looks something like this:
# Base URLs
base-url.picnic-service-1=https://picnic-service-1-prod.nl.picnicdomain.com
base-url.picnic-service-2=https://picnic-service-2-prod.nl.picnicdomain.com# LWC
lwc.some-enum-order-in-list=VALUE1,VALUE2,VALUE3
lwc.some-label=Dit is een nederlands label
Where the base-urls might be used for Named Credentials, while the lwc properties can be used for localization of text or design.
These scripts allow us to easily apply and revert the property replacements whenever needed. Pushing to a scratch org? Apply! Pushing to git? Revert! Creating multiple package versions? Apply, create a version, revert! This enables our developers to easily work with properties without ever losing their work.
Multi-line properties
You might have noticed a folder called templates in the above folder structure. They are used to manage multi-line properties. When dealing with property values that span more than one line, a simple .properties file was not enough, since the compiler works on a line-by-line basis when parsing the properties files. Instead, a placeholder can be added with the format {{FILE:property.name}}. In the properties file, we then give as property value the name of a file. These files are then stored in the templates folder for each country/environment combo. This way we can, for example, have org-specific email templates, field formulas or even whole sections of code!
How our solution fixes our issues
With this new approach, we avoid many of the problems we had before. Now, if a ticket requires an org-specific change (e.g. a translated label), all the relevant changes can be included in a single PR. This ensures those changes are reviewed properly, with the right context, and all approved and merged at the same time. This also makes it easier to revert any change, since we only have one main trunk in git containing all our source code. That means that the develop branch is our true single source of truth for our Salesforce Customer Success implementation.
When performing releases, the release master can be sure that every change included in the new version has been properly reviewed. No switching between branches is needed, all that needs to be done to prepare each org’s version is to run the orchestrator to compile the replacements, and then apply each org’s properties before generating that package version. To further secure the release process, a parallel project worked on a script for releasing packages, which takes as parameters the country, the environment, the username of the release master and the desired version number. This script does all the work for us: runs the orchestrator and applies the properties for the right country/environment, does some validations that the correct org will be chosen, generates a package version with the right source code, promotes it (if the desired environment is production) and then installs it in the right org, leaving the source code clean and ready for the next package version generation.
A big benefit to our solution is that since it revolves around text-based replacement in source files, this approach could be adapted for any method of deploying code from local files to a Salesforce org! There’s nothing that ties our approach specifically to unlocked packages, so companies using managed packages or the metadata API to deploy code to an org can also benefit from a solution like ours.
Future work
Of course, the solution is not perfect. As with any software feature, there are always improvements to be made.
The first idea is to migrate the tool to package in a more efficient and user-friendly language like Python. This would make it easier to extend and update the scripts, and we could make use of the extensive knowledge and help of our Python-expert colleagues at Picnic. Moving the scripts to Python would also help with set-up, as there are quite a few prerequisites, configurations and shell utilities needed to install before a new developer can start using the scripts. For Python, the only prerequisite is installing Python!
A significant challenge is that running the orchestrator script takes quite some time. For most use cases this is fine, since a developer only needs to run it again if they have either cleared their local, generated files, or if any changes were made to properties. However, during CI/CD builds (where generally a new VM is spun up for each build), the orchestrator needs to be run each time. The script already has some concurrency features, but a possible fix would be to enable parameterization of this, so that CI/CD builds can have a different value than developers running the script manually.
Another challenge is the topic of package dependencies. Dependencies in Salesforce are defined in terms of the package version ID, which means that if we’re generating different package versions for each org, any dependent packages also need to depend on different package versions. Currently, we are only using these scripts in child packages (meaning, they don’t have any dependents), but we foresee the need to apply these scripts also for parent packages.
Finally, there are always opportunities for extra automation. One option is to create our own wrapper scripts for sfdx push/pull commands, to help developers avoid issues where they forget to run the apply or revert scripts before syncing their code with a scratch org. The less friction we have in routine development tasks, the more efficient and happy our developers become!
Conclusions
Earlier in this post I mentioned how releasing a new CS package version took 5 hours. That included manual verification of code on different branches, triple/quadruple-reading the commands issued to make sure the right version would be installed in the right environment, generating package versions for our Dutch and German orgs and finally installing them.
With the org-specific properties scripts, in conjunction with the new package-release script, doing the same for our Dutch and German org in addition to our brand new French org, takes only 30 minutes which is a decrease of 90% in time spent by one developer! We gained not only tremendous time savings, but we also minimized the stress experienced by the release master of the week, resulting in happier developers who don’t dread the week they are scheduled to perform releases.
With Salesforce, as with every programming language, platform or library, there are always limitations and challenges we need to tackle or work around. The approach usually taken here at Picnic is to figure out the best, cleanest and most efficient way of working around these limitations, while still adhering to best practices of Salesforce, clean code and Picnic standards in general. If that sounds like something you want to do, then make sure to check out our available positions — our Salesforce team is growing fast!
Salesforce Developer
Salesforce Service Cloud Developer
Javascript Developer (Salesforce)
and many more!
