This section describes everything we do in our development cycle to reach a high integrity level.
We have a Git repository for all our applications. By taking this approach instead of dividing applications into smaller repositories, we achieve the following:
Centralized source of truth: Everything regarding the application can be found in a single place.
Centralized knowledge: Teams have an all-inclusive knowledge of the application, as they spend their time working in the same repository.
Standardization: Standardizing a project (folder structure, naming conventions, etc.) is easier when there is only one repository where everyone works, as there is no need to duplicate efforts or synchronize repositories.
Everything as code
As mentioned in the "Everything as Code" subsection, we try to keep as much material as possible versioned in our Git repository. Application integrity becomes a matter of keeping a healthy source code. This is after making the source code the only variable that affects an application.
Please read this.
Infrastructure as Code (IaC)
All our infrastructure is versioned in our Git repository written as code. Such code can be deployed anywhere and has all the properties of any other source code, such as auditability, history, revert capabilities, etc.
By having our infrastructure written as code, we can recreate it on a daily basis. Regenerating our infrastructure every day brings the following advantages:
Any injected Trojans or malicious scripts are removed.
Having new servers every 24 hours lets us avoid availability and performance issues generated by memory leaks and unreleased resources.
Having the capability of deploying our infrastructure from zero to production in an automated process.
The infrastructure code can be audited, and changes can only be made by changing such code. This provides full transparency on what was changed, when, and who did it. Also, no administrative protocols like ssh or administrative accounts are needed.
We run an Application Build Process for every change a developer wants to introduce to the application's source code via a merge request. The Application Build Process includes steps like the following:
End to end tests
Commit message tests
Commit deltas tests
Creation of an ephemeral environment
Ephemeral environment tests
By always building and testing everything, we can guarantee that every change is compliant with the application's quality standards.
We recognize that not all the steps of a building process can be automated, especially some tests. That is why developers also need to ask a peer to review their code changes before their merge requests can go to production. Reviewers usually evaluate code quality, commit message coherence, and other semantic properties of the change.
Peer reviewing also becomes an activity where product teams discuss philosophies, standards, and future plans for the application. This is an ideal space for senior developers to guide juniors on the right path.
In addition to running an automated building process for every change, we also run an automatic deployment process. Once a merge request is accepted, an additional Continuous Deployment (CD) pipeline is triggered, automatically deploying a new production version based on the new source code.
Instead of having long-term development environments like staging, we use testing environments that are created during a Continuous Integration (CI) pipeline. We call them ephemeral environments, as they only exist in pipeline time. These environments are created on demand when a developer triggers a CI pipeline. They are also written as code, regenerable and immutable, allowing us to certify that a new version of an application is stable and secure before it reaches production environments. Once a change reaches production, its ephemeral environment is destroyed forever. Ephemeral environments only contain mocked data; they do not share any data with production.
Trunk Based Development
We use Trunk Based Development to keep only one long-term trunk branch. That branch is the source of truth regarding what code is running in the production environments.
Merge requests made by developers cannot be bigger than 200 deltas of code. A delta consists of either a removed or an added line of code. The following are some advantages of working with micro-changes:
Merge requests are small and easy to review by peer reviewers.
Introducing critical bugs to production becomes harder as changes are smaller.
In case something goes wrong with a deployment, identifying the error within those 200 deltas is easier.
Developers go to production multiple times a day, so no code goes stale.
Users of the application see it evolve on a daily basis.
One branch per developer
Developers can only have one short-term branch with their names (employeeatfluid) for every application. Once they develop a portion of code (200 deltas maximum), they run the CI phase, create a merge request, and ask for peer review. If everything goes well, their branch is merged to the trunk branch, their changes are deployed to production, and their short-term branch is deleted.
Isolated and sudo-less dependencies
Some of our dependencies
do not require OS libraries like
they are completely built from scratch,
thus guaranteeing total reproducibility.
these dependencies do not require
any administrative privileges like
They are entirely built on user space,
considerably reducing the possibility
of compromising OS core files.
No dependency auto-update
All external dependencies are pinned to a specific version (this is highly related to the immutability property), meaning that to update a dependency, a developer must do the following:
Change the version in the source code.
Run all CI tests on the generated ephemeral environment with the new dependency version.
Get the change approved by a colleague after running a peer review.
If all tests and the peer review are passed, a new production version with the updated dependency will be automatically deployed.