This section describes everything we do in our development cycle to reach a high integrity level.
We have a Git repository for all our applications. By taking this approach instead of dividing applications into smaller repositories, we get the following:
Centralized source of truth: Everything regarding the application can be found in a single place.
Centralized knowledge: Teams have an all-inclusive knowledge of the application, as they spend their time working in the same repository.
Standardization: Standardizing a project (folder structure, naming conventions, etc.) is easier when there is only one repo where everyone works, as there is no need to duplicate efforts or synchronize repositories.
As mentioned in the "Everything as code" previous section, we try to keep as much as possible versioned in our Git repository. The application integrity becomes a matter of keeping a healthy source code, after making the source code the only variable affecting an application.
Our entire infrastructure is versioned in our Git repository written as code. Such code can be deployed anywhere and has all the properties of any other source code, such as auditability, history, revert capabilities, etc.
By having our infrastructure written as code, we can recreate it on a daily basis. Regenerating our infrastructure every day brings the following advantages:
Any injected trojans or malicious scripts are removed.
Having fresh new servers every 24 hours lets us avoid availability and performance issues generated by memory leaks and not released resources.
Having the capability of deploying our infrastructure from zero (0) to production in an automated process.
The infrastructure code can be audited, and changes can only be made by changing such code. This provides full transparency on what was changed, when, and who did it. Also, no administrative protocols like ssh or administrative accounts are needed.
We run an Application Build Process for every change a developer wants to introduce to the source code of the application via Merge Request. The Application Build Process includes steps like the following:
End to end tests
Commit message tests
Commit deltas tests
Creation of Ephemeral environment
Ephemeral environment tests
By always building and testing everything, we can guarantee that every change is compliant with the application’s quality standards.
We recognize that not all the steps of a building process can be automatized, especially some tests. That is why developers also need to ask a peer to review their code changes before their Merge Requests can go to production. Reviewers usually evaluate code quality, commit message coherence, and other semantic properties of the change.
Peer reviewing also becomes an activity where product teams discuss philosophies, standards, and future plans for the application. This space is ideal for senior developers to guide juniors on the right path.
In addition to running an automated building process for every change, we also run an automatic deployment process. Once a Merge Request is accepted, an additional Continuous Deployment pipeline is triggered, automatically deploying a new production version based on the new source code.
Instead of having long-term development environments like staging, we use testing environments that are created during a CI pipeline. We call them ephemeral environments, as they only exist in pipeline time. These environments are created on demand when a developer triggers a CI pipeline. They are also written as code, regenerable and immutable, allowing us to certify that a new version of an application is stable and secure before it reaches production environments. Once a change reaches production, its ephemeral environment is destroyed forever.
We use Trunk based development to keep only one long-term master branch. That branch is the source of truth regarding what code is running in the production environments.
Merge requests made by developers cannot be bigger than 200 deltas of code. A delta consists of either a removed or an added line of code. The following are some advantages of working with micro-changes:
Merge requests are small and easy to review by peer reviewers.
Introducing critical bugs to production becomes harder as changes are smaller.
In case something goes wrong with a deployment, identifying the error within those 200 deltas is easier.
Developers go to production multiple times a day, so no code goes stale.
Users of the application see it evolve on a daily basis.
Developers can only have one short-term branch with their names (employeeatfluid) for every application. Once they develop a portion of code (200 deltas maximum), they run the Continuous Integration phase, create a Merge Request, and ask for peer review. If everything goes well, their branch is merged to the master branch, their changes are deployed to production, and their short-term branch is deleted.
Some of our dependencies do not require OS libraries like libc. Instead, they are completely built from scratch, thus guaranteeing total reproducibility.
Additionally, these dependencies do not require any administrative privileges like sudo. They are entirely built on user space, considerably reducing the possibility of compromising OS core files.
All external dependencies are pinned to a specific version (this is highly related to the immutability property), meaning that to update a dependency, a developer must do the following:
Change the version in the source code.
Run all CI tests on the generated ephemeral environment with the new dependency version.
Get the change approved by a colleague after running a peer review.
In case all tests and peer review pass, a new production version with the updated dependency will be automatically deployed.