Developing for integrity

This section describes everything we do in our development cycle to reach a high integrity level.

Monorepo

We have a Git repository for all our applications. By taking this approach, instead of dividing applications into smaller repositories, we achieve the following:

Centralized source of truth: Everything regarding the application can be found in a single place.
Centralized knowledge: Teams have an all-inclusive knowledge of the application, as they spend their time working in the same repository.
Standardization: Standardizing a project (folder structure, naming conventions, etc.) is easier when there is only one repository where everyone works, as there is no need to duplicate efforts or synchronize repositories.

Everything as code

As mentioned in the “Everything as code” article , we try to keep as much material as possible versioned in our Git repository. Application integrity becomes a matter of keeping a healthy source code. This is after making the source code the only variable that affects an application.

Governance model

Fluid Attacks’ repository operates as a Free and Open Source Software project driven by internal contributions. Our development lifecycle is guided by a “most recent developer is responsible” model, which ensures clear ownership and support boundaries while preventing burnout by shifting accountability as code evolves. Within this framework, technical insights flow upward while business vision flows downward, with the CTO maintaining ultimate responsibility and final decision-making authority for the entire ecosystem.

Infrastructure as code

All our infrastructure is versioned in our Git repository , written as code. Such code can be deployed anywhere and has all the properties of any other source code, such as auditability, history, revert capabilities, etc.

Regenerative infrastructure

By having our infrastructure written as code, we can recreate it on a daily basis. Regenerating our infrastructure every day brings the following advantages:

Any injected Trojans or malicious scripts are removed.
Having new servers every 24 hours lets us avoid availability and performance issues generated by memory leaks and unreleased resources.
Having the capability of deploying our infrastructure from zero to production in an automated process.

Immutable infrastructure

The infrastructure code can be audited, and changes can only be made by changing such code . This provides full transparency on what was changed, when, and who did it . Also, no administrative protocols like SSH or administrative accounts are needed.

Continuous integration

We run an Application Build Process for every change a developer wants to introduce to the application’s source code via a merge request . The Application Build Process includes steps like the following:

Exploit tests
Linting tests
Compilation tests
Unit tests
End-to-end tests
Commit message tests
Commit deltas tests
Creation of an ephemeral environment
Ephemeral environment tests

By always building and testing everything, we can guarantee that every change is compliant with the application’s quality standards.

Peer review

We recognize that not all the steps of a building process can be automated, especially some tests. That is why developers also need to ask a peer to review their code changes before their merge requests can go to production. Reviewers usually evaluate code quality, commit message coherence, and other semantic properties of the change. (At least one review is required, and anyone other than the author of the merge request can approve.)

Peer reviewing also becomes an activity where product teams discuss philosophies, standards, and future plans for the application. This is an ideal space for senior developers to guide juniors on the right path.

Continuous deployment

In addition to running an automated building process for every change, we also run an automatic deployment process. Once a merge request is accepted, an additional Continuous Deployment (CD) pipeline is triggered, automatically deploying a new production version based on the new source code.

Ephemeral environments

Instead of having long-term development environments like staging, we use testing environments that are created during a Continuous Integration (CI) pipeline. We call them ephemeral environments, as they only exist in pipeline time. These environments are created on demand when a developer triggers a CI pipeline. They are also written as code, regenerable and immutable, allowing us to certify that a new version of an application is stable and secure before it reaches production environments. Once a change reaches production, its ephemeral environment is destroyed forever. Ephemeral environments only contain mocked data; they do not share any data with production.

Trunk-based development

We use trunk-based development to keep only one long-term trunk branch. That branch is the source of truth regarding what code is running in the production environments.

Micro-changes

Merge requests made by developers cannot be bigger than 250 deltas of code. A delta consists of either a removed or an added line of code. The following are some advantages of working with micro-changes:

Merge requests are small and easy to review by peer reviewers.
Introducing critical bugs to production becomes harder as changes are smaller.
In case something goes wrong with a deployment, identifying the error within those 250 deltas is easier.
Developers go to production multiple times a day, so no code goes stale.
Users of the application see it evolve on a daily basis.

Short-lived branches

Developers work on short-lived branches that are one or two lowercase words optionally separated by a hyphen (e.g., feature or john-feature). Once they develop a portion of code (250 deltas maximum), they run the CI phase, create a merge request, and ask for peer review. If everything goes well, their branch is merged to the trunk branch, their changes are deployed to production, and their short-term branch is deleted.

Isolated and sudo-less dependencies

Some of our dependencies do not require OS libraries like libc. Instead, they are completely built from scratch, thus guaranteeing total reproducibility.

Additionally, these dependencies do not require any administrative privileges like sudo. They are entirely built on user space, considerably reducing the possibility of compromising OS core files.

No dependency auto-update

All external dependencies are pinned to a specific version (this is highly related to the immutability property), meaning that to update a dependency, a developer must do the following:

Change the version in the source code.
Run all CI tests on the generated ephemeral environment with the new dependency version.
Get the change approved by a colleague after running a peer review.

If all tests and the peer review are passed, a new production version with the updated dependency will be automatically deployed.