Engineering metrics

Rationale

Engineering metrics help Fluid Attacks gain insights into the development process, both individually and collectively. We use a combination of tools, including GitLab  and QuickSight , to gather and analyze these metrics. The main reasons we chose this way over other alternatives are the following:

• Creating custom metrics and dashboards is fairly simple due to
  • having high standards for our code contributions and overall backlog management: commit messages, deltas, and issue labels;
  • our engineering methodology: micro-changes, trunk-based development, ephemeral environments, monorepo, standard commit messages, among other things;
  • controlling all the above from our CI process.
• It provides custom metrics that we consider more useful over time than well-known standards like DORA or DevEx (we tend to outperform when benchmarked using standard metrics).
• It still allows us to measure what the industry considers standard metrics and benchmark our team.
• It lets us manage access using Okta, AWS roles, and QuickSight to assign dashboards only to those who need them.
• It gives us full customization capabilities as an in-house project.
• It sends periodic reports to dashboard users with customizable frequency and timing.
• It allows filtering individual and collective data using both fixed and custom ranges.
• It was built over services that we already use heavily.

Alternatives

The following alternatives were considered but not chosen for the following reasons:

• Pluralsight Flow: Flow integrates seamlessly with GitLab — our primary development platform — and GitHub, enabling us to track metrics for our open-source projects like Makes. Flow provides valuable insights into efficiency, impact, and complexity based on developers’ code contributions. It also delivers weekly digests summarizing work focus and recognizing key contributions. Additionally, Flow allows filtering of individual and collective data across custom time ranges and offers a REST API for exporting data. Its integration with Okta ensures centralized authorization, enhancing security and access management. We stopped using it because, due to our engineering methodology, creating custom metrics and dashboards is fairly simple.
• Waydev: Waydev provides dashboards for team leaders to track both individual and team performance. It quantifies impact using deltas, file types, and the spread of changes in the codebase, while also evaluating developer efficiency based on recent edits. It measures the risk and complexity of commits and allows for team metric comparisons through separate dashboards. Waydev offers DORA metrics, tracks MR reviews for collaboration insights, and calculates the Knowledge Sharing Index (KSI). It displays MR activity over time and includes benchmarks. Developer Experience (DevEx) tools enable custom surveys, and hygiene metrics ensure good project management practices, with periodic report generation. Its price is comparable to Pluralsight Flow’s. However, we have covered most of its features with our in-house solution, except for commit complexity, the KSI index, and DevEx. We are currently evaluating the value of these features to determine if we should implement them in-house.
• Jellyfish: To take full advantage of the platform, issue tracking must be done in Jira, which leaves us primarily with coding metrics already covered by Pluralsight Flow and GitLab. It has good features related to DevFinOps and engineering focus. However, we currently cover this using GitLab’s data in a custom dashboard on AWS QuickSight. The price is the same as Pluralsight Flow’s.
• LinearB: To take full advantage of the platform, issue tracking must be done in Jira, which leaves us primarily with coding metrics already covered by Pluralsight Flow and GitLab. It offers benchmarking for coding and productivity metrics, but after reviewing relevant publications, we determined that benchmarking is not significant enough to warrant a change in our service, as we have achieved substantial progress in metrics. The price is the same as Pluralsight Flow’s.
• Swarmia: It does not support GitLab integration.
• Allstacks: While it provides good dashboards that utilize heat maps and bar charts to illustrate a team’s active hours, it lacks relevant features compared to Pluralsight Flow and can be 10% to 40% more expensive per contributor.
• Keypup: It only offers basic analytics and has no relevant features compared to Pluralsight Flow.
• DX: It focuses on qualitative and quantitative insights, using both standard and custom metrics, as well as surveys to track developers’ experiences. It can be approximately 20% more expensive per contributor compared to Pluralsight Flow.

Usage

We use our engineering metrics for the following purposes:

• Gaining insights from engineers, both individually and collectively.
• Gaining insights from our continuous integration and continuous deployment processes.
• Reviewing team focus over custom timelines.
• Obtaining commit insights, including size, time of day, and frequency (average per day/month).
• Creating custom dashboards and metrics to serve our team, product, and business.