BigCodeBench

Last updated: Oct 30, 2026

Rationale

Fluid Attacks uses BigCodeBench to evaluate models from providers, for example, Anthropic, for its AI features.

These are the primary reasons why Fluid Attacks selected it and not other alternatives:

It is for code generation and hard programming tasks that use imports and function calls.
Its benchmark uses a dataset that is similar to the prompts that Fluid Attacks uses.
It has a user-centric approach with many different conditions from practical use, and it uses Stack Overflow as a source.
It calculates ratings with two different prompting methods and their average.
It is open source.

HumanEval: It is not for code generation. It is only for Python, and its test dataset is very small. Thus, it is hard to get useful conclusions.
MBPP (Mostly Basic Python Problems): It is only for Python, and its problems are very easy.
RepoBench: It measures code auto completion. This problem is related, but it is not the primary goal of Fluid Attacks.
CRUXEval: It evaluates how correctly models understand code to predict function inputs or outputs. This is not very important for the refactoring work of Fluid Attacks.
DS-1000: It is for data science problems, and not for code generation.

Fluid Attacks uses BigCodeBench to find the best model for the features that suggest vulnerability fixes in its platform and VS Code extension.