InfiCoder-Eval: Systematically Evaluating Question-Answering for Code Large Language Models

Large language models for code (code LLMs) have made huge progress. Evaluation benchmarks for code LLMs, such as HumanEval, DS-1000, and MBPP, predominantly focus on code generation. But they are insufficient to evaluate code LLMs’ multifaceted ability. To fill this gap, we propose InfiCoder-Eval, a large-scale free-form question-answering (QA) benchmark for code. InfiCoder-Eval comprises 270 carefully picked high-quality StackOverflow questions, covering 18 programming languages. To tackle the evaluation challenge, InfiCoder-Eval includes an evaluation framework integrating four types of model-free metrics, and domain experts design the concrete criteria for each question. As confirmed with human experiments, InfiCoder-Eval evaluation aligns with humans better than model-based evaluation and runs much faster at the same time. We conduct a systematic evaluation with InfiCoder-Eval for more than 30 code LLMs, leading to several interesting findings. For example, though open-source code LLMs show competitive performance with proprietary models in code generation (e.g., HumanEval), they still have a large gap compared to proprietary ones in InfiCoder-Eval and even the best proprietary LLM (GPT4) is still far from perfect (best open-source model Deepseek-Coder 33B Instruct achieves 50.34% and GPT4 achieves 59.13%). Furthermore, our detailed analysis reveals several weaknesses of current code LLMs. Benchmark, evaluation tools, and detailed results are all publicly available.

InfiCoder-Eval comprises 270 carefully picked high-quality Stack Overflow questions, covering 18 programming languages, and largely following the natural question distribution of Stack Overflow.

We recruited five domain experts to create the benchmark and annotate the correctness evaluation criteria. Specifically, the InfiCoder-Eval framework integrates four types of model-free metrics for evaluating the correctness: keywords matching, blank filling, unit testing, and dialogue similarity.

Below is the question type, metric type, and length statistics.

Existing benchmarks weigh heavily on code generation, unit-test-based evaluation, and a limited set of programming languages. InfiCoder-Eval processes a much higher diversity to reflect real-world code LLMs’ usage scenarios and is far from saturation.

Each question contains a system prompt and content prompt. For questions whose responses are mainly in natural language, the system prompt is For other questions, the system prompt is We then format the system prompt and content prompt following each model's default instruction template. If no instruction template specified, we use the prompt format

We adopt best@10 as the main evaluation metric, where 10 responses are sampled and evaluated for each question and the best score per question is recorded and summed up. Throughout the evaluation, we set sampling temperature T to be 0.2 and top p cut-off threshold to be 0.9. We leave the exploration of other hyperparameters as the future work.

For score computation, we treat each question equally with one point each. Since the question frequency largely follows the Stack Overflow distribution, this score can be explained as how well the model responds to Stack Overflow questions. Given 270 questions in the benchmark, the full score is 270, and we by default report the percentage score (achieved score divided by the full score which is 270). The one point for each question can be further decomposed into a few scoring points within each question. For example, a question may contain four keywords with weights 2, 1, 1, and 1 each. Then, matching each keyword can contribute to 0.4, 0.2, 0.2, and 0.2 points respectively to the final score.

Step 0: Setup

1. Convert or save your model weights in Hugging Face Transformers format.
2. Clone the two repositories: Inference Repo and Evaluation Repo.

No requirement on the local directory paths.

3. Set global environment variable:
export INFERENCE_REPO_PATH=[evaluation repo]/batched_prompts/suite_v2.0.0_dev.csv

Step 1: Generate Response for Your Model

1. Set the working directory to Inference Repo.

The inference repo is forked and slightly modified from bigcode-evaluation-harness framework. We leverage its function for inference.

2. Determine the prompt format to use, which corresponds to task name.

We support these format for now: code-ffqa-v2 (the default one, system + '\n' + content), code-ffqa-v2-endn (system + '\n' + content + '\n'), code-ffqa-v2-deepseek-chat (deepseek-coder-instruct format), code-ffqa-v2-baichuan2 (baichuan2 models format), code-ffqa-v2-zypher (zypher-7b-beta format), code-ffqa-v2-octo (octopack model format), code-ffqa-v2-wizard (wizard-python model format), code-ffqa-v2-phi (phi-1.5 model format), and code-ffqa-v2-inficoder (our InfiCoder model format).

Feel free to contribute by adding your model format, which is easy - just slightly modify bigcode_eval/tasks/code_ffqa_v200.py a bit.

3. Run batch inference to generate responses for question prompts.
accelerate launch [inference repo dir]/main.py --model [your model path / hugging face hub path] --tasks [determined task name above] --batch_size [batch_size] --precision bf16 --n_samples 30 --do_sample True --temperature 0.2 --top_p 0.9 --save_generations --save_references --trust_remote_code --generation_only --max_new_tokens 1024 --save_generations_path [output raw response file path].json --eos='[EOS string]'

This command will output two files in your working directory: [output raw response file path].json which stores responses and references.json which stores case names as the index.

4. Export responses and case names to evaluation-capatible csv file.
python3 [inference repo dir]/ffqa_processor.py [output raw response file path].json references.json [response csv file].csv

This command will join the two output files above into one csv file [response csv file].csv which can be processed by the evaluation framework below.

Step 2: Evaluation (Dev Set)

1. Setup the evaluation framework: Evaluation Repo.

At this point, we only support Linux environment.

Run pip3 install -r requirements.txt, then ./setup.sh (time costly, usually 1-2 hours) which installs necessary compilers and packages for multi-lingual execution environment.

2. Check the evaluation environment.

Run python3 env_check.py to check and fix the environment incompatibility according to the console output. If the console output is "You're good to go.", then we can proceed.

3. Unpack the csv output.

Unpack the csv output file from the previous inference step into a directory where each response is stored in a separate txt.

python3 adaptors/csv_response_unpacker.py [response csv file].csv [response save dir]

We recommend to save the responses in a directory in responses/, i.e., let [response save dir]=responses/.... The above script will create the [response save dir] directory if it does not exist.

4. Run main evaluation.
python3 grader_main.py suite_v2.0.0_dev.yaml [response save dir]

The evaluation takes around 15 min - 45 min.

When it finishes, there are two output files: results/suite_v2.0.0_dev_[response save dir base name].txt (short summary) and results/suite_v2.0.0_dev_[response save dir base name].yaml (all details).

You can also customized the output paths by --result_summary_path and --result_detail_path arguments respectively.

5. Get statistics and print the results.
python3 print_result_stat.py [result detail path] [summary txt path]

In console output and [summary txt path], a nice table will be printed, including the overall score and percentage and the sub-scores for each question type, metric type, and programming language.

Step 3: Evaluate (Test Set)

Available upon request (email us).

You can also give us feedback in the discussion issue posts of our repositories:

• 
• 
•