lemon42-ai/minified-diverseful-multilabels

A minified, clean and annotated version of DiverseVul

Dataset Summary

This is a minified, clean and deduplicated version of the DiverseVul dataset. <br> We publish this version to help practionners in their code vulnerability detection research. <br>

Data Structure & Overview

• —Number of samples: 23847
• —Features: func (the C/C++ code)cwe (the CWE weakness, see table below)
• —Supported Programming Languages: C/C++
• —Supported CWE Weaknesses: | Label | Description | |---------|-------------------------------------------------------| | CWE-119 | Improper Restriction of Operations within the Bounds of a Memory Buffer | | CWE-125 | Out-of-bounds Read | | CWE-20 | Improper Input Validation | | CWE-416 | Use After Free | | CWE-703 | Improper Check or Handling of Exceptional Conditions | | CWE-787 | Out-of-bounds Write | | safe | Safe code |

Cleaning & Preprocessing

The original dataset contains 18,945 vulnerable functions spanning 150 CWEs and 330,492 non-vulnerable functions extracted from 7,514 commits, according to the original paper. <br>

We constructed the minified version through the following steps: We cleaned the original dataset by removing null values and removing deduplicates (keeping the first occurance only). We then randomly sampled 19271 'safe' functions that have between 50 and 595 tokens (tokenized using ModernBERT-base's tokenizer, special tokens excluded). The majority of function have between 200 and 595, a small fraction that have between 50 and 200 tokens were randomly added for the sake of diversification. The safe label's token distribution is as follows:

To construct the unsafe labels, we only kept the six most frequent CWE weaknesses (see table above). <br> We also noticed that the unsafe functions are very long (token wise): The average token length was around 1200 and the median token length was arround 512. <br> We built this dataset to use it for fine-tuning ModernBERT-base. For compute reasons, we decided to only keep functions that have less than 595 tokens, just like the safe label. In case you want different token lengths, we provide the github repository to re construct the dataset. The unsafe labels have the following token distribution:

Application

We publish this minified dataset to help researchers with low compute resources in their code vulnerability research projects. <br> This dataset was also used to fine tune ThreatDetect-C-Cpp.

More Details & Acknowledgements

You can find all the code on our Github Repository. We deeply thank the mainteners of the DiverseVul dataset.