REQBench is a large-scale benchmark with 2,095 test cases, each labeled with compatibility information. This large-scale dataset serves as a benchmark for evaluating compatible requirements inference approaches of TPL upgrades in Python projects.

Each .json file in the label directory represents a different scenario used to determine compatibility in TPL upgrades:

The source code of each Python project can be accessed at Zenodo.

./
├── 3d-ken-burns.tar.gz # Project source code
...

To extract the project, run:

cd /home/usr/
mkdir projects
tar -xzvf projectName.tar.gz -C projects

Each test case in REQBench can be regarded as an independent test project. To use PCREQ to infer compatible requirements in REQBench, you need a configuration file for each test case.

All test cases are defined in configure/test.json. Use configure/run.py to batch-generate every config.json at once.

Step 1: Edit the three paths at the top of configure/run.py to match your local environment:

knowledge_path = "/dataset/lei/" # Change to your knowledge directory
proj_path = "/dataset/lei/projects/" # Change to your projects directory
requirements_path = "/dataset/lei/requirements/" # Change to your requirements directory

Step 2: Run the generator from the configure/ directory:

cd configure
python run.py

This creates the directory tree configure/<project>/<library>/<version>/config.json for all 2,095 test cases.

Step 3: Copy the generated config to PCREQ/configure and run PCREQ:

cp configure/PyTorch-ENet/torch/1.2.0/config.json /path/to/PCREQ/configure/
cd /path/to/PCREQ
python main.py --config config.json

Alternatively, create a config.json by hand:

{
 "projPath": "/home/usr/projects/project",
 "requirementsPath": "/home/usr/requirements/project/requirements.txt",
 "targetLibrary": "pillow",
 "startVersion": "6.2.0",
 "targetVersion": "7.0.0",
 "pythonVersion": "3.7",
 "knowledgePath": "/home/usr/knowledge/"
}

Then, copy the configuration file to PCREQ/configure and run the command as follows:

python main.py --config config.json

REQBench is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). See LICENSE.txt for details.