-
-
Notifications
You must be signed in to change notification settings - Fork 309
Proposal: Standardize User-Agent headers in Importers to prevent 403 blocking #2123
-
Hi everyone! 👋
I am a new contributor looking into the importers module. While analyzing how we fetch data from upstream sources (like GitHub, GitLab, and NVD), I noticed that we don't seem to have a standardized User-Agent strategy.
The Problem
Many importers currently use the default library User-Agents (e.g., python-requests/x.x.x or aiohttp/x.x.x).
From my experience, many modern WAFs (Web Application Firewalls) and APIs block these default agents, resulting in silent failures or 403 Forbidden errors. This makes our data collection less reliable.
The Proposal
I would like to propose standardizing our network identity across the project.
1. Centralize the Configuration
Add a constant in vulnerablecode/settings.py:
# Example VULNERABLECODE_USER_AGENT = "VulnerableCode/1.0 (+[https://github.com/aboutcode-org/vulnerablecode](https://github.com/aboutcode-org/vulnerablecode))"
Beta Was this translation helpful? Give feedback.