🚀 HeadlessX v1.3.0

License: MIT Version Node.js Playwright

GitHub Stars GitHub Forks Docker CI/CD

Discussions Open Source Contributors Issues PRs Welcome

HeadlessX Demo

The future of intelligent web scraping is here

Roadmap

🎯 Revolutionary Features Coming:

• 🤖 AI-Powered Admin Panel - Intelligent task management & automation
• 🎨 Modern React Frontend - Sleek, responsive dashboard interface
• 🧠 Smart Automation - AI-driven scraping strategies & optimization
• 📊 Advanced Analytics - Real-time insights & performance metrics
• 🔄 Workflow Builder - Visual scraping pipeline creation
• 🎛️ Enterprise Controls - Advanced user management & permissions

Transform your web scraping experience with the next generation of HeadlessX

✨ v1.3.0 Key Features

🛡️ Advanced Anti-Detection Engine

🧠 Human-like Behavioral Simulation

🌐 WAF Bypass Capabilities

📊 Comprehensive Device Profiles

🌐 Website: https://your-subdomain.yourdomain.com
🔗 API: https://your-subdomain.yourdomain.com/api
🛡️ Stealth: https://your-subdomain.yourdomain.com/api/render/stealth
🧪 Testing: https://your-subdomain.yourdomain.com/api/test-fingerprint
📱 Profiles: https://your-subdomain.yourdomain.com/api/profiles
🔧 Health: https://your-subdomain.yourdomain.com/api/health
📊 Status: https://your-subdomain.yourdomain.com/api/status?token=YOUR_AUTH_TOKEN

🏗️ Enhanced Anti-Detection Architecture v1.3.0

v1.3.0 Key Enhancements:

v1.3.0 Architecture Overview:

┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Routes │───▶│ Controllers │───▶│ Services │
│ (api.js) │ │ (rendering.js)│ │ (browser.js) │
│ (admin.js) │ │ (profiles.js) │ │ (stealth.js) │
└─────────────────┘ │ (detection.js)│ │ (interaction.js)
 │ └─────────────────┘ └─────────────────┘
 ▼ │ │
┌─────────────────┐ ▼ ▼
│ Middleware │ ┌─────────────────┐ ┌─────────────────┐
│ (auth.js) │ │ Utils │ │ Config │
│ (error.js) │ │ (logger.js) │ │ (index.js) │
│ (analyzer.js) │ │ (helpers.js) │ │ (browser.js) │
└─────────────────┘ │ (validator.js)│ │ (profiles/) │
 │ └─────────────────┘ └─────────────────┘
 ▼ │ │
┌─────────────────┐ ▼ ▼
│ Fingerprinting │ ┌─────────────────┐ ┌─────────────────┐
│ (canvas-spoof) │ │ Behavioral │ │ Evasion │
│ (webgl-spoof) │ │ (mouse-movement)│ │ (cloudflare) │
│ (audio-context) │ │ (keyboard-dyn) │ │ (datadome) │
│ (webrtc-ctrl) │ │ (scroll-pattern)│ │ (waf-bypass) │
│ (hardware-noise)│ │ (attention-mod) │ │ (tls-fingerpr) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
 │ │ │
 ▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Testing │ │ Development │ │ Profiles │
│ (test-framework)│ │ (dev-tools) │ │ (chrome-prof) │
│ (detection-test)│ │ (profile-gen) │ │ (mobile-prof) │
│ (performance) │ │ (fingerpr-test) │ │ (firefox-prof) │
└─────────────────┘ └─────────────────┘ └─────────────────┘

🚀 Deployment Guide

🐳 Docker Deployment (Recommended)

# Install Docker (if needed)
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Deploy HeadlessX
git clone https://github.com/SaifyXPRO/HeadlessX.git
cd HeadlessX
cp .env.example .env
nano .env # Configure DOMAIN, SUBDOMAIN, AUTH_TOKEN
# Start services
docker-compose up -d
# Optional: Setup SSL
apt install certbot python3-certbot-nginx
certbot --nginx -d your-subdomain.yourdomain.com

docker-compose ps # Check status
docker-compose logs headlessx # View logs
docker-compose restart # Restart services
docker-compose down # Stop services

🔧 Node.js + PM2 Deployment

# Automated setup (recommended)
git clone https://github.com/SaifyXPRO/HeadlessX.git
cd HeadlessX
cp .env.example .env
nano .env # Configure environment
chmod +x scripts/setup.sh
sudo ./scripts/setup.sh # Installs dependencies, builds website, starts PM2

# Copy and configure nginx site
sudo cp nginx/headlessx.conf /etc/nginx/sites-available/headlessx
# Replace placeholders with your actual domain
sudo sed -i 's/SUBDOMAIN.DOMAIN.COM/your-subdomain.yourdomain.com/g' /etc/nginx/sites-available/headlessx
# Enable the site
sudo ln -sf /etc/nginx/sites-available/headlessx /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default
# Test and reload nginx
sudo nginx -t && sudo systemctl reload nginx

sudo apt update && sudo apt upgrade -y
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs build-essential
npm install && npm run build
sudo npm install -g pm2
npm run pm2:start

npm run pm2:status # Check status
npm run pm2:logs # View logs
npm run pm2:restart # Restart server
npm run pm2:stop # Stop server

💻 Development Setup

git clone https://github.com/SaifyXPRO/HeadlessX.git
cd HeadlessX
cp .env.example .env
nano .env # Set AUTH_TOKEN, DOMAIN=localhost, SUBDOMAIN=headlessx
# Make scripts executable
chmod +x scripts/*.sh
# Install dependencies
npm install
cd website && npm install && npm run build && cd ..
# Start development server
npm start # Access at http://localhost:3000

🌐 API Routes & Structure

HeadlessX Routes:
├── /favicon.ico → Favicon
├── /robots.txt → SEO robots file
├── /api/health → Health check (no auth required)
├── /api/status → Server status (requires token)
├── /api/render → Full page rendering
├── /api/html → HTML extraction 
├── /api/content → Clean text extraction
├── /api/screenshot → Screenshot generation
├── /api/pdf → PDF generation
└── /api/batch → Batch URL processing

🚀 API Examples & HTTP Integrations

Quick Health Check (No Auth)

curl https://your-subdomain.yourdomain.com/api/health

🔧 cURL Examples

🛡️ v1.3.0 Anti-Detection Rendering (Maximum Stealth)

curl -X POST "https://your-subdomain.yourdomain.com/api/render/stealth?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "url": "https://example.com",
 "profile": "desktop-chrome",
 "stealthMode": "maximum",
 "behaviorSimulation": true,
 "timeout": 30000
 }'

📱 Mobile Device Simulation

curl -X POST "https://your-subdomain.yourdomain.com/api/render?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "url": "https://example.com",
 "profile": "iphone-14-pro",
 "geolocation": {"latitude": 40.7128, "longitude": -74.0060},
 "behaviorSimulation": true
 }'

🧪 Test Anti-Detection Capabilities

curl -X POST "https://your-subdomain.yourdomain.com/api/test-fingerprint?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "profile": "desktop-chrome",
 "testCanvas": true,
 "testWebGL": true,
 "testAudio": true
 }'

📊 Get Available Device Profiles

curl "https://your-subdomain.yourdomain.com/api/profiles?token=YOUR_AUTH_TOKEN"

🎭 Behavioral Simulation with WAF Bypass

curl -X POST "https://your-subdomain.yourdomain.com/api/render?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "url": "https://example.com",
 "profile": "desktop-firefox",
 "cloudflareBypass": true,
 "datadomeBypass": true,
 "mouseMovement": "natural",
 "keyboardDynamics": "human",
 "timeout": 45000
 }'

Extract HTML Content

curl -X POST "https://your-subdomain.yourdomain.com/api/html?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{"url": "https://example.com", "timeout": 30000}'

Generate Screenshot

curl "https://your-subdomain.yourdomain.com/api/screenshot?token=YOUR_AUTH_TOKEN&url=https://example.com&fullPage=true" \
 -o screenshot.png

Extract Text Only

curl -X POST "https://your-subdomain.yourdomain.com/api/text?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{"url": "https://example.com", "waitForSelector": "main"}'

Generate PDF

curl -X POST "https://your-subdomain.yourdomain.com/api/pdf?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{"url": "https://example.com", "format": "A4"}' \
 -o document.pdf

🤖 Make.com (Integromat) Integration

{
 "url": "https://your-subdomain.yourdomain.com/api/html",
 "method": "POST",
 "headers": {
 "Content-Type": "application/json"
 },
 "qs": {
 "token": "YOUR_AUTH_TOKEN"
 },
 "body": {
 "url": "{{url_to_scrape}}",
 "timeout": 30000,
 "waitForSelector": "{{optional_selector}}"
 }
}

⚡ Zapier Integration

{
 "url": "{{url_from_trigger}}",
 "timeout": 30000,
 "humanBehavior": true
}

🔗 n8n Integration

{
 "url": "https://your-subdomain.yourdomain.com/api/html",
 "method": "POST",
 "authentication": "queryAuth",
 "query": {
 "token": "YOUR_AUTH_TOKEN"
 },
 "headers": {
 "Content-Type": "application/json"
 },
 "body": {
 "url": "={{$json.url}}",
 "timeout": 30000,
 "humanBehavior": true
 }
}

🐍 Python Example

import requests
def scrape_with_headlessx(url, token):
 response = requests.post(
 "https://your-subdomain.yourdomain.com/api/html",
 params={"token": token},
 json={
 "url": url,
 "timeout": 30000,
 "humanBehavior": True
 }
 )
 return response.json()
# Usage
result = scrape_with_headlessx("https://example.com", "YOUR_TOKEN")
print(result['html'])

🟨 JavaScript/Node.js Example

const axios = require('axios');
async function scrapeWithHeadlessX(url, token) {
 try {
 const response = await axios.post(
 `https://your-subdomain.yourdomain.com/api/html?token=${token}`,
 {
 url: url,
 timeout: 30000,
 humanBehavior: true
 }
 );
 return response.data;
 } catch (error) {
 console.error('Scraping failed:', error.message);
 throw error;
 }
}
// Usage
scrapeWithHeadlessX('https://example.com', 'YOUR_TOKEN')
 .then(result => console.log(result.html))
 .catch(error => console.error(error));

🔄 Batch Processing Example

curl -X POST "https://your-subdomain.yourdomain.com/api/batch?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "urls": [
 "https://example1.com",
 "https://example2.com",
 "https://example3.com"
 ],
 "timeout": 30000,
 "humanBehavior": true
 }'

Batch Processing

curl -X POST "https://your-subdomain.yourdomain.com/api/batch?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "urls": ["https://example.com", "https://httpbin.org"],
 "format": "text",
 "options": {"timeout": 30000}
 }'

📁 Project Structure

HeadlessX v1.3.0 - Enhanced Anti-Detection Architecture/
├── 📂 src/ # Modular application source
│ ├── 📂 config/ # Configuration management
│ │ ├── index.js # Main configuration loader
│ │ └── browser.js # Browser-specific settings
│ ├── 📂 utils/ # Utility functions
│ │ ├── errors.js # Error handling & categorization
│ │ ├── logger.js # Structured logging
│ │ └── helpers.js # Common utilities
│ ├── 📂 services/ # Business logic services
│ │ ├── browser.js # Browser lifecycle management
│ │ ├── stealth.js # Anti-detection techniques
│ │ ├── interaction.js # Human-like behavior
│ │ └── rendering.js # Core rendering logic
│ ├── 📂 middleware/ # Express middleware
│ │ ├── auth.js # Authentication
│ │ └── error.js # Error handling
│ ├── 📂 controllers/ # Request handlers
│ │ ├── system.js # Health & status endpoints
│ │ ├── rendering.js # Main rendering endpoints
│ │ ├── batch.js # Batch processing
│ │ └── get.js # GET endpoints & docs
│ ├── 📂 routes/ # Route definitions
│ │ ├── api.js # API route mappings
│ │ └── static.js # Static file serving
│ ├── app.js # Main application setup
│ ├── server.js # Entry point for PM2
│ └── rate-limiter.js # Rate limiting implementation
├── 📂 website/ # Next.js website (unchanged)
│ ├── app/ # Next.js 13+ app directory
│ ├── components/ # React components
│ ├── .env.example # Website environment template
│ ├── next.config.js # Next.js configuration
│ └── package.json # Website dependencies
├── 📂 scripts/ # Deployment & management scripts
│ ├── setup.sh # Automated installation (updated)
│ ├── update_server.sh # Server update script (updated)
│ ├── verify-domain.sh # Domain verification
│ └── test-routing.sh # Integration testing
├── 📂 nginx/ # Nginx configuration
│ └── headlessx.conf # Nginx proxy config
├── 📂 docker/ # Docker deployment (updated)
│ ├── Dockerfile # Container definition
│ └── docker-compose.yml # Docker Compose setup
├── ecosystem.config.js # PM2 configuration (moved to root)
├── .env.example # Environment template (updated)
├── package.json # Server dependencies (updated)
├── docs/
│ └── MODULAR_ARCHITECTURE.md # Architecture documentation
└── README.md # This file

🛠️ Development

Local Development

# 1. Install dependencies
npm install
# 2. Build website
cd website
npm install
npm run build
cd ..
# 3. Set environment variables
export AUTH_TOKEN="development_token_123"
export DOMAIN="localhost"
export SUBDOMAIN="headlessx"
# 4. Start server
npm start # Uses src/app.js
# 5. Access locally
# Website: http://localhost:3000
# API: http://localhost:3000/api/health

Testing Integration

# Test server and website integration
bash scripts/test-routing.sh localhost
# Test with environment variables
bash scripts/verify-domain.sh

⚙️ Configuration

🌐 Environment Variables (.env)

cp .env.example .env
nano .env

# Security Token (Generate a secure random string)
AUTH_TOKEN=your_secure_token_here
# Domain Configuration 
DOMAIN=yourdomain.com
SUBDOMAIN=headlessx
# Optional: Browser Settings
BROWSER_TIMEOUT=60000
MAX_CONCURRENT_BROWSERS=5
# Optional: Server Settings
PORT=3000
NODE_ENV=production

🌐 Nginx Domain Setup

# The setup script automatically replaces domain placeholders
sudo ./scripts/setup.sh

# Copy nginx configuration
sudo cp nginx/headlessx.conf /etc/nginx/sites-available/headlessx
# Replace domain placeholders (replace with your actual domain)
sudo sed -i 's/SUBDOMAIN.DOMAIN.COM/headlessx.yourdomain.com/g' /etc/nginx/sites-available/headlessx
# Example: If your domain is "api.example.com"
sudo sed -i 's/SUBDOMAIN.DOMAIN.COM/api.example.com/g' /etc/nginx/sites-available/headlessx
# Enable site and reload nginx
sudo ln -sf /etc/nginx/sites-available/headlessx /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

📊 API Reference

🔧 Core Endpoints

🔑 Authentication

📖 Complete Documentation

📊 Monitoring & Troubleshooting

🔍 Health Checks

curl https://your-subdomain.yourdomain.com/api/health
curl "https://your-subdomain.yourdomain.com/api/status?token=YOUR_TOKEN"

📋 Log Management

# PM2 logs
npm run pm2:logs
pm2 logs headlessx --lines 100
# Docker logs
docker-compose logs -f headlessx
# Nginx logs
sudo tail -f /var/log/nginx/access.log

🔄 Updates

git pull origin main
npm run build # Rebuild website
npm run pm2:restart # PM2
# OR
docker-compose restart # Docker

🔧 Common Issues

chmod +x scripts/generate-lockfiles.sh
./scripts/generate-lockfiles.sh # Generate lock files
# OR
npm install --production # Use install instead

npm install # Install dependencies

sudo apt update && sudo apt install -y \
 libatk1.0-0t64 libatk-bridge2.0-0t64 libcups2t64 \
 libatspi2.0-0t64 libasound2t64 libxcomposite1

sudo npm install -g pm2
chmod +x scripts/setup.sh # Make script executable
pm2 start config/ecosystem.config.js
pm2 logs headlessx # Check errors

# Make all scripts executable
chmod +x scripts/*.sh
# Or use the quick setup
chmod +x scripts/quick-setup.sh && ./scripts/quick-setup.sh

# Use dedicated Playwright setup script
chmod +x scripts/setup-playwright.sh
./scripts/setup-playwright.sh
# Or install manually:
sudo apt update && sudo apt install -y \
 libgtk-3-0t64 libpangocairo-1.0-0 libcairo-gobject2 \
 libgdk-pixbuf-2.0-0 libdrm2 libxss1 libxrandr2 \
 libasound2t64 libatk1.0-0t64 libnss3
# Install only Chromium (most stable)
npx playwright install chromium
# Alternative: Use Docker (avoids dependency issues)
docker-compose up -d

🔐 Security Features

🤝 Contributing

Ways to Contribute

Development Workflow

� Community

General Q&A Ideas Show & Tell

Get Help & Share Knowledge

Community Guidelines

�📄 License

🆘 Support & Resources

Quick Links

HeadlessX v1.3.0 - The most advanced open-source anti-detection web scraping solution.

GitHub stars GitHub forks GitHub watchers

Made with 🚀 by developers, for developers