🚀 HeadlessX v1.2.0

✨ Key Features

🎯 Quick Start

# 1. Clone and configure
git clone https://github.com/SaifyXPRO/HeadlessX.git
cd HeadlessX
# Quick setup (makes scripts executable + creates .env)
chmod +x scripts/quick-setup.sh && ./scripts/quick-setup.sh
# Then edit: nano .env # Update DOMAIN, SUBDOMAIN, and AUTH_TOKEN

🌐 Website: https://your-subdomain.yourdomain.com
🔧 Health: https://your-subdomain.yourdomain.com/api/health
📊 Status: https://your-subdomain.yourdomain.com/api/status?token=YOUR_AUTH_TOKEN

🏗️ New Modular Architecture v1.2.0

Key Improvements:

Architecture Overview:

┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Routes │───▶│ Controllers │───▶│ Services │
│ (api.js) │ │ (rendering.js)│ │ (browser.js) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
 │ │ │
 ▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Middleware │ │ Utils │ │ Config │
│ (auth.js) │ │ (logger.js) │ │ (index.js) │
└─────────────────┘ └─────────────────┘ └─────────────────┘

🚀 Deployment Guide

🐳 Docker Deployment (Recommended)

# Install Docker (if needed)
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER
# Deploy HeadlessX
git clone https://github.com/SaifyXPRO/HeadlessX.git
cd HeadlessX
cp .env.example .env
nano .env # Configure DOMAIN, SUBDOMAIN, AUTH_TOKEN
# Start services
docker-compose up -d
# Optional: Setup SSL
sudo apt install certbot
sudo certbot --standalone -d your-subdomain.yourdomain.com

docker-compose ps # Check status
docker-compose logs headlessx # View logs
docker-compose restart # Restart services
docker-compose down # Stop services

🔧 Node.js + PM2 Deployment

# Automated setup (recommended)
git clone https://github.com/SaifyXPRO/HeadlessX.git
cd HeadlessX
cp .env.example .env
nano .env # Configure environment
chmod +x scripts/setup.sh
sudo ./scripts/setup.sh # Installs dependencies, builds website, starts PM2

# Copy and configure nginx site
sudo cp nginx/headlessx.conf /etc/nginx/sites-available/headlessx
# Replace placeholders with your actual domain
sudo sed -i 's/SUBDOMAIN.DOMAIN.COM/your-subdomain.yourdomain.com/g' /etc/nginx/sites-available/headlessx
# Enable the site
sudo ln -sf /etc/nginx/sites-available/headlessx /etc/nginx/sites-enabled/
sudo rm -f /etc/nginx/sites-enabled/default
# Test and reload nginx
sudo nginx -t && sudo systemctl reload nginx

sudo apt update && sudo apt upgrade -y
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs build-essential
npm install && npm run build
sudo npm install -g pm2
npm run pm2:start

npm run pm2:status # Check status
npm run pm2:logs # View logs
npm run pm2:restart # Restart server
npm run pm2:stop # Stop server

💻 Development Setup

git clone https://github.com/SaifyXPRO/HeadlessX.git
cd HeadlessX
cp .env.example .env
nano .env # Set AUTH_TOKEN, DOMAIN=localhost, SUBDOMAIN=headlessx
# Make scripts executable
chmod +x scripts/*.sh
# Install dependencies
npm install
cd website && npm install && npm run build && cd ..
# Start development server
npm start # Access at http://localhost:3000

🌐 API Routes & Structure

HeadlessX Routes:
├── /favicon.ico → Favicon
├── /robots.txt → SEO robots file
├── /api/health → Health check (no auth required)
├── /api/status → Server status (requires token)
├── /api/render → Full page rendering
├── /api/html → HTML extraction 
├── /api/content → Clean text extraction
├── /api/screenshot → Screenshot generation
├── /api/pdf → PDF generation
└── /api/batch → Batch URL processing

🚀 API Examples & HTTP Integrations

Quick Health Check (No Auth)

curl https://your-subdomain.yourdomain.com/api/health

🔧 cURL Examples

Extract HTML Content

curl -X POST "https://your-subdomain.yourdomain.com/api/html?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{"url": "https://example.com", "timeout": 30000}'

Generate Screenshot

curl "https://your-subdomain.yourdomain.com/api/screenshot?token=YOUR_AUTH_TOKEN&url=https://example.com&fullPage=true" \
 -o screenshot.png

Extract Text Only

curl -X POST "https://your-subdomain.yourdomain.com/api/text?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{"url": "https://example.com", "waitForSelector": "main"}'

Generate PDF

curl -X POST "https://your-subdomain.yourdomain.com/api/pdf?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{"url": "https://example.com", "format": "A4"}' \
 -o document.pdf

🤖 Make.com (Integromat) Integration

{
 "url": "https://your-subdomain.yourdomain.com/api/html",
 "method": "POST",
 "headers": {
 "Content-Type": "application/json"
 },
 "qs": {
 "token": "YOUR_AUTH_TOKEN"
 },
 "body": {
 "url": "{{url_to_scrape}}",
 "timeout": 30000,
 "waitForSelector": "{{optional_selector}}"
 }
}

⚡ Zapier Integration

{
 "url": "{{url_from_trigger}}",
 "timeout": 30000,
 "humanBehavior": true
}

🔗 n8n Integration

{
 "url": "https://your-subdomain.yourdomain.com/api/html",
 "method": "POST",
 "authentication": "queryAuth",
 "query": {
 "token": "YOUR_AUTH_TOKEN"
 },
 "headers": {
 "Content-Type": "application/json"
 },
 "body": {
 "url": "={{$json.url}}",
 "timeout": 30000,
 "humanBehavior": true
 }
}

🐍 Python Example

import requests
def scrape_with_headlessx(url, token):
 response = requests.post(
 "https://your-subdomain.yourdomain.com/api/html",
 params={"token": token},
 json={
 "url": url,
 "timeout": 30000,
 "humanBehavior": True
 }
 )
 return response.json()
# Usage
result = scrape_with_headlessx("https://example.com", "YOUR_TOKEN")
print(result['html'])

🟨 JavaScript/Node.js Example

const axios = require('axios');
async function scrapeWithHeadlessX(url, token) {
 try {
 const response = await axios.post(
 `https://your-subdomain.yourdomain.com/api/html?token=${token}`,
 {
 url: url,
 timeout: 30000,
 humanBehavior: true
 }
 );
 return response.data;
 } catch (error) {
 console.error('Scraping failed:', error.message);
 throw error;
 }
}
// Usage
scrapeWithHeadlessX('https://example.com', 'YOUR_TOKEN')
 .then(result => console.log(result.html))
 .catch(error => console.error(error));

🔄 Batch Processing Example

curl -X POST "https://your-subdomain.yourdomain.com/api/batch?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "urls": [
 "https://example1.com",
 "https://example2.com",
 "https://example3.com"
 ],
 "timeout": 30000,
 "humanBehavior": true
 }'

Batch Processing

curl -X POST "https://your-subdomain.yourdomain.com/api/batch?token=YOUR_AUTH_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "urls": ["https://example.com", "https://httpbin.org"],
 "format": "text",
 "options": {"timeout": 30000}
 }'

📁 Project Structure

HeadlessX v1.2.0 - Modular Architecture/
├── 📂 src/ # Modular application source
│ ├── 📂 config/ # Configuration management
│ │ ├── index.js # Main configuration loader
│ │ └── browser.js # Browser-specific settings
│ ├── 📂 utils/ # Utility functions
│ │ ├── errors.js # Error handling & categorization
│ │ ├── logger.js # Structured logging
│ │ └── helpers.js # Common utilities
│ ├── 📂 services/ # Business logic services
│ │ ├── browser.js # Browser lifecycle management
│ │ ├── stealth.js # Anti-detection techniques
│ │ ├── interaction.js # Human-like behavior
│ │ └── rendering.js # Core rendering logic
│ ├── 📂 middleware/ # Express middleware
│ │ ├── auth.js # Authentication
│ │ └── error.js # Error handling
│ ├── 📂 controllers/ # Request handlers
│ │ ├── system.js # Health & status endpoints
│ │ ├── rendering.js # Main rendering endpoints
│ │ ├── batch.js # Batch processing
│ │ └── get.js # GET endpoints & docs
│ ├── 📂 routes/ # Route definitions
│ │ ├── api.js # API route mappings
│ │ └── static.js # Static file serving
│ ├── app.js # Main application setup
│ ├── server.js # Entry point for PM2
│ └── rate-limiter.js # Rate limiting implementation
├── 📂 website/ # Next.js website (unchanged)
│ ├── app/ # Next.js 13+ app directory
│ ├── components/ # React components
│ ├── .env.example # Website environment template
│ ├── next.config.js # Next.js configuration
│ └── package.json # Website dependencies
├── 📂 scripts/ # Deployment & management scripts
│ ├── setup.sh # Automated installation (updated)
│ ├── update_server.sh # Server update script (updated)
│ ├── verify-domain.sh # Domain verification
│ └── test-routing.sh # Integration testing
├── 📂 nginx/ # Nginx configuration
│ └── headlessx.conf # Nginx proxy config
├── 📂 docker/ # Docker deployment (updated)
│ ├── Dockerfile # Container definition
│ └── docker-compose.yml # Docker Compose setup
├── ecosystem.config.js # PM2 configuration (moved to root)
├── .env.example # Environment template (updated)
├── package.json # Server dependencies (updated)
├── MODULAR_ARCHITECTURE.md # Architecture documentation
└── README.md # This file

🛠️ Development

Local Development

# 1. Install dependencies
npm install
# 2. Build website
cd website
npm install
npm run build
cd ..
# 3. Set environment variables
export AUTH_TOKEN="development_token_123"
export DOMAIN="localhost"
export SUBDOMAIN="headlessx"
# 4. Start server
npm start # Uses src/app.js
# 5. Access locally
# Website: http://localhost:3000
# API: http://localhost:3000/api/health

Testing Integration

# Test server and website integration
bash scripts/test-routing.sh localhost
# Test with environment variables
bash scripts/verify-domain.sh

⚙️ Configuration

🌐 Environment Variables (.env)

cp .env.example .env
nano .env

# Security Token (Generate a secure random string)
AUTH_TOKEN=your_secure_token_here
# Domain Configuration 
DOMAIN=yourdomain.com
SUBDOMAIN=headlessx
# Optional: Browser Settings
BROWSER_TIMEOUT=60000
MAX_CONCURRENT_BROWSERS=5
# Optional: Server Settings
PORT=3000
NODE_ENV=production

🌐 Nginx Domain Setup

# The setup script automatically replaces domain placeholders
sudo ./scripts/setup.sh

# Copy nginx configuration
sudo cp nginx/headlessx.conf /etc/nginx/sites-available/headlessx
# Replace domain placeholders (replace with your actual domain)
sudo sed -i 's/SUBDOMAIN.DOMAIN.COM/headlessx.yourdomain.com/g' /etc/nginx/sites-available/headlessx
# Example: If your domain is "api.example.com"
sudo sed -i 's/SUBDOMAIN.DOMAIN.COM/api.example.com/g' /etc/nginx/sites-available/headlessx
# Enable site and reload nginx
sudo ln -sf /etc/nginx/sites-available/headlessx /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

📊 API Reference

🔧 Core Endpoints

🔑 Authentication

📖 Complete Documentation

📊 Monitoring & Troubleshooting

🔍 Health Checks

curl https://your-subdomain.yourdomain.com/api/health
curl "https://your-subdomain.yourdomain.com/api/status?token=YOUR_TOKEN"

📋 Log Management

# PM2 logs
npm run pm2:logs
pm2 logs headlessx --lines 100
# Docker logs
docker-compose logs -f headlessx
# Nginx logs
sudo tail -f /var/log/nginx/access.log

🔄 Updates

git pull origin main
npm run build # Rebuild website
npm run pm2:restart # PM2
# OR
docker-compose restart # Docker

🔧 Common Issues

chmod +x scripts/generate-lockfiles.sh
./scripts/generate-lockfiles.sh # Generate lock files
# OR
npm install --production # Use install instead

npm install # Install dependencies

sudo apt update && sudo apt install -y \
 libatk1.0-0t64 libatk-bridge2.0-0t64 libcups2t64 \
 libatspi2.0-0t64 libasound2t64 libxcomposite1

sudo npm install -g pm2
chmod +x scripts/setup.sh # Make script executable
pm2 start config/ecosystem.config.js
pm2 logs headlessx # Check errors

# Make all scripts executable
chmod +x scripts/*.sh
# Or use the quick setup
chmod +x scripts/quick-setup.sh && ./scripts/quick-setup.sh

# Use dedicated Playwright setup script
chmod +x scripts/setup-playwright.sh
./scripts/setup-playwright.sh
# Or install manually:
sudo apt update && sudo apt install -y \
 libgtk-3-0t64 libpangocairo-1.0-0 libcairo-gobject2 \
 libgdk-pixbuf-2.0-0 libdrm2 libxss1 libxrandr2 \
 libasound2t64 libatk1.0-0t64 libnss3
# Install only Chromium (most stable)
npx playwright install chromium
# Alternative: Use Docker (avoids dependency issues)
docker-compose up -d

🔐 Security Features

🤝 Contributing

📄 License

🆘 Support

🎯 Built by SaifyXPRO