🎓 AI Proctoring Engine

🌟 Features

Visual Monitoring

Audio Monitoring

Behavioral Pattern Detection

Browser Telemetry

Architecture

📦 Installation

npm install @timadey/proctor

yarn add @timadey/proctor

🚀 Quick Start

import { ProctoringEngine } from '@timadey/proctor';
// 1. Initialize the engine
const engine = ProctoringEngine.getInstance({
 // Enable/disable modules
 enableVisualDetection: true,
 enableAudioMonitoring: true,
 enablePatternDetection: true,
 enableBrowserTelemetry: true,
 
 // Callbacks
 onEvent: (event) => {
 console.log('Proctoring event:', event);
 // Send to your backend
 },
 
 onBehavioralPattern: (pattern) => {
 console.warn('Suspicious pattern detected:', pattern);
 // Alert supervisor
 }
});
// 2. Initialize modules
await engine.initialize();
// 3. Start proctoring
const videoElement = document.getElementById('webcam');
engine.start(videoElement);
// 4. Get session summary anytime
const summary = engine.getSessionSummary();
console.log('Suspicious score:', summary.suspiciousScore);
// 5. Stop when exam ends
engine.stop();

📚 Table of Contents

🏗️ Architecture

┌────────────────────────────────────────┐
│ ProctoringEngine (Orchestrator) │
│ ┌─────────────┐ ┌──────────────┐ │
│ │EventManager │ │StateManager │ │
│ └─────────────┘ └──────────────┘ │
└────────────┬───────────────────────────┘
 │
 ┌────────┼────────┬────────┐
 ▼ ▼ ▼ ▼
┌────────┐ ┌────┐ ┌────┐ ┌────────┐
│Visual │ │Audio│ │Pat │ │Browser │
│Module │ │Mod │ │Mod │ │Telemetry│
└────────┘ └────┘ └────┘ └────────┘

Core Components

🧩 Modules

VisualDetectionModule

// Access directly if needed
const visualState = engine.visualModule.getState();
console.log('Current gaze:', visualState.currentGazeDirection);
console.log('Number of faces:', visualState.numFaces);
console.log('Mouth moving:', visualState.isMouthMoving);
console.log('Object detected:', visualState.suspiciousObjectDetected);

AudioMonitoringModule

// Access audio state
const audioState = engine.audioModule.getState();
console.log('Is talking:', audioState.isTalking);
console.log('Is whispering:', audioState.isWhispering);
console.log('Audio level:', audioState.currentAudioLevel, 'dB');
console.log('Total talking time:', audioState.totalTalkingDuration, 'ms');

PatternDetectionModule

// Get pattern summary
const patterns = engine.patternModule.getPatternSummary();
console.log('Suspicious patterns detected:', patterns);

BrowserTelemetryModule

// Get telemetry summary
const telemetry = engine.telemetryModule.getSummary();
console.log('Tab switches:', telemetry.tabSwitches);
console.log('Copy attempts:', telemetry.copyAttempts);
console.log('Paste attempts:', telemetry.pasteAttempts);

⚙️ Configuration

Complete Configuration Example

const engine = ProctoringEngine.getInstance({
 // ===== Module Toggles =====
 enableVisualDetection: true,
 enableAudioMonitoring: true,
 enablePatternDetection: true,
 enableBrowserTelemetry: true,
 
 // ===== Visual Detection Options =====
 detectionFPS: 10, // Frame processing rate (5-30)
 stabilityFrames: 15, // Frames before event triggers
 gazeThreshold: 20, // Degrees for gaze deviation
 yawThreshold: 25, // Degrees for head rotation
 pitchThreshold: 20, // Degrees for head tilt
 prolongedGazeAwayDuration: 5000, // ms for prolonged gaze
 mouthOpenRatioThreshold: 0.15, // Mouth aspect ratio threshold
 
 // ===== Audio Monitoring Options =====
 talkingThreshold: -45, // dB for talking detection
 whisperThreshold: -55, // dB for whisper detection
 audioSampleInterval: 100, // Audio check interval (ms)
 prolongedTalkingDuration: 3000, // ms for prolonged talking
 
 // ===== Pattern Detection Options =====
 suspiciousPatternThreshold: 3, // Events to trigger pattern
 patternDetectionWindow: 10000, // Time window (ms)
 
 // ===== Callbacks =====
 onEvent: (event) => {
 // Handle individual events
 console.log('Event:', event);
 },
 
 onBehavioralPattern: (pattern) => {
 // Handle detected patterns (critical)
 console.warn('Pattern:', pattern);
 },
 
 onStatusChange: (status) => {
 // Engine status: 'initializing', 'loading-models', 'ready', 'error'
 console.log('Status:', status);
 },
 
 onError: (error) => {
 // Handle errors
 console.error('Error:', error);
 }
});

Selective Module Configuration

// Lightweight mode - only browser telemetry
const lightEngine = ProctoringEngine.getInstance({
 enableVisualDetection: false,
 enableAudioMonitoring: false,
 enablePatternDetection: false,
 enableBrowserTelemetry: true
});
// Heavy mode - full monitoring
const heavyEngine = ProctoringEngine.getInstance({
 enableVisualDetection: true,
 enableAudioMonitoring: true,
 enablePatternDetection: true,
 enableBrowserTelemetry: true,
 detectionFPS: 15 // Higher FPS for more accuracy
});
// Custom mode - visual + patterns only
const customEngine = ProctoringEngine.getInstance({
 enableVisualDetection: true,
 enableAudioMonitoring: false,
 enablePatternDetection: true,
 enableBrowserTelemetry: true
});

Runtime Configuration Updates

// Update configuration during session
engine.updateOptions({
 gazeThreshold: 30, // More lenient
 detectionFPS: 5 // Reduce CPU usage
});
// Update individual modules
engine.visualModule.updateOptions({
 detectionFPS: 8
});
engine.audioModule.updateOptions({
 talkingThreshold: -40
});

📡 Events

Event Structure

{
 event: 'TALKING_DETECTED', // Event type
 lv: 8, // Severity level (1-10)
 ts: 1703098765432, // Timestamp (Unix ms)
 source: 'audio', // Module source
 sessionDuration: 123456, // Time since session start (ms)
 
 // Event-specific metadata
 duration: 5000, // Duration of behavior (ms)
 level: -40, // Audio level (dB)
 direction: 'left', // Direction (for gaze/head)
 severity: 'high', // Human-readable severity
 extractedFeatures: {} // The face and hand features extracted from the frame
}

Event Types by Severity

Critical (9-10)

High (7-8)

Medium (5-6)

Low (1-4)

🎯 Behavioral Patterns

Pattern: Suspicious Triple Pattern

onBehavioralPattern: (pattern) => {
 if (pattern.pattern === 'suspiciousTriplePattern') {
 // This is extremely suspicious
 alertSupervisor('Student likely cheating');
 flagExamForReview();
 }
}

Pattern: Looking Left/Right + Whispering

Pattern: Mouth Covered + Audio

Pattern: Object + Looking Away

Pattern: Multiple Faces + Audio

Custom Pattern Detection

// Add custom pattern
engine.patternModule.patterns.myCustomPattern = {
 name: 'myCustomPattern',
 severity: 8,
 events: [],
 count: 0,
 lastTriggered: 0,
 check: (visualState, audioState) => {
 // Your custom logic
 return visualState.numFaces === 0 && 
 audioState.isTalking;
 }
};

📖 API Reference

ProctoringEngine

getInstance(options)

const engine = ProctoringEngine.getInstance(options);

async initialize()

await engine.initialize();

start(videoElement)

const video = document.getElementById('webcam');
engine.start(video);

stop()

engine.stop();

updateOptions(options)

engine.updateOptions({ detectionFPS: 5 });

getSessionSummary()

const summary = engine.getSessionSummary();
/*
{
 sessionDuration: 1800000,
 sessionStartTime: 1703098765432,
 sessionEndTime: 1703100565432,
 totalEvents: 45,
 eventCounts: {...},
 eventsBySeverity: {...},
 patterns: {...},
 visualState: {...},
 audioState: {...},
 suspiciousScore: 127
}
*/

getLogs()

const logs = engine.getLogs();

clearLogs()

engine.clearLogs();

calculateSuspiciousScore()

const score = engine.calculateSuspiciousScore();
// 0-50: Normal
// 51-100: Some suspicious activity
// 101-200: Concerning behavior
// 201+: High probability of cheating

destroy()

engine.destroy();

StateManager

subscribe(callback)

const unsubscribe = engine.stateManager.subscribe((state) => {
 console.log('State updated:', state);
});
// Unsubscribe later
unsubscribe();

getCompleteState()

const state = engine.stateManager.getCompleteState();

getVisualState()

const visual = engine.stateManager.getVisualState();

getAudioState()

const audio = engine.stateManager.getAudioState();

EventManager

getAllEvents()

const events = engine.eventManager.getAllEvents();

getEventsByType(type)

const tabSwitches = engine.eventManager.getEventsByType('TAB_SWITCHED');

getEventsBySeverity(minSeverity)

const critical = engine.eventManager.getEventsBySeverity(9);

getSummary()

const summary = engine.eventManager.getSummary();

💡 Examples

Basic Exam Proctoring

import { ProctoringEngine } from './ProctoringEngine.js';
class SimpleProctor {
 constructor() {
 this.engine = ProctoringEngine.getInstance({
 onEvent: (e) => console.log('Event:', e.event),
 onBehavioralPattern: (p) => alert(`Warning: ${p.pattern}`)
 });
 }
 async start() {
 // Get camera
 const stream = await navigator.mediaDevices.getUserMedia({
 video: true
 });
 const video = document.getElementById('video');
 video.srcObject = stream;
 await video.play();
 // Start proctoring
 await this.engine.initialize();
 this.engine.start(video);
 }
 stop() {
 const summary = this.engine.getSessionSummary();
 console.log('Final score:', summary.suspiciousScore);
 this.engine.stop();
 }
}
const proctor = new SimpleProctor();
await proctor.start();

Advanced Integration with Backend

class AdvancedProctor {
 constructor(examId, studentId) {
 this.examId = examId;
 this.studentId = studentId;
 this.ws = null; // WebSocket connection
 
 this.engine = ProctoringEngine.getInstance({
 onEvent: (event) => this.handleEvent(event),
 onBehavioralPattern: (pattern) => this.handlePattern(pattern)
 });
 }
 async start() {
 // Connect to backend via WebSocket
 this.ws = new WebSocket('wss://api.example.com/proctoring');
 
 // Setup camera
 const stream = await navigator.mediaDevices.getUserMedia({
 video: { width: 1280, height: 720 }
 });
 const video = document.getElementById('video');
 video.srcObject = stream;
 await video.play();
 // Initialize and start
 await this.engine.initialize();
 this.engine.start(video);
 // Subscribe to state for real-time updates
 this.engine.stateManager.subscribe((state) => {
 this.sendStateUpdate(state);
 });
 // Periodic summaries
 this.summaryInterval = setInterval(() => {
 this.sendSummary();
 }, 30000); // Every 30 seconds
 }
 handleEvent(event) {
 // Send event to backend via REST
 fetch('/api/proctoring/event', {
 method: 'POST',
 headers: { 'Content-Type': 'application/json' },
 body: JSON.stringify({
 examId: this.examId,
 studentId: this.studentId,
 event: event
 })
 });
 // Send via WebSocket for real-time monitoring
 if (this.ws && this.ws.readyState === WebSocket.OPEN) {
 this.ws.send(JSON.stringify({
 type: 'EVENT',
 data: event
 }));
 }
 // Show to student if critical
 if (event.lv >= 8) {
 this.showWarning(event.event);
 }
 }
 handlePattern(pattern) {
 // Critical alert - send immediately
 fetch('/api/proctoring/critical', {
 method: 'POST',
 headers: { 'Content-Type': 'application/json' },
 body: JSON.stringify({
 examId: this.examId,
 studentId: this.studentId,
 pattern: pattern,
 timestamp: Date.now()
 }),
 keepalive: true // Ensure delivery
 });
 // Alert supervisor via WebSocket
 if (this.ws && this.ws.readyState === WebSocket.OPEN) {
 this.ws.send(JSON.stringify({
 type: 'CRITICAL_PATTERN',
 data: pattern
 }));
 }
 // Show strong warning to student
 this.showCriticalWarning(pattern.pattern);
 }
 sendStateUpdate(state) {
 if (this.ws && this.ws.readyState === WebSocket.OPEN) {
 this.ws.send(JSON.stringify({
 type: 'STATE_UPDATE',
 data: {
 examId: this.examId,
 studentId: this.studentId,
 state: state
 }
 }));
 }
 }
 sendSummary() {
 const summary = this.engine.getSessionSummary();
 
 fetch('/api/proctoring/summary', {
 method: 'POST',
 headers: { 'Content-Type': 'application/json' },
 body: JSON.stringify({
 examId: this.examId,
 studentId: this.studentId,
 summary: summary
 })
 });
 }
 showWarning(eventType) {
 const warnings = {
 'TALKING_DETECTED': 'Please remain quiet during the exam.',
 'TAB_SWITCHED': 'Do not switch tabs during the exam.',
 'MULTIPLE_FACES': 'Multiple people detected. Only you should be visible.',
 'PERSON_LEFT': 'You have left the exam area.',
 };
 const message = warnings[eventType] || 'Suspicious activity detected.';
 const warning = document.getElementById('warning');
 warning.textContent = message;
 warning.classList.add('show');
 
 setTimeout(() => {
 warning.classList.remove('show');
 }, 5000);
 }
 showCriticalWarning(patternName) {
 const modal = document.getElementById('critical-modal');
 modal.querySelector('.message').textContent = 
 `Critical violation detected: ${patternName}. This exam may be flagged for review.`;
 modal.classList.add('show');
 
 setTimeout(() => {
 modal.classList.remove('show');
 }, 10000);
 }
 async stop() {
 // Clear interval
 if (this.summaryInterval) {
 clearInterval(this.summaryInterval);
 }
 // Get final data
 const summary = this.engine.getSessionSummary();
 const logs = this.engine.getLogs();
 // Send final report
 const finalData = {
 examId: this.examId,
 studentId: this.studentId,
 summary: summary,
 logs: logs,
 endTime: Date.now()
 };
 // Use sendBeacon for reliability during unload
 const blob = new Blob([JSON.stringify(finalData)], {
 type: 'application/json'
 });
 navigator.sendBeacon('/api/proctoring/finalize', blob);
 // Close WebSocket
 if (this.ws) {
 this.ws.close();
 }
 // Destroy engine
 this.engine.destroy();
 }
}
// Usage
const proctor = new AdvancedProctor('exam-123', 'student-456');
await proctor.start();
// On exam submit
document.getElementById('submit-btn').addEventListener('click', async () => {
 await proctor.stop();
 // Submit exam answers...
});
// On page unload
window.addEventListener('beforeunload', () => {
 proctor.stop();
});

React Integration

import React, { useEffect, useRef, useState } from 'react';
import { ProctoringEngine } from './ProctoringEngine';
function ExamProctoring({ examId, studentId }) {
 const videoRef = useRef(null);
 const engineRef = useRef(null);
 
 const [status, setStatus] = useState('initializing');
 const [events, setEvents] = useState([]);
 const [score, setScore] = useState(0);
 const [warning, setWarning] = useState('');
 useEffect(() => {
 let mounted = true;
 const initialize = async () => {
 try {
 // Setup camera
 const stream = await navigator.mediaDevices.getUserMedia({
 video: { width: 1280, height: 720 }
 });
 
 if (videoRef.current) {
 videoRef.current.srcObject = stream;
 await videoRef.current.play();
 }
 // Initialize engine
 const engine = ProctoringEngine.getInstance({
 onEvent: (event) => {
 if (mounted) {
 setEvents(prev => [event, ...prev].slice(0, 20));
 }
 },
 onBehavioralPattern: (pattern) => {
 if (mounted) {
 setWarning(`⚠️ ${pattern.pattern} detected`);
 setTimeout(() => setWarning(''), 5000);
 }
 },
 onStatusChange: (newStatus) => {
 if (mounted) setStatus(newStatus);
 }
 });
 engineRef.current = engine;
 await engine.initialize();
 
 if (videoRef.current) {
 engine.start(videoRef.current);
 }
 // Update score periodically
 const interval = setInterval(() => {
 if (engineRef.current && mounted) {
 const summary = engineRef.current.getSessionSummary();
 setScore(summary.suspiciousScore);
 }
 }, 5000);
 return () => {
 clearInterval(interval);
 };
 } catch (error) {
 console.error('Initialization failed:', error);
 if (mounted) setStatus('error');
 }
 };
 initialize();
 return () => {
 mounted = false;
 if (engineRef.current) {
 engineRef.current.stop();
 engineRef.current.destroy();
 }
 };
 }, [examId, studentId]);
 return (
 <div className="proctoring-container">
 <video 
 ref={videoRef} 
 autoPlay 
 playsInline 
 muted
 className="proctoring-video"
 />
 
 <div className="proctoring-info">
 <div className="status">
 Status: <span className={status}>{status}</span>
 </div>
 <div className="score">
 Suspicious Score: <span>{score}</span>
 </div>
 </div>
 {warning && (
 <div className="warning-banner">
 {warning}
 </div>
 )}
 <div className="event-log">
 <h3>Recent Events</h3>
 {events.map((event, i) => (
 <div key={i} className={`event severity-${event.lv}`}>
 {new Date(event.ts).toLocaleTimeString()} - {event.event}
 </div>
 ))}
 </div>
 </div>
 );
}
export default ExamProctoring;

🌐 Browser Support

Requirements

Permissions Required

// Camera permission
await navigator.mediaDevices.getUserMedia({ video: true });
// Microphone permission (handled internally)
// Requested automatically by AudioMonitoringModule

⚡ Performance

Optimization Tips

// Lower FPS for better performance
engine.updateOptions({ detectionFPS: 5 });

// Fewer false positives, better performance
engine.updateOptions({ stabilityFrames: 20 });

// Only enable what you need
ProctoringEngine.getInstance({
 enableVisualDetection: true,
 enableAudioMonitoring: false, // Disable if not needed
 enablePatternDetection: true,
 enableBrowserTelemetry: true
});

Performance Metrics

🔒 Security

Data Privacy

Best Practices

// Show consent dialog before starting
const consent = await showConsentDialog();
if (consent) {
 await proctor.start();
}

// Always use secure connections
fetch('https://api.example.com/proctoring/event', {
 method: 'POST',
 // ...
});

// Clear logs after exam
window.addEventListener('beforeunload', () => {
 engine.clearLogs();
});

// Adjust for students with disabilities
engine.updateOptions({
 gazeThreshold: 35, // More lenient
 prolongedGazeAwayDuration: 10000
});

🐛 Troubleshooting

Camera Not Working

// Check permission
navigator.permissions.query({ name: 'camera' })
 .then(result => {
 console.log('Camera permission:', result.state);
 if (result.state === 'denied') {
 alert('Please allow camera access');
 }
 });

Audio Not Detected

// Check microphone permission
navigator.permissions.query({ name: 'microphone' })
 .then(result => {
 console.log('Microphone permission:', result.state);
 });
// Check if audio module initialized
if (!engine.audioModule.isSetup) {
 console.warn('Audio monitoring not available');
}

High CPU Usage

// Reduce detection FPS
engine.updateOptions({ detectionFPS: 5 });
// Disable unnecessary modules
engine.updateOptions({
 enableAudioMonitoring: false
});

Models Not Loading

// Check network connection
// MediaPipe models loaded from CDN
// Ensure CDN is accessible: https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/

🤝 Contributing

Development Setup

# Clone repository
git clone https://github.com/yourusername/ai-proctoring-engine.git
# Install dependencies
cd ai-proctoring-engine
npm install
# Run tests
npm test
# Build
npm run build

📄 License

🙏 Acknowledgments

📞 Support

🗺️ Roadmap