An intelligent workout tracking system that uses computer vision and machine learning to automatically detect and count exercise repetitions in real-time videos.

WorkoutTracker combines MediaPipe pose detection with a Temporal Convolutional Network (TCN) to perform two main tasks:

• Exercise Classification: Identify the type of exercise (push-ups, squats, pull-ups, dips, no-exercise)
• Repetition Segmentation: Detect and count individual repetitions within the exercise

1. Raw Videos → MediaPipe pose extraction → Joint angle features
2. Manual Labels → CSV files with repetition start markers
3. Gaussian Augmentation → Smooth temporal labels around rep markers
4. Multitask Dataset → Combined features + classification + segmentation labels

• TCN Backbone: 8-layer temporal convolutional network with residual connections
• Multi-Head Attention: Captures global temporal dependencies
• Dual Outputs: Classification head + segmentation head
• Balanced Training: Focal loss with class balancing for better recall

WorkoutTracker/
├── data/
│ ├── raw/ # Original exercise videos
│ │ ├── push-ups/ # Push-up videos
│ │ ├── squats/ # Squat videos
│ │ ├── pull-ups/ # Pull-up videos
│ │ ├── dips/ # Dip videos
│ │ └── no-exercise/ # Non-exercise videos
│ ├── labels/ # Manual CSV labels
│ │ ├── push-ups/ # Push-up labels
│ │ ├── squats/ # Squat labels
│ │ ├── pull-ups/ # Pull-up labels
│ │ ├── dips/ # Dip labels
│ │ └── no_exercise/ # No-exercise labels
│ └── processed/ # Generated datasets
│ └── multitask_dataset.npz
├── models/ # Trained models
│ └── main/ # Current best model
│ ├── main.keras # Model weights
│ └── training_history.npy
├── src/
│ ├── core/ # Dataset building
│ │ ├── dataset_builder.py
│ │ └── improved_dataset_builder.py
│ ├── training/ # Model training
│ │ ├── trainer.py
│ │ ├── model.py
│ │ └── balanced_generator.py
│ ├── demo/ # Demo applications
│ │ ├── demo.py
│ │ └── live/
│ └── utils/ # Utilities
│ ├── video_labeler.py
│ └── csv_format_converter.py
├── demo_output/ # Demo results
└── requirements.txt

# Clone the repository
git clone <repository-url>
cd WorkoutTracker
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt

Place your exercise videos in the appropriate directories:

• data/raw/push-ups/ - Push-up videos
• data/raw/squats/ - Squat videos
• data/raw/pull-ups/ - Pull-up videos
• data/raw/dips/ - Dip videos
• data/raw/no-exercise/ - Non-exercise videos

Use the video labeling tool to create CSV files with repetition markers:

python src/utils/video_labeler.py data/raw/push-ups/video1.mp4

This creates data/labels/push-ups/video1.csv with frame-by-frame labels.

Create the dataset from your labeled videos:

python build_dataset.py

This generates data/processed/multitask_dataset.npz with:

• Features: (N, 30, 25) - 30-frame windows of 25 joint angles
• Classification labels: (N,) - Exercise type (0-4)
• Segmentation labels: (N, 30) - Per-frame repetition probability

Train the model:

python train.py

This creates models/main/main.keras with the trained model.

Analyze a video and generate results:

python src/demo/demo.py --video data/test_videos/test0.mp4 --output demo_output

Real-time exercise detection from webcam:

cd src/demo/live
./start_live_demo.sh

The multitask_dataset.npz is created by:

1. Feature Extraction: MediaPipe pose detection → 25 joint angles per frame
2. Temporal Windowing: 30-frame sliding windows (1 second at 30 FPS)
3. Label Augmentation: Gaussian smoothing around repetition markers
4. Class Balancing: Includes no-exercise samples for better generalization

Instead of binary 0/1 labels, we use Gaussian augmentation:

• Center (rep start): Probability = 1.0
• ±4 frames: Probability = 0.5
• ±12 frames: Probability ≈ 0.1
• Creates smooth temporal patterns for better training

• Total Sequences: ~46,700
• Window Size: 30 frames (1 second)
• Features: 25 joint angles per frame
• Classes: 5 (push-ups, squats, pull-ups, dips, no-exercise)
• Positive Samples: ~9.9% (repetition frames)

• Input: (batch_size, 30, 25) - 30 frames ×ばつ 25 joint angles
• TCN Backbone: 8 residual blocks with dilated convolutions
• Attention: Multi-head attention for global temporal dependencies
• Outputs:
  • Classification: 5 classes (softmax)
  • Segmentation: 30 probabilities (sigmoid)

• Optimizer: Adam (lr=5e-4)
• Loss: Focal Loss (γ=1.0, α=0.5) + Binary Crossentropy
• Balanced Sampling: 20% positive, 80% negative samples
• Augmentation: Gaussian label smoothing
• Regularization: Dropout (0.25), Early stopping (patience=20)

Analyzes pre-recorded videos and generates:

• Annotated video with detected repetitions
• Analysis plot showing exercise classification and repetition detection
• Repetition count and confidence scores

Usage:

python src/demo/demo.py --video path/to/video.mp4 --output output_directory

Real-time exercise detection from webcam:

• Live pose detection and skeleton overlay
• Real-time repetition counting
• Exercise type classification

Usage:

cd src/demo/live
./start_live_demo.sh
## 📈 Performance
### Current Model Metrics
- **Classification Accuracy**: ~99.9%
- **Segmentation Precision**: ~97%
- **Segmentation Recall**: ~38%
- **AUC**: ~0.93
### Model Comparison
- **Robust Model**: Better generalization, fewer false positives
- **Gaussian Filtered**: Improved temporal consistency
- **Improved Recall**: Better detection of repetitions
## 🔧 Configuration
### Dataset Building
Edit `build_dataset.py` to modify:
- Window size (default: 30 frames)
- Gaussian augmentation parameters
- No-exercise ratio
### Model Training
Edit `src/training/trainer.py` to modify:
- Model architecture (filters, layers, dropout)
- Loss function parameters
- Training hyperparameters
### Demo Settings
Edit `src/demo/demo.py` to modify:
- Model path
- Output format
- Visualization settings
## 📄 License
This project is licensed under the MIT License - see the LICENSE file for details.