perf-prof is a comprehensive Linux system-level analysis tool for long-term performance monitoring with low overhead, broad compatibility, and high reliability.
perf-prof is a user-space performance profiling tool built on top of libperf, libtraceevent, and libbpf. It provides real-time analysis without writing event data to disk, processing everything in memory and discarding it immediately after use.
- In-Memory Processing: Events are processed in memory and discarded immediately - no persistent storage overhead
- Broad Compatibility: Works with older Linux kernels (requires perf_event support)
- User-Space Implementation: Safe execution with rapid iteration capability
- Modular Architecture: 30+ specialized profilers for different analysis scenarios
- Low Overhead: Kernel-level filtering reduces data transfer to user space
- Real-Time Analysis: Process events as they occur with immediate feedback
# Install required dependencies yum install -y xz-devel elfutils-libelf-devel # Optional: Install eBPF dependencies yum install -y llvm bpftool
# Clone the repository git clone https://github.com/OpenCloudOS/perf-prof.git cd perf-prof # Build the project make # Verbose build make V=1 # Clean build artifacts make clean
# Using CROSS_COMPILE make CROSS_COMPILE=aarch64-linux-gnu- # Using LLVM make LLVM=1
# List all profilers ./perf-prof -h # List all tracepoint events ./perf-prof list
# Profile CPU usage at 997Hz with call graph ./perf-prof profile -F 997 -g # Generate flame graph ./perf-prof profile -F 997 -g --flame-graph cpu.folded # Profile user-space only on specific CPU ./perf-prof profile -F 997 -C 0-3 --exclude-kernel --than 30
# Detect kernel memory leaks ./perf-prof kmemleak --alloc "kmem:kmalloc//ptr=ptr/size=bytes_alloc/stack/" \ --free "kmem:kfree//ptr=ptr/" --order -m 128 -g
# Monitor task states (R, S, D, T, I)
./perf-prof task-state -i 1000# Trace specific events ./perf-prof trace -e sched:sched_wakeup,sched:sched_switch -i 1000 # Trace with filtering ./perf-prof trace -e "sched:sched_wakeup/prio<10/" -i 1000
perf-prof provides 30+ specialized profilers organized by category:
- profile - CPU performance sampling analysis
- oncpu - Monitor processes running on CPU
- kmemleak - Memory leak detection
- kmemprof - Memory allocation profiling
- page-faults - Page fault tracking
- task-state - Process state monitoring (R, S, D, T, I)
- rundelay - Scheduling run delay analysis
- sched-migrate - Process migration monitoring
- blktrace - Block device I/O tracking
- kvm-exit - KVM exit latency analysis
- kvmmmu - KVM MMU mapping observation
- hwstat - Hardware state monitoring (cycles, IPC)
- llcstat - Last-level cache monitoring
- tlbstat - dTLB monitoring
- ldlat-loads - Intel load latency counting
- ldlat-stores - Intel store instruction counting
- split-lock - x86 split lock detection
- hrtimer - High-resolution conditional sampling
- irq-off - Interrupt disabled detection
- watchdog - Hard/soft lock detection
- sql - SQL aggregation analysis with SQLite
- top - Key-value statistical analysis
- multi-trace - Multi-event relationship analysis
- syscalls - System call latency analysis
- python - Custom Python script event processing
- trace - Event tracking
- list - List tracepoint events
- expr - Expression-based test tool
- usdt - User Statically Defined Tracing
- breakpoint - Kernel/user space hardware breakpoints
- kcore - Read kernel memory
- misc - Miscellaneous tracking
perf-prof follows a three-layer event selection specification:
# List all events ./perf-prof list # Filter by category ./perf-prof list | grep -E "^(sched:|kmem:|timer:|irq:)"
# View event fields ./perf-prof trace -e sched:sched_wakeup help # Multiple events ./perf-prof trace -e sched:sched_wakeup,sched:sched_switch help
EVENT: sys:name[/filter/ATTR/ATTR/.../]
kprobe:func[/filter/ATTR/ATTR/.../]
kretprobe:func[/filter/ATTR/ATTR/.../]
uprobe:func@"file"[/filter/ATTR/ATTR/.../]
uretprobe:func@"file"[/filter/ATTR/ATTR/.../]
# Numeric comparison ./perf-prof trace -e "sched:sched_wakeup/pid>1000/" ./perf-prof trace -e "sched:sched_wakeup/prio<10/" # String matching ./perf-prof trace -e 'sched:sched_wakeup/comm=="java"/' ./perf-prof trace -e 'sched:sched_wakeup/comm~"pyth*"/' # Logical combinations ./perf-prof trace -e "sched:sched_wakeup/pid>1000 && prio<10/"
stack # Enable call stack alias=str # Event alias max-stack=int # Max stack depth key=EXPR # Event key for correlation top-by=EXPR # Sort field comm=EXPR # Process name display ptr=EXPR # Pointer field size=EXPR # Size field num=EXPR # Number distribution field
# profiler help with examples ./perf-prof trace -h ./perf-prof task-state -h # Event help with field information ./perf-prof trace -e sched:sched_wakeup help ./perf-prof kmemleak --alloc kmem:kmalloc --free kmem:kfree help
- Main Options Reference - Complete command-line options
- profile - CPU performance analysis
- task-state - Process state monitoring
- multi-trace - Multi-event analysis
- sql - SQL aggregation analysis
- top - Key-value statistics
- kmemleak - Memory leak detection
- kvm-exit - KVM exit analysis
- blktrace - Block device I/O tracking
- trace - Event tracing
- python - Python script event processing
- Event Filtering - Trace event filter syntax
- Expressions - Expression language reference
- README - 中文版 - Chinese README
# Run all tests cd tests pytest # Run specific test file pytest test_profile.py # Run with custom runtime and memory leak check pytest --runtime=20 --memleak-check=2000
perf-prof/
├── *.c # Core profiler modules (30+ profilers)
├── lib/ # Base libraries (libperf, libtraceevent, libbpf)
├── arch/ # Architecture-specific code
├── bpf-skel/ # BPF skeleton programs
├── filter/ # Event filters (BPF, tracepoint, PMU)
├── sqlite/ # SQLite amalgamation source code and extension modules
├── include/ # Included header files
├── tests/ # Test suite
├── docs/ # Documentation
└── skills/ # AI-assisted analysis skill packages
Monitoring Framework:
monitor.c/h- Core frameworktep.c/h- Trace event parsertrace_helpers.c/h- Trace event utilitiesstack_helpers.c/h- Stack traversal and symbol resolution
Profiling Units:
- Each profiler is an independent
.cfile - Registered via
PROFILER_REGISTER()macro - Supports
init,deinit,interval,read,samplecallbacks
Event Source → Filters → Ring Buffer → Sort → Profiler → Output
- Create source file
new_profiler.c - Implement
profilerstructure with required callbacks - Define
name,desc,argc,option - Register with
PROFILER_REGISTER() - Add to
Buildfile:perf-prof-y += new_profiler.o - Add test in
tests/directory
We welcome contributions! Please follow these guidelines:
- Fork the repository
- Create a feature branch
- Write tests for new functionality
- Ensure all tests pass
- Submit a pull request
This project is licensed under GPLv2. See LICENSE for details.
- Follow Linux kernel coding style
- Write clear, maintainable code
- Include tests for new features
- Document public APIs and interfaces
Built with components from the Linux kernel:
- libperf
- libtraceevent
- libbpf
- Additional utility libraries