PULP – Precompressed Upstream Layer Pipeline

Copyright François Gauthier - Superwired-Labs

License Platform MSVC C11 Build Check Version

Stop feeding raw logs to expensive storage/network.

PULP is a native C library for Windows x64 that compresses structured data at the source, inside your own process, before it ever hits disk or network. Lossless semantic deduplication paired with LZ4 shrinks data ×ばつ; IP anonymisation is applied inline; and a zero-allocation on the producer hot path sustains 20M+ logs/sec with fully deterministic memory.

Link the DLL, call one function per log, and let compressed binary shards accumulate. A companion CLI (PulpReader) decodes them back to text whenever you need.

Key numbers (6‐core Ryzen 5 Pro 8640HS, NVMe SSD):

• Up to 21.7M logs/second sustained throughput (High-Performance mode, 6 threads, ~105 MB RAM)
• 8.5M logs/second at ~16 MB RAM (Eco mode, single thread, ultra-low footprint)
• ×ばつ lossless end-to-end compression on realistic structured logs (preprocessing + LZ4)
• 0 logs lost, 0 backpressure under heavy adversarial stress

PULP's benchmark throughput measures the complete end-to-end pipeline operating on realistic workloads:

Summary: While standard tools measure how fast they can discard or process logs in memory, PULP processes, sanitises, compresses, and writes full-fidelity logs to physical disk in a single pass.

Verify the numbers yourself: LogProducer ships with embedded test datasets. Drop the DLL into the folder, run it on your hardware, results in under 5 minutes.

Live benhmark on Youtube:
PULP Live Benchmark Demo

Most ultra-high-performance telemetry tools (Vector, eBPF-based agents, Fluent Bit) are designed Linux-first. On Windows Server, which powers critical IIS infrastructure, high-throughput enterprise .NET apps, and low-latency C++ services, developers are often left with two undesirable options:

1. Heavy managed logging frameworks that cause Garbage Collection (GC) pauses and high CPU overhead under heavy traffic spikes.
2. Cross-platform ports that wrap Linux paradigms, losing performance through abstraction layers.

PULP was built to solve this exact problem.

Rather than using generic cross-platform wrappers, PULP leverages bare-metal Windows primitives:

• Direct Thread-Local Storage (TLS) for zero-contention hot path ingestion.
• Native Windows Thread Pool API for asynchronous compression tasks.

By committing fully to the Win32 ecosystem, PULP delivers line-rate ingestion with zero GC impact and a deterministic memory footprint under any load.

• Blazing fast – zero‐allocation producer hot path, per‐thread TLS caches, zero-eviction temporal cache, custom IP masking.
• On‐the‐fly compression – lossless semantic precompression paired with LZ4 reduces I/O volume before writing to disk.
• Deterministic memory footprint – no memory spikes during activity surges; the architecture absorbs load seamlessly.
• Dual‐mode IP anonymisation – configurable masking for IPv4 and IPv6, including support for CIDR, zones, and compressed addresses.
• Inter-thread atomic numeration – easily reconstruct the absolute sequence order of millions of events/logs.
• Automatic file rotation – smooth handle swapping and asynchronous deferred close eliminate write stalls.
• Resilient disk handling – automatic fallback to a backup path on disk full, access denied, or path not found.
• Resilient file format – Each log file is a sequence of independent blocks (batches), making the storage resilient to corruption.
• Rich telemetry – JSON statistics endpoint (cache hit ratios, compression ratios, throughput, backpressure, and more).
• Dual licensing – AGPLv3 for open‐source use, or a commercial license for closed‐source products.

Measured on a Lenovo ThinkPad P14s Gen 5 (Ryzen 5 Pro 8640HS, 96 GB RAM, NVMe SSD, Windows 11).

Workload: 1 000 unique URLs ×ばつ 5 000 unique IPs, random distribution.

High-Performance Mode (6 caller threads):

• 250,000,000 logs in 11.52 seconds
• 21.71M logs/second sustained
• ~105MB RAM footprint
• LZ4-only ratio: ×ばつ (67% of original)
• End-to-end ratio: ×ばつ (preprocessing + LZ4: 19.4% of the original)
• 0 logs lost, 0 backpressure events

Economy Mode (single caller thread):

• 250,000,000 logs in 29.52 seconds
• 8.47M logs/second
• ~16MB RAM footprint
• LZ4-only ratio: ×ばつ (67% of original)
• End-to-end ratio: ×ばつ (preprocessing + LZ4: 21.5% of the original)
• 0 logs lost, 0 backpressure events

Performance depends on the hardware, the number of threads, the data entropy, cardinality, and the parameters passed to the initialisation function.

Ran the benchmark? Open an issue with your configuration (CPU, dataset used, PulpInit parameters,...) + throughput !

• Windows 10 / 11 or Windows Server 2016+ (x64).
• Visual Studio 2022 (solution provided).

A complete working example is provided in the LogProducer project. All values are 'per-thread'. All the caller threads use the same init values. The DLL is intended to be called by a pool of (or one unique) long lived thread.

#include "pulp.h"
int main() {
 // One‐time initialisation
 uint8_t rc = PulpInit(
 "C:\\Logs", // primary log folder
 "D:\\BackupLogs", // backup folder (optional, can be "")
 "C:\\Errors", // error log folder
 1, // enable/disable atomic inter‐thread sequence IDs
 ANON_IP_2, // enable/disable anonymization on last 4 octets (IPv4) / 4 hextets (IPv6)
 1, // enable/disable strip URL query parameters (if URLs are to be logged)
 128, // file rotation every 128 batches (must be power of 2)
 COMPRESSION_BALANCED, // LZ4 compression level
 BATCH_8MB, // 8MB active buffer or 262 144 in-flight logs
 DICT_256K // 256k slots cache
 );
 if (rc != RTN_OK) {
 // Handle initialisation error
 return 1;
 }
 // Write a log/evt entry from any thread
 uint16_t res = PulpWrite(
 12, // Numeric identifier of the operation (HTTP method, ICMP type, DNS opcode, etc.), user defined
 "https://www.resource/admin/overview.jpeg", // Generic resource reference (URL, domain name, ICMP message, etc.) 563 char MAX, see API header for truncation politic.
 40, // Length of the resource string without the terminating char.
 200, // Generic response or error code (HTTP status, DNS RCODE, ICMP code/type, etc.)
 "172.21.22.23", // Target address (IPv4/IPv6, hostname, DNS server, etc.). Do NOT try IP_ANON on non-IP endpoints.
 12, // Length of the endpoint string
 4520, // Duration in milliseconds (0–65535)
 8, // Data size bucket (0 = 1–5 KB, 1 = 5–10 KB, etc.), user defined
 123, // Free bitmask (bit 0 = encrypted, bit 1 = protocol version, bit 2 = fragmented, etc.), user defined
 1780008801123456 // High-resolution timestamp in microseconds (UTC ISO-8601) ("2026-07-29T14:53:21.123456Z" as per the example)
 );
 
 uint8_t backpressure = res & 0xFF;
 uint8_t error = res >> 8;
 // Graceful shutdown
 PulpShutdown();
 return 0;
}

The LogProducer project lets you verify PULP's performance on your own hardware in minutes.

1. Clone the repository and open PULP.sln in Visual Studio 2022.
2. Build the solution in Release | x64 configuration.
3. Open LogProducer.vcxproj in a new Visual Studio instance and build it in Release | x64
4. Copy Pulp.dll and Pulp.lib from PULP/x64/Release/ into LogProducer/x64/Release/.
5. Customize the test (paths, number of threads, PULP options) in LogProducer.cpp – it's straightforward.
6. Click Run.

Test datasets are included. Results (throughput, compression ratio, memory footprint, lost logs) are printed directly to the console.

The public API is declared in pulp.h and exported by pulp.dll.

For detailed parameter descriptions, see the API header pulp.h.

How the dictionary achieves high compression with few slots: Even with only 16K slots, PULP can handle millions of unique URLs/IPs over time. The dictionary persists across batches; once an entry is cached, all future occurrences reference the same index. You only need enough slots to hold your current hot working set (frequently repeated values), not every unique string ever seen. Undersized dictionaries perform very well, while oversized dictionaries can be slightly detrimental. The L1/L2/L3 hash cascade ensures fast lookup even under cache pressure. See the API header pulp.h for more details.

Number of unique URL/IP values the per‐thread cache can hold.

Controls the flush threshold and the maximum in-flight logs (lost on a hard crash, power outage, etc. The write queue can also hold batches waiting to be processed).

Flush is triggered by buffer pressure only (no timer). Call PulpShutdown() to flush remaining logs on shutdown.

Since compression isn't the bottleneck, COMPRESSION_BALANCED is generally recommended.

⚠ Only use IP anonymisation on actual IP addresses. Do not apply it to hostnames or other non‐IP strings.

PULP is designed to be glued into your existing infrastructure. As a C native Library it can be interfaced with virtually everything. It writes compressed .bin files locally; you decide how to move and process them.

Hereafter are some implementations ideas, which would require adding only basic extensions to the project (2 & 3) or are ready to use out of the box (1 & 4).

Just link PULP, call PulpWrite(), and the compressed shards accumulate in your log folder. Use PulpReader later to decode them.

All servers write to \\nas\logs\. A scheduled task on the central machine runs PulpReader against new files and pipes the output to your observability platform.

# Example PowerShell script, with json format and output filter extensions (not included in the sources, but easily implementable)
Get-ChildItem \\nas\logs\ -Filter *.bin | ForEach-Object {
 PulpReader.exe --input $_.FullName --format json --filter "http_code>=400" |
 Invoke-RestMethod -Uri "https://api.datadog.com/v1/input" -Method Post
}

PulpReader.exe --input C:\Logs\shard_12345.bin --format json --filter "http_code>=500" |
 curl -X POST https://api.datadog.com/v1/input -H "Content-Type: application/json" -d @-

PulpReader.exe E:\Archives2円025-01-01.bin E:\Archives2円025-01-01.txt

Logging infrastructure and cloud providers charge heavily for network egress and storage capacity. By compressing logs directly at the source, PULP reduces the data footprint by ×ばつ to ×ばつ before it leaves your application process, depending on log structure and entropy.

The figures below are illustrative, based on a conservative ×ばつ compression factor. You can measure savings on your own workloads using the included LogProducer benchmark project.

*Savings reflect bandwidth, local/cold storage, and self-hosted cluster capacity (Elastic, Loki).
A lifetime commercial license typically pays for itself within weeks or days on high-volume nodes.

PulpWrite() → TLS context → URL/IP cache → active buffer
 ↑ ↓ (buffer full)
 | PulpFlush()
 | ↓
 | BuildDictionaryInMemory()
 | ↓
 | CompressionTask (thread pool)
 | ↓
 +——— WritePool_Enqueue() → WriteThread → WriteFile()
 ↓
 RotationThread (file creation, handle swap)

Every thread owns its own cache and active buffer – no lock contention on the hot path. The write pool is a multi‐producer / multi‐consumer queue synchronised with slim reader‐writer locks and condition variables.

For a deep dive into the design choices and performance measurements that drive PULP's architecture, see this companion technical article: Processing 250M logs in 11.5s on a laptop with on-the-fly ×ばつ compression

PULP uses a block-based binary format engineered for high-throughput streaming writes and total thread-level isolation. A .bin archive is a continuous sequence of independent, self-contained compressed blocks (batches).

+-----------------------------------------------------------------------+
| LOG FILE |
| +-------------------+ +-------------------+ +-------------------+ |
| | Block 0 (Batch) | | Block 1 (Batch) | ... | Block N (Batch) | |
| +-------------------+ +-------------------+ +-------------------+ |
+-----------------------------------------------------------------------+

Each block on disk consists of a lightweight header followed immediately by the payload:

Once decompressed, the payload contains two primary sections delimited by 64-bit canary markers (DICT_BEGIN and DICT_END):

The first portion of the decompressed payload consists of a contiguous array of fixed-size 32‐byte SerializedEntry structures. Each entry captures a single log event with zero memory alignment padding gaps:

To guarantee stream integrity, prevent out‐of‐bounds reading, and detect partial block writes (e.g., after an unexpected crash), PULP uses two 64‐bit magic constants:

• DICT_BEGIN (0xDEADBEEFCAFEBABE): Marks the exact boundary between the SerializedEntry record array and the dictionary section.
• DICT_END (0xBEEFBABEDEADCAFE): Marks the end of the decompressed block payload.

Following DICT_BEGIN, the dictionary is split into two independent parts:

• URL Dictionary: starts with a uint32_t count of entries, then that many URL entries.
• IP Dictionary: starts with a uint32_t count of entries, then that many IP entries.

Each individual entry has the following binary layout:

Parser Behavior (pulpReader.c): The reader processes the array up to DICT_BEGIN to extract all log events, builds a sparse lookup table from the dictionary entries up to DICT_END, and reconstructs full text log entries on the fly with zero dynamic memory allocation overhead on the producer hot path.

PulpGetStats() returns a JSON snapshot (caller must free with PulpFreeStats()):

{
 "cache_hit_resource_L1" : 249937807
 "cache_hit_resource_L1_%" : 99.98
 "cache_hit_resource_L2" : 62193
 "cache_hit_resource_L2_%" : 0.02
 "cache_hit_resource_L3" : 0
 "cache_hit_resource_L3_%" : 0.00
 "cache_hit_endpoint_L1" : 250000000
 "cache_hit_endpoint_L1_%" : 100.00
 "cache_hit_endpoint_L2" : 0
 "cache_hit_endpoint_L2_%" : 0.00
 "cache_hit_endpoint_L3" : 0
 "cache_hit_endpoint_L3_%" : 0.00
 "resource_cache_probes_total" : 206998729
 "endpoint_cache_probes_total" : 4944735
 "resource_cache_insert_to_step_ratio" : 1.21
 "endpoint_cache_insert_to_step_ratio" : 50.56
 "resource_cache_probes_depth_max" : 32
 "endpoint_cache_probes_depth_max" : 0
 "resource_cache_fullprobescan_total" : 0
 "endpoint_cache_fullprobescan_total" : 0
 "log_processed_total" : 250000000
 "batch_flushed_total" : 15
 "batch_compressed_total" : 15
 "batch_written_total" : 15
 "writer_waitfile_max" : 0
 "backpressure_count" : 0
 "compression_lz4_ratio_avg" : 0.71
 "compression_failure_total" : 0
 "lost_logs_total" : 0
 "log_rotation" : 1
 "log_refused_total" : 0
 "rotation_resync_total" : 0
 "throughput millions logs/s" : 26.49 
}

PulpGetStats() introduces a ~2 s measurement window. Call it at most every few seconds with a dedicated thread.

• Windows x64 only. The code uses Windows‐specific APIs (TLS, SRW locks, thread pools). A Linux port is planned.
• No transactional durability for in‐flight data. Logs in the active buffer are lost on a hard crash. Loss is bounded to BATCH_SIZE / 32 entries per thread + the write queue if any (it's usually empty due to the speed of the write pool). Batches already flushed to disk are safe.
• Flush triggered by throughput only – no background timer. Low‐traffic applications should use the smallest batch size (15 600 logs) probably with a unique long lived thread.
• Fixed log schema. The PulpWrite() signature accepts a fixed set of fields (operation, resource, endpoint, status, timing, flags). These cover HTTP, DNS, ICMP and similar tabular log formats. Fully arbitrary structured fields would require extending the source.

Contributors must adhere to the Contributor Licence Agreement (CLA.md) located at the root of the project. Take time to read it before contributing. The document also states the project's philosophy regarding the dual-license.

PULP was created by François Gauthier – Founder & Software Architect, Superwired-labs.

PULP is dual‐licensed:

• Open Source – GNU Affero General Public License v3.0 (AGPLv3)
• Commercial – a proprietary license for closed‐source products

Full license texts are in the LICENSES/ folder.
For commercial conditions & pricing, contact fgauthier [at] superwired-labs [dot] com.

PULP is open-source software maintained by Superwired-Labs. If PULP saves you storage costs or accelerates your infrastructure, consider supporting its ongoing development:

• ⭐ Star this repo if it helps you; it's the simplest way to support the project.
• Open Source Sponsorship: Support the project via GitHub Sponsors or PAYPAL to help fund roadmap features and maintenance.
• Commercial Licensing & Consulting: For custom binary layouts, dedicated support, or integration services into enterprise architectures, contact us at fgauthier[at]superwired-labs[dot]com or read our Commercial Licensing Guide.

See THIRD_PARTY.md for details.