Spotify Builds External Index to Enable Low Latency Point Queries on its Data Lake

Follow us on

Spotify has introduced Random Access Parquet (RAP), a storage architecture that enables low-latency point queries directly against data stored in its data lake, allowing online services and AI applications to retrieve individual records without replicating datasets into operational databases. RAP adds an external indexing layer over Apache Parquet files, enabling interactive lookups while continuing to use the same datasets for analytics, machine learning, and online serving.

Spotify explained that modern data lakes have become the central repository for analytical and AI workloads, but retrieving individual records remains inefficient because distributed query engines such as Trino and BigQuery are optimized for analytical scans rather than key-based lookups. Although cloud object stores such as Google Cloud Storage now provide millisecond access latency, query planning, metadata traversal, and file discovery can add significant overhead for point queries. Spotify noted that it stores petabytes of online data in Bigtable while exabytes reside in its Google Cloud Storage-based data lake, making large-scale replication into serving databases increasingly costly.

RAP addresses this challenge by introducing an external index that maps lookup keys, such as user IDs, directly to Parquet files and row locations. Instead of scanning thousands of files, a query resolves the key through the index before issuing a targeted ranged read against object storage. As new data is written into Apache Iceberg tables, an index builder generates append-only index fragments without modifying immutable Parquet files. Spotify said the approach allows the same datasets to support analytical processing, machine learning pipelines, notebooks, AI agents, and latency-sensitive online applications without maintaining duplicate storage systems.

Spotify's announcement follows broader efforts to extend open data lake technologies beyond analytical processing. Google Cloud recently described an Apache Iceberg based lakehouse architecture for AI applications that similarly seeks to reduce data duplication while enabling operational access to data. Unlike that approach, RAP introduces a dedicated external indexing layer optimized for point lookups while remaining compatible with existing Parquet files and Iceberg tables.

The architecture also generated discussion within the data engineering community. Andrew Lamb highlighted RAP as an example of extending open data formats for interactive workloads. In a separate LinkedIn discussion, Vikas Singh argued that improvements in cloud object storage performance have shifted more of the latency associated with point queries toward query planning and metadata access, an area that RAP is designed to reduce through precomputed indexes.

Spotify also described several storage layout optimizations that reduce point query latency. These include sorting data by lookup key to reduce the number of files accessed, grouping related records together, interleaving value columns so multiple attributes can be retrieved through a single contiguous read, and using covering indexes that can satisfy some queries without reading Parquet files. According to Spotify, these techniques trade modest increases in file or index size for fewer storage operations, allowing some point queries to be served through a single ranged read of only a few kilobytes.

Interleaved value column layout enables related values from multiple columns (Source: Spotify Blog Post)

Spotify also supports secondary indexes, enabling efficient querying across multiple lookup dimensions, such as buyer ID or seller ID, without rewriting Parquet files. Hash-based indexes support exact lookups, while sorted indexes enable range queries. Spotify said secondary indexes are managed at the serving layer, allowing new access paths without changing data pipelines while continuing to use the same Parquet datasets for both analytical scans and interactive point lookups. Storage layout techniques such as Z ordering and Hilbert curves can further improve data locality for secondary lookup dimensions.

Related Topics:

• Development
• Architecture & Design
• AI, ML & Data Engineering
• Low Latency
• Data
• Distributed Systems
• Data Lake
• Software Engineering
• Google Cloud
• AI Architecture
• Big Data
• IndexedDB
• Apache
• Apache Iceberg
• Query

We protect your privacy.