rama-sail-graph

CI

Status: Public Preview — This project is functional and tested but not yet production-hardened. APIs may change. Feedback and contributions welcome.

A Rama-backed RDF quad store with SPARQL query evaluation and HTTP endpoint.

rama-sail-graph integrates the Rama distributed data processing framework with the RDF4J SAIL API, enabling distributed RDF quad storage with SPARQL query support.

• Quad Storage: Four index PStates (SPOC, POSC, OSPC, CSPO) for efficient lookups across any access pattern
• SPARQL Query Engine: Server-side query execution via Rama topologies — joins, filters, aggregates, ORDER BY, OPTIONAL, UNION, BIND, VALUES, ASK
• RDF-Star / SPARQL 1.2 Triple Terms: Store and query triple terms (<< s p o >>) as subjects and objects — enables provenance, confidence, and temporal annotations
• Query Optimization: Cardinality-based join ordering, self-join optimization, colocated subject joins, batch property lookups, materialized type views
• Statistics: Per-predicate and global cardinality tracking for adaptive query planning
• Physical Deletes: Idempotent add/delete with physical index removal
• SPARQL HTTP Endpoint: W3C SPARQL Protocol compatible, works with tools like gdotv
• Observability: Prometheus metrics (/metrics), health checks (/health), query latency histograms, connection tracking
• Namespace Management: Full prefix/IRI namespace support

• Java 17+
• Leiningen
• Rama 1.8.0 (see Rama installation)

This project is open source under the Apache License 2.0.

You can use, modify, and ship this code in commercial products, subject to Apache 2.0 notice requirements. However, this project depends on Rama, which has its own platform licensing terms. Per Red Planet Labs, Rama includes an embedded free license for clusters up to two Supervisor nodes; larger clusters require a separate paid license.

There is no published Maven/Clojars artifact yet — consume the library directly from GitHub as a git dependency.

deps.edn (git dependency):

{:deps {io.github.Shtanglitza/rama-sail-graph
 {:git/sha "<full-40-char-commit-sha>"}}}

Pin :git/sha to a commit on main. Add a matching :git/tag once release tags are published.

Leiningen: Leiningen has no native git-dependency support. Clone the repo and install it into your local Maven cache:

git clone https://github.com/Shtanglitza/rama-sail-graph.git
cd rama-sail-graph
lein install

Then add the locally installed coordinate to project.clj:

[rama-sail-graph "0.2.0-SNAPSHOT"]

Note: Rama 1.8.0 is required at runtime. Add the Red Planet Labs Maven repository to resolve the Rama dependency.

# Check compilation (~30s)
lein check
# Run unit tests (~5s, no Rama cluster needed)
lein test :only rama-sail.sail.optimizer-test rama-sail.sail.adapter-test
# Run full test suite (~2-5min, starts in-process Rama cluster)
lein test
# Run RDF4J SAIL compliance tests only
lein test :only rama-sail.sail.compliance-test
# Run RDF-star / triple term integration tests
lein test :only rama-sail.sail.rdf-star-test
# Run W3C SPARQL 1.2 eval-triple-terms conformance suite
lein test :only rama-sail.sail.w3c.sparql12-test
# Run W3C SPARQL 1.1 property path conformance suite
lein test :only rama-sail.sail.w3c.sparql11-property-path-test

# Start with in-process cluster (dev mode). Read-only by default.
lein run -m rama-sail.server.sparql -- --port 7200 --mode ipc
# Start with external Rama cluster
lein run -m rama-sail.server.sparql -- --port 7200 --mode cluster --host localhost --rama-port 1973
# Enable SPARQL UPDATE (off by default), require a Bearer token, and restrict CORS
lein run -m rama-sail.server.sparql -- --port 7200 --mode ipc \
 --allow-updates --auth-token s3cret --cors-origin https://app.example
# Query
curl -G http://localhost:7200/sparql --data-urlencode "query=SELECT * WHERE { ?s ?p ?o } LIMIT 10"
# Update (only when the server was started with --allow-updates + --auth-token)
curl -X POST http://localhost:7200/sparql \
 -H "Authorization: Bearer s3cret" \
 -H "Content-Type: application/sparql-update" \
 --data 'INSERT DATA { <http://ex/a> <http://ex/b> <http://ex/c> }'

Server flags:

Note: The endpoint is read-only by default — a form-encoded POST is a CORS "simple request", so leaving UPDATE open would let any web page issue DELETE WHERE { ?s ?p ?o }. Enable writes explicitly with --allow-updates, and gate them with --auth-token and a non-wildcard --cors-origin. The endpoint still provides no rate limiting or transport security (TLS); front it with a reverse proxy for internet-facing deployments.

(require '[com.rpl.rama :as rama])
(require '[rama-sail.core :as core])
;; In-process cluster (dev/test)
(with-open [ipc (com.rpl.rama.test/create-ipc)]
 (com.rpl.rama.test/launch-module! ipc core/RdfStorageModule {:tasks 4 :threads 2})
 (let [module-name (rama/get-module-name core/RdfStorageModule)
 depot (rama/foreign-depot ipc module-name "*triple-depot")]
 ;; Add triples
 (rama/foreign-append! depot [:add ["<alice>" "<knows>" "<bob>" "::rama-internal::default-graph"]])
 ;; Query
 (rama/foreign-invoke-query
 (rama/foreign-query ipc module-name "find-triples")
 "<alice>" nil nil nil)))

(require '[rama-sail.sail.adapter :as sail])
(import '[org.eclipse.rdf4j.repository.sail SailRepository])
(let [rsail (sail/create-rama-sail ipc module-name)
 repo (SailRepository. rsail)]
 (.init repo)
 (with-open [conn (.getConnection repo)]
 ;; Use standard RDF4J API for SPARQL queries
 (let [query (.prepareTupleQuery conn "SELECT * WHERE { ?s ?p ?o } LIMIT 10")]
 (with-open [result (.evaluate query)]
 (doseq [bs (iterator-seq result)]
 (println bs))))))

Triple terms allow statements about statements — attach provenance, confidence scores, or temporal metadata directly to triples:

;; Store a triple with a confidence annotation
(rama/foreign-append! depot [:add ["<< <alice> <knows> <bob> >>" "<confidence>" "\"0.95\"^^<http://www.w3.org/2001/XMLSchema#double>" "::rama-internal::default-graph"]])

# Find all annotations on a specific triple
SELECT ?pred ?val WHERE {
 << <http://ex/alice> <http://ex/knows> <http://ex/bob> >> ?pred ?val .
}
# Extract components from annotated triples
SELECT ?s ?p ?o ?confidence WHERE {
 << ?s ?p ?o >> <http://ex/confidence> ?confidence .
 FILTER(?confidence > 0.8)
}
# Provenance tracking via SPARQL UPDATE
INSERT { << ?s ?p ?o >> <http://ex/source> ?g }
WHERE { GRAPH ?g { ?s ?p ?o } }

Triple terms are stored as first-class values in all four quad indexes, using << <s> <p> <o> >> canonical serialization. Nested triple terms are supported to arbitrary depth.

Rama module with quad-based storage using four complementary indices. The module is defined in core.clj and assembled from composable building blocks in module/indexer.clj (PStates, ETL), module/namespaces.clj (prefix management), and module/queries.clj (query topologies):

Deletion uses physical index removal — deleted quads are removed from all four indices immediately. Adds and deletes are idempotent (set semantics for indices, $$quad-tx-time tracking for statistics correctness).

Triple terms (<< s p o >>) are handled across the stack:

RamaSail passes 40 of 40 tests in the RDF4J RDFStoreTest compliance suite (rdf4j-sail-testsuite 5.2.0), which validates:

• Statement CRUD (add, remove, get, has) with pattern matching and wildcards
• Value round-trips (IRIs, BNodes, typed literals, decimals, timezones, long URIs/literals)
• Named graph / context management (add to context, list contexts, clear)
• Transaction semantics (commit, rollback, isolation)
• Namespace management (get, set, remove, clear)
• Duplicate handling and statement counting
• Concurrent add-while-querying
• Dual connections — two separate SailConnection instances operating concurrently on the same store, each maintaining independent transaction state
• BNode reuse — blank node identity preserved when the same BNode is used across multiple statements within a connection

SPARQL operators handled server-side via Rama topologies:

Some unsupported operators are automatically handled by falling back to RDF4J's DefaultEvaluationStrategy, which evaluates locally via getStatements. This ensures correct results but without distributed execution benefits. SERVICE is a special case and is currently rejected explicitly rather than evaluated through fallback.

REPL verification showed that basic nested SELECT subqueries can compile into Rama plans and run server-side, so subquery support is at least partial and should not be described as fallback-only.

The project includes a vendored copy of the W3C eval-triple-terms test suite (41 tests) in test/resources/w3c/sparql12/eval-triple-terms/.

Quarantine breakdown:

The quarantined tests will become unblocked as RDF4J ships RDF 1.2 parser support (no released version exists as of April 2026) and as rdf:reifies semantics are implemented.

The project includes a vendored copy of the W3C property-path test suite (33 tests) in test/resources/w3c/sparql11/property-path/.

Prometheus metrics are exposed at /metrics when running the SPARQL endpoint:

• ramasail_query_latency_seconds — query execution latency histogram
• ramasail_queries_total — query count by status (success, timeout, error, fallback)
• ramasail_active_connections — active SAIL connections gauge
• ramasail_query_result_size — result size summary (p50, p90, p99)
• ramasail_transactions_total — transaction count by status
• ramasail_transaction_ops_total — triple operations by type (add, del)

A /health endpoint is also available for liveness checks.

The project includes a Berlin SPARQL Benchmark (BSBM)-compatible benchmark suite with synthetic data generation. Implements 9 of 12 original BSBM SELECT queries (Q1–Q5, Q7, Q8, Q10, Q11) plus 4 custom join-focused queries. Q6 (INSERT), Q9 (DESCRIBE), and Q12 (CONSTRUCT) are omitted as they test write/export operations rather than read performance. Benchmarks can run against an in-process cluster (IPC) for development or a real Rama cluster for production-like numbers.

Datasets must be generated before running benchmarks:

lein with-profile +dev run -m clojure.main -e "
(require '[rama-sail.bench.bsbm.infra.data-generator :refer [generate-standard-datasets!]])
(generate-standard-datasets!)
"

This creates three datasets in test/resources/bsbm/:

lein test :only rama-sail.bench.bsbm.bsbm-bench/test-bsbm-small

cd $RAMA_HOME
# Start ZooKeeper, conductor, and supervisor (each in background)
./rama devZookeeper &
sleep 5
./rama conductor &
sleep 10
./rama supervisor &
sleep 10
# Verify cluster is ready
./rama conductorReady # should print: true
./rama numSupervisors # should print: 1

# From the project directory
lein uberjar
# From the Rama installation directory
cd $RAMA_HOME
./rama deploy --action launch \
 --jar /path/to/rama-sail-graph/target/rama-sail-graph-0.2.0-SNAPSHOT-standalone.jar \
 --module "rama-sail.core/RdfStorageModule" \
 --tasks 4 --threads 2 --workers 1
# Verify module is running
./rama moduleStatus "rama-sail.core/RdfStorageModule"

# From the project directory
# Load data + run all queries including joins
lein run -m rama-sail.bench.cluster.cluster-bench :host localhost :load true :joins true
# Re-run without reloading data
lein run -m rama-sail.bench.cluster.cluster-bench :host localhost :joins true
# Joins only
lein run -m rama-sail.bench.cluster.cluster-bench :host localhost :joins-only true
# Custom dataset and iteration count
lein run -m rama-sail.bench.cluster.cluster-bench \
 :host localhost \
 :dataset test/resources/bsbm/dataset_1000.nt \
 :load true \
 :iterations 100 \
 :joins true

The suite includes 9 BSBM queries and 4 join-focused queries:

These benchmark results were recorded on this development machine:

• Host: MacBook Pro
• Chip: Apple M5
• CPU: 10 cores (4 performance, 6 efficiency)
• Memory: 24 GB
• OS: macOS 26.4
• Runtime/config: Rama 1.8.0, single-node cluster, 6 GB heap, 4 tasks, 2 threads
• Benchmark settings: 50 iterations, 5 warmup

With query optimizations (LIMIT pushdown to BGPs, adaptive filter selectivity, AND decomposition for filter pushdown), p50 latencies scale near-linearly across 6K to 59K triples. Most queries show 1.0x for a 10x data increase.

The following optimizations are applied automatically during query planning:

• LIMIT pushdown: Propagates LIMIT into BGPs, joins, and self-joins for early termination — truncates quads before binding construction. Pushes through transparent operators (PROJECT, BIND), with safety multipliers for FILTER (4x) and DISTINCT (2x). Correctly blocks at ORDER BY for BGP targets (ORDER BY needs all rows to sort)
• Adaptive filter selectivity: Estimates filter selectivity by operator type (equality=0.05, range=0.3, inequality=0.8) instead of a flat 0.3, improving join ordering decisions
• AND decomposition: Splits compound AND filters into independent conjuncts that can be pushed separately to each join side
• Selinger-style join ordering: Dynamic programming optimizer for multi-way join chains using predicate statistics
• Self-join optimization: Detects same-predicate joins with inequality filters and uses group-and-pair generation instead of Cartesian product
• Colocated subject joins: Exploits subject hash partitioning for partition-local joins
• Batch property enrichment: Replaces N individual property lookups with single batch query
• Materialized type views: Pre-computed indices for rdf:type pattern lookups

• Query fallback: Some unsupported SPARQL operators fall back to RDF4J's local evaluation strategy, which may not scale for large datasets; SERVICE is explicitly unsupported
• RDF 1.2 partial: Triple terms work via RDF-star (<< s p o >>), but RDF 1.2 <<( s p o )>> syntax, ~ :reifier syntax, {| |} annotations, and rdf:reifies semantics are not yet supported. 5/41 W3C SPARQL 1.2 conformance tests pass; 36 are quarantined
• Triple term decomposition is client-side: Extracting subject/predicate/object from triple terms uses string parsing on the SAIL client, not distributed Rama topologies. Constant triple-term lookups are O(1) distributed, but variable decomposition and component filtering are post-fetch
• Query timeout: Cancellation is best-effort; Rama cluster queries may continue after client timeout
• SPARQL endpoint: No authentication, rate limiting, or TLS — intended for development use
• No CI: Tests are run locally; CI configuration is not yet included

Copyright (c) 2025-2026 Shtanglitza. Licensed under the Apache License 2.0. See LICENSE, NOTICE, and COPYRIGHT.md.