Skip to content

Navigation Menu

Sign in
Sign up

Repository files navigation

lean-s3 npm badge

A server-side S3 API for the regular user. lean-s3 tries to provide the 80% of S3 that most people use. It is heavily inspired by Bun's S3 API. Requires a supported Node.js version.

Elevator Pitch

import { S3Client } from "lean-s3";
const client = new S3Client({
 // All of these are _required_
 // lean-s3 doesn't guess any of these. See below for common values for most providers
 endpoint: env.S3_ENDPOINT,
 accessKeyId: env.S3_ACCESS_KEY,
 secretAccessKey: env.S3_SECRET_KEY,
 region: "auto",
 bucket: env.S3_BUCKET, // required here, but you can specify a different bucket later
});
const test = client.file("/test.json");
const exists = await test.exists();
console.log("Does it exist?", exists);
if (exists) {
 console.log("Seems so. How big is it?");
 const stat = await test.stat();
 console.log("Object size in bytes:", stat.size);
 console.log("Its contents:");
 console.log(await test.text()); // If it's JSON: `await test.json()`
 // Delete the object:
 // await test.delete();
 // copy object to a different bucket in a different region
 const otherFile = client.file("/new-file.json", {
 bucket: "foo-bucket",
 region: "bar-region",
 });
 await otherFile.write(test);
 // Copy the object to a new key in the same bucket
 await test.copyTo("/test-copy.json");
 // You can also copy to a different bucket
 await test.copyTo("/test.json", {
 destinationBucket: "foo-bucket",
 });
 const firstBytesFile = test.slice(0, 100); // lazy-evaluated slicing
 const firstBytes = await firstBytesFile.bytes(); // evaluated using HTTP range requests
 console.log("First 100 bytes:", firstBytes);
}
console.log("Pre-signing URL for clients:");
const url = test.presign({ method: "PUT" }); // synchronous, no await needed
console.log(url);

Installation

# choose your PM
npm install lean-s3
yarn add lean-s3
pnpm add lean-s3

Why?

@aws-sdk/client-s3 is cumbersome to use and doesn't align well with the current web standards. It is focused on providing a great experience when used in conjunction with other AWS services. This comes at the cost of performance and package size:

$ npm i @aws-sdk/client-s3 @aws-sdk/s3-request-presigner
$ du -sh node_modules
21M	node_modules
# vs
$ npm i lean-s3
$ du -sh node_modules
1,8M	node_modules

lean-s3 is so lean that it is ~1.8MB just to do a couple of HTTP requests BUT...

Due to the scalability, portability and AWS integrations of @aws-sdk/client-s3, pre-signing URLs is async and performs poorly in high-performance scenarios. By taking different trade-offs, lean-s3 can presign URLs much faster. I promise! This is the reason you cannot use lean-s3 in the browser.

lean-s3 is currently about 30x faster than AWS SDK when it comes to pre-signing URLs:

benchmark avg (min ... max) p75 / p99
-------------------------------------------- ----------
@aws-sdk/s3-request-presigner 140.12 μs/iter 143.53 μs
 (96.47 μs ... 674.16 μs) 348.75 μs
 ( 3.10 kb ... 1.98 mb) 226.95 kb
s3mini 34.83 μs/iter 33.58 μs
 (26.91 μs ... 611.92 μs) 81.47 μs
 ( 1.29 kb ... 334.05 kb) 29.82 kb
lean-s3 13.82 μs/iter 14.03 μs
 (12.01 μs ... 382.96 μs) 21.40 μs
 ( 2.41 kb ... 595.80 kb) 5.20 kb
aws4fetch 44.79 μs/iter 43.36 μs
 (33.95 μs ... 972.53 μs) 90.25 μs
 ( 80.00 b ... 741.64 kb) 33.90 kb
minio client 58.11 μs/iter 56.44 μs
 (46.77 μs ... 4.03 ms) 92.10 μs
 ( 1.25 kb ... 437.31 kb) 21.57 kb
summary
 lean-s3
 2.52x faster than s3mini
 3.24x faster than aws4fetch
 4.2x faster than minio client
 10.14x faster than @aws-sdk/s3-request-presigner

Don't trust this benchmark and run it yourself. I am just some random internet guy trying to tell you how much better this s3 client is. For PUT operations, it is ~1.5x faster than @aws-sdk/client-s3. We still work on improving these numbers.

See BENCHMARKS.md for more numbers and how to run it yourself. PRs for improving the benchmarks are welcome!

Why not lean-s3?

Don't use lean-s3 if you

  • need a broader set of S3 operations.
  • need a tight integration into the AWS ecosystem.
  • need browser support.
  • are already using @aws-sdk/client-s3 and don't have any issues with it.
  • are using Bun. Bun ships with a great built-in S3 API.

I need feature X

We try to keep this library small. If you happen to need something that is not supported, maybe using the AWS SDK is an option for you. If you think that it is something that >80% of users of this library will need at some point, feel free to open an issue.

See DESIGN_DECISIONS.md to read about why this library is the way it is.

Supported Operations

Bucket Operations

Object Operations

Example Configurations

Hetzner Object Storage

const client = new S3Client({
 endpoint: "https://fsn1.your-objectstorage.com", // "fsn1" may be different depending on your selected data center
 region: "auto",
 bucket: "<your-bucket-name>",
 accessKeyId: process.env.S3_ACCESS_KEY_ID, // <your-access-key-id>,
 secretAccessKey: process.env.S3_SECRET_KEY, // <your-secret-key>,
});

Cloudflare R2

const client = new S3Client({
 endpoint: "https://<account-id>.r2.cloudflarestorage.com",
 region: "auto",
 bucket: "<your-bucket-name>",
 accessKeyId: process.env.S3_ACCESS_KEY_ID, // <your-access-key-id>,
 secretAccessKey: process.env.S3_SECRET_KEY, // <your-secret-key>,
});

Amazon AWS S3

const client = new S3Client({
 // keep {bucket} and {region} placeholders (they are used internally).
 endpoint: "https://{bucket}.s3.{region}.amazonaws.com",
 region: "<your-region>",
 bucket: "<your-bucket-name>",
 accessKeyId: process.env.S3_ACCESS_KEY_ID, // <your-access-key-id>,
 secretAccessKey: process.env.S3_SECRET_KEY, // <your-secret-key>,
});

Backblaze B2

// Docs: https://www.backblaze.com/apidocs/introduction-to-the-s3-compatible-api
const client = new S3Client({
 // keep {region} placeholders (it is used internally).
 endpoint: "https://s3.{region}.backblazeb2.com",
 region: "<your-region>", // something like "eu-central-003"
 bucket: "<your-bucket-name>",
 accessKeyId: process.env.S3_ACCESS_KEY_ID, // <your-application-key-id>,
 secretAccessKey: process.env.S3_SECRET_KEY, // <your-application-key>,
});

Popular S3 provider missing? Open an issue or file a PR!

Tested On

To ensure compability across various providers and self-hosted services, all tests are run on:

  • Amazon AWS S3
  • Hetzner Object Storage
  • Cloudflare R2
  • Garage
  • Backblaze B2
  • Minio
  • LocalStack
  • RustFS
  • Ceph
  • S3Mock

About

A small and fast S3 client without the clutter.

Topics

Resources

Contributing

Stars

42 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages

AltStyle によって変換されたページ (->オリジナル) /