- Rust 99.6%
- Shell 0.4%
|
Peter Gerber
226b65a1eb
Fixes:
warning: using `chunks_exact` with a constant chunk size
--> src/hash.rs:226:63
|
226 | ..._mut().zip(hex.as_bytes().chunks_exact(2)) {
| ^^^^^^^^^^^^^^^
|
help: consider using `as_chunks::<2>()` instead
--> src/hash.rs:226:63
|
226 | ..._mut().zip(hex.as_bytes().chunks_exact(2)) {
| ^^^^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#chunks_exact_to_as_chunks
= note: `#[warn(clippy::chunks_exact_to_as_chunks)]` on by default
|
||
|---|---|---|
| src | Clippy: slice::prefer as_chunks() over slice::chunks_exact() | |
| test_utils | rustfmt: reformat using imports_granularity=Item | |
| tests | rustfmt: reformat using imports_granularity=Item | |
| .gitignore | Implement handling objects in file system | |
| .gitlab-ci.yml | CI: use correct host for publishing package | |
| Cargo.toml | Update DEB maintainer | |
| clippy.toml | Ban various hard-to-read methods | |
| CODEOWNERS | Add CODEOWNERS file | |
| deny.toml | Deny: allow CDLA-Permissive-2.0 license | |
| LICENSE | Initial commit | |
| minio-wrapper | CI: increase Minio concurrent request limit to 1000 | |
| README.md | README: update link to documentation | |
Tocco S3
Tool to manage S3 and S3 buckups.
Documentation at https://toccoag.gitlab.io/devops/tocco-s3.
Testing
It's easiest to run the tests against Minio, a simple S3 implentation, running locally. A helper script is available which runs Minio in Docker and allows running tests against it.
Run tests:
./minio-wrapper cargo test
Run tests, including those needing a Cloudscale API key:
export TOCCO_S3_TEST_CLOUDSCALE_API_TOKEN=...
./minio-wrapper cargo test --features tests_with_secrets
Requires a read/write key available here.
Run tests against Cloudscale's S3:
-
Create user
tco s3 create-user --force s3-test -
Run tests using printed key:
export S3_ENDPOINT=https://objects.rma.cloudscale.ch export S3_KEY_ID=... export S3_SECRET_KEY=... cargo test -
Update access_key / secret_key in
~/.s3cfg. -
List buckets
s3cmd ls -
Remove any remaining buckets
s3cmd rb --recursive s3://... -
Remove user
tco s3 remove-user --force s3-test
Run benchmarks:
./minio-wrapper cargo +nightly bench --features nightly
Benchmarks create archives in /tmp with large numbers of objects. Depending
on your setup, you may not have enough space there or wish to test against
another file system. Use TMPDIR environment variable to change directory.
Debugging the State DB
The DB is a simple SQLite DB that can be opened with sqlite3:
$ sqlite3 state.sqlite
SQLite version 3.40.1 2022年12月28日 14:03:47
Enter ".help" for usage hints.
sqlite>
Use .schema or .fullschema to see DB schema:
sqlite> .schema
CREATE TABLE key_value (
...
) STRICT;
...
Table object stores the hash and last-seen timestamp of every object. For
reasons of efficiency, they are stored in binary and seconds since Unix Epoch,
respectively. Use hex() and datetime() to get readable output:
sqlite> SELECT lower(hex(hash)), datetime(last_seen, 'unixepoch') FROM object LIMIT 3;
8f5ba9389cef8dc174e97c4e3a2ddc758a4bf8d66f26334dcc148d7f654940b3|2025年01月15日 19:04:23
459d0ac2cba72d24f85bf9638539cb26ffe4e014db833d62a38e6796bcfc1113|2025年01月15日 19:04:23
2b16aeb5a2866e7e8807ba1de0f5c5cced3044c7f598f8976a15324bcd2515b2|2025年01月15日 19:04:23