Implement content addressable file storage #1

Open
opened 2026年06月17日 21:12:05 +02:00 by jackolantern · 0 comments
Collaborator
Copy link

Proposed filesystem structure:

- firefly/
 - topic/
 - <topic name>
 - <share name>.<share hash>.<share.ext>
 - content/
 - <hash>

Uploads and downloads live together harmoniously in the /firefly/content directory.

Consider a directory containing two files /foo/hello.png and /foo/goodbye.png. If a user shares this directory then two symlinks /firefly/content/<hash of /foo/hello.png> and /firefly/content/<hash of /foo/goodbye.png> will be created in the obvious way. A link of this sort is called a "content link"

If a file is added to the foo directory a content link is created if and only if the hash of the new file is not the name of a file in the content directory. If a file is removed from foo then the corresponding content link is also removed. If a file in foo is replaced by a file with a different hash then it is equivalent to removing the original file and adding the new file.

If the user fetches a share from the network then the file is stored in /firefly/content with the appropriate hash, and a symlink to the content is created in the relevant topic directory.


Create function that takes a root and paths from that root, and symlinks every path (contained in that root) to a new location, recreating the folder structure of that root


Application state should be persisted somewhere so that on startup the hashes of the files in the shared directories do not need to be re-computed.
The state should include - for each shared file - the path to the file, the created time and last modified time of the file, the siize of the file, and the hash of the file. On startup the create time, modified time, and file size from the restored state are compared to the files on disk. If they are different then the hash is recomputed and the content links / share links are updated as appropriate

Proposed filesystem structure: ``` - firefly/ - topic/ - <topic name> - <share name>.<share hash>.<share.ext> - content/ - <hash> ``` Uploads and downloads live together harmoniously in the `/firefly/content` directory. Consider a directory containing two files `/foo/hello.png` and `/foo/goodbye.png`. If a user shares this directory then two symlinks `/firefly/content/<hash of /foo/hello.png>` and `/firefly/content/<hash of /foo/goodbye.png>` will be created in the obvious way. A link of this sort is called a "content link" If a file is added to the `foo` directory a content link is created if and only if the hash of the new file is not the name of a file in the content directory. If a file is removed from `foo` then the corresponding content link is also removed. If a file in `foo` is replaced by a file with a different hash then it is equivalent to removing the original file and adding the new file. If the user fetches a share from the network then the file is stored in `/firefly/content` with the appropriate hash, and a symlink to the content is created in the relevant topic directory. --- Create function that takes a root and paths from that root, and symlinks every path (contained in that root) to a new location, recreating the folder structure of that root --- Application state should be persisted somewhere so that on startup the hashes of the files in the shared directories do not need to be re-computed. The state should include - for each shared file - the path to the file, the created time and last modified time of the file, the siize of the file, and the hash of the file. On startup the create time, modified time, and file size from the restored state are compared to the files on disk. If they are different then the hash is recomputed and the content links / share links are updated as appropriate
jackolantern changed title from (削除) test (削除ここまで) to Implement file storage 2026年06月17日 21:18:34 +02:00
eaon changed title from (削除) Implement file storage (削除ここまで) to Implement content addressable file storage 2026年06月17日 21:26:09 +02:00
Sign in to join this conversation.
No Branch/Tag specified
No results found.
No results found.
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
eaon/firefly#1
Reference in a new issue
eaon/firefly
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?