-
Notifications
You must be signed in to change notification settings - Fork 1.3k
feature: saving files as blob in database #1326
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
DanielAuerX
commented
Apr 29, 2025
some clean up tasks etc are needed. just wanted to see if there is interest in a feature like this.
LinkinStars
commented
Apr 30, 2025
Although it is possible to save files in the database, we do not recommend it. Since the number of files can be large, saving them in the database not only causes performance problems with queries, it can also be very troublesome when it comes time to back up database data. You should be able to find some bad things about storing files in databases in the web. All in all we still don't recommend users to store files in the database. Storing them in the file system or a third-party cloud provider using a plugin is good.
mgerhardy
commented
Apr 30, 2025
+1 for this feature.
It is quite troublesome to keep backups and restores in sync if you have multiple infrastructure components. It's also a cost aspect for us.
+1 for this PR
I would also like to have that ability as we are using Apache Answers also.
At the moment we have to align the backups of the filesystem AND the database backup as well.
Both of them must match to each other.
If we use the ability to store the files as blobs within the database, we would only have to care about a single backup component.
We could use the toggle that @DanielAuerX already provided to explicitly enable the blob feature, by default we simply set FILE_STORAGE_MODE to 'fs'.
What do you say @LinkinStars ? :)
Cheers,
Steven
LinkinStars
commented
Apr 30, 2025
Thanks for the feedback, it's important for us to hear more varied suggestions. Of course, there are advantages and disadvantages to putting files in the database. I understand that this can be turned off by default by setting ENV to keep the existing feature unaffected. Other than that, we'll mainly be looking at maintenance costs.
Let's hear more ideas from others before we decide on this feature. @sy-records @kumfo
861cff6 to
88a3d5c
Compare
DanielAuerX
commented
May 1, 2025
When I was looking into how files are deleted, I noticed some issues with the file_record table. This is actually related to the problem of keeping the file system and the db in sync.
- Not all uploaded files (branding and avatar) are added to the file_record, leading to the file_record being incomplete
- If a question (or answer) is deleted, it is not being checked if the text body contained a reference to a file/image. Therefore, the question is deleted, but the file_record entry is not being updated and the image remains in the file system.
- The file record_entity is not registered in init_data.go, therefore the table is not being created in the first place and no files are deleted at all. This also causes an error log whenever the clean up cron job is running.
I might missed or misunderstood some things, but I am quite sure there is something wrong here
LinkinStars
commented
May 3, 2025
@DanielAuerX Thanks for the feedback.
- Yes, for these files were not tried to record before, it may be better to add them.
- Cleanup is based on a time set by the user, not immediately after deletion. FYI:
answer/internal/base/cron/cron.go
Line 81 in 3f1ed50
if s.serviceConfig.CleanUpUploads { - Very important feedback. Would you be interested in submitting another separate PR to help fix this?
DanielAuerX
commented
May 3, 2025
Yes sure, I will prepare a PR
sy-records
commented
May 9, 2025
Thank you for your contribution.
I also think it would add extra maintenance costs, after all, most use local storage or 3rd party cloud storage. If this can exist as a plugin, that's good.
- [x] create file_record table
- [x] avatar and branding files are added to file_record
- [x] branding files are being deleted
- [x] avatar files are being deleted
- [x] reload latest avatar (frontend) after backend state is being
updated
problems addressed in the pr:
- clean up job fails, because it cannot access file_record table
- avatar and branding files are not added to the file_record table
- avatar and branding files are never deleted
- after an avatar is being updated/deleted, the old file is still being
requested due to browser caching. This is causing error logs ("no such
file or directory") in the backend.
cf. conversation in [pr
1326](#1326)
---------
Co-authored-by: broccoli <lekker.broccoli@gmail.com>
Co-authored-by: LinkinStars <linkinstar@foxmail.com>
mgerhardy
commented
May 27, 2025
For me it's not about the extra maintenance costs - but the extra infrastructure costs that we would have to pay when we have to use local storage (persistent volumes in k8s) and a postgres database. It's a lot easier to maintain and reduces production costs if we only have to use the (already existing) postgres database. Adding S3 or even local storage is not that easy in cluster environments.
+ files (e.g. branding pictues, attachments) can be saved as a blob in a database. The feature can be turned on and off (feature toggle) with the env var FILE_STORAGE_MODE + a new controller with the endpoints /file/branding etc. had to be created since the files are statically linked when fs is used. + the files plus metadata are stored in the table 'file'
88a3d5c to
20534ba
Compare
setting storage mode once in a central place, so the logic "if env_var == 'db' then ..." does not have to be repeated
mingcheng
commented
Jun 28, 2025
Thank you for your contribution. I think the test code is necessary because this is a BIG PR.
DanielAuerX
commented
Jun 28, 2025
@mingcheng I can add tests, no problem. However, if I understood @LinkinStars and @sy-records responses correctly, this pr (feature "saving files as blob" within the answer core) was not gonna be merged?
If its going to be merged, I will continue and finish the pr.
joyqi
commented
Jul 17, 2025
Thanks for the contribution!
I personally think this idea might be better implemented as a plugin under the Storage type. However, it seems that the current Storage interface lacks a way to directly handle HTTP requests. Perhaps we could consider adding such an interface, which would open up more possibilities for ideas like this one.
files (e.g. branding pictues, attachments) can be saved as a blob in a database. The feature can be turned on and off (feature toggle) with the env var FILE_STORAGE_MODE
a new controller with the endpoints /file/branding etc. had to be created since the files are statically linked when fs is used.
the files plus metadata are stored in the table 'file'