Add template silverfish.service as a user service #166

this says /usr/local/bin but the [Service] below says /usr/bin/silverfish

Can we maybe do something a bit fancier with maybe https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#StartLimitIntervalSec=interval &co, so the service stop restarting if it fails repeatedly in a time window, then pauses trying to restart for a while, then resumes trying to restart? I don't know what kind of stuff systemd can do here, but I think at least limiting restarts a bit would be good to do so folks don't get into error restart loops.

This was something systemctl security-analyze complained about, but I wasn't sure what an appropriate root would be in combination with DynamicUser so I doubt this was the best choice.

RootDirectory chroots inyo that directory, so it should probanly be /.

WorkingDirectory=~ sets it to the DynamicUser's home (/var/lib/silverfish I believe).

That's what I'd set it to

i agree with @jalil that RootDirectory should probably be /, i ran into some issues trying to run this as written

@jalil, I'm unable to set WorkingDirectory=~ due to dynamic user; this is what gets logged:

silverfish.service: WorkingDirectory=~ is not allowed under DynamicUser=yes. Refusing.

Note that the service doesn't actually run right now; it attempts to start then complains about the keygen step.

I kinda wonder how we should handle this because those keys are sensitive and I don’t know how regular sensitive things are set these days outside of virtual hosts with a Secrets feature that sets env vars at runtime. Does systemd have anything for this?

System credentials look like they would be a good fit here.

There are missing $ in those variables

And yes, systemd credentials are the perfect thing for this, you can decrypt a file and pass its value as an env var, or as a path to a decrypted temp file. You can also store the secret in plain text but make it only readable by root and systemd will make it available to the service anyways.

and as mentioned in my other comment, things that can be environment variables can just be set as environment variables and you don't need to pass them as CLI args.

@jalil you can do @include <some path> in the silverfish configs. Maybe we can pass --config <decrypted file> and that file can then @include "/etc/path/to/actual/config"?

Yup, that can be done

These settings will prevent the service from restarting more than 3 times in 120 seconds; I chose these numbers arbitrarily and am open to suggestions on better ones.

I could not find a way to recover from a rate limited restart within systemd, my understanding is this might need to be done manually but I am new to this depth of systemd stuff.

Looks fine to me then

looks good to me, behaved as expected when i tested it.

Note this isn't used or passed in yet.

silverfish does a thing where environment variables can be passed in instead of command line arguments. The conversion is automatic so to pass in —foo-bar you would set SILVERFISH_FOO_BAR instead. They would get processed the same.

the %D specifier is new enough (introduced in systemd v256, released may 2024) that my debian bookworm machine (still in full support til mid-2026, and LTS into 2028) can't parse it. could you please remove %D from these to improve portability?

the other specifiers seem to parse ok on that machine, which is running systemd 252.

this expands to /var/lib/themes, could we change it to %S/silverfish/themes?

This is probably because StateDirectory=silverfish isn't set

When I try to add StateDirectory=silverfish, I get the following namespace related failures:

silverfish.service: Failed to set up mount namespacing: /var/lib/private/silverfish: No such file or directory
silverfish.service: Failed at step NAMESPACE spawning /usr/local/bin/silverfish: No such file or directory

This is the only directory I've been setting up, I'm not sure if they depend on each other or if it's incompatible with another setting?

This is my forgejo runner's service:

[Unit]
After=network-online.target
Description=Gitea Actions Runner
Wants=network-online.target
[Service]
Environment="HOME=/var/lib/gitea-runner/native"
Environment="LOCALE_ARCHIVE=/nix/store/9vmxdn26pggj3glazmb68vhcf4smflvz-glibc-locales-2.40-66/lib/locale/locale-archive"
Environment="PATH=[REDACTED (too long)]"
Environment="TZDIR=/nix/store/xaa75rd44q62nc9mrbvym9d1m6gy0fj8-tzdata-2025b/share/zoneinfo"
DynamicUser=true
EnvironmentFile=/run/secrets/services/forgejo-runner/token
ExecStart=/nix/store/3dsj66ykgdx8m9hrp44wy2zdn6shfi3p-forgejo-runner-11.3.1/bin/act_runner daemon --config /nix/store/2bd7b0gqaka25j5f839gpcfl06bh75na-config.yaml
ExecStartPre=/nix/store/fvz2kxp6difjx3yrbjc648hqhhrvsq9h-gitea-register-runner-native
Restart=on-failure
RestartSec=2
StateDirectory=gitea-runner
User=gitea-runner
WorkingDirectory=-/var/lib/gitea-runner/native
[Install]
WantedBy=multi-user.target

oh yeah should probably want network-online.target yeah?

@zkat maybe I misunderstood your previous comments, but this was the only way I could pass the environment variables to silverfish successfully.

Other things I tried:

1. Passing only the env vars:

ExecStart=/usr/local/bin/silverfish serve $SILVERFISH_CONFIG_DIR $SILVERFISH_TMP_PATH $SILVERFISH_UPLOADS_PATH $SILVERFISH_THEMES_PATH $SILVERFISH_DATABASE_PATH

Resulted in error: unexpected argument '/config' found; they were not recognized as the flag args.
2. Adjusting the env vars to better match the flag values, eg SILVERFISH_UPLOADS_PATH becomes SILVERFISH_UPLOADS-PATH: resulted in
Invalid environment assignment, ignoring: SILVERFISH_UPLOADS-PATH=/uploads

You don't need to pass them to the CLI, it picks them up from the Environment

Yeah just don’t pass in extra arguments at all. As long as they’re set in the env you’re good

My misunderstanding, thank you!

Maybe this is a silly question, but for working with the config file and credentials, where do I put the encrypted configuration file so that exists ahead of time for the service?

I would have expected to be able to put a config directory in /usr/local based on what I have here, but that does not work.

This I have no idea about. I didn’t even know until this PR that systemd supported this kind of thing!!

I use sops-nix which decrypts secrets to /run/secrets/ (which is a symlink to /run/secrets.d/$GENERATION so you can update the secrets without overwriting the files).

This is used for example:

• Configuring a forgejo runner: EnvironmentFile=/run/secrets/services/forgejo-runner/token
• Configuring harmonia (a nix cache): LoadCredential=sign-key-0:/run/secrets/services/harmonia/key

I don't know how you'd do this in general, but you could nudge people create a /secrets folder (chmod 700 /secrets) and load the encrypted config from there.

I didn't include the systemctl enable silverfish.service and systemctl start silverfish.service steps here, let me know if I should to be completely exhaustive!

I had to drop RootDirectory as a setting to get the service running. Otherwise I'd get No Such File or Directory errors when trying to use StateDirectory properly but I'm not 100% on why that's the case.

https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#LoadCredential=ID:PATH provides a good overview of this; I chose /etc/silverfish somewhat arbitrarily.

1. I'm still passing env vars here because I could not get the service working without passing them directly unfortunately 😕 I'm not sure why.
2. I decided to encrypt the entire config and pass it in directly for this initial pass at a service file because it worked with my understanding of what LoadCredentialsEncrypted does. I like Kat's suggestion better but it would require knowing things about CREDENTIALS_DIRECTORY that seems to be managed by systemd at runtime.

it looks like we also need to install a theme? unless i'm misunderstanding something about first-time bringup. (the service crashed due to no index.html, for me)

@cjdesno the default theme should be baked into the binary (via the DefaultTheme struct). It uses rust-embed to bake it in, which I know works differently between debug and release builds by default... maybe that's what could've caused the error you were seeing?

anyway, for this PR, I feel like the systemd service should assume that the Silverfish binary itself will provide and/or configure the default theme

I agree with Kyle.
Ideally these steps will be short lived and removed once there's a complete pipeline for release/installation, since they're pretty technical as it is. I don't want to get more in the weeds or prescriptive than I have to with them.

--db-path should be --database-path

Also the database path is defined later in the file only as a template path, so it's challenging to get this step done. i had to know that %S expands to /var/lib, and then run the service and let it fail to create the dir with correct permissions, and then i was able to run this step (but again by already knowing that %S means /var/lib on my machine). Maybe we can add another note here to help with that, or something?

For sure, I'll just hardcode it for now. %S could mean something different for user managed services but since this is going under system, I think we're safe to keep it simple.

if instead of SILVERFISH_UPLOADS_PATH you set SF_CONFIG_UPLOADS_PATH, and SF_CONFIG_DATABASE_PATH, etc, then you don't need to pass the command-line args in the exec line because silverfish will detect the envs automatically.

Thanks! I was trying SILVERFISH_... before; this explains why it wasn't working 😅

These need to be SF_CONFIG_UPLOADS_PATH, SF_CONFIG_DATABASE_PATH, etc (you need to add CONFIG after SF in all of them)

What I get for not double checking!
Thanks!

there's actually a --save flag for keygen that will let you combine steps 1 and 2 here.

Do we want

After=network-online.target
Wants=network-online.target

here too?

(削除) A bit of a technicallity, but we may not want network-online.target since we don't need to be online for silverfish to run, we can still accept connections from the LAN (and NetworkManager has known issues that are resolved by not checking that it is online when activating network-online.target https://wiki.archlinux.org/title/NetworkManager#NetworkManager-wait-online). (削除ここまで)

(削除) I think the correct target is network.target, since we need to be able to access the network, but not necessarily be online. (削除ここまで)

network-online.target seems to be correct, I just have bad personal experiences with it (as a user of a linux PC, not as a server admin)

sounds good, then