Increase ENDLESS_TASK_TIMEOUT to 72h? #2202

zyphlar commented 2025年11月07日 20:16:52 +01:00

Comment

Hi there! We at CoMaps have our own runners for beefy long-running Actions, and benefit greatly from having our CI/CD ecosystem all in one place here at Codeberg, but our map generation job can run for days (currently 5 days but ideally 3 or less once we optimize a problematic straggler.) I've edited the runner settings so the jobs actually keep going behind the scenes, the only issue is that Codeberg gives up on the job and marks it as failed... but doesn't actually force-stop the job with the runner. Which is a good thing, I like my jobs finishing behind the scenes! But it's also unfortunate, since it looks failed and stopped and causes a little panic.

Hopefully it's not too many resources to hold open a job at the coordinator level, right? And the primary resource issue is the runners themselves, who should actually be doing the bulk of any cancelling?

### Comment Hi there! We at CoMaps have our own runners for beefy long-running Actions, and benefit greatly from having our CI/CD ecosystem all in one place here at Codeberg, but our map generation job can run for days (currently 5 days but ideally 3 or less once we optimize a problematic straggler.) I've edited the runner settings so the jobs actually keep going behind the scenes, the only issue is that Codeberg gives up on the job and marks it as failed... but doesn't actually force-stop the job with the runner. Which is a good thing, I like my jobs finishing behind the scenes! But it's also unfortunate, since it looks failed and stopped and causes a little panic. Hopefully it's not too many resources to hold open a job at the coordinator level, right? And the primary resource issue is the runners themselves, who should actually be doing the bulk of any cancelling?

Gusted commented 2025年11月08日 22:11:25 +01:00

The current value is 3 hours and I'm not exactly sure who is benefiting from this (as the job is still continuing by the runner). I don't feel strongly about changing this.

@zyphlar wrote in #2202 (comment):

Hopefully it's not too many resources to hold open a job at the coordinator level, right? And the primary resource issue is the runners themselves, who should actually be doing the bulk of any cancelling?

You might be better suited for this, does the job still continuing outputting after it being marked as failed? All what stopping the task does is updating the status in the database.

CC @fnetX, do you have anything for changing this value?

The current value is 3 hours and I'm not exactly sure who is benefiting from this (as the job is still continuing by the runner). I don't feel strongly about changing this. @zyphlar wrote in https://codeberg.org/Codeberg/Community/issues/2202#issue-2645660: > Hopefully it's not too many resources to hold open a job at the coordinator level, right? And the primary resource issue is the runners themselves, who should actually be doing the bulk of any cancelling? You might be better suited for this, does the job still continuing outputting after it being marked as failed? All what stopping the task does is updating the status in the database. CC @fnetX, do you have anything for changing this value?

fnetX commented 2025年11月09日 00:13:01 +01:00

No, I don't have strong objections. I have increased them in Codeberg-Infrastructure/build-deploy-forgejo@747d8b7c22 already. While I see the value in cancelling jobs that take too long, I think this is better done on the runner level to give project admins the control. I don't see a good reason for enforcing this on Codeberg's level.

No, I don't have strong objections. I have increased them in https://codeberg.org/Codeberg-Infrastructure/build-deploy-forgejo/commit/747d8b7c22db487a001e42642ba31c02a74747fb already. While I see the value in cancelling jobs that take too long, I think this is better done on the runner level to give project admins the control. I don't see a good reason for enforcing this on Codeberg's level.

fnetX commented 2025年11月09日 00:16:07 +01:00

Oh, and re-reading my commit message: @zyphlar It is actually relevant if your jobs produce no output, because then ZOMBIE_TASK_TIMEOUT also applies. If they produce debug output, we should be fine.

@Gusted I'll have to look into the exact behaviour of Actions (or you tell me if you know). I have questions about the behaviour of running jobs with no log output. Ideally, if the runner continously reports them as still running, they should not be canceled. However, I would want that jobs where the runner simply disappears (e.g. because of server shutdowns, network issues, crashes ... and thus cannot mark the job as failed) will run into some sort of timeout, because this is then only our responsibility.

So: Runner actively reports job as running: We shouldn't cancel, not our responsibility.
Runner disappearing and no longer marking the job as running: We need to assume the runner has failed and it becomes our responsibility to consider the job failed after some time.

Oh, and re-reading my commit message: @zyphlar It is actually relevant if your jobs produce no output, because then ZOMBIE_TASK_TIMEOUT also applies. If they produce debug output, we should be fine. @Gusted I'll have to look into the exact behaviour of Actions (or you tell me if you know). I have questions about the behaviour of running jobs with no log output. Ideally, if the runner continously reports them as still running, they should not be canceled. However, I would want that jobs where the runner simply disappears (e.g. because of server shutdowns, network issues, crashes ... and thus cannot mark the job as failed) will run into some sort of timeout, because this is then only our responsibility. So: Runner actively reports job as running: We shouldn't cancel, not our responsibility. Runner disappearing and no longer marking the job as running: We need to assume the runner has failed and it becomes our responsibility to consider the job failed after some time.

zyphlar commented 2025年11月09日 00:26:17 +01:00

Sounds good, thanks! The symptom is codeberg.org/*/actions/* looks like the job has failed mysteriously after 24h and some minutes, even with recent log output, but investigating the Forgejo runner (docker on our server) shows the job actually continuing just fine. Then after that job has finished, it moves on to the next job just fine (with successful codeberg.org output for that subsequent job.)

While we're at it, if CoMaps is using some resource excessively feel free to open an issue and @ me or @pastk -- thanks for everything you do! Making the map generation process public and reproduceable and automated has been a big goal of mine.

Sounds good, thanks! The symptom is `codeberg.org/*/actions/*` looks like the job has failed mysteriously after 24h and some minutes, even with recent log output, but investigating the Forgejo runner (docker on our server) shows the job actually continuing just fine. Then after that job has finished, it moves on to the next job just fine (with successful codeberg.org output for that subsequent job.) While we're at it, if CoMaps is using some resource excessively feel free to open an issue and @ me or @pastk -- thanks for everything you do! Making the map generation process public and reproduceable and automated has been a big goal of mine.

fnetX commented 2025年11月23日 00:09:55 +01:00

Hi! I have finally made the config changes in Codeberg-Infrastructure/build-deploy-forgejo@f16fc1b3d7 and they'll be included in the next deployment, likely in a few days. I'm closing this issue, I suppose you'll notice once the behaviour changed :)

Please let us know once you have optimized the builds for a shorter runtime, so we can ideally lower the threshold again (currently at 5d).

I'm very happy to support CoMaps, and I hope that the last anti-feature flag can be removed from F-Droid some day. If there is anything else that we can do for you (e.g. actions/meta#54), please let us know.

Hi! I have finally made the config changes in https://codeberg.org/Codeberg-Infrastructure/build-deploy-forgejo/commit/f16fc1b3d7e3e8e9b76ff546192bcecc67217499 and they'll be included in the next deployment, likely in a few days. I'm closing this issue, I suppose you'll notice once the behaviour changed :) Please let us know once you have optimized the builds for a shorter runtime, so we can ideally lower the threshold again (currently at 5d). I'm very happy to support CoMaps, and I hope that the last anti-feature flag can be removed from F-Droid some day. If there is anything else that we can do for you (e.g. https://codeberg.org/actions/meta/issues/54), please let us know.

zyphlar commented 2025年11月23日 01:44:45 +01:00

Thanks so much! I think we're at 36-ish hours currently.

Thanks so much! I think we're at 36-ish hours currently.