RecordedOutput - Job Status Updates Throughout Job #1064

I think I'm mostly ok with it as a feature. The main hesitation I have with any feature that modifies job rows during execution is the potential for deadlocks or other bad interactions if transactions aren't closed properly or just if they're blocking system operations like completions, cancellation, etc from happening.

Use at your own risk, I suppose?

Yeah, so this is the concern for sure.

We also are only updating the data every 30 seconds, but the concern is still valid.....think of it more like a heartbeat more than anything else with very small amounts of data. For example, we log how many bytes has been processed to this point and loop iterations.

I think we could do something where we check to make sure the job is not completed before updating too, but we would need to wrap the logic in a lock of some sort, but I'm not 100% sure how postgres handles this and if we can put a row lock intentionally and what the repercussions of that would be. I believe below is roughly the right idea.

BEGIN;
-- Lock the row (skips if not found)
SELECT * FROM river_job
WHERE id = 123
FOR UPDATE;
-- Now check and update (this assumes you fetched the row above)
UPDATE river_job
SET metadata = 'data goes here'
WHERE id = 123
AND state != 'completed';
COMMIT;

Hey guys,

Just coming back to this. Is it worth me putting together a PR for this? Is there an appetite for this for others?

Took a rough stab at what this might look like in #1098.