feat(runtime): Implement streaming execution output (fixes #1813) - #1814

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/vscode

Inputs:

• PR number: 1814
• Commit SHA: 55336697342c3e7988e9121fccf6b2709246c98d

Checks will be approved automatically on success.

Hi @tinodj — first, a genuine apology for the long silence here. This PR (and #1813) sat since early December without a proper review; the team was stretched thin and PRs didn't get the attention they deserved. That's on us. We now have more bandwidth to work through the backlog and invest in improvements, and I really appreciate your patience — and the fact that you filed a clear issue and took a run at fixing it.

On the change itself: the intent is spot-on, but I don't think this approach can deliver streaming output, and I want to explain why so we can steer it toward something that does help.

The limitation is in the execution API, not the client. This run path goes through the legacy 1.2/commands execution API via ExecutionContext.execute → Command. The onStatusUpdate callback you're wiring up does fire on every status poll (see Command.response() in src/sdk-extensions/Command.ts), so that part works — but the callback only ever receives result.results populated once the command reaches a terminal state (Finished/Error/Cancelled). While status === "Running", results is empty, so onStatusUpdate has nothing to emit and the output still arrives all at once at the end.

You can see this in the SDK model too: the only output-shaping fields on Results are pos and truncated, which handle large results being paginated/size-capped — there's no field carrying incremental stdout produced over time. So no client-side callback wiring can surface partial output the API doesn't return. (Real streaming needs API/server-side support, which is a bigger effort on our side.)

A couple of smaller notes:

• The status poll uses exponential backoff capped at ~10s, so even if partial results were available, the cadence wouldn't be truly "real-time."
• (result: any) / (result.results as any).data drop the typing the surrounding code relies on.

Good news — there's a way to get live output today: if you run the file via the Databricks Connect run/debug mode instead of "Upload and Run File", execution happens locally through Spark Connect and streams stdout to the integrated terminal in real time. That's the path we'd recommend for interactive, long-running scripts right now, and it's the one we're investing in making frictionless.

Where I think we can still get value on this path — #1813 actually describes two problems, and the second one is fixable client-side:

1. Output doesn't appear live — blocked on API-level streaming support for the Upload-and-Run path; can't be solved from the extension today (use the Databricks Connect mode above in the meantime).
2. "If the script fails with an exception, none of the printed output appears" — this is a separate, real bug. On failure the API returns only the traceback and the earlier print output is dropped; we can capture stdout in resources/python/bootstrap.py and surface it alongside the error.

Also worth noting: this is a duplicate of the older #1325, which reports the same thing — we'll consolidate those.