Skip to content

Navigation Menu

Sign in
Sign up

feat(python-setup): report manual-setup opt-out telemetry - #2170

Open
rugpanov wants to merge 3 commits into
main from
feat/manual-setup-optout-telemetry
Open

feat(python-setup): report manual-setup opt-out telemetry #2170
rugpanov wants to merge 3 commits into
main from
feat/manual-setup-optout-telemetry

Conversation

@rugpanov

@rugpanov rugpanov commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up telemetry deferred from #2158 (manual opt-out for uv-native Python setup). Makes the opt-out measurable — both the deliberate action and the standing prevalence across sessions.

What & why

Today the only signal for a manual opt-out is the generic commandExecution event that fires for the "Use manual setup" button, and opting out by editing databricks.python.environmentSetup directly in settings is invisible. This adds:

  1. python_env.manual_setup.optout — recorded when the opt-out command write succeeds (not on a failed write). Fields:
    • scope: workspace (a folder is open) | global (no-folder fallback).
    • source: error_popup (the E_FETCH "Use manual setup" button) | command_palette.
  2. setupMode on python_env.setup.detected (once per session): uv (uv-native flow), pip (legacy/non-uv — the exact manager stays in primaryManager), or fallback-pip (opted out). The auto|manual setting is recoverable from it (fallback-pipmanual). Derived by resolveSetupMode, kept beside isUvSetupSuitable so "which flow is in effect" has one source of truth.

Categorical data only — no paths, package names, or PII (enforced by the allowlist discipline in telemetry/).

Out of scope

The dashboard panel that visualises these events is a separate follow-up — the events need a released build to reach users first (the same "not released yet" gap the existing reportOffered counters have).

Verification

  • yarn build (tsc) + yarn test:lint clean.
  • yarn test:unit green (1061 passing), including new unit tests for resolveSetupMode, recordManualSetupOptout, emitDetection's setupMode, and the popup passing {source: "error_popup"}.

This pull request and its description were written by Isaac.

rugpanov commented Sep 3, 2026
edited
Loading

Copy link
Copy Markdown
Contributor Author

🤖 Integration tests ❌ 1 of 41 test jobs failed for 10fccf93 (40 passed).
View run

rugpanov commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Integration tests triggered for 0fdde38e — ⏳ running.
View run

rugpanov marked this pull request as ready for review September 3, 2026 13:48

rugpanov commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Integration tests ✅ passed for 98267bc8.
View run

rugpanov and others added 3 commits September 7, 2026 11:12
*Why*
The manual opt-out added in #2158 (databricks.python.environmentSetup:
manual, plus the "Use manual setup" button on the E_FETCH failure) is only
observable today via the generic commandExecution event, and opting out by
editing the setting directly is invisible. We can't measure how often users
fall back to a manually managed environment, or from where.
*What*
- New python_env.manual_setup.optout event, recorded when the opt-out command
 write succeeds, with scope (workspace|global) and source (error_popup|
 command_palette). The E_FETCH popup tags its invocation as error_popup;
 palette invocations read as command_palette.
- Add setupMode (uv | pip | fallback-pip) to the once-per-session
 python_env.setup.detected event so opt-out prevalence is measurable per
 session. Derived in one place (resolveSetupMode, beside isUvSetupSuitable):
 manual => fallback-pip; auto splits by uv-suitability into uv / pip. The
 exact manager stays in primaryManager.
Categorical data only; no paths, package names, or PII.
*Verification*
- yarn build (tsc) and yarn test:lint clean.
- yarn test:unit green (1061 passing). Added unit tests: resolveSetupMode
 cases; recordManualSetupOptout scope/source; emitDetection setupMode for
 uv / pip / fallback-pip; and the popup passing {source: "error_popup"}.
Co-authored-by: Isaac <no-reply@databricks.com>
*Why*
Multi-source review of the opt-out telemetry raised: the event over-counted
when the setting was already manual; the command-handler glue was untested; and
two type names collided / broke the repo's OptOut casing.
*What*
- Record the opt-out only on a genuine auto->manual transition (not on every
 successful write), so repeat invocations while already manual don't inflate
 the count. Extract the handler into optOutOfAutomatedPythonSetup with injected
 deps, covered by unit tests (transition, already-manual, no-folder, write
 failure).
- Rename the reported-flow type PythonEnvSetupMode -> ReportedSetupMode to avoid
 collision with PythonEnvironmentSetupMode / PythonSetupMode.
- Rename Optout -> OptOut in identifiers to match the repo's optOutOfInstallPrompt.
- Clarify the setupMode field comment: emitted per (trigger, project); measure
 prevalence by distinct user, not raw event share.
- Use import type for the now type-only telemetry imports.
*Verification*
- yarn build (tsc) and yarn test:lint clean; yarn test:unit green (1065 passing).
Co-authored-by: Isaac <no-reply@databricks.com>
The helper injects out the VS Code UI globals (window/workspace/commands) but
still imports the ConfigurationTarget enum as a runtime value, so its tests run
under the extension-host harness. Reword the doc-comment that claimed 'without
VS Code globals'.
Co-authored-by: Isaac <no-reply@databricks.com>
rugpanov force-pushed the feat/manual-setup-optout-telemetry branch from 98267bc to 9cf28f7 Compare September 7, 2026 09:15

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/vscode

Inputs:

  • PR number: 2170
  • Commit SHA: 9cf28f767b26a79b7a54ac6e9e255b6a5208c60b

Checks will be approved automatically on success.

rugpanov commented Sep 7, 2026
edited
Loading

Copy link
Copy Markdown
Contributor Author

🤖 Integration tests ✅ passed for 9cf28f76.
View run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /