Skip to content

Navigation Menu

Sign in
Sign up

_remote_debugging: Impossible stack traces (cross calls, orphans, wrong lines) with cache_frames=True #151424

Closed
Labels

Description

Bug report

Bug description:

Basically, when profiling with Tachyon I'm seeing stack traces that are just impossible.

Truth to be told, this is an absolutely pathological script with completely unstable (flutter frames) stack traces. That said, I'd expect them to be rejected (thus increasing the error rate), not corrupted.

As before, I've spotted this issue - albeit not that catastrophically - while observing my vm_remap runs.

Reproduction

def leaf_a(): return sum(range(50))
def leaf_b(): return sum(range(50))
def hot_a(): return leaf_a()
def hot_b(): return leaf_b()
while True:
 hot_a(); hot_b();

Then:

2026年06月12日T20:56:38.335765000+0200 maurycy@gimel /Users/maurycy/work/cpython (main f4f1020?) % sudo ./python.exe -m profiling.sampling run --collapsed -o /tmp/stacks.txt -d 10 repro.py
Captured 10,000 samples in 10.00 seconds
Sample rate: 999.99 samples/sec
Error rate: 16.50
Collapsed stack output written to /tmp/stacks.txt
2026年06月12日T20:56:48.704707000+0200 maurycy@gimel /Users/maurycy/work/cpython (main f4f1020?) % cat /tmp/stacks.txt
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_b:4;repro.py:leaf_a:1 1721
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_b:4;repro.py:leaf_b:2 1436
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_a:3;repro.py:leaf_b:2 1430
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6 1170
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_a:3;repro.py:leaf_a:1 1075
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_a:3 413
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_b:4 396
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:leaf_a:1 262
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:leaf_b:2 207
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_a:-1;repro.py:leaf_b:2 75
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_b:-1;repro.py:leaf_a:1 69
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_b:-1;repro.py:leaf_b:2 41
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_a:-1;repro.py:leaf_a:1 31
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_a:3;repro.py:leaf_b:-1 11
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_b:4;repro.py:leaf_a:-1 9
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:leaf_a:-1 2
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:5;repro.py:hot_a:-1;repro.py:leaf_b:2 1
tid:12100288;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_a:3;repro.py:leaf_a:-1 1
2026年06月12日T20:58:42.867874000+0200 maurycy@gimel /Users/maurycy/work/cpython (main f4f1020?) %

More precisely, out of 8476 successful stacks, 4316 (50.9%) are impossible.

Cross:

tid:12252325;...;repro.py:<module>:6;repro.py:hot_b:4;repro.py:leaf_a:1 1736
tid:12252325;...;repro.py:<module>:6;repro.py:hot_a:3;repro.py:leaf_b:2 1584
tid:12252325;...;repro.py:<module>:6;repro.py:hot_b:-1;repro.py:leaf_a:1 23
tid:12252325;...;repro.py:<module>:6;repro.py:hot_a:-1;repro.py:leaf_b:2 14
tid:12252325;...;repro.py:<module>:6;repro.py:hot_a:3;repro.py:leaf_b:-1 7
tid:12252325;...;repro.py:<module>:6;repro.py:hot_b:4;repro.py:leaf_a:-1 4
tid:12252325;...;repro.py:<module>:5;repro.py:hot_a:3;repro.py:leaf_b:2 2
tid:12252325;...;repro.py:<module>:5;repro.py:hot_b:4;repro.py:leaf_a:1 2

Orphans:

tid:12252325;...;repro.py:<module>:6;repro.py:leaf_b:2 464
tid:12252325;...;repro.py:<module>:6;repro.py:leaf_a:1 462
tid:12252325;...;repro.py:<module>:5;repro.py:leaf_b:2 1
tid:12252325;...;repro.py:<module>:6;repro.py:leaf_a:-1 1

Wrong lines:

tid:12252325;...;repro.py:<module>:6;repro.py:hot_a:-1;repro.py:leaf_a:1 7
tid:12252325;...;repro.py:<module>:6;repro.py:hot_b:-1;repro.py:leaf_b:2 4
tid:12252325;...;repro.py:<module>:5;repro.py:hot_b:4;repro.py:leaf_b:2 4
tid:12252325;...;repro.py:<module>:5;repro.py:hot_a:3;repro.py:leaf_a:1 1

Very important caveat here is that it's just 1 level deep, there are no generators etc.,

Isolation to cache_frames=True

diff --git i/Lib/profiling/sampling/sample.py w/Lib/profiling/sampling/sample.py
index 50ccc57566d..16de26b2dc5 100644
--- i/Lib/profiling/sampling/sample.py
+++ w/Lib/profiling/sampling/sample.py
@@ -110,7 +110,7 @@ def _new_unwinder(self, native, gc, opcodes, skip_non_matching_threads):
 gc=gc,
 opcodes=opcodes,
 skip_non_matching_threads=skip_non_matching_threads,
- cache_frames=True,
+ cache_frames=os.environ.get("PROFILING_CACHE_FRAMES", "1") != "0",
 stats=self.collect_stats,
 **kwargs
 )
2026年06月13日T02:57:07.911742000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (main f4f1020*?) % sudo PROFILING_CACHE_FRAMES=0
./python.exe -m profiling.sampling run --collapsed -o /tmp/stacks.txt -d 10 repro.py
Captured 10,000 samples in 10.00 seconds
Sample rate: 999.99 samples/sec
Error rate: 15.09
Collapsed stack output written to /tmp/stacks.txt
2026年06月13日T02:57:18.021250000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (main f4f1020*?) % cat /tmp/stacks.txt
tid:13147584;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_b:4;repro.py:leaf_b:2 3337
tid:13147584;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_a:3;repro.py:leaf_a:1 3257
tid:13147584;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6 1264
tid:13147584;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_a:3 332
tid:13147584;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_b:4 282
tid:13147584;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_a:3;repro.py:leaf_a:-1 9
tid:13147584;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_b:4;repro.py:leaf_b:-1 5
tid:13147584;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:5 2
tid:13147584;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:leaf_b:2 2
tid:13147584;<frozen runpy>:_run_module_as_main:201;<frozen runpy>:_run_code:87;repro.py:<module>:6;repro.py:hot_b:4;repro.py:leaf_a:1 1

...so now it's only 0,20% impossible stack traces (leaving figuring out which as a trivial excercise to the reader.)

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      AltStyle によって変換されたページ (->オリジナル) /