The problem was reported here and here and it hasn't been fixed since 2018!

When you set a custom x-axis range, double clicking/reset axis/autoscale actions reset the custom range. The expected behaviour is to respect the custom ranges.

The current workaround is a function that creates a Div (with Graph/Figure) directly in the layout that can then be updated later via callbacks.