It's true that moving more logic and state management to the client side often increase page load times - but it also allows you (potentially) to make certain interactions much faster.
When you have real data on the client instead of just a bunch of markup, you can be a lot smarter about how and when you make additional AJAX requests. Optimistic updates can make a huge difference in perceived performance.