Investigate recent performance degradation #129

Sometime in the past couple of months, we went from processing literally 1m reqs/s to barely managing 9k reqs/s on my machine. This is almost certainly database access related but we might just be doing too much access, or we could be caching certain things.

We need to investigate what happened and fix it. Current performance levels are unacceptable.

A clue: Someone on mastodon pointed me to https://htmlpreview.github.io/?https://github.com/diesel-rs/metrics/blob/results/plots/table_bench_trivial_query_sqlite.html, and that might mean we need to choose something other than sqlx for most of our sqlite-related stuff (I still think it's nice to use it for migrations!).

I'm gonna be taking a look at this to see if there's easy issues to resolve.

@pyrox one thing I just noticed, since I have my logs enabled now, is that we're creating and setting session cookies unconditionally, even if someone is logged out. This is... wildly unnecessary.

also: https://docs.rs/tower-sessions/latest/tower_sessions/#layered-caching might be beneficial, alongside not creating those sessions at all for open pages.

We should perhaps use moka in general for various things??

Tried adding a cache to the sessions store with the layered cache provided by tower_sessions.

Before: screenshot-19_14-58-01

After: screenshot-19_15-18-24

Turns out, this cache actually decreases performance, using the plow load testing tool. Therefore, at this moment, I don't believe this is useful. However, I am using purely unauthenticated requests, so maybe it's better with authenticated requests.

idea that we might spin up a separate issue for: "precompress Pages into brotli/gzip", so we have them pre-saved in the database in their compressed formats, and serve those directly. We compress them whenever data gets saved.

this whole thing is cursed

I'm thinking doing regular benchmarks might be helpful, maybe automated on PR to monitor performance changes if possible.
Additionally we should be able to check out past revisions and see how performance has shifted over time

I don't think it's prudent to put our CI machine under load. It's not like one of the random "free resources" machines you get out of GitHub

What about weekly benchmarking on CI? so we have a range to bisect for regressions? And maybe add a perf-sensitive label that triggers CI benchmarks?

I could do that, I have some experience with those kinds of setups

I wonder what @conjured/seeing-eye thinks about this. I'm just kinda like hm about putting that machine under too much load.

Maybe we could do it without CI, we'd just have to remember to actually do it.

My latest notes on this is that we're creating session cookies for any and all visitors, even when they're not logged in or trying to log in. People should be able to browse the site without a session cookie getting created. Just gotta fix the middleware.

I'm gonna investigate this

https://github.com/maxcountryman/axum-login/discussions/307#discussioncomment-15324144 this seems to be a way to make it possible to have our context-middleware conditionally do AuthSession extraction