-
Notifications
You must be signed in to change notification settings - Fork 299
Issue #337 #366
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue #337 #366
Conversation
...ormance expectations (also flake8)
...viewset to help with prefetching includes.
Add Django Debug Toolbar, and add a failing test for query explosions
Add a helper for specifying a prefetch_for_related attribute on your viewset to help with prefetching includes.
5f56a2d
to
d71e526
Compare
Codecov Report
@@ Coverage Diff @@ ## develop #366 +/- ## ========================================= + Coverage 92.81% 93% +0.19% ========================================= Files 51 51 Lines 2644 2659 +15 ========================================= + Hits 2454 2473 +19 + Misses 190 186 -4
Continue to review full report at Codecov.
|
d71e526
to
a75a613
Compare
a75a613
to
f924ac8
Compare
Looks great!
@jsenecal It looks like Travis is failing because of isort (curiously, only Python 3.4 🤔 ). I think isort gets a little tricky because the config file is trying to be correct for the rest_framework_json_api, but it also has to satisfy the included example package. I'm guessing you can solve this by including --thirdparty packaging
to the isort command that checks example
.
$ isort --check-only --verbose --recursive --diff --thirdparty pytest \ --thirdparty polymorphic --thirdparty pytest_factoryboy \ --thirdparty packaging example
I'll look at the code closer to give it some review.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this ignoring manage.py
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@aidanlister must have a custom one (introduced by da0c94c).
We're not using it to start the app anyways ($ django-admin.py runserver --settings=example.settings
)
example/settings/dev.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since example
is what runs on the demo site, should we leave the form version on by default instead of the perf version so that people can add extra data to play with it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess it would make sense indeed - let me fix this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👏 I'm a fan of any time a *
import can be removed.
I like what's here a lot!
What do you think about documentation for this stuff? IMO, features don't really exist until they are documented so that users know how to use them. With that stated, it doesn't matter to me if the features are documented in this branch or in some other branch as long as there is some prose before in the docs before the next release.
Once Travis is green, I'm ok with merging. If you decide to add some docs, I can give it another look.
Thanks for pulling this together with @aidanlister. 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still have an issue with the way relations are extracted first level... I had to prefetch things that I shouldn't have to only to make tests pass... Feels like the queryset might be over reaching to overcome an issue with drf_jsonapi.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After further investigation, the problem does not occur inside extract_relationships - Still looking
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jsenecal the problem is in two places, setting use_pk_only_optimization to False, and extract_attributes. You have to whack them both, which made it so hard to find the source of the explosion.
I think you should probably remove the select_related in this because the tests should fail until we fix that other issue?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this - I merged already but will update my branch to reflect this conversation
@jsenecal thanks for including my cite, might be better to give some context there though? Let me write something:
Using the helper to prefetch, rather than attempting to minimise queries via select_related might give you better performance depending on the characteristics of your data and database.
For example:
If you have a single model, e.g. Book, which has four relations e.g. Author, Publisher, CopyrightHolder, Category.
To display 25 books and related models, you would need to either do:
a) 1 query via selected_related, e.g. SELECT * FROM books LEFT JOIN author LEFT JOIN publisher LEFT JOIN CopyrightHolder LEFT JOIN Category
b) 4 small queries via prefetch_related.
If you have 1M books, 50k authors, 10k categories, 10k copyrightholders in the select_related scenario, you've just created a in-memory table with 1e18 rows which will likely exhaust any available memory and slow your database to crawl. The prefetch_related case will issue 4 queries, but they will be small and fast queries.
No need to cite me, my paragraph in that issue is about the query explosion from displaying FKs -- let's merge this and we'll keep working out of that issue.
Uh oh!
There was an error while loading. Please reload this page.
Thanks to @aidanlister for his contribution on this.