Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Commit 8bb123c

Browse files
Anton-Shutiksliverc
authored andcommitted
Allow defining of select_related per include (#600)
1 parent f3e67a7 commit 8bb123c

File tree

5 files changed

+125
-21
lines changed

5 files changed

+125
-21
lines changed

‎CHANGELOG.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,12 +14,22 @@ any parts of the framework not mentioned in the documentation should generally b
1414

1515
* Add support for Django 2.2
1616

17+
### Changed
18+
19+
* Allow to define `select_related` per include using [select_for_includes](https://django-rest-framework-json-api.readthedocs.io/en/stable/usage.html#performance-improvements)
20+
* Reduce number of queries to calculate includes by using `select_related` when possible
21+
1722
### Fixed
1823

1924
* Avoid exception when trying to include skipped relationship
2025
* Don't swallow `filter[]` params when there are several
2126
* Fix DeprecationWarning regarding collections.abc import in Python 3.7
22-
* Allow OPTIONS request to be used on RelationshipView.
27+
* Allow OPTIONS request to be used on RelationshipView
28+
29+
### Deprecated
30+
31+
* Deprecate `PrefetchForIncludesHelperMixin` use `PreloadIncludesMixin` instead
32+
* Deprecate `AutoPrefetchMixin` use `AutoPreloadMixin` instead
2333

2434
## [2.7.0] - 2019年01月14日
2535

‎docs/usage.md‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -823,17 +823,23 @@ class QuestSerializer(serializers.ModelSerializer):
823823

824824
Be aware that using included resources without any form of prefetching **WILL HURT PERFORMANCE** as it will introduce m\*(n+1) queries.
825825

826-
A viewset helper was designed to allow for greater flexibility and it is automatically available when subclassing
826+
A viewset helper was therefore designed to automatically preload data when possible. Such is automatically available when subclassing `ModelViewSet`.
827+
828+
It also allows to define custom `select_related` and `prefetch_related` for each requested `include` when needed in special cases:
829+
827830
`rest_framework_json_api.views.ModelViewSet`:
828831
```python
829832
from rest_framework_json_api import views
830833

831834
# When MyViewSet is called with ?include=author it will dynamically prefetch author and author.bio
832835
class MyViewSet(views.ModelViewSet):
833836
queryset = Book.objects.all()
837+
select_for_includes = {
838+
'author': ['author__bio'],
839+
}
834840
prefetch_for_includes = {
835841
'__all__': [],
836-
'author': ['author', 'author__bio'],
842+
'all_authors': [Prefetch('all_authors', queryset=Author.objects.select_related('bio'))],
837843
'category.section': ['category']
838844
}
839845
```
@@ -848,7 +854,7 @@ class MyReadOnlyViewSet(views.ReadOnlyModelViewSet):
848854

849855
The special keyword `__all__` can be used to specify a prefetch which should be done regardless of the include, similar to making the prefetch yourself on the QuerySet.
850856

851-
Using the helper to prefetch, rather than attempting to minimise queries via select_related might give you better performance depending on the characteristics of your data and database.
857+
Using the helper to prefetch, rather than attempting to minimise queries via `select_related` might give you better performance depending on the characteristics of your data and database.
852858

853859
For example:
854860

@@ -861,11 +867,11 @@ a) 1 query via selected_related, e.g. SELECT * FROM books LEFT JOIN author LEFT
861867
b) 4 small queries via prefetch_related.
862868

863869
If you have 1M books, 50k authors, 10k categories, 10k copyrightholders
864-
in the select_related scenario, you've just created a in-memory table
870+
in the `select_related` scenario, you've just created a in-memory table
865871
with 1e18 rows which will likely exhaust any available memory and
866872
slow your database to crawl.
867873

868-
The prefetch_related case will issue 4 queries, but they will be small and fast queries.
874+
The `prefetch_related` case will issue 4 queries, but they will be small and fast queries.
869875
<!--
870876
### Relationships
871877
### Errors

‎example/tests/test_performance.py‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,16 @@ def test_query_count_include_author(self):
5353
4. Author types prefetched
5454
5. Entries prefetched
5555
"""
56-
with self.assertNumQueries(5):
56+
with self.assertNumQueries(4):
5757
response = self.client.get('/comments?include=author&page[size]=25')
5858
self.assertEqual(len(response.data['results']), 25)
59+
60+
def test_query_select_related_entry(self):
61+
""" We expect a list view with an include have two queries:
62+
63+
1. Primary resource COUNT query
64+
2. Primary resource SELECT + SELECT RELATED writer(author) and bio
65+
"""
66+
with self.assertNumQueries(2):
67+
response = self.client.get('/comments?include=writer&page[size]=25')
68+
self.assertEqual(len(response.data['results']), 25)

‎example/views.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
from rest_framework_json_api.filters import OrderingFilter, QueryParameterValidationFilter
1313
from rest_framework_json_api.pagination import JsonApiPageNumberPagination
1414
from rest_framework_json_api.utils import format_drf_errors
15-
from rest_framework_json_api.views import ModelViewSet, RelationshipView
15+
from rest_framework_json_api.views import ModelViewSet, RelationshipView, PreloadIncludesMixin
1616

1717
from example.models import Author, Blog, Comment, Company, Entry, Project, ProjectType
1818
from example.serializers import (
@@ -184,6 +184,9 @@ class AuthorViewSet(ModelViewSet):
184184
class CommentViewSet(ModelViewSet):
185185
queryset = Comment.objects.all()
186186
serializer_class = CommentSerializer
187+
select_for_includes = {
188+
'writer': ['author__bio']
189+
}
187190
prefetch_for_includes = {
188191
'__all__': [],
189192
'author': ['author__bio', 'author__entries'],
@@ -197,9 +200,12 @@ def get_queryset(self, *args, **kwargs):
197200
return super(CommentViewSet, self).get_queryset()
198201

199202

200-
class CompanyViewset(ModelViewSet):
203+
class CompanyViewset(PreloadIncludesMixin, viewsets.ModelViewSet):
201204
queryset = Company.objects.all()
202205
serializer_class = CompanySerializer
206+
prefetch_for_includes = {
207+
'current_project': ['current_project'],
208+
}
203209

204210

205211
class ProjectViewset(ModelViewSet):

‎rest_framework_json_api/views.py‎

Lines changed: 84 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import warnings
12

23
from django.core.exceptions import ImproperlyConfigured
34
from django.db.models import Model
@@ -31,6 +32,13 @@
3132

3233

3334
class PrefetchForIncludesHelperMixin(object):
35+
36+
def __init__(self, *args, **kwargs):
37+
warnings.warn("PrefetchForIncludesHelperMixin is deprecated. "
38+
"Use PreloadIncludesMixin instead",
39+
DeprecationWarning)
40+
super(PrefetchForIncludesHelperMixin, self).__init__(*args, **kwargs)
41+
3442
def get_queryset(self):
3543
"""
3644
This viewset provides a helper attribute to prefetch related models
@@ -62,33 +70,86 @@ class MyViewSet(viewsets.ModelViewSet):
6270
return qs
6371

6472

65-
class AutoPrefetchMixin(object):
73+
class PreloadIncludesMixin(object):
74+
"""
75+
This mixin provides a helper attributes to select or prefetch related models
76+
based on the include specified in the URL.
77+
78+
__all__ can be used to specify a prefetch which should be done regardless of the include
79+
80+
.. code:: python
81+
82+
# When MyViewSet is called with ?include=author it will prefetch author and authorbio
83+
class MyViewSet(viewsets.ModelViewSet):
84+
queryset = Book.objects.all()
85+
prefetch_for_includes = {
86+
'__all__': [],
87+
'category.section': ['category']
88+
}
89+
select_for_includes = {
90+
'__all__': [],
91+
'author': ['author', 'author__authorbio'],
92+
}
93+
"""
94+
95+
def get_select_related(self, include):
96+
return getattr(self, 'select_for_includes', {}).get(include, None)
97+
98+
def get_prefetch_related(self, include):
99+
return getattr(self, 'prefetch_for_includes', {}).get(include, None)
100+
101+
def get_queryset(self, *args, **kwargs):
102+
qs = super(PreloadIncludesMixin, self).get_queryset(*args, **kwargs)
103+
104+
included_resources = get_included_resources(self.request)
105+
for included in included_resources + ['__all__']:
106+
107+
select_related = self.get_select_related(included)
108+
if select_related is not None:
109+
qs = qs.select_related(*select_related)
110+
111+
prefetch_related = self.get_prefetch_related(included)
112+
if prefetch_related is not None:
113+
qs = qs.prefetch_related(*prefetch_related)
114+
115+
return qs
116+
117+
118+
class AutoPreloadMixin(object):
119+
66120
def get_queryset(self, *args, **kwargs):
67121
""" This mixin adds automatic prefetching for OneToOne and ManyToMany fields. """
68-
qs = super(AutoPrefetchMixin, self).get_queryset(*args, **kwargs)
122+
qs = super(AutoPreloadMixin, self).get_queryset(*args, **kwargs)
69123
included_resources = get_included_resources(self.request)
70124

71-
for included in included_resources:
125+
for included in included_resources + ['__all__']:
126+
# If include was not defined, trying to resolve it automatically
72127
included_model = None
73128
levels = included.split('.')
74129
level_model = qs.model
130+
# Suppose we can do select_related by default
131+
can_select_related = True
75132
for level in levels:
76133
if not hasattr(level_model, level):
77134
break
78135
field = getattr(level_model, level)
79136
field_class = field.__class__
80137

81138
is_forward_relation = (
82-
issubclass(field_class, ForwardManyToOneDescriptor) or
83-
issubclass(field_class, ManyToManyDescriptor)
139+
issubclass(field_class, (ForwardManyToOneDescriptor, ManyToManyDescriptor))
84140
)
85141
is_reverse_relation = (
86-
issubclass(field_class, ReverseManyToOneDescriptor) or
87-
issubclass(field_class, ReverseOneToOneDescriptor)
142+
issubclass(field_class, (ReverseManyToOneDescriptor, ReverseOneToOneDescriptor))
88143
)
89144
if not (is_forward_relation or is_reverse_relation):
90145
break
91146

147+
# Figuring out if relation should be select related rather than prefetch_related
148+
# If at least one relation in the chain is not "selectable" then use "prefetch"
149+
can_select_related &= (
150+
issubclass(field_class, (ForwardManyToOneDescriptor, ReverseOneToOneDescriptor))
151+
)
152+
92153
if level == levels[-1]:
93154
included_model = field
94155
else:
@@ -104,11 +165,23 @@ def get_queryset(self, *args, **kwargs):
104165
level_model = model_field.model
105166

106167
if included_model is not None:
107-
qs = qs.prefetch_related(included.replace('.', '__'))
168+
if can_select_related:
169+
qs = qs.select_related(included.replace('.', '__'))
170+
else:
171+
qs = qs.prefetch_related(included.replace('.', '__'))
108172

109173
return qs
110174

111175

176+
class AutoPrefetchMixin(AutoPreloadMixin):
177+
178+
def __init__(self, *args, **kwargs):
179+
warnings.warn("AutoPrefetchMixin is deprecated. "
180+
"Use AutoPreloadMixin instead",
181+
DeprecationWarning)
182+
super(AutoPrefetchMixin, self).__init__(*args, **kwargs)
183+
184+
112185
class RelatedMixin(object):
113186
"""
114187
This mixin handles all related entities, whose Serializers are declared in "related_serializers"
@@ -186,15 +259,14 @@ def get_related_instance(self):
186259
raise NotFound
187260

188261

189-
class ModelViewSet(AutoPrefetchMixin,
190-
PrefetchForIncludesHelperMixin,
262+
class ModelViewSet(AutoPreloadMixin,
263+
PreloadIncludesMixin,
191264
RelatedMixin,
192265
viewsets.ModelViewSet):
193266
pass
194267

195268

196-
class ReadOnlyModelViewSet(AutoPrefetchMixin,
197-
PrefetchForIncludesHelperMixin,
269+
class ReadOnlyModelViewSet(AutoPreloadMixin,
198270
RelatedMixin,
199271
viewsets.ReadOnlyModelViewSet):
200272
pass

0 commit comments

Comments
(0)

AltStyle によって変換されたページ (->オリジナル) /