How to implement Asynchronous Processing? · django-json-api/django-rest-framework-json-api · Discussion #1012

Recommendations from the JSON:API specs

JSON:API defines the following to handle async processing for long running tasks:

POST /photos HTTP/1.1

The request SHOULD return a status 202 Accepted with a link in the Content-Location header.

HTTP/1.1 202 Accepted
Content-Type: application/vnd.api+json
Content-Location: https://example.com/photos/queue-jobs/5234
{
 "data": {
 "type": "queue-jobs",
 "id": "5234",
 "attributes": {
 "status": "Pending request, waiting other process"
 },
 "links": {
 "self": "/photos/queue-jobs/5234"
 }
 }
}

-- json:api

My implementation to handle long running create endpoint

To follow that recommendation from the json:api specs if implemented it as following:

Long running create endpoint

from django_celery_results.models import TaskResult
from rest_framework_json_api.views import ModelViewSet
class OgcServiceViewSet(ModelViewSet):
 queryset = OgcService.objects.all()
 serializer_classes = {
 'default': OgcServiceSerializer,
 'create': OgcServiceCreateSerializer
 }
 filterset_fields = {
 'id': ('exact', 'lt', 'gt', 'gte', 'lte', 'in'),
 'title': ('icontains', 'iexact', 'contains'),
 }
 search_fields = ('id', 'title',)
 def get_serializer_class(self):
 return self.serializer_classes.get(self.action, self.serializer_classes['default'])
 def create(self, request, *args, **kwargs):
 serializer = self.get_serializer(data=request.data)
 serializer.is_valid(raise_exception=True)
 task = build_ogc_service.delay(data=serializer.data)
 task_result, created = TaskResult.objects.get_or_create(task_id=task.id)
 serialized_task_result = TaskResultSerializer(task_result)
 serialized_task_result_data = serialized_task_result.data
 # meta object is None... we need to set it to an empty dict to prevend uncaught runtime exceptions
 meta = serialized_task_result_data.get("meta", None)
 if not meta:
 serialized_task_result_data.update({"meta": {}})
 headers = self.get_success_headers(serialized_task_result_data)
 return Response(serialized_task_result_data, status=status.HTTP_202_ACCEPTED, headers=headers)
 def get_success_headers(self, data):
 try:
 return {'Content-Location': str(data[api_settings.URL_FIELD_NAME])}
 except (TypeError, KeyError):
 return {}

Create serializer with different field set

class OgcServiceCreateSerializer(ModelSerializer):
 # TODO: implement included serializer for ServiceAuthentication
 # included_serializers = {
 # 'auth': ServiceAuthentication,
 # }
 class Meta:
 model = OgcService
 fields = ("get_capabilities_url", )

TaskResult serializer

from django_celery_results.models import TaskResult
from registry.models.jobs import RegisterOgcServiceJob
from rest_framework_json_api.serializers import ModelSerializer
class TaskResultSerializer(ModelSerializer):
 class Meta:
 model = TaskResult
 fields = "__all__"

Response

The create enpoint is creating the celery job fine and response with the content like bellow:

Response Body

{
 "data": {
 "type": "OgcService",
 "id": "8",
 "attributes": {
 "task_id": "23e945e2-4c63-45bf-a76d-b64062db930e",
 "task_name": null,
 "task_args": null,
 "task_kwargs": null,
 "status": "PENDING",
 "worker": null,
 "content_type": "",
 "content_encoding": "",
 "result": null,
 "date_created": "2021年11月22日T09:34:11.940341+01:00",
 "date_done": "2021年11月22日T09:34:11.940404+01:00",
 "traceback": null,
 "meta": {}
 }
 }
}

Response headers

 allow: GET,POST,HEAD,OPTIONS content-encoding: gzip content-language: en content-length: 238 content-type: application/vnd.api+json referrer-policy: same-origin server-timing: SQLPanel_sql_time;dur=3.191709518432617;desc="SQL 2 queries" vary: Accept-Language,Cookie,Accept-Encoding,Origin x-content-type-options: nosniff x-frame-options: DENY

Quesion

As you can see in the response body, the type differs to the resource type of the used serializer. How can dynamically set the type to the correct "type": "TaskResult" ?
How can i add url field to provide resource urls as meta information on the TaskResultSerializer without naming all fields explicit?

Replies: 2 comments 4 replies

sliverc
Nov 23, 2021
Maintainer

I did not notice before that there is a recommendation from JSON:API spec on async calls... Great!

As far as I understand your example it would be enough if you set resource_name in the create serializer like the following.

class OgcServiceCreateSerializer:
 class Meta:
 resource_name = 'TaskResult'

I hope I understand you correctly here. DJA removes the URLs from fields as this is presented in the links (see here). You can write a custom metadata class which overwrites get_serializer_info though. To follow the JSON:API spec recommendation though you would need to overwrite get_links in your view to set the correct self link.

4 replies

@jokiefer

jokiefer Nov 24, 2021
Author

As far as I understand your example it would be enough if you set resource_name in the create serializer like the following.

That did not properly work. I defined the resource_name on OgcServiceCreateSerializer and/or on TaskResultSerializer. The endpoint will always response with:

{
 "errors": [
 {
 "detail": "The resource object's type (OgcService) is not the type that constitute the collection represented by the endpoint (TaskResult).",
 "status": "409",
 "source": {
 "pointer": "/data"
 },
 "code": "error"
 }
 ]
}

on given POST data:

{
	"data": {
		"attributes": {
			"get_capabilities_url": "http://geo5.service24.rlp.de/wms/karte_rp.fcgi?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetCapabilities"
		},
		"relationships": {
			"owned_by_org": {
				"data": {
					"id": "b2fdc463-87cb-4deb-b39b-187f7616d1df",
					"type": "Organization"
				}
			}
		},
		"type": "OgcService"
	}
}

This is correct, cause JSON:API spec defines the following:

A server MUST return 409 Conflict when processing a POST request in which the resource object’s type is not among the type(s) that constitute the collection represented by the endpoint.

But i did not find a constraint in the JSON:API spec that does not allow on a endpoint to response with a different resource object type (TaskResult). Here is a discussion on the jsonapi.org forum, which handles about that.

So for my request the response looks like (with mismatching resource object type) :

{
 "data": {
 "type": "OgcService",
 "id": "9",
 "attributes": {
 "task_meta": null,
 "task_id": "d8146e14-d1ef-46d5-a6d2-be0de7017703",
 "task_name": null,
 "task_args": null,
 "task_kwargs": null,
 "status": "PENDING",
 "worker": null,
 "content_type": "",
 "content_encoding": "",
 "result": null,
 "date_created": "2021年11月24日T09:12:35.056767+01:00",
 "date_done": "2021年11月24日T09:12:35.056860+01:00",
 "traceback": null
 },
 "links": {
 "self": "http://testserver/api/v1/registry/task-results/9/"
 }
 }
}

Here is the link to the current implemented code: https://github.com/mrmap-community/mrmap/blob/a215b1c11bfff1f17d13c27d3e7b3ca88c3e3f87/backend/registry/api/views/service.py#L108

I hope I understand you correctly here. DJA removes the URLs from fields as this is presented in the links (see here). You can write a custom metadata class which overwrites get_serializer_info though. To follow the JSON:API spec recommendation though you would need to overwrite get_links in your view to set the correct self link.

OK - i understood.
My solution was it to add the field as HyperlinkedIdentityField:

class WebMapServiceSerializer(ModelSerializer):
 url = HyperlinkedIdentityField(
 view_name='registry:taskresult-detail',
 )
 class Meta:
 model = WebMapService
 fields = "__all__"

I struggled a bit with the view_name, cause i forgot to pass the app_name as well.

@sliverc

sliverc Nov 30, 2021
Maintainer

True this error message makes sense. If we wanna support this async recommendation we properly would need a special case when response status code is 202 which than also handles Content-Location. I haven't investigated how this could be accomplished yet. Suggestions are welcome.

@jokiefer

jokiefer Dec 1, 2021
Author

I think the handling of the resource_name and Content-Location could be done with a little switch in the renderer:

class JSONRenderer(renderers.JSONRenderer):
 ...
 def render(self, data, accepted_media_type=None, renderer_context=None):
 if serializer is not None:
 # Extract root meta for any type of serializer
 json_api_meta.update(self.extract_root_meta(serializer, serializer_data))
 if getattr(serializer, "many", False):
 ...
 else:
 ...
 if response.status_code == 202:
 # handle async processing as recommended https://jsonapi.org/recommendations/#asynchronous-processing
 resource_name = utils.get_resource_type_from_serializer(serializer)
 response.headers.update({'Content-Location': serializer_data[api_settings.URL_FIELD_NAME]})
 resource_instance = serializer.instance
 json_api_data = self.build_json_resource_obj(
 fields,
 serializer_data,
 resource_instance,
 resource_name,
 serializer,
 force_type_resolution,
 )

@sliverc

sliverc Dec 2, 2021
Maintainer

Hmm not so sure. During parsing wouldn't this still lead to the same error as when resource_name of the created serializer is set differently? Also in terms of features the renderer is already fairly complex and we need to work on that to make it easier and more maintainable. So I want to try to avoid making it more complex especially when adding optional features like async support. An idea (not thought through yet) but what about putting the support of async calls into a AsyncResponse class?

sliverc
Nov 30, 2021
Maintainer

Additionally as a workaround what you could do is set resource_name to False in your OgcServiceCreateSerializer. This means DJA rendering will be skipped and the data will be returned as you pass it on to Response. This means in your code you have to structure the data as it is required by the JSON:API async recommendation. Not ideal but should work as a workaround.

0 replies

How to implement Asynchronous Processing? #1012

Uh oh!

jokiefer Nov 22, 2021

Recommendations from the JSON:API specs

My implementation to handle long running create endpoint

Long running create endpoint

Create serializer with different field set

TaskResult serializer

Response

Response Body

Response headers

Quesion

Replies: 2 comments · 4 replies

Uh oh!

sliverc Nov 23, 2021 Maintainer

Uh oh!

Uh oh!

jokiefer Nov 24, 2021 Author

Uh oh!

sliverc Nov 30, 2021 Maintainer

Uh oh!

jokiefer Dec 1, 2021 Author

Uh oh!

sliverc Dec 2, 2021 Maintainer

Uh oh!

sliverc Nov 30, 2021 Maintainer

jokiefer
Nov 22, 2021

Replies: 2 comments 4 replies

sliverc
Nov 23, 2021
Maintainer

jokiefer Nov 24, 2021
Author

sliverc Nov 30, 2021
Maintainer

jokiefer Dec 1, 2021
Author

sliverc Dec 2, 2021
Maintainer

sliverc
Nov 30, 2021
Maintainer