Serving several external APIs in Django + Angular application

Question

I'm working on a web-based app that uses Django and Angular. This app uses several external APIs to fetch environmental data from different monitoring networks. We then use these data to perform forecasts and gain insights about environmental issues. The APIs belong to different owners so the way the data is retrieved, the form of the urls, the data organization, etc. is super heterogeneous.

They way we are currently working is like this: for each external API we create a new Django app and in each app views.py we create three django views:

def get_stations(request) -> JSONResponse:
"""Fetch stations as points (lat and long) that can be plotted on the frontend"""
def get_parameters(request) -> JSONResponse:
"""Fetch the parameters available by station (this view is called after clicking on a station)"""
def get_timeseries(request) -> JSON Response:
"""Fetch the events from a specific parameter (called after clicking on a parameter)"""

Naturally, each of this views calls different functions that request the data from the APIs and each of them return a normalized response (basically a dictionary with predefined keys). For instance, the get_parameters view for a NASA Api (just an example) would return a List with parameters like this (the keys are manually passed).

parameter = {"ID": "PCP00123", "Name": "Precipitation", "Location_ID": "ATK_01", "API_source": "NASA", "Units": "mm"}

The numbers of APIs we are connecting to is growing fast and, given that I have little experience with the Django framework and software development in general, I'm a little bit concerned about the future of the app. Therefore, I would like to know:

How could I improve the design to make the app more robust?

• Greg Burghardt: You have a lot of good information, but I don't see where you've asked a question. Can you edit your post to identify the software design problem you are trying to solve?
• Community: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
• stray_dog: @GregBurghardt Thank you. I have removed unnecesary text and made my questions more explicit.
• Greg Burghardt: Thank you for the update! The Q&A format of this community has a restriction of one question per post. I see one vote to close as needing focus. Can you edit your post to focus on one of the questions?
• CPlus: Are there any ways in particular that you find that current approach is not robust enough?

Answer 1

Note: I am basing my answer on the assumption that "Django + Angular" means "Angular front-end with Django back-end".

If you haven't already, I would look into Django REST Framework. This is by far the easiest way to create custom APIs with Django.

Now, from what you describe, you should really only need one Django app. Let me break this problem down into two parts: (1) handling the growing number of external APIs being used; and (2) using the external APIs to return the necessary data to the user.

Part 1: Addressing the Scaling Issue

One of the best parts about Django is that you can manage data very easily with it. I would start by creating a database specifically for this application and connecting it to your Django project.

From here, you should create a model, that is meant to represent an individual external API, in the models.py file. You'll likely have 3 fields in this model, one for each of the 3 endpoints that you have given us as an example. You can also add other fields if you would like (maybe a "name" field for readability?). After the model is created to your liking, add this model to the admin page (admin.py).

Upon completion of the above, run the Django server. It may ask you to run migrations, which the output goes into further detail on how to do so. Once the server is running, navigate to "/admin" on the server URL. It will prompt you to log in to your account (assuming that you've created one already). After logging in, you should see the model that you created earlier. From here, you can begin to add all of your external APIs.

Part 2: Getting the Data

With the proper data structure in place, you are now ready to use the data that you've stored. As mentioned before, you'll want to use the Django REST Framework.

In views.py, you will create all of the endpoints that you will need for the application, including:

• A GET request for all of the external APIs you have available, which should return the data for each of those APIs, including its unique ID.
• A GET request for each of the views that you mentioned before
  • Instead of manually-entered URLs, you will use the new model to grab the correct URL from the database based on a given ID (the documentation on models that I linked before tells you how to do this)
  • After getting the proper URL, you will call that API as normal and return the data that you receive.

Once the views have been created, go into urls.py and create URLs for each endpoint. The most important part of this is being able request data based on an external API's unique ID, as that allows us to keep the API isolated to a single app. Make sure that each of the views you have created can accept this unique ID so that it can return the correct data.

With everything in place, you should be able to navigate to whatever URL you would like in the browser (thanks to Django REST Framework), and if successful, it will show you the data that you need.

• stray_dog: Thanks for commenting! Firstly, yes. I meant angular frontend and django backend. Secondly, why do I need to create a database? As it is right now, I'm getting the data from external parties. I don't really want to store anything. Is there any benefit in this case? Thank you very much.
• yogos15: @stray_dog The database isn't for the actual data you are getting from the external APIs. The database is exclusively for storing all of the URLs for these external APIs. When scaling this application, creating a separate app within your Django project for every single new API just doesn't make sense in the long run. Having a database where you store all of the API URLs allows you to add more APIs without actually modifying the Django code.