Python: Handling different HTTP status codes using the requests library

Question

I am using the requests library to make requests to an API. However, I am not sure how to handle different status codes. I am particularly interested in handling status code 500, as that is part of my task.

1: Is a try/except block like I have an acceptable way of handling error codes?

2: How should I handle 500 status code? Would retrying 5 times once per second, then exiting be acceptable? If so, I don't know how to implement that without wrapping the whole function in a for loop (which seems bad practice).

Here is my function that makes HTTP requests:

def make_request(type, endpoint, h, data=None):
 try:
 url = f'https://example.net/not-important/{endpoint}'
 if type == 'GET':
 r = requests.get(url, headers=h)
 elif type == 'POST':
 r = requests.post(url, headers=h, json=data)
 # r = requests.get('https://httpstat.us/500')
 r.raise_for_status()
 except requests.exceptions.HTTPError as e:
 if e.response.status_code == 500:
 # 500 logic here. retry?
 else:
 print(f'An error occurred: {e}\n{e.response.status_code}')
 sys.exit(1)
 else:
 return r.json()

Answer 1

I suggest you implement exponential back-off. For example, first try after x seconds and then twice as long as the last wait time. This is the common practice for retries. Recursion often looks cleaner than a loop. You could give your make_request function an additional parameter with a default value of 0 that counts retries. As always with recursion make sure to have a break point e.g. 5 retries to avoid infinite recursion.

• Reinderien: Recursion often looks cleaner than a loop seems dubious and subjective. For this instance I really don't think it's the right tool.
• Bindestrich: @Reinderien I agree, but OP did not want a loop, so I offered an alternative.
• Reinderien: OP suspected that wrapping in a loop is bad practice, but I disagree. Some modicum of reassurance may help them see a different perspective.
• Bergi: @Reinderien Especially for handling different errors with different strategies, and only some of them by retries, recursion is often cleaner. Also a loop around a try block that continues only if an exception is thrown is a bit weird, and usually requires break or early return in the good case, or an extra state variable. Recursion is much more straightforward and requires less lines of code and less levels of indentation.

Answer 2

First, you should reduce to one call to request. You don't need to separate by method; just pass type to the method argument.

Don't call a variable type; that's a built-in.

Don't call an argument h; just call it headers.

Add PEP484 type hints to your signature, something like

def make_request(
 method: Literal['GET', 'POST'],
 endpoint: str,
 headers: dict[str, str],
 data: Optional[dict[str, Any]] = None,
) -> dict[str, Any]:

Don't exit() from a utility function. If you want to propagate a (possibly fatal) error, just raise or perhaps raise MyException('An error occurred' ...) from e.

Answer 3

1: Is a try/except block like I have an acceptable way of handling error codes?

By default, requests library does not raise exceptions based on HTTP status codes. Instead, status code is returned in status_code attribute of response object:

>>> r = requests.get('https://httpbin.org/get')
>>> r.status_code
200

Exceptions, though, can be raised using raise_for_status method, as you did.

However, in my opinion, it makes little sense to raise exception only to catch it int the very next line. So I'd get rid of try/except and raise_for_status and just inspect status_code:

def make_request(type, endpoint, h, data=None):
 url = f'https://example.net/not-important/{endpoint}'
 r = requests.request(type, url, headers=h, json=data)
 if r.status_code == 200:
 return r.json() # ok
 elif r.status_code == 500:
 pass # not ok, handle 500
 else:
 pass # not ok, handle other codes