Error Handling in Python

When interacting with APIs and other systems in Python, errors can happen. 404, 501, 429?? What do these numbers even mean?? Let's discuss some common error codes that might happen and how we might want to deal with them within our script. We'll work through a extract script built to extract data from TFL's Bike Point API.

Common Errors

Standard HTTP status codes are grouped into groups based on the things they relate to.

• 1xx - Informational response
• 2xx - Success
• 3xx - Redirection
• 4xx - Client error
• 5xx - Server error

When calling our API, the codes we're after are the 2xx codes. These would indicate that the call was successful. If we receive any other status codes, we want to think about how our script reacts to these codes. For example, if it's a server error (5xx), it might be worth just trying the call again after a small delay. However, a 401 error (authentication issue) will likely require you to go back into your code and make some changes (e.g. ensuring your credentials are correct). In this case you'd want to stop the script running so you can go and make those changes.

• Note - There may be some information on the codes you could receive in the API documentation, so keep an eye out for that!

Error Handling Loops

Step 1: Set up the retry settings

max_retry = 5
attempt = 0
delay = 10

Before the loop starts, we define three variables:

• max_retry: the maximum number of times we’ll try the call before giving up
• attempt: a counter that tracks how many tries we’ve used so far
• delay: how many seconds to wait between attempts

Without a cap on retries, a broken API could leave your script looping forever. The API may also have a rate limit, capping how many times you can attempt to access it. This logic could be useful to avoid hitting that limit.

Step 2: Make the call inside a while loop

while attempt < max_retry:
    response = requests.get(url)
    status = response.status_code

The loop keeps running as long as we have attempts left. The request and the status code live inside the loop because they need to be refreshed on every attempt. If we made the call outside the loop, we’d just be checking the same stale response over and over.

Step 3: Decide what to do based on the status code

This is where the groups of codes from earlier come in. Our script has three paths.

Path 1: Success (2xx)

if 200 <= status < 300: 
    data = response.json()

    if len(data) > 0:
        try:
            with open(filename, 'w') as file:
                json.dump(data, file)
            print(f'{filename} was successfully saved :)')
            logger.info(f'File {filename} was successfully saved')
        except Exception as e:
            print(f'An error has occured {e}')
            logger.error(f'An error has occured {e}')
        break
    else:
        print('No data returned')
        logger.warning('No data returned')
        break

If the call worked, we convert the response to Python and check that it actually contains something.

A quick side note, An API can return a 2xx code but still have an empty payload, to avoid just saving an empty file, we can set up a try/except around the file write which catches problems on our side.

Notice the break statements. Once we’ve succeeded (or reached a result that retrying won’t change), we exit the loop. There’s no point calling the API again.

Path 2: Worth retrying

elif status >= 500:
    time.sleep(delay)
    attempt += 1
    print(f'Status_code: {status}. Retrying, attempt number {attempt}')
    logger.info(f'Status_code: {status}. Retrying, attempt number {attempt}')

A 5xx error means something went wrong on the server’s end. It might be a temporary blip, so waiting a few seconds and trying again is a sensible response. In this case, we set our delay variable to 10 seconds at the start. We can use time.sleep(delay), add one to our attempt counter, and let the loop go round again.

There may be other specific error codes that you want to use this logic for (e.g. a timeout, certain client side errors, etc...).

Path 3: Stop and fix it

else:
    print(f'Error, Status code {status}, Fix it')
    logger.error(f'Error, Status code {status}, Fix it')
    break

Other codes such as a 401 (bad credentials) or a 404 (wrong URL), indicates a problem with our request. Retrying won’t help, because we’d just get the same error five times. Instead, we log the problem and break out of the loop so we can go and fix the code.

All together, what could have been just a few lines of code set up to just call the API becomes a decently sized script which can look intimidating if you're new to python but it can save you lots of headache trying to figure out why your script isn't working. This is extremely useful in a business context where efficiency is important.

Full Script

max_retry = 5
attempt = 0
delay = 10

while attempt < max_retry:
    response = requests.get(url)
    status = response.status_code

if 200 <= status < 300: 
    data = response.json()

    if len(data) > 0:
        try:
            with open(filename, 'w') as file:
                json.dump(data, file)
            print(f'{filename} was successfully saved :)')
            logger.info(f'File {filename} was successfully saved')
        except Exception as e:
            print(f'An error has occured {e}')
            logger.error(f'An error has occured {e}')
        break
    else:
        print('No data returned')
        logger.warning('No data returned')
        break

elif status >= 500:
    time.sleep(delay)
    attempt += 1
    print(f'Status_code: {status}. Retrying, attempt number {attempt}')
    logger.info(f'Status_code: {status}. Retrying, attempt number {attempt}')

else:
    print(f'Error, Status code {status}, Fix it')
    logger.error(f'Error, Status code {status}, Fix it')
    break