Python requests.request returning garbage

Question

Calling an api. For almost all calls the response comes clean but for a few there is garbage in the returned json. Executing the garbage one in the Firefox browser with the same parameters returns clean json. Working code for testing:

import requests
headers = {
 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:144.0) Gecko/20100101 Firefox/144.0',
 'Accept': 'application/json, text/javascript, */*; q=0.01',
 'Accept-Language': 'en-US,en;q=0.8,pt-BR;q=0.5,pt;q=0.3',
 'Accept-Encoding': 'gzip, deflate, br, zstd',
 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
 'X-Requested-With': 'XMLHttpRequest',
 'Origin': 'https://veiculos.fipe.org.br',
 'DNT': '1',
 'Sec-GPC': '1',
 'Connection': 'keep-alive',
 'Referer': 'https://veiculos.fipe.org.br/',
 'Cookie': 'ROUTEID=.5',
 'Sec-Fetch-Dest': 'empty',
 'Sec-Fetch-Mode': 'cors',
 'Sec-Fetch-Site': 'same-origin',
 'Priority': 'u=0',
 'TE': 'trailers',
}
def api_call(url, headers, parameters):
 try:
 r = requests.request('post', url, headers=headers, data=parameters)
 print (r.content)
 r_json = r.json()
 r.close()
 return r_json
 except Exception as e:
 print(repr(e))
url = 'https://veiculos.fipe.org.br/api/veiculos/ConsultarAnoModelo'
# First call returning clean json
parameters = [('codigoModelo', 6906), ('codigoTabelaReferencia', 327), ('codigoTipoVeiculo', 1), ('CodigoMarca', 189)]
r_json = api_call(url, headers, parameters)
# Second call returning garbage
parameters = [('codigoModelo', 6340), ('codigoTabelaReferencia', 327), ('codigoTipoVeiculo', 1), ('CodigoMarca', 189)]
r_json = api_call(url, headers, parameters)

The second call always raises a JSONDecodeError because of the initial sequence \x8b\x15\x80 and the final sequence \x03

b'[{"Label":"2016 Gasolina","Value":"2016-1"},{"Label":"2014 Gasolina","Value":"2014-1"}]'
b'\x8b\x15\x80[{"Label":"2011 Gasolina","Value":"2011-1"}]\x03'
JSONDecodeError('Expecting value: line 1 column 1 (char 0)')

The buggy one executed in the browser:

enter image description here

Any ideas?

Thanks to a comment now I know it only errors when run in a Python virtual environment (venv).

• Booboo: When I run this from a command prompt I do not get an exception. How are you executing the posted code in a browser? Firefox does not understand Python, so I don't see how.
• charlotte: Does it return the same garbage characters at the start/end if you run the same request twice in a row?
• Clodoaldo Neto: @charlotte Yes. It is totally repeatable.
• Clodoaldo Neto: @Booboo You pointed me in the right direction. When I run it in a virtual environment (venv) it errors. But in a system host terminal it does ok.
• charlotte: that is weird. maybe you could try re-installing requests or setting up your venv from scratch?

Answer 1

The response is encoded using Brotli compression:

>>> r.headers['content-encoding']
'br'

If the brotli package is available, requests imports it and uses it to decode the response. Otherwise, it does anything (no error reported either). This explains the different behavior in different environments.

How to fix it: ensure the brotli package is installed. (Alternatively: do not include 'br' with the 'Accept-Encoding' header)

• jsbueno: That. It was quite a surprise for me - and that this default Brotli Python package is a bit huge at a few megabytes, and will prevent "free-threading" on Python 3.14t
• Gabriel Genellina: Just omit 'br' from the Accept-Encoding header - an API call like this does not benefit from compression.

Answer 2

The problem occcurs in the second api_call(). Because the response r contains �� at start (r.text)

��[{"Label":"2011 Gasolina","Value":"2011-1"}]

and that's why r.json() throws the JSONDecodeError

JSONDecodeError('Expecting value: line 1 column 1 (char 0)')

One way to fix is to simply remove the br encoding from the Accept-Encoding header.

'Accept-Encoding': 'gzip, deflate, zstd'

code:

import requests
headers = {
 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:144.0) Gecko/20100101 Firefox/144.0',
 'Accept': 'application/json, text/javascript, */*; q=0.01',
 'Accept-Language': 'en-US,en;q=0.8,pt-BR;q=0.5,pt;q=0.3',
 'Accept-Encoding': 'gzip, deflate, zstd',
 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
 'X-Requested-With': 'XMLHttpRequest',
 'Origin': 'https://veiculos.fipe.org.br',
 'DNT': '1',
 'Sec-GPC': '1',
 'Connection': 'keep-alive',
 'Referer': 'https://veiculos.fipe.org.br/',
 'Cookie': 'ROUTEID=.5',
 'Sec-Fetch-Dest': 'empty',
 'Sec-Fetch-Mode': 'cors',
 'Sec-Fetch-Site': 'same-origin',
 'Priority': 'u=0',
 'TE': 'trailers',
}
def api_call(url, headers, parameters):
 try:
 r = requests.request('post', url, headers=headers, data=parameters)
 r_json = r.json()
 print(r_json)
 r.close()
 return r_json
 except Exception as e:
 print(repr(e))
url = 'https://veiculos.fipe.org.br/api/veiculos/ConsultarAnoModelo'
# First call returning clean json
parameters = [('codigoModelo', 6906), ('codigoTabelaReferencia', 327), ('codigoTipoVeiculo', 1), ('CodigoMarca', 189)]
r_json = api_call(url, headers, parameters)
# Second call returning garbage
parameters = [('codigoModelo', 6340), ('codigoTabelaReferencia', 327), ('codigoTipoVeiculo', 1), ('CodigoMarca', 189)]
r_json = api_call(url, headers, parameters)

output:

[{'Label': '2016 Gasolina', 'Value': '2016-1'}, {'Label': '2014 Gasolina', 'Value': '2014-1'}]
[{'Label': '2011 Gasolina', 'Value': '2011-1'}]

Answer 3

Your code is unnecessarily complicated. The API only needs a User-Agent to work properly.

You can simplify it as follows:

from requests import post as POST
URL = "https://veiculos.fipe.org.br/api/veiculos/ConsultarAnoModelo"
UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 15_7_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.0 Safari/605.1.15"
DATA = [
 ("codigoModelo", 6906),
 ("codigoTabelaReferencia", 327),
 ("codigoTipoVeiculo", 1),
 ("CodigoMarca", 189),
]
HEADERS = {"User-Agent": UA}
with POST(URL, data=DATA, headers=HEADERS) as response:
 response.raise_for_status()
 print(response.json())

Output:

[{'Label': '2016 Gasolina', 'Value': '2016-1'}, {'Label': '2014 Gasolina', 'Value': '2014-1'}]