Python: HTTPS using a proxy server with requests

Question

I'm using MyIPHide. I downloaded their client software, installed it and have the service turned on.

I can access https websites fine with a browser but I cannot use requests to get the pages

This works:

import requests
IP=requests.get('http://api.ipify.org').text
proxyDict = { "http" : IP,
 "https" : IP
 }
url='http://www.cnn.com'
r=requests.get(url,proxies=proxyDict)

This doesn't:

url='https://www.cnn.com'
r=requests.get(url,proxies=proxyDict)

only difference is http vs https

here is the traceback:

File "C:\Python27\lib\site-packages\requests\adapters.py", line 502, in send
 raise ProxyError(e, request=request)
ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))

I've tried other https websites, they all don't work.

I have also emailed support at MyIPHide. They said all proxies support https, which is true when I use a browser only.

One work around that works is if I use Selenium and get the page, then use driver.page_source for text.

It's not a proxy server problem because I have bought a private proxy server address through sslprivateproxy.com and put in the IP and port and I still get the same errors.

I'm using Python 2.7.15 and requests 2.20.1. Non proxy use of requests works, ie:

import requests
url='https://www.cnn.com'
r=requests.get(url)
>>> r
<Response [200]>
>>>

Also tried python 3.6 with requests 2.20.1 --> same results.

• KC.: did you add the port? {"https":"ip:port"}. Or try {"https":"183.88.219.163:57457"}
• jason: i don't think you need to. there is no port specified.
• KC.: If my proxy work for you that means the issue occured on your proxy
• jason: @kcorlidy. Thanks for the proxy. It looks like it also doesn't work. ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003CD9208>: Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it',)))
• KC.: yes, not all nodes work for me too. But also you can find one on free-proxy-list.net. If you still go wrong. Tell me your platform,Python version and requests version, in order to i can get same error as you then find a solution.

Answer 1

On requests it based on PoolManager size, so if you want to have more connection in pool you can reset the size of HTTPAdapter(). The way like that(to ensure if it works you can set to 0).

import requests
from requests.adapters import HTTPAdapter
proxy = {"http":"118.174.233.31:51726",
 "https":"43.254.132.86:50659"} # free proxy,often need to modify
with requests.Session() as se:
 # pool_connections=pool_maxsize=0 -> pool closed
 se.mount('https://', HTTPAdapter(pool_connections=100,pool_maxsize=100)) 
 se.mount('http://', HTTPAdapter(pool_connections=100,pool_maxsize=100))
 print(se.adapters["https://"]._pool_maxsize)
 #print(se.get("https://github.com"))
 print(se.get('http://api.ipify.org',proxies=proxy).text)
#100
#118.174.233.31

• KC.: Ran on Windows10, Python27, requests-2.20.1
• jason: Thanks for trying, but I still get the same error when I make a page request. url='https://www.cnn.com' r=se.get(url,proxies=proxy) gives me ProxyError: HTTPSConnectionPool(host='www.cnn.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
• KC.: what about your requests_version, and can you access website without https proxy?
• jason: yes I can. requests == 2.18.4. I just upgraded to 2.20.1. Same error. I'm running python 2.7.15