自建免费代理IP池
- 自动爬取互联网上公开的免费代理IP(目前已支持西刺代理、快代理、IP181)
- 周期性验证代理IP有效性
- 提供http接口获取可用IP
系统架构
- /db 数据库操作
- /schedule 后台周期任务
- /spider 爬虫
- /util 通用工具
- /web web服务
- /log 日志存储文件夹
- config.py 全局配置
- main.py 启动入口
- 下载整个项目
- 安装Python3
- 安装Python包
pip install -r requirements.txt
- 安装MySQL数据库
- 初始化建表 db/proxy.sql
- 修改配置文件 config.py
- 运行 main.py
| 参数名 |
类型 |
必填 |
参数位置 |
描述 |
默认值 |
| check_in_hour |
float |
否 |
urlParam |
代理最后验证时间(小时)以内 |
24 |
| response_time_in_second |
float |
否 |
urlParam |
代理响应时间(秒)以内 |
null |
| protocol |
string |
否 |
urlParam |
代理网络协议,http/https |
null |
| anonymity |
string |
否 |
urlParam |
代理匿名性,transparent/anonymous/high_anonymous |
null |
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import requests
url = "http://localhost:9999/get"
querystring = {"anonymity":"high_anonymous","response_time_in_second":"1.5"}
response = requests.request("GET", url, params=querystring)
print(response.json())
{
"ret": 0,
"data": {
"anonymity": "high_anonymous",
"check_time": "2017-12-20 13:55:17",
"country": "CN",
"export_address": [
"120.25.253.234"
],
"from": "快代理",
"grab_time": "2017-12-20 13:54:55",
"host": "120.25.253.234",
"port": "8118",
"protocol": "http",
"response_time": 1.45
}
}
| 参数名 |
类型 |
必填 |
参数位置 |
描述 |
默认值 |
| check_in_hour |
float |
否 |
urlParam |
代理最后验证时间(小时)以内 |
24 |
| response_time_in_second |
float |
否 |
urlParam |
代理响应时间(秒)以内 |
null |
| protocol |
string |
否 |
urlParam |
代理网络协议,http/https |
null |
| anonymity |
string |
否 |
urlParam |
代理匿名性,transparent/anonymous/high_anonymous |
null |
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import requests
url = "http://localhost:9999/get_all"
querystring = {"anonymity":"high_anonymous","response_time_in_second":"1.5","protocol":"https"}
response = requests.request("GET", url, params=querystring)
print(response.json())
{
"ret": 0,
"data": [
{
"anonymity": "high_anonymous",
"check_time": "2017-12-20 14:10:25",
"country": "CN",
"export_address": [
"118.114.77.47"
],
"from": "西刺代理",
"grab_time": "2017-12-20 14:09:36",
"host": "118.114.77.47",
"port": "8080",
"protocol": "https",
"response_time": 1.41
},
{
"anonymity": "high_anonymous",
"check_time": "2017-12-20 13:09:40",
"country": "CN",
"export_address": [
"119.29.178.21"
],
"from": "西刺代理",
"grab_time": "2017-12-14 16:17:52",
"host": "119.29.178.21",
"port": "8118",
"protocol": "https",
"response_time": 1.11
}
]
}
