A complete-auto IP pool, provider IP location visualization.
并行网络爬虫IP池自动筛选系统
This project provide a web API for auto Proxy Pool, automatically crawl IP proxy from xici.com, 66ip.cn, kuaidaili.com and some GitHub opensource ip proxy project. Once you install this project on your Linux server, you can get a dynamic proxy from your server port 5555.
Random select a Available IP from database.
Find Accurate IP location
- Clone this project
git clone https://github.com/KylinC/SmartProxyPool.git
- install redis and other dependent packages (sever)
// install redis
sudo apt-get install redis-server
//install packages
cd SmartProxy
pip install -r requirements.txt
- Change the redis' password to yours (SmartProxy/SmartProxy/config.py)
# Scheduler Switch TESTER_ENABLED = True GETTER_ENABLED = True API_ENABLED = True # Check Loop TESTER_CYCLE = 20 # CRAWLER Loop GETTER_CYCLE = 300 # API Configuration API_HOST = '127.0.0.1' API_PORT = 5000 ##### Flask Configuration # Aim Website TEST_URL = 'http://www.baidu.com' ##### Aim Website you want to crawler # Redis Database locstion REDIS_HOST = '127.0.0.1' ###### Support Your Remote Redis Database # Redis port REDIS_PORT = 6379 # Redis password,default = None REDIS_PASSWORD = "YourPassword" ##### Change to Your Redis Password REDIS_KEY = 'proxies' # selection initial ruler MAX_SCORE = 100 MIN_SCORE = 0 INITIAL_SCORE = 10 # selection parameter RANDOM_INTERVAL = [90, 100] # redis mox data number POOL_UPPER_THRESHOLD = 50000 # accept response code VALID_STATUS_CODES = [200, 302] # ip test batch BATCH_TEST_SIZE = 10 # baiduMap AK AK = "YourAK" #### Change to your Baidu Map API AK
- turn on your redis server(refers to SmartProxy/instructions)
- Run it on your server
python run.py
View in Browser(recommend: Google Chrome) at http://YourServerIP:YourPort/
Access in WebSpider Programm at http://YourServerIP:YourPort/random
According to Python3WebSpider written by Qingcai Cui.