Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Commit ad82d46

Browse files
authored
正式环境支持多种方法运行server (#138)
* update dockerfile and build.yaml * update run app mode in prod - update readme * fix alpine python install gevent * fix #138 
1 parent f37f3ce commit ad82d46

File tree

6 files changed

+68
-20
lines changed

6 files changed

+68
-20
lines changed

‎Dockerfile

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
FROM python:3.7-alpine AS build
22
COPY requirements.txt .
33
RUN apk update &&\
4-
apk add --no-cache gcc g++ libffi-dev openssl-dev libxml2-dev libxslt-dev &&\
4+
apk add --no-cache gcc g++ libffi-dev openssl-dev libxml2-dev libxslt-dev build-base musl-dev &&\
5+
pip install -U pip &&\
56
pip install --timeout 30 --user --no-cache-dir --no-warn-script-location -r requirements.txt
67

78
FROM python:3.7-alpine

‎README.md

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ cd ProxyPool
3838

3939
安装方法自行搜索即可。
4040

41+
官方 Docker Hub 镜像:[germey/proxypool](https://hub.docker.com/r/germey/proxypool)
42+
4143
### 常规方式
4244

4345
常规方式要求有 Python 环境、Redis 环境,具体要求如下:
@@ -100,25 +102,20 @@ docker-compose -f build.yml up
100102
设置 host、port、password,如果 password 为空可以设置为空字符串,示例如下:
101103

102104
```shell script
103-
export REDIS_HOST='localhost'
104-
export REDIS_PORT=6379
105-
export REDIS_PASSWORD=''
106-
export REDIS_DB=0
105+
export PROXYPOOL_REDIS_HOST='localhost'
106+
export PROXYPOOL_REDIS_PORT=6379
107+
export PROXYPOOL_REDIS_PASSWORD=''
108+
export PROXYPOOL_REDIS_DB=0
107109
```
108110

109111
或者只设置连接字符串:
110112

111113
```shell script
112-
export REDIS_CONNECTION_STRING='redis://[password]@host:port/db'
113-
```
114-
115-
如果没有密码也要设置为:
116-
117-
```shell script
118-
export REDIS_CONNECTION_STRING='redis://@host:port/db'
114+
export PROXYPOOL_REDIS_CONNECTION_STRING='redis://localhost'
119115
```
120116

121-
这里连接字符串的格式需要符合 `redis://[password]@host:port/db` 的格式,注意不要遗漏 `@`。
117+
这里连接字符串的格式需要符合 `redis://[:password@]host[:port][/database]` 的格式,
118+
中括号参数可以省略,port默认是6379,database默认是0,密码默认为空。
122119

123120
以上两种设置任选其一即可。
124121

@@ -233,6 +230,8 @@ get random proxy 116.196.115.209:8080
233230

234231
- APP_ENV:运行环境,可以设置 dev、test、prod,即开发、测试、生产环境,默认 dev
235232
- APP_DEBUG:调试模式,可以设置 true 或 false,默认 true
233+
- APP_PROD_METHOD: 正式环境启动应用方式,默认是`gevent`,
234+
可选:`tornado`,`meinheld`(分别需要安装tornado或meinheld模块)
236235

237236
### Redis 连接
238237

‎proxypool/processors/server.py

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
from flask import Flask, g
22
from proxypool.storages.redis import RedisClient
3-
from proxypool.setting import API_HOST, API_PORT, API_THREADED
3+
from proxypool.setting import API_HOST, API_PORT, API_THREADED, IS_DEV
44

55

66
__all__ = ['app']
77

88
app = Flask(__name__)
9+
if IS_DEV:
10+
app.debug = True
911

1012

1113
def get_conn():
@@ -46,8 +48,9 @@ def get_proxy_all():
4648
conn = get_conn()
4749
proxies = conn.all()
4850
proxies_string = ''
49-
for proxy in proxies:
50-
proxies_string += str(proxy) + '\n'
51+
if proxies:
52+
for proxy in proxies:
53+
proxies_string += str(proxy) + '\n'
5154

5255
return proxies_string
5356

‎proxypool/scheduler.py

Lines changed: 40 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
from proxypool.processors.server import app
44
from proxypool.processors.getter import Getter
55
from proxypool.processors.tester import Tester
6-
from proxypool.setting import CYCLE_GETTER, CYCLE_TESTER, API_HOST, API_THREADED, API_PORT, ENABLE_SERVER, \
6+
from proxypool.setting import CYCLE_GETTER, CYCLE_TESTER, API_HOST, \
7+
API_THREADED, API_PORT, ENABLE_SERVER, IS_PROD, APP_PROD_METHOD, \
78
ENABLE_GETTER, ENABLE_TESTER, IS_WINDOWS
89
from loguru import logger
910

@@ -56,7 +57,42 @@ def run_server(self):
5657
if not ENABLE_SERVER:
5758
logger.info('server not enabled, exit')
5859
return
59-
app.run(host=API_HOST, port=API_PORT, threaded=API_THREADED)
60+
if IS_PROD:
61+
if APP_PROD_METHOD == 'gevent':
62+
try:
63+
from gevent.pywsgi import WSGIServer
64+
except ImportError as e:
65+
logger.exception(e)
66+
else:
67+
http_server = WSGIServer((API_HOST, API_PORT), app)
68+
http_server.serve_forever()
69+
70+
elif APP_PROD_METHOD == 'tornado':
71+
try:
72+
from tornado.wsgi import WSGIContainer
73+
from tornado.httpserver import HTTPServer
74+
from tornado.ioloop import IOLoop
75+
except ImportError as e:
76+
logger.exception(e)
77+
else:
78+
http_server = HTTPServer(WSGIContainer(app))
79+
http_server.listen(API_PORT)
80+
IOLoop.instance().start()
81+
82+
elif APP_PROD_METHOD == "meinheld":
83+
try:
84+
import meinheld
85+
except ImportError as e:
86+
logger.exception(e)
87+
else:
88+
meinheld.listen((API_HOST, API_PORT))
89+
meinheld.run(app)
90+
91+
else:
92+
logger.error("unsupported APP_PROD_METHOD")
93+
return
94+
else:
95+
app.run(host=API_HOST, port=API_PORT, threaded=API_THREADED)
6096

6197
def run(self):
6298
global tester_process, getter_process, server_process
@@ -71,13 +107,13 @@ def run(self):
71107
if ENABLE_GETTER:
72108
getter_process = multiprocessing.Process(
73109
target=self.run_getter)
74-
logger.info(f'starting getter, pid{getter_process.pid}...')
110+
logger.info(f'starting getter, pid{getter_process.pid}...')
75111
getter_process.start()
76112

77113
if ENABLE_SERVER:
78114
server_process = multiprocessing.Process(
79115
target=self.run_server)
80-
logger.info(f'starting server, pid{server_process.pid}...')
116+
logger.info(f'starting server, pid{server_process.pid}...')
81117
server_process.start()
82118

83119
tester_process and tester_process.join()

‎proxypool/setting.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,12 @@
2222
APP_PROD = IS_PROD = APP_ENV == PROD_MODE
2323
APP_TEST = IS_TEST = APP_ENV == TEST_MODE
2424

25+
# Which WSGI container is used to run applications
26+
# - gevent: pip install gevent
27+
# - tornado: pip install tornado
28+
# - meinheld: pip install meinheld
29+
APP_PROD_METHOD = env.str('APP_PROD_METHOD', "gevent").lower()
30+
2531
# redis host
2632
REDIS_HOST = env.str('PROXYPOOL_REDIS_HOST',
2733
env.str('REDIS_HOST', '127.0.0.1'))

‎requirements.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,6 @@ redis==3.5.3
1111
lxml==4.6.5
1212
fake_headers==1.0.2
1313
maxminddb_geolite2==2018.703
14+
gevent>=21.1.0
15+
tornado>=6.0
16+
meinheld>=1.0.0

0 commit comments

Comments
(0)

AltStyle によって変換されたページ (->オリジナル) /