homepage

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: ftplib: Add client-side SSL session resumption
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Florian Wickert, Mark.Ribau, Ye.Wang, alex, christian.heimes, dstufft, giampaolo.rodola, janssen, larsschellhas, pitrou, python-dev, robr, xiang.zhang
Priority: low Keywords: patch

Created on 2013年11月05日 04:02 by Ye.Wang, last changed 2022年04月11日 14:57 by admin.

Files
File name Uploaded Description Edit
ftplib-FTPS-bug.txt Ye.Wang, 2013年11月05日 04:02 Python source and command prompt input/output
reuse_session.diff Alex Warhawk, 2015年10月03日 11:35 proof-of-concept session reuse review
implement_ssl_session_reuse.patch Alex Warhawk, 2015年10月07日 11:51 Patch which implements SSL session reuse and fixes corresponding FTP_TLS issue review
implement_ssl_session_reuse_3.6.patch Alex Warhawk, 2015年10月08日 09:40 Patch which implements SSL session reuse and fixes corresponding FTP_TLS issue targeted at 3.6 review
SSLSession-support.patch christian.heimes, 2016年09月04日 15:27 review
sesstime.py christian.heimes, 2016年09月05日 06:23
SSLSession-support-2.patch christian.heimes, 2016年09月05日 20:19 review
SSL-client-side-SSL-session-resumption.patch christian.heimes, 2016年09月08日 21:25 review
ftplib_session.patch robr, 2017年01月24日 15:12
Messages (27)
msg202193 - (view) Author: Ye Wang (Ye.Wang) Date: 2013年11月05日 04:02
According to RFC4217 (Securing FTP with TLS, aka the FTPS spec), 
http://tools.ietf.org/html/rfc4217.html#section-10.2
" It is reasonable for the server to insist that the data connection
 uses a TLS cached session. This might be a cache of a previous data
 connection or of a cleared control connection. If this is the reason
 for the refusal to allow the data transfer, then the '522' reply
 should indicate this.
 Note: This has an important impact on client design, but allows
 servers to minimize the cycles used during TLS negotiation by
 refusing to perform a full negotiation with a previously
 authenticated client."
It appears that vsftpd server implemented exactly that by enforcing the "SSL session reuse between the control and data connection".
http://scarybeastsecurity.blogspot.com/2009/02/vsftpd-210-released.html
Looking at the source of Python core library ftplib.py, there isn't any regard to the idea of SSL session reuse between data connection vs. control connection (correct me if I am wrong here. I've tried FTP_TLS.transfercmd(cmd[, rest])¶, didn't work). 
This issue is well documented on other FTP clients that supports FTPS, I.E. WinSCP: http://winscp.net/tracker/show_bug.cgi?id=668
See test log file attached. A vsftpd server with "require_ssl_reuse" set to true in vsftpd.conf would do the trick and can be reproduced.
msg216674 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2014年04月17日 08:40
Interesting, I wasn't aware of this FTP(S) feature.
Unfortunately RFC-4217 really doesn't say much about how this should be done but it definitively looks like something worth having.
AFAIU this looks like something which should be implemented by servers though, not clients.
msg216679 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014年04月17日 09:42
Yuck. Is there a public FTP server available somewhere with this "feature"?
msg216680 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014年04月17日 09:57
The RFC is unhelpfully lousy. It's not enough to process a "522" error, since that can be triggered for different reasons. You also somehow have to interpret the error text to detect that session reuse is indeed mandated by the server.
Regardless, to progress with this we would first need to implement client-side SSL session reuse, which necessitates a bunch of additional APIs (since which session is to be reused is a decision made by user code), and a new opaque type to carry SSL_SESSION objects...
(see issue #8106)
msg225078 - (view) Author: Mark Ribau (Mark.Ribau) Date: 2014年08月08日 16:29
Adding Python v2.7 as also exhibiting this behavior. 
Some people over on Stack Overflow have done some things to work around the issue via subclassing, but I'm not sure their solutions are "correct", so much as have useful side effects. (For example, when only the server has a key/cert and the client does not, how is that handled for reuse?)
http://stackoverflow.com/questions/12164470/python-ftp-tls-connection-issue 
msg252205 - (view) Author: Alex Warhawk (Alex Warhawk) Date: 2015年10月03日 11:35
I encountered this problem recently and could not find a fix, so i tried fixing it myself.
Note that the patch attached is my first contribution to cpython as well as the first time I used the C extension mechanism. Therefore I do not consider the patch polished enough to be just merged upstream.
Maybe it helps in solving this issue.
The attached patch is based on:
changeset: 79113:ec373d762213
branch: 2.7
msg252467 - (view) Author: Alex Warhawk (Alex Warhawk) Date: 2015年10月07日 11:51
Based on the proof-of-concept patch I submitted a few days ago I have built a more sophisticated patch. Please review it and let me know about necessary changes.
msg252468 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2015年10月07日 11:55
This is supposed to be a new feature hence the patch should be targeted against Python 3.6, definitively not 2.7.
msg252527 - (view) Author: Alex Warhawk (Alex Warhawk) Date: 2015年10月08日 09:40
I have re-targeted the patch for 3.6. It is not a 1 to 1 port of the prior one, but quite similar.
msg252530 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2015年10月08日 10:35
Thanks for your patch. There might be a simpler way. By default a SSLContext only caches server sessions. You can enable client session caching with:
 SSL_CTX_set_session_cache_mode(ctx, SSL_SESS_CACHE_CLIENT)
This may be sufficient for FTP over TLS since both sockets are created from the same context.
 
The new patch has a flaw. With the new SSLSession object a user could attempt to reuse a SSLSession with a different SSLContext. That's going to break OpenSSL.
From SSL_set_session(3)
NOTES
 SSL_SESSION objects keep internal link information about the session cache list, when being inserted into one SSL_CTX object's session cache. One SSL_SESSION object, regardless of its reference count, must therefore only be used with one SSL_CTX object (and the SSL objects created from this SSL_CTX object).
msg252543 - (view) Author: Alex Warhawk (Alex Warhawk) Date: 2015年10月08日 14:57
Thanks for the heads up Christian I'll try enabling client session caching. If this does not work I'll try to adapt the patch to only allow session reusing within the same context.
msg252635 - (view) Author: Alex Warhawk (Alex Warhawk) Date: 2015年10月09日 18:51
Even after enabling client cache one still has to call SSL_set_session. See documentation of SSL_CTX_set_session_cache_mode point SSL_SESS_CACHE_CLIENT.
I started thinking about not exposing a SSL_SESSION object to the user but rather extending wrap_socket to take an already established socket as argument and use that socket's session object. This way I can ensure that both sockets share the same SSL context
I am not really convinced by this idea myself, what do you think about this? Any better ideas?
msg274367 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016年09月04日 15:27
Here is my take on the SSLSession feature. The patch provides a SSLSession type, SSLSocket.session getter/setter and SSLSocket.session_reused getter. The setter makes sure that the session can only set for client sockets from the same SSLContext and before handshake. Tests and documentation need some improvements.
https://github.com/tiran/cpython/commits/feature/openssl_session 
msg274391 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016年09月05日 06:23
Performance improvements with session resumption is quite noticeable. I see between 17 and 21% improvement for 10 requests GET http://pypi.python.org/pypi in a simple benchmark:
session resumption: 1.264sec
no session: 1.535sec
 82.4%
msg274394 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2016年09月05日 07:51
Patch LGTM. But one thing is that every time it returns a new instance of SSL.Session. That means ssl_sock.session == ssl_sock.session will always return False right now. Is it useful to make it comparable?
msg274396 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016年09月05日 10:33
Xiang, good point! I have added richcompare to SSLSession (based on session id). My branch on github implements a couple more fixes and improvements.
msg274417 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016年09月05日 18:26
Note to future me:
Don't forget to take care of X.509 client authentication. A server is allowed to bypass client cert validation when a SSL session is resumed. SSLContext.load_cert_chain() should invalidate session caches. (CVE-2016-5419 https://curl.haxx.se/docs/adv_20160803A.html)
msg274893 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016年09月07日 21:03
Session resumption is currently broken in OpenSSL 1.1.0, https://github.com/openssl/openssl/issues/1550 
msg275161 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016年09月08日 21:25
This patch implements a workaround for OpenSSL 1.1.0.
msg275703 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016年09月10日 21:45
New changeset 6f2644738876 by Christian Heimes in branch 'default':
Issue #19500: Add client-side SSL session resumption to the ssl module.
https://hg.python.org/cpython/rev/6f2644738876 
msg275704 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016年09月10日 21:46
I have committed the feature with rudimentary documentation. I will provide more documentation and an example before 3.6.0b2.
msg286190 - (view) Author: Rob Reilink (robr) Date: 2017年01月24日 15:12
With this code in place, ftplib should / could also be updated to support session resumption. This would fix bugs with connections to FTP servers that require session resumption [1], [2]
In ftplib.FTP_TLS.ntransfercmd, just add a reference to the current session in the wrap_socket call (maybe make this an option to do session resumption or not; I don't know if it could break something)
Proposed patch is attached.
[1] http://stackoverflow.com/questions/14659154/ftpes-session-reuse-required
[2] https://forum.filezilla-project.org/viewtopic.php?t=36903 
msg312885 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018年02月26日 08:24
It's now an ftplib issue.
It's too late to land a new feature in 3.7 because we have reached feature freeze.
msg370001 - (view) Author: Florian Wickert (Florian Wickert) Date: 2020年05月26日 15:21
Using vsftpd 3.0.3 with ssl_enable=YES and require_ssl_reuse=YES I get the following error when I try to upload a file with conn.storbinary():
FTP command: Client "192.168.178.115", "STOR something.bin"
FTP response: Client "192.168.178.115", "150 Ok to send data."
FTP response: Client "x.x.x.x", "150 Ok to send data."
DEBUG: Client "x.x.x.x", "DATA connection terminated without SSL shutdown. Buggy client! Integrity of upload cannot be asserted."
FTP response: Client "x.x.x.x", "426 Failure reading network stream."
This also happens when SSL reuse is disabled on the server side.
Uploading with FileZilla works and there is basically no difference in the log until the error happens.
msg377412 - (view) Author: Lars Schellhas (larsschellhas) Date: 2020年09月23日 16:58
Excuse me, but why is this issue still open and unfixed? There are already proposed fixes and this issue has been around for nearly 7 years now. 
Filezilla has forwarded the responsibility to us for this issue (https://trac.filezilla-project.org/ticket/10700), so please, let's just fix it.
msg377413 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020年09月23日 18:23
I have provided necessary infrastructure for TLS 1.2 session resumption in the ssl module a couple of releases ago. But nobody has taken ownership of this ftplib issue and provided a full fix with regression tests and documentation update.
If you like to see ftplib enhanced to support session resumption, please provide a PR with regression tests. Demanding a fix won't accelerate work on the issue.
msg377419 - (view) Author: Lars Schellhas (larsschellhas) Date: 2020年09月23日 19:34
@Christian Heimes, you are absolutely right. I'm sorry if I came off rude. 
Actually, I just had a rough day at work with the end of it being the realisation that this missing fix would solve my day-filling issue from today.
Furthermore, I've actually returned here, just to read through the whole thread again, hoping to find a way to contribute to it's resolution.
However, this is my first time contributing to Python, but I'm eager to learn and dive into the necessary resources.
Could you give me a hint about where to start?
History
Date User Action Args
2022年04月11日 14:57:53adminsetgithub: 63699
2020年09月23日 19:34:39larsschellhassetmessages: + msg377419
2020年09月23日 18:23:35christian.heimessetstage: patch review -> needs patch
messages: + msg377413
versions: + Python 3.10, - Python 3.8
2020年09月23日 17:26:50Stephen Ashsetnosy: - Stephen Ash
2020年09月23日 16:58:06larsschellhassetnosy: + larsschellhas
messages: + msg377412
2020年05月26日 15:21:56Florian Wickertsetnosy: + Florian Wickert
messages: + msg370001
2018年02月26日 08:24:16christian.heimessetpriority: high -> low
title: Add client-side SSL session resumption -> ftplib: Add client-side SSL session resumption
versions: + Python 3.8, - Python 3.6
messages: + msg312885

assignee: christian.heimes ->
components: - Documentation, SSL
2017年01月24日 15:12:17robrsetfiles: + ftplib_session.patch
nosy: + robr
messages: + msg286190

2016年09月15日 07:49:05christian.heimessetcomponents: + SSL
2016年09月10日 21:46:20christian.heimessetpriority: normal -> high
assignee: christian.heimes
messages: + msg275704

components: + Documentation
2016年09月10日 21:45:02python-devsetnosy: + python-dev
messages: + msg275703
2016年09月09日 06:47:38Alex Warhawksetnosy: - Alex Warhawk
2016年09月09日 00:23:58Stephen Ashsetnosy: + Stephen Ash
2016年09月08日 23:09:48christian.heimeslinkissue25437 superseder
2016年09月08日 21:25:28christian.heimessetfiles: + SSL-client-side-SSL-session-resumption.patch

messages: + msg275161
2016年09月08日 15:28:50christian.heimeslinkissue8106 superseder
2016年09月08日 15:25:17christian.heimeslinkissue24542 superseder
2016年09月07日 21:03:12christian.heimessetmessages: + msg274893
2016年09月05日 20:19:11christian.heimessetfiles: + SSLSession-support-2.patch
2016年09月05日 18:26:27christian.heimessetmessages: + msg274417
2016年09月05日 10:33:08christian.heimessetmessages: + msg274396
2016年09月05日 07:51:11xiang.zhangsetnosy: + xiang.zhang
messages: + msg274394
2016年09月05日 06:23:07christian.heimessetfiles: + sesstime.py

messages: + msg274391
2016年09月04日 15:29:08christian.heimessetnosy: + alex

type: behavior -> enhancement
stage: patch review
title: Error when connecting to FTPS servers not supporting SSL session resuming -> Add client-side SSL session resumption
2016年09月04日 15:27:36christian.heimessetfiles: + SSLSession-support.patch

messages: + msg274367
2015年10月09日 18:51:57Alex Warhawksetmessages: + msg252635
2015年10月08日 14:57:57Alex Warhawksetmessages: + msg252543
2015年10月08日 10:35:05christian.heimessetmessages: + msg252530
2015年10月08日 09:40:46Alex Warhawksetfiles: + implement_ssl_session_reuse_3.6.patch

messages: + msg252527
2015年10月07日 11:55:48giampaolo.rodolasetmessages: + msg252468
versions: + Python 3.6, - Python 2.7, Python 3.4, Python 3.5
2015年10月07日 11:51:46Alex Warhawksetfiles: + implement_ssl_session_reuse.patch

messages: + msg252467
2015年10月03日 11:35:55Alex Warhawksetfiles: + reuse_session.diff

nosy: + Alex Warhawk
messages: + msg252205

keywords: + patch
2014年08月10日 23:41:57pitrousetversions: + Python 3.4
2014年08月08日 16:29:48Mark.Ribausetnosy: + Mark.Ribau

messages: + msg225078
versions: + Python 2.7
2014年04月17日 09:57:05pitrousetnosy: + janssen, christian.heimes, dstufft
messages: + msg216680
2014年04月17日 09:42:46pitrousetnosy: + pitrou
messages: + msg216679
2014年04月17日 08:40:06giampaolo.rodolasetnosy: + giampaolo.rodola

messages: + msg216674
versions: - Python 3.1, Python 2.7, Python 3.2, Python 3.3, Python 3.4
2013年11月05日 04:02:10Ye.Wangcreate

AltStyle によって変換されたページ (->オリジナル) /