homepage

I find that urllib fails on ftp if:
- I start more than 1 transfer, each in its own thread
- I let those finish
- I again start more than 1 transfer, each in its own thread
One of the latter xfers will fail with:
 [Errno ftp error] 200 Type set to A
I have attached sample code that demonstrates the problem.
Notes:
- I have seen the problem on more than one ftp server, using 
different sets of sample files.
- An explicit ";type=i" suffix on binary file URLs does not help. I have 
not tried text files with explicit "type=a".

OP supplied a detailed test. Needs confirmation.

I was able to get some error output with the code of the OP. However, I only saw the "opposite" message, such as:
Retrieval of 'ftp://ftp.gnome.org/pub/debian/dists/stable/main/source/Sources.bz2' failed with error: [Errno ftp error] 200 Switching to Binary mode.
I had to change the used FTP server and path because I had problems connecting to that server.
Even with the guidelines by the OP, I could trigger the bug only occasionally.
I converted the script to be executable with manual intervention (see attachment). Still, sometimes both files are also successfully downloaded in the second iteration (i. e. the "outer" iteration which downloads both files).
These results are from a test with Python 2.6.6. I plan to continue testing with the tip of the Python 2.7 branch.

> I converted the script to be executable with manual intervention (see attachment).
This should have been "without manual intervention". :)

I can confirm the bug for the Python 2.7 tip (changeset b11e7bc76d07) after using the script urllibftpbug-non-interactive.py.

After running the adapted test script three times for Python 3 tip (changeset c5b0585624ef), I didn't get an error message / exception.

The traceback and its context for the exception raised in Python 2.7 is:
...
################################################################################################### 99 %
################################################################################################### 99 %
#################################################################################################### 100 %
Done retrieving 'ftp://ftp.gnome.org/pub/debian/dists/stable/main/source/Sources.bz2'; read 4539023 bytes
Start retrieving 'ftp://ftp.gnome.org/pub/debian/dists/stable/main/source/Sources.bz2'
Start retrieving 'ftp://ftp.gnome.org/pub/debian/dists/stable/main/source/Sources.bz2'
Retrieval of 'ftp://ftp.gnome.org/pub/debian/dists/stable/main/source/Sources.bz2' failed with error: [Errno ftp error] 200 Switching to Binary mode.
Traceback (most recent call last):
 File "urllibftpbug-non-interactive.py", line 62, in _getTask
 self._fromFile = urllib.urlopen(self.fromURL)
 File "/home/schwa/sd/python/cpython/Lib/urllib.py", line 84, in urlopen
 return opener.open(url)
 File "/home/schwa/sd/python/cpython/Lib/urllib.py", line 205, in open
 return getattr(self, name)(url)
 File "/home/schwa/sd/python/cpython/Lib/urllib.py", line 548, in open_ftp
 (fp, retrlen) = self.ftpcache[key].retrfile(file, type)
 File "/home/schwa/sd/python/cpython/Lib/urllib.py", line 886, in retrfile
 conn = self.ftp.ntransfercmd(cmd)
 File "/home/schwa/sd/python/cpython/Lib/ftplib.py", line 326, in ntransfercmd
 host, port = self.makepasv()
 File "/home/schwa/sd/python/cpython/Lib/ftplib.py", line 304, in makepasv
 host, port = parse227(self.sendcmd('PASV'))
 File "/home/schwa/sd/python/cpython/Lib/ftplib.py", line 790, in parse227
 raise error_reply, resp
IOError: [Errno ftp error] 200 Switching to Binary mode.
Total bytes: 4539023
 0 %
 0 %
 0 %
...
I printed the traceback by adding traceback.print_exc() at the end of the _getTask method:
 except Exception, e:
 self._cleanup(e)
 traceback.print_exc()

New changeset a4d418e8010d by Senthil Kumaran in branch '2.7':
Fix closes issue1067702 The problem with close multiple ftp transfers were due cases where sockets/file were not closed immediately. Tightned those cases and failure is no longer observed.
http://hg.python.org/cpython/rev/a4d418e8010d 

Hi Senthil,
I don't yet understand what was going on before it resulted in the traceback. I also don't understand _why_ the patch fixes _this_ bug. (That's not to say it doesn't, but I think it's not obvious either. :-) )
Were you able to reproduce the exception with my attached script before you did the change? Here in the hotel I have a much faster internet connection than I had yesterday at the sprint (where lots of people shared the uplink), and now I can't reproduce the exception after running the test script three times, even _without_ having your change applied.

Hello Stefen, 
Yes, I was able to reproduce the issue without the patch and at the
same time, noticed that the problem was not observed in other branches
(py3k), then saw the differences where the possible socket close
delays can happen and fixed it. As it was multiple threads closely
being closed it was most probably due to delay in closing socket
connection that lead the problem in the first place.
If it is required, then manually this can be verified by setting up an
artificial delay in the network layer and trying multiple ftp
transfers. (No, I did not do that to verify. :))

The problem is reproducible on latest python 2.7 package (2.7.12).
I tried the same scenario on 2.7.10 and it worked fine. I am not sure if this issue can be reopened or should I create a new one?
In my case first transfer succeeds but second ftp transfer fails with the error:
[Errno ftp error] 200 Type set to I
I am using urllib.urlretrieve(url, local_path) to retrieve two files (one by one) from FTP server.