Message158418
| Author |
Robert.Elsner |
| Recipients |
Robert.Elsner |
| Date |
2012年04月16日.12:10:11 |
| SpamBayes Score |
-1.0 |
| Marked as misclassified |
Yes |
| Message-id |
<1334578212.47.0.0704752159065.issue14596@psf.upfronthosting.co.za> |
| In-reply-to |
| Content |
When unpacking multiple files with _variable_ length, struct unpack leaks massive amounts of memory. The corresponding functions from numpy (fromfile) or the array (fromfile) standard lib module behave as expected.
I prepared a minimal testcase illustrating the problem on
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
[GCC 4.4.5] on linux2
This is a severe limitation when reading big files where performance is critical. The struct.Struct class does not display this behavior. Note that the variable length of the buffer is necessary to reproduce the problem (as is usually the case with real data files).
I suspect this is due to some internal buffer in the struct module not being freed after use.
I did not test on later Python versions, but could not find a related bug in the tracker. |
|
History
|
|---|
| Date |
User |
Action |
Args |
| 2012年04月16日 12:10:12 | Robert.Elsner | set | recipients:
+ Robert.Elsner |
| 2012年04月16日 12:10:12 | Robert.Elsner | set | messageid: <1334578212.47.0.0704752159065.issue14596@psf.upfronthosting.co.za> |
| 2012年04月16日 12:10:11 | Robert.Elsner | link | issue14596 messages |
| 2012年04月16日 12:10:11 | Robert.Elsner | create |
|