homepage

I'm trying to create a CAB file containing about 69MB of data, in 4555 files. msilib fails in CAB.commit():
$ python build-msi-installer.py /c/UpLib/1.7.9 ~/uplib 1.7.9 ./uplib-1.7.9.msi
c:\Documents and Settings\wjanssen\uplib\win32\uplib-1.7.9.msi
install_location c:/UpLib/1.7.9
c:\docume~1\wjanssen\locals~1\temp\tmpu5dwz6 68943045
gc: collecting generation 0...
gc: objects in each generation: 670 702 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 707 1364 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 699 2031 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 794 2680 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 730 3373 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 765 4018 8255
gc: done, 0.0000s elapsed.
gc: collecting generation 0...
gc: objects in each generation: 741 4697 8255
gc: done, 0.0000s elapsed.
(1, 4555, None, '#prereqs', None, None)
Traceback (most recent call last):
 File "build-msi-installer.py", line 780, in <module>
 p.run()
 File "build-msi-installer.py", line 243, in run
 self.add_files()
 File "build-msi-installer.py", line 312, in add_files
 cab.commit(self.db)
 File "c:\Python26\lib\msilib\__init__.py", line 223, in commit
 [(1, self.index, None, "#"+self.name, None, None)])
 File "c:\Python26\lib\msilib\__init__.py", line 97, in add_data
 v = db.OpenView("SELECT * FROM `%s`" % table)
_msi.MSIError: 1: 2229 2: c:\Documents and Settings\wjanssen\uplib\win32\uplib-1.7.9.msi 3: Media 4: SELECT * FROM `Media` 
gc: collecting generation 2...
gc: objects in each generation: 416 5351 8254
gc: done, 0.0000s elapsed.
$
(I added a bit of code to print out the Media table record, and the size of the file created with FCICreate.)
This works fine if I choose a slightly smaller subset, such as /c/UpLib/1.7.9/lib, for instance. So I'm guessing this is a memory problem; at some point there's just not enough memory to open the MSI file and load the Media table.
I opened the MSI file with orca and sure enough, there is no Media table in it. Not sure how to take that, though; perhaps the table only appears after a record has been written to it.
Another reason to allow multiple CAB files in a single installer?

> Another reason to allow multiple CAB files in a single installer?
I'm still curious as to what the first reason is.

Ummm, just in case the packager wanted to.
In my case, I was putting the files which were registered as part of the installation in one CAB file, and another set of temporary files which were used by some of the installation scripts, but not part of the installation file set, in a second CAB file. I've since changed that second package to a zip file.

So, I subclassed msilib.CAB, and re-wrote commit() so that the CAB file is created in a different process, a la Tools/msi/msilib.py. Still have the same problem, though. So now I'm thinking it's not a memory problem, but I'm wondering if there's something about the length of the path names in the CAB file, or some such.

You could also try to commit the MSI file in-between, which may release memory.

Yes, I've tried that. No joy. Right now I'm trying an approach which
packages each top-level directory as a separate cab.
What I'm finding is that if I get up around 4200 files, it breaks,
regardless of the file sizes. Out of curiosity, how many files are in the
Python MSI file?
Bill
On Tue, Apr 27, 2010 at 11:37 PM, Martin v. Löwis <report@bugs.python.org>wrote:
>
> Martin v. Löwis <martin@v.loewis.de> added the comment:
>
> You could also try to commit the MSI file in-between, which may release
> memory.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue8552>
> _______________________________________
>

I've now been able to build my installer.
I applied Travis Oliphant's patch from http://bugs.python.org/issue2399 to Lib/msilib/__init__.py, then added a __del__ method to the Directory class:
 def __del__(self):
 if self._numfiles_wo_commit > 0:
 self.db.Commit()
 self.db = None
 self.keyfiles = None
 self.cab = None
 self.ids = None
This seems to release enough memory that I can deal with a larger number of files.
I don't think there's a good way to write a unit test for this, though. Success or failure will depend on the machine you're running it on, I think.

Another bit of info. It's the frequent commits that seem to fix the problem; when I comment those out of the __del__ method, it fails as before. I also notice that the finished installer is about twice the size of the two data files in it (the CAB file and my zip file). Seems like a lot of overhead.

Ah, found the size problem -- I was measuring something in 512 blocks not 1KB blocks. Never mind.

It looks like it turned out that there is nothing specific in this issue that isn't covered by issue 2399.