matplotlib-devel Mailing List for matplotlib (Page 2)

On Thu, Jan 27, 2011 at 12:57 PM, Pauli Virtanen <pa...@ik...> wrote:
> 2011年1月27日 12:39:48 -0500, Darren Dale wrote:
> [clip]
>> Me too. I just posted the latest version of the repository to
>> github.com/darrendale/matplotlib.git . Its ~42MB, but it has a bunch of
>> unreachable objects. As soon as we figure out how to git rid of them, I
>> think we will be ready to freeze the svn repo and wrap this up.
>
> Unreachable from where? How do you know there are unreachable
> objects?
>
> Note that the snippet
>
>  git fsck --unreachable HEAD $(git for-each-ref --format="%(objectname)" refs/heads)
>
> only checks for objects unreachable from branches (by definition,
> stuff under refs/heads). However, there's also other stuff under refs/:
> tags and hidden branches. Especially the postprocess.sh script hides
> some branches.
>
> To see all that is there, check the output from
>
>  git for-each-ref
Oh, I didn't understand what I was doing with the git fsck command.
Still, Even after removing the the largest blob in the repo with
run git filter-branch --index-filter \
 'git rm --cached --ignore-unmatch release/osx/matplotlib-0.98.5.tar.gz' \
 -- 750059aa09340^..
the blob still exists, but is not associated with a commit according to
git log --pretty=oneline -- release/osx/matplotlib-0.98.5.tar.gz
That blob accounts for 1/4 of the total size of the repo. It would be
nice to get rid of it, if possible.
Darren

to, 2011年01月27日 kello 13:44 -0500, Darren Dale kirjoitti:
[clip]
> Still, Even after removing the the largest blob in the repo with
> 
> run git filter-branch --index-filter \
> 'git rm --cached --ignore-unmatch release/osx/matplotlib-0.98.5.tar.gz' \
> -- 750059aa09340^..
>
> the blob still exists, but is not associated with a commit according to
> 
> git log --pretty=oneline -- release/osx/matplotlib-0.98.5.tar.gz
> 
> That blob accounts for 1/4 of the total size of the repo. It would be
> nice to get rid of it, if possible.
I think "git log" will show you only the current branch by default. Do
 git log --pretty=oneline --all -- release/osx/matplotlib-0.98.5.tar.gz
to get all branches, and do
 for branch in `git for-each-ref --format='%(refname)'`; do S=`git log --pretty=oneline $branch -- release/osx/matplotlib-0.98.5.tar.gz`; if test -n "$S"; then echo "$branch"; echo "$S"; fi; done
to see which refs have the commits containing it.
Similarly, git-filter-branch rewrites only the current branch unless
told otherwise. To filter everything, it's best to do
 git filter-branch --index-filter \
 	'git rm --cached --ignore-unmatch release/osx/matplotlib-0.98.5.tar.gz' \
 	-- `git for-each-ref --format="750059aa09340^..%(refname)"`
Note that all branches and tags should be filtered in the same way:
since rewriting changes the hashes of all following commits, you end up
with incompatible histories otherwise.
After that, I get down to 34 MB.
-- 
Pauli Virtanen

2011年1月27日 12:39:48 -0500, Darren Dale wrote:
[clip]
> Me too. I just posted the latest version of the repository to
> github.com/darrendale/matplotlib.git . Its ~42MB, but it has a bunch of
> unreachable objects. As soon as we figure out how to git rid of them, I
> think we will be ready to freeze the svn repo and wrap this up.
Unreachable from where? How do you know there are unreachable
objects?
Note that the snippet
 git fsck --unreachable HEAD $(git for-each-ref --format="%(objectname)" refs/heads)
only checks for objects unreachable from branches (by definition,
stuff under refs/heads). However, there's also other stuff under refs/:
tags and hidden branches. Especially the postprocess.sh script hides
some branches.
To see all that is there, check the output from
 git for-each-ref
-- 
Pauli Virtanen

On Thu, Jan 27, 2011 at 12:00 PM, John Hunter <jd...@gm...> wrote:
> On Thu, Jan 27, 2011 at 10:26 AM, Darren Dale <dsd...@gm...> wrote:
>> On Tue, Jan 25, 2011 at 11:37 PM, Jae-Joon Lee <lee...@gm...> wrote:
>>> And some of the biggest blobs are associated with svg and pdf files.
>>> It seems possible to remove these files (if needed) from the
>>> repository using "git-filter-branch" to further reduce the size, but
>>> I'm not sure if we need that.
>>
>> Some of this appears to be the result of some svn tags referencing the
>> contents of trunk/htdocs. It looks like I can cut 25 MB off the size
>> of the repo.
>
> That feels pretty good to me, considering that the compressed release
> tarball is 13MB. If we can get the whole history for twice that, then
> you must be getting pretty close to the limit.
Me too. I just posted the latest version of the repository to
github.com/darrendale/matplotlib.git . Its ~42MB, but it has a bunch
of unreachable objects. As soon as we figure out how to git rid of
them, I think we will be ready to freeze the svn repo and wrap this
up.

On Thu, Jan 27, 2011 at 10:26 AM, Darren Dale <dsd...@gm...> wrote:
> On Tue, Jan 25, 2011 at 11:37 PM, Jae-Joon Lee <lee...@gm...> wrote:
>> And some of the biggest blobs are associated with svg and pdf files.
>> It seems possible to remove these files (if needed) from the
>> repository using "git-filter-branch" to further reduce the size, but
>> I'm not sure if we need that.
>
> Some of this appears to be the result of some svn tags referencing the
> contents of trunk/htdocs. It looks like I can cut 25 MB off the size
> of the repo.
That feels pretty good to me, considering that the compressed release
tarball is 13MB. If we can get the whole history for twice that, then
you must be getting pretty close to the limit.
JDH

On Tue, Jan 25, 2011 at 11:37 PM, Jae-Joon Lee <lee...@gm...> wrote:
> And some of the biggest blobs are associated with svg and pdf files.
> It seems possible to remove these files (if needed) from the
> repository using "git-filter-branch" to further reduce the size, but
> I'm not sure if we need that.
Some of this appears to be the result of some svn tags referencing the
contents of trunk/htdocs. It looks like I can cut 25 MB off the size
of the repo.

On Wed, Jan 26, 2011 at 6:35 AM, Jae-Joon Lee <lee...@gm...> wrote:
> On Wed, Jan 26, 2011 at 7:28 PM, Pauli Virtanen <pa...@ik...> wrote:
>> The complete magic stanza is:
>>
>> git reflog expire --expire=0 --all
>> git prune
>> git repack -f -a -d
>> git gc --prune=0
>
> Wonderful!
> With this, I get about 40 MB!
How did you manage to do that? Using these steps (which have been
added to the postprocess.sh script):
---
run git filter-branch --index-filter \
 'git rm --cached --ignore-unmatch release/osx/matplotlib-0.98.5.tar.gz' \
 -- 750059aa09340^..
rm -Rf refs/original
rm -Rf logs
run git reflog expire --expire=0 --all
run git prune
run git repack -f -a -d --depth=250 --window=250
run git gc --prune=0
---
I only get down to 65 MB, at which point "git fsck --unreachable HEAD"
indicates a slew of unreachable blobs, trees, and commits, one of
which is:
3b8b6c010f8ce59afac1e811b1bbc3efc21b770a release/osx/matplotlib-0.98.5.tar.gz
yet "git log --pretty=oneline -- release/osx/matplotlib-0.98.5.tar.gz"
shows it is no longer associated with any commits. Rerunning the
"reflog/prune/repack/gc" block doesn't seem to help at this point.

On Wednesday, January 26, 2011, Darren Dale <dsd...@gm...> wrote:
> On Wed, Jan 26, 2011 at 7:44 AM, Darren Dale <dsd...@gm...> wrote:
>> Last night I noticed that, in the git repo, the commit messages
>> produced by svnmerge.py still contain a lot of svn-specific
>> information. Pauli's conversion script includes a step that filters
>> out two lines at the end of each commit containing some svn metadata,
>> but for svnmerge commits we still end up with:
>>
>>  Merged revisions 8933 via svnmerge from
>>  https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_maint
>>
>>  ........
>>   r8933 | weathergod | 2011年01月22日 10:35:26 -0600 (Sat, 22 Jan
>> 2011) | 3 lines
>>
>>   Fixing problem where reversed colormaps of
>> LinearSegmentedColormaps were not initialized properly.
>>   Thanks to LittleBigBrain for reporting and Friedrich Romstedt
>> for making the original patch.
>>  ........
>>
>> Do we live with this, or try to replace references to svn commits with
>> their git hash and references to svn branches with the git ones?
>
> I think I just convinced myself that replacing svn references with git
> hashes would be a Herculean task. One could generate a
> svn_rev:git_hash mapping, but then changing the commit message
> actually yields a new git hash, breaking the mapping. Its not an
> impossible task, but it would be a lot of work and difficult to check.
> Leaving the svn path and revision information will probably be more
> reliable. "git log --all" will therefore yield enough information that
> we can easily identify the provenance of a cherry pick and find it in
> the git history:
>
> commit adb1a7f67daf955a1af3a86d42b5181767c18819
> Author: Ben Root <ben...@gm...>
> Date:  Sat Jan 22 16:40:21 2011 +0000
>
>  Merged revisions 8933 via svnmerge from
>  https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_main
>
>  ........
>   r8933 | weathergod | 2011年01月22日 10:35:26 -0600 (2011年1月22日) | 3 line
>
>   Fixing problem where reversed colormaps of LinearSegmentedColormaps were n
>   Thanks to LittleBigBrain for reporting and Friedrich Romstedt for making t
>  ........
>
>  svn path=/trunk/matplotlib/; revision=8934
>
> commit 2fa57c710607496a93000bfb3191bdd422f518cc
> Author: Ben Root <ben...@gm...>
> Date:  Sat Jan 22 16:35:26 2011 +0000
>
>  Fixing problem where reversed colormaps of LinearSegmentedColormaps were not
>  Thanks to LittleBigBrain for reporting and Friedrich Romstedt for making the
>
>  svn path=/branches/v1_0_maint/; revision=8933
>
>
> Is this satisfactory?
>
> Darren
>
I think it is fine. Maybe a nice convenience script would be one that
would return a got hash for a svn revision number (and vice verse?).
With that, I would be more than happy with how the logs are.
Ben Root

On Wed, Jan 26, 2011 at 7:44 AM, Darren Dale <dsd...@gm...> wrote:
> Last night I noticed that, in the git repo, the commit messages
> produced by svnmerge.py still contain a lot of svn-specific
> information. Pauli's conversion script includes a step that filters
> out two lines at the end of each commit containing some svn metadata,
> but for svnmerge commits we still end up with:
>
>  Merged revisions 8933 via svnmerge from
>  https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_maint
>
>  ........
>   r8933 | weathergod | 2011年01月22日 10:35:26 -0600 (Sat, 22 Jan
> 2011) | 3 lines
>
>   Fixing problem where reversed colormaps of
> LinearSegmentedColormaps were not initialized properly.
>   Thanks to LittleBigBrain for reporting and Friedrich Romstedt
> for making the original patch.
>  ........
>
> Do we live with this, or try to replace references to svn commits with
> their git hash and references to svn branches with the git ones?
I think I just convinced myself that replacing svn references with git
hashes would be a Herculean task. One could generate a
svn_rev:git_hash mapping, but then changing the commit message
actually yields a new git hash, breaking the mapping. Its not an
impossible task, but it would be a lot of work and difficult to check.
Leaving the svn path and revision information will probably be more
reliable. "git log --all" will therefore yield enough information that
we can easily identify the provenance of a cherry pick and find it in
the git history:
commit adb1a7f67daf955a1af3a86d42b5181767c18819
Author: Ben Root <ben...@gm...>
Date: Sat Jan 22 16:40:21 2011 +0000
 Merged revisions 8933 via svnmerge from
 https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_main
 ........
 r8933 | weathergod | 2011年01月22日 10:35:26 -0600 (2011年1月22日) | 3 line
 Fixing problem where reversed colormaps of LinearSegmentedColormaps were n
 Thanks to LittleBigBrain for reporting and Friedrich Romstedt for making t
 ........
 svn path=/trunk/matplotlib/; revision=8934
commit 2fa57c710607496a93000bfb3191bdd422f518cc
Author: Ben Root <ben...@gm...>
Date: Sat Jan 22 16:35:26 2011 +0000
 Fixing problem where reversed colormaps of LinearSegmentedColormaps were not
 Thanks to LittleBigBrain for reporting and Friedrich Romstedt for making the
 svn path=/branches/v1_0_maint/; revision=8933
Is this satisfactory?
Darren

On 01/25/2011 06:53 PM, Eric Firing wrote:
[...]
>
> Darren,
>
> It looks like at least some of the problem is the origin/unit_support:
>
> efiring@manini:~/test/matplotlib.git.ddale$ git checkout origin/unit_support
> Checking out files: 100% (4514/4514), done.
> Note: checking out 'origin/unit_support'.
> [...]
> HEAD is now at 8d705be... refactoring, moved units conversion to
> units.UnitsManager
> efiring@manini:~/test/matplotlib.git.ddale$ ls
> course CVSROOT htdocs matplotlib scipy06 toolkits users_guide
>
> It appears to have branched trunk, not trunk/matplotlib. Junk it!
>
> I think that losing some bits of history from the git repo is entirely
> acceptable. The history is still in the svn repo, if anyone really
> needs to dig back into the earliest recorded origins of unit_support.
>
> Eric
Or, maybe a rule like this will work?
match /branches/unit_support/matplotlib/
 repository matplotlib
 branch unit_support
end match
(I don't know how significant the parentheses are; I am guessing they 
are for regular expressions, which are not needed here.)
Eric

On Wed, Jan 26, 2011 at 6:42 AM, Pauli Virtanen <pa...@ik...> wrote:
> 2011年1月26日 20:35:03 +0900, Jae-Joon Lee wrote:
>> On Wed, Jan 26, 2011 at 7:28 PM, Pauli Virtanen
>> <pa...@ik...> wrote:
>>> The complete magic stanza is:
>>>
>>> git reflog expire --expire=0 --all
>>> git prune
>>> git repack -f -a -d
>>> git gc --prune=0
>>
>> Wonderful!
>> With this, I get about 40 MB!
>
> Some additional compression can be obtained by passing also suitable
> values for the --window and --depth flags for git repack.
>
> http://metalinguist.wordpress.com/2007/12/06/the-woes-of-git-gc-aggressive-and-how-git-deltas-work/
Thank you all for the helpful feedback. I'll work on this again this evening.
Last night I noticed that, in the git repo, the commit messages
produced by svnmerge.py still contain a lot of svn-specific
information. Pauli's conversion script includes a step that filters
out two lines at the end of each commit containing some svn metadata,
but for svnmerge commits we still end up with:
 Merged revisions 8933 via svnmerge from
 https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_maint
 ........
 r8933 | weathergod | 2011年01月22日 10:35:26 -0600 (2011年1月22日) | 3 lines
 Fixing problem where reversed colormaps of
LinearSegmentedColormaps were not initialized properly.
 Thanks to LittleBigBrain for reporting and Friedrich Romstedt
for making the original patch.
 ........
Do we live with this, or try to replace references to svn commits with
their git hash and references to svn branches with the git ones? I
tried merging the v.1.0.x branch into master, which applied pretty
cleanly, with only a few minor merge conflicts:
$ git merge origin/v1.0.x
Auto-merging CHANGELOG
CONFLICT (content): Merge conflict in CHANGELOG
Auto-merging examples/api/quad_bezier.py
CONFLICT (content): Merge conflict in examples/api/quad_bezier.py
Auto-merging lib/matplotlib/__init__.py
CONFLICT (content): Merge conflict in lib/matplotlib/__init__.py
Auto-merging lib/matplotlib/axes.py
Auto-merging lib/matplotlib/backends/backend_macosx.py
Auto-merging lib/matplotlib/backends/backend_ps.py
Auto-merging lib/matplotlib/backends/backend_qt4.py
Auto-merging lib/matplotlib/tests/test_axes.py
Auto-merging lib/matplotlib/text.py
Auto-merging lib/matplotlib/ticker.py
CONFLICT (content): Merge conflict in lib/matplotlib/ticker.py
Auto-merging src/_gtkagg.cpp
Auto-merging src/_macosx.m
CONFLICT (content): Merge conflict in src/_macosx.m
Here is the CHANGELOG:
<<<<<<< HEAD
2011年01月13日 Added zdir and offset arguments to contourf3d to
 bring contourf3d in feature parity with contour3d. - BVR
2011年01月04日 Tag 1.0.1 for release at r8896
2011年01月03日 Added display of ticker offset to 3d plots. - BVR
2011年01月03日 Turn off tick labeling on interior subplots for
 pyplots.subplots when sharex/sharey is True. - JDH
2010年12月29日 Implment axes_divider.HBox and VBox. -JJL
=======
2011年01月04日 Tag 1.0.1 for release at r8896
>>>>>>> origin/v1.0.x

2011年1月25日 18:12:35 -0500, Darren Dale wrote:
[clip]
> Pauli: could I trouble you to have a look at my rules file, maybe you
> will notice something I overlooked?
> (https://github.com/darrendale/mpl2git/blob/master/matplotlib.rules) Any
> other ideas?
As an additional check, I'd suggest checking that the history graph
in each repository is simple. Find all root (parentless) commits:
 git rev-list --parents master | egrep "^[a-f0-9]{40}$" # for master branch
 git rev-list --parents --all | egrep "^[a-f0-9]{40}$" # all branches
There should be only one root commit, unless there is a very good
reason to have several.
If there are multiple root commits, this may indicate that
the history was splintered into several parts (e.g. due to complicated 
SVN moves), which needs manual grafting to connect the parts.
For perfectionists, I'd also suggest eyeballing the "gitk --all" DAG.
-- 
Pauli Virtanen

2011年1月26日 20:35:03 +0900, Jae-Joon Lee wrote:
> On Wed, Jan 26, 2011 at 7:28 PM, Pauli Virtanen
> <pa...@ik...> wrote:
>> The complete magic stanza is:
>>
>> git reflog expire --expire=0 --all
>> git prune
>> git repack -f -a -d
>> git gc --prune=0
> 
> Wonderful!
> With this, I get about 40 MB!
Some additional compression can be obtained by passing also suitable 
values for the --window and --depth flags for git repack.
http://metalinguist.wordpress.com/2007/12/06/the-woes-of-git-gc-aggressive-and-how-git-deltas-work/
	Pauli

On Wed, Jan 26, 2011 at 7:28 PM, Pauli Virtanen <pa...@ik...> wrote:
> The complete magic stanza is:
>
> git reflog expire --expire=0 --all
> git prune
> git repack -f -a -d
> git gc --prune=0
Wonderful!
With this, I get about 40 MB!
Regards,
-JJ

2011年1月26日 13:37:59 +0900, Jae-Joon Lee wrote:
[clip]
> I spend an hour to figure out how we can delete these unreachable
> objects. But it turned out that the answer seems to be simple.
> 
> $ git repack -ad
The complete magic stanza is:
git reflog expire --expire=0 --all
git prune
git repack -f -a -d
git gc --prune=0

On 01/25/2011 01:38 PM, Darren Dale wrote:
> On Tue, Jan 25, 2011 at 6:12 PM, Darren Dale<dsd...@gm...> wrote:
>> There is still an outstanding issue that must be taken care of before
>> we migrate. The conversion routines create a basemap repository out of
>> trunk/toolkits/basemap, and matplotlib repository out of
>> trunk/matplotlib. Still, the matplotlib repo (at
>> github.com/darrendale/matplotlib) is over 200 MB. One can search the
>> objects in the large packfile, and find that there are still
>> references to basemap data in the matplotlib repo. I don't know how it
>> got in there, nor how to remove it.
>
Darren,
It looks like at least some of the problem is the origin/unit_support:
efiring@manini:~/test/matplotlib.git.ddale$ git checkout origin/unit_support
Checking out files: 100% (4514/4514), done.
Note: checking out 'origin/unit_support'.
[...]
HEAD is now at 8d705be... refactoring, moved units conversion to 
units.UnitsManager
efiring@manini:~/test/matplotlib.git.ddale$ ls
course CVSROOT htdocs matplotlib scipy06 toolkits users_guide
It appears to have branched trunk, not trunk/matplotlib. Junk it!
I think that losing some bits of history from the git repo is entirely 
acceptable. The history is still in the svn repo, if anyone really 
needs to dig back into the earliest recorded origins of unit_support.
Eric
> I went through the exercise of identifying the largest blob, as
> described near the end of http://progit.org/book/ch9-7.html :
>
> $ git verify-pack -v
> objects/pack/pack-fa44ca56d7ec3964e562494f2fe08203143074bd.idx | sort
> -k 3 -n | tail -3
> 3b8b6c010f8ce59afac1e811b1bbc3efc21b770a blob 9154481 9089827 62749144
> 6328b70e665b58ed7f5aa1e110418cbb3facc07a blob 9331200 94297 156884507
> f784efc1518b10dff33673ad9a7a1ac3a7d107d5 blob 51399604 14333430 162328624
>
> $ git rev-list --objects --all | grep f784efc1518b10dff
> f784efc1518b10dff33673ad9a7a1ac3a7d107d5 toolkits/basemap/data/gshhs_h.txt
>
> This shell script is supposed to identify which commits have that blob
> in their tree (http://stackoverflow.com/questions/223678/git-which-commit-has-this-blob):
>
> ---
> #!/bin/sh
> obj_name="1ドル"
> shift
> git log "$@" --pretty=format:'%T %h %s' \
> | while read tree commit subject ; do
> if git ls-tree -r $tree | grep -q "$obj_name" ; then
> echo $commit "$subject"
> fi
> done
> ---
>
> but it comes up empty, so now I'm stuck. Any ideas would be greatly appreciated.
>
> ------------------------------------------------------------------------------
> Special Offer-- Download ArcSight Logger for FREE (a 49ドル USD value)!
> Finally, a world-class log management solution at an even better price-free!
> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
> February 28th, so secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsight-sfd2d
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

On Wed, Jan 26, 2011 at 8:38 AM, Darren Dale <dsd...@gm...> wrote:
> On Tue, Jan 25, 2011 at 6:12 PM, Darren Dale <dsd...@gm...> wrote:
>> There is still an outstanding issue that must be taken care of before
>> we migrate. The conversion routines create a basemap repository out of
>> trunk/toolkits/basemap, and matplotlib repository out of
>> trunk/matplotlib. Still, the matplotlib repo (at
>> github.com/darrendale/matplotlib) is over 200 MB. One can search the
>> objects in the large packfile, and find that there are still
>> references to basemap data in the matplotlib repo. I don't know how it
>> got in there, nor how to remove it.
>
> I went through the exercise of identifying the largest blob, as
> described near the end of http://progit.org/book/ch9-7.html :
>
> $ git verify-pack -v
> objects/pack/pack-fa44ca56d7ec3964e562494f2fe08203143074bd.idx | sort
> -k 3 -n | tail -3
> 3b8b6c010f8ce59afac1e811b1bbc3efc21b770a blob  9154481 9089827 62749144
> 6328b70e665b58ed7f5aa1e110418cbb3facc07a blob  9331200 94297 156884507
> f784efc1518b10dff33673ad9a7a1ac3a7d107d5 blob  51399604 14333430 162328624
>
> $ git rev-list --objects --all | grep f784efc1518b10dff
> f784efc1518b10dff33673ad9a7a1ac3a7d107d5 toolkits/basemap/data/gshhs_h.txt
>
> This shell script is supposed to identify which commits have that blob
> in their tree (http://stackoverflow.com/questions/223678/git-which-commit-has-this-blob):
>
> ---
> #!/bin/sh
> obj_name="1ドル"
> shift
> git log "$@" --pretty=format:'%T %h %s' \
> | while read tree commit subject ; do
>  if git ls-tree -r $tree | grep -q "$obj_name" ; then
>    echo $commit "$subject"
>  fi
> done
> ---
>
> but it comes up empty, so now I'm stuck. Any ideas would be greatly appreciated.
>
First of all, I must clarify that I'm not a git expert by any means.
I suspected this could be some dangling objects within the repository,
which could be side effects of svn2git. After some googling, I found
that
$ git fsck --unreachable HEAD $(git for-each-ref
--format="%(objectname)" refs/heads)
This gave me 2774 objects which includes the blob of
"toolkits/basemap/data/gshhs_h.txt".
Since they are unreachable, I suppose that they can be simply removed.
I spend an hour to figure out how we can delete these unreachable
objects. But it turned out that the answer seems to be simple.
$ git repack -ad
Now there is no unreachable object reported and this seems to reduce
the total size down to ~140 MB.
Now the biggest blob is for "release/osx/matplotlib-0.98.5.tar.gz". and
$ git log -r -- release/osx/matplotlib-0.98.5.tar.gz
works as expected.
And some of the biggest blobs are associated with svg and pdf files.
It seems possible to remove these files (if needed) from the
repository using "git-filter-branch" to further reduce the size, but
I'm not sure if we need that.
IHTH,
-JJ
> ------------------------------------------------------------------------------
> Special Offer-- Download ArcSight Logger for FREE (a 49ドル USD value)!
> Finally, a world-class log management solution at an even better price-free!
> Download using promo code Free_Logger_4_Dev2Dev. Offer expires
> February 28th, so secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsight-sfd2d
> _______________________________________________
> Matplotlib-devel mailing list
> Mat...@li...
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
>

On Tue, Jan 25, 2011 at 6:12 PM, Darren Dale <dsd...@gm...> wrote:
> There is still an outstanding issue that must be taken care of before
> we migrate. The conversion routines create a basemap repository out of
> trunk/toolkits/basemap, and matplotlib repository out of
> trunk/matplotlib. Still, the matplotlib repo (at
> github.com/darrendale/matplotlib) is over 200 MB. One can search the
> objects in the large packfile, and find that there are still
> references to basemap data in the matplotlib repo. I don't know how it
> got in there, nor how to remove it.
I went through the exercise of identifying the largest blob, as
described near the end of http://progit.org/book/ch9-7.html :
$ git verify-pack -v
objects/pack/pack-fa44ca56d7ec3964e562494f2fe08203143074bd.idx | sort
-k 3 -n | tail -3
3b8b6c010f8ce59afac1e811b1bbc3efc21b770a blob 9154481 9089827 62749144
6328b70e665b58ed7f5aa1e110418cbb3facc07a blob 9331200 94297 156884507
f784efc1518b10dff33673ad9a7a1ac3a7d107d5 blob 51399604 14333430 162328624
$ git rev-list --objects --all | grep f784efc1518b10dff
f784efc1518b10dff33673ad9a7a1ac3a7d107d5 toolkits/basemap/data/gshhs_h.txt
This shell script is supposed to identify which commits have that blob
in their tree (http://stackoverflow.com/questions/223678/git-which-commit-has-this-blob):
---
#!/bin/sh
obj_name="1ドル"
shift
git log "$@" --pretty=format:'%T %h %s' \
| while read tree commit subject ; do
 if git ls-tree -r $tree | grep -q "$obj_name" ; then
 echo $commit "$subject"
 fi
done
---
but it comes up empty, so now I'm stuck. Any ideas would be greatly appreciated.

On Tue, Jan 25, 2011 at 3:06 PM, Darren Dale <dsd...@gm...> wrote:
> On Tue, Jan 25, 2011 at 1:31 PM, Pauli Virtanen <pa...@ik...> wrote:
>> 2011年1月25日 12:19:37 -0500, Darren Dale wrote:
>>> There is a potential problem converting the entire basemap history to
>>> git. In svn commit 4418, trunk/toolkits had basemap and basemap-testing
>>> directories. In commit 4419, basemap was renamed basemap-0.9.6.1, so
>>> there was only basemap-0.9.6.1 and basemap-testing. In commit 4420,
>>> basemap-testing is renamed basemap. The git history only goes back as
>>> far as svn4420, it looks like the conversion routines get confused by
>>> the temporary absence of the basemap directory.
>>>
>>> I'm trying to find a workaround, but if I can't... ?
>>
>> You can maybe do it like this:
>>
>> 1) Write matplotlib.rules so that all of the directories where basemap
>> stuff has been ends in the basemap repository. (I'm assuming this does
>> not error out...)
>
> Aha! I thought I had tried that. Thanks.
>
>> 2) This will create a number of separate heads in the basemap repo that
>> do not share common history.
>>
>> 3) Add graft rules in matplotlib.grafts to stitch the disconnected
>> history graphs together.
>
> Mercifully, the latest checkout of svn2git seems to take care of that.
> I've developed a wicked headache.
>
> Jeff, the repository is temporarily available at
> https://github.com/darrendale/basemap . It would be really helpful if
> you would have a look at the network graph at
> https://github.com/darrendale/basemap/network to make sure there are
> no surprises, maybe clone the repository and check that the working
> directory is identical to your svn checkout.
There is still an outstanding issue that must be taken care of before
we migrate. The conversion routines create a basemap repository out of
trunk/toolkits/basemap, and matplotlib repository out of
trunk/matplotlib. Still, the matplotlib repo (at
github.com/darrendale/matplotlib) is over 200 MB. One can search the
objects in the large packfile, and find that there are still
references to basemap data in the matplotlib repo. I don't know how it
got in there, nor how to remove it.
Jeff: was there ever any basemap data committed directly to trunk/matplotlib?
Pauli: could I trouble you to have a look at my rules file, maybe you
will notice something I overlooked?
(https://github.com/darrendale/mpl2git/blob/master/matplotlib.rules)
Any other ideas?
Thanks
Darren

On Tue, Jan 25, 2011 at 1:31 PM, Pauli Virtanen <pa...@ik...> wrote:
> 2011年1月25日 12:19:37 -0500, Darren Dale wrote:
>> There is a potential problem converting the entire basemap history to
>> git. In svn commit 4418, trunk/toolkits had basemap and basemap-testing
>> directories. In commit 4419, basemap was renamed basemap-0.9.6.1, so
>> there was only basemap-0.9.6.1 and basemap-testing. In commit 4420,
>> basemap-testing is renamed basemap. The git history only goes back as
>> far as svn4420, it looks like the conversion routines get confused by
>> the temporary absence of the basemap directory.
>>
>> I'm trying to find a workaround, but if I can't... ?
>
> You can maybe do it like this:
>
> 1) Write matplotlib.rules so that all of the directories where basemap
> stuff has been ends in the basemap repository. (I'm assuming this does
> not error out...)
Aha! I thought I had tried that. Thanks.
> 2) This will create a number of separate heads in the basemap repo that
> do not share common history.
>
> 3) Add graft rules in matplotlib.grafts to stitch the disconnected
> history graphs together.
Mercifully, the latest checkout of svn2git seems to take care of that.
I've developed a wicked headache.
Jeff, the repository is temporarily available at
https://github.com/darrendale/basemap . It would be really helpful if
you would have a look at the network graph at
https://github.com/darrendale/basemap/network to make sure there are
no surprises, maybe clone the repository and check that the working
directory is identical to your svn checkout.
Darren

2011年1月25日 12:19:37 -0500, Darren Dale wrote:
> There is a potential problem converting the entire basemap history to
> git. In svn commit 4418, trunk/toolkits had basemap and basemap-testing
> directories. In commit 4419, basemap was renamed basemap-0.9.6.1, so
> there was only basemap-0.9.6.1 and basemap-testing. In commit 4420,
> basemap-testing is renamed basemap. The git history only goes back as
> far as svn4420, it looks like the conversion routines get confused by
> the temporary absence of the basemap directory.
> 
> I'm trying to find a workaround, but if I can't... ?
You can maybe do it like this:
1) Write matplotlib.rules so that all of the directories where basemap 
stuff has been ends in the basemap repository. (I'm assuming this does 
not error out...)
2) This will create a number of separate heads in the basemap repo that 
do not share common history.
3) Add graft rules in matplotlib.grafts to stitch the disconnected 
history graphs together.
This happened also with Numpy: part of the old history had a this sort of 
a rename and so part of the history was not connected to the main graph. 
So I just stitched the graphs together manually.
-- 
Pauli Virtanen

There is a potential problem converting the entire basemap history to
git. In svn commit 4418, trunk/toolkits had basemap and
basemap-testing directories. In commit 4419, basemap was renamed
basemap-0.9.6.1, so there was only basemap-0.9.6.1 and
basemap-testing. In commit 4420, basemap-testing is renamed basemap.
The git history only goes back as far as svn4420, it looks like the
conversion routines get confused by the temporary absence of the
basemap directory.
I'm trying to find a workaround, but if I can't... ?

Sorry it was my fault: /dev/shm was not mounted in the chroot where I
built the package so multiprocessing wasn't able to create semaphores.
It's building fine now.
On Thu, Jan 20, 2011 at 22:58, Sandro Tosi <mo...@de...> wrote:
> Hi all,
> I'm fighting against this weirdness since some days: if I build mpl
> debian package on my system, it builds normally, but if I build the
> package in a clean chroot it fails, and I only noticed because I want
> to create a link in doc/build/html/_static dir, that's missing. and
> I'm wondering why! I can't seem to find a clear answer to that, so I'm
> asking for any advice yo might have.
>
> The full log of the package build in the chroot is at [1] (it's quite
> large expanded); at the end of the file there are some ls -lR to try
> to identify what's going wrong.
>
> [1] http://people.debian.org/~morph/matplotlib_1.0.1-1_amd64.build.bz2
>
> Thanks in advance,
> --
> Sandro Tosi (aka morph, morpheus, matrixhasu)
> My website: http://matrixhasu.altervista.org/
> Me at Debian: http://wiki.debian.org/SandroTosi
>
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

On Mon, Jan 24, 2011 at 9:18 AM, John Hunter <jd...@gm...> wrote:
> On Mon, Jan 24, 2011 at 8:11 AM, Darren Dale <dsd...@gm...> wrote:
>
>> That said, I have the conversion routines set up right now to create a
>> separate "toolkits" repository, which currently includes all the
>> contents of trunk/toolkits. I can tailor this further to split the
>> toolkits up and create a repository for each, if thats how people want
>> to do it. Those repos could either be hosted as separate repositories
>> under the matplotlib github organization, or maybe better as a
>> repository under a separate github project. In case of the latter, I
>> can temporarily publish the repository at my own account, long enough
>> for Jeff to clone it (don't fork it using githubs fork button!) and
>> push it to its final destination.
>
> I think mplsize should just be moved into lib/mpl_toolkits so it can
> be picked up by a default mpl installation. It's small enough that it
> probably doesn't need it's own repo. What do you think Andrew?
I guess this can be done after the conversion: split it into its own
repo, fix the hierarchy and then do an external merge from that repo
to draw it into the matplotlib repo under lib/mpl_toolkits.

On 1/24/11 7:11 AM, Darren Dale wrote:
> On Mon, Jan 24, 2011 at 8:48 AM, John Hunter<jd...@gm...> wrote:
>> On Mon, Jan 24, 2011 at 7:18 AM, Jeff Whitaker<js...@fa...> wrote:
>>
>>> Regarding basemap, what do people recommend? Should I create a separate
>>> github project for basemap and it's data?
>> I think that would be ideal. Unlink svn, in git you can't have full
>> featured subdirectory checkouts, and since some of the projects,
>> basemap included are too large to include with the core, the best
>> solution is to put them in a separate repository. This complicates
>> keeping versions in sync, but it seems like the best solution.
> That said, I have the conversion routines set up right now to create a
> separate "toolkits" repository, which currently includes all the
> contents of trunk/toolkits. I can tailor this further to split the
> toolkits up and create a repository for each, if thats how people want
> to do it. Those repos could either be hosted as separate repositories
> under the matplotlib github organization, or maybe better as a
> repository under a separate github project. In case of the latter, I
> can temporarily publish the repository at my own account, long enough
> for Jeff to clone it (don't fork it using githubs fork button!) and
> push it to its final destination.
>
> Darren
Darren: I think having them as separate repos under the matplotlib 
organization is fine. If you could do that I would be grateful.
-Jeff