matplotlib-devel Mailing List for matplotlib

On Thu, Jan 27, 2011 at 4:18 PM, Pauli Virtanen <pa...@ik...> wrote:
> to, 2011年01月27日 kello 13:44 -0500, Darren Dale kirjoitti:
> [clip]
>> Still, Even after removing the the largest blob in the repo with
>>
>> run git filter-branch --index-filter \
>>  'git rm --cached --ignore-unmatch release/osx/matplotlib-0.98.5.tar.gz' \
>>  -- 750059aa09340^..
>>
>> the blob still exists, but is not associated with a commit according to
>>
>> git log --pretty=oneline -- release/osx/matplotlib-0.98.5.tar.gz
>>
>> That blob accounts for 1/4 of the total size of the repo. It would be
>> nice to get rid of it, if possible.
>
> I think "git log" will show you only the current branch by default. Do
>
>    git log --pretty=oneline --all -- release/osx/matplotlib-0.98.5.tar.gz
>
> to get all branches, and do
>
>    for branch in `git for-each-ref --format='%(refname)'`; do S=`git log --pretty=oneline $branch -- release/osx/matplotlib-0.98.5.tar.gz`; if test -n "$S"; then echo "$branch"; echo "$S"; fi; done
>
> to see which refs have the commits containing it.
>
> Similarly, git-filter-branch rewrites only the current branch unless
> told otherwise. To filter everything, it's best to do
>
>    git filter-branch --index-filter \
>        'git rm --cached --ignore-unmatch release/osx/matplotlib-0.98.5.tar.gz' \
>        -- `git for-each-ref --format="750059aa09340^..%(refname)"`
>
> Note that all branches and tags should be filtered in the same way:
> since rewriting changes the hashes of all following commits, you end up
> with incompatible histories otherwise.
>
> After that, I get down to 34 MB.
You are brilliant. If you send me your address off-list, I'll send you
a bottle of scotch, or tequila, or a doughnut, or whatever you want.

On Thu, Jan 27, 2011 at 12:57 PM, Pauli Virtanen <pa...@ik...> wrote:
> 2011年1月27日 12:39:48 -0500, Darren Dale wrote:
> [clip]
>> Me too. I just posted the latest version of the repository to
>> github.com/darrendale/matplotlib.git . Its ~42MB, but it has a bunch of
>> unreachable objects. As soon as we figure out how to git rid of them, I
>> think we will be ready to freeze the svn repo and wrap this up.
>
> Unreachable from where? How do you know there are unreachable
> objects?
>
> Note that the snippet
>
>  git fsck --unreachable HEAD $(git for-each-ref --format="%(objectname)" refs/heads)
>
> only checks for objects unreachable from branches (by definition,
> stuff under refs/heads). However, there's also other stuff under refs/:
> tags and hidden branches. Especially the postprocess.sh script hides
> some branches.
>
> To see all that is there, check the output from
>
>  git for-each-ref
Oh, I didn't understand what I was doing with the git fsck command.
Still, Even after removing the the largest blob in the repo with
run git filter-branch --index-filter \
 'git rm --cached --ignore-unmatch release/osx/matplotlib-0.98.5.tar.gz' \
 -- 750059aa09340^..
the blob still exists, but is not associated with a commit according to
git log --pretty=oneline -- release/osx/matplotlib-0.98.5.tar.gz
That blob accounts for 1/4 of the total size of the repo. It would be
nice to get rid of it, if possible.
Darren

to, 2011年01月27日 kello 13:44 -0500, Darren Dale kirjoitti:
[clip]
> Still, Even after removing the the largest blob in the repo with
> 
> run git filter-branch --index-filter \
> 'git rm --cached --ignore-unmatch release/osx/matplotlib-0.98.5.tar.gz' \
> -- 750059aa09340^..
>
> the blob still exists, but is not associated with a commit according to
> 
> git log --pretty=oneline -- release/osx/matplotlib-0.98.5.tar.gz
> 
> That blob accounts for 1/4 of the total size of the repo. It would be
> nice to get rid of it, if possible.
I think "git log" will show you only the current branch by default. Do
 git log --pretty=oneline --all -- release/osx/matplotlib-0.98.5.tar.gz
to get all branches, and do
 for branch in `git for-each-ref --format='%(refname)'`; do S=`git log --pretty=oneline $branch -- release/osx/matplotlib-0.98.5.tar.gz`; if test -n "$S"; then echo "$branch"; echo "$S"; fi; done
to see which refs have the commits containing it.
Similarly, git-filter-branch rewrites only the current branch unless
told otherwise. To filter everything, it's best to do
 git filter-branch --index-filter \
 	'git rm --cached --ignore-unmatch release/osx/matplotlib-0.98.5.tar.gz' \
 	-- `git for-each-ref --format="750059aa09340^..%(refname)"`
Note that all branches and tags should be filtered in the same way:
since rewriting changes the hashes of all following commits, you end up
with incompatible histories otherwise.
After that, I get down to 34 MB.
-- 
Pauli Virtanen

2011年1月27日 12:39:48 -0500, Darren Dale wrote:
[clip]
> Me too. I just posted the latest version of the repository to
> github.com/darrendale/matplotlib.git . Its ~42MB, but it has a bunch of
> unreachable objects. As soon as we figure out how to git rid of them, I
> think we will be ready to freeze the svn repo and wrap this up.
Unreachable from where? How do you know there are unreachable
objects?
Note that the snippet
 git fsck --unreachable HEAD $(git for-each-ref --format="%(objectname)" refs/heads)
only checks for objects unreachable from branches (by definition,
stuff under refs/heads). However, there's also other stuff under refs/:
tags and hidden branches. Especially the postprocess.sh script hides
some branches.
To see all that is there, check the output from
 git for-each-ref
-- 
Pauli Virtanen

On Thu, Jan 27, 2011 at 12:00 PM, John Hunter <jd...@gm...> wrote:
> On Thu, Jan 27, 2011 at 10:26 AM, Darren Dale <dsd...@gm...> wrote:
>> On Tue, Jan 25, 2011 at 11:37 PM, Jae-Joon Lee <lee...@gm...> wrote:
>>> And some of the biggest blobs are associated with svg and pdf files.
>>> It seems possible to remove these files (if needed) from the
>>> repository using "git-filter-branch" to further reduce the size, but
>>> I'm not sure if we need that.
>>
>> Some of this appears to be the result of some svn tags referencing the
>> contents of trunk/htdocs. It looks like I can cut 25 MB off the size
>> of the repo.
>
> That feels pretty good to me, considering that the compressed release
> tarball is 13MB. If we can get the whole history for twice that, then
> you must be getting pretty close to the limit.
Me too. I just posted the latest version of the repository to
github.com/darrendale/matplotlib.git . Its ~42MB, but it has a bunch
of unreachable objects. As soon as we figure out how to git rid of
them, I think we will be ready to freeze the svn repo and wrap this
up.

On Thu, Jan 27, 2011 at 10:26 AM, Darren Dale <dsd...@gm...> wrote:
> On Tue, Jan 25, 2011 at 11:37 PM, Jae-Joon Lee <lee...@gm...> wrote:
>> And some of the biggest blobs are associated with svg and pdf files.
>> It seems possible to remove these files (if needed) from the
>> repository using "git-filter-branch" to further reduce the size, but
>> I'm not sure if we need that.
>
> Some of this appears to be the result of some svn tags referencing the
> contents of trunk/htdocs. It looks like I can cut 25 MB off the size
> of the repo.
That feels pretty good to me, considering that the compressed release
tarball is 13MB. If we can get the whole history for twice that, then
you must be getting pretty close to the limit.
JDH

On Tue, Jan 25, 2011 at 11:37 PM, Jae-Joon Lee <lee...@gm...> wrote:
> And some of the biggest blobs are associated with svg and pdf files.
> It seems possible to remove these files (if needed) from the
> repository using "git-filter-branch" to further reduce the size, but
> I'm not sure if we need that.
Some of this appears to be the result of some svn tags referencing the
contents of trunk/htdocs. It looks like I can cut 25 MB off the size
of the repo.

On Wed, Jan 26, 2011 at 6:35 AM, Jae-Joon Lee <lee...@gm...> wrote:
> On Wed, Jan 26, 2011 at 7:28 PM, Pauli Virtanen <pa...@ik...> wrote:
>> The complete magic stanza is:
>>
>> git reflog expire --expire=0 --all
>> git prune
>> git repack -f -a -d
>> git gc --prune=0
>
> Wonderful!
> With this, I get about 40 MB!
How did you manage to do that? Using these steps (which have been
added to the postprocess.sh script):
---
run git filter-branch --index-filter \
 'git rm --cached --ignore-unmatch release/osx/matplotlib-0.98.5.tar.gz' \
 -- 750059aa09340^..
rm -Rf refs/original
rm -Rf logs
run git reflog expire --expire=0 --all
run git prune
run git repack -f -a -d --depth=250 --window=250
run git gc --prune=0
---
I only get down to 65 MB, at which point "git fsck --unreachable HEAD"
indicates a slew of unreachable blobs, trees, and commits, one of
which is:
3b8b6c010f8ce59afac1e811b1bbc3efc21b770a release/osx/matplotlib-0.98.5.tar.gz
yet "git log --pretty=oneline -- release/osx/matplotlib-0.98.5.tar.gz"
shows it is no longer associated with any commits. Rerunning the
"reflog/prune/repack/gc" block doesn't seem to help at this point.

On Wednesday, January 26, 2011, Darren Dale <dsd...@gm...> wrote:
> On Wed, Jan 26, 2011 at 7:44 AM, Darren Dale <dsd...@gm...> wrote:
>> Last night I noticed that, in the git repo, the commit messages
>> produced by svnmerge.py still contain a lot of svn-specific
>> information. Pauli's conversion script includes a step that filters
>> out two lines at the end of each commit containing some svn metadata,
>> but for svnmerge commits we still end up with:
>>
>>  Merged revisions 8933 via svnmerge from
>>  https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_maint
>>
>>  ........
>>   r8933 | weathergod | 2011年01月22日 10:35:26 -0600 (Sat, 22 Jan
>> 2011) | 3 lines
>>
>>   Fixing problem where reversed colormaps of
>> LinearSegmentedColormaps were not initialized properly.
>>   Thanks to LittleBigBrain for reporting and Friedrich Romstedt
>> for making the original patch.
>>  ........
>>
>> Do we live with this, or try to replace references to svn commits with
>> their git hash and references to svn branches with the git ones?
>
> I think I just convinced myself that replacing svn references with git
> hashes would be a Herculean task. One could generate a
> svn_rev:git_hash mapping, but then changing the commit message
> actually yields a new git hash, breaking the mapping. Its not an
> impossible task, but it would be a lot of work and difficult to check.
> Leaving the svn path and revision information will probably be more
> reliable. "git log --all" will therefore yield enough information that
> we can easily identify the provenance of a cherry pick and find it in
> the git history:
>
> commit adb1a7f67daf955a1af3a86d42b5181767c18819
> Author: Ben Root <ben...@gm...>
> Date:  Sat Jan 22 16:40:21 2011 +0000
>
>  Merged revisions 8933 via svnmerge from
>  https://matplotlib.svn.sourceforge.net/svnroot/matplotlib/branches/v1_0_main
>
>  ........
>   r8933 | weathergod | 2011年01月22日 10:35:26 -0600 (2011年1月22日) | 3 line
>
>   Fixing problem where reversed colormaps of LinearSegmentedColormaps were n
>   Thanks to LittleBigBrain for reporting and Friedrich Romstedt for making t
>  ........
>
>  svn path=/trunk/matplotlib/; revision=8934
>
> commit 2fa57c710607496a93000bfb3191bdd422f518cc
> Author: Ben Root <ben...@gm...>
> Date:  Sat Jan 22 16:35:26 2011 +0000
>
>  Fixing problem where reversed colormaps of LinearSegmentedColormaps were not
>  Thanks to LittleBigBrain for reporting and Friedrich Romstedt for making the
>
>  svn path=/branches/v1_0_maint/; revision=8933
>
>
> Is this satisfactory?
>
> Darren
>
I think it is fine. Maybe a nice convenience script would be one that
would return a got hash for a svn revision number (and vice verse?).
With that, I would be more than happy with how the logs are.
Ben Root