266157 – sys-kernel/libre-sources ebuild request

1) FSF started new "libre" fork of the Linux kernel without non-free blobs:
http://directory.fsf.org/project/linux/
The main version of the Linux kernel that is being maintained by Linus Torvalds, Andrew Morton, et al., contains proprietary code in the form of binary blobs, so in the interest of freedom, we are providing a link to a version of the kernel in which this proprietary code has been removed so that it is entirely free software.
2) FSF Latin America (FSFLA) maintains repository of the libre sources:
http://fsfla.org/svnwiki/selibre/linux-libre/
http://www.fsfla.org/~lxoliva/fsfla/linux-libre/
linux-libre@fsfla.org
3) Some people on FSLA mailing list requested libre-sources ebuild for Gentoo.
http://www.fsfla.org/pipermail/linux-libre/2009-January/000371.html
4) I post this request here :)
Reproducible: Always
Steps to Reproduce:
Expected Results: 
sys-kernel/libre-sources in Portage ;)

Created attachment 191488 [details]
libre-sources-2.6.29.3 ebuild
This should work nicely. Also renaming it to the appropriate upstream release should all work as expected.

Created attachment 191489 [details]
libre-sources-2.6.29.3 ebuild (using deblob scripts)
This is a version which grabs the vanilla kernel and applies patches etc., and runs their deblob script. I wouldn't recommend it, simply as it takes in the region of hours of time and a gigabyte or so of RAM to complete.

Thanks a lot! :)
I didn't believe it to appear. I thought it was needed only for me and very very few people. I've tried to write this ebuild myself, but I'm novice in gentoo and so my ebuild didn't work...
Thank you for the help!
P.S. Does it need to apply any gentoo-specific patches to this sources (like in sys-kernel/gentoo-sources)?

(In reply to comment #3)
> Thanks a lot! :)
> I didn't believe it to appear. I thought it was needed only for me and very
> very few people. I've tried to write this ebuild myself, but I'm novice in
> gentoo and so my ebuild didn't work...
> Thank you for the help!
My pleasure - glad you find it useful. I myself had been waiting for an ebuild to appear, but finally decided the best way to make that happen would be to write one myself :)
> P.S. Does it need to apply any gentoo-specific patches to this sources (like in
> sys-kernel/gentoo-sources)?
It doesn't apply any patches, it's just the straight (deblobbed) vanilla linux kernel from kernel.org . The gentoo-sources guys, in their wisdom, don't use many patches, as they emphasise pushing stuff upstream (see http://dev.gentoo.org/~dsd/genpatches/about.htm).
It would be very easy to modify the usedeblob ebuild to use their patches, but as it takes SO LONG to actually deblob, and there aren't many patches we miss out on by not fetching those from gentoo-sources, I'm not enormously motivated to write a version. If it would be useful to anyone let me know and I'll knock one up for you.

Created attachment 192011 [details]
libre-sources-2.6.29.3 ebuild
Added ~amd64 keyword, after email correspondance reporting it working without problem on that arch.

Created attachment 192142 [details]
libre-sources-2.6.29.4 ebuild
Ebuild for latest version (no change from the last ebuild, just a rename).

Thanks a lot, I hope this appears in the tree soon. I'm working with it and aside from a few 404s till the sources could be grabbed and having to generate the Manifest everything seems fine :-)

Works fine on all my servers!
Thanks for the ebuilds. :)

Created attachment 195056 [details]
kernel-libre eclass
I've put togethor a new eclass (attached), which makes it much easier to write linux libre - based ebuilds for different kernel patchsets.
This is required for the new ebuilds (to follow).

Created attachment 195057 [details]
libre-sources-2.6.30 ebuild
New libre-sources ebuild, using the kernel-libre eclass.

Created attachment 195059 [details]
libregentoo-soources

Created attachment 195060 [details]
libregentoo-sources-2.6.30-r1 ebuild
A new ebuild of the blob-free kernel, with the gentoo kernel patchset. Requires kernel-libre eclass.

Comment on attachment 195059 [details]
libregentoo-soources
(sorry, finger slipped, see below comment for correct ebuild - ignore this one)

Created attachment 195062 [details]
librehardened-sources-2.6.29 ebuild
A new ebuild of the blob-free kernel, with the gentoo hardened patchset. Requires
kernel-libre eclass.

Thanks again! :) The ebuild is alive!
P.S. Did you measure, how much time does it take to fully deblob the kernel source? And on what machine?

(In reply to comment #15)
> P.S. Did you measure, how much time does it take to fully deblob the kernel
> source? And on what machine?
It takes over four hours (I got bored and gave up around then), on my 1.6GHz machine. And it used around 2GB RAM throughout.
The primary author of the deblob script is currently working on making it much faster, see http://www.fsfla.org/pipermail/linux-libre/2009-May/000610.html for more details on that.

Created attachment 195855 [details]
libre-sources-2.6.29.5 ebuild
Just renaming the ebuild.

Created attachment 196628 [details]
libre-sources-2.6.29.6 ebuild
Just renaming the ebuild.

Created attachment 196943 [details]
libregentoo-sources-2.6.30-r2.ebuild

Created attachment 197978 [details]
libre-sources-2.6.29.6.1 ebuild
Just renaming the ebuild.

Created attachment 197980 [details]
libre-sources-2.6.30.1 ebuild
sorry, it was a duplicate of 2.6.29.6 (2.6.29.6-libre1 != 2.6.29.6.1-libre)
The new ebuild is 2.6.30.1-libre

Nick, what do you think about lzma tarbolls? Is it possible to portage to unpack them? They are about 10Mb smaller...
I tried it mysel just to replace tar.bz2 with tar.lz, but it didn't help...
May be some keys to unpack, or something else?

(In reply to comment #22)
> Nick, what do you think about lzma tarbolls? Is it possible to portage to
> unpack them? They are about 10Mb smaller...
They're lzip compressed, which is based on lzma, but not the same (s ee http://www.nongnu.org/lzip/lzip.html ). While portage can unpack lzma, it cannot lzip, at present.
You can uncompress lzip archives using the app-arch/lzip package.
We could quite easily manually unpack the archive in the src-unpack function using app-arch/lzip. However as app-arch/lzip is not marked stable at the moment, portage has no in-built lzip capability, and the other kernel ebuilds use bzip2, I don't think it's worth it to add it.
If you think it's really useful I could add it as a use-flag option, though.
Thoughts?

I agree with you, that if portage can't deal with lzip yet, making the ebuild non-standard way is now worth thing (somewhat like rpm2tgz unpacking).
Also it's not so big problem to donwload 50 Mb instead of 40.
Let's wait for app-arch/lzip to beacme stable... :)

sorry:
s/is now worth thing/is NOT worth thing/
s/beacme/became/

Created attachment 198448 [details]
libregentoo-sources-2.6.30-r3 ebuild
Ebuild of linux libre kernel using the latest gentoo patchset. Requires kernel-libre.eclass.
I'll try to keep an eye on the status of lzip in gentoo; let me know here if you see anything which warrants reconsidering its use. And thanks Tim for your continued input and interest :-)

Congratulations, Nick! :)
Your ebuilds are publicly avaliable now: http://gentoo-overlays.zugaina.org/njw/
I think, it's your own repository, isn't it? ;)

Created attachment 199039 [details]
libregentoo-sources-2.6.30-r4
Ebuild of linux libre kernel using the latest gentoo patchset. Requires
kernel-libre.eclass.
And yes, Tim, that is my overlay (originally to be found at http://git.njw.me.uk/cgit/cgit.cgi/njw-gentoo-local/). Just got it added to layman - most exciting! You sure noticed quickly ;-)

Created attachment 199764 [details]
libre-sources-2.6.30.4 ebuild
Expected to be in njw repository :)

OK, libre-sources 2.6.30.4 is now in the njw repository. Thanks for the prod!

Created attachment 200034 [details]
kernel-libre eclass
New version of the libre kernel eclass which is more reliable with libre-sources version numbering.

Created attachment 201340 [details]
libregentoo-sources-2.6.30-r5 ebuild
Ebuild incorporating the latest gentoo patchset into the libre kernel.
Requires kernel-libre.eclass
Also available from the njw overlay.

Created attachment 201770 [details]
libre-sources-2.6.30.5 ebuild
Latest linux-libre kernel.
Requires kernel-libre.eclass
Also available from the njw overlay.

Thanks, Nick.
5 minutes ago I didn't found this ebuild and have moved to the "libregentoo" ebuild serie, which I like more, than simply "libre" :)
It was a surprise, that it didn't take some hours to deblob the tree! Why?
And also, I think, your repository is better place for the eclass and all these ebuilds, than attachments to this bugreport. Here, in bugzilla, attachments are hard to maintain; we can't delete obsolete ones...
What's your opinion? Why do you need to duplicate these ebuilds here?
P.S. To all the gentoo kernel list:
What are the reasons to not to include these ebuilds and eclass in portage tree?
What can we do for it?

> ... moved to the "libregentoo" ebuild serie, which I like more, than simply "libre" :)
> It was a surprise, that it didn't take some hours to deblob the tree! Why?
The gentoo-sources ebuild takes the vanilla kernel tree and applies a few patches to it.
My libregentoo-sources ebuild does the same with the libre kernel tree as its base. So no deblobbing is necessary. It does check the gentoo patches for blobs, and warn the user if any are found (which seems exceedingly unlikely).
> And also, I think, your repository is better place for the eclass and all these
> ebuilds, than attachments to this bugreport. Here, in bugzilla, attachments are
> hard to maintain; we can't delete obsolete ones...
> What's your opinion? Why do you need to duplicate these ebuilds here?
I've been pushing all of the ebuilds to bugzilla too in case people don't want to use my overlay. Perhaps it would be sensible to only post non-bump changes here?
The ebuilds in my overlay are always available on the web, at http://git.njw.me.uk/cgit/cgit.cgi/njw-gentoo-local/tree/sys-kernel
> What are the reasons to not to include these ebuilds and eclass in portage tree?
Well the simplest answer is probably that I haven't explicitly asked them. That said, if there is any desire from gentoo kernel developers to include any of these kernel versions in portage I'd be happy to take the role of maintainer (or help out any other way, for that matter).
Perhaps reworking my effort to add a "libre" or "linux-libre" use flag to the existing kernels (by extending kernel-2.eclass) would be more reasonable to the gentoo devs? I'll look into this more when I have more time (in a couple of weeks), and would be very interested to hear any feedback on this idea.

sys-kernel/libregentoo-sources-2.6.31 version bump
sys-kernel/libre-sources-2.6.31 version bump
http://www.linux-libre.fsfla.org/pub/linux-libre/releases/2.6.31-libre/ 

Created attachment 203898 [details]
libre-sources 2.6.31 ebuild
Damn Tim, you're fast with those requests ;-)
This is the libre-sources-2.6.31 ebuild, the libregentoo one will follow.
Both of these are now in the njw repository.

Created attachment 203899 [details]
libregentoo-sources 2.6.31 ebuild

Thanks, Nick! :)

any reason the libregentoo-sources-2.6.31 has keywords="~x86" only. Can I change that line and have it work on my amd64 box?

(In reply to comment #40)
> any reason the libregentoo-sources-2.6.31 has keywords="~x86" only. Can I
> change that line and have it work on my amd64 box?
The reason is only that no-one has (to my knowledge) tested it on a non-x86 platform yet. It should work fine, so please do try it.
Let us know if it works for you (as I say, it certainly should), and I'll add ~amd64 to keywords.

Nick, could you explain me something about your eclass?
What name should I give to the ebuild for 2.6.31.4-libre1 (or may be 2.6.31.4-libre2 if it would be) sources to comply with package syntax?
And how the ${LIBRE_VERSION} in eclass is determined?
sys-kernel/libregentoo-sources-2.6.31.4-libre1 and
sys-kernel/libregentoo-sources-2.6.31.4_libre1
didn't make the ebuild utility happy:
"does not follow correct package syntax" it said! :)

(In reply to comment #42)
> Nick, could you explain me something about your eclass?
> What name should I give to the ebuild for 2.6.31.4-libre1 (or may be
> 2.6.31.4-libre2 if it would be) sources to comply with package syntax?
> And how the ${LIBRE_VERSION} in eclass is determined?
Hi Tim, good to hear from you.
$LIBRE_VER is just set in the ebuild itself, so just set LIBRE_VER=1 in the ebuild(s) which need it.
I've fallen a little behind with the ebuilds, (though they don't take long), I'll update them tonight. Or, if you want to (and feel like being awesome), post new ebuilds here, and I'll add them to the overlay. Thanks to the eclass it's just a case of renaming, and adding the LIBRE_VER varbiable if necessary, and of course basic testing that it merges correctly.

(In reply to comment #43)
> I'll update them tonight. Or, if you want to (and feel like being awesome),
> post new ebuilds here, and I'll add them to the overlay.
Right, all should be up to date now, using the correct LIBRE_VER, and with the latest patchset ebuilds available.
Attachments of working libregentoo-sources and libre-sources follow.

Created attachment 207206 [details]
libre-sources-2.6.31.3 ebuild
Ebuild for libre-sources 2.6.31.3, with updated libre script version.

Created attachment 207208 [details]
libregentoo-sources-2.6.31-r3 ebuild
Ebuild for libregentoo-sources-2.6.31-r3, with updated libre script version.

Created attachment 208411 [details]
libregentoo-sources-2.6.31-r4 ebuild

Created attachment 208412 [details]
libre-sources-2.6.31.4 ebuild

Awesome, thanks for adding those Tim, added them to my repository. I also just the libre-sources-2.6.31.5 ebuild to my repository.
In other news, I've added a linux libre kernel with realtime patches to my repository (called librert-sources). I'll add it as an attachment to this bug in a while; I'm hoping to first fix a problem with the deblob-check script incorrectly identifying the patches as containing a blob (at the moment the ebuild just skips the test).

> In other news, I've added a linux libre kernel with realtime patches to my
> repository (called librert-sources).
Thanks to you, Nick, we now have much more ebuilds, than I asked in first comment. :)
P.S. I post here in bugzilla just the ebuilds, not manifest for them. Please, update it in your repository... For now I get this error: :(
Checking for possible errors......
Checking 'linux-2.6.31-libre1.tar.bz2'... ok
Checking 'genpatches-2.6.31-5.base.tar.bz2'... not in Manifest
Checking 'genpatches-2.6.31-5.extras.tar.bz2'... not in Manifest
Checking 'deblob-check-2.6.31'... ok
Fetch error:
 * In program paludis (--continue-on-failure if-satisfied --with-unused-dependencies --dl-new-slots as-needed --log-level silent) -ip everything:
 * When performing install action from command line:
 * When executing install task:
 * When performing pretend actions:
 * When fetching 'sys-kernel/libregentoo-sources-2.6.31-r4:2.6.31-r4::njw':
 * Fetch of 'sys-kernel/libregentoo-sources-2.6.31-r4:2.6.31-r4::njw' failed
 * File 'genpatches-2.6.31-5.base.tar.bz2': failed integrity checks: Not in Manifest
 * File 'genpatches-2.6.31-5.extras.tar.bz2': failed integrity checks: Not in Manifest
P.P.S. How can I reply to some post in this bug tracker? I don't see any button or link for this... May be, it can be done by replying to email?

Something wrong with GENPATCHES?
# paludis -i =libregentoo-sources-2.6.31-r4
...
Checking all patches are clean of blobs
/usr/portage/distfiles/genpatches-2.6.31-5.base.tar.bz2
 * Warning: A file in UNIPATCH_LIST_GENPATCHES appears to contain blobs.
 * Disabling UNIPATCH_LIST_GENPATCHES for now.
 * Please report this to http://bugs.gentoo.org/266157
...
It was the same with previous version...

(In reply to comment #50)
> P.S. I post here in bugzilla just the ebuilds, not manifest for them. Please,
> update it in your repository... For now I get this error: :(
Oops, yes, I have a tendency to forget to update the manifests (as paludis, which I tend to use, will continue without them). Anyway, I've added them now. I'll tweak my settings so it's harder to forget ;-) 
> P.P.S. How can I reply to some post in this bug tracker? I don't see any button
> or link for this... May be, it can be done by replying to email?
What do you mean, reply to a post in the bug tracker (you're clearly competent at replying to comments here)? Feel free to email me at the address registered here if you want. 
(In reply to comment #51)
> Something wrong with GENPATCHES?
> 
> # paludis -i =libregentoo-sources-2.6.31-r4
> ...
> Checking all patches are clean of blobs
> /usr/portage/distfiles/genpatches-2.6.31-5.base.tar.bz2
> * Warning: A file in UNIPATCH_LIST_GENPATCHES appears to contain blobs.
> * Disabling UNIPATCH_LIST_GENPATCHES for now.
> * Please report this to http://bugs.gentoo.org/266157
> ...
> It was the same with previous version...
Aha, sloppily I didn't notice this. Very likely it isn't a problem with GENPATCHES, but with the recent change I made in the kernel-libre eclass. I'll have a look tomorrow evening. It should be easy enough to fix, I'll let you know when I have.
Thanks for keeping me on my toes ;-)

(In reply to comment #51)
> Checking all patches are clean of blobs
> /usr/portage/distfiles/genpatches-2.6.31-5.base.tar.bz2
> * Warning: A file in UNIPATCH_LIST_GENPATCHES appears to contain blobs.
> * Disabling UNIPATCH_LIST_GENPATCHES for now.
> * Please report this to http://bugs.gentoo.org/266157 
Huh, well it turns out one of the device tables added in the 2.6.31.2 patchset is incorrectly identified as containing a blob by the current deblob script.
i asked the linux-libre list for help to whitelist them, but in the meantime i'll turn off the patch checking for these ebuilds.

(In reply to comment #53)
> Huh, well it turns out one of the device tables added in the 2.6.31.2 patchset
> is incorrectly identified as containing a blob by the current deblob script.
> 
> i asked the linux-libre list for help to whitelist them, but in the meantime
> i'll turn off the patch checking for these ebuilds.
Also, the patch doesn't apply cleanly, as it's not expecting deblobbed messages where it wants to make changes. I'll patch this up tonight, but in the longer term will look into properly deblobbing the patches so that they'll always apply cleanly, even to deblobbed sections of code.

(In reply to comment #53)
> What do you mean, reply to a post in the bug tracker (you're clearly competent
> at replying to comments here)? Feel free to email me at the address registered
> here if you want.
Yes, I was rather lazy to try to discover it myself :)
I meant making "In reply to comment #" links and quoting comments automatically...
The bugzilla engine makes the "reply" links in javascript, which was turned off by default in my NoScript, so i couldn't see them. It's all OK now :)
P.S. Thanks for SKIP_PATCH_DEBLOB="1" ;-)
P.P.S. Nick, could you explain me some unclear thing in version numbers of these ebuilds: why the latest libre-sources is 2.6.31.5 while the latest libregentoo-sources (and gentoo-sources) is 2.6.31-r4 ?

Created attachment 213127 [details]
libregentoo-sources-2.6.32-r1 ebuild 
libregentoo-sources-2.6.32-r1 version bump ;)
Nick, do you accept pull requests in your njw repository? And in what branch? ;)
In simple cases, when only renaming of ebuild is needed, I temporary put the ebuilds in my own repository for internal use (http://gitorious.org/okelly_overlay), until they appear in your "official" :) repository.
If you wish, I can put them directly in some branch of your repository instead... I don't want to be the lead maintainer of these ebuilds :), I just want to have them as soon as possible after they are released.

P.S. In 2.6.32 branch deblob patches seem to apply clearly. So no need is in SKIP_PATCH_DEBLOB="1" for now...

(In reply to comment #56)
> libregentoo-sources-2.6.32-r1 version bump ;)
Awesome, thanks a lot Tim - I've had plenty on my plate recently, so I'm glad you bumped this for me. I'll add them to my overlay this weekend.
> If you wish, I can put them directly in some branch of your repository
> instead...
I don't think that would be doable at the moment as my git server currently only has an git+ssh capabilities. Perhaps I should set up a linux-libre-gentoo repository on gitorious, which we could both push to? Or you could ;-) Thoughts?
> I don't want to be the lead maintainer of these ebuilds :), I just
> want to have them as soon as possible after they are released.
Sure, I quite understand. You just version-bumping and testing is very much appreciated, and reduces the pressure & dependence on me, which is great :-)

Hi,
I tested the libregentoo-sources-2.6.32-r1 ebuild on x86. Besides having to grab the manifest and the missing files from http://www.linux-libre.fsfla.org/pub/linux-libre/releases/2.6.32-libre/, it emerged, compiled and ran perfectly with no problems.
Why is this not in the main tree? Is it due to lack of testing or has it just not been brought to the attention of the devs? 
Is there anything I could do to assist?

Reading http://www.fsfla.org/svn/fsfla/software/linux-libre/scripts/README it looks like all they do is repackaging linux tree with some scripts. I guess we may do the same in our kernel-2 eclass (depending on USE flag). This way we'll have to maintain one ebiuld less and we'll make possible to install all our sources packages blob free. Have anybody considered this way?

I think, it would be the best way to maintain these ebuilds.
And Nick White already proposed the same: "Perhaps reworking my effort to add a "libre" or "linux-libre" use flag to the existing kernels (by extending kernel-2.eclass) would be more reasonable to the gentoo devs?"
He had already asked the gentoo kernel team to do this, and I join him in his request.

(In reply to comment #61)
> He had already asked the gentoo kernel team to do this, and I join him in his
> request.
I guess some test implementation will help here.

(In reply to comment #59)
> Why is this not in the main tree? Is it due to lack of testing or has it just
> not been brought to the attention of the devs? 
Basically as I haven't really brought it to attention of the developers.
(In reply to comment #60)
> Reading http://www.fsfla.org/svn/fsfla/software/linux-libre/scripts/README it
> looks like all they do is repackaging linux tree with some scripts. I guess we
> may do the same in our kernel-2 eclass (depending on USE flag). This way we'll
> have to maintain one ebiuld less and we'll make possible to install all our
> sources packages blob free. Have anybody considered this way?
I did consider this, and wrote a rough working ebuild for it (which could be generalised easily to the kernel-2 eclass), see the attachment #191489 [details]: libre-sources-2.6.29.3-usedeblob.ebuild. However the deblobbing scripts used by linux-libre last I checked (couple of months ago) took hours on a reasonable computer, and as such I lost interest in that approach. See comments #2, #4 and #16 above. There was some work on making the scripts faster, but I don't think they have come to anything yet.
One other issue is that the deblob scripts typically are released a few days after the vanilla kernel releases, so there will be a gap without deblob scripts being available. A similar issue was one reason for using separate icecat and firefox ebuilds, rather than implementing it as a USE flag (see bug #233164). But that does take more maintenance, and a clear message along the lines of "A deblob script for version x.x.x is not currently available" would probably suffice here.
I do like the USE flag approach, and will have another look at it. I'll have a go at creating a patch to add this functionality to kernel-2.eclass, and post the results here.
If anyone in the kernel herd has views on the right way for this to go for gentoo, I'd be very happy to hear them.

(In reply to comment #63)
> the deblobbing scripts used by
> linux-libre last I checked (couple of months ago) took hours on a reasonable
> computer, and as such I lost interest in that approach. See comments #2, #4 and
> #16 above. There was some work on making the scripts faster, but I don't think
> they have come to anything yet.
I tested the latest scripts, and they take my 1.6GHz computer about 24 hours (in which time it's essentially unusable) to deblob the latest kernel. Not great.
However, I found that the deblob script can also run just removing whole files rather than replacing things it regards as bad with '/* DEBLOBBED */'. This might make a few things not work which worked previously (if they had fine code in the same file as the bad code), but practically I don't think it's likely to make much difference. And when running in this mode, the deblob script only takes about a minute, so it's usable.
So I modified gentoo's kernel-2 eclass, with a 'deblob' use flag (which I think should be described in use.desc as 'Remove components the FSF considers nonfree'). I'll attach the patch and a full copy of the new version.
I'm interested to hear how this more savage deblobbing goes for people - if it causes problems for anyone where the libre-sources ebuilds worked, then we should probably have the deblob use flag in the eclass, but also add libre-sources to the tree, for a canonical vanilla deblobbed version.
There are still a couple of problems which need to be addressed with the eclass changes, though.
 - The deblob script is generally released a couple of days after new major releases of the kernel, before which time the script won't be available, so fetching will fail. Would this cause a problem for manifest generation? Should we show a notice explaining this if fetching the script fails?
 - The deblob script is liable to change occasionally, without the URL changing (as it's a symbolic link to the script location). This would presumably screw up manifests, and previously downloaded versions of the script would not be replaced. We're using the symbolic link to the script, as otherwise the name of the directory holding it is liable to change, on new revisions of the script, which we can't easily automatically detect (e.g. the 2.6.31 scripts are in the directory 2.6.31-libre2 whereas the 2.6.32 scripts are in 2.6.32-libre - see http://www.linux-libre.fsfla.org/pub/linux-libre/releases/).
 Any advice from gentoo devs on the best course of action would be very warmly received. Thanks in advance.

Created attachment 215031 [details, diff]
Patch of kernel-2 eclass adding deblob use flag

Created attachment 215033 [details]
Full kernel-2 eclass adding deblob use flag

I don't feel that this belongs in the eclass, or that USE flags should make such modifications to the installed sources.
What I would suggest is creating a package that simply installs this deblob script, and maybe a wrapper to make it even easier to use. Then interested users could just run something like "emerge fsf-kernel-deblob; fsf-kernel-deblob /usr/src/linux"
You could even make such a package optionally install a portage hook which would automatically run the deblob script as soon as any new kernel is installed.

dsd: the one advantage to the USE flag or seperate ebuilds is that they will be automatically pulled in by ACCEPT_LICENSES, while your suggestion will cause a block to occur.

Also, the deblob scripts depend on a particular major kernel version. I could package up all of the scripts, and then create a wrapper script which detects the kernel version and chooses the appropriate deblob script. However then I'd have to release new versions with each release kernel, and it generally sounds like more work and maintenance than is necessary.
dsd: Why doesn't a USE flag in an eclass sound like the right idea to you?
Robert: For ACCEPT_LICENSES to differentiate between linux and linux-libre, the linux sources ebuilds would have to include in LICENSE something else identifying it as containing non-FSF-approved bits.

# emerge libre-sources
<...>
Saving to: `/usr/portage/distfiles/linux-2.6.32-libre.tar.bz2'
100%[=============================================>] 62,634,375 786K/s in 75s
2010年01月28日 21:45:10 (819 KB/s) - `/usr/portage/distfiles/linux-2.6.32-libre.tar.bz2' saved [62634375/62634375]
(u'Failed on RMD160 verification', '4bb5a049bd993068f0093497c754142cb3efcfa5', u'283a589f1620b867c6d149b946e599c32a11b1f6')
!!! Fetched file: linux-2.6.32-libre.tar.bz2 VERIFY FAILED!
!!! Reason: Failed on RMD160 verification
!!! Got: 4bb5a049bd993068f0093497c754142cb3efcfa5
!!! Expected: 283a589f1620b867c6d149b946e599c32a11b1f6
Refetching... File renamed to '/usr/portage/distfiles/linux-2.6.32-libre.tar.bz2._checksum_failure_.LqVcOR'
!!! Couldn't download 'linux-2.6.32-libre.tar.bz2'. Aborting.
<...>
P.S. Is this a right place to post this?

Created attachment 218889 [details, diff]
Patch of kernel-2 eclass adding a deblob use flag
Updated kernel-2 eclass patch fixing typo

Created attachment 218891 [details]
Full kernel-2 eclass adding deblob use flag
Updated kernel-2 eclass fixing typo

(In reply to comment #70)
> # emerge libre-sources
> <...>
> (u'Failed on RMD160 verification', '4bb5a049bd993068f0093497c754142cb3efcfa5',
Hi Dmitry,
Thanks for reporting, that was due to them changing their tarball without changing the name, annoyingly. I updated the Manifest in my overlay now, and it should all be working again.
> P.S. Is this a right place to post this?
Exactly the right place. Sorry I took so long to reply.

The modified kernel-2 eclass worked great for me. I tested it with the latest gentoo-sources and it emerged, compiled and ran fine. I for one would like to see it committed.
(In reply to comment #69)
> For ACCEPT_LICENSES to differentiate between linux and linux-libre, the
> linux sources ebuilds would have to include in LICENSE something else
> identifying it as containing non-FSF-approved bits.
Perhaps something like LICENSE="GPL-2 !deblob? ( freedist )" could be used.

I haven't heard from dsd for nearly 3 months now, so I'm going to poke this a bit.
dsd:
Do you have any actual objections to this going in? If not, I'm going to merge it.
Tim:
1. Please update your ebuilds to reflect 2.6.33
2. mark your old attachments here as obsolete if they are.
Nick:
1. Please update your eclass changes for the 4th generation of the deblob script.
2. mark your old attachments here as obsolete if they are.
3. Have you tested the deblob eclass changes on ALL of the kernel source packages in the tree?
3.1. If you have not, please introduce a whitelist flag of the ebuilds that we can deblob safely.
Vincent:
I had updated the LICENSE block in the kernel to reflect that it contains blobs covered under freedist for the moment.
Your blogpost today misrepresents the kernel team. They did not refuse ("for no reason" you claim), that is clearly evident by the fact that this bug is still open, it's simply that there was insufficient discussion.

(In reply to comment #75)
> 3. Have you tested the deblob eclass changes on ALL of the kernel source
> packages in the tree?
I'd be happy to do some testing, to fill in any gaps. Are you just interested in the latest-stable for each package?
> Vincent:
> I had updated the LICENSE block in the kernel to reflect that it contains blobs
> covered under freedist for the moment.
> Your blogpost today misrepresents the kernel team. They did not refuse ("for no
> reason" you claim), that is clearly evident by the fact that this bug is still
> open, it's simply that there was insufficient discussion.
Many thanks for updating the license. Sorry, I hadn't meant to misrepresent the team, I (incorrectly) assumed that Greg's refusal was the final word.

(In reply to comment #76)
> I'd be happy to do some testing, to fill in any gaps. Are you just interested
> in the latest-stable for each package?
Latest stable for each arch, and latest ~arch for each.

Created attachment 226395 [details]
libregentoo-sources-2.6.33 ebuild
> Tim:
> 1. Please update your ebuilds to reflect 2.6.33
> 2. mark your old attachments here as obsolete if they are.
...done.

(In reply to comment #75)
> I haven't heard from dsd for nearly 3 months now, so I'm going to poke this a
> bit.
Happy to see things moving forward - sorry I haven't looked at this yet, I've been on holiday. I'll update things and reply further tonight.

(In reply to comment #75)
> 2. mark your old attachments here as obsolete if they are.
Done, I marked all of the ebuilds as obsolete, as we're happily going the kernel-2 eclass route instead.
> 1. Please update your eclass changes for the 4th generation of the deblob
> script.
The eclass works fine with the new deblob script. The new generation is faster, so we could potentially include the extra deblob-check functionality (which will scrub files with blobs, rather than just deletes them). I haven't yet checked how long this takes (the old version took upwards of 12 hours!), but will do so tonight, and we can decide whether the extra complexity and processing time is worth it for us.
> 3. Have you tested the deblob eclass changes on ALL of the kernel source
> packages in the tree?
Not yet. What constitutes proper testing? Checking that all of the kernel sources are installed sanely with the new eclass changes? Checking that they build for me? Checking that they run for me? I'm happy to do so, but it'll take some time, as on my hardware kernel builds take around 30 minutes to complete. I'll start with the latest stable of each kernel in sys-kernels.
> 3.1. If you have not, please introduce a whitelist flag of the ebuilds that we
> can deblob safely.
I'd rather just test everything over the next few days and we can then commit the eclass changes confident that they're robust for all the kernels, rather than rush it and use a whitelist.

(In reply to comment #80)
> > 3. Have you tested the deblob eclass changes on ALL of the kernel source
> > packages in the tree?
> 
> Not yet. What constitutes proper testing? 
I'd be happy with _every_ kernel sources package unpacking correctly, but I would prefer to know that compilation of those kernels isn't broken by the deblob. Latest stable and latest ~arch for each.
{cell,gentoo,git,hardened,mips,mm,openvz,sparc,tuxonice,usermode,vanilla,vserver,xbox,xen,zen}-sources
> > 3.1. If you have not, please introduce a whitelist flag of the ebuilds that we
> > can deblob safely.
> I'd rather just test everything over the next few days and we can then commit
> the eclass changes confident that they're robust for all the kernels, rather
> than rush it and use a whitelist.
If any of the above kernel sources fail, we're going to need a white/blacklist anyway.

Tim/Nick:
your 2.6.33 there still references the kernel-libre eclass that was nixed.
Is it meant to be using the deblob script or the pre-blobbed sources from upstream (like the 2.6.29.3 ebuild).
I think presenting BOTH choices of blob-removal and pre-trimmed would be a good move.

(In reply to comment #80)
> The eclass works fine with the new deblob script. The new generation is faster,
> so we could potentially include the extra deblob-check functionality (which
> will scrub files with blobs, rather than just deletes them). I haven't yet
> checked how long this takes (the old version took upwards of 12 hours!), but
> will do so tonight, and we can decide whether the extra complexity and
> processing time is worth it for us.
I tested the new deblob-check script, and while much faster and lighter, it still took about 30 minutes to run against the 2.6.31 kernel, compared to 30 seconds for the basic deblob script. So I think we should just keep things how they are at present (of course we could create another flag - deblob-thorough or similar, but I expect this is of too limited interest to be worthwhile).
(In reply to comment #82)
> Tim/Nick:
> your 2.6.33 there still references the kernel-libre eclass that was nixed.
> Is it meant to be using the deblob script or the pre-blobbed sources from
> upstream (like the 2.6.29.3 ebuild).
> 
> I think presenting BOTH choices of blob-removal and pre-trimmed would be a good
> move.
As the deblob script only takes 30 seconds, and it produces essentially the same kernel as the pre-trimmed ones, I don't see much value in keeping a libre-sources ebuild around. More maintenance work for very little value. If you disagree we'll probably have to lightly rewrite the kernel-libre eclass to ensure the deblob script is not called by kernel-2 on the already deblobbed sources.
(In reply to comment #81)
> I'd be happy with _every_ kernel sources package unpacking correctly, but I
> would prefer to know that compilation of those kernels isn't broken by the
> deblob. Latest stable and latest ~arch for each.
> {cell,gentoo,git,hardened,mips,mm,openvz,sparc,tuxonice,usermode,vanilla,vserver,xbox,xen,zen}-sources
OK, I've tested latest unpack and build of stable x86 vanilla-sources and gentoo-sources so far (both worked great). I guess I'll take responsibility for testing stable x86 and ~x86 the xen, zen, vanilla, gentoo and hardened sources, for now. So Vincent (and anyone else particularly inspired), go crazy with the others, and check they unpack and build.

(In reply to comment #83)
> OK, I've tested latest unpack and build of stable x86 vanilla-sources and
> gentoo-sources so far (both worked great). I guess I'll take responsibility for
> testing stable x86 and ~x86 the xen, zen, vanilla, gentoo and hardened sources,
> for now. So Vincent (and anyone else particularly inspired), go crazy with the
> others, and check they unpack and build.
I'll take ~x86 and stable x86 (where available) for git, mm, openvz, tuxonice, usermode, vserver and xbox. Together, that will cover all the x86.

First, the ones that unpacked and built successfully:
Stable x86:
- tuxonice-sources-2.6.30-r6 Succeeded
~x86:
- git-sources-2.6.34_rc3-r10 Succeeded
- mm-sources-2.6.28_rc2-r1 Succeeded
- tuxonice-sources-2.6.33-r1 Succeeded
- vserver-sources-2.3.0.36.30.4 Succeeded
And the failures (note that the deblob script didn't exist prior to 2.6.27):
Stable x86:
- No stable {git,mm,usermode,xbox}-sources
- vserver-sources-2.2.0.7 Failed (no deblob-2.6.22)
- openvz-sources-2.6.27.3.1 Failed (see below*)
~x86:
- usermode-sources-2.6.18-r2 Failed (no deblob-2.6.18)
- xbox-sources-2.6.16.26 Failed (no deblob-2.6.16)
- openvz-sources-2.6.27.4.1 Failed (see below*)
 
* - Both openvz-sources installed fine, but none of the blobs were actually removed. I pasted the relevant output below. There's usually hundreds of messages about files it's removed, and a quick look in firmware/ confirms that it hasn't removed anything.
>>> Compiling source in /var/tmp/portage/sys-kernel/openvz-sources-2.6.27.3.1/work/linux-2.6.27-openvz-chistyakov.1 ...
>>> Running deblob script ...
/var/tmp/portage/sys-kernel/openvz-sources-2.6.27.3.1/distdir/: /var/tmp/portage/sys-kernel/ openvz-sources-2.6.27.3.1/distdir/: is a directory
>>> Source compiled.

I'm afraid I won't be able to test or debug things for a few days, due to a water in laptop incident (noo). Thanks alot Vincent for the testing so far - do go ahead and test other kernel sources I was planning to take if you have time.
Looks like a basic whitelist will be needed, though probably not more complex than a list of kernel versions the libre scripts are available for (I expect the openvz issues will be easily fixable).

(In reply to comment #86)
> I'm afraid I won't be able to test or debug things for a few days, due to a
> water in laptop incident (noo). Thanks alot Vincent for the testing so far - do
> go ahead and test other kernel sources I was planning to take if you have time.
I'm busy with exams this weekend, but I can do them on Monday if noone else has got to them by then.

Successes:
Stable x86
- {vanilla,gentoo}-sources tested by Nick
- hardened-sources-2.6.28-r9 Succeeded
~x86
- vanilla-sources-2.6.34_rc5 Succeeded
- gentoo-sources-2.6.33-r1 Succeeded
- hardened-sources-2.6.29 Succeeded
- zen-sources-2.6.33_p1 Succeeded
And the failures:
Stable x86
- No stable zen-sources
- xen-sources-2.6.18-r12 Failed (no deblob-2.6.18)
~x86
- xen-sources-2.6.32-r1 Failed (see below*)
* - Perhaps an inconsistency with deblob-2.6.32?
>>> Running deblob script ...
deblob-check: `--force' given too late or out of the proper sequence.
deblob-check: The order of arguments is significant, see the usage.
>>> Source compiled.

ok, so let's see about a whitelist. This also brings us to one problem/issue.
If you have a deblob'd version installed, and then want to upgrade to something that does NOT yet have a deblob script available, what course of action should be taken?
- Do like SSH's support for HPN/LPK, and abort.
- don't deblob and just display a warning.

Since the whole point on deblobing is to be and by extension stay "free", only the first can be an option, no?

I fully agree with Robert: people, who want to use indeed free sources should know their sources are either fully deblobbed, or just not installed. :)

FSF-LA must have updated the deblob-2.6.32 script, as the new one works fine, which solves the xen problem. As for openvz, they're the only ebuilds that touch KERNEL_URI, and KV_MAJOR.. etc, so I've made a patch that moves things around a little and fixes the issue. So with the new patch (to be attached):
Stable x86
- openvz-sources-2.6.27.4.1-r1 Succeeded
~x86
- Latest openvz is stable
- xen-sources-2.6.32-r1 Succeeded
I also installed one of the gentoo-sources to check that the patch didn't mess anything else up. So as far as I can see, the only problems now, are when the deblob script doesn't exist for a particular version.

Created attachment 229129 [details, diff]
Deblob patch that fixes openvz-sources issue
When installing openvz-sources, KV_MAJOR, KV_MINOR and KV_PATCH are all empty in detect_version. Also, adding the deblob script to KERNEL_URI in detect_version doesn't work with openvz-sources, as it seems to be overwritten by the ebuilds.

Can we get a merged list of the test results? Other than that, the patch looks good.

(In reply to comment #94)
> Can we get a merged list of the test results?
Stable x86:
- gentoo-sources Succeeded
- hardened-sources-2.6.28-r9 Succeeded
- openvz-sources-2.6.27.4.1-r1 Succeeded
- tuxonice-sources-2.6.30-r6 Succeeded
- vanilla-sources Succeeded
- vserver-sources-2.2.0.7 Failed (no deblob-2.6.22)
- xen-sources-2.6.18-r12 Failed (no deblob-2.6.18)
~x86:
- gentoo-sources-2.6.33-r1 Succeeded
- git-sources-2.6.34_rc3-r10 Succeeded
- hardened-sources-2.6.29 Succeeded
- mm-sources-2.6.28_rc2-r1 Succeeded
- openvz-sources-2.6.27.4.1-r1 Succeeded (same as stable)
- tuxonice-sources-2.6.33-r1 Succeeded
- usermode-sources-2.6.18-r2 Failed (no deblob-2.6.18)
- vanilla-sources-2.6.34_rc5 Succeeded
- vserver-sources-2.3.0.36.30.4 Succeeded
- xbox-sources-2.6.16.26 Failed (no deblob-2.6.16)
- xen-sources-2.6.32-r1 Succeeded
- zen-sources-2.6.33_p1 Succeeded
Excluded: 
- No stable {git,mm,usermode,xbox,zen}-sources
- No {cell,mips,sparc}-sources for x86

Ok, that leaves the following sources to test:
cell ck mips sparc
Just pass in the relevant arches for ACCEPT_KEYWORDS please.

I'm testing those other sources now.
We're going to need a whitelist somehow to say that deblob is only available for >=2.6.27.
Also, ran into a sandbox violation, how were you testing?
ACCESS DENIED fchmodat: /dev/shm/portage/sys-kernel/ck-sources-2.6.33/distdir/deblob-2.6.33
chmod: changing permissions of `/dev/shm/portage/sys-kernel/ck-sources-2.6.33/distdir/deblob-2.6.33': Permission denied
>>> Source unpacked in /dev/shm/portage/sys-kernel/ck-sources-2.6.33/work
--------------------------- ACCESS VIOLATION SUMMARY ---------------------------
LOG FILE "/var/log/sandbox/sandbox-25735.log"
VERSION 1.0
FORMAT: F - Function called
FORMAT: S - Access Status
FORMAT: P - Path as passed to function
FORMAT: A - Absolute Path (not canonical)
FORMAT: R - Canonical Path
FORMAT: C - Command Line
F: fchmodat
S: deny
P: /dev/shm/portage/sys-kernel/ck-sources-2.6.33/distdir/deblob-2.6.33
A: /dev/shm/portage/sys-kernel/ck-sources-2.6.33/distdir/deblob-2.6.33
R: /home/gentoo/distfiles/deblob-2.6.33
C: chmod +x /dev/shm/portage/sys-kernel/ck-sources-2.6.33/distdir/deblob-2.6.33 
--------------------------------------------------------------------------------

(In reply to comment #97)
> Also, ran into a sandbox violation, how were you testing?
Ah, sorry, I saw the top of that and thought it was just a
warning from the deblob script. Am I missing something in
FEATURES that makes it more verbose?
> I'm testing those other sources now.
I'd just finished when I saw you saw your message, so I'll
post this anyway:
- No stable {cell,ck,mips}-sources
- No ~sparc for sparc-sources
- cell-sources-2.6.24-r1 (~ppc) Failed (no deblob-2.6.24)
- ck-sources-2.6.33 (~x86) Succeeded
- sparc-sources-2.4.34 (sparc) Failed (no deblob-2.4.34)
- mips-sources-2.6.31.12 (~mips) Failed (see below*)
* - The mips-sources ebuilds touch SRC_URI after inheriting
kernel-2, but KERNEL_URI is included, so I'm not sure how
the deblob script dissappears.
>>> Running deblob script ... 
>>> sh: /var/tmp/portage/sys-kernel/mips-sources-2.6.31.12/distdir/deblob-2.6.31: No such file or directory

Created attachment 229169 [details, diff]
kernel-2.eclass_libre20100426.0.diff
This patch fixes all the issues I have noted so far. However, there were some pending ones in the previous patch as well. Details next.

There are a LOT of kernels that should be offering USE=deblob, but aren't.
# ACCEPT_KEYWORDS="* ~*" emerge --nodeps -pv $(find *-sources -name '*.ebuild' |sed -e 's,.*/,=,g' -e 's,.ebuild,,ドルg' |egrep -v -- '-99|-2\.[24]\.|-2.6.(1|2[0-6])|openvz-sources-2.6.32|xen-sources-2.6.31|zen-sources-2.6.33_rc' ) |grep -v deblob
(You might need to adjust the end of that grep for stuff that is in package.mask).
These are the packages that would be merged, in order:
[ebuild N ] sys-kernel/hardened-sources-2.6.29 USE="-build -symlink" 328 kB
[ebuild N ] sys-kernel/mips-sources-2.6.29.1 USE="-build -cobalt -impactdebug -ip27 -ip28 -ip30 -ip32r10k -symlink" 105 kB
[ebuild N ] sys-kernel/mips-sources-2.6.28.9-r1 USE="-build -cobalt -ip27 -ip28 -ip30 -ip32r10k -symlink" 280 kB
[ebuild N ] sys-kernel/mips-sources-2.6.31.12 USE="-build -cobalt -impactdebug -ip27 -ip28 -ip30 -ip32r10k -symlink" 0 kB
[ebuild N ] sys-kernel/mips-sources-2.6.27.21-r1 USE="-build -cobalt -ip27 -ip28 -ip30 -ip32r10k -symlink" 415 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.32.1 USE="-build -symlink" 16 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.10 USE="-build -symlink" 151 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.44 USE="-build -symlink" 428 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.35 USE="-build -symlink" 397 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.31.5 USE="-build -symlink" 108 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.32.3 USE="-build -symlink" 95 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.12 USE="-build -symlink" 195 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.31.11 USE="-build -symlink" 236 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.41 USE="-build -symlink" 422 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.32 USE="-build -symlink" 393 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.31.2 USE="-build -symlink" 79 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.32 USE="-build -symlink" 0 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.32.5 USE="-build -symlink" 128 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.29.5 USE="-build -symlink" 112 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.43 USE="-build -symlink" 427 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.39 USE="-build -symlink" 408 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.31.9 USE="-build -symlink" 224 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.34 USE="-build -symlink" 393 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.28.9 USE="-build -symlink" 0 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.31.4 USE="-build -symlink" 89 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.32.7 USE="-build -symlink" 170 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.32.2 USE="-build -symlink" 62 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.31.10 USE="-build -symlink" 236 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.45 USE="-build -symlink" 431 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.40 USE="-build -symlink" 420 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.36 USE="-build -symlink" 399 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.32.4 USE="-build -symlink" 120 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.42 USE="-build -symlink" 424 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.38 USE="-build -symlink" 403 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.31.8 USE="-build -symlink" 197 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.27.33 USE="-build -symlink" 393 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.31.3 USE="-build -symlink" 79 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.30.10 USE="-build -symlink" 137 kB
[ebuild N ] sys-kernel/vanilla-sources-2.6.32.6 USE="-build -symlink" 138 kB
[ebuild N ] sys-kernel/vserver-sources-2.3.0.36.28 USE="-build -symlink" 231 kB
[ebuild N ] sys-kernel/zen-sources-2.6.31_p11 USE="-build -symlink" 1,994 kB
[ebuild N ] sys-kernel/zen-sources-2.6.32_p5 USE="-build -symlink" 1,268 kB
Total: 125 packages (125 new), Size of downloads: 85,232 kB

I added ACCEPT_LICENSE="*" to the front of that and got (just after syncing):
-------
These are the packages that would be merged, in order:
... done!
[ebuild N ] sys-kernel/mips-sources-2.6.27.21-r1 USE="-build -cobalt -ip27 -ip28 -ip30 -ip32r10k -symlink" 415 kB
[ebuild N ] sys-kernel/mips-sources-2.6.29.1 USE="-build -cobalt -impactdebug -ip27 -ip28 -ip30 -ip32r10k -symlink" 105 kB
[ebuild N ] sys-kernel/mips-sources-2.6.31.12 USE="-build -cobalt -impactdebug -ip27 -ip28 -ip30 -ip32r10k -symlink" 0 kB
[ebuild N ] sys-kernel/mips-sources-2.6.28.9-r1 USE="-build -cobalt -ip27 -ip28 -ip30 -ip32r10k -symlink" 280 kB
Total: 126 packages (124 new, 1 in new slot, 1 reinstall), Size of downloads: 56,787 kB
--------
It's strange that we get different results, perhaps because I exclude metadata/cache from rsync? I also tried it before applying your patch (i.e with my old one), and everything had the deblob flag.
I added a couple of echos in, and it seems that all the mips-sources are failing this line (and going to the else, hence no IUSE="deblob".. etc):
if kernel_is ge 2 6 27 && [[ "${PN}" != "libre-sources" ]] ; then

(In reply to comment #101)
> It's strange that we get different results, perhaps because I exclude
> metadata/cache from rsync? I also tried it before applying your patch (i.e with my old one), and everything had the deblob flag.
Yup, was bad cache.
 
> I added a couple of echos in, and it seems that all the mips-sources are
> failing this line (and going to the else, hence no IUSE="deblob".. etc):
Thanks, this lead me to a latent bug in kernel-2 with certain sources ebuilds that had either no CKV/OKV, or did weird version things. Fixed now (affected openvz-sources, mips-sources, and zen-sources-9999).

Comment on attachment 229169 [details, diff]
kernel-2.eclass_libre20100426.0.diff
Eclass changes merged, and all -sources manifests/metadata.xml updated.

Ok, the only thing left is adding libre-sources as a new package now, for users that want pre-stripped sources. I'll try to get to that tomorrow, but I've got a very busy work day on my calendar.

(In reply to comment #104)
> Ok, the only thing left is adding libre-sources as a new package now, for users
> that want pre-stripped sources. I'll try to get to that tomorrow, but I've got
> a very busy work day on my calendar.
> 
any chance we could include the RT kernel for a libre kernel package? i have no idea how to write an ebuild, but Nick White has one in his NJW overlay. i would be happy to build and test it. maybe someday help write the ebuild, though not at this moment.
 just a suggestion, and thanks for making the available to Gentoo.
peace.

(In reply to comment #105)
> any chance we could include the RT kernel for a libre kernel package? i have
> no idea how to write an ebuild, but Nick White has one in his NJW overlay. i
> would be happy to build and test it. maybe someday help write the ebuild,
> though not at this moment.
Hi Wayne. Happily, once this is finished, it should hopefully work with pretty much any linux kernel sources. When I created the test rt-sources libre ebuild there was no wierdness, so it should all be good. It'd be good to test though, sure.
(I have a new keyboard now!) Thanks everyone for doing lots of work :-)

(In reply to comment #106)
> (In reply to comment #105)
> > any chance we could include the RT kernel for a libre kernel package? i have
> > no idea how to write an ebuild, but Nick White has one in his NJW overlay. i
> > would be happy to build and test it. maybe someday help write the ebuild,
> > though not at this moment.
> 
> Hi Wayne. Happily, once this is finished, it should hopefully work with pretty
> much any linux kernel sources. When I created the test rt-sources libre ebuild
> there was no wierdness, so it should all be good. It'd be good to test though,
> sure.
> 
> (I have a new keyboard now!) Thanks everyone for doing lots of work :-)
> 
ahoy Nick. thanks for the good news and the work. i am not sure if you mean that there is something for me to build now (besides what is in your overlay), or i should be waiting for a RT/real time libre ebuild to show up in the Gentoo portage to test? please let me know how i can help when you have the time.
thanks

Hi Nick. Thanks for the ebuilds! I'm happy to see them in main tree. Your new deblob script also works well and very fast.
Do you know what happened with the latest kernel ebuilds? Why isn't there deblob flag? The deblob script on http://www.linux-libre.fsfla.org/pub/linux-libre/releases/2.6.34-libre/ is ready since 18 of May...

(In reply to comment #108)
> Do you know what happened with the latest kernel ebuilds? Why isn't there
> deblob flag?
The eclass is hardcoded with a max version, so that if a new kernel is put in the tree before the corresponding deblob script has been written, it won't fail to merge. So it's just the case that the busy devs haven't yet noticed, or had time to check for, the release of a new script.
Robin: Perhaps for 'kernel_is gt 2 6 ${DEBLOB_MAX_VERSION} && use deblob', it could show a brief message saying that the script isn't yet available for that version, or something similar. It might help to alleviate confusion of why it won't deblob.

eclass updated.
We can't add a message like that because the only point at which we can evaluate it to decide if USE=deblob is available is during dependency calculation and output is not allowed during that phase.
Later on, during pkg_setup, if we have 'use blob', we'll get QA warnings/errors depending how strict the setup is AND we will not get the use flag from the user, because it wasn't declared in IUSE.
I still haven't gotten an updated libre-sources package, please open a new bug for it and any other source packages you want.
For future DEBLOB_MAX_VERSION bumps, please open a new bug as well.

ahoy all,
 first, thanks for the the deblob USE flag. really a great feature for us trying to use Gentoo libre.
 i am having some problems with the deblob flag enabled on some RT kernels. i tried the deblob keyword on the latest (and 2 previous) RT kernels from the pro-audio overlay, and while the non-libre versions build fine, the libre versions i have yet to compile. i get errors like these below:
 LD drivers/bluetooth/built-in.o
 CC [M] drivers/bluetooth/hci_vhci.o
 CC [M] drivers/bluetooth/btmrvl_main.o
 CC [M] drivers/bluetooth/btmrvl_debugfs.o
 CC [M] drivers/bluetooth/hci_ldisc.o
 CC [M] drivers/bluetooth/hci_h4.o
 CC [M] drivers/bluetooth/hci_bcsp.o
 CC [M] drivers/bluetooth/hci_ll.o
 LD [M] drivers/bluetooth/hci_uart.o
 CC [M] drivers/bluetooth/bpa10x.o
 CC [M] drivers/bluetooth/btusb.o
 CC [M] drivers/bluetooth/btsdio.o
 LD [M] drivers/bluetooth/btmrvl.o
make[2]: *** No rule to make target `drivers/bluetooth/btmrvl_sdio.c', needed by `drivers/bluetooth/btmrvl_sdio.o'. Stop.
make[1]: *** [drivers/bluetooth] Error 2
make: *** [drivers] Error 2
 CC [M] drivers/media/video/cx25840/cx25840-core.o
 CC [M] drivers/media/video/cx25840/cx25840-audio.o
make[4]: *** No rule to make target `drivers/media/video/cx25840/cx25840-firmware.o', needed by `drivers/media/video/cx25840/cx25840.o'. Stop.
make[3]: *** [drivers/media/video/cx25840] Error 2
make[2]: *** [drivers/media/video] Error 2
make[1]: *** [drivers/media] Error 2
make: *** [drivers] Error 2
 LD drivers/net/built-in.o
 CC [M] drivers/net/mii.o
 CC [M] drivers/net/ipg.o
 CC [M] drivers/net/jme.o
 CC [M] drivers/net/sis190.o
 CC [M] drivers/net/yellowfin.o
 CC [M] drivers/net/ns83820.o
make[2]: *** No rule to make target `drivers/net/bnx2.c', needed by `drivers/net/bnx2.o'. Stop.
make[1]: *** [drivers/net] Error 2
make: *** [drivers] Error 2
in the case of the Bluetooth error, disabling Bluetooth in .config solves that for now. i have been trying to track down the various config options to disable, but hunting via "menu makeconfig", i have not been able to disable enough options to let the kernel build yet.
	i read the log for the deblob'd kernel install (via elogv), and i could not find where to post bugs/errors. a quick google search also turned up nothing on the errors. i figured this was a good place to start.
thanks again.
peace

wayne:
1. please open a new bug.
2. please state exact which kernel sources and version.

(In reply to comment #112)
> wayne:
> 1. please open a new bug.
> 2. please state exact which kernel sources and version.
> 
thanks and apologies. first time reporter, long time listener ;)
peace

(In reply to comment #112)
> wayne:
> 1. please open a new bug.
> 2. please state exact which kernel sources and version.
> 
OK, here is the bug. also happening with the latest gentoo-sources as well.
http://bugs.gentoo.org/show_bug.cgi?id=324505 

ahoy all,
 has anyone had trouble emerging the latest stable sys-kernel/gentoo-sources-2.6.37-r4 with the deblob USE flag enabled? i am getting the following error:
 cp deblob-.. failed
Call stack: │
│ ebuild.sh, line 56: Called src_unpack │
│ environment, line 3030: Called kernel-2_src_unpack │
│ environment, line 2231: Called die │
│The specific snippet of code: │
│ cp "${DISTDIR}/${DEBLOB_A}" "${T}" || die "cp ${DEBLOB_A} failed"; │
│ │
│If you need support, post the output of 'emerge --info =sys-kernel/gentoo-sources-2.6.37-r4', │
│the complete build log and the output of 'emerge -pqv =sys-kernel/gentoo-sources-2.6.37-r4'. │
│The complete build log is located at '/var/log/portage/sys-kernel:gentoo-sources-2.6.37-r4:20110411-163140.log'. │
│The ebuild environment file is located at '/var/tmp/portage/sys-kernel/gentoo-sources-2.6.37-r4/temp/environment'. │
│S: '/var/tmp/portage/sys-kernel/gentoo-sources-2.6.37-r4/work/linux-2.6.37-gentoo-r4'
it looks like the deblob script is missing? is the libre kernel just behind the current release? thanks in advance.
peace, w

(In reply to comment #115)
> has anyone had trouble emerging the latest stable
> sys-kernel/gentoo-sources-2.6.37-r4 with the deblob USE flag enabled? i am
> getting the following error:
Hi Wayne, yeah, this is bug 359865, see there for progress.

(In reply to comment #116)
> (In reply to comment #115)
> > has anyone had trouble emerging the latest stable
> > sys-kernel/gentoo-sources-2.6.37-r4 with the deblob USE flag enabled? i am
> > getting the following error:
> 
> Hi Wayne, yeah, this is bug 359865, see there for progress.
thanks Nick. i missed that in my "libre deblob" bugzilla search.