[opensuse-kernel] procedure to extract & build btrfs-unstable module & progs against opensuse Kernel:HEAD kernel sources?
I'm running uname -a Linux TEST 2.6.33-rc6-12-xen #1 SMP 2010-02-05 18:11:23 +0100 x86_64 x86_64 x86_64 GNU/Linux and have, ls -al /usr/src | grep rc6 lrwxrwxrwx 1 root root 19 2010-02-07 09:53 linux -> linux-2.6.33-rc6-12 drwxr-xr-x 24 root root 4096 2010-02-06 07:04 linux-2.6.33-rc6-12/ drwxr-xr-x 3 root root 4096 2010-02-06 01:09 linux-2.6.33-rc6-12-obj/ I'd like to experiment with *this* kernel and latest/unstable btrfs. Although the btrfs wiki (http://btrfs.wiki.kernel.org/index.php/Btrfs_source_repositories) refers to a 'standalone' btrfs source tree, it's apparently ~ 1 year out of date, and no longer maintained. Afaict, the latest btrfs sources are only available in git clone git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable.git git clone git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs-unstable.git that pull a complete kernel tree ... Reading Git docs/howtos, I note: "With Git it is not possible to pull only a single directory, because of the hash code nature of Git, you must take all or nothing.", so I guess I'm stuck with pulling the whole non-opensuse kernel tree, and either patching it with opensuse's patches etc, or extracting all (?) the btrfs-unstable bits, and swapping them into the opensuse kernel sources. What would be the "right procedure" for extracting the btrfs & btrfs-progs sources from the kernel.org trees, and building the module/progs against my in-place, fully-patched, opensuse kernel's sources? -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Sunday 2010-02-07 19:06, 0bo0 wrote:
Reading Git docs/howtos, I note: "With Git it is not possible to pull only a single directory, because of the hash code nature of Git, you must take all or nothing."
Well, the hash code nature actually allows a party to download single bits. The top commit, root dir listing, fs/ listing, btrfs/ listing and btrfs files amount to about ~60 objects only (out of the 2 million that the entire history is approaching). It just so is that this ultrasparse/shallow download mode is not attainable using typical command lines currently. There is only clone --depth=1 for now; I guess this is because many of the git subcommands expect to work with a commit that has a complete filetree attached at least, but I see no reason why it should not be possible to download few objects without the intention to create a .git/ control dir afterwards.
, so I guess I'm stuck with pulling the whole non-opensuse kernel tree, and either patching it with opensuse's patches etc, or extracting all (?) the btrfs-unstable bits, and swapping them into the opensuse kernel sources.
You make a diff between kernel_a:fs/btrfs/ and btrfs_clone:fs/btrfs/, and apply that on top of the suse one. And *praying* that the new code does not rely on an incompatible change of a struct foo that the rest of the old kernel depends on at the same time! -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Sun, Feb 7, 2010 at 1:09 PM, Jan Engelhardt
Well, the hash code nature actually allows a party to download single bits. The top commit, root dir listing, fs/ listing,
You make a diff between kernel_a:fs/btrfs/ and btrfs_clone:fs/btrfs/, and apply that on top of the suse one.
I'll give that a try.
And *praying* that the new code does not rely on an incompatible change of a struct foo that the rest of the old kernel depends on at the same time!
Clear. Which is why I'd asked on this list a couple of days ago whether an opensuse/obs repo tracking the btrfs-unstable changes already exists. that'd be preferable ... Lots of useful changes (e.g., additional btrfsctl options in the tools, etc) are being rolled into the unstable tree, but not promoted to the stable one that, i think, opensuse Kernel:HEAD builds track. I'm not interested in yet another "just use the vanilla kernels" discussion@ irc, lists, etc -- and would much prefer to stay with *suse built/managed kernel development. Thanks. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Sun, Feb 7, 2010 at 1:43 PM, 0bo0 <0.bugs.only.0@gmail.com> wrote:
I'll give that a try.
as a first step, i tried to rebuild the btrfs module for the in-place kernel -- using unmodified, repo-installed source. e.g., cd /usr/src/linux-2.6.33-rc6-12 make mrproper cd /usr/src/linux-2.6.33-rc6-12-obj/x86_64/xen make mrproper make oldconfig make prepare make modules_prepare make SUBDIRS=scripts/mod make SUBDIRS=fs/btrfs/ modules after which, ls -al fs/btrfs/btrfs.ko \ /lib/modules/2.6.33-rc6-12-xen/kernel/fs/btrfs/btrfs.ko \ -rw-r--r-- 1 root root 8071997 2010-02-07 15:43 fs/btrfs/btrfs.ko -rw-r--r-- 1 root root 825136 2010-02-06 01:25 /lib/modules/2.6.33-rc6-12-xen/kernel/fs/btrfs/btrfs.ko that's a rather large difference! guessing it's debug symbols, I strip fs/btrfs/btrfs.ko ls -al fs/btrfs/btrfs.ko -rw-r--r-- 1 root root 464776 2010-02-07 15:43 fs/btrfs/btrfs.ko but size is still off from the original kernel mod. any hints as to where the difference lies? compiler flags maybe? thanks. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Sun, Feb 7, 2010 at 3:47 PM, 0bo0 <0.bugs.only.0@gmail.com> wrote:
any hints as to where the difference lies? compiler flags maybe?
agh! --- strip xxx +++ strip --strip-debug xxx does the trick. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 02/07/2010 10:20 PM, 0bo0 wrote:
On Sun, Feb 7, 2010 at 3:47 PM, 0bo0 <0.bugs.only.0@gmail.com> wrote:
any hints as to where the difference lies? compiler flags maybe?
agh!
--- strip xxx +++ strip --strip-debug xxx
does the trick.
Have you looked at the diff in the actual code between Kernel:HEAD and btrfs-unstable? I see a 126 line patch with extremely minor changes. -Jeff -- Jeff Mahoney SUSE Labs -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Mon, Feb 8, 2010 at 8:57 AM, Jeff Mahoney
Have you looked at the diff in the actual code between Kernel:HEAD and btrfs-unstable?
I see a 126 line patch with extremely minor changes.
Yes, I did ... and discovered that the features/fixes I'm after have _not_ been, in fact, added to the unstable tree, but rather exist in yet another, unmerged location ... "Chris does not merge patches into the tree until they are pushed to Linus. Sometimes he creates "experimental" branches with code for testing but I don't think he has done that recently. You can find proposed unmerged patches at: http://patchwork.kernel.org/project/linux-btrfs/list/" Until that message, I'd not yet heard of the 'patchwork' site. :-( The btrfs wiki, referenced even in the *suse man pages, is, at best, a bit sparse on such info ... The good news is that, as you've pointed out, the diffs are not that significant ... once I figured out the right steps to the process, both the unstable module and tools build against the Suse kernel sources fine: modinfo btrfs filename: /lib/modules/2.6.33-rc6-12-xen/kernel/fs/btrfs/btrfs.ko license: GPL srcversion: 14CB8D6D8EF3012C156FDC1 depends: libcrc32c,zlib_deflate vermagic: 2.6.33-rc6-12-xen SMP mod_unload modversions Xen ls -al /usr/local/bin/btrfs* -rwxr-xr-x 1 root root 655556 2010-02-07 19:33 /usr/local/bin/btrfsck -rwxr-xr-x 1 root root 570459 2010-02-07 19:33 /usr/local/bin/btrfsctl -rwxr-xr-x 1 root root 576799 2010-02-07 19:33 /usr/local/bin/btrfs-debug-tree -rwxr-xr-x 1 root root 574634 2010-02-07 19:33 /usr/local/bin/btrfs-map-logical -rwxr-xr-x 1 root root 573254 2010-02-07 19:33 /usr/local/bin/btrfs-show -rwxr-xr-x 1 root root 569360 2010-02-07 19:33 /usr/local/bin/btrfs-vol btrfs-show Label: TEST uuid: 85aa9ac8-0089-4dd3-b8b2-3c0cbb96c924 Total devices 4 FS bytes used 32.00KB devid 3 size 931.51GB used 2.01GB path /dev/sdc devid 4 size 931.51GB used 2.01GB path /dev/sdd devid 1 size 931.51GB used 2.03GB path /dev/sda devid 2 size 931.51GB used 2.01GB path /dev/sdb Btrfs v0.19-4-gab8fb4c Atm I'm digging through the patchwork patches, and pulling/applying one at a time ... as well as stress-testing the btrfs arrays. Even though it'd be useful to "get out in front" and be testing btrfs-unstable+patches against *Suse Kernel:HEAD I'm guessing that having that all nice-n-tidy in an obs repo isn't likely to happen ... Thanks. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 02/08/2010 12:14 PM, 0bo0 wrote:
On Mon, Feb 8, 2010 at 8:57 AM, Jeff Mahoney
wrote: Have you looked at the diff in the actual code between Kernel:HEAD and btrfs-unstable?
I see a 126 line patch with extremely minor changes.
Yes, I did ... and discovered that the features/fixes I'm after have _not_ been, in fact, added to the unstable tree, but rather exist in yet another, unmerged location ...
"Chris does not merge patches into the tree until they are pushed to Linus. Sometimes he creates "experimental" branches with code for testing but I don't think he has done that recently.
You can find proposed unmerged patches at:
http://patchwork.kernel.org/project/linux-btrfs/list/"
Until that message, I'd not yet heard of the 'patchwork' site. :-( The btrfs wiki, referenced even in the *suse man pages, is, at best, a bit sparse on such info ...
The good news is that, as you've pointed out, the diffs are not that significant ... once I figured out the right steps to the process, both the unstable module and tools build against the Suse kernel sources fine:
modinfo btrfs filename: /lib/modules/2.6.33-rc6-12-xen/kernel/fs/btrfs/btrfs.ko license: GPL srcversion: 14CB8D6D8EF3012C156FDC1 depends: libcrc32c,zlib_deflate vermagic: 2.6.33-rc6-12-xen SMP mod_unload modversions Xen
ls -al /usr/local/bin/btrfs* -rwxr-xr-x 1 root root 655556 2010-02-07 19:33 /usr/local/bin/btrfsck -rwxr-xr-x 1 root root 570459 2010-02-07 19:33 /usr/local/bin/btrfsctl -rwxr-xr-x 1 root root 576799 2010-02-07 19:33 /usr/local/bin/btrfs-debug-tree -rwxr-xr-x 1 root root 574634 2010-02-07 19:33 /usr/local/bin/btrfs-map-logical -rwxr-xr-x 1 root root 573254 2010-02-07 19:33 /usr/local/bin/btrfs-show -rwxr-xr-x 1 root root 569360 2010-02-07 19:33 /usr/local/bin/btrfs-vol
btrfs-show Label: TEST uuid: 85aa9ac8-0089-4dd3-b8b2-3c0cbb96c924 Total devices 4 FS bytes used 32.00KB devid 3 size 931.51GB used 2.01GB path /dev/sdc devid 4 size 931.51GB used 2.01GB path /dev/sdd devid 1 size 931.51GB used 2.03GB path /dev/sda devid 2 size 931.51GB used 2.01GB path /dev/sdb
Btrfs v0.19-4-gab8fb4c
Atm I'm digging through the patchwork patches, and pulling/applying one at a time ... as well as stress-testing the btrfs arrays.
Even though it'd be useful to "get out in front" and be testing btrfs-unstable+patches against *Suse Kernel:HEAD I'm guessing that having that all nice-n-tidy in an obs repo isn't likely to happen ...
It's possible. There's already the drivers:filesystems btrfs package. I haven't updated it in ages, though. The big issue now is that the current btrfs is based on 2.6.32+ which fundamentally changed the way filesystem writeback happens. It's a lot of work to get it working again on older kernels and I'd argue that if you're willing to test a file system during the development stage, you're probably ok running the bleeding edge kernel anyway. -Jeff -- Jeff Mahoney SUSE Labs -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Wed, Feb 10, 2010 at 11:12 AM, Jeff Mahoney
It's possible. There's already the drivers:filesystems btrfs package. I haven't updated it in ages, though. The big issue now is that the current btrfs is based on 2.6.32+ which fundamentally changed the way filesystem writeback happens. It's a lot of work to get it working again on older kernels and I'd argue that if you're willing to test a file system during the development stage, you're probably ok running the bleeding edge kernel anyway.
personally, i'm not at all interested in getting it working on older kernels. rather, i'm interested in testing/preparing for what's coming ... soon. atm i'm running, uname -ri 2.6.33-rc6-12-xen x86_64 from Kernel:HEAD for OpenSuse12. which is, iiuc, about as "bleeding edge" as I can get within the opensuse repos. (pvops 'bleeds more', but i'm honestly not sure where in opensuse-land those kernels are ....) still, *that* kernel, built from sources pulled kernel upstream and patched w/ *suse 'magic', only include the "stable" btrfs tree. which is not the "unstable" tree which *has* been pushed in the last "for linus" branch. and most certainly does not include the "up -n- coming" fixes & capabiilities in the patchwork trees, waiting for the next merge window. my only point/suggestion is based on the fact btrfs is coming, and that -- eventually -- it will be implemented by *suse in some version of release kernels. it'd be nice to be able to have it as thoroughly vetted as possible before that release -- and I'd argue that that consists of testing the btrfs code as early as possibly in the dev cycle, not just after the merge window's been closed, and we're into rc kernels. this challenge stems from the 'policy' of having four btrfs-related trees -- stable, unstable, for-linus and patchwork (not really a tree -- yet) -- where the relevant, recent work is being done in the patchwork additions, and all of these being out of sync with the upstream mainstream kernel development, which -- of course -- is where the opensuse kernels are based. sure, there's the option of DIY'ing -- which is what I'm doing atm -- but, as you might well images, filing a bugs abt it @ *suse will get generally ignored because it's "not a supported option" &/or "not a priority", and upstream will argue that we should not use the distro's kernels ... or at least not *suse's (they strongly suggest Fedora12/13 as a better alternative :-/) -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Wed, Feb 10, 2010 at 11:44 AM, 0bo0 <0.bugs.only.0@gmail.com> wrote:
but, as you might well images, filing a bugs abt it @ *suse will get
jeesh ... one those days. that's "but, as you might well imagine, filing a bug abt it @ *suse will get" -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 02/10/2010 02:44 PM, 0bo0 wrote:
On Wed, Feb 10, 2010 at 11:12 AM, Jeff Mahoney
wrote: It's possible. There's already the drivers:filesystems btrfs package. I haven't updated it in ages, though. The big issue now is that the current btrfs is based on 2.6.32+ which fundamentally changed the way filesystem writeback happens. It's a lot of work to get it working again on older kernels and I'd argue that if you're willing to test a file system during the development stage, you're probably ok running the bleeding edge kernel anyway.
personally, i'm not at all interested in getting it working on older kernels. rather, i'm interested in testing/preparing for what's coming ... soon.
atm i'm running,
uname -ri 2.6.33-rc6-12-xen x86_64
from Kernel:HEAD for OpenSuse12.
which is, iiuc, about as "bleeding edge" as I can get within the opensuse repos. (pvops 'bleeds more', but i'm honestly not sure where in opensuse-land those kernels are ....)
still, *that* kernel, built from sources pulled kernel upstream and patched w/ *suse 'magic', only include the "stable" btrfs tree. which is not the "unstable" tree which *has* been pushed in the last "for linus" branch. and most certainly does not include the "up -n- coming" fixes & capabiilities in the patchwork trees, waiting for the next merge window.
my only point/suggestion is based on the fact btrfs is coming, and that -- eventually -- it will be implemented by *suse in some version of release kernels.
it'd be nice to be able to have it as thoroughly vetted as possible before that release -- and I'd argue that that consists of testing the btrfs code as early as possibly in the dev cycle, not just after the merge window's been closed, and we're into rc kernels. this challenge stems from the 'policy' of having four btrfs-related trees -- stable, unstable, for-linus and patchwork (not really a tree -- yet) -- where the relevant, recent work is being done in the patchwork additions, and all of these being out of sync with the upstream mainstream kernel development, which -- of course -- is where the opensuse kernels are based.
sure, there's the option of DIY'ing -- which is what I'm doing atm -- but, as you might well images, filing a bugs abt it @ *suse will get generally ignored because it's "not a supported option" &/or "not a priority", and upstream will argue that we should not use the distro's kernels ... or at least not *suse's (they strongly suggest Fedora12/13 as a better alternative :-/)
Hrm. The for-linus branch seems to be pretty outdated. The master branch was last touched a week ago and the for-linus branch was last touched 7 weeks ago. I'll get the package up and running again. I don't really want to include the patchworks patches in it since those are everything that someone has posted to the mailing list - not everything that is up for consideration. -Jeff -- Jeff Mahoney SUSE Labs -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Thu, Feb 11, 2010 at 1:00 PM, Jeff Mahoney
Hrm. The for-linus branch seems to be pretty outdated.
It is. Iiuc, some mystical combination of master branch plus patchwork patches are pushed to it ~ the time of each kernel mainline merge window. I think. Getting a straight answer on if/how/when that happens has been -- challenging.
The master branch was last touched a week ago and the for-linus branch was last touched 7 weeks ago.
Yes. *Some* development is done in the master branch. Some is not, and goes to the patchwork trees. Which when? Who knows :-/
I'll get the package up and running again.
I don't really want to include the patchworks patches in it since those are everything that someone has posted to the mailing list - not everything that is up for consideration.
I certainly see your point. Then again, some of the key 'tweaks' _are_ in patchwork. As long as a *suse repo include the sources for a 'zypper si', then, of course, pulling/applying a selection of the patches becomes more manageable, if not trivial. Fwiw, the 'use case' perspective that I try to keep in mind -- while testing around here -- is a modern, almost-bleeding-edge stack of mainline-based kernel:HEAD, Xen4 & latest KVM, with up-to-date ext4, md & btrfs (all RAID modes). I'd argue that building/testing that "core stack" has very different rationale, complexity, and associated time-horizons etc than doing the same for bleeding edge "Icon Packs & Twitter clients" ... Anyway, enuf discussion. Thanks. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 02/11/2010 04:29 PM, 0bo0 wrote:
On Thu, Feb 11, 2010 at 1:00 PM, Jeff Mahoney
wrote: Hrm. The for-linus branch seems to be pretty outdated.
It is. Iiuc, some mystical combination of master branch plus patchwork patches are pushed to it ~ the time of each kernel mainline merge window. I think. Getting a straight answer on if/how/when that happens has been -- challenging.
The master branch was last touched a week ago and the for-linus branch was last touched 7 weeks ago.
Yes. *Some* development is done in the master branch. Some is not, and goes to the patchwork trees. Which when? Who knows :-/
I'll get the package up and running again.
I don't really want to include the patchworks patches in it since those are everything that someone has posted to the mailing list - not everything that is up for consideration.
I certainly see your point. Then again, some of the key 'tweaks' _are_ in patchwork. As long as a *suse repo include the sources for a 'zypper si', then, of course, pulling/applying a selection of the patches becomes more manageable, if not trivial.
Fwiw, the 'use case' perspective that I try to keep in mind -- while testing around here -- is a modern, almost-bleeding-edge stack of mainline-based kernel:HEAD, Xen4 & latest KVM, with up-to-date ext4, md & btrfs (all RAID modes).
I'd argue that building/testing that "core stack" has very different rationale, complexity, and associated time-horizons etc than doing the same for bleeding edge "Icon Packs & Twitter clients" ...
Anyway, enuf discussion. Thanks.
Do you have a build service account? If not, get one. Then you can create a branch of the drivers:filesystems btrfs package and add all the patches you want and even have an easy repo to use as a result. -Jeff -- Jeff Mahoney SUSE Labs -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Thu, Feb 11, 2010 at 1:34 PM, Jeff Mahoney
Do you have a build service account? If not, get one. Then you can create a branch of the drivers:filesystems btrfs package and add all the patches you want and even have an easy repo to use as a result.
Sure. If when osc ls Server returned an error: HTTP Error 401: Unauthorized and login to http://api.opensuse.org stop refusing credentials that demonstrably work via web interface @ obs site ... Iiuc, that's been a problem for quite a few people for quite awhile (weeks ...). At least noone around here can get it to work. Anyway, point taken. Thanks. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
participants (3)
-
0bo0
-
Jan Engelhardt
-
Jeff Mahoney