[opensuse-factory] openSUSE:Factory git mirror update
Hi, our experimental git mirror of openSUSE:Factory got a new feature: It now makes single commits from most individual updates with some useful info in the commit message. This makes the whole history much nicer to read: https://github.com/bmwiedemann/openSUSE/commits/master In the usual case of SRs it links back to the SR on OBS, because that is where most of the discussion might have happened: https://github.com/bmwiedemann/openSUSE/commit/d1afc9699 but sometimes direct commits happen and they are shown with the commit message and a link to the OBS change https://github.com/bmwiedemann/openSUSE/commit/b2786e5c8 Ideas for future work: In theory we could also track these packages' devel projects on a 'devel' branch In a similar fashion we could also track openSUSE:Leap:15.* on stable/leap15_N branches to allow offline diff We could try to enable osc build to work from such git dirs. Either this uses the OBS server-side logic to resolve dependencies to get the usual .osc/_buildinfo* and .osc/_buildconfig* files or we re-implement or re-use OBS server logic to allow builds when offline. And then (in the foggy far future) it would be possible to make a gateway back from github pull-requests or gitlab merge-requests to submitting SRs in OBS. Or we re-think how OBS handles its source files until then. See https://www.youtube.com/watch?v=iY_ADUQiiQI&t=787 Ciao Bernhard M.
On Wed, Nov 6, 2019 at 10:46 AM Bernhard M. Wiedemann <bernhardout@lsmod.de> wrote:
Hi,
our experimental git mirror of openSUSE:Factory got a new feature: It now makes single commits from most individual updates with some useful info in the commit message.
This makes the whole history much nicer to read: https://github.com/bmwiedemann/openSUSE/commits/master
In the usual case of SRs it links back to the SR on OBS, because that is where most of the discussion might have happened:
https://github.com/bmwiedemann/openSUSE/commit/d1afc9699
but sometimes direct commits happen and they are shown with the commit message and a link to the OBS change
https://github.com/bmwiedemann/openSUSE/commit/b2786e5c8
Ideas for future work:
In theory we could also track these packages' devel projects on a 'devel' branch
In a similar fashion we could also track openSUSE:Leap:15.* on stable/leap15_N branches to allow offline diff
We could try to enable osc build to work from such git dirs. Either this uses the OBS server-side logic to resolve dependencies to get the usual .osc/_buildinfo* and .osc/_buildconfig* files or we re-implement or re-use OBS server logic to allow builds when offline.
And then (in the foggy far future) it would be possible to make a gateway back from github pull-requests or gitlab merge-requests to submitting SRs in OBS. Or we re-think how OBS handles its source files until then. See https://www.youtube.com/watch?v=iY_ADUQiiQI&t=787
This is very cool! Perhaps, if we could stand up a Pagure instance (with dist-git[1][2] maybe?), could we have it so that it creates per-package Git repos with this data? That would make it equivalent to how other distributions managing packaging in Git (Fedora, OpenMandriva, ALT, PLD, etc.) represent it. We could do interesting things like having branches mapping to Fedora, CentOS/RHEL, openSUSE, SLE, etc. so that packaging contents can be trivially compared, cherry-picked, etc. This is something that Fedora and CentOS will be doing soon leveraging the repoSpanner backend for Pagure[3][4]. I think it would be possible to also integrate Fedora/CentOS into an openSUSE Pagure and openSUSE in the Fedora Pagure instance. [1]: https://github.com/release-engineering/dist-git [2]: https://pagure.io/pagure-dist-git [3]: https://github.com/repoSpanner/repoSpanner [4]: https://lists.centos.org/pipermail/centos-devel/2019-April/017308.html -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 06/11/2019 16.59, Neal Gompa wrote:
This is very cool!
Perhaps, if we could stand up a Pagure instance (with dist-git[1][2] maybe?), could we have it so that it creates per-package Git repos with this data? That would make it equivalent to how other distributions managing packaging in Git (Fedora, OpenMandriva, ALT, PLD, etc.) represent it.
and https://aur.archlinux.org/<packetname>.git OTOH other distros use a single repo: https://github.com/gentoo/gentoo https://github.com/felixonmars/archlinux-packages https://git.savannah.gnu.org/git/guix.git https://github.com/NixOS/nixpkgs https://github.com/void-linux/void-packages None of those use rpm though. I can imagine that a single repo has advantages when you want multiple related changes to go in together or do mass-updates across a distribution. Per packet-repos certainly have their advantages, too. E.g. you can delete packages. Permission handling might be more straight-forward. It would even be possible to have both variants in parallel with auto-sync. Ciao Bernhard M.
Hi, you might have read before about my git mirror of our openSUSE:Factory OBS repo: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/2... Meanwhile, there was also Alberto's approach with obsgit https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/2... but that is not why I am writing this. I wanted to tell about the per-package split repos that had been proposed by Neal Gompa back then in 2019. Those are now all created on our pagure and auto-updated alongside the large repo. Here are some examples: 1. https://code.opensuse.org/package/kernel-source/commits/master gives a nice view of when what kernel version entered Factory and you can click any of the commits to find the SR message and a link to the SR on OBS. 2. https://code.opensuse.org/package/bash/blob/master/f/bash.spec#_200 Can link to individual lines to help discussions with remote people 3. https://code.opensuse.org/package/bash/blame/bash.spec?identifier=master#_90 Git blame is fast, both online and offline. 4. https://code.opensuse.org/package/_project/c/b7c851ac89f186ecf2a4261376b0ef9... also tracks prjconf changes More Advantages are: * easy to do full-text search * offline access to history * distributed repo * increased visibility of changes * ability to download patch files without account * cryptograpically signed commits to impede tampering with history * even works on Thursdays (when we do maintenance of our infrastructure) * does not carry tarballs and other large binaries, so the whole repo is below 7GB atm and as before there are still countless ways on how we could extend that: * track Leap, Backports and devel repos * allow PRs to package repos and proxy those into OBS for a more gitty workflow and wouldn't it be nice, if we could have packages in OBS without tarballs, with a reference to an upstream git repo and commit id? It would still need a git clone --depth 1 in OBS, to ensure that we can build even if the original git server/commit vanishes. But having a well-organized clone of all our upstream git repos would still be a nice thing for other reasons, e.g. to continue maintaining orphaned packages. What are your thoughts and ideas? Ciao Bernhard M.
On Saturday 2021-10-02 10:12, Bernhard M. Wiedemann wrote:
wouldn't it be nice, if we could have packages in OBS without tarballs, with a reference to an upstream git repo and commit id? It would still need a git clone --depth 1 in OBS, to ensure that we can build even if the original git server/commit vanishes.
``--depth 1`` mandates that the vanishing project has correctly made a git tag (or at least, a branch head) for every release. There is bound to be someone who messes that up too, so on the OBS side, there will need to be a way to use the implied default of infinite depth.
On 02/10/2021 11.08, Jan Engelhardt wrote:
On Saturday 2021-10-02 10:12, Bernhard M. Wiedemann wrote:
wouldn't it be nice, if we could have packages in OBS without tarballs, with a reference to an upstream git repo and commit id? It would still need a git clone --depth 1 in OBS, to ensure that we can build even if the original git server/commit vanishes.
``--depth 1`` mandates that the vanishing project has correctly made a git tag (or at least, a branch head) for every release. There is bound to be someone who messes that up too, so on the OBS side, there will need to be a way to use the implied default of infinite depth.
It turns out, it is possible to shallow-clone a commit-id since git 2.5 Found out via https://stackoverflow.com/questions/31278902/how-to-shallow-clone-a-specific... My test shows that it works with github: git init git remote add origin https://github.com/bmwiedemann/openSUSE.git git fetch --depth 1 origin e07bd24453d6bcdeb0b7d0ae2ab2ec5edc9282b4 git checkout FETCH_HEAD du -s .git 227676 .git and then we can tag it with release-numbers on our side to prevent garbage-collection and as a pointer to the releases using that commit-id. Ciao Bernhard M.
On Sun, Oct 3, 2021 at 2:43 AM Bernhard M. Wiedemann <bernhardout@lsmod.de> wrote:
On 02/10/2021 11.08, Jan Engelhardt wrote:
On Saturday 2021-10-02 10:12, Bernhard M. Wiedemann wrote:
wouldn't it be nice, if we could have packages in OBS without tarballs, with a reference to an upstream git repo and commit id? It would still need a git clone --depth 1 in OBS, to ensure that we can build even if the original git server/commit vanishes.
``--depth 1`` mandates that the vanishing project has correctly made a git tag (or at least, a branch head) for every release. There is bound to be someone who messes that up too, so on the OBS side, there will need to be a way to use the implied default of infinite depth.
It turns out, it is possible to shallow-clone a commit-id since git 2.5 Found out via https://stackoverflow.com/questions/31278902/how-to-shallow-clone-a-specific...
My test shows that it works with github: git init git remote add origin https://github.com/bmwiedemann/openSUSE.git git fetch --depth 1 origin e07bd24453d6bcdeb0b7d0ae2ab2ec5edc9282b4 git checkout FETCH_HEAD du -s .git 227676 .git
and then we can tag it with release-numbers on our side to prevent garbage-collection and as a pointer to the releases using that commit-id.
Pagure uses libgit2 underneath through pygit2, so as soon as it's supported in libgit2, we can have the capability: https://github.com/libgit2/libgit2/issues/3058 -- 真実はいつも一つ！/ Always, there's only one truth!
Bernhard M. Wiedemann wrote:
[...] I wanted to tell about the per-package split repos that had been proposed by Neal Gompa back then in 2019. Those are now all created on our pagure and auto-updated alongside the large repo.
Cool project, thanks for continuing that :-) Would it be possible to get rid of the dot files with OBS metadata somehow? Feels ugly :-) Maybe a tag could be used instead?
and wouldn't it be nice, if we could have packages in OBS without tarballs, with a reference to an upstream git repo and commit id?
Or just branch the upstream repo and add the packaging related changes on top. Some packages more or less already went that route. Eg. dracut: https://code.opensuse.org/package/dracut/blob/master/f/_service So for packages like that it would make sense to not mirror what's in OBS into code.o.o but rather use the repo listed in _servicedata directly. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer HRB 36809 (AG Nürnberg)
On Mon, Oct 4, 2021 at 4:40 AM Ludwig Nussel <ludwig.nussel@suse.de> wrote:
Bernhard M. Wiedemann wrote:
[...] I wanted to tell about the per-package split repos that had been proposed by Neal Gompa back then in 2019. Those are now all created on our pagure and auto-updated alongside the large repo.
Cool project, thanks for continuing that :-) Would it be possible to get rid of the dot files with OBS metadata somehow? Feels ugly :-) Maybe a tag could be used instead?
and wouldn't it be nice, if we could have packages in OBS without tarballs, with a reference to an upstream git repo and commit id?
Or just branch the upstream repo and add the packaging related changes on top. Some packages more or less already went that route. Eg. dracut: https://code.opensuse.org/package/dracut/blob/master/f/_service
So for packages like that it would make sense to not mirror what's in OBS into code.o.o but rather use the repo listed in _servicedata directly.
That would make it considerably more difficult to examine the packaging alone. And in general, I'm not a huge fan of source-git/merged source trees/etc style where the packaging and the software packaging are in one place. It leads to complicated Git histories and makes it difficult for people to figure out what's going on. It also makes it too easy to fork a project unintentionally. And in the dracut case, the long-term goal is to eliminate openSUSE/dracut entirely. We should *not* have so many patches that we want to do something like this. I don't want to have to go through another exercise where I had to work to pay off ~10 years of tech debt to eliminate ~600 patches that openSUSE dracut had. I'd like for openSUSE/systemd to go away eventually for similar reasons. -- 真実はいつも一つ！/ Always, there's only one truth!
participants (4)
-
Bernhard M. Wiedemann -
Jan Engelhardt -
Ludwig Nussel -
Neal Gompa