[uyuni-devel] Feature request: Add missing Debian headers to repository metadata before signing
Hello Uyuni devel,
this is a feature request for sysadmins, managing Debian / Ubuntu
systems with Uyuni.
Although there has been a lot of work to support Ubuntu / Debian, there
is still some major thing missing.
Several metadata headers are missing for Debian packages, that causes
major issues on Debian / Ubuntu systems.
One header is the "Multi-Arch" header, that is missing. This missing
header causes packages to be installable over and over again. "apt" does
not correctly "get it", that this package is already the newest
installed and suggests these packages when searching for updates all the
time.
Also some dependencies cannot be successfully solved, because of this
missing header.
So the workaround most Debian / Ubuntu admins are doing right now, is to
- sync the data into Uyuni
- wait for spacewalk to generate the "Packages.gz" file
- start - possibly below - script to add the missing "Multi-Arch" header
to the "Packages.gz" file
- sign the file
Currently, I don't think that there are any efforts to tune the Uyuni
database to also support the missing headers of Debian packages.
So a further workaround could be to include such a script to add the
missing headers "while or after" the "Packages.gz" gets created (or has
been created), but before the file gets "signed".
This could be added somewhere within the "DebRepositoryWriter.java"
file, where the "signing" of the channel starts.
First, I only had written a script to only add the "Multi-Arch" header.
But this script below adds all headers missing for a packages.
For this to work, you - of course - need the original "Packages.gz" file
from Debian / Ubuntu with *all* headers present.
Think about it. Would be cool, if this part could also be integrated
into Uyuni.
Just an idea and still used as workaround....
Best regards,
Robert
#!/usr/bin/env python -tt
# Basically, this should only add one major header
# 'Multi-Arch' to the Debian Packages file, but now
# we add all missing headers.
#
# This only works correctly, if you are syncing the Debian repos
# directly to one channel within Spacewalk or Uyuni
#
# You also need python-debian (which is installed on
# Spacewalk server) to run this script
#
# Author: Robert Paschedag
Hi As I am from the RPM world I do not know what this multiarch field is and how it is set. Could we set it to some default value in the java code? What values can it have? How somebody define what value is correct? Maybe you can provide us a bit more background? Thanks Am Donnerstag, 23. Mai 2019, 22:01:23 CEST schrieb Robert Paschedag:
Hello Uyuni devel,
this is a feature request for sysadmins, managing Debian / Ubuntu systems with Uyuni.
Although there has been a lot of work to support Ubuntu / Debian, there is still some major thing missing.
Several metadata headers are missing for Debian packages, that causes major issues on Debian / Ubuntu systems.
One header is the "Multi-Arch" header, that is missing. This missing header causes packages to be installable over and over again. "apt" does not correctly "get it", that this package is already the newest installed and suggests these packages when searching for updates all the time.
Also some dependencies cannot be successfully solved, because of this missing header.
So the workaround most Debian / Ubuntu admins are doing right now, is to
- sync the data into Uyuni - wait for spacewalk to generate the "Packages.gz" file - start - possibly below - script to add the missing "Multi-Arch" header to the "Packages.gz" file - sign the file
Currently, I don't think that there are any efforts to tune the Uyuni database to also support the missing headers of Debian packages.
So a further workaround could be to include such a script to add the missing headers "while or after" the "Packages.gz" gets created (or has been created), but before the file gets "signed".
This could be added somewhere within the "DebRepositoryWriter.java" file, where the "signing" of the channel starts.
First, I only had written a script to only add the "Multi-Arch" header. But this script below adds all headers missing for a packages.
For this to work, you - of course - need the original "Packages.gz" file from Debian / Ubuntu with *all* headers present.
Think about it. Would be cool, if this part could also be integrated into Uyuni.
Just an idea and still used as workaround....
Best regards, Robert
#!/usr/bin/env python -tt
# Basically, this should only add one major header # 'Multi-Arch' to the Debian Packages file, but now # we add all missing headers. # # This only works correctly, if you are syncing the Debian repos # directly to one channel within Spacewalk or Uyuni # # You also need python-debian (which is installed on # Spacewalk server) to run this script # # Author: Robert Paschedag
# Version: 20181107 import sys import os from debian.deb822 import *
if len(sys.argv) < 3: print "Usage: %s
" % sys.argv[0] sys.exit(1) spacewalk_file = sys.argv[1] original_file = sys.argv[2]
if not os.path.isfile(spacewalk_file): print "Error: Inputfile '%s' not available." % spacewalk_file sys.exit(1) if not os.path.isfile(original_file): print "Error: Inputfile '%s' not available." % original_file sys.exit(1)
spacewalk_packages = {} original_packages = {}
with open(spacewalk_file, 'r') as pkgs: for pkg in Packages.iter_paragraphs(pkgs): spacewalk_packages[pkg['Package'] + pkg['Version'] + pkg['Architecture']] = pkg
with open(original_file, 'r') as orig_file: for pkg in Packages.iter_paragraphs(orig_file): p = pkg['Package'] v = pkg['Version'] a = pkg['Architecture'] if spacewalk_packages.has_key(p + v + a): # found package. Check for missing headers for header in pkg.keys(): if not header in spacewalk_packages[p + v + a].keys(): spacewalk_packages[p + v + a][header] = pkg[header]
# open new file new_package = open(spacewalk_file + '.new', 'w') for pkg in spacewalk_packages.values(): pkg.dump(new_package) new_package.write("\n") new_package.close() sys.exit(0)
uyuni-test:/usr/local/sbin #
-- Regards Michael Calmer -------------------------------------------------------------------------- Michael Calmer SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg T: +49 (0) 911 74053 0 F: +49 (0) 911 74053575 - e-mail: Michael.Calmer@suse.com -------------------------------------------------------------------------- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: uyuni-devel+unsubscribe@opensuse.org To contact the owner, e-mail: uyuni-devel+owner@opensuse.org
Hi,
I don't think that you can set some default value for that header on every package.
See also https://wiki.ubuntu.com/MultiarchSpec#Dependencies_involving_Architecture:_a...
for more information.
Robert
sent from my mobile device
-------- Originale Nachricht --------
Von: Michael Calmer
Hello Uyuni devel,
this is a feature request for sysadmins, managing Debian / Ubuntu systems with Uyuni.
Although there has been a lot of work to support Ubuntu / Debian, there is still some major thing missing.
Several metadata headers are missing for Debian packages, that causes major issues on Debian / Ubuntu systems.
One header is the "Multi-Arch" header, that is missing. This missing header causes packages to be installable over and over again. "apt" does not correctly "get it", that this package is already the newest installed and suggests these packages when searching for updates all the time.
Also some dependencies cannot be successfully solved, because of this missing header.
So the workaround most Debian / Ubuntu admins are doing right now, is to
- sync the data into Uyuni - wait for spacewalk to generate the "Packages.gz" file - start - possibly below - script to add the missing "Multi-Arch" header to the "Packages.gz" file - sign the file
Currently, I don't think that there are any efforts to tune the Uyuni database to also support the missing headers of Debian packages.
So a further workaround could be to include such a script to add the missing headers "while or after" the "Packages.gz" gets created (or has been created), but before the file gets "signed".
This could be added somewhere within the "DebRepositoryWriter.java" file, where the "signing" of the channel starts.
First, I only had written a script to only add the "Multi-Arch" header. But this script below adds all headers missing for a packages.
For this to work, you - of course - need the original "Packages.gz" file from Debian / Ubuntu with *all* headers present.
Think about it. Would be cool, if this part could also be integrated into Uyuni.
Just an idea and still used as workaround....
Best regards, Robert
#!/usr/bin/env python -tt
# Basically, this should only add one major header # 'Multi-Arch' to the Debian Packages file, but now # we add all missing headers. # # This only works correctly, if you are syncing the Debian repos # directly to one channel within Spacewalk or Uyuni # # You also need python-debian (which is installed on # Spacewalk server) to run this script # # Author: Robert Paschedag
# Version: 20181107 import sys import os from debian.deb822 import *
if len(sys.argv) < 3: print "Usage: %s
" % sys.argv[0] sys.exit(1) spacewalk_file = sys.argv[1] original_file = sys.argv[2]
if not os.path.isfile(spacewalk_file): print "Error: Inputfile '%s' not available." % spacewalk_file sys.exit(1) if not os.path.isfile(original_file): print "Error: Inputfile '%s' not available." % original_file sys.exit(1)
spacewalk_packages = {} original_packages = {}
with open(spacewalk_file, 'r') as pkgs: for pkg in Packages.iter_paragraphs(pkgs): spacewalk_packages[pkg['Package'] + pkg['Version'] + pkg['Architecture']] = pkg
with open(original_file, 'r') as orig_file: for pkg in Packages.iter_paragraphs(orig_file): p = pkg['Package'] v = pkg['Version'] a = pkg['Architecture'] if spacewalk_packages.has_key(p + v + a): # found package. Check for missing headers for header in pkg.keys(): if not header in spacewalk_packages[p + v + a].keys(): spacewalk_packages[p + v + a][header] = pkg[header]
# open new file new_package = open(spacewalk_file + '.new', 'w') for pkg in spacewalk_packages.values(): pkg.dump(new_package) new_package.write("\n") new_package.close() sys.exit(0)
uyuni-test:/usr/local/sbin #
-- Regards Michael Calmer -------------------------------------------------------------------------- Michael Calmer SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg T: +49 (0) 911 74053 0 F: +49 (0) 911 74053575 - e-mail: Michael.Calmer@suse.com -------------------------------------------------------------------------- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: uyuni-devel+unsubscribe@opensuse.org To contact the owner, e-mail: uyuni-devel+owner@opensuse.org -- To unsubscribe, e-mail: uyuni-devel+unsubscribe@opensuse.org To contact the owner, e-mail: uyuni-devel+owner@opensuse.org
Hi Robert, Thanks for reporting this. My first impression after checking our code is that we'll have to adapt spacewalk-repo-sync to take this header into account for each package. There might be also some changes needed in the database in order to support this. Anyway I'll create an issue in Uyuni to discuss the possible solutions. Regards, On 5/25/19 9:49 PM, Robert Paschedag wrote:
Hi,
I don't think that you can set some default value for that header on every package.
See also https://wiki.ubuntu.com/MultiarchSpec#Dependencies_involving_Architecture:_a...
for more information.
Robert
sent from my mobile device
-------- Originale Nachricht -------- Von: Michael Calmer
Gesendet: Fri May 24 15:50:27 GMT+02:00 2019 An: uyuni-devel@opensuse.org Betreff: Re: [uyuni-devel] Feature request: Add missing Debian headers to repository metadata before signing Hi
As I am from the RPM world I do not know what this multiarch field is and how it is set.
Could we set it to some default value in the java code? What values can it have? How somebody define what value is correct?
Maybe you can provide us a bit more background?
Thanks
Am Donnerstag, 23. Mai 2019, 22:01:23 CEST schrieb Robert Paschedag:
Hello Uyuni devel,
this is a feature request for sysadmins, managing Debian / Ubuntu systems with Uyuni.
Although there has been a lot of work to support Ubuntu / Debian, there is still some major thing missing.
Several metadata headers are missing for Debian packages, that causes major issues on Debian / Ubuntu systems.
One header is the "Multi-Arch" header, that is missing. This missing header causes packages to be installable over and over again. "apt" does not correctly "get it", that this package is already the newest installed and suggests these packages when searching for updates all the time.
Also some dependencies cannot be successfully solved, because of this missing header.
So the workaround most Debian / Ubuntu admins are doing right now, is to
- sync the data into Uyuni - wait for spacewalk to generate the "Packages.gz" file - start - possibly below - script to add the missing "Multi-Arch" header to the "Packages.gz" file - sign the file
Currently, I don't think that there are any efforts to tune the Uyuni database to also support the missing headers of Debian packages.
So a further workaround could be to include such a script to add the missing headers "while or after" the "Packages.gz" gets created (or has been created), but before the file gets "signed".
This could be added somewhere within the "DebRepositoryWriter.java" file, where the "signing" of the channel starts.
First, I only had written a script to only add the "Multi-Arch" header. But this script below adds all headers missing for a packages.
For this to work, you - of course - need the original "Packages.gz" file from Debian / Ubuntu with *all* headers present.
Think about it. Would be cool, if this part could also be integrated into Uyuni.
Just an idea and still used as workaround....
Best regards, Robert
#!/usr/bin/env python -tt
# Basically, this should only add one major header # 'Multi-Arch' to the Debian Packages file, but now # we add all missing headers. # # This only works correctly, if you are syncing the Debian repos # directly to one channel within Spacewalk or Uyuni # # You also need python-debian (which is installed on # Spacewalk server) to run this script # # Author: Robert Paschedag
# Version: 20181107 import sys import os from debian.deb822 import *
if len(sys.argv) < 3: print "Usage: %s
" % sys.argv[0] sys.exit(1) spacewalk_file = sys.argv[1] original_file = sys.argv[2]
if not os.path.isfile(spacewalk_file): print "Error: Inputfile '%s' not available." % spacewalk_file sys.exit(1) if not os.path.isfile(original_file): print "Error: Inputfile '%s' not available." % original_file sys.exit(1)
spacewalk_packages = {} original_packages = {}
with open(spacewalk_file, 'r') as pkgs: for pkg in Packages.iter_paragraphs(pkgs): spacewalk_packages[pkg['Package'] + pkg['Version'] + pkg['Architecture']] = pkg
with open(original_file, 'r') as orig_file: for pkg in Packages.iter_paragraphs(orig_file): p = pkg['Package'] v = pkg['Version'] a = pkg['Architecture'] if spacewalk_packages.has_key(p + v + a): # found package. Check for missing headers for header in pkg.keys(): if not header in spacewalk_packages[p + v + a].keys(): spacewalk_packages[p + v + a][header] = pkg[header]
# open new file new_package = open(spacewalk_file + '.new', 'w') for pkg in spacewalk_packages.values(): pkg.dump(new_package) new_package.write("\n") new_package.close() sys.exit(0)
uyuni-test:/usr/local/sbin #
-- --- Matei Albu SUSE Manager Development Team -- To unsubscribe, e-mail: uyuni-devel+unsubscribe@opensuse.org To contact the owner, e-mail: uyuni-devel+owner@opensuse.org
Hi yes, this might be needed. But in this case we should do a quick RFC (https://github.com/uyuni-project/uyuni-rfc) to design it properly. I think we need to trust the metadata before we do this and this needs a bit more implementation details. Am Montag, 27. Mai 2019, 16:12:10 CEST schrieb Matei Albu:
Hi Robert,
Thanks for reporting this.
My first impression after checking our code is that we'll have to adapt spacewalk-repo-sync to take this header into account for each package. There might be also some changes needed in the database in order to support this.
Anyway I'll create an issue in Uyuni to discuss the possible solutions.
Regards,
On 5/25/19 9:49 PM, Robert Paschedag wrote:
Hi,
I don't think that you can set some default value for that header on every package.
See also https://wiki.ubuntu.com/MultiarchSpec#Dependencies_involving_Architecture:_a...
for more information.
Robert
sent from my mobile device
-------- Originale Nachricht -------- Von: Michael Calmer
Gesendet: Fri May 24 15:50:27 GMT+02:00 2019 An: uyuni-devel@opensuse.org Betreff: Re: [uyuni-devel] Feature request: Add missing Debian headers to repository metadata before signing Hi
As I am from the RPM world I do not know what this multiarch field is and how it is set.
Could we set it to some default value in the java code? What values can it have? How somebody define what value is correct?
Maybe you can provide us a bit more background?
Thanks
Am Donnerstag, 23. Mai 2019, 22:01:23 CEST schrieb Robert Paschedag:
Hello Uyuni devel,
this is a feature request for sysadmins, managing Debian / Ubuntu systems with Uyuni.
Although there has been a lot of work to support Ubuntu / Debian, there is still some major thing missing.
Several metadata headers are missing for Debian packages, that causes major issues on Debian / Ubuntu systems.
One header is the "Multi-Arch" header, that is missing. This missing header causes packages to be installable over and over again. "apt" does not correctly "get it", that this package is already the newest installed and suggests these packages when searching for updates all the time.
Also some dependencies cannot be successfully solved, because of this missing header.
So the workaround most Debian / Ubuntu admins are doing right now, is to
- sync the data into Uyuni - wait for spacewalk to generate the "Packages.gz" file - start - possibly below - script to add the missing "Multi-Arch" header to the "Packages.gz" file - sign the file
Currently, I don't think that there are any efforts to tune the Uyuni database to also support the missing headers of Debian packages.
So a further workaround could be to include such a script to add the missing headers "while or after" the "Packages.gz" gets created (or has been created), but before the file gets "signed".
This could be added somewhere within the "DebRepositoryWriter.java" file, where the "signing" of the channel starts.
First, I only had written a script to only add the "Multi-Arch" header. But this script below adds all headers missing for a packages.
For this to work, you - of course - need the original "Packages.gz" file from Debian / Ubuntu with *all* headers present.
Think about it. Would be cool, if this part could also be integrated into Uyuni.
Just an idea and still used as workaround....
Best regards, Robert
#!/usr/bin/env python -tt
# Basically, this should only add one major header # 'Multi-Arch' to the Debian Packages file, but now # we add all missing headers. # # This only works correctly, if you are syncing the Debian repos # directly to one channel within Spacewalk or Uyuni # # You also need python-debian (which is installed on # Spacewalk server) to run this script # # Author: Robert Paschedag
# Version: 20181107 import sys import os from debian.deb822 import *
if len(sys.argv) < 3: print "Usage: %s
" % sys.argv[0] sys.exit(1) spacewalk_file = sys.argv[1] original_file = sys.argv[2]
if not os.path.isfile(spacewalk_file): print "Error: Inputfile '%s' not available." % spacewalk_file sys.exit(1) if not os.path.isfile(original_file): print "Error: Inputfile '%s' not available." % original_file sys.exit(1)
spacewalk_packages = {} original_packages = {}
with open(spacewalk_file, 'r') as pkgs: for pkg in Packages.iter_paragraphs(pkgs): spacewalk_packages[pkg['Package'] + pkg['Version'] + pkg['Architecture']] = pkg
with open(original_file, 'r') as orig_file: for pkg in Packages.iter_paragraphs(orig_file): p = pkg['Package'] v = pkg['Version'] a = pkg['Architecture'] if spacewalk_packages.has_key(p + v + a): # found package. Check for missing headers for header in pkg.keys(): if not header in spacewalk_packages[p + v + a].keys(): spacewalk_packages[p + v + a][header] = pkg[header]
# open new file new_package = open(spacewalk_file + '.new', 'w') for pkg in spacewalk_packages.values(): pkg.dump(new_package) new_package.write("\n") new_package.close() sys.exit(0)
uyuni-test:/usr/local/sbin #
-- Regards Michael Calmer -------------------------------------------------------------------------- Michael Calmer SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg T: +49 (0) 911 74053 0 F: +49 (0) 911 74053575 - e-mail: Michael.Calmer@suse.com -------------------------------------------------------------------------- SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: uyuni-devel+unsubscribe@opensuse.org To contact the owner, e-mail: uyuni-devel+owner@opensuse.org
participants (3)
-
Matei Albu
-
Michael Calmer
-
Robert Paschedag