[opensuse] non-standard characters in 11.3 rpm data causing problems with createrepo
Guys,
Just fyi, creatrepo throws info warning on the foreign characters included in
rpm metadata. Earlier versions of createrepo would choke on these. You can see
the packages and characters causing the problems in the createrepo output below:
464/1381 - i586/gcalctool-5.30.2-1.2.i586.rpm
iso-8859-1 encoding on A French dictionary for the ASpell spell checker.
Authors:
--------
R�mi Vanicat
David C. Rankin wrote:
Just fyi, creatrepo throws info warning on the foreign characters included in rpm metadata. Earlier versions of createrepo would choke on these. You can see the packages and characters causing the problems in the createrepo output below: [snip] The repositories are still built, but something needs to be done to standardize the allowed character set for rpm metadata or I guess the whole OS needs to be updated to handle the non-standard one.
Us dratted non-domestics, huh, with our funny names and things :) I thought something *was* done to solve issues like this. It's called Unicode. What locale settings do you have? Are they inherited from some old ASCII or Latin1 system, perhaps? Or are you saying the repo software is broken? Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 10/13/2010 04:55 AM, Dave Howorth wrote:
Us dratted non-domestics, huh, with our funny names and things :)
Oh no, you'd be surprised how many of us domestics use the funny things too.
I thought something *was* done to solve issues like this. It's called Unicode. What locale settings do you have? Are they inherited from some old ASCII or Latin1 system, perhaps?
That is a good question. It looks like there is a wide difference among distributions: 11.3 06:56 zephyr:~> noc /etc/sysconfig/language RC_LANG="en_US.UTF-8" RC_LC_ALL="" RC_LC_MESSAGES="" RC_LC_CTYPE="" RC_LC_COLLATE="POSIX" RC_LC_TIME="" RC_LC_NUMERIC="" RC_LC_MONETARY="" RC_LC_PAPER="" ROOT_USES_LANG="yes" AUTO_DETECT_UTF8="no" INSTALLED_LANGUAGES="en_US" INPUT_METHOD="" 11.0 07:02 alchemy:/home/david/suse/pkg/sle_11 # noc /etc/sysconfig/language INPUT_METHOD="" RC_LANG="en_US.UTF-8" RC_LC_ALL="" RC_LC_MESSAGES="" RC_LC_CTYPE="" RC_LC_COLLATE="" RC_LC_TIME="" RC_LC_NUMERIC="" RC_LC_MONETARY="" RC_LC_PAPER="" ROOT_USES_LANG="ctype" AUTO_DETECT_UTF8="no" INSTALLED_LANGUAGES="" Arch 07:04 archangel:~> noc /etc/locale.gen en_US.UTF-8 UTF-8 en_US ISO-8859-1
Or are you saying the repo software is broken?
No, it still works, it just throws out chunks of metadata when it hits the funky characters while createrepo is creating/updating the rpms: 07:05 nirvana:/home/backup/rpms/data> rpm --version rpm (RPM) 5.1.9 07:05 nirvana:/home/backup/rpms/data> createrepo --version createrepo 0.9.8 I guess I could always add 'createrepo --update -d ${REPODIR} > /dev/null', but that seems like a cludge when createrepo has run clean for years before. What locale setting do I need to change to eliminate the problem? -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
* David C. Rankin (drankinatty@suddenlinkmail.com) [20101013 14:11]:
What locale setting do I need to change to eliminate the problem?
You would have to change locale based on the encoding used in the rpm data (most probably the specs) which I guess is rather impractical. I'd say the OBS should run an additional check on .spec files to ensure that Author names are encoded in utf8. I'll discuss that with collegues. Philipp -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
* Philipp Thomas (pth@suse.de) [20101019 17:13]:
I'd say the OBS should run an additional check on .spec files to ensure that Author names are encoded in utf8. I'll discuss that with collegues.
Actually rpmlint *does* give a warning when a spec file isn't encoded in utf8.But as it's only a warning, you should just bug the maintainer of a given package to fix the spec file. Philipp -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 10/19/2010 10:32 AM, Philipp Thomas wrote:
* Philipp Thomas (pth@suse.de) [20101019 17:13]:
I'd say the OBS should run an additional check on .spec files to ensure that Author names are encoded in utf8. I'll discuss that with collegues.
Thanks Philipp, That sound like the smart way to standardize the spec file.
Actually rpmlint *does* give a warning when a spec file isn't encoded in utf8.But as it's only a warning, you should just bug the maintainer of a given package to fix the spec file.
Philipp
I like your first thought better. Standardizing the content seems more logical than trying to chase down each package maintainer that has a spec throwing character encoding warnings. I don't mind bugging the 4-5 maintainers mentioned in the original post, but among your colleagues, it would be nice if packagers were aware that non-utf8 encoding causes warnings and that their use is specs should be avoided. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tue, 19 Oct 2010 16:30:27 -0500, "David C. Rankin"
I like your first thought better. Standardizing the content seems more logical than trying to chase down each package maintainer that has a spec throwing character encoding warnings.
That would require rpmlint to treat it as an error with a high enough badness to stop package creation. Make a feature request on openFATE, disscuss it on opensuse-factory or make an enhancement request for rpmlint via bugzilla. Philipp -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 10/19/2010 10:32 AM, Philipp Thomas wrote:
* Philipp Thomas (pth@suse.de) [20101019 17:13]:
I'd say the OBS should run an additional check on .spec files to ensure that Author names are encoded in utf8. I'll discuss that with collegues.
Actually rpmlint *does* give a warning when a spec file isn't encoded in utf8.But as it's only a warning, you should just bug the maintainer of a given package to fix the spec file.
Philipp
Philipp, While we're on the topic, why does suse still use PreReq in packaging? The yum/creatrepo upstream guys had to create a patch to make createrepo 0.9.8 work with my suse repos. Without it, the following errors were received on metadata update: <snip> AttributeError: 'module' object has no attribute 'RPMSENSE_PREREQ' The patch required was: --- yum/packages.py.orig +++ yum/packages.py @@ -1353,11 +1353,14 @@ is a pre-requires or a not""" # FIXME this should probably be put in rpmUtils.miscutils since # - that's what it is + RPMSENSE_PREREQ = (1 << 6) + RPMSENSE_SCRIPT_PRE = (1 << 9) + RPMSENSE_SCRIPT_POST = (1 << 10) if flag is not None: # Note: RPMSENSE_PREREQ == 0 since rpm-4.4'ish - if flag & (rpm.RPMSENSE_PREREQ | - rpm.RPMSENSE_SCRIPT_PRE | - rpm.RPMSENSE_SCRIPT_POST): + if flag & (RPMSENSE_PREREQ | + RPMSENSE_SCRIPT_PRE | + RPMSENSE_SCRIPT_POST): return 1 return 0 There comments were: "only suse uses PreReq nowadays", so the question is "Why?" -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tue, 19 Oct 2010 16:49:42 -0500, "David C. Rankin"
While we're on the topic, why does suse still use PreReq in packaging?
I don't know. Maybe Darix (CCed, that's why I quoted in full) knows the answer to that?
yum/creatrepo upstream guys had to create a patch to make createrepo 0.9.8 work with my suse repos. Without it, the following errors were received on metadata update:
<snip> AttributeError: 'module' object has no attribute 'RPMSENSE_PREREQ'
The patch required was:
--- yum/packages.py.orig +++ yum/packages.py @@ -1353,11 +1353,14 @@ is a pre-requires or a not""" # FIXME this should probably be put in rpmUtils.miscutils since # - that's what it is + RPMSENSE_PREREQ = (1 << 6) + RPMSENSE_SCRIPT_PRE = (1 << 9) + RPMSENSE_SCRIPT_POST = (1 << 10) if flag is not None: # Note: RPMSENSE_PREREQ == 0 since rpm-4.4'ish - if flag & (rpm.RPMSENSE_PREREQ | - rpm.RPMSENSE_SCRIPT_PRE | - rpm.RPMSENSE_SCRIPT_POST): + if flag & (RPMSENSE_PREREQ | + RPMSENSE_SCRIPT_PRE | + RPMSENSE_SCRIPT_POST): return 1 return 0
Their comments were: "only suse uses PreReq nowadays", so the question is "Why?"
BTW, what do you use createrepo for? Philipp -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On 10/19/2010 07:36 PM, Philipp Thomas wrote:
BTW, what do you use createrepo for?
I have done a lot of work on a set of scripts that maintain a local repository of my opensuse installs on my local server. The scripts handle the collection and rsync of installed/updated packages from from /var/cache/zypp/packages to a local server, separates all packages into openSUSE_11.X and factory_11.X repositories, cleans the repo of old versions and then rebuilds and signs the repositories... and then builds html pages for the packages with repoview. The original directory structure from /var/cache/zypp/packages is then moved from / to /home/back/rpms/suse_11.X on the local box (to get the storage off /) where the original structure is parsed for duplicates/old versions which are then moved to /home/back/rpms/suse_11.X-old. That way if I ever have to reload after a hdw failure (or if I royally screw something up), I simply use my local repo to re-install with all updates already current without using a bit of bandwidth. Plus, I'm protected when a release hits eol and suse pulls all the repos before I have time or a reason to reinstall all my boxes. Further, on the local box I have /home/back/rpms/suse_11.X-old in case an update for the current release breaks something and the working versions of the packages on build service no longer exist. (kernel:/HEAD is a prime example) So -- without createrepo working correctly, my little scheme kind of falls apart... -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (4)
-
Dave Howorth -
David C. Rankin -
Philipp Thomas -
Philipp Thomas