openSUSE Build Service October 2020

buildservice@lists.opensuse.org

28 participants
22 discussions

[opensuse-buildservice] handling one thousand source tarballs in one package

by Ludwig Nussel

Hi, I'm trying to build a package from source that uses npm modules. Doing so involves more than one thousand(!) upstream tarballs. In the hope to make that more manageable I wrote a PoC script¹ to produuce a file includable from the spec and to also download the tarballs. Uploading them to OBS as part of the source is rather annoying though as you can't see the forest for all the trees anymore then. So would be convenient if a service like download_files or download_url could do that on server side so the files would be hidden as _service:* AFAICS services do not have access to previously generated results² though so would have to download those files on every source change which seems rather wasteful and also takes time. OTOH there's a hack in OBS to have a cache directory for tar_scm³ only. So would it be possible to either let services access the previous results or have a cache dir for all? Any other ideas how to handle one thousand source tarballs? cu Ludwig [1] https://github.com/lnussel/nodejs-tarballs [2] https://github.com/openSUSE/osc/blob/7b5d10552352295ebdd21193b6dfbf830e7d23… [3] https://github.com/openSUSE/open-build-service/blob/master/src/backend/run-… -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer HRB 36809 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

1 month

[opensuse-buildservice] Compiler Cache (Ccache) support in Open Build Service

by Sumit Jamgade

Hello, Long running compilation can now leverage ccache[1] to speed up the build process. This feature can be enabled per package in the project config: BuildFlags: useccahe:<packagename> example: https://build.opensuse.org/projects/home:sjamgade:branches:science:machinel… Once enabled per package, it will be enabled for all architectures and repositories. https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.prjconfig.… (grep ccache) --- The ccache tar file is collected only on *successful builds with changed result*. It is extracted for every build at /.ccache/ if enabled. This build artifact is not versioned and only the latest tar is stored. It can be downloaded from the webui or osc. https://build.opensuse.org/package/binaries/home:sjamgade:branches:science:… osc getbinaries --ccache -M standard home:sjamgade:branches:science:machinelearning tensorflow openSUSE_Tumbleweed x86_64 _ccache.tar osc getbinaries --ccache $prj $pkg $repo $arch _ccache.tar This archive can be further used for local builds too. osc build --pkg-ccache /full/path/to/_ccache. (please use the latest osc) --- Ccache config file is stored in /.ccache/ccache.conf and is writable by the build user. So, it could also be disabled (or configured as per package requirements) echo "disable = true" >> /.ccache/ccache.conf For possible build time comparisons: https://build.opensuse.org/packages/tensorflow:standard/job_history/home:sj… https://w3.nue.suse.com/~sjamgade/ccachereport.html Thank you, Sumit Jamgade and OBS Team [1] - https://ccache.dev/ PS: Some packages (eg: tensorflow) need extra plumbing in spec file to enable access to /.ccache and use of ccache cmd: export CC_PREFIX=ccache; ln -s /.ccache/ ~/.ccache -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

2 months, 3 weeks

[opensuse-buildservice] ARM Architectures on OBS Instance

by Kyle Edwards

Hello all, I recently added aarch64 and armv7el architectures to my OBS instance. I'm building for Ubuntu 16.04 and Ubuntu 20.04. However, both aarch64 and armv7el seem to be unavailable for Ubuntu 16.04 - they both say "unresolvable." When I look for more details, I see a long list of "nothing provides <package>." Is there not an aarch64 and armv7el build available for Ubuntu 16.04 on OBS? In addition, for Ubuntu 20.04, armv7el says "The repository setup is broken, build or publish is not possible." What does this mean? So far the only one of these four builds that I've managed to get working is aarch64 on Ubuntu 20.04. Side note: is armv7el the right architecture to use? There's a bunch of different arm-32 architectures, and it's not clear how the OBS arches correspond to the arches for other distros. Kyle -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

3 months, 3 weeks

[opensuse-buildservice] Need help with debugging an endless loop of events - bs_sched never takes a break!

by Srinidhi B

Hello List, I need some help in debugging a situation in my private OBS (2.9.x) instance that has been going on for a couple of months!! The schedulers keep scanning projects in a loop - never taking a break. On a Sunday, after 12 hours and 20 minutes of runtime, here are some numbers: # grep -ci "calculating project dependencies" /bs/log/scheduler_*.log /bs/log/scheduler_i586.log:19032 /bs/log/scheduler_local.log:10256 /bs/log/scheduler_x86_64.log:18915 # grep -ci "waiting for an event" /bs/log/scheduler_*.log /bs/log/scheduler_i586.log:0 /bs/log/scheduler_local.log:19150 /bs/log/scheduler_x86_64.log:0 I have not managed to find any hints as to what is causing this loop. Any help or pointers would be greatly appreciated! Let me share what I've found / done so far: 0. This problem started after we applied all available updates for SLES12 SP3 on July 31st. We had to, of course, reboot the system after that. I want to plan an upgrade to SLES12 SP5 but the current situation is making it very difficult to plan the upgrade. I just don't know if I'll making things even worse with an upgrade! 1. Dumped scheduler state by running `obs_admin --dump-state x86_64` and looked inside the state file. A few observations: * 'lookat' element has 1,230 items. * 'lookat_oob' element has 11 items. * 'retryevents' element has 66 items. * Number of items in 'prpfinished' and 'prps' do not match. Difference is 106. * 'changed_med' has 2 entries. 2. /bs/events/lastevents is always filled. After stopping all backend services, there are almost 5,000 entries in the file. 3. /srv/www/obs/api/log/backend_access.log has backend calls which do not show up in the production.log file. I thought, all backend calls must go through the API. Since today is Sunday, I suspected that the project consistency check could have been triggered. While it was indeed triggered, that job was finished 4 hours 23 minutes: Performed ConsistencyCheckJob (Job ID: c492970a-bfd7-4683-abf1-816f5eb5a6b6) from DelayedJob(consistency_check) in 259020.22ms 4. I have checked clockworkd.ruby2.5.clock.output to see if there are any strange jobs that are getting triggered. But the problem that I've always faced with this log is that it only logs when a job is triggered while it never contains when a job finished. When I try to run the "jobs:check" command, I get a MySQL error: $ run_in_api rails.ruby2.5 jobs:check rails aborted! ActiveRecord::StatementInvalid: Mysql2::Error: Unknown column 'created_at' in 'where clause': SELECT COUNT(*) FROM `delayed_jobs` WHERE (attempts = 0 AND created_at < '2020-10-25 10:26:42.114022') bin/rails:4:in `require' bin/rails:4:in `<main>' Caused by: Mysql2::Error: Unknown column 'created_at' in 'where clause' bin/rails:4:in `require' bin/rails:4:in `<main>' Tasks: TOP => jobs:check (See full trace by running task with --trace) Clearly, I'm going nowhere with this and I'm desperately trying to improve the situation. Any hints or pointers would be greatly appreciated. Regards, Srinidhi. N��r��y隊Z)z{.��Wlz��qﮞ˛��m�)z{.��+�:�{Zr�az�'z��j)h��Ǜ�)]��Ǿ� ޮ�^�ˬz��

3 months, 3 weeks

[opensuse-buildservice] Strange differences in 15.1 vs. 15.2 OBS builds

by Manfred Hollstein

Hi there, while this appears to be Rust specific, it is not as I've seen similar strange differences in other complex build situations (e.g. MozillaFirefox), too. The build of rust for Leap 15.1 failed due to the following results of RPMLINT <https://build.opensuse.org/package/live_build_log/devel:languages:rust/rust…>: [15022s] rust-src.noarch: E: env-script-interpreter (Badness: 9) /usr/lib/rustlib/src/rust/library/backtrace/ci/android-sdk.sh /usr/bin/env sh [15022s] rust-src.noarch: E: env-script-interpreter (Badness: 9) /usr/lib/rustlib/src/rust/library/stdarch/ci/dox.sh /usr/bin/env bash [15022s] rust-src.noarch: E: env-script-interpreter (Badness: 9) /usr/lib/rustlib/src/rust/library/stdarch/ci/run-docker.sh /usr/bin/env sh [15022s] rust-src.noarch: E: env-script-interpreter (Badness: 9) /usr/lib/rustlib/src/rust/library/stdarch/ci/run.sh /usr/bin/env sh [15022s] rust-src.noarch: E: env-script-interpreter (Badness: 9) /usr/lib/rustlib/src/rust/library/stdarch/ci/style.sh /usr/bin/env sh [15022s] This script uses 'env' as an interpreter. For the rpm runtime dependency [15022s] detection to work, the shebang #!/usr/bin/env python needs to be patched into [15022s] #!/usr/bin/python otherwise the package dependency generator merely adds a [15022s] dependency on /usr/bin/env rather than the actual interpreter /usr/bin/python. [15022s] Alternatively, if the file should not be executed, then ensure that it is not [15022s] marked as executable or don't install it in a path that is reserved for [15022s] executables. [15022s] ... [15022s] 12 packages and 0 specfiles checked; 6 errors, 27 warnings. Interestingly the same messages are shown in the build log for Leap 15.2, but this build succeeded <https://build.opensuse.org/package/live_build_log/devel:languages:rust/rust…> Here there is one error less (due to not having to build the included LLVM, but rather use the system provided one), but still 5 errors. I worked around it (see <https://build.opensuse.org/request/show/845063>), but I don't understand why OBS behaves differently for OSs older than 15.2 and 15.2 onwards (incl. Tumbleweed). Does anyone have an idea? Fix? TIA, cheers. l8er manfred

3 months, 4 weeks

[opensuse-buildservice] How to specify DEB_BUILD_OPTIONS / environment variables

by Harald Welte

Dear OBS list, I've tried to find out how I can set environment variables while building dpkg packages via OBS. I would have assumed that maybe the project config would provide something like that, but I failed to find any. The concrete usage example is to pass something like DEB_BUILD_OPTIONS=parallel=N which then most sane debian packages turn into "-jN" during make. I also grep'ed for DEB_BUILD_OPTIONS in the OBS source code, but there were no hits. Right now I'm watching a 16-core machine building a Linux kernel with "-j1" since I cannot figure out what is the designated way in OBS to specify such environment variables. Thanks in advance for any assistance. Regards, Harald p.s.: I previously had to switch from public build.opensuse.org to a local installation of OBS due to the public build.opensuse.org workers consistently failing builds with "no space left on device" while trying to build a linux kernel package. And I already had stripped it down to only one variant + flavor + architecture. Not a complaint, just noticing it. I would have expected kernel builds not to be that uncommon. -- - Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/ ============================================================================ "Privacy in residential applications is a desirable marketing option." (ETSI EN 300 175-7 Ch. A6) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

3 months, 4 weeks

[opensuse-buildservice] Trying to add ARM architectures to my OBS installation

by Kyle Edwards

Hello, I am trying to add aarch64 and armv7hl to my OBS server. In the web interface, I check the boxes for aarch64 and armv7hl (and also uncheck the one for armv7l, since it was checked for some reason.) However, the changes don't seem to take - when I leave the page and come back, aarch64 and armv7hl are unchecked, and armv7l is still checked. Am I doing this right? I don't see an "update" button anywhere on the page (and the "update" button on the main configuration tab doesn't seem to help.) Kyle -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

3 months, 4 weeks

[opensuse-buildservice] Installing RPM fails

by Roger Oberholtzer

I have just built a couple RPMs as a user on OBS. The repository is not a new one. But the package is new. The package built okay. But when I try to install the RPM, I get this: Download (curl) error for 'http://download.opensuse.org/repositories/home:/rogeroberholtzer/openSUSE_L…': Error code: Curl error 60 Error message: SSL certificate problem: self signed certificate in certificate chain I do not know what this means. It happens in more than one environment. So I suspect that something has happened on OBS related to certificates. But I really do not know what. I googled and found things like this: https://support.microfocus.com/kb/doc.php?id=7024620 I tried what it said, but it did not help. Other solutions were less clear. And most were many years old. This problem has just happened. I do not know if it is an OBS issue or a local issue. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

3 months, 4 weeks

[opensuse-buildservice] Add Fedora 33 to openSUSE Build Service?

by Neal Gompa

Hey, Fedora 33 is going to be released on Tuesday. Can we get it added to OBS now? -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

4 months

[opensuse-buildservice] New repo for mass building of packages from CRAN

by Detlef Steuer

Hi Adrian, I have created d:l:R:autoCRAN to hold the generated packages and bring most of cran.r-project.org to all users of SuSE Linux. Before I start populating that directory I want to state my plans and ask if such a usage is ok with OBS. At the moment cran holds 16.5k libraries for the R language, around 11k libraries can be created already automatically, but I'm actively improving the script and hope to reach 90% of CRAN soon(tm). That means there will be >= 15k packages in the repo with some heavy dependencies and huge recompiles when a core package changes. My experience in the home:dsteuer repo showed that the packages compile relatively fast on OBS' big machines. Therefore I don't expect trouble for OBS. All packages will be compiled successfully outside of OBS before uploading (modulo bugs in the scripts, of course). So I don't expect too much bitrot there. All packages will be kept uptodate through cron jobs outside of OBS and will only be uploaded after the updates compile successfully, too. I expect 20 to 50 updates a day after the initial upload. Following our former discussions: is it ok for me to push CRAN to OBS, even with this number of packages? Should I throttle the initial upload to a package a minute or every second minute or is it ok to push after each build? My guess is around two packages a minute for a few days, if I push after each successful build. Any hints from your side? Of course I expect to be stopped doing the upload, if OBS has any problems with the number of packages. But I'm optimistic. Until I hit the wall of 10k packages in a home project everything went smoothly :-) For a start I added Tumbleweed, SLE_15_SP2 and Leap_15.2 as targets. If you think it is sensible, I can use only Tumbleweed for a start. After filling the repo I would like to add all SuSE releases for which R-base builds. Regards Detlef -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org

4 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openSUSE Build Service October 2020