Hi,
I'm trying to build a package from source that uses npm modules. Doing
so involves more than one thousand(!) upstream tarballs. In the hope to
make that more manageable I wrote a PoC script¹ to produuce a file
includable from the spec and to also download the tarballs. Uploading
them to OBS as part of the source is rather annoying though as you can't
see the forest for all the trees anymore then.
So would be convenient if a service like download_files or download_url
could do that on server side so the files would be hidden as
_service:*
AFAICS services do not have access to previously generated results²
though so would have to download those files on every source change
which seems rather wasteful and also takes time.
OTOH there's a hack in OBS to have a cache directory for tar_scm³ only.
So would it be possible to either let services access the previous
results or have a cache dir for all? Any other ideas how to handle one
thousand source tarballs?
cu
Ludwig
[1] https://github.com/lnussel/nodejs-tarballs
[2]
https://github.com/openSUSE/osc/blob/7b5d10552352295ebdd21193b6dfbf830e7d23…
[3]
https://github.com/openSUSE/open-build-service/blob/master/src/backend/run-…
--
(o_ Ludwig Nussel
//\
V_/_ http://www.suse.com/
SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer
HRB 36809 (AG Nürnberg)
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org
Hello,
Long running compilation can now leverage ccache[1] to speed up the
build process. This feature can be enabled per package in the project
config:
BuildFlags: useccahe:<packagename>
example:
https://build.opensuse.org/projects/home:sjamgade:branches:science:machinel…
Once enabled per package, it will be enabled for all architectures and
repositories.
https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.prjconfig.…
(grep ccache)
---
The ccache tar file is collected only on *successful builds with changed
result*. It is extracted for every build at /.ccache/ if enabled.
This build artifact is not versioned and only the latest tar is stored.
It can be downloaded from the webui or osc.
https://build.opensuse.org/package/binaries/home:sjamgade:branches:science:…
osc getbinaries --ccache -M standard
home:sjamgade:branches:science:machinelearning tensorflow
openSUSE_Tumbleweed x86_64 _ccache.tar osc getbinaries --ccache $prj
$pkg $repo $arch _ccache.tar
This archive can be further used for local builds too.
osc build --pkg-ccache /full/path/to/_ccache.
(please use the latest osc)
---
Ccache config file is stored in /.ccache/ccache.conf and is writable by
the build user. So, it could also be disabled (or configured as per
package requirements)
echo "disable = true" >> /.ccache/ccache.conf
For possible build time comparisons:
https://build.opensuse.org/packages/tensorflow:standard/job_history/home:sj…https://w3.nue.suse.com/~sjamgade/ccachereport.html
Thank you,
Sumit Jamgade and OBS Team
[1] - https://ccache.dev/
PS:
Some packages (eg: tensorflow) need extra plumbing in spec file to
enable access to /.ccache and use of ccache cmd:
export CC_PREFIX=ccache;
ln -s /.ccache/ ~/.ccache
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org
Hello all,
I recently added aarch64 and armv7el architectures to my OBS instance.
I'm building for Ubuntu 16.04 and Ubuntu 20.04. However, both aarch64
and armv7el seem to be unavailable for Ubuntu 16.04 - they both say
"unresolvable." When I look for more details, I see a long list of
"nothing provides <package>." Is there not an aarch64 and armv7el build
available for Ubuntu 16.04 on OBS?
In addition, for Ubuntu 20.04, armv7el says "The repository setup is
broken, build or publish is not possible." What does this mean? So far
the only one of these four builds that I've managed to get working is
aarch64 on Ubuntu 20.04.
Side note: is armv7el the right architecture to use? There's a bunch of
different arm-32 architectures, and it's not clear how the OBS arches
correspond to the arches for other distros.
Kyle
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org
Hello List,
I need some help in debugging a situation in my private OBS (2.9.x) instance
that has been going on for a couple of months!!
The schedulers keep scanning projects in a loop - never taking a break. On a
Sunday, after 12 hours and 20 minutes of runtime, here are some numbers:
# grep -ci "calculating project dependencies" /bs/log/scheduler_*.log
/bs/log/scheduler_i586.log:19032
/bs/log/scheduler_local.log:10256
/bs/log/scheduler_x86_64.log:18915
# grep -ci "waiting for an event" /bs/log/scheduler_*.log
/bs/log/scheduler_i586.log:0
/bs/log/scheduler_local.log:19150
/bs/log/scheduler_x86_64.log:0
I have not managed to find any hints as to what is causing this loop. Any help
or pointers would be greatly appreciated!
Let me share what I've found / done so far:
0. This problem started after we applied all available updates for SLES12 SP3 on
July 31st. We had to, of course, reboot the system after that. I want to plan an
upgrade to SLES12 SP5 but the current situation is making it very difficult to
plan the upgrade.
I just don't know if I'll making things even worse with an upgrade!
1. Dumped scheduler state by running `obs_admin --dump-state x86_64` and looked
inside the state file. A few observations:
* 'lookat' element has 1,230 items.
* 'lookat_oob' element has 11 items.
* 'retryevents' element has 66 items.
* Number of items in 'prpfinished' and 'prps' do not match. Difference is 106.
* 'changed_med' has 2 entries.
2. /bs/events/lastevents is always filled. After stopping all backend services,
there are almost 5,000 entries in the file.
3. /srv/www/obs/api/log/backend_access.log has backend calls which do not show
up in the production.log file. I thought, all backend calls must go through the
API.
Since today is Sunday, I suspected that the project consistency check could have
been triggered. While it was indeed triggered, that job was finished 4 hours 23
minutes:
Performed ConsistencyCheckJob (Job ID: c492970a-bfd7-4683-abf1-816f5eb5a6b6) from DelayedJob(consistency_check) in 259020.22ms
4. I have checked clockworkd.ruby2.5.clock.output to see if there are any
strange jobs that are getting triggered. But the problem that I've always faced
with this log is that it only logs when a job is triggered while it never
contains when a job finished.
When I try to run the "jobs:check" command, I get a MySQL error:
$ run_in_api rails.ruby2.5 jobs:check
rails aborted!
ActiveRecord::StatementInvalid: Mysql2::Error: Unknown column 'created_at' in 'where clause': SELECT COUNT(*) FROM `delayed_jobs` WHERE (attempts = 0 AND created_at < '2020-10-25 10:26:42.114022')
bin/rails:4:in `require'
bin/rails:4:in `<main>'
Caused by:
Mysql2::Error: Unknown column 'created_at' in 'where clause'
bin/rails:4:in `require'
bin/rails:4:in `<main>'
Tasks: TOP => jobs:check
(See full trace by running task with --trace)
Clearly, I'm going nowhere with this and I'm desperately trying to improve the
situation. Any hints or pointers would be greatly appreciated.
Regards,
Srinidhi.
N�����r��y隊Z)z{.���Wlz��qﮞ˛���m�)z{.��+�:�{Zr�az�'z��j)h���Ǜ�)]���Ǿ� ޮ�^�ˬz��
Hi there,
while this appears to be Rust specific, it is not as I've seen similar
strange differences in other complex build situations (e.g.
MozillaFirefox), too.
The build of rust for Leap 15.1 failed due to the following results of
RPMLINT <https://build.opensuse.org/package/live_build_log/devel:languages:rust/rust…>:
[15022s] rust-src.noarch: E: env-script-interpreter (Badness: 9) /usr/lib/rustlib/src/rust/library/backtrace/ci/android-sdk.sh /usr/bin/env sh
[15022s] rust-src.noarch: E: env-script-interpreter (Badness: 9) /usr/lib/rustlib/src/rust/library/stdarch/ci/dox.sh /usr/bin/env bash
[15022s] rust-src.noarch: E: env-script-interpreter (Badness: 9) /usr/lib/rustlib/src/rust/library/stdarch/ci/run-docker.sh /usr/bin/env sh
[15022s] rust-src.noarch: E: env-script-interpreter (Badness: 9) /usr/lib/rustlib/src/rust/library/stdarch/ci/run.sh /usr/bin/env sh
[15022s] rust-src.noarch: E: env-script-interpreter (Badness: 9) /usr/lib/rustlib/src/rust/library/stdarch/ci/style.sh /usr/bin/env sh
[15022s] This script uses 'env' as an interpreter. For the rpm runtime dependency
[15022s] detection to work, the shebang #!/usr/bin/env python needs to be patched into
[15022s] #!/usr/bin/python otherwise the package dependency generator merely adds a
[15022s] dependency on /usr/bin/env rather than the actual interpreter /usr/bin/python.
[15022s] Alternatively, if the file should not be executed, then ensure that it is not
[15022s] marked as executable or don't install it in a path that is reserved for
[15022s] executables.
[15022s] ...
[15022s] 12 packages and 0 specfiles checked; 6 errors, 27 warnings.
Interestingly the same messages are shown in the build log for Leap
15.2, but this build succeeded
<https://build.opensuse.org/package/live_build_log/devel:languages:rust/rust…>
Here there is one error less (due to not having to build the included
LLVM, but rather use the system provided one), but still 5 errors.
I worked around it (see <https://build.opensuse.org/request/show/845063>),
but I don't understand why OBS behaves differently for OSs older than
15.2 and 15.2 onwards (incl. Tumbleweed).
Does anyone have an idea? Fix?
TIA, cheers.
l8er
manfred
Dear OBS list,
I've tried to find out how I can set environment variables while
building dpkg packages via OBS. I would have assumed that maybe the
project config would provide something like that, but I failed to find
any.
The concrete usage example is to pass something like DEB_BUILD_OPTIONS=parallel=N
which then most sane debian packages turn into "-jN" during make.
I also grep'ed for DEB_BUILD_OPTIONS in the OBS source code, but there
were no hits.
Right now I'm watching a 16-core machine building a Linux kernel with "-j1" since
I cannot figure out what is the designated way in OBS to specify such environment
variables.
Thanks in advance for any assistance.
Regards,
Harald
p.s.: I previously had to switch from public build.opensuse.org to
a local installation of OBS due to the public build.opensuse.org workers
consistently failing builds with "no space left on device" while trying
to build a linux kernel package. And I already had stripped it down to
only one variant + flavor + architecture. Not a complaint, just
noticing it. I would have expected kernel builds not to be that
uncommon.
--
- Harald Welte <laforge(a)gnumonks.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org
Hello,
I am trying to add aarch64 and armv7hl to my OBS server. In the web
interface, I check the boxes for aarch64 and armv7hl (and also uncheck
the one for armv7l, since it was checked for some reason.) However, the
changes don't seem to take - when I leave the page and come back,
aarch64 and armv7hl are unchecked, and armv7l is still checked. Am I
doing this right? I don't see an "update" button anywhere on the page
(and the "update" button on the main configuration tab doesn't seem to
help.)
Kyle
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org
I have just built a couple RPMs as a user on OBS. The repository is
not a new one. But the package is new.
The package built okay. But when I try to install the RPM, I get this:
Download (curl) error for
'http://download.opensuse.org/repositories/home:/rogeroberholtzer/openSUSE_L…':
Error code: Curl error 60
Error message: SSL certificate problem: self signed certificate in
certificate chain
I do not know what this means. It happens in more than one
environment. So I suspect that something has happened on OBS related
to certificates. But I really do not know what.
I googled and found things like this:
https://support.microfocus.com/kb/doc.php?id=7024620
I tried what it said, but it did not help. Other solutions were less
clear. And most were many years old. This problem has just happened.
I do not know if it is an OBS issue or a local issue.
--
Roger Oberholtzer
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org
Hey,
Fedora 33 is going to be released on Tuesday. Can we get it added to OBS now?
--
真実はいつも一つ!/ Always, there's only one truth!
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org
Hi Adrian,
I have created d:l:R:autoCRAN to hold the generated packages
and bring most of cran.r-project.org to all users of SuSE Linux.
Before I start populating that directory I want to state my plans
and ask if such a usage is ok with OBS.
At the moment cran holds 16.5k libraries for the R language, around 11k
libraries can be created already automatically, but I'm actively
improving the script and hope to reach 90% of CRAN soon(tm).
That means there will be >= 15k packages in the repo with some heavy
dependencies and huge recompiles when a core package changes. My
experience in the home:dsteuer repo showed that the packages compile
relatively fast on OBS' big machines. Therefore I don't expect
trouble for OBS.
All packages will be compiled successfully outside of OBS
before uploading (modulo bugs in the scripts, of course). So I don't
expect too much bitrot there.
All packages will be kept uptodate through cron jobs outside of OBS and
will only be uploaded after the updates compile successfully, too.
I expect 20 to 50 updates a day after the initial upload.
Following our former discussions: is it ok for me to push CRAN to OBS,
even with this number of packages?
Should I throttle the initial upload to a package a minute or every
second minute or is it ok to push after each build? My guess is around
two packages a minute for a few days, if I push after each successful
build. Any hints from your side?
Of course I expect to be stopped doing the upload, if OBS has any
problems with the number of packages. But I'm optimistic. Until I hit
the wall of 10k packages in a home project everything went smoothly :-)
For a start I added Tumbleweed, SLE_15_SP2 and Leap_15.2 as targets. If
you think it is sensible, I can use only Tumbleweed for a start. After
filling the repo I would like to add all SuSE releases for which R-base
builds.
Regards
Detlef
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-buildservice+owner(a)opensuse.org