Re: [opensuse-buildservice] .oscignore to ignore tarballs downloaded by source service?

newer
[opensuse-buildservice] Download...

older
[opensuse-buildservice] Remove...

Michal Marek

13 Apr 2011 13 Apr '11

20:20

(I got this reply privately, but I think it was meant for the list) On 13.4.2011 15:28, Michal Vyskocil wrote:

...

On Wed, Apr 13, 2011 at 11:36:25AM +0200, Michal Marek wrote:

...
Hi,

I see there is no way to set ignore patterns on a per-package basis, i.e. some .oscignore or rather _oscignore file. Would you accept a patch that adds such feature?

The problem I'm trying to solve is following: The kernel-source package is maintained in a git repository and a script exists to generate a source package that can be built using rpmbuild, osc build or the buildservice. To avoid having to upload the 70M+ linux tarball each time you branch the package in the buildservice, I would like to use the download_url service to let the server download it. But at the same

Well, as osc build runs all service files by default on each build, which is braindead, as people will be really confused why their build has a delay (during it they can download 70MB big file ...).

I don't observe this with osc-0.130.1 2, osc build uses the existing _service:download_url:linux-2.6.39-rc3.tar.bz2 tarball.

...

...
time, I would like to have the tarball locally as before, so that plain rpmbuild works. But then I need to tell osc to ignore the linux tarball and avoid messages like this:

$ osc ci File "linux-2.6.38.tar.bz2" is listed in package meta but does not exist. (s)kip/(r)emove/(e)dit file lists/(c)ommit/(A)bort?

You can either skip this check by -f or check_filelist in ~/.oscrc, see source code

3087 if conf.config['check_filelist'] and not opts.force: 3088 check_filelist_before_commit(pacs)

I'm aware of the global config, but this is really specific to one package.

...

As an original author I'd say we can remove this one, as this is a duplicate of source-validator.

Great, that would solve my problem. I don't care if the tarball shows up with a question mark in 'osc status', as long as it doesn't complain during commit.

...

If you want to write a patch, then consider reverting of existing braindead osc build logic and add add --service, --force-service, or --run-service option to force a _service execution. Or in the most clever case, save a time of last execution and compare a timestamp, so the _service files will be triggered only in case of change.

The current logic works fine for my specific use case. Michal -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Show replies by date

Michal Marek

13 Apr 13 Apr

20:55

New subject: [opensuse-buildservice] [PATCH] Do not check for untracked files before commit

This duplicates the work of the source validator and is just annoying. Suggested-by: Michal Vyskocil Signed-off-by: Michal Marek --- osc/commandline.py | 3 --- osc/core.py | 27 --------------------------- 2 files changed, 0 insertions(+), 30 deletions(-) diff --git a/osc/commandline.py b/osc/commandline.py index 0e4673c..b0831be 100644 --- a/osc/commandline.py +++ b/osc/commandline.py @@ -3418,9 +3418,6 @@ Please submit there instead, or use --nodevelproject to force direct submission. pacs = findpacs(args) - if conf.config['check_filelist'] and not opts.force: - check_filelist_before_commit(pacs) - if conf.config['do_package_tracking'] and len(pacs) > 0: prj_paths = {} single_paths = [] diff --git a/osc/core.py b/osc/core.py index f41661e..f083753 100644 --- a/osc/core.py +++ b/osc/core.py @@ -5903,33 +5903,6 @@ def get_commit_msg(wc_dir, pacs): store_unlink_file(wc_dir, '_commit_msg') return msg -def check_filelist_before_commit(pacs): - - # warn if any of files has a ? status (usually a patch, or new source was not added to meta) - for p in pacs: - # no files given as argument? Take all files in current dir - if not p.todo: - p.todo = p.filenamelist + p.filenamelist_unvers - p.todo.sort() - for f in [f for f in p.todo if not os.path.isdir(f)]: - if not f.startswith('_service:') and not f.startswith('_service_') and p.status(f) in ('?', '!'): - print 'File "%s" found, but not listed in package meta.' % f - resp = raw_input('(s)kip/(r)emove/(e)dit file lists/(c)ommit/(A)bort? ') - if resp in ('s', 'S'): - continue - elif resp in ('r', 'R', 'd', 'D'): - p.process_filelist(['r ? %s' % f]) - elif resp in ('e', 'E'): - try: - p.edit_filelist() - except ValueError: - print >>sys.stderr, "Error during processiong of file list." - raise oscerr.UserAbort() - elif resp in ('c', 'C'): - break - else: - raise oscerr.UserAbort() - def print_request_list(apiurl, project, package = None, states = ('new','review',), force = False): """ prints list of pending requests for the specified project/package if "check_for_request_on_action" -- 1.7.4.1 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Ludwig Nussel

14 Apr 14 Apr

07:41

New subject: [opensuse-buildservice] [PATCH] Do not check for untracked files before commit

Michal Marek wrote:

...

This duplicates the work of the source validator and is just annoying.

The source validator doesn't check whether a referenced file is actually tracked by osc though, or does it? cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Michal Marek

12:05

New subject: [opensuse-buildservice] [PATCH v2] Do not check for untracked files before commit

Michal Marek

12:07

New subject: [opensuse-buildservice] [PATCH v2] Do not check for untracked files before commit

This duplicates the work of the source validator and is just annoying. Suggested-by: Michal Vyskocil Signed-off-by: Michal Marek --- Sorry, here is the actual v2 patch that deletes more unused functions. Michal --- osc/commandline.py | 15 +------ osc/core.py | 120 ---------------------------------------------------- 2 files changed, 1 insertions(+), 134 deletions(-) diff --git a/osc/commandline.py b/osc/commandline.py index 0e4673c..15fc016 100644 --- a/osc/commandline.py +++ b/osc/commandline.py @@ -3350,7 +3350,7 @@ Please submit there instead, or use --nodevelproject to force direct submission. @cmdln.option('-F', '--file', metavar='FILE', help='read log message from FILE, \'-\' denotes standard input.') @cmdln.option('-f', '--force', default=False, action="store_true", - help='force commit - do not tests a file list') + help='ignored') @cmdln.option('--skip-validation', default=False, action="store_true", help='Skip the source validation') @cmdln.option('--verbose-validation', default=False, action="store_true", @@ -3361,16 +3361,6 @@ Please submit there instead, or use --nodevelproject to force direct submission. Upload content which is changed in your working copy, to the repository server. - Optionally checks the state of a working copy, if found a file with - unknown state, it requests an user input: - * skip - don't change anything, just move to another file - * remove - remove a file from dir - * edit file list - edit filelist using EDITOR - * commit - don't check anything and commit package - * abort - abort commit - this is default value - This can be supressed by check_filelist config item, or -f/--force - command line option. - examples: osc ci # current dir osc ci <dir> @@ -3418,9 +3408,6 @@ Please submit there instead, or use --nodevelproject to force direct submission. pacs = findpacs(args) - if conf.config['check_filelist'] and not opts.force: - check_filelist_before_commit(pacs) - if conf.config['do_package_tracking'] and len(pacs) > 0: prj_paths = {} single_paths = [] diff --git a/osc/core.py b/osc/core.py index f41661e..9d86292 100644 --- a/osc/core.py +++ b/osc/core.py @@ -2035,99 +2035,6 @@ rev: %s si.execute(self.absdir, mode, singleservice) os.chdir(curdir) - def prepare_filelist(self): - """Prepare a list of files, which will be processed by process_filelist - method. This allows easy modifications of a file list in commit - phase. - """ - if not self.todo: - self.todo = self.filenamelist + self.filenamelist_unvers - self.todo.sort() - - ret = "" - for f in [f for f in self.todo if not os.path.isdir(f)]: - action = 'leave' - status = self.status(f) - if status == 'S': - continue - if status == '!': - action = 'remove' - ret += "%s %s %s\n" % (action, status, f) - - ret += """ -# Edit a filelist for package \'%s\' -# Commands: -# l, leave = leave a file as is -# r, remove = remove a file -# a, add = add a file -# -# If you remove file from a list, it will be unchanged -# If you remove all, commit will be aborted""" % self.name - - return ret - - def edit_filelist(self): - """Opens a package list in editor for editing. This allows easy - modifications of it just by simple text editing - """ - - import tempfile - (fd, filename) = tempfile.mkstemp(prefix = 'osc-filelist', suffix = '.txt') - f = os.fdopen(fd, 'w') - f.write(self.prepare_filelist()) - f.close() - mtime_orig = os.stat(filename).st_mtime - - while 1: - run_editor(filename) - mtime = os.stat(filename).st_mtime - if mtime_orig < mtime: - filelist = open(filename).readlines() - os.unlink(filename) - break - else: - raise oscerr.UserAbort() - - return self.process_filelist(filelist) - - def process_filelist(self, filelist): - """Process a filelist - it add/remove or leave files. This depends on - user input. If no file is processed, it raises an ValueError - """ - - loop = False - for line in [l.strip() for l in filelist if (l[0] != "#" or l.strip() != '')]: - - foo = line.split(' ') - if len(foo) == 4: - action, state, name = (foo[0], ' ', foo[3]) - elif len(foo) == 3: - action, state, name = (foo[0], foo[1], foo[2]) - else: - break - action = action.lower() - loop = True - - if action in ('r', 'remove'): - if self.status(name) == '?': - os.unlink(name) - if name in self.todo: - self.todo.remove(name) - else: - self.delete_file(name, True) - elif action in ('a', 'add'): - if self.status(name) != '?': - print "Cannot add file %s with state %s, skipped" % (name, self.status(name)) - else: - self.addfile(name) - elif action in ('l', 'leave'): - pass - else: - raise ValueError("Unknow action `%s'" % action) - - if not loop: - raise ValueError("Empty filelist") - def revert(self, filename): if not filename in self.filenamelist and not filename in self.to_be_added: raise oscerr.OscIOError(None, 'file \'%s\' is not under version control' % filename) @@ -5903,33 +5810,6 @@ def get_commit_msg(wc_dir, pacs): store_unlink_file(wc_dir, '_commit_msg') return msg -def check_filelist_before_commit(pacs): - - # warn if any of files has a ? status (usually a patch, or new source was not added to meta) - for p in pacs: - # no files given as argument? Take all files in current dir - if not p.todo: - p.todo = p.filenamelist + p.filenamelist_unvers - p.todo.sort() - for f in [f for f in p.todo if not os.path.isdir(f)]: - if not f.startswith('_service:') and not f.startswith('_service_') and p.status(f) in ('?', '!'): - print 'File "%s" found, but not listed in package meta.' % f - resp = raw_input('(s)kip/(r)emove/(e)dit file lists/(c)ommit/(A)bort? ') - if resp in ('s', 'S'): - continue - elif resp in ('r', 'R', 'd', 'D'): - p.process_filelist(['r ? %s' % f]) - elif resp in ('e', 'E'): - try: - p.edit_filelist() - except ValueError: - print >>sys.stderr, "Error during processiong of file list." - raise oscerr.UserAbort() - elif resp in ('c', 'C'): - break - else: - raise oscerr.UserAbort() - def print_request_list(apiurl, project, package = None, states = ('new','review',), force = False): """ prints list of pending requests for the specified project/package if "check_for_request_on_action" -- 1.7.4.1 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Michal Marek

12:05

New subject: [opensuse-buildservice] [PATCH] Also check if files are tracked by osc

Signed-off-by: Michal Marek --- 20-files-present-and-referenced | 23 +++++++++++++++++++---- 1 files changed, 19 insertions(+), 4 deletions(-) diff --git a/20-files-present-and-referenced b/20-files-present-and-referenced index 146bd84..a456e0e 100755 --- a/20-files-present-and-referenced +++ b/20-files-present-and-referenced @@ -135,11 +135,26 @@ for i in `cat $TMPDIR/sources` ; do done mv $TMPDIR/sources.t $TMPDIR/sources +check_tracked() +{ + local file=${1##*/} + + if ! test -f "$DIR_TO_CHECK/$file"; then + echo "$file mentioned in spec file does not exist." + return 1 + fi + if test -f "$DESTINATIONDIR/$file"; then + return 0 + fi + if grep -qsFx "$file" "$DESTINATIONDIR/_to_be_added"; then + return 0 + fi + echo "$file mentioned in spec file is not tracked." + return 1 +} + for HASTOBETHER in `cat $TMPDIR/sources` ; do - test -f $DIR_TO_CHECK/$HASTOBETHER || { - echo $HASTOBETHER mentioned in spec file does not exist. - RETURN=1 - } + check_tracked "$HASTOBETHER" || RETURN=1 done # -- 1.7.4.1 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Michal Marek

15 Apr 15 Apr

09:06

New subject: [opensuse-buildservice] [PATCH] Also check if files are tracked by osc

Hi Rudi, can you have a look at this patch? I also created a merge request on gitorious: https://gitorious.org/opensuse/osc-source_validator/merge_requests/1 Michal On 14.4.2011 14:05, Michal Marek wrote:

...

Signed-off-by: Michal Marek --- 20-files-present-and-referenced | 23 +++++++++++++++++++---- 1 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/20-files-present-and-referenced b/20-files-present-and-referenced index 146bd84..a456e0e 100755 --- a/20-files-present-and-referenced +++ b/20-files-present-and-referenced @@ -135,11 +135,26 @@ for i in `cat $TMPDIR/sources` ; do done mv $TMPDIR/sources.t $TMPDIR/sources

+check_tracked() +{ + local file=${1##*/} + + if ! test -f "$DIR_TO_CHECK/$file"; then + echo "$file mentioned in spec file does not exist." + return 1 + fi + if test -f "$DESTINATIONDIR/$file"; then + return 0 + fi + if grep -qsFx "$file" "$DESTINATIONDIR/_to_be_added"; then + return 0 + fi + echo "$file mentioned in spec file is not tracked." + return 1 +} + for HASTOBETHER in `cat $TMPDIR/sources` ; do - test -f $DIR_TO_CHECK/$HASTOBETHER || { - echo $HASTOBETHER mentioned in spec file does not exist. - RETURN=1 - } + check_tracked "$HASTOBETHER" || RETURN=1 done

#

-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Ruediger Oertel

12:05

New subject: [opensuse-buildservice] [PATCH] Also check if files are tracked by osc

On Friday 15 April 2011 11:06:22 Michal Marek wrote:

...

Hi Rudi, can you have a look at this patch? I also created a merge request on gitorious: https://gitorious.org/opensuse/osc-source_validator/merge_requests/1

well, this just opens up another hole to keep arbitrary files in the sources without having them mentioned in the specfile. What's the rationale to do this (and potentially not being able to rebuild a package from it's src-rpm) ?

...

Michal

On 14.4.2011 14:05, Michal Marek wrote:

...
Signed-off-by: Michal Marek --- 20-files-present-and-referenced | 23 +++++++++++++++++++---- 1 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/20-files-present-and-referenced b/20-files-present-and-referenced index 146bd84..a456e0e 100755 --- a/20-files-present-and-referenced +++ b/20-files-present-and-referenced @@ -135,11 +135,26 @@ for i in `cat $TMPDIR/sources` ; do done mv $TMPDIR/sources.t $TMPDIR/sources

+check_tracked() +{ + local file=${1##*/} + + if ! test -f "$DIR_TO_CHECK/$file"; then + echo "$file mentioned in spec file does not exist." + return 1 + fi + if test -f "$DESTINATIONDIR/$file"; then + return 0 + fi + if grep -qsFx "$file" "$DESTINATIONDIR/_to_be_added"; then + return 0 + fi + echo "$file mentioned in spec file is not tracked." + return 1 +} + for HASTOBETHER in `cat $TMPDIR/sources` ; do - test -f $DIR_TO_CHECK/$HASTOBETHER || { - echo $HASTOBETHER mentioned in spec file does not exist. - RETURN=1 - } + check_tracked "$HASTOBETHER" || RETURN=1 done

#

-- with kind regards (mit freundlichem Grinsen), Ruediger Oertel (ro@novell.com,ro@suse.de,bugfinder@t-online.de) ---------------------------------------------------------------------- Linux fatou 2.6.38-2-desktop #1 SMP PREEMPT 2011-03-15 17:12:01 +0100 x86_64 Key fingerprint = 17DC 6553 86A7 384B 53C5 CA5C 3CE4 F2E7 23F2 B417 SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Michal Marek

12:19

New subject: [opensuse-buildservice] [PATCH] Also check if files are tracked by osc

On 15.4.2011 14:05, Ruediger Oertel wrote:

...

On Friday 15 April 2011 11:06:22 Michal Marek wrote:

...
Hi Rudi, can you have a look at this patch? I also created a merge request on gitorious: https://gitorious.org/opensuse/osc-source_validator/merge_requests/1

well, this just opens up another hole to keep arbitrary files in the sources without having them mentioned in the specfile. What's the rationale to do this (and potentially not being able to rebuild a package from it's src-rpm) ?

I don't understand you now. What I'm trying to achieve is to check not only if a referenced file is in the source directory, but if it is also tracked by osc (i.e. if the packager didn't accidentally forget to 'osc add' it). The opposite check, if everything in the source directory is also referenced by the specfile, comes a few lines later (after "# now check if everything is marked in spec files.") and I didn't modify it. At least I didn't intend to :). The rationale for this is that a similar check exists in osc and they overlap a bit, so I'm removing the osc check and adding the missing bits here. Michal -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Ruediger Oertel

12:42

New subject: [opensuse-buildservice] [PATCH] Also check if files are tracked by osc

On Friday 15 April 2011 14:19:02 Michal Marek wrote:

...

I don't understand you now. What I'm trying to achieve is to check not only if a referenced file is in the source directory, but if it is also tracked by osc (i.e. if the packager didn't accidentally forget to 'osc add' it). The opposite check, if everything in the source directory is also referenced by the specfile, comes a few lines later (after "# now check if everything is marked in spec files.") and I didn't modify it. At least I didn't intend to :).

The rationale for this is that a similar check exists in osc and they overlap a bit, so I'm removing the osc check and adding the missing bits here.

argh ... sorry, my oversight. thanks for the explanation. -- with kind regards (mit freundlichem Grinsen), Ruediger Oertel (ro@novell.com,ro@suse.de,bugfinder@t-online.de) ---------------------------------------------------------------------- Linux fatou 2.6.38-2-desktop #1 SMP PREEMPT 2011-03-15 17:12:01 +0100 x86_64 Key fingerprint = 17DC 6553 86A7 384B 53C5 CA5C 3CE4 F2E7 23F2 B417 SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

4781

Age (days ago)

4783

Last active (days ago)

List overview

Download

9 comments

3 participants

participants (3)

Ludwig Nussel
Michal Marek
Ruediger Oertel

Re: [opensuse-buildservice] .oscignore to ignore tarballs downloaded by source service?

tags

participants (3)