Hello, on Montag, 7. Juli 2008, Marko Jung wrote:

after a lot of inspiring conversations with co-workers at SUSE and friends from the open source community I finally published a first abstract draft how we could add some metrics to openSUSE build service to create a trust system.

http://en.opensuse.org/Build_Service/Concepts/Trust

Your text looks like you have invested lots of time :-) Nevertheless I found some things I'd like to comment on: Despite all proposed values make an effort to evaluate a package and not its producers, this project does a lot of data mining. Therefore, it has to be evaluated if there are any violations of privacy. Very valid point! In addition to this, all metrics which would enable performance measurement of Novell employees should be disabled by default as soon as the employee flag is set for an account. Please use the same defaults for everybody. If you think something should be disabled by default for Novell employees, there's a great chance community packagers also don't want to have it ;-) (Yes, you'll probably miss some data because of this. But you'll also miss several complaints about not respecting privacy.) Quality & maintenance Without a strong connection between packages and bugs, it is nearly impossible to get quality and security metrics for packages. Therefore, I strongly recommend to integrate a possibility to assign a bug to a package, like all other major distributions do in Novell's Bugzilla. Agreed - a strong connection to bugzilla would be very helpful. On the other hand, the build service team should consider creating an own bug tracker because it may not be desirable to have all bugs for all packages from the openSUSE build service within Novell's bug tracking system. The preferred way would probably be an own bug tracking system for each build service project. I only see disadvantages when using separate bug trackers: - searching for bugs is more complicated (you have to search multiple places if a package is also in the distribution) - it's difficult to make a bug in a buildservice package a bug in the distribution (and the other way round) - you'll need to copy&paste everything to another bugtracker. - and, BTW, I don't see why it would do any harm to b.n.c if all bugs go there ;-) My proposal: - create a product "build service packages" on bugzilla.novell.com (optional: split home:* to a separate product) - automatically create a component for each project in the build service, with the configured bugowner as default assignee and QA contact - let the "report a bug" link preselect the component (aka project) and pre-populate the description with package name and version - if a package has a different bugowner than the project, I propose to set the assignee to the package bug owner and QA to the project bug owner - oh, and please make sure the bugzilla team finally fixes bug 374374 (non-public, short summary: user lookup is disabled for community members :-( ) Regards, Christian Boltz -- Come on, just change ftp to ftp4 and go on. 1000 OpenSUSE users more or less a day really is peanuts. ;-)) [Eberhard Moenkeberg in opensuse] --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

On Monday 07 July 2008, Marko Jung wrote:

http://en.opensuse.org/Build_Service/Concepts/Trust

[quote rpmlint] A first implementation should only check whether any test failed and possibly how many problems have been found. As a subsequent improvement, I would suggest to add a parser to identify the failed tests and to further add a severity measure for the identified problems. [/quote] We already have such a system with rpmlint: each check has an associated "badness" score that is cumulated. However, it is a weak indication of trust: checks can be filtered, and if one chooses to trust a package that has certain "fatal" checks filtered depends on quite some factors that one probably can not easily judge. [quality checks] I think the most obvious ones here are missing: no implicit file conflicts, no "evil" reverse dependencies (supplements, enhances), no conflicts or obsoletes on core packages. nonstandardized %pre/%post scripts are also something that should lower trust. Number of services that are installed at boot, firewall exceptions etc. [Reverse build dependencies] I think the paragraph is the wrong way around: a package doesn't need a high trust level if it is a leaf - package. if it is a core package with many other things depending on it, we don't want to have "newbie changes" in there. Note that this is not only about buildtime but also about runtime dependencies. [action based ratings] I agree that actions are the best indicator for trust - an accepted submission request should grow trust between the two parties, permanently declined requests lower it. submission "dialogs" however again indicate an ongoing relationship. [package history / version numbers] The easiest way is probably comparing timestamps of the files in the source tarball versus timestamp of the spec file. or something like that. In any case I think this is not really useful as an indicator of trust: if the time between upstream and package is really short, then this either indicates a script at work or some spoofing going on. if the time is long, it does not mean much either: it could be that the previous packager dropped dead and somebody else finally came around updating the package. this situation should increase the trust in the package instead of lowering it. Overall, I think the conceptual description is not very clear and is lacking some fundamental decisions, which make evaluation of the proposed concept difficult: a) is trust an attribute of an identity, of modifications, projects, packages or binaries? I think this is the most important question to answer. according to the introduction, trust appears to be a relationship between identities. identities are however only loosely connected to projects or packages. the proposed implementation suggests that the trust is related to a project (lowest trustlevel of all maintainers). The implementation proposal is btw flawed - trust has to be sticky, otherwise I can trojan a project and afterwards remove myself from the package maintainership list, and the package would immediately gain high trust level again. I suggest to split and clarify the definition of trust as a relationship between and as an entity of modifications (submit requests or OBS commits). It is not generally true that a brillant packager of package foo is also good when he touches package bar. b) the novell/opensuse employee exceptions in the trust scheme do not make it trustworthy to outsiders. an opensuse employee is not inherently less vulnerable to screwing something up. it should be possible to reduce those exceptions to a bare minimum, or make them symmetric so that any outside group of people can gain the same privileges vice versa. c) the quoted definition for trust is quite good: A trusted party is presumed to seek to fulfill ethical codes, law and policies. Therefore it would be useful to propose a list of policies (law's) that allow, when being fulfilled, being an indicator of trust. One obvious example: the source of tarballs should be verifyable, be it URL, gpg signature of the packager or some other way defined by the upstream project. Other indicators are easy to come up with. for each of those, the impact on the trust should be elaborated: how bad is it if a tarball is updated but the signature is not? how bad is it if a signature is removed upon a version upgrade? how good is it if somebody adds a signature or if somebody verifies the tarball? Note that trusting somebody doesn't mean that some weird datamining determined him to be trustworthy. it means that, a particular person seeked to fulfill previous promises, ethical standards and policies. A clear prerequirement here is: the list of promises and policies to be defined from the start. d) category ratings&Acceptance: This is a difficult part. how much impact has a rating from a total stranger? how do you define trust for a rating from somebody to whom you have no trust relationship with? Also, are ratings related to an identity, to a package, to a particular package version? can people grant cookies (virtual beer) to people who fix their bugreports in the package? can they express distrust to a package when a version upgrade, when being installed, hoses their system? and if they do so, is the distrust bouncing back to the packager, to the lastest change, to the package..? if they rate a package, does that increase trust in the packager, in the one doing the latest change, in the package..? e) we want to reward people for complying to policies and for being active. we don't necessarily want to reward them for screwing up, but as time goes by, shit happens. the stickyness of bad behaviour in trust systems like e.g. ebay cause people to change their identity quite frequently. we would like to avoid that. I would propose therefore to track badness of packages independently of "badness" of identities. it should be possible to get rid of "mistrust" by dropping a really bad package (perhaps it was all upstreams fault, really), and to gain trust by being more active and interacting with more people. A person who seeks relationship is more trustworthy than one who does not. A reward system might be a good indicator and motivational factor for people to participate in the trust scheme. an important factor for that is to define privileges or advantages and to make the rules for the trust as simple and as fair as possible. Greetings, Dirk --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org