Martin Schlander wrote:

One big problem with the current situation is that noone knows what to test and when. One of the main reasons I usually don't test early alphas is that I expect everything to change dramatically later on anyway, hence thinking testing is probably pointless at that point anyway - but I guess some components _are_ ready for widespread testing and that point, we just need to know which.

I partially agree with this. When 11.0 GM was released, there was a critical bug in the installation process which affected only Japanese users. https://bugzilla.novell.com/show_bug.cgi?id=385172#c34 The most annoying was, that the bug seemed to be fixed in RC1, which we non-Novell persons can test, but reproduced in RC4, which only Novell persons can test, and the bug was not fixed in GM. However, during the 11.1 release process, there was a call for feature testing. http://lists.opensuse.org/archive/opensuse-testing/2008-11/msg00005.html But you can see most of the Test Results for listed features remain "Idle". http://en.opensuse.org/Testing:Features_11.1 We could know *what to test and when* in this case, but as far as I can see, not so many people participated the testing. I think we have to analyze why many people don't have keen interest to testing. Do we need to spread the word much more ? Do we need to clarify what and how things to be tested and where the result should be reported ? Or, do we need to prepare benefits for dedicated testers ? IMHO, although testing itself is very important, it would be more important to figure out a way to participate testing easily, so that as many people as possible can feel "openSUSE is *our* distribution".

Additionally maybe some automation could be set up where openfate would spam this list when a new feature reaches a certain status, so as to let us know it might be something that needed testing.

openFATE screening team has just begun its activities and we are now brainstorming what can and should we do. So your opinion will be inputted. ;-) Best, -- _/_/ Satoru Matsumoto - openSUSE Member - Japan _/_/ _/_/ Marketing/Weekly News/openFATE Screening Team _/_/ _/_/ mail: helios_reds_at_gmx.net / irc: HeliosReds _/_/ _/_/ http://blog.geeko.jp/author/heliosreds _/_/ -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

On Thursday 19 March 2009 10:32:55 Tilman Schmidt wrote:

Satoru Matsumoto schrieb:

...

However, during the 11.1 release process, there was a call for feature testing. http://lists.opensuse.org/archive/opensuse-testing/2008-11/msg00005.html But you can see most of the Test Results for listed features remain "Idle". http://en.opensuse.org/Testing:Features_11.1 We could know *what to test and when* in this case, but as far as I can see, not so many people participated the testing.

I think we have to analyze why many people don't have keen interest to testing. Do we need to spread the word much more ? Do we need to clarify what and how things to be tested and where the result should be reported ? Or, do we need to prepare benefits for dedicated testers ?

This particular call for testing had two reasons why I didn't consider participating:

a) It called for testing new features, but I am much more interested, and consider it much more important, to make sure the old features still work correctly, ie. there are no regressions.

We need both IMO. We need on the one hand a defined set of use-cases that the distribution is tested against - so that in the end we can say: All use-cases were tested and that's their status so we know exactly which fail and can base a decision of releasing on that. On the other hand, we need to test new features as well.

b) "Basic reqirements: * install openSUSE 11.1 beta 4 * meet in #opensuse-testing on irc.freenode.org" I do not use IRC.

So, let's look at other ways of coordinating as well. IRC can help a lot but should not be a prerequisite for coordinated testing.

Apart from that, I must admit that my testing enthusiasm has been somewhat dampened by a recent series of WONTFIX resolutions for problems I reported, a few of them with closing remarks that seemed to imply what I reported was "my problem". So I guess positive feedback from developers that test reports are appreciated, taken seriously, and the reported issues addressed, would go a long way in encouraging more testers.

If you have any concrete examples, let's discuss these. Andreas -- Andreas Jaeger, Director Platform / openSUSE, aj@suse.de SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126

On Thursday 19 March 2009 19:10:07 Tilman Schmidt wrote:

Andreas Jaeger schrieb:

...

On Thursday 19 March 2009 10:32:55 Tilman Schmidt wrote:

...

Apart from that, I must admit that my testing enthusiasm has been somewhat dampened by a recent series of WONTFIX resolutions for problems I reported, a few of them with closing remarks that seemed to imply what I reported was "my problem". So I guess positive feedback from developers that test reports are appreciated, taken seriously, and the reported issues addressed, would go a long way in encouraging more testers.

If you have any concrete examples, let's discuss these.

Ok:

https://bugzilla.novell.com/show_bug.cgi?id=234529 - complete with patch and support offer, but after two years of waiting it won't be merged because of "no resources for ISDN anymore"

https://bugzilla.novell.com/show_bug.cgi?id=302004 - WONTFIXed after a year and a half of inactivity, with a comment I frankly don't understand at all (except for the "send patch" bit)

https://bugzilla.novell.com/show_bug.cgi?id=309382 https://bugzilla.novell.com/show_bug.cgi?id=301997 - seriously strange - I reported the problem as fixed, with no reaction from anyone, then a year later it is "set to WONTFIX because LATER and REMIND resolutions have been removed"

https://bugzilla.novell.com/show_bug.cgi?id=332696 https://bugzilla.novell.com/show_bug.cgi?id=239695 - not recent, but perfect illustrations of the point I'm trying to make

Besides the first one, all are old examples. We have to become better and I hope we do - but people are also overloaded and they will not fix everything, some stuff also needs to go upstream. If in the future, you have concrete problems, please speak up - but I'm not following up with developers on old bugs. thanks for the list, Andreas -- Andreas Jaeger, Director Platform / openSUSE, aj@suse.de SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126

Tilman Schmidt a écrit :

On Thu, 26 Mar 2009 16:11 +0100, "Andreas Jaeger" wrote:

- Obviously my citing that case here didn't have any effect either, because the entry is still RESOLVED/WONTFIX as of today.

is the maintainer still maintaining it? jdd -- http://www.dodin.net http://valerie.dodin.org http://www.youtube.com/watch?v=t-eic8MSSfM http://www.facebook.com/profile.php?id=1412160445 -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

Tilman Schmidt a écrit :

I don't know, nor do I know how to find out. I have a hunch that he might not be working for SUSE/Novell anymore, but I may be completely wrong.

Does it matter? Should it matter?

of course, it matters! jdd -- http://www.dodin.net http://valerie.dodin.org http://www.youtube.com/watch?v=t-eic8MSSfM http://www.facebook.com/profile.php?id=1412160445 -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

* Tilman Schmidt [04-12-09 06:50]:

Ok, that splits my question in three:

a) How do I get my patch included if he is still maintaining it?

Send it to the packager.

b) How do I get my patch included if he isn't maintaining it anymore?

Build the package yourself, or beg that someone else will pick it up.

c) How do I find out which of the two is the case?

I have found that *usually* the last entry in the "changelog" will provide the name/email addr of the packager.

Which are, of course, instances of general questions about the way openSUSE want's to handle external contributions:

a) What happens to contributions if a maintainer is so overburdened he or she doesn't even have the capacity to approve contributions that are otherwise ready for inclusion?

Request/assume those duties, fork, or do without.

b) What happens to contributions if a maintainer leaves?

Someone else picks it up or it dies.

c) How can contributors find their responsible maintainer (or absence of same)?

rpm -qi <package-name> or read the latest entry in the changelog -- Patrick Shanahan Plainfield, Indiana, USA HOG # US1244711 http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://counter.li.org -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

Hi, Il giorno lun, 13/04/2009 alle 23.25 +0200, Tilman Schmidt ha scritto:

When I started trying to help improving (Open)SUSE, the advice I got was that the best way to submit a patch was to open a Bugzilla entry describing the problem the patch was going to solve, and attach the proposed patch to it. Is that no longer true, and is it now acceptable to send personal mail to the packager of the package the file to be patched happens to be distributed in?

I think opening a bugreport and proposing a patch is still the way to go. But of course someone at openSUSE/Novell should confirm. Maybe we (community) can write a sort of how-to on the wiki to explain how to submit a patch, if not there already once a procedure is defined.

How can I build the standard openSUSE package hwinfo myself and get my version included instead of the current one?

To have your patch in the official package, it must be approved by some "insider", of course.

Ok. So ISDN in openSUSE dies. SuSE was once the distribution of choice for ISDN users. I guess that time is over.

I would say there is a lower number of users, so probably priorities changed. However, I agree with you it should not be so hard to get a working patch included. P.S. We are working on the creation of a community testing team (see thread on this ML). Why don't you take part to it? There will be a discussion to set it up during the openSUSE community week. Best, A. -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

Martin Schlander schrieb:

Mandag den 13. april 2009 23:25:47 skrev Tilman Schmidt:

...

When I started trying to help improving (Open)SUSE, the advice I got was that the best way to submit a patch was to open a Bugzilla entry describing the problem the patch was going to solve, and attach the proposed patch to it. Is that no longer true, and is it now acceptable to send personal mail to the packager of the package the file to be patched happens to be distributed in?

It's true if the patch is for openSUSE maintained code or packaging.

But if the patch is for upstream software, clearly the patch should be sent upstream and not to bnc.

AFAICT neither YaST (the software exposing the deficiency) nor hwinfo (the package containing the file that needs to be patched) are upstream software in that sense. But hwinfo doesn't seem to be openSUSE maintained either. So apparently there's a third category. How should that be handled? Thanks, Tilman PS: What is "bnc"? -- Tilman Schmidt E-Mail: tilman@imap.cc Bonn, Germany Diese Nachricht besteht zu 100% aus wiederverwerteten Bits. Ungeöffnet mindestens haltbar bis: (siehe Rückseite)

...

I think we have to analyze why many people don't have keen interest to testing

real tesing have to be done on a day to day use (apart from install stuff). So, IMHO, the tesing should divided in steps: * first step: install. use the basic dvd and network, etc... only with the basic config. If it installs: pass, if it don't->bugzilla. We are just now on this side with 10.2 * make a live cd with just this basic software and possibly all the hardware modules available. Testing all network and wlan is extremely important * make a general call on opensuse list: try to install the basis, try the live cd and report pass/don't pass with the hardware info (very important) - this is fast and easy, so many people is able to do * try to install and use most usually used aps. For me it's Mozilla and Thunderbird, then digikam. With this, I begin to be able to work with the new install. then go on with more apps, of course. Once a module or an app is validated by a sufficient number of users, slowdown any new feature on it. you can without doubt (you, any reader here) add some important apps jdd -- http://www.dodin.net http://valerie.dodin.org http://www.youtube.com/watch?v=t-eic8MSSfM http://www.facebook.com/profile.php?id=1412160445 -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

Il giorno mar, 07/04/2009 alle 17.50 +0200, Holger Sickenberg ha scritto:

I was following this thread quite interested and my summary of it is what I did find out before also:

It seem to be quite hard to find people that are really interested in testing openSUSE (by various reasons).

So I really like your idea of creating such a testing team and would be glad if you/we do succeed with it. I think it would be worth to spend efforts into such a task and I'm willing to help. If there is anything I can do in enabling this team just let me know. You will get my full support.

Hi Holeger, sorry for my late answer! Thanks for the encouragement and for the interest in my idea. Indeed it seems hard to attract people to do that! Anyway, I still have some hope: I found some volunteer from the local Italian community (suseitalia.org) and they seem quite motivated to help. Anyway, we should really discuss on how to proceed in setting things up. Ideally it would be nice to start with alphas, but being the first time we try to do something like this, anytime is OK. At the moment I don't understand if there is interest for this idea on the Novell side though (Are you at N, btw?). I just had some quick talk with Zonker, who showed his interest, and who said he would have contacted Oliver Ries at suse.de to know what the existing infrastructure is. Still no answer. If you know something about that, it would be useful to talk about it too. :-) Let's talk soon, when you have time. Best, Alberto -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

Hello, On Thu, Apr 09, 2009 at 08:47:37AM +0200, Holger Sickenberg wrote:

...

Anyway, we should really discuss on how to proceed in setting things up. Ideally it would be nice to start with alphas, but being the first time we try to do something like this, anytime is OK.

As far as I know there will not be any alpha and beta versions but something like milestones with 11.2 - so yes: starting to test with the first milestone make sense.

one of the main problems is (excuse my french) the management of it. As in what needs to be tested, how it should be tested or how it was tested, what has already been tested, how can we organize who tests what, how do test results make it back into Suse? How is the time line for opening Testopia to the outside, perhaps (initially) only for a selected "openSUSE test group"? Then we would have a test plan, would be able to avoid unneccessary double testing, and results would automatically find their way back to Suse QA.

I volunteered to take over the action item of the Testing/QA section of the CommunityWeek (http://en.opensuse.org/CommunityWeek). So maybe we can use that as kind of kick-off for the testing team.

That's a good idea. I'm particularly interested in the questions above, the organisation. Just doing isolated, random testing is not going to be very efficient. Regards, Rasmus -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

On Thu, Apr 09, 2009 at 09:25:36AM -0400, Patrick Shanahan wrote:

* Rasmus Plewe [04-09-09 04:55]:

...

How is the time line for opening Testopia to the outside, perhaps (initially) only for a selected "openSUSE test group"? Then we would have a test plan, would be able to avoid unneccessary double testing, and results would automatically find their way back to Suse QA.

Maybe that should read: Opening TestSUSE

The name of the tool used internally is Testopia, not TestSUSE. Regards, Rasmus -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

Hi, maybe we should first set the team up, and then worry about how tools are called? ;-) Regards, A. Il giorno gio, 09/04/2009 alle 16.09 -0400, Patrick Shanahan ha scritto:

I understand that, but maybe it shouldn't be, afterall it is for openSUSE. A little brand recognition :^)

-- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

Hi Holger, thanks for you reply! Il giorno gio, 09/04/2009 alle 08.47 +0200, Holger Sickenberg ha scritto:

As far as I know there will not be any alpha and beta versions but something like milestones with 11.2 - so yes: starting to test with the first milestone make sense.

OK.

...

At the moment I don't understand if there is interest for this idea on the Novell side though (Are you at N, btw?). I just had some quick talk with Zonker, who showed his interest, and who said he would have contacted Oliver Ries at suse.de to know what the existing infrastructure is. Still no answer. If you know something about that, it would be useful to talk about it too. :-)

Let's talk soon, when you have time.

There is interest by Novell: Zonker already told you about that. Yes, I'm also working for SUSE/N (Oliver is my boss) and I'm quite interested as well. Which information do you like to have?

Yup, Henne clarified the situation to me yesterday. Mainly I was interested to know what tools are available and some general idea of how testing works at SUSE.

I volunteered to take over the action item of the Testing/QA section of the CommunityWeek (http://en.opensuse.org/CommunityWeek). So maybe we can use that as kind of kick-off for the testing team.

Yeah. This sounds a good idea. I'm at work during that week, however it should not be overbusy, and I can find some time.

PS: I will be on vacation the next couple of days but should read my e-mails anyway. Maybe some answers will last a little bit longer.

No prob! Enjoy your vacation :-) Best, Alberto -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

Hi Holger, Il giorno ven, 10/04/2009 alle 11.20 +0200, Holger Sickenberg ha scritto:

Do you have that already or do you want me to tell something about the current way of testing at SUSE?

We did not talk about the current way of testing with Henne, so it would be great if you could tell me something :-)

Feel free to add your thoughts and ideas to the Testing/QA section of the wiki page mentioned above.

OK. Thanks, A. -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

Holger Sickenberg a écrit :

1)Installation/Update Tests Test installing and updating from the former openSUSE version (for 11.1 it was 11.0)

testing the install is pretty easy (need only to keep a partition free), testing *update* seems to me much more difficult, at least it would need some sort of HOWTO: howto backup previous install, just in case things go wrong or to be able to do it again (g4l?) Install should have a list of all the kind of hardware needing detailed tests use of automated HW list could allow to ask one defined tester who have one particular HW :-) all this seems mostly kernel directed? couldn't we also use the other distribution bugzilla to know what is wrong or share infos on a given kernel?

All test cases and results are stored in the tool "Testopia" (an enhancement to Bugzilla) which is unfortunately just accessable for people inside Novell.

this is odd and should be the first priority (at least open to test team or members)

- test of installation/updates on a wider hardware base - better test coverage of feature tests - creating and executing of test cases for base system and applications

true thanks jdd -- http://www.dodin.net http://valerie.dodin.org http://www.youtube.com/watch?v=t-eic8MSSfM http://www.facebook.com/profile.php?id=1412160445 -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org

Holger Sickenberg wrote:

Hi Alberto,

Hi guys, my name is Christian Hueller, i also work in the QA department at Novell/SUSE and also would like to offer help and information to a community quality group.

...

...

Do you have that already or do you want me to tell something about the current way of testing at SUSE?

We did not talk about the current way of testing with Henne, so it would be great if you could tell me something :-)

Going not too much into details: current testing of openSUSE is divided in three parts: 1)Installation/Update Tests Test installing and updating from the former openSUSE version (for 11.1 it was 11.0) of every milestone (beta) version both on x86 and x86_64. (If time permits also installation tests on PlayStation 3) 2)Automated Kernel Test Running a subset of available Kernel Test Suites on every milestone version (x86/x86_64) 3)Feature Tests Based on the list of new features test if features are implemented and working as expected (see http://en.opensuse.org/Testing:Features_11.1)

All test cases and results are stored in the tool "Testopia" (an enhancement to Bugzilla) which is unfortunately just accessable for people inside Novell.

We are working on opening it asap, however this might still be some time into the future. In case somebody is interested in the tool itself, one can find it's project pages here: http://www.mozilla.org/projects/testopia/.

With the help of the openSUSE community (developers/testing team) I do see room for improvement for milestone testing in following areas: - test of installation/updates on a wider hardware base - better test coverage of feature tests - creating and executing of test cases for base system and applications (also useful for maintenance updates of packages during product lifetime)

I hope this information is a good base for further discussions.

Let me add some more bits, especially areas where we are still expanding our efforts. o Benchmarking is becoming a real big item in QA. Kernel performance benchmarks showed some performance degradation in recent kernel releases which pointed to on the one hand still insufficient benchmarks and the need for some process to check for regression on the other hand. o Automated testing and coverage (outside the kernel) - currently mostly CLI based applications - next big item: GUI testing, since QT4 is used, LDTP (http://ldtp.freedesktop.org/wiki/) is becoming an option * Related to both above: Open Source Test Suites. There are plenty available, however reviewing and adopting them eats up time, so this also is an ongoing process. Here I see a lot of gain from more people simply looking and talking about them in a public discussion. o Documentation. It is there, though it could be better - as always ;) o much more I'm just not thinking of right now. There are three pages on opensuse.org right now (maybe more I'm not aware of), which I think can cause confusion or lead to missing information. I see these currently: http://en.opensuse.org/QA -> Redirects to http://en.opensuse.org/QA_Team, an Overview of QA personnel at Novell/SUSE http://opensuse.org/QA -> Empty page http://en.opensuse.org/Testing -> Testing page created around 11.1, mostly of 11.1 related test material My idea would be to use http://opensuse.org/QA as kind of entry point and then link to all other QA/Testing related pages from there, but thats just an suggestion, I do not want to tell you what to do but mere try to support you in the decisions that you make and give input where I have it. I'll send out some presentations and documentation if it is up to date and valid, just to give an better impression what QA is doing in here. If you have questions, please ask, we'll try to help the best we can. Have fun and thanks, Chris

Best wishes, Holgi

-- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-testing+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-testing+help@opensuse.org