Author: coolo Date: Mon Oct 15 11:53:02 2007 New Revision: 7534 URL: http://svn.opensuse.org/viewcvs/zypp?rev=7534&view=rev Log: learned ruby in two hours - implemented expected and unexpected failures and passes: ==> 241 tests: (203/0) passed, (38/0) failed <== (everything expected) Modified: trunk/sat-solver/testsuite/README.FAILS trunk/sat-solver/testsuite/runtest.rb Modified: trunk/sat-solver/testsuite/README.FAILS URL: http://svn.opensuse.org/viewcvs/zypp/trunk/sat-solver/testsuite/README.FAILS?rev=7534&r1=7533&r2=7534&view=diff ============================================================================== --- trunk/sat-solver/testsuite/README.FAILS (original) +++ trunk/sat-solver/testsuite/README.FAILS Mon Oct 15 11:53:02 2007 @@ -1,72 +1,64 @@ -basic-exercises/Bug299819-2-test.xml -yast-tests/Bug159696-test.xml +./data.libzypp/basic-exercises/Bug299819-2-test.xml +./data.libzypp/yast-tests/Bug159696-test.xml installs recommends -basic-exercises/exercise-27-force-test.xml +./data.libzypp/basic-exercises/exercise-27-force-test.xml removes "uninstallables" - weired decision -yast-tests/exercise-xfree-test.xml -yast-tests/exercise-obscureobsoletes-01-test.xml -yast-tests/exercise-kdelibsobs-force-test.xml -yast-tests/exercise-corcircle-test.xml -yast-tests/exercise-04obsoletes-09-test.xml -yast-tests/exercise-04obsoletes-06-test.xml -yast-tests/exercise-04obsoletes-01-test.xml -yast-tests/exercise-02conflict-07-test.xml -yast-tests/exercise-02conflict-09-force-test.xml -yast-tests/exercise-02conflict-09-test.xml -yast-tests/exercise-02conflict-12-test.xml -yast-tests/exercise-02conflict-13-force-test.xml -basic-exercises/exercise-60-test.xml -basic-exercises/exercise-48-test.xml -basic-exercises/exercise-40-test.xml +./data.libzypp/yast-tests/exercise-xfree-test.xml +./data.libzypp/yast-tests/exercise-obscureobsoletes-01-test.xml +./data.libzypp/yast-tests/exercise-kdelibsobs-force-test.xml +./data.libzypp/yast-tests/exercise-corcircle-test.xml +./data.libzypp/yast-tests/exercise-04obsoletes-09-test.xml +./data.libzypp/yast-tests/exercise-04obsoletes-06-test.xml +./data.libzypp/yast-tests/exercise-04obsoletes-01-test.xml +./data.libzypp/yast-tests/exercise-02conflict-09-force-test.xml +./data.libzypp/yast-tests/exercise-02conflict-09-test.xml +./data.libzypp/yast-tests/exercise-02conflict-12-test.xml +./data.libzypp/yast-tests/exercise-02conflict-13-force-test.xml +./data.libzypp/basic-exercises/exercise-60-test.xml +./data.libzypp/basic-exercises/exercise-48-test.xml +./data.libzypp/basic-exercises/exercise-40-test.xml does not want to remove obsoletes -yast-tests/kernel3-test.xml -yast-tests/tpctl-bigsmp-test.xml -yast-tests/Bug156513-test.xml -yast-tests/kernel2-test.xml +./data.libzypp/yast-tests/kernel3-test.xml +./data.libzypp/yast-tests/tpctl-bigsmp-test.xml +./data.libzypp/yast-tests/Bug156513-test.xml +./data.libzypp/yast-tests/kernel2-test.xml installs 3 kernels and not just the default as wanted -yast-tests/exercise-inconsistent6-force-test.xml -basic-exercises/exercise-36-test.xml -yast-tests/bug155368-force-test.xml +./data.libzypp/yast-tests/exercise-inconsistent6-force-test.xml +./data.libzypp/basic-exercises/exercise-36-test.xml +./data.libzypp/yast-tests/bug155368-force-test.xml libzypp removes broken pack, mls changes version -basic-exercises/exercise-44-delete-glibc-force-test.xml -basic-exercises/exercise-42-force-test.xml -basic-exercises/exercise-32-test.xml -basic-exercises/exercise-35-test.xml -basic-exercises/exercise-37-test.xml -basic-exercises/exercise-44-test.xml +./data.libzypp/basic-exercises/exercise-44-delete-glibc-force-test.xml +./data.libzypp/basic-exercises/exercise-42-force-test.xml +./data.libzypp/basic-exercises/exercise-32-test.xml +./data.libzypp/basic-exercises/exercise-35-test.xml +./data.libzypp/basic-exercises/exercise-37-test.xml +./data.libzypp/basic-exercises/exercise-44-test.xml libzypp (de)installs less packages - looks like a libzypp bug to me -yast-tests/exercise-06upgrade-test.xml -simple-tests/vendor-test.xml -kernel-tests/1_2_1-default-test.xml -kernel-tests/2_2_1-default-test.xml -kernel-tests/2_2_2-default-test.xml -kernel-tests/1_2_2-default-test.xml -kernel-tests/1_2_2-smp-test.xml -distupgrade-tests/unmaintained-nodelete-single-test.xml +./data.libzypp/yast-tests/exercise-06upgrade-test.xml distupgrade unhandled -basic-exercises/exercise-38-test.xml +./data.libzypp/basic-exercises/exercise-38-test.xml forceuninstall unhandled -basic-exercises/exercise-39-force-test.xml +./data.libzypp/basic-exercises/exercise-39-force-test.xml mls removes a package not fullfilling requirements on install of another -basic-exercises/exercise-47-test.xml -basic-exercises/exercise-43-test.xml +./data.libzypp/basic-exercises/exercise-47-test.xml +./data.libzypp/basic-exercises/exercise-43-test.xml the test case's solution is not what libzypp is doing (INVALID) -basic-exercises/provide-1-test.xml +./data.libzypp/basic-exercises/provide-1-test.xml addProvide is not supported -basic-exercises/require-1-test.xml +./data.libzypp/basic-exercises/require-1-test.xml B requires c, but nothing providing C is installed -yast-tests/Bug162887-test.xml +./data.libzypp/yast-tests/Bug162887-test.xml mls installs sendmail, libzypp prefers postfix as recommended by selections Modified: trunk/sat-solver/testsuite/runtest.rb URL: http://svn.opensuse.org/viewcvs/zypp/trunk/sat-solver/testsuite/runtest.rb?rev=7534&r1=7533&r2=7534&view=diff ============================================================================== --- trunk/sat-solver/testsuite/runtest.rb (original) +++ trunk/sat-solver/testsuite/runtest.rb Mon Oct 15 11:53:02 2007 @@ -20,8 +20,19 @@ $tests = Array.new $deptestomatic = File.join( Dir.getwd, "deptestomatic" ) +$topdir = Dir.getwd +$fails = Array.new + +class CompareResult + Incomplete = 0 + KnownFailure = 1 + UnexpectedFailure = 2 + KnownPass = 3 + UnexpectedPass = 4 +end class Solution + # poor mans diff def Solution.filediff name1, name2 begin @@ -91,29 +102,47 @@ def Solution.compare sname, rname unless File.readable?( sname ) STDERR.puts "Cannot open #{sname}" - return false + return CompareResult::Incomplete end unless File.readable?( rname ) STDERR.puts "Cannot open #{rname}" - return false + return CompareResult::Incomplete end solutions = Solution.read sname results = Solution.read rname - return true if (solutions.empty? && results.empty?) + if (solutions.empty? && results.empty?) + if ( $fails.member?( sname ) ) + STDERR.puts "#{rname} passed" + return CompareResult::UnexpectedPass + else + return CompareResult::KnownPass + end + end + r = results.first solutions.each { |s| - return true if s == r + if s == r + if ( $fails.member?( sname ) ) + STDERR.puts "#{rname} passed" + return CompareResult::UnexpectedPass + else + return CompareResult::KnownPass + end + end } - #STDERR.puts "#{rname} failed" + if ( $fails.member?( sname ) ) + return CompareResult::KnownFailure + end + STDERR.puts "#{rname} failed" system( "./diffres #{sname} #{rname}") #STDERR.puts "Solution:" #pp solutions.first #STDERR.puts "Result:" #pp r - return false + return CompareResult::UnexpectedFailure end end @@ -122,7 +151,10 @@ class Tester < Test::Unit::TestCase def test_run - passed = 0 + upassed = 0 + epassed = 0 + ufailed = 0 + efailed = 0 puts "#{$tests.size} tests ahead" $tests.sort! $tests.each { |test| @@ -137,13 +169,21 @@ sname = File.join( dir, "#{basename}.solution" ) rname = File.join( dir, "#{basename}.result" ) result = Solution.compare( sname, rname ) - passed += 1 if result + case result + when CompareResult::UnexpectedFailure + ufailed += 1 + when CompareResult::UnexpectedPass + upassed += 1 + when CompareResult::KnownFailure + efailed += 1 + when CompareResult::KnownPass + epassed += 1 + end # assert( ) # puts "(cd #{File.dirname(test)}; #{$deptestomatic} #{basename}.xml > #{basename}.result)" end } - failed = $tests.size - passed - puts "\n\t==> #{$tests.size} tests: #{passed} passed, #{failed} failed <==\n" + puts "\n\t==> #{$tests.size} tests: (#{epassed}/#{upassed}) passed, (#{efailed}/#{ufailed}) failed <==\n" end end @@ -210,9 +250,17 @@ ARGV.shift end +IO.foreach( "README.FAILS") { |line| + line.chomp + if ( line !~ /^\s/ ) + line = line[0..-6] + ".solution" + $fails << line + end +} + r = Runner.new ARGV.each { |arg| - wd = Dir.getwd unless arg[0,1] == "/" + wd = "." unless arg[0,1] == "/" r.run wd, arg, recurse } -- To unsubscribe, e-mail: zypp-commit+unsubscribe@opensuse.org For additional commands, e-mail: zypp-commit+help@opensuse.org