Mailinglist Archive: zypp-commit (301 mails)

< Previous Next >
[zypp-commit] r7534 - in /trunk/sat-solver/testsuite: README.FAILS runtest.rb
  • From: coolo@xxxxxxxxxxxxxxxx
  • Date: Mon, 15 Oct 2007 09:53:02 -0000
  • Message-id: <20071015095302.D9B4045675@xxxxxxxxxxxxxxxx>
Author: coolo
Date: Mon Oct 15 11:53:02 2007
New Revision: 7534

URL: http://svn.opensuse.org/viewcvs/zypp?rev=7534&view=rev
Log:
learned ruby in two hours - implemented expected and unexpected
failures and passes:
  ==> 241 tests: (203/0) passed, (38/0) failed <==

(everything expected)

Modified:
    trunk/sat-solver/testsuite/README.FAILS
    trunk/sat-solver/testsuite/runtest.rb

Modified: trunk/sat-solver/testsuite/README.FAILS
URL: http://svn.opensuse.org/viewcvs/zypp/trunk/sat-solver/testsuite/README.FAILS?rev=7534&r1=7533&r2=7534&view=diff
==============================================================================
--- trunk/sat-solver/testsuite/README.FAILS (original)
+++ trunk/sat-solver/testsuite/README.FAILS Mon Oct 15 11:53:02 2007
@@ -1,72 +1,64 @@
-basic-exercises/Bug299819-2-test.xml
-yast-tests/Bug159696-test.xml
+./data.libzypp/basic-exercises/Bug299819-2-test.xml
+./data.libzypp/yast-tests/Bug159696-test.xml
        installs recommends
 
-basic-exercises/exercise-27-force-test.xml
+./data.libzypp/basic-exercises/exercise-27-force-test.xml
        removes "uninstallables" - weired decision
 
-yast-tests/exercise-xfree-test.xml
-yast-tests/exercise-obscureobsoletes-01-test.xml
-yast-tests/exercise-kdelibsobs-force-test.xml
-yast-tests/exercise-corcircle-test.xml
-yast-tests/exercise-04obsoletes-09-test.xml
-yast-tests/exercise-04obsoletes-06-test.xml
-yast-tests/exercise-04obsoletes-01-test.xml
-yast-tests/exercise-02conflict-07-test.xml
-yast-tests/exercise-02conflict-09-force-test.xml
-yast-tests/exercise-02conflict-09-test.xml
-yast-tests/exercise-02conflict-12-test.xml
-yast-tests/exercise-02conflict-13-force-test.xml
-basic-exercises/exercise-60-test.xml
-basic-exercises/exercise-48-test.xml
-basic-exercises/exercise-40-test.xml
+./data.libzypp/yast-tests/exercise-xfree-test.xml
+./data.libzypp/yast-tests/exercise-obscureobsoletes-01-test.xml
+./data.libzypp/yast-tests/exercise-kdelibsobs-force-test.xml
+./data.libzypp/yast-tests/exercise-corcircle-test.xml
+./data.libzypp/yast-tests/exercise-04obsoletes-09-test.xml
+./data.libzypp/yast-tests/exercise-04obsoletes-06-test.xml
+./data.libzypp/yast-tests/exercise-04obsoletes-01-test.xml
+./data.libzypp/yast-tests/exercise-02conflict-09-force-test.xml
+./data.libzypp/yast-tests/exercise-02conflict-09-test.xml
+./data.libzypp/yast-tests/exercise-02conflict-12-test.xml
+./data.libzypp/yast-tests/exercise-02conflict-13-force-test.xml
+./data.libzypp/basic-exercises/exercise-60-test.xml
+./data.libzypp/basic-exercises/exercise-48-test.xml
+./data.libzypp/basic-exercises/exercise-40-test.xml
        does not want to remove obsoletes
 
-yast-tests/kernel3-test.xml
-yast-tests/tpctl-bigsmp-test.xml
-yast-tests/Bug156513-test.xml
-yast-tests/kernel2-test.xml
+./data.libzypp/yast-tests/kernel3-test.xml
+./data.libzypp/yast-tests/tpctl-bigsmp-test.xml
+./data.libzypp/yast-tests/Bug156513-test.xml
+./data.libzypp/yast-tests/kernel2-test.xml
        installs 3 kernels and not just the default as wanted
 
-yast-tests/exercise-inconsistent6-force-test.xml
-basic-exercises/exercise-36-test.xml
-yast-tests/bug155368-force-test.xml
+./data.libzypp/yast-tests/exercise-inconsistent6-force-test.xml
+./data.libzypp/basic-exercises/exercise-36-test.xml
+./data.libzypp/yast-tests/bug155368-force-test.xml
        libzypp removes broken pack, mls changes version
 
-basic-exercises/exercise-44-delete-glibc-force-test.xml
-basic-exercises/exercise-42-force-test.xml
-basic-exercises/exercise-32-test.xml
-basic-exercises/exercise-35-test.xml
-basic-exercises/exercise-37-test.xml
-basic-exercises/exercise-44-test.xml
+./data.libzypp/basic-exercises/exercise-44-delete-glibc-force-test.xml
+./data.libzypp/basic-exercises/exercise-42-force-test.xml
+./data.libzypp/basic-exercises/exercise-32-test.xml
+./data.libzypp/basic-exercises/exercise-35-test.xml
+./data.libzypp/basic-exercises/exercise-37-test.xml
+./data.libzypp/basic-exercises/exercise-44-test.xml
        libzypp (de)installs less packages - looks like a libzypp bug to me
 
-yast-tests/exercise-06upgrade-test.xml
-simple-tests/vendor-test.xml
-kernel-tests/1_2_1-default-test.xml
-kernel-tests/2_2_1-default-test.xml
-kernel-tests/2_2_2-default-test.xml
-kernel-tests/1_2_2-default-test.xml
-kernel-tests/1_2_2-smp-test.xml
-distupgrade-tests/unmaintained-nodelete-single-test.xml
+./data.libzypp/yast-tests/exercise-06upgrade-test.xml
        distupgrade unhandled
 
-basic-exercises/exercise-38-test.xml
+./data.libzypp/basic-exercises/exercise-38-test.xml
        forceuninstall unhandled
 
-basic-exercises/exercise-39-force-test.xml
+./data.libzypp/basic-exercises/exercise-39-force-test.xml
        mls removes a package not fullfilling requirements on install of another
 
-basic-exercises/exercise-47-test.xml
-basic-exercises/exercise-43-test.xml
+./data.libzypp/basic-exercises/exercise-47-test.xml
+./data.libzypp/basic-exercises/exercise-43-test.xml
        the test case's solution is not what libzypp is doing (INVALID)
        
-basic-exercises/provide-1-test.xml
+./data.libzypp/basic-exercises/provide-1-test.xml
        addProvide is not supported
 
-basic-exercises/require-1-test.xml
+./data.libzypp/basic-exercises/require-1-test.xml
        B requires c, but nothing providing C is installed
 
-yast-tests/Bug162887-test.xml
+./data.libzypp/yast-tests/Bug162887-test.xml
        mls installs sendmail, libzypp prefers postfix as recommended by selections
 

Modified: trunk/sat-solver/testsuite/runtest.rb
URL: http://svn.opensuse.org/viewcvs/zypp/trunk/sat-solver/testsuite/runtest.rb?rev=7534&r1=7533&r2=7534&view=diff
==============================================================================
--- trunk/sat-solver/testsuite/runtest.rb (original)
+++ trunk/sat-solver/testsuite/runtest.rb Mon Oct 15 11:53:02 2007
@@ -20,8 +20,19 @@
 
 $tests = Array.new
 $deptestomatic = File.join( Dir.getwd, "deptestomatic" )
+$topdir = Dir.getwd
+$fails = Array.new
+
+class CompareResult
+  Incomplete = 0
+  KnownFailure = 1
+  UnexpectedFailure = 2
+  KnownPass = 3
+  UnexpectedPass = 4
+end
 
 class Solution
+
   # poor mans diff
   def Solution.filediff name1, name2
     begin
@@ -91,29 +102,47 @@
   def Solution.compare sname, rname
     unless File.readable?( sname )
       STDERR.puts "Cannot open #{sname}"
-      return false
+      return CompareResult::Incomplete
     end
     unless File.readable?( rname )
       STDERR.puts "Cannot open #{rname}"
-      return false
+      return CompareResult::Incomplete
     end
     
     solutions = Solution.read sname
     results = Solution.read rname
     
-    return true if (solutions.empty? && results.empty?)
+    if (solutions.empty? && results.empty?)
+      if ( $fails.member?( sname ) )
+       STDERR.puts "#{rname} passed"
+       return CompareResult::UnexpectedPass
+      else
+       return CompareResult::KnownPass  
+      end
+    end
+
     r = results.first
     solutions.each { |s|
-      return true if s == r
+      if s == r
+       if ( $fails.member?( sname ) )
+         STDERR.puts "#{rname} passed"
+         return CompareResult::UnexpectedPass
+       else
+         return CompareResult::KnownPass  
+       end
+      end
     }
     
-    #STDERR.puts "#{rname} failed"
+    if ( $fails.member?( sname ) )
+      return CompareResult::KnownFailure
+    end
+    STDERR.puts "#{rname} failed"
     system( "./diffres #{sname} #{rname}")
     #STDERR.puts "Solution:"
     #pp solutions.first
     #STDERR.puts "Result:"
     #pp r
-    return false
+    return CompareResult::UnexpectedFailure
   end
 
 end
@@ -122,7 +151,10 @@
 class Tester < Test::Unit::TestCase
   
   def test_run
-    passed = 0
+    upassed = 0
+    epassed = 0
+    ufailed = 0
+    efailed = 0
     puts "#{$tests.size} tests ahead"
     $tests.sort!
     $tests.each { |test|
@@ -137,13 +169,21 @@
         sname = File.join( dir, "#{basename}.solution" )
         rname = File.join( dir, "#{basename}.result" )
        result = Solution.compare( sname, rname ) 
-       passed += 1 if result
+       case result
+       when CompareResult::UnexpectedFailure
+         ufailed += 1
+       when CompareResult::UnexpectedPass
+         upassed += 1
+       when CompareResult::KnownFailure
+         efailed += 1
+       when CompareResult::KnownPass
+         epassed += 1
+       end
 #        assert(  )
 #      puts "(cd #{File.dirname(test)}; #{$deptestomatic} #{basename}.xml > #{basename}.result)" 
       end
     }
-    failed = $tests.size - passed
-    puts "\n\t==> #{$tests.size} tests: #{passed} passed, #{failed} failed <==\n"
+      puts "\n\t==> #{$tests.size} tests: (#{epassed}/#{upassed}) passed, (#{efailed}/#{ufailed}) failed <==\n"
   end
 end
 
@@ -210,9 +250,17 @@
   ARGV.shift
 end
 
+IO.foreach( "README.FAILS") { |line|
+  line.chomp
+  if ( line !~ /^\s/ )
+    line = line[0..-6] + ".solution"
+    $fails << line
+  end
+}
+
 r = Runner.new
 
 ARGV.each { |arg|
-  wd = Dir.getwd unless arg[0,1] == "/"
+  wd = "." unless arg[0,1] == "/"
   r.run wd, arg, recurse
 }

--
To unsubscribe, e-mail: zypp-commit+unsubscribe@xxxxxxxxxxxx
For additional commands, e-mail: zypp-commit+help@xxxxxxxxxxxx

< Previous Next >
This Thread
  • No further messages