Bug 85446

Summary: new-run-webkit-tests reports unexpected pass of pixel tests when pixel testing is disabled
Product: WebKit Reporter: Filip Pizlo <fpizlo>
Component: Tools / TestsAssignee: Dirk Pranke <dpranke>
Status: RESOLVED FIXED    
Severity: Normal CC: abarth, dpranke, eric, fpizlo, ojan, simon.fraser
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: All   
OS: All   
Bug Depends on:    
Bug Blocks: 88680    
Attachments:
Description Flags
Screenshot
none
Patch simon.fraser: review+

Filip Pizlo
Reported 2012-05-02 18:45:45 PDT
If new-run-webkit-tests does not run a test, it should not report anything about that test; in particular, reporting that it unexpectedly passed when it wasn't run at all is very confusing.
Attachments
Screenshot (58.61 KB, image/png)
2012-06-18 17:35 PDT, Simon Fraser (smfr) 		no flags
Details
Patch (3.71 KB, patch)
2012-06-18 18:45 PDT, Dirk Pranke 		simon.fraser: review+
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Dirk Pranke
Comment 1 2012-06-18 17:26:01 PDT
We used to have a bug where we would log 'unexpected pass' for expected image failures (bug 36771), but that was fixed years ago. When I run NRWT now, I don't see any "unexpected pass" messages for tests expected to fail with IMAGE failures. However, I'm not sure I understand what you mean by "wasn't run at all"; you mean if a test foo.html is skipped, you're seeing unexpected pass for foo.html ? If you can still reproduce this, can you tell me what command line you're running with?
Simon Fraser (smfr)
Comment 2 2012-06-18 17:35:47 PDT
Created attachment 148211 [details] Screenshot Here's the issue. I did run-webkit-tests --debug (i.e. NO pixel testing) but the results page said that sometests were expected to fail, but passed. That's nonsensical, because only IMAGE failures were expected, but I'm not testing pixel results.
Dirk Pranke
Comment 3 2012-06-18 18:25:45 PDT
I see, I didn't realize we were talking about results.html ... this is happening because the html/js page doesn't know if pixel testing was disabled or not. This should be easy to fix.
Dirk Pranke
Comment 4 2012-06-18 18:45:46 PDT
Created attachment 148221 [details] Patch
Dirk Pranke
Comment 5 2012-06-19 11:03:36 PDT
Committed r120736: <http://trac.webkit.org/changeset/120736>
Note You need to log in before you can comment on or make changes to this bug.