Bug 152257

Summary: Mac EWS bots frequently fail several wpt tests
Product: WebKit Reporter: youenn fablet <youennf>
Component: Tools / TestsAssignee: youenn fablet <youennf>
Status: RESOLVED FIXED    
Severity: Normal CC: ap, commit-queue, lforschler, ryanhaddad
Priority: P2    
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=152297
https://bugs.webkit.org/show_bug.cgi?id=152298
https://bugs.webkit.org/show_bug.cgi?id=152335
Bug Depends on:    
Bug Blocks: 152334    
Attachments:
Description Flags
Patch none

youenn fablet
Reported 2015-12-14 09:59:43 PST
Some WPT tests have difficulties passing in Mac EWS bots. But I cannot find flakiness in the dashboard. imported/w3c/web-platform-tests/dom/interfaces.html [ Failure ] imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/contenttype_datauri_01.html [ Failure ] imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/contenttype_datauri_02.html [ Failure ] imported/w3c/web-platform-tests/dom/nodes/Node-isEqualNode.xhtml [ Failure ] imported/w3c/web-platform-tests/html/dom/interfaces.html [ Failure ] Is there a way to solve this?
Attachments
Patch (10.58 KB, patch)
2015-12-16 03:37 PST, youenn fablet 		no flags
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Ryan Haddad
Comment 1 2015-12-14 10:03:32 PST
(In reply to comment #0) > Some WPT tests have difficulties passing in Mac EWS bots. > But I cannot find flakiness in the dashboard. > imported/w3c/web-platform-tests/dom/interfaces.html [ Failure ] > > imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/ > contenttype_datauri_01.html [ Failure ] > > imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/ > contenttype_datauri_02.html [ Failure ] > imported/w3c/web-platform-tests/dom/nodes/Node-isEqualNode.xhtml [ Failure > ] > imported/w3c/web-platform-tests/html/dom/interfaces.html [ Failure ] > > Is there a way to solve this? We noticed this on Friday, but don't have a solution yet. We are going to try to reproduce the failure locally to figure out what is causing the bots to get into this state.
Alexey Proskuryakov
Comment 2 2015-12-14 10:06:53 PST
My theory is that this happens when running the tests multiple times in the same build directory (slaves on the dashboard always download and unarchive a fresh build directory).
Alexey Proskuryakov
Comment 3 2015-12-14 20:25:45 PST
There is also this output in the logs, even after deleting the build directory: ////////// WPT tests are not using the same testharness.js file as other WebKit Layout tests. WebKit testharness.js might need to be updated according WPT testharness.js. //////////
youenn fablet
Comment 4 2015-12-15 02:30:52 PST
(In reply to comment #3) > There is also this output in the logs, even after deleting the build > directory: > > ////////// > WPT tests are not using the same testharness.js file as other WebKit Layout > tests. > WebKit testharness.js might need to be updated according WPT testharness.js. > ////////// It is strange that I do not see it in my Linux machine, while it should. Thanks for the information, I am fixing it in bug 152257. I do not think though that this will help Mac EWS bots.
Alexey Proskuryakov
Comment 5 2015-12-15 14:46:49 PST
So what should we be looking for? It definitely looks like a harness issue that lines are off by one.
Alexey Proskuryakov
Comment 6 2015-12-15 17:10:27 PST
Looks like this simply happens on all truly clean builds. Here is what I did, and then I could reproduce this locally: 1. Deleted the build directory. 2. Deleted all unversioned files in Tools and in LayoutTests subdirectories. 3. Deleted all files with svn:ignore in Tools and in LayoutTests subdirectories. Youenn, does this provide enough information to fix this?
youenn fablet
Comment 7 2015-12-16 03:36:39 PST
(In reply to comment #5) > So what should we be looking for? It definitely looks like a harness issue > that lines are off by one. OK, there is a difference of an empty line between WPT and WK testharness.js. Patch for bug 152297 removes that difference, but this will be difficult to deploy it on all bots smoothly. The first step might be to mark these tests as failing temporarily. Then we might need to ensure bots reload the "web-platform-modules/resources" submodule. I will do the first step shortly.
youenn fablet
Comment 8 2015-12-16 03:37:20 PST
Created attachment 267455 [details] Patch
WebKit Commit Bot
Comment 9 2015-12-16 04:46:04 PST
The commit-queue encountered the following flaky tests while processing attachment 267455 [details]: transitions/default-timing-function.html bug 138901 (author: simon.fraser@apple.com) The commit-queue is continuing to process your patch.
WebKit Commit Bot
Comment 10 2015-12-16 04:46:06 PST
The commit-queue encountered the following flaky tests while processing attachment 267455 [details]: The commit-queue is continuing to process your patch.
WebKit Commit Bot
Comment 11 2015-12-16 05:30:38 PST
Comment on attachment 267455 [details] Patch Clearing flags on attachment: 267455 Committed r194142: <http://trac.webkit.org/changeset/194142>
WebKit Commit Bot
Comment 12 2015-12-16 05:30:42 PST
All reviewed patches have been landed. Closing bug.
youenn fablet
Comment 13 2015-12-16 06:44:57 PST
Some additional tests should temporarily be made flaky: LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/interfaces.html LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/responsetype.htm LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/timeout-sync.html LayoutTests/imported/w3c/web-platform-tests/dom/interfaces.htmlLayoutTests/imported/w3c/web-platform-tests/html/dom/interfaces.html
Note You need to log in before you can comment on or make changes to this bug.