Bug 171960

Summary:

Import FileAPI WPT tests

Product:

WebKit

Reporter:

Ben Kelly <ben>

Component:

WebCore Misc.

Assignee:

Ben Kelly <ben>

Status:

RESOLVED FIXED

Severity:

Normal

CC:

buildbot, cdumez, commit-queue, rniwa, youennf

Priority:

Version:

WebKit Nightly Build

Hardware:

Unspecified

OS:

Unspecified

Bug Depends on:

Bug Blocks:

172099, 171781

Attachments:

Description	Flags
Patch	none
Archive of layout-test-results from ews100 for mac-elcapitan	none
Archive of layout-test-results from ews107 for mac-elcapitan-wk2	none
Archive of layout-test-results from ews114 for mac-elcapitan	none
Patch	none
Patch	none
Archive of layout-test-results from ews101 for mac-elcapitan	none
Archive of layout-test-results from ews104 for mac-elcapitan-wk2	none
Archive of layout-test-results from ews113 for mac-elcapitan	none
Archive of layout-test-results from ews123 for ios-simulator-wk2	none
Patch	none
Patch	none
Archive of layout-test-results from ews100 for mac-elcapitan	none

Ben Kelly

Reported 2017-05-10 18:46:21 PDT

In bug 171781 I'd like to make a change to File which is tested by the WPT FileAPI tests. These tests haven't been imported into webkit yet, though. Lets import the tests first and then I'll make the change in bug 171781 on top of them. That way it will be easier to see what changes.

Attachments
Patch (186.28 KB, patch) 2017-05-10 18:55 PDT, Ben Kelly	no flags	Details Formatted Diff Diff
Archive of layout-test-results from ews100 for mac-elcapitan (771.60 KB, application/zip) 2017-05-10 20:03 PDT, Build Bot	no flags	Details
Archive of layout-test-results from ews107 for mac-elcapitan-wk2 (936.90 KB, application/zip) 2017-05-10 20:12 PDT, Build Bot	no flags	Details
Archive of layout-test-results from ews114 for mac-elcapitan (1.53 MB, application/zip) 2017-05-10 20:14 PDT, Build Bot	no flags	Details
Patch (8.25 KB, patch) 2017-05-10 20:26 PDT, Ben Kelly	no flags	Details Formatted Diff Diff
Patch (160.99 KB, patch) 2017-05-13 18:09 PDT, Ben Kelly	no flags	Details Formatted Diff Diff
Archive of layout-test-results from ews101 for mac-elcapitan (774.12 KB, application/zip) 2017-05-13 19:18 PDT, Build Bot	no flags	Details
Archive of layout-test-results from ews104 for mac-elcapitan-wk2 (907.62 KB, application/zip) 2017-05-13 19:23 PDT, Build Bot	no flags	Details
Archive of layout-test-results from ews113 for mac-elcapitan (1.58 MB, application/zip) 2017-05-13 19:30 PDT, Build Bot	no flags	Details
Archive of layout-test-results from ews123 for ios-simulator-wk2 (772.96 KB, application/zip) 2017-05-13 19:48 PDT, Build Bot	no flags	Details
Patch (161.70 KB, patch) 2017-05-14 20:56 PDT, Ben Kelly	no flags	Details Formatted Diff Diff
Patch (161.50 KB, patch) 2017-05-16 17:28 PDT, Ben Kelly	no flags	Details Formatted Diff Diff
Archive of layout-test-results from ews100 for mac-elcapitan (1.12 MB, application/zip) 2017-05-17 02:56 PDT, Build Bot	no flags	Details
Show Obsolete (11) View All Add attachment proposed patch, testcase, etc.

Ben Kelly

Comment 1 2017-05-10 18:55:07 PDT

Created attachment 309675 [details] Patch

Build Bot

Comment 2 2017-05-10 20:03:29 PDT

Comment on attachment 309675 [details] Patch Attachment 309675 [details] did not pass mac-ews (mac): Output: http://webkit-queues.webkit.org/results/3715266 New failing tests: imported/w3c/web-platform-tests/FileAPI/url/url_createobjecturl_file_img-manual.html

Build Bot

Comment 3 2017-05-10 20:03:31 PDT

Created attachment 309679 [details] Archive of layout-test-results from ews100 for mac-elcapitan The attached test failures were seen while running run-webkit-tests on the mac-ews. Bot: ews100 Port: mac-elcapitan Platform: Mac OS X 10.11.6

Build Bot

Comment 4 2017-05-10 20:12:44 PDT

Comment on attachment 309675 [details] Patch Attachment 309675 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/3715294 New failing tests: imported/w3c/web-platform-tests/FileAPI/url/url_createobjecturl_file_img-manual.html

Build Bot

Comment 5 2017-05-10 20:12:45 PDT

Created attachment 309681 [details] Archive of layout-test-results from ews107 for mac-elcapitan-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews107 Port: mac-elcapitan-wk2 Platform: Mac OS X 10.11.6

Build Bot

Comment 6 2017-05-10 20:14:33 PDT

Comment on attachment 309675 [details] Patch Attachment 309675 [details] did not pass mac-debug-ews (mac): Output: http://webkit-queues.webkit.org/results/3715275 New failing tests: imported/w3c/web-platform-tests/FileAPI/url/url_createobjecturl_file_img-manual.html

Build Bot

Comment 7 2017-05-10 20:14:34 PDT

Created attachment 309682 [details] Archive of layout-test-results from ews114 for mac-elcapitan The attached test failures were seen while running run-webkit-tests on the mac-debug-ews. Bot: ews114 Port: mac-elcapitan Platform: Mac OS X 10.11.6

Chris Dumez

Comment 8 2017-05-10 20:15:28 PDT

Are you using the importer script in Tools/Scripts? Your patch seems to contain manual tests.

Ben Kelly

Comment 9 2017-05-10 20:21:05 PDT

(In reply to Chris Dumez from comment #8) > Are you using the importer script in Tools/Scripts? Your patch seems to > contain manual tests. Ah, no. I was manually doing it. I'll take a look at the script.

Ben Kelly

Comment 10 2017-05-10 20:26:29 PDT

Created attachment 309684 [details] Patch

Ben Kelly

Comment 11 2017-05-10 20:27:22 PDT

Comment on attachment 309684 [details] Patch Wrong bug number. This was supposed to go to bug 171781.

Chris Dumez

Comment 12 2017-05-10 20:35:31 PDT

(In reply to Ben Kelly from comment #9) > (In reply to Chris Dumez from comment #8) > > Are you using the importer script in Tools/Scripts? Your patch seems to > > contain manual tests. > > Ah, no. I was manually doing it. I'll take a look at the script. Yes, please do. The script does a lot of things for you. Personally, I do: Tools/Scripts/import-w3c-tests FileAPI -l -s /Volumes/Data Note: I have my web-platform-tests checkout under /Volumes/Data.

Ben Kelly

Comment 13 2017-05-10 20:38:34 PDT

(In reply to Chris Dumez from comment #12) > Yes, please do. The script does a lot of things for you. I guess it probably would have handled the ".worker.js" WPT magic for me. > Personally, I do: > Tools/Scripts/import-w3c-tests FileAPI -l -s /Volumes/Data > > Note: I have my web-platform-tests checkout under /Volumes/Data. Perfect. I'll try this tomorrow night. Thanks!

Chris Dumez

Comment 14 2017-05-10 20:39:56 PDT

(In reply to Ben Kelly from comment #13) > (In reply to Chris Dumez from comment #12) > > Yes, please do. The script does a lot of things for you. > > I guess it probably would have handled the ".worker.js" WPT magic for me. Youenn is more familiar but I believe it does indeed. > > > Personally, I do: > > Tools/Scripts/import-w3c-tests FileAPI -l -s /Volumes/Data > > > > Note: I have my web-platform-tests checkout under /Volumes/Data. > > Perfect. I'll try this tomorrow night. Thanks!

youenn fablet

Comment 15 2017-05-10 20:45:12 PDT

(In reply to Chris Dumez from comment #12) > (In reply to Ben Kelly from comment #9) > > (In reply to Chris Dumez from comment #8) > > > Are you using the importer script in Tools/Scripts? Your patch seems to > > > contain manual tests. > > > > Ah, no. I was manually doing it. I'll take a look at the script. > > Yes, please do. The script does a lot of things for you. > > Personally, I do: > Tools/Scripts/import-w3c-tests FileAPI -l -s /Volumes/Data > > Note: I have my web-platform-tests checkout under /Volumes/Data. I think the recommended way is to do: Tools/Scripts/import-w3c-tests web-platform-tests/FileAPI Now that csswg-test is a thing of the past, we should update the script so that you would only need to type: Tools/Scripts/import-w3c-tests FileAPI If you really want the latests tests, you can use the '-t' option. It will take some time the first time at it will clone the WPT repo in WebKitBuild repository so be patient. The idea is that this programmatically-controlled clone will also be used to export changes from WebKit. The exporter will create the necessary commits/branches/remotes.

youenn fablet

Comment 16 2017-05-10 20:47:13 PDT

(In reply to Ben Kelly from comment #13) > (In reply to Chris Dumez from comment #12) > > Yes, please do. The script does a lot of things for you. > > I guess it probably would have handled the ".worker.js" WPT magic for me. Yes, WebKit test infra requires a .html file to run the test so the importer adds a dummy file, which is never actually served through WPT. > > Personally, I do: > > Tools/Scripts/import-w3c-tests FileAPI -l -s /Volumes/Data > > > > Note: I have my web-platform-tests checkout under /Volumes/Data. > > Perfect. I'll try this tomorrow night. Thanks! Again, Tools/Scripts/import-w3c-tests web-platform-tests/FileAPI !

Ben Kelly

Comment 17 2017-05-13 18:09:31 PDT

Created attachment 310043 [details] Patch

Ben Kelly

Comment 18 2017-05-13 18:12:35 PDT

Note, when I run this patch locally I get this failure: Regressions: Unexpected image-only failures (1) imported/w3c/web-platform-tests/FileAPI/url/url_xmlhttprequest_img.html [ ImageOnlyFailure ] I'm not sure the correct way to handle that here. If the test is flakey, should I disable it in some way?

Chris Dumez

Comment 19 2017-05-13 18:45:37 PDT

(In reply to Ben Kelly from comment #18) > Note, when I run this patch locally I get this failure: > > Regressions: Unexpected image-only failures (1) > imported/w3c/web-platform-tests/FileAPI/url/url_xmlhttprequest_img.html [ > ImageOnlyFailure ] > > I'm not sure the correct way to handle that here. If the test is flakey, > should I disable it in some way? Yes, you can disable it by adding this line: imported/w3c/web-platform-tests/FileAPI/url/url_xmlhttprequest_img.html [ ImageOnlyFailure ] to LayoutTests/TestExpectations Ideally, you'd file a bug for this failure and prepend the bug URL to the line like so: webkit.org/b/12345 imported/w3c/web-platform-tests/FileAPI/url/url_xmlhttprequest_img.html [ ImageOnlyFailure ]

Ben Kelly

Comment 20 2017-05-13 18:48:13 PDT

(In reply to Chris Dumez from comment #19) > Yes, you can disable it by adding this line: > imported/w3c/web-platform-tests/FileAPI/url/url_xmlhttprequest_img.html [ > ImageOnlyFailure ] > > to LayoutTests/TestExpectations Well, the weird thing is I auto-generated the expectations by running the tests after the import. Did the test pass that time, but fail after? Or does that auto-generated expectation feature not work for image failures? I guess I worry about adding a test that sometimes passes and sometimes fails.

Chris Dumez

Comment 21 2017-05-13 18:56:23 PDT

(In reply to Ben Kelly from comment #20) > (In reply to Chris Dumez from comment #19) > > Yes, you can disable it by adding this line: > > imported/w3c/web-platform-tests/FileAPI/url/url_xmlhttprequest_img.html [ > > ImageOnlyFailure ] > > > > to LayoutTests/TestExpectations > > Well, the weird thing is I auto-generated the expectations by running the > tests after the import. Did the test pass that time, but fail after? Or > does that auto-generated expectation feature not work for image failures? > > I guess I worry about adding a test that sometimes passes and sometimes > fails. Looks like a ref-test so the expected result is an HTML file which comes from web-platform-tests and is not updated/generated by our tools (unlike tests with text expectations).

Ben Kelly

Comment 22 2017-05-13 19:04:06 PDT

(In reply to Chris Dumez from comment #21) > Looks like a ref-test so the expected result is an HTML file which comes > from web-platform-tests and is not updated/generated by our tools (unlike > tests with text expectations). Perfect. Thanks for clearing that up. I'll let EWS finish since I already started it in case there are other problems I need to deal with and then upload a patch with a fixed expecation tomorrow.

Build Bot

Comment 23 2017-05-13 19:18:50 PDT

Comment on attachment 310043 [details] Patch Attachment 310043 [details] did not pass mac-ews (mac): Output: http://webkit-queues.webkit.org/results/3735835 New failing tests: imported/w3c/web-platform-tests/FileAPI/url/url_xmlhttprequest_img.html

Build Bot

Comment 24 2017-05-13 19:18:51 PDT

Created attachment 310050 [details] Archive of layout-test-results from ews101 for mac-elcapitan The attached test failures were seen while running run-webkit-tests on the mac-ews. Bot: ews101 Port: mac-elcapitan Platform: Mac OS X 10.11.6

Build Bot

Comment 25 2017-05-13 19:23:50 PDT

Comment on attachment 310043 [details] Patch Attachment 310043 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.webkit.org/results/3735843 New failing tests: imported/w3c/web-platform-tests/FileAPI/url/url_xmlhttprequest_img.html

Build Bot

Comment 26 2017-05-13 19:23:51 PDT

Created attachment 310052 [details] Archive of layout-test-results from ews104 for mac-elcapitan-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: ews104 Port: mac-elcapitan-wk2 Platform: Mac OS X 10.11.6

Build Bot

Comment 27 2017-05-13 19:30:57 PDT

Comment on attachment 310043 [details] Patch Attachment 310043 [details] did not pass mac-debug-ews (mac): Output: http://webkit-queues.webkit.org/results/3735838 New failing tests: imported/w3c/web-platform-tests/FileAPI/url/url_xmlhttprequest_img.html

Build Bot

Comment 28 2017-05-13 19:30:58 PDT

Created attachment 310053 [details] Archive of layout-test-results from ews113 for mac-elcapitan The attached test failures were seen while running run-webkit-tests on the mac-debug-ews. Bot: ews113 Port: mac-elcapitan Platform: Mac OS X 10.11.6

Build Bot

Comment 29 2017-05-13 19:48:03 PDT

Comment on attachment 310043 [details] Patch Attachment 310043 [details] did not pass ios-sim-ews (ios-simulator-wk2): Output: http://webkit-queues.webkit.org/results/3735874 New failing tests: imported/w3c/web-platform-tests/FileAPI/url/url_xmlhttprequest_img.html

Build Bot

Comment 30 2017-05-13 19:48:05 PDT

Created attachment 310055 [details] Archive of layout-test-results from ews123 for ios-simulator-wk2 The attached test failures were seen while running run-webkit-tests on the ios-sim-ews. Bot: ews123 Port: ios-simulator-wk2 Platform: Mac OS X 10.11.6

Ben Kelly

Comment 31 2017-05-14 20:56:21 PDT

Created attachment 310105 [details] Patch

Ben Kelly

Comment 32 2017-05-14 20:57:13 PDT

Updated the patch based on comment 19. Unfortunately I couldn't test locally as my tree seems busted after updating this evening. Let's see what EWS says.

youenn fablet

Comment 33 2017-05-14 21:58:27 PDT

Comment on attachment 310105 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=310105&action=review > LayoutTests/tests-options.json:12 > + "slow" This change is probably not needed anymore.

Ben Kelly

Comment 34 2017-05-15 06:12:14 PDT

(In reply to youenn fablet from comment #33) > > LayoutTests/tests-options.json:12 > > + "slow" > > This change is probably not needed anymore. Hmm, can you explain why? AFAICT this was put there by the import script.

youenn fablet

Comment 35 2017-05-15 09:07:41 PDT

(In reply to Ben Kelly from comment #34) > (In reply to youenn fablet from comment #33) > > > LayoutTests/tests-options.json:12 > > > + "slow" > > > > This change is probably not needed anymore. > > Hmm, can you explain why? AFAICT this was put there by the import script. This test is a manual test. Test importer by default does not import such test. In your patch, it does not seem to be added anymore. Was this change done with your last script importer run? If so, there might be a bug in the importer.

Chris Dumez

Comment 36 2017-05-15 09:29:35 PDT

The patch looks good besides Youenn's comment that should be addressed. r=me with Youenn's fix included.

Ben Kelly

Comment 37 2017-05-15 10:02:26 PDT

(In reply to youenn fablet from comment #35) > This test is a manual test. Test importer by default does not import such > test. > In your patch, it does not seem to be added anymore. > Was this change done with your last script importer run? > If so, there might be a bug in the importer. No, I started from scratch with the importer script. So I'm pretty sure the importer script added this line.

youenn fablet

Comment 38 2017-05-15 10:05:31 PDT

(In reply to Ben Kelly from comment #37) > (In reply to youenn fablet from comment #35) > > This test is a manual test. Test importer by default does not import such > > test. > > In your patch, it does not seem to be added anymore. > > Was this change done with your last script importer run? > > If so, there might be a bug in the importer. > > No, I started from scratch with the importer script. So I'm pretty sure the > importer script added this line. Looking at the importer, it indeed sounds like a bug in the way we compute _slow_tests. I'll fix it.

youenn fablet

Comment 39 2017-05-15 22:39:14 PDT

(In reply to youenn fablet from comment #38) > (In reply to Ben Kelly from comment #37) > > (In reply to youenn fablet from comment #35) > > > This test is a manual test. Test importer by default does not import such > > > test. > > > In your patch, it does not seem to be added anymore. > > > Was this change done with your last script importer run? > > > If so, there might be a bug in the importer. > > > > No, I started from scratch with the importer script. So I'm pretty sure the > > importer script added this line. > > Looking at the importer, it indeed sounds like a bug in the way we compute > _slow_tests. > I'll fix it. Fixed as per http://trac.webkit.org/changeset/216900.

Ben Kelly

Comment 40 2017-05-16 07:09:19 PDT

(In reply to youenn fablet from comment #39) > Fixed as per http://trac.webkit.org/changeset/216900. Thanks! Sorry I didn't get to fix this patch last night. I'll fix it up and flag for review tonight.

Ben Kelly

Comment 41 2017-05-16 17:28:11 PDT

Created attachment 310328 [details] Patch

Build Bot

Comment 42 2017-05-17 02:56:11 PDT

Comment on attachment 310328 [details] Patch Attachment 310328 [details] did not pass mac-ews (mac): Output: http://webkit-queues.webkit.org/results/3760445 New failing tests: http/tests/media/hls/video-duration-accessibility.html http/tests/media/hls/video-controls-live-stream.html

Build Bot

Comment 43 2017-05-17 02:56:12 PDT

Created attachment 310367 [details] Archive of layout-test-results from ews100 for mac-elcapitan The attached test failures were seen while running run-webkit-tests on the mac-ews. Bot: ews100 Port: mac-elcapitan Platform: Mac OS X 10.11.6

Ben Kelly

Comment 44 2017-05-17 06:12:40 PDT

(In reply to Build Bot from comment #42) > New failing tests: > http/tests/media/hls/video-duration-accessibility.html > http/tests/media/hls/video-controls-live-stream.html Hmm... Not sure how adding FileAPI WPT tests could cause these media tests to fail.

youenn fablet

Comment 45 2017-05-17 08:33:09 PDT

(In reply to Ben Kelly from comment #44) > (In reply to Build Bot from comment #42) > > New failing tests: > > http/tests/media/hls/video-duration-accessibility.html > > http/tests/media/hls/video-controls-live-stream.html > > Hmm... Not sure how adding FileAPI WPT tests could cause these media tests > to fail. Bots sometimes have such flaky regressions.

WebKit Commit Bot

Comment 46 2017-05-17 09:04:39 PDT

Comment on attachment 310328 [details] Patch Clearing flags on attachment: 310328 Committed r216975: <http://trac.webkit.org/changeset/216975>

WebKit Commit Bot

Comment 47 2017-05-17 09:04:41 PDT

All reviewed patches have been landed. Closing bug.

Note You need to log in before you can comment on or make changes to this bug.