Bug 37104

Two more failures likely caused by the same root bug: http/tests/xmlhttprequest/logout.html https://bugs.webkit.org/show_bug.cgi?id=33301 http/tests/xmlhttprequest/frame-load-cancelled-abort.html https://bugs.webkit.org/show_bug.cgi?id=33462

This sounds like something that needs to be reproducible locally to investigate.

Looks like Exception 101 is just XMLHttpRequestException::NETWORK_ERR.

In bug 37235 I've fixed it so that we can actually run: run-webkit-tests --iterations 100 http/tests/xmlhttprequest without expected failures.

Running: run-webkit-tests --iterations 1000 http/tests/xmlhttprequest/cross-origin-authorization.html on SL seems to hang every 10 tests or so for a couple seconds. Strange.

http/tests/xmlhttprequest/cross-origin-no-authorization.html exhibits the same strange "pause every 10 or so" behavior. Watching the apache access_log, the hang point seems random. The last request entry in the log before the hang is not consistent. Watching "top" I only see DumpRenderTree ever taking time, never httpd.

Created attachment 52808 [details] Reduced http/tests/xmlhttprequest/cross-origin-authorization.html which shows hang when run repeatedly I used: yes LayoutTests/http/tests/xmlhttprequest/cross-origin-authorization.html | head -n 1000 | xargs WebKitBuild/Debug/DumpRenderTree | grep Time to run this test repeatedly, and noticed that every 20 or so runs it would hang for 2 seconds. I'm still investigating to find out what part is hanging.

I'm investigating the hanging because I suspect it's related to the XHR exceptions that this bug is actually about.

(In reply to comment #7) > I used: > yes LayoutTests/http/tests/xmlhttprequest/cross-origin-authorization.html | > head -n 1000 | xargs WebKitBuild/Debug/DumpRenderTree | grep Time > to run this test repeatedly, and noticed that every 20 or so runs it would hang > for 2 seconds. This original testing was on a Debug build on a 4-core Mac Pro running Snow Leopard. I was also able to reproduce similar symptoms on a Release build on a 2-core Mac Book Pro running Leopard.

I updated the test to call GCController.collect() at the end of each test (after printing the time taken), and that did not make the hang go away. Hopefully that rules out the idea of GC causing this hang.

Adam and I have now reduced the test case to: <pre id="console"></pre> <script> var startTime = new Date; layoutTestController.dumpAsText(); function log(message) { document.getElementById('console').appendChild(document.createTextNode(message + '\n')); } var req = new XMLHttpRequest; req.open("GET", "http://localhost:8000/xxxxx", false); req.send(); log("Time: " + (new Date - startTime)); </script> See the instructions for running it under DRT above. If you change this to use same-origin (127.0.0.1), we don't hang. But the test as is will hang for about 1 second every 50 requests or so.

Created attachment 52811 [details] An even better reduction The 124th XHR seems to most commonly fail!

Firefox reproduced the failure as well. Suggesting I may be confused. Chrome throws an error: (111) Connection refused

while curl "http://localhost:8000/xxxxxx" -s -w "Time: %{time_total}\n" | grep Time ; do; done; reproduces the hang. Suggesting that this is not WebKit, but rather some lower-level system service (like lookupd?) or apache itself.

More data: while [ true ]; do; curl "http://localhost:8000/xxxxxx" -s -w "Time: %{time_total}\n" | grep Time | grep -v "0.0"; done; reproduces. However, using "127.0.0.1" instead of "localhost": while [ true ]; do; curl "http://127.0.0.1:8000/xxxxxx" -s -w "Time: %{time_total}\n" | grep Time | grep -v "0.0"; done; does not reproduce. This still faults Apache or DNS resolution (or reverse lookup) on the machine (which used to be done by lookupd, but I don't know how things are done in Leopard/SnowLeopard).

Reduced test case: run this command in a loop (with no server listening on port 9999): echo | nc 127.0.0.1 9999 Sometimes it will hang for 1+ seconds. Other examples: dig localhost <-- no hang ping -c 1 127.0.0.1 <-- does hang (often for 11 seconds) curl -s http://localhost:9999/xxx <-- does hang If you try talking to a remote server (like webblaze.org), you don't get any hangs. My read is that this is a bug in Mac OS X listening sockets.

Are you always testing on 10.5.8?

The issue (thanks to Rob Braun and Maciej) is that our Apache is only listening on IPv4 loopback (127.0.0.1) and not IPv6 loopback (::1). When CFNetwork (or any other network layer) resolves "localhost" to an address it gets back both 127.0.0.1 and ::1 in random order. When the connection fails, it resolves again, and tries again. If the network layer tries again, it may or may not get the IPv4 address first on the second resolve. If we're failing often enough on enough requests, the kernel will start to throttle back connection reset responses (which was what was causing the 1s or 2s pauses in several of our reduced test cases). We suspect this can lead to both the random "network error" exceptions as well as the timeouts. The fix is to make our various apache configs listen on both the IPv4 loopback addresses as well as the IPv6 addresses. Unfortunately during this process I also discovered that we have some (half-broken) support for changing HTTP port numbers in our run-webkit-tests and run-webkit-httpd scripts. I did not make any attempt to fix those, I just added a FIXME since the fix for this bug makes that half-broken support more broken.

Created attachment 52824 [details] Patch

Comment on attachment 52824 [details] Patch I bet websockets have the same bug.

Comment on attachment 52824 [details] Patch Clearing flags on attachment: 52824 Committed r57249: <http://trac.webkit.org/changeset/57249>

All reviewed patches have been landed. Closing bug.

This is great news! 1. What will this do to Apache 1 on Tiger? Or to Apache on machines without IPv6 support? 2. Does this work with "run-webkit-httpd -a"? 3. Does run-webkit-websocketserver need to be fixed? 4. Did you verify that this is the actual cause of problems by disabling IPv6 on your machine?

Guys, you are too fast...

> Guys, you are too fast... Sorry. We've been working on this bug for hours and talked about the fix over dinner. > 1. What will this do to Apache 1 on Tiger? Or to Apache on machines without > IPv6 support? I think apache will fail to bind to the interface but otherwise be ok. We'll find out for sure by what the bot does. > 2. Does this work with "run-webkit-httpd -a"? I don't know what that option does, but we can look into it. > 3. Does run-webkit-websocketserver need to be fixed? Yes. > 4. Did you verify that this is the actual cause of problems by disabling IPv6 > on your machine? We ran lots of experiments, but we didn't run that particular experiment. The configuration is definitely wrong. We're unsure of how much flakiness this will fix (hopefully a lot). We'll find out.

> > 2. Does this work with "run-webkit-httpd -a"? > > I don't know what that option does, but we can look into it. That's part of the runtime configuration of Apache listening. Eric says that all broken.

> 2. Does this work with "run-webkit-httpd -a"? The short answer is yes. These directives are additive.

> Sorry. Fast isn't bad :) > > 2. Does this work with "run-webkit-httpd -a"? > I don't know what that option does, but we can look into it. It binds to all interfaces - useful when you want to run tests over network from another machine for some reason. > The configuration is definitely wrong. Agreed.

> Unfortunately during this process I also discovered that we have some > (half-broken) support for changing HTTP port numbers It's completely broken, because tests rely on hardcoded port numbers (I initially hoped that test authors would be careful to not depend on those). This support should be just removed.

Tiger needs more fixeration. Reopening.

Created attachment 52826 [details] Patch

http://trac.webkit.org/changeset/57249 might have broken Tiger Intel Release

Committed r57250: <http://trac.webkit.org/changeset/57250>

Created attachment 52842 [details] Patch

Committed r57260: <http://trac.webkit.org/changeset/57260>