Security

(public)

User Story

User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3) Gecko/2008020511 Firefox/3.0b3
3b3 spends a lot longer 'looking up' servers than 3b2 or Safari do. This is not a local DNS issue; it occurs even when running a caching DNS server and using only cached domains.
This makes sites that load data from a lot of different places almost unusable. 10 seconds to load www.cnn.com vs 3 in Safari.
Reproducible: Always
Steps to Reproduce:
Load a website you haven't just visited or reload one you haven't refreshed in 3 minutes or so.
Marked this as a major bug because it is a major hit to usability.

Probably related to bug 408881 (re-enabling of IPv6 for Mac).
Can you please check this, by going to about:config, and set network.dns.disableIPv6 to true (create the entry if necessary) ? You'll probably have to restart Firefox too.

(In reply to comment #1)
> Can you please check this, by going to about:config, and set
> network.dns.disableIPv6 to true (create the entry if necessary) ? You'll
> probably have to restart Firefox too.
This appears to fix the problem. I do have ipv6 turned off for my main internet interface (configure ipv6 for tcp/ip under Airport prefs is set to 'off'). However, I have additional virtual interfaces (parallels and a vpn) that utilize IPv6.

I think that you're using a DNS server which is behind the VPN interface (your company ?), and the Mac OS X tried to to AAAA lookups first, because it knows it's an IPv6 capable interface. At least, that how I think the OS works, to provide adequate backwards compatibility. Then it fell back to A-lookups to the same DNS server, or to a DNS server on your main interface (the ISP ?).
Maybe you can mark that VPN interface as IPv4 only, if you think that the DNS server on that network is having problems with IPv6.

same experience here - seen the same effect on two different networks (company and my home DSL), and disabling IPv6 resolved it.
I believe, however, that some fix should be done before release, as this was behaving without problems in FF2 in the same networks (makes sense because it probably didn't have the IPv6 DNS support) and I'd assume that it'll affect many users for whom this is beyond their capabilities. It will make the product look slow as compared to Safari on all Mac and possibly other systems.

So, a few thoughts:
- we'll definitely relnote this (need to be using ipv6 and unroutable)
- we probably want to turn the pref to be off by default
- we can do that in a stability and support release
- we should monitor to see how widespread a problem this is

The problem also manisfests with FF3rc2 on my OS X 10.5.3 G5 PPC box: setting network.dns.disableIPv6 to true does make this version actually usable. Network (home) is via cable, but is using IPv4, not using IPv6.
FF3rc2 actually appears to be a bit faster on OS X 10.4.11 with network.dns.disableIPv6 set to "false" default.

(In reply to comment #5)
> - we probably want to turn the pref to be off by default
This would be unfortunate. The pref was off by default on OS X (only)
in FF2, and the original reason for this is long resolved. FF3 will be
out for a long time and IPv6 is only going to continue to gain in
prominence during that time (as the number of sites where multilevel NAT
breaks AJAX etc when attempting to use IPv4 will only increase).
The question we should be focussing on in this case is why FF3rc3 is slower
than Safari. Safari does IPv6 lookups in the same scenarios that FF3rc3
does IPv6 lookups. Why is Safari faster? What is it doing better? Is
it ordering its DNS queries better?
What is the exact URL typed for the poor behavior?
Is Firefox scanning a (long?) DNS searchlist with AAAA before falling back to A?

I have a windows XP and I just installed ipv6. I started to experience very slow response time from website with firefox.
I did a packet capture and there are many requests for AAAA for many ocsp and CRL verification. The problem is they don't answer to AAAA yet. I don't know why the firefox features for ocsp request any times something that don't exist...
I put a packet capture in <a href='http://www.heypasteit.com/clip/0J7W'>this paste</a>.
The sequence start at 20:01:47.950204 and it takes 20 seconds to finally find out there is nothing there...
For google it's faster but, still it freezes the browser for 5 seconds. I don't think that disabling ipv6 is a good solution as I want ipv6 and I will not switched it off.
If I can help more in anyway I will very happy to help.
John@wedebugyou.com
<a href='http://www.wedebugyou.com'>www.wedebugyou.com</a>

(In reply to John from comment #12)
> I have a windows XP and I just installed ipv6. I started to experience very
> slow response time from website with firefox.
[..]
>
> The sequence start at 20:01:47.950204 and it takes 20 seconds to finally
> find out there is nothing there...
Josh, do we have bugs around which cover improvements for DNS lookups via IPv6?

(In reply to John from comment #12)
> I have a windows XP and I just installed ipv6. I started to experience very
> slow response time from website with firefox.
According to bug 388836 (is it still accurate ?), we're not using IPv6 for OCSP.

Here is how to reproduce.
You need an OSX with ipv6 enabled. Also note that it's not platform dependant but it affects Mac, Windows and Linux.
By default ocsp is enabled after an install. So verify that it's enable. Then browse https://www.google.com . It will be long. Then go to https://www.hotmail.com and you will see that it's also long.
If you check in the pastebin I posted before, http://www.heypasteit.com/clip/0J7W
you can check at frame 20:01:47.950204
It's querying ocsp for 20 seconds and the browser is frozen during that time.
If you go to about:config and change the value of network.dns.ipv4OnlyDomains and you put the values in this list http://www.wedebugyou.com/ipv4domains.txt
After that the browser is fast to browse google, hotmail, ebay, etc in https
Jean

(In reply to Josh Aas (Mozilla Corporation) from comment #14)
> (In reply to Henrik Skupin (:whimboo) from comment #13)
>
> > Josh, do we have bugs around which cover improvements for DNS lookups via
> > IPv6?
>
> Not that I know of. Maybe Steve knows.
None of my cc'd or assigned DNS bugs are related directly to IPV6. So, I did a bugzilla search :)
Bug 680932 suggests that falling back to IPv4 is not working right.
Bug 697415 is to do with IPv6 failing using Teredo tunnels.
Bug 653953 relates to IPv6 failing with SSL.
...
John, from comment 16 it seems like you only see this with OCSP - is that correct, or do you have other IPv6 connection issues? Trying to determine if it's OCSP specific or a more general issue that you just happen to observe with OCSP.
In comment 12 you say Google is faster - do you mean Chrome or a google site? Trying to determine if it's FF or something on your network. Do you see it on multiple networks? You said you see it on multiple OSes, so it's probably not your machine.
From the log you refer to in comment 16, it looks like the DNS resolver tries AAAA queries first, then fails and and eventually falls back to IPv4. So, it may be related to bug 680932.

Sorry I am not very familiar with submitting bugs.
I will try to submit more details. :)
I checked the Bug 680932 and Bug 653953 and I can't 100% confirm that what I experience are link but, I am positive that they might also be link.
So I don't think my network as connectivity or networking issue. I checked everything and with Internet explorer in my network everything is fine. (sorry to mention IE but, I tested with it to eliminate hypothesis.)
So to give more details about my network to better understand, I have a dual-stack network. I use ipv4 and ipv6 with a tunnel-broker.
I can confirm that when I browse a website with firefox that uses https then it's very slow and sometimes it feels like it just don't open.
For the rest of the example I will use https://www.hotmail.com as a testing case. I could chose also any other https website like https://www.google.com and they all suffer the same issue.
So I am not sure what do you mean with only OCSP specific website. Is an https website always using OCSP? So if yes, then yes I observed extreme slowness with all HTTPS website when I have ipv6 and ipv4 enabled.
In comment 12 I said google is faster I meant https://www.google.com but, let's continue with https://www.hotmail.com since the capture I included in comment 12 is about this https website.
Since, I don't have multiple ipv6 enabled networks it's hard for me to test on another one. I have a friend who have one too, I will check with him if he experiences the same issues. I can't guarantee that he will reply soon as he lives on another continent.
This behaviour with firefox in my network it happens with firefox 16.0.1 in Windows XP SP3 and I also tested with firefox 16.0 under Mac OS X 10.6.8
I don't have any linux with me at the moment but, if it's necessary in the next days I could install one and run the same tests.
Now, if I completely disable OCSP in the options, all the internet browsing is as fast as with ipv4. As a human I can't notice a difference.
I don't think it's good to disable OCSP. Is it a problem?
So I reenabled OCSP and I tried to understand what is wrong and how to temporarily "patch" the bug.
Now here are the very details troubleshooting and "patch" I applied.
We need to refer to http://www.heypasteit.com/clip/0J7W that I pasted in comment 12. The first packet 20:01:35.243087 is my first dns queries to go to https://www.hotmail.com
At this moment, I see the browser doing stuffs, it starts to go to hotmail and it seems to load something...
Then at 20:01:47.950204 it does a ipv6 dns queris for AAAA record of evintl-ocsp.verisign.com
Then 1 second later it retries with another dns server, then it retries again 1 second later with my first dns server, then it retries I think 7 times, then it tries a second ocsp server that, I guess, is hard-coded somewhere in mozilla.
Finally, around 20 seconds later, it figures out that ocsp is not ipv6 reachable and then try over a ipv4 dns queries with a A record.
Then https://www.hotmail.com loads.
Now why is ocsp querying so many times? I don't know.
What I did to bring back normal speed to my firefox browser in Windows and MAC, is to force firefox to query over ipv4 some non-working ipv6 domain.
- In firefox browse to the address about:config.
- Search for the key network.dns.ipv4OnlyDomains and double-click on it.
- Then you need to put some servers to force a dns query over ipv4. Put all the servers from this up to date list http://www.wedebugyou.com/ipv4domains.txt .
- Completely close all the browsers and reopen a new one.
I will update the lists if I found more ocsp non working over ipv6.
As soon as I do that, firefox browsing with ipv6 for https website is fast again.
Now it seems that Bug 680932 and Bug 653953 could potentially be link to this behaviour. Should I also post all of this troubleshooting there too?
Anyway I remain available for more questions or troubleshooting.
thanks
Jean

Wow, now I remove all the "patch" I applied and I can't reproduce the problem.
Now it seems that the OCSP provider added a better ipv6 to ipv4 name resolution.
See this port 53 tcpdump now. http://www.heypasteit.com/clip/0JBO
Check line 10:37:03.946087 and then it returns a CNAME and a A record to ocsp.verisign.net., A 199.7.57.72
I am sure it was not doing that before. I tried many times and over 2 days.
Maybe it was a temporary issue with my provider. I am sorry for this confusion. I guess you can disregard all of my comments as I can't reproduce this issue.
Sorry again.

I am also experiencing this bug.
System: Debian Wheezy 7.7 64 bit
CPU: 6 core 3.4 GHz
Memory: 12 GB
Version: 35 64 bit (from mozilla downloads)
DNS: Running a local caching DNS server.
I am also experiencing similar problems.
When running Firefox 64 bit, DNS lookups frequently take forever (5+ seconds) and so do some "connecting" stages (but sending,waiting,receiving is usually quick). I've tested this out in an empty profile as well as my main profile. I've done all the tricks I could find online, disable dns ipv6, dns prefecthing, increase DNS cache entry counts, etc. Nothing fixed the problem, except using the 32 bit version of Firefox instead. When I use the exact same version of Firefox but 32 bit instead of 64 bit, under the exact same profile, it is lightning fast. No waiting for "Looking up blah.com" or anything. So, what is different in the 64 bit version than the 32 bit version that causes extremely slow lookups?

I am experiencing this slow down too.
System: Ubuntu 14.04
CPU: AMD Phenom II x4
Memory: 16GB
Version: 37.0.1 64-Bit (Ubuntu package)
DNS: Running a local caching DNS server
Network: NFS network, with /home folders (including browser cache) hosted on fileserver (1 Gigabit Ethernet)
Vivaldi and Chrome are both snappier than Firefox when browsing to various sites.
Firefox seems to comparably take an eternity when resolving a domain name ("waiting for xyz site").
Interestingly after a while (after trying connect to 5-10 sites) it seems to get better.
Then close all firefox instances, wait a few minutes and start firefox again. And again dns lookup is slow for the first 1 to 3 sites.
Chrome and Vivaldi have no such issues.

+1 - I noticed this problem the last few weeks, found this bug report, disabled ipv6 in about:config, and now things are okay again. So at least I found the earlier workaround to work, but I thought I'd +1 the bug.

Update:
I notice some significant improvement with Firefox 39 regarding DNS lookup. The issue seems to be fixed, but slow page-loading in general is not entirely resolved yet. Some pages (usually sub-pages in my case) take a long time, with the wheel spinning counter-clock-wise (or flipping back and forth between clockwise and counter-clockwise). When a sublink is clicked before the page is entirely loaded, it appears to me Firefox is prioritizing to finish loading elements that are linked to the current page (i.e. advertisements or stats) from other, slow responding servers, instead of aborting the whole loading process and immediately start loading the sub-page. This can cause a delay of 10-30 or even more seconds as opposed to webkit or Blink based browsers which load the same subpage immediately.

Correction regarding my update from 2015-07-23:
There's no improvement unfortunately. The reason for a better performance was that my 2nd DNS server wasn't working correctly due to corrupted data. This is fixed now and firefox is again super slow as always.

the following workaround helped to improve the problem significantly:
1. open firefox
2. type in about:config in the address bar
3. right click on the properties and select New > Integer in the context menu
4. Enter network.dnsCacheExpiration as the preference name and 0 as the integer value
more details: http://ccm.net/faq/555-disabling-the-dns-cache-in-mozilla-firefox

(In reply to psnizek from comment #28)
> the following workaround helped to improve the problem significantly:
1 - The fact that _disabling_ the DNS cache helps speed things up for you is certainly an indication that the actual DNS resolves are not at all slow since you then rather introduce more resolves and yet you say that runs faster for you.
2 - In the problematic case when you're using the DNS cache with its default settings (or have you altered it in any other way?), does the problem then appear with all sites or just specific ones? Are the problematic sites resolving to IPv4 or IPv6 addresses?
is this still version 37?

Hi Daniel,
1 - There is a significant improvement, even though sometimes I experience some waiting time of 10-20 seconds (connecting to server) on an initial access to a website. However, subsequent connections now usually take anything from 1-3 seconds. This is definitely an improvement that happened after I have disabled the DNS caching in firefox. All in all I still perceive firefox to be quite a bit slower than his peers. Yes, our DNS is OK in terms of speed.
2 - I am testing following URLs: spiegel.de, zeit.de, autobild.de, automotorundsport.de, heise.de. The built-in firefox dns caching hasn't been played with at all, with the exception that I switched it off recently.
3 - version 39.0.3 on Ubuntu 14.04.2 LTS it is.
I like to mention also, that my /home folders are on an NFS share (1gbits/s ethernet) and not on a local disk. Firefox is writing/reading all data (including cache) into/from a folder within this structure, which is probably not the best idea - but chrome and all his derivatives on the opposite side handle this perfectly well). I noticed that firefox particularly doesn't like this solution as it tends to grey out every once in a while (probably depending on network traffic or busy fileserver IO stack).
I hope I was able to provide some useful information.
Regards

psnizek,
I don't doubt that you experience a performance problem, but your description really doesn't make it sound like it is due to "very slow dns lookups" and thus this is not the right bug entry for it. I would urge you find another one that more closely matches your symptoms or perhaps create a new one if there isn't one such already. Feel free to cc me on it.
Then, it would be really cool if you could run a single-shot test with an installation that isn't on the NFS share just to see if that changes anything so that we can rule out the risk that this little detail actually has anything to do with your problem?
I use Firefox on 64bit Debian (which should be quite similar to Ubuntu) all the time and I've done so for a long time through a range of Firefox versions - I've never experienced these problems you mention. I also retried both version 38 and Nightly 42 right now on several of the suggested domains you have problems with, without any noticeable degradation in speed.
Would it be possible to do a wireshark recording of what happens over the network when you visit one of these sites and get such a slow response? I assume you get these problems on ordinary HTTP sites as well, and then such a trace would be the most beneficial (to have us rule out TLS related things).
We would also greatly appreciate if you would consider doing a log with Firefox (https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging) to allow us to even closer see what timings and things that lead to this madness.

Hi Daniel,
I gave it a second thought after reading your message. Indeed you seem to be right.
I don't know if I really should create a "new bug" entry, because of I think Firefox works as designed - it scribbles everything to disk creating large numbers of disk I/O operations. So I don't view it as a bug in the classic sense. And I also agree this doesn't belong into this "Slow DNS"-related forum. I therefore apologize that I still do my final post in here.
It appears all is caused by the NFS3 share that is simply overwhelmed by the number of I/O operations coming from firefox (checked disk stack on the fs and the raid was constantly busy when using firefox as browser). Our raid has a (very!) weak random I/O performance, especially for small blocks. This would also explain, why firefox requires so much time to terminate itself when exiting the app. It's because of it is busy to finish up with queued disk I/O operations.
A second indicator is that I failed to reproduce the firefox issue on an other machine that is connected to the same network, but using its own local drive instead of the nfs share (Elementary OS Luna, Firefox 39).
I see two solutions:
1) use local hard drive as target for all firefox disk I/O operations
2) attach an SSD drive as write-back caching device to the LVG by harnessing dm-cache. This does away with the NFS bottleneck related to random I/O operations, but won't solve the issue of possibly eating a hefty portion of the ethernet bandwith.
We decided to do 2) anyway because of other reasons, but I need to investigate how to configure firefox to use a user-defined folder for all disk I/O operations.
I can't send you wireshark logs. But I did a firefox log. If you're interested I can send you the file (90MB)! Let me know where I should upload it please (ideally not to a public place).
best regards
peter

Just wanted to say I hit this bug today OVER A VPN connection (when I'm in office things work fine). The slowdown was so bad it made Firefox unusable. Disabling ipv6 fixed the issue on Firefox 49.0.2 - thanks for the help

Shame that major bugs like this one have been around for 10+ years and still not have been fixed but are claimed "resolved".
No wonder that users get annoyed with Firefox and turn away from it.
I had the same issue (5 seconds needless delay for fresh DNS lookups) with the latest FF 54.0 on Ubuntu 17.04 64-bit.
As reported by various others here and elsewhere, workarounds are setting network.dns.disableIPv6 or switching to 32-bit.
As commented for instance on https://askubuntu.com/questions/317676/firefox-slow-dns-lookup-due-to-ipv6-in-ubuntu-13-04
a proper solution would be to do IPv4 and IPv6 lookups in parallel, or even better: not to try IPv6 if it is unavailable.

I have given up writing new bug reports, providing test data, and the like, as a waste of effort.
It's a recurring pattern in my experience with Bugzilla that I come across some (major) bug that has been reported years before, discussed to some extent, wrongly closed by a person who aims at cutting down the growing and growing mess of bug reports but apparently don't have sufficient technical insight and does not try reproducing the issue. Later part of the bug reports get re-opened after repeated complaints that things still do not work, and eventually neglected because no developer cares or has time for fixing the bug and users have found workarounds or give up on Mozilla. Instead, new bug reports pop up reporting the same issues again, and a new cycle begins.
While discussion and data gathering is necessary to some extent, as well as checking for duplicate reports (which would be much fewer if bugs were solved in due time) and closing invalid ones, all this does not really solve any bug unless the respective code actually gets improved.
If I had power on this project I would block new features until all critical and major bugs have been provably fixed.

David,
I deeply appreciate your devotion to still spend time and tell us about your problems. But...
You're mixing several complaints in a comment to a very old closed bug report. While you're free to do that, it probably just adds to your frustration instead of helping us and yourself to close in to actually find a solution to the issues you have.
It is *highly* unlikely that you suffer from this same old bug if you're using a modern Firefox browser since so much of the code and architecture has changed since. By adding comments and mixing symptoms in the same closed bug report it also makes it very hard for us to work on the report.
If you have a decent way to reproduce your problems, then please tell us.
Firefox uses the underlying system functions for name resolving (and they work really fine for hundreds of millions of users) and it does try to do both IPv4 and IPv6 in parallel (Happy eyeballs).

So why do I then still get that very annoying behaviour people have been complaining about since a decade?
And why are the well-known workarounds still applicable and have the expected effect?
No matter what may have been changed internally, the bad user experience on 64-bit Linux is still the same.
Again, take a vanilla Ubuntu 17.04 64-bit installation with its current vanilla 64-bit Firefox installation,
take a default (empty) profile and you will be able to reproduce the issue easily. Did you try it out? I guess not.
When you visit an address that has not been resolved recently (say, google.pl), it takes 10 seconds to load it.
After setting network.dns.disableIPv6, when you visit some other similar address (say, google.ch), it takes 1 second.
After switching back again to the default configuration and doing a further test (say, google.es), again 10 seconds.
Don't you get this?
I also tried 32- and 64-bit Firefox on 64-bit Win7 today and was not able to reproduce the issue there.
With both 32- and 64-bit Firefox on my 64-bit Mac OS 10.8.5 system, the loading time difference between having the network.dns.disableIPv6 option set or not is only some 2 seconds (rather than 5 to 10 seconds as on Linux), but still noticeable.

It's good information about this issue with recent Firefox versions on 64-bit Linux. However I am not able to reproduce it on:
1) Ubuntu 14.04 with Firefox nightly
2) Ubuntu 16.04 with Firefox 54.0
There is a bug 1122907 exactly about this issue on 64-bit Linux, let's have more discussion there.

Hi David,
I remember that I got the bug too and couldn't find anything useful other than to disable ipv6.
If you still can reproduce the issue on linux, could you kindly run those 4 commands please?
ip -4 addr
ip -6 addr
ip -4 route show
ip -6 route show
Thanks
John

Oops, correction of the last couple of lines above:
When I disable IPv6 system-wide:
> sudo sysctl net.ipv6.conf.all.disable_ipv6=1
the delays disappear as well, and the output of the above commands with "-6" become empty (as expected).
BTW, I don't think the issue is OCSP related, since setting security.OCSP.enabled to 0 did not affect the (DNS lookup) delay.

Hi again,
it seems that you are running ip in dual stack with only partial ipv6 support.
Indeed you have an ipv6 subnet but you are missing an ipv6 default route.
If you try to ping ipv4.google.com it should work but if you try to ping with ipv6, it should fail.
Can you try these?
ping ipv4.google.com
ping6 ipv6.google.com
$ ping6 ipv6.google.com
PING ipv6.google.com(iad23s44-in-x0e.1e100.net) 56 data bytes
64 bytes from iad23s44-in-x0e.1e100.net: icmp_seq=1 ttl=56 time=23.1 ms
64 bytes from iad23s44-in-x0e.1e100.net: icmp_seq=2 ttl=56 time=20.5 ms
Let us know the output of these and I guess that from there we will get closer to this mysterious bug.
Cheers,
John