Bug 1952697 - Add a timer for native DNS resolutions with DNSServiceQueryRecord on OSX r=necko-reviewers,kershaw

The callback passed to DNSServiceQueryRecord never gets called
if a domain does not exist, no error gets thrown, and select
on the resulting FD never returns.
This patch adds a 5 second timeout, but it's not a comprehensive fix.
Opening a page that has lots of non-existant domains would still end
up delaying page load in other tabs by 5 seconds.

Differential Revision: https://phabricator.services.mozilla.com/D249450

modules/libpref/init/StaticPrefList.yaml

@@ -13973,6 +13973,13 @@
   value: 20000
   mirror: always
 
+# When resolving a native HTTPS query with native APIs
+# the MacOS implementation has a max timeout
+- name: network.dns.native_https_timeout_mac
+  type: RelaxedAtomicInt32
+  value: 5000
+  mirror: always
+
 # When this pref is true, we copy the host name to a fresh string before
 # calling into getaddrinfo.
 - name: network.dns.copy_string_before_call

netwerk/dns/PlatformDNSMac.cpp

@@ -136,12 +136,21 @@ nsresult ResolveHTTPSRecordImpl(const nsACString& aHost,
   FD_ZERO(&readfds);
   FD_SET(fd, &readfds);
 
-  int result = select(fd + 1, &readfds, NULL, NULL, NULL);
+  // If the domain queried results in NXDOMAIN, then QueryCallback will
+  // never get called, and select will hang forever. We need to use a
+  // timeout so that select() eventually returns.
+  struct timeval timeout;
+  timeout.tv_sec = StaticPrefs::network_dns_native_https_timeout_mac();
+  timeout.tv_usec = 0;
+
+  int result = select(fd + 1, &readfds, NULL, NULL, &timeout);
   if (result > 0 && FD_ISSET(fd, &readfds)) {
     // Process the result
     DNSServiceProcessResult(sdRef);
   } else if (result < 0) {
     LOG("select() failed");
+  } else if (result == 0) {
+    LOG("select timed out");
   }
 
   // Cleanup