Closed Bug 408889 Opened 17 years ago Closed 17 years ago

figure out what the heck is wrong with the crash report submit URL on Vista

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
x86
Windows Vista
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: ted, Assigned: justin)

References

(
URL
)

Details

We can't submit crash reports on Vista currently because the certificate revocation check fails.  This was a problem earlier, and it was fixed by changing the reverse DNS for the crash-reports.mozilla.com host.  It stopped working again ~Dec 7th, and it's definitely a server side problem, as builds before that date display the problem.  You can test the behavior by visiting https://crash-reports.mozilla.com/submit/ in IE7 on Vista, or on XP by changing the pref "Check for server certificate revocation" (in Internet Options -> Advanced -> Security) and then visiting that URL.  If the problem exists you will see a dialog that says "Revocation information for the security certificate for this site is
not available.  Do you want to proceed?" (at least on XP/IE6).

Filing this separately and blocking bug 390568 so 390568 can stay blocking-1.9+.
Severity: normal → major
Can you get to https://addons.mozilla.org or https://www.mozilla.com/ ?  I can't duplicate this with my XP VM.
I just tried this under vista with IE7 and it wfm - is there something special I need to do?  It seems to just redirect to crash-stats.mozilla.com...
mrz: with the pref set, in XP/IE6, I get the same dialog at both of those URLs.  Clicking "yes" once will get rid of the dialog until I restart.

Justin:  Now that I try it, it appears to work fine in IE7.  I think Microsoft may have worked around this in IE7, so only IE6 or apps using WinINET display the problem.

I downloaded and compiled a WinINET sample from MSDN:
http://msdn.microsoft.com/archive/default.asp?url=/archive/en-us/samples/internet/networking/asyncdemo/default.asp
http://people.mozilla.com/~tmielczarek/AsyncDemo.exe

It's sort of a crappy GUI, but you can put in two URLs and hit "download resources".  If I put https://www.mozilla.org/ in the first URL box, and https://www.microsoft.com/ in the second, and hit download, the microsoft one gives me content, and the mozilla one gives an error.  If you scroll down the "callbacks" list, you'll see that it lists error 12057.  (The message looks something like: Closed: REQUEST_COMPLETE (8) Error (12057) encountered)

That's a non-helpful error. The only real difference I can see is that Mozilla's certificates are chained from XRamp.  

The GeoTrust certificate that's right now running in China isn't chained and works.  If you add:

59.151.50.30 addons.mozilla.org

to your hosts file and try that one, you'll get content back.  It too is a wild card certificate.

Either IE7 doesn't like XRamp's CRL or doesn't like XRamp's chain.  I'm not sure how to tell which nor am I sure how to fix it other than getting a site specific certificate for crash-reports.mozilla.com. 

Anyone?
btw, if I use any of the XRamp certs, I get that error - none of the GeoTrust certs fail.
A possibly helpful tidbit: any HTTPS url ending with "mozilla.com." or "mozilla.org." (notice the trailing dot, that's the closing period of a Fully-Qualified Domain Name) fails in both browsers. The error in that case seems to be that the URLs "*.mozilla.org." and "*.mozilla.com." are missing from the certificates (only the dotless variants are there). Maybe the revocation information of the Mozilla domains is (accidentally or intentionally) at a domain that has the FQDN closing dot. (A quick test is visiting the url https://bugzilla.mozilla.org./show_bug.cgi?id=408889 in any browser that supports SSL/TLS. It is supposed to show this bug, but instead it will throw an error about an invalid certificate.)
I have never seen a url or hostname with a trailing period after the host name.  Should these even work?
Yes, they should work. In nearly all cases, it's safe to omit the trailing period. However, there are cases when omission might cause ambiguity. Consider "www.msn.com", for example. The actual domain name in that case is something along the lines of "msn.com.ms.akadns.net.", which has "msn.com" set as a shorthand. Microsoft's DNS servers make sure that "www.msn.com" ends up the same as just "msn.com". However, there is also a separate domain name, "msn.com.", which is an FQDN, and not a shorthand. It redirects to "www.msn.com", but itself has a "www" subdomain (full form: "www.msn.com."), which points to a server once used for testing new MSN services. (The trailing dot ensures that the domain doesn't get expanded to "www.msn.com.ms.akadns.net.".) When looking up "www.msn.com", an incomplete domain name, Microsoft's DNS servers are configured to redirect to msn.com.ms.akadns.net. (or something like that), but typing "www.msn.com.", a complete, fully-qualified domain name, will trigger no completion. Another example is the domain "http://museum.", which is the canonical form of www.museum. Because the "www" subdomain doesn't have its own DNS record assigned, it gets the record of the TLD, that is, "museum.". Also, try typing "com." or "net." into Firefox or any other browser. It should bring up "www.com" or "www.net". (Similarly, the easiest way to access the Dot TK site is "tk.", which is the canonical form of "www.tk", a redirect to "dot.tk".) Note however, that "org." doesn't work, because W3C actually has the "www" subdomain of the "org" TLD registered, as opposed to the TLD itself. So, the "www" version is not a fallback address. (Note however, that many programs got screwed up when they get such an address. This is due to an incorrect implementation of the DNS specification.)
(In reply to comment #7)
> A possibly helpful tidbit: any HTTPS url ending with "mozilla.com." or
> "mozilla.org." (notice the trailing dot, that's the closing period of a
> Fully-Qualified Domain Name) fails in both browsers. 

Do you have an example of a URL ending in either that does work?  In my testing yesterday it was only the xramp wildcard cert sites - the geotrust ones work just fine (www.mozilla.org is geotrust, www.mozilla.com is xramp).
Talked to Ted - this is part of the crash reporting tool in Fx build off of WinINET.  It isn't really clear where the error and I can't find a way to view XRamp's CRL.  This error:

"Revocation information for the security certificate for this site is not available.  Do you want to proceed?"

Isn't helpful since as best as I can tell, the CRL that the cert mentions is available.  Error 12057 is even less meaningful.

I want to do a little more debugging but I suspect a single host cert might just be easiest.
Assignee: server-ops → mrz
Just FYI, error 12057 is ERROR_INTERNET_SEC_CERT_REV_FAILED--"Security certificate
revocation failed".


Does the CRL contain both crash-reports.mozilla.com AND dyna-crashreports.nslb.sj.mozilla.com (ad possibly the IP 63.245.209.57)? Maybe IE has a weird habit to check CRLs against canonical URLs. (Maybe the FQDN versions are also needed, that is, "crash-reports.mozilla.com." and "dyna-crashreports.nslb.sj.mozilla.com.")
It should contain neither - the cert is issues for *.mozilla.org and CRLs list certificate serial numbers that should be considered revoked.  

But I have no idea how to view the contents of the CRL - if you do, you're welcome to look at it and let me know.  The CRL URI is in the cert details.
mail.mozilla.com uses a host specific Xramp chained SSL certificate and fails with 12057.  mail.mozilla.com resolves to 10.2.72.15 and 10.2.72.15 resolves to mail.mozilla.com. 

build.mozilla.org uses a host specific GeoTrust non-chained SSL certificate and works (fails on web auth but otherwise works).  Internally for me build.mozilla.org is an A RR to 10.2.74.128 which is really dm-wwwbuild.mozilla.org so in this case, DNS is not at issue.
Attaching CSR - 

-----BEGIN CERTIFICATE REQUEST-----
MIICNjCCAZ8CAQAwgcMxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
MRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRwwGgYDVQQKExNNb3ppbGxhIENvcnBv
cmF0aW9uMR4wHAYDVQQLExVNb3ppbGxhIENyYXNoIFJlcG9ydHMxIjAgBgNVBAMT
GWNyYXNoLXJlcG9ydHMubW96aWxsYS5jb20xJTAjBgkqhkiG9w0BCQEWFmhvc3Rt
YXN0ZXJAbW96aWxsYS5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMrr
jxQWtgh6xJkFb6DebjldmLr0IU3SUymHaMcos6ISJ4w8IkkGGJS+59sMpLGR6bMZ
mioH4dpSZqwQqCYPpMQxi8XjdkeIzxP8Q2+01lYYK/fTVqp2jh3TWwOk1gbiNBzY
YYxxkhXOR5yjj6pfSHdWBJZZJRajYo/xddzmq5p9AgMBAAGgMjAwBgkqhkiG9w0B
CQ4xIzAhMBEGCWCGSAGG+EIBAQQEAwIGQDAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3
DQEBBAUAA4GBAJkSxFxy1EjWNnDlmmoD7rktEq7hy6IDmpKsWcVonWJgm2NneFi+
8X59DjzBQaufH8HssuJIW8r3lRuG3ITPz4h/jQwwC0Okas1499vGhhqgM6upvj9V
GuzhF0uDZsilDcXFJ/ZasjGvszGo6HLwxPN5fdv69CkYqMx2yJODi0Q1
-----END CERTIFICATE REQUEST-----
Assignee: mrz → justin
here's what I get when I load https://bugzilla.mozilla.org. in Firefox:

Secure Connection Failed

      

      
      
      

      
        
        

          

bugzilla.mozilla.org. uses an invalid security certificate.

The certificate is only valid for *.mozilla.org.

(Error code: ssl_error_bad_cert_domain)

        


        
        


    *   This could be a problem with the server's configuration, or it could be
      someone trying to impersonate the server.

    *   If you have connected to this server successfully in the past, the error may
      be temporary, and you can try again later.
That's due to an error in the certificate, which is missing the url "*.mozilla.org.". The message looks quite confusing, because a period is visible after "org". However. that period is in fact a full-stop at the end of a sentence, that is, if the included URL was really "*.mozilla.org.", then the page would show "*.mozilla.org..". (I was also quite puzzled when I saw that the certificate is not for mozilla.org., but instead, it's for mozilla.org. The period is just plain confusing there.)
I don't think that has anything to do with the problem at hand, but feel free to demonstrate otherwise.  If you can't though, please don't clutter up this bug with irrelevant information.

Comment #17 and #18 make sense - the Common Name for the certificate is "*.mozilla.org" without any trailing period.  You'll see the same certificate warnings if you goto https://mail.google.com./ 

These errors, however, only appear for me in Fx3/Minefield - I don't see these errors in Safari.  

In any event, these aren't related to this bug - maybe a new bug ?
mrz: any update on this?  I think we should just do whatever we have to do (new cert or whatever) to make this work, even if we're working around some broken behavior in Windows.

Ted - sorry for the delay - I had attached a CSR before the holidays.  I'll make sure that gets put through this week.
cert is on order.
Do we have an ETA on the cert?
I've been after geotrust daily for a status, but it seems out sales person has left.  I'll do what I can to push this along...
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
verified fixed , i can send now crashreports via vista without problems. Thanks justin and ted.
Status: RESOLVED → VERIFIED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.