Closed Bug 390568 Opened 17 years ago Closed 17 years ago

HttpSendRequest returning ERROR_INTERNET_SEC_CERT_REV_FAILED (crash reporter fails to submit when IE7 is installed) (also Vista)

Categories

(Toolkit :: Crash Reporting, defect, P1)

Product:
Toolkit
Component:
Crash Reporting
Platform:
x86
Windows Vista
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla1.9beta3

People

(Reporter: vlad, Assigned: mrz)

References

Details

(Keywords: relnote) 

On my Vista machine, I have yet to be able to submit a crash report.  HttpSendRequest is returning FALSE, and GetLastError is returning ERROR_INTERNET_SEC_CERT_REV_FAILED (12057).  This means "Security certificate revocation failed".

I found some stuff at http://www.archivum.info/microsoft.public.inetsdk.programming.wininet/2005-07/msg00096.html -- and I tried using HttpSendRequestEx and setting that flag, but that didn't help (HttpSendRequestEx was still returning FALSE, and GetLastError was 12057).  Not sure what's going on.

Vista Ultimate, UAC is disabled.  I can visit the crash reporter submission URL fine from IE (and get a "GET not allowed" page).
Not a fix but I was able to submit crash reports by unchecking "Check for server certificate revocation" in IE on Vista
This also happens on on Win XP with IE6 or IE7. i.e. Enabling "Check for
server certificate revocation" in IE causes the submission to fail right after communicating with XRamp. Not as obvious with IE6 as it's not on by default. I guess it was enabled by default with IE7/Vista. 

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a8pre) Gecko/2007081818 Minefield/3.0a8pre ID:2007081818
Summary: HttpSendRequest returning ERROR_INTERNET_SEC_CERT_REV_FAILED → HttpSendRequest returning ERROR_INTERNET_SEC_CERT_REV_FAILED (crash reporter fails to submit on Vista)
Brian, dunno if you have tested the Breakpad Win32 crash report sender code on Vista, but consensus here is that it doesn't work.
Flags: blocking1.9?
I get the feeling bryner doesn't read his bugmail.  Mark, I don't know if you're aware of this, but the win32 sender code appears to be broken by default on Vista.

Some data points:
1) Enabling this option in WinXP/IE6, then visiting https://bugzilla.mozilla.org (any ssl mozilla.org site will work) gives me a security alert dialog: "Revocation information for the security certificate for this site is not available.  Do you want to proceed?"
2) Accessing the same site on Vista/IE7 does not give the error dialog.

What server is breakpad trying to connect to?  Is there anything in the chain that lists a CRL that's not reachable?
The submit URL is: https://crash-reports.mozilla.com/submit

Viewing the cert chain, I can load all of the CRL URLs that I see.

Yup, and OpenSSL is able to parse all of those CRLs, too.
Assignee: nobody → ted.mielczarek
Flags: blocking1.9? → blocking1.9+
I'm sort of stuck on this, I could use help if anyone has the time.

(In reply to comment #6)
> Some data points:
> 1) Enabling this option in WinXP/IE6, then visiting
> https://bugzilla.mozilla.org (any ssl mozilla.org site will work) gives me a
> security alert dialog: "Revocation information for the security certificate > for this site is not available.  Do you want to proceed?"

I'm not seeing this using IE6 in WinXP SP2 on my work machine.


I don't know if this is somehow related or not, but you say that crashes get submitted to crash-reports.mozilla.com.  This resolves to the ip address of 63.245.209.57, but if I do a reverse lookup on 63.245.209.57 I get dyna-crashreports.nslb.sj.mozilla.com.  Trying to resolve dyna-crashreports.nslb.sj.mozilla.com results in an error of it not being found.

While the crash report is trying to submit, doing a netstat -a displays dyna-crashreports.nslb.sj.mozilla.com as the address that the crash reporter is connecting to.

If I got to https://crash-reports.mozilla.com in my browser, everything is fine.
If I go to https://63.245.209.57 in my browser I get a certificate mismatch error.
Going to https://dyna-crashreports.nslb.sj.mozilla.com completely fails since the domain lookup fails.
I don't think that has anything to do with it, that's just our IT infrastructure.  We're not accessing the host by its IP or the canonical name there, we're accessing it via crash-reports.mozilla.com.

dyna-crashreports.nslb.sj.mozilla.com now resolves to the same ip as crash-reports.mozilla.com.
This seems to WFM on Vista now.  Wow!  Thanks, Michael, I think that was it!

http://crash-stats.mozilla.com/report/index/401d8fe8-5fd8-11dc-ac7a-001a4bd43e5c
(not that this is a particularly useful report)

If I had to take a stab at what might have been be happening, I'd say that the revocation check used the ip address instead of (or in addition to the hostname) and then tried to correlate it back to a hostname which then failed because that hostname didn't have an ip. 

I'm not sure exactly why it would do this (maybe to prevent DNS or hostname spoofing?), but it's the only thing I can think of that would cause the revocation check to pass after making dyna-crashreports.nslb.sj.mozilla.com resolve correctly.

At least it works.
Assignee: ted.mielczarek → aravind
Resolving FIXED per testing.
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
reopening, this is happening again using today's nightly on Vista.  ted asked me to move to IT component, doing so per his request.
Status: RESOLVED → REOPENED
Component: Breakpad Integration → Server Operations
Flags: blocking1.9+
Product: Toolkit → mozilla.org
Resolution: FIXED → ---
Version: unspecified → other
Looks like the reverse DNS is still good.  Is there anything else that would have changed about this configuration to make this bug show up again?

Ted and Aravind were talking about this on IRC, but I had to signoff before I heard if there was any resolution. Seems to me this would be a blocker for Beta 2, but there is no way for me to nominate it under this product/component.
Assignee: aravind → nobody
Status: REOPENED → NEW
Component: Server Operations → Breakpad Integration
Product: mozilla.org → Toolkit
Version: other → Trunk
Nominating to get this on the radar. Currently we are unable to submit crash reports on Vista using 2007121008.
Flags: blocking1.9?
Target Milestone: --- → mozilla1.9 M10
Blocks, but not M10. Possibly IT related, but we should hammer down the solution.
Flags: blocking1.9? → blocking1.9+
Priority: -- → P1
Target Milestone: mozilla1.9 M10 → mozilla1.9 M11
Keywords: relnote
The last checkin to crashreporter code was on 12/4, so marcia is going to test an older build to try to verify if this is a server-side regression.

Still fails to submit report using  Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b2pre) Gecko/2007120305 Minefield/3.0b2pre.

(In reply to comment #22)
> The last checkin to crashreporter code was on 12/4, so marcia is going to test
> an older build to try to verify if this is a server-side regression.
> 


https://dyna-crashreports.nslb.sj.mozilla.com is now a redirect to http://www.mozilla.com. Maybe that's the problem.
Just to be clear, the error message that I see logged on Vista is: Crash report submission failed: Unknown error, error code: 0x00002f19.
(In reply to comment #25)
> Just to be clear, the error message that I see logged on Vista is: Crash report
> submission failed: Unknown error, error code: 0x00002f19.
> 

Did you try with an older build?
Yes, See Comment 23 - It fails on that build as well. I can go further back if needed.

(In reply to comment #26)
> (In reply to comment #25)
> > Just to be clear, the error message that I see logged on Vista is: Crash report
> > submission failed: Unknown error, error code: 0x00002f19.
> > 
> 
> Did you try with an older build?
> 


I actually just got this on Windows XP as well.  IE6, not sure if I twiddled any settings or what.

Whatever this is, it's a server side issue.  If I set the "Check for server certificate revocation" option in IE, and visit https://crash-reports.mozilla.com/submit in IE, I get an error dialog--"Revocation information for the security certificate for this site is not available.  Do you want to proceed?".  This is using IE6 on WinXP.

Is the client trying to access https://dyna-crashreports.nslb.sj.mozilla.com, or is it actually https://dyna-crashreports.nslb.sj.mozilla.com.? (Notice the trailing dot!) When I visit https://dyna-crashreports.nslb.sj.mozilla.com (without the FQDN closing dot), it redirects to the crash statistics site, but going to https://dyna-crashreports.nslb.sj.mozilla.com. (with FQDN period) produces a certificate error. (It looks like the certificate contains the "*.mozilla.com" domain, but not the canonical "*.mozilla.com." one. The same error appears when visiting "https://bugzilla.mozilla.org.", this time the "*.mozilla.org." domain is the missing one.)
The client loads https://crash-reports.mozilla.com/submit directly.  I don't know if anything strange happens after that.

I can't submit crash reports under Vista either. My last report I could send was in Fx 3.0b2pre build 2007120405. A change must have landed between then and 3.0b3pre. I get just "Crash report submission failed." Where's good ol' Talkback when you need it?
Please see bug 408889, this is an issue with the Windows HTTP code and certain kinds of certificates.  We are aware of the problem and are working on it.

note: this seems not to be only a vista problem. I was running into this same Problem [12/29/07 02:25:13] Crash report submission failed: Unknown error, error code: 0x00002f19 with Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9b3pre) Gecko/2007122805 Minefield/3.0b3pre and thats a Windows XP x64 SP2 Platform
Indeed, it's IE7-specific, not Vista-specific. Changing summary.
Summary: HttpSendRequest returning ERROR_INTERNET_SEC_CERT_REV_FAILED (crash reporter fails to submit on Vista) → HttpSendRequest returning ERROR_INTERNET_SEC_CERT_REV_FAILED (crash reporter fails to submit when IE7 is installed)
See comment 2, that's known.  Leave Vista in the summary so people can find it.
Summary: HttpSendRequest returning ERROR_INTERNET_SEC_CERT_REV_FAILED (crash reporter fails to submit when IE7 is installed) → HttpSendRequest returning ERROR_INTERNET_SEC_CERT_REV_FAILED (crash reporter fails to submit when IE7 is installed) (also Vista)
Oddly enough, while confirming a bug today re: a printing crash, I actually did get one breakpad report to submit on Vista (prior to today on this machine I always received a crash submission failure). Anyway, the report shows up in my submitted directory - http://crash-stats.mozilla.com/report/index/48b26c45-bfcd-11dc-9dfe-001a4bd46e84?date=2008-01-10-22. Running Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b3pre) Gecko/2008011005 Minefield/3.0b3pre.
I just had a crash, and no attempt to report this seems to have been made. Firefox 3 keeps on crashing often, and the ability to send crash reports had better be fixed very soon if developers want to analyze crash data. Elevate severity of this bug!

Using FF on Vista Home Premium with UAC turned on.
Here is the cert - mrz is going to install it now...

-----BEGIN CERTIFICATE-----
MIIDTDCCArWgAwIBAgIDCJSOMA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT
MRAwDgYDVQQKEwdFcXVpZmF4MS0wKwYDVQQLEyRFcXVpZmF4IFNlY3VyZSBDZXJ0
aWZpY2F0ZSBBdXRob3JpdHkwHhcNMDgwMTE1MjEyMjE0WhcNMTAwMTE1MjEyMjE0
WjCB1jELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGWNyYXNoLXJlcG9ydHMubW96aWxs
YS5jb20xEzARBgNVBAsTCkdUNTg5NzcyOTUxMTAvBgNVBAsTKFNlZSB3d3cuZ2Vv
dHJ1c3QuY29tL3Jlc291cmNlcy9jcHMgKGMpMDgxNzA1BgNVBAsTLkRvbWFpbiBD
b250cm9sIFZhbGlkYXRlZCAtIFF1aWNrU1NMIFByZW1pdW0oUikxIjAgBgNVBAMT
GWNyYXNoLXJlcG9ydHMubW96aWxsYS5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0A
MIGJAoGBAMrrjxQWtgh6xJkFb6DebjldmLr0IU3SUymHaMcos6ISJ4w8IkkGGJS+
59sMpLGR6bMZmioH4dpSZqwQqCYPpMQxi8XjdkeIzxP8Q2+01lYYK/fTVqp2jh3T
WwOk1gbiNBzYYYxxkhXOR5yjj6pfSHdWBJZZJRajYo/xddzmq5p9AgMBAAGjga4w
gaswDgYDVR0PAQH/BAQDAgTwMB0GA1UdDgQWBBTd2p7ARtLT3nVeh8a/M5NdAUWy
TzA6BgNVHR8EMzAxMC+gLaArhilodHRwOi8vY3JsLmdlb3RydXN0LmNvbS9jcmxz
L3NlY3VyZWNhLmNybDAfBgNVHSMEGDAWgBRI5mj5K9KylddH2CMgEE8zmJCf1DAd
BgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDQYJKoZIhvcNAQEFBQADgYEA
Pm4LBvtD8ysPdAZsSVbQ6/8rOGtGqCsBJRtmcyF3FKeIBZQEGokheqeiofDL5HOW
m7gcMk1sH+j9ibKg9YLpN5WwT9ZbCnWkAmVUgUrxx+OL2yVtBKdqiVpa+LW5TiFm
cW9WpswFRN/W+5lOZNLUVHxPz6imzOGPhCctDl1kctM=
-----END CERTIFICATE-----
Assignee: nobody → mrz
cert installed.
Status: NEW → RESOLVED
Closed: 17 years ago → 17 years ago
Resolution: --- → FIXED
(In reply to comment #39)
> I just had a crash, and no attempt to report this seems to have been made.
> Firefox 3 keeps on crashing often, and the ability to send crash reports had
> better be fixed very soon if developers want to analyze crash data. Elevate
> severity of this bug!
> 
> Using FF on Vista Home Premium with UAC turned on.
> 

Hey Roman.  Thanks for your help - this bug was already the highest severity - we were not ignoring it.


verified fixed using Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b3pre) Gecko/2008011504 Minefield/3.0b3pre. Tested on two different machines running Vista Ultimate (laptop and desktop) as well as desktop running Vista Home Premium.
Status: RESOLVED → VERIFIED
Can someone check that it still works properly on Windows 2000?  I believe "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9b3) Gecko/2008020514 Firefox/3.0b3" may have broken the crashreporting on Win2000.
(In reply to comment #44)
> Can someone check that it still works properly on Windows 2000?  I believe
> "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9b3) Gecko/2008020514
> Firefox/3.0b3" may have broken the crashreporting on Win2000.

John: see bug 382124...
Thanks, it appears that there may be a workaround for my problem in bug 382124.  Sorry for using the wrong bug.
You need to log in before you can comment on or make changes to this bug.