521538 - (autodiscover) Support Microsoft's autodiscover protocol (as in Exchange 2007/2010)

Status: RESOLVED DUPLICATE of bug 1500105

(Thunderbird :: Account Manager, enhancement)

Description

[Andy Lyttle]

In addition to the various other ways Thunderbird can determine how an e-mail account should be configured, we should implement Microsoft's autodiscovery mechanism, for compatibility with servers that support this for Outlook 2007.  Lord knows it's not perfect, but there will be servers out there that implement this because Outlook supports it, that won't bother implementing whatever far-superior mechanisms the Mozilla project can come up with.

This should be a third-from-last resort, just ahead of taking a wild stab in the dark to see if we can guess something that works, and making the user enter everything manually.  By all means, try better options first.

The best documentation I've been able to find is this Word document:
http://office.microsoft.com/download/afile.aspx?AssetID=AM102105061033

So here's the basic idea:  first we figure out where to send the user's e-mail address.  Next, we submit a query in XML format via HTTP POST.  Third, we parse the resulting XML data, which can indicate either a redirect to a different URL (go back to step 2), or configuration data (yay!).  For security reasons, we only accept configuration data over HTTPS, never over HTTP (but we can accept a redirect over HTTP, with user approval).

So let's say the user has entered "[e-mail]".  To figure out where to submit our query, we take a few stabs:
1) POST to https://example.com/autodiscover/autodiscover.xml
2) POST to https://autodiscover.example.com/autodiscover/autodiscover.xml
3) GET http://autodiscover.example.com/autodiscover/autodiscover.xml
4) check DNS for an SRV record _autodiscover._tcp.example.com, and POST to /autodiscover/autodiscover.xml on that host

In case 3, we don't submit the e-mail address; we only accept a redirect to an HTTPS URL.  When we get one, prompt the user with the HTTPS URL before POSTing to it.

In case 4, prompt the user with the HTTPS URL we've cobbled together from the SRV record.  It seems to me that using an SRV record for this purpose is ridiculous, but it's better than being limited to the first three options, which is what Outlook 2007 was limited to before Service Pack 1.

Once we've got an HTTPS URL to POST to, we submit this:
<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/requestschema/2006">
    <Request>
        <EMailAddress>[e-mail]</EMailAddress>
        <AcceptableResponseSchema>http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a</AcceptableResponseSchema>
    </Request>
</Autodiscover>

See the Word document for further details.

Comment 1

[Ludovic Hirlimann [:Usul]]

What would this offer over the current plans about the ISPdb (https://wiki.mozilla.org/Thunderbird/ISPDB) beside bringing some support for coporate using exchange ? I don't see much value in implementing this.

Comment 2

[Andy Lyttle]

I work for a company that hosts e-mail for several thousand domains, and we're growing.  Most of these domains only have a handful of users; many have just one.  Our customers are non-technical; occasionally they give us their registrar's login credentials so we can change their MX records for them, because they can't figure out how to do it themselves (we prefer not to do this, but when the customer is frustrated, we try to help however we can).

The vast majority of our user base is running Microsoft Outlook, or using webmail.  Of the rest, most are using either Outlook Express, Windows Mail, Apple Mail, or Entourage.  Very few are running anything else (Thunderbird, Eudora, etc.).  There's a huge incentive for us to support autodiscovery for Outlook, and zero incentive to do anything to support any sort of autoconfiguration system for any other client.  I'm sure the numbers add up differently for home users, but these are business clients, and Outlook is what they're using.

Outlook autodiscovery is relatively new and not widely supported yet (including by the company I work for).  As awareness increases, I expect more providers to add support, and I'd like to see Thunderbird be able to take advantage of it.

My views are my own, and not necessarily those of my employer. ;-)

Comment 3

[David Ascher (:davida)]

Exchange Autodiscovery would be definitely nice (we have a lot of users who currently have to configure exchange by hand, and it would automatically support thousands and thousands of servers).  I would suggest that someone first experiment with making an add-on to the existing autoconfig dialog.  One of the known existing requirements (from talking to the microsoft team) is asking for username _and_ email address, a requirement that we wanted to avoid in the first version of the autoconfig dialog.

Comment 4

[Andy Lyttle]

I'm not sure I understand what you're talking about.  Outlook only asks for the display name, e-mail address, and password; of these, only the e-mail address is submitted to the server.  The username is supplied by the server, along with the rest of the settings.  Thunderbird also asks for the display name and e-mail address when setting up a new e-mail account, so there should be no problem there (the user will be prompted for their password as needed, after everything has been set up).  No additional dialog should be needed.

Comment 5

[Ben Bucksch (:BenB)]

I discussed with Microsoft about this a while ago.
Results (their claims):
- The Exchange 2007 autodiscover protocol does *not* supply IMAP, POP, SMTP
  servers. Only the proprietary MS protocols.
- The Exchange 2010 autodiscover protocol is different, based on SOAP.
  It does supply IMAP and co. It is not supported by Exchange 2007.
- Both protocols require you to authenticate via an unspecified
  authentication protocol, e.g. the proprietary NTLM. Only then does
  the Exchange server return the configuration
  (even though it's just hostnames etc.).
- To authenticate, you need the username (and password). Whoever,
  the username is part of what we want to autodiscover, as it's often
  unclear what it is (or the user forgot it), and even if I know it,
  the exact form requires varies between systems:
  - with and without domain
  - with @ or // as domain separator
  - email address or other username as username
  I very often find myself not able to log in and set up an account,
  for this very reason, at various ISPs and company mail servers.
  You may not have seen a username prompt, because either:
  - The app guesses the username and tries various variations.
    That's a heuristic and not discovery. We already do that when we fail
    to autodiscover the configuration.
  - NTLM is used, which is a single-signon. Windows uses your windows
    login and sends that to an authentication server that it already knows.

I asked Microsoft to add a simple service where I can pass the email-address and password, and if it matches, the service returns the username or the whole configuration. That would be very simple to implement, but Microsoft refused to do any engineering for us whatsoever, by the argument "it works in iPhone".

Comment 6

[Andy Lyttle]

This bug is specifically about implementing autodiscovery for POP3/IMAP as it's supported in Outlook, regardless of what may or may not work in which versions of Exchange.  This can be implemented on the server side with a simple Perl or PHP script.  No pre-authentication is required; you don't need a username and password, just an e-mail address.  The implementation I wrote doesn't even check to see whether the first part of the e-mail address you've supplied is associated with a valid mailbox; it just returns what the correct configuration would be if it were (and I've verified that Outlook uses the username provided by my script, not something guessed heuristically).

I'm not sure to what extent this is supported on iPhone currently.  Apparently SRV records are not supported (as with Outlook 2007 before SP1).  SRV records are supported by Mail on Snow Leopard (Mac OS X 10.6), but Mail only supports Exchange 2007 servers and will ignore configuration data for POP3 or IMAP, so it's quite possible that iPhone Mail does the same.  I know RIM's Blackberry Internet Service supports autodiscovery in some form, but I haven't had a chance to test it yet to see what works and what doesn't.

Comment 7

[Ben Bucksch (:BenB)]

> This bug is specifically about implementing autodiscovery for POP3/IMAP
> as it's supported in Outlook, regardless of what may or may not work
> in which versions of Exchange.

That's pointless for us. The goal would be to autoconfigure Thunderbird with Exchange servers in companies, nothing else.

ISPs (including you) who want to allow autoconfigure should use the protocol shown https://wiki.mozilla.org/Thunderbird:Autoconfiguration (once we implement the ISP part).
If this bug is to make Thunderbird compatible with your Perl implementation or that of other hosters, that's WONTFIX as far as I'm concerned. I therefore change the summary from "Outlook" to "Exchange". IF we add Exchange support and it happens to work with yours, fine.

Comment 8

[Ben Bucksch (:BenB)]

BTW, thanks for the info on Mail.app and iPhone. If you can find out how they find the username and do the authentication, that could be useful.

Comment 9

[David Ascher (:davida)]

I'd love someone to take on extending the autoconfig protocol in an add-on.  Figuring out the impact of something like this would be much easier if we had working code to start with.  The autoconfig dialog is XUL/JS, and I'd strongly advocate for taking whatever patches to it are necessary to _allow_ such an add-on to work (in trunk).

Comment 10

[Ben Bucksch (:BenB)]

If somebody wants to take this on, the relevant code is in
mailnews/base/prefs/content/accountcreation/
and you'd need to write something like fetchConfig.js with a custom readXML.js.
I had an unfinished exchangeAutodiscover.js, but I lost it. Should still be easy to write.

Comment 11

[Ben Bucksch (:BenB)]

Attachment: Example for implementation, exchangeDiscover.js

Actually, I searched and found it again. Here it is.
As said, it's unfinished. But it may help to get somebody to see how to implement it.
It's half a year old, so small changes may be required to adapt to the trunk source.

Comment 12

[Ben Bucksch (:BenB)]

Attachment: Changes required for fetchHTTP.js

You'll need this patch for fetchHTTP.js, which adds the POST of body values
and an Abortable for combining several network calls, as may be useful for implement the REDIRECT feature in the autodiscover protocol.
Again, this patch might be bitrotten.

Also, you'll of course need to add a call to the exchangeAutodiscover class in emailWizard.js, after the fetchConfig() calls.

Comment 13

[Ben Bucksch (:BenB)]

Comment on attachment 409947 [details]
Example for implementation, exchangeDiscover.js

The XML reading function is currently reading our format.

This is really just throwing my mid-stage implementation out, in case it's useful to anybody.

Comment 14

[Andy Lyttle]

Ben, I'm confused:  if you're saying Exchange 2007 does not support autodiscovery for POP3/IMAP protocols, then..  what are we trying to discover?  Thunderbird doesn't support Exchange accounts, does it?

Comment 15

[Ben Bucksch (:BenB)]

Right. I don't see any way to support Exchange 2007 autodiscover, given what Microsoft said, for that reason and the login problem, that's why I stopped.

Comment 16

[Andy Lyttle]

Well then I'm changing the bug title back.

We have a chicken-and-egg problem here.  Not many servers support this, because a lot of clients don't support it, which is because not a lot of servers support it.  If it were more widely supported, users would expect it, which would put pressure on server admins to support it, which would make it more widely supported.

As a server admin, I wouldn't mind supporting Mozilla's proprietary autodiscovery protocol alongside Microsoft's proprietary autodiscovery protocol, but Microsoft's is already widely deployed on the client side and Mozilla's isn't even clearly defined yet.  Microsoft's protocol sucks, but at least it's defined pretty clearly.

Comment 17

[Ben Bucksch (:BenB)]

> We have a chicken-and-egg problem here.  Not many servers support this, because
> a lot of clients don't support it, which is because not a lot of servers
> support it.  If it were more widely supported, users would expect it, which
> would put pressure on server admins to support it, which would make it more
> widely supported.

You see, that's the point: I do *not* want that to happen. In fact, quite the opposite, I want to prevent that from happening. Mozilla's stated goals are open protocols, and helping spread a proprietary protocol from Microsoft, which is not very well designed, and they even change themselves entirely with every release, is definitely not what we want. In fact, we might go as far as actively *prevent* that from happening, if that's a serious risk. If one or two hosters implement MS's protocol, and we use it, so be it, but I don't want to help it spread the net, as you stated as goal. I want *our* protocol to be the new standard.

If we support the MS Exchange autodiscover protocol, the *only* goal is to support Exchange. Because we have no other option there.

Therefore, I change the title back from "Outlook" to "Exchange" (Sorry), because it would be WONTFIX as-is and I'd be forced to file a new bug.

There is an actual difference between your request and Exchange support, as we saw, as your own Perl implementation and Exchange differ: Exchange 2007 doesn't give IMAP configs, and both Exchange 2007 and 2010 require authentication. 

According to Microsoft - I couldn't verify it. If somebody can prove that wrong, all the better.

Comment 18

[Andy Lyttle]

I don't see Mozilla's solution as being any less proprietary than Microsoft's.  Both are open, both are documented (or will be, in Mozilla's case), both are controlled by a single vendor.  Mozilla's is better designed (or will be, when it's completed), but Microsoft's is adequate in most situations.

I'm still confused as to what you hope to accomplish here:  if Exchange 2007 cannot be made to serve POP3/IMAP config data, then what are you trying to add support for?

Another reason why supporting Microsoft's protocol is a good idea:  the protocol exists and is documented, which gives us something solid we can actually implement now.  Once that's in place, adding support for other autodiscovery methods should be easy; the framework will be there and it's just a matter of adjusting the details.

By the way, I mentioned that RIM's BIS supports some form of autodiscovery; I've confirmed that they do not support SRV records, and they do try to guess the configuration heuristically.  I don't know whether they try to fetch autodiscover.xml by other means, since I don't have that set up.  Specifically, they cycle through pop3s/pop3/imaps/imap, trying the full e-mail address as the username, then just the first part as the username.

Comment 19

[Ben Bucksch (:BenB)]

FIXED in bug 1500105.