Encryption – Frank Leonhardt's Blog

GMail can’t send to sendmail

Gmail Fail

What’s happening with Google? Their Internet engineering used to be spot on. They’re generally a bunch of clever guys, and they follow standards and their stuff just works. Or did. Lately their halo has been getting a bit tarnished, and problems with GMail are a good case in point.

It all started quietly around a month ago on the 6th August. About a week later, people started complaining that users sending mail to them from GMail were getting bounce messages. It looks like Google had rolled out a broken software update, but they’re keeping a low profile on the subject.

After a great deal of investigation it appeared that their new MTA was attempting to make a STARTTLS connection when delivering mail on port 25. STARTTLS is a mechanism that allows encryption on a standard unencrypted channel. Basically, the sender tries a STARTTLS command and if the receiver supports it, returns a reply of “okay” and the remainder of the connection is encrypted using TLS. unfortunately Google’s implementation, which had been working for years, is now broken. The GMail lusers got a bounce back a week later that said it couldn’t negotiate a STARTTLS connection. No further explanation has been forthcoming. STARTTLS should work, and if it doesn’t GMail should try again without using it, but doesn’t.

On the servers I’ve examined there is no problem with STARTTLS. Other MTAs are continuing to use it. All certificate diagnostics pass. Presumably Google has changed the specification as to what kind of TLS/SSL its going to work with, as, presumably, it’s not happy working with all types. Not all servers have this problem. But Google isn’t telling anyone what they’ve done, at least not so far. Working out what’s wrong with their new specification using trial and error takes a while, and I have yet to find a combination that works. And besides, it’s not Google’s place to tell recipients what kind of encryption they should be using, especially when the default state is unencrypted.

Google does offer a troubleshooter but it doesn’t cover this eventuality. There is an option to report other problems, but to date I’ve had no response.

So what’s the solution? The only method I’ve found that works is to disable STARTTLS on Port 25. This means that Google can’t try and fail, and go in to sulk mode. And here’s the bit you’ve probably been waiting for: how to do it.

Assuming you have an access DB configured for sendmail, (the norm) you need to add an extra line somewhere and makemap it:

srv_features: S

On FreeBSD this file is /etc/mail/access and you can make it active using make run from the /etc/mail directory. But you probably knew that.

The srv_features stuff basically tells sendmail which services to advertise as being available. STARTTLS is option ‘S’, with a lower-case letter meaning “advertise it”, and an upper-case meaning “don’t advertise it”. This over-rides defaults, and all we want to do here is stop advertising STARTTLS. If it’s not advertised, Google doesn’t try using it (at least for now).

You might want to read this sendmail documentation for more information in the normal Sendmail easy-to-understand(!) format. If that doesn’t do it for you, look at section 5.1.4.15 of the manual, available in PDF here.

Now Google may defend this state of affairs by saying that they’re implementing something odd with STARTTLS for “security reasons”. There may even be some justification in this. If I knew what they’d changed I might be able to comment on that, but I can’t. However, even if this was the case, they’d be wrong in principle. Since the dawn of Internet email we’ve had RFCs telling us how things should work. You can’t just change the way you do things and expect everyone else to change to suit you, however large you are. And it’s possible that what Google has done is RFC compliant, even if it is bonkers. There are unspecified aspects in RFCs, and some grey areas. However, anyone who’s been around for long enough will know that Sendmail is the de-facto MTA. If you have an argument about the interpretation of an RFC, you can settle it by asking the question “Does it work with sendmail?” If it doesn’t, it’s your problem.

And while we’re at it, it’s really good of Google to stop anyone reading your email while it’s in transit (could they be thinking of the NSA here?) After all, you don’t want email sent through GMail to be readable by anyone until they’re delivered, do you? The only snag is that they are still being read and analysed, by Google. Of course. Email is never secure unless you have end-to-end encryption, and by definition, you can’t get this with a webmail service.

26-August-1523-September-16

Problems receiving mail from GMail – STARTTLS is a bad idea

Gmail Fail

Note: You may wish to read this follow-up article, which contains a solution.

A couple of weeks ago, users started complaining that people using GMAIL (and possibly iCloud) were having their emails bounced back to them from my servers. This is odd – most complaints on the Internet are from users of dodgy hosting companies having their mail rejected by GMail as likely spam. But I haven’t blacklisted Google, and all other mail is working, so they must have been mistaken.

But as soon as I could, I tried it for myself. And sure enough, a bounce came back. The relevent bit is:

Technical details of temporary failure:
TLS Negotiation failed: generic::failed_precondition:
               starttls error (0): protocol error

On fishing around in Sendmail logs, I found evidence that this has been going on all over the place:

sm-mta[84848]: STARTTLS=server, error: accept failed=-1, SSL_error=1, 
               errno=0, retry=-1, relay=mail-qg0-f50.google.com [209.85.192.50]
sm-mta[84848]: STARTTLS=server: 84848:error:1408A0C1:SSL
               routines:SSL3_GET_CLIENT_HELLO:no shared cipher:/usr/src/secure
               /lib/libssl/../../../crypto/openssl/ssl/s3_srvr.c:1073:
sm-mta[84848]: t7QJXCPI084848: mail-qg0-f50.google.com [209.85.192.50] did
               not issue MAIL/EXPN/VRFY/ETRN during connection to MTA

Oh my! The STARTTLS stuff isn’t working because there’s no shared cypher! Hang on a minute, there isn’t supposed to be. Who told Google they could use STARTTLS on port 25. It’d be neat if it worked, but it’s not configured – at least not with a certificate from a public CA. It actually works just fine if you are cool with self-signed (private) certificates. So what is Google playing at?

In the wake of Edward Snowden, people have started worrying about this kind of thing, so companies like Google are trying to be seen doing something about it, and encrypting mail might seem like a good idea. Unfortunately STARTTLS is a bad idea. The rationale behind STARTTLS was to add encryption to a previously unencrypted port’s traffic. If the sender issued a STARTTLS as part of the protocol it could switch in to TLS mode if it knew how; otherwise it would just work as normal. The IETF was very keen on this in the late 1990’s as an easy fix, citing all sorts of iffy reasons, generally to do with having two ports; one standard and one encrypted. They thought it would be confusing, requiring different URLs and not allow for opportunistic automatic encryption of the kind Google seems to be attempting.

As far as I’m concerned, this is rubbish. Having clearly defined encrypted and unencrypted ports means you know where you are. It either is or it isn’t. If you say something must be encrypted, turn off the unencrypted port. STARTTLS allows a fall-back to plain text if you specify the clear text port; and if you have a man-in-the-middle you’ll never know that the STARTTLS was stripped from the negotiations. It opens up a vulnerability that need not be there, all for the sake of saving a port. And time is on my side in this argument. Since 1999 the implementation of encrypted ports has really taken off, with https, smtps (in spite of 465 being rescinded), imaps – you name it – all servers and clients support it and you know where you are.

So what’s this sudden clamouring for the insecure STARTTLS? Naivety on the part of the large internet companies, or a plot to make people think their email traffic is now safe from snoopers when its not?

I’ve reported this problem and I await an answer from Google, but my best guess is that they’re speculatively using STARTTLS, and then barfing and throwing their toys from the pram when it doesn’t work because the verify can’t be done. Having thought about it, I’m okay with the idea of trying STARTTLS as long as you don’t mind about the CA used for the certificate; and if you can’t negotiate a TLS link, go back to plain text. In many ways it’d be better to use the well known port 465 for TLS, and if it can’t be opened, go to plain text on 25. Except there’s no guarantee that port 465 is on the same server as port 25, and it’s normally configured to require SASL authentication. As everyone knows, apart from Google it seems, assumption is the mother of all foul-ups.

Encryption is a good idea, but making assumptions about Port 25 being anything other that straight SMTP is asking for trouble.

21-December-1224-December-12

Bitlocker, PGP and TrueCrypt encryption broken (sort-of)

ElcomSoft has released a utility called Forensic Disk Decryptor that can get the data off encrypted hard drives without knowing the password. According to their web site it:

Decrypts information stored in three most popular crypto containers
Mounts encrypted BitLocker, PGP and TrueCrypt volumes

Amazing!

In complete decryption mode, Elcomsoft Forensic Disk Decryptor will automatically decrypt the entire content of the encrypted container, providing investigators with full, unrestricted access to absolutely all information stored on encrypted volumes.

Wow!

Reading the technical details further, it’s not quite so amazing – they haven’t found a back-door to these encryption algorithms. Instead they’re examining the machine’s core (memory/RAM in user-land parlance) and pinching the key when they find it. This does, unfortunately, require that the machine in question is already running and decryption to be taking place ‘cos its user has already entered the password. This isn’t has hopeless as it sounds as there may be a core-dump (hibernation file) kicking around on an unencrypted hard disk, and indeed this is a known technique (one of very few) for getting data off these drives. Other methods are scaring your suspect with a slap on the wrist if they don’t cough up the password, or running a trojan on the suspect’s PC (questionable legality).

According to ElcomSoft’s CEO, Vladimir Katalov, “Our customers asked us for a tool like this for a long, long time. We’re finally releasing a product that’s able to access encrypted volumes produced by all three popular crypto containers.”

ElcomSoft is a company that certainly knows what it’s doing, and this tool appears to automate a process that’s a PITA to do manually, but Mr. Katalov’s miraculous claims for the product shouldn’t unduly worry the user’s of this technology. It’s probably a good tool but it can’t do anything that wasn’t possible before.