Gmail – Frank Leonhardt's Blog

GMail can’t send to sendmail

Gmail Fail

What’s happening with Google? Their Internet engineering used to be spot on. They’re generally a bunch of clever guys, and they follow standards and their stuff just works. Or did. Lately their halo has been getting a bit tarnished, and problems with GMail are a good case in point.

It all started quietly around a month ago on the 6th August. About a week later, people started complaining that users sending mail to them from GMail were getting bounce messages. It looks like Google had rolled out a broken software update, but they’re keeping a low profile on the subject.

After a great deal of investigation it appeared that their new MTA was attempting to make a STARTTLS connection when delivering mail on port 25. STARTTLS is a mechanism that allows encryption on a standard unencrypted channel. Basically, the sender tries a STARTTLS command and if the receiver supports it, returns a reply of “okay” and the remainder of the connection is encrypted using TLS. unfortunately Google’s implementation, which had been working for years, is now broken. The GMail lusers got a bounce back a week later that said it couldn’t negotiate a STARTTLS connection. No further explanation has been forthcoming. STARTTLS should work, and if it doesn’t GMail should try again without using it, but doesn’t.

On the servers I’ve examined there is no problem with STARTTLS. Other MTAs are continuing to use it. All certificate diagnostics pass. Presumably Google has changed the specification as to what kind of TLS/SSL its going to work with, as, presumably, it’s not happy working with all types. Not all servers have this problem. But Google isn’t telling anyone what they’ve done, at least not so far. Working out what’s wrong with their new specification using trial and error takes a while, and I have yet to find a combination that works. And besides, it’s not Google’s place to tell recipients what kind of encryption they should be using, especially when the default state is unencrypted.

Google does offer a troubleshooter but it doesn’t cover this eventuality. There is an option to report other problems, but to date I’ve had no response.

So what’s the solution? The only method I’ve found that works is to disable STARTTLS on Port 25. This means that Google can’t try and fail, and go in to sulk mode. And here’s the bit you’ve probably been waiting for: how to do it.

Assuming you have an access DB configured for sendmail, (the norm) you need to add an extra line somewhere and makemap it:

srv_features: S

On FreeBSD this file is /etc/mail/access and you can make it active using make run from the /etc/mail directory. But you probably knew that.

The srv_features stuff basically tells sendmail which services to advertise as being available. STARTTLS is option ‘S’, with a lower-case letter meaning “advertise it”, and an upper-case meaning “don’t advertise it”. This over-rides defaults, and all we want to do here is stop advertising STARTTLS. If it’s not advertised, Google doesn’t try using it (at least for now).

You might want to read this sendmail documentation for more information in the normal Sendmail easy-to-understand(!) format. If that doesn’t do it for you, look at section 5.1.4.15 of the manual, available in PDF here.

Now Google may defend this state of affairs by saying that they’re implementing something odd with STARTTLS for “security reasons”. There may even be some justification in this. If I knew what they’d changed I might be able to comment on that, but I can’t. However, even if this was the case, they’d be wrong in principle. Since the dawn of Internet email we’ve had RFCs telling us how things should work. You can’t just change the way you do things and expect everyone else to change to suit you, however large you are. And it’s possible that what Google has done is RFC compliant, even if it is bonkers. There are unspecified aspects in RFCs, and some grey areas. However, anyone who’s been around for long enough will know that Sendmail is the de-facto MTA. If you have an argument about the interpretation of an RFC, you can settle it by asking the question “Does it work with sendmail?” If it doesn’t, it’s your problem.

And while we’re at it, it’s really good of Google to stop anyone reading your email while it’s in transit (could they be thinking of the NSA here?) After all, you don’t want email sent through GMail to be readable by anyone until they’re delivered, do you? The only snag is that they are still being read and analysed, by Google. Of course. Email is never secure unless you have end-to-end encryption, and by definition, you can’t get this with a webmail service.

26-August-1523-September-16

Problems receiving mail from GMail – STARTTLS is a bad idea

Gmail Fail

Note: You may wish to read this follow-up article, which contains a solution.

A couple of weeks ago, users started complaining that people using GMAIL (and possibly iCloud) were having their emails bounced back to them from my servers. This is odd – most complaints on the Internet are from users of dodgy hosting companies having their mail rejected by GMail as likely spam. But I haven’t blacklisted Google, and all other mail is working, so they must have been mistaken.

But as soon as I could, I tried it for myself. And sure enough, a bounce came back. The relevent bit is:

Technical details of temporary failure:
TLS Negotiation failed: generic::failed_precondition:
               starttls error (0): protocol error

On fishing around in Sendmail logs, I found evidence that this has been going on all over the place:

sm-mta[84848]: STARTTLS=server, error: accept failed=-1, SSL_error=1, 
               errno=0, retry=-1, relay=mail-qg0-f50.google.com [209.85.192.50]
sm-mta[84848]: STARTTLS=server: 84848:error:1408A0C1:SSL
               routines:SSL3_GET_CLIENT_HELLO:no shared cipher:/usr/src/secure
               /lib/libssl/../../../crypto/openssl/ssl/s3_srvr.c:1073:
sm-mta[84848]: t7QJXCPI084848: mail-qg0-f50.google.com [209.85.192.50] did
               not issue MAIL/EXPN/VRFY/ETRN during connection to MTA

Oh my! The STARTTLS stuff isn’t working because there’s no shared cypher! Hang on a minute, there isn’t supposed to be. Who told Google they could use STARTTLS on port 25. It’d be neat if it worked, but it’s not configured – at least not with a certificate from a public CA. It actually works just fine if you are cool with self-signed (private) certificates. So what is Google playing at?

In the wake of Edward Snowden, people have started worrying about this kind of thing, so companies like Google are trying to be seen doing something about it, and encrypting mail might seem like a good idea. Unfortunately STARTTLS is a bad idea. The rationale behind STARTTLS was to add encryption to a previously unencrypted port’s traffic. If the sender issued a STARTTLS as part of the protocol it could switch in to TLS mode if it knew how; otherwise it would just work as normal. The IETF was very keen on this in the late 1990’s as an easy fix, citing all sorts of iffy reasons, generally to do with having two ports; one standard and one encrypted. They thought it would be confusing, requiring different URLs and not allow for opportunistic automatic encryption of the kind Google seems to be attempting.

As far as I’m concerned, this is rubbish. Having clearly defined encrypted and unencrypted ports means you know where you are. It either is or it isn’t. If you say something must be encrypted, turn off the unencrypted port. STARTTLS allows a fall-back to plain text if you specify the clear text port; and if you have a man-in-the-middle you’ll never know that the STARTTLS was stripped from the negotiations. It opens up a vulnerability that need not be there, all for the sake of saving a port. And time is on my side in this argument. Since 1999 the implementation of encrypted ports has really taken off, with https, smtps (in spite of 465 being rescinded), imaps – you name it – all servers and clients support it and you know where you are.

So what’s this sudden clamouring for the insecure STARTTLS? Naivety on the part of the large internet companies, or a plot to make people think their email traffic is now safe from snoopers when its not?

I’ve reported this problem and I await an answer from Google, but my best guess is that they’re speculatively using STARTTLS, and then barfing and throwing their toys from the pram when it doesn’t work because the verify can’t be done. Having thought about it, I’m okay with the idea of trying STARTTLS as long as you don’t mind about the CA used for the certificate; and if you can’t negotiate a TLS link, go back to plain text. In many ways it’d be better to use the well known port 465 for TLS, and if it can’t be opened, go to plain text on 25. Except there’s no guarantee that port 465 is on the same server as port 25, and it’s normally configured to require SASL authentication. As everyone knows, apart from Google it seems, assumption is the mother of all foul-ups.

Encryption is a good idea, but making assumptions about Port 25 being anything other that straight SMTP is asking for trouble.

12-January-158-September-15

Interesting security issue with Google Apps for Education

I’ve come across a feature of Google Apps for Education that people should really be aware of. It goes like this…

When a school or college signs up for Google Apps for Education, a single email account is used to register a local administrator. This administrator then has control over the sub-accounts, including creation, passwords and monitoring. This would be someone at the school you can trust, right? Because they have access to all your children’s data. And it’s only for school use, so where’s the problem?

Well here’s the problem: that data will probably include a GMail account, and they may not be using it for education-related matters. Creepy. Assuming you trust the monitor, do you snoop on the pupils for their own protection or leave it completely unmoderated, with all the implications for child safety. You’re between a rock and a hard place. By forcing pupils to use an insecure channel you’re responsible for the consequences: if you look you could be accused of voyeurism; if you don’t you can be accused of allowing abuse which you could have prevented.

And it gets worse, because you’re basically logging in using a Google Account. How many people log out when they’re finished? And if a child logs in on a home computer and someone else uses it afterwards without realising, the administrator at the school gets to snoop on data inadvertently added to the account by other members of the household.

Are you a parent, and were you aware of this? You are now!

If you’re a school, my advice is to (a) monitor the monitor; and (b) make sure children know to log out after use; and (c) make very sure that you have parents’ specific permission to allow their children to use the system, being aware of the above. If not and you end up monitoring someone you don’t have permission to (i.e. not your pupil), you’re probably looking at an offence under the Misuse of Computer Act 1990 in the UK, and a class action law suit in the USA. Remember that school in Philadelphia that took snapshots using students’ Macbook webcams without telling anyone? (Robbins v. Lower Merion School District). There was no suggestion of foul play, just naivety on the part of the school district. And it cost them $600K to settle, plus a great deal of embarrassment.

8-November-148-September-15

Google Apps for Schools – how safe are they?

So-called Group Work is probably the bane of every tutor in higher education, myself included. As to the poor students having to collaborate; it’s always the motivated one dragging the hangers-on and possibly university’s resident idiot along with them. It’s a nightmare. The most common complaint is that they never turn up to meetings to work on the project because it’s too difficult to organise. Yeah, right!

So this week, one of my colleges persuaded me to get them all working with Google Apps. The theory is that they don’t need to be co-located in time or space to work on a common document. I suspect the lack of physical presence will actually make it easier for some of the group to loaf off, but perhaps I’ve been at this too long to be optimistic.

Google Apps, on the other hand, is gaining ground in education. Cloud-based applications that allow easy sharing of documents has to be a good thing, and I have to say I’m very impressed at the ability of several people to edit the same document at once. And it comes with the ultimate feature that will guarantee sales – it’s free.

When I say “free”, that means that Google gets to harvest your personal data instead of hard cash, and feed you targeted advertising. And this is a worry. You may be okay with this, but if it’s to be adopted in colleges or schools, supposing some students aren’t as relaxed about it? Those in the know keep away from Facebook for just this reasons, but it’s optional. If you make Google Apps part of coursework you’re forcing students to accept terms they’d otherwise reject.

So, in 2006, Google announced Google Apps for Education, with the advertising stripped out. It’s actually a pretty good deal. Features may change over time, but it’s basically business version of Google Apps with one difference – it’s also free.

Unsurprisingly, Microsoft is really hacked off about this. They’ve been giving their Windows and Office software to educational establishments at a huge discount (or free) in order to get kids hooked on it, and as a result we have a generation that believes Microsoft Office is necessary to do anything. Kids come out of education knowing nothing else, which forces companies to purchase Microsoft Office at the full price in order to make them feel at home.

So, free or otherwise, Google Apps is probably more suited to college use, and Microsoft isn’t going to like it, so is fighting back with lawyers (no surprise there).

For example, last year Microsoft backed a bill in the US state of Massachusetts to block the use of Google Apps in schools.

To quote: “An Act prohibiting service providers who offer cloud computing services to K-12 educational institutions from processing student data for commercial purposes.”

Pernicious as Microsoft’s education offering is, this bill does have a point and I find myself siding with Microsoft for once. In fact I’d go further – no one should be forced to use applications collecting personal data, even in further or higher education.

This is becoming more relevant as I understand many schools are now considering the use of Google for Education. If their students are under 18, how can they even give informed consent? And once the parents understand the issues, who would give consent on their behalf? In most Judistictions, you need to be 13 or over (or 16+ in some parts of Europe) before you are allowed by Google to have a Google account, so it’s not like Google isn’t sensitive to the issue.

My sources inside the chocolate box tell me that the new Apps for Education will be advert free. When pushed, there was no guarantee that tracking wouldn’t happen – only that no adverts would be shown in the Apps themselves. Whether they will appear, based on tracking data, on other web sites remains to be seen and when the child reaches an “appropriate” age they’ll come with years of profile data. I’m awaiting clarification from Google on this matter.

(Update: Google has now publically declared that they will not scan Apps for Education data for advertising purposes, however the devil is in the detail. They don’t say that they don’t scan it for other profiling reasons. And then I found this court document, unearthed by SafeGov, in which Google’s own lawyers admit that they do profile students email and suchlike, meaning they can target adverts in other circumstances.)

And then there’s the question of whether it’s a secure environment. Well, no, it’s not. But that applies to Office 365, most LMS (see blogs passim) and anything else that has public messaging – in this case GMail. Given the problems I’ve had with users of freemail accounts, including GMail, I can’t help but question of the wisdom of allowing children access to it. When you’re signed up for Apps for Education you are supposed to be getting 24/7 support from Google, unlike Joe Public. Whether this helps resolve the issues remains to be seen. It’s also possible to turn off features centrally, such as Chat (an obvious thing to disable). Unfortunately, if you do turn off GMail there’s no other closed
messaging system to use instead.

As with my earlier papers and articles concerning LMS systems, I’m not saying that Google Apps are inherently insecure. In fact, I’ve got a lot of confidence that Google data centres, in particular, are robust. If Google does deliver on it’s data use policy, and is providing this service free of charge and with no strings attached, that’s great news. Microsoft has had their way for far to long for it to be healthy. Google has stated that as Google was born out of a research project at Stanford, they now want to give something back to education and that’s their only motive. It’s nothing to do with scuppering Microsoft; how could you possibly think that?

Like all Internet connect IT for use in schools, it’s the social risks that worry me the most, such as abuse of Internet email. If your school plans to use Google Apps, Office 365 or any other system with open email, just ask to see the risk assessment first.

That said, I’d still prefer to see educational establishments return to the open source model; Linux if you must, and OpenOffice. Computing by and for the people. Or perhaps those days are gone. We’re already stuck with a generation that now believes computing comes from large companies like Google and Microsoft. Sadly, I feel that it’s unlikely that most will have the technical talent in-house to make it happen.

Update:

Some of the concerns expressed here about data usage have now been addressed after Google signed up to this code of conduct IN THE USA.

9-April-139-April-13

Cybercriminals: Microsoft’s X-EIP is your friend.

Since January 2013, and without any fanfare, Microsoft has stopped including the originating IP address of Hotmail emails in the headers. Instead, an ominously named X-EIP has appeared in its place, consisting of random characters.

Originating IP addresses are the only means to verifying the source of an email. This is important to prevent fraud, detect crime and block spam. It can’t be used by a recipient to positively identify a sender, but by contacting the relevant ISP about it, the location can be pinpointed relatively quickly and the ISP can take action against a customer based on a complaint. Even home users can check that the IP address their friend’s email came from is in the right country, rather than a cyber-café in some remote and lawless part of the world.

So why has Microsoft done this? After much waiting for a reply, this is the best I have got:

My name is **** and I am a Senior Support Analyst for Microsoft. I am part of the Hotmail Escalations Team handling this issue.

In the pursuit of protecting the privacy of our users, Microsoft has opted to mask the X-Originating IP address. This is a planned change on the part of Microsoft in order to secure the well-being and safety of our customers.

Microsoft is in the path of continuously improving the online safety and security of its users. Any feedback regarding this concern would be treated with utmost attention.

We appreciate your patience and understanding regarding this matter.

Thank you.
Best Regards, etc.

Note the “wellbeing and safety of [their] customers” in the above. Which of their customers need this protection? Well paedophiles wishing transfer material with their mates anonymously will love it. As will fraudsters, cyber-bullies and anyone else wishing to send untraceable emails.

Having analysed the new encrypted codes, they’re not a one-to-one encryption of an IP address. Two emails from the same address will have different codes, so decoding them won’t be easy at all. It’s likely that it’s a one-way hash, meaning Microsoft will need to go back through its records to find out where an email came from, and they’re only going to do that with a court order, I suspect.

And that’s not good enough – tracking cybercrime is an immediate activity, so such things can be shut down quickly. The Internet is self-policing; there’s no time for court orders, and no point if you’re crossing international boundaries. If you know the IP address some malware came from, it’s possible to get hold of the sender’s ISP and have the feed quenched within minutes, or if coming from a commercial or academic institution, the network administrators could be around to catch them in the act. Microsoft has extended this process from minutes to weeks, losing any reputation for responsibility it had with Hotmail (not much I’ll grant you) and promoting its service to the cyber criminal.

However, Microsoft is not alone. Google has been doing this for years with Gmail. Is this a cynical attempt by Microsoft to follow Google’s shameful lead?

There are some cases where anonymous email is a good idea, such as when sending emails from a country where free speech is aggressively discouraged. There is no need for this with a mainstream email service; it’s just a feature provided to encourage new users with something to hide.