Genii Weblog
A test designed to provoke failure?
Wed 7 Jan 2009, 10:02 AM
Tweetby Ben Langhinrichs
In my earlier Email Standards Project post, I showed the relatively dismal results demonstrated by both Outlook 2007 and Lotus Notes 8, with the caveat that the Notes 8 mail client actually rendered much, much better than this. But finding that out made me look back at the test results for the so-called "Acid Test" used by the project. To the right are results from this page.
Only three clients are marked as "Poor", Lotus Notes 8, Outlook 2007 and Google Gmail, while clients such as AOL Webmail and Yahoo! Mail (classic) are marked as "Excellent". Does that really pass the smell test? When three of the most popular business email systems, with hundreds of millions of customers between them, are all marked "Poor" with extensive critiques, and webmail clients are all marked as excellent, one must suspect a bit of researcher bias.
A few clues can be found in the critiques made:
Margin, padding and floats are vital to any design. Outlook 2007 supports none of them, which means even the most elementary standards-based design fails to render.While these seem like nice features to support, are they "vital to any design" when it comes to email? We are not talking about web design here, but email. These are all features that I am working on with CoexLinks, and I don't mean to minimize the cumulative impact entirely, but "vital to any design"?
One of the core recommendations we are proposing is support for embedded CSS in the head. While support for inline CSS is probably better than nothing, it isn't a solid alternative for true, standards-based support. Therefore, because of our recommendation, our acid test does not include inline CSS.I'm sorry, but this strikes me as awfully arbitrary. My extensive experience with MIME generation is that many e-mail systems will not allow a head at all, so inline CSS has been a requirement. Perhaps that is not a perfect world, but again, we are not talking about web design, where CSS is often best consolidated into external files or in the head, but rather email, where all CSS has to be sent along with, and so is often equally valid inline as in the head.
Lotus Notes is widely used, but primarily by people who work in organizations that mandate its use. It's an application which has a lot to offer the business world, but when it comes to email rendering Lotus Notes falls behind. Lotus Notes supports a number of items from our list, but they matter little when the most important properties fail. And they can even be detrimental to readability as with its support of color while not supporting background-color.Pardon me, but your bias is showing rather blatantly, especially for a group that didn't know enough about Notes on their own to test anything but IMAP/POP3. That first sentence is both blatantly biased and quite likely true of any business based email system, besides being absolutely unprovable. As for the solitary specific, while I agree that the lack of background-color is an issue for some emails, it feels like there was no real interest in giving Notes a chance.
But perhaps the most damning evidence of bias is what was left out entirely. Lotus Notes 8 is judged by IMAP/POP3, which is bad enough, but there is not the slightest mention of iNotes. Outlook 2007 is judged by its rich client, but not by Outlook Web Access. Yet AOL is judged only by its webmail, without the slightest mention of its heavily used client. So, why would anybody do that? Because it allows you to show the state of the clients as shown here, without having to mention that you are cherry-picking the evidence to "prove" some original belief, namely that web mail is the "correct" mail, and that the less the mail client impacts the presentation the better. Those may be valid opinions, but they should not dictate a test which is designed to make sure the leading mail systems fail, but presented as an objective test.
Copyright © 2009 Genii Software Ltd.
What has been said: