Alexa Top 100 Accessibility

The following data comes from automated accessibility testing of the Alexa Top 100 US websites (minus the pr0n, search engines, social networks and sites primarily driven from user content) using AQUA and lists their performance from worst to best (based on density of errors per page). This information comes with the important caveat that it is limited to automated testing. That being said, my own experience has been that poor performance in automated accessibility testing is strongly correlated with poor performance in manual accessibility testing as well.

Site
Name
Errors Warnings Pages Tested Avg Per Page (Errors +
Warnings)
Avg. Per Page (Errors Only)
LA Times 75 46 1 121.00 75.00
Wal Mart 23691 9404 500 66.19 47.38
IMDB 22746 35552 496 117.54 45.86
CNN 22255 18354 498 81.54 44.69
Answers 21663 2477 500 48.28 43.33
Roadrunner 183772 131066 4979 63.23 36.91
Huffington Post 17652 2697 499 40.78 35.37
Best Buy 383 1886 12 189.08 31.92
Fox News 11456 5478 361 46.91 31.73
Weather Channel 13868 13035 491 54.79 28.24
Washington Post 12850 19892 464 70.56 27.69
NFL 12104 27059 498 78.64 24.31
CNET 11863 34408 499 92.73 23.77
eHow 3987 1464 199 27.39 20.04
Wall St. Journal 201 243 16 27.75 12.56
New York Times 6085 6458 496 25.29 12.27
Microsoft 4824 15804 431 47.86 11.19
Apple 4690 6705 500 22.79 9.38
Chase 2908 848 361 10.40 8.06
BBC 3823 10364 483 29.37 7.92
PayPal 2992 4822 498 15.69 6.01
AOL 60 56 10 11.60 6.00
About.com 833 401 142 8.69 5.87
Netflix 2544 1814 474 9.19 5.37
Verizon Wireless 1447 940 275 8.68 5.26
Comcast 1803 745 368 6.92 4.90
USPS 1158 3289 338 13.16 3.43
Wells Fargo 1176 3831 479 10.45 2.46
UPS 2 272 1 274.00 2.00
Bank of America 489 6316 314 21.67 1.56

Some Definitions

Error
In this context, an “error” is something that was found using a test which us clearly ‘pass/fail’ in nature. Things such as missing alt attributes for images fall into this category.
Warning
In this context, a “warning” is something that was found using a test which does not have a clear ‘pass/fail’ criteria but would instead require human inspection to verify.
If you are interested in learning about the next generation in Web Accessibility Testing, sign up for the release of Tenon.io
If you or your organization need help with accessibility consulting, strategy, or accessible web development, email me directly at karl@karlgroves.com or call me at +1 443-875-7343. Download Resume [MS Word]

6 Comments

  • Posted December 5, 2011 at 6:26 pm   Permalink

    The danger here is that without context, some could look at these stats and make some sweeping judgments about how accessible certain sites are over others. The fact that there is such a variability in the number of pages tested, coupled with the fact that we have no idea about whether pages tested were plain HTML or built using rich internet technologies, or both (which could be within a site or between sites) makes me discomforted with such a ranking without much more information.

    • karlgroves
      Posted December 5, 2011 at 6:33 pm   Permalink

      Jennison, I would concede that in a purely scientific setting it would be important to supply additional details with this data including exactly when testing happened, what tool was used, what tests were performed, exactly which pages were tested, etc. In practice, however, I’ve yet to experience a website who performed poorly in automated testing but performed well in manual or use case testing. The converse, admittedly, can often be true. In the future I plan on doing something that calculates error density-per-page using the document’s size.

      • Posted December 5, 2011 at 6:45 pm   Permalink

        I humbly submit a few questions.
        What tool was used to test these pages?
        How were the pages selected?
        Why such variability in the number of pages tested?
        What were these pages tested against (e.g., WCAG 2.0 AA, Section 508)?

        • karlgroves
          Posted December 5, 2011 at 8:17 pm   Permalink

          I tested these sites (as well as most US Federal Government websites) using AQUA, a tool I’ve been developing for a long time. Over the last year or so, I’ve been working on revamping the way automated testing works – specifically when it comes to speed. The tests weren’t meant to be a test of these sites but rather a test of AQUA’s ability to spider & test rapidly and accurately. I chose these sites because they would be large and complex which afforded me the opportunity to really throttle the spidering capability. I set it to test up to 500 pages per site. Anything less than 500 means that it reached as many pages as it could before experiencing one of any number of items which would cause it to stop (the list of things which would cause it to stop is too numerous and boring to go into here). There’s a weird anomaly with Roadrunner that made it go far beyond 500 on that site. These were tests of WCAG 2.0 AA.

  • Posted December 6, 2011 at 12:27 pm   Permalink

    Hey, finally got to register to this joint. :)

    Just meant to say your last sentence really made me smile: “…poor performance in automated testing is strongly correlated with poor performance in manual testing as well.”

    That’s a hell of an understatement. “Strongly” doesn’t even begin to describe it. Even more so – I’d say that on websites where automated testing fails, it’s actually a waste of everybody’s time and money to bother going even further.

    I really feel that until we’ve reached the point where automated testing still fails, there’s no real point in evaluating for anything else. The basics need to be covered first.

  • Posted October 6, 2013 at 1:57 am   Permalink

    Since a lot of my work doing a11y for Postal is still standing 3 years later, and one of my favorite devs of all time is still working it, I’m happy to see that number so low. I have occasionally had some of my students test the site manually for practice and given my results to my friend as well.
    I would think the news sources would get better numbers. I wonder if Al-Jazeera America is better since the site is so much newer and cleaner than the others.
    I know they are nowhere near a top 100 though.

2 Trackbacks

Post a Comment