The MotherEffingToolConfuser

Many academic papers dealing with web accessibility make use of automated tools as part of their research. For instance, one recently published paper did a comparative analysis of the Forbes 250. Unfortunately, the tool in use did not test the DOM of the pages it analyzed.  On one hand, you can say that at least all of the sites compared were tested against identical criteria, which I concede is an important consideration. I would still argue that the data is still invalid because the measurements were taken with a flawed tool. I think of it like doing a study of temperatures using a broken thermometer. You’ll never get an accurate measurement if you use the wrong tool.

Consequently, I created the “Mother Effing Tool Confuser” (naming idea shamelessly stolen from Paul Irish’s Mother Effing Text Shadow and Mother Effing HSL). Anyway, the first criteria against which any automated testing tool must be measured is this: Does the tool test the DOM? Be careful here, because some vendors will say that they test the DOM, but it is a browser DOM that matters. There’s a difference between “making” a DOM and actually using a headless browser to test the browser DOM. Nearly all modern programming languages have the ability to make a DOM (primarily best suited for processing XML files). Java has jDOM, PHP has a handful of DOM classes in the standard library, and there are similar built in functionalities in JSP, C++ and others. These are not the same as the browser DOM. The true DOM that must be tested is the browser DOM. Essentially this means the tool either resides in the browser as a plug-in or has its own headless browser.

Go to Mother Effing Tool Confuser with your favorite tool to see me holding a puppy. Then come back here to comment. How did your favorite tool do?

If you are interested in learning about the next generation in Web Accessibility Testing, give Tenon.io a try.
If you or your organization need help with accessibility consulting, strategy, or accessible web development, email me directly at karl@karlgroves.com or call me at +1 443-875-7343. Download Resume [MS Word]

5 Comments

  • patrickdunphy
    Posted January 31, 2013 at 2:06 pm   Permalink

    Quick question – are your referring to commercial tools specifically or all types of tools?

    I enjoy testing for accessibility & I’d like to think I’m quite competent doing so.

    Generally I refer people to use WAVE, FAE & aChecker when they want to learn if site/page X is accessible. This being said, I always preface that manual testing is a must for more indepth & thorough results.

    Having run all 3 against your test page, I’m now experiencing some self doubt..

    I did think that Wave in particular was interesting – 3 different tests produced 3 different results.

    aChecker results:
    Known Problems(10)
    Likely Problems (6)
    Potential Problems (72)
    HTML Validation (5)
    CSS Validation (29)

    Wave by URL – 5 errors

    Wave w/plugin – 0 results

    Wave 5:
    6 errors
    5 alerts
    1 features
    17 structural elements
    3 HTML5/ARIA elements
    1 contrast errors

    FAE
    Navigation & Orientation – 100% Pass
    Text Equivalents – 100% Pass
    Scripting – 100% Pass
    Styling – 66% Pass, 16% Warn, 16% Fail
    HTML Standards – 50% Pass, 50% Warn

  • Dan Mouyard
    Posted February 11, 2013 at 11:34 pm   Permalink

    This is an awesome tool!

    @patrickdunphy – I think the Firefox extension version of WAVE is more accurate since it’s using the browser’s DOM.

    My current favorite tool is the HTML_CodeSniffer bookmarklet, but it shows an error on the select element because none of the options has the selected attribute.

  • Joshua
    Posted February 12, 2013 at 11:30 am   Permalink

    Great page to have to test some of the automated tools available!

    I checked the page using SSBBartGroup’s AMP tool and received the following:

    508.22: 100%
    WCAG 2.0 lvl A: 100%
    WCAG 2.0 lvl AA: 100%

    Violations: None

  • Joshua
    Posted February 12, 2013 at 7:28 pm   Permalink

    Tested tool HTML_CodeSniffer by Squiz:

    508: 0 errors, 0 warnings, 3 notices
    WCAG 2.0 lvl A: 1 error, 3 warnings, 20 notices
    WCAG 2.0 lvl AA: 2 errors, 3 warnings, 22 notices

    The report stated errors within the drop down menu, while the warnings were stating “Are you sure you want to use instead of headers?”

  • Joshua
    Posted February 12, 2013 at 7:31 pm   Permalink

    Tested tool HTML_CodeSniffer by Squiz:

    508: 0 errors, 0 warnings, 3 notices
    WCAG 2.0 lvl A: 1 error, 3 warnings, 20 notices
    WCAG 2.0 lvl AA: 2 errors, 3 warnings, 22 notices

    The report stated errors within the drop down menu, while the warnings were stating “Are you sure you want to use the emphasis tag instead of headers?”

Post a Comment