CSS test suites: lessons learned

presenter

Tantek Çelik, Microsoft

event

20040303 W3C Plenary Meeting QA Panel

CSS1 test suite

• first CSS test suite
• small number of folks created, drove and managed. many in the group and public also made contributions.
• simple navigation limited by what one person could do.
  • toc with sections
  • flattened linear traversal next/prev links.
• implementations use test suites more than the spec
  • feature with more tests = more interoperable e.g. border-style
  • feature with fewer tests = less interoperable e.g. display:list-item which still has interoperability problems today.
  • interoperability may be limited to what was tested e.g. float
  • features may be expanded by what was tested e.g. white-space on inline elements.
    • spec said white-space only on block level elements
    • spec errata'd to reflect interoperability
    • css2.1 incorporated the expansion
• need completion deadline
  • easy to keep tweaking
  • 'done' = taken seriously kind of like how "last call" is often treated as "first call".
  • 'completed' test suite = better interoperability
  • tweak later anyway set an essentially arbitrary deadline to call it done, don't worry, it won't stop you from tweaking it later.

Selectors test suite

• Most recent CSS test suite
• also small number of folks
• new purpose: help spec exit CR : demonstrate two interoperable implementations for each feature.
• created Implementation Report template
  • simple header
  • rows are features
  • columns are markup languages
  • cells are tests for a feature in a language
  • feature-centric view revealed missing tests e.g. scroll to :active row, CSS2(.1) :active, applies to non-hyperlinks as well.
• sample report
  • test numbers link to tests files (click on test link to go directly to a test and verify it and back)
  • pass: green, right aligned
  • fail: red, on the left
  • na: gray, in the middle
  • "at a glance" test results
  • whole feature: solid block of green in a cell which is necessary to satisfy CR exit criteria.
  • a feature across multiple languages
  • implementations want "column of green"

Conclusions

1. test suites help verify and even debug specs
2. although more tests = more interoperable, need to "complete" the test suite to get use
3. corollary: beware of features (or specs) with few or no tests
4. "at a glance" implementation reports provide a "reality check"