CSS test suites: lessons learned
- Tantek Çelik, Microsoft
- 20040303 W3C Plenary Meeting QA Panel
CSS1 test suite
- first CSS test suite
- small number of folks created, drove and managed. many in the group and public also made contributions.
- simple navigation limited by what one person could do.
- implementations use test suites more than the spec
- need completion deadline
- easy to keep tweaking
- 'done' = taken seriously kind of like how "last call" is often treated as "first call".
- 'completed' test suite = better interoperability
- tweak later anyway
set an essentially arbitrary deadline to call it done, don't worry, it won't stop you from tweaking it later.
Selectors test suite
- Most recent CSS test suite
- also small number of folks
- new purpose: help spec exit CR : demonstrate two interoperable implementations for each feature.
- created Implementation Report template
- simple header
- rows are features
- columns are markup languages
- cells are tests for a feature in a language
- feature-centric view revealed missing tests e.g. scroll to :active row, CSS2(.1) :active, applies to non-hyperlinks as well.
- sample report
- test numbers link to tests files (click on test link to go directly to a test and verify it and back)
- pass: green, right aligned
- fail: red, on the left
- na: gray, in the middle
- "at a glance" test results
- whole feature: solid block of green in a cell which is necessary to satisfy CR exit criteria.
- a feature across multiple languages
- implementations want "column of green"
- test suites help verify and even debug specs
- although more tests = more interoperable,
need to "complete" the test suite to get use
- corollary: beware of features (or specs) with few or no tests
- "at a glance" implementation reports provide a "reality check"