Tokenizer tests
===============

The test format is [JSON](http://www.json.org/). This has the advantage
that the syntax allows backward-compatible extensions to the tests and
the disadvantage that it is relatively verbose.

Basic Structure
---------------

    {"tests": [
        {"description": "Test description",
        "input": "input_string",
        "output": [expected_output_tokens],
        "initialStates": [initial_states],
        "lastStartTag": last_start_tag,
        "ignoreErrorOrder": ignore_error_order
        }
    ]}

Multiple tests per file are allowed simply by adding more objects to the
"tests" list.

`description`, `input` and `output` are always present. The other values
are optional.

### Test set-up

`test.input` is a string containing the characters to pass to the
tokenizer. Specifically, it represents the characters of the **input
stream**, and so implementations are expected to perform the processing
described in the spec's **Preprocessing the input stream** section
before feeding the result to the tokenizer.

If `test.doubleEscaped` is present and `true`, then `test.input` is not
quite as described above. Instead, it must first be subjected to another
round of unescaping (i.e., in addition to any unescaping involved in the
JSON import), and the result of *that* represents the characters of the
input stream. Currently, the only unescaping required by this option is
to convert each sequence of the form \\uHHHH (where H is a hex digit)
into the corresponding Unicode code point. (Note that this option also
affects the interpretation of `test.output`.)

`test.initialStates` is a list of strings, each being the name of a
tokenizer state. The test should be run once for each string, using it
to set the tokenizer's initial state for that run. If
`test.initialStates` is omitted, it defaults to `["data state"]`.

`test.lastStartTag` is a lowercase string that should be used as "the
tag name of the last start tag to have been emitted from this
tokenizer", referenced in the spec's definition of **appropriate end tag
token**. If it is omitted, it is treated as if "no start tag has been
emitted from this tokenizer".

### Test results

`test.output` is a list of tokens, ordered with the first produced by
the tokenizer the first (leftmost) in the list. The list must mach the
**complete** list of tokens that the tokenizer should produce. Valid
tokens are:

    ["DOCTYPE", name, public_id, system_id, correctness]
    ["StartTag", name, {attributes}*, true*]
    ["StartTag", name, {attributes}]
    ["EndTag", name]
    ["Comment", data]
    ["Character", data]
    "ParseError"

`public_id` and `system_id` are either strings or `null`. `correctness`
is either `true` or `false`; `true` corresponds to the force-quirks flag
being false, and vice-versa.

When the self-closing flag is set, the `StartTag` array has `true` as
its fourth entry. When the flag is not set, the array has only three
entries for backwards compatibility.

All adjacent character tokens are coalesced into a single
`["Character", data]` token.

If `test.doubleEscaped` is present and `true`, then every string within
`test.output` must be further unescaped (as described above) before
comparing with the tokenizer's output.

`test.ignoreErrorOrder` is a boolean value indicating that the order of
`ParseError` tokens relative to other tokens in the output stream is
unimportant, and implementations should ignore such differences between
their output and `expected_output_tokens`. (This is used for errors
emitted by the input stream preprocessing stage, since it is useful to
test that code but it is undefined when the errors occur). If it is
omitted, it defaults to `false`.

xmlViolation tests
------------------

`tokenizer/xmlViolation.test` differs from the above in a couple of
ways:

-   The name of the single member of the top-level JSON object is
    "xmlViolationTests" instead of "tests".
-   Each test's expected output assumes that implementation is applying
    the tweaks given in the spec's "Coercing an HTML DOM into an
    infoset" section.