SParseval: Evaluation metrics for parsing speech

While both spoken and written language processing stand to benefit from parsing, the standard Parseval metrics (Black et al., 1991) andtheir canonical implementation (Sekine and Collins, 1997) are only useful for text. The Parseval metrics are undefined when the words
input to the parser do not match the words in the gold standard parse tree exactly, and word errors are unavoidable with automatic speech
recognition (ASR) systems. To fill this gap, we have developed a publicly available tool for scoring parses that implements a variety
of metrics which can handle mismatches in words and segmentations, including: alignment-based bracket evaluation, alignment-based
dependency evaluation, and a dependency evaluation that does not require alignment. We describe the different metrics, how to use the
tool, and the outcome of an extensive set of experiments on the sensitivity of the metrics.