Coding book:
http://www.hcrc.ed.ac.uk/~jeanc/
Authors: Carletta, J. C., Isard,
A., Isard, S., Kowtko, J., Doherty-Sneddon, G. and
Anderson, A.
Title: HCRC Dialogue Structure
Coding Manual
Human Communication Research
Centre HCRC TR-82, University of Edinburgh, Edinburgh, Scotland; 1996
A slightly shortened version
of the coding instructions can be found in
Authors: Carletta, J. C., Isard,
A., Isard, S., Kowtko, J., Doherty-Sneddon, G. and Anderson, A.
Title: The Reliability of a
Dialogue Structure Coding Scheme. Computational Linguistics, 23, 13-31.
1997
Number of annotators:
Main Map Task corpus was annotated
using four different coders. In all, at least 50 people have tried
the scheme, with around a dozen research projects employing it. Most of
the annotators were PhD students in linguistics or psychology; one was
just someone with a degree in modern languages.
Number of dialogues:
128 in the original Map Task
corpus (English) , plus at least as many again coded using the same scheme
or minor variants, comprised of Map Task in other languages,
in other conditions (audio-only, video-mediated, children), and dialogues
for other tasks (e.g., travel planning, financial services simulations,
simpler children's tasks).
Evaluations of scheme:
Full results published in Carletta,
J. C., Isard, A., Isard, S., Kowtko, J., Doherty-Sneddon, G. and Anderson,
A. (1997) The Reliability of a Dialogue Structure Coding Scheme. Computational
Linguistics, 23, 13-31.
For act segmentation, k
=.92 (N=4079,k=4).
For act classification on main
corpus, k =.83
(N=563,k=4).
For act classification using
naive coders and written instructions, k
=.67
(N=139,k=3; agreement k
=.69 when coding developer added to pool).
For main distinction between
initiation, response, or ready, using naive coders, k
=.84.
Disagreements were between CHECK
and QUERY-YN, INSTRUCT and CLARIFY, and ACKNOWLEDGE, READY, and REPLY-Y.
Underlying task:
Linguistically motivated, but
developed on map task (and therefore likely to be missing categories for
goal negotiation).
List of phenomena annotated:
Primarily dialogue acts, but
the papers also describe coding and reliability for higher level discourse
structure built from the acts, in terms of goal-oriented dialogue games
and transactions relating to dialogue planning divisions. These
levels are not part of the MATE specification.
Markup language:
Current mark-up specified in
an HCRC internal document; SGML-conformant and based on the TEI.
Existence of annotation tools:
No tools publicly available;
in-house tools for move coding operate in python using LT-XML and Tk, and
in Microsoft Word. Nb can be configured to perform the annotation
(implemented in TCL/Tk).
No semi-automatic annotation available.
Usability:
Has been used to structure the
dialogue planning element of an SDS, to learn how to mark dialogue moves
based on topic spotting, and to train the relationship between prosody
and move type.
Contact address:
Maptask@cogsci.ed.ac.uk
Last ModificationL 27.8.1998 by Marion
Klein