Human judgements play a key role in the development and the assessment of linguistic resources and methods in Computational Linguistics. [...]
We invite papers about experiments that collect human judgements for Computational Linguistic purposes, with a particular focus on linguistic tasks that are controversial from a theoretical point of view (e.g., some coding tasks having to do with semantics or pragmatics). Such experimental tasks are usually difficult to design and interpret, and they typically result in mediocre inter-rater reliability.
So let me think. "Coding tasks having to do with semantics and pragmatics" "typically result in mediocre inter-rater reliability". Seriously? So are we back to the 50s and Chomsky's concept of "ungrammaticality"? The dominance of syntax and the marginalization of semantics and pragmatics as "subjective"?
Now that we finally made it through enough of Chomsky 50+ years later, now that CL is finally breaking free of attempts to formalize semantics, now that we have finally figured it out how to relate language and information theory, we now willingly take a turn back and look at "human judgements"? Why? Language is definitely not created in a vacuum. Virtually every level of natural language (and hence also of CL) is potentially subjective in that it inevitably reflects the 'theory' of the linguist who looks at it. There is no way around this. Claiming that "some coding" is subjective implies that some other "coding" is not. Well, the point is that if it is not, then it has nothing to do with *natural* language.
