The quality control issue has always been the focus of Crowdsourcing research. However, existing surveys predominantly concentrate on quality control in Boolean tasks, which are generally formulated as simple label classification, ranking, or numerical prediction. Ubiquitous open-ended tasks like question-answering, translation, and semantic segmentation have not been sufficiently discussed due to several challenges, like large to infinite answer spaces, complex task structure, dynamic and multidimensional worker abilities and non-unique acceptable answers.
In order to discuss and summarize the quality control methods applicable to open-ended tasks in crowdsourcing, a research team led by Hailong Sun published their survey on 15 June 2026 in Frontiers of Computer Science co-published by Higher Education Press and Springer Nature.
As shown in Figure 1, the team introduced the quality control framework through task flow and answer flow, emphasizing the necessity of contextual dependencies and dynamic optimization in open-ended tasks, thereby providing intuitive support for method design and validation.
Beyond that, they proposed a two-tier framework to categorize related works. As shown in Figure 2, the first tier presents a comprehensive overview of the quality model, covering essential aspects including tasks, workers, answers, and the system. The second tier further refines this classification by breaking it down into more detailed categories: 'quality dimensions,' 'evaluation metrics,' and 'design decisions.' This breakdown provides deeper insights into the internal structure of the quality control model for each aspect.
This paper thoroughly investigate how these quality control methods are implemented in state-of-the-art works and discuss key challenges and pointed out that the potential future research directions includes leveraging Large Language Models (LLMs) to enhance answer generation and evaluation, and exploring cross-task generalization capabilities of the quality control methods.
Frontiers of Computer Science
Experimental study
Not applicable
Quality control in open-ended crowdsourcing: a survey
15-Jun-2026