Análise quantitativa da concordância entre pares de avaliadores de recursos digitais educacionais: um estudo de caso com dados do repositório Merlot

Resumo

Community evaluation (through comments and ratings) is one of the most common strategies adopted by learning resources repositories to assessing quality of the resources. In these cases, the final rating of a given resource is calculated by the means of all ratings given by this community. Despite its wide adoption and consolidation, this strategy normally hides the distribution of the ratings and the nuances of the assessments given from users with different backgrounds and disciplines of interest. The present paper analyzes the inter-rater agreement between pairs of reviewers of the repository Merlot, using the Intra-class correlation coefficient (ICC). The results show reviewers with different disciplines of interest and with different professional background tend to disagree more about the quality of the resources.