HTML EN
PDF EN
FLIP
XML
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Revista de Arquitectura is an open access journal. More information...
Authors retain copyright and grant to the Revista de Arquitectura the right of first publication, which will be simultaneously subject to the Creative Commons (CC) BY-NC license.
Authors will sign a non-exclusive distribution license for the published version of the article by completing (RevArq FP03 Permission to Reproduce).
Self-archiving will comply with SHERPA/RoMEO guidelines and the Green classification.
To see in detail these guidelines, please consult...
Abstract
A performance assessment involves examinees creating a product or developing a process, which is evaluated by several raters. The Multi-faceted Rasch Measurement Model (MFRM), an extension of the Rasch Model, allows quantifying diverse attributes associated with measurement quality in this type of assessments, including the degree of inter-rater agreement (inter-rater reliability), which is an essential requirement for validity. Data from a performance test, currently applied for selection purposes in the undergraduate program of the School of Architecture at the University of Costa Rica (UCR), were analyzed with MFRM. Four data sets were used, from 2015 to 2018 test administrations, each one having between 600 and 800 applicants. Each examinee’s product was evaluated by three raters. The rater teams had between 12 and 15 members. The first three years showed a high degree of variability between raters’ severities, extending over 2 logits on the Rasch Scale. Modifications were introduced in the 2018 application, aiming to improve inter-rater reliability. The corresponding analyses showed a relevant decrease in the dispersions of raters’ severities, with a range of 1.09 logits. The study illustrates the benefits of the MFRM Model for analyzing rater data and improving the technical quality of a high- stakes performance assessment.
Keywords:
References
Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43(4), 561-573. https://www.springer.com/journal/11336
Eckes, T. (2011). Many- Facet Rash Measurement. In Grotjahn, R and Sigott, G (Eds.), Introduction to Many-Facet Rasch Measurement (2nd ed.). Peter Lang. https://www.researchgate.net/publication/228465956_Many-facet_Rasch_measurement
Hernández, O. (2015). Informe PH, Prueba de Habilidad 2014 - ingreso 2015. Escuela de Arquitectura-UCR. [2014 Skills Test Report for 2015 admission. Architecture School. University of Costa Rica]. https://issuu.com/olmanarq/docs/informe_ph-2014_arquis
Hernández, O. (2018). Informe PH, Prueba de Habilidad 2017 - ingreso 2018. Escuela de Arquitectura-UCR. [2017 Skills Test Report for 2018 admission. Architecture School. University of Costa Rica]. https://issuu.com/olmanarq/docs/informe_ph-2017
Lane, S. & Stone, C.A. (2006). Performance Assessment. In R. L. Brennan (Ed.), Educational Measurement (pp. 387-431). Praeger.
Linacre, J. M. (1989). Many-facet Rasch measurement. MESA Press.
Linacre, J. M. & Wright, B. D. (2002). Construction of measures from many-facet data. Journal of Applied Measurement, 3(4), 486-512. http://jampress.org/
Linacre, J. M. (2010). A user’s guide to Facets: Rasch model computer programs. Winsteps.
Linacre, J. M. (2015). Facet Rasch Measurement computer program (Version 3.71.3). Winsteps.
Martínez, R. (2010). La evaluación del desempeño. [Performance assessment]. Papeles del Psicólogo, 31(1), 85-96. http://www.papelesdelpsicologo.es/pdf/1799.pdf
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174. https://doi.org/10.1007/BF02296272
Myford, C. M. & Wolfe, E. W. (2004). Detecting and Measuring Rater Effects Using Many-Facet Rasch Measurement: Part I. In E. V. Smith & R. M. Smith (Eds.), Introduction to Rasch Measurement (pp. 460-517). JAM Press.
Prieto, G. (2015). Análisis de un test de desempeño en expresión escrita mediante el modelo de MFRM. [Analysis of a performance test in written expression using the MFRM model]. Actualidades en Psicología, 29(119), 03-19. http://dx.doi.org/10.15517/ap.v29i119.19822
Prieto, G. & Nieto, E. (2014). Analysis of rater severity on written expression exam using Many Faceted Rasch Measurement. Psicológica, 35(2), 385-397. https://www.researchgate.net/publication/288462542_Analysis_of_rater_severity_on_written_expression_exam_using_Many_Faceted_Rasch_Measurement
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. MESA Press. https://doi.org/10.1177/014662168100500413
Reference by
