Investigating expert vs rater consensus agreement, inter and intra-rater reliability of two fundamental movement skills for the locomotor subscale of the FG-COMPASS

Abstract

The Furtado-Gallagher Computerized Observational Movement Pattern Assessment System (FG-COMPASS) is an observational assessment tool that uses sequential decisions to assess fundamental movement skill proficiency. The current version of the test has three locomotor and five manipulative skills. This study aimed to assess expert-rater agreement and inter/intra rater reliability of two new scales to be added to the locomotor subtest. This study was divided into two phases. In Phase I, 60 children between the ages of 5 and 10 were filmed performing the skills of gallop and vertical jump. An expert in motor behavior classified the videotapes using the newly created rating scales. Next, eight video clips were selected for training purposes and 24 video clips for testing purposes. In Phase II, 30 undergraduate and graduate students served as raters and underwent a training session prior the testing session. Participants were instructed not to classify the video clips based on the apparent age of the children as skill levels were distributed across all age levels. Further, to avoid guessing, participants were not told how many videos of each level they would be rating. Unlike the training sessions, participants did not receive feedback during testing sessions. Weighted kappa (Kw) was used to analyze the data. The results suggested a “very good” agreement between expert and rater consensus for vertical jump (Kw = .96) and gallop (Kw=.89). Inter-rater reliability resulted in “very good” agreement for vertical jump (Kw=.92) and a “good” agreement for gallop (Kw=.78). Intra-rater reliability resulted in a “very good” agreement for both vertical jump (Kw=.96) and gallop (Kw=.85). Future studies should (a) assess the replicability of the results in live settings and (b) test the feasibility of the mobile app and its web-based training tool.

comments powered by Disqus