Exercise 1: Language learning class
A teacher designed three sets of class materials (sets A, B, and C) for a language learning class. The teacher wants to know whether students’ learning outcome differ depending on the set of material used, and/or whether they receive a 10-min revision session after the class. 60 students participated in a round of trial classes: 10 students took part in the class with material set A, and received revision after class; 10 students also took part in the class with material set A, but did not receive revision; and so on.
This study is a 3 x 2 design with Material set (A vs. B vs. C) and Revision (revision vs. no revision).
- Download the data Here.
- Read the data into a data frame.
data1 <- read.csv('language-class.csv', header = TRUE)- Determine which statistical test to use to answer the teacher’s question.
- How many factors are there in the study?
- Are the factor(s) between- or within-subject?
- How many levels does each factor have?
There are two factors in this study, Material set and Revision. Each factor has two levels. Both factors are between-subjects. Therefore the suitable test is the two-way between-subject ANOVA.
- Visualise the data using your preferred type of plot and get descriptive statistics.
library(ggplot2)
data1_groupmean <- aggregate(score ~ material+revision, data = data1, FUN = mean)
ggplot(data1_groupmean, aes(x = material, y = score, group=revision, color = revision)) +
geom_point() +
geom_line() +
ylim(c(50,100)) +
theme_light() 
library(psych)
describeBy(score ~ material*revision, data=data1)##
## Descriptive statistics by group
## material: A
## revision: no_revision
## vars n mean sd median trimmed mad min max range skew kurtosis se
## score 1 10 62.3 11.14 61.5 62.62 12.6 44 78 34 -0.3 -1.24 3.52
## ------------------------------------------------------------
## material: B
## revision: no_revision
## vars n mean sd median trimmed mad min max range skew kurtosis se
## score 1 10 77.5 9.41 77.5 77.12 8.9 64 94 30 0.24 -1.2 2.97
## ------------------------------------------------------------
## material: C
## revision: no_revision
## vars n mean sd median trimmed mad min max range skew kurtosis se
## score 1 10 84.4 8.07 86.5 85 7.41 69 95 26 -0.56 -0.99 2.55
## ------------------------------------------------------------
## material: A
## revision: revision
## vars n mean sd median trimmed mad min max range skew kurtosis se
## score 1 10 78.5 9.79 80 78.75 11.86 63 92 29 -0.15 -1.53 3.1
## ------------------------------------------------------------
## material: B
## revision: revision
## vars n mean sd median trimmed mad min max range skew kurtosis se
## score 1 10 76.4 11.22 73.5 76.5 13.34 59 93 34 0.03 -1.43 3.55
## ------------------------------------------------------------
## material: C
## revision: revision
## vars n mean sd median trimmed mad min max range skew kurtosis se
## score 1 10 91 6.31 89 91 6.67 82 100 18 0.23 -1.6 1.99- Check whether the data meet the assumptions and run the test.
data1_result <- aov(score ~ material*revision, data = data1)
summary(data1_result)## Df Sum Sq Mean Sq F value Pr(>F)
## material 2 3052 1525.9 16.982 1.9e-06 ***
## revision 1 785 784.8 8.735 0.00462 **
## material:revision 2 751 375.6 4.180 0.02051 *
## Residuals 54 4852 89.8
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1data1_residuals <- residuals(object = data1_result)
shapiro.test(data1_residuals)##
## Shapiro-Wilk normality test
##
## data: data1_residuals
## W = 0.97604, p-value = 0.2846library(car)
leveneTest(score ~ material*revision, data = data1)## Levene's Test for Homogeneity of Variance (center = median)
## Df F value Pr(>F)
## group 5 0.5797 0.7153
## 54Different ways to do pair-wise comparisons: In the two-way between-subject example in the handout, I simply did pairwise comparisons between each possible combination of gropus. This is totally ok, like this:
# Two-way between-subject ANOVA post-hocs
# This is the method I showed you in the handout, which does all possible pair-wise comparisons
data1$group <- paste('set_', data1$material, '_', data1$revision, sep = '')
data1$group <- as.factor(data1$group)
pairwise.t.test(data1$score, data1$group, p.adjust.method = 'bonferroni')##
## Pairwise comparisons using t tests with pooled SD
##
## data: data1$score and data1$group
##
## set_A_no_revision set_A_revision set_B_no_revision
## set_A_revision 0.0052 - -
## set_B_no_revision 0.0108 1.0000 -
## set_B_revision 0.0238 1.0000 1.0000
## set_C_no_revision 4.5e-05 1.0000 1.0000
## set_C_revision 1.5e-07 0.0706 0.0361
## set_B_revision set_C_no_revision
## set_A_revision - -
## set_B_no_revision - -
## set_B_revision - -
## set_C_no_revision 0.9677 -
## set_C_revision 0.0167 1.0000
##
## P value adjustment method: bonferroniHowever we can also approach this in a more careful way: you might wonder what’s the point in comparisons such as set_A_revision vs. set_B_no-revision? These comparisons don’t sound very informative to the research question of this study.
Can’t I do something that simply lets me know whether for each set of materials, there is an effect of revision or not (I only care about set_A_revision vs. set_A_no-revision; set_B_revision vs. set_B_no-revision; etc.)? You can, of course, and in fact in the Mixed-ANOVA example in the handout, I did something very similar. When you have a very clear idea of what comparisons will be useful for your research question, then all you need are those comparisons you planned to do.
Let’s say we only want to know whether, for each set of materials, revision significantly changes students’ performance:
# Let's look at whether revision has a significant effect for each material set.
library(rstatix)
data1_pt <- data1 %>%
group_by(material) %>%
rstatix::pairwise_t_test(score ~ revision, p.adjust.method = "bonferroni")
data1_pt## # A tibble: 3 × 10
## material .y. group1 group2 n1 n2 p p.sig…¹ p.adj p.adj…²
## * <chr> <chr> <chr> <chr> <int> <int> <dbl> <chr> <dbl> <chr>
## 1 A score no_revision revisi… 10 10 0.00282 ** 0.00282 **
## 2 B score no_revision revisi… 10 10 0.815 ns 0.815 ns
## 3 C score no_revision revisi… 10 10 0.0566 ns 0.0566 ns
## # … with abbreviated variable names ¹p.signif, ²p.adj.signifYou might notice that the p-values are slightly different, this is because we are not doing the ‘useless’ comparisons, so we don’t need to adjust the p-values that much.
Alternatively (or in addition), if you care about whether the sets of materials differ when student get the revision session, and whether the sets of materials differ when students don’t get the revision session, you can do something like this:
# we can also look at whether the sets of materials differ, when students revised vs. when they didn't revise.
data1_pt2 <- data1 %>%
group_by(revision) %>%
rstatix::pairwise_t_test(score ~ material, p.adjust.method = "bonferroni")
data1_pt2## # A tibble: 6 × 10
## revision .y. group1 group2 n1 n2 p p.signif p.adj p.adj…¹
## * <chr> <chr> <chr> <chr> <int> <int> <dbl> <chr> <dbl> <chr>
## 1 no_revision score A B 10 10 0.0015 ** 4.5 e-3 **
## 2 no_revision score A C 10 10 0.0000211 **** 6.33e-5 ****
## 3 no_revision score B C 10 10 0.12 ns 3.61e-1 ns
## 4 revision score A B 10 10 0.619 ns 1 e+0 ns
## 5 revision score A C 10 10 0.00583 ** 1.75e-2 *
## 6 revision score B C 10 10 0.00165 ** 4.94e-3 **
## # … with abbreviated variable name ¹p.adj.signifIt’s up to you how you want to approach your pair-wise comparisons.
- Find out the sizes of the effects.
library(lsr)
etaSquared(data1_result)## eta.sq eta.sq.part
## material 0.32328529 0.3861152
## revision 0.08314044 0.1392329
## material:revision 0.07958275 0.1340738- Report your findings in a short paragraph.
Two-way between-subjects ANOVA revealed significant main effects of Material set (F(2, 54)=16.98, p<0.001, \(\eta^2\)=0.39) and Revision (F(1, 54)=8.74, p=0.005, \(\eta^2\)=0.14), as well as a significant interaction (F(2, 54)=4.18, p=0.02, \(\eta^2\)=0.13). Post-hoc pair-wise t-tests with Bonferroni correction revealed that the effect of Revision was only sigificant for Material set A (p=0.003), but not the other two sets of materials. (Alternatively or in addition: Without revision, Material sets B and C were significantly more effective than Material set A (both p’s<0.01), while there was no significant difference between sets B and C. With revision, Material set A became as effective as Material set B; however, set C was still significantly more effective than both A and B (both p’s<0.01). )
6. Remember we need by-participant (or by-item) means rather than raw data. Convert raw data to by-participant means. Hint: 
6. Run the statistical test.