The Wild Animal Modeling Wiki
Testing significance of random effects
Summary
This page provides an overview of how to test the significance of random effects using log-likelihood ratio tests (in ASReml, ASReml-R, WOMBAT). To do this, you compare the log-likelihoods of models with and without the appropriate random effect - if removing the random effect causes a large enough drop in log-likelihood then one can say the effect is statistically significant. Commonly we assume that a test statistic equal to twice the difference in log-lkelihoods is distributed as chisquare with DF equal to the number of additional parameters in the more complex model (1 if testng a single random effect). There is a sensible argument that this is overly conservative and so variations on this are also used.
It is also possible to get an idea of if a random effect will be significant by examining the parameter estimate and standard error of the variance component. As a rule of thumb, if the estimate is more than twice the standard error then it is likely that the estimate will be significant, although this should always be checked analytically.
MCMCglmm is not likelihood based, instead DIC can be used for model selection.
Table of contents
Definitions
This code shows how to test the significance of the random effect 'RANDOMEFFECT' using log-likelihood ratio tests.
Response variable: Size
Fixed effects: Intercept (mu) & SEX
Random effects: Additive genetic variance & RANDOMEFFECT
Data containing phenotypic information: phenotypicdata
Data containing pedigree data:pedigreedata
Sample Code
ASReml
Fit a model with and without a particular random effect. Then use log likelihoods reported in the primary results file to perform a ratio test.ASReml analysis of size ANIMAL !P SIZE SEX !A #sex as a factor pedigreedata.ped !skip 1 phenotypicdata.dat !skip 1 !dopart 1 #change part to be run required !part 1 SIZE ~ mu SEX !r ANIMAL RANDOMEFFECT !part 2 SIZE ~ mu SEX !r ANIMAL
ASReml-R
model1<-asreml(fixed=SIZE~1+SEX ,random=~ped(ANIMAL,var=T,init=1)+RANDOMEFFECT ,data=phenotypicdata ,ginverse=list(ANIMAL=ainv), na.method.X="omit', na.method.Y="omit') model2<-asreml(fixed=SIZE~1+SEX ,random=~ped(ANIMAL,var=T,init=1) ,data=phenotypicdata ,ginverse=list(ANIMAL=ainv), na.method.X="omit', na.method.Y="omit') #calculate the chi-squared stat for the log-likelihood ratio test 2*(model1$loglik-model2$loglik) #calculate the associated significance 1-pchisq(2*(model1$loglik-model2$loglik),1)
MCMCglmm
We cannot use the Highest Posterior Density intervals (i.e. the HPDinterval() R function) to assess the statistical support for random effects. This is because random effects are constrained to positive values. However, by default, MCMCglmm calculates the Deviance Information Criterion (DIC), and this can be used to assess whether increases in model fit justify the inclusion of any particular effect:
model1.3$DIC model1.4$DIC
WOMBAT
To test the significance of a random effect (here calledRANDOMEFFECT
), we compare models with and without the random effect of interest, and compare the final log-likelihoods as reported in SumEstimates.out
.Fitting a model with
RANDOMEFFECT
usingMODEL FIX SEX RAN ANIMAL NRM RAN RANDOMEFFECT TRAIT SIZE END
And compare the log-likelihood of this model to a model without
RANDOMEFFECT
usingMODEL FIX SEX RAN ANIMAL NRM TRAIT SIZE END