Testing significance of random effects

Summary

This page provides an overview of how to test the significance of random effects using log-likelihood ratio tests (in ASReml, ASReml-R, WOMBAT). To do this, you compare the log-likelihoods of models with and without the appropriate random effect - if removing the random effect causes a large enough drop in log-likelihood then one can say the effect is statistically significant. Commonly we assume that a test statistic equal to twice the difference in log-lkelihoods is distributed as chisquare with DF equal to the number of additional parameters in the more complex model (1 if testng a single random effect). There is a sensible argument that this is overly conservative and so variations on this are also used.

It is also possible to get an idea of if a random effect will be significant by examining the parameter estimate and standard error of the variance component. As a rule of thumb, if the estimate is more than twice the standard error then it is likely that the estimate will be significant, although this should always be checked analytically.

MCMCglmm is not likelihood based, instead DIC can be used for model selection.

Table of contents

• Summary
• Definitions
• Sample Code
  • ASReml
  • ASReml-R
  • MCMCglmm
  • WOMBAT

Definitions

This code shows how to test the significance of the random effect 'RANDOMEFFECT' using log-likelihood ratio tests.

Response variable: Size
Fixed effects: Intercept (mu) & SEX
Random effects: Additive genetic variance & RANDOMEFFECT
Data containing phenotypic information: phenotypicdata
Data containing pedigree data:pedigreedata

Sample Code

ASReml

Fit a model with and without a particular random effect. Then use log likelihoods reported in the primary results file to perform a ratio test.

ASReml analysis of size  						 

   ANIMAL       !P 
   SIZE
   SEX          !A  #sex as a factor

pedigreedata.ped      !skip 1   
phenotypicdata.dat    !skip 1    !dopart 1 #change part to be run required
 
!part 1
SIZE ~ mu SEX !r ANIMAL  RANDOMEFFECT

!part 2
SIZE ~ mu SEX !r ANIMAL

ASReml-R

model1<-asreml(fixed=SIZE~1+SEX
           ,random=~ped(ANIMAL,var=T,init=1)+RANDOMEFFECT
           ,data=phenotypicdata
           ,ginverse=list(ANIMAL=ainv), na.method.X="omit', na.method.Y="omit')

model2<-asreml(fixed=SIZE~1+SEX
           ,random=~ped(ANIMAL,var=T,init=1)
           ,data=phenotypicdata
           ,ginverse=list(ANIMAL=ainv), na.method.X="omit', na.method.Y="omit')

#calculate the chi-squared stat for the log-likelihood ratio test
2*(model1$loglik-model2$loglik) 

#calculate the associated significance
1-pchisq(2*(model1$loglik-model2$loglik),1)

MCMCglmm

We cannot use the Highest Posterior Density intervals (i.e. the HPDinterval() R function) to assess the statistical support for random effects. This is because random effects are constrained to positive values. However, by default, MCMCglmm calculates the Deviance Information Criterion (DIC), and this can be used to assess whether increases in model fit justify the inclusion of any particular effect:

model1.3$DIC

model1.4$DIC

WOMBAT

To test the significance of a random effect (here called RANDOMEFFECT), we compare models with and without the random effect of interest, and compare the final log-likelihoods as reported in SumEstimates.out.

Fitting a model with RANDOMEFFECT using

MODEL
  FIX SEX
  RAN ANIMAL NRM
  RAN RANDOMEFFECT
  TRAIT SIZE
END

And compare the log-likelihood of this model to a model without RANDOMEFFECT using

MODEL
  FIX SEX
  RAN ANIMAL NRM
  TRAIT SIZE
END

---