Analysis of Benefits

In intervention research we:

Ø     Randomly assign subjects;

Ø     Intervene with the Treatment group;

Ø     Record test scores:

o      after the intervention (Post scores)

o      and from before the intervention as well (Pre scores), for comparison.    

 

So, how much benefit has the treatment contributed?

 

The Post scores are obviously the Outcome measure, as these show student performance after the intervention and should reflect the benefits of the intervention.  Specifically, the Treatment group should show improved performance, as compared to the Control group.

 

Yet prior qualities also contribute to the Outcome, so we have a second factor to consider.  The two principal contributing factors are: 

Ø     Intervention impact;  

Ø     Prior qualities, including prior academic achievement and also aptitude and social factors that would influence the outcome measures.

The task then is to separate the two. 

 

The Pre-Intervention scores are our available measure of prior qualities which we would expect to contribute to the Outcome. 

 

How much do the Pre scores contribute to the Outcome variance?   As an excellent estimate, simply correlate the Pre to Post scores.  The contribution size is the square of the attained correlation coefficient.  A Pre–Post correlation of r=.87 means that the Pre scores account for 75% of the Post variance, while a correlation of r=.50 says that the Pre scores account for 25% of the variance.   

 

Since we can accurately estimate the contribution of the Pre scores, we can remove it from the Post scores.  We thus attain a close estimate of what the Outcome would be if all students had started with the same initial scores.  

 

Analysis of Covariance (ANCOVA):  As a practical matter, such contributions are ordinarily calculated by multiple regression statistics. 

Ø     In the Excel spreadsheet, in three adjacent columns, enter for each subject:

1.    Intervention condition (use 1=Treatment, 0=Control)

2.    Pre scores (the covariate)  

3.    Post scores (the outcome)  

Ø     Open the regression program.

o      Install the Analysis ToolPak (press F1; type in Analysis ToolPak; follow instructions).   

o      Select the program (click on Tools, Data Analysis; Regression)

o      Select the Post scores as the "Y" or "Dependent" variable, and the Intervention and Pre scores together as the "X" or "Independent" variables.

Ø     The program then calculates Beta coefficients and statistical significance for the contributing factors.  The square of the Beta coefficients provide an accurate estimate of the Outcome variance accounted for by Pre scores and by the Intervention. 

 

Alternate analyses can be used, including simple change scores.  

 

Change Scores.  We can subtract the Pre scores from the Post scores, to calculate a simple change score.  The problem here is that the Pre scores may contribute a considerable amount to the Post scores, or contribute very little.  So far as the Pre scores are only loosely related to outcome, subtracting the full score introduces unnecessary error. 

 

See:  Analysis of Covariance (ANCOVA) from StatSoft

 

Return to Research page