The Classic Solomon Four-Group Analysis

Why preserve the classic analysis?

The Solomon four-group design has a long methodological history. Earlier approaches analyzed different pieces of the design sequentially, using tests that came to be described as Tests A through I.

solomonR reproduces this historical workflow for two reasons:

1. it remains useful for teaching how the Solomon design developed; and
2. it allows researchers to reproduce analyses based on historical recommendations.

The classic workflow should not be interpreted as the default modern analysis merely because it is historically important. solomonR therefore distinguishes the historical procedure from the package’s modern model-based and randomization-based approaches.

The implementation draws especially on the historical discussions of Huck and Sandler (1973), Walton Braver and Braver (1988), and later simulation work examining the operating characteristics of conditional Solomon procedures.

The four groups

A Solomon four-group design contains:

Groups 1 and 2 permit a pretest-adjusted treatment comparison.

Groups 3 and 4 provide a treatment comparison that cannot itself have been influenced by administration of the pretest.

Together, the four groups permit investigation of both the treatment effect and possible pretest sensitization.

Example data

solomonR includes an example dataset with all four Solomon groups.

library(solomonR)

data(solomon_example)

demo_preview <- head(
  solomon_example,
  6
)

demo_preview$y_post <- round(
  demo_preview$y_post,
  2
)

demo_preview$y_pre <- round(
  demo_preview$y_pre,
  2
)

knitr::kable(
  demo_preview,
  align = "rrrr"
)

The pretest variable is structurally absent for participants who were not pretested. Those missing values are therefore part of the design, not ordinary missing observations.

Running the historical analysis

The complete historical analysis is available through fit_solomon_classic():

classic <- with(
  solomon_example,
  fit_solomon_classic(
    y_post,
    treat,
    pretested,
    y_pre
  )
)

classic
#> Classic Solomon analysis (historical teaching workflow)
#> -------------------------------------------------------
#> Selected pretested-group method: Test E (ancova)
#> Historical decision path (1988 flow): A -> D -> E -> H -> I
#> 
#> Historical Tests A-I
#> --------------------
#> All tests are shown below. Tests marked [PATH] were reached by
#> the historical decision sequence for these data.
#> 
#> [PATH] Test A: Pretest x Treatment interaction               F(1, 116) = 0.29, p = 0.589
#>        Test B: Treatment effect among pretested groups       F(1, 116) = 0.49, p = 0.486
#>        Test C: Treatment effect among unpretested groups     F(1, 116) = 2.14, p = 0.146
#> [PATH] Test D: Treatment main effect                         F(1, 116) = 2.34, p = 0.129
#> [PATH] Test E: ANCOVA treatment effect                       F(1, 57) = 0.59, p = 0.444
#>        Test F: Gain-score treatment effect                   F(1, 58) = 0.46, p = 0.499
#>        Test G: Repeated-measures Treatment x Time interaction F(1, 58) = 0.46, p = 0.499
#> [PATH] Test H: Posttest-only treatment effect                t(58) = 1.66, p = 0.103
#> [PATH] Test I: Walton Braver & Braver (1988) Stouffer combination Z = 1.69, p = 0.090 [E + H (ANCOVA + posttest-only)]
#> 
#> Historical interpretation
#> -------------------------
#> Historical pathway: no treatment test in the selected A-I sequence reaches the specified alpha level. 
#> 
#> Groups 3-4 effect size: Hedges g = 0.423, 95% CI [-0.086, 0.938] (noncentral t)
#> 
#> History/maturation check (historical; Mai et al., 2020)
#>   O6 - O1: control posttest vs. treated-group pretest: difference = 1.467, t(58) = 0.54, p = 0.590
#>   O6 - O3: control posttest vs. control-group pretest: difference = 1.533, t(58) = 0.61, p = 0.543
#> 
#> Caution: Test I, the Walton Braver & Braver (1988) Stouffer combination, is
#> reproduced for historical teaching and replication. Later simulation
#> work (see Sawilowsky et al., 1994) raised concerns about Type I error
#> for the conditional meta-analytic sequence; it is not the default
#> modern inferential recommendation in solomonR.

The printed output reports all of the historical tests but marks the tests actually reached by the historical decision sequence with [PATH].

This distinction is important.

A test can be calculated for teaching or inspection without having been reached under the historical conditional procedure.

plot_classic_flow() draws the conditional sequence as a decision tree and highlights the route these data took, with the p-value of each test reached:

plot_classic_flow(classic)

The caption repeats the caution discussed below: the sequence ending in Test I is kept for teaching and replication, not as a recommended analysis.

Tests A-D: the four-group posttest model

The first part of the classic workflow is based on the four-group posttest model.

classic$tests$A$result
classic$tests$D$result

Tests E-G: analyses within the pretested groups

Historical approaches proposed several ways to make use of the baseline measurement available in Groups 1 and 2.

classic$tests$E$result

classic$tests$F$result

with(solomon_example, plot_solomon_change(y_post, treat, pretested, y_pre))

classic$tests$G$result

fit_solomon_classic(
  y_post,
  treat,
  pretested,
  y_pre,
  pretested_test = "gain"
)

Test H: posttest-only comparison

Test H compares Groups 3 and 4, which did not receive the pretest.

classic$tests$H$result

Because these groups were never pretested, this comparison provides a direct treatment contrast uncontaminated by administration of a baseline measure.

solomonR also reports Hedges’ \(g\) for this posttest-only comparison when available.

cat(
  sprintf(
    "Hedges' g = %.2f, 95%% CI [%.2f, %.2f]",
    classic$g_post["g"],
    classic$g_post["lower"],
    classic$g_post["upper"]
  )
)
#> Hedges' g = 0.42, 95% CI [-0.09, 0.94]

Test I: Walton Braver & Braver (1988) Stouffer combination

Walton Braver and Braver (1988) proposed combining evidence from the pretested and unpretested treatment comparisons using Stouffer’s method.

When requested, solomonR reproduces this historical analysis using directional one-tailed p-values aligned with the same treatment direction.

cat(
  sprintf(
    "Stouffer Z = %.2f, p = %s",
    classic$tests$I$result$z,
    if (
      classic$tests$I$result$p.value < .001
    ) {
      "< .001"
    } else {
      sub(
        "^0",
        "",
        sprintf(
          "%.3f",
          classic$tests$I$result$p.value
        )
      )
    }
  )
)
#> Stouffer Z = 1.69, p = .090

This detail matters. A directional one-tailed p-value is not obtained correctly by mechanically dividing every two-sided p-value by two without considering the sign of the effect. The package’s replication of the published error rates found that the published simulations appear to have combined the tests without regard to sign. solomonR refers the combined z to its two-tailed p-value, as Walton Braver and Braver report it in their worked example (1988, p. 153). That is why they report far fewer Test I rejections than the procedure as defined produces (see Historical Tests: Replicating the Published Error Rates).

Versions of the decision sequence

The sequence changed after it was published, and fit_solomon_classic() reproduces each version through its flow argument.

• 1988, the default. In the original sequence, testing stops at the first significant test among D, the selected pretested-groups test, and H; Test I is reached only when all of them are nonsignificant (Walton Braver & Braver, 1988, pp. 151–153).
• 1990. Replying to Sawilowsky and Markman (1990a), the authors agreed that using the meta-analysis selectively introduces selection bias. Once Tests A and D are nonsignificant, the analyst “should complete all the tests through Test I” (Braver & Walton Braver, 1990, p. 322), and Test I is regarded as the most definitive test.
• 1995. A later revision removed Test D. It is unpublished and known from Sawilowsky (1996, p. 2), so it is cited as Walton Braver and Braver (1995, as cited in Sawilowsky, 1996). The remaining tests follow the 1988 rule, as in Sawilowsky’s simulation of the revised sequence.

classic_1990 <- with(
  solomon_example,
  fit_solomon_classic(y_post, treat, pretested, y_pre, flow = "1990")
)

classic_1990$path_string
#> [1] "A -> D -> E -> H -> I"
classic_1990$conclusion
#> [1] "Historical pathway (1990 amendment): Tests A and D are nonsignificant, so every test through Test I is run and Test I is regarded as the most definitive. Test I does not reach the specified alpha level (Test E p = 0.444, Test H p = 0.103, Test I p = 0.090). Interpret this in light of later Type I error critiques."

The tests themselves are identical in every version; only the path and the conclusion differ. For the 1995 version, alpha_allocation applies the test-wise significance levels that Sawilowsky (1996, Table 4) proposed to control the experiment-wise error rate:

classic_1995 <- with(
  solomon_example,
  fit_solomon_classic(y_post, treat, pretested, y_pre, flow = "1995",
                      alpha_allocation = "method2_conservative")
)

classic_1995$path_string
#> [1] "A -> E -> H -> I"

History and maturation

The four-group design also allows a check that does not involve the treatment. The unpretested control group’s posttest (O6) was measured at the same time as the other posttests, but its participants received neither the pretest nor the treatment. Comparing it with the pretests of the pretested groups (O1 and O3), which were measured before any treatment, estimates the combined effect of history and maturation between the two occasions. Mai et al. (2020, p. 8) report these comparisons as independent-samples t tests, which fit_solomon_classic() returns as history:

classic$history
#>                                            comparison estimate statistic df
#> 1 O6 - O1: control posttest vs. treated-group pretest 1.466667 0.5415318 58
#> 2 O6 - O3: control posttest vs. control-group pretest 1.533333 0.6121767 58
#>     p.value  conf.low conf.high
#> 1 0.5902153 -3.954718  6.888051
#> 2 0.5428135 -3.480416  6.547082

The check assumes random assignment and a measure that is comparable at the two occasions.

Why Test I is treated cautiously

The Stouffer procedure is included because it is part of the history of the Solomon design and may be required when reproducing historical analyses.

It is not the default modern recommendation in solomonR.

Later simulation work showed that conditional analysis sequences can have undesirable experiment-wise Type I error properties. In particular, choosing later tests based on the statistical significance of earlier tests changes the operating characteristics of the overall procedure. Sawilowsky et al. (1994) found that the 1988 sequence falsely declared an effect about 14% of the time at a nominal 5% per test.

The package’s own replication found similar rates for the sequence as Walton Braver and Braver define it. With Test I judged two-tailed, as in their worked example, the rate was 13.5% to 13.7% for the 1988 and 1995 versions, and Sawilowsky’s (1996) alpha allocations stayed above their targets.

For contemporary applied work, researchers should therefore consider a single prespecified model or a randomization-based analysis rather than automatically following a significance-driven testing tree.

See the unified GLM vignette for the primary modern observed-variable workflow:

vignette("glm-solomon", package = "solomonR")

Historical analysis versus modern analysis

The two approaches answer related questions but serve different purposes.

The historical procedure is therefore preserved rather than erased, while modern methods are available alongside it. A History of the Solomon Design and Its Analysis tells how each version arose and how the debate over it unfolded.

All works cited in solomonR are listed, with notes on how the package uses them, on the References page.

References

Braver, S. L., & Walton Braver, M. C. (1990). Meta-analysis for Solomon four-group designs reconsidered: A reply to Sawilowsky and Markman. Perceptual and Motor Skills, 71(1), 321–322. https://doi.org/10.2466/pms.1990.71.1.321

Huck, S. W., & Sandler, H. M. (1973). A note on the Solomon 4-group design: Appropriate statistical analyses. The Journal of Experimental Education, 42(2), 54–55. https://doi.org/10.1080/00220973.1973.11011460

Mai, N. N., Takahashi, Y., & Oo, M. M. (2020). Testing the effectiveness of transfer interventions using Solomon four-group designs. Education Sciences, 10(4), Article 92. https://doi.org/10.3390/educsci10040092

Sawilowsky, S. S. (1996, June 23). Controlling experiment-wise Type I error of meta-analysis in the Solomon four-group design [Paper presentation]. First International Conference on Multiple Comparisons, Tel Aviv, Israel. https://digitalcommons.wayne.edu/coe_tbf/29/

Sawilowsky, S. S., Kelley, D. L., Blair, R. C., & Markman, B. S. (1994). Meta-analysis and the Solomon four-group design. The Journal of Experimental Education, 62(4), 361–376. https://doi.org/10.1080/00220973.1994.9944140

Sawilowsky, S. S., & Markman, B. S. (1990a). Another look at the power of meta-analysis in the Solomon four-group design. Perceptual and Motor Skills, 71(1), 177–178. https://doi.org/10.2466/pms.1990.71.1.177

Walton Braver, M. C., & Braver, S. L. (1988). Statistical treatment of the Solomon four-group design: A meta-analytic approach. Psychological Bulletin, 104(1), 150–154. https://doi.org/10.1037/0033-2909.104.1.150