Solomon Methods: History, Recommendations, and Extensions

Four-group analysis of posttest scores (Tests A-D)

Label: historical procedure.

• Estimates: all four contrasts from a two-by-two model of posttest scores (Test A: Pretest x Treatment; B and C: the simple treatment effects; D: the equal-weighted average).
• Assumptions: independent observations, normally distributed errors, and equal variances across the four groups.
• Limitations: the pretest scores in Groups 1 and 2 are ignored, which gives up precision. Campbell and Stanley (1963/1966, p. 25) suggested analyzing the posttest scores with a two-by-two analysis of variance and, if the main and interactive effects of pretesting are negligible, an analysis of covariance of the pretested groups’ posttests with the pretest as covariate. Choosing later tests according to earlier significance tests makes the overall procedure conditional.
• Sources: Campbell and Stanley (1963/1966); Huck and Sandler (1973).

Analyses within the pretested groups (Tests E-G)

Label: historical procedure.

• Estimates: Treatment | pretested, using the pretest information. Test E is analysis of covariance (ANCOVA), Test F compares gain scores, and Test G is the two-occasion repeated-measures interaction, which is algebraically equivalent to Test F.
• Assumptions: for ANCOVA, a common pretest-posttest slope in both groups; a gain score implicitly assumes that slope equals one.
• Limitations: only Groups 1 and 2 are used. In randomized studies, ANCOVA has more power than change scores; in nonrandomized comparisons, ANCOVA can be more biased (Van Breukelen, 2006). Because Solomon groups are randomized, ANCOVA is generally the stronger choice.
• Sources: Huck and Sandler (1973); Van Breukelen (2006).

Posttest-only comparison (Test H)

Label: historical procedure.

• Estimates: Treatment | unpretested from Groups 3 and 4.
• Assumptions: independent observations, normal errors, equal variances in the two groups.
• Limitations: half of the sample is ignored.
• Sources: Huck and Sandler (1973).

Meta-analytic combination (Test I)

Label: historical procedure.

• Estimates: nothing directly. Test I combines one-tailed p-values from the pretested and unpretested treatment comparisons with Stouffer’s method (Stouffer et al., 1949), as proposed by Walton Braver and Braver (1988). The combined z is judged by its two-tailed p-value, as in their worked example (p. 153).
• Assumptions: both tests address the same directional hypothesis, and the combination is specified in advance.
• Limitations: in a Monte Carlo study, the conditional sequence that ends in Test I had an experiment-wise Type I error rate nearly three times the nominal alpha, and traditional procedures used without conditioning were more powerful (Sawilowsky et al., 1994). The proposal was debated in published exchanges between the two research groups. solomonR reproduces Test I for replication but does not recommend it as a default analysis.
• Sources: Walton Braver and Braver (1988); Sawilowsky et al. (1994).

One prespecified analysis with explicit contrasts

Label: contemporary recommendation.

Rather than following a sequence of significance-driven decisions, contemporary practice specifies the analysis and its target quantities in advance (Lundberg et al., 2021). In randomized experiments, adjusting for a baseline covariate such as the pretest is justified for estimating treatment effects, and heteroskedasticity-robust standard errors protect that inference (Lin, 2013).

Heteroskedasticity-consistent and small-sample inference

Label: contemporary recommendation.

• HC3 standard errors are recommended as a routine choice, especially below about 250 observations (MacKinnon & White, 1985; Long & Ervin, 2000; Hayes & Cai, 2007). fit_solomon_glm() uses HC3 by default.
• t reference distributions with residual degrees of freedom are used for Gaussian models. Normal-reference robust intervals can under-cover in small samples (Imbens & Kolesár, 2016), and t with n - p degrees of freedom is the choice evaluated in recent guidance for behavioral research (Rajh-Weber et al., 2025).
• Clustered data use the CR2 estimator with Satterthwaite degrees of freedom (Bell & McCaffrey, 2002; Pustejovsky & Tipton, 2018).

Randomization inference

Label: contemporary recommendation.

• Tests: perm_solomon() permutes treatment labels within pretest conditions, which reproduces how treatment was assigned. Its p-value is valid for the sharp null hypothesis of no effect for any participant, and the +1 correction keeps it from being zero (Phipson & Smyth, 2010). Studentized statistics make such tests robust when only an average effect is hypothesized to be zero (DiCiccio & Romano, 2017; Wu & Ding, 2021).
• Clustered designs: the permuted unit must be the randomized unit, so for fits with a cluster variable whole clusters are permuted. The statistic uses covariate-adjusted cluster-level summaries (Gail et al., 1996; Hayes & Moulton, 2017, ch. 10), and designs with few enough allocations are enumerated exactly.
• Limitations: designs that assign treatment to participants within clusters, and stratified or restricted randomization of clusters, are not supported.

Equivalence testing for sensitization

Label: contemporary recommendation.

A nonsignificant Pretest x Treatment test does not show that sensitization is absent. equivalence_solomon() applies the two one-sided tests procedure (Schuirmann, 1987) against bounds set in advance at the smallest effect size of interest (Lakens, 2017; Lakens et al., 2018).

Effect sizes with confidence intervals

Label: contemporary recommendation.

Standardized mean differences use noncentral t intervals (Cumming & Finch, 2001; Kelley, 2007), and partial R-squared uses noncentral F intervals when conventional covariance is used (Steiger, 2004). Hedges’ g corrects the small-sample bias of the standardized mean difference (Hedges, 1981).

Missing data

Label: contemporary recommendation, applied to the Solomon design.

• Structural absence: Groups 3 and 4 have no pretest by design. That absence is the manipulation, so it is never imputed. This differs from planned missing-data designs, in which unmeasured values exist (Graham et al., 2006).
• Incidental missing pretests: because treatment is randomized, simple approaches such as mean imputation with a missingness indicator retain participants without biasing the treatment effect (White & Thompson, 2005; Groenwold et al., 2012).
• Missing outcomes: the classification of missing-data mechanisms follows Rubin (1976) and Little and Rubin (2019).
• Missing posttests: if posttests are missing at random given group and, in the pretested groups, the pretest, the model fitted to the observed posttests is a valid main analysis, and multiple imputation under the same assumption agrees with it (Carpenter et al., 2023, p. 256). Whether the conclusion survives departures from that assumption is a sensitivity analysis. fit_solomon_mi() shifts the imputed posttests of each group by a chosen offset (Carpenter et al., 2023, section 10.3), and tipping_point_solomon() reports the offset at which a contrast’s conclusion changes (White et al., 2011). Offsets that differ between the pretested groups bear on the sensitization contrast: in the package’s simulation study, a departure confined to one pretested group biased the analyses that assume missing at random by 0.10 to 0.16 SD.
• Tools: check_solomon_missing() and validate_solomon() report each category.

Assumption diagnostics as description

Label: contemporary recommendation.

Choosing an analysis because a preliminary test of an assumption was or was not significant can distort Type I error rates (Zimmerman, 2004). check_solomon_assumptions() therefore reports Brown-Forsythe tests (Brown & Forsythe, 1974), normality tests, and a slope-homogeneity test as descriptions rather than gates.

Nonrandomized Solomon designs

Label: contemporary recommendation.

Solomon designs are often run with intact groups, as in El Karkri et al. (2025a), who assigned one class to each condition. Without random assignment, the contrasts are differences between groups rather than treatment effects. Selection bias is the largest threat to internal validity in quasi-experimental research (Edmonds & Kennedy, 2017, p. 7), and with instrumentation it is the threat most common in quasi-experimental Solomon designs (p. 94).

• Baseline in the pretested arms. baseline_solomon() compares the pretested groups’ pretests. In El Karkri et al. (2025a, p. 10), the pretested classes already differed at baseline, 9.61 against 7.86.
• No baseline in the unpretested arms. The unpretested arms form the posttest-only control group design, which relies on randomization rather than a pretest for the equivalence of its groups (Campbell & Stanley, 1963/1966, p. 25). Without random assignment they form a static-group comparison, for which there are “no formal means of certifying that the groups would have been equivalent” (p. 12). Selection bias in their comparison, which isolates pretest sensitization, therefore cannot be checked or adjusted for with the study’s own data.
• Reporting. report_solomon(fit, design = list(assignment = "nonrandom")) describes differences rather than effects and names these threats.

Repeated posttests

Label: contemporary recommendation, applied to the Solomon design.

When the posttest is measured on several occasions, fit_solomon_mmrm() fits a mixed model for repeated measures with an unstructured within-participant covariance estimated by restricted maximum likelihood (Laird & Ware, 1982; Mallinckrodt et al., 2008). Likelihood-based analyses remain valid when participants drop out depending on their earlier observed scores, whereas per-occasion analyses of those still observed, and unweighted generalized estimating equations, require missingness completely at random (Fitzmaurice et al., 2011).

• Covariance by pretest condition. Estimating the covariance separately for pretested and unpretested participants is solomonR’s adaptation: the pretest adjustment makes the residual covariance differ between them by design.
• Evidence. In the package’s simulation study, the contrasts were unbiased under such dropout, with coverage of 0.937 to 0.962. Per-occasion complete-case analyses were biased by up to 0.09 SD, and a covariance shared by all four groups misstated standard errors by up to 17%.

Designs with several treatments

Label: contemporary recommendation, applied to the Solomon design.

With k treatments and a control, each with and without a pretest, the design has 2(k + 1) groups (Steyn, 2009). Published studies analyzed such designs as overlapping four-group designs: one for each treatment against the control (McCarthy & Tucker, 2002), those and one with the treatments pooled (Steyn, 2005), or one for each pair of conditions (Mai et al., 2020). Those analyses share groups, so their tests are dependent, and their number is not allowed for.

• Estimates: the four Solomon contrasts for each treatment against the control, for every pair of conditions, or for planned comparisons given as weights, from one model fitted to all the groups.
• Tests: omnibus Wald tests of each contrast across the conditions, then the comparisons, with p-values adjusted within each contrast by Holm’s (1979) procedure. The confidence intervals are not adjusted.
• Assumptions: those of the unified model; a common pretest-posttest slope across the pretested groups.
• Function: fit_solomon_glm(control = ). The article “Designs With Several Treatments” works through an example.
• Validation: in the package’s simulation study (issue #45), the contrasts were unbiased, coverage of the 95% intervals was 0.939 to 0.967, and the familywise error rates of the Holm-adjusted comparisons were at most 0.059. The pre-specified rule for error control was not met: with three treatments and 10 participants per group, the omnibus tests of Condition | pretested and Condition | unpretested rejected in 0.055 to 0.069 of replications at the .05 level. The analysis is therefore experimental. Overlapping four-group analyses found at least one significant interaction in 8% to 22% of replications when no treatment was sensitized.

Full-information maximum likelihood

Label: published Solomon proposal.

• Estimates: all four contrasts, using the pretest in Groups 1 and 2 and separate residual variances for pretested and unpretested participants (van Engelenburg, 1999).
• Assumptions: normal errors; a common pretest-posttest slope in the pretested groups; equal variances within each pretest condition.
• Limitations: its point estimates coincide with separate regressions in the two pretest conditions. Its default Wald intervals use a normal reference distribution and were too narrow with small groups in the package’s simulation validation. A small-sample option, inference = "satterthwaite", uses unbiased residual variances and Welch-Satterthwaite degrees of freedom (Satterthwaite, 1946; Welch, 1947); the function warns when groups are small and no option has been chosen.
• Function: fit_solomon_ml().

Steyn’s sequence of tests

Label: published Solomon proposal.

• Estimates: no joint estimate of the Solomon contrasts. It is a sequence of tests: of the equivalence of the pretested groups, history and maturation, testing, the pretest-intervention interaction (Tests A-I of Walton Braver & Braver, 1988, for each treatment against the control), reliability, regression to the mean, and attrition, and then of the treatments’ effects by one-way analyses of variance with post hoc tests (Steyn, 2009). The post hoc tests are Scheffé’s (1953) tests, as Steyn
  2005. used them.
• Assumptions: equal variances and independent groups in each test.
• Limitations: many tests, each at the nominal level; Steyn (2009) notes that several analyses of variance capitalize on chance. It tests the pretest-intervention interaction separately for each treatment, unadjusted, and its further comparison of each treatment’s pretested and unpretested groups mixes the pretest’s own effect with sensitization. solomonR follows a pre-publication draft of the article, which the author provided.
• Function: fit_solomon_steyn(), for one or several treatments.

Unified GLM with a design-coded pretest

Label: solomonR extension, built on contemporary recommendations.

• Estimates: all four contrasts from one model, y ~ treat * pretested + pre_obs. pre_obs equals the pretest in Groups 1 and 2 and 0 in Groups 3 and 4, so no group is dropped.
• Assumptions: a common pretest slope in the pretested groups; independent observations (or clusters with CR2).
• Limitations: because adjusting for the pretest reduces residual variance only in the pretested groups, the model is heteroskedastic whenever the pretest predicts the posttest. HC3 or CR2 covariance addresses this.
• Checks: its estimate of Treatment | pretested equals the ANCOVA estimate (Test E), and its estimate of Treatment | unpretested equals Test C. The package tests these identities.
• Function: fit_solomon_glm().

Structural equation models

Label: solomonR extension, built on established SEM methods.

• Estimates: the four contrasts from a multi-group mean structure (fit_solomon_sem()), or on a latent outcome measured by several items (fit_solomon_sem_latent()).
• Assumptions: latent mean comparisons require scalar measurement invariance across groups (Meredith, 1993; Vandenberg & Lance, 2000). Models are estimated with lavaan (Rosseel, 2012).
• Limitations: the four-group observed mean model is saturated, so its fit indices are not diagnostic. Latent contrasts are on the latent-variable scale and are not directly comparable with observed-score contrasts.

Method comparison and design checks

Label: solomonR extension.

compare_solomon_methods() places the estimates from several analyses side by side and explains which analyses are not comparable. validate_solomon() and check_solomon_missing() check the design before analysis.

Power simulation

Label: solomonR extension, validated by simulation.

power_solomon() reports the rejection rate of each Solomon test with its Monte Carlo standard error, naming the estimand and true effect behind every row. A pre-specified study of 126 scenarios found that the 2x2 ANOVA interaction matched its analytic benchmark in every scenario, that the historical Test I held its nominal size under the complete null, and that no fit failed. Rejection rates for the GLM tests are conservative with small cells, following the HC3 standard errors used by default, so treat them as a lower bound with 20 or fewer participants per cell. The article “Validating power_solomon()” reports the study in full. plan_solomon() inverts the same calculations to find the smallest design that reaches a target power, either analytically or by simulating the package’s own GLM test.