--- title: "Solomon Methods: History, Recommendations, and Extensions" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Solomon Methods: History, Recommendations, and Extensions} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ## How to read this guide This guide describes every analysis in `solomonR`: what it estimates, what it assumes, where it falls short, and where it comes from. Each method carries one of four labels. | Label | Meaning | |---|---| | **Historical procedure** | Proposed in the Solomon literature and kept for teaching and replication. | | **Contemporary recommendation** | A general methodological recommendation from current literature, applied here to the Solomon design. | | **Published Solomon proposal** | A Solomon-specific method proposed in the literature but not yet widely established. | | **solomonR extension** | An implementation choice or combination introduced by this package. | A label describes where a method comes from, not whether it is correct. Historical procedures can still give valid estimates, and package extensions are tested against the methods they build on. ## The design and its questions Solomon (1949) added two posttest-only groups to the pretest-posttest control-group design so that the effect of taking a pretest could be separated from the effect of the treatment. Campbell and Stanley (1963/1966, pp. 13, 24) later classified it among the true experimental designs. | Group | Pretest | Treatment | Posttest | |:-----:|:-------:|:---------:|:--------:| | 1 | Yes | Yes | Yes | | 2 | Yes | No | Yes | | 3 | No | Yes | Yes | | 4 | No | No | Yes | Every analysis in `solomonR` estimates some of four contrasts in mean posttest scores, always as treatment minus control: | Contrast | Question | |---|---| | Treatment \| pretested | Does the treatment work among participants who took the pretest (Group 1 vs. 2)? | | Treatment \| unpretested | Does the treatment work among participants who did not (Group 3 vs. 4)? | | Pretest x Treatment | Does the treatment effect depend on pretesting? This is pretest sensitization. | | ATE (avg over pretest) | What is the treatment effect averaged equally over both pretest conditions? | Naming the target quantity before choosing an estimator makes it clear when different analyses answer the same question and when they do not (Lundberg et al., 2021). ## Historical procedures The historical analyses are available through `fit_solomon_classic()` and are described in more detail in `vignette("classic-solomon")`. ### Four-group analysis of posttest scores (Tests A-D) **Label:** historical procedure. - **Estimates:** all four contrasts from a two-by-two model of posttest scores (Test A: Pretest x Treatment; B and C: the simple treatment effects; D: the equal-weighted average). - **Assumptions:** independent observations, normally distributed errors, and equal variances across the four groups. - **Limitations:** the pretest scores in Groups 1 and 2 are ignored, which gives up precision. Campbell and Stanley (1963/1966, p. 25) suggested analyzing the posttest scores with a two-by-two analysis of variance and, if the main and interactive effects of pretesting are negligible, an analysis of covariance of the pretested groups' posttests with the pretest as covariate. Choosing later tests according to earlier significance tests makes the overall procedure conditional. - **Sources:** Campbell and Stanley (1963/1966); Huck and Sandler (1973). ### Analyses within the pretested groups (Tests E-G) **Label:** historical procedure. - **Estimates:** Treatment \| pretested, using the pretest information. Test E is analysis of covariance (ANCOVA), Test F compares gain scores, and Test G is the two-occasion repeated-measures interaction, which is algebraically equivalent to Test F. - **Assumptions:** for ANCOVA, a common pretest-posttest slope in both groups; a gain score implicitly assumes that slope equals one. - **Limitations:** only Groups 1 and 2 are used. In randomized studies, ANCOVA has more power than change scores; in nonrandomized comparisons, ANCOVA can be more biased (Van Breukelen, 2006). Because Solomon groups are randomized, ANCOVA is generally the stronger choice. - **Sources:** Huck and Sandler (1973); Van Breukelen (2006). ### Posttest-only comparison (Test H) **Label:** historical procedure. - **Estimates:** Treatment \| unpretested from Groups 3 and 4. - **Assumptions:** independent observations, normal errors, equal variances in the two groups. - **Limitations:** half of the sample is ignored. - **Sources:** Huck and Sandler (1973). ### Meta-analytic combination (Test I) **Label:** historical procedure. - **Estimates:** nothing directly. Test I combines one-tailed p-values from the pretested and unpretested treatment comparisons with Stouffer's method (Stouffer et al., 1949), as proposed by Walton Braver and Braver (1988). The combined z is judged by its two-tailed p-value, as in their worked example (p. 153). - **Assumptions:** both tests address the same directional hypothesis, and the combination is specified in advance. - **Limitations:** in a Monte Carlo study, the conditional sequence that ends in Test I had an experiment-wise Type I error rate nearly three times the nominal alpha, and traditional procedures used without conditioning were more powerful (Sawilowsky et al., 1994). The proposal was debated in published exchanges between the two research groups. `solomonR` reproduces Test I for replication but does not recommend it as a default analysis. - **Sources:** Walton Braver and Braver (1988); Sawilowsky et al. (1994). ## Contemporary recommendations ### One prespecified analysis with explicit contrasts **Label:** contemporary recommendation. Rather than following a sequence of significance-driven decisions, contemporary practice specifies the analysis and its target quantities in advance (Lundberg et al., 2021). In randomized experiments, adjusting for a baseline covariate such as the pretest is justified for estimating treatment effects, and heteroskedasticity-robust standard errors protect that inference (Lin, 2013). ### Heteroskedasticity-consistent and small-sample inference **Label:** contemporary recommendation. - **HC3 standard errors** are recommended as a routine choice, especially below about 250 observations (MacKinnon & White, 1985; Long & Ervin, 2000; Hayes & Cai, 2007). `fit_solomon_glm()` uses HC3 by default. - **t reference distributions** with residual degrees of freedom are used for Gaussian models. Normal-reference robust intervals can under-cover in small samples (Imbens & Kolesár, 2016), and t with n - p degrees of freedom is the choice evaluated in recent guidance for behavioral research (Rajh-Weber et al., 2025). - **Clustered data** use the CR2 estimator with Satterthwaite degrees of freedom (Bell & McCaffrey, 2002; Pustejovsky & Tipton, 2018). ### Randomization inference **Label:** contemporary recommendation. - **Tests:** `perm_solomon()` permutes treatment labels within pretest conditions, which reproduces how treatment was assigned. Its p-value is valid for the sharp null hypothesis of no effect for any participant, and the +1 correction keeps it from being zero (Phipson & Smyth, 2010). Studentized statistics make such tests robust when only an average effect is hypothesized to be zero (DiCiccio & Romano, 2017; Wu & Ding, 2021). - **Clustered designs:** the permuted unit must be the randomized unit, so for fits with a `cluster` variable whole clusters are permuted. The statistic uses covariate-adjusted cluster-level summaries (Gail et al., 1996; Hayes & Moulton, 2017, ch. 10), and designs with few enough allocations are enumerated exactly. - **Limitations:** designs that assign treatment to participants within clusters, and stratified or restricted randomization of clusters, are not supported. ### Equivalence testing for sensitization **Label:** contemporary recommendation. A nonsignificant Pretest x Treatment test does not show that sensitization is absent. `equivalence_solomon()` applies the two one-sided tests procedure (Schuirmann, 1987) against bounds set in advance at the smallest effect size of interest (Lakens, 2017; Lakens et al., 2018). ### Effect sizes with confidence intervals **Label:** contemporary recommendation. Standardized mean differences use noncentral t intervals (Cumming & Finch, 2001; Kelley, 2007), and partial R-squared uses noncentral F intervals when conventional covariance is used (Steiger, 2004). Hedges' g corrects the small-sample bias of the standardized mean difference (Hedges, 1981). ### Missing data **Label:** contemporary recommendation, applied to the Solomon design. - **Structural absence:** Groups 3 and 4 have no pretest by design. That absence is the manipulation, so it is never imputed. This differs from planned missing-data designs, in which unmeasured values exist (Graham et al., 2006). - **Incidental missing pretests:** because treatment is randomized, simple approaches such as mean imputation with a missingness indicator retain participants without biasing the treatment effect (White & Thompson, 2005; Groenwold et al., 2012). - **Missing outcomes:** the classification of missing-data mechanisms follows Rubin (1976) and Little and Rubin (2019). - **Missing posttests:** if posttests are missing at random given group and, in the pretested groups, the pretest, the model fitted to the observed posttests is a valid main analysis, and multiple imputation under the same assumption agrees with it (Carpenter et al., 2023, p. 256). Whether the conclusion survives departures from that assumption is a sensitivity analysis. `fit_solomon_mi()` shifts the imputed posttests of each group by a chosen offset (Carpenter et al., 2023, section 10.3), and `tipping_point_solomon()` reports the offset at which a contrast's conclusion changes (White et al., 2011). Offsets that differ between the pretested groups bear on the sensitization contrast: in the package's simulation study, a departure confined to one pretested group biased the analyses that assume missing at random by 0.10 to 0.16 SD. - **Tools:** `check_solomon_missing()` and `validate_solomon()` report each category. ### Assumption diagnostics as description **Label:** contemporary recommendation. Choosing an analysis because a preliminary test of an assumption was or was not significant can distort Type I error rates (Zimmerman, 2004). `check_solomon_assumptions()` therefore reports Brown-Forsythe tests (Brown & Forsythe, 1974), normality tests, and a slope-homogeneity test as descriptions rather than gates. ### Nonrandomized Solomon designs **Label:** contemporary recommendation. Solomon designs are often run with intact groups, as in El Karkri et al. (2025a), who assigned one class to each condition. Without random assignment, the contrasts are differences between groups rather than treatment effects. Selection bias is the largest threat to internal validity in quasi-experimental research (Edmonds & Kennedy, 2017, p. 7), and with instrumentation it is the threat most common in quasi-experimental Solomon designs (p. 94). - **Baseline in the pretested arms.** `baseline_solomon()` compares the pretested groups' pretests. In El Karkri et al. (2025a, p. 10), the pretested classes already differed at baseline, 9.61 against 7.86. - **No baseline in the unpretested arms.** The unpretested arms form the posttest-only control group design, which relies on randomization rather than a pretest for the equivalence of its groups (Campbell & Stanley, 1963/1966, p. 25). Without random assignment they form a static-group comparison, for which there are "no formal means of certifying that the groups would have been equivalent" (p. 12). Selection bias in their comparison, which isolates pretest sensitization, therefore cannot be checked or adjusted for with the study's own data. - **Reporting.** `report_solomon(fit, design = list(assignment = "nonrandom"))` describes differences rather than effects and names these threats. ### Repeated posttests **Label:** contemporary recommendation, applied to the Solomon design. When the posttest is measured on several occasions, `fit_solomon_mmrm()` fits a mixed model for repeated measures with an unstructured within-participant covariance estimated by restricted maximum likelihood (Laird & Ware, 1982; Mallinckrodt et al., 2008). Likelihood-based analyses remain valid when participants drop out depending on their earlier observed scores, whereas per-occasion analyses of those still observed, and unweighted generalized estimating equations, require missingness completely at random (Fitzmaurice et al., 2011). - **Covariance by pretest condition.** Estimating the covariance separately for pretested and unpretested participants is solomonR's adaptation: the pretest adjustment makes the residual covariance differ between them by design. - **Evidence.** In the package's simulation study, the contrasts were unbiased under such dropout, with coverage of 0.937 to 0.962. Per-occasion complete-case analyses were biased by up to 0.09 SD, and a covariance shared by all four groups misstated standard errors by up to 17%. ### Designs with several treatments **Label:** contemporary recommendation, applied to the Solomon design. With k treatments and a control, each with and without a pretest, the design has 2(k + 1) groups (Steyn, 2009). Published studies analyzed such designs as overlapping four-group designs: one for each treatment against the control (McCarthy & Tucker, 2002), those and one with the treatments pooled (Steyn, 2005), or one for each pair of conditions (Mai et al., 2020). Those analyses share groups, so their tests are dependent, and their number is not allowed for. - **Estimates:** the four Solomon contrasts for each treatment against the control, for every pair of conditions, or for planned comparisons given as weights, from one model fitted to all the groups. - **Tests:** omnibus Wald tests of each contrast across the conditions, then the comparisons, with p-values adjusted within each contrast by Holm's (1979) procedure. The confidence intervals are not adjusted. - **Assumptions:** those of the unified model; a common pretest-posttest slope across the pretested groups. - **Function:** `fit_solomon_glm(control = )`. The article "Designs With Several Treatments" works through an example. - **Validation:** in the package's simulation study (issue #45), the contrasts were unbiased, coverage of the 95% intervals was 0.939 to 0.967, and the familywise error rates of the Holm-adjusted comparisons were at most 0.059. The pre-specified rule for error control was not met: with three treatments and 10 participants per group, the omnibus tests of Condition | pretested and Condition | unpretested rejected in 0.055 to 0.069 of replications at the .05 level. The analysis is therefore experimental. Overlapping four-group analyses found at least one significant interaction in 8% to 22% of replications when no treatment was sensitized. ## Published Solomon proposals ### Full-information maximum likelihood **Label:** published Solomon proposal. - **Estimates:** all four contrasts, using the pretest in Groups 1 and 2 and separate residual variances for pretested and unpretested participants (van Engelenburg, 1999). - **Assumptions:** normal errors; a common pretest-posttest slope in the pretested groups; equal variances within each pretest condition. - **Limitations:** its point estimates coincide with separate regressions in the two pretest conditions. Its default Wald intervals use a normal reference distribution and were too narrow with small groups in the package's simulation validation. A small-sample option, `inference = "satterthwaite"`, uses unbiased residual variances and Welch-Satterthwaite degrees of freedom (Satterthwaite, 1946; Welch, 1947); the function warns when groups are small and no option has been chosen. - **Function:** `fit_solomon_ml()`. ### Steyn's sequence of tests **Label:** published Solomon proposal. - **Estimates:** no joint estimate of the Solomon contrasts. It is a sequence of tests: of the equivalence of the pretested groups, history and maturation, testing, the pretest-intervention interaction (Tests A-I of Walton Braver & Braver, 1988, for each treatment against the control), reliability, regression to the mean, and attrition, and then of the treatments' effects by one-way analyses of variance with post hoc tests (Steyn, 2009). The post hoc tests are Scheffé's (1953) tests, as Steyn (2005) used them. - **Assumptions:** equal variances and independent groups in each test. - **Limitations:** many tests, each at the nominal level; Steyn (2009) notes that several analyses of variance capitalize on chance. It tests the pretest-intervention interaction separately for each treatment, unadjusted, and its further comparison of each treatment's pretested and unpretested groups mixes the pretest's own effect with sensitization. solomonR follows a pre-publication draft of the article, which the author provided. - **Function:** `fit_solomon_steyn()`, for one or several treatments. ## solomonR extensions ### Unified GLM with a design-coded pretest **Label:** solomonR extension, built on contemporary recommendations. - **Estimates:** all four contrasts from one model, `y ~ treat * pretested + pre_obs`. `pre_obs` equals the pretest in Groups 1 and 2 and 0 in Groups 3 and 4, so no group is dropped. - **Assumptions:** a common pretest slope in the pretested groups; independent observations (or clusters with CR2). - **Limitations:** because adjusting for the pretest reduces residual variance only in the pretested groups, the model is heteroskedastic whenever the pretest predicts the posttest. HC3 or CR2 covariance addresses this. - **Checks:** its estimate of Treatment \| pretested equals the ANCOVA estimate (Test E), and its estimate of Treatment \| unpretested equals Test C. The package tests these identities. - **Function:** `fit_solomon_glm()`. ### Structural equation models **Label:** solomonR extension, built on established SEM methods. - **Estimates:** the four contrasts from a multi-group mean structure (`fit_solomon_sem()`), or on a latent outcome measured by several items (`fit_solomon_sem_latent()`). - **Assumptions:** latent mean comparisons require scalar measurement invariance across groups (Meredith, 1993; Vandenberg & Lance, 2000). Models are estimated with lavaan (Rosseel, 2012). - **Limitations:** the four-group observed mean model is saturated, so its fit indices are not diagnostic. Latent contrasts are on the latent-variable scale and are not directly comparable with observed-score contrasts. ### Method comparison and design checks **Label:** solomonR extension. `compare_solomon_methods()` places the estimates from several analyses side by side and explains which analyses are not comparable. `validate_solomon()` and `check_solomon_missing()` check the design before analysis. ### Power simulation **Label:** solomonR extension, validated by simulation. `power_solomon()` reports the rejection rate of each Solomon test with its Monte Carlo standard error, naming the estimand and true effect behind every row. A pre-specified study of 126 scenarios found that the 2x2 ANOVA interaction matched its analytic benchmark in every scenario, that the historical Test I held its nominal size under the complete null, and that no fit failed. Rejection rates for the GLM tests are conservative with small cells, following the HC3 standard errors used by default, so treat them as a lower bound with 20 or fewer participants per cell. The article "Validating power_solomon()" reports the study in full. `plan_solomon()` inverts the same calculations to find the smallest design that reaches a target power, either analytically or by simulating the package's own GLM test. ## Summary | Method | Label | Contrasts | Pretest used | Key assumption | |---|---|---|---|---| | Tests A-D | Historical | All four | No | Equal variances across four groups | | Tests E-G | Historical | Treatment \| pretested | Yes | Common slope (E); slope of one (F, G) | | Test H | Historical | Treatment \| unpretested | No | Equal variances in Groups 3-4 | | Test I | Historical | None (combined p-value) | Yes | Same directional hypothesis | | Unified GLM | solomonR extension | All four | Yes | Common slope; HC3 or CR2 inference | | Randomization test | Contemporary | One contrast (test only) | Optional | Permute the randomized unit | | Equivalence test | Contemporary | One contrast | As fitted | Bounds fixed in advance | | Maximum likelihood | Published proposal | All four | Yes | Normal errors; separate variances by pretest condition | | SEM (observed) | solomonR extension | All four | Optional | Multi-group mean structure | | SEM (latent) | solomonR extension | All four (latent scale) | Optional | Scalar invariance | | Multiple imputation with offsets | Contemporary | All four | Yes | Missing posttests shifted by stated offsets | | Mixed model for repeated measures | Contemporary | All four, at each occasion | Yes | Dropout missing at random; covariance by pretest condition | | Joint model for several treatments | Contemporary | All four, for each comparison | Yes | Common slope; p-values adjusted across comparisons | | Steyn's sequence | Published proposal | Per treatment, through Tests A-I; none jointly | In its checks | Equal variances; many unadjusted tests | All works cited in solomonR are listed, with notes on how the package uses them, on the [References](https://juhalt.github.io/solomonR/articles/references.html) page. ## References Bell, R. M., & McCaffrey, D. F. (2002). Bias reduction in standard errors for linear regression with multi-stage samples. *Survey Methodology, 28*(2), 169–181. Brown, M. B., & Forsythe, A. B. (1974). Robust tests for the equality of variances. *Journal of the American Statistical Association, 69*(346), 364–367. https://doi.org/10.1080/01621459.1974.10482955 Campbell, D. T., & Stanley, J. C. (1966). *Experimental and quasi-experimental designs for research*. Rand McNally. (Original work published 1963) Carpenter, J. R., Bartlett, J. W., Morris, T. P., Wood, A. M., Quartagno, M., & Kenward, M. G. (2023). *Multiple imputation and its application* (2nd ed.). Wiley. https://doi.org/10.1002/9781119756118 Cumming, G., & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. *Educational and Psychological Measurement, 61*(4), 532–574. https://doi.org/10.1177/00131640121971374 DiCiccio, C. J., & Romano, J. P. (2017). Robust permutation tests for correlation and regression coefficients. *Journal of the American Statistical Association, 112*(519), 1211–1220. https://doi.org/10.1080/01621459.2016.1202117 Edmonds, W. A., & Kennedy, T. D. (2017). *An applied guide to research designs: Quantitative, qualitative, and mixed methods* (2nd ed.). SAGE Publications. https://doi.org/10.4135/9781071802779 El Karkri, M., Quesada, A., & Romero-Ariza, M. (2025a). The dual impact of pretest sensitisation and the cognitive acceleration through science education programme in the Solomon four-group design. *Brain Sciences, 16*(1), Article 64. https://doi.org/10.3390/brainsci16010064 Fitzmaurice, G. M., Laird, N. M., & Ware, J. H. (2011). *Applied longitudinal analysis* (2nd ed.). Wiley. https://doi.org/10.1002/9781119513469 Gail, M. H., Mark, S. D., Carroll, R. J., Green, S. B., & Pee, D. (1996). On design considerations and randomization-based inference for community intervention trials. *Statistics in Medicine, 15*(11), 1069–1092. Graham, J. W., Taylor, B. J., Olchowski, A. E., & Cumsille, P. E. (2006). Planned missing data designs in psychological research. *Psychological Methods, 11*(4), 323–343. https://doi.org/10.1037/1082-989X.11.4.323 Groenwold, R. H. H., White, I. R., Donders, A. R. T., Carpenter, J. R., Altman, D. G., & Moons, K. G. M. (2012). Missing covariate data in clinical research: When and when not to use the missing-indicator method for analysis. *Canadian Medical Association Journal, 184*(11), 1265–1269. https://doi.org/10.1503/cmaj.110977 Hayes, A. F., & Cai, L. (2007). Using heteroskedasticity-consistent standard error estimators in OLS regression: An introduction and software implementation. *Behavior Research Methods, 39*(4), 709–722. https://doi.org/10.3758/BF03192961 Hayes, R. J., & Moulton, L. H. (2017). *Cluster randomised trials* (2nd ed.). Chapman and Hall/CRC. https://doi.org/10.4324/9781315370286 Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. *Journal of Educational Statistics, 6*(2), 107–128. https://doi.org/10.3102/10769986006002107 Holm, S. (1979). A simple sequentially rejective multiple test procedure. *Scandinavian Journal of Statistics, 6*(2), 65–70. https://www.jstor.org/stable/4615733 Huck, S. W., & Sandler, H. M. (1973). A note on the Solomon 4-group design: Appropriate statistical analyses. *The Journal of Experimental Education, 42*(2), 54–55. https://doi.org/10.1080/00220973.1973.11011460 Imbens, G. W., & Kolesár, M. (2016). Robust standard errors in small samples: Some practical advice. *The Review of Economics and Statistics, 98*(4), 701–712. https://doi.org/10.1162/REST_a_00552 Kelley, K. (2007). Confidence intervals for standardized effect sizes: Theory, application, and implementation. *Journal of Statistical Software, 20*(8), 1–24. https://doi.org/10.18637/jss.v020.i08 Laird, N. M., & Ware, J. H. (1982). Random-effects models for longitudinal data. *Biometrics, 38*(4), 963–974. https://doi.org/10.2307/2529876 Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. *Social Psychological and Personality Science, 8*(4), 355–362. https://doi.org/10.1177/1948550617697177 Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. *Advances in Methods and Practices in Psychological Science, 1*(2), 259–269. https://doi.org/10.1177/2515245918770963 Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique. *The Annals of Applied Statistics, 7*(1), 295–318. https://doi.org/10.1214/12-AOAS583 Little, R. J. A., & Rubin, D. B. (2019). *Statistical analysis with missing data* (3rd ed.). Wiley. https://doi.org/10.1002/9781119482260 Long, J. S., & Ervin, L. H. (2000). Using heteroscedasticity consistent standard errors in the linear regression model. *The American Statistician, 54*(3), 217–224. https://doi.org/10.1080/00031305.2000.10474549 Lundberg, I., Johnson, R., & Stewart, B. M. (2021). What is your estimand? Defining the target quantity connects statistical evidence to theory. *American Sociological Review, 86*(3), 532–565. https://doi.org/10.1177/00031224211004187 MacKinnon, J. G., & White, H. (1985). Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties. *Journal of Econometrics, 29*(3), 305–325. https://doi.org/10.1016/0304-4076(85)90158-7 Mai, N. N., Takahashi, Y., & Oo, M. M. (2020). Testing the effectiveness of transfer interventions using Solomon four-group designs. *Education Sciences, 10*(4), Article 92. https://doi.org/10.3390/educsci10040092 Mallinckrodt, C. H., Lane, P. W., Schnell, D., Peng, Y., & Mancuso, J. P. (2008). Recommendations for the primary analysis of continuous endpoints in longitudinal clinical trials. *Drug Information Journal, 42*(4), 303–319. https://doi.org/10.1177/009286150804200402 McCarthy, A. M., & Tucker, M. L. (2002). Encouraging community service through service learning. *Journal of Management Education, 26*(6), 629–647. https://doi.org/10.1177/1052562902238322 Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. *Psychometrika, 58*(4), 525–543. https://doi.org/10.1007/BF02294825 Phipson, B., & Smyth, G. K. (2010). Permutation p-values should never be zero: Calculating exact p-values when permutations are randomly drawn. *Statistical Applications in Genetics and Molecular Biology, 9*(1), Article 39. https://doi.org/10.2202/1544-6115.1585 Pustejovsky, J. E., & Tipton, E. (2018). Small-sample methods for cluster-robust variance estimation and hypothesis testing in fixed effects models. *Journal of Business & Economic Statistics, 36*(4), 672–683. https://doi.org/10.1080/07350015.2016.1247004 Rajh-Weber, H., Huber, S. E., & Arendasy, M. (2025). A practice-oriented guide to statistical inference in linear modeling for non-normal or heteroskedastic error distributions. *Behavior Research Methods, 57*(12), Article 338. https://doi.org/10.3758/s13428-025-02801-4 Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. *Journal of Statistical Software, 48*(2), 1–36. https://doi.org/10.18637/jss.v048.i02 Rubin, D. B. (1976). Inference and missing data. *Biometrika, 63*(3), 581–592. https://doi.org/10.1093/biomet/63.3.581 Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. *Biometrics Bulletin, 2*(6), 110–114. https://doi.org/10.2307/3002019 Sawilowsky, S. S., Kelley, D. L., Blair, R. C., & Markman, B. S. (1994). Meta-analysis and the Solomon four-group design. *The Journal of Experimental Education, 62*(4), 361–376. https://doi.org/10.1080/00220973.1994.9944140 Scheffé, H. (1953). A method for judging all contrasts in the analysis of variance. *Biometrika, 40*(1–2), 87–104. https://doi.org/10.1093/biomet/40.1-2.87 Schuirmann, D. J. (1987). A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. *Journal of Pharmacokinetics and Biopharmaceutics, 15*(6), 657–680. https://doi.org/10.1007/BF01068419 Solomon, R. L. (1949). An extension of control group design. *Psychological Bulletin, 46*(2), 137–150. https://doi.org/10.1037/h0062958 Steiger, J. H. (2004). Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. *Psychological Methods, 9*(2), 164–182. https://doi.org/10.1037/1082-989X.9.2.164 Steyn, R. (2005). *Self-evaluasie en die vorming van selfdoeltreffendheidspersepsies* [Self-evaluation and the forming of self-efficacy perceptions] [Doctoral thesis, University of South Africa]. Unisa Institutional Repository. https://hdl.handle.net/10500/1745 Steyn, R. (2009). Re-designing the Solomon four-group: Can we improve on this exemplary model? *Design Principles and Practices: An International Journal—Annual Review, 3*(1), 383–394. https://doi.org/10.18848/1833-1874/CGP/v03i01/37588 Stouffer, S. A., Suchman, E. A., DeVinney, L. C., Star, S. A., & Williams, R. M., Jr. (1949). *The American soldier: Adjustment during army life* (Vol. 1). Princeton University Press. Van Breukelen, G. J. P. (2006). ANCOVA versus change from baseline had more power in randomized studies and more bias in nonrandomized studies. *Journal of Clinical Epidemiology, 59*(9), 920–925. https://doi.org/10.1016/j.jclinepi.2006.02.007 Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. *Organizational Research Methods, 3*(1), 4–70. https://doi.org/10.1177/109442810031002 van Engelenburg, G. (1999). *Statistical analysis for the Solomon four-group design* (Research Report 99-06). University of Twente. ERIC. https://eric.ed.gov/?id=ED435692 Walton Braver, M. C., & Braver, S. L. (1988). Statistical treatment of the Solomon four-group design: A meta-analytic approach. *Psychological Bulletin, 104*(1), 150–154. https://doi.org/10.1037/0033-2909.104.1.150 Welch, B. L. (1947). The generalization of "Student's" problem when several different population variances are involved. *Biometrika, 34*(1–2), 28–35. https://doi.org/10.1093/biomet/34.1-2.28 White, I. R., Horton, N. J., Carpenter, J., & Pocock, S. J. (2011). Strategy for intention to treat analysis in randomised trials with missing outcome data. *BMJ, 342*, Article d40. https://doi.org/10.1136/bmj.d40 White, I. R., & Thompson, S. G. (2005). Adjusting for partially missing baseline measurements in randomized trials. *Statistics in Medicine, 24*(7), 993–1007. https://doi.org/10.1002/sim.1981 Wu, J., & Ding, P. (2021). Randomization tests for weak null hypotheses in randomized experiments. *Journal of the American Statistical Association, 116*(536), 1898–1913. https://doi.org/10.1080/01621459.2020.1750415 Zimmerman, D. W. (2004). A note on preliminary tests of equality of variances. *British Journal of Mathematical and Statistical Psychology, 57*(1), 173–181. https://doi.org/10.1348/000711004849222