---
title: "Getting Started: Analyzing a Solomon Four-Group Study"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting Started: Analyzing a Solomon Four-Group Study}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

This guide is for graduate students and applied researchers who need to
analyze, interpret, and report a Solomon four-group study. It follows one
example from checking the data to writing up the results, and it explains
where each recommendation comes from. For the full origin, assumptions, and
limitations of every method, see `vignette("solomon-methods")`.

## The path through a Solomon study

This guide covers the analysis. The steps before and after it each have
their own article:

1. **Decide** whether a Solomon design suits the question:
   [Should I Use a Solomon Design?](https://juhalt.github.io/solomonR/articles/should-i-use-a-solomon-design.html)
2. **Plan** the sample size:
   [Planning a Solomon Study](https://juhalt.github.io/solomonR/articles/planning.html)
3. **Analyze** the data: this guide, with the details of every method in
   `vignette("solomon-methods")`.
4. **Report** the design, the estimates, and the references:
   [Reporting a Solomon Study](https://juhalt.github.io/solomonR/articles/reporting.html)
5. **Synthesize** published studies:
   [Reanalysis and Synthesis](https://juhalt.github.io/solomonR/articles/synthesis.html)

The historical analysis has its own place:

- [A History of the Solomon Design and Its Analysis](https://juhalt.github.io/solomonR/articles/history.html) tells
  how it developed.
- `vignette("classic-solomon")` shows how to reproduce it. It is kept for
  teaching and replication, not recommended for new studies.

```{r load}
library(solomonR)
```

## The design and its three questions

Solomon (1949) extended the pretest-posttest control-group design by adding
two groups that receive no pretest:

| Group | Pretest | Treatment | Posttest |
|:-----:|:-------:|:---------:|:--------:|
| 1 | Yes | Yes | Yes |
| 2 | Yes | No | Yes |
| 3 | No | Yes | Yes |
| 4 | No | No | Yes |

The four groups let a study ask three questions:

1. **Does the treatment work?** Compare treatment with control.
2. **Does taking the pretest change posttest scores?** Compare pretested with
   unpretested groups.
3. **Does taking the pretest change the treatment effect?** This is
   *pretest sensitization*: the Pretest x Treatment contrast.

Sensitization should be tested rather than assumed. A systematic review of
ten Solomon four-group studies with behavioral outcomes found only weak
evidence that assessments interact with interventions, and concluded that
too few rigorous studies exist to say whether such interactions occur
(McCambridge et al., 2011).

`solomonR` reports four contrasts in mean posttest scores, always as
treatment minus control:

| Contrast | Meaning |
|---|---|
| Treatment \| pretested | Treatment effect among pretested participants (Groups 1 vs. 2) |
| Treatment \| unpretested | Treatment effect among unpretested participants (Groups 3 vs. 4) |
| Pretest x Treatment | Difference between those two effects (sensitization) |
| ATE (avg over pretest) | The two effects averaged with equal weight |

## The example data

`solomon_example` is a simulated study with 30 participants per group and
scores on a 0-100 scale. Because it was simulated, the truth is known:

- the treatment effect is **5 points** in both pretest conditions, so there
  is **no sensitization** and the ATE is 5 points;
- taking the pretest raises posttest scores by **2 points**; and
- the pretest and posttest correlate at about 0.6.

```{r example-data}
data(solomon_example)
head(solomon_example)
```

Pretest scores are missing for Groups 3 and 4 by design: those participants
never took the pretest.

Every analysis function takes the same data arguments, in the same order:
the posttest `y_post`, the treatment indicator `treat` (1 = treatment), the
pretest indicator `pretested` (1 = pretested), and the pretest `y_pre`. Pass
them as vectors, or give a data frame as `data` and name its columns. The
columns need not have these names: with your own data you might write
`fit_solomon_glm(score2, group, took_pretest, score1, data = mydata)`.

## Step 1: Check the design

Before fitting a model, confirm that all four groups are present, the design
indicators are coded 0/1, and missing values are the kind you expect.

```{r validate}
validate_solomon(y_post, treat, pretested, y_pre, data = solomon_example)
```

`check_solomon_missing()` separates pretests that are absent by design from
incidental missing values. Structurally absent pretests are never imputed:
not giving the pretest is the experimental manipulation. Incidental missing
values have their own supported responses (White & Thompson, 2005; Little &
Rubin, 2019).

```{r missing}
check_solomon_missing(y_post, treat, pretested, y_pre, data = solomon_example)
```

A schematic of the design, in the notation of Campbell and Stanley (1963/1966),
shows each group's size and posttest mean, and flags any group that is empty
or too small to estimate its variability:

```{r design-plot, fig.width = 8, fig.height = 3.5, fig.alt = "Diagram of the four Solomon groups, showing each group's size and posttest mean."}
plot_solomon_design(y_post, treat, pretested, data = solomon_example)
```

## Step 2: See how the design was analyzed historically

Early treatments of the design analyzed posttest scores with a two-by-two
analysis of variance and moved on to further tests depending on which
earlier tests were significant (Campbell & Stanley, 1963/1966, p. 25; Huck & Sandler,
1973). Walton Braver and Braver (1988) added a meta-analytic combination, Test I.
`fit_solomon_classic()` reproduces this sequence and marks the tests the
historical path reaches with `[PATH]`.

```{r classic}
classic <- fit_solomon_classic(y_post, treat, pretested, y_pre,
                               data = solomon_example)
classic
```

Here the sequence runs `r classic$path_string`: each treatment test is
nonsignificant, so the path keeps going until it reaches Test I. Choosing
the next test because the last one was not significant is exactly the kind
of conditional procedure that inflates Type I error. In a Monte Carlo study,
the sequence ending in Test I had an experiment-wise Type I error rate
nearly three times the nominal alpha (Sawilowsky et al., 1994). The
historical workflow is worth knowing, but it should not be the primary
analysis of a new study.

## Step 3: Fit the recommended analysis

Contemporary practice specifies one analysis and its target quantities in
advance (Lundberg et al., 2021). For a randomized Solomon study with a
continuous outcome, `fit_solomon_glm()` estimates all four contrasts from a
single model that adjusts for the pretest in the groups that took it. It
uses heteroskedasticity-consistent (HC3) standard errors by default (Long &
Ervin, 2000; Hayes & Cai, 2007) with t tests on the residual degrees of
freedom.

```{r glm}
fit <- fit_solomon_glm(y_post, treat, pretested, y_pre, data = solomon_example)
fit
```

```{r glm-values, include = FALSE}
effect <- function(name) fit$effects[fit$effects$contrast == name, ]
ate <- effect("ATE (avg over pretest)")
sens <- effect("Pretest x Treatment")
fmt <- function(x, digits = 1) formatC(x, format = "f", digits = digits)
fmt_p <- function(p) if (p < .001) "< .001" else sub("^0", "", formatC(p, format = "f", digits = 3))
```

Compare the estimates with the truth. The true ATE is 5 points; this sample
estimates `r fmt(ate$estimate)` points, with a 95% confidence interval from
`r fmt(ate$conf.low)` to `r fmt(ate$conf.high)`. The interval contains the
true value, and it also contains zero, so the test against zero is not
significant (p = `r fmt_p(ate$p.value)`).

This is not a failure of the method. Estimates vary from sample to sample,
and with 30 participants per group this design has roughly 85% power for the
ATE and less for the treatment effect within each pretest condition. About
one study in seven with this design would miss the effect. The confidence
interval is the honest summary: it shows that effects anywhere from about
zero to about six points are compatible with these data.

Planning the sample size in advance is how a study avoids this situation.
`plan_solomon()` finds the smallest design that reaches a target power for
each Solomon estimand. Using the values that generated these data (a
treatment effect of 5 points, a posttest standard deviation of 10, and a
pretest-posttest correlation of 0.6), 90% power requires:

```{r plan}
plan_solomon(power = 0.90, delta = 5, sens = 0, rho = 0.6, sigma = 10)
```

The sensitization row is empty because these data were generated without
sensitization, and no sample size gives power against a zero effect.
Detecting sensitization is far more demanding than detecting the average
effect, because the sensitization contrast has four times the sampling
variance of the ATE. Planning to detect sensitization of 5 points with 80%
power:

```{r plan-sensitization}
plan_solomon(power = 0.80, delta = 5, sens = 5, rho = 0.6, sigma = 10,
             estimand = "sensitization")
```

A Solomon study powered only for the ATE is usually underpowered for the
question the design exists to answer. `?plan_solomon` also shows how to
trade pretested against unpretested participants when pretesting is costly.

Power curves show the same planning problem across sample sizes. Each cross
marks the design `plan_solomon()` returns for the target:

```{r power-curves, fig.width = 8, fig.height = 5, fig.alt = "Power curves for the average treatment effect and the two simple treatment effects across sample sizes, with crosses marking the designs that reach 90% power."}
plot_power_solomon(
  n = seq(10, 150, by = 5), delta = 5, sens = 0, rho = 0.6, sigma = 10,
  estimand = c("ate", "pretested", "unpretested"), target = 0.90
)
```

## Step 4: Ask whether sensitization is negligible

The Pretest x Treatment estimate is `r fmt(sens$estimate)` points
(p = `r fmt_p(sens$p.value)`). A nonsignificant result does **not** show that
sensitization is absent. To make that claim, use an equivalence test against
a smallest effect size of interest set *before* seeing the data (Lakens,
2017; Lakens et al., 2018).

Suppose the researchers decided in advance that sensitization smaller than
half a standard deviation, 5 points on this test, would not matter:

```{r equivalence}
equivalence_solomon(fit, bounds = 5)
```

The 90% confidence interval extends beyond the bounds, so these data can
neither show sensitization nor rule out sensitization as large as 5 points.
"Inconclusive" is a legitimate and informative result; it tells readers that
a larger study would be needed to settle the question.

The sensitization itself is easiest to see in the group means.
`plot_sensitization()` draws model-adjusted means, so the difference between
the two lines' rises from control to treatment is exactly the Pretest x
Treatment estimate above. Hollow points show the observed means:

```{r sensitization-plot, fig.width = 6, fig.height = 4, fig.alt = "Model-based posttest means by treatment for pretested and unpretested participants, with observed means as hollow points and the equivalence bounds for the interaction."}
plot_sensitization(fit, bounds = 5)
```

A forest plot shows all four Solomon contrasts at once, with the equivalence
bounds shaded on the sensitization row:

```{r effects-plot, fig.width = 7, fig.height = 3.5, fig.alt = "Forest plot of the four Solomon contrasts with 95% intervals, with the equivalence bounds shaded on the sensitization row."}
plot_solomon_effects(fit, bounds = 5)
```

The sensitization interval extends beyond the shaded band, so these data
cannot rule out sensitization as large as 5 points. The figure shows the
fit's 95% intervals; the equivalence test uses the narrower 90% interval,
which also extends beyond the bounds.

## Step 5: Check how the conclusions depend on the analysis

`compare_solomon_methods()` fits several analyses to the same data and lines
up their estimates. In a randomized experiment with a continuous outcome,
they all target the same contrasts; pretest adjustment changes precision
rather than the target (Lin, 2013).

```{r compare}
compare_solomon_methods(
  y_post, treat, pretested, y_pre,
  methods = c("glm", "ml", "classic"),
  data = solomon_example
)
```

Several estimates are identical, but their intervals differ. The
maximum-likelihood analysis of van Engelenburg (1999) uses large-sample Wald
intervals by default. In the package's simulation validation, those
intervals were too narrow when groups were small, so `fit_solomon_ml()` warns
when it is used with small groups and offers a small-sample option:

```{r ml-default}
ml <- fit_solomon_ml(y_post, treat, pretested, y_pre, data = solomon_example)
```

```{r ml-small}
ml_small <- fit_solomon_ml(y_post, treat, pretested, y_pre,
                           inference = "satterthwaite", data = solomon_example)
ml_small
```

## Step 6: Report the results

A report should state the design and group sizes, the estimand, the
analysis and how its uncertainty was computed, the estimate with its
confidence interval, and how sensitization was assessed. For these data:

> Participants were randomly assigned to the four groups of a Solomon
> four-group design (n = 30 per group). The primary analysis was a linear
> model of posttest scores with treatment, pretesting, their interaction, and
> the pretest score in the pretested groups, using HC3 standard errors and t
> tests with `r fit$effects$df[1]` degrees of freedom. The equal-weighted
> average treatment effect was `r fmt(ate$estimate)` points, 95% CI
> [`r fmt(ate$conf.low)`, `r fmt(ate$conf.high)`], p = `r fmt_p(ate$p.value)`.
> The Pretest x Treatment contrast was `r fmt(sens$estimate)` points, 95% CI
> [`r fmt(sens$conf.low)`, `r fmt(sens$conf.high)`]. An equivalence test
> against a prespecified smallest effect of interest of 5 points was
> inconclusive.

`report_solomon()` drafts such a paragraph, with the APA 7 references for
exactly the methods the analysis used. What was pre-specified must be
supplied; the helper never infers it, and the interpretation remains yours:

```{r report}
report_solomon(fit, design = list(prespecified = TRUE))
```

## A second example: when estimators disagree

In `solomon_example`, the pretested groups started out nearly identical and
the pretest predicted the posttest well, so different ways of using the
pretest agreed. The older `solomon_demo` data set shows what can happen
otherwise. Among its pretested participants, the treatment group's mean
pretest is about 4.5 points higher by chance, and the pretest and posttest
are slightly *negatively* correlated.

```{r demo}
data(solomon_demo)
demo_classic <- fit_solomon_classic(y_post, treat, pretested, y_pre,
                                    data = solomon_demo)
data.frame(
  analysis = c("Unadjusted (Test B)", "ANCOVA (Test E)", "Gain score (Test F)"),
  estimate = round(c(
    demo_classic$tests$B$result$estimate,
    demo_classic$tests$E$result$estimate,
    demo_classic$tests$F$result$estimate
  ), 2)
)
```

All three estimate the treatment effect among pretested participants, yet
they disagree. A gain score subtracts the whole pretest difference, which
implicitly assumes a pretest-posttest slope of one; ANCOVA estimates the
slope from the data. In randomized studies ANCOVA has more power than change
scores (Van Breukelen, 2006), which is one reason the recommended analysis
adjusts for the pretest rather than analyzing gains.

## Choosing an analysis

| Goal | Function | Why |
|---|---|---|
| Plan the study and register its analysis | `plan_solomon()`, `analysis_plan_solomon()` | Sample size, then an editable plan for preregistration |
| Check coding, groups, and missing data | `validate_solomon()`, `check_solomon_missing()` | Catch problems before modeling |
| Primary analysis of a continuous outcome | `fit_solomon_glm()` | One prespecified model with explicit contrasts and HC3 standard errors |
| Participants nested in classrooms or sites | `fit_solomon_glm(robust = "CR2", cluster = ...)` | Cluster-robust small-sample tests (Pustejovsky & Tipton, 2018) |
| Test whether sensitization is negligible | `equivalence_solomon()` | A nonsignificant test is not evidence of absence |
| Missing posttests | `fit_solomon_mi()`, `tipping_point_solomon()` | How far missing posttests would have to depart from missing at random to change a conclusion |
| Posttests on several occasions | `fit_solomon_mmrm()` | Valid when participants drop out depending on earlier scores |
| Several treatments and a control | `fit_solomon_glm(control = )` | One model for all the groups, with comparisons adjusted for their number |
| Inference that rests only on random assignment | `perm_solomon()` | Randomization test of no treatment effect |
| Likelihood-based analysis | `fit_solomon_ml()` | Separate residual variances; use `inference = "satterthwaite"` with small groups |
| Outcomes measured by several items | `fit_solomon_sem_latent()` | Latent contrasts under measurement invariance |
| Teach or reproduce the historical analysis | `fit_solomon_classic()` | Tests A-I and the historical decision path |
| See how conclusions depend on the analysis | `compare_solomon_methods()` | Estimates and intervals side by side |

## Reading list

**Historical sources**

- Solomon (1949) introduced the design and the problem it solves.
- Campbell and Stanley (1963/1966, pp. 13, 24–25) placed it among the true
  experimental designs and described the posttest analysis of variance
  followed by ANCOVA.
- Huck and Sandler (1973) discussed appropriate analyses for the design.
- Walton Braver and Braver (1988) proposed combining the pretested and unpretested
  comparisons with Stouffer's method (Test I).
- Sawilowsky et al. (1994) showed by simulation that the conditional sequence
  ending in Test I inflates experiment-wise Type I error.

**Contemporary methodology**

- Lundberg et al. (2021) explain why an analysis should begin by defining its
  estimand.
- Lin (2013) justifies regression adjustment in randomized experiments.
- Long and Ervin (2000) and Hayes and Cai (2007) recommend
  heteroskedasticity-consistent standard errors.
- Pustejovsky and Tipton (2018) describe small-sample inference for clustered
  data.
- Lakens (2017) and Lakens et al. (2018) introduce equivalence testing for
  psychological research.
- Van Breukelen (2006) compares ANCOVA with change scores.
- Little and Rubin (2019) and White and Thompson (2005) cover missing data,
  including missing baseline measurements in randomized trials.
- McCambridge et al. (2011) review the empirical evidence on whether research
  assessments sensitize participants.

All works cited in solomonR are listed, with notes on how the package uses
them, on the [References](https://juhalt.github.io/solomonR/articles/references.html) page.

## References
Campbell, D. T., & Stanley, J. C. (1966). *Experimental and
quasi-experimental designs for research*. Rand McNally. (Original work
published 1963)

Hayes, A. F., & Cai, L. (2007). Using heteroskedasticity-consistent standard
error estimators in OLS regression: An introduction and software
implementation. *Behavior Research Methods, 39*(4), 709–722.
https://doi.org/10.3758/BF03192961

Huck, S. W., & Sandler, H. M. (1973). A note on the Solomon 4-group design:
Appropriate statistical analyses. *The Journal of Experimental Education,
42*(2), 54–55. https://doi.org/10.1080/00220973.1973.11011460

Lakens, D. (2017). Equivalence tests: A practical primer for t tests,
correlations, and meta-analyses. *Social Psychological and Personality
Science, 8*(4), 355–362. https://doi.org/10.1177/1948550617697177

Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence testing for
psychological research: A tutorial. *Advances in Methods and Practices in
Psychological Science, 1*(2), 259–269.
https://doi.org/10.1177/2515245918770963

Lin, W. (2013). Agnostic notes on regression adjustments to experimental
data: Reexamining Freedman's critique. *The Annals of Applied Statistics,
7*(1), 295–318. https://doi.org/10.1214/12-AOAS583

Little, R. J. A., & Rubin, D. B. (2019). *Statistical analysis with missing
data* (3rd ed.). Wiley. https://doi.org/10.1002/9781119482260

Long, J. S., & Ervin, L. H. (2000). Using heteroscedasticity consistent
standard errors in the linear regression model. *The American Statistician,
54*(3), 217–224. https://doi.org/10.1080/00031305.2000.10474549

Lundberg, I., Johnson, R., & Stewart, B. M. (2021). What is your estimand?
Defining the target quantity connects statistical evidence to theory.
*American Sociological Review, 86*(3), 532–565.
https://doi.org/10.1177/00031224211004187

McCambridge, J., Butor-Bhavsar, K., Witton, J., & Elbourne, D. (2011). Can
research assessments themselves cause bias in behaviour change trials? A
systematic review of evidence from Solomon 4-group studies. *PLoS ONE,
6*(10), Article e25223. https://doi.org/10.1371/journal.pone.0025223

Pustejovsky, J. E., & Tipton, E. (2018). Small-sample methods for
cluster-robust variance estimation and hypothesis testing in fixed effects
models. *Journal of Business & Economic Statistics, 36*(4), 672–683.
https://doi.org/10.1080/07350015.2016.1247004

Sawilowsky, S. S., Kelley, D. L., Blair, R. C., & Markman, B. S. (1994).
Meta-analysis and the Solomon four-group design. *The Journal of
Experimental Education, 62*(4), 361–376.
https://doi.org/10.1080/00220973.1994.9944140

Solomon, R. L. (1949). An extension of control group design. *Psychological
Bulletin, 46*(2), 137–150. https://doi.org/10.1037/h0062958

Van Breukelen, G. J. P. (2006). ANCOVA versus change from baseline had more
power in randomized studies and more bias in nonrandomized studies. *Journal
of Clinical Epidemiology, 59*(9), 920–925.
https://doi.org/10.1016/j.jclinepi.2006.02.007

van Engelenburg, G. (1999). *Statistical analysis for the Solomon four-group
design* (Research Report 99-06). University of Twente. ERIC.
https://eric.ed.gov/?id=ED435692

Walton Braver, M. C., & Braver, S. L. (1988). Statistical treatment of the
Solomon four-group design: A meta-analytic approach. *Psychological
Bulletin, 104*(1), 150–154. https://doi.org/10.1037/0033-2909.104.1.150

White, I. R., & Thompson, S. G. (2005). Adjusting for partially missing
baseline measurements in randomized trials. *Statistics in Medicine, 24*(7),
993–1007. https://doi.org/10.1002/sim.1981