Use the design that matches the question posed to judges rather than selecting an index after data collection.
sort_validity() for
Psa/Csv, exact target-count screening, strongest-competitor diagnostics,
and scale-level Colquitt norms.rating_validity() for HTC/HTD, repeated-measures inference,
planned target-versus-orbiting contrasts, and scale-level norms.expert_validity()
with the corresponding explicit mode.delphi_validity() for consensus against a threshold fixed
before the study, and stability between rounds.The designs can complement one another during scale development, but
their statistics are not interchangeable. Two further questions apply to
any of them: whether the conclusions depend on the particular judges who
served (judge_validity()), and whether the items cover the
blueprint of the domain (domain_validity()).
For item sorts, plan judge N in relation to the exact retention rule
and a plausible true target-assignment probability.
sort_power() provides exact planning probabilities; avoid a
universal judge-count rule of thumb.
For construct-rating studies, power depends on the number of judges, number of construct definitions, within-judge target-orbiting separation, and missing profiles. Report the effective complete-judge N itemwise.
For expert panels, panel-size sensitivity is part of the statistic. CVR exact critical counts and common CVI review guidelines therefore need to be interpreted with the actual effective N, not a nominal panel size that ignores missingness.
The reporting-examples vignette expands this sequence
into reusable methods and results scaffolds. Treat those examples as
reporting patterns rather than fixed language that must be copied
verbatim.
sort_dat <- read_example("sort_example.csv")
rating_dat <- read_example("rating_example.csv")
expert_rel <- read_example("expert_relevance_example.csv")
sort_fit <- sort_validity(sort_dat)
rating_fit <- rating_validity(rating_dat, scale_min = 1, scale_max = 5)
expert_fit <- expert_validity(
as.matrix(expert_rel[setdiff(names(expert_rel), "expert")]),
mode = "relevance", lo = 1, hi = 4
)These files are synthetic and generated by
data-raw/build-example-data.R in the source repository.
They deliberately include both supported and review-worthy items so
documentation exercises realistic output paths without depending on
random-number generation.
sort_fit$results[c(
"item", "target", "n", "n_target", "competitor",
"psa", "csv", "p_value", "status", "recommendation"
)]
#> item target n n_target competitor psa csv p_value status
#> 1 A1 A 20 18 B; C 0.90 0.85 0.0002012253 Supported
#> 2 A2 A 20 15 B 0.75 0.60 0.0206947327 Supported
#> 3 B1 B 20 17 A 0.85 0.75 0.0012884140 Supported
#> 4 B2 B 20 13 A 0.65 0.40 0.1315879822 Review
#> 5 C1 C 20 18 A; B 0.90 0.85 0.0002012253 Supported
#> 6 C2 C 20 14 B 0.70 0.50 0.0576591492 Review
#> recommendation
#> 1 Retain
#> 2 Retain
#> 3 Retain
#> 4 Review
#> 5 Retain
#> 6 ReviewAt the target-scale level, report mean Psa/Csv and the benchmark set actually used. Do not convert Colquitt’s scale-level norms into individual-item cutoffs.
rating_fit$results[c(
"item", "target", "n_complete", "strongest_competitor",
"htc", "htd", "p_value", "max_contrast_p", "status", "recommendation"
)]
#> item target n_complete strongest_competitor htc htd p_value
#> 1 A1 A 24 C 0.9333333 0.6666667 1.274299e-19
#> 2 A2 A 24 B 0.7333333 0.2083333 7.375102e-08
#> 3 B1 B 24 C 0.9333333 0.6666667 1.274299e-19
#> 4 B2 B 24 A 0.7333333 0.2083333 7.375102e-08
#> 5 C1 C 24 B 0.9333333 0.6666667 1.274299e-19
#> 6 C2 C 24 B 0.7333333 0.2083333 7.375102e-08
#> max_contrast_p status recommendation
#> 1 1.122784e-13 Supported Retain
#> 2 5.173102e-02 Review Review
#> 3 1.122784e-13 Supported Retain
#> 4 5.173102e-02 Review Review
#> 5 1.122784e-13 Supported Retain
#> 6 5.173102e-02 Review Review
rating_fit$scale_summary
#> target n_items n_htc n_htd n_retain n_review n_insufficient mean_htc
#> 1 A 2 2 2 1 1 0 0.8333333
#> 2 B 2 2 2 1 1 0 0.8333333
#> 3 C 2 2 2 1 1 0 0.8333333
#> htc_strength mean_htd htd_strength overall_strength orbiting_r benchmark_set
#> 1 Weak 0.4375 Very Strong Weak NA overall
#> 2 Weak 0.4375 Very Strong Weak NA overall
#> 3 Weak 0.4375 Very Strong Weak NA overall
#> evidence
#> 1 The weaker of HTC and HTD falls in the Weak band of published scales (Colquitt et al., 2019); review item wording, construct boundaries, and the choice of orbiting constructs, and consider pretesting the revised items again.
#> 2 The weaker of HTC and HTD falls in the Weak band of published scales (Colquitt et al., 2019); review item wording, construct boundaries, and the choice of orbiting constructs, and consider pretesting the revised items again.
#> 3 The weaker of HTC and HTD falls in the Weak band of published scales (Colquitt et al., 2019); review item wording, construct boundaries, and the choice of orbiting constructs, and consider pretesting the revised items again.Report the repeated-measures design and target-versus-orbiting contrasts. For a review item, naming the strongest competitor is often more informative than a standalone p value.
expert_fit$results[c(
"item", "N", "V", "ci_low", "ci_high", "I_CVI",
"kappa_mod", "status", "recommendation"
)]
#> item N V ci_low ci_high I_CVI kappa_mod status
#> 1 Item1 8 1.0000000 0.8620238 1.0000000 1.00 1.0000000 Supported
#> 2 Item2 8 0.9166667 0.7415120 0.9768412 1.00 1.0000000 Supported
#> 3 Item3 8 0.8333333 0.6414693 0.9332132 1.00 1.0000000 Supported
#> 4 Item4 8 0.5833333 0.3883467 0.7553240 0.75 0.7192982 Review
#> 5 Item5 8 0.4166667 0.2446760 0.6116533 0.25 0.1578947 Review
#> recommendation
#> 1 Strong support
#> 2 Strong support
#> 3 Strong support
#> 4 Review
#> 5 Review
expert_fit$scale_summary
#> n_items n_experts_min n_experts_max mean_Aiken_V S_CVI_Ave S_CVI_UA agreement
#> 1 5 8 8 0.75 0.8 0.6 0.6934253
#> agreement_low agreement_high n_strong_support n_support n_review
#> 1 0.07142857 0.8457143 3 0 2
#> n_insufficient
#> 1 0Essentiality and congruence require different expert tasks. Do not place CVR, CVI, Aiken V, and IOC in one generic threshold table as if they answer the same question.
signal_detection() and
reproducibility_phi() remain auxiliary helpers. They can be
useful when researchers later compare pretest decisions with CFA/IRT
retention or an independent replication pretest, but they are not
required parts of the flagship workflows.
pretest_supported <- sort_fit$results$status == "Supported"
later_retained <- c(TRUE, TRUE, TRUE, FALSE, TRUE, FALSE)
signal_detection(pretest_supported, later_retained)
#> Retention decisions compared with the actual outcome
#>
#> Actual
#> Predicted Retain Not retained
#> Retain 4 0
#> Not retained 0 2
#>
#> accuracy = 1.00, sensitivity = 1.00, specificity = 1.00, phi = 1.00,
#> chi-square(1) = 6.00, p = .014.
replication_supported <- c(TRUE, TRUE, TRUE, FALSE, TRUE, TRUE)
reproducibility_phi(pretest_supported, replication_supported)
#> Retention decisions in two pretests
#>
#> Pretest2
#> Pretest1 Retain Not retained
#> Retain 4 0
#> Not retained 1 1
#>
#> phi = .63, chi-square(1) = 2.40, p = .121.Each planning function answers the question for one design exactly, from the decision rule the analysis will apply, rather than from a rule of thumb.
For an item sort, sort_power() gives the probability
that an item reaches the exact retention count, for each panel size and
each rate at which you expect judges to choose the target:
sort_power(N = c(20, 30, 40), true_p = c(.60, .70, .80))
#> contentvalidR item-sort planning
#> --------------------------------
#> Retention rule: Howard-Melloy exact test (p0 = .50, alpha = .05)
#>
#> judges required minimum Psa power at .60 power at .70 power at .80
#> 20 15/20 .75 .13 .42 .80
#> 30 20/30 .67 .29 .73 .97
#> 40 26/40 .65 .32 .81 .99
#>
#> required: target assignments an item needs to be retained. minimum Psa: the
#> same as a proportion. power at p: the exact probability of reaching the
#> required count if each judge assigns the item to its target with probability
#> p.For an expert panel, expert_power() gives the
probability that an item clears the I-CVI criterion, for each panel size
and an assumed probability that one expert endorses it:
expert_power(n_experts = 3:8, prob = 0.9)
#> contentvalidR expert-panel planning
#> -----------------------------------
#> Criterion: Exact binomial planning against the panel-size I-CVI guideline
#>
#> Probability that an item clears the criterion
#> experts required prob = .90
#> 3 3/3 .73
#> 4 4/4 .66
#> 5 5/5 .59
#> 6 5/6 .89
#> 7 6/7 .85
#> 8 7/8 .81
#>
#> required: endorsements the criterion needs from the panel. prob: the
#> probability you assume that one expert endorses the item.
#>
#> Note the step at six experts. Lynn's criterion requires unanimity with three
#> to five experts and allows one disagreement from six, so a sixth expert
#> relaxes the criterion while a fourth or fifth makes unanimity harder. That is
#> a property of the guideline, not of the items.
#>
#> This table reports the consequences of the panel sizes you asked about. It
#> does not recommend one. `prob` is an assumption you supply, so treat the
#> result as conditional on it and report the value you assumed.Note the step. Lynn’s (1986) criterion requires every expert to agree on panels of up to five, and allows one disagreement from six, so a fourth or fifth expert lowers the probability of clearing while a sixth raises it sharply. That is a property of the guideline, not of the items, and the package reports it rather than smoothing it away.
When a panel has already rated the items, generalizability theory (Brennan, 2001; Crocker et al., 1988) estimates how many judges the same design would need to reach a given dependability:
judge_ratings <- rbind(
c(4, 4, 4, 3, 2, 2), c(4, 4, 3, 4, 2, 1), c(4, 3, 4, 4, 1, 2),
c(3, 4, 4, 4, 2, 2), c(4, 4, 4, 4, 2, 1), c(4, 3, 4, 3, 1, 2),
c(4, 4, 3, 4, 2, 2), c(2, 2, 2, 2, 1, 1)
)
dimnames(judge_ratings) <- list(paste0("Judge", 1:8), paste0("Item", 1:6))
gtheory_content(judge_ratings)
#> contentvalidR generalizability analysis
#> ---------------------------------------
#> Design: items x judges, crossed, one rating per cell
#> Items: 6 | Judges: 8
#>
#> Observed design
#> Generalizability coefficient (relative, rank ordering): .97
#> Dependability coefficient (absolute, fixed standard): .94
#> Status: Supported
#> With 8 judges, absolute decisions about these items would generalize
#> dependably to another panel of the same size (Phi = .94). Judge differences
#> account for 15.6% of total variance.
#>
#> Variance components
#> source df MS estimate used % of total
#> item 5 7.77 0.94 0.94 66.8
#> judge 7 1.57 0.22 0.22 15.6
#> residual 35 0.25 0.25 0.25 17.5
#>
#> estimate: the ANOVA estimate of each variance component. used: the same with
#> a negative estimate set to 0, which the coefficients use (Brennan, 2001).
#>
#> Judges needed to reach each coefficient
#> target relative (G) absolute (Phi)
#> .70 1 2
#> .80 2 2
#> .90 3 5
#>
#> A dependability coefficient describes generalization over judges only. It is
#> not evidence that the items cover the intended content domain.The dependability coefficient (Phi) concerns the absolute level of
the ratings and is lowered by differences in judge severity, which is
usually what a content-validity decision rests on. A target no realistic
panel reaches shows as NA: the judges barely distinguished
the items.
Across pretest rounds, compare_rounds() sets each item’s
status side by side and checks whether the analysis settings changed
between rounds, so a relaxed criterion cannot read as an improved
item.
Pre-register quantitative screening criteria when feasible, preserve qualitative judge feedback, report effective N after missingness, and archive the construct definitions and item wording used in the pretest. A content-validation statistic is evidence from a designed judgment task; it is not a substitute for defining and sampling the construct domain.
Brennan, R. L. (2001). Generalizability theory. Springer. https://doi.org/10.1007/978-1-4757-3456-0
Crocker, L., Llabre, M., & Miller, M. D. (1988). The generalizability of content validity ratings. Journal of Educational Measurement, 25(4), 287–299. https://doi.org/10.1111/j.1745-3984.1988.tb00309.x
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. https://doi.org/10.1097/00006199-198611000-00017