---
title: "Expert-Panel Content Validation: Relevance, Essentiality, and Congruence"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Expert-Panel Content Validation: Relevance, Essentiality, and Congruence}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
library(contentvalidR)
```

## Why expert-panel methods need separate modes

Expert-panel content validation is not one statistical task. A relevance rating,
an essential/not-essential judgment, and an item-objective congruence judgment
ask experts different questions and therefore support different indices.
`expert_validity()` uses an explicit `mode` so those designs are not treated as
interchangeable.

The three modes are:

- **relevance**: Aiken's V plus CVI and modified kappa, with a panel-level
  agreement coefficient;
- **essentiality**: Lawshe's CVR with exact binomial inference; and
- **congruence**: Rovinelli-Hambleton IOC.

All three are quantitative complements to qualitative expert comments, construct
coverage, comprehensibility review, and other parts of the content-validity
argument.

## Relevance: Aiken V, score intervals, and CVI

Suppose six experts rate item relevance from 1 (not relevant) to 4 (highly
relevant):

```{r relevance}
R <- matrix(
  c(4,4,4,4,4,4,
    4,4,4,3,4,4,
    4,3,4,4,3,4,
    3,3,4,3,2,3),
  nrow = 6,
  dimnames = list(NULL, paste0("Item", 1:4))
)

fit <- expert_validity(R, mode = "relevance", lo = 1, hi = 4, seed = 1)
fit
summary(fit)
```

Aiken's V rescales the bounded expert ratings to the 0-1 interval. The default
confidence interval is the score interval proposed by Penfield and Giacobbi
(2004), rather than a simulation-dependent bootstrap interval. Bootstrap
intervals remain available through the low-level function:

```{r aiken-bootstrap}
aikens_v(R, lo = 1, hi = 4, ci = "bootstrap", B = 200, seed = 1)
```

For CVI, the workflow dichotomizes ratings at `relevance_cut`. On a 1-4 scale
the default is 3, so ratings of 3 or 4 count as relevant. Declare a different
threshold if the study protocol used one.

The workflow reports Lynn's (1986) panel-size criterion for the I-CVI as a
**review aid**: with three to five experts all must agree, from six one may
disagree, and from nine two may (7 of 9, the .78 usually quoted). Lynn's table
stops at ten experts; beyond that the package holds her lowest proportion, 7 of
9. The criterion is not presented as universal proof that an item is or is not
content valid. Modified kappa provides
a chance-corrected complement to I-CVI.

At the scale level, S-CVI/Ave and S-CVI/UA are reported together. S-CVI/Ave is
generally less brittle than universal agreement, but both should be interpreted
alongside the distribution of item-level evidence.

## Panel-level agreement

I-CVI and modified kappa describe one item at a time. Relevance mode also
reports how consistently the panel rated the whole item set, as one coefficient
with a bootstrap interval:

```{r agreement}
fit$scale_summary[, c("agreement", "agreement_low", "agreement_high")]
fit$details$agreement
```

The default coefficient is Krippendorff's alpha. It accepts any number of
experts and missing ratings, and Zapf et al. (2016) recommend it when ratings
are ordinal or incomplete, which describes most expert panels. It is a general
reliability coefficient (Hayes & Krippendorff, 2007) rather than one developed
for content validity; no publication applying it specifically to
content-validity panels was found.

Choose the measurement level that matches the rating scale. Relevance ratings
are treated as ordinal by default. `agreement_level = "interval"` treats the
distances between scale points as equal, and `"nominal"` treats every
disagreement as equally serious:

```{r agreement-level}
expert_validity(R, mode = "relevance", lo = 1, hi = 4,
                agreement_level = "interval", agreement_B = 0)$scale_summary$agreement
```

### Why alpha can be low when experts agree

Alpha compares the disagreement within items with the disagreement expected if
the same ratings were scattered across items at random. When a panel rates
nearly every item 4, very little disagreement is expected by chance, so a few 3s
pull alpha down even though most rating pairs are identical. Feinstein and
Cicchetti (1990) described the same pattern for kappa. The output reports the
share of identical rating pairs next to alpha so the two can be read together.
A low alpha alongside a high share of identical pairs is not by itself evidence
of a poor panel.

### Gwet's AC1

Gwet's (2008) AC1 was designed to stay high in that situation, and it is
available with `agreement = "ac1"`. It is never the default. Vach and Gerke
(2023) show that AC1 rises as ratings concentrate in one category even when
agreement does not change, and that it can be above zero when experts rate
independently. Its output always repeats that critique. In relevance mode, AC1
is computed on the relevant/not-relevant decision at `relevance_cut`:

```{r agreement-ac1}
ac1_fit <- expert_validity(R, mode = "relevance", lo = 1, hi = 4,
                           agreement = "ac1", agreement_B = 0)
ac1_fit$details$agreement
```

### The interval

The interval resamples items with all of their ratings intact, the procedure
Zapf et al. (2016) evaluated; they found that Krippendorff's original bootstrap,
which ignores dependence between raters, reached only about 60% coverage. The
interval varies slightly between runs, so set `seed` to make it reproducible, or
set `agreement_B = 0` to skip it. `panel_agreement()` runs the same analysis on
any rater-by-item matrix.

## Essentiality: Lawshe CVR with exact critical values

Lawshe's task asks experts whether an item is essential. With twelve experts:

```{r essentiality}
expert_validity(c(10, 8, 6), mode = "essentiality", N = 12)
```

`cvr()` derives the smallest essential count whose one-sided binomial upper-tail
probability is no greater than `alpha`. This makes the panel-size dependency
explicit and follows the exact-probability logic revisited by Ayre and Scally
(2014).

Judge-by-item binary data can be supplied directly:

```{r essential-matrix}
E <- cbind(
  Item1 = c(1,1,1,1,1,1,1,1),
  Item2 = c(1,1,1,1,1,0,0,0)
)
expert_validity(E, mode = "essentiality")
```

A failure to clear the exact criterion is labeled `Review`, not automatic
deletion. Expert rationales and domain coverage matter when deciding whether an
item should be rewritten, retained for breadth, or removed.

### Lawshe's table and its recalculation, for comparison

Lawshe (1975) published a table of minimum CVR values, and Wilson, Pan and
Schumsky (2012) recalculated it. `legacy = TRUE` prints both beside the exact
decision, without changing it:

```{r essentiality-legacy}
old <- options(contentvalidR.show_key = FALSE)
expert_validity(c(9, 8, 7), mode = "essentiality", N = 9, legacy = TRUE)
options(old)
```

Nine experts show why the table was questioned. Eight of nine gives a CVR of
.778, just under Lawshe's .78, so his table asks for all nine, while with eight
experts it accepts seven. Wilson et al. describe the anomaly: the minimum rises
steadily from 40 experts down to nine, then drops at eight. The exact test
needs eight of nine.

In relevance mode, `legacy = TRUE` adds Fleiss' (1971) kappa on the
relevant/not-relevant decision and the Polit and Beck (2006) benchmarks for
S-CVI/Ave and S-CVI/UA. It also adds the content validity coefficient (Ccv)
of Hernández-Nieto (2002), which is widely cited, beside Aiken's V. Read it as a
cautionary example rather than evidence. Ccv is the mean rating divided by the
scale maximum, minus a "chance" term, (1/J)^J, that depends only on the number
of judges and is .00032 for five of them. So on a scale starting at 0 it is
Aiken's V, and on a scale starting at 1 it cannot fall below one over the
scale maximum. It uses only the mean, so despite the book's claim it cannot
reflect agreement: in the book's own Table 7, ratings of 1, 3, 4, 5, 2 and of
3, 3, 3, 3, 3 both score .60. Its .80 cutoff is stated without derivation, and
the book's own examples do not keep to it. `?expert_validity` sets out each of
these with page references.

## Congruence: item-objective alignment

IOC uses expert ratings of -1, 0, and +1 for item-objective congruence. A target
mapping lets the workflow compare intended and competing objectives:

```{r congruence}
d <- expand.grid(
  item = c("I1", "I2"),
  judge = 1:4,
  objective = c("A", "B")
)
d$target_objective <- ifelse(d$item == "I1", "A", "B")
d$score <- ifelse(d$objective == d$target_objective, 1, -1)

expert_validity(d, mode = "congruence")
```

The workflow reports target IOC, the strongest competitor, and their margin.
This is a diagnostic comparison, not a manufactured significance test. If no
target mapping is provided, all IOC cells are returned descriptively.

## Missing ratings

Missing data are never silently ignored by default. Set `na.rm = TRUE` only when
itemwise/cellwise deletion matches the study protocol. Effective expert counts
and missing counts are then reported so downstream interpretation uses the
actual panel size.

## Plotting expert evidence

Each mode uses a plot matched to the expert task rather than forcing unlike
indices into one generic chart.

```{r expert-plots, fig.width=7, fig.height=4, fig.alt=c("Expert relevance plot: for each item, Aiken's V (filled) and I-CVI (open), each with its 95% interval, and a dashed line at the I-CVI criterion for the panel size.", "Essentiality plot: each item's observed CVR (filled) joined to the CVR the exact test needs (open), with a dotted line at zero.", "Congruence plot: each item's IOC for its intended objective (filled) joined to its IOC for the strongest competing objective (open), with a dotted line at zero.")}
plot(expert_validity(R, mode = "relevance", lo = 1, hi = 4))
plot(expert_validity(c(10, 8, 6), mode = "essentiality", N = 12))
plot(expert_validity(d, mode = "congruence"))
```

Relevance mode displays Aiken's V with its score interval and overlays I-CVI as
a separate marker. Essentiality mode displays observed CVR against the exact
panel-specific critical CVR. Congruence mode connects target IOC to the strongest
competitor so the alignment margin is visually explicit. These displays are
diagnostic summaries; they do not create new validity thresholds.

An index reduces each item's ratings to one number, and the relevance plot
shows that number. The distribution view shows the ratings behind it, as
diverging stacked bars (Heiberger & Robbins, 2014). Each bar splits at the
relevance cut: ratings below it extend left and ratings at or above it extend
right, so the right-hand length is the I-CVI, read against the dashed criterion
line.

```{r expert-distribution, fig.width=7, fig.height=4, fig.alt="Rating distributions for four items rated by six experts, as bars split at the relevance cut of 3, with a dashed line at the I-CVI criterion of .83. Items 1 to 3 are rated relevant by all six experts, Item 1 entirely with 4s; Item 4 is rated relevant by five of six, mostly with 3s."}
plot(fit, type = "distribution",
     labels = c("Not relevant", "Somewhat", "Quite", "Highly relevant"))
```

Item1 and Item3 have the same I-CVI, but every expert gave Item1 a 4, while
two gave Item3 a 3. The index cannot tell them apart; the bars can, and so can
Aiken's V, which uses the whole scale. The figure is gray by default, as an APA
figure is printed. For slides or a poster, `apa = FALSE` draws it in color:
brown for ratings below the cut and teal for ratings at or above it. The
symbol beside each bar still carries the decision, so no reading depends on
color.

## Reporting

A concise methods/results description should identify:

1. who the experts were and why they were qualified;
2. the exact task and response scale;
3. the index and inference/CI procedure used, and for relevance ratings the
   agreement coefficient and its measurement level;
4. the panel size, including item-specific missingness;
5. quantitative item and scale evidence; and
6. how expert comments, construct coverage, and comprehensibility informed the
   final item decisions.

A content-validity coefficient is evidence about a defined expert task. It is
not, by itself, a complete validity argument.

## References

Aiken, L. R. (1980). Content validity and reliability of single items or
questionnaires. *Educational and Psychological Measurement, 40*(4), 955–959. https://doi.org/10.1177/001316448004000419

Ayre, C., & Scally, A. J. (2014). Critical values for Lawshe's content validity
ratio: Revisiting the original methods of calculation. *Measurement and
Evaluation in Counseling and Development, 47*(1), 79–86.
https://doi.org/10.1177/0748175613513808

Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I.
The problems of two paradoxes. *Journal of Clinical Epidemiology, 43*(6),
543–549. https://doi.org/10.1016/0895-4356(90)90158-L

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters.
*Psychological Bulletin, 76*(5), 378–382. https://doi.org/10.1037/h0031619

Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the
presence of high agreement. *British Journal of Mathematical and Statistical
Psychology, 61*(1), 29–48. https://doi.org/10.1348/000711006X126600

Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard
reliability measure for coding data. *Communication Methods and Measures, 1*(1),
77–89. https://doi.org/10.1080/19312450709336664

Heiberger, R. M., & Robbins, N. B. (2014). Design of diverging stacked bar
charts for Likert scales and other applications. *Journal of Statistical
Software, 57*(5), 1–32. https://doi.org/10.18637/jss.v057.i05

Hernández-Nieto, R. (2002). *Contributions to statistical analysis: The
coefficients of proportional variance, content validity and kappa*.
BookSurge.

Krippendorff, K. (2011). *Computing Krippendorff's alpha-reliability*. Annenberg
School for Communication, University of Pennsylvania.
https://www.asc.upenn.edu/sites/default/files/2021-03/Computing%20Krippendorff%27s%20Alpha-Reliability.pdf

Lawshe, C. H. (1975). A quantitative approach to content validity. *Personnel
Psychology, 28*(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x

Lynn, M. R. (1986). Determination and quantification of content validity.
*Nursing Research, 35*(6), 382–385.
https://doi.org/10.1097/00006199-198611000-00017

Penfield, R. D., & Giacobbi, P. R., Jr. (2004). Applying a score confidence
interval to Aiken's item content-relevance index. *Measurement in Physical
Education and Exercise Science, 8*(4), 213–225.
https://doi.org/10.1207/S15327841MPEE0804_3

Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure
you know what's being reported? Critique and recommendations. *Research in
Nursing & Health, 29*(5), 489–497. https://doi.org/10.1002/nur.20147

Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable
indicator of content validity? Appraisal and recommendations. *Research in
Nursing & Health, 30*(4), 459–467. https://doi.org/10.1002/nur.20199

Rovinelli, R. J., & Hambleton, R. K. (1977). On the use of content specialists
in the assessment of criterion-referenced test item validity. *Dutch Journal of
Educational Research, 2*, 49–60.

Turner, R. C., & Carlson, L. (2003). Indexes of item-objective congruence for
multidimensional items. *International Journal of Testing, 3*(2), 163–171.
https://doi.org/10.1207/S15327574IJT0302_5

Vach, W., & Gerke, O. (2023). Gwet's AC1 is not a substitute for Cohen's kappa:
A comparison of basic properties. *MethodsX, 10*, 102212.
https://doi.org/10.1016/j.mex.2023.102212

Wilson, F. R., Pan, W., & Schumsky, D. A. (2012). Recalculation of the critical
values for Lawshe's content validity ratio. *Measurement and Evaluation in
Counseling and Development, 45*(3), 197–210.
https://doi.org/10.1177/0748175612440286

Zapf, A., Castell, S., Morawietz, L., & Karch, A. (2016). Measuring inter-rater
reliability for nominal data: Which coefficients and confidence intervals are
appropriate? *BMC Medical Research Methodology, 16*, 93.
https://doi.org/10.1186/s12874-016-0200-9