---
title: "Manuscript-Ready Reporting Examples"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Manuscript-Ready Reporting Examples}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
library(contentvalidR)
read_example <- function(name) {
  utils::read.csv(
    system.file("extdata", name, package = "contentvalidR"),
    stringsAsFactors = FALSE
  )
}
```

## Purpose

This vignette provides reporting scaffolds for the three flagship workflows. The
examples are intentionally conservative: statistical screening is described as
**evidence for retention or review**, not as proof that an item or scale is
content valid. Final decisions should also consider construct-domain coverage,
item wording, qualitative judge feedback, and the intended use of the measure.

The bundled CSV files are synthetic, deterministic, and regenerated from
`data-raw/build-example-data.R`. They are useful for reproducing the examples
and for seeing the expected input shape before analyzing a new study.

## Item-sort study

```{r sort-fit}
sort_dat <- read_example("sort_example.csv")
sort_fit <- sort_validity(sort_dat)
sort_sum <- summary(sort_fit)
sort_fit$results
sort_fit$scale_summary
```

### Methods scaffold

Report who completed the sort, how the construct definitions were presented,
the available assignment alternatives, and the a priori screening rule. A
concise methods statement can follow this structure:

> Candidate items were evaluated in an item-sort pretest in which judges
> assigned each item to the construct definition that best represented its
> content. We quantified definitional correspondence using the proportion of
> substantive agreement (Psa) and definitional distinctiveness using the
> substantive-validity coefficient (Csv). Item-level screening used the exact
> target-assignment test described by Howard and Melloy (2016), with the null
> target-assignment probability and alpha specified a priori. Target-scale mean
> Psa and Csv were interpreted against Colquitt et al. (2019) norms only when the
> judge population matched the intended use of those norms.

### Results scaffold

For this bundled example, `r sort_fit$design$n_raters` judges evaluated
`r sort_fit$design$n_items` items. `r sort_sum$n_supported` items met the exact
screening criterion, `r sort_sum$n_review` were flagged for review, and
`r sort_sum$n_insufficient` had insufficient usable assignments.

A manuscript table can usually be built directly from:

```{r sort-table}
sort_fit$results[c(
  "item", "target", "n", "n_target", "competitor",
  "psa", "csv", "p_value", "status", "recommendation"
)]
```

Do not report `Review` as synonymous with deletion. A review flag identifies an
item for substantive inspection; retaining an item for domain coverage can be a
reasonable decision when that rationale is documented.

## Construct-rating study

```{r rating-fit}
rating_dat <- read_example("rating_example.csv")
rating_fit <- rating_validity(rating_dat, scale_min = 1, scale_max = 5)
rating_sum <- summary(rating_fit)
rating_fit$results
rating_fit$scale_summary
```

### Methods scaffold

> Judges rated every candidate item against each focal and orbiting construct
> definition using the same response scale. We summarized correspondence with
> HTC and distinctiveness with HTD. Because the same judges rated the competing
> definitions, item-level inference used a repeated-measures design. Planned
> paired contrasts compared each item's intended definition with every orbiting
> definition; omnibus Greenhouse-Geisser-corrected inference was used when
> applicable. Scale-level HTC/HTD norms from Colquitt et al. (2019) were treated
> as empirical benchmarks rather than universal item cutoffs.

### Results scaffold

The example contains `r rating_fit$design$n_items` items rated by
`r rating_fit$design$n_raters` judges against
`r rating_fit$design$n_constructs_observed` construct definitions.
`r rating_sum$n_supported` items were supported by the complete screening rule
and `r rating_sum$n_review` were flagged for review.

```{r rating-table}
rating_fit$results[c(
  "item", "target", "n_complete", "strongest_competitor",
  "htc", "htd", "p_value", "max_contrast_p", "status", "recommendation"
)]
```

For review items, report the strongest orbiting competitor. That information
turns a generic statement about weak distinctiveness into a specific diagnostic
about where construct overlap may be occurring.

## Expert-panel study

### Relevance

```{r expert-relevance}
expert_rel <- read_example("expert_relevance_example.csv")
expert_rel_matrix <- as.matrix(expert_rel[setdiff(names(expert_rel), "expert")])
expert_fit <- expert_validity(
  expert_rel_matrix,
  mode = "relevance",
  lo = 1,
  hi = 4
)
expert_sum <- summary(expert_fit)
expert_fit$results
expert_fit$scale_summary
```

> Experts rated the relevance of each candidate item on a bounded ordinal
> scale. We summarized relevance using Aiken's V with Penfield-Giacobbi score
> confidence intervals and calculated I-CVI with Polit-Beck-Owen modified kappa.
> S-CVI/Ave and S-CVI/UA were reported at the scale level. Panel-size CVI
> guidelines were used as review aids and were considered alongside written
> expert feedback and construct coverage.

In this example, `r expert_fit$design$n_judges` experts evaluated
`r expert_fit$design$n_items` items. The workflow identifies
`r expert_sum$n_supported` supported items and `r expert_sum$n_review` review
items under its quantitative rules.

### Essentiality

```{r expert-essentiality}
expert_ess <- read_example("expert_essentiality_example.csv")
expert_ess_matrix <- as.matrix(expert_ess[setdiff(names(expert_ess), "expert")])
ess_fit <- expert_validity(expert_ess_matrix, mode = "essentiality")
ess_fit$results
```

For an essentiality task, state that CVR is tied to a different expert judgment
than relevance. Report the effective panel size and exact critical essential
count for each item rather than borrowing a relevance/CVI threshold.

### Congruence

```{r expert-congruence}
expert_ioc <- read_example("expert_congruence_example.csv")
ioc_fit <- expert_validity(expert_ioc, mode = "congruence")
ioc_fit$results
```

For IOC, report the intended objective, target IOC, strongest competing
objective, and target-minus-competitor margin. The margin is diagnostic evidence
about alignment; it is not a newly invented significance test.

## Figures across review stages

Most studies gather more than one kind of content evidence. `content_evidence()`
brings the stages together, in the order they ran, and draws two figures for a
paper or poster. Here the twelve items of `vignette("one-item-set-both-stages")`
are first rated for relevance by eight experts, and the items the panel carries
go on to the item sort. The relevance ratings are constructed for this example;
`data-raw/build-walkthrough-panel.R` says what each item's ratings are built to
show.

```{r evidence}
relevance <- read_example("walkthrough_relevance.csv")
panel <- content_handoff(expert_validity(
  as.matrix(relevance[setdiff(names(relevance), "expert")]),
  mode = "relevance", lo = 1, hi = 4, agreement = "none"
))
sorted <- read_example("walkthrough_sort.csv")
sort_stage <- sort_validity(sorted[sorted$item %in% panel$items, ])
evidence <- content_evidence(`Relevance panel` = panel, `Item sort` = sort_stage)
evidence
```

The flow diagram follows the items through the stages. It is modeled on the
PRISMA 2020 flow diagram for systematic reviews (Page et al., 2021), with items
in place of studies, and it is the figure a Method section needs: what each
stage reviewed, what it held back and why, and what went forward.

```{r evidence-flow, fig.width = 7.5, fig.height = 3.2, fig.alt = "Item flow diagram. Stage 1, the relevance panel, reviewed 12 items with 8 experts and held back EF5, with an I-CVI of .50 against a criterion of .88. Stage 2, the item sort, reviewed the remaining 11 items with 20 judges and held back TF5, with a Psa of .30 against a criterion of .75. Ten items were carried forward, five in each facet."}
plot(evidence, type = "flow")
```

The evidence profile sets the stages side by side, item by item. Each panel
shows the statistic that stage's decision read, its 95% interval, and its
criterion as a dashed line; a filled symbol met the criterion and an open one
was flagged for review.

```{r evidence-profile, fig.width = 7.5, fig.height = 5.5, fig.alt = "Item evidence profile with one panel per stage. Left, the I-CVI for twelve items with 95% intervals and a dashed criterion at .88; every item meets it except EF5, at .50. Right, Psa for the eleven items the panel carried, with a dashed criterion at .75; every item meets it except TF5, at .30, and EF6 sits exactly on the line. A last column marks EF5 as held back by the relevance panel and TF5 by the item sort."}
plot(evidence)
```

Read across a row and the sources can disagree. Every expert rated `TF5`
relevant; only the sort shows that judges place it in the other facet. Read
down a panel and the intervals speak: an I-CVI of 1.00 from eight experts has
an interval reaching down to about .68, a reminder of how small a panel is.

Both figures are gray by default, as an APA figure is printed. For slides or a
poster, `apa = FALSE` draws them in color, teal for evidence that met its
criterion and brown for evidence under review. The symbols carry the same
decisions, so no reading depends on color.

```{r evidence-color, fig.width = 7.5, fig.height = 5.5, fig.alt = "The same item evidence profile in color: items that met each criterion in teal, and EF5 in the relevance panel and TF5 in the item sort in brown, with open symbols."}
plot(evidence, apa = FALSE)
```

A caption for the profile might read: *Content evidence for twelve draft
items. Left: the share of eight experts rating each item relevant (I-CVI), with
95% Wilson score intervals and Lynn's (1986) criterion of 7 of 8. Right: the
share of 20 judges assigning each item to its intended facet (Psa), with 95%
Wilson score intervals and the exact test's criterion of 15 of 20 (Howard &
Melloy, 2016). Filled symbols met the criterion; open symbols were flagged for
review.*

## Minimum reproducibility statement

At minimum, a manuscript or supplement should identify the package version and
record the analysis settings that determine the results. For a fitted workflow:

```{r reproducibility}
packageVersion("contentvalidR")
sort_fit$settings
sort_fit$design
```

A strong reproducibility supplement should also archive the item wording,
construct definitions, judge instructions, anonymized response data when
permitted, and the script that reproduces all tables and figures.

## Language to avoid

Avoid claims such as "the scale was proven content valid" or "items failing the
cutoff were invalid." The workflows quantify evidence from a defined pretest.
They do not replace a theory-based definition of the construct domain or the
researcher's responsibility to justify substantive item decisions.

## References

Colquitt, J. A., Sabey, T. B., Rodell, J. B., & Hill, E. T. (2019). Content
validation guidelines: Evaluation criteria for definitional correspondence and
definitional distinctiveness. *Journal of Applied Psychology, 104*(10),
1243–1265. https://doi.org/10.1037/apl0000406

Howard, M. C., & Melloy, R. C. (2016). Evaluating item-sort task methods: The
presentation of a new statistical significance formula and methodological best
practices. *Journal of Business and Psychology, 31*(1), 173–186.
https://doi.org/10.1007/s10869-015-9404-y

Lynn, M. R. (1986). Determination and quantification of content validity.
*Nursing Research, 35*(6), 382–385.
https://doi.org/10.1097/00006199-198611000-00017

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C.,
Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou,
R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T.,
Loder, E. W., Mayo-Wilson, E., McDonald, S., . . . Moher, D. (2021). The PRISMA
2020 statement: An updated guideline for reporting systematic reviews. *BMJ,
372*, Article n71. https://doi.org/10.1136/bmj.n71