---
title: "One Item Set, Both Stages"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{One Item Set, Both Stages}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
library(contentvalidR)
```

## The claim this walkthrough is testing

Content review and empirical screening are usually described as two stages of
one process, and then demonstrated separately, on different data, in different
packages. That arrangement hides the only interesting question: what happens
when the two stages disagree about the same item.

They can disagree in both directions. An item can read badly to a panel and be
caught before anyone collects a response. An item can read perfectly, be kept
by every judge, and then carry almost none of the construct when people answer
it. The second case is the reason a pretest is not a substitute for data.

This vignette follows twelve items through both stages. The data are
**simulated** and were built so that specific items misbehave in specific ways.
Nothing here is a real instrument or a real sample. The advantage of simulated
data is that the right answer is known in advance, so you can check whether
each stage found what was actually put there.

This is the joint walkthrough for `contentvalidR` and its partner package
`nomologR`, which handles the empirical stage. The empirical steps below use
base R, so the whole vignette runs with `contentvalidR` alone. Where the
handoff leaves this package, the walkthrough shows the `nomologR` calls that
pick it up.

## The construct and the items

The construct is *Study Persistence*: the tendency to keep working on academic
tasks when they become difficult or uninteresting. It has two facets, effort
regulation (`EF`) and task focus (`TF`). A third construct, test anxiety
(`TA`), is offered to the judges as a distractor and has no items of its own.

```{r items}
path <- function(f) system.file("extdata", f, package = "contentvalidR")
items <- read.csv(path("walkthrough_items.csv"), stringsAsFactors = FALSE)

items[c("item", "facet", "stem")]
```

Each item was also given a job, and the file records what it is, so you can
check the stages against the design rather than take this vignette's word for
anything. Two are simply written the other way round; five more are meant to
cause trouble:

```{r roles}
subset(items, !startsWith(role, "ordinary"))[c("item", "role")]
```

## Stage one: the expert panel

Twenty judges sorted each item into `EF`, `TF`, or `TA`. `sort_validity()`
applies the exact target-count test of Howard and Melloy (2016): with twenty
judges and two plausible answers, an item needs fifteen assignments to its
target to meet the criterion.

```{r sort}
sorted <- read.csv(path("walkthrough_sort.csv"), stringsAsFactors = FALSE)
panel <- sort_validity(sorted)

panel$results[c("item", "target", "n_target", "competitor", "n_other_max",
                "csv", "recommendation")]
```

Two items are flagged, and for different reasons.

`EF5` ("I get tense when I fall behind on coursework") still drew more
assignments to effort regulation than anywhere else, but eight of twenty judges
read it as test anxiety. Its target wins and the criterion is not met.

`TF5` ("I work hard to stay on top of my reading") is worse: thirteen judges
put it under effort regulation and only six under task focus, so a competing
facet beat the target outright. Its content validity index for sorting is
negative, which is what a negative `csv` means.

```{r flagged}
panel$results[panel$results$recommendation == "Review",
              c("item", "csv", "issue")]
```

`EF6` is worth a second look for the opposite reason. It met the criterion by
exactly one judge:

```{r borderline}
panel$results[panel$results$item == "EF6",
              c("item", "n_target", "critical_n_target", "p_value",
                "recommendation")]
```

A result that close to the line is a reason to look at the item, not a
prediction that it will fail later. Keep that in mind; `EF6` comes back.

## The handoff

`content_handoff()` packages the decision. By default only items whose status
is `Supported` travel forward, so `EF5` and `TF5` stop here.

Two facts about the instrument travel with it as well, and neither is
something the panel could know. Two items are written the other way round, so
that agreeing with them means *less* persistence, and respondents will answer
on a one-to-five scale. The panel sorted items into facets and never saw that
scale, so both have to come from you:

```{r handoff}
h <- content_handoff(panel,
                     reverse_keyed = items$item[items$reverse_worded],
                     response_scale = c(1, 5))

h$items
h$scales
```

Nothing is thrown away. The two flagged items stay in the record, marked as not
carried, and every item carries its keying, the rejected ones included, since a
reader recoding the response file needs all of them:

```{r evidence}
h$item_evidence[c("item", "carried", "status", "recommendation", "keying")]
```

## Stage two: what the responses say

Four hundred people answered all twelve items on a five-point scale. The
response file keeps the items the panel rejected, because the point of a
walkthrough is to be able to see what would have happened had they been kept.

```{r responses}
responses <- read.csv(path("walkthrough_responses.csv"),
                      stringsAsFactors = FALSE)
str(responses[1:5])
```

### Recode first, and see why it matters

Before anything is correlated, the reverse-worded items have to be turned
round. Skip that and look at what happens to `TF2`, "I switch between tasks
while I study":

```{r unrecoded}
raw <- as.matrix(responses[h$items])
corrected <- function(X, item, set) {
  rest <- setdiff(set, item)
  stats::cor(X[, item], rowSums(X[, rest, drop = FALSE]))
}
round(corrected(raw, "TF2", h$scales$TF), 2)
```

A negative corrected item-total correlation is about as strong a case against
an item as item analysis produces, and here it means nothing of the kind. It is
a coding error.
The same number on a correctly coded item would be evidence that the item
measures the opposite of its facet, and the advice in the two cases is
opposite: fix the scoring, or drop the item. The response data cannot tell you
which case you are in. The handoff can, because it recorded the keying:

```{r recode}
ev <- h$item_evidence
key <- setNames(ev$keying, ev$item)[h$items]
top <- ev$response_min[1] + ev$response_max[1]

carried <- raw
carried[, key == -1] <- top - carried[, key == -1]

round(corrected(carried, "TF2", h$scales$TF), 2)
```

Recoding uses the scale's limits, not the lowest and highest answers anyone
gave, which is why the handoff carries them: had nobody chosen 5, reversing
against the observed maximum would shift every recoded answer down a point.

### Two factors

Now take the ten carried items, recoded, and ask for two factors:

```{r factors}
fa <- stats::factanal(carried, factors = 2, rotation = "varimax")

round(unclass(fa$loadings), 2)
```

Seven of the ten behave as the panel expected. Three do not, and none of the
three could have been caught by reading the item.

**`EF4` passed content review and carries almost nothing.** Eighteen of twenty
judges sorted "I keep to the study schedule I set for myself" under effort
regulation, which is a fair reading of the words. It is really about planning,
so it shares little variance with the rest of the facet:

```{r ef4}
round(unclass(fa$loadings)["EF4", ], 2)
```

**`TF4` passed content review and belongs to both facets.** "I keep working
through a task without taking breaks" is as much effort as focus, and the
response data say so even though the judges saw only one of the two:

```{r tf4}
round(unclass(fa$loadings)["TF4", ], 2)
```

And `EF6`, the item that met the content criterion by a single judge, is
unremarkable here:

```{r ef6}
round(unclass(fa$loadings)["EF6", ], 2)
```

A borderline content result and an empirical failure are different things. The
item the panel worried about is fine; the item eighteen of twenty judges agreed
on is the one that fails.

## The item you should keep anyway

So far the empirical stage has been the one that knows better. `EF3` is the
case that stops that from becoming the moral of the story.

```{r ef3}
round(unclass(fa$loadings)["EF3", ], 2)
```

That is low enough for a screening rule written as "flag anything under .40"
to catch it. Compare it with `EF4`, which we have just thrown out, and the two
look like the same kind of problem:

```{r compare}
round(c(EF3 = corrected(carried, "EF3", h$scales$EF),
        EF4 = corrected(carried, "EF4", h$scales$EF)), 2)
```

They are not the same problem, and the correlation cannot tell you that. The
distribution can:

```{r spread}
data.frame(
  sd = round(apply(carried, 2, stats::sd), 2),
  top_two = round(colMeans(carried >= 4), 2)
)
```

`EF3` is "I finish the assignments that count toward my grade." Almost
everybody does: 96% of respondents pick one of the top two categories, and its
standard deviation is less than half of every other item's. A correlation is
bounded by how much the two variables vary, so an item nearly everyone answers
the same way cannot correlate strongly with anything, however well it measures
the construct. `EF4`, by contrast, has an entirely ordinary spread — its low
correlation has no such excuse, and is telling you the item is about something
else.

The content argument then decides it. `EF3` is the only item in the set about
completing required work; every other effort-regulation item is about how the
work *feels* — boring, dull, unappealing. Drop `EF3` for its correlation and
the scale still has five items, but it no longer covers a part of the domain
the panel defined. That is a content-validity loss that no empirical index
reports, because no empirical index knows what the domain was.

Neither stage overrules the other. `EF4` is the case for not trusting a panel
on its own; `EF3` is the case for not trusting a screen on its own.

## What a second stage adds beyond factors

The response file carries a `cohort` variable, and one item answers differently
in the two cohorts:

```{r cohort}
shifts <- vapply(h$items, function(i) {
  m <- tapply(responses[[i]], responses$cohort, mean)
  unname(m["B"] - m["A"])
}, numeric(1))

round(sort(shifts), 2)
```

`TF6` ("I put my phone away while I study") sits well outside the rest. Nothing
about the item's wording predicts that, and no panel could have. Whether the
difference matters is a measurement-invariance question, which belongs to the
empirical stage.

The handoff is what carries the decision into that stage. Pass the handoff
itself rather than `h$items`, so that the held-back items, their reasons, the
keying, and the response scale travel with the names. `nomologR`, the partner
package for the empirical stage, reads it that way. It is not a dependency of
this package, so the calls below are shown rather than run:

```{r nomologr, eval=FALSE}
library(nomologR)

# Screen the carried items, using the keying declared at content review.
screened <- nomo_screen(responses, items = h)

# Form the subscales content review defined, and run the empirical stage.
run <- nomo_run(responses, scales = h)
```

`nomologR`'s article
[From content review to empirical screening](https://juhalt.github.io/nomologR/articles/content-review.html)
continues this walkthrough on the same responses. It runs the empirical stage
with `nomologR`'s own tools, from the raw screen and the recode through the
factor models to invariance across the two cohorts. Its guided workflow
shows the same calls on a handoff, in the section
[Starting from content review](https://juhalt.github.io/nomologR/articles/guided-workflow.html#starting-from-content-review).

## The moral, stated plainly

Surviving content review is evidence about relevance, representation, and
whether experts read an item the way it was meant. It is not evidence that the
item measures anything. Of the ten items this panel carried forward, one
carries almost no common variance and one belongs to a facet the judges never
considered. Both read well. That is not a failure of the panel; it is the
boundary of what a panel can see.

The reverse holds just as firmly. A screening index is a number about a sample,
and it does not know which part of the construct an item was there to cover.
`EF3` would be dropped by a rule and kept by anyone who had read the domain
definition.

So the two stages are worth running in that order not because the second
corrects the first, but because each sees something the other cannot. Content
review removes items no amount of data would rescue, and it removes them before
anyone spends a sample on them. Empirical screening asks the question content
review could not. Neither one gets to decide alone, which is why the handoff
carries the evidence forward rather than just the surviving names.

## Reproducing the data

Every file used here is generated by `data-raw/build-walkthrough-data.R` in the
package sources, which uses only base R and is deterministic. It states the
full generating model: the two-factor structure, every loading, the factor
correlation, the response thresholds, and the single cohort shift.
`nomologR` ships the item and response files unchanged, as
`nomo_demo_walkthrough_items` and `nomo_demo_walkthrough`, together with this
script, and its companion article reads them. Its guided workflow starts from
the same handoff but simulates its own responses, so the screening numbers
there are not the ones shown here.

## References

Howard, M. C., & Melloy, R. C. (2016). Evaluating item-sort task methods: The
presentation of a new statistical significance formula and methodological best
practices. *Journal of Business and Psychology, 31*(1), 173–186.
https://doi.org/10.1007/s10869-015-9404-y