Construct-Rating Content Validation: Hinkin-Tracey to Colquitt

library(contentvalidR)

What the rating workflow asks

The Hinkin and Tracey (1999) content-rating procedure asks judges to evaluate how well each item corresponds to each construct definition under consideration. The typical design is fully crossed within judges: the same judge rates an item against the intended definition and against one or more orbiting definitions.

That design provides two complementary kinds of evidence:

Definitional correspondence: does the item strongly match its intended construct?
Definitional distinctiveness: does it match the intended construct more strongly than plausible orbiting constructs?

contentvalidR keeps those questions separate rather than reducing the study to a single coefficient.

Example data

rating_dat <- expand.grid(
  item = c("A1", "A2", "A3", "B1"),
  rater = 1:24,
  construct = c("A", "B", "C")
)
rating_dat$target_construct <- ifelse(rating_dat$item == "B1", "B", "A")

rating_dat$rating <- ifelse(
  rating_dat$construct == rating_dat$target_construct,
  pmin(5, pmax(1, round(rnorm(nrow(rating_dat), 4.4, .6)))),
  pmin(5, pmax(1, round(rnorm(nrow(rating_dat), 2.2, .8))))
)

Each item-judge combination appears once for every construct definition. A duplicated item-rater-construct row is treated as a data error.

HTC: definitional correspondence

Following Colquitt et al. (2019), the Hinkin-Tracey correspondence index is

\[ HTC = \frac{\bar{x}_{target}}{a}, \]

where \(a\) is the number of response anchors when ratings use a 1-to-\(a\) scale. contentvalidR can also accept an equally spaced integer scale such as 0-to-4; it shifts that scale internally to the equivalent 1-to-5 anchor metric before computing HTC.

htc(rating_dat, scale_min = 1, scale_max = 5)
#> Hinkin-Tracey correspondence (HTC; Colquitt et al., 2019)
#> 
#>  item target judges target mean HTC
#>    A1      A     24        4.38 .88
#>    A2      A     24        4.46 .89
#>    A3      A     24        4.42 .88
#>    B1      B     24        4.58 .92
#> 
#> HTC expresses the mean target rating as a share of the 5-point scale.

Higher HTC means stronger correspondence with the intended definition.

HTD: definitional distinctiveness

HTD compares intended-definition ratings with orbiting-definition ratings:

\[ HTD = \frac{\text{average}(x_{target} - x_{orbiting})}{a - 1}. \]

It ranges from -1 to 1. Positive values favor the intended definition; negative values indicate that orbiting definitions are rated more highly on average.

htd(rating_dat, scale_min = 1, scale_max = 5)
#> Hinkin-Tracey distinctiveness (HTD; Colquitt et al., 2019)
#> 
#>  item target judges target mean competitor competitor mean HTD
#>    A1      A     24        4.38          B            2.38 .56
#>    A2      A     24        4.46          B            2.00 .63
#>    A3      A     24        4.42          C            2.25 .55
#>    B1      B     24        4.58          C            2.17 .61
#> 
#> competitor: the other construct with the highest mean rating. HTD itself
#> averages the gap over every other construct.

The item-level table also identifies the strongest orbiting competitor. That is often more useful for revision than merely knowing that distinctiveness is weak.

Repeated-measures item screening

The same judges provide multiple construct ratings, so those observations are not independent. anova_content() uses a one-way repeated-measures ANOVA for the standard fully crossed design and follows it with planned paired comparisons of the intended definition against every orbiting definition.

aov_out <- anova_content(rating_dat, design = "within")
aov_out
#> Content-validity ANOVA (Hinkin & Tracey, 1999)
#> 
#>  item target judges                  F test      p partial eta^2 contrast p met
#>    A1      A     24  F(1.98, 45.65) = 72.64 < .001           .76     < .001 yes
#>    A2      A     24 F(1.62, 37.22) = 105.82 < .001           .82     < .001 yes
#>    A3      A     24  F(2.00, 45.97) = 82.25 < .001           .78     < .001 yes
#>    B1      B     24 F(1.85, 42.61) = 108.29 < .001           .82     < .001 yes
#> 
#> Within-judge omnibus tests are Greenhouse-Geisser corrected, so their degrees
#> of freedom are fractional.
#> contrast p: the largest p among the planned target-versus-other contrasts;
#> met: whether every one of them met the screening criterion. attr(x,
#> "contrasts") holds each contrast, and `strongest_competitor` the construct
#> rated closest to the target.
attr(aov_out, "contrasts")
#>   item design target competitor  n mean_target mean_competitor mean_diff
#> 1   A1 within      A          B 24    4.375000        2.375000  2.000000
#> 2   A1 within      A          C 24    4.375000        1.916667  2.458333
#> 3   A2 within      A          B 24    4.458333        2.000000  2.458333
#> 4   A2 within      A          C 24    4.458333        1.875000  2.583333
#> 5   A3 within      A          B 24    4.416667        2.208333  2.208333
#> 6   A3 within      A          C 24    4.416667        2.250000  2.166667
#> 7   B1 within      B          A 24    4.583333        2.083333  2.500000
#> 8   B1 within      B          C 24    4.583333        2.166667  2.416667
#>           t df            p        p_adj       dz pass
#> 1  9.591663 23 8.344091e-10 8.344091e-10 1.957890 TRUE
#> 2 11.336315 23 3.409941e-11 3.409941e-11 2.314016 TRUE
#> 3 10.552406 23 1.372752e-10 1.372752e-10 2.154001 TRUE
#> 4 17.643975 23 3.644893e-15 3.644893e-15 3.601561 TRUE
#> 5 11.072214 23 5.409806e-11 5.409806e-11 2.260106 TRUE
#> 6 11.021286 23 5.918580e-11 5.918580e-11 2.249711 TRUE
#> 7 13.133926 23 1.786074e-12 1.786074e-12 2.680951 TRUE
#> 8 14.269216 23 3.241706e-13 3.241706e-13 2.912692 TRUE

The omnibus F test asks whether the item’s mean ratings differ somewhere across definitions. The planned contrasts ask the more direct content-validity question: is the target mean higher than each orbiting mean?

With more than two construct definitions, the conventional repeated-measures F test assumes sphericity. contentvalidR reports that historical omnibus test but does not hide the assumption. The planned target-versus-orbiting comparisons are therefore important diagnostic evidence rather than decorative post-hoc tests.

Recommended workflow

fit <- rating_validity(
  rating_dat,
  scale_min = 1,
  scale_max = 5
)
fit
#> contentvalidR construct-rating analysis
#> ---------------------------------------
#> Items: 4 | Judges: 24 | Target constructs: 2 | Constructs rated: 3
#> Design: within-judge ratings on a 1 to 5 scale
#> Test: one-way repeated-measures ANOVA (Greenhouse-Geisser corrected omnibus
#> p) plus planned paired target-versus-orbiting contrasts; planned-contrast
#> adjustment: none.
#> Judges: naive, meaning drawn from the kind of people who will answer the
#> items.
#> 
#> 4 of 4 items meet the full item-level screening criterion.
#> 
#> Item-level evidence
#>  item target decision  n HTC HTD omnibus p contrast p competitor
#>    A1      A   Retain 24 .88 .56    < .001     < .001          B
#>    A2      A   Retain 24 .89 .63    < .001     < .001          B
#>    A3      A   Retain 24 .88 .55    < .001     < .001          C
#>    B1      B   Retain 24 .92 .61    < .001     < .001          C
#> 
#> n: judges who rated the item against every construct. omnibus p: do the
#> item's ratings differ across constructs (Greenhouse-Geisser corrected).
#> contrast p: the largest p among the planned target-versus-orbiting contrasts,
#> so every contrast is at or below it.
#> 
#> Target-scale Colquitt benchmarks
#>  target items mean HTC   HTC level mean HTD   HTD level
#>       A     3      .88      Strong      .58 Very Strong
#>       B     1      .92 Very Strong      .61 Very Strong
#> Benchmark set: Overall (not correlation-normed)
#> 
#> Colquitt labels are empirical percentile norms for scale-level HTC and HTD
#> averages, not universal cutoffs. HTC is an average rating and HTD is a
#> difference between ratings, so they sit on different scales with different
#> typical values. A high HTC can be labeled Weak in the same analysis where a
#> much smaller HTD is labeled Very Strong. Compare each index against its own
#> benchmark, never against the other index's number.
#> 
#> What these columns mean
#>   HTC -- Hinkin-Tracey Correspondence. Mean rating against the intended
#>       definition, as a share of the rating scale (0 to 1).
#>   HTD -- Hinkin-Tracey Distinctiveness. How far that mean exceeds the
#>       closest rival's, as a share of the scale (usually small).
#> 
#> What the decisions mean
#>   Retain -- rated highest against its intended construct, with every
#>       planned contrast meeting the screening criterion.
#> 
#> Full definitions: contentvalid_glossary(). To hide this key:
#> options(contentvalidR.show_key = FALSE).
#> 
#> 'Review' is not an automatic deletion decision. Consider construct
#> definitions, item wording, orbiting-construct choice, domain coverage, and
#> qualitative judge feedback.
summary(fit)
#> Summary: construct-rating content-validity evidence
#> ---------------------------------------------------
#> Retain: 4 of 4 | Review: 0 of 4
#> 
#> Scale-level evidence
#>  target items retain review mean HTC   HTC level mean HTD   HTD level
#>       A     3      3      0      .88      Strong      .58 Very Strong
#>       B     1      1      0      .92 Very Strong      .61 Very Strong
#>      overall
#>       Strong
#>  Very Strong
#> 
#> A: The weaker of HTC and HTD falls in the Strong band of published scales
#>   (Colquitt et al., 2019).
#> B: The weaker of HTC and HTD falls in the Very Strong band of published
#>   scales (Colquitt et al., 2019).
#> 
#> All analyzed items met the item-level inferential screening criterion.
#> 
#> Interpret these results alongside theory, domain coverage, and qualitative
#> feedback. The analysis does not by itself establish comprehensiveness or the
#> full content-validity argument.

The item-level recommendation has deliberately limited meaning:

Retain: the item cleared the package’s inferential screening rule in this pretest.
Review: the full screening rule was not met; inspect wording, construct overlap, and judge feedback.
Insufficient data: too few complete judge profiles are available for the repeated-measures comparison.

Review is not an instruction to delete an item. Content coverage can be harmed by mechanical item deletion.

Scale-level Colquitt norms

Colquitt et al. (2019) created empirical norms from scale-level averages of HTC and HTD across 112 published scales. rating_validity() therefore averages item HTC/HTD within each target scale before assigning those descriptive normative labels.

fit$scale_summary
#>   target n_items n_htc n_htd n_retain n_review n_insufficient  mean_htc
#> 1      A       3     3     3        3        0              0 0.8833333
#> 2      B       1     1     1        1        0              0 0.9166667
#>   htc_strength  mean_htd htd_strength overall_strength orbiting_r benchmark_set
#> 1       Strong 0.5781250  Very Strong           Strong         NA       overall
#> 2  Very Strong 0.6145833  Very Strong      Very Strong         NA       overall
#>                                                                                               evidence
#> 1      The weaker of HTC and HTD falls in the Strong band of published scales (Colquitt et al., 2019).
#> 2 The weaker of HTC and HTD falls in the Very Strong band of published scales (Colquitt et al., 2019).
colquitt_benchmarks("htc")
#> Benchmarks for HTC (Colquitt et al., 2019): Overall (not correlation-normed)
#> 
#>         band percentile minimum
#>  Very Strong  80th-99th     .91
#>       Strong  60th-79th     .87
#>     Moderate  40th-59th     .84
#>         Weak  20th-39th     .60
#>      Lack of   0th-19th    none
#> 
#> A scale-level mean at or above a band's minimum falls in that band. The bands
#> are percentiles of published scales, not validity cutoffs.
colquitt_benchmarks("htd")
#> Benchmarks for HTD (Colquitt et al., 2019): Overall (not correlation-normed)
#> 
#>         band percentile minimum
#>  Very Strong  80th-99th     .35
#>       Strong  60th-79th     .27
#>     Moderate  40th-59th     .18
#>         Weak  20th-39th     .04
#>      Lack of   0th-19th    none
#> 
#> A scale-level mean at or above a band's minimum falls in that band. The bands
#> are percentiles of published scales, not validity cutoffs.

The overall bands are empirical percentile standing, not universal validity cutoffs. If the average correlation between a focal scale and its orbiting scales is known, correlation-conditional norms can be requested:

rating_validity(
  rating_dat,
  orbiting_r = c(A = .42, B = .55)
)$scale_summary
#>   target n_items n_htc n_htd n_retain n_review n_insufficient  mean_htc
#> 1      A       3     3     3        3        0              0 0.8833333
#> 2      B       1     1     1        1        0              0 0.9166667
#>   htc_strength  mean_htd htd_strength overall_strength orbiting_r benchmark_set
#> 1     Moderate 0.5781250  Very Strong         Moderate       0.42      moderate
#> 2  Very Strong 0.6145833  Very Strong      Very Strong       0.55      stronger
#>                                                                                                                                                                        evidence
#> 1 The weaker of HTC and HTD falls in the Moderate band of published scales (Colquitt et al., 2019); inspect the weaker items and construct overlap before finalizing the scale.
#> 2                                                                          The weaker of HTC and HTD falls in the Very Strong band of published scales (Colquitt et al., 2019).

A given level of distinctiveness can be more impressive when the focal and orbiting constructs are known to correlate strongly.

Naive versus expert judges

Colquitt et al.’s normative distributions were developed using naive judges representative of substantive target populations. Their paper cautions against applying those norms to expert panels. The package therefore separates calculation from norm applicability:

rating_validity(rating_dat, judge_type = "expert")$scale_summary
#>   target n_items n_htc n_htd n_retain n_review n_insufficient  mean_htc
#> 1      A       3     3     3        3        0              0 0.8833333
#> 2      B       1     1     1        1        0              0 0.9166667
#>   htc_strength  mean_htd htd_strength overall_strength orbiting_r benchmark_set
#> 1         <NA> 0.5781250         <NA>             <NA>         NA       overall
#> 2         <NA> 0.6145833         <NA>             <NA>         NA       overall
#>                                                                                                        evidence
#> 1 HTC/HTD are reported descriptively; Colquitt et al. (2019) normative labels are suppressed for expert judges.
#> 2 HTC/HTD are reported descriptively; Colquitt et al. (2019) normative labels are suppressed for expert judges.

HTC/HTD are still computed, but the Colquitt labels are suppressed.

Missing ratings

For HTD and repeated-measures inference, a judge must have a usable rating for every construct definition presented for that item. Incomplete profiles are excluded itemwise and counted explicitly in the output. This preserves the paired design rather than quietly treating incomplete repeated observations as independent data.

Plotting

The original one-index views remain available:

plot(fit, metric = "htc")

HTC for each item on a 0 to 1 scale; filled points are retained items and open points are items to review.

plot(fit, metric = "htd")

HTD for each item on a -1 to 1 scale, with a dotted line at zero; filled points are retained items and open points are items to review.

A correspondence-distinctiveness evidence map displays HTC and HTD together:

plot(fit, type = "map")

Construct-rating evidence map: each item's HTC on the horizontal axis against its HTD on the vertical axis. Filled points are retained items, open points are items to review, and triangles mark each target scale's mean.

Target-scale averages are shown as diamonds and items needing review are labeled by default. As with the item-sort map, Colquitt norm regions are not drawn across individual items because those benchmarks were constructed from scale averages.

The target-versus-competitor gap plot makes the Hinkin-Tracey mean-rating logic more directly visible:

plot(fit, type = "profile")

Construct-rating profile: for each item, its mean rating against the intended definition (filled) and against the strongest competing definition (open), joined by a line that is dashed for items to review.

Filled points are intended-definition means, open points are the strongest orbiting-definition means, and the connecting segment is the observed content distinctiveness gap. A reversed segment immediately identifies an item whose strongest competitor outrates its intended definition. This is a graphical extension of the mean-rating tables used in the original procedure, not a new statistical cutoff.

Reporting

A useful report should identify:

the construct definitions and orbiting constructs shown to judges;
the judge population and recruitment method;
the response anchors and rating instructions;
item-level HTC, HTD, repeated-measures omnibus results, and planned contrasts;
the strongest orbiting competitor for items needing review;
target-scale mean HTC/HTD and the norm set used, if applicable; and
qualitative feedback and substantive decisions made after the pretest.

The quantitative analysis is evidence about definitional correspondence and distinctiveness. It does not by itself demonstrate that the item pool comprehensively samples the full construct domain.

References

Colquitt, J. A., Sabey, T. B., Rodell, J. B., & Hill, E. T. (2019). Content validation guidelines: Evaluation criteria for definitional correspondence and definitional distinctiveness. Journal of Applied Psychology, 104(10), 1243–1265. https://doi.org/10.1037/apl0000406

Hinkin, T. R., & Tracey, J. B. (1999). An analysis of variance approach to content validation. Organizational Research Methods, 2(2), 175–186. https://doi.org/10.1177/109442819922004