Reached response accuracy
Score what students answered correctly among valid reached responses. Cleaner as an accuracy or ability claim, but can understate timed productivity concerns.
2026 BOY Scoring Options
A concise comparison of candidate methods for reporting 2026 Beginning-of-Year early numeracy screener performance, with provisional judgements organised around subtest representation, precision and uncertainty, and classification stability.
Purpose and context
This memo compares scoring methods for two decisions: the single global BOY score and the student-level subtest scores. The immediate reporting need is one global score for dashboard labels and bands, plus a defensible way to describe subtest profiles.
The longer-term direction is a two-axis accuracy × pace report, but that architecture is not the immediate BOY single-score decision.
Decision 1: Global score options
| Method | Description | Subtest representation | Precision / uncertainty | Classification stability |
|---|---|---|---|---|
| All-items timed-performance IRT | Unidimensional + testlet model. One global score from all retained items. Timed subtests use timed-performance scoring. | ◐Uses all included items, but longer subtests naturally carry more influence. | ●Student-level uncertainty is available; strongest fit for a global score precision story. | ●Current benchmark for movement checks. Stability must be read against challengers, not assumed as validity. |
| Equal-subtest timed-performance IRT | One global IRT score, with item contributions weighted so each subtest has equal design influence. | ●Balances all five subtests by design. | ●Student-level uncertainty is available; check whether equal weighting changes precision near cut points. | ◐Compare against all-items timed-performance IRT using percentile movement and high-concern cut-point crossings. |
| Hierarchical global-plus-subtest timed-performance model | Joint model estimating a general score plus subtest-specific deviations. | ●Represents all subtests and explicitly models subtest profiles. | ●Strong posterior uncertainty story, especially for subtest profiles and shrinkage. | △Compared with all-items timed-performance IRT, the global score can move students materially. Global use needs explicit justification. |
| Equal-subtest raw-score composite | Each subtest is scored directly from responses, then standardised and averaged equally. | ●Transparent equal-subtest representation. | △Limited uncertainty handling; short subtests can have coarse score resolution and many ties. | ◐Useful as a sanity check, not a strong standalone psychometric endpoint. |
| Selected-subtest reliability-weighted marker | Separate subtest calibrations. A selected subset feeds the global ability marker, weighted by reliability. Pace/fluency is separate. | ◐Only selected subtests contribute to global ability. That selection is a major construct decision. | ◐Reliability weights favour more precise selected subtests; uncertainty from excluded subtests is not represented. | ○The deck reviewed appears to use a 2025 cohort base. Run a 2026 comparison before treating it as a BOY scoring option. |
Response-treatment sensitivity
Reached-only accuracy versus timed-performance scoring. This shows how much the global score depends on treating trailing unreached timed items as zero credit.
Subtest-weighting sensitivity
Equal-subtest timed-performance IRT versus all-items timed-performance IRT. This shows whether unequal subtest lengths materially affect student ordering.
Decision 2: Subtest score options
| Method | Description | Subtest representation | Precision / uncertainty | Classification stability |
|---|---|---|---|---|
| Standalone subtest IRT | Each subtest is modelled separately. Timed subtests use timed-performance inputs where applicable. | ●Clean subtest-specific score for each probe. | ◐Subtest SEs are available, but the method does not borrow information across subtests. | ○Compare directly against hierarchical subtest scores by same student ID. |
| Hierarchical modelled subtest scores | Subtest scores are estimated inside the joint model as general level plus subtest-specific deviation. | ●Represents all subtests while modelling each student’s profile. | ●Posterior uncertainty and shrinkage make this strongest for uncertainty-qualified subtest profiles. | ◐May shift subtest interpretation relative to standalone IRT. Needs same-student agreement review. |
| Raw/proportion subtest scores | Direct score from responses, such as correct over denominator or reached accuracy. | ●Highly transparent for teachers and audit. | △Transparent, but uncertainty is mostly denominator size, discreteness and tied-score behaviour. | ◐Useful companion method, but should not be the only evidence for subtest bands. |
Subtest-score method agreement
Hierarchical modelled subtest percentiles versus standalone subtest percentiles. Each dot is the same student ID. The black line is equal percentile rank; amber dashed lines mark the 15th and 35th percentiles.
Accuracy vs timed performance
Correct work over the timed item set
Trailing unreached items receive zero credit. This intentionally blends accuracy, reach, pace and productivity.
Accuracy × pace
Report accuracy and pace separately. Conceptually cleaner, but not the immediate BOY single-score decision.
Precision / uncertainty evidence
The scorecard uses precision to mean student-score uncertainty and band stability, not just reliability. The headline question is whether students can be placed confidently enough for dashboard bands and subtest profiles.
Global score precision
| Method | Year level | Posterior SD, median | 90% interval width, median | Approx. reliability | Read |
|---|---|---|---|---|---|
| All-items timed-performance IRT | Foundation | 0.491 | 1.614 | 0.665 | Usable global precision, but intervals are not narrow. |
| Equal-subtest timed-performance IRT | Foundation | 0.507 | 1.667 | 0.666 | Similar precision to all-items; weighting does not obviously improve precision. |
| H1 global score | Foundation | 0.334 | 1.097 | 0.909 | More precise on its own scale, but it is a different global construct. |
| All-items timed-performance IRT | Year 1 | 0.372 | 1.222 | 0.811 | Stronger global precision than Foundation. |
| Equal-subtest timed-performance IRT | Year 1 | 0.402 | 1.322 | 0.796 | Slightly wider uncertainty than all-items. |
| H1 global score | Year 1 | 0.309 | 1.017 | 0.955 | Most precise global posterior, but global interpretation differs from the screener-index score. |
Band uncertainty from hierarchical posterior intervals
| Score type | Year level | Median 90% interval width, percentile points | One-band interval | Three-band interval | Read |
|---|---|---|---|---|---|
| Global | Foundation | 30.8 | 49.6% | 12.7% | Many global intervals cross at least one dashboard band cut. |
| Global | Year 1 | 21.5 | 57.2% | 5.5% | Year 1 global bands are more stable. |
| Subtest | Foundation | 45.6 | 35.3% | 33.0% | Foundation subtest profiles need cautious, uncertainty-aware interpretation. |
| Subtest | Year 1 | 34.4 | 43.3% | 23.0% | Year 1 subtest profiles are more stable, but still not point-precise. |
Subtest precision signals
| Evidence | Foundation | Year 1 | Read |
|---|---|---|---|
| Standalone subtest reliability range | 0.602 to 0.927 across achievement subtests | 0.727 to 0.940 across achievement subtests | Some subtests are much stronger measurement instruments than others. |
| H1 subtest posterior SD, median | 0.454 to 0.728 across subtests | see subtest summary appendix | Shorter or noisier subtests have wider subtest uncertainty. |
| Raw/proportion score resolution | Coarse on short subtests, visible as tied percentile bands | Coarse where denominators are short | Raw scores are transparent, but tied scores limit fine-grained profile interpretation. |
Evidence gaps and checks
- No 2026 external outcome validation is available yet.
- PAT and teacher-rating validation should be added when usable 2026 outcome files are available.
- School, class, device and session checks should be added when 2026 administration metadata are available.
- The selected-subtest reliability-weighted marker reviewed here appears based on a 2025 cohort. Run a 2026 comparison before treating it as a BOY scoring option.
- All method-comparison plots should retain same-student joins in the production export.
Appendix
Model specifications
Notation: student i, item j, subtest s(j), testlet t(j), scored response y. Equations below are schematic, intended to document the scoring architecture rather than every implementation detail.
All-items timed-performance IRT
yij ~ Bernoulli(pij)logit(pij) = θi − bj + ui,t(j)ui,t ~ Normal(0, σtestlet)
Each retained item contributes once. For timed subtests, y uses timed-performance scoring.
Equal-subtest timed-performance IRT
weighted log likelihood = Σi Σj ws(j) · log P(yij | θi, bj, ui,t(j))ws ∝ 1 / number of retained items in subtest s
The item model is similar, but likelihood contributions are weighted so subtests have equal design influence.
Hierarchical global-plus-subtest model
θi,s = θi,global + δi,sδi,s ~ Normal(0, σs)yij ~ item model using θi,s(j)
The model estimates a general score and subtest-specific deviations. Subtest scores are posterior summaries of θ_i,s.
Equal-subtest raw-score composite
rawi,s = scored crediti,s / denominatori,szi,s = standardise(rawi,s within year × subtest)compositei = means(zi,s), over reportable subtests
This is direct observed scoring, not an IRT model.
Selected-subtest reliability-weighted marker
zi,s = standardise(separately calibrated subtest θi,s)markeri = Σs in selected set rs · zi,s / Σs in selected set rs
Only selected subtests enter the global ability marker. Reliability weights r_s affect influence, but do not by themselves define construct importance.
Timed-performance rule
Valid reached responses are scored 0/1. Trailing unreached items receive zero credit. Intermittent, contract, QC or other non-comparable missing rows remain missing.
timed-performance yij = 1, correct valid reached responsetimed-performance yij = 0, incorrect valid reached responsetimed-performance yij = 0, trailing unreached timed itemtimed-performance yij = missing, otherwise
Reliability and construct validity
Reliability can inform precision, uncertainty, shrinkage and reportability. It should not automatically determine construct importance. A reliable subtest can be narrow or misaligned; a noisier subtest can still represent an important domain.
Plot checks
The global sensitivity plots use release-candidate comparison exports. The subtest agreement plots join the GUID-keyed standalone subtest score export to hierarchical subtest posterior means by year level, subtest and the same student GUID. Both scores are converted to within-year × subtest percentile ranks for the deployed plots.