How we measure engagement

Two instruments people answer about themselves, one method that reads what they actually did, and the reason we need both.

A theory about how people engage with AI is only as good as the measures that test it. We use two kinds, and they answer different questions.

Self-report asks people how they use AI and what they believe about it. It is cheap, it scales to hundreds of people, and it captures beliefs that no transcript shows.

Conversation coding reads a person's actual AI conversations and labels each turn with the mode it shows. It is slower, but it records behavior rather than a person's memory of it.

The fourth finding on the next step is the reason both exist: what people report and what their conversations show barely correspond.

The instruments

1.AI Interaction Typology
How often a person uses each of the eight modes (Never to Always), 29 items across the eight mode scales, plus five 5-point agreement scales on managing the engagement: AI Calibration, AI Metacognition, Cognitive Offloading, AI Dependency and AI Self-Efficacy.
2.AI Perceptions Survey
What a person believes about AI across seven dimensions (Personal Utility, Societal Benefit, Identity Threat, Existential Risk, Epistemic Wonder, Ethical Concern, Economic Displacement Anxiety), four 7-point items each, plus three Process Valuation scales on whether AI augments or displaces the parts of work a person values.
3.Conversation coding
Each user turn in a conversation is assigned one of the eight modes by a trained classifier, checked against human coders. The engagement score is computed from those turn labels, not from the surveys.

Every item is public

All 86 survey items, their response anchors and their reliabilities are on the measurement scales page, with a CSV download for survey tools. The wording is verbatim from the validation studies; changing it means the published reliabilities no longer apply. How the score is built from coded turns is on How the Score Works.

Most scales are reliable; three mode scales are still short

Twenty of the scales have been estimated, and seventeen reach Cronbach's α of .70 or better, with the seven perception dimensions between .81 and .89. The Production Assistant, Verification Agent and Tutor scales have only three items each and sit between .55 and .62.

What this does not show

Low α on a three-item scale mostly reflects its length. Two new items for each are being written and piloted, and until then those three scales should be reported with a caveat and not used to judge individuals. Calibration, Metacognition and Dependency have no reliability estimate yet.

Instrument record canon-v1 (frozen 2026-09-22); scale diagnostics on the student perception sample.

The classifier agrees with human coders well, but that is not yet confirmed on unseen data

On the current coding protocol, agreement between the classifier and human coders reached a Krippendorff's α of .785, up from .638 in the previous version.

What this does not show

That figure may be in-sample: the protocol was refined on some of the same conversations it was scored on. A 280-conversation held-out set has been drawn but not yet coded, and the next protocol allows a turn to carry more than one mode.

Measures registry, classification agreement (v6).

Quick check

Why does the research code actual conversations instead of relying on the surveys alone?

What this semester's classroom data can and cannot show

The fall 2026 classroom survey used an earlier version of the instruments. It administered 44 of the 59 typology items, so Calibration and Metacognition have no usable score this semester, and it used a different version of the Perceptions survey, so none of the seven perception dimensions can be computed from it. New studies, including the Prolific studies, use the instruments on the scales page.