This trace has been submitted and is locked. Contact the researcher if you need to change it.
Question
— End of trace —
Task B · Human annotation
Identify the action.
Follow the reasoning.
The segments are already fixed. Read each one in order and choose the single action it explicitly performs.
Annotator handbook · Task B
Identify the action.
Follow the reasoning.
Read the full trace and label its segments in order. Your task is to identify what the text explicitly does at each step.
Do not use AI.
Do not use ChatGPT, Gemini, Copilot, or any other AI tool to interpret the trace, choose labels, or write annotation notes. Make every annotation decision yourself. Email sign-in only provides access to this website.
01The annotation unit
One segment is one explicitly expressed local reasoning operation. Keep its lead-in, operands, and stated result together when interpreting it. For this task, the LLM-judge segmentation is already supplied and boundaries are fixed. Do not rewrite the trace or change its segmentation. Flag a segment if its boundary makes one label inappropriate.
02The action set
Assign exactly one primary action label to each segment that you can resolve. Judge the operation explicitly performed, regardless of whether its reasoning is correct or successful.
1 · Construct
Build reasoning
Introduces or derives content: setup, recalled facts, inference, or calculation.
Since x + 2 = 5, x = 3.
2 · Inspect
Check reasoning
Checks, verifies, compares, or diagnoses existing reasoning or content, without replacing it yet.
Substituting x = 3 gives 3 + 2 = 5, so it checks out.
3 · Revise
Correct locally
Explicitly corrects, replaces, or withdraws local content while retaining the overall method.
I made an arithmetic error: that sum is 12, not 10.
4 · Redirect
Change direction
Changes the method, strategy, plan, case, or branch of the reasoning.
This counting approach is unwieldy; instead, use a symmetry argument.
5 · Nonaction
No substantive update
Filler, formatting, or repetition without a substantive reasoning update.
Hmm. Let me think.
03Distinguish close cases
- CONSTRUCT or INSPECT? A new derivation builds reasoning. An explicit check evaluates content already introduced. Recalculation used to check a prior result can be INSPECT.
- INSPECT or REVISE? Identifying or diagnosing an error is INSPECT. Explicitly replacing, correcting, or withdrawing content is REVISE.
- REVISE or REDIRECT? A local correction retains the method. Switching the strategy, method, plan, case, or branch is REDIRECT.
- NONACTION is a real label. Use it for filler or repetition with no substantive update. It is not a substitute for uncertainty.
- Read the whole segment. “Wait,” “let me check,” or “actually” alone does not determine the action. Consider what the segment actually performs in context. A stated intention alone does not demonstrate a completed check.
- Use context without hindsight. Earlier segments clarify the operation and its referents. Later success or failure does not change its action identity.
04Work through a trace
- Read the question and orient yourself in the full trace.
- Read consecutive segments. The active segment is highlighted in the trace and shown in the labeling panel.
- Select one action using the buttons or keys 1–5. With auto-advance enabled, selecting a label moves to the next segment.
- If uncertain, flag the segment and explain why. You may keep a tentative label, or leave it unlabeled for adjudication. For multiple operations or poor boundaries, use the segmentation issue flag reason in your note.
- Review all unresolved and flagged items. Submit only after every segment has a label or an explained flag.
1–5 label
J / → next
K / ← previous
N next unresolved
F flag
0 clear label
Independent annotation
Do not consult other annotators or an answer key. Do not assess mathematical correctness, repair the reasoning, or infer unstated actions. LLM-judge action labels are hidden to support independent decisions.
Drafts save to your account. Check the save status before leaving. Submitted work is locked until the researcher reopens it.
Practice · Not scored
Label two short passages first.
The segments are fixed. Choose one action for each, then compare with a reference. Practice answers are not part of the study; they only help you calibrate.
Your traces
Four shared traces.
Every annotator labels the same segments, independently. Work in any order and at your own pace; drafts are saved automatically. Once you submit a trace it is locked.
All traces submitted
Thank you.
Every label counts.
Your labels for all four traces have been received. You can close this page.