Reality Monitoring Task with Feedback — ps-parser branch¶
Runs the two-phase RM word-pair task with trial-level corrective feedback using the ps-parser API.
Task structure¶
| Phase | Trials | Parser | Response fields |
|---|---|---|---|
| Encoding | imagined_N, perceived_N |
Response_part_1_rm |
Word_2 (str), Rating (0–100) |
| Test | test:imagined_N, test:perceived_N |
Response_part_2_rm |
Judgment (internal/external), Confidence (1–6) |
What is new vs the old archive version¶
Old archive (parser='dynamic') |
ps-parser branch | |
|---|---|---|
| Parser routing | Hardcoded conditional edges in LangGraph | Per-trial "parser" key in task JSON |
| Card-level parser | card_in.parser = 'dynamic' |
card_in.parser = '0' (per-trial JSON handles it) |
| Feedback bool | feedback = '1' |
feedback = True |
| Feedback handler | Re-instantiated every trial | Instantiated once per simulation |
| Cross-trial word list | Module-level global (breaks with N>1) | self.all_words_in_use — per-instance |
Model: Groq — openai/gpt-oss-120b
1. Setup¶
import os, sys, ast, json, nltk
from pathlib import Path
from dotenv import load_dotenv
nltk.download('words', quiet=True)
load_dotenv()
# ── LLM ─────────────────────────────────────────────────────────────────────
# psychscanner automatically loads GROQ_API_KEY from the environment
MODEL_NAME = 'openai/gpt-oss-120b'
MODEL_FAMILY = 'groq'
# ── Paths ─────────────────────────────────────────────────────────────────
EXAMPLES_DIR = Path(os.getcwd()) # notebook lives in examples/
# Update TASKS_DIR to point to your local RM task files
TASKS_DIR = Path(os.getenv('RM_TASKS_DIR', Path.cwd() / 'tasks'))
RUN_DIR = EXAMPLES_DIR / '_rm_fb_tutorial_runs'
task_files = {
'demo_5t' : TASKS_DIR / 'rm_2op_convo_5t_feedback_allparts.json',
'demo_5t_rev' : TASKS_DIR / 'rm_2op_convo_5t_feedback_allparts_r.json',
'full' : TASKS_DIR / 'rm_2op_convo_feedback_allparts.json',
'full_rev' : TASKS_DIR / 'rm_2op_convo_feedback_allparts_r.json',
}
for name, p in task_files.items():
print(f' {name:12} -> {p.name} exists={p.exists()}')
demo_5t -> rm_2op_convo_5t_feedback_allparts.json exists=True demo_5t_rev -> rm_2op_convo_5t_feedback_allparts_r.json exists=True full -> rm_2op_convo_feedback_allparts.json exists=True full_rev -> rm_2op_convo_feedback_allparts_r.json exists=True
import psychscanner as psy
from psychscanner import ExpCard, ExpCardInit, ScannerModel
from psychscanner.parsers import Response_part_1_rm, Response_part_2_rm
if str(EXAMPLES_DIR) not in sys.path:
sys.path.insert(0, str(EXAMPLES_DIR))
from rm_msg_injection import Stim_Trial_Injection
print('psychscanner version:', psy.__version__)
print('Response_part_1_rm fields:', list(Response_part_1_rm.model_fields.keys()))
print('Response_part_2_rm fields:', list(Response_part_2_rm.model_fields.keys()))
psychscanner version: 0.1.0 Response_part_1_rm fields: ['Word_2', 'Rating'] Response_part_2_rm fields: ['Judgment', 'Confidence']
/opt/anaconda3/envs/psyscan/lib/python3.11/site-packages/langgraph/checkpoint/base/__init__.py:17: LangChainPendingDeprecationWarning: The default value of `allowed_objects` will change in a future version. Pass an explicit value (e.g., allowed_objects='messages' or allowed_objects='core') to suppress this warning. from langgraph.checkpoint.serde.jsonplus import JsonPlusSerializer
2. Per-trial parser routing in the task JSON¶
The old archive used parser='dynamic', which hardcoded two LangGraph nodes
(runnable_resp1node / runnable_resp2node) and routed between them based on
trcode. That sentinel no longer exists in ps-parser.
The task JSON files have been updated: every trial now carries a "parser" key
that names the registered parser class to use for that trial. The ps-parser
branch reads this key at invocation time and resolves the class from the
PARSER_REGISTRY — no hardcoding needed.
Encoding trial: "parser": "Response_part_1_rm" → Word_2, Rating
Test trial: "parser": "Response_part_2_rm" → Judgment, Confidence
task_json = json.loads(task_files['demo_5t'].read_text())
print('Task name :', task_json['taskname'])
print('Chain type :', task_json['chain_type'])
print('Top-level parser (fallback):', task_json['parser']) # '0' = none
print()
items = task_json['items']
print(f'{"trcode":<24} {"trial parser":<22} {"stimulus / corrAns"}')
print('-' * 80)
for trcode, trials in items.items():
t = trials[0]
stim = t.get('stimulus', {})
detail = stim.get('Word_Pair', stim.get('Test_Word', stim))
corr = t.get('corrAns', '')
extra = f'corrAns={corr}' if corr else str(detail)[:45]
print(f"{trcode:<24} {t.get('parser','(none)'):<22} {extra}")
Task name : rm_2op_5t_convo_fb
Chain type : task
Top-level parser (fallback): 0
trcode trial parser stimulus / corrAns
--------------------------------------------------------------------------------
imagined_1 Response_part_1_rm {'word_1': 'sandpaper', 'word_2': '________'}
perceived_2 Response_part_1_rm {'word_1': 'sugar', 'word_2': 'sweet'}
imagined_3 Response_part_1_rm {'word_1': 'suburb', 'word_2': '________'}
perceived_4 Response_part_1_rm {'word_1': 'chair', 'word_2': 'cushion'}
perceived_5 Response_part_1_rm {'word_1': 'letter', 'word_2': 'tip'}
imagined_6 Response_part_1_rm {'word_1': 'building', 'word_2': '________'}
imagined_7 Response_part_1_rm {'word_1': 'blouse', 'word_2': '________'}
perceived_8 Response_part_1_rm {'word_1': 'pin', 'word_2': 'haystack'}
imagined_9 Response_part_1_rm {'word_1': 'fire', 'word_2': '________'}
perceived_11 Response_part_1_rm {'word_1': 'sky', 'word_2': 'airplane'}
test:imagined_1 Response_part_2_rm corrAns=internal
test:perceived_2 Response_part_2_rm corrAns=external
test:imagined_3 Response_part_2_rm corrAns=internal
test:perceived_4 Response_part_2_rm corrAns=external
test:perceived_5 Response_part_2_rm corrAns=external
test:imagined_6 Response_part_2_rm corrAns=internal
test:imagined_7 Response_part_2_rm corrAns=internal
test:perceived_8 Response_part_2_rm corrAns=external
test:imagined_9 Response_part_2_rm corrAns=internal
test:perceived_11 Response_part_2_rm corrAns=external
3. Feedback handler — Stim_Trial_Injection¶
rm_msg_injection.py provides Stim_Trial_Injection(FeedbackBase).
Because ps-parser instantiates it once per simulation, self.all_words_in_use
accumulates words across all encoding trials for that participant.
on_response receives a plain dict already parsed by TaskRunner:
- Encoding:
{'Word_2': '...', 'Rating': 75.0} - Test:
{'Judgment': 'internal', 'Confidence': 5}
from rm_msg_injection import generate_fb_response
# ── Imagined encoding trial — correct imagined word ───────────────────────────
trial_enc = {
'trcode' : 'imagined_1',
'stimulus' : {'Word_Pair': {'word_1': 'sandpaper', 'word_2': '________'}},
}
words_seen = []
fb_enc = generate_fb_response(trial_enc, {'Word_2': 'velvet', 'Rating': 60.0}, words_seen)
print('Encoding feedback (imagined, correct):')
print(json.dumps(json.loads(fb_enc), indent=2))
print('words tracked:', words_seen)
Encoding feedback (imagined, correct):
{
"Feedback on previous response": {
"response feedback": "**CORRECT: Last trial was an imagined trial type. Your 'Word_2' correctly followed the given instructions.**",
"rating feedback": "**CORRECT: Your 'rating' value is in the instructed rating scale range.**",
"overall feedback": "**CORRECT Answer!** Well answered, good job!"
}
}
words tracked: ['sandpaper', 'velvet']
# ── Test trial — wrong judgment ───────────────────────────────────────────────
trial_test = {
'trcode' : 'test:imagined_1',
'stimulus': {'Test_Word': 'sandpaper'},
'corrAns' : 'internal',
}
fb_test = generate_fb_response(trial_test, {'Judgment': 'external', 'Confidence': 3}, [])
print('Test feedback (wrong judgment):')
print(json.dumps(json.loads(fb_test), indent=2))
Test feedback (wrong judgment):
{
"Feedback on previous response": {
"response feedback": "**INCORRECT: The correct 'Judgment' was 'internal'. It might also be incorrect due to poor formatting. Carefully follow all trial instructions about the source-judgment options.**",
"rating feedback": "**CORRECT: Your 'Confidence' value is in the instructed rating scale range (1\u20136).**",
"overall feedback": "**INCORRECT Answer!** Follow all trial instructions more accurately."
}
}
# Old archive API vs ps-parser API
print('=== Card-level differences: old archive vs ps-parser ===')
print()
print('OLD (archive):')
print(" card_in.parser = 'dynamic' # hardcoded trcode routing in LangGraph")
print(" card_in.feedback = '1' # string")
print()
print('NEW (ps-parser):')
print(" card_in.parser = '0' # no card-level parser; per-trial JSON 'parser' field used")
print(" card_in.feedback = True # bool ('1' still works for backward compat)")
=== Card-level differences: old archive vs ps-parser ===
OLD (archive):
card_in.parser = 'dynamic' # hardcoded trcode routing in LangGraph
card_in.feedback = '1' # string
NEW (ps-parser):
card_in.parser = '0' # no card-level parser; per-trial JSON 'parser' field used
card_in.feedback = True # bool ('1' still works for backward compat)
card_in = ExpCardInit()
# ── Model ─────────────────────────────────────────────────────────────────────
card_in.model = MODEL_NAME
card_in.family = MODEL_FAMILY
card_in.parameters = {'temperature': 0}
# ── Task ─────────────────────────────────────────────────────────────────────
card_in.task_file = task_files['demo_5t'] # 10 encoding + 10 test trials
card_in.parser = '0' # no card-level default — per-trial JSON 'parser' field handles routing
# encoding → Response_part_1_rm, test → Response_part_2_rm
# ── Memory ───────────────────────────────────────────────────────────────────
card_in.memory = 'Convo' # full conversation across all trials
card_in.chain_type = 'task' # single thread per participant
# ── Participant ───────────────────────────────────────────────────────────────
card_in.cogtype = 'no'
card_in.nsim = 1 # 1 simulated participant; set to 100 for a full study
# ── Feedback ─────────────────────────────────────────────────────────────────
card_in.feedback = True # bool (was '1' in the old archive)
card_in.feedback_fn = Stim_Trial_Injection # class — instantiated once per simulation
# ── Output ───────────────────────────────────────────────────────────────────
card_in.proj_dir = RUN_DIR
card_in.projectname = 'RM_FB_TUTORIAL'
card_in.tunnel_status = '0'
expcard = ExpCard(card_in)
print('card-level parser resolved to:', expcard.parser) # None — per-trial JSON overrides
----<PROJECT AND DATA ROOT DIRECTORY>---- Project root dir: /Users/saurabhext/Documents/PSYCHSCANNER/psychscanner/examples/_rm_fb_tutorial_runs Simulation data root dir: /Users/saurabhext/Documents/PSYCHSCANNER/psychscanner/examples/_rm_fb_tutorial_runs/RM_FB_TUTORIAL/rm_2op_5t_convo_fb/groq_openai/gpt-oss-120b_Convo ----<>---- card-level parser resolved to: None
5. Run¶
scanner = ScannerModel(expcard=expcard)
simulation = scanner.run()
print(f'\nDone — {len(simulation)} participant(s), {len(simulation[0])} trials each.')
--<chat model>-- profile={'max_input_tokens': 131072, 'max_output_tokens': 32768, 'image_inputs': False, 'audio_inputs': False, 'video_inputs': False, 'image_outputs': False, 'audio_outputs': False, 'video_outputs': False, 'reasoning_output': True, 'tool_calling': True} client=<groq.resources.chat.completions.Completions object at 0x152c003d0> async_client=<groq.resources.chat.completions.AsyncCompletions object at 0x151eb1310> model_name='openai/gpt-oss-120b' temperature=1e-08 model_kwargs={} groq_api_key=SecretStr('**********')
2026-05-07 02:12:51.631 | CRITICAL | psychscanner.session_tunnel.session_tunnel:create_tunnel:152 - BEGIN
TOTAL RUNS: 1 RESUME IDX: None ----<>---- task running
20it [04:02, 12.14s/it] 2026-05-07 02:16:54.436 | INFO | psychscanner.session_tunnel.session_tunnel:scan_checkpoint:184 - scan-checkpoint
----<scanned runs>---- i = 0
2026-05-07 02:16:54.441 | CRITICAL | psychscanner.session_tunnel.session_tunnel:end_checkpoint:163 - END
Done — 1 participant(s), 20 trials each.
6. Results¶
def decode(pred_resp):
content = pred_resp.content if hasattr(pred_resp, 'content') else str(pred_resp)
try:
return ast.literal_eval(content)
except Exception:
return {'_raw': content[:80]}
trials = simulation[0]
# ── Encoding phase ────────────────────────────────────────────────────────────
enc_trials = [t for t in trials if 'test' not in t['trcode']]
test_trials = [t for t in trials if 'test' in t['trcode']]
print('ENCODING PHASE')
print(f'{"trcode":<20} {"Word_2":<16} {"Rating":<8} {"feedback sent?"}')
print('-' * 65)
for t in enc_trials:
r = decode(t['pred_resp'])
print(f"{t['trcode']:<20} {str(r.get('Word_2','?')):<16} "
f"{str(r.get('Rating','?')):<8} {'yes' if t.get('fb_response') else 'no'}")
print()
print('TEST PHASE')
print(f'{"trcode":<24} {"Judgment":<12} {"corrAns":<12} {"Conf":<6} {"correct?"}')
print('-' * 65)
correct = 0
for t in test_trials:
r = decode(t['pred_resp'])
j = str(r.get('Judgment', '?')).lower()
c = str(t.get('corrAns', '')).lower()
ok = j == c
if ok:
correct += 1
print(f"{t['trcode']:<24} {j:<12} {c:<12} {str(r.get('Confidence','?')):<6} {'✓' if ok else '✗'}")
print()
print(f'Source-monitoring accuracy: {correct}/{len(test_trials)} = {correct/len(test_trials):.0%}')
ENCODING PHASE trcode Word_2 Rating feedback sent? ----------------------------------------------------------------- imagined_1 abrasive 85.0 yes perceived_2 sweet 92.0 yes imagined_3 neighborhood 90.0 yes perceived_4 cushion 88.0 yes perceived_5 tip 30.0 yes imagined_6 construction 85.0 yes imagined_7 shirt 80.0 yes perceived_8 haystack 25.0 yes imagined_9 smoke 92.0 yes perceived_11 airplane 85.0 yes TEST PHASE trcode Judgment corrAns Conf correct? ----------------------------------------------------------------- test:imagined_1 internal internal 6 ✓ test:perceived_2 external external 6 ✓ test:imagined_3 internal internal 6 ✓ test:perceived_4 external external 6 ✓ test:perceived_5 external external 6 ✓ test:imagined_6 internal internal 6 ✓ test:imagined_7 internal internal 6 ✓ test:perceived_8 external external 6 ✓ test:imagined_9 internal internal 6 ✓ test:perceived_11 external external 6 ✓ Source-monitoring accuracy: 10/10 = 100%
# Show the full feedback JSON injected after the first encoding trial
t = enc_trials[0]
print(f"Trial: {t['trcode']}")
print(f"Response: {decode(t['pred_resp'])}")
print()
if t['fb_response']:
print('Feedback injected before next trial:')
try:
print(json.dumps(json.loads(t['fb_response']), indent=2))
except Exception:
print(t['fb_response'])
else:
print('No feedback for this trial.')
Trial: imagined_1
Response: {'Word_2': 'abrasive', 'Rating': 85.0}
Feedback injected before next trial:
{
"Feedback on previous response": {
"response feedback": "**CORRECT: Last trial was an imagined trial type. Your 'Word_2' correctly followed the given instructions.**",
"rating feedback": "**CORRECT: Your 'rating' value is in the instructed rating scale range.**",
"overall feedback": "**CORRECT Answer!** Well answered, good job!"
}
}
7. Using the callable form instead of per-trial JSON (alternative)¶
If you prefer to keep the routing logic in Python rather than in the task JSON,
you can use a Form B callable at the card level and leave the task JSON
trial "parser" field absent.
from psychscanner.parsers import Response_part_1_rm, Response_part_2_rm
# Form B: callable at card level — routes by trcode
rm_parser_dispatch = lambda trcode: (
Response_part_2_rm if 'test' in trcode else Response_part_1_rm
)
# Verify routing
print('encode trial ->', rm_parser_dispatch('imagined_1'))
print('test trial ->', rm_parser_dispatch('test:imagined_1'))
print()
print('To use: set card_in.parser = rm_parser_dispatch')
print('Per-trial JSON "parser" (Form A) still takes priority over this callable.')
encode trial -> <class 'psychscanner.datasets.prompts.parser_tasks.Response_part_1_rm'> test trial -> <class 'psychscanner.datasets.prompts.parser_tasks.Response_part_2_rm'> To use: set card_in.parser = rm_parser_dispatch Per-trial JSON "parser" (Form A) still takes priority over this callable.
8. Scaling to many participants¶
Change nsim and optionally switch to the full 20-pair task or the reversed
counterbalance list. Each participant gets its own Stim_Trial_Injection
instance so word lists never leak between them.
# Uncomment to run a full multi-participant study
#
# card_in.nsim = 100
# card_in.task_file = task_files['full'] # 20 encoding + 20 test trials
# # or reversed counterbalance list:
# # card_in.task_file = task_files['full_rev']
# card_in.tunnel_status = '1' # enable checkpointing for long runs
#
# expcard_full = ExpCard(card_in)
# scanner_full = ScannerModel(expcard=expcard_full)
# simulation_full = scanner_full.run()
print('Scaling config ready — uncomment above to run.')
Scaling config ready — uncomment above to run.