Reality Monitoring Task — Parsers across Memory & Feedback configurations¶

Walks through how the same Reality-Monitoring (RM) word-pair paradigm is run under four different psychscanner configurations. Task JSONs are trimmed from the rmllm dissertation repo (data/raw/runtasks2/) and live in examples/tasks/.

#	Variant	`chain_type`	`memory`	`feedback`	Parser (card)	Response shape
1	Single-turn (4-field)	`item`	`SingleTurn`	off	`'1'` → `ResponseRmStEI`	Word_2, Rating, Judgment, Confidence
2	Trial chain (4-step)	`trial`	`Convo`	off	`'1'` → `AllResponseRMEI`	Union-dispatched per stimulus step
3	Episodic convo, no fb	`task`	`Convo`	off	`'dynamic'` → `Response_part_1_rm`	Word_2, Rating per trial
4	Episodic convo + fb	`task`	`Convo`	on	`'dynamic'` → `Response_part_1_rm`	Word_2, Rating + feedback injection

Variants 3 & 4 replicate the exact experiment configuration from rmllm (run_rm_task_convo_no_fb.py and run_rm_task_convo_fb.py). The feedback class (Stim_Trial_Injection) is adapted directly from rmllm/rmllm/rm_msg_injection.py.

Model: smollm2:360m-instruct-fp16 via Ollama — the local small model used throughout the psychscanner tutorials. Structured-output parsers (Pydantic schemas) enforce the response shape, so even a 360 M model produces correctly-typed output across all four variants. The real rmllm experiments used gemma3:12b; reference output from that model is shown in Section 9. To reproduce the full-quality experiment, change MODEL_NAME in the setup cell below to 'gemma3:12b'.

1. Setup¶

Requires Ollama running locally with smollm2:360m-instruct-fp16 pulled (ollama pull smollm2:360m-instruct-fp16). To use the model from the real rmllm experiments instead, set MODEL_NAME = 'gemma3:12b'.

In [1]:

Copied!





import ast
import json
import shutil
from pathlib import Path

from langchain_core.messages import HumanMessage
from psychscanner import ExpCard, ExpCardInit, ScannerModel
from psychscanner.parsers import (
    list_parsers,
    get_parser,
    ResponseRmStEI,
    AllResponseRMEI,
)
from psychscanner.datasets.prompts.parser_tasks import (
    Response_part_1_rm,
    Response_part_2_rm,
)

TASKS_DIR = Path.cwd() / 'tasks'
RUN_DIR   = Path.cwd() / '_rm_tutorial_runs'
if RUN_DIR.exists():
    shutil.rmtree(RUN_DIR)

# ── Model selection ────────────────────────────────────────────────────────────
# smollm2:360m-instruct-fp16  → fast local model, works for all 4 variants
#                               because structured-output parsers enforce the schema.
# gemma3:12b                  → same model used in published rmllm experiments;
#                               produces richer, more realistic responses.
MODEL_NAME   = 'smollm2:360m-instruct-fp16'
MODEL_FAMILY = 'ollama'
# ──────────────────────────────────────────────────────────────────────────────

task_files = {
    'singleturn'  : TASKS_DIR / 'rm_singleturn_demo.json',
    'trialchain'  : TASKS_DIR / 'rm_trialchain_demo.json',
    'episodic'    : TASKS_DIR / 'rm_episodic_demo.json',
    'episodic_fb' : TASKS_DIR / 'rm_episodic_fb_demo.json',
}
for name, p in task_files.items():
    print(f'  {name:13} -> {p.name}  (exists={p.exists()})')
import ast
import json
import shutil
from pathlib import Path

from langchain_core.messages import HumanMessage
from psychscanner import ExpCard, ExpCardInit, ScannerModel
from psychscanner.parsers import (
    list_parsers,
    get_parser,
    ResponseRmStEI,
    AllResponseRMEI,
)
from psychscanner.datasets.prompts.parser_tasks import (
    Response_part_1_rm,
    Response_part_2_rm,
)

TASKS_DIR = Path.cwd() / 'tasks'
RUN_DIR   = Path.cwd() / '_rm_tutorial_runs'
if RUN_DIR.exists():
    shutil.rmtree(RUN_DIR)

# ── Model selection ────────────────────────────────────────────────────────────
# smollm2:360m-instruct-fp16  → fast local model, works for all 4 variants
#                               because structured-output parsers enforce the schema.
# gemma3:12b                  → same model used in published rmllm experiments;
#                               produces richer, more realistic responses.
MODEL_NAME   = 'smollm2:360m-instruct-fp16'
MODEL_FAMILY = 'ollama'
# ──────────────────────────────────────────────────────────────────────────────

task_files = {
    'singleturn'  : TASKS_DIR / 'rm_singleturn_demo.json',
    'trialchain'  : TASKS_DIR / 'rm_trialchain_demo.json',
    'episodic'    : TASKS_DIR / 'rm_episodic_demo.json',
    'episodic_fb' : TASKS_DIR / 'rm_episodic_fb_demo.json',
}
for name, p in task_files.items():
    print(f'  {name:13} -> {p.name}  (exists={p.exists()})')

  singleturn    -> rm_singleturn_demo.json  (exists=True)
  trialchain    -> rm_trialchain_demo.json  (exists=True)
  episodic      -> rm_episodic_demo.json  (exists=True)
  episodic_fb   -> rm_episodic_fb_demo.json  (exists=True)

2. Inspect the four task JSONs¶

In [2]:

Copied!





print(f'{"variant":13} {"taskname":24} {"tasktype":18} {"chain_type":11} {"parser":22} items')
print('-' * 105)
for name, p in task_files.items():
    d = json.loads(p.read_text())
    print(f'{name:13} {d["taskname"]:24} {d["tasktype"]:18} {d["chain_type"]:11} {d["parser"]:22} {len(d["items"]):>5}')
print(f'{"variant":13} {"taskname":24} {"tasktype":18} {"chain_type":11} {"parser":22} items')
print('-' * 105)
for name, p in task_files.items():
    d = json.loads(p.read_text())
    print(f'{name:13} {d["taskname"]:24} {d["tasktype"]:18} {d["chain_type"]:11} {d["parser"]:22} {len(d["items"]):>5}')

variant       taskname                 tasktype           chain_type  parser                 items
---------------------------------------------------------------------------------------------------------
singleturn    zs_rm_2op                sc                 item        ResponseRmStEI             4
trialchain    zs_rm_2op_tc             sc                 trial       AllResponseRMEI            2
episodic      rm_2op_convo_nofb        episodic_system    task        dynamic                    4
episodic_fb   rm_2op_convo_fb          episodic_system    task        dynamic                    4

3. RM parsers used in these variants¶

ResponseRmStEI (V1): All four judgments in one structured call.
AllResponseRMEI (V2): Union that routes each step of a trial-chain to the right sub-schema.
Response_part_1_rm (V3/V4): What the 'dynamic' router selects for encoding trials (imagined / perceived). Two fields: Word_2 + Rating.
Response_part_2_rm (V3/V4 test trials): Judgment + Confidence — returned for test trials in the full task.

In [3]:

Copied!





for cls in (ResponseRmStEI, AllResponseRMEI, Response_part_1_rm, Response_part_2_rm):
    print(f'{cls.__name__}  (module: {cls.__module__.split(".")[-1]})')
    for fname, fld in cls.model_fields.items():
        ann = str(fld.annotation).replace('typing.', '')
        print(f'  - {fname}: {ann}')
    print()
for cls in (ResponseRmStEI, AllResponseRMEI, Response_part_1_rm, Response_part_2_rm):
    print(f'{cls.__name__}  (module: {cls.__module__.split(".")[-1]})')
    for fname, fld in cls.model_fields.items():
        ann = str(fld.annotation).replace('typing.', '')
        print(f'  - {fname}: {ann}')
    print()

ResponseRmStEI  (module: parser_tasks)
  - Word_2: <class 'str'>
  - Rating: <class 'float'>
  - Judgment: Literal['external', 'internal']
  - Confidence: Literal[1, 2, 3, 4, 5, 6]

AllResponseRMEI  (module: parser_tasks)
  - response: Union[psychscanner.datasets.prompts.parser_tasks.Word2, psychscanner.datasets.prompts.parser_tasks.RelatednessRating, psychscanner.datasets.prompts.parser_tasks.JudgmentEI, psychscanner.datasets.prompts.parser_tasks.Confidence16]

Response_part_1_rm  (module: parser_tasks)
  - Word_2: <class 'str'>
  - Rating: <class 'float'>

Response_part_2_rm  (module: parser_tasks)
  - Judgment: Literal['internal', 'external']
  - Confidence: Literal[1, 2, 3, 4, 5, 6]

4. Helper functions¶

In [4]:

Copied!





def make_card(variant, *, memory, chain_type, parser='1', feedback=False, feedback_fn=None):
    """Build an ExpCardInit for the given tutorial variant.

    Parameters
    ----------
    parser : str
        '1'       → resolve class name from task JSON via the parsers registry.
        'dynamic' → use built-in RM routing (Response_part_1_rm for encoding
                    trials, Response_part_2_rm for test trials).
    """
    card = ExpCardInit()
    card.proj_dir       = RUN_DIR / variant
    card.projectname    = variant
    card.model          = MODEL_NAME
    card.family         = MODEL_FAMILY
    card.parameters     = {'temperature': 0}
    card.task_file      = task_files[variant]
    card.parser         = parser
    card.cogtype        = 'no'
    card.nsim           = 1
    card.tunnel_status  = '0'
    card.memory         = memory
    card.chain_type     = chain_type
    card.feedback       = feedback
    card.feedback_fn    = feedback_fn
    return card


def parse_pred_resp(pred_resp):
    """Decode the stringified-dict AIMessage content back to a Python object."""
    content = pred_resp.content if hasattr(pred_resp, 'content') else str(pred_resp)
    try:
        return ast.literal_eval(content)
    except Exception:
        return {'_raw': content[:120]}


def show_trials(trials, label=''):
    if label:
        print(f'--- {label} ---')
    print(f'  {"trcode":15}  parsed response')
    print(f'  {"-"*15}  {"-"*65}')
    for t in trials:
        parsed = parse_pred_resp(t['pred_resp'])
        print(f'  {t["trcode"]:15}  {parsed}')
def make_card(variant, *, memory, chain_type, parser='1', feedback=False, feedback_fn=None):
    """Build an ExpCardInit for the given tutorial variant.

    Parameters
    ----------
    parser : str
        '1'       → resolve class name from task JSON via the parsers registry.
        'dynamic' → use built-in RM routing (Response_part_1_rm for encoding
                    trials, Response_part_2_rm for test trials).
    """
    card = ExpCardInit()
    card.proj_dir       = RUN_DIR / variant
    card.projectname    = variant
    card.model          = MODEL_NAME
    card.family         = MODEL_FAMILY
    card.parameters     = {'temperature': 0}
    card.task_file      = task_files[variant]
    card.parser         = parser
    card.cogtype        = 'no'
    card.nsim           = 1
    card.tunnel_status  = '0'
    card.memory         = memory
    card.chain_type     = chain_type
    card.feedback       = feedback
    card.feedback_fn    = feedback_fn
    return card


def parse_pred_resp(pred_resp):
    """Decode the stringified-dict AIMessage content back to a Python object."""
    content = pred_resp.content if hasattr(pred_resp, 'content') else str(pred_resp)
    try:
        return ast.literal_eval(content)
    except Exception:
        return {'_raw': content[:120]}


def show_trials(trials, label=''):
    if label:
        print(f'--- {label} ---')
    print(f'  {"trcode":15}  parsed response')
    print(f'  {"-"*15}  {"-"*65}')
    for t in trials:
        parsed = parse_pred_resp(t['pred_resp'])
        print(f'  {t["trcode"]:15}  {parsed}')

5. Variant 1 — Single-turn (`chain_type=item`, `memory=SingleTurn`)¶

Each word pair is an independent call. The parser ResponseRmStEI captures all four judgments in a single structured response: Word_2 (imagined or perceived word) · Rating (0–100 relatedness) · Judgment (external/internal) · Confidence (1–6).

Parser is resolved via the registry: card.parser='1' reads the "parser" field from the task JSON ("ResponseRmStEI") and looks it up in PARSER_REGISTRY.

In [5]:

Copied!





card_v1  = make_card('singleturn', memory='SingleTurn', chain_type='item', parser='1')
exp_v1   = ExpCard(card_v1)
print(f'Resolved parser: {exp_v1.parser.__name__}  (module: {exp_v1.parser.__module__.split(".")[-1]})')
scanner_v1 = ScannerModel(expcard=exp_v1)
trials_v1  = scanner_v1.run()[0]
print()
show_trials(trials_v1, label='V1 — singleturn / item / no feedback')
card_v1  = make_card('singleturn', memory='SingleTurn', chain_type='item', parser='1')
exp_v1   = ExpCard(card_v1)
print(f'Resolved parser: {exp_v1.parser.__name__}  (module: {exp_v1.parser.__module__.split(".")[-1]})')
scanner_v1 = ScannerModel(expcard=exp_v1)
trials_v1  = scanner_v1.run()[0]
print()
show_trials(trials_v1, label='V1 — singleturn / item / no feedback')

----<PROJECT AND DATA ROOT DIRECTORY>----

	Project root dir: /Users/saurabhext/Documents/PSYCHSCANNER/psychscanner/examples/_rm_tutorial_runs/singleturn

	Simulation data root dir: /Users/saurabhext/Documents/PSYCHSCANNER/psychscanner/examples/_rm_tutorial_runs/singleturn/singleturn/zs_rm_2op/ollama_smollm2:360m-instruct-fp16_SingleTurn

----<>----

Resolved parser: ResponseRmStEI  (module: parser_tasks)
{'temperature': 0}
--<chat model>-- model='smollm2:360m-instruct-fp16' temperature=0.0

2026-05-04 16:04:21.479 | CRITICAL | psychscanner.session_tunnel.session_tunnel:create_tunnel:152 - BEGIN

TOTAL RUNS: 1	RESUME IDX: None

----<>---- task running

0it [00:00, ?it/s]

1it [00:21, 21.94s/it]

2it [00:28, 13.12s/it]

3it [00:34,  9.69s/it]

4it [00:38,  7.63s/it]

4it [00:38,  9.74s/it]

2026-05-04 16:05:00.478 | INFO     | psychscanner.session_tunnel.session_tunnel:scan_checkpoint:184 - scan-checkpoint

----<scanned runs>---- i = 0

2026-05-04 16:05:00.482 | CRITICAL | psychscanner.session_tunnel.session_tunnel:end_checkpoint:163 - END

--- V1 — singleturn / item / no feedback ---
  trcode           parsed response
  ---------------  -----------------------------------------------------------------
  imagined_1       {'Word_2': 'sandpaper', 'Rating': 3.0, 'Judgment': 'external', 'Confidence': 4}
  perceived_2      {'Word_2': 'sweet', 'Rating': 3.0, 'Judgment': 'external', 'Confidence': 4}
  imagined_3       {'Word_2': 'suburb', 'Rating': 3.0, 'Judgment': 'external', 'Confidence': 4}
  perceived_4      {'Word_2': 'cushion', 'Rating': 3.0, 'Judgment': 'external', 'Confidence': 4}

6. Variant 2 — Trial chain (`chain_type=trial`, `memory=Convo`)¶

Each item key (e.g. imagined_1) holds 4 sequential stimuli that walk the model through:

identify word_2 → 2. rate relatedness → 3. judge source → 4. report confidence.

Parser AllResponseRMEI = Union[Word2, RelatednessRating, JudgmentEI, Confidence16]: LangChain picks the right sub-schema for each step automatically. With chain_type='trial' + memory='Convo', prior steps within the same trial are visible; trials don't share history with each other.

In [6]:

Copied!





card_v2  = make_card('trialchain', memory='Convo', chain_type='trial', parser='1')
exp_v2   = ExpCard(card_v2)
print(f'Resolved parser: {exp_v2.parser.__name__}  (module: {exp_v2.parser.__module__.split(".")[-1]})')
scanner_v2 = ScannerModel(expcard=exp_v2)
trials_v2  = scanner_v2.run()[0]
print()
show_trials(trials_v2, label='V2 — trialchain / trial / no feedback')
card_v2  = make_card('trialchain', memory='Convo', chain_type='trial', parser='1')
exp_v2   = ExpCard(card_v2)
print(f'Resolved parser: {exp_v2.parser.__name__}  (module: {exp_v2.parser.__module__.split(".")[-1]})')
scanner_v2 = ScannerModel(expcard=exp_v2)
trials_v2  = scanner_v2.run()[0]
print()
show_trials(trials_v2, label='V2 — trialchain / trial / no feedback')

----<PROJECT AND DATA ROOT DIRECTORY>----

	Project root dir: /Users/saurabhext/Documents/PSYCHSCANNER/psychscanner/examples/_rm_tutorial_runs/trialchain

	Simulation data root dir: /Users/saurabhext/Documents/PSYCHSCANNER/psychscanner/examples/_rm_tutorial_runs/trialchain/trialchain/zs_rm_2op_tc/ollama_smollm2:360m-instruct-fp16_Convo

----<>----

Resolved parser: AllResponseRMEI  (module: parser_tasks)
{'temperature': 0}
--<chat model>-- model='smollm2:360m-instruct-fp16' temperature=0.0

2026-05-04 16:05:00.637 | CRITICAL | psychscanner.session_tunnel.session_tunnel:create_tunnel:152 - BEGIN

TOTAL RUNS: 1	RESUME IDX: None

----<>---- task running

0it [00:00, ?it/s]

1it [00:05,  5.49s/it]

2it [00:12,  6.16s/it]

3it [00:17,  5.75s/it]

4it [00:27,  7.33s/it]

5it [00:29,  5.46s/it]

6it [00:36,  6.16s/it]

7it [00:41,  5.64s/it]

8it [00:54,  7.90s/it]

8it [00:54,  6.76s/it]

2026-05-04 16:05:54.833 | INFO     | psychscanner.session_tunnel.session_tunnel:scan_checkpoint:184 - scan-checkpoint

----<scanned runs>---- i = 0

2026-05-04 16:05:54.838 | CRITICAL | psychscanner.session_tunnel.session_tunnel:end_checkpoint:163 - END

--- V2 — trialchain / trial / no feedback ---
  trcode           parsed response
  ---------------  -----------------------------------------------------------------
  imagined_1       {'response': {'Word_2': 'sandpaper'}}
  imagined_1       {'response': {'Relatedness_Rating': 100.0}}
  imagined_1       {'response': {'Judgment': 'external'}}
  imagined_1       {'response': {'Confidence': 5}}
  perceived_2      {'response': {'Word_2': 'sweet'}}
  perceived_2      {'response': {'Relatedness_Rating': 100.0}}
  perceived_2      {'response': {'Judgment': 'external'}}
  perceived_2      {'response': {'Confidence': 5}}

7. Variant 3 — Episodic conversation, no feedback (`chain_type=task`, `memory=Convo`)¶

This replicates the run_rm_task_convo_no_fb.py configuration from the rmllm repo.

Key settings:

tasktype='episodic_system' → each trial overrides the system message with its own per-trial instructions.
chain_type='task' + memory='Convo' → all trials share a single LangGraph thread; the model sees its own previous responses as conversation history.
parser='dynamic' → built-in routing: Response_part_1_rm for encoding trials (imagined_* / perceived_*), Response_part_2_rm for recognition/test trials.

No feedback is injected, so the model cannot see whether its prior judgments were right.

In [7]:

Copied!





# parser='1' is set directly — not routed through the registry
card_v3  = make_card('episodic', memory='Convo', chain_type='task', parser='1')
exp_v3   = ExpCard(card_v3)
print(f'Parser mode: {exp_v3.parser}')
scanner_v3 = ScannerModel(expcard=exp_v3)
trials_v3  = scanner_v3.run()[0]
print()
show_trials(trials_v3, label='V3 — episodic / task / no feedback')
# parser='1' is set directly — not routed through the registry
card_v3  = make_card('episodic', memory='Convo', chain_type='task', parser='1')
exp_v3   = ExpCard(card_v3)
print(f'Parser mode: {exp_v3.parser}')
scanner_v3 = ScannerModel(expcard=exp_v3)
trials_v3  = scanner_v3.run()[0]
print()
show_trials(trials_v3, label='V3 — episodic / task / no feedback')

----<PROJECT AND DATA ROOT DIRECTORY>----

	Project root dir: /Users/saurabhext/Documents/PSYCHSCANNER/psychscanner/examples/_rm_tutorial_runs/episodic

	Simulation data root dir: /Users/saurabhext/Documents/PSYCHSCANNER/psychscanner/examples/_rm_tutorial_runs/episodic/episodic/rm_2op_convo_nofb/ollama_smollm2:360m-instruct-fp16_Convo

----<>----

Parser mode: dynamic
{'temperature': 0}

--<chat model>-- model='smollm2:360m-instruct-fp16' temperature=0.0

2026-05-04 16:05:55.052 | CRITICAL | psychscanner.session_tunnel.session_tunnel:create_tunnel:152 - BEGIN

TOTAL RUNS: 1	RESUME IDX: None

----<>---- task running

0it [00:00, ?it/s]

1it [00:08,  8.86s/it]

2it [00:11,  5.41s/it]

3it [00:14,  4.03s/it]

4it [00:16,  3.34s/it]

4it [00:16,  4.13s/it]

2026-05-04 16:06:11.611 | INFO     | psychscanner.session_tunnel.session_tunnel:scan_checkpoint:184 - scan-checkpoint

----<scanned runs>---- i = 0

2026-05-04 16:06:11.615 | CRITICAL | psychscanner.session_tunnel.session_tunnel:end_checkpoint:163 - END

--- V3 — episodic / task / no feedback ---
  trcode           parsed response
  ---------------  -----------------------------------------------------------------
  imagined_1       {'Word_2': 'sandpaper', 'Rating': 0.5}
  perceived_2      {'Word_2': 'sugar', 'Rating': 0.5}
  imagined_3       {'Word_2': 'suburb', 'Rating': 0.5}
  perceived_4      {'Word_2': 'chair', 'Rating': 0.5}

8. Variant 4 — Episodic conversation with feedback (`chain_type=task`, `memory=Convo`)¶

This replicates the run_rm_task_convo_fb.py configuration from the rmllm repo.

The feedback class Stim_Trial_Injection is adapted directly from rmllm/rmllm/rm_msg_injection.py. It implements the 3-method feedback contract expected by TaskRunner:

Method	When called	What it does
`__init__(trial_data)`	Once per trial, before anything	Store trial reference
`update_trial_stim(finput, fb_response)`	Before LLM call	Merge previous feedback + current stimulus into one JSON HumanMessage
`generate_feedback(trdata, pred_dict, input_dict, parser_status, trial_item_collector)`	After LLM call	Validate response; return JSON feedback string

Feedback logic:

Imagined trial (word_2 == '________'): checks the imagined word is novel (not reused from prior trials).
Perceived trial (word_2 is a real word): checks the model reported the exact given word.
Emits a structured JSON message with response feedback, rating feedback, and overall feedback.

Design note: TaskRunner instantiates a fresh Stim_Trial_Injection per trial, so cross-trial state (used-word list) lives in a module-level variable _used_words.
Call _reset_used_words() before starting a new run to avoid bleed-over from prior cells.

In [8]:

Copied!





import json
from langchain_core.messages import HumanMessage
from psychscanner import FeedbackBase

# Module-level word tracker — mirrors all_words_in_use in the original rmllm code.
_used_words = []

def _reset_used_words():
    """Reset the cross-trial word tracker before a new experiment run."""
    global _used_words
    _used_words = []


def _update_input_with_feedback(input_dict, fb_response):
    """Merge feedback from the previous trial with the current stimulus."""
    trial_dict = json.loads(input_dict["inputs"][0].content)
    merged_msg = HumanMessage(json.dumps(
        {**json.loads(fb_response), "current_trial": trial_dict},
        indent=4,
    ))
    input_dict["inputs"] = [merged_msg]
    return input_dict


class Stim_Trial_Injection(FeedbackBase):
    """Per-trial feedback injector for RM word-pair tasks.

    Adapted from rmllm/rmllm/rm_msg_injection.py.
    Implements the FeedbackBase contract used by psychscanner's TaskRunner.
    """

    def on_response(self, trial: dict, response: dict) -> str:
        """Evaluate word-pair RM response and return feedback JSON."""
        stim  = trial["stimulus"]
        word1 = stim["Word_Pair"]["word_1"].strip()
        word2 = stim["Word_Pair"]["word_2"].strip()

        try:
            vals     = list(response.values())
            word_ans = str(vals[0])    # Word_2
            rating   = vals[1]         # Rating
        except Exception:
            word_ans, rating = str(response), ""

        _used_words.append(word1)

        if "__" in word2:                              # imagined trial
            if word_ans.lower() in [w.lower() for w in _used_words]:
                _used_words.append(word_ans)
                response_fb = (
                    "**INCORRECT: Last trial was an imagined trial type. "
                    "Your 'Word_2' was already used in a previous trial. "
                    "Always imagine a novel word not used before.**"
                )
            else:
                _used_words.append(word_ans)
                response_fb = (
                    "**CORRECT: Last trial was an imagined trial type, "
                    "your 'Word_2' correctly followed the given instructions.**"
                )
        else:                                          # perceived trial
            _used_words.append(word2)
            _used_words.append(word_ans)
            if word2.lower() == word_ans.lower():
                response_fb = (
                    "**CORRECT: Last trial was a perceived trial type, "
                    "your 'Word_2' value correctly followed the trial instruction.**"
                )
            else:
                response_fb = (
                    f"**INCORRECT: Last trial was a perceived trial type. "
                    f"The second word was '{word2}' but you responded '{word_ans}'. "
                    f"Report the given word exactly as-is.**"
                )

        try:
            r = float(rating)
            rate_fb = (
                "**CORRECT: Your 'rating' value is in the instructed rating scale range.**"
                if 0 <= r <= 100 else
                "**INCORRECT: Your 'rating' value is outside the valid range (0-100).**"
            )
        except Exception:
            rate_fb = "**INCORRECT: Your 'rating' value could not be parsed as a number.**"

        if "INCORRECT" in response_fb or "INCORRECT" in rate_fb:
            overall_fb = "**INCORRECT Answer! Follow all trial instructions more accurately.**"
        else:
            overall_fb = "**CORRECT Answer! Well answered, Good Job!**"

        return json.dumps(
            {"Feedback on previous response": {
                "response feedback": response_fb,
                "rating feedback":   rate_fb,
                "overall feedback":  overall_fb,
            }},
            indent=4,
        )
import json
from langchain_core.messages import HumanMessage
from psychscanner import FeedbackBase

# Module-level word tracker — mirrors all_words_in_use in the original rmllm code.
_used_words = []

def _reset_used_words():
    """Reset the cross-trial word tracker before a new experiment run."""
    global _used_words
    _used_words = []


def _update_input_with_feedback(input_dict, fb_response):
    """Merge feedback from the previous trial with the current stimulus."""
    trial_dict = json.loads(input_dict["inputs"][0].content)
    merged_msg = HumanMessage(json.dumps(
        {**json.loads(fb_response), "current_trial": trial_dict},
        indent=4,
    ))
    input_dict["inputs"] = [merged_msg]
    return input_dict


class Stim_Trial_Injection(FeedbackBase):
    """Per-trial feedback injector for RM word-pair tasks.

    Adapted from rmllm/rmllm/rm_msg_injection.py.
    Implements the FeedbackBase contract used by psychscanner's TaskRunner.
    """

    def on_response(self, trial: dict, response: dict) -> str:
        """Evaluate word-pair RM response and return feedback JSON."""
        stim  = trial["stimulus"]
        word1 = stim["Word_Pair"]["word_1"].strip()
        word2 = stim["Word_Pair"]["word_2"].strip()

        try:
            vals     = list(response.values())
            word_ans = str(vals[0])    # Word_2
            rating   = vals[1]         # Rating
        except Exception:
            word_ans, rating = str(response), ""

        _used_words.append(word1)

        if "__" in word2:                              # imagined trial
            if word_ans.lower() in [w.lower() for w in _used_words]:
                _used_words.append(word_ans)
                response_fb = (
                    "**INCORRECT: Last trial was an imagined trial type. "
                    "Your 'Word_2' was already used in a previous trial. "
                    "Always imagine a novel word not used before.**"
                )
            else:
                _used_words.append(word_ans)
                response_fb = (
                    "**CORRECT: Last trial was an imagined trial type, "
                    "your 'Word_2' correctly followed the given instructions.**"
                )
        else:                                          # perceived trial
            _used_words.append(word2)
            _used_words.append(word_ans)
            if word2.lower() == word_ans.lower():
                response_fb = (
                    "**CORRECT: Last trial was a perceived trial type, "
                    "your 'Word_2' value correctly followed the trial instruction.**"
                )
            else:
                response_fb = (
                    f"**INCORRECT: Last trial was a perceived trial type. "
                    f"The second word was '{word2}' but you responded '{word_ans}'. "
                    f"Report the given word exactly as-is.**"
                )

        try:
            r = float(rating)
            rate_fb = (
                "**CORRECT: Your 'rating' value is in the instructed rating scale range.**"
                if 0 <= r <= 100 else
                "**INCORRECT: Your 'rating' value is outside the valid range (0-100).**"
            )
        except Exception:
            rate_fb = "**INCORRECT: Your 'rating' value could not be parsed as a number.**"

        if "INCORRECT" in response_fb or "INCORRECT" in rate_fb:
            overall_fb = "**INCORRECT Answer! Follow all trial instructions more accurately.**"
        else:
            overall_fb = "**CORRECT Answer! Well answered, Good Job!**"

        return json.dumps(
            {"Feedback on previous response": {
                "response feedback": response_fb,
                "rating feedback":   rate_fb,
                "overall feedback":  overall_fb,
            }},
            indent=4,
        )

In [9]:

Copied!





# Reset the used-word tracker before the V4 run
_reset_used_words()

card_v4 = make_card(
    'episodic_fb',
    memory='Convo', chain_type='task',
    parser='1',
    feedback=True, feedback_fn=Stim_Trial_Injection,
)
exp_v4   = ExpCard(card_v4)
print(f'Parser mode: {exp_v4.parser}')
scanner_v4 = ScannerModel(expcard=exp_v4)
trials_v4  = scanner_v4.run()[0]
print()
show_trials(trials_v4, label='V4 — episodic / task / +feedback (Stim_Trial_Injection)')
# Reset the used-word tracker before the V4 run
_reset_used_words()

card_v4 = make_card(
    'episodic_fb',
    memory='Convo', chain_type='task',
    parser='1',
    feedback=True, feedback_fn=Stim_Trial_Injection,
)
exp_v4   = ExpCard(card_v4)
print(f'Parser mode: {exp_v4.parser}')
scanner_v4 = ScannerModel(expcard=exp_v4)
trials_v4  = scanner_v4.run()[0]
print()
show_trials(trials_v4, label='V4 — episodic / task / +feedback (Stim_Trial_Injection)')

----<PROJECT AND DATA ROOT DIRECTORY>----

	Project root dir: /Users/saurabhext/Documents/PSYCHSCANNER/psychscanner/examples/_rm_tutorial_runs/episodic_fb

	Simulation data root dir: /Users/saurabhext/Documents/PSYCHSCANNER/psychscanner/examples/_rm_tutorial_runs/episodic_fb/episodic_fb/rm_2op_convo_fb/ollama_smollm2:360m-instruct-fp16_Convo

----<>----

Parser mode: dynamic
{'temperature': 0}
--<chat model>-- model='smollm2:360m-instruct-fp16' temperature=0.0

2026-05-04 16:06:11.712 | CRITICAL | psychscanner.session_tunnel.session_tunnel:create_tunnel:152 - BEGIN

TOTAL RUNS: 1	RESUME IDX: None

----<>---- task running

0it [00:00, ?it/s]

1it [00:10, 10.93s/it]

2it [00:15,  7.43s/it]

3it [00:20,  6.00s/it]

4it [00:25,  5.61s/it]

4it [00:25,  6.31s/it]

2026-05-04 16:06:36.956 | INFO     | psychscanner.session_tunnel.session_tunnel:scan_checkpoint:184 - scan-checkpoint

----<scanned runs>---- i = 0

2026-05-04 16:06:36.963 | CRITICAL | psychscanner.session_tunnel.session_tunnel:end_checkpoint:163 - END

--- V4 — episodic / task / +feedback (Stim_Trial_Injection) ---
  trcode           parsed response
  ---------------  -----------------------------------------------------------------
  imagined_1       {'Word_2': 'sandpaper', 'Rating': 0.5}
  perceived_2      {'Word_2': 'sugar', 'Rating': 0.5}
  imagined_3       {'Word_2': 'suburb', 'Rating': 0.5}
  perceived_4      {'Word_2': 'cushion', 'Rating': 0.5}

9. Reference: real rmllm experiment output (gemma3:12b, 100 simulations)¶

The rmllm repo (data/external/exp2/10t_100/ollama_gemma3_12b_Convo/) stores .psyscan files from running run_rm_task_convo_no_fb.py with 100 LLM-personas × 10 trials each.

The cell below loads the first simulation's first 3 trials to show the exact output format produced by the same model and task configuration used in V3 above.

In [10]:

Copied!





from pathlib import Path
import json

RMLLM_REPO = Path.home() / (
    'Library/CloudStorage/OneDrive-UniversityofFlorida/'
    '2020_2025_reality_monitoring_dissertation/D/rmllm'
)
ref_file = (
    RMLLM_REPO
    / 'data/external/exp2/10t_100/ollama_gemma3_12b_Convo'
    / '0-20251101_185522.psyscan'
)

if ref_file.exists():
    with open(ref_file) as fh:
        ref_data = json.load(fh)
    print('Real experiment — first 4 trials (gemma3:12b, no feedback, 10-trial task):')
    print(f'  {"trcode":15}  pred_resp content')
    print(f'  {"-"*15}  {"-"*45}')
    for trial in ref_data['taskdata'][:4]:
        print(f'  {trial["trcode"]:15}  {trial["pred_resp"]["content"]}')
else:
    print('rmllm repo not found at expected path; skipping reference data.')
    print()
    print('Expected output (from the real experiment):')
    ref_sample = [
        ('imagined_1',  "{'Word_2': 'rough',   'Rating': 75.0}"),
        ('perceived_2', "{'Word_2': 'sweet',   'Rating': 95.0}"),
        ('imagined_3',  "{'Word_2': 'highway', 'Rating': 15.0}"),
        ('perceived_4', "{'Word_2': 'cushion', 'Rating': 88.0}"),
    ]
    for trcode, content in ref_sample:
        print(f'  {trcode:15}  {content}')
from pathlib import Path
import json

RMLLM_REPO = Path.home() / (
    'Library/CloudStorage/OneDrive-UniversityofFlorida/'
    '2020_2025_reality_monitoring_dissertation/D/rmllm'
)
ref_file = (
    RMLLM_REPO
    / 'data/external/exp2/10t_100/ollama_gemma3_12b_Convo'
    / '0-20251101_185522.psyscan'
)

if ref_file.exists():
    with open(ref_file) as fh:
        ref_data = json.load(fh)
    print('Real experiment — first 4 trials (gemma3:12b, no feedback, 10-trial task):')
    print(f'  {"trcode":15}  pred_resp content')
    print(f'  {"-"*15}  {"-"*45}')
    for trial in ref_data['taskdata'][:4]:
        print(f'  {trial["trcode"]:15}  {trial["pred_resp"]["content"]}')
else:
    print('rmllm repo not found at expected path; skipping reference data.')
    print()
    print('Expected output (from the real experiment):')
    ref_sample = [
        ('imagined_1',  "{'Word_2': 'rough',   'Rating': 75.0}"),
        ('perceived_2', "{'Word_2': 'sweet',   'Rating': 95.0}"),
        ('imagined_3',  "{'Word_2': 'highway', 'Rating': 15.0}"),
        ('perceived_4', "{'Word_2': 'cushion', 'Rating': 88.0}"),
    ]
    for trcode, content in ref_sample:
        print(f'  {trcode:15}  {content}')

Real experiment — first 4 trials (gemma3:12b, no feedback, 10-trial task):
  trcode           pred_resp content
  ---------------  ---------------------------------------------
  imagined_1       {'Word_2': 'rough', 'Rating': 75.0}
  perceived_2      {'Word_2': 'sweet', 'Rating': 98.0}
  imagined_3       {'Word_2': 'residential', 'Rating': 85.0}
  perceived_4      {'Word_2': 'cushion', 'Rating': 78.0}

10. Side-by-side summary¶

Same word pairs, four different parser + memory + feedback wirings:

Variant	Parser path	Response keys	Memory scope
V1	registry → `ResponseRmStEI`	Word_2, Rating, Judgment, Confidence	none (per-item)
V2	registry → `AllResponseRMEI`	response.{Word_2 \| Relatedness_Rating \| Judgment \| Confidence}	trial thread
V3	dynamic → `Response_part_1_rm`	Word_2, Rating	task thread (full history)
V4	dynamic → `Response_part_1_rm`	Word_2, Rating (+ feedback injected between trials)	task thread (full history)

In [11]:

Copied!





summary = {
    'V1 singleturn / item / no fb'   : trials_v1,
    'V2 trialchain / trial / no fb'  : trials_v2,
    'V3 episodic / task / no fb'     : trials_v3,
    'V4 episodic / task / +feedback' : trials_v4,
}
for label, trials in summary.items():
    print(f'=== {label} ({len(trials)} trials) ===')
    for t in trials:
        parsed = parse_pred_resp(t['pred_resp'])
        print(f'  {t["trcode"]:15} -> {parsed}')
    print()
summary = {
    'V1 singleturn / item / no fb'   : trials_v1,
    'V2 trialchain / trial / no fb'  : trials_v2,
    'V3 episodic / task / no fb'     : trials_v3,
    'V4 episodic / task / +feedback' : trials_v4,
}
for label, trials in summary.items():
    print(f'=== {label} ({len(trials)} trials) ===')
    for t in trials:
        parsed = parse_pred_resp(t['pred_resp'])
        print(f'  {t["trcode"]:15} -> {parsed}')
    print()

=== V1 singleturn / item / no fb (4 trials) ===
  imagined_1      -> {'Word_2': 'sandpaper', 'Rating': 3.0, 'Judgment': 'external', 'Confidence': 4}
  perceived_2     -> {'Word_2': 'sweet', 'Rating': 3.0, 'Judgment': 'external', 'Confidence': 4}
  imagined_3      -> {'Word_2': 'suburb', 'Rating': 3.0, 'Judgment': 'external', 'Confidence': 4}
  perceived_4     -> {'Word_2': 'cushion', 'Rating': 3.0, 'Judgment': 'external', 'Confidence': 4}

=== V2 trialchain / trial / no fb (8 trials) ===
  imagined_1      -> {'response': {'Word_2': 'sandpaper'}}
  imagined_1      -> {'response': {'Relatedness_Rating': 100.0}}
  imagined_1      -> {'response': {'Judgment': 'external'}}
  imagined_1      -> {'response': {'Confidence': 5}}
  perceived_2     -> {'response': {'Word_2': 'sweet'}}
  perceived_2     -> {'response': {'Relatedness_Rating': 100.0}}
  perceived_2     -> {'response': {'Judgment': 'external'}}
  perceived_2     -> {'response': {'Confidence': 5}}

=== V3 episodic / task / no fb (4 trials) ===
  imagined_1      -> {'Word_2': 'sandpaper', 'Rating': 0.5}
  perceived_2     -> {'Word_2': 'sugar', 'Rating': 0.5}
  imagined_3      -> {'Word_2': 'suburb', 'Rating': 0.5}
  perceived_4     -> {'Word_2': 'chair', 'Rating': 0.5}

=== V4 episodic / task / +feedback (4 trials) ===
  imagined_1      -> {'Word_2': 'sandpaper', 'Rating': 0.5}
  perceived_2     -> {'Word_2': 'sugar', 'Rating': 0.5}
  imagined_3      -> {'Word_2': 'suburb', 'Rating': 0.5}
  perceived_4     -> {'Word_2': 'cushion', 'Rating': 0.5}

Key takeaways¶

parser='1' (registry) vs parser='dynamic'
'1' reads the class name from the task JSON and resolves it through PARSER_REGISTRY — safe, no eval(), works for any registered parser.
'dynamic' is a hardcoded RM routing built into single_turn_convo.py that switches between two parsers based on trcode.
Removing format templates from instructions
The original rm_2op.json includes a response_format block with literal placeholder strings like "<your word…>". When using structured output (Pydantic schema), this block must be omitted — otherwise the model echoes back the placeholder text instead of its own answer. V1's task JSON (rm_singleturn_demo.json) omits it.
chain_type controls thread persistence
item → stateless; trial → shared thread across stimuli within one trial; task → one thread for the whole task.
Stim_Trial_Injection feedback format
The feedback injector wraps the feedback JSON and the next trial's stimulus into a single HumanMessage, giving the model both correction signal and new task in one turn. This mirrors exactly what run_rm_task_convo_fb.py does in the rmllm experiments.
Cross-trial state in feedback
TaskRunner reinstantiates Stim_Trial_Injection every trial. Cross-trial state (word tracking) must live in a module-level variable, not on self. Reset with _reset_used_words() before each new run. This is a known design limitation — on the planning list to replace with a stateful FeedbackEngine class.