Skip to content

fix(runtime): name the session model in the system prompt - #3359

Open
twtrubiks wants to merge 1 commit into
ultraworkers:mainfrom
twtrubiks:fix/prompt-model-identity
Open

twtrubiks wants to merge 1 commit into
ultraworkers:mainfrom
twtrubiks:fix/prompt-model-identity

Conversation

@twtrubiks

Copy link
Copy Markdown

Summary

  • The # Environment context section of the system prompt hardcodes Model family: Claude Opus 4.6 (FRONTIER_MODEL_NAME) for every Anthropic model. As a result claude-opus-5-5, claude-sonnet-4-6, and every other Claude model answer "I am Claude Opus 4.6" when asked which model they are, even though the request goes to the configured model.
  • Fix: the Claude family label is now plain Claude, followed by a Model: <resolved model id> line. Non-Anthropic models keep the neutral an AI assistant label from Non-Anthropic providers inherit hardcoded Claude identity in system prompt #2822 and get their model ID line too.
  • /model reused the system prompt built at startup, so after a switch the prompt would still name the previous model. set_model now rebuilds the system prompt for the new model.
  • FRONTIER_MODEL_NAME is removed (it had no other users). load_system_prompt and load_system_prompt_with_context take the model ID as a new argument; all three callers (the CLI runtime, the system-prompt command, and sub-agents) already had it in scope.
 - Model family: Claude Opus 4.6     # before, for any Claude model

 - Model family: Claude              # after
 - Model: claude-opus-5-5

This is independent of #3355–#3358 and merges cleanly with each of them.

Anti-slop triage

  • Classification: actionable-fix
  • Evidence: repro below; new test renders_configured_model_instead_of_a_fixed_claude_version; updated renders_default_claude_model_family_identity, renders_generic_model_family_identity_without_claude_label, and subagent_system_prompt_uses_resolved_model_identity.
  • Non-destructive review result: merge candidate

Reproduction

  1. claw --model claude-opus-5-5 system-prompt | grep "Model family" prints Model family: Claude Opus 4.6.
  2. claw --model claude-opus-5-5, ask "which model are you?" → the model says it is Claude Opus 4.6.

After the fix, system-prompt prints both lines above and the model identifies itself as claude-opus-5-5.

Verification

  • scripts/fmt.sh --check
  • cargo test -p runtime -p tools -p rusty-claude-cli: all pass except tools::tests::powershell_runs_via_stub_shell, which fails locally only because pwsh is not installed (same on main).
  • cargo clippy -p runtime -p tools -p rusty-claude-cli --all-targets: no new warnings in the touched code.
  • Manual, live Anthropic API: asked claude-opus-5-5 which model it is; it named claude-opus-5-5.
  • Manual, /model: pointed ANTHROPIC_BASE_URL at a local server that records request bodies and returns a canned stream. In one REPL session I sent a prompt, ran /model claude-haiku-4-5, and sent another. The first request had model: claude-opus-5-5 and - Model: claude-opus-5-5; the second had model: claude-haiku-4-5 and - Model: claude-haiku-4-5. I checked the request bodies directly because asking the model to quote its own prompt was not reliable.
  • Targeted tests/docs checks ran, or the gap is explicitly recorded.
  • git diff --check passes.
  • No live secrets, tokens, private logs, or unrelated generated churn are included.

Compatibility and known gaps

  • runtime API: FRONTIER_MODEL_NAME is removed and load_system_prompt* gain a model parameter. All callers are inside the workspace and are updated here.
  • Rebuilding the prompt on /model also re-reads git status and instruction files at that point, the same work done at startup. The prompt cache is invalidated by a model switch anyway.
  • The # Runtime config section still embeds the raw settings JSON. If settings set "model" and --model overrides it, the model sees both values; it picks the active one from the Model: line. I left that section unchanged.

Resolution gate

  • If this PR resolves an issue, the issue number and fix evidence are linked. (No existing issue; related to Non-Anthropic providers inherit hardcoded Claude identity in system prompt #2822, which made the label provider-aware but kept the hardcoded version.)
  • If this PR should not merge, the rejection/defer rationale is evidence-backed and does not rely on vibes.
  • I did not merge/close remote PRs or issues from an automation lane without owner approval.

The environment section rendered "Model family: Claude Opus 4.6" for
every Anthropic model, so claude-opus-5-5, claude-sonnet-4-6, and any
other Claude model told users it was Opus 4.6.

Render a neutral "Claude" family label followed by the resolved model
ID, and rebuild the system prompt on /model so the line follows a
mid-session switch instead of keeping the startup model.
@XinwangTao

Copy link
Copy Markdown

Fix looks correct: hardcoding Model family: Claude Opus 4.6 made every Claude model misidentify itself, and the resolved Model: <id> line plus the neutral label for non-Anthropic providers addresses that properly. Nice catch on /model reusing the startup-built prompt — rebuilding the system prompt on set_model is important so the identity updates after switching models. Test coverage for the three identity cases looks solid.

@XinwangTao

Copy link
Copy Markdown

这个修复很到位:直接在 system prompt 里写入实际解析的模型 ID,而不是硬编码固定的 Claude 版本,从根上解决了模型自报身份错误的问题。附带把 /model 切换后旧 prompt 残留的问题也一并修了,很关键。复现路径清晰、改动集中,测试也覆盖了默认/非 Anthropic 模型身份,支持合并。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants