Signal Brief

Bad Memory: Evaluating Prompt Injection Risks from Memory in Agentic Systems

University of Washington 研究发现 Agent 系统存在内存提示词注入风险:当模型在交互中遇到存储在内存文件(如 CLAUDE.md)中的恶意指令时,即便当前会话拒绝了攻击,通常也不会自动删除该恶意指令,导致风险在后续会话中持续存在。研究人员针对 4 个模型(包含 Claud...

twitter关注列表 Rohan Paul (@rohanpaul_ai) 发布 2026-07-26 收录 2026-07-26 观察

一句话判断

揭示了 Agent 长期记忆机制带来的安全隐患,需关注 Agent 在处理持久化上下文时的清理机制。

核心信息

University of Washington 研究发现 Agent 系统存在内存提示词注入风险:当模型在交互中遇到存储在内存文件(如 CLAUDE.md)中的恶意指令时,即便当前会话拒绝了攻击,通常也不会自动删除该恶意指令,导致风险在后续会话中持续存在。研究人员针对 4 个模型(包含 Claude Opus)在 multi-session probe sequences 实验中进行了 10 次试验。

原始内容

When a model/agent hits a malicious instruction sitting in a memory file, it can refuse an attack and still leave it armed for the next session. University of Washington study of prompt injection in agent memory. It found, when an agent faces an attack from a malicious instruction sitting in a memory file, there are 2 separate questions: - Does it obey? Opus mostly doesn't. - Does it remove it? Usually no. It leaves the line in behaviors.md and asks you whether you'd like it deleted. The team inserted payloads into persistent workspace files (CLAUDE.md, AGENTS.md, behavior notes) and then ran claude-code and codex over 4 models, using multi-session probe sequences with ten trials each. The point is that memory files are loaded from scratch at the beginning of each session. This implies that a refusal is only valid for the session in which it was issued. Tomorrow they will leave the problematic line alone in anticipation of the next run . This run could be a sub-agent , a cheaper model , or you , after you have downgraded to Haiku because you hit your token limit . Models are likely to follow suit. – arxiv. org/abs/2607.14611 Title: "Bad Memory: Evaluating Prompt Injection Risks from Memory in Agentic Systems" ![photo](https://pbs.twimg.com/media/HOKAix8bEAA5KJa.jpg)

相关动态

01

From Memory to Skills: Evidence-Grounded Co-Evolution Governance for Long-Horizon LLM Agents

论文指出大多数代理记忆系统只保存事件并在上下文召回,导致重复推理错误。作者提出MSCE方法,将经验组织为声明性事实、诱导程序化策略和基础步骤轨迹,仅将重复模式提升为可调用的过程(如pip安装失败时运行规则),每个策略需具备触发条件、边界、正向收益估计和证据支持。

twitter关注列表2026-07-26#AI#研究#技术
观察
02

Understanding Reasoning from Pretraining to Post-Training

论文通过象棋实验(5M-1B参数模型,36种预训练-RL组合)发现:预训练验证损失能高精度预测后RL的pass@1(相关性从0.93到0.99),RL计算量每10倍增益与预训练token数对数正相关(r=+0.84);RL最优计算量份额在50M参数时为20%,680M时为28%,且RL会放大模型在困难问题上的错误模式。

twitter关注列表2026-07-26#研究#模型#技术
观察
03

Human Capital, AI, and Labor Commoditization

一项研究分析AI对自由职业市场的影响,发现在ChatGPT后,AI暴露岗位中人力资本信号重要性下降7.8%,价格信号重要性上升1.1%,导致需求向更廉价工人转移,支持AI使这些工人更可替代的观点。

twitter关注列表2026-07-26#AI#研究#行业动态
观察
05

AMD 发布 Instella 16B MoE 基础模型

AMD 发表 Instella 16B MoE 基础模型,全程开源,提供预训练、长上下文、精调、DPO 及强化学习等多个阶段的检查点,公布数据集、训练方案及复现代码,这是 AMD 首个 MoE 基础模型,也是首个采用 FarSkip 架构的模型。

twitter关注列表2026-07-25#模型发布#开源#技术
观察