Whitepaper: Exception-Driven Cognition in Agentic Workers

Shame, Surprise, Curiosity, and Distrust as Signals for Memory, Learning, and Governance

Tony Wood
London, UK

Abstract

As agentic artificial intelligence systems transition from episodic task execution to continuous operation, the design of memory becomes a critical and under-examined challenge. Prevailing approaches treat memory as an accumulation problem, prioritising exhaustive logging and post-hoc retrieval. This paper argues that such approaches are misaligned with effective learning and decision-making in long-running agents. Drawing on principles from human cognition and organisational learning, we propose an exception-driven memory framework in which memory formation is gated by four operational signals: surprise, shame, curiosity, and distrust. These signals function as indicators of significance rather than anthropomorphic emotions. When triggered, they initiate structured memory creation, classified as discovery or rediscovery, and weighted by recency and durability. We present the theoretical motivation for this framework, describe its implementation within agentic systems, and outline an experimentation rig used to evaluate its effectiveness. The contribution of this paper is a practical, testable model for memory, learning, and governance in agentic workers that reduces noise, improves retrieval relevance, and embeds accountability directly into system behaviour.

1. Introduction

Agentic AI systems are increasingly designed to operate continuously, maintaining persistent state, engaging in extended interactions, and executing sequences of decisions over long time horizons. While significant research has focused on reasoning, planning, and tool use in such systems, comparatively little attention has been paid to how these agents should remember. Memory is often implemented as an unfiltered historical record, relying on downstream retrieval mechanisms to extract relevance. This approach scales poorly as agents accumulate experience and leads to declining signal-to-noise ratios in decision support.

In contrast, human cognition demonstrates a strikingly different approach to memory. Humans do not store experience exhaustively. Instead, memory formation is highly selective, privileging events that violate expectation, expose responsibility, introduce novelty, or signal risk. This selectivity enables learning, adaptability, and governance within bounded cognitive resources.

This paper explores how analogous principles can be operationalised in agentic workers. We propose an exception-driven cognition model in which memory formation is explicitly gated by signals of significance. Specifically, we focus on four signals that recur across human learning, organisational failure analysis, and system governance: surprise, shame, curiosity, and distrust. We argue that these signals provide a compact yet expressive basis for determining when memory should be created, how it should be classified, and how it should influence future behaviour.

2. Background and Related Work

2.1 Memory in Artificial Agents

Memory in artificial agents has traditionally been approached from a data management perspective, emphasising storage capacity, indexing, and retrieval efficiency. Techniques such as episodic memory buffers, vector databases, and retrieval-augmented generation focus on improving access to stored information rather than on selective memory formation. While effective for short-term tasks, these approaches face challenges in long-running systems where memory growth leads to retrieval dilution and increased latency.

2.2 Human Cognition and Selective Memory

Research in cognitive psychology consistently demonstrates that memory is selective and event-driven. Salient events, particularly those involving surprise or emotional arousal, are more likely to be encoded into long-term memory. Importantly, this selectivity is not merely emotional but functional, enabling adaptive behaviour under uncertainty.

2.3 Organisational Learning and Governance

In organisational contexts, learning failures are often attributed not to lack of information but to failure to retain, retrieve, or act upon prior knowledge. Concepts such as “lessons learned” and “institutional memory” highlight the importance of rediscovery, where known issues reoccur due to insufficient integration into governance mechanisms.

This paper situates itself at the intersection of these literatures, proposing a model that bridges individual cognition, organisational learning, and agentic system design.

3. Conceptual Framework: Exception-Driven Cognition

3.1 Memory as a Gated Process

We begin from the premise that memory formation should be an intentional act triggered by significance rather than an automatic consequence of activity. In this model, routine events are not stored. Memory is created only when a signal indicates that the event has implications for future behaviour.

3.2 Operational Signals

The framework defines four primary signals that gate memory formation.

3.2.1 Surprise

Surprise is defined as a deviation between expected and observed outcomes that exceeds acceptable tolerance. In agentic systems, surprise indicates model failure or environmental change. It serves as the primary trigger for discovery, prompting revision of internal representations.

3.2.2 Shame

Shame is conceptualised as a responsibility recognition signal rather than an affective state. It arises when a failure occurs that should have been anticipated or prevented by existing processes or controls. In agentic workers, shame highlights process deficiencies and drives corrective learning without attributing blame to individuals.

3.2.3 Curiosity

Curiosity represents the detection of novelty or ambiguity that may hold future relevance. Unlike surprise, curiosity does not imply error or failure. Instead, it supports exploratory learning and capability expansion. Curiosity-driven memories often begin with low confidence and gain significance through repetition.

3.2.4 Distrust

Distrust functions as a risk detection signal, triggered by inconsistency, misalignment, or unverifiable information. It underpins governance, security, and compliance functions within agentic systems and ensures that trust is conditional rather than assumed.

4. Memory Classification: Discovery and Rediscovery

Upon signal activation, the agent evaluates whether the event represents a discovery or a rediscovery.

A discovery introduces genuinely new knowledge into the agent’s model of the world. A rediscovery occurs when the event matches existing memory, indicating repetition or persistence. This distinction is critical, as rediscovery highlights systemic issues and governance failures rather than informational gaps.

Explicit modelling of rediscovery allows agentic systems to recognise recurring patterns and avoid repeating known mistakes.

5. Temporal Weighting: Recency and Durability

Memory relevance is not static. Recent events often carry greater weight in volatile domains, while certain knowledge retains relevance regardless of age. The proposed framework therefore applies temporal weighting based on both recency and durability, allowing agents to adapt to change without erasing foundational constraints such as legal or ethical obligations.

6. Implementation in Agentic Workers

The framework is implemented through intentional memory gating within agentic architectures. Memory objects are structured to include signal type, contextual information, impact assessment, actions taken, outcomes, confidence, and temporal metadata. Memory propagation is layered, distinguishing between local agent memory and shared organisational memory.

This approach supports autonomy while enabling collective learning.

7. Experimental Methodology

To evaluate the framework, an experimentation rig has been developed to simulate continuous agent operation under realistic conditions. The rig introduces routine activity interspersed with injected anomalies designed to trigger the four signals. Metrics include memory growth rate, retrieval relevance, decision quality, and recurrence of known errors.

While full quantitative results are ongoing, early qualitative findings indicate reduced memory noise and improved retrieval alignment with decision-critical contexts.

8. Limitations and Risks

The framework introduces risks that must be managed. Excessive curiosity can lead to noise, shame can degrade into blame, distrust can become paranoia, and surprise can be triggered by trivial variance. These risks are mitigated through thresholding, corroboration requirements, and human-in-the-loop oversight.

9. Discussion

The proposed framework shifts the design of agentic memory from passive storage to active judgement. By embedding significance detection into memory formation, agentic workers can align learning with responsibility and governance. This approach also provides a conceptual bridge between human cognition and artificial systems without relying on anthropomorphic assumptions.

10. Conclusion

As agentic systems scale, the limiting factor will not be computational capacity but discernment. This paper presents an exception-driven model of cognition that enables agentic workers to remember selectively, learn effectively, and govern responsibly. By operationalising shame, surprise, curiosity, and distrust as signals rather than emotions, the framework offers a practical path toward accountable, adaptive agentic systems.

References (Indicative)

• Argyris, C., & Schön, D. (1978). Organizational Learning.

• Kahneman, D. (2011). Thinking, Fast and Slow.

• Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning.

• Tulving, E. (1983). Elements of Episodic Memory.