Skip to main content
Hindsight is State-of-the-Art on Memory for AI Agents | Read the paper β†’
πŸ€–
Using a coding agent? Run this to install the Hindsight docs skill:
npx skills add https://github.com/vectorize-io/hindsight --skill hindsight-docs

Memory Defense

Hindsight scrubs secrets and PII from retain content using a 45-pattern regex set. Each match is replaced with a [REDACTED:type] marker before content reaches memory units or the document body. The feature is configured per bank and disabled by default.

How it works​

Memory Defense is opt-in per bank. The extension is always present, but it sits dormant until you give a bank a policy that turns it on. When a policy is set, every memory the agent writes to that bank is scanned before it lands in storage. When the scanner recognizes a credential, an API key, a database connection string, or a known PII format, the matched substring is replaced with a redaction marker like [REDACTED:github_token].

The scrubbed version is what actually gets stored. Memory units and document bodies persist the redacted text, so future recall responses, exports, and reflect operations never see the original secret.

A policy only affects future retain calls on the bank where it is set. Existing memories are not retroactively scanned when you add or change a policy.

Configuring Memory Defense​

Memory Defense is configured per bank via the bank's memory_defense config field. You can set the policy at bank creation time or update it later via PATCH /v1/{tenant}/banks/{bank_id}/config.

The open-source version implements the sensitive_data rule with two possible actions:

  • redact β€” replace each matched secret with a [REDACTED:type] marker and store the scrubbed memory.
  • block β€” drop any item that contains a match. If every item in a retain request is blocked, the call returns 422.

A minimal policy:

{
"memory_defense":{
"enabled":true,
"rules":[
{"on":"sensitive_data","action":"redact"}
]
}
}

Once that policy is on a bank, every retain to that bank is screened with the 45 redaction patterns documented below.

Existing memories are not retroactively scanned

Enabling Memory Defense on a bank only affects future retain calls. Memories already in the bank are not re-scanned or modified when you add or change a policy. If you need to scrub a bank that already contains unredacted content, you have to re-ingest the affected memories or remove them manually.

Disabled by default​

Memory Defense is off on every bank until you set a policy. A bank with no memory_defense field, with enabled: false, or with no sensitive_data rule is treated identically: the extension returns ALLOW and content passes through unchanged. To stop redacting on a bank that has it on, set enabled: false or remove the policy.

Notifications​

When an item is redacted or blocked, Hindsight fires a memory_defense.triggered webhook if a webhook on the bank is subscribed to that event type. The payload reports the action taken, the document ID, and which redaction patterns matched β€” useful for routing security alerts to a SIEM or Slack. Clean items fire no event.

The same redact/block decisions are also recorded as memory_defense entries in the audit log (when audit logging is enabled), with the action and matched pattern labels in the entry metadata.

Patterns covered​

The 45 bundled patterns cover the categories below.

AI and LLM providers​

LabelCatches
anthropic_keysk-ant-...
openai_key, openai_project_key, openai_admin_keysk-..., sk-proj-..., sk-admin-...
google_api_keyAIza... (39 chars)
google_oauth_tokenya29.<token>
xai_keyxai-...
groq_keygsk_...
huggingface_tokenhf_...
replicate_tokenr8_...
perplexity_keypplx-...
databricks_tokendapi<hex32>

Cloud providers​

LabelCatches
aws_access_keyAKIA<16>
aws_session_tokenASIA<16>
digitalocean_tokendop_v1_<hex64>

Source control and CI​

LabelCatches
github_fg_patgithub_pat_...
github_tokenghp_<36>
github_app_tokenghs_<36>
github_user_tokenghu_<36>
github_refreshghr_<36>
github_oauthgho_<36>
gitlab_patglpat-...
npm_tokennpm_...
pypi_tokenpypi-AgEIcHlwaS5vcmc...

Payment processors​

LabelCatches
stripe_secretsk_live_..., sk_test_...
stripe_restrictedrk_live_..., rk_test_...
square_tokensq0...
braintree_tokenaccess_token$production$...

Communications and email​

LabelCatches
slack_tokenxoxb-, xoxp-, xoxa-, xoxr-
slack_webhookhttps://hooks.slack.com/services/...
twilio_api_keySK<hex32>
twilio_account_sidAC<hex32>
sendgrid_keySG.<22>.<43>
mailgun_keykey-<32>
discord_bot<MNO><23>.<6>.<27>
telegram_bot<8-10 digits>:<35>

Commerce​

LabelCatches
shopify_tokenshpat_<hex32>

Database connection strings​

LabelCatches
db_url_postgrespostgres://user:pass@host or postgresql://...
db_url_mysqlmysql://user:pass@host
db_url_mongodbmongodb://user:pass@host or mongodb+srv://...

Private keys, JWTs, and generic credentials​

LabelCatches
private_key_pem-----BEGIN ... PRIVATE KEY----- PEM blocks
jwteyJ<header>.eyJ<payload>.<signature>

PII (US defaults)​

LabelCatches
credit_card13 to 19 digits with regular separators
ssn_us123-45-6789 shape

AltStyle γ«γ‚ˆγ£γ¦ε€‰ζ›γ•γ‚ŒγŸγƒšγƒΌγ‚Έ (->γ‚ͺγƒͺγ‚ΈγƒŠγƒ«) /