Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Add SQLite-based memory service implementation #2768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Raman369AI wants to merge 33 commits into google:main
base: main
Choose a base branch
Loading
from Raman369AI:main

Conversation

@Raman369AI
Copy link

@Raman369AI Raman369AI commented Aug 28, 2025
edited
Loading

Summary

This PR introduces a new SqliteMemoryService that provides persistent storage for agent memory using SQLite with async operations.

🔄 UPDATED: Addressed all reviewer feedback with significant performance and robustness improvements based on code review suggestions.

Key Improvements Implemented

1. ⚡ Performance: FTS5 Full-Text Search

Issue: Original implementation fetched all memory entries and performed keyword matching in Python, causing performance bottlenecks with large datasets.

Solution: Implemented SQLite FTS5 (Full-Text Search) for database-level search optimization:

  • FTS5 virtual table for efficient full-text indexing
  • Native MATCH queries instead of Python-based filtering
  • Automatic synchronization triggers between main and FTS5 tables
  • Query preparation with FTS5-specific character escaping

Performance Impact: O(n) → O(log n) search complexity, scales to millions of entries

2. 🚀 Performance: Lazy Initialization

Issue: _init_db() called repeatedly on every method invocation, adding unnecessary overhead.

Solution: Implemented one-time lazy initialization pattern:

  • asyncio.Lock for thread-safe initialization
  • _ensure_initialized() method with double-checked locking
  • Database setup occurs only once on first operation

Performance Impact: Eliminated redundant database setup calls

3. 🛡️ Robustness: Specific Exception Handling

Issue: Generic Exception catching masked specific errors and hindered debugging.

Solution: Implemented precise error handling:

  • Catch specific aiosqlite.Error instead of generic Exception
  • Proper error handling during database initialization
  • Graceful degradation with meaningful error messages

Reliability Impact: Better debugging, error recovery, and system stability

Architecture & Features

  • Persistent SQLite storage with automatic table creation
  • Async operations using aiosqlite for non-blocking database access
  • User isolation by app_name and user_id with efficient filtering
  • FTS5 virtual table with automatic synchronization
  • Robust error handling with specific exception types
  • Additional utility methods: clear_memory() and get_memory_stats()
  • Duplicate prevention with UNIQUE constraints
  • Content filtering for empty or invalid events

Files Changed

  • src/google/adk/memory/sqlite_memory_service.py - Enhanced with FTS5 and optimizations
  • tests/unittests/memory/test_sqlite_memory_service.py - Expanded test suite (32 tests)
  • src/google/adk/memory/__init__.py - Export SqliteMemoryService
  • pyproject.toml - Added aiosqlite>=0.20.0 dependency
  • .gitignore - Added SQLite database files and development entries

Database Schema

-- Main table for memory storage
CREATE TABLE memory_entries (
 id INTEGER PRIMARY KEY AUTOINCREMENT,
 app_name TEXT NOT NULL,
 user_id TEXT NOT NULL,
 session_id TEXT NOT NULL,
 author TEXT,
 timestamp REAL NOT NULL,
 content_json TEXT NOT NULL,
 text_content TEXT NOT NULL,
 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
 UNIQUE(app_name, user_id, session_id, timestamp, author)
);
-- FTS5 virtual table for efficient search
CREATE VIRTUAL TABLE memory_entries_fts USING fts5(
 app_name, user_id, text_content,
 content='memory_entries', content_rowid='id'
);
-- Auto-sync triggers
CREATE TRIGGER memory_entries_ai AFTER INSERT ON memory_entries ...
CREATE TRIGGER memory_entries_ad AFTER DELETE ON memory_entries ...
CREATE TRIGGER memory_entries_au AFTER UPDATE ON memory_entries ...

Testing Results ✅

  • 32 comprehensive test cases (8 new tests added for improvements)
  • All tests passing (32/32 ✅)
  • Enhanced coverage including:
    • FTS5 table creation and synchronization
    • Lazy initialization with asyncio.Lock
    • FTS5 query preparation and edge cases
    • Database error handling and recovery scenarios
    • Empty query validation
    • Performance with multiple memory entries

Test Execution

pytest tests/unittests/memory/test_sqlite_memory_service.py -v
# Result: 32 passed in 3.43s ✅

Performance Comparison

Before (Original):

# Fetched ALL entries, filtered in Python
rows = await cursor.fetchall()
for row in rows:
 if keyword_matches_in_python(row): # O(n) per query
 memories.append(row)

After (FTS5 Optimized):

# Database-level search with FTS5
SELECT m.* FROM memory_entries_fts f
JOIN memory_entries m ON f.rowid = m.id
WHERE f.memory_entries_fts MATCH ? # O(log n) with indexes

Breaking Changes

None - this is a backward-compatible enhancement that maintains the same public API.

Dependencies

  • aiosqlite>=0.20.0, <1.0.0 for async SQLite operations

Usage Example

from google.adk.memory import SqliteMemoryService
# High-performance persistent memory
memory_service = SqliteMemoryService(db_path='./agent_memory.db')
# Use in agent configuration
root_agent = Agent(
 name='my_agent',
 model='gemini-2.0-flash',
 memory_service=memory_service,
 # ... other config
)

This implementation now addresses all reviewer feedback and provides production-ready performance that scales efficiently from development to enterprise use cases.

Ready for review! 🚀

magarafael reacted with eyes emoji
This commit introduces a new SqliteMemoryService that provides persistent
storage for agent memory using SQLite with async operations via aiosqlite.
Features:
- Persistent storage with SQLite database
- Async operations using aiosqlite for non-blocking database access
- Keyword-based search matching InMemoryMemoryService behavior
- User isolation by app_name and user_id
- Automatic database initialization with optimized indexes
- Additional utility methods: clear_memory() and get_memory_stats()
- Duplicate entry prevention with UNIQUE constraints
- Content filtering for empty or invalid events
Implementation follows existing ADK patterns:
- Extends BaseMemoryService with @OverRide decorators
- Similar interface to InMemoryMemoryService and VertexAi services
- Comprehensive test coverage with 24 test cases
- Code formatted with project standards (isort + pyink)
Dependencies:
- Added aiosqlite>=0.20.0 dependency to pyproject.toml
- Updated memory module __init__.py to export SqliteMemoryService
Testing:
- Complete test suite in tests/unittests/memory/test_sqlite_memory_service.py
- All 24 tests passing covering initialization, CRUD operations, search,
 user isolation, memory management, and edge cases
Copy link

google-cla bot commented Aug 28, 2025

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @Raman369AI, I'm Gemini Code Assist1 ! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new SqliteMemoryService designed to provide persistent storage for agent memory. It utilizes SQLite with aiosqlite for asynchronous operations, offering a robust solution for local and small-scale deployments. The service supports keyword-based search, ensures user isolation, and includes database optimizations for efficient data handling. This new feature bridges the gap between simple in-memory solutions and more complex cloud-based services, providing a production-ready alternative without affecting existing functionalities.

Highlights

  • New Persistent Memory Service: A new SqliteMemoryService has been introduced, providing persistent storage for agent memory using SQLite.
  • Asynchronous Database Operations: The service leverages aiosqlite for asynchronous database operations, ensuring non-blocking performance.
  • Keyword-based Search: Memory search is keyword-based, aligning with the behavior of InMemoryMemoryService for consistency.
  • User and Application Isolation: The implementation ensures user isolation by filtering memory entries based on app_name and user_id.
  • Database Optimization: Database optimization includes the creation of indexes for faster searches and efficient timestamp ordering.
  • Memory Management Utilities: Utility methods clear_memory() and get_memory_stats() are provided for memory management and insights.
  • Robustness and Data Integrity: The service includes mechanisms for duplicate prevention using unique constraints and content filtering for empty or invalid events.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Collaborator

adk-bot commented Aug 28, 2025

Response from ADK Triaging Agent

Hello @Raman369AI, thank you for your contribution!

Before we can review this PR, could you please sign the Contributor License Agreement (CLA)? You can find more information at https://cla.developers.google.com/.

Additionally, this PR introduces a new feature. Could you please create a GitHub issue that describes this feature and associate it with this PR?

Thank you!

@adk-bot adk-bot added bot triaged services [Component] This issue is related to runtime services, e.g. sessions, memory, artifacts, etc labels Aug 28, 2025
Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a SqliteMemoryService for persistent agent memory, which is a great addition. The implementation is well-structured and includes a comprehensive test suite. My review focuses on a few areas for improvement, primarily concerning performance and robustness. I've identified a significant performance bottleneck in the search functionality and suggest using SQLite's FTS5 extension for a more scalable solution. I've also pointed out opportunities to optimize database initialization and refine exception handling. Overall, this is a solid contribution that will be even better with these changes.

Address reviewer feedback with significant performance and robustness improvements:
**Performance Optimizations:**
- Implement SQLite FTS5 (Full-Text Search) for efficient database-level search
- Replace Python-based keyword matching with native FTS5 MATCH queries
- Add lazy one-time database initialization with asyncio.Lock
- Remove redundant _init_db() calls from every method
**Robustness Improvements:**
- Replace generic Exception handling with specific aiosqlite.Error
- Add proper error handling during database initialization
- Improve query preparation with FTS5-specific escaping
- Add database triggers to keep FTS5 table synchronized
**New Features:**
- FTS5 virtual table for scalable full-text search
- Query preparation function with special character handling
- Enhanced error recovery and graceful degradation
**Testing:**
- Add 8 new test cases covering FTS5 functionality
- Test lazy initialization behavior
- Test FTS5 query preparation and edge cases
- Test comprehensive error handling scenarios
- All 32 tests passing with improved coverage
This addresses the performance bottleneck in search_memory() and ensures
the service scales efficiently with large datasets while maintaining
reliability through proper error handling.
Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a well-designed SqliteMemoryService for persistent agent memory, leveraging aiosqlite and FTS5 for efficient, asynchronous operations. The implementation is robust, with features like lazy initialization and specific exception handling. The accompanying test suite is comprehensive. My review includes a critical fix to prevent potential data loss in the clear_memory method, along with suggestions to improve internationalization support in search queries and adhere to Python's standard coding practices. I've also recommended an additional test case to cover the identified critical issue.

Raman369AI and others added 6 commits September 6, 2025 20:24
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
- Move GITHUB_TOKEN validation from module import time to function call time
- This prevents ValueError during module imports when GITHUB_TOKEN is not yet set
- Update utils.py to use lazy token loading via get_github_token() function
- Apply fix to both adk_pr_triaging_agent and adk_triaging_agent
- Fixes CI/CD issue where triaging bots fail during workflow initialization
Copy link
Author

closes #2976

Copy link
Collaborator

@hangfei hangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the contribution!



class SqliteMemoryService(BaseMemoryService):
"""An async SQLite-based memory service for persistent storage with FTS5 search.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we choose FTS5 search?

Copy link
Author

@Raman369AI Raman369AI Sep 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My use case would need a FTS5 search for string matching , I am exposing the same DB to another agent's tool to filter strings and obtain the required data.

Copy link
Collaborator

hangfei commented Sep 19, 2025

@GWeale @DeanChensj Please review this.

Copy link
Collaborator

hangfei commented Sep 19, 2025

@Raman369AI Could you fix the linter checks and tests?

Raman369AI reacted with thumbs up emoji

Raman369AI and others added 4 commits September 19, 2025 01:26
- Fix line length formatting for test function name
- Ensure compliance with project style guidelines
- All 34 tests still passing after formatting
Addresses reviewer feedback for PR google#2768 
Copy link
Author

Hi I did fix the linter checks ans tests , can you please review the same. > @Raman369AI Could you fix the linter checks and tests?

hangfei and others added 3 commits September 19, 2025 11:59
- Add explicit Dict and Any imports from typing module
- Replace dict return type with Dict[str, Any] for Python 3.9 compatibility
- Ensures compatibility across Python 3.9-3.13 versions
- Fixes CI test failures on Python 3.9
- Add development-only Python 3.9 compatibility testing scripts
- Prevents accidental inclusion of temporary testing files
- Files: check_py39_compatibility.py, test_runtime_compatibility.py,
 test_py39_docker.py, test_simple_py39.py, test_core_py39.py
Copy link
Author

Hi, I have updated the compatibility with 3.9 and other versions of python, can you please review the same.

@Raman369AI Could you fix the linter checks and tests?

- Changed from creating asyncio.Lock() in __init__ to lazy initialization
- Prevents 'RuntimeError: There is no current event loop' in Python 3.9
- Lock is now created only when needed in _ensure_initialized()
- Maintains thread safety and proper initialization semantics
- Fixes all SQLite memory service unit test failures in CI
Tested with Docker across Python 3.9-3.12, all versions pass.
- Added test_sqlite_docker.py and test_real_sqlite_docker.py
- These are development-only testing files for validating the Python 3.9 fix
- Keep the repo clean by excluding temporary testing scripts
hangfei and others added 3 commits September 22, 2025 11:00
...API operation parser
- Updated OperationParser to use 'object' as default return type instead of 'Any'
- This provides better type consistency for OpenAPI operations
- Improves schema validation and type inference
fix: Change default return schema type from 'Any' to 'object' in OpenAPI operation parser
Copy link
Author

Please review the same, all issues have been addressed.

@boyangsvl boyangsvl linked an issue Oct 8, 2025 that may be closed by this pull request
Copy link
Author

@Raman369AI Could you fix the linter checks and tests?

Please review the request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DeanChensj DeanChensj Awaiting requested review from DeanChensj

@hangfei hangfei Awaiting requested review from hangfei

1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

services [Component] This issue is related to runtime services, e.g. sessions, memory, artifacts, etc

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

SQLite-based memory service implementation

AltStyle によって変換されたページ (->オリジナル) /