Skip to content

Implement a database abstraction layer that will enable the use of different DB backends. - #282

Open

Davidyz wants to merge 96 commits into

main from

Open

Implement a database abstraction layer that will enable the use of different DB backends. #282
Davidyz wants to merge 96 commits into
main from
feat/db_layer

Commits

Commits on Sep 22, 2025

WIP(cli): database abstraction layer
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
cdba77c
WIP(cli): Add wip chromadb connector
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
dac5f75
feat(db): Improve database abstraction and vectorisation process
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
8bda33f
feat(db): Implement delete and drop methods for database connectors
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
6f42c52
refactor(cli): `drop` now use the DB adapter layer.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
9028981
fix(cli): default db_url for chroma0
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
416d7c8
fix(cli): minor fixes.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
8cb0e2d
feat(cli): implement database builder with lazy import
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
1f457f7
refactor(cli): `ls` in CLI mode now use the DB adapter layer.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
40b6d97
docs about database connectors.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
129ebd4
feat(cli): support excluding files in queries.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
fdc6e27
fix query filter
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
984cf15
feat(db): Implement config update and replace methods for database connectors
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
b082fa6
remove some parameters.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
9494a76
update `chroma0` connector to adopt the new API definition.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
6263c1b
fix(cli): fix default value of `excluded_files`
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
fd3dd3c
fix(cli): improve assertions for `metadatas`, `documents` and `ids`
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
79e540e
refactor(cli): Use DB adapter layer for `ls` and `rm` commands
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
1bea28a
docs(db): Improve docstrings for database connectors
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
4dd184e
docs(db): Add documentation for database connectors
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
b5e3d79
feat(db): Implement orphan removal functionality in the database connector
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
a34d84e
feat(db): Refactor database connectors to use embeddings and improve listing
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
5f8b65f
refactor(db): Implement `get_chunks` method in the database connectors
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
b578328
wip(db): port `vectorise` command to the new db layer
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
a5546ee
fix(db): Refactor `list_collection_content` to accept keyword arguments
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
46b52c0
feat(cli): Refactor vectorise command to use DB adapter layer
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
29f53d3
feat(cli): Refactor `update` command to use DB adapter layer
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
35db49f
refactor(cli): Refactor query command to use DB adapter layer
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
dcf8309
feat(db): Enforce Chroma v0 client version
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
3f7b591
feat(deps): Update chromadb dependency to latest and add `chromadb==0.6.3` optional dependency
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
beb1d8b
fix(cli): Fix query command by removing deprecated code and chunking queries
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
c3ec51f
refactor(cli): Refactor mcp_main to use DB adapter layer
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
92cdc9d
refactor(cli): Refactor query command to use `preprocess_query_keywords` function
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
265a213
fix(cli): Remove all references to chromadb-related APIs.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
5e3a0cc
fix(chroma0): fix import origin for `get_uuid`
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
80a8893
refactor(lsp): Refactor LSP server to use DB adapter layer
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
7facf78
refactor(cli): Refactor `clean` command to use DB adapter layer
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
7281c45
refactor(cli)!: remove `common.py`
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
236d57f
docs(database): Clarify terminology in database README
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
f064fa3
feat(chroma0): add graceful shutdown for bundled chroma server
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
07d8c53
test(chroma0): Add unittests.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
5fdb3d1
remove unnecessary test
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
aed9e6d
tests(mcp): fix tests.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
bce3e28
fix(mcp): Improve error handling and test coverage
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
a335964
fix(lsp): use correct file lists
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
8c29871
tests(chroma0): make sure database is mocked.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
3478ac7
fix(lsp): remove collection-related code and improved test coverage.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
61628eb
tests(cli): Improve test coverage for `ls` subcommand and handle `CollectionNotFoundError`
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
3949592
tests(cli): refactor tests and coverage for `files rm`
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
301525b
tests(cli): Refactor query subcommand and add tests
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
341fc24
fix(lsp): Validate include parameters in query command
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
03432d4
tests(cli): Refactor drop subcommand and add exception handling
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
7b9a633
feat(cli): Improve test coverage for `vectorise`
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
333d663
tests(cli): Improve test coverage and refactor update subcommand
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
c46118e
tests(cli): Refactor tests and improve coverage for `main`
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
b6d23d5
tests: Refactor database types and improve test coverage
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
e3518bf
tests(cli): Refactor config and cleanup path handling
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
a6a2d72
chore(cli): mark some code as nocover
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
c7e27d6
pin to chroma 0.6.3 for now.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
2ef9349
fix: use `os.path.samefile` for accurate directory comparison
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
7edec0c
fix error message
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
108b195
docs(database): Document the database connector API
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
c15051f
tests(chroma0): Add more tests to chroma0
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
778d84e
tests(cli): Improve test coverage for SpecResolver
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
b05701b
tests: Add tests for database types
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
e64d297
tests(chroma0): improve test coverages.
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
5894d7a
docs(database): update database configuration documentation
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
734e330
Auto generate docs
Davidyz Davidyz
committed
3abc848
docs(database): Document helper methods and error handling in the database API
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
3c4f759
docs: Document database configuration and connector development
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
35c554f
Auto generate docs
Davidyz Davidyz
committed
ff775cb
docs: clarify default chroma version
Zhe YuDavidyz
Zhe Yu
authored and
Davidyz
committed
4aa138d
Auto generate docs
Davidyz Davidyz
committed
424ccb4

Commits on Sep 25, 2025

feat(cli): Report skipped files in vectorise stats
author
Zhe Yu
committed
1df845e

Commits on Oct 3, 2025

build(cli): add `chroma0` dep group.
author
Zhe Yu
committed
34421db
Auto generate docs
Davidyzgithub-actions[bot]
Davidyz
authored and
github-actions[bot]
committed
7baa033
chore(cli): Use chroma0 for CI in test workflow
author
Zhe Yu
committed
11aecc4
chore(cli): Document how to install extra dependencies
author
Zhe Yu
committed
4f12fd5
docs(cli): reflect packaging changes.
author
Zhe Yu
committed
dc44ba9
Auto generate docs
Davidyzgithub-actions[bot]
Davidyz
authored and
github-actions[bot]
committed
d064d99
tests(db): Add database connector initialization test
author
Zhe Yu
committed
dad1eb9

Commits on Oct 4, 2025

tests(db): skip tests when dependency's not met
author
Zhe Yu
committed
78495a9
chore: extra coverage args via env var.
author
Zhe Yu
committed
1684d14
feat(db): Check ChromaDB version on startup
author
Zhe Yu
committed
5c13bea

Commits on Oct 5, 2025

refactor(cli): remove obsolete opts
author
Zhe Yu
committed
ff1c3da
fix(cli): Fixes db_url config retrieval
author
Zhe Yu
committed
d84fa81
refactor(chroma0): extract some stuff that can be reused by chromadb (1.x)
author
Zhe Yu
committed
db10d56
fix(db): Upgrade chromadb version check and include enums
author
Zhe Yu
committed
6a20a0e
feat(chroma): a WIP chromadb connector for chroma 1.x
author
Zhe Yu
committed
5ea7d74

Commits on Oct 6, 2025

Commits on Oct 8, 2025

feat(db): Add inter-process and inter-thread locks for ChromaDB connector
Davidyz Davidyz
committed
29cb6aa

Commits on Oct 10, 2025

Commits on Nov 13, 2025

Merge branch 'main' into feat/db_layer
Davidyz Davidyz
committed
799e1fe