Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Add table/column versioning info to root directory structure #267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rzats wants to merge 1 commit into questdb:main
base: main
Choose a base branch
Loading
from rzats:rzatserkovnyi/column-versioning
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 53 additions & 19 deletions documentation/concept/root-directory-structure.md
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,31 +94,65 @@ organised as follows:
- If a given column has an [index](/docs/concept/indexes/), then there will also
be an `index_file`, for example `mycolumn.k`

:::info Table and Column Versioning

QuestDB uses a **copy-on-write** mechanism for data updates, which means that when you update existing data, new versions of column files are created rather than modifying existing ones. This ensures data consistency and allows readers to continue accessing the previous version while updates are in progress.

For more details on how versioning works, see the [Updating data](/docs/operations/updating-data/#column-versions) documentation.

:::

The table also stores metadata in `_meta` files:

```filestructure
├── db
│ ├── Table
│ │ │
│ │ ├── Partition 1
│ │ │ ├── _archive
│ │ │ ├── column1.d
│ │ │ ├── column2.d
│ │ │ ├── column2.k
│ │ │ └── ...
│ │ ├── Partition 2
│ │ │ ├── _archive
│ │ │ ├── column1.d
│ │ │ ├── column2.d
│ │ │ ├── column2.k
│ │ │ └── ...
│ │ │
│ │ ├── _meta
│ │ ├── _txn
│ │ └── _cv
│ └── table_1.lock
│ ├── Table
│ │ │
│ │ ├── Partition 1
│ │ │ ├── _archive
│ │ │ ├── column1.d
│ │ │ ├── column2.d
│ │ │ ├── column2.k
│ │ │ └── ...
│ │ ├── Partition 2
│ │ │ ├── _archive
│ │ │ ├── column1.d
│ │ │ ├── column2.d
│ │ │ ├── column2.k
│ │ │ └── ...
│ │ │
│ │ ├── _meta
│ │ ├── _txn
│ │ └── _cv
│ └── table_1.lock
```

:::note Column Versioning in Practice

When data is updated, you may see multiple versions of column files in the directory structure. For example, after updating `column1`, you might see:

```filestructure
├── Partition 1
│ ├── column1.d # Original version
│ ├── column1.d.1 # New version after update
│ ├── column2.d # Unchanged column
│ └── ...
```

The `_cv` (column version) metadata file tracks which version of each column is currently active. Older versions are automatically cleaned up by the vacuum process.

:::

### Metadata files

Each table directory contains several metadata files:

| file | description |
|------|-------------|
| `_meta` | Table metadata including schema information |
| `_txn` | Transaction metadata for consistency |
| `_cv` | Column version metadata - tracks which version of each column is currently active |

If the table is not partitioned, data is stored in a directory called `default`:

```filestructure
Expand Down

AltStyle によって変換されたページ (->オリジナル) /