Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

feat: 2pc logic for resharding#1088

Open
yogesh1801 wants to merge 2 commits into
pgdogdev:main from
yogesh1801:feat-2pc-resharding
Open

feat: 2pc logic for resharding #1088
yogesh1801 wants to merge 2 commits into
pgdogdev:main from
yogesh1801:feat-2pc-resharding

Conversation

@yogesh1801

@yogesh1801 yogesh1801 commented Jun 18, 2026
edited by levkk
Loading

Copy link
Copy Markdown

Use two-phase commit for COPY commands when resharding. This allows for a clean rollback on error.

CLAassistant commented Jun 18, 2026
edited
Loading

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@levkk levkk requested a review from meskill June 18, 2026 14:29

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 12.19512% with 36 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...src/backend/replication/logical/subscriber/copy.rs 10.25% 35 Missing ⚠️
...src/frontend/client/query_engine/two_pc/manager.rs 50.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

meskill commented Jun 18, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Could you please uncomment the tests in the integration/copy_data/run.sh and make sure you have 2pc enabled for them? We had issues with these tests and I'd like to check if 2pc fixes them

Copy link
Copy Markdown
Author

@meskill these tests pass for me even if two_pc flag is disabled

meskill commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

@meskill these tests pass for me even if two_pc flag is disabled

yeah, they are just flaky sometimes on ci. don't worry about it, I'll be just checking ci status

levkk commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

@meskill are we ok to merge or do we need anything else?

Comment on lines +288 to +289
TwoPcPhase::Phase1 => format!("PREPARE TRANSACTION '{txn}_{shard}'"),
TwoPcPhase::Phase2 => format!("COMMIT PREPARED '{txn}_{shard}'"),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should work rn since the same logic is used for tx name as in the binding

let shard_name = format!("{}_{}", name, shard);

but it could diverge if something changed on one side. It's especially important since the rollback is used exactly from the binding.

Can we share some of the logic of the query or tx name creation between here and binding.rs? that'll prevent possible issues

Comment thread pgdog/src/backend/replication/logical/subscriber/copy.rs Outdated
let query = match phase {
TwoPcPhase::Phase1 => format!("PREPARE TRANSACTION '{txn}_{shard}'"),
TwoPcPhase::Phase2 => format!("COMMIT PREPARED '{txn}_{shard}'"),
_ => unreachable!(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There could be some issues due to the nature of cli calls i.e. when we run pgdog data-sync instead admin command on the running instance:

  • the Manager::monitor should be running anyway, but we don't wait for it's shutdown in the cli path. That means if we fail the prepared transactions could be left on some shards and they won't be cleaned up.
  • wal is also is not enabled in cli path, but on running instance it'll be enabled. We can get different behavior depending how exactly the copy command was run.

We probably should unify this and make sure we wait for shutdown and enable wal for the cli path. But that also means we should think how the old transactions should be recovered..

@yogesh1801 yogesh1801 Jun 21, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should i take up this as another PR, after thinking about crash recovery

}

for result in join_all(futures).await {
result?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if it's really a problem, but the commit/rollback for prepared transactions happens inside Manager::monitor handler and it's async to this code meaning we may actually start new attempt before the manager logic is fired and the cleanup could happen in the middle of new copy attempt.

@levkk wdyt? should we care about it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@meskill meskill meskill requested changes

@levkk levkk levkk approved these changes

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AltStyle によって変換されたページ (->オリジナル) /