-
-
Notifications
You must be signed in to change notification settings - Fork 760
ETH/ETC - Some found blocks flagged as Uncle are actually Orphaned and this scenario breaks the payment system
#1600
-
Hi everyone,
I'm in the process of implementing ETC, i have it hashing away, finding blocks and sending payments on ETC testnet aka Mordor. However, i found myself facing a case where a found block is flagged by the software as Uncle but it is actually an Orphaned (Someone else on the network beat me to it). And it totally breaks the payment process:
[2023年01月12日 23:43:27.4207] [I] [PayoutManager] Processing payments for pool etc1
[2023年01月12日 23:43:27.7151] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973152
[2023年01月12日 23:43:27.7151] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973152
[2023年01月12日 23:43:27.7151] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973154
[2023年01月12日 23:43:27.7182] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973154
[2023年01月12日 23:43:27.7182] [I] [etc1] [Ethereum Payout Handler] Found another uncle from block 7973153 in the DB. Continuing search for uncle.
[2023年01月12日 23:43:27.7182] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973160
[2023年01月12日 23:43:27.7182] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973160
[2023年01月12日 23:43:27.7182] [I] [etc1] [Ethereum Payout Handler] Unlocked uncle for block 7973160 at height 7973159 worth 0.125 ETC
[2023年01月12日 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973152
[2023年01月12日 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973152
[2023年01月12日 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973154
[2023年01月12日 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973154
[2023年01月12日 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Found another uncle from block 7973153 in the DB. Continuing search for uncle.
[2023年01月12日 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973160
[2023年01月12日 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973160
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked uncle for block 7973160 at height 7973159 worth 0.125 ETC
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973152
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973152
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973154
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973154
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Found another uncle from block 7973153 in the DB. Continuing search for uncle.
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973160
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973160
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked uncle for block 7973160 at height 7973159 worth 0.125 ETC
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973167 worth 4 ETC
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973173 worth 4 ETC
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973176 worth 4 ETC
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973178 worth 4 ETC
[2023年01月12日 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973181 worth 4 ETC
[2023年01月12日 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973185 worth 4 ETC
[2023年01月12日 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973187 worth 4 ETC
[2023年01月12日 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973192 worth 4 ETC
[2023年01月12日 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973193 worth 4 ETC
[2023年01月12日 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973199 worth 4 ETC
[2023年01月12日 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973202 worth 4 ETC
[2023年01月12日 23:43:27.7416] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973154
[2023年01月12日 23:43:27.7416] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973154
[2023年01月12日 23:43:27.7424] [I] [etc1] [Ethereum Payout Handler] Found another uncle from block 7973153 in the DB. Continuing search for uncle.
[2023年01月12日 23:43:27.7424] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973160
[2023年01月12日 23:43:27.7424] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973160
[2023年01月12日 23:43:27.7424] [I] [etc1] [Ethereum Payout Handler] Unlocked uncle for block 7973160 at height 7973159 worth 0.125 ETC
[2023年01月12日 23:43:27.7424] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973207 worth 4 ETC
[2023年01月12日 23:43:27.7424] [I] [PayoutManager] Processing payments for pool etc1, block 7973159
[2023年01月12日 23:43:27.7560] [I] [PPLNS Payment] Fetching page 0 of shares for pool etc1, block 7973159
[2023年01月12日 23:43:27.7560] [I] [PPLNS Payment] Balance-calculation for pool etc1, block 7973159 completed with accumulated score 0.5 (100%)
[2023年01月12日 23:43:27.7560] [I] [PPLNS Payment] Crediting 0x0b8e5bafb7758872c3e237a172ec04b4206f1a5f with 0.12238 ETC for 429.5M (429496729.6) shares for block 7973159
[2023年01月12日 23:43:27.7586] [I] [PPLNS Payment] Fetching page 0 of discarded shares for pool etc1, block 7973159
[2023年01月12日 23:43:27.7586] [I] [PPLNS Payment] 429.5M (429496729.6) total discarded shares, block 7973159
[2023年01月12日 23:43:27.7586] [I] [PPLNS Payment] 0x0b8e5bafb7758872c3e237a172ec04b4206f1a5f = 429.5M (429496729.6) discarded shares, block 7973159
[2023年01月12日 23:43:27.7586] [I] [PPLNS Payment] Deleting 1 discarded shares before 2023年01月12日T23:27:44.5990000Z
[2023年01月12日 23:43:27.7586] [I] [PPLNS Payment] 429.5M (429496729.6) shares contributed to a total payout of 0.12238 ETC (100.00% of block reward) to 1 addresses
[2023年01月12日 23:43:27.7623] [I] [PayoutManager] Processing payments for pool etc1, block 7973159
[2023年01月12日 23:43:27.7623] [I] [PPLNS Payment] Fetching page 0 of shares for pool etc1, block 7973159
[2023年01月12日 23:43:27.7623] [I] [PPLNS Payment] Balance-calculation for pool etc1, block 7973159 completed with accumulated score 0.5 (100%)
[2023年01月12日 23:43:27.7623] [I] [PPLNS Payment] Crediting 0x0b8e5bafb7758872c3e237a172ec04b4206f1a5f with 0.12238 ETC for 429.5M (429496729.6) shares for block 7973159
[2023年01月12日 23:43:27.7623] [I] [PPLNS Payment] Fetching page 0 of discarded shares for pool etc1, block 7973159
[2023年01月12日 23:43:27.7623] [I] [PPLNS Payment] 429.5M (429496729.6) total discarded shares, block 7973159
[2023年01月12日 23:43:27.7623] [I] [PPLNS Payment] 0x0b8e5bafb7758872c3e237a172ec04b4206f1a5f = 429.5M (429496729.6) discarded shares, block 7973159
[2023年01月12日 23:43:27.7623] [I] [PPLNS Payment] Deleting 1 discarded shares before 2023年01月12日T23:27:57.9027720Z
[2023年01月12日 23:43:27.7623] [I] [PPLNS Payment] 429.5M (429496729.6) shares contributed to a total payout of 0.12238 ETC (100.00% of block reward) to 1 addresses
[2023年01月12日 23:43:27.7775] [E] [PayoutManager] [etc1] Payment processing failed System.InvalidOperationException: This NpgsqlTransaction has completed; it is no longer usable.
at Npgsql.NpgsqlTransaction.CheckReady()
at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlTransaction.Rollback()
at Miningcore.Extensions.ConnectionFactoryExtensions.RunTx(IConnectionFactory factory, Func`3 action, Boolean autoCommit, IsolationLevel isolation) in /home/ceedii/miningcore/src/Miningcore/Extensions/ConnectionFactoryExtensions.cs:line 61
at Miningcore.Payments.PayoutManager.UpdatePoolBalancesAsync(IMiningPool pool, PoolConfig poolConfig, IPayoutHandler handler, IPayoutScheme scheme, CancellationToken ct) in /home/ceedii/miningcore/src/Miningcore/Payments/PayoutManager.cs:line 164
at Miningcore.Payments.PayoutManager.ProcessPoolsAsync(CancellationToken ct) in /home/ceedii/miningcore/src/Miningcore/Payments/PayoutManager.cs:line 104 at Npgsql.NpgsqlTransaction.CheckReady()
at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlTransaction.Rollback()
at Miningcore.Extensions.ConnectionFactoryExtensions.RunTx(IConnectionFactory factory, Func`3 action, Boolean autoCommit, IsolationLevel isolation) in /home/ceedii/miningcore/src/Miningcore/Extensions/ConnectionFactoryExtensions.cs:line 61
at Miningcore.Payments.PayoutManager.UpdatePoolBalancesAsync(IMiningPool pool, PoolConfig poolConfig, IPayoutHandler handler, IPayoutScheme scheme, CancellationToken ct) in /home/ceedii/miningcore/src/Miningcore/Payments/PayoutManager.cs:line 164
at Miningcore.Payments.PayoutManager.ProcessPoolsAsync(CancellationToken ct) in /home/ceedii/miningcore/src/Miningcore/Payments/PayoutManager.cs:line 104
The faulty block in the log is 7973153, it is declared in the database as Uncle but it is actually an Orphaned:
Database
23 | etc1 | 7973153 | 769441 | confirmed | uncle | 1 | NULL | | 0x0b8e5bafb7758872c3e237a172ec04b4206f1a5f | 0.125000000000 | NULL | NULL | 2023年01月12日 23:26:33.701667+00
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
ETC testnet MORDOR block explorer - https://blockscout.com/etc/mordor/block/7973153
As you can see, it's not an Uncle but an Orphaned, someone else beat me to it, my pool wallet address is 0x421Afb2ce225D3A2d3DD6e63Fe57E124B40e20Af, which is not the address which found that block :)
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 13 comments 1 reply
-
I managed to solve half of my issue, which are indeed blocks recorded in my database, which are indeed Uncle but not mined by my pool wallet but someone else at the same time on the network, with following the advice of the dev from that PR #1501 👍🏽
However, the error NpgsqlTransaction has completed; it is no longer usable. still persists, i even tried to add the timeout in the config.json for PostreSQL.
I have a suspicion that it is a RAM space issue, since i am running everything on a laptop with only 8GB shared with the APU. When everything run, i have barely 1GB left.
I will continue to test and report my findings.
Beta Was this translation helpful? Give feedback.
All reactions
-
After further investigation and testing, i can confirm that the suggestion in PR #1501 from @jon4hz to increase BlockSearchOffset works perfectly and seems to cover all the inherent cases withUncle, including my issue with Uncle recorded in the database but mined by another pool wallet on the network 👍 Only one question remains on my side, what if the two uncles are very very far away from each other, in my case on the ETC Mordor testnet, i had to increase BlockSearchOffset up to 3200 🔥 🚀
There is one last case scenario which is very annoying on the ETC Mordor testnet, some Uncle blocks are duplicated in the blockchain:
Screenshot_2023年01月14日_02-22-19
And they are also logically inserted in the database:
Screenshot_2023年01月14日_02-22-00
In this case scenario, the payout will be totally broken: This NpgsqlTransaction has completed; it is no longer usable. And it will not resume as long as a manual deletion of the duplicated block won't be performed in the database.
Lastly, RAM space left is not the issue at all here, i was able to run the pool and sending payouts with less than 1Go RAM left.
Beta Was this translation helpful? Give feedback.
All reactions
-
Your last issue was probably caused because a block mined by your pool included an orphan, which was also mined by you. At least this is what it looks like to me from what I see in the screenshot of the explorer.
As it seems this use case is not supported at the moment.
I assume miningcore just detects the block and tries to insert it a second time, when looking for that uncle. This violates the constraint CONSTRAINT BLOCKS_POOL_HEIGHT UNIQUE (poolid, blockheight, type) DEFERRABLE INITIALLY DEFERRED leading to the npgsql error.
You probably have to tune the ClassifyBlocksAsync method and add a check to look in the db for existing blocks with the same height. Something similar to what I did in #1501 but for blocks of type Block and not just Uncle.
Beta Was this translation helpful? Give feedback.
All reactions
-
Thanks for the reply @jon4hz, i know you are a busy man :)
According to the ETC Mordor testnet explorer, both blocks has been actually mined by my pool wallet :/
Here the link to my pool wallet https://blockscout.com/etc/mordor/address/0x421Afb2ce225D3A2d3DD6e63Fe57E124B40e20Af/validations#address-tabs, just search for block 7980798, they have the same block height but NOT the same block hash.
The first one, declared as type block
Screenshot_2023年01月15日_00-55-27
The second one, declared as type uncle
Screenshot_2023年01月15日_00-55-51
According to the ETC Mordor testnet block explorer, they both have been found by my pool wallet 🤯
I got that answer from a dev in the ETC discord group:
Screenshot_2023年01月15日_02-30-45
Beta Was this translation helpful? Give feedback.
All reactions
-
It might be a rare case but as you can see, it is possible that a block mined by you includes an uncle block which was also mined by you.
Miningcore just doesn't seem to handle that scenario correctly.
Beta Was this translation helpful? Give feedback.
All reactions
-
It might be a rare case but as you can see, it is possible that a block mined by you includes an uncle block which was also mined by you.
Miningcore just doesn't seem to handle that scenario correctly.
Yes, definitely, but your method GetBlockByHeightAsync goes against the concept of Uncle BlockChain and even the database design:
public async Task<Block> GetBlockByHeightAsync(IDbConnection con, string poolId, long height)
{
const string query = @"SELECT * FROM blocks WHERE poolid = @poolId AND blockheight = @height";
var entity = await con.QuerySingleOrDefaultAsync<Entities.Block>(new CommandDefinition(query, new { poolId, height }));
return entity == null ? null : mapper.Map<Block>(entity);
}
You decided that a block is unique based on the poolId and the blockheight. It's not not. it's poolId, blockheight and type. My bug and this bug #1080 (comment) are related, we trigger an error through your method since there is two duplicated blockHeight in our database . Your method should have been something like that, especially since from what i see, you are using it only for finding uncle:
public async Task<Block> GetBlockByPoolHeightAndTypeAsync(IDbConnection con, string poolId, long height, string type)
{
const string query = @"SELECT * FROM blocks WHERE poolid = @poolId AND blockheight = @height AND type = @type";
return (await con.QueryAsync<Entities.Block>(query, new
{
poolId,
height,
type
}))
.Select(mapper.Map<Block>)
.FirstOrDefault();
}
Beta Was this translation helpful? Give feedback.
All reactions
-
Yes, definitely, but your method GetBlockByHeightAsync goes against the concept of Uncle BlockChain and even the database design
It does not. My PR simply aimed to fix another bug for which it wasn't necessary to check for a specific block type in the db request.
Just changing that method most likely wont fix your issue, however it might be necessary to cover your use-case.
Beta Was this translation helpful? Give feedback.
All reactions
-
Yes, definitely, but your method GetBlockByHeightAsync goes against the concept of Uncle BlockChain and even the database design
It does not. My PR simply aimed to fix another bug for which it wasn't necessary to check for a specific block type in the db request.
Just changing that method most likely wont fix your issue, however it might be necessary to cover your use-case.
I should have been more specific apparently because i thought you aware of a choice made in the function GetBlockByHeightAsync but let's analyze it. It uses a C# function called QuerySingleOrDefaultAsync. A quick search warns about something very important and non-negligible:
Screenshot_2023年01月17日_03-16-47
Anyway, i got it fixed and it does not bother me anymore and the payment system can resume with the mitigation i made before the uncle process.
Duplicate blockHeight can be processed properly.
Screenshot_2023年01月16日_15-26-04
Screenshot_2023年01月16日_15-44-50
There is still an inherent issue with database timeout which happens when there is a lot of pending shares with a lot of Uncle in between but i don't think there is nothing which can really be done about it. But from i what i see so far, the blocks are unlocked properly and the miner wallet balance is also updated properly.
Beta Was this translation helpful? Give feedback.
All reactions
-
I see, I wasn't aware that QuerySingleOrDefaultAsync throws an exception if there are multiple results.
So was specifying the blockType in the query all it took to fix the bug?
Beta Was this translation helpful? Give feedback.
All reactions
-
Nevermind, saw the commits in #1514, amazing work :D
Beta Was this translation helpful? Give feedback.
All reactions
-
I see, I wasn't aware that
QuerySingleOrDefaultAsyncthrows an exception if there are multiple results.So was specifying the
blockTypein the query all it took to fix the bug?
Hey no worry, i understood you were not aware, i actually found out about that SQL function kinda luckily when i was looking to make my patch. I thought you knew since you seem to be a real MVP when it comes to miningcore lol
And thanks for the kind words! Have a great weekend ahead :)
Beta Was this translation helpful? Give feedback.
All reactions
-
Please consider using the repository's discussion feature instead of issues folks 😄
Beta Was this translation helpful? Give feedback.
All reactions
-
did u fix the problem @blackmennewstyle i just now having this problem when mining on other coins using ethash algo, my pool paying reward when someone else found the block in the network
Beta Was this translation helpful? Give feedback.
All reactions
-
You will have to be more specific about which coin(s) you are talking about because that discussion was related to ETH, ETC and also UBQ. That issue regarding ETH, ETC and UBQ has been fixed on the dev branch with even an ultimate evolution: #1608
For any other fork of ETH, you will have to dig in it yourself sadly. The notion of uncles can be drastically different on other projects.
Beta Was this translation helpful? Give feedback.