Massive performance improvement...#363

Open

typhoonsimon wants to merge 8 commits into

iquidus:master from

typhoonsimon:master

Open

Massive performance improvement... #363
typhoonsimon wants to merge 8 commits into
iquidus:master from
typhoonsimon:master

Conversation

@typhoonsimon

@typhoonsimon typhoonsimon commented Mar 14, 2020 •

edited by uaktags

Loading

Copy link

Copy Markdown

.. for index page and address view page. Don't forget that these additional mongo indexes must be created to achieve the level of performance of http://lcp.altcoinwarz.com/

db.txes.createIndex({total: 1})
db.txes.createIndex({total: -1})
db.txes.createIndex({blockindex: 1})
db.txes.createIndex({blockindex: -1})
addresstxes
db.addresstxes.createIndex({a_id: 1, blockindex: -1})```

typhoonsimon added 7 commits

March 14, 2020 16:13

@typhoonsimon


 Gigantic performance improvemet

917b9c6

... over large chains. Added indexes and brought back last_txs setting (set to 0 will display the whole chain in the index, but without sacrificing server or mongo performance).

@typhoonsimon


 Rewritten peers sync way

8f9945b

@typhoonsimon


 Hybrid display fix

0a10086

@typhoonsimon


 Hybrid display fix

45c5f53

@typhoonsimon


 Hybrid display fix

ca20343

@typhoonsimon


 Update README.md

5a28894

@typhoonsimon


 Update README.md

263140e

@uaktags uaktags linked an issue

Mar 14, 2020

that may be closed by this pull request

Please help, paging doesn't work, I am getting MAD ! #360

Open

@uaktags uaktags requested a review from TheHolyRoger

March 14, 2020 09:15

@typhoonsimon

typhoonsimon commented Mar 14, 2020

Copy link

Copy Markdown

Author

Ah sorry, that was a debug, I will fix it.

@typhoonsimon


 Add files via upload

4f7f61b

@typhoonsimon

typhoonsimon commented Mar 14, 2020

Copy link

Copy Markdown

Author

OK I have put back the correct port settings, sorry about that.

@typhoonsimon

typhoonsimon commented Mar 14, 2020

Copy link

Copy Markdown

Author

it should be noted that there is a diff between count and countDocuments, and by virtue of mongos documentation this should be avoided.

Do you want me to set it back to countDocuments ?

@uaktags

uaktags commented Mar 14, 2020

Copy link

Copy Markdown

Collaborator

it should be noted that there is a diff between count and countDocuments, and by virtue of mongos documentation this should be avoided.

Do you want me to set it back to countDocuments ?

I'm not entirely sure yet. I need to go through and double check its usage. My fear is improper shutdowns will cause the count to be off, but this is simply the total count of txes for an address. It's not something I deem mission critical compared to say the actual values of transactions/amounts/etc.

Performance wise, count is indeed much faster, but it's also susceptible to inaccuracies.

@typhoonsimon

typhoonsimon commented Mar 15, 2020

Copy link

Copy Markdown

Author

it should be noted that there is a diff between count and countDocuments, and by virtue of mongos documentation this should be avoided.

Do you want me to set it back to countDocuments ?

I'm not entirely sure yet. I need to go through and double check its usage. My fear is improper shutdowns will cause the count to be off, but this is simply the total count of txes for an address. It's not something I deem mission critical compared to say the actual values of transactions/amounts/etc.

Performance wise, count is indeed much faster, but it's also susceptible to inaccuracies.

That's exactly what I thought. It's the same as count without lock on MySQL InnoDB, the count is not accurate but is very fast. In this case, worst that can happen is the the page number display is not accurate. But transactions per address are slow, I guess we could live with that given the cost of the countDocuments().

@uaktags

uaktags commented Mar 15, 2020

Copy link

Copy Markdown

Collaborator

it should be noted that there is a diff between count and countDocuments, and by virtue of mongos documentation this should be avoided.

Do you want me to set it back to countDocuments ?

I'm not entirely sure yet. I need to go through and double check its usage. My fear is improper shutdowns will cause the count to be off, but this is simply the total count of txes for an address. It's not something I deem mission critical compared to say the actual values of transactions/amounts/etc.
Performance wise, count is indeed much faster, but it's also susceptible to inaccuracies.

That's exactly what I thought. It's the same as count without lock on MySQL InnoDB, the count is not accurate but is very fast. In this case, worst that can happen is the the page number display is not accurate. But transactions per address are slow, I guess we could live with that given the cost of the countDocuments().

I agree. I would however counter to provide a settings config so that explorer admins can choose to have accuracy. The performance impact from if/then statements would be negligible.

@buzzkillb buzzkillb mentioned this pull request

Apr 29, 2020

Not loading transactions #367

Closed

@allavett

allavett commented Jun 1, 2020 •

edited

Loading

Copy link

Copy Markdown

Any update on the ETA of this pull request being finalized and merged?

Also, @typhoonsimon where does the need come to index both descending and ascending directions, does it really give the desired result?

.. Don't forget that these additional mongo indexes must be created to achieve the level of performance of http://lcp.altcoinwarz.com/
db.txes.createIndex({total: 1})
db.txes.createIndex({total: -1})
db.txes.createIndex({blockindex: 1})
db.txes.createIndex({blockindex: -1})

@uaktags

uaktags commented Jul 9, 2020

Copy link

Copy Markdown

Collaborator

I'm going to add this to my testing and merges with uaktags:explorer#58. Once testing is complete, I'll push accept and merge this. Hopefully end of week for an eta

uaktags added a commit to uaktags/explorer that referenced this pull request

Jul 10, 2020

@uaktags


 Testing iquidus#363

0915b39

TheHolyRoger

TheHolyRoger requested changes

Jul 16, 2020

View reviewed changes

@TheHolyRoger TheHolyRoger left a comment

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per my comments this needs adjusting and/or duplicating into latest TX and latest BLOCK function calls

views/layout.pug

$.ajax({url: '/ext/summary', success: function(json){

$("#supply").text(parseInt(parseFloat(json.data[0].supply).toFixed(0)).toLocaleString('en'));

$("#difficulty").text(parseFloat(json.data[0].difficulty).toFixed(2));

$("#difficulty").text(json.data[0].difficulty);

@TheHolyRoger TheHolyRoger Jul 16, 2020 •

edited

Loading

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we changing this? If you want this formatted differently then we should add config options for it in a separate PR
Edit: sorry my fault I broke this for hybrid diff

@TheHolyRoger TheHolyRoger Jul 16, 2020

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b7180dc

scripts/peers.js

}, function() {

exit();

// insert all at once after creation

@TheHolyRoger TheHolyRoger Jul 16, 2020 •

edited

Loading

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks good, much better way of doing it

lib/database.js

return cb(txs, count);

}

});

lib.get_blockcount(function(blockcount) {

@TheHolyRoger TheHolyRoger Jul 16, 2020

Copy link

Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@typhoonsimon As per the function's name "get_last_txs_ajax" - this function does NOT show latest blocks - it shows Latest Transactions, we cannot use the blockcount call here.

This does look very efficient though and would work almost as is for an additional function for latest blocks, which we could then set in the config for the user to display latest TX or latest BLOCKS on the front page.

In terms of making this code work for latest TXs what's the performance difference of Tx.countDocuments vs Tx.count here?

tx_count might be a stat we should keep updated via block indexing?

@TheHolyRoger

TheHolyRoger commented Jul 16, 2020 •

edited

Loading

Copy link

Copy Markdown

Collaborator

Merged the index declarations to 'main' here: 3f81b75 - these alone seem to have improved paging performance for me so thanks for that @typhoonsimon

@TheHolyRoger

TheHolyRoger commented Jul 16, 2020

Copy link

Copy Markdown

Collaborator

d611f8f This patch seems to add a mild speed improvement until we resolve this PR but thanks for your help so far @typhoonsimon

@TheHolyRoger

TheHolyRoger commented Jul 16, 2020

Copy link

Copy Markdown

Collaborator

Difficulty display fixed: b7180dc

This was linked to issues Jul 19, 2020

Blocks Page not Loading #391

Open

Add option for Latest Txs vs Latest Blocks on front page #238

Open

Labels

None yet

4 participants

@typhoonsimon @uaktags @allavett @TheHolyRoger

Conversation

@typhoonsimon typhoonsimon commented Mar 14, 2020 • edited by uaktags Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

typhoonsimon commented Mar 14, 2020

Uh oh!

typhoonsimon commented Mar 14, 2020

Uh oh!

typhoonsimon commented Mar 14, 2020

Uh oh!

uaktags commented Mar 14, 2020

Uh oh!

typhoonsimon commented Mar 15, 2020

Uh oh!

uaktags commented Mar 15, 2020

Uh oh!

allavett commented Jun 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uaktags commented Jul 9, 2020

Uh oh!

@TheHolyRoger TheHolyRoger left a comment

Choose a reason for hiding this comment

Uh oh!

@TheHolyRoger TheHolyRoger Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

@TheHolyRoger TheHolyRoger Jul 16, 2020

Choose a reason for hiding this comment

Uh oh!

@TheHolyRoger TheHolyRoger Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

@TheHolyRoger TheHolyRoger Jul 16, 2020

Choose a reason for hiding this comment

Uh oh!

TheHolyRoger commented Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TheHolyRoger commented Jul 16, 2020

Uh oh!

TheHolyRoger commented Jul 16, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

@typhoonsimon typhoonsimon commented Mar 14, 2020 •

edited by uaktags

Loading

allavett commented Jun 1, 2020 •

edited

Loading

@TheHolyRoger TheHolyRoger Jul 16, 2020 •

edited

Loading

@TheHolyRoger TheHolyRoger Jul 16, 2020 •

edited

Loading

TheHolyRoger commented Jul 16, 2020 •

edited

Loading