-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Count line of processor in /proc/cpuinfo on linux, that matches...#4512
Open
sergey-dryabzhinsky wants to merge 7 commits intofacebook:dev from
Open
Count line of processor in /proc/cpuinfo on linux, that matches... #4512sergey-dryabzhinsky wants to merge 7 commits intofacebook:dev from
sergey-dryabzhinsky wants to merge 7 commits intofacebook:dev from
Conversation
cpu cores even in lxc containers
Contributor
Author
Backup compression on proxmox3 server / debian7. Cpu: intel core i7 6850k 3.6 GHz, 12 cores.
W/o patch:
time zstdmt -14 -vv -f vzdump-openvz-20001-2025_10_09-15_57_53.tar
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Note: 6 physical core(s) detected
Compressing with 6 worker threads
--zstd=wlog=22,clog=22,hlog=23,slog=5,mml=5,tlen=32,strat=6
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=6 --content-size
Decompression will require 4194304 B of memory
vzdump-openvz-20001-2025_10_09-15_57_53.tar : 22.04% (1803284480 B => 397499354 B, vzdump-openvz-20001-20
25_10_09-15_57_53.tar.zst)
vzdump-openvz-20001-2025_10_09-15_57_53.tar : Completed in 26.51 sec (cpu load : 489%)
real 0m26.573s
user 2m8.388s
sys 0m1.313s
Contributor
Author
sergey-dryabzhinsky
commented
Oct 18, 2025
With patch:
time zstdmt -14 -vv -f vzdump-openvz-20001-2025_10_09-15_57_53.tar
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Note: 12 physical core(s) detected
Compressing with 12 worker threads
--zstd=wlog=22,clog=22,hlog=23,slog=5,mml=5,tlen=32,strat=6
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=12 --content-size
Decompression will require 4194304 B of memory
vzdump-openvz-20001-2025_10_09-15_57_53.tar : 22.04% (1803284480 B => 397499354 B, vzdump-openvz-20001-2025_10_09-15_57_53.tar.zst)
vzdump-openvz-20001-2025_10_09-15_57_53.tar : Completed in 17.84 sec (cpu load : 916%)
real 0m17.990s
user 2m42.266s
sys 0m1.353s
System load greater but process run faster.
Contributor
Author
Second server: proxmox7, debian11, cpu intel xeon E3-1230 v6 @ 3.50GHz, 8 cores.
W/o patch:
time zstdmt -14 -vv -f vzdump-openvz-60047-2023_08_18-18_01_53.tar
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
Note: 4 physical core(s) detected
--zstd=wlog=22,clog=22,hlog=23,slog=5,mml=5,tlen=32,strat=6
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=4 --content-size
Decompression will require 4194304 B of memory
vzdump-openvz-60047-2023_08_18-18_01_53.tar : 25.32% (580116480 B => 146903865 B, vzdump-openvz-60047-2023_08_18-18_01_53.tar.zst)
vzdump-openvz-60047-2023_08_18-18_01_53.tar : Completed in 12.85 sec (cpu load : 375%)
real 0m12.890s
user 0m47.804s
sys 0m0.433s
Contributor
Author
With patch:
time zstdmt -14 -vv -f vzdump-openvz-60047-2023_08_18-18_01_53.tar
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Note: 8 physical core(s) detected
Compressing with 8 worker threads
--zstd=wlog=22,clog=22,hlog=23,slog=5,mml=5,tlen=32,strat=6
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=8 --content-size
Decompression will require 4194304 B of memory
vzdump-openvz-60047-2023_08_18-18_01_53.tar : 25.01% (580116480 B => 145070412 B, vzdump-openvz-60047-2023_08_18-18_01_53.tar.zst)
vzdump-openvz-60047-2023_08_18-18_01_53.tar : Completed in 9.25 sec (cpu load : 673%)
real 0m9.279s
user 1m1.568s
sys 0m0.703s
Contributor
Author
sergey-dryabzhinsky
commented
Oct 18, 2025
Failed action not related
Contributor
Cyan4973
commented
Oct 18, 2025
If you prefer zstdmt or -T0 to count logical cores as opposed to physical ones, there is already the --auto-threads=logical command.
Contributor
Author
sergey-dryabzhinsky
commented
Oct 19, 2025
@Cyan4973 Thanx, I'll check it
Contributor
Author
sergey-dryabzhinsky
commented
Oct 19, 2025
That works:
zstdmt -vv --auto-threads=logical /dev/zero
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Note: 8 logical core(s) detected
Compressing with 8 worker threads
--zstd=wlog=21,clog=16,hlog=17,slog=1,mml=5,tlen=0,strat=2
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=8 --content-size
zstd: /dev/zero is not a regular file -- ignored
Question: why it is not default behaviour?
Contributor
Cyan4973
commented
Oct 29, 2025
It's a question of choice.
The original choice was: one thread per physical code.
I already explained in another thread the reasoning leading to that choice.
Now, this is not immutable, we could change it if we feel that's a better option.
This is less a question of code than a question of properly documenting why a different default would be better.
Contributor
Author
sergey-dryabzhinsky
commented
Oct 30, 2025
Maybe this behaviour could be changed at compile time to let distributors choose what they want?
Contributor
Cyan4973
commented
Oct 31, 2025
Maybe this behaviour could be changed at compile time to let distributors choose what they want?
A compilation variable would do the trick
Adx new defined compile var
Use new defined var to enable-disable hardcode
Add comments
Fix comment
Contributor
Author
I added compilation "variable" through cflags -DZSTD_PARSE_FULL_CPUINFO , so my code stay disabled by default. I'll add other PR with another compile var later
Contributor
Author
sergey-dryabzhinsky
commented
Nov 6, 2025
Work still in progress.
@sergey-dryabzhinsky
sergey-dryabzhinsky
changed the title
(削除) Count line of processor in /proc/cpuinfo on linux, that matches (削除ここまで)
(追記) [WIP] Count line of processor in /proc/cpuinfo on linux, that matches... (追記ここまで)
Nov 6, 2025
Change var name
Fixed prepocessor check on new var
Contributor
Author
sergey-dryabzhinsky
commented
Nov 6, 2025
All done I think.
@sergey-dryabzhinsky
sergey-dryabzhinsky
changed the title
(削除) [WIP] Count line of processor in /proc/cpuinfo on linux, that matches... (削除ここまで)
(追記) Count line of processor in /proc/cpuinfo on linux, that matches... (追記ここまで)
Nov 6, 2025
Contributor
Author
sergey-dryabzhinsky
commented
Nov 6, 2025
Not shure how to add this to docs.
Contributor
Author
sergey-dryabzhinsky
commented
Nov 7, 2025
Ready for review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
cpu cores even in lxc containers
Address issue #4510