What is the best way to obtain the 7-piece tablebases?
Discussion of anything and everything relating to chess playing software and machines.
Moderator: Ras
Re: What is the best way to obtain the 7-piece tablebases?
Thank you, that looks useful for Windows users. I wonder if there is something like that for Mac.
If you are not cash strapped, buy VMware's Fusion https://www.vmware.com/products/fusion.html and install Windoze in a VM. There may be some freeware Windows emulator too, but I don't do Mac or any Apple products, so I wouldn't be much help beyond the Fusion suggestion.
- Dann Corbit
- Posts: 12803
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
- Contact:
Re: What is the best way to obtain the 7-piece tablebases?
Post by Dann Corbit »
I guess that people will not seriously start on 8 man tablebase files for five more years.
Then, they will require about the same effort as the 7 man files did.
Then, they will require about the same effort as the 7 man files did.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
- Ajedrecista
- Posts: 2146
- Joined: Wed Jul 13, 2011 9:04 pm
- Location: Madrid, Spain.
- Contact:
Re: What is the best way to obtain the 7-piece tablebases?
Post by Ajedrecista »
Hello Dann:
[引用]
NULP (Number of Unique Legal Positions in chess endgames)
The number of unique legal positions (NULP) in 8-man EGTB should be circa 3.8176e+16, which is around 90.07 times the number of unique legal positions in 7-man EGTB (circa 4.2384e+14). The breakdown of NULP by endgame type is here:
https://kirill-kryukov.com/chess/nulp/r ... ndgame.txt
I downloaded an Excel with interesting EGTB statistics through 3-man to 8-man long ago, including difference in material (Q = 9, R = 5, B = 3, N = 3, P = 1), pawnful or pawnless, number of pieces by side (4 vs 4, 5 vs 3, ...). I downloaded it from CCRL Endgame Tablebase Forum and it is still downloadable:
analysis of 1 specific 6-7-8 piece position
(I found the original source thanks to an old post by myself!).
I combined some data of 8-man EGTB: NULP by endgame type, difference in material, pawnful or pawnless and number of pieces by side. Summary:
Endgames can be ordered in many ways: by size, by difference in material and then by size...
Many of those 2520 endgames are very umbalanced. I get the following results applying filters in Excel:
Similar analysis can be done using endgames plus side to move instead of only endgames:
https://kirill-kryukov.com/chess/nulp/r ... by-esm.txt
Regards from Spain.
Ajedrecista.
[引用]
Dann Corbit wrote: ↑ Sat Jun 20, 2020 10:03 am
I guess that people will not seriously start on 8 man tablebase files for five more years.
Then, they will require about the same effort as the 7 man files did.
I agree with you. According to this site:Then, they will require about the same effort as the 7 man files did.
NULP (Number of Unique Legal Positions in chess endgames)
The number of unique legal positions (NULP) in 8-man EGTB should be circa 3.8176e+16, which is around 90.07 times the number of unique legal positions in 7-man EGTB (circa 4.2384e+14). The breakdown of NULP by endgame type is here:
https://kirill-kryukov.com/chess/nulp/r ... ndgame.txt
I downloaded an Excel with interesting EGTB statistics through 3-man to 8-man long ago, including difference in material (Q = 9, R = 5, B = 3, N = 3, P = 1), pawnful or pawnless, number of pieces by side (4 vs 4, 5 vs 3, ...). I downloaded it from CCRL Endgame Tablebase Forum and it is still downloadable:
analysis of 1 specific 6-7-8 piece position
(I found the original source thanks to an old post by myself!).
I combined some data of 8-man EGTB: NULP by endgame type, difference in material, pawnful or pawnless and number of pieces by side. Summary:
Code: Select all
Endgames Positions Percentage
===================================================================
4 vs 4 (no P) 210 1,273,144,250,260,640 3.33 %
4 vs 4 (P) 420 10,459,782,892,146,840 27.40 %
5 vs 3 (no P) 350 1,944,492,729,833,666 5.09 %
5 vs 3 (P) 700 15,851,450,424,099,256 41.52 %
6 vs 2 (no P) 224 819,877,739,413,186 2.15 %
6 vs 2 (P) 406 6,535,348,354,167,395 17.12 %
7 vs 1 (no P) 84 148,530,422,041,225 0.39 %
7 vs 1 (P) 126 1,143,680,065,786,037 3.00 %
-------------------------------------------------------------------
TOTAL 2520 38,176,306,877,748,245 100.00 %Many of those 2520 endgames are very umbalanced. I get the following results applying filters in Excel:
Code: Select all
Side 1: UPPERCASE.
Side 2: lowercase.
#x: number of x.
Difference = ABS[9*(#Q - #q) + 5*(#R - #r) + 3*(#B - #b) + 3*(#N - #n) + 1*(#P - #p)]
Difference Endgames
=======================
0 157
2 265
4 256
6 260
8 229
10 212
12 202
14 170
16 145
18 136
20 104
22 85
24 75
26 55
28 41
30 38
32 24
34 18
36 16
38 10
40 6
42 7
44 3
46 2
48 2
50 1
52 0
54 1
-----------------------
TOTAL 2520https://kirill-kryukov.com/chess/nulp/r ... by-esm.txt
Regards from Spain.
Ajedrecista.
Re: What is the best way to obtain the 7-piece tablebases?
[引用]
Lets say you need 100 times as much ram and hard drive for each extra piece so 9 pieces needs 10 pb Ram and 180 Pb hard drive.
This seems to be within the capability of Summit supercomputer
https://lichess.org/blog/W3WeMyQAACQAdf ... e-complete
7-piece Syzygy tablebases are complete
..
Bojun Guo (also one of the top contributors of computing power for Stockfish testing) started generating the first tables in May, expertly managing two machines, one of them with 384 threads (8x Intel Xeon Platinum 8168) and more than 1 TB of RAM (later distributed differently).
..
The number of unique legal 7-piece positions is 423,836,835,667,331. Syzygy tablebases store all their information in 18.4 TB, so at around 0.35 bits per position. This is much more compact than the proprietary 100 TB Lomonosov tablebases.
https://www.tomshardware.com/news/us-su ... 37367.html
In total, the Summit supercomputer has at is disposal 9,216 Power9 processors, over 10PB of DDR4, HBM, and non-volatile memory, 250PB of storage, and 27,648 Tesla V100 accelerators — which was certainly not a bad way to spend 200ドル million dollars.
yurikvelo wrote: ↑ Fri Jun 19, 2020 11:42 pm
This article from the Lomonosov tablebase page estimates that the size of an 8-man tablebase will be ~100x larger than a 7-man table base and will take a minimum of 10,000 TB of disk space and 50 TB of RAM, limiting its completion to a top-10 super computer
Syzygy 7-piece was about 1 Tb of Ram and 18Tb hard diskThis article from the Lomonosov tablebase page estimates that the size of an 8-man tablebase will be ~100x larger than a 7-man table base and will take a minimum of 10,000 TB of disk space and 50 TB of RAM, limiting its completion to a top-10 super computer
Lets say you need 100 times as much ram and hard drive for each extra piece so 9 pieces needs 10 pb Ram and 180 Pb hard drive.
This seems to be within the capability of Summit supercomputer
https://lichess.org/blog/W3WeMyQAACQAdf ... e-complete
7-piece Syzygy tablebases are complete
..
Bojun Guo (also one of the top contributors of computing power for Stockfish testing) started generating the first tables in May, expertly managing two machines, one of them with 384 threads (8x Intel Xeon Platinum 8168) and more than 1 TB of RAM (later distributed differently).
..
The number of unique legal 7-piece positions is 423,836,835,667,331. Syzygy tablebases store all their information in 18.4 TB, so at around 0.35 bits per position. This is much more compact than the proprietary 100 TB Lomonosov tablebases.
https://www.tomshardware.com/news/us-su ... 37367.html
In total, the Summit supercomputer has at is disposal 9,216 Power9 processors, over 10PB of DDR4, HBM, and non-volatile memory, 250PB of storage, and 27,648 Tesla V100 accelerators — which was certainly not a bad way to spend 200ドル million dollars.
- Dann Corbit
- Posts: 12803
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
- Contact:
Re: What is the best way to obtain the 7-piece tablebases?
Post by Dann Corbit »
[引用]
duncan wrote: ↑ Sun Jun 21, 2020 2:25 am
The number of unique legal 7-piece positions is 423,836,835,667,331. Syzygy tablebases store all their information in 18.4 TB, so at around 0.35 bits per position. This is much more compact than the proprietary 100 TB Lomonosov tablebases.
But I guess that if the Syzygy tablebase format were changed to DTM, much of the space advantage would vanish.
The number of unique legal 7-piece positions is 423,836,835,667,331. Syzygy tablebases store all their information in 18.4 TB, so at around 0.35 bits per position. This is much more compact than the proprietary 100 TB Lomonosov tablebases.
My suspicion is that while the Russian university allowed their supercomputer to be used for TB computation, the Summit supercomputer will be booked with jobs until it dies.
https://www.tomshardware.com/news/us-su ... 37367.html
In total, the Summit supercomputer has at is disposal 9,216 Power9 processors, over 10PB of DDR4, HBM, and non-volatile memory, 250PB of storage, and 27,648 Tesla V100 accelerators — which was certainly not a bad way to spend 200ドル million dollars.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
- Nordlandia
- Posts: 2822
- Joined: Fri Sep 25, 2015 9:38 pm
- Location: Sortland, Norway
Re: What is the best way to obtain the 7-piece tablebases?
Post by Nordlandia »
Is it possible to schedule the download in chronological order. Chrome only accept one download at a time. My point is that once the download has finished for one particular table, the next start immediately, without any time wasted for nothing.
I don't know if it possible to download one from Sesse website and one from Lichess mirror concurrently ?
I don't know if it possible to download one from Sesse website and one from Lichess mirror concurrently ?
Re: What is the best way to obtain the 7-piece tablebases?
You have free online access to the 7-piece TB here https://syzygy-tables.info/ and even better for direct use in chess programs for analyzing purposes with the Fritz 17 or ChessBase 14 / 15 software.
Re: What is the best way to obtain the 7-piece tablebases?
[引用]
The tool I posted does download one after another, with the only delay being the time to hash the prior file. The cumulative hashing time is probably less than 1/60th of the total time to acquire the files. You don't want to use them without checking the file integrity.
You could cut the list of files into two text files and launch the tool twice, once for each site. So the short answer is: Yes, limited only by your creative thinking.
Nordlandia wrote: ↑ Sun Jun 21, 2020 10:17 am
Is it possible to schedule the download in chronological order. Chrome only accept one download at a time. My point is that once the download has finished for one particular table, the next start immediately, without any time wasted for nothing.
I don't know if it possible to download one from Sesse website and one from Lichess mirror concurrently ?
The only benefit of downloading from two different sites simultaneously would be in the case that your internet connection is faster than the path to the fastest repository. Otherwise you need two internet pipes, one for each download (i.e. home and office computers both downloading). If you have a fast internet connection >50Mb/s, pulling from multiple sites would be faster.I don't know if it possible to download one from Sesse website and one from Lichess mirror concurrently ?
The tool I posted does download one after another, with the only delay being the time to hash the prior file. The cumulative hashing time is probably less than 1/60th of the total time to acquire the files. You don't want to use them without checking the file integrity.
You could cut the list of files into two text files and launch the tool twice, once for each site. So the short answer is: Yes, limited only by your creative thinking.
Re: What is the best way to obtain the 7-piece tablebases?
[引用]
Around 2,000 TB of disk space would probably be enough to store the full 8-piece set (WDL+DTZ).
duncan wrote: ↑ Sun Jun 21, 2020 2:25 am
[引用]
Lets say you need 100 times as much ram and hard drive for each extra piece so 9 pieces needs 10 pb Ram and 180 Pb hard drive.
64 TB of RAM (shared by all cpu cores) would be enough for the current generator (with some minor tweaking).yurikvelo wrote: ↑ Fri Jun 19, 2020 11:42 pm
This article from the Lomonosov tablebase page estimates that the size of an 8-man tablebase will be ~100x larger than a 7-man table base and will take a minimum of 10,000 TB of disk space and 50 TB of RAM, limiting its completion to a top-10 super computer
Syzygy 7-piece was about 1 Tb of Ram and 18Tb hard diskThis article from the Lomonosov tablebase page estimates that the size of an 8-man tablebase will be ~100x larger than a 7-man table base and will take a minimum of 10,000 TB of disk space and 50 TB of RAM, limiting its completion to a top-10 super computer
Lets say you need 100 times as much ram and hard drive for each extra piece so 9 pieces needs 10 pb Ram and 180 Pb hard drive.
Around 2,000 TB of disk space would probably be enough to store the full 8-piece set (WDL+DTZ).
Supercomputers typically are clusters of computing nodes, with each computing node having direct access to only a small portion of total RAM memory. The current generator cannot work with that. It was designed to generate 6-piece TBs on a single machine with at least 16GB of RAM and preferably a bit more (which would have been a ridiculous requirement when Nalimov wrote his generator, but was quite reasonable in 2012). It would need 64 TB of RAM shared by all cores to generate 8-piece TBs. To tackle 8 piece, it is probably best to first write a new generator designed for the hardware that one may hope to be available 5-10 years from now.This seems to be within the capability of Summit supercomputer
Re: What is the best way to obtain the 7-piece tablebases?
[引用]
HPE Integrity MC990 X Server has a maximum of 48 Tb. If this is upgraded in the future to 64 Tb, would it run your software for 8 pieces ?
https://buy.hpe.com/us/en/servers/missi ... 1008798952
Do you need a high performance server with greater in-memory computing capacity, reliability, and cost-efficiency? The HPE Integrity MC990 X Server delivers in-memory computing performance for Linux®-based applications at unparalleled scale with mission-critical reliability and modular flexibility. An advanced symmetric multiprocessing (SMP) system designed for data-intensive workloads, the HPE MC990 X Server features the enterprise-class Intel® Xeon® E7-8800/4800 v4 processors and robust reliability, availability, and serviceability. The 5U modular chassis contains 4 sockets with up to 192 threads. By adding chassis and leveraging high bandwidth, low latency NUMAlink technology, the HPE MC990 X Server can scale as a single system from 4 to 32 sockets and from 1 to 48 TB of cache-coherent shared memory.
syzygy wrote: ↑ Sun Jun 21, 2020 10:44 pm
[引用]
Around 2,000 TB of disk space would probably be enough to store the full 8-piece set (WDL+DTZ).
Thanks for this.duncan wrote: ↑ Sun Jun 21, 2020 2:25 am
64 TB of RAM (shared by all cpu cores) would be enough for the current generator (with some minor tweaking).yurikvelo wrote: ↑ Fri Jun 19, 2020 11:42 pmSyzygy 7-piece was about 1 Tb of Ram and 18Tb hard disk
This article from the Lomonosov tablebase page estimates that the size of an 8-man tablebase will be ~100x larger than a 7-man table base and will take a minimum of 10,000 TB of disk space and 50 TB of RAM, limiting its completion to a top-10 super computer
Lets say you need 100 times as much ram and hard drive for each extra piece so 9 pieces needs 10 pb Ram and 180 Pb hard drive.
Around 2,000 TB of disk space would probably be enough to store the full 8-piece set (WDL+DTZ).
Supercomputers typically are clusters of computing nodes, with each computing node having direct access to only a small portion of total RAM memory. The current generator cannot work with that. It was designed to generate 6-piece TBs on a single machine with at least 16GB of RAM and preferably a bit more (which would have been a ridiculous requirement when Nalimov wrote his generator, but was quite reasonable in 2012). It would need 64 TB of RAM shared by all cores to generate 8-piece TBs. To tackle 8 piece, it is probably best to first write a new generator designed for the hardware that one may hope to be available 5-10 years from now.This seems to be within the capability of Summit supercomputer
HPE Integrity MC990 X Server has a maximum of 48 Tb. If this is upgraded in the future to 64 Tb, would it run your software for 8 pieces ?
https://buy.hpe.com/us/en/servers/missi ... 1008798952
Do you need a high performance server with greater in-memory computing capacity, reliability, and cost-efficiency? The HPE Integrity MC990 X Server delivers in-memory computing performance for Linux®-based applications at unparalleled scale with mission-critical reliability and modular flexibility. An advanced symmetric multiprocessing (SMP) system designed for data-intensive workloads, the HPE MC990 X Server features the enterprise-class Intel® Xeon® E7-8800/4800 v4 processors and robust reliability, availability, and serviceability. The 5U modular chassis contains 4 sockets with up to 192 threads. By adding chassis and leveraging high bandwidth, low latency NUMAlink technology, the HPE MC990 X Server can scale as a single system from 4 to 32 sockets and from 1 to 48 TB of cache-coherent shared memory.