On 14.10.2020 16:00, Pablo Galindo Salgado wrote: >> Would it be possible to get the data for older runs back, so that > it's easier to find the changes which caused the slowdown ? > > Unfortunately no. The reasons are that that data was misleading because > different points were computed with a different version of pyperformance and > therefore with different packages (and therefore different code). So the > points > could not be compared among themselves. > > Also, past data didn't include 3.9 commits because the data gathering was not > automated and it didn't run in a long time :(
Make sense. Would it be possible rerun the tests with the current setup for say the last 1000 revisions or perhaps a subset of these (e.g. every 10th revision) to try to binary search for the revision which introduced the change ? > On 2020年10月14日 at 14:57, M.-A. Lemburg <[email protected] > <mailto:[email protected]>> wrote: > > Hi Pablo, > > thanks for pointing this out. > > Would it be possible to get the data for older runs back, so that > it's easier to find the changes which caused the slowdown ? > > Going to the timeline, it seems that the system only has data > for Oct 14 (today): > > > https://speed.python.org/timeline/#/?exe=12&ben=regex_dna&env=1&revs=1000&equid=off&quarts=on&extr=on&base=none > > In addition to unpack_sequence, the regex_dna test has slowed > down a lot compared to Py3.8. > > > https://github.com/python/pyperformance/blob/master/pyperformance/benchmarks/bm_unpack_sequence.py > > https://github.com/python/pyperformance/blob/master/pyperformance/benchmarks/bm_regex_dna.py > > Thanks. > > On 14.10.2020 15:16, Pablo Galindo Salgado wrote: > > Hi! > > > > I have updated the branch benchmarks in the pyperformance server and > now they > > include 3.9. There are > > some benchmarks that are faster but on the other hand some benchmarks > are > > substantially slower, pointing > > at a possible performance regression in 3.9 in some aspects. In > particular > some > > tests like "unpack sequence" are > > almost 20% slower. As there are some other tests were 3.9 is faster, is > not fair > > to conclude that 3.9 is slower, but > > this is something we should look into in my opinion. > > > > You can check these benchmarks I am talking about by: > > > > * Go here: https://speed.python.org/comparison/ > > * In the left bar, select "lto-pgo latest in branch '3.9'" and "lto-pgo > latest > > in branch '3.8'" > > * To better read the plot, I would recommend to select a "Normalization" > to the > > 3.8 branch (this is in the top part of the page) > > and to check the "horizontal" checkbox. > > > > These benchmarks are very stable: I have executed them several times > over the > > weekend yielding the same results and, > > more importantly, they are being executed on a server specially > prepared to > > running reproducible benchmarks: CPU affinity, > > CPU isolation, CPU pinning for NUMA nodes, CPU frequency is fixed, CPU > governor > > set to performance mode, IRQ affinity is > > disabled for the benchmarking CPU nodes...etc so you can trust these > numbers. > > > > I kindly suggest for everyone interested in trying to improve the 3.9 > (and > > master) performance, to review these benchmarks > > and try to identify the problems and fix them or to find what changes > introduced > > the regressions in the first place. All benchmarks > > are the ones being executed by the pyperformance suite > > (https://github.com/python/pyperformance) so you can execute them > > locally if you need to. > > > > --- > > > > On a related note, I am also working on the speed.python.org > <http://speed.python.org> > > <http://speed.python.org> server to provide more automation and > > ideally some integrations with GitHub to detect performance > regressions. For > > now, I have done the following: > > > > * Recompute benchmarks for all branches using the same version of > > pyperformance (except master) so they can > > be compared with each other. This can only be seen in the > "Comparison" > > tab: https://speed.python.org/comparison/ > > * I am setting daily builds of the master branch so we can detect > performance > > regressions with daily granularity. These > > daily builds will be located in the "Changes" and "Timeline" tabs > > (https://speed.python.org/timeline/). > > * Once the daily builds are working as expected, I plan to work on > trying to > > automatically comment or PRs or on bpo if > > we detect that a commit has introduced some notable performance > regression. > > > > Regards from sunny London, > > Pablo Galindo Salgado. > > > > _______________________________________________ > > python-committers mailing list -- [email protected] > <mailto:[email protected]> > > To unsubscribe send an email to [email protected] > <mailto:[email protected]> > > https://mail.python.org/mailman3/lists/python-committers.python.org/ > > Message archived at > > https://mail.python.org/archives/list/[email protected]/message/G3LB4BCAY7T7WG22YQJNQ64XA4BXBCT4/ > > Code of Conduct: https://www.python.org/psf/codeofconduct/ > > > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Oct 14 2020) > >>> Python Projects, Coaching and Support ... https://www.egenix.com/ > >>> Python Product Development ... https://consulting.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > https://www.egenix.com/company/contact/ > https://www.malemburg.com/ > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 14 2020) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ _______________________________________________ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/TYG6SQHN6RUN3DKXGEJKO3SVO2PH3OMO/ Code of Conduct: http://python.org/psf/codeofconduct/