homepage

In 3.6 we get --enable-optimizations. One thing I find annoyed with this flag is that it makes some build commands compile from scratch since it always clears the environment first. For example, with the commands listed at the top of Makefile:
./configure
make 
make test
make install
It compiles 3 times and considering the optimized build needs more time every time, it lasts long.
I am not sure this is a problem and feel free to close it.

Since optimizations uses PGO which should do a fresh run, that's why there's the clearing of the state.
I'm personally not bothered by it as you really should only being doing an optimized build once when you do the first install and then you're done. But if or anyone else can come up with a way to not clear out the results while not accidentally using stale PGO-collected stats or builds then I don't see why we can't speed it up.

That's why I want to enable only --with-lto

Another complaint from #29889.

I've opened a PR#1478 that I believe fixes the issue.

New changeset a1054c3b0037d4c2a5492e79fc193f36245366c7 by Victor Stinner (torsava) in branch 'master':
bpo-29243: Fix Makefile with respect to --enable-optimizations (#1478)
https://github.com/python/cpython/commit/a1054c3b0037d4c2a5492e79fc193f36245366c7

I closed my issue #29641 as a duplicate of this one.

New changeset 03b8a378dfa46372b96790f82c85e9b72518f1bf by Victor Stinner (torsava) in branch '3.6':
[3.6] bpo-29243: Fix Makefile with respect to --enable-optimizations (GH-1478) (#1518)
https://github.com/python/cpython/commit/03b8a378dfa46372b96790f82c85e9b72518f1bf

New changeset 8489409bbfabb2ddc30ed55c9f4d679a3710ebe4 by Victor Stinner (torsava) in branch '3.5':
[3.5] bpo-29243: Fix Makefile with respect to --enable-optimizations (GH-1478) (#1520)
https://github.com/python/cpython/commit/8489409bbfabb2ddc30ed55c9f4d679a3710ebe4

New changeset a473a73d0cb42c534a3047bbf781b3c592fc33ca by Victor Stinner (torsava) in branch '2.7':
[2.7] bpo-29243: Fix Makefile with respect to --enable-optimizations (GH-1478) (#1522)
https://github.com/python/cpython/commit/a473a73d0cb42c534a3047bbf781b3c592fc33ca

So now we have to run make if we want to trigger the profile procedure since other make commands like make test won't trigger the procedure by itself, right?

Xiang Zhang added the comment:
> So now we have to run make if we want to trigger the profile procedure since other make commands like make test won't trigger the procedure by itself, right?
I don't think that the patch changes the behaviour of "make": it only
changes the behaviour of "make install" and "make test" (and a few
others).
To be honest, I don't know if Makefile knows when a rebuild is needed
and how it works. To run benchmarks, I always rebuild Python from
scratch, even from an empty directory!

> I don't think that the patch changes the behaviour of "make": it only
> changes the behaviour of "make install" and "make test" (and a few
> others).
I don't mean "make" is changed. I mean the latter. Before, a single "make test" or "make install" will do all things for you, including the profile procedure. Now they seems not. You have to run "make" before them to trigger the profile procedure.
Actually I am quite okay with this change. But don't know others.

Yes, this appears true.
I'm okay with it because I've always considered the make, test, and install as different command invocations. The package building automation I am aware of tends to do the same. Our buildbots keep the steps separate.
But we need document this requirement somewhere.
I think if we wanted test and install to accurately figure out if they need to redo the profile build and profiling run, we'd need make to detect that _any_ of its possible profile build+run input files are more recent than the last profiling run. ugh.

Oh wait, I expected that "make test" would build Python with PGO, but it doesn't anymore. Would it be possible to change the Makefile to make sure that "make python" (on Linux, "make python.exe" on macOS) would build Python using PGO?
I tried to hack something, but I'm lost in dependencies... "make profile-opt" runs "$(MAKE) build_all_generate_profile" which runs "$(MAKE) build_all CFLAGS_NODIST=..." and build_all depends on "$(BUILDPYTHON)".
I'm not sure that it's possible to make everything automatic because of the high number of targets and dependencies.
To be honest, I'm not excited by ./configure --enable-optimizations. I was happy with an explicit "make profile-opt".

> Oh wait, I expected that "make test" would build Python with PGO, but it
> doesn't anymore. Would it be possible to change the Makefile to make sure
> that "make python" (on Linux, "make python.exe" on macOS) would build Python
> using PGO?
>
> I tried to hack something, but I'm lost in dependencies... "make profile-opt"
> runs "$(MAKE) build_all_generate_profile" which runs "$(MAKE) build_all
> CFLAGS_NODIST=..." and build_all depends on "$(BUILDPYTHON)".
>
> I'm not sure that it's possible to make everything automatic because of the
> high number of targets and dependencies.
In addition, profile-opt would have to rewritten to test whether there's
already an interpreter built with PGO. And if there is, regenerate it only if
the main target is `all` or `profile-opt`, but not when someone runs `make
test`, `make install`, etc.
It's maybe too radical, but we could change it so that if you run `make test`, `make install` and the like and there's no interpreter built already to test or install, the user is told to run `make` first.

To be honest, I'm not excited by ./configure --enable-optimizations. I was happy with an explicit "make profile-opt".
The configure flag was added because not everything can be enabled just in the Makefile and so that people have a single flag to set to turn on any and all optimizations that work consistently across OSs (it used to be LTO+PGO until we discovered how universally broken LTO seems to be).

Is the issue done? Can it be closed?