Differential PGO build method

The article is an Email I sent to tete009, and I think the method should be open for other builders, because the method can save lots of PGO build time when building the similar source code repeatly.


Hi, Tete

I use the differential build method between small version number changes, because they are almost the same source code (especially the hot code), which can save second/third/… PGO training time.

Take 32.x for the example,
when building 32.0
I use

mozmake -f client.mk realbuild MOZ_PROFILE_GENERATE=1 MOZ_PGO_INSTRUMENTED=1 2>&1 | tee vc-pgo-1.log

mozmake -C ../obju-vc-pgo package MOZ_PGO_INSTRUMENTED=1 MOZ_INTERNAL_SIGNING_FORMAT= MOZ_EXTERNAL_SIGNING_FORMAT=

PogoSafeMode=1 MOZ_PGO_INSTRUMENTED=1 JARLOG_FILE=jarlog/en-US.log EXTRA_TEST_ARGS=10 mozmake -C ../obju-vc-pgo pgo-profile-run

<take the other yourself PGO training ...> --- may cost much more time, depending on your PGO training things

mozmake -f client.mk realbuild MOZ_PROFILE_USE=1 2>&1 | tee vc-pgo-2.log

when building 32.0.1

I will generate official 32.0->32.0.1 patch first (You also can get it from Mozilla HG):

diff -ruNa mozilla-release-32.0 mozilla-release-32.0.1 > 32.0.1.patch

Modify 32.0.1.patch, delete the Non-text changes
enter 32.0 obju-vc-pgo path, delete all *.rc in sub-directory ( It’s Mozilla Build’s bug, *.rc can’t be updated when using differential method )
enter 32.0 source code path and then

patch -p1 < ../32.0.1.patch

It’s very important to point that, if 32.0.1.patch has header files changes, you should make sure obju-vc-pgo/dist’s subdirectory’s header files which the patch has should also be updated manually yourself. ( I think it’s a Mozilla Build’s bug )

then

mozmake -f client.mk realbuild MOZ_PROFILE_USE=1 2>&1 | tee vc-pgo-32.0.1.log

You can see new build will use PGO information automatically.

32.0.2 and 32.0.3 is very similar.

The Differential Build Method has much advantage:
1. save PGO training time except the first edition PGO training
2. It’s very useful for the editions whose code are almost the same, especial each small version number changes (32.0,32.0.1,32.0.2,32.0.3) and ESR edition
3. Improve the build stability a lot. As long as the first PGO edition has no PGO bugs (compiler or linker or PGO information or some another things), the second/third/forth/fifth/… also has no problem.

The Differential Build Method also has disadvantage:
The changes code can’t use PGO information, so these code will use normal build (that is, will exclude these code in PGO information)

for example, 32.0.1 change XUL module code, xul PGO build information will show:
32.0

  Reading PGD file 1: xul.pgd
   正在创建库 xul.lib 和对象 xul.exp
正在生成代码
0 of 0 ( 0.0%) original invalid call sites were matched.
0 new call sites were added.
295731 of 295731 (100.00%) profiled functions will be compiled for speed
512963 of 2303978 inline instances were from dead/cold paths
295712 of 295731 functions (100.0%) were optimized using profile data, and the rest of the functions were optimized without using profile data
461056952657 of 461056952657 instructions (100.0%) were optimized using profile data, and the rest of the instructions were optimized without using profile data

32.0.1

  Reading PGD file 1: xul.pgd
   正在创建库 xul.lib 和对象 xul.exp
正在生成代码
130 of 151 (86.1%) original invalid call sites were matched.
64 new call sites were added.
295729 of 295729 (100.00%) profiled functions will be compiled for speed
461106 of 2270673 inline instances were from dead/cold paths
295709 of 295731 functions (99.9%) were optimized using profile data, and the rest of the functions were optimized without using profile data
460988845138 of 461056952657 instructions (99.9%) were optimized using profile data, and the rest of the instructions were optimized without using profile data

You can see some code will be in invalid call, and some code will in new call, but these code is very few and not hot code most time, which doesn’t affect the firefox’s performance.

So in the end, as long as the source code changes not so much, we can use differential build method to speed up the PGO build time and improve the stability a lot.

And if Mozilla noticed my reported bug (https://bugzilla.mozilla.org/show_bug.cgi?id=866009), the differential method will save more and more time.

Advertisements