Doubling performance of pkgsrc/mk

●To: tech-pkg%pkgsrc.org@localhost ●Subject: Doubling performance of pkgsrc/mk ●From: Jonathan Perkin <jperkin%mnx.io@localhost> ●Date: Thu, 20 Oct 2022 16:13:15 +0100

Hi,

I got fed up with pkgsrc being slow again, so have spent a bit of time recently improving performance.

Here are some benchmarks for the build of pkgtools/url2pkg (chosen as its "build" time is very short so exposes any delays with the infrastructure):


  +----------+---------+--------+-------+
  | OS       | SmartOS | NetBSD | macOS |
  +----------+---------+--------+-------+
  | Baseline |   12.5s |  10.3s |  3.0s |
  | Patched  |    5.7s |   5.3s |  1.4s |
  +----------+---------+--------+-------+

All of the builds were done on very different hardware so the OPSYS differences aren't interesting, but the patched times show a consistent 2x speedup regardless.

I'm obviously testing this work a lot as it touches a huge amount of very important code, but I'd be intested in feedback early on, as well as reports of testing from anyone who wants to try it out.


The branch is here:

  https://github.com/TritonDataCenter/pkgsrc/tree/feature/performance/trunk

I may occasionally rebase against trunk as well as squashing/improving patches, so expect to sometimes see a forced update.

As with previous performance work the bulk of this is simply thanks to reducing the number of fork+exec, using DTrace to identify areas that can be optimised. On SmartOS the baseline build invoked 2391 processes, the patched build only 877.

The good news is there's still a bunch of things that can be improved, most notably around tools creation which is still very expensive.


Cheers,

--
Jonathan Perkin   -   mnx.io   -   pkgsrc.smartos.org
Open Source Complete Cloud   www.tritondatacenter.com

●Follow-Ups: ●Re: Doubling performance of pkgsrc/mk ●From: Jonathan Perkin ●Re: Doubling performance of pkgsrc/mk ●From: John Klos ●Re: Doubling performance of pkgsrc/mk ●From: Adam ●Re: Doubling performance of pkgsrc/mk ●From: Thomas Klausner

●Prev by Date: Re: CA bundle for cadaver/neon [PATCH] ●Next by Date: Re: CA bundle for cadaver/neon ●Previous by Thread: Re: pkgsrc-trunk-arm64 macOS 11.3 Darwin 21.6.0/aarch64 2022-10-19 13:12 ●Next by Thread: Re: Doubling performance of pkgsrc/mk ●Indexes: ● reverse Date ● reverse Thread ● Old Index Home | Main Index | Thread Index | Old Index