Compare commits

..
Author SHA1 Message Date
Jörg Thalheim fdab0c716b single-user-install: fix cp flags on freebsd 2025-08-07 22:45:08 +02:00
Jörg Thalheim d7ba936c36 single-user-install: just call uname once 2025-08-07 22:45:08 +02:00
1ff041082c Add sandboxed building for FreeBSD using jails
Co-Authored-By: Artemis Tosini <me@artem.ist>
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2025-08-07 22:45:08 +02:00
Audrey Dutcher 8f91140253 Don't use a redirected subshell in tests/restricted.sh (fails on FreeBSD) 2025-08-07 22:45:08 +02:00
Jörg Thalheim 49b385af00 Merge pull request #13711 from Mic92/chroot-builder
Factor out `ChrootDerivationBuilder`
2025-08-07 22:40:24 +02:00
John Ericson d4ef822add Factor out ChrootDerivationBuilder
This will allow us to have non-Linux chroot-using sandboxed builds.
2025-08-07 13:47:16 -04:00
Sergei Zimmerman 6b6a1e14ce Merge pull request #13708 from NixOS/parallel-mark-2
Enable parallel marking in boehm-gc
2025-08-07 13:56:06 +03:00
Eelco Dolstra 0d559f0c13 Increase the initial Boehm GC mark stack size
If the mark stack size is too small, it greatly inhibits parallel
marking, which is very bad for performance on multi-core systems.
2025-08-07 11:45:06 +02:00
Eelco Dolstra 4c1c4f79a3 Move setting GC_THREADS into eval-gc.hh 2025-08-07 11:44:20 +02:00
Eelco Dolstra a3ff648f90 Call GC_allow_register_threads() to enable parallel marking in Boehm GC 2025-08-07 11:44:07 +02:00
John Ericson c4c3203b7b Merge pull request #13706 from xokdvium/meson-format
meson: Apply formatting universally
2025-08-06 20:50:44 -04:00
Sergei Zimmerman 041a733139 Merge pull request #13705 from obsidiansystems/consolidate-path-functions
Move some `MixStoreDirMethods` members to the right `.cc` file
2025-08-07 03:39:02 +03:00
John Ericson e07440665c Move some MixStoreDirMethods members to the right .cc file
I had not wanted to cause unncessary churn before, but now that we've
bitten the bullet with the Big Reformat, I feel it is the right time.

Future readers will appreciate that the declarations and definitions
files are one-to-one as they should be, and `store-api.cc` is good to
shrink in any event.

I don't think there are outstanding PRs changing this code either. (I
had some for a while, but they are all merged.)
2025-08-06 20:13:15 -04:00
Sergei Zimmerman cf10085940 git-blame-ignore-revs: Add the meson reformatting commit 2025-08-07 03:01:40 +03:00
Sergei Zimmerman 385e2c3542 meson: Apply formatting universally
Now that we have applied the [1] patch, the diff is much
nicer and less noisy.

[1]: https://www.github.com/mesonbuild/meson/pull/14861
2025-08-07 02:58:29 +03:00
Sergei Zimmerman bf80696ed9 meson: Get rid of multiline array formatting hack 2025-08-07 02:57:02 +03:00
Sergei Zimmerman da028aa454 maintainers: Fetch patch for multiline formatting in meson 2025-08-07 02:56:59 +03:00
John Ericson 9ff4c446df Merge pull request #3425 from mkg20001/pr
Add user@address:port support
2025-08-06 17:16:58 -04:00
49ba06175e Add user@address:port support
This patch allows users to specify the connection port
in the store URLS like so:

```
nix store info --store "ssh-ng://localhost:22" --json
```

Previously this failed with: `error: failed to start SSH connection to 'localhost:22'`,
because the code did not distinguish the port from the hostname. This
patch remedies that problem by introducing a ParsedURL::Authority type
for working with parsed authority components of URIs.

Now that the URL parsing code is less ad-hoc we can
add more long-awaited fixes for specifying SSH connection
ports in store URIs.

Builds upon the work from bd1d2d1041.

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2025-08-06 23:48:14 +03:00
tomberek 55f6ff36c2 Merge pull request #13526 from philiptaron/issue-12786/canonicaliseTimestampAndPermissions-on-dir-better
libstore: always canonicalize directory permissions
2025-08-06 16:30:13 -04:00
Jörg Thalheim 1376e27156 Merge pull request #13702 from h0nIg/revert-13351-push-rrzylpqynznw
Revert "docker: make sure `nix config check` works"
2025-08-06 20:49:11 +02:00
h0nIg 02cc215221 Revert "docker: make sure nix config check works" 2025-08-06 20:33:44 +02:00
Jörg Thalheim c76222e87e Merge pull request #13694 from NaN-git/opt-parseString
speedup derivation parsing by optimizing `parseString`
2025-08-06 20:23:27 +02:00
John Ericson c98af65da6 Merge pull request #13697 from obsidiansystems/base64-trailing-test
Base64 trailing test (from Snix, thanks)
2025-08-06 13:13:17 -04:00
John Ericson e25ab029ae Merge pull request #13693 from obsidiansystems/more-base-files
Clean up Base* code
2025-08-06 13:02:01 -04:00
Philip TaronandRobert Hensing c38987e04a libstore: always canonicalize directory permissions
Prior to this patch, mode 0444 is not updated to 0555 for directories.
That means for instance 0554 is canonicalized, but not 0444.

We don't believe this has any implications for backwards compatibility,
because directories do not have permissions in NAR format and so are
always 0555 after deserialization, and store paths with wrong
permissions can’t be copied to another host.

Co-authored-by: Robert Hensing <robert@roberthensing.nl>
2025-08-06 10:58:46 -05:00
John Ericson ead795404d Base64 trailing test (from Snix, thanks) 2025-08-06 11:52:11 -04:00
John EricsonandSergei Zimmerman 991831227e Clean up Base* code
Make it separate from Hash, since other things can be base-encoded too.

This isn't really needed for Nix, but it makes the code easier to read
e.g. for someone reimplementing this stuff in a different language. (Of
course, Base16/Base64 should be gotten off-the-shelf, but now the hash
code, which is more bespoke, is less cluttered with the parts that would
be from some library.)

Many reimplementations of "Nix32" and our hash type already exist, so
this cleanup is coming years too late, but I say better late than never
/ it is always good to nudge the code in the direction of being a
"living spec".

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-08-06 11:51:44 -04:00
Sergei Zimmerman 664f06c94c Merge pull request #13699 from xokdvium/fix-dev-shell-build
nix/profile.cc: Fix header include path
2025-08-06 13:31:06 +03:00
Sergei Zimmerman 53492c0ecd Merge pull request #13555 from k1gen/no-url-literals-test
Add functional test for no-url-literals experimental feature
2025-08-06 11:29:08 +03:00
Sergei Zimmerman 68f61d3fcf nix/profile.cc: Fix header include path
This wasn't caught by CI for aea312dae3
due to weird componentized build reasons.
2025-08-06 11:01:39 +03:00
Philipp Otterbein 5e94fe5693 speedup derivation parsing 2025-08-05 22:00:30 +02:00
Jörg Thalheim fd6e383a63 Merge pull request #13689 from xokdvium/dead-code
libstore: Remove dead variable
2025-08-05 14:16:00 +02:00
Jörg Thalheim 55ff55fefa Merge pull request #13684 from the-sun-will-rise-tomorrow/identity-compression
Recognize "identity" compression method
2025-08-05 14:07:01 +02:00
Jörg Thalheim cb84ac203b Merge pull request #13687 from xokdvium/move-nix2-sources
nix-cli: Move nix2 binaries sources properly into nix subproject
2025-08-05 14:06:29 +02:00
Jörg Thalheim 4a7285c649 Merge pull request #13688 from xokdvium/odr-toplevel-types
treewide: Move private types and code into anonymous namespaces
2025-08-05 14:05:16 +02:00
Sergei Zimmerman e3d7f20afe libstore: Remove dead variable 2025-08-05 14:03:38 +02:00
Sergei Zimmerman 866d5e6cf4 treewide: Sprinkle more anonymous namespace for classes private to TUs
This code should be private to the corresponding translation units.
2025-08-05 02:25:40 +03:00
Sergei Zimmerman 0118e5ea5d libutil: Move Ctx type from the nix namespace to Hash class
Same as previous commit. This really should not be a part of
the `nix` namespace. Otherwise the doxygen documentation is
really confusing.
2025-08-05 02:20:28 +03:00
Sergei Zimmerman 5ee0d5669e libstore: Move State to an anonymous namespace
Having a State class in the nix namespace is asking
for ODR trouble. This class is already private to the
translation unit, let's move it into an anonymous namespace.
2025-08-05 02:15:21 +03:00
Sergei Zimmerman aea312dae3 nix-cli: Move nix2 binaries sources properly into nix subproject
There have been prior concerns about reogranizing the repo, but
this seems like a trivial simplification which will not interfere
with either our packaging or the modular builds in nixpkgs.
2025-08-05 02:10:21 +03:00
Sergei Zimmerman e5a8ee45b7 Merge pull request #13686 from xokdvium/ci-coverage
ci: Collect code coverage in tests
2025-08-05 00:41:41 +03:00
Sergei ZimmermanandJade Lovelace 4333a9d5a8 ci: Collect code coverage in tests
This adds the necessary infrastructure to collect
code coverage in CI, which could be useful to look
at munually or track consistently via something like
codecov.

Co-authored-by: Jade Lovelace <lix@jade.fyi>
2025-08-04 23:50:02 +03:00
John Ericson 0889960869 Merge pull request #13685 from obsidiansystems/hash-sentinal-encapsulation
Encapsulate `invalidBase32`, avoid 0xFF magic number
2025-08-04 16:01:16 -04:00
John Ericson 23c87d8a21 Encapsulate invalidBase32, avoid 0xFF magic number
This keeps things fast by making the function inline, but also prevents
people from having to know about the `0xFF` implementation detail
directly, instead making one go through a `std::optional` (which could be
fused away with a sufficiently smart compiler).

Additionally, the base "nix32" implementation is moved to its own header
file pair, as it is logically distinct and prior to the `Hash` data
type. It would probably be nice to do this with all the hash format
implementations.
2025-08-04 15:32:38 -04:00
John Ericson 6ab8cbe31a Merge pull request #13680 from avnik/avnik/fast-base32
Improve base-32 hash decoding performance with reverse map
2025-08-04 14:22:59 -04:00
Jörg Thalheim c7af923865 Merge pull request #13445 from xokdvium/simplify-util-url
libutil: Use Boost.URL for URI parsing
2025-08-04 19:46:58 +02:00
WxNzEMof 0a96b7e627 Recognize "identity" compression method
Some binary caches (incorrectly) use this header to indicate lack of
compression, inspired by the valid "identity" token in the
"Accept-Encoding" header.
2025-08-04 17:00:38 +00:00
John Ericson 7abfc9b92a Merge pull request #13682 from xokdvium/refscan-benchmark
libstore-tests: Add reference scanning benchmark
2025-08-03 23:29:40 -04:00
Sergei Zimmerman 20df0e5fa7 libstore-tests: Add reference scanning benchmark
This benchmark should provide a relatively realistic
scenario for reference scanning.

As shown by the following results, reference scanning code
is already plenty fast and is definitely not a bottleneck:

```
BM_RefScanSinkRandom/10000               1672 ns         1682 ns       413354 bytes_per_second=5.53691Gi/s
BM_RefScanSinkRandom/100000             11217 ns        11124 ns        64341 bytes_per_second=8.37231Gi/s
BM_RefScanSinkRandom/1000000           205745 ns       204421 ns         3360 bytes_per_second=4.55591Gi/s
BM_RefScanSinkRandom/5000000          1208407 ns      1201046 ns          597 bytes_per_second=3.87713Gi/s
BM_RefScanSinkRandom/10000000         2534397 ns      2523344 ns          273 bytes_per_second=3.69083Gi/s
```

(Measurements on Ryzen 5900X via `nix build --file ci/gha/tests componentTests.nix-store-tests-run -L`)
2025-08-04 02:11:14 +03:00
Alexander V. Nikolaev 4bfc007a7b Improve base-32 hash decoding performance with reverse map
The changes include:

* Defining nix32Chars as a constexpr char[].
* Adding a constexpr std::array<unsigned char, 256> (reverseNix32Map) to map characters to their base-32 digit values at compile time.
* Replacing the slow character search loop with a direct lookup using reverseNix32Map.
* Removing std::once_flag/isBase32 logic in references.cc in favor of reverseNix32Map

Signed-off-by: Alexander V. Nikolaev <avn@avnik.info>
2025-08-03 19:00:56 +03:00
John Ericson 51a32e4645 Merge pull request #13679 from xokdvium/build-benchmarks-in-ci
libstore-tests: Build benchmarks in GHA CI
2025-08-02 19:44:04 -04:00
John Ericson dbccfc277f Merge pull request #13678 from xokdvium/pch-disable-gcc
meson: Disable PCH for GCC
2025-08-02 19:42:17 -04:00
Sergei Zimmerman ea1f67393d libstore-tests: Build and run benchmarks in CI
This changes our GHA CI and nix-store-tests packaging
to build and run the benchmarks. This does not affect
the default packaging - the overrides apply only for the
GHA CI.
2025-08-03 01:09:04 +03:00
Sergei Zimmerman b2b2f2dc53 libstore-tests: Split bench-main into a separate file
This makes it easier to add new benchmarks.
2025-08-03 01:03:40 +03:00
Sergei Zimmerman b8301b2c25 libstore-tests: Add nix-store-benchmarks as a meson benchmark 2025-08-03 01:03:37 +03:00
Sergei Zimmerman 5db50e3f77 meson: Disable PCH for GCC
GCC doesn't really benefit as much as Clang does from
using precompiled headers. Another aspect to consider is that
clangd doesn't really like GCC's PCH flags in the compilation database,
so GCC based devshells would continue to work with clangd.

This also has the slight advantage of ensuring that our includes are in
order, since we build with both Clang and GCC.
2025-08-03 00:08:40 +03:00
Oleksandr Knyshuk 6e8f76482e Add functional test for no-url-literals experimental feature
Closes: #13533
2025-07-31 15:34:49 +02:00
John Ericson bbd14173b5 Merge pull request #13263 from NixOS/structured-attrs-in-drv
Store `StructuredAttrs` directly in `Derivation`
2025-07-30 16:14:14 -04:00
John Ericson 8652b6b417 Store StructuredAttrs directly in Derivation
Instead of parsing a structured attrs at some later point, we parsed it
right away when parsing the A-Term format, and likewise serialize it to
`__json = <JSON dump>` when serializing a derivation to A-Term.

The JSON format can directly contain the JSON structured attrs without
so encoding it, so we just do that.
2025-07-29 17:28:16 -04:00
John Ericson b062730665 Merge pull request #13569 from Mic92/benchmark
add derivation parser benchmark
2025-07-29 14:41:10 -04:00
Robert Hensing c85a014ef9 Merge pull request #13489 from k1gen/add-warn-short-path-literals
Add warn-short-path-literals setting
2025-07-29 20:32:56 +02:00
Robert Hensing 6365cf1446 Merge pull request #13572 from xokdvium/nixos-tests-parametrize
tests/nixos: Parametrize on the system
2025-07-29 20:29:24 +02:00
Jörg Thalheim 1989dd7bf9 add derivation parser benchmark
the current identified bottlenecks are parseString in derivations.cc and dirOf (because of std::filessystem creation).
2025-07-29 16:51:55 +02:00
Oleksandr KnyshukandRobert Hensing 6d46dc9f6a Add warn-short-path-literals setting
Add a new setting to warn about path literals that don't start with "." or "/". When enabled,
expressions like `foo/bar` will emit a warning suggesting to use `./foo/bar` instead.

A functional test is included.

The setting defaults to false for backward compatibility but could eventually default to true in
the future.

Closes: #13374

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2025-07-29 15:47:58 +02:00
Sergei Zimmerman 1c9e11a11f tests/nixos: Parametrize on the system
All of the existing tests only run on x86_64-linux and
the whole `nixpkgsFor` makes it hard to reuse those for
e.g. running the nixos tests with a sanitizer build of nix.
This rips off the bandaid and removes the `nixpkgsFor` parameter
in favor of a single instance of nixpkgs with a separate `nixComponents`.
2025-07-29 01:24:03 +03:00
John Ericson 47f5e5fbef Merge pull request #13567 from xokdvium/split-sanitizers-ci
ci: Run sanitizer tests in a separate job
2025-07-28 17:21:19 -04:00
Sergei Zimmerman 413f782176 ci: Run sanitizer tests in a separate job
This should speed up the CI somewhat by parallelizing
the work across the matrix of configurations.
2025-07-28 23:37:04 +03:00
Jörg Thalheim 9c9473aa9b Merge pull request #13566 from xokdvium/refactor-ci-tests
flake: Factor out checks into ci/gha/tests for reusability
2025-07-28 21:11:13 +02:00
Robert Hensing 5bd68f24aa Merge pull request #13543 from obsidiansystems/git-256
Implement support for Git hashing with SHA-256
2025-07-28 21:00:32 +02:00
Sergei Zimmerman 9913ec55ba flake: Factor out checks into ci/gha/tests for reusability
This moves out the checks that get run in GHA CI into ci/gha/tests
folder and splits those into `topLevel` and `componentTests` attributes.
The idea behind this is to make it easier to parametrize tests that can
be run with sanitizers in order to run those as a matrix of jobs. The same
can be said for static builds.

Existing stdenv selection infrastructure via `lib.makeComponents` would
also allow us to switch over to using `clangStdenv` to significantly speed
up pre-merge CI (though the default stdenv would still be used for non-overridable
topLevel checks, like installer artifacts).
2025-07-28 03:26:48 +03:00
Sergei Zimmerman cff021898d Merge pull request #13525 from alurm/patch-1
doc/advanced-attributes.md: remove stray backslashes in the rendered output
2025-07-28 01:34:58 +03:00
John Ericson 44a8f8d02b Merge pull request #13558 from xokdvium/nix-cpu-system-fix
meson: Fix `nix_system_cpu` for MIPS and 32 bit ARM systems
2025-07-26 19:18:57 -04:00
Sergei Zimmerman a2561b853a Merge pull request #13557 from Jayman2000/sometimes-build-manual-when-cross-compiling
Sometimes build `nix-manual` when cross compiling
2025-07-27 01:28:34 +03:00
Sergei Zimmerman 60d124b36e meson: Fix nix_system_cpu for MIPS and 32 bit ARM systems
Prior patches in 54dc5314e8
and 6db6190002 fixed the default
system double for i686 and ppc/ppc64. This also patch also covers
32 bit arm and mips. ARM cpu names are taken from host_machine.cpu()
for a lack of a better option, but host_machine.cpu_family() is
preferred, since that is supposed to be somewhat standard for cross
files. Endianness is handled correctly by looking at host_machine.endian().

This also updates the documentation to be up to date to how system cpu
is translated from the host_machine specification.
2025-07-27 00:50:53 +03:00
Jason Yundt 5e407e6abb Sometimes build nix-manual when cross compiling
Before this change, if you were cross compiling Nix, then the nix-manual
subproject would never get built. In some situations, it makes sense to
not build the nix-manual subproject when cross compiling. For example,
if the build system is x86_64 and the host system is riscv64, then it
makes sense to not build the manual. Building the manual requires
executing certain build artifacts, and you can’t run x86_64 executables
on riscv64 systems.

That being said, there are some situations where it does make sense to
build the nix-manual subproject when cross compiling. For example, if
the build system is x86_64 and the host system is i686, then it doesn’t
make sense to not build the manual. You can run i686 executables on
x86_64 systems just fine.

This change makes it so that the nix-manual subproject will sometimes
get built when cross compiling. Specifically, the nix-manual subproject
will get built as long as the doc-gen option is enabled and the build
system is capable of running host binaries.

---

The main motivation behind this change is to fix this Nixpkgs issue [1].
Building pkgs.nixStatic counts as cross compiling Nix, and
pkgs.nixStatic is supposed to produce a man output. Building
pkgs.nixStatic currently fails because it isn’t actually producing a man
output. That issue will go away once this commit gets backported to Nix
2.28.x.

[1]: <https://github.com/NixOS/nixpkgs/issues/426410>
2025-07-26 15:00:02 -04:00
John Ericson d21e3f88ec Implement support for Git hashing with SHA-256
SHA-256 is Git's next hash algorithm. The world is still basically stuck
on SHA-1 with git, but shouldn't be. We can at least do our part to get
ready.

On the C++ implementation side, only a little bit of generalization was
needed, and that was fairly straight-forward. The tests (unit and
system) were actually bigger, and care was taken to make sure they were
all cover both algorithms equally.
2025-07-25 10:19:08 -04:00
Jörg Thalheim 7f4acb9f10 Merge pull request #13553 from roberth/fix-url
Fix documentation link
2025-07-25 11:37:58 +02:00
Robert Hensing 49f411c08c Update developer facing links to nix.dev
Not broken, but it's a tiny bit quicker
2025-07-25 10:59:44 +02:00
Robert Hensing b7782809cb docs, messages: Resolve nixos.org/manual/nix/stable redirect
Loads a bit quicker
2025-07-25 10:18:13 +02:00
Robert Hensing 35835e0644 Fix documentation link
The file was renamed. We've also moved to nix.dev, but that was
redirected properly.
Closes #13488
2025-07-25 10:18:11 +02:00
Sergei Zimmerman d4c562c6ff Merge pull request #13546 from xokdvium/dogfood-ci-boolean
ci: Make it actually possible to disable dogfooding
2025-07-25 02:18:00 +03:00
Sergei Zimmerman f0695e177f ci: Make it actually possible to disable dogfooding
Github composite actions are a real treat. Boolean inputs
are not actually booleans but rather strings [1].

[1]: https://www.github.com/actions/runner/issues/2238
2025-07-25 00:31:04 +03:00
Eelco Dolstra 2b22871ea2 Merge pull request #13535 from NixOS/fix-i686
Fix nix_system_cpu on i686-linux
2025-07-24 19:13:06 +02:00
Eelco Dolstra 54dc5314e8 Fix nix_system_cpu on i686-linux
Fixes #13532.
2025-07-24 18:06:06 +02:00
John Ericson f0485520cc Merge pull request #13530 from KAction/contrib/0/nar-format/out
Fix documentation of the NAR archive structure
2025-07-24 11:17:33 -04:00
Eelco Dolstra cbdb497c79 Merge pull request #13478 from NixOS/posix-source-accessor-concurrent-map
PosixSourceAccessor: Use concurrent_flat_map
2025-07-24 11:19:37 +02:00
Robert Hensing 3543a73195 Merge pull request #13456 from fzakaria/git-pack-ref-cache
Add a test case where fetchGit is failing to cache due to packed-refs
2025-07-23 23:04:29 +02:00
Robert Hensing eed1d370d0 Merge pull request #13521 from fzakaria/fzakaria/issue-11266
Unpeel reference for git+file
2025-07-23 22:36:44 +02:00
Jörg Thalheim a3f6508761 Merge pull request #13520 from xokdvium/ppc-meson-endian
meson: Correctly handle endianness for PowerPC CPU families
2025-07-23 22:35:58 +02:00
Jörg Thalheim 7ff426627d Merge pull request #13506 from xokdvium/simplify-installer-tests
ci: Simplify installer tests
2025-07-23 22:25:38 +02:00
Sergei Zimmerman c6d0d33e4d ci: Simplify installer tests 2025-07-23 21:52:23 +02:00
Eelco Dolstra 3a67cafa0d Merge branch 'master' (reformat) 2025-07-23 21:22:27 +02:00
Eelco Dolstra 72e8f7cd35 Merge branch 'master' (pre-reformat) 2025-07-23 21:22:19 +02:00
Alan Urmancheev bd29dc67b7 doc/advanced-attributes.md: remove stray backslashes in the rendered output
They have appeared literally because the was an empty line after them.
2025-07-23 17:32:52 +04:00
Dmitry Bogatov 143b87c1a9 Fix documentation of the NAR archive structure
For regular, non-executable files, there is no str("") between str("regular")
and str("contents"). Note that str("") is exactly 8 zero bytes, while just ""
is actual empty string (0 bytes).
2025-07-22 20:00:00 -04:00
Sergei Zimmerman 9671275198 Merge pull request #13522 from juhp/patch-1
meson: libfetchers needs libgit2 1.9+
2025-07-22 21:45:17 +03:00
Jens Petersen ff6ab3b213 meson: libfetchers needs libgit2 1.9+
libfetchers uses `git_mempack_write_thin_pack` which was introduced in libgit2-1.9.0

This avoids error like:

../src/libfetchers/git-utils.cc: In member function ‘virtual void nix::GitRepoImpl::flush()’:
../src/libfetchers/git-utils.cc:270:13: error: ‘git_mempack_write_thin_pack’ was not declared in this scope
  270 |             git_mempack_write_thin_pack(mempack_backend, packBuilder.get())
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~

on older libgit2 (like 1.7.2 in Centos Stream 10)
2025-07-22 22:58:55 +08:00
Farid Zakaria aadfb682d4 Fix fetchGit functional tests to peel as well 2025-07-21 22:01:05 -07:00
Farid Zakaria 98858148dc Add unit test 2025-07-21 21:56:11 -07:00
Farid Zakaria 33ceea6099 Unpeel reference for git+file
If the reference for git+file is an annotated tag, the revision will
differ than when it's fetched using other fetchers such as `github:`
since Github seems to automatiacally peel to the underlying commit.

Turns out that rev-parse has the capability through it's syntax to
request the underlying commit by "peeling" using the `^{commit}` syntax.

This is safe to apply in all scenarios where the goal is to get an
underlying commit.

fixes #11266
2025-07-21 21:10:41 -07:00
John Ericson 3cbcceee02 Merge pull request #13519 from xokdvium/pch-instantiate-templates
meson: Further optimize compile times with PCH template instantiations
2025-07-21 22:39:45 -04:00
Sergei Zimmerman ebd311b7b7 meson: Correctly handle endianness for PowerPC CPU families
I've missed this while reviewing 6db6190002.
I only built big endian ppc64, so that didn't occur to me.

From meson manual:

> Those porting from autotools should note that Meson does not add
> endianness to the name of the cpu_family. For example, autotools will
> call little endian PPC64 "ppc64le", Meson will not, you must also check
> the .endian() value of the machine for this information.

This code should handle that correctly.
2025-07-22 03:27:27 +03:00
Farid Zakaria 0c32b0c8c3 Added comment to test case 2025-07-22 02:52:53 +03:00
Farid Zakaria 48083028ac Add a test case for failing git cache
builtins.fetchGit is not using the cached Git directory if
packed-references are used.

This is because the ref file for the fetchGit `refs/heads/master` is
used to check the mtime for whether to cache or not.

Let's at least codify this failure in a test case.
2025-07-22 02:52:53 +03:00
Sergei Zimmerman 9bc6c30d97 meson: Further optimize compile times with PCH template instantiations
This is a follow-up to 6ec50ba736, which
also almost halves the compile times on clang for subprojects that use PCH.

`-fpch-instantiate-templates` is a clang-only option to force the instantiation
of templates once in the PCH itself, not all of the translation units that
it gets included to. This really cuts down on the overhead from nlohmann::json
and std::format code:

 48244 ms: nlohmann::basic_json<>::parse<const char *> (76 times, avg 634 ms)
 36193 ms: nlohmann::basic_json<>::basic_json (310 times, avg 116 ms)
 28307 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (76 times, avg 372 ms)
 20334 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (76 times, avg 267 ms)
 17387 ms: nlohmann::basic_json<>::json_value::json_value (389 times, avg 44 ms)
 16822 ms: std::vformat_to<std::__format::_Sink_iter<char>> (76 times, avg 221 ms)
 16771 ms: std::__format::__do_vformat_to<std::__format::_Sink_iter<char>, char... (76 times, avg 220 ms)
 12160 ms: std::vformat_to<std::__format::_Sink_iter<wchar_t>> (76 times, avg 160 ms)
 12127 ms: std::__format::__do_vformat_to<std::__format::_Sink_iter<wchar_t>, w... (76 times, avg 159 ms)
 10397 ms: nlohmann::detail::json_sax_dom_callback_parser<nlohmann::basic_json<... (76 times, avg 136 ms)
  9118 ms: nlohmann::basic_json<>::data::data (76 times, avg 119 ms)

Initially done by Jade Lovelace <lix@jade.fyi> in https://gerrit.lix.systems/c/lix/+/1842.
We are doing basically the same, but unconditionally. It would be
a huge pain to add a pch option for all subprojects to just support the
usecase of using clangd in a gcc devshell.

In total, this basically halves the frontend times for nix-util and nix-store
to the point that the most expensive part of the build is linking.

(nix-store before):

```
**** Time summary:
Compilation (77 times):
  Parsing (frontend):          243.4 s
  Codegen & opts (backend):    140.3 s
```

(nix-store after):

```
**** Time summary:
Compilation (77 times):
  Parsing (frontend):          120.2 s
  Codegen & opts (backend):    141.2 s
```
2025-07-22 02:39:06 +03:00
Sergei Zimmerman e2b0ff18f8 Merge pull request #13514 from OPNA2608/fix/meson-powerpc-cpu-names
treewide: Fix Meson CPU names for powerpc CPUs
2025-07-22 01:48:22 +03:00
OPNA2608 6db6190002 treewide: Fix Meson CPU names for powerpc CPUs 2025-07-21 19:03:49 +02:00
Robert Hensing 6ec50ba736 Merge pull request #13512 from xokdvium/speed-up-builds-pch
Reintroduce precompiled headers
2025-07-21 12:38:52 +02:00
Sergei Zimmerman 7e184bbc29 Merge pull request #13510 from xokdvium/build-profiling 2025-07-21 11:44:12 +03:00
Sergei Zimmerman 9dae9f6cab nix-cli: Speed up builds by using precompiled headers 2025-07-20 22:02:01 +03:00
Sergei Zimmerman 46469983e7 libcmd: Speed up builds by using precompiled headers 2025-07-20 22:01:59 +03:00
Sergei Zimmerman af6c84b523 libexpr: Speed up builds by using precompiled headers 2025-07-20 22:01:56 +03:00
Sergei Zimmerman 33d031095f libstore: Speed up builds by using precompiled headers 2025-07-20 22:01:53 +03:00
Sergei Zimmerman e920e28734 libutil: Speed up builds by using precompiled headers 2025-07-20 22:01:51 +03:00
Sergei Zimmerman 9c58c8f739 ci: Add build profiling job
This adds a GHA jobs to help analyze build times
and its regressions. It is based on `clangStdenv` with `-ftime-trace`
together with `ClangBuildAnalyzer` to prepare markdown summary for
individual components.

This also has the minor benefit of dogfooding CA and impure derivations.
2025-07-20 18:26:26 +03:00
Sergei Zimmerman c70e1433ab Merge pull request #13073 from roberth/first-class-splicing
First class cross
2025-07-19 17:58:10 +03:00
Robert Hensing 9b9e49e8e4 overlays.default: Only set nix
We should keep users' pkgs tidy.
2025-07-19 16:27:55 +03:00
Robert Hensing e70dac8244 doc: Add brief comments to the components
Users can access these through the `lib.makeComponents` return value,
so it's helpful to briefly explain some of them.

This doesn't replace `meta.description`, but supplements it.
(TODO: improve `meta.description`)
2025-07-19 16:27:55 +03:00
Robert Hensing 0468c1d69a flake.nix: Create package sets independent of overlay
This was carefully refactored by inlining some Nixpkgs definitions
like `generateSplicesForMkScope`, so the memoization properties
should remain the same.
2025-07-19 16:27:55 +03:00
Sergei Zimmerman a54284cbc7 rl-next: Add release note about IPv6 Scoped Addresses in URIs 2025-07-18 21:24:03 +03:00
Sergei ZimmermanandJörg Thalheim bd1d2d1041 libutil: Use Boost.URL in parseURL
Boost.URL is a significantly more RFC-compliant parser
than what libutil currently has a bundle of incomprehensible
regexes.

One aspect of this change is that RFC4007 ZoneId IPv6 literals
are represented in URIs according to RFC6874 [1].

Previously they were represented naively like so: [fe80::818c:da4d:8975:415c\%enp0s25].
This is not entirely correct, because the percent itself has to be pct-encoded:

> "%" is always treated as
   an escape character in a URI, so, according to the established URI
   syntax [RFC3986] any occurrences of literal "%" symbols in a URI MUST
   be percent-encoded and represented in the form "%25".  Thus, the
   scoped address fe80::a%en1 would appear in a URI as
   http://[fe80::a%25en1].

[1]: https://datatracker.ietf.org/doc/html/rfc6874

Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
2025-07-18 21:24:01 +03:00
Sergei Zimmerman d020f21a2a libutil: Use default operator== for ParsedURL
The default comparison operator can be generated
by the compiler since C++20.
2025-07-18 21:23:42 +03:00
Sergei Zimmerman ad449c0288 libutil: Refactor percentDecode,percentEncode to use Boost.URL
The myriad of hand-rolled URL parsing and validation code
is a constant source of problems. Regexes are not a great way
of writing parsers and there's a history of getting them wrong.
Boost.URL is a good library we can outsource most of the heavy
lifting to.
2025-07-18 21:23:40 +03:00
Sergei Zimmerman d9053390ce libutil-test-support: Add HasSubstrIgnoreANSIMatcher
This matcher is useful for checking error messages, which
always contain ANSI escapes.
2025-07-18 21:23:38 +03:00
Sergei Zimmerman ffc9bfb66d lib{store,flake}-tests: Add test for spaces in URIs
These cases do not seem to be covered by the test suite at all.
2025-07-18 21:23:35 +03:00
Sergei Zimmerman b8d223a210 Merge pull request #13108 from DeterminateSystems/check-format
Rip off the bandaid: Format the codebase with clang-format
2025-07-18 20:38:50 +03:00
Graham Christensen fb493ad7ca Update .git-blame-ignore-revs to ignore the mass reformatting 2025-07-18 12:47:27 -04:00
Graham Christensen e4f62e4608 Apply clang-format universally.
* It is tough to contribute to a project that doesn't use a formatter,
* It is extra hard to contribute to a project which has configured the formatter, but ignores it for some files
* Code formatting makes it harder to hide obscure / weird bugs by accident or on purpose,

Let's rip the bandaid off?

Note that PRs currently in flight should be able to be merged relatively easily by applying `clang-format` to their tip prior to merge.
2025-07-18 12:47:27 -04:00
Graham Christensen 41bf87ec70 Update clang-format with fixing namespace coments, and separate definition blocks 2025-07-18 12:46:51 -04:00
Graham Christensen e7af2e6566 Drop a ton of files that should just get formatted 2025-07-18 12:46:51 -04:00
Graham Christensen 6896761d79 Add sed 2025-07-18 12:46:51 -04:00
Graham Christensen ee9b57cbf5 format.sh: support looping until it is happy 2025-07-18 12:46:51 -04:00
Sergei Zimmerman 60c48b7058 Merge pull request #13501 from k1gen/fix-rsync-requirement
Require rsync in nix-manual meson.build
2025-07-18 00:43:27 +03:00
Oleksandr Knyshuk cfb8a31885 Require rsync in nix-manual meson.build
Closes: #13313
2025-07-17 22:59:37 +02:00
Jörg Thalheim 599c1d78ed Merge pull request #13496 from k1gen/fix-help-shell-command
Fix `nix help shell` by properly handling command aliases
2025-07-17 19:40:47 +02:00
Jörg Thalheim 4c95086879 Merge pull request #13493 from Mic92/clang-tidy-virtual-methods
Fix virtual method calls during construction
2025-07-17 17:41:18 +02:00
Oleksandr Knyshuk d678b071d6 Make nix help shell work by handling aliases properly
Previously, `nix help shell` failed with "Nix has no subcommand 'shell'" despite `nix shell --help`
working correctly. This happened because the `shell` command is actually an alias for `env shell`,
and the help system wasn't resolving aliases when looking up documentation.

This patch modifies the `showHelp` function to check for and resolve aliases before generating the
manpage name, ensuring that shorthand commands like `shell` get proper help documentation.

Closes: #13431
2025-07-17 17:26:56 +02:00
Jörg Thalheim 32a11a64c1 Merge pull request #13491 from Mic92/clang-tidy-windows-fixes
Fix Windows header inclusions for clang-tidy
2025-07-17 16:46:54 +02:00
Jörg Thalheim 36d451ddf4 Merge pull request #13476 from NixOS/improve-ignored-exceptions
Improve rendering of ignored exceptions
2025-07-17 15:57:59 +02:00
Jörg Thalheim a4d03ba527 Merge pull request #13370 from NixOS/unit-tests-home
Give unit tests access to a $HOME directory
2025-07-17 15:56:26 +02:00
Jörg Thalheim 44963da787 Fix virtual method calls during construction in LocalBinaryCacheStore
Move init() call from constructor to openStore() method to avoid calling
virtual methods during object construction. This prevents undefined
behavior when virtual methods are called before the object is fully
constructed.
2025-07-17 15:45:32 +02:00
Jörg Thalheim f12f96bcbb Fix virtual method calls during construction in S3BinaryCacheStoreImpl
Move init() call from constructor to openStore() method to avoid calling
virtual methods during object construction. This prevents undefined
behavior when virtual methods are called before the object is fully
constructed.
2025-07-17 15:45:32 +02:00
Jörg Thalheim 6681933643 Fix Windows header inclusions for clang-tidy
Move windows-error.hh includes inside _WIN32 guards to prevent
clang-tidy errors when analyzing these files on non-Windows platforms.
2025-07-17 15:40:46 +02:00
tomberek de1d91f711 Merge pull request #13486 from fzakaria/issue-12935
Add helpful messages when file:// used as tarball
2025-07-17 02:51:31 -04:00
tomberek 5ced59a21b Merge pull request #13449 from fzakaria/issue-10431
Add error message when git returns non-0 for fetch
2025-07-17 02:50:13 -04:00
Farid Zakaria 196c21c5a0 Add helpful messages when file:// used as tarball
When `file://` is used accidentally in a flake as the source it is
expected to be a tarball by default.

Add some friendlier error messages to either inform the user this is not
in fact a tarball or if it's a git directory, let them know they can use
`git+file`.

fixes #12935
2025-07-16 21:17:07 -07:00
Eelco Dolstra 17c94ca89e Merge pull request #13469 from drupol/push-nnwkpwsowkqm
docker: set default parameters values
2025-07-16 21:02:51 +02:00
Farid Zakaria a4f548fed1 Fix FetchGit test 2025-07-16 08:04:16 -07:00
Farid Zakaria fb6f494d35 merge stderr to stdout so we can emit it 2025-07-16 08:04:16 -07:00
Farid Zakaria 8e8416387c Add error message when git returns non-0 for fetch
Users have complained that fetchGit is flaky however the culprit is
likely that `git fetch` was unable itself to download the repository for
whatever reason (i.e. poor network etc..)

Nothing was checking the status of `git fetch` and the error message
that would eventually surface to the users were that the commit was not
found.

Add explicit error checking for status code from `git fetch` and return
a message earlier on to indicate that the failure was from that point.

fixes #10431
2025-07-16 08:04:16 -07:00
Eelco Dolstra 1272c4957f Merge pull request #13442 from NixOS/boost-minver
Require Boost 1.82.0 or higher
2025-07-16 14:54:14 +02:00
Sergei Zimmerman d7c340afbf Merge pull request #13473 from xokdvium/dogfood-ci 2025-07-15 22:35:43 +03:00
Eelco Dolstra 8e98f62a6e PosixSourceAccessor: Use concurrent_flat_map 2025-07-15 19:49:07 +02:00
Eelco Dolstra 7b2f24d688 Improve handleSQLiteBusy() message
Closes https://github.com/NixOS/nix/pull/10319.
2025-07-15 19:09:45 +02:00
Eelco Dolstra fde6068874 Improve rendering of ignored exceptions
Instead of

  error (ignored): error: SQLite database '...' is busy

we now get

  error (ignored): SQLite database '...' is busy
2025-07-15 19:09:40 +02:00
Eelco Dolstra 6abc29bba5 Move boost version check to libutil 2025-07-15 15:17:33 +02:00
Sergei Zimmerman 3b3c02160d ci: Dogfood nix from master for vm_tests and flake_regressions
This should provide more coverage for the build from master that
is being dogfooded.
2025-07-14 17:03:12 +03:00
Sergei Zimmerman 04f6974d2c ci: Dogfood Nix from master 2025-07-14 17:03:10 +03:00
Eelco Dolstra e8314e69ab Merge pull request #13460 from m4dc4p/m4dc4p/handle-expired-tokens
Better Handling for Expired Credentials
2025-07-14 11:51:30 +02:00
Eelco Dolstra 03bf582be3 Merge pull request #13468 from xokdvium/darwin-sandbox-sockets
libstore: fix Unix sockets in the build directory on sandboxed macOS
2025-07-14 11:13:36 +02:00
Pol Dellaiera 3e9a100bdf docker: set default parameters values 2025-07-13 22:49:12 +02:00
Emily 5cd94436f5 libstore: fix Unix sockets in the build directory on sandboxed macOS
We’re already allowing `/tmp` anyway, so this should be harmless,
and it fixes a regression in the default configuration caused by
moving the build directories out of `temp-dir`. (For instance, that
broke the Lix `guessOrInventPath.sockets` test.)

Note that removing `/tmp` breaks quite a few builds, so although it may
be a good idea in general it would require work on the Nixpkgs side.

Fixes: 749afbbe99fd7b45f828b72628252feba9241362
Change-Id: I6a6a69645f429bc50d4cb24283feda3d3091f534

(This is a cherry-pick of commit d1db3e5fa3faa43b3d2f2e2e843e9cfc1e6e1b71)

Lix patch: https://gerrit.lix.systems/c/lix/+/3500
2025-07-13 19:19:38 +03:00
m4dc4pandEelco Dolstra f786c0b8d1 Update src/libstore/s3-binary-cache-store.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-07-12 08:06:09 -07:00
m4dc4pandEelco Dolstra 22d6969d66 Update src/libstore/s3-binary-cache-store.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-07-12 08:05:52 -07:00
Eelco Dolstra b124512388 Merge pull request #13447 from de11n/fix-gc-docs
Fix documentation for GC w.r.t. symlinks
2025-07-12 09:47:39 +02:00
Justin Bailey a48632f2e0 Better Handling for Expired Credentials
When AWS credentials expired, in some scenarios they led to the
nix process aborting with an error similar to ' Unable to parse ExceptionName: ExpiredToken'.

This change updates the S3 handling code such that those errors are treated like 403s or 404s.

Closes #13459
2025-07-11 17:30:47 -07:00
Sergei Zimmerman ab3cd76e73 Merge pull request #13455 from DeterminateSystems/gustavderdrache/fix-sandbox-ifdef
Address ifdef problem with macOS/BSD sandboxing
2025-07-12 02:09:14 +03:00
gustavderdrache 8e5814d972 CI: Roll nix version to 2.29.1
This works around the macOS issue that the prior commit addresses.
2025-07-11 18:46:01 -04:00
gustavderdrache e2ef2cfcbc Address ifdef problem with macOS/BSD sandboxing 2025-07-11 18:09:07 -04:00
John Ericson b196922bcc Merge pull request #13450 from xokdvium/formal-call-unreachable
libexpr: Fix invalid handling of errors for imported functions
2025-07-11 14:26:07 -04:00
Sergei Zimmerman 6e78cc90d3 libexpr: Fix invalid handling of errors for imported functions
c39cc00404 has added assertions for
all Value accesses and the following case has started failing with
an `unreachable`:

(/tmp/fun.nix):

```nix
{a}: a
```

```
$ nix eval --impure --expr 'import /tmp/fun.nix {a="a";b="b";}'
```

This would crash:

```
terminating due to unexpected unrecoverable internal error: Unexpected condition in getStorage at ../include/nix/expr/value.hh:844
```

This is not a regression, but rather surfaces an existing problem, which previously
was left undiagnosed. In the case of an import `fun` is the `import` primOp, so that read is invalid
and previously this resulted in an access into an inactive union member, which is UB.
The correct thing to use is `vCur`. Identical problem also affected the case of a missing argument.

Add previously failing test cases to the functional/lang test suite.

Fixes #13448.
2025-07-11 20:20:48 +03:00
Elliot Cameron a17f377f69 Fix documentation for GC w.r.t. symlinks 2025-07-10 16:19:43 -04:00
Eelco Dolstra a783468333 Merge pull request #13444 from roberth/fix-lessThan-doc
Fix `builtins.lessThan` doc
2025-07-10 20:32:47 +02:00
Eelco Dolstra e82ad5d9e2 Merge pull request #13441 from NixOS/fix-fetch-closure-warning
fetchClosure: Fix gcc warning
2025-07-10 20:32:10 +02:00
Robert Hensing fc03b89ff4 Fix lessThan doc 2025-07-10 17:50:43 +02:00
Eelco Dolstra ca9f2028b0 Simplify SymbolTable::create() 2025-07-10 12:28:34 +02:00
Eelco Dolstra 74a144ce98 Require Boost 1.81.0 or higher
Note: this version of Boost was released in December 2022.
2025-07-10 11:53:36 +02:00
Eelco Dolstra aa18dc54dc fetchClosure: Fix gcc warning
Fixes:

[261/394] Linking target src/libexpr/libnixexpr.so
In function ‘copy’,
    inlined from ‘__ct ’ at /nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/basic_string.h:688:23,
    inlined from ‘operator+’ at /nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/basic_string.h:3735:43,
    inlined from ‘operator()’ at ../src/libexpr/primops/fetchClosure.cc:127:58,
    inlined from ‘prim_fetchClosure’ at ../src/libexpr/primops/fetchClosure.cc:132:88:
/nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/char_traits.h:427:56: warning: ‘__builtin_memcpy’ writing 74 bytes into a region of size 16 overflows the destination [-Wstringop-overflow=]
  427 |         return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n));
      |                                                        ^
../src/libexpr/primops/fetchClosure.cc: In function ‘prim_fetchClosure’:
../src/libexpr/primops/fetchClosure.cc:132:88: note: at offset 16 into destination object ‘<anonymous>’ of size 32
  132 |             fromPath = state.coerceToStorePath(attr.pos, *attr.value, context, attrHint());
      |                                                                                        ^
2025-07-10 11:41:32 +02:00
tomberek 485cbfc7f1 Merge pull request #13437 from NixOS/respect-transitive-locks
lockFlake(): When updating a lock, respect the input's lock file
2025-07-10 00:28:55 -04:00
Sergei Zimmerman b19e9acc03 Merge pull request #13402 from DavHau/build-cores
build-cores: detect cores automatically if set to 0
2025-07-09 23:06:55 +03:00
Eelco Dolstra 95437b90fc lockFlake(): When updating a lock, respect the input's lock file 2025-07-09 19:19:15 +02:00
Eelco Dolstra 3cfecd9a80 Merge pull request #13435 from h0nIg/patch-4
docker: fix nixConf
2025-07-09 11:16:55 +02:00
h0nIg 9857c0bb52 docker: fix nixConf - fmt 2025-07-09 09:34:50 +02:00
h0nIg 8a1f471b66 docker: fix nixConf 2025-07-09 09:30:11 +02:00
John Ericson 7caac53ae4 Merge pull request #13433 from ConnorBaker/patch-1
globals.hh: fix broken link to nspawn example
2025-07-08 16:43:40 -04:00
Connor Baker a16491375a globals.hh: fix broken link to nspawn example
The substitution included the `.` at the end of the URL, breaking it.
2025-07-08 12:59:16 -07:00
Eelco Dolstra 98a883b51a Merge pull request #13427 from jsoo1/remove-p
installers, tests: remove --preserve=mode from cp invocations
2025-07-08 17:09:42 +02:00
Eelco Dolstra 6d0194b17f Merge pull request #13429 from NixOS/bump-2.31.0
Bump version to 2.31.0
2025-07-08 17:03:02 +02:00
Eelco Dolstra 06665e27f4 Update .mergify.yml 2025-07-08 16:18:53 +02:00
Eelco Dolstra 4fa99d743e release-process.md: Remove unnecessary step 2025-07-08 16:15:45 +02:00
Eelco Dolstra 723903da3c Bump version 2025-07-08 16:14:50 +02:00
John Soo 87299e466d installers, tests: remove --preserve=mode from cp invocations
-p preserves xattrs and acls which can be incompatible between
filesystems

Unfortunately keep -p on darwin because the bsd coreutils do not
support --preserve.

Fixes #13426
2025-07-07 15:34:18 -06:00
John Ericson cf0df2607d Merge pull request #13240 from obsidiansystems/dyn-drv-take-3
Get rid of `addWantedOutputs`
2025-07-07 11:39:45 -04:00
Eelco Dolstra 7781560724 libstore-tests: Fix impurity trying to access the Nix daemon
This failed on macOS:

  nix-store-tests-run> C++ exception with description "../nix_api_store.cc:33: nix_err_code(ctx) != NIX_OK, message: error: getting status of '/nix/var/nix/daemon-socket/socket': Operation not permitted" thrown in the test body.
2025-07-07 11:42:20 +02:00
Eelco Dolstra f29acd5bbc Use writableTmpDirAsHomeHook 2025-07-07 11:12:55 +02:00
Eelco Dolstra 01388b3e78 Give unit tests access to a $HOME directory
Also, don't try to access cache.nixos.org in the libstore unit tests.
2025-07-07 11:02:51 +02:00
DavHau fb5e22e318 build-cores: detect cores automatically if set to 0
This changes makes nix detect a machines available cores automatically whenever build-cores is set to 0.

So far, nix simply passed NIX_BUILD_CORES=0 whenever build-cores is set to 0. (only when build-cores is unset it was detecting cores automatically)

The behavior of passing NIX_BUILD_CORES=0 leads to a performance penalty when sourcing nixpkgs' generic builder's `setup.sh`, as setup.sh has to execute `nproc`. This significantly slows down sourcing of setup.sh
2025-07-07 10:33:42 +07:00
John EricsonandSergei Zimmerman a55806a0dd Get rid of addWantedOutputs
This is just a code cleanup; it should not be behavior change.

`addWantedOutputs` is removed by introducing `DerivationTrampolineGoal`.

`DerivationGoal` now only tracks a single output, and is back to
tracking a plain store path `drvPath`, not a deriving path one. Its
`addWantedOutputs` method is gone. These changes will allow subsequent
PRs to simplify it greatly.

Because the purpose of each goal is back to being immutable, we can also
once again make `Goal::buildResult` a public field, and get rid of the
`getBuildResult` method. This simplifies things also.

`DerivationTrampolineGoal` is, as the nane is supposed to indicate, a
cheap "trampoline" goal. It takes immutable sets of wanted outputs, and
just kicks of `DerivationGoal`s for them. Since now "actual work" is
done in these goals, it is not wasteful to have separate ones for
separate sets of outputs, even if those outputs (and the derivations
they are from) overlap.

This design is described in more detail in the doc comments on the goal
types, which I've now greatly expanded.

---

This separation of concerns will make it possible for future work on
issues like #11928, and to continue the path of having more goal types,
but each goal type does fewer things (issue #12628).

---

This commit in some sense reverts
f4f28cdd0e, but that one kept around
`addWantedOutputs`. I am quite sure it was having two layers of goals
with `addWantedOutputs` that caused the issues --- restarting logic like
`addWantedOutputs` has is very tempermental! In this version of the
change, we have *zero* layers of `addWantedOutputs` --- no goal type
needs it, or otherwise has a mutable objective --- and so I think this
change is safe.

Co-authored-by: Sergei Zimmerman <145775305+xokdvium@users.noreply.github.com>
2025-06-20 18:43:35 -04:00
John Ericson 47b7d910e9 Properly prefix libfetchers header
This is a minor oversight from 012453d1e6.
2025-06-20 18:42:23 -04:00
317 changed files with 5449 additions and 3619 deletions

No files matched your search

+3 -3
View File
@@ -1,4 +1,4 @@
# bulk initial re-formatting with clang-format
95d9c13716e0000f46f5279367fdecb5b4545923 # !autorebase ./maintainers/format.sh --until-stable
# nixfmt 1.0.0
7518fdc1f8f9468b3c3c310cbf4bfebb7d848692 # !autorebase ./maintainers/format.sh --until-stable
e4f62e46088919428a68bd8014201dc8e379fed7 # !autorebase ./maintainers/format.sh --until-stable
# meson re-formatting
385e2c3542c707d95e3784f7f6d623f67e77ab61 # !autorebase ./maintainers/format.sh --until-stable
+1 -1
View File
@@ -45,7 +45,7 @@ assignees: ''
- [ ] checked [latest Nix manual] \([source])
- [ ] checked [open bug issues and pull requests] for possible duplicates
[latest Nix manual]: https://nixos.org/manual/nix/unstable/
[latest Nix manual]: https://nix.dev/manual/nix/development/
[source]: https://github.com/NixOS/nix/tree/master/doc/manual/source
[open bug issues and pull requests]: https://github.com/NixOS/nix/labels/bug
+1 -1
View File
@@ -30,7 +30,7 @@ assignees: ''
- [ ] checked [latest Nix manual] \([source])
- [ ] checked [open feature issues and pull requests] for possible duplicates
[latest Nix manual]: https://nixos.org/manual/nix/unstable/
[latest Nix manual]: https://nix.dev/manual/nix/development/
[source]: https://github.com/NixOS/nix/tree/master/doc/manual/source
[open feature issues and pull requests]: https://github.com/NixOS/nix/labels/feature
+1 -1
View File
@@ -38,7 +38,7 @@ assignees: ''
- [ ] checked [latest Nix manual] \([source])
- [ ] checked [open installer issues and pull requests] for possible duplicates
[latest Nix manual]: https://nixos.org/manual/nix/unstable/
[latest Nix manual]: https://nix.dev/manual/nix/development/
[source]: https://github.com/NixOS/nix/tree/master/doc/manual/source
[open installer issues and pull requests]: https://github.com/NixOS/nix/labels/installer
@@ -22,7 +22,7 @@ assignees: ''
- [ ] checked [latest Nix manual] \([source])
- [ ] checked [open documentation issues and pull requests] for possible duplicates
[latest Nix manual]: https://nixos.org/manual/nix/unstable/
[latest Nix manual]: https://nix.dev/manual/nix/development/
[source]: https://github.com/NixOS/nix/tree/master/doc/manual/source
[open documentation issues and pull requests]: https://github.com/NixOS/nix/labels/documentation
+1 -75
View File
@@ -4,29 +4,15 @@ inputs:
dogfood:
description: "Whether to use Nix installed from the latest artifact from master branch"
required: true # Be explicit about the fact that we are using unreleased artifacts
experimental-installer:
description: "Whether to use the experimental installer to install Nix"
default: false
experimental-installer-version:
description: "Version of the experimental installer to use. If `latest`, the newest artifact from the default branch is used."
# TODO: This should probably be pinned to a release after https://github.com/NixOS/experimental-nix-installer/pull/49 lands in one
default: "latest"
extra_nix_config:
description: "Gets appended to `/etc/nix/nix.conf` if passed."
install_url:
description: "URL of the Nix installer"
required: false
default: "https://releases.nixos.org/nix/nix-2.29.1/install"
tarball_url:
description: "URL of the Nix tarball to use with the experimental installer"
required: false
default: "https://releases.nixos.org/nix/nix-2.30.1/install"
github_token:
description: "Github token"
required: true
use_cache:
description: "Whether to setup github actions cache (not implemented currently)"
default: false
required: false
runs:
using: "composite"
steps:
@@ -51,74 +37,14 @@ runs:
gh run download "$RUN_ID" --repo "$DOGFOOD_REPO" -n "$INSTALLER_ARTIFACT" -D "$INSTALLER_DOWNLOAD_DIR"
echo "installer-path=file://$INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$INSTALLER_DOWNLOAD_DIR" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
echo "::notice ::Dogfooding Nix installer from master (https://github.com/$DOGFOOD_REPO/actions/runs/$RUN_ID)"
env:
GH_TOKEN: ${{ inputs.github_token }}
DOGFOOD_REPO: "NixOS/nix"
- name: "Gather system info for experimental installer"
shell: bash
if: ${{ inputs.experimental-installer == 'true' }}
run: |
echo "::notice Using experimental installer from $EXPERIMENTAL_INSTALLER_REPO (https://github.com/$EXPERIMENTAL_INSTALLER_REPO)"
if [ "$RUNNER_OS" == "Linux" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="linux"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
elif [ "$RUNNER_OS" == "macOS" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="darwin"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_OS: $RUNNER_OS"
exit 1
fi
if [ "$RUNNER_ARCH" == "X64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=x86_64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
elif [ "$RUNNER_ARCH" == "ARM64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=aarch64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_ARCH: $RUNNER_ARCH"
exit 1
fi
echo "EXPERIMENTAL_INSTALLER_ARTIFACT=nix-installer-$EXPERIMENTAL_INSTALLER_ARCH-$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
env:
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- name: "Download latest experimental installer"
shell: bash
id: download-latest-experimental-installer
if: ${{ inputs.experimental-installer == 'true' && inputs.experimental-installer-version == 'latest' }}
run: |
RUN_ID=$(gh run list --repo "$EXPERIMENTAL_INSTALLER_REPO" --workflow ci.yml --branch main --status success --json databaseId --jq ".[0].databaseId")
EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR="$GITHUB_WORKSPACE/$EXPERIMENTAL_INSTALLER_ARTIFACT"
mkdir -p "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
gh run download "$RUN_ID" --repo "$EXPERIMENTAL_INSTALLER_REPO" -n "$EXPERIMENTAL_INSTALLER_ARTIFACT" -D "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
# Executable permissions are lost in artifacts
find $EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR -type f -exec chmod +x {} +
echo "installer-path=$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
env:
GH_TOKEN: ${{ inputs.github_token }}
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- uses: cachix/install-nix-action@c134e4c9e34bac6cab09cf239815f9339aaaf84e # v31.5.1
if: ${{ inputs.experimental-installer != 'true' }}
with:
# Ternary operator in GHA: https://www.github.com/actions/runner/issues/409#issuecomment-752775072
install_url: ${{ inputs.dogfood == 'true' && format('{0}/install', steps.download-nix-installer.outputs.installer-path) || inputs.install_url }}
install_options: ${{ inputs.dogfood == 'true' && format('--tarball-url-prefix {0}', steps.download-nix-installer.outputs.installer-path) || '' }}
extra_nix_config: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/nix-installer-action@786fff0690178f1234e4e1fe9b536e94f5433196 # v20
if: ${{ inputs.experimental-installer == 'true' }}
with:
diagnostic-endpoint: ""
# TODO: It'd be nice to use `artifacts.nixos.org` for both of these, maybe through an `/experimental-installer/latest` endpoint? or `/commit/<hash>`?
local-root: ${{ inputs.experimental-installer-version == 'latest' && steps.download-latest-experimental-installer.outputs.installer-path || '' }}
source-url: ${{ inputs.experimental-installer-version != 'latest' && 'https://artifacts.nixos.org/experimental-installer/tag/${{ inputs.experimental-installer-version }}/${{ env.EXPERIMENTAL_INSTALLER_ARTIFACT }}' || '' }}
nix-package-url: ${{ inputs.dogfood == 'true' && steps.download-nix-installer.outputs.tarball-path || (inputs.tarball_url || '') }}
extra-conf: ${{ inputs.extra_nix_config }}
+145 -22
View File
@@ -15,11 +15,10 @@ jobs:
fetch-depth: 0
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
dogfood: true
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
use_cache: false
- run: nix flake show --all-systems --json
tests:
@@ -30,9 +29,21 @@ jobs:
- scenario: on ubuntu
runs-on: ubuntu-24.04
os: linux
instrumented: false
primary: true
stdenv: stdenv
- scenario: on macos
runs-on: macos-14
os: darwin
instrumented: false
primary: true
stdenv: stdenv
- scenario: on ubuntu (with sanitizers / coverage)
runs-on: ubuntu-24.04
os: linux
instrumented: true
primary: false
stdenv: clangStdenv
name: tests ${{ matrix.scenario }}
runs-on: ${{ matrix.runs-on }}
timeout-minutes: 60
@@ -43,22 +54,46 @@ jobs:
- uses: ./.github/actions/install-nix-action
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
dogfood: false
dogfood: true
# The sandbox would otherwise be disabled by default on Darwin
extra_nix_config: |
sandbox = true
max-jobs = 1
- uses: DeterminateSystems/magic-nix-cache-action@main
# Since ubuntu 22.30, unprivileged usernamespaces are no longer allowed to map to the root user:
# https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
- run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
if: matrix.os == 'linux'
- run: scripts/build-checks
- run: scripts/prepare-installer-for-github-actions
- name: Run component tests
run: |
nix build --file ci/gha/tests/wrapper.nix componentTests -L \
--arg withInstrumentation ${{ matrix.instrumented }} \
--argstr stdenv "${{ matrix.stdenv }}"
- name: Run flake checks and prepare the installer tarball
run: |
ci/gha/tests/build-checks
ci/gha/tests/prepare-installer-for-github-actions
if: ${{ matrix.primary }}
- name: Collect code coverage
run: |
nix build --file ci/gha/tests/wrapper.nix codeCoverage.coverageReports -L \
--arg withInstrumentation ${{ matrix.instrumented }} \
--argstr stdenv "${{ matrix.stdenv }}" \
--out-link coverage-reports
cat coverage-reports/index.txt >> $GITHUB_STEP_SUMMARY
if: ${{ matrix.instrumented }}
- name: Upload coverage reports
uses: actions/upload-artifact@v4
with:
name: coverage-reports
path: coverage-reports/
if: ${{ matrix.instrumented }}
- name: Upload installer tarball
uses: actions/upload-artifact@v4
with:
name: installer-${{matrix.os}}
path: out/*
if: ${{ matrix.primary }}
installer_test:
needs: [tests]
@@ -81,13 +116,13 @@ jobs:
with:
name: installer-${{matrix.os}}
path: out
- name: Serving installer
id: serving_installer
run: ./scripts/serve-installer-for-github-actions
- name: Looking up the installer tarball URL
id: installer-tarball-url
run: echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
- uses: cachix/install-nix-action@v31
with:
install_url: 'http://localhost:8126/install'
install_options: "--tarball-url-prefix http://localhost:8126/"
install_url: ${{ format('{0}/install', steps.installer-tarball-url.outputs.installer-url) }}
install_options: ${{ format('--tarball-url-prefix {0}', steps.installer-tarball-url.outputs.installer-url) }}
- run: sudo apt install fish zsh
if: matrix.os == 'linux'
- run: brew install fish
@@ -99,20 +134,84 @@ jobs:
- run: exec bash -c "nix-channel --add https://releases.nixos.org/nixos/unstable/nixos-23.05pre466020.60c1d71f2ba nixpkgs"
- run: exec bash -c "nix-channel --update && nix-env -iA nixpkgs.hello && hello"
# Steps to test CI automation in your own fork.
# 1. Sign-up for https://hub.docker.com/
# 2. Store your dockerhub username as DOCKERHUB_USERNAME in "Repository secrets" of your fork repository settings (https://github.com/$githubuser/nix/settings/secrets/actions)
# 3. Create an access token in https://hub.docker.com/settings/security and store it as DOCKERHUB_TOKEN in "Repository secrets" of your fork
check_secrets:
permissions:
contents: none
name: Check Docker secrets present for installer tests
runs-on: ubuntu-24.04
outputs:
docker: ${{ steps.secret.outputs.docker }}
steps:
- name: Check for secrets
id: secret
env:
_DOCKER_SECRETS: ${{ secrets.DOCKERHUB_USERNAME }}${{ secrets.DOCKERHUB_TOKEN }}
run: |
echo "::set-output name=docker::${{ env._DOCKER_SECRETS != '' }}"
docker_push_image:
name: Push docker image to DockerHub and GHCR
needs: [tests, vm_tests]
if: github.event_name == 'push' && github.ref_name == 'master'
uses: ./.github/workflows/docker-push.yml
with:
ref: ${{ github.sha }}
is_master: true
needs: [tests, vm_tests, check_secrets]
permissions:
contents: read
packages: write
secrets:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
if: >-
needs.check_secrets.outputs.docker == 'true' &&
github.event_name == 'push' &&
github.ref_name == 'master'
runs-on: ubuntu-24.04
steps:
- name: Check for secrets
id: secret
env:
_DOCKER_SECRETS: ${{ secrets.DOCKERHUB_USERNAME }}${{ secrets.DOCKERHUB_TOKEN }}
run: |
echo "::set-output name=docker::${{ env._DOCKER_SECRETS != '' }}"
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v31
with:
install_url: https://releases.nixos.org/nix/nix-2.20.3/install
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: echo NIX_VERSION="$(nix --experimental-features 'nix-command flakes' eval .\#nix.version | tr -d \")" >> $GITHUB_ENV
- run: nix --experimental-features 'nix-command flakes' build .#dockerImage -L
- run: docker load -i ./result/image.tar.gz
- run: docker tag nix:$NIX_VERSION ${{ secrets.DOCKERHUB_USERNAME }}/nix:$NIX_VERSION
- run: docker tag nix:$NIX_VERSION ${{ secrets.DOCKERHUB_USERNAME }}/nix:master
# We'll deploy the newly built image to both Docker Hub and Github Container Registry.
#
# Push to Docker Hub first
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- run: docker push ${{ secrets.DOCKERHUB_USERNAME }}/nix:$NIX_VERSION
- run: docker push ${{ secrets.DOCKERHUB_USERNAME }}/nix:master
# Push to GitHub Container Registry as well
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Push image
run: |
IMAGE_ID=ghcr.io/${{ github.repository_owner }}/nix
# Change all uppercase to lowercase
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
docker tag nix:$NIX_VERSION $IMAGE_ID:$NIX_VERSION
docker tag nix:$NIX_VERSION $IMAGE_ID:latest
docker push $IMAGE_ID:$NIX_VERSION
docker push $IMAGE_ID:latest
# deprecated 2024-02-24
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
vm_tests:
runs-on: ubuntu-24.04
@@ -120,7 +219,7 @@ jobs:
- uses: actions/checkout@v4
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
dogfood: true
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
@@ -151,8 +250,32 @@ jobs:
path: flake-regressions/tests
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
dogfood: true
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: nix build -L --out-link ./new-nix && PATH=$(pwd)/new-nix/bin:$PATH MAX_FLAKES=25 flake-regressions/eval-all.sh
profile_build:
needs: tests
runs-on: ubuntu-24.04
timeout-minutes: 60
if: >-
github.event_name == 'push' &&
github.ref_name == 'master'
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: ./.github/actions/install-nix-action
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
dogfood: true
extra_nix_config: |
experimental-features = flakes nix-command ca-derivations impure-derivations
max-jobs = 1
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: |
nix build -L --file ./ci/gha/profile-build buildTimeReport --out-link build-time-report.md
cat build-time-report.md >> $GITHUB_STEP_SUMMARY
-101
View File
@@ -1,101 +0,0 @@
name: "Push Docker Image"
on:
workflow_call:
inputs:
ref:
description: "Git ref to build the docker image from"
required: true
type: string
is_master:
description: "Whether run from master branch"
required: true
type: boolean
secrets:
DOCKERHUB_USERNAME:
required: true
DOCKERHUB_TOKEN:
required: true
permissions: {}
jobs:
# Steps to test CI automation in your own fork.
# 1. Sign-up for https://hub.docker.com/
# 2. Store your dockerhub username as DOCKERHUB_USERNAME in "Repository secrets" of your fork repository settings (https://github.com/$githubuser/nix/settings/secrets/actions)
# 3. Create an access token in https://hub.docker.com/settings/security and store it as DOCKERHUB_TOKEN in "Repository secrets" of your fork
check_secrets:
permissions:
contents: none
name: Check presence of secrets
runs-on: ubuntu-24.04
outputs:
docker: ${{ steps.secret.outputs.docker }}
steps:
- name: Check for DockerHub secrets
id: secret
env:
_DOCKER_SECRETS: ${{ secrets.DOCKERHUB_USERNAME }}${{ secrets.DOCKERHUB_TOKEN }}
run: |
echo "docker=${{ env._DOCKER_SECRETS != '' }}" >> $GITHUB_OUTPUT
push:
name: Push docker image to DockerHub and GHCR
needs: [check_secrets]
permissions:
contents: read
packages: write
if: needs.check_secrets.outputs.docker == 'true'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
fetch-depth: 0
ref: ${{ inputs.ref }}
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
extra_nix_config: |
experimental-features = flakes nix-command
- run: echo NIX_VERSION="$(nix eval .\#nix.version | tr -d \")" >> $GITHUB_ENV
- run: nix build .#dockerImage -L
- run: docker load -i ./result/image.tar.gz
# We'll deploy the newly built image to both Docker Hub and Github Container Registry.
#
# Push to Docker Hub first
- name: Login to Docker Hub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Push to Docker Hub
env:
IS_MASTER: ${{ inputs.is_master }}
DOCKERHUB_REPO: ${{ secrets.DOCKERHUB_USERNAME }}/nix
run: |
docker tag nix:$NIX_VERSION $DOCKERHUB_REPO:$NIX_VERSION
docker push $DOCKERHUB_REPO:$NIX_VERSION
if [ "$IS_MASTER" = "true" ]; then
docker tag nix:$NIX_VERSION $DOCKERHUB_REPO:master
docker push $DOCKERHUB_REPO:master
fi
# Push to GitHub Container Registry as well
- name: Login to GitHub Container Registry
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Push to GHCR
env:
IS_MASTER: ${{ inputs.is_master }}
run: |
IMAGE_ID=ghcr.io/${{ github.repository_owner }}/nix
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
docker tag nix:$NIX_VERSION $IMAGE_ID:$NIX_VERSION
docker push $IMAGE_ID:$NIX_VERSION
if [ "$IS_MASTER" = "true" ]; then
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
fi
-80
View File
@@ -1,80 +0,0 @@
name: Upload Release
on:
workflow_dispatch:
inputs:
eval_id:
description: "Hydra evaluation ID"
required: true
type: number
is_latest:
description: "Mark as latest release"
required: false
type: boolean
default: false
permissions:
contents: read
id-token: write
packages: write
jobs:
release:
runs-on: ubuntu-24.04
environment: releases
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- uses: ./.github/actions/install-nix-action
with:
dogfood: false # Use stable version
use_cache: false # Don't want any cache injection shenanigans
extra_nix_config: |
experimental-features = nix-command flakes
- name: Set NIX_PATH from flake input
run: |
NIXPKGS_PATH=$(nix build --inputs-from .# nixpkgs#path --print-out-paths --no-link)
# Shebangs with perl have issues. Pin nixpkgs this way. nix shell should maybe
# get the same uberhack that nix-shell has to support it.
echo "NIX_PATH=nixpkgs=$NIXPKGS_PATH" >> "$GITHUB_ENV"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708 # v5.1.1
with:
role-to-assume: "arn:aws:iam::080433136561:role/nix-release"
role-session-name: nix-release-oidc-${{ github.run_id }}
aws-region: eu-west-1
- name: Disable containerd image store
run: |
# Docker 28+ defaults to the containerd image store, which
# pushes layers uncompressed instead of gzip. OCI clients
# that only support gzip (e.g. go-containerregistry) fail
# with "gzip: invalid header". Disabling the containerd
# snapshotter restores the classic storage driver, which
# preserves gzip-compressed layers through the
# `docker load` / `docker push` pipeline.
echo '{"features":{"containerd-snapshotter":false}}' | sudo tee /etc/docker/daemon.json > /dev/null
sudo systemctl restart docker
- name: Login to Docker Hub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Upload release
run: |
./maintainers/upload-release.pl \
${{ inputs.eval_id }} \
--skip-git
env:
IS_LATEST: ${{ inputs.is_latest && '1' || '' }}
- name: Push to GHCR
run: |
DOCKER_OWNER="ghcr.io/$(echo '${{ github.repository_owner }}' | tr '[A-Z]' '[a-z]')/nix"
./maintainers/upload-release.pl \
${{ inputs.eval_id }} \
--skip-git \
--skip-s3 \
--docker-owner "$DOCKER_OWNER"
env:
IS_LATEST: ${{ inputs.is_latest && '1' || '' }}
+11
View File
@@ -150,3 +150,14 @@ pull_request_rules:
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.30
conditions:
- label=backport 2.30-maintenance
actions:
backport:
branches:
- "2.30-maintenance"
labels:
- automatic backport
- merge-queue
+1 -1
View File
@@ -1 +1 @@
2.30.4
2.31.0
+1 -1
View File
@@ -89,7 +89,7 @@ Check out the [security policy](https://github.com/NixOS/nix/security/policy).
## Making changes to the Nix manual
The Nix reference manual is hosted on https://nixos.org/manual/nix.
The Nix reference manual is hosted on https://nix.dev/manual/nix.
The underlying source files are located in [`doc/manual/source`](./doc/manual/source).
For small changes you can [use GitHub to edit these files](https://docs.github.com/en/repositories/working-with-files/managing-files/editing-files)
For larger changes see the [Nix reference manual](https://nix.dev/manual/nix/development/development/contributing.html).
+101
View File
@@ -0,0 +1,101 @@
{
nixFlake ? builtins.getFlake ("git+file://" + toString ../../..),
system ? builtins.currentSystem,
pkgs ? nixFlake.inputs.nixpkgs.legacyPackages.${system},
}:
let
inherit (pkgs) lib;
nixComponentsInstrumented =
(nixFlake.lib.makeComponents {
inherit pkgs;
getStdenv = p: p.clangStdenv;
}).overrideScope
(
_: _: {
mesonComponentOverrides = finalAttrs: prevAttrs: {
outputs = (prevAttrs.outputs or [ "out" ]) ++ [ "buildprofile" ];
nativeBuildInputs = [ pkgs.clangbuildanalyzer ] ++ prevAttrs.nativeBuildInputs or [ ];
__impure = true;
env = {
CFLAGS = "-ftime-trace";
CXXFLAGS = "-ftime-trace";
};
preBuild = ''
ClangBuildAnalyzer --start $PWD
'';
postBuild = ''
ClangBuildAnalyzer --stop $PWD $buildprofile
'';
};
}
);
componentsToProfile = {
"nix-util" = { };
"nix-util-c" = { };
"nix-util-test-support" = { };
"nix-util-tests" = { };
"nix-store" = { };
"nix-store-c" = { };
"nix-store-test-support" = { };
"nix-store-tests" = { };
"nix-fetchers" = { };
"nix-fetchers-c" = { };
"nix-fetchers-tests" = { };
"nix-expr" = { };
"nix-expr-c" = { };
"nix-expr-test-support" = { };
"nix-expr-tests" = { };
"nix-flake" = { };
"nix-flake-c" = { };
"nix-flake-tests" = { };
"nix-main" = { };
"nix-main-c" = { };
"nix-cmd" = { };
"nix-cli" = { };
};
componentDerivationsToProfile = builtins.intersectAttrs componentsToProfile nixComponentsInstrumented;
componentBuildProfiles = lib.mapAttrs (
n: v: lib.getOutput "buildprofile" v
) componentDerivationsToProfile;
buildTimeReport =
pkgs.runCommand "build-time-report"
{
__impure = true;
__structuredAttrs = true;
nativeBuildInputs = [ pkgs.clangbuildanalyzer ];
inherit componentBuildProfiles;
}
''
{
echo "# Build time performance profile for components:"
echo
echo "This reports the build profile collected via \`-ftime-trace\` for each component."
echo
} >> $out
for name in "''\${!componentBuildProfiles[@]}"; do
{
echo "<details><summary><strong>$name</strong></summary>"
echo
echo '````'
ClangBuildAnalyzer --analyze "''\${componentBuildProfiles[$name]}"
echo '````'
echo
echo "</details>"
} >> $out
done
'';
in
{
inherit buildTimeReport;
inherit componentDerivationsToProfile;
}
File renamed without changes.
+211
View File
@@ -0,0 +1,211 @@
{
nixFlake ? builtins.getFlake ("git+file://" + toString ../../..),
system ? builtins.currentSystem,
pkgs ? nixFlake.inputs.nixpkgs.legacyPackages.${system},
getStdenv ? p: p.stdenv,
componentTestsPrefix ? "",
withSanitizers ? false,
withCoverage ? false,
...
}:
let
inherit (pkgs) lib;
hydraJobs = nixFlake.hydraJobs;
packages' = nixFlake.packages.${system};
stdenv = (getStdenv pkgs);
enableSanitizersLayer = finalAttrs: prevAttrs: {
mesonFlags =
(prevAttrs.mesonFlags or [ ])
++ [
# Run all tests with UBSAN enabled. Running both with ubsan and
# without doesn't seem to have much immediate benefit for doubling
# the GHA CI workaround.
#
# TODO: Work toward enabling "address,undefined" if it seems feasible.
# This would maybe require dropping Boost coroutines and ignoring intentional
# memory leaks with detect_leaks=0.
(lib.mesonOption "b_sanitize" "undefined")
]
++ (lib.optionals stdenv.cc.isClang [
# https://www.github.com/mesonbuild/meson/issues/764
(lib.mesonBool "b_lundef" false)
]);
};
collectCoverageLayer = finalAttrs: prevAttrs: {
env =
let
# https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#the-code-coverage-workflow
coverageFlags = [
"-fprofile-instr-generate"
"-fcoverage-mapping"
];
in
{
CFLAGS = toString coverageFlags;
CXXFLAGS = toString coverageFlags;
};
# Done in a pre-configure hook, because $NIX_BUILD_TOP needs to be substituted.
preConfigure =
prevAttrs.preConfigure or ""
+ ''
mappingFlag=" -fcoverage-prefix-map=$NIX_BUILD_TOP/${finalAttrs.src.name}=${finalAttrs.src}"
CFLAGS+="$mappingFlag"
CXXFLAGS+="$mappingFlag"
'';
};
componentOverrides =
(lib.optional withSanitizers enableSanitizersLayer)
++ (lib.optional withCoverage collectCoverageLayer);
in
rec {
nixComponents =
(nixFlake.lib.makeComponents {
inherit pkgs;
inherit getStdenv;
}).overrideScope
(
final: prev: {
nix-store-tests = prev.nix-store-tests.override { withBenchmarks = true; };
mesonComponentOverrides = lib.composeManyExtensions componentOverrides;
}
);
/**
Top-level tests for the flake outputs, as they would be built by hydra.
These tests generally can't be overridden to run with sanitizers.
*/
topLevel = {
installerScriptForGHA = hydraJobs.installerScriptForGHA.${system};
installTests = hydraJobs.installTests.${system};
nixpkgsLibTests = hydraJobs.tests.nixpkgsLibTests.${system};
rl-next = pkgs.buildPackages.runCommand "test-rl-next-release-notes" { } ''
LANG=C.UTF-8 ${pkgs.changelog-d}/bin/changelog-d ${../../../doc/manual/rl-next} >$out
'';
repl-completion = pkgs.callPackage ../../../tests/repl-completion.nix { inherit (packages') nix; };
/**
Checks for our packaging expressions.
This shouldn't build anything significant; just check that things
(including derivations) are _set up_ correctly.
*/
packaging-overriding =
let
nix = packages'.nix;
in
assert (nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src.patches == [ pkgs.emptyFile ];
if pkgs.stdenv.buildPlatform.isDarwin then
lib.warn "packaging-overriding check currently disabled because of a permissions issue on macOS" pkgs.emptyFile
else
# If this fails, something might be wrong with how we've wired the scope,
# or something could be broken in Nixpkgs.
pkgs.testers.testEqualContents {
assertion = "trivial patch does not change source contents";
expected = "${../../..}";
actual =
# Same for all components; nix-util is an arbitrary pick
(nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src;
};
};
componentTests =
(lib.concatMapAttrs (
pkgName: pkg:
lib.concatMapAttrs (testName: test: {
"${componentTestsPrefix}${pkgName}-${testName}" = test;
}) (pkg.tests or { })
) nixComponents)
// lib.optionalAttrs (pkgs.stdenv.hostPlatform == pkgs.stdenv.buildPlatform) {
"${componentTestsPrefix}nix-functional-tests" = nixComponents.nix-functional-tests;
};
codeCoverage =
let
componentsTestsToProfile =
(builtins.mapAttrs (n: v: nixComponents.${n}.tests.run) {
"nix-util-tests" = { };
"nix-store-tests" = { };
"nix-fetchers-tests" = { };
"nix-expr-tests" = { };
"nix-flake-tests" = { };
})
// {
inherit (nixComponents) nix-functional-tests;
};
coverageProfileDrvs = lib.mapAttrs (
n: v:
v.overrideAttrs (
finalAttrs: prevAttrs: {
outputs = (prevAttrs.outputs or [ "out" ]) ++ [ "profraw" ];
env = {
LLVM_PROFILE_FILE = "${placeholder "profraw"}/%m";
};
}
)
) componentsTestsToProfile;
coverageProfiles = lib.mapAttrsToList (n: v: lib.getOutput "profraw" v) coverageProfileDrvs;
mergedProfdata =
pkgs.runCommand "merged-profdata"
{
__structuredAttrs = true;
nativeBuildInputs = [ pkgs.llvmPackages.libllvm ];
inherit coverageProfiles;
}
''
rawProfiles=()
for dir in "''\${coverageProfiles[@]}"; do
rawProfiles+=($dir/*)
done
llvm-profdata merge -sparse -output $out "''\${rawProfiles[@]}"
'';
coverageReports =
let
nixComponentDrvs = lib.filter (lib.isDerivation) (lib.attrValues nixComponents);
in
pkgs.runCommand "code-coverage-report"
{
nativeBuildInputs = [
pkgs.llvmPackages.libllvm
];
__structuredAttrs = true;
nixComponents = nixComponentDrvs;
}
''
# ${toString (lib.map (v: v.src) nixComponentDrvs)}
binaryFiles=()
for dir in "''\${nixComponents[@]}"; do
readarray -t filesInDir < <(find "$dir" -type f -executable)
binaryFiles+=("''\${filesInDir[@]}")
done
arguments=$(concatStringsSep " -object " binaryFiles)
llvm-cov show $arguments -instr-profile ${mergedProfdata} -output-dir $out -format=html
{
echo "# Code coverage summary (generated via \`llvm-cov\`):"
echo
echo '```'
llvm-cov report $arguments -instr-profile ${mergedProfdata} -format=text -use-color=false
echo '```'
echo
} >> $out/index.txt
'';
in
assert withCoverage;
assert stdenv.cc.isClang;
{
inherit coverageProfileDrvs mergedProfdata coverageReports;
};
}
+16
View File
@@ -0,0 +1,16 @@
{
nixFlake ? builtins.getFlake ("git+file://" + toString ../../..),
system ? builtins.currentSystem,
pkgs ? nixFlake.inputs.nixpkgs.legacyPackages.${system},
stdenv ? "stdenv",
componentTestsPrefix ? "",
withInstrumentation ? false,
}@args:
import ./. (
args
// {
getStdenv = p: p.${stdenv};
withSanitizers = withInstrumentation;
withCoverage = withInstrumentation;
}
)
+3 -10
View File
@@ -24,15 +24,8 @@ def map_contents_recursively(transformer):
def process_command:
.[0] as $context |
.[1] as $body |
# mdbook 0.5.x uses 'items' instead of 'sections'
if $body.items then
$body + {
items: $body.items | map(map_contents_recursively(if $context.renderer == "html" then transform_anchors_html else transform_anchors_strip end)),
}
else
$body + {
sections: $body.sections | map(map_contents_recursively(if $context.renderer == "html" then transform_anchors_html else transform_anchors_strip end)),
}
end;
$body + {
sections: $body.sections | map(map_contents_recursively(if $context.renderer == "html" then transform_anchors_html else transform_anchors_strip end)),
};
process_command
+9
View File
@@ -23,3 +23,12 @@ renderers = ["html"]
command = "jq --from-file ./anchors.jq"
[output.markdown]
[output.linkcheck]
# no Internet during the build (in the sandbox)
follow-web-links = false
# mdbook-linkcheck does not understand [foo]{#bar} style links, resulting in
# excessive "Potential incomplete link" warnings. No other kind of warning was
# produced at the time of writing.
warning-policy = "ignore"
+24 -17
View File
@@ -1,4 +1,5 @@
project('nix-manual',
project(
'nix-manual',
version : files('.version'),
meson_version : '>= 1.1',
license : 'LGPL-2.1-or-later',
@@ -8,43 +9,45 @@ nix = find_program('nix', native : true)
mdbook = find_program('mdbook', native : true)
bash = find_program('bash', native : true)
rsync = find_program('rsync', required : true, native : true)
pymod = import('python')
python = pymod.find_installation('python3')
nix_env_for_docs = {
'HOME': '/dummy',
'NIX_CONF_DIR': '/dummy',
'NIX_SSL_CERT_FILE': '/dummy/no-ca-bundle.crt',
'NIX_STATE_DIR': '/dummy',
'NIX_CONFIG': 'cores = 0',
'HOME' : '/dummy',
'NIX_CONF_DIR' : '/dummy',
'NIX_SSL_CERT_FILE' : '/dummy/no-ca-bundle.crt',
'NIX_STATE_DIR' : '/dummy',
'NIX_CONFIG' : 'cores = 0',
}
nix_for_docs = [nix, '--experimental-features', 'nix-command']
nix_for_docs = [ nix, '--experimental-features', 'nix-command' ]
nix_eval_for_docs_common = nix_for_docs + [
'eval',
'-I', 'nix=' + meson.current_source_dir(),
'-I',
'nix=' + meson.current_source_dir(),
'--store', 'dummy://',
'--impure',
]
nix_eval_for_docs = nix_eval_for_docs_common + '--raw'
conf_file_json = custom_target(
command : nix_for_docs + ['config', 'show', '--json'],
command : nix_for_docs + [ 'config', 'show', '--json' ],
capture : true,
output : 'conf-file.json',
env : nix_env_for_docs,
)
language_json = custom_target(
command: [nix, '__dump-language'],
command : [ nix, '__dump-language' ],
output : 'language.json',
capture : true,
env : nix_env_for_docs,
)
nix3_cli_json = custom_target(
command : [nix, '__dump-cli'],
command : [ nix, '__dump-cli' ],
capture : true,
output : 'nix.json',
env : nix_env_for_docs,
@@ -78,13 +81,14 @@ manual = custom_target(
'manual',
command : [
bash,
'-euo', 'pipefail',
'-euo',
'pipefail',
'-c',
'''
@0@ @INPUT0@ @CURRENT_SOURCE_DIR@ > @DEPFILE@
@0@ @INPUT1@ summary @2@ < @CURRENT_SOURCE_DIR@/source/SUMMARY.md.in > @2@/source/SUMMARY.md
sed -e 's|@version@|@3@|g' < @INPUT2@ > @2@/book.toml
rsync -r --include='*.md' @CURRENT_SOURCE_DIR@/ @2@/
@4@ -r --include='*.md' @CURRENT_SOURCE_DIR@/ @2@/
(cd @2@; RUST_LOG=warn @1@ build -d @2@ 3>&2 2>&1 1>&3) | { grep -Fv "because fragment resolution isn't implemented" || :; } 3>&2 2>&1 1>&3
rm -rf @2@/manual
mv @2@/html @2@/manual
@@ -94,6 +98,7 @@ manual = custom_target(
mdbook.full_path(),
meson.current_build_dir(),
meson.project_version(),
rsync.full_path(),
),
],
input : [
@@ -118,8 +123,8 @@ manual = custom_target(
],
depfile : 'manual.d',
env : {
'RUST_LOG': 'info',
'MDBOOK_SUBSTITUTE_SEARCH': meson.current_build_dir() / 'source',
'RUST_LOG' : 'info',
'MDBOOK_SUBSTITUTE_SEARCH' : meson.current_build_dir() / 'source',
},
)
manual_html = manual[0]
@@ -131,7 +136,8 @@ install_subdir(
)
nix_nested_manpages = [
[ 'nix-env',
[
'nix-env',
[
'delete-generations',
'install',
@@ -146,7 +152,8 @@ nix_nested_manpages = [
'upgrade',
],
],
[ 'nix-store',
[
'nix-store',
[
'add-fixed',
'add',
+19 -16
View File
@@ -6,6 +6,7 @@
ninja,
lowdown-unsandboxed,
mdbook,
mdbook-linkcheck,
jq,
python3,
rsync,
@@ -45,22 +46,24 @@ mkMesonDerivation (finalAttrs: {
];
# Hack for sake of the dev shell
passthru.externalNativeBuildInputs = [
meson
ninja
(lib.getBin lowdown-unsandboxed)
mdbook
jq
python3
rsync
changelog-d
]
++ lib.optionals (!officialRelease) [
# When not an official release, we likely have changelog entries that have
# yet to be rendered.
# When released, these are rendered into a committed file to save a dependency.
changelog-d
];
passthru.externalNativeBuildInputs =
[
meson
ninja
(lib.getBin lowdown-unsandboxed)
mdbook
mdbook-linkcheck
jq
python3
rsync
changelog-d
]
++ lib.optionals (!officialRelease) [
# When not an official release, we likely have changelog entries that have
# yet to be rendered.
# When released, these are rendered into a committed file to save a dependency.
changelog-d
];
nativeBuildInputs = finalAttrs.passthru.externalNativeBuildInputs ++ [
nix-cli
@@ -0,0 +1,6 @@
---
synopsis: "`build-cores = 0` now auto-detects CPU cores"
prs: [13402]
---
When `build-cores` is set to `0`, nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
+13
View File
@@ -0,0 +1,13 @@
---
synopsis: "Add support for user@address:port syntax in store URIs"
prs: [3425]
issues: [7044]
---
It's now possible to specify the port used for the SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable. This has now been fixed.
This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
- `ssh://127.0.0.1:2222`
- `ssh://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
- `ssh-ng://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
@@ -0,0 +1,6 @@
---
synopsis: "Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874"
prs: [13445]
---
Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.
@@ -1,6 +0,0 @@
---
synopsis: "Temporary build directories no longer include derivation names"
prs: [13839]
---
Temporary build directories created during derivation builds no longer include the derivation name in their path to avoid build failures when the derivation name is too long. This change ensures predictable prefix lengths for build directories under `/nix/var/nix/builds`.
+1
View File
@@ -128,6 +128,7 @@
- [Development](development/index.md)
- [Building](development/building.md)
- [Testing](development/testing.md)
- [Benchmarking](development/benchmarking.md)
- [Debugging](development/debugging.md)
- [Documentation](development/documentation.md)
- [CLI guideline](development/cli-guideline.md)
+5 -13
View File
@@ -1,13 +1,12 @@
xp_features_json = custom_target(
command : [nix, '__dump-xp-features'],
command : [ nix, '__dump-xp-features' ],
capture : true,
output : 'xp-features.json',
)
experimental_features_shortlist_md = custom_target(
command : nix_eval_for_docs + [
'--expr',
'import @INPUT0@ (builtins.fromJSON (builtins.readFile ./@INPUT1@))',
'--expr', 'import @INPUT0@ (builtins.fromJSON (builtins.readFile ./@INPUT1@))',
],
input : [
'../../generate-xp-features-shortlist.nix',
@@ -19,14 +18,8 @@ experimental_features_shortlist_md = custom_target(
)
nix3_cli_files = custom_target(
command : [
python.full_path(),
'@INPUT0@',
'@OUTPUT@',
'--'
] + nix_eval_for_docs + [
'--expr',
'import @INPUT1@ true (builtins.readFile ./@INPUT2@)',
command : [ python.full_path(), '@INPUT0@', '@OUTPUT@', '--' ] + nix_eval_for_docs + [
'--expr', 'import @INPUT1@ true (builtins.readFile ./@INPUT2@)',
],
input : [
'../../remove_before_wrapper.py',
@@ -40,8 +33,7 @@ nix3_cli_files = custom_target(
conf_file_md_body = custom_target(
command : [
nix_eval_for_docs,
'--expr',
'import @INPUT0@ { prefix = "conf"; } (builtins.fromJSON (builtins.readFile ./@INPUT1@))',
'--expr', 'import @INPUT0@ { prefix = "conf"; } (builtins.fromJSON (builtins.readFile ./@INPUT1@))',
],
capture : true,
input : [
@@ -0,0 +1,187 @@
# Running Benchmarks
This guide explains how to build and run performance benchmarks in the Nix codebase.
## Overview
Nix uses the [Google Benchmark](https://github.com/google/benchmark) framework for performance testing. Benchmarks help measure and track the performance of critical operations like derivation parsing.
## Building Benchmarks
Benchmarks are disabled by default and must be explicitly enabled during the build configuration. For accurate results, use a debug-optimized release build.
### Development Environment Setup
First, enter the development shell which includes the necessary dependencies:
```bash
nix develop .#native-ccacheStdenv
```
### Configure Build with Benchmarks
From the project root, configure the build with benchmarks enabled and optimization:
```bash
cd build
meson configure -Dbenchmarks=true -Dbuildtype=debugoptimized
```
The `debugoptimized` build type provides:
- Compiler optimizations for realistic performance measurements
- Debug symbols for profiling and analysis
- Balance between performance and debuggability
### Build the Benchmarks
Build the project including benchmarks:
```bash
ninja
```
This will create benchmark executables in the build directory. Currently available:
- `build/src/libstore-tests/nix-store-benchmarks` - Store-related performance benchmarks
Additional benchmark executables will be created as more benchmarks are added to the codebase.
## Running Benchmarks
### Basic Usage
Run benchmark executables directly. For example, to run store benchmarks:
```bash
./build/src/libstore-tests/nix-store-benchmarks
```
As more benchmark executables are added, run them similarly from their respective build directories.
### Filtering Benchmarks
Run specific benchmarks using regex patterns:
```bash
# Run only derivation parser benchmarks
./build/src/libstore-tests/nix-store-benchmarks --benchmark_filter="derivation.*"
# Run only benchmarks for hello.drv
./build/src/libstore-tests/nix-store-benchmarks --benchmark_filter=".*hello.*"
```
### Output Formats
Generate benchmark results in different formats:
```bash
# JSON output
./build/src/libstore-tests/nix-store-benchmarks --benchmark_format=json > results.json
# CSV output
./build/src/libstore-tests/nix-store-benchmarks --benchmark_format=csv > results.csv
```
### Advanced Options
```bash
# Run benchmarks multiple times for better statistics
./build/src/libstore-tests/nix-store-benchmarks --benchmark_repetitions=10
# Set minimum benchmark time (useful for micro-benchmarks)
./build/src/libstore-tests/nix-store-benchmarks --benchmark_min_time=2
# Compare against baseline
./build/src/libstore-tests/nix-store-benchmarks --benchmark_baseline=baseline.json
# Display time in custom units
./build/src/libstore-tests/nix-store-benchmarks --benchmark_time_unit=ms
```
## Writing New Benchmarks
To add new benchmarks:
1. Create a new `.cc` file in the appropriate `*-tests` directory
2. Include the benchmark header:
```cpp
#include <benchmark/benchmark.h>
```
3. Write benchmark functions:
```cpp
static void BM_YourBenchmark(benchmark::State & state)
{
// Setup code here
for (auto _ : state) {
// Code to benchmark
}
}
BENCHMARK(BM_YourBenchmark);
```
4. Add the file to the corresponding `meson.build`:
```meson
benchmarks_sources = files(
'your-benchmark.cc',
# existing benchmarks...
)
```
## Profiling with Benchmarks
For deeper performance analysis, combine benchmarks with profiling tools:
```bash
# Using Linux perf
perf record ./build/src/libstore-tests/nix-store-benchmarks
perf report
```
### Using Valgrind Callgrind
Valgrind's callgrind tool provides detailed profiling information that can be visualized with kcachegrind:
```bash
# Profile with callgrind
valgrind --tool=callgrind ./build/src/libstore-tests/nix-store-benchmarks
# Visualize the results with kcachegrind
kcachegrind callgrind.out.*
```
This provides:
- Function call graphs
- Instruction-level profiling
- Source code annotation
- Interactive visualization of performance bottlenecks
## Continuous Performance Testing
```bash
# Save baseline results
./build/src/libstore-tests/nix-store-benchmarks --benchmark_format=json > baseline.json
# Compare against baseline in CI
./build/src/libstore-tests/nix-store-benchmarks --benchmark_baseline=baseline.json
```
## Troubleshooting
### Benchmarks not building
Ensure benchmarks are enabled:
```bash
meson configure build | grep benchmarks
# Should show: benchmarks true
```
### Inconsistent results
- Ensure your system is not under heavy load
- Disable CPU frequency scaling for consistent results
- Run benchmarks multiple times with `--benchmark_repetitions`
## See Also
- [Google Benchmark documentation](https://github.com/google/benchmark/blob/main/docs/user_guide.md)
+1 -2
View File
@@ -1,7 +1,6 @@
experimental_feature_descriptions_md = custom_target(
command : nix_eval_for_docs + [
'--expr',
'import @INPUT0@ (builtins.fromJSON (builtins.readFile @INPUT1@))',
'--expr', 'import @INPUT0@ (builtins.fromJSON (builtins.readFile @INPUT1@))',
],
input : [
'../../generate-xp-features.nix',
@@ -160,7 +160,6 @@ See the [corresponding section in the derivation output page](@docroot@/store/de
## Other output modifications
- [`unsafeDiscardReferences`]{#adv-attr-unsafeDiscardReferences}\
When using [structured attributes](#adv-attr-structuredAttrs), the
attribute `unsafeDiscardReferences` is an attribute set with a boolean value for each output name.
If set to `true`, it disables scanning the output for runtime dependencies.
@@ -195,7 +194,6 @@ See the [corresponding section in the derivation output page](@docroot@/store/de
[`builder`]: ./derivations.md#attr-builder
- [`requiredSystemFeatures`]{#adv-attr-requiredSystemFeatures}\
If a derivation has the `requiredSystemFeatures` attribute, then Nix will only build it on a machine that has the corresponding features set in its [`system-features` configuration](@docroot@/command-ref/conf-file.md#conf-system-features).
For example, setting
+3 -9
View File
@@ -1,19 +1,13 @@
builtins_md = custom_target(
command : [
python.full_path(),
'@INPUT0@',
'@OUTPUT@',
'--'
] + nix_eval_for_docs + [
'--expr',
'(builtins.readFile @INPUT3@) + import @INPUT1@ (builtins.fromJSON (builtins.readFile ./@INPUT2@)) + (builtins.readFile @INPUT4@)',
command : [ python.full_path(), '@INPUT0@', '@OUTPUT@', '--' ] + nix_eval_for_docs + [
'--expr', '(builtins.readFile @INPUT3@) + import @INPUT1@ (builtins.fromJSON (builtins.readFile ./@INPUT2@)) + (builtins.readFile @INPUT4@)',
],
input : [
'../../remove_before_wrapper.py',
'../../generate-builtins.nix',
language_json,
'builtins-prefix.md',
'builtins-suffix.md'
'builtins-suffix.md',
],
output : 'builtins.md',
env : nix_env_for_docs,
+3 -2
View File
@@ -1,7 +1,8 @@
summary_rl_next = custom_target(
command : [
bash,
'-euo', 'pipefail',
'-euo',
'pipefail',
'-c',
'''
if [ -e "@INPUT@" ]; then
@@ -12,6 +13,6 @@ summary_rl_next = custom_target(
input : [
rl_next_generated,
],
capture: true,
capture : true,
output : 'SUMMARY-rl-next.md',
)
@@ -12,7 +12,7 @@ $ ln -s /nix/store/d718ef...-foo /nix/var/nix/gcroots/bar
That is, after this command, the garbage collector will not remove
`/nix/store/d718ef...-foo` or any of its dependencies.
Subdirectories of `prefix/nix/var/nix/gcroots` are also searched for
symlinks. Symlinks to non-store paths are followed and searched for
roots, but symlinks to non-store paths *inside* the paths reached in
that way are not followed to prevent infinite recursion.
Subdirectories of `prefix/nix/var/nix/gcroots` are searched
recursively. Symlinks to store paths count as roots. Symlinks to
non-store paths are ignored, unless the non-store path is itself a
symlink to a store path.
+1 -1
View File
@@ -24,7 +24,7 @@ nar-obj-inner
| str("type"), str("directory") directory
;
regular = [ str("executable"), str("") ], str("contents"), str(contents);
regular = [ str("executable") ], str("contents"), str(contents);
symlink = str("target"), str(target);
+1 -1
View File
@@ -48,6 +48,6 @@
* `nix run` is now stricter in what it accepts: members of the `apps`
flake output are now required to be apps (as defined in [the
manual](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-run.html#apps)),
manual](https://nix.dev/manual/nix/stable/command-ref/new-cli/nix3-run.html#apps)),
and members of `packages` or `legacyPackages` must be derivations
(not apps).
+2 -3
View File
@@ -12,11 +12,10 @@
The [`builder`](./derivation/index.md#builder) is executed as follows:
- A temporary directory is created where the build will take place. The
- A temporary directory is created under the directory specified by
`TMPDIR` (default `/tmp`) where the build will take place. The
current directory is changed to this directory.
See the per-store [`build-dir`](@docroot@/store/types/local-store.md#store-local-store-build-dir) setting for more information.
- The environment is cleared and set to the derivation attributes, as
specified above.
+2 -8
View File
@@ -1,12 +1,6 @@
types_dir = custom_target(
command : [
python.full_path(),
'@INPUT0@',
'@OUTPUT@',
'--'
] + nix_eval_for_docs + [
'--expr',
'import @INPUT1@ (builtins.fromJSON (builtins.readFile ./@INPUT2@)).stores',
command : [ python.full_path(), '@INPUT0@', '@OUTPUT@', '--' ] + nix_eval_for_docs + [
'--expr', 'import @INPUT1@ (builtins.fromJSON (builtins.readFile ./@INPUT2@)).stores',
],
input : [
'../../remove_before_wrapper.py',
-4
View File
@@ -41,10 +41,6 @@ def recursive_replace(data: dict[str, t.Any], book_root: Path, search_path: Path
return data | dict(
sections = [recursive_replace(section, book_root, search_path) for section in sections],
)
case {'items': items}:
return data | dict(
items = [recursive_replace(item, book_root, search_path) for item in items],
)
case {'Chapter': chapter}:
path_to_chapter = Path(chapter['path'])
chapter_content = chapter['content']
+73 -72
View File
@@ -1,10 +1,10 @@
{
# Core dependencies
pkgs,
lib,
dockerTools,
runCommand,
buildPackages,
pkgs ? import <nixpkgs> { },
lib ? pkgs.lib,
dockerTools ? pkgs.dockerTools,
runCommand ? pkgs.runCommand,
buildPackages ? pkgs.buildPackages,
# Image configuration
name ? "nix",
tag ? "latest",
@@ -28,24 +28,24 @@
},
Cmd ? [ (lib.getExe bashInteractive) ],
# Default Packages
nix,
bashInteractive,
coreutils-full,
gnutar,
gzip,
gnugrep,
which,
curl,
less,
wget,
man,
cacert,
findutils,
iana-etc,
gitMinimal,
openssh,
nix ? pkgs.nix,
bashInteractive ? pkgs.bashInteractive,
coreutils-full ? pkgs.coreutils-full,
gnutar ? pkgs.gnutar,
gzip ? pkgs.gzip,
gnugrep ? pkgs.gnugrep,
which ? pkgs.which,
curl ? pkgs.curl,
less ? pkgs.less,
wget ? pkgs.wget,
man ? pkgs.man,
cacert ? pkgs.cacert,
findutils ? pkgs.findutils,
iana-etc ? pkgs.iana-etc,
gitMinimal ? pkgs.gitMinimal,
openssh ? pkgs.openssh,
# Other dependencies
shadow,
shadow ? pkgs.shadow,
}:
let
defaultPkgs = [
@@ -65,60 +65,61 @@ let
iana-etc
gitMinimal
openssh
]
++ extraPkgs;
] ++ extraPkgs;
users = {
users =
{
root = {
uid = 0;
shell = lib.getExe bashInteractive;
home = "/root";
gid = 0;
groups = [ "root" ];
description = "System administrator";
};
nobody = {
uid = 65534;
shell = lib.getExe' shadow "nologin";
home = "/var/empty";
gid = 65534;
groups = [ "nobody" ];
description = "Unprivileged account (don't use!)";
};
}
// lib.optionalAttrs (uid != 0) {
"${uname}" = {
uid = uid;
shell = lib.getExe bashInteractive;
home = "/home/${uname}";
gid = gid;
groups = [ "${gname}" ];
description = "Nix user";
};
}
// lib.listToAttrs (
map (n: {
name = "nixbld${toString n}";
value = {
uid = 30000 + n;
gid = 30000;
groups = [ "nixbld" ];
description = "Nix build user ${toString n}";
root = {
uid = 0;
shell = lib.getExe bashInteractive;
home = "/root";
gid = 0;
groups = [ "root" ];
description = "System administrator";
};
}) (lib.lists.range 1 32)
);
groups = {
root.gid = 0;
nixbld.gid = 30000;
nobody.gid = 65534;
}
// lib.optionalAttrs (gid != 0) {
"${gname}".gid = gid;
};
nobody = {
uid = 65534;
shell = lib.getExe' shadow "nologin";
home = "/var/empty";
gid = 65534;
groups = [ "nobody" ];
description = "Unprivileged account (don't use!)";
};
}
// lib.optionalAttrs (uid != 0) {
"${uname}" = {
uid = uid;
shell = lib.getExe bashInteractive;
home = "/home/${uname}";
gid = gid;
groups = [ "${gname}" ];
description = "Nix user";
};
}
// lib.listToAttrs (
map (n: {
name = "nixbld${toString n}";
value = {
uid = 30000 + n;
gid = 30000;
groups = [ "nixbld" ];
description = "Nix build user ${toString n}";
};
}) (lib.lists.range 1 32)
);
groups =
{
root.gid = 0;
nixbld.gid = 30000;
nobody.gid = 65534;
}
// lib.optionalAttrs (gid != 0) {
"${gname}".gid = gid;
};
userToPasswd = (
k:
Generated
+4 -4
View File
@@ -63,16 +63,16 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1756178832,
"narHash": "sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8=",
"lastModified": 1747179050,
"narHash": "sha256-qhFMmDkeJX9KJwr5H32f1r7Prs7XbQWtO0h3V0a0rFY=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "d98ce345cdab58477ca61855540999c86577d19d",
"rev": "adaa24fbf46737f3f1b5497bf64bae750f82942e",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-25.05-small",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
+208 -112
View File
@@ -1,7 +1,7 @@
{
description = "The purely functional package manager";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05-small";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.nixpkgs-23-11.url = "github:NixOS/nixpkgs/a62e6edd6d5e1fa0329b8653c801147986f8d446";
@@ -32,7 +32,7 @@
let
inherit (nixpkgs) lib;
officialRelease = true;
officialRelease = false;
linux32BitSystems = [ "i686-linux" ];
linux64BitSystems = [
@@ -131,31 +131,107 @@
}
);
overlayFor =
getStdenv: final: prev:
/**
Produce the `nixComponents` and `nixDependencies` package sets (scopes) for
a given `pkgs` and `getStdenv`.
*/
packageSetsFor =
let
stdenv = getStdenv final;
/**
Removes a prefix from the attribute names of a set of splices.
This is a completely uninteresting and exists for compatibility only.
Example:
```nix
renameSplicesFrom "pkgs" { pkgsBuildBuild = ...; ... }
=> { buildBuild = ...; ... }
```
*/
renameSplicesFrom = prefix: x: {
buildBuild = x."${prefix}BuildBuild";
buildHost = x."${prefix}BuildHost";
buildTarget = x."${prefix}BuildTarget";
hostHost = x."${prefix}HostHost";
hostTarget = x."${prefix}HostTarget";
targetTarget = x."${prefix}TargetTarget";
};
/**
Adds a prefix to the attribute names of a set of splices.
This is a completely uninteresting and exists for compatibility only.
Example:
```nix
renameSplicesTo "self" { buildBuild = ...; ... }
=> { selfBuildBuild = ...; ... }
```
*/
renameSplicesTo = prefix: x: {
"${prefix}BuildBuild" = x.buildBuild;
"${prefix}BuildHost" = x.buildHost;
"${prefix}BuildTarget" = x.buildTarget;
"${prefix}HostHost" = x.hostHost;
"${prefix}HostTarget" = x.hostTarget;
"${prefix}TargetTarget" = x.targetTarget;
};
/**
Takes a function `f` and returns a function that applies `f` pointwise to each splice.
Example:
```nix
mapSplices (x: x * 10) { buildBuild = 1; buildHost = 2; ... }
=> { buildBuild = 10; buildHost = 20; ... }
```
*/
mapSplices =
f:
{
buildBuild,
buildHost,
buildTarget,
hostHost,
hostTarget,
targetTarget,
}:
{
buildBuild = f buildBuild;
buildHost = f buildHost;
buildTarget = f buildTarget;
hostHost = f hostHost;
hostTarget = f hostTarget;
targetTarget = f targetTarget;
};
in
{
nixStable = prev.nix;
args@{
pkgs,
getStdenv ? pkgs: pkgs.stdenv,
}:
let
nixComponentsSplices = mapSplices (
pkgs': (packageSetsFor (args // { pkgs = pkgs'; })).nixComponents
) (renameSplicesFrom "pkgs" pkgs);
nixDependenciesSplices = mapSplices (
pkgs': (packageSetsFor (args // { pkgs = pkgs'; })).nixDependencies
) (renameSplicesFrom "pkgs" pkgs);
# A new scope, so that we can use `callPackage` to inject our own interdependencies
# without "polluting" the top level "`pkgs`" attrset.
# This also has the benefit of providing us with a distinct set of packages
# we can iterate over.
# The `2` suffix is here because otherwise it interferes with `nixVersions.latest`, which is used in daemon compat tests.
nixComponents2 =
nixComponents =
lib.makeScopeWithSplicing'
{
inherit (final) splicePackages;
inherit (final.nixDependencies2) newScope;
inherit (pkgs) splicePackages;
inherit (nixDependencies) newScope;
}
{
otherSplices = final.generateSplicesForMkScope "nixComponents2";
otherSplices = renameSplicesTo "self" nixComponentsSplices;
f = import ./packaging/components.nix {
inherit (final) lib;
inherit (pkgs) lib;
inherit officialRelease;
pkgs = final;
inherit pkgs;
src = self;
maintainers = [ ];
};
@@ -163,29 +239,71 @@
# The dependencies are in their own scope, so that they don't have to be
# in Nixpkgs top level `pkgs` or `nixComponents2`.
# The `2` suffix is here because otherwise it interferes with `nixVersions.latest`, which is used in daemon compat tests.
nixDependencies2 =
nixDependencies =
lib.makeScopeWithSplicing'
{
inherit (final) splicePackages;
inherit (final) newScope; # layered directly on pkgs, unlike nixComponents2 above
inherit (pkgs) splicePackages;
inherit (pkgs) newScope; # layered directly on pkgs, unlike nixComponents2 above
}
{
otherSplices = final.generateSplicesForMkScope "nixDependencies2";
otherSplices = renameSplicesTo "self" nixDependenciesSplices;
f = import ./packaging/dependencies.nix {
inherit inputs stdenv;
pkgs = final;
inherit inputs pkgs;
stdenv = getStdenv pkgs;
};
};
# If the package set is largely empty, we should(?) return empty sets
# This is what most package sets in Nixpkgs do. Otherwise, we get
# an error message that indicates that some stdenv attribute is missing,
# and indeed it will be missing, as seemingly `pkgsTargetTarget` is
# very incomplete.
fixup = lib.mapAttrs (k: v: if !(pkgs ? nix) then { } else v);
in
fixup {
inherit nixDependencies;
inherit nixComponents;
};
overlayFor =
getStdenv: final: prev:
let
packageSets = packageSetsFor {
inherit getStdenv;
pkgs = final;
};
in
{
nixStable = prev.nix;
# The `2` suffix is here because otherwise it interferes with `nixVersions.latest`, which is used in daemon compat tests.
nixComponents2 = packageSets.nixComponents;
# The dependencies are in their own scope, so that they don't have to be
# in Nixpkgs top level `pkgs` or `nixComponents2`.
# The `2` suffix is here because otherwise it interferes with `nixVersions.latest`, which is used in daemon compat tests.
nixDependencies2 = packageSets.nixDependencies;
nix = final.nixComponents2.nix-cli;
};
in
{
# A Nixpkgs overlay that overrides the 'nix' and
# 'nix-perl-bindings' packages.
overlays.default = overlayFor (p: p.stdenv);
overlays.internal = overlayFor (p: p.stdenv);
/**
A Nixpkgs overlay that sets `nix` to something like `packages.<system>.nix-everything`,
except dependencies aren't taken from (flake) `nix.inputs.nixpkgs`, but from the Nixpkgs packages
where the overlay is used.
*/
overlays.default =
final: prev:
let
packageSets = packageSetsFor { pkgs = final; };
in
{
nix = packageSets.nixComponents.nix-everything;
};
hydraJobs = import ./packaging/hydra.nix {
inherit
@@ -202,43 +320,11 @@
checks = forAllSystems (
system:
{
installerScriptForGHA = self.hydraJobs.installerScriptForGHA.${system};
installTests = self.hydraJobs.installTests.${system};
nixpkgsLibTests = self.hydraJobs.tests.nixpkgsLibTests.${system};
rl-next =
let
pkgs = nixpkgsFor.${system}.native;
in
pkgs.buildPackages.runCommand "test-rl-next-release-notes" { } ''
LANG=C.UTF-8 ${pkgs.changelog-d}/bin/changelog-d ${./doc/manual/rl-next} >$out
'';
repl-completion = nixpkgsFor.${system}.native.callPackage ./tests/repl-completion.nix { };
/**
Checks for our packaging expressions.
This shouldn't build anything significant; just check that things
(including derivations) are _set up_ correctly.
*/
packaging-overriding =
let
pkgs = nixpkgsFor.${system}.native;
nix = self.packages.${system}.nix;
in
assert (nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src.patches == [ pkgs.emptyFile ];
if pkgs.stdenv.buildPlatform.isDarwin then
lib.warn "packaging-overriding check currently disabled because of a permissions issue on macOS" pkgs.emptyFile
else
# If this fails, something might be wrong with how we've wired the scope,
# or something could be broken in Nixpkgs.
pkgs.testers.testEqualContents {
assertion = "trivial patch does not change source contents";
expected = "${./.}";
actual =
# Same for all components; nix-util is an arbitrary pick
(nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src;
};
}
(import ./ci/gha/tests {
inherit system;
pkgs = nixpkgsFor.${system}.native;
nixFlake = self;
}).topLevel
// (lib.optionalAttrs (builtins.elem system linux64BitSystems)) {
dockerImage = self.hydraJobs.dockerImage.${system};
}
@@ -251,58 +337,20 @@
# Add "passthru" tests
//
flatMapAttrs
{
"" = {
pkgs = nixpkgsFor.${system}.native;
};
}
(
{
# Run all tests with UBSAN enabled. Running both with ubsan and
# without doesn't seem to have much immediate benefit for doubling
# the GHA CI workaround.
#
# TODO: Work toward enabling "address,undefined" if it seems feasible.
# This would maybe require dropping Boost coroutines and ignoring intentional
# memory leaks with detect_leaks=0.
"" = rec {
nixpkgs = nixpkgsFor.${system}.native;
nixComponents = nixpkgs.nixComponents2.overrideScope (
nixCompFinal: nixCompPrev: {
mesonComponentOverrides = _finalAttrs: prevAttrs: {
mesonFlags =
(prevAttrs.mesonFlags or [ ])
# TODO: Macos builds instrumented with ubsan take very long
# to run functional tests.
++ lib.optionals (!nixpkgs.stdenv.hostPlatform.isDarwin) [
(lib.mesonOption "b_sanitize" "undefined")
];
};
}
);
};
}
// lib.optionalAttrs (!nixpkgsFor.${system}.native.stdenv.hostPlatform.isDarwin) {
# TODO: enable static builds for darwin, blocked on:
# https://github.com/NixOS/nixpkgs/issues/320448
# TODO: disabled to speed up GHA CI.
# "static-" = {
# nixpkgs = nixpkgsFor.${system}.native.pkgsStatic;
# };
}
)
(
nixpkgsPrefix:
{
nixpkgs,
nixComponents ? nixpkgs.nixComponents2,
}:
flatMapAttrs nixComponents (
pkgName: pkg:
flatMapAttrs pkg.tests or { } (
testName: test: {
"${nixpkgsPrefix}${pkgName}-${testName}" = test;
}
)
)
// lib.optionalAttrs (nixpkgs.stdenv.hostPlatform == nixpkgs.stdenv.buildPlatform) {
"${nixpkgsPrefix}nix-functional-tests" = nixComponents.nix-functional-tests;
}
nixpkgsPrefix: args:
(import ./ci/gha/tests (
args
// {
nixFlake = self;
componentTestsPrefix = nixpkgsPrefix;
}
)).componentTests
)
// devFlake.checks.${system} or { }
);
@@ -465,5 +513,53 @@
default = self.devShells.${system}.native;
}
);
lib = {
/**
Creates a package set for a given Nixpkgs instance and stdenv.
# Inputs
- `pkgs`: The Nixpkgs instance to use.
- `getStdenv`: _Optional_ A function that takes a package set and returns the stdenv to use.
This needs to be a function in order to support cross compilation - the `pkgs` passed to `getStdenv` can be `pkgsBuildHost` or any other variation needed.
# Outputs
The return value is a fresh Nixpkgs scope containing all the packages that are defined in the Nix repository,
as well as some internals and parameters, which may be subject to change.
# Example
```console
nix repl> :lf NixOS/nix
nix-repl> ps = lib.makeComponents { pkgs = import inputs.nixpkgs { crossSystem = "riscv64-linux"; }; }
nix-repl> ps
{
appendPatches = «lambda appendPatches @ ...»;
callPackage = «lambda callPackageWith @ ...»;
overrideAllMesonComponents = «lambda overrideSource @ ...»;
overrideSource = «lambda overrideSource @ ...»;
# ...
nix-everything
# ...
nix-store
nix-store-c
# ...
}
```
*/
makeComponents =
{
pkgs,
getStdenv ? pkgs: pkgs.stdenv,
}:
let
packageSets = packageSetsFor { inherit getStdenv pkgs; };
in
packageSets.nixComponents;
};
};
}
+23 -115
View File
@@ -37,118 +37,29 @@
fi
''}";
};
meson-format = {
enable = true;
files = "(meson.build|meson.options)$";
entry = "${pkgs.writeScript "format-meson" ''
#!${pkgs.runtimeShell}
for file in "$@"; do
${lib.getExe pkgs.meson} format -ic ${../meson.format} "$file"
done
''}";
excludes = [
# We haven't applied formatting to these files yet
''^doc/manual/meson.build$''
''^doc/manual/source/command-ref/meson.build$''
''^doc/manual/source/development/meson.build$''
''^doc/manual/source/language/meson.build$''
''^doc/manual/source/meson.build$''
''^doc/manual/source/release-notes/meson.build$''
''^doc/manual/source/store/meson.build$''
''^misc/bash/meson.build$''
''^misc/fish/meson.build$''
''^misc/launchd/meson.build$''
''^misc/meson.build$''
''^misc/systemd/meson.build$''
''^misc/zsh/meson.build$''
''^nix-meson-build-support/$''
''^nix-meson-build-support/big-objs/meson.build$''
''^nix-meson-build-support/common/meson.build$''
''^nix-meson-build-support/deps-lists/meson.build$''
''^nix-meson-build-support/export/meson.build$''
''^nix-meson-build-support/export-all-symbols/meson.build$''
''^nix-meson-build-support/generate-header/meson.build$''
''^nix-meson-build-support/libatomic/meson.build$''
''^nix-meson-build-support/subprojects/meson.build$''
''^scripts/meson.build$''
''^src/external-api-docs/meson.build$''
''^src/internal-api-docs/meson.build$''
''^src/libcmd/include/nix/cmd/meson.build$''
''^src/libcmd/meson.build$''
''^src/libcmd/nix-meson-build-support$''
''^src/libexpr/include/nix/expr/meson.build$''
''^src/libexpr/meson.build$''
''^src/libexpr/nix-meson-build-support$''
''^src/libexpr-c/meson.build$''
''^src/libexpr-c/nix-meson-build-support$''
''^src/libexpr-test-support/meson.build$''
''^src/libexpr-test-support/nix-meson-build-support$''
''^src/libexpr-tests/meson.build$''
''^src/libexpr-tests/nix-meson-build-support$''
''^src/libfetchers/include/nix/fetchers/meson.build$''
''^src/libfetchers/meson.build$''
''^src/libfetchers/nix-meson-build-support$''
''^src/libfetchers-c/meson.build$''
''^src/libfetchers-c/nix-meson-build-support$''
''^src/libfetchers-tests/meson.build$''
''^src/libfetchers-tests/nix-meson-build-support$''
''^src/libflake/include/nix/flake/meson.build$''
''^src/libflake/meson.build$''
''^src/libflake/nix-meson-build-support$''
''^src/libflake-c/meson.build$''
''^src/libflake-c/nix-meson-build-support$''
''^src/libflake-tests/meson.build$''
''^src/libflake-tests/nix-meson-build-support$''
''^src/libmain/include/nix/main/meson.build$''
''^src/libmain/meson.build$''
''^src/libmain/nix-meson-build-support$''
''^src/libmain-c/meson.build$''
''^src/libmain-c/nix-meson-build-support$''
''^src/libstore/include/nix/store/meson.build$''
''^src/libstore/meson.build$''
''^src/libstore/nix-meson-build-support$''
''^src/libstore/unix/include/nix/store/meson.build$''
''^src/libstore/unix/meson.build$''
''^src/libstore/windows/meson.build$''
''^src/libstore-c/meson.build$''
''^src/libstore-c/nix-meson-build-support$''
''^src/libstore-test-support/include/nix/store/tests/meson.build$''
''^src/libstore-test-support/meson.build$''
''^src/libstore-test-support/nix-meson-build-support$''
''^src/libstore-tests/meson.build$''
''^src/libstore-tests/nix-meson-build-support$''
''^src/libutil/meson.build$''
''^src/libutil/nix-meson-build-support$''
''^src/libutil/unix/include/nix/util/meson.build$''
''^src/libutil/unix/meson.build$''
''^src/libutil/windows/meson.build$''
''^src/libutil-c/meson.build$''
''^src/libutil-c/nix-meson-build-support$''
''^src/libutil-test-support/include/nix/util/tests/meson.build$''
''^src/libutil-test-support/meson.build$''
''^src/libutil-test-support/nix-meson-build-support$''
''^src/libutil-tests/meson.build$''
''^src/libutil-tests/nix-meson-build-support$''
''^src/nix/meson.build$''
''^src/nix/nix-meson-build-support$''
''^src/perl/lib/Nix/meson.build$''
''^src/perl/meson.build$''
''^tests/functional/ca/meson.build$''
''^tests/functional/common/meson.build$''
''^tests/functional/dyn-drv/meson.build$''
''^tests/functional/flakes/meson.build$''
''^tests/functional/git-hashing/meson.build$''
''^tests/functional/local-overlay-store/meson.build$''
''^tests/functional/meson.build$''
''^src/libcmd/meson.options$''
''^src/libexpr/meson.options$''
''^src/libstore/meson.options$''
''^src/libutil/meson.options$''
''^src/libutil-c/meson.options$''
''^src/nix/meson.options$''
''^src/perl/meson.options$''
];
};
meson-format =
let
meson = pkgs.meson.overrideAttrs {
doCheck = false;
doInstallCheck = false;
patches = [
(pkgs.fetchpatch {
url = "https://github.com/mesonbuild/meson/commit/38d29b4dd19698d5cad7b599add2a69b243fd88a.patch";
hash = "sha256-PgPBvGtCISKn1qQQhzBW5XfknUe91i5XGGBcaUK4yeE=";
})
];
};
in
{
enable = true;
files = "(meson.build|meson.options)$";
entry = "${pkgs.writeScript "format-meson" ''
#!${pkgs.runtimeShell}
for file in "$@"; do
${lib.getExe meson} format -ic ${../meson.format} "$file"
done
''}";
};
nixfmt-rfc-style = {
enable = true;
excludes = [
@@ -262,8 +173,6 @@
''^tests/functional/gc-concurrent\.sh$''
''^tests/functional/gc-concurrent2\.builder\.sh$''
''^tests/functional/gc-non-blocking\.sh$''
''^tests/functional/git-hashing/common\.sh$''
''^tests/functional/git-hashing/simple\.sh$''
''^tests/functional/hash-convert\.sh$''
''^tests/functional/impure-derivations\.sh$''
''^tests/functional/impure-eval\.sh$''
@@ -339,7 +248,6 @@
''^tests/functional/user-envs\.builder\.sh$''
''^tests/functional/user-envs\.sh$''
''^tests/functional/why-depends\.sh$''
''^src/libutil-tests/data/git/check-data\.sh$''
];
};
};
@@ -1,110 +0,0 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGPtMiwBEAC0sFZW2QW/OaDjKm5zGRpDvHXDsMIUtlHfoi5ce8pocC63W05o
FSXbUZjZ1VfYO8lT8DFANCzTkiXYaZx0cPRG2pVY4AOQZDNFt5XrAyvw496XCAIM
DTYGFLjCqgjPt9RUFEy4MyHPJTEpB0x3rXgT4ILNu9vsj9Q0vttps7SpbZ3Ldq5H
o/BBbLW77q/vNjpYzCbBIXF7ycUGpnNv9Go/WuiDnrBMcyxh+8kjjIHB5cxZSnjJ
DUv681+m83v+gLZQGX/jexQrrf5JpS0X9qEnhGLrNUDhtyv5ud3Je4EfamkjLVVC
RlNLofgflOCsl/tP80i+K7S1QdKhUALxuJ6H0prYUflGBDxDyC8XYuJ62TT0OUpa
vJvgwVlCq8/jq+ykYQXlbuBVOzi5wAuI4l3+HqreSQYPSiwe+6N590Zbafdv1fvN
WFtZKCTGMqfyaaAnppioH9/+NWkI2AQxaYVasYM/JEYvY9pJgA7alh51jHW4JglP
ErypKfBKPKJID0QENqYoa3bDDCihuNWhgQf9dxzPlj2ckd35Zb6w4DfuSmtjaa9D
o0jZVY1JbFuxBqP09+saVPrxLHgmPxjcdzPGQQtAqdO2vyJXNEGLFMoVEZPNaLo3
QmcIJnT7oSck+4vGfOYtWUHXQynu/Tnwsv2XkA/uyw8HNe+RRMqv/apnzQARAQAB
tCdTZXJnZWkgWmltbWVybWFuIDxzZXJnZWlAemltbWVybWFuLmZvbz6JAlEEEwEK
ADsCGwEFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQQVim9TDqIC5fZRYRMU+upj
RI4d+QUCaUbsaAIZAQAKCRAU+upjRI4d+VdDD/492HRaJ/8V7R7VUzkafmb2Hb28
SLf7oiB8Uq9I7SukiDEaIT1fUhquYWQ9KWpPRNR1TX6ApXnIeuJRMGFoDVIRnmnr
cKnYYXfqqc81VxIyKvaumB7KWbS7G4Nbor8AH1ouOOOMMS50OTJOWQA4A26inIuG
n+7L8MeS5aT+3uNKDoTKsidC47vnaxNMcke1taPfbfo7vn69PsRCM/g9/7TQYU8b
6xp+pM9Ao9nJneRk2YCpsGYRrWTpaik0DFKnfpPKJM/yunhtGLF2IYAp3l1mvHPK
nnzo92zjpQuZwazEIK+23V1vRT4IjM2BewbJPAzf2/UuxEjjgNQm0tOtH2JhFNeB
VM0BVrGxWrrwrsmv6lWghTtBc6zRWyHrj/rpjtVQNmeKYrHWJeXwVz1rqgGPmB2N
k0MZD1UjHHhEs1Cntn7yLmxTPztRJCtR+euRu81Uo2NAvrMJ4xsDjaM0LeLTnzjV
9AsPjD188dOFyz7VExZum4+XaaEJ41FIPLEqU3U0GAa6stEy0ylSlIN4x9aiXXVW
xfzHHchS5jK6QAjuZxN8t01GRactNylINRf7uoTECFZTtXNqfeuk0HQxBH0LuKVE
0PxJbcNI4mVWw1KgTJ8PUVC1IXP3sEpqPdJOYiRXgnpcS26fWOBu0aQ4mhxiaJhr
/zBfrkLEqp20TdNDZLkCDQRj7TM1ARAAt73xO24curnHTTgXkkVMMRzcMLx3Mb1a
2FuddxC5hzTpEpw01L91UBrXVJEg9K2KAwP5CtCLgPCqXr47Tm7krvHxWwBksgY/
6aHRsoPQfCFUZHc0aiO+C4NCzR+aEeGKn66Oc1Hq9oUTpDgiBWhsuEPiyA1OSGF0
4L0jeTCqfm68kWp4PIK9yuugkdDsoyj6TonuMsb3V5ctHLqop9KH+eHSkUTPo+Lk
+bxaeAOJ1UfbohgbRbrYKAfsaghhOMDH3R1w2pvtUJz+sDbuQsiPFTqbxsXDTFws
H4N/AQCYnnvOhqEek2sOEZ19bJXt5UrAr10mX4PGmAkWqE1JWBxpOKG3BXSGOTu1
3dFhQfPMK+PmvUrs0kcWQr53K/aRUdKKhIfTcMfkqYTGPK5HclHph24WjXj3QFFA
SjksQTdm6486ZmLZK4CTbAFOPfTF/aWg8gu9v4ihdq6lqHNNXxv2xBAChcd59H7p
D5zy9z8SpwWR9V5JDmlF6HWIIau1c6lSsQq1xHvYM8EuPe03vJvor+2u/cn5zYF1
5ZxAuPI2i5vtavg1s8ZGAAogJ9dVcP36LdJfL9quXWvmovkd//qHIepBB+l/zQio
ZRDZlIcfV3Xycaqsb5OqHGARHE0097koipMt5y/iXlqG4Ruue6Idb8bW96EKpaWj
kKy/iNfQfQMAEQEAAYkEcgQYAQoAJgIbAhYhBBWKb1MOogLl9lFhExT66mNEjh35
BQJpRuz3BQkJHCDCAkDBdCAEGQEKAB0WIQRK3hK0WyJ4BicGpcmpsLVXymMjJQUC
Y+0zNQAKCRCpsLVXymMjJbdSD/9+f1FOOeGDAJI6Duo5fsWnf4xJJdtQtDbz6d2A
SeDapxeJ3zWfKBD0wu5sISEa0uiWsYSmLtsa2SqVAKHlEaMGRR+tkBMPQ+rvgI4c
62YjGTgm+IPd+NFIn+ixFU1hpinTh+KhUEoeOwWCvKs9nZfSG9vkienfiG0bBxo2
zrvBzXA50x5hbUL+ghKu/AVfN9qZDwh30O4KZTwk4g4cM9SeaQa4YvHYIS3IEhDZ
hGybwrrqV9cs92ln4IJw9WCy9QReBNrdeFgC4+3ziUp1QsG3RvqrtuMttwBVC1Z3
bj5QjLLOREhhodfvk98t9yVkragObb4rGrLo1mWuF0c4mJGvXwnrqhCMvzv4M+0T
Zdrmw6YpGkGOaOPghVuwoTtqSAkl+zFWIJS89jidvkYG3EqKAkgLKog/TQReCq13
HWrF8cMck+Rf2K8k26q/RNZaA9ZUKjLExzz8lsWmd2C7rvkGLrlxnzxz0gGyNR3Q
KK74vcPhqeABt2GSkHtEXZFFA9IVVzwlRWK3e0S+mVQnZVjNL+cBPn3/hZHMLesB
CucyYZv+DxvT+JkYXBkGSw4s3hpABqGym7gdPUIa0q4rbBFG6xP5sLLBG4yru8vV
2dyCMmFqRuxpT49uNfyQ6Vj+dobN6qHnP/9NwfzOixXYBHXR6LBqb/M+iCiJaaIn
uiRLHwkQFPrqY0SOHfkQpA//W51vj8meuz7snRO+vZFcjLneFFzqfh1Jdz8IqDpO
CkI5pBJmi8e0oSe6r68MkahiQLlYPwm7d+sjHvJhPWipNKWq/uwCgBs+Ac1lpPXR
MwLbrZukcLMYlLmb2MrCKmjcMt0BZsZKBNYL3a3X9nHgwXdeqFYS4WQDMCCc09lz
9YqfdoEsqRO4qN7D0hFqnwjOzb34ixZ6UO8a8ekY9QKxAgWc9fJWGMg6Pjdg4qsK
nqymOIAdGVOJdoRM46wKGVBvbsF2gNfQU4XyzgJo5vHGFwJm6EoSnODlL5e2wsQh
uN1oqBt/8ef/plloMEqVBweUBATqSqjRF6IhhYJvWVuQHQL1p1vnV9FebiVj34ir
Z8ID+o0AnTJcclbUcDwannGJ0cuDcPhk/v/ahVuoMERCi12qnMBo5B/e6Omyh1yB
4pbf4GATGGQipDQG75eC/kP2GQEqJP5WYN0Ar8Le/AA/2xyL7upW0yIByyXCwGEb
JRwEgU3+bPyu58bFt8Pftit6J7rA3oBVVMOPrYH5eZwRaj5m2RptwKGL6BfHnhNv
ZqmCq9EBGX6L1NI0xHMjEFfXJ8jU01XdfG8nCqkwqsHwslXLhqjJphfHcx89YwbV
/15GCuURAv1cKe/7277sOhcvP/QpQqSWgvYExHw8PeFJcTYtF2NrRgNwcQsWS1Rj
gXa5Ag0EY+0zcAEQANC5N6kSfezuucAgi+X3BD+MT37mxQyvICSggEJf1LDSmy0+
bnvD7setL8CP9etTA2fcVNYKI1oboMyhoCnsRP2jDdv1iXOI/hZg4wSb/D1yUkae
fUpxv3Wuci2QKavH2MfraDD7BFMbsQeMcHtn4Rk216T6jndZHnzT1Ih7iX0XeQPb
li5fojOiZssgWAVT4HPXFCJB6lI35Hjp35oRYwrtMmu5INinZ79n9h1igGtt1ItZ
b7rQKNd772Jxcn4UU71ovORSL/xT5i5sxZ+evQOxkpqUAokMOFaoHcOXLmA1NsFv
yryXHK4Ioq9ap2jKlLTWkJWjua9JZ4AmKhbvT8X4ELxIKSCAdJKAWP8ZHbXNu5MD
aznyzZQLxSO7uFvu356De75mI5iohZNj5wB5Wju71pBiorTKVj4+iJ4e+xVIzFdG
hFC0DehNcl2t9w/y8qHwIQ1yUAjXHLXq0/2jsVeH6bU5q/MsgvUP1jcFe0eyOpxy
CDvyFdzZFbI57TnB/fvcZTRZ5ewXMFpH8gzuoFzAjUAP95UjYKgaGdrNPNIy28Ii
4zhvdghei2+n9jgiMfcGQg8lyfH5yF0vWWWynX0KcJsRwEZoL2EauVdwq4PcYOoU
pQFhpcreCjD4LdZ4yRU4InbhcUogXjrQ9Dz01TbPmQD5b5iso21bCEFBXrhzABEB
AAGJAjwEGAEKACYCGwwWIQQVim9TDqIC5fZRYRMU+upjRI4d+QUCaUbs9wUJCRwg
hwAKCRAU+upjRI4d+X/XD/sH5xvHPfTJq52v8weFmB52up+DzqG2lyhGdoUQ1Muw
dRDLTLXLJrFdfpoOo7/j4Scr0rdc7/dpCn0DLcPuCoPxu+SkjEnVehFmZrGSv7Ga
x9dHr3DBh42fdlX/U/EnDuyosY0JU1gNF2/6FIA+bTTOFE3RxfN906RjslYQDjMZ
UAlSeLYHOZofdltI0YIr32vrxgdWQGZXPxU4XusDUc0z163OO+TGg7iUNWFZP5Qj
ubM7e0YbDX0NPIshk8us99YJmrWnhaix1/W5ryO3DXiGaQ7XFi9u7QofRqvRIctg
QXavdepkzJow9V9qpMECAJePIuICq7rm+xy+njjbuF436W7390bfVBwRr+FPADsl
jgQP4KvY5rykss30kheom8wNEbveWkhH5oTfH9b7O4KXJfpfJzrlgOWp2BD9JL8t
/M4HvFXTr2a75H/QbHK5OFrZeGATuv9OTxv7EZvnrPXU+DYTFldpu7TrNNqKCoj3
ZyXmc3Hhg5kskDhfHJppaeOayuhMOpT3ud1MFzROY5SLVIH8rBR12KUgsCUYQcGs
Iy0+0QvEGkjb4cAH1NK3VlbqVNsy1RmqRt2B28R2ueewDfTOoqkzt4MmzLqTdnAx
mTqmHmkEKhEf3K4MRNUPO2yieUg2COk5l6x9HhAnoxxeOZrTmcMsPY/UViG2HEPm
ybkCDQRj7TRDARAA9DZuKdfKq4Bs2+NwxC0aplljWOl8VIsEVg+Q8agD7/HU6/b6
Dry0njtWybn2x6Axf/nUdeOC01Fi1lmht/fpj6mRkgAvd/V6P10xnsUoykPSDSTh
P25MFFGW3JAA82bwdJ4AJpEQvTZG2nTb3237vlBiI1qHQrac8GYkju2O4UfySRN6
7cyi7bMf2pjWBBOEhaNy4b6CMDsb32P/N5J7sTE/TXgrS+u4ITIgjzSrkUkh5Z+B
8QVRa7xPIDZJdvZWTEXWu5fgRPZvxbr154GIkWJkFzlDoB1UcO56/uzRUuKhEV6o
HW3LMUuWdPMjpHpq8hrL0G2rDniJFUtbDFzHdZK1LUU3T2BJM8rjI3D/euph+IDT
27vl5qo72zCYE/iKzx4FMLZcQvx1kUAxkPX8l+dzZEwKeRIIpFDxQvatRtl+z0bM
jbkpDb+Yjv66sC4dYRpgTTGX6rok0PWHR3IxDNzyf2j8zQ4LFJ+rVBM1GjGSt6mG
j9TeL8CVeiSp4SuJ7I/FJVPHsKb50m+BDzeB31qTydNqh2kKr0DVAUa+TUsCr7e0
OYr8WE2adJcRXIW0qw50xXF+W7/05GqSCVD0dpeOUdBTQTsSkQmM3/0hcj9aVo9e
UDCM9RF0WRqiDAoHzJFfg+ztamkQI5HO6CklC4Ok22qrHRf6HDNYSuT6QFkAEQEA
AYkCPQQYAQoAJwMbIAQWIQQVim9TDqIC5fZRYRMU+upjRI4d+QUCaUbs9wUJCRwf
tAAKCRAU+upjRI4d+Y+cD/9yllG6uo934pcHNsVppZBfREFwSc8ywlbosCuSVpay
PjSqgrWwDrnqrsk0F2kUdC6rR3BIcXbn+lA9KqylH+cCXAJCkh8EDq6TlQ7Lt5EV
w1U0MAMXOyxPwDymQ/BO+iDyjXWkRRYgbF5XiFhCfGeuKyhkhACisAgNZ1uA1P5k
0SJYc14YfEhQkB46Y20SpfVHRsQ46FyNB6GHbmTmfoO8La8VTh++7GBdh85HfvkG
VNQ3wpi5oXsOLN9+MJOezc0XsW2LQsKQj1/J7QKzGh+lxN5cemsA5aqPzh8dyxeT
0lYRFp4AHkimqGUomVpRkbegMIPxXqOE+ZAmsddErw0UtmrKxcmMptOJwNgYzEgu
++2vtqerL/NYp+wsdcWaBjCz2F3NiwHgNli7NSB/FPwucZZ5gN5C4SnmeFzrGdHg
Oy+tQUN6ayQKljHeBO7CjMlsFNo/dcVrEMa1ShxBMqlj/6ivoEhktLz0Nru4FwNU
xE5SJYDYfpjD7Ws8y4LoXgWXjFHrMO6N9GzqLN/e8LT7I+w4ps2MrgJ8QSrelmQ3
rjkxp3uWp5v2lqy4rLfpi9iB6zIAeoN2eU1yOM9joxOYMxKYaYeYyP1Mm90wFol8
LcTSaN+tVniPddBiL6zvsGBEMbCR9XN3EQ+mErbuw5ovWBOCrr+dvN3FxvD11y4J
7w==
=mXYP
-----END PGP PUBLIC KEY BLOCK-----
@@ -1,51 +0,0 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQENBFZu2zwBCADfatenjH3cvhlU6AeInvp4R0JmPBG942aghFj1Qh57smRcO5Bv
y9mqrX3UDdmVvu58V3k1k9/GzPnAG1t+c7ohdymv/AMuNY4pE2sfxx7bX+mncTHX
5wthipn8kTNm4WjREjCJM1Bm5sozzEZetED3+0/dWlnHl8b38evnLsD+WbSrDPVp
o6M6Eg9IfMwTfcXzdmLmSnGolBWDQ9i1a0x0r3o+sDW5UTnr7jVP+zILcnOZ1Ewl
Rn9OJ4Qg3ULM7WTMDYpKH4BO7RLR3aJgmsFAHp17vgUnzzFBZ10MCS3UOyUNoyph
xo3belf7Q9nrHcSNbqSeQuBnW/vafAZUreAlABEBAAG0IkVlbGNvIERvbHN0cmEg
PGVkb2xzdHJhQGdtYWlsLmNvbT6JATwEEwEIACYCGyMHCwkIBwMCAQYVCAIJCgsE
FgIDAQIeAQIXgAUCVm7etAIZAQAKCRCBcLRybXGY3q51B/96qt41tmcDSzrj/UTl
O6rErfW5zFvVsJTZ95Duwu87t/DVhw5lKBQcjALqVddufw1nMzyN/tSOMVDW8xe4
wMEdcU4+QAMzNX80enuyinsw1glxfLcK0+VbTvqNIfw0sG3MjPqNs6cK2VRfMHK4
paJjytBVICszNX9TfjLyIpKKoSSo1vqnT47LDZ5GIMy7l9Cs2sO/rqQHSPcR79yz
8m8tbHpDDEMZmJeklckKP2QoiqnHiIvlisDxLclYnUmNaPdaN/f++qZz5Yqvu1n+
sNUBA5eLaZH64Uy2SwtABxO3JPJ8nQ2+SFZ7ocFm4Gcdv4aM+Ura9S6fvM91tEJp
yAQOiQE5BBMBCAAjBQJWbts8AhsjBwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AA
CgkQgXC0cm1xmN6sIAgAielxO8zJREqEkA2xudg/o4e9ZlNZ3X1NvY8OzJH/qlB2
SmwKqwifhtbC1K0uavXA7eaxdtd2zrI+Yq7IooUyv7juMjHTZhLcFbR5iVkQ4Mfp
JmeHXJ/ChYKxD5mMj/C3WbCZ91oCSNZ6Iyi5fvQj/691OC4q+y/2NEUcOI8D8cw8
XKHbKtceFYc+nZmdOv3ZZrNTSN/kszGViNNLKgnpPdDVPtLp+vjXtbmitiFG2HL/
WfbJ+3Gh2Yr1Vy3O9dWKH++e1AmIv7WWqmUjRFVpqC/wr7/BLaScWT8WKF5vkshU
gq8Ez1/cuizsgs3wQIZWgXKQK5njvwnbKg+Zmh/uGbQmRWVsY28gRG9sc3RyYSA8
ZWVsY28uZG9sc3RyYUB0d2VhZy5pbz6JAU4EEwEIADgWIQS1QdVTAScOC88Vyl2B
cLRybXGY3gUCXELt4gIbIwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRCBcLRy
bXGY3ujFCADfS5D1xHU8KH6TpqgssSggYVq62Wwn/Ga+4XPPetM+ajcXagyH6SwB
mxlHICcnv9xC93ryiTI10P1ADJl+aBsI66wEdHBU+ty4RTDy4JZNUPtmRCk9LhSc
mtUO3ry/wtWkRLdJxP49hg7BbQvWoU0M6WODp7SJjPKPWNX64mzHBeOuy+DqGCbM
lpGNCvW8ahU/ewbm7+xwWmzqLDoWzXjHsdF4QdzMVM/vkAgWEP4y0wEqFASzIYaR
GNEkBWU4OQVq5Bdm9+wWWAgsbM0FJAQl0GDqnz4QxWzxxCAAXdbh9F5ffafWYsA9
bise4ZQLkvYo6iUnrcFm4dtZbT8iL3gptCtFZWxjbyBEb2xzdHJhIDxlZWxjby5k
b2xzdHJhQGxvZ2ljYmxveC5jb20+iQE5BBMBCAAjBQJWbt6nAhsjBwsJCAcDAgEG
FQgCCQoLBBYCAwECHgECF4AACgkQgXC0cm1xmN4b/wf8DApMV/jSPEpibekrUPQu
Ye3Z8cxBQuRm/nOPowtPEH/ShAevrCdRiob2nuEZWNoqZ2e5/+6ud07Hs9bslvco
cDv1jeY1dof1idxfKhH3kfSpuD2XJhuzQBxBqOrIlCS/rdnW+Y9wOGD7+bs9QpcA
IyAeQGLLkfggAxaGYQ2Aev8pS7i3a/+lOWbFhcTe02I49KemCOJqBorG5FfILLNr
DjO3EoutNGpuz6rZvc/BlymphWBoAdUmxgoObr7NYWgw9pI8WeE6C7bbSOO7p5aQ
spWXU7Hm17DkzsVDpaJlyClllqK+DdKza5oWlBMe/P02jD3Y+0P/2rCCyQQwmH3D
RbkBDQRWbts8AQgA0g556xc08dH5YNEjbCwEt1j+XoRnV4+GfbSJIXOl9joIgzRC
4IaijvL8+4biWvX7HiybfvBKto0XB1AWLZRC3jWKX5p74I77UAcrD+VQ/roWQqlJ
BKbiQMlRYEsj/5Xnf72G90IP4DAFKvNl+rLChe+jUySA91BCtrYoP75Sw1BE9Cyz
xEtm4WUzKAJdXI+ZTBttA2Nbqy+GSuzBs7fSKDwREJaZmVrosvmns+pQVG4WPWf4
0l4mPguDQmZ9wSWZvBDkpG7AgHYDRYRGkMbAGsVfc6cScN2VsSTa6cbeeAEowKxM
qx9RbY3WOq6aKAm0qDvow1nl7WwXwe8K0wQxfQARAQABiQEfBBgBCAAJBQJWbts8
AhsMAAoJEIFwtHJtcZjeuAAH/0YNz2Qe1IAEO5oqEZNFOccL4KxVPrBhWUen83/b
C6PjOnOqv6q5ztAcms88WIKxBlfzIfq+dzJcbKVS/H7TEXgcaC+7EYW8sJVEsipN
BtEZ3LQNJ5coDjm7WZygniah1lfXNuiritAXduK5FWNNndqGArEaeZ8Shzdo/Uyi
b9lOsBIL6xc2ZcnX5f+rTu02LCEtEb0FwCycZLEWYf8hG4k8uttIOZOC+CLk/k8d
kBmPikMwUVTTV0CdT1cemQKdTaoAaK+kurF6FYXwcnjhRlHrisSt/tVMEwTw4LUM
3MYf6qfjjvE4HlDwZal8th7ccoQp/flfJIuRv85xCcKK+PI=
=u5cX
-----END PGP PUBLIC KEY BLOCK-----
-13
View File
@@ -1,13 +0,0 @@
# Maintainer GPG Keys
Release tags are signed by members of the [Nix maintainer team](https://nixos.org/community/teams/nix/) as part of the [release process](../release-process.md). This directory contains the public GPG keys used for signing.
## Keys
- **Eelco Dolstra**
GPG Fingerprint: `B541 D553 0127 0E0B CF15 CA5D 8170 B472 6D71 98DE`
- **Sergei Zimmerman**
GPG Fingerprint: [`158A 6F53 0EA2 02E5 F651 6113 14FA EA63 448E 1DF9`](https://keys.openpgp.org/vks/v1/by-fingerprint/158A6F530EA202E5F651611314FAEA63448E1DF9)
<!-- TODO: Add keys for other Nix team members -->
+11 -13
View File
@@ -5,11 +5,11 @@
The release process is intended to create the following for each
release:
* A signed Git tag (public keys in `maintainers/keys/`)
* A Git tag
* Binary tarballs in https://releases.nixos.org/?prefix=nix/
* Docker images (arm64 and amd64 variants, uploaded to DockerHub and GHCR)
* Docker images
* Closures in https://cache.nixos.org
@@ -39,10 +39,6 @@ release:
* Proof-read / edit / rearrange the release notes if needed. Breaking changes
and highlights should go to the top.
* Run `maintainers/release-credits` to make sure the credits script works
and produces a sensible output. Some emails might not automatically map to
a GitHub handle.
* Push.
```console
@@ -101,17 +97,21 @@ release:
evaluation ID (e.g. `1780832` in
`https://hydra.nixos.org/eval/1780832`).
* Tag the release:
* Tag the release and upload the release artifacts to
[`releases.nixos.org`](https://releases.nixos.org/) and [Docker Hub](https://hub.docker.com/):
```console
$ IS_LATEST=1 ./maintainers/upload-release.pl --skip-docker --skip-s3 --project-root $PWD <EVAL-ID>
$ IS_LATEST=1 ./maintainers/upload-release.pl <EVAL-ID>
```
Note: `IS_LATEST=1` causes the `latest-release` branch to be
force-updated. This is used by the `nixos.org` website to get the
[latest Nix manual](https://nixos.org/manual/nixpkgs/unstable/).
* Trigger the [`upload-release.yml` workflow](https://github.com/NixOS/nix/actions/workflows/upload-release.yml) via `workflow_dispatch` trigger. At the top click `Run workflow` -> select the current release branch from `Use workflow from` -> fill in `Hydra evaluation ID` with `<EVAL-ID>` value from previous steps -> click `Run workflow`. Wait for the run to be approved by `NixOS/nix-team` (or bypass checks if warranted). Wait for the workflow to succeed.
TODO: This script requires the right AWS credentials. Document.
TODO: This script currently requires a
`/home/eelco/Dev/nix-pristine`.
TODO: trigger nixos.org netlify: https://docs.netlify.com/configure-builds/build-hooks/
@@ -176,18 +176,16 @@ release:
* Wait for the desired evaluation of the maintenance jobset to finish
building.
* Tag the release
* Run
```console
$ IS_LATEST=1 ./maintainers/upload-release.pl --skip-docker --skip-s3 --project-root $PWD <EVAL-ID>
$ IS_LATEST=1 ./maintainers/upload-release.pl <EVAL-ID>
```
Omit `IS_LATEST=1` when creating a point release that is not on the
most recent stable branch. This prevents `nixos.org` to going back
to an older release.
* Trigger the [`upload-release.yml` workflow](https://github.com/NixOS/nix/actions/workflows/upload-release.yml) via `workflow_dispatch` trigger. At the top click `Run workflow` -> select the current release branch from `Use workflow from` -> fill in `Hydra evaluation ID` with `<EVAL-ID>` value from previous steps -> click `Run workflow`. Wait for the run to be approved by `NixOS/nix-team` (or bypass checks if warranted). Wait for the workflow to succeed.
* Bump the version number of the release branch as above (e.g. to
`2.12.2`).
+82 -128
View File
@@ -1,8 +1,7 @@
#! /usr/bin/env nix-shell
#! nix-shell -i perl -p awscli2 perl perlPackages.LWPUserAgent perlPackages.LWPProtocolHttps perlPackages.FileSlurp perlPackages.NetAmazonS3 perlPackages.GetoptLongDescriptive gnupg1
#! nix-shell -i perl -p perl perlPackages.LWPUserAgent perlPackages.LWPProtocolHttps perlPackages.FileSlurp perlPackages.NetAmazonS3 gnupg1
use strict;
use Getopt::Long::Descriptive;
use Data::Dumper;
use File::Basename;
use File::Path;
@@ -14,30 +13,7 @@ use Net::Amazon::S3;
delete $ENV{'shell'}; # shut up a LWP::UserAgent.pm warning
my ($opt, $usage) = describe_options(
'%c %o <eval-id>',
[ 'skip-docker', 'Skip Docker image upload' ],
[ 'skip-git', 'Skip Git tagging' ],
[ 'skip-s3', 'Skip S3 upload' ],
[ 'docker-owner=s', 'Docker image owner', { default => 'nixos/nix' } ],
[ 'project-root=s', 'Pristine git repository path' ],
[ 's3-endpoint=s', 'Custom S3 endpoint' ],
[ 's3-host=s', 'S3 host', { default => 's3-eu-west-1.amazonaws.com' } ],
[],
[ 'help|h', 'Show this help message', { shortcircuit => 1 } ],
[],
[ 'Environment variables:' ],
[ 'AWS_ACCESS_KEY_ID' ],
[ 'AWS_SECRET_ACCESS_KEY' ],
[ 'AWS_SESSION_TOKEN For OIDC' ],
[ 'IS_LATEST Set to "1" to mark as latest release' ],
);
print($usage->text), exit if $opt->help;
my $evalId = $ARGV[0] or do { print STDERR $usage->text; exit 1 };
die "--project-root is required unless --skip-git is specified\n" unless $opt->skip_git || $opt->project_root;
my $evalId = $ARGV[0] or die "Usage: $0 EVAL-ID\n";
my $releasesBucketName = "nix-releases";
my $channelsBucketName = "nix-channels";
@@ -86,38 +62,25 @@ File::Path::make_path($narCache);
my $binaryCache = "https://cache.nixos.org/?local-nar-cache=$narCache";
# S3 setup.
my $aws_access_key_id = $ENV{'AWS_ACCESS_KEY_ID'};
my $aws_secret_access_key = $ENV{'AWS_SECRET_ACCESS_KEY'};
my $aws_session_token = $ENV{'AWS_SESSION_TOKEN'};
my $aws_access_key_id = $ENV{'AWS_ACCESS_KEY_ID'} or die "No AWS_ACCESS_KEY_ID given.";
my $aws_secret_access_key = $ENV{'AWS_SECRET_ACCESS_KEY'} or die "No AWS_SECRET_ACCESS_KEY given.";
my ($s3, $releasesBucket, $s3_channels, $channelsBucket);
my $s3 = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
retry => 1,
host => "s3-eu-west-1.amazonaws.com",
});
unless ($opt->skip_s3) {
$aws_access_key_id or die "No AWS_ACCESS_KEY_ID given.";
$aws_secret_access_key or die "No AWS_SECRET_ACCESS_KEY given.";
my $releasesBucket = $s3->bucket($releasesBucketName) or die;
$s3 = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
$aws_session_token ? (aws_session_token => $aws_session_token) : (),
retry => 1,
host => $opt->s3_host,
secure => ($opt->s3_endpoint && $opt->s3_endpoint =~ /^http:/) ? 0 : 1,
});
my $s3_us = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
retry => 1,
});
$releasesBucket = $s3->bucket($releasesBucketName) or die;
$s3_channels = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
$aws_session_token ? (aws_session_token => $aws_session_token) : (),
retry => 1,
$opt->s3_endpoint ? (host => $opt->s3_host) : (),
$opt->s3_endpoint ? (secure => ($opt->s3_endpoint =~ /^http:/) ? 0 : 1) : (),
});
$channelsBucket = $s3_channels->bucket($channelsBucketName) or die;
}
my $channelsBucket = $s3_us->bucket($channelsBucketName) or die;
sub getStorePath {
my ($jobName, $output) = @_;
@@ -152,12 +115,11 @@ sub copyManual {
File::Path::remove_tree("$tmpDir/manual.tmp", {safe => 1});
}
my $awsEndpoint = $opt->s3_endpoint ? "--endpoint-url " . $opt->s3_endpoint : "";
system("aws $awsEndpoint s3 sync '$tmpDir/manual' s3://$releasesBucketName/$releaseDir/manual") == 0
system("aws s3 sync '$tmpDir/manual' s3://$releasesBucketName/$releaseDir/manual") == 0
or die "syncing manual to S3\n";
}
copyManual unless $opt->skip_s3;
copyManual;
sub downloadFile {
my ($jobName, $productNr, $dstName) = @_;
@@ -196,12 +158,30 @@ sub downloadFile {
return $sha256_expected;
}
# Upload docker images.
downloadFile("binaryTarball.i686-linux", "1");
downloadFile("binaryTarball.x86_64-linux", "1");
downloadFile("binaryTarball.aarch64-linux", "1");
downloadFile("binaryTarball.x86_64-darwin", "1");
downloadFile("binaryTarball.aarch64-darwin", "1");
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv6l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv7l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.riscv64-unknown-linux-gnu", "1");
};
warn "$@" if $@;
downloadFile("installerScript", "1");
# Upload docker images to dockerhub.
my $dockerManifest = "";
my $dockerManifestLatest = "";
my $haveDocker = 0;
unless ($opt->skip_docker) {
for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
my $system = $platforms->[0];
my $dockerPlatform = $platforms->[1];
@@ -215,8 +195,8 @@ for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
print STDERR "loading docker image for $dockerPlatform...\n";
system("docker load -i $tmpDir/$fn") == 0 or die;
my $tag = $opt->docker_owner . ":$version-$dockerPlatform";
my $latestTag = $opt->docker_owner . ":latest-$dockerPlatform";
my $tag = "nixos/nix:$version-$dockerPlatform";
my $latestTag = "nixos/nix:latest-$dockerPlatform";
print STDERR "tagging $version docker image for $dockerPlatform...\n";
system("docker tag nix:$version $tag") == 0 or die;
@@ -239,94 +219,68 @@ for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
}
if ($haveDocker) {
my $dockerOwner = $opt->docker_owner;
print STDERR "creating multi-platform docker manifest...\n";
system("docker manifest rm $dockerOwner:$version");
system("docker manifest create $dockerOwner:$version $dockerManifest") == 0 or die;
system("docker manifest rm nixos/nix:$version");
system("docker manifest create nixos/nix:$version $dockerManifest") == 0 or die;
if ($isLatest) {
print STDERR "creating latest multi-platform docker manifest...\n";
system("docker manifest rm $dockerOwner:latest");
system("docker manifest create $dockerOwner:latest $dockerManifestLatest") == 0 or die;
system("docker manifest rm nixos/nix:latest");
system("docker manifest create nixos/nix:latest $dockerManifestLatest") == 0 or die;
}
print STDERR "pushing multi-platform docker manifest...\n";
system("docker manifest push $dockerOwner:$version") == 0 or die;
system("docker manifest push nixos/nix:$version") == 0 or die;
if ($isLatest) {
print STDERR "pushing latest multi-platform docker manifest...\n";
system("docker manifest push $dockerOwner:latest") == 0 or die;
system("docker manifest push nixos/nix:latest") == 0 or die;
}
}
}
# Upload nix-fallback-paths.nix.
write_file("$tmpDir/fallback-paths.nix",
"{\n" .
" x86_64-linux = \"" . getStorePath("build.nix-everything.x86_64-linux") . "\";\n" .
" i686-linux = \"" . getStorePath("build.nix-everything.i686-linux") . "\";\n" .
" aarch64-linux = \"" . getStorePath("build.nix-everything.aarch64-linux") . "\";\n" .
" riscv64-linux = \"" . getStorePath("buildCross.nix-everything.riscv64-unknown-linux-gnu.x86_64-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.nix-everything.x86_64-darwin") . "\";\n" .
" aarch64-darwin = \"" . getStorePath("build.nix-everything.aarch64-darwin") . "\";\n" .
"}\n");
# Upload release files to S3.
unless ($opt->skip_s3) {
downloadFile("binaryTarball.i686-linux", "1");
downloadFile("binaryTarball.x86_64-linux", "1");
downloadFile("binaryTarball.aarch64-linux", "1");
downloadFile("binaryTarball.x86_64-darwin", "1");
downloadFile("binaryTarball.aarch64-darwin", "1");
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv6l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv7l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.riscv64-unknown-linux-gnu", "1");
};
warn "$@" if $@;
downloadFile("installerScript", "1");
for my $fn (glob "$tmpDir/*") {
my $name = basename($fn);
next if $name eq "manual";
my $dstKey = "$releaseDir/" . $name;
unless (defined $releasesBucket->head_key($dstKey)) {
print STDERR "uploading $fn to s3://$releasesBucketName/$dstKey...\n";
# Upload nix-fallback-paths.nix.
write_file("$tmpDir/fallback-paths.nix",
"{\n" .
" x86_64-linux = \"" . getStorePath("build.nix-everything.x86_64-linux") . "\";\n" .
" i686-linux = \"" . getStorePath("build.nix-everything.i686-linux") . "\";\n" .
" aarch64-linux = \"" . getStorePath("build.nix-everything.aarch64-linux") . "\";\n" .
" riscv64-linux = \"" . getStorePath("buildCross.nix-everything.riscv64-unknown-linux-gnu.x86_64-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.nix-everything.x86_64-darwin") . "\";\n" .
" aarch64-darwin = \"" . getStorePath("build.nix-everything.aarch64-darwin") . "\";\n" .
"}\n");
my $configuration = ();
$configuration->{content_type} = "application/octet-stream";
for my $fn (glob "$tmpDir/*") {
my $name = basename($fn);
next if $name eq "manual";
my $dstKey = "$releaseDir/" . $name;
unless (defined $releasesBucket->head_key($dstKey)) {
print STDERR "uploading $fn to s3://$releasesBucketName/$dstKey...\n";
my $configuration = ();
$configuration->{content_type} = "application/octet-stream";
if ($fn =~ /.sha256|install|\.nix$/) {
$configuration->{content_type} = "text/plain";
}
$releasesBucket->add_key_filename($dstKey, $fn, $configuration)
or die $releasesBucket->err . ": " . $releasesBucket->errstr;
if ($fn =~ /.sha256|install|\.nix$/) {
$configuration->{content_type} = "text/plain";
}
}
# Update the "latest" symlink.
$channelsBucket->add_key(
"nix-latest/install", "",
{ "x-amz-website-redirect-location" => "https://releases.nixos.org/$releaseDir/install" })
or die $channelsBucket->err . ": " . $channelsBucket->errstr
if $isLatest;
$releasesBucket->add_key_filename($dstKey, $fn, $configuration)
or die $releasesBucket->err . ": " . $releasesBucket->errstr;
}
}
# Update the "latest" symlink.
$channelsBucket->add_key(
"nix-latest/install", "",
{ "x-amz-website-redirect-location" => "https://releases.nixos.org/$releaseDir/install" })
or die $channelsBucket->err . ": " . $channelsBucket->errstr
if $isLatest;
# Tag the release in Git.
unless ($opt->skip_git) {
chdir($opt->project_root) or die "Cannot chdir to " . $opt->project_root . ": $!";
system("git remote update origin") == 0 or die;
system("git tag --force --sign $version $nixRev -m 'Tagging release $version'") == 0 or die;
system("git push origin refs/tags/$version") == 0 or die;
system("git push --force-with-lease origin $nixRev:refs/heads/latest-release") == 0 or die if $isLatest;
}
chdir("/home/eelco/Dev/nix-pristine") or die;
system("git remote update origin") == 0 or die;
system("git tag --force --sign $version $nixRev -m 'Tagging release $version'") == 0 or die;
system("git push --tags") == 0 or die;
system("git push --force-with-lease origin $nixRev:refs/heads/latest-release") == 0 or die if $isLatest;
File::Path::remove_tree($narCache, {safe => 1});
File::Path::remove_tree($tmpDir, {safe => 1});
+1 -2
View File
@@ -8,7 +8,6 @@ project(
subproject_dir : 'src',
default_options : [
'localstatedir=/nix/var',
# hack for trailing newline
],
meson_version : '>= 1.1',
)
@@ -29,7 +28,7 @@ subproject('nix')
if get_option('doc-gen')
subproject('internal-api-docs')
subproject('external-api-docs')
if not meson.is_cross_build()
if meson.can_run_host_binaries()
subproject('nix-manual')
endif
endif
+7
View File
@@ -20,3 +20,10 @@ option(
value : true,
description : 'Build language bindings (e.g. Perl)',
)
option(
'benchmarks',
type : 'boolean',
value : false,
description : 'Build benchmarks (requires gbenchmark)',
)
+1 -1
View File
@@ -9,5 +9,5 @@ configure_file(
# 'storedir' : store_dir,
# 'localstatedir' : localstatedir,
# 'bindir' : bindir,
},
},
)
+1 -1
View File
@@ -2,5 +2,5 @@ if host_machine.system() == 'windows'
# libexpr's primops creates a large object
# Without the following flag, we'll get errors when cross-compiling to mingw32:
# Fatal error: can't write 66 bytes to section .text of src/libexpr/libnixexpr.dll.p/primops.cc.obj: 'file too big'
add_project_arguments([ '-Wa,-mbig-obj' ], language: 'cpp')
add_project_arguments([ '-Wa,-mbig-obj' ], language : 'cpp')
endif
@@ -18,3 +18,25 @@ add_project_arguments(
'-Wno-deprecated-declarations',
language : 'cpp',
)
# GCC doesn't benefit much from precompiled headers.
do_pch = cxx.get_id() == 'clang'
# This is a clang-only option for improving build times.
# It forces the instantiation of templates in the PCH itself and
# not every translation unit it's included in.
# It's available starting from clang 11, which is old enough to not
# bother checking the version.
# This feature helps in particular with the expensive nlohmann::json template
# instantiations in libutil and libstore.
if cxx.get_id() == 'clang'
add_project_arguments('-fpch-instantiate-templates', language : 'cpp')
endif
# Clang gets grumpy about missing libasan symbols if -shared-libasan is not
# passed when building shared libs, at least on Linux
if cxx.get_id() == 'clang' and ('address' in get_option('b_sanitize') or 'undefined' in get_option(
'b_sanitize',
))
add_project_link_arguments('-shared-libasan', language : 'cpp')
endif
@@ -6,7 +6,7 @@
# *interface*.
#
# See `man pkg-config` for some details.
deps_private = [ ]
deps_private = []
# These are public dependencies with pkg-config files. Public is the
# opposite of private: these dependencies are used in installed header
@@ -23,14 +23,14 @@ deps_private = [ ]
# N.B. For distributions that care about "ABI" stability and not just
# "API" stability, the private dependencies also matter as they can
# potentially affect the public ABI.
deps_public = [ ]
deps_public = []
# These are subproject deps (type == "internal"). They are other
# packages in `/src` in this repo. The private vs public distinction is
# the same as above.
deps_private_subproject = [ ]
deps_public_subproject = [ ]
deps_private_subproject = []
deps_public_subproject = []
# These are dependencencies without pkg-config files. Ideally they are
# just private, but they may also be public (e.g. boost).
deps_other = [ ]
deps_other = []
@@ -5,7 +5,7 @@ if host_machine.system() == 'cygwin' or host_machine.system() == 'windows'
# and not detail with this yet.
#
# TODO do not do this, and instead do fine-grained export annotations.
linker_export_flags = ['-Wl,--export-all-symbols']
linker_export_flags = [ '-Wl,--export-all-symbols' ]
else
linker_export_flags = []
endif
+14 -11
View File
@@ -1,12 +1,12 @@
requires_private = []
foreach dep : deps_private_subproject
requires_private += dep.name()
requires_private += dep.name()
endforeach
requires_private += deps_private
requires_public = []
requires_public = []
foreach dep : deps_public_subproject
requires_public += dep.name()
requires_public += dep.name()
endforeach
requires_public += deps_public
@@ -14,7 +14,7 @@ extra_pkg_config_variables = get_variable('extra_pkg_config_variables', {})
extra_cflags = []
if not meson.project_name().endswith('-c')
extra_cflags += ['-std=c++2a']
extra_cflags += [ '-std=c++2a' ]
endif
import('pkgconfig').generate(
@@ -29,10 +29,13 @@ import('pkgconfig').generate(
variables : extra_pkg_config_variables,
)
meson.override_dependency(meson.project_name(), declare_dependency(
include_directories : include_dirs,
link_with : this_library,
compile_args : ['-std=c++2a'],
dependencies : deps_public_subproject + deps_public,
variables : extra_pkg_config_variables,
))
meson.override_dependency(
meson.project_name(),
declare_dependency(
include_directories : include_dirs,
link_with : this_library,
compile_args : [ '-std=c++2a' ],
dependencies : deps_public_subproject + deps_public,
variables : extra_pkg_config_variables,
),
)
@@ -1,7 +1,12 @@
bash = find_program('bash', native: true)
bash = find_program('bash', native : true)
gen_header = generator(
bash,
arguments : [ '-c', '{ echo \'R"__NIX_STR(\' && cat @INPUT@ && echo \')__NIX_STR"\'; } > "$1"', '_ignored_argv0', '@OUTPUT@' ],
arguments : [
'-c',
'{ echo \'R"__NIX_STR(\' && cat @INPUT@ && echo \')__NIX_STR"\'; } > "$1"',
'_ignored_argv0',
'@OUTPUT@',
],
output : '@PLAINNAME@.gen.hh',
)
+22 -8
View File
@@ -54,12 +54,12 @@ let
preConfigure =
prevAttrs.preConfigure or ""
+
# Update the repo-global .version file.
# Symlink ./.version points there, but by default only workDir is writable.
''
chmod u+w ./.version
echo ${finalAttrs.version} > ./.version
'';
# Update the repo-global .version file.
# Symlink ./.version points there, but by default only workDir is writable.
''
chmod u+w ./.version
echo ${finalAttrs.version} > ./.version
'';
};
localSourceLayer =
@@ -148,8 +148,7 @@ let
nativeBuildInputs = [
meson
ninja
]
++ prevAttrs.nativeBuildInputs or [ ];
] ++ prevAttrs.nativeBuildInputs or [ ];
mesonCheckFlags = prevAttrs.mesonCheckFlags or [ ] ++ [
"--print-errorlogs"
];
@@ -366,18 +365,33 @@ in
nix-cmd = callPackage ../src/libcmd/package.nix { };
/**
The Nix command line interface. Note that this does not include its tests, whereas `nix-everything` does.
*/
nix-cli = callPackage ../src/nix/package.nix { version = fineVersion; };
nix-functional-tests = callPackage ../tests/functional/package.nix {
version = fineVersion;
};
/**
The manual as would be published on https://nix.dev/reference/nix-manual
*/
nix-manual = callPackage ../doc/manual/package.nix { version = fineVersion; };
/**
Doxygen pages for C++ code
*/
nix-internal-api-docs = callPackage ../src/internal-api-docs/package.nix { version = fineVersion; };
/**
Doxygen pages for the public C API
*/
nix-external-api-docs = callPackage ../src/external-api-docs/package.nix { version = fineVersion; };
nix-perl-bindings = callPackage ../src/perl/package.nix { };
/**
Combined package that has the CLI, libraries, and (assuming non-cross, no overrides) it requires that all tests succeed.
*/
nix-everything = callPackage ../packaging/everything.nix { } // {
# Note: no `passthru.overrideAllMesonComponents` etc
# This would propagate into `nix.overrideAttrs f`, but then discard
+14 -3
View File
@@ -50,9 +50,19 @@ scope: {
requiredSystemFeatures = [ ];
};
boehmgc = pkgs.boehmgc.override {
enableLargeConfig = true;
};
boehmgc =
(pkgs.boehmgc.override {
enableLargeConfig = true;
}).overrideAttrs
(attrs: {
# Increase the initial mark stack size to avoid stack
# overflows, since these inhibit parallel marking (see
# GC_mark_some()). To check whether the mark stack is too
# small, run Nix with GC_PRINT_STATS=1 and look for messages
# such as `Mark stack overflow`, `No room to copy back mark
# stack`, and `Grew mark stack to ... frames`.
NIX_CFLAGS_COMPILE = "-DINITIAL_MARK_STACK_SIZE=1048576";
});
# TODO Hack until https://github.com/NixOS/nixpkgs/issues/45462 is fixed.
boost =
@@ -62,6 +72,7 @@ scope: {
"--with-context"
"--with-coroutine"
"--with-iostreams"
"--with-url"
];
enableIcu = false;
}).overrideAttrs
+13 -11
View File
@@ -71,16 +71,17 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
# We use this shell with the local checkout, not unpackPhase.
src = null;
env = {
# For `make format`, to work without installing pre-commit
_NIX_PRE_COMMIT_HOOKS_CONFIG = "${(pkgs.formats.yaml { }).generate "pre-commit-config.yaml"
modular.pre-commit.settings.rawConfig
}";
}
// lib.optionalAttrs stdenv.hostPlatform.isLinux {
CC_LD = "mold";
CXX_LD = "mold";
};
env =
{
# For `make format`, to work without installing pre-commit
_NIX_PRE_COMMIT_HOOKS_CONFIG = "${(pkgs.formats.yaml { }).generate "pre-commit-config.yaml"
modular.pre-commit.settings.rawConfig
}";
}
// lib.optionalAttrs stdenv.hostPlatform.isLinux {
CC_LD = "mold";
CXX_LD = "mold";
};
mesonFlags =
map (transformFlag "libutil") (ignoreCrossFile pkgs.nixComponents2.nix-util.mesonFlags)
@@ -125,7 +126,8 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
++ lib.optional stdenv.hostPlatform.isLinux pkgs.buildPackages.mold-wrapped;
buildInputs =
attrs.buildInputs or [ ]
[ pkgs.gbenchmark ]
++ attrs.buildInputs or [ ]
++ pkgs.nixComponents2.nix-util.buildInputs
++ pkgs.nixComponents2.nix-store.buildInputs
++ pkgs.nixComponents2.nix-store-tests.externalBuildInputs
+32 -31
View File
@@ -47,25 +47,25 @@
}:
let
libs = {
inherit
nix-util
nix-util-c
nix-store
nix-store-c
nix-fetchers
nix-fetchers-c
nix-expr
nix-expr-c
nix-flake
nix-flake-c
nix-main
nix-main-c
nix-cmd
;
}
//
lib.optionalAttrs
libs =
{
inherit
nix-util
nix-util-c
nix-store
nix-store-c
nix-fetchers
nix-fetchers-c
nix-expr
nix-expr-c
nix-flake
nix-flake-c
nix-main
nix-main-c
nix-cmd
;
}
// lib.optionalAttrs
(!stdenv.hostPlatform.isStatic && stdenv.buildPlatform.canExecute stdenv.hostPlatform)
{
# Currently fails in static build
@@ -127,19 +127,20 @@ stdenv.mkDerivation (finalAttrs: {
*/
dontFixup = true;
checkInputs = [
# Make sure the unit tests have passed
nix-util-tests.tests.run
nix-store-tests.tests.run
nix-expr-tests.tests.run
nix-fetchers-tests.tests.run
nix-flake-tests.tests.run
checkInputs =
[
# Make sure the unit tests have passed
nix-util-tests.tests.run
nix-store-tests.tests.run
nix-expr-tests.tests.run
nix-fetchers-tests.tests.run
nix-flake-tests.tests.run
# Make sure the functional tests have passed
nix-functional-tests
]
++
lib.optionals (!stdenv.hostPlatform.isStatic && stdenv.buildPlatform.canExecute stdenv.hostPlatform)
# Make sure the functional tests have passed
nix-functional-tests
]
++ lib.optionals
(!stdenv.hostPlatform.isStatic && stdenv.buildPlatform.canExecute stdenv.hostPlatform)
[
# Perl currently fails in static build
# TODO: Split out tests into a separate derivation?
+3 -1
View File
@@ -240,7 +240,9 @@ in
# System tests.
tests =
import ../tests/nixos {
inherit lib nixpkgs nixpkgsFor;
inherit lib nixpkgs;
pkgs = nixpkgsFor.x86_64-linux.native;
nixComponents = nixpkgsFor.x86_64-linux.native.nixComponents2;
inherit (self.inputs) nixpkgs-23-11;
}
// {
+10 -8
View File
@@ -26,8 +26,10 @@ if [ -z "$HOME" ]; then
exit 1
fi
OS="$(uname -s)"
# macOS support for 10.12.6 or higher
if [ "$(uname -s)" = "Darwin" ]; then
if [ "$OS" = "Darwin" ]; then
IFS='.' read -r macos_major macos_minor macos_patch << EOF
$(sw_vers -productVersion)
EOF
@@ -39,11 +41,11 @@ EOF
fi
# Determine if we could use the multi-user installer or not
if [ "$(uname -s)" = "Linux" ]; then
echo "Note: a multi-user installation is possible. See https://nixos.org/manual/nix/stable/installation/installing-binary.html#multi-user-installation" >&2
if [ "$OS" = "Linux" ]; then
echo "Note: a multi-user installation is possible. See https://nix.dev/manual/nix/stable/installation/installing-binary.html#multi-user-installation" >&2
fi
case "$(uname -s)" in
case "$OS" in
"Darwin")
INSTALL_MODE=daemon;;
*)
@@ -60,7 +62,7 @@ while [ $# -gt 0 ]; do
ACTION=install
;;
--no-daemon)
if [ "$(uname -s)" = "Darwin" ]; then
if [ "$OS" = "Darwin" ]; then
printf '\e[1;31mError: --no-daemon installs are no-longer supported on Darwin/macOS!\e[0m\n' >&2
exit 1
fi
@@ -96,7 +98,7 @@ while [ $# -gt 0 ]; do
echo " providing multi-user support and better isolation for local builds."
echo " Both for security and reproducibility, this method is recommended if"
echo " supported on your platform."
echo " See https://nixos.org/manual/nix/stable/installation/installing-binary.html#multi-user-installation"
echo " See https://nix.dev/manual/nix/stable/installation/installing-binary.html#multi-user-installation"
echo ""
echo " --no-daemon: Simple, single-user installation that does not require root and is"
echo " trivial to uninstall."
@@ -144,7 +146,7 @@ if ! [ -e "$dest" ]; then
fi
if ! [ -w "$dest" ]; then
echo "$0: directory $dest exists, but is not writable by you. This could indicate that another user has already performed a single-user installation of Nix on this system. If you wish to enable multi-user support see https://nixos.org/manual/nix/stable/installation/multi-user.html. If you wish to continue with a single-user install for $USER please run 'chown -R $USER $dest' as root." >&2
echo "$0: directory $dest exists, but is not writable by you. This could indicate that another user has already performed a single-user installation of Nix on this system. If you wish to enable multi-user support see https://nix.dev/manual/nix/stable/installation/multi-user.html. If you wish to continue with a single-user install for $USER please run 'chown -R $USER $dest' as root." >&2
exit 1
fi
@@ -167,7 +169,7 @@ for i in $(cd "$self/store" >/dev/null && echo ./*); do
rm -rf "$i_tmp"
fi
if ! [ -e "$dest/store/$i" ]; then
if [ "$(uname -s)" = "Darwin" ]; then
if [ "$OS" = "Darwin" ] || [ "$OS" = "FreeBSD" ]; then
cp -RPp "$self/store/$i" "$i_tmp"
else
cp -RP --preserve=ownership,timestamps "$self/store/$i" "$i_tmp"
+4 -4
View File
@@ -2,19 +2,19 @@ configure_file(
input : 'nix-profile.sh.in',
output : 'nix-profile.sh',
configuration : {
'localstatedir': localstatedir,
}
'localstatedir' : localstatedir,
},
)
foreach rc : [ '.sh', '.fish', '-daemon.sh', '-daemon.fish' ]
configure_file(
input : 'nix-profile' + rc + '.in',
input : 'nix-profile' + rc + '.in',
output : 'nix' + rc,
install : true,
install_dir : get_option('profile-dir'),
install_mode : 'rw-r--r--',
configuration : {
'localstatedir': localstatedir,
'localstatedir' : localstatedir,
},
)
endforeach
@@ -1,22 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ ! -d out ]]; then
echo "run prepare-installer-for-github-actions first"
exit 1
fi
cd out
PORT=${PORT:-8126}
nohup python -m http.server "$PORT" >/dev/null 2>&1 &
pid=$!
while ! curl -s "http://localhost:$PORT"; do
sleep 1
if ! kill -0 $pid; then
echo "Failed to start http server"
exit 1
fi
done
echo 'To install nix, run the following command:'
echo "sh <(curl http://localhost:$PORT/install) --tarball-url-prefix http://localhost:$PORT"
+4 -3
View File
@@ -1,4 +1,5 @@
project('nix-external-api-docs',
project(
'nix-external-api-docs',
version : files('.version'),
meson_version : '>= 1.1',
license : 'LGPL-2.1-or-later',
@@ -10,7 +11,7 @@ doxygen_cfg = configure_file(
input : 'doxygen.cfg.in',
output : 'doxygen.cfg',
configuration : {
'PROJECT_NUMBER': meson.project_version(),
'PROJECT_NUMBER' : meson.project_version(),
'OUTPUT_DIRECTORY' : meson.current_build_dir(),
'src' : fs.parent(fs.parent(meson.project_source_root())),
},
@@ -20,7 +21,7 @@ doxygen = find_program('doxygen', native : true, required : true)
custom_target(
'external-api-docs',
command : [ doxygen , doxygen_cfg ],
command : [ doxygen, doxygen_cfg ],
input : [
doxygen_cfg,
],
+1 -3
View File
@@ -57,9 +57,7 @@ INPUT = \
@src@/libutil/args \
@src@/libutil-tests \
@src@/libutil-test-support/tests \
@src@/nix \
@src@/nix-env \
@src@/nix-store
@src@/nix
# If the MACRO_EXPANSION tag is set to YES, doxygen will expand all macro names
# in the source code. If set to NO, only conditional compilation will be
+4 -3
View File
@@ -1,4 +1,5 @@
project('nix-internal-api-docs',
project(
'nix-internal-api-docs',
version : files('.version'),
meson_version : '>= 1.1',
license : 'LGPL-2.1-or-later',
@@ -10,7 +11,7 @@ doxygen_cfg = configure_file(
input : 'doxygen.cfg.in',
output : 'doxygen.cfg',
configuration : {
'PROJECT_NUMBER': meson.project_version(),
'PROJECT_NUMBER' : meson.project_version(),
'OUTPUT_DIRECTORY' : meson.current_build_dir(),
'BUILD_ROOT' : meson.build_root(),
'src' : fs.parent(fs.parent(meson.project_source_root())) / 'src',
@@ -21,7 +22,7 @@ doxygen = find_program('doxygen', native : true, required : true)
custom_target(
'internal-api-docs',
command : [ doxygen , doxygen_cfg ],
command : [ doxygen, doxygen_cfg ],
input : [
doxygen_cfg,
],
@@ -82,7 +82,7 @@ private:
};
/**
* @param baseDir Optional [base directory](https://nixos.org/manual/nix/unstable/glossary#gloss-base-directory)
* @param baseDir Optional [base directory](https://nix.dev/manual/nix/development/glossary#gloss-base-directory)
*/
SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * baseDir = nullptr);
+1 -1
View File
@@ -1,6 +1,6 @@
# Public headers directory
include_dirs = [include_directories('../..')]
include_dirs = [ include_directories('../..') ]
headers = files(
'built-path.hh',
+16 -7
View File
@@ -1,4 +1,6 @@
project('nix-cmd', 'cpp',
project(
'nix-cmd',
'cpp',
version : files('.version'),
default_options : [
'cpp_std=c++2a',
@@ -16,8 +18,7 @@ subdir('nix-meson-build-support/deps-lists')
configdata = configuration_data()
deps_private_maybe_subproject = [
]
deps_private_maybe_subproject = []
deps_public_maybe_subproject = [
dependency('nix-util'),
dependency('nix-store'),
@@ -31,11 +32,18 @@ subdir('nix-meson-build-support/subprojects')
nlohmann_json = dependency('nlohmann_json', version : '>= 3.9')
deps_public += nlohmann_json
lowdown = dependency('lowdown', version : '>= 0.9.0', required : get_option('markdown'))
lowdown = dependency(
'lowdown',
version : '>= 0.9.0',
required : get_option('markdown'),
)
deps_private += lowdown
configdata.set('HAVE_LOWDOWN', lowdown.found().to_int())
# The API changed slightly around terminal initialization.
configdata.set('HAVE_LOWDOWN_1_4', lowdown.version().version_compare('>= 1.4.0').to_int())
configdata.set(
'HAVE_LOWDOWN_1_4',
lowdown.version().version_compare('>= 1.4.0').to_int(),
)
readline_flavor = get_option('readline-flavor')
if readline_flavor == 'editline'
@@ -50,7 +58,7 @@ endif
configdata.set(
'USE_READLINE',
(readline_flavor == 'readline').to_int(),
description: 'Use readline instead of editline',
description : 'Use readline instead of editline',
)
config_priv_h = configure_file(
@@ -89,9 +97,10 @@ this_library = library(
config_priv_h,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
link_args: linker_export_flags,
link_args : linker_export_flags,
prelink : true, # For C++ static initializers
install : true,
cpp_pch : do_pch ? [ 'pch/precompiled-headers.hh' ] : [],
)
install_headers(headers, subdir : 'nix/cmd', preserve_path : true)
+3 -3
View File
@@ -2,14 +2,14 @@
option(
'markdown',
type: 'feature',
description: 'Enable Markdown rendering in the Nix binary (requires lowdown)',
type : 'feature',
description : 'Enable Markdown rendering in the Nix binary (requires lowdown)',
)
option(
'readline-flavor',
type : 'combo',
choices : ['editline', 'readline'],
choices : [ 'editline', 'readline' ],
value : 'editline',
description : 'Which library to use for nice line editing with the Nix language REPL',
)
+1 -2
View File
@@ -53,8 +53,7 @@ mkMesonLibrary (finalAttrs: {
buildInputs = [
({ inherit editline readline; }.${readlineFlavor})
]
++ lib.optional enableMarkdown lowdown;
] ++ lib.optional enableMarkdown lowdown;
propagatedBuildInputs = [
nix-util
+4
View File
@@ -0,0 +1,4 @@
#include "nix/cmd/installables.hh"
#include "nix/expr/eval.hh"
#include "nix/util/util.hh"
#include "nix/flake/flake.hh"
+5 -3
View File
@@ -1,4 +1,6 @@
project('nix-expr-c', 'cpp',
project(
'nix-expr-c',
'cpp',
version : files('.version'),
default_options : [
'cpp_std=c++2a',
@@ -33,7 +35,7 @@ sources = files(
'nix_api_value.cc',
)
include_dirs = [include_directories('.')]
include_dirs = [ include_directories('.') ]
headers = files(
'nix_api_expr.h',
@@ -50,7 +52,7 @@ this_library = library(
sources,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
link_args: linker_export_flags,
link_args : linker_export_flags,
prelink : true, # For C++ static initializers
install : true,
)
@@ -6,5 +6,4 @@ headers = files(
'libexpr.hh',
'nix_api_expr.hh',
'value/context.hh',
# hack for trailing newline
)
+5 -4
View File
@@ -1,4 +1,6 @@
project('nix-expr-test-support', 'cpp',
project(
'nix-expr-test-support',
'cpp',
version : files('.version'),
default_options : [
'cpp_std=c++2a',
@@ -14,8 +16,7 @@ cxx = meson.get_compiler('cpp')
subdir('nix-meson-build-support/deps-lists')
deps_private_maybe_subproject = [
]
deps_private_maybe_subproject = []
deps_public_maybe_subproject = [
dependency('nix-util'),
dependency('nix-util-test-support'),
@@ -47,7 +48,7 @@ this_library = library(
include_directories : include_dirs,
# TODO: Remove `-lrapidcheck` when https://github.com/emil-e/rapidcheck/pull/326
# is available. See also ../libutil/build.meson
link_args: linker_export_flags + ['-lrapidcheck'],
link_args : linker_export_flags + [ '-lrapidcheck' ],
prelink : true, # For C++ static initializers
install : true,
)
+7 -6
View File
@@ -1,4 +1,6 @@
project('nix-expr-tests', 'cpp',
project(
'nix-expr-tests',
'cpp',
version : files('.version'),
default_options : [
'cpp_std=c++2a',
@@ -19,8 +21,7 @@ deps_private_maybe_subproject = [
dependency('nix-expr-c'),
dependency('nix-expr-test-support'),
]
deps_public_maybe_subproject = [
]
deps_public_maybe_subproject = []
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/export-all-symbols')
@@ -62,7 +63,7 @@ sources = files(
'value/value.cc',
)
include_dirs = [include_directories('.')]
include_dirs = [ include_directories('.') ]
this_exe = executable(
@@ -72,7 +73,7 @@ this_exe = executable(
dependencies : deps_private_subproject + deps_private + deps_other,
include_directories : include_dirs,
# TODO: -lrapidcheck, see ../libutil-support/build.meson
link_args: linker_export_flags + ['-lrapidcheck'],
link_args : linker_export_flags + [ '-lrapidcheck' ],
install : true,
)
@@ -80,7 +81,7 @@ test(
meson.project_name(),
this_exe,
env : {
'_NIX_TEST_UNIT_DATA': meson.current_source_dir() / 'data',
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
},
protocol : 'gtest',
)
+3 -2
View File
@@ -15,8 +15,6 @@
# include <pthread_np.h>
# endif
# include <gc/gc.h>
# include <gc/gc_cpp.h>
# include <gc/gc_allocator.h>
# include <boost/coroutine2/coroutine.hpp>
@@ -53,6 +51,9 @@ static inline void initGCReal()
GC_INIT();
/* Enable parallel marking. */
GC_allow_register_threads();
/* Register valid displacements in case we are using alignment niches
for storing the type information. This way tagged pointers are considered
to be valid, even when they are not aligned. */
+1 -1
View File
@@ -1812,7 +1812,7 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res
Nix attempted to evaluate a function as a top level expression; in
this case it must have its arguments supplied either by default
values, or passed explicitly with '--arg' or '--argstr'. See
https://nixos.org/manual/nix/stable/language/constructs.html#functions.)",
https://nix.dev/manual/nix/stable/language/syntax.html#functions.)",
symbols[i.name])
.atPos(i.pos)
.withFrame(*fun.lambda().env, *fun.lambda().fun)
+2 -1
View File
@@ -3,12 +3,13 @@
#include <cstddef>
// For `NIX_USE_BOEHMGC`, and if that's set, `GC_THREADS`
// For `NIX_USE_BOEHMGC`
#include "nix/expr/config.hh"
#if NIX_USE_BOEHMGC
# define GC_INCLUDE_NEW
# define GC_THREADS 1
# include <gc/gc.h>
# include <gc/gc_cpp.h>
@@ -6,9 +6,6 @@
#include "nix/expr/eval-error.hh"
#include "nix/expr/eval-settings.hh"
// For `NIX_USE_BOEHMGC`, and if that's set, `GC_THREADS`
#include "nix/expr/config.hh"
namespace nix {
/**
@@ -327,6 +327,21 @@ struct EvalSettings : Config
This option can be enabled by setting `NIX_ABORT_ON_WARN=1` in the environment.
)"};
Setting<bool> warnShortPathLiterals{
this,
false,
"warn-short-path-literals",
R"(
If set to true, the Nix evaluator will warn when encountering relative path literals
that don't start with `./` or `../`.
For example, with this setting enabled, `foo/bar` would emit a warning
suggesting to use `./foo/bar` instead.
This is useful for improving code readability and making path literals
more explicit.
)"};
};
/**
+2 -2
View File
@@ -1,13 +1,13 @@
# Public headers directory
include_dirs = [include_directories('../..')]
include_dirs = [ include_directories('../..') ]
config_pub_h = configure_file(
configuration : configdata_pub,
output : 'config.hh',
)
headers = [config_pub_h] + files(
headers = [ config_pub_h ] + files(
'attr-path.hh',
'attr-set.hh',
'eval-cache.hh',
+2 -27
View File
@@ -7,12 +7,7 @@
#include "nix/util/error.hh"
#include <boost/version.hpp>
#define USE_FLAT_SYMBOL_SET (BOOST_VERSION >= 108100)
#if USE_FLAT_SYMBOL_SET
# include <boost/unordered/unordered_flat_set.hpp>
#else
# include <boost/unordered/unordered_set.hpp>
#endif
#include <boost/unordered/unordered_flat_set.hpp>
namespace nix {
@@ -234,13 +229,7 @@ private:
* Transparent lookup of string view for a pointer to a ChunkedVector entry -> return offset into the store.
* ChunkedVector references are never invalidated.
*/
#if USE_FLAT_SYMBOL_SET
boost::unordered_flat_set<SymbolStr, SymbolStr::Hash, SymbolStr::Equal> symbols{SymbolStr::chunkSize};
#else
using SymbolValueAlloc = std::pmr::polymorphic_allocator<SymbolStr>;
boost::unordered_set<SymbolStr, SymbolStr::Hash, SymbolStr::Equal, SymbolValueAlloc> symbols{
SymbolStr::chunkSize, {&buffer}};
#endif
public:
@@ -252,19 +241,7 @@ public:
// Most symbols are looked up more than once, so we trade off insertion performance
// for lookup performance.
// FIXME: make this thread-safe.
return [&]<typename T>(T && key) -> Symbol {
if constexpr (requires { symbols.insert<T>(key); }) {
auto [it, _] = symbols.insert<T>(key);
return Symbol(*it);
} else {
auto it = symbols.find<T>(key);
if (it != symbols.end())
return Symbol(*it);
it = symbols.emplace(key).first;
return Symbol(*it);
}
}(SymbolStr::Key{store, s, stringAlloc});
return Symbol(*symbols.insert(SymbolStr::Key{store, s, stringAlloc}).first);
}
std::vector<SymbolStr> resolve(const std::vector<Symbol> & symbols) const
@@ -309,5 +286,3 @@ struct std::hash<nix::Symbol>
return std::hash<decltype(s.id)>{}(s.id);
}
};
#undef USE_FLAT_SYMBOL_SET
+10 -17
View File
@@ -1,4 +1,6 @@
project('nix-expr', 'cpp',
project(
'nix-expr',
'cpp',
version : files('.version'),
default_options : [
'cpp_std=c++2a',
@@ -17,8 +19,7 @@ subdir('nix-meson-build-support/deps-lists')
configdata_pub = configuration_data()
configdata_priv = configuration_data()
deps_private_maybe_subproject = [
]
deps_private_maybe_subproject = []
deps_public_maybe_subproject = [
dependency('nix-util'),
dependency('nix-store'),
@@ -39,8 +40,8 @@ endforeach
boost = dependency(
'boost',
modules : ['container', 'context'],
include_type: 'system',
modules : [ 'container', 'context' ],
include_type : 'system',
)
# boost is a public dependency, but not a pkg-config dependency unfortunately, so we
# put in `deps_other`.
@@ -60,8 +61,6 @@ if bdw_gc.found()
define_value = cxx.has_function(funcspec).to_int()
configdata_priv.set(define_name, define_value)
endforeach
# Affects ABI, because it changes what bdw_gc itself does!
configdata_pub.set('GC_THREADS', 1)
endif
# Used in public header. Affects ABI!
configdata_pub.set('NIX_USE_BOEHMGC', bdw_gc.found().to_int())
@@ -70,14 +69,8 @@ toml11 = dependency(
'toml11',
version : '>=3.7.0',
method : 'cmake',
include_type: 'system',
include_type : 'system',
)
configdata_priv.set(
'HAVE_TOML11_4',
toml11.version().version_compare('>= 4.0.0').to_int(),
)
deps_other += toml11
config_priv_h = configure_file(
@@ -119,8 +112,7 @@ lexer_tab = custom_target(
command : [
'flex',
'-Cf', # Use full scanner tables
'--outfile',
'@OUTPUT0@',
'--outfile', '@OUTPUT0@',
'--header-file=' + '@OUTPUT1@',
'@INPUT0@',
],
@@ -181,9 +173,10 @@ this_library = library(
generated_headers,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
link_args: linker_export_flags,
link_args : linker_export_flags,
prelink : true, # For C++ static initializers
install : true,
cpp_pch : do_pch ? [ 'pch/precompiled-headers.hh' ] : [],
)
install_headers(headers, subdir : 'nix/expr', preserve_path : true)
+3 -1
View File
@@ -1,3 +1,5 @@
option('gc', type : 'feature',
option(
'gc',
type : 'feature',
description : 'enable garbage collection in the Nix expression evaluator (requires Boehm GC)',
)
+2 -4
View File
@@ -70,15 +70,13 @@ mkMesonLibrary (finalAttrs: {
nix-util
nix-store
nix-fetchers
]
++ finalAttrs.passthru.externalPropagatedBuildInputs;
] ++ finalAttrs.passthru.externalPropagatedBuildInputs;
# Hack for sake of the dev shell
passthru.externalPropagatedBuildInputs = [
boost
nlohmann_json
]
++ lib.optional enableGC boehmgc;
] ++ lib.optional enableGC boehmgc;
mesonFlags = [
(lib.mesonEnable "gc" enableGC)
+9
View File
@@ -365,6 +365,15 @@ string_parts_interpolated
path_start
: PATH {
std::string_view literal({$1.p, $1.l});
/* check for short path literals */
if (state->settings.warnShortPathLiterals && literal.front() != '/' && literal.front() != '.') {
logWarning({
.msg = HintFmt("relative path literal '%s' should be prefixed with '.' for clarity: './%s'. (" ANSI_BOLD "warn-short-path-literals" ANSI_NORMAL " = true)", literal, literal),
.pos = state->positions[CUR_POS]
});
}
Path path(absPath(literal, state->basePath.path.abs()));
/* add back in the trailing '/' to the first segment */
if (literal.size() > 1 && literal.back() == '/')
+1
View File
@@ -0,0 +1 @@
#include "nix/expr/eval.hh"
+29 -29
View File
@@ -1363,7 +1363,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
/* Check whether attributes should be passed as a JSON file. */
using nlohmann::json;
std::optional<json> jsonObject;
std::optional<StructuredAttrs> jsonObject;
auto pos = v.determinePos(noPos);
auto attr = attrs->find(state.sStructuredAttrs);
if (attr != attrs->end()
@@ -1372,7 +1372,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
pos,
"while evaluating the `__structuredAttrs` "
"attribute passed to builtins.derivationStrict"))
jsonObject = json::object();
jsonObject = StructuredAttrs{.structuredAttrs = json::object()};
/* Check whether null attributes should be ignored. */
bool ignoreNulls = false;
@@ -1484,7 +1484,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
if (i->name == state.sStructuredAttrs)
continue;
jsonObject->emplace(key, printValueAsJSON(state, true, *i->value, pos, context));
jsonObject->structuredAttrs.emplace(key, printValueAsJSON(state, true, *i->value, pos, context));
if (i->name == state.sBuilder)
drv.builder = state.forceString(*i->value, context, pos, context_below);
@@ -1532,23 +1532,26 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
} else {
auto s = state.coerceToString(pos, *i->value, context, context_below, true).toOwned();
drv.env.emplace(key, s);
if (i->name == state.sBuilder)
drv.builder = std::move(s);
else if (i->name == state.sSystem)
drv.platform = std::move(s);
else if (i->name == state.sOutputHash)
outputHash = std::move(s);
else if (i->name == state.sOutputHashAlgo)
outputHashAlgo = parseHashAlgoOpt(s);
else if (i->name == state.sOutputHashMode)
handleHashMode(s);
else if (i->name == state.sOutputs)
handleOutputs(tokenizeString<Strings>(s));
else if (i->name == state.sJson)
if (i->name == state.sJson) {
warn(
"In derivation '%s': setting structured attributes via '__json' is deprecated, and may be disallowed in future versions of Nix. Set '__structuredAttrs = true' instead.",
drvName);
drv.structuredAttrs = StructuredAttrs::parse(s);
} else {
drv.env.emplace(key, s);
if (i->name == state.sBuilder)
drv.builder = std::move(s);
else if (i->name == state.sSystem)
drv.platform = std::move(s);
else if (i->name == state.sOutputHash)
outputHash = std::move(s);
else if (i->name == state.sOutputHashAlgo)
outputHashAlgo = parseHashAlgoOpt(s);
else if (i->name == state.sOutputHashMode)
handleHashMode(s);
else if (i->name == state.sOutputs)
handleOutputs(tokenizeString<Strings>(s));
}
}
}
@@ -1560,8 +1563,10 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
}
if (jsonObject) {
drv.env.emplace("__json", jsonObject->dump());
jsonObject.reset();
/* The only other way `drv.structuredAttrs` can be set is when
`jsonObject` is not set. */
assert(!drv.structuredAttrs);
drv.structuredAttrs = std::move(*jsonObject);
}
/* Everything in the context of the strings in the derivation
@@ -1933,13 +1938,8 @@ static void prim_dirOf(EvalState & state, const PosIdx pos, Value ** args, Value
NixStringContext context;
auto path = state.coerceToString(
pos, *args[0], context, "while evaluating the first argument passed to 'builtins.dirOf'", false, false);
auto pos = path->rfind('/');
if (pos == path->npos)
v.mkStringMove(".", context);
else if (pos == 0)
v.mkStringMove("/", context);
else
v.mkString(path->substr(0, pos), context);
auto dir = dirOf(*path);
v.mkString(dir, context);
}
}
@@ -4268,9 +4268,9 @@ static RegisterPrimOp primop_lessThan({
.name = "__lessThan",
.args = {"e1", "e2"},
.doc = R"(
Return `true` if the number *e1* is less than the number *e2*, and
`false` otherwise. Evaluation aborts if either *e1* or *e2* does not
evaluate to a number.
Return `true` if the value *e1* is less than the value *e2*, and `false` otherwise.
Evaluation aborts if either *e1* or *e2* does not evaluate to a number, string or path.
Furthermore, it aborts if *e2* does not match *e1*'s type according to the aforementioned classification of number, string or path.
)",
.fun = prim_lessThan,
});
+22 -97
View File
@@ -1,140 +1,73 @@
#include "nix/expr/primops.hh"
#include "nix/expr/eval-inline.hh"
#include "expr-config-private.hh"
#include <sstream>
#include <toml.hpp>
namespace nix {
#if HAVE_TOML11_4
/**
* This is what toml11 < 4.0 did when choosing the subsecond precision.
* TOML 1.0.0 spec doesn't define how sub-millisecond ranges should be handled and calls it
* implementation defined behavior. For a lack of a better choice we stick with what older versions
* of toml11 did [1].
*
* [1]: https://github.com/ToruNiina/toml11/blob/dcfe39a783a94e8d52c885e5883a6fbb21529019/toml/datetime.hpp#L282
*/
static size_t normalizeSubsecondPrecision(toml::local_time lt)
{
auto millis = lt.millisecond;
auto micros = lt.microsecond;
auto nanos = lt.nanosecond;
if (millis != 0 || micros != 0 || nanos != 0) {
if (micros != 0 || nanos != 0) {
if (nanos != 0)
return 9;
return 6;
}
return 3;
}
return 0;
}
/**
* Normalize date/time formats to serialize to the same strings as versions prior to toml11 4.0.
*
* Several things to consider:
*
* 1. Sub-millisecond range is represented the same way as in toml11 versions prior to 4.0. Precisioun is rounded
* towards the next multiple of 3 or capped at 9 digits.
* 2. Seconds must be specified. This may become optional in (yet unreleased) TOML 1.1.0, but 1.0.0 defined local time
* in terms of RFC3339 [1].
* 3. date-time separator (`t`, `T` or space ` `) is canonicalized to an upper T. This is compliant with RFC3339
* [1] 5.6:
* > Applications that generate this format SHOULD use upper case letters.
*
* [1]: https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
*/
static void normalizeDatetimeFormat(toml::value & t)
{
if (t.is_local_datetime()) {
auto & ldt = t.as_local_datetime();
t.as_local_datetime_fmt() = {
.delimiter = toml::datetime_delimiter_kind::upper_T,
// https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
.has_seconds = true, // Mandated by TOML 1.0.0
.subsecond_precision = normalizeSubsecondPrecision(ldt.time),
};
return;
}
if (t.is_offset_datetime()) {
auto & odt = t.as_offset_datetime();
t.as_offset_datetime_fmt() = {
.delimiter = toml::datetime_delimiter_kind::upper_T,
// https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
.has_seconds = true, // Mandated by TOML 1.0.0
.subsecond_precision = normalizeSubsecondPrecision(odt.time),
};
return;
}
if (t.is_local_time()) {
auto & lt = t.as_local_time();
t.as_local_time_fmt() = {
.has_seconds = true, // Mandated by TOML 1.0.0
.subsecond_precision = normalizeSubsecondPrecision(lt),
};
return;
}
}
#endif
static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Value & val)
{
auto toml = state.forceStringNoCtx(*args[0], pos, "while evaluating the argument passed to builtins.fromTOML");
std::istringstream tomlStream(std::string{toml});
auto visit = [&](auto & self, Value & v, toml::value t) -> void {
std::function<void(Value &, toml::value)> visit;
visit = [&](Value & v, toml::value t) {
switch (t.type()) {
case toml::value_t::table: {
auto table = toml::get<toml::table>(t);
auto attrs = state.buildBindings(table.size());
size_t size = 0;
for (auto & i : table) {
(void) i;
size++;
}
auto attrs = state.buildBindings(size);
for (auto & elem : table) {
forceNoNullByte(elem.first);
self(self, attrs.alloc(elem.first), elem.second);
visit(attrs.alloc(elem.first), elem.second);
}
v.mkAttrs(attrs);
} break;
;
case toml::value_t::array: {
auto array = toml::get<std::vector<toml::value>>(t);
auto list = state.buildList(array.size());
for (const auto & [n, v] : enumerate(list))
self(self, *(v = state.allocValue()), array[n]);
visit(*(v = state.allocValue()), array[n]);
v.mkList(list);
} break;
;
case toml::value_t::boolean:
v.mkBool(toml::get<bool>(t));
break;
;
case toml::value_t::integer:
v.mkInt(toml::get<int64_t>(t));
break;
;
case toml::value_t::floating:
v.mkFloat(toml::get<NixFloat>(t));
break;
;
case toml::value_t::string: {
auto s = toml::get<std::string_view>(t);
forceNoNullByte(s);
v.mkString(s);
} break;
;
case toml::value_t::local_datetime:
case toml::value_t::offset_datetime:
case toml::value_t::local_date:
case toml::value_t::local_time: {
if (experimentalFeatureSettings.isEnabled(Xp::ParseTomlTimestamps)) {
#if HAVE_TOML11_4
normalizeDatetimeFormat(t);
#endif
auto attrs = state.buildBindings(2);
attrs.alloc("_type").mkString("timestamp");
std::ostringstream s;
@@ -147,24 +80,16 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Va
throw std::runtime_error("Dates and times are not supported");
}
} break;
;
case toml::value_t::empty:
v.mkNull();
break;
;
}
};
try {
visit(
visit,
val,
toml::parse(
tomlStream,
"fromTOML" /* the "filename" */
#if HAVE_TOML11_4
,
toml::spec::v(1, 0, 0) // Be explicit that we are parsing TOML 1.0.0 without extensions
#endif
));
visit(val, toml::parse(tomlStream, "fromTOML" /* the "filename" */));
} catch (std::exception & e) { // TODO: toml::syntax_error
state.error<EvalError>("while parsing TOML: %s", e.what()).atPos(pos).debugThrow();
}
+5 -3
View File
@@ -1,4 +1,6 @@
project('nix-fetchers-c', 'cpp',
project(
'nix-fetchers-c',
'cpp',
version : files('.version'),
default_options : [
'cpp_std=c++2a',
@@ -35,7 +37,7 @@ sources = files(
'nix_api_fetchers.cc',
)
include_dirs = [include_directories('.')]
include_dirs = [ include_directories('.') ]
headers = files(
'nix_api_fetchers.h',
@@ -53,7 +55,7 @@ this_library = library(
sources,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
link_args: linker_export_flags,
link_args : linker_export_flags,
prelink : true, # For C++ static initializers
install : true,
)
+59 -1
View File
@@ -3,20 +3,28 @@
#include <gmock/gmock.h>
#include <git2/global.h>
#include <git2/repository.h>
#include <git2/signature.h>
#include <git2/types.h>
#include <git2/object.h>
#include <git2/tag.h>
#include <gtest/gtest.h>
#include "nix/util/fs-sink.hh"
#include "nix/util/serialise.hh"
#include "nix/fetchers/git-lfs-fetch.hh"
#include <git2/blob.h>
#include <git2/tree.h>
namespace nix {
class GitUtilsTest : public ::testing::Test
{
// We use a single repository for all tests.
std::filesystem::path tmpDir;
std::unique_ptr<AutoDelete> delTmpDir;
protected:
std::filesystem::path tmpDir;
public:
void SetUp() override
{
@@ -115,4 +123,54 @@ TEST_F(GitUtilsTest, sink_hardlink)
}
};
TEST_F(GitUtilsTest, peel_reference)
{
// Create a commit in the repo
git_repository * rawRepo = nullptr;
ASSERT_EQ(git_repository_open(&rawRepo, tmpDir.string().c_str()), 0);
// Create a blob
git_oid blob_oid;
const char * blob_content = "hello world";
ASSERT_EQ(git_blob_create_from_buffer(&blob_oid, rawRepo, blob_content, strlen(blob_content)), 0);
// Create a tree with that blob
git_treebuilder * builder = nullptr;
ASSERT_EQ(git_treebuilder_new(&builder, rawRepo, nullptr), 0);
ASSERT_EQ(git_treebuilder_insert(nullptr, builder, "file.txt", &blob_oid, GIT_FILEMODE_BLOB), 0);
git_oid tree_oid;
ASSERT_EQ(git_treebuilder_write(&tree_oid, builder), 0);
git_treebuilder_free(builder);
git_tree * tree = nullptr;
ASSERT_EQ(git_tree_lookup(&tree, rawRepo, &tree_oid), 0);
// Create a commit
git_signature * sig = nullptr;
ASSERT_EQ(git_signature_now(&sig, "nix", "nix@example.com"), 0);
git_oid commit_oid;
ASSERT_EQ(git_commit_create_v(&commit_oid, rawRepo, "HEAD", sig, sig, nullptr, "initial commit", tree, 0), 0);
// Lookup our commit
git_object * commit_object = nullptr;
ASSERT_EQ(git_object_lookup(&commit_object, rawRepo, &commit_oid, GIT_OBJECT_COMMIT), 0);
// Create annotated tag
git_oid tag_oid;
ASSERT_EQ(git_tag_create(&tag_oid, rawRepo, "v1", commit_object, sig, "annotated tag", 0), 0);
auto repo = openRepo();
// Use resolveRef to get peeled object
auto resolved = repo->resolveRef("refs/tags/v1");
// Now assert that we have unpeeled it!
ASSERT_STREQ(resolved.gitRev().c_str(), git_oid_tostr_s(&commit_oid));
git_signature_free(sig);
git_repository_free(rawRepo);
}
} // namespace nix
+7 -6
View File
@@ -1,4 +1,6 @@
project('nix-fetchers-tests', 'cpp',
project(
'nix-fetchers-tests',
'cpp',
version : files('.version'),
default_options : [
'cpp_std=c++2a',
@@ -19,8 +21,7 @@ deps_private_maybe_subproject = [
dependency('nix-fetchers'),
dependency('nix-fetchers-c'),
]
deps_public_maybe_subproject = [
]
deps_public_maybe_subproject = []
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/export-all-symbols')
@@ -44,7 +45,7 @@ sources = files(
'public-key.cc',
)
include_dirs = [include_directories('.')]
include_dirs = [ include_directories('.') ]
this_exe = executable(
@@ -53,7 +54,7 @@ this_exe = executable(
dependencies : deps_private_subproject + deps_private + deps_other,
include_directories : include_dirs,
# TODO: -lrapidcheck, see ../libutil-support/build.meson
link_args: linker_export_flags + ['-lrapidcheck'],
link_args : linker_export_flags + [ '-lrapidcheck' ],
# get main from gtest
install : true,
)
@@ -62,7 +63,7 @@ test(
meson.project_name(),
this_exe,
env : {
'_NIX_TEST_UNIT_DATA': meson.current_source_dir() / 'data',
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
},
protocol : 'gtest',
)
+5 -2
View File
@@ -44,16 +44,19 @@ static void downloadToSink(
static std::string getLfsApiToken(const ParsedURL & url)
{
assert(url.authority.has_value());
// FIXME: Not entirely correct.
auto [status, output] = runProgram(
RunOptions{
.program = "ssh",
.args = {*url.authority, "git-lfs-authenticate", url.path, "download"},
.args = {url.authority->to_string(), "git-lfs-authenticate", url.path, "download"},
});
if (output.empty())
throw Error(
"git-lfs-authenticate: no output (cmd: ssh %s git-lfs-authenticate %s download)",
url.authority.value_or(""),
url.authority.value_or(ParsedURL::Authority{}).to_string(),
url.path);
auto queryResp = nlohmann::json::parse(output);
+15 -3
View File
@@ -2,6 +2,7 @@
#include "nix/fetchers/git-lfs-fetch.hh"
#include "nix/fetchers/cache.hh"
#include "nix/fetchers/fetch-settings.hh"
#include "nix/util/base-n.hh"
#include "nix/util/finally.hh"
#include "nix/util/processes.hh"
#include "nix/util/signals.hh"
@@ -360,7 +361,13 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
Hash resolveRef(std::string ref) override
{
Object object;
if (git_revparse_single(Setter(object), *this, ref.c_str()))
// Using the rev-parse notation which libgit2 supports, make sure we peel
// the ref ultimately down to the underlying commit.
// This is to handle the case where it may be an annotated tag which itself has
// an object_id.
std::string peeledRef = ref + "^{commit}";
if (git_revparse_single(Setter(object), *this, peeledRef.c_str()))
throw Error("resolving Git reference '%s': %s", ref, git_error_last()->message);
auto oid = git_object_id(object.get());
return toHash(*oid);
@@ -540,7 +547,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
append(gitArgs, {"--depth", "1"});
append(gitArgs, {std::string("--"), url, refspec});
runProgram(
auto [status, output] = runProgram(
RunOptions{
.program = "git",
.lookupPath = true,
@@ -548,7 +555,12 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
// we're using --quiet for now. Should process its stderr.
.args = gitArgs,
.input = {},
.mergeStderrToStdout = true,
.isInteractive = true});
if (status > 0) {
throw Error("Failed to fetch git repository %s : %s", url, output);
}
}
void verifyCommit(const Hash & rev, const std::vector<fetchers::PublicKey> & publicKeys) override
@@ -597,7 +609,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
// Calculate sha256 fingerprint from public key and escape the regex symbol '+' to match the key literally
std::string keyDecoded;
try {
keyDecoded = base64Decode(k.key);
keyDecoded = base64::decode(k.key);
} catch (Error & e) {
e.addTrace({}, "while decoding public key '%s' used for git signature", k.key);
}
+11 -2
View File
@@ -463,7 +463,10 @@ struct GitInputScheme : InputScheme
// repo, treat as a remote URI to force a clone.
static bool forceHttp = getEnv("_NIX_FORCE_HTTP") == "1"; // for testing
auto url = parseURL(getStrAttr(input.attrs, "url"));
bool isBareRepository = url.scheme == "file" && !pathExists(url.path + "/.git");
// Why are we checking for bare repository?
// well if it's a bare repository we want to force a git fetch rather than copying the folder
bool isBareRepository = url.scheme == "file" && pathExists(url.path) && !pathExists(url.path + "/.git");
//
// FIXME: here we turn a possibly relative path into an absolute path.
// This allows relative git flake inputs to be resolved against the
@@ -481,6 +484,12 @@ struct GitInputScheme : InputScheme
"See https://github.com/NixOS/nix/issues/12281 for details.",
url);
}
// If we don't check here for the path existence, then we can give libgit2 any directory
// and it will initialize them as git directories.
if (!pathExists(url.path)) {
throw Error("The path '%s' does not exist.", url.path);
}
repoInfo.location = std::filesystem::absolute(url.path);
} else {
if (url.scheme == "file")
@@ -621,7 +630,7 @@ struct GitInputScheme : InputScheme
auto localRefFile = ref.compare(0, 5, "refs/") == 0 ? cacheDir / ref : cacheDir / "refs/heads" / ref;
bool doFetch;
bool doFetch = false;
time_t now = time(0);
/* If a rev was specified, we need to fetch if it's not in the
@@ -1,4 +1,4 @@
#include "fetchers.hh"
#include "nix/fetchers/fetchers.hh"
namespace nix::fetchers {
@@ -1,4 +1,4 @@
include_dirs = [include_directories('../..')]
include_dirs = [ include_directories('../..') ]
headers = files(
'attrs.hh',
+6 -5
View File
@@ -1,4 +1,6 @@
project('nix-fetchers', 'cpp',
project(
'nix-fetchers',
'cpp',
version : files('.version'),
default_options : [
'cpp_std=c++2a',
@@ -16,8 +18,7 @@ subdir('nix-meson-build-support/deps-lists')
configuration_data()
deps_private_maybe_subproject = [
]
deps_private_maybe_subproject = []
deps_public_maybe_subproject = [
dependency('nix-util'),
dependency('nix-store'),
@@ -27,7 +28,7 @@ subdir('nix-meson-build-support/subprojects')
nlohmann_json = dependency('nlohmann_json', version : '>= 3.9')
deps_public += nlohmann_json
libgit2 = dependency('libgit2')
libgit2 = dependency('libgit2', version : '>= 1.9')
deps_private += libgit2
subdir('nix-meson-build-support/common')
@@ -62,7 +63,7 @@ this_library = library(
sources,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
link_args: linker_export_flags,
link_args : linker_export_flags,
prelink : true, # For C++ static initializers
install : true,
)
Loaded 100 of 317 files, more files were not shown because too many files have changed in this diff. Show more