Compare commits

..
Author SHA1 Message Date
Eelco Dolstra 1de323e9df Merge branch 'master' (reformat) 2025-07-23 21:23:25 +02:00
Eelco Dolstra 4eb2df09d1 Merge branch 'master' (pre-reformat) 2025-07-23 21:23:16 +02:00
Sergei Zimmerman b8d223a210 Merge pull request #13108 from DeterminateSystems/check-format
Rip off the bandaid: Format the codebase with clang-format
2025-07-18 20:38:50 +03:00
Graham Christensen fb493ad7ca Update .git-blame-ignore-revs to ignore the mass reformatting 2025-07-18 12:47:27 -04:00
Graham Christensen e4f62e4608 Apply clang-format universally.
* It is tough to contribute to a project that doesn't use a formatter,
* It is extra hard to contribute to a project which has configured the formatter, but ignores it for some files
* Code formatting makes it harder to hide obscure / weird bugs by accident or on purpose,

Let's rip the bandaid off?

Note that PRs currently in flight should be able to be merged relatively easily by applying `clang-format` to their tip prior to merge.
2025-07-18 12:47:27 -04:00
Graham Christensen 41bf87ec70 Update clang-format with fixing namespace coments, and separate definition blocks 2025-07-18 12:46:51 -04:00
Graham Christensen e7af2e6566 Drop a ton of files that should just get formatted 2025-07-18 12:46:51 -04:00
Graham Christensen 6896761d79 Add sed 2025-07-18 12:46:51 -04:00
Graham Christensen ee9b57cbf5 format.sh: support looping until it is happy 2025-07-18 12:46:51 -04:00
Sergei Zimmerman 60c48b7058 Merge pull request #13501 from k1gen/fix-rsync-requirement
Require rsync in nix-manual meson.build
2025-07-18 00:43:27 +03:00
Oleksandr Knyshuk cfb8a31885 Require rsync in nix-manual meson.build
Closes: #13313
2025-07-17 22:59:37 +02:00
Jörg Thalheim 599c1d78ed Merge pull request #13496 from k1gen/fix-help-shell-command
Fix `nix help shell` by properly handling command aliases
2025-07-17 19:40:47 +02:00
Jörg Thalheim 4c95086879 Merge pull request #13493 from Mic92/clang-tidy-virtual-methods
Fix virtual method calls during construction
2025-07-17 17:41:18 +02:00
Oleksandr Knyshuk d678b071d6 Make nix help shell work by handling aliases properly
Previously, `nix help shell` failed with "Nix has no subcommand 'shell'" despite `nix shell --help`
working correctly. This happened because the `shell` command is actually an alias for `env shell`,
and the help system wasn't resolving aliases when looking up documentation.

This patch modifies the `showHelp` function to check for and resolve aliases before generating the
manpage name, ensuring that shorthand commands like `shell` get proper help documentation.

Closes: #13431
2025-07-17 17:26:56 +02:00
Jörg Thalheim 32a11a64c1 Merge pull request #13491 from Mic92/clang-tidy-windows-fixes
Fix Windows header inclusions for clang-tidy
2025-07-17 16:46:54 +02:00
Jörg Thalheim 36d451ddf4 Merge pull request #13476 from NixOS/improve-ignored-exceptions
Improve rendering of ignored exceptions
2025-07-17 15:57:59 +02:00
Jörg Thalheim a4d03ba527 Merge pull request #13370 from NixOS/unit-tests-home
Give unit tests access to a $HOME directory
2025-07-17 15:56:26 +02:00
Jörg Thalheim 44963da787 Fix virtual method calls during construction in LocalBinaryCacheStore
Move init() call from constructor to openStore() method to avoid calling
virtual methods during object construction. This prevents undefined
behavior when virtual methods are called before the object is fully
constructed.
2025-07-17 15:45:32 +02:00
Jörg Thalheim f12f96bcbb Fix virtual method calls during construction in S3BinaryCacheStoreImpl
Move init() call from constructor to openStore() method to avoid calling
virtual methods during object construction. This prevents undefined
behavior when virtual methods are called before the object is fully
constructed.
2025-07-17 15:45:32 +02:00
Jörg Thalheim 6681933643 Fix Windows header inclusions for clang-tidy
Move windows-error.hh includes inside _WIN32 guards to prevent
clang-tidy errors when analyzing these files on non-Windows platforms.
2025-07-17 15:40:46 +02:00
tomberek de1d91f711 Merge pull request #13486 from fzakaria/issue-12935
Add helpful messages when file:// used as tarball
2025-07-17 02:51:31 -04:00
tomberek 5ced59a21b Merge pull request #13449 from fzakaria/issue-10431
Add error message when git returns non-0 for fetch
2025-07-17 02:50:13 -04:00
Farid Zakaria 196c21c5a0 Add helpful messages when file:// used as tarball
When `file://` is used accidentally in a flake as the source it is
expected to be a tarball by default.

Add some friendlier error messages to either inform the user this is not
in fact a tarball or if it's a git directory, let them know they can use
`git+file`.

fixes #12935
2025-07-16 21:17:07 -07:00
Eelco Dolstra 17c94ca89e Merge pull request #13469 from drupol/push-nnwkpwsowkqm
docker: set default parameters values
2025-07-16 21:02:51 +02:00
Farid Zakaria a4f548fed1 Fix FetchGit test 2025-07-16 08:04:16 -07:00
Farid Zakaria fb6f494d35 merge stderr to stdout so we can emit it 2025-07-16 08:04:16 -07:00
Farid Zakaria 8e8416387c Add error message when git returns non-0 for fetch
Users have complained that fetchGit is flaky however the culprit is
likely that `git fetch` was unable itself to download the repository for
whatever reason (i.e. poor network etc..)

Nothing was checking the status of `git fetch` and the error message
that would eventually surface to the users were that the commit was not
found.

Add explicit error checking for status code from `git fetch` and return
a message earlier on to indicate that the failure was from that point.

fixes #10431
2025-07-16 08:04:16 -07:00
Eelco Dolstra 1272c4957f Merge pull request #13442 from NixOS/boost-minver
Require Boost 1.82.0 or higher
2025-07-16 14:54:14 +02:00
Sergei Zimmerman d7c340afbf Merge pull request #13473 from xokdvium/dogfood-ci 2025-07-15 22:35:43 +03:00
Eelco Dolstra 3dc1b99be5 AllowListSourceAccessor: Make thread-safe 2025-07-15 19:44:13 +02:00
Eelco Dolstra 7b2f24d688 Improve handleSQLiteBusy() message
Closes https://github.com/NixOS/nix/pull/10319.
2025-07-15 19:09:45 +02:00
Eelco Dolstra fde6068874 Improve rendering of ignored exceptions
Instead of

  error (ignored): error: SQLite database '...' is busy

we now get

  error (ignored): SQLite database '...' is busy
2025-07-15 19:09:40 +02:00
Eelco Dolstra 6abc29bba5 Move boost version check to libutil 2025-07-15 15:17:33 +02:00
Sergei Zimmerman 3b3c02160d ci: Dogfood nix from master for vm_tests and flake_regressions
This should provide more coverage for the build from master that
is being dogfooded.
2025-07-14 17:03:12 +03:00
Sergei Zimmerman 04f6974d2c ci: Dogfood Nix from master 2025-07-14 17:03:10 +03:00
Eelco Dolstra e8314e69ab Merge pull request #13460 from m4dc4p/m4dc4p/handle-expired-tokens
Better Handling for Expired Credentials
2025-07-14 11:51:30 +02:00
Eelco Dolstra 03bf582be3 Merge pull request #13468 from xokdvium/darwin-sandbox-sockets
libstore: fix Unix sockets in the build directory on sandboxed macOS
2025-07-14 11:13:36 +02:00
Pol Dellaiera 3e9a100bdf docker: set default parameters values 2025-07-13 22:49:12 +02:00
Emily 5cd94436f5 libstore: fix Unix sockets in the build directory on sandboxed macOS
We’re already allowing `/tmp` anyway, so this should be harmless,
and it fixes a regression in the default configuration caused by
moving the build directories out of `temp-dir`. (For instance, that
broke the Lix `guessOrInventPath.sockets` test.)

Note that removing `/tmp` breaks quite a few builds, so although it may
be a good idea in general it would require work on the Nixpkgs side.

Fixes: 749afbbe99fd7b45f828b72628252feba9241362
Change-Id: I6a6a69645f429bc50d4cb24283feda3d3091f534

(This is a cherry-pick of commit d1db3e5fa3faa43b3d2f2e2e843e9cfc1e6e1b71)

Lix patch: https://gerrit.lix.systems/c/lix/+/3500
2025-07-13 19:19:38 +03:00
m4dc4pandEelco Dolstra f786c0b8d1 Update src/libstore/s3-binary-cache-store.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-07-12 08:06:09 -07:00
m4dc4pandEelco Dolstra 22d6969d66 Update src/libstore/s3-binary-cache-store.cc
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-07-12 08:05:52 -07:00
Eelco Dolstra b124512388 Merge pull request #13447 from de11n/fix-gc-docs
Fix documentation for GC w.r.t. symlinks
2025-07-12 09:47:39 +02:00
Justin Bailey a48632f2e0 Better Handling for Expired Credentials
When AWS credentials expired, in some scenarios they led to the
nix process aborting with an error similar to ' Unable to parse ExceptionName: ExpiredToken'.

This change updates the S3 handling code such that those errors are treated like 403s or 404s.

Closes #13459
2025-07-11 17:30:47 -07:00
Sergei Zimmerman ab3cd76e73 Merge pull request #13455 from DeterminateSystems/gustavderdrache/fix-sandbox-ifdef
Address ifdef problem with macOS/BSD sandboxing
2025-07-12 02:09:14 +03:00
gustavderdrache 8e5814d972 CI: Roll nix version to 2.29.1
This works around the macOS issue that the prior commit addresses.
2025-07-11 18:46:01 -04:00
gustavderdrache e2ef2cfcbc Address ifdef problem with macOS/BSD sandboxing 2025-07-11 18:09:07 -04:00
John Ericson b196922bcc Merge pull request #13450 from xokdvium/formal-call-unreachable
libexpr: Fix invalid handling of errors for imported functions
2025-07-11 14:26:07 -04:00
Sergei Zimmerman 6e78cc90d3 libexpr: Fix invalid handling of errors for imported functions
c39cc00404 has added assertions for
all Value accesses and the following case has started failing with
an `unreachable`:

(/tmp/fun.nix):

```nix
{a}: a
```

```
$ nix eval --impure --expr 'import /tmp/fun.nix {a="a";b="b";}'
```

This would crash:

```
terminating due to unexpected unrecoverable internal error: Unexpected condition in getStorage at ../include/nix/expr/value.hh:844
```

This is not a regression, but rather surfaces an existing problem, which previously
was left undiagnosed. In the case of an import `fun` is the `import` primOp, so that read is invalid
and previously this resulted in an access into an inactive union member, which is UB.
The correct thing to use is `vCur`. Identical problem also affected the case of a missing argument.

Add previously failing test cases to the functional/lang test suite.

Fixes #13448.
2025-07-11 20:20:48 +03:00
Elliot Cameron a17f377f69 Fix documentation for GC w.r.t. symlinks 2025-07-10 16:19:43 -04:00
Eelco Dolstra a783468333 Merge pull request #13444 from roberth/fix-lessThan-doc
Fix `builtins.lessThan` doc
2025-07-10 20:32:47 +02:00
Eelco Dolstra e82ad5d9e2 Merge pull request #13441 from NixOS/fix-fetch-closure-warning
fetchClosure: Fix gcc warning
2025-07-10 20:32:10 +02:00
Robert Hensing fc03b89ff4 Fix lessThan doc 2025-07-10 17:50:43 +02:00
Eelco Dolstra ca9f2028b0 Simplify SymbolTable::create() 2025-07-10 12:28:34 +02:00
Eelco Dolstra 74a144ce98 Require Boost 1.81.0 or higher
Note: this version of Boost was released in December 2022.
2025-07-10 11:53:36 +02:00
Eelco Dolstra aa18dc54dc fetchClosure: Fix gcc warning
Fixes:

[261/394] Linking target src/libexpr/libnixexpr.so
In function ‘copy’,
    inlined from ‘__ct ’ at /nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/basic_string.h:688:23,
    inlined from ‘operator+’ at /nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/basic_string.h:3735:43,
    inlined from ‘operator()’ at ../src/libexpr/primops/fetchClosure.cc:127:58,
    inlined from ‘prim_fetchClosure’ at ../src/libexpr/primops/fetchClosure.cc:132:88:
/nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/char_traits.h:427:56: warning: ‘__builtin_memcpy’ writing 74 bytes into a region of size 16 overflows the destination [-Wstringop-overflow=]
  427 |         return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n));
      |                                                        ^
../src/libexpr/primops/fetchClosure.cc: In function ‘prim_fetchClosure’:
../src/libexpr/primops/fetchClosure.cc:132:88: note: at offset 16 into destination object ‘<anonymous>’ of size 32
  132 |             fromPath = state.coerceToStorePath(attr.pos, *attr.value, context, attrHint());
      |                                                                                        ^
2025-07-10 11:41:32 +02:00
tomberek 485cbfc7f1 Merge pull request #13437 from NixOS/respect-transitive-locks
lockFlake(): When updating a lock, respect the input's lock file
2025-07-10 00:28:55 -04:00
Sergei Zimmerman b19e9acc03 Merge pull request #13402 from DavHau/build-cores
build-cores: detect cores automatically if set to 0
2025-07-09 23:06:55 +03:00
Eelco Dolstra 95437b90fc lockFlake(): When updating a lock, respect the input's lock file 2025-07-09 19:19:15 +02:00
Eelco Dolstra 3cfecd9a80 Merge pull request #13435 from h0nIg/patch-4
docker: fix nixConf
2025-07-09 11:16:55 +02:00
h0nIg 9857c0bb52 docker: fix nixConf - fmt 2025-07-09 09:34:50 +02:00
h0nIg 8a1f471b66 docker: fix nixConf 2025-07-09 09:30:11 +02:00
John Ericson 7caac53ae4 Merge pull request #13433 from ConnorBaker/patch-1
globals.hh: fix broken link to nspawn example
2025-07-08 16:43:40 -04:00
Connor Baker a16491375a globals.hh: fix broken link to nspawn example
The substitution included the `.` at the end of the URL, breaking it.
2025-07-08 12:59:16 -07:00
Eelco Dolstra 98a883b51a Merge pull request #13427 from jsoo1/remove-p
installers, tests: remove --preserve=mode from cp invocations
2025-07-08 17:09:42 +02:00
Eelco Dolstra 6d0194b17f Merge pull request #13429 from NixOS/bump-2.31.0
Bump version to 2.31.0
2025-07-08 17:03:02 +02:00
Eelco Dolstra 06665e27f4 Update .mergify.yml 2025-07-08 16:18:53 +02:00
Eelco Dolstra 4fa99d743e release-process.md: Remove unnecessary step 2025-07-08 16:15:45 +02:00
Eelco Dolstra 723903da3c Bump version 2025-07-08 16:14:50 +02:00
John Soo 87299e466d installers, tests: remove --preserve=mode from cp invocations
-p preserves xattrs and acls which can be incompatible between
filesystems

Unfortunately keep -p on darwin because the bsd coreutils do not
support --preserve.

Fixes #13426
2025-07-07 15:34:18 -06:00
John Ericson cf0df2607d Merge pull request #13240 from obsidiansystems/dyn-drv-take-3
Get rid of `addWantedOutputs`
2025-07-07 11:39:45 -04:00
Eelco Dolstra ab5fc30d67 Merge pull request #13424 from NixOS/release-notes
Release notes for Nix 2.30.0
2025-07-07 17:22:57 +02:00
Eelco Dolstra b9b791dd51 Merge pull request #13425 from NixOS/tomberek.update_setuid_test
fix: make setuid tests use new build-dir location /nix/var/nix/builds
2025-07-07 16:50:57 +02:00
Robert Hensing 9f8df6878f doc: Add more links
Mostly in the 2.30 release notes
2025-07-07 16:47:02 +02:00
Eelco Dolstra 58e07c3291 Sort contributors 2025-07-07 16:17:06 +02:00
Thomas Bereknyei 9e7655f440 fix: make setuid tests use new build-dir location /nix/var/nix/builds 2025-07-07 10:13:40 -04:00
Eelco Dolstra 8c71de202f Add link 2025-07-07 15:50:51 +02:00
Eelco Dolstra a492493d97 Rearrange release notes 2025-07-07 15:10:25 +02:00
Eelco Dolstra 2c0343ec51 # Contributors -> ## Contributors 2025-07-07 15:00:49 +02:00
Eelco Dolstra f5312492c1 Add manual release notes 2025-07-07 14:53:11 +02:00
Eelco Dolstra 19c4e78d97 Typo 2025-07-07 14:19:57 +02:00
Eelco Dolstra 8a9e625ba5 release notes: 2.30.0 2025-07-07 14:09:18 +02:00
Eelco Dolstra 864ccb6e63 Merge pull request #13356 from SimSaladin/sim/fix-erase-output-paths
fix throwing output paths out of sandbox paths
2025-07-07 12:15:23 +02:00
Robert Hensing c512fd1a4b Merge pull request #13420 from NixOS/query-missing
Introduce MissingPaths struct for queryMissing() / printMissing()
2025-07-07 11:48:06 +02:00
Eelco Dolstra 7781560724 libstore-tests: Fix impurity trying to access the Nix daemon
This failed on macOS:

  nix-store-tests-run> C++ exception with description "../nix_api_store.cc:33: nix_err_code(ctx) != NIX_OK, message: error: getting status of '/nix/var/nix/daemon-socket/socket': Operation not permitted" thrown in the test body.
2025-07-07 11:42:20 +02:00
Eelco Dolstra 99c20e6c3e Merge pull request #13328 from h0nIg/patch-3
docker: add docu references & remove duplicate code
2025-07-07 11:31:59 +02:00
Eelco Dolstra f29acd5bbc Use writableTmpDirAsHomeHook 2025-07-07 11:12:55 +02:00
Eelco Dolstra 01388b3e78 Give unit tests access to a $HOME directory
Also, don't try to access cache.nixos.org in the libstore unit tests.
2025-07-07 11:02:51 +02:00
DavHau fb5e22e318 build-cores: detect cores automatically if set to 0
This changes makes nix detect a machines available cores automatically whenever build-cores is set to 0.

So far, nix simply passed NIX_BUILD_CORES=0 whenever build-cores is set to 0. (only when build-cores is unset it was detecting cores automatically)

The behavior of passing NIX_BUILD_CORES=0 leads to a performance penalty when sourcing nixpkgs' generic builder's `setup.sh`, as setup.sh has to execute `nproc`. This significantly slows down sourcing of setup.sh
2025-07-07 10:33:42 +07:00
Sergei Zimmerman 6bf997e0bd Merge pull request #13419 from NixOS/fix-indent 2025-07-04 20:27:32 +03:00
Eelco Dolstra f039f6886a nix-build: Drop unnecessary call to queryMissing()
This is already done by Worker::run().
2025-07-04 16:34:34 +02:00
Eelco Dolstra 5d308ccca5 printMissing(): Take a MissingPaths argument 2025-07-04 16:34:30 +02:00
Eelco Dolstra af05ce0f6d queryMissing(): Return a struct
...instead of having a bunch of pass-by-reference arguments.
2025-07-04 16:34:24 +02:00
Eelco Dolstra eb97d8c170 Fix indentation of "Unable to start any build" error message
The use of R"(...)" added a bunch of unnecessary whitespace, e.g.

  error:
                             Unable to start any build;
                             either increase '--max-jobs' or enable remote builds.

                             For more information run 'man nix.conf' and search for '/machines'.

Now we get

  error: Unable to start any build; either increase '--max-jobs' or enable remote builds.

         For more information run 'man nix.conf' and search for '/machines'.
2025-07-04 15:21:31 +02:00
Eelco Dolstra d4f67fd46d Merge pull request #13369 from NixOS/null-revision-not-locked
Git fetcher: Do not consider a null revision (i.e. workdir) to be locked
2025-07-03 19:56:21 +02:00
Robert Hensing c79b8018c0 Merge pull request #13407 from xokdvium/smaller-value
libexpr: Reduce the size of Value down to 16 bytes (on 64 bit systems)
2025-07-03 00:12:47 +02:00
Sergei Zimmerman 5a20a48f13 libexpr: Reduce the size of Value down to 16 bytes
This shaves off a very significand amount of memory used
for evaluation as well as reduces the GC-managed heap.

Previously the union discriminator (InternalType) was
stored as a separate field in the Value, which takes up
whole 8 bytes due to padding needed for member alignment.
This effectively wasted 7 whole bytes of memory. Instead
of doing that InternalType is instead packed into pointer
alignment niches. As it turns out, there's more than enough
unused bits there for the bit packing to be effective.

See the doxygen comment in the ValueStorage specialization
for more details.

This does not add any performance overhead, even though
we now consistently assert the InternalType in all getters.

This can also be made atomic with a double width compare-and-swap
instruction on x86_64 (CMPXCHG16B instruction) for parallel evaluation.
2025-07-02 22:27:37 +03:00
Sergei Zimmerman e73fcf7b53 libexpr: Use proxy ListView for all Value list accesses
This also makes it possible to make `payload` field private
in the `ValueStorage` class template.
2025-07-02 21:57:02 +03:00
Sergei Zimmerman c39cc00404 libexpr: Factor out Payload union to a default implementation of ValueStorage
This factors out most of the value representation into a mixin class.
`finishValue` is now gone for good and replaced with a simple template
function `setStorage` which derives the type information/disriminator from
the type of the argument. Likewise, reading of the value goes through function
template `getStorage`.

An empty type `Null` is introduced to make the bijection InternalType <-> C++ type
complete.
2025-07-02 21:51:15 +03:00
Sergei Zimmerman 810455f1b8 libexpr: Simplify Value::is* methods by introducing isa function template 2025-07-02 21:51:12 +03:00
Sergei Zimmerman 1a033ee4ee libexpr: Use single tSmallList Value discriminator for small lists 2025-07-02 21:51:09 +03:00
Sergei Zimmerman ea32580c9b libexpr: Format value.hh
The following commits will touch this file significantly, so
it's better to get the formatting out of the way first.
2025-07-02 21:51:07 +03:00
Robert Hensing 04a731b4b9 Merge pull request #13387 from NaN-git/opt-listToAttrs
libexpr: don't allocate additional set in `builtins.listToAttrs`
2025-07-02 11:55:39 +02:00
Robert Hensing 4bf23d242d Merge pull request #13416 from xokdvium/pragma-once-missing
libfetchers: Add missing include guard to git-lfs-fetch.hh
2025-07-02 11:52:03 +02:00
Sergei Zimmerman d16af1d099 libfetchers: Add missing include guard to git-lfs-fetch.hh
This is a publicly installed header without a header guard.
Doesn't seem right.
2025-07-02 00:57:35 +03:00
Eelco Dolstra 5879ab1577 Merge pull request #13400 from NixOS/fix-deep-overrides
Fix deep flake input overrides
2025-07-01 19:34:55 +02:00
Eelco Dolstra 4458a9061b Merge pull request #13406 from xokdvium/libflake-dead-code
libflake: Remove unused maybeParseFlakeRef and maybeParseFlakeRefWith…
2025-07-01 17:03:00 +02:00
Eelco Dolstra 9600eaef2e Merge pull request #13412 from xokdvium/read-directory-caching
libutil: Use caching `directory_entry` API in `PosixSourceAccessor::r…
2025-07-01 16:59:40 +02:00
Sergei Zimmerman 8708e9a526 libutil: Use caching directory_entry API in PosixSourceAccessor::readDirectory
Previous use of symlink_status() always translated into a stat call, leading
to huge performance penalties for by-name-overlay in nixpkgs. The comment
below references the possible caching, but that seemed to be erroneous, since
the correct way to make use of the caching API is by calling a bunch of `is_*`
functions [1]. For example, here's how libstdc++ does that [2], [3].

This translates to great nixpkgs eval performance improvements:

```
Benchmark 1: GC_INITIAL_HEAP_SIZE=4G result/bin/nix-instantiate ../nixpkgs -A hello --readonly-mode
  Time (mean ± σ):     186.7 ms ±   6.7 ms    [User: 121.3 ms, System: 64.9 ms]
  Range (min … max):   179.4 ms … 201.6 ms    16 runs

Benchmark 2: GC_INITIAL_HEAP_SIZE=4G nix-instantiate ../nixpkgs -A hello --readonly-mode
  Time (mean ± σ):     230.6 ms ±   5.0 ms    [User: 126.9 ms, System: 103.1 ms]
  Range (min … max):   225.1 ms … 241.4 ms    13 runs
```

[1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0317r1.html
[2]: https://github.com/gcc-mirror/gcc/blob/8ea555b7b4725dbc5d9286f729166cd54ce5b615/libstdc%2B%2B-v3/include/bits/fs_dir.h#L341-L348
[3]: https://github.com/gcc-mirror/gcc/blob/8ea555b7b4725dbc5d9286f729166cd54ce5b615/libstdc%2B%2B-v3/include/bits/fs_dir.h#L161-L163
2025-06-30 23:29:07 +03:00
Philipp Otterbein fa3d7e6f68 libexpr: don't allocate additional set in builtins.listToAttrs 2025-06-29 01:41:16 +02:00
Sergei Zimmerman 75412ebc30 libflake: Remove unused maybeParseFlakeRef and maybeParseFlakeRefWithFragment
These wrappers don't seem to be used anywhere in and out of tree.
Also the declaration in the header has an incorrect function name `maybeParseFlake`.

Closes #11948
2025-06-27 23:33:54 +03:00
h0nIg ba12adc0f9 format 2025-06-26 23:37:39 +02:00
h0nIg 8fbc27af46 enhancements 2025-06-26 23:33:27 +02:00
h0nIg 2481e9a203 Merge branch 'master' into patch-3 2025-06-26 23:31:24 +02:00
Sergei Zimmerman acfdacc971 Merge pull request #13401 from NixOS/split-flake-command
Move FlakeCommand into a header, allow separate registration of subcommands
2025-06-26 19:02:59 +03:00
Eelco Dolstra eaced1e0d2 Move FlakeCommand into a header, allow separate registration of subcommands
This allows us to start splitting up src/nix/flake.cc.
2025-06-26 17:14:32 +02:00
Eelco Dolstra b415faceca Don't allow flake inputs to have both a flakeref and a follows
Having both doesn't make sense so it's best to disallow it. If this
causes issues we could turn into a warning.
2025-06-26 12:56:54 +02:00
Eelco DolstraandSebastian Ullrich 637c4f3ad7 Add tests for deep overrides
Taken from https://github.com/NixOS/nix/pull/6621.

Co-authored-by: Sebastian Ullrich <sebasti@nullri.ch>
2025-06-26 12:56:49 +02:00
Eelco Dolstra 7c0ea143d8 Fix deep overrides
An override like

  inputs.foo.inputs.bar.inputs.nixpkgs.follows = "nixpkgs";

implicitly set `inputs.foo.inputs.bar` to `flake:bar`, which led to an
unexpected error like

  error: cannot find flake 'flake:bar' in the flake registries

We now no longer create a parent override (like for `foo.bar` in the
example above) if it doesn't set an explicit ref or follows
attribute. We only recursively apply its child overrides.

Fixes https://github.com/NixOS/nix/issues/8325, https://github.com/DeterminateSystems/nix-src/issues/95, https://github.com/NixOS/nix/issues/12083, https://github.com/NixOS/nix/issues/5790.
2025-06-26 12:56:39 +02:00
Robert Hensing df2d5f276a Merge pull request #13395 from xokdvium/libstore-tests-leaks
libstore-tests: Don't leak memory in tests
2025-06-25 22:41:00 +02:00
Sergei Zimmerman 55d12dfc5d libstore-tests: Don't leak memory in tests
We shouldn't leak memory in unit tests in order
to make enabling ASAN easier.
2025-06-25 00:07:58 +03:00
Eelco Dolstra 448cfb71ea Merge remote-tracking branch 'cve/fod-cves-master' 2025-06-24 16:02:58 +02:00
Robert Hensing f9afc1e68c Merge pull request #13393 from egorkonovalov/patch-1
Fix typo: remove an extra word
2025-06-23 16:07:14 +02:00
Egor Konovalov df21f24987 Fix link
Remove extra `realise`
2025-06-23 11:26:59 +02:00
Robert Hensing 6a74590063 Merge pull request #13388 from NaN-git/opt-string_view
libexpr: further removal of `std::string` copies
2025-06-23 09:05:28 +02:00
Eelco Dolstra 37685b1c9c Fix Darwin test failure in repl.sh
Fixes

  error:
         … while processing sandbox path '/private/tmp/nix-shell.0MDgyx/nix-test/ca/repl/store/nix/var/nix/builds/nix-build-simple.drv-65916-3910734210' (/private/tmp/nix-shell.0MDgyx/nix-test/ca/repl/store)

         error: 'nix' is too short to be a valid store path

which happened because we were now putting the build directory
underneath the store directory.
2025-06-22 16:49:44 +02:00
Eelco Dolstra 2e2fe4cb07 Cleanup 2025-06-22 16:49:44 +02:00
Eelco Dolstra 5acf50a327 Disallow the build directory having world-writable parents 2025-06-22 16:49:44 +02:00
eldritch horrors 88b7db1ba4 libstore: Don't default build-dir to temp-dir, store setting
If a build directory is accessible to other users it is possible to
smuggle data in and out of build directories. Usually this is only
a build purity problem, but in combination with other issues it can
be used to break out of a build sandbox. to prevent this we default
to using a subdirectory of nixStateDir (which is more restrictive).

(cherry picked from pennae Lix commit 55b416f6897fb0d8a9315a530a9b7f0914458ded)
(store setting done by roberth)
2025-06-22 16:49:44 +02:00
Eelco Dolstra 9af4c267c6 Chown structured attr files safely 2025-06-22 16:48:34 +02:00
Eelco Dolstra a4b5584fb1 Replace 'bool sync' with an enum for clarity
And drop writeFileAndSync().
2025-06-22 16:48:34 +02:00
Eelco Dolstra 479c356510 Drop guessOrInventPathFromFD()
No need to do hacky stuff like that when we already know the original path.
2025-06-22 16:48:34 +02:00
Eelco Dolstra e5dc3e84f0 Tweak comment 2025-06-22 16:48:34 +02:00
Raito Bezarius 4ea4813753 libstore: ensure that temporary directory is always 0o000 before deletion
In the case the deletion fails, we should ensure that the temporary
directory cannot be used for nefarious purposes.

Change-Id: I498a2dd0999a74195d13642f44a5de1e69d46120
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-22 16:48:33 +02:00
Raito Bezarius 5ec047f348 libutil: ensure that _deletePath does NOT use absolute paths with dirfds
When calling `_deletePath` with a parent file descriptor, `openat` is
made effective by using relative paths to the directory file descriptor.

To avoid the problem, the signature is changed to resist misuse with an
assert in the prologue of the function.

Change-Id: I6b3fc766bad2afe54dc27d47d1df3873e188de96
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-22 16:48:33 +02:00
Raito Bezarius 4e59d3fdb2 libstore: ensure that passAsFile is created in the original temp dir
This ensures that `passAsFile` data is created inside the expected
temporary build directory by `openat()` from the parent directory file
descriptor.

This avoids a TOCTOU which is part of the attack chain of CVE-????.

Change-Id: Ie5273446c4a19403088d0389ae8e3f473af8879a
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-22 16:48:33 +02:00
Raito Bezarius 034f59bbb9 libutil: writeFile variant for file descriptors
`writeFile` lose its `sync` boolean flag to make things simpler.

A new `writeFileAndSync` function is created and all call sites are
converted to it.

Change-Id: Ib871a5283a9c047db1e4fe48a241506e4aab9192
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-22 16:48:33 +02:00
Raito Bezarius 002d202653 libstore: chown to builder variant for file descriptors
We use it immediately for the build temporary directory.

Change-Id: I180193c63a2b98721f5fb8e542c4e39c099bb947
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-22 16:48:33 +02:00
Raito Bezarius 6a5b6ad3b7 libstore: open build directory as a dirfd as well
We now keep around a proper AutoCloseFD around the temporary directory
which we plan to use for openat operations and avoiding the build
directory being swapped out while we are doing something else.

Change-Id: I18d387b0f123ebf2d20c6405cd47ebadc5505f2a
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-22 16:48:33 +02:00
Raito Bezarius 7226a116a0 libutil: guess or invent a path from file descriptors
This is useful for certain error recovery paths (no pun intended) that
does not thread through the original path name.

Change-Id: I2d800740cb4f9912e64c923120d3f977c58ccb7e
Signed-off-by: Raito Bezarius <raito@lix.systems>
2025-06-22 16:48:33 +02:00
John EricsonandSergei Zimmerman a55806a0dd Get rid of addWantedOutputs
This is just a code cleanup; it should not be behavior change.

`addWantedOutputs` is removed by introducing `DerivationTrampolineGoal`.

`DerivationGoal` now only tracks a single output, and is back to
tracking a plain store path `drvPath`, not a deriving path one. Its
`addWantedOutputs` method is gone. These changes will allow subsequent
PRs to simplify it greatly.

Because the purpose of each goal is back to being immutable, we can also
once again make `Goal::buildResult` a public field, and get rid of the
`getBuildResult` method. This simplifies things also.

`DerivationTrampolineGoal` is, as the nane is supposed to indicate, a
cheap "trampoline" goal. It takes immutable sets of wanted outputs, and
just kicks of `DerivationGoal`s for them. Since now "actual work" is
done in these goals, it is not wasteful to have separate ones for
separate sets of outputs, even if those outputs (and the derivations
they are from) overlap.

This design is described in more detail in the doc comments on the goal
types, which I've now greatly expanded.

---

This separation of concerns will make it possible for future work on
issues like #11928, and to continue the path of having more goal types,
but each goal type does fewer things (issue #12628).

---

This commit in some sense reverts
f4f28cdd0e, but that one kept around
`addWantedOutputs`. I am quite sure it was having two layers of goals
with `addWantedOutputs` that caused the issues --- restarting logic like
`addWantedOutputs` has is very tempermental! In this version of the
change, we have *zero* layers of `addWantedOutputs` --- no goal type
needs it, or otherwise has a mutable objective --- and so I think this
change is safe.

Co-authored-by: Sergei Zimmerman <145775305+xokdvium@users.noreply.github.com>
2025-06-20 18:43:35 -04:00
John Ericson 47b7d910e9 Properly prefix libfetchers header
This is a minor oversight from 012453d1e6.
2025-06-20 18:42:23 -04:00
John Ericson b9b510d692 Merge pull request #13383 from xokdvium/meson-format-multiline
Restore multiline formatting of lists in meson files
2025-06-20 18:19:03 -04:00
Sergei Zimmerman a4dcce36c9 Merge pull request #13382 from synalice/patch-3
Update docs
2025-06-20 23:18:11 +03:00
Sergei Zimmerman 6ef683cb2a Restore multiline formatting of lists in meson files
Applies a workaround to enforce multiline formatting
of lists to reduce code churn introduced in 93a42a5971.
2025-06-20 23:12:36 +03:00
Nikita Krasnov 785f3867fd Update docs 2025-06-20 21:19:13 +03:00
Robert Hensing 42ea2724a8 Merge pull request #13353 from lucperkins/messages-present-tense
Rework future tense in user-facing messages
2025-06-20 10:43:03 +02:00
Luc Perkins 3132aba8e4 Fix broken test 2025-06-19 15:23:10 -07:00
Luc Perkins 9c120596ec Merge remote-tracking branch 'upstream/master' into messages-present-tense 2025-06-19 10:51:50 -07:00
Jörg Thalheim aa1629ca35 Merge pull request #13378 from NixOS/improve-rosetta-hint
Improve the Rosetta installation hint
2025-06-19 16:38:10 +02:00
Jörg Thalheim a8a5537109 Merge pull request #13376 from NixOS/use-magic-nix-cache
Revert "Drop magic-nix-cache"
2025-06-19 14:10:55 +02:00
Eelco Dolstra 20ba6be749 Improve the Rosetta installation hint
The Nix daemon detects supported system types at start time, so it
needs to be restarted to detect x86_64-darwin support.
2025-06-19 13:58:51 +02:00
John Ericson d254c840b5 Merge pull request #13349 from obsidiansystems/structured-attrs-json
Introduce top-level `structuredAttrs` field in JSON derivation format
2025-06-18 16:35:42 -04:00
Eelco Dolstra 9b57573bae Revert "Drop magic-nix-cache"
This reverts commit 9cc8be2674 since
magic-nix-cache works again (thanks @jchv).
2025-06-18 18:06:24 +02:00
Eelco Dolstra fc47178d0d Merge pull request #13372 from synalice/patch-1
Fix broken link
2025-06-18 17:54:04 +02:00
Eelco Dolstra 197484ce1f Merge pull request #13373 from synalice/patch-2
Fix typo
2025-06-18 17:52:38 +02:00
Luc Perkins d6710b4c04 Merge remote-tracking branch 'upstream/master' into messages-present-tense 2025-06-18 08:24:23 -07:00
Luc Perkins d2a25fbe51 Fix Nix formatting changes 2025-06-18 08:23:37 -07:00
Eelco Dolstra cd97c545d9 Merge pull request #13371 from vcunat/p/jq-1.8.0_master
tests: fixup with jq-1.8.0
2025-06-18 17:17:24 +02:00
Nikita Krasnov 86dda9884a Fix typo 2025-06-18 12:46:53 +03:00
Nikita Krasnov da76bc0cac Fix broken link 2025-06-18 12:40:07 +03:00
Vladimír Čunát 77f6b6532f tests: fixup with jq-1.8.0 2025-06-18 10:11:50 +02:00
Eelco Dolstra bb32fcdf30 Merge pull request #13368 from wolfgangwalther/race-state-creation
libstore: fix race condition when creating state directories
2025-06-17 11:33:29 +02:00
Eelco Dolstra 59c7dac867 Git fetcher: Do not consider a null revision (i.e. workdir) to be locked 2025-06-17 10:55:23 +02:00
Wolfgang Walther d64c922164 libstore: fix race condition when creating state directories
Running parallel nix in nix can lead to multiple instances trying to
create the state directories and failing on the `createSymlink` step,
because the link already exists.

`replaceSymlink` is already idempotent, so let's use that.

Resolves #2706
2025-06-17 08:45:29 +02:00
John Ericson cdb417854b Merge pull request #13366 from drupol/push-smvorxlvxusx
docker: add basics OpenContainers labels
2025-06-16 18:11:53 -04:00
tomberek af01fea5d6 Merge pull request #12623 from xokdvium/ord-safe-sort
Switch builtins.sort to a custom stable PeekSort
2025-06-16 05:11:23 -04:00
Pol Dellaiera bb44347fac docker: expose config.Cmd as parameter 2025-06-16 08:16:39 +02:00
Pol Dellaiera 18dc96269d docker: add basics OpenContainers labels 2025-06-16 08:16:39 +02:00
Philipp Otterbein c1aaa970c7 libexpr: further removal of std::string copies 2025-06-15 21:56:45 +02:00
Sergei Zimmerman f9170a84f6 tests/functional/lang: Add sort stability test for lists langer than 16 elements
libstdc++'s std::stable_sort and new builtins.sort implementation
special-case ranges with length less than or equal to 16 and delegate
to insertionsort.

Having a larger e2e test would allow catching sort stability issues
at functional level as well.
2025-06-15 16:52:08 +00:00
Sergei Zimmerman ddcfc81ff1 libexpr: Document requirements for comparator passed to builtins.sort 2025-06-15 16:52:05 +00:00
Sergei Zimmerman 351d898c43 libexpr: Switch builtins.sort primop to use peeksort
This prevents C++ level undefined behavior from affecting
the evaluator. Stdlib implementation details should not affect
eval, regardless of the build platform. Even erroneous usage
of `builtins.sort` should not make it possible to crash the
evaluator or produce results that depend on the host platform.
2025-06-15 16:52:03 +00:00
Sergei Zimmerman b2596a7615 libutil: Add custom PeekSort implementation
Unlike std::sort and std::stable_sort, this implementation
does not lead to out-of-bounds memory reads or other undefined
behavior when the predicate is not strict weak ordering.

This makes it possible to use this function in libexpr for
builtins.sort, where an incorrectly implemented comparator
in the user nix code currently can crash and burn the evaluator
by invoking C++ UB.
2025-06-15 16:52:01 +00:00
Jörg Thalheim e73fcd008a Merge pull request #13364 from drupol/push-qspryqrvmpkt
docker: remove last use of `pkgs.`
2025-06-15 06:09:52 +02:00
Pol Dellaiera e27a062783 docker: remove last use of pkgs.
Follow-up of https://github.com/NixOS/nix/pull/13354
2025-06-14 10:38:20 +02:00
Jörg Thalheim 46853c467d Merge pull request #13361 from jayeshv/patch-1
Fix a minor typo
2025-06-14 02:01:05 +02:00
jayeshv 699db04df3 Fix a minor typo 2025-06-13 12:28:27 +02:00
Jörg Thalheim a1fe09d2c8 Merge pull request #13350 from xokdvium/meson-format
flake: Add meson formatter
2025-06-13 11:37:32 +02:00
Jörg Thalheim 84743c84c0 Merge pull request #13351 from drupol/push-rrzylpqynznw
docker: make sure `nix config check` works
2025-06-13 07:41:27 +02:00
Jörg Thalheim 82397e220d Merge branch 'master' into push-rrzylpqynznw 2025-06-13 07:06:11 +02:00
Jörg Thalheim 918ac6b2fc Merge pull request #13354 from drupol/push-sopslzmmlqxu
docker: use `callPackage`, parametrise the image build
2025-06-13 07:03:23 +02:00
Jörg Thalheim d12e42489a Merge pull request #13360 from xokdvium/duplicate-list-elems-overload
libexpr: Remove non-const overload of `listElems`
2025-06-13 07:03:03 +02:00
John Ericson f8c1ac9515 Introduce top-level structuredAttrs field in JSON derivation format
Makes the behavoral change of #13263 without the underlying refactor.
Hopefully this clearly safe from a perf and GC perspective, and will
make it easier to benchmark #13263.
2025-06-12 20:45:47 -04:00
Sergei Zimmerman 7b46eb9958 libexpr: Remove non-const overload of listElems
This overload isn't actually necessary anywhere and
doesn't make much sense. The pointers to `Value`s are
themselves const, but the `Value`s are mutable.
A non-const member function implies that the object itself
can be modified but this doesn't make much sense considering
the return type: `Value * const * `, which is a pointer
to a constant array of pointers to mutable values.
2025-06-12 22:29:05 +00:00
Robert Hensing c437e27abc Merge pull request #13357 from xokdvium/more-getters
libexpr: Use more `Value` getters instead of reading `payload` directly (NFC)
2025-06-12 23:03:20 +02:00
Sergei Zimmerman bc6b52aff0 libexpr: Add and use pathAccessor getter 2025-06-12 20:01:38 +00:00
Sergei Zimmerman e4df189123 libexpr: Add and use pathStr getter 2025-06-12 19:57:46 +00:00
Sergei Zimmerman c041d71406 libexpr: Add and use app getter 2025-06-12 19:53:44 +00:00
Sergei Zimmerman f07a9f863e libexpr: Add and use primOpApp getter 2025-06-12 19:51:44 +00:00
Sergei Zimmerman 441fa86e82 libexpr: Add and use thunk getter 2025-06-12 19:48:42 +00:00
Sergei Zimmerman 6587e7bcff libexpr: Add and use lambda getter 2025-06-12 19:42:50 +00:00
Samuli Thomasson 0f6cb33763 fix throwing output paths out of sandbox paths
It seems obvious that erasing any output paths from pathsInChroot needs
to happen after getPathsInSandbox(), not before.

Signed-off-by: Samuli Thomasson <samuli.thomasson@pm.me>
2025-06-12 21:36:04 +02:00
Pol Dellaiera 6eb4ee6855 docker: replace git with gitMinimal 2025-06-12 19:50:52 +02:00
Pol Dellaiera 5862f38d00 docker: use callPackage, parametrise the image build 2025-06-12 19:50:31 +02:00
Pol Dellaiera 5abaf361a4 docker: reduce duplicates, use coreutils-full 2025-06-12 19:06:48 +02:00
Luc Perkins ab10fddc6e Rework future tense in user-facing messages 2025-06-12 09:07:36 -07:00
Pol Dellaiera 57c72dee9b docker: make sure nix config check works 2025-06-12 11:01:17 +02:00
Sergei Zimmerman 93a42a5971 flake: Add meson formatter
This adds a meson.format file that mostly mirrors the projects
meson style and a pre-commit hook to enforce this style.
Some low-diff files are formatted.
2025-06-11 22:08:03 +00:00
John Ericson 371fcf91c3 Merge pull request #13294 from obsidiansystems/fix-13293
Fix #13293
2025-06-11 16:55:43 -04:00
John Ericson 5267718472 Fix #13293
We move the `assertPathValidity` to where we know what the wanted
outputs are.
2025-06-11 16:15:26 -04:00
John Ericson f42eaf2c8e Create test for #13293
It currently fails, before the fix.
2025-06-11 16:11:38 -04:00
John Ericson d46ce52fac Merge pull request #13347 from lucperkins/fix-broken-conf-link
Fix broken link in configuration description
2025-06-11 15:58:03 -04:00
John Ericson 35cb138a93 Merge pull request #13348 from NixOS/repl-test-robust-2
Make the repl test more robust
2025-06-11 15:57:38 -04:00
Eelco Dolstra 9eb46e9cc0 Make the repl test more robust
Seen in https://github.com/DeterminateSystems/nix-src/actions/runs/15590867877/job/43909540271:

  nix-functional-tests> grep: repl_output: No such file or directory
  nix-functional-tests> +(repl.sh:174) cat repl_output

This is because there is a small possibility that the `nix repl` child
process hasn't created `repl_output` yet. So make sure it exists.
2025-06-11 20:53:28 +02:00
Luc Perkins 525078c59d Fix broken link in configuration description 2025-06-11 08:52:04 -07:00
Robert Hensing b3c1b70c19 Merge pull request #13345 from xokdvium/use-value-getters
libexpr: Use value getters (NFC)
2025-06-11 00:57:39 +02:00
Sergei Zimmerman 408873c2f7 libexpr: Use c_str getter 2025-06-10 13:37:06 +00:00
Sergei Zimmerman 77f5f50ec2 libexpr: Use context getter 2025-06-10 13:37:04 +00:00
Sergei Zimmerman c2aaa68c2c libexpr: Use primOp getter 2025-06-10 13:37:01 +00:00
Robert Hensing c5b1be46b4 Merge pull request #11406 from kstrafe/master
nix repl: Print which variables are just loaded
2025-06-07 13:53:13 +02:00
Robert Hensing 102259898c Merge pull request #13258 from NaN-git/opt-symbol-table
Optimize symbol table
2025-06-07 13:16:14 +02:00
h0nIg 2caccbed11 docker: shrink code - use buildenv.manifest 2025-06-06 23:54:15 +02:00
John Ericson 530efc3121 Merge pull request #13334 from NixOS/repl-incomplete-parse
repl: Don't wait on incomplete parses from imported file
2025-06-06 14:08:15 -04:00
Eelco Dolstra 91b3573770 Rethrow non-EOF errors 2025-06-06 17:09:01 +02:00
Eelco Dolstra d8b067b549 repl: Don't wait on incomplete parses from imported file
Fixes #13332.
2025-06-06 14:14:57 +02:00
Jörg Thalheim 37b4407c5c Merge pull request #13284 from getchoo-contrib/getchoo/override-input-local-registry
lockFlake(): Allow registry lookups for overridden inputs
2025-06-06 10:11:27 +02:00
Jörg Thalheim 540db8036d Merge pull request #13326 from NixOS/no-double-copy
docker.nix: Prevent double copy of nixpkgs source tree
2025-06-06 10:09:31 +02:00
Kevin Robert Stravers 13e3704329 nix repl: Add :ll to show all recently loaded variables
Invoking `:ll` will start a pager with all variables which have just
been loaded by `:lf`, `:l`, or by a flake provided to `nix repl` as an
argument.

https://github.com/NixOS/nix/issues/11404
2025-06-05 17:33:04 -04:00
Kevin Robert Stravers 3c9b9b13af nix repl: Print which variables are just loaded
When we run `nix repl nixpkgs` we get "Added 6 variables". This is not
useful as it doesn't tell us which variables the flake has exported to
our global repl scope.

This patch prints the name of each variable that was just loaded. We
currently cap printing to 20 variables in order to avoid excessive
prints.

https://github.com/NixOS/nix/issues/11404
2025-06-05 17:32:57 -04:00
h0nIg e72a0ad8c3 docker: add docu references & remove duplicate code 2025-06-05 23:28:47 +02:00
Eelco Dolstra 0a87ba0e39 Prevent double copy of nixpkgs source tree 2025-06-05 13:46:08 +02:00
Eelco Dolstra 2afc84fddf Merge pull request #13323 from NixOS/git-accessor-thread-safe
GitSourceAccessor: Make thread-safe
2025-06-05 09:33:03 +02:00
Eelco Dolstra 4bce2d723d GitSourceAccessor: Make thread-safe 2025-06-04 21:39:15 +02:00
Jörg Thalheim 4751cbef63 Merge pull request #13321 from poperigby/docker-map-attrs-flatten-replacement
Modify docker.nix to use mapAttrsToList instead of mapAttrsFlatten
2025-06-04 06:49:28 +02:00
PopeRigby cfc15d6921 Modify docker.nix to use mapAttrsToList instead of mapAttrsFlatten
The latter alias is deprecated in favor of the former, and produces a
warning.
2025-06-03 13:26:09 -07:00
Jörg Thalheim 4b8f74b3a3 Merge pull request #13320 from DeterminateSystems/keep-failed-remote-builders-warning
Clarify that `--keep-failed` with remote builders will keep the failed build directory on that builder
2025-06-03 19:28:20 +02:00
Cole Helbling 54aa73b19b fixup: only show "you can rerun" message if the derivation's platform is supported on this machine 2025-06-03 08:38:50 -07:00
Cole Helbling 0712339912 --keep-failed with remote builders will keep the failed build directory on that builder 2025-06-03 07:50:29 -07:00
Jörg Thalheim 6a761c1a60 Merge pull request #13296 from xokdvium/empty-accessor-prefix
Clear `displayPrefix` in `makeEmptySourceAccessor`
2025-06-03 01:32:46 +02:00
Jörg Thalheim e72f19eb28 Merge pull request #13300 from NixOS/deletePath-keep-going
deletePath(): Keep going when encountering an undeletable file
2025-06-02 18:00:10 +02:00
Eelco Dolstra 6b6d3dcf34 deletePath(): Keep going when encountering an undeletable file
This should reduce the impact of #5207.
2025-06-02 14:54:01 +02:00
Eelco Dolstra 97462bde31 Merge pull request #13305 from donottellmetonottellyou/donottellmetonottellyou/document-nix-channel-tarball-ttl
add documentation of tarball-ttl to nix-channel
2025-06-02 14:37:59 +02:00
Eelco Dolstra 86767349d5 Merge pull request #13304 from fzakaria/small-fix
Fix overriding gtest with gmock
2025-06-02 14:02:47 +02:00
Eelco Dolstra 7ad4426b8b Merge pull request #13310 from xokdvium/cleanup-position
libutil: Use `std::shared_ptr<const Pos>` and simplify `Pos` class constructors
2025-06-02 14:02:25 +02:00
Eelco Dolstra 97e3c3fff8 Merge pull request #13309 from NaN-git/fix-substring
libexpr: fix various overflows and type mismatches
2025-06-02 14:01:48 +02:00
Eelco Dolstra 16e4b3b7df Merge pull request #13311 from xokdvium/dedup-prim-op-app-prim-op
libexpr: Deduplicate `Value::primOpAppPrimOp`
2025-06-02 14:00:29 +02:00
Philipp Otterbein afd9c78508 libexpr: fix various overflows and type mismatches 2025-06-02 02:06:48 +02:00
Sergei Zimmerman 9563b509ff libexpr: Deduplicate Value::primOpAppPrimOp
`getPrimOp` function was basically identical to existing
`Value::primOpAppPrimOp` modulo some trivial differences.
Makes sense to reuse existing code for that.
2025-06-01 21:16:01 +00:00
Sergei Zimmerman cdb8567473 libutil: Don't explicitly default special member functions
Since all of the member types are copyable/movable the compiler
will generate all of those by default anyway.
2025-06-01 20:55:28 +00:00
Sergei Zimmerman b73e706589 libutil: Use std::shared_ptr<const Pos> instead of std::shared_ptr<Pos>
There's actually no mutation happening so there's no point in using
a mutable shared_ptr. Furthermore, this makes it much more evident to
the reader that no actual mutation (especially in multithreaded case)
is happening.

Also get rid of redundant constructor that isn't actually used anywhere
other than `Pos::operator std::shared_ptr<Pos>` which just passes in &*this,
(identical to just `this`), which can't be nullptr.
2025-06-01 20:55:26 +00:00
Jade MaskerandValentin Gagarin c0ceaa2d5d add reference to the tarball-ttl documentation
Co-authored-by: Valentin Gagarin <valentin@gagarin.work>
2025-06-01 15:56:46 -04:00
Jade MaskerandValentin Gagarin 633d39109b remove overly verbose mention of fetchTarball
Co-authored-by: Valentin Gagarin <valentin@gagarin.work>
2025-06-01 15:55:49 -04:00
Philipp Otterbein ed4e512dcd symbol-table: reference entries instead of allocating Values 2025-06-01 21:00:01 +02:00
Philipp Otterbein 94bbaddb93 symbol-table: reduce memory usage and use boost::unordered_flat_set 2025-06-01 19:20:11 +02:00
Jade Lynn Masker 6badd21b6a add documentation of tarball-ttl to nix-channel 2025-05-31 19:05:29 -04:00
Farid Zakaria 58e34a2d27 Overriding gtest with gmock
How did this work before...

* Added .direnv/ to gitignore
2025-05-31 07:41:27 -07:00
Eelco Dolstra 587b5f5361 Merge pull request #13297 from NixOS/split-linux-builder
Split LinuxDerivationBuilder
2025-05-30 12:35:01 +02:00
Eelco Dolstra 908129eb22 Cleanup 2025-05-30 11:54:54 +02:00
John Ericson 81f5d6763d Merge pull request #13299 from xokdvium/restore-packaging-overriding
flake: Restore `packaging-overriding` check
2025-05-29 22:31:15 -04:00
John Ericson 1a73f7aa6d Merge pull request #13298 from xokdvium/drop-pre-commit-override
flake: Drop `pre-commit` override
2025-05-29 22:22:57 -04:00
Sergei Zimmerman 4fa991a680 flake: Restore packaging-overriding check
The underlying bug seems to have been fixed in diffoscope 293 [1] [2].
Our nixpkgs input has 295.

[1]: https://github.com/NixOS/nixpkgs/pull/393381#issuecomment-2766703347
[2]: https://diffoscope.org/news/diffoscope-292-released/
2025-05-29 22:35:50 +00:00
Sergei Zimmerman dd80f16376 flake: Drop pre-commit override
`pre-commit` builds fine with the flake's input nixpkgs
on i686-linux.
2025-05-29 21:57:08 +00:00
Eelco Dolstra 4dc419eaec Split LinuxDerivationBuilder
This restores doing seccomp/personality initialization even when
sandboxing is disabled.

https://hydra.nixos.org/build/298482132
2025-05-29 21:57:25 +02:00
Eelco Dolstra b7fd872147 Cleanup 2025-05-29 21:39:22 +02:00
Sergei Zimmerman fba1bb0c13 Clear displayPrefix in makeEmptySourceAccessor
Judging by the comment for `makeEmptySourceAccessor` the prefix has
to be empty:

> Return a source accessor that contains only an empty root directory.

Fixes #13295.
2025-05-29 19:35:12 +00:00
John Ericson ba96067535 Merge pull request #13290 from xokdvium/drop-pch
Drop precompiled-headers.h
2025-05-28 18:46:19 -04:00
Sergei Zimmerman a653184926 Drop precompiled-headers.h
Since the migration to meson precompiled-headers.h isn't actually used
anymore and is just confusing. Meson can't handle shared pch across
subprojects [1] and without that there's no performances benefit of PCH
at all. Also rolling our own support for that isn't trivial. See [2] for
an example of how that would look like.

[1]: https://github.com/mesonbuild/meson/issues/4350
[2]: https://github.com/WayfireWM/wayfire/blob/22bc8b647351c45829baac570b828c8f852111dc/plugins/meson.build
2025-05-28 21:42:33 +00:00
John Ericson 69603a1348 Merge pull request #13282 from xokdvium/prim-match-tests
tests/functional: Add more language tests for `builtins.match`
2025-05-28 17:11:44 -04:00
John Ericson 20226c85bc Merge pull request #13273 from NixOS/deprecate-structured-attrs-hack
Deprecate hacky way of making structured attrs
2025-05-28 13:40:23 -04:00
John Ericson 5e9744c331 Merge pull request #13287 from NixOS/fix-freebsd-build
Fix FreeBSD builds
2025-05-28 13:37:29 -04:00
Eelco Dolstra 1c0ff5295b Merge pull request #13286 from NixOS/fix-warning
Fix warning when `HAVE_EMBEDDED_SANDBOX_SHELL` is not set
2025-05-28 19:36:53 +02:00
John EricsonandEelco Dolstra 7577d2d3ae Deprecate hacky way of making structured attrs
The method tested for in the previous commit is now deprecated.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-05-28 12:59:04 -04:00
John Ericson 24f5d7a9c3 Fix warning when HAVE_EMBEDDED_SANDBOX_SHELL is not set
Clang doesn't like the double indent that is needed for the `if...else`
that is CPP'd away. Adding braces is fine in the `if...else...` case,
and fine as a naked block in the CPP'd away case, and properly-indented
both ways.
2025-05-28 12:49:13 -04:00
John Ericson 8d725fdcb0 Fix FreeBSD builds 2025-05-28 12:47:33 -04:00
John Ericson cc926c332c Merge pull request #13283 from NixOS/fix-mingw
Fix mingw build
2025-05-28 12:31:56 -04:00
John Ericson a353b2f4b2 Test hacky way of making structured attrs 2025-05-28 11:52:17 -04:00
Seth Flynn d0a2323829 lockFlake(): Allow registry lookups for overridden inputs
Fixes #13144
2025-05-28 00:24:41 -04:00
Eelco Dolstra 6686b54077 Fix mingw build
https://hydra.nixos.org/build/298331457
2025-05-28 00:48:10 +02:00
Sergei Zimmerman d555d6b404 tests/functional: Add more language tests for builtins.match
These tests have been collected from nixpkgs f870c6ccc8951fc48aeb293cf3e98ade6ac42668
usage of builtins.match for x86_64-linux eval system. At most 2 matching and
non-matching cases are included for each encountered regex. This should
hopefully add more confidence when possibly trying to switch the regex implementation
in the future.
2025-05-27 22:28:13 +00:00
John Ericson de71cebc3a Merge pull request #13281 from NixOS/freebsd-utils
Prepare for FreeBSD sandboxing support
2025-05-27 15:37:18 -04:00
625dce659a Prepare for FreeBSD sandboxing support
This is the utility changes from #9968, which were easier to rebase
first.

I (@Ericson2314) didn't write this code; I just rebased it.

Co-Authored-By: Artemis Tosini <me@artem.ist>
Co-Authored-By: Audrey Dutcher <audrey@rhelmot.io>
2025-05-27 14:54:50 -04:00
John Ericson 653a93ac0f Merge pull request #13276 from NixOS/split-derivation-builder
Move platform-specific code out of `DerivationBuilder`
2025-05-27 13:39:02 -04:00
Eelco Dolstra cf9d962086 Remove unused variable 2025-05-27 17:54:02 +02:00
Jörg Thalheim f7c94cd908 Merge pull request #13278 from DeterminateSystems/gustavderdrache/system-builderror
Make platform checks throw BuildError like other failures
2025-05-27 16:39:55 +02:00
Jörg Thalheim 9ddac1b516 Merge pull request #13277 from zimbatm/flake-archive-no-check-sigs
nix flake archive: add --no-check-sigs option
2025-05-27 16:38:42 +02:00
Jörg Thalheim 161bf86457 Merge pull request #13279 from DeterminateSystems/gustavderdrache/trace-import-from-derivation
Emit warnings for IFDs with new `trace-import-from-derivation` option
2025-05-27 16:37:29 +02:00
Eelco Dolstra f9fdf94e12 Fix macOS build 2025-05-27 15:25:51 +02:00
gustavderdracheandEelco Dolstra d80f0fb15a Add test for output warning to ensure stability
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-05-27 09:18:51 -04:00
gustavderdracheandEelco Dolstra ce89c8c114 Log warnings on IFD with new option
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-05-27 09:18:51 -04:00
gustavderdracheandCole Helbling b04962b33b Make platform checks throw BuildError like other failures
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2025-05-27 08:57:13 -04:00
Eelco Dolstra 352ca238a9 Move cgroup support 2025-05-27 14:06:32 +02:00
Eelco Dolstra b27e684ca5 Inline initTmpDir() 2025-05-27 14:00:28 +02:00
Eelco Dolstra 774678b87f Get rid of tmpDirInSandbox variable 2025-05-27 14:00:21 +02:00
Eelco Dolstra c9b55fa3f0 Move autoDelChroot 2025-05-27 14:00:16 +02:00
Eelco Dolstra 9e2151d839 Move seccomp code 2025-05-27 14:00:12 +02:00
Eelco Dolstra b623fe8d14 Move doBind() 2025-05-27 14:00:07 +02:00
Eelco Dolstra 9f8f3968e3 Eliminate useChroot 2025-05-27 13:59:40 +02:00
Eelco Dolstra 67408807d8 Move pathsInChroot 2025-05-27 13:57:41 +02:00
Eelco Dolstra 5653bf5e0a DerivationBuilder: Move Linux/Darwin-specific code into subclasses 2025-05-27 13:42:22 +02:00
zimbatm 80a4293486 nix flake archive: add --no-check-sigs option
Allows to copy the archive to a remote host and not get

    error: cannot add path '/nix/store/01x2k4nlxcpyd85nnr0b9gm89rm8ff4x-source' because it lacks a signature by a trusted key
2025-05-27 10:05:21 +00:00
John Ericson 74a107d364 Merge pull request #13270 from NixOS/no-optional-drv-type
DerivationBuilderImpl: Drop std::optional from derivationType
2025-05-27 03:46:06 -04:00
Eelco Dolstra aaca9711fc DerivationBuilderImpl: Drop std::optional from derivationType
No point in computing this lazily, since it's pretty much the first
thing the DerivationBuilder does.
2025-05-27 09:04:29 +02:00
Jörg Thalheim 6adee1145f Merge pull request #13275 from RossComputerGuy/fix/static-deps-nix
Remove propagated-build-inputs when static
2025-05-27 09:01:12 +02:00
Tristan Ross d07852b5f3 Remove propagated-build-inputs when static 2025-05-26 21:54:19 -07:00
John Ericson a2f65dd43b Merge pull request #13274 from NixOS/drop-magic-nix-cache
Drop magic-nix-cache
2025-05-26 20:44:10 -04:00
Eelco Dolstra 9cc8be2674 Drop magic-nix-cache
This no longer works, see https://determinate.systems/posts/magic-nix-cache-free-tier-eol/.
2025-05-26 23:30:16 +02:00
John Ericson 32d19040e7 Merge pull request #13268 from NixOS/derivation-builder-cleanup
Remove `buildUser` from `DerivationBuilder`
2025-05-26 12:10:57 -04:00
Eelco Dolstra 93ae95be83 Remove duplicate comments on DerivationBuilderImpl overriden methods
Having the exact same doc comments isn't very useful/maintainable.
2025-05-26 16:05:54 +02:00
Eelco Dolstra 69914e4b3c Remove buildUser from DerivationBuilder
The use of a `buildUser` is an implementation detail of some types of
sandboxes that shouldn't exposed.
2025-05-26 16:05:35 +02:00
Jörg Thalheim ed3e2a71ea Merge pull request #12994 from fricklerhandwerk/reword-derivation
glossary: re-introduce "derivation"
2025-05-26 14:58:52 +02:00
Eelco Dolstra 170f6021cf Merge pull request #13265 from Mic92/fix-shallow-clone-subset
Improve errors when we are trying to access a git repository with partial history (+ fix fetchGit on these repos)
2025-05-26 14:53:08 +02:00
Jörg Thalheim 5419d82547 tests/fetchGit: work around something that looks a bash parsing quirk.
Before we got something like this but only inside the VM test:

 vm-test-run-functional-tests-on-nixos_user> machine # fetchGit.sh: line 286: unexpected EOF while looking for matching `)'

We now try to do not too much in a single line, so that the bash parser does not get confused.
This also seems more readable and better quoted.
2025-05-26 12:44:31 +02:00
Jörg Thalheim b1ccfaa080 git/revCount: improve error message when we have incomplete git history 2025-05-26 12:44:31 +02:00
Jörg Thalheim 0479db934c fetchGit: don't compute revCount on shallow repository
This can never work and leads to a crash bug.
2025-05-26 09:56:44 +02:00
Jörg Thalheim 247f166f87 Merge pull request #13245 from stefanboca/sb/push-yoorqzwsxmzn
nix-profile{,-daemon}.fish: check for profile in XDG_DATA_HOME
2025-05-26 00:14:40 +02:00
Jörg Thalheim d929eed168 Merge pull request #13262 from xokdvium/typos
Fix various typos in source code
2025-05-25 23:42:21 +02:00
Sergei Zimmerman 114de63d88 Fix various typos in source code
This only touches code comments, class names, documentation,
enumeration names and tests.
2025-05-25 20:14:11 +00:00
Jörg Thalheim 4777734974 Merge pull request #13261 from xokdvium/eval-profiler-derivations
libexpr: Include derivation names in the call stack profile
2025-05-25 21:15:54 +02:00
Sergei Zimmerman 9e97ecabb6 libexpr: Include derivation names in the call stack profile
This makes the profiler much more useful by actually distiguishing
different derivations being evaluated. This does make the implementation
a bit more convoluted, but I think it's worth it.
2025-05-25 15:53:01 +00:00
Sergei Zimmerman a76c76a9d5 libexpr: Make getAttr a member function of EvalState 2025-05-25 15:52:58 +00:00
Sergei Zimmerman 128750225d libexpr: Pass mutable EvalState to EvalProfiler
Sometimes the profiler might want to do evaluation (e.g. for getting
derivation names). This is not ideal, but is really necessary
to make the profiler stack traces useful for end users.
2025-05-25 15:52:56 +00:00
Jörg Thalheim 543cee1c92 Merge pull request #13244 from stefanboca/sb/push-lmnqpuztqxmz
nix-profile{,-daemon}.fish: fix do not source twice
2025-05-25 17:48:25 +02:00
Stefan Boca 751f50f4ad nix-profile{,-daemon}.fish: check for profile in XDG_DATA_HOME
...and also NIX_STATE_HOME in nix-profile.fish. This is directly
translated from the bash scripts and makes the fish scripts equivalent
in functionality to the bash scripts.

Note that nix-profile.fish checks for NIX_STATE_HOME and
nix-profile-daemon.fish does not, so the two scripts are no longer
identical.
2025-05-25 17:15:32 +02:00
Jörg Thalheim 087f268615 Merge pull request #13256 from NixOS/fix-substitution-crash
Fix nlohmann error in fromStructuredAttrs()
2025-05-25 07:54:28 +02:00
tomberek 68f8e1c32e Merge pull request #13257 from NixOS/robust-s3-test
Make the S3 test more robust
2025-05-24 14:10:54 -04:00
Eelco Dolstra ca9696748a Make the S3 test more robust
Waiting for the minio unit is apparently not reliable enough, so let's
also wait for the port.
2025-05-24 01:30:51 +02:00
Eelco Dolstra c66eb9cef7 Add test 2025-05-24 00:40:06 +02:00
Eelco Dolstra d877b0c0cc fromStructuredAttrs(): Don't crash if exportReferencesGraph is a string
Fixes

  error: [json.exception.type_error.302] type must be array, but is string

and other crashes.

Fixes #13254.
2025-05-24 00:17:03 +02:00
Eelco Dolstra fa6e10ea6a Don't use 'callback' object that we may have moved out of 2025-05-24 00:14:03 +02:00
Jörg Thalheim 81884c36a3 Merge pull request #13248 from xokdvium/logger-exceptions
libmain: Catch logger exceptions in `handleExceptions`
2025-05-23 15:26:13 +02:00
Jörg Thalheim 2d20cd8085 Merge pull request #13253 from xokdvium/pkgconf-c
export/meson: Don't require `-std=c++2a` for -c libraries in `.pc` files
2025-05-23 15:17:32 +02:00
Sergei Zimmerman d8da8f0cd6 export/meson: Don't require -std=c++2a for -c libraries in .pc files 2025-05-23 12:06:59 +00:00
Jörg Thalheim 4cc312a6e1 Merge pull request #13251 from Mic92/json-log-path
Add `json-log-path` setting (revisted)
2025-05-23 11:25:40 +02:00
Jörg Thalheim 7cef4559fe util/json-log-path: document unix sockets and concurrency issues 2025-05-23 10:45:12 +02:00
Eelco Dolstra 7240fb198f Add json-log-path setting
This setting specifies a path (which can be a regular file or Unix
domain socket) that receives a copy of all Nix log messages (in JSON
format).
2025-05-23 10:45:12 +02:00
Eelco Dolstra 0087188d47 Add convenience function for connecting to a Unix domain socket 2025-05-23 10:45:12 +02:00
Jörg Thalheim 906cc88f4e Merge pull request #13220 from xokdvium/flamegraph
Stack sampling flamegraph profiler
2025-05-23 10:42:49 +02:00
Jörg Thalheim 5b4806ab3c Merge pull request #13249 from NixOS/more-robust-json-logger
Make the JSON logger more robust
2025-05-23 10:25:16 +02:00
Jörg Thalheim 76358748da Merge pull request #13143 from jfly/issue-8034-nix-fmt
Expose flake directory to `nix fmt` as `PRJ_ROOT` env var
2025-05-23 10:24:26 +02:00
Eelco Dolstra 9f680874c5 Make the JSON logger more robust
We now ignore connection / write errors.
2025-05-23 03:27:57 +02:00
Sergei Zimmerman 90d1ff4805 libmain: Catch logger exceptions in handleExceptions
Avoid std::terminate in case logging code also
throws.
2025-05-22 23:08:59 +00:00
Jeremy Fleischman 17eb2e8400 Expose flake directory to nix fmt as PRJ_ROOT env var
This was discussed in https://github.com/NixOS/nix/issues/8034. I
personally like `PRJ_ROOT`, which hopefully avoids some ambiguity
around with subflakes.

I only implemented this for `nix fmt` because it doesn't let you point
at a flake not on your filesystem.

macOS compilation fixes
2025-05-22 19:18:53 +02:00
Jörg Thalheim b4bea57667 Merge pull request #13241 from fzakaria/lix-2100
cherry-pick https://gerrit.lix.systems/c/lix/+/2100
2025-05-22 18:56:40 +02:00
Sergei Zimmerman 2190bf2006 doc: Add eval-profiler documentation and release note 2025-05-21 22:16:31 +00:00
John Ericson 76a4d4c291 Merge pull request #13186 from obsidiansystems/dyn-drv-without-new-goal-type
Restore dynamic derivations!
2025-05-21 18:11:32 -04:00
John Ericson 57348b677b Restore dynamic derivations!
This method does *not* create a new type of goal. We instead just make
`DerivationGoal` more sophisticated, which is much easier to do now that
`DerivationBuildingGoal` has been split from it (and so many fields are
gone, or or local variables instead).

This avoids the need for a secondarily trampoline goal that interacted
poorly with `addWantedOutputs`. That, I hope, will mean the bugs from
before do not reappear.

There may in fact be a reason to introduce such a trampoline in the
future, but it would only happen in conjunction with getting rid of
`addWantedOutputs`.

Restores the functionality (and tests) that was reverted in
f4f28cdd0e.
2025-05-21 17:31:41 -04:00
John Ericson bf5d544d3b Merge pull request #13181 from obsidiansystems/derivation-building-goal
Split `DerivationGoal` in two
2025-05-21 16:44:16 -04:00
Sergei Zimmerman 33141cd133 tests/functional: Add tests for flamegraph profiler
The tests are mostly based on existing `function-trace.sh`
with some tests for corner cases.
2025-05-21 20:15:22 +00:00
Sergei ZimmermanandJörg Thalheim 5e74c0e4d6 libexpr: Add SampleStack stack-sampling profiler
This patch adds support for a native stack sampling
profiler to the evaluator, which saves a collapsed stack
profile information to a configurable location.

Introduced options (in `EvalSettings`):

- `eval-profile-file` - path to the collected profile file.
- `eval-profiler-frequency` - sampling frequency.
- `eval-profiler` - enumeration option for enabling the profiler.

  Currently only `flamegraph` is supported, but having this an
  enumeration rather than a boolean switch leaves the door open
  for other profiler variants (e.g. tracy).

Profile includes the following information on best-effort basis (e.g. some lambdas might
have an undefined name). Callstack information contains:

- Call site location (where the function gets called).
- Primop/lambda name of the function being called.
- Functors/partial applications don't have a name attached to them unlike special-cased primops and lambads.

For cases where callsite location isn't available we have to resort to providing
the location where the lambda itself is defined. This removes some of the confusing
`«none»:0` locations in the profile from previous attempts.

Example usage with piping directly into zstd for compression:

```
nix eval --no-eval-cache nixpkgs#nixosTests.gnome \
  --eval-profiler flamegraph \
  --eval-profile-file >(zstd -of nix.profile.zstd)
```

Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
2025-05-21 20:15:19 +00:00
Stefan Boca b9ed3ae36e nix-profile{,-daemon}.fish: fix do not source twice
Commit b36637c8f7 set
`__ETC_PROFILE_NIX_SOURCED` globally, but this is not enough to prevent
the script from being run again by child shells, because the
variable was not exported and thus not inherited by any child process.
Exporting the variable also agrees with the bash scripts.

Notably, the old behavior broke `nix develop -c fish` in some cases,
because the profile bin directory got prepended to the path, causing
binaries from the profile to override binareis from the devshell.
2025-05-21 11:29:47 -07:00
Farid Zakaria 6c85a90b87 Change to atomic<uint32_t> 2025-05-21 10:07:04 -07:00
Farid Zakaria 8fd15ac228 Add canon for path 2025-05-21 10:05:23 -07:00
Farid Zakaria 9e26549c2b Consolidate tempName and makeTempPath 2025-05-21 09:01:49 -07:00
Farid ZakariaandLily Ballard 6aed9d877c cherry-pick https://gerrit.lix.systems/c/lix/+/2100
Cherry-pick https://gerrit.lix.systems/c/lix/+/2100

This change fixes a potential concurrency failure when accessing random
which is not thread safe.

Co-authored-by: Lily Ballard <lily@ballards.net>
2025-05-21 08:49:09 -07:00
Stefan Boca f627b8c721 nix-profile{,-daemon}.fish: format with fish_indent 2025-05-20 22:11:39 -07:00
Eelco Dolstra e22142e11a Merge pull request #13224 from NixOS/nix-profile-add
Rename `nix profile install` to `nix profile add`
2025-05-20 20:15:31 +02:00
Eelco Dolstra 6d59fa03fd Add release note 2025-05-20 18:24:13 +02:00
John Ericson 3b617e471b Split DerivationGoal in two
This separation of concerns is generally good, but in particular sets up
for removing `addWantedOutputs` next.
2025-05-20 11:54:53 -04:00
John Ericson d1295448e0 Copy files before split
Same technique as 6c2a7fdc49.
2025-05-20 11:54:52 -04:00
John Ericson a6c5d56af7 Merge pull request #13177 from obsidiansystems/less-useDerivation
Remove `useDerivation`
2025-05-20 11:39:48 -04:00
Eelco DolstraandJörg Thalheim 803c0086f3 Update src/nix/profile-add.md
Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2025-05-20 15:39:40 +02:00
Jörg Thalheim fad975b58f Merge pull request #13231 from gwennlbh/docs-attrset-functor
docs: clarify attrset __functor
2025-05-20 09:03:13 +02:00
Jörg Thalheim 7616d290a0 Merge pull request #13232 from gwennlbh/clarify-implication-operator
docs: add another equivalence for the implication operator
2025-05-20 09:01:29 +02:00
Jörg Thalheim 6867a4688a Merge pull request #13233 from xokdvium/function-args-opt
libexpr: Use `attrs.alreadySorted()` in primop_functionArgs
2025-05-20 08:29:42 +02:00
John Ericson fe41bdefd7 Merge pull request #13234 from xokdvium/transparent-comparator-map
Use C++14 N3657 transparent comparator for `StringMap` and `StringPairs` (NFC)
2025-05-19 17:55:23 -04:00
Sergei Zimmerman 49d026083b Use transparent comparators for StringMap and StringPairs 2025-05-19 20:38:19 +00:00
Sergei Zimmerman 8ee513379a Use StringMap instead of std::map<std::string, std::string> throughout the codebase 2025-05-19 20:33:28 +00:00
Sergei Zimmerman a4bfd559f1 libexpr: Use attrs.alreadySorted() in primop_functionArgs
Formals are already sorted as it's an invariant of `Formals`.
2025-05-19 19:53:20 +00:00
Gwenn Le Bihan 51151c2c28 docs: add another equivalence for the implication operator
the second equivalence, using a if-else expression, aligns much closer to how most humans think about implication, adding it might help some people :)
2025-05-19 17:37:29 +02:00
Gwenn Le Bihan 5c512a4ee1 docs: clarify attrset __functor 2025-05-19 16:58:16 +02:00
Eelco Dolstra 8c10104e9e Merge pull request #13228 from Mic92/json-utils
libutil-tests/json-utils: fix -Werror=sign-compare error
2025-05-19 16:15:09 +02:00
Robert Hensing f18af849fe Merge pull request #13170 from MattSturgeon/fix/call-flake/rel-path
Use correct parent `outPath` for relative path inputs
2025-05-19 14:13:54 +02:00
Jörg Thalheim 1290b7e53d libutil-tests/json-utils: fix -Werror=sign-compare error
I am on a newer different nixpkgs branch, so I am getting this error
2025-05-19 09:26:24 +02:00
Jörg Thalheim e7078d4de1 Merge pull request #13142 from xokdvium/gc-root-boost-regex
libstore: Use `boost::regex` for GC root discovery
2025-05-18 21:44:41 +02:00
Jörg Thalheim 638b7ec6c5 Merge pull request #13219 from xokdvium/eval-profiler
libexpr: Add `EvalProfiler` and use it for `FunctionCallTrace`
2025-05-18 21:35:43 +02:00
Jörg Thalheim f2671886f5 Merge pull request #13223 from NixOS/remove-global-fetcher-cache
Remove global fetcher cache
2025-05-18 15:58:53 +02:00
Sergei Zimmerman fa6f69f9c5 libexpr: Use EvalProfiler for FunctionCallTrace
This wires up the {pre,post}FunctionCallHook machinery
in EvalState::callFunction and migrates FunctionCallTrace
to use the new EvalProfiler mechanisms for tracing.

Note that branches when the hook gets called are marked with [[unlikely]]
as a hint to the compiler that this is not a hot path. For non-tracing
evaluation this should be a 100% predictable branch, so the performance
cost is nonexistent.

Some measurements to prove support this point:

```
nix build .#nix-cli
nix build github:nixos/nix/d692729759e4e370361cc5105fbeb0e33137ca9e#nix-cli --out-link before
```

(Before)

```
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g before/bin/nix eval nixpkgs#gnome --no-eval-cache" --warmup 4
Benchmark 1: GC_INITIAL_HEAP_SIZE=16g before/bin/nix eval nixpkgs#gnome --no-eval-cache
  Time (mean ± σ):      2.517 s ±  0.032 s    [User: 1.464 s, System: 0.476 s]
  Range (min … max):    2.464 s …  2.557 s    10 runs
```

(After)

```
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g result/bin/nix eval nixpkgs#gnome --no-eval-cache" --warmup 4
Benchmark 1: GC_INITIAL_HEAP_SIZE=16g result/bin/nix eval nixpkgs#gnome --no-eval-cache
  Time (mean ± σ):      2.499 s ±  0.022 s    [User: 1.448 s, System: 0.478 s]
  Range (min … max):    2.472 s …  2.537 s    10 runs
```
2025-05-18 11:55:39 +00:00
Sergei Zimmerman 6b4a86a9e3 libexpr: Add EvalProfiler
This patch adds an EvalProfiler and MultiEvalProfiler that can be used
to insert hooks into the evaluation for the purposes of function tracing
(what function-trace currently does) or for flamegraph/tracy profilers.

See the following commits for how this is supposed to be integrated into
the evaluator and performance considerations.
2025-05-18 11:55:37 +00:00
Matt Sturgeon 2922e3082e Add release note for non-flake inputs having sourceInfo 2025-05-18 00:45:33 +01:00
Matt Sturgeon 72232bc28a tests/functional/flakes/non-flake-inputs: Test non-flake inputs having sourceInfo 2025-05-18 00:40:09 +01:00
Robert Hensing eaee0b4740 tests/function/flakes/relative-paths: Test #13164 2025-05-18 00:40:08 +01:00
Matt Sturgeon 46beb9af76 Use correct parent outPath for relative path inputs
Ensure relative path inputs are relative to the parent node's _actual_
`outPath`, instead of the subtly different `sourceInfo.outPath`.

Additionally, non-flake inputs now also have a `sourceInfo` attribute.

This fixes the relationship between `self.outPath` and
`self.sourceInfo.outPath` in some edge cases.

Fixes #13164
2025-05-18 00:38:17 +01:00
Eelco Dolstra 5f75738fd6 Install 'nix profile add' manpage 2025-05-17 21:18:04 +02:00
Eelco Dolstra fd407141e1 Make nix profile install an alias of nix profile add 2025-05-17 21:07:24 +02:00
Graham Christensen b37d1cdd8e Rename nix profile install to nix profile add. 2025-05-17 21:07:24 +02:00
Eelco Dolstra efcb9e36a9 Remove global fetcher cache
The cache is now part of fetchers::Settings.
2025-05-17 19:54:32 +02:00
Eelco Dolstra 7d89e46f65 Sync: Support moving out of another Sync 2025-05-17 19:38:36 +02:00
Eelco Dolstra e5e5c819dd Merge pull request #13221 from roberth/doc-rl-docroot
doc: Render verbatim `@docroot@` on contributing page
2025-05-17 19:18:59 +02:00
Robert Hensing 147a34c573 doc: Render verbatim @docroot@ on contributing page
In rendered form:

```diff
-Add references to the manual using ..
+Add references to the manual using [links like this](@docroot@/example.md)
```
2025-05-17 17:50:49 +02:00
Eelco Dolstra 39f9f5d1b6 Merge pull request #13206 from obsidiansystems/goal-init-dedup
Deduplicate the goal creation functions
2025-05-17 09:07:51 +02:00
Jörg Thalheim d692729759 Merge pull request #13212 from pbsds/lexical-illusions
docs: remove repeated "allowedReferences" and other lexical illusion
2025-05-16 16:59:23 +02:00
Jörg Thalheim 041d2374dd Merge pull request #13213 from pbsds/deriving-path-encoding-anchor
docs: fix duplicate anchor
2025-05-16 11:28:31 +02:00
Peder Bergebakken Sundt 4c50cf798e docs: fix duplicate anchor
`#deriving-path-encoding` is defined in two places, I _think_ this is the correct one to change.
2025-05-16 10:27:29 +02:00
Peder Bergebakken Sundt ea5302c4a2 docs: remove lexical illusions detected with write-good
I made this this non-markdown aware tool somewhat behave with some cursed fd+pandoc invocations
2025-05-16 09:52:19 +02:00
Peder Bergebakken Sundt cb16cd707c docs: remove repeated "allowedReferences"
This is what write-good lints as a "lexical illusion"
2025-05-16 09:20:55 +02:00
John Ericson b21fc05047 Merge pull request #13211 from xokdvium/pos-table
libexpr: Actually cache line information in PosTable
2025-05-16 00:37:57 -04:00
Sergei Zimmerman 5ea81f5b8f libexpr: Actually cache line information in PosTable
Previous code had a sneaky bug due to which no caching
actually happened:

```cpp
auto linesForInput = (*lines)[origin->offset];
```

That should have been:
```cpp
auto & linesForInput = (*lines)[origin->offset];
```

See [1].

Now that it also makes sense to make the cache bound in side
in order not to memoize all the sources without freeing any memory.
The default cache size has been chosen somewhat arbitrarily to be ~64k
origins. For reference, 25.05 nixpkgs has ~50k .nix files.

Simple benchmark:

```nix
let
  pkgs = import <nixpkgs> { };
in
builtins.foldl' (acc: el: acc + el.line) 0 (
  builtins.genList (x: builtins.unsafeGetAttrPos "gcc" pkgs) 10000
)
```

(After)

```
$ hyperfine "result/bin/nix eval -f ./test.nix"
Benchmark 1: result/bin/nix eval -f ./test.nix
  Time (mean ± σ):     292.7 ms ±   3.9 ms    [User: 131.0 ms, System: 120.5 ms]
  Range (min … max):   288.1 ms … 300.5 ms    10 runs
```

(Before)

```
hyperfine "nix eval -f ./test.nix"
Benchmark 1: nix eval -f ./test.nix
  Time (mean ± σ):     666.7 ms ±   6.4 ms    [User: 428.3 ms, System: 191.2 ms]
  Range (min … max):   659.7 ms … 681.3 ms    10 runs
```

If the origin happens to be a `all-packages.nix` or similar in size then the
difference is much more dramatic.

[1]: https://www.github.com/lix-project/lix/commit/22e3f0e9875082be7f4eec8e3caeb134a7f1c05f
2025-05-15 23:07:25 +00:00
Sergei Zimmerman 4711720efe libutil: Add LRUCache::getOrNullptr
For heavier objects it doesn't make sense to return
a std::optional with the copy of the data, when it
can be used by const reference.
2025-05-15 22:28:41 +00:00
John Ericson dc1a5135ee Merge pull request #13207 from xokdvium/drop-bear
dev-shell: Drop bear dependency
2025-05-15 17:48:08 -04:00
Sergei Zimmerman 67535263a5 dev-shell: Drop bear dependency
Since the autotools-based build system has been removed
and meson already generates compile database there's no
need to have it in the devshell.
2025-05-15 21:13:13 +00:00
John Ericson 7bd9eef772 Deduplicate the goal creation functions
The weak reference logic is the same in both these cases, and if/when I
get rid `addWantedOutputs`, also in the `DerivationGoal` case.
2025-05-15 16:59:48 -04:00
John Ericson 01207fd101 Remove useDerivation
Try to make `DerivationGoal` care less whether we're working from an
in-memory derivation or not.

It's a clean-up in its own right, but it will also help with other
cleanups under the umbrella of #12628.
2025-05-15 13:40:26 -04:00
John Ericson c1085ce849 Get rid of virtual Goal::init()
Now, each class provides the initial coroutine by value. This avoids
some sketchy virtual function stuff, and will also be further put to
good use in the next commit.
2025-05-15 13:40:26 -04:00
Jörg Thalheim 0d3750e902 Merge pull request #13202 from NixOS/fix-nix-copy-closure-test
Fix nix-copy-closure VM test
2025-05-15 17:50:12 +02:00
Eelco Dolstra d626348f42 Fix nix-copy-closure VM test
https://hydra.nixos.org/build/297112538
2025-05-15 12:55:08 +02:00
Jörg Thalheim ea09198f93 Merge pull request #13197 from NixOS/bump-nixpkgs
Update Nixpkgs to fix static builds
2025-05-15 12:20:06 +02:00
Eelco Dolstra 0b46025f7a Merge pull request #13196 from Mic92/store-config-base
rename StoreDirConfigItself to StoreDirConfigBase
2025-05-15 12:03:11 +02:00
Jörg Thalheim 2dacdd110c Merge pull request #13195 from NixOS/remove-nix_2_18
Remove otherNixes.nix_2_18
2025-05-15 11:43:12 +02:00
Eelco Dolstra 3ba49d7ec2 flake.lock: Update
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/f02fddb8acef29a8b32f10a335d44828d7825b78?narHash=sha256-IgBWhX7A2oJmZFIrpRuMnw5RAufVnfvOgHWgIdds%2Bhc%3D' (2025-05-01)
  → 'github:NixOS/nixpkgs/adaa24fbf46737f3f1b5497bf64bae750f82942e?narHash=sha256-qhFMmDkeJX9KJwr5H32f1r7Prs7XbQWtO0h3V0a0rFY%3D' (2025-05-13)
2025-05-15 11:31:34 +02:00
Jörg Thalheim 2dd2142754 rename StoreDirConfigItself to StoreDirConfigBase
context: https://github.com/NixOS/nix/pull/13154#discussion_r2081904653
2025-05-15 11:24:42 +02:00
Eelco Dolstra 3285b68044 Merge pull request #13176 from obsidiansystems/revert-incomplete-closure-feature
Revert #77
2025-05-15 11:13:31 +02:00
Eelco Dolstra bc85e20fb9 Remove otherNixes.nix_2_18
Nixpkgs no longer has Nix 2.18, so this fails to evaluate.
2025-05-15 11:10:21 +02:00
John Ericson c3e7e88889 Merge pull request #13194 from NixOS/bump-2.30.0
Bump 2.30.0
2025-05-14 20:40:24 -04:00
John EricsonandEelco Dolstra 99cb85cd37 Revert "If a substitute closure is incomplete, build dependencies, then retry the substituter"
As summarized in
https://github.com/NixOS/nix/issues/77#issuecomment-2843228280 the
motivation is that the complicated retry logic this introduced was
making the cleanup task #12628 harder to accomplish. It was not easy to
ascertain just what policy / semantics the extra control-flow was
implementing, in order to figure out a different way to implementing it
either.

After talking to Eelco about it, he decided we could just....get rid of
the feature entirely! It's a bit scary removing a decade+ old feature,
but I think he is right. See the release notes for more explanation.

This reverts commit 299141ecbd.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-05-14 20:16:40 -04:00
John Ericson 95e1f463c8 .mergify.yml: Add backport 2.29-maintenance entry 2025-05-14 19:59:22 -04:00
John Ericson 6591a4119c Bump version 2025-05-14 19:59:22 -04:00
Sergei Zimmerman 18a5589f9a libstore: Depend on boost_regex explicitly 2025-05-13 08:51:46 +00:00
Sergei Zimmerman f3090ef703 packaging/dependencies: Use boost without enableIcu
This reduces the closure size on master by 40MiB.

```
$ nix build github:nixos/nix/1e822bd4149a8bce1da81ee2ad9404986b07914c#nix-store --out-link closure-on-master
$ nix build .#nix-store -L --out-link closure-without-icu
$ nix path-info --closure-size -h ./closure-on-master
/nix/store/8gwr38m5h6p7245ji9jv28a2a11w1isx-nix-store-2.29.0pre  124.4 MiB
$ nix path-info --closure-size -h ./closure-without-icu
/nix/store/k0gwfykjqpnmaqbwh23nk55lhanc9g24-nix-store-2.29.0pre   86.6 MiB
```
2025-05-13 08:47:24 +00:00
Sergei Zimmerman 3a1301cd6d libstore: Use boost::regex for GC root discovery
As it turns out using `std::regex` is actually the bottleneck
for root discovery. Just substituting `std::` -> `boost::`
makes root discovery twice as fast (3x if counting only userspace time).

Some rather ad-hoc measurements to motivate the switch:

(On master)

```
nix build github:nixos/nix/1e822bd4149a8bce1da81ee2ad9404986b07914c#nix-cli --out-link result-1e822bd4149a8bce1da81ee2ad9404986b07914c
taskset -c 2,3 hyperfine "result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0"
Benchmark 1: result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0
  Time (mean ± σ):     481.6 ms ±   3.9 ms    [User: 336.2 ms, System: 142.0 ms]
  Range (min … max):   474.6 ms … 487.7 ms    10 runs
```

(After this patch)

```
taskset -c 2,3 hyperfine "result/bin/nix store gc --dry-run --max 0"
Benchmark 1: result/bin/nix store gc --dry-run --max 0
  Time (mean ± σ):     254.7 ms ±   9.7 ms    [User: 111.1 ms, System: 141.3 ms]
  Range (min … max):   246.5 ms … 281.3 ms    10 runs
```

`boost::regex` is a drop-in replacement for `std::regex`, but much faster.
Doing a simple before/after comparison doesn't surface any change in behavior:

```
result/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
```
2025-05-06 21:58:52 +00:00
Valentin Gagarin 82d598282d don't emphasize 'store derivation' 2025-04-29 10:55:38 +02:00
Valentin Gagarin 8db8371a48 glossary: re-introduce "derivation"
The "derivation" is one of the key concepts and captures the most distinctive aspect of Nix:
that we work with a certain type data (linked files) in a certain manner (using pure functions).

Here we finally arrange all the important pieces to show how they belong
together, while referring to the respective reference documentation for details.

This change also unbreaks downstream links to `//glossary#gloss-derivation`,
which had been broken by removing the glossary entry
2025-04-11 19:47:28 +02:00
334 changed files with 9253 additions and 5150 deletions

No files matched your search

+1 -3
View File
@@ -1,4 +1,2 @@
# bulk initial re-formatting with clang-format
0e35cd6f3e27760976ead16fb45008ece0185aad # !autorebase ./maintainers/format.sh --until-stable
# nixfmt 1.0.0
d6aebd884790ae82c21ef2b58a010026ce757eaf # !autorebase ./maintainers/format.sh --until-stable
e4f62e46088919428a68bd8014201dc8e379fed7 # !autorebase ./maintainers/format.sh --until-stable
@@ -9,7 +9,7 @@ inputs:
install_url:
description: "URL of the Nix installer"
required: false
default: "https://releases.nixos.org/nix/nix-2.29.1/install"
default: "https://releases.nixos.org/nix/nix-2.30.1/install"
github_token:
description: "Github token"
required: true
@@ -19,7 +19,7 @@ runs:
- name: "Download nix install artifact from master"
shell: bash
id: download-nix-installer
if: inputs.dogfood == 'true'
if: ${{ inputs.dogfood }}
run: |
RUN_ID=$(gh run list --repo "$DOGFOOD_REPO" --workflow ci.yml --branch master --status success --json databaseId --jq ".[0].databaseId")
@@ -45,6 +45,6 @@ runs:
- uses: cachix/install-nix-action@c134e4c9e34bac6cab09cf239815f9339aaaf84e # v31.5.1
with:
# Ternary operator in GHA: https://www.github.com/actions/runner/issues/409#issuecomment-752775072
install_url: ${{ inputs.dogfood == 'true' && format('{0}/install', steps.download-nix-installer.outputs.installer-path) || inputs.install_url }}
install_options: ${{ inputs.dogfood == 'true' && format('--tarball-url-prefix {0}', steps.download-nix-installer.outputs.installer-path) || '' }}
install_url: ${{ inputs.dogfood && format('{0}/install', steps.download-nix-installer.outputs.installer-path) || inputs.install_url }}
install_options: ${{ inputs.dogfood && format('--tarball-url-prefix {0}', steps.download-nix-installer.outputs.installer-path) || '' }}
extra_nix_config: ${{ inputs.extra_nix_config }}
+4 -4
View File
@@ -15,7 +15,7 @@ jobs:
fetch-depth: 0
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
dogfood: true
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
@@ -42,7 +42,7 @@ jobs:
- uses: ./.github/actions/install-nix-action
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
dogfood: false
dogfood: true
# The sandbox would otherwise be disabled by default on Darwin
extra_nix_config: |
sandbox = true
@@ -184,7 +184,7 @@ jobs:
- uses: actions/checkout@v4
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
dogfood: true
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
@@ -215,7 +215,7 @@ jobs:
path: flake-regressions/tests
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
dogfood: true
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
+22
View File
@@ -139,3 +139,25 @@ pull_request_rules:
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.29
conditions:
- label=backport 2.29-maintenance
actions:
backport:
branches:
- "2.29-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.30
conditions:
- label=backport 2.30-maintenance
actions:
backport:
branches:
- "2.30-maintenance"
labels:
- automatic backport
- merge-queue
+1 -1
View File
@@ -1 +1 @@
2.29.2
2.31.0
+5 -3
View File
@@ -8,6 +8,7 @@ nix = find_program('nix', native : true)
mdbook = find_program('mdbook', native : true)
bash = find_program('bash', native : true)
rsync = find_program('rsync', required: true, native: true)
pymod = import('python')
python = pymod.find_installation('python3')
@@ -84,7 +85,7 @@ manual = custom_target(
@0@ @INPUT0@ @CURRENT_SOURCE_DIR@ > @DEPFILE@
@0@ @INPUT1@ summary @2@ < @CURRENT_SOURCE_DIR@/source/SUMMARY.md.in > @2@/source/SUMMARY.md
sed -e 's|@version@|@3@|g' < @INPUT2@ > @2@/book.toml
rsync -r --include='*.md' @CURRENT_SOURCE_DIR@/ @2@/
@4@ -r --include='*.md' @CURRENT_SOURCE_DIR@/ @2@/
(cd @2@; RUST_LOG=warn @1@ build -d @2@ 3>&2 2>&1 1>&3) | { grep -Fv "because fragment resolution isn't implemented" || :; } 3>&2 2>&1 1>&3
rm -rf @2@/manual
mv @2@/html @2@/manual
@@ -94,6 +95,7 @@ manual = custom_target(
mdbook.full_path(),
meson.current_build_dir(),
meson.project_version(),
rsync.full_path(),
),
],
input : [
@@ -246,11 +248,11 @@ nix3_manpages = [
'nix3-nar',
'nix3-path-info',
'nix3-print-dev-env',
'nix3-profile',
'nix3-profile-add',
'nix3-profile-diff-closures',
'nix3-profile-history',
'nix3-profile-install',
'nix3-profile-list',
'nix3-profile',
'nix3-profile-remove',
'nix3-profile-rollback',
'nix3-profile-upgrade',
+18 -17
View File
@@ -46,23 +46,24 @@ mkMesonDerivation (finalAttrs: {
];
# Hack for sake of the dev shell
passthru.externalNativeBuildInputs = [
meson
ninja
(lib.getBin lowdown-unsandboxed)
mdbook
mdbook-linkcheck
jq
python3
rsync
changelog-d
]
++ lib.optionals (!officialRelease) [
# When not an official release, we likely have changelog entries that have
# yet to be rendered.
# When released, these are rendered into a committed file to save a dependency.
changelog-d
];
passthru.externalNativeBuildInputs =
[
meson
ninja
(lib.getBin lowdown-unsandboxed)
mdbook
mdbook-linkcheck
jq
python3
rsync
changelog-d
]
++ lib.optionals (!officialRelease) [
# When not an official release, we likely have changelog entries that have
# yet to be rendered.
# When released, these are rendered into a committed file to save a dependency.
changelog-d
];
nativeBuildInputs = finalAttrs.passthru.externalNativeBuildInputs ++ [
nix-cli
@@ -0,0 +1,6 @@
---
synopsis: "`build-cores = 0` now auto-detects CPU cores"
prs: [13402]
---
When `build-cores` is set to `0`, nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
@@ -1,17 +0,0 @@
---
synopsis: Non-flake inputs now contain a `sourceInfo` attribute
issues: 13164
prs: 13170
---
Flakes have always a `sourceInfo` attribute which describes the source of the flake.
The `sourceInfo.outPath` is often identical to the flake's `outPath`, however it can differ when the flake is located in a subdirectory of its source.
Non-flake inputs (i.e. inputs with `flake = false`) can also be located at some path _within_ a wider source.
This usually happens when defining a relative path input within the same source as the parent flake, e.g. `inputs.foo.url = ./some-file.nix`.
Such relative inputs will now inherit their parent's `sourceInfo`.
This also means it is now possible to use `?dir=subdir` on non-flake inputs.
This iterates on the work done in 2.26 to improve relative path support ([#10089](https://github.com/NixOS/nix/pull/10089)),
and resolves a regression introduced in 2.28 relating to nested relative path inputs ([#13164](https://github.com/NixOS/nix/issues/13164)).
+2
View File
@@ -57,6 +57,7 @@
- [Tuning Cores and Jobs](advanced-topics/cores-vs-jobs.md)
- [Verifying Build Reproducibility](advanced-topics/diff-hook.md)
- [Using the `post-build-hook`](advanced-topics/post-build-hook.md)
- [Evaluation profiler](advanced-topics/eval-profiler.md)
- [Command Reference](command-ref/index.md)
- [Common Options](command-ref/opt-common.md)
- [Common Environment Variables](command-ref/env-common.md)
@@ -136,6 +137,7 @@
- [Contributing](development/contributing.md)
- [Releases](release-notes/index.md)
{{#include ./SUMMARY-rl-next.md}}
- [Release 2.30 (2025-07-07)](release-notes/rl-2.30.md)
- [Release 2.29 (2025-05-14)](release-notes/rl-2.29.md)
- [Release 2.28 (2025-04-02)](release-notes/rl-2.28.md)
- [Release 2.27 (2025-03-03)](release-notes/rl-2.27.md)
@@ -0,0 +1,33 @@
# Using the `eval-profiler`
Nix evaluator supports [evaluation](@docroot@/language/evaluation.md)
[profiling](<https://en.wikipedia.org/wiki/Profiling_(computer_programming)>)
compatible with `flamegraph.pl`. The profiler samples the nix
function call stack at regular intervals. It can be enabled with the
[`eval-profiler`](@docroot@/command-ref/conf-file.md#conf-eval-profiler)
setting:
```console
$ nix-instantiate "<nixpkgs>" -A hello --eval-profiler flamegraph
```
Stack sampling frequency and the output file path can be configured with
[`eval-profile-file`](@docroot@/command-ref/conf-file.md#conf-eval-profile-file)
and [`eval-profiler-frequency`](@docroot@/command-ref/conf-file.md#conf-eval-profiler-frequency).
By default the collected profile is saved to `nix.profile` file in the current working directory.
The collected profile can be directly consumed by `flamegraph.pl`:
```console
$ flamegraph.pl nix.profile > flamegraph.svg
```
The line information in the profile contains the location of the [call
site](https://en.wikipedia.org/wiki/Call_site) position and the name of the
function being called (when available). For example:
```
/nix/store/x9wnkly3k1gkq580m90jjn32q9f05q2v-source/pkgs/top-level/default.nix:167:5:primop import
```
Here `import` primop is called at `/nix/store/x9wnkly3k1gkq580m90jjn32q9f05q2v-source/pkgs/top-level/default.nix:167:5`.
+8 -12
View File
@@ -215,18 +215,14 @@ nix build .#nix-everything-x86_64-w64-mingw32
For historic reasons and backward-compatibility, some CPU and OS identifiers are translated as follows:
| `host_machine.cpu_family()` | `host_machine.endian()` | Nix |
|-----------------------------|-------------------------|---------------------|
| `x86` | | `i686` |
| `arm` | | `host_machine.cpu()`|
| `ppc` | `little` | `powerpcle` |
| `ppc64` | `little` | `powerpc64le` |
| `ppc` | `big` | `powerpc` |
| `ppc64` | `big` | `powerpc64` |
| `mips` | `little` | `mipsel` |
| `mips64` | `little` | `mips64el` |
| `mips` | `big` | `mips` |
| `mips64` | `big` | `mips64` |
| `config.guess` | Nix |
|----------------------------|---------------------|
| `amd64` | `x86_64` |
| `i*86` | `i686` |
| `arm6` | `arm6l` |
| `arm7` | `arm7l` |
| `linux-gnu*` | `linux` |
| `linux-musl*` | `linux` |
## Compilation environments
+15 -5
View File
@@ -31,9 +31,22 @@
The industry term for storage and retrieval systems using [content addressing](#gloss-content-address). A Nix store also has [input addressing](#gloss-input-addressed-store-object), and metadata.
- [derivation]{#gloss-derivation}
A derivation can be thought of as a [pure function](https://en.wikipedia.org/wiki/Pure_function) that produces new [store objects][store object] from existing store objects.
Derivations are implemented as [operating system processes that run in a sandbox](@docroot@/store/building.md#builder-execution).
This sandbox by default only allows reading from store objects specified as inputs, and only allows writing to designated [outputs][output] to be [captured as store objects](@docroot@/store/building.md#processing-outputs).
A derivation is typically specified as a [derivation expression] in the [Nix language], and [instantiated][instantiate] to a [store derivation].
There are multiple ways of obtaining store objects from store derivatons, collectively called [realisation][realise].
[derivation]: #gloss-derivation
- [store derivation]{#gloss-store-derivation}
A single build task.
A [derivation] represented as a [store object].
See [Store Derivation](@docroot@/store/derivation/index.md#store-derivation) for details.
[store derivation]: #gloss-store-derivation
@@ -57,10 +70,7 @@
- [derivation expression]{#gloss-derivation-expression}
A description of a [store derivation] in the Nix language.
The output(s) of a derivation are store objects.
Derivations are typically specified in Nix expressions using the [`derivation` primitive](./language/derivations.md).
These are translated into store layer *derivations* (implicitly by `nix-env` and `nix-build`, or explicitly by `nix-instantiate`).
A description of a [store derivation] using the [`derivation` primitive](./language/derivations.md) in the [Nix language].
[derivation expression]: #gloss-derivation-expression
@@ -53,23 +53,13 @@ Derivations can declare some infrequently used optional attributes.
- [`__structuredAttrs`]{#adv-attr-structuredAttrs}\
If the special attribute `__structuredAttrs` is set to `true`, the other derivation
attributes are serialised into a file in JSON format. The environment variable
`NIX_ATTRS_JSON_FILE` points to the exact location of that file both in a build
and a [`nix-shell`](../command-ref/nix-shell.md). This obviates the need for
[`passAsFile`](#adv-attr-passAsFile) since JSON files have no size restrictions,
unlike process environments.
attributes are serialised into a file in JSON format.
It also makes it possible to tweak derivation settings in a structured way; see
[`outputChecks`](#adv-attr-outputChecks) for example.
This obviates the need for [`passAsFile`](#adv-attr-passAsFile) since JSON files have no size restrictions, unlike process environments.
It also makes it possible to tweak derivation settings in a structured way;
see [`outputChecks`](#adv-attr-outputChecks) for example.
As a convenience to Bash builders,
Nix writes a script that initialises shell variables
corresponding to all attributes that are representable in Bash. The
environment variable `NIX_ATTRS_SH_FILE` points to the exact
location of the script, both in a build and a
[`nix-shell`](../command-ref/nix-shell.md). This includes non-nested
(associative) arrays. For example, the attribute `hardening.format = true`
ends up as the Bash associative array element `${hardening[format]}`.
See the [corresponding section in the derivation page](@docroot@/store/derivation/index.md#structured-attrs) for further details.
> **Warning**
>
+1 -1
View File
@@ -1,6 +1,6 @@
# Nix Language
The Nix language is designed for conveniently creating and composing *derivations* – precise descriptions of how contents of existing files are used to derive new files.
The Nix language is designed for conveniently creating and composing [derivations](@docroot@/glossary.md#gloss-derivation) – precise descriptions of how contents of existing files are used to derive new files.
> **Tip**
>
+2 -2
View File
@@ -225,8 +225,8 @@ passed in first , e.g.,
```nix
let add = { __functor = self: x: x + self.x; };
inc = add // { x = 1; };
in inc 1
inc = add // { x = 1; }; # inc is { x = 1; __functor = (...) }
in inc 1 # equivalent of `add.__functor add 1` i.e. `1 + self.x`
```
evaluates to `2`. This can be used to attach metadata to a function
@@ -12,7 +12,7 @@ $ ln -s /nix/store/d718ef...-foo /nix/var/nix/gcroots/bar
That is, after this command, the garbage collector will not remove
`/nix/store/d718ef...-foo` or any of its dependencies.
Subdirectories of `prefix/nix/var/nix/gcroots` are also searched for
symlinks. Symlinks to non-store paths are followed and searched for
roots, but symlinks to non-store paths *inside* the paths reached in
that way are not followed to prevent infinite recursion.
Subdirectories of `prefix/nix/var/nix/gcroots` are searched
recursively. Symlinks to store paths count as roots. Symlinks to
non-store paths are ignored, unless the non-store path is itself a
symlink to a store path.
@@ -91,3 +91,7 @@ is a JSON object with the following fields:
* `env`:
The environment passed to the `builder`.
* `structuredAttrs`:
[Strucutured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
Structured attributes are JSON, and thus embedded as-is.
+1 -1
View File
@@ -269,7 +269,7 @@
e.g. `--warn-large-path-threshold 100M`.
# Contributors
## Contributors
This release was made possible by the following 43 contributors:
+1 -1
View File
@@ -77,7 +77,7 @@
`<nix/fetchurl.nix>` is also known as the builtin derivation builder `builtin:fetchurl`. It's not to be confused with the evaluation-time function `builtins.fetchurl`, which was not affected by this issue.
# Contributors
## Contributors
This release was made possible by the following 58 contributors:
+1 -1
View File
@@ -76,7 +76,7 @@
- Evaluation caching now works for dirty Git workdirs [#11992](https://github.com/NixOS/nix/pull/11992)
# Contributors
## Contributors
This release was made possible by the following 45 contributors:
+1 -1
View File
@@ -47,7 +47,7 @@
blake3-34P4p+iZXcbbyB1i4uoF7eWCGcZHjmaRn6Y7QdynLwU=
```
# Contributors
## Contributors
This release was made possible by the following 21 contributors:
+1 -1
View File
@@ -82,7 +82,7 @@ This completes the infrastructure overhaul for the [RFC 132](https://github.com/
Although this change is not as critical, we figured it would be good to do this API change at the same time, also.
Also note that we try to keep the C API compatible, but we decided to break this function because it was young and likely not in widespread use yet. This frees up time to make important progress on the rest of the C API.
# Contributors
## Contributors
This earlier-than-usual release was made possible by the following 16 contributors:
+1 -1
View File
@@ -111,7 +111,7 @@ This fact is counterbalanced by the fact that most of those changes are bug fixe
This in particular prevents parts of GCC 14's diagnostics from being improperly filtered away.
# Contributors
## Contributors
This release was made possible by the following 40 contributors:
+153
View File
@@ -0,0 +1,153 @@
# Release 2.30.0 (2025-07-07)
## Backward-incompatible changes and deprecations
- [`build-dir`] no longer defaults to `$TMPDIR`
The directory in which temporary build directories are created no longer defaults
to `TMPDIR` or `/tmp`, to avoid builders making their directories
world-accessible. This behavior allowed escaping the build sandbox and can
cause build impurities even when not used maliciously. We now default to `builds`
in `NIX_STATE_DIR` (which is `/nix/var/nix/builds` in the default configuration).
- Deprecate manually making structured attrs using the `__json` attribute [#13220](https://github.com/NixOS/nix/pull/13220)
The proper way to create a derivation using [structured attrs] in the Nix language is by using `__structuredAttrs = true` with [`builtins.derivation`].
However, by exploiting how structured attrs are implementated, it has also been possible to create them by setting the `__json` environment variable to a serialized JSON string.
This sneaky alternative method is now deprecated, and may be disallowed in future versions of Nix.
[structured attrs]: @docroot@/language/advanced-attributes.md#adv-attr-structuredAttrs
[`builtins.derivation`]: @docroot@/language/builtins.html#builtins-derivation
- Rename `nix profile install` to [`nix profile add`] [#13224](https://github.com/NixOS/nix/pull/13224)
The command `nix profile install` has been renamed to [`nix profile add`] (though the former is still available as an alias). This is because the verb "add" is a better antonym for the verb "remove" (i.e. `nix profile remove`). Nix also does not have install hooks or general behavior often associated with "installing".
## Performance improvements
This release has a number performance improvements, in particular:
- Reduce the size of value from 24 to 16 bytes [#13407](https://github.com/NixOS/nix/pull/13407)
This shaves off a very significant amount of memory used for evaluation (~20% percent reduction in maximum heap size and ~17% in total bytes).
## Features
- Add [stack sampling evaluation profiler] [#13220](https://github.com/NixOS/nix/pull/13220)
The Nix evaluator now supports [stack sampling evaluation profiling](@docroot@/advanced-topics/eval-profiler.md) via the [`--eval-profiler flamegraph`] setting.
It outputs collapsed call stack information to the file specified by
[`--eval-profile-file`] (`nix.profile` by default) in a format directly consumable
by `flamegraph.pl` and compatible tools like [speedscope](https://speedscope.app/).
Sampling frequency can be configured via [`--eval-profiler-frequency`] (99 Hz by default).
Unlike the existing [`--trace-function-calls`], this profiler includes the name of the function
being called when it's available.
- [`nix repl`] prints which variables were loaded [#11406](https://github.com/NixOS/nix/pull/11406)
Instead of `Added <n> variables` it now prints the first 10 variables that were added to the global scope.
- `nix flake archive`: Add [`--no-check-sigs`] option [#13277](https://github.com/NixOS/nix/pull/13277)
This is useful when using [`nix flake archive`] with the destination set to a remote store.
- Emit warnings for IFDs with [`trace-import-from-derivation`] option [#13279](https://github.com/NixOS/nix/pull/13279)
While we have the setting [`allow-import-from-derivation`] to deny import-from-derivation (IFD), sometimes users would like to observe IFDs during CI processes to gradually phase out the idiom. The new setting `trace-import-from-derivation`, when set, logs a simple warning to the console.
- `json-log-path` setting [#13003](https://github.com/NixOS/nix/pull/13003)
New setting [`json-log-path`] that sends a copy of all Nix log messages (in JSON format) to a file or Unix domain socket.
- Non-flake inputs now contain a `sourceInfo` attribute [#13164](https://github.com/NixOS/nix/issues/13164) [#13170](https://github.com/NixOS/nix/pull/13170)
Flakes have always had a `sourceInfo` attribute which describes the source of the flake.
The `sourceInfo.outPath` is often identical to the flake's `outPath`. However, it can differ when the flake is located in a subdirectory of its source.
Non-flake inputs (i.e. inputs with [`flake = false`]) can also be located at some path _within_ a wider source.
This usually happens when defining a relative path input within the same source as the parent flake, e.g. `inputs.foo.url = ./some-file.nix`.
Such relative inputs will now inherit their parent's `sourceInfo`.
This also means it is now possible to use `?dir=subdir` on non-flake inputs.
This iterates on the work done in 2.26 to improve relative path support ([#10089](https://github.com/NixOS/nix/pull/10089)),
and resolves a regression introduced in 2.28 relating to nested relative path inputs ([#13164](https://github.com/NixOS/nix/issues/13164)).
## Miscellaneous changes
- [`builtins.sort`] uses PeekSort [#12623](https://github.com/NixOS/nix/pull/12623)
Previously it used libstdc++'s `std::stable_sort()`. However, that implementation is not reliable if the user-supplied comparison function is not a strict weak ordering.
- Revert incomplete closure mixed download and build feature [#77](https://github.com/NixOS/nix/issues/77) [#12628](https://github.com/NixOS/nix/issues/12628) [#13176](https://github.com/NixOS/nix/pull/13176)
Since Nix 1.3 ([commit `299141e`] in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference).
This feature is now removed.
In the worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute.
In practice, however, we doubt even more building is very likely to happen.
Remote stores that are missing dependencies in arbitrary ways (e.g. corruption) don't seem to be very common.
On the contrary, when remote stores fail to implement the [closure property](@docroot@/store/store-object.md#closure-property), it is usually an *intentional* choice on the part of the remote store, because it wishes to serve as an "overlay" store over another store, such as `https://cache.nixos.org`.
If an "incomplete closure" is encountered in that situation, the right fix is not to do some sort of "franken-building" as this feature implemented, but instead to make sure both substituters are enabled in the settings.
(In the future, we should make it easier for remote stores to indicate this to clients, to catch settings that won't work in general before a missing dependency is actually encountered.)
## Contributors
This release was made possible by the following 32 contributors:
- Cole Helbling [**(@cole-h)**](https://github.com/cole-h)
- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
- Egor Konovalov [**(@egorkonovalov)**](https://github.com/egorkonovalov)
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
- Graham Christensen [**(@grahamc)**](https://github.com/grahamc)
- gustavderdrache [**(@gustavderdrache)**](https://github.com/gustavderdrache)
- Gwenn Le Bihan [**(@gwennlbh)**](https://github.com/gwennlbh)
- h0nIg [**(@h0nIg)**](https://github.com/h0nIg)
- Jade Masker [**(@donottellmetonottellyou)**](https://github.com/donottellmetonottellyou)
- jayeshv [**(@jayeshv)**](https://github.com/jayeshv)
- Jeremy Fleischman [**(@jfly)**](https://github.com/jfly)
- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
- Jonas Chevalier [**(@zimbatm)**](https://github.com/zimbatm)
- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
- kstrafe [**(@kstrafe)**](https://github.com/kstrafe)
- Luc Perkins [**(@lucperkins)**](https://github.com/lucperkins)
- Matt Sturgeon [**(@MattSturgeon)**](https://github.com/MattSturgeon)
- Nikita Krasnov [**(@synalice)**](https://github.com/synalice)
- Peder Bergebakken Sundt [**(@pbsds)**](https://github.com/pbsds)
- pennae [**(@pennae)**](https://github.com/pennae)
- Philipp Otterbein
- Pol Dellaiera [**(@drupol)**](https://github.com/drupol)
- PopeRigby [**(@poperigby)**](https://github.com/poperigby)
- Raito Bezarius
- Robert Hensing [**(@roberth)**](https://github.com/roberth)
- Samuli Thomasson [**(@SimSaladin)**](https://github.com/SimSaladin)
- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
- Seth Flynn [**(@getchoo)**](https://github.com/getchoo)
- Stefan Boca [**(@stefanboca)**](https://github.com/stefanboca)
- tomberek [**(@tomberek)**](https://github.com/tomberek)
- Tristan Ross [**(@RossComputerGuy)**](https://github.com/RossComputerGuy)
- Valentin Gagarin [**(@fricklerhandwerk)**](https://github.com/fricklerhandwerk)
- Vladimír Čunát [**(@vcunat)**](https://github.com/vcunat)
- Wolfgang Walther [**(@wolfgangwalther)**](https://github.com/wolfgangwalther)
<!-- markdown links -->
[stack sampling evaluation profiler]: @docroot@/advanced-topics/eval-profiler.md
[`--eval-profiler`]: @docroot@/command-ref/conf-file.md#conf-eval-profiler
[`--eval-profiler flamegraph`]: @docroot@/command-ref/conf-file.md#conf-eval-profiler
[`--trace-function-calls`]: @docroot@/command-ref/conf-file.md#conf-trace-function-calls
[`--eval-profile-file`]: @docroot@/command-ref/conf-file.md#conf-eval-profile-file
[`--eval-profiler-frequency`]: @docroot@/command-ref/conf-file.md#conf-eval-profiler-frequency
[`build-dir`]: @docroot@/command-ref/conf-file.md#conf-build-dir
[`nix profile add`]: @docroot@/command-ref/new-cli/nix3-profile-add.md
[`nix repl`]: @docroot@/command-ref/new-cli/nix3-repl.md
[`nix flake archive`]: @docroot@/command-ref/new-cli/nix3-flake-archive.md
[`json-log-path`]: @docroot@/command-ref/conf-file.md#conf-json-log-path
[`trace-import-from-derivation`]: @docroot@/command-ref/conf-file.md#conf-trace-import-from-derivation
[`allow-import-from-derivation`]: @docroot@/command-ref/conf-file.md#conf-allow-import-from-derivation
[`builtins.sort`]: @docroot@/language/builtins.md#builtins-sort
[`flake = false`]: @docroot@/command-ref/new-cli/nix3-flake.md?highlight=false#flake-inputs
[`--no-check-sigs`]: @docroot@/command-ref/new-cli/nix3-flake-archive.md#opt-no-check-sigs
[commit `299141e`]: https://github.com/NixOS/nix/commit/299141ecbd08bae17013226dbeae71e842b4fdd7
@@ -138,6 +138,17 @@ See [Wikipedia](https://en.wikipedia.org/wiki/Argv) for details.
Environment variables which will be passed to the [builder](#builder) executable.
#### Structured Attributes {#structured-attrs}
Nix also has special support for embedding JSON in the derivations.
The environment variable `NIX_ATTRS_JSON_FILE` points to the exact location of that file both in a build and a [`nix-shell`](@docroot@/command-ref/nix-shell.md).
As a convenience to Bash builders, Nix writes a script that initialises shell variables corresponding to all attributes that are representable in Bash.
The environment variable `NIX_ATTRS_SH_FILE` points to the exact location of the script, both in a build and a [`nix-shell`](@docroot@/command-ref/nix-shell.md).
This includes non-nested (associative) arrays.
For example, the attribute `hardening.format = true` ends up as the Bash associative array element `${hardening[format]}`.
### Placeholders
Placeholders are opaque values used within the [process creation fields] to [store objects] for which we don't yet know [store path]s.
+138 -102
View File
@@ -1,6 +1,11 @@
{
# Core dependencies
pkgs ? import <nixpkgs> { },
lib ? pkgs.lib,
dockerTools ? pkgs.dockerTools,
runCommand ? pkgs.runCommand,
buildPackages ? pkgs.buildPackages,
# Image configuration
name ? "nix",
tag ? "latest",
bundleNixpkgs ? true,
@@ -14,81 +19,107 @@
gid ? 0,
uname ? "root",
gname ? "root",
Labels ? {
"org.opencontainers.image.title" = "Nix";
"org.opencontainers.image.source" = "https://github.com/NixOS/nix";
"org.opencontainers.image.vendor" = "Nix project";
"org.opencontainers.image.version" = nix.version;
"org.opencontainers.image.description" = "Nix container image";
},
Cmd ? [ (lib.getExe bashInteractive) ],
# Default Packages
nix ? pkgs.nix,
bashInteractive ? pkgs.bashInteractive,
coreutils-full ? pkgs.coreutils-full,
gnutar ? pkgs.gnutar,
gzip ? pkgs.gzip,
gnugrep ? pkgs.gnugrep,
which ? pkgs.which,
curl ? pkgs.curl,
less ? pkgs.less,
wget ? pkgs.wget,
man ? pkgs.man,
cacert ? pkgs.cacert,
findutils ? pkgs.findutils,
iana-etc ? pkgs.iana-etc,
gitMinimal ? pkgs.gitMinimal,
openssh ? pkgs.openssh,
# Other dependencies
shadow ? pkgs.shadow,
}:
let
defaultPkgs =
with pkgs;
[
nix
bashInteractive
coreutils-full
gnutar
gzip
gnugrep
which
curl
less
wget
man
cacert.out
findutils
iana-etc
git
openssh
]
++ extraPkgs;
defaultPkgs = [
nix
bashInteractive
coreutils-full
gnutar
gzip
gnugrep
which
curl
less
wget
man
cacert.out
findutils
iana-etc
gitMinimal
openssh
] ++ extraPkgs;
users = {
users =
{
root = {
uid = 0;
shell = "${pkgs.bashInteractive}/bin/bash";
home = "/root";
gid = 0;
groups = [ "root" ];
description = "System administrator";
};
nobody = {
uid = 65534;
shell = "${pkgs.shadow}/bin/nologin";
home = "/var/empty";
gid = 65534;
groups = [ "nobody" ];
description = "Unprivileged account (don't use!)";
};
}
// lib.optionalAttrs (uid != 0) {
"${uname}" = {
uid = uid;
shell = "${pkgs.bashInteractive}/bin/bash";
home = "/home/${uname}";
gid = gid;
groups = [ "${gname}" ];
description = "Nix user";
};
}
// lib.listToAttrs (
map (n: {
name = "nixbld${toString n}";
value = {
uid = 30000 + n;
gid = 30000;
groups = [ "nixbld" ];
description = "Nix build user ${toString n}";
root = {
uid = 0;
shell = lib.getExe bashInteractive;
home = "/root";
gid = 0;
groups = [ "root" ];
description = "System administrator";
};
}) (lib.lists.range 1 32)
);
groups = {
root.gid = 0;
nixbld.gid = 30000;
nobody.gid = 65534;
}
// lib.optionalAttrs (gid != 0) {
"${gname}".gid = gid;
};
nobody = {
uid = 65534;
shell = lib.getExe' shadow "nologin";
home = "/var/empty";
gid = 65534;
groups = [ "nobody" ];
description = "Unprivileged account (don't use!)";
};
}
// lib.optionalAttrs (uid != 0) {
"${uname}" = {
uid = uid;
shell = lib.getExe bashInteractive;
home = "/home/${uname}";
gid = gid;
groups = [ "${gname}" ];
description = "Nix user";
};
}
// lib.listToAttrs (
map (n: {
name = "nixbld${toString n}";
value = {
uid = 30000 + n;
gid = 30000;
groups = [ "nixbld" ];
description = "Nix build user ${toString n}";
};
}) (lib.lists.range 1 32)
);
groups =
{
root.gid = 0;
nixbld.gid = 30000;
nobody.gid = 65534;
}
// lib.optionalAttrs (gid != 0) {
"${gname}".gid = gid;
};
userToPasswd = (
k:
@@ -145,30 +176,29 @@ let
"${k}:x:${toString gid}:${lib.concatStringsSep "," members}";
groupContents = (lib.concatStringsSep "\n" (lib.attrValues (lib.mapAttrs groupToGroup groups)));
defaultNixConf = {
sandbox = "false";
build-users-group = "nixbld";
trusted-public-keys = [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" ];
};
toConf =
with pkgs.lib.generators;
toKeyValue {
mkKeyValue = mkKeyValueDefault {
mkValueString = v: if lib.isList v then lib.concatStringsSep " " v else mkValueStringDefault { } v;
} " = ";
};
nixConfContents =
(lib.concatStringsSep "\n" (
lib.mapAttrsToList (
n: v:
let
vStr = if builtins.isList v then lib.concatStringsSep " " v else v;
in
"${n} = ${vStr}"
) (defaultNixConf // nixConf)
))
+ "\n";
nixConfContents = toConf (
{
sandbox = false;
build-users-group = "nixbld";
trusted-public-keys = [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" ];
}
// nixConf
);
userHome = if uid == 0 then "/root" else "/home/${uname}";
baseSystem =
let
nixpkgs = pkgs.path;
channel = pkgs.runCommand "channel-nixos" { inherit bundleNixpkgs; } ''
channel = runCommand "channel-nixos" { inherit bundleNixpkgs; } ''
mkdir $out
if [ "$bundleNixpkgs" ]; then
ln -s ${
@@ -180,11 +210,8 @@ let
echo "[]" > $out/manifest.nix
fi
'';
rootEnv = pkgs.buildPackages.buildEnv {
name = "root-profile-env";
paths = defaultPkgs;
};
manifest = pkgs.buildPackages.runCommand "manifest.nix" { } ''
# doc/manual/source/command-ref/files/manifest.nix.md
manifest = buildPackages.runCommand "manifest.nix" { } ''
cat > $out <<EOF
[
${lib.concatStringsSep "\n" (
@@ -213,11 +240,15 @@ let
]
EOF
'';
profile = pkgs.buildPackages.runCommand "user-environment" { } ''
mkdir $out
cp -a ${rootEnv}/* $out/
ln -s ${manifest} $out/manifest.nix
'';
profile = buildPackages.buildEnv {
name = "root-profile-env";
paths = defaultPkgs;
postBuild = ''
mv $out/manifest $out/manifest.nix
'';
inherit manifest;
};
flake-registry-path =
if (flake-registry == null) then
null
@@ -226,7 +257,7 @@ let
else
flake-registry;
in
pkgs.runCommand "base-system"
runCommand "base-system"
{
inherit
passwdContents
@@ -249,6 +280,7 @@ let
set -x
mkdir -p $out/etc
# may get replaced by pkgs.dockerTools.caCertificates
mkdir -p $out/etc/ssl/certs
ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt $out/etc/ssl/certs
@@ -276,19 +308,23 @@ let
mkdir -p $out${userHome}
mkdir -p $out/nix/var/nix/profiles/per-user/${uname}
# see doc/manual/source/command-ref/files/profiles.md
ln -s ${profile} $out/nix/var/nix/profiles/default-1-link
ln -s /nix/var/nix/profiles/default-1-link $out/nix/var/nix/profiles/default
# see doc/manual/source/command-ref/files/channels.md
ln -s ${channel} $out/nix/var/nix/profiles/per-user/${uname}/channels-1-link
ln -s /nix/var/nix/profiles/per-user/${uname}/channels-1-link $out/nix/var/nix/profiles/per-user/${uname}/channels
# see doc/manual/source/command-ref/files/default-nix-expression.md
mkdir -p $out${userHome}/.nix-defexpr
ln -s /nix/var/nix/profiles/per-user/${uname}/channels $out${userHome}/.nix-defexpr/channels
echo "${channelURL} ${channelName}" > $out${userHome}/.nix-channels
# may get replaced by pkgs.dockerTools.binSh & pkgs.dockerTools.usrBinEnv
mkdir -p $out/bin $out/usr/bin
ln -s ${pkgs.coreutils}/bin/env $out/usr/bin/env
ln -s ${pkgs.bashInteractive}/bin/bash $out/bin/sh
ln -s ${lib.getExe' coreutils-full "env"} $out/usr/bin/env
ln -s ${lib.getExe bashInteractive} $out/bin/sh
''
+ (lib.optionalString (flake-registry-path != null) ''
@@ -297,13 +333,13 @@ let
globalFlakeRegistryPath="$nixCacheDir/flake-registry.json"
ln -s ${flake-registry-path} $out$globalFlakeRegistryPath
mkdir -p $out/nix/var/nix/gcroots/auto
rootName=$(${pkgs.nix}/bin/nix --extra-experimental-features nix-command hash file --type sha1 --base32 <(echo -n $globalFlakeRegistryPath))
rootName=$(${lib.getExe' nix "nix"} --extra-experimental-features nix-command hash file --type sha1 --base32 <(echo -n $globalFlakeRegistryPath))
ln -s $globalFlakeRegistryPath $out/nix/var/nix/gcroots/auto/$rootName
'')
);
in
pkgs.dockerTools.buildLayeredImageWithNixDb {
dockerTools.buildLayeredImageWithNixDb {
inherit
name
@@ -329,7 +365,7 @@ pkgs.dockerTools.buildLayeredImageWithNixDb {
'';
config = {
Cmd = [ (lib.getExe pkgs.bashInteractive) ];
inherit Cmd Labels;
User = "${toString uid}:${toString gid}";
Env = [
"USER=${uname}"
Generated
+4 -4
View File
@@ -63,16 +63,16 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1756178832,
"narHash": "sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8=",
"lastModified": 1747179050,
"narHash": "sha256-qhFMmDkeJX9KJwr5H32f1r7Prs7XbQWtO0h3V0a0rFY=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "d98ce345cdab58477ca61855540999c86577d19d",
"rev": "adaa24fbf46737f3f1b5497bf64bae750f82942e",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-25.05-small",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
+21 -39
View File
@@ -1,7 +1,7 @@
{
description = "The purely functional package manager";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05-small";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.nixpkgs-23-11.url = "github:NixOS/nixpkgs/a62e6edd6d5e1fa0329b8653c801147986f8d446";
@@ -32,7 +32,7 @@
let
inherit (nixpkgs) lib;
officialRelease = true;
officialRelease = false;
linux32BitSystems = [ "i686-linux" ];
linux64BitSystems = [
@@ -179,19 +179,6 @@
};
nix = final.nixComponents2.nix-cli;
# See https://github.com/NixOS/nixpkgs/pull/214409
# Remove when fixed in this flake's nixpkgs
pre-commit =
if prev.stdenv.hostPlatform.system == "i686-linux" then
(prev.pre-commit.override (o: {
dotnet-sdk = "";
})).overridePythonAttrs
(o: {
doCheck = false;
})
else
prev.pre-commit;
};
in
@@ -233,28 +220,24 @@
This shouldn't build anything significant; just check that things
(including derivations) are _set up_ correctly.
*/
# Disabled due to a bug in `testEqualContents` (see
# https://github.com/NixOS/nix/issues/12690).
/*
packaging-overriding =
let
pkgs = nixpkgsFor.${system}.native;
nix = self.packages.${system}.nix;
in
assert (nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src.patches == [ pkgs.emptyFile ];
if pkgs.stdenv.buildPlatform.isDarwin then
lib.warn "packaging-overriding check currently disabled because of a permissions issue on macOS" pkgs.emptyFile
else
# If this fails, something might be wrong with how we've wired the scope,
# or something could be broken in Nixpkgs.
pkgs.testers.testEqualContents {
assertion = "trivial patch does not change source contents";
expected = "${./.}";
actual =
# Same for all components; nix-util is an arbitrary pick
(nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src;
};
*/
packaging-overriding =
let
pkgs = nixpkgsFor.${system}.native;
nix = self.packages.${system}.nix;
in
assert (nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src.patches == [ pkgs.emptyFile ];
if pkgs.stdenv.buildPlatform.isDarwin then
lib.warn "packaging-overriding check currently disabled because of a permissions issue on macOS" pkgs.emptyFile
else
# If this fails, something might be wrong with how we've wired the scope,
# or something could be broken in Nixpkgs.
pkgs.testers.testEqualContents {
assertion = "trivial patch does not change source contents";
expected = "${./.}";
actual =
# Same for all components; nix-util is an arbitrary pick
(nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src;
};
}
// (lib.optionalAttrs (builtins.elem system linux64BitSystems)) {
dockerImage = self.hydraJobs.dockerImage.${system};
@@ -421,8 +404,7 @@
dockerImage =
let
pkgs = nixpkgsFor.${system}.native;
image = import ./docker.nix {
inherit pkgs;
image = pkgs.callPackage ./docker.nix {
tag = pkgs.nix.version;
};
in
@@ -166,5 +166,24 @@
"the-tumultuous-unicorn-of-darkness@gmx.com": "TheTumultuousUnicornOfDarkness",
"dev@rodney.id.au": "rvl",
"pe@pijul.org": "P-E-Meunier",
"yannik@floxdev.com": "ysndr"
"yannik@floxdev.com": "ysndr",
"73017521+egorkonovalov@users.noreply.github.com": "egorkonovalov",
"raito@lix.systems": null,
"nikita.nikita.krasnov@gmail.com": "synalice",
"lucperkins@gmail.com": "lucperkins",
"vladimir.cunat@nic.cz": "vcunat",
"walther@technowledgy.de": "wolfgangwalther",
"jayesh.mail@gmail.com": "jayeshv",
"samuli.thomasson@pm.me": "SimSaladin",
"kevin@stravers.net": "kstrafe",
"poperigby@mailbox.org": "poperigby",
"cole.helbling@determinate.systems": "cole-h",
"donottellmetonottellyou@gmail.com": "donottellmetonottellyou",
"getchoo@tuta.io": "getchoo",
"alex.ford@determinate.systems": "gustavderdrache",
"stefan.r.boca@gmail.com": "stefanboca",
"gwenn.lebihan7@gmail.com": "gwennlbh",
"hey@ewen.works": "gwennlbh",
"matt@sturgeon.me.uk": "MattSturgeon",
"pbsds@hotmail.com": "pbsds"
}
@@ -146,5 +146,21 @@
"ajlekcahdp4": "Alexander Romanov",
"Valodim": "Vincent Breitmoser",
"rvl": "Rodney Lorrimar",
"whatsthecraic": "Dean De Leo"
"whatsthecraic": "Dean De Leo",
"gwennlbh": "Gwenn Le Bihan",
"donottellmetonottellyou": "Jade Masker",
"kstrafe": null,
"synalice": "Nikita Krasnov",
"poperigby": "PopeRigby",
"MattSturgeon": "Matt Sturgeon",
"lucperkins": "Luc Perkins",
"gustavderdrache": null,
"SimSaladin": "Samuli Thomasson",
"getchoo": "Seth Flynn",
"stefanboca": "Stefan Boca",
"wolfgangwalther": "Wolfgang Walther",
"pbsds": "Peder Bergebakken Sundt",
"egorkonovalov": "Egor Konovalov",
"jayeshv": "jayeshv",
"vcunat": "Vladim\u00edr \u010cun\u00e1t"
}
+112
View File
@@ -37,6 +37,118 @@
fi
''}";
};
meson-format = {
enable = true;
files = "(meson.build|meson.options)$";
entry = "${pkgs.writeScript "format-meson" ''
#!${pkgs.runtimeShell}
for file in "$@"; do
${lib.getExe pkgs.meson} format -ic ${../meson.format} "$file"
done
''}";
excludes = [
# We haven't applied formatting to these files yet
''^doc/manual/meson.build$''
''^doc/manual/source/command-ref/meson.build$''
''^doc/manual/source/development/meson.build$''
''^doc/manual/source/language/meson.build$''
''^doc/manual/source/meson.build$''
''^doc/manual/source/release-notes/meson.build$''
''^doc/manual/source/store/meson.build$''
''^misc/bash/meson.build$''
''^misc/fish/meson.build$''
''^misc/launchd/meson.build$''
''^misc/meson.build$''
''^misc/systemd/meson.build$''
''^misc/zsh/meson.build$''
''^nix-meson-build-support/$''
''^nix-meson-build-support/big-objs/meson.build$''
''^nix-meson-build-support/common/meson.build$''
''^nix-meson-build-support/deps-lists/meson.build$''
''^nix-meson-build-support/export/meson.build$''
''^nix-meson-build-support/export-all-symbols/meson.build$''
''^nix-meson-build-support/generate-header/meson.build$''
''^nix-meson-build-support/libatomic/meson.build$''
''^nix-meson-build-support/subprojects/meson.build$''
''^scripts/meson.build$''
''^src/external-api-docs/meson.build$''
''^src/internal-api-docs/meson.build$''
''^src/libcmd/include/nix/cmd/meson.build$''
''^src/libcmd/meson.build$''
''^src/libcmd/nix-meson-build-support$''
''^src/libexpr/include/nix/expr/meson.build$''
''^src/libexpr/meson.build$''
''^src/libexpr/nix-meson-build-support$''
''^src/libexpr-c/meson.build$''
''^src/libexpr-c/nix-meson-build-support$''
''^src/libexpr-test-support/meson.build$''
''^src/libexpr-test-support/nix-meson-build-support$''
''^src/libexpr-tests/meson.build$''
''^src/libexpr-tests/nix-meson-build-support$''
''^src/libfetchers/include/nix/fetchers/meson.build$''
''^src/libfetchers/meson.build$''
''^src/libfetchers/nix-meson-build-support$''
''^src/libfetchers-c/meson.build$''
''^src/libfetchers-c/nix-meson-build-support$''
''^src/libfetchers-tests/meson.build$''
''^src/libfetchers-tests/nix-meson-build-support$''
''^src/libflake/include/nix/flake/meson.build$''
''^src/libflake/meson.build$''
''^src/libflake/nix-meson-build-support$''
''^src/libflake-c/meson.build$''
''^src/libflake-c/nix-meson-build-support$''
''^src/libflake-tests/meson.build$''
''^src/libflake-tests/nix-meson-build-support$''
''^src/libmain/include/nix/main/meson.build$''
''^src/libmain/meson.build$''
''^src/libmain/nix-meson-build-support$''
''^src/libmain-c/meson.build$''
''^src/libmain-c/nix-meson-build-support$''
''^src/libstore/include/nix/store/meson.build$''
''^src/libstore/meson.build$''
''^src/libstore/nix-meson-build-support$''
''^src/libstore/unix/include/nix/store/meson.build$''
''^src/libstore/unix/meson.build$''
''^src/libstore/windows/meson.build$''
''^src/libstore-c/meson.build$''
''^src/libstore-c/nix-meson-build-support$''
''^src/libstore-test-support/include/nix/store/tests/meson.build$''
''^src/libstore-test-support/meson.build$''
''^src/libstore-test-support/nix-meson-build-support$''
''^src/libstore-tests/meson.build$''
''^src/libstore-tests/nix-meson-build-support$''
''^src/libutil/meson.build$''
''^src/libutil/nix-meson-build-support$''
''^src/libutil/unix/include/nix/util/meson.build$''
''^src/libutil/unix/meson.build$''
''^src/libutil/windows/meson.build$''
''^src/libutil-c/meson.build$''
''^src/libutil-c/nix-meson-build-support$''
''^src/libutil-test-support/include/nix/util/tests/meson.build$''
''^src/libutil-test-support/meson.build$''
''^src/libutil-test-support/nix-meson-build-support$''
''^src/libutil-tests/meson.build$''
''^src/libutil-tests/nix-meson-build-support$''
''^src/nix/meson.build$''
''^src/nix/nix-meson-build-support$''
''^src/perl/lib/Nix/meson.build$''
''^src/perl/meson.build$''
''^tests/functional/ca/meson.build$''
''^tests/functional/common/meson.build$''
''^tests/functional/dyn-drv/meson.build$''
''^tests/functional/flakes/meson.build$''
''^tests/functional/git-hashing/meson.build$''
''^tests/functional/local-overlay-store/meson.build$''
''^tests/functional/meson.build$''
''^src/libcmd/meson.options$''
''^src/libexpr/meson.options$''
''^src/libstore/meson.options$''
''^src/libutil/meson.options$''
''^src/libutil-c/meson.options$''
''^src/nix/meson.options$''
''^src/perl/meson.options$''
];
};
nixfmt-rfc-style = {
enable = true;
excludes = [
+1 -1
View File
@@ -157,7 +157,7 @@ section_title="Release $version_full ($DATE)"
if ! $IS_PATCH; then
echo
echo "# Contributors"
echo "## Contributors"
echo
VERSION=$version_full ./maintainers/release-credits
fi
-4
View File
@@ -39,10 +39,6 @@ release:
* Proof-read / edit / rearrange the release notes if needed. Breaking changes
and highlights should go to the top.
* Run `maintainers/release-credits` to make sure the credits script works
and produces a sensible output. Some emails might not automatically map to
a GitHub handle.
* Push.
```console
+5 -2
View File
@@ -1,13 +1,16 @@
# This is just a stub project to include all the others as subprojects
# for development shell purposes
project('nix-dev-shell', 'cpp',
project(
'nix-dev-shell',
'cpp',
version : files('.version'),
subproject_dir : 'src',
default_options : [
'localstatedir=/nix/var',
# hack for trailing newline
],
meson_version : '>= 1.1'
meson_version : '>= 1.1',
)
# Internal Libraries
+7
View File
@@ -0,0 +1,7 @@
indent_by = ' '
space_array = true
kwargs_force_multiline = false
wide_colon = true
group_arg_value = true
indent_before_comments = ' '
use_editor_config = true
+12 -3
View File
@@ -1,13 +1,22 @@
# vim: filetype=meson
option('doc-gen', type : 'boolean', value : false,
option(
'doc-gen',
type : 'boolean',
value : false,
description : 'Generate documentation',
)
option('unit-tests', type : 'boolean', value : true,
option(
'unit-tests',
type : 'boolean',
value : true,
description : 'Build unit tests',
)
option('bindings', type : 'boolean', value : true,
option(
'bindings',
type : 'boolean',
value : true,
description : 'Build language bindings (e.g. Perl)',
)
+2 -1
View File
@@ -1 +1,2 @@
d @localstatedir@/nix/daemon-socket 0755 root root - -
d @localstatedir@/nix/daemon-socket 0755 root root - -
d @localstatedir@/nix/builds 0755 root root 7d -
@@ -1,19 +0,0 @@
# This attempts to translate meson cpu_family and cpu_name specified via
# --cross-file [1] into a nix *system double*. Nixpkgs mostly respects ([2]) the
# conventions outlined in [1].
#
# [1]: https://mesonbuild.com/Reference-tables.html#cpu-families
# [2]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/lib/meson.nix
nix_system_cpu = {'ppc64' : 'powerpc64', 'ppc' : 'powerpc', 'x86' : 'i686'}.get(
host_machine.cpu_family(),
host_machine.cpu_family(),
)
if (host_machine.cpu_family() in [ 'ppc64', 'ppc' ]) and host_machine.endian() == 'little'
nix_system_cpu += 'le'
elif host_machine.cpu_family() in [ 'mips64', 'mips' ] and host_machine.endian() == 'little'
nix_system_cpu += 'el'
elif host_machine.cpu_family() == 'arm'
nix_system_cpu = host_machine.cpu()
endif
@@ -2,5 +2,5 @@ if host_machine.system() == 'windows'
# https://learn.microsoft.com/en-us/cpp/porting/modifying-winver-and-win32-winnt?view=msvc-170
# #define _WIN32_WINNT_WIN8 0x0602
# We currently don't use any API which requires higher than this.
add_project_arguments([ '-D_WIN32_WINNT=0x0602' ], language: 'cpp')
add_project_arguments([ '-D_WIN32_WINNT=0x0602' ], language : 'cpp')
endif
+7 -8
View File
@@ -54,12 +54,12 @@ let
preConfigure =
prevAttrs.preConfigure or ""
+
# Update the repo-global .version file.
# Symlink ./.version points there, but by default only workDir is writable.
''
chmod u+w ./.version
echo ${finalAttrs.version} > ./.version
'';
# Update the repo-global .version file.
# Symlink ./.version points there, but by default only workDir is writable.
''
chmod u+w ./.version
echo ${finalAttrs.version} > ./.version
'';
};
localSourceLayer =
@@ -148,8 +148,7 @@ let
nativeBuildInputs = [
meson
ninja
]
++ prevAttrs.nativeBuildInputs or [ ];
] ++ prevAttrs.nativeBuildInputs or [ ];
mesonCheckFlags = prevAttrs.mesonCheckFlags or [ ] ++ [
"--print-errorlogs"
];
+11 -10
View File
@@ -71,16 +71,17 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
# We use this shell with the local checkout, not unpackPhase.
src = null;
env = {
# For `make format`, to work without installing pre-commit
_NIX_PRE_COMMIT_HOOKS_CONFIG = "${(pkgs.formats.yaml { }).generate "pre-commit-config.yaml"
modular.pre-commit.settings.rawConfig
}";
}
// lib.optionalAttrs stdenv.hostPlatform.isLinux {
CC_LD = "mold";
CXX_LD = "mold";
};
env =
{
# For `make format`, to work without installing pre-commit
_NIX_PRE_COMMIT_HOOKS_CONFIG = "${(pkgs.formats.yaml { }).generate "pre-commit-config.yaml"
modular.pre-commit.settings.rawConfig
}";
}
// lib.optionalAttrs stdenv.hostPlatform.isLinux {
CC_LD = "mold";
CXX_LD = "mold";
};
mesonFlags =
map (transformFlag "libutil") (ignoreCrossFile pkgs.nixComponents2.nix-util.mesonFlags)
+32 -31
View File
@@ -47,25 +47,25 @@
}:
let
libs = {
inherit
nix-util
nix-util-c
nix-store
nix-store-c
nix-fetchers
nix-fetchers-c
nix-expr
nix-expr-c
nix-flake
nix-flake-c
nix-main
nix-main-c
nix-cmd
;
}
//
lib.optionalAttrs
libs =
{
inherit
nix-util
nix-util-c
nix-store
nix-store-c
nix-fetchers
nix-fetchers-c
nix-expr
nix-expr-c
nix-flake
nix-flake-c
nix-main
nix-main-c
nix-cmd
;
}
// lib.optionalAttrs
(!stdenv.hostPlatform.isStatic && stdenv.buildPlatform.canExecute stdenv.hostPlatform)
{
# Currently fails in static build
@@ -127,19 +127,20 @@ stdenv.mkDerivation (finalAttrs: {
*/
dontFixup = true;
checkInputs = [
# Make sure the unit tests have passed
nix-util-tests.tests.run
nix-store-tests.tests.run
nix-expr-tests.tests.run
nix-fetchers-tests.tests.run
nix-flake-tests.tests.run
checkInputs =
[
# Make sure the unit tests have passed
nix-util-tests.tests.run
nix-store-tests.tests.run
nix-expr-tests.tests.run
nix-fetchers-tests.tests.run
nix-flake-tests.tests.run
# Make sure the functional tests have passed
nix-functional-tests
]
++
lib.optionals (!stdenv.hostPlatform.isStatic && stdenv.buildPlatform.canExecute stdenv.hostPlatform)
# Make sure the functional tests have passed
nix-functional-tests
]
++ lib.optionals
(!stdenv.hostPlatform.isStatic && stdenv.buildPlatform.canExecute stdenv.hostPlatform)
[
# Perl currently fails in static build
# TODO: Split out tests into a separate derivation?
-63
View File
@@ -1,63 +0,0 @@
#include <algorithm>
#include <array>
#include <atomic>
#include <cassert>
#include <cctype>
#include <chrono>
#include <climits>
#include <cmath>
#include <condition_variable>
#include <cstddef>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <exception>
#include <functional>
#include <future>
#include <iostream>
#include <limits>
#include <list>
#include <locale>
#include <map>
#include <memory>
#include <mutex>
#include <numeric>
#include <optional>
#include <queue>
#include <random>
#include <regex>
#include <set>
#include <sstream>
#include <stack>
#include <stdexcept>
#include <string>
#include <thread>
#include <unordered_map>
#include <unordered_set>
#include <vector>
#include <boost/format.hpp>
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <signal.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>
#ifndef _WIN32
# include <grp.h>
# include <netdb.h>
# include <pwd.h>
# include <sys/resource.h>
# include <sys/select.h>
# include <sys/socket.h>
# include <sys/utsname.h>
# include <sys/wait.h>
# include <termios.h>
#endif
#include <nlohmann/json.hpp>
+7 -2
View File
@@ -834,8 +834,13 @@ install_from_extracted_nix() {
(
cd "$EXTRACTED_NIX_PATH"
_sudo "to copy the basic Nix files to the new store at $NIX_ROOT/store" \
cp -RPp ./store/* "$NIX_ROOT/store/"
if is_os_darwin; then
_sudo "to copy the basic Nix files to the new store at $NIX_ROOT/store" \
cp -RPp ./store/* "$NIX_ROOT/store/"
else
_sudo "to copy the basic Nix files to the new store at $NIX_ROOT/store" \
cp -RP --preserve=ownership,timestamps ./store/* "$NIX_ROOT/store/"
fi
_sudo "to make the new store non-writable at $NIX_ROOT/store" \
chmod -R ugo-w "$NIX_ROOT/store/"
+5 -1
View File
@@ -167,7 +167,11 @@ for i in $(cd "$self/store" >/dev/null && echo ./*); do
rm -rf "$i_tmp"
fi
if ! [ -e "$dest/store/$i" ]; then
cp -RPp "$self/store/$i" "$i_tmp"
if [ "$(uname -s)" = "Darwin" ]; then
cp -RPp "$self/store/$i" "$i_tmp"
else
cp -RP --preserve=ownership,timestamps "$self/store/$i" "$i_tmp"
fi
chmod -R a-w "$i_tmp"
chmod +w "$i_tmp"
mv "$i_tmp" "$dest/store/$i"
+9 -1
View File
@@ -323,9 +323,17 @@ static int main_build_remote(int argc, char ** argv)
drv.inputSrcs = store->parseStorePathSet(inputs);
optResult = sshStore->buildDerivation(*drvPath, (const BasicDerivation &) drv);
auto & result = *optResult;
if (!result.success())
if (!result.success()) {
if (settings.keepFailed) {
warn(
"The failed build directory was kept on the remote builder due to `--keep-failed`.%s",
(settings.thisSystem == drv.platform || settings.extraPlatforms.get().count(drv.platform) > 0)
? " You can re-run the command with `--builders ''` to disable remote building for this invocation."
: "");
}
throw Error(
"build of '%s' on '%s' failed: %s", store->printStorePath(*drvPath), storeUri, result.errorMsg);
}
} else {
copyClosure(*store, *sshStore, StorePathSet{*drvPath}, NoRepair, NoCheckSigs, substitute);
auto res = sshStore->buildPathsWithResults({DerivedPath::Built{
+9 -5
View File
@@ -33,8 +33,12 @@ EvalSettings evalSettings{
auto flakeRef = parseFlakeRef(fetchSettings, std::string{rest}, {}, true, false);
debug("fetching flake search path element '%s''", rest);
auto [accessor, lockedRef] = flakeRef.resolve(state.store).lazyFetch(state.store);
auto storePath =
nix::fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName());
auto storePath = nix::fetchToStore(
state.fetchSettings,
*state.store,
SourcePath(accessor),
FetchMode::Copy,
lockedRef.input.getName());
state.allowPath(storePath);
return state.storePath(storePath);
},
@@ -175,7 +179,7 @@ SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * bas
{
if (EvalSettings::isPseudoUrl(s)) {
auto accessor = fetchers::downloadTarball(state.store, state.fetchSettings, EvalSettings::resolvePseudoUrl(s));
auto storePath = fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy);
auto storePath = fetchToStore(state.fetchSettings, *state.store, SourcePath(accessor), FetchMode::Copy);
return state.storePath(storePath);
}
@@ -183,8 +187,8 @@ SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * bas
experimentalFeatureSettings.require(Xp::Flakes);
auto flakeRef = parseFlakeRef(fetchSettings, std::string(s.substr(6)), {}, true, false);
auto [accessor, lockedRef] = flakeRef.resolve(state.store).lazyFetch(state.store);
auto storePath =
nix::fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName());
auto storePath = nix::fetchToStore(
state.fetchSettings, *state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName());
state.allowPath(storePath);
return state.storePath(storePath);
}
+1 -1
View File
@@ -336,7 +336,7 @@ struct MixEnvironment : virtual Args
StringSet keepVars;
StringSet unsetVars;
std::map<std::string, std::string> setVars;
StringMap setVars;
bool ignoreEnvironment;
MixEnvironment();
@@ -5,6 +5,7 @@
#include "nix/util/canon-path.hh"
#include "nix/main/common-args.hh"
#include "nix/expr/search-path.hh"
#include "nix/expr/eval-settings.hh"
#include <filesystem>
@@ -17,27 +18,25 @@ struct Settings;
}
class EvalState;
struct EvalSettings;
struct CompatibilitySettings;
class Bindings;
struct SourcePath;
namespace flake {
struct Settings;
}
/**
* @todo Get rid of global setttings variables
* @todo Get rid of global settings variables
*/
extern fetchers::Settings fetchSettings;
/**
* @todo Get rid of global setttings variables
* @todo Get rid of global settings variables
*/
extern EvalSettings evalSettings;
/**
* @todo Get rid of global setttings variables
* @todo Get rid of global settings variables
*/
extern flake::Settings flakeSettings;
+1 -1
View File
@@ -43,7 +43,7 @@ std::optional<DerivedPathWithInfo>
InstallableValue::trySinglePathToDerivedPaths(Value & v, const PosIdx pos, std::string_view errorCtx)
{
if (v.type() == nPath) {
auto storePath = fetchToStore(*state->store, v.path(), FetchMode::Copy);
auto storePath = fetchToStore(state->fetchSettings, *state->store, v.path(), FetchMode::Copy);
return {{
.path =
DerivedPath::Opaque{
+2 -2
View File
@@ -198,7 +198,7 @@ SourceExprCommand::SourceExprCommand()
.shortName = 'f',
.description =
"Interpret [*installables*](@docroot@/command-ref/new-cli/nix.md#installables) as attribute paths relative to the Nix expression stored in *file*. "
"If *file* is the character -, then a Nix expression will be read from standard input. "
"If *file* is the character -, then a Nix expression is read from standard input. "
"Implies `--impure`.",
.category = installablesCategory,
.labels = {"file"},
@@ -818,7 +818,7 @@ RawInstallablesCommand::RawInstallablesCommand()
void RawInstallablesCommand::applyDefaultInstallables(std::vector<std::string> & rawInstallables)
{
if (rawInstallables.empty()) {
// FIXME: commands like "nix profile install" should not have a
// FIXME: commands like "nix profile add" should not have a
// default, probably.
rawInstallables.push_back(".");
}
+1 -2
View File
@@ -53,8 +53,7 @@ mkMesonLibrary (finalAttrs: {
buildInputs = [
({ inherit editline readline; }.${readlineFlavor})
]
++ lib.optional enableMarkdown lowdown;
] ++ lib.optional enableMarkdown lowdown;
propagatedBuildInputs = [
nix-util
+2
View File
@@ -2,6 +2,8 @@
#include <cstdio>
#include <signal.h>
#if USE_READLINE
# include <readline/history.h>
# include <readline/readline.h>
+55 -13
View File
@@ -66,6 +66,7 @@ struct NixRepl : AbstractNixRepl, detail::ReplCompleterMixin, gc
const static int envSize = 32768;
std::shared_ptr<StaticEnv> staticEnv;
Value lastLoaded;
Env * env;
int displ;
StringSet varNames;
@@ -96,6 +97,7 @@ struct NixRepl : AbstractNixRepl, detail::ReplCompleterMixin, gc
void loadFiles();
void loadFlakes();
void reloadFilesAndFlakes();
void showLastLoaded();
void addAttrsToScope(Value & attrs);
void addVarToScope(const Symbol name, Value & v);
Expr * parseString(std::string s);
@@ -165,6 +167,8 @@ static std::ostream & showDebugTrace(std::ostream & out, const PosTable & positi
return out;
}
MakeError(IncompleteReplExpr, ParseError);
static bool isFirstRepl = true;
ReplExitStatus NixRepl::mainLoop()
@@ -213,16 +217,8 @@ ReplExitStatus NixRepl::mainLoop()
default:
unreachable();
}
} catch (ParseError & e) {
if (e.msg().find("unexpected end of file") != std::string::npos) {
// For parse errors on incomplete input, we continue waiting for the next line of
// input without clearing the input so far.
continue;
} else {
printMsg(lvlError, e.msg());
}
} catch (EvalError & e) {
printMsg(lvlError, e.msg());
} catch (IncompleteReplExpr &) {
continue;
} catch (Error & e) {
printMsg(lvlError, e.msg());
} catch (Interrupted & e) {
@@ -307,7 +303,7 @@ StringSet NixRepl::completePrefix(const std::string & prefix)
} catch (BadURL & e) {
// Quietly ignore BadURL flake-related errors.
} catch (FileNotFound & e) {
// Quietly ignore non-existent file beeing `import`-ed.
// Quietly ignore non-existent file being `import`-ed.
}
}
@@ -390,6 +386,7 @@ ProcessLineResult NixRepl::processLine(std::string line)
<< " current profile\n"
<< " :l, :load <path> Load Nix expression and add it to scope\n"
<< " :lf, :load-flake <ref> Load Nix flake and add it to scope\n"
<< " :ll, :last-loaded Show most recently loaded variables added to scope\n"
<< " :p, :print <expr> Evaluate and print expression recursively\n"
<< " Strings are printed directly, without escaping.\n"
<< " :q, :quit Exit nix-repl\n"
@@ -478,6 +475,10 @@ ProcessLineResult NixRepl::processLine(std::string line)
loadFlake(arg);
}
else if (command == ":ll" || command == ":last-loaded") {
showLastLoaded();
}
else if (command == ":r" || command == ":reload") {
state->resetFileCache();
reloadFilesAndFlakes();
@@ -493,7 +494,7 @@ ProcessLineResult NixRepl::processLine(std::string line)
auto path = state->coerceToPath(noPos, v, context, "while evaluating the filename to edit");
return {path, 0};
} else if (v.isLambda()) {
auto pos = state->positions[v.payload.lambda.fun->pos];
auto pos = state->positions[v.lambda().fun->pos];
if (auto path = std::get_if<SourcePath>(&pos.origin))
return {*path, pos.line};
else
@@ -768,6 +769,16 @@ void NixRepl::initEnv()
varNames.emplace(state->symbols[i.first]);
}
void NixRepl::showLastLoaded()
{
RunPager pager;
for (auto & i : *lastLoaded.attrs()) {
std::string_view name = state->symbols[i.name];
logger->cout(name);
}
}
void NixRepl::reloadFilesAndFlakes()
{
initEnv();
@@ -820,6 +831,27 @@ void NixRepl::addAttrsToScope(Value & attrs)
staticEnv->sort();
staticEnv->deduplicate();
notice("Added %1% variables.", attrs.attrs()->size());
lastLoaded = attrs;
const int max_print = 20;
int counter = 0;
std::ostringstream loaded;
for (auto & i : attrs.attrs()->lexicographicOrder(state->symbols)) {
if (counter >= max_print)
break;
if (counter > 0)
loaded << ", ";
printIdentifier(loaded, state->symbols[i->name]);
counter += 1;
}
notice("%1%", loaded.str());
if (attrs.attrs()->size() > max_print)
notice("... and %1% more; view with :ll", attrs.attrs()->size() - max_print);
}
void NixRepl::addVarToScope(const Symbol name, Value & v)
@@ -841,7 +873,17 @@ Expr * NixRepl::parseString(std::string s)
void NixRepl::evalString(std::string s, Value & v)
{
Expr * e = parseString(s);
Expr * e;
try {
e = parseString(s);
} catch (ParseError & e) {
if (e.msg().find("unexpected end of file") != std::string::npos)
// For parse errors on incomplete input, we continue waiting for the next line of
// input without clearing the input so far.
throw IncompleteReplExpr(e.msg());
else
throw;
}
e->eval(*state, *env, v);
state->forceValue(v, v.determinePos(noPos));
}
+2 -2
View File
@@ -252,7 +252,7 @@ const char * nix_get_path_string(nix_c_context * context, const nix_value * valu
// We could use v.path().to_string().c_str(), but I'm concerned this
// crashes. Looks like .path() allocates a CanonPath with a copy of the
// string, then it gets the underlying data from that.
return v.payload.path.path;
return v.pathStr();
}
NIXC_CATCH_ERRS_NULL
}
@@ -324,7 +324,7 @@ nix_value * nix_get_list_byidx(nix_c_context * context, const nix_value * value,
try {
auto & v = check_value_in(value);
assert(v.type() == nix::nList);
auto * p = v.listElems()[ix];
auto * p = v.listView()[ix];
nix_gc_incref(nullptr, p);
if (p != nullptr)
state->state.forceValue(*p, nix::noPos);
@@ -1,9 +1,10 @@
# Public headers directory
include_dirs = [include_directories('../../..')]
include_dirs = [ include_directories('../../..') ]
headers = files(
'libexpr.hh',
'nix_api_expr.hh',
'value/context.hh',
# hack for trailing newline
)
+5 -5
View File
@@ -392,7 +392,7 @@ TEST_F(ErrorTraceTest, filterSource)
HintFmt("expected a function but found %s: %s", "a list", Uncolored("[ ]")),
HintFmt("while evaluating the first argument passed to builtins.filterSource"));
// Usupported by store "dummy"
// Unsupported by store "dummy"
// ASSERT_TRACE2("filterSource (_: 1) ./.",
// TypeError,
@@ -574,7 +574,7 @@ TEST_F(ErrorTraceTest, mapAttrs)
HintFmt("expected a set but found %s: %s", "a list", Uncolored("[ ]")),
HintFmt("while evaluating the second argument passed to builtins.mapAttrs"));
// XXX: defered
// XXX: deferred
// ASSERT_TRACE2("mapAttrs \"\" { foo.bar = 1; }",
// TypeError,
// HintFmt("attempt to call something which is not a function but %s", "a string"),
@@ -605,9 +605,9 @@ TEST_F(ErrorTraceTest, zipAttrsWith)
HintFmt("expected a set but found %s: %s", "an integer", Uncolored(ANSI_CYAN "1" ANSI_NORMAL)),
HintFmt("while evaluating a value of the list passed as second argument to builtins.zipAttrsWith"));
// XXX: How to properly tell that the fucntion takes two arguments ?
// XXX: How to properly tell that the function takes two arguments ?
// The same question also applies to sort, and maybe others.
// Due to lazyness, we only create a thunk, and it fails later on.
// Due to laziness, we only create a thunk, and it fails later on.
// ASSERT_TRACE2("zipAttrsWith (_: 1) [ { foo = 1; } ]",
// TypeError,
// HintFmt("attempt to call something which is not a function but %s", "an integer"),
@@ -825,7 +825,7 @@ TEST_F(ErrorTraceTest, genList)
HintFmt("expected a function but found %s: %s", "an integer", Uncolored(ANSI_CYAN "1" ANSI_NORMAL)),
HintFmt("while evaluating the first argument passed to builtins.genList"));
// XXX: defered
// XXX: deferred
// ASSERT_TRACE2("genList (x: x + \"foo\") 2 #TODO",
// TypeError,
// HintFmt("cannot add %s to an integer", "a string"),
+69 -49
View File
@@ -170,8 +170,8 @@ TEST_F(PrimOpTest, attrValues)
{
auto v = eval("builtins.attrValues { x = \"foo\"; a = 1; }");
ASSERT_THAT(v, IsListOfSize(2));
ASSERT_THAT(*v.listElems()[0], IsIntEq(1));
ASSERT_THAT(*v.listElems()[1], IsStringEq("foo"));
ASSERT_THAT(*v.listView()[0], IsIntEq(1));
ASSERT_THAT(*v.listView()[1], IsStringEq("foo"));
}
TEST_F(PrimOpTest, getAttr)
@@ -284,8 +284,8 @@ TEST_F(PrimOpTest, catAttrs)
{
auto v = eval("builtins.catAttrs \"a\" [{a = 1;} {b = 0;} {a = 2;}]");
ASSERT_THAT(v, IsListOfSize(2));
ASSERT_THAT(*v.listElems()[0], IsIntEq(1));
ASSERT_THAT(*v.listElems()[1], IsIntEq(2));
ASSERT_THAT(*v.listView()[0], IsIntEq(1));
ASSERT_THAT(*v.listView()[1], IsIntEq(2));
}
TEST_F(PrimOpTest, functionArgs)
@@ -341,6 +341,7 @@ TEST_F(PrimOpTest, elemtAt)
TEST_F(PrimOpTest, elemtAtOutOfBounds)
{
ASSERT_THROW(eval("builtins.elemAt [0 1 2 3] 5"), Error);
ASSERT_THROW(eval("builtins.elemAt [0] 4294967296"), Error);
}
TEST_F(PrimOpTest, head)
@@ -363,7 +364,8 @@ TEST_F(PrimOpTest, tail)
{
auto v = eval("builtins.tail [ 3 2 1 0 ]");
ASSERT_THAT(v, IsListOfSize(3));
for (const auto [n, elem] : enumerate(v.listItems()))
auto listView = v.listView();
for (const auto [n, elem] : enumerate(listView))
ASSERT_THAT(*elem, IsIntEq(2 - static_cast<int>(n)));
}
@@ -376,17 +378,17 @@ TEST_F(PrimOpTest, map)
{
auto v = eval("map (x: \"foo\" + x) [ \"bar\" \"bla\" \"abc\" ]");
ASSERT_THAT(v, IsListOfSize(3));
auto elem = v.listElems()[0];
auto elem = v.listView()[0];
ASSERT_THAT(*elem, IsThunk());
state.forceValue(*elem, noPos);
ASSERT_THAT(*elem, IsStringEq("foobar"));
elem = v.listElems()[1];
elem = v.listView()[1];
ASSERT_THAT(*elem, IsThunk());
state.forceValue(*elem, noPos);
ASSERT_THAT(*elem, IsStringEq("foobla"));
elem = v.listElems()[2];
elem = v.listView()[2];
ASSERT_THAT(*elem, IsThunk());
state.forceValue(*elem, noPos);
ASSERT_THAT(*elem, IsStringEq("fooabc"));
@@ -396,7 +398,7 @@ TEST_F(PrimOpTest, filter)
{
auto v = eval("builtins.filter (x: x == 2) [ 3 2 3 2 3 2 ]");
ASSERT_THAT(v, IsListOfSize(3));
for (const auto elem : v.listItems())
for (const auto elem : v.listView())
ASSERT_THAT(*elem, IsIntEq(2));
}
@@ -416,7 +418,8 @@ TEST_F(PrimOpTest, concatLists)
{
auto v = eval("builtins.concatLists [[1 2] [3 4]]");
ASSERT_THAT(v, IsListOfSize(4));
for (const auto [i, elem] : enumerate(v.listItems()))
auto listView = v.listView();
for (const auto [i, elem] : enumerate(listView))
ASSERT_THAT(*elem, IsIntEq(static_cast<int>(i) + 1));
}
@@ -461,7 +464,8 @@ TEST_F(PrimOpTest, genList)
auto v = eval("builtins.genList (x: x + 1) 3");
ASSERT_EQ(v.type(), nList);
ASSERT_EQ(v.listSize(), 3u);
for (const auto [i, elem] : enumerate(v.listItems())) {
auto listView = v.listView();
for (const auto [i, elem] : enumerate(listView)) {
ASSERT_THAT(*elem, IsThunk());
state.forceValue(*elem, noPos);
ASSERT_THAT(*elem, IsIntEq(static_cast<int>(i) + 1));
@@ -475,7 +479,8 @@ TEST_F(PrimOpTest, sortLessThan)
ASSERT_EQ(v.listSize(), 6u);
const std::vector<int> numbers = {42, 77, 147, 249, 483, 526};
for (const auto [n, elem] : enumerate(v.listItems()))
auto listView = v.listView();
for (const auto [n, elem] : enumerate(listView))
ASSERT_THAT(*elem, IsIntEq(numbers[n]));
}
@@ -487,17 +492,17 @@ TEST_F(PrimOpTest, partition)
auto right = v.attrs()->get(createSymbol("right"));
ASSERT_NE(right, nullptr);
ASSERT_THAT(*right->value, IsListOfSize(2));
ASSERT_THAT(*right->value->listElems()[0], IsIntEq(23));
ASSERT_THAT(*right->value->listElems()[1], IsIntEq(42));
ASSERT_THAT(*right->value->listView()[0], IsIntEq(23));
ASSERT_THAT(*right->value->listView()[1], IsIntEq(42));
auto wrong = v.attrs()->get(createSymbol("wrong"));
ASSERT_NE(wrong, nullptr);
ASSERT_EQ(wrong->value->type(), nList);
ASSERT_EQ(wrong->value->listSize(), 3u);
ASSERT_THAT(*wrong->value, IsListOfSize(3));
ASSERT_THAT(*wrong->value->listElems()[0], IsIntEq(1));
ASSERT_THAT(*wrong->value->listElems()[1], IsIntEq(9));
ASSERT_THAT(*wrong->value->listElems()[2], IsIntEq(3));
ASSERT_THAT(*wrong->value->listView()[0], IsIntEq(1));
ASSERT_THAT(*wrong->value->listView()[1], IsIntEq(9));
ASSERT_THAT(*wrong->value->listView()[2], IsIntEq(3));
}
TEST_F(PrimOpTest, concatMap)
@@ -507,7 +512,8 @@ TEST_F(PrimOpTest, concatMap)
ASSERT_EQ(v.listSize(), 6u);
const std::vector<int> numbers = {1, 2, 0, 3, 4, 0};
for (const auto [n, elem] : enumerate(v.listItems()))
auto listView = v.listView();
for (const auto [n, elem] : enumerate(listView))
ASSERT_THAT(*elem, IsIntEq(numbers[n]));
}
@@ -671,6 +677,18 @@ TEST_F(PrimOpTest, substringSmallerString)
ASSERT_THAT(v, IsStringEq("n"));
}
TEST_F(PrimOpTest, substringHugeStart)
{
auto v = eval("builtins.substring 4294967296 5 \"nixos\"");
ASSERT_THAT(v, IsStringEq(""));
}
TEST_F(PrimOpTest, substringHugeLength)
{
auto v = eval("builtins.substring 0 4294967296 \"nixos\"");
ASSERT_THAT(v, IsStringEq("nixos"));
}
TEST_F(PrimOpTest, substringEmptyString)
{
auto v = eval("builtins.substring 1 3 \"\"");
@@ -752,8 +770,8 @@ TEST_F(PrimOpTest, derivation)
auto v = eval("derivation");
ASSERT_EQ(v.type(), nFunction);
ASSERT_TRUE(v.isLambda());
ASSERT_NE(v.payload.lambda.fun, nullptr);
ASSERT_TRUE(v.payload.lambda.fun->hasFormals());
ASSERT_NE(v.lambda().fun, nullptr);
ASSERT_TRUE(v.lambda().fun->hasFormals());
}
TEST_F(PrimOpTest, currentTime)
@@ -769,7 +787,8 @@ TEST_F(PrimOpTest, splitVersion)
ASSERT_THAT(v, IsListOfSize(4));
const std::vector<std::string_view> strings = {"1", "2", "3", "git"};
for (const auto [n, p] : enumerate(v.listItems()))
auto listView = v.listView();
for (const auto [n, p] : enumerate(listView))
ASSERT_THAT(*p, IsStringEq(strings[n]));
}
@@ -858,12 +877,12 @@ TEST_F(PrimOpTest, split1)
auto v = eval("builtins.split \"(a)b\" \"abc\"");
ASSERT_THAT(v, IsListOfSize(3));
ASSERT_THAT(*v.listElems()[0], IsStringEq(""));
ASSERT_THAT(*v.listView()[0], IsStringEq(""));
ASSERT_THAT(*v.listElems()[1], IsListOfSize(1));
ASSERT_THAT(*v.listElems()[1]->listElems()[0], IsStringEq("a"));
ASSERT_THAT(*v.listView()[1], IsListOfSize(1));
ASSERT_THAT(*v.listView()[1]->listView()[0], IsStringEq("a"));
ASSERT_THAT(*v.listElems()[2], IsStringEq("c"));
ASSERT_THAT(*v.listView()[2], IsStringEq("c"));
}
TEST_F(PrimOpTest, split2)
@@ -872,17 +891,17 @@ TEST_F(PrimOpTest, split2)
auto v = eval("builtins.split \"([ac])\" \"abc\"");
ASSERT_THAT(v, IsListOfSize(5));
ASSERT_THAT(*v.listElems()[0], IsStringEq(""));
ASSERT_THAT(*v.listView()[0], IsStringEq(""));
ASSERT_THAT(*v.listElems()[1], IsListOfSize(1));
ASSERT_THAT(*v.listElems()[1]->listElems()[0], IsStringEq("a"));
ASSERT_THAT(*v.listView()[1], IsListOfSize(1));
ASSERT_THAT(*v.listView()[1]->listView()[0], IsStringEq("a"));
ASSERT_THAT(*v.listElems()[2], IsStringEq("b"));
ASSERT_THAT(*v.listView()[2], IsStringEq("b"));
ASSERT_THAT(*v.listElems()[3], IsListOfSize(1));
ASSERT_THAT(*v.listElems()[3]->listElems()[0], IsStringEq("c"));
ASSERT_THAT(*v.listView()[3], IsListOfSize(1));
ASSERT_THAT(*v.listView()[3]->listView()[0], IsStringEq("c"));
ASSERT_THAT(*v.listElems()[4], IsStringEq(""));
ASSERT_THAT(*v.listView()[4], IsStringEq(""));
}
TEST_F(PrimOpTest, split3)
@@ -891,37 +910,37 @@ TEST_F(PrimOpTest, split3)
ASSERT_THAT(v, IsListOfSize(5));
// First list element
ASSERT_THAT(*v.listElems()[0], IsStringEq(""));
ASSERT_THAT(*v.listView()[0], IsStringEq(""));
// 2nd list element is a list [ "" null ]
ASSERT_THAT(*v.listElems()[1], IsListOfSize(2));
ASSERT_THAT(*v.listElems()[1]->listElems()[0], IsStringEq("a"));
ASSERT_THAT(*v.listElems()[1]->listElems()[1], IsNull());
ASSERT_THAT(*v.listView()[1], IsListOfSize(2));
ASSERT_THAT(*v.listView()[1]->listView()[0], IsStringEq("a"));
ASSERT_THAT(*v.listView()[1]->listView()[1], IsNull());
// 3rd element
ASSERT_THAT(*v.listElems()[2], IsStringEq("b"));
ASSERT_THAT(*v.listView()[2], IsStringEq("b"));
// 4th element is a list: [ null "c" ]
ASSERT_THAT(*v.listElems()[3], IsListOfSize(2));
ASSERT_THAT(*v.listElems()[3]->listElems()[0], IsNull());
ASSERT_THAT(*v.listElems()[3]->listElems()[1], IsStringEq("c"));
ASSERT_THAT(*v.listView()[3], IsListOfSize(2));
ASSERT_THAT(*v.listView()[3]->listView()[0], IsNull());
ASSERT_THAT(*v.listView()[3]->listView()[1], IsStringEq("c"));
// 5th element is the empty string
ASSERT_THAT(*v.listElems()[4], IsStringEq(""));
ASSERT_THAT(*v.listView()[4], IsStringEq(""));
}
TEST_F(PrimOpTest, split4)
{
auto v = eval("builtins.split \"([[:upper:]]+)\" \" FOO \"");
ASSERT_THAT(v, IsListOfSize(3));
auto first = v.listElems()[0];
auto second = v.listElems()[1];
auto third = v.listElems()[2];
auto first = v.listView()[0];
auto second = v.listView()[1];
auto third = v.listView()[2];
ASSERT_THAT(*first, IsStringEq(" "));
ASSERT_THAT(*second, IsListOfSize(1));
ASSERT_THAT(*second->listElems()[0], IsStringEq("FOO"));
ASSERT_THAT(*second->listView()[0], IsStringEq("FOO"));
ASSERT_THAT(*third, IsStringEq(" "));
}
@@ -942,15 +961,15 @@ TEST_F(PrimOpTest, match3)
{
auto v = eval("builtins.match \"a(b)(c)\" \"abc\"");
ASSERT_THAT(v, IsListOfSize(2));
ASSERT_THAT(*v.listElems()[0], IsStringEq("b"));
ASSERT_THAT(*v.listElems()[1], IsStringEq("c"));
ASSERT_THAT(*v.listView()[0], IsStringEq("b"));
ASSERT_THAT(*v.listView()[1], IsStringEq("c"));
}
TEST_F(PrimOpTest, match4)
{
auto v = eval("builtins.match \"[[:space:]]+([[:upper:]]+)[[:space:]]+\" \" FOO \"");
ASSERT_THAT(v, IsListOfSize(1));
ASSERT_THAT(*v.listElems()[0], IsStringEq("FOO"));
ASSERT_THAT(*v.listView()[0], IsStringEq("FOO"));
}
TEST_F(PrimOpTest, match5)
@@ -969,7 +988,8 @@ TEST_F(PrimOpTest, attrNames)
// ensure that the list is sorted
const std::vector<std::string_view> expected{"a", "x", "y", "z"};
for (const auto [n, elem] : enumerate(v.listItems()))
auto listView = v.listView();
for (const auto [n, elem] : enumerate(listView))
ASSERT_THAT(*elem, IsStringEq(expected[n]));
}
+1 -1
View File
@@ -168,7 +168,7 @@ TEST_P(AttrSetMergeTrvialExpressionTest, attrsetMergeLazy)
// Usually Nix rejects duplicate keys in an attrset but it does allow
// so if it is an attribute set that contains disjoint sets of keys.
// The below is equivalent to `{a.b = 1; a.c = 2; }`.
// The attribute set `a` will be a Thunk at first as the attribuets
// The attribute set `a` will be a Thunk at first as the attributes
// have to be merged (or otherwise computed) and that is done in a lazy
// manner.
+1 -1
View File
@@ -98,7 +98,7 @@ findAlongAttrPath(EvalState & state, const std::string & attrPath, Bindings & au
if (*attrIndex >= v->listSize())
throw AttrPathNotFound("list index %1% in selection path '%2%' is out of range", *attrIndex, attrPath);
v = v->listElems()[*attrIndex];
v = v->listView()[*attrIndex];
pos = noPos;
}
}
+1 -1
View File
@@ -651,7 +651,7 @@ std::vector<std::string> AttrCursor::getListOfStrings()
std::vector<std::string> res;
for (auto & elem : v.listItems())
for (auto elem : v.listView())
res.push_back(
std::string(root->state.forceStringNoCtx(*elem, noPos, "while evaluating an attribute for caching")));
+8
View File
@@ -4,6 +4,7 @@
#include "nix/util/config-global.hh"
#include "nix/util/serialise.hh"
#include "nix/expr/eval-gc.hh"
#include "nix/expr/value.hh"
#include "expr-config-private.hh"
@@ -52,6 +53,13 @@ static inline void initGCReal()
GC_INIT();
/* Register valid displacements in case we are using alignment niches
for storing the type information. This way tagged pointers are considered
to be valid, even when they are not aligned. */
if constexpr (detail::useBitPackedValueStorage<sizeof(void *)>)
for (std::size_t i = 1; i < sizeof(std::uintptr_t); ++i)
GC_register_displacement(i);
GC_set_oom_fn(oomHandler);
/* Set the initial heap size to something fairly big (25% of
+49
View File
@@ -0,0 +1,49 @@
#include "nix/expr/eval-profiler-settings.hh"
#include "nix/util/configuration.hh"
#include "nix/util/logging.hh" /* Needs to be included before config-impl.hh */
#include "nix/util/config-impl.hh"
#include "nix/util/abstract-setting-to-json.hh"
#include <nlohmann/json.hpp>
namespace nix {
template<>
EvalProfilerMode BaseSetting<EvalProfilerMode>::parse(const std::string & str) const
{
if (str == "disabled")
return EvalProfilerMode::disabled;
else if (str == "flamegraph")
return EvalProfilerMode::flamegraph;
else
throw UsageError("option '%s' has invalid value '%s'", name, str);
}
template<>
struct BaseSetting<EvalProfilerMode>::trait
{
static constexpr bool appendable = false;
};
template<>
std::string BaseSetting<EvalProfilerMode>::to_string() const
{
if (value == EvalProfilerMode::disabled)
return "disabled";
else if (value == EvalProfilerMode::flamegraph)
return "flamegraph";
else
unreachable();
}
NLOHMANN_JSON_SERIALIZE_ENUM(
EvalProfilerMode,
{
{EvalProfilerMode::disabled, "disabled"},
{EvalProfilerMode::flamegraph, "flamegraph"},
});
/* Explicit instantiation of templates */
template class BaseSetting<EvalProfilerMode>;
} // namespace nix
+355
View File
@@ -0,0 +1,355 @@
#include "nix/expr/eval-profiler.hh"
#include "nix/expr/nixexpr.hh"
#include "nix/expr/eval.hh"
#include "nix/util/lru-cache.hh"
namespace nix {
void EvalProfiler::preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) {}
void EvalProfiler::postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
}
void MultiEvalProfiler::preFunctionCallHook(
EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
for (auto & profiler : profilers) {
if (profiler->getNeededHooks().test(Hook::preFunctionCall))
profiler->preFunctionCallHook(state, v, args, pos);
}
}
void MultiEvalProfiler::postFunctionCallHook(
EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
for (auto & profiler : profilers) {
if (profiler->getNeededHooks().test(Hook::postFunctionCall))
profiler->postFunctionCallHook(state, v, args, pos);
}
}
EvalProfiler::Hooks MultiEvalProfiler::getNeededHooksImpl() const
{
Hooks hooks;
for (auto & p : profilers)
hooks |= p->getNeededHooks();
return hooks;
}
void MultiEvalProfiler::addProfiler(ref<EvalProfiler> profiler)
{
profilers.push_back(profiler);
invalidateNeededHooks();
}
namespace {
class PosCache : private LRUCache<PosIdx, Pos>
{
const EvalState & state;
public:
PosCache(const EvalState & state)
: LRUCache(524288) /* ~40MiB */
, state(state)
{
}
Pos lookup(PosIdx posIdx)
{
auto posOrNone = LRUCache::get(posIdx);
if (posOrNone)
return *posOrNone;
auto pos = state.positions[posIdx];
upsert(posIdx, pos);
return pos;
}
};
struct LambdaFrameInfo
{
ExprLambda * expr;
/** Position where the lambda has been called. */
PosIdx callPos = noPos;
std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const;
auto operator<=>(const LambdaFrameInfo & rhs) const = default;
};
/** Primop call. */
struct PrimOpFrameInfo
{
const PrimOp * expr;
/** Position where the primop has been called. */
PosIdx callPos = noPos;
std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const;
auto operator<=>(const PrimOpFrameInfo & rhs) const = default;
};
/** Used for functor calls (attrset with __functor attr). */
struct FunctorFrameInfo
{
PosIdx pos;
std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const;
auto operator<=>(const FunctorFrameInfo & rhs) const = default;
};
struct DerivationStrictFrameInfo
{
PosIdx callPos = noPos;
std::string drvName;
std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const;
auto operator<=>(const DerivationStrictFrameInfo & rhs) const = default;
};
/** Fallback frame info. */
struct GenericFrameInfo
{
PosIdx pos;
std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const;
auto operator<=>(const GenericFrameInfo & rhs) const = default;
};
using FrameInfo =
std::variant<LambdaFrameInfo, PrimOpFrameInfo, FunctorFrameInfo, DerivationStrictFrameInfo, GenericFrameInfo>;
using FrameStack = std::vector<FrameInfo>;
/**
* Stack sampling profiler.
*/
class SampleStack : public EvalProfiler
{
/* How often stack profiles should be flushed to file. This avoids the need
to persist stack samples across the whole evaluation at the cost
of periodically flushing data to disk. */
static constexpr std::chrono::microseconds profileDumpInterval = std::chrono::milliseconds(2000);
Hooks getNeededHooksImpl() const override
{
return Hooks().set(preFunctionCall).set(postFunctionCall);
}
FrameInfo getPrimOpFrameInfo(const PrimOp & primOp, std::span<Value *> args, PosIdx pos);
public:
SampleStack(EvalState & state, std::filesystem::path profileFile, std::chrono::nanoseconds period)
: state(state)
, sampleInterval(period)
, profileFd([&]() {
AutoCloseFD fd = toDescriptor(open(profileFile.string().c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0660));
if (!fd)
throw SysError("opening file %s", profileFile);
return fd;
}())
, posCache(state)
{
}
[[gnu::noinline]] void
preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
[[gnu::noinline]] void
postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
void maybeSaveProfile(std::chrono::time_point<std::chrono::high_resolution_clock> now);
void saveProfile();
FrameInfo getFrameInfoFromValueAndPos(const Value & v, std::span<Value *> args, PosIdx pos);
SampleStack(SampleStack &&) = default;
SampleStack & operator=(SampleStack &&) = delete;
SampleStack(const SampleStack &) = delete;
SampleStack & operator=(const SampleStack &) = delete;
~SampleStack();
private:
/** Hold on to an instance of EvalState for symbolizing positions. */
EvalState & state;
std::chrono::nanoseconds sampleInterval;
AutoCloseFD profileFd;
FrameStack stack;
std::map<FrameStack, uint32_t> callCount;
std::chrono::time_point<std::chrono::high_resolution_clock> lastStackSample =
std::chrono::high_resolution_clock::now();
std::chrono::time_point<std::chrono::high_resolution_clock> lastDump = std::chrono::high_resolution_clock::now();
PosCache posCache;
};
FrameInfo SampleStack::getPrimOpFrameInfo(const PrimOp & primOp, std::span<Value *> args, PosIdx pos)
{
auto derivationInfo = [&]() -> std::optional<FrameInfo> {
/* Here we rely a bit on the implementation details of libexpr/primops/derivation.nix
and derivationStrict primop. This is not ideal, but is necessary for
the usefulness of the profiler. This might actually affect the evaluation,
but the cost shouldn't be that high as to make the traces entirely inaccurate. */
if (primOp.name == "derivationStrict") {
try {
/* Error context strings don't actually matter, since we ignore all eval errors. */
state.forceAttrs(*args[0], pos, "");
auto attrs = args[0]->attrs();
auto nameAttr = state.getAttr(state.sName, attrs, "");
auto drvName = std::string(state.forceStringNoCtx(*nameAttr->value, pos, ""));
return DerivationStrictFrameInfo{.callPos = pos, .drvName = std::move(drvName)};
} catch (...) {
/* Ignore all errors, since those will be diagnosed by the evaluator itself. */
}
}
return std::nullopt;
}();
return derivationInfo.value_or(PrimOpFrameInfo{.expr = &primOp, .callPos = pos});
}
FrameInfo SampleStack::getFrameInfoFromValueAndPos(const Value & v, std::span<Value *> args, PosIdx pos)
{
/* NOTE: No actual references to garbage collected values are not held in
the profiler. */
if (v.isLambda())
return LambdaFrameInfo{.expr = v.lambda().fun, .callPos = pos};
else if (v.isPrimOp()) {
return getPrimOpFrameInfo(*v.primOp(), args, pos);
} else if (v.isPrimOpApp())
/* Resolve primOp eagerly. Must not hold on to a reference to a Value. */
return PrimOpFrameInfo{.expr = v.primOpAppPrimOp(), .callPos = pos};
else if (state.isFunctor(v)) {
const auto functor = v.attrs()->get(state.sFunctor);
if (auto pos_ = posCache.lookup(pos); std::holds_alternative<std::monostate>(pos_.origin))
/* HACK: In case callsite position is unresolved. */
return FunctorFrameInfo{.pos = functor->pos};
return FunctorFrameInfo{.pos = pos};
} else
/* NOTE: Add a stack frame even for invalid cases (e.g. when calling a non-function). This is what
* trace-function-calls does. */
return GenericFrameInfo{.pos = pos};
}
[[gnu::noinline]] void
SampleStack::preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
stack.push_back(getFrameInfoFromValueAndPos(v, args, pos));
auto now = std::chrono::high_resolution_clock::now();
if (now - lastStackSample > sampleInterval) {
callCount[stack] += 1;
lastStackSample = now;
}
/* Do this in preFunctionCallHook because we might throw an exception, but
callFunction uses Finally, which doesn't play well with exceptions. */
maybeSaveProfile(now);
}
[[gnu::noinline]] void
SampleStack::postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
if (!stack.empty())
stack.pop_back();
}
std::ostream & LambdaFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const
{
if (auto pos = posCache.lookup(callPos); std::holds_alternative<std::monostate>(pos.origin))
/* HACK: To avoid dubious «none»:0 in the generated profile if the origin can't be resolved
resort to printing the lambda location instead of the callsite position. */
os << posCache.lookup(expr->getPos());
else
os << pos;
if (expr->name)
os << ":" << state.symbols[expr->name];
return os;
}
std::ostream & GenericFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const
{
os << posCache.lookup(pos);
return os;
}
std::ostream & FunctorFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const
{
os << posCache.lookup(pos) << ":functor";
return os;
}
std::ostream & PrimOpFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const
{
/* Sometimes callsite position can have an unresolved origin, which
leads to confusing «none»:0 locations in the profile. */
auto pos = posCache.lookup(callPos);
if (!std::holds_alternative<std::monostate>(pos.origin))
os << posCache.lookup(callPos) << ":";
os << *expr;
return os;
}
std::ostream &
DerivationStrictFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const
{
/* Sometimes callsite position can have an unresolved origin, which
leads to confusing «none»:0 locations in the profile. */
auto pos = posCache.lookup(callPos);
if (!std::holds_alternative<std::monostate>(pos.origin))
os << posCache.lookup(callPos) << ":";
os << "primop derivationStrict:" << drvName;
return os;
}
void SampleStack::maybeSaveProfile(std::chrono::time_point<std::chrono::high_resolution_clock> now)
{
if (now - lastDump >= profileDumpInterval)
saveProfile();
else
return;
/* Save the last dump timepoint. Do this after actually saving data to file
to not account for the time doing the flushing to disk. */
lastDump = std::chrono::high_resolution_clock::now();
/* Free up memory used for stack sampling. This might be very significant for
long-running evaluations, so we shouldn't hog too much memory. */
callCount.clear();
}
void SampleStack::saveProfile()
{
auto os = std::ostringstream{};
for (auto & [stack, count] : callCount) {
auto first = true;
for (auto & pos : stack) {
if (first)
first = false;
else
os << ";";
std::visit([&](auto && info) { info.symbolize(state, os, posCache); }, pos);
}
os << " " << count;
writeLine(profileFd.get(), std::move(os).str());
/* Clear ostringstream. */
os.str("");
os.clear();
}
}
SampleStack::~SampleStack()
{
/* Guard against cases when we are already unwinding the stack. */
try {
saveProfile();
} catch (...) {
ignoreExceptionInDestructor();
}
}
} // namespace
ref<EvalProfiler> makeSampleStackProfiler(EvalState & state, std::filesystem::path profileFile, uint64_t frequency)
{
/* 0 is a special value for sampling stack after each call. */
std::chrono::nanoseconds period = frequency == 0
? std::chrono::nanoseconds{0}
: std::chrono::nanoseconds{std::nano::den / frequency / std::nano::num};
return make_ref<SampleStack>(state, profileFile, period);
}
} // namespace nix
+91 -63
View File
@@ -2,6 +2,7 @@
#include "nix/expr/eval-settings.hh"
#include "nix/expr/primops.hh"
#include "nix/expr/print-options.hh"
#include "nix/expr/symbol-table.hh"
#include "nix/util/exit.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
@@ -89,21 +90,16 @@ std::string printValue(EvalState & state, Value & v)
return out.str();
}
Value * Value::toPtr(SymbolStr str) noexcept
{
return const_cast<Value *>(str.valuePtr());
}
void Value::print(EvalState & state, std::ostream & str, PrintOptions options)
{
printValue(state, str, *this, options);
}
const Value * getPrimOp(const Value & v)
{
const Value * primOp = &v;
while (primOp->isPrimOpApp()) {
primOp = primOp->payload.primOpApp.left;
}
assert(primOp->isPrimOp());
return primOp;
}
std::string_view showType(ValueType type, bool withArticle)
{
#define WA(a, w) withArticle ? a " " w : w
@@ -139,13 +135,13 @@ std::string showType(const Value & v)
// Allow selecting a subset of enum values
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wswitch-enum"
switch (v.internalType) {
switch (v.getInternalType()) {
case tString:
return v.payload.string.context ? "a string with context" : "a string";
return v.context() ? "a string with context" : "a string";
case tPrimOp:
return fmt("the built-in function '%s'", std::string(v.payload.primOp->name));
return fmt("the built-in function '%s'", std::string(v.primOp()->name));
case tPrimOpApp:
return fmt("the partially applied built-in function '%s'", std::string(getPrimOp(v)->payload.primOp->name));
return fmt("the partially applied built-in function '%s'", v.primOpAppPrimOp()->name);
case tExternal:
return v.external()->showType();
case tThunk:
@@ -163,13 +159,13 @@ PosIdx Value::determinePos(const PosIdx pos) const
// Allow selecting a subset of enum values
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wswitch-enum"
switch (internalType) {
switch (getInternalType()) {
case tAttrs:
return attrs()->pos;
case tLambda:
return payload.lambda.fun->pos;
return lambda().fun->pos;
case tApp:
return payload.app.left->determinePos(pos);
return app().left->determinePos(pos);
default:
return pos;
}
@@ -178,11 +174,10 @@ PosIdx Value::determinePos(const PosIdx pos) const
bool Value::isTrivial() const
{
return internalType != tApp && internalType != tPrimOpApp
&& (internalType != tThunk
|| (dynamic_cast<ExprAttrs *>(payload.thunk.expr)
&& ((ExprAttrs *) payload.thunk.expr)->dynamicAttrs.empty())
|| dynamic_cast<ExprLambda *>(payload.thunk.expr) || dynamic_cast<ExprList *>(payload.thunk.expr));
return !isa<tApp, tPrimOpApp>()
&& (!isa<tThunk>()
|| (dynamic_cast<ExprAttrs *>(thunk().expr) && ((ExprAttrs *) thunk().expr)->dynamicAttrs.empty())
|| dynamic_cast<ExprLambda *>(thunk().expr) || dynamic_cast<ExprList *>(thunk().expr));
}
static Symbol getName(const AttrName & name, EvalState & state, Env & env)
@@ -227,6 +222,7 @@ EvalState::EvalState(
, sRight(symbols.create("right"))
, sWrong(symbols.create("wrong"))
, sStructuredAttrs(symbols.create("__structuredAttrs"))
, sJson(symbols.create("__json"))
, sAllowedReferences(symbols.create("allowedReferences"))
, sAllowedRequisites(symbols.create("allowedRequisites"))
, sDisallowedReferences(symbols.create("disallowedReferences"))
@@ -387,6 +383,19 @@ EvalState::EvalState(
);
createBaseEnv(settings);
/* Register function call tracer. */
if (settings.traceFunctionCalls)
profiler.addProfiler(make_ref<FunctionCallTrace>());
switch (settings.evalProfilerMode) {
case EvalProfilerMode::flamegraph:
profiler.addProfiler(
makeSampleStackProfiler(*this, settings.evalProfileFile.get(), settings.evalProfilerFrequency));
break;
case EvalProfilerMode::disabled:
break;
}
}
EvalState::~EvalState() {}
@@ -499,7 +508,7 @@ void EvalState::addConstant(const std::string & name, Value * v, Constant info)
/* Install value the base environment. */
staticBaseEnv->vars.emplace_back(symbols.create(name), baseEnvDispl);
baseEnv.values[baseEnvDispl++] = v;
getBuiltins().payload.attrs->push_back(Attr(symbols.create(name2), v));
const_cast<Bindings *>(getBuiltins().attrs())->push_back(Attr(symbols.create(name2), v));
}
}
@@ -518,20 +527,22 @@ std::ostream & operator<<(std::ostream & output, const PrimOp & primOp)
const PrimOp * Value::primOpAppPrimOp() const
{
Value * left = payload.primOpApp.left;
Value * left = primOpApp().left;
while (left && !left->isPrimOp()) {
left = left->payload.primOpApp.left;
left = left->primOpApp().left;
}
if (!left)
return nullptr;
assert(left->isPrimOp());
return left->primOp();
}
void Value::mkPrimOp(PrimOp * p)
{
p->check();
finishValue(tPrimOp, {.primOp = p});
setStorage(p);
}
Value * EvalState::addPrimOp(PrimOp && primOp)
@@ -565,7 +576,7 @@ Value * EvalState::addPrimOp(PrimOp && primOp)
else {
staticBaseEnv->vars.emplace_back(envName, baseEnvDispl);
baseEnv.values[baseEnvDispl++] = v;
getBuiltins().payload.attrs->push_back(Attr(symbols.create(primOp.name), v));
const_cast<Bindings *>(getBuiltins().attrs())->push_back(Attr(symbols.create(primOp.name), v));
}
return v;
@@ -599,7 +610,7 @@ std::optional<EvalState::Doc> EvalState::getDoc(Value & v)
};
}
if (v.isLambda()) {
auto exprLambda = v.payload.lambda.fun;
auto exprLambda = v.lambda().fun;
std::ostringstream s;
std::string name;
@@ -645,7 +656,7 @@ std::optional<EvalState::Doc> EvalState::getDoc(Value & v)
Value & functor = *v.attrs()->find(sFunctor)->value;
Value * vp[] = {&v};
Value partiallyApplied;
// The first paramater is not user-provided, and may be
// The first parameter is not user-provided, and may be
// handled by code that is opaque to the user, like lib.const = x: y: y;
// So preferably we show docs that are relevant to the
// "partially applied" function returned by e.g. `const`.
@@ -1495,7 +1506,14 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
{
auto _level = addCallDepth(pos);
auto trace = settings.traceFunctionCalls ? std::make_unique<FunctionCallTrace>(positions[pos]) : nullptr;
auto neededHooks = profiler.getNeededHooks();
if (neededHooks.test(EvalProfiler::preFunctionCall)) [[unlikely]]
profiler.preFunctionCallHook(*this, fun, args, pos);
Finally traceExit_{[&]() {
if (profiler.getNeededHooks().test(EvalProfiler::postFunctionCall)) [[unlikely]]
profiler.postFunctionCallHook(*this, fun, args, pos);
}};
forceValue(fun, pos);
@@ -1516,11 +1534,11 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
if (vCur.isLambda()) {
ExprLambda & lambda(*vCur.payload.lambda.fun);
ExprLambda & lambda(*vCur.lambda().fun);
auto size = (!lambda.arg ? 0 : 1) + (lambda.hasFormals() ? lambda.formals->formals.size() : 0);
Env & env2(allocEnv(size));
env2.up = vCur.payload.lambda.env;
env2.up = vCur.lambda().env;
Displacement displ = 0;
@@ -1552,7 +1570,7 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
symbols[i.name])
.atPos(lambda.pos)
.withTrace(pos, "from call site")
.withFrame(*vCur.payload.lambda.env, lambda)
.withFrame(*vCur.lambda().env, lambda)
.debugThrow();
}
env2.values[displ++] = i.def->maybeThunk(*this, env2);
@@ -1580,7 +1598,7 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
.atPos(lambda.pos)
.withTrace(pos, "from call site")
.withSuggestions(suggestions)
.withFrame(*vCur.payload.lambda.env, lambda)
.withFrame(*vCur.lambda().env, lambda)
.debugThrow();
}
unreachable();
@@ -1654,7 +1672,7 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
Value * primOp = &vCur;
while (primOp->isPrimOpApp()) {
argsDone++;
primOp = primOp->payload.primOpApp.left;
primOp = primOp->primOpApp().left;
}
assert(primOp->isPrimOp());
auto arity = primOp->primOp()->arity;
@@ -1670,8 +1688,8 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
Value * vArgs[maxPrimOpArity];
auto n = argsDone;
for (Value * arg = &vCur; arg->isPrimOpApp(); arg = arg->payload.primOpApp.left)
vArgs[--n] = arg->payload.primOpApp.right;
for (Value * arg = &vCur; arg->isPrimOpApp(); arg = arg->primOpApp().left)
vArgs[--n] = arg->primOpApp().right;
for (size_t i = 0; i < argsLeft; ++i)
vArgs[argsDone + i] = args[i];
@@ -1769,15 +1787,14 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res
}
}
if (!fun.isLambda() || !fun.payload.lambda.fun->hasFormals()) {
if (!fun.isLambda() || !fun.lambda().fun->hasFormals()) {
res = fun;
return;
}
auto attrs =
buildBindings(std::max(static_cast<uint32_t>(fun.payload.lambda.fun->formals->formals.size()), args.size()));
auto attrs = buildBindings(std::max(static_cast<uint32_t>(fun.lambda().fun->formals->formals.size()), args.size()));
if (fun.payload.lambda.fun->formals->ellipsis) {
if (fun.lambda().fun->formals->ellipsis) {
// If the formals have an ellipsis (eg the function accepts extra args) pass
// all available automatic arguments (which includes arguments specified on
// the command line via --arg/--argstr)
@@ -1785,7 +1802,7 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res
attrs.insert(v);
} else {
// Otherwise, only pass the arguments that the function accepts
for (auto & i : fun.payload.lambda.fun->formals->formals) {
for (auto & i : fun.lambda().fun->formals->formals) {
auto j = args.get(i.name);
if (j) {
attrs.insert(*j);
@@ -1798,7 +1815,7 @@ values, or passed explicitly with '--arg' or '--argstr'. See
https://nixos.org/manual/nix/stable/language/constructs.html#functions.)",
symbols[i.name])
.atPos(i.pos)
.withFrame(*fun.payload.lambda.env, *fun.payload.lambda.fun)
.withFrame(*fun.lambda().env, *fun.lambda().fun)
.debugThrow();
}
}
@@ -1969,9 +1986,10 @@ void EvalState::concatLists(
auto list = buildList(len);
auto out = list.elems;
for (size_t n = 0, pos = 0; n < nrLists; ++n) {
auto l = lists[n]->listSize();
auto listView = lists[n]->listView();
auto l = listView.size();
if (l)
memcpy(out + pos, lists[n]->listElems(), l * sizeof(Value *));
memcpy(out + pos, listView.data(), l * sizeof(Value *));
pos += l;
}
v.mkList(list);
@@ -2134,8 +2152,8 @@ void EvalState::forceValueDeep(Value & v)
// If the value is a thunk, we're evaling. Otherwise no trace necessary.
auto dts = debugRepl && i.value->isThunk() ? makeDebugTraceStacker(
*this,
*i.value->payload.thunk.expr,
*i.value->payload.thunk.env,
*i.value->thunk().expr,
*i.value->thunk().env,
i.pos,
"while evaluating the attribute '%1%'",
symbols[i.name])
@@ -2149,7 +2167,7 @@ void EvalState::forceValueDeep(Value & v)
}
else if (v.isList()) {
for (auto v2 : v.listItems())
for (auto v2 : v.listView())
recurse(*v2);
}
};
@@ -2211,7 +2229,16 @@ bool EvalState::forceBool(Value & v, const PosIdx pos, std::string_view errorCtx
return v.boolean();
}
bool EvalState::isFunctor(Value & fun)
Bindings::const_iterator EvalState::getAttr(Symbol attrSym, const Bindings * attrSet, std::string_view errorCtx)
{
auto value = attrSet->find(attrSym);
if (value == attrSet->end()) {
error<TypeError>("attribute '%s' missing", symbols[attrSym]).withTrace(noPos, errorCtx).debugThrow();
}
return value;
}
bool EvalState::isFunctor(const Value & fun) const
{
return fun.type() == nAttrs && fun.attrs()->find(sFunctor) != fun.attrs()->end();
}
@@ -2249,8 +2276,8 @@ std::string_view EvalState::forceString(Value & v, const PosIdx pos, std::string
void copyContext(const Value & v, NixStringContext & context, const ExperimentalFeatureSettings & xpSettings)
{
if (v.payload.string.context)
for (const char ** p = v.payload.string.context; *p; ++p)
if (v.context())
for (const char ** p = v.context(); *p; ++p)
context.insert(NixStringContextElem::parse(*p, xpSettings));
}
@@ -2331,7 +2358,7 @@ BackedStringView EvalState::coerceToString(
return !canonicalizePath && !copyToStore
? // FIXME: hack to preserve path literals that end in a
// slash, as in /foo/${x}.
v.payload.path.path
v.pathStr()
: copyToStore ? store->printStorePath(copyPathToStore(context, v.path()))
: std::string(v.path().path.abs());
}
@@ -2375,7 +2402,8 @@ BackedStringView EvalState::coerceToString(
if (v.isList()) {
std::string result;
for (auto [n, v2] : enumerate(v.listItems())) {
auto listView = v.listView();
for (auto [n, v2] : enumerate(listView)) {
try {
result += *coerceToString(
pos,
@@ -2412,6 +2440,7 @@ StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePat
auto dstPath = dstPathCached ? *dstPathCached : [&]() {
auto dstPath = fetchToStore(
fetchSettings,
*store,
path.resolveSymlinks(SymlinkResolution::Ancestors),
settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
@@ -2453,7 +2482,7 @@ SourcePath EvalState::coerceToPath(const PosIdx pos, Value & v, NixStringContext
}
}
/* Any other value should be coercable to a string, interpreted
/* Any other value should be coercible to a string, interpreted
relative to the root filesystem. */
auto path = coerceToString(pos, v, context, errorCtx, false, false, true).toOwned();
if (path == "" || path[0] != '/')
@@ -2594,14 +2623,14 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st
return;
case nPath:
if (v1.payload.path.accessor != v2.payload.path.accessor) {
if (v1.pathAccessor() != v2.pathAccessor()) {
error<AssertionError>(
"path '%s' is not equal to path '%s' because their accessors are different",
ValuePrinter(*this, v1, errorPrintOptions),
ValuePrinter(*this, v2, errorPrintOptions))
.debugThrow();
}
if (strcmp(v1.payload.path.path, v2.payload.path.path) != 0) {
if (strcmp(v1.pathStr(), v2.pathStr()) != 0) {
error<AssertionError>(
"path '%s' is not equal to path '%s'",
ValuePrinter(*this, v1, errorPrintOptions),
@@ -2625,7 +2654,7 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st
}
for (size_t n = 0; n < v1.listSize(); ++n) {
try {
assertEqValues(*v1.listElems()[n], *v2.listElems()[n], pos, errorCtx);
assertEqValues(*v1.listView()[n], *v2.listView()[n], pos, errorCtx);
} catch (Error & e) {
e.addTrace(positions[pos], "while comparing list element %d", n);
throw;
@@ -2772,8 +2801,7 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v
case nPath:
return
// FIXME: compare accessors by their fingerprint.
v1.payload.path.accessor == v2.payload.path.accessor
&& strcmp(v1.payload.path.path, v2.payload.path.path) == 0;
v1.pathAccessor() == v2.pathAccessor() && strcmp(v1.pathStr(), v2.pathStr()) == 0;
case nNull:
return true;
@@ -2782,7 +2810,7 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v
if (v1.listSize() != v2.listSize())
return false;
for (size_t n = 0; n < v1.listSize(); ++n)
if (!eqValues(*v1.listElems()[n], *v2.listElems()[n], pos, errorCtx))
if (!eqValues(*v1.listView()[n], *v2.listView()[n], pos, errorCtx))
return false;
return true;
@@ -2834,7 +2862,7 @@ bool EvalState::fullGC()
GC_gcollect();
// Check that it ran. We might replace this with a version that uses more
// of the boehm API to get this reliably, at a maintenance cost.
// We use a 1K margin because technically this has a race condtion, but we
// We use a 1K margin because technically this has a race condition, but we
// probably won't encounter it in practice, because the CLI isn't concurrent
// like that.
return GC_get_bytes_since_gc() < 1024;
@@ -2987,7 +3015,7 @@ void EvalState::printStatistics()
// XXX: overrides earlier assignment
topObj["symbols"] = json::array();
auto & list = topObj["symbols"];
symbols.dump([&](const std::string & s) { list.emplace_back(s); });
symbols.dump([&](std::string_view s) { list.emplace_back(s); });
}
if (outPath == "-") {
std::cerr << topObj.dump(2) << std::endl;
@@ -3115,7 +3143,7 @@ std::optional<SourcePath> EvalState::resolveLookupPathPath(const LookupPath::Pat
if (EvalSettings::isPseudoUrl(value)) {
try {
auto accessor = fetchers::downloadTarball(store, fetchSettings, EvalSettings::resolvePseudoUrl(value));
auto storePath = fetchToStore(*store, SourcePath(accessor), FetchMode::Copy);
auto storePath = fetchToStore(fetchSettings, *store, SourcePath(accessor), FetchMode::Copy);
return finish(this->storePath(storePath));
} catch (Error & e) {
logWarning({.msg = HintFmt("Nix search path entry '%1%' cannot be downloaded, ignoring", value)});
+6 -5
View File
@@ -3,19 +3,20 @@
namespace nix {
FunctionCallTrace::FunctionCallTrace(const Pos & pos)
: pos(pos)
void FunctionCallTrace::preFunctionCallHook(
EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
auto duration = std::chrono::high_resolution_clock::now().time_since_epoch();
auto ns = std::chrono::duration_cast<std::chrono::nanoseconds>(duration);
printMsg(lvlInfo, "function-trace entered %1% at %2%", pos, ns.count());
printMsg(lvlInfo, "function-trace entered %1% at %2%", state.positions[pos], ns.count());
}
FunctionCallTrace::~FunctionCallTrace()
void FunctionCallTrace::postFunctionCallHook(
EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
auto duration = std::chrono::high_resolution_clock::now().time_since_epoch();
auto ns = std::chrono::duration_cast<std::chrono::nanoseconds>(duration);
printMsg(lvlInfo, "function-trace exited %1% at %2%", pos, ns.count());
printMsg(lvlInfo, "function-trace exited %1% at %2%", state.positions[pos], ns.count());
}
} // namespace nix
+5 -4
View File
@@ -115,7 +115,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
state->forceList(*i->value, i->pos, "while evaluating the 'outputs' attribute of a derivation");
/* For each output... */
for (auto elem : i->value->listItems()) {
for (auto elem : i->value->listView()) {
std::string output(
state->forceStringNoCtx(*elem, i->pos, "while evaluating the name of an output of a derivation"));
@@ -166,7 +166,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
if (!outTI->isList())
throw errMsg;
Outputs result;
for (auto elem : outTI->listItems()) {
for (auto elem : outTI->listView()) {
if (elem->type() != nString)
throw errMsg;
auto out = outputs.find(elem->c_str());
@@ -216,7 +216,7 @@ bool PackageInfo::checkMeta(Value & v)
{
state->forceValue(v, v.determinePos(noPos));
if (v.type() == nList) {
for (auto elem : v.listItems())
for (auto elem : v.listView())
if (!checkMeta(*elem))
return false;
return true;
@@ -427,7 +427,8 @@ static void getDerivations(
}
else if (v.type() == nList) {
for (auto [n, elem] : enumerate(v.listItems())) {
auto listView = v.listView();
for (auto [n, elem] : enumerate(listView)) {
std::string pathPrefix2 = addToPath(pathPrefix, fmt("%d", n));
if (getDerivation(state, *elem, pathPrefix2, drvs, done, ignoreAssertionFailures))
getDerivations(state, *elem, pathPrefix2, autoArgs, drvs, done, ignoreAssertionFailures);
+3 -3
View File
@@ -89,9 +89,9 @@ Env & EvalState::allocEnv(size_t size)
void EvalState::forceValue(Value & v, const PosIdx pos)
{
if (v.isThunk()) {
Env * env = v.payload.thunk.env;
Env * env = v.thunk().env;
assert(env || v.isBlackhole());
Expr * expr = v.payload.thunk.expr;
Expr * expr = v.thunk().expr;
try {
v.mkBlackhole();
// checkInterrupt();
@@ -105,7 +105,7 @@ void EvalState::forceValue(Value & v, const PosIdx pos)
throw;
}
} else if (v.isApp())
callFunction(*v.payload.app.left, *v.payload.app.right, v, pos);
callFunction(*v.app().left, *v.app().right, v, pos);
}
[[gnu::always_inline]]
@@ -0,0 +1,16 @@
#pragma once
///@file
#include "nix/util/configuration.hh"
namespace nix {
enum struct EvalProfilerMode { disabled, flamegraph };
template<>
EvalProfilerMode BaseSetting<EvalProfilerMode>::parse(const std::string & str) const;
template<>
std::string BaseSetting<EvalProfilerMode>::to_string() const;
} // namespace nix
@@ -0,0 +1,114 @@
#pragma once
/**
* @file
*
* Evaluation profiler interface definitions and builtin implementations.
*/
#include "nix/util/ref.hh"
#include <vector>
#include <span>
#include <bitset>
#include <optional>
#include <filesystem>
namespace nix {
class EvalState;
class PosIdx;
struct Value;
class EvalProfiler
{
public:
enum Hook {
preFunctionCall,
postFunctionCall,
};
static constexpr std::size_t numHooks = Hook::postFunctionCall + 1;
using Hooks = std::bitset<numHooks>;
private:
std::optional<Hooks> neededHooks;
protected:
/** Invalidate the cached neededHooks. */
void invalidateNeededHooks()
{
neededHooks = std::nullopt;
}
/**
* Get which hooks need to be called.
*
* This is the actual implementation which has to be defined by subclasses.
* Public API goes through the needsHooks, which is a
* non-virtual interface (NVI) which caches the return value.
*/
virtual Hooks getNeededHooksImpl() const
{
return Hooks{};
}
public:
/**
* Hook called in the EvalState::callFunction preamble.
* Gets called only if (getNeededHooks().test(Hook::preFunctionCall)) is true.
*
* @param state Evaluator state.
* @param v Function being invoked.
* @param args Function arguments.
* @param pos Function position.
*/
virtual void preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos);
/**
* Hook called on EvalState::callFunction exit.
* Gets called only if (getNeededHooks().test(Hook::postFunctionCall)) is true.
*
* @param state Evaluator state.
* @param v Function being invoked.
* @param args Function arguments.
* @param pos Function position.
*/
virtual void postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos);
virtual ~EvalProfiler() = default;
/**
* Get which hooks need to be invoked for this EvalProfiler instance.
*/
Hooks getNeededHooks()
{
if (neededHooks.has_value())
return *neededHooks;
return *(neededHooks = getNeededHooksImpl());
}
};
/**
* Profiler that invokes multiple profilers at once.
*/
class MultiEvalProfiler : public EvalProfiler
{
std::vector<ref<EvalProfiler>> profilers;
[[gnu::noinline]] Hooks getNeededHooksImpl() const override;
public:
MultiEvalProfiler() = default;
/** Register a profiler instance. */
void addProfiler(ref<EvalProfiler> profiler);
[[gnu::noinline]] void
preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
[[gnu::noinline]] void
postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
};
ref<EvalProfiler> makeSampleStackProfiler(EvalState & state, std::filesystem::path profileFile, uint64_t frequency);
} // namespace nix
+54 -10
View File
@@ -1,6 +1,7 @@
#pragma once
///@file
#include "nix/expr/eval-profiler-settings.hh"
#include "nix/util/configuration.hh"
#include "nix/util/source-path.hh"
@@ -12,7 +13,7 @@ struct PrimOp;
struct EvalSettings : Config
{
/**
* Function used to interpet look path entries of a given scheme.
* Function used to interpret look path entries of a given scheme.
*
* The argument is the non-scheme part of the lookup path entry (see
* `LookupPathHooks` below).
@@ -139,7 +140,7 @@ struct EvalSettings : Config
false,
"restrict-eval",
R"(
If set to `true`, the Nix evaluator will not allow access to any
If set to `true`, the Nix evaluator doesn't allow access to any
files outside of
[`builtins.nixPath`](@docroot@/language/builtins.md#builtins-nixPath),
or to URIs outside of
@@ -161,6 +162,17 @@ struct EvalSettings : Config
- [`builtins.storePath`](@docroot@/language/builtins.md#builtins-storePath)
)"};
Setting<bool> traceImportFromDerivation{
this,
false,
"trace-import-from-derivation",
R"(
By default, Nix allows [Import from Derivation](@docroot@/language/import-from-derivation.md).
When this setting is `true`, Nix logs a warning indicating that it performed such an import.
This option has no effect if `allow-import-from-derivation` is disabled.
)"};
Setting<bool> enableImportFromDerivation{
this,
true,
@@ -168,9 +180,9 @@ struct EvalSettings : Config
R"(
By default, Nix allows [Import from Derivation](@docroot@/language/import-from-derivation.md).
With this option set to `false`, Nix will throw an error when evaluating an expression that uses this feature,
With this option set to `false`, Nix throws an error when evaluating an expression that uses this feature,
even when the required store object is readily available.
This ensures that evaluation will not require any builds to take place,
This ensures that evaluation doesn't require any builds to take place,
regardless of the state of the store.
)"};
@@ -195,8 +207,8 @@ struct EvalSettings : Config
false,
"trace-function-calls",
R"(
If set to `true`, the Nix evaluator will trace every function call.
Nix will print a log message at the "vomit" level for every function
If set to `true`, the Nix evaluator traces every function call.
Nix prints a log message at the "vomit" level for every function
entrance and function exit.
function-trace entered undefined position at 1565795816999559622
@@ -211,6 +223,38 @@ struct EvalSettings : Config
`flamegraph.pl`.
)"};
Setting<EvalProfilerMode> evalProfilerMode{
this,
EvalProfilerMode::disabled,
"eval-profiler",
R"(
Enables evaluation profiling. The following modes are supported:
* `flamegraph` stack sampling profiler. Outputs folded format, one line per stack (suitable for `flamegraph.pl` and compatible tools).
Use [`eval-profile-file`](#conf-eval-profile-file) to specify where the profile is saved.
See [Using the `eval-profiler`](@docroot@/advanced-topics/eval-profiler.md).
)"};
Setting<Path> evalProfileFile{
this,
"nix.profile",
"eval-profile-file",
R"(
Specifies the file where [evaluation profile](#conf-eval-profiler) is saved.
)"};
Setting<uint32_t> evalProfilerFrequency{
this,
99,
"eval-profiler-frequency",
R"(
Specifies the sampling rate in hertz for sampling evaluation profilers.
Use `0` to sample the stack after each function call.
See [`eval-profiler`](#conf-eval-profiler).
)"};
Setting<bool> useEvalCache{
this,
true,
@@ -226,8 +270,8 @@ struct EvalSettings : Config
false,
"ignore-try",
R"(
If set to true, ignore exceptions inside 'tryEval' calls when evaluating nix expressions in
debug mode (using the --debugger flag). By default the debugger will pause on all exceptions.
If set to true, ignore exceptions inside 'tryEval' calls when evaluating Nix expressions in
debug mode (using the --debugger flag). By default the debugger pauses on all exceptions.
)"};
Setting<bool> traceVerbose{
@@ -245,7 +289,7 @@ struct EvalSettings : Config
"debugger-on-trace",
R"(
If set to true and the `--debugger` flag is given, the following functions
will enter the debugger like [`builtins.break`](@docroot@/language/builtins.md#builtins-break).
enter the debugger like [`builtins.break`](@docroot@/language/builtins.md#builtins-break):
* [`builtins.trace`](@docroot@/language/builtins.md#builtins-trace)
* [`builtins.traceVerbose`](@docroot@/language/builtins.md#builtins-traceVerbose)
@@ -273,7 +317,7 @@ struct EvalSettings : Config
false,
"abort-on-warn",
R"(
If set to true, [`builtins.warn`](@docroot@/language/builtins.md#builtins-warn) will throw an error when logging a warning.
If set to true, [`builtins.warn`](@docroot@/language/builtins.md#builtins-warn) throws an error when logging a warning.
This will give you a stack trace that leads to the location of the warning.
+15 -5
View File
@@ -3,6 +3,7 @@
#include "nix/expr/attr-set.hh"
#include "nix/expr/eval-error.hh"
#include "nix/expr/eval-profiler.hh"
#include "nix/util/types.hh"
#include "nix/expr/value.hh"
#include "nix/expr/nixexpr.hh"
@@ -221,10 +222,11 @@ public:
PosTable positions;
const Symbol sWith, sOutPath, sDrvPath, sType, sMeta, sName, sValue, sSystem, sOverrides, sOutputs, sOutputName,
sIgnoreNulls, sFile, sLine, sColumn, sFunctor, sToString, sRight, sWrong, sStructuredAttrs, sAllowedReferences,
sAllowedRequisites, sDisallowedReferences, sDisallowedRequisites, sMaxSize, sMaxClosureSize, sBuilder, sArgs,
sContentAddressed, sImpure, sOutputHash, sOutputHashAlgo, sOutputHashMode, sRecurseForDerivations, sDescription,
sSelf, sEpsilon, sStartSet, sOperator, sKey, sPath, sPrefix, sOutputSpecified;
sIgnoreNulls, sFile, sLine, sColumn, sFunctor, sToString, sRight, sWrong, sStructuredAttrs, sJson,
sAllowedReferences, sAllowedRequisites, sDisallowedReferences, sDisallowedRequisites, sMaxSize, sMaxClosureSize,
sBuilder, sArgs, sContentAddressed, sImpure, sOutputHash, sOutputHashAlgo, sOutputHashMode,
sRecurseForDerivations, sDescription, sSelf, sEpsilon, sStartSet, sOperator, sKey, sPath, sPrefix,
sOutputSpecified;
const Expr::AstSymbols exprSymbols;
@@ -565,6 +567,11 @@ public:
const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
std::string_view forceStringNoCtx(Value & v, const PosIdx pos, std::string_view errorCtx);
/**
* Get attribute from an attribute set and throw an error if it doesn't exist.
*/
Bindings::const_iterator getAttr(Symbol attrSym, const Bindings * attrSet, std::string_view errorCtx);
template<typename... Args>
[[gnu::noinline]]
void addErrorTrace(Error & e, const Args &... formatArgs) const;
@@ -769,7 +776,7 @@ public:
*/
void assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::string_view errorCtx);
bool isFunctor(Value & fun);
bool isFunctor(const Value & fun) const;
void callFunction(Value & fun, std::span<Value *> args, Value & vRes, const PosIdx pos);
@@ -939,6 +946,9 @@ private:
typedef std::map<ExprLambda *, size_t> FunctionCalls;
FunctionCalls functionCalls;
/** Evaluation/call profiler. */
MultiEvalProfiler profiler;
void incrFunctionCall(ExprLambda * fun);
typedef std::map<PosIdx, size_t> AttrSelects;
+15 -6
View File
@@ -2,15 +2,24 @@
///@file
#include "nix/expr/eval.hh"
#include <chrono>
#include "nix/expr/eval-profiler.hh"
namespace nix {
struct FunctionCallTrace
class FunctionCallTrace : public EvalProfiler
{
const Pos pos;
FunctionCallTrace(const Pos & pos);
~FunctionCallTrace();
Hooks getNeededHooksImpl() const override
{
return Hooks().set(preFunctionCall).set(postFunctionCall);
}
public:
FunctionCallTrace() = default;
[[gnu::noinline]] void
preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
[[gnu::noinline]] void
postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
};
} // namespace nix
+2
View File
@@ -14,6 +14,8 @@ headers = [config_pub_h] + files(
'eval-error.hh',
'eval-gc.hh',
'eval-inline.hh',
'eval-profiler-settings.hh',
'eval-profiler.hh',
'eval-settings.hh',
'eval.hh',
'function-trace.hh',
+3
View File
@@ -391,6 +391,9 @@ struct Formal
struct Formals
{
typedef std::vector<Formal> Formals_;
/**
* @pre Sorted according to predicate (std::tie(a.name, a.pos) < std::tie(b.name, b.pos)).
*/
Formals_ formals;
bool ellipsis;
@@ -1,6 +1,7 @@
#pragma once
#include "nix/expr/value.hh"
#include "nix/expr/symbol-table.hh"
namespace nix {
+174 -54
View File
@@ -1,54 +1,30 @@
#pragma once
///@file
#include <list>
#include <map>
#include <unordered_map>
#include "nix/util/types.hh"
#include <memory_resource>
#include "nix/expr/value.hh"
#include "nix/util/chunked-vector.hh"
#include "nix/util/error.hh"
#include <boost/version.hpp>
#include <boost/unordered/unordered_flat_set.hpp>
namespace nix {
/**
* This class mainly exists to give us an operator<< for ostreams. We could also
* return plain strings from SymbolTable, but then we'd have to wrap every
* instance of a symbol that is fmt()ed, which is inconvenient and error-prone.
*/
class SymbolStr
class SymbolValue : protected Value
{
friend class SymbolStr;
friend class SymbolTable;
private:
const std::string * s;
uint32_t size_;
uint32_t idx;
explicit SymbolStr(const std::string & symbol)
: s(&symbol)
{
}
SymbolValue() = default;
public:
bool operator==(std::string_view s2) const
operator std::string_view() const noexcept
{
return *s == s2;
}
const char * c_str() const
{
return s->c_str();
}
operator const std::string_view() const
{
return *s;
}
friend std::ostream & operator<<(std::ostream & os, const SymbolStr & symbol);
bool empty() const
{
return s->empty();
return {c_str(), size_};
}
};
@@ -59,33 +35,35 @@ public:
*/
class Symbol
{
friend class SymbolStr;
friend class SymbolTable;
private:
uint32_t id;
explicit Symbol(uint32_t id)
explicit Symbol(uint32_t id) noexcept
: id(id)
{
}
public:
Symbol()
Symbol() noexcept
: id(0)
{
}
explicit operator bool() const
[[gnu::always_inline]]
explicit operator bool() const noexcept
{
return id > 0;
}
auto operator<=>(const Symbol other) const
auto operator<=>(const Symbol other) const noexcept
{
return id <=> other.id;
}
bool operator==(const Symbol other) const
bool operator==(const Symbol other) const noexcept
{
return id == other.id;
}
@@ -93,6 +71,145 @@ public:
friend class std::hash<Symbol>;
};
/**
* This class mainly exists to give us an operator<< for ostreams. We could also
* return plain strings from SymbolTable, but then we'd have to wrap every
* instance of a symbol that is fmt()ed, which is inconvenient and error-prone.
*/
class SymbolStr
{
friend class SymbolTable;
constexpr static size_t chunkSize{8192};
using SymbolValueStore = ChunkedVector<SymbolValue, chunkSize>;
const SymbolValue * s;
struct Key
{
using HashType = boost::hash<std::string_view>;
SymbolValueStore & store;
std::string_view s;
std::size_t hash;
std::pmr::polymorphic_allocator<char> & alloc;
Key(SymbolValueStore & store, std::string_view s, std::pmr::polymorphic_allocator<char> & stringAlloc)
: store(store)
, s(s)
, hash(HashType{}(s))
, alloc(stringAlloc)
{
}
};
public:
SymbolStr(const SymbolValue & s) noexcept
: s(&s)
{
}
SymbolStr(const Key & key)
{
auto size = key.s.size();
if (size >= std::numeric_limits<uint32_t>::max()) {
throw Error("Size of symbol exceeds 4GiB and cannot be stored");
}
// for multi-threaded implementations: lock store and allocator here
const auto & [v, idx] = key.store.add(SymbolValue{});
if (size == 0) {
v.mkString("", nullptr);
} else {
auto s = key.alloc.allocate(size + 1);
memcpy(s, key.s.data(), size);
s[size] = '\0';
v.mkString(s, nullptr);
}
v.size_ = size;
v.idx = idx;
this->s = &v;
}
bool operator==(std::string_view s2) const noexcept
{
return *s == s2;
}
[[gnu::always_inline]]
const char * c_str() const noexcept
{
return s->c_str();
}
[[gnu::always_inline]] operator std::string_view() const noexcept
{
return *s;
}
friend std::ostream & operator<<(std::ostream & os, const SymbolStr & symbol);
[[gnu::always_inline]]
bool empty() const noexcept
{
return s->size_ == 0;
}
[[gnu::always_inline]]
size_t size() const noexcept
{
return s->size_;
}
[[gnu::always_inline]]
const Value * valuePtr() const noexcept
{
return s;
}
explicit operator Symbol() const noexcept
{
return Symbol{s->idx + 1};
}
struct Hash
{
using is_transparent = void;
using is_avalanching = std::true_type;
std::size_t operator()(SymbolStr str) const
{
return Key::HashType{}(*str.s);
}
std::size_t operator()(const Key & key) const noexcept
{
return key.hash;
}
};
struct Equal
{
using is_transparent = void;
bool operator()(SymbolStr a, SymbolStr b) const noexcept
{
// strings are unique, so that a pointer comparison is OK
return a.s == b.s;
}
bool operator()(SymbolStr a, const Key & b) const noexcept
{
return a == b.s;
}
[[gnu::always_inline]]
bool operator()(const Key & a, SymbolStr b) const noexcept
{
return operator()(b, a);
}
};
};
/**
* Symbol table used by the parser and evaluator to represent and look
* up identifiers and attributes efficiently.
@@ -101,11 +218,18 @@ class SymbolTable
{
private:
/**
* Map from string view (backed by ChunkedVector) -> offset into the store.
* SymbolTable is an append only data structure.
* During its lifetime the monotonic buffer holds all strings and nodes, if the symbol set is node based.
*/
std::pmr::monotonic_buffer_resource buffer;
std::pmr::polymorphic_allocator<char> stringAlloc{&buffer};
SymbolStr::SymbolValueStore store{16};
/**
* Transparent lookup of string view for a pointer to a ChunkedVector entry -> return offset into the store.
* ChunkedVector references are never invalidated.
*/
std::unordered_map<std::string_view, uint32_t> symbols;
ChunkedVector<std::string, 8192> store{16};
boost::unordered_flat_set<SymbolStr, SymbolStr::Hash, SymbolStr::Equal> symbols{SymbolStr::chunkSize};
public:
@@ -117,13 +241,7 @@ public:
// Most symbols are looked up more than once, so we trade off insertion performance
// for lookup performance.
// FIXME: make this thread-safe.
auto it = symbols.find(s);
if (it != symbols.end())
return Symbol(it->second + 1);
const auto & [rawSym, idx] = store.add(s);
symbols.emplace(rawSym, idx);
return Symbol(idx + 1);
return Symbol(*symbols.insert(SymbolStr::Key{store, s, stringAlloc}).first);
}
std::vector<SymbolStr> resolve(const std::vector<Symbol> & symbols) const
@@ -137,12 +255,14 @@ public:
SymbolStr operator[](Symbol s) const
{
if (s.id == 0 || s.id > store.size())
uint32_t idx = s.id - uint32_t(1);
if (idx >= store.size())
unreachable();
return SymbolStr(store[s.id - 1]);
return store[idx];
}
size_t size() const
[[gnu::always_inline]]
size_t size() const noexcept
{
return store.size();
}
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -2,7 +2,7 @@
#include <cstddef>
// inluding the generated headers twice leads to errors
// including the generated headers twice leads to errors
#ifndef BISON_HEADER
# include "lexer-tab.hh"
# include "parser-tab.hh"
+2 -6
View File
@@ -72,12 +72,6 @@ toml11 = dependency(
method : 'cmake',
include_type: 'system',
)
configdata_priv.set(
'HAVE_TOML11_4',
toml11.version().version_compare('>= 4.0.0').to_int(),
)
deps_other += toml11
config_priv_h = configure_file(
@@ -146,6 +140,8 @@ sources = files(
'eval-cache.cc',
'eval-error.cc',
'eval-gc.cc',
'eval-profiler-settings.cc',
'eval-profiler.cc',
'eval-settings.cc',
'eval.cc',
'function-trace.cc',
+1 -1
View File
@@ -612,7 +612,7 @@ void ExprLambda::setDocComment(DocComment docComment)
size_t SymbolTable::totalSize() const
{
size_t n = 0;
dump([&](const std::string & s) { n += s.size(); });
dump([&](SymbolStr s) { n += s.size(); });
return n;
}
+2 -4
View File
@@ -70,15 +70,13 @@ mkMesonLibrary (finalAttrs: {
nix-util
nix-store
nix-fetchers
]
++ finalAttrs.passthru.externalPropagatedBuildInputs;
] ++ finalAttrs.passthru.externalPropagatedBuildInputs;
# Hack for sake of the dev shell
passthru.externalPropagatedBuildInputs = [
boost
nlohmann_json
]
++ lib.optional enableGC boehmgc;
] ++ lib.optional enableGC boehmgc;
mesonFlags = [
(lib.mesonEnable "gc" enableGC)
+213 -153
View File
@@ -14,6 +14,7 @@
#include "nix/expr/value-to-xml.hh"
#include "nix/expr/primops.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/util/sort.hh"
#include <boost/container/small_vector.hpp>
#include <nlohmann/json.hpp>
@@ -93,11 +94,16 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
if (drvs.empty())
return {};
if (isIFD && !settings.enableImportFromDerivation)
error<IFDError>(
"cannot build '%1%' during evaluation because the option 'allow-import-from-derivation' is disabled",
drvs.begin()->to_string(*store))
.debugThrow();
if (isIFD) {
if (!settings.enableImportFromDerivation)
error<IFDError>(
"cannot build '%1%' during evaluation because the option 'allow-import-from-derivation' is disabled",
drvs.begin()->to_string(*store))
.debugThrow();
if (settings.traceImportFromDerivation)
warn("built '%1%' during evaluation due to an import from derivation", drvs.begin()->to_string(*store));
}
/* Build/substitute the context. */
std::vector<DerivedPath> buildReqs;
@@ -350,7 +356,7 @@ static RegisterPrimOp primop_import(
> }
> ```
>
> then the following `foo.nix` will give an error:
> then the following `foo.nix` throws an error:
>
> ```nix
> # foo.nix
@@ -420,7 +426,7 @@ void prim_importNative(EvalState & state, const PosIdx pos, Value ** args, Value
void prim_exec(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.exec");
auto elems = args[0]->listElems();
auto elems = args[0]->listView();
auto count = args[0]->listSize();
if (count == 0)
state.error<EvalError>("at least one argument to 'exec' required").atPos(pos).debugThrow();
@@ -435,7 +441,7 @@ void prim_exec(EvalState & state, const PosIdx pos, Value ** args, Value & v)
false)
.toOwned();
Strings commandArgs;
for (unsigned int i = 1; i < args[0]->listSize(); ++i) {
for (size_t i = 1; i < count; ++i) {
commandArgs.push_back(state
.coerceToString(
pos,
@@ -689,7 +695,7 @@ struct CompareValues
// Note: we don't take the accessor into account
// since it's not obvious how to compare them in a
// reproducible way.
return strcmp(v1->payload.path.path, v2->payload.path.path) < 0;
return strcmp(v1->pathStr(), v2->pathStr()) < 0;
case nList:
// Lexicographic comparison
for (size_t i = 0;; i++) {
@@ -697,8 +703,8 @@ struct CompareValues
return false;
} else if (i == v1->listSize()) {
return true;
} else if (!state.eqValues(*v1->listElems()[i], *v2->listElems()[i], pos, errorCtx)) {
return (*this)(v1->listElems()[i], v2->listElems()[i], "while comparing two list elements");
} else if (!state.eqValues(*v1->listView()[i], *v2->listView()[i], pos, errorCtx)) {
return (*this)(v1->listView()[i], v2->listView()[i], "while comparing two list elements");
}
}
default:
@@ -718,25 +724,13 @@ struct CompareValues
typedef std::list<Value *, gc_allocator<Value *>> ValueList;
static Bindings::const_iterator
getAttr(EvalState & state, Symbol attrSym, const Bindings * attrSet, std::string_view errorCtx)
{
auto value = attrSet->find(attrSym);
if (value == attrSet->end()) {
state.error<TypeError>("attribute '%s' missing", state.symbols[attrSym])
.withTrace(noPos, errorCtx)
.debugThrow();
}
return value;
}
static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceAttrs(*args[0], noPos, "while evaluating the first argument passed to builtins.genericClosure");
/* Get the start set. */
auto startSet = getAttr(
state, state.sStartSet, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
auto startSet = state.getAttr(
state.sStartSet, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
state.forceList(
*startSet->value,
@@ -744,7 +738,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar
"while evaluating the 'startSet' attribute passed as argument to builtins.genericClosure");
ValueList workSet;
for (auto elem : startSet->value->listItems())
for (auto elem : startSet->value->listView())
workSet.push_back(elem);
if (startSet->value->listSize() == 0) {
@@ -753,8 +747,8 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar
}
/* Get the operator. */
auto op = getAttr(
state, state.sOperator, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
auto op = state.getAttr(
state.sOperator, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
state.forceFunction(
*op->value, noPos, "while evaluating the 'operator' attribute passed as argument to builtins.genericClosure");
@@ -775,8 +769,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar
noPos,
"while evaluating one of the elements generated by (or initially passed to) builtins.genericClosure");
auto key = getAttr(
state,
auto key = state.getAttr(
state.sKey,
e->attrs(),
"in one of the attrsets generated by (or initially passed to) builtins.genericClosure");
@@ -795,7 +788,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar
"while evaluating the return value of the `operator` passed to builtins.genericClosure");
/* Add the values returned by the operator to the work set. */
for (auto elem : newElements.listItems()) {
for (auto elem : newElements.listView()) {
state.forceValue(*elem, noPos); // "while evaluating one one of the elements returned by the `operator`
// passed to builtins.genericClosure");
workSet.push_back(elem);
@@ -983,7 +976,7 @@ static void prim_ceil(EvalState & state, const PosIdx pos, Value ** args, Value
if (arg != res) {
state
.error<EvalError>(
"Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occured in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.",
"Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occurred in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.",
arg,
res)
.atPos(pos)
@@ -1004,7 +997,7 @@ static RegisterPrimOp primop_ceil({
a NixInt and if `*number* < -9007199254740992` or `*number* > 9007199254740992`.
If the datatype of *number* is neither a NixInt (signed 64-bit integer) nor a NixFloat
(IEEE-754 double-precision floating-point number), an evaluation error will be thrown.
(IEEE-754 double-precision floating-point number), an evaluation error is thrown.
)",
.fun = prim_ceil,
});
@@ -1038,7 +1031,7 @@ static void prim_floor(EvalState & state, const PosIdx pos, Value ** args, Value
if (arg != res) {
state
.error<EvalError>(
"Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occured in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.",
"Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occurred in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.",
arg,
res)
.atPos(pos)
@@ -1075,7 +1068,7 @@ static void prim_tryEval(EvalState & state, const PosIdx pos, Value ** args, Val
ReplExitStatus (*savedDebugRepl)(ref<EvalState> es, const ValMap & extraEnv) = nullptr;
if (state.debugRepl && state.settings.ignoreExceptionsDuringTry) {
/* to prevent starting the repl from exceptions withing a tryEval, null it. */
/* to prevent starting the repl from exceptions within a tryEval, null it. */
savedDebugRepl = state.debugRepl;
state.debugRepl = nullptr;
}
@@ -1104,15 +1097,15 @@ static RegisterPrimOp primop_tryEval({
Try to shallowly evaluate *e*. Return a set containing the
attributes `success` (`true` if *e* evaluated successfully,
`false` if an error was thrown) and `value`, equalling *e* if
successful and `false` otherwise. `tryEval` will only prevent
successful and `false` otherwise. `tryEval` only prevents
errors created by `throw` or `assert` from being thrown.
Errors `tryEval` will not catch are for example those created
Errors `tryEval` doesn't catch are, for example, those created
by `abort` and type errors generated by builtins. Also note that
this doesn't evaluate *e* deeply, so `let e = { x = throw ""; };
in (builtins.tryEval e).success` will be `true`. Using
in (builtins.tryEval e).success` is `true`. Using
`builtins.deepSeq` one can get the expected result:
`let e = { x = throw ""; }; in
(builtins.tryEval (builtins.deepSeq e e)).success` will be
(builtins.tryEval (builtins.deepSeq e e)).success` is
`false`.
`tryEval` intentionally does not return the error message, because that risks bringing non-determinism into the evaluation result, and it would become very difficult to improve error reporting without breaking existing expressions.
@@ -1211,7 +1204,7 @@ static RegisterPrimOp primop_trace({
If the
[`debugger-on-trace`](@docroot@/command-ref/conf-file.md#conf-debugger-on-trace)
option is set to `true` and the `--debugger` flag is given, the
interactive debugger will be started when `trace` is called (like
interactive debugger is started when `trace` is called (like
[`break`](@docroot@/language/builtins.md#builtins-break)).
)",
.fun = prim_trace,
@@ -1263,7 +1256,7 @@ static RegisterPrimOp primop_warn({
If the
[`abort-on-warn`](@docroot@/command-ref/conf-file.md#conf-abort-on-warn)
option is set, the evaluation will be aborted after the warning is printed.
option is set, the evaluation is aborted after the warning is printed.
This is useful to reveal the stack trace of the warning, when the context is non-interactive and a debugger can not be launched.
)",
.fun = prim_warn,
@@ -1282,7 +1275,7 @@ static void prim_second(EvalState & state, const PosIdx pos, Value ** args, Valu
* Derivations
*************************************************************/
static void derivationStrictInternal(EvalState & state, const std::string & name, const Bindings * attrs, Value & v);
static void derivationStrictInternal(EvalState & state, std::string_view name, const Bindings * attrs, Value & v);
/* Construct (as a unobservable side effect) a Nix derivation
expression that performs the derivation described by the argument
@@ -1298,10 +1291,9 @@ static void prim_derivationStrict(EvalState & state, const PosIdx pos, Value **
auto attrs = args[0]->attrs();
/* Figure out the name first (for stack backtraces). */
auto nameAttr =
getAttr(state, state.sName, attrs, "in the attrset passed as argument to builtins.derivationStrict");
auto nameAttr = state.getAttr(state.sName, attrs, "in the attrset passed as argument to builtins.derivationStrict");
std::string drvName;
std::string_view drvName;
try {
drvName = state.forceStringNoCtx(
*nameAttr->value, pos, "while evaluating the `name` attribute passed to builtins.derivationStrict");
@@ -1365,7 +1357,7 @@ static void checkDerivationName(EvalState & state, std::string_view drvName)
}
}
static void derivationStrictInternal(EvalState & state, const std::string & drvName, const Bindings * attrs, Value & v)
static void derivationStrictInternal(EvalState & state, std::string_view drvName, const Bindings * attrs, Value & v)
{
checkDerivationName(state, drvName);
@@ -1474,7 +1466,7 @@ static void derivationStrictInternal(EvalState & state, const std::string & drvN
command-line arguments to the builder. */
else if (i->name == state.sArgs) {
state.forceList(*i->value, pos, context_below);
for (auto elem : i->value->listItems()) {
for (auto elem : i->value->listView()) {
auto s = state
.coerceToString(
pos, *elem, context, "while evaluating an element of the argument list", true)
@@ -1508,7 +1500,7 @@ static void derivationStrictInternal(EvalState & state, const std::string & drvN
/* Require ‘outputs’ to be a list of strings. */
state.forceList(*i->value, pos, context_below);
Strings ss;
for (auto elem : i->value->listItems())
for (auto elem : i->value->listView())
ss.emplace_back(state.forceStringNoCtx(*elem, pos, context_below));
handleOutputs(ss);
}
@@ -1553,6 +1545,10 @@ static void derivationStrictInternal(EvalState & state, const std::string & drvN
handleHashMode(s);
else if (i->name == state.sOutputs)
handleOutputs(tokenizeString<Strings>(s));
else if (i->name == state.sJson)
warn(
"In derivation '%s': setting structured attributes via '__json' is deprecated, and may be disallowed in future versions of Nix. Set '__structuredAttrs = true' instead.",
drvName);
}
}
@@ -1756,7 +1752,7 @@ static RegisterPrimOp primop_placeholder({
.name = "placeholder",
.args = {"output"},
.doc = R"(
Return at
Return an
[output placeholder string](@docroot@/store/derivation/index.md#output-placeholder)
for the specified *output* that will be substituted by the corresponding
[output path](@docroot@/glossary.md#gloss-output-path)
@@ -1919,7 +1915,7 @@ static RegisterPrimOp primop_baseNameOf({
After this, the *base name* is returned as previously described, assuming `/` as the directory separator. (Note that evaluation must be platform independent.)
This is somewhat similar to the [GNU `basename`](https://www.gnu.org/software/coreutils/manual/html_node/basename-invocation.html) command, but GNU `basename` will strip any number of trailing slashes.
This is somewhat similar to the [GNU `basename`](https://www.gnu.org/software/coreutils/manual/html_node/basename-invocation.html) command, but GNU `basename` strips any number of trailing slashes.
)",
.fun = prim_baseNameOf,
});
@@ -2000,7 +1996,7 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value ** args, Va
LookupPath lookupPath;
for (auto v2 : args[0]->listItems()) {
for (auto v2 : args[0]->listView()) {
state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.findFile");
std::string prefix;
@@ -2011,7 +2007,7 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value ** args, Va
pos,
"while evaluating the `prefix` attribute of an element of the list passed to builtins.findFile");
i = getAttr(state, state.sPath, v2->attrs(), "in an element of the __nixPath");
i = state.getAttr(state.sPath, v2->attrs(), "in an element of the __nixPath");
NixStringContext context;
auto path =
@@ -2027,7 +2023,7 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value ** args, Va
try {
auto rewrites = state.realiseContext(context);
path = rewriteStrings(path, rewrites);
path = rewriteStrings(std::move(path), rewrites);
} catch (InvalidPathError & e) {
state.error<EvalError>("cannot find '%1%', since path '%2%' is not valid", path, e.path)
.atPos(pos)
@@ -2036,8 +2032,8 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value ** args, Va
lookupPath.elements.emplace_back(
LookupPath::Elem{
.prefix = LookupPath::Prefix{.s = prefix},
.path = LookupPath::Path{.s = path},
.prefix = LookupPath::Prefix{.s = std::move(prefix)},
.path = LookupPath::Path{.s = std::move(path)},
});
}
@@ -2128,9 +2124,9 @@ static RegisterPrimOp primop_findFile(
> ]
> ```
>
> and a *lookup-path* value `"nixos-config"` will cause Nix to try `/home/eelco/Dev/nixos-config` and `/etc/nixos` in that order and return the first path that exists.
> and a *lookup-path* value `"nixos-config"` causes Nix to try `/home/eelco/Dev/nixos-config` and `/etc/nixos` in that order and return the first path that exists.
If `path` starts with `http://` or `https://`, it is interpreted as the URL of a tarball that will be downloaded and unpacked to a temporary location.
If `path` starts with `http://` or `https://`, it is interpreted as the URL of a tarball to be downloaded and unpacked to a temporary location.
The tarball must consist of a single top-level directory.
The URLs of the tarballs from the official `nixos.org` channels can be abbreviated as `channel:<channel-name>`.
@@ -2277,7 +2273,7 @@ static RegisterPrimOp primop_readDir({
Return the contents of the directory *path* as a set mapping
directory entries to the corresponding file type. For instance, if
directory `A` contains a regular file `B` and another directory
`C`, then `builtins.readDir ./A` will return the set
`C`, then `builtins.readDir ./A` returns the set
```nix
{ B = "regular"; C = "directory"; }
@@ -2314,8 +2310,8 @@ static RegisterPrimOp primop_outputOf({
[input placeholder string](@docroot@/store/derivation/index.md#input-placeholder)
if needed.
If the derivation has a statically-known output path (i.e. the derivation output is input-addressed, or fixed content-addresed), the output path will just be returned.
But if the derivation is content-addressed or if the derivation is itself not-statically produced (i.e. is the output of another derivation), an input placeholder will be returned instead.
If the derivation has a statically-known output path (i.e. the derivation output is input-addressed, or fixed content-addressed), the output path is returned.
But if the derivation is content-addressed or if the derivation is itself not-statically produced (i.e. is the output of another derivation), an input placeholder is returned instead.
*`derivation reference`* must be a string that may contain a regular store path to a derivation, or may be an input placeholder reference.
If the derivation is produced by a derivation, you must explicitly select `drv.outPath`.
@@ -2328,7 +2324,7 @@ static RegisterPrimOp primop_outputOf({
"out"
```
will return a input placeholder for the output of the output of `myDrv`.
returns an input placeholder for the output of the output of `myDrv`.
This primop corresponds to the `^` sigil for [deriving paths](@docroot@/glossary.md#gloss-deriving-paths), e.g. as part of installable syntax on the command line.
)",
@@ -2505,10 +2501,9 @@ static RegisterPrimOp primop_fromJSON({
static void prim_toFile(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
NixStringContext context;
std::string name(
state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.toFile"));
std::string contents(
state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.toFile"));
auto name = state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.toFile");
auto contents =
state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.toFile");
StorePathSet refs;
@@ -2679,6 +2674,7 @@ static void addPath(
if (!expectedHash || !state.store->isValidPath(*expectedStorePath)) {
auto dstPath = fetchToStore(
state.fetchSettings,
*state.store,
path.resolveSymlinks(),
settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
@@ -2721,12 +2717,12 @@ static RegisterPrimOp primop_filterSource({
>
> `filterSource` should not be used to filter store paths. Since
> `filterSource` uses the name of the input directory while naming
> the output directory, doing so will produce a directory name in
> the output directory, doing so produces a directory name in
> the form of `<hash2>-<hash>-<name>`, where `<hash>-<name>` is
> the name of the input directory. Since `<hash>` depends on the
> unfiltered directory, the name of the output directory will
> indirectly depend on files that are filtered out by the
> function. This will trigger a rebuild even when a filtered out
> unfiltered directory, the name of the output directory
> indirectly depends on files that are filtered out by the
> function. This triggers a rebuild even when a filtered out
> file is changed. Use `builtins.path` instead, which allows
> specifying the name of the output directory.
@@ -2741,8 +2737,8 @@ static RegisterPrimOp primop_filterSource({
}
```
However, if `source-dir` is a Subversion working copy, then all
those annoying `.svn` subdirectories will also be copied to the
However, if `source-dir` is a Subversion working copy, then all of
those annoying `.svn` subdirectories are also copied to the
store. Worse, the contents of those directories may change a lot,
causing lots of spurious rebuilds. With `filterSource` you can
filter out the `.svn` directories:
@@ -2762,8 +2758,8 @@ static RegisterPrimOp primop_filterSource({
`"regular"`, `"directory"`, `"symlink"` or `"unknown"` (for other
kinds of files such as device nodes or fifos — but note that those
cannot be copied to the Nix store, so if the predicate returns
`true` for them, the copy will fail). If you exclude a directory,
the entire corresponding subtree of *e2* will be excluded.
`true` for them, the copy fails). If you exclude a directory,
the entire corresponding subtree of *e2* is excluded.
)",
.fun = prim_filterSource,
});
@@ -2771,7 +2767,7 @@ static RegisterPrimOp primop_filterSource({
static void prim_path(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
std::optional<SourcePath> path;
std::string name;
std::string_view name;
Value * filterFun = nullptr;
auto method = ContentAddressMethod::Raw::NixArchive;
std::optional<Hash> expectedHash;
@@ -2843,7 +2839,7 @@ static RegisterPrimOp primop_path({
- sha256\
When provided, this is the expected hash of the file at the
path. Evaluation will fail if the hash is incorrect, and
path. Evaluation fails if the hash is incorrect, and
providing a hash allows `builtins.path` to be used even when the
`pure-eval` nix config option is on.
)",
@@ -2863,7 +2859,7 @@ static void prim_attrNames(EvalState & state, const PosIdx pos, Value ** args, V
auto list = state.buildList(args[0]->attrs()->size());
for (const auto & [n, i] : enumerate(*args[0]->attrs()))
(list[n] = state.allocValue())->mkString(state.symbols[i.name]);
list[n] = Value::toPtr(state.symbols[i.name]);
std::sort(list.begin(), list.end(), [](Value * v1, Value * v2) { return strcmp(v1->c_str(), v2->c_str()) < 0; });
@@ -2918,7 +2914,7 @@ void prim_getAttr(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto attr = state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.getAttr");
state.forceAttrs(*args[1], pos, "while evaluating the second argument passed to builtins.getAttr");
auto i = getAttr(state, state.symbols.create(attr), args[1]->attrs(), "in the attribute set under consideration");
auto i = state.getAttr(state.symbols.create(attr), args[1]->attrs(), "in the attribute set under consideration");
// !!! add to stack trace?
if (state.countCalls && i->pos)
state.attrSelects[i->pos]++;
@@ -2964,7 +2960,7 @@ static RegisterPrimOp primop_unsafeGetAttrPos(
.fun = prim_unsafeGetAttrPos,
});
// access to exact position information (ie, line and colum numbers) is deferred
// access to exact position information (ie, line and column numbers) is deferred
// due to the cost associated with calculating that information and how rarely
// it is used in practice. this is achieved by creating thunks to otherwise
// inaccessible primops that are not exposed as __op or under builtins to turn
@@ -2976,7 +2972,7 @@ static RegisterPrimOp primop_unsafeGetAttrPos(
// but each type of thunk has an associated runtime cost in the current evaluator.
// as with black holes this cost is too high to justify another thunk type to check
// for in the very hot path that is forceValue.
static struct LazyPosAcessors
static struct LazyPosAccessors
{
PrimOp primop_lineOfPos{.arity = 1, .fun = [](EvalState & state, PosIdx pos, Value ** args, Value & v) {
v.mkInt(state.positions[PosIdx(args[0]->integer().value)].line);
@@ -2987,7 +2983,7 @@ static struct LazyPosAcessors
Value lineOfPos, columnOfPos;
LazyPosAcessors()
LazyPosAccessors()
{
lineOfPos.mkPrimOp(&primop_lineOfPos);
columnOfPos.mkPrimOp(&primop_columnOfPos);
@@ -3053,7 +3049,7 @@ static void prim_removeAttrs(EvalState & state, const PosIdx pos, Value ** args,
// 64: large enough to fit the attributes of a derivation
boost::container::small_vector<Attr, 64> names;
names.reserve(args[1]->listSize());
for (auto elem : args[1]->listItems()) {
for (auto elem : args[1]->listView()) {
state.forceStringNoCtx(
*elem, pos, "while evaluating the values of the second argument passed to builtins.removeAttrs");
names.emplace_back(state.symbols.create(elem->string_view()), nullptr);
@@ -3094,28 +3090,51 @@ static void prim_listToAttrs(EvalState & state, const PosIdx pos, Value ** args,
{
state.forceList(*args[0], pos, "while evaluating the argument passed to builtins.listToAttrs");
auto attrs = state.buildBindings(args[0]->listSize());
// Step 1. Sort the name-value attrsets in place using the memory we allocate for the result
auto listView = args[0]->listView();
size_t listSize = listView.size();
auto & bindings = *state.allocBindings(listSize);
using ElemPtr = decltype(&bindings[0].value);
std::set<Symbol> seen;
for (auto v2 : args[0]->listItems()) {
for (const auto & [n, v2] : enumerate(listView)) {
state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.listToAttrs");
auto j = getAttr(state, state.sName, v2->attrs(), "in a {name=...; value=...;} pair");
auto j = state.getAttr(state.sName, v2->attrs(), "in a {name=...; value=...;} pair");
auto name = state.forceStringNoCtx(
*j->value,
j->pos,
"while evaluating the `name` attribute of an element of the list passed to builtins.listToAttrs");
auto sym = state.symbols.create(name);
if (seen.insert(sym).second) {
auto j2 = getAttr(state, state.sValue, v2->attrs(), "in a {name=...; value=...;} pair");
attrs.insert(sym, j2->value, j2->pos);
}
// (ab)use Attr to store a Value * * instead of a Value *, so that we can stabilize the sort using the Value * *
bindings[n] = Attr(sym, std::bit_cast<Value *>(&v2));
}
v.mkAttrs(attrs);
std::sort(&bindings[0], &bindings[listSize], [](const Attr & a, const Attr & b) {
// Note that .value is actually a Value * * that corresponds to the position in the list
return a < b || (!(a > b) && std::bit_cast<ElemPtr>(a.value) < std::bit_cast<ElemPtr>(b.value));
});
// Step 2. Unpack the bindings in place and skip name-value pairs with duplicate names
Symbol prev;
for (size_t n = 0; n < listSize; n++) {
auto attr = bindings[n];
if (prev == attr.name) {
continue;
}
// Note that .value is actually a Value * *; see earlier comments
Value * v2 = *std::bit_cast<ElemPtr>(attr.value);
auto j = state.getAttr(state.sValue, v2->attrs(), "in a {name=...; value=...;} pair");
prev = attr.name;
bindings.push_back({prev, j->value, j->pos});
}
// help GC and clear end of allocated array
for (size_t n = bindings.size(); n < listSize; n++) {
bindings[n] = Attr{};
}
v.mkAttrs(&bindings);
}
static RegisterPrimOp primop_listToAttrs({
@@ -3235,7 +3254,7 @@ static void prim_catAttrs(EvalState & state, const PosIdx pos, Value ** args, Va
SmallValueVector<nonRecursiveStackReservation> res(args[1]->listSize());
size_t found = 0;
for (auto v2 : args[1]->listItems()) {
for (auto v2 : args[1]->listView()) {
state.forceAttrs(
*v2, pos, "while evaluating an element in the list passed as second argument to builtins.catAttrs");
if (auto i = v2->attrs()->get(attrName))
@@ -3243,7 +3262,7 @@ static void prim_catAttrs(EvalState & state, const PosIdx pos, Value ** args, Va
}
auto list = state.buildList(found);
for (unsigned int n = 0; n < found; ++n)
for (size_t n = 0; n < found; ++n)
list[n] = res[n];
v.mkList(list);
}
@@ -3275,15 +3294,21 @@ static void prim_functionArgs(EvalState & state, const PosIdx pos, Value ** args
if (!args[0]->isLambda())
state.error<TypeError>("'functionArgs' requires a function").atPos(pos).debugThrow();
if (!args[0]->payload.lambda.fun->hasFormals()) {
if (!args[0]->lambda().fun->hasFormals()) {
v.mkAttrs(&state.emptyBindings);
return;
}
auto attrs = state.buildBindings(args[0]->payload.lambda.fun->formals->formals.size());
for (auto & i : args[0]->payload.lambda.fun->formals->formals)
const auto & formals = args[0]->lambda().fun->formals->formals;
auto attrs = state.buildBindings(formals.size());
for (auto & i : formals)
attrs.insert(i.name, state.getBool(i.def), i.pos);
v.mkAttrs(attrs);
/* Optimization: avoid sorting bindings. `formals` must already be sorted according to
(std::tie(a.name, a.pos) < std::tie(b.name, b.pos)) predicate, so the following assertion
always holds:
assert(std::is_sorted(attrs.alreadySorted()->begin(), attrs.alreadySorted()->end()));
.*/
v.mkAttrs(attrs.alreadySorted());
}
static RegisterPrimOp primop_functionArgs({
@@ -3311,9 +3336,8 @@ static void prim_mapAttrs(EvalState & state, const PosIdx pos, Value ** args, Va
auto attrs = state.buildBindings(args[1]->attrs()->size());
for (auto & i : *args[1]->attrs()) {
Value * vName = state.allocValue();
Value * vName = Value::toPtr(state.symbols[i.name]);
Value * vFun2 = state.allocValue();
vName->mkString(state.symbols[i.name]);
vFun2->mkApp(args[0], vName);
attrs.alloc(i.name).mkApp(vFun2, i.value);
}
@@ -3356,7 +3380,7 @@ static void prim_zipAttrsWith(EvalState & state, const PosIdx pos, Value ** args
state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.zipAttrsWith");
state.forceList(*args[1], pos, "while evaluating the second argument passed to builtins.zipAttrsWith");
const auto listItems = args[1]->listItems();
const auto listItems = args[1]->listView();
for (auto & vElem : listItems) {
state.forceAttrs(
@@ -3378,8 +3402,7 @@ static void prim_zipAttrsWith(EvalState & state, const PosIdx pos, Value ** args
auto attrs = state.buildBindings(attrsSeen.size());
for (auto & [sym, elem] : attrsSeen) {
auto name = state.allocValue();
name->mkString(state.symbols[sym]);
auto name = Value::toPtr(state.symbols[sym]);
auto call1 = state.allocValue();
call1->mkApp(args[0], name);
auto call2 = state.allocValue();
@@ -3450,12 +3473,12 @@ static void prim_elemAt(EvalState & state, const PosIdx pos, Value ** args, Valu
NixInt::Inner n =
state.forceInt(*args[1], pos, "while evaluating the second argument passed to 'builtins.elemAt'").value;
state.forceList(*args[0], pos, "while evaluating the first argument passed to 'builtins.elemAt'");
if (n < 0 || (unsigned int) n >= args[0]->listSize())
if (n < 0 || std::make_unsigned_t<NixInt::Inner>(n) >= args[0]->listSize())
state.error<EvalError>("'builtins.elemAt' called with index %d on a list of size %d", n, args[0]->listSize())
.atPos(pos)
.debugThrow();
state.forceValue(*args[0]->listElems()[n], pos);
v = *args[0]->listElems()[n];
state.forceValue(*args[0]->listView()[n], pos);
v = *args[0]->listView()[n];
}
static RegisterPrimOp primop_elemAt({
@@ -3474,8 +3497,8 @@ static void prim_head(EvalState & state, const PosIdx pos, Value ** args, Value
state.forceList(*args[0], pos, "while evaluating the first argument passed to 'builtins.head'");
if (args[0]->listSize() == 0)
state.error<EvalError>("'builtins.head' called on an empty list").atPos(pos).debugThrow();
state.forceValue(*args[0]->listElems()[0], pos);
v = *args[0]->listElems()[0];
state.forceValue(*args[0]->listView()[0], pos);
v = *args[0]->listView()[0];
}
static RegisterPrimOp primop_head({
@@ -3500,7 +3523,7 @@ static void prim_tail(EvalState & state, const PosIdx pos, Value ** args, Value
auto list = state.buildList(args[0]->listSize() - 1);
for (const auto & [n, v] : enumerate(list))
v = args[0]->listElems()[n + 1];
v = args[0]->listView()[n + 1];
v.mkList(list);
}
@@ -3534,7 +3557,7 @@ static void prim_map(EvalState & state, const PosIdx pos, Value ** args, Value &
auto list = state.buildList(args[1]->listSize());
for (const auto & [n, v] : enumerate(list))
(v = state.allocValue())->mkApp(args[0], args[1]->listElems()[n]);
(v = state.allocValue())->mkApp(args[0], args[1]->listView()[n]);
v.mkList(list);
}
@@ -3568,16 +3591,17 @@ static void prim_filter(EvalState & state, const PosIdx pos, Value ** args, Valu
state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.filter");
SmallValueVector<nonRecursiveStackReservation> vs(args[1]->listSize());
auto len = args[1]->listSize();
SmallValueVector<nonRecursiveStackReservation> vs(len);
size_t k = 0;
bool same = true;
for (unsigned int n = 0; n < args[1]->listSize(); ++n) {
for (size_t n = 0; n < len; ++n) {
Value res;
state.callFunction(*args[0], *args[1]->listElems()[n], res, noPos);
state.callFunction(*args[0], *args[1]->listView()[n], res, noPos);
if (state.forceBool(
res, pos, "while evaluating the return value of the filtering function passed to builtins.filter"))
vs[k++] = args[1]->listElems()[n];
vs[k++] = args[1]->listView()[n];
else
same = false;
}
@@ -3607,7 +3631,7 @@ static void prim_elem(EvalState & state, const PosIdx pos, Value ** args, Value
{
bool res = false;
state.forceList(*args[1], pos, "while evaluating the second argument passed to builtins.elem");
for (auto elem : args[1]->listItems())
for (auto elem : args[1]->listView())
if (state.eqValues(*args[0], *elem, pos, "while searching for the presence of the given element in the list")) {
res = true;
break;
@@ -3629,10 +3653,11 @@ static RegisterPrimOp primop_elem({
static void prim_concatLists(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.concatLists");
auto listView = args[0]->listView();
state.concatLists(
v,
args[0]->listSize(),
args[0]->listElems(),
listView.data(),
pos,
"while evaluating a value of the list passed to builtins.concatLists");
}
@@ -3672,7 +3697,8 @@ static void prim_foldlStrict(EvalState & state, const PosIdx pos, Value ** args,
if (args[2]->listSize()) {
Value * vCur = args[1];
for (auto [n, elem] : enumerate(args[2]->listItems())) {
auto listView = args[2]->listView();
for (auto [n, elem] : enumerate(listView)) {
Value * vs[]{vCur, elem};
vCur = n == args[2]->listSize() - 1 ? &v : state.allocValue();
state.callFunction(*args[0], vs, *vCur, pos);
@@ -3715,7 +3741,7 @@ static void anyOrAll(bool any, EvalState & state, const PosIdx pos, Value ** arg
: "while evaluating the return value of the function passed to builtins.all";
Value vTmp;
for (auto elem : args[1]->listItems()) {
for (auto elem : args[1]->listView()) {
state.callFunction(*args[0], *elem, vTmp, pos);
bool res = state.forceBool(vTmp, pos, errorCtx);
if (res == any) {
@@ -3761,12 +3787,12 @@ static void prim_genList(EvalState & state, const PosIdx pos, Value ** args, Val
{
auto len_ = state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.genList").value;
if (len_ < 0)
if (len_ < 0 || std::make_unsigned_t<NixInt::Inner>(len_) > std::numeric_limits<size_t>::max())
state.error<EvalError>("cannot create list of size %1%", len_).atPos(pos).debugThrow();
size_t len = size_t(len_);
// More strict than striclty (!) necessary, but acceptable
// More strict than strictly (!) necessary, but acceptable
// as evaluating map without accessing any values makes little sense.
state.forceFunction(*args[0], noPos, "while evaluating the first argument passed to builtins.genList");
@@ -3811,7 +3837,7 @@ static void prim_sort(EvalState & state, const PosIdx pos, Value ** args, Value
auto list = state.buildList(len);
for (const auto & [n, v] : enumerate(list))
state.forceValue(*(v = args[1]->listElems()[n]), pos);
state.forceValue(*(v = args[1]->listView()[n]), pos);
auto comparator = [&](Value * a, Value * b) {
/* Optimization: if the comparator is lessThan, bypass
@@ -3830,10 +3856,14 @@ static void prim_sort(EvalState & state, const PosIdx pos, Value ** args, Value
vBool, pos, "while evaluating the return value of the sorting function passed to builtins.sort");
};
/* FIXME: std::sort can segfault if the comparator is not a strict
weak ordering. What to do? std::stable_sort() seems more
resilient, but no guarantees... */
std::stable_sort(list.begin(), list.end(), comparator);
/* NOTE: Using custom implementation because std::sort and std::stable_sort
are not resilient to comparators that violate strict weak ordering. Diagnosing
incorrect implementations is a O(n^3) problem, so doing the checks is much more
expensive that doing the sorting. For this reason we choose to use sorting algorithms
that are can't be broken by invalid comprators. peeksort (mergesort)
doesn't misbehave when any of the strict weak order properties is
violated - output is always a reordering of the input. */
peeksort(list.begin(), list.end(), comparator);
v.mkList(list);
}
@@ -3855,6 +3885,32 @@ static RegisterPrimOp primop_sort({
This is a stable sort: it preserves the relative order of elements
deemed equal by the comparator.
*comparator* must impose a strict weak ordering on the set of values
in the *list*. This means that for any elements *a*, *b* and *c* from the
*list*, *comparator* must satisfy the following relations:
1. Transitivity
```nix
comparator a b && comparator b c -> comparator a c
```
1. Irreflexivity
```nix
comparator a a == false
```
1. Transitivity of equivalence
```nix
let equiv = a: b: (!comparator a b && !comparator b a); in
equiv a b && equiv b c -> equiv a c
```
If the *comparator* violates any of these properties, then `builtins.sort`
reorders elements in an unspecified manner.
)",
.fun = prim_sort,
});
@@ -3868,8 +3924,8 @@ static void prim_partition(EvalState & state, const PosIdx pos, Value ** args, V
ValueVector right, wrong;
for (unsigned int n = 0; n < len; ++n) {
auto vElem = args[1]->listElems()[n];
for (size_t n = 0; n < len; ++n) {
auto vElem = args[1]->listView()[n];
state.forceValue(*vElem, pos);
Value res;
state.callFunction(*args[0], *vElem, res, pos);
@@ -3927,7 +3983,7 @@ static void prim_groupBy(EvalState & state, const PosIdx pos, Value ** args, Val
ValueVectorMap attrs;
for (auto vElem : args[1]->listItems()) {
for (auto vElem : args[1]->listView()) {
Value res;
state.callFunction(*args[0], *vElem, res, pos);
auto name = state.forceStringNoCtx(
@@ -3983,8 +4039,8 @@ static void prim_concatMap(EvalState & state, const PosIdx pos, Value ** args, V
SmallTemporaryValueVector<conservativeStackReservation> lists(nrLists);
size_t len = 0;
for (unsigned int n = 0; n < nrLists; ++n) {
Value * vElem = args[1]->listElems()[n];
for (size_t n = 0; n < nrLists; ++n) {
Value * vElem = args[1]->listView()[n];
state.callFunction(*args[0], *vElem, lists[n], pos);
state.forceList(
lists[n],
@@ -3995,10 +4051,11 @@ static void prim_concatMap(EvalState & state, const PosIdx pos, Value ** args, V
auto list = state.buildList(len);
auto out = list.elems;
for (unsigned int n = 0, pos = 0; n < nrLists; ++n) {
auto l = lists[n].listSize();
for (size_t n = 0, pos = 0; n < nrLists; ++n) {
auto listView = lists[n].listView();
auto l = listView.size();
if (l)
memcpy(out + pos, lists[n].listElems(), l * sizeof(Value *));
memcpy(out + pos, listView.data(), l * sizeof(Value *));
pos += l;
}
v.mkList(list);
@@ -4206,9 +4263,9 @@ static RegisterPrimOp primop_lessThan({
.name = "__lessThan",
.args = {"e1", "e2"},
.doc = R"(
Return `true` if the number *e1* is less than the number *e2*, and
`false` otherwise. Evaluation aborts if either *e1* or *e2* does not
evaluate to a number.
Return `true` if the value *e1* is less than the value *e2*, and `false` otherwise.
Evaluation aborts if either *e1* or *e2* does not evaluate to a number, string or path.
Furthermore, it aborts if *e2* does not match *e1*'s type according to the aforementioned classification of number, string or path.
)",
.fun = prim_lessThan,
});
@@ -4258,6 +4315,7 @@ static RegisterPrimOp primop_toString({
non-negative. */
static void prim_substring(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
using NixUInt = std::make_unsigned_t<NixInt::Inner>;
NixInt::Inner start =
state
.forceInt(
@@ -4277,12 +4335,10 @@ static void prim_substring(EvalState & state, const PosIdx pos, Value ** args, V
// Negative length may be idiomatically passed to builtins.substring to get
// the tail of the string.
if (len < 0) {
len = std::numeric_limits<NixInt::Inner>::max();
}
auto _len = std::numeric_limits<std::string::size_type>::max();
// Special-case on empty substring to avoid O(n) strlen
// This allows for the use of empty substrings to efficently capture string context
// This allows for the use of empty substrings to efficiently capture string context
if (len == 0) {
state.forceValue(*args[2], pos);
if (args[2]->type() == nString) {
@@ -4291,11 +4347,15 @@ static void prim_substring(EvalState & state, const PosIdx pos, Value ** args, V
}
}
if (len >= 0 && NixUInt(len) < _len) {
_len = len;
}
NixStringContext context;
auto s = state.coerceToString(
pos, *args[2], context, "while evaluating the third argument (the string) passed to builtins.substring");
v.mkString((unsigned int) start >= s->size() ? "" : s->substr(start, len), context);
v.mkString(NixUInt(start) >= s->size() ? "" : s->substr(start, _len), context);
}
static RegisterPrimOp primop_substring({
@@ -4369,7 +4429,7 @@ static void prim_convertHash(EvalState & state, const PosIdx pos, Value ** args,
state.forceAttrs(*args[0], pos, "while evaluating the first argument passed to builtins.convertHash");
auto inputAttrs = args[0]->attrs();
auto iteratorHash = getAttr(state, state.symbols.create("hash"), inputAttrs, "while locating the attribute 'hash'");
auto iteratorHash = state.getAttr(state.symbols.create("hash"), inputAttrs, "while locating the attribute 'hash'");
auto hash = state.forceStringNoCtx(*iteratorHash->value, pos, "while evaluating the attribute 'hash'");
auto iteratorHashAlgo = inputAttrs->get(state.symbols.create("hashAlgo"));
@@ -4378,8 +4438,8 @@ static void prim_convertHash(EvalState & state, const PosIdx pos, Value ** args,
ha = parseHashAlgo(
state.forceStringNoCtx(*iteratorHashAlgo->value, pos, "while evaluating the attribute 'hashAlgo'"));
auto iteratorToHashFormat = getAttr(
state, state.symbols.create("toHashFormat"), args[0]->attrs(), "while locating the attribute 'toHashFormat'");
auto iteratorToHashFormat = state.getAttr(
state.symbols.create("toHashFormat"), args[0]->attrs(), "while locating the attribute 'toHashFormat'");
HashFormat hf = parseHashFormat(
state.forceStringNoCtx(*iteratorToHashFormat->value, pos, "while evaluating the attribute 'toHashFormat'"));
@@ -4603,7 +4663,7 @@ void prim_split(EvalState & state, const PosIdx pos, Value ** args, Value & v)
// Add a list for matched substrings.
const size_t slen = match.size() - 1;
// Start at 1, beacause the first match is the whole string.
// Start at 1, because the first match is the whole string.
auto list2 = state.buildList(slen);
for (const auto & [si, v2] : enumerate(list2)) {
if (!match[si + 1].matched)
@@ -4687,7 +4747,7 @@ static void prim_concatStringsSep(EvalState & state, const PosIdx pos, Value **
res.reserve((args[1]->listSize() + 32) * sep.size());
bool first = true;
for (auto elem : args[1]->listItems()) {
for (auto elem : args[1]->listView()) {
if (first)
first = false;
else
@@ -4722,14 +4782,14 @@ static void prim_replaceStrings(EvalState & state, const PosIdx pos, Value ** ar
.atPos(pos)
.debugThrow();
std::vector<std::string> from;
std::vector<std::string_view> from;
from.reserve(args[0]->listSize());
for (auto elem : args[0]->listItems())
for (auto elem : args[0]->listView())
from.emplace_back(state.forceString(
*elem, pos, "while evaluating one of the strings to replace passed to builtins.replaceStrings"));
std::unordered_map<size_t, std::string> cache;
auto to = args[1]->listItems();
std::unordered_map<size_t, std::string_view> cache;
auto to = args[1]->listView();
NixStringContext context;
auto s = state.forceString(
@@ -4986,7 +5046,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings)
.type = nInt,
.doc = R"(
Return the [Unix time](https://en.wikipedia.org/wiki/Unix_time) at first evaluation.
Repeated references to that name will re-use the initially obtained value.
Repeated references to that name re-use the initially obtained value.
Example:
@@ -5001,7 +5061,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings)
1683705525
```
The [store path](@docroot@/store/store-path.md) of a derivation depending on `currentTime` will differ for each evaluation, unless both evaluate `builtins.currentTime` in the same second.
The [store path](@docroot@/store/store-path.md) of a derivation depending on `currentTime` differs for each evaluation, unless both evaluate `builtins.currentTime` in the same second.
)",
.impureOnly = true,
});
@@ -5195,7 +5255,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings)
/* Now that we've added all primops, sort the `builtins' set,
because attribute lookups expect it to be sorted. */
getBuiltins().payload.attrs->sort();
const_cast<Bindings *>(getBuiltins().attrs())->sort();
staticBaseEnv->sort();
+2 -2
View File
@@ -235,7 +235,7 @@ static RegisterPrimOp primop_getContext(
The string context tracks references to derivations within a string.
It is represented as an attribute set of [store derivation](@docroot@/glossary.md#gloss-store-derivation) paths mapping to output names.
Using [string interpolation](@docroot@/language/string-interpolation.md) on a derivation will add that derivation to the string context.
Using [string interpolation](@docroot@/language/string-interpolation.md) on a derivation adds that derivation to the string context.
For example,
```nix
@@ -308,7 +308,7 @@ static void prim_appendContext(EvalState & state, const PosIdx pos, Value ** arg
.atPos(i.pos)
.debugThrow();
}
for (auto elem : attr->value->listItems()) {
for (auto elem : attr->value->listView()) {
auto outputName =
state.forceStringNoCtx(*elem, attr->pos, "while evaluating an output name within a string context");
context.emplace(
+2 -2
View File
@@ -131,7 +131,7 @@ static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value ** args
for (auto & attr : *args[0]->attrs()) {
const auto & attrName = state.symbols[attr.name];
auto attrHint = [&]() -> std::string {
return "while evaluating the '" + attrName + "' attribute passed to builtins.fetchClosure";
return fmt("while evaluating the attribute '%s' passed to builtins.fetchClosure", attrName);
};
if (attrName == "fromPath") {
@@ -212,7 +212,7 @@ static RegisterPrimOp primop_fetchClosure({
.doc = R"(
Fetch a store path [closure](@docroot@/glossary.md#gloss-closure) from a binary cache, and return the store path as a string with context.
This function can be invoked in three ways, that we will discuss in order of preference.
This function can be invoked in three ways that we will discuss in order of preference.
**Fetch a content-addressed store path**
+11 -10
View File
@@ -204,7 +204,7 @@ static void fetchTree(
else
state
.error<EvalError>(
"in pure evaluation mode, '%s' will not fetch unlocked input '%s'", fetcher, input.to_string())
"in pure evaluation mode, '%s' doesn't fetch unlocked input '%s'", fetcher, input.to_string())
.atPos(pos)
.debugThrow();
}
@@ -257,7 +257,7 @@ static RegisterPrimOp primop_fetchTree({
That is, `fetchTree` is idempotent.
Downloads are cached in `$XDG_CACHE_HOME/nix`.
The remote source will be fetched from the network if both are true:
The remote source is fetched from the network if both are true:
- A NAR hash is supplied and the corresponding store path is not [valid](@docroot@/glossary.md#gloss-validity), that is, not available in the store
> **Note**
@@ -319,7 +319,7 @@ static RegisterPrimOp primop_fetchTree({
- `"tarball"`
Download a tar archive and extract it into the Nix store.
This has the same underyling implementation as [`builtins.fetchTarball`](@docroot@/language/builtins.md#builtins-fetchTarball)
This has the same underlying implementation as [`builtins.fetchTarball`](@docroot@/language/builtins.md#builtins-fetchTarball)
- `url` (String, required)
@@ -352,7 +352,7 @@ static RegisterPrimOp primop_fetchTree({
> **Note**
>
> If the URL points to a local directory, and no `ref` or `rev` is given, Nix will only consider files added to the Git index, as listed by `git ls-files` but use the *current file contents* of the Git working directory.
> If the URL points to a local directory, and no `ref` or `rev` is given, Nix only considers files added to the Git index, as listed by `git ls-files` but use the *current file contents* of the Git working directory.
- `ref` (String, optional)
@@ -570,11 +570,12 @@ static void fetch(
// TODO: fetching may fail, yet the path may be substitutable.
// https://github.com/NixOS/nix/issues/4313
auto storePath = unpack ? fetchToStore(
state.fetchSettings,
*state.store,
fetchers::downloadTarball(state.store, state.fetchSettings, *url),
FetchMode::Copy,
name)
: fetchers::downloadFile(state.store, *url, name).storePath;
: fetchers::downloadFile(state.store, state.fetchSettings, *url, name).storePath;
if (expectedHash) {
auto hash = unpack ? state.store->queryPathInfo(storePath)->narHash
@@ -709,7 +710,7 @@ static RegisterPrimOp primop_fetchGit({
This option has no effect once `shallow` cloning is enabled.
By default, the `ref` value is prefixed with `refs/heads/`.
As of 2.3.0, Nix will not prefix `refs/heads/` if `ref` starts with `refs/`.
As of 2.3.0, Nix doesn't prefix `refs/heads/` if `ref` starts with `refs/`.
- `submodules` (default: `false`)
@@ -868,7 +869,7 @@ static RegisterPrimOp primop_fetchGit({
}
```
Nix will refetch the branch according to the [`tarball-ttl`](@docroot@/command-ref/conf-file.md#conf-tarball-ttl) setting.
Nix refetches the branch according to the [`tarball-ttl`](@docroot@/command-ref/conf-file.md#conf-tarball-ttl) setting.
This behavior is disabled in [pure evaluation mode](@docroot@/command-ref/conf-file.md#conf-pure-eval).
@@ -879,9 +880,9 @@ static RegisterPrimOp primop_fetchGit({
```
If the URL points to a local directory, and no `ref` or `rev` is
given, `fetchGit` will use the current content of the checked-out
files, even if they are not committed or added to Git's index. It will
only consider files added to the Git repository, as listed by `git ls-files`.
given, `fetchGit` uses the current content of the checked-out
files, even if they are not committed or added to Git's index. It
only considers files added to the Git repository, as listed by `git ls-files`.
)",
.fun = prim_fetchGit,
});
+22 -97
View File
@@ -1,140 +1,73 @@
#include "nix/expr/primops.hh"
#include "nix/expr/eval-inline.hh"
#include "expr-config-private.hh"
#include <sstream>
#include <toml.hpp>
namespace nix {
#if HAVE_TOML11_4
/**
* This is what toml11 < 4.0 did when choosing the subsecond precision.
* TOML 1.0.0 spec doesn't define how sub-millisecond ranges should be handled and calls it
* implementation defined behavior. For a lack of a better choice we stick with what older versions
* of toml11 did [1].
*
* [1]: https://github.com/ToruNiina/toml11/blob/dcfe39a783a94e8d52c885e5883a6fbb21529019/toml/datetime.hpp#L282
*/
static size_t normalizeSubsecondPrecision(toml::local_time lt)
{
auto millis = lt.millisecond;
auto micros = lt.microsecond;
auto nanos = lt.nanosecond;
if (millis != 0 || micros != 0 || nanos != 0) {
if (micros != 0 || nanos != 0) {
if (nanos != 0)
return 9;
return 6;
}
return 3;
}
return 0;
}
/**
* Normalize date/time formats to serialize to the same strings as versions prior to toml11 4.0.
*
* Several things to consider:
*
* 1. Sub-millisecond range is represented the same way as in toml11 versions prior to 4.0. Precisioun is rounded
* towards the next multiple of 3 or capped at 9 digits.
* 2. Seconds must be specified. This may become optional in (yet unreleased) TOML 1.1.0, but 1.0.0 defined local time
* in terms of RFC3339 [1].
* 3. date-time separator (`t`, `T` or space ` `) is canonicalized to an upper T. This is compliant with RFC3339
* [1] 5.6:
* > Applications that generate this format SHOULD use upper case letters.
*
* [1]: https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
*/
static void normalizeDatetimeFormat(toml::value & t)
{
if (t.is_local_datetime()) {
auto & ldt = t.as_local_datetime();
t.as_local_datetime_fmt() = {
.delimiter = toml::datetime_delimiter_kind::upper_T,
// https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
.has_seconds = true, // Mandated by TOML 1.0.0
.subsecond_precision = normalizeSubsecondPrecision(ldt.time),
};
return;
}
if (t.is_offset_datetime()) {
auto & odt = t.as_offset_datetime();
t.as_offset_datetime_fmt() = {
.delimiter = toml::datetime_delimiter_kind::upper_T,
// https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
.has_seconds = true, // Mandated by TOML 1.0.0
.subsecond_precision = normalizeSubsecondPrecision(odt.time),
};
return;
}
if (t.is_local_time()) {
auto & lt = t.as_local_time();
t.as_local_time_fmt() = {
.has_seconds = true, // Mandated by TOML 1.0.0
.subsecond_precision = normalizeSubsecondPrecision(lt),
};
return;
}
}
#endif
static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Value & val)
{
auto toml = state.forceStringNoCtx(*args[0], pos, "while evaluating the argument passed to builtins.fromTOML");
std::istringstream tomlStream(std::string{toml});
auto visit = [&](auto & self, Value & v, toml::value t) -> void {
std::function<void(Value &, toml::value)> visit;
visit = [&](Value & v, toml::value t) {
switch (t.type()) {
case toml::value_t::table: {
auto table = toml::get<toml::table>(t);
auto attrs = state.buildBindings(table.size());
size_t size = 0;
for (auto & i : table) {
(void) i;
size++;
}
auto attrs = state.buildBindings(size);
for (auto & elem : table) {
forceNoNullByte(elem.first);
self(self, attrs.alloc(elem.first), elem.second);
visit(attrs.alloc(elem.first), elem.second);
}
v.mkAttrs(attrs);
} break;
;
case toml::value_t::array: {
auto array = toml::get<std::vector<toml::value>>(t);
auto list = state.buildList(array.size());
for (const auto & [n, v] : enumerate(list))
self(self, *(v = state.allocValue()), array[n]);
visit(*(v = state.allocValue()), array[n]);
v.mkList(list);
} break;
;
case toml::value_t::boolean:
v.mkBool(toml::get<bool>(t));
break;
;
case toml::value_t::integer:
v.mkInt(toml::get<int64_t>(t));
break;
;
case toml::value_t::floating:
v.mkFloat(toml::get<NixFloat>(t));
break;
;
case toml::value_t::string: {
auto s = toml::get<std::string_view>(t);
forceNoNullByte(s);
v.mkString(s);
} break;
;
case toml::value_t::local_datetime:
case toml::value_t::offset_datetime:
case toml::value_t::local_date:
case toml::value_t::local_time: {
if (experimentalFeatureSettings.isEnabled(Xp::ParseTomlTimestamps)) {
#if HAVE_TOML11_4
normalizeDatetimeFormat(t);
#endif
auto attrs = state.buildBindings(2);
attrs.alloc("_type").mkString("timestamp");
std::ostringstream s;
@@ -147,24 +80,16 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Va
throw std::runtime_error("Dates and times are not supported");
}
} break;
;
case toml::value_t::empty:
v.mkNull();
break;
;
}
};
try {
visit(
visit,
val,
toml::parse(
tomlStream,
"fromTOML" /* the "filename" */
#if HAVE_TOML11_4
,
toml::spec::v(1, 0, 0) // Be explicit that we are parsing TOML 1.0.0 without extensions
#endif
));
visit(val, toml::parse(tomlStream, "fromTOML" /* the "filename" */));
} catch (std::exception & e) { // TODO: toml::syntax_error
state.error<EvalError>("while parsing TOML: %s", e.what()).atPos(pos).debugThrow();
}
+1 -1
View File
@@ -1,6 +1,6 @@
generated_headers += gen_header.process(
'derivation.nix',
preserve_path_from: meson.project_source_root(),
preserve_path_from : meson.project_source_root(),
)
sources += files(
+4 -2
View File
@@ -46,11 +46,13 @@ void printAmbiguous(
break;
}
case nList:
if (seen && v.listSize() && !seen->insert(v.listElems()).second)
/* Use pointer to the Value instead of pointer to the elements, because
that would need to explicitly handle the case of SmallList. */
if (seen && v.listSize() && !seen->insert(&v).second)
str << "«repeated»";
else {
str << "[ ";
for (auto v2 : v.listItems()) {
for (auto v2 : v.listView()) {
if (v2)
printAmbiguous(*v2, symbols, str, seen, depth - 1);
else
+6 -6
View File
@@ -415,8 +415,8 @@ private:
if (depth < options.maxDepth) {
increaseIndent();
output << "[";
auto listItems = v.listItems();
auto prettyPrint = shouldPrettyPrintList(listItems);
auto listItems = v.listView();
auto prettyPrint = shouldPrettyPrintList(listItems.span());
size_t currentListItemsPrinted = 0;
@@ -453,13 +453,13 @@ private:
if (v.isLambda()) {
output << "lambda";
if (v.payload.lambda.fun) {
if (v.payload.lambda.fun->name) {
output << " " << state.symbols[v.payload.lambda.fun->name];
if (v.lambda().fun) {
if (v.lambda().fun->name) {
output << " " << state.symbols[v.lambda().fun->name];
}
std::ostringstream s;
s << state.positions[v.payload.lambda.fun->pos];
s << state.positions[v.lambda().fun->pos];
output << " @ " << filterANSIEscapes(toView(s));
}
} else if (v.isPrimOp()) {
+1 -1
View File
@@ -75,7 +75,7 @@ json printValueAsJSON(
case nList: {
out = json::array();
int i = 0;
for (auto elem : v.listItems()) {
for (auto elem : v.listView()) {
try {
out.push_back(printValueAsJSON(state, strict, *elem, pos, context, copyToStore));
} catch (Error & e) {
+8 -8
View File
@@ -129,7 +129,7 @@ static void printValueAsXML(
case nList: {
XMLOpenElement _(doc, "list");
for (auto v2 : v.listItems())
for (auto v2 : v.listView())
printValueAsXML(state, strict, location, *v2, doc, context, drvsSeen, pos);
break;
}
@@ -142,20 +142,20 @@ static void printValueAsXML(
}
XMLAttrs xmlAttrs;
if (location)
posToXML(state, xmlAttrs, state.positions[v.payload.lambda.fun->pos]);
posToXML(state, xmlAttrs, state.positions[v.lambda().fun->pos]);
XMLOpenElement _(doc, "function", xmlAttrs);
if (v.payload.lambda.fun->hasFormals()) {
if (v.lambda().fun->hasFormals()) {
XMLAttrs attrs;
if (v.payload.lambda.fun->arg)
attrs["name"] = state.symbols[v.payload.lambda.fun->arg];
if (v.payload.lambda.fun->formals->ellipsis)
if (v.lambda().fun->arg)
attrs["name"] = state.symbols[v.lambda().fun->arg];
if (v.lambda().fun->formals->ellipsis)
attrs["ellipsis"] = "1";
XMLOpenElement _(doc, "attrspat", attrs);
for (auto & i : v.payload.lambda.fun->formals->lexicographicOrder(state.symbols))
for (auto & i : v.lambda().fun->formals->lexicographicOrder(state.symbols))
doc.writeEmptyElement("attr", singletonAttrs("name", state.symbols[i.name]));
} else
doc.writeEmptyElement("varpat", singletonAttrs("name", state.symbols[v.payload.lambda.fun->arg]));
doc.writeEmptyElement("varpat", singletonAttrs("name", state.symbols[v.lambda().fun->arg]));
break;
}
+2 -2
View File
@@ -93,9 +93,9 @@ bool getBoolAttr(const Attrs & attrs, const std::string & name)
return *s;
}
std::map<std::string, std::string> attrsToQuery(const Attrs & attrs)
StringMap attrsToQuery(const Attrs & attrs)
{
std::map<std::string, std::string> query;
StringMap query;
for (auto & attr : attrs) {
if (auto v = std::get_if<uint64_t>(&attr.second)) {
query.insert_or_assign(attr.first, fmt("%d", *v));
+6 -3
View File
@@ -1,4 +1,5 @@
#include "nix/fetchers/cache.hh"
#include "nix/fetchers/fetch-settings.hh"
#include "nix/util/users.hh"
#include "nix/store/sqlite.hh"
#include "nix/util/sync.hh"
@@ -148,10 +149,12 @@ struct CacheImpl : Cache
}
};
ref<Cache> getCache()
ref<Cache> Settings::getCache() const
{
static auto cache = std::make_shared<CacheImpl>();
return ref<Cache>(cache);
auto cache(_cache.lock());
if (!*cache)
*cache = std::make_shared<CacheImpl>();
return ref<Cache>(*cache);
}
} // namespace nix::fetchers
+4 -2
View File
@@ -1,5 +1,6 @@
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/fetchers/fetch-settings.hh"
namespace nix {
@@ -12,6 +13,7 @@ fetchers::Cache::Key makeFetchToStoreCacheKey(
}
StorePath fetchToStore(
const fetchers::Settings & settings,
Store & store,
const SourcePath & path,
FetchMode mode,
@@ -27,7 +29,7 @@ StorePath fetchToStore(
if (!filter && path.accessor->fingerprint) {
cacheKey = makeFetchToStoreCacheKey(std::string{name}, *path.accessor->fingerprint, method, path.path.abs());
if (auto res = fetchers::getCache()->lookupStorePath(*cacheKey, store)) {
if (auto res = settings.getCache()->lookupStorePath(*cacheKey, store)) {
debug("store path cache hit for '%s'", path);
return res->storePath;
}
@@ -49,7 +51,7 @@ StorePath fetchToStore(
debug(mode == FetchMode::DryRun ? "hashed '%s'" : "copied '%s' to '%s'", path, store.printStorePath(storePath));
if (cacheKey && mode == FetchMode::Copy)
fetchers::getCache()->upsert(*cacheKey, store, {}, storePath);
settings.getCache()->upsert(*cacheKey, store, {}, storePath);
return storePath;
}
+3 -2
View File
@@ -134,7 +134,7 @@ ParsedURL Input::toURL() const
return scheme->toURL(*this);
}
std::string Input::toURLString(const std::map<std::string, std::string> & extraQuery) const
std::string Input::toURLString(const StringMap & extraQuery) const
{
auto url = toURL();
for (auto & attr : extraQuery)
@@ -200,7 +200,8 @@ std::pair<StorePath, Input> Input::fetchToStore(ref<Store> store) const
try {
auto [accessor, result] = getAccessorUnchecked(store);
auto storePath = nix::fetchToStore(*store, SourcePath(accessor), FetchMode::Copy, result.getName());
auto storePath =
nix::fetchToStore(*settings, *store, SourcePath(accessor), FetchMode::Copy, result.getName());
auto narHash = store->queryPathInfo(storePath)->narHash;
result.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
+5 -4
View File
@@ -1,4 +1,5 @@
#include "nix/fetchers/filtering-source-accessor.hh"
#include "nix/util/sync.hh"
namespace nix {
@@ -56,8 +57,8 @@ void FilteringSourceAccessor::checkAccess(const CanonPath & path)
struct AllowListSourceAccessorImpl : AllowListSourceAccessor
{
std::set<CanonPath> allowedPrefixes;
std::unordered_set<CanonPath> allowedPaths;
SharedSync<std::set<CanonPath>> allowedPrefixes;
SharedSync<std::unordered_set<CanonPath>> allowedPaths;
AllowListSourceAccessorImpl(
ref<SourceAccessor> next,
@@ -72,12 +73,12 @@ struct AllowListSourceAccessorImpl : AllowListSourceAccessor
bool isAllowed(const CanonPath & path) override
{
return allowedPaths.contains(path) || path.isAllowed(allowedPrefixes);
return allowedPaths.readLock()->contains(path) || path.isAllowed(*allowedPrefixes.readLock());
}
void allowPrefix(CanonPath prefix) override
{
allowedPrefixes.insert(std::move(prefix));
allowedPrefixes.lock()->insert(std::move(prefix));
}
};
+73 -46
View File
@@ -1,6 +1,7 @@
#include "nix/fetchers/git-utils.hh"
#include "nix/fetchers/git-lfs-fetch.hh"
#include "nix/fetchers/cache.hh"
#include "nix/fetchers/fetch-settings.hh"
#include "nix/util/finally.hh"
#include "nix/util/processes.hh"
#include "nix/util/signals.hh"
@@ -321,11 +322,16 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
for (size_t n = 0; n < git_commit_parentcount(commit->get()); ++n) {
git_commit * parent;
if (git_commit_parent(&parent, commit->get(), n))
if (git_commit_parent(&parent, commit->get(), n)) {
throw Error(
"getting parent of Git commit '%s': %s",
"Failed to retrieve the parent of Git commit '%s': %s. "
"This may be due to an incomplete repository history. "
"To resolve this, either enable the shallow parameter in your flake URL (?shallow=1) "
"or add set the shallow parameter to true in builtins.fetchGit, "
"or fetch the complete history for this branch.",
*git_commit_id(commit->get()),
git_error_last()->message);
}
todo.push(Commit(parent));
}
}
@@ -370,7 +376,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
if (git_config_iterator_glob_new(Setter(it), config.get(), "^submodule\\..*\\.(path|url|branch)$"))
throw Error("iterating over .gitmodules: %s", git_error_last()->message);
std::map<std::string, std::string> entries;
StringMap entries;
while (true) {
git_config_entry * entry = nullptr;
@@ -534,7 +540,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
append(gitArgs, {"--depth", "1"});
append(gitArgs, {std::string("--"), url, refspec});
runProgram(
auto [status, output] = runProgram(
RunOptions{
.program = "git",
.lookupPath = true,
@@ -542,7 +548,12 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
// we're using --quiet for now. Should process its stderr.
.args = gitArgs,
.input = {},
.mergeStderrToStdout = true,
.isInteractive = true});
if (status > 0) {
throw Error("Failed to fetch git repository %s : %s", url, output);
}
}
void verifyCommit(const Hash & rev, const std::vector<fetchers::PublicKey> & publicKeys) override
@@ -582,7 +593,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
});
/* Evaluate result through status code and checking if public
key fingerprints appear on stderr. This is neccessary
key fingerprints appear on stderr. This is necessary
because the git command might also succeed due to the
commit being signed by gpg keys that are present in the
users key agent. */
@@ -607,19 +618,18 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
throw Error("Commit signature verification on commit %s failed: %s", rev.gitRev(), output);
}
Hash treeHashToNarHash(const Hash & treeHash) override
Hash treeHashToNarHash(const fetchers::Settings & settings, const Hash & treeHash) override
{
auto accessor = getAccessor(treeHash, false, "");
fetchers::Cache::Key cacheKey{"treeHashToNarHash", {{"treeHash", treeHash.gitRev()}}};
if (auto res = fetchers::getCache()->lookup(cacheKey))
if (auto res = settings.getCache()->lookup(cacheKey))
return Hash::parseAny(fetchers::getStrAttr(*res, "narHash"), HashAlgorithm::SHA256);
auto narHash = accessor->hashPath(CanonPath::root);
fetchers::getCache()->upsert(
cacheKey, fetchers::Attrs({{"narHash", narHash.to_string(HashFormat::SRI, true)}}));
settings.getCache()->upsert(cacheKey, fetchers::Attrs({{"narHash", narHash.to_string(HashFormat::SRI, true)}}));
return narHash;
}
@@ -653,29 +663,39 @@ ref<GitRepo> GitRepo::openRepo(const std::filesystem::path & path, bool create,
struct GitSourceAccessor : SourceAccessor
{
ref<GitRepoImpl> repo;
Object root;
std::optional<lfs::Fetch> lfsFetch = std::nullopt;
struct State
{
ref<GitRepoImpl> repo;
Object root;
std::optional<lfs::Fetch> lfsFetch = std::nullopt;
};
Sync<State> state_;
GitSourceAccessor(ref<GitRepoImpl> repo_, const Hash & rev, bool smudgeLfs)
: repo(repo_)
, root(peelToTreeOrBlob(lookupObject(*repo, hashToOID(rev)).get()))
: state_{State{
.repo = repo_,
.root = peelToTreeOrBlob(lookupObject(*repo_, hashToOID(rev)).get()),
.lfsFetch = smudgeLfs ? std::make_optional(lfs::Fetch(*repo_, hashToOID(rev))) : std::nullopt,
}}
{
if (smudgeLfs)
lfsFetch = std::make_optional(lfs::Fetch(*repo, hashToOID(rev)));
}
std::string readBlob(const CanonPath & path, bool symlink)
{
const auto blob = getBlob(path, symlink);
auto state(state_.lock());
if (lfsFetch) {
if (lfsFetch->shouldFetch(path)) {
const auto blob = getBlob(*state, path, symlink);
if (state->lfsFetch) {
if (state->lfsFetch->shouldFetch(path)) {
StringSink s;
try {
// FIXME: do we need to hold the state lock while
// doing this?
auto contents =
std::string((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
lfsFetch->fetch(contents, path, s, [&s](uint64_t size) { s.s.reserve(size); });
state->lfsFetch->fetch(contents, path, s, [&s](uint64_t size) { s.s.reserve(size); });
} catch (Error & e) {
e.addTrace({}, "while smudging git-lfs file '%s'", path);
throw;
@@ -694,15 +714,18 @@ struct GitSourceAccessor : SourceAccessor
bool pathExists(const CanonPath & path) override
{
return path.isRoot() ? true : (bool) lookup(path);
auto state(state_.lock());
return path.isRoot() ? true : (bool) lookup(*state, path);
}
std::optional<Stat> maybeLstat(const CanonPath & path) override
{
if (path.isRoot())
return Stat{.type = git_object_type(root.get()) == GIT_OBJECT_TREE ? tDirectory : tRegular};
auto state(state_.lock());
auto entry = lookup(path);
if (path.isRoot())
return Stat{.type = git_object_type(state->root.get()) == GIT_OBJECT_TREE ? tDirectory : tRegular};
auto entry = lookup(*state, path);
if (!entry)
return std::nullopt;
@@ -730,6 +753,8 @@ struct GitSourceAccessor : SourceAccessor
DirEntries readDirectory(const CanonPath & path) override
{
auto state(state_.lock());
return std::visit(
overloaded{
[&](Tree tree) {
@@ -746,7 +771,7 @@ struct GitSourceAccessor : SourceAccessor
return res;
},
[&](Submodule) { return DirEntries(); }},
getTree(path));
getTree(*state, path));
}
std::string readLink(const CanonPath & path) override
@@ -760,7 +785,9 @@ struct GitSourceAccessor : SourceAccessor
*/
std::optional<Hash> getSubmoduleRev(const CanonPath & path)
{
auto entry = lookup(path);
auto state(state_.lock());
auto entry = lookup(*state, path);
if (!entry || git_tree_entry_type(entry) != GIT_OBJECT_COMMIT)
return std::nullopt;
@@ -771,7 +798,7 @@ struct GitSourceAccessor : SourceAccessor
std::unordered_map<CanonPath, TreeEntry> lookupCache;
/* Recursively look up 'path' relative to the root. */
git_tree_entry * lookup(const CanonPath & path)
git_tree_entry * lookup(State & state, const CanonPath & path)
{
auto i = lookupCache.find(path);
if (i != lookupCache.end())
@@ -783,7 +810,7 @@ struct GitSourceAccessor : SourceAccessor
auto name = path.baseName().value();
auto parentTree = lookupTree(*parent);
auto parentTree = lookupTree(state, *parent);
if (!parentTree)
return nullptr;
@@ -813,29 +840,29 @@ struct GitSourceAccessor : SourceAccessor
return res;
}
std::optional<Tree> lookupTree(const CanonPath & path)
std::optional<Tree> lookupTree(State & state, const CanonPath & path)
{
if (path.isRoot()) {
if (git_object_type(root.get()) == GIT_OBJECT_TREE)
return dupObject<Tree>((git_tree *) &*root);
if (git_object_type(state.root.get()) == GIT_OBJECT_TREE)
return dupObject<Tree>((git_tree *) &*state.root);
else
return std::nullopt;
}
auto entry = lookup(path);
auto entry = lookup(state, path);
if (!entry || git_tree_entry_type(entry) != GIT_OBJECT_TREE)
return std::nullopt;
Tree tree;
if (git_tree_entry_to_object((git_object **) (git_tree **) Setter(tree), *repo, entry))
if (git_tree_entry_to_object((git_object **) (git_tree **) Setter(tree), *state.repo, entry))
throw Error("looking up directory '%s': %s", showPath(path), git_error_last()->message);
return tree;
}
git_tree_entry * need(const CanonPath & path)
git_tree_entry * need(State & state, const CanonPath & path)
{
auto entry = lookup(path);
auto entry = lookup(state, path);
if (!entry)
throw Error("'%s' does not exist", showPath(path));
return entry;
@@ -844,16 +871,16 @@ struct GitSourceAccessor : SourceAccessor
struct Submodule
{};
std::variant<Tree, Submodule> getTree(const CanonPath & path)
std::variant<Tree, Submodule> getTree(State & state, const CanonPath & path)
{
if (path.isRoot()) {
if (git_object_type(root.get()) == GIT_OBJECT_TREE)
return dupObject<Tree>((git_tree *) &*root);
if (git_object_type(state.root.get()) == GIT_OBJECT_TREE)
return dupObject<Tree>((git_tree *) &*state.root);
else
throw Error("Git root object '%s' is not a directory", *git_object_id(root.get()));
throw Error("Git root object '%s' is not a directory", *git_object_id(state.root.get()));
}
auto entry = need(path);
auto entry = need(state, path);
if (git_tree_entry_type(entry) == GIT_OBJECT_COMMIT)
return Submodule();
@@ -862,16 +889,16 @@ struct GitSourceAccessor : SourceAccessor
throw Error("'%s' is not a directory", showPath(path));
Tree tree;
if (git_tree_entry_to_object((git_object **) (git_tree **) Setter(tree), *repo, entry))
if (git_tree_entry_to_object((git_object **) (git_tree **) Setter(tree), *state.repo, entry))
throw Error("looking up directory '%s': %s", showPath(path), git_error_last()->message);
return tree;
}
Blob getBlob(const CanonPath & path, bool expectSymlink)
Blob getBlob(State & state, const CanonPath & path, bool expectSymlink)
{
if (!expectSymlink && git_object_type(root.get()) == GIT_OBJECT_BLOB)
return dupObject<Blob>((git_blob *) &*root);
if (!expectSymlink && git_object_type(state.root.get()) == GIT_OBJECT_BLOB)
return dupObject<Blob>((git_blob *) &*state.root);
auto notExpected = [&]() {
throw Error(expectSymlink ? "'%s' is not a symlink" : "'%s' is not a regular file", showPath(path));
@@ -880,7 +907,7 @@ struct GitSourceAccessor : SourceAccessor
if (path.isRoot())
notExpected();
auto entry = need(path);
auto entry = need(state, path);
if (git_tree_entry_type(entry) != GIT_OBJECT_BLOB)
notExpected();
@@ -895,7 +922,7 @@ struct GitSourceAccessor : SourceAccessor
}
Blob blob;
if (git_tree_entry_to_object((git_object **) (git_blob **) Setter(blob), *repo, entry))
if (git_tree_entry_to_object((git_object **) (git_blob **) Setter(blob), *state.repo, entry))
throw Error("looking up file '%s': %s", showPath(path), git_error_last()->message);
return blob;
+34 -11
View File
@@ -463,7 +463,10 @@ struct GitInputScheme : InputScheme
// repo, treat as a remote URI to force a clone.
static bool forceHttp = getEnv("_NIX_FORCE_HTTP") == "1"; // for testing
auto url = parseURL(getStrAttr(input.attrs, "url"));
bool isBareRepository = url.scheme == "file" && !pathExists(url.path + "/.git");
// Why are we checking for bare repository?
// well if it's a bare repository we want to force a git fetch rather than copying the folder
bool isBareRepository = url.scheme == "file" && pathExists(url.path) && !pathExists(url.path + "/.git");
//
// FIXME: here we turn a possibly relative path into an absolute path.
// This allows relative git flake inputs to be resolved against the
@@ -481,6 +484,12 @@ struct GitInputScheme : InputScheme
"See https://github.com/NixOS/nix/issues/12281 for details.",
url);
}
// If we don't check here for the path existence, then we can give libgit2 any directory
// and it will initialize them as git directories.
if (!pathExists(url.path)) {
throw Error("The path '%s' does not exist.", url.path);
}
repoInfo.location = std::filesystem::absolute(url.path);
} else {
if (url.scheme == "file")
@@ -499,11 +508,15 @@ struct GitInputScheme : InputScheme
return repoInfo;
}
uint64_t getLastModified(const RepoInfo & repoInfo, const std::filesystem::path & repoDir, const Hash & rev) const
uint64_t getLastModified(
const Settings & settings,
const RepoInfo & repoInfo,
const std::filesystem::path & repoDir,
const Hash & rev) const
{
Cache::Key key{"gitLastModified", {{"rev", rev.gitRev()}}};
auto cache = getCache();
auto cache = settings.getCache();
if (auto res = cache->lookup(key))
return getIntAttr(*res, "lastModified");
@@ -515,11 +528,15 @@ struct GitInputScheme : InputScheme
return lastModified;
}
uint64_t getRevCount(const RepoInfo & repoInfo, const std::filesystem::path & repoDir, const Hash & rev) const
uint64_t getRevCount(
const Settings & settings,
const RepoInfo & repoInfo,
const std::filesystem::path & repoDir,
const Hash & rev) const
{
Cache::Key key{"gitRevCount", {{"rev", rev.gitRev()}}};
auto cache = getCache();
auto cache = settings.getCache();
if (auto revCountAttrs = cache->lookup(key))
return getIntAttr(*revCountAttrs, "revCount");
@@ -613,7 +630,7 @@ struct GitInputScheme : InputScheme
auto localRefFile = ref.compare(0, 5, "refs/") == 0 ? cacheDir / ref : cacheDir / "refs/heads" / ref;
bool doFetch;
bool doFetch = false;
time_t now = time(0);
/* If a rev was specified, we need to fetch if it's not in the
@@ -692,11 +709,11 @@ struct GitInputScheme : InputScheme
Attrs infoAttrs({
{"rev", rev.gitRev()},
{"lastModified", getLastModified(repoInfo, repoDir, rev)},
{"lastModified", getLastModified(*input.settings, repoInfo, repoDir, rev)},
});
if (!getShallowAttr(input))
infoAttrs.insert_or_assign("revCount", getRevCount(repoInfo, repoDir, rev));
infoAttrs.insert_or_assign("revCount", getRevCount(*input.settings, repoInfo, repoDir, rev));
printTalkative("using revision %s of repo '%s'", rev.gitRev(), repoInfo.locationToArg());
@@ -810,7 +827,10 @@ struct GitInputScheme : InputScheme
auto rev = repoInfo.workdirInfo.headRev.value_or(nullRev);
input.attrs.insert_or_assign("rev", rev.gitRev());
input.attrs.insert_or_assign("revCount", rev == nullRev ? 0 : getRevCount(repoInfo, repoPath, rev));
if (!getShallowAttr(input)) {
input.attrs.insert_or_assign(
"revCount", rev == nullRev ? 0 : getRevCount(*input.settings, repoInfo, repoPath, rev));
}
verifyCommit(input, repo);
} else {
@@ -826,7 +846,9 @@ struct GitInputScheme : InputScheme
input.attrs.insert_or_assign(
"lastModified",
repoInfo.workdirInfo.headRev ? getLastModified(repoInfo, repoPath, *repoInfo.workdirInfo.headRev) : 0);
repoInfo.workdirInfo.headRev
? getLastModified(*input.settings, repoInfo, repoPath, *repoInfo.workdirInfo.headRev)
: 0);
return {accessor, std::move(input)};
}
@@ -887,7 +909,8 @@ struct GitInputScheme : InputScheme
bool isLocked(const Input & input) const override
{
return (bool) input.getRev();
auto rev = input.getRev();
return rev && rev != nullRev;
}
};
Loaded 100 of 334 files, more files were not shown because too many files have changed in this diff. Show more