Compare commits

..
606 Commits
Author SHA1 Message Date
Eelco Dolstra c98026e982 Lazy trees v2
This adds a setting 'lazy-trees' that causes flake inputs to be
"mounted" as virtual filesystems on top of /nix/store as random
"virtual" store paths. Only when the store path is actually used as a
dependency of a store derivation do we materialize ("devirtualize")
the input by copying it to its content-addressed location in the
store.

String contexts determine when devirtualization happens. One wrinkle
is that there are cases where we had store paths without proper
contexts, in particular when the user does `toString <path>` (where
<path> is a source tree in the Nix store) and passes the result to a
derivation. This usage was always broken, since it can result in
derivations that lack correct references. But to ensure that we don't
change evaluation results, we introduce a new type of context that
results in devirtualization but not in store references. We also now
print a warning about this.
2025-05-19 16:45:53 +02:00
Eelco Dolstra 8c10104e9e Merge pull request #13228 from Mic92/json-utils
libutil-tests/json-utils: fix -Werror=sign-compare error
2025-05-19 16:15:09 +02:00
Robert Hensing f18af849fe Merge pull request #13170 from MattSturgeon/fix/call-flake/rel-path
Use correct parent `outPath` for relative path inputs
2025-05-19 14:13:54 +02:00
Jörg Thalheim 1290b7e53d libutil-tests/json-utils: fix -Werror=sign-compare error
I am on a newer different nixpkgs branch, so I am getting this error
2025-05-19 09:26:24 +02:00
Jörg Thalheim e7078d4de1 Merge pull request #13142 from xokdvium/gc-root-boost-regex
libstore: Use `boost::regex` for GC root discovery
2025-05-18 21:44:41 +02:00
Jörg Thalheim 638b7ec6c5 Merge pull request #13219 from xokdvium/eval-profiler
libexpr: Add `EvalProfiler` and use it for `FunctionCallTrace`
2025-05-18 21:35:43 +02:00
Jörg Thalheim f2671886f5 Merge pull request #13223 from NixOS/remove-global-fetcher-cache
Remove global fetcher cache
2025-05-18 15:58:53 +02:00
Sergei Zimmerman fa6f69f9c5 libexpr: Use EvalProfiler for FunctionCallTrace
This wires up the {pre,post}FunctionCallHook machinery
in EvalState::callFunction and migrates FunctionCallTrace
to use the new EvalProfiler mechanisms for tracing.

Note that branches when the hook gets called are marked with [[unlikely]]
as a hint to the compiler that this is not a hot path. For non-tracing
evaluation this should be a 100% predictable branch, so the performance
cost is nonexistent.

Some measurements to prove support this point:

```
nix build .#nix-cli
nix build github:nixos/nix/d692729759e4e370361cc5105fbeb0e33137ca9e#nix-cli --out-link before
```

(Before)

```
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g before/bin/nix eval nixpkgs#gnome --no-eval-cache" --warmup 4
Benchmark 1: GC_INITIAL_HEAP_SIZE=16g before/bin/nix eval nixpkgs#gnome --no-eval-cache
  Time (mean ± σ):      2.517 s ±  0.032 s    [User: 1.464 s, System: 0.476 s]
  Range (min … max):    2.464 s …  2.557 s    10 runs
```

(After)

```
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g result/bin/nix eval nixpkgs#gnome --no-eval-cache" --warmup 4
Benchmark 1: GC_INITIAL_HEAP_SIZE=16g result/bin/nix eval nixpkgs#gnome --no-eval-cache
  Time (mean ± σ):      2.499 s ±  0.022 s    [User: 1.448 s, System: 0.478 s]
  Range (min … max):    2.472 s …  2.537 s    10 runs
```
2025-05-18 11:55:39 +00:00
Sergei Zimmerman 6b4a86a9e3 libexpr: Add EvalProfiler
This patch adds an EvalProfiler and MultiEvalProfiler that can be used
to insert hooks into the evaluation for the purposes of function tracing
(what function-trace currently does) or for flamegraph/tracy profilers.

See the following commits for how this is supposed to be integrated into
the evaluator and performance considerations.
2025-05-18 11:55:37 +00:00
Matt Sturgeon 2922e3082e Add release note for non-flake inputs having sourceInfo 2025-05-18 00:45:33 +01:00
Matt Sturgeon 72232bc28a tests/functional/flakes/non-flake-inputs: Test non-flake inputs having sourceInfo 2025-05-18 00:40:09 +01:00
Robert Hensing eaee0b4740 tests/function/flakes/relative-paths: Test #13164 2025-05-18 00:40:08 +01:00
Matt Sturgeon 46beb9af76 Use correct parent outPath for relative path inputs
Ensure relative path inputs are relative to the parent node's _actual_
`outPath`, instead of the subtly different `sourceInfo.outPath`.

Additionally, non-flake inputs now also have a `sourceInfo` attribute.

This fixes the relationship between `self.outPath` and
`self.sourceInfo.outPath` in some edge cases.

Fixes #13164
2025-05-18 00:38:17 +01:00
Eelco Dolstra efcb9e36a9 Remove global fetcher cache
The cache is now part of fetchers::Settings.
2025-05-17 19:54:32 +02:00
Eelco Dolstra 7d89e46f65 Sync: Support moving out of another Sync 2025-05-17 19:38:36 +02:00
Eelco Dolstra e5e5c819dd Merge pull request #13221 from roberth/doc-rl-docroot
doc: Render verbatim `@docroot@` on contributing page
2025-05-17 19:18:59 +02:00
Robert Hensing 147a34c573 doc: Render verbatim @docroot@ on contributing page
In rendered form:

```diff
-Add references to the manual using ..
+Add references to the manual using [links like this](@docroot@/example.md)
```
2025-05-17 17:50:49 +02:00
Eelco Dolstra 39f9f5d1b6 Merge pull request #13206 from obsidiansystems/goal-init-dedup
Deduplicate the goal creation functions
2025-05-17 09:07:51 +02:00
Jörg Thalheim d692729759 Merge pull request #13212 from pbsds/lexical-illusions
docs: remove repeated "allowedReferences" and other lexical illusion
2025-05-16 16:59:23 +02:00
Jörg Thalheim 041d2374dd Merge pull request #13213 from pbsds/deriving-path-encoding-anchor
docs: fix duplicate anchor
2025-05-16 11:28:31 +02:00
Peder Bergebakken Sundt 4c50cf798e docs: fix duplicate anchor
`#deriving-path-encoding` is defined in two places, I _think_ this is the correct one to change.
2025-05-16 10:27:29 +02:00
Peder Bergebakken Sundt ea5302c4a2 docs: remove lexical illusions detected with write-good
I made this this non-markdown aware tool somewhat behave with some cursed fd+pandoc invocations
2025-05-16 09:52:19 +02:00
Peder Bergebakken Sundt cb16cd707c docs: remove repeated "allowedReferences"
This is what write-good lints as a "lexical illusion"
2025-05-16 09:20:55 +02:00
John Ericson b21fc05047 Merge pull request #13211 from xokdvium/pos-table
libexpr: Actually cache line information in PosTable
2025-05-16 00:37:57 -04:00
Sergei Zimmerman 5ea81f5b8f libexpr: Actually cache line information in PosTable
Previous code had a sneaky bug due to which no caching
actually happened:

```cpp
auto linesForInput = (*lines)[origin->offset];
```

That should have been:
```cpp
auto & linesForInput = (*lines)[origin->offset];
```

See [1].

Now that it also makes sense to make the cache bound in side
in order not to memoize all the sources without freeing any memory.
The default cache size has been chosen somewhat arbitrarily to be ~64k
origins. For reference, 25.05 nixpkgs has ~50k .nix files.

Simple benchmark:

```nix
let
  pkgs = import <nixpkgs> { };
in
builtins.foldl' (acc: el: acc + el.line) 0 (
  builtins.genList (x: builtins.unsafeGetAttrPos "gcc" pkgs) 10000
)
```

(After)

```
$ hyperfine "result/bin/nix eval -f ./test.nix"
Benchmark 1: result/bin/nix eval -f ./test.nix
  Time (mean ± σ):     292.7 ms ±   3.9 ms    [User: 131.0 ms, System: 120.5 ms]
  Range (min … max):   288.1 ms … 300.5 ms    10 runs
```

(Before)

```
hyperfine "nix eval -f ./test.nix"
Benchmark 1: nix eval -f ./test.nix
  Time (mean ± σ):     666.7 ms ±   6.4 ms    [User: 428.3 ms, System: 191.2 ms]
  Range (min … max):   659.7 ms … 681.3 ms    10 runs
```

If the origin happens to be a `all-packages.nix` or similar in size then the
difference is much more dramatic.

[1]: https://www.github.com/lix-project/lix/commit/22e3f0e9875082be7f4eec8e3caeb134a7f1c05f
2025-05-15 23:07:25 +00:00
Sergei Zimmerman 4711720efe libutil: Add LRUCache::getOrNullptr
For heavier objects it doesn't make sense to return
a std::optional with the copy of the data, when it
can be used by const reference.
2025-05-15 22:28:41 +00:00
John Ericson dc1a5135ee Merge pull request #13207 from xokdvium/drop-bear
dev-shell: Drop bear dependency
2025-05-15 17:48:08 -04:00
Sergei Zimmerman 67535263a5 dev-shell: Drop bear dependency
Since the autotools-based build system has been removed
and meson already generates compile database there's no
need to have it in the devshell.
2025-05-15 21:13:13 +00:00
John Ericson 7bd9eef772 Deduplicate the goal creation functions
The weak reference logic is the same in both these cases, and if/when I
get rid `addWantedOutputs`, also in the `DerivationGoal` case.
2025-05-15 16:59:48 -04:00
Jörg Thalheim 0d3750e902 Merge pull request #13202 from NixOS/fix-nix-copy-closure-test
Fix nix-copy-closure VM test
2025-05-15 17:50:12 +02:00
Eelco Dolstra d626348f42 Fix nix-copy-closure VM test
https://hydra.nixos.org/build/297112538
2025-05-15 12:55:08 +02:00
Jörg Thalheim ea09198f93 Merge pull request #13197 from NixOS/bump-nixpkgs
Update Nixpkgs to fix static builds
2025-05-15 12:20:06 +02:00
Eelco Dolstra 0b46025f7a Merge pull request #13196 from Mic92/store-config-base
rename StoreDirConfigItself to StoreDirConfigBase
2025-05-15 12:03:11 +02:00
Jörg Thalheim 2dacdd110c Merge pull request #13195 from NixOS/remove-nix_2_18
Remove otherNixes.nix_2_18
2025-05-15 11:43:12 +02:00
Eelco Dolstra 3ba49d7ec2 flake.lock: Update
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/f02fddb8acef29a8b32f10a335d44828d7825b78?narHash=sha256-IgBWhX7A2oJmZFIrpRuMnw5RAufVnfvOgHWgIdds%2Bhc%3D' (2025-05-01)
  → 'github:NixOS/nixpkgs/adaa24fbf46737f3f1b5497bf64bae750f82942e?narHash=sha256-qhFMmDkeJX9KJwr5H32f1r7Prs7XbQWtO0h3V0a0rFY%3D' (2025-05-13)
2025-05-15 11:31:34 +02:00
Jörg Thalheim 2dd2142754 rename StoreDirConfigItself to StoreDirConfigBase
context: https://github.com/NixOS/nix/pull/13154#discussion_r2081904653
2025-05-15 11:24:42 +02:00
Eelco Dolstra 3285b68044 Merge pull request #13176 from obsidiansystems/revert-incomplete-closure-feature
Revert #77
2025-05-15 11:13:31 +02:00
Eelco Dolstra bc85e20fb9 Remove otherNixes.nix_2_18
Nixpkgs no longer has Nix 2.18, so this fails to evaluate.
2025-05-15 11:10:21 +02:00
John Ericson c3e7e88889 Merge pull request #13194 from NixOS/bump-2.30.0
Bump 2.30.0
2025-05-14 20:40:24 -04:00
John EricsonandEelco Dolstra 99cb85cd37 Revert "If a substitute closure is incomplete, build dependencies, then retry the substituter"
As summarized in
https://github.com/NixOS/nix/issues/77#issuecomment-2843228280 the
motivation is that the complicated retry logic this introduced was
making the cleanup task #12628 harder to accomplish. It was not easy to
ascertain just what policy / semantics the extra control-flow was
implementing, in order to figure out a different way to implementing it
either.

After talking to Eelco about it, he decided we could just....get rid of
the feature entirely! It's a bit scary removing a decade+ old feature,
but I think he is right. See the release notes for more explanation.

This reverts commit 299141ecbd.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-05-14 20:16:40 -04:00
John Ericson 95e1f463c8 .mergify.yml: Add backport 2.29-maintenance entry 2025-05-14 19:59:22 -04:00
John Ericson 6591a4119c Bump version 2025-05-14 19:59:22 -04:00
John Ericson 45726dcc55 Merge pull request #13189 from NixOS/rl-next
Add release notes
2025-05-14 19:58:59 -04:00
John Ericson f70796309d Merge pull request #13193 from xokdvium/lru-cache
libutil: Less unnecessary copying in `LRUCache`
2025-05-14 19:29:53 -04:00
John Ericson 1b22f0252d Merge pull request #13192 from NixOS/store-open-reg-header
Factor out `resolveStoreConfig`
2025-05-14 19:20:05 -04:00
John Ericson 6e9350d53f Add little blurb at the top of 2.29 release notes 2025-05-14 19:05:27 -04:00
John Ericson 29e78faa59 Slightly sort 2.29 release notes 2025-05-14 18:46:35 -04:00
John Ericson 646f3064a3 Cache more contributors 2025-05-14 18:44:39 -04:00
John Ericson d5e7359155 release notes: 2.29.0 2025-05-14 18:43:53 -04:00
John EricsonandRobert Hensing d7481b4b63 Factor out resolveStoreConfig
Leverage #10766 to show how we can now resolve a store configuration
without actually opening the store for that resolved configuration.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2025-05-14 18:30:25 -04:00
Robert Hensing 12b7132209 Merge pull request #13188 from NixOS/store-open-reg-header
Split out `store-open.hh` and `store-registration.hh`
2025-05-15 00:21:32 +02:00
John Ericson 18894c934f Merge remote-tracking branch 'upstream/master' into rl-next 2025-05-14 18:14:55 -04:00
Sergei Zimmerman d955b401a7 libutil: Sprinkle some noexcept on LRUCache::{size,clear}
The underlying containers are already noexcept to destroy and dtors are
noexcept in general.
2025-05-14 22:14:46 +00:00
John Ericson 8557f08327 Merge pull request #13190 from roberth/render-upcoming-notes
Render upcoming notes
2025-05-14 18:14:23 -04:00
Robert Hensing 2df638fd38 doc/rl-next: Edit 2025-05-15 00:12:42 +02:00
John Ericson b7b1ca9a3f doc/rl-next: Add repl-reload-flakes 2025-05-14 18:11:51 -04:00
Sergei Zimmerman 2f2e04142e libutil: Simplify LRUCache::get by using list splice
Splicing the list element to the back can be done in
a much simpler and concise way without the need for
erasing and re-inserting the element. Doing it this
way is equivalent to just moving node pointers around,
whereas inserting/erasing allocates/deallocates new nodes.
2025-05-14 22:05:53 +00:00
John Ericson e056cdc2a9 doc/rl-next: Add eval-cache-errors 2025-05-14 18:03:31 -04:00
John Ericson 3bc6bf3659 doc/rl-next: add osc-8-xterm-style-fix 2025-05-14 17:56:32 -04:00
Jörg Thalheim bc774d4365 doc/rl-next: retry-on-http-429 2025-05-14 23:51:19 +02:00
Robert Hensing bfecd13e9b doc/rl-next; add pretty-json 2025-05-14 23:43:27 +02:00
Sergei Zimmerman cd61e922ff libutil: Use heterogeneous lookup for LRUCache
This gets rid of some ugly std::string_view -> std::string
conversions, which are an eye-sore and lead to extra copying.
2025-05-14 21:42:35 +00:00
John Ericson 44ca04b944 doc/rl-next: nix-formatter 2025-05-14 17:41:50 -04:00
Jörg Thalheim c3684efa95 doc/rl-next: faster-blake3 2025-05-14 23:29:49 +02:00
Jörg Thalheim 121f7ca5fd doc/rl-next: add duplicate-copy 2025-05-14 23:24:35 +02:00
Sergei Zimmerman 90d70aa4c9 libutil: Format lru-cache.hh
Rip off the band-aid for further refactors. The diff is
very small, so it makes to get it out of the way first.
2025-05-14 21:23:13 +00:00
John Ericson eca100eb7d Revert "doc/rl-next: add better-flake-git-errors"
This was already released on 2.28.0

This reverts commit 721b09a69f.
2025-05-14 17:22:24 -04:00
John Ericson 1d23d883ad Revert "doc/rl-next: Add repl-and-logging"
It is already released on 2.28.0

This reverts commit db76487f41.
2025-05-14 17:20:37 -04:00
Jörg Thalheim 5ca242018d doc/rl-next: add multiple-signatures 2025-05-14 23:16:59 +02:00
John Ericson 0db10fc875 Merge pull request #12977 from Mic92/docs
Docs: fix "building" documentation w.r.t. meson
2025-05-14 17:10:29 -04:00
Jörg Thalheim cdb332990e doc/rl-next: add s3-sts-authentication 2025-05-14 23:05:46 +02:00
Robert Hensing 132c20872f doc/manual: Render unreleased release notes on master
This renders the rl-next notes when `officialRelease = false`, which
corresponds to the case where we're not on a release branch.

Previously we had disabled this behavior because changelog-d is
somewhat of a heavy dependency, being the only Haskell package.
However, we now have new circumstances that topple the tradeoff.

- We render `master` docs to https://nix.dev/manual/nix/development/release-notes/rl-next.html

- `.#manual` is a separate build now, so `nix build nix/foo` is
  not affected by the increased closure of build input outputs.

Because of these factors, I believe adding this functionality back
is more valuable, as we can use it to

- Previous release notes

- Showcase the upcoming release to the community
2025-05-14 23:03:33 +02:00
Jörg Thalheim 14b4a208c7 doc/rl-next: add fixed-s3-progress-bar 2025-05-14 23:01:46 +02:00
John Ericson db76487f41 doc/rl-next: Add repl-and-logging 2025-05-14 16:58:57 -04:00
Jörg Thalheim 721b09a69f doc/rl-next: add better-flake-git-errors 2025-05-14 22:57:07 +02:00
Jörg Thalheim 7f07a94b64 doc/rl-next: add repl-continuation 2025-05-14 22:49:05 +02:00
Jörg Thalheim 13dafac243 doc/rl-next: add substituter-timeout 2025-05-14 22:46:08 +02:00
Jörg Thalheim cbdd0c1035 doc/rl-next: add nix-flake-show 2025-05-14 22:45:57 +02:00
Jörg Thalheim 39d10229e5 doc/rl-next: add github-host-attr 2025-05-14 22:45:38 +02:00
Jörg Thalheim c12fd7b319 docs/building: fix attribute for nix-cli-ccacheStdenv 2025-05-14 22:22:01 +02:00
723f2c7352 docs: update cross compilation section from autotools to meson
fixes https://github.com/NixOS/nix/issues/12934

Update doc/manual/source/development/building.md

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>

Update doc/manual/source/development/building.md

Co-authored-by: John Ericson <git@JohnEricson.me>
2025-05-14 22:22:01 +02:00
Robert Hensing af6e44b116 doc/rl-next: Add c-api 2025-05-14 22:21:01 +02:00
John Ericson d972f9e2e2 Split out store-open.hh and store-registration.hh
The existing header is a bit too big. Now the following use-cases are
separated, and get their own headers:

- Using or implementing an arbitrary store: remaining `store-api.hh`

  This is closer to just being about the `Store` (and `StoreConfig`)
  classes, as one would expect.

- Opening a store from a textual description: `store-open.hh`

  Opening an aribtrary store implementation like this requires some sort
  of store registration mechanism to exists, but the caller doesn't need
  to know how it works. This just exposes the functions which use such a
  mechanism, without exposing the mechanism itself

- Registering a store implementation: `store-registration.hh`

  This requires understanding how the mechanism actually works, and the
  mechanism in question involves templated machinery in headers we
  rather not expose to things that don't need it, as it would slow down
  compilation for no reason.
2025-05-14 16:07:57 -04:00
Jörg Thalheim a70140b55a fix various typos in docs 2025-05-14 21:29:17 +02:00
Jörg Thalheim e088ab3eaf Merge pull request #13154 from obsidiansystems/split-store-config
Stores no longer inherit from their configs
2025-05-14 21:08:40 +02:00
Jörg Thalheim addb9f8418 Merge pull request #13180 from Jaculabilis/reload-flakes
nix repl: remember :load-flake calls for :reload
2025-05-14 20:35:58 +02:00
Jörg Thalheim a714d3bb3c Merge pull request #13184 from NixOS/doc-references
Document more references concepts
2025-05-14 19:33:43 +02:00
John Ericson b287438476 Document more references concepts 2025-05-14 12:57:06 -04:00
Eelco Dolstra 893b7a5fc7 Merge pull request #13178 from xokdvium/fix-doc-hash
docs: Fix miscellaneous typos and formatting issues
2025-05-14 15:24:16 +02:00
Tim Van Baak fb510a9e50 nix repl: remember :load-flake calls for :reload
Fixes #8753
2025-05-13 20:56:41 -07:00
Sergei Zimmerman 20a724d131 docs: Fix miscellaneous typos and formatting issues 2025-05-13 22:20:11 +00:00
John EricsonandSergei Zimmerman 934918ba16 Stores no longer inherit from their configs
Fix #10766

See that ticket for details.

Progress (I hope!) towards #11139.

Co-Authored-By: Sergei Zimmerman <xokdvium@proton.me>
2025-05-13 15:56:35 -04:00
Jörg Thalheim f0f196cef0 Merge pull request #13175 from xokdvium/optimise-fetchtarball
libutil/tarfile: Create the scratch `std::vector` only once
2025-05-13 14:21:26 +02:00
Sergei Zimmerman 7628155d2b libutil/tarfile: Create the scratch std::vector only once
I can't find a good way to benchmark in isolation from the
git cache, but common sense dictates that creating (and destroying)
a 131KiB std::vector for each regular file from the archive imposes
quite a significant overhead regardless of the IO bound git cache.

AFAICT there is no reason to keep a copy of the data since
it always gets fed into the sink and there are no coroutines/threads
in sight.
2025-05-13 10:50:46 +00:00
Jörg Thalheim 1ac4bf122b Merge pull request #13173 from NixOS/better-truncated-tar-error
Improve 'cannot read file from tarball' error
2025-05-13 12:34:27 +02:00
Jörg Thalheim 542d6f3c79 Merge pull request #13174 from NixOS/test-lock-files
Test lock file contents more precisely
2025-05-13 12:34:06 +02:00
Sergei Zimmerman 18a5589f9a libstore: Depend on boost_regex explicitly 2025-05-13 08:51:46 +00:00
Sergei Zimmerman f3090ef703 packaging/dependencies: Use boost without enableIcu
This reduces the closure size on master by 40MiB.

```
$ nix build github:nixos/nix/1e822bd4149a8bce1da81ee2ad9404986b07914c#nix-store --out-link closure-on-master
$ nix build .#nix-store -L --out-link closure-without-icu
$ nix path-info --closure-size -h ./closure-on-master
/nix/store/8gwr38m5h6p7245ji9jv28a2a11w1isx-nix-store-2.29.0pre  124.4 MiB
$ nix path-info --closure-size -h ./closure-without-icu
/nix/store/k0gwfykjqpnmaqbwh23nk55lhanc9g24-nix-store-2.29.0pre   86.6 MiB
```
2025-05-13 08:47:24 +00:00
Eelco Dolstra 824e0d51fe Test lock file contents more precisely 2025-05-13 10:40:24 +02:00
Eelco Dolstra bdb3f613dd Improve 'cannot read file from tarball' error
It now says e.g.

  error: cannot read file from tarball: Truncated tar archive detected while reading data
2025-05-13 10:32:12 +02:00
Jörg Thalheim ed521760bc Merge pull request #13161 from NixOS/remove-final
nix flake prefetch: Remove __final
2025-05-13 08:50:36 +02:00
Jörg Thalheim 0f985fea11 Merge pull request #13138 from NixOS/register-builtin-builders
Register builtin builders
2025-05-13 08:50:19 +02:00
Jörg Thalheim 6fdb170fb3 Merge pull request #13159 from NixOS/ignore-dir
Avoid unnecessarily updating old lock files with 'dir' parameters
2025-05-12 22:34:50 +02:00
Jörg Thalheim b87b1cbc75 Merge pull request #13165 from bam80/patch-1
index.md: add warning for installing as root
2025-05-12 22:25:50 +02:00
Jörg Thalheim c3745e7895 Merge pull request #13169 from NixOS/improve-build-failure-error
Improve build failure error messages
2025-05-12 22:25:31 +02:00
Eelco Dolstra 5a84237209 Improve build failure error messages
They're now laid out in a more readable way, and they shows the output
paths (if known).
2025-05-12 15:06:54 +02:00
Andrey Butirsky 3cc16d13f0 index.md: add warning for installing as root
warning: installing Nix as root is not supported by this script!
performing a single-user installation of Nix...
copying Nix to /nix/store.......................................................
......
warning: the group 'nixbld' specified in 'build-users-group' does not exist
warning: the group 'nixbld' specified in 'build-users-group' does not exist
installing 'nix-2.28.3'
error: the group 'nixbld' specified in 'build-users-group' does not exist
/tmp/nix-binary-tarball-unpack.2j3lCU0A89/unpack/nix-2.28.3-x86_64-linux/install: unable to install Nix into your default profile
2025-05-10 18:13:58 +03:00
Eelco Dolstra cdbe788c1f Merge pull request #13155 from Valodim/patch-2
Reduce maxLayers to 70 in docker build
2025-05-09 17:12:38 +02:00
Eelco Dolstra da953d6d39 Add test 2025-05-09 16:55:13 +02:00
Eelco Dolstra c686de6505 nix flake prefetch: Remove __final
This is currently an internal attribute, not intended to be shown to
users.

Fixes #13150.
2025-05-09 16:44:48 +02:00
Eelco Dolstra d00682beb2 Backward compatibility hack for dealing with dir in URL-style flakerefs 2025-05-09 10:10:42 +02:00
Vincent Breitmoser 6d0f174cd9 Reduce maxLayers to 70 in docker build
The nixos/nix docker image is built using `buildLayeredImage`, which spreads the nix store over a configured number of layers. This number was set to create an image with 100 layers. Because there is a limit of (typically) 127 layers in AUFS, this only left 27 layers to build on top. At the same time, nearly half of the created layers were only <100kb in size, many even <10kb, negating the intended advantage in cachability.

This commit moves the tradeoff a bit by reducing the number of layers to 70.

Layer sizes for the 2.28.3 nixos/nix image: https://hub.docker.com/layers/nixos/nix/2.28.3/images/sha256-d078d7153763895fce17c5fbbdeb86fcfcac414ca0ba875d413c1df57be19931
2025-05-08 09:56:14 +02:00
Sergei Zimmerman 3a1301cd6d libstore: Use boost::regex for GC root discovery
As it turns out using `std::regex` is actually the bottleneck
for root discovery. Just substituting `std::` -> `boost::`
makes root discovery twice as fast (3x if counting only userspace time).

Some rather ad-hoc measurements to motivate the switch:

(On master)

```
nix build github:nixos/nix/1e822bd4149a8bce1da81ee2ad9404986b07914c#nix-cli --out-link result-1e822bd4149a8bce1da81ee2ad9404986b07914c
taskset -c 2,3 hyperfine "result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0"
Benchmark 1: result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0
  Time (mean ± σ):     481.6 ms ±   3.9 ms    [User: 336.2 ms, System: 142.0 ms]
  Range (min … max):   474.6 ms … 487.7 ms    10 runs
```

(After this patch)

```
taskset -c 2,3 hyperfine "result/bin/nix store gc --dry-run --max 0"
Benchmark 1: result/bin/nix store gc --dry-run --max 0
  Time (mean ± σ):     254.7 ms ±   9.7 ms    [User: 111.1 ms, System: 141.3 ms]
  Range (min … max):   246.5 ms … 281.3 ms    10 runs
```

`boost::regex` is a drop-in replacement for `std::regex`, but much faster.
Doing a simple before/after comparison doesn't surface any change in behavior:

```
result/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
```
2025-05-06 21:58:52 +00:00
John Ericson 1e822bd414 Merge pull request #13141 from obsidiansystems/fix-windows-warning
Fix windows warning
2025-05-06 15:42:16 -04:00
John Ericson 1594d4b879 Fix windows warning 2025-05-06 15:02:10 -04:00
Jörg Thalheim 10358c630b Merge pull request #13139 from NixOS/singleton-pattern
Simplify plugin registrations
2025-05-06 09:31:09 +02:00
Eelco Dolstra 060c34b664 Attempt to fix macOS build 2025-05-06 08:50:14 +02:00
Eelco Dolstra 47989a2124 Simplify ConfigRegistrations 2025-05-05 09:42:16 +02:00
Eelco Dolstra f59ccb468e Simplify Implementations registration 2025-05-05 08:41:23 +02:00
Eelco Dolstra 93844a5998 Simplify registerInputScheme() 2025-05-05 08:35:59 +02:00
Eelco Dolstra e7c0906521 Simplify RegisterCommand 2025-05-05 08:28:12 +02:00
Eelco Dolstra 4de7a986d4 Simplify RegisterPrimOp 2025-05-05 08:26:29 +02:00
Eelco Dolstra b7add9736c Simplify RegisterLegacyCommand 2025-05-05 08:22:53 +02:00
Eelco Dolstra 1479305001 Simplify RegisterBuiltinBuilder 2025-05-05 08:16:09 +02:00
Eelco Dolstra c7a84b9160 Pass tmpDirInSandbox to the builtin builders 2025-05-04 22:19:34 +02:00
Jörg Thalheim bd80a4f176 Merge pull request #13137 from xokdvium/regex-cache-transparent
libexpr: Use C++20 heterogeneous lookup for RegexCache
2025-05-04 22:18:07 +02:00
Eelco Dolstra fe0124fe17 Put the builder context in a struct 2025-05-04 22:14:11 +02:00
Eelco Dolstra 40bbad3be5 Allow dynamic registration of builtin builders 2025-05-04 22:09:45 +02:00
Sergei Zimmerman 36c583dae0 libexpr: Use C++20 heterogeneous lookup for RegexCache 2025-05-04 16:03:57 +00:00
John Ericson 2676ae7ca6 Merge pull request #12676 from silvanshade/blake3-tbb
Implement memory-mapped IO and multi-threading for BLAKE3 hashing
2025-05-04 10:58:53 -04:00
Jörg Thalheim 86bf01bc84 Merge pull request #13136 from not-my-profile/fix-matrix-links
fix(docs): update Matrix channel links
2025-05-04 13:57:50 +02:00
Martin Fischer 81683a845b fix(docs): update Matrix channel links 2025-05-04 12:17:48 +02:00
Jörg Thalheim 469a6371ec Merge pull request #13121 from tomberek/tomberek.redirect_http
fix: allow redirected HTTP uploads
2025-05-03 09:33:19 +02:00
Jörg Thalheim 7808aa2eee Merge pull request #13129 from xokdvium/transparent-comparator
Use transparent comparators for `std::set<std::string>` (NFC)
2025-05-03 09:04:52 +02:00
Jörg Thalheim bd643359a2 Merge pull request #13130 from xokdvium/symbol-table-chore
libexpr: Remove unused field from SymbolTable::symbols and emplace in…
2025-05-03 08:59:05 +02:00
silvanshade 7db388f597 Implement multi-threaded BLAKE3 hashing 2025-05-02 15:29:22 -06:00
silvanshade b1783ff615 Implement memory-mapped IO for Sinks 2025-05-02 15:29:22 -06:00
silvanshade a87c3711b6 Update flake nixpkgs 2025-05-02 15:29:22 -06:00
Sergei Zimmerman 161c5dbf39 libexpr: Remove unused field from SymbolTable::symbols and emplace into the ChunkedVector
Remove outdated and no longer relevant TODO. It's more confusing
now, since symbol table must now be addressed by uint32_t indices
in order to keep Attr size down to 16 bytes on 64 bit machines.
2025-05-02 20:42:47 +00:00
Sergei Zimmerman ebb836d499 Use transparent comparators for std::set<std::string> (NFC)
This patch finally applies the transition to std::less<>,
which is a transparent comparator. There's no functional
change and string lookups in sets are now more efficient
and don't produce temporaries (e.g. set.find(std::string_view{"key"})).
2025-05-02 17:43:02 +00:00
Sergei Zimmerman 5278cd2396 libstore: Introduce WorkerProto::FeatureSet alias
Unfortunately Feature is just an alias to `std::string`
and not a new-type, so a ton of code relies on it being
exactly a `std::string`.

Using transparent comparators just for StringSet necessitates
using it here as well.
2025-05-02 17:40:34 +00:00
Sergei Zimmerman 55815ec225 treewide: Use PathSet alias consistently instead of std::set<Path> 2025-05-02 17:40:31 +00:00
Sergei Zimmerman d8c97d8073 treewide: Use StringSet alias consistently instead of std::set<std::string>
The intention is to switch to transparent comparators from N3657 for
ordered set containers for strings and using the alias consistently
would simplify things.
2025-05-02 17:40:29 +00:00
Eelco Dolstra a976a46ee8 Merge pull request #13123 from Mic92/filesystem-refactoring
Drop fs alias in favour of std::filesystem
2025-05-02 12:52:12 +02:00
Jörg Thalheim 95cf0d31df Merge pull request #13126 from xokdvium/flex-full
libexpr: Improve lexer performance by using full scanner tables (-Cf)
2025-05-02 10:28:37 +02:00
Sergei Zimmerman 86a3fad085 libexpr: Improve lexer performance by using full scanner tables (-Cf)
This trades off some executable size for measurable lexer performance
improvements.

Note on the explicitly enabling 8bit scanner.
This is needed due to the default behavior of flex (excerpt from the manual [1]):

> Flex’s default behavior is to generate an 8-bit scanner unless you
> use the ‘-Cf’ or ‘-CF’, in which case flex defaults to generating
> 7-bit scanners unless your site was always configured to generate 8-bit
> scanners.

Some quantifyable metrics:

Nixpkgs revision: a6e3f45acf4e817532a861ab0eda4ab5485fecc1
Parsing the largest file in nixpkgs: pkgs/development/haskell-modules/hackage-packages.nix.

(Before this patch)

```
$ nix build github:nixos/nix/9fe3077d4#nix-expr
$ du --apparent-size result/lib/libnixexpr.so
2518    result/lib/libnixexpr.so
$ nix build github:nixos/nix/9fe3077d4#nix-cli
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g \
    result/bin/nix-instantiate --parse               \
    ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix > /dev/null"
  Time (mean ± σ):     375.5 ms ±   6.3 ms    [User: 316.9 ms, System: 56.7 ms]
  Range (min … max):   368.5 ms … 388.3 ms    10 runs
```

(After the patch)

```
$ nix build .#nix-expr
$ du --apparent-size result/lib/libnixexpr.so
2685    result/lib/libnixexpr.so
$ nix build .#nix-cli
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g \
    result/bin/nix-instantiate --parse               \
    ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix > /dev/null"
  Time (mean ± σ):     326.8 ms ±   4.9 ms    [User: 269.5 ms, System: 55.3 ms]
  Range (min … max):   319.7 ms … 335.5 ms    10 runs
```

Overall, the change is roughly:

- 2518KiB -> 2685KiB ~ 150 KiB of machine code
- 375ms -> 325ms ~ 50ms

The perf uplift for eval-heavy test cases is obviously less noticeable,
but it doesn't make sense not to take this free perf win.

[1]: https://westes.github.io/flex/manual/Options-Affecting-Scanner-Behavior.html#Options-Affecting-Scanner-Behavior
2025-05-01 23:10:04 +00:00
Thomas Bereknyei 90deb665eb fix: allow redirected HTTP uploads
When a PUT is redirected, some of the data can be sent by curl before headers are read. This means the subsequent PUT operation needs to seek back to origin.
2025-05-01 11:11:49 -04:00
Jörg Thalheim 9fe3077d47 Merge pull request #13014 from jfly/update-nix_fmt-man-page
Update `nix fmt` man page with official formatter example
2025-05-01 15:11:12 +02:00
Jeremy Fleischman 6f71d8a9c2 Update nix fmt man page with official formatter example
The current example relies upon [nixfmt's deprecated tree traversal
behavior](https://github.com/NixOS/nixfmt/pull/240). The simplest
alternative is the new `nixfmt-tree` wrapper for `nixfmt`/`treefmt`.
2025-05-01 14:30:43 +02:00
Jörg Thalheim 979d5a7cae Drop fs alias in favour of std::filesystem
Since we dropped fs::symlink_exists, we no longer have a need for the fs
namespace. Having less abstractions makes it easier to lookup the
functions in reference documentations.
2025-05-01 14:24:34 +02:00
Jörg Thalheim 5b59be914d Replace symlink_exists with pathExists
As it turns out the orignal implementation of symlink_exists cannot be
used in Nix because it did now std::filesystem::filesystem_error.
The new implementation fixes that but is now actually the same as
pathExists except for the path type.
2025-05-01 14:22:35 +02:00
Jörg Thalheim 143fb88ceb Merge pull request #13122 from Mic92/directory-iterator
Replace all instances of std::filesystem::directory_iterator with DirectoryIterator
2025-05-01 14:22:21 +02:00
Jörg Thalheim 1c4496f4e5 replace all instances of std::filesystem::directory_iterator with DirectoryIterator 2025-05-01 11:54:26 +02:00
Jörg ThalheimandSergei Zimmerman 7ccc0d591f add DirectoryIterator to re-throw std::filesystem::filesystem_error
Co-authored-by: Sergei Zimmerman <145775305+xokdvium@users.noreply.github.com>
2025-05-01 11:54:13 +02:00
Jörg Thalheim b5dc8181b0 Merge pull request #13107 from fricklerhandwerk/ivory-tower
docs: don't mention Haskell
2025-05-01 10:27:42 +02:00
Jörg Thalheim 5b900120a2 Merge pull request #13117 from NixOS/file-pure-eval
Complain when using --pure-eval with --file
2025-05-01 10:26:41 +02:00
Jörg Thalheim d5cefa625c Merge pull request #13120 from Mic92/fix-quoting
replaceSymlink: fix quoting in error message
2025-05-01 10:25:22 +02:00
Jörg Thalheim 408746cba6 replaceSymlink: fix quoting in error message 2025-05-01 08:19:18 +02:00
mergify[bot] 4548dd1abb Merge pull request #13112 from NaN-git/fix-json-getInteger
bugfix in getInteger(const nlohmann::json &) and add bounds checks
2025-04-30 21:20:49 +00:00
Jörg Thalheim d155bb9012 Merge pull request #13063 from jfly/add-nix-fmt-print-command-option
Add `nix formatter build` and `nix formatter run` commands
2025-04-30 22:39:48 +02:00
Eelco Dolstra d46f741cdf Complain when using --pure-eval with --file
This never worked and cannot work because in pure eval mode, the
evaluator doesn't have access to the file.
2025-04-30 22:29:27 +02:00
Jeremy Fleischman 5089f1292d Refactor, use MixOutLinkByDefault 2025-04-29 18:40:02 -07:00
Jeremy Fleischman 7df7bde306 Refactor, extract some shared code into UnresolvedApp::build 2025-04-29 18:40:02 -07:00
Robert Hensing e14346c7da Refactor, dedup nix formatter attribute methods 2025-04-29 18:40:02 -07:00
Jeremy Fleischman ba6b617e75 Add nix formatter build command
`nix formatter build` is sort of like `nix build`: it builds, links, and
prints a path to the formatter program:

    $ nix formatter build
    /nix/store/cb9w44vkhk2x4adfxwgdkkf5gjmm856j-treefmt/bin/treefmt

Note that unlike `nix build`, this prints the full path to the program,
not just the store path (in the example above that would be
`/nix/store/cb9w44vkhk2x4adfxwgdkkf5gjmm856j-treefmt`).

Motivation
----------

I maintain a vim plugin that automatically runs `nix fmt` on files on
save. Since `nix fmt` can be quite slow due to nix evaluation, I choose
to cache the `nix fmt `entrypoint. This was very awkward to do, see the
implementation for details:
https://github.com/nvimtools/none-ls.nvim/blob/786460723170bda9e9f95c55a382d21436575297/lua/null-ls/builtins/formatting/nix_flake_fmt.lua#L83-L110.

I recently discovered that my implementation was buggy (it didn't handle
flakes that expose a `formatter` package, such as nixpkgs), so I had to
rework the implementation:
https://github.com/nvimtools/none-ls.nvim/pull/272.

With the new `nix formatter build` command, I can delete all this akward
code, and it will be easier for other folks to build performant editor
integrations for `nix fmt`.
2025-04-29 18:40:02 -07:00
Jeremy Fleischman 5ea7b97147 refactor: create a new nix formatter run command alias for nix fmt
This refactor shouldn't change much except add a new `nix formatter run`
command. This creates space for the new `nix formatter build` command,
which I'll be introducing in the next commit.
2025-04-29 18:40:02 -07:00
Philipp Otterbein 788be3f964 bugfix in getInteger(const nlohmann::json &) and add bounds checks
improve error messages, too
2025-04-30 01:38:48 +02:00
Eelco Dolstra 9099b7dd87 Merge pull request #13109 from trofi/osc-8-xterm-style-fix
libutil: amend OSC 8 escape stripping for xterm-style separator
2025-04-29 22:39:04 +02:00
Jörg Thalheim 0e2dc8774c Merge pull request #13105 from roberth/refactor-cli-out-link
Factor out `MixOutLinkByDefault`
2025-04-29 21:19:34 +02:00
Sergei Trofimovich e322b714dc libutil: amend OSC 8 escape stripping for xterm-style separator
Before the change `nix` was stripping warning flags
reported by `gcc-14` too eagerly:

    $ nix build -f. texinfo4
    error: builder for '/nix/store/i9948l91s3df44ip5jlpp6imbrcs646x-texinfo-4.13a.drv' failed with exit code 2;
           last 25 log lines:
           >  1495 | info_tag (mbi_iterator_t iter, int handle, size_t *plen)
           >       |                                            ~~~~~~~~^~~~
           > window.c:1887:39: error: passing argument 4 of 'printed_representation' from incompatible pointer type []
           >  1887 |                                       &replen);
           >       |                                       ^~~~~~~
           >       |                                       |
           >       |                                       int *

After the change the compiler flag remains:

    $ ~/patched.nix build -f. texinfo4
    error: builder for '/nix/store/i9948l91s3df44ip5jlpp6imbrcs646x-texinfo-4.13a.drv' failed with exit code 2;
       last 25 log lines:
       >  1495 | info_tag (mbi_iterator_t iter, int handle, size_t *plen)
       >       |                                            ~~~~~~~~^~~~
       > window.c:1887:39: error: passing argument 4 of 'printed_representation' from incompatible pointer type [-Wincompatible-pointer-types]
       >  1887 |                                       &replen);
       >       |                                       ^~~~~~~
       >       |                                       |
       >       |                                       int *

Note the difference in flag rendering around the warning.

https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda has a
good sumamry of why it happens. Befomre the change `nix` was handling
just one form or URL separator:

    $ printf '\e]8;;http://example.com\e\\This is a link\e]8;;\e\\\n'

Now it also handled another for (used by gcc-14`):

    printf '\e]8;;http://example.com\aThis is a link\e]8;;\a\n'

While at it fixed accumulation of trailing escape `\e\\` symbol.
2025-04-29 16:12:18 +01:00
Valentin Gagarin cb9182f9ed docs: don't mention Haskell
sometimes it's these little things that let beginners stumble at the
first step...

mentioning one potentially foreign concept while introducing an entirely
new concept is asking enough already.
2025-04-29 10:58:34 +02:00
Robert Hensing 2ec1303286 Merge pull request #13103 from obsidiansystems/dedup-static-functions
Delete dead code
2025-04-28 18:00:48 +02:00
John Ericson 46030181d4 Delete dead code
We had multiple copies of some static functions after splitting out
`DerivationBuilder` by mistake.
2025-04-28 11:19:36 -04:00
Robert Hensing 9d1f00e31d Factor out MixOutLinkByDefault 2025-04-28 16:54:00 +02:00
mergify[bot] 3d39864b96 Merge pull request #13099 from roberth/doc-evaluation
doc/manual: Add language/evaluation
2025-04-28 08:20:45 +00:00
Eelco Dolstra 5565679b45 Merge pull request #13100 from NaN-git/fix-musl-deadlock
allocate SimpleLogger before forking
2025-04-28 09:01:06 +02:00
Philipp Otterbein 4e95f662db allocate SimpleLogger before forking 2025-04-28 00:46:44 +02:00
Robert Hensing c81202b358 Merge pull request #13096 from NixOS/no-use-registries
getFlake(): Don't use registries for refetching
2025-04-27 00:42:51 +02:00
Robert Hensing ee59af99f8 Merge pull request #13098 from roberth/fix-string-ctor
Fix flake-c out of bounds access
2025-04-26 18:46:11 +02:00
Robert Hensing a525c7e991 doc/manual: Add language/evaluation 2025-04-25 18:06:03 +02:00
Robert Hensing 94916136dc Fix flake-c out of bounds access
The explicit include is needed for clangd to not get confused somehow,
which is also what threw me off initially and made me pick the wrong
constructor.
The (pointer, number, number) constructor first constructs a C string
and then takes a substring from that, but we didn't specify that the
buffer needs to be NUL-terminated, and then what would be the point of
the size argument anyway...

basic_string.h:

>	basic_string(const _Tp& __t, size_type __pos, size_type __n,
>		     const _Alloc& __a = _Alloc())
>	: basic_string(_S_to_string_view(__t).substr(__pos, __n), __a) { }

Valgrind on nixops4/rust/nix-flake tests:

==1344422== Conditional jump or move depends on uninitialised value(s)
==1344422==    at 0x48513E8: strlen (vg_replace_strmem.c:505)
==1344422==    by 0x488E941: UnknownInlinedFun (char_traits.h:391)
==1344422==    by 0x488E941: UnknownInlinedFun (string_view:141)
==1344422==    by 0x488E941: UnknownInlinedFun (basic_string.h:790)
==1344422==    by 0x488E941: nix_flake_reference_and_fragment_from_string (nix_api_flake.cc:81)
==1344422==    by 0x127332: nix_flake::FlakeReference::parse_with_fragment (lib.rs:123)
2025-04-25 17:07:32 +02:00
Eelco Dolstra 3f811c2373 Merge pull request #13094 from xokdvium/file-content-address-fmt-string
libutil: Add missing format arguments to UsageError ctor
2025-04-25 15:21:57 +02:00
Eelco Dolstra 953ec00794 getFlake(): Don't use registries for refetching
`newLockedRef` is already resolved so there is no need to re-resolve
it.
2025-04-25 13:46:42 +02:00
Sergei Zimmerman 9fff868e39 libutil: Add missing format arguments to UsageError ctor
Once again found by an automated migration to `std::format`.
I've tested that boost::format works fine with `std::string_view`
arguments.
2025-04-25 13:35:16 +03:00
Jörg Thalheim 27047570b5 Merge pull request #13086 from xokdvium/bad-format-string
libutil: Fix invalid boost format string in infinite symlink recursion error
2025-04-25 11:20:04 +02:00
Jörg Thalheim 96a1740942 Merge pull request #13088 from NixOS/fix-ignore-local-registries
Fix ignore local registries
2025-04-25 11:19:25 +02:00
Jörg Thalheim 94edfb1a3b Merge pull request #13087 from xokdvium/libutil-git-permission-format
libutil: Use correct argument to Error format ctor
2025-04-25 11:18:58 +02:00
Jörg Thalheim 6405d6822d tests/flakes: add regression test for resolving user flakes 2025-04-25 10:34:41 +02:00
Sergei Zimmerman 1b5c8aac12 libutil: Use correct argument to Error format ctor
It seems that the intention was to format a number in base 8 (as
suggested by the %o format specifier), but `perms` is a `std::string`
and not a number. Looks like `rawMode` is the correct thing to use here.
2025-04-24 22:27:03 +00:00
Sergei Zimmerman bfb357c40b libutil: Fix invalid boost format string in infinite symlink recursion error
Found while working on an automated migration to `std::format`.
2025-04-24 21:00:24 +00:00
mergify[bot] b287f33090 Merge pull request #13085 from NixOS/bump-nixpkgs
flake.lock: Update
2025-04-24 20:23:53 +00:00
John Ericson 3e7d85dfdb flake.lock: Update
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/f675531bc7e6657c10a18b565cfebd8aa9e24c14?narHash=sha256-gbl9hE39nQRpZaLjhWKmEu5ejtQsgI5TWYrIVVJn30U%3D' (2025-04-09)
  → 'github:NixOS/nixpkgs/8a2f738d9d1f1d986b5a4cd2fd2061a7127237d7?narHash=sha256-sPwcCYuiEopaafePqlG826tBhctuJsLx/mhKKM5Fmjo%3D' (2025-04-23)
2025-04-24 13:43:53 -04:00
Jörg Thalheim a8fb23b716 Merge pull request #13079 from cole-h/cole-h/push-romznoxnzppy
nix-cli: restore binary-dist artifact to Hydra static builds
2025-04-24 19:29:59 +02:00
Eelco Dolstra 68de26d38a lockFlake(): Allow registry lookups for the top-level flake
Fixes #13050.
2025-04-24 18:59:10 +02:00
Cole Helbling e1b68244ad nix-cli: restore binary-dist artifact to Hydra static builds 2025-04-24 08:46:47 -07:00
Eelco Dolstra dda265f09a Reapply "Actually ignore system/user registries during locking"
This reverts commit 3b5f0d9fb3.
2025-04-24 17:33:27 +02:00
Jörg Thalheim 7880afcacd Merge pull request #13076 from Mic92/fix-ignore-local-registrie
Revert "Actually ignore system/user registries during locking"
2025-04-24 13:33:31 +02:00
Jörg Thalheim 3b5f0d9fb3 Revert "Actually ignore system/user registries during locking"
This reverts commit 77d4316353.
2025-04-24 11:28:11 +02:00
Robert Hensing 8a1c40b927 Merge pull request #13065 from roberth/escapeShellArg
Rename `shellEscape` -> `escapeShellArgAlways`
2025-04-23 23:35:15 +02:00
Robert Hensing 1e5b1d9973 Rename shellEscape -> escapeShellArgAlways
This name is close to the Nixpkgs lib function `escapeShellArg`,
making it easier to find.

A friendlier function with the same behavior as lib could be added
later.
2025-04-23 22:56:07 +02:00
John Ericson cccc26dcb1 Merge pull request #13059 from VitGottwald/patch-1
Use python3 packages in deps
2025-04-23 14:43:06 -04:00
John Ericson 9eb608cbef Merge pull request #13070 from DeterminateSystems/fix-windows
Don't build MonitorFdHup test on Windows
2025-04-23 14:23:15 -04:00
John Ericson 714662e675 Merge pull request #13068 from DeterminateSystems/fix-warning
DerivationBuilder: Fix compiler warning
2025-04-23 14:10:55 -04:00
John Ericson 052d5cb3fe Merge pull request #13067 from DeterminateSystems/move-libflake
Move libflake/flake/* to libflake
2025-04-23 14:03:35 -04:00
John Ericson 36fa9b90d8 Merge pull request #13069 from DeterminateSystems/fix-freebsd
Fix signedness error on FreeBSD
2025-04-23 14:02:13 -04:00
Eelco Dolstra a9b6213221 Don't build MonitorFdHup on Windows
https://hydra.nixos.org/build/295398462
2025-04-23 19:51:13 +02:00
Eelco Dolstra fd0835e7d7 Fix signedness error on FreeBSD
https://hydra.nixos.org/build/295398446
2025-04-23 17:41:35 +02:00
Eelco Dolstra d07541be29 DerivationBuilder: Fix compiler warning
Fixes

  ../src/libstore/unix/build/derivation-builder.cc:1130:86: warning: the compiler can assume that the address of ‘nix::DerivationBuilderParams::drv’ will never be NULL [-Waddress]
   1130 |     if (useChroot && settings.preBuildHook != "" && dynamic_cast<const Derivation *>(&drv)) {
        |                                                                                        ^~~~

Assuming this check was left over from the time `drv` could be a
`BasicDerivation`.
2025-04-23 17:28:09 +02:00
Eelco Dolstra bc67e47298 Move libflake/flake/* to libflake 2025-04-23 17:20:42 +02:00
John Ericson 876f676d90 Merge pull request #13060 from VitGottwald/patch-3
Fix perl example
2025-04-22 13:28:54 -04:00
Vit Gottwald 9f94a1b9fb Put every package on its own line 2025-04-22 10:45:15 +02:00
Vit Gottwald aa96bf2faf Fix perl example
The perl example does not work with http://nixos.org because it redirects.

Updating the url to https requires additional package.
2025-04-22 09:49:34 +02:00
Vit Gottwald aff7facdf6 Use python3 packages in deps
To be consistent with the sample that uses the packages directly
in inline shebang.
2025-04-22 09:23:56 +02:00
John Ericson 155411397d Merge pull request #13055 from obsidiansystems/inlined-resolvedFinished
Inline `DerivationGoal::resolvedFinished`
2025-04-20 19:11:38 -04:00
John Ericson 54b7ccc466 Merge pull request #13053 from obsidiansystems/no-more-local-derivation-goal
Get rid of `LocalDerivationGoal`
2025-04-20 18:50:05 -04:00
John Ericson 16f640a9b2 Inline DerivationGoal::resolvedFinished
`resolvedDrvGoal` can just become a local variable!
2025-04-20 18:32:19 -04:00
Robert Hensing 15fa95f925 Merge pull request #13052 from rhendric/rhendric/increase-429-delay
libstore: increase retry delay for 429
2025-04-21 00:31:15 +02:00
John EricsonandRobert Hensing 4e586149df Get rid of LocalDerivationGoal
I split it out before to try to separate the building logic, but now we
have the much better `DerivationBuilder` abstraction for that. With that
change, I think `LocalDerivationGoal` has outlived its usefulness.

We just inline it back into `DerivationGoal`, and do so with minimal
`#ifdef` for Windows.

Note that the order of statements in `~DerivationGoal` is different than
it was after the `~LocalDerivationGoal` split, but it is *restored* to
the way it original was before --- evidently I did the split slightly
wrong, but nobody noticed, probably because the order doesn't actually
matter.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2025-04-20 18:09:41 -04:00
John Ericson bebef8f0c4 Merge pull request #13010 from obsidiansystems/bump-nixpkgs
flake.lock: Update Nixpkgs
2025-04-20 17:59:15 -04:00
mergify[bot] 933ed27dd9 Merge pull request #13054 from roberth/fix-non-virt-dtor
Fix non-virtual destructor warning
2025-04-20 21:50:32 +00:00
Robert Hensing 047f2bc1af refactor: Extract RETRY_TIME constants in filetransfer 2025-04-20 23:29:22 +02:00
John EricsonandRobert Hensing ef36806898 Explain the use of "2" in the overlay
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2025-04-20 17:20:54 -04:00
Robert Hensing 363ee6dfcb Fix non-virtual destructor warning 2025-04-20 23:05:21 +02:00
Robert Hensing b257ea94e3 Fix pkgs.nixVersions and installTests
... by moving our stuff out of the way from upstream's
`nixComponents` and `nixDependencies` attrsets.

(I prefer not to use overlays, but let's make it work this way
first)
2025-04-20 22:20:52 +02:00
John Ericson f186491db9 Merge pull request #13047 from obsidiansystems/derivation-building-2
Remove double indirection from `DerivationBuilder` params
2025-04-20 15:19:35 -04:00
Ryan Hendrickson 85420b8537 libstore: increase retry delay for 429
A 429 (Too Many Requests) error should not be retried after a quarter of
a second; that's just silly. GitHub recommends a minute.
2025-04-20 14:49:03 -04:00
John Ericson 5eeeb44f79 Remove double indirection from DerivationBuilder params
Now that `DerivationBuilder` is created after the underlying data has
already been initialized, we can just refer this data normally, with a
direct reference.

Only `parsedDrv` takes a (borrowing) pointer, because independent of
initialization the derivation may or may not have structured attrs.
2025-04-18 12:26:05 -04:00
John Ericson 2b2ea218d9 Delay constructing a DerivationBuilder until we're about to build
This makes the simplification in the following commit possible.
2025-04-18 12:26:05 -04:00
John Ericson 27907e6cac Add documentation for DerivationBuilder::{parsedDrv, drvOptions} 2025-04-18 12:26:05 -04:00
Jörg Thalheim f683a555bf Merge pull request #12423 from ilya-bobyr/fish-profile-local-state-bin-in-PATH
nix-profile.fish: Add local state bin to $PATH
2025-04-18 07:27:04 +02:00
Eelco Dolstra 75679b323b Merge pull request #13043 from Mic92/fix-build
fix armv7/i686 build
2025-04-17 17:03:19 +02:00
Jörg Thalheim b3bbbd2e71 fix armv7/i686 build 2025-04-17 13:56:11 +02:00
Jörg Thalheim e27d804929 Merge pull request #13039 from obsidiansystems/derivation-builder-in-separate-file
Move `DerivationBuilder` to its own files
2025-04-17 13:25:37 +02:00
Jörg Thalheim 6dcbac4278 Merge pull request #13040 from ilya-bobyr/fish-profile-set-MANPATH
nix-profile-daemon.fish: Set MANPATH
2025-04-17 10:24:19 +02:00
Jörg Thalheim 6fe0545bf3 Merge pull request #13041 from ilya-bobyr/fish-profile-do-not-test-USER
nix-profile.fish: Do not check $USER
2025-04-17 10:23:59 +02:00
Illia Bobyr 66ae8f4f44 nix-profile.fish: Do not check $USER
While it seems unlikely that `$USER` will be unset while `$HOME` is set,
as `$USER` is not used in the script and as `nix-profile-daemon.fish` is
not checking `$USER`, it seems better to remove this check.

`nix-profile.fish` and `nix-profile-daemon.fish` now become identical.
2025-04-16 17:16:51 -07:00
Illia Bobyr 470c521bcc nix-profile-daemon.fish: Set MANPATH
There seems to be no good reason for `nix-profile.fish` to set
`$MANPATH` while it being unset when `nix-profile-daemon.fish` is used.
2025-04-16 17:14:47 -07:00
Illia Bobyr fc8c11be48 nix-profile.fish: Add local state dir bin to $PATH
It seems reasonable to add both `$HOME/.profile/bin` and
`@localstatedir@/nix/profiles/default/bin` to `$PATH` for both user
local and daemon based nix execution.  Nix daemon execution mode does
not affect these path.
2025-04-16 17:11:56 -07:00
John Ericson 9792d6bbd9 Move DerivationBuilder to its own file/header
The building logic is now free of the scheduling logic!

(The interface between them is just what is in the new header. This
makes it much easier to audit, and shrink over time.)
2025-04-16 18:59:17 -04:00
John Ericson 6c2a7fdc49 Copy local-derivation-goal.cc to derivation-builder.{cc,hh}
This is done to prior to splitting, just like
05cc5a8587 for
68f4c728ec.
2025-04-16 18:59:17 -04:00
mergify[bot] b509502bd5 Merge pull request #12663 from obsidiansystems/local-derivation-goal-encapsulation
Separate derivation building from the scheduler
2025-04-16 22:33:25 +00:00
John Ericson d98c0dbe99 Start separating scheduling from building
We have a new `DerivationBuilder` struct, and `DerivationBuilderParams`
`DerivationBuilderCallbacks` supporting it.

`LocalDerivationGoal` doesn't subclass any of these, so we are ready to
now move them out to a new file!
2025-04-16 17:54:56 -04:00
John Ericson f81c06accf Gut LocalDerivationGoal::tryLocalBuild
Now, most of it is in two new functions:
`LocalDerivationGoal::{,un}repareBuild`.

This might seems like a step backwards from coroutines --- now we have
more functions, and are stuck with class vars --- but I don't think it
needs to be.

There's a few options here:

- (Re)introduce coroutines for the isolated building logic. We could use the
  same coroutines types, or simpler ones specialized to this use-case.
  The `tryLocalBuild` caller can still use `Goal::Co`, and just will
  manually "pump" this inner coroutine.

- Return closures from each step. This is sort of like coroutines by
  hand, but it still allows us to stop writing down the local variables
  in each type.

  Being able to fully-use RAII again would be very nice!

- Keep top-level first-order functions like now, but make more
  functional. Instead of having one state object (`DerivationBuilder`)
  for all steps (setup, run, teardown), we can have separate structs for
  the live variables at each point we consume and return.

  This at least avoids "are these variables active at this time?"
  questions, but doesn't give us the full benefit of RAII as we must
  manually ensure FIFO create/destroy orders still.

One thing to note is that by keeping the `outputLock` unlocking in
`tryLocalBuild`, we are arguably uncovering a rebuild scheduling vs
building distinction, as the output locks are pretty squarely a
scheduling concern. It's nice that the builder doesn't need to know
about them at all.
2025-04-16 17:39:45 -04:00
John Ericson ae7f411a18 Remove some unused includes
This is unreleated to the other commits in this PR.
2025-04-16 17:39:22 -04:00
John Ericson e83ef7a477 Make appendLogTailErrorMsg as class method after all
The other parameters it took were somewhat implementation-specific.
2025-04-16 15:40:59 -04:00
Jörg Thalheim 89ba6dff66 Merge pull request #13032 from obsidiansystems/more-outputs-test
Improve derivation "advanced attrs" test, fix almost-bug
2025-04-15 19:08:35 +02:00
Jörg Thalheim 009ff8e5a7 Merge pull request #13013 from NaN-git/fix-ceil-floor
libexpr: fix UB in builtins.ceil and builtins.floor
2025-04-15 18:56:46 +02:00
Jörg Thalheim cf6da9234c Merge pull request #12410 from obsidiansystems/derivation-options-2
Scrap `ParsedDerivation` for parts
2025-04-15 18:36:11 +02:00
John Ericson 32409dd7d7 Remove stray assignment side affect in lambda
This was almost a bug! It wasn't simply because another assignment would
clobber it later.
2025-04-15 12:29:17 -04:00
John Ericson a0b2b75f59 Derivation "advanced attrs" test: Ensure fields are set to distinct values
We had fields set to the same values before in our test data. This is
not a problem per-se, but does mean we wouldn't catch certain mixups.
Now, the fields are set to distinct values (where possible), which makes
the test more robust.
2025-04-15 12:29:14 -04:00
Jörg Thalheim 99a16c5203 Merge pull request #13026 from roberth/issue-13018
Fix issue #13018, `sourceInfo` strict in `outputs`
2025-04-15 13:51:02 +02:00
Jörg Thalheim 22c928f9c0 Merge pull request #12992 from DeterminateSystems/input-cache
Rename FlakeCache to InputCache and move it to libfetchers
2025-04-15 13:32:39 +02:00
Jörg Thalheim 13a0853f04 Merge pull request #13024 from obsidiansystems/derivation-options-exportReferencesGraph
Move `exportReferencesGraph` to `DerivationOptions`
2025-04-15 13:28:00 +02:00
Jörg Thalheim 995d4f4aa7 Merge pull request #13022 from obsidiansystems/derivation-options
Test derivation options with content-addressing too
2025-04-15 13:27:09 +02:00
Robert Hensing 2109a5a206 fix: Evaluate flake parent source without evaluating its outputs
This requires that we refer to the `sourceInfo` instead of the
`result`. However, `sourceInfo` does not create a chain of basedir
resolution, so we add that back with `flakeDir`.
2025-04-15 13:22:53 +02:00
Robert Hensing 9de9410f29 call-flake.nix: allNodes.${key} -> allNodes.${key}.result 2025-04-15 13:22:53 +02:00
Robert Hensing 674375b021 call-flake.nix: refactor: Bring mapAttrs into scope 2025-04-15 13:22:53 +02:00
John Ericson d8be4f618f Scrap ParsedDerivation for parts
Only a much smaller `StructuredAttrs` remains, the rest is is now moved
to `DerivationOptions`.

This gets us quite close to `std::optional<StructuredAttrs>` and
`DerivationOptions` being included in `Derivation` as fields.
2025-04-14 16:14:41 -04:00
John EricsonandHaeNoe 1e31b60043 Limit ParsedDerivation just to the derivation's environment
This moves us towards getting rid of `ParsedDerivation` and just having
`DerivationOptions`.

Co-Authored-By: HaeNoe <git@haenoe.party>
2025-04-14 15:46:55 -04:00
John Ericson d285b80033 Move exportReferencesGraph to DerivationOptions
Tests are updated accordingly.
2025-04-14 14:24:36 -04:00
John Ericson 307dbe9914 Test derivation options with content-addressing too
Now, both the unit and functional tests relating to derivation options
are tested both ways -- with input addressing and content-addressing
derivations.
2025-04-14 13:43:46 -04:00
John Ericson 7acc229c8f Use the same variable for content addressing in functional tests
`CONTENT_ADDRESSED` -> `NIX_TESTS_CA_BY_DEFAULT`
2025-04-14 11:20:07 -04:00
Eelco Dolstra b3aa07009d Merge pull request #13019 from NixOS/mention-blake3
Mention BLAKE3 in the Nix 2.27 release notes
2025-04-14 15:47:38 +02:00
Eelco Dolstra c0ed07755a Mention BLAKE3 in the Nix 2.27 release notes 2025-04-14 15:18:29 +02:00
Eelco Dolstra 4966217b6a Move the InputCache to EvalState 2025-04-14 14:29:14 +02:00
mergify[bot] b1fe93fab7 Merge pull request #13015 from obsidiansystems/derivation-options
`ParsedDerivation`: don't take `drvPath`
2025-04-14 11:14:34 +00:00
Jörg Thalheim 3f3fd2c94b Merge pull request #12976 from picnoir/pic/multisign
store URI: introduce multiple signatures support
2025-04-14 11:20:41 +02:00
Picnoir 7ea536fe84 Narinfo sign: multiple signatures variant
This is a small optimization used when we're signing a narinfo for
multiple keys in one go. Using this sign variant, we only compute the
NAR fingerprint once, then sign it with all the keys.
2025-04-14 10:30:47 +02:00
John Ericson 0123640009 ParsedDerivation: don't take drvPath
It is just use for adding context to errors, but we have `addTrace` to
do that. Let the callers do that instead.

The callers doing so is a bit duplicated, yes, but this will get better
once `DerivationOptions` is included in `Derivation`.
2025-04-13 18:21:13 -04:00
Philipp Otterbein 56d37656ac libexpr: fix UB in builtins.ceil and builtins.floor
tighten and fix specification of both builtins
2025-04-13 04:36:09 +02:00
mergify[bot] 0e1323c041 Merge pull request #13009 from anthowan/patch-1
Fix typo in string context docs
2025-04-12 23:57:57 +00:00
John Ericson f6df573a91 flake.lock: Update Nixpkgs
This fixes evaluation for Windows. There are unfortunately deps that
still don't build, but this can be fixed next.

Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/52faf482a3889b7619003c0daec593a1912fddc1?narHash=sha256-6hl6L/tRnwubHcA4pfUUtk542wn2Om%2BD4UnDhlDW9BE%3D' (2025-03-30)
  → 'github:NixOS/nixpkgs/f675531bc7e6657c10a18b565cfebd8aa9e24c14?narHash=sha256-gbl9hE39nQRpZaLjhWKmEu5ejtQsgI5TWYrIVVJn30U%3D' (2025-04-09)
2025-04-12 19:37:20 -04:00
Anthony Wang f64b8957c7 Fix typo in string context docs 2025-04-12 19:17:27 -04:00
Jörg Thalheim 71567373b6 Merge pull request #13008 from Mic92/aliases
Move alias support from NixArgs to MultiCommand + test
2025-04-12 11:06:09 +02:00
Jörg Thalheim b1b75e1d7a tests/functional: add test for alias commands 2025-04-12 10:20:30 +02:00
Jörg Thalheim 837f349a99 Merge pull request #12999 from fricklerhandwerk/language
Nix language reference: highlight characteristic features
2025-04-12 10:13:47 +02:00
Jörg Thalheim 3094e39dc4 Merge pull request #12646 from whatsthecraic/feature/s3-auth-iam-role
S3: opt-in the STSProfileCredentialsProvider
2025-04-12 00:30:22 +02:00
Dean De Leo 51073607ea S3: opt-in the STSProfileCredentialsProvider
The STSProfileCredentialsProviders allows to assume a specific IAM role
when accessing an S3 repository. Sometimes this is needed to obtain the
permissions to operate on the bucket.
2025-04-11 23:47:56 +02:00
Jörg Thalheim a46ca4f02f Merge pull request #12538 from NaN-git/fix-s3-progress
libstore S3: fix progress bar and make file transfers interruptible
2025-04-11 23:44:53 +02:00
Philipp Otterbein 49f757c24a add isInterrupted() call and replace some checkInterrupt() occurrences 2025-04-11 22:48:10 +02:00
Philipp Otterbein db297d3dda libstore: same progress bar behavior for PUT and POST requests
- no differentiation between uploads and downloads in CLI
2025-04-11 22:48:10 +02:00
Philipp Otterbein 9da01e69f9 libstore S3: fix progress bar and make file transfers interruptible 2025-04-11 22:48:10 +02:00
Robert Hensing bbfe39ef27 Merge pull request #12995 from fricklerhandwerk/link-building
glossary: link "building" from "realisation"
2025-04-11 00:15:12 +02:00
Robert Hensing 7a0e38a1aa Merge pull request #12998 from fricklerhandwerk/packages
glossary: refine the definition of "package"
2025-04-11 00:14:23 +02:00
Valentin Gagarin bf65bc7eb7 Nix language reference: highlight characteristic features
Nix shipping with Yet Another Programming Language is often questioned
among beginners. This change highlights distinctive aspects of the Nix
language to ease the learning curve and better orient readers around
what really matters for using Nix.

Since it's on topic, this change also polishes the wording on the motivation for string contexts.
2025-04-10 23:50:55 +02:00
Valentin Gagarin a0d3003bf2 glossary: refine the definition of "package"
This change follows the definition from aptitude, but using precise notions from Nix:

> package managers deal with packages: collections of files that are
> bundled together and can be installed and removed as a group.
> [...]
> If a package A depends upon another package B, then B is required
> for A to operate properly.
> [...]
> The job of a package manager is to present an interface which assists
> the user in managing the collection of packages installed on his or her system.
>
> -- <https://www.debian.org/doc/manuals/aptitude/pr01s02.en.html>

An interesting addition:

> Packages are abstractions defining the granularity at which users can act
> (add, remove, upgrade, etc.) on available software.
> A distribution is a collection of packages maintained (hopefully) coherently.
>
> -- Package Upgrades in FOSS Distributions: Details and Challenges
>    (Roberto Di Cosmo, Stefano Zacchiroli; 2009) <https://arxiv.org/pdf/0902.1610>

Notably these quotes and this change don't say anything about installation,
or what it means for software to be available. In practice, this is
handled downstream, e.g. in NixOS or Home Manager. Nix historically
provides rudimentary facilities for package management such as
`nix-env`, but I claim they are widely agreed upon being discouraged,
with plenty of arguments provided in <https://stop-using-nix-env.privatevoid.net>.

Similarly, the specific structure of packages is determined downstream,
since Nix is policy-free:

> Nix is policy-free; it provides mechanisms to implement various deployment policies, but does not enforce a specific one.
>
> -- The Purely Functional Software Deployment Model (Eelco Dolstra; 2006) <https://edolstra.github.io/pubs/phd-thesis.pdf>

Specifically, Nix mechanisms do not define what a package is supposed to be:

> It's worth noting that the Nix language is intended as a DSL for package and configuration management, but it has no notions of "packages" or "configurations".
>
> -- <https://gist.github.com/edolstra/29ce9d8ea399b703a7023073b0dbc00d>

This is why we say, Nix *allows* denoting packages in a certain way, but
doesn't enforce any particular way.
2025-04-10 22:50:28 +02:00
Eelco Dolstra 6cd2b4e169 Move alias support from NixArgs to MultiCommand
This allows subcommands to declare aliases, e.g. `nix store ping` is
now a proper alias of `nix store info`.
2025-04-10 19:48:55 +02:00
Valentin Gagarin fc5b1d2344 glossary: link "building" from "realisation" 2025-04-10 16:56:19 +02:00
Eelco Dolstra e3042f10af Move getAccessorCached() to InputCache 2025-04-10 10:24:15 +02:00
Eelco Dolstra 012453d1e6 Move the input cache into libfetchers 2025-04-10 10:24:15 +02:00
Eelco Dolstra c7f8147282 Rename FlakeCache -> InputCache and key it on Inputs instead of FlakeRefs 2025-04-10 10:24:15 +02:00
Eelco Dolstra 3f3cc6f438 Merge pull request #12991 from Mic92/cgroup
linux/cgroup: delete double quote in error message
2025-04-10 10:11:24 +02:00
Eelco Dolstra 26cb166bca Merge pull request #12531 from obsidiansystems/store-accessor-root
`Store::getFSAccessor`: Do not include the store dir
2025-04-10 09:55:03 +02:00
Jörg Thalheim b3f80283a6 linux/cgroup: delete double quote in error message 2025-04-10 09:31:48 +02:00
Eelco Dolstra 9d3595646d nix shell: Resolve symlinks in storeFS
`storeFS` is the `MountedSourceAccessor` that wraps `store->getFSAccessor()`.
2025-04-09 17:34:19 -04:00
John EricsonandEelco Dolstra eb643d034f Store::getFSAccessor: Do not include the store dir
Rather than "mounting" the store inside an empty virtual filesystem,
just return the store as a virtual filesystem. This is more modular.

(FWIW, it also supports two long term hopes of mind:

1. More capability-based Nix language mode. I dream of a "super pure
   eval" where you can only use relative path literals (See #8738), and
   any `fetchTree`-fetched stuff + the store are all disjoint (none is
   mounted in another) file systems.

2. Windows, where the store dir may include drive letters, etc., and is
   thus unsuitable to be the prefix of any `CanonPath`s.

)

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-04-09 17:34:18 -04:00
Jörg Thalheim 2455bda91b Merge pull request #12986 from obsidiansystems/fix-machine-commment-semicolon
Fix another machine config parsing bug
2025-04-09 22:12:26 +02:00
John Ericson b74b0f4e1c Fix another machine config parsing bug
We were ignorning the result of `trim`, and after my last change we were
also trimmming too early.
2025-04-09 15:33:09 -04:00
Jörg Thalheim d45067177e Merge pull request #12984 from obsidiansystems/fix-machine-commment-semicolon
Fix `;` and `#` bug in machine file parsing
2025-04-09 20:26:03 +02:00
Robert Hensing fc77a89d53 Merge pull request #12699 from rvl/subflake-locking-test
tests/functional/flakes: Add test case for subflake locking
2025-04-09 19:19:34 +02:00
John Ericson f8b13cce19 Fix ; and # bug in machine file parsing
Comments go to the end of the line, not merely the next ; *or* \n. Fix
by splitting on `;` *within* lines, and test.
2025-04-09 12:38:42 -04:00
Jörg Thalheim ea48a318ba Merge pull request #12979 from DeterminateSystems/fix-ignore-local-registrie
Actually ignore system/user registries during locking
2025-04-09 18:32:54 +02:00
Jörg Thalheim 6d50980c2f Merge pull request #12978 from SuperSandro2000/patch-2
Fix meson warning about meson_version 1.1
2025-04-09 18:31:47 +02:00
Eelco Dolstra 77d4316353 Actually ignore system/user registries during locking
Something went wrong in #12068 so this didn't work. Also added a test.
2025-04-09 17:59:51 +02:00
Sandro f9a8fdecf5 Fix meson warning about meson_version 1.1
meson.options requires that we set meson_version to at least 1.1

similar to #12956
2025-04-09 15:31:34 +02:00
Picnoir e12369a68e store URI: introduce multiple signatures support
Add a `secretKeyFiles` URI parameter in the store URIs receiving a
coma-separated list of Nix signing keyfiles.

For instance:

  nix copy --to "file:///tmp/store?secret-keys=/tmp/key1,/tmp/key2" \
    "$(nix build --print-out-paths nixpkgs#hello)"

The keys passed through this new store URI parameter are merged with
the key specified in the `secretKeyFile` parameter, if any.

We'd like to rotate the signing key for cache.nixos.org. To simplify
the transition, we'd like to sign the new paths with two keys: the new
one and the current one. With this, the cache can support nix
configurations only trusting the new key and legacy configurations
only trusting the current key.

See https://github.com/NixOS/rfcs/pull/149 for more informations
behind the motivation.
2025-04-09 13:30:37 +02:00
Robert Hensing e76bbe413e Merge pull request #12877 from roberth/c-api-libflake-override-input
C API: Flake loading, input overriding
2025-04-08 09:01:51 +02:00
John Ericson 73d3159ba0 Merge pull request #12967 from DeterminateSystems/trailing-commas
Add trailing commas on addFlag incantations
2025-04-07 19:14:31 -04:00
Graham Christensen 9b47b2b217 format as required 2025-04-07 17:24:41 -04:00
Graham Christensen 06acbd37bd Add trailing commas on addFlag incantations 2025-04-07 17:18:15 -04:00
John Ericson 596389a5f6 Merge pull request #12958 from DeterminateSystems/unlinked-tests
Fix some unlinked tests
2025-04-07 16:41:07 -04:00
John Ericson 6789202af0 Merge pull request #12962 from DeterminateSystems/clang-tidy-warnings
Fix some clang-tidy warnings
2025-04-07 16:37:02 -04:00
John Ericson 3bcc70b64f Merge pull request #12959 from DeterminateSystems/fchmodat2-compat-private
Keep fchmodat2-compat.hh private
2025-04-07 15:17:46 -04:00
John Ericson 856f8e0e49 Merge pull request #12961 from DeterminateSystems/lexer-helpers-private
Make lexer-helpers.hh internal to fix a clang-tidy error
2025-04-07 14:28:14 -04:00
Eelco Dolstra c0ad5d36c4 Fix some clang-tidy warnings 2025-04-07 19:55:33 +02:00
Eelco Dolstra d1ba01235f Merge pull request #12956 from SuperSandro2000/patch-2
Fix meson warnings on minimum version
2025-04-07 19:50:36 +02:00
Eelco Dolstra 8be24f58f2 Make lexer-helpers.hh internal to fix a clang-tidy error 2025-04-07 18:21:08 +02:00
Eelco Dolstra 04e9dc27ac Keep fchmodat2-compat.hh private
Since it references store-config-private.hh.
2025-04-07 18:19:42 +02:00
Eelco Dolstra 611fd806cb Remove unused tracing-file-system-object-sink.{hh,cc} 2025-04-07 17:10:28 +02:00
Eelco Dolstra 340fa00d52 Fix/run monitorfdhup test 2025-04-07 17:09:42 +02:00
Sandro 14a829acbb Fix meson warnings on minimum version
nix> meson.build:216: WARNING: Project targets '>= 1.1' but uses feature introduced in '1.4.0': fs.name with build_tgt, custom_tgt, and custom_idx.
nix> meson.build:222: WARNING: Project targets '>= 1.1' but uses feature introduced in '1.4.0': fs.name with build_tgt, custom_tgt, and custom_idx.
nix> meson.build:235: WARNING: Project targets '>= 1.1' but uses feature introduced in '1.4.0': fs.name with build_tgt, custom_tgt, and custom_idx.
nix> meson.build:236: WARNING: Project targets '>= 1.1' but uses feature introduced in '1.4.0': fs.name with build_tgt, custom_tgt, and custom_idx.
nix> meson.build:242: WARNING: Project targets '>= 1.1' but uses feature introduced in '1.4.0': fs.name with build_tgt, custom_tgt, and custom_idx.
2025-04-07 15:06:10 +02:00
Jörg Thalheim 2960d7adf5 Merge pull request #12951 from Mic92/fix-race-condition
tests/functional/repl: fix race condition
2025-04-07 14:18:05 +02:00
Jörg Thalheim 1de951d31d tests/functional/repl: fix race condition
the sleep 1 is not enough in some circumstances. Switching to a fifo
helps.
2025-04-07 14:00:19 +02:00
Jörg Thalheim 3ff19be1f7 Merge pull request #12944 from roberth/maintainers-release-notes-fail
maintainers/release-notes: Let it fail
2025-04-07 10:51:08 +02:00
mergify[bot] 07204ff6e5 Merge pull request #12940 from NixOS/header-cleanup
Header cleanup
2025-04-07 00:09:49 +00:00
Robert Hensing da36c34db7 maintainers/release-notes: Let it fail
Fail when a command fails.

Basic error handling was missing, which would lead to errors getting
obscured a bit by subsequent successful logging.
2025-04-07 01:46:43 +02:00
mergify[bot] 1dc7e6c4dc Merge pull request #12936 from ajlekcahdp4/master
libflake: add lock file path to invalid json error
2025-04-06 23:32:50 +00:00
John Ericson 7a7fe350d5 Get rid of raw -D defines, always use private config files
Now that we have the private vs public distinction, we can do this
without leaking information downstream.
2025-04-06 18:53:42 -04:00
John Ericson 3294b22a68 Clean some header related things.
Revert most of "Hack together a fix for the public headers"

- The `libmain` change is kept, and one more libmain change is made.
  (Need to update Meson and Nix per the package alike).

- The S3 situation is fixed in a different way: the variable is public
  now, used in the header, and fixed accordingly.

- Fix TODO for `HAVE_EMBEDDED_SANDBOX_SHELL`

This reverts commit 2b51250534.
2025-04-06 18:53:38 -04:00
mergify[bot] 3fcdccb8ce Merge pull request #12939 from NixOS/2.28-release-notes
2.28 release notes for master
2025-04-06 21:41:00 +00:00
Robert Hensing 78e2832d7d Edit rl-2.28
(cherry picked from commit 1ca3ee1287)
2025-04-06 17:03:38 -04:00
Robert Hensing 4ac9bc08ea Fix maintainers/release-credits output
(cherry picked from commit b87b3d79f2)
2025-04-06 17:03:36 -04:00
Robert Hensing 93248bd92c doc/rl-2.28: Add contributors
(cherry picked from commit fea87a94e6)
2025-04-06 17:03:36 -04:00
Robert Hensing f261c03686 chore: Update contributor handle caches
(cherry picked from commit 6687ce2a6d)
2025-04-06 17:03:35 -04:00
John Ericson 682a28a76b release notes: 2.28.0
(cherry picked from commit 703f0fbe74)
2025-04-06 17:03:34 -04:00
Alexander Romanov e3873aa1a0 libflake: add lock file path to invalid json error
Previously, when lock file contained invalid JSON nix reported a parser
error without specifying the file it came from.

This change adds flake.lock file path to the error message to avoid
confusion.
2025-04-06 22:52:46 +03:00
mergify[bot] a56aaf26c2 Merge pull request #12937 from roberth/undefined-macros
Fix undefined macro errors
2025-04-06 16:25:22 +00:00
Robert Hensing 77b4bb74d5 Fix undefined macro errors 2025-04-06 17:43:17 +02:00
Jörg Thalheim cf5e59911b Merge pull request #12931 from roberth/add-Wundef-and-fix-public-headers
Add `-Wundef` and fix public headers
2025-04-05 08:43:43 +02:00
Robert Hensing ba89da8fa2 Fix more -Wundef, in darwin context 2025-04-05 01:04:58 +02:00
Robert Hensing 2b51250534 Hack together a fix for the public headers
Please fix this.
2025-04-05 00:59:58 +02:00
Robert Hensing 59ced3da96 Add -Wundef to make #if FOO an error if not defined
This commit has all the straightforward stuff.
2025-04-05 00:45:19 +02:00
Robert Hensing 8bd342c660 Merge pull request #12911 from Mic92/no-copy-twice
create cache entry for paths already in the nix store
2025-04-04 13:41:36 +02:00
Jörg Thalheim 61c6210dbf create cache entry for paths already in the nix store
This allows path:/nix/store/* paths to not be copied twice to the nix
store.
2025-04-04 12:45:22 +02:00
Jörg Thalheim 6c3476784b Merge pull request #12919 from roberth/darwin-minversion
Reinstate Darwin minversion
2025-04-04 08:38:22 +02:00
Robert Hensing 4be92e7b82 packaging/dependency: Clarify darwinMinVersion 2025-04-03 23:22:33 +02:00
Robert Hensing 5c4a4aeed7 Revert "remove obsolete stdenv darwinMinVersion override"
This reverts commit d91310bb32.

> Some packages require setting a non-default deployment target
> (or minimum version) to gain access to certain APIs. You do
> that using the darwinMinVersionHook, which takes the deployment
> target version as a parameter.

-- https://github.com/NixOS/nixpkgs/blob/60b54c7aee3c0cefde72d1a151bb7d3a46361ca2/doc/stdenv/platform-notes.chapter.md#what-is-a-deployment-target-or-minimum-version-sec-darwin-troubleshooting-using-deployment-targets

This will again solve error:

    ../nix_api_expr.cc:38:18: error: aligned allocation function of type 'void *(std::size_t, std::align_val_t)' is only available on macOS 10.13 or newer

-- https://hydra.nixos.org/build/294088946
2025-04-03 23:22:19 +02:00
Jörg Thalheim cb972c789a Merge pull request #12912 from DeterminateSystems/substituted-path-display
Set path display for substituted inputs
2025-04-03 14:49:51 +02:00
Eelco Dolstra 4a397cfb80 Set path display for substituted inputs 2025-04-03 14:14:22 +02:00
Jörg Thalheim 5e19252de9 Merge pull request #12902 from Mic92/signed-comparison
Fix -Wsign-compare errors
2025-04-03 10:30:18 +02:00
Jörg Thalheim c0912a12bf Merge pull request #12846 from allrealmsoflife/clearer-continuation-prompt
repl: improve continuation prompt for incomplete expressions
2025-04-03 09:52:16 +02:00
Jörg Thalheim bbc4977fed local-derivation-goal: fix sized comparison when checking sandbox profile length 2025-04-03 09:50:53 +02:00
Jörg Thalheim 681d7f7e57 Fix -Wsign-compare errors 2025-04-03 09:48:53 +02:00
mergify[bot] c4b81184e4 Merge pull request #12886 from Mic92/symlink-exists
symlink_exists: wrap exceptions into nix exception
2025-04-02 23:13:24 +00:00
mergify[bot] 1831948c20 Merge pull request #12885 from NixOS/fix-windows-build
Fix windows build
2025-04-02 22:34:41 +00:00
Jörg Thalheim 779687854f symlink_exists: wrap exceptions into nix exception 2025-04-02 23:48:16 +02:00
Jörg Thalheim 2ace512a70 Merge pull request #12896 from Mic92/no-dangling-reference
Fix -Wdangling-reference
2025-04-02 23:45:55 +02:00
Eelco Dolstra 0f723769b7 Fix -Wdangling-reference 2025-04-02 20:51:45 +00:00
Robert Hensing 33e638dc1d Merge pull request #12863 from Mic92/libgit2
libgit2: use upstream version if possible
2025-04-02 22:44:42 +02:00
Robert Hensing ff328e1dad Merge pull request #12876 from Mic92/connect-timeout
decrease connect-timeout to 5s
2025-04-02 22:42:36 +02:00
Jörg Thalheim d81b51d84b Merge pull request #12888 from DeterminateSystems/fix-maintainers
Fix jobset evaluation
2025-04-02 22:36:29 +02:00
Eelco Dolstra 93d8f62057 Remove meta.maintainers
Some of the maintainer attribute names got changed in nixos-unstable
(e.g. "edolstra" is now "eelco") but we want this flake to work on
nixos-24.11. So just get rid of them.
2025-04-02 22:02:15 +02:00
Eelco Dolstra 7eb76186ba Update meta.maintainers field for nixos-unstable 2025-04-02 21:39:02 +02:00
John Ericson 652a628d1c Fix windows build 2025-04-02 15:17:26 -04:00
Jörg Thalheim 26b33aa1b0 Merge pull request #12869 from DeterminateSystems/empty-git-repo
Apply makeNotAllowedError to empty Git repos
2025-04-02 20:32:18 +02:00
Eelco Dolstra 67e957b636 Apply makeNotAllowedError to empty repos 2025-04-02 19:57:49 +02:00
Robert Hensing 8c903e0402 nix-flake-c: Add lock flags
Going with a slightly more limited, high level API supporting the
three main use cases.
This should allow the underlying code to evolve more freely.
2025-04-02 18:29:42 +02:00
Robert Hensing 1a3789e222 fix: nix_clear_err in nix_flake_* functions 2025-04-02 18:29:42 +02:00
Robert Hensing a0a1d00370 nix-flake-c: Add basic flakeref parsing and locking 2025-04-02 18:29:42 +02:00
Robert Hensing 60bffbd41b nix-fetchers-c: Init with settings object
Also make it a dependency of nix-flake-c; we'll need that.
2025-04-02 18:29:42 +02:00
Robert Hensing 1061a0965a nix-flake-c: Add missing bits 2025-04-02 18:02:32 +02:00
Robert Hensing 05e5bd2140 Docs 2025-04-02 18:02:32 +02:00
Robert Hensing 02360dd65c nix-expr: Expose nix_api_expr_internal.h intentionally
This is required for other bindings like nix-flake-c to hook into
nix-expr-c appropriately.
The `_internal` part should be a sufficient deterrent normally,
and it may also be useful for bindings that migrate from the C++
interface.
2025-04-02 18:02:32 +02:00
Robert Hensing 60b4b220d8 test: Fixup test name 2025-04-02 18:02:32 +02:00
Robert Hensing 0dc9b6b7c7 libutil-tests-support: Add file/line to ctx errors 2025-04-02 18:02:32 +02:00
Jörg Thalheim bef91a618a decrease connect-timeout to 5s
For people self-hosting caches that can be occasionally down, the
default timeout is very long. This is annoying if you are trying to
update your binary cache at the same time you are trying to update
another machine. Same if cachix has one of its rare hiccups.

We tested this value of 5s in srvos now for years and we like to travel
around the world with shitty internet, so it should be still reasonable
high.
2025-04-02 18:01:19 +02:00
mergify[bot] 70dcd738ef Merge pull request #12870 from DeterminateSystems/improve-git-error
Git fetcher: Improve error message for untracked files
2025-04-02 15:39:35 +00:00
Jörg Thalheim 7a6570a11c Merge pull request #12836 from NixOS/component-in-header-path
Expose the nix component in header include paths
2025-04-02 15:29:22 +02:00
Eelco Dolstra f15681df26 Make Git error messages more consistent 2025-04-02 12:58:53 +02:00
Eelco Dolstra 277c29a64b Tweak error message 2025-04-02 12:58:11 +02:00
Graham Christensen 62e2304891 Improve and fix the error message when a file is not tracked by Git 2025-04-02 12:58:06 +02:00
Robert Hensing cf409fd250 Merge pull request #12868 from NixOS/bump-2.29.0
Bump version
2025-04-02 00:23:30 +02:00
John Ericson fc1de4a610 Bump version 2025-04-01 14:26:00 -04:00
mergify[bot] 9ed5482545 Merge pull request #12810 from roberth/packaging-sync
Packaging improvements from Nixpkgs
2025-04-01 17:49:35 +00:00
Jörg Thalheim d91310bb32 remove obsolete stdenv darwinMinVersion override
we are more up-to-date now:

nix-repl> stdenv.hostPlatform.darwinMinVersion
"11.3"
2025-04-01 19:20:43 +02:00
Jörg Thalheim 0b61b758fb libgit2: use upstream version if possible
we don't seem to use libgit2 for fetching via ssh, hence it shouldn't
matter if it's using libssh or the ssh binary.
2025-04-01 19:09:21 +02:00
Robert HensingandPeder Bergebakken Sundt 27d71b21fc packaging: finalAttrs.doCheck -> finalAttrs.finalPackage.doCheck
This includes the logic that disables checks on cross appropriately.

Co-authored-by: Peder Bergebakken Sundt <pbsds@hotmail.com>
2025-04-01 18:51:09 +02:00
Robert HensingandMic92 1172e49a3a packaging: Various improvements
Co-authored-by: Mic92 <Mic92@users.noreply.github.com>
2025-04-01 18:51:09 +02:00
Jörg Thalheim c57e2486df Merge pull request #12853 from roberth/flake-nixos-unstable
flake: nixpkgs: 24.11 -> nixos-unstable
2025-04-01 18:49:10 +02:00
John Ericson cc24766fa6 Expose the nix component in header include paths
For example, instead of doing

    #include "nix/store-config.hh"
    #include "nix/derived-path.hh"

Now do

    #include "nix/store/config.hh"
    #include "nix/store/derived-path.hh"

This was originally planned in the issue, and also recent requested by
Eelco.

Most of the change is purely mechanical. There is just one small
additional issue. See how, in the example above, we took this
opportunity to also turn `<comp>-config.hh` into `<comp>/config.hh`.
Well, there was already a `nix/util/config.{cc,hh}`. Even though there
is not a public configuration header for libutil (which also would be
called `nix/util/config.{cc,hh}`) that's still confusing, To avoid any
such confusion, we renamed that to `nix/util/configuration.{cc,hh}`.

Finally, note that the libflake headers already did this, so we didn't
need to do anything to them. We wouldn't want to mistakenly get
`nix/flake/flake/flake.hh`!

Progress on #7876
2025-04-01 11:40:42 -04:00
Robert Hensing 58b657b976 tests/nixos: Work around network-online.target inactivity 2025-04-01 16:36:47 +02:00
Robert Hensing 4de73df8bf Merge pull request #12765 from Mic92/clan-tidy-cleanups
Add various clang tidy fixes
2025-04-01 15:36:50 +02:00
Robert Hensing 55297f865c Format
clang-format: 18.1.8 -> 19.1.7
2025-04-01 15:33:49 +02:00
Robert Hensing c212035d94 flake: nixpkgs: 24.11 -> nixos-unstable
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/48d12d5e70ee91fe8481378e540433a7303dbf6a?narHash=sha256-1Noao/H%2BN8nFB4Beoy8fgwrcOQLVm9o4zKW1ODaqK9E%3D' (2024-12-16)
  → 'github:NixOS/nixpkgs/52faf482a3889b7619003c0daec593a1912fddc1?narHash=sha256-6hl6L/tRnwubHcA4pfUUtk542wn2Om%2BD4UnDhlDW9BE%3D' (2025-03-30)
2025-04-01 15:19:54 +02:00
mergify[bot] cfc4db85a5 Merge pull request #12850 from Mic92/header
Revert "Create script to symlink headers to old location"
2025-04-01 12:50:04 +00:00
mergify[bot] fadd86011f Merge pull request #12814 from silvanshade/link-with-mold
Use mold instead of GNU ld for faster builds
2025-04-01 12:09:05 +00:00
Jörg Thalheim c69e11ac46 Revert "Create script to symlink headers to old location"
This reverts commit c6a176be62.

Polluting the source tree with untracked symlinks that can end up
dangling may be unsound.
2025-04-01 13:55:41 +02:00
Jörg Thalheim cfeb9404a8 remove deprecated gold linker
Binutils deprecated the gold linker, so do we.
For development we now use mold instead,
we can potentially also use it for release builds later.
2025-04-01 12:52:29 +02:00
Eelco Dolstra db70061f0b Merge pull request #12809 from DeterminateSystems/handle-failed-cached-values
Throw CachedEvalError if a cached value exists but has type "failed"
2025-04-01 11:49:09 +02:00
Eelco Dolstra 2ec1eba7ca Merge remote-tracking branch 'origin/master' into handle-failed-cached-values 2025-04-01 11:00:27 +02:00
allrealmsoflife d3ebbb37e8 repl: improve continuation prompt for incomplete expressions
Previously, when users entered an incomplete expression in the REPL,
the continuation prompt was just 10 blank spaces, which looked invisible
and gave the impression that the REPL had stalled.

This change updates the prompt to "         > ", aligning it visually
with 'nix-repl> ' and clearly indicating that the REPL is waiting for
more input.

Fixes: https://github.com/NixOS/nix/issues/12702
2025-04-01 10:25:40 +02:00
mergify[bot] 3f5f2c4734 Merge pull request #12798 from obsidiansystems/clean-config-headers
Cleanup config headers
2025-04-01 07:47:32 +00:00
Jörg Thalheim 6fe39566d2 Merge pull request #12805 from thomie/nix-daemon-source-once
nix-daemon: source nix-profile-daemon.sh only once
2025-04-01 08:46:13 +02:00
John Ericson c204e307ac Cleanup config headers
There are two big changes:

1. Public and private config is now separated. Configuration variables
   that are only used internally do not go in a header which is
   installed.

   (Additionally, libutil has a unix-specific private config header,
   which should only be used in unix-specific code. This keeps things a
   bit more organized, in a purely private implementation-internal way.)

2. Secondly, there is no more `-include`. There are very few config
   items that need to be publically exposed, so now it is feasible to
   just make the headers that need them just including the (public)
   configuration header.

And there are also a few more small cleanups on top of those:

- The configuration files have better names.

- The few CPP variables that remain exposed in the public headers are
  now also renamed to always start with `NIX_`. This ensures they should
  not conflict with variables defined elsewhere.

- We now always use `#if` and not `#ifdef`/`#ifndef` for our
  configuration variables, which helps avoid bugs by requiring that
  variables must be defined in all cases.
2025-03-31 23:28:36 -04:00
Jörg Thalheim 0e1da295e9 Merge pull request #12817 from NixOS/link-header-script
Create script to symlink headers to old location
2025-04-01 00:48:01 +02:00
John Ericson c6a176be62 Create script to symlink headers to old location
See comments on the script; this is supposed to avoid breaking muscle
memory without complicating the build system (which proved harder than I
thought too) or not doing the header hygiene change at all.

link-headers: use pathlib consistenly and fix type errors
2025-03-31 21:32:27 +02:00
mergify[bot] 5a8dedc45c Merge pull request #12764 from NixOS/header-file-namespacing
Separate headers from source files
2025-03-31 18:03:22 +00:00
John Ericson 38dfb40fdb Merge pull request #12815 from NixOS/mergify-yml-2.28
.mergify.yml: Add backport 2.28-maintenance entry
2025-03-31 13:18:11 -04:00
silvanshade 3546e574ca Use mold instead of GNU ld for faster builds 2025-03-31 10:38:11 -06:00
Jörg Thalheim 97ff7330ef Merge pull request #12812 from DeterminateSystems/disable-packaging-overriding
Disable packaging-overriding test
2025-03-31 18:30:23 +02:00
John Ericson f3e1c47f47 Separate headers from source files
The short answer for why we need to do this is so we can consistently do
`#include "nix/..."`. Without this change, there are ways to still make
that work, but they are hacky, and they have downsides such as making it
harder to make sure headers from the wrong Nix library (e..g.
`libnixexpr` headers in `libnixutil`) aren't being used.

The C API alraedy used `nix_api_*`, so its headers are *not* put in
subdirectories accordingly.

Progress on #7876

We resisted doing this for a while because it would be annoying to not
have the header source file pairs close by / easy to change file
path/name from one to the other. But I am ameliorating that with
symlinks in the next commit.
2025-03-31 12:20:25 -04:00
John Ericson 326548bae5 Cleanup config header for libcmd
- Since it's now private, give it a rename. Note that I want to switch the
  word order on the public ones too.

- Since it is only needed by two files, just include there rather than
  the nasty blanket-forced thing.
2025-03-31 12:20:25 -04:00
John Ericson c29b2d1058 .mergify.yml: Add backport 2.28-maintenance entry 2025-03-31 12:15:28 -04:00
Eelco Dolstra a4be66828a Disable packaging-overriding
Fixes #12690.
2025-03-31 16:37:36 +02:00
Eelco Dolstra 5a35745949 AttrCursor::Parent: shared_ptr -> ref 2025-03-31 15:14:10 +02:00
Eelco Dolstra 8b438fccb4 Throw CachedEvalError if a cached value exists but has type "failed"
Otherwise you get unhelpful errors like

  error: 'apps' is not an attribute set

Fixes #12762.
2025-03-31 15:11:39 +02:00
Thomas Miedema 2b4e3fa144 nix-daemon: source nix-profile-daemon.sh only once
On my system (Ubuntu 24.04 with nix installed using
https://zero-to-nix.com/), I noticed that my PATH
contained multiple times the following entries:

  /home/thomas/.nix-profile/bin
  /nix/var/nix/profiles/default/bin

Fix it by inserting a missing `export`, to make
sure `nix-daemon.sh` is really only executed once.
2025-03-29 20:23:16 +01:00
John Ericson 3f13cc0f87 Merge pull request #12794 from obsidiansystems/fix-windows-build
Fix windows build
2025-03-28 21:04:14 -04:00
John Ericson 99041b4d84 Fix windows build
PR #12767 accidentally broke it.
2025-03-28 20:04:51 -04:00
John Ericson 1cfbd489f5 Merge pull request #12789 from DeterminateSystems/nix-daemon-no-store
nix daemon: Don't open the store
2025-03-28 20:01:11 -04:00
Eelco Dolstra 9590167290 nix daemon: Don't open the store
This makes it behave the same as nix-daemon. Opening the store in the
parent can cause a SIGBUS in libsqlite in the child:

  #0  0x00007f141cf6f789 in __memset_avx2_unaligned_erms () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
  #1  0x00007f141c322fe8 in walIndexAppend () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
  #2  0x00007f141c3711a2 in pagerWalFrames () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
  #3  0x00007f141c38317e in sqlite3PagerCommitPhaseOne.part.0 () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
  #4  0x00007f141c383555 in sqlite3BtreeCommitPhaseOne.part.0 () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
  #5  0x00007f141c384797 in sqlite3VdbeHalt () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
  #6  0x00007f141c3b8f60 in sqlite3VdbeExec () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
  #7  0x00007f141c3bbfef in sqlite3_step () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
  #8  0x00007f141c3bd0e5 in sqlite3_exec () from /nix/store/bbd59cqw259149r2ddk4w1q0lr2fch8c-sqlite-3.46.1/lib/libsqlite3.so.0
  #9  0x00007f141da140e0 in nix::SQLiteTxn::commit() () from /nix/store/1m4r8s7s1v54zq9isncvjgia02bffxlz-determinate-nix-store-3.1.0/lib/libnixstore.so
  #10 0x00007f141d9ce69c in nix::LocalStore::registerValidPaths(std::map<nix::StorePath, nix::ValidPathInfo, std::less<nix::StorePath>, std::allocator<std::pair<nix::StorePath const, nix::ValidPathInfo> > > const&)::{lambda()#1}::operator()() const () from /nix/store/1m4r8s7s1v54zq9isncvjgia02bffxlz-determinate-nix-store-3.1.0/lib/libnixstore.so
2025-03-28 19:29:50 +01:00
John Ericson fd98f30e4e Merge pull request #12758 from roberth/lutimes-error
libutil: Fix error message
2025-03-28 13:06:43 -04:00
John Ericson f1197f7601 Merge pull request #12768 from Mic92/improve-error-messages
substitution-goal: convert assert into an Error
2025-03-28 13:05:46 -04:00
John Ericson c36a9b8e1b Merge pull request #12773 from roberth/config-h-low-hanging-fruit
Unexpose config headers (low hanging fruit only)
2025-03-28 12:43:28 -04:00
Robert Hensing a2e43cd8e6 Merge pull request #12771 from roberth/suppress-parser-warning
nix-expr/parser: Suppress warning
2025-03-28 15:55:05 +00:00
John Ericson a08b0c742a Merge pull request #12772 from NixOS/light-packaging-cleanus
Light packaging cleanups
2025-03-28 11:53:48 -04:00
Robert Hensing 1e60ebeea0 Merge pull request #12767 from Mic92/fs-stuff-2
use createDirs consistently everywhere
2025-03-28 15:53:33 +00:00
Robert Hensing b86a76044e Unexpose config headers (low hanging fruit only)
- Some headers were completely redundant and have been removed.
- Other headers have been turned private.
- Unnecessary meson.build code has been removed.
- libutil-tests now has a private config header, where previously
  it had none. This removes the need to expose a package version
  macro publicly.
2025-03-28 15:17:54 +00:00
John Ericson e4c571c2f1 Add a withAWS flag to libstore
Nixpkgs wants this, at least.
2025-03-28 11:08:15 -04:00
John Ericson fb9c9ee35a Link the right issue about the bad AWS pkg-config
It is https://github.com/aws/aws-sdk-cpp/issues/2673
2025-03-28 10:50:46 -04:00
John Ericson 459fb59493 Remove boost env vars
https://github.com/NixOS/nixpkgs/issues/86131 is now fixed!
2025-03-28 10:45:27 -04:00
John Ericson ffdce51cd5 Fix a bunch of missing meson boilerplate
These other libraries need this too
2025-03-28 10:38:08 -04:00
Robert Hensing 1df69c3920 Merge pull request #12766 from Mic92/fs-stuff
git-utils: add trace context to writeFull
2025-03-28 13:53:29 +00:00
Robert Hensing aa689b96e6 nix-expr/parser: Suppress warning
We rely on `yyerror()` instead.

> The variable yynerrs contains the number of syntax errors reported so
> far.
> Normally this variable is global; but if you request a pure parser
> (see A Pure (Reentrant) Parser) then it is a local variable which only
> the actions can access.

https://www.gnu.org/software/bison/manual/html_node/Error-Reporting-Function.html
2025-03-28 13:50:24 +00:00
John Ericson a26a15d05c Merge pull request #12759 from roberth/c-api-libflake-settings
C API / settings: remove nix-flake-c global init
2025-03-27 12:38:25 -04:00
Robert Hensing 0c75581d8b Move call-flake.nix to nix-flake
As suggested by Ericson2314 in review
https://github.com/NixOS/nix/pull/12759#issuecomment-2755352343
2025-03-27 14:29:07 +00:00
Jörg Thalheim 7a6ce75aea substitution-goal: convert assert into an Error
This is to get more context on https://github.com/NixOS/nix/issues/12761
2025-03-27 14:13:57 +01:00
Jörg Thalheim ca165f09c0 Merge pull request #12583 from ulucs/ulucs/skip-ifds
`nix flake show`: Skip IFDs instead of throwing
2025-03-27 11:40:49 +01:00
Jörg Thalheim a8217f2642 use createDirs consistently everywhere 2025-03-27 11:28:11 +01:00
Jörg Thalheim 86de15f4fc git-utils: add trace context to writeFull 2025-03-27 11:26:30 +01:00
Jörg Thalheim 5c3682d7a1 libstore/daemon: make sure monitor is not considered "unused" 2025-03-27 11:24:30 +01:00
Jörg Thalheim b050db951b test/ca-fd-leak: fix clang-tidy lints 2025-03-27 11:24:28 +01:00
Jörg Thalheim 7e540059a3 git/getStringUntil: fix uninitialized stack variable
at least clang-tidy is not convinced that this initialized.
If this is not the case, the impact should be small and hopefully also
more robust if changed.
2025-03-27 11:24:27 +01:00
Jörg Thalheim 05082ea1c5 libstore/local-store: fix linting warning about unused variable 2025-03-27 11:24:25 +01:00
Uluc Sengil fcf5966488 skip ifds in nix flake show instead of throwing 2025-03-27 11:16:35 +01:00
Robert Hensing 6fc9651d57 nix-expr: remove EvalSettings::addPrimOp, add const
Not required for a struct and potentially confusing.
2025-03-26 19:22:40 +00:00
Robert Hensing 6309bbb426 Merge pull request #12760 from Prince213/docs-unsafeGetAttrPos
docs: document unsafeGetAttrPos
2025-03-26 13:04:13 +00:00
Robert Hensing d48101109d nix-flake: Move primops registration to configureEvalSettings 2025-03-26 11:15:02 +00:00
Robert Hensing 6a192ec0cd C API: (breaking) remove nix-flake-c global init 2025-03-26 11:15:02 +00:00
Sizhe Zhao 39ce478283 docs: document unsafeGetAttrPos 2025-03-26 18:48:57 +08:00
Robert Hensing 3c4c0953e0 nix-expr: Add primops to EvalSettings 2025-03-26 09:32:43 +00:00
Robert Hensing 1cffcd91a9 libutil: Fix error message
I encountered this with a misconfigured libutil. I doubt that a
non-lutimes config is viable, because tests were failing.
2025-03-26 09:16:27 +00:00
mergify[bot] 9b0f455609 Merge pull request #12105 from trofi/local-derivation-goal-better-reference-bug
local-derivation-goal: improve "illegal reference" error
2025-03-25 14:54:27 +00:00
Sergei Trofimovich 4d72e0f73b tests/functional/check-refs.nix: format newly added test 2025-03-25 06:12:19 +00:00
Sergei Trofimovich 1e7c7244cf tests/functional/check-refs.sh: guard test12 against too old nix daemon
Otherwise without the change the test fails on nix-2.26 as:

    error: derivation contains an illegal reference specifier 'dev'

Note: the error message does not match intended change.
2025-03-25 06:12:19 +00:00
Robert Hensing f4def47c89 test illegal reference specifier error message 2025-03-25 06:12:19 +00:00
Robert Hensing 2b4d461c14 checkRefs: use concatMapStringsSep 2025-03-25 06:12:19 +00:00
Robert Hensing 32898dc46a nix-util: Use small_vector in concatMapStringsSep 2025-03-25 06:12:19 +00:00
Robert Hensing f3dbaa3f54 nix-util: Add concatMapStrings 2025-03-25 06:12:19 +00:00
Sergei Trofimovich bbdc3197a9 local-derivation-goal: improve "illegal reference" error
Before the change "illegal reference" was hard to interpret as it did
not mention what derivation actually hits it.

Today's `nixpkgs` example:

Before the change:

    $ nix build --no-link -f. postgresql_14
    ...
    error: derivation contains an illegal reference specifier 'man'

After the change:

    $ nix build --no-link -f. postgresql_14
    ...
    error: derivation '/nix/store/bxp6g57limvwiga61vdlyvhy7i8rp6wd-postgresql-14.15.drv' output check for 'lib' contains an illegal reference specifier 'man', expected store path or output name (one of [debug, dev, doc, lib, out])
2025-03-25 06:12:19 +00:00
mergify[bot] 5a4e306cc5 Merge pull request #12747 from obsidiansystems/more-goal-waitForAWhile
Use `Goal::waitForAWhile` in a few more places
2025-03-24 17:28:20 +00:00
mergify[bot] 07321575ca Merge pull request #12716 from Mic92/fix-script
tests/functional: use script flags compatible with macOS
2025-03-24 16:51:27 +00:00
mergify[bot] 7da2e5ba6a Merge pull request #12745 from obsidiansystems/append-log-tail-error-msg-type
`appendLogTailErrorMsg`: Take a "smaller" arugment
2025-03-24 16:05:32 +00:00
Las 3cb38e8ab9 Use Goal::waitForAWhile in a few more places 2025-03-24 11:46:55 -04:00
Jörg ThalheimandJohn Ericson f02c57dbfa tests/functional: use script flags compatible with macOS
Co-authored-by: John Ericson <git@JohnEricson.me>
2025-03-24 16:38:20 +01:00
John Ericson c121daf331 appendLogTailErrorMsg: Take a "smaller" arugment
We just need a `const Store &`, not a `Worker &`.
2025-03-24 11:24:16 -04:00
Robert Hensing 1988dc0c73 Merge pull request #12734 from Mic92/doc-fix
packaging/everything.nix: make sure doc and manpage outputs are symlinks
2025-03-24 13:08:46 +00:00
Jörg Thalheim ebcb13842b Merge pull request #12736 from Mic92/monitor-fd-hup
`MonitorFdHup::~MonitorFdHup`: use proper close method instead of lib…
2025-03-24 13:15:27 +01:00
Jörg Thalheim 87a34a45ff MonitorFdHup::~MonitorFdHup: use proper close method instead of libc close()
Otherwise closing it again will cause an EBADF in the AutoCloseFd class.
2025-03-24 12:09:54 +01:00
Kirens 15dfeb9182 make sure doc and manpage outputs are symlinks
Part of https://github.com/NixOS/nixpkgs/pull/392549
The doc and manpage fix already happend in 0ddfbc5939
2025-03-24 09:30:28 +01:00
John Ericson ff17dd2a9a Merge pull request #12730 from xokdvium/repl-shell-env
libcmd/repl: Fix missing runNix in repl
2025-03-23 23:53:26 -04:00
Jörg Thalheim 648c095da5 Merge pull request #12714 from picnoir/pic/monitorhup-fix-pthread-cancellation
MonitorFdHup: replace pthread_cancel trick with a notification pipe
2025-03-24 00:53:34 +01:00
John Ericson 49f486d8e0 MonitorFdHup: Don't sleep anymore
After the previous commit it should not be necessary. Furthermore, if we
*do* sleep, we'll exacerbate a race condition (in conjunction with
getting rid of the thread cancellation) that will cause test failures.
2025-03-23 19:15:27 -04:00
Jade Lovelace 9b3352c3c8 daemon: remove workaround for macOS kernel bug that seems fixed
This was filed as https://github.com/nixos/nix/issues/7584, but as far
as I can tell, the previous solution of POLLHUP works just fine on macOS
14. I've also tested on an ancient machine with macOS 10.15.7, which
also has POLLHUP work correctly.

It's possible this might regress some older versions of macOS that have
a kernel bug, but I went looking through the history on the sources and
didn't find anything that looked terribly convincingly like a bug fix
between 2020 and today. If such a broken version exists, it seems pretty
reasonable to suggest simply updating the OS.

Change-Id: I178a038baa000f927ea2cbc4587d69d8ab786843

Based off of commit 69e2ee5b25752ba5fd8644cef56fb9d627ca4a64. Ericson2314 added
additional other information.
2025-03-23 19:05:45 -04:00
Félix Baylac Jacqué 1c636284a3 MonitorFdHup: replace pthread_cancel trick with a notification pipe
On https://github.com/NixOS/nix/issues/8946, we faced a surprising
behaviour wrt. exception when using pthread_cancel. In a nutshell when
a thread is inside a catch block and it's getting pthread_cancel by
another one, then the original exception is bubbled up and crashes the
process.

We now poll on the notification pipe from the thread and exit when the
main thread closes its end. This solution does not exhibit surprising
behaviour wrt. exceptions.

Co-authored-by: Mic92 <joerg@thalheim.io>

Fixes https://github.com/NixOS/nix/issues/8946

See also Lix https://gerrit.lix.systems/c/lix/+/1605 which is very
similar by coincidence. Pulled a comment from that.
2025-03-23 18:34:52 -04:00
John Ericson cb95791198 MonitorFdHup: introduce a num_fds variable
Better than just putting `1` in multiple spots.
2025-03-23 18:23:07 -04:00
John Ericson d028bb4c4a MonitorFdHup: Cleanup a bit with designated initializers 2025-03-23 18:23:07 -04:00
Jörg Thalheim 8e0bc2c3a8 MonitorFdHup: raise explicit SysError rather unreachable
Syscalls can fail for many reasons and we don't want to loose the errno
and error context.
2025-03-23 18:22:48 -04:00
Sergei Zimmerman d371aadb2b tests/functional: Add regression test for broken :sh in repl
Can't really test `:u` because it needs <nixpkgs>.
2025-03-23 22:13:40 +00:00
Sergei Zimmerman 44055dc09d libcmd/repl: Fix missing runNix in repl
Without this :u, :sh and :i repl commands fail with:

> Cannot run 'nix-shell'/`nix-env` because no method of calling the Nix
> CLI was provided. This is a configuration problem pertaining to how
> this program was built.

Remove the default ctor argument as it evidently makes catching
refactoring bugs much harder. `NixRepl` implementation lives completely
in `repl.cc`, so we can be as explicit as necessary.
2025-03-23 22:13:14 +00:00
Sergei Zimmerman 8066e4b0c3 libcmd/repl: Make AbstractNixRepl::create respect its store argument
The only reference (according to clangd) to this function also uses `openStore`,
so this is a no-op.
2025-03-23 22:10:43 +00:00
John Ericson 041394b741 monitor-fd.hh: Format
It's a pretty small diff, so let's just start formatting before we make
other changes.
2025-03-23 18:00:36 -04:00
Robert Hensing bfc05d2e3b Merge pull request #12723 from xokdvium/fix-man-doc-outputs
packaging/everything.nix: Fix doc and man outputs
2025-03-23 11:53:13 +00:00
Sergei Zimmerman 0ddfbc5939 packaging/everything.nix: Fix doc and man outputs
We want the $doc, $man outputs to be symlinks pointing to nix-manual and
nix-manual.man. Creating the directories first makes the `ln` command
produce symlink $doc/${nix-manual} instead.

```
$file /nix/store/q4dwlnd36gpfajgfcp6hca2xwy068wjq-nix-2.27.1-man/rwh8ky3k040wyrywl8k2v5b3csdfbdg7-nix-manual-2.27.1-man

/nix/store/q4dwlnd36gpfajgfcp6hca2xwy068wjq-nix-2.27.1-man/rwh8ky3k040wyrywl8k2v5b3csdfbdg7-nix-manual-2.27.1-man:
  symbolic link to /nix/store/rwh8ky3k040wyrywl8k2v5b3csdfbdg7-nix-manual-2.27.1-man
```

This is the reason `nix-env --help` is once again broken on 2.26/2.27/master
after 4108529.
2025-03-22 12:16:47 +00:00
Eelco Dolstra d975d32d67 Merge pull request #12719 from NixOS/fix-links
rl-2.27.md: Fix GitHub links
2025-03-21 20:48:06 +01:00
Eelco Dolstra be5a455a1a rl-2.27.md: Fix GitHub links
https://discourse.nixos.org/t/nix-2-27-0-released/62003/2?u=edolstra
2025-03-21 20:23:46 +01:00
Eelco Dolstra cb66d3eea9 Merge pull request #12718 from NixOS/bump-2.28.0
Bump version
2025-03-21 20:10:24 +01:00
Eelco Dolstra a1f05dc669 Bump version 2025-03-21 19:32:21 +01:00
Robert Hensing 2172c17b5e Merge pull request #12701 from mightyiam/nix-infrec-base-error
stack overflow is EvalBaseError
2025-03-21 14:10:22 +00:00
mergify[bot] f0b7b37425 Merge pull request #12697 from NixOS/worker-abstraction
Make Goal code use abstractions over interations with Worker
2025-03-21 10:38:29 +00:00
Shahar "Dawn" Or 23c7a45a05 stack overflow is EvalBaseError 2025-03-20 17:43:20 +00:00
The Tumultuous Unicorn Of DarknessandJohn Ericson 83ec81789a Use feature_level field from libcpuid (#12674)
Close #11375

Co-authored-by: John Ericson <git@JohnEricson.me>
2025-03-20 13:11:55 -04:00
Rodney Lorrimar 1bc82d1c86 tests/functional/flakes: Add test case for subflake locking
This adds a test case where the lockfile of a relative path flake
dependency is updated.

It was reported by a user here: https://discourse.nixos.org/t/updating-local-subflakes-inputs-when-building-root-flake/61682

I think this test case relates to issue #7730.

Because the issue is not resolved, this test case would fail without
the `|| true` clause.
2025-03-20 13:28:05 +08:00
Las 36e5aa6c7d Make Goal code use abstractions over interations with Worker
Instead of calling `worker.waitForAWhile(shared_from_this())` etc.,
the subclasses of Goal instead call protected functions defined in Goal
that abstract over these.

The code for awaiting has also been heavily simplified.
Instead of calling `addWaitee`, then suspending,
`co_await await(waitees)` is called once, which also handles the suspend.

The end-goal is to remove all manual `co_await Suspend{}`s.
2025-03-19 18:36:43 -04:00
Jörg Thalheim c97f779dbb Merge pull request #12692 from obsidiansystems/small-derivation-goal-cleanups
Small derivation goal cleanups
2025-03-19 23:30:23 +01:00
John Ericson 3d333e0aff Merge pull request #12667 from NixOS/in-dir-cleanup
Cleanup `isInDir` and  `isDirOrInDir`
2025-03-19 18:01:59 -04:00
Jörg Thalheim 12ad06c104 Merge pull request #12687 from Mic92/handle-eagain
libutil/file-descriptor: handle EAGAIN in read/write operations
2025-03-19 22:46:05 +01:00
John Ericson 91e90aaee0 Clean up one path computation with / operator
Because of the previous commit, we need to use `std::filesystem::path`
anyways.
2025-03-19 17:14:55 -04:00
John Ericson d3de22b2be isInDir and isDirOrInDir: Clean up with std::filesystem
The behavior *does* change, per the tests, but I think the new behavior
is less buggy.
2025-03-19 17:13:21 -04:00
John Ericson 3286728e40 Simplify isInDir usage with isDirOrInDir 2025-03-19 17:12:49 -04:00
Jörg Thalheim 2790f5f9ae libutil/file-descriptor: handle EAGAIN in read/write operations
We now see exception beeing thrown when remote building in master
because of writing to a non-blocking file descriptor from our json logger.

> #0  0x00007f2ea97aea9c in __pthread_kill_implementation () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
> #1  0x00007f2ea975c576 in raise () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
> #2  0x00007f2ea9744935 in abort () from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
> #3  0x00007f2ea99e8c2b in __gnu_cxx::__verbose_terminate_handler() [clone .cold] () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
> #4  0x00007f2ea99f820a in __cxxabiv1::__terminate(void (*)()) () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
> #5  0x00007f2ea99f8275 in std::terminate() () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
> #6  0x00007f2ea99f84c7 in __cxa_throw () from /nix/store/ybjcla5bhj8g1y84998pn4a2drfxybkv-gcc-13.3.0-lib/lib/libstdc++.so.6
> #7  0x00007f2eaa5035c2 in nix::writeFull (fd=2, s=..., allowInterrupts=true) at ../unix/file-descriptor.cc:43
> #8  0x00007f2eaa5633c4 in nix::JSONLogger::write (this=this@entry=0x249a7d40, json=...) at /nix/store/4krab2h0hd4wvxxmscxrw21pl77j4i7j-gcc-13.3.0/include/c++/13.3.0/bits/char_traits.h:358
> #9  0x00007f2eaa5658d7 in nix::JSONLogger::logEI (this=<optimized out>, ei=...) at ../logging.cc:242
> #10 0x00007f2ea9c5d048 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x249a7d40) at /nix/store/a7cq5bqh0ryvnkv4m19ffchnvi8l9qx6-nix-util-2.27.0-dev/include/nix/logging.hh:108
> #11 nix::handleExceptions (programName="nix", fun=...) at ../shared.cc:343
> #12 0x0000000000465b1f in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/4krab2h0hd4wvxxmscxrw21pl77j4i7j-gcc-13.3.0/include/c++/13.3.0/bits/allocator.h:163
> (gdb) frame 10
> #10 0x00007f2ea9c5d048 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x249a7d40) at /nix/store/a7cq5bqh0ryvnkv4m19ffchnvi8l9qx6-nix-util-2.27.0-dev/include/nix/logging.hh:108
> 108             logEI(ei);

So far only drainFD sets the non-blocking flag on a "readable" file descriptor,
while this is a "writeable" file descriptor.
It's not clear to me yet, why we see logs after that point, but it's
also not that bad to handle EAGAIN in read/write functions after all.
2025-03-19 22:02:18 +01:00
tomberek 5df1975246 Merge pull request #12570 from P-E-Meunier/macos-fix
Fix macos sandbox issue
2025-03-19 16:19:59 -04:00
Eelco Dolstra 0fa3edc921 Merge pull request #12682 from Mic92/logging
libfetchers: fix double quote in path printed in logger
2025-03-19 20:37:11 +01:00
300465c7b8 Fix macos sandbox issue
Co-authored-by: John Ericson <git@JohnEricson.me>

Co-authored-by: Poliorcetics <poliorcetics@users.noreply.github.com>
2025-03-19 17:09:18 +00:00
Jörg Thalheim 314e9fbeda libfetchers: fix double quote in path printed in logger 2025-03-19 16:04:16 +01:00
mergify[bot] d10f9488fe Merge pull request #12658 from obsidiansystems/local-derivation-goal-hide-and-split
Move `RestrictedStore` into its own file+header
2025-03-19 10:00:34 +00:00
Jörg Thalheim 59ef89d886 Merge pull request #12677 from oldshensheep/master
Improve the documentation of store path
2025-03-19 10:24:38 +01:00
Jörg Thalheim 5ba9949b82 Merge pull request #12578 from roberth/complete-hydra-build-jobs
packaging: Make hydraJobs.build.* complete
2025-03-19 10:11:19 +01:00
oldshensheep 355a923e81 Improve the documentation of store path
# Conflicts:
#	doc/manual/source/protocols/store-path.md
2025-03-19 13:45:10 +08:00
John Ericson f497711aa8 Merge pull request #12624 from KAction/store-path-doc
Improve the documentation of the store path protocol
2025-03-18 12:36:01 -04:00
Robert Hensing 95b0971031 Merge pull request #12652 from roberth/cli-json-pretty
nix-cli: Add --json --pretty / --no-pretty
2025-03-18 12:29:43 +00:00
Robert Hensing 9ff8309f47 Remove commented code
Some four years old; time to go
2025-03-18 08:14:07 +00:00
John Ericson 1c022077ea Get rid of on usage pair of actLock
Now that we have coroutines, we can go back to loops and regular RAII,
which is must less error-proone!

I look forward to removing the other instances!
2025-03-17 11:07:25 -04:00
John Ericson 7f8d348f3d Move derivationType from DerivationGoal to LocalDerivationGoal
The super class doesn't actually care.
2025-03-17 11:07:25 -04:00
John Ericson 5283589542 RestrictedStore: Move some definitions outside of the type declaration
Even when the type is not currently declared in a header, I still consider this a
more future-proof style.
2025-03-17 11:01:18 -04:00
John Ericson 5026d5af95 Move RestrictedStore into its own file+header
Perhaps more significantly, it no longer knows about
`LocalDerivationGoal`, and without any effort it also compiles on
Windows just fine. (`local-derivation-goal.{cc,hh}` is currently skipped
on Windows.)
2025-03-17 10:54:48 -04:00
John Ericson 2cfd031511 Merge pull request #12662 from obsidiansystems/local-derivation-goal-hide
Local derivation goal hide
2025-03-16 17:13:22 -04:00
Robert Hensing e4bda20918 Merge pull request #12586 from xokdvium/refactor/chmod-if-needed
{libutil,libstore}: Factor out chmodIfNeeded
2025-03-15 09:06:28 +00:00
John Ericson 7f2b7b8bd1 Do not expose LocalDerivationGoal implementation
We just need to expose construction functions.
2025-03-14 15:57:24 -04:00
John Ericson d572533774 Move signPathInfo to Store
Motivation is the same for moving `signRealisation` in
db8439c328.
2025-03-14 15:57:24 -04:00
Robert Hensing 1bff2aeec0 Merge pull request #12645 from xokdvium/debugger-perf
Make debugger significantly faster
2025-03-14 13:21:12 +00:00
Robert Hensing fe00dfbd56 nix-cli: Add --json --pretty / --no-pretty
Default: istty(stdout)

This refactors `nix develop` internals a bit to use the `json` type
more. The assertion now operates in the in-memory json instead of
re-parsing it. While this is technically a weaker guarantee, we
should be able to rely on the library to get this right. It's its
most essential purpose.
2025-03-14 12:39:13 +00:00
Sergei Zimmerman adbd08399c {libexpr,libcmd}: Make debugger significantly faster
The underlying issue is that debugger code path was
calling PosTable::operator[] in each eval method.
This has become incredibly expensive since 5d9fdab3de.

While we are it it, I've reworked the code to
not use std::shared_ptr where it really isn't necessary.

As I've documented in previous commits, this is actually
more a workaround for recursive header dependencies now
and is only necessary in `error.hh` code.

Some ad-hoc benchmarking:

After this commit:

```
Benchmark 1: nix eval nixpkgs#hello --impure --ignore-try --no-eval-cache --debugger
  Time (mean ± σ):     784.2 ms ±   7.1 ms    [User: 561.4 ms, System: 147.7 ms]
  Range (min … max):   773.5 ms … 792.6 ms    10 runs
```

On master 3604c7c51:

```
Benchmark 1: nix eval nixpkgs#hello --impure --ignore-try --no-eval-cache --debugger
  Time (mean ± σ):     22.914 s ±  0.178 s    [User: 18.524 s, System: 4.151 s]
  Range (min … max):   22.738 s … 23.290 s    10 runs
```
2025-03-13 16:24:30 +00:00
Jörg Thalheim e9af7a0749 Merge pull request #12647 from DeterminateSystems/lock-json-logger
JSONLogger: Acquire a lock to prevent log messages from clobbering each other
2025-03-13 16:43:09 +01:00
Eelco Dolstra d0227f8d02 JSONLogger: Acquire a lock to prevent log messages from clobbering each other 2025-03-13 16:02:10 +01:00
Sergei Zimmerman 50123f2a56 libutil: Fix Pos::getSourcePath
Previous implementation didn't actually check if
std::get_if returned a nullptr:

std::optional<SourcePath> getSourcePath() const {
    return *std::get_if<SourcePath>(&origin);
}
2025-03-13 13:29:08 +00:00
Sergei Zimmerman bf12aedf2e libutil: Document hacks and problems around Pos class
This should provide context for follow-up commits in
the patch series.
2025-03-13 13:29:08 +00:00
Sergei Zimmerman a53b184e63 {libutil,libexpr}: Move pos-idx,pos-table code to libutil
All of this code doesn't actually depend on anything from
libexpr. Because Pos is so tigtly coupled with Error, it
makes sense to have in the same library.
2025-03-13 13:29:08 +00:00
John Ericson b8eaf1b322 Merge pull request #12643 from obsidiansystems/delete-dead-goal-var
Remove unused parameter to the goal constructor
2025-03-12 16:45:26 -07:00
John Ericson ecdcba27c5 Remove unused parameter to the goal constructor
It has been unused since 37fca662b0.
2025-03-12 19:09:54 -04:00
John Ericson 1055c9fd14 Merge pull request #12630 from L-as/me/clean-up-drv-goal
Clean up derivation goals a bit
2025-03-12 15:52:05 -07:00
Jörg Thalheim af4c587ae3 Merge pull request #12596 from obsidiansystems/adv-attrs-organize
Advanced attributes organize
2025-03-12 23:21:48 +01:00
Jörg Thalheim cba1a2155a Merge pull request #12567 from obsidiansystems/slightly-rework-drv-resolution
Rework derivation input resolution
2025-03-12 23:15:45 +01:00
Jörg Thalheim f4fd570ae3 Merge pull request #12642 from Mic92/shallow-clone
libfetchers/git: fix caching head when using shallow clones
2025-03-12 23:10:31 +01:00
John Ericson dc0bc7f0a3 Make debug message more precise 2025-03-12 18:09:38 -04:00
John Ericson 99d0dd3a43 Simplify hook error status logic
The simplification here is due to a long-standing bug, but it is not
worth fixing the bug at this time. Instead we've finally written up an
issue for the bug, and referenced the issue number in the code.
2025-03-12 18:09:38 -04:00
John Ericson 06af9cb532 Inline the try-catch BuildError in the hook case
In the local building case, there is many things which can through
`BuildError`, but in the hook case there is just this one. We can
therefore simplify the code by "cinching" down the logic just to the
spot the error is thrown.

There is other code outside `libstore/build` which also uses
`BuildError`, but I believe those cases are mistakes. The point of
`BuildError` is the narrow technical use-cases of "errors which should
not be fatal with `--keep-going`". Using it outside the
building/scheduling code doesn't really make sense in that regard. It
seems likely that those usages were instead merely because "oh, this
error has something to do with building, so I guess `BuildError` is
better than `Error`".

It is quite likely that I myself used `BuildError` incorrectly as
described above :).
2025-03-12 18:09:38 -04:00
John Ericson a39ed67180 Do no store timestamps in the build result in the build hook case
The variables are only set by CGroup mechanisms in `killSandbox` in the
local build. In the build hook case, these variables will not be set, so
there is nothing to do.
2025-03-12 18:09:38 -04:00
Las db8439c328 Remove signRealisation from drv goal
We can move this method from `LocalStore` to `Store` --- even if we only
want the actual builder to sign things in many cases, there is no reason
to try to enforce this policy by spurious moving the method to a
subclass.

Now, we might technically sign class, but CA derivations is
experimental, and @Ericson2314 is going to revisit all this stuff with
issue #11896 anyways.
2025-03-12 18:09:38 -04:00
Las 0e7e1f5b57 Remove registerOutputs from drv goal
Easy to inline in one spot, and assert in the other.
2025-03-12 18:09:38 -04:00
Las a87589a035 Simplify local drv goal a bit more
- `chrootParentDir` can be a local variable instead of a class variable.

- `getChildStatus` can be inlined. Again, we have the `assert(!hook);`
  in the local building case, which makes for a simpler thing inlined.
2025-03-12 18:09:38 -04:00
Las 4b521f14ac Remove privateNetwork variable from local drv goal
Can just inline its definition, it was immutable.
2025-03-12 18:09:38 -04:00
John Ericson 87824bca6b Avoid pointless mutation
The code that was in between is now gone. We can just set `st` correctly
the first time.
2025-03-12 18:08:58 -04:00
John Ericson 145aa2f118 Remove dead hook code in LocalDerivationGoal::tryLocalBuild
The `assert` above proves that `hook` is not set.
2025-03-12 18:08:10 -04:00
Las 75feeecd5d Start simplifying {Local,}DerivationGoal cleanup code
Thanks to the previous commit, we can inline all these small callbacks.
In the build-hook case, they were empty, and now they disappear
entirely.

While `LocalDerivationGoal` can be used in the hook case (we use it
based on whether we have a local store, not based on whether we are
using the build hook, a decision which comes later), the previous
commit's inline moved the code into a spot where we know we are cleaning
up after local building, *not* after running the build hook. This allows
for much more simplification.
2025-03-12 18:05:08 -04:00
Las e87ba85705 Inline buildDone from DerivationGoal into use sites
The basic idea is that while we have duplicated this function, we now
have one call-site in the local build case, and one call site in the
build hook case. This unlocks big opportunities to specialize each copy,
since they really shouldn't be doing the same things. By the time we are
are done, there should not be much duplication left.

See #12628 for further info.
2025-03-12 18:00:07 -04:00
John Ericson 1de97bbe2e Factor out "last 10 log lines" error message code
This will help avoid duplication later. In particular, the next commit
will not need to duplicate as much.
2025-03-12 18:00:07 -04:00
John EricsonandJörg Thalheim 637aa0944d Advanced attributes organize
This is supposed to firstly improve the docs as they are, and secondly
hint at how the core conceptual information ought to be moved to the
store derivation section of the manual.

Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
2025-03-12 17:45:51 -04:00
Jörg Thalheim e9f1761597 libfetchers/git: fix caching head when using shallow clones
the old code was using the wrong cache directory, which lead to a
warning on every nix flake update

Update src/libfetchers/git.cc
2025-03-12 22:32:05 +01:00
Robert Hensing 0068e225e0 Merge pull request #12639 from DeterminateSystems/remove-NIX_HELD_LOCKS
Remove use of $NIX_HELD_LOCKS
2025-03-12 21:10:08 +00:00
Eelco Dolstra 78eed85ba3 Remove use of $NIX_HELD_LOCKS
This variable was once used to communicate already acquired store path
locks between Nix and the build hook, but this hasn't been the case
since 9bcb4d2dd9. So let's get rid of
it.
2025-03-12 21:28:56 +01:00
Eelco Dolstra 9c5a8e9352 Merge pull request #12636 from Mic92/crash-handler
port crash-handler from lix to nix
2025-03-12 20:59:20 +01:00
Jörg Thalheim cac1168afd Merge pull request #12424 from ilya-bobyr/fish-profile-unify-ca-bundle.crt-search
nix-profile.fish: Look for ca-bundle.crt in $NIX_PROFILES
2025-03-12 17:07:57 +01:00
Jörg Thalheim c9a3101bff Merge pull request #12603 from NaN-git/fix-curl-retry
libstore: curl retry: reset content-encoding and don't use string after move
2025-03-12 13:47:51 +01:00
Jade Lovelaceandeldritch horrors 163f94412a port crash-handler from lix to nix
It was first introduced in https://git.lix.systems/lix-project/lix/commit/19e0ce2c03d8e0baa16998b086665664c420c1df

In Nix we only register the crash handler in main instead of initNix,
because library user may want to use their own crash handler.

Sample output:

Mar 12 08:38:06 eve nix[2303762]: Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
Mar 12 08:38:06 eve nix[2303762]: Exception: nix::SysError: error: writing to file: Resource temporarily unavailable
Mar 12 08:38:06 eve nix[2303762]: Stack trace:
Mar 12 08:38:06 eve nix[2303762]:  0# 0x000000000076876A in nix
                                   1# 0x00007FDA40E9F20A in /nix/store/2lhklm5aizx30qbw49acnrrzkj9lbmij-gcc-14-20241116-lib/lib/libstdc++.so.6
                                   2# std::unexpected() in /nix/store/2lhklm5aizx30qbw49acnrrzkj9lbmij-gcc-14-20241116-lib/lib/libstdc++.so.6
                                   3# 0x00007FDA40E9F487 in /nix/store/2lhklm5aizx30qbw49acnrrzkj9lbmij-gcc-14-20241116-lib/lib/libstdc++.so.6
                                   4# nix::writeFull(int, std::basic_string_view<char, std::char_traits<char> >, bool) in /home/joerg/git/nix/inst/lib/libnixutil.so
                                   5# nix::writeLine(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) in /home/joerg/git/nix/inst/lib/libnixutil.so
                                   6# nix::JSONLogger::write(nlohmann::json_abi_v3_11_3::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_11_3::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void> const&) in /home/joerg/git/nix/inst/lib/libnixutil.so
                                   7# nix::JSONLogger::logEI(nix::ErrorInfo const&) in /home/joerg/git/nix/inst/lib/libnixutil.so
                                   8# nix::Logger::logEI(nix::Verbosity, nix::ErrorInfo) in nix
                                   9# nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) in /home/joerg/git/nix/inst/lib/libnixmain.so
                                  10# 0x000000000087A563 in nix
                                  11# 0x00007FDA40BD41FE in /nix/store/6q2mknq81cyscjmkv72fpcsvan56qhmg-glibc-2.40-66/lib/libc.so.6
                                  12# __libc_start_main in /nix/store/6q2mknq81cyscjmkv72fpcsvan56qhmg-glibc-2.40-66/lib/libc.so.6
                                  13# 0x00000000006F4DF5 in nix

Co-authored-by: eldritch horrors <pennae@lix.systems>
2025-03-12 10:23:02 +01:00
Dmitry BogatovandJohn Ericson affd9bbab7 Update doc/manual/source/protocols/store-path.md
Co-authored-by: John Ericson <git@JohnEricson.me>
2025-03-11 12:30:21 -04:00
Jörg Thalheim 8e8edb5bf8 Merge pull request #12615 from xokdvium/ubsan-checks
flake: Enable UBSAN for checks
2025-03-11 14:22:20 +01:00
Jörg Thalheim 341628a037 Merge pull request #12618 from fzakaria/minor-text-fixes
Fix minor documentation typos
2025-03-11 14:13:14 +01:00
Jörg Thalheim d1b21e812d Merge pull request #12633 from xokdvium/wstring-ctor-fiasco
libutil/windows: Finally use the correct constructor for std::wstring
2025-03-11 13:52:29 +01:00
Sergei Zimmerman 24fbb456ba libutil/windows: Finally use the correct constructor for std::wstring
C++ is very intuitive /s [1]. Fixes #12631.

[1]: https://godbolt.org/z/jMa9GP5sq
2025-03-11 10:57:38 +00:00
mergify[bot] cacab33f0d Merge pull request #12629 from NixOS/dependabot/github_actions/cachix/install-nix-action-31
build(deps): bump cachix/install-nix-action from 30 to 31
2025-03-10 23:12:44 +00:00
dependabot[bot] 3387d5a3c4 build(deps): bump cachix/install-nix-action from 30 to 31
Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from 30 to 31.
- [Release notes](https://github.com/cachix/install-nix-action/releases)
- [Commits](https://github.com/cachix/install-nix-action/compare/v30...v31)

---
updated-dependencies:
- dependency-name: cachix/install-nix-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-10 22:58:40 +00:00
Dmitry Bogatov a0facb2aba Improve the documentation of the store path protocol
1. Fix confusing wording that might imply unnecessary double-hashing.
2. Add references to specifics of base-32 encoding.
3. Fix incorrect description that sha256 hash of `fingerprint` is
   truncated. "Truncated" is actual wording used in Nix theses, but it has
   unusual meaning, that is better conveyed by word "compressed", which is
   used by the reference C++ implementation.
4. Clarify details of base16 encoding.
2025-03-08 19:00:00 -05:00
Sergei Zimmerman 874587516c flake: Enable UBSAN for checks
Doing this makes catching non-obvious bugs easier. GHA CI workload is
already a concern and there isn't much benefit in running the tests with
and without sanitizers at the same time, so UBSAN is enabled for default
checks.

This change doesn't affect production builds in any way, but is rather a
step in the direction of improving automated testing during development.

Relates to #10969.
2025-03-09 00:24:47 +03:00
Robert Hensing a047dec120 Merge pull request #12620 from nix-windows/c-api/leaks
c-api: fix a few memory leaks
2025-03-08 22:06:59 +01:00
John Ericson fb4d55c227 Merge pull request #12619 from nix-windows/tests/fix-rapidcheck-arbitraries
rapidcheck: change to working arbitrary instances
2025-03-08 08:42:12 -08:00
Brian McKenna 79b019ec4f c-api: fix a few memory leaks 2025-03-08 20:23:07 +11:00
Brian McKenna c58202c6f9 DerivedPathTest: disable prop_legacy_round_rip until fixed 2025-03-08 19:51:25 +11:00
Brian McKenna c82ef825d4 coerceToSingleDerivedPathUnchecked: pass through experimental features
This fixes a few of the property tests, now that the property tests
are actually generating arbitrary data - some of that data now
requiring experimental features to function properly.
2025-03-08 19:14:58 +11:00
Brian McKenna 9a04f1e732 rapidcheck: change to working arbitrary instances
Here we're switching to combinators instead of dereference operator.
It turns out the dereference operator was being executed upon test
setup, meaning that we were only using a only single value for each of
the executions of the property tests! Really not good.

And on Windows, we instead get:

    operator* is not allowed in this context

https://github.com/emil-e/rapidcheck/blob/ff6af6fc683159deb51c543b065eba14dfcf329b/src/gen/detail/GenerationHandler.cpp#L16C31-L16C71

Now a few of the property tests fail, because we're generating cases
which haven't been exercised before.
2025-03-08 19:08:29 +11:00
Farid Zakaria 33493b9ead Fix minor documentation typos
Was reading the store chapter and came across a few small typos
and edits.
2025-03-07 23:07:03 -08:00
Eelco Dolstra 47e23811ff Merge pull request #12613 from Mic92/minor-cleanups
Fix double quoting in some error messages
2025-03-07 18:38:48 +01:00
Jörg Thalheim 3b1e23af09 libstore/remote-store: avoid old-style casting for maxConnections
Type-checking works better this way as (type) style casting is too
permissive.
2025-03-07 04:00:26 +01:00
Jörg Thalheim 3b0b2fd8d6 libflake: fix double quoting when updating flakes 2025-03-07 04:00:24 +01:00
Jörg Thalheim 559a2d1bc7 libfetchers/git: fix double quoting in error message 2025-03-07 04:00:22 +01:00
Sergei Zimmerman 82f337de10 {libutil,libstore}: Factor out chmodIfNeeded
Using std::filesystem::path directly because we need .c_str()
anyway to interact with chmod. Path/string views don't have to be
null-terminated.
2025-03-05 02:43:11 +03:00
Robert Hensing 2c1447b7f2 Merge pull request #12604 from NixOS/issue-12599
Fix chopped up repl output
2025-03-04 20:09:16 +01:00
Robert Hensing 1e1c587073 repl: Fix :print corruption
See preceding commits.
2025-03-04 19:48:22 +01:00
Robert Hensing 30694b5d8a refactor: RAII logger suspension 2025-03-04 19:48:22 +01:00
Robert Hensing 880489051a repl: Fix value printing corruption
The resume call would get some non-flushed(?) data.
Extending the pause to include the newline makes the complete flush
part of the pause.
2025-03-04 19:48:22 +01:00
Robert Hensing 49d8ee5359 progress-bar: Make pause/resume nestable 2025-03-04 19:48:22 +01:00
Philipp Otterbein b129fc8237 libstore: curl retry: reset content-encoding and don't use string after move 2025-03-04 18:05:33 +01:00
Robert Hensing a5cf291dde Merge pull request #12600 from roberth/packaging-typo-preConfigure
packaging: Typo in setVersionLayer / preConfigure
2025-03-04 17:44:47 +01:00
Robert Hensing dcaea8cb1c packaging: Typo in setVersionLayer / preConfigure
Apparently dead code in our use case, but good to keep nonetheless.
Credit: ztzg in https://github.com/NixOS/nix/pull/12498#pullrequestreview-2658031853
2025-03-04 17:22:24 +01:00
John Ericson 24463dd025 Merge pull request #11672 from fricklerhandwerk/at-pattern-default
doc: note that @-pattern is accessible in default values
2025-03-04 00:57:17 -05:00
Valentin GagarinandRobert Hensing 9c3dd34cfe doc: note that function bindings are accessible in default values
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2025-03-04 00:44:46 -05:00
Robert Hensing efbd4c1ebb Merge pull request #12442 from NixOS/store-derivation-options
Expand manual on derivation outputs
2025-03-03 23:11:51 +01:00
Robert Hensing 1e00d14c29 manual: Edit 2025-03-03 19:09:24 +01:00
John Ericson a58e0584f5 Rework derivation input resolution
I refactored the way that input resolution works in `DerivationGoal`. To
be honest, it is probably unclear to the reader whether this new way is
better or worse. I suppose *intrinsic* motivation, I can say that

- the more structured use of `inputGoal` (a local variable) is better
  than the shotgrun approach with `inputDrvOutputs`

- A virtual `waiteeDone` was a hack, and now it's gone.

However, the *real* motivation of this is not the above things, but that
it is needed for my mammoth refactor fixing #11897 and #11928.

It is nice that this step could come first, rather than making that
refactor even bigger.
2025-03-03 10:31:56 -05:00
John Ericson 8fdb50761d SingleDerivedPath should be const in recursive data structures 2025-03-03 10:31:23 -05:00
Robert Hensing 89b495520d Merge pull request #12591 from obsidiansystems/temp-revert-no-dyn-drv
Once again temporarily revert "Adapt scheduler to work with dynamic derivations"
2025-03-02 21:57:56 +01:00
John Ericson f4f28cdd0e Revert "Revert "Revert "Adapt scheduler to work with dynamic derivations"""
The bug reappeared after all, and the fix introduced a different bug. I
just reverted on 2.27 first, in #12576, but upon further introspection
and discussion with @roberth, with preparing for and travelling to
Planet Nix I will not be able to fix it on `master` soon enough for a
revert to not be warranted here in the meantime also.

This reverts commit c98525235f.
2025-03-02 15:34:21 -05:00
Robert Hensing 7cfc52fd42 Merge pull request #12587 from NixOS/mergiy-2-27-backports
.mergify.yml: Add backport 2.27-maintenance entry
2025-03-02 14:08:49 +01:00
Eelco Dolstra 67aaed7f99 Merge pull request #12588 from fricklerhandwerk/exit-fricklerhandwerk
remove fricklerhandwerk from CODEOWNERS
2025-03-02 12:28:41 +01:00
Valentin Gagarin 0358007da3 remove fricklerhandwerk from CODEOWNERS
stepping aside as a Nix maintainer:
https://discourse.nixos.org/t/time-to-step-aside/61050
2025-03-02 00:06:15 +01:00
Robert Hensing 529cbea343 .mergify.yml: Add backport 2.27-maintenance entry 2025-03-01 22:54:57 +01:00
Sergei Zimmerman c99edc840c libutil/file-system.hh: Fix typos 2025-03-01 18:44:48 +00:00
John Ericson 77f22db567 Merge pull request #12582 from roberth/nix-everything-symlink-megafarm
packaging/everything.nix: Use a multi-output derivation
2025-02-28 12:47:11 -05:00
Robert Hensing 41085295ab packaging/everything.nix: Use a multi-output derivation
This should fix a few packaging regressions.

`dev` also includes a merged `includes/`, which may be helpful until
inter-component includes are fixed properly.
2025-02-28 17:40:32 +01:00
Eelco Dolstra d8a7c50495 Merge pull request #12580 from ysndr/fix/fetchers/host-in-locked-url
Add host attribute of github/gitlab flakerefs to URL serialization
2025-02-28 15:41:02 +01:00
Yannik Sander 954e9101ba Add host attribute of github/gitlab flakerefs to URL serialization
`GitArchiveInputScheme::toUrl` currently drops the `host` attribute,
creating invalid urls when locking `github:` or `gitlab:` urls pointing to alterative instances and serializing the input back to a url.

```
❯ cat flake.nix
{
  inputs.gnome-2048 = {
    url = "gitlab:GNOME/gnome-2048?host=gitlab.gnome.org";
    flake = false;
  };

  outputs = inputs: {};
}
f1xb57354q79t_jpw5_h79cw0000gq/T/tmp.MOBbzbpT35
❯ nix flake metadata
warning: creating lock file '/private/var/folders/fb/f1xb57354q79t_jpw5_h79cw0000gq/T/tmp.MOBbzbpT35/flake.lock':
• Added input 'gnome-2048':
    'gitlab:GNOME/gnome-2048/70e0e430ca4bf590990433a3abdce6b631d50e6e?narHash=sha256-bya45ug2mDSU4SMn0fSBlZCuPl9y15B12ubKeb2A58s%3D' (2025-02-21)
Resolved URL:  path:/private/var/folders/fb/f1xb57354q79t_jpw5_h79cw0000gq/T/tmp.MOBbzbpT35
Locked URL:    path:/private/var/folders/fb/f1xb57354q79t_jpw5_h79cw0000gq/T/tmp.MOBbzbpT35?lastModified=1740744684&narHash=sha256-nxUL/JiTYbZX2c1XiN/TC6aA1hf%2B1YXsUvhL7ASY2uE%3D
Path:          /nix/store/f4xczpwhdxs8gal1rika1c5bvhyd472l-source
Last modified: 2025-02-28 13:11:24
Inputs:
└───gnome-2048: gitlab:GNOME/gnome-2048/70e0e430ca4bf590990433a3abdce6b631d50e6e?narHash=sha256-bya45ug2mDSU4SMn0fSBlZCuPl9y15B12ubKeb2A58s%3D (2025-02-21 23:18:46)
```

Note the gnome-2048 input url missing the original host query.

The Url after this commit:

```
[...]
Inputs:
└───gnome-2048: gitlab:GNOME/gnome-2048/70e0e430ca4bf590990433a3abdce6b631d50e6e?host=gitlab.gnome.org&narHash=sha256-bya45ug2mDSU4SMn0fSBlZCuPl9y15B12ubKeb2A58s%3D (2025-02-21 23:18:46)
```
2025-02-28 13:14:16 +01:00
John Ericson 92c4789ec7 Merge pull request #12573 from tomberek/tomberek.update_meeting
fix: update work meeting calendar link
2025-02-27 16:37:45 -05:00
Robert Hensing d6139a339b packaging: Make hydraJobs.build.* complete 2025-02-27 20:45:09 +01:00
John Ericson 2aa6e0f084 Expand manual on derivation outputs
Note, this includes some text adapted from from Eelco's dissertation
2025-02-27 02:13:36 -05:00
Thomas Bereknyei 068cdfafb8 fix: update work meeting calendar link 2025-02-26 15:45:57 -05:00
Illia Bobyr 23f07fa974 nix-profile.fish: Look for ca-bundle.crt in $NIX_PROFILES
There seems to be no good reason for `nix-profile.fish` and
`nix-profile-daemon.fish` to differ in how they look for the location of
the `ca-bundle.crt` that might be installed by one of the packages.

As `$NIX_PROFILES` points to user local paths, not checking there is
strictly less useful, it seems?
2025-02-05 01:42:21 -08:00
389 changed files with 7794 additions and 4320 deletions

No files matched your search

+1 -10
View File
@@ -11,16 +11,7 @@
.github/CODEOWNERS @edolstra
# Documentation of built-in functions
src/libexpr/primops.cc @roberth @fricklerhandwerk
# Documentation of settings
src/libexpr/eval-settings.hh @fricklerhandwerk
src/libstore/globals.hh @fricklerhandwerk
# Documentation
doc/manual @fricklerhandwerk
maintainers/*.md @fricklerhandwerk
src/**/*.md @fricklerhandwerk
src/libexpr/primops.cc @roberth
# Libstore layer
/src/libstore @ericson2314
+1 -1
View File
@@ -3,7 +3,7 @@
- Thanks for your contribution!
- To remove the stale label, just leave a new comment.
- _How to find the right people to ping?_ &rarr; [`git blame`](https://git-scm.com/docs/git-blame) to the rescue! (or GitHub's history and blame buttons.)
- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on [Matrix - #nix:nixos.org](https://matrix.to/#/#nix:nixos.org).
- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on [Matrix - #users:nixos.org](https://matrix.to/#/#users:nixos.org).
## Suggestions for PRs
+4 -4
View File
@@ -13,7 +13,7 @@ jobs:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v30
- uses: cachix/install-nix-action@v31
- run: nix --experimental-features 'nix-command flakes' flake show --all-systems --json
tests:
@@ -34,7 +34,7 @@ jobs:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v30
- uses: cachix/install-nix-action@v31
with:
# The sandbox would otherwise be disabled by default on Darwin
extra_nix_config: |
@@ -77,7 +77,7 @@ jobs:
- name: Serving installer
id: serving_installer
run: ./scripts/serve-installer-for-github-actions
- uses: cachix/install-nix-action@v30
- uses: cachix/install-nix-action@v31
with:
install_url: 'http://localhost:8126/install'
install_options: "--tarball-url-prefix http://localhost:8126/"
@@ -131,7 +131,7 @@ jobs:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v30
- uses: cachix/install-nix-action@v31
with:
install_url: https://releases.nixos.org/nix/nix-2.20.3/install
- uses: DeterminateSystems/magic-nix-cache-action@main
+22
View File
@@ -128,3 +128,25 @@ pull_request_rules:
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.28
conditions:
- label=backport 2.28-maintenance
actions:
backport:
branches:
- "2.28-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.29
conditions:
- label=backport 2.29-maintenance
actions:
backport:
branches:
- "2.29-maintenance"
labels:
- automatic backport
- merge-queue
+1 -1
View File
@@ -1 +1 @@
2.28.4
2.30.0
+1 -1
View File
@@ -31,7 +31,7 @@ Today, a world-wide developer community contributes to Nix and the ecosystem tha
- [Nixpkgs](https://github.com/NixOS/nixpkgs) is [the largest, most up-to-date free software repository in the world](https://repology.org/repositories/graphs)
- [NixOS](https://github.com/NixOS/nixpkgs/tree/master/nixos) is a Linux distribution that can be configured fully declaratively
- [Discourse](https://discourse.nixos.org/)
- [Matrix](https://matrix.to/#/#nix:nixos.org)
- Matrix: [#users:nixos.org](https://matrix.to/#/#users:nixos.org) for user support and [#nix-dev:nixos.org](https://matrix.to/#/#nix-dev:nixos.org) for development
## License
+1
View File
@@ -33,6 +33,7 @@ let
{
settings,
doc,
uri-schemes,
experimentalFeature,
}:
let
-1
View File
@@ -281,7 +281,6 @@ nix3_manpages = [
'nix3-store',
'nix3-store-optimise',
'nix3-store-path-from-hash-part',
'nix3-store-ping',
'nix3-store-prefetch-file',
'nix3-store-repair',
'nix3-store-sign',
+20 -10
View File
@@ -11,6 +11,8 @@
python3,
rsync,
nix-cli,
changelog-d,
officialRelease,
# Configuration Options
@@ -44,16 +46,24 @@ mkMesonDerivation (finalAttrs: {
];
# Hack for sake of the dev shell
passthru.externalNativeBuildInputs = [
meson
ninja
(lib.getBin lowdown-unsandboxed)
mdbook
mdbook-linkcheck
jq
python3
rsync
];
passthru.externalNativeBuildInputs =
[
meson
ninja
(lib.getBin lowdown-unsandboxed)
mdbook
mdbook-linkcheck
jq
python3
rsync
changelog-d
]
++ lib.optionals (!officialRelease) [
# When not an official release, we likely have changelog entries that have
# yet to be rendered.
# When released, these are rendered into a committed file to save a dependency.
changelog-d
];
nativeBuildInputs = finalAttrs.passthru.externalNativeBuildInputs ++ [
nix-cli
+1
View File
@@ -374,6 +374,7 @@ const redirects = {
},
"glossary.html": {
"gloss-local-store": "store/types/local-store.html",
"package-attribute-set": "#package",
"gloss-chroot-store": "store/types/local-store.html",
"gloss-content-addressed-derivation": "#gloss-content-addressing-derivation",
},
@@ -0,0 +1,17 @@
---
synopsis: Non-flake inputs now contain a `sourceInfo` attribute
issues: 13164
prs: 13170
---
Flakes have always a `sourceInfo` attribute which describes the source of the flake.
The `sourceInfo.outPath` is often identical to the flake's `outPath`, however it can differ when the flake is located in a subdirectory of its source.
Non-flake inputs (i.e. inputs with `flake = false`) can also be located at some path _within_ a wider source.
This usually happens when defining a relative path input within the same source as the parent flake, e.g. `inputs.foo.url = ./some-file.nix`.
Such relative inputs will now inherit their parent's `sourceInfo`.
This also means it is now possible to use `?dir=subdir` on non-flake inputs.
This iterates on the work done in 2.26 to improve relative path support ([#10089](https://github.com/NixOS/nix/pull/10089)),
and resolves a regression introduced in 2.28 relating to nested relative path inputs ([#13164](https://github.com/NixOS/nix/issues/13164)).
+17
View File
@@ -0,0 +1,17 @@
---
synopsis: Revert incomplete closure mixed download and build feature
issues: [77, 12628]
prs: [13176]
---
Since Nix 1.3 (299141ecbd08bae17013226dbeae71e842b4fdd7 in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference).
This feature is now removed.
Worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute.
In practice, however, we doubt even the more building is very likely to happen.
Remote stores that are missing dependencies in arbitrary ways (e.g. corruption) don't seem to be very common.
On the contrary, when remote stores fail to implement the [closure property](@docroot@/store/store-object.md#closure-property), it is usually an *intentional* choice on the part of the remote store, because it wishes to serve as an "overlay" store over another store, such as `https://cache.nixos.org`.
If an "incomplete closure" is encountered in that situation, the right fix is not to do some sort of "franken-building" as this feature implemented, but instead to make sure both substituters are enabled in the settings.
(In the future, we should make it easier for remote stores to indicate this to clients, to catch settings that won't work in general before a missing dependency is actually encountered.)
+2
View File
@@ -33,6 +33,7 @@
- [Data Types](language/types.md)
- [String context](language/string-context.md)
- [Syntax and semantics](language/syntax.md)
- [Evaluation](language/evaluation.md)
- [Variables](language/variables.md)
- [String literals](language/string-literals.md)
- [Identifiers](language/identifiers.md)
@@ -135,6 +136,7 @@
- [Contributing](development/contributing.md)
- [Releases](release-notes/index.md)
{{#include ./SUMMARY-rl-next.md}}
- [Release 2.29 (2025-05-14)](release-notes/rl-2.29.md)
- [Release 2.28 (2025-04-02)](release-notes/rl-2.28.md)
- [Release 2.27 (2025-03-03)](release-notes/rl-2.27.md)
- [Release 2.26 (2025-01-22)](release-notes/rl-2.26.md)
+9 -4
View File
@@ -242,16 +242,21 @@ print(t)
```
Similarly, the following is a Perl script that specifies that it
requires Perl and the `HTML::TokeParser::Simple` and `LWP` packages:
requires Perl and the `HTML::TokeParser::Simple`, `LWP` and
`LWP::Protocol::Https` packages:
```perl
#! /usr/bin/env nix-shell
#! nix-shell -i perl --packages perl perlPackages.HTMLTokeParserSimple perlPackages.LWP
#! nix-shell -i perl
#! nix-shell --packages perl
#! nix-shell --packages perlPackages.HTMLTokeParserSimple
#! nix-shell --packages perlPackages.LWP
#! nix-shell --packages perlPackages.LWPProtocolHttps
use HTML::TokeParser::Simple;
# Fetch nixos.org and print all hrefs.
my $p = HTML::TokeParser::Simple->new(url => 'http://nixos.org/');
my $p = HTML::TokeParser::Simple->new(url => 'https://nixos.org/');
while (my $token = $p->get_tag("a")) {
my $href = $token->get_attr("href");
@@ -316,7 +321,7 @@ contains:
```nix
with import <nixpkgs> {};
runCommand "dummy" { buildInputs = [ python pythonPackages.prettytable ]; } ""
runCommand "dummy" { buildInputs = [ python3 python3Packages.prettytable ]; } ""
```
The script's file name is passed as the first argument to the interpreter specified by the `-i` flag.
@@ -45,10 +45,19 @@ symlink.
[output paths]: @docroot@/glossary.md#gloss-output-path
- `--references`
Prints the set of [references] of the store paths
*paths*, that is, their immediate dependencies. (For *all*
dependencies, use `--requisites`.)
[references]: @docroot@/glossary.md#gloss-reference
- `--requisites` / `-R`
Prints out the [closure] of the store path *paths*.
Prints out the set of [*requisites*][requisite] (better known as the [closure]) of the store path *paths*.
[requisite]: @docroot@/glossary.md#gloss-requisite
[closure]: @docroot@/glossary.md#gloss-closure
This query has one option:
@@ -65,29 +74,25 @@ symlink.
dependencies) is obtained by distributing the closure of a store
derivation and specifying the option `--include-outputs`.
- `--references`
Prints the set of [references] of the store paths
*paths*, that is, their immediate dependencies. (For *all*
dependencies, use `--requisites`.)
[references]: @docroot@/glossary.md#gloss-reference
- `--referrers`
Prints the set of *referrers* of the store paths *paths*, that is,
Prints the set of [*referrers*][referrer] of the store paths *paths*, that is,
the store paths currently existing in the Nix store that refer to
one of *paths*. Note that contrary to the references, the set of
referrers is not constant; it can change as store paths are added or
removed.
[referrer]: @docroot@/glossary.md#gloss-referrer
- `--referrers-closure`
Prints the closure of the set of store paths *paths* under the
referrers relation; that is, all store paths that directly or
[referrers relation][referrer]; that is, all store paths that directly or
indirectly refer to one of *paths*. These are all the path currently
in the Nix store that are dependent on *paths*.
[referrer]: @docroot@/glossary.md#gloss-referrer
- `--deriver` / `-d`
Prints the [deriver] that was used to build the store paths *paths*. If
@@ -20,8 +20,9 @@ prs: 1238
Here's one or more paragraphs that describe the change.
- It's markdown
- Add references to the manual using @docroot@
- Add references to the manual using [links like this](@_at_docroot@/example.md)
```
<!-- for the raw markdown readers: that means using @docroot@ -->
Significant changes should add the following header, which moves them to the top.
+37 -15
View File
@@ -38,6 +38,13 @@
[store derivation]: #gloss-store-derivation
- [directed acyclic graph]{#gloss-directed-acyclic-graph}
A [directed acyclic graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) is graph whose edges are given a direction ("a to b" is not the same edge as "b to a"), and for which no possible path (created by joining together edges) forms a cycle.
DAGs are very important to Nix.
In particular, the non-self-[references][reference] of [store object][store object] form a cycle.
- [derivation path]{#gloss-derivation-path}
A [store path] which uniquely identifies a [store derivation].
@@ -71,9 +78,8 @@
This can be achieved by:
- Fetching a pre-built [store object] from a [substituter]
- Running the [`builder`](@docroot@/language/derivations.md#attr-builder) executable as specified in the corresponding [store derivation]
- [Building](@docroot@/store/building.md) the corresponding [store derivation]
- Delegating to a [remote machine](@docroot@/command-ref/conf-file.md#conf-builders) and retrieving the outputs
<!-- TODO: link [running] to build process page, #8888 -->
See [`nix-store --realise`](@docroot@/command-ref/nix-store/realise.md) for a detailed description of the algorithm.
@@ -156,6 +162,8 @@
non-[fixed-output](#gloss-fixed-output-derivation)
derivation.
See [input-addressing derivation outputs](store/derivation/outputs/input-address.md) for details.
- [content-addressed store object]{#gloss-content-addressed-store-object}
A [store object] which is [content-addressed](#gloss-content-address),
@@ -215,23 +223,25 @@
> **Example**
>
> Building and deploying software using Nix entails writing Nix expressions as a high-level description of packages and compositions thereof.
> Building and deploying software using Nix entails writing Nix expressions to describe [packages][package] and compositions thereof.
- [reference]{#gloss-reference}
A [store object] `O` is said to have a *reference* to a store object `P` if a [store path] to `P` appears in the contents of `O`.
An edge from one [store object] to another.
Store objects can refer to both other store objects and themselves.
References from a store object to itself are called *self-references*.
References other than a self-reference must not form a cycle.
See [References](@docroot@/store/store-object.md#references) for details.
[reference]: #gloss-reference
See [References](@docroot@/store/store-object.md#references) for details.
- [reachable]{#gloss-reachable}
A store path `Q` is reachable from another store path `P` if `Q`
is in the *closure* of the *references* relation.
See [References](@docroot@/store/store-object.md#references) for details.
- [closure]{#gloss-closure}
The closure of a store path is the set of store paths that are
@@ -248,8 +258,21 @@
to a store object at path `Q`, then `Q` is in the closure of `P`. Further, if `Q`
references `R` then `R` is also in the closure of `P`.
See [References](@docroot@/store/store-object.md#references) for details.
[closure]: #gloss-closure
- [requisite]{#gloss-requisite}
A store object [reachable] by a path (chain of references) from a given [store object].
The [closure] is the set of requisites.
See [References](@docroot@/store/store-object.md#references) for details.
- [referrer]{#gloss-reference}
A reversed edge from one [store object] to another.
- [output]{#gloss-output}
A [store object] produced by a [store derivation].
@@ -330,18 +353,17 @@
- [package]{#package}
1. A software package; a collection of files and other data.
A software package; files that belong together for a particular purpose, and metadata.
2. A [package attribute set].
Nix represents files as [file system objects][file system object], and how they belong together is encoded as [references][reference] between [store objects][store object] that contain these file system objects.
- [package attribute set]{#package-attribute-set}
The [Nix language] allows denoting packages in terms of [attribute sets](@docroot@/language/types.md#attribute-set) containing:
- attributes that refer to the files of a package, typically in the form of [derivation outputs](#output),
- attributes with metadata, such as information about how the package is supposed to be used.
An [attribute set](@docroot@/language/types.md#attribute-set) containing the attribute `type = "derivation";` (derivation for historical reasons), as well as other attributes, such as
- attributes that refer to the files of a [package], typically in the form of [derivation outputs](#output),
- attributes that declare something about how the package is supposed to be installed or used,
- other metadata or arbitrary attributes.
The exact shape of these attribute sets is up to convention.
[package attribute set]: #package-attribute-set
[package]: #package
- [string interpolation]{#gloss-string-interpolation}
+2
View File
@@ -30,6 +30,8 @@ $ curl -L https://nixos.org/nix/install | sh -s -- --daemon
> Single-user is not supported on Mac.
> `warning: installing Nix as root is not supported by this script!`
This installation has less requirements than the multi-user install, however it
cannot offer equivalent sharing, isolation, or security.
+2 -2
View File
@@ -1,8 +1,8 @@
# Introduction
Nix is a _purely functional package manager_. This means that it
treats packages like values in purely functional programming languages
such as Haskell — they are built by functions that don’t have
treats packages like values in a purely functional programming language
— packages are built by functions that don’t have
side-effects, and they never change after they have been built. Nix
stores packages in the _Nix store_, usually the directory
`/nix/store`, where each package has its own unique subdirectory such
@@ -79,6 +79,8 @@ Derivations can declare some infrequently used optional attributes.
## Output checks
See the [corresponding section in the derivation output page](@docroot@/store/derivation/outputs/index.md).
- [`allowedReferences`]{#adv-attr-allowedReferences}\
The optional attribute `allowedReferences` specifies a list of legal
references (dependencies) of the output of the builder. For example,
+77
View File
@@ -0,0 +1,77 @@
# Evaluation
Evaluation is the process of turning a Nix expression into a [Nix value](types.md).
This happens by a number of rules, such as:
- Constructing values from literals.
For example the number literal `1` is turned into the number value `1`.
- Applying operators
For example the addition operator `+` is applied to two number values to produce a new number value.
- Applying built-in functions
For example the expression `builtins.isInt 1` is evaluated to `true`.
- Applying user-defined functions
For example the expression `(x: x + 1) 10` can[*](#laziness) be thought of rewriting `x` in the function body to the argument, `10 + 1`, which is then evaluated to `11`.
These rules are applied as needed, driven by the specific use of the expression. For example, this can occur in the Nix command line interface or interactively with the [repl (read-eval-print loop)](@docroot@/command-ref/new-cli/nix3-repl.md), which is a useful tool when learning about evaluation.
# Details
## Values {#values}
Nix values can be thought of as a subset of Nix expressions.
For example, the expression `1 + 2` is not a value, because it can be reduced to `3`. The expression `3` is a value, because it cannot be reduced any further.
Evaluation normally happens by applying rules to the "head" of the expression, which is the outermost part of the expression. The head of an expression like `[ 1 2 ]` is the list literal (`[ a1 a2 ]`), for `1 + 2` it is the addition operator (`+`), and for `f 1` it is the function application "operator" (` `).
After applying all possible rules to the head until no rules can be applied, the expression is in "weak head normal form" (WHNF). This means that the outermost constructor of the expression is evaluated, but the inner values may or may not be. "Weak" only signifies that the expression may be a function. This is an historical or academic artifact, and Nix has no use for the non-weak "head normal form".
## Laziness and thunks {#laziness}
The Nix language implements _call by need_ (as opposed to _call by value_ or _call by reference_). <!-- No wikipedia link, which would be a huge distraction. --> Call by need is commonly known as laziness in functional programming, as it is a specific implementation of the concept where evaluation is deferred until the result is required, aiming to only evaluate the parts of an expression that are needed to produce the final result.
Furthermore, the result of evaluation is preserved, in values, in `let` bindings, in function _parameters_, which behave a lot like `let` bindings, but with the notable exception of function _calls_. Results of function calls rely on being put into `let` bindings, etc to be reused. <!-- which would be prohibitively expensive and too strict, or we wouldn't have a cache key for the argument -->
When discussing the process of evaluation in lower level terms, we may define values not as a subset of expressions, but separately, where each "value" is either a data constructor, a function or a _thunk_. A thunk is a delayed computation, represented by an expression reference and a "closure" &ndash; the values for the lexical scope around the delayed expression.
As a user of the language, you generally don't have to think about thunks, as they are not part of the language semantics, but you may encounter them in the repl, in the [C API] or in discussions.
## Strictness
Instead of thinking about thunks, it is often more productive to think in terms of _strictness_.
This term is used in functional programming to refer to the opposite of laziness, i.e. not just for something like error propagation. It refers to the need to evaluate certain expressions before evaluation can produce any result.
Statements about strictness usually implicitly refer to weak head normal form.
For example, we can say that the following function is strict in its argument:
```nix
x: isAttrs x || isFunction x
```
The above function must be strict in its argument `x` because determining its type requires evaluating `x` to at least some degree.
The following function is not strict in its argument:
```nix
x: { isOk = isAttrs x || isFunction x; }
```
It is not strict, because it can return the attribute set before evaluating `x`.
The attribute value for `isOk` _is_ strict in `x`.
A function with a _set pattern_ is always strict in its argument, as a consequence of checking the argument's type and/or attribute names:
```nix
let f = { ... }: "ok";
in f (throw "kablam")
=> error: kablam
```
However, a set pattern does not add any strictness beyond WHNF of the attribute set argument.
```nix
let f = orig@{ x, ... }: "ok";
in f { x = throw "error"; y = throw "error"; }
=> "ok"
```
[C API]: @docroot@/c-api.md
+8 -1
View File
@@ -11,7 +11,14 @@ The language is:
- *domain-specific*
It comes with [built-in functions](@docroot@/language/builtins.md) to integrate with the Nix store, which manages files and performs the derivations declared in the Nix language.
The Nix language is purpose-built for working with text files.
Its most characteristic features are:
- [File system path primitives](@docroot@/language/types.md#type-path), for accessing source files
- [Indented strings](@docroot@/language/string-literals.md) and [string interpolation](@docroot@/language/string-interpolation.md), for creating file contents
- [Strings with contexts](@docroot@/language/string-context.md), for transparently linking files
It comes with [built-in functions](@docroot@/language/builtins.md) to integrate with the [Nix store](@docroot@/store/index.md), which manages files and enables [realising](@docroot@/glossary.md#gloss-realise) derivations declared in the Nix language.
- *declarative*
+2 -2
View File
@@ -13,8 +13,8 @@ The purpose of string contexts is to collect non-string values attached to strin
[string concatenation](./operators.md#string-concatenation),
[string interpolation](./string-interpolation.md),
and similar operations.
The idea is that a user can combine together values to create a build instructions for derivations without manually keeping track of where they come from.
Then the Nix language implicitly does that bookkeeping to efficiently obtain the closure of derivation inputs.
The idea is that a user can reference other files when creating text files through Nix expressions, without manually keeping track of the exact paths.
Nix will ensure that the all referenced files are accessible – that all [store paths](@docroot@/glossary.md#gloss-store-path) are [valid](@docroot@/glossary.md#gloss-validity).
> **Note**
>
+160
View File
@@ -0,0 +1,160 @@
# Release 2.29.0 (2025-05-14)
After the special backport-based release of Nix 2.28 (timed to coincide with Nixpkgs 25.05), the release process is back to normal with 2.29.
As such, we have slightly more weeks of work from `master` (since 2.28 was branched from 2.27) than usual.
This fact is counterbalanced by the fact that most of those changes are bug fixes rather than larger new features.
- Prettified JSON output on the terminal [#12555](https://github.com/NixOS/nix/issues/12555) [#12652](https://github.com/NixOS/nix/pull/12652)
This makes the output easier to read.
Scripts are mostly unaffected because for those, stdout will be a file or a pipe, not a terminal, and for those, the old single-line behavior applies.
`--json --pretty` can be passed to enable it even if the output is not a terminal.
If your script creates a pseudoterminal for Nix's stdout, you can pass `--no-pretty` to disable the new behavior.
- Repl: improve continuation prompt for incomplete expressions [#12846](https://github.com/NixOS/nix/pull/12846)
Improved REPL user experience by updating the continuation prompt from invisible blank spaces to a visible `" > "`, enhancing clarity when entering multi-line expressions.
- REPL `:load-flake` and `:reload` now work together [#8753](https://github.com/NixOS/nix/issues/8753) [#13180](https://github.com/NixOS/nix/pull/13180)
Previously, `:reload` only reloaded the files specified with `:load` (or on the command line).
Now, it also works with the flakes specified with `:load-flake` (or on the command line).
This makes it correctly reload everything that was previously loaded, regardless of what sort of thing (plain file or flake) each item is.
- Increase retry delays on HTTP 429 Too Many Requests [#13052](https://github.com/NixOS/nix/pull/13052)
When downloading Nix, the retry delay was previously set to 0.25 seconds. It has now been increased to 1 minute to better handle transient CI errors, particularly on GitHub.
- S3: opt-in the STSProfileCredentialsProvider [#12646](https://github.com/NixOS/nix/pull/12646)
Added support for STS-based authentication for S3-based binary caches, i.e. enabling seamless integration with `aws sso login`.
- Reduce connect timeout for http substituter [#12876](https://github.com/NixOS/nix/pull/12876)
Previously, the Nix setting `connect-timeout` had no limit. It is now set to `5s`, offering a more practical default for users self-hosting binary caches, which may occasionally become unavailable, such as during updates.
- C API: functions for locking and loading a flake [#10435](https://github.com/NixOS/nix/issues/10435) [#12877](https://github.com/NixOS/nix/pull/12877) [#13098](https://github.com/NixOS/nix/pull/13098)
This release adds functions to the C API for handling the loading of flakes. Previously, this had to be worked around by using `builtins.getFlake`.
C API consumers and language bindings now have access to basic locking functionality.
It does not expose the full locking API, so that the implementation can evolve more freely.
Locking is controlled with the functions, which cover the common use cases for consuming a flake:
- `nix_flake_lock_flags_set_mode_check`
- `nix_flake_lock_flags_set_mode_virtual`
- `nix_flake_lock_flags_set_mode_write_as_needed`
- `nix_flake_lock_flags_add_input_override`, which also enables `virtual`
This change also introduces the new `nix-fetchers-c` library, whose single purpose for now is to manage the (`nix.conf`) settings for the built-in fetchers.
More details can be found in the [C API documentation](@docroot@/c-api.md).
- No longer copy flakes that are in the nix store [#10435](https://github.com/NixOS/nix/issues/10435) [#12877](https://github.com/NixOS/nix/pull/12877) [#13098](https://github.com/NixOS/nix/pull/13098)
Previously, we would duplicate entries like `path:/nix/store/*` back into the Nix store.
This was prominently visible for pinned system flake registry entries in NixOS, e.g., when running `nix run nixpkgs#hello`.
- Consistently preserve error messages from cached evaluation [#12762](https://github.com/NixOS/nix/issues/12762) [#12809](https://github.com/NixOS/nix/pull/12809)
In one code path, we are not returning the errors cached from prior evaluation, but instead throwing generic errors stemming from the lack of value (due to the error).
These generic error messages were far less informative.
Now we consistently return the original error message.
- Faster blake3 hashing [#12676](https://github.com/NixOS/nix/pull/12676)
The implementation for blake3 hashing is now multi-threaded and used memory-mapped IO.
Benchmark results can be found the [pull request](https://github.com/NixOS/nix/pull/12676).
- Fix progress bar for S3 binary caches and make file transfers interruptible [#12877](https://github.com/NixOS/nix/issues/12877) [#13098](https://github.com/NixOS/nix/issues/13098) [#12538](https://github.com/NixOS/nix/pull/12538)
The progress bar now correctly display upload/download progress for S3 up/downloads. S3 uploads are now interruptible.
- Add host attribute of github/gitlab flakerefs to URL serialization [#12580](https://github.com/NixOS/nix/pull/12580)
Resolved an issue where `github:` or `gitlab:` URLs lost their `host` attribute when written to a lockfile, resulting in invalid URLs.
- Multiple signatures support in store urls [#12976](https://github.com/NixOS/nix/pull/12976)
Added support for a `secretKeyFiles` URI parameter in Nix store URIs, allowing multiple signing key files to be specified as a comma-separated list.
This enables signing paths with multiple keys. This helps with [RFC #149](https://github.com/NixOS/rfcs/pull/149) to enable binary cache key rotation in the NixOS infra.
Example usage:
```bash
nix copy --to "file:///tmp/store?secret-keys=/tmp/key1,/tmp/key2" \
"$(nix build --print-out-paths nixpkgs#hello)"
```
- nix flake show now skips over import-from-derivation [#4265](https://github.com/NixOS/nix/issues/4265) [#12583](https://github.com/NixOS/nix/pull/12583)
Previously, if a flake contained outputs relying on [import from derivation](@docroot@/language/import-from-derivation.md) during evaluation, `nix flake show` would fail to display the rest of the flake. The updated behavior skips such outputs, allowing the rest of the flake to be shown.
- Add `nix formatter build` and `nix formatter run` commands [#13063](https://github.com/NixOS/nix/pull/13063)
`nix formatter run` is an alias for `nix fmt`. Nothing new there.
`nix formatter build` is sort of like `nix build`: it builds, links, and prints a path to the formatter program:
```
$ nix formatter build
/nix/store/cb9w44vkhk2x4adfxwgdkkf5gjmm856j-treefmt/bin/treefmt
```
Note that unlike `nix build`, this prints the full path to the program, not just the store path (in the example above that would be `/nix/store/cb9w44vkhk2x4adfxwgdkkf5gjmm856j-treefmt`).
- Amend OSC 8 escape stripping for xterm-style separator [#13109](https://github.com/NixOS/nix/pull/13109)
Improve terminal escape code filtering to understand a second type of hyperlink escape codes.
This in particular prevents parts of GCC 14's diagnostics from being improperly filtered away.
# Contributors
This release was made possible by the following 40 contributors:
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
- The Tumultuous Unicorn Of Darkness [**(@TheTumultuousUnicornOfDarkness)**](https://github.com/TheTumultuousUnicornOfDarkness)
- Robert Hensing [**(@roberth)**](https://github.com/roberth)
- Félix [**(@picnoir)**](https://github.com/picnoir)
- Valentin Gagarin [**(@fricklerhandwerk)**](https://github.com/fricklerhandwerk)
- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
- Vincent Breitmoser [**(@Valodim)**](https://github.com/Valodim)
- Brian McKenna [**(@puffnfresh)**](https://github.com/puffnfresh)
- ulucs [**(@ulucs)**](https://github.com/ulucs)
- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
- Andrey Butirsky [**(@bam80)**](https://github.com/bam80)
- Dean De Leo [**(@whatsthecraic)**](https://github.com/whatsthecraic)
- Las Safin [**(@L-as)**](https://github.com/L-as)
- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
- Shahar "Dawn" Or [**(@mightyiam)**](https://github.com/mightyiam)
- Ryan Hendrickson [**(@rhendric)**](https://github.com/rhendric)
- Rodney Lorrimar [**(@rvl)**](https://github.com/rvl)
- Erik Nygren [**(@Kirens)**](https://github.com/Kirens)
- Cole Helbling [**(@cole-h)**](https://github.com/cole-h)
- Martin Fischer [**(@not-my-profile)**](https://github.com/not-my-profile)
- Graham Christensen [**(@grahamc)**](https://github.com/grahamc)
- Vit Gottwald [**(@VitGottwald)**](https://github.com/VitGottwald)
- silvanshade [**(@silvanshade)**](https://github.com/silvanshade)
- Illia Bobyr [**(@ilya-bobyr)**](https://github.com/ilya-bobyr)
- Jeremy Fleischman [**(@jfly)**](https://github.com/jfly)
- Ruby Rose [**(@oldshensheep)**](https://github.com/oldshensheep)
- Sergei Trofimovich [**(@trofi)**](https://github.com/trofi)
- Tim [**(@Jaculabilis)**](https://github.com/Jaculabilis)
- Anthony Wang [**(@anthowan)**](https://github.com/anthowan)
- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
- Sandro [**(@SuperSandro2000)**](https://github.com/SuperSandro2000)
- tomberek [**(@tomberek)**](https://github.com/tomberek)
- Dmitry Bogatov [**(@KAction)**](https://github.com/KAction)
- Sizhe Zhao [**(@Prince213)**](https://github.com/Prince213)
- jade [**(@lf-)**](https://github.com/lf-)
- Pierre-Etienne Meunier [**(@P-E-Meunier)**](https://github.com/P-E-Meunier)
- Alexander Romanov [**(@ajlekcahdp4)**](https://github.com/ajlekcahdp4)
- Domagoj Mišković [**(@allrealmsoflife)**](https://github.com/allrealmsoflife)
- Thomas Miedema [**(@thomie)**](https://github.com/thomie)
- Yannik Sander [**(@ysndr)**](https://github.com/ysndr)
- Philipp Otterbein
- Dmitry Bogatov
+1 -1
View File
@@ -282,7 +282,7 @@ type DerivingPath = ConstantPath | OutputPath;
Under this extended model, `DerivingPath`s are thus inductively built up from a root `ConstantPath`, wrapped with zero or more outer `OutputPath`s.
### Encoding {#deriving-path-encoding}
### Encoding {#deriving-path-encoding-higher-order}
The encoding is adjusted in the natural way, encoding the `drv` field recursively using the same deriving path encoding.
The result of this is that it is possible to have a chain of `^<output-name>` at the end of the final string, as opposed to just a single one.
+59 -2
View File
@@ -4,7 +4,64 @@ A Nix store is a collection of *store objects* with *references* between them.
A store object consists of
- A [file system object](./file-system-object.md) as data
- A set of [store paths](./store-path.md) as references to other store objects
- A set of [store paths](./store-path.md) as references to store objects
### References
Store objects can refer to both other store objects and themselves.
References from a store object to itself are called *self-references*.
Store objects and their references form a directed graph, where the store objects are the vertices, and the references are the edges.
In particular, the edge corresponding to a reference is from the store object that contains the reference, and to the store object that the store path (which is the reference) refers to.
References other than a self-reference must not form a cycle.
The graph of references excluding self-references thus forms a [directed acyclic graph].
[directed acyclic graph]: @docroot@/glossary.md#gloss-directed acyclic graph
We can take the [transitive closure] of the references graph, which any pair of store objects have an edge not if there is a single reference from the first to the second, but a path of one or more references from the first to the second.
The *requisites* of a store object are all store objects reachable by paths of references which start with given store object's references.
[transitive closure]: https://en.wikipedia.org/wiki/Transitive_closure
We can also take the [transpose graph] ofthe references graph, where we reverse the orientation of all edges.
The *referrers* of a store object are the store objects that reference it.
[transpose graph]: https://en.wikipedia.org/wiki/Transpose_graph
One can also combine both concepts: taking the transitive closure of the transposed references graph.
The *referrers closure* of a store object are the store objects that can reach the given store object via paths of references.
> **Note**
>
> Care must be taken to distinguish between the intrinsic and extrinsic properties of store objects.
> We can create graphs from the store objects in a store, but the contents of the store is not, in general fixed, and may instead change over time.
>
> - The references of a store object --- the set of store paths called the references --- is a field of a store object, and thus intrinsic by definition.
Regardless of what store contains the store object in question, and what else that store may or may not contain, the references are the same.
>
> - The requisites of a store object are almost intrinsic --- some store paths due not precisely refer to a unique single store object.
> Exactly what store object is being referenced, and what in turn *its* references are, depends on the store in question.
> Different stores that disagree.
>
> - The referrers of a store object are completely extrinsic, and depends solely on the store which contains that store object, not the store object itself.
> Other store objects which refer to the store object in question may be added or removed from the store.
### Immutability
Store objects are [immutable](https://en.wikipedia.org/wiki/Immutable_object):
Once created, they do not change until they are deleted.
Once created, they do not change nor can any store object they reference be changed.
> **Note**
>
> Stores which support atomically deleting multiple store objects allow more flexibility while still upholding this property.
### Closure property
A store can only contain a store object if it also contains all the store objects it refers to.
> **Note**
>
> The "closure property" isn't meant to prohibit, for example, [lazy loading](https://en.wikipedia.org/wiki/Lazy_loading) of store objects.
> However, the "closure property" and immutability in conjunction imply that any such lazy loading ought to be deterministic.
+3
View File
@@ -57,6 +57,9 @@ def recursive_replace(data: dict[str, t.Any], book_root: Path, search_path: Path
).replace(
'@docroot@',
("../" * len(path_to_chapter.parent.parts) or "./")[:-1]
).replace(
'@_at_',
'@'
),
sub_items = [
recursive_replace(sub_item, book_root, search_path)
+1 -1
View File
@@ -7,7 +7,7 @@
channelName ? "nixpkgs",
channelURL ? "https://nixos.org/channels/nixpkgs-unstable",
extraPkgs ? [ ],
maxLayers ? 100,
maxLayers ? 70,
nixConf ? { },
flake-registry ? null,
uid ? 0,
Generated
+4 -4
View File
@@ -63,16 +63,16 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1734359947,
"narHash": "sha256-1Noao/H+N8nFB4Beoy8fgwrcOQLVm9o4zKW1ODaqK9E=",
"lastModified": 1747179050,
"narHash": "sha256-qhFMmDkeJX9KJwr5H32f1r7Prs7XbQWtO0h3V0a0rFY=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "48d12d5e70ee91fe8481378e540433a7303dbf6a",
"rev": "adaa24fbf46737f3f1b5497bf64bae750f82942e",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "release-24.11",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
+4 -2
View File
@@ -1,7 +1,7 @@
{
description = "The purely functional package manager";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/release-24.11";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.nixpkgs-23-11.url = "github:NixOS/nixpkgs/a62e6edd6d5e1fa0329b8653c801147986f8d446";
@@ -32,7 +32,7 @@
let
inherit (nixpkgs) lib;
officialRelease = true;
officialRelease = false;
linux32BitSystems = [ "i686-linux" ];
linux64BitSystems = [
@@ -357,6 +357,7 @@
"nix-store-tests" = { };
"nix-fetchers" = { };
"nix-fetchers-c" = { };
"nix-fetchers-tests" = { };
"nix-expr" = { };
@@ -365,6 +366,7 @@
"nix-expr-tests" = { };
"nix-flake" = { };
"nix-flake-c" = { };
"nix-flake-tests" = { };
"nix-main" = { };
+2 -2
View File
@@ -37,7 +37,7 @@ The team is on Github as [@NixOS/nix-team](https://github.com/orgs/NixOS/teams/n
The team meets twice a week (times are denoted in the [Europe/Amsterdam](https://en.m.wikipedia.org/wiki/Time_in_the_Netherlands) time zone):
- Discussion meeting: [Wednesday 21:00-22:00 Europe/Amsterdam](https://www.google.com/calendar/event?eid=ZG5rZzNyajRjajducGV2NGY5aGkzYWIwdnJfMjAyNDA1MDhUMTkwMDAwWiBiOW81MmZvYnFqYWs4b3E4bGZraGczdDBxZ0Bn)
- Discussion meeting: Wednesday 21:00-22:00 Europe/Amsterdam see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
1. Triage issues and pull requests from the [No Status](#no-status) column (30 min)
2. Discuss issues and pull requests from the [To discuss](#to-discuss) column (30 min).
@@ -46,7 +46,7 @@ The team meets twice a week (times are denoted in the [Europe/Amsterdam](https:/
- mark it as draft if it is blocked on the contributor
- escalate it back to the team by moving it to To discuss, and leaving a comment as to why the issue needs to be discussed again.
- Work meeting: [Mondays 14:00-16:00 Europe/Amsterdam](https://www.google.com/calendar/event?eid=Ym52NDdzYnRic2NzcDcybjZiNDhpNzhpa3NfMjAyNDA1MTNUMTIwMDAwWiBiOW81MmZvYnFqYWs4b3E4bGZraGczdDBxZ0Bn)
- Work meeting: Mondays 14:00-16:00 Europe/Amsterdam see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
1. Code review on pull requests from [In review](#in-review).
2. Other chores and tasks.
@@ -152,5 +152,19 @@
"kaction@disroot.org": "KAction",
"serenity@kaction.cc": null,
"dev@erik.work": "Kirens",
"felix@alternativebit.fr": "picnoir"
"felix@alternativebit.fr": "picnoir",
"butirsky@gmail.com": "bam80",
"look@my.amazin.horse": "Valodim",
"jeremyfleischman@gmail.com": "jfly",
"vit.gottwald@gmail.com": "VitGottwald",
"a@unnamed.website": "anthowan",
"hello@whatsthecraic.net": "whatsthecraic",
"alex.rom23@mail.ru": "ajlekcahdp4",
"domagoj@tuta.com": "allrealmsoflife",
"uluc.sengil@gmail.com": "ulucs",
"prc.zhao@outlook.com": "Prince213",
"the-tumultuous-unicorn-of-darkness@gmx.com": "TheTumultuousUnicornOfDarkness",
"dev@rodney.id.au": "rvl",
"pe@pijul.org": "P-E-Meunier",
"yannik@floxdev.com": "ysndr"
}
@@ -133,5 +133,18 @@
"oldshensheep": "Ruby Rose",
"KAction": "Dmitry Bogatov",
"thomie": "Thomas Miedema",
"Kirens": "Erik Nygren"
"Kirens": "Erik Nygren",
"Prince213": "Sizhe Zhao",
"anthowan": "Anthony Wang",
"jfly": "Jeremy Fleischman",
"VitGottwald": "Vit Gottwald",
"bam80": "Andrey Butirsky",
"ulucs": null,
"P-E-Meunier": "Pierre-Etienne Meunier",
"ysndr": "Yannik Sander",
"TheTumultuousUnicornOfDarkness": "The Tumultuous Unicorn Of Darkness",
"ajlekcahdp4": "Alexander Romanov",
"Valodim": "Vincent Breitmoser",
"rvl": "Rodney Lorrimar",
"whatsthecraic": "Dean De Leo"
}
+4 -5
View File
@@ -284,8 +284,8 @@
''^src/libstore/build/goal\.cc$''
''^src/libstore/include/nix/store/build/goal\.hh$''
''^src/libstore/unix/build/hook-instance\.cc$''
''^src/libstore/unix/build/local-derivation-goal\.cc$''
''^src/libstore/unix/include/nix/store/build/local-derivation-goal\.hh$''
''^src/libstore/unix/build/derivation-builder\.cc$''
''^src/libstore/unix/include/nix/store/build/derivation-builder\.hh$''
''^src/libstore/build/substitution-goal\.cc$''
''^src/libstore/include/nix/store/build/substitution-goal\.hh$''
''^src/libstore/build/worker\.cc$''
@@ -360,7 +360,6 @@
''^src/libutil/linux/namespaces\.cc$''
''^src/libutil/logging\.cc$''
''^src/libutil/include/nix/util/logging\.hh$''
''^src/libutil/include/nix/util/lru-cache\.hh$''
''^src/libutil/memory-source-accessor\.cc$''
''^src/libutil/include/nix/util/memory-source-accessor\.hh$''
''^src/libutil/include/nix/util/pool\.hh$''
@@ -605,8 +604,8 @@
''^tests/functional/flakes/prefetch\.sh$''
''^tests/functional/flakes/run\.sh$''
''^tests/functional/flakes/show\.sh$''
''^tests/functional/fmt\.sh$''
''^tests/functional/fmt\.simple\.sh$''
''^tests/functional/formatter\.sh$''
''^tests/functional/formatter\.simple\.sh$''
''^tests/functional/gc-auto\.sh$''
''^tests/functional/gc-concurrent\.builder\.sh$''
''^tests/functional/gc-concurrent\.sh$''
-83
View File
@@ -1,83 +0,0 @@
#!/usr/bin/env python3
# This script must be run from the root of the Nix repository.
#
# For include path hygiene, we need to put headers in a separate
# directory than sources. But during development, it is nice to paths
# that are similar for headers and source files, e.g.
# `foo/bar/baz.{cc,hh}`, e.g. for less typing when opening one file, and
# then opening the other file.
#
# This script symlinks the headers next to the source files to
# facilitate such a development workflows. It also updates
# `.git/info/exclude` so that the symlinks are not accidentally committed
# by mistake.
from pathlib import Path
import subprocess
import os
def main() -> None:
# Path to the source directory
GIT_TOPLEVEL = Path(
subprocess.run(
["git", "rev-parse", "--show-toplevel"],
text=True,
stdout=subprocess.PIPE,
check=True,
).stdout.strip()
)
# Get header files from git
result = subprocess.run(
["git", "-C", str(GIT_TOPLEVEL), "ls-files", "*/include/nix/**.hh"],
text=True,
stdout=subprocess.PIPE,
check=True,
)
header_files = result.stdout.strip().split("\n")
header_files.sort()
links = []
for file_str in header_files:
project_str, header_str = file_str.split("/include/nix/", 1)
project = Path(project_str)
header = Path(header_str)
# Reconstruct the full path (relative to SRC_DIR) to the header file.
file = project / "include" / "nix" / header
# The symlink should be created at "project/header", i.e. next to the project's sources.
link = project / header
# Compute a relative path from the symlink's parent directory to the actual header file.
relative_source = os.path.relpath(
GIT_TOPLEVEL / file, GIT_TOPLEVEL / link.parent
)
# Create the symbolic link.
full_link_path = GIT_TOPLEVEL / link
full_link_path.parent.mkdir(parents=True, exist_ok=True)
if full_link_path.is_symlink():
full_link_path.unlink()
full_link_path.symlink_to(relative_source)
links.append(link)
# Generate .gitignore file
gitignore_path = GIT_TOPLEVEL / ".git" / "info" / "exclude"
gitignore_path.parent.mkdir(parents=True, exist_ok=True)
with gitignore_path.open("w") as gitignore:
gitignore.write("# DO NOT EDIT! Autogenerated\n")
gitignore.write(
"# Symlinks for headers to be next to sources for development\n"
)
gitignore.write('# Run "maintainers/link-headers" to regenerate\n\n')
gitignore.write('# Run "maintainers/link-headers" to regenerate\n\n')
for link in links:
gitignore.write(f"/{link}\n")
if __name__ == "__main__":
main()
+3 -1
View File
@@ -6,7 +6,8 @@ project('nix-dev-shell', 'cpp',
subproject_dir : 'src',
default_options : [
'localstatedir=/nix/var',
]
],
meson_version : '>= 1.1'
)
# Internal Libraries
@@ -33,6 +34,7 @@ endif
# External C wrapper libraries
subproject('libutil-c')
subproject('libstore-c')
subproject('libfetchers-c')
subproject('libexpr-c')
subproject('libflake-c')
subproject('libmain-c')
@@ -12,6 +12,7 @@ add_project_arguments(
'-Werror=switch-enum',
'-Werror=undef',
'-Werror=unused-result',
'-Werror=sign-compare',
'-Wignored-qualifiers',
'-Wimplicit-fallthrough',
'-Wno-deprecated-declarations',
+2 -7
View File
@@ -160,13 +160,6 @@ let
];
separateDebugInfo = !stdenv.hostPlatform.isStatic;
hardeningDisable = lib.optional stdenv.hostPlatform.isStatic "pie";
env =
prevAttrs.env or { }
// lib.optionalAttrs (
stdenv.isLinux
&& !(stdenv.hostPlatform.isStatic && stdenv.system == "aarch64-linux")
&& !(stdenv.hostPlatform.useLLVM or false)
) { LDFLAGS = "-fuse-ld=gold"; };
};
mesonLibraryLayer = finalAttrs: prevAttrs: {
@@ -218,6 +211,7 @@ in
{
version = baseVersion + versionSuffix;
inherit versionSuffix;
inherit officialRelease;
inherit maintainers;
inherit filesetToSource;
@@ -340,6 +334,7 @@ in
nix-store-tests = callPackage ../src/libstore-tests/package.nix { };
nix-fetchers = callPackage ../src/libfetchers/package.nix { };
nix-fetchers-c = callPackage ../src/libfetchers-c/package.nix { };
nix-fetchers-tests = callPackage ../src/libfetchers-tests/package.nix { };
nix-expr = callPackage ../src/libexpr/package.nix { };
+3
View File
@@ -38,6 +38,7 @@ scope: {
aws-sdk-cpp =
(pkgs.aws-sdk-cpp.override {
apis = [
"identity-management"
"s3"
"transfer"
];
@@ -60,7 +61,9 @@ scope: {
"--with-container"
"--with-context"
"--with-coroutine"
"--with-iostreams"
];
enableIcu = false;
}).overrideAttrs
(old: {
# Need to remove `--with-*` to use `--with-libraries=...`
+13 -7
View File
@@ -71,12 +71,17 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
# We use this shell with the local checkout, not unpackPhase.
src = null;
env = {
# For `make format`, to work without installing pre-commit
_NIX_PRE_COMMIT_HOOKS_CONFIG = "${(pkgs.formats.yaml { }).generate "pre-commit-config.yaml"
modular.pre-commit.settings.rawConfig
}";
};
env =
{
# For `make format`, to work without installing pre-commit
_NIX_PRE_COMMIT_HOOKS_CONFIG = "${(pkgs.formats.yaml { }).generate "pre-commit-config.yaml"
modular.pre-commit.settings.rawConfig
}";
}
// lib.optionalAttrs stdenv.hostPlatform.isLinux {
CC_LD = "mold";
CXX_LD = "mold";
};
mesonFlags =
map (transformFlag "libutil") (ignoreCrossFile pkgs.nixComponents2.nix-util.mesonFlags)
@@ -116,7 +121,8 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
]
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
);
)
++ lib.optional stdenv.hostPlatform.isLinux pkgs.buildPackages.mold-wrapped;
buildInputs =
attrs.buildInputs or [ ]
+3
View File
@@ -15,6 +15,7 @@
nix-store-tests,
nix-fetchers,
nix-fetchers-c,
nix-fetchers-tests,
nix-expr,
@@ -54,6 +55,7 @@ let
nix-store
nix-store-c
nix-fetchers
nix-fetchers-c
nix-expr
nix-expr-c
nix-flake
@@ -230,6 +232,7 @@ stdenv.mkDerivation (finalAttrs: {
"nix-expr"
"nix-expr-c"
"nix-fetchers"
"nix-fetchers-c"
"nix-flake"
"nix-flake-c"
"nix-main"
+1
View File
@@ -48,6 +48,7 @@ let
"nix-store-test-support"
"nix-store-tests"
"nix-fetchers"
"nix-fetchers-c"
"nix-fetchers-tests"
"nix-expr"
"nix-expr-c"
+7
View File
@@ -62,6 +62,13 @@ else
end
end
# Only use MANPATH if it is already set. In general `man` will just simply
# pick up `.nix-profile/share/man` because is it close to `.nix-profile/bin`
# which is in the $PATH. For more info, run `manpath -d`.
if set --query MANPATH
set --export --prepend --path MANPATH "$NIX_LINK/share/man"
end
add_path "@localstatedir@/nix/profiles/default/bin"
add_path "$NIX_LINK/bin"
+9 -1
View File
@@ -1,5 +1,5 @@
# Only execute this file once per shell.
if test -z "$HOME" || test -z "$USER" || \
if test -z "$HOME" || \
test -n "$__ETC_PROFILE_NIX_SOURCED"
exit
end
@@ -53,6 +53,13 @@ else if test -e "$NIX_LINK/etc/ssl/certs/ca-bundle.crt" # fall back to cacert in
set --export NIX_SSL_CERT_FILE "$NIX_LINK/etc/ssl/certs/ca-bundle.crt"
else if test -e "$NIX_LINK/etc/ca-bundle.crt" # old cacert in Nix profile
set --export NIX_SSL_CERT_FILE "$NIX_LINK/etc/ca-bundle.crt"
else
# Fall back to what is in the nix profiles, favouring whatever is defined last.
for i in (string split ' ' $NIX_PROFILES)
if test -e "$i/etc/ssl/certs/ca-bundle.crt"
set --export NIX_SSL_CERT_FILE "$i/etc/ssl/certs/ca-bundle.crt"
end
end
end
# Only use MANPATH if it is already set. In general `man` will just simply
@@ -62,6 +69,7 @@ if set --query MANPATH
set --export --prepend --path MANPATH "$NIX_LINK/share/man"
end
add_path "@localstatedir@/nix/profiles/default/bin"
add_path "$NIX_LINK/bin"
# Cleanup
+5 -5
View File
@@ -16,7 +16,7 @@
#include "nix/store/globals.hh"
#include "nix/util/serialise.hh"
#include "nix/store/build-result.hh"
#include "nix/store/store-api.hh"
#include "nix/store/store-open.hh"
#include "nix/util/strings.hh"
#include "nix/store/derivations.hh"
#include "nix/store/local-store.hh"
@@ -42,9 +42,9 @@ static AutoCloseFD openSlotLock(const Machine & m, uint64_t slot)
return openLockFile(fmt("%s/%s-%d", currentLoad, escapeUri(m.storeUri.render()), slot), true);
}
static bool allSupportedLocally(Store & store, const std::set<std::string>& requiredFeatures) {
static bool allSupportedLocally(Store & store, const StringSet& requiredFeatures) {
for (auto & feature : requiredFeatures)
if (!store.systemFeatures.get().count(feature)) return false;
if (!store.config.systemFeatures.get().count(feature)) return false;
return true;
}
@@ -85,7 +85,7 @@ static int main_build_remote(int argc, char * * argv)
that gets cleared on reboot, but it wouldn't work on macOS. */
auto currentLoadName = "/current-load";
if (auto localStore = store.dynamic_pointer_cast<LocalFSStore>())
currentLoad = std::string { localStore->stateDir } + currentLoadName;
currentLoad = std::string { localStore->config.stateDir } + currentLoadName;
else
currentLoad = settings.nixStateDir + currentLoadName;
@@ -113,7 +113,7 @@ static int main_build_remote(int argc, char * * argv)
auto amWilling = readInt(source);
auto neededSystem = readString(source);
drvPath = store->parseStorePath(readString(source));
auto requiredFeatures = readStrings<std::set<std::string>>(source);
auto requiredFeatures = readStrings<StringSet>(source);
/* It would be possible to build locally after some builds clear out,
so don't show the warning now: */
+10 -5
View File
@@ -3,7 +3,7 @@
#include "nix/cmd/command.hh"
#include "nix/cmd/markdown.hh"
#include "nix/store/store-api.hh"
#include "nix/store/store-open.hh"
#include "nix/store/local-fs-store.hh"
#include "nix/store/derivations.hh"
#include "nix/expr/nixexpr.hh"
@@ -14,12 +14,10 @@
namespace nix {
RegisterCommand::Commands * RegisterCommand::commands = nullptr;
nix::Commands RegisterCommand::getCommandsFor(const std::vector<std::string> & prefix)
{
nix::Commands res;
for (auto & [name, command] : *RegisterCommand::commands)
for (auto & [name, command] : RegisterCommand::commands())
if (name.size() == prefix.size() + 1) {
bool equal = true;
for (size_t i = 0; i < prefix.size(); ++i)
@@ -40,7 +38,7 @@ nlohmann::json NixMultiCommand::toJSON()
void NixMultiCommand::run()
{
if (!command) {
std::set<std::string> subCommandTextLines;
StringSet subCommandTextLines;
for (auto & [name, _] : commands)
subCommandTextLines.insert(fmt("- `%s`", name));
std::string markdownError =
@@ -397,4 +395,11 @@ void createOutLinks(const std::filesystem::path & outLink, const BuiltPaths & bu
}
}
void MixOutLinkBase::createOutLinksMaybe(const std::vector<BuiltPathWithResult> & buildables, ref<Store> & store)
{
if (outLink != "")
if (auto store2 = store.dynamic_pointer_cast<LocalFSStore>())
createOutLinks(outLink, toBuiltPaths(buildables), *store2);
}
}
+20 -7
View File
@@ -9,7 +9,7 @@
#include "nix/fetchers/registry.hh"
#include "nix/flake/flakeref.hh"
#include "nix/flake/settings.hh"
#include "nix/store/store-api.hh"
#include "nix/store/store-open.hh"
#include "nix/cmd/command.hh"
#include "nix/fetchers/tarball.hh"
#include "nix/fetchers/fetch-to-store.hh"
@@ -18,7 +18,6 @@
namespace nix {
namespace fs { using namespace std::filesystem; }
fetchers::Settings fetchSettings;
@@ -35,7 +34,12 @@ EvalSettings evalSettings {
auto flakeRef = parseFlakeRef(fetchSettings, std::string { rest }, {}, true, false);
debug("fetching flake search path element '%s''", rest);
auto [accessor, lockedRef] = flakeRef.resolve(state.store).lazyFetch(state.store);
auto storePath = nix::fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName());
auto storePath = nix::fetchToStore(
state.fetchSettings,
*state.store,
SourcePath(accessor),
FetchMode::Copy,
lockedRef.input.getName());
state.allowPath(storePath);
return state.storePath(storePath);
},
@@ -123,8 +127,8 @@ MixEvalArgs::MixEvalArgs()
.category = category,
.labels = {"original-ref", "resolved-ref"},
.handler = {[&](std::string _from, std::string _to) {
auto from = parseFlakeRef(fetchSettings, _from, fs::current_path().string());
auto to = parseFlakeRef(fetchSettings, _to, fs::current_path().string());
auto from = parseFlakeRef(fetchSettings, _from, std::filesystem::current_path().string());
auto to = parseFlakeRef(fetchSettings, _to, std::filesystem::current_path().string());
fetchers::Attrs extraAttrs;
if (to.subdir != "") extraAttrs["dir"] = to.subdir;
fetchers::overrideRegistry(from.input, to.input, extraAttrs);
@@ -178,7 +182,11 @@ SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * bas
state.store,
state.fetchSettings,
EvalSettings::resolvePseudoUrl(s));
auto storePath = fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy);
auto storePath = fetchToStore(
state.fetchSettings,
*state.store,
SourcePath(accessor),
FetchMode::Copy);
return state.storePath(storePath);
}
@@ -186,7 +194,12 @@ SourcePath lookupFileArg(EvalState & state, std::string_view s, const Path * bas
experimentalFeatureSettings.require(Xp::Flakes);
auto flakeRef = parseFlakeRef(fetchSettings, std::string(s.substr(6)), {}, true, false);
auto [accessor, lockedRef] = flakeRef.resolve(state.store).lazyFetch(state.store);
auto storePath = nix::fetchToStore(*state.store, SourcePath(accessor), FetchMode::Copy, lockedRef.input.getName());
auto storePath = nix::fetchToStore(
state.fetchSettings,
*state.store,
SourcePath(accessor),
FetchMode::Copy,
lockedRef.input.getName());
state.allowPath(storePath);
return state.storePath(storePath);
}
+47 -7
View File
@@ -18,7 +18,7 @@ extern char ** savedArgv;
class EvalState;
struct Pos;
class Store;
class LocalFSStore;
struct LocalFSStore;
static constexpr Command::Category catHelp = -1;
static constexpr Command::Category catSecondary = 100;
@@ -285,13 +285,16 @@ struct StorePathCommand : public StorePathsCommand
struct RegisterCommand
{
typedef std::map<std::vector<std::string>, std::function<ref<Command>()>> Commands;
static Commands * commands;
static Commands & commands()
{
static Commands commands;
return commands;
}
RegisterCommand(std::vector<std::string> && name, std::function<ref<Command>()> command)
{
if (!commands)
commands = new Commands;
commands->emplace(name, command);
commands().emplace(name, command);
}
static nix::Commands getCommandsFor(const std::vector<std::string> & prefix);
@@ -363,7 +366,7 @@ void completeFlakeRefWithFragment(
const Strings & defaultFlakeAttrPaths,
std::string_view prefix);
std::string showVersions(const std::set<std::string> & versions);
std::string showVersions(const StringSet & versions);
void printClosureDiff(
ref<Store> store, const StorePath & beforePath, const StorePath & afterPath, std::string_view indent);
@@ -374,4 +377,41 @@ void printClosureDiff(
*/
void createOutLinks(const std::filesystem::path & outLink, const BuiltPaths & buildables, LocalFSStore & store);
}
/** `outLink` parameter, `createOutLinksMaybe` method. See `MixOutLinkByDefault`. */
struct MixOutLinkBase : virtual Args
{
/** Prefix for any output symlinks. Empty means do not write an output symlink. */
Path outLink;
MixOutLinkBase(const std::string & defaultOutLink)
: outLink(defaultOutLink)
{
}
void createOutLinksMaybe(const std::vector<BuiltPathWithResult> & buildables, ref<Store> & store);
};
/** `--out-link`, `--no-link`, `createOutLinksMaybe` */
struct MixOutLinkByDefault : MixOutLinkBase, virtual Args
{
MixOutLinkByDefault()
: MixOutLinkBase("result")
{
addFlag({
.longName = "out-link",
.shortName = 'o',
.description = "Use *path* as prefix for the symlinks to the build results. It defaults to `result`.",
.labels = {"path"},
.handler = {&outLink},
.completer = completePath,
});
addFlag({
.longName = "no-link",
.description = "Do not create symlinks to the build results.",
.handler = {&outLink, Path("")},
});
}
};
} // namespace nix
@@ -21,6 +21,7 @@ struct App
struct UnresolvedApp
{
App unresolved;
std::vector<BuiltPathWithResult> build(ref<Store> evalStore, ref<Store> store);
App resolve(ref<Store> evalStore, ref<Store> store);
};
+6 -3
View File
@@ -12,12 +12,15 @@ typedef std::function<void(int, char * *)> MainFunction;
struct RegisterLegacyCommand
{
typedef std::map<std::string, MainFunction> Commands;
static Commands * commands;
static Commands & commands() {
static Commands commands;
return commands;
}
RegisterLegacyCommand(const std::string & name, MainFunction fun)
{
if (!commands) commands = new Commands;
(*commands)[name] = fun;
commands()[name] = fun;
}
};
+1 -1
View File
@@ -72,7 +72,7 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths()
auto newOutputs = std::visit(overloaded {
[&](const ExtendedOutputsSpec::Default & d) -> OutputsSpec {
std::set<std::string> outputsToInstall;
StringSet outputsToInstall;
for (auto & output : packageInfo.queryOutputs(false, true))
outputsToInstall.insert(output.first);
if (outputsToInstall.empty())
+1 -1
View File
@@ -117,7 +117,7 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths()
.drvPath = makeConstantStorePathRef(std::move(drvPath)),
.outputs = std::visit(overloaded {
[&](const ExtendedOutputsSpec::Default & d) -> OutputsSpec {
std::set<std::string> outputsToInstall;
StringSet outputsToInstall;
if (auto aOutputSpecified = attr->maybeGetAttr(state->sOutputSpecified)) {
if (aOutputSpecified->getBool()) {
if (auto aOutputName = attr->maybeGetAttr("outputName"))
+3 -2
View File
@@ -45,7 +45,7 @@ ref<InstallableValue> InstallableValue::require(ref<Installable> installable)
std::optional<DerivedPathWithInfo> InstallableValue::trySinglePathToDerivedPaths(Value & v, const PosIdx pos, std::string_view errorCtx)
{
if (v.type() == nPath) {
auto storePath = fetchToStore(*state->store, v.path(), FetchMode::Copy);
auto storePath = fetchToStore(state->fetchSettings, *state->store, v.path(), FetchMode::Copy);
return {{
.path = DerivedPath::Opaque {
.path = std::move(storePath),
@@ -57,7 +57,8 @@ std::optional<DerivedPathWithInfo> InstallableValue::trySinglePathToDerivedPaths
else if (v.type() == nString) {
return {{
.path = DerivedPath::fromSingle(
state->coerceToSingleDerivedPath(pos, v, errorCtx)),
state->devirtualize(
state->coerceToSingleDerivedPath(pos, v, errorCtx))),
.info = make_ref<ExtraPathInfo>(),
}};
}
+6 -4
View File
@@ -31,8 +31,6 @@
namespace nix {
namespace fs { using namespace std::filesystem; }
void completeFlakeInputAttrPath(
AddCompletions & completions,
ref<EvalState> evalState,
@@ -343,7 +341,7 @@ void completeFlakeRefWithFragment(
auto flakeRefS = std::string(prefix.substr(0, hash));
// TODO: ideally this would use the command base directory instead of assuming ".".
auto flakeRef = parseFlakeRef(fetchSettings, expandTilde(flakeRefS), fs::current_path().string());
auto flakeRef = parseFlakeRef(fetchSettings, expandTilde(flakeRefS), std::filesystem::current_path().string());
auto evalCache = openEvalCache(*evalState,
std::make_shared<flake::LockedFlake>(lockFlake(
@@ -491,7 +489,11 @@ Installables SourceExprCommand::parseInstallables(
throw UsageError("'--file' and '--expr' are exclusive");
// FIXME: backward compatibility hack
if (file) evalSettings.pureEval = false;
if (file) {
if (evalSettings.pureEval && evalSettings.pureEval.overridden)
throw UsageError("'--file' is not compatible with '--pure-eval'");
evalSettings.pureEval = false;
}
auto state = getEvalState();
auto vFile = state->allocValue();
-7
View File
@@ -1,7 +0,0 @@
#include "nix/cmd/legacy.hh"
namespace nix {
RegisterLegacyCommand::Commands * RegisterLegacyCommand::commands = 0;
}
-1
View File
@@ -71,7 +71,6 @@ sources = files(
'installable-flake.cc',
'installable-value.cc',
'installables.cc',
'legacy.cc',
'markdown.cc',
'misc-store-flags.cc',
'network-proxy.cc',
+1 -1
View File
@@ -135,7 +135,7 @@ static constexpr const char * promptForType(ReplPromptType promptType)
case ReplPromptType::ReplPrompt:
return "nix-repl> ";
case ReplPromptType::ContinuationPrompt:
return " ";
return " > "; // 9 spaces + >
}
assert(false);
}
+1 -1
View File
@@ -12,7 +12,7 @@
#include "nix/expr/eval-settings.hh"
#include "nix/expr/attr-path.hh"
#include "nix/util/signals.hh"
#include "nix/store/store-api.hh"
#include "nix/store/store-open.hh"
#include "nix/store/log-store.hh"
#include "nix/cmd/common-eval-args.hh"
#include "nix/expr/get-drvs.hh"
+1 -3
View File
@@ -37,13 +37,11 @@ include_dirs = [include_directories('.')]
headers = files(
'nix_api_expr.h',
'nix_api_expr_internal.h',
'nix_api_external.h',
'nix_api_value.h',
)
# TODO move this header to libexpr, maybe don't use it in tests?
headers += files('nix_api_expr_internal.h')
subdir('nix-meson-build-support/export-all-symbols')
subdir('nix-meson-build-support/windows-version')
+3 -1
View File
@@ -199,7 +199,9 @@ EvalState * nix_state_create(nix_c_context * context, const char ** lookupPath_c
!= NIX_OK)
return nullptr;
return nix_eval_state_build(context, builder);
auto *state = nix_eval_state_build(context, builder);
nix_eval_state_builder_free(builder);
return state;
}
void nix_state_free(EvalState * state)
+5
View File
@@ -286,6 +286,11 @@ nix_err nix_gc_incref(nix_c_context * context, const void * object);
/**
* @brief Decrement the garbage collector reference counter for the given object
*
* We also provide typed `nix_*_decref` functions, which are
* - safer to use
* - easier to integrate when deriving bindings
* - allow more flexibility
*
* @param[out] context Optional, stores error information
* @param[in] object The object to stop referencing
*/
@@ -23,6 +23,11 @@ struct Arbitrary<NixStringContextElem::DrvDeep> {
static Gen<NixStringContextElem::DrvDeep> arbitrary();
};
template<>
struct Arbitrary<NixStringContextElem::Path> {
static Gen<NixStringContextElem::Path> arbitrary();
};
template<>
struct Arbitrary<NixStringContextElem> {
static Gen<NixStringContextElem> arbitrary();
@@ -15,6 +15,15 @@ Gen<NixStringContextElem::DrvDeep> Arbitrary<NixStringContextElem::DrvDeep>::arb
});
}
Gen<NixStringContextElem::Path> Arbitrary<NixStringContextElem::Path>::arbitrary()
{
return gen::map(gen::arbitrary<StorePath>(), [](StorePath storePath) {
return NixStringContextElem::Path{
.storePath = storePath,
};
});
}
Gen<NixStringContextElem> Arbitrary<NixStringContextElem>::arbitrary()
{
return gen::mapcat(
@@ -30,6 +39,9 @@ Gen<NixStringContextElem> Arbitrary<NixStringContextElem>::arbitrary()
case 2:
return gen::map(
gen::arbitrary<NixStringContextElem::Built>(), [](NixStringContextElem a) { return a; });
case 3:
return gen::map(
gen::arbitrary<NixStringContextElem::Path>(), [](NixStringContextElem a) { return a; });
default:
assert(false);
}
+7 -7
View File
@@ -33,7 +33,7 @@ namespace nix {
ASSERT_EQ(PrintToString(e.info().msg),
PrintToString(HintFmt("puppy")));
auto trace = e.info().traces.rbegin();
ASSERT_EQ(e.info().traces.size(), 2);
ASSERT_EQ(e.info().traces.size(), 2u);
ASSERT_EQ(PrintToString(trace->hint),
PrintToString(HintFmt("doggy")));
trace++;
@@ -54,8 +54,8 @@ namespace nix {
} catch (Error & e2) {
e.addTrace(state.positions[noPos], "beans2");
//e2.addTrace(state.positions[noPos], "Something", "");
ASSERT_TRUE(e.info().traces.size() == 2);
ASSERT_TRUE(e2.info().traces.size() == 0);
ASSERT_TRUE(e.info().traces.size() == 2u);
ASSERT_TRUE(e2.info().traces.size() == 0u);
ASSERT_FALSE(&e.info() == &e2.info());
}
}
@@ -71,7 +71,7 @@ namespace nix {
} catch (BaseError & e) { \
ASSERT_EQ(PrintToString(e.info().msg), \
PrintToString(message)); \
ASSERT_EQ(e.info().traces.size(), 1) << "while testing " args << std::endl << e.what(); \
ASSERT_EQ(e.info().traces.size(), 1u) << "while testing " args << std::endl << e.what(); \
auto trace = e.info().traces.rbegin(); \
ASSERT_EQ(PrintToString(trace->hint), \
PrintToString(HintFmt("while calling the '%s' builtin", name))); \
@@ -90,7 +90,7 @@ namespace nix {
} catch (BaseError & e) { \
ASSERT_EQ(PrintToString(e.info().msg), \
PrintToString(message)); \
ASSERT_EQ(e.info().traces.size(), 2) << "while testing " args << std::endl << e.what(); \
ASSERT_EQ(e.info().traces.size(), 2u) << "while testing " args << std::endl << e.what(); \
auto trace = e.info().traces.rbegin(); \
ASSERT_EQ(PrintToString(trace->hint), \
PrintToString(context)); \
@@ -112,7 +112,7 @@ namespace nix {
} catch (BaseError & e) { \
ASSERT_EQ(PrintToString(e.info().msg), \
PrintToString(message)); \
ASSERT_EQ(e.info().traces.size(), 3) << "while testing " args << std::endl << e.what(); \
ASSERT_EQ(e.info().traces.size(), 3u) << "while testing " args << std::endl << e.what(); \
auto trace = e.info().traces.rbegin(); \
ASSERT_EQ(PrintToString(trace->hint), \
PrintToString(context1)); \
@@ -137,7 +137,7 @@ namespace nix {
} catch (BaseError & e) { \
ASSERT_EQ(PrintToString(e.info().msg), \
PrintToString(message)); \
ASSERT_EQ(e.info().traces.size(), 4) << "while testing " args << std::endl << e.what(); \
ASSERT_EQ(e.info().traces.size(), 4u) << "while testing " args << std::endl << e.what(); \
auto trace = e.info().traces.rbegin(); \
ASSERT_EQ(PrintToString(trace->hint), \
PrintToString(context1)); \
+1 -1
View File
@@ -222,7 +222,7 @@ TEST_F(nix_api_expr_test, nix_expr_realise_context)
names.push_back(name);
}
std::sort(names.begin(), names.end());
ASSERT_EQ(3, names.size());
ASSERT_EQ(3u, names.size());
EXPECT_THAT(names[0], testing::StrEq("just-a-file"));
EXPECT_THAT(names[1], testing::StrEq("letsbuild"));
EXPECT_THAT(names[2], testing::StrEq("not-actually-built-yet.drv"));
+3
View File
@@ -63,6 +63,9 @@ TEST_F(nix_api_expr_test, nix_expr_eval_external)
std::string string_value;
nix_get_string(nullptr, valueResult, OBSERVE_STRING(string_value));
ASSERT_STREQ("nix-external<MyExternalValueDesc( 42 )>", string_value.c_str());
nix_state_free(stateResult);
nix_state_free(stateFn);
}
}
+7 -7
View File
@@ -134,12 +134,12 @@ TEST_F(nix_api_expr_test, nix_build_and_init_list_invalid)
{
ASSERT_EQ(nullptr, nix_get_list_byidx(ctx, nullptr, state, 0));
assert_ctx_err();
ASSERT_EQ(0, nix_get_list_size(ctx, nullptr));
ASSERT_EQ(0u, nix_get_list_size(ctx, nullptr));
assert_ctx_err();
ASSERT_EQ(nullptr, nix_get_list_byidx(ctx, value, state, 0));
assert_ctx_err();
ASSERT_EQ(0, nix_get_list_size(ctx, value));
ASSERT_EQ(0u, nix_get_list_size(ctx, value));
assert_ctx_err();
}
@@ -163,7 +163,7 @@ TEST_F(nix_api_expr_test, nix_build_and_init_list)
ASSERT_EQ(42, nix_get_int(ctx, nix_get_list_byidx(ctx, value, state, 0)));
ASSERT_EQ(43, nix_get_int(ctx, nix_get_list_byidx(ctx, value, state, 1)));
ASSERT_EQ(nullptr, nix_get_list_byidx(ctx, value, state, 2));
ASSERT_EQ(10, nix_get_list_size(ctx, value));
ASSERT_EQ(10u, nix_get_list_size(ctx, value));
ASSERT_STREQ("a list", nix_get_typename(ctx, value));
ASSERT_EQ(NIX_TYPE_LIST, nix_get_type(ctx, value));
@@ -180,7 +180,7 @@ TEST_F(nix_api_expr_test, nix_build_and_init_attr_invalid)
assert_ctx_err();
ASSERT_EQ(nullptr, nix_get_attr_name_byidx(ctx, nullptr, state, 0));
assert_ctx_err();
ASSERT_EQ(0, nix_get_attrs_size(ctx, nullptr));
ASSERT_EQ(0u, nix_get_attrs_size(ctx, nullptr));
assert_ctx_err();
ASSERT_EQ(false, nix_has_attr_byname(ctx, nullptr, state, "no-value"));
assert_ctx_err();
@@ -191,7 +191,7 @@ TEST_F(nix_api_expr_test, nix_build_and_init_attr_invalid)
assert_ctx_err();
ASSERT_EQ(nullptr, nix_get_attr_name_byidx(ctx, value, state, 0));
assert_ctx_err();
ASSERT_EQ(0, nix_get_attrs_size(ctx, value));
ASSERT_EQ(0u, nix_get_attrs_size(ctx, value));
assert_ctx_err();
ASSERT_EQ(false, nix_has_attr_byname(ctx, value, state, "no-value"));
assert_ctx_err();
@@ -215,7 +215,7 @@ TEST_F(nix_api_expr_test, nix_build_and_init_attr)
nix_make_attrs(ctx, value, builder);
nix_bindings_builder_free(builder);
ASSERT_EQ(2, nix_get_attrs_size(ctx, value));
ASSERT_EQ(2u, nix_get_attrs_size(ctx, value));
nix_value * out_value = nix_get_attr_byname(ctx, value, state, "a");
ASSERT_EQ(42, nix_get_int(ctx, out_value));
@@ -374,7 +374,7 @@ TEST_F(nix_api_expr_test, nix_value_init_apply_lazy_arg)
auto n = nix_get_attrs_size(ctx, r);
assert_ctx_ok();
ASSERT_EQ(1, n);
ASSERT_EQ(1u, n);
// nix_get_attr_byname isn't lazy (it could have been) so it will throw the exception
nix_value * foo = nix_get_attr_byname(ctx, r, state, "foo");
+25 -5
View File
@@ -56,11 +56,31 @@ namespace nix {
TEST_F(PrimOpTest, ceil) {
auto v = eval("builtins.ceil 1.9");
ASSERT_THAT(v, IsIntEq(2));
auto intMin = eval("builtins.ceil (-4611686018427387904 - 4611686018427387904)");
ASSERT_THAT(intMin, IsIntEq(std::numeric_limits<NixInt::Inner>::min()));
ASSERT_THROW(eval("builtins.ceil 1.0e200"), EvalError);
ASSERT_THROW(eval("builtins.ceil -1.0e200"), EvalError);
ASSERT_THROW(eval("builtins.ceil (1.0e200 * 1.0e200)"), EvalError); // inf
ASSERT_THROW(eval("builtins.ceil (-1.0e200 * 1.0e200)"), EvalError); // -inf
ASSERT_THROW(eval("builtins.ceil (1.0e200 * 1.0e200 - 1.0e200 * 1.0e200)"), EvalError); // nan
// bugs in previous Nix versions
ASSERT_THROW(eval("builtins.ceil (4611686018427387904 + 4611686018427387903)"), EvalError);
ASSERT_THROW(eval("builtins.ceil (-4611686018427387904 - 4611686018427387903)"), EvalError);
}
TEST_F(PrimOpTest, floor) {
auto v = eval("builtins.floor 1.9");
ASSERT_THAT(v, IsIntEq(1));
auto intMin = eval("builtins.ceil (-4611686018427387904 - 4611686018427387904)");
ASSERT_THAT(intMin, IsIntEq(std::numeric_limits<NixInt::Inner>::min()));
ASSERT_THROW(eval("builtins.ceil 1.0e200"), EvalError);
ASSERT_THROW(eval("builtins.ceil -1.0e200"), EvalError);
ASSERT_THROW(eval("builtins.ceil (1.0e200 * 1.0e200)"), EvalError); // inf
ASSERT_THROW(eval("builtins.ceil (-1.0e200 * 1.0e200)"), EvalError); // -inf
ASSERT_THROW(eval("builtins.ceil (1.0e200 * 1.0e200 - 1.0e200 * 1.0e200)"), EvalError); // nan
// bugs in previous Nix versions
ASSERT_THROW(eval("builtins.ceil (4611686018427387904 + 4611686018427387903)"), EvalError);
ASSERT_THROW(eval("builtins.ceil (-4611686018427387904 - 4611686018427387903)"), EvalError);
}
TEST_F(PrimOpTest, tryEvalFailure) {
@@ -204,7 +224,7 @@ namespace nix {
auto v = eval("builtins.listToAttrs []");
ASSERT_THAT(v, IsAttrsOfSize(0));
ASSERT_EQ(v.type(), nAttrs);
ASSERT_EQ(v.attrs()->size(), 0);
ASSERT_EQ(v.attrs()->size(), 0u);
}
TEST_F(PrimOpTest, listToAttrsNotFieldName) {
@@ -383,7 +403,7 @@ namespace nix {
TEST_F(PrimOpTest, genList) {
auto v = eval("builtins.genList (x: x + 1) 3");
ASSERT_EQ(v.type(), nList);
ASSERT_EQ(v.listSize(), 3);
ASSERT_EQ(v.listSize(), 3u);
for (const auto [i, elem] : enumerate(v.listItems())) {
ASSERT_THAT(*elem, IsThunk());
state.forceValue(*elem, noPos);
@@ -394,7 +414,7 @@ namespace nix {
TEST_F(PrimOpTest, sortLessThan) {
auto v = eval("builtins.sort builtins.lessThan [ 483 249 526 147 42 77 ]");
ASSERT_EQ(v.type(), nList);
ASSERT_EQ(v.listSize(), 6);
ASSERT_EQ(v.listSize(), 6u);
const std::vector<int> numbers = { 42, 77, 147, 249, 483, 526 };
for (const auto [n, elem] : enumerate(v.listItems()))
@@ -414,7 +434,7 @@ namespace nix {
auto wrong = v.attrs()->get(createSymbol("wrong"));
ASSERT_NE(wrong, nullptr);
ASSERT_EQ(wrong->value->type(), nList);
ASSERT_EQ(wrong->value->listSize(), 3);
ASSERT_EQ(wrong->value->listSize(), 3u);
ASSERT_THAT(*wrong->value, IsListOfSize(3));
ASSERT_THAT(*wrong->value->listElems()[0], IsIntEq(1));
ASSERT_THAT(*wrong->value->listElems()[1], IsIntEq(9));
@@ -424,7 +444,7 @@ namespace nix {
TEST_F(PrimOpTest, concatMap) {
auto v = eval("builtins.concatMap (x: x ++ [0]) [ [1 2] [3 4] ]");
ASSERT_EQ(v.type(), nList);
ASSERT_EQ(v.listSize(), 6);
ASSERT_EQ(v.listSize(), 6u);
const std::vector<int> numbers = { 1, 2, 0, 3, 4, 0 };
for (const auto [n, elem] : enumerate(v.listItems()))
+1 -1
View File
@@ -74,7 +74,7 @@ std::pair<Value *, PosIdx> findAlongAttrPath(EvalState & state, const std::strin
auto a = v->attrs()->get(state.symbols.create(attr));
if (!a) {
std::set<std::string> attrNames;
StringSet attrNames;
for (auto & attr : *v->attrs())
attrNames.insert(std::string(state.symbols[attr.name]));
+12 -9
View File
@@ -492,7 +492,7 @@ Value & AttrCursor::forceValue()
Suggestions AttrCursor::getSuggestionsForAttr(Symbol name)
{
auto attrNames = getAttrs();
std::set<std::string> strAttrNames;
StringSet strAttrNames;
for (auto & name : attrNames)
strAttrNames.insert(std::string(root->state.symbols[name]));
@@ -618,18 +618,21 @@ string_t AttrCursor::getStringWithContext()
if (auto s = std::get_if<string_t>(&cachedValue->second)) {
bool valid = true;
for (auto & c : s->second) {
const StorePath & path = std::visit(overloaded {
[&](const NixStringContextElem::DrvDeep & d) -> const StorePath & {
return d.drvPath;
const StorePath * path = std::visit(overloaded {
[&](const NixStringContextElem::DrvDeep & d) -> const StorePath * {
return &d.drvPath;
},
[&](const NixStringContextElem::Built & b) -> const StorePath & {
return b.drvPath->getBaseStorePath();
[&](const NixStringContextElem::Built & b) -> const StorePath * {
return &b.drvPath->getBaseStorePath();
},
[&](const NixStringContextElem::Opaque & o) -> const StorePath & {
return o.path;
[&](const NixStringContextElem::Opaque & o) -> const StorePath * {
return &o.path;
},
[&](const NixStringContextElem::Path & p) -> const StorePath * {
return nullptr;
},
}, c.raw);
if (!root->state.store->isValidPath(path)) {
if (!path || !root->state.store->isValidPath(*path)) {
valid = false;
break;
}
+1
View File
@@ -110,5 +110,6 @@ template class EvalErrorBuilder<UndefinedVarError>;
template class EvalErrorBuilder<MissingArgumentError>;
template class EvalErrorBuilder<InfiniteRecursionError>;
template class EvalErrorBuilder<InvalidPathError>;
template class EvalErrorBuilder<IFDError>;
}
+48
View File
@@ -0,0 +1,48 @@
#include "nix/expr/eval-profiler.hh"
#include "nix/expr/nixexpr.hh"
namespace nix {
void EvalProfiler::preFunctionCallHook(
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
}
void EvalProfiler::postFunctionCallHook(
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
}
void MultiEvalProfiler::preFunctionCallHook(
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
for (auto & profiler : profilers) {
if (profiler->getNeededHooks().test(Hook::preFunctionCall))
profiler->preFunctionCallHook(state, v, args, pos);
}
}
void MultiEvalProfiler::postFunctionCallHook(
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
for (auto & profiler : profilers) {
if (profiler->getNeededHooks().test(Hook::postFunctionCall))
profiler->postFunctionCallHook(state, v, args, pos);
}
}
EvalProfiler::Hooks MultiEvalProfiler::getNeededHooksImpl() const
{
Hooks hooks;
for (auto & p : profilers)
hooks |= p->getNeededHooks();
return hooks;
}
void MultiEvalProfiler::addProfiler(ref<EvalProfiler> profiler)
{
profilers.push_back(profiler);
invalidateNeededHooks();
}
}
+92 -23
View File
@@ -15,10 +15,12 @@
#include "nix/expr/print.hh"
#include "nix/fetchers/filtering-source-accessor.hh"
#include "nix/util/memory-source-accessor.hh"
#include "nix/util/mounted-source-accessor.hh"
#include "nix/expr/gc-small-vector.hh"
#include "nix/util/url.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/fetchers/tarball.hh"
#include "nix/fetchers/input-cache.hh"
#include "parser-tab.hh"
@@ -246,6 +248,30 @@ EvalState::EvalState(
}
, repair(NoRepair)
, emptyBindings(0)
, storeFS(
makeMountedSourceAccessor(
{
{CanonPath::root, makeEmptySourceAccessor()},
/* In the pure eval case, we can simply require
valid paths. However, in the *impure* eval
case this gets in the way of the union
mechanism, because an invalid access in the
upper layer will *not* be caught by the union
source accessor, but instead abort the entire
lookup.
This happens when the store dir in the
ambient file system has a path (e.g. because
another Nix store there), but the relocated
store does not.
TODO make the various source accessors doing
access control all throw the same type of
exception, and make union source accessor
catch it, so we don't need to do this hack.
*/
{CanonPath(store->storeDir), makeFSSourceAccessor(dirOf(store->toRealPath(StorePath::dummy)))}
}))
, rootFS(
({
/* In pure eval mode, we provide a filesystem that only
@@ -259,17 +285,9 @@ EvalState::EvalState(
/nix/store while using a chroot store. */
auto accessor = getFSSourceAccessor();
auto realStoreDir = dirOf(store->toRealPath(StorePath::dummy));
if (settings.pureEval || store->storeDir != realStoreDir) {
auto storeFS = makeMountedSourceAccessor(
{
{CanonPath::root, makeEmptySourceAccessor()},
{CanonPath(store->storeDir), makeFSSourceAccessor(realStoreDir)}
});
accessor = settings.pureEval
? storeFS
: makeUnionSourceAccessor({accessor, storeFS});
}
accessor = settings.pureEval
? storeFS.cast<SourceAccessor>()
: makeUnionSourceAccessor({accessor, storeFS});
/* Apply access control if needed. */
if (settings.restrictEval || settings.pureEval)
@@ -291,6 +309,7 @@ EvalState::EvalState(
)}
, store(store)
, buildStore(buildStore ? buildStore : store)
, inputCache(fetchers::InputCache::create())
, debugRepl(nullptr)
, debugStop(false)
, trylevel(0)
@@ -351,6 +370,10 @@ EvalState::EvalState(
);
createBaseEnv(settings);
/* Register function call tracer. */
if (settings.traceFunctionCalls)
profiler.addProfiler(make_ref<FunctionCallTrace>());
}
@@ -947,7 +970,16 @@ void EvalState::mkPos(Value & v, PosIdx p)
auto origin = positions.originOf(p);
if (auto path = std::get_if<SourcePath>(&origin)) {
auto attrs = buildBindings(3);
attrs.alloc(sFile).mkString(path->path.abs());
if (path->accessor == rootFS && store->isInStore(path->path.abs()))
// FIXME: only do this for virtual store paths?
attrs.alloc(sFile).mkString(path->path.abs(),
{
NixStringContextElem::Path{
.storePath = store->toStorePath(path->path.abs()).first
}
});
else
attrs.alloc(sFile).mkString(path->path.abs());
makePositionThunks(*this, p, attrs.alloc(sLine), attrs.alloc(sColumn));
v.mkAttrs(attrs);
} else
@@ -1133,6 +1165,7 @@ void EvalState::resetFileCache()
{
fileEvalCache.clear();
fileParseCache.clear();
inputCache->clear();
}
@@ -1424,7 +1457,7 @@ void ExprSelect::eval(EvalState & state, Env & env, Value & v)
} else {
state.forceAttrs(*vAttrs, pos, "while selecting an attribute");
if (!(j = vAttrs->attrs()->get(name))) {
std::set<std::string> allAttrNames;
StringSet allAttrNames;
for (auto & attr : *vAttrs->attrs())
allAttrNames.insert(std::string(state.symbols[attr.name]));
auto suggestions = Suggestions::bestMatches(allAttrNames, state.symbols[name]);
@@ -1504,9 +1537,14 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
{
auto _level = addCallDepth(pos);
auto trace = settings.traceFunctionCalls
? std::make_unique<FunctionCallTrace>(positions[pos])
: nullptr;
auto neededHooks = profiler.getNeededHooks();
if (neededHooks.test(EvalProfiler::preFunctionCall)) [[unlikely]]
profiler.preFunctionCallHook(*this, fun, args, pos);
Finally traceExit_{[&](){
if (profiler.getNeededHooks().test(EvalProfiler::postFunctionCall)) [[unlikely]]
profiler.postFunctionCallHook(*this, fun, args, pos);
}};
forceValue(fun, pos);
@@ -1581,7 +1619,7 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
user. */
for (auto & i : *args[0]->attrs())
if (!lambda.formals->has(i.name)) {
std::set<std::string> formalNames;
StringSet formalNames;
for (auto & formal : lambda.formals->formals)
formalNames.insert(std::string(symbols[formal.name]));
auto suggestions = Suggestions::bestMatches(formalNames, symbols[i.name]);
@@ -2067,7 +2105,7 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
else if (firstType == nFloat)
v.mkFloat(nf);
else if (firstType == nPath) {
if (!context.empty())
if (hasContext(context))
state.error<EvalError>("a string that refers to a store path cannot be appended to a path").atPos(pos).withFrame(env, *this).debugThrow();
v.mkPath(state.rootPath(CanonPath(str())));
} else
@@ -2266,7 +2304,10 @@ std::string_view EvalState::forceStringNoCtx(Value & v, const PosIdx pos, std::s
{
auto s = forceString(v, pos, errorCtx);
if (v.context()) {
error<EvalError>("the string '%1%' is not allowed to refer to a store path (such as '%2%')", v.string_view(), v.context()[0]).withTrace(pos, errorCtx).debugThrow();
NixStringContext context;
copyContext(v, context);
if (hasContext(context))
error<EvalError>("the string '%1%' is not allowed to refer to a store path (such as '%2%')", v.string_view(), v.context()[0]).withTrace(pos, errorCtx).debugThrow();
}
return s;
}
@@ -2315,6 +2356,9 @@ BackedStringView EvalState::coerceToString(
}
if (v.type() == nPath) {
// FIXME: instead of copying the path to the store, we could
// return a virtual store path that lazily copies the path to
// the store in devirtualize().
return
!canonicalizePath && !copyToStore
? // FIXME: hack to preserve path literals that end in a
@@ -2322,7 +2366,16 @@ BackedStringView EvalState::coerceToString(
v.payload.path.path
: copyToStore
? store->printStorePath(copyPathToStore(context, v.path()))
: std::string(v.path().path.abs());
: ({
auto path = v.path();
if (path.accessor == rootFS && store->isInStore(path.path.abs())) {
context.insert(
NixStringContextElem::Path{
.storePath = store->toStorePath(path.path.abs()).first
});
}
std::string(path.path.abs());
});
}
if (v.type() == nAttrs) {
@@ -2401,10 +2454,11 @@ StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePat
? *dstPathCached
: [&]() {
auto dstPath = fetchToStore(
fetchSettings,
*store,
path.resolveSymlinks(SymlinkResolution::Ancestors),
settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
path.baseName(),
computeBaseName(path),
ContentAddressMethod::Raw::NixArchive,
nullptr,
repair);
@@ -2459,7 +2513,7 @@ StorePath EvalState::coerceToStorePath(const PosIdx pos, Value & v, NixStringCon
auto path = coerceToString(pos, v, context, errorCtx, false, false, true).toOwned();
if (auto storePath = store->maybeParseStorePath(path))
return *storePath;
error<EvalError>("path '%1%' is not in the Nix store", path).withTrace(pos, errorCtx).debugThrow();
error<EvalError>("cannot coerce '%s' to a store path because it is not a subpath of the Nix store", path).withTrace(pos, errorCtx).debugThrow();
}
@@ -2485,6 +2539,11 @@ std::pair<SingleDerivedPath, std::string_view> EvalState::coerceToSingleDerivedP
[&](NixStringContextElem::Built && b) -> SingleDerivedPath {
return std::move(b);
},
[&](NixStringContextElem::Path && p) -> SingleDerivedPath {
error<EvalError>(
"string '%s' has no context",
s).withTrace(pos, errorCtx).debugThrow();
},
}, ((NixStringContextElem &&) *context.begin()).raw);
return {
std::move(derivedPath),
@@ -3068,6 +3127,11 @@ SourcePath EvalState::findFile(const LookupPath & lookupPath, const std::string_
auto res = (r / CanonPath(suffix)).resolveSymlinks();
if (res.pathExists()) return res;
// Backward compatibility hack: throw an exception if access
// to this path is not allowed.
if (auto accessor = res.accessor.dynamic_pointer_cast<FilteringSourceAccessor>())
accessor->checkAccess(res.path);
}
if (hasPrefix(path, "nix/"))
@@ -3103,7 +3167,7 @@ std::optional<SourcePath> EvalState::resolveLookupPathPath(const LookupPath::Pat
store,
fetchSettings,
EvalSettings::resolvePseudoUrl(value));
auto storePath = fetchToStore(*store, SourcePath(accessor), FetchMode::Copy);
auto storePath = fetchToStore(fetchSettings, *store, SourcePath(accessor), FetchMode::Copy);
return finish(this->storePath(storePath));
} catch (Error & e) {
logWarning({
@@ -3138,6 +3202,11 @@ std::optional<SourcePath> EvalState::resolveLookupPathPath(const LookupPath::Pat
if (path.resolveSymlinks().pathExists())
return finish(std::move(path));
else {
// Backward compatibility hack: throw an exception if access
// to this path is not allowed.
if (auto accessor = path.accessor.dynamic_pointer_cast<FilteringSourceAccessor>())
accessor->checkAccess(path.path);
logWarning({
.msg = HintFmt("Nix search path entry '%1%' does not exist, ignoring", value)
});
+8 -4
View File
@@ -3,16 +3,20 @@
namespace nix {
FunctionCallTrace::FunctionCallTrace(const Pos & pos) : pos(pos) {
void FunctionCallTrace::preFunctionCallHook(
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
auto duration = std::chrono::high_resolution_clock::now().time_since_epoch();
auto ns = std::chrono::duration_cast<std::chrono::nanoseconds>(duration);
printMsg(lvlInfo, "function-trace entered %1% at %2%", pos, ns.count());
printMsg(lvlInfo, "function-trace entered %1% at %2%", state.positions[pos], ns.count());
}
FunctionCallTrace::~FunctionCallTrace() {
void FunctionCallTrace::postFunctionCallHook(
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
auto duration = std::chrono::high_resolution_clock::now().time_since_epoch();
auto ns = std::chrono::duration_cast<std::chrono::nanoseconds>(duration);
printMsg(lvlInfo, "function-trace exited %1% at %2%", pos, ns.count());
printMsg(lvlInfo, "function-trace exited %1% at %2%", state.positions[pos], ns.count());
}
}
@@ -54,6 +54,7 @@ MakeError(TypeError, EvalError);
MakeError(UndefinedVarError, EvalError);
MakeError(MissingArgumentError, EvalError);
MakeError(InfiniteRecursionError, EvalError);
MakeError(IFDError, EvalBaseError);
struct InvalidPathError : public EvalError
{
@@ -0,0 +1,113 @@
#pragma once
/**
* @file
*
* Evaluation profiler interface definitions and builtin implementations.
*/
#include "nix/util/ref.hh"
#include <vector>
#include <span>
#include <bitset>
#include <optional>
namespace nix {
class EvalState;
class PosIdx;
struct Value;
class EvalProfiler
{
public:
enum Hook {
preFunctionCall,
postFunctionCall,
};
static constexpr std::size_t numHooks = Hook::postFunctionCall + 1;
using Hooks = std::bitset<numHooks>;
private:
std::optional<Hooks> neededHooks;
protected:
/** Invalidate the cached neededHooks. */
void invalidateNeededHooks()
{
neededHooks = std::nullopt;
}
/**
* Get which hooks need to be called.
*
* This is the actual implementation which has to be defined by subclasses.
* Public API goes through the needsHooks, which is a
* non-virtual interface (NVI) which caches the return value.
*/
virtual Hooks getNeededHooksImpl() const
{
return Hooks{};
}
public:
/**
* Hook called in the EvalState::callFunction preamble.
* Gets called only if (getNeededHooks().test(Hook::preFunctionCall)) is true.
*
* @param state Evaluator state.
* @param v Function being invoked.
* @param args Function arguments.
* @param pos Function position.
*/
virtual void
preFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos);
/**
* Hook called on EvalState::callFunction exit.
* Gets called only if (getNeededHooks().test(Hook::postFunctionCall)) is true.
*
* @param state Evaluator state.
* @param v Function being invoked.
* @param args Function arguments.
* @param pos Function position.
*/
virtual void
postFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos);
virtual ~EvalProfiler() = default;
/**
* Get which hooks need to be invoked for this EvalProfiler instance.
*/
Hooks getNeededHooks()
{
if (neededHooks.has_value())
return *neededHooks;
return *(neededHooks = getNeededHooksImpl());
}
};
/**
* Profiler that invokes multiple profilers at once.
*/
class MultiEvalProfiler : public EvalProfiler
{
std::vector<ref<EvalProfiler>> profilers;
[[gnu::noinline]] Hooks getNeededHooksImpl() const override;
public:
MultiEvalProfiler() = default;
/** Register a profiler instance. */
void addProfiler(ref<EvalProfiler> profiler);
[[gnu::noinline]] void
preFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
[[gnu::noinline]] void
postFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
};
}
@@ -247,6 +247,11 @@ struct EvalSettings : Config
This option can be enabled by setting `NIX_ABORT_ON_WARN=1` in the environment.
)"};
Setting<bool> lazyTrees{this, false, "lazy-trees",
R"(
If set to true, flakes and trees fetched by [`builtins.fetchTree`](@docroot@/language/builtins.md#builtins-fetchTree) are only copied to the Nix store when they're used as a dependency of a derivation. This avoids copying (potentially large) source trees unnecessarily.
)"};
};
/**
+51 -1
View File
@@ -3,6 +3,7 @@
#include "nix/expr/attr-set.hh"
#include "nix/expr/eval-error.hh"
#include "nix/expr/eval-profiler.hh"
#include "nix/util/types.hh"
#include "nix/expr/value.hh"
#include "nix/expr/nixexpr.hh"
@@ -33,13 +34,18 @@ namespace nix {
constexpr size_t maxPrimOpArity = 8;
class Store;
namespace fetchers { struct Settings; }
namespace fetchers {
struct Settings;
struct InputCache;
struct Input;
}
struct EvalSettings;
class EvalState;
class StorePath;
struct SingleDerivedPath;
enum RepairFlag : bool;
struct MemorySourceAccessor;
struct MountedSourceAccessor;
namespace eval_cache {
class EvalCache;
}
@@ -265,6 +271,11 @@ public:
/** `"unknown"` */
Value vStringUnknown;
/**
* The accessor corresponding to `store`.
*/
const ref<MountedSourceAccessor> storeFS;
/**
* The accessor for the root filesystem.
*/
@@ -295,6 +306,8 @@ public:
RootValue vImportedDrvToDerivation = nullptr;
ref<fetchers::InputCache> inputCache;
/**
* Debugger
*/
@@ -439,6 +452,15 @@ public:
void checkURI(const std::string & uri);
/**
* Mount an input on the Nix store.
*/
StorePath mountInput(
fetchers::Input & input,
const fetchers::Input & originalInput,
ref<SourceAccessor> accessor,
bool requireLockable);
/**
* Parse a Nix expression from the specified file.
*/
@@ -548,6 +570,18 @@ public:
std::optional<std::string> tryAttrsToString(const PosIdx pos, Value & v,
NixStringContext & context, bool coerceMore = false, bool copyToStore = true);
StorePath devirtualize(
const StorePath & path,
StringMap * rewrites = nullptr);
SingleDerivedPath devirtualize(
const SingleDerivedPath & path,
StringMap * rewrites = nullptr);
std::string devirtualize(
std::string_view s,
const NixStringContext & context);
/**
* String coercion.
*
@@ -563,6 +597,19 @@ public:
StorePath copyPathToStore(NixStringContext & context, const SourcePath & path);
/**
* Compute the base name for a `SourcePath`. For non-store paths,
* this is just `SourcePath::baseName()`. But for store paths, for
* backwards compatibility, it needs to be `<hash>-source`,
* i.e. as if the path were copied to the Nix store. This results
* in a "double-copied" store path like
* `/nix/store/<hash1>-<hash2>-source`. We don't need to
* materialize /nix/store/<hash2>-source though. Still, this
* requires reading/hashing the path twice.
*/
std::string computeBaseName(const SourcePath & path);
/**
* Path coercion.
*
@@ -893,6 +940,9 @@ private:
typedef std::map<ExprLambda *, size_t> FunctionCalls;
FunctionCalls functionCalls;
/** Evaluation/call profiler. */
MultiEvalProfiler profiler;
void incrFunctionCall(ExprLambda * fun);
typedef std::map<PosIdx, size_t> AttrSelects;
+15 -6
View File
@@ -2,15 +2,24 @@
///@file
#include "nix/expr/eval.hh"
#include <chrono>
#include "nix/expr/eval-profiler.hh"
namespace nix {
struct FunctionCallTrace
class FunctionCallTrace : public EvalProfiler
{
const Pos pos;
FunctionCallTrace(const Pos & pos);
~FunctionCallTrace();
Hooks getNeededHooksImpl() const override
{
return Hooks().set(preFunctionCall).set(postFunctionCall);
}
public:
FunctionCallTrace() = default;
[[gnu::noinline]] void
preFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
[[gnu::noinline]] void
postFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
};
}
+1
View File
@@ -14,6 +14,7 @@ headers = [config_pub_h] + files(
'eval-error.hh',
'eval-gc.hh',
'eval-inline.hh',
'eval-profiler.hh',
'eval-settings.hh',
'eval.hh',
'function-trace.hh',
+6 -1
View File
@@ -27,7 +27,12 @@ constexpr size_t conservativeStackReservation = 16;
struct RegisterPrimOp
{
typedef std::vector<PrimOp> PrimOps;
static PrimOps * primOps;
static PrimOps & primOps()
{
static PrimOps primOps;
return primOps;
}
/**
* You can register a constant by passing an arity of 0. fun
@@ -15,10 +15,10 @@ namespace nix {
* See: https://github.com/NixOS/nix/issues/9730
*/
void printAmbiguous(
Value &v,
const SymbolTable &symbols,
std::ostream &str,
std::set<const void *> *seen,
EvalState & state,
Value & v,
std::ostream & str,
std::set<const void *> * seen,
int depth);
}
+10 -7
View File
@@ -81,26 +81,29 @@ public:
class SymbolTable
{
private:
std::unordered_map<std::string_view, std::pair<const std::string *, uint32_t>> symbols;
/**
* Map from string view (backed by ChunkedVector) -> offset into the store.
* ChunkedVector references are never invalidated.
*/
std::unordered_map<std::string_view, uint32_t> symbols;
ChunkedVector<std::string, 8192> store{16};
public:
/**
* converts a string into a symbol.
* Converts a string into a symbol.
*/
Symbol create(std::string_view s)
{
// Most symbols are looked up more than once, so we trade off insertion performance
// for lookup performance.
// TODO: could probably be done more efficiently with transparent Hash and Equals
// on the original implementation using unordered_set
// FIXME: make this thread-safe.
auto it = symbols.find(s);
if (it != symbols.end()) return Symbol(it->second.second + 1);
if (it != symbols.end())
return Symbol(it->second + 1);
const auto & [rawSym, idx] = store.add(std::string(s));
symbols.emplace(rawSym, std::make_pair(&rawSym, idx));
const auto & [rawSym, idx] = store.add(s);
symbols.emplace(rawSym, idx);
return Symbol(idx + 1);
}
+32 -1
View File
@@ -54,10 +54,35 @@ struct NixStringContextElem {
*/
using Built = SingleDerivedPath::Built;
/**
* A store path that will not result in a store reference when
* used in a derivation or toFile.
*
* When you apply `builtins.toString` to a path value representing
* a path in the Nix store (as is the case with flake inputs),
* historically you got a string without context
* (e.g. `/nix/store/...-source`). This is broken, since it allows
* you to pass a store path to a derivation/toFile without a
* proper store reference. This is especially a problem with lazy
* trees, since the store path is a virtual path that doesn't
* exist.
*
* For backwards compatibility, and to warn users about this
* unsafe use of `toString`, we keep track of such strings as a
* special type of context.
*/
struct Path
{
StorePath storePath;
GENERATE_CMP(Path, me->storePath);
};
using Raw = std::variant<
Opaque,
DrvDeep,
Built
Built,
Path
>;
Raw raw;
@@ -82,4 +107,10 @@ struct NixStringContextElem {
typedef std::set<NixStringContextElem> NixStringContext;
/**
* Returns false if `context` has no elements other than
* `NixStringContextElem::Path`.
*/
bool hasContext(const NixStringContext & context);
}
+1
View File
@@ -1,3 +1,4 @@
%option 8bit
%option reentrant bison-bridge bison-locations
%option align
%option noyywrap
+2
View File
@@ -112,6 +112,7 @@ lexer_tab = custom_target(
],
command : [
'flex',
'-Cf', # Use full scanner tables
'--outfile',
'@OUTPUT0@',
'--header-file=' + '@OUTPUT1@',
@@ -139,6 +140,7 @@ sources = files(
'eval-cache.cc',
'eval-error.cc',
'eval-gc.cc',
'eval-profiler.cc',
'eval-settings.cc',
'eval.cc',
'function-trace.cc',
+6 -1
View File
@@ -179,7 +179,12 @@ static Expr * makeCall(PosIdx pos, Expr * fn, Expr * arg) {
%%
start: expr { state->result = $1; };
start: expr {
state->result = $1;
// This parser does not use yynerrs; suppress the warning.
(void) yynerrs;
};
expr: expr_function;
+85
View File
@@ -1,5 +1,7 @@
#include "nix/store/store-api.hh"
#include "nix/expr/eval.hh"
#include "nix/util/mounted-source-accessor.hh"
#include "nix/fetchers/fetch-to-store.hh"
namespace nix {
@@ -18,4 +20,87 @@ SourcePath EvalState::storePath(const StorePath & path)
return {rootFS, CanonPath{store->printStorePath(path)}};
}
StorePath EvalState::devirtualize(const StorePath & path, StringMap * rewrites)
{
if (auto mount = storeFS->getMount(CanonPath(store->printStorePath(path)))) {
auto storePath = fetchToStore(
fetchSettings,
*store,
SourcePath{ref(mount)},
settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
path.name());
assert(storePath.name() == path.name());
if (rewrites)
rewrites->emplace(path.hashPart(), storePath.hashPart());
return storePath;
} else
return path;
}
SingleDerivedPath EvalState::devirtualize(const SingleDerivedPath & path, StringMap * rewrites)
{
if (auto o = std::get_if<SingleDerivedPath::Opaque>(&path.raw()))
return SingleDerivedPath::Opaque{devirtualize(o->path, rewrites)};
else
return path;
}
std::string EvalState::devirtualize(std::string_view s, const NixStringContext & context)
{
StringMap rewrites;
for (auto & c : context)
if (auto o = std::get_if<NixStringContextElem::Opaque>(&c.raw))
devirtualize(o->path, &rewrites);
return rewriteStrings(std::string(s), rewrites);
}
std::string EvalState::computeBaseName(const SourcePath & path)
{
if (path.accessor == rootFS) {
if (auto storePath = store->maybeParseStorePath(path.path.abs())) {
warn(
"Performing inefficient double copy of path '%s' to the store. "
"This can typically be avoided by rewriting an attribute like `src = ./.` "
"to `src = builtins.path { path = ./.; name = \"source\"; }`.",
path);
return std::string(
fetchToStore(fetchSettings, *store, path, FetchMode::DryRun, storePath->name()).to_string());
}
}
return std::string(path.baseName());
}
StorePath EvalState::mountInput(
fetchers::Input & input, const fetchers::Input & originalInput, ref<SourceAccessor> accessor, bool requireLockable)
{
auto storePath = settings.lazyTrees
? StorePath::random(input.getName())
: fetchToStore(fetchSettings, *store, accessor, FetchMode::Copy, input.getName());
allowPath(storePath); // FIXME: should just whitelist the entire virtual store
storeFS->mount(CanonPath(store->printStorePath(storePath)), accessor);
if (requireLockable && (!settings.lazyTrees || !input.isLocked()) && !input.getNarHash()) {
auto narHash = accessor->hashPath(CanonPath::root);
input.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
}
// FIXME: what to do with the NAR hash in lazy mode?
if (!settings.lazyTrees && originalInput.getNarHash()) {
auto expected = originalInput.computeStorePath(*store);
if (storePath != expected)
throw Error(
(unsigned int) 102,
"NAR hash mismatch in input '%s', expected '%s' but got '%s'",
originalInput.to_string(),
store->printStorePath(storePath),
store->printStorePath(expected));
}
return storePath;
}
}
+121 -34
View File
@@ -14,6 +14,7 @@
#include "nix/expr/value-to-xml.hh"
#include "nix/expr/primops.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/util/mounted-source-accessor.hh"
#include <boost/container/small_vector.hpp>
#include <nlohmann/json.hpp>
@@ -75,7 +76,10 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
ensureValid(b.drvPath->getBaseStorePath());
},
[&](const NixStringContextElem::Opaque & o) {
ensureValid(o.path);
// We consider virtual store paths valid here. They'll
// be devirtualized if needed elsewhere.
if (!storeFS->getMount(CanonPath(store->printStorePath(o.path))))
ensureValid(o.path);
if (maybePathsOut)
maybePathsOut->emplace(o.path);
},
@@ -85,13 +89,16 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
if (maybePathsOut)
maybePathsOut->emplace(d.drvPath);
},
[&](const NixStringContextElem::Path & p) {
// FIXME: do something?
},
}, c.raw);
}
if (drvs.empty()) return {};
if (isIFD && !settings.enableImportFromDerivation)
error<EvalBaseError>(
error<IFDError>(
"cannot build '%1%' during evaluation because the option 'allow-import-from-derivation' is disabled",
drvs.begin()->to_string(*store)
).debugThrow();
@@ -888,18 +895,40 @@ static void prim_ceil(EvalState & state, const PosIdx pos, Value * * args, Value
{
auto value = state.forceFloat(*args[0], args[0]->determinePos(pos),
"while evaluating the first argument passed to builtins.ceil");
v.mkInt(ceil(value));
auto ceilValue = ceil(value);
bool isInt = args[0]->type() == nInt;
constexpr NixFloat int_min = std::numeric_limits<NixInt::Inner>::min(); // power of 2, so that no rounding occurs
if (ceilValue >= int_min && ceilValue < -int_min) {
v.mkInt(ceilValue);
} else if (isInt) {
// a NixInt, e.g. INT64_MAX, can be rounded to -int_min due to the cast to NixFloat
state.error<EvalError>("Due to a bug (see https://github.com/NixOS/nix/issues/12899) the NixInt argument %1% caused undefined behavior in previous Nix versions.\n\tFuture Nix versions might implement the correct behavior.", args[0]->integer().value).atPos(pos).debugThrow();
} else {
state.error<EvalError>("NixFloat argument %1% is not in the range of NixInt", args[0]->fpoint()).atPos(pos).debugThrow();
}
// `forceFloat` casts NixInt to NixFloat, but instead NixInt args shall be returned unmodified
if (isInt) {
auto arg = args[0]->integer();
auto res = v.integer();
if (arg != res) {
state.error<EvalError>("Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occured in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.", arg, res).atPos(pos).debugThrow();
}
}
}
static RegisterPrimOp primop_ceil({
.name = "__ceil",
.args = {"double"},
.args = {"number"},
.doc = R"(
Converts an IEEE-754 double-precision floating-point number (*double*) to
the next higher integer.
Rounds and converts *number* to the next higher NixInt value if possible, i.e. `ceil *number* >= *number*` and
`ceil *number* - *number* < 1`.
If the datatype is neither an integer nor a "float", an evaluation error will be
thrown.
An evaluation error is thrown, if there exists no such NixInt value `ceil *number*`.
Due to bugs in previous Nix versions an evaluation error might be thrown, if the datatype of *number* is
a NixInt and if `*number* < -9007199254740992` or `*number* > 9007199254740992`.
If the datatype of *number* is neither a NixInt (signed 64-bit integer) nor a NixFloat
(IEEE-754 double-precision floating-point number), an evaluation error will be thrown.
)",
.fun = prim_ceil,
});
@@ -907,18 +936,40 @@ static RegisterPrimOp primop_ceil({
static void prim_floor(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
auto value = state.forceFloat(*args[0], args[0]->determinePos(pos), "while evaluating the first argument passed to builtins.floor");
v.mkInt(floor(value));
auto floorValue = floor(value);
bool isInt = args[0]->type() == nInt;
constexpr NixFloat int_min = std::numeric_limits<NixInt::Inner>::min(); // power of 2, so that no rounding occurs
if (floorValue >= int_min && floorValue < -int_min) {
v.mkInt(floorValue);
} else if (isInt) {
// a NixInt, e.g. INT64_MAX, can be rounded to -int_min due to the cast to NixFloat
state.error<EvalError>("Due to a bug (see https://github.com/NixOS/nix/issues/12899) the NixInt argument %1% caused undefined behavior in previous Nix versions.\n\tFuture Nix versions might implement the correct behavior.", args[0]->integer().value).atPos(pos).debugThrow();
} else {
state.error<EvalError>("NixFloat argument %1% is not in the range of NixInt", args[0]->fpoint()).atPos(pos).debugThrow();
}
// `forceFloat` casts NixInt to NixFloat, but instead NixInt args shall be returned unmodified
if (isInt) {
auto arg = args[0]->integer();
auto res = v.integer();
if (arg != res) {
state.error<EvalError>("Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occured in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.", arg, res).atPos(pos).debugThrow();
}
}
}
static RegisterPrimOp primop_floor({
.name = "__floor",
.args = {"double"},
.args = {"number"},
.doc = R"(
Converts an IEEE-754 double-precision floating-point number (*double*) to
the next lower integer.
Rounds and converts *number* to the next lower NixInt value if possible, i.e. `floor *number* <= *number*` and
`*number* - floor *number* < 1`.
If the datatype is neither an integer nor a "float", an evaluation error will be
thrown.
An evaluation error is thrown, if there exists no such NixInt value `floor *number*`.
Due to bugs in previous Nix versions an evaluation error might be thrown, if the datatype of *number* is
a NixInt and if `*number* < -9007199254740992` or `*number* > 9007199254740992`.
If the datatype of *number* is neither a NixInt (signed 64-bit integer) nor a NixFloat
(IEEE-754 double-precision floating-point number), an evaluation error will be thrown.
)",
.fun = prim_floor,
});
@@ -1408,6 +1459,10 @@ static void derivationStrictInternal(
/* Everything in the context of the strings in the derivation
attributes should be added as dependencies of the resulting
derivation. */
StringMap rewrites;
std::optional<std::string> drvS;
for (auto & c : context) {
std::visit(overloaded {
/* Since this allows the builder to gain access to every
@@ -1430,11 +1485,24 @@ static void derivationStrictInternal(
drv.inputDrvs.ensureSlot(*b.drvPath).value.insert(b.output);
},
[&](const NixStringContextElem::Opaque & o) {
drv.inputSrcs.insert(o.path);
drv.inputSrcs.insert(state.devirtualize(o.path, &rewrites));
},
[&](const NixStringContextElem::Path & p) {
if (!drvS) drvS = drv.unparse(*state.store, true);
if (drvS->find(p.storePath.to_string()) != drvS->npos) {
auto devirtualized = state.devirtualize(p.storePath, &rewrites);
warn(
"Using 'builtins.derivation' to create a derivation named '%s' that references the store path '%s' without a proper context. "
"The resulting derivation will not have a correct store reference, so this is unreliable and may stop working in the future.",
drvName,
state.store->printStorePath(devirtualized));
}
},
}, c.raw);
}
drv.applyRewrites(rewrites);
/* Do we have all required attributes? */
if (drv.builder == "")
state.error<EvalError>("required attribute 'builder' missing")
@@ -2338,10 +2406,21 @@ static void prim_toFile(EvalState & state, const PosIdx pos, Value * * args, Val
std::string contents(state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.toFile"));
StorePathSet refs;
StringMap rewrites;
for (auto c : context) {
if (auto p = std::get_if<NixStringContextElem::Opaque>(&c.raw))
refs.insert(p->path);
else if (auto p = std::get_if<NixStringContextElem::Path>(&c.raw)) {
if (contents.find(p->storePath.to_string()) != contents.npos) {
auto devirtualized = state.devirtualize(p->storePath, &rewrites);
warn(
"Using 'builtins.toFile' to create a file named '%s' that references the store path '%s' without a proper context. "
"The resulting file will not have a correct store reference, so this is unreliable and may stop working in the future.",
name,
state.store->printStorePath(devirtualized));
}
}
else
state.error<EvalError>(
"files created by %1% may not reference derivations, but %2% references %3%",
@@ -2351,6 +2430,8 @@ static void prim_toFile(EvalState & state, const PosIdx pos, Value * * args, Val
).atPos(pos).debugThrow();
}
contents = rewriteStrings(contents, rewrites);
auto storePath = settings.readOnlyMode
? state.store->makeFixedOutputPathFromCA(name, TextInfo {
.hash = hashString(HashAlgorithm::SHA256, contents),
@@ -2500,7 +2581,9 @@ static void addPath(
{}));
if (!expectedHash || !state.store->isValidPath(*expectedStorePath)) {
// FIXME: make this lazy?
auto dstPath = fetchToStore(
state.fetchSettings,
*state.store,
path.resolveSymlinks(),
settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
@@ -2530,7 +2613,7 @@ static void prim_filterSource(EvalState & state, const PosIdx pos, Value * * arg
"while evaluating the second argument (the path to filter) passed to 'builtins.filterSource'");
state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.filterSource");
addPath(state, pos, path.baseName(), path, args[0], ContentAddressMethod::Raw::NixArchive, std::nullopt, v, context);
addPath(state, pos, state.computeBaseName(path), path, args[0], ContentAddressMethod::Raw::NixArchive, std::nullopt, v, context);
}
static RegisterPrimOp primop_filterSource({
@@ -2775,7 +2858,13 @@ static void prim_unsafeGetAttrPos(EvalState & state, const PosIdx pos, Value * *
static RegisterPrimOp primop_unsafeGetAttrPos(PrimOp {
.name = "__unsafeGetAttrPos",
.args = {"s", "set"},
.arity = 2,
.doc = R"(
`unsafeGetAttrPos` returns the position of the attribute named *s*
from *set*. This is used by Nixpkgs to provide location information
in error messages.
)",
.fun = prim_unsafeGetAttrPos,
});
@@ -4261,9 +4350,7 @@ struct RegexCache
{
struct State
{
// TODO use C++20 transparent comparison when available
std::unordered_map<std::string_view, std::regex> cache;
std::list<std::string> keys;
std::unordered_map<std::string, std::regex, StringViewHash, std::equal_to<>> cache;
};
Sync<State> state_;
@@ -4274,8 +4361,14 @@ struct RegexCache
auto it = state->cache.find(re);
if (it != state->cache.end())
return it->second;
state->keys.emplace_back(re);
return state->cache.emplace(state->keys.back(), std::regex(state->keys.back(), std::regex::extended)).first->second;
/* No std::regex constructor overload from std::string_view, but can be constructed
from a pointer + size or an iterator range. */
return state->cache
.emplace(
std::piecewise_construct,
std::forward_as_tuple(re),
std::forward_as_tuple(/*s=*/re.data(), /*count=*/re.size(), std::regex::extended))
.first->second;
}
};
@@ -4659,13 +4752,9 @@ static RegisterPrimOp primop_splitVersion({
*************************************************************/
RegisterPrimOp::PrimOps * RegisterPrimOp::primOps;
RegisterPrimOp::RegisterPrimOp(PrimOp && primOp)
{
if (!primOps) primOps = new PrimOps;
primOps->push_back(std::move(primOp));
primOps().push_back(std::move(primOp));
}
@@ -4919,14 +5008,12 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings)
)",
});
if (RegisterPrimOp::primOps)
for (auto & primOp : *RegisterPrimOp::primOps)
if (experimentalFeatureSettings.isEnabled(primOp.experimentalFeature))
{
auto primOpAdjusted = primOp;
primOpAdjusted.arity = std::max(primOp.args.size(), primOp.arity);
addPrimOp(std::move(primOpAdjusted));
}
for (auto & primOp : RegisterPrimOp::primOps())
if (experimentalFeatureSettings.isEnabled(primOp.experimentalFeature)) {
auto primOpAdjusted = primOp;
primOpAdjusted.arity = std::max(primOp.args.size(), primOp.arity);
addPrimOp(std::move(primOpAdjusted));
}
for (auto & primOp : evalSettings.extraPrimOps) {
auto primOpAdjusted = primOp;
+24 -4
View File
@@ -7,9 +7,15 @@ namespace nix {
static void prim_unsafeDiscardStringContext(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
NixStringContext context;
NixStringContext context, filtered;
auto s = state.coerceToString(pos, *args[0], context, "while evaluating the argument passed to builtins.unsafeDiscardStringContext");
v.mkString(*s);
for (auto & c : context)
if (auto * p = std::get_if<NixStringContextElem::Path>(&c.raw))
filtered.insert(*p);
v.mkString(*s, filtered);
}
static RegisterPrimOp primop_unsafeDiscardStringContext({
@@ -21,12 +27,19 @@ static RegisterPrimOp primop_unsafeDiscardStringContext({
.fun = prim_unsafeDiscardStringContext,
});
bool hasContext(const NixStringContext & context)
{
for (auto & c : context)
if (!std::get_if<NixStringContextElem::Path>(&c.raw))
return true;
return false;
}
static void prim_hasContext(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
NixStringContext context;
state.forceString(*args[0], context, pos, "while evaluating the argument passed to builtins.hasContext");
v.mkBool(!context.empty());
v.mkBool(hasContext(context));
}
static RegisterPrimOp primop_hasContext({
@@ -103,7 +116,7 @@ static void prim_addDrvOutputDependencies(EvalState & state, const PosIdx pos, V
NixStringContext context;
auto s = state.coerceToString(pos, *args[0], context, "while evaluating the argument passed to builtins.addDrvOutputDependencies");
auto contextSize = context.size();
auto contextSize = context.size();
if (contextSize != 1) {
state.error<EvalError>(
"context of string '%s' must have exactly one element, but has %d",
@@ -136,6 +149,11 @@ static void prim_addDrvOutputDependencies(EvalState & state, const PosIdx pos, V
above does not make much sense. */
return std::move(c);
},
[&](const NixStringContextElem::Path & p) -> NixStringContextElem::DrvDeep {
state.error<EvalError>(
"`addDrvOutputDependencies` does not work on a string without context"
).atPos(pos).debugThrow();
},
}, context.begin()->raw) }),
};
@@ -206,6 +224,8 @@ static void prim_getContext(EvalState & state, const PosIdx pos, Value * * args,
[&](NixStringContextElem::Opaque && o) {
contextInfos[std::move(o.path)].path = true;
},
[&](NixStringContextElem::Path && p) {
},
}, ((NixStringContextElem &&) i).raw);
}
+1 -1
View File
@@ -1,5 +1,5 @@
#include "nix/expr/primops.hh"
#include "nix/store/store-api.hh"
#include "nix/store/store-open.hh"
#include "nix/store/realisation.hh"
#include "nix/store/make-content-addressed.hh"
#include "nix/util/url.hh"
+1 -1
View File
@@ -64,7 +64,7 @@ static void prim_fetchMercurial(EvalState & state, const PosIdx pos, Value * * a
if (rev) attrs.insert_or_assign("rev", rev->gitRev());
auto input = fetchers::Input::fromAttrs(state.fetchSettings, std::move(attrs));
auto [storePath, input2] = input.fetchToStore(state.store);
auto [storePath, accessor, input2] = input.fetchToStore(state.store);
auto attrs2 = state.buildBindings(8);
state.mkStorePathString(storePath, attrs2.alloc(state.sOutPath));
+7 -4
View File
@@ -10,6 +10,8 @@
#include "nix/util/url.hh"
#include "nix/expr/value-to-json.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/fetchers/input-cache.hh"
#include "nix/util/mounted-source-accessor.hh"
#include <nlohmann/json.hpp>
@@ -204,11 +206,11 @@ static void fetchTree(
throw Error("input '%s' is not allowed to use the '__final' attribute", input.to_string());
}
auto [storePath, input2] = input.fetchToStore(state.store);
auto cachedInput = state.inputCache->getAccessor(state.store, input, fetchers::UseRegistries::No);
state.allowPath(storePath);
auto storePath = state.mountInput(cachedInput.lockedInput, input, cachedInput.accessor, true);
emitTreeAttrs(state, storePath, input2, v, params.emptyRevFallback, false);
emitTreeAttrs(state, storePath, cachedInput.lockedInput, v, params.emptyRevFallback, false);
}
static void prim_fetchTree(EvalState & state, const PosIdx pos, Value * * args, Value & v)
@@ -537,11 +539,12 @@ static void fetch(EvalState & state, const PosIdx pos, Value * * args, Value & v
auto storePath =
unpack
? fetchToStore(
state.fetchSettings,
*state.store,
fetchers::downloadTarball(state.store, state.fetchSettings, *url),
FetchMode::Copy,
name)
: fetchers::downloadFile(state.store, *url, name).storePath;
: fetchers::downloadFile(state.store, state.fetchSettings, *url, name).storePath;
if (expectedHash) {
auto hash = unpack
+14 -10
View File
@@ -7,10 +7,10 @@ namespace nix {
// See: https://github.com/NixOS/nix/issues/9730
void printAmbiguous(
Value &v,
const SymbolTable &symbols,
std::ostream &str,
std::set<const void *> *seen,
EvalState & state,
Value & v,
std::ostream & str,
std::set<const void *> * seen,
int depth)
{
checkInterrupt();
@@ -26,9 +26,13 @@ void printAmbiguous(
case nBool:
printLiteralBool(str, v.boolean());
break;
case nString:
printLiteralString(str, v.string_view());
case nString: {
NixStringContext context;
copyContext(v, context);
// FIXME: make devirtualization configurable?
printLiteralString(str, state.devirtualize(v.string_view(), context));
break;
}
case nPath:
str << v.path().to_string(); // !!! escaping?
break;
@@ -40,9 +44,9 @@ void printAmbiguous(
str << "«repeated»";
else {
str << "{ ";
for (auto & i : v.attrs()->lexicographicOrder(symbols)) {
str << symbols[i->name] << " = ";
printAmbiguous(*i->value, symbols, str, seen, depth - 1);
for (auto & i : v.attrs()->lexicographicOrder(state.symbols)) {
str << state.symbols[i->name] << " = ";
printAmbiguous(state, *i->value, str, seen, depth - 1);
str << "; ";
}
str << "}";
@@ -56,7 +60,7 @@ void printAmbiguous(
str << "[ ";
for (auto v2 : v.listItems()) {
if (v2)
printAmbiguous(*v2, symbols, str, seen, depth - 1);
printAmbiguous(state, *v2, str, seen, depth - 1);
else
str << "(nullptr)";
str << " ";
+5 -1
View File
@@ -249,7 +249,11 @@ private:
void printString(Value & v)
{
printLiteralString(output, v.string_view(), options.maxStringLength, options.ansiColors);
NixStringContext context;
copyContext(v, context);
std::ostringstream s;
printLiteralString(s, v.string_view(), options.maxStringLength, options.ansiColors);
output << state.devirtualize(s.str(), context);
}
void printPath(Value & v)
+3 -1
View File
@@ -7,9 +7,11 @@
#include <iomanip>
#include <nlohmann/json.hpp>
namespace nix {
using json = nlohmann::json;
// TODO: rename. It doesn't print.
json printValueAsJSON(EvalState & state, bool strict,
Value & v, const PosIdx pos, NixStringContext & context, bool copyToStore)
{
+9
View File
@@ -57,6 +57,11 @@ NixStringContextElem NixStringContextElem::parse(
.drvPath = StorePath { s.substr(1) },
};
}
case '@': {
return NixStringContextElem::Path {
.storePath = StorePath { s.substr(1) },
};
}
default: {
// Ensure no '!'
if (s.find("!") != std::string_view::npos) {
@@ -100,6 +105,10 @@ std::string NixStringContextElem::to_string() const
res += '=';
res += d.drvPath.to_string();
},
[&](const NixStringContextElem::Path & p) {
res += '@';
res += p.storePath.to_string();
},
}, raw);
return res;
+1
View File
@@ -0,0 +1 @@
../../.version
+65
View File
@@ -0,0 +1,65 @@
project('nix-fetchers-c', 'cpp',
version : files('.version'),
default_options : [
'cpp_std=c++2a',
# TODO(Qyriad): increase the warning level
'warning_level=1',
'errorlogs=true', # Please print logs for tests that fail
],
meson_version : '>= 1.1',
license : 'LGPL-2.1-or-later',
)
cxx = meson.get_compiler('cpp')
subdir('nix-meson-build-support/deps-lists')
deps_private_maybe_subproject = [
dependency('nix-util'),
dependency('nix-store'),
dependency('nix-fetchers'),
]
deps_public_maybe_subproject = [
dependency('nix-util-c'),
dependency('nix-store-c'),
]
subdir('nix-meson-build-support/subprojects')
add_project_arguments(
language : 'cpp',
)
subdir('nix-meson-build-support/common')
sources = files(
'nix_api_fetchers.cc',
)
include_dirs = [include_directories('.')]
headers = files(
'nix_api_fetchers.h',
'nix_api_fetchers_internal.hh',
)
# TODO move this header to libexpr, maybe don't use it in tests?
headers += files('nix_api_fetchers.h')
subdir('nix-meson-build-support/export-all-symbols')
subdir('nix-meson-build-support/windows-version')
this_library = library(
'nixfetchersc',
sources,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
link_args: linker_export_flags,
prelink : true, # For C++ static initializers
install : true,
)
install_headers(headers, preserve_path : true)
libraries_private = []
subdir('nix-meson-build-support/export')
+1
View File
@@ -0,0 +1 @@
../../nix-meson-build-support
Loaded 100 of 389 files, more files were not shown because too many files have changed in this diff. Show more