Compare commits

..
1 Commits
Author SHA1 Message Date
Eelco Dolstra c98026e982 Lazy trees v2
This adds a setting 'lazy-trees' that causes flake inputs to be
"mounted" as virtual filesystems on top of /nix/store as random
"virtual" store paths. Only when the store path is actually used as a
dependency of a store derivation do we materialize ("devirtualize")
the input by copying it to its content-addressed location in the
store.

String contexts determine when devirtualization happens. One wrinkle
is that there are cases where we had store paths without proper
contexts, in particular when the user does `toString <path>` (where
<path> is a source tree in the Nix store) and passes the result to a
derivation. This usage was always broken, since it can result in
derivations that lack correct references. But to ensure that we don't
change evaluation results, we introduce a new type of context that
results in devirtualization but not in store references. We also now
print a warning about this.
2025-05-19 16:45:53 +02:00
318 changed files with 4886 additions and 9610 deletions

No files matched your search

-3
View File
@@ -14,8 +14,6 @@ jobs:
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v31
with:
install_url: "https://releases.nixos.org/nix/nix-2.29.1/install"
- run: nix --experimental-features 'nix-command flakes' flake show --all-systems --json
tests:
@@ -38,7 +36,6 @@ jobs:
fetch-depth: 0
- uses: cachix/install-nix-action@v31
with:
install_url: "https://releases.nixos.org/nix/nix-2.29.1/install"
# The sandbox would otherwise be disabled by default on Darwin
extra_nix_config: |
sandbox = true
-3
View File
@@ -47,6 +47,3 @@ result-*
.DS_Store
flake-regressions
# direnv
.direnv/
+1 -1
View File
@@ -1 +1 @@
2.30.1
2.30.0
+2 -2
View File
@@ -246,11 +246,11 @@ nix3_manpages = [
'nix3-nar',
'nix3-path-info',
'nix3-print-dev-env',
'nix3-profile',
'nix3-profile-add',
'nix3-profile-diff-closures',
'nix3-profile-history',
'nix3-profile-install',
'nix3-profile-list',
'nix3-profile',
'nix3-profile-remove',
'nix3-profile-rollback',
'nix3-profile-upgrade',
@@ -0,0 +1,17 @@
---
synopsis: Non-flake inputs now contain a `sourceInfo` attribute
issues: 13164
prs: 13170
---
Flakes have always a `sourceInfo` attribute which describes the source of the flake.
The `sourceInfo.outPath` is often identical to the flake's `outPath`, however it can differ when the flake is located in a subdirectory of its source.
Non-flake inputs (i.e. inputs with `flake = false`) can also be located at some path _within_ a wider source.
This usually happens when defining a relative path input within the same source as the parent flake, e.g. `inputs.foo.url = ./some-file.nix`.
Such relative inputs will now inherit their parent's `sourceInfo`.
This also means it is now possible to use `?dir=subdir` on non-flake inputs.
This iterates on the work done in 2.26 to improve relative path support ([#10089](https://github.com/NixOS/nix/pull/10089)),
and resolves a regression introduced in 2.28 relating to nested relative path inputs ([#13164](https://github.com/NixOS/nix/issues/13164)).
+17
View File
@@ -0,0 +1,17 @@
---
synopsis: Revert incomplete closure mixed download and build feature
issues: [77, 12628]
prs: [13176]
---
Since Nix 1.3 (299141ecbd08bae17013226dbeae71e842b4fdd7 in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference).
This feature is now removed.
Worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute.
In practice, however, we doubt even the more building is very likely to happen.
Remote stores that are missing dependencies in arbitrary ways (e.g. corruption) don't seem to be very common.
On the contrary, when remote stores fail to implement the [closure property](@docroot@/store/store-object.md#closure-property), it is usually an *intentional* choice on the part of the remote store, because it wishes to serve as an "overlay" store over another store, such as `https://cache.nixos.org`.
If an "incomplete closure" is encountered in that situation, the right fix is not to do some sort of "franken-building" as this feature implemented, but instead to make sure both substituters are enabled in the settings.
(In the future, we should make it easier for remote stores to indicate this to clients, to catch settings that won't work in general before a missing dependency is actually encountered.)
-2
View File
@@ -57,7 +57,6 @@
- [Tuning Cores and Jobs](advanced-topics/cores-vs-jobs.md)
- [Verifying Build Reproducibility](advanced-topics/diff-hook.md)
- [Using the `post-build-hook`](advanced-topics/post-build-hook.md)
- [Evaluation profiler](advanced-topics/eval-profiler.md)
- [Command Reference](command-ref/index.md)
- [Common Options](command-ref/opt-common.md)
- [Common Environment Variables](command-ref/env-common.md)
@@ -137,7 +136,6 @@
- [Contributing](development/contributing.md)
- [Releases](release-notes/index.md)
{{#include ./SUMMARY-rl-next.md}}
- [Release 2.30 (2025-07-07)](release-notes/rl-2.30.md)
- [Release 2.29 (2025-05-14)](release-notes/rl-2.29.md)
- [Release 2.28 (2025-04-02)](release-notes/rl-2.28.md)
- [Release 2.27 (2025-03-03)](release-notes/rl-2.27.md)
@@ -1,33 +0,0 @@
# Using the `eval-profiler`
Nix evaluator supports [evaluation](@docroot@/language/evaluation.md)
[profiling](<https://en.wikipedia.org/wiki/Profiling_(computer_programming)>)
compatible with `flamegraph.pl`. The profiler samples the nix
function call stack at regular intervals. It can be enabled with the
[`eval-profiler`](@docroot@/command-ref/conf-file.md#conf-eval-profiler)
setting:
```console
$ nix-instantiate "<nixpkgs>" -A hello --eval-profiler flamegraph
```
Stack sampling frequency and the output file path can be configured with
[`eval-profile-file`](@docroot@/command-ref/conf-file.md#conf-eval-profile-file)
and [`eval-profiler-frequency`](@docroot@/command-ref/conf-file.md#conf-eval-profiler-frequency).
By default the collected profile is saved to `nix.profile` file in the current working directory.
The collected profile can be directly consumed by `flamegraph.pl`:
```console
$ flamegraph.pl nix.profile > flamegraph.svg
```
The line information in the profile contains the location of the [call
site](https://en.wikipedia.org/wiki/Call_site) position and the name of the
function being called (when available). For example:
```
/nix/store/x9wnkly3k1gkq580m90jjn32q9f05q2v-source/pkgs/top-level/default.nix:167:5:primop import
```
Here `import` primop is called at `/nix/store/x9wnkly3k1gkq580m90jjn32q9f05q2v-source/pkgs/top-level/default.nix:167:5`.
@@ -53,11 +53,6 @@ This command has the following operations:
Download the Nix expressions of subscribed channels and create a new generation.
Update all channels if none is specified, and only those included in *names* otherwise.
> **Note**
>
> Downloaded channel contents are cached.
> Use `--tarball-ttl` or the [`tarball-ttl` configuration option](@docroot@/command-ref/conf-file.md#conf-tarball-ttl) to change the validity period of cached downloads.
- `--list-generations`
Prints a list of all the current existing generations for the
+5 -15
View File
@@ -31,22 +31,9 @@
The industry term for storage and retrieval systems using [content addressing](#gloss-content-address). A Nix store also has [input addressing](#gloss-input-addressed-store-object), and metadata.
- [derivation]{#gloss-derivation}
A derivation can be thought of as a [pure function](https://en.wikipedia.org/wiki/Pure_function) that produces new [store objects][store object] from existing store objects.
Derivations are implemented as [operating system processes that run in a sandbox](@docroot@/store/building.md#builder-execution).
This sandbox by default only allows reading from store objects specified as inputs, and only allows writing to designated [outputs][output] to be [captured as store objects](@docroot@/store/building.md#processing-outputs).
A derivation is typically specified as a [derivation expression] in the [Nix language], and [instantiated][instantiate] to a [store derivation].
There are multiple ways of obtaining store objects from store derivatons, collectively called [realisation][realise].
[derivation]: #gloss-derivation
- [store derivation]{#gloss-store-derivation}
A [derivation] represented as a [store object].
A single build task.
See [Store Derivation](@docroot@/store/derivation/index.md#store-derivation) for details.
[store derivation]: #gloss-store-derivation
@@ -70,7 +57,10 @@
- [derivation expression]{#gloss-derivation-expression}
A description of a [store derivation] using the [`derivation` primitive](./language/derivations.md) in the [Nix language].
A description of a [store derivation] in the Nix language.
The output(s) of a derivation are store objects.
Derivations are typically specified in Nix expressions using the [`derivation` primitive](./language/derivations.md).
These are translated into store layer *derivations* (implicitly by `nix-env` and `nix-build`, or explicitly by `nix-instantiate`).
[derivation expression]: #gloss-derivation-expression
@@ -25,7 +25,7 @@ This performs the default type of installation for your platform:
We recommend the multi-user installation if it supports your platform and you can authenticate with `sudo`.
The installer can be configured with various command line arguments and environment variables.
The installer can configured with various command line arguments and environment variables.
To show available command line flags:
```console
@@ -53,13 +53,23 @@ Derivations can declare some infrequently used optional attributes.
- [`__structuredAttrs`]{#adv-attr-structuredAttrs}\
If the special attribute `__structuredAttrs` is set to `true`, the other derivation
attributes are serialised into a file in JSON format.
attributes are serialised into a file in JSON format. The environment variable
`NIX_ATTRS_JSON_FILE` points to the exact location of that file both in a build
and a [`nix-shell`](../command-ref/nix-shell.md). This obviates the need for
[`passAsFile`](#adv-attr-passAsFile) since JSON files have no size restrictions,
unlike process environments.
This obviates the need for [`passAsFile`](#adv-attr-passAsFile) since JSON files have no size restrictions, unlike process environments.
It also makes it possible to tweak derivation settings in a structured way;
see [`outputChecks`](#adv-attr-outputChecks) for example.
It also makes it possible to tweak derivation settings in a structured way; see
[`outputChecks`](#adv-attr-outputChecks) for example.
See the [corresponding section in the derivation page](@docroot@/store/derivation/index.md#structured-attrs) for further details.
As a convenience to Bash builders,
Nix writes a script that initialises shell variables
corresponding to all attributes that are representable in Bash. The
environment variable `NIX_ATTRS_SH_FILE` points to the exact
location of the script, both in a build and a
[`nix-shell`](../command-ref/nix-shell.md). This includes non-nested
(associative) arrays. For example, the attribute `hardening.format = true`
ends up as the Bash associative array element `${hardening[format]}`.
> **Warning**
>
+1 -1
View File
@@ -1,6 +1,6 @@
# Nix Language
The Nix language is designed for conveniently creating and composing [derivations](@docroot@/glossary.md#gloss-derivation) – precise descriptions of how contents of existing files are used to derive new files.
The Nix language is designed for conveniently creating and composing *derivations* – precise descriptions of how contents of existing files are used to derive new files.
> **Tip**
>
+1 -1
View File
@@ -196,7 +196,7 @@ All comparison operators are implemented in terms of `<`, and the following equi
## Logical implication
Equivalent to `!`*b1* `||` *b2* (or `if` *b1* `then` *b2* `else true`)
Equivalent to `!`*b1* `||` *b2*.
[Logical implication]: #logical-implication
+2 -2
View File
@@ -225,8 +225,8 @@ passed in first , e.g.,
```nix
let add = { __functor = self: x: x + self.x; };
inc = add // { x = 1; }; # inc is { x = 1; __functor = (...) }
in inc 1 # equivalent of `add.__functor add 1` i.e. `1 + self.x`
inc = add // { x = 1; };
in inc 1
```
evaluates to `2`. This can be used to attach metadata to a function
@@ -91,7 +91,3 @@ is a JSON object with the following fields:
* `env`:
The environment passed to the `builder`.
* `structuredAttrs`:
[Strucutured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
Structured attributes are JSON, and thus embedded as-is.
+1 -1
View File
@@ -269,7 +269,7 @@
e.g. `--warn-large-path-threshold 100M`.
## Contributors
# Contributors
This release was made possible by the following 43 contributors:
+1 -1
View File
@@ -77,7 +77,7 @@
`<nix/fetchurl.nix>` is also known as the builtin derivation builder `builtin:fetchurl`. It's not to be confused with the evaluation-time function `builtins.fetchurl`, which was not affected by this issue.
## Contributors
# Contributors
This release was made possible by the following 58 contributors:
+1 -1
View File
@@ -76,7 +76,7 @@
- Evaluation caching now works for dirty Git workdirs [#11992](https://github.com/NixOS/nix/pull/11992)
## Contributors
# Contributors
This release was made possible by the following 45 contributors:
+1 -1
View File
@@ -47,7 +47,7 @@
blake3-34P4p+iZXcbbyB1i4uoF7eWCGcZHjmaRn6Y7QdynLwU=
```
## Contributors
# Contributors
This release was made possible by the following 21 contributors:
+1 -1
View File
@@ -82,7 +82,7 @@ This completes the infrastructure overhaul for the [RFC 132](https://github.com/
Although this change is not as critical, we figured it would be good to do this API change at the same time, also.
Also note that we try to keep the C API compatible, but we decided to break this function because it was young and likely not in widespread use yet. This frees up time to make important progress on the rest of the C API.
## Contributors
# Contributors
This earlier-than-usual release was made possible by the following 16 contributors:
+1 -1
View File
@@ -111,7 +111,7 @@ This fact is counterbalanced by the fact that most of those changes are bug fixe
This in particular prevents parts of GCC 14's diagnostics from being improperly filtered away.
## Contributors
# Contributors
This release was made possible by the following 40 contributors:
-153
View File
@@ -1,153 +0,0 @@
# Release 2.30.0 (2025-07-07)
## Backward-incompatible changes and deprecations
- [`build-dir`] no longer defaults to `$TMPDIR`
The directory in which temporary build directories are created no longer defaults
to `TMPDIR` or `/tmp`, to avoid builders making their directories
world-accessible. This behavior allowed escaping the build sandbox and can
cause build impurities even when not used maliciously. We now default to `builds`
in `NIX_STATE_DIR` (which is `/nix/var/nix/builds` in the default configuration).
- Deprecate manually making structured attrs using the `__json` attribute [#13220](https://github.com/NixOS/nix/pull/13220)
The proper way to create a derivation using [structured attrs] in the Nix language is by using `__structuredAttrs = true` with [`builtins.derivation`].
However, by exploiting how structured attrs are implementated, it has also been possible to create them by setting the `__json` environment variable to a serialized JSON string.
This sneaky alternative method is now deprecated, and may be disallowed in future versions of Nix.
[structured attrs]: @docroot@/language/advanced-attributes.md#adv-attr-structuredAttrs
[`builtins.derivation`]: @docroot@/language/builtins.html#builtins-derivation
- Rename `nix profile install` to [`nix profile add`] [#13224](https://github.com/NixOS/nix/pull/13224)
The command `nix profile install` has been renamed to [`nix profile add`] (though the former is still available as an alias). This is because the verb "add" is a better antonym for the verb "remove" (i.e. `nix profile remove`). Nix also does not have install hooks or general behavior often associated with "installing".
## Performance improvements
This release has a number performance improvements, in particular:
- Reduce the size of value from 24 to 16 bytes [#13407](https://github.com/NixOS/nix/pull/13407)
This shaves off a very significant amount of memory used for evaluation (~20% percent reduction in maximum heap size and ~17% in total bytes).
## Features
- Add [stack sampling evaluation profiler] [#13220](https://github.com/NixOS/nix/pull/13220)
The Nix evaluator now supports [stack sampling evaluation profiling](@docroot@/advanced-topics/eval-profiler.md) via the [`--eval-profiler flamegraph`] setting.
It outputs collapsed call stack information to the file specified by
[`--eval-profile-file`] (`nix.profile` by default) in a format directly consumable
by `flamegraph.pl` and compatible tools like [speedscope](https://speedscope.app/).
Sampling frequency can be configured via [`--eval-profiler-frequency`] (99 Hz by default).
Unlike the existing [`--trace-function-calls`], this profiler includes the name of the function
being called when it's available.
- [`nix repl`] prints which variables were loaded [#11406](https://github.com/NixOS/nix/pull/11406)
Instead of `Added <n> variables` it now prints the first 10 variables that were added to the global scope.
- `nix flake archive`: Add [`--no-check-sigs`] option [#13277](https://github.com/NixOS/nix/pull/13277)
This is useful when using [`nix flake archive`] with the destination set to a remote store.
- Emit warnings for IFDs with [`trace-import-from-derivation`] option [#13279](https://github.com/NixOS/nix/pull/13279)
While we have the setting [`allow-import-from-derivation`] to deny import-from-derivation (IFD), sometimes users would like to observe IFDs during CI processes to gradually phase out the idiom. The new setting `trace-import-from-derivation`, when set, logs a simple warning to the console.
- `json-log-path` setting [#13003](https://github.com/NixOS/nix/pull/13003)
New setting [`json-log-path`] that sends a copy of all Nix log messages (in JSON format) to a file or Unix domain socket.
- Non-flake inputs now contain a `sourceInfo` attribute [#13164](https://github.com/NixOS/nix/issues/13164) [#13170](https://github.com/NixOS/nix/pull/13170)
Flakes have always had a `sourceInfo` attribute which describes the source of the flake.
The `sourceInfo.outPath` is often identical to the flake's `outPath`. However, it can differ when the flake is located in a subdirectory of its source.
Non-flake inputs (i.e. inputs with [`flake = false`]) can also be located at some path _within_ a wider source.
This usually happens when defining a relative path input within the same source as the parent flake, e.g. `inputs.foo.url = ./some-file.nix`.
Such relative inputs will now inherit their parent's `sourceInfo`.
This also means it is now possible to use `?dir=subdir` on non-flake inputs.
This iterates on the work done in 2.26 to improve relative path support ([#10089](https://github.com/NixOS/nix/pull/10089)),
and resolves a regression introduced in 2.28 relating to nested relative path inputs ([#13164](https://github.com/NixOS/nix/issues/13164)).
## Miscellaneous changes
- [`builtins.sort`] uses PeekSort [#12623](https://github.com/NixOS/nix/pull/12623)
Previously it used libstdc++'s `std::stable_sort()`. However, that implementation is not reliable if the user-supplied comparison function is not a strict weak ordering.
- Revert incomplete closure mixed download and build feature [#77](https://github.com/NixOS/nix/issues/77) [#12628](https://github.com/NixOS/nix/issues/12628) [#13176](https://github.com/NixOS/nix/pull/13176)
Since Nix 1.3 ([commit `299141e`] in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference).
This feature is now removed.
In the worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute.
In practice, however, we doubt even more building is very likely to happen.
Remote stores that are missing dependencies in arbitrary ways (e.g. corruption) don't seem to be very common.
On the contrary, when remote stores fail to implement the [closure property](@docroot@/store/store-object.md#closure-property), it is usually an *intentional* choice on the part of the remote store, because it wishes to serve as an "overlay" store over another store, such as `https://cache.nixos.org`.
If an "incomplete closure" is encountered in that situation, the right fix is not to do some sort of "franken-building" as this feature implemented, but instead to make sure both substituters are enabled in the settings.
(In the future, we should make it easier for remote stores to indicate this to clients, to catch settings that won't work in general before a missing dependency is actually encountered.)
## Contributors
This release was made possible by the following 32 contributors:
- Cole Helbling [**(@cole-h)**](https://github.com/cole-h)
- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
- Egor Konovalov [**(@egorkonovalov)**](https://github.com/egorkonovalov)
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
- Graham Christensen [**(@grahamc)**](https://github.com/grahamc)
- gustavderdrache [**(@gustavderdrache)**](https://github.com/gustavderdrache)
- Gwenn Le Bihan [**(@gwennlbh)**](https://github.com/gwennlbh)
- h0nIg [**(@h0nIg)**](https://github.com/h0nIg)
- Jade Masker [**(@donottellmetonottellyou)**](https://github.com/donottellmetonottellyou)
- jayeshv [**(@jayeshv)**](https://github.com/jayeshv)
- Jeremy Fleischman [**(@jfly)**](https://github.com/jfly)
- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
- Jonas Chevalier [**(@zimbatm)**](https://github.com/zimbatm)
- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
- kstrafe [**(@kstrafe)**](https://github.com/kstrafe)
- Luc Perkins [**(@lucperkins)**](https://github.com/lucperkins)
- Matt Sturgeon [**(@MattSturgeon)**](https://github.com/MattSturgeon)
- Nikita Krasnov [**(@synalice)**](https://github.com/synalice)
- Peder Bergebakken Sundt [**(@pbsds)**](https://github.com/pbsds)
- pennae [**(@pennae)**](https://github.com/pennae)
- Philipp Otterbein
- Pol Dellaiera [**(@drupol)**](https://github.com/drupol)
- PopeRigby [**(@poperigby)**](https://github.com/poperigby)
- Raito Bezarius
- Robert Hensing [**(@roberth)**](https://github.com/roberth)
- Samuli Thomasson [**(@SimSaladin)**](https://github.com/SimSaladin)
- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
- Seth Flynn [**(@getchoo)**](https://github.com/getchoo)
- Stefan Boca [**(@stefanboca)**](https://github.com/stefanboca)
- tomberek [**(@tomberek)**](https://github.com/tomberek)
- Tristan Ross [**(@RossComputerGuy)**](https://github.com/RossComputerGuy)
- Valentin Gagarin [**(@fricklerhandwerk)**](https://github.com/fricklerhandwerk)
- Vladimír Čunát [**(@vcunat)**](https://github.com/vcunat)
- Wolfgang Walther [**(@wolfgangwalther)**](https://github.com/wolfgangwalther)
<!-- markdown links -->
[stack sampling evaluation profiler]: @docroot@/advanced-topics/eval-profiler.md
[`--eval-profiler`]: @docroot@/command-ref/conf-file.md#conf-eval-profiler
[`--eval-profiler flamegraph`]: @docroot@/command-ref/conf-file.md#conf-eval-profiler
[`--trace-function-calls`]: @docroot@/command-ref/conf-file.md#conf-trace-function-calls
[`--eval-profile-file`]: @docroot@/command-ref/conf-file.md#conf-eval-profile-file
[`--eval-profiler-frequency`]: @docroot@/command-ref/conf-file.md#conf-eval-profiler-frequency
[`build-dir`]: @docroot@/command-ref/conf-file.md#conf-build-dir
[`nix profile add`]: @docroot@/command-ref/new-cli/nix3-profile-add.md
[`nix repl`]: @docroot@/command-ref/new-cli/nix3-repl.md
[`nix flake archive`]: @docroot@/command-ref/new-cli/nix3-flake-archive.md
[`json-log-path`]: @docroot@/command-ref/conf-file.md#conf-json-log-path
[`trace-import-from-derivation`]: @docroot@/command-ref/conf-file.md#conf-trace-import-from-derivation
[`allow-import-from-derivation`]: @docroot@/command-ref/conf-file.md#conf-allow-import-from-derivation
[`builtins.sort`]: @docroot@/language/builtins.md#builtins-sort
[`flake = false`]: @docroot@/command-ref/new-cli/nix3-flake.md?highlight=false#flake-inputs
[`--no-check-sigs`]: @docroot@/command-ref/new-cli/nix3-flake-archive.md#opt-no-check-sigs
[commit `299141e`]: https://github.com/NixOS/nix/commit/299141ecbd08bae17013226dbeae71e842b4fdd7
+2 -13
View File
@@ -138,17 +138,6 @@ See [Wikipedia](https://en.wikipedia.org/wiki/Argv) for details.
Environment variables which will be passed to the [builder](#builder) executable.
#### Structured Attributes {#structured-attrs}
Nix also has special support for embedding JSON in the derivations.
The environment variable `NIX_ATTRS_JSON_FILE` points to the exact location of that file both in a build and a [`nix-shell`](@docroot@/command-ref/nix-shell.md).
As a convenience to Bash builders, Nix writes a script that initialises shell variables corresponding to all attributes that are representable in Bash.
The environment variable `NIX_ATTRS_SH_FILE` points to the exact location of the script, both in a build and a [`nix-shell`](@docroot@/command-ref/nix-shell.md).
This includes non-nested (associative) arrays.
For example, the attribute `hardening.format = true` ends up as the Bash associative array element `${hardening[format]}`.
### Placeholders
Placeholders are opaque values used within the [process creation fields] to [store objects] for which we don't yet know [store path]s.
@@ -173,7 +162,7 @@ There are two types of placeholder, corresponding to the two cases where this pr
> **Explanation**
>
> In general, we need to [realise] a [store object] in order to be sure to have a store object for it.
> In general, we need to realise [realise] a [store object] in order to be sure to have a store object for it.
> But for these two cases this is either impossible or impractical:
>
> - In the output case this is impossible:
@@ -200,7 +189,7 @@ This ensures that there is a canonical [store path] used to refer to the derivat
> **Note**
>
> Currently, the canonical encoding for every derivation is the "ATerm" format,
> but this is subject to change for the types of derivations which are not yet stable.
> but this is subject to change for types derivations which are not yet stable.
Regardless of the format used, when serializing a derivation to a store object, that store object will be content-addressed.
+2 -2
View File
@@ -18,14 +18,14 @@ In particular, the edge corresponding to a reference is from the store object th
References other than a self-reference must not form a cycle.
The graph of references excluding self-references thus forms a [directed acyclic graph].
[directed acyclic graph]: @docroot@/glossary.md#gloss-directed-acyclic-graph
[directed acyclic graph]: @docroot@/glossary.md#gloss-directed acyclic graph
We can take the [transitive closure] of the references graph, which any pair of store objects have an edge not if there is a single reference from the first to the second, but a path of one or more references from the first to the second.
The *requisites* of a store object are all store objects reachable by paths of references which start with given store object's references.
[transitive closure]: https://en.wikipedia.org/wiki/Transitive_closure
We can also take the [transpose graph] of the references graph, where we reverse the orientation of all edges.
We can also take the [transpose graph] ofthe references graph, where we reverse the orientation of all edges.
The *referrers* of a store object are the store objects that reference it.
[transpose graph]: https://en.wikipedia.org/wiki/Transpose_graph
+61 -99
View File
@@ -1,11 +1,6 @@
{
# Core dependencies
pkgs,
lib,
dockerTools,
runCommand,
buildPackages,
# Image configuration
pkgs ? import <nixpkgs> { },
lib ? pkgs.lib,
name ? "nix",
tag ? "latest",
bundleNixpkgs ? true,
@@ -19,60 +14,36 @@
gid ? 0,
uname ? "root",
gname ? "root",
Labels ? {
"org.opencontainers.image.title" = "Nix";
"org.opencontainers.image.source" = "https://github.com/NixOS/nix";
"org.opencontainers.image.vendor" = "Nix project";
"org.opencontainers.image.version" = nix.version;
"org.opencontainers.image.description" = "Nix container image";
},
Cmd ? [ (lib.getExe bashInteractive) ],
# Default Packages
nix,
bashInteractive,
coreutils-full,
gnutar,
gzip,
gnugrep,
which,
curl,
less,
wget,
man,
cacert,
findutils,
iana-etc,
gitMinimal,
openssh,
# Other dependencies
shadow,
}:
let
defaultPkgs = [
nix
bashInteractive
coreutils-full
gnutar
gzip
gnugrep
which
curl
less
wget
man
cacert.out
findutils
iana-etc
gitMinimal
openssh
] ++ extraPkgs;
defaultPkgs =
with pkgs;
[
nix
bashInteractive
coreutils-full
gnutar
gzip
gnugrep
which
curl
less
wget
man
cacert.out
findutils
iana-etc
git
openssh
]
++ extraPkgs;
users =
{
root = {
uid = 0;
shell = lib.getExe bashInteractive;
shell = "${pkgs.bashInteractive}/bin/bash";
home = "/root";
gid = 0;
groups = [ "root" ];
@@ -81,7 +52,7 @@ let
nobody = {
uid = 65534;
shell = lib.getExe' shadow "nologin";
shell = "${pkgs.shadow}/bin/nologin";
home = "/var/empty";
gid = 65534;
groups = [ "nobody" ];
@@ -92,7 +63,7 @@ let
// lib.optionalAttrs (uid != 0) {
"${uname}" = {
uid = uid;
shell = lib.getExe bashInteractive;
shell = "${pkgs.bashInteractive}/bin/bash";
home = "/home/${uname}";
gid = gid;
groups = [ "${gname}" ];
@@ -176,42 +147,41 @@ let
"${k}:x:${toString gid}:${lib.concatStringsSep "," members}";
groupContents = (lib.concatStringsSep "\n" (lib.attrValues (lib.mapAttrs groupToGroup groups)));
toConf =
with pkgs.lib.generators;
toKeyValue {
mkKeyValue = mkKeyValueDefault {
mkValueString = v: if lib.isList v then lib.concatStringsSep " " v else mkValueStringDefault { } v;
} " = ";
};
defaultNixConf = {
sandbox = "false";
build-users-group = "nixbld";
trusted-public-keys = [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" ];
};
nixConfContents = toConf (
{
sandbox = false;
build-users-group = "nixbld";
trusted-public-keys = [ "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" ];
}
// nixConf
);
nixConfContents =
(lib.concatStringsSep "\n" (
lib.mapAttrsFlatten (
n: v:
let
vStr = if builtins.isList v then lib.concatStringsSep " " v else v;
in
"${n} = ${vStr}"
) (defaultNixConf // nixConf)
))
+ "\n";
userHome = if uid == 0 then "/root" else "/home/${uname}";
baseSystem =
let
nixpkgs = pkgs.path;
channel = runCommand "channel-nixos" { inherit bundleNixpkgs; } ''
channel = pkgs.runCommand "channel-nixos" { inherit bundleNixpkgs; } ''
mkdir $out
if [ "$bundleNixpkgs" ]; then
ln -s ${
builtins.path {
path = nixpkgs;
name = "source";
}
} $out/nixpkgs
ln -s ${nixpkgs} $out/nixpkgs
echo "[]" > $out/manifest.nix
fi
'';
# doc/manual/source/command-ref/files/manifest.nix.md
manifest = buildPackages.runCommand "manifest.nix" { } ''
rootEnv = pkgs.buildPackages.buildEnv {
name = "root-profile-env";
paths = defaultPkgs;
};
manifest = pkgs.buildPackages.runCommand "manifest.nix" { } ''
cat > $out <<EOF
[
${lib.concatStringsSep "\n" (
@@ -240,15 +210,11 @@ let
]
EOF
'';
profile = buildPackages.buildEnv {
name = "root-profile-env";
paths = defaultPkgs;
postBuild = ''
mv $out/manifest $out/manifest.nix
'';
inherit manifest;
};
profile = pkgs.buildPackages.runCommand "user-environment" { } ''
mkdir $out
cp -a ${rootEnv}/* $out/
ln -s ${manifest} $out/manifest.nix
'';
flake-registry-path =
if (flake-registry == null) then
null
@@ -257,7 +223,7 @@ let
else
flake-registry;
in
runCommand "base-system"
pkgs.runCommand "base-system"
{
inherit
passwdContents
@@ -280,7 +246,6 @@ let
set -x
mkdir -p $out/etc
# may get replaced by pkgs.dockerTools.caCertificates
mkdir -p $out/etc/ssl/certs
ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt $out/etc/ssl/certs
@@ -308,23 +273,20 @@ let
mkdir -p $out${userHome}
mkdir -p $out/nix/var/nix/profiles/per-user/${uname}
# see doc/manual/source/command-ref/files/profiles.md
ln -s ${profile} $out/nix/var/nix/profiles/default-1-link
ln -s /nix/var/nix/profiles/default-1-link $out/nix/var/nix/profiles/default
ln -s /nix/var/nix/profiles/default $out${userHome}/.nix-profile
# see doc/manual/source/command-ref/files/channels.md
ln -s ${channel} $out/nix/var/nix/profiles/per-user/${uname}/channels-1-link
ln -s /nix/var/nix/profiles/per-user/${uname}/channels-1-link $out/nix/var/nix/profiles/per-user/${uname}/channels
# see doc/manual/source/command-ref/files/default-nix-expression.md
mkdir -p $out${userHome}/.nix-defexpr
ln -s /nix/var/nix/profiles/per-user/${uname}/channels $out${userHome}/.nix-defexpr/channels
echo "${channelURL} ${channelName}" > $out${userHome}/.nix-channels
# may get replaced by pkgs.dockerTools.binSh & pkgs.dockerTools.usrBinEnv
mkdir -p $out/bin $out/usr/bin
ln -s ${lib.getExe' coreutils-full "env"} $out/usr/bin/env
ln -s ${lib.getExe bashInteractive} $out/bin/sh
ln -s ${pkgs.coreutils}/bin/env $out/usr/bin/env
ln -s ${pkgs.bashInteractive}/bin/bash $out/bin/sh
''
+ (lib.optionalString (flake-registry-path != null) ''
@@ -333,13 +295,13 @@ let
globalFlakeRegistryPath="$nixCacheDir/flake-registry.json"
ln -s ${flake-registry-path} $out$globalFlakeRegistryPath
mkdir -p $out/nix/var/nix/gcroots/auto
rootName=$(${lib.getExe' nix "nix"} --extra-experimental-features nix-command hash file --type sha1 --base32 <(echo -n $globalFlakeRegistryPath))
rootName=$(${pkgs.nix}/bin/nix --extra-experimental-features nix-command hash file --type sha1 --base32 <(echo -n $globalFlakeRegistryPath))
ln -s $globalFlakeRegistryPath $out/nix/var/nix/gcroots/auto/$rootName
'')
);
in
dockerTools.buildLayeredImageWithNixDb {
pkgs.dockerTools.buildLayeredImageWithNixDb {
inherit
name
@@ -365,7 +327,7 @@ dockerTools.buildLayeredImageWithNixDb {
'';
config = {
inherit Cmd Labels;
Cmd = [ "${userHome}/.nix-profile/bin/bash" ];
User = "${toString uid}:${toString gid}";
Env = [
"USER=${uname}"
+38 -20
View File
@@ -32,7 +32,7 @@
let
inherit (nixpkgs) lib;
officialRelease = true;
officialRelease = false;
linux32BitSystems = [ "i686-linux" ];
linux64BitSystems = [
@@ -179,6 +179,19 @@
};
nix = final.nixComponents2.nix-cli;
# See https://github.com/NixOS/nixpkgs/pull/214409
# Remove when fixed in this flake's nixpkgs
pre-commit =
if prev.stdenv.hostPlatform.system == "i686-linux" then
(prev.pre-commit.override (o: {
dotnet-sdk = "";
})).overridePythonAttrs
(o: {
doCheck = false;
})
else
prev.pre-commit;
};
in
@@ -220,24 +233,28 @@
This shouldn't build anything significant; just check that things
(including derivations) are _set up_ correctly.
*/
packaging-overriding =
let
pkgs = nixpkgsFor.${system}.native;
nix = self.packages.${system}.nix;
in
assert (nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src.patches == [ pkgs.emptyFile ];
if pkgs.stdenv.buildPlatform.isDarwin then
lib.warn "packaging-overriding check currently disabled because of a permissions issue on macOS" pkgs.emptyFile
else
# If this fails, something might be wrong with how we've wired the scope,
# or something could be broken in Nixpkgs.
pkgs.testers.testEqualContents {
assertion = "trivial patch does not change source contents";
expected = "${./.}";
actual =
# Same for all components; nix-util is an arbitrary pick
(nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src;
};
# Disabled due to a bug in `testEqualContents` (see
# https://github.com/NixOS/nix/issues/12690).
/*
packaging-overriding =
let
pkgs = nixpkgsFor.${system}.native;
nix = self.packages.${system}.nix;
in
assert (nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src.patches == [ pkgs.emptyFile ];
if pkgs.stdenv.buildPlatform.isDarwin then
lib.warn "packaging-overriding check currently disabled because of a permissions issue on macOS" pkgs.emptyFile
else
# If this fails, something might be wrong with how we've wired the scope,
# or something could be broken in Nixpkgs.
pkgs.testers.testEqualContents {
assertion = "trivial patch does not change source contents";
expected = "${./.}";
actual =
# Same for all components; nix-util is an arbitrary pick
(nix.appendPatches [ pkgs.emptyFile ]).libs.nix-util.src;
};
*/
}
// (lib.optionalAttrs (builtins.elem system linux64BitSystems)) {
dockerImage = self.hydraJobs.dockerImage.${system};
@@ -404,7 +421,8 @@
dockerImage =
let
pkgs = nixpkgsFor.${system}.native;
image = pkgs.callPackage ./docker.nix {
image = import ./docker.nix {
inherit pkgs;
tag = pkgs.nix.version;
};
in
@@ -166,24 +166,5 @@
"the-tumultuous-unicorn-of-darkness@gmx.com": "TheTumultuousUnicornOfDarkness",
"dev@rodney.id.au": "rvl",
"pe@pijul.org": "P-E-Meunier",
"yannik@floxdev.com": "ysndr",
"73017521+egorkonovalov@users.noreply.github.com": "egorkonovalov",
"raito@lix.systems": null,
"nikita.nikita.krasnov@gmail.com": "synalice",
"lucperkins@gmail.com": "lucperkins",
"vladimir.cunat@nic.cz": "vcunat",
"walther@technowledgy.de": "wolfgangwalther",
"jayesh.mail@gmail.com": "jayeshv",
"samuli.thomasson@pm.me": "SimSaladin",
"kevin@stravers.net": "kstrafe",
"poperigby@mailbox.org": "poperigby",
"cole.helbling@determinate.systems": "cole-h",
"donottellmetonottellyou@gmail.com": "donottellmetonottellyou",
"getchoo@tuta.io": "getchoo",
"alex.ford@determinate.systems": "gustavderdrache",
"stefan.r.boca@gmail.com": "stefanboca",
"gwenn.lebihan7@gmail.com": "gwennlbh",
"hey@ewen.works": "gwennlbh",
"matt@sturgeon.me.uk": "MattSturgeon",
"pbsds@hotmail.com": "pbsds"
"yannik@floxdev.com": "ysndr"
}
@@ -146,21 +146,5 @@
"ajlekcahdp4": "Alexander Romanov",
"Valodim": "Vincent Breitmoser",
"rvl": "Rodney Lorrimar",
"whatsthecraic": "Dean De Leo",
"gwennlbh": "Gwenn Le Bihan",
"donottellmetonottellyou": "Jade Masker",
"kstrafe": null,
"synalice": "Nikita Krasnov",
"poperigby": "PopeRigby",
"MattSturgeon": "Matt Sturgeon",
"lucperkins": "Luc Perkins",
"gustavderdrache": null,
"SimSaladin": "Samuli Thomasson",
"getchoo": "Seth Flynn",
"stefanboca": "Stefan Boca",
"wolfgangwalther": "Wolfgang Walther",
"pbsds": "Peder Bergebakken Sundt",
"egorkonovalov": "Egor Konovalov",
"jayeshv": "jayeshv",
"vcunat": "Vladim\u00edr \u010cun\u00e1t"
"whatsthecraic": "Dean De Leo"
}
+3 -115
View File
@@ -37,118 +37,6 @@
fi
''}";
};
meson-format = {
enable = true;
files = "(meson.build|meson.options)$";
entry = "${pkgs.writeScript "format-meson" ''
#!${pkgs.runtimeShell}
for file in "$@"; do
${lib.getExe pkgs.meson} format -ic ${../meson.format} "$file"
done
''}";
excludes = [
# We haven't applied formatting to these files yet
''^doc/manual/meson.build$''
''^doc/manual/source/command-ref/meson.build$''
''^doc/manual/source/development/meson.build$''
''^doc/manual/source/language/meson.build$''
''^doc/manual/source/meson.build$''
''^doc/manual/source/release-notes/meson.build$''
''^doc/manual/source/store/meson.build$''
''^misc/bash/meson.build$''
''^misc/fish/meson.build$''
''^misc/launchd/meson.build$''
''^misc/meson.build$''
''^misc/systemd/meson.build$''
''^misc/zsh/meson.build$''
''^nix-meson-build-support/$''
''^nix-meson-build-support/big-objs/meson.build$''
''^nix-meson-build-support/common/meson.build$''
''^nix-meson-build-support/deps-lists/meson.build$''
''^nix-meson-build-support/export/meson.build$''
''^nix-meson-build-support/export-all-symbols/meson.build$''
''^nix-meson-build-support/generate-header/meson.build$''
''^nix-meson-build-support/libatomic/meson.build$''
''^nix-meson-build-support/subprojects/meson.build$''
''^scripts/meson.build$''
''^src/external-api-docs/meson.build$''
''^src/internal-api-docs/meson.build$''
''^src/libcmd/include/nix/cmd/meson.build$''
''^src/libcmd/meson.build$''
''^src/libcmd/nix-meson-build-support$''
''^src/libexpr/include/nix/expr/meson.build$''
''^src/libexpr/meson.build$''
''^src/libexpr/nix-meson-build-support$''
''^src/libexpr-c/meson.build$''
''^src/libexpr-c/nix-meson-build-support$''
''^src/libexpr-test-support/meson.build$''
''^src/libexpr-test-support/nix-meson-build-support$''
''^src/libexpr-tests/meson.build$''
''^src/libexpr-tests/nix-meson-build-support$''
''^src/libfetchers/include/nix/fetchers/meson.build$''
''^src/libfetchers/meson.build$''
''^src/libfetchers/nix-meson-build-support$''
''^src/libfetchers-c/meson.build$''
''^src/libfetchers-c/nix-meson-build-support$''
''^src/libfetchers-tests/meson.build$''
''^src/libfetchers-tests/nix-meson-build-support$''
''^src/libflake/include/nix/flake/meson.build$''
''^src/libflake/meson.build$''
''^src/libflake/nix-meson-build-support$''
''^src/libflake-c/meson.build$''
''^src/libflake-c/nix-meson-build-support$''
''^src/libflake-tests/meson.build$''
''^src/libflake-tests/nix-meson-build-support$''
''^src/libmain/include/nix/main/meson.build$''
''^src/libmain/meson.build$''
''^src/libmain/nix-meson-build-support$''
''^src/libmain-c/meson.build$''
''^src/libmain-c/nix-meson-build-support$''
''^src/libstore/include/nix/store/meson.build$''
''^src/libstore/meson.build$''
''^src/libstore/nix-meson-build-support$''
''^src/libstore/unix/include/nix/store/meson.build$''
''^src/libstore/unix/meson.build$''
''^src/libstore/windows/meson.build$''
''^src/libstore-c/meson.build$''
''^src/libstore-c/nix-meson-build-support$''
''^src/libstore-test-support/include/nix/store/tests/meson.build$''
''^src/libstore-test-support/meson.build$''
''^src/libstore-test-support/nix-meson-build-support$''
''^src/libstore-tests/meson.build$''
''^src/libstore-tests/nix-meson-build-support$''
''^src/libutil/meson.build$''
''^src/libutil/nix-meson-build-support$''
''^src/libutil/unix/include/nix/util/meson.build$''
''^src/libutil/unix/meson.build$''
''^src/libutil/windows/meson.build$''
''^src/libutil-c/meson.build$''
''^src/libutil-c/nix-meson-build-support$''
''^src/libutil-test-support/include/nix/util/tests/meson.build$''
''^src/libutil-test-support/meson.build$''
''^src/libutil-test-support/nix-meson-build-support$''
''^src/libutil-tests/meson.build$''
''^src/libutil-tests/nix-meson-build-support$''
''^src/nix/meson.build$''
''^src/nix/nix-meson-build-support$''
''^src/perl/lib/Nix/meson.build$''
''^src/perl/meson.build$''
''^tests/functional/ca/meson.build$''
''^tests/functional/common/meson.build$''
''^tests/functional/dyn-drv/meson.build$''
''^tests/functional/flakes/meson.build$''
''^tests/functional/git-hashing/meson.build$''
''^tests/functional/local-overlay-store/meson.build$''
''^tests/functional/meson.build$''
''^src/libcmd/meson.options$''
''^src/libexpr/meson.options$''
''^src/libstore/meson.options$''
''^src/libutil/meson.options$''
''^src/libutil-c/meson.options$''
''^src/nix/meson.options$''
''^src/perl/meson.options$''
];
};
nixfmt-rfc-style = {
enable = true;
excludes = [
@@ -193,6 +81,7 @@
# We haven't applied formatting to these files yet
''^doc/manual/redirects\.js$''
''^doc/manual/theme/highlight\.js$''
''^precompiled-headers\.h$''
''^src/build-remote/build-remote\.cc$''
''^src/libcmd/built-path\.cc$''
''^src/libcmd/include/nix/cmd/built-path\.hh$''
@@ -256,6 +145,7 @@
''^src/libexpr/include/nix/expr/value-to-json\.hh$''
''^src/libexpr/value-to-xml\.cc$''
''^src/libexpr/include/nix/expr/value-to-xml\.hh$''
''^src/libexpr/include/nix/expr/value\.hh$''
''^src/libexpr/value/context\.cc$''
''^src/libexpr/include/nix/expr/value/context\.hh$''
''^src/libfetchers/attrs\.cc$''
@@ -386,8 +276,6 @@
''^src/libstore/store-api\.cc$''
''^src/libstore/include/nix/store/store-api\.hh$''
''^src/libstore/include/nix/store/store-dir-config\.hh$''
''^src/libstore/build/derivation-building-goal\.cc$''
''^src/libstore/include/nix/store/build/derivation-building-goal\.hh$''
''^src/libstore/build/derivation-goal\.cc$''
''^src/libstore/include/nix/store/build/derivation-goal\.hh$''
''^src/libstore/build/drv-output-substitution-goal\.cc$''
@@ -469,7 +357,7 @@
''^src/libutil/json-utils\.cc$''
''^src/libutil/include/nix/util/json-utils\.hh$''
''^src/libutil/linux/cgroup\.cc$''
''^src/libutil/linux/linux-namespaces\.cc$''
''^src/libutil/linux/namespaces\.cc$''
''^src/libutil/logging\.cc$''
''^src/libutil/include/nix/util/logging\.hh$''
''^src/libutil/memory-source-accessor\.cc$''
+1 -1
View File
@@ -157,7 +157,7 @@ section_title="Release $version_full ($DATE)"
if ! $IS_PATCH; then
echo
echo "## Contributors"
echo "# Contributors"
echo
VERSION=$version_full ./maintainers/release-credits
fi
+2 -5
View File
@@ -1,16 +1,13 @@
# This is just a stub project to include all the others as subprojects
# for development shell purposes
project(
'nix-dev-shell',
'cpp',
project('nix-dev-shell', 'cpp',
version : files('.version'),
subproject_dir : 'src',
default_options : [
'localstatedir=/nix/var',
# hack for trailing newline
],
meson_version : '>= 1.1',
meson_version : '>= 1.1'
)
# Internal Libraries
-7
View File
@@ -1,7 +0,0 @@
indent_by = ' '
space_array = true
kwargs_force_multiline = false
wide_colon = true
group_arg_value = true
indent_before_comments = ' '
use_editor_config = true
+3 -12
View File
@@ -1,22 +1,13 @@
# vim: filetype=meson
option(
'doc-gen',
type : 'boolean',
value : false,
option('doc-gen', type : 'boolean', value : false,
description : 'Generate documentation',
)
option(
'unit-tests',
type : 'boolean',
value : true,
option('unit-tests', type : 'boolean', value : true,
description : 'Build unit tests',
)
option(
'bindings',
type : 'boolean',
value : true,
option('bindings', type : 'boolean', value : true,
description : 'Build language bindings (e.g. Perl)',
)
+1 -2
View File
@@ -1,2 +1 @@
d @localstatedir@/nix/daemon-socket 0755 root root - -
d @localstatedir@/nix/builds 0755 root root 7d -
d @localstatedir@/nix/daemon-socket 0755 root root - -
+1 -7
View File
@@ -11,18 +11,12 @@ endforeach
requires_public += deps_public
extra_pkg_config_variables = get_variable('extra_pkg_config_variables', {})
extra_cflags = []
if not meson.project_name().endswith('-c')
extra_cflags += ['-std=c++2a']
endif
import('pkgconfig').generate(
this_library,
filebase : meson.project_name(),
name : 'Nix',
description : 'Nix Package Manager',
extra_cflags : extra_cflags,
extra_cflags : ['-std=c++2a'],
requires : requires_public,
requires_private : requires_private,
libraries_private : libraries_private,
@@ -2,5 +2,5 @@ if host_machine.system() == 'windows'
# https://learn.microsoft.com/en-us/cpp/porting/modifying-winver-and-win32-winnt?view=msvc-170
# #define _WIN32_WINNT_WIN8 0x0602
# We currently don't use any API which requires higher than this.
add_project_arguments([ '-D_WIN32_WINNT=0x0602' ], language : 'cpp')
add_project_arguments([ '-D_WIN32_WINNT=0x0602' ], language: 'cpp')
endif
-14
View File
@@ -166,17 +166,6 @@ let
outputs = prevAttrs.outputs or [ "out" ] ++ [ "dev" ];
};
fixupStaticLayer = finalAttrs: prevAttrs: {
postFixup =
prevAttrs.postFixup or ""
+ lib.optionalString (stdenv.hostPlatform.isStatic) ''
# HACK: Otherwise the result will have the entire buildInputs closure
# injected by the pkgsStatic stdenv
# <https://github.com/NixOS/nixpkgs/issues/83667>
rm -f $out/nix-support/propagated-build-inputs
'';
};
# Work around weird `--as-needed` linker behavior with BSD, see
# https://github.com/mesonbuild/meson/issues/3593
bsdNoLinkAsNeeded =
@@ -312,7 +301,6 @@ in
scope.sourceLayer
setVersionLayer
mesonLayer
fixupStaticLayer
scope.mesonComponentOverrides
];
mkMesonExecutable = mkPackageBuilder [
@@ -322,7 +310,6 @@ in
setVersionLayer
mesonLayer
mesonBuildLayer
fixupStaticLayer
scope.mesonComponentOverrides
];
mkMesonLibrary = mkPackageBuilder [
@@ -333,7 +320,6 @@ in
setVersionLayer
mesonBuildLayer
mesonLibraryLayer
fixupStaticLayer
scope.mesonComponentOverrides
];
+63
View File
@@ -0,0 +1,63 @@
#include <algorithm>
#include <array>
#include <atomic>
#include <cassert>
#include <cctype>
#include <chrono>
#include <climits>
#include <cmath>
#include <condition_variable>
#include <cstddef>
#include <cstdint>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <exception>
#include <functional>
#include <future>
#include <iostream>
#include <limits>
#include <list>
#include <locale>
#include <map>
#include <memory>
#include <mutex>
#include <numeric>
#include <optional>
#include <queue>
#include <random>
#include <regex>
#include <set>
#include <sstream>
#include <stack>
#include <stdexcept>
#include <string>
#include <thread>
#include <unordered_map>
#include <unordered_set>
#include <vector>
#include <boost/format.hpp>
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
#include <signal.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>
#ifndef _WIN32
# include <grp.h>
# include <netdb.h>
# include <pwd.h>
# include <sys/resource.h>
# include <sys/select.h>
# include <sys/socket.h>
# include <sys/utsname.h>
# include <sys/wait.h>
# include <termios.h>
#endif
#include <nlohmann/json.hpp>
+2 -7
View File
@@ -834,13 +834,8 @@ install_from_extracted_nix() {
(
cd "$EXTRACTED_NIX_PATH"
if is_os_darwin; then
_sudo "to copy the basic Nix files to the new store at $NIX_ROOT/store" \
cp -RPp ./store/* "$NIX_ROOT/store/"
else
_sudo "to copy the basic Nix files to the new store at $NIX_ROOT/store" \
cp -RP --preserve=ownership,timestamps ./store/* "$NIX_ROOT/store/"
fi
_sudo "to copy the basic Nix files to the new store at $NIX_ROOT/store" \
cp -RPp ./store/* "$NIX_ROOT/store/"
_sudo "to make the new store non-writable at $NIX_ROOT/store" \
chmod -R ugo-w "$NIX_ROOT/store/"
+1 -5
View File
@@ -167,11 +167,7 @@ for i in $(cd "$self/store" >/dev/null && echo ./*); do
rm -rf "$i_tmp"
fi
if ! [ -e "$dest/store/$i" ]; then
if [ "$(uname -s)" = "Darwin" ]; then
cp -RPp "$self/store/$i" "$i_tmp"
else
cp -RP --preserve=ownership,timestamps "$self/store/$i" "$i_tmp"
fi
cp -RPp "$self/store/$i" "$i_tmp"
chmod -R a-w "$i_tmp"
chmod +w "$i_tmp"
mv "$i_tmp" "$dest/store/$i"
+5 -30
View File
@@ -1,16 +1,17 @@
# Only execute this file once per shell.
if test -z "$HOME" || test -n "$__ETC_PROFILE_NIX_SOURCED"
if test -z "$HOME" || \
test -n "$__ETC_PROFILE_NIX_SOURCED"
exit
end
set --global --export __ETC_PROFILE_NIX_SOURCED 1
set --global __ETC_PROFILE_NIX_SOURCED 1
# Local helpers
function add_path --argument-names new_path
if type -q fish_add_path
# fish 3.2.0 or newer
fish_add_path --prepend --global $new_path
fish_add_path --prepend --global $new_path
else
# older versions of fish
if not contains $new_path $fish_user_paths
@@ -23,33 +24,7 @@ end
# Set up the per-user profile.
set --local NIX_LINK "$HOME/.nix-profile"
set --local NIX_LINK_NEW
if test -n "$XDG_STATE_HOME"
set NIX_LINK_NEW "$XDG_STATE_HOME/nix/profile"
else
set NIX_LINK_NEW "$HOME/.local/state/nix/profile"
end
if test -e "$NIX_LINK_NEW"
if test -t 2; and test -e "$NIX_LINK"
set --local warning "\033[1;35mwarning:\033[0m "
printf "$warning Both %s and legacy %s exist; using the former.\n" "$NIX_LINK_NEW" "$NIX_LINK" 1>&2
if test (realpath "$NIX_LINK") = (realpath "$NIX_LINK_NEW")
printf " Since the profiles match, you can safely delete either of them.\n" 1>&2
else
# This should be an exceptionally rare occasion: the only way to get it would be to
# 1. Update to newer Nix;
# 2. Remove .nix-profile;
# 3. Set the $NIX_LINK_NEW to something other than the default user profile;
# 4. Roll back to older Nix.
# If someone did all that, they can probably figure out how to migrate the profile.
printf "$warning Profiles do not match. You should manually migrate from %s to %s.\n" "$NIX_LINK" "$NIX_LINK_NEW" 1>&2
end
end
set NIX_LINK "$NIX_LINK_NEW"
end
set --local NIX_LINK $HOME/.nix-profile
# Set up environment.
# This part should be kept in sync with nixpkgs:nixos/modules/programs/environment.nix
+5 -35
View File
@@ -1,16 +1,17 @@
# Only execute this file once per shell.
if test -z "$HOME" || test -n "$__ETC_PROFILE_NIX_SOURCED"
if test -z "$HOME" || \
test -n "$__ETC_PROFILE_NIX_SOURCED"
exit
end
set --global --export __ETC_PROFILE_NIX_SOURCED 1
set --global __ETC_PROFILE_NIX_SOURCED 1
# Local helpers
function add_path --argument-names new_path
if type -q fish_add_path
# fish 3.2.0 or newer
fish_add_path --prepend --global $new_path
fish_add_path --prepend --global $new_path
else
# older versions of fish
if not contains $new_path $fish_user_paths
@@ -23,38 +24,7 @@ end
# Set up the per-user profile.
set --local NIX_LINK
if test -n "$NIX_STATE_HOME"
set NIX_LINK "$NIX_STATE_HOME/.nix-profile"
else
set NIX_LINK "$HOME/.nix-profile"
set --local NIX_LINK_NEW
if test -n "$XDG_STATE_HOME"
set NIX_LINK_NEW "$XDG_STATE_HOME/nix/profile"
else
set NIX_LINK_NEW "$HOME/.local/state/nix/profile"
end
if test -e "$NIX_LINK_NEW"
if test -t 2; and test -e "$NIX_LINK"
set --local warning "\033[1;35mwarning:\033[0m "
printf "$warning Both %s and legacy %s exist; using the former.\n" "$NIX_LINK_NEW" "$NIX_LINK" 1>&2
if test (realpath "$NIX_LINK") = (realpath "$NIX_LINK_NEW")
printf " Since the profiles match, you can safely delete either of them.\n" 1>&2
else
# This should be an exceptionally rare occasion: the only way to get it would be to
# 1. Update to newer Nix;
# 2. Remove .nix-profile;
# 3. Set the $NIX_LINK_NEW to something other than the default user profile;
# 4. Roll back to older Nix.
# If someone did all that, they can probably figure out how to migrate the profile.
printf "$warning Profiles do not match. You should manually migrate from %s to %s.\n" "$NIX_LINK" "$NIX_LINK_NEW" 1>&2
end
end
set NIX_LINK "$NIX_LINK_NEW"
end
end
set --local NIX_LINK $HOME/.nix-profile
# Set up environment.
# This part should be kept in sync with nixpkgs:nixos/modules/programs/environment.nix
+1 -10
View File
@@ -329,17 +329,8 @@ connected:
drv.inputSrcs = store->parseStorePathSet(inputs);
optResult = sshStore->buildDerivation(*drvPath, (const BasicDerivation &) drv);
auto & result = *optResult;
if (!result.success()) {
if (settings.keepFailed) {
warn(
"The failed build directory was kept on the remote builder due to `--keep-failed`.%s",
(settings.thisSystem == drv.platform || settings.extraPlatforms.get().count(drv.platform) > 0)
? " You can re-run the command with `--builders ''` to disable remote building for this invocation."
: ""
);
}
if (!result.success())
throw Error("build of '%s' on '%s' failed: %s", store->printStorePath(*drvPath), storeUri, result.errorMsg);
}
} else {
copyClosure(*store, *sshStore, StorePathSet {*drvPath}, NoRepair, NoCheckSigs, substitute);
auto res = sshStore->buildPathsWithResults({
+1 -1
View File
@@ -336,7 +336,7 @@ struct MixEnvironment : virtual Args
StringSet keepVars;
StringSet unsetVars;
StringMap setVars;
std::map<std::string, std::string> setVars;
bool ignoreEnvironment;
MixEnvironment();
@@ -5,7 +5,6 @@
#include "nix/util/canon-path.hh"
#include "nix/main/common-args.hh"
#include "nix/expr/search-path.hh"
#include "nix/expr/eval-settings.hh"
#include <filesystem>
@@ -16,23 +15,25 @@ class Store;
namespace fetchers { struct Settings; }
class EvalState;
struct EvalSettings;
struct CompatibilitySettings;
class Bindings;
struct SourcePath;
namespace flake { struct Settings; }
/**
* @todo Get rid of global settings variables
* @todo Get rid of global setttings variables
*/
extern fetchers::Settings fetchSettings;
/**
* @todo Get rid of global settings variables
* @todo Get rid of global setttings variables
*/
extern EvalSettings evalSettings;
/**
* @todo Get rid of global settings variables
* @todo Get rid of global setttings variables
*/
extern flake::Settings flakeSettings;
+2 -1
View File
@@ -57,7 +57,8 @@ std::optional<DerivedPathWithInfo> InstallableValue::trySinglePathToDerivedPaths
else if (v.type() == nString) {
return {{
.path = DerivedPath::fromSingle(
state->coerceToSingleDerivedPath(pos, v, errorCtx)),
state->devirtualize(
state->coerceToSingleDerivedPath(pos, v, errorCtx))),
.info = make_ref<ExtraPathInfo>(),
}};
}
+2 -2
View File
@@ -199,7 +199,7 @@ SourceExprCommand::SourceExprCommand()
.shortName = 'f',
.description =
"Interpret [*installables*](@docroot@/command-ref/new-cli/nix.md#installables) as attribute paths relative to the Nix expression stored in *file*. "
"If *file* is the character -, then a Nix expression is read from standard input. "
"If *file* is the character -, then a Nix expression will be read from standard input. "
"Implies `--impure`.",
.category = installablesCategory,
.labels = {"file"},
@@ -849,7 +849,7 @@ RawInstallablesCommand::RawInstallablesCommand()
void RawInstallablesCommand::applyDefaultInstallables(std::vector<std::string> & rawInstallables)
{
if (rawInstallables.empty()) {
// FIXME: commands like "nix profile add" should not have a
// FIXME: commands like "nix profile install" should not have a
// default, probably.
rawInstallables.push_back(".");
}
-2
View File
@@ -2,8 +2,6 @@
#include <cstdio>
#include <signal.h>
#if USE_READLINE
#include <readline/history.h>
#include <readline/readline.h>
+13 -55
View File
@@ -69,7 +69,6 @@ struct NixRepl
const static int envSize = 32768;
std::shared_ptr<StaticEnv> staticEnv;
Value lastLoaded;
Env * env;
int displ;
StringSet varNames;
@@ -96,7 +95,6 @@ struct NixRepl
void loadFiles();
void loadFlakes();
void reloadFilesAndFlakes();
void showLastLoaded();
void addAttrsToScope(Value & attrs);
void addVarToScope(const Symbol name, Value & v);
Expr * parseString(std::string s);
@@ -160,8 +158,6 @@ static std::ostream & showDebugTrace(std::ostream & out, const PosTable & positi
return out;
}
MakeError(IncompleteReplExpr, ParseError);
static bool isFirstRepl = true;
ReplExitStatus NixRepl::mainLoop()
@@ -209,8 +205,16 @@ ReplExitStatus NixRepl::mainLoop()
default:
unreachable();
}
} catch (IncompleteReplExpr &) {
continue;
} catch (ParseError & e) {
if (e.msg().find("unexpected end of file") != std::string::npos) {
// For parse errors on incomplete input, we continue waiting for the next line of
// input without clearing the input so far.
continue;
} else {
printMsg(lvlError, e.msg());
}
} catch (EvalError & e) {
printMsg(lvlError, e.msg());
} catch (Error & e) {
printMsg(lvlError, e.msg());
} catch (Interrupted & e) {
@@ -290,7 +294,7 @@ StringSet NixRepl::completePrefix(const std::string & prefix)
} catch (BadURL & e) {
// Quietly ignore BadURL flake-related errors.
} catch (FileNotFound & e) {
// Quietly ignore non-existent file being `import`-ed.
// Quietly ignore non-existent file beeing `import`-ed.
}
}
@@ -374,7 +378,6 @@ ProcessLineResult NixRepl::processLine(std::string line)
<< " current profile\n"
<< " :l, :load <path> Load Nix expression and add it to scope\n"
<< " :lf, :load-flake <ref> Load Nix flake and add it to scope\n"
<< " :ll, :last-loaded Show most recently loaded variables added to scope\n"
<< " :p, :print <expr> Evaluate and print expression recursively\n"
<< " Strings are printed directly, without escaping.\n"
<< " :q, :quit Exit nix-repl\n"
@@ -465,10 +468,6 @@ ProcessLineResult NixRepl::processLine(std::string line)
loadFlake(arg);
}
else if (command == ":ll" || command == ":last-loaded") {
showLastLoaded();
}
else if (command == ":r" || command == ":reload") {
state->resetFileCache();
reloadFilesAndFlakes();
@@ -484,7 +483,7 @@ ProcessLineResult NixRepl::processLine(std::string line)
auto path = state->coerceToPath(noPos, v, context, "while evaluating the filename to edit");
return {path, 0};
} else if (v.isLambda()) {
auto pos = state->positions[v.lambda().fun->pos];
auto pos = state->positions[v.payload.lambda.fun->pos];
if (auto path = std::get_if<SourcePath>(&pos.origin))
return {*path, pos.line};
else
@@ -761,16 +760,6 @@ void NixRepl::initEnv()
varNames.emplace(state->symbols[i.first]);
}
void NixRepl::showLastLoaded()
{
RunPager pager;
for (auto & i : *lastLoaded.attrs()) {
std::string_view name = state->symbols[i.name];
logger->cout(name);
}
}
void NixRepl::reloadFilesAndFlakes()
{
@@ -824,27 +813,6 @@ void NixRepl::addAttrsToScope(Value & attrs)
staticEnv->sort();
staticEnv->deduplicate();
notice("Added %1% variables.", attrs.attrs()->size());
lastLoaded = attrs;
const int max_print = 20;
int counter = 0;
std::ostringstream loaded;
for (auto & i : attrs.attrs()->lexicographicOrder(state->symbols)) {
if (counter >= max_print)
break;
if (counter > 0)
loaded << ", ";
printIdentifier(loaded, state->symbols[i->name]);
counter += 1;
}
notice("%1%", loaded.str());
if (attrs.attrs()->size() > max_print)
notice("... and %1% more; view with :ll", attrs.attrs()->size() - max_print);
}
@@ -869,17 +837,7 @@ Expr * NixRepl::parseString(std::string s)
void NixRepl::evalString(std::string s, Value & v)
{
Expr * e;
try {
e = parseString(s);
} catch (ParseError & e) {
if (e.msg().find("unexpected end of file") != std::string::npos)
// For parse errors on incomplete input, we continue waiting for the next line of
// input without clearing the input so far.
throw IncompleteReplExpr(e.msg());
else
throw;
}
Expr * e = parseString(s);
e->eval(*state, *env, v);
state->forceValue(v, v.determinePos(noPos));
}
+2 -2
View File
@@ -252,7 +252,7 @@ const char * nix_get_path_string(nix_c_context * context, const nix_value * valu
// We could use v.path().to_string().c_str(), but I'm concerned this
// crashes. Looks like .path() allocates a CanonPath with a copy of the
// string, then it gets the underlying data from that.
return v.pathStr();
return v.payload.path.path;
}
NIXC_CATCH_ERRS_NULL
}
@@ -324,7 +324,7 @@ nix_value * nix_get_list_byidx(nix_c_context * context, const nix_value * value,
try {
auto & v = check_value_in(value);
assert(v.type() == nix::nList);
auto * p = v.listView()[ix];
auto * p = v.listElems()[ix];
nix_gc_incref(nullptr, p);
if (p != nullptr)
state->state.forceValue(*p, nix::noPos);
@@ -1,10 +1,9 @@
# Public headers directory
include_dirs = [ include_directories('../../..') ]
include_dirs = [include_directories('../../..')]
headers = files(
'libexpr.hh',
'nix_api_expr.hh',
'value/context.hh',
# hack for trailing newline
)
@@ -23,6 +23,11 @@ struct Arbitrary<NixStringContextElem::DrvDeep> {
static Gen<NixStringContextElem::DrvDeep> arbitrary();
};
template<>
struct Arbitrary<NixStringContextElem::Path> {
static Gen<NixStringContextElem::Path> arbitrary();
};
template<>
struct Arbitrary<NixStringContextElem> {
static Gen<NixStringContextElem> arbitrary();
@@ -15,6 +15,15 @@ Gen<NixStringContextElem::DrvDeep> Arbitrary<NixStringContextElem::DrvDeep>::arb
});
}
Gen<NixStringContextElem::Path> Arbitrary<NixStringContextElem::Path>::arbitrary()
{
return gen::map(gen::arbitrary<StorePath>(), [](StorePath storePath) {
return NixStringContextElem::Path{
.storePath = storePath,
};
});
}
Gen<NixStringContextElem> Arbitrary<NixStringContextElem>::arbitrary()
{
return gen::mapcat(
@@ -30,6 +39,9 @@ Gen<NixStringContextElem> Arbitrary<NixStringContextElem>::arbitrary()
case 2:
return gen::map(
gen::arbitrary<NixStringContextElem::Built>(), [](NixStringContextElem a) { return a; });
case 3:
return gen::map(
gen::arbitrary<NixStringContextElem::Path>(), [](NixStringContextElem a) { return a; });
default:
assert(false);
}
+5 -5
View File
@@ -458,7 +458,7 @@ namespace nix {
HintFmt("expected a function but found %s: %s", "a list", Uncolored("[ ]")),
HintFmt("while evaluating the first argument passed to builtins.filterSource"));
// Unsupported by store "dummy"
// Usupported by store "dummy"
// ASSERT_TRACE2("filterSource (_: 1) ./.",
// TypeError,
@@ -636,7 +636,7 @@ namespace nix {
HintFmt("expected a set but found %s: %s", "a list", Uncolored("[ ]")),
HintFmt("while evaluating the second argument passed to builtins.mapAttrs"));
// XXX: deferred
// XXX: defered
// ASSERT_TRACE2("mapAttrs \"\" { foo.bar = 1; }",
// TypeError,
// HintFmt("attempt to call something which is not a function but %s", "a string"),
@@ -666,9 +666,9 @@ namespace nix {
HintFmt("expected a set but found %s: %s", "an integer", Uncolored(ANSI_CYAN "1" ANSI_NORMAL)),
HintFmt("while evaluating a value of the list passed as second argument to builtins.zipAttrsWith"));
// XXX: How to properly tell that the function takes two arguments ?
// XXX: How to properly tell that the fucntion takes two arguments ?
// The same question also applies to sort, and maybe others.
// Due to laziness, we only create a thunk, and it fails later on.
// Due to lazyness, we only create a thunk, and it fails later on.
// ASSERT_TRACE2("zipAttrsWith (_: 1) [ { foo = 1; } ]",
// TypeError,
// HintFmt("attempt to call something which is not a function but %s", "an integer"),
@@ -877,7 +877,7 @@ namespace nix {
HintFmt("expected a function but found %s: %s", "an integer", Uncolored(ANSI_CYAN "1" ANSI_NORMAL)),
HintFmt("while evaluating the first argument passed to builtins.genList"));
// XXX: deferred
// XXX: defered
// ASSERT_TRACE2("genList (x: x + \"foo\") 2 #TODO",
// TypeError,
// HintFmt("cannot add %s to an integer", "a string"),
+2 -2
View File
@@ -32,8 +32,8 @@ deps_private += rapidcheck
gtest = dependency('gtest')
deps_private += gtest
gmock = dependency('gmock')
deps_private += gmock
gtest = dependency('gmock')
deps_private += gtest
configdata = configuration_data()
configdata.set_quoted('PACKAGE_VERSION', meson.project_version())
+49 -67
View File
@@ -150,8 +150,8 @@ namespace nix {
TEST_F(PrimOpTest, attrValues) {
auto v = eval("builtins.attrValues { x = \"foo\"; a = 1; }");
ASSERT_THAT(v, IsListOfSize(2));
ASSERT_THAT(*v.listView()[0], IsIntEq(1));
ASSERT_THAT(*v.listView()[1], IsStringEq("foo"));
ASSERT_THAT(*v.listElems()[0], IsIntEq(1));
ASSERT_THAT(*v.listElems()[1], IsStringEq("foo"));
}
TEST_F(PrimOpTest, getAttr) {
@@ -250,8 +250,8 @@ namespace nix {
TEST_F(PrimOpTest, catAttrs) {
auto v = eval("builtins.catAttrs \"a\" [{a = 1;} {b = 0;} {a = 2;}]");
ASSERT_THAT(v, IsListOfSize(2));
ASSERT_THAT(*v.listView()[0], IsIntEq(1));
ASSERT_THAT(*v.listView()[1], IsIntEq(2));
ASSERT_THAT(*v.listElems()[0], IsIntEq(1));
ASSERT_THAT(*v.listElems()[1], IsIntEq(2));
}
TEST_F(PrimOpTest, functionArgs) {
@@ -301,7 +301,6 @@ namespace nix {
TEST_F(PrimOpTest, elemtAtOutOfBounds) {
ASSERT_THROW(eval("builtins.elemAt [0 1 2 3] 5"), Error);
ASSERT_THROW(eval("builtins.elemAt [0] 4294967296"), Error);
}
TEST_F(PrimOpTest, head) {
@@ -320,8 +319,7 @@ namespace nix {
TEST_F(PrimOpTest, tail) {
auto v = eval("builtins.tail [ 3 2 1 0 ]");
ASSERT_THAT(v, IsListOfSize(3));
auto listView = v.listView();
for (const auto [n, elem] : enumerate(listView))
for (const auto [n, elem] : enumerate(v.listItems()))
ASSERT_THAT(*elem, IsIntEq(2 - static_cast<int>(n)));
}
@@ -332,17 +330,17 @@ namespace nix {
TEST_F(PrimOpTest, map) {
auto v = eval("map (x: \"foo\" + x) [ \"bar\" \"bla\" \"abc\" ]");
ASSERT_THAT(v, IsListOfSize(3));
auto elem = v.listView()[0];
auto elem = v.listElems()[0];
ASSERT_THAT(*elem, IsThunk());
state.forceValue(*elem, noPos);
ASSERT_THAT(*elem, IsStringEq("foobar"));
elem = v.listView()[1];
elem = v.listElems()[1];
ASSERT_THAT(*elem, IsThunk());
state.forceValue(*elem, noPos);
ASSERT_THAT(*elem, IsStringEq("foobla"));
elem = v.listView()[2];
elem = v.listElems()[2];
ASSERT_THAT(*elem, IsThunk());
state.forceValue(*elem, noPos);
ASSERT_THAT(*elem, IsStringEq("fooabc"));
@@ -351,7 +349,7 @@ namespace nix {
TEST_F(PrimOpTest, filter) {
auto v = eval("builtins.filter (x: x == 2) [ 3 2 3 2 3 2 ]");
ASSERT_THAT(v, IsListOfSize(3));
for (const auto elem : v.listView())
for (const auto elem : v.listItems())
ASSERT_THAT(*elem, IsIntEq(2));
}
@@ -368,8 +366,7 @@ namespace nix {
TEST_F(PrimOpTest, concatLists) {
auto v = eval("builtins.concatLists [[1 2] [3 4]]");
ASSERT_THAT(v, IsListOfSize(4));
auto listView = v.listView();
for (const auto [i, elem] : enumerate(listView))
for (const auto [i, elem] : enumerate(v.listItems()))
ASSERT_THAT(*elem, IsIntEq(static_cast<int>(i)+1));
}
@@ -407,8 +404,7 @@ namespace nix {
auto v = eval("builtins.genList (x: x + 1) 3");
ASSERT_EQ(v.type(), nList);
ASSERT_EQ(v.listSize(), 3u);
auto listView = v.listView();
for (const auto [i, elem] : enumerate(listView)) {
for (const auto [i, elem] : enumerate(v.listItems())) {
ASSERT_THAT(*elem, IsThunk());
state.forceValue(*elem, noPos);
ASSERT_THAT(*elem, IsIntEq(static_cast<int>(i)+1));
@@ -421,8 +417,7 @@ namespace nix {
ASSERT_EQ(v.listSize(), 6u);
const std::vector<int> numbers = { 42, 77, 147, 249, 483, 526 };
auto listView = v.listView();
for (const auto [n, elem] : enumerate(listView))
for (const auto [n, elem] : enumerate(v.listItems()))
ASSERT_THAT(*elem, IsIntEq(numbers[n]));
}
@@ -433,17 +428,17 @@ namespace nix {
auto right = v.attrs()->get(createSymbol("right"));
ASSERT_NE(right, nullptr);
ASSERT_THAT(*right->value, IsListOfSize(2));
ASSERT_THAT(*right->value->listView()[0], IsIntEq(23));
ASSERT_THAT(*right->value->listView()[1], IsIntEq(42));
ASSERT_THAT(*right->value->listElems()[0], IsIntEq(23));
ASSERT_THAT(*right->value->listElems()[1], IsIntEq(42));
auto wrong = v.attrs()->get(createSymbol("wrong"));
ASSERT_NE(wrong, nullptr);
ASSERT_EQ(wrong->value->type(), nList);
ASSERT_EQ(wrong->value->listSize(), 3u);
ASSERT_THAT(*wrong->value, IsListOfSize(3));
ASSERT_THAT(*wrong->value->listView()[0], IsIntEq(1));
ASSERT_THAT(*wrong->value->listView()[1], IsIntEq(9));
ASSERT_THAT(*wrong->value->listView()[2], IsIntEq(3));
ASSERT_THAT(*wrong->value->listElems()[0], IsIntEq(1));
ASSERT_THAT(*wrong->value->listElems()[1], IsIntEq(9));
ASSERT_THAT(*wrong->value->listElems()[2], IsIntEq(3));
}
TEST_F(PrimOpTest, concatMap) {
@@ -452,8 +447,7 @@ namespace nix {
ASSERT_EQ(v.listSize(), 6u);
const std::vector<int> numbers = { 1, 2, 0, 3, 4, 0 };
auto listView = v.listView();
for (const auto [n, elem] : enumerate(listView))
for (const auto [n, elem] : enumerate(v.listItems()))
ASSERT_THAT(*elem, IsIntEq(numbers[n]));
}
@@ -598,16 +592,6 @@ namespace nix {
ASSERT_THAT(v, IsStringEq("n"));
}
TEST_F(PrimOpTest, substringHugeStart){
auto v = eval("builtins.substring 4294967296 5 \"nixos\"");
ASSERT_THAT(v, IsStringEq(""));
}
TEST_F(PrimOpTest, substringHugeLength){
auto v = eval("builtins.substring 0 4294967296 \"nixos\"");
ASSERT_THAT(v, IsStringEq("nixos"));
}
TEST_F(PrimOpTest, substringEmptyString){
auto v = eval("builtins.substring 1 3 \"\"");
ASSERT_THAT(v, IsStringEq(""));
@@ -672,8 +656,8 @@ namespace nix {
auto v = eval("derivation");
ASSERT_EQ(v.type(), nFunction);
ASSERT_TRUE(v.isLambda());
ASSERT_NE(v.lambda().fun, nullptr);
ASSERT_TRUE(v.lambda().fun->hasFormals());
ASSERT_NE(v.payload.lambda.fun, nullptr);
ASSERT_TRUE(v.payload.lambda.fun->hasFormals());
}
TEST_F(PrimOpTest, currentTime) {
@@ -687,8 +671,7 @@ namespace nix {
ASSERT_THAT(v, IsListOfSize(4));
const std::vector<std::string_view> strings = { "1", "2", "3", "git" };
auto listView = v.listView();
for (const auto [n, p] : enumerate(listView))
for (const auto [n, p] : enumerate(v.listItems()))
ASSERT_THAT(*p, IsStringEq(strings[n]));
}
@@ -778,12 +761,12 @@ namespace nix {
auto v = eval("builtins.split \"(a)b\" \"abc\"");
ASSERT_THAT(v, IsListOfSize(3));
ASSERT_THAT(*v.listView()[0], IsStringEq(""));
ASSERT_THAT(*v.listElems()[0], IsStringEq(""));
ASSERT_THAT(*v.listView()[1], IsListOfSize(1));
ASSERT_THAT(*v.listView()[1]->listView()[0], IsStringEq("a"));
ASSERT_THAT(*v.listElems()[1], IsListOfSize(1));
ASSERT_THAT(*v.listElems()[1]->listElems()[0], IsStringEq("a"));
ASSERT_THAT(*v.listView()[2], IsStringEq("c"));
ASSERT_THAT(*v.listElems()[2], IsStringEq("c"));
}
TEST_F(PrimOpTest, split2) {
@@ -791,17 +774,17 @@ namespace nix {
auto v = eval("builtins.split \"([ac])\" \"abc\"");
ASSERT_THAT(v, IsListOfSize(5));
ASSERT_THAT(*v.listView()[0], IsStringEq(""));
ASSERT_THAT(*v.listElems()[0], IsStringEq(""));
ASSERT_THAT(*v.listView()[1], IsListOfSize(1));
ASSERT_THAT(*v.listView()[1]->listView()[0], IsStringEq("a"));
ASSERT_THAT(*v.listElems()[1], IsListOfSize(1));
ASSERT_THAT(*v.listElems()[1]->listElems()[0], IsStringEq("a"));
ASSERT_THAT(*v.listView()[2], IsStringEq("b"));
ASSERT_THAT(*v.listElems()[2], IsStringEq("b"));
ASSERT_THAT(*v.listView()[3], IsListOfSize(1));
ASSERT_THAT(*v.listView()[3]->listView()[0], IsStringEq("c"));
ASSERT_THAT(*v.listElems()[3], IsListOfSize(1));
ASSERT_THAT(*v.listElems()[3]->listElems()[0], IsStringEq("c"));
ASSERT_THAT(*v.listView()[4], IsStringEq(""));
ASSERT_THAT(*v.listElems()[4], IsStringEq(""));
}
TEST_F(PrimOpTest, split3) {
@@ -809,36 +792,36 @@ namespace nix {
ASSERT_THAT(v, IsListOfSize(5));
// First list element
ASSERT_THAT(*v.listView()[0], IsStringEq(""));
ASSERT_THAT(*v.listElems()[0], IsStringEq(""));
// 2nd list element is a list [ "" null ]
ASSERT_THAT(*v.listView()[1], IsListOfSize(2));
ASSERT_THAT(*v.listView()[1]->listView()[0], IsStringEq("a"));
ASSERT_THAT(*v.listView()[1]->listView()[1], IsNull());
ASSERT_THAT(*v.listElems()[1], IsListOfSize(2));
ASSERT_THAT(*v.listElems()[1]->listElems()[0], IsStringEq("a"));
ASSERT_THAT(*v.listElems()[1]->listElems()[1], IsNull());
// 3rd element
ASSERT_THAT(*v.listView()[2], IsStringEq("b"));
ASSERT_THAT(*v.listElems()[2], IsStringEq("b"));
// 4th element is a list: [ null "c" ]
ASSERT_THAT(*v.listView()[3], IsListOfSize(2));
ASSERT_THAT(*v.listView()[3]->listView()[0], IsNull());
ASSERT_THAT(*v.listView()[3]->listView()[1], IsStringEq("c"));
ASSERT_THAT(*v.listElems()[3], IsListOfSize(2));
ASSERT_THAT(*v.listElems()[3]->listElems()[0], IsNull());
ASSERT_THAT(*v.listElems()[3]->listElems()[1], IsStringEq("c"));
// 5th element is the empty string
ASSERT_THAT(*v.listView()[4], IsStringEq(""));
ASSERT_THAT(*v.listElems()[4], IsStringEq(""));
}
TEST_F(PrimOpTest, split4) {
auto v = eval("builtins.split \"([[:upper:]]+)\" \" FOO \"");
ASSERT_THAT(v, IsListOfSize(3));
auto first = v.listView()[0];
auto second = v.listView()[1];
auto third = v.listView()[2];
auto first = v.listElems()[0];
auto second = v.listElems()[1];
auto third = v.listElems()[2];
ASSERT_THAT(*first, IsStringEq(" "));
ASSERT_THAT(*second, IsListOfSize(1));
ASSERT_THAT(*second->listView()[0], IsStringEq("FOO"));
ASSERT_THAT(*second->listElems()[0], IsStringEq("FOO"));
ASSERT_THAT(*third, IsStringEq(" "));
}
@@ -856,14 +839,14 @@ namespace nix {
TEST_F(PrimOpTest, match3) {
auto v = eval("builtins.match \"a(b)(c)\" \"abc\"");
ASSERT_THAT(v, IsListOfSize(2));
ASSERT_THAT(*v.listView()[0], IsStringEq("b"));
ASSERT_THAT(*v.listView()[1], IsStringEq("c"));
ASSERT_THAT(*v.listElems()[0], IsStringEq("b"));
ASSERT_THAT(*v.listElems()[1], IsStringEq("c"));
}
TEST_F(PrimOpTest, match4) {
auto v = eval("builtins.match \"[[:space:]]+([[:upper:]]+)[[:space:]]+\" \" FOO \"");
ASSERT_THAT(v, IsListOfSize(1));
ASSERT_THAT(*v.listView()[0], IsStringEq("FOO"));
ASSERT_THAT(*v.listElems()[0], IsStringEq("FOO"));
}
TEST_F(PrimOpTest, match5) {
@@ -880,8 +863,7 @@ namespace nix {
// ensure that the list is sorted
const std::vector<std::string_view> expected { "a", "x", "y", "z" };
auto listView = v.listView();
for (const auto [n, elem] : enumerate(listView))
for (const auto [n, elem] : enumerate(v.listItems()))
ASSERT_THAT(*elem, IsStringEq(expected[n]));
}
+1 -1
View File
@@ -143,7 +143,7 @@ namespace nix {
// Usually Nix rejects duplicate keys in an attrset but it does allow
// so if it is an attribute set that contains disjoint sets of keys.
// The below is equivalent to `{a.b = 1; a.c = 2; }`.
// The attribute set `a` will be a Thunk at first as the attributes
// The attribute set `a` will be a Thunk at first as the attribuets
// have to be merged (or otherwise computed) and that is done in a lazy
// manner.
+1 -1
View File
@@ -95,7 +95,7 @@ std::pair<Value *, PosIdx> findAlongAttrPath(EvalState & state, const std::strin
if (*attrIndex >= v->listSize())
throw AttrPathNotFound("list index %1% in selection path '%2%' is out of range", *attrIndex, attrPath);
v = v->listView()[*attrIndex];
v = v->listElems()[*attrIndex];
pos = noPos;
}
+12 -9
View File
@@ -618,18 +618,21 @@ string_t AttrCursor::getStringWithContext()
if (auto s = std::get_if<string_t>(&cachedValue->second)) {
bool valid = true;
for (auto & c : s->second) {
const StorePath & path = std::visit(overloaded {
[&](const NixStringContextElem::DrvDeep & d) -> const StorePath & {
return d.drvPath;
const StorePath * path = std::visit(overloaded {
[&](const NixStringContextElem::DrvDeep & d) -> const StorePath * {
return &d.drvPath;
},
[&](const NixStringContextElem::Built & b) -> const StorePath & {
return b.drvPath->getBaseStorePath();
[&](const NixStringContextElem::Built & b) -> const StorePath * {
return &b.drvPath->getBaseStorePath();
},
[&](const NixStringContextElem::Opaque & o) -> const StorePath & {
return o.path;
[&](const NixStringContextElem::Opaque & o) -> const StorePath * {
return &o.path;
},
[&](const NixStringContextElem::Path & p) -> const StorePath * {
return nullptr;
},
}, c.raw);
if (!root->state.store->isValidPath(path)) {
if (!path || !root->state.store->isValidPath(*path)) {
valid = false;
break;
}
@@ -721,7 +724,7 @@ std::vector<std::string> AttrCursor::getListOfStrings()
std::vector<std::string> res;
for (auto elem : v.listView())
for (auto & elem : v.listItems())
res.push_back(std::string(root->state.forceStringNoCtx(*elem, noPos, "while evaluating an attribute for caching")));
if (root->db)
-8
View File
@@ -4,7 +4,6 @@
#include "nix/util/config-global.hh"
#include "nix/util/serialise.hh"
#include "nix/expr/eval-gc.hh"
#include "nix/expr/value.hh"
#include "expr-config-private.hh"
@@ -53,13 +52,6 @@ static inline void initGCReal()
GC_INIT();
/* Register valid displacements in case we are using alignment niches
for storing the type information. This way tagged pointers are considered
to be valid, even when they are not aligned. */
if constexpr (detail::useBitPackedValueStorage<sizeof(void *)>)
for (std::size_t i = 1; i < sizeof(std::uintptr_t); ++i)
GC_register_displacement(i);
GC_set_oom_fn(oomHandler);
/* Set the initial heap size to something fairly big (25% of
-49
View File
@@ -1,49 +0,0 @@
#include "nix/expr/eval-profiler-settings.hh"
#include "nix/util/configuration.hh"
#include "nix/util/logging.hh" /* Needs to be included before config-impl.hh */
#include "nix/util/config-impl.hh"
#include "nix/util/abstract-setting-to-json.hh"
#include <nlohmann/json.hpp>
namespace nix {
template<>
EvalProfilerMode BaseSetting<EvalProfilerMode>::parse(const std::string & str) const
{
if (str == "disabled")
return EvalProfilerMode::disabled;
else if (str == "flamegraph")
return EvalProfilerMode::flamegraph;
else
throw UsageError("option '%s' has invalid value '%s'", name, str);
}
template<>
struct BaseSetting<EvalProfilerMode>::trait
{
static constexpr bool appendable = false;
};
template<>
std::string BaseSetting<EvalProfilerMode>::to_string() const
{
if (value == EvalProfilerMode::disabled)
return "disabled";
else if (value == EvalProfilerMode::flamegraph)
return "flamegraph";
else
unreachable();
}
NLOHMANN_JSON_SERIALIZE_ENUM(
EvalProfilerMode,
{
{EvalProfilerMode::disabled, "disabled"},
{EvalProfilerMode::flamegraph, "flamegraph"},
});
/* Explicit instantiation of templates */
template class BaseSetting<EvalProfilerMode>;
}
+8 -315
View File
@@ -1,18 +1,20 @@
#include "nix/expr/eval-profiler.hh"
#include "nix/expr/nixexpr.hh"
#include "nix/expr/eval.hh"
#include "nix/util/lru-cache.hh"
namespace nix {
void EvalProfiler::preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) {}
void EvalProfiler::preFunctionCallHook(
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
}
void EvalProfiler::postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
void EvalProfiler::postFunctionCallHook(
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
}
void MultiEvalProfiler::preFunctionCallHook(
EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
for (auto & profiler : profilers) {
if (profiler->getNeededHooks().test(Hook::preFunctionCall))
@@ -21,7 +23,7 @@ void MultiEvalProfiler::preFunctionCallHook(
}
void MultiEvalProfiler::postFunctionCallHook(
EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
for (auto & profiler : profilers) {
if (profiler->getNeededHooks().test(Hook::postFunctionCall))
@@ -43,313 +45,4 @@ void MultiEvalProfiler::addProfiler(ref<EvalProfiler> profiler)
invalidateNeededHooks();
}
namespace {
class PosCache : private LRUCache<PosIdx, Pos>
{
const EvalState & state;
public:
PosCache(const EvalState & state)
: LRUCache(524288) /* ~40MiB */
, state(state)
{
}
Pos lookup(PosIdx posIdx)
{
auto posOrNone = LRUCache::get(posIdx);
if (posOrNone)
return *posOrNone;
auto pos = state.positions[posIdx];
upsert(posIdx, pos);
return pos;
}
};
struct LambdaFrameInfo
{
ExprLambda * expr;
/** Position where the lambda has been called. */
PosIdx callPos = noPos;
std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const;
auto operator<=>(const LambdaFrameInfo & rhs) const = default;
};
/** Primop call. */
struct PrimOpFrameInfo
{
const PrimOp * expr;
/** Position where the primop has been called. */
PosIdx callPos = noPos;
std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const;
auto operator<=>(const PrimOpFrameInfo & rhs) const = default;
};
/** Used for functor calls (attrset with __functor attr). */
struct FunctorFrameInfo
{
PosIdx pos;
std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const;
auto operator<=>(const FunctorFrameInfo & rhs) const = default;
};
struct DerivationStrictFrameInfo
{
PosIdx callPos = noPos;
std::string drvName;
std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const;
auto operator<=>(const DerivationStrictFrameInfo & rhs) const = default;
};
/** Fallback frame info. */
struct GenericFrameInfo
{
PosIdx pos;
std::ostream & symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const;
auto operator<=>(const GenericFrameInfo & rhs) const = default;
};
using FrameInfo =
std::variant<LambdaFrameInfo, PrimOpFrameInfo, FunctorFrameInfo, DerivationStrictFrameInfo, GenericFrameInfo>;
using FrameStack = std::vector<FrameInfo>;
/**
* Stack sampling profiler.
*/
class SampleStack : public EvalProfiler
{
/* How often stack profiles should be flushed to file. This avoids the need
to persist stack samples across the whole evaluation at the cost
of periodically flushing data to disk. */
static constexpr std::chrono::microseconds profileDumpInterval = std::chrono::milliseconds(2000);
Hooks getNeededHooksImpl() const override
{
return Hooks().set(preFunctionCall).set(postFunctionCall);
}
FrameInfo getPrimOpFrameInfo(const PrimOp & primOp, std::span<Value *> args, PosIdx pos);
public:
SampleStack(EvalState & state, std::filesystem::path profileFile, std::chrono::nanoseconds period)
: state(state)
, sampleInterval(period)
, profileFd([&]() {
AutoCloseFD fd = toDescriptor(open(profileFile.string().c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0660));
if (!fd)
throw SysError("opening file %s", profileFile);
return fd;
}())
, posCache(state)
{
}
[[gnu::noinline]] void
preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
[[gnu::noinline]] void
postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
void maybeSaveProfile(std::chrono::time_point<std::chrono::high_resolution_clock> now);
void saveProfile();
FrameInfo getFrameInfoFromValueAndPos(const Value & v, std::span<Value *> args, PosIdx pos);
SampleStack(SampleStack &&) = default;
SampleStack & operator=(SampleStack &&) = delete;
SampleStack(const SampleStack &) = delete;
SampleStack & operator=(const SampleStack &) = delete;
~SampleStack();
private:
/** Hold on to an instance of EvalState for symbolizing positions. */
EvalState & state;
std::chrono::nanoseconds sampleInterval;
AutoCloseFD profileFd;
FrameStack stack;
std::map<FrameStack, uint32_t> callCount;
std::chrono::time_point<std::chrono::high_resolution_clock> lastStackSample =
std::chrono::high_resolution_clock::now();
std::chrono::time_point<std::chrono::high_resolution_clock> lastDump = std::chrono::high_resolution_clock::now();
PosCache posCache;
};
FrameInfo SampleStack::getPrimOpFrameInfo(const PrimOp & primOp, std::span<Value *> args, PosIdx pos)
{
auto derivationInfo = [&]() -> std::optional<FrameInfo> {
/* Here we rely a bit on the implementation details of libexpr/primops/derivation.nix
and derivationStrict primop. This is not ideal, but is necessary for
the usefulness of the profiler. This might actually affect the evaluation,
but the cost shouldn't be that high as to make the traces entirely inaccurate. */
if (primOp.name == "derivationStrict") {
try {
/* Error context strings don't actually matter, since we ignore all eval errors. */
state.forceAttrs(*args[0], pos, "");
auto attrs = args[0]->attrs();
auto nameAttr = state.getAttr(state.sName, attrs, "");
auto drvName = std::string(state.forceStringNoCtx(*nameAttr->value, pos, ""));
return DerivationStrictFrameInfo{.callPos = pos, .drvName = std::move(drvName)};
} catch (...) {
/* Ignore all errors, since those will be diagnosed by the evaluator itself. */
}
}
return std::nullopt;
}();
return derivationInfo.value_or(PrimOpFrameInfo{.expr = &primOp, .callPos = pos});
}
FrameInfo SampleStack::getFrameInfoFromValueAndPos(const Value & v, std::span<Value *> args, PosIdx pos)
{
/* NOTE: No actual references to garbage collected values are not held in
the profiler. */
if (v.isLambda())
return LambdaFrameInfo{.expr = v.lambda().fun, .callPos = pos};
else if (v.isPrimOp()) {
return getPrimOpFrameInfo(*v.primOp(), args, pos);
} else if (v.isPrimOpApp())
/* Resolve primOp eagerly. Must not hold on to a reference to a Value. */
return PrimOpFrameInfo{.expr = v.primOpAppPrimOp(), .callPos = pos};
else if (state.isFunctor(v)) {
const auto functor = v.attrs()->get(state.sFunctor);
if (auto pos_ = posCache.lookup(pos); std::holds_alternative<std::monostate>(pos_.origin))
/* HACK: In case callsite position is unresolved. */
return FunctorFrameInfo{.pos = functor->pos};
return FunctorFrameInfo{.pos = pos};
} else
/* NOTE: Add a stack frame even for invalid cases (e.g. when calling a non-function). This is what
* trace-function-calls does. */
return GenericFrameInfo{.pos = pos};
}
[[gnu::noinline]] void
SampleStack::preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
stack.push_back(getFrameInfoFromValueAndPos(v, args, pos));
auto now = std::chrono::high_resolution_clock::now();
if (now - lastStackSample > sampleInterval) {
callCount[stack] += 1;
lastStackSample = now;
}
/* Do this in preFunctionCallHook because we might throw an exception, but
callFunction uses Finally, which doesn't play well with exceptions. */
maybeSaveProfile(now);
}
[[gnu::noinline]] void
SampleStack::postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
if (!stack.empty())
stack.pop_back();
}
std::ostream & LambdaFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const
{
if (auto pos = posCache.lookup(callPos); std::holds_alternative<std::monostate>(pos.origin))
/* HACK: To avoid dubious «none»:0 in the generated profile if the origin can't be resolved
resort to printing the lambda location instead of the callsite position. */
os << posCache.lookup(expr->getPos());
else
os << pos;
if (expr->name)
os << ":" << state.symbols[expr->name];
return os;
}
std::ostream & GenericFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const
{
os << posCache.lookup(pos);
return os;
}
std::ostream & FunctorFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const
{
os << posCache.lookup(pos) << ":functor";
return os;
}
std::ostream & PrimOpFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const
{
/* Sometimes callsite position can have an unresolved origin, which
leads to confusing «none»:0 locations in the profile. */
auto pos = posCache.lookup(callPos);
if (!std::holds_alternative<std::monostate>(pos.origin))
os << posCache.lookup(callPos) << ":";
os << *expr;
return os;
}
std::ostream &
DerivationStrictFrameInfo::symbolize(const EvalState & state, std::ostream & os, PosCache & posCache) const
{
/* Sometimes callsite position can have an unresolved origin, which
leads to confusing «none»:0 locations in the profile. */
auto pos = posCache.lookup(callPos);
if (!std::holds_alternative<std::monostate>(pos.origin))
os << posCache.lookup(callPos) << ":";
os << "primop derivationStrict:" << drvName;
return os;
}
void SampleStack::maybeSaveProfile(std::chrono::time_point<std::chrono::high_resolution_clock> now)
{
if (now - lastDump >= profileDumpInterval)
saveProfile();
else
return;
/* Save the last dump timepoint. Do this after actually saving data to file
to not account for the time doing the flushing to disk. */
lastDump = std::chrono::high_resolution_clock::now();
/* Free up memory used for stack sampling. This might be very significant for
long-running evaluations, so we shouldn't hog too much memory. */
callCount.clear();
}
void SampleStack::saveProfile()
{
auto os = std::ostringstream{};
for (auto & [stack, count] : callCount) {
auto first = true;
for (auto & pos : stack) {
if (first)
first = false;
else
os << ";";
std::visit([&](auto && info) { info.symbolize(state, os, posCache); }, pos);
}
os << " " << count;
writeLine(profileFd.get(), std::move(os).str());
/* Clear ostringstream. */
os.str("");
os.clear();
}
}
SampleStack::~SampleStack()
{
/* Guard against cases when we are already unwinding the stack. */
try {
saveProfile();
} catch (...) {
ignoreExceptionInDestructor();
}
}
} // namespace
ref<EvalProfiler> makeSampleStackProfiler(EvalState & state, std::filesystem::path profileFile, uint64_t frequency)
{
/* 0 is a special value for sampling stack after each call. */
std::chrono::nanoseconds period = frequency == 0
? std::chrono::nanoseconds{0}
: std::chrono::nanoseconds{std::nano::den / frequency / std::nano::num};
return make_ref<SampleStack>(state, profileFile, period);
}
}
+111 -93
View File
@@ -2,7 +2,6 @@
#include "nix/expr/eval-settings.hh"
#include "nix/expr/primops.hh"
#include "nix/expr/print-options.hh"
#include "nix/expr/symbol-table.hh"
#include "nix/util/exit.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
@@ -16,6 +15,7 @@
#include "nix/expr/print.hh"
#include "nix/fetchers/filtering-source-accessor.hh"
#include "nix/util/memory-source-accessor.hh"
#include "nix/util/mounted-source-accessor.hh"
#include "nix/expr/gc-small-vector.hh"
#include "nix/util/url.hh"
#include "nix/fetchers/fetch-to-store.hh"
@@ -90,16 +90,20 @@ std::string printValue(EvalState & state, Value & v)
return out.str();
}
Value * Value::toPtr(SymbolStr str) noexcept
{
return const_cast<Value *>(str.valuePtr());
}
void Value::print(EvalState & state, std::ostream & str, PrintOptions options)
{
printValue(state, str, *this, options);
}
const Value * getPrimOp(const Value &v) {
const Value * primOp = &v;
while (primOp->isPrimOpApp()) {
primOp = primOp->payload.primOpApp.left;
}
assert(primOp->isPrimOp());
return primOp;
}
std::string_view showType(ValueType type, bool withArticle)
{
#define WA(a, w) withArticle ? a " " w : w
@@ -125,12 +129,12 @@ std::string showType(const Value & v)
// Allow selecting a subset of enum values
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wswitch-enum"
switch (v.getInternalType()) {
case tString: return v.context() ? "a string with context" : "a string";
switch (v.internalType) {
case tString: return v.payload.string.context ? "a string with context" : "a string";
case tPrimOp:
return fmt("the built-in function '%s'", std::string(v.primOp()->name));
return fmt("the built-in function '%s'", std::string(v.payload.primOp->name));
case tPrimOpApp:
return fmt("the partially applied built-in function '%s'", v.primOpAppPrimOp()->name);
return fmt("the partially applied built-in function '%s'", std::string(getPrimOp(v)->payload.primOp->name));
case tExternal: return v.external()->showType();
case tThunk: return v.isBlackhole() ? "a black hole" : "a thunk";
case tApp: return "a function application";
@@ -145,10 +149,10 @@ PosIdx Value::determinePos(const PosIdx pos) const
// Allow selecting a subset of enum values
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wswitch-enum"
switch (getInternalType()) {
switch (internalType) {
case tAttrs: return attrs()->pos;
case tLambda: return lambda().fun->pos;
case tApp: return app().left->determinePos(pos);
case tLambda: return payload.lambda.fun->pos;
case tApp: return payload.app.left->determinePos(pos);
default: return pos;
}
#pragma GCC diagnostic pop
@@ -157,12 +161,13 @@ PosIdx Value::determinePos(const PosIdx pos) const
bool Value::isTrivial() const
{
return
!isa<tApp, tPrimOpApp>()
&& (!isa<tThunk>()
|| (dynamic_cast<ExprAttrs *>(thunk().expr)
&& ((ExprAttrs *) thunk().expr)->dynamicAttrs.empty())
|| dynamic_cast<ExprLambda *>(thunk().expr)
|| dynamic_cast<ExprList *>(thunk().expr));
internalType != tApp
&& internalType != tPrimOpApp
&& (internalType != tThunk
|| (dynamic_cast<ExprAttrs *>(payload.thunk.expr)
&& ((ExprAttrs *) payload.thunk.expr)->dynamicAttrs.empty())
|| dynamic_cast<ExprLambda *>(payload.thunk.expr)
|| dynamic_cast<ExprList *>(payload.thunk.expr));
}
@@ -208,7 +213,6 @@ EvalState::EvalState(
, sRight(symbols.create("right"))
, sWrong(symbols.create("wrong"))
, sStructuredAttrs(symbols.create("__structuredAttrs"))
, sJson(symbols.create("__json"))
, sAllowedReferences(symbols.create("allowedReferences"))
, sAllowedRequisites(symbols.create("allowedRequisites"))
, sDisallowedReferences(symbols.create("disallowedReferences"))
@@ -266,7 +270,7 @@ EvalState::EvalState(
exception, and make union source accessor
catch it, so we don't need to do this hack.
*/
{CanonPath(store->storeDir), store->getFSAccessor(settings.pureEval)},
{CanonPath(store->storeDir), makeFSSourceAccessor(dirOf(store->toRealPath(StorePath::dummy)))}
}))
, rootFS(
({
@@ -281,12 +285,9 @@ EvalState::EvalState(
/nix/store while using a chroot store. */
auto accessor = getFSSourceAccessor();
auto realStoreDir = dirOf(store->toRealPath(StorePath::dummy));
if (settings.pureEval || store->storeDir != realStoreDir) {
accessor = settings.pureEval
? storeFS
: makeUnionSourceAccessor({accessor, storeFS});
}
accessor = settings.pureEval
? storeFS.cast<SourceAccessor>()
: makeUnionSourceAccessor({accessor, storeFS});
/* Apply access control if needed. */
if (settings.restrictEval || settings.pureEval)
@@ -373,17 +374,9 @@ EvalState::EvalState(
/* Register function call tracer. */
if (settings.traceFunctionCalls)
profiler.addProfiler(make_ref<FunctionCallTrace>());
switch (settings.evalProfilerMode) {
case EvalProfilerMode::flamegraph:
profiler.addProfiler(makeSampleStackProfiler(
*this, settings.evalProfileFile.get(), settings.evalProfilerFrequency));
break;
case EvalProfilerMode::disabled:
break;
}
}
EvalState::~EvalState()
{
}
@@ -502,7 +495,7 @@ void EvalState::addConstant(const std::string & name, Value * v, Constant info)
/* Install value the base environment. */
staticBaseEnv->vars.emplace_back(symbols.create(name), baseEnvDispl);
baseEnv.values[baseEnvDispl++] = v;
const_cast<Bindings *>(getBuiltins().attrs())->push_back(Attr(symbols.create(name2), v));
getBuiltins().payload.attrs->push_back(Attr(symbols.create(name2), v));
}
}
@@ -524,15 +517,13 @@ std::ostream & operator<<(std::ostream & output, const PrimOp & primOp)
const PrimOp * Value::primOpAppPrimOp() const
{
Value * left = primOpApp().left;
Value * left = payload.primOpApp.left;
while (left && !left->isPrimOp()) {
left = left->primOpApp().left;
left = left->payload.primOpApp.left;
}
if (!left)
return nullptr;
assert(left->isPrimOp());
return left->primOp();
}
@@ -540,7 +531,7 @@ const PrimOp * Value::primOpAppPrimOp() const
void Value::mkPrimOp(PrimOp * p)
{
p->check();
setStorage(p);
finishValue(tPrimOp, { .primOp = p });
}
@@ -572,7 +563,7 @@ Value * EvalState::addPrimOp(PrimOp && primOp)
else {
staticBaseEnv->vars.emplace_back(envName, baseEnvDispl);
baseEnv.values[baseEnvDispl++] = v;
const_cast<Bindings *>(getBuiltins().attrs())->push_back(Attr(symbols.create(primOp.name), v));
getBuiltins().payload.attrs->push_back(Attr(symbols.create(primOp.name), v));
}
return v;
@@ -609,7 +600,7 @@ std::optional<EvalState::Doc> EvalState::getDoc(Value & v)
};
}
if (v.isLambda()) {
auto exprLambda = v.lambda().fun;
auto exprLambda = v.payload.lambda.fun;
std::ostringstream s;
std::string name;
@@ -656,7 +647,7 @@ std::optional<EvalState::Doc> EvalState::getDoc(Value & v)
Value & functor = *v.attrs()->find(sFunctor)->value;
Value * vp[] = {&v};
Value partiallyApplied;
// The first parameter is not user-provided, and may be
// The first paramater is not user-provided, and may be
// handled by code that is opaque to the user, like lib.const = x: y: y;
// So preferably we show docs that are relevant to the
// "partially applied" function returned by e.g. `const`.
@@ -979,7 +970,16 @@ void EvalState::mkPos(Value & v, PosIdx p)
auto origin = positions.originOf(p);
if (auto path = std::get_if<SourcePath>(&origin)) {
auto attrs = buildBindings(3);
attrs.alloc(sFile).mkString(path->path.abs());
if (path->accessor == rootFS && store->isInStore(path->path.abs()))
// FIXME: only do this for virtual store paths?
attrs.alloc(sFile).mkString(path->path.abs(),
{
NixStringContextElem::Path{
.storePath = store->toStorePath(path->path.abs()).first
}
});
else
attrs.alloc(sFile).mkString(path->path.abs());
makePositionThunks(*this, p, attrs.alloc(sLine), attrs.alloc(sColumn));
v.mkAttrs(attrs);
} else
@@ -1566,13 +1566,13 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
if (vCur.isLambda()) {
ExprLambda & lambda(*vCur.lambda().fun);
ExprLambda & lambda(*vCur.payload.lambda.fun);
auto size =
(!lambda.arg ? 0 : 1) +
(lambda.hasFormals() ? lambda.formals->formals.size() : 0);
Env & env2(allocEnv(size));
env2.up = vCur.lambda().env;
env2.up = vCur.payload.lambda.env;
Displacement displ = 0;
@@ -1602,7 +1602,7 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
symbols[i.name])
.atPos(lambda.pos)
.withTrace(pos, "from call site")
.withFrame(*vCur.lambda().env, lambda)
.withFrame(*fun.payload.lambda.env, lambda)
.debugThrow();
}
env2.values[displ++] = i.def->maybeThunk(*this, env2);
@@ -1629,7 +1629,7 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
.atPos(lambda.pos)
.withTrace(pos, "from call site")
.withSuggestions(suggestions)
.withFrame(*vCur.lambda().env, lambda)
.withFrame(*fun.payload.lambda.env, lambda)
.debugThrow();
}
unreachable();
@@ -1701,7 +1701,7 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
Value * primOp = &vCur;
while (primOp->isPrimOpApp()) {
argsDone++;
primOp = primOp->primOpApp().left;
primOp = primOp->payload.primOpApp.left;
}
assert(primOp->isPrimOp());
auto arity = primOp->primOp()->arity;
@@ -1717,8 +1717,8 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
Value * vArgs[maxPrimOpArity];
auto n = argsDone;
for (Value * arg = &vCur; arg->isPrimOpApp(); arg = arg->primOpApp().left)
vArgs[--n] = arg->primOpApp().right;
for (Value * arg = &vCur; arg->isPrimOpApp(); arg = arg->payload.primOpApp.left)
vArgs[--n] = arg->payload.primOpApp.right;
for (size_t i = 0; i < argsLeft; ++i)
vArgs[argsDone + i] = args[i];
@@ -1824,14 +1824,14 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res
}
}
if (!fun.isLambda() || !fun.lambda().fun->hasFormals()) {
if (!fun.isLambda() || !fun.payload.lambda.fun->hasFormals()) {
res = fun;
return;
}
auto attrs = buildBindings(std::max(static_cast<uint32_t>(fun.lambda().fun->formals->formals.size()), args.size()));
auto attrs = buildBindings(std::max(static_cast<uint32_t>(fun.payload.lambda.fun->formals->formals.size()), args.size()));
if (fun.lambda().fun->formals->ellipsis) {
if (fun.payload.lambda.fun->formals->ellipsis) {
// If the formals have an ellipsis (eg the function accepts extra args) pass
// all available automatic arguments (which includes arguments specified on
// the command line via --arg/--argstr)
@@ -1839,7 +1839,7 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res
attrs.insert(v);
} else {
// Otherwise, only pass the arguments that the function accepts
for (auto & i : fun.lambda().fun->formals->formals) {
for (auto & i : fun.payload.lambda.fun->formals->formals) {
auto j = args.get(i.name);
if (j) {
attrs.insert(*j);
@@ -1849,7 +1849,7 @@ Nix attempted to evaluate a function as a top level expression; in
this case it must have its arguments supplied either by default
values, or passed explicitly with '--arg' or '--argstr'. See
https://nixos.org/manual/nix/stable/language/constructs.html#functions.)", symbols[i.name])
.atPos(i.pos).withFrame(*fun.lambda().env, *fun.lambda().fun).debugThrow();
.atPos(i.pos).withFrame(*fun.payload.lambda.env, *fun.payload.lambda.fun).debugThrow();
}
}
}
@@ -2007,10 +2007,9 @@ void EvalState::concatLists(Value & v, size_t nrLists, Value * const * lists, co
auto list = buildList(len);
auto out = list.elems;
for (size_t n = 0, pos = 0; n < nrLists; ++n) {
auto listView = lists[n]->listView();
auto l = listView.size();
auto l = lists[n]->listSize();
if (l)
memcpy(out + pos, listView.data(), l * sizeof(Value *));
memcpy(out + pos, lists[n]->listElems(), l * sizeof(Value *));
pos += l;
}
v.mkList(list);
@@ -2106,7 +2105,7 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
else if (firstType == nFloat)
v.mkFloat(nf);
else if (firstType == nPath) {
if (!context.empty())
if (hasContext(context))
state.error<EvalError>("a string that refers to a store path cannot be appended to a path").atPos(pos).withFrame(env, *this).debugThrow();
v.mkPath(state.rootPath(CanonPath(str())));
} else
@@ -2163,7 +2162,7 @@ void EvalState::forceValueDeep(Value & v)
try {
// If the value is a thunk, we're evaling. Otherwise no trace necessary.
auto dts = debugRepl && i.value->isThunk()
? makeDebugTraceStacker(*this, *i.value->thunk().expr, *i.value->thunk().env, i.pos,
? makeDebugTraceStacker(*this, *i.value->payload.thunk.expr, *i.value->payload.thunk.env, i.pos,
"while evaluating the attribute '%1%'", symbols[i.name])
: nullptr;
@@ -2175,7 +2174,7 @@ void EvalState::forceValueDeep(Value & v)
}
else if (v.isList()) {
for (auto v2 : v.listView())
for (auto v2 : v.listItems())
recurse(*v2);
}
};
@@ -2243,18 +2242,8 @@ bool EvalState::forceBool(Value & v, const PosIdx pos, std::string_view errorCtx
return v.boolean();
}
Bindings::const_iterator EvalState::getAttr(Symbol attrSym, const Bindings * attrSet, std::string_view errorCtx)
{
auto value = attrSet->find(attrSym);
if (value == attrSet->end()) {
error<TypeError>("attribute '%s' missing", symbols[attrSym])
.withTrace(noPos, errorCtx)
.debugThrow();
}
return value;
}
bool EvalState::isFunctor(const Value & fun) const
bool EvalState::isFunctor(Value & fun)
{
return fun.type() == nAttrs && fun.attrs()->find(sFunctor) != fun.attrs()->end();
}
@@ -2297,8 +2286,8 @@ std::string_view EvalState::forceString(Value & v, const PosIdx pos, std::string
void copyContext(const Value & v, NixStringContext & context, const ExperimentalFeatureSettings & xpSettings)
{
if (v.context())
for (const char * * p = v.context(); *p; ++p)
if (v.payload.string.context)
for (const char * * p = v.payload.string.context; *p; ++p)
context.insert(NixStringContextElem::parse(*p, xpSettings));
}
@@ -2315,7 +2304,10 @@ std::string_view EvalState::forceStringNoCtx(Value & v, const PosIdx pos, std::s
{
auto s = forceString(v, pos, errorCtx);
if (v.context()) {
error<EvalError>("the string '%1%' is not allowed to refer to a store path (such as '%2%')", v.string_view(), v.context()[0]).withTrace(pos, errorCtx).debugThrow();
NixStringContext context;
copyContext(v, context);
if (hasContext(context))
error<EvalError>("the string '%1%' is not allowed to refer to a store path (such as '%2%')", v.string_view(), v.context()[0]).withTrace(pos, errorCtx).debugThrow();
}
return s;
}
@@ -2364,14 +2356,26 @@ BackedStringView EvalState::coerceToString(
}
if (v.type() == nPath) {
// FIXME: instead of copying the path to the store, we could
// return a virtual store path that lazily copies the path to
// the store in devirtualize().
return
!canonicalizePath && !copyToStore
? // FIXME: hack to preserve path literals that end in a
// slash, as in /foo/${x}.
v.pathStr()
v.payload.path.path
: copyToStore
? store->printStorePath(copyPathToStore(context, v.path()))
: std::string(v.path().path.abs());
: ({
auto path = v.path();
if (path.accessor == rootFS && store->isInStore(path.path.abs())) {
context.insert(
NixStringContextElem::Path{
.storePath = store->toStorePath(path.path.abs()).first
});
}
std::string(path.path.abs());
});
}
if (v.type() == nAttrs) {
@@ -2412,8 +2416,7 @@ BackedStringView EvalState::coerceToString(
if (v.isList()) {
std::string result;
auto listView = v.listView();
for (auto [n, v2] : enumerate(listView)) {
for (auto [n, v2] : enumerate(v.listItems())) {
try {
result += *coerceToString(pos, *v2, context,
"while evaluating one element of the list",
@@ -2455,7 +2458,7 @@ StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePat
*store,
path.resolveSymlinks(SymlinkResolution::Ancestors),
settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
path.baseName(),
computeBaseName(path),
ContentAddressMethod::Raw::NixArchive,
nullptr,
repair);
@@ -2496,7 +2499,7 @@ SourcePath EvalState::coerceToPath(const PosIdx pos, Value & v, NixStringContext
}
}
/* Any other value should be coercible to a string, interpreted
/* Any other value should be coercable to a string, interpreted
relative to the root filesystem. */
auto path = coerceToString(pos, v, context, errorCtx, false, false, true).toOwned();
if (path == "" || path[0] != '/')
@@ -2510,7 +2513,7 @@ StorePath EvalState::coerceToStorePath(const PosIdx pos, Value & v, NixStringCon
auto path = coerceToString(pos, v, context, errorCtx, false, false, true).toOwned();
if (auto storePath = store->maybeParseStorePath(path))
return *storePath;
error<EvalError>("path '%1%' is not in the Nix store", path).withTrace(pos, errorCtx).debugThrow();
error<EvalError>("cannot coerce '%s' to a store path because it is not a subpath of the Nix store", path).withTrace(pos, errorCtx).debugThrow();
}
@@ -2536,6 +2539,11 @@ std::pair<SingleDerivedPath, std::string_view> EvalState::coerceToSingleDerivedP
[&](NixStringContextElem::Built && b) -> SingleDerivedPath {
return std::move(b);
},
[&](NixStringContextElem::Path && p) -> SingleDerivedPath {
error<EvalError>(
"string '%s' has no context",
s).withTrace(pos, errorCtx).debugThrow();
},
}, ((NixStringContextElem &&) *context.begin()).raw);
return {
std::move(derivedPath),
@@ -2637,14 +2645,14 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st
return;
case nPath:
if (v1.pathAccessor() != v2.pathAccessor()) {
if (v1.payload.path.accessor != v2.payload.path.accessor) {
error<AssertionError>(
"path '%s' is not equal to path '%s' because their accessors are different",
ValuePrinter(*this, v1, errorPrintOptions),
ValuePrinter(*this, v2, errorPrintOptions))
.debugThrow();
}
if (strcmp(v1.pathStr(), v2.pathStr()) != 0) {
if (strcmp(v1.payload.path.path, v2.payload.path.path) != 0) {
error<AssertionError>(
"path '%s' is not equal to path '%s'",
ValuePrinter(*this, v1, errorPrintOptions),
@@ -2668,7 +2676,7 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st
}
for (size_t n = 0; n < v1.listSize(); ++n) {
try {
assertEqValues(*v1.listView()[n], *v2.listView()[n], pos, errorCtx);
assertEqValues(*v1.listElems()[n], *v2.listElems()[n], pos, errorCtx);
} catch (Error & e) {
e.addTrace(positions[pos], "while comparing list element %d", n);
throw;
@@ -2811,8 +2819,8 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v
case nPath:
return
// FIXME: compare accessors by their fingerprint.
v1.pathAccessor() == v2.pathAccessor()
&& strcmp(v1.pathStr(), v2.pathStr()) == 0;
v1.payload.path.accessor == v2.payload.path.accessor
&& strcmp(v1.payload.path.path, v2.payload.path.path) == 0;
case nNull:
return true;
@@ -2820,7 +2828,7 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v
case nList:
if (v1.listSize() != v2.listSize()) return false;
for (size_t n = 0; n < v1.listSize(); ++n)
if (!eqValues(*v1.listView()[n], *v2.listView()[n], pos, errorCtx)) return false;
if (!eqValues(*v1.listElems()[n], *v2.listElems()[n], pos, errorCtx)) return false;
return true;
case nAttrs: {
@@ -2867,7 +2875,7 @@ bool EvalState::fullGC() {
GC_gcollect();
// Check that it ran. We might replace this with a version that uses more
// of the boehm API to get this reliably, at a maintenance cost.
// We use a 1K margin because technically this has a race condition, but we
// We use a 1K margin because technically this has a race condtion, but we
// probably won't encounter it in practice, because the CLI isn't concurrent
// like that.
return GC_get_bytes_since_gc() < 1024;
@@ -3020,7 +3028,7 @@ void EvalState::printStatistics()
// XXX: overrides earlier assignment
topObj["symbols"] = json::array();
auto &list = topObj["symbols"];
symbols.dump([&](std::string_view s) { list.emplace_back(s); });
symbols.dump([&](const std::string & s) { list.emplace_back(s); });
}
if (outPath == "-") {
std::cerr << topObj.dump(2) << std::endl;
@@ -3119,6 +3127,11 @@ SourcePath EvalState::findFile(const LookupPath & lookupPath, const std::string_
auto res = (r / CanonPath(suffix)).resolveSymlinks();
if (res.pathExists()) return res;
// Backward compatibility hack: throw an exception if access
// to this path is not allowed.
if (auto accessor = res.accessor.dynamic_pointer_cast<FilteringSourceAccessor>())
accessor->checkAccess(res.path);
}
if (hasPrefix(path, "nix/"))
@@ -3189,6 +3202,11 @@ std::optional<SourcePath> EvalState::resolveLookupPathPath(const LookupPath::Pat
if (path.resolveSymlinks().pathExists())
return finish(std::move(path));
else {
// Backward compatibility hack: throw an exception if access
// to this path is not allowed.
if (auto accessor = path.accessor.dynamic_pointer_cast<FilteringSourceAccessor>())
accessor->checkAccess(path.path);
logWarning({
.msg = HintFmt("Nix search path entry '%1%' does not exist, ignoring", value)
});
+2 -2
View File
@@ -4,7 +4,7 @@
namespace nix {
void FunctionCallTrace::preFunctionCallHook(
EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
auto duration = std::chrono::high_resolution_clock::now().time_since_epoch();
auto ns = std::chrono::duration_cast<std::chrono::nanoseconds>(duration);
@@ -12,7 +12,7 @@ void FunctionCallTrace::preFunctionCallHook(
}
void FunctionCallTrace::postFunctionCallHook(
EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos)
{
auto duration = std::chrono::high_resolution_clock::now().time_since_epoch();
auto ns = std::chrono::duration_cast<std::chrono::nanoseconds>(duration);
+4 -5
View File
@@ -117,7 +117,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
state->forceList(*i->value, i->pos, "while evaluating the 'outputs' attribute of a derivation");
/* For each output... */
for (auto elem : i->value->listView()) {
for (auto elem : i->value->listItems()) {
std::string output(state->forceStringNoCtx(*elem, i->pos, "while evaluating the name of an output of a derivation"));
if (withPaths) {
@@ -159,7 +159,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
/* ^ this shows during `nix-env -i` right under the bad derivation */
if (!outTI->isList()) throw errMsg;
Outputs result;
for (auto elem : outTI->listView()) {
for (auto elem : outTI->listItems()) {
if (elem->type() != nString) throw errMsg;
auto out = outputs.find(elem->c_str());
if (out == outputs.end()) throw errMsg;
@@ -206,7 +206,7 @@ bool PackageInfo::checkMeta(Value & v)
{
state->forceValue(v, v.determinePos(noPos));
if (v.type() == nList) {
for (auto elem : v.listView())
for (auto elem : v.listItems())
if (!checkMeta(*elem)) return false;
return true;
}
@@ -400,8 +400,7 @@ static void getDerivations(EvalState & state, Value & vIn,
}
else if (v.type() == nList) {
auto listView = v.listView();
for (auto [n, elem] : enumerate(listView)) {
for (auto [n, elem] : enumerate(v.listItems())) {
std::string pathPrefix2 = addToPath(pathPrefix, fmt("%d", n));
if (getDerivation(state, *elem, pathPrefix2, drvs, done, ignoreAssertionFailures))
getDerivations(state, *elem, pathPrefix2, autoArgs, drvs, done, ignoreAssertionFailures);
+3 -3
View File
@@ -89,9 +89,9 @@ Env & EvalState::allocEnv(size_t size)
void EvalState::forceValue(Value & v, const PosIdx pos)
{
if (v.isThunk()) {
Env * env = v.thunk().env;
Env * env = v.payload.thunk.env;
assert(env || v.isBlackhole());
Expr * expr = v.thunk().expr;
Expr * expr = v.payload.thunk.expr;
try {
v.mkBlackhole();
//checkInterrupt();
@@ -106,7 +106,7 @@ void EvalState::forceValue(Value & v, const PosIdx pos)
}
}
else if (v.isApp())
callFunction(*v.app().left, *v.app().right, v, pos);
callFunction(*v.payload.app.left, *v.payload.app.right, v, pos);
}
@@ -1,16 +0,0 @@
#pragma once
///@file
#include "nix/util/configuration.hh"
namespace nix {
enum struct EvalProfilerMode { disabled, flamegraph };
template<>
EvalProfilerMode BaseSetting<EvalProfilerMode>::parse(const std::string & str) const;
template<>
std::string BaseSetting<EvalProfilerMode>::to_string() const;
}
@@ -11,7 +11,6 @@
#include <span>
#include <bitset>
#include <optional>
#include <filesystem>
namespace nix {
@@ -62,7 +61,8 @@ public:
* @param args Function arguments.
* @param pos Function position.
*/
virtual void preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos);
virtual void
preFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos);
/**
* Hook called on EvalState::callFunction exit.
@@ -73,7 +73,8 @@ public:
* @param args Function arguments.
* @param pos Function position.
*/
virtual void postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos);
virtual void
postFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos);
virtual ~EvalProfiler() = default;
@@ -104,11 +105,9 @@ public:
void addProfiler(ref<EvalProfiler> profiler);
[[gnu::noinline]] void
preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
preFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
[[gnu::noinline]] void
postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
postFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
};
ref<EvalProfiler> makeSampleStackProfiler(EvalState & state, std::filesystem::path profileFile, uint64_t frequency);
}
+15 -44
View File
@@ -1,7 +1,6 @@
#pragma once
///@file
#include "nix/expr/eval-profiler-settings.hh"
#include "nix/util/configuration.hh"
#include "nix/util/source-path.hh"
@@ -13,7 +12,7 @@ struct PrimOp;
struct EvalSettings : Config
{
/**
* Function used to interpret look path entries of a given scheme.
* Function used to interpet look path entries of a given scheme.
*
* The argument is the non-scheme part of the lookup path entry (see
* `LookupPathHooks` below).
@@ -132,7 +131,7 @@ struct EvalSettings : Config
Setting<bool> restrictEval{
this, false, "restrict-eval",
R"(
If set to `true`, the Nix evaluator doesn't allow access to any
If set to `true`, the Nix evaluator will not allow access to any
files outside of
[`builtins.nixPath`](@docroot@/language/builtins.md#builtins-nixPath),
or to URIs outside of
@@ -152,24 +151,14 @@ struct EvalSettings : Config
)"
};
Setting<bool> traceImportFromDerivation{
this, false, "trace-import-from-derivation",
R"(
By default, Nix allows [Import from Derivation](@docroot@/language/import-from-derivation.md).
When this setting is `true`, Nix logs a warning indicating that it performed such an import.
This option has no effect if `allow-import-from-derivation` is disabled.
)"
};
Setting<bool> enableImportFromDerivation{
this, true, "allow-import-from-derivation",
R"(
By default, Nix allows [Import from Derivation](@docroot@/language/import-from-derivation.md).
With this option set to `false`, Nix throws an error when evaluating an expression that uses this feature,
With this option set to `false`, Nix will throw an error when evaluating an expression that uses this feature,
even when the required store object is readily available.
This ensures that evaluation doesn't require any builds to take place,
This ensures that evaluation will not require any builds to take place,
regardless of the state of the store.
)"};
@@ -188,8 +177,8 @@ struct EvalSettings : Config
Setting<bool> traceFunctionCalls{this, false, "trace-function-calls",
R"(
If set to `true`, the Nix evaluator traces every function call.
Nix prints a log message at the "vomit" level for every function
If set to `true`, the Nix evaluator will trace every function call.
Nix will print a log message at the "vomit" level for every function
entrance and function exit.
function-trace entered undefined position at 1565795816999559622
@@ -204,29 +193,6 @@ struct EvalSettings : Config
`flamegraph.pl`.
)"};
Setting<EvalProfilerMode> evalProfilerMode{this, EvalProfilerMode::disabled, "eval-profiler",
R"(
Enables evaluation profiling. The following modes are supported:
* `flamegraph` stack sampling profiler. Outputs folded format, one line per stack (suitable for `flamegraph.pl` and compatible tools).
Use [`eval-profile-file`](#conf-eval-profile-file) to specify where the profile is saved.
See [Using the `eval-profiler`](@docroot@/advanced-topics/eval-profiler.md).
)"};
Setting<Path> evalProfileFile{this, "nix.profile", "eval-profile-file",
R"(
Specifies the file where [evaluation profile](#conf-eval-profiler) is saved.
)"};
Setting<uint32_t> evalProfilerFrequency{this, 99, "eval-profiler-frequency",
R"(
Specifies the sampling rate in hertz for sampling evaluation profilers.
Use `0` to sample the stack after each function call.
See [`eval-profiler`](#conf-eval-profiler).
)"};
Setting<bool> useEvalCache{this, true, "eval-cache",
R"(
Whether to use the flake evaluation cache.
@@ -236,8 +202,8 @@ struct EvalSettings : Config
Setting<bool> ignoreExceptionsDuringTry{this, false, "ignore-try",
R"(
If set to true, ignore exceptions inside 'tryEval' calls when evaluating Nix expressions in
debug mode (using the --debugger flag). By default the debugger pauses on all exceptions.
If set to true, ignore exceptions inside 'tryEval' calls when evaluating nix expressions in
debug mode (using the --debugger flag). By default the debugger will pause on all exceptions.
)"};
Setting<bool> traceVerbose{this, false, "trace-verbose",
@@ -249,7 +215,7 @@ struct EvalSettings : Config
Setting<bool> builtinsTraceDebugger{this, false, "debugger-on-trace",
R"(
If set to true and the `--debugger` flag is given, the following functions
enter the debugger like [`builtins.break`](@docroot@/language/builtins.md#builtins-break):
will enter the debugger like [`builtins.break`](@docroot@/language/builtins.md#builtins-break).
* [`builtins.trace`](@docroot@/language/builtins.md#builtins-trace)
* [`builtins.traceVerbose`](@docroot@/language/builtins.md#builtins-traceVerbose)
@@ -271,7 +237,7 @@ struct EvalSettings : Config
Setting<bool> builtinsAbortOnWarn{this, false, "abort-on-warn",
R"(
If set to true, [`builtins.warn`](@docroot@/language/builtins.md#builtins-warn) throws an error when logging a warning.
If set to true, [`builtins.warn`](@docroot@/language/builtins.md#builtins-warn) will throw an error when logging a warning.
This will give you a stack trace that leads to the location of the warning.
@@ -281,6 +247,11 @@ struct EvalSettings : Config
This option can be enabled by setting `NIX_ABORT_ON_WARN=1` in the environment.
)"};
Setting<bool> lazyTrees{this, false, "lazy-trees",
R"(
If set to true, flakes and trees fetched by [`builtins.fetchTree`](@docroot@/language/builtins.md#builtins-fetchTree) are only copied to the Nix store when they're used as a dependency of a derivation. This avoids copying (potentially large) source trees unnecessarily.
)"};
};
/**
+39 -8
View File
@@ -37,6 +37,7 @@ class Store;
namespace fetchers {
struct Settings;
struct InputCache;
struct Input;
}
struct EvalSettings;
class EvalState;
@@ -44,6 +45,7 @@ class StorePath;
struct SingleDerivedPath;
enum RepairFlag : bool;
struct MemorySourceAccessor;
struct MountedSourceAccessor;
namespace eval_cache {
class EvalCache;
}
@@ -213,7 +215,7 @@ public:
const Symbol sWith, sOutPath, sDrvPath, sType, sMeta, sName, sValue,
sSystem, sOverrides, sOutputs, sOutputName, sIgnoreNulls,
sFile, sLine, sColumn, sFunctor, sToString,
sRight, sWrong, sStructuredAttrs, sJson,
sRight, sWrong, sStructuredAttrs,
sAllowedReferences, sAllowedRequisites, sDisallowedReferences, sDisallowedRequisites,
sMaxSize, sMaxClosureSize,
sBuilder, sArgs,
@@ -272,7 +274,7 @@ public:
/**
* The accessor corresponding to `store`.
*/
const ref<SourceAccessor> storeFS;
const ref<MountedSourceAccessor> storeFS;
/**
* The accessor for the root filesystem.
@@ -450,6 +452,15 @@ public:
void checkURI(const std::string & uri);
/**
* Mount an input on the Nix store.
*/
StorePath mountInput(
fetchers::Input & input,
const fetchers::Input & originalInput,
ref<SourceAccessor> accessor,
bool requireLockable);
/**
* Parse a Nix expression from the specified file.
*/
@@ -542,11 +553,6 @@ public:
std::string_view forceString(Value & v, NixStringContext & context, const PosIdx pos, std::string_view errorCtx, const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
std::string_view forceStringNoCtx(Value & v, const PosIdx pos, std::string_view errorCtx);
/**
* Get attribute from an attribute set and throw an error if it doesn't exist.
*/
Bindings::const_iterator getAttr(Symbol attrSym, const Bindings * attrSet, std::string_view errorCtx);
template<typename... Args>
[[gnu::noinline]]
void addErrorTrace(Error & e, const Args & ... formatArgs) const;
@@ -564,6 +570,18 @@ public:
std::optional<std::string> tryAttrsToString(const PosIdx pos, Value & v,
NixStringContext & context, bool coerceMore = false, bool copyToStore = true);
StorePath devirtualize(
const StorePath & path,
StringMap * rewrites = nullptr);
SingleDerivedPath devirtualize(
const SingleDerivedPath & path,
StringMap * rewrites = nullptr);
std::string devirtualize(
std::string_view s,
const NixStringContext & context);
/**
* String coercion.
*
@@ -579,6 +597,19 @@ public:
StorePath copyPathToStore(NixStringContext & context, const SourcePath & path);
/**
* Compute the base name for a `SourcePath`. For non-store paths,
* this is just `SourcePath::baseName()`. But for store paths, for
* backwards compatibility, it needs to be `<hash>-source`,
* i.e. as if the path were copied to the Nix store. This results
* in a "double-copied" store path like
* `/nix/store/<hash1>-<hash2>-source`. We don't need to
* materialize /nix/store/<hash2>-source though. Still, this
* requires reading/hashing the path twice.
*/
std::string computeBaseName(const SourcePath & path);
/**
* Path coercion.
*
@@ -736,7 +767,7 @@ public:
*/
void assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::string_view errorCtx);
bool isFunctor(const Value & fun) const;
bool isFunctor(Value & fun);
void callFunction(Value & fun, std::span<Value *> args, Value & vRes, const PosIdx pos);
@@ -17,9 +17,9 @@ public:
FunctionCallTrace() = default;
[[gnu::noinline]] void
preFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
preFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
[[gnu::noinline]] void
postFunctionCallHook(EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
postFunctionCallHook(const EvalState & state, const Value & v, std::span<Value *> args, const PosIdx pos) override;
};
}
-1
View File
@@ -14,7 +14,6 @@ headers = [config_pub_h] + files(
'eval-error.hh',
'eval-gc.hh',
'eval-inline.hh',
'eval-profiler-settings.hh',
'eval-profiler.hh',
'eval-settings.hh',
'eval.hh',
-3
View File
@@ -306,9 +306,6 @@ struct Formal
struct Formals
{
typedef std::vector<Formal> Formals_;
/**
* @pre Sorted according to predicate (std::tie(a.name, a.pos) < std::tie(b.name, b.pos)).
*/
Formals_ formals;
bool ellipsis;
@@ -1,7 +1,6 @@
#pragma once
#include "nix/expr/value.hh"
#include "nix/expr/symbol-table.hh"
namespace nix {
@@ -16,10 +15,10 @@ namespace nix {
* See: https://github.com/NixOS/nix/issues/9730
*/
void printAmbiguous(
Value &v,
const SymbolTable &symbols,
std::ostream &str,
std::set<const void *> *seen,
EvalState & state,
Value & v,
std::ostream & str,
std::set<const void *> * seen,
int depth);
}
+45 -187
View File
@@ -1,65 +1,16 @@
#pragma once
///@file
#include <memory_resource>
#include "nix/expr/value.hh"
#include <list>
#include <map>
#include <unordered_map>
#include "nix/util/types.hh"
#include "nix/util/chunked-vector.hh"
#include "nix/util/error.hh"
#include <boost/version.hpp>
#define USE_FLAT_SYMBOL_SET (BOOST_VERSION >= 108100)
#if USE_FLAT_SYMBOL_SET
# include <boost/unordered/unordered_flat_set.hpp>
#else
# include <boost/unordered/unordered_set.hpp>
#endif
namespace nix {
class SymbolValue : protected Value
{
friend class SymbolStr;
friend class SymbolTable;
uint32_t size_;
uint32_t idx;
SymbolValue() = default;
public:
operator std::string_view() const noexcept
{
return {c_str(), size_};
}
};
/**
* Symbols have the property that they can be compared efficiently
* (using an equality test), because the symbol table stores only one
* copy of each string.
*/
class Symbol
{
friend class SymbolStr;
friend class SymbolTable;
private:
uint32_t id;
explicit Symbol(uint32_t id) noexcept : id(id) {}
public:
Symbol() noexcept : id(0) {}
[[gnu::always_inline]]
explicit operator bool() const noexcept { return id > 0; }
auto operator<=>(const Symbol other) const noexcept { return id <=> other.id; }
bool operator==(const Symbol other) const noexcept { return id == other.id; }
friend class std::hash<Symbol>;
};
/**
* This class mainly exists to give us an operator<< for ostreams. We could also
* return plain strings from SymbolTable, but then we'd have to wrap every
@@ -69,130 +20,58 @@ class SymbolStr
{
friend class SymbolTable;
constexpr static size_t chunkSize{8192};
using SymbolValueStore = ChunkedVector<SymbolValue, chunkSize>;
private:
const std::string * s;
const SymbolValue * s;
struct Key
{
using HashType = boost::hash<std::string_view>;
SymbolValueStore & store;
std::string_view s;
std::size_t hash;
std::pmr::polymorphic_allocator<char> & alloc;
Key(SymbolValueStore & store, std::string_view s, std::pmr::polymorphic_allocator<char> & stringAlloc)
: store(store)
, s(s)
, hash(HashType{}(s))
, alloc(stringAlloc) {}
};
explicit SymbolStr(const std::string & symbol): s(&symbol) {}
public:
SymbolStr(const SymbolValue & s) noexcept : s(&s) {}
SymbolStr(const Key & key)
{
auto size = key.s.size();
if (size >= std::numeric_limits<uint32_t>::max()) {
throw Error("Size of symbol exceeds 4GiB and cannot be stored");
}
// for multi-threaded implementations: lock store and allocator here
const auto & [v, idx] = key.store.add(SymbolValue{});
if (size == 0) {
v.mkString("", nullptr);
} else {
auto s = key.alloc.allocate(size + 1);
memcpy(s, key.s.data(), size);
s[size] = '\0';
v.mkString(s, nullptr);
}
v.size_ = size;
v.idx = idx;
this->s = &v;
}
bool operator == (std::string_view s2) const noexcept
bool operator == (std::string_view s2) const
{
return *s == s2;
}
[[gnu::always_inline]]
const char * c_str() const noexcept
const char * c_str() const
{
return s->c_str();
}
[[gnu::always_inline]]
operator std::string_view () const noexcept
operator const std::string_view () const
{
return *s;
}
friend std::ostream & operator <<(std::ostream & os, const SymbolStr & symbol);
[[gnu::always_inline]]
bool empty() const noexcept
bool empty() const
{
return s->size_ == 0;
return s->empty();
}
};
[[gnu::always_inline]]
size_t size() const noexcept
{
return s->size_;
}
/**
* Symbols have the property that they can be compared efficiently
* (using an equality test), because the symbol table stores only one
* copy of each string.
*/
class Symbol
{
friend class SymbolTable;
[[gnu::always_inline]]
const Value * valuePtr() const noexcept
{
return s;
}
private:
uint32_t id;
explicit operator Symbol() const noexcept
{
return Symbol{s->idx + 1};
}
explicit Symbol(uint32_t id): id(id) {}
struct Hash
{
using is_transparent = void;
using is_avalanching = std::true_type;
public:
Symbol() : id(0) {}
std::size_t operator()(SymbolStr str) const
{
return Key::HashType{}(*str.s);
}
explicit operator bool() const { return id > 0; }
std::size_t operator()(const Key & key) const noexcept
{
return key.hash;
}
};
auto operator<=>(const Symbol other) const { return id <=> other.id; }
bool operator==(const Symbol other) const { return id == other.id; }
struct Equal
{
using is_transparent = void;
bool operator()(SymbolStr a, SymbolStr b) const noexcept
{
// strings are unique, so that a pointer comparison is OK
return a.s == b.s;
}
bool operator()(SymbolStr a, const Key & b) const noexcept
{
return a == b.s;
}
[[gnu::always_inline]]
bool operator()(const Key & a, SymbolStr b) const noexcept
{
return operator()(b, a);
}
};
friend class std::hash<Symbol>;
};
/**
@@ -203,46 +82,29 @@ class SymbolTable
{
private:
/**
* SymbolTable is an append only data structure.
* During its lifetime the monotonic buffer holds all strings and nodes, if the symbol set is node based.
*/
std::pmr::monotonic_buffer_resource buffer;
std::pmr::polymorphic_allocator<char> stringAlloc{&buffer};
SymbolStr::SymbolValueStore store{16};
/**
* Transparent lookup of string view for a pointer to a ChunkedVector entry -> return offset into the store.
* Map from string view (backed by ChunkedVector) -> offset into the store.
* ChunkedVector references are never invalidated.
*/
#if USE_FLAT_SYMBOL_SET
boost::unordered_flat_set<SymbolStr, SymbolStr::Hash, SymbolStr::Equal> symbols{SymbolStr::chunkSize};
#else
using SymbolValueAlloc = std::pmr::polymorphic_allocator<SymbolStr>;
boost::unordered_set<SymbolStr, SymbolStr::Hash, SymbolStr::Equal, SymbolValueAlloc> symbols{SymbolStr::chunkSize, {&buffer}};
#endif
std::unordered_map<std::string_view, uint32_t> symbols;
ChunkedVector<std::string, 8192> store{16};
public:
/**
* Converts a string into a symbol.
*/
Symbol create(std::string_view s) {
Symbol create(std::string_view s)
{
// Most symbols are looked up more than once, so we trade off insertion performance
// for lookup performance.
// FIXME: make this thread-safe.
return [&]<typename T>(T && key) -> Symbol {
if constexpr (requires { symbols.insert<T>(key); }) {
auto [it, _] = symbols.insert<T>(key);
return Symbol(*it);
} else {
auto it = symbols.find<T>(key);
if (it != symbols.end())
return Symbol(*it);
auto it = symbols.find(s);
if (it != symbols.end())
return Symbol(it->second + 1);
it = symbols.emplace(key).first;
return Symbol(*it);
}
}(SymbolStr::Key{store, s, stringAlloc});
const auto & [rawSym, idx] = store.add(s);
symbols.emplace(rawSym, idx);
return Symbol(idx + 1);
}
std::vector<SymbolStr> resolve(const std::vector<Symbol> & symbols) const
@@ -256,14 +118,12 @@ public:
SymbolStr operator[](Symbol s) const
{
uint32_t idx = s.id - uint32_t(1);
if (idx >= store.size())
if (s.id == 0 || s.id > store.size())
unreachable();
return store[idx];
return SymbolStr(store[s.id - 1]);
}
[[gnu::always_inline]]
size_t size() const noexcept
size_t size() const
{
return store.size();
}
@@ -287,5 +147,3 @@ struct std::hash<nix::Symbol>
return std::hash<decltype(s.id)>{}(s.id);
}
};
#undef USE_FLAT_SYMBOL_SET
File diff suppressed because it is too large. Load diff
+32 -1
View File
@@ -54,10 +54,35 @@ struct NixStringContextElem {
*/
using Built = SingleDerivedPath::Built;
/**
* A store path that will not result in a store reference when
* used in a derivation or toFile.
*
* When you apply `builtins.toString` to a path value representing
* a path in the Nix store (as is the case with flake inputs),
* historically you got a string without context
* (e.g. `/nix/store/...-source`). This is broken, since it allows
* you to pass a store path to a derivation/toFile without a
* proper store reference. This is especially a problem with lazy
* trees, since the store path is a virtual path that doesn't
* exist.
*
* For backwards compatibility, and to warn users about this
* unsafe use of `toString`, we keep track of such strings as a
* special type of context.
*/
struct Path
{
StorePath storePath;
GENERATE_CMP(Path, me->storePath);
};
using Raw = std::variant<
Opaque,
DrvDeep,
Built
Built,
Path
>;
Raw raw;
@@ -82,4 +107,10 @@ struct NixStringContextElem {
typedef std::set<NixStringContextElem> NixStringContext;
/**
* Returns false if `context` has no elements other than
* `NixStringContextElem::Path`.
*/
bool hasContext(const NixStringContext & context);
}
+1 -1
View File
@@ -2,7 +2,7 @@
#include <cstddef>
// including the generated headers twice leads to errors
// inluding the generated headers twice leads to errors
#ifndef BISON_HEADER
# include "lexer-tab.hh"
# include "parser-tab.hh"
-1
View File
@@ -140,7 +140,6 @@ sources = files(
'eval-cache.cc',
'eval-error.cc',
'eval-gc.cc',
'eval-profiler-settings.cc',
'eval-profiler.cc',
'eval-settings.cc',
'eval.cc',
+1 -1
View File
@@ -606,7 +606,7 @@ void ExprLambda::setDocComment(DocComment docComment) {
size_t SymbolTable::totalSize() const
{
size_t n = 0;
dump([&] (SymbolStr s) { n += s.size(); });
dump([&] (const std::string & s) { n += s.size(); });
return n;
}
+85
View File
@@ -1,5 +1,7 @@
#include "nix/store/store-api.hh"
#include "nix/expr/eval.hh"
#include "nix/util/mounted-source-accessor.hh"
#include "nix/fetchers/fetch-to-store.hh"
namespace nix {
@@ -18,4 +20,87 @@ SourcePath EvalState::storePath(const StorePath & path)
return {rootFS, CanonPath{store->printStorePath(path)}};
}
StorePath EvalState::devirtualize(const StorePath & path, StringMap * rewrites)
{
if (auto mount = storeFS->getMount(CanonPath(store->printStorePath(path)))) {
auto storePath = fetchToStore(
fetchSettings,
*store,
SourcePath{ref(mount)},
settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
path.name());
assert(storePath.name() == path.name());
if (rewrites)
rewrites->emplace(path.hashPart(), storePath.hashPart());
return storePath;
} else
return path;
}
SingleDerivedPath EvalState::devirtualize(const SingleDerivedPath & path, StringMap * rewrites)
{
if (auto o = std::get_if<SingleDerivedPath::Opaque>(&path.raw()))
return SingleDerivedPath::Opaque{devirtualize(o->path, rewrites)};
else
return path;
}
std::string EvalState::devirtualize(std::string_view s, const NixStringContext & context)
{
StringMap rewrites;
for (auto & c : context)
if (auto o = std::get_if<NixStringContextElem::Opaque>(&c.raw))
devirtualize(o->path, &rewrites);
return rewriteStrings(std::string(s), rewrites);
}
std::string EvalState::computeBaseName(const SourcePath & path)
{
if (path.accessor == rootFS) {
if (auto storePath = store->maybeParseStorePath(path.path.abs())) {
warn(
"Performing inefficient double copy of path '%s' to the store. "
"This can typically be avoided by rewriting an attribute like `src = ./.` "
"to `src = builtins.path { path = ./.; name = \"source\"; }`.",
path);
return std::string(
fetchToStore(fetchSettings, *store, path, FetchMode::DryRun, storePath->name()).to_string());
}
}
return std::string(path.baseName());
}
StorePath EvalState::mountInput(
fetchers::Input & input, const fetchers::Input & originalInput, ref<SourceAccessor> accessor, bool requireLockable)
{
auto storePath = settings.lazyTrees
? StorePath::random(input.getName())
: fetchToStore(fetchSettings, *store, accessor, FetchMode::Copy, input.getName());
allowPath(storePath); // FIXME: should just whitelist the entire virtual store
storeFS->mount(CanonPath(store->printStorePath(storePath)), accessor);
if (requireLockable && (!settings.lazyTrees || !input.isLocked()) && !input.getNarHash()) {
auto narHash = accessor->hashPath(CanonPath::root);
input.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
}
// FIXME: what to do with the NAR hash in lazy mode?
if (!settings.lazyTrees && originalInput.getNarHash()) {
auto expected = originalInput.computeStorePath(*store);
if (storePath != expected)
throw Error(
(unsigned int) 102,
"NAR hash mismatch in input '%s', expected '%s' but got '%s'",
originalInput.to_string(),
store->printStorePath(storePath),
store->printStorePath(expected));
}
return storePath;
}
}
+188 -209
View File
@@ -14,7 +14,7 @@
#include "nix/expr/value-to-xml.hh"
#include "nix/expr/primops.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/util/sort.hh"
#include "nix/util/mounted-source-accessor.hh"
#include <boost/container/small_vector.hpp>
#include <nlohmann/json.hpp>
@@ -76,7 +76,10 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
ensureValid(b.drvPath->getBaseStorePath());
},
[&](const NixStringContextElem::Opaque & o) {
ensureValid(o.path);
// We consider virtual store paths valid here. They'll
// be devirtualized if needed elsewhere.
if (!storeFS->getMount(CanonPath(store->printStorePath(o.path))))
ensureValid(o.path);
if (maybePathsOut)
maybePathsOut->emplace(o.path);
},
@@ -86,24 +89,19 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
if (maybePathsOut)
maybePathsOut->emplace(d.drvPath);
},
[&](const NixStringContextElem::Path & p) {
// FIXME: do something?
},
}, c.raw);
}
if (drvs.empty()) return {};
if (isIFD) {
if (!settings.enableImportFromDerivation)
error<IFDError>(
"cannot build '%1%' during evaluation because the option 'allow-import-from-derivation' is disabled",
drvs.begin()->to_string(*store)
).debugThrow();
if (settings.traceImportFromDerivation)
warn(
"built '%1%' during evaluation due to an import from derivation",
drvs.begin()->to_string(*store)
);
}
if (isIFD && !settings.enableImportFromDerivation)
error<IFDError>(
"cannot build '%1%' during evaluation because the option 'allow-import-from-derivation' is disabled",
drvs.begin()->to_string(*store)
).debugThrow();
/* Build/substitute the context. */
std::vector<DerivedPath> buildReqs;
@@ -346,7 +344,7 @@ static RegisterPrimOp primop_import({
> }
> ```
>
> then the following `foo.nix` throws an error:
> then the following `foo.nix` will give an error:
>
> ```nix
> # foo.nix
@@ -415,7 +413,7 @@ void prim_importNative(EvalState & state, const PosIdx pos, Value * * args, Valu
void prim_exec(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.exec");
auto elems = args[0]->listView();
auto elems = args[0]->listElems();
auto count = args[0]->listSize();
if (count == 0)
state.error<EvalError>("at least one argument to 'exec' required").atPos(pos).debugThrow();
@@ -424,7 +422,7 @@ void prim_exec(EvalState & state, const PosIdx pos, Value * * args, Value & v)
"while evaluating the first element of the argument passed to builtins.exec",
false, false).toOwned();
Strings commandArgs;
for (size_t i = 1; i < count; ++i) {
for (unsigned int i = 1; i < args[0]->listSize(); ++i) {
commandArgs.push_back(
state.coerceToString(pos, *elems[i], context,
"while evaluating an element of the argument passed to builtins.exec",
@@ -652,7 +650,7 @@ struct CompareValues
// Note: we don't take the accessor into account
// since it's not obvious how to compare them in a
// reproducible way.
return strcmp(v1->pathStr(), v2->pathStr()) < 0;
return strcmp(v1->payload.path.path, v2->payload.path.path) < 0;
case nList:
// Lexicographic comparison
for (size_t i = 0;; i++) {
@@ -660,8 +658,8 @@ struct CompareValues
return false;
} else if (i == v1->listSize()) {
return true;
} else if (!state.eqValues(*v1->listView()[i], *v2->listView()[i], pos, errorCtx)) {
return (*this)(v1->listView()[i], v2->listView()[i], "while comparing two list elements");
} else if (!state.eqValues(*v1->listElems()[i], *v2->listElems()[i], pos, errorCtx)) {
return (*this)(v1->listElems()[i], v2->listElems()[i], "while comparing two list elements");
}
}
default:
@@ -679,17 +677,31 @@ struct CompareValues
typedef std::list<Value *, gc_allocator<Value *>> ValueList;
static Bindings::const_iterator getAttr(
EvalState & state,
Symbol attrSym,
const Bindings * attrSet,
std::string_view errorCtx)
{
auto value = attrSet->find(attrSym);
if (value == attrSet->end()) {
state.error<TypeError>("attribute '%s' missing", state.symbols[attrSym]).withTrace(noPos, errorCtx).debugThrow();
}
return value;
}
static void prim_genericClosure(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
state.forceAttrs(*args[0], noPos, "while evaluating the first argument passed to builtins.genericClosure");
/* Get the start set. */
auto startSet = state.getAttr(state.sStartSet, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
auto startSet = getAttr(state, state.sStartSet, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
state.forceList(*startSet->value, noPos, "while evaluating the 'startSet' attribute passed as argument to builtins.genericClosure");
ValueList workSet;
for (auto elem : startSet->value->listView())
for (auto elem : startSet->value->listItems())
workSet.push_back(elem);
if (startSet->value->listSize() == 0) {
@@ -698,7 +710,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value * * a
}
/* Get the operator. */
auto op = state.getAttr(state.sOperator, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
auto op = getAttr(state, state.sOperator, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
state.forceFunction(*op->value, noPos, "while evaluating the 'operator' attribute passed as argument to builtins.genericClosure");
/* Construct the closure by applying the operator to elements of
@@ -715,7 +727,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value * * a
state.forceAttrs(*e, noPos, "while evaluating one of the elements generated by (or initially passed to) builtins.genericClosure");
auto key = state.getAttr(state.sKey, e->attrs(), "in one of the attrsets generated by (or initially passed to) builtins.genericClosure");
auto key = getAttr(state, state.sKey, e->attrs(), "in one of the attrsets generated by (or initially passed to) builtins.genericClosure");
state.forceValue(*key->value, noPos);
if (!doneKeys.insert(key->value).second) continue;
@@ -727,7 +739,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value * * a
state.forceList(newElements, noPos, "while evaluating the return value of the `operator` passed to builtins.genericClosure");
/* Add the values returned by the operator to the work set. */
for (auto elem : newElements.listView()) {
for (auto elem : newElements.listItems()) {
state.forceValue(*elem, noPos); // "while evaluating one one of the elements returned by the `operator` passed to builtins.genericClosure");
workSet.push_back(elem);
}
@@ -899,7 +911,7 @@ static void prim_ceil(EvalState & state, const PosIdx pos, Value * * args, Value
auto arg = args[0]->integer();
auto res = v.integer();
if (arg != res) {
state.error<EvalError>("Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occurred in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.", arg, res).atPos(pos).debugThrow();
state.error<EvalError>("Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occured in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.", arg, res).atPos(pos).debugThrow();
}
}
}
@@ -916,7 +928,7 @@ static RegisterPrimOp primop_ceil({
a NixInt and if `*number* < -9007199254740992` or `*number* > 9007199254740992`.
If the datatype of *number* is neither a NixInt (signed 64-bit integer) nor a NixFloat
(IEEE-754 double-precision floating-point number), an evaluation error is thrown.
(IEEE-754 double-precision floating-point number), an evaluation error will be thrown.
)",
.fun = prim_ceil,
});
@@ -940,7 +952,7 @@ static void prim_floor(EvalState & state, const PosIdx pos, Value * * args, Valu
auto arg = args[0]->integer();
auto res = v.integer();
if (arg != res) {
state.error<EvalError>("Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occurred in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.", arg, res).atPos(pos).debugThrow();
state.error<EvalError>("Due to a bug (see https://github.com/NixOS/nix/issues/12899) a loss of precision occured in previous Nix versions because the NixInt argument %1% was rounded to %2%.\n\tFuture Nix versions might implement the correct behavior.", arg, res).atPos(pos).debugThrow();
}
}
}
@@ -974,7 +986,7 @@ static void prim_tryEval(EvalState & state, const PosIdx pos, Value * * args, Va
ReplExitStatus (* savedDebugRepl)(ref<EvalState> es, const ValMap & extraEnv) = nullptr;
if (state.debugRepl && state.settings.ignoreExceptionsDuringTry)
{
/* to prevent starting the repl from exceptions within a tryEval, null it. */
/* to prevent starting the repl from exceptions withing a tryEval, null it. */
savedDebugRepl = state.debugRepl;
state.debugRepl = nullptr;
}
@@ -1003,15 +1015,15 @@ static RegisterPrimOp primop_tryEval({
Try to shallowly evaluate *e*. Return a set containing the
attributes `success` (`true` if *e* evaluated successfully,
`false` if an error was thrown) and `value`, equalling *e* if
successful and `false` otherwise. `tryEval` only prevents
successful and `false` otherwise. `tryEval` will only prevent
errors created by `throw` or `assert` from being thrown.
Errors `tryEval` doesn't catch are, for example, those created
Errors `tryEval` will not catch are for example those created
by `abort` and type errors generated by builtins. Also note that
this doesn't evaluate *e* deeply, so `let e = { x = throw ""; };
in (builtins.tryEval e).success` is `true`. Using
in (builtins.tryEval e).success` will be `true`. Using
`builtins.deepSeq` one can get the expected result:
`let e = { x = throw ""; }; in
(builtins.tryEval (builtins.deepSeq e e)).success` is
(builtins.tryEval (builtins.deepSeq e e)).success` will be
`false`.
`tryEval` intentionally does not return the error message, because that risks bringing non-determinism into the evaluation result, and it would become very difficult to improve error reporting without breaking existing expressions.
@@ -1109,7 +1121,7 @@ static RegisterPrimOp primop_trace({
If the
[`debugger-on-trace`](@docroot@/command-ref/conf-file.md#conf-debugger-on-trace)
option is set to `true` and the `--debugger` flag is given, the
interactive debugger is started when `trace` is called (like
interactive debugger will be started when `trace` is called (like
[`break`](@docroot@/language/builtins.md#builtins-break)).
)",
.fun = prim_trace,
@@ -1158,7 +1170,7 @@ static RegisterPrimOp primop_warn({
If the
[`abort-on-warn`](@docroot@/command-ref/conf-file.md#conf-abort-on-warn)
option is set, the evaluation is aborted after the warning is printed.
option is set, the evaluation will be aborted after the warning is printed.
This is useful to reveal the stack trace of the warning, when the context is non-interactive and a debugger can not be launched.
)",
.fun = prim_warn,
@@ -1180,7 +1192,7 @@ static void prim_second(EvalState & state, const PosIdx pos, Value * * args, Val
static void derivationStrictInternal(
EvalState & state,
std::string_view name,
const std::string & name,
const Bindings * attrs,
Value & v);
@@ -1198,9 +1210,9 @@ static void prim_derivationStrict(EvalState & state, const PosIdx pos, Value * *
auto attrs = args[0]->attrs();
/* Figure out the name first (for stack backtraces). */
auto nameAttr = state.getAttr(state.sName, attrs, "in the attrset passed as argument to builtins.derivationStrict");
auto nameAttr = getAttr(state, state.sName, attrs, "in the attrset passed as argument to builtins.derivationStrict");
std::string_view drvName;
std::string drvName;
try {
drvName = state.forceStringNoCtx(*nameAttr->value, pos, "while evaluating the `name` attribute passed to builtins.derivationStrict");
} catch (Error & e) {
@@ -1259,7 +1271,7 @@ static void checkDerivationName(EvalState & state, std::string_view drvName)
static void derivationStrictInternal(
EvalState & state,
std::string_view drvName,
const std::string & drvName,
const Bindings * attrs,
Value & v)
{
@@ -1367,7 +1379,7 @@ static void derivationStrictInternal(
command-line arguments to the builder. */
else if (i->name == state.sArgs) {
state.forceList(*i->value, pos, context_below);
for (auto elem : i->value->listView()) {
for (auto elem : i->value->listItems()) {
auto s = state.coerceToString(pos, *elem, context,
"while evaluating an element of the argument list",
true).toOwned();
@@ -1399,7 +1411,7 @@ static void derivationStrictInternal(
/* Require ‘outputs’ to be a list of strings. */
state.forceList(*i->value, pos, context_below);
Strings ss;
for (auto elem : i->value->listView())
for (auto elem : i->value->listItems())
ss.emplace_back(state.forceStringNoCtx(*elem, pos, context_below));
handleOutputs(ss);
}
@@ -1428,8 +1440,6 @@ static void derivationStrictInternal(
else if (i->name == state.sOutputHashMode) handleHashMode(s);
else if (i->name == state.sOutputs)
handleOutputs(tokenizeString<Strings>(s));
else if (i->name == state.sJson)
warn("In derivation '%s': setting structured attributes via '__json' is deprecated, and may be disallowed in future versions of Nix. Set '__structuredAttrs = true' instead.", drvName);
}
}
@@ -1449,6 +1459,10 @@ static void derivationStrictInternal(
/* Everything in the context of the strings in the derivation
attributes should be added as dependencies of the resulting
derivation. */
StringMap rewrites;
std::optional<std::string> drvS;
for (auto & c : context) {
std::visit(overloaded {
/* Since this allows the builder to gain access to every
@@ -1471,11 +1485,24 @@ static void derivationStrictInternal(
drv.inputDrvs.ensureSlot(*b.drvPath).value.insert(b.output);
},
[&](const NixStringContextElem::Opaque & o) {
drv.inputSrcs.insert(o.path);
drv.inputSrcs.insert(state.devirtualize(o.path, &rewrites));
},
[&](const NixStringContextElem::Path & p) {
if (!drvS) drvS = drv.unparse(*state.store, true);
if (drvS->find(p.storePath.to_string()) != drvS->npos) {
auto devirtualized = state.devirtualize(p.storePath, &rewrites);
warn(
"Using 'builtins.derivation' to create a derivation named '%s' that references the store path '%s' without a proper context. "
"The resulting derivation will not have a correct store reference, so this is unreliable and may stop working in the future.",
drvName,
state.store->printStorePath(devirtualized));
}
},
}, c.raw);
}
drv.applyRewrites(rewrites);
/* Do we have all required attributes? */
if (drv.builder == "")
state.error<EvalError>("required attribute 'builder' missing")
@@ -1635,7 +1662,7 @@ static RegisterPrimOp primop_placeholder({
.name = "placeholder",
.args = {"output"},
.doc = R"(
Return an
Return at
[output placeholder string](@docroot@/store/derivation/index.md#output-placeholder)
for the specified *output* that will be substituted by the corresponding
[output path](@docroot@/glossary.md#gloss-output-path)
@@ -1800,7 +1827,7 @@ static RegisterPrimOp primop_baseNameOf({
After this, the *base name* is returned as previously described, assuming `/` as the directory separator. (Note that evaluation must be platform independent.)
This is somewhat similar to the [GNU `basename`](https://www.gnu.org/software/coreutils/manual/html_node/basename-invocation.html) command, but GNU `basename` strips any number of trailing slashes.
This is somewhat similar to the [GNU `basename`](https://www.gnu.org/software/coreutils/manual/html_node/basename-invocation.html) command, but GNU `basename` will strip any number of trailing slashes.
)",
.fun = prim_baseNameOf,
});
@@ -1882,7 +1909,7 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value * * args, V
LookupPath lookupPath;
for (auto v2 : args[0]->listView()) {
for (auto v2 : args[0]->listItems()) {
state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.findFile");
std::string prefix;
@@ -1890,7 +1917,7 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value * * args, V
if (i != v2->attrs()->end())
prefix = state.forceStringNoCtx(*i->value, pos, "while evaluating the `prefix` attribute of an element of the list passed to builtins.findFile");
i = state.getAttr(state.sPath, v2->attrs(), "in an element of the __nixPath");
i = getAttr(state, state.sPath, v2->attrs(), "in an element of the __nixPath");
NixStringContext context;
auto path = state.coerceToString(pos, *i->value, context,
@@ -1899,7 +1926,7 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value * * args, V
try {
auto rewrites = state.realiseContext(context);
path = rewriteStrings(std::move(path), rewrites);
path = rewriteStrings(path, rewrites);
} catch (InvalidPathError & e) {
state.error<EvalError>(
"cannot find '%1%', since path '%2%' is not valid",
@@ -1909,8 +1936,8 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value * * args, V
}
lookupPath.elements.emplace_back(LookupPath::Elem {
.prefix = LookupPath::Prefix { .s = std::move(prefix) },
.path = LookupPath::Path { .s = std::move(path) },
.prefix = LookupPath::Prefix { .s = prefix },
.path = LookupPath::Path { .s = path },
});
}
@@ -1999,9 +2026,9 @@ static RegisterPrimOp primop_findFile(PrimOp {
> ]
> ```
>
> and a *lookup-path* value `"nixos-config"` causes Nix to try `/home/eelco/Dev/nixos-config` and `/etc/nixos` in that order and return the first path that exists.
> and a *lookup-path* value `"nixos-config"` will cause Nix to try `/home/eelco/Dev/nixos-config` and `/etc/nixos` in that order and return the first path that exists.
If `path` starts with `http://` or `https://`, it is interpreted as the URL of a tarball to be downloaded and unpacked to a temporary location.
If `path` starts with `http://` or `https://`, it is interpreted as the URL of a tarball that will be downloaded and unpacked to a temporary location.
The tarball must consist of a single top-level directory.
The URLs of the tarballs from the official `nixos.org` channels can be abbreviated as `channel:<channel-name>`.
@@ -2148,7 +2175,7 @@ static RegisterPrimOp primop_readDir({
Return the contents of the directory *path* as a set mapping
directory entries to the corresponding file type. For instance, if
directory `A` contains a regular file `B` and another directory
`C`, then `builtins.readDir ./A` returns the set
`C`, then `builtins.readDir ./A` will return the set
```nix
{ B = "regular"; C = "directory"; }
@@ -2183,8 +2210,8 @@ static RegisterPrimOp primop_outputOf({
[input placeholder string](@docroot@/store/derivation/index.md#input-placeholder)
if needed.
If the derivation has a statically-known output path (i.e. the derivation output is input-addressed, or fixed content-addressed), the output path is returned.
But if the derivation is content-addressed or if the derivation is itself not-statically produced (i.e. is the output of another derivation), an input placeholder is returned instead.
If the derivation has a statically-known output path (i.e. the derivation output is input-addressed, or fixed content-addresed), the output path will just be returned.
But if the derivation is content-addressed or if the derivation is itself not-statically produced (i.e. is the output of another derivation), an input placeholder will be returned instead.
*`derivation reference`* must be a string that may contain a regular store path to a derivation, or may be an input placeholder reference.
If the derivation is produced by a derivation, you must explicitly select `drv.outPath`.
@@ -2197,7 +2224,7 @@ static RegisterPrimOp primop_outputOf({
"out"
```
returns an input placeholder for the output of the output of `myDrv`.
will return a input placeholder for the output of the output of `myDrv`.
This primop corresponds to the `^` sigil for [deriving paths](@docroot@/glossary.md#gloss-deriving-paths), e.g. as part of installable syntax on the command line.
)",
@@ -2375,14 +2402,25 @@ static RegisterPrimOp primop_fromJSON({
static void prim_toFile(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
NixStringContext context;
auto name = state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.toFile");
auto contents = state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.toFile");
std::string name(state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.toFile"));
std::string contents(state.forceString(*args[1], context, pos, "while evaluating the second argument passed to builtins.toFile"));
StorePathSet refs;
StringMap rewrites;
for (auto c : context) {
if (auto p = std::get_if<NixStringContextElem::Opaque>(&c.raw))
refs.insert(p->path);
else if (auto p = std::get_if<NixStringContextElem::Path>(&c.raw)) {
if (contents.find(p->storePath.to_string()) != contents.npos) {
auto devirtualized = state.devirtualize(p->storePath, &rewrites);
warn(
"Using 'builtins.toFile' to create a file named '%s' that references the store path '%s' without a proper context. "
"The resulting file will not have a correct store reference, so this is unreliable and may stop working in the future.",
name,
state.store->printStorePath(devirtualized));
}
}
else
state.error<EvalError>(
"files created by %1% may not reference derivations, but %2% references %3%",
@@ -2392,6 +2430,8 @@ static void prim_toFile(EvalState & state, const PosIdx pos, Value * * args, Val
).atPos(pos).debugThrow();
}
contents = rewriteStrings(contents, rewrites);
auto storePath = settings.readOnlyMode
? state.store->makeFixedOutputPathFromCA(name, TextInfo {
.hash = hashString(HashAlgorithm::SHA256, contents),
@@ -2541,6 +2581,7 @@ static void addPath(
{}));
if (!expectedHash || !state.store->isValidPath(*expectedStorePath)) {
// FIXME: make this lazy?
auto dstPath = fetchToStore(
state.fetchSettings,
*state.store,
@@ -2572,7 +2613,7 @@ static void prim_filterSource(EvalState & state, const PosIdx pos, Value * * arg
"while evaluating the second argument (the path to filter) passed to 'builtins.filterSource'");
state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.filterSource");
addPath(state, pos, path.baseName(), path, args[0], ContentAddressMethod::Raw::NixArchive, std::nullopt, v, context);
addPath(state, pos, state.computeBaseName(path), path, args[0], ContentAddressMethod::Raw::NixArchive, std::nullopt, v, context);
}
static RegisterPrimOp primop_filterSource({
@@ -2583,12 +2624,12 @@ static RegisterPrimOp primop_filterSource({
>
> `filterSource` should not be used to filter store paths. Since
> `filterSource` uses the name of the input directory while naming
> the output directory, doing so produces a directory name in
> the output directory, doing so will produce a directory name in
> the form of `<hash2>-<hash>-<name>`, where `<hash>-<name>` is
> the name of the input directory. Since `<hash>` depends on the
> unfiltered directory, the name of the output directory
> indirectly depends on files that are filtered out by the
> function. This triggers a rebuild even when a filtered out
> unfiltered directory, the name of the output directory will
> indirectly depend on files that are filtered out by the
> function. This will trigger a rebuild even when a filtered out
> file is changed. Use `builtins.path` instead, which allows
> specifying the name of the output directory.
@@ -2603,8 +2644,8 @@ static RegisterPrimOp primop_filterSource({
}
```
However, if `source-dir` is a Subversion working copy, then all of
those annoying `.svn` subdirectories are also copied to the
However, if `source-dir` is a Subversion working copy, then all
those annoying `.svn` subdirectories will also be copied to the
store. Worse, the contents of those directories may change a lot,
causing lots of spurious rebuilds. With `filterSource` you can
filter out the `.svn` directories:
@@ -2624,8 +2665,8 @@ static RegisterPrimOp primop_filterSource({
`"regular"`, `"directory"`, `"symlink"` or `"unknown"` (for other
kinds of files such as device nodes or fifos — but note that those
cannot be copied to the Nix store, so if the predicate returns
`true` for them, the copy fails). If you exclude a directory,
the entire corresponding subtree of *e2* is excluded.
`true` for them, the copy will fail). If you exclude a directory,
the entire corresponding subtree of *e2* will be excluded.
)",
.fun = prim_filterSource,
});
@@ -2633,7 +2674,7 @@ static RegisterPrimOp primop_filterSource({
static void prim_path(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
std::optional<SourcePath> path;
std::string_view name;
std::string name;
Value * filterFun = nullptr;
auto method = ContentAddressMethod::Raw::NixArchive;
std::optional<Hash> expectedHash;
@@ -2699,7 +2740,7 @@ static RegisterPrimOp primop_path({
- sha256\
When provided, this is the expected hash of the file at the
path. Evaluation fails if the hash is incorrect, and
path. Evaluation will fail if the hash is incorrect, and
providing a hash allows `builtins.path` to be used even when the
`pure-eval` nix config option is on.
)",
@@ -2721,7 +2762,7 @@ static void prim_attrNames(EvalState & state, const PosIdx pos, Value * * args,
auto list = state.buildList(args[0]->attrs()->size());
for (const auto & [n, i] : enumerate(*args[0]->attrs()))
list[n] = Value::toPtr(state.symbols[i.name]);
(list[n] = state.allocValue())->mkString(state.symbols[i.name]);
std::sort(list.begin(), list.end(),
[](Value * v1, Value * v2) { return strcmp(v1->c_str(), v2->c_str()) < 0; });
@@ -2779,7 +2820,8 @@ void prim_getAttr(EvalState & state, const PosIdx pos, Value * * args, Value & v
{
auto attr = state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.getAttr");
state.forceAttrs(*args[1], pos, "while evaluating the second argument passed to builtins.getAttr");
auto i = state.getAttr(
auto i = getAttr(
state,
state.symbols.create(attr),
args[1]->attrs(),
"in the attribute set under consideration"
@@ -2826,7 +2868,7 @@ static RegisterPrimOp primop_unsafeGetAttrPos(PrimOp {
.fun = prim_unsafeGetAttrPos,
});
// access to exact position information (ie, line and column numbers) is deferred
// access to exact position information (ie, line and colum numbers) is deferred
// due to the cost associated with calculating that information and how rarely
// it is used in practice. this is achieved by creating thunks to otherwise
// inaccessible primops that are not exposed as __op or under builtins to turn
@@ -2838,7 +2880,7 @@ static RegisterPrimOp primop_unsafeGetAttrPos(PrimOp {
// but each type of thunk has an associated runtime cost in the current evaluator.
// as with black holes this cost is too high to justify another thunk type to check
// for in the very hot path that is forceValue.
static struct LazyPosAccessors {
static struct LazyPosAcessors {
PrimOp primop_lineOfPos{
.arity = 1,
.fun = [] (EvalState & state, PosIdx pos, Value * * args, Value & v) {
@@ -2854,7 +2896,7 @@ static struct LazyPosAccessors {
Value lineOfPos, columnOfPos;
LazyPosAccessors()
LazyPosAcessors()
{
lineOfPos.mkPrimOp(&primop_lineOfPos);
columnOfPos.mkPrimOp(&primop_columnOfPos);
@@ -2920,7 +2962,7 @@ static void prim_removeAttrs(EvalState & state, const PosIdx pos, Value * * args
// 64: large enough to fit the attributes of a derivation
boost::container::small_vector<Attr, 64> names;
names.reserve(args[1]->listSize());
for (auto elem : args[1]->listView()) {
for (auto elem : args[1]->listItems()) {
state.forceStringNoCtx(*elem, pos, "while evaluating the values of the second argument passed to builtins.removeAttrs");
names.emplace_back(state.symbols.create(elem->string_view()), nullptr);
}
@@ -2962,48 +3004,25 @@ static void prim_listToAttrs(EvalState & state, const PosIdx pos, Value * * args
{
state.forceList(*args[0], pos, "while evaluating the argument passed to builtins.listToAttrs");
// Step 1. Sort the name-value attrsets in place using the memory we allocate for the result
auto listView = args[0]->listView();
size_t listSize = listView.size();
auto & bindings = *state.allocBindings(listSize);
using ElemPtr = decltype(&bindings[0].value);
auto attrs = state.buildBindings(args[0]->listSize());
for (const auto & [n, v2] : enumerate(listView)) {
std::set<Symbol> seen;
for (auto v2 : args[0]->listItems()) {
state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.listToAttrs");
auto j = state.getAttr(state.sName, v2->attrs(), "in a {name=...; value=...;} pair");
auto j = getAttr(state, state.sName, v2->attrs(), "in a {name=...; value=...;} pair");
auto name = state.forceStringNoCtx(*j->value, j->pos, "while evaluating the `name` attribute of an element of the list passed to builtins.listToAttrs");
auto sym = state.symbols.create(name);
// (ab)use Attr to store a Value * * instead of a Value *, so that we can stabilize the sort using the Value * *
bindings[n] = Attr(sym, std::bit_cast<Value *>(&v2));
}
std::sort(&bindings[0], &bindings[listSize], [](const Attr & a, const Attr & b) {
// Note that .value is actually a Value * * that corresponds to the position in the list
return a < b || (!(a > b) && std::bit_cast<ElemPtr>(a.value) < std::bit_cast<ElemPtr>(b.value));
});
// Step 2. Unpack the bindings in place and skip name-value pairs with duplicate names
Symbol prev;
for (size_t n = 0; n < listSize; n++) {
auto attr = bindings[n];
if (prev == attr.name) {
continue;
if (seen.insert(sym).second) {
auto j2 = getAttr(state, state.sValue, v2->attrs(), "in a {name=...; value=...;} pair");
attrs.insert(sym, j2->value, j2->pos);
}
// Note that .value is actually a Value * *; see earlier comments
Value * v2 = *std::bit_cast<ElemPtr>(attr.value);
}
auto j = state.getAttr(state.sValue, v2->attrs(), "in a {name=...; value=...;} pair");
prev = attr.name;
bindings.push_back({prev, j->value, j->pos});
}
// help GC and clear end of allocated array
for (size_t n = bindings.size(); n < listSize; n++) {
bindings[n] = Attr{};
}
v.mkAttrs(&bindings);
v.mkAttrs(attrs);
}
static RegisterPrimOp primop_listToAttrs({
@@ -3123,14 +3142,14 @@ static void prim_catAttrs(EvalState & state, const PosIdx pos, Value * * args, V
SmallValueVector<nonRecursiveStackReservation> res(args[1]->listSize());
size_t found = 0;
for (auto v2 : args[1]->listView()) {
for (auto v2 : args[1]->listItems()) {
state.forceAttrs(*v2, pos, "while evaluating an element in the list passed as second argument to builtins.catAttrs");
if (auto i = v2->attrs()->get(attrName))
res[found++] = i->value;
}
auto list = state.buildList(found);
for (size_t n = 0; n < found; ++n)
for (unsigned int n = 0; n < found; ++n)
list[n] = res[n];
v.mkList(list);
}
@@ -3162,21 +3181,15 @@ static void prim_functionArgs(EvalState & state, const PosIdx pos, Value * * arg
if (!args[0]->isLambda())
state.error<TypeError>("'functionArgs' requires a function").atPos(pos).debugThrow();
if (!args[0]->lambda().fun->hasFormals()) {
if (!args[0]->payload.lambda.fun->hasFormals()) {
v.mkAttrs(&state.emptyBindings);
return;
}
const auto &formals = args[0]->lambda().fun->formals->formals;
auto attrs = state.buildBindings(formals.size());
for (auto & i : formals)
auto attrs = state.buildBindings(args[0]->payload.lambda.fun->formals->formals.size());
for (auto & i : args[0]->payload.lambda.fun->formals->formals)
attrs.insert(i.name, state.getBool(i.def), i.pos);
/* Optimization: avoid sorting bindings. `formals` must already be sorted according to
(std::tie(a.name, a.pos) < std::tie(b.name, b.pos)) predicate, so the following assertion
always holds:
assert(std::is_sorted(attrs.alreadySorted()->begin(), attrs.alreadySorted()->end()));
.*/
v.mkAttrs(attrs.alreadySorted());
v.mkAttrs(attrs);
}
static RegisterPrimOp primop_functionArgs({
@@ -3204,8 +3217,9 @@ static void prim_mapAttrs(EvalState & state, const PosIdx pos, Value * * args, V
auto attrs = state.buildBindings(args[1]->attrs()->size());
for (auto & i : *args[1]->attrs()) {
Value * vName = Value::toPtr(state.symbols[i.name]);
Value * vName = state.allocValue();
Value * vFun2 = state.allocValue();
vName->mkString(state.symbols[i.name]);
vFun2->mkApp(args[0], vName);
attrs.alloc(i.name).mkApp(vFun2, i.value);
}
@@ -3248,7 +3262,7 @@ static void prim_zipAttrsWith(EvalState & state, const PosIdx pos, Value * * arg
state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.zipAttrsWith");
state.forceList(*args[1], pos, "while evaluating the second argument passed to builtins.zipAttrsWith");
const auto listItems = args[1]->listView();
const auto listItems = args[1]->listItems();
for (auto & vElem : listItems) {
state.forceAttrs(*vElem, noPos, "while evaluating a value of the list passed as second argument to builtins.zipAttrsWith");
@@ -3269,7 +3283,8 @@ static void prim_zipAttrsWith(EvalState & state, const PosIdx pos, Value * * arg
auto attrs = state.buildBindings(attrsSeen.size());
for (auto & [sym, elem] : attrsSeen) {
auto name = Value::toPtr(state.symbols[sym]);
auto name = state.allocValue();
name->mkString(state.symbols[sym]);
auto call1 = state.allocValue();
call1->mkApp(args[0], name);
auto call2 = state.allocValue();
@@ -3341,14 +3356,14 @@ static void prim_elemAt(EvalState & state, const PosIdx pos, Value * * args, Val
{
NixInt::Inner n = state.forceInt(*args[1], pos, "while evaluating the second argument passed to 'builtins.elemAt'").value;
state.forceList(*args[0], pos, "while evaluating the first argument passed to 'builtins.elemAt'");
if (n < 0 || std::make_unsigned_t<NixInt::Inner>(n) >= args[0]->listSize())
if (n < 0 || (unsigned int) n >= args[0]->listSize())
state.error<EvalError>(
"'builtins.elemAt' called with index %d on a list of size %d",
n,
args[0]->listSize()
).atPos(pos).debugThrow();
state.forceValue(*args[0]->listView()[n], pos);
v = *args[0]->listView()[n];
state.forceValue(*args[0]->listElems()[n], pos);
v = *args[0]->listElems()[n];
}
static RegisterPrimOp primop_elemAt({
@@ -3369,8 +3384,8 @@ static void prim_head(EvalState & state, const PosIdx pos, Value * * args, Value
state.error<EvalError>(
"'builtins.head' called on an empty list"
).atPos(pos).debugThrow();
state.forceValue(*args[0]->listView()[0], pos);
v = *args[0]->listView()[0];
state.forceValue(*args[0]->listElems()[0], pos);
v = *args[0]->listElems()[0];
}
static RegisterPrimOp primop_head({
@@ -3395,7 +3410,7 @@ static void prim_tail(EvalState & state, const PosIdx pos, Value * * args, Value
auto list = state.buildList(args[0]->listSize() - 1);
for (const auto & [n, v] : enumerate(list))
v = args[0]->listView()[n + 1];
v = args[0]->listElems()[n + 1];
v.mkList(list);
}
@@ -3430,7 +3445,7 @@ static void prim_map(EvalState & state, const PosIdx pos, Value * * args, Value
auto list = state.buildList(args[1]->listSize());
for (const auto & [n, v] : enumerate(list))
(v = state.allocValue())->mkApp(
args[0], args[1]->listView()[n]);
args[0], args[1]->listElems()[n]);
v.mkList(list);
}
@@ -3464,16 +3479,15 @@ static void prim_filter(EvalState & state, const PosIdx pos, Value * * args, Val
state.forceFunction(*args[0], pos, "while evaluating the first argument passed to builtins.filter");
auto len = args[1]->listSize();
SmallValueVector<nonRecursiveStackReservation> vs(len);
SmallValueVector<nonRecursiveStackReservation> vs(args[1]->listSize());
size_t k = 0;
bool same = true;
for (size_t n = 0; n < len; ++n) {
for (unsigned int n = 0; n < args[1]->listSize(); ++n) {
Value res;
state.callFunction(*args[0], *args[1]->listView()[n], res, noPos);
state.callFunction(*args[0], *args[1]->listElems()[n], res, noPos);
if (state.forceBool(res, pos, "while evaluating the return value of the filtering function passed to builtins.filter"))
vs[k++] = args[1]->listView()[n];
vs[k++] = args[1]->listElems()[n];
else
same = false;
}
@@ -3502,7 +3516,7 @@ static void prim_elem(EvalState & state, const PosIdx pos, Value * * args, Value
{
bool res = false;
state.forceList(*args[1], pos, "while evaluating the second argument passed to builtins.elem");
for (auto elem : args[1]->listView())
for (auto elem : args[1]->listItems())
if (state.eqValues(*args[0], *elem, pos, "while searching for the presence of the given element in the list")) {
res = true;
break;
@@ -3524,8 +3538,7 @@ static RegisterPrimOp primop_elem({
static void prim_concatLists(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
state.forceList(*args[0], pos, "while evaluating the first argument passed to builtins.concatLists");
auto listView = args[0]->listView();
state.concatLists(v, args[0]->listSize(), listView.data(), pos, "while evaluating a value of the list passed to builtins.concatLists");
state.concatLists(v, args[0]->listSize(), args[0]->listElems(), pos, "while evaluating a value of the list passed to builtins.concatLists");
}
static RegisterPrimOp primop_concatLists({
@@ -3563,8 +3576,7 @@ static void prim_foldlStrict(EvalState & state, const PosIdx pos, Value * * args
if (args[2]->listSize()) {
Value * vCur = args[1];
auto listView = args[2]->listView();
for (auto [n, elem] : enumerate(listView)) {
for (auto [n, elem] : enumerate(args[2]->listItems())) {
Value * vs []{vCur, elem};
vCur = n == args[2]->listSize() - 1 ? &v : state.allocValue();
state.callFunction(*args[0], vs, *vCur, pos);
@@ -3606,7 +3618,7 @@ static void anyOrAll(bool any, EvalState & state, const PosIdx pos, Value * * ar
: "while evaluating the return value of the function passed to builtins.all";
Value vTmp;
for (auto elem : args[1]->listView()) {
for (auto elem : args[1]->listItems()) {
state.callFunction(*args[0], *elem, vTmp, pos);
bool res = state.forceBool(vTmp, pos, errorCtx);
if (res == any) {
@@ -3653,12 +3665,12 @@ static void prim_genList(EvalState & state, const PosIdx pos, Value * * args, Va
{
auto len_ = state.forceInt(*args[1], pos, "while evaluating the second argument passed to builtins.genList").value;
if (len_ < 0 || std::make_unsigned_t<NixInt::Inner>(len_) > std::numeric_limits<size_t>::max())
if (len_ < 0)
state.error<EvalError>("cannot create list of size %1%", len_).atPos(pos).debugThrow();
size_t len = size_t(len_);
// More strict than strictly (!) necessary, but acceptable
// More strict than striclty (!) necessary, but acceptable
// as evaluating map without accessing any values makes little sense.
state.forceFunction(*args[0], noPos, "while evaluating the first argument passed to builtins.genList");
@@ -3704,7 +3716,7 @@ static void prim_sort(EvalState & state, const PosIdx pos, Value * * args, Value
auto list = state.buildList(len);
for (const auto & [n, v] : enumerate(list))
state.forceValue(*(v = args[1]->listView()[n]), pos);
state.forceValue(*(v = args[1]->listElems()[n]), pos);
auto comparator = [&](Value * a, Value * b) {
/* Optimization: if the comparator is lessThan, bypass
@@ -3721,14 +3733,10 @@ static void prim_sort(EvalState & state, const PosIdx pos, Value * * args, Value
return state.forceBool(vBool, pos, "while evaluating the return value of the sorting function passed to builtins.sort");
};
/* NOTE: Using custom implementation because std::sort and std::stable_sort
are not resilient to comparators that violate strict weak ordering. Diagnosing
incorrect implementations is a O(n^3) problem, so doing the checks is much more
expensive that doing the sorting. For this reason we choose to use sorting algorithms
that are can't be broken by invalid comprators. peeksort (mergesort)
doesn't misbehave when any of the strict weak order properties is
violated - output is always a reordering of the input. */
peeksort(list.begin(), list.end(), comparator);
/* FIXME: std::sort can segfault if the comparator is not a strict
weak ordering. What to do? std::stable_sort() seems more
resilient, but no guarantees... */
std::stable_sort(list.begin(), list.end(), comparator);
v.mkList(list);
}
@@ -3750,32 +3758,6 @@ static RegisterPrimOp primop_sort({
This is a stable sort: it preserves the relative order of elements
deemed equal by the comparator.
*comparator* must impose a strict weak ordering on the set of values
in the *list*. This means that for any elements *a*, *b* and *c* from the
*list*, *comparator* must satisfy the following relations:
1. Transitivity
```nix
comparator a b && comparator b c -> comparator a c
```
1. Irreflexivity
```nix
comparator a a == false
```
1. Transitivity of equivalence
```nix
let equiv = a: b: (!comparator a b && !comparator b a); in
equiv a b && equiv b c -> equiv a c
```
If the *comparator* violates any of these properties, then `builtins.sort`
reorders elements in an unspecified manner.
)",
.fun = prim_sort,
});
@@ -3789,8 +3771,8 @@ static void prim_partition(EvalState & state, const PosIdx pos, Value * * args,
ValueVector right, wrong;
for (size_t n = 0; n < len; ++n) {
auto vElem = args[1]->listView()[n];
for (unsigned int n = 0; n < len; ++n) {
auto vElem = args[1]->listElems()[n];
state.forceValue(*vElem, pos);
Value res;
state.callFunction(*args[0], *vElem, res, pos);
@@ -3847,7 +3829,7 @@ static void prim_groupBy(EvalState & state, const PosIdx pos, Value * * args, Va
ValueVectorMap attrs;
for (auto vElem : args[1]->listView()) {
for (auto vElem : args[1]->listItems()) {
Value res;
state.callFunction(*args[0], *vElem, res, pos);
auto name = state.forceStringNoCtx(res, pos, "while evaluating the return value of the grouping function passed to builtins.groupBy");
@@ -3902,8 +3884,8 @@ static void prim_concatMap(EvalState & state, const PosIdx pos, Value * * args,
SmallTemporaryValueVector<conservativeStackReservation> lists(nrLists);
size_t len = 0;
for (size_t n = 0; n < nrLists; ++n) {
Value * vElem = args[1]->listView()[n];
for (unsigned int n = 0; n < nrLists; ++n) {
Value * vElem = args[1]->listElems()[n];
state.callFunction(*args[0], *vElem, lists[n], pos);
state.forceList(lists[n], lists[n].determinePos(args[0]->determinePos(pos)), "while evaluating the return value of the function passed to builtins.concatMap");
len += lists[n].listSize();
@@ -3911,11 +3893,10 @@ static void prim_concatMap(EvalState & state, const PosIdx pos, Value * * args,
auto list = state.buildList(len);
auto out = list.elems;
for (size_t n = 0, pos = 0; n < nrLists; ++n) {
auto listView = lists[n].listView();
auto l = listView.size();
for (unsigned int n = 0, pos = 0; n < nrLists; ++n) {
auto l = lists[n].listSize();
if (l)
memcpy(out + pos, listView.data(), l * sizeof(Value *));
memcpy(out + pos, lists[n].listElems(), l * sizeof(Value *));
pos += l;
}
v.mkList(list);
@@ -4177,20 +4158,22 @@ static RegisterPrimOp primop_toString({
non-negative. */
static void prim_substring(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
using NixUInt = std::make_unsigned_t<NixInt::Inner>;
NixInt::Inner start = state.forceInt(*args[0], pos, "while evaluating the first argument (the start offset) passed to builtins.substring").value;
if (start < 0)
state.error<EvalError>("negative start position in 'substring'").atPos(pos).debugThrow();
NixInt::Inner len = state.forceInt(*args[1], pos, "while evaluating the second argument (the substring length) passed to builtins.substring").value;
// Negative length may be idiomatically passed to builtins.substring to get
// the tail of the string.
auto _len = std::numeric_limits<std::string::size_type>::max();
if (len < 0) {
len = std::numeric_limits<NixInt::Inner>::max();
}
// Special-case on empty substring to avoid O(n) strlen
// This allows for the use of empty substrings to efficiently capture string context
// This allows for the use of empty substrings to efficently capture string context
if (len == 0) {
state.forceValue(*args[2], pos);
if (args[2]->type() == nString) {
@@ -4199,14 +4182,10 @@ static void prim_substring(EvalState & state, const PosIdx pos, Value * * args,
}
}
if (len >= 0 && NixUInt(len) < _len) {
_len = len;
}
NixStringContext context;
auto s = state.coerceToString(pos, *args[2], context, "while evaluating the third argument (the string) passed to builtins.substring");
v.mkString(NixUInt(start) >= s->size() ? "" : s->substr(start, _len), context);
v.mkString((unsigned int) start >= s->size() ? "" : s->substr(start, len), context);
}
static RegisterPrimOp primop_substring({
@@ -4277,7 +4256,7 @@ static void prim_convertHash(EvalState & state, const PosIdx pos, Value * * args
state.forceAttrs(*args[0], pos, "while evaluating the first argument passed to builtins.convertHash");
auto inputAttrs = args[0]->attrs();
auto iteratorHash = state.getAttr(state.symbols.create("hash"), inputAttrs, "while locating the attribute 'hash'");
auto iteratorHash = getAttr(state, state.symbols.create("hash"), inputAttrs, "while locating the attribute 'hash'");
auto hash = state.forceStringNoCtx(*iteratorHash->value, pos, "while evaluating the attribute 'hash'");
auto iteratorHashAlgo = inputAttrs->get(state.symbols.create("hashAlgo"));
@@ -4285,7 +4264,7 @@ static void prim_convertHash(EvalState & state, const PosIdx pos, Value * * args
if (iteratorHashAlgo)
ha = parseHashAlgo(state.forceStringNoCtx(*iteratorHashAlgo->value, pos, "while evaluating the attribute 'hashAlgo'"));
auto iteratorToHashFormat = state.getAttr(state.symbols.create("toHashFormat"), args[0]->attrs(), "while locating the attribute 'toHashFormat'");
auto iteratorToHashFormat = getAttr(state, state.symbols.create("toHashFormat"), args[0]->attrs(), "while locating the attribute 'toHashFormat'");
HashFormat hf = parseHashFormat(state.forceStringNoCtx(*iteratorToHashFormat->value, pos, "while evaluating the attribute 'toHashFormat'"));
v.mkString(Hash::parseAny(hash, ha).to_string(hf, hf == HashFormat::SRI));
@@ -4510,7 +4489,7 @@ void prim_split(EvalState & state, const PosIdx pos, Value * * args, Value & v)
// Add a list for matched substrings.
const size_t slen = match.size() - 1;
// Start at 1, because the first match is the whole string.
// Start at 1, beacause the first match is the whole string.
auto list2 = state.buildList(slen);
for (const auto & [si, v2] : enumerate(list2)) {
if (!match[si + 1].matched)
@@ -4591,7 +4570,7 @@ static void prim_concatStringsSep(EvalState & state, const PosIdx pos, Value * *
res.reserve((args[1]->listSize() + 32) * sep.size());
bool first = true;
for (auto elem : args[1]->listView()) {
for (auto elem : args[1]->listItems()) {
if (first) first = false; else res += sep;
res += *state.coerceToString(pos, *elem, context, "while evaluating one element of the list of strings to concat passed to builtins.concatStringsSep");
}
@@ -4619,13 +4598,13 @@ static void prim_replaceStrings(EvalState & state, const PosIdx pos, Value * * a
"'from' and 'to' arguments passed to builtins.replaceStrings have different lengths"
).atPos(pos).debugThrow();
std::vector<std::string_view> from;
std::vector<std::string> from;
from.reserve(args[0]->listSize());
for (auto elem : args[0]->listView())
for (auto elem : args[0]->listItems())
from.emplace_back(state.forceString(*elem, pos, "while evaluating one of the strings to replace passed to builtins.replaceStrings"));
std::unordered_map<size_t, std::string_view> cache;
auto to = args[1]->listView();
std::unordered_map<size_t, std::string> cache;
auto to = args[1]->listItems();
NixStringContext context;
auto s = state.forceString(*args[2], context, pos, "while evaluating the third argument passed to builtins.replaceStrings");
@@ -4863,7 +4842,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings)
.type = nInt,
.doc = R"(
Return the [Unix time](https://en.wikipedia.org/wiki/Unix_time) at first evaluation.
Repeated references to that name re-use the initially obtained value.
Repeated references to that name will re-use the initially obtained value.
Example:
@@ -4878,7 +4857,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings)
1683705525
```
The [store path](@docroot@/store/store-path.md) of a derivation depending on `currentTime` differs for each evaluation, unless both evaluate `builtins.currentTime` in the same second.
The [store path](@docroot@/store/store-path.md) of a derivation depending on `currentTime` will differ for each evaluation, unless both evaluate `builtins.currentTime` in the same second.
)",
.impureOnly = true,
});
@@ -5054,7 +5033,7 @@ void EvalState::createBaseEnv(const EvalSettings & evalSettings)
/* Now that we've added all primops, sort the `builtins' set,
because attribute lookups expect it to be sorted. */
const_cast<Bindings *>(getBuiltins().attrs())->sort();
getBuiltins().payload.attrs->sort();
staticBaseEnv->sort();
+26 -6
View File
@@ -7,9 +7,15 @@ namespace nix {
static void prim_unsafeDiscardStringContext(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
NixStringContext context;
NixStringContext context, filtered;
auto s = state.coerceToString(pos, *args[0], context, "while evaluating the argument passed to builtins.unsafeDiscardStringContext");
v.mkString(*s);
for (auto & c : context)
if (auto * p = std::get_if<NixStringContextElem::Path>(&c.raw))
filtered.insert(*p);
v.mkString(*s, filtered);
}
static RegisterPrimOp primop_unsafeDiscardStringContext({
@@ -21,12 +27,19 @@ static RegisterPrimOp primop_unsafeDiscardStringContext({
.fun = prim_unsafeDiscardStringContext,
});
bool hasContext(const NixStringContext & context)
{
for (auto & c : context)
if (!std::get_if<NixStringContextElem::Path>(&c.raw))
return true;
return false;
}
static void prim_hasContext(EvalState & state, const PosIdx pos, Value * * args, Value & v)
{
NixStringContext context;
state.forceString(*args[0], context, pos, "while evaluating the argument passed to builtins.hasContext");
v.mkBool(!context.empty());
v.mkBool(hasContext(context));
}
static RegisterPrimOp primop_hasContext({
@@ -103,7 +116,7 @@ static void prim_addDrvOutputDependencies(EvalState & state, const PosIdx pos, V
NixStringContext context;
auto s = state.coerceToString(pos, *args[0], context, "while evaluating the argument passed to builtins.addDrvOutputDependencies");
auto contextSize = context.size();
auto contextSize = context.size();
if (contextSize != 1) {
state.error<EvalError>(
"context of string '%s' must have exactly one element, but has %d",
@@ -136,6 +149,11 @@ static void prim_addDrvOutputDependencies(EvalState & state, const PosIdx pos, V
above does not make much sense. */
return std::move(c);
},
[&](const NixStringContextElem::Path & p) -> NixStringContextElem::DrvDeep {
state.error<EvalError>(
"`addDrvOutputDependencies` does not work on a string without context"
).atPos(pos).debugThrow();
},
}, context.begin()->raw) }),
};
@@ -206,6 +224,8 @@ static void prim_getContext(EvalState & state, const PosIdx pos, Value * * args,
[&](NixStringContextElem::Opaque && o) {
contextInfos[std::move(o.path)].path = true;
},
[&](NixStringContextElem::Path && p) {
},
}, ((NixStringContextElem &&) i).raw);
}
@@ -240,7 +260,7 @@ static RegisterPrimOp primop_getContext({
The string context tracks references to derivations within a string.
It is represented as an attribute set of [store derivation](@docroot@/glossary.md#gloss-store-derivation) paths mapping to output names.
Using [string interpolation](@docroot@/language/string-interpolation.md) on a derivation adds that derivation to the string context.
Using [string interpolation](@docroot@/language/string-interpolation.md) on a derivation will add that derivation to the string context.
For example,
```nix
@@ -312,7 +332,7 @@ static void prim_appendContext(EvalState & state, const PosIdx pos, Value * * ar
name
).atPos(i.pos).debugThrow();
}
for (auto elem : attr->value->listView()) {
for (auto elem : attr->value->listItems()) {
auto outputName = state.forceStringNoCtx(*elem, attr->pos, "while evaluating an output name within a string context");
context.emplace(NixStringContextElem::Built {
.drvPath = makeConstantStorePathRef(namePath),
+2 -2
View File
@@ -124,7 +124,7 @@ static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value * * arg
for (auto & attr : *args[0]->attrs()) {
const auto & attrName = state.symbols[attr.name];
auto attrHint = [&]() -> std::string {
return fmt("while evaluating the attribute '%s' passed to builtins.fetchClosure", attrName);
return "while evaluating the '" + attrName + "' attribute passed to builtins.fetchClosure";
};
if (attrName == "fromPath") {
@@ -214,7 +214,7 @@ static RegisterPrimOp primop_fetchClosure({
.doc = R"(
Fetch a store path [closure](@docroot@/glossary.md#gloss-closure) from a binary cache, and return the store path as a string with context.
This function can be invoked in three ways that we will discuss in order of preference.
This function can be invoked in three ways, that we will discuss in order of preference.
**Fetch a content-addressed store path**
+1 -1
View File
@@ -64,7 +64,7 @@ static void prim_fetchMercurial(EvalState & state, const PosIdx pos, Value * * a
if (rev) attrs.insert_or_assign("rev", rev->gitRev());
auto input = fetchers::Input::fromAttrs(state.fetchSettings, std::move(attrs));
auto [storePath, input2] = input.fetchToStore(state.store);
auto [storePath, accessor, input2] = input.fetchToStore(state.store);
auto attrs2 = state.buildBindings(8);
state.mkStorePathString(storePath, attrs2.alloc(state.sOutPath));
+14 -12
View File
@@ -10,6 +10,8 @@
#include "nix/util/url.hh"
#include "nix/expr/value-to-json.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/fetchers/input-cache.hh"
#include "nix/util/mounted-source-accessor.hh"
#include <nlohmann/json.hpp>
@@ -191,7 +193,7 @@ static void fetchTree(
input.to_string());
else
state.error<EvalError>(
"in pure evaluation mode, '%s' doesn't fetch unlocked input '%s'",
"in pure evaluation mode, '%s' will not fetch unlocked input '%s'",
fetcher, input.to_string()).atPos(pos).debugThrow();
}
@@ -204,11 +206,11 @@ static void fetchTree(
throw Error("input '%s' is not allowed to use the '__final' attribute", input.to_string());
}
auto [storePath, input2] = input.fetchToStore(state.store);
auto cachedInput = state.inputCache->getAccessor(state.store, input, fetchers::UseRegistries::No);
state.allowPath(storePath);
auto storePath = state.mountInput(cachedInput.lockedInput, input, cachedInput.accessor, true);
emitTreeAttrs(state, storePath, input2, v, params.emptyRevFallback, false);
emitTreeAttrs(state, storePath, cachedInput.lockedInput, v, params.emptyRevFallback, false);
}
static void prim_fetchTree(EvalState & state, const PosIdx pos, Value * * args, Value & v)
@@ -243,7 +245,7 @@ static RegisterPrimOp primop_fetchTree({
That is, `fetchTree` is idempotent.
Downloads are cached in `$XDG_CACHE_HOME/nix`.
The remote source is fetched from the network if both are true:
The remote source will be fetched from the network if both are true:
- A NAR hash is supplied and the corresponding store path is not [valid](@docroot@/glossary.md#gloss-validity), that is, not available in the store
> **Note**
@@ -305,7 +307,7 @@ static RegisterPrimOp primop_fetchTree({
- `"tarball"`
Download a tar archive and extract it into the Nix store.
This has the same underlying implementation as [`builtins.fetchTarball`](@docroot@/language/builtins.md#builtins-fetchTarball)
This has the same underyling implementation as [`builtins.fetchTarball`](@docroot@/language/builtins.md#builtins-fetchTarball)
- `url` (String, required)
@@ -338,7 +340,7 @@ static RegisterPrimOp primop_fetchTree({
> **Note**
>
> If the URL points to a local directory, and no `ref` or `rev` is given, Nix only considers files added to the Git index, as listed by `git ls-files` but use the *current file contents* of the Git working directory.
> If the URL points to a local directory, and no `ref` or `rev` is given, Nix will only consider files added to the Git index, as listed by `git ls-files` but use the *current file contents* of the Git working directory.
- `ref` (String, optional)
@@ -681,7 +683,7 @@ static RegisterPrimOp primop_fetchGit({
This option has no effect once `shallow` cloning is enabled.
By default, the `ref` value is prefixed with `refs/heads/`.
As of 2.3.0, Nix doesn't prefix `refs/heads/` if `ref` starts with `refs/`.
As of 2.3.0, Nix will not prefix `refs/heads/` if `ref` starts with `refs/`.
- `submodules` (default: `false`)
@@ -840,7 +842,7 @@ static RegisterPrimOp primop_fetchGit({
}
```
Nix refetches the branch according to the [`tarball-ttl`](@docroot@/command-ref/conf-file.md#conf-tarball-ttl) setting.
Nix will refetch the branch according to the [`tarball-ttl`](@docroot@/command-ref/conf-file.md#conf-tarball-ttl) setting.
This behavior is disabled in [pure evaluation mode](@docroot@/command-ref/conf-file.md#conf-pure-eval).
@@ -851,9 +853,9 @@ static RegisterPrimOp primop_fetchGit({
```
If the URL points to a local directory, and no `ref` or `rev` is
given, `fetchGit` uses the current content of the checked-out
files, even if they are not committed or added to Git's index. It
only considers files added to the Git repository, as listed by `git ls-files`.
given, `fetchGit` will use the current content of the checked-out
files, even if they are not committed or added to Git's index. It will
only consider files added to the Git repository, as listed by `git ls-files`.
)",
.fun = prim_fetchGit,
});
+1 -1
View File
@@ -1,6 +1,6 @@
generated_headers += gen_header.process(
'derivation.nix',
preserve_path_from : meson.project_source_root(),
preserve_path_from: meson.project_source_root(),
)
sources += files(
+16 -14
View File
@@ -7,10 +7,10 @@ namespace nix {
// See: https://github.com/NixOS/nix/issues/9730
void printAmbiguous(
Value &v,
const SymbolTable &symbols,
std::ostream &str,
std::set<const void *> *seen,
EvalState & state,
Value & v,
std::ostream & str,
std::set<const void *> * seen,
int depth)
{
checkInterrupt();
@@ -26,9 +26,13 @@ void printAmbiguous(
case nBool:
printLiteralBool(str, v.boolean());
break;
case nString:
printLiteralString(str, v.string_view());
case nString: {
NixStringContext context;
copyContext(v, context);
// FIXME: make devirtualization configurable?
printLiteralString(str, state.devirtualize(v.string_view(), context));
break;
}
case nPath:
str << v.path().to_string(); // !!! escaping?
break;
@@ -40,9 +44,9 @@ void printAmbiguous(
str << "«repeated»";
else {
str << "{ ";
for (auto & i : v.attrs()->lexicographicOrder(symbols)) {
str << symbols[i->name] << " = ";
printAmbiguous(*i->value, symbols, str, seen, depth - 1);
for (auto & i : v.attrs()->lexicographicOrder(state.symbols)) {
str << state.symbols[i->name] << " = ";
printAmbiguous(state, *i->value, str, seen, depth - 1);
str << "; ";
}
str << "}";
@@ -50,15 +54,13 @@ void printAmbiguous(
break;
}
case nList:
/* Use pointer to the Value instead of pointer to the elements, because
that would need to explicitly handle the case of SmallList. */
if (seen && v.listSize() && !seen->insert(&v).second)
if (seen && v.listSize() && !seen->insert(v.listElems()).second)
str << "«repeated»";
else {
str << "[ ";
for (auto v2 : v.listView()) {
for (auto v2 : v.listItems()) {
if (v2)
printAmbiguous(*v2, symbols, str, seen, depth - 1);
printAmbiguous(state, *v2, str, seen, depth - 1);
else
str << "(nullptr)";
str << " ";
+11 -7
View File
@@ -249,7 +249,11 @@ private:
void printString(Value & v)
{
printLiteralString(output, v.string_view(), options.maxStringLength, options.ansiColors);
NixStringContext context;
copyContext(v, context);
std::ostringstream s;
printLiteralString(s, v.string_view(), options.maxStringLength, options.ansiColors);
output << state.devirtualize(s.str(), context);
}
void printPath(Value & v)
@@ -415,8 +419,8 @@ private:
if (depth < options.maxDepth) {
increaseIndent();
output << "[";
auto listItems = v.listView();
auto prettyPrint = shouldPrettyPrintList(listItems.span());
auto listItems = v.listItems();
auto prettyPrint = shouldPrettyPrintList(listItems);
size_t currentListItemsPrinted = 0;
@@ -453,13 +457,13 @@ private:
if (v.isLambda()) {
output << "lambda";
if (v.lambda().fun) {
if (v.lambda().fun->name) {
output << " " << state.symbols[v.lambda().fun->name];
if (v.payload.lambda.fun) {
if (v.payload.lambda.fun->name) {
output << " " << state.symbols[v.payload.lambda.fun->name];
}
std::ostringstream s;
s << state.positions[v.lambda().fun->pos];
s << state.positions[v.payload.lambda.fun->pos];
output << " @ " << filterANSIEscapes(toView(s));
}
} else if (v.isPrimOp()) {
+3 -2
View File
@@ -7,9 +7,10 @@
#include <iomanip>
#include <nlohmann/json.hpp>
namespace nix {
using json = nlohmann::json;
// TODO: rename. It doesn't print.
json printValueAsJSON(EvalState & state, bool strict,
Value & v, const PosIdx pos, NixStringContext & context, bool copyToStore)
@@ -73,7 +74,7 @@ json printValueAsJSON(EvalState & state, bool strict,
case nList: {
out = json::array();
int i = 0;
for (auto elem : v.listView()) {
for (auto elem : v.listItems()) {
try {
out.push_back(printValueAsJSON(state, strict, *elem, pos, context, copyToStore));
} catch (Error & e) {
+7 -7
View File
@@ -114,7 +114,7 @@ static void printValueAsXML(EvalState & state, bool strict, bool location,
case nList: {
XMLOpenElement _(doc, "list");
for (auto v2 : v.listView())
for (auto v2 : v.listItems())
printValueAsXML(state, strict, location, *v2, doc, context, drvsSeen, pos);
break;
}
@@ -126,18 +126,18 @@ static void printValueAsXML(EvalState & state, bool strict, bool location,
break;
}
XMLAttrs xmlAttrs;
if (location) posToXML(state, xmlAttrs, state.positions[v.lambda().fun->pos]);
if (location) posToXML(state, xmlAttrs, state.positions[v.payload.lambda.fun->pos]);
XMLOpenElement _(doc, "function", xmlAttrs);
if (v.lambda().fun->hasFormals()) {
if (v.payload.lambda.fun->hasFormals()) {
XMLAttrs attrs;
if (v.lambda().fun->arg) attrs["name"] = state.symbols[v.lambda().fun->arg];
if (v.lambda().fun->formals->ellipsis) attrs["ellipsis"] = "1";
if (v.payload.lambda.fun->arg) attrs["name"] = state.symbols[v.payload.lambda.fun->arg];
if (v.payload.lambda.fun->formals->ellipsis) attrs["ellipsis"] = "1";
XMLOpenElement _(doc, "attrspat", attrs);
for (auto & i : v.lambda().fun->formals->lexicographicOrder(state.symbols))
for (auto & i : v.payload.lambda.fun->formals->lexicographicOrder(state.symbols))
doc.writeEmptyElement("attr", singletonAttrs("name", state.symbols[i.name]));
} else
doc.writeEmptyElement("varpat", singletonAttrs("name", state.symbols[v.lambda().fun->arg]));
doc.writeEmptyElement("varpat", singletonAttrs("name", state.symbols[v.payload.lambda.fun->arg]));
break;
}
+9
View File
@@ -57,6 +57,11 @@ NixStringContextElem NixStringContextElem::parse(
.drvPath = StorePath { s.substr(1) },
};
}
case '@': {
return NixStringContextElem::Path {
.storePath = StorePath { s.substr(1) },
};
}
default: {
// Ensure no '!'
if (s.find("!") != std::string_view::npos) {
@@ -100,6 +105,10 @@ std::string NixStringContextElem::to_string() const
res += '=';
res += d.drvPath.to_string();
},
[&](const NixStringContextElem::Path & p) {
res += '@';
res += p.storePath.to_string();
},
}, raw);
return res;
+2 -2
View File
@@ -89,9 +89,9 @@ bool getBoolAttr(const Attrs & attrs, const std::string & name)
return *s;
}
StringMap attrsToQuery(const Attrs & attrs)
std::map<std::string, std::string> attrsToQuery(const Attrs & attrs)
{
StringMap query;
std::map<std::string, std::string> query;
for (auto & attr : attrs) {
if (auto v = std::get_if<uint64_t>(&attr.second)) {
query.insert_or_assign(attr.first, fmt("%d", *v));
+18 -19
View File
@@ -134,7 +134,7 @@ ParsedURL Input::toURL() const
return scheme->toURL(*this);
}
std::string Input::toURLString(const StringMap & extraQuery) const
std::string Input::toURLString(const std::map<std::string, std::string> & extraQuery) const
{
auto url = toURL();
for (auto & attr : extraQuery)
@@ -189,34 +189,30 @@ bool Input::contains(const Input & other) const
}
// FIXME: remove
std::pair<StorePath, Input> Input::fetchToStore(ref<Store> store) const
std::tuple<StorePath, ref<SourceAccessor>, Input> Input::fetchToStore(ref<Store> store) const
{
if (!scheme)
throw Error("cannot fetch unsupported input '%s'", attrsToJSON(toAttrs()));
auto [storePath, input] = [&]() -> std::pair<StorePath, Input> {
try {
auto [accessor, result] = getAccessorUnchecked(store);
try {
auto [accessor, result] = getAccessorUnchecked(store);
auto storePath = nix::fetchToStore(*settings, *store, SourcePath(accessor), FetchMode::Copy, result.getName());
auto storePath = nix::fetchToStore(*settings, *store, SourcePath(accessor), FetchMode::Copy, result.getName());
auto narHash = store->queryPathInfo(storePath)->narHash;
result.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
auto narHash = store->queryPathInfo(storePath)->narHash;
result.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
result.attrs.insert_or_assign("__final", Explicit<bool>(true));
result.attrs.insert_or_assign("__final", Explicit<bool>(true));
assert(result.isFinal());
assert(result.isFinal());
checkLocks(*this, result);
checkLocks(*this, result);
return {storePath, result};
} catch (Error & e) {
e.addTrace({}, "while fetching the input '%s'", to_string());
throw;
}
}();
return {std::move(storePath), input};
return {std::move(storePath), accessor, result};
} catch (Error & e) {
e.addTrace({}, "while fetching the input '%s'", to_string());
throw;
}
}
void Input::checkLocks(Input specified, Input & result)
@@ -234,6 +230,9 @@ void Input::checkLocks(Input specified, Input & result)
if (auto prevNarHash = specified.getNarHash())
specified.attrs.insert_or_assign("narHash", prevNarHash->to_string(HashFormat::SRI, true));
if (auto narHash = result.getNarHash())
result.attrs.insert_or_assign("narHash", narHash->to_string(HashFormat::SRI, true));
for (auto & field : specified.attrs) {
auto field2 = result.attrs.find(field.first);
if (field2 != result.attrs.end() && field.second != field2->second)
+6 -1
View File
@@ -20,9 +20,14 @@ bool FilteringSourceAccessor::pathExists(const CanonPath & path)
}
std::optional<SourceAccessor::Stat> FilteringSourceAccessor::maybeLstat(const CanonPath & path)
{
return isAllowed(path) ? next->maybeLstat(prefix / path) : std::nullopt;
}
SourceAccessor::Stat FilteringSourceAccessor::lstat(const CanonPath & path)
{
checkAccess(path);
return next->maybeLstat(prefix / path);
return next->lstat(prefix / path);
}
SourceAccessor::DirEntries FilteringSourceAccessor::readDirectory(const CanonPath & path)
+40 -68
View File
@@ -322,17 +322,8 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
for (size_t n = 0; n < git_commit_parentcount(commit->get()); ++n) {
git_commit * parent;
if (git_commit_parent(&parent, commit->get(), n)) {
throw Error(
"Failed to retrieve the parent of Git commit '%s': %s. "
"This may be due to an incomplete repository history. "
"To resolve this, either enable the shallow parameter in your flake URL (?shallow=1) "
"or add set the shallow parameter to true in builtins.fetchGit, "
"or fetch the complete history for this branch.",
*git_commit_id(commit->get()),
git_error_last()->message
);
}
if (git_commit_parent(&parent, commit->get(), n))
throw Error("getting parent of Git commit '%s': %s", *git_commit_id(commit->get()), git_error_last()->message);
todo.push(Commit(parent));
}
}
@@ -377,7 +368,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
if (git_config_iterator_glob_new(Setter(it), config.get(), "^submodule\\..*\\.(path|url|branch)$"))
throw Error("iterating over .gitmodules: %s", git_error_last()->message);
StringMap entries;
std::map<std::string, std::string> entries;
while (true) {
git_config_entry * entry = nullptr;
@@ -596,7 +587,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
});
/* Evaluate result through status code and checking if public
key fingerprints appear on stderr. This is necessary
key fingerprints appear on stderr. This is neccessary
because the git command might also succeed due to the
commit being signed by gpg keys that are present in the
users key agent. */
@@ -665,40 +656,28 @@ ref<GitRepo> GitRepo::openRepo(const std::filesystem::path & path, bool create,
struct GitSourceAccessor : SourceAccessor
{
struct State
{
ref<GitRepoImpl> repo;
Object root;
std::optional<lfs::Fetch> lfsFetch = std::nullopt;
};
Sync<State> state_;
ref<GitRepoImpl> repo;
Object root;
std::optional<lfs::Fetch> lfsFetch = std::nullopt;
GitSourceAccessor(ref<GitRepoImpl> repo_, const Hash & rev, bool smudgeLfs)
: state_{
State {
.repo = repo_,
.root = peelToTreeOrBlob(lookupObject(*repo_, hashToOID(rev)).get()),
.lfsFetch = smudgeLfs ? std::make_optional(lfs::Fetch(*repo_, hashToOID(rev))) : std::nullopt,
}
}
: repo(repo_)
, root(peelToTreeOrBlob(lookupObject(*repo, hashToOID(rev)).get()))
{
if (smudgeLfs)
lfsFetch = std::make_optional(lfs::Fetch(*repo, hashToOID(rev)));
}
std::string readBlob(const CanonPath & path, bool symlink)
{
auto state(state_.lock());
const auto blob = getBlob(path, symlink);
const auto blob = getBlob(*state, path, symlink);
if (state->lfsFetch) {
if (state->lfsFetch->shouldFetch(path)) {
if (lfsFetch) {
if (lfsFetch->shouldFetch(path)) {
StringSink s;
try {
// FIXME: do we need to hold the state lock while
// doing this?
auto contents = std::string((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
state->lfsFetch->fetch(contents, path, s, [&s](uint64_t size){ s.s.reserve(size); });
lfsFetch->fetch(contents, path, s, [&s](uint64_t size){ s.s.reserve(size); });
} catch (Error & e) {
e.addTrace({}, "while smudging git-lfs file '%s'", path);
throw;
@@ -717,18 +696,15 @@ struct GitSourceAccessor : SourceAccessor
bool pathExists(const CanonPath & path) override
{
auto state(state_.lock());
return path.isRoot() ? true : (bool) lookup(*state, path);
return path.isRoot() ? true : (bool) lookup(path);
}
std::optional<Stat> maybeLstat(const CanonPath & path) override
{
auto state(state_.lock());
if (path.isRoot())
return Stat { .type = git_object_type(state->root.get()) == GIT_OBJECT_TREE ? tDirectory : tRegular };
return Stat { .type = git_object_type(root.get()) == GIT_OBJECT_TREE ? tDirectory : tRegular };
auto entry = lookup(*state, path);
auto entry = lookup(path);
if (!entry)
return std::nullopt;
@@ -756,8 +732,6 @@ struct GitSourceAccessor : SourceAccessor
DirEntries readDirectory(const CanonPath & path) override
{
auto state(state_.lock());
return std::visit(overloaded {
[&](Tree tree) {
DirEntries res;
@@ -775,7 +749,7 @@ struct GitSourceAccessor : SourceAccessor
[&](Submodule) {
return DirEntries();
}
}, getTree(*state, path));
}, getTree(path));
}
std::string readLink(const CanonPath & path) override
@@ -789,9 +763,7 @@ struct GitSourceAccessor : SourceAccessor
*/
std::optional<Hash> getSubmoduleRev(const CanonPath & path)
{
auto state(state_.lock());
auto entry = lookup(*state, path);
auto entry = lookup(path);
if (!entry || git_tree_entry_type(entry) != GIT_OBJECT_COMMIT)
return std::nullopt;
@@ -802,7 +774,7 @@ struct GitSourceAccessor : SourceAccessor
std::unordered_map<CanonPath, TreeEntry> lookupCache;
/* Recursively look up 'path' relative to the root. */
git_tree_entry * lookup(State & state, const CanonPath & path)
git_tree_entry * lookup(const CanonPath & path)
{
auto i = lookupCache.find(path);
if (i != lookupCache.end()) return i->second.get();
@@ -812,7 +784,7 @@ struct GitSourceAccessor : SourceAccessor
auto name = path.baseName().value();
auto parentTree = lookupTree(state, *parent);
auto parentTree = lookupTree(*parent);
if (!parentTree) return nullptr;
auto count = git_tree_entrycount(parentTree->get());
@@ -841,29 +813,29 @@ struct GitSourceAccessor : SourceAccessor
return res;
}
std::optional<Tree> lookupTree(State & state, const CanonPath & path)
std::optional<Tree> lookupTree(const CanonPath & path)
{
if (path.isRoot()) {
if (git_object_type(state.root.get()) == GIT_OBJECT_TREE)
return dupObject<Tree>((git_tree *) &*state.root);
if (git_object_type(root.get()) == GIT_OBJECT_TREE)
return dupObject<Tree>((git_tree *) &*root);
else
return std::nullopt;
}
auto entry = lookup(state, path);
auto entry = lookup(path);
if (!entry || git_tree_entry_type(entry) != GIT_OBJECT_TREE)
return std::nullopt;
Tree tree;
if (git_tree_entry_to_object((git_object * *) (git_tree * *) Setter(tree), *state.repo, entry))
if (git_tree_entry_to_object((git_object * *) (git_tree * *) Setter(tree), *repo, entry))
throw Error("looking up directory '%s': %s", showPath(path), git_error_last()->message);
return tree;
}
git_tree_entry * need(State & state, const CanonPath & path)
git_tree_entry * need(const CanonPath & path)
{
auto entry = lookup(state, path);
auto entry = lookup(path);
if (!entry)
throw Error("'%s' does not exist", showPath(path));
return entry;
@@ -871,16 +843,16 @@ struct GitSourceAccessor : SourceAccessor
struct Submodule { };
std::variant<Tree, Submodule> getTree(State & state, const CanonPath & path)
std::variant<Tree, Submodule> getTree(const CanonPath & path)
{
if (path.isRoot()) {
if (git_object_type(state.root.get()) == GIT_OBJECT_TREE)
return dupObject<Tree>((git_tree *) &*state.root);
if (git_object_type(root.get()) == GIT_OBJECT_TREE)
return dupObject<Tree>((git_tree *) &*root);
else
throw Error("Git root object '%s' is not a directory", *git_object_id(state.root.get()));
throw Error("Git root object '%s' is not a directory", *git_object_id(root.get()));
}
auto entry = need(state, path);
auto entry = need(path);
if (git_tree_entry_type(entry) == GIT_OBJECT_COMMIT)
return Submodule();
@@ -889,16 +861,16 @@ struct GitSourceAccessor : SourceAccessor
throw Error("'%s' is not a directory", showPath(path));
Tree tree;
if (git_tree_entry_to_object((git_object * *) (git_tree * *) Setter(tree), *state.repo, entry))
if (git_tree_entry_to_object((git_object * *) (git_tree * *) Setter(tree), *repo, entry))
throw Error("looking up directory '%s': %s", showPath(path), git_error_last()->message);
return tree;
}
Blob getBlob(State & state, const CanonPath & path, bool expectSymlink)
Blob getBlob(const CanonPath & path, bool expectSymlink)
{
if (!expectSymlink && git_object_type(state.root.get()) == GIT_OBJECT_BLOB)
return dupObject<Blob>((git_blob *) &*state.root);
if (!expectSymlink && git_object_type(root.get()) == GIT_OBJECT_BLOB)
return dupObject<Blob>((git_blob *) &*root);
auto notExpected = [&]()
{
@@ -911,7 +883,7 @@ struct GitSourceAccessor : SourceAccessor
if (path.isRoot()) notExpected();
auto entry = need(state, path);
auto entry = need(path);
if (git_tree_entry_type(entry) != GIT_OBJECT_BLOB)
notExpected();
@@ -926,7 +898,7 @@ struct GitSourceAccessor : SourceAccessor
}
Blob blob;
if (git_tree_entry_to_object((git_object * *) (git_blob * *) Setter(blob), *state.repo, entry))
if (git_tree_entry_to_object((git_object * *) (git_blob * *) Setter(blob), *repo, entry))
throw Error("looking up file '%s': %s", showPath(path), git_error_last()->message);
return blob;
+4 -6
View File
@@ -15,6 +15,7 @@
#include "nix/fetchers/fetch-settings.hh"
#include "nix/util/json-utils.hh"
#include "nix/util/archive.hh"
#include "nix/util/mounted-source-accessor.hh"
#include <regex>
#include <string.h>
@@ -798,10 +799,8 @@ struct GitInputScheme : InputScheme
auto rev = repoInfo.workdirInfo.headRev.value_or(nullRev);
input.attrs.insert_or_assign("rev", rev.gitRev());
if (!getShallowAttr(input)) {
input.attrs.insert_or_assign("revCount",
rev == nullRev ? 0 : getRevCount(*input.settings, repoInfo, repoPath, rev));
}
input.attrs.insert_or_assign("revCount",
rev == nullRev ? 0 : getRevCount(*input.settings, repoInfo, repoPath, rev));
verifyCommit(input, repo);
} else {
@@ -883,8 +882,7 @@ struct GitInputScheme : InputScheme
bool isLocked(const Input & input) const override
{
auto rev = input.getRev();
return rev && rev != nullRev;
return (bool) input.getRev();
}
};
+8 -1
View File
@@ -175,7 +175,7 @@ struct GitArchiveInputScheme : InputScheme
return input;
}
// Search for the longest possible match starting from the beginning and ending at either the end or a path segment.
// Search for the longest possible match starting from the begining and ending at either the end or a path segment.
std::optional<std::string> getAccessToken(const fetchers::Settings & settings, const std::string & host, const std::string & url) const override
{
auto tokens = settings.accessTokens.get();
@@ -335,6 +335,13 @@ struct GitArchiveInputScheme : InputScheme
false,
"«" + input.to_string() + "»");
if (!input.settings->trustTarballsFromGitForges)
// FIXME: computing the NAR hash here is wasteful if
// copyInputToStore() is just going to hash/copy it as
// well.
input.attrs.insert_or_assign("narHash",
accessor->hashPath(CanonPath::root).to_string(HashFormat::SRI, true));
return {accessor, input};
}
@@ -37,7 +37,7 @@ std::optional<bool> maybeGetBoolAttr(const Attrs & attrs, const std::string & na
bool getBoolAttr(const Attrs & attrs, const std::string & name);
StringMap attrsToQuery(const Attrs & attrs);
std::map<std::string, std::string> attrsToQuery(const Attrs & attrs);
Hash getRevAttr(const Attrs & attrs, const std::string & name);
Loaded 100 of 318 files, more files were not shown because too many files have changed in this diff. Show more