It would have been nice to use std::views::enumerate here, but
it uses a signed difference type for the value_type:
> value_type = std::tuple<difference_type, ranges::range_value_t<Base>>
zip + iota has the same semantics as the code used to have, so there's
no behavior change here.
This is a nicer separation of concerns --- `DerivationBuilder` just
mounts the extra paths you tell it too, and the outside world is
responsible for making sure those extra paths make sense.
Since the closure only depends on global settings, and not
per-derivation information, we also have the option of moving this up
further and caching it across all local builds. (I only just realized
this after having done this refactor. I am not doing that change at this
time, however.)
Now, `DerivationBuilder` only concerns itself with `finalEnv` and
`extraFiles`, in straightforward unconditional code. All the fancy
desugaring logic is consolidated in `DerivationBuildingGoal`.
We should better share the pulled-out logic with `nix-shell`/`nix
develop`, which would fill in some missing features, arguably fixing
bugs.
I think this is a better separation of concerns. `DerivationBuilder`
doesn't need to to the final, query-heavy details about how these things
are constructed. It just operates on the level of "simple, stupid" files
and environment variables.
As much as I prefer rewriting the parsed rather than unparsed JSON for
elegance, this gets in the way of the separation of concerns that I am
trying to do.
As a practical matter, any rewriting that this did will also be done by
the second round of rewriting that remains below, so removing this code
should have no effect.
This is needed to rearrange include order, but I also think it is a good
thing anyways, as we seek to reduce the use of global settings variables
over time.
This avoids problems with older versions of Nix that don't put the
caches in WAL mode. That's generally not a problem, until you do something like
nix build --print-out-paths ... | cachix
which deadlocks because cachix tries to switch the caches to truncate
mode, which requires exclusive access. But the first process cannot
make progress because the cachix process isn't reading from the pipe.
With "truncate" mode, if we try to write to the database while another
process has an active write transaction, we'll block until the other
transaction finishes. This is a problem for the evaluation cache in
particular, since it uses long-running transactions.
WAL mode does not have this issue: it just returns "busy" right away,
so Nix will print
error (ignored): SQLite database '/home/eelco/.cache/nix/eval-cache-v5/...' is busy
and stop trying to write to the evaluation cache. (This was the
intended/original behaviour, see AttrDb::doSQLite().)
Compilers in nixpkgs have caught up and major distros
should also have recent enough compilers. It would be
nice to have newer features like more full featured
ranges and deducing this.
About time we upgraded our nixpkgs flake input. Ideally
we'd have automation to do this.
Flake lock file updates:
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/adaa24fbf46737f3f1b5497bf64bae750f82942e?narHash=sha256-qhFMmDkeJX9KJwr5H32f1r7Prs7XbQWtO0h3V0a0rFY%3D' (2025-05-13)
→ 'github:NixOS/nixpkgs/cd32a774ac52caaa03bcfc9e7591ac8c18617ced?narHash=sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI%3D' (2025-08-17)
This caused RemoteStore::queryPathInfoUncached() to mark the
connection as invalid (see
RemoteStore::ConnectionHandle::~ConnectionHandle()), causing it to
disconnect and reconnect after every lookup of an invalid path. This
caused huge slowdowns in conjunction with
19f89eb684 and lazy-trees.
Period '.' is a special branch name in the gitsubmodule file which
represents the branch of the parent repository [1].
We add support for this by registering the ref of the InputAccessor to
be that of the parent input if '.' is encountered.
Fixes#13215
[1]: man gitmodules
lowdown >= 1.4.0 supports LOWDOWN_TERM_NORELLINK to render
absolute urls. This is useful, since we want to keep links to
web resources and such intact.
Fix description of `NIX_CONF_DIR`. It currently say that it defaults to `prefix/etc/nix`, which would mean `/nix/etc/nix` on default installations, and contradicts the description in `conf-file-prefix.md`.
This fix makes the description of `NIX_CONF_DIR` consistent with `conf-file-prefix.md`, assuming that the latter is correct.
Turns out we didn't have tests for some of the important behavior introduced
for flake reference fragments and url queries [1]. This is rather important
and is relied upon by existing tooling. This fixes up these exact cases before
handing off the URL to the Boost.URL parser.
To the best of my knowledge this implements the same behavior as prior regex-based
parser did [2]:
> fragmentRegex = "(?:" + pcharRegex + "|[/? \"^])*";
> queryRegex = "(?:" + pcharRegex + "|[/? \"])*";
[1]: 9c0a09f09f
[2]: https://github.com/NixOS/nix/blob/2.30.2/src/libutil/include/nix/util/url-parts.hh
It's about time we added debuggers to the dev-shell. Having it in build inputs
does some magic so pretty printers for standard library types work better.
Since this goal has no (goal-wide) notion of "wanted outputs" (we're
building the derivation, and thus making all outputs), we should have
`initialOutputs` for all outputs, and if we're missing one that's an
internal error caused by a bug in Nix.
Concretely, `DerivationBuildingGoal::gaveUpOnSubstitution` now clearly
does create `initialOutputs` for all outputs, whereas a few commits ago
that was not obvious, so I feel confident in saying that this invariant
that should be upheld, in fact is upheld.
`scatchOutputs` is initialized for every initial output, so the same
change to it follows for the same reasons.
This is just more honest, since we downcasted it to `LocalStore` in many
places. We had the downcast before because it wasn't needed in the hook
case, just the local building case, but now that `DerivationBuilder` is
separated and just does the building case, we have formalized the
boundary where the single downcast should occur.
No derivation goal type has a notion of variable wanted outputs any
more. They either want them all, or they just care about a single
output, in which case we would just store this information for the one
output in question.
We can cut out some gratuitous inhertence as follows:
- `MixStoreDirMethods` -> `StoreDirConfig`
- `StoreDirConfig` deleted because no longer needed. It is just folded
into `StoreConfig`.
- `StoreDirConfigBase` -> `StoreConfigBase` same trick still needed, but
now is for `StoreConfig` not `StoreDirConfig`
Here's how we got here:
1. I once factored out `StoreDirConfig` in #6236.
2. I factored out `MixStoreDirMethods` in #13154.
But, I didn't realize at point (2) that we didn't need `StoreDirConfig`
anymore, all uses of `StoreDirConfig` could instead be uses of
`MixStoreDirMethods`. Now I am doing that, and renaming
`MixStoreDirMethods` to just `StoreDirConfig` to reduce churn.
This leads to a use-after free, because staticOutputHashes returns a temporary
object that dies before we can do a `return *mOutputHash`.
This is most likely the cause for random failures in Hydra [1].
[1]: https://hydra.nixos.org/build/305091330/nixlog/2
Old code completely ignored query parameters and it seems ok to keep
that behavior. There's a lot of code out there that parses nix code
like nix-output-monitor and it can't parse messages like:
> copying path '/nix/store/wha2hi4yhkjmccqhivxavbfspsg1wrsj-source' from 'https://cache.nixos.org' to 'local://'...
Let's not break these tools without a good reason. This goes in line
with what other code does by ignoring parameters in logs.
The issue is just in detecting the shorthand notations for the store
reference - not in printing the url in logs.
By default the daemon opens a local store with ?path-info-cache-size=0,
so that leads to the erronenous 'local://'.
The problem with old code was that it used getUri for both the `diskCache`
as well as logging. This is really bad because it mixes the textual human
readable representation with the caching.
Also using getUri for the cache key is really problematic for the S3 store,
since it doesn't include the `endpoint` in the cache key, so it's totally broken.
This starts separating the logging / cache concerns by introducing a
`getHumanReadableURI` that should only be used for logging. The caching
logic now instead uses `getReference().render(/*withParams=*/false)` exclusively.
This would need to be fixed in follow-ups, because that's really fragile and
broken for some store types (but it was already broken before).
Move output result filtering logic and assert just into the branch where
it is not obviously a no op / meeting the assertion.
Add a comment too, while we are at it.
Now that `DerivationGoal::checkPathValidity` is legible, we can see that
it only sets `outputKnown`, and doesn't read it. Likewise, with
co-routines, we don't have tiny scopes that make local variables
difficult. Between these two things, we can simply have
`checkPathValidity` return what it finds, rather than mutate some state,
and update everyting to use local variables.
The same transformation could probably be done to the other derivation
goal types (which currently, unfortunately, contain their own
`checkPathValidity`s, though they are diverging, and we hope and believe
that they continue to diverge).
`Store::queryPartialDerivationOutputMap` is nothing but checking
statically-known output paths, and then `Store::queryRealisation`, and
we were doing both of those things already. Inline that and simplify,
again taking advantage of the fact that we only care about one output.
Rather than having store implementations return a free-form URI string,
have them return a `StoreReference`. This reflects that fact that this
method is supposed to invert `resolveStoreConfig`, which goes from a
`StoreReference` to some `StoreConfig` concrete derived class (based on
the registry).
`StoreConfig::getUri` is kept only as a convenience for the common case
that we want to immediately render the `StoreReference`.
A few tests were changed to use `local://` not `local`, since
`StoreReference` does not encode the `local` and `daemon` shorthands
(and instead desugars them to `local://` and `unix://` right away). I
think that is fine. `local` and `daemon` still work as input.
* Adds support for NIX_SSHOPTS
* Properly uses the parsed port from URL (fixes#13337)
* Don't guess the HTTP endpoint, use the response of git-lfs-authenticate
* Add an SSH Git LFS test
* Removed some unused test code
Clang refused to do a narrowing conversion in an initializer list:
```
local-keys.cc:56:90: note: insert an explicit cast to silence this issue
return name + ":" + base64::encode(std::as_bytes(std::span<const unsigned char>{sig, sigLen}));
^~~~~~
static_cast<size_type>( )
```
This addresses several changes from toml11 4.0 bump in
nixpkgs [1].
1. Added more regression tests for timestamp formats.
Special attention needs to be paid to the precision
of the subsecond range for local-time. Prior versions select the closest
(upwards) multiple of 3 with a hard cap of 9 digits.
2. Normalize local datetime and offset datetime to always
use the uppercase separator `T`. This is actually the issue
surfaced in [2]. This canonicalization is basically a requirement
by (a certain reading) of rfc3339 section 5.6 [3].
3. If using toml11 >= 4.0 also keep the old behavior wrt
to the number of digits used for subsecond part of the local-time.
Newer versions cap it at 6 digits unconditionally.
[1]: https://www.github.com/NixOS/nixpkgs/pull/331649
[2]: https://www.github.com/NixOS/nix/issues/11441
[3]: https://datatracker.ietf.org/doc/html/rfc3339
Current test suite doesn't cover the subsecond formatting at
all and toml11 is quite finicky with that. We should at the very
least test its behavior to avoid silent breakages on updates.
Previously `getUri` didn't include store query parameters,
`ssh-ng` didn't include any information at all and the local
store didn't have the path:
```
$ nix store info --store "local?root=/tmp/aaa&require-sigs=false"
Store URL: local
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh-ng://localhost?remote-program=nix-daemon"
Store URL: ssh-ng://
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh://localhost?remote-program=nix-store"
Store URL: ssh://localhost
```
This commit changes this to:
```
$ nix store info --store "local?root=/tmp/aaa&require-sigs=false"
Store URL: local?require-sigs=false&root=/tmp/aaa
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh-ng://localhost?remote-program=nix-daemon"
Store URL: ssh-ng://localhost?remote-program=nix-daemon
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh://localhost?remote-program=nix-store"
Store URL: ssh://localhost?remote-program=nix-store
```
The clang-analyzer incorrectly flags a use-after-free for GC-managed objects
when used with std::unique_ptr. Since NixRepl inherits from gc, its memory
is properly managed by Boehm GC and this is a false positive.
Added NOLINTNEXTLINE directive to suppress the warning.
Fixes usage of `#` symbol in the reference name.
This also seems to identify several deficiencies in the libgit2 refname
validation code wrt to DEL symbol and a singular `@` symbol [1].
[1]: https://git-scm.com/docs/git-check-ref-format#_description
This change was necessary when we were using `nix flake check` for CI
(see 6db6b269ed). Now this is not really
necessary, because we are running the checks in a much saner way, that
doesn't use up too much memory for evaluation.
Sometime ago we lost the coverage job in the midst of
meson migration. Until we have something like codecov
it'd be very useful to restore this job with the html
reports and historical metrics.
As a bonus we get more coverage metrics by switching to
LLVM tooling from LCOV.
The implicit dependency on refLength (which is the StorePath::HashLen)
is not good. Also the companion tests and benchmarks are already in libstore-tests.
This doesn't seem to be used anywhere at the moment.
It might be used out-of-tree, but this is a small convenience
function that is not worth keeping without in-tree usage.
With this I'm able to do a fresh config + meson test with all dependencies
correctly propagated.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
This was removed in https://github.com/NixOS/nix/pull/11152. However,
we need it for the multi-threaded evaluator, because otherwise Boehm
GC will crash while scanning the thread stack:
#0 GC_push_all_eager (bottom=<optimized out>, top=<optimized out>) at extra/../mark.c:1488
#1 0x00007ffff74691d5 in GC_push_all_stack_sections (lo=<optimized out>, hi=<optimized out>, traced_stack_sect=0x0) at extra/../mark_rts.c:704
#2 GC_push_all_stacks () at extra/../pthread_stop_world.c:876
#3 GC_default_push_other_roots () at extra/../os_dep.c:2893
#4 0x00007ffff746235c in GC_mark_some (cold_gc_frame=0x7ffee8ecaa50 "`\304G\367\377\177") at extra/../mark.c:374
#5 0x00007ffff7465a8d in GC_stopped_mark (stop_func=stop_func@entry=0x7ffff7453c80 <GC_never_stop_func>) at extra/../alloc.c:875
#6 0x00007ffff7466724 in GC_try_to_collect_inner (stop_func=0x7ffff7453c80 <GC_never_stop_func>) at extra/../alloc.c:624
#7 0x00007ffff7466a22 in GC_collect_or_expand (needed_blocks=needed_blocks@entry=1, ignore_off_page=ignore_off_page@entry=0, retry=retry@entry=0) at extra/../alloc.c:1688
#8 0x00007ffff746878f in GC_allocobj (gran=<optimized out>, kind=<optimized out>) at extra/../alloc.c:1798
#9 GC_generic_malloc_inner (lb=<optimized out>, k=k@entry=1) at extra/../malloc.c:193
#10 0x00007ffff746cd40 in GC_generic_malloc_many (lb=<optimized out>, k=<optimized out>, result=<optimized out>) at extra/../mallocx.c:477
#11 0x00007ffff746cf35 in GC_malloc_kind (bytes=120, kind=1) at extra/../thread_local_alloc.c:187
#12 0x00007ffff796ede5 in nix::allocBytes (n=<optimized out>, n=<optimized out>) at ../src/libexpr/include/nix/expr/eval-inline.hh:19
This is because it will use the stack pointer of the coroutine, so it
will scan a region of memory that doesn't exist, e.g.
Stack for thread 0x7ffea4ff96c0 is [0x7ffe80197af0w,0x7ffea4ffa000)
(where 0x7ffe80197af0w is the sp of the coroutine and 0x7ffea4ffa000
is the base of the thread stack).
We don't scan coroutine stacks, because currently they don't have GC
roots (there is no evaluation happening in coroutines). So there is
currently no need to restore the other parts of the original patch,
such as BoehmGCStackAllocator.
I had not wanted to cause unncessary churn before, but now that we've
bitten the bullet with the Big Reformat, I feel it is the right time.
Future readers will appreciate that the declarations and definitions
files are one-to-one as they should be, and `store-api.cc` is good to
shrink in any event.
I don't think there are outstanding PRs changing this code either. (I
had some for a while, but they are all merged.)
- No more private constructor that is kinda weird
- Two new static functions, `baseFromSize` and `baseFromSize`, that do
one thing, and one thing only (simple).
- Two `Hash::parse*` that previously used the private constructor now
can use these two functions directly.
- The remaining `Hash::parseAny*` methods, which are inherently more
complex, are written in terms of a `parseAnyHelper` static function
which is also complex, but keeps the complexity in one spot.
This patch allows users to specify the connection port
in the store URLS like so:
```
nix store info --store "ssh-ng://localhost:22" --json
```
Previously this failed with: `error: failed to start SSH connection to 'localhost:22'`,
because the code did not distinguish the port from the hostname. This
patch remedies that problem by introducing a ParsedURL::Authority type
for working with parsed authority components of URIs.
Now that the URL parsing code is less ad-hoc we can
add more long-awaited fixes for specifying SSH connection
ports in store URIs.
Builds upon the work from bd1d2d1041.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Prior to this patch, mode 0444 is not updated to 0555 for directories.
That means for instance 0554 is canonicalized, but not 0444.
We don't believe this has any implications for backwards compatibility,
because directories do not have permissions in NAR format and so are
always 0555 after deserialization, and store paths with wrong
permissions can’t be copied to another host.
Co-authored-by: Robert Hensing <robert@roberthensing.nl>
Make it separate from Hash, since other things can be base-encoded too.
This isn't really needed for Nix, but it makes the code easier to read
e.g. for someone reimplementing this stuff in a different language. (Of
course, Base16/Base64 should be gotten off-the-shelf, but now the hash
code, which is more bespoke, is less cluttered with the parts that would
be from some library.)
Many reimplementations of "Nix32" and our hash type already exist, so
this cleanup is coming years too late, but I say better late than never
/ it is always good to nudge the code in the direction of being a
"living spec".
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Having a State class in the nix namespace is asking
for ODR trouble. This class is already private to the
translation unit, let's move it into an anonymous namespace.
There have been prior concerns about reogranizing the repo, but
this seems like a trivial simplification which will not interfere
with either our packaging or the modular builds in nixpkgs.
This adds the necessary infrastructure to collect
code coverage in CI, which could be useful to look
at munually or track consistently via something like
codecov.
Co-authored-by: Jade Lovelace <lix@jade.fyi>
This keeps things fast by making the function inline, but also prevents
people from having to know about the `0xFF` implementation detail
directly, instead making one go through a `std::optional` (which could be
fused away with a sufficiently smart compiler).
Additionally, the base "nix32" implementation is moved to its own header
file pair, as it is logically distinct and prior to the `Hash` data
type. It would probably be nice to do this with all the hash format
implementations.
Some binary caches (incorrectly) use this header to indicate lack of
compression, inspired by the valid "identity" token in the
"Accept-Encoding" header.
This benchmark should provide a relatively realistic
scenario for reference scanning.
As shown by the following results, reference scanning code
is already plenty fast and is definitely not a bottleneck:
```
BM_RefScanSinkRandom/10000 1672 ns 1682 ns 413354 bytes_per_second=5.53691Gi/s
BM_RefScanSinkRandom/100000 11217 ns 11124 ns 64341 bytes_per_second=8.37231Gi/s
BM_RefScanSinkRandom/1000000 205745 ns 204421 ns 3360 bytes_per_second=4.55591Gi/s
BM_RefScanSinkRandom/5000000 1208407 ns 1201046 ns 597 bytes_per_second=3.87713Gi/s
BM_RefScanSinkRandom/10000000 2534397 ns 2523344 ns 273 bytes_per_second=3.69083Gi/s
```
(Measurements on Ryzen 5900X via `nix build --file ci/gha/tests componentTests.nix-store-tests-run -L`)
The changes include:
* Defining nix32Chars as a constexpr char[].
* Adding a constexpr std::array<unsigned char, 256> (reverseNix32Map) to map characters to their base-32 digit values at compile time.
* Replacing the slow character search loop with a direct lookup using reverseNix32Map.
* Removing std::once_flag/isBase32 logic in references.cc in favor of reverseNix32Map
Signed-off-by: Alexander V. Nikolaev <avn@avnik.info>
This changes our GHA CI and nix-store-tests packaging
to build and run the benchmarks. This does not affect
the default packaging - the overrides apply only for the
GHA CI.
GCC doesn't really benefit as much as Clang does from
using precompiled headers. Another aspect to consider is that
clangd doesn't really like GCC's PCH flags in the compilation database,
so GCC based devshells would continue to work with clangd.
This also has the slight advantage of ensuring that our includes are in
order, since we build with both Clang and GCC.
Instead of parsing a structured attrs at some later point, we parsed it
right away when parsing the A-Term format, and likewise serialize it to
`__json = <JSON dump>` when serializing a derivation to A-Term.
The JSON format can directly contain the JSON structured attrs without
so encoding it, so we just do that.
Add a new setting to warn about path literals that don't start with "." or "/". When enabled,
expressions like `foo/bar` will emit a warning suggesting to use `./foo/bar` instead.
A functional test is included.
The setting defaults to false for backward compatibility but could eventually default to true in
the future.
Closes: #13374
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
All of the existing tests only run on x86_64-linux and
the whole `nixpkgsFor` makes it hard to reuse those for
e.g. running the nixos tests with a sanitizer build of nix.
This rips off the bandaid and removes the `nixpkgsFor` parameter
in favor of a single instance of nixpkgs with a separate `nixComponents`.
This command fetches all inputs of a flake in parallel.
Example runtime for
$ chmod -R u+w /tmp/nix2; rm -rf /tmp/nix2; rm ~/.cache/nix/fetcher-cache-v3.sqlite*; rm -rf ~/.cache/nix/tarball-cache/ ~/.cache/nix/gitv3/; time nix flake prefetch-inputs --store /tmp/nix2 https://api.flakehub.com/f/pinned/informalsystems/cosmos.nix/0.3.0/018ce9ed-d0be-7ce5-81b6-a3c6e3ae1187/source.tar.gz
with http-connections = 1:
real 4m11.859s
user 2m6.931s
sys 0m25.619s
and http-connections = 25 (the default):
real 0m57.146s
user 2m49.506s
sys 0m36.008s
This moves out the checks that get run in GHA CI into ci/gha/tests
folder and splits those into `topLevel` and `componentTests` attributes.
The idea behind this is to make it easier to parametrize tests that can
be run with sanitizers in order to run those as a matrix of jobs. The same
can be said for static builds.
Existing stdenv selection infrastructure via `lib.makeComponents` would
also allow us to switch over to using `clangStdenv` to significantly speed
up pre-merge CI (though the default stdenv would still be used for non-overridable
topLevel checks, like installer artifacts).
Prior patches in 54dc5314e8
and 6db6190002 fixed the default
system double for i686 and ppc/ppc64. This also patch also covers
32 bit arm and mips. ARM cpu names are taken from host_machine.cpu()
for a lack of a better option, but host_machine.cpu_family() is
preferred, since that is supposed to be somewhat standard for cross
files. Endianness is handled correctly by looking at host_machine.endian().
This also updates the documentation to be up to date to how system cpu
is translated from the host_machine specification.
Before this change, if you were cross compiling Nix, then the nix-manual
subproject would never get built. In some situations, it makes sense to
not build the nix-manual subproject when cross compiling. For example,
if the build system is x86_64 and the host system is riscv64, then it
makes sense to not build the manual. Building the manual requires
executing certain build artifacts, and you can’t run x86_64 executables
on riscv64 systems.
That being said, there are some situations where it does make sense to
build the nix-manual subproject when cross compiling. For example, if
the build system is x86_64 and the host system is i686, then it doesn’t
make sense to not build the manual. You can run i686 executables on
x86_64 systems just fine.
This change makes it so that the nix-manual subproject will sometimes
get built when cross compiling. Specifically, the nix-manual subproject
will get built as long as the doc-gen option is enabled and the build
system is capable of running host binaries.
---
The main motivation behind this change is to fix this Nixpkgs issue [1].
Building pkgs.nixStatic counts as cross compiling Nix, and
pkgs.nixStatic is supposed to produce a man output. Building
pkgs.nixStatic currently fails because it isn’t actually producing a man
output. That issue will go away once this commit gets backported to Nix
2.28.x.
[1]: <https://github.com/NixOS/nixpkgs/issues/426410>
SHA-256 is Git's next hash algorithm. The world is still basically stuck
on SHA-1 with git, but shouldn't be. We can at least do our part to get
ready.
On the C++ implementation side, only a little bit of generalization was
needed, and that was fairly straight-forward. The tests (unit and
system) were actually bigger, and care was taken to make sure they were
all cover both algorithms equally.
For regular, non-executable files, there is no str("") between str("regular")
and str("contents"). Note that str("") is exactly 8 zero bytes, while just ""
is actual empty string (0 bytes).
libfetchers uses `git_mempack_write_thin_pack` which was introduced in libgit2-1.9.0
This avoids error like:
../src/libfetchers/git-utils.cc: In member function ‘virtual void nix::GitRepoImpl::flush()’:
../src/libfetchers/git-utils.cc:270:13: error: ‘git_mempack_write_thin_pack’ was not declared in this scope
270 | git_mempack_write_thin_pack(mempack_backend, packBuilder.get())
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
on older libgit2 (like 1.7.2 in Centos Stream 10)
If the reference for git+file is an annotated tag, the revision will
differ than when it's fetched using other fetchers such as `github:`
since Github seems to automatiacally peel to the underlying commit.
Turns out that rev-parse has the capability through it's syntax to
request the underlying commit by "peeling" using the `^{commit}` syntax.
This is safe to apply in all scenarios where the goal is to get an
underlying commit.
fixes#11266
I've missed this while reviewing 6db6190002.
I only built big endian ppc64, so that didn't occur to me.
From meson manual:
> Those porting from autotools should note that Meson does not add
> endianness to the name of the cpu_family. For example, autotools will
> call little endian PPC64 "ppc64le", Meson will not, you must also check
> the .endian() value of the machine for this information.
This code should handle that correctly.
builtins.fetchGit is not using the cached Git directory if
packed-references are used.
This is because the ref file for the fetchGit `refs/heads/master` is
used to check the mtime for whether to cache or not.
Let's at least codify this failure in a test case.
This is a follow-up to 6ec50ba736, which
also almost halves the compile times on clang for subprojects that use PCH.
`-fpch-instantiate-templates` is a clang-only option to force the instantiation
of templates once in the PCH itself, not all of the translation units that
it gets included to. This really cuts down on the overhead from nlohmann::json
and std::format code:
48244 ms: nlohmann::basic_json<>::parse<const char *> (76 times, avg 634 ms)
36193 ms: nlohmann::basic_json<>::basic_json (310 times, avg 116 ms)
28307 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (76 times, avg 372 ms)
20334 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (76 times, avg 267 ms)
17387 ms: nlohmann::basic_json<>::json_value::json_value (389 times, avg 44 ms)
16822 ms: std::vformat_to<std::__format::_Sink_iter<char>> (76 times, avg 221 ms)
16771 ms: std::__format::__do_vformat_to<std::__format::_Sink_iter<char>, char... (76 times, avg 220 ms)
12160 ms: std::vformat_to<std::__format::_Sink_iter<wchar_t>> (76 times, avg 160 ms)
12127 ms: std::__format::__do_vformat_to<std::__format::_Sink_iter<wchar_t>, w... (76 times, avg 159 ms)
10397 ms: nlohmann::detail::json_sax_dom_callback_parser<nlohmann::basic_json<... (76 times, avg 136 ms)
9118 ms: nlohmann::basic_json<>::data::data (76 times, avg 119 ms)
Initially done by Jade Lovelace <lix@jade.fyi> in https://gerrit.lix.systems/c/lix/+/1842.
We are doing basically the same, but unconditionally. It would be
a huge pain to add a pch option for all subprojects to just support the
usecase of using clangd in a gcc devshell.
In total, this basically halves the frontend times for nix-util and nix-store
to the point that the most expensive part of the build is linking.
(nix-store before):
```
**** Time summary:
Compilation (77 times):
Parsing (frontend): 243.4 s
Codegen & opts (backend): 140.3 s
```
(nix-store after):
```
**** Time summary:
Compilation (77 times):
Parsing (frontend): 120.2 s
Codegen & opts (backend): 141.2 s
```
This adds a GHA jobs to help analyze build times
and its regressions. It is based on `clangStdenv` with `-ftime-trace`
together with `ClangBuildAnalyzer` to prepare markdown summary for
individual components.
This also has the minor benefit of dogfooding CA and impure derivations.
Users can access these through the `lib.makeComponents` return value,
so it's helpful to briefly explain some of them.
This doesn't replace `meta.description`, but supplements it.
(TODO: improve `meta.description`)
This was carefully refactored by inlining some Nixpkgs definitions
like `generateSplicesForMkScope`, so the memoization properties
should remain the same.
Boost.URL is a significantly more RFC-compliant parser
than what libutil currently has a bundle of incomprehensible
regexes.
One aspect of this change is that RFC4007 ZoneId IPv6 literals
are represented in URIs according to RFC6874 [1].
Previously they were represented naively like so: [fe80::818c:da4d:8975:415c\%enp0s25].
This is not entirely correct, because the percent itself has to be pct-encoded:
> "%" is always treated as
an escape character in a URI, so, according to the established URI
syntax [RFC3986] any occurrences of literal "%" symbols in a URI MUST
be percent-encoded and represented in the form "%25". Thus, the
scoped address fe80::a%en1 would appear in a URI as
http://[fe80::a%25en1].
[1]: https://datatracker.ietf.org/doc/html/rfc6874
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
The myriad of hand-rolled URL parsing and validation code
is a constant source of problems. Regexes are not a great way
of writing parsers and there's a history of getting them wrong.
Boost.URL is a good library we can outsource most of the heavy
lifting to.
* It is tough to contribute to a project that doesn't use a formatter,
* It is extra hard to contribute to a project which has configured the formatter, but ignores it for some files
* Code formatting makes it harder to hide obscure / weird bugs by accident or on purpose,
Let's rip the bandaid off?
Note that PRs currently in flight should be able to be merged relatively easily by applying `clang-format` to their tip prior to merge.
Previously, `nix help shell` failed with "Nix has no subcommand 'shell'" despite `nix shell --help`
working correctly. This happened because the `shell` command is actually an alias for `env shell`,
and the help system wasn't resolving aliases when looking up documentation.
This patch modifies the `showHelp` function to check for and resolve aliases before generating the
manpage name, ensuring that shorthand commands like `shell` get proper help documentation.
Closes: #13431
The default constructor for Attr was not initializing the value pointer,
which could lead to undefined behavior when the uninitialized pointer is
accessed. This was caught by clang-tidy's UninitializedObject check.
This fixes the warning:
1 uninitialized field at the end of the constructor call
[clang-analyzer-optin.cplusplus.UninitializedObject]
The s3.hh public header was incorrectly including store-config-private.hh
instead of the public config.hh. Since NIX_WITH_S3_SUPPORT is defined in
the public config, this caused clang-tidy to report it as undefined.
Move init() call from constructor to openStore() method to avoid calling
virtual methods during object construction. This prevents undefined
behavior when virtual methods are called before the object is fully
constructed.
Move init() call from constructor to openStore() method to avoid calling
virtual methods during object construction. This prevents undefined
behavior when virtual methods are called before the object is fully
constructed.
When `file://` is used accidentally in a flake as the source it is
expected to be a tarball by default.
Add some friendlier error messages to either inform the user this is not
in fact a tarball or if it's a git directory, let them know they can use
`git+file`.
fixes#12935
Users have complained that fetchGit is flaky however the culprit is
likely that `git fetch` was unable itself to download the repository for
whatever reason (i.e. poor network etc..)
Nothing was checking the status of `git fetch` and the error message
that would eventually surface to the users were that the commit was not
found.
Add explicit error checking for status code from `git fetch` and return
a message earlier on to indicate that the failure was from that point.
fixes#10431
We’re already allowing `/tmp` anyway, so this should be harmless,
and it fixes a regression in the default configuration caused by
moving the build directories out of `temp-dir`. (For instance, that
broke the Lix `guessOrInventPath.sockets` test.)
Note that removing `/tmp` breaks quite a few builds, so although it may
be a good idea in general it would require work on the Nixpkgs side.
Fixes: 749afbbe99fd7b45f828b72628252feba9241362
Change-Id: I6a6a69645f429bc50d4cb24283feda3d3091f534
(This is a cherry-pick of commit d1db3e5fa3faa43b3d2f2e2e843e9cfc1e6e1b71)
Lix patch: https://gerrit.lix.systems/c/lix/+/3500
When AWS credentials expired, in some scenarios they led to the
nix process aborting with an error similar to ' Unable to parse ExceptionName: ExpiredToken'.
This change updates the S3 handling code such that those errors are treated like 403s or 404s.
Closes#13459
c39cc00404 has added assertions for
all Value accesses and the following case has started failing with
an `unreachable`:
(/tmp/fun.nix):
```nix
{a}: a
```
```
$ nix eval --impure --expr 'import /tmp/fun.nix {a="a";b="b";}'
```
This would crash:
```
terminating due to unexpected unrecoverable internal error: Unexpected condition in getStorage at ../include/nix/expr/value.hh:844
```
This is not a regression, but rather surfaces an existing problem, which previously
was left undiagnosed. In the case of an import `fun` is the `import` primOp, so that read is invalid
and previously this resulted in an access into an inactive union member, which is UB.
The correct thing to use is `vCur`. Identical problem also affected the case of a missing argument.
Add previously failing test cases to the functional/lang test suite.
Fixes#13448.
Fixes:
[261/394] Linking target src/libexpr/libnixexpr.so
In function ‘copy’,
inlined from ‘__ct ’ at /nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/basic_string.h:688:23,
inlined from ‘operator+’ at /nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/basic_string.h:3735:43,
inlined from ‘operator()’ at ../src/libexpr/primops/fetchClosure.cc:127:58,
inlined from ‘prim_fetchClosure’ at ../src/libexpr/primops/fetchClosure.cc:132:88:
/nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/char_traits.h:427:56: warning: ‘__builtin_memcpy’ writing 74 bytes into a region of size 16 overflows the destination [-Wstringop-overflow=]
427 | return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n));
| ^
../src/libexpr/primops/fetchClosure.cc: In function ‘prim_fetchClosure’:
../src/libexpr/primops/fetchClosure.cc:132:88: note: at offset 16 into destination object ‘<anonymous>’ of size 32
132 | fromPath = state.coerceToStorePath(attr.pos, *attr.value, context, attrHint());
| ^
-p preserves xattrs and acls which can be incompatible between
filesystems
Unfortunately keep -p on darwin because the bsd coreutils do not
support --preserve.
Fixes#13426
This failed on macOS:
nix-store-tests-run> C++ exception with description "../nix_api_store.cc:33: nix_err_code(ctx) != NIX_OK, message: error: getting status of '/nix/var/nix/daemon-socket/socket': Operation not permitted" thrown in the test body.
This changes makes nix detect a machines available cores automatically whenever build-cores is set to 0.
So far, nix simply passed NIX_BUILD_CORES=0 whenever build-cores is set to 0. (only when build-cores is unset it was detecting cores automatically)
The behavior of passing NIX_BUILD_CORES=0 leads to a performance penalty when sourcing nixpkgs' generic builder's `setup.sh`, as setup.sh has to execute `nproc`. This significantly slows down sourcing of setup.sh
The use of R"(...)" added a bunch of unnecessary whitespace, e.g.
error:
Unable to start any build;
either increase '--max-jobs' or enable remote builds.
For more information run 'man nix.conf' and search for '/machines'.
Now we get
error: Unable to start any build; either increase '--max-jobs' or enable remote builds.
For more information run 'man nix.conf' and search for '/machines'.
This shaves off a very significand amount of memory used
for evaluation as well as reduces the GC-managed heap.
Previously the union discriminator (InternalType) was
stored as a separate field in the Value, which takes up
whole 8 bytes due to padding needed for member alignment.
This effectively wasted 7 whole bytes of memory. Instead
of doing that InternalType is instead packed into pointer
alignment niches. As it turns out, there's more than enough
unused bits there for the bit packing to be effective.
See the doxygen comment in the ValueStorage specialization
for more details.
This does not add any performance overhead, even though
we now consistently assert the InternalType in all getters.
This can also be made atomic with a double width compare-and-swap
instruction on x86_64 (CMPXCHG16B instruction) for parallel evaluation.
This factors out most of the value representation into a mixin class.
`finishValue` is now gone for good and replaced with a simple template
function `setStorage` which derives the type information/disriminator from
the type of the argument. Likewise, reading of the value goes through function
template `getStorage`.
An empty type `Null` is introduced to make the bijection InternalType <-> C++ type
complete.
Previous use of symlink_status() always translated into a stat call, leading
to huge performance penalties for by-name-overlay in nixpkgs. The comment
below references the possible caching, but that seemed to be erroneous, since
the correct way to make use of the caching API is by calling a bunch of `is_*`
functions [1]. For example, here's how libstdc++ does that [2], [3].
This translates to great nixpkgs eval performance improvements:
```
Benchmark 1: GC_INITIAL_HEAP_SIZE=4G result/bin/nix-instantiate ../nixpkgs -A hello --readonly-mode
Time (mean ± σ): 186.7 ms ± 6.7 ms [User: 121.3 ms, System: 64.9 ms]
Range (min … max): 179.4 ms … 201.6 ms 16 runs
Benchmark 2: GC_INITIAL_HEAP_SIZE=4G nix-instantiate ../nixpkgs -A hello --readonly-mode
Time (mean ± σ): 230.6 ms ± 5.0 ms [User: 126.9 ms, System: 103.1 ms]
Range (min … max): 225.1 ms … 241.4 ms 13 runs
```
[1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0317r1.html
[2]: 8ea555b7b4/libstdc%2B%2B-v3/include/bits/fs_dir.h (L341-L348)
[3]: 8ea555b7b4/libstdc%2B%2B-v3/include/bits/fs_dir.h (L161-L163)
These wrappers don't seem to be used anywhere in and out of tree.
Also the declaration in the header has an incorrect function name `maybeParseFlake`.
Closes#11948
Fixes
error:
… while processing sandbox path '/private/tmp/nix-shell.0MDgyx/nix-test/ca/repl/store/nix/var/nix/builds/nix-build-simple.drv-65916-3910734210' (/private/tmp/nix-shell.0MDgyx/nix-test/ca/repl/store)
error: 'nix' is too short to be a valid store path
which happened because we were now putting the build directory
underneath the store directory.
If a build directory is accessible to other users it is possible to
smuggle data in and out of build directories. Usually this is only
a build purity problem, but in combination with other issues it can
be used to break out of a build sandbox. to prevent this we default
to using a subdirectory of nixStateDir (which is more restrictive).
(cherry picked from pennae Lix commit 55b416f6897fb0d8a9315a530a9b7f0914458ded)
(store setting done by roberth)
In the case the deletion fails, we should ensure that the temporary
directory cannot be used for nefarious purposes.
Change-Id: I498a2dd0999a74195d13642f44a5de1e69d46120
Signed-off-by: Raito Bezarius <raito@lix.systems>
When calling `_deletePath` with a parent file descriptor, `openat` is
made effective by using relative paths to the directory file descriptor.
To avoid the problem, the signature is changed to resist misuse with an
assert in the prologue of the function.
Change-Id: I6b3fc766bad2afe54dc27d47d1df3873e188de96
Signed-off-by: Raito Bezarius <raito@lix.systems>
This ensures that `passAsFile` data is created inside the expected
temporary build directory by `openat()` from the parent directory file
descriptor.
This avoids a TOCTOU which is part of the attack chain of CVE-????.
Change-Id: Ie5273446c4a19403088d0389ae8e3f473af8879a
Signed-off-by: Raito Bezarius <raito@lix.systems>
`writeFile` lose its `sync` boolean flag to make things simpler.
A new `writeFileAndSync` function is created and all call sites are
converted to it.
Change-Id: Ib871a5283a9c047db1e4fe48a241506e4aab9192
Signed-off-by: Raito Bezarius <raito@lix.systems>
We use it immediately for the build temporary directory.
Change-Id: I180193c63a2b98721f5fb8e542c4e39c099bb947
Signed-off-by: Raito Bezarius <raito@lix.systems>
We now keep around a proper AutoCloseFD around the temporary directory
which we plan to use for openat operations and avoiding the build
directory being swapped out while we are doing something else.
Change-Id: I18d387b0f123ebf2d20c6405cd47ebadc5505f2a
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is useful for certain error recovery paths (no pun intended) that
does not thread through the original path name.
Change-Id: I2d800740cb4f9912e64c923120d3f977c58ccb7e
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is just a code cleanup; it should not be behavior change.
`addWantedOutputs` is removed by introducing `DerivationTrampolineGoal`.
`DerivationGoal` now only tracks a single output, and is back to
tracking a plain store path `drvPath`, not a deriving path one. Its
`addWantedOutputs` method is gone. These changes will allow subsequent
PRs to simplify it greatly.
Because the purpose of each goal is back to being immutable, we can also
once again make `Goal::buildResult` a public field, and get rid of the
`getBuildResult` method. This simplifies things also.
`DerivationTrampolineGoal` is, as the nane is supposed to indicate, a
cheap "trampoline" goal. It takes immutable sets of wanted outputs, and
just kicks of `DerivationGoal`s for them. Since now "actual work" is
done in these goals, it is not wasteful to have separate ones for
separate sets of outputs, even if those outputs (and the derivations
they are from) overlap.
This design is described in more detail in the doc comments on the goal
types, which I've now greatly expanded.
---
This separation of concerns will make it possible for future work on
issues like #11928, and to continue the path of having more goal types,
but each goal type does fewer things (issue #12628).
---
This commit in some sense reverts
f4f28cdd0e, but that one kept around
`addWantedOutputs`. I am quite sure it was having two layers of goals
with `addWantedOutputs` that caused the issues --- restarting logic like
`addWantedOutputs` has is very tempermental! In this version of the
change, we have *zero* layers of `addWantedOutputs` --- no goal type
needs it, or otherwise has a mutable objective --- and so I think this
change is safe.
Co-authored-by: Sergei Zimmerman <145775305+xokdvium@users.noreply.github.com>
Running parallel nix in nix can lead to multiple instances trying to
create the state directories and failing on the `createSymlink` step,
because the link already exists.
`replaceSymlink` is already idempotent, so let's use that.
Resolves#2706
libstdc++'s std::stable_sort and new builtins.sort implementation
special-case ranges with length less than or equal to 16 and delegate
to insertionsort.
Having a larger e2e test would allow catching sort stability issues
at functional level as well.
This prevents C++ level undefined behavior from affecting
the evaluator. Stdlib implementation details should not affect
eval, regardless of the build platform. Even erroneous usage
of `builtins.sort` should not make it possible to crash the
evaluator or produce results that depend on the host platform.
Unlike std::sort and std::stable_sort, this implementation
does not lead to out-of-bounds memory reads or other undefined
behavior when the predicate is not strict weak ordering.
This makes it possible to use this function in libexpr for
builtins.sort, where an incorrectly implemented comparator
in the user nix code currently can crash and burn the evaluator
by invoking C++ UB.
Makes the behavoral change of #13263 without the underlying refactor.
Hopefully this clearly safe from a perf and GC perspective, and will
make it easier to benchmark #13263.
This overload isn't actually necessary anywhere and
doesn't make much sense. The pointers to `Value`s are
themselves const, but the `Value`s are mutable.
A non-const member function implies that the object itself
can be modified but this doesn't make much sense considering
the return type: `Value * const * `, which is a pointer
to a constant array of pointers to mutable values.
It seems obvious that erasing any output paths from pathsInChroot needs
to happen after getPathsInSandbox(), not before.
Signed-off-by: Samuli Thomasson <samuli.thomasson@pm.me>
This adds a meson.format file that mostly mirrors the projects
meson style and a pre-commit hook to enforce this style.
Some low-diff files are formatted.
Seen in https://github.com/DeterminateSystems/nix-src/actions/runs/15590867877/job/43909540271:
nix-functional-tests> grep: repl_output: No such file or directory
nix-functional-tests> +(repl.sh:174) cat repl_output
This is because there is a small possibility that the `nix repl` child
process hasn't created `repl_output` yet. So make sure it exists.
Add helpful context when opening the Nix database lock fails due to
permission errors. Instead of just showing "Permission denied", now
provides guidance about possible causes:
- Running as non-root in a single-user Nix installation
- Nix daemon may have crashed
Invoking `:ll` will start a pager with all variables which have just
been loaded by `:lf`, `:l`, or by a flake provided to `nix repl` as an
argument.
https://github.com/NixOS/nix/issues/11404
When we run `nix repl nixpkgs` we get "Added 6 variables". This is not
useful as it doesn't tell us which variables the flake has exported to
our global repl scope.
This patch prints the name of each variable that was just loaded. We
currently cap printing to 20 variables in order to avoid excessive
prints.
https://github.com/NixOS/nix/issues/11404
`getPrimOp` function was basically identical to existing
`Value::primOpAppPrimOp` modulo some trivial differences.
Makes sense to reuse existing code for that.
There's actually no mutation happening so there's no point in using
a mutable shared_ptr. Furthermore, this makes it much more evident to
the reader that no actual mutation (especially in multithreaded case)
is happening.
Also get rid of redundant constructor that isn't actually used anywhere
other than `Pos::operator std::shared_ptr<Pos>` which just passes in &*this,
(identical to just `this`), which can't be nullptr.
Judging by the comment for `makeEmptySourceAccessor` the prefix has
to be empty:
> Return a source accessor that contains only an empty root directory.
Fixes#13295.
Since the migration to meson precompiled-headers.h isn't actually used
anymore and is just confusing. Meson can't handle shared pch across
subprojects [1] and without that there's no performances benefit of PCH
at all. Also rolling our own support for that isn't trivial. See [2] for
an example of how that would look like.
[1]: https://github.com/mesonbuild/meson/issues/4350
[2]: 22bc8b6473/plugins/meson.build
Clang doesn't like the double indent that is needed for the `if...else`
that is CPP'd away. Adding braces is fine in the `if...else...` case,
and fine as a naked block in the CPP'd away case, and properly-indented
both ways.
These tests have been collected from nixpkgs f870c6ccc8951fc48aeb293cf3e98ade6ac42668
usage of builtins.match for x86_64-linux eval system. At most 2 matching and
non-matching cases are included for each encountered regex. This should
hopefully add more confidence when possibly trying to switch the regex implementation
in the future.
This is the utility changes from #9968, which were easier to rebase
first.
I (@Ericson2314) didn't write this code; I just rebased it.
Co-Authored-By: Artemis Tosini <me@artem.ist>
Co-Authored-By: Audrey Dutcher <audrey@rhelmot.io>
Allows to copy the archive to a remote host and not get
error: cannot add path '/nix/store/01x2k4nlxcpyd85nnr0b9gm89rm8ff4x-source' because it lacks a signature by a trusted key
Before we got something like this but only inside the VM test:
vm-test-run-functional-tests-on-nixos_user> machine # fetchGit.sh: line 286: unexpected EOF while looking for matching `)'
We now try to do not too much in a single line, so that the bash parser does not get confused.
This also seems more readable and better quoted.
This makes the profiler much more useful by actually distiguishing
different derivations being evaluated. This does make the implementation
a bit more convoluted, but I think it's worth it.
Sometimes the profiler might want to do evaluation (e.g. for getting
derivation names). This is not ideal, but is really necessary
to make the profiler stack traces useful for end users.
...and also NIX_STATE_HOME in nix-profile.fish. This is directly
translated from the bash scripts and makes the fish scripts equivalent
in functionality to the bash scripts.
Note that nix-profile.fish checks for NIX_STATE_HOME and
nix-profile-daemon.fish does not, so the two scripts are no longer
identical.
This was discussed in https://github.com/NixOS/nix/issues/8034. I
personally like `PRJ_ROOT`, which hopefully avoids some ambiguity
around with subflakes.
I only implemented this for `nix fmt` because it doesn't let you point
at a flake not on your filesystem.
macOS compilation fixes
This method does *not* create a new type of goal. We instead just make
`DerivationGoal` more sophisticated, which is much easier to do now that
`DerivationBuildingGoal` has been split from it (and so many fields are
gone, or or local variables instead).
This avoids the need for a secondarily trampoline goal that interacted
poorly with `addWantedOutputs`. That, I hope, will mean the bugs from
before do not reappear.
There may in fact be a reason to introduce such a trampoline in the
future, but it would only happen in conjunction with getting rid of
`addWantedOutputs`.
Restores the functionality (and tests) that was reverted in
f4f28cdd0e.
This patch adds support for a native stack sampling
profiler to the evaluator, which saves a collapsed stack
profile information to a configurable location.
Introduced options (in `EvalSettings`):
- `eval-profile-file` - path to the collected profile file.
- `eval-profiler-frequency` - sampling frequency.
- `eval-profiler` - enumeration option for enabling the profiler.
Currently only `flamegraph` is supported, but having this an
enumeration rather than a boolean switch leaves the door open
for other profiler variants (e.g. tracy).
Profile includes the following information on best-effort basis (e.g. some lambdas might
have an undefined name). Callstack information contains:
- Call site location (where the function gets called).
- Primop/lambda name of the function being called.
- Functors/partial applications don't have a name attached to them unlike special-cased primops and lambads.
For cases where callsite location isn't available we have to resort to providing
the location where the lambda itself is defined. This removes some of the confusing
`«none»:0` locations in the profile from previous attempts.
Example usage with piping directly into zstd for compression:
```
nix eval --no-eval-cache nixpkgs#nixosTests.gnome \
--eval-profiler flamegraph \
--eval-profile-file >(zstd -of nix.profile.zstd)
```
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Commit b36637c8f7 set
`__ETC_PROFILE_NIX_SOURCED` globally, but this is not enough to prevent
the script from being run again by child shells, because the
variable was not exported and thus not inherited by any child process.
Exporting the variable also agrees with the bash scripts.
Notably, the old behavior broke `nix develop -c fish` in some cases,
because the profile bin directory got prepended to the path, causing
binaries from the profile to override binareis from the devshell.
This wires up the {pre,post}FunctionCallHook machinery
in EvalState::callFunction and migrates FunctionCallTrace
to use the new EvalProfiler mechanisms for tracing.
Note that branches when the hook gets called are marked with [[unlikely]]
as a hint to the compiler that this is not a hot path. For non-tracing
evaluation this should be a 100% predictable branch, so the performance
cost is nonexistent.
Some measurements to prove support this point:
```
nix build .#nix-cli
nix build github:nixos/nix/d692729759e4e370361cc5105fbeb0e33137ca9e#nix-cli --out-link before
```
(Before)
```
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g before/bin/nix eval nixpkgs#gnome --no-eval-cache" --warmup 4
Benchmark 1: GC_INITIAL_HEAP_SIZE=16g before/bin/nix eval nixpkgs#gnome --no-eval-cache
Time (mean ± σ): 2.517 s ± 0.032 s [User: 1.464 s, System: 0.476 s]
Range (min … max): 2.464 s … 2.557 s 10 runs
```
(After)
```
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g result/bin/nix eval nixpkgs#gnome --no-eval-cache" --warmup 4
Benchmark 1: GC_INITIAL_HEAP_SIZE=16g result/bin/nix eval nixpkgs#gnome --no-eval-cache
Time (mean ± σ): 2.499 s ± 0.022 s [User: 1.448 s, System: 0.478 s]
Range (min … max): 2.472 s … 2.537 s 10 runs
```
This patch adds an EvalProfiler and MultiEvalProfiler that can be used
to insert hooks into the evaluation for the purposes of function tracing
(what function-trace currently does) or for flamegraph/tracy profilers.
See the following commits for how this is supposed to be integrated into
the evaluator and performance considerations.
Ensure relative path inputs are relative to the parent node's _actual_
`outPath`, instead of the subtly different `sourceInfo.outPath`.
Additionally, non-flake inputs now also have a `sourceInfo` attribute.
This fixes the relationship between `self.outPath` and
`self.sourceInfo.outPath` in some edge cases.
Fixes#13164
Previous code had a sneaky bug due to which no caching
actually happened:
```cpp
auto linesForInput = (*lines)[origin->offset];
```
That should have been:
```cpp
auto & linesForInput = (*lines)[origin->offset];
```
See [1].
Now that it also makes sense to make the cache bound in side
in order not to memoize all the sources without freeing any memory.
The default cache size has been chosen somewhat arbitrarily to be ~64k
origins. For reference, 25.05 nixpkgs has ~50k .nix files.
Simple benchmark:
```nix
let
pkgs = import <nixpkgs> { };
in
builtins.foldl' (acc: el: acc + el.line) 0 (
builtins.genList (x: builtins.unsafeGetAttrPos "gcc" pkgs) 10000
)
```
(After)
```
$ hyperfine "result/bin/nix eval -f ./test.nix"
Benchmark 1: result/bin/nix eval -f ./test.nix
Time (mean ± σ): 292.7 ms ± 3.9 ms [User: 131.0 ms, System: 120.5 ms]
Range (min … max): 288.1 ms … 300.5 ms 10 runs
```
(Before)
```
hyperfine "nix eval -f ./test.nix"
Benchmark 1: nix eval -f ./test.nix
Time (mean ± σ): 666.7 ms ± 6.4 ms [User: 428.3 ms, System: 191.2 ms]
Range (min … max): 659.7 ms … 681.3 ms 10 runs
```
If the origin happens to be a `all-packages.nix` or similar in size then the
difference is much more dramatic.
[1]: 22e3f0e987
Try to make `DerivationGoal` care less whether we're working from an
in-memory derivation or not.
It's a clean-up in its own right, but it will also help with other
cleanups under the umbrella of #12628.
Now, each class provides the initial coroutine by value. This avoids
some sketchy virtual function stuff, and will also be further put to
good use in the next commit.
As summarized in
https://github.com/NixOS/nix/issues/77#issuecomment-2843228280 the
motivation is that the complicated retry logic this introduced was
making the cleanup task #12628 harder to accomplish. It was not easy to
ascertain just what policy / semantics the extra control-flow was
implementing, in order to figure out a different way to implementing it
either.
After talking to Eelco about it, he decided we could just....get rid of
the feature entirely! It's a bit scary removing a decade+ old feature,
but I think he is right. See the release notes for more explanation.
This reverts commit 299141ecbd.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
As it turns out using `std::regex` is actually the bottleneck
for root discovery. Just substituting `std::` -> `boost::`
makes root discovery twice as fast (3x if counting only userspace time).
Some rather ad-hoc measurements to motivate the switch:
(On master)
```
nix build github:nixos/nix/1e822bd4149a8bce1da81ee2ad9404986b07914c#nix-cli --out-link result-1e822bd4149a8bce1da81ee2ad9404986b07914c
taskset -c 2,3 hyperfine "result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0"
Benchmark 1: result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0
Time (mean ± σ): 481.6 ms ± 3.9 ms [User: 336.2 ms, System: 142.0 ms]
Range (min … max): 474.6 ms … 487.7 ms 10 runs
```
(After this patch)
```
taskset -c 2,3 hyperfine "result/bin/nix store gc --dry-run --max 0"
Benchmark 1: result/bin/nix store gc --dry-run --max 0
Time (mean ± σ): 254.7 ms ± 9.7 ms [User: 111.1 ms, System: 141.3 ms]
Range (min … max): 246.5 ms … 281.3 ms 10 runs
```
`boost::regex` is a drop-in replacement for `std::regex`, but much faster.
Doing a simple before/after comparison doesn't surface any change in behavior:
```
result/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
```
The "derivation" is one of the key concepts and captures the most distinctive aspect of Nix:
that we work with a certain type data (linked files) in a certain manner (using pure functions).
Here we finally arrange all the important pieces to show how they belong
together, while referring to the respective reference documentation for details.
This change also unbreaks downstream links to `//glossary#gloss-derivation`,
which had been broken by removing the glossary entry
2025-04-11 19:47:28 +02:00
914 changed files with 39197 additions and 30122 deletions
@@ -53,6 +53,11 @@ This command has the following operations:
Download the Nix expressions of subscribed channels and create a new generation.
Update all channels if none is specified, and only those included in *names* otherwise.
> **Note**
>
> Downloaded channel contents are cached.
> Use `--tarball-ttl` or the [`tarball-ttl` configuration option](@docroot@/command-ref/conf-file.md#conf-tarball-ttl) to change the validity period of cached downloads.
-`--list-generations`
Prints a list of all the current existing generations for the
This guide explains how to build and run performance benchmarks in the Nix codebase.
## Overview
Nix uses the [Google Benchmark](https://github.com/google/benchmark) framework for performance testing. Benchmarks help measure and track the performance of critical operations like derivation parsing.
## Building Benchmarks
Benchmarks are disabled by default and must be explicitly enabled during the build configuration. For accurate results, use a debug-optimized release build.
### Development Environment Setup
First, enter the development shell which includes the necessary dependencies:
```bash
nix develop .#native-ccacheStdenv
```
### Configure Build with Benchmarks
From the project root, configure the build with benchmarks enabled and optimization:
The industry term for storage and retrieval systems using [content addressing](#gloss-content-address). A Nix store also has [input addressing](#gloss-input-addressed-store-object), and metadata.
- [derivation]{#gloss-derivation}
A derivation can be thought of as a [pure function](https://en.wikipedia.org/wiki/Pure_function) that produces new [store objects][store object] from existing store objects.
Derivations are implemented as [operating system processes that run in a sandbox](@docroot@/store/building.md#builder-execution).
This sandbox by default only allows reading from store objects specified as inputs, and only allows writing to designated [outputs][output] to be [captured as store objects](@docroot@/store/building.md#processing-outputs).
A derivation is typically specified as a [derivation expression] in the [Nix language], and [instantiated][instantiate] to a [store derivation].
There are multiple ways of obtaining store objects from store derivatons, collectively called [realisation][realise].
[derivation]: #gloss-derivation
- [store derivation]{#gloss-store-derivation}
A single build task.
A [derivation] represented as a [store object].
See [Store Derivation](@docroot@/store/derivation/index.md#store-derivation) for details.
If the special attribute `__structuredAttrs` is set to `true`, the other derivation
attributes are serialised into a file in JSON format. The environment variable
`NIX_ATTRS_JSON_FILE` points to the exact location of that file both in a build
and a [`nix-shell`](../command-ref/nix-shell.md). This obviates the need for
[`passAsFile`](#adv-attr-passAsFile) since JSON files have no size restrictions,
unlike process environments.
attributes are serialised into a file in JSON format.
It also makes it possible to tweak derivation settings in a structured way; see
[`outputChecks`](#adv-attr-outputChecks) for example.
This obviates the need for [`passAsFile`](#adv-attr-passAsFile) since JSON files have no size restrictions, unlike process environments.
It also makes it possible to tweak derivation settings in a structured way;
see [`outputChecks`](#adv-attr-outputChecks) for example.
As a convenience to Bash builders,
Nix writes a script that initialises shell variables
corresponding to all attributes that are representable in Bash. The
environment variable `NIX_ATTRS_SH_FILE` points to the exact
location of the script, both in a build and a
[`nix-shell`](../command-ref/nix-shell.md). This includes non-nested
(associative) arrays. For example, the attribute `hardening.format = true`
ends up as the Bash associative array element `${hardening[format]}`.
See the [corresponding section in the derivation page](@docroot@/store/derivation/index.md#structured-attrs) for further details.
> **Warning**
>
> If set to `true`, other advanced attributes such as [`allowedReferences`](#adv-attr-allowedReferences), [`allowedReferences`](#adv-attr-allowedReferences), [`allowedRequisites`](#adv-attr-allowedRequisites),
> If set to `true`, other advanced attributes such as [`allowedReferences`](#adv-attr-allowedReferences), [`allowedRequisites`](#adv-attr-allowedRequisites),
[`disallowedReferences`](#adv-attr-disallowedReferences) and [`disallowedRequisites`](#adv-attr-disallowedRequisites), maxSize, and maxClosureSize.
will have no effect.
@@ -170,7 +160,6 @@ See the [corresponding section in the derivation output page](@docroot@/store/de
If a derivation has the `requiredSystemFeatures` attribute, then Nix will only build it on a machine that has the corresponding features set in its [`system-features` configuration](@docroot@/command-ref/conf-file.md#conf-system-features).
The Nix language is designed for conveniently creating and composing *derivations*– precise descriptions of how contents of existing files are used to derive new files.
The Nix language is designed for conveniently creating and composing [derivations](@docroot@/glossary.md#gloss-derivation)– precise descriptions of how contents of existing files are used to derive new files.
`<nix/fetchurl.nix>` is also known as the builtin derivation builder `builtin:fetchurl`. It's not to be confused with the evaluation-time function `builtins.fetchurl`, which was not affected by this issue.
# Contributors
## Contributors
This release was made possible by the following 58 contributors:
@@ -82,7 +82,7 @@ This completes the infrastructure overhaul for the [RFC 132](https://github.com/
Although this change is not as critical, we figured it would be good to do this API change at the same time, also.
Also note that we try to keep the C API compatible, but we decided to break this function because it was young and likely not in widespread use yet. This frees up time to make important progress on the rest of the C API.
# Contributors
## Contributors
This earlier-than-usual release was made possible by the following 16 contributors:
The directory in which temporary build directories are created no longer defaults
to `TMPDIR` or `/tmp`, to avoid builders making their directories
world-accessible. This behavior allowed escaping the build sandbox and can
cause build impurities even when not used maliciously. We now default to `builds`
in `NIX_STATE_DIR` (which is `/nix/var/nix/builds` in the default configuration).
- Deprecate manually making structured attrs using the `__json` attribute [#13220](https://github.com/NixOS/nix/pull/13220)
The proper way to create a derivation using [structured attrs] in the Nix language is by using `__structuredAttrs = true` with [`builtins.derivation`].
However, by exploiting how structured attrs are implementated, it has also been possible to create them by setting the `__json` environment variable to a serialized JSON string.
This sneaky alternative method is now deprecated, and may be disallowed in future versions of Nix.
- Rename `nix profile install` to [`nix profile add`] [#13224](https://github.com/NixOS/nix/pull/13224)
The command `nix profile install` has been renamed to [`nix profile add`] (though the former is still available as an alias). This is because the verb "add" is a better antonym for the verb "remove" (i.e. `nix profile remove`). Nix also does not have install hooks or general behavior often associated with "installing".
## Performance improvements
This release has a number performance improvements, in particular:
- Reduce the size of value from 24 to 16 bytes [#13407](https://github.com/NixOS/nix/pull/13407)
This shaves off a very significant amount of memory used for evaluation (~20% percent reduction in maximum heap size and ~17% in total bytes).
The Nix evaluator now supports [stack sampling evaluation profiling](@docroot@/advanced-topics/eval-profiler.md) via the [`--eval-profiler flamegraph`] setting.
It outputs collapsed call stack information to the file specified by
[`--eval-profile-file`] (`nix.profile` by default) in a format directly consumable
by `flamegraph.pl` and compatible tools like [speedscope](https://speedscope.app/).
Sampling frequency can be configured via [`--eval-profiler-frequency`] (99 Hz by default).
Unlike the existing [`--trace-function-calls`], this profiler includes the name of the function
being called when it's available.
- [`nix repl`] prints which variables were loaded [#11406](https://github.com/NixOS/nix/pull/11406)
Instead of `Added <n> variables` it now prints the first 10 variables that were added to the global scope.
This is useful when using [`nix flake archive`] with the destination set to a remote store.
- Emit warnings for IFDs with [`trace-import-from-derivation`] option [#13279](https://github.com/NixOS/nix/pull/13279)
While we have the setting [`allow-import-from-derivation`] to deny import-from-derivation (IFD), sometimes users would like to observe IFDs during CI processes to gradually phase out the idiom. The new setting `trace-import-from-derivation`, when set, logs a simple warning to the console.
New setting [`json-log-path`] that sends a copy of all Nix log messages (in JSON format) to a file or Unix domain socket.
- Non-flake inputs now contain a `sourceInfo` attribute [#13164](https://github.com/NixOS/nix/issues/13164) [#13170](https://github.com/NixOS/nix/pull/13170)
Flakes have always had a `sourceInfo` attribute which describes the source of the flake.
The `sourceInfo.outPath` is often identical to the flake's `outPath`. However, it can differ when the flake is located in a subdirectory of its source.
Non-flake inputs (i.e. inputs with [`flake = false`]) can also be located at some path _within_ a wider source.
This usually happens when defining a relative path input within the same source as the parent flake, e.g. `inputs.foo.url = ./some-file.nix`.
Such relative inputs will now inherit their parent's `sourceInfo`.
This also means it is now possible to use `?dir=subdir` on non-flake inputs.
This iterates on the work done in 2.26 to improve relative path support ([#10089](https://github.com/NixOS/nix/pull/10089)),
and resolves a regression introduced in 2.28 relating to nested relative path inputs ([#13164](https://github.com/NixOS/nix/issues/13164)).
Previously it used libstdc++'s `std::stable_sort()`. However, that implementation is not reliable if the user-supplied comparison function is not a strict weak ordering.
Since Nix 1.3 ([commit `299141e`] in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference).
This feature is now removed.
In the worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute.
In practice, however, we doubt even more building is very likely to happen.
Remote stores that are missing dependencies in arbitrary ways (e.g. corruption) don't seem to be very common.
On the contrary, when remote stores fail to implement the [closure property](@docroot@/store/store-object.md#closure-property), it is usually an *intentional* choice on the part of the remote store, because it wishes to serve as an "overlay" store over another store, such as `https://cache.nixos.org`.
If an "incomplete closure" is encountered in that situation, the right fix is not to do some sort of "franken-building" as this feature implemented, but instead to make sure both substituters are enabled in the settings.
(In the future, we should make it easier for remote stores to indicate this to clients, to catch settings that won't work in general before a missing dependency is actually encountered.)
## Contributors
This release was made possible by the following 32 contributors:
-`build-cores = 0` now auto-detects CPU cores [#13402](https://github.com/NixOS/nix/pull/13402)
When `build-cores` is set to `0`, Nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
Fixed some outstanding issues with Git LFS and SSH.
* Added support for `NIX_SSHOPTS`.
* Properly use the parsed port from URL.
* Better use of the response of `git-lfs-authenticate` to determine API endpoint when the API is not exposed on port 443.
- Add support for `user@address:port` syntax in store URIs [#7044](https://github.com/NixOS/nix/issues/7044) [#3425](https://github.com/NixOS/nix/pull/3425)
It's now possible to specify the port used for SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or the `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable.
This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
- Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874 [#13445](https://github.com/NixOS/nix/pull/13445)
Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.
- Use WAL mode for SQLite cache databases [#13800](https://github.com/NixOS/nix/pull/13800)
Previously, Nix used SQLite's "truncate" mode for caches. However, this could cause a Nix process to block if another process was updating the cache. This was a problem for the flake evaluation cache in particular, since it uses long-running transactions. Thus, concurrent Nix commands operating on the same flake could be blocked for an unbounded amount of time. WAL mode avoids this problem.
This change required updating the versions of the SQLite caches. For instance, `eval-cache-v5.sqlite` is now `eval-cache-v6.sqlite`.
- Enable parallel marking in bdwgc [#13708](https://github.com/NixOS/nix/pull/13708)
Previously marking was done by only one thread, which takes a long time if the heap gets big. Enabling parallel marking speeds up evaluation a lot, for example (on a Ryzen 9 5900X 12-Core):
*`nix search nixpkgs` from 24.3s to 18.9s.
* Evaluating the `NixOS/nix/2.21.2` flake regression test from 86.1s to 71.2s.
- New command `nix flake prefetch-inputs` [#13565](https://github.com/NixOS/nix/pull/13565)
This command fetches all inputs of a flake in parallel. This can be a lot faster than the serialized on-demand fetching during regular flake evaluation. The downside is that it may fetch inputs that aren't normally used.
This setting, when enabled, causes Nix to emit warnings when encountering relative path literals that don't start with `.` or `/`, for instance suggesting that `foo/bar` should be rewritten to `./foo/bar`.
- When updating a lock, respect the input's lock file [#13437](https://github.com/NixOS/nix/pull/13437)
For example, if a flake has a lock for `a` and `a/b`, and we change the flakeref for `a`, previously Nix would fetch the latest version of `b` rather than using the lock for `b` from `a`.
- Implement support for Git hashing with SHA-256 [#13543](https://github.com/NixOS/nix/pull/13543)
The experimental support for [Git-hashing](@docroot@/development/experimental-features.md#xp-feature-git-hashing) store objects now also includes support for SHA-256, not just SHA-1, in line with upstream Git.
## Contributors
This release was made possible by the following 34 contributors:
- John Soo [**(@jsoo1)**](https://github.com/jsoo1)
- Alan Urmancheev [**(@alurm)**](https://github.com/alurm)
@@ -9,7 +9,7 @@ This is where Nix distinguishes itself.
## Store Derivation {#store-derivation}
A derivation is a specification for running an executable on precisely defined input to produce on more [store objects][store object].
A derivation is a specification for running an executable on precisely defined input to produce one or more [store objects][store object].
These store objects are known as the derivation's *outputs*.
Derivations are *built*, in which case the process is spawned according to the spec, and when it exits, required to leave behind files which will (after post-processing) become the outputs of the derivation.
@@ -138,6 +138,17 @@ See [Wikipedia](https://en.wikipedia.org/wiki/Argv) for details.
Environment variables which will be passed to the [builder](#builder) executable.
#### Structured Attributes {#structured-attrs}
Nix also has special support for embedding JSON in the derivations.
The environment variable `NIX_ATTRS_JSON_FILE` points to the exact location of that file both in a build and a [`nix-shell`](@docroot@/command-ref/nix-shell.md).
As a convenience to Bash builders, Nix writes a script that initialises shell variables corresponding to all attributes that are representable in Bash.
The environment variable `NIX_ATTRS_SH_FILE` points to the exact location of the script, both in a build and a [`nix-shell`](@docroot@/command-ref/nix-shell.md).
This includes non-nested (associative) arrays.
For example, the attribute `hardening.format = true` ends up as the Bash associative array element `${hardening[format]}`.
### Placeholders
Placeholders are opaque values used within the [process creation fields] to [store objects] for which we don't yet know [store path]s.
@@ -162,7 +173,7 @@ There are two types of placeholder, corresponding to the two cases where this pr
> **Explanation**
>
> In general, we need to realise [realise] a [store object] in order to be sure to have a store object for it.
> In general, we need to [realise] a [store object] in order to be sure to have a store object for it.
> But for these two cases this is either impossible or impractical:
>
> - In the output case this is impossible:
@@ -189,7 +200,7 @@ This ensures that there is a canonical [store path] used to refer to the derivat
> **Note**
>
> Currently, the canonical encoding for every derivation is the "ATerm" format,
> but this is subject to change for types derivations which are not yet stable.
> but this is subject to change for the types of derivations which are not yet stable.
Regardless of the format used, when serializing a derivation to a store object, that store object will be content-addressed.
@@ -282,7 +293,7 @@ type DerivingPath = ConstantPath | OutputPath;
Under this extended model, `DerivingPath`s are thus inductively built up from a root `ConstantPath`, wrapped with zero or more outer `OutputPath`s.
@@ -23,7 +23,7 @@ The output spec for an output with a fixed content addresses additionally contai
> **Design note**
>
> In principle, the output spec could also specify the references the store object should have, since the references and file system objects are equally parts of a content-addressed store object proper that contribute to its content-addressed.
> However, at this time, the references are not not done because all fixed content-addressed outputs are required to have no references (including no self-reference).
> However, at this time, the references are not done because all fixed content-addressed outputs are required to have no references (including no self-reference).
>
> Also in principle, rather than specifying the references and file system object data with separate hashes, a single hash that constraints both could be used.
> This could be done with the final store path's digest, or better yet, the hash that will become the store path's digest before it is truncated.
@@ -116,7 +116,7 @@ Because the derivation output is not fixed (just like with [input addressing]),
> (The "environment", in this case, consists of attributes such as the Operating System Nix runs atop, along with the operating-system-specific privileges that Nix has been granted.
> Because of how conventional operating systems like macos, Linux, etc. work, granting builders *fewer* privileges may ironically require that Nix be run with *more* privileges.)
That said, derivations producing floating content-addressed outputs may declare their builders as impure (like the builders of derivations producing producing fixed outputs).
That said, derivations producing floating content-addressed outputs may declare their builders as impure (like the builders of derivations producing fixed outputs).
This is provisionally supported as part of the [`impure-derivations`][xp-feature-impure-derivations] experimental feature.
### Compatibility negotiation
@@ -144,7 +144,7 @@ A *deterministic* content-addressing derivation should produce outputs with the
The choice of provisional store path can be thought of as an impurity, since it is an arbitrary choice.
If provisional outputs paths are deterministically chosen, we are in the first branch of part (1).
The builder the data it produces based on it in arbitrary ways, but this gets us closer to to [input addressing].
The builder the data it produces based on it in arbitrary ways, but this gets us closer to [input addressing].
Deterministically choosing the provisional path may be considered "complete sandboxing" by removing an impurity, but this is unsatisfactory
@@ -83,7 +83,7 @@ The rules for this are fairly concise:
- A content-addressing derivation may be pure or impure
- If it is impure, it may be be fixed (typical), or it may be floating if the additional [`impure-derivations`][xp-feature-impure-derivations] experimental feature is enabled.
- If it is impure, it may be fixed (typical), or it may be floating if the additional [`impure-derivations`][xp-feature-impure-derivations] experimental feature is enabled.
We can take the [transitive closure] of the references graph, which any pair of store objects have an edge not if there is a single reference from the first to the second, but a path of one or more references from the first to the second.
The *requisites* of a store object are all store objects reachable by paths of references which start with given store object's references.
# Determine if we could use the multi-user installer or not
if["$(uname -s)"="Linux"];then
echo"Note: a multi-user installation is possible. See https://nixos.org/manual/nix/stable/installation/installing-binary.html#multi-user-installation" >&2
if["$OS"="Linux"]||["$OS"="FreeBSD"];then
echo"Note: a multi-user installation is possible. See https://nix.dev/manual/nix/stable/installation/installing-binary.html#multi-user-installation" >&2
fi
case"$(uname -s)" in
case"$OS" in
"Darwin")
INSTALL_MODE=daemon;;
*)
@@ -60,7 +62,7 @@ while [ $# -gt 0 ]; do
ACTION=install
;;
--no-daemon)
if["$(uname -s)"="Darwin"];then
if["$OS"="Darwin"];then
printf'\e[1;31mError: --no-daemon installs are no-longer supported on Darwin/macOS!\e[0m\n' >&2
exit1
fi
@@ -96,7 +98,7 @@ while [ $# -gt 0 ]; do
echo" providing multi-user support and better isolation for local builds."
echo" Both for security and reproducibility, this method is recommended if"
echo" supported on your platform."
echo" See https://nixos.org/manual/nix/stable/installation/installing-binary.html#multi-user-installation"
echo" See https://nix.dev/manual/nix/stable/installation/installing-binary.html#multi-user-installation"
echo""
echo" --no-daemon: Simple, single-user installation that does not require root and is"
echo" trivial to uninstall."
@@ -123,6 +125,13 @@ while [ $# -gt 0 ]; do
done
if["$INSTALL_MODE"="daemon"];then
# Check for bash on systems that don't have it by default
printf'\e[1;31mError: bash is required for multi-user installation but was not found.\e[0m\n' >&2
printf'Please install bash first:\n' >&2
printf' pkg install bash\n' >&2
exit1
fi
printf'\e[1;31mSwitching to the Multi-user Installer\e[0m\n'
exec"$self/install-multi-user"$ACTION
exit0
@@ -144,7 +153,7 @@ if ! [ -e "$dest" ]; then
fi
if ! [ -w "$dest"];then
echo"$0: directory $dest exists, but is not writable by you. This could indicate that another user has already performed a single-user installation of Nix on this system. If you wish to enable multi-user support see https://nixos.org/manual/nix/stable/installation/multi-user.html. If you wish to continue with a single-user install for $USER please run 'chown -R $USER$dest' as root." >&2
echo"$0: directory $dest exists, but is not writable by you. This could indicate that another user has already performed a single-user installation of Nix on this system. If you wish to enable multi-user support see https://nix.dev/manual/nix/stable/installation/multi-user.html. If you wish to continue with a single-user install for $USER please run 'chown -R $USER$dest' as root." >&2
exit1
fi
@@ -167,7 +176,11 @@ for i in $(cd "$self/store" >/dev/null && echo ./*); do
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
