Closes#10585
As it turns out, libseccomp maintains an internal syscall table and
validates each rule against it. This means that when using libseccomp
2.5.4 or older, one may pass `452` as syscall number against it, but
since it doesn't exist in the internal structure, `libseccomp` will refuse
to create a filter for that. This happens with nixpkgs-23.11, i.e. on
stable NixOS and when building Nix against the project's flake.
To work around that
* a backport of libseccomp 2.5.5 on upstream nixpkgs has been
scheduled[1].
* the package now uses libseccomp 2.5.5 on its own already. This is to
provide a quick fix since the correct fix for 23.11 is still a staging cycle
away.
It must not be possible to build a Nix with an incompatible libseccomp
version (nothing can be built in a sandbox on Linux!), so configure.ac
rejects libseccomp if `__SNR_fchmodat2` is not defined.
We still need the compat header though since `SCMP_SYS(fchmodat2)`
internally transforms this into `__SNR_fchmodat2` which points to
`__NR_fchmodat2` from glibc 2.39, so it wouldn't build on glibc 2.38.
The updated syscall table from libseccomp 2.5.5 is NOT used for that
step, but used later, so we need both, our compat header and their
syscall table 🤷
[1] https://github.com/NixOS/nixpkgs/pull/306070
I've added the new local.mk to the package sources. While this
should not be needed for the build, it is the simplest solution,
and won't cause many extra rebuilds, because the file won't change
very often.
This reverts commit 62feb5ca09263c78ddb692836228223e5b58d3ae.
It runs as part of the functional tests, which control the environment,
solving some of the problems a default config has when run in the
sandbox.
With Linux kernel >=6.6 & glibc 2.39 a `fchmodat2(2)` is available that
isn't filtered away by the libseccomp sandbox.
Being able to use this to bypass that restriction has surprising results
for some builds such as lxc[1]:
> With kernel ≥6.6 and glibc 2.39, lxc's install phase uses fchmodat2,
> which slips through 9b88e52846/src/libstore/build/local-derivation-goal.cc (L1650-L1663).
> The fixupPhase then uses fchmodat, which fails.
> With older kernel or glibc, setting the suid bit fails in the
> install phase, which is not treated as fatal, and then the
> fixup phase does not try to set it again.
Please note that there are still ways to bypass this sandbox[2] and this is
mostly a fix for the breaking builds.
This change works by creating a syscall filter for the `fchmodat2`
syscall (number 452 on most systems). The problem is that glibc 2.39
and seccomp 2.5.5 are needed to have the correct syscall number available
via `__NR_fchmodat2` / `__SNR_fchmodat2`, but this flake is still on
nixpkgs 23.11. To have this change everywhere and not dependent on the
glibc this package is built against, I added a header
"fchmodat2-compat.hh" that sets the syscall number based on the
architecture. On most platforms its 452 according to glibc with a few
exceptions:
$ rg --pcre2 'define __NR_fchmodat2 (?!452)'
sysdeps/unix/sysv/linux/x86_64/x32/arch-syscall.h
58:#define __NR_fchmodat2 1073742276
sysdeps/unix/sysv/linux/mips/mips64/n32/arch-syscall.h
67:#define __NR_fchmodat2 6452
sysdeps/unix/sysv/linux/mips/mips64/n64/arch-syscall.h
62:#define __NR_fchmodat2 5452
sysdeps/unix/sysv/linux/mips/mips32/arch-syscall.h
70:#define __NR_fchmodat2 4452
sysdeps/unix/sysv/linux/alpha/arch-syscall.h
59:#define __NR_fchmodat2 562
I tested the change by adding the diff below as patch to
`pkgs/tools/package-management/nix/common.nix` & then built a VM from
the following config using my dirty nixpkgs master:
{
vm = { pkgs, ... }: {
virtualisation.writableStore = true;
virtualisation.memorySize = 8192;
virtualisation.diskSize = 12 * 1024;
nix.package = pkgs.nixVersions.nix_2_21;
};
}
The original issue can be triggered via
nix build -L github:nixos/nixpkgs/d6dc19adbda4fd92fe9a332327a8113eaa843894#lxc \
--extra-experimental-features 'nix-command flakes'
however the problem disappears with this patch applied.
Closes#10424
[1] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2031073804
[2] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2030844251
With Nix 2.3, it was possible to pass a subpath of a store path to
exportReferencesGraph:
with import <nixpkgs> {};
let
hello = writeShellScriptBin "hello" ''
echo ${toString builtins.currentTime}
'';
in
writeClosure [ "${hello}/bin/hello" ]
This regressed with Nix 2.4, with a very confusing error message, that
presumably indicates it was unintentional:
error: path '/nix/store/3gl7kgjr4pwf03f0x70dgx9ln3bhl7zc-hello/bin/hello' is not in the Nix store
At this point many features are stripped out, but this works:
- Can run libnix{util,store,expr} unit tests
- Can run some Nix commands
Co-Authored-By volth <volth@volth.com>
Co-Authored-By Brian McKenna <brian@brianmckenna.org>
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_expr.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_expr.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_external.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_external.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_value.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_value.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_store.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_store.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_util.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_util.h'
See https://github.com/NixOS/nix/pull/8699#discussion_r1554312181
Casting a function pointer to `void*` is undefined behavior in the C
spec, since there are platforms with different sizes for these two kinds
of pointers. A safe alternative might be `void (*callback)()`
https://github.com/NixOS/nix/pull/10456 fixed the addition of symlink
store paths to the sandbox, but also made it so that the hardcoded
sandbox paths (like `/etc/hosts`) were now bind-mounted without
following the possible symlinks. This made these files unreadable if
there were symlinks (because the sandbox would now contain a symlink to
an unreachable file rather than the underlying file).
In particular, this broke FOD derivations on NixOS as `/etc/hosts` is a
symlink there.
Fix that by canonicalizing all these hardcoded sandbox paths before
adding them to the sandbox.
Like always declining; local builds only, as can be inferred from the
docs. (Not worth spending too many words on this pretty obvious
behavior, I think. Also, plans to remove it? https://github.com/NixOS/nix/issues/1221)
This requires moving resolveSymlinks() into SourceAccessor. Also, it
requires LocalStoreAccessor::maybeLstat() to work on parents of the
store (to avoid an error like "/nix is not in the store").
Fixes#10375.
Instead of relying on setup script to set output variables when
structured attributes are enabled, iterate over the values of an
outputs associative array.
See also
374fa3532e/pkgs/stdenv/generic/setup.sh (L23-L26)
Bind-mounting symlinks is apparently not possible, which is why the
thing was failing.
Fortunately, symlinks are small, so we can fallback to copy them at no cost.
Fix https://github.com/NixOS/nix/issues/9579
Co-authored-by: Artturin <Artturin@artturin.com>
Now that we have a few things identifying content address methods by
name, we should be consistent about it.
Move up the `parseHashAlgoOpt` for tidiness too.
Discussed this change for consistency's sake as part of #8876
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
This requires `--substitute-on-destination` if you want the remote side
to substitute instead of copying if possible.
For completeness sake, document it here.
Also, the stable Nix from nixpkgs is still 2.18, so more folks may
stumble upon this when this is bumped, so I'd expect this to be actually
useful.
Closes#10182
This was used in only one place, namely builtins.fetchurl with an
expected hash. Since this can cause similar issues as described
in #9814 and #9905 with the "locked" flag for fetchTarball and fetchTree,
let's just remove it.
Note that if an expected hash is given and the hash algorithm is
SHA-256, then we will never do a download anyway if the resulting
store path already exists. So removing the "locked" flag will only
cause potentially unnecessary HTTP requests (subject to the tarball
TTL) for non-SHA-256 hashes.
This probably snuck in in a refactor using truthiness or so. The
trustedness flag was having the optional fullness checked, rather than
the actual contained trust level.
Also adds some tests.
```
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix store ping
warning: 'nix store ping' is a deprecated alias for 'nix store info'
Store URL: daemon
Version: 2.20.4
Trusted: 0
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix doctor
warning: 'doctor' is a deprecated alias for 'config check'
[PASS] PATH contains only one nix version.
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: daemon
```
The script at `/nix/store/...-nix-2.21.0/etc/profile.d/nix-daemon.sh` was leaving behind a variable, which was visible in the user's shell environment, but not used outside the script.
code blocks, if not surrounded by empty lines, have the language
tags (in these cases, always `nix`) show up in the output of :doc.
for example:
nix-repl> :doc builtins.parseFlakeRef
Synopsis: builtins.parseFlakeRef flake-ref
Parse a flake reference, and return its exploded form.
For example: nix builtins.parseFlakeRef
"github:NixOS/nixpkgs/23.05?dir=lib" evaluates to: nix { dir =
"lib"; owner = "NixOS"; ref = "23.05"; repo = "nixpkgs"; type =
"github"; }
is now instead:
nix-repl> :doc builtins.parseFlakeRef
Synopsis: builtins.parseFlakeRef flake-ref
Parse a flake reference, and return its exploded form.
For example:
| builtins.parseFlakeRef "github:NixOS/nixpkgs/23.05?dir=lib"
evaluates to:
| { dir = "lib"; owner = "NixOS"; ref = "23.05"; repo = "nixpkgs"; type = "github"; }
When querying all paths in a binary cache store, the path's representation
is `<hash>-x` (where `x` is the value of `MissingName`) because the .narinfo
filenames only contain the hash.
Before cc46ea1630 this worked correctly,
because the entire path info was read and the path from this
representation was printed, i.e. in the form `<hash>-<name>`. Since then
however, the direct result from `queryAllValidPaths()` was used as `path`.
Added a regression test to make sure the behavior remains correct.
This was part of approved PR #10021. Unfortunately that one is stalled
on a peculiar Linux test timeout, so trying to get bits of it merged
first to bisect failure.
Forcing a conditional include, vs making the headers content
conditional, I think is more maintainable.
It is also how the other platform-specific headers (like
`namespaces.hh`) have been adapted.
A possible use of them might have been to figure out the paths
(which can now be retrieved with maybePathsOut), but I have
not found evidence that it was used this way, and it would have
been broken, because non-CA outputs weren't recorded in the map.
It's a little weird we don't check the return status for these, but
changing that would introduce risk so I did not.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
This introduces new utility functions to get elements from JSON — in an ergonomic way and with nice error messages if the expected type does not match.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
* show Nix logo in the manual
the location of files is hard-coded by mdBook.
there is also seems to be no way to define custom templates, therefore
all styling has to be done in the CSS override.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This splits files and adds new identifiers in preperation for supporting
windows, but no Windows-specific code is actually added yet.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This patch makes `makeDecompressionSink` strip only a single layer
of compression specified via method. This fixes erroneous decompression
of doubly-compressed NARs fetched with curl.
This function is nice for more than `PosixSourceAccessor`. We can make a
few things simpler with it.
Note that the error logic slightly changes in some of the call sites, in
that we also count `ENOTDIR` and not just `ENOENT` as not having the
file, but that should be fine.
HintFmt(string) invokes the HintFmt("%s", literal) constructor,
which is not what we want here. Add a constructor with a proper name
and call that.
Next step: rename all the other ones to HintFmt::literal(string).
Fixes https://github.com/NixOS/nix/issues/10238
Thunks are now overwritten by a helper function
`Value::finishValue(newType, payload)` (where `payload` is the
original anonymous union inside `Value`). This helps to ensure we
never update a value elsewhere, since that would be incompatible with
parallel evaluation (i.e. after a value has transitioned from being a
thunk to being a non-thunk, it should be immutable).
There were two places where this happened: `Value::mkString()` and
`ExprAttrs::eval()`.
This PR also adds a bunch of accessor functions for value contents,
like `Value::integer()` to access the integer field in the union.
I realized it was checking NAR hashes before of added objects, which
makes little sense --- we don't really care about ancillary NAR hashes.
Now, the bottom `nix store add` tests compare the CA field with a git
hash to hashes calculated by Git. This matches top `nix hash path` ones
in using git as a source of truth.
Fixes an instance of
nix: src/libutil/util.cc:139: nix::Path nix::canonPath(PathView, bool): Assertion `path != ""' failed.
... which I've been getting in one of my shells for some reason.
I have yet to find out why TMPDIR was empty, but it's no reason for
Nix to break.
I was using by mistake the .#nix-clangStdenv shell to retrieve clangd.
This clangd is unusable with the project and constantly segfaults.
Let's explicitly state which shell the user should use in the docs.
I don't really understand the source of this segfault. I assume it's
related to a clang version incompatibility. (16.0.6 for
.#nix-clangStdenv 14.0.6 for .#native-clangStdenvPackages)
Fixes
terminate called after throwing an instance of 'boost::wrapexcept<boost::io::too_few_args>'
what(): boost::too_few_args: format-string referred to more arguments than were passed
Aborted (core dumped)
for type errors in AttrCursor.
- Align the “frequent” release cycle with the calendar
- The 6-month release cycle is hard to keep track of. A monthly
release will make it much easier to remember the release date.
- Officialise the support for a stable version maintained for as long as NixOS stable
- This is already the case in practice, it just happens that the
“stable” Nixpkgs version is whichever version was deemed
stable-enough at the time of the NixOS release.
Officialise that by cutting a new major release alongside each NixOS one.
Note that this breaks whatever semver compatibility Nix might pretend to
have, but I don't think it makes sense any way.
In a daemon-based Nix setup, some options cannot be overridden by a
client unless the client's user is considered trusted.
Currently, if an untrusted user tries to override one of those
options, we are silently ignoring it.
This can be pretty confusing in certain situations.
e.g. a user thinks he disabled the sandbox when in reality he did not.
We are now sending a warning message letting know the user some options
have been ignored.
Related to #1761.
This is a cherry-pick of 9e0f5f803f.
The above commit has been reverted by
a59e77d9e5 to prevent spamming warnings
with experimental features, but these are now totally ignored on the
daemon side, so there's no reason for the revert any more.
Previously, `state.mkList()` would set the type of the value to tList
and allocate the list vector, but it would not initialize the values
in the list. This has two problems:
* If an exception occurs, the list is left in an undefined state.
* More importantly, for multithreaded evaluation, if a value
transitions from thunk to non-thunk, it should be final (i.e. other
threads should be able to access the value safely).
To address this, there now is a `ListBuilder` class (analogous to
`BindingsBuilder`) to build the list vector prior to the call to
`Value::mkList()`. Typical usage:
auto list = state.buildList(size);
for (auto & v : list)
v = ... set value ...;
vRes.mkList(list);
* Add regression test
* Fix 'no repo' test so it doesn't succeed if the data is still in cache
* Use git_revparse_single inside git-utils instead of reimplementing the same logic.
Add `runHook preInstallCheck` to the overriden `installCheckPhase` used
for the non-build case.
In particular, this allow the fix from 2a34510776
to also apply there.
Previously, errors while printing values in `nix repl` would be printed
in `«error: ...»` brackets rather than displayed normally:
```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
«error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.»
```
Now, errors will be displayed normally if they're emitted at the
top-level of an expression:
```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
error:
… in the condition of the assert statement
at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/lib/customisation.nix:268:17:
267| in commonAttrs // {
268| drvPath = assert condition; drv.drvPath;
| ^
269| outPath = assert condition; drv.outPath;
… in the left operand of the OR (||) operator
at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/passthrufun.nix:28:45:
27| if lib.isDerivation value then
28| lib.extendDerivation (valid value || throw "${name} should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.") {} value
| ^
29| else
(stack trace truncated; use '--show-trace' to show the full trace)
error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.
```
Errors emitted in nested structures (like e.g. when printing `nixpkgs`)
will still be printed in brackets.
Strings are now printed directly when evaluated by `:print`, rather than
escaped. This makes it easier to debug multi-line strings or strings
containing quotes, like the results of `builtins.readFile`,
`lib.toShellArg`, and so on.
```
nix-repl> "cuppy\ndog\ncity"
"cuppy\ndog\ncity"
nix-repl> :p "cuppy\ndog\ncity"
cuppy
dog
city
```
Fixes this very long warning, which I'll only include the first line of:
/nix/store/8wrjhrycpshhc3b41xmjwvgqr2m3yajq-libcxx-16.0.6-dev/include/c++/v1/__memory/construct_at.h:66:5: warning: destructor called on non-final 'RegexMatcher' that has virtual functions but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor]
__loc->~_Tp();
`nix eval` forces values and prints derivations as attribute sets, so
commands that print derivations (e.g. `nix eval nixpkgs#bash`) will
infinitely loop and segfault.
Printing derivations as `.drv` paths makes `nix eval` complete as
expected. Further work is needed, but this is better than a segfault.
`nix-env -qaP`'s output has changed a bit because of https://github.com/NixOS/nix/issues/10132.
Although that's a bit annoying, it isn't nearly as problematic as the
evaluation changes that this test is supposed to catch. So it's find to
just update the hash for the time being and fix the issue later
(properly fixing the issue will very likely change the hash any way).
Currently there isn't a convenient way to check for multiline output. In
addition, these outputs will easily change and having a diff between the
expected an the actual output upon failures is convenient.
This wasn't caught by CI because #10149 and #10152 pass
individually... It doesn't happen on lazy-trees either because we
never try to fetch relative path flakes (#10089).
we now keep not a table of all positions, but a table of all origins and
their sizes. position indices are now direct pointers into the virtual
concatenation of all parsed contents. this slightly reduces memory usage
and time spent in the parser, at the cost of not being able to report
positions if the total input size exceeds 4GiB. this limit is not unique
to nix though, rustc and clang also limit their input to 4GiB (although
at least clang refuses to process inputs that are larger, we will not).
this new 4GiB limit probably will not cause any problems for quite a
while, all of nixpkgs together is less than 100MiB in size and already
needs over 700MiB of memory and multiple seconds just to parse. 4GiB
worth of input will easily take multiple minutes and over 30GiB of
memory without even evaluating anything. if problems *do* arise we can
probably recover the old table-based system by adding some tracking to
Pos::Origin (or increasing the size of PosIdx outright), but for time
being this looks like more complexity than it's worth.
since we now need to read the entire input again to determine the
line/column of a position we'll make unsafeGetAttrPos slightly lazy:
mostly the set it returns is only used to determine the file of origin
of an attribute, not its exact location. the thunks do not add
measurable runtime overhead.
notably this change is necessary to allow changing the parser since
apparently nothing supports nix's very idiosyncratic line ending choice
of "anything goes", making it very hard to calculate line/column
positions in the parser (while byte offsets are very easy).
this needs a string comparison because there seems to be no other way to
get that information out of bison. usually the location info is going to
be correct (pointing at a bad token), but since EOF isn't a token as
such it'll be wrong in that this case.
this hasn't shown up much so far because a single line ending *is* a
token, so any file formatted in the usual manner (ie, ending in a line
ending) would have its EOF position reported correctly.
the parser treats a plain \r as a newline, error reports do not. this
can lead to interesting divergences if anything makes use of this
feature, with error reports pointing to wrong locations in the input (or
even outside the input altogether).
previously we reported the error at the beginning of the binding
block (for plain inherits) or the beginning of the attr list (for
inherit-from), effectively hiding where exactly the error happened.
this also carries over to runtime positions of attributes in sets as
reported by unsafeGetAttrPos. we're not worried about this changing
observable eval behavior because it *is* marked unsafe, and the new
behavior is much more useful.
we already normalize attr order to lexicographic, doing the same for
formals makes sense. doubly so because the order of formals would
otherwise depend on the context of the expression, which is not quite as
useful as one might expect.
the parser modifies its inputs, which means that sharing them between
the error context reporting system and the parser itself can confuse the
reporting system. usually this led to early truncation of error context
reports which, while not dangerous, can be quite confusing.
* Convert all InputScheme::fetch() methods to getAccessor().
* Add checkLocks() method for checking lock attributes.
* Rename fetch() to fetchToStore().
This is the case for e.g. dirty Git workdirs, where we would get
$ nix flake metadata
Resolved URL: git+file:///home/eelco/Dev/nix-master
Locked URL: git+file:///home/eelco/Dev/nix-master
The "lockedRef" field is a misnomer, since it can be unlocked
(e.g. for a dirty Git workdir). In that case, `nix profile upgrade`
needs to assume that the package can have changed, and perform an
upgrade.
When reviewing old PRs, I found that #9997 adds some code to ensure one
particular assert is always present. But, removing asserts isn't
something we do in our own release builds either in the flake here or in
nixpkgs, and is plainly a bad idea that increases support burden,
especially if other distros make bad choices of build flags in their Nix
packaging.
For context, the assert macro in the C standard is defined to do nothing
if NDEBUG is set.
There is no way in our build system to set -DNDEBUG without manually
adding it to CFLAGS, so this is simply a configuration we do not use.
Let's ban it at compile time.
I put this preprocessor directive in src/libutil.cc because it is not
obvious where else to put it, and it seems like the most logical file
since you are not getting a usable nix without it.
Directly fail if a flakeref points to something that isn't a directory
instead of falling back to the logic of trying to look up the hierarchy
to find a valid flake root.
Fix https://github.com/NixOS/nix/issues/9868
After https://github.com/NixOS/nix/pull/10071, the CI was trying to push
ghcr.io/nixos/nix:master for backwards-compatibility, but the image was
not tagged as such, causing the job to fail.
Fix this.
It is possible to exfiltrate a file descriptor out of the build sandbox
of FODs, and use it to modify the store path after it has been
registered.
To avoid that issue, don't register the output of the build, but a copy
of it (that will be free of any leaked file descriptor).
`NIX_HARDENING_ENABLE` causes `_FORTIFY_SOURCE` to be defined.
This isn't compatible with `-O0`, and the compiler will happily remind
us about it at every call, spamming the terminal with warnings and stack
traces.
We don't really care hardening in that case, so just disable it if we
pass `OPTIMIZE=0`.
Just `stdenv.isDarwin` isn't enough because it doesn't apply to the
build platform, which mean that cross packages building from darwin to
another platform will have `isDarwin` set to false.
Replace it by `stdenv.buildPlatform.isDarwin`.
Instead, serialize as NAR and send that over, then rehash sever side.
This is alorithmically simpler, but comes at the cost of a newer
parameter to `Store::addToStoreFromDump`.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
desugaring inherit-from to syntactic duplication of the source expr also
duplicates side effects of the source expr (such as trace calls) and
expensive computations (such as derivationStrict).
Flakes still reside in the Nix store (so there shouldn't be any change
in behaviour), but they are now accessed via the rootFS
accessor. Since rootFS implements access checks, we no longer have to
worry about flake.{nix,lock} or their parents being symlinks that
escape from the flake.
Extracted from the lazy-trees branch.
"hash type" -> "hash algorithm" in all comments, documentation, and
messages.
ht -> ha, [Hh]ashType -> [HhashAlgo] for all local variables and
function arguments. No API change is made.
Continuation of 5334c9c792 and 837b889c41.
After commit 91b6833686 (" Move tests to separate directories, and
document"), previously-built test executables are now tracked by Git,
which is annoying for developers.
This patch add .gitignore rules to ignore the obsolete test directories
to solve such problem and enhance developer experience.
The sandbox rule `(allow network* (local ip))` doesn't do what it
implies. Adding this rule permits all network traffic. We should be
matching on (remote ip "localhost:*")` instead.
This seems to have found one actual bug in fs-sink.cc: the symlink case
was falling into the regular file case, which can't possibly be
intentional, right?
Docker uses "latest" as the default label instead of "master".
This change will allow to docker run ghcr.io/nixos/nix without having to
specify the label.
It keeps the :master label on docker hub for back-compat.
When a file conflict arises during a package install a suggestion is
made to remove the old entry. This was previously done using the
installable URLs of the old entry. These URLs are quite verbose and
often do not equal the URL of the existing entry.
This change uses the recently introduced profile entry name for the
suggestion, resulting in a simpler output.
The improvement is easily seen in the change to the functional test.
- `nix store add` supports text hashing
With functional test ensuring it matches `builtins.toFile`.
- Factored-out flags for both commands
- Move all common reusable flags to `libcmd`
- They are not part of the *definition* of the CLI infra, just a usag
of it.
- The `libstore` flag couldn't go in `args.hh` in libutil anyways,
would be awkward for it to live alone
- Shuffle around `Cmd*` hierarchy so flags for deprecated commands don't
end up on the new ones
This PR reduces the creation of short-lived basic_json objects while
parsing flake.lock files. For large flake.lock files (~1.5MB) I was
observing ~60s being spent for trivial nix build operations while
after this change it is now taking ~1.6s.
It's better to just check whether the input has all the attributes
needed to consider itself locked (e.g. whether a Git input has an
'rev' attribute).
Also, the 'locked' field was actually incorrect for Git inputs: it
would be set to true even for dirty worktrees. As a result, we got
away with using fetchTree() internally even though fetchTree()
requires a locked input in pure mode. In particular, this allowed
'--override-input' to work by accident.
The fix is to pass a set of "overrides" to call-flake.nix for all the
unlocked inputs (i.e. the top-level flake and any --override-inputs).
This fixes warnings like
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
when using the daemon and the user didn't actually set those settings.
Note: this also hides those settings from `nix config show`, but that
seems a good thing.
`canonPath` and `absPath` work on native paths, and so should switch
between supporting Unix paths and Windows paths accordingly.
The templating is because `CanonPath`, which shares the implementation,
should always be Unix style. It is the pure "nix-native" path type for
virtual file operations --- it is part of Nix's "business logic", and
should not vary with the host OS accordingly.
The core `CanonPath` constructors were using `absPath`, but `absPath` in
some situations does IO which is not appropriate. It turns out that
these constructors avoided those situations, and thus were pure, but it
was far from obvious this was the case.
To remedy the situation, abstract the core algorithm from `canonPath` to
use separately in `CanonPath` without any IO. No we know by-construction
that those constructors are pure.
That leaves `CanonPath::fromCWD` as the only operation which uses IO /
is impure. Add docs on it, and `CanonPath` as a whole, explaining the
situation.
This is also necessary to support Windows paths on windows without
messing up `CanonPath`. But, I think it is good even without that.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
There is no longer an `importTarball` method. Instead, there is a
`unpackTarfileToSink` function (back in libutil). The caller can use
thisw with the `getParseSink` method we added in the last commit easily
enough.
In addition, tarball cache functionality is separated from `git-utils`
and moved into `tarball-cache`. This ensures we are separating mechanism
and policy.
There is now a separation of:
1. A `FileSystemObjectSink` for writing to git repos
2. Adapting libarchive to use that parse sink.
The prepares a proper separation of concerns.
A command like
rm -rf ~/.cache/nix/tarball-cache/ ~/.cache/nix/fetcher-cache-v1.sqlite*; nix flake metadata 'git+file:///home/eelco/Dev/nixpkgs?rev=9463103069725474698139ab10f17a9d125da859'
was spending about 84% of its runtime in lookup(), specifically in
git_tree_entry_bypath(). (The reading of blobs is less than 3%.)
It appears libgit2 doesn't do a lot of caching of trees, so we now
make sure that when we look up a path, we add all its parents, and all
the immediate children of the parents (since we have them in memory
anyway), to our own cache.
This speed up the command above from 17.2s to 7.8s on my machine.
Fixes (or at least should improve a lot) #9684.
Commit 83c067c0fa changed `builtins.pathExists`
to resolve symlinks before checking for existence. Consequently, if the path
refers to a symlink itself, existence of the target of the symlink (instead of
the symlink itself) was checked. Restore the previous behavior by skipping
symlink resolution in the last component.
No outward facing behavior is changed.
Older methods with same names that operate on on method + algo pair (for
old-style `<method>:algo`) are renamed to `*WithAlgo`.)
The functions are unit-tested in the same way the names for the hash
algorithms are tested.
* reword description of the `cores` setting
- be precise about the `builder` executable
- clearly distinguish between `builder` and job parallelism
- clarify the role of `mkDerivation` in the example
- remove prose for the default, it's shown programmatically
- mention relation to `max-jobs`
- move all reference documentation to the `builders` configuration setting
- reword documentation on machine specification, add examples
- disable showing the default value, as it rendered as `@/dummy/machines`, which is wrong
- highlight the examples
- link to the configuration docs for distributed builds
- builder -> build machine
Co-authored-by: Janik H <janik@aq0.de>
the interesting information is on the proper pages, and is now presented
a bit more prominently.
the paragraph was a bit confusing to read, also because an anchor link
to an inline definition was in the middle of the sentence. "local store"
now has its own glossary entry.
The symbolic form in use here doesn't seem to have an effect
in either the BSD or coreutils install commands, leaving the
daemon plist with empty permissions. This seems to cause its
own problems.
I think I've got the right symbolic syntax now :)
for plain inherits this is really just a stylistic choice, but for
inherit-from it actually fixes an exponential size increase problem
during expr printing (as may happen during assertion failure reporting,
on during duplicate attr detection in the parser)
this also has the effect of sorting let bindings lexicographically
rather than by symbol creation order as was previously done, giving a
better canonicalization in the process.
There's probably more that can be said, but I thought it might be helpful to put something here about how to access elements of a list for folks coming from more or less any other programming language. If this is rarely used, it might be nice to add to the documentation something about why it's rarely used.
Commit d536c57e87 inadvertedly broke build and
installation of all non-autogenerated manual pages (in particular, all the ones
documenting the stable CLI), by moving the definition of the man-pages variable
in doc/manual/local.mk after its usage in mk/lib.mk. Move including the former
earlier so that the correct order is restored.
how the different invocations relate to each other seems be
confusing, which is relatable because one has to wire it up in your head
while reading. an explicit reference should make it unambiguous and
easier to notice due to links being highlighted.
autoconf authors apparently decided that setting `-O2` by default was a good idea. I disagree, and Nix has its own way of deciding that (with `OPTIMIZE={0,1}`). Explicitly set `CFLAGS` and `CXXFLAGS` in the configure script to disable that behaviour.
Fix#9965
When I started contributing to Nix, I found the mix of definitions and
names in `fmt.hh` to be rather confusing, especially the small
difference between `hintfmt` and `hintformat`. I've renamed many classes
and added documentation to most definitions.
- `formatHelper` is no longer exported.
- `fmt`'s documentation is now with `fmt` rather than (misleadingly)
above `formatHelper`.
- `yellowtxt` is renamed to `Magenta`.
`yellowtxt` wraps its value with `ANSI_WARNING`, but `ANSI_WARNING`
has been equal to `ANSI_MAGENTA` for a long time. Now the name is
updated.
- `normaltxt` is renamed to `Uncolored`.
- `hintfmt` has been merged into `hintformat` as extra constructor
functions.
- `hintformat` has been renamed to `hintfmt`.
- The single-argument `hintformat(std::string)` constructor has been
renamed to a static member `hintformat::interpolate` to avoid pitfalls
with using user-generated strings as format strings.
As discussed in the last Nix team meeting (2024-02-95), this method
doesn't belong because `CanonPath` is a virtual/ideal absolute path
format, not used in file systems beyond the native OS format for which a
"current working directory" is defined.
Progress towards #9205
Pretty-print values in the REPL by printing each item in a list or
attrset on a separate line. When possible, single-item lists and
attrsets are printed on one line, as long as they don't contain a nested
list, attrset, or thunk.
Before:
```
{ attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
```
After:
```
{
attrs = {
a = {
b = {
c = { };
};
};
};
list = [ 1 ];
list' = [
1
2
3
];
}
```
Some tools which consume the "nix print-dev-env" rc script (such as
"nix-direnv") are sensitive to the use of unbound variables. They use
"set -u".
The "nix print-dev-env" rc script initially unsets "shellHook", then
loads variables from the derivation, and then evaluates "shellHook".
However, most derivations don't have a "shellHook" attribute.
So users get the error "shellHook: unbound variable". This can be
demonstrated with the command:
nix print-dev-env nixpkgs#hello | bash -u
This commit changes the rc script to provide an empty fallback value
for the "shellHook" variable.
Closes: #7951#8253
It is entirely possible for the path to be an empty string and many
unit tests actually pass it as an empty string (e.g. both_roundrip or
turnsEmptyPathIntoCWD). In this case, without this patch, absPath will
perform a one-byte out-of-bounds access.
This was discovered while enabling the nix test suite on Alpine where
we compile all software with `-D_GLIBCXX_ASSERTIONS=1`, thus resulting
in a test failure on Alpine.
While preparing PRs like #9753, I've had to change error messages in
dozens of code paths. It would be nice if instead of
EvalError("expected 'boolean' but found '%1%'", showType(v))
we could write
TypeError(v, "boolean")
or similar. Then, changing the error message could be a mechanical
refactor with the compiler pointing out places the constructor needs to
be changed, rather than the error-prone process of grepping through the
codebase. Structured errors would also help prevent the "same" error
from having multiple slightly different messages, and could be a first
step towards error codes / an error index.
This PR reworks the exception infrastructure in `libexpr` to
support exception types with different constructor signatures than
`BaseError`. Actually refactoring the exceptions to use structured data
will come in a future PR (this one is big enough already, as it has to
touch every exception in `libexpr`).
The core design is in `eval-error.hh`. Generally, errors like this:
state.error("'%s' is not a string", getAttrPathStr())
.debugThrow<TypeError>()
are transformed like this:
state.error<TypeError>("'%s' is not a string", getAttrPathStr())
.debugThrow()
The type annotation has moved from `ErrorBuilder::debugThrow` to
`EvalState::error`.
As discussed in the maintainer meeting on 2024-01-29.
Mainly this is to avoid a situation where the name is parsed and
treated as a file name, mostly to protect users.
.-* and ..-* are also considered invalid because they might strip
on that separator to remove versions. Doesn't really work, but that's
what we decided, and I won't argue with it, because .-* probably
doesn't seem to have a real world application anyway.
We do still permit a 1-character name that's just "-", which still
poses a similar risk in such a situation. We can't start disallowing
trailing -, because a non-zero number of users will need it and we've
seen how annoying and painful such a change is.
What matters most is preventing a situation where . or .. can be
injected, and to just get this done.
GitArchiveInputScheme now streams tarballs into a Git repository. This
deduplicates data a lot, e.g. when you're fetching different revisions
of the Nixpkgs repo. It also warns if the tree hash returned by GitHub
doesn't match the tree hash of the imported tarball.
These docs explain the implementation relative to the local store
originals. The original declaration of virtual methods can still be
consulted for proper interface-level documentation.
As an optimisation for LocalStore, we read all the store directory entries into
a set. Checking for membership of this set is much faster than a stat syscall.
However for LocalOverlayStore, the lower store directory is expected to contain
a vast number of entries and reading them all can take a very long time.
So instead of enumerating them all upfront, we call pathExists as needed. This
means making stat syscalls for each store path, but the upper layer is expected
to be relatively small compared to the lower store so that should be okay.
Grouping our tests should make it easier to understand the intent than
one long poorly-arranged list. It also is convenient for running just
the tests for a specific component when working on that component.
We need at least one test group so this isn't dead code; I decided to
collect the tests for the `ca-derivations` and `dynamic-derivations`
experimental features in groups. Do
```bash
make ca.test-group -jN
```
and
```bash
make dyn-drv.test-group -jN
```
to try running just them.
I originally did this as part of #8397 for being able to just the local
overlay store alone. I am PRing it separately now so we can separate
general infra from new features.
This requires switching on SQLITE_OPEN_URI because there is no open flag to
make the database immutable. Without immutable, sqlite will still attempt to
create journal and wal files, even when the database is opened read-only.
https://www.sqlite.org/c3ref/open.html
The immutable parameter is a boolean query parameter that indicates that the
database file is stored on read-only media. When immutable is set, SQLite
assumes that the database file cannot be changed, even by a process with higher
privilege, and so the database is opened read-only and all locking and change
detection is disabled.
Nix does not manage the overlayfs mount point itself, but the correct
functioning of the overlay store does depend on this mount point being set up
correctly. Rather than just assume this is the case, check that the lowerdir
and upperdir options are what we expect them to be. This check is on by
default, but can be disabled if needed.
- [ ] User documentation in the [manual](..doc/manual/src)
- [ ] User documentation in the [manual](./doc/manual/src)
- [ ] API documentation in header files
- [ ] Code and comments are self-explanatory
- [ ] Commit message explains **why** the change was made
- [ ] New feature or incompatible change: updated [release notes](./doc/manual/src/release-notes/rl-next.md)
- [ ] New feature or incompatible change: [add a release note](https://nixos.org/manual/nix/stable/contributing/hacking#add-a-release-note)
7. If you need additional feedback or help to getting pull request into shape, ask other contributors using [@mentions](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#mentioning-people-and-teams).
[test "$ENABLE_BUILD" == "no" && test "$ENABLE_UNIT_TESTS" == "yes"],
[AC_MSG_ERROR([Cannot enable unit tests when building overall is disabled. Please do not pass '--enable-unit-tests' or do not pass '--disable-build'.])])
then you need to ensure that the `PATH` of non-interactive login shells
contains Nix.
> **Warning**
>
> If you are building via the Nix daemon, it is the Nix daemon user account (that is, `root`) that should have SSH access to a user (not necessarily `root`) on the remote machine.
>
> If you can’t or don’t want to configure `root` to be able to access the remote machine, you can use a private Nix store instead by passing e.g. `--store ~/my-nix` when running a Nix command from the local machine.
The list of remote machines can be specified on the command line or in
the Nix configuration file. The former is convenient for testing. For
example, the following command allows you to build a derivation for
`x86_64-darwin` on a Linux machine:
The [list of remote build machines](@docroot@/command-ref/conf-file.md#conf-builders) can be specified on the command line or in the Nix configuration file.
For example, the following command allows you to build a derivation for `x86_64-darwin` on a Linux machine:
```console
$ uname
@@ -60,97 +52,20 @@ $ cat ./result
Darwin
```
It is possible to specify multiple builders separated by a semicolon or
a newline, e.g.
It is possible to specify multiple build machines separated by a semicolon or a newline, e.g.
This is the equivalent of invoking `nix-env --delete-generations old` on each found profile.
This is the equivalent of invoking [`nix-env --delete-generations old`](@docroot@/command-ref/nix-env/delete-generations.md#generations-old) on each found profile.
[binary format emulation]: https://nixos.org/manual/nixos/stable/options.html#opt-boot.binfmt.emulatedSystems
Given such a setup, executing the build only requires selecting the respective attribute.
@@ -166,7 +167,10 @@ or for Nix with the [`flakes`] and [`nix-command`] experimental features enabled
$ nix build .#packages.aarch64-linux.default
```
Cross-compiled builds are available for ARMv6 (`armv6l-linux`) and ARMv7 (`armv7l-linux`).
Cross-compiled builds are available for:
-`armv6l-linux`
-`armv7l-linux`
-`riscv64-linux`
Add more [system types](#system-type) to `crossSystems` in `flake.nix` to bootstrap Nix on unsupported platforms.
### Building for multiple platforms at once
@@ -196,7 +200,7 @@ In order to facilitate this, Nix has some support for being built out of tree
## System type
Nix uses a string with he following format to identify the *system type* or *platform* it runs on:
Nix uses a string with the following format to identify the *system type* or *platform* it runs on:
```
<cpu>-<os>[-<abi>]
@@ -258,10 +262,10 @@ See [supported compilation environments](#compilation-environments) and instruct
To use the LSP with your editor, you first need to [set up `clangd`](https://clangd.llvm.org/installation#project-setup) by running:
```console
make clean && bear -- make -j$NIX_BUILD_CORES default check install
make compile_commands.json
```
Configure your editor to use the `clangd` from the shell, either by running it inside the development shell, or by using [nix-direnv](https://github.com/nix-community/nix-direnv) and [the appropriate editor plugin](https://github.com/direnv/direnv/wiki#editor-integration).
Configure your editor to use the `clangd` from the `.#native-clangStdenvPackages` shell. You can do that either by running it inside the development shell, or by using [nix-direnv](https://github.com/nix-community/nix-direnv) and [the appropriate editor plugin](https://github.com/direnv/direnv/wiki#editor-integration).
> **Note**
>
@@ -269,6 +273,29 @@ Configure your editor to use the `clangd` from the shell, either by running it i
> Some other editors (e.g. Emacs, Vim) need a plugin to support LSP servers in general (e.g. [lsp-mode](https://github.com/emacs-lsp/lsp-mode) for Emacs and [vim-lsp](https://github.com/prabirshrestha/vim-lsp) for vim).
> Editor-specific setup is typically opinionated, so we will not cover it here in more detail.
## Formatting and pre-commit hooks
You may run the formatters as a one-off using:
```console
make format
```
If you'd like to run the formatters before every commit, install the hooks:
```
pre-commit-hooks-install
```
This installs [pre-commit](https://pre-commit.com) using [cachix/git-hooks.nix](https://github.com/cachix/git-hooks.nix).
When making a commit, pay attention to the console output.
If it fails, run `git add --patch` to approve the suggestions _and commit again_.
To refresh pre-commit hook's config file, do the following:
1. Exit the development shell and start it again by running `nix develop`.
2. If you also use the pre-commit hook, also run `pre-commit-hooks-install` again.
## Add a release note
`doc/manual/rl-next` contains release notes entries for all unreleased changes.
- Fetching a pre-built [store object] from a [substituter]
- Running the [`builder`](@docroot@/language/derivations.md#attr-builder) executable as specified in the corresponding [derivation]
- Delegating to a [remote builder](@docroot@/advanced-topics/distributed-builds.html) and retrieving the outputs
- Delegating to a [remote machine](@docroot@/command-ref/conf-file.md#conf-builders) and retrieving the outputs
<!-- TODO: link [running] to build process page, #8888 -->
See [`nix-store --realise`](@docroot@/command-ref/nix-store/realise.md) for a detailed description of the algorithm.
@@ -59,23 +59,12 @@
- [store]{#gloss-store}
A collection of store objects, with operations to manipulate that collection.
See [Nix store](./store/index.md) for details.
A collection of [store objects][store object], with operations to manipulate that collection.
See [Nix Store](./store/index.md) for details.
There are many types of stores.
See [`nix help-stores`](@docroot@/command-ref/new-cli/nix3-help-stores.md) for a complete list.
From the perspective of the location where Nix is invoked, the Nix store can be referred to _local_ or _remote_.
Only a [local store]{#gloss-local-store} exposes a location in the file system of the machine where Nix is invoked that allows access to store objects, typically `/nix/store`.
Local stores can be used for building [derivations](#gloss-derivation).
See [Local Store](@docroot@/command-ref/new-cli/nix3-help-stores.md#local-store) for details.
There are many types of stores, see [Store Types](./store/types/index.md) for details.
[store]: #gloss-store
[local store]: #gloss-local-store
- [chroot store]{#gloss-chroot-store}
A [local store] whose canonical path is anything other than `/nix/store`.
- [binary cache]{#gloss-binary-cache}
@@ -87,7 +76,7 @@
- [store path]{#gloss-store-path}
The location of a [store object](@docroot@/store/index.md#store-object) in the file system, i.e., an immediate child of the Nix store directory.
The location of a [store object] in the file system, i.e., an immediate child of the Nix store directory.
> **Example**
>
@@ -97,7 +86,7 @@
[store path]: #gloss-store-path
- [file system object]{#gloss-store-object}
- [file system object]{#gloss-file-system-object}
The Nix data model for representing simplified file system data.
@@ -226,6 +215,9 @@
[output path]: #gloss-output-path
- [output closure]{#gloss-output-closure}\
The [closure] of an [output path]. It only contains what is [reachable] from the output.
- [deriver]{#gloss-deriver}
The [store derivation] that produced an [output path].
@@ -243,6 +235,7 @@
- All paths in the store path's [closure] are valid.
The location from which relative paths are resolved.
- For expressions in a file, the base directory is the directory containing that file.
This is analogous to the directory of a [base URL](https://datatracker.ietf.org/doc/html/rfc1808#section-3.3).
<!-- which is sufficient for resolving non-empty URLs -->
<!--
The wording here may look awkward, but it's for these reasons:
* "with --expr": it's a flag, and not an option with an accompanying value
* "written in": the expression itself must be written as an argument,
whereas the more natural "passed as an argument" allows an interpretation
where the expression could be passed by file name.
-->
- For expressions written in command line arguments with [`--expr`](@docroot@/command-ref/opt-common.html#opt-expr), the base directory is the current working directory.
> Writing to the [local store](@docroot@/store/types/local-store.md) with a newer version of Nix, for example by building derivations with [`nix-build`](@docroot@/command-ref/nix-build.md) or [`nix-store --realise`](@docroot@/command-ref/nix-store/realise.md), may change the database schema!
> Reverting to an older version of Nix may therefore require purging the store database before it can be used.
@@ -188,9 +188,13 @@ Derivations can declare some infrequently used optional attributes.
}
```
The `outputHashAlgo` attribute specifies the hash algorithm used to
compute the hash. It can currently be `"sha1"`, `"sha256"` or
`"sha512"`.
The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The "nix32" encoding is an adaptation of base-32 encoding.
The [`convertHash`](@docroot@/language/builtins.md#builtins-convertHash) function shows how to convert between different encodings, and the [`nix-hash` command](../command-ref/nix-hash.md) has information about obtaining the hash for some contents, as well as converting to and from encodings.
The `outputHashAlgo` attribute specifies the hash algorithm used to compute the hash.
It can currently be `"sha1"`, `"sha256"`, `"sha512"`, or `null`.
`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format.
The `outputHashMode` attribute determines how the hash is computed.
It must be one of the following two values:
@@ -203,16 +207,16 @@ Derivations can declare some infrequently used optional attributes.
This is the default.
- `"recursive"`\
The hash is computed over the NAR archive dump of the output
- `"recursive"` or `"nar"`\
The hash is computed over the [NAR archive](@docroot@/glossary.md#gloss-nar) dump of the output
(i.e., the result of [`nix-store --dump`](@docroot@/command-ref/nix-store/dump.md)). In
this case, the output can be anything, including a directory
tree.
The `outputHash` attribute, finally, must be a string containing
the hash in either hexadecimal or base-32 notation. (See the
[`nix-hash` command](../command-ref/nix-hash.md) for information
about converting to and from base-32 notation.)
`"recursive"` is the traditional way of indicating this,
and is supported since 2005 (virtually the entire history of Nix).
`"nar"` is more clear, and consistent with other parts of Nix (such as the CLI),
however support for it is only added in Nix version 2.21.
If this attribute is set to `true` and [distributed building is enabled](../advanced-topics/distributed-builds.md), then, if possible, the derivation will be built locally instead of being forwarded to a remote machine.
If this attribute is set to `true` and [distributed building is enabled](@docroot@/command-ref/conf-file.md#conf-builders), then, if possible, the derivation will be built locally instead of being forwarded to a remote machine.
This is useful for derivations that are cheapest to build locally.
Passing an expression `expr` that evaluates to a [store path](@docroot@/glossary.md#gloss-store-path) to any built-in function which reads from the filesystem constitutes Import From Derivation (IFD):
The Nix language is designed for conveniently creating and composing *derivations*– precise descriptions of how contents of existing files are used to derive new files.
It is:
> **Tip**
>
> These pages are written as a reference.
> If you are learning Nix, nix.dev has a good [introduction to the Nix language](https://nix.dev/tutorials/nix-language).
The language is:
- *domain-specific*
@@ -432,6 +438,32 @@ This is an incomplete overview of language features, by example.
</td>
</tr>
<tr>
<td>
`inherit pkgs src;`
</td>
<td>
Adds the variables to the current scope (attribute set or `let` binding).
Desugars to `pkgs = pkgs; src = src;`
</td>
</tr>
<tr>
<td>
`inherit (pkgs) lib stdenv;`
</td>
<td>
Adds the attributes, from the attribute set in parentheses, to the current scope (attribute set or `let` binding).
Desugars to `lib = pkgs.lib; stdenv = pkgs.stdenv;`
@@ -105,9 +107,9 @@ An expression that is interpolated must evaluate to one of the following:
A string interpolates to itself.
A path in an interpolated expression is first copied into the Nix store, and the resulting string is the [store path] of the newly created [store object](../glossary.md#gloss-store-object).
A path in an interpolated expression is first copied into the Nix store, and the resulting string is the [store path] of the newly created [store object](@docroot@/glossary.md#gloss-store-object).
is not a path: it's parsed as an expression that selects the
attribute `sh` from the variable `builder`. If the file name is
relative, i.e., if it does not begin with a slash, it is made
absolute at parse time relative to the directory of the Nix
expression that contained it. For instance, if a Nix expression in
absolute at parse time relative to the [base directory](@docroot@/glossary.md#gloss-base-directory).
For instance, if a Nix expression in
`/foo/bar/bla.nix` refers to `../xyzzy/fnord.nix`, the absolute path
is `/foo/xyzzy/fnord.nix`.
@@ -107,13 +107,13 @@
e.g. `~/foo` would be equivalent to `/home/edolstra/foo` for a user
whose home directory is `/home/edolstra`.
For instance, evaluating `"${./foo.txt}"` will cause `foo.txt` in the current directory to be copied into the Nix store and result in the string `"/nix/store/<hash>-foo.txt"`.
For instance, evaluating `"${./foo.txt}"` will cause `foo.txt` in the base directory to be copied into the Nix store and result in the string `"/nix/store/<hash>-foo.txt"`.
Note that the Nix language assumes that all input files will remain _unchanged_ while evaluating a Nix expression.
For example, assume you used a file path in an interpolated string during a `nix repl` session.
Later in the same session, after having changed the file contents, evaluating the interpolated string with the file path again might not return a new [store path], since Nix might not re-read the file contents.
This is the complete specification of the Nix Archive format.
The Nix Archive format closely follows the abstract specification of a [file system object] tree,
because it is designed to serialize exactly that data structure.
[file system object]: @docroot@/store/file-system-object.md
The format of this specification is close to [Extended Backus–Naur form](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form), with the exception of the `str(..)` function / parameterized rule, which length-prefixes and pads strings.
This makes the resulting binary format easier to parse.
Regular users do *not* need to know this information.
But for those interested in exactly how Nix works, e.g. if they are reimplementing it, this information can be useful.
This is the complete specification for how store paths are calculated.
The format of this specification is close to [Extended Backus–Naur form](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form), but must deviate for a few things such as hash functions which we treat as bidirectional for specification purposes.
Regular users do *not* need to know this information --- store paths can be treated as black boxes computed from the properties of the store objects they refer to.
But for those interested in exactly how Nix works, e.g. if they are reimplementing it, this information can be useful.
## Store path proper
```ebnf
store-path =store-dir "/"digest "-"name
```
where
-`name` = the name of the store object.
-`store-dir` = the [store directory](@docroot@/store/store-path.md#store-directory)
-`digest` = base-32 representation of the first 160 bits of a [SHA-256] hash of `fingerprint`
This the hash part of the store name
## Fingerprint
- ```ebnf
fingerprint = type ":" sha256 ":" inner-digest ":" store ":" name
```
Note that it includes the location of the store as well as the name to make sure that changes to either of those are reflected in the hash
(e.g. you won't get `/nix/store/<digest>-name1` and `/nix/store/<digest>-name2`, or `/gnu/store/<digest>-name1`, with equal hash parts).
- `type` = one of:
- ```ebnf
| "text" ( ":" store-path )*
```
for encoded derivations written to the store.
The optional trailing store paths are the references of the store object.
- ```ebnf
| "source" ( ":" store-path )*
```
For paths copied to the store and hashed via a [Nix Archive (NAR)] and [SHA-256][sha-256].
Just like in the text case, we can have the store objects referenced by their paths.
Additionally, we can have an optional `:self` label to denote self reference.
- ```ebnf
| "output:" id
```
For either the outputs built from derivations,
paths copied to the store hashed that area single file hashed directly, or the via a hash algorithm other than [SHA-256][sha-256].
(in that case "source" is used; this is only necessary for compatibility).
`id` is the name of the output (usually, "out").
For content-addressed store objects, `id`, is always "out".
- `inner-digest` = base-16 representation of a SHA-256 hash of `inner-fingerprint`
## Inner fingerprint
- `inner-fingerprint` = one of the following based on `type`:
- if `type` = `"text:" ...`:
the string written to the resulting store path.
- if `type` = `"source:" ...`:
the the hash of the [Nix Archive (NAR)] serialization of the [file system object](@docroot@/store/file-system-object.md) of the store object.
- if `type` = `"output:" id`:
- For input-addressed derivation outputs:
the [ATerm](@docroot@/protocols/derivation-aterm.md) serialization of the derivation modulo fixed output derivations.
- For content-addressed store paths:
```ebnf
"fixed:out:" rec algo ":" hash ":"
```
where
- `rec` = one of:
- ```ebnf
| ""
```
(empty string) for hashes of the flat (single file) serialization
- ```ebnf
| "r:"
```
hashes of the for [Nix Archive (NAR)] (arbitrary file system object) serialization
- ```ebnf
| "git:"
```
hashes of the [Git blob/tree](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects) [Merkel tree](https://en.wikipedia.org/wiki/Merkle_tree) format
- ```ebnf
algo = "md5" | "sha1" | "sha256"
```
- `hash` = base-16 representation of the path or flat hash of the contents of the path (or expected contents of the path for fixed-output derivations).
Note that `id` = `"out"`, regardless of the name part of the store path.
Also note that NAR + SHA-256 must not use this case, and instead must use the `type` = `"source:" ...` case.
The `type` = `"source:" ...` and `type` = `"output:out"` grammars technically overlap in purpose,
in that both can represent data hashed by its SHA-256 NAR serialization.
The original reason for this way of computing names was to prevent name collisions (for security).
For instance, the thinking was that it shouldn't be feasible to come up with a derivation whose output path collides with the path for a copied source.
The former would have an `inner-fingerprint` starting with `output:out:`, while the latter would have an `inner-fingerprint` starting with `source:`.
Since `64519cfd657d024ae6e2bb74cb21ad21b886fd2a` (2008), however, it was decided that separating derivation-produced vs manually-hashed content-addressed data like this was not useful.
Now, data that is content-addressed with SHA-256 + NAR-serialization always uses the `source:...` construction, regardless of how it was produced (manually or by derivation).
This allows freely switching between using [fixed-output derivations](@docroot@/glossary.md#gloss-fixed-output-derivation) for fetching, and fetching out-of-band and then manually adding.
The Nix release cycle is calendar-based as follows:
Nix has a release cycle of roughly 6 weeks.
Notable changes and additions are announced in the release notes for each version.
Bugfixes can be backported on request to previous Nix releases.
We typically backport only as far back as the Nix version used in the latest NixOS release, which is announced in the [NixOS release notes](https://nixos.org/manual/nixos/stable/release-notes.html#ch-release-notes).
Backports never skip releases.
If a feature is backported to version `x.y`, it must also be available in version `x.(y+1)`.
This ensures that upgrading from an older version with backports is still safe and no backported functionality will go missing.
The supported Nix versions are:
- The latest release
- The version used in the stable NixOS release, which is announced in the [NixOS release notes](https://nixos.org/manual/nixos/stable/release-notes.html#ch-release-notes).
Bugfixes and security issues are backported to every supported version.
As the choice of hash formats is no longer binary, the `--base16` flag is also added
to explicitly specify the Base16 format, which is still the default.
* The special handling of an [installable](../command-ref/new-cli/nix.md#installables) with `.drv` suffix being interpreted as all of the given [store derivation](../glossary.md#gloss-store-derivation)'s output paths is removed, and instead taken as the literal store path that it represents.
* The special handling of an [installable](../command-ref/new-cli/nix.md#installables) with `.drv` suffix being interpreted as all of the given [store derivation](@docroot@/glossary.md#gloss-store-derivation)'s output paths is removed, and instead taken as the literal store path that it represents.
The new `^` syntax for store paths introduced in Nix 2.13 allows explicitly referencing output paths of a derivation.
Using this is better and more clear than relying on the now-removed `.drv` special handling.
- Fix a fixed-output derivation sandbox escape (CVE-2024-27297)
Cooperating Nix derivations could send file descriptors to files in the Nix
store to each other via Unix domain sockets in the abstract namespace. This
allowed one derivation to modify the output of the other derivation, after Nix
has registered the path as "valid" and immutable in the Nix database.
In particular, this allowed the output of fixed-output derivations to be
modified from their expected content.
This isn't the case any more.
- CLI options `--arg-from-file` and `--arg-from-stdin` [#10122](https://github.com/NixOS/nix/pull/10122)
The new CLI option `--arg-from-file`*name**path* passes the contents
of file *path* as a string value via the function argument *name* to a
Nix expression. Similarly, the new option `--arg-from-stdin`*name*
reads the contents of the string from standard input.
- Concise error printing in `nix repl` [#9928](https://github.com/NixOS/nix/pull/9928)
Previously, if an element of a list or attribute set threw an error while
evaluating, `nix repl` would print the entire error (including source location
information) inline. This output was clumsy and difficult to parse:
```
nix-repl> { err = builtins.throw "uh oh!"; }
{ err = «error:
… while calling the 'throw' builtin
at «string»:1:9:
1| { err = builtins.throw "uh oh!"; }
| ^
error: uh oh!»; }
```
Now, only the error message is displayed, making the output much more readable.
```
nix-repl> { err = builtins.throw "uh oh!"; }
{ err = «error: uh oh!»; }
```
However, if the whole expression being evaluated throws an error, source
locations and (if applicable) a stack trace are printed, just like you'd expect:
```
nix-repl> builtins.throw "uh oh!"
error:
… while calling the 'throw' builtin
at «string»:1:1:
1| builtins.throw "uh oh!"
| ^
error: uh oh!
```
- `--debugger` can now access bindings from `let` expressions [#8827](https://github.com/NixOS/nix/issues/8827) [#9918](https://github.com/NixOS/nix/pull/9918)
Breakpoints and errors in the bindings of a `let` expression can now access
those bindings in the debugger. Previously, only the body of `let` expressions
could access those bindings.
- Enter the `--debugger` when `builtins.trace` is called if `debugger-on-trace` is set [#9914](https://github.com/NixOS/nix/pull/9914)
If the `debugger-on-trace` option is set and `--debugger` is given,
`builtins.trace` calls will behave similarly to `builtins.break` and will enter
the debug REPL. This is useful for determining where warnings are being emitted
from.
- Debugger prints source position information [#9913](https://github.com/NixOS/nix/pull/9913)
The `--debugger` now prints source location information, instead of the
pointers of source location information. Before:
```
nix-repl> :bt
0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
0x600001522598
```
After:
```
0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
- The `--debugger` will start more reliably in `let` expressions and function calls [#6649](https://github.com/NixOS/nix/issues/6649) [#9917](https://github.com/NixOS/nix/pull/9917)
Previously, if you attempted to evaluate this file with the debugger:
```nix
let
a = builtins.trace "before inner break" (
builtins.break "hello"
);
b = builtins.trace "before outer break" (
builtins.break a
);
in
b
```
Nix would correctly enter the debugger at `builtins.break a`, but if you asked
it to `:continue`, it would skip over the `builtins.break "hello"` expression
entirely.
Now, Nix will correctly enter the debugger at both breakpoints.
- Nested debuggers are no longer supported [#9920](https://github.com/NixOS/nix/pull/9920)
Previously, evaluating an expression that throws an error in the debugger would
enter a second, nested debugger:
```
nix-repl> builtins.throw "what"
error: what
Starting REPL to allow you to inspect the current state of the evaluator.
Welcome to Nix 2.18.1. Type :? for help.
nix-repl>
```
Now, it just prints the error message like `nix repl`:
```
nix-repl> builtins.throw "what"
error:
… while calling the 'throw' builtin
at «string»:1:1:
1| builtins.throw "what"
| ^
error: what
```
- Consistent order of function arguments in printed expressions [#9874](https://github.com/NixOS/nix/pull/9874)
Function arguments are now printed in lexicographic order rather than the internal, creation-time based symbol order.
- Fix duplicate attribute error positions for `inherit` [#9874](https://github.com/NixOS/nix/pull/9874)
When an `inherit` caused a duplicate attribute error the position of the error was not reported correctly, placing the error with the inherit itself or at the start of the bindings block instead of the offending attribute name.
- `inherit (x) ...` evaluates `x` only once [#9847](https://github.com/NixOS/nix/pull/9847)
`inherit (x) a b ...` now evaluates the expression `x` only once for all inherited attributes rather than once for each inherited attribute.
This does not usually have a measurable impact, but side-effects (such as `builtins.trace`) would be duplicated and expensive expressions (such as derivations) could cause a measurable slowdown.
- Store paths are allowed to start with `.` [#912](https://github.com/NixOS/nix/issues/912) [#9091](https://github.com/NixOS/nix/pull/9091) [#9095](https://github.com/NixOS/nix/pull/9095) [#9120](https://github.com/NixOS/nix/pull/9120) [#9121](https://github.com/NixOS/nix/pull/9121) [#9122](https://github.com/NixOS/nix/pull/9122) [#9130](https://github.com/NixOS/nix/pull/9130) [#9219](https://github.com/NixOS/nix/pull/9219) [#9224](https://github.com/NixOS/nix/pull/9224) [#9867](https://github.com/NixOS/nix/pull/9867)
Leading periods were allowed by accident in Nix 2.4. The Nix team has considered this to be a bug, but this behavior has since been relied on by users, leading to unnecessary difficulties.
From now on, leading periods are supported. The names `.` and `..` are disallowed, as well as those starting with `.-` or `..-`.
Nix versions that denied leading periods are documented [in the issue](https://github.com/NixOS/nix/issues/912#issuecomment-1919583286).
- Introduction of `--regex` and `--all` in `nix profile remove` and `nix profile upgrade` [#10166](https://github.com/NixOS/nix/pull/10166)
Previously the command-line arguments for `nix profile remove` and `nix profile upgrade` matched the package entries using regular expression.
For instance:
```
nix profile remove '.*vim.*'
```
This would remove all packages that contain `vim` in their name.
In most cases, only singular package names were used to remove and upgrade packages. Mixing this with regular expressions sometimes lead to unintended behavior. For instance, `python3.1` could match `python311`.
To avoid unintended behavior, the arguments are now only matching exact names.
Matching using regular expressions is still possible by using the new `--regex` flag:
```
nix profile remove --regex '.*vim.*'
```
One of the most useful cases for using regular expressions was to upgrade all packages. This was previously accomplished by:
```
nix profile upgrade '.*'
```
With the introduction of the `--all` flag, this now becomes more straightforward:
```
nix profile upgrade --all
```
- Visual clutter in `--debugger` is reduced [#9919](https://github.com/NixOS/nix/pull/9919)
Before:
```
info: breakpoint reached
Starting REPL to allow you to inspect the current state of the evaluator.
Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
nix-repl> :continue
error: uh oh
Starting REPL to allow you to inspect the current state of the evaluator.
Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
nix-repl>
```
After:
```
info: breakpoint reached
Nix 2.20.0pre20231222_dirty debugger
Type :? for help.
nix-repl> :continue
error: uh oh
nix-repl>
```
- Cycle detection in `nix repl` is simpler and more reliable [#8672](https://github.com/NixOS/nix/issues/8672) [#9926](https://github.com/NixOS/nix/pull/9926)
The cycle detection in `nix repl`, `nix eval`, `builtins.trace`, and everywhere
else values are printed is now simpler and matches the cycle detection in
The `repl-flake` experimental feature has been removed. The `nix repl` command now works like the rest of the new CLI in that `nix repl {path}` now tries to load a flake at `{path}` (or fails if the `flakes` experimental feature isn't enabled).
### Other changes
-`nix eval` prints derivations as `.drv` paths [#10200](https://github.com/NixOS/nix/pull/10200)
`nix eval` will now print derivations as their `.drv` paths, rather than as
attribute sets. This makes commands like `nix eval nixpkgs#bash` terminate
instead of infinitely looping into recursive self-referential attributes:
// FIXME indentation carefully made for context, this is messed up.
.description=R"(
Howtocomputethehashoftheinput.
Oneof:
-`nar`(thedefault):Serialisestheinputasanarchive(followingthe[_NixArchiveFormat_](https://edolstra.github.io/pubs/phd-thesis.pdf#page=101)) and passes that to the hash function.
// FIXME indentation carefully made for context, this is messed up.
.description=R"(
Howtocomputethecontent-addressofthestoreobject.
Oneof:
-`nar`(thedefault):Serialisestheinputasanarchive(followingthe[_NixArchiveFormat_](https://edolstra.github.io/pubs/phd-thesis.pdf#page=101)) and passes that to the hash function.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
