The recent fix for CVE-2024-38531 broke the sandbox on macOS
completely. As it’s not practical to use `chroot(2)` on
macOS, the build takes place in the main filesystem tree, and the
world‐unreadable wrapper directory prevents the build from accessing
its `$TMPDIR` at all.
The macOS sandbox probably shouldn’t be treated as any kind of a
security boundary in its current state, but this specific vulnerability
wasn’t possible to exploit on macOS anyway, as creating `set{u,g}id`
binaries is blocked by sandbox policy.
Locking down the build sandbox further may be a good idea in future,
but it already has significant compatibility issues. For now, restore
the previous status quo on macOS.
Thanks to @alois31 for helping me come to a better understanding of
the vulnerability.
Fixes: 1d3696f0fbCloses: #11002
(cherry picked from commit af2e1142b1)
After the fix for CVE-2024-38531, this was only removing the nested
build directory, rather than the top‐level temporary directory.
Fixes: 1d3696f0fb
(cherry picked from commit 76e4adfaac)
This is useful for testing/debugging and maybe for sharing eval caches
(since it tells you what file in ~/.cache/nix/eval-cache-v5 to copy).
(cherry picked from commit 1ff186fc6e)
GitHub Actions seems to have magically switched architectures
without changing their identifiers.
See 2813ee66cb/README.md (available-images)
Maybe they have more complete documentation elsewhere, but it
seems to be incapable of selecting a runner based on architecture.
(cherry picked from commit df3e92ff96)
Previously, the .chroot directory had permission 750 or 755 (depending
on the uid-range system feature) and was owned by root/nixbld. This
makes it possible for any nixbld user (if uid-range is disabled) or
any user (if uid-range is enabled) to inspect the contents of the
chroot of an active build and maybe interfere with it (e.g. via /tmp
in the chroot, which has 1777 permission).
To prevent this, the root is now a subdirectory of .chroot, which has
permission 700 and is owned by root/root.
(cherry picked from commit ede95b1fc1)
Instead of running the builds under
`$TMPDIR/{unique-build-directory-owned-by-the-build-user}`, run them
under `$TMPDIR/{unique-build-directory-owned-by-the-daemon}/{subdir-owned-by-the-build-user}`
where the build directory is only readable and traversable by the daemon user.
This achieves two things:
1. It prevents builders from making their build directory world-readable
(or even writeable), which would allow the outside world to interact
with them.
2. It prevents external processes running as the build user (either
because that somehow leaked, maybe as a consequence of 1., or because
`build-users` isn't in use) from gaining access to the build
directory.
(cherry picked from commit 1d3696f0fb)
Previously (in cfc18a7739), we forgot to
compare the algo at all. This means we keep the same ordering as before
by making the stuff we always have compared take priority.
(cherry picked from commit 25a9894943)
* docs: fix python nix-shell example
This Python code snippet depended on Python 2 which has been marked as insecure in 24.05.
I modernized the example so new users will not be surprised upon copying and pasting the snippet for exploration.
Co-authored-by: John Ericson <git@JohnEricson.me>
Before, `-lnixutil` was just stuck in `nix-store.pc`, but that doesn't
seem so nice.
This prepares us to distribute `libnixutil` in a separate package if we
want, but it should be a good change either way. I suspect it wasn't
done before because libutil was an extra unstable interface, but I don't
think we need worry about that. *All* the C++ is less stable than the C
(or that's the goal at least).
For what it's worth, Lix also created this pkg-config file *en passant*
during their rename:
c97e17144e (diff-3c4f60cc44a0e35444c7f45331cfa50f76637118)
... so that we may perhaps later extend the interface.
Note that Nixpkgs' lib.warn already requires a string coercible
argument, so this is reasonable. Also note that string coercible
values aren't all strings, but in practice, for warn, they are.
Progress on #10832
This doesn't switch to auto-deriving the fields, but by defining `<=>`
we allow deriving `<=>` in downstream types where `Hash` is used.
Fixes assertion failure if outputsToInstall is empty by defaulting to the "out"
output. That is, behavior between the following commands should be consistent:
$ nix build --no-link --json .#nothing-to-install-no-out
error: derivation '/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nothing-to-install-no-out.drv' does not have wanted outputs 'out'
$ nix build --no-link --file default.nix --json nothing-to-install-no-out
error: derivation '/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nothing-to-install-no-out.drv' does not have wanted outputs 'out'
Real-world example of this issue:
$ nix build --json .#.legacyPackages.aarch64-linux.texlive.pkgs.iwona
error: derivation '/nix/store/dj0h6b0pnlnan5nidnhqa0bmzq4rv6sx-iwona-0.995b.drv' does not have wanted outputs 'out'
$ git rev-parse HEAD
eee33247cf6941daea8398c976bd2dda7962b125
$ nix build --json --file . texlive.pkgs.iwona
nix: src/libstore/outputs-spec.hh:46: nix::OutputsSpec::Names::Names(std::set<std::__cxx11::basic_string<char> >&&): Assertion `!empty()' failed.
Aborted (core dumped)
1. Fix build by making the legacy SSH Storey's secret `logFD` setting
not a setting on Windows. (It doesn't make sense to specify `void *`
handles by integer cross-proccess, I don't think.)
2. Move some files that don't need to be Unix-only anymore back to their
original locations.
Before:
$ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
fetching git input 'git+file:///home/linus/projects/lix'
fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: [json.exception.type_error.302] type must be string, but is null
After:
/tmp/inst/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent
warning: unknown experimental feature 'repl-flake'
error:
… while updating the lock file of flake 'git+file:///home/joerg/git/nix?ref=refs/heads/master&rev=62693c2c37c8edd92f95114eb1387b461fc671df'
… while updating the flake input 'nixpkgs'
… while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent'
error: No commits returned by GitLab API -- does the git ref really exist?
Adapted from: 3df013597d
This turns errors like:
error: flake output attribute 'hydraJobs' is not a derivation or path
into errors like:
error: expected flake output attribute 'hydraJobs' to be a derivation or
path but found a set: { binaryTarball = «thunk»; build = «thunk»; etc> }
This change affects all InstallableFlake commands.
Source: 20981461d4
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
* docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation
Both of these still needs their own actual documentation, but they are
at least now mentioned that they exist and what they're enabled by.
Co-authored-by: Qyriad <qyriad@qyriad.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
- Get a rump derivation goal: hook instance will come later, local
derivation goal will come after that.
- Start cleaning up the channel / waiting code with an abstraction.
By moving `host` to the config, we can do a lot further cleanups and
dedups. This anticipates a world where we always go `StoreReference` ->
`*StoreConfig` -> `Store*` rather than skipping the middle step too.
Progress on #10766
Progress on https://github.com/NixOS/hydra/issues/1164
This increases test coverage, and gets the worker protocol ready to be
used by Hydra.
Why don't we just try to use the store interface in Hydra? Well, the
problem is that the store interface works on connection pools, with each
opreation getting potentially a different connection, but the way temp
roots work requires that we keep one logical "transaction" (temp root
session) using the same connection.
The longer-term solution probably is making connections themselves
implement the store interface, but that is something that builds on
this, so I feel OK that this is not churn in the wrong direction.
Fixes#9584
We don't want to rely on how C assigns numbers for enums in the wire
format. Sure, this is totally determined by the ABI, but it obscures the
code and makes it harder to safely change the enum definition (should we
need to) without accidentally breaking the wire format.
Do this instead of an unchecked cast
I redid this to use the serialisation framework (including a unit test),
but I am keeping the reference to credit Jade for spotting the issue.
Change-Id: Icf6af7935e8f139bef36b40ad475e973aa48855c
(adapted from commit 2a7a824d83dc5fb33326b8b89625685f283a743b)
Co-Authored-By: Jade Lovelace <lix@jade.fyi>
File not found while importing is not currently caught by the tab-completion handler.
Original bug report: https://git.lix.systems/lix-project/lix/issues/340
Fix has been adapted from https://gerrit.lix.systems/c/lix/+/1189
Example crash:
$ cat /tmp/foo.nix
{
someImport = import ./this_file_does_not_exist;
}
$ ./src/nix/nix repl --file /tmp/foo.nix
warning: unknown experimental feature 'repl-flake'
Nix 2.23.0pre20240517_dirty
Type :? for help.
Loading installable ''...
Added 1 variables.
nix-repl> someImport.<TAB>
This way we can commit the same amount of stack size (64 MB) without a conditional.
Includes nix, libnixexpr-tests, libnixfetchers-tests, libnixstore-tests, libnixutil-tests.
https://github.com/NixOS/nix/pull/10555 added a check requiring
that output parameters always have an uninitialized Value as argument.
Unfortunately the output parameter of the primop callback received
a thunk instead.
See the comment for implementation considerations.
This was accidentally removed in
e989c83b44. I restored it and also did a
few other cleanups:
- Make a static method for namespacing purposes
- Put the test files in the data dir with the other test data
- Avoid mutating globals in the machine config tests
This will be used by Hydra.
* reword documentation on `nix-copy-closure`
- one sentence per line
- be more precise with respect to which Nix stores are being accessed
- make a clear distinction between store paths and store objects
- add links to definitions of terms
- clarify which machine is which
- --to and --from don't take arguments
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In particular `local://<path>` and `unix://` (without any path) now
work, and mean the same things as `local` and `daemon`, respectively. We
thus now have the opportunity to desguar `local` and `daemon` early.
This will allow me to make a change to
https://github.com/NixOS/nix/pull/9839 requested during review to
desugar those earlier.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
1. Hydra currently queries for multiple path infos at once, so let us
make a connection item for that.
2. The minimum of the two versions should always be used, see #9584.
(The issue remains open because the daemon protocol needs to be
likewise updated.)
The JSON format no longer uses the legacy ATerm `r:` prefixing nonsese,
but separate fields.
Progress on #9866
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
- add links to definitions of terms
- one sentence per line
- be more specific about which store is used for the import
- clearly distinguish store paths and store objects
- make a recommendation to use `nix-copy-closure` for efficient SSH transfers
Basically I'd expect the same behavior as with `nix-build`, i.e.
with `--keep-going` the hash-mismatch error of each failing
fixed-output derivation is shown.
The approach is derived from `Store::buildPaths` (`entry-point.cc`):
instead of throwing the first build-result, check if there are any build
errors and if so, display all of them and throw after that.
Unfortunately, the BuildResult struct doesn't have an `ErrorInfo`
(there's a FIXME for that at least), so I have to construct my own here.
This is a rather cheap bugfix and I decided against touching too many
parts of libstore for that (also I don't know if that's in line with the
ongoing refactoring work).
Closes https://git.lix.systems/lix-project/lix/issues/302
Change-Id: I378ab984fa271e6808c6897c45e0f070eb4c6fac
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
On several occasions I've found myself confused when trying to delete
a store path, because I am told it's still alive, but
nix-store --query --roots doesn't show anything. Let's save future
users this confusion by mentioning that a path might be alive due to
having referrers, not just roots.
given `nix-copy-closure` exists, it doesn't make much sense to do
nix-store --export $paths | nix-store --import --store ssh://foo@bar
since that dumps everything rather than granularly transferring store
objects as needed.
therefore, pick an example where dumping the entire closure into a file
actually makes a difference, such as when deploying to airgapped systems.
In addition:
- Take the opportunity to add a bunch more missing hyperlinks, too.
- Remove some glossary entries that are now subsumed by dedicated pages.
We used to not be able to do this without breaking link fragments, but
now we can, so pick up where we left off.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
- add links to definitions of terms
- one sentence per line
- be more specific about which store is used for the import
- clearly distinguish store paths and store objects
- make a recommendation to use `nix-copy-closure` for efficient SSH transfers
the individual commands' documentation should provide enough examples to
make sense of the options and judge what to use and when. proper guides,
which would require a more elaborate setup to show off Nix's
capabilities are out of scope for the reference manual.
* doc: convention improvements for copying closure
use -P, which only considers executables but not shell builtins
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
previously the test directory could have been left untouched before executing
a test when `init.sh` was not run - and sometimes it isn't
supposed to be run - which made the test suite highly stateful and thus
behaving surprisingly on multiple runs.
pararameterisation is not actually needed the way things are currently
set up, and it confused me when trying to understand what the code does.
all but one test sources vars-and-functions.sh, which nominally only
defines variables, but in practice is always coupled with the actual
initialisation. while the cleaner way of making this more legible would
be to source variables and initialisation separately, this would produce
a huge diff.
the change requires a few small fixes to keep the tests working:
- only create test home directory during initialisation
that vars-and-functions.sh wrote to the file system seems not write
- fix creation of the test directory
due to statefulness, the test home directory was implicitly creating
the test root, too. decoupling that made it apparent that this was
probably not intentional, and certainly confusing.
- only source vars-and-functions.sh if init.sh is not needed
there is one test case that only needs a helper function but no
initialisation side effects
- remove some unnecessary cleanups and split parts of re-used test code
there were confusing bits in how initialisation code was repurposed,
which break if trying to refactor the outer layers naively...
This is useful for diagnosing whether an evaluation is copying large
paths to the store. Example:
$ nix build .#packages.x86_64-linux.default --large-path-warning-threshold 1000000
warning: copied large path '/home/eelco/Dev/nix-master/' to the store (6271792 bytes)
warning: copied large path '«github:NixOS/nixpkgs/b550fe4b4776908ac2a861124307045f8e717c8e?narHash=sha256-7kkJQd4rZ%2BvFrzWu8sTRtta5D1kBG0LSRYAfhtmMlSo%3D»/' to the store (155263768 bytes)
warning: copied large path '«github:libgit2/libgit2/45fd9ed7ae1a9b74b957ef4f337bc3c8b3df01b5?narHash=sha256-oX4Z3S9WtJlwvj0uH9HlYcWv%2Bx1hqp8mhXl7HsLu2f0%3D»/' to the store (22175416 bytes)
warning: copied large path '/nix/store/z985088mcd6w23qwdlirsinnyzayagki-source' to the store (5885872 bytes)
This fixes https://github.com/NixOS/nixpkgs/pull/290775 by not expanding aliases
when sourcing the stdenv setup script. The way bash handles aliases is to expand
them when a function is defined, not when it is used. I.e.:
$ alias echo="echo bar "
$ echo foo
bar foo
$ xyzzy() { echo foo; }
$ shopt -u expand_aliases
$ xyzzy
bar foo
$ xyzzy2() { echo foo; }
$ xyzzy2
foo
The problem is that ~/.bashrc is sourced before the stdenv setup, and bashrc
commonly sets aliases for ‘cp’, ‘mv’ and ‘rm’ which you don’t want to take
effect in the stdenv derivation builders. The original commit introducing this
feature (5fd8cf7667) even mentioned this very
alias.
The only way to avoid this is to disable aliases entirely while sourcing the
stdenv setup, and reenable them afterwards.
Passing the commit message as an argument causes update failures on repositories with lots of flake inputs. In some cases, the commit message is over 250,000 bytes.
the old `copyFile` was just a wrapper that was calling the `copy`
function. This wrapper function is removed and the `copy` function is
renamed to `copyFile`.
Building derivations is a lot harder, but the downloading goals is
portable enough.
The "common channel" code is due to Volth. I wonder if there is a way we
can factor it out into separate functions / files to avoid some
within-function CPP.
Co-authored-by: volth <volth@volth.com>
The warning was done to handle older Nix releases that didn't have
Docker images (091f232896), but this was
a bad idea because it causes us to silently skip uploading Docker
images if e.g. Hydra hasn't finished building them yet.
Issue #10648.
In streaming mode, libarchive doesn't handle symlinks in zip files
correctly. So write the entire file to disk so libarchive can access
it in random-access mode.
Fixes#10649. This was broken in cabee98152.
builtins.strictDerivation returns an attribute set with drvPath and
output paths. For some reason, current implementation forbids drv
instead of drvPath.
Remove `isLink` in favor of `std::filesystem::is_link`
This is one step closer to eventually getting rid of most of our file system utils (in `file-system.cc`) in favor of the `std::filesystem`.
- specify meeting times in terms of a time zone rather than standard
time (the first encompasses standard time changes)
- add information on who can participate and how
- unrelated but still important: add GitHub handle to contact the team
Sometimes we read a directory with children we cannot stat. It's a pitty
we even try to stat at all (wasteful) in the `DT_UNKNOWN` case, but at
least this should get rid of the failure.
This was changed in #10611, which caused the derivation paths of
anything using builtin:fetchurl to change (i.e. all of
Nixpkgs). However, impureEnvVars doesn't actually do anything for
builtin:fetchurl, so we can just set it to its historical value.
This makes for shorter and more portable code.
The only tricky part is catching exceptions: I just searched for near by
`catch (Error &)` or `catch (SysError &)` and adjusted them to `catch
(std::filesystem::filesystem_error &)` according to my human judgement.
Good for windows portability; will help @siddhantk232 with his GSOC
project.
Different parts of the project honor different sets of proxy environment
variables. With this commit all parts of the project will honor the same
set of proxy environment variables.
---------
Co-authored-by: Your Name <you@example.com>
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Now that SourcePath uses a SourceAccessor instead of an InputAccessor,
we can use it in function signatures instead of passing a
SourceAccessor and CanonPath separately.
Make sure that `extraSandboxProfile` is set before we check whether it's
empty or not (in the `sandbox=true` case).
Also adds a test case for this.
Co-Authored-By: Artemis Tosini <lix@artem.ist>
Co-Authored-By: Eelco Dolstra <edolstra@gmail.com>
After the removal of the InputAccessor::fetchToStore() method, the
only remaining functionality in InputAccessor was `fingerprint` and
`getLastModified()`, and there is no reason to keep those in a
separate class.
Fix formatting violations, update blacklist to reflect moved files.
PR #10556 passed CI before the new formating rules were added, and our
CI has the race condition of allowing old results, resulting in master
getting broken.
This missing GC root wasn't much of a problem before, because the
heap would end up with a reference to the `baseEnv` pretty soon,
but when unit testing, the construction of `EvalState` doesn't
necessarily happen well before GC runs for the first time.
Found while unit testing the Rust bindings that currently reside
at https://github.com/nixops4/nixops4/tree/main/rust
When trying the „nix-store info“ commands on this page I received the error "error: 'info' is not a recognised command". According to https://github.com/NixOS/nix/issues/9349 info seems to have been an alias for ping. So why not just replace info with ping?
This makes it match the current pattern:
- `package.nix` assumes deps are right version
- Overlay in `flake.nix` creates `*-nix` package variations
- Overlay manually passes in those packages to `package.nix`
* move single-user uninstall to the end
this is not the default method of installation, and therefore irrelevant
for most users.
* move the backup restore instructions to the first step
for most users we can expect that the system-wide shell init files were
not ever touched, so we can as well tell them to do the most likely
thing.
from experience, while it's not necessarily safe to just mess with these
files, most people are simply confused by the complexity of
instructions.
* provide more detailed instructions for using `sudo vifs`
we can expect most beginners not to ever have used `vi`, and they will
probably need some hand-holding.
* express instructions as a script
Co-authored-by: wamirez <wamirez@protonmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Closes#10585
As it turns out, libseccomp maintains an internal syscall table and
validates each rule against it. This means that when using libseccomp
2.5.4 or older, one may pass `452` as syscall number against it, but
since it doesn't exist in the internal structure, `libseccomp` will refuse
to create a filter for that. This happens with nixpkgs-23.11, i.e. on
stable NixOS and when building Nix against the project's flake.
To work around that
* a backport of libseccomp 2.5.5 on upstream nixpkgs has been
scheduled[1].
* the package now uses libseccomp 2.5.5 on its own already. This is to
provide a quick fix since the correct fix for 23.11 is still a staging cycle
away.
It must not be possible to build a Nix with an incompatible libseccomp
version (nothing can be built in a sandbox on Linux!), so configure.ac
rejects libseccomp if `__SNR_fchmodat2` is not defined.
We still need the compat header though since `SCMP_SYS(fchmodat2)`
internally transforms this into `__SNR_fchmodat2` which points to
`__NR_fchmodat2` from glibc 2.39, so it wouldn't build on glibc 2.38.
The updated syscall table from libseccomp 2.5.5 is NOT used for that
step, but used later, so we need both, our compat header and their
syscall table 🤷
[1] https://github.com/NixOS/nixpkgs/pull/306070
Add a method to check if a value has been initialized. This helps avoid
segfaults when calling `type()`.
Useful in the context of the new C API.
Closes#10524
I've added the new local.mk to the package sources. While this
should not be needed for the build, it is the simplest solution,
and won't cause many extra rebuilds, because the file won't change
very often.
This reverts commit 62feb5ca09263c78ddb692836228223e5b58d3ae.
It runs as part of the functional tests, which control the environment,
solving some of the problems a default config has when run in the
sandbox.
Windows now has some basic Unix Domain Socket support, see
https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/
Building `nix daemon` on Windows I've left for later, because the daemon
currently forks per connection but this is not an option on Windows. But
we can get the client part working right away.
With Linux kernel >=6.6 & glibc 2.39 a `fchmodat2(2)` is available that
isn't filtered away by the libseccomp sandbox.
Being able to use this to bypass that restriction has surprising results
for some builds such as lxc[1]:
> With kernel ≥6.6 and glibc 2.39, lxc's install phase uses fchmodat2,
> which slips through 9b88e52846/src/libstore/build/local-derivation-goal.cc (L1650-L1663).
> The fixupPhase then uses fchmodat, which fails.
> With older kernel or glibc, setting the suid bit fails in the
> install phase, which is not treated as fatal, and then the
> fixup phase does not try to set it again.
Please note that there are still ways to bypass this sandbox[2] and this is
mostly a fix for the breaking builds.
This change works by creating a syscall filter for the `fchmodat2`
syscall (number 452 on most systems). The problem is that glibc 2.39
and seccomp 2.5.5 are needed to have the correct syscall number available
via `__NR_fchmodat2` / `__SNR_fchmodat2`, but this flake is still on
nixpkgs 23.11. To have this change everywhere and not dependent on the
glibc this package is built against, I added a header
"fchmodat2-compat.hh" that sets the syscall number based on the
architecture. On most platforms its 452 according to glibc with a few
exceptions:
$ rg --pcre2 'define __NR_fchmodat2 (?!452)'
sysdeps/unix/sysv/linux/x86_64/x32/arch-syscall.h
58:#define __NR_fchmodat2 1073742276
sysdeps/unix/sysv/linux/mips/mips64/n32/arch-syscall.h
67:#define __NR_fchmodat2 6452
sysdeps/unix/sysv/linux/mips/mips64/n64/arch-syscall.h
62:#define __NR_fchmodat2 5452
sysdeps/unix/sysv/linux/mips/mips32/arch-syscall.h
70:#define __NR_fchmodat2 4452
sysdeps/unix/sysv/linux/alpha/arch-syscall.h
59:#define __NR_fchmodat2 562
I tested the change by adding the diff below as patch to
`pkgs/tools/package-management/nix/common.nix` & then built a VM from
the following config using my dirty nixpkgs master:
{
vm = { pkgs, ... }: {
virtualisation.writableStore = true;
virtualisation.memorySize = 8192;
virtualisation.diskSize = 12 * 1024;
nix.package = pkgs.nixVersions.nix_2_21;
};
}
The original issue can be triggered via
nix build -L github:nixos/nixpkgs/d6dc19adbda4fd92fe9a332327a8113eaa843894#lxc \
--extra-experimental-features 'nix-command flakes'
however the problem disappears with this patch applied.
Closes#10424
[1] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2031073804
[2] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2030844251
With Nix 2.3, it was possible to pass a subpath of a store path to
exportReferencesGraph:
with import <nixpkgs> {};
let
hello = writeShellScriptBin "hello" ''
echo ${toString builtins.currentTime}
'';
in
writeClosure [ "${hello}/bin/hello" ]
This regressed with Nix 2.4, with a very confusing error message, that
presumably indicates it was unintentional:
error: path '/nix/store/3gl7kgjr4pwf03f0x70dgx9ln3bhl7zc-hello/bin/hello' is not in the Nix store
At this point many features are stripped out, but this works:
- Can run libnix{util,store,expr} unit tests
- Can run some Nix commands
Co-Authored-By volth <volth@volth.com>
Co-Authored-By Brian McKenna <brian@brianmckenna.org>
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_expr.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_expr.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_external.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_external.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_value.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_value.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_store.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_store.h'
local.mk:5: warning: overriding recipe for target 'outputs/dev/include/nix/nix_api_util.h'
local.mk:5: warning: ignoring old recipe for target 'outputs/dev/include/nix/nix_api_util.h'
This also reworks the Mercurial fetcher (which was still using the
old cache interface) to have two distinct cache mappings:
* A ref-to-rev mapping, which is store-independent.
* A rev-to-store-path mapping.
Code operating on store objects (including creating them) should, in
general, use `ContentAddressMethod` rather than `FileIngestionMethod`.
See also dfc876531f which included some
similar refactors.
See https://github.com/NixOS/nix/pull/8699#discussion_r1554312181
Casting a function pointer to `void*` is undefined behavior in the C
spec, since there are platforms with different sizes for these two kinds
of pointers. A safe alternative might be `void (*callback)()`
https://github.com/NixOS/nix/pull/10456 fixed the addition of symlink
store paths to the sandbox, but also made it so that the hardcoded
sandbox paths (like `/etc/hosts`) were now bind-mounted without
following the possible symlinks. This made these files unreadable if
there were symlinks (because the sandbox would now contain a symlink to
an unreachable file rather than the underlying file).
In particular, this broke FOD derivations on NixOS as `/etc/hosts` is a
symlink there.
Fix that by canonicalizing all these hardcoded sandbox paths before
adding them to the sandbox.
Like always declining; local builds only, as can be inferred from the
docs. (Not worth spending too many words on this pretty obvious
behavior, I think. Also, plans to remove it? https://github.com/NixOS/nix/issues/1221)
This requires moving resolveSymlinks() into SourceAccessor. Also, it
requires LocalStoreAccessor::maybeLstat() to work on parents of the
store (to avoid an error like "/nix is not in the store").
Fixes#10375.
Instead of relying on setup script to set output variables when
structured attributes are enabled, iterate over the values of an
outputs associative array.
See also
374fa3532e/pkgs/stdenv/generic/setup.sh (L23-L26)
Bind-mounting symlinks is apparently not possible, which is why the
thing was failing.
Fortunately, symlinks are small, so we can fallback to copy them at no cost.
Fix https://github.com/NixOS/nix/issues/9579
Co-authored-by: Artturin <Artturin@artturin.com>
Now that we have a few things identifying content address methods by
name, we should be consistent about it.
Move up the `parseHashAlgoOpt` for tidiness too.
Discussed this change for consistency's sake as part of #8876
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
This requires `--substitute-on-destination` if you want the remote side
to substitute instead of copying if possible.
For completeness sake, document it here.
Also, the stable Nix from nixpkgs is still 2.18, so more folks may
stumble upon this when this is bumped, so I'd expect this to be actually
useful.
Closes#10182
This was used in only one place, namely builtins.fetchurl with an
expected hash. Since this can cause similar issues as described
in #9814 and #9905 with the "locked" flag for fetchTarball and fetchTree,
let's just remove it.
Note that if an expected hash is given and the hash algorithm is
SHA-256, then we will never do a download anyway if the resulting
store path already exists. So removing the "locked" flag will only
cause potentially unnecessary HTTP requests (subject to the tarball
TTL) for non-SHA-256 hashes.
This probably snuck in in a refactor using truthiness or so. The
trustedness flag was having the optional fullness checked, rather than
the actual contained trust level.
Also adds some tests.
```
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix store ping
warning: 'nix store ping' is a deprecated alias for 'nix store info'
Store URL: daemon
Version: 2.20.4
Trusted: 0
m1@6876551b-255d-4cb0-af02-8a4f17b27e2e ~ % nix doctor
warning: 'doctor' is a deprecated alias for 'config check'
[PASS] PATH contains only one nix version.
[PASS] All profiles are gcroots.
[PASS] Client protocol matches store protocol.
[INFO] You are trusted by store uri: daemon
```
The script at `/nix/store/...-nix-2.21.0/etc/profile.d/nix-daemon.sh` was leaving behind a variable, which was visible in the user's shell environment, but not used outside the script.
code blocks, if not surrounded by empty lines, have the language
tags (in these cases, always `nix`) show up in the output of :doc.
for example:
nix-repl> :doc builtins.parseFlakeRef
Synopsis: builtins.parseFlakeRef flake-ref
Parse a flake reference, and return its exploded form.
For example: nix builtins.parseFlakeRef
"github:NixOS/nixpkgs/23.05?dir=lib" evaluates to: nix { dir =
"lib"; owner = "NixOS"; ref = "23.05"; repo = "nixpkgs"; type =
"github"; }
is now instead:
nix-repl> :doc builtins.parseFlakeRef
Synopsis: builtins.parseFlakeRef flake-ref
Parse a flake reference, and return its exploded form.
For example:
| builtins.parseFlakeRef "github:NixOS/nixpkgs/23.05?dir=lib"
evaluates to:
| { dir = "lib"; owner = "NixOS"; ref = "23.05"; repo = "nixpkgs"; type = "github"; }
When querying all paths in a binary cache store, the path's representation
is `<hash>-x` (where `x` is the value of `MissingName`) because the .narinfo
filenames only contain the hash.
Before cc46ea1630 this worked correctly,
because the entire path info was read and the path from this
representation was printed, i.e. in the form `<hash>-<name>`. Since then
however, the direct result from `queryAllValidPaths()` was used as `path`.
Added a regression test to make sure the behavior remains correct.
This was part of approved PR #10021. Unfortunately that one is stalled
on a peculiar Linux test timeout, so trying to get bits of it merged
first to bisect failure.
Forcing a conditional include, vs making the headers content
conditional, I think is more maintainable.
It is also how the other platform-specific headers (like
`namespaces.hh`) have been adapted.
A possible use of them might have been to figure out the paths
(which can now be retrieved with maybePathsOut), but I have
not found evidence that it was used this way, and it would have
been broken, because non-CA outputs weren't recorded in the map.
It's a little weird we don't check the return status for these, but
changing that would introduce risk so I did not.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
This introduces new utility functions to get elements from JSON — in an ergonomic way and with nice error messages if the expected type does not match.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
* show Nix logo in the manual
the location of files is hard-coded by mdBook.
there is also seems to be no way to define custom templates, therefore
all styling has to be done in the CSS override.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This splits files and adds new identifiers in preperation for supporting
windows, but no Windows-specific code is actually added yet.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This patch makes `makeDecompressionSink` strip only a single layer
of compression specified via method. This fixes erroneous decompression
of doubly-compressed NARs fetched with curl.
This function is nice for more than `PosixSourceAccessor`. We can make a
few things simpler with it.
Note that the error logic slightly changes in some of the call sites, in
that we also count `ENOTDIR` and not just `ENOENT` as not having the
file, but that should be fine.
HintFmt(string) invokes the HintFmt("%s", literal) constructor,
which is not what we want here. Add a constructor with a proper name
and call that.
Next step: rename all the other ones to HintFmt::literal(string).
Fixes https://github.com/NixOS/nix/issues/10238
Thunks are now overwritten by a helper function
`Value::finishValue(newType, payload)` (where `payload` is the
original anonymous union inside `Value`). This helps to ensure we
never update a value elsewhere, since that would be incompatible with
parallel evaluation (i.e. after a value has transitioned from being a
thunk to being a non-thunk, it should be immutable).
There were two places where this happened: `Value::mkString()` and
`ExprAttrs::eval()`.
This PR also adds a bunch of accessor functions for value contents,
like `Value::integer()` to access the integer field in the union.
I realized it was checking NAR hashes before of added objects, which
makes little sense --- we don't really care about ancillary NAR hashes.
Now, the bottom `nix store add` tests compare the CA field with a git
hash to hashes calculated by Git. This matches top `nix hash path` ones
in using git as a source of truth.
Fixes an instance of
nix: src/libutil/util.cc:139: nix::Path nix::canonPath(PathView, bool): Assertion `path != ""' failed.
... which I've been getting in one of my shells for some reason.
I have yet to find out why TMPDIR was empty, but it's no reason for
Nix to break.
I was using by mistake the .#nix-clangStdenv shell to retrieve clangd.
This clangd is unusable with the project and constantly segfaults.
Let's explicitly state which shell the user should use in the docs.
I don't really understand the source of this segfault. I assume it's
related to a clang version incompatibility. (16.0.6 for
.#nix-clangStdenv 14.0.6 for .#native-clangStdenvPackages)
Fixes
terminate called after throwing an instance of 'boost::wrapexcept<boost::io::too_few_args>'
what(): boost::too_few_args: format-string referred to more arguments than were passed
Aborted (core dumped)
for type errors in AttrCursor.
- Align the “frequent” release cycle with the calendar
- The 6-month release cycle is hard to keep track of. A monthly
release will make it much easier to remember the release date.
- Officialise the support for a stable version maintained for as long as NixOS stable
- This is already the case in practice, it just happens that the
“stable” Nixpkgs version is whichever version was deemed
stable-enough at the time of the NixOS release.
Officialise that by cutting a new major release alongside each NixOS one.
Note that this breaks whatever semver compatibility Nix might pretend to
have, but I don't think it makes sense any way.
In a daemon-based Nix setup, some options cannot be overridden by a
client unless the client's user is considered trusted.
Currently, if an untrusted user tries to override one of those
options, we are silently ignoring it.
This can be pretty confusing in certain situations.
e.g. a user thinks he disabled the sandbox when in reality he did not.
We are now sending a warning message letting know the user some options
have been ignored.
Related to #1761.
This is a cherry-pick of 9e0f5f803f.
The above commit has been reverted by
a59e77d9e5 to prevent spamming warnings
with experimental features, but these are now totally ignored on the
daemon side, so there's no reason for the revert any more.
Previously, `state.mkList()` would set the type of the value to tList
and allocate the list vector, but it would not initialize the values
in the list. This has two problems:
* If an exception occurs, the list is left in an undefined state.
* More importantly, for multithreaded evaluation, if a value
transitions from thunk to non-thunk, it should be final (i.e. other
threads should be able to access the value safely).
To address this, there now is a `ListBuilder` class (analogous to
`BindingsBuilder`) to build the list vector prior to the call to
`Value::mkList()`. Typical usage:
auto list = state.buildList(size);
for (auto & v : list)
v = ... set value ...;
vRes.mkList(list);
* Add regression test
* Fix 'no repo' test so it doesn't succeed if the data is still in cache
* Use git_revparse_single inside git-utils instead of reimplementing the same logic.
Add `runHook preInstallCheck` to the overriden `installCheckPhase` used
for the non-build case.
In particular, this allow the fix from 2a34510776
to also apply there.
Previously, errors while printing values in `nix repl` would be printed
in `«error: ...»` brackets rather than displayed normally:
```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
«error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.»
```
Now, errors will be displayed normally if they're emitted at the
top-level of an expression:
```
nix-repl> legacyPackages.aarch64-darwin.pythonPackages.APScheduler
error:
… in the condition of the assert statement
at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/lib/customisation.nix:268:17:
267| in commonAttrs // {
268| drvPath = assert condition; drv.drvPath;
| ^
269| outPath = assert condition; drv.outPath;
… in the left operand of the OR (||) operator
at /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/passthrufun.nix:28:45:
27| if lib.isDerivation value then
28| lib.extendDerivation (valid value || throw "${name} should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.") {} value
| ^
29| else
(stack trace truncated; use '--show-trace' to show the full trace)
error: Package ‘python-2.7.18.7’ in /nix/store/6s0m1qc31zw3l3kq0q4wd5cp3lqpkq0q-source/pkgs/development/interpreters/python/cpython/2.7/default.nix:335 is marked as insecure, refusing to evaluate.
```
Errors emitted in nested structures (like e.g. when printing `nixpkgs`)
will still be printed in brackets.
Strings are now printed directly when evaluated by `:print`, rather than
escaped. This makes it easier to debug multi-line strings or strings
containing quotes, like the results of `builtins.readFile`,
`lib.toShellArg`, and so on.
```
nix-repl> "cuppy\ndog\ncity"
"cuppy\ndog\ncity"
nix-repl> :p "cuppy\ndog\ncity"
cuppy
dog
city
```
Fixes this very long warning, which I'll only include the first line of:
/nix/store/8wrjhrycpshhc3b41xmjwvgqr2m3yajq-libcxx-16.0.6-dev/include/c++/v1/__memory/construct_at.h:66:5: warning: destructor called on non-final 'RegexMatcher' that has virtual functions but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor]
__loc->~_Tp();
`nix eval` forces values and prints derivations as attribute sets, so
commands that print derivations (e.g. `nix eval nixpkgs#bash`) will
infinitely loop and segfault.
Printing derivations as `.drv` paths makes `nix eval` complete as
expected. Further work is needed, but this is better than a segfault.
`nix-env -qaP`'s output has changed a bit because of https://github.com/NixOS/nix/issues/10132.
Although that's a bit annoying, it isn't nearly as problematic as the
evaluation changes that this test is supposed to catch. So it's find to
just update the hash for the time being and fix the issue later
(properly fixing the issue will very likely change the hash any way).
Currently there isn't a convenient way to check for multiline output. In
addition, these outputs will easily change and having a diff between the
expected an the actual output upon failures is convenient.
This wasn't caught by CI because #10149 and #10152 pass
individually... It doesn't happen on lazy-trees either because we
never try to fetch relative path flakes (#10089).
we now keep not a table of all positions, but a table of all origins and
their sizes. position indices are now direct pointers into the virtual
concatenation of all parsed contents. this slightly reduces memory usage
and time spent in the parser, at the cost of not being able to report
positions if the total input size exceeds 4GiB. this limit is not unique
to nix though, rustc and clang also limit their input to 4GiB (although
at least clang refuses to process inputs that are larger, we will not).
this new 4GiB limit probably will not cause any problems for quite a
while, all of nixpkgs together is less than 100MiB in size and already
needs over 700MiB of memory and multiple seconds just to parse. 4GiB
worth of input will easily take multiple minutes and over 30GiB of
memory without even evaluating anything. if problems *do* arise we can
probably recover the old table-based system by adding some tracking to
Pos::Origin (or increasing the size of PosIdx outright), but for time
being this looks like more complexity than it's worth.
since we now need to read the entire input again to determine the
line/column of a position we'll make unsafeGetAttrPos slightly lazy:
mostly the set it returns is only used to determine the file of origin
of an attribute, not its exact location. the thunks do not add
measurable runtime overhead.
notably this change is necessary to allow changing the parser since
apparently nothing supports nix's very idiosyncratic line ending choice
of "anything goes", making it very hard to calculate line/column
positions in the parser (while byte offsets are very easy).
this needs a string comparison because there seems to be no other way to
get that information out of bison. usually the location info is going to
be correct (pointing at a bad token), but since EOF isn't a token as
such it'll be wrong in that this case.
this hasn't shown up much so far because a single line ending *is* a
token, so any file formatted in the usual manner (ie, ending in a line
ending) would have its EOF position reported correctly.
the parser treats a plain \r as a newline, error reports do not. this
can lead to interesting divergences if anything makes use of this
feature, with error reports pointing to wrong locations in the input (or
even outside the input altogether).
previously we reported the error at the beginning of the binding
block (for plain inherits) or the beginning of the attr list (for
inherit-from), effectively hiding where exactly the error happened.
this also carries over to runtime positions of attributes in sets as
reported by unsafeGetAttrPos. we're not worried about this changing
observable eval behavior because it *is* marked unsafe, and the new
behavior is much more useful.
we already normalize attr order to lexicographic, doing the same for
formals makes sense. doubly so because the order of formals would
otherwise depend on the context of the expression, which is not quite as
useful as one might expect.
the parser modifies its inputs, which means that sharing them between
the error context reporting system and the parser itself can confuse the
reporting system. usually this led to early truncation of error context
reports which, while not dangerous, can be quite confusing.
* Convert all InputScheme::fetch() methods to getAccessor().
* Add checkLocks() method for checking lock attributes.
* Rename fetch() to fetchToStore().
This is the case for e.g. dirty Git workdirs, where we would get
$ nix flake metadata
Resolved URL: git+file:///home/eelco/Dev/nix-master
Locked URL: git+file:///home/eelco/Dev/nix-master
The "lockedRef" field is a misnomer, since it can be unlocked
(e.g. for a dirty Git workdir). In that case, `nix profile upgrade`
needs to assume that the package can have changed, and perform an
upgrade.
When reviewing old PRs, I found that #9997 adds some code to ensure one
particular assert is always present. But, removing asserts isn't
something we do in our own release builds either in the flake here or in
nixpkgs, and is plainly a bad idea that increases support burden,
especially if other distros make bad choices of build flags in their Nix
packaging.
For context, the assert macro in the C standard is defined to do nothing
if NDEBUG is set.
There is no way in our build system to set -DNDEBUG without manually
adding it to CFLAGS, so this is simply a configuration we do not use.
Let's ban it at compile time.
I put this preprocessor directive in src/libutil.cc because it is not
obvious where else to put it, and it seems like the most logical file
since you are not getting a usable nix without it.
Directly fail if a flakeref points to something that isn't a directory
instead of falling back to the logic of trying to look up the hierarchy
to find a valid flake root.
Fix https://github.com/NixOS/nix/issues/9868
After https://github.com/NixOS/nix/pull/10071, the CI was trying to push
ghcr.io/nixos/nix:master for backwards-compatibility, but the image was
not tagged as such, causing the job to fail.
Fix this.
It is possible to exfiltrate a file descriptor out of the build sandbox
of FODs, and use it to modify the store path after it has been
registered.
To avoid that issue, don't register the output of the build, but a copy
of it (that will be free of any leaked file descriptor).
`NIX_HARDENING_ENABLE` causes `_FORTIFY_SOURCE` to be defined.
This isn't compatible with `-O0`, and the compiler will happily remind
us about it at every call, spamming the terminal with warnings and stack
traces.
We don't really care hardening in that case, so just disable it if we
pass `OPTIMIZE=0`.
Just `stdenv.isDarwin` isn't enough because it doesn't apply to the
build platform, which mean that cross packages building from darwin to
another platform will have `isDarwin` set to false.
Replace it by `stdenv.buildPlatform.isDarwin`.
Instead, serialize as NAR and send that over, then rehash sever side.
This is alorithmically simpler, but comes at the cost of a newer
parameter to `Store::addToStoreFromDump`.
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
desugaring inherit-from to syntactic duplication of the source expr also
duplicates side effects of the source expr (such as trace calls) and
expensive computations (such as derivationStrict).
Flakes still reside in the Nix store (so there shouldn't be any change
in behaviour), but they are now accessed via the rootFS
accessor. Since rootFS implements access checks, we no longer have to
worry about flake.{nix,lock} or their parents being symlinks that
escape from the flake.
Extracted from the lazy-trees branch.
"hash type" -> "hash algorithm" in all comments, documentation, and
messages.
ht -> ha, [Hh]ashType -> [HhashAlgo] for all local variables and
function arguments. No API change is made.
Continuation of 5334c9c792 and 837b889c41.
After commit 91b6833686 (" Move tests to separate directories, and
document"), previously-built test executables are now tracked by Git,
which is annoying for developers.
This patch add .gitignore rules to ignore the obsolete test directories
to solve such problem and enhance developer experience.
The sandbox rule `(allow network* (local ip))` doesn't do what it
implies. Adding this rule permits all network traffic. We should be
matching on (remote ip "localhost:*")` instead.
This seems to have found one actual bug in fs-sink.cc: the symlink case
was falling into the regular file case, which can't possibly be
intentional, right?
Docker uses "latest" as the default label instead of "master".
This change will allow to docker run ghcr.io/nixos/nix without having to
specify the label.
It keeps the :master label on docker hub for back-compat.
When a file conflict arises during a package install a suggestion is
made to remove the old entry. This was previously done using the
installable URLs of the old entry. These URLs are quite verbose and
often do not equal the URL of the existing entry.
This change uses the recently introduced profile entry name for the
suggestion, resulting in a simpler output.
The improvement is easily seen in the change to the functional test.
- `nix store add` supports text hashing
With functional test ensuring it matches `builtins.toFile`.
- Factored-out flags for both commands
- Move all common reusable flags to `libcmd`
- They are not part of the *definition* of the CLI infra, just a usag
of it.
- The `libstore` flag couldn't go in `args.hh` in libutil anyways,
would be awkward for it to live alone
- Shuffle around `Cmd*` hierarchy so flags for deprecated commands don't
end up on the new ones
This PR reduces the creation of short-lived basic_json objects while
parsing flake.lock files. For large flake.lock files (~1.5MB) I was
observing ~60s being spent for trivial nix build operations while
after this change it is now taking ~1.6s.
It's better to just check whether the input has all the attributes
needed to consider itself locked (e.g. whether a Git input has an
'rev' attribute).
Also, the 'locked' field was actually incorrect for Git inputs: it
would be set to true even for dirty worktrees. As a result, we got
away with using fetchTree() internally even though fetchTree()
requires a locked input in pure mode. In particular, this allowed
'--override-input' to work by accident.
The fix is to pass a set of "overrides" to call-flake.nix for all the
unlocked inputs (i.e. the top-level flake and any --override-inputs).
This fixes warnings like
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
when using the daemon and the user didn't actually set those settings.
Note: this also hides those settings from `nix config show`, but that
seems a good thing.
`canonPath` and `absPath` work on native paths, and so should switch
between supporting Unix paths and Windows paths accordingly.
The templating is because `CanonPath`, which shares the implementation,
should always be Unix style. It is the pure "nix-native" path type for
virtual file operations --- it is part of Nix's "business logic", and
should not vary with the host OS accordingly.
for plain inherits this is really just a stylistic choice, but for
inherit-from it actually fixes an exponential size increase problem
during expr printing (as may happen during assertion failure reporting,
on during duplicate attr detection in the parser)
this also has the effect of sorting let bindings lexicographically
rather than by symbol creation order as was previously done, giving a
better canonicalization in the process.
Previously, the "file:./" prefix was not correctly recognized in
fixGitURL; instead, it was mistaken as a file path, which resulted in a
parsed url of the form "file://file:./".
This commit fixes the issue by properly detecting the "file:" prefix.
Note, however, that unlike "file://", the "file:./" URI is _not_
standardized, but has been widely used to referred to relative file
paths. In particular, the "git+file:./" did work for nix<=2.18, and was
broken since nix 2.19.0.
Finally, this commit fixes the issue completely for the 2.19 series, but
is still inadequate for the 2.20 series due to new behaviors from the
switch to libgit2. However, it does improve the correctness of parsing
even though it is not yet a complete solution.
These docs explain the implementation relative to the local store
originals. The original declaration of virtual methods can still be
consulted for proper interface-level documentation.
As an optimisation for LocalStore, we read all the store directory entries into
a set. Checking for membership of this set is much faster than a stat syscall.
However for LocalOverlayStore, the lower store directory is expected to contain
a vast number of entries and reading them all can take a very long time.
So instead of enumerating them all upfront, we call pathExists as needed. This
means making stat syscalls for each store path, but the upper layer is expected
to be relatively small compared to the lower store so that should be okay.
Grouping our tests should make it easier to understand the intent than
one long poorly-arranged list. It also is convenient for running just
the tests for a specific component when working on that component.
We need at least one test group so this isn't dead code; I decided to
collect the tests for the `ca-derivations` and `dynamic-derivations`
experimental features in groups. Do
```bash
make ca.test-group -jN
```
and
```bash
make dyn-drv.test-group -jN
```
to try running just them.
I originally did this as part of #8397 for being able to just the local
overlay store alone. I am PRing it separately now so we can separate
general infra from new features.
This requires switching on SQLITE_OPEN_URI because there is no open flag to
make the database immutable. Without immutable, sqlite will still attempt to
create journal and wal files, even when the database is opened read-only.
https://www.sqlite.org/c3ref/open.html
The immutable parameter is a boolean query parameter that indicates that the
database file is stored on read-only media. When immutable is set, SQLite
assumes that the database file cannot be changed, even by a process with higher
privilege, and so the database is opened read-only and all locking and change
detection is disabled.
Nix does not manage the overlayfs mount point itself, but the correct
functioning of the overlay store does depend on this mount point being set up
correctly. Rather than just assume this is the case, check that the lowerdir
and upperdir options are what we expect them to be. This check is on by
default, but can be disabled if needed.
@@ -27,6 +27,8 @@ Check out the [security policy](https://github.com/NixOS/nix/security/policy).
1. Search for related issues that cover what you're going to work on.
It could help to mention there that you will work on the issue.
We strongly recommend first-time contributors not to propose new features but rather fix tightly-scoped problems in order to build trust and a working relationship with maintainers.
Issues labeled [good first issue](https://github.com/NixOS/nix/labels/good%20first%20issue) should be relatively easy to fix and are likely to get merged quickly.
Pull requests addressing issues labeled [idea approved](https://github.com/NixOS/nix/labels/idea%20approved) or [RFC](https://github.com/NixOS/nix/labels/RFC) are especially welcomed by maintainers and will receive prioritised review.
@@ -67,7 +69,7 @@ Check out the [security policy](https://github.com/NixOS/nix/security/policy).
- [ ] API documentation in header files
- [ ] Code and comments are self-explanatory
- [ ] Commit message explains **why** the change was made
- [ ] New feature or incompatible change: updated [release notes](./doc/manual/src/release-notes/rl-next.md)
- [ ] New feature or incompatible change: [add a release note](https://nixos.org/manual/nix/stable/contributing/hacking#add-a-release-note)
7. If you need additional feedback or help to getting pull request into shape, ask other contributors using [@mentions](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#mentioning-people-and-teams).
[test "$ENABLE_BUILD" == "no" && test "$ENABLE_UNIT_TESTS" == "yes"],
[AC_MSG_ERROR([Cannot enable unit tests when building overall is disabled. Please do not pass '--enable-unit-tests' or do not pass '--disable-build'.])])
The build directory has been hardened against interference with the outside world by nesting it inside another directory owned by (and only readable by) the daemon user.
synopsis: Store paths are allowed to start with `.`
issues: 912
prs: 9867 9091 9095 9120 9121 9122 9130 9219 9224
---
Leading periods were allowed by accident in Nix 2.4. The Nix team has considered this to be a bug, but this behavior has since been relied on by users, leading to unnecessary difficulties.
From now on, leading periods are officially, definitively supported. The names `.` and `..` are disallowed, as well as those starting with `.-` or `..-`.
Nix versions that denied leading periods are documented [in the issue](https://github.com/NixOS/nix/issues/912#issuecomment-1919583286).
`nix-copy-closure` - copy a closure to or from a remote machine via SSH
`nix-copy-closure` - copy store objects to or from a remote machine via SSH
# Synopsis
`nix-copy-closure`
[`--to` | `--from`]
[`--to` | `--from`]
[`--gzip`]
[`--include-outputs`]
[`--use-substitutes` | `-s`]
[`-v`]
_user@machine__paths_
[_user_@]_machine_[:_port_]_paths_
# Description
`nix-copy-closure` gives you an easy and efficient way to exchange
software between machines. Given one or more Nix store _paths_ on the
local machine, `nix-copy-closure` computes the closure of those paths
(i.e. all their dependencies in the Nix store), and copies all paths
in the closure to the remote machine via the `ssh` (Secure Shell)
command. With the `--from` option, the direction is reversed: the
closure of _paths_ on a remote machine is copied to the Nix store on
the local machine.
Given _paths_ from one machine, `nix-copy-closure` computes the [closure](@docroot@/glossary.md#gloss-closure) of those paths (i.e. all their dependencies in the Nix store), and copies [store objects](@docroot@/glossary.md#gloss-store-object) in that closure to another machine via SSH.
It doesn’t copy store objects that are already present on the other machine.
This command is efficient because it only sends the store paths
that are missing on the target machine.
> **Note**
>
> While the Nix store to use on the local machine can be specified on the command line with the [`--store`](@docroot@/command-ref/conf-file.md#conf-store) option, the Nix store to be accessed on the remote machine can only be [configured statically](@docroot@/command-ref/conf-file.md#configuration-file) on that remote machine.
Since `nix-copy-closure` calls `ssh`, you may be asked to type in the
appropriate password or passphrase. In fact, you may be asked _twice_
because `nix-copy-closure` currently connects twice to the remote
machine, first to get the set of paths missing on the target machine,
and second to send the dump of those paths. When using public key
authentication, you can avoid typing the passphrase with `ssh-agent`.
Since `nix-copy-closure` calls `ssh`, you may need to authenticate with the remote machine.
In fact, you may be asked for authentication _twice_ because `nix-copy-closure` currently connects twice to the remote machine: first to get the set of paths missing on the target machine, and second to send the dump of those paths.
When using public key authentication, you can avoid typing the passphrase with `ssh-agent`.
# Options
-`--to`\
Copy the closure of _paths_ from the local Nix store to the Nix
store on _machine_. This is the default.
-`--to`
-`--from`\
Copy the closure of _paths_ from the Nix store on _machine_ to the
local Nix store.
Copy the closure of _paths_ from a Nix store accessible from the local machine to the Nix store on the remote _machine_.
This is the default behavior.
-`--from`
Copy the closure of _paths_ from the Nix store on the remote _machine_ to the local machine's specified Nix store.
-`--gzip`
-`--gzip`\
Enable compression of the SSH connection.
-`--include-outputs`\
-`--include-outputs`
Also copy the outputs of [store derivation]s included in the closure.
Attempt to download missing paths on the target machine using Nix’s
substitute mechanism. Any paths that cannot be substituted on the
target are still copied normally from the source. This is useful,
for instance, if the connection between the source and target
machine is slow, but the connection between the target machine and
`nixos.org` (the default binary cache server) is
fast.
-`--use-substitutes` / `-s`
-`-v`\
Show verbose output.
Attempt to download missing store objects on the target from [substituters](@docroot@/command-ref/conf-file.md#conf-substituters).
Any store objects that cannot be substituted on the target are still copied normally from the source.
This is useful, for instance, if the connection between the source and target machine is slow, but the connection between the target machine and `cache.nixos.org` (the default binary cache server) is fast.
{{#include ./opt-common.md}}
# Environment variables
-`NIX_SSHOPTS`\
Additional options to be passed to `ssh` on the command
line.
-`NIX_SSHOPTS`
Additional options to be passed to `ssh` on the command line.
{{#include ./env-common.md}}
# Examples
Copy Firefox with all its dependencies to a remote machine:
> **Example**
>
> Copy GNU Hello with all its dependencies to a remote machine:
>
> ```shell-session
> $ storePath="$(nix-build '<nixpkgs>' -I nixpkgs=channel:nixpkgs-unstable -A hello --no-out-link)"
@@ -47,39 +47,83 @@ These pages can be viewed offline:
Example: `nix-env --help --install`
# Package sources
`nix-env` can obtain packages from multiple sources:
- An attribute set of derivations from:
- The [default Nix expression](@docroot@/command-ref/files/default-nix-expression.md) (by default)
- A Nix file, specified via `--file`
- A [profile](@docroot@/command-ref/files/profiles.md), specified via `--from-profile`
- A Nix expression that is a function which takes default expression as argument, specified via `--from-expression`
- A [store path](@docroot@/store/store-path.md)
# Selectors
Several commands, such as `nix-env --query` and `nix-env --install`, take a list of
arguments that specify the packages on which to operate. These are
extended regular expressions that must match the entire name of the
package. (For details on regular expressions, see **regex**(7).) The match is
case-sensitive. The regular expression can optionally be followed by a
dash and a version number; if omitted, any version of the package will
match. Here are some examples:
Several operations, such as [`nix-env --query`](./nix-env/query.md) and [`nix-env --install`](./nix-env/install.md), take a list of *arguments* that specify the packages on which to operate.
-`firefox`\
Matches the package name `firefox` and any version.
Packages are identified based on a `name` part and a `version` part of a [symbolic derivation name](@docroot@/language/derivations.md#attr-names):
-`firefox-32.0`\
Matches the package name `firefox` and version `32.0`.
-`name`: Everything up to but not including the first dash (`-`) that is *not* followed by a letter.
-`version`: The rest, excluding the separating dash.
-`gtk\\+`\
Matches the package name `gtk+`. The `+` character must be escaped
using a backslash to prevent it from being interpreted as a
quantifier, and the backslash must be escaped in turn with another
backslash to ensure that the shell passes it on.
> **Example**
>
> `nix-env` parses the symbolic derivation name `apache-httpd-2.0.48` as:
>
> ```json
> {
> "name": "apache-httpd",
> "version": "2.0.48"
> }
> ```
-`.\*`\
Matches any package name. This is the default for most commands.
> **Example**
>
> `nix-env` parses the symbolic derivation name `firefox.*` as:
>
> ```json
> {
> "name": "firefox.*",
> "version": ""
> }
> ```
-`'.*zip.*'`\
Matches any package name containing the string `zip`. Note the dots:
`'*zip*'` does not work, because in a regular expression, the
character `*` is interpreted as a quantifier.
The `name` parts of the *arguments* to `nix-env` are treated as extended regular expressions and matched against the `name` parts of derivation names in the package source.
The match is case-sensitive.
The regular expression can optionally be followed by a dash (`-`) and a version number; if omitted, any version of the package will match.
For details on regular expressions, see [**regex**(7)](https://linux.die.net/man/7/regex).
-`'.*(firefox|chromium).*'`\
Matches any package name containing the strings `firefox` or
`chromium`.
> **Example**
>
> Common patterns for finding package names with `nix-env`:
>
> - `firefox`
>
> Matches the package name `firefox` and any version.
>
> - `firefox-32.0`
>
> Matches the package name `firefox` and version `32.0`.
>
> - `gtk\\+`
>
> Matches the package name `gtk+`.
> The `+` character must be escaped using a backslash (`\`) to prevent it from being interpreted as a quantifier, and the backslash must be escaped in turn with another backslash to ensure that the shell passes it on.
>
> - `.\*`
>
> Matches any package name.
> This is the default for most commands.
>
> - `'.*zip.*'`
>
> Matches any package name containing the string `zip`.
> Note the dots: `'*zip*'` does not work, because in a regular expression, the character `*` is interpreted as a quantifier.
>
> - `'.*(firefox|chromium).*'`
>
> Matches any package name containing the strings `firefox` or `chromium`.
The install operation creates a new user environment.
The `--install` operation creates a new user environment.
It is based on the current generation of the active [profile](@docroot@/command-ref/files/profiles.md), to which a set of [store paths] described by *args* is added.
The arguments *args* map to store paths in a number of possible ways:
- By default, *args* is a set of [derivation] names denoting derivations in the [default Nix expression].
These are [realised], and the resulting output paths are installed.
Currently installed derivations with a name equal to the name of a derivation being added are removed unless the option `--preserve-installed` is specified.
@@ -50,7 +49,7 @@ The arguments *args* map to store paths in a number of possible ways:
Show the attribute paths of available packages with [`nix-env --query`](./query.md):
```console
nix-env --query --available --attr-path`
nix-env --query --available --attr-path
```
- If `--from-profile` *path* is given, *args* is a set of names
`nix-store --export` - export store paths to a Nix Archive
`nix-store --export` - export store paths to a [Nix Archive]
## Synopsis
@@ -8,16 +8,22 @@
## Description
The operation `--export` writes a serialisation of the specified store
paths to standard output in a format that can be imported into another
Nix store with `nix-store --import`. This is like `nix-store
--dump`, except that the NAR archive produced by that command doesn’t
contain the necessary meta-information to allow it to be imported into
another Nix store (namely, the set of references of the path).
The operation `--export` writes a serialisation of the given [store objects](@docroot@/glossary.md#gloss-store-object) to standard output in a format that can be imported into another [Nix store](@docroot@/store/index.md) with [`nix-store --import`](./import.md).
This command does not produce a *closure* of the specified paths, so if
a store path references other store paths that are missing in the target
Nix store, the import will fail.
> **Warning**
>
> This command *does not* produce a [closure](@docroot@/glossary.md#gloss-closure) of the specified store paths.
> Trying to import a store object that refers to store paths not available in the target Nix store will fail.
>
> Use [`nix-store --query`](@docroot@/command-ref/nix-store/query.md) to obtain the closure of a store path.
This command is different from [`nix-store --dump`](./dump.md), which produces a [Nix archive](@docroot@/glossary.md#gloss-nar) that *does not* contain the set of [references](@docroot@/glossary.md#gloss-reference) of a given store path.
> **Note**
>
> For efficient transfer of closures to remote machines over SSH, use [`nix-copy-closure`](@docroot@/command-ref/nix-copy-closure.md).
The operation `--import` reads a serialisation of a set of store paths
produced by `nix-store --export` from standard input and adds those
store paths to the Nix store. Paths that already exist in the Nix store
are ignored. If a path refers to another path that doesn’t exist in the
Nix store, the import fails.
The operation `--import` reads a serialisation of a set of [store objects](@docroot@/glossary.md#gloss-store-object) produced by [`nix-store --export`](./export.md) from standard input, and adds those store objects to the specified [Nix store](@docroot@/store/index.md).
Paths that already exist in the target Nix store are ignored.
If a path [refers](@docroot@/glossary.md#gloss-reference) to another path that doesn’t exist in the target Nix store, the import fails.
> **Note**
>
> For efficient transfer of closures to remote machines over SSH, use [`nix-copy-closure`](@docroot@/command-ref/nix-copy-closure.md).
{{#include ./opt-common.md}}
{{#include ../opt-common.md}}
{{#include ../env-common.md}}
# Examples
> **Example**
>
> Given a closure of GNU Hello as a file:
>
> ```shell-session
> $ storePath="$(nix-build '<nixpkgs>' -I nixpkgs=channel:nixpkgs-unstable -A hello --no-out-link)"
> Import the closure into a [remote SSH store](@docroot@/store/types/ssh-store.md) using the [`--store`](@docroot@/command-ref/conf-file.md#conf-store) option:
@@ -389,88 +389,6 @@ colors, no emojis and using ASCII instead of Unicode symbols). The same should
happen when TTY is not detected on STDERR. We should not display progress /
status section, but only print warnings and errors.
## Returning future proof JSON
The schema of JSON output should allow for backwards compatible extension. This section explains how to achieve this.
Two definitions are helpful here, because while JSON only defines one "key-value"
object type, we use it to cover two use cases:
- **dictionary**: a map from names to value that all have the same type. In
C++ this would be a `std::map` with string keys.
- **record**: a fixed set of attributes each with their own type. In C++, this
would be represented by a `struct`.
It is best not to mix these use cases, as that may lead to incompatibilities when the schema changes. For example, adding a record field to a dictionary breaks consumers that assume all JSON object fields to have the same meaning and type.
This leads to the following guidelines:
- The top-level (root) value must be a record.
Otherwise, one can not change the structure of a command's output.
- The value of a dictionary item must be a record.
Otherwise, the item type can not be extended.
- List items should be records.
Otherwise, one can not change the structure of the list items.
If the order of the items does not matter, and each item has a unique key that is a string, consider representing the list as a dictionary instead. If the order of the items needs to be preserved, return a list of records.
- Streaming JSON should return records.
An example of a streaming JSON format is [JSON lines](https://jsonlines.org/), where each line represents a JSON value. These JSON values can be considered top-level values or list items, and they must be records.
### Examples
This is bad, because all keys must be assumed to be store types:
```json
{
"local":{...},
"remote":{...},
"http":{...}
}
```
This is good, because the it is extensible at the root, and is somewhat self-documenting:
```json
{
"storeTypes":{"local":{...},...},
"pluginSupport":true
}
```
While the dictionary of store types seems like a very complete response at first, a use case may arise that warrants returning additional information.
For example, the presence of plugin support may be crucial information for a client to proceed when their desired store type is missing.
The following representation is bad because it is not extensible:
```json
{"outputs":["out""bin"]}
```
However, simply converting everything to records is not enough, because the order of outputs must be preserved:
```json
{"outputs":{"bin":{},"out":{}}}
```
The first item is the default output. Deriving this information from the outputs ordering is not great, but this is how Nix currently happens to work.
While it is possible for a JSON parser to preserve the order of fields, we can not rely on this capability to be present in all JSON libraries.
This representation is extensible and preserves the ordering:
@@ -144,6 +144,7 @@ Nix can be built for various platforms, as specified in [`flake.nix`]:
-`aarch64-darwin`
-`armv6l-linux`
-`armv7l-linux`
-`riscv64-linux`
In order to build Nix for a different platform than the one you're currently
on, you need a way for your current Nix installation to build code for that
@@ -166,7 +167,10 @@ or for Nix with the [`flakes`] and [`nix-command`] experimental features enabled
$ nix build .#packages.aarch64-linux.default
```
Cross-compiled builds are available for ARMv6 (`armv6l-linux`) and ARMv7 (`armv7l-linux`).
Cross-compiled builds are available for:
-`armv6l-linux`
-`armv7l-linux`
-`riscv64-linux`
Add more [system types](#system-type) to `crossSystems` in `flake.nix` to bootstrap Nix on unsupported platforms.
### Building for multiple platforms at once
@@ -196,7 +200,7 @@ In order to facilitate this, Nix has some support for being built out of tree
## System type
Nix uses a string with he following format to identify the *system type* or *platform* it runs on:
Nix uses a string with the following format to identify the *system type* or *platform* it runs on:
```
<cpu>-<os>[-<abi>]
@@ -258,10 +262,10 @@ See [supported compilation environments](#compilation-environments) and instruct
To use the LSP with your editor, you first need to [set up `clangd`](https://clangd.llvm.org/installation#project-setup) by running:
```console
make clean && bear -- make -j$NIX_BUILD_CORES default check install
make compile_commands.json
```
Configure your editor to use the `clangd` from the shell, either by running it inside the development shell, or by using [nix-direnv](https://github.com/nix-community/nix-direnv) and [the appropriate editor plugin](https://github.com/direnv/direnv/wiki#editor-integration).
Configure your editor to use the `clangd` from the `.#native-clangStdenvPackages` shell. You can do that either by running it inside the development shell, or by using [nix-direnv](https://github.com/nix-community/nix-direnv) and [the appropriate editor plugin](https://github.com/direnv/direnv/wiki#editor-integration).
> **Note**
>
@@ -269,6 +273,29 @@ Configure your editor to use the `clangd` from the shell, either by running it i
> Some other editors (e.g. Emacs, Vim) need a plugin to support LSP servers in general (e.g. [lsp-mode](https://github.com/emacs-lsp/lsp-mode) for Emacs and [vim-lsp](https://github.com/prabirshrestha/vim-lsp) for vim).
> Editor-specific setup is typically opinionated, so we will not cover it here in more detail.
## Formatting and pre-commit hooks
You may run the formatters as a one-off using:
```console
make format
```
If you'd like to run the formatters before every commit, install the hooks:
```
pre-commit-hooks-install
```
This installs [pre-commit](https://pre-commit.com) using [cachix/git-hooks.nix](https://github.com/cachix/git-hooks.nix).
When making a commit, pay attention to the console output.
If it fails, run `git add --patch` to approve the suggestions _and commit again_.
To refresh pre-commit hook's config file, do the following:
1. Exit the development shell and start it again by running `nix develop`.
2. If you also use the pre-commit hook, also run `pre-commit-hooks-install` again.
## Add a release note
`doc/manual/rl-next` contains release notes entries for all unreleased changes.
Nix consumes and produces JSON in a variety of contexts.
These guidelines ensure consistent practices for all our JSON interfaces, for ease of use, and so that experience in one part carries over to another.
## Extensibility
The schema of JSON input and output should allow for backwards compatible extension.
This section explains how to achieve this.
Two definitions are helpful here, because while JSON only defines one "key-value" object type, we use it to cover two use cases:
- **dictionary**: a map from names to value that all have the same type.
In C++ this would be a `std::map` with string keys.
- **record**: a fixed set of attributes each with their own type.
In C++, this would be represented by a `struct`.
It is best not to mix these use cases, as that may lead to incompatibilities when the schema changes.
For example, adding a record field to a dictionary breaks consumers that assume all JSON object fields to have the same meaning and type, and dictionary items with a colliding name can not be represented anymore.
This leads to the following guidelines:
- The top-level (root) value must be a record.
Otherwise, one can not change the structure of a command's output.
- The value of a dictionary item must be a record.
Otherwise, the item type can not be extended.
- List items should be records.
Otherwise, one can not change the structure of the list items.
If the order of the items does not matter, and each item has a unique key that is a string, consider representing the list as a dictionary instead.
If the order of the items needs to be preserved, return a list of records.
- Streaming JSON should return records.
An example of a streaming JSON format is [JSON lines](https://jsonlines.org/), where each line represents a JSON value.
These JSON values can be considered top-level values or list items, and they must be records.
### Examples
This is bad, because all keys must be assumed to be store types:
```json
{
"local":{...},
"remote":{...},
"http":{...}
}
```
This is good, because the it is extensible at the root, and is somewhat self-documenting:
```json
{
"storeTypes":{"local":{...},...},
"pluginSupport":true
}
```
While the dictionary of store types seems like a very complete response at first, a use case may arise that warrants returning additional information.
For example, the presence of plugin support may be crucial information for a client to proceed when their desired store type is missing.
The following representation is bad because it is not extensible:
```json
{"outputs":["out""bin"]}
```
However, simply converting everything to records is not enough, because the order of outputs must be preserved:
```json
{"outputs":{"bin":{},"out":{}}}
```
The first item is the default output. Deriving this information from the outputs ordering is not great, but this is how Nix currently happens to work.
While it is possible for a JSON parser to preserve the order of fields, we can not rely on this capability to be present in all JSON libraries.
This representation is extensible and preserves the ordering:
@@ -60,7 +60,7 @@ The unit tests are defined using the [googletest] and [rapidcheck] frameworks.
> ```
The tests for each Nix library (`libnixexpr`, `libnixstore`, etc..) live inside a directory `tests/unit/${library_name_without-nix}`.
Given a interface (header) and implementation pair in the original library, say, `src/libexpr/value/context.{hh,cc}`, we write tests for it in `tests/unit/libexpr/tests/value/context.cc`, and (possibly) declare/define additional interfaces for testing purposes in `tests/unit/libexpr-support/tests/value/context.{hh,cc}`.
Given an interface (header) and implementation pair in the original library, say, `src/libexpr/value/context.{hh,cc}`, we write tests for it in `tests/unit/libexpr/tests/value/context.cc`, and (possibly) declare/define additional interfaces for testing purposes in `tests/unit/libexpr-support/tests/value/context.{hh,cc}`.
Data for unit tests is stored in a `data` subdir of the directory for each unit test executable.
For example, `libnixstore` code is in `src/libstore`, and its test data is in `tests/unit/libstore/data`.
@@ -162,14 +162,14 @@ ran test tests/functional/${testName}.sh... [PASS]
Software Heritage's writing on [*Intrinsic and Extrinsic identifiers*](https://www.softwareheritage.org/2020/07/09/intrinsic-vs-extrinsic-identifiers) is also a good introduction to the value of content-addressing over other referencing schemes.
Besides content addressing, the Nix store also uses [input addressing](#gloss-input-addressed-store-object).
- [derivation]{#gloss-derivation}
A description of a build task. The result of a derivation is a
@@ -86,7 +105,7 @@
[store path]: #gloss-store-path
- [file system object]{#gloss-store-object}
- [file system object]{#gloss-file-system-object}
The Nix data model for representing simplified file system data.
@@ -118,9 +137,12 @@
- [content-addressed store object]{#gloss-content-addressed-store-object}
A [store object] whose [store path] is determined by its contents.
A [store object] which is [content-addressed](#gloss-content-address),
i.e. whose [store path] is determined by its contents.
This includes derivations, the outputs of [content-addressed derivations](#gloss-content-addressed-derivation), and the outputs of [fixed-output derivations](#gloss-fixed-output-derivation).
See [Content-Addressing Store Objects](@docroot@/store/store-object/content-address.md) for details.
- [substitute]{#gloss-substitute}
A substitute is a command invocation stored in the [Nix database] that
@@ -215,6 +237,20 @@
[output path]: #gloss-output-path
- [output closure]{#gloss-output-closure}\
The [closure] of an [output path]. It only contains what is [reachable] from the output.
- [deriving path]{#gloss-deriving-path}
Deriving paths are a way to refer to [store objects][store object] that ar not yet [realised][realise].
This is necessary because, in general and particularly for [content-addressed derivations][content-addressed derivation], the [output path] of an [output] is not known in advance.
There are two forms:
- *constant*: just a [store path]
It can be made [valid][validity] by copying it into the store: from the evaluator, command line interface or another store.
- *output*: a pair of a [store path] to a [derivation] and an [output] name.
- [deriver]{#gloss-deriver}
The [store derivation] that produced an [output path].
@@ -252,13 +288,15 @@
See [installables](./command-ref/new-cli/nix.md#installables) for [`nix` commands](./command-ref/new-cli/nix.md) (experimental) for details.
- [NAR]{#gloss-nar}
- [Nix Archive (NAR)]{#gloss-nar}
A *N*ix *AR*chive. This is a serialisation of a path in the Nix
store. It can contain regular files, directories and symbolic
links. NARs are generated and unpacked using `nix-store --dump`
and `nix-store --restore`.
See [Nix Archive](store/file-system-object/content-address.html#serial-nix-archive) for details.
- [`∅`]{#gloss-emtpy-set}
The empty set symbol. In the context of profile history, this denotes a package is not present in a particular version of the profile.
The location from which relative paths are resolved.
- For expressions in a file, the base directory is the directory containing that file.
This is analogous to the directory of a [base URL](https://datatracker.ietf.org/doc/html/rfc1808#section-3.3).
<!-- which is sufficient for resolving non-empty URLs -->
<!--
The wording here may look awkward, but it's for these reasons:
* "with --expr": it's a flag, and not an option with an accompanying value
* "written in": the expression itself must be written as an argument,
whereas the more natural "passed as an argument" allows an interpretation
where the expression could be passed by file name.
-->
- For expressions written in command line arguments with [`--expr`](@docroot@/command-ref/opt-common.html#opt-expr), the base directory is the current working directory.
@@ -188,38 +188,49 @@ Derivations can declare some infrequently used optional attributes.
}
```
The `outputHashAlgo` attribute specifies the hash algorithm used to
compute the hash. It can currently be `"sha1"`, `"sha256"` or
`"sha512"`.
The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The "nix32" encoding is an adaptation of base-32 encoding.
The [`convertHash`](@docroot@/language/builtins.md#builtins-convertHash) function shows how to convert between different encodings, and the [`nix-hash` command](../command-ref/nix-hash.md) has information about obtaining the hash for some contents, as well as converting to and from encodings.
The `outputHashAlgo` attribute specifies the hash algorithm used to compute the hash.
It can currently be `"sha1"`, `"sha256"`, `"sha512"`, or `null`.
`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format.
The `outputHashMode` attribute determines how the hash is computed.
It must be one of the following two values:
It must be one of the following values:
- `"flat"`\
The output must be a non-executable regular file. If it isn’t,
the build fails. The hash is simply computed over the contents
of that file (so it’s equal to what Unix commands like
@@ -141,7 +141,7 @@ It outputs an attribute set, and produces a [store derivation] as a side effect
By default, a derivation produces a single output called `out`.
However, derivations can produce multiple outputs.
This allows the associated [store objects](@docroot@/glossary.md#gloss-store-object) and their [closures](@docroot@/glossary.md#gloss-closure) to be copied or garbage-collected separately.
This allows the associated [store objects](@docroot@/store/store-object.md) and their [closures](@docroot@/glossary.md#gloss-closure) to be copied or garbage-collected separately.
Passing an expression `expr` that evaluates to a [store path](@docroot@/glossary.md#gloss-store-path) to any built-in function which reads from the filesystem constitutes Import From Derivation (IFD):
Passing an expression `expr` that evaluates to a [store path](@docroot@/store/store-path.md) to any built-in function which reads from the filesystem constitutes Import From Derivation (IFD):
The Nix language is designed for conveniently creating and composing *derivations*– precise descriptions of how contents of existing files are used to derive new files.
It is:
> **Tip**
>
> These pages are written as a reference.
> If you are learning Nix, nix.dev has a good [introduction to the Nix language](https://nix.dev/tutorials/nix-language).
The language is:
- *domain-specific*
@@ -47,7 +53,7 @@ This is an incomplete overview of language features, by example.
@@ -124,7 +142,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
Null value
[Null](@docroot@/language/values.md#type-null) value
</td>
</tr>
@@ -136,7 +154,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
An integer
An [integer](@docroot@/language/values.md#type-number)
</td>
</tr>
@@ -148,7 +166,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A floating point number
A [floating point number](@docroot@/language/values.md#type-number)
</td>
</tr>
@@ -160,7 +178,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
An absolute path
An absolute [path](@docroot@/language/values.md#type-path)
</td>
</tr>
@@ -172,7 +190,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A path relative to the file containing this Nix expression
A [path](@docroot@/language/values.md#type-path) relative to the file containing this Nix expression
</td>
</tr>
@@ -184,7 +202,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A home path. Evaluates to the `"<user's home directory>/.config"`.
A home [path](@docroot@/language/values.md#type-path). Evaluates to the `"<user's home directory>/.config"`.
</td>
</tr>
@@ -196,7 +214,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
Search path for Nix files. Value determined by [`$NIX_PATH` environment variable](../command-ref/env-common.md#env-NIX_PATH).
A [lookup path](@docroot@/language/constructs/lookup-path.md) for Nix files. Value determined by [`$NIX_PATH` environment variable](../command-ref/env-common.md#env-NIX_PATH).
</td>
</tr>
@@ -220,7 +238,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A set with attributes named `x` and `y`
An [attribute set](@docroot@/language/values.md#attribute-set) with attributes named `x` and `y`
</td>
</tr>
@@ -244,7 +262,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A recursive set, equivalent to `{ x = "foo"; y = "foobar"; }`
A [recursive set](@docroot@/language/constructs.md#recursive-sets), equivalent to `{ x = "foo"; y = "foobar"; }`.
</td>
</tr>
@@ -260,7 +278,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
Lists with three elements.
[Lists](@docroot@/language/values.md#list) with three elements.
</td>
</tr>
@@ -344,7 +362,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
Attribute selection (evaluates to `1`)
[Attribute selection](@docroot@/language/values.md#attribute-set) (evaluates to `1`)
</td>
</tr>
@@ -356,7 +374,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
Attribute selection with default (evaluates to `3`)
[Attribute selection](@docroot@/language/values.md#attribute-set) with default (evaluates to `3`)
</td>
</tr>
@@ -392,7 +410,7 @@ This is an incomplete overview of language features, by example.
@@ -452,7 +500,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A function that expects an integer and returns it increased by 1
A [function](@docroot@/language/constructs.md#functions) that expects an integer and returns it increased by 1.
</td>
</tr>
@@ -464,7 +512,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
Curried function, equivalent to `x: (y: x + y)`. Can be used like a function that takes two arguments and returns their sum.
Curried [function](@docroot@/language/constructs.md#functions), equivalent to `x: (y: x + y)`. Can be used like a function that takes two arguments and returns their sum.
</td>
</tr>
@@ -476,7 +524,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A function call (evaluates to 101)
A [function](@docroot@/language/constructs.md#functions) call (evaluates to 101)
</td>
</tr>
@@ -488,7 +536,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A function bound to a variable and subsequently called by name (evaluates to 103)
A [function](@docroot@/language/constructs.md#functions) bound to a variable and subsequently called by name (evaluates to 103)
</td>
</tr>
@@ -500,7 +548,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A function that expects a set with required attributes `x` and `y` and concatenates them
A [function](@docroot@/language/constructs.md#functions) that expects a set with required attributes `x` and `y` and concatenates them
</td>
</tr>
@@ -512,7 +560,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A function that expects a set with required attribute `x` and optional `y`, using `"bar"` as default value for `y`
A [function](@docroot@/language/constructs.md#functions) that expects a set with required attribute `x` and optional `y`, using `"bar"` as default value for `y`
</td>
</tr>
@@ -524,7 +572,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A function that expects a set with required attributes `x` and `y` and ignores any other attributes
A [function](@docroot@/language/constructs.md#functions) that expects a set with required attributes `x` and `y` and ignores any other attributes
</td>
</tr>
@@ -538,7 +586,7 @@ This is an incomplete overview of language features, by example.
</td>
<td>
A function that expects a set with required attributes `x` and `y`, and binds the whole set to `args`
A [function](@docroot@/language/constructs.md#functions) that expects a set with required attributes `x` and `y`, and binds the whole set to `args`
</td>
</tr>
@@ -562,7 +610,8 @@ This is an incomplete overview of language features, by example.
</td>
<td>
Load and return Nix expression in given file
Load and return Nix expression in given file.
See [import](@docroot@/language/builtins.md#builtins-import).
</td>
</tr>
@@ -574,7 +623,8 @@ This is an incomplete overview of language features, by example.
</td>
<td>
Apply a function to every element of a list (evaluates to `[ 2 4 6 ]`)
Apply a function to every element of a list (evaluates to `[ 2 4 6 ]`).
See [`map`](@docroot@/language/builtins.md#builtins-map).
The idea is that a user can combine together values to create a build instructions for derivations without manually keeping track of where they come from.
Then the Nix language implicitly does that bookkeeping to efficiently obtain the closure of derivation inputs.
> **Note**
>
> String contexts are *not* explicitly manipulated in idiomatic Nix language code.
[`buitins.unsafeDiscardStringContext`](./builtins.md#builtins-unsafeDiscardStringContext) will make a copy of a string, but with an empty string context.
The returned string can be used in more ways, e.g. by operators that require the string context to be empty.
The requirement to explicitly discard the string context in such use cases helps ensure that string context elements are not lost by mistake.
The "unsafe" marker is only there to remind that Nix normally guarantees that dependencies are tracked, whereas the returned string has lost them.
## Constructing string contexts
[`builtins.appendContext`] will create a copy of a string, but with additional string context elements.
The context is specified explicitly by an [attribute set] in the format that [`builtins.hasContext`] produces.
A string with arbitrary contexts can be made like this:
1. Create a string with the desired string context elements.
(The contents of the string do not matter.)
2. Dump its context with [`builtins.getContext`].
3. Combine it with a base string and repeated [`builtins.appendContext`] calls.
@@ -107,9 +107,9 @@ An expression that is interpolated must evaluate to one of the following:
A string interpolates to itself.
A path in an interpolated expression is first copied into the Nix store, and the resulting string is the [store path] of the newly created [store object](../glossary.md#gloss-store-object).
A path in an interpolated expression is first copied into the Nix store, and the resulting string is the [store path] of the newly created [store object](@docroot@/store/store-object.md).
*Paths*, e.g., `/bin/sh` or `./builder.sh`. A path must contain at
least one slash to be recognised as such. For instance, `builder.sh`
is not a path: it's parsed as an expression that selects the
attribute `sh` from the variable `builder`. If the file name is
relative, i.e., if it does not begin with a slash, it is made
absolute at parse time relative to the directory of the Nix
expression that contained it. For instance, if a Nix expression in
`/foo/bar/bla.nix` refers to `../xyzzy/fnord.nix`, the absolute path
is `/foo/xyzzy/fnord.nix`.
*Paths* are distinct from strings and can be expressed by path literals such as `./builder.sh`.
If the first component of a path is a `~`, it is interpreted as if
the rest of the path were relative to the user's home directory.
e.g. `~/foo` would be equivalent to `/home/edolstra/foo` for a user
whose home directory is `/home/edolstra`.
Paths are suitable for referring to local files, and are often preferable over strings.
- Path values do not contain trailing slashes, `.` and `..`, as they are resolved when evaluating a path literal.
- Path literals are automatically resolved relative to their [base directory](@docroot@/glossary.md#gloss-base-directory).
- The files referred to by path values are automatically copied into the Nix store when used in a string interpolation or concatenation.
- Tooling can recognize path literals and provide additional features, such as autocompletion, refactoring automation and jump-to-file.
For instance, evaluating `"${./foo.txt}"` will cause `foo.txt` in the current directory to be copied into the Nix store and result in the string `"/nix/store/<hash>-foo.txt"`.
A path literal must contain at least one slash to be recognised as such.
For instance, `builder.sh` is not a path:
it's parsed as an expression that selects the attribute `sh` from the variable `builder`.
Note that the Nix language assumes that all input files will remain _unchanged_ while evaluating a Nix expression.
Path literals may also refer to absolute paths by starting with a slash.
> **Note**
>
> Absolute paths make expressions less portable.
> In the case where a function translates a path literal into an absolute path string for a configuration file, it is recommended to write a string literal instead.
> This avoids some confusion about whether files at that location will be used during evaluation.
> It also avoids unintentional situations where some function might try to copy everything at the location into the store.
If the first component of a path is a `~`, it is interpreted such that the rest of the path were relative to the user's home directory.
For example, `~/foo` would be equivalent to `/home/edolstra/foo` for a user whose home directory is `/home/edolstra`.
Path literals that start with `~` are not allowed in [pure](@docroot@/command-ref/conf-file.md#conf-pure-eval) evaluation.
Paths can be used in [string interpolation] and string concatenation.
For instance, evaluating `"${./foo.txt}"` will cause `foo.txt` from the same directory to be copied into the Nix store and result in the string `"/nix/store/<hash>-foo.txt"`.
Note that the Nix language assumes that all input files will remain _unchanged_ while evaluating a Nix expression.
For example, assume you used a file path in an interpolated string during a `nix repl` session.
Later in the same session, after having changed the file contents, evaluating the interpolated string with the file path again might not return a new [store path], since Nix might not re-read the file contents.
Later in the same session, after having changed the file contents, evaluating the interpolated string with the file path again might not return a new [store path], since Nix might not re-read the file contents. Use `:r` to reset the repl as needed.
[store path]: ../glossary.md#gloss-store-path
[store path]: @docroot@/store/store-path.md
Paths can include [string interpolation] and can themselves be [interpolated in other expressions].
Path literals can also include [string interpolation], besides being [interpolated into other expressions].
[interpolated in other expressions]: ./string-interpolation.md#interpolated-expressions
[interpolated into other expressions]: ./string-interpolation.md#interpolated-expressions
At least one slash (`/`) must appear *before* any interpolated expression for the result to be recognized as a path.
`a.${foo}/b.${bar}` is a syntactically valid division operation.
`a.${foo}/b.${bar}` is a syntactically valid number division operation.
`./a.${foo}/b.${bar}` is a path.
[Lookup paths](./constructs/lookup-path.md) such as `<nixpkgs>` resolve to path values.
[Lookup path](./constructs/lookup-path.md) literals such as `<nixpkgs>` also resolve to path values.
@@ -18,10 +18,30 @@ is a JSON object with the following fields:
Information about the output paths of the derivation.
This is a JSON object with one member per output, where the key is the output name and the value is a JSON object with these fields:
*`path`: The output path.
*`path`:
The output path, if it is known in advanced.
Otherwise, `null`.
*`method`:
For an output which will be [content addresed], a string representing the [method](@docroot@/store/store-object/content-address.md) of content addressing that is chosen.
For fixed-output derivations, the hashing algorithm (e.g. `sha256`), optionally prefixed by `r:` if `hash` denotes a NAR hash rather than a flat file hash.
For an output which will be [content addresed], the name of the hash algorithm used.
Valid algorithm strings are:
-`md5`
-`sha1`
-`sha256`
-`sha512`
*`hash`:
For fixed-output derivations, the expected content hash in base-16.
@@ -32,7 +52,8 @@ is a JSON object with the following fields:
[file system object]: @docroot@/store/file-system-object.md
The format of this specification is close to [Extended Backus–Naur form](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form), with the exception of the `str(..)` function / parameterized rule, which length-prefixes and pads strings.
This makes the resulting binary format easier to parse.
Regular users do *not* need to know this information.
But for those interested in exactly how Nix works, e.g. if they are reimplementing it, this information can be useful.
This is the complete specification for how store paths are calculated.
This is the complete specification for how [store path]s are calculated.
The format of this specification is close to [Extended Backus–Naur form](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form), but must deviate for a few things such as hash functions which we treat as bidirectional for specification purposes.
Regular users do *not* need to know this information --- store paths can be treated as black boxes computed from the properties of the store objects they refer to.
But for those interested in exactly how Nix works, e.g. if they are reimplementing it, this information can be useful.
- ["Nix Archive"](@docroot@/store/store-object/content-address.md#method-nix-archive) if the hash algorithm is not [SHA-256].
`id` is the name of the output (usually, "out").
For content-addressed store objects, `id`, is always "out".
@@ -89,15 +100,20 @@ where
- `rec` = one of:
- ```ebnf
| ""
```
(empty string) for hashes of the flat (single file) serialization
- ```ebnf
| "r:"
```
hashes of the for [Nix Archive (NAR)] (arbitrary file system object) serialization
- ```ebnf
| ""
| "git:"
```
(empty string) for hashes of the flat (single file) serialization
hashes of the [Git blob/tree](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects) [Merkel tree](https://en.wikipedia.org/wiki/Merkle_tree) format
- ```ebnf
algo = "md5" | "sha1" | "sha256"
@@ -108,8 +124,8 @@ where
Note that `id` = `"out"`, regardless of the name part of the store path.
Also note that NAR + SHA-256 must not use this case, and instead must use the `type` = `"source:" ...` case.
The Nix release cycle is calendar-based as follows:
Nix has a release cycle of roughly 6 weeks.
Notable changes and additions are announced in the release notes for each version.
Bugfixes can be backported on request to previous Nix releases.
We typically backport only as far back as the Nix version used in the latest NixOS release, which is announced in the [NixOS release notes](https://nixos.org/manual/nixos/stable/release-notes.html#ch-release-notes).
Backports never skip releases.
If a feature is backported to version `x.y`, it must also be available in version `x.(y+1)`.
This ensures that upgrading from an older version with backports is still safe and no backported functionality will go missing.
The supported Nix versions are:
- The latest release
- The version used in the stable NixOS release, which is announced in the [NixOS release notes](https://nixos.org/manual/nixos/stable/release-notes.html#ch-release-notes).
Bugfixes and security issues are backported to every supported version.
As the choice of hash formats is no longer binary, the `--base16` flag is also added
to explicitly specify the Base16 format, which is still the default.
* The special handling of an [installable](../command-ref/new-cli/nix.md#installables) with `.drv` suffix being interpreted as all of the given [store derivation](../glossary.md#gloss-store-derivation)'s output paths is removed, and instead taken as the literal store path that it represents.
* The special handling of an [installable](../command-ref/new-cli/nix.md#installables) with `.drv` suffix being interpreted as all of the given [store derivation](@docroot@/glossary.md#gloss-store-derivation)'s output paths is removed, and instead taken as the literal store path that it represents.
The new `^` syntax for store paths introduced in Nix 2.13 allows explicitly referencing output paths of a derivation.
Using this is better and more clear than relying on the now-removed `.drv` special handling.
- Fix a fixed-output derivation sandbox escape (CVE-2024-27297)
Cooperating Nix derivations could send file descriptors to files in the Nix
store to each other via Unix domain sockets in the abstract namespace. This
allowed one derivation to modify the output of the other derivation, after Nix
has registered the path as "valid" and immutable in the Nix database.
In particular, this allowed the output of fixed-output derivations to be
modified from their expected content.
This isn't the case any more.
- CLI options `--arg-from-file` and `--arg-from-stdin` [#10122](https://github.com/NixOS/nix/pull/10122)
The new CLI option `--arg-from-file`*name**path* passes the contents
of file *path* as a string value via the function argument *name* to a
Nix expression. Similarly, the new option `--arg-from-stdin`*name*
reads the contents of the string from standard input.
- Concise error printing in `nix repl` [#9928](https://github.com/NixOS/nix/pull/9928)
Previously, if an element of a list or attribute set threw an error while
evaluating, `nix repl` would print the entire error (including source location
information) inline. This output was clumsy and difficult to parse:
```
nix-repl> { err = builtins.throw "uh oh!"; }
{ err = «error:
… while calling the 'throw' builtin
at «string»:1:9:
1| { err = builtins.throw "uh oh!"; }
| ^
error: uh oh!»; }
```
Now, only the error message is displayed, making the output much more readable.
```
nix-repl> { err = builtins.throw "uh oh!"; }
{ err = «error: uh oh!»; }
```
However, if the whole expression being evaluated throws an error, source
locations and (if applicable) a stack trace are printed, just like you'd expect:
```
nix-repl> builtins.throw "uh oh!"
error:
… while calling the 'throw' builtin
at «string»:1:1:
1| builtins.throw "uh oh!"
| ^
error: uh oh!
```
- `--debugger` can now access bindings from `let` expressions [#8827](https://github.com/NixOS/nix/issues/8827) [#9918](https://github.com/NixOS/nix/pull/9918)
Breakpoints and errors in the bindings of a `let` expression can now access
those bindings in the debugger. Previously, only the body of `let` expressions
could access those bindings.
- Enter the `--debugger` when `builtins.trace` is called if `debugger-on-trace` is set [#9914](https://github.com/NixOS/nix/pull/9914)
If the `debugger-on-trace` option is set and `--debugger` is given,
`builtins.trace` calls will behave similarly to `builtins.break` and will enter
the debug REPL. This is useful for determining where warnings are being emitted
from.
- Debugger prints source position information [#9913](https://github.com/NixOS/nix/pull/9913)
The `--debugger` now prints source location information, instead of the
pointers of source location information. Before:
```
nix-repl> :bt
0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
0x600001522598
```
After:
```
0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
- The `--debugger` will start more reliably in `let` expressions and function calls [#6649](https://github.com/NixOS/nix/issues/6649) [#9917](https://github.com/NixOS/nix/pull/9917)
Previously, if you attempted to evaluate this file with the debugger:
```nix
let
a = builtins.trace "before inner break" (
builtins.break "hello"
);
b = builtins.trace "before outer break" (
builtins.break a
);
in
b
```
Nix would correctly enter the debugger at `builtins.break a`, but if you asked
it to `:continue`, it would skip over the `builtins.break "hello"` expression
entirely.
Now, Nix will correctly enter the debugger at both breakpoints.
- Nested debuggers are no longer supported [#9920](https://github.com/NixOS/nix/pull/9920)
Previously, evaluating an expression that throws an error in the debugger would
enter a second, nested debugger:
```
nix-repl> builtins.throw "what"
error: what
Starting REPL to allow you to inspect the current state of the evaluator.
Welcome to Nix 2.18.1. Type :? for help.
nix-repl>
```
Now, it just prints the error message like `nix repl`:
```
nix-repl> builtins.throw "what"
error:
… while calling the 'throw' builtin
at «string»:1:1:
1| builtins.throw "what"
| ^
error: what
```
- Consistent order of function arguments in printed expressions [#9874](https://github.com/NixOS/nix/pull/9874)
Function arguments are now printed in lexicographic order rather than the internal, creation-time based symbol order.
- Fix duplicate attribute error positions for `inherit` [#9874](https://github.com/NixOS/nix/pull/9874)
When an `inherit` caused a duplicate attribute error the position of the error was not reported correctly, placing the error with the inherit itself or at the start of the bindings block instead of the offending attribute name.
- `inherit (x) ...` evaluates `x` only once [#9847](https://github.com/NixOS/nix/pull/9847)
`inherit (x) a b ...` now evaluates the expression `x` only once for all inherited attributes rather than once for each inherited attribute.
This does not usually have a measurable impact, but side-effects (such as `builtins.trace`) would be duplicated and expensive expressions (such as derivations) could cause a measurable slowdown.
- Store paths are allowed to start with `.` [#912](https://github.com/NixOS/nix/issues/912) [#9091](https://github.com/NixOS/nix/pull/9091) [#9095](https://github.com/NixOS/nix/pull/9095) [#9120](https://github.com/NixOS/nix/pull/9120) [#9121](https://github.com/NixOS/nix/pull/9121) [#9122](https://github.com/NixOS/nix/pull/9122) [#9130](https://github.com/NixOS/nix/pull/9130) [#9219](https://github.com/NixOS/nix/pull/9219) [#9224](https://github.com/NixOS/nix/pull/9224) [#9867](https://github.com/NixOS/nix/pull/9867)
Leading periods were allowed by accident in Nix 2.4. The Nix team has considered this to be a bug, but this behavior has since been relied on by users, leading to unnecessary difficulties.
From now on, leading periods are supported. The names `.` and `..` are disallowed, as well as those starting with `.-` or `..-`.
Nix versions that denied leading periods are documented [in the issue](https://github.com/NixOS/nix/issues/912#issuecomment-1919583286).
- Introduction of `--regex` and `--all` in `nix profile remove` and `nix profile upgrade` [#10166](https://github.com/NixOS/nix/pull/10166)
Previously the command-line arguments for `nix profile remove` and `nix profile upgrade` matched the package entries using regular expression.
For instance:
```
nix profile remove '.*vim.*'
```
This would remove all packages that contain `vim` in their name.
In most cases, only singular package names were used to remove and upgrade packages. Mixing this with regular expressions sometimes lead to unintended behavior. For instance, `python3.1` could match `python311`.
To avoid unintended behavior, the arguments are now only matching exact names.
Matching using regular expressions is still possible by using the new `--regex` flag:
```
nix profile remove --regex '.*vim.*'
```
One of the most useful cases for using regular expressions was to upgrade all packages. This was previously accomplished by:
```
nix profile upgrade '.*'
```
With the introduction of the `--all` flag, this now becomes more straightforward:
```
nix profile upgrade --all
```
- Visual clutter in `--debugger` is reduced [#9919](https://github.com/NixOS/nix/pull/9919)
Before:
```
info: breakpoint reached
Starting REPL to allow you to inspect the current state of the evaluator.
Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
nix-repl> :continue
error: uh oh
Starting REPL to allow you to inspect the current state of the evaluator.
Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
nix-repl>
```
After:
```
info: breakpoint reached
Nix 2.20.0pre20231222_dirty debugger
Type :? for help.
nix-repl> :continue
error: uh oh
nix-repl>
```
- Cycle detection in `nix repl` is simpler and more reliable [#8672](https://github.com/NixOS/nix/issues/8672) [#9926](https://github.com/NixOS/nix/pull/9926)
The cycle detection in `nix repl`, `nix eval`, `builtins.trace`, and everywhere
else values are printed is now simpler and matches the cycle detection in
The `repl-flake` experimental feature has been removed. The `nix repl` command now works like the rest of the new CLI in that `nix repl {path}` now tries to load a flake at `{path}` (or fails if the `flakes` experimental feature isn't enabled).
### Other changes
-`nix eval` prints derivations as `.drv` paths [#10200](https://github.com/NixOS/nix/pull/10200)
`nix eval` will now print derivations as their `.drv` paths, rather than as
attribute sets. This makes commands like `nix eval nixpkgs#bash` terminate
instead of infinitely looping into recursive self-referential attributes:
- Make `nix build --keep-going` consistent with `nix-build --keep-going`
This means that if e.g. multiple fixed-output derivations fail to
build, all hash mismatches are displayed.
- Modify `nix derivation {add,show}` JSON format [#9866](https://github.com/NixOS/nix/issues/9866) [#10722](https://github.com/NixOS/nix/pull/10722)
The JSON format for derivations has been slightly revised to better conform to our [JSON guidelines](@docroot@/contributing/cli-guideline.md#returning-future-proof-json).
In particular, the hash algorithm and content addressing method of content-addresed derivation outputs are now separated into two fields `hashAlgo` and `method`,
rather than one field with an arcane `:`-separated format.
This JSON format is only used by the experimental `nix derivation` family of commands, at this time.
Future revisions are expected as the JSON format is still not entirely in compliance even after these changes.
- Warn on unknown settings anywhere in the command line [#10701](https://github.com/NixOS/nix/pull/10701)
All `nix` commands will now properly warn when an unknown option is specified anywhere in the command line.
Before:
```console
$ nix-instantiate --option foobar baz --expr '{}'
warning: unknown setting 'foobar'
$ nix-instantiate '{}' --option foobar baz --expr
$ nix eval --expr '{}' --option foobar baz
{ }
```
After:
```console
$ nix-instantiate --option foobar baz --expr '{}'
warning: unknown setting 'foobar'
$ nix-instantiate '{}' --option foobar baz --expr
warning: unknown setting 'foobar'
$ nix eval --expr '{}' --option foobar baz
warning: unknown setting 'foobar'
{ }
```
- `nix env shell` is the new `nix shell`, and `nix shell` remains an accepted alias [#10504](https://github.com/NixOS/nix/issues/10504) [#10807](https://github.com/NixOS/nix/pull/10807)
This is part of an effort to bring more structure to the CLI subcommands.
`nix env` will be about the process environment.
Future commands may include `nix env run` and `nix env print-env`.
It is also somewhat analogous to the [planned](https://github.com/NixOS/nix/issues/10504) `nix dev shell` (currently `nix develop`), which is less about environment variables, and more about running a development shell, which is a more powerful command, but also requires more setup.
- Flake operations that expect derivations now print the failing value and its type [#10778](https://github.com/NixOS/nix/pull/10778)
In errors like `flake output attribute 'nixosConfigurations.yuki.config' is not a derivation or path`, the message now includes the failing value and type.
Before:
```
error: flake output attribute 'nixosConfigurations.yuki.config' is not a derivation or path
````
After:
```
error: expected flake output attribute 'nixosConfigurations.yuki.config' to be a derivation or path but found a set: { appstream = «thunk»; assertions = «thunk»; boot = { bcache = «thunk»; binfmt = «thunk»; binfmtMiscRegistrations = «thunk»; blacklistedKernelModules = «thunk»; bootMount = «thunk»; bootspec = «thunk»; cleanTmpDir = «thunk»; consoleLogLevel = «thunk»; «43 attributes elided» }; «48 attributes elided» }
```
- `fetchTree` now fetches Git repositories shallowly by default [#10028](https://github.com/NixOS/nix/pull/10028)
`builtins.fetchTree` now clones Git repositories shallowly by default, which reduces network traffic and disk usage significantly in many cases.
Previously, the default behavior was to clone the full history of a specific tag or branch (e.g. `ref`) and only afterwards extract the files of one specific revision.
From now on, the `ref` and `allRefs` arguments will be ignored, except if shallow cloning is disabled by setting `shallow = false`.
The defaults for `builtins.fetchGit` remain unchanged. Here, shallow cloning has to be enabled manually by passing `shallow = true`.
- Store object info JSON format now uses `null` rather than omitting fields [#9995](https://github.com/NixOS/nix/pull/9995)
The [store object info JSON format](@docroot@/protocols/json/store-object-info.md), used for e.g. `nix path-info`, no longer omits fields to indicate absent information, but instead includes the fields with a `null` value.
For example, `"ca": null` is used to to indicate a store object that isn't content-addressed rather than omitting the `ca` field entirely.
This makes records of this sort more self-describing, and easier to consume programmatically.
We will follow this design principle going forward;
the [JSON guidelines](@docroot@/contributing/json-guideline.md) in the contributing section have been updated accordingly.
- Large path warnings [#10661](https://github.com/NixOS/nix/pull/10661)
Nix can now warn when evaluation of a Nix expression causes a large
path to be copied to the Nix store. The threshold for this warning can
be configured using [the `warn-large-path-threshold`
For many operations, Nix needs to calculate [a content addresses](@docroot@/glossary.md#gloss-content-address) of [a file system object][file system object].
Usually this is needed as part of
[content addressing store objects](../store-object/content-address.md),
since store objects always have a root file system object.
But some command-line utilities also just work on "raw" file system objects, not part of any store object.
Every content addressing scheme Nix uses ultimately involves feeding data into a [hash function](https://en.wikipedia.org/wiki/Hash_function), and getting back an opaque fixed-size digest which is deemed a content address.
The various *methods* of content addressing thus differ in how abstract data (in this case, a file system object and its descendents) are fed into the hash function.
## Serialising File System Objects { #serial }
The simplest method is to serialise the entire file system object tree into a single binary string, and then hash that binary string, yielding the content address.
In this section we describe the currently-supported methods of serialising file system objects.
### Flat { #serial-flat }
A single file object can just be hashed by its contents.
This is not enough information to encode the fact that the file system object is a file,
but if we *already* know that the FSO is a single non-executable file by other means, it is sufficient.
Because the hashed data is just the raw file, as is, this choice is good for compatibility with other systems.
For example, Unix commands like `sha256sum` or `sha1sum` will produce hashes for single files that match this.
### Nix Archive (NAR) { #serial-nix-archive }
For the other cases of [file system objects][file system object], especially directories with arbitrary descendents, we need a more complex serialisation format.
Examples of such serialisations are the ZIP and TAR file formats.
However, for our purposes these formats have two problems:
- They do not have a canonical serialisation, meaning that given an FSO, there can
be many different serialisations.
For instance, TAR files can have variable amounts of padding between archive members;
and some archive formats leave the order of directory entries undefined.
This would be bad because we use serialisation to compute cryptographic hashes over file system objects, and for those hashes to be useful as a content address or for integrity checking, uniqueness is crucial.
Otherwise, correct hashes would report false mismatches, and the store would fail to find the content.
- They store more information than we have in our notion of FSOs, such as time stamps.
This can cause FSOs that Nix should consider equal to hash to different values on different machines, just because the dates differ.
- As a practical consideration, the TAR format is the only truly universal format in the Unix environment.
It has many problems, such as an inability to deal with long file names and files larger than 2^33 bytes.
Current implementations such as GNU Tar work around these limitations in various ways.
For these reasons, Nix has its very own archive format—the Nix Archive (NAR) format,
which is carefully designed to avoid the problems described above.
The exact specification of the Nix Archive format is in `protocols/nix-archive.md`
## Content addressing File System Objects beyond a single serialisation pass
Serialising the entire tree and then hashing that binary string is not the only option for content addressing, however.
Another technique is that of a [Merkle graph](https://en.wikipedia.org/wiki/Merkle_tree), where previously computed hashes are included in subsequent byte strings to be hashed.
In particular, the Merkle graphs can match the original graph structure of file system objects:
we can first hash (serialised) child file system objects, and then hash parent objects using the hashes of their children in the serialisation (to be hashed) of the parent file system objects.
Currently, there is one such Merkle DAG content addressing method supported.
> This method is part of the [`git-hashing`][xp-feature-git-hashing] experimental feature.
Git's file system model is very close to Nix's, and so Git's content addressing method is a pretty good fit.
Just as with regular Git, files and symlinks are hashed as git "blobs", and directories are hashed as git "trees".
However, one difference between Nix's and Git's file system model needs special treatment.
Plain files, executable files, and symlinks are not differentiated as distinctly addressable objects, but by their context: by the directory entry that refers to them.
That means so long as the root object is a directory, there is no problem:
every non-directory object is owned by a parent directory, and the entry that refers to it provides the missing information.
However, if the root object is not a directory, then we have no way of knowing which one of an executable file, non-executable file, or symlink it is supposed to be.
In response to this, we have decided to treat a bare file as non-executable file.
This is similar to do what we do with [flat serialisation](#serial-flat), which also lacks this information.
To avoid an address collision, attempts to hash a bare executable file or symlink will result in an error (just as would happen for flat serialisation also).
Thus, Git can encode some, but not all of Nix's "File System Objects", and this sort of content-addressing is likewise partial.
In the future, we may support a Git-like hash for such file system objects, or we may adopt another Merkle DAG format which is capable of representing all Nix file system objects.
of the store object, and not any other information, which would not be an intrinsic property of that store object.
For the full specification of the algorithms involved, see the [specification of store path digests][sp-spec].
[File System Object]: ../file-system-object.md
[Store Object]: ../store-object.md
[Store Path]: ../store-path.md
## Content addressing each part of a store object
### File System Objects
With all currently supported store object content addressing methods, the file system object is always [content-addressed][fso-ca] first, and then that hash is incorporated into content address computation for the store object.
### References
With all currently supported store object content addressing methods,
other objects are referred to by their regular (string-encoded-) [store paths][Store Path].
Self-references however cannot be referred to by their path, because we are in the midst of describing how to compute that path!
> The alternative would require finding as hash function fixed point, i.e. the solution to an equation in the form
> ```
> digest = hash(..... || digest || ....)
> ```
> which is computationally infeasible.
> As far as we know, this is equivalent to finding a hash collision.
Instead we just have a "has self reference" boolean, which will end up affecting the digest.
### Name and Store Directory
These two items affect the digest in a way that is standard for store path digest computations and not specific to content-addressing.
Consult the [specification of store path digests][sp-spec] for further details.
## Content addressing Methods
For historical reasons, we don't support all features in all combinations.
Each currently supported method of content addressing chooses a single method of file system object hashing, and may offer some restrictions on references.
The names and store directories are unrestricted however.
### Flat { #method-flat }
This uses the corresponding [Flat](../file-system-object/content-address.md#serial-flat) method of file system object content addressing.
References are not supported: store objects with flat hashing *and* references can not be created.
### Text { #method-text }
This also uses the corresponding [Flat](../file-system-object/content-address.md#serial-flat) method of file system object content addressing.
References to other store objects are supported, but self references are not.
This is the only store-object content-addressing method that is not named identically with a corresponding file system object method.
It is somewhat obscure, mainly used for "drv files"
(derivations serialized as store objects in their ["ATerm" file format](@docroot@/protocols/derivation-aterm.md)).
Prefer another method if possible.
### Nix Archive { #method-nix-archive }
This uses the corresponding [Nix Archive](../file-system-object/content-address.md#serial-nix-archive) method of file system object content addressing.
References (to other store objects and self references alike) are supported so long as the hash algorithm is SHA-256, but not (neither kind) otherwise.
### Git { #method-git }
> **Warning**
>
> This method is part of the [`git-hashing`][xp-feature-git-hashing] experimental feature.
This uses the corresponding [Git](../file-system-object/content-address.md#serial-git) method of file system object content addressing.
References are not supported.
Only SHA-1 is supported at this time.
If [SHA-256-based Git](https://git-scm.com/docs/hash-function-transition)
becomes more widespread, this restriction will be revisited.
1. Triage issues and pull requests from the [No Status](#no-status) column (30 min)
2. Discuss issues and pull requests from the [To discuss](#to-discuss) column (30 min).
@@ -49,15 +51,19 @@ The team meets twice a week:
- mark it as draft if it is blocked on the contributor
- escalate it back to the team by moving it to To discuss, and leaving a comment as to why the issue needs to be discussed again.
- Work meeting: [Mondays 13:00-15:00 CET](https://calendar.google.com/calendar/event?eid=NTM1MG1wNGJnOGpmOTZhYms3bTB1bnY5cWxfMjAyMjExMjFUMTIwMDAwWiBiOW81MmZvYnFqYWs4b3E4bGZraGczdDBxZ0Bn)
- Work meeting: [Mondays 13:00-15:00 Europe/Amsterdam](https://www.google.com/calendar/event?eid=Ym52NDdzYnRic2NzcDcybjZiNDhpNzhpa3NfMjAyNDA1MTNUMTIwMDAwWiBiOW81MmZvYnFqYWs4b3E4bGZraGczdDBxZ0Bn)
1. Code review on pull requests from [In review](#in-review).
2. Other chores and tasks.
Meeting notes are collected on a [collaborative scratchpad](https://pad.lassul.us/Cv7FpYx-Ri-4VjUykQOLAw).
Notes on issues and pull requests are posted as comments and linked from the meeting notes, so they are easy to find from both places.
Notes on issues and pull requests are posted as comments and linked from the meeting notes, so they are can be found from both places.
[All meeting notes](https://discourse.nixos.org/search?expanded=true&q=Nix%20team%20meeting%20minutes%20%23%20%23dev%3Anix%20in%3Atitle%20order%3Alatest_topic) are published on Discourse under the [Nix category](https://discourse.nixos.org/c/dev/nix/50).
Team meetings are generally open to anyone interested.
We can make exceptions to discuss sensitive issues, such as security incidents or people matters.
Contact any team member to get a calendar invite for reminders and updates.
## Project board protocol
The team uses a [GitHub project board](https://github.com/orgs/NixOS/projects/19/views/1) for tracking its work.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
