This PR reduces the creation of short-lived basic_json objects while
parsing flake.lock files. For large flake.lock files (~1.5MB) I was
observing ~60s being spent for trivial nix build operations while
after this change it is now taking ~1.6s.
(cherry picked from commit 7fd0de38c6)
(cherry picked from commit 89cadf5d64)
this part must have been moved quite a while ago, but apparently so far
no one noticed
(cherry picked from commit 6db805b3d1)
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
up to now, those were managed outside of this repo, which as
unsurprisingly a real hassle to deal with if one wanted to prevent URLs
from breaking when moving pages around. this change removes a large part
of the friction involved in moving content in the Nix manual.
possible next steps for further automation:
- check for content that moved and warn if it's not reachable from
links that were valid prior to a change
- create redirect rules automatically based on this information
(cherry picked from commit 2b7016cc56)
https://github.com/NixOS/nixpkgs/pull/269064 makes rapidcheck be build
as a shared lib, but that broke Nix because the `-lrapidcheck` was
missing. This fixes that (and doesn't break Nix what the library is a
static archive as today).
(cherry picked from commit 46131567da)
This has been the behaviour before Nix 2.4. It was dropped in a rewrite
in 759947bf72, allowing the creation of
store paths that aren't considered valid by older Nix versions or other
Nix tooling.
Nix 2.4 didn't ship in NixOS until 22.05, and stdenv.mkDerivation in
nixpkgs drops leading periods since April 2022, so it's unlikely anyone
is relying on the current lax behaviour.
Closes#9091.
Change-Id: I4a57bd9899e1b0dba56870ae5a1b680918a18ce9
(cherry picked from commit 24bda0c7b3)
Without the change build with `-D_GLIBCXX_ASSERTIONS` exposes testsuite
assertion:
$ gdb src/libexpr/tests/libnixexpr-tests
Reading symbols from src/libexpr/tests/libnixexpr-tests...
(gdb) break __glibcxx_assert_fail
(gdb) run
(gdb) bt
in std::__glibcxx_assert_fail(char const*, int, char const*, char const*)@plt () from /mnt/archive/big/git/nix/src/libexpr/libnixexpr.so
in std::basic_string_view<char, std::char_traits<char> >::operator[] (this=0x7fffffff56c0, __pos=4)
at /nix/store/r74fw2j8rx5idb0w8s1s6ynwwgs0qmh9-gcc-14.0.0/include/c++/14.0.0/string_view:258
in nix::SearchPath::Prefix::suffixIfPotentialMatch (this=0x7fffffff5780, path=...) at src/libexpr/search-path.cc:15
in nix::SearchPathElem_suffixIfPotentialMatch_partialPrefix_Test::TestBody (this=0x555555a17540) at src/libexpr/tests/search-path.cc:62
As string sizes are usigned types `(a - b) > 0` effectively means
`a != b`. While the intention should be `a > b`.
The change fixes test suite pass.
(cherry picked from commit b74962c92b)
Will need to do subclass-specific implementations in the next commit.
This isn't because there will be multiple variations of the daemon
protocol (whew!) but because different clients pick and choose different
parts to use.
This makes it more useful. In general, the derivation will be in one
store, and the realisation info is in another.
This also helps us avoid duplication. See how `resolveDerivedPath` is
now simpler because it uses `queryPartialDerivationOutputMap`. In #8369
we get more flavors of derived path, and need more code to resolve them
all, and this problem only gets worse.
The fact that we need a new method to deal with the multiple dispatch is
unfortunate, but this generally relates to the fact that `Store` is a
sub-par interface, too bulky/unwieldy and conflating separate concerns.
Solving that is out of scope of this PR.
This is part of the RFC 92 work. See tracking issue #6316
the original change broke many pre-existing anchor links.
also change formatting of the constants listing slightly:
- the type should not be part of the anchor
- add highlight to the "impure only" note
* clarify wording on args@ default handling
Most importantly use shorter sentences and emphasize the key point that defaults aren't taken into account
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
this is a how-to guide which should not be in the reference manual.
it also refers to `nix-env`, which should not be the first thing readers
of the reference manual encounter, as it behaves very differently in
spirit from the rest of Nix.
slightly reword the documentation to be more concise and informative.
Grouping our tests should make it easier to understand the intent than
one long poorly-arranged list. It also is convenient for running just
the tests for a specific component when working on that component.
We need at least one test group so this isn't dead code; I decided to
collect the tests for the `ca-derivations` and `dynamic-derivations`
experimental features in groups. Do
```bash
make ca.test-group -jN
```
and
```bash
make dyn-drv.test-group -jN
```
to try running just them.
I originally did this as part of #8397 for being able to just the local
overlay store alone. I am PRing it separately now so we can separate
general infra from new features.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
We were bedeviled by sandboxing issues when working on the layered
store. The problem ended up being that when we have nested nix builds,
and the inner store is inside the build dir (e.g. store is
`/build/nix-test/$name/store`, build dir is `/build`) bind mounts
clobber each other and store paths cannot be found.
After thoroughly cleaning up `local-derivation-goal.cc`, we might be
able to make that work. But that is a lot of work. For now, we just fail
earlier with a proper error message.
Finally, test this: nested sandboxing without the problematic store dir
should work, and with should fail with the expected error message.
Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
When we pipe to `>(...)` like that, we unfortunately don't wait for the
process to finish. Better to just substitute the file.
Also, use the "unified" diff output that people (including myself) are
more familiar with, thanks to Git.
* Lang now verifies errors and parse output
* Some new miscellaneous tests
* Easy way to update the tests
* Document workflow in manual
* Use `!` not `~` as separater char for sed
It is confusing to use `~` when we are talking about paths and home
directories!
* Test test suite itself (`test/lang-test/infra.sh`)
Additionally, run shellcheck on `tests/lang.sh` to help ensure it is
correct, now that is is more complex.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
- Better types
- Own header / C++ file pair
- Test factored out methods
- Pass parsed thing around more than strings
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Whereas `ContentAddressWithReferences` is a sum type complex because different
varieties support different notions of reference, and
`ContentAddressMethod` is a nested enum to support that,
`ContentAddress` can be a simple pair of a method and hash.
`ContentAddress` does not need to be a sum type on the outside because
the choice of method doesn't effect what type of hashes we can use.
Co-Authored-By: Cale Gibbard <cgibbard@gmail.com>
'resolvedRef' was incorrect, since a resolved ref is one after
registry resolution, which may still be unlocked (e.g. 'nixpkgs' ->
'github:NixOS/nixpkgs').
If we call `adjustLoc`, the global variable `prev_yylloc` is shared
between threads and racy.
Currently, nix itself does not concurrently parsing files, but this is
helpful for libexpr users. (The parser is thread-safe except this.)
When explicitly requested by the caller, as suggested in the meeting
(https://github.com/NixOS/nix/pull/8090#issuecomment-1531139324)
> @edolstra: { toPath } vs { fromPath } is too implicit
I've opted for the `inputAddressed = true` requirement, because it
we did not agree on renaming the path attributes.
> @roberth: more explicit
> @edolstra: except for the direction; not immediately clear in which direction the rewriting happens
This is in fact the most explicit syntax and a bit redundant, which is
good, because that redundancy lets us deliver an error message that
reminds expression authors that CA provides a better experience to
their users.
* nix flake check: improve error message if overlay is not a lambda
Suppose you have an overlay like this
{
inputs = { /* ... */ };
outputs = { flake-utils, ... }: flake-utils.lib.eachDefaultSystem
(system: {
overlays.default = final: prev: {
};
});
}
then `nix flake check` (correctly) fails because `overlays` are supposed
to have the structure `overlays.<name> = final: prev: exp`. However, the
error-message is a little bit counter-intuitive:
error: overlay does not take an argument named 'final'
While one might guess where the error actually comes from because the
trace above says `… while checking the overlay 'overlays.x86_64-linux'`
this is still pretty confusing because it complains about an argument
not being named `final` even though that's evidently the case.
With this change, the error-message actually makes it clear what's
wrong:
[ma27@carsten:~/Projects/nix/tmp]$ nix flake check --extra-experimental-features 'nix-command flakes' path:$(pwd)
error:
… while checking flake output 'overlays'
at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:
3| outputs = { ... }: {
4| overlays.x86_64-linux.snens = final: prev: {
| ^
5| kek = throw "snens";
… while checking the overlay 'overlays.x86_64-linux'
at /nix/store/clgblnxx003hyrq8qkz5ab6kgqkck6qc-source/flake.nix:4:5:
3| outputs = { ... }: {
4| overlays.x86_64-linux.snens = final: prev: {
| ^
5| kek = throw "snens";
error: overlay is not a lambda, but a set instead
I got very confused trying to keep all the `first` and `second` straight
reading the code, *especially* as there is also another `(boolean,
string)` pair type also being used.
Named fields is much better.
There are other cleanups that we can do (for example, the existing
TODO), but we can do them later. Doing them now would just make this
harder to review.
- Improved API docs from comment
- Exit codes are for `nix-build`, not just `nix-store --release`
- Make note in tests so the magic numbers are not surprising
Picking up where #8387 left off.
Deleting store info corrected (there is a foot-gun in Nix with
`--delete-generations old`!)
Also a few things are cleaned up based on feedback.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Previously it was not possible to open a local store when its database is on a read-only filesystem. Obviously a store on a read-only filesystem cannot be modified, but it would still be useful to be able to query it.
This change adds a new read-only setting to LocalStore. When set to true, Nix will skip operations that fail when the database is on a read-only filesystem (acquiring big-lock, schema migration, etc), and the store database will be opened in immutable mode.
Co-authored-by: Ben Radford <benradf@users.noreply.github.com>
Co-authored-by: cidkidnix <cidkidnix@protonmail.com>
Co-authored-by: Dylan Green <67574902+cidkidnix@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Uninstall instructions were moved to their own page in #8267. The
overall section link was redirected in #8286, but platform-specific
links (which I give out frequently when I triage installer trouble)
weren't included.
Pass this around instead of `Source &` and `Sink &` directly. This will
give us something to put the protocol version on once the time comes.
To do this ergonomically, we need to expose `RemoteStore::Connection`,
so do that too. Give it some more API docs while we are at it.
The motivation is exactly the same as for the last commit. In addition,
this anticipates us formally defining separate serialisers for the serve
protocol.
See API docs on that struct for why. The pasing as as template argument
doesn't yet happen in that commit, but will instead happen in later
commit.
Also make `WorkerOp` (now `Op`) and enum struct. This led us to catch
that two operations were not handled!
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This is generally a fine practice: Putting implementations in headers
makes them harder to read and slows compilation. Unfortunately it is
necessary for templates, but we can ameliorate that by putting them in a
separate header. Only files which need to instantiate those templates
will need to include the header with the implementation; the rest can
just include the declaration.
This is now documenting in the contributing guide.
Also, it just happens that these polymorphic serializers are the
protocol agnostic ones. (Worker and serve protocol have the same logic
for these container types.) This means by doing this general template
cleanup, we are also getting a head start on better indicating which
code is protocol-specific and which code is shared between protocols.
- Greatly expand API docs
- Clean up code in misc ways
- Instead of a complicated single loop on generations, do different
operations in successive subsequent steps.
- Avoid `ref` in one place where `&` is fine
- Just return path instead of mutating an argument in `makeName`
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
While this is not actually a notion in the implementation, it is
explicitly described in the thesis and quite important for understanding
how the store works.
Co-authored-by: John Ericson <git@JohnEricson.me>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Rather than doing `allowEmpty` as boolean, have separate types and use
`std::optional`. This makes it harder to forget the possibility of an
empty path.
The `build-hook` setting was categorized as a `PathSetting`, but
actually it was split into arguments. No good! Now, it is
`Setting<Strings>` which actually reflects what it means and how it is
used.
Because of the subtyping, we now also have support for
`Setting<std::optional<String>>` in general. I imagine this can be used
to clean up many more settings also.
The code accidentally conflated `std::string::size_type` and `long unsigned int`.
This was fine on 64bits machines where they are apparently the same in
practice, but not on 32bits. Fix that by using `std::string::size_type`
everywhere.
A library shouldn't require changes to the caller's argument handling,
especially if it doesn't have to, and indeed we don't have to.
This changes the lookup order to prioritize the hardcoded path to nix
if it exists. The static executable still finds itself through /proc
and the like.
Introduce what substituters "are" in the configuration option entry.
Remove arbitrary line breaks for easier editing in the future.
Link glossary some more.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: John Ericson <git@JohnEricson.me>
The remaining constructor RegisterPrimOp::RegisterPrimOp(Info && info)
allows specifying the documentation in .args and .doc members of the
Info structure.
Commit 8ec1ba0210 removed all uses of the removed constructor in the
nix binary. Here, we remove the constructor completely as well as its
use in a plugin test. According to #8515, we didn't promis to maintain
compatibility with external plugins.
Fixes#8515
`filesystem.cc` is the only place where `createSymlink()` is used with three arguments:
in the definition of `replaceSymlink()` with three parameters that _is not used at all_.
Closes#8495
Previously, for tarball flakes, we recorded the original URL of the
tarball flake, rather than the URL to which it ultimately
redirects. Thus, a flake URL like
http://example.org/patchelf-latest.tar that redirects to
http://example.org/patchelf-<revision>.tar was not really usable. We
couldn't record the redirected URL, because sites like GitHub redirect
to CDN URLs that we can't rely on to be stable.
So now we use the redirected URL only if the server returns the
`x-nix-is-immutable` or `x-amz-meta-nix-is-immutable` headers in its
response.
This will allow documenting them (in later commits).
Note that we keep the old constructor even if it is no longer used by
Nix code, because it is used in tests/plugins/plugintest.cc, which
suggests that it might be used by some external plugin.
This is necessary when we're in a chroot environment, where the
process root is not the same as the root of the mount namespace
(e.g. in nixos-enter).
Fixes#7602.
Currently `fromTOML` throws an exception when encountering a timestamp
since the Nix language lacks a way to represent them.
This patch changes this beaviour and makes `fromTOML` parse timestamps as
attrsets of the format
{ _type = "timestamp"; value = "1979-05-27T07:32:00Z"; }
This is guarded by an experimental feature flag to leave room for iterating on the representation.
packages and configurations are not really a concept in Nix or the Nix language. the idea of transforming files into other files clearly captures what it's all about, and the new phrasing should make the term "derivation" more obvious both in terms of meaning and origin.
* Document manual migration for use-xdg-base-directories
As there's currently no automatic migration for use-xdg-base-directories
option, add instructions for manual migration to the option's
description.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Add support to --list-generations
as another way to say
nix-env --profile /nix/var/nix/profiles/per-user/$USER/channels --list-generations
the way we did for nix-channel --rollback [generation id]
And fix a test failure in the sandbox due to /home
existing on Darwin but not being accessible in the sandbox since it's a
symlink to /System/Volumes/Data/home, see
https://github.com/NixOS/nix/actions/runs/4205378453/jobs/7297384658#step:6:2127:
C++ exception with description "error: getting status of /home/schnitzel/darmstadt/pommes: Operation not permitted" thrown in the test body.
On Linux this wasn't a problem because there /home doesn't exist in the sandbox
The primop `builtins.replaceStrings` currently always strictly evaluates the
replacement strings, however time and space are wasted for their computation
if the corresponding pattern do not occur in the input string. This commit
makes the evaluation of the replacement strings lazy by deferring their
evaluation to when the corresponding pattern are matched and memoize the result
for efficient retrieval on subsequent matches.
The testcases for replaceStrings was updated to check for lazy evaluation
of the replacements. A note was also added in the release notes to
document the behavior change.
macOS Ventura ships with it's own version of diff. Try to output a
similar diff with Apple diff as with GNU diff, instead of failing
Helps https://github.com/NixOS/nix/issues/7286
When encountering a build error, Nix moves the output paths out of the
chroot into their final location (for “easier debugging of build
failures”). However this was broken for chroot stores as it was moving
it to the _logical_ location, not the _physical_ one.
Fix it by moving to the physical (_real_) location.
Fix https://github.com/NixOS/nix/issues/8395
The `hashed-mirrors` option did use to have this default value,
but it was removed and re-added with an empty default value.
As the autogenerated docs show the (actual) default values from code,
remove this incorrect reference from the docs.
I was updating my nix.conf settings after a few years and noticed this.
Using abstract types like can help cut down on compilation time, both
from scratch, and especially incremental builds during development. The
idea is that `worker-protocol.hh` can declare all the (de)serializers, but
only again abstract types; when code needs to use some (de)serializers, it can
include headers just for the data types it needs to (de)serialize.
`store-api.hh` in particular is a bit of a sledgehammer, and the data
types we want to serialize have their own headers.
This is the more typically way to do [Argument-dependent
lookup](https://en.cppreference.com/w/cpp/language/adl)-leveraging
generic serializers in C++. It makes the relationship between the `read`
and `write` methods more clear and rigorous, and also looks more
familiar to users coming from other languages that do not have C++'s
libertine ad-hoc overloading.
I am returning to this because during the review in
https://github.com/NixOS/nix/pull/6223, it came up as something that
would make the code easier to read --- easier today hopefully already,
but definitely easier if we were have multiple codified protocols with
code sharing between them as that PR seeks to accomplish.
If I recall correctly, the main criticism of this the first time around
(in 2020) was that having to specify the type when writing, e.g.
`WorkerProto<MyType>::write`, was too verbose and cumbersome. This is
now addressed with the `workerProtoWrite` wrapper function.
This method is also the way `nlohmann::json`, which we have used for a
number of years now, does its serializers, for what its worth.
This reverts commit 45a0ed82f0. That
commit in turn reverted 9ab07e99f5.
This is good in general, but in particular ensures when we heavily
refactor it in the next commit there is less likelihood for an
unintentional change in behavior to sneak in.
These items are not templates, and they declared in
`worker-protocol.hh`; therefore they should live in a
`worker-protocol.cc`.
Anything else needlessly diverges from convention. After all, it is not
like this code is only used in `remote-store.cc`; it is also used in
`daemon.cc`. There is no good reason to place it with the client
implementation or the server implementation when it used equally by
both.
it's more likely for readers to find it right there.
this also slightly rewords examples to make them stand out better.
in the long run there probably needs to be a dedicated section on formal syntax, and better highlighting of examples.
e.g. nix-env -e subversion => nix-env --uninstall subversion
The aim is to make the documentation less cryptic for newcomers and the
long options are more self-documenting.
The change was made with the following script:
<https://github.com/aschmolck/convert-short-nix-opts-to-long-ones>
and sanity checked visually.
- If the element comes from a flake, print the full flakeref (with the
fragment part) and not just the reference to the flake itself
- If the element doesn't come from a flake, print its store path(s)
This is a bit too verbose, but has the advantages of being correct (and
not crashing), so it's strictly better than the previous situation
Fix https://github.com/NixOS/nix/issues/8284
This gives some more context and should clarify why it works that way.
Also link it from the section on `NIX_USER_CONF_FILES`.
Co-authored-by: John Ericson <git@JohnEricson.me>
They were improperly added in 8a93b5a551.
They were not `.gitignore`d because they were stale in that commit --
build artifacts no longer used that name by then and so `.gitignore` was
updated accordingly.
As discussed in #7417, it would be good to make more string values work
as installables. That is to say, if an installable refers to a value,
and the value is a string, it used to not work at all, since #7484, it
works somewhat, and this PR make it work some more.
The new cases that are added for `BuiltPath` contexts:
- Fixed input- or content-addressed derivation:
```
nix-repl> hello.out.outPath
"/nix/store/jppfl2bp1zhx8sgs2mgifmsx6dv16mv2-hello-2.12"
nix-repl> :p builtins.getContext hello.out.outPath
{ "/nix/store/c7jrxqjhdda93lhbkanqfs07x2bzazbm-hello-2.12.drv" = { outputs = [ "out" ]; }; }
The string matches the specified single output of that derivation, so
it should also be valid.
- Floating content-addressed derivation:
```
nix-repl> (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath
"/1a08j26xqc0zm8agps8anxpjji410yvsx4pcgyn4bfan1ddkx2g0"
nix-repl> :p builtins.getContext (hello.overrideAttrs (_: { __contentAddressed = true; })).out.outPath
{ "/nix/store/qc645pyf9wl37c6qvqzaqkwsm1gp48al-hello-2.12.drv" = { outputs = [ "out" ]; }; }
```
The string is not a path but a placeholder, however it also matches
the context, and because it is a CA derivation we have no better
option. This should also be valid.
We may also want to think about richer attrset based values (also
discussed in that issue and #6507), but this change "completes" our
string-based building blocks, from which the others can be desugared
into or at least described/document/taught in terms of.
Progress towards #7417
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
`/etc/bash.bashrc` is backed up as `/etc/bash.bashrc.backup-before-nix`,
but since other changes might have been introduced in the meantime we can't
just tell the user to revert.
add anchor to `builtins.derivation` and list some built-in functions that are
exposed in the global scope.
I decided not to list everything, because we probably don't want to
encourage people using them that way.
Fixes#8309
This regression was because both `CmdDevelop` and `CmdPrintDevEnv` were
switched to be `InstallableValueCommand` subclasses, but actually
neither should have been.
The `nixpkgsFlakeRef` method should indeed not be on the base
installable class, because "flake refs" and "nixpkgs" are not
installable-wide notions, but that doesn't mean these commands should
only accept installable values.
This fixes a bug in commands like `nix eval' which would emit invalid attribute
sets if they contained reserved keywords such as "assert", "let", etc.
These keywords will not be quoted when printed, making them valid expressions.
All keywords recognized by the lexer are quoted except "or", which does not
require quotation.
fe5509df caused only wanted outputs to be passed to the
post-build-hook, which resulted in paths being built
without ever going into the hook.
This commit adds a (currently failing) test for this.
* doc rendering: add functions to scope explicitly
this especially helps beginners with code readability, since the origin
of names is always immediately visible.
it's probably better not to show the manifest file documentation in the
command-specific pages, because these are implementation details that are not really practically useful.
this means no additional hassle for building the manual, but clutters
the table of contents a bit.
placed in a subsection of the binary install, the instructions are hard
to find. putting them in a separate page that is shown in the table of
contents should make it easier for users to find what they need when
they need it.
Previously, we relied on the `shutdown()` function to terminate `accept()`
calls on a listening socket. However, this approach did not work on macOS as
the waiting `accept()` call is not considered a connected socket, resulting in
an `ENOTCONN` error. Instead, we now close the listening socket to terminate
the `accept()` call.
Additionally, we fixed a resource management issue where we set the
`daemonSocket` variable to -1, triggering resource cleanup and causing the
`stopDaemon` function to be called twice. This resulted in errors as the socket
was already closed by the time the second `stopDaemon` call was made. Instead of
setting `daemonSocket` to -1, we now release the socket using the `release()`
method on a unique pointer. This properly transfers ownership and allows for
correct resource cleanup.
These changes ensure proper behavior and resource management for the
recursive-nix feature on macOS.
Motivation
`PathSet` is not correct because string contexts have other forms
(`Built` and `DrvDeep`) that are not rendered as plain store paths.
Instead of wrongly using `PathSet`, or "stringly typed" using
`StringSet`, use `std::std<StringContextElem>`.
-----
In support of this change, `NixStringContext` is now defined as
`std::std<StringContextElem>` not `std:vector<StringContextElem>`. The
old definition was just used by a `getContext` method which was only
used by the eval cache. It can be deleted altogether since the types are
now unified and the preexisting `copyContext` function already suffices.
Summarizing the previous paragraph:
Old:
- `value/context.hh`: `NixStringContext = std::vector<StringContextElem>`
- `value.hh`: `NixStringContext Value::getContext(...)`
- `value.hh`: `copyContext(...)`
New:
- `value/context.hh`: `NixStringContext = std::set<StringContextElem>`
- `value.hh`: `copyContext(...)`
----
The string representation of string context elements no longer contains
the store dir. The diff of `src/libexpr/tests/value/context.cc` should
make clear what the new representation is, so we recommend reviewing
that file first. This was done for two reasons:
Less API churn:
`Value::mkString` and friends did not take a `Store` before. But if
`NixStringContextElem::{parse, to_string}` *do* take a store (as they
did before), then we cannot have the `Value` functions use them (in
order to work with the fully-structured `NixStringContext`) without
adding that argument.
That would have been a lot of churn of threading the store, and this
diff is already large enough, so the easier and less invasive thing to
do was simply make the element `parse` and `to_string` functions not
take the `Store` reference, and the easiest way to do that was to simply
drop the store dir.
Space usage:
Dropping the `/nix/store/` (or similar) from the internal representation
will safe space in the heap of the Nix programming being interpreted. If
the heap contains many strings with non-trivial contexts, the saving
could add up to something significant.
----
The eval cache version is bumped.
The eval cache serialization uses `NixStringContextElem::{parse,
to_string}`, and since those functions are changed per the above, that
means the on-disk representation is also changed.
This is simply done by changing the name of the used for the eval cache
from `eval-cache-v4` to eval-cache-v5`.
----
To avoid some duplication `EvalCache::mkPathString` is added to abstract
over the simple case of turning a store path to a string with just that
string in the context.
Context
This PR picks up where #7543 left off. That one introduced the fully
structured `NixStringContextElem` data type, but kept `PathSet context`
as an awkward middle ground between internal `char[][]` interpreter heap
string contexts and `NixStringContext` fully parsed string contexts.
The infelicity of `PathSet context` was specifically called out during
Nix team group review, but it was agreeing that fixing it could be left
as future work. This is that future work.
A possible follow-up step would be to get rid of the `char[][]`
evaluator heap representation, too, but it is not yet clear how to do
that. To use `NixStringContextElem` there we would need to get the STL
containers to GC pointers in the GC build, and I am not sure how to do
that.
----
PR #7543 effectively is writing the inverse of a `mkPathString`,
`mkOutputString`, and one more such function for the `DrvDeep` case. I
would like that PR to have property tests ensuring it is actually the
inverse as expected.
This PR sets things up nicely so that reworking that PR to be in that
more elegant and better tested way is possible.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
In other words, use a plain `ContentAddress` not
`ContentAddressWithReferences` for `DerivationOutput::CAFixed`.
Supporting fixed output derivations with (fixed) references would be a
cool feature, but it is out of scope at this moment.
Recently, I encountered the "NAR info file 'xxxx' is corrupt" error
with my binary cache. The message is not helpful in determining, which
kind of corruption happened. The file, fetched with curl, looked
reasonably.
This commit adds more information to the error message, which should
allow debugging and hopefully fixing the problem.
We finally test the status quo of remote build trust in a number of
ways. We create a new experimental feature on `nix-daemon` to do so.
PR #3921, which improves the situation with trustless remote building,
will build upon these changes. This code / tests was pull out of there
to make this, so everything is easier to review, and in particular we
test before and after so the new behavior in that PR is readily apparent
from the testsuite diff alone.
Issues:
1. Features gated on disabled experimental settings should warn and be
ignored, not silently succeed.
2. Experimental settings in the same config "batch" (file or env var)
as the enabling of the experimental feature should work.
3. For (2), the order should not matter.
These are analogous to the issues @roberth caught with my changes for
arg handling, but they are instead for config handling.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
In many cases we are dealing with a collection of realisations, they are
all outputs of the same derivation. In that case, we don't need
"derivation hashes modulos" to be part of our map key, because the
output names alone will be unique. Those hashes are still part of the
realisation proper, so we aren't loosing any information, we're just
"normalizing our schema" by narrowing the "primary key".
Besides making our data model a bit "tighter" this allows us to avoid a
double `for` loop in `DerivationGoal::waiteeDone`. The inner `for` loop
was previously just to select the output we cared about without knowing
its hash. Now we can just select the output by name directly.
Note that neither protocol is changed as part of this: we are still
transferring `DrvOutputs` over the wire for `BuildResult`s. I would only
consider revising this once #6223 is merged, and we can mention protocol
versions inside factored-out serialization logic. Until then it is
better not change anything because it would come a the cost of code
reuse.
If my memory is correct, @edolstra objected to modifying `wantedOutputs`
upon falling back to doing a build (as we did before), because we should
only modify it in response to new requests --- *actual* wants --- and
not because we are "incidentally" building all the outptus beyond what
may have been requested.
That's a fair point, and the alternative is to replace the boolean soup
with proper enums: Instead of modifying `wantedOuputs` som more, we'll
modify `needsRestart` to indicate we are passed the need.
In https://github.com/NixOS/nix/pull/6311#discussion_r834863823, I
realized since derivation goals' wanted outputs can "grow" due to
overlapping dependencies (See `DerivationGoal::addWantedOutputs`, called
by `Worker::makeDerivationGoalCommon`), the previous bug fix had an
unfortunate side effect of causing more pointless rebuilds.
In paticular, we have this situation:
1. Goal made from `DerivedPath::Built { foo, {a} }`.
2. Goal gives on on substituting, starts building.
3. Goal made from `DerivedPath::Built { foo, {b} }`, in fact is just
modified original goal.
4. Though the goal had gotten as far as building, so all outputs were
going to be produced, `addWantedOutputs` no longer knows that and so
the goal is flagged to be restarted.
This might sound far-fetched with input-addressed drvs, where we usually
basically have all our goals "planned out" before we start doing
anything, but with CA derivation goals and especially RFC 92, where *drv
resolution* means goals are created after some building is completed, it
is more likely to happen.
So the first thing to do was restore the clearing of `wantedOutputs` we
used to do, and then filter the outputs in `buildPathsWithResults` to
only get the ones we care about.
But fix also has its own side effect in that the `DerivedPath` in the
`BuildResult` in `DerivationGoal` cannot be trusted; it is merely the
*first* `DerivedPath` for which this goal was originally created.
To remedy this, I made `BuildResult` be like it was before, and instead
made `KeyedBuildResult` be a subclass wit the path. Only
`buildPathsWithResults` returns `KeyedBuildResult`s, everything else
just becomes like it was before, where the "key" is unambiguous from
context.
I think separating the "primary key" field(s) from the other fields is
good practical in general anyways. (I would like to do the same thing
for `ValidPathInfo`.) Among other things, it allows constructions like
`std::map<Key, ThingWithKey>` where doesn't contain duplicate keys and
just precludes the possibility of those duplicate keys being out of
sync.
We might leverage the above someday to overload `buildPathsWithResults`
to take a *set* of return a *map* per the above.
-----
Unfortunately, we need to avoid C++20 strictness on designated
initializers.
(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)
No having that yet, maybe it would be better to not make
`KeyedBuildResult` a subclass to just avoid this.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
As requested by @roberth, it is good to call out the specific instances
we care about, which is `!` for the RPC protocols, and `^` for humans.
This doesn't take advantage of parametricity as much, but since the
human and computer interfaces are good to decouple anyways (we don't
care if they drift further apart over time in the slightest) some
separation and slight duplication is fine.
Also, unit test both round trips.
More progress on issue #5729
The method trivially generalizes to be store-implementation-agnostic, in
fact.
However, we force it to continue to be unimplemented with `RemoteStore`
and `LegacySSHStore` because the implementation we'd get via the
generalization is probably not the one users expect. This keeps our
hands untied to do it right going forward.
For more about the tension between the scheduler logic being
store-type-agnostic and remote stores doing their own scheduling, see
issues #5025 and #5056.
- Create a glossary entry for experimental features.
- Have the man page experimental feature notice link `nix-commmand`.
(Eventually this should be programmed, based on whether the command is
experimental, and if so what experimental feature does it depend on.)
- Document which installables depend on which experimental features.
I tried to use the same style (bold warning and block quote) that the
top of the man page uses.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
The warning message should produce an installable name that can be
passed to `nix build`, `nix path-info`, etc. again. Since the CLI
expects that the .drv path and the output names are separated by
a caret, the warning message must also separate the .drv path and output
names with a caret.
However, `DerivedPath::Built.to_string()` uses an exclamation point as
the separator instead. This commit adds a `separator` argument to the
to_string method.
This changes the warning message from:
If this command is now failing try again with '/nix/store/foo.drv!*'
to:
If this command is now failing try again with '/nix/store/foo.drv^*'
More progress on issue #5729.
Instead of having it by the default method in `Store` itself, have it be
the implementation in `DummyStore` and `LegacySSHStore`. Then just the
implementations which fail to provide the method pay the "penalty" of
dealing with the icky `unimplemented` function for non-compliance.
Combined with my other recent PRs, this finally makes `Store` have no
`unsupported` calls!
This is somewhat hacky fix just for 2.15. I unintentionally hid them
from the manual, when no one wanted to hide them that (including
myself). I also required the experimental feature to be enabled in an
order-dependent way, which is not good.
The simplest fix for this immanent release is just to always show them,
and always allow them to be set.
Effectively undoes some changes from aa663b7e89
Getting the occasional SQLITE_BUSY is expected when the database is being
accessed concurrently. The retry will likely succeed so it is pointless to warn
immediately. Instead we track how long each retrySQLite block has been running,
and only begin warning after a second has elapsed (and then every 10 seconds
subsequently).
How signals should be handled depends on what kind of process Nix
is integrated into. The signal handler thread used by the stand-alone
Nix commands / processes may not work well in the context of other
runtime systems, such as those of Python, Perl, or Haskell.
libutil is a dependency of libstore, so it should always be
initialized as such.
libutil is also a dependency of libmain. Being explicit about this
dependency might be good, but not worth the slight code complexity
until the library structure gets more advanced.
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
Quote
Why not initLibExpr()? initGC() is essentially that, but
detectStackOverflow is not an instance of the init function concept, as
it may have to be invoked more than once per process.
Furthermore, renaming initGC to initLibExpr is more trouble than it's
worth at this time.
This code is bad. We shouldn't unset variables in programs whose
children may need them. Fixing one issue at a time, so postponing.
See https://github.com/NixOS/nix/issues/7731
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
It is required for the sandbox, which is a libstore responsibility;
not just libmain.
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
The goal of this reordering is to make initLibStore self-sufficient
in a following commit.
Part of an effort to make it easier to initialize the right things,
by moving code into the appropriate libraries.
Using libstore without loading the config file is risky, as sqlite
may then be misconfigured. See https://github.com/cachix/cachix/issues/475
* Finish converting existing comments for internal API docs
99% of this was just reformatting existing comments. Only two exceptions:
- Expanded upon `BuildResult::status` compat note
- Split up file-level `symbol-table.hh` doc comments to get
per-definition docs
Also fixed a few whitespace goofs, turning leading tabs to spaces and
removing trailing spaces.
Picking up from #8133
* Fix two things from comments
* Use triple-backtick not indent for `dumpPath`
* Convert GNU-style `\`..'` quotes to markdown style in API docs
This will render correctly.
This is non-breaking change in the to-JSON direction. This *is* a
breaking change in the from-JSON direction, but we don't care, as that
is brand new in this PR.
`nix show-derivation --help` currently has the sole public documentation
of this format, it is updated accordingly.
The `write` name is ambiguous and could lead to some funny bugs like
https://github.com/NixOS/nix/pull/8173#issuecomment-1500009480. So
rename it to the more explicit `writeUnbuffered`.
Besides, this method shouldn't be (and isn't) used outside of the class
implementation, so mark it `protected`.
This makes it more symetrical to `BufferedSource` which uses a
`protected readUnbuffered` method.
This function returns true or false depending on whether the Nix client
is trusted or not. Mostly relevant when speaking to a remote store with
a daemon.
We include this information in `nix ping store` and `nix doctor`
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
They are put in the manual separate pages under the new overarching
description of experimental features.
The settings page just lists the valid experimental feature names (so
people know what a valid setting entry looks like), with links to those
pages. It doesn't attempt to describe each experimental feature as that
is too much information for the configuration settings section.
This introduces the SourcePath type from lazy-trees as an abstraction
for accessing files from inputs that may not be materialized in the
real filesystem (e.g. Git repositories). Currently, however, it's just
a wrapper around CanonPath, so it shouldn't change any behaviour. (On
lazy-trees, SourcePath is a <InputAccessor, CanonPath> tuple.)
the semantics are not explained in the referenced section any more, they
have been moved to the documentation for common options in the new CLI [0].
[0]: 703d863a48
Instead of constructing a markdown list in C++ (which involved all sorts
of nasty string literals), export some JSON and assemble it with the
manual build system.
Besides following the precedent set with other dumped data, this is a
better separate of content and presentation; if we decide for example we
want to display this information in a different way, or in a different
section of the manual, it will become much easier to do so.
switch statements must now match all enum values or disable the
warning.
Explicit is good. This has helped us find two bugs, after solving
another one by debugging.
From now on, adding to an enum will raise errors where they are
not explicitly handled, which is good for productivity, and helps
us decide the correct behavior in all usages.
Notably still excluded from this though are the cases where the
warning is disabled by local pragmas.
fromTOML.cc did not build despite a top-level pragma, so I've had
to resort to a makefile solution for that.
Prior to this, there was an ad-hoc whitelist in `main.cc`. Now, every
command states its stability.
In a future PR, we will adjust the manual to take advantage of this new
information in the JSON.
(It will be easier to do that once we have some experimental feature
docs to link too; see #5930 and #7798.)
The code is not local-store-specific, so we should share it with all
stores. More uniform behavior is better, and a less store-specific
functionality is more maintainable.
This fixes a FIXME added in f73d911628 by @edolstra himself.
I think we want to ensure that all new items in headers are documented,
and the documentation on modified items is kept up to date.
It will take a while to document the backlog of undocumented things, but
we can at least ensure that new items don't extend that backlog.
Was probably an overlook of when the tests were first added, but that
now messes-up with the `nix-channel --update` that happens down the line
(and can't access the network since we're inside a Nix build)
otherwise the order of found `.md` files will influence if `@docroot@`
is replaced before them being included, which may mess up relative
links.
the weirdest thing about it is that the mess-up happens
deterministically on macOS, but deterministically doesn't happen on
Linux!
Documentation on "classic" commands with many sub-commands are
notoriously hard to discover due to lack of overview and anchor links.
Additionally the information on common options and environment variables
is not accessible offline in man pages, and therefore often overlooked
by readers.
With this change, each sub-command of nix-store and nix-env gets its
own page in the manual (listed in the table of contents), and each own
man page.
Also, man pages for each subcommand now (again) list common options
and environment variables. While this makes each page quite long and
some common parameters don't apply, this should still make it easier
to navigate as that additional information was not accessible on the
command line at all.
It is now possible to run 'nix-store --<subcommand> --help` to display
help pages for the given subcommand.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This fixes the issue that `nix-build`, without experimental feature
'nix-command' enabled, recommends the experimental CLI `nix log` to view
build logs. Now it'll recommend the stable `nix-store -l` CLI instead.
Fixes https://github.com/NixOS/nix/issues/8118
in order to make the development process more transparent for everyone,
all pull requests should go through the triage process before getting
merged.
this ensures that all team members are aware of what is going on, and
that rationale for decisions is kept track of in the meeting notes for
posterity. (ideally all that should go into the commit history, but this
is a more invasive process change that needs further deliberation.)
having all team members take at least a brief but conscious look at each
change may also help with keeping our quality standards, as more
reviewers are more likely to remind each other of our shared values.
See the note in the test.
We don't want these flags showing up for commands where they are
irrelevant.
Eventually, this needs a proper fix, but it need not be a blocker for
stabilize: for a quick-n-dirty punt, just put these flags behind the
`nix-command` unstable feature.
This is fine because they are only relevant for commands which we don't
need to stabilize for a while.
`legacyPackages` of nixpkgs trigger eval errors in `hasContent`, causing
the whole `legacyPackages` being skipped. We should treat it as
has-content in that case.
Instead of having a bunch of optional fields, have a few subclasses
which can have mandatory fields.
Additionally, the new `getExtraPathInfo`, and `nixpkgsFlakeRef`, are
moved to `InstallableValue`.
I did these things because https://github.com/NixOS/rfcs/pull/134 ; with
these things moved to `InstallableValue`, the base `Installable` no
longer depends on libexpr! This is a major step towards that.
Also, add a bunch of doc comments for sake of the internal API docs.
Otherwise, a trace consisting of
frame
frame
frame
non-frame
... would reach the non-frame and print the suggestion, even though
it would have ignored the non-frame anyway.
This resulted in a peculariar situation where --show-trace would have
no apparent effect, as the trace was actually already complete.
Pause logger before starting SSH connections, and resume it after the
connection is established, so that SSH password prompts are not erased
by the logger's updates.
Otherwise, when running as root and user namespaces are enabled,
opening the slave fails with EPERM.
Fixes "opening pseudoterminal slave: Permission denied" followed by a
hang (https://hydra.nixos.org/build/213104244), and "error: getting
sandbox mount namespace: No such file or directory" (#8072), which
happens when the child fails very quickly and consequently reading
/proc/<child>/ns fails.
If we conditionally "declare" the argument, as we did before, based upon
weather the feature is enabled, commands like
nix --experimental-features=foo ... --thing-gated-on-foo
won't work, because the experimental feature isn't enabled until *after*
we start parsing.
Instead, allow arguments to also be associated with experimental
features (just as we did for builtins and settings), and then the
command line parser will filter out the experimental ones.
Since the effects of arguments (handler functions) are performed right
away, we get the required behavior: earlier arguments can enable later
arguments enabled!
There is just one catch: we want to keep non-positional
flags...non-positional. So if
nix --experimental-features=foo ... --thing-gated-on-foo
works, then
nix --thing-gated-on-foo --experimental-features=foo ...
should also work.
This is not my favorite long-term solution, but for now this is
implemented by delaying the requirement of needed experimental features
until *after* all the arguments have been parsed.
We hide them in various ways if the experimental feature isn't enabled.
To do this, we had to move the experimental features list out of
libnixstore, because the setting machinary itself depends on it. To do
that, we made a new `ExperimentalFeatureSettings`.
Currently it gives a 500 error with "Do not know how to serve path
'/nix/store/bym5sm8z2wpavnvzancb9gjdlgyzs1l8-nix-internal-api-docs-2.15.0pre20230320_e37f436/share/doc/nix/internal-api'."
I noticed a regression in the lazy-trees branch, which I'm trying to
capture with this test. While the tests succeeds in master, the
lazy-trees branch gives the following error message:
error: access to path
'/build/nix-test/tests/flakes/flake-in-submodule/rootRepo/submodule/flake.nix'
is forbidden because it is not under Git control; maybe you should
'git add' it to the repository
'/build/nix-test/tests/flakes/flake-in-submodule/rootRepo'?
This provides a platform-independent way to configure the SSL
certificates file in the Nix daemon. Previously we provided
instructions for overriding the environment variable in launchd, but
that obviously doesn't work with systemd. Now we can just tell users
to add
ssl-cert-file = /etc/ssl/my-certificate-bundle.crt
to their nix.conf.
PRs that don't increase our ongoing obligations (i.e. by adding new
features) but do increase test coverage of existing features are good
things to merge for the health of the project, and thus good to
prioritize.
These methods would previously fail on the other `Installable`s, so
moving them to this class is more correct as to where they actually
work.
Additionally, a `InstallableValueCommand` is created to make it easier
(or rather no worse than before) to write commands that just work on
`InstallableValue`s.
Besides being a cleanup to avoid failing default methods, this gets us
closer to https://github.com/NixOS/rfcs/pull/134.
- Try not to put cryptic "99" in many places
Factor out `exit 99` into `skipTest` function
- Alows make sure skipping a test is done with a reason
`skipTest` takes a mandatory argument
- Separate pure conditionals vs side-effectful test skipping.
"require daemon" already had this, but "sandbox support" did not.
Already, we had classes like `BuiltPathsCommand` and `StorePathsCommand`
which provided alternative `run` virtual functions providing the
implementation with more arguments. This was a very nice and easy way to
make writing command; just fill in the virtual functions and it is
fairly clear what to do.
However, exception to this pattern were `Installable{,s}Command`. These
two classes instead just had a field where the installables would be
stored, and various side-effecting `prepare` and `load` machinery too
fill them in. Command would wish out those fields.
This isn't so clear to use.
What this commit does is make those command classes like the others,
with richer `run` functions.
Not only does this restore the pattern making commands easier to write,
it has a number of other benefits:
- `prepare` and `load` are gone entirely! One command just hands just
hands off to the next.
- `useDefaultInstallables` because `defaultInstallables`. This takes
over `prepare` for the one case that needs it, and provides enough
flexiblity to handle `nix repl`'s idiosyncratic migration.
- We can use `ref` instead of `std::shared_ptr`. The former must be
initialized (so it is like Rust's `Box` rather than `Option<Box>`,
This expresses the invariant that the installable are in fact
initialized much better.
This is possible because since we just have local variables not
fields, we can stop worrying about the not-yet-initialized case.
- Fewer lines of code! (Finally I have a large refactor that makes the
number go down not up...)
- `nix repl` is now implemented in a clearer way.
The last item deserves further mention. `nix repl` is not like the other
installable commands because instead working from once-loaded
installables, it needs to be able to load them again and again.
To properly support this, we make a new superclass
`RawInstallablesCommand`. This class has the argument parsing and
completion logic, but does *not* hand off parsed installables but
instead just the raw string arguments.
This is exactly what `nix repl` needs, and allows us to instead of
having the logic awkwardly split between `prepare`,
`useDefaultInstallables,` and `load`, have everything right next to each
other. I think this will enable future simplifications of that argument
defaulting logic, but I am saving those for a future PR --- best to keep
code motion and more complicated boolean expression rewriting separate
steps.
The "diagnostic ignored `-Woverloaded-virtual`" pragma helps because C++
doesn't like our many `run` methods. In our case, we don't mind the
shadowing it all --- it is *intentional* that the derived class only
provides a `run` method, and doesn't call any of the overridden `run`
methods.
Helps with https://github.com/NixOS/rfcs/pull/134
Add the --base64 and --sri flags for the Base64 and SRI format output.
Add the --base16 flag to explicitly specify the hexadecimal format.
Add the --to-base64 and --to-sri flag to convert a hash to the above
mentioned format.
Hopefully this fixes "unexpected EOF" failures on macOS
(#3137, #3605, #7242, #7702).
The problem appears to be that under some circumstances, macOS
discards the output written to the slave side of the
pseudoterminal. Hence the parent never sees the "sandbox initialized"
message from the child, even though it succeeded. The conditions are:
* The child finishes very quickly. That's why this bug is likely to
trigger in nix-env tests, since that uses a builtin builder. Adding
a short sleep before the child exits makes the problem go away.
* The parent has closed its duplicate of the slave file
descriptor. This shouldn't matter, since the child has a duplicate
as well, but it does. E.g. moving the close to the bottom of
startBuilder() makes the problem go away. However, that's not a
solution because it would make Nix hang if the child dies before
sending the "sandbox initialized" message.
* The system is under high load. E.g. "make installcheck -j16" makes
the issue pretty reproducible, while it's very rare under "make
installcheck -j1".
As a fix/workaround, we now open the pseudoterminal slave in the
child, rather than the parent. This removes the second condition
(i.e. the parent no longer needs to close the slave fd) and I haven't
been able to reproduce the "unexpected EOF" with this.
This allows having multiple separate lockfiles for a single
project, which can be useful for testing against different versions of
nixpkgs; it also allows tracking custom input overrides for remote
flakes without requiring local clones of these flakes.
For example, if I want to build Nix against my locally pinned nixpkgs,
and have a lock file tracking this override independently of future
updates to said nixpkgs:
nix flake lock --output-lock-file /tmp/nix-flake.lock --override-input nixpkgs flake:nixpkgs
nix build --reference-lock-file /tmp/nix-flake.lock
Co-Authored-By: Will Fancher <elvishjerricco@gmail.com>
The motivation is as stated in issue #7814: even though the the C++ API
is internal and unstable, people still want it to be well documented for
sake of learning, code review, and other purposes that aren't predicated
on it being stable.
Fixes#7814
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
I saw this random failure in https://hydra.nixos.org/build/211811692:
error: opening /proc/15307/fd: No such process
while running nix-collect-garbage in a readfile-context.sh. This is
because we're not handling ESRCH errors reading /proc/<pid>/fd. So
just move the read inside the try/catch where we do handle it.
`nix copy` operations did not show progress. This is quite confusing.
Add a `progressSink` which displays the progress during `copyPaths`,
pretty much copied from `copyStorePath`.
Fixes https://github.com/NixOS/nix/issues/8000
Use `set -u` and `set -o pipefail` to catch accidental mistakes and
failures more strongly.
- `set -u` catches the use of undefined variables
- `set -o pipefail` catches failures (like `set -e`) earlier in the
pipeline.
This makes the tests a bit more robust. It is nice to read code not
worrying about these spurious success paths (via uncaught) errors
undermining the tests. Indeed, I caught some bugs doing this.
There are a few tests where we run a command that should fail, and then
search its output to make sure the failure message is one that we
expect. Before, since the `grep` was the last command in the pipeline
the exit code of those failing programs was silently ignored. Now with
`set -o pipefail` it won't be, and we have to do something so the
expected failure doesn't accidentally fail the test.
To do that we use `expect` and a new `expectStderr` to check for the
exact failing exit code. See the comments on each for why.
`grep -q` is replaced with `grepQuiet`, see the comments on that
function for why.
`grep -v` when we just want the exit code is replaced with `grepInverse,
see the comments on that function for why.
`grep -q -v` together is, surprise surprise, replaced with
`grepQuietInverse`, which is both combined.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
This was failing because the check for the existence of the
'installcheck' target failed silently, so the whole phase got
skipped. It works by running 'make -n installcheck 2> /dev/null',
which however barfs with
/nix/store/039g378vc3pc3dvi9dzdlrd0i4q93qwf-binutils-2.39/bin/ld.gold: error: cannot open tests/plugins/plugintest.o: No such file or directory
Fixes#8004.
This was causing random failures in tests/ca/substitute.ca: 'nix copy
--file ./content-addressed.nix' wouldn't get the default installable
'.' applied in InstallablesCommand::load(), so it would do nothing.
The curl download can outlive DrvOutputSubstitutionGoal (if some other
error occurs), so at shutdown setting the promise to an exception will
fail because 'this' is no longer valid in the callback. This can
manifest itself as a segfault, "corrupted double-linked list" or hang.
the term was hard to discover, as its definition and explanation were in
a very long document lacking an overview section.
search did not help because it occurs so often.
- clarify wording in the definition
- add an overview of installable types
- add "installable" to glossary
- link to definition from occurrences of the term
- be more precise about where store derivation outputs are processed
- installable Nix expressions must evaluate to a derivation
Co-authored-by: Adam Joseph <54836058+amjoseph-nixpkgs@users.noreply.github.com>
Presently when nix says something like:
```
these 486 paths will be fetched (511.54 MiB download, 6458.64 MiB unpacked):
...path1
...path2
...path3
...
...
...path486
```
It sorts path1, path2, path3, ..., path486 in lexicographic order of the
store path.
After this commit, nix will show path1, path2, path3, ..., path486 sorted by
StorePath name() (basically everything after the hash) rather than the store path.
This makes it easier to review what exactly is being downloaded at a glance,
especially when many paths need to be fetched.
We make sure the env var paths are actually set (ie. not "") before
sending them to the canonicalization function. If we forget to do so,
the user will end up facing a puzzled failed assertion internal error.
We issue a non-failing warning as a stop-gap measure. We could want to
revisit this to issue a detailed failing error message in the future.
For brand new installations, neither NIX_LINK_NEW
(`$XDG_STATE_HOME/nix/profile` or `~/.local/state/nix/profile`), nor
NIX_LINK (`~/.nix-profile`) will exist.
This restores functionality to nix-env, which is relied upon by GitHub
Actions such as https://github.com/cachix/cachix-action and the Nixpkgs
EditorConfig (and other) CI.
Currently the valid key is only present when the path is invalid, which
makes checking path validity more complex than it should be. With this
change, the valid key can always be used to check if a path is valid
The release notes document the change in behavior, I don't include it
here so there is no risk to it getting out of sync.
> Motivation
>> Plumbing CLI should be simple
Store derivation installations are intended as "plumbing": very simple
utilities for advanced users and scripts, and not what regular users
interact with. (Similarly, regular Git users will use branch and tag
names not explicit hashes for most things.)
The plumbing CLI should prize simplicity over convenience; that is its
raison d'etre. If the user provides a path, we should treat it the same
way not caring what sort of path it is.
>> Scripting
This is especially important for the scripting use-case. when arbitrary
paths are sent to e.g. `nix copy` and the script author wants consistent
behavior regardless of what those store paths are. Otherwise the script
author needs to be careful to filter out `.drv` ones, and then run `nix
copy` again with those paths and `--derivation`. That is not good!
>> Surprisingly low impact
Only two lines in the tests need changing, showing that the impact of
this is pretty light.
Many command, like `nix log` will continue to work with just the
derivation passed as before. This because we used to:
- Special case the drv path and replace it with it's outputs (what this
gets rid of).
- Turn those output path *back* into the original drv path.
Now we just skip that entire round trip!
> Context
Issue #7261 lays out a broader vision for getting rid of `--derivation`,
and has this as one of its dependencies. But we can do this with or
without that.
`Installable::toDerivations` is changed to handle the case of a
`DerivedPath::Opaque` ending in `.drv`, which is new: it simply doesn't
need to do any extra work in that case. On this basis, commands like
`nix {show-derivation,log} /nix/store/...-foo.drv` still work as before,
as described above.
When testing older daemons, the post-build-hook will be run against the
old CLI, so we need the old version of the post-build-hook to support
that use-case.
Co-authored-by: Travis A. Everett <travis.a.everett@gmail.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Resolves#7437 for new `nix` commands only by adding a `--stdin` flag.
If paths are also passed on the cli they will be combined with the ones
from standard input.
One of our CI machines installs Nix via the official script and then
sources the nix-profile.sh script to setup the environment. However, it
doesn't have XDG_STATE_HOME set, which causes sourcing the script to
fail.
Whenever a file conflict happens during "nix profile install" an error
is shown that was previously thrown inside builtins.buildEnv.
We catch BuildProfileConflictError here so that we can provide the user
with more useful instructions on what to do next.
Most notably, we give the user concrete commands to use with all
parameters already filled in. This avoids the need for the user to look
up these commands in manual pages.
At the moment an Error is thrown that only holds an error message
regarding `nix-env` and `nix profile`. These tools make use of
builtins.buildEnv, but buildEnv is also used in other places. These
places are unrelated to Nix profiles, so the error shouldn't mention
these tools.
This generic error is now BuildEnvFileConflictError, which holds more
contextual information about the files that were conflicting while
building the environment.
Building without tests is useful for bootstrapping with a smaller footprint
or running the tests in a separate derivation. Otherwise, we do compile and
run them.
This isn't fine grained as to allow picking `check` but not `installcheck`
or vice versa, but it's good enough for now.
I've tried to use Nixpkgs' `checkInputs`, but those inputs weren't discovered
properly by the configure script. We can emulate its behavior very well though.
so far there were no even remotely measurable objectives, only a general
purpose statement.
this change is intended to focus the team's work on what I (and many
others I talked to) perceive to be the main pain point in the
development process.
Co-authored-by: solene.rapenne@tweag.io
Split `common.sh` into the vars and functions definitions vs starting
the daemon (and possibly other initialization logic). This way,
`init.sh` can just `source` the former. Trying to start the daemon
before `nix.conf` is written will fail because `nix daemon` requires
`--experimental-features 'nix-command'`.
`killDaemon` is idempotent, so it's safe to call when no daemon is
running.
`startDaemon` and `killDaemon` use the PID (which is now exported to
subshells) to decide whether there is work to be done, rather than
`NIX_REMOTE`, which might conceivably be set differently even if a
daemon is running.
`startDaemon` and `killDaemon` can save/restore the old `NIX_REMOTE` as
`NIX_REMOTE_OLD`.
`init.sh` kills daemon before deleting everything (including the daemon
socket).
`init.sh` is tested on its own. We used to do that. I deleted it in
4720853129 but I am not sure why. Better
to just restore it; at one point working on this every other test
passed, so seems good to check whether `init.sh` can be run twice.
We don't *need* to run `init.sh` twice, but I want to try to make our
tests as robust as possible so that manual debugging (where tests for
better or worse might be run ways that we didn't expect) is less
fragile.
It would be incorrect to say that the `sourceInfo` has an `outPath`
that isn't the root. `sourceInfo` is about the root, whereas only
the flake may not be about the root. Thanks Eelco for pointing that
out.
Some dependencies supposed to be skipped in the cross build, along with
not using the gold linker. But in https://github.com/NixOS/nix/pull/6538
this was accidentally not preserved.
Also since https://github.com/NixOS/nix/pull/6538 we saw some new
aarch64-linux static build failures. This is a first attempt to try to
fix those failures. If this is not sufficient, there are other things we
can try next.
It is independent of SourceExprCommand, which is about parsing
installables, except for the fact that parsing installables is one of
the many things influenced by read-only mode.
- `nixpkgsFor` does all of native, static, cross, and the different stdenvs.
- The main Nix derivation is no longer duplicated for static.
- DRY nixpkgs.lib and lib.genAttrs calls.
If this documentation is inaccurate in any way please do not hesitate to suggest corrections.
My understanding of this function is strictly from reading the source code and some limited experience implementing fetchers.
We are looking for *$ because it indicate that it was constructed with a new but
not release. De-referencing shallow copy so deleting as whole might create
dangling pointer that's why we move it so we delete a empty containers + the
nice perf boost.
For static builds, we need to propagate all the static library
dependencies to the link of the program. E.g. if libstore-tests-exe
depends on libnixstore-tests, and libnixstore-tests depends on
libstore, then libstore-tests-exe needs to link against libstore.
https://hydra.nixos.org/build/209007480
- Refer to current version in readme
- Split into flakes and non-flakes section
- Change order to move nix-build to the end, since people often start
with it in the beginning.
- Use proper "Note" syntax
- Add notes about editor integration
- Move information about target platforms and stdenvs into separate
sections
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Alexander Bantyev <alexander.bantyev@tweag.io>
Co-authored-by: Théophane Hufschmitt <theophane.hufschmitt@tweag.io>
Adding a test to ensure there is no regression.
The tests that are split out of `tests/build.sh` are ones that don't yet
work with CA derivation. I have not yet evaluated whether they should or
not.
This behavior, reported missing in issue #4661, already got fixed in
PR #4818, but didn't get a test case then.
Nixpkgs on aarch64-linux is currently stuck on GCC 9
(https://github.com/NixOS/nixpkgs/issues/208412) and using gcc11Stdenv
doesn't work either.
So use c++2a instead of c++20 for now. Unfortunately this means we
can't use some C++20 features for now (like std::span).
Descriptions for commandline flags may not include newlines and should
be rather short for display in a shell. Truncate the description string
of a flag on '\n' or '.' to and add an ellipsis if needed.
XDG Base Directory is a standard for locations for storing various
files. Nix has a few files which seem to fit in the standard, but
currently use a custom location directly in the user's ~, polluting
it:
- ~/.nix-profile
- ~/.nix-defexpr
- ~/.nix-channels
This commit adds a config option (use-xdg-base-directories) to follow
the XDG spec and instead use the following locations:
- $XDG_STATE_HOME/nix/profile
- $XDG_STATE_HOME/nix/defexpr
- $XDG_STATE_HOME/nix/channels
If $XDG_STATE_HOME is not set, it is assumed to be ~/.local/state.
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
Co-authored-by: Tim Fenney <kodekata@gmail.com>
Co-authored-by: pasqui23 <pasqui23@users.noreply.github.com>
Co-authored-by: Artturin <Artturin@artturin.com>
Co-authored-by: John Ericson <Ericson2314@Yahoo.com>
Fixes#3898
The entire `BinaryCaches` row used to get replaced after it became
stale according to the `timestamp` column. In a concurrent scenario,
this leads to foreign key conflicts as different instances of the
in-process `state.caches` cache now differ, with the consequence that
the older process still tries to use the `id` number of the old record.
Furthermore, this phenomenon appears to have caused the cache for
actual narinfos to be erased about every week, while the default
ttl for narinfos was supposed to be 30 days.
This is slightly more accurate considering that an outdated record
may exist in the persistent cache. Possibly-outdated records are
quite relevant as they may be foreign keys to more recent information
that we want to keep, but we will not return them here.
In unprivileged podman containers, /proc is not fully visible (there
are other filesystems mounted on subdirectories of /proc). Therefore
we can't mount a new /proc in the sandbox that matches the PID
namespace of the sandbox. So this commit automatically disables
sandboxing if /proc is not fully visible.
This didn't work because sandboxing doesn't work in Docker. However,
the sandboxing check is done lazily - after clone(CLONE_NEWNS) fails,
we retry with sandboxing disabled. But at that point, we've already
done UID allocation under the assumption that user namespaces are
enabled.
So let's get rid of the "goto fallback" logic and just detect early
whether user / mount namespaces are enabled.
This commit also gets rid of a compatibility hack for some ancient
Linux kernels (<2.13).
Previously we would completely refetch the submodules from the
network, even though the repo might already have them. Now we copy the
.git/modules directory from the repo as an optimisation. This speeds
up evaluating
builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }
(where /path/to/blender already has the needed submodules) from 121s
to 57s.
This is still pretty inefficient and a hack, but a better solution is
best done on the lazy-trees branch.
This change also help in the case where the repo already has the
submodules but the origin is unfetchable for whatever reason
(e.g. there have been cases where Nix in a GitHub action doesn't have
the right authentication set up).
We cannot use 'actualUrl', because for file:// repos that's not the
original URL that the repo was fetched from. This is a problem since
submodules may be relative to the original URL.
Fixes e.g.
nix eval --impure --json --expr 'builtins.fetchTree { type = "git"; url = "/path/to/blender"; submodules = true; }'
where /path/to/blender is a clone of
https://github.com/blender/blender.git (which has several relative
submodules like '../blender-addons.git').
Previously, build-remote would show a warning if all build slots were
taken, even if they would open up later. This caused a lot of spam in
the logs. Disable this warning when maxJobs > 0.
See #6263
`clang11StdenvPackages` does not exist
```
│ └───x86_64-linux
│ ├───ccacheStdenv: development environment 'nix'
│ ├───clang11Stdenv: development environment 'nix'
│ ├───clangStdenv: development environment 'nix'
│ ├───default: development environment 'nix'
│ ├───gccStdenv: development environment 'nix'
│ ├───libcxxStdenv: development environment 'nix'
│ └───stdenv: development environment 'nix'
```
Per the old FIXME, this flag was on too many commands, and mostly
ignored. Now it is just on the commands where it actually has an effect.
Per https://github.com/NixOS/nix/issues/7261, I would still like to get
rid of it entirely, but that is a separate project. This change should
be good with or without doing that.
`nix app` had something called `InstallableDerivedPath` which is
actually the same thing. We go with the later's name because it has
become more correct.
I originally did this change (more hurriedly) as part of #6225 --- a
mini store-only Nix and a full Nix need to share this code. In the first
RFC meeting for https://github.com/NixOS/rfcs/pull/134 we discussed how
some splitting of the massive `installables.cc` could begin prior, as
that is a good thing anyways. (@edolstra's words, not mine!) This would
be one such PR.
tl;dr: With this 1 line change I was able to get a speedup of 1.5x on 1Gbit/s
wan connections by enabling zstd compression in nginx.
Also nix already supported all common compression format for http
transfer, webservers usually only enable them if they are advertised
through the Accept-Encoding header.
This pull requests makes nix advertises content compression support for
zstd, br, gzip and deflate.
It's particular useful to add transparent compression for binary caches
that serve packages from the host nix store in particular nix-serve,
nix-serve-ng and harmonia.
I tried so far gzip, brotli and zstd, whereas only zstd was able to bring
me performance improvements for 1Gbit/s WAN connections.
The following nginx configuration was used in combination with the
[zstd module](https://github.com/tokers/zstd-nginx-module) and
[harmonia](https://github.com/nix-community/harmonia/)
```nix
{
services.nginx.virtualHosts."cache.yourhost.com" = {
locations."/".extraConfig = ''
proxy_pass http://127.0.0.1:5000;
proxy_set_header Host $host;
proxy_redirect http:// https://;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
zstd on;
zstd_types application/x-nix-archive;
'';
};
}
```
For testing I unpacked a linux kernel tarball to the nix store using
this command `nix-prefetch-url --unpack https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.8.tar.gz`.
Before:
```console
$ nix build && rm -rf /tmp/hello && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwbb88sw36fvah-linux-6.1.8.tar.gz'
warning: Git tree '/scratch/joerg/nix' is dirty
real 0m18,375s
user 0m2,889s
sys 0m1,558s
```
After:
```console
$ nix build && rm -rf /tmp/hello && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwb
b88sw36fvah-linux-6.1.8.tar.gz'
real 0m11,884s
user 0m4,130s
sys 0m1,439s
```
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
Update src/libstore/filetransfer.cc
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
These settings are not needed for libstore at all, they are just used by
the nix daemon *command* for authorization on unix domain sockets. My
moving them to a new configuration struct just in that file, we avoid
them leaking anywhere else.
Also, it is good to break up the mammoth `Settings` struct in general.
Issue #5638 tracks this.
The message is not changed because I do not want to regress in
convenience to the user. Just saying "this connection is not trusted"
doesn't tell them out to fix the issue. The ideal thing to do would be
to somehow parameterize `processCommand` on how the error should be
displayed, so different sorts of connections can display different
information to the user based on how authentication is performed for the
connection in question. This, however, is a good bit more work, so it is
left for the future.
This came up with me thinking about the tcp:// store (#5265). The larger
project is not TCP *per se*, but the idea that it should be possible for
something else to manage access control to services like the Nix Daemon,
and those services simply trust or trust the incoming connection as they
are told. This is a more capability-oriented way of thinking about trust
than "every server implements its own auth separately" as we are used to today.
Its very great that libstore itself already implements just this model,
and so via this refactor I basically want to "enshrine" that so it
continues to be the case.
Since #7478 it's mandatory that `initLibStore()` is called for store
operations. However that's not the case when running `openStore()` in
Perl using the perl-bindings. That breaks e.g. `hydra-eval-jobset` when
built against Nix 2.13 which uses small portions of the store API.
With the switch to C++20, the rules became more strict, and we can no
longer initialize base classes. Make them comments instead.
(BTW
https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html
this offers some new syntax for this use-case. Hopefully this will be
adopted and we can eventually use it.)
I don't think the `narHash` is in need of documentation more than the
other undocumented fields, but regardless this change has nothing to do
with that field and so we should leave the comment as is.
`&` without space before is far more common on this codebase than I
thought, so it is not worth changing just this one file. Maybe we will
adopt a formatter someday but until then this is fine.
For frameworks it's important that structures are as lazy as possible
to prevent infinite recursions, performance issues and errors that
aren't related to the thing to evaluate. As a consequence, they have
to emit more attributes than strictly (sic) necessary.
However, these attributes with empty values are not useful to the user
so we omit them.
Commit 14bc3ce3d6 (0.13~43) changed the
timestamps in the Nix store from 0 to 1. Update the nix-store man
page to match.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
- `AC_LANG_PUSH(C++)` is needed for the header check
- The library check is hopeless (without lots of third-party macros I
don't feel like getting) because name mangling
Pkg-config would make all this easier. I previously opened
https://github.com/emil-e/rapidcheck/issues/302, I should write a PR
too.
The references set seems to have been unused since `LegacySSHStore`
references were first created in
caa5793b4a.
The method decls never were upstream, and accidentally added by me in
062533f7cd (probably due to `git rerere`).
Sorry!
This reduces the diff from #3746.
No other getDefaultFlakeAttrPaths implementation has this trailing dot,
and the dot can show up in error messages like:
error: flake '...' does not provide attribute 'packages.x86_64-linux.', ...
- Clarify doc comments, Installables::getCursors returns non-empty
vector
- Use vector::at in Installable::getCursor instead of checking for empty
vector and throwing an exception with error message.
Handle the case where none of getActualAttrPaths() actually exists,
in which case instead of returning an empty vector.
This fixes the case where the user misspells the attribute name in nix
search. Instead of getting no search results, now it shows an error with
suggestions.
Also remove InstallableFlake::getCursor() override since it's now
equivalent to the base class version.
Avoid needless work and throwing away invariants.
These conversions date back to when `StorePath` was in Rust and there
were issues with it missing utility methods.
Previously, getDefaultNixPath was called too early: at initialisation
time, before CLI and config have been processed, when `restrictEval` and
`pureEval` both have their default value `false`. Call it when
initialising the EvalState instead, and use `setDefault`.
clangStdenv compiles with a single warning:
```
warning: destructor called on non-final 'nix::PosAdapter' that has virtual functions but non-virtual destructor [-Wdelete-non-abstract-non-virtual-dtor]
```
This fixes the warning by making the destructor of PosAdapter virtual,
deffering to the correct destructor from the concrete child classes.
This has no impact in the end, as none of these classes have specific
destructors.
Technicaly, it may be faster not to have this indirection, but as per
the warning, there is only one place where we have to delete abstract
PosAdapter values.
Not worth bikesheding I guess.
It is still necessary.
Please do your research, or f ask the author, which happens to be me.
An evaluator like this is not an environment where "it compiles, so
it works" will ever hold.
This reverts commit 1c40182b12.
Add an `$` at the end of the `grep` regex. Without it, `checkRef foo`
would always imply `checkRef foo.drv`. We want to tell these situations
apart to more precisely test what is going on.
The issue *seems* to be the cross jobs, which are missing the `CXXFLAGS`
needed to get rapidcheck.
PR #6538 would be really nice to resurrect which will prevent the
`configureFlags` from going out of sync between the regular build and
the cross build again.
Allows checking directory entry type of a single file/directory.
This was added to optimize the use of `builtins.readDir` on some
filesystems and operating systems which cannot detect this information
using POSIX's `readdir`.
Previously `builtins.readDir` would eagerly use system calls to lookup
these filetypes using other interfaces; this change makes these
operations lazy in the attribute values for each file with application
of `builtins.readFileType`.
We had some local variables left over from the older (more
complicated) implementation of this function. They should all be unused,
but one wasn't by mistake.
Delete them all, and replace the one that was still in use as intended.
The original `builtins.getContext` test from
1d757292d0 would have caught this. The
problem is that b30be6b450 adding
`builtins.appendContext` modified that test to make it test too much at
once, rather than adding a separate test.
We now have isolated tests for both functions, and also a property test
showing everything put together (in the form of an eta rule for strings
with context). This is better coverage and properly reproduces the bug.
It's used as the “system” profile in a bunch of places, so better not
touch it. Besides, it doesn't hurt to keep it since it's owned by root
any way, so it doesn't have the `chown` problem that the user profiles
had and that led to wanting to move them on the client-side.
Rather than using `/nix/var/nix/{profiles,gcroots}/per-user/`, put the user
profiles and gcroots under `$XDG_DATA_DIR/nix/{profiles,gcroots}`.
This means that the daemon no longer needs to manage these paths itself
(they are fully handled client-side). In particular, it doesn’t have to
`chown` them anymore (removing one need for root).
This does change the layout of the gc-roots created by nix-env, and is
likely to break some stuff, so I’m not sure how to properly handle that.
Originally there was no `path-info.*`, then there was `path-info.hh`,
then there was `path-info.cc`, but only for new things. Moving this
stuff over makes everything consistent.
Among all the characters that are allowed in a URL, both the percentage
sign "%" and the single quotation mark "'" needs escaping when written
as a environment variable in a systemd service file. While the single
quotation mark may be rare, the percentage sign is widely used to escape
characters in a URL. This is especially common in proxy setting, where
username and password may contain special characters that need
percentage escaping. This patch applies the following replacements:
% -> %%
' -> \'
Instead of needing to run `nix show-config --json | jq -r
'."warn-dirty".value'` to view the value of `warn-dirty`, you can now
run `nix show-config warn-dirty`.
This should be a non-empty set, and so we don't want people doing this
by accident. We remove the zero-0 constructor with a little inheritance
trickery.
`DerivedPath::Built` and `DerivationGoal` were previously using a
regular set with the convention that the empty set means all outputs.
But it is easy to forget about this rule when processing those sets.
Using `OutputSpec` forces us to get it right.
It appears that on current macOS versions, our use of poll() to detect
client disconnects no longer works. As a workaround, poll() for
POLLRDNORM, since this *will* wake up when the client has
disconnected. The downside is that it also wakes up when input is
available. So just sleep for a bit in that case. This means that on
macOS, a client disconnect may take up to a second to be detected,
but that's better than not being detected at all.
Fixes#7584.
John has been part of every meeting since the beginning.
He took on a lot of work on behalf of the team, and provided useful suggestions in discussions, advocating for stability, reasonable design decisions, and maintainable code.
He was in general highly productive within the team process, and repeatedly helped us to keep focus on our stated goals.
Specifically, early on he suggested to gather more experience with the team reviews in order derive our values for the project encode a more structured approach to guiding contributions, which is slowly bearing fruit these days.
John is already the contributor with the most code changes to date (only topped by principal author Eelco), and is well-known to be highly knowledgeable about both high-level design and low-level internals of the code base.
He has continued to offer high quality work during the team's operation, which resulted in many pull requests getting merged that further the team's goals.
It is due time for John to be come an official team member and be granted merge access that he will surely exercise with the great care he is known for.
This way the links are clearly within the manual (ie not absolute paths),
while allowing snippets to reference the documentation root reliably,
regardless of at which base url they're included.
mdbook-linkcheck is not consistent about its warning setting.
It disables some warnings, but not the warnings about lack of
fragment checking support; hence the extra filtering.
Prior to this change, we had a bunch of ad-hoc string manipulation code
scattered around. This made it hard to figure out what data model for
string contexts is.
Now, we still store string contexts most of the time as encoded strings
--- I was wary of the performance implications of changing that --- but
whenever we parse them we do so only through the
`NixStringContextElem::parse` method, which handles all cases. This
creates a data type that is very similar to `DerivedPath` but:
- Represents the funky `=<drvpath>` case as properly distinct from the
others.
- Only encodes a single output, no wildcards and no set, for the
"built" case.
(I would like to deprecate `=<path>`, after which we are in spitting
distance of `DerivedPath` and could maybe get away with fewer types, but
that is another topic for another day.)
this form is much easier to maintain (also with minimal diffs), and
allows for more details on each operator.
this change a purely mechanical transformation, without changing any contents.
macOS doesn't have user namespacing, so the gid of the builder needs
to be nixbld. The logic got "has sandboxing enabled" confused with
"has user namespaces".
Fixes#7529.
This basically reverts 6e5165b773.
It fixes errors like
sandbox-exec: <internal init prelude>:292:47: unable to open sandbox-minimal.sb: not found
when trying to run a development Nix installed in a user's home
directory.
Also, we're trying to minimize the number of installed files
to make it possible to deploy Nix as a single statically-linked
binary.
The `fish_add_path` function is only available for fish 3.2.0 or newer,
and not on older versions.
This commit adds an alternative way to update the PATH when
`fish_add_path` does not exist.
Adds a new boolean structured attribute
`outputChecks.<output>.unsafeDiscardReferences` which disables scanning
an output for runtime references.
__structuredAttrs = true;
outputChecks.out.unsafeDiscardReferences = true;
This is useful when creating filesystem images containing their own embedded Nix
store: they are self-contained blobs of data with no runtime dependencies.
Setting this attribute requires the experimental feature
`discard-references` to be enabled.
Previously addTempRoot() acquired the LocalStore state lock and waited
for the garbage collector to reply. If the garbage collector is in the
same process (as it the case with auto-GC), this would deadlock as
soon as the garbage collector thread needs the LocalStore state lock.
So now addTempRoot() uses separate Syncs for the state that it
needs. As long at the auto-GC thread doesn't call addTempRoot() (which
it shouldn't), it shouldn't deadlock.
Fixes#3224.
This also moves the file handle into its own Sync object so we're not
holding the _state while acquiring the file lock. There was no real
deadlock risk here since locking a newly created file cannot block,
but it's still a bit nicer.
This has the same goal as b13fd4c58e81b2b2b0d72caa5ce80de861622610,but
achieves it in a different way in order to not break
`nix why-depends --derivation`.
In principle, this should avoid deadlocks where two instances of Nix are
holding a shared lock on big-lock and are both waiting to get an
exclusive lock.
However, it seems like `flock(2)` is supposed to do this automatically,
so it's not clear whether this is actually where the problem comes from.
Without the change checks issue the fllowing warning:
$ nix flake check
trace: warning: The option `nix.useSandbox' defined in `makeTest parameters' has been renamed to `nix.settings.sandbox'.
trace: warning: The option `nix.useSandbox' defined in `makeTest parameters' has been renamed to `nix.settings.sandbox'.
trace: warning: The option `nix.maxJobs' defined in `makeTest parameters' has been renamed to `nix.settings.max-jobs'.
...
This makes 'nix develop' set the Linux personality in the same way
that the actual build does, allowing a command like 'nix develop
nix#devShells.i686-linux.default' on x86_64-linux to work correctly.
Without this, the error is lost, and it makes for a hard to debug
situation. Also remove some of the busyness inside the sqlite_open_v2
args.
The errcode returned is not the extended one. The only way to make open
return an extended code, would be to add SQLITE_OPEN_EXRESCODE to the
flags. In the future it might be worth making this change,
which would also simplify the existing SQLiteError code.
First, logic is consolidated in the shell script instead of being spread
between them and makefiles. That makes understanding what is going on a
little easier.
This would not be super interesting by itself, but it gives us a way to
debug tests more easily. *That* in turn I hope is much more compelling.
See the updated manual for details.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
* doc/book.toml: Improve config
- `title` value will be added to the HTML <title> - here</title>
- `git-repository-url` adds a link to the GitHub repo in the top right corner
- `edit-url-template` adds an edit link, inviting contributions
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
This makes 'nix build' work on paths (which will be copied to the
store) and store paths (returned as is). E.g. the following flake
output attributes can be built using 'nix build .#foo':
foo = ./src;
foo = self.outPath;
foo = builtins.fetchTarball { ... };
foo = (builtins.fetchTree { .. }).outPath;
foo = builtins.fetchTree { .. } + "/README.md";
foo = builtins.storePath /nix/store/...;
Note that this is potentially risky, e.g.
foo = /.;
will cause Nix to try to copy the entire file system to the store.
What doesn't work yet:
foo = self;
foo = builtins.fetchTree { .. };
because we don't handle attrsets with an outPath attribute in it yet,
and
foo = builtins.storePath /nix/store/.../README.md;
since result symlinks have to point to a store path currently (rather
than a file inside a store path).
Fixes#7417.
They did not include the detailed error message, losing essential
information for troubleshooting.
Example message:
warning: creating statement 'insert or rplace into NARs(cache, hashPart, namePart, url, compression, fileHash, fileSize, narHash, narSize, refs, deriver, sigs, ca, timestamp, present) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)': at offset 10: SQL logic error, near "rplace": syntax error (in '/tmp/nix-shell.grQ6f7/nix-test/tests/binary-cache/test-home/.cache/nix/binary-cache-v6.sqlite')
It's not the best example; more important information will be in
the message for e.g. a constraint violation.
I don't see why this specific error is printed as a warning, but
that's for another commit.
Unsetting `build-users-group` (without `auto-allocate-uids` enabled)
gives the following error:
```
src/libstore/lock.cc:25: static std::unique_ptr<nix::UserLock> nix::SimpleUserLock::acquire(): Assertion `settings.buildUsersGroup != ""' failed.
```
Fix the logic in `useBuildUsers` and document the default value
for `build-users-group`.
This makes the position object used in exceptions abstract, with a
method getSource() to get the source code of the file in which the
error originated. This is needed for lazy trees because source files
don't necessarily exist in the filesystem, and we don't want to make
libutil depend on the InputAccessor type in libfetcher.
Make everything be in the form "while ..." (most things were already),
and in particular *don't* use other propositions that must go after or
before specific "while ..." clauses to make sense.
When debugging nix expressions the outermost trace tends to be more useful
than the innermost. It is therefore printed last to save developers from
scrolling.
We used to set enforceDeterminism to true in the settings (by default)
and thus did send a non-zero value over the wire. The value should
probably be ignored as it should only matter if nrRounds is non-zero
as well.
Having the old code here where the value is expected to be zero only
works with the same version of Nix where we are sending zero. We
should always test this against older Nix versions being client or
server as otherwise upgrade in larger networks might be a pain.
Fixes 8e0946e8df
Fix#6209
When trying to run `nix log <installable>`, try first to resolve the derivation pointed to
by `<installable>` as it is the resolved one that holds the build log.
This has a couple of shortcomings:
1. It’s expensive as it requires re-reading the derivation
2. It’s brittle because if the derivation doesn’t exist anymore or can’t
be resolved (which is the case if any one of its build inputs is missing),
then we can’t access the log anymore
However, I don’t think we can do better (at least not right now).
The alternatives I see are:
1. Copy the build log for the un-resolved derivation. But that means a
lot of duplication
2. Store the results of the resolving in the db. Which might be the best
long-term solution, but leads to a whole new class of potential
issues.
this avoids incorrect rendering on the man pages, since `lowdown`
neither parses the anchor syntax nor HTML.
this should rather be fixed in lowdown, as adding more anchors
would otherwise produce ever more noise and error-prone repetition.
* docs: Use secret-key-files when demonstrating post-build-hooks
The docs used to recommend calling `nix store sign` in a post-build
hook, but on more recent versions of nix, this results in unsigned
store paths being copied into binary caches. See
https://github.com/NixOS/nix/issues/6960 for details.
Instead, use the `secret-key-files` config option, which signs all
locally-built derivations with the private key.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
These only functioned if a very narrow combination of conditions held:
- The result path does not yet exist (--check did not result in
repeated builds), AND
- The result path is not available from any configured substituters, AND
- No remote builders that can build the path are available.
If any of these do not hold, a derivation would be built 0 or 1 times
regardless of the repeat option. Thus, remove it to avoid confusion.
being too specific about it requires more maintenance (or otherwise
produced more confusion and churn), since these points of contact change
over time.
this is a quick half-fix for command line examples, as discussed
discussed in [1].
[1]: https://github.com/NixOS/nix/pull/7389
examples which look like this
$ foo bar
baz
are confusing for Unix shell beginners, because it's hard to discern
what is supposed to be entered into the actual command line when the
convention of prefixing `$` is not known, as barely any real-world shell
looks that way any more.
this change prevents selecting the prompt part with the mouse in the
HTML representation of the Nix manual.
it does not prevent selecting the output part of the shell example.
it also does not address that the copy button provided by mdBook takes
the entire sample, including the prompts, into the clipboard.
The old way was not correct.
Here is an example:
```
$ nix-instantiate --eval --expr 'let x = a: throw "asdf"; in x 1' --show-trace
error: asdf
… while evaluating 'x'
at «string»:1:9:
1| let x = a: throw "asdf"; in x 1
| ^
… from call site
at «string»:1:29:
1| let x = a: throw "asdf"; in x 1
| ^
```
and yet also:
```
$ nix-instantiate --eval --expr 'let x = a: throw "asdf"; in x' --show-trace
<LAMBDA>
```
Here is the thing: in both cases we are evaluating `x`!
Nix is a higher-order languages, and functions are a sort of value. When
we write `x = a: ...`, `a: ...` is the expression that `x` is being
defined to be, and that is already a value. Therefore, we should *never*
get an trace that says "while evaluating `x`", because evaluating `a:
...` is *trival* and nothing happens during it!
What is actually happening here is we are applying `x` and evaluating
its *body* with arguments substituted for parameters. I think the
simplest way to say is just "while *calling* `x`", and so that is what I
changed it to.
We need to close the GC server socket before shutting down the active
GC client connections, otherwise a client may (re)connect and get
ECONNRESET. But also handle ECONNRESET for resilience.
Fixes random failures like
GC socket disconnected
connecting to '/tmp/nix-shell.y07M0H/nix-test/default/var/nix/gc-socket/socket'
sending GC root '/tmp/nix-shell.y07M0H/nix-test/default/store/kb5yzija0f1x5xkqkgclrdzldxj6nnc6-non-blocking'
reading GC root from client: error: unexpected EOF reading a line
1 store paths deleted, 0.00 MiB freed
error: reading from file: Connection reset by peer
in gc-non-blocking.sh.
It calls strlen() on the input (rather than simply copying at most
`size` bytes), which can fail if the input is not zero-terminated and
is inefficient in any case.
Fixes#7347.
why-depends assumed that we knew the output path of the second argument.
For CA derivations, we might not know until it's built. One way to solve
this would be to build the second installable to get the output path.
In this case we don't need to, though. If the first installable (A)
depends on the second (B), then getting the store path of A will
necessitate having the store path B. The contrapositive is, if the store
path of B is not known (i.e. it's a CA derivation which hasn't been
built), then A does not depend on B.
We shouldn't skip this if the supplementary group list is empty,
because then the sandbox won't drop the supplementary groups of the
parent (like "root").
The new experimental feature 'cgroups' enables the use of cgroups for
all builds. This allows better containment and enables setting
resource limits and getting some build stats.
It occurred when a output of the dependency was already available,
so it didn't need rebuilding and didn't get added to the
inputDrvOutputs.
This process-related info wasn't suitable for the purpose of finding
the actual input paths for the builder. It is better to do this in
absolute terms by querying the store.
This change is needed to support aws-sdk-cpp 1.10 and newer.
I opted not to make this dependent on the sdk version because
the crt dependency has been in the interface of the older
sdk as well, and it was only coincidence that libstore didn't
make use of any privately defined symbols directly.
When calling `builtins.readFile` on a store path, the references of that
path are currently added to the resulting string's context.
This change makes those references the *possible* context of the string,
but filters them to keep only the references whose hash actually appears
in the string, similarly to what is done for determining the runtime
references of a path.
Cgroups are now only used for derivations that require the uid-range
range feature. This allows auto UID allocation even on systems that
don't have cgroups (like macOS).
Also, make things work on modern systems that use cgroups v2 (where
there is a single hierarchy and no "systemd" controller).
after discussing this with multiple people, I'm convinced that "build
task" is more precise: a derivation is not an action, but inert until it
is built. also it's easier to pronounce.
proposal: use "build task" for the generic concept "description of how
to derive new files from the contents of existing files". then it will
be easier to distinguish what we mean by "derivation" (a specific data
structure and Nix language value type) and "store derivation" (a
serialisation of a derivation into a file in the Nix store).
readline is not re-entrant, so entering the debugger from the
completioncallback results in an eventual segfault.
The workaround is to temporarily disable the debugger when searching
for possible completions.
In addition to consistency, the fancy "Copy to clipboard" button on the
website will copy the prompt character. Retaining the prompt character
would mean having to edit each command after pasting in the terminal.
Call it as `['nix', '__build-remote', ... ]` rather than the previous
`["__build-remote", "nix __build-remote", ... ]` which seemed to have
been most likely unintended
The description of the --profile option talks about the "update" operation.
This is probably meant for operations such as "nix profile install", but the
same option is reused in other subcommands, which do not update the profile,
such as "nix profile {list,history,diff-closures}".
We update the description to make sense in both contexts.
Currently, Nix passes `-a` when it runs commands on a remote machine via
SSH, which disables agent forwarding. This causes issues when the
`ForwardAgent` option is set in SSH config files, as the command line
operation always overrides those.
In particular, this causes issues if the command being run is `sudo`
and the remote machine is configured with the equivalent of NixOS's
`security.pam.enableSSHAgentAuth` option. Not allowing SSH agent
forwarding can cause authentication to fail unexpectedly.
This can currently be worked around by setting `NIX_SSHOPTS="-A"`, but
we should defer to the options in the SSH config files to be least
surprising for users.
This is a really old test case (which was originally written before the
proper Nix syntax). The tested deep comparison behavior was implemented
and reverted soon after due to performance problems, but it has been
restored in today's Nix again (thanks to the derivation comparison
optimization, presumably).
Specifically, explain why Nix does not _re_evaluate paths during a
`nix repl` session. This is a thing that bit me while playing around
with paths and antiquotation in `nix repl` while reading the Nix
language tutorial at https://nix.dev/tutorials/nix-language.
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
* Clarify the documentation of foldl': That the arguments are forced
before application (?) of `op` is necessarily true. What is important
to stress is that we force every application of `op`, even when the
value turns out to be unused.
* Move the example before the comment about strictness to make it less
confusing: It is a general example and doesn't really showcase anything
about foldl' strictness.
* Add test cases which nail down aspects of foldl' strictness:
* The initial accumulator value is not forced unconditionally.
* Applications of op are forced.
* The list elements are not forced unconditionally.
After we've send "\2\n" to the parent, we can't send a serialized
exception anymore. It will show up garbled like
$ nix-build --store /tmp/nix --expr 'derivation { name = "foo"; system = "x86_64-linux"; builder = "/foo/bar"; }'
this derivation will be built:
/nix/store/xmdip0z5x1zqpp6gnxld3vqng7zbpapp-foo.drv
building '/nix/store/xmdip0z5x1zqpp6gnxld3vqng7zbpapp-foo.drv'...
ErrorErrorEexecuting '/foo/bar': No such file or directory
error: builder for '/nix/store/xmdip0z5x1zqpp6gnxld3vqng7zbpapp-foo.drv' failed with exit code 1
While trying to use an alternate directory for my Nix installation, I
noticed that nix's output didn't reflect the updated state
directory. This patch corrects that and now prints the warning before
attempting to create the directory (if the directory creation fails,
it wouldn't have been obvious why nix was attempting to create the
directory in the first place).
With this patch, I now get the following warning:
warning: '/home/deck/.var/app/org.nixos.nix/var/nix' does not
exist, so Nix will use '/home/deck/.local/share/nix/root' as a
chroot store
This commit adds a test covering the discrepancy between parseDrvName's
implementation and documentation (the discrepancy was eliminated in the previous
commit).
The documentation for `parseDrvName` does not agree with the implementation when
the derivation name contains a dash which is followed by something that is
neither a letter nor a digit. This commit corrects the documentation to agree
with the implementation.
Previously the MANPATH was set even if MANPATH was empty beforehand
which resulted in a MANPATH of only ~/.nix-profile/share/man which
omitted the default man page directory (commonly /opt/local/share/man)
from man page results.
If we don't have any github token, we won't be able to fetch private
repos, but we are also more likely to run into API limits since
we don't have a token. To mitigate this only ever use the github api
if we actually have a token.
Older versions of Fish (such as those bundled with Ubuntu LTS 22.04) do
not support return outside of functions. We need to use the equivalent
exit instead.
The current definition of `intersectAttrs` is incorrect:
> Return a set consisting of the attributes in the set e2 that also exist in the
> set e1.
Recall that (Nix manual, section 5.1):
> An attribute set is a collection of name-value-pairs (called attributes)
According to the existing description of `intersectAttrs`, the following should
evaluate to the empty set, since no key-value *pair* (i.e. attribute) exists in
both sets:
```
builtins.intersectAttrs { x=3; } {x="foo";}
```
And yet:
```
nix-repl> builtins.intersectAttrs { x=3; } {x="foo";}
{ x = "foo"; }
```
Clearly the intent here was for the *names* of the resulting attribute set to be
the intersection of the *names* of the two arguments, and for the values of the
resulting attribute set to be the values from the second argument.
This commit corrects the definition, making it match the implementation and intent.
Make sure that people who run Nix in non-interactive mode (and so don't have the possibility to interactively accept the individual flake configuration settings) are aware of this flag.
Fix#7086
These settings seem harmless, they control the same polling
functionality that timeout does, but with different behavior. Should
be safe for untrusted users to pass in.
I just had a colleague get confused by the previous phrase for good
reason. "valid" sounds like an *objective* criterion, e.g. and *invalid
signature* would be one that would be trusted by no one, e.g. because it
misformatted or something.
What is actually going is that there might be a signature which is
perfectly valid to *someone else*, but not to the user, because they
don't trust the corresponding public key. This is a *subjective*
criterion, because it depends on the arbitrary and personal choice of
which public keys to trust.
I therefore think "trustworthy" is a better adjective to use. Whether
something is worthy of trust is clearly subjective, and then "trust"
within that word nicely evokes `trusted-public-keys` and friends.
The history is not critical to the functionality of nix repl, so it's
enough to warn here, rather than refuse to start if the directory Nix
thinks the history should live in can't be created.
- call close explicitly in writeFile to prevent the close exception
from being ignored
- fsync after writing schema file to flush data to disk
- fsync schema file parent to flush metadata to disk
https://github.com/NixOS/nix/issues/7064
Remove the `verify TLS: Nix CA file = 'blah'` message that Nix used to print when fetching anything as it's both useless (`libcurl` prints the same info in its logs) and misleading (gives the impression that a new TLS connection is being established which might not be the case because of multiplexing. See #7011 )
Mainly:
- Try to triangulate between narrative that framed this as
a new/easy process and the need for a reference that will
not quickly grow stale.
- Fix a ~continuity issue where the text was talking about
"your Cachix cache" before saying that you'd need to make
a Cachix cache to enable the installer tests.
- Adopt suggestion on titling, and nest subtitles in the
installer test section.
This commit adds an optional `__impure` parameter to fetchurl.nix, which allows
the caller to use `libfetcher`'s fetcher in an impure derivation. This allows
nixpkgs' patch-normalizing fetcher (fetchpatch) to be rewritten to use nix's
internal fetchurl, thereby eliminating the awkward "you can't use fetchpatch
here" banners scattered all over the place.
See also: https://github.com/NixOS/nixpkgs/pull/188587
This runs the installer in a QEMU VM. Unlike the old installer test
that ran inside a declaratively built RedHat/Debian image, this uses
an image from Vagrant.
Before this patch, installing Nix using the Fish shell did not
work because Fish wasn't configured to add Nix to the PATH. Some
options in #1512 offered workarounds, but they typically involve
extra plugins or packages.
This patch adds native, out-of-the-box support for the Fish shell.
Note that Fish supports a `conf.d` directory, which is intended
for exactly use cases like this: software projects distributing
shell snippets. This patch takes advantage of it. The installer
doesn't append any Nix loader behavior to any Fish config file.
Because of that, the uninstall process is smooth and a reinstall
obliterates the existing nix.fish files that we place instead of
bothering the user with a backup / manual removal.
Both single-user and multi-user cases are covered. It has been
tested on Ubuntu, and a Mac with MacPorts, homebrew, and the
Fish installer pkg.
Closes#1512
Co-authored-by: Graham Christensen <graham@grahamc.com>
Implements the approach suggested by feedback on PR #6994, where
tempdir paths are created in the store (now with an exclusive lock).
As part of this work, the currently-broken and unused
`createTempDirInStore` function is updated to create an exclusive lock
on the temp directory in the store.
The GC now makes a non-blocking attempt to lock any store directories
that "look like" the temp directories created by this function, and if
it can't acquire one, ignores the directory.
Stdenv sets this to a bash that doesn't have readline/completion
support, so running 'nix (develop|shell)' inside a 'nix develop' gives
you a crippled shell. So let's just ignore the derivation's $SHELL.
This could break interactive use of build phases that use $SHELL, but
they appear to be fairly rare.
renaming section headers and changing manually set `id`s will break URLs
in the wild.
this change allows keeping track of all changes to ensure backwards
compatibility.
Disables the SA_RESTART behavior on macOS which causes:
> Restarting of pending calls is requested by setting the SA_RESTART bit
> in sa_flags. The affected system calls include read(2), write(2),
> sendto(2), recvfrom(2), sendmsg(2) and recvmsg(2) on a communications
> channel or a slow device (such as a terminal, but not a regular file)
> and during a wait(2) or ioctl(2).
From: https://man.openbsd.org/sigaction#SA_RESTART
This being set on macOS caused a bug where read() calls to the daemon
socket were blocking after a SIGINT was received. As a result,
checkInterrupt was never reached even though the signal was received
by the signal handler thread.
On Linux, SA_RESTART is disabled by default. This probably effects
other BSDs but I don’t have the ability to test it there right now.
readDerivation is pretty slow, and while it may not be significant for
some use cases, on things like ghc-nix where we have thousands of
derivations is really slows things down.
So, this just doesn’t do the impure derivation check if the impure
derivation experimental feature is disabled. Perhaps we could cache
the result of isPure() and keep the check, but this is a quick fix to
for the slowdown introduced with impure derivations features in 2.8.0.
This matches the behavior of bash. We don’t want to add a space after
completion on attrs. Uses -S.
Switches to new compadd style comppletions instead of _describe.
Shouldn’t have any negative issues from what I can tell.
The darwin_stop_world implementation is slightly different. sp goes to
altstack_lo instead of lo in this case. Assuming that is an
implementation detail.
But the fix is the same, when we detect alstack_lo outside of the
expected stack range, we reset it to hi - stack_limit.
Here stack_limit is calculated with pthread_get_stacksize_np since
that is the BSD equivalent to pthread_attr_getstacksize.
A [recent-ish change](https://github.com/NixOS/nix/pull/6676) logs a warning when a potentially counterintuitive situation happens.
This now causes the multi-user installer to [emit a warning](https://github.com/NixOS/nixpkgs/issues/189043) when it's doing
the "seed the Nix database" step via a low-level `nix-store --load-db` invocation.
`nix-store` functionality implementations don't actually use profiles or channels or homedir as far as i can tell. So why are we
hitting this code at all?
Well, the current command approach for functionality here builds a [fat `nix` binary](https://github.com/NixOS/nix/blob/master/src/nix/local.mk#L23-L26) which has _all_ the functionality of
previous individual binaries (nix-env, nix-store, etc) bundled in, then [uses the invocation name](https://github.com/NixOS/nix/blob/master/src/nix/main.cc#L274-L277) to select the
set of commands to expose. `nix` itself has this behavior, even when just trying to parse the (sub)command and arguments:
```
dave @ davembp2
$ nix
error: no subcommand specified
Try 'nix --help' for more information.
dave @ davembp2
$ sudo nix
warning: $HOME ('/Users/dave') is not owned by you, falling back to the one defined in the 'passwd' file
error: no subcommand specified
Try 'nix --help' for more information.
dave @ davembp2
$ HOME=~root sudo nix
error: no subcommand specified
Try 'nix --help' for more information.
```
This behavior can also be seen pretty easily with an arbitrary `nix-store` invocation:
```
dave @ davembp2
$ nix-store --realize
dave @ davembp2
$ sudo nix-store --realize # what installer is doing now
warning: $HOME ('/Users/dave') is not owned by you, falling back to the one defined in the 'passwd' file
dave @ davembp2
$ sudo HOME=~root nix-store --realize # what this PR effectively does
dave @ davembp2
$
```
this simplifies the setup a lot, and avoids weird looking `./file.md`
links showing up.
it also does not show regular URLs any more. currently the command
reference only has few of them, and not showing them in the offline
documentation is hopefully not a big deal.
instead of building more special-case solutions, clumsily preprocessing
the input, or issuing verbal rules on dealing with URLs, should better
be solved sustainably by not rendering relative links in `lowdown`:
https://github.com/kristapsdz/lowdown/issues/105
Nix veterans intuitively know what the following terms mean. They are
used in several places in the nix documentation, but never defined:
- local store
- remote store
- binary cache
- substituter
In particular, I found the last two terms to be confusingly similar.
Let's give definitions for them.
A mips64el Linux MIPS kernel can execute userspace code using any of
three ABIs:
mips64el-linux-*abin64
mips64el-linux-*abin32
mipsel-linux-*
The first of these is the native 64-bit ABI, and the only ABI with
64-bit pointers; this is sometimes called "n64". The last of these is
the old legacy 32-bit ABI, whose binaries can execute natively on
32-bit MIPS hardware; this is sometimes called "o32".
The second ABI, "n32" is essentially the 64-bit ABI with 32-bit
pointers and address space. Hardware 64-bit integer/floating
arithmetic is still allowed, as well as the much larger mips64
register set and more-efficient calling convention.
Let's enable seccomp filters for all of these. Likewise for big
endian (mips64-linux-*).
In particular, this means that derivations can output derivations. But
that ramification isn't (yet!) useful as we would want, since there is
no way to have a dependent derivation that is itself a dependent
derivation.
"uid-range" provides 65536 UIDs to a build and runs the build as root
in its user namespace. "systemd-cgroup" allows the build to mount the
systemd cgroup controller (needed for running systemd-nspawn and NixOS
containers).
Also, add a configuration option "auto-allocate-uids" which is needed
to enable these features, and some experimental feature gates.
So to enable support for containers you need the following in
nix.conf:
experimental-features = auto-allocate-uids systemd-cgroup
auto-allocate-uids = true
system-features = uid-range systemd-cgroup
2^18 was overkill. The idea was to enable multiple containers to run
inside a build. However, those containers can use the same UID range -
we don't really care about perfect isolation between containers inside
a build.
Also, run builds in a cgroup namespace (ensuring /proc/self/cgroup
doesn't leak information about the outside world) and mount /sys. This
enables running systemd-nspawn and thus NixOS containers in a Nix
build.
Rather than rely on a nixbld group, we now allocate UIDs/GIDs
dynamically starting at a configurable ID (872415232 by default).
Also, we allocate 2^18 UIDs and GIDs per build, and run the build as
root in its UID namespace. (This should not be the default since it
breaks some builds. We probably should enable this conditional on a
requiredSystemFeature.) The goal is to be able to run (NixOS)
containers in a build. However, this will also require some cgroup
initialisation.
The 2^18 UIDs/GIDs is intended to provide enough ID space to run
multiple containers per build, e.g. for distributed NixOS tests.
Please include relevant [release notes](https://github.com/NixOS/nix/blob/master/doc/manual/src/release-notes/rl-next.md) as needed.
**Testing**
If this issue is a regression or something that should block release, please consider including a test either in the [testsuite](https://github.com/NixOS/nix/tree/master/tests) or as a [hydraJob]( https://github.com/NixOS/nix/blob/master/flake.nix#L396) so that it can be part of the [automatic checks](https://hydra.nixos.org/jobset/nix/master).
Welcome and thank you for your interest in contributing to Nix!
We appreciate your support.
Reading and following these guidelines will help us make the contribution process easy and effective for everyone involved.
## Report a bug
1. Check on the [GitHub issue tracker](https://github.com/NixOS/nix/issues) if your bug was already reported.
2. If you were not able to find the bug or feature [open a new issue](https://github.com/NixOS/nix/issues/new/choose)
3. The issue templates will guide you in specifying your issue.
The more complete the information you provide, the more likely it can be found by others and the more useful it is in the future.
Make sure reported bugs can be reproduced easily.
4. Once submitted, do not expect issues to be picked up or solved right away.
The only way to ensure this, is to [work on the issue yourself](#making-changes-to-nix).
## Report a security vulnerability
Check out the [security policy](https://github.com/NixOS/nix/security/policy).
## Making changes to Nix
1. Check for [pull requests](https://github.com/NixOS/nix/pulls) that might already cover the contribution you are about to make.
There are many open pull requests that might already do what you intent to work on.
You can use [labels](https://github.com/NixOS/nix/labels) to filter for relevant topics.
2. Search for related issues that cover what you're going to work on. It could help to mention there that you will work on the issue.
Issues labeled ["good first issue"](https://github.com/NixOS/nix/labels/good-first-issue) should be relatively easy to fix and are likely to get merged quickly.
Pull requests addressing issues labeled ["idea approved"](https://github.com/NixOS/nix/labels/idea%20approved) are especially welcomed by maintainers and will receive prioritised review.
3. Check the [Nix reference manual](https://nixos.org/manual/nix/unstable/contributing/hacking.html) for information on building Nix and running its tests.
For contributions to the command line interface, please check the [CLI guidelines](https://nixos.org/manual/nix/unstable/contributing/cli-guideline.html).
4. Make your changes!
5. [Create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) for your changes.
* [Mark the pull request as draft](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/changing-the-stage-of-a-pull-request) if you're not done with the changes.
* Make sure to have [a clean history of commits on your branch by using rebase](https://www.digitalocean.com/community/tutorials/how-to-rebase-and-update-a-pull-request).
* Link related issues in your pull request to inform interested parties and future contributors about your change.
If your pull request closes one or multiple issues, note that in the description using `Closes: #<number>`, as it will then happen automatically when your change is merged.
6. Do not expect your pull request to be reviewed immediately.
Nix maintainers follow a [structured process for reviews and design decisions](https://github.com/NixOS/nix/tree/master/maintainers#project-board-protocol), which may or may not prioritise your work.
7. If you need additional feedback or help to getting pull request into shape, ask other contributors using [@mentions](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#mentioning-people-and-teams).
## Making changes to the Nix manual
The Nix reference manual is hosted on https://nixos.org/manual/nix.
The underlying source files are located in [`doc/manual/src`](./doc/manual/src).
For small changes you can [use GitHub to edit these files](https://docs.github.com/en/repositories/working-with-files/managing-files/editing-files)
For larger changes see the [Nix reference manual](https://nixos.org/manual/nix/unstable/contributing/hacking.html).
## Getting help
Whenever you're stuck or do not know how to proceed, you can always ask for help.
The appropriate channels to do so can be found on the [NixOS Community](https://nixos.org/community/) page.
building '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' (round 1/2)...
building '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' (round 2/2)...
output '/nix/store/6xg356v9gl03hpbbg8gws77n19qanh02-unstable' of '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' differs from '/nix/store/6xg356v9gl03hpbbg8gws77n19qanh02-unstable.check' from previous round
> If you are building via the Nix daemon, it is the Nix daemon user
>account (that is, `root`) that should have SSH access to the remote
> machine. If you can’t or don’t want to configure `root` to be able to
> access to remote machine, you can use a private Nix store instead by
> passing e.g. `--store ~/my-nix`.
> If you are building via the Nix daemon, it is the Nix daemon user account (that is, `root`) that should have SSH access to a user (not necessarily `root`) on the remote machine.
>
> If you can’t or don’t want to configure `root` to be able to access the remote machine, you can use a private Nix store instead by passing e.g. `--store ~/my-nix` when running a Nix command from the local machine.
The list of remote machines can be specified on the command line or in
the Nix configuration file. The former is convenient for testing. For
Then, add the public key and the cache URL to your `nix.conf`'s
`trusted-public-keys` and `substituters` options:
Then update [`nix.conf`](../command-ref/conf-file.md) on any machine that will access the cache.
Add the cache URL to [`substituters`](../command-ref/conf-file.md#conf-substituters) and the public key to [`trusted-public-keys`](../command-ref/conf-file.md#conf-trusted-public-keys):
Machines that build for the cache must sign derivations using the private key.
On those machines, add the path to the key file to the [`secret-key-files`](../command-ref/conf-file.md#conf-secret-key-files) field in their [`nix.conf`](../command-ref/conf-file.md):
secret-key-files = /etc/nix/key.private
We will restart the Nix daemon in a later step.
# Implementing the build hook
@@ -52,14 +57,12 @@ set -eu
set -f # disable globbing
exportIFS=' '
echo"Signing paths"$OUT_PATHS
nix store sign --key-file /etc/nix/key.private $OUT_PATHS
The following [concept map] shows its main components (rectangles), the objects they operate on (rounded rectangles), and their interactions (connecting phrases):
At the top is the [command line interface](../command-ref/command-ref.md), translating from invocations of Nix executables to interactions with the underlying layers.
At the top is the [command line interface](../command-ref/command-ref.md) that drives the underlying layers.
Below that is the [Nix expression language](../expressions/expression-language.md), a [purely functional][purely-functional-programming] configuration language.
It is used to compose expressions which ultimately evaluate to self-contained *build plans*, used to derive *build results* from referenced *build inputs*.
The [Nix language](../language/index.md) evaluator transforms Nix expressions into self-contained *build plans*, which are used to derive *build results* from referenced *build inputs*.
The command line and Nix language are what users interact with most.
The command line interface and Nix expressions are what users deal with most.
> **Note**
> The Nix language itself does not have a notion of *packages* or *configurations*.
> As far as we are concerned here, the inputs and results of a build plan are just data.
Underlying these is the [Nix store](./store/store.md), a mechanism to keep track of build plans, data, and references between them.
It can also execute build plans to produce new data.
Underlying the command line interface and the Nix language evaluator is the [Nix store](../glossary.md#gloss-store), a mechanism to keep track of build plans, data, and references between them.
It can also execute build plans to produce new data, which are made available to the operating system as files.
A build plan is a series of *build tasks*.
Each build task has a special build input which is used as *build instructions*.
A build plan itself is a series of *build tasks*, together with their build inputs.
> **Important**
> A build task in Nix is called [derivation](../glossary.md#gloss-derivation).
Each build task has a special build input executed as *build instructions* in order to perform the build.
The result of a build task can be input to another build task.
The following [data flow diagram] shows a build plan for illustration.
Build inputs used as instructions to a build task are marked accordingly:
Nix does not assign any semantics to symbolic links.
File system objects and their children form a tree.
A bare file or symlink can be a root file system object.
Nix does not encode any other file system notions such as [hard links](https://en.m.wikipedia.org/wiki/Hard_link), [permissions](https://en.m.wikipedia.org/wiki/File-system_permissions), timestamps, or other metadata.
## Examples of file system objects
A plain file:
```
50 B, executable: false
```
An executable file:
```
122 KB, executable: true
```
A symlink:
```
-> /usr/bin/sh
```
A directory with contents:
```
├── bin
│ └── hello: 35 KB, executable: true
└── share
├── info
│ └── hello.info: 36 KB, executable: false
└── man
└── man1
└── hello.1.gz: 790 B, executable: false
```
A directory that contains a symlink and other directories:
If the store has a [file system representation](./store.md#files-and-processes), this directory contains the store’s [file system objects](#file-system-object), which can be addressed by [store paths](#store-path).
This means a store path is not just derived from the referenced store object itself, but depends on the store the store object is in.
> **Note**
> The store directory defaults to `/nix/store`, but is in principle arbitrary.
It is important which store a given store object belongs to:
Files in the store object can contain store paths, and processes may read these paths.
Nix can only guarantee [referential integrity](store/closure.md) if store paths do not cross store boundaries.
Therefore one can only copy store objects to a different store if
- the source and target stores' directories match
or
- the store object in question has no references, that is, contains no store paths.
One cannot copy a store object to a store with a different store directory.
Instead, it has to be rebuilt, together with all its dependencies.
It is in general not enough to replace the store directory string in file contents, as this may render executables unusable by invalidating their internal offsets or checksums.
# Digest
In a [store path](#store-path), the [digest][digest] is the output of a [cryptographic hash function][hash] of either all *inputs* involved in building the referenced store object or its actual *contents*.
Store objects are therefore said to be either [input-addressed](#input-addressing) or [content-addressed](#content-addressing).
> **Historical Note**
> The 20 byte restriction is because originally digests were [SHA-1][sha-1] hashes.
> Nix now uses [SHA-256][sha-256], and longer hashes are still reduced to 20 bytes for compatibility.
When a new store object is built, Nix scans its file contents for store paths to construct its set of references.
The special format of a store path's [digest](#digest) allows reliably detecting it among arbitrary data.
Nix uses the [closure](store.md#closure) of build inputs to derive the list of allowed store paths, to avoid false positives.
This way, scanning files captures run time dependencies without the user having to declare them explicitly.
Doing it at build time and persisting references in the store object avoids repeating this time-consuming operation.
> **Note**
> In practice, it is sometimes still necessary for users to declare certain dependencies explicitly, if they are to be preserved in the build result's closure.
This depends on the specifics of the software to build and run.
>
> For example, Java programs are compressed after compilation, which obfuscates any store paths they may refer to and prevents Nix from automatically detecting them.
## Input Addressing
Input addressing means that the digest derives from how the store object was produced, namely its build inputs and build plan.
To compute the hash of a store object one needs a deterministic serialisation, i.e., a binary string representation which only changes if the store object changes.
Nix has a custom serialisation format called Nix Archive (NAR)
Store object references of this sort can *not* be validated from the content of the store object.
Rather, a cryptographic signature has to be used to indicate that someone is vouching for the store object really being produced from a build plan with that digest.
## Content Addressing
Content addressing means that the digest derives from the store object's contents, namely its file system objects and references.
If one knows content addressing was used, one can recalculate the reference and thus verify the store object.
Content addressing is currently only used for the special cases of source files and "fixed-output derivations", where the contents of a store object are known in advance.
Content addressing of build results is still an [experimental feature subject to some restrictions](https://github.com/tweag/rfcs/blob/cas-rfc/rfcs/0062-content-addressed-paths.md).
Store objects can be build inputs, build results, or build tasks.
Store objects are [immutable][immutable-object]: once created, they do not change until they are deleted.
## Reference
A store object reference is an [opaque][opaque-data-type], [unique identifier][unique-identifier]:
The only way to obtain references is by adding or building store objects.
A reference will always point to exactly one store object.
## Operations
A Nix store can *add*, *retrieve*, and *delete* store objects.
[ data ]
|
V
[ store ] ---> add ----> [ store' ]
|
V
[ reference ]
<!-- -->
[ reference ]
|
V
[ store ] ---> get
|
V
[ store object ]
<!-- -->
[ reference ]
|
V
[ store ] --> delete --> [ store' ]
It can *perform builds*, that is, create new store objects by transforming build inputs into build outputs, using instructions from the build tasks.
[ reference ]
|
V
[ store ] --> build --(maybe)--> [ store' ]
|
V
[ reference ]
As it keeps track of references, it can [garbage-collect][garbage-collection] unused store objects.
[ store ] --> collect garbage --> [ store' ]
## Files and Processes
Nix maps between its store model and the [Unix paradigm][unix-paradigm] of [files and processes][file-descriptor], by encoding immutable store objects and opaque identifiers as file system primitives: files and directories, and paths.
That allows processes to resolve references contained in files and thus access the contents of store objects.
Store objects are therefore implemented as the pair of
All of these systems share features of [declarative programming][declarative-programming] languages, a key insight first put forward by Eelco Dolstra et al. in [Imposing a Memory Management Discipline on Software Deployment][immdsd] (2004), elaborated in his PhD thesis [The Purely Functional Software Deployment Model][phd-thesis] (2006), and further refined by Andrey Mokhov et al. in [Build Systems à la Carte][bsalc] (2018).
Nix stores ensure [referential integrity][referential-integrity]: for each store object in the store, all the store objects it references must also be in the store.
The set of all store objects reachable by following references from a given initial set of store objects is called a *closure*.
Adding, building, copying and deleting store objects must be done in a way that preserves referential integrity:
- A newly added store object cannot have references, unless it is a build task.
- Build results must only refer to store objects in the closure of the build inputs.
Building a store object will add appropriate references, according to the build task.
- Store objects being copied must refer to objects already in the destination store.
Recursive copying must either proceed in dependency order or be atomic.
- We can only safely delete store objects which are not reachable from any reference still in use.
<!-- more details in section on garbage collection, link to it once it exists -->
By default Nix reads settings from the following places:
Nix supports a variety of configuration settings, which are read from configuration files or taken as command line flags.
- The system-wide configuration file `sysconfdir/nix/nix.conf` (i.e.
`/etc/nix/nix.conf` on most systems), or `$NIX_CONF_DIR/nix.conf` if
`NIX_CONF_DIR` is set. Values loaded in this file are not forwarded
to the Nix daemon. The client assumes that the daemon has already
loaded them.
## Configuration file
- If `NIX_USER_CONF_FILES` is set, then each path separated by `:`
will be loaded in reverse order.
By default Nix reads settings from the following places, in that order:
Otherwise it will look for `nix/nix.conf` files in `XDG_CONFIG_DIRS`
and `XDG_CONFIG_HOME`. If unset, `XDG_CONFIG_DIRS` defaults to
`/etc/xdg`, and `XDG_CONFIG_HOME` defaults to `$HOME/.config`
as per [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html).
1. The system-wide configuration file `sysconfdir/nix/nix.conf` (i.e. `/etc/nix/nix.conf` on most systems), or `$NIX_CONF_DIR/nix.conf` if [`NIX_CONF_DIR`](./env-common.md#env-NIX_CONF_DIR) is set.
- If `NIX_CONFIG` is set, its contents is treated as the contents of
a configuration file.
Values loaded in this file are not forwarded to the Nix daemon.
The client assumes that the daemon has already loaded them.
The configuration files consist of `name = value` pairs, one per
line. Other files can be included with a line like `include path`,
where *path* is interpreted relative to the current conf file and a
missing file is an error unless `!include` is used instead. Comments
start with a `#` character. Here is an example configuration file:
1. If [`NIX_USER_CONF_FILES`](./env-common.md#env-NIX_USER_CONF_FILES) is set, then each path separated by `:` will be loaded in reverse order.
keep-outputs = true # Nice for developers
keep-derivations = true # Idem
Otherwise it will look for `nix/nix.conf` files in `XDG_CONFIG_DIRS` and [`XDG_CONFIG_HOME`](./env-common.md#env-XDG_CONFIG_HOME).
If unset, `XDG_CONFIG_DIRS` defaults to `/etc/xdg`, and `XDG_CONFIG_HOME` defaults to `$HOME/.config` as per [XDG Base Directory Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html).
You can override settings on the command line using the `--option`
flag, e.g. `--option keep-outputs false`. Every configuration setting
also has a corresponding command line flag, e.g. `--max-jobs 16`; for
Boolean settings, there are two flags to enable or disable the setting
(e.g. `--keep-failed` and `--no-keep-failed`).
1. If [`NIX_CONFIG`](./env-common.md#env-NIX_CONFIG) is set, its contents are treated as the contents of a configuration file.
A configuration setting usually overrides any previous value. However,
you can prefix the name of the setting by `extra-` to *append* to the
previous value. For instance,
### File format
substituters = a b
extra-substituters = c d
Configuration files consist of `name = value` pairs, one per line.
Comments start with a `#` character.
defines the `substituters` setting to be `a b c d`. This is also
available as a command line flag (e.g. `--extra-substituters`).
Example:
The following settings are currently available:
```
keep-outputs = true # Nice for developers
keep-derivations = true # Idem
```
Other files can be included with a line like `include <path>`, where `<path>` is interpreted relative to the current configuration file.
A missing file is an error unless `!include` is used instead.
A configuration setting usually overrides any previous value.
However, for settings that take a list of items, you can prefix the name of the setting by `extra-` to *append* to the previous value.
For instance,
```
substituters = a b
extra-substituters = c d
```
defines the `substituters` setting to be `a b c d`.
Unknown option names are not an error, and are simply ignored with a warning.
## Command line flags
Configuration options can be set on the command line, overriding the values set in the [configuration file](#configuration-file):
- Every configuration setting has corresponding command line flag (e.g. `--max-jobs 16`).
Boolean settings do not need an argument, and can be explicitly disabled with the `no-` prefix (e.g. `--keep-failed` and `--no-keep-failed`).
Unknown option names are invalid flags (unless there is already a flag with that name), and are rejected with an error.
- The flag `--option <name> <value>` is interpreted exactly like a `<name> = <value>` in a setting file.
Unknown option names are ignored with a warning.
The `extra-` prefix is supported for settings that take a list of items (e.g. `--extra-trusted users alice` or `--option extra-trusted-users alice`).
If `NIX_PATH` is not set at all, Nix will fall back to the following list in [impure](@docroot@/command-ref/conf-file.md#conf-pure-eval) and [unrestricted](@docroot@/command-ref/conf-file.md#conf-restrict-eval) evaluation mode:
will cause Nix to look for paths relative to `/home/eelco/Dev` and
`/etc/nixos`, in this order. It is also possible to match paths
The source for the default [Nix expressions](@docroot@/language/index.md) used by [`nix-env`]:
-`~/.nix-defexpr`
-`$XDG_STATE_HOME/nix/defexpr` if [`use-xdg-base-directories`] is set to `true`.
It is loaded as follows:
- If the default expression is a file, it is loaded as a Nix expression.
- If the default expression is a directory containing a `default.nix` file, that `default.nix` file is loaded as a Nix expression.
- If the default expression is a directory without a `default.nix` file, then its contents (both files and subdirectories) are loaded as Nix expressions.
The expressions are combined into a single attribute set, each expression under an attribute with the same name as the original file or subdirectory.
Subdirectories without a `default.nix` file are traversed recursively in search of more Nix expressions, but the names of these intermediate directories are not added to the attribute paths of the default Nix expression.
Then, the resulting expression is interpreted like this:
- If the expression is an attribute set, it is used as the default Nix expression.
- If the expression is a function, an empty set is passed as argument and the return value is used as the default Nix expression.
For example, if the default expression contains two files, `foo.nix` and `bar.nix`, then the default Nix expression will be equivalent to
```nix
{
foo=import~/.nix-defexpr/foo.nix;
bar=import~/.nix-defexpr/bar.nix;
}
```
The file [`manifest.nix`](@docroot@/command-ref/files/manifest.nix.md) is always ignored.
The command [`nix-channel`] places a symlink to the user's current [channels profile](@docroot@/command-ref/files/channels.md) in this directory.
This makes all subscribed channels available as attributes in the default expression.
## User channel link
A symlink that ensures that [`nix-env`] can find your channels:
-`~/.nix-defexpr/channels`
-`$XDG_STATE_HOME/defexpr/channels` if [`use-xdg-base-directories`] is set to `true`.
This symlink points to:
-`$XDG_STATE_HOME/profiles/channels` for regular users
-`$NIX_STATE_DIR/profiles/per-user/root/channels` for `root`
In a multi-user installation, you may also have `~/.nix-defexpr/channels_root`, which links to the channels of the root user.[`nix-env`]: ../nix-env.md
The manifest file records the provenance of the packages that are installed in a [profile](./profiles.md) managed by [`nix profile`](@docroot@/command-ref/new-cli/nix3-profile.md) (experimental).
Here is an example of what the file might look like after installing `zoom-us` from Nixpkgs:
The manifest file records the provenance of the packages that are installed in a [profile](./profiles.md) managed by [`nix-env`](@docroot@/command-ref/nix-env.md).
Here is an example of how this file might look like after installing `hello` from Nixpkgs:
A directory that contains links to profiles managed by [`nix-env`] and [`nix profile`]:
-`$XDG_STATE_HOME/nix/profiles` for regular users
-`$NIX_STATE_DIR/profiles/per-user/root` if the user is `root`
A profile is a directory of symlinks to files in the Nix store.
### Filesystem layout
Profiles are versioned as follows. When using a profile named *path*, *path* is a symlink to *path*`-`*N*`-link`, where *N* is the version of the profile.
In turn, *path*`-`*N*`-link` is a symlink to a path in the Nix store.
For example:
```console
$ ls -l ~alice/.local/state/nix/profiles/profile*
lrwxrwxrwx 1 alice users 14 Nov 25 14:35 /home/alice/.local/state/nix/profiles/profile -> profile-7-link
lrwxrwxrwx 1 alice users 51 Oct 28 16:18 /home/alice/.local/state/nix/profiles/profile-5-link -> /nix/store/q69xad13ghpf7ir87h0b2gd28lafjj1j-profile
lrwxrwxrwx 1 alice users 51 Oct 29 13:20 /home/alice/.local/state/nix/profiles/profile-6-link -> /nix/store/6bvhpysd7vwz7k3b0pndn7ifi5xr32dg-profile
lrwxrwxrwx 1 alice users 51 Nov 25 14:35 /home/alice/.local/state/nix/profiles/profile-7-link -> /nix/store/mp0x6xnsg0b8qhswy6riqvimai4gm677-profile
```
Each of these symlinks is a root for the Nix garbage collector.
The contents of the store path corresponding to each version of the
profile is a tree of symlinks to the files of the installed packages,
- [`manifest.nix`](@docroot@/command-ref/files/manifest.nix.md) used by [`nix-env`](@docroot@/command-ref/nix-env.md).
- [`manifest.json`](@docroot@/command-ref/files/manifest.json.md) used by [`nix profile`](@docroot@/command-ref/new-cli/nix3-profile.md) (experimental).
## User profile link
A symbolic link to the user's current profile:
-`~/.nix-profile`
-`$XDG_STATE_HOME/nix/profile` if [`use-xdg-base-directories`] is set to `true`.
By default, this symlink points to:
-`$XDG_STATE_HOME/nix/profiles/profile` for regular users
-`$NIX_STATE_DIR/profiles/per-user/root/profile` for `root`
The `PATH` environment variable should include `/bin` subdirectory of the profile link (e.g. `~/.nix-profile/bin`) for the user environment to be visible to the user.
The [installer](@docroot@/installation/installing-binary.md) sets this up by default, unless you enable [`use-xdg-base-directories`].
A Nix channel is a mechanism that allows you to automatically stay
up-to-date with a set of pre-built Nix expressions. A Nix channel is
just a URL that points to a place containing a set of Nix expressions.
Channels are a mechanism for referencing remote Nix expressions and conveniently retrieving their latest version.
To see the list of official NixOS channels, visit
<https://nixos.org/channels>.
The moving parts of channels are:
- The official channels listed at <https://nixos.org/channels>
- The user-specific list of [subscribed channels](#subscribed-channels)
- The [downloaded channel contents](#channels)
- The [Nix expression search path](@docroot@/command-ref/conf-file.md#conf-nix-path), set with the [`-I` option](#opt-i) or the [`NIX_PATH` environment variable](#env-NIX_PATH)
> **Note**
>
> The state of a subscribed channel is external to the Nix expressions relying on it.
> This may limit reproducibility.
>
> Dependencies on other Nix expressions can be declared explicitly with:
> - [`fetchurl`](@docroot@/language/builtins.md#builtins-fetchurl), [`fetchTarball`](@docroot@/language/builtins.md#builtins-fetchTarball), or [`fetchGit`](@docroot@/language/builtins.md#builtins-fetchGit) in Nix expressions
> - the [`-I` option](@docroot@/command-ref/opt-common.md#opt-I) in command line invocations
This command has the following operations:
-`--add`*url* \[*name*\]\
Adds a channel named *name*with URL*url* to the list of subscribed
channels. If *name* is omitted, it defaults to the last component of
*url*, with the suffixes `-stable` or `-unstable` removed.
Add a channel *name*located at*url* to the list of subscribed channels.
If *name* is omitted, default to the last component of*url*, with the suffixes `-stable` or `-unstable` removed.
> **Note**
>
> `--add` does not automatically perform an update.
> Use `--update` explicitly.
A channel URL must point to a directory containing a file `nixexprs.tar.gz`.
At the top level, that tarball must contain a single directory with a `default.nix` file that serves as the channel’s entry point.
-`--remove`*name*\
Removes the channel named *name* from the list of subscribed
channels.
Remove the channel *name* from the list of subscribed channels.
-`--list`\
Prints the names and URLs of all subscribed channels on standard
output.
Print the names and URLs of all subscribed channels on standard output.
-`--update` \[*names*…\]\
Downloads the Nix expressions of all subscribed channels (or only
those included in *names* if specified) and makes them the default
for `nix-env` operations (by symlinking them from the directory
`~/.nix-defexpr`).
Download the Nix expressions of subscribed channels and create a new generation.
Update all channels if none is specified, and only those included in *names* otherwise.
-`--list-generations`\
Prints a list of all the current existing generations for the
prior to running`nix-collect-garbage` (or just `nix-store --gc`) without any flags.
> **Note**
>
> Deleting previous configurations makes rollbacks to them impossible.
These flags should be used with care, because they potentially delete generations of profiles used by other users on the system.
## Locations searched for profiles
`nix-collect-garbage` cannot know about all profiles; that information doesn't exist.
Instead, it looks in a few locations, and acts on all profiles it finds there:
1. The default profile locations as specified in the [profiles] section of the manual.
2. > **NOTE**
>
> Not stable; subject to change
>
> Do not rely on this functionality; it just exists for migration purposes and is may change in the future.
> These deprecated paths remain a private implementation detail of Nix.
`$NIX_STATE_DIR/profiles` and `$NIX_STATE_DIR/profiles/per-user`.
With the exception of `$NIX_STATE_DIR/profiles/per-user/root` and `$NIX_STATE_DIR/profiles/default`, these directories are no longer used by other commands.
`nix-collect-garbage` looks there anyways in order to clean up profiles from older versions of Nix.
# Options
These options are for deleting old [profiles] prior to deleting unreachable [store objects].
Delete all generations of profiles older than the specified amount (except for the generations that were active at that point in time).
*period* is a value such as `30d`, which would mean 30 days.
This is the equivalent of invoking [`nix-env --delete-generations <period>`](@docroot@/command-ref/nix-env/delete-generations.md#generations-time) on each found profile.
See the documentation of that command for additional information about the *period* argument.
@@ -389,6 +389,88 @@ colors, no emojis and using ASCII instead of Unicode symbols). The same should
happen when TTY is not detected on STDERR. We should not display progress /
status section, but only print warnings and errors.
## Returning future proof JSON
The schema of JSON output should allow for backwards compatible extension. This section explains how to achieve this.
Two definitions are helpful here, because while JSON only defines one "key-value"
object type, we use it to cover two use cases:
- **dictionary**: a map from names to value that all have the same type. In
C++ this would be a `std::map` with string keys.
- **record**: a fixed set of attributes each with their own type. In C++, this
would be represented by a `struct`.
It is best not to mix these use cases, as that may lead to incompatibilities when the schema changes. For example, adding a record field to a dictionary breaks consumers that assume all JSON object fields to have the same meaning and type.
This leads to the following guidelines:
- The top-level (root) value must be a record.
Otherwise, one can not change the structure of a command's output.
- The value of a dictionary item must be a record.
Otherwise, the item type can not be extended.
- List items should be records.
Otherwise, one can not change the structure of the list items.
If the order of the items does not matter, and each item has a unique key that is a string, consider representing the list as a dictionary instead. If the order of the items needs to be preserved, return a list of records.
- Streaming JSON should return records.
An example of a streaming JSON format is [JSON lines](https://jsonlines.org/), where each line represents a JSON value. These JSON values can be considered top-level values or list items, and they must be records.
### Examples
This is bad, because all keys must be assumed to be store implementations:
```json
{
"local":{...},
"remote":{...},
"http":{...}
}
```
This is good, because the it is extensible at the root, and is somewhat self-documenting:
```json
{
"storeTypes":{"local":{...},...},
"pluginSupport":true
}
```
While the dictionary of store types seems like a very complete response at first, a use case may arise that warrants returning additional information.
For example, the presence of plugin support may be crucial information for a client to proceed when their desired store type is missing.
The following representation is bad because it is not extensible:
```json
{"outputs":["out""bin"]}
```
However, simply converting everything to records is not enough, because the order of outputs must be preserved:
```json
{"outputs":{"bin":{},"out":{}}}
```
The first item is the default output. Deriving this information from the outputs ordering is not great, but this is how Nix currently happens to work.
While it is possible for a JSON parser to preserve the order of fields, we can not rely on this capability to be present in all JSON libraries.
This representation is extensible and preserves the ordering:
CLIs don't always make it clear when an action has taken place. For every
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
