`tryToBuild` threaded a single `PathLocks outputLocks` by reference
across all build phases and managed a `std::unique_ptr<Activity> actLock`
with explicit `if (!actLock)` guards and `.reset()` calls around the hook
retry loop. This commit introduces coroutine lambdas for the three phases:
`tryHookLoop` owns a `PathLocks` in a scoped block for the first attempt
and per-iteration in the retry loop, `tryBuildLocally` acquires its own
`PathLocks`, and the hook-wait `Activity` is a stack variable scoped to
the postpone block.
When `max-jobs = 0` and no remote builders are available, Nix reported
"required system or feature not available" even though the system and
features matched fine. The `canBuildLocally` lambda returned a plain
`bool`, conflating a configuration knob (`max-jobs = 0`) with actual
incompatibility (wrong platform, missing features). It also short-circuited
on the first failing check, so a user with both a platform mismatch and
missing features would only see one of the two.
This commit replaces the bool with a `LocalBuildRejection` struct whose
`WrongLocalStore` variant collects all applicable failures into
`badPlatform`, `missingFeatures`, and an orthogonal `maxJobsZero` flag.
Platform mismatch and missing features now produce separate error
paragraphs, and all applicable reasons appear in a single message.
The local-build capability check also now returns
`std::variant<LocalBuildCapability, LocalBuildRejection>`, bundling
the `LocalStore &` and optional `ExternalBuilder *` together.
This comes in two parts: a `nix store roots-daemon` command that
can run as root and list runtime roots,
and client logic to find runtime roots for a `LocalStore` by connecting
to that daemon.
This may be useful with an unprivileged nix daemon, as it would
otherwise be unable to find runtime roots from process open files
and maps.
Using nix::unreachable() in getInternalType() and type() turns
out to be quite expensive and prevents inlining. Also Value::type
got compiled to a jump table which has a high overhead from indirect
jumps. Using an explicit lookup table turns out to be more efficient.
This does mean that we lose out on nice diagnostics from nix::unreachable
calls, but this code is probably one of the hottests functions in the whole
evaluator, so I think the tradeoff is worth it. The nixUnreachableWhenHardened
boils down to nix::unreachable when UBSan is enabled so we still have good
coverage there.
This makes sure that ExprVar::eval inlines lookupVar call. In practice
this seems to reduce instruction count by ~2%, though it doesn't have
a statistically significant impact on the wall time.
This reduces the churn when changing up the order of
values in a follow-up commit. This should have been done
from the start ideally to improve readability.
The external-builders test expands `$PATH` into a heredoc without quotes,
so any `PATH` entry containing spaces causes bash to parse the line as a
command instead of an assignment, failing the test.
The tab completion handler in `completePrefix` only caught `ParseError`,
`EvalError`, `BadURL`, and `FileNotFound`. Other error types like
`JSONParseError` (which derives from `Error`, not `EvalError`) escaped
the catch block and propagated through editline's C code as undefined
behavior, crashing the REPL. This happened when tab-completing
expressions like `(builtins.fromJSON "invalid").` where evaluation
throws a non-`EvalError` exception.
This commit marks `completionCallback` and `listPossibleCallback` as
`noexcept` with function-try-blocks that catch all exceptions at the
C/C++ boundary, preventing any exception from reaching editline.
Fixes#15133.
When a URL like `github:nixos/nixpkgs/nixpkgs.git?ref=<hash>` (using
`ref` instead of `rev`) failed the github input scheme, it fell
through to `parsePathFlakeRefWithFragment` which constructed a `path:`
`ParsedURL` with an empty authority but a relative path. This violated
RFC 3986 section 3.3 (authority present requires path starting with
`/`), causing an assertion failure in `renderAuthorityAndPath` when
`PathInputScheme` tried to format the URL for an error message.
This commit only sets the authority on absolute paths. Relative paths
get `std::nullopt` for authority, which is the correct representation
per the URL spec.
Fixes#15196. Fixes#14830.
`file-descriptor.{cc,hh}` was getting too big, split out
`file-system-at.{cc,hh}` for the FD-based file system stuff,
`file-descriptor.{cc,hh}` will only be for the fundamental primitives
that are file-system agnostic and work on almost all file types.
Review with `git show --color-moved` to see that this is indeed all
moving.
Docker 28+ defaults to the containerd image store, which pushes layers
uncompressed instead of gzip. The GHA runner image updated Docker to
29.x (actions/runner-images#13633), causing the `nixos/nix:2.33.3`
image to balloon from 138 MB to 505 MB, with all 70 layers pushed as
`application/vnd.docker.image.rootfs.diff.tar` instead of `.tar.gzip`.
OCI clients that only support gzip (e.g. `go-containerregistry`, used
by Concourse CI) fail with "gzip: invalid header".
This commit disables the containerd snapshotter in the release workflow
before any Docker operations, restoring the classic storage driver that
preserves gzip compression through the `docker load` / `docker push`
pipeline.
Fixes#15246
Linux, macOS, and all 3 BSDs have it (according to man page google
search), so let's just drop this. Support for not having it was added in
d03f0d4117 in 2006, things have changed in
the last 20 years!
When ref::cast() fails, the error message was cryptic ("null pointer
cast to ref"). Now it throws a proper bad_ref_cast (a std::bad_cast
subclass) with a clear message showing the actual types involved:
ref<nix::Base> cannot be cast to ref<nix::Derived>
This also adds a demangle.hh utility.
ref<Derived> was already implicitly convertible to ref<Base>, but the
mechanism was unclear and error messages for rejected downcasts were
more cryptic than necessary. This change:
- Adds RefImplicitlyUpcastableTo concept to constrain the conversion
operator, making the intent explicit and improving error messages
- Documents .cast() and .dynamic_pointer_cast() as alternatives for
explicit downcasting
- Adds unit tests for covariance behavior
The old check rejected any relative path whose first character was a
dot, producing false negatives for valid descendants like `.ssh` or
`.config`. This commit changes the logic such that now it inspects the
first path component via `path::begin()`, only rejects `.` and `..`
rather than anything dot-prefixed. Fixes#15207.
libutil tests were crashing on Windows due to issues finding `environ`.
Replace process creation of `getEnv` with a new `getEnvOs` function that
uses native windows APIs.
Also convert a bunch of `RunOptions` fields to use `OsString` to better
reflect the underlying interfaces.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
The fake cacert didn't have subjectAltName for 127.0.0.1, so the test
was failing for a different reason. Also `tries` setting wasn't being respected.
There's no callsite specifying it in the request, so just use the one specified
in the FileTransferSettings and remove the fields from the FileTransferRequest.
This commit removes the `nixStore` member from `Settings` and instead
computes the default Nix store directory directly in
`StoreConfigBase::getDefaultNixStoreDir()` from env vars
(`NIX_STORE_DIR`, `NIX_STORE`) or the compile-time default. The method
is made public so callers that previously reached through the global
`settings.nixStore` can use it instead.
Progress on #5638
nix store gc: prints number of paths that would be freed, but not bytes
nix-collect-garbage: ditto
nix-store --gc: retains current behavior
It would be very non-trivial to also compute the bytes that would be
freed, due to hardlinking in the store.
Also adds checking for incompatible mixing of dry-run and max-freed
options.
Resolves#5704
Signed-off-by: Lisanna Dettwyler <lisanna.dettwyler@gmail.com>
This commit inlines `DerivationOptions::willBuildLocally` and
`DerivationOptions::canBuildLocally` into their sole call site in
`DerivationBuildingGoal::tryToBuild`. The `canBuildLocally` logic is now
a lambda capturing the surrounding context, and `willBuildLocally` is
replaced by `drvOptions.preferLocalBuild && canBuildLocally`.
Progress on #5638
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
This commit removes the `getMachines` free function and inlines `Machine::parseConfig({settings.thisSystem}, settings.getWorkerSettings().builders)` at its two call sites in `worker.cc` and `build-remote.cc`. The wrapper just forwarded to `Machine::parseConfig` with global settings, so inlining it removes an unnecessary layer of indirection and makes the global dependency explicit at each call site.
Progress on #5638
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
This commit moves `nixStateDir` and `useXDGBaseDirectories` into a dedicated `ProfileDirsOptions` struct and threads it through the profile directory functions (`profilesDir`, `rootProfilesDir`, `defaultChannelsDir`, `rootChannelsDir`, `getDefaultProfile`) so they no longer read from the global `Settings` object directly. This follows the same pattern as `LocalSettings`, `WorkerSettings`, and `NarInfoDiskCacheSettings`.
Progress on #5638
Co-authored-by: Amaan Qureshi <git@amaanq.com>
This commit moves `ttlNegativeNarInfoCache` and `ttlPositiveNarInfoCache` into a dedicated `NarInfoDiskCacheSettings` struct that `Settings` privately inherits from, following the same pattern as `LocalSettings`, `LogFileSettings`, and `WorkerSettings`.
`NarInfoDiskCache` now takes explicit `NarInfoDiskCacheSettings` and `SQLiteSettings` in its constructor instead of reading from the global. The singleton `getNarInfoDiskCache()` is replaced with a `NarInfoDiskCache::get()` static method that accepts these settings, though they are only used on the first call (subsequent calls return the cached instance regardless of arguments).
Progress on #5638
It was a crude hack that this one low-level function was dependent on
the high-level read-only mode setting --- all the more so because rather
than making derivation writing fail, that setting made it silently
"succeed" why not actually writing the derivation. (Also, for context,
we didn't have an such behavior for any other store-mutating operations,
just for this one function.)
I have gotten rid of the wrapper, and updated the call sites
accordingly.
- For the ones that should remain dependent on this setting, I made this
explicit, and added a comment.
- For others, surrounding operations assumed writability (e.g. we had
written something before, or were about to try to read back the
written derivation after), and so I just made those do the underlying
`Store::writeDerivation` operation.
This makes `acquireUserLock` take an explicit stateDir parameter,
since it was previously reaching into the global settings object
just to read `nixStateDir` for constructing the userpool paths.
Progress on #5638
This commit introduces a `getReadOnly` method on the store config that returns if the current store is read only or not. This is then used in subtitution, so we fail gracefully with a nice error message if only the individual store is read-only.
As a bonus, it gets us one step closer to getting rid of the global because we can use the per-store method instead.
Progress on #5638
S3 binary caches now use virtual-hosted-style URLs by default for
standard AWS endpoints. Path-style endpoints (s3.region.amazonaws.com)
only serve HTTP/1.1, preventing HTTP/2 multiplexing and causing TCP
TIME_WAIT socket exhaustion under high concurrency. Virtual-hosted-style
endpoints (bucket.s3.region.amazonaws.com) support HTTP/2, enabling
multiplexing with the existing CURLPIPE_MULTIPLEX configuration.
Add a new `addressing-style` store option (auto/path/virtual) to control
this behavior. `auto` (default) uses virtual-hosted-style for standard
AWS endpoints and path-style for custom endpoints. `path` forces
path-style for backwards compatibility. `virtual` forces virtual-hosted-
style for all endpoints including custom ones.
Fixes: https://github.com/NixOS/nix/issues/15208
Idle connections in libcurl's connection pool can be silently dropped by
the OS or intermediate firewalls/NATs before they can be reused, forcing
new TCP connections to be created. This is especially problematic for
HTTP/1.1 endpoints where multiplexing is unavailable.
Enable TCP keep-alive with a 60-second idle/interval on all curl easy
handles to prevent idle connection drops and improve connection reuse.
This commit moves `pollInterval`, `maxSubstitutionJobs`, `postBuildHook`, and `logLines` into a dedicated `WorkerSettings` struct that `Settings` privately inherits from, as they are only used by the build worker subsystem. This follows the same pattern as `LocalSettings` and `LogFileSettings`.
This expands to __GCC_DESTRUCTIVE_SIZE, which is also 64 (at least in the x86_64 stdenv).
Let the compiler decide what's the appropriate cache line size is. Also, on aarch64-darwin
the cache line size 128 bytes, so the previous fix didn't actually get rid of false sharing
reliably. Clang does this [1] [2], so it overestimates the sizes somewhat, but that's still enough
for avoiding false sharing on darwin.
[1]: a289341ded/clang/lib/Frontend/InitPreprocessor.cpp (L1331-L1339)
[2]: 6f51f8e0f9/clang/lib/Basic/Targets/AArch64.h (L262-L264)
In the https-store tests, a `TestHttpBinaryCacheStoreConfig` is constructed with a call to format to create the cache uri. This commit adds a constructor to `HttpBinaryCacheStoreConfig` to remove the need for this call, and updates the test type to leverage this so we're no longer manually calling fmt on a string to format the port.
This commit makes `FileTransfer` self-contained by giving it a reference
to `FileTransferSettings` instead of reading from the global. It also
adds an optional `FileTransfer` parameter to `HttpBinaryCacheStore` so
callers can inject their own instance.
The main motivation is test isolation. The HTTPS store tests now create
custom `FileTransferSettings` with the test CA certificate and pass it
through `makeFileTransfer()`, avoiding global state mutation entirely.
nix-nswrapper allows running nix in its own user namespace,
believing it is root and with access to build users for sandboxing
with auto-allocate-uids, while it is actually unprivileged.
It is used to wrap nix, and an example of its use has been
added to the unprivileged daemon functional tests.
Running it does not require any elevated privileges,
only uids and gids allocated in /etc/sub{uid,gid}
The `caFile`, `netrcFile`, and `downloadSpeed` settings are only used by
the file transfer subsystem but lived in the global `Settings` class.
This moves them to `FileTransferSettings` where they belong.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
`hashedMirrors` is only relevant to local builds (it is consumed by
`builtin:fetchurl` during derivation building) but lived in the global
`Settings` class. This moves it to `LocalSettings` where it belongs
and threads it through `BuiltinBuilderContext` so `fetchurl.cc` reads
it from the context instead of reaching into `settings` directly.
The underlying mechanism is now in a new `fetchBuildLog` function I put
in `libcmd`.
I am putting it in here and not in libstore because I have some doubts
about `getDefaultSubstituters`, so I would like to keep it in a more
"peripheral" part of the codebase for now.
The `storeUri`, `substituters`, and `trustedSubstituters` settings now
store typed `StoreReference` values directly instead of raw strings,
so callers work with the real types without manual parsing.
This is a reworked version of #10761.
Instead of constructing a `StoreReference` and letting `openStore()` resolve it,
this commit creates the store config directly and calls `openStore()` on it. This
avoids the indirection through the store registry.
Stores hold their config as `ref<const Config>` or `const Config &`,
so `Setting<>` fields are already immutable after store construction.
The field-level `const` is redundant and prevents pre-construction
mutation which is sometimes useful. This commit updates these settings by dropping the `const` qualifier, as it's not needed.
The CLI flags `--from`, `--to`, `--eval-store`, and substituter URIs now
parse to `StoreReference` at the argument boundary. `fetchClosure` uses
`StoreReference::parse` instead of `parseURL`. This also adds
`operator<=>` to `StoreReference`.
This commit makes `nix daemon` inherit from `StoreConfigCommand`
instead of `Command`, so that it receives a `StoreConfig` to open
and serve stores with. This cleans up a few things (removes
`openUncachedStore` helper, passes `storeConfig` through
`daemonLoop`/`runDaemon` instead of opening stores ad-hoc) and will
allow further cleanups.
It's apparently a common footgun in Linux that the death signal isn't
preserved across calls to setuid/setgid. If nix-build gets SIGKILL-ed
while a build is running that would lead to a runaway build process that
would get reparented to init/systemd.
This is pretty easy to reproduce with the following derivation:
derivation {
name = "pdeathsig-repro";
system = builtins.currentSystem;
builder = "/bin/sh";
args = [
"-c"
''
while :; do :; done
''
];
}
And the reproduction script:
sudo nix-build repro.nix &
sleep 3
BUILDER=$(pgrep -u nixbld1)
sudo kill -9 $(pgrep -f 'nix-build.*repro')
sleep 1
ps -p $BUILDER -o pid,ppid,user,comm
To address this we have to restore the death signal after all the calls
to setuid/setgid. This is done in a helper function preserveDeathSignal
that takes a callback to avoid code duplication.
See: https://github.com/golang/go/issues/9686
This is a fairly simple function, isolated from the rest of libmain
and could be useful if new programs are made that are not part of the
main nix-cli subproject.
This commit extracts the Unix domain socket server loop (`PeerInfo`,
`getPeerInfo`, and the systemd socket activation / poll / accept loop)
from `src/nix/unix/daemon.cc` into a reusable `unix::serveUnixSocket`
function in `libcmd`.
Diagnosed by UBSan in hydra [1]:
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /nix/store/m1k4nxs8r0fl0pjxqp5n37vxgms7gdlb-gcc-14.3.0/include/c++/14.3.0/bits/move.h:234:11 in
+(prefetch.sh:6) path=
++(prefetch.sh:6) onError
++(/build/source/tests/functional/common/functions.sh:241) set +x
prefetch.sh: test failed at:
main in prefetch.sh:6
[1]: https://hydra.nixos.org/build/321098757/nixlog/1
To get `bashInteractive`, `nix develop` currently
[gets a flake reference to nixpkgs](3b473c4be5/src/nix/develop.cc (L650-L658)),
either from [`inputs.nixpkgs`](3b473c4be5/src/libcmd/installable-flake.cc (L204))
or [`<nixpkgs>`](3b473c4be5/src/libcmd/include/nix/cmd/installable-flake.hh (L87)).
Currently, this is done by name for any (locked) reference.
For any `nixpkgs` reference lacking a `flake.nix`, `nix develop` thus errors:
```
error (ignored): path '.../flake.nix' does not exist
```
While the registry does not expose info to help check this,
flake inputs expose the `flake` boolean.
This means that given `inputs.<name>.flake = false;`,
we may know in advance that our reference should not be to a flake.
This change incorporates this, so that given `inputs.nixpkgs.flake = false;`,
`nix develop` will fall back `<nixpkgs>` to use its flake for `bashInteractive`.
While Nixpkgs itself of course does expose a flake,
this check is relevant in particular for dummy inputs
(e.g. inputs using `{ url = "file:/dev/null"; flake = false; }`),
which may be overridden by `--override-input` or `.follows`,
yet do not expose (flake) code themselves.
Signed-off-by: cinereal <cinereal@riseup.net>
Extract the logic for determining whether a derivation should be resolved
before building into a dedicated method. Then use that to not resolve
unnecessarily in `nix-shell`.
The recent conversion of WorkerProto::Version from unsigned int to a
struct exposed a latent issue: `.version = 16` was being interpreted
as aggregate initialization `{.major = 16, .minor = 0}` rather than
the intended wire format value.
This commit adds -Werror=c99-designator to catch this class of bugs at
compile time. (The bug itself was fixed in
7eb23c15f39ca413a5f3cc0d3ab630311b4709be.)
For background:
The hardcoded version was originally the integer 16, which in the old
wire format (major << 8 | minor) meant version 0.16. However, the
version checks only compared minor versions via GET_PROTOCOL_MINOR(),
so this worked by accident.
After the Version struct conversion, the aggregate initialization
{.major = 16, .minor = 0} happened to still work because 16 > 1 in
lexicographic comparison against {1, 16}.
The correct version is {1, 16} because:
- The worker protocol uses major version 1 (all checks are {1, x})
- Version 1.16 is when ultimate/sigs/ca fields were added
- This matches the serialization check `>= {1, 16}`
Fixes abort on :ll if nothing has been loaded yet. Also gets rid of
redundant openStore() calls that were dead code (store can be extracted
from EvalState already) and arguably openStore is a layer violation.
Also catches EPIPE in case the pager gets interrupted to avoid superfluous
error messages.
This commit embeds the negotiated `FeatureSet` directly into `WorkerProto::Version` and introduces a `Number` inner type with total ordering, so that `Version` itself (number + features) only has partial ordering. This is a follow-up to #15155, cleaning up the separate `features` fields on `ReadConn`/`WriteConn`.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
This improves the performance of parseNarListing, which is used by commands
like `nix nar ls` when the underlying source allows cheap seeks (like StringSource
or FdSource that does lseek).
For `nix nar ls` of a NAR for linux source tarball this cuts down the runtime almost
in half (from 300ms -> 175ms).
This commit replaces the `GET_PROTOCOL_MINOR(version)` macros with a proper `WorkerProto::Version` struct. As a bonus, this also fixes some version checks that were incorrectly ignoring the major version number.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
This change is required for implementing the unprivileged garbage collection daemon,
but it may also be useful to reduce code duplication and separate out OS-specific
garbage collector roots implementations in the future.
Caching NAR hashes instead of store paths makes the cache more
general, because we can always compute the store path from the NAR
hash, but not the other way around. This is useful for lazy trees,
where we want to compute the NAR hash of an input with caching.
The global `Settings` struct contained many settings that only apply to
local builds or the local store (sandbox configuration, GC settings,
build user groups, etc.). This commit extracts these into a dedicated
`LocalSettings` struct in its own header, along with `GCSettings` and
`AutoAllocateUidSettings`.
This improves code organization and prepares for eventually making these
per-store settings in the future. Settings are accessed via
`getLocalSettings()` from the global settings object or through
`LocalStoreConfig::getLocalSettings()` for store-specific access.
Useful for commands that need a `StoreConfig` but do not want to open
the store, as a `StoreCommand` would do.
At the same time, make copy commands more clear by making store choice
for `updateProfile` explicit and removing the unused `getDstStore` function
in `StoreCommand`. This was a layer violation, as `StoreCommand` does
not have a concept of a source/destination store distinction.
This is purely a fix to use CreateFileW in mingw builds. Also adds some
FIXMEs for suspicious symlink following on truncation that can probably
be tightened down without any problems (other than nix-channel), but for
now this is a no-op change other than consistently using O_CLOEXEC, which
is harmless.
We now support `LISTEN_FDS` values greater than 1, per the systemd
socket activation spec.
These changes are by @edolstra, taken from #5265. This is just that PR
*without* the TCP parts, which I gathered are the controversial parts.
Hopefully this remainder is not so controversial.
Review with indentation ignored, because some code has moved inside a
new loop but otherwise is mostly unchanged.
aws-sdk-cpp used to include a checksum for uploads (CRC64 since ~September 2025).
Content-MD5 [1] should be universally supported by all s3 compatible services, since the SDK used
to include it unconditionally too.
[1]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
This commit consolidates the four separate boolean flags
(`permanentFailure`, `timedOut`, `hashMismatch`, & `checkMismatch`) into
a single `ExitStatusFlags` struct with methods for computing exit status
codes and updating from failure status.
The explicit serializer added in
bfdd124837 is the right place to adjust
values for sake of wire protocol compat. The protocol-agnostic `Worker`
code where it was before is the wrong spot.
(That spot was originally chosen because the back compat logic predates
having an explicit serializer for this data type to use instead.)
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
This test insisted on placing profiles in NIX_STATE_DIR, but all
packages were removed from the profile immediately after so they did not
act as garbage collector roots. Switch to directly calling nix-build,
allowing the test to run in VMs without NIX_STATE_DIR.
This commit refactors the `canonicalisePathMetaData` function to take an
options struct instead of individual parameters with platform-specific
`#ifdef`s.
The struct contains a `uidRange` field (Unix only) for build user
ownership validation, and an `ignoredAcls` field for ACLs to skip when
removing extended attributes
Due to a typo in quoteRegexChars, finding runtime garbage collection roots
was failing on paths that contained a dot, or any other regex chars that would
have to be replaced.
When fixing that error, also add tests to make sure gc continues to
work.
This introduces a `PersonalityArgs` struct to pass named arguments to `setPersonality`. The `impersonateLinux26` setting is now passed from the call site rather than read from settings inside the function.
The C++ rule of five suggests that when a custom destructor is needed
then several other functions are as well. The lack of those makes
certain operations challenging
We use a different fstat on posix and windows systems,
and not all fstat users were using the correct one.
Factor out fstat to make the change easier.
See also a13de50df3 for other stat
functions
This cleans up the logic for checking if the worker's store is a valid
local store when we're not hooking it. If we have a local store, we then
pass that as an argument to `DerivationBuildingGoal::buildLocally`,
rather than checking inside the function itself.
The two settings `envKeepDerivations` and `upgradeNixStorePathUrl` were
only used in one command each, so it makes more sense to move them to
their own files. This commit moves them both into a small self-contained
settings struct and registers them with the global config.
Currently, tests fail when the host system has `commit.gpgsign` or
`tag.gpgsign` enabled at the system level (in my case, a custom path
located at `/etc/git/config`), since the signing key is unavailable in
the test sandbox.
The tests set `HOME=$TEST_HOME` to isolate themselves, which bypasses
the user-level git config (`~/.gitconfig`). However, if a user sets the
system-level config via `GIT_CONFIG_GLOBAL` or `GIT_CONFIG_SYSTEM`, it
still applies, causing commits to fail when signing is enabled there.
In this PR, I explicitly set `GIT_CONFIG_GLOBAL` and `GIT_CONFIG_SYSTEM`
to `/dev/null` so that the user's system config is never read from or
written to. I've also replaced `git config --global protocol.file.allow
always` with `GIT_CONFIG_*` environment variables to avoid writing to
`/dev/null`.
The test was checking for `$stdenv` but the `fixed` derivation doesn't
actually have stdenv, it just has `FOO`. I've updated it to check the
value of `FOO` instead.
The settings related to diff hook (`run-diff-hook` and `diff-hook`) are
a little redundant and don't need to be leaked in derivation-builder
when computing the diff hook path to execute.
Instead of directly using both `runDiffHook` and `diffHook` settings in
derivation-builder, we can just encapsulate the logic to determine
whether or not we have a diff hook executable to run in a helper
function. We also mark `handleDiffHook` as static.
- Extract destructor logic into named methods (`deletePath()`,
`unmount()`, and `remove()`) that can be called explicitly. These ones
will throw exceptions normally, unlike the destructor which must quash
them to avoid double-exceptions.
- Use `std::filesystem::path` in `AutoUnmount` (changed from `Path`)
- Remove `del` field from `AutoRemoveJail`, using `INVALID_JAIL`
sentinel value instead.
- Add move assignment operators implemented via friend `swap` functions
for all three RAII classes.
- Remove old `reset(...)` methods that took parameters. These were a bit
misleading --- do they cancel or immediately destroy? --- and doing it
explicitly with cancel and then assignment is not hard.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
The previous implementation double-quoted the _args variable by escaping
each argument individually and then wrapping them all in single quotes,
producing output like: _args=''-e' 'arg1' 'arg2''
This fix concatenates all arguments into a single string first, then
escapes that string once, producing correct output like:
_args='-e arg1 arg2'
This prevents potential command injection issues when the output is
sourced in shell scripts.
Fixes#14327
Use wrappers to make error handling easier.
On Windows we are using proper 64-bit time and size info.
We still have the problem of no `lstat` on Windows, but this will be
dealt with in future PRs.
Follows the same pattern as `GCSettings`: extract UID allocation
settings
into a dedicated struct that Settings inherits privately from.
The current settings infrastructure prevents correct data modeling that
would allow `autoAllocateUids` to be a
`std::optional<AutoAllocateUidSettings>`.
To compensate, the getter `getAutoAllocateUidSettings()` returns a
pointer -
nullptr when disabled, providing the optional-like semantics we want.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
settings.buildHook.setDefault was running after nix.conf was parsed,
causing whatever value settings.buildHook had to be clobbered.
Re-arrange the logic so that the default is set before nix.conf is
parsed, so that custom build hooks can be used by specifying them in
nix.conf.
Signed-off-by: Lisanna Dettwyler <lisanna.dettwyler@gmail.com>
This is no longer necessary and produces an eval warning:
> evaluation warning: The 'pie' hardening flag has been removed in favor of enabling PIE by default in compilers and should no longer be used.
This was first introduced in 2200f315da, but
is no longer necessary since the switch to meson.
Previously AWS CRT logs went directly to stderr via ApiHandle::InitializeLogging,
causing log spam that didn't respect Nix's verbosity settings.
This implements a custom aws_logger using the aws-c-common C API that:
- Routes all AWS logs through nix::logger
- Maps AWS log levels conservatively (ERROR/WARN -> lvlInfo) since the SDK
treats expected conditions like missing IMDS as errors
- Prefixes messages with (aws) for clarity
- Respects Nix's verbosity flags (-v, -vv, etc.)
This unsures that we call the correct virtual functions when destroying a particular
DerivationBuilder.
Usually the order of destructors is in the reverse order of inheritance:
ChrootLinuxDerivationBuilder -> ChrootDerivationBuilder -> DerivationBuilderImpl
autoDelChroot was being destroyed before the DerivationBuilderImpl::killChild was
run and it would fail to clean up the chroot directory, since there were still processes
writing to it. Note that ChrootLinuxDerivationBuilder::killSandbox was never run in
the interrupted case at all, since virtual functions in destructors do not call derived class
methods.
I could reproduce the issue with the following derivation:
let
pkgs = import <nixpkgs> { };
in
pkgs.runCommand "chroot-cleanup-race" { } ''
mkdir -p $out
for i in $(seq 1 200); do
(
mkfifo $out/fifo$i
cat $out/fifo$i > /dev/null &
while true; do
: > $out/file$i
done
) &
done
sleep 0.05
echo done > $out/main
''
While interrupting it manually when it would hang.
Wrapping the unique pointer in a custom deleter function we can run all
of the necessary clean up code consistently and calling the right virtual
functions. Ideally we'd have a lint that bans the usage of virtual functions
in destructors completely.
This addresses the concerns with network isolation that have been raised
previously [1] by only running the tests by default in a network namespace.
This way all networks tests are independent of each other and do not bind
to ports in the host namespace.
This is much neater than doing these sorts of tests in functional suite.
[1]: https://github.com/NixOS/nix/pull/14266#issuecomment-3411261285
Don't add the whole store to sandbox-paths unconditionally. Exposing
the entire store defeats the purpose of sandboxing, and when the test
store is the same as the system store (NixOS VM), it causes an obscure
"Permission denied" error.
Only add sandbox-paths when not on NixOS, indicating a separate test
store that needs access to system store build tools.
All current NixOS functional VM tests have a daemon as root with the
tests running as different unprivileged users.
The new `functional_unprivileged-daemon` test runs the daemon and the
nix functional tests as separate unprivileged users.
Users may want to run an unprivileged daemon on non-NixOS systems
where the administrator does not fully trust nix, but multiple users
want to use nix for their own purposes. It could also be useful in
concert with an overlay-mount store, where the nix daemon cannot
modify the derivations used by the system, and thus a nix vulnerability
would not lead to root code execution.
When running nix as an unprivileged user it may not be able to write to
all paths in the nix store. Ignore deletion failures to fix tests that
run `nix-collect-garbage` in this configuration.
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
We are now seeing. I guess we are out with the cache. When the API responds with 418 (I'm a teapot)
it seems like the only reasonable solution is to oblige.
error: unable to download 'http://127.0.0.1:37515/7ms9f25xyxavf32pvdc3vb28nzzmkbn3.narinfo': HTTP error 418
response body:
GitHub API error: GitHub Actions Cache throttled Magic Nix Cache. Not trying to use it again on this run.
This means that `RestoreSink` can work in the TOCTOU-resilliant way on
Windows too. And it also bodes will for the upcoming OS source accessor
improvements.
A few misc little refactors around error handling and whatnot are done
along the way too. (No more attempt to support pre Windows Vista! lol.)
This cannot be realiably automatically tested until we have a newer
version of Wine, but it does build, so I am inclined to say we just try
it for now.
This PR follows the same approach as #15043 and the
[`LogFileSettings`](https://github.com/NixOS/nix/pull/15051)
extraction:
- `GCSettings` struct inherits from virtual `Config`
- `Settings` privately inherits from it
- Accessed through `getGCSettings()`
The new method on `LocalStoreConfig` anticipates on making these
settings per-store. 0b606aad46 added both
the autoGC and periodic wakeups, which is why we think they are related.
When an upload fails with a transient HTTP error (e.g., S3 rate limiting
with HTTP 503), retries would fail with "curl error: Failed to open/read
local data from file/application" because the upload source was already
exhausted from the previous attempt.
Restart the source in init() to ensure it's at the beginning for both
first attempts (no-op) and retries (necessary fix).
Fixes: #15023
Progress on #5638
Replace the SQLite constructor's mode parameter with a Settings struct
that includes both the open mode and useWAL flag. This makes the
dependency on useSQLiteWAL explicit at call sites rather than having
it read from the global settings inside the constructor.
All call sites now explicitly pass settings.useSQLiteWAL, preparing
for downstream work where stores can pass their own settings instead
of relying on the global.
Some S3-compatible services (like GCS) modify the Accept-Encoding header
in transit, which breaks AWS SigV4 signature verification since curl's
implementation signs all headers including Accept-Encoding.
Fixes: #15019
It is possible that the `nix` executable is installed but not `nix-env`
(this may be unusual but for example in Fedora we have a separate
`nix-legacy` subpackage, which includes the `nix-env` symlink).
The current error message:
```
$ nix config check --verbose
Running checks against store uri: local
[FAIL] Multiple versions of nix found in PATH:
```
when there is no nix-env in PATH is confusing.
This change makes the error message precise for the missing nix-env case.
Use
```
git show --color-moved --patience --color-moved-ws=ignore-all-space
```
to review and see that this is mostly code motion.
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
Introduce a new `Signature` struct that represents a cryptographic
signature
along with the key name that produced it. This provides:
- Structured representation instead of colon-separated strings
- Type-safe parsing with `Signature::parse()`
- Serialization with `to_string()`
- JSON serialization/deserialization
- Batch parsing with `parseMany<Container>()`
- Batch serialization with `toStrings()`
This is scaffolding for future changes that will use this type
throughout the codebase.
This will once and for all get rid of all double-quoting issues. On windows the quoting
is doubly bad because it escaped all \ to \\, which is very bad for error messages. In
order to prevent future regression std::filesystem::path formatting now must use a special
type PathFmt (like Magenta). In the future we could even change how we render filesystem paths.
Instead of the stringly typed code we should use an enum class, this is
more clear and less error-prone. Also adds settings implementations for
CompressionAlgo and std::optional<CompressionAlgo>. The first is used
for NAR compression, since we never accepted empty strings there:
error: unknown compression method ''
The other one is used for optional .narinfo, .ls, and log/ compression.
Those treated empty strings as compression being disabled. The same exact
semantics is kept.
This has the benefit of improving error messages for incorrect values:
error: option 'compression' has invalid value 'bz'
Did you mean one of br, xz or lz4?
The docs were out of date. Since 8a0c00b856 Nix
supports all compression algorithms exposed by libarchive (if it's built with
native support for them). Let's be honest about it in the docs.
This avoids the wall of text like, because ThreadPool doesn't print interrupts
on shutdowns.
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
Without this we can abort by throwing an exception in the destructor:
[24/635/2958 copied (3.8/26.0 GiB)] copying path '/nix/store/ncd2iic2nwxwhqsf4gp9sdybkwnwz20b-ruby3.3-mini_portile2-2.8.9' from 'ssh-ng://localhost:22'
Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
Exception: nix::Interrupted: error: interrupted by the user
Stack trace:
0# 0x00000000004AFFE9 in result/bin/nix
1# 0x00007F946290A1AA in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
2# __cxa_call_terminate in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
3# __gxx_personality_v0 in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
4# 0x00007F946283FA19 in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libgcc_s.so.1
5# _Unwind_RaiseException in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libgcc_s.so.1
6# __cxa_throw in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
7# 0x00007F94635D82D0 in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
8# nix::Pid::wait() in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
9# nix::Pid::~Pid() in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
forced_unwind is thrown by Boost.Context when destroying the coroutine.
This lead to us resetting the remote connection for each narFromPath
with the ssh-ng:// store, so copying was very slow.
This makes all addToStore operations that use these source accessors
constant memory regardless of file sizes. Also make the other overload
altogether and relegate it to the base class as a non-virtual method to
avoid such mistakes.
This factors out the helper function from seekableGetNarBytes into copyFdRange
and adds some more sanity checks for offset/length truncation/wrapping at that
API boundary where we work with NAR-style offsets and convert to native off_t.
Instead of mutating the file pointer we can instead safely do
preads. That makes the local-nar-info cache once again thread safe
without the overhead of reopening the file that we used to have prior
to b9b6defca6 which broke the thread safety
by persisting the file descriptor.
Without the change the build fails for me as:
../unix/file-descriptor.cc:404:70: error: 'RESOLVE_BENEATH' was not declared in this scope
404 | dirFd, path.rel_c_str(), flags, static_cast<uint64_t>(mode), RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS);
| ^~~~~~~~~~~~~~~
This happens for 2 reasons:
1. `__NR_openat2` constant was not pulled in from the according headers
and as a result `<linux/openat2.h>` was not included.
2. `define HAVE_OPENAT2 0` build is broken: refers to missing
`RESOLVE_BENEATH` normally pulled in from `<linux/openat2.h>`
This changes fixes both.
Document the nix-cache-info file format used by binary caches, including
the StoreDir, WantMassQuery, and Priority fields, their behavior, and
links to related store options.
This is the usual conventions on windows.
See https://learn.microsoft.com/en-us/windows/win32/shell/knownfolderid and
https://github.com/adrg/xdg for examples of the mapping of XDG paths to Windows
known folders.
Additionally, on Windows, this allows us to dispense with a hard-coded
default for `nixConfDir`, which is both nice (fewer compile-time
configuration options) and necessary, because we don't know what drive
the `ProgramData` directory will live on.
Tested on wine.
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
The test data was using invalid signature strings like "asdf" and
"qwer" which don't follow the required "name:base64signature" format.
This updates them to use properly formatted signatures with valid
base64-encoded data.
Document the Nix32 base-32 variant used for store path digests and
hash output. The new page covers:
- The 32-character alphabet (omitting e, o, u, t)
- Byte order differences from base-16 encoding
Also update references throughout the manual to link to the new page.
Change "cannot build missing derivation" to "failed to obtain derivation of"
since the path (e.g. '...drv^out') is a derivation output, not a derivation.
The message could be improved further to resolve ambiguity when multiple
outputOf links are involved, but for now we err on the side of brevity
since this message is already merged into larger error messages with
other context from the Worker and CLI.
When !keepGoing and a goal fails, other goals are cancelled and
remain with exitCode == ecBusy. These cancelled goals have a default
BuildResult::Failure{} with empty errorMsg.
Previously, buildPathsWithResults would return these cancelled goals,
and throwBuildErrors would report them as failures. When only one such
cancelled goal was present, it would throw an error with an empty
message like:
error: build of '/nix/store/...drv^*' failed:
Now we skip goals with ecBusy since their state is indeterminate.
Cancelled goals could be reported, but this keeps the output relevant.
Other indeterminate goal states were already not being reported, for
instance: derivations that weren't started for being blocked on a
concurrency limit, or blocked on a currently building dependency.
When keepGoing=false and a build fails, other goals are cancelled.
Previously, these cancelled goals were reported in the "build of ...
failed" error message alongside actual failures. This was misleading
since cancelled goals didn't actually fail - they were never tried.
Update the test to expect only the actual failure (hash mismatch) to
be reported, not the cancelled goals.
DerivationTrampolineGoal is the top-level goal whose buildResult is
returned by buildPathsWithResults. When it failed without setting
buildResult.inner, buildPathsWithResults would return failures with
empty errorMsg, producing error messages like:
error: failed to build attribute 'checks.x86_64-linux.foo',
build of '/nix/store/...drv^*' failed:
(note the empty message after "failed:")
Use the new doneFailure helper to ensure buildResult is populated
with meaningful error information.
There can be a long time between the creation of `TransferItem` and
the start of the curl download, which can lead to misleading download
durations and progress bar status. So now we create the `Activity` and
update `startTime` when curl actually starts the download.
Previously, calling queryValidPaths() with a large number (e.g. 100K)
of store paths failed because Nix immediately creates a `TransferItem`
for each .narinfo, which is then registered as a handle with
curl. However curl appears to scale poorly internally: even though
only a few downloads are actually started (up to the
connections/streams limits), it spends a lot of CPU time dealing with
the inactive handles. So the curl thread is sitting at 100% CPU, the
active downloads stall and time out, and everything grind to a halt.
So now we limit the number of curl handles to http-connections *
5. With this, fetching 100K .narinfo files from localhost succeeds in
~15 seconds.
This operation has been deprecated since
09a6321aeb (July 2012). It was used by
client versions <= 11, which is below `MINIMUM_PROTOCOL_VERSION`
(currently 18).
We can get rid of `NarMember`, because it is just `NarListing` in
disguise! The use of `std::variant` makes clear that certain stat fields
we don't care about in the non-regular-file case too.
Relative paths (e.g., "relative/repo") would crash in renderAuthorityAndPath()
because an empty authority was set, violating RFC 3986 section 3.3 which
requires paths to start with "/" when an authority is present.
Fix by only setting authority for absolute paths:
- Absolute paths: file:///path (empty authority)
- Relative paths: file:path (no authority)
Also reject SCP-like URLs without a user (e.g., "github.com:path") with a
clear error message, since proper support requires careful implementation,
which is not something I can do right now.
Relative paths like `file:./foo.tar.gz` have never worked for tarballs
because curl rejects relative file: URLs. Previously this resulted in
cryptic curl errors. Now we reject them early with a clear message
explaining that relative paths are not supported because there is no
defined base directory to resolve them against.
See https://github.com/NixOS/nix/issues/12281
When a goal with an active child process is destroyed (e.g., during
failure cascades without `--keep-going`), the child process gets killed
but `childTerminated` was never called. This left stale entries in the
worker's `children` list.
Fix this by ensuring `childTerminated` is called from destructors:
- `DerivationBuilderImpl::killChild` now calls `childTerminated` via
the `miscMethods` callback.
- `HookInstance` gains an `onKillChild` callback that is invoked from
its destructor when killing the process. `buildWithHook` sets this
callback to call `childTerminated`.
To make these calls safe from destructors (where the goal object may be
partially destroyed), add a new overload of `Worker::childTerminated`
that takes an explicit `JobCategory` parameter instead of calling the
virtual method `Goal::jobCategory`. The original overload still exists
for convenience for normal (non-destructor) call sites.
Add helpers to the base Goal class that set buildResult and call amDone,
ensuring buildResult is always populated when a goal terminates.
Derived class helpers now call the base class versions. This reorders
operations: previously buildResult was set before bookkeeping (counter
resets, worker stats), now it's set after. This is safe because the
bookkeeping code (mcExpectedBuilds.reset(), worker.doneBuilds++,
worker.updateProgress(), etc.) only accesses worker counters, not
buildResult.
It correctly models the is-a relation. This will be useful for doing a dynamic_cast in
downstream code that wants to copy from a file descriptor to a file descriptor.
Nix currently doesn't do any resource control, and Delegate=yes turns on all the controllers.
In particular, this enables using cpusets with cgroups V1 alongside the Nix daemon.
This also adds a utility for opening a file descriptor from a path in readonly mode.
Previous commit helps a bit with error handling, since now we just throw a NativeSysError.
This way each consumer of NativeSysError doesn't have to
also conditionally include the windows-error.hh, which is very cumbersome.
And we can't include windows-error.hh in error.hh because of a circular import.
Previously builtins.readDir would return an empty attribute set
instead of barfing on non-existent paths. This is a regression from
2.32 for impure eval.
Using fchmodat after a fstatat in deletePath has a slight TOCTOU
window. We can plug it by using fchmodat (the libc wrapper with
AT_SYMLINK_NOFOLLOW), but it tries fchmodat2 and falls back to the
O_PATH trick while failing when procfs isn't mounted. We can do a bit
better than that and also cache whether syscalls are unsupported to
avoid the repeated context switching that glibc would impose.
Also tests the fallback path. It's only for kernels older than 6.6 and
when procfs isn't accessible that we fall back to the racy fchmodat
without AT_SYMLINK_NOFOLLOW.
What previously used to be:
openat(AT_FDCWD, "/tmp/store-race/nix/var/nix/builds", O_RDONLY) = 11
newfstatat(11, "nix-2704212-84654554", {st_mode=S_IFDIR|000, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(11, "nix-2704212-84654554", 040700) = 0
Is now a TOCTOU-free sequence of syscalls:
openat(AT_FDCWD, "/tmp/store-race/nix/var/nix/builds", O_RDONLY) = 11
newfstatat(11, "nix-2704953-1733606057", {st_mode=S_IFDIR|000, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat2(11, "nix-2704953-1733606057", 040700, AT_SYMLINK_NOFOLLOW) = 0
Or if the fchmodat2 is not supported:
openat(11, "nix-2705443-3010460784", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 12
fstat(12, {st_mode=S_IFDIR|000, st_size=3, ...}) = 0
chmod("/proc/self/fd/12", 040700) = 0
openat(11, "nix-2705443-3010460784", O_RDONLY|O_NOFOLLOW|O_DIRECTORY) = 12
This prevents a potentially arbitrary chmod that follows symlinks,
though the race window is very small. Also in the case that fchmodat2
isn't supported we could instead open the /proc/self/fd/N path instead
of using openat, but that's pretty much equivalent. We only care
about ensuring that the thing we chmodded wasn't a symlink since
fchmodat follows symlinks and the support for AT_SYMLINK_NOFOLLOW
in libc for that is pretty spotty on Linux. E.g. glibc fails if the
AT_SYMLINK_NOFOLLOW is specified and procfs isn't available even on
regular files. The patch also includes a test that uses a user namespace
on Linux to test this exact scenario (though it's rather exotic).
This makes the logic much easier to follow. Unlike before, the use of
separate functions is not making us pass a gazillion arguments or use
the crutch of class variables.
There was a bunch of logic in there which was, effectively, using the
build hook, rather than deciding *whether* to use the build hook. We
want it to only be the latter.
Pulling in the java into the tests closure for just testing a piece of code
for the docs (and the tests actually are wrong, since a correct parser must *reject*
those NARs). This is too much of an ask to maintain for zero benefit. I already had
to disable it basically everywhere, because it works only on linux.
It can be revisited in the future, but considering that it's not exercised anywhere and
shouldn't be used anywhere other than a toy example for the docs I think it's best to drop
it.
This is no longer needed (best I can tell), since nix docker
images now get uploaded to GHCR as part of the release process too
and they contain both aarch64 and x86_64 instead of only x86_64.
I messed up and accidentally configured the S3 client to use the same
host as the nix-releases bucket, but nix-channels is us-east-1 and
nix-releases is eu-west-1.
This workflow is supposed to automate release uploads by using OIDC
for AWS setup. DockerHub still uses long-lived credentials, but that's
not fixable. In a follow-up we could set up release uploads to GHCR too.
Previously it was only Eeclo doing releases that were signed with
B541D55301270E0BCF15CA5D8170B4726D7198DE. Other linux distributions
have the expectation (rightfully so) that our tags are signed. Let's
document this.
We could do cross-signing to make tracing the chain of trust easier
for all Nix team members [1].
[1]: https://nixos.org/community/teams/nix/
This allows for testing with a local minio deployment like:
./upload-release.pl --skip-docker --skip-git --s3-endpoint http://localhost:9000 --s3-host localhost:9000 1821360
Add a test case that explicitly demonstrates NAR hashing of a directory
without using a filter. Add comments to clarify what each test case is
testing (NAR vs flat hashing).
The sha256 parameter documentation said "file at the path" but it
works with directories too (using NAR hashing). Link to the
content-address documentation instead of duplicating information.
Remove include of signals.hh from signals-impl.hh to fix
misc-header-include-cycle warning. The impl header is only included
from signals.hh which already provides the necessary declarations.
When using designated initializers, clang-tidy warns about skipped
fields. Explicitly initialize pos to {} to silence the
clang-diagnostic-missing-designated-field-initializers warning.
Runs the tests against the new daemon as well as the cli.
This more reliably shares the artifact (not relying directly on github
actions cache). We've seen github evict our caches super fast, so it would
be nice to move away from it entirely if possible.
Best reviewed with -w --color-moved. This just moves the code
into a separate workflow. This will allow us to reuse it in
the release job for github releng of releases.
When a remote SSH client disconnects during a long-running operation
like addToStore(), the nix-daemon can deadlock in a circular wait:
- Process A (SSH daemon): blocked reading from downstream store socket,
waiting for response from local daemon
- Process B (local daemon): blocked reading from upstream socket,
waiting for more NAR data from SSH daemon
The existing interrupt mechanism (ReceiveInterrupts + MonitorFdHup)
correctly detects the SSH disconnect and sets _isInterrupted, but the
daemon remains blocked in read() on the downstream store connection.
Even though SIGUSR1 causes read() to return EINTR, the circular
dependency prevents forward progress.
Fix this by adding shutdownConnections() to RemoteStore that calls
shutdown(fd, SHUT_RDWR) on all tracked connection file descriptors.
Register an interrupt callback in processConnection() that invokes
this method when the store is a RemoteStore. This causes any blocking
read() to return 0 (EOF), breaking the circular wait and allowing
both processes to exit cleanly.
The fix tracks connection FDs in a synchronized set, populated when
connections are created by the Pool factory. On interrupt, all FDs
are shut down regardless of whether they're idle or in-use.
Makes the error messages render paths correctly, also introduces
a new hierarchy of error classes for SourceAccessor related errors
that we might want to handle differently (e.g. like when doing a readFile
on a directory and such). This should make it easier to implement better
UnionSourceAccessor and AllowListSourceAccessor by catching these errors
consistently.
See #8188. Resolves issues about the error not
being actionable, but I am not marking it closing
yet because of further discussion about the naming
of these flags in the thread.
`nix build --rebuild` (and others)
will fail if the derivation has not been built
before, because it runs a check build and
confirms that the build was deterministic.
It may be unclear to users that --rebuild will fail
if the derivation has never been built before,
because the flag makes no indication that a
determinism check occurs.
The error message does
not help clear this up, or provide any actionable
steps, and at first glance seems to indicate that
the derivation being built is invalid, rather than
just not present in the store:
```
error: some outputs of '...' are not valid, so checking is not possible
```
We can suggest to the user the following (correct)
rewrites. This list of commands that may result in
the error is comprehensive.
- `nix build --rebuild` to `nix build` or `nix build --repair`
- `nix-build --check` to `nix-build` or `nix-build --repair`
- `nix-store --realise --check` to `nix-store --realise` or `nix-store --realise --repair`
Wording is based on that in the documentation:
```
(nix build)
--repair During evaluation, rewrite missing or
corrupted files in the Nix store. During
building, rebuild missing or corrupted
store paths.
(nix-build)
--repair Fix corrupted or missing store paths by
redownloading or rebuilding them. Note
that this is slow because it requires
computing a cryptographic hash of the
contents of every path in the closure
of the build. Also note the warning
under nix-store --repair-path.
(nix-store --realise)
--repair Fix corrupted or missing store paths by
redownloading or rebuilding them. (etc)
```
Make the C API error message more explicit about what went wrong and
why it's invalid. The new message explains that a zero-length path was
passed and clarifies that it would refer to the flake itself.
Updates the unit test to match the new error message.
Make the error message more explicit about what went wrong and why
it's invalid. The new message explains that a zero-length path was
passed and clarifies that it would refer to the flake itself.
An empty path refers to the flake itself, not an input. Apply the same
type safety to inputUpdates as inputOverrides.
The deprecated --update-input flag (deprecated since Nix 2.4) and the
modern 'nix flake update' command now properly reject empty paths.
Includes functional tests for both commands.
Wraps InputAttrPath with compile-time guarantee of non-emptiness.
Replaces obscure .back() calls with domain-specific inputName() method.
An empty path refers to the flake itself, making it nonsensical for
input override operations. The type system now prevents this.
- getDerivations() filters attribute names with std::regex_match, which runs the regex engine for every attribute visited during nixpkgs scanning.
- BM_GetDerivationsAttrScan/10000_mean: 3.338 ms → 1.506 ms (≈ -54.9%)
- RegexCache::get() returned std::regex by value, copying the compiled regex on every cache hit.
- Store the compiled regex behind std::shared_ptr<const std::regex> and return the shared pointer instead, so callers reuse the same compiled object.
- BM_EvalManyBuiltinsMatchSameRegex_mean improved about 8%
Remove the per-call reserve() inside printString to avoid linear-growth reallocations when called in loops (e.g. printStrings). Derivation::unparse already pre-reserves a large buffer, so this remains efficient while preserving amortized growth behavior when the initial estimate is exceeded.
Testing with 10 derivations is sufficient to verify performance
characteristics. The larger test cases (50, 200) don't provide
additional insight and slow down the benchmark unnecessarily.
The separate checkInvariants loop after addValidPath was added in 2014
(d210cdc43) to work around an assertion failure:
nix-store: derivations.cc:242: Assertion 'store.isValidPath(i->first)' failed.
At that time, hashDerivationModulo() contained assert(store.isValidPath(...))
which required input derivations to be registered as valid in the database
before computing their hash. The workaround was to:
1. Call addValidPath with checkOutputs=false
2. Add all references to the database
3. Run checkInvariants in a separate loop after paths were valid
In 2020 (bccff827d), the isValidPath assertion was removed to fix a
deadlock in IFD through the daemon (issue #4235). The fix changed
hashDerivationModulo to use readInvalidDerivation, which reads directly
from the filesystem without requiring database validity.
This made the separate checkInvariants loop unnecessary, but nobody
noticed the code could be simplified. The comment "We can't do this in
addValidPath() above, because the references might not be valid yet"
became stale.
Now we simply call addValidPath() with the default checkOutputs=true,
which runs checkInvariants internally using the already-parsed
derivation. This commit eliminates the separate loop over derivations.
- LocalStore::registerValidPaths() parsed derivations twice: once in addValidPath() and again when calling checkInvariants(), despite already having loaded the derivation.
- Plumb the parsed Derivation out of addValidPath() and reuse it for the invariant check pass, falling back to re-parsing only when a derivation wasn’t newly registered in this call.
- BM_RegisterValidPathsDerivations/200_mean runs 32% faster
rsync was only used to copy source files while following symlinks.
Replace with tar --dereference, which serves the same purpose.
Tried plain cp but couldn't get it to work reliably. tar is already
a test dependency.
Add tests/functional/derivation to fileset to include the symlink
targets.
Fixes#14776
While working on #12464, I realized this method was not correct in this
case. With the current binary cache format, it is harmless, since we
don't create arbitrary directories, but with my change, we started to.
Regardless of whether we need it or not, I think it is better if the
function just does the right thing.
For windows we should live fully in the HANDLE land instead
of converting back-n-forth (which sometimes is destructive).
Using native API is much better for this.
It doesn't track the number of bytes deleted, but since this code is
security critical also we can split unix and windows implementations.
If the need arises we can implement a smarter recursive deletion function
ourselves in the future.
Review with --color-moved.
This at least makes canonPath not consider the drive letter as a path
component. There still some issues with it on windows, but at least
this gets us through some of the libutil-tests.
Also since we don't want to change which env variables nix considers
we don't use std::filesystem::temp_directory_path and implement the
windows version directly.
- Skip packages that don't build for Windows when building for windows
- Automatically disable kaitai / json schema, fixing todo
- Skip native build of Nix for manual
Should be pretty self-explanatory. We didn't really have unit tests
for the filesystem source accessor. Now we do and this will be immensely
useful for implementing a unix-only smarter accessor that doesn't suffer
from TOCTOU on symlinks.
We now have a nice separation of concerns: `DrvOutputSubstitutionGoal`
is *just* for getting realisations, and `PathSubstitutionGoal` is just
for fetching store objects.
The fetching of store objects that this used to do is now moved to the
caller.
This progress on #11896. It introduces some issues temporarily which
will be fixed when #11928 is fixed.
The SQL tables are left in place because there is no point inducing a
migration now, when we will be immediately landing more changes after
this that also require schema changes. They will simply be ignored by in
this commit, and so all data will be preserved.
Error messages now include suggestions like:
error: unknown compression method 'bzip'
Did you mean one of bzip2, gzip, lzip, grzip or lrzip?
Also a bit of progress on making the compression code use less stringly
typed compression type, which is good because it's easy to confuse
which strings are accepted where (e.g. Content-Encoding should be able
to accept x-gzip, but it shouldn't be exposed in NAR decompression and
so on). An enum cleanly separates the concerns of parsing strings / handling
libarchive write/read filters.
The previous error message was ambiguous about which specific directory failed the check.
This commit updates checkNotWorldWritable to return the failing path so it can be included in the error message, making debugging easier.
Ref #14787
This really doesn't really fixes the problem of the symlink, but it
solves the progress of getting windows working.
TODO: find out if it's a bug from meason & make a feature request to
avoid symlinks or generate symlinks upon build and git ignore, but still
goes back to the issue of is this a bug or do we need to make a feature
requests.
Co-authored-by: John Ericson <git@JohnEricson.me>
- More concise
- Also checks error messages
- Checks more error codes
The nature of that bug is that if the first command's exit status is
correctly 101 and not 1, the rest should be correctly 101, 100, etc.
too.
Unfortunately previous tarball caches had loose objects written to
them and subsequent switch to thin packfiles. This results in possibly
broken thin packfiles when the loose objects backend is disabled. Thin
packfiles do not necessarily contain the whole closure of objects.
When packfilesOnly is true we end up with an inconsistent state where
a tree lives in a packfiles which refers to a blob in the loose objects
backend.
In the future we might want to nuke old cache directories and repack
the tarball cache.
The SSO provider was unconditionally setting profile_name_override to
the (potentially empty) profile string from the S3 URL. When profile
was empty, this prevented the AWS CRT SDK from falling back to the
AWS_PROFILE environment variable.
Only set profile_name_override when a profile is explicitly specified
in the URL, allowing the SDK's built-in AWS_PROFILE handling to work.
The default (empty) profile case was using CreateCredentialsProviderChainDefault
which didn't properly support role_arn/source_profile based role assumption via
STS because TLS context wasn't being passed to the Profile provider.
This change unifies the credential chain for all profiles (default and named),
ensuring:
- Consistent behavior between default and named profiles
- Proper TLS context is passed for STS operations
- SSO support works for both cases
Add validation for TLS context and client bootstrap initialization,
with appropriate error messages when these fail. The TLS context failure
is now a warning that gracefully disables SSO, while bootstrap failure
throws since it's required for all providers.
This enables seamless AWS SSO authentication for S3 binary caches
without requiring users to manually export credentials.
This adds SSO support by calling aws_credentials_provider_new_sso() from
the C library directly. It builds a custom credential chain: Env → SSO →
Profile → IMDS
The SSO provider requires a TLS context for HTTPS connections to SSO
endpoints, which is created once and shared across all providers.
Good to explicitly declare things to not accidentally do twice the work by
preventing that kind of misuse.
This is essentially just cppcoreguidelines-special-member-functions lint
in clang-tidy.
This is mostly theoretical, but the code was calling getenv("TZ")
twice: once to check if it's non-null, and again to get its value.
This creates a potential race condition where the environment could
change between calls.
The nix_api_store.cc tests and derivation-parser-bench.cc were using raw
getenv() calls or unsafe .value() calls on optional, which would segfault
when passed to std::filesystem::path constructor if the
_NIX_TEST_UNIT_DATA environment variable was not set.
The previous implementation called .value() on std::optional without
checking if it had a value. When _NIX_TEST_UNIT_DATA was not set, this
would throw std::bad_optional_access or cause a segfault in code that
used the raw getenv() result.
The new implementation checks the optional first and throws an Error
with a helpful message directing users to run tests via meson. The
example includes --gdb since this situation may arise when trying to
debug tests without knowing about meson's test infrastructure.
Pretty self-explanatory. More RAII is good and unclutters the already heavily overloaded
destructors from ownership logic. Not yet touching CURL *req because that would be too churny.
Fetching gcc-15.2.0.tar.gz I get a warning about UTF8 archive names. This
now mentions problematic pathnames.
warning: getting archive member 'gcc-15.2.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Äfoo.go': Pathname can't be converted from UTF-8 to current locale.
warning: getting archive member 'gcc-15.2.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Ämain.go': Pathname can't be converted from UTF-8 to current locale.
Also apparently libarchive depends on locale (yikes). Fixing reproducibility issues
that stem from this is a separate issue. At least having the warning actually mention
the pathname should be useful enough even though it's not actionable.
At least using the default locale yields something sane:
builtins.readDir "${gcc}/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir"
{
"Äfoo.go" = "regular";
"Ämain.go" = "regular";
}
Fix intermittent SIGSEGV (exit code 139) on macOS when running
nix-shell and shebang tests inside the nix sandbox.
The foo, bar, and ruby test scripts were created without shebangs,
which causes intermittent crashes when executed via command
substitution on macOS. Adding proper shebangs resolves the flakiness.
Potentially closes: #13106
This matches what we just did for `nix path-info`, and I hope will allow
us to avoiding any more breaking changes to this command for the
foreseeable future.
Previously the daemon didn't validate that it got a valid GCAction
and did a naive C-style cast to the enum. This is certainly unintentional,
albeit mostly harmless in practice.
The fact that we were introducing a conversion from the output of `nix
path-info` into the input of `builtins.fetchTree` was the deciding
factor. We want scripting outputs into inputs like that to be easy.
Since JSON strings and objects are trivially distinguishable, we still
have the option of introducing the JSON format as an alternative input
scheme in the future, should we want to. (The output format would still
be SRI in that case, presumably.)
Floating CA outputs just have a hash algorith, not a whole hash. It is
fixed ones which are a pair of a method and a hash, just like the `ca`
field of store object info.
This restores compatibility with Nix 2.18, which behaved this
way. Note that this doesn't scan for the actually visible references.
Unlike in Nix 2.18, we only do this for paths with context, i.e. it
applies to `builtins.storePath "/nix/store/bla..."` but not
`"/nix/store/bla..."`. We don't want the latter because it shouldn't
matter whether a source file happens to be in the Nix store.
This was lost after 2.32 while making the accessor lazy. We can restore the support
for it pretty easily. Also this is significant optimization for nix nar cat.
E.g. with a NAR of a linux repo this speeds up by ~3x:
Benchmark 1: nix nar cat /tmp/linux.nar README
Time (mean ± σ): 737.2 ms ± 5.6 ms [User: 298.1 ms, System: 435.7 ms]
Range (min … max): 728.6 ms … 746.9 ms 10 runs
Benchmark 2: build/src/nix/nix nar cat /tmp/linux.nar README
Time (mean ± σ): 253.5 ms ± 2.9 ms [User: 56.4 ms, System: 196.3 ms]
Range (min … max): 248.1 ms … 258.7 ms 12 runs
The whole NarAccessor -> listing -> lazy NarAccessor is very weird. Source
can now be seek-ed over when supported, so we can support it pretty easily.
Alternatively we could also make it single-pass very easily with a custom
FileSystemObjectSink. It will get removed in a follow-up commit anyway.
It's not evaling on hydra currently and is only part of the docs. Support
for this it best effort at best so we should not build this in hydra, considering
the amount of effort required to support this. I would even consider dropping these
checks and component altogether, since there doesn't seem to be much interest in maintaining
these docs from the core team anyway.
This somehow fails https://hydra.nixos.org/build/315675349/nixlog/1. I don't know the exact
details, but it seems that something goes very wrong with LTO and sanitizers that lead to the
string literal to be moved? Instead of relying on the string literal deduplication to provide
a consistent address we can use a global. That should have a single address (modulo wonky copy
relocations).
I didn't do things quite right in 496e43ec72:
- Forgot to remove the now-redundant `isAllowed` check.
- Called the non-virtual, not the superclass's impl, in
`addDependencyPrep`, causing bad recursion / UB.
Doing this fixes a crash I encountered with manual testing an Nix Ninja
--- hopefully we will get Nix Ninja or similar in a NixOS test longer
term to defend against this thing happening again.
Add tests for function equality covering both direct comparisons and
comparisons within composite types (lists and attribute sets).
Tests verify:
- Direct function comparisons always return false
- Value identity optimization in composite types allows identical
functions to compare as equal when both references point to the
same function value
Correct and clarify evaluation semantics including to help users
understand Nix language behavior without unnecessarily pinning down
the implementation.
Add `html-manual` Meson option to allow building manpages without the
HTML manual, removing the mdbook dependency for manpage-only builds.
Changes:
- Add `html-manual` Meson option (default: true)
- Make HTML manual build conditional in meson.build
- Add `buildHtmlManual` parameter to package.nix
- Conditional outputs: ["out" "man"] when enabled, ["out"] when disabled
- Make mdbook/rsync/json-schema-for-humans dependencies conditional
- Add `nix-manual-manpages-only` package variant
This allows systems that only need manpages to avoid the mdbook build
dependency while preserving full functionality for HTML manual builds.
Add configurable documentation URLs that change based on whether this is
an official release or development build:
- Nix builds:
- Development (officialRelease = false): Use /latest/ URLs
- Official releases (officialRelease = true): Use versioned URLs with
MAJOR.MINOR only (e.g., /2.33/ instead of /2.33.0/)
- Plain meson builds: Default to versioned URLs (official-release = true)
Changes:
- Add --doc-url parameter to expand-includes.py
- Add meson option 'official-release' (defaults to true for Meson builds)
- Compute doc_url in meson.build based on version and official-release
- Forward Nix officialRelease variable to Meson in package.nix
- Update render-manpage.sh to pass doc-url parameter
This allows distros (Fedora, etc.) to have stable versioned URLs by default,
while Nix development builds point to /latest/ for up-to-date documentation.
Add standalone markdown preprocessor to generate manpages without requiring
mdbook's Rust toolchain. This removes a significant build dependency for
manpage generation while keeping the HTML manual (mdbook) working unchanged.
Changes:
- Add expand-includes.py: Python 3 script that recursively expands
{{#include}} directives, resolves @docroot@ to nix.dev URLs, and handles
@generated@/ paths for build-generated files
- Update render-manpage.sh: Replace mdbook-based implementation with
standalone version that uses expand-includes.py + lowdown
- Update meson.build: All 134 manpage targets now use standalone renderer
with proper dependencies (expand-includes.py, experimental-features-shortlist)
- Fix nix-hash.md: Remove extra parenthesis in markdown link syntax
Benefits:
- No mdbook/Rust toolchain required for manpage builds
- Manpages contain nix.dev/latest URLs instead of broken relative paths
- Fixes bug where mdbook didn't expand experimental-features-shortlist.md
- 98.5% identical output to mdbook (2 files differ, both acceptable)
All 134 manpages (131 section 1, 2 section 5, 1 section 8) build successfully.
We have the machinery to make a more informative error, telling the
user what format was actually encountered, and not just that it is not
the format that was requested.
The S3BinaryCacheStoreConfig::doc() function was returning a minimal
hardcoded 3-line string instead of including the comprehensive
documentation from s3-binary-cache-store.md.
This was introduced in PR #13752 which moved the prose documentation to
the markdown file but forgot to update the doc() function to use it.
The previous commit hacked it into the output of `nix path-info --json`,
this cleans that up my making it an actual field of that data type, and
part of the canonical JSON serializers for it (and `ValidPathInfo` and
`NarInfo`).
Beyond cleaning up the JSON code, this also opens the doors to things
like:
- Binary caches that contain store objects that don't all belong in the
same store directory
- Relocatable store objects which carefully don't mention any store
directory by absolute path, and instead use relative paths for
anything. (#9549)
As discussed today at great length in the Nix meeting, we don't want to
break the format, but we also don't want to impede the improvement of
JSON formats. The solution is to add a new flag for control the output
format.
Note that prior to the release, we may want to replace `--json
--json-format N` with `--json=N`, but this is being left for a separate
PR, as we don't yet have `=` support for CLI flags.
Fix#14532.
As discussed on the call today:
1. We'll stick with `format = "base16"` and `hash = "<hash>"`, not do
`base16 = "<hash>"`, in order to be forward compatible with
supporting more versioning formats.
The motivation we discussed for someday *possibly* doing this is
making it easier to write very slap-dash lang2nix tools that create
(not consume) derivations with dynamic derivations.
2. We will remove support for non-base16 (and make that the default, not
base64) in `Hash`, so this is strictly forward contingency, *not*
yet something we support. (And also not something we have concrete
plans to start supporting.)
This leads to incredibly wasteful refreshes (see [^]) when oids are not found.
Since we are writing the pack files only once per unpacking we should not bother
with this refreshing at all.
This brings down the number of syscalls during `nix flake metadata "https://releases.nixos.org/nixos/25.05/nixos-25.05.813095.1c8ba8d3f763/nixexprs.tar.xz" --store "dummy://?read-only=false"`
Down from 576334 to just 6235 (100x less syscalls):
(Before)
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ------------------
32.98 0.625288 3 162898 getdents64
29.58 0.560686 3 163514 81917 openat
15.01 0.284509 3 81819 186 newfstatat
10.99 0.208349 2 81601 close
10.56 0.200145 2 81552 fstat
All these are coming from [2] and are totally useless.
(After)
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ------------------
76.47 0.108558 247 438 20 futex
6.55 0.009292 18 513 munmap
3.30 0.004680 7 639 492 openat
2.68 0.003803 10 359 write
2.30 0.003268 2 1146 read
2.26 0.003215 3 870 mmap
[^]: 58d9363f02/include/git2/sys/odb_backend.h (L68-L75)
[2]: 58d9363f02/src/libgit2/odb_pack.c (L517-L546)
Fixes#14628
- Remove mdbook-linkcheck dependency and configuration (was blocking
upgrades to mdbook 0.5.0+, configured with warning-policy = "ignore"
due to false positives, and redundant with lychee-based link checking)
- Update substitute.py and anchors.jq to handle 'items' (mdbook 0.5.x)
in addition to 'sections' (mdbook 0.4.x), as per mdbook 0.5.0
changelog: "Book::sections was renamed to Book::items"
https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#05-migration-guide
Instead we can just seek back in the file - duh. Also this makes use
of the anonymous temp file facility, since that is much safer (no need
window where the we don't have an open file descriptor for it).
With the addition of "delete" method we can no longer rely on
just concatenating "ing" to get the continuous form of the verb.
Also some use-cases actually need a noun instead.
queryRealisationUncached was crashing with an assertion failure when
ca-derivations experimental feature is not enabled, because the SQLite
statements for realisations are only initialized when ca-derivations
is enabled.
Return nullptr (no realisation found) when ca-derivations is disabled,
matching the behavior of other CA-related functions like registerDrvOutput
which check for the feature before proceeding.
They are flagged by Fedora CI checks as a potential security issue.
Use of such raw Right-to-Left control characters in source code is
strongly discouraged
also update release-credits-handle-to-name.json
We can precompute the exact information we need for topo sorting and
store it in `PerhapsNeedToRegister`. Depending on how `topoSort` works,
this is easy a performance improvement or just completely harmless.
Co-Authored-By: Bernardo Meurer Costa <beme@anthropic.com>
Build the inverse of `scratchOuputs` before running topoSort, avoiding
quadratic complexity when determining which outputs reference each
other. This fixes the FIXME comment about building the inverted map up
front.
Inspired by Lix commit 10c04ce84 / Change Id
Ibdd46e7b2e895bfeeebc173046d1297b41998181, but ended up being completely
different code.
Co-Authored-By: Maximilian Bosch <maximilian@mbosch.me>
Co-Authored-By: Bernardo Meurer Costa <beme@anthropic.com>
This was failing under ASAN in https://hydra.nixos.org/build/315173638/nixlog/1.
ASAN uses a bit more stack space and the default max call depth is not enough.
Not sure what's so special about this particular test.
This test is now pointless and the comment is outdated. Also the test fails
on 32 bit systems with:
../nix_api_value_internal.cc:22: Failure
Expected equality of these values:
sizeof(nix::Value)
Which is: 12
sizeof(nix_value)
Which is: 8
It just happeneded to work because Value is 16 bytes and nix_value was also 16 bytes.
Also get rid of a pointless inline in new_nix_value, since it's already static and
inline there does nothing.
When an SSH connection dies during a remote build, MonitorFdHup correctly
detects the disconnect and calls triggerInterrupt(). However, without
ReceiveInterrupts instantiated, no SIGUSR1 is sent to interrupt the
blocking read() syscall. This causes the daemon to hang indefinitely
while holding file locks, blocking subsequent builds.
The fix instantiates ReceiveInterrupts in processConnection(), which
registers a callback to send SIGUSR1 to the current thread when
triggerInterrupt() is called. This allows the blocking read() to return
with EINTR, causing checkInterrupt() to throw and the daemon to exit
cleanly.
This pattern is already used in ThreadPool::doWork() and
SubstitutionGoal for the same purpose.
Clarifies that the first positional argument is always treated as the
installable, even after --. Adds tests to prevent accidental change.
Addresses https://github.com/NixOS/nix/issues/13994
Previously inputFromAttrs() didn't do any validation. inputFromURL()
now calls inputFromAttrs(), so we only need to validate in one place.
Fixes#14655.
It's very weird it doesn't work here, but I don't mind not debugging
this now as I just added this part of the functional test --- it's
already better than it was before.
Add several new functions to the C API:
StorePath operations:
- nix_store_path_hash: Extract the hash part from a store path
- nix_store_create_from_parts: Construct a store path from hash and name
Derivation operations:
- nix_derivation_clone: Clone a derivation
- nix_derivation_to_json: Serialize a derivation to JSON
Store operations:
- nix_store_drv_from_store_path: Load a derivation from a store path
Test the new functions, and improve documentation of some existing
functions to better distinguish them, also.
Co-authored-by: Tristan Ross <tristan.ross@determinate.systems>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Before, had some funny logic with an unnecessary is CA enabled branch,
and erroneous use of the comma operator. Now, take advantage of the new
`Derivation::fillInOutputPaths` to fill in input addresses (and output
path env vars) in a much-more lightweight manner.
Also, fix `nix develop` on fixed-output derivations so that weird things
don't happen when we have that experimental feature enabled.
As a slight behavior change, if the original derivation was
content-addressing this one will be too, but I really don't think that
matters --- if anything, it is a slight improvement for users that have
already opted into content-addressing anyways.
When this function is called as a coroutine (e.g. when it's called by
`copyStorePath()`), the code after `decompressor->finish()` is never
reached because the coroutine is destroyed when the caller reaches the
end of the NAR. So put that code in a `LambdaSink` destructor.
Implements a safe no symlink following primitive operation for opening file descriptors.
This is unix-only for the time being, since windows doesn't really suffer from symlink
races, since they are admin-only.
Tested with enosys --syscall openat2 as well.
I missed this because I assumed all Exprs were recursed into by
bindVars, but ExprLet's ExprAttrs field is not really its own AST node,
so it doesn't get recursed into.
The single-string syntax '>=8.16.0 <8.17.0' only applied the lower
bound, causing curl 8.17.0 to be incorrectly rejected. Split into two
separate version_compare() calls for compatibility with Meson 1.1,
since multi-argument syntax requires Meson 1.8+.
printAmbiguous (used by nix-instantiate --eval and nix-env) had a depth
parameter, but all callers passed INT_MAX, effectively disabling the
limit. The function relied on the C++ stack to eventually overflow,
which could cause uncontrolled SIGSEGV crashes on deeply nested
pre-forced structures.
Now printAmbiguous checks depth against max-call-depth (default 10000)
and throws StackOverflowError with a proper trace, consistent with
other recursive value traversal functions.
The function signature is updated to take EvalState& to access the
settings and throw proper errors. The depth parameter now counts up
from 0 instead of down from INT_MAX.
Non-cyclic structures can be infinitely deep when values are lazily
produced (e.g., `let f = n: { inner = f (n + 1); }; in f 0`). Since f
returns immediately with a thunk, Nix call depth stays at 1, but
Printer::print recurses on the C++ stack when printing.
We check print depth against max-call-depth rather than incrementing
the callDepth counter, because accessing an attribute is not a call.
StackOverflowError is always re-thrown because stack overflow is a
serious condition that expressions should avoid, unlike say `throw`,
which can be part of legitimate expression patterns.
This partially reverts commit bc6b9ce.
This transformation is unsound and thread unsafe. Internal libgit2
structures must *never* be shared between threads. This causes
internal odb corruption with e.g.:
nix flake prefetch-inputs:
error:
… while fetching the input 'github:nixos/nixpkgs/89c2b2330e733d6cdb5eae7b899326930c2c0648?narHash=sha256-Stk9ZYRkGrnnpyJ4eqt9eQtdFWRRIvMxpNRf4sIegnw%3D'
error: adding a file to a tree builder: failed to insert entry: invalid object specified - upload-image.sh
error:
… while fetching the input 'github:NixOS/nixpkgs/a8d610af3f1a5fb71e23e08434d8d61a466fc942?narHash=sha256-v5afmLjn/uyD9EQuPBn7nZuaZVV9r%2BJerayK/4wvdWA%3D'
error: adding a file to a tree builder: failed to insert entry: invalid object specified - outline.nix
double free or corruption (!prev)
Thread 21 "nix" received signal SIGABRT, Aborted.
nix-instantiate on deeply nested structures with recurseForDerivations
(e.g., `let x = { recurseForDerivations = true; more = x; }; in x`)
caused an uncontrolled OS-level stack overflow with no Nix stack trace.
Fix by adding call depth tracking to getDerivations, integrating with
Nix's existing max-call-depth mechanism. Now produces a controlled
"stack overflow; max-call-depth exceeded" error with a proper stack
trace.
Without this dependency, e.g. `meson compile nix-instantiate`
would produce a broken symlink, or the `nix` it points to may be
stale.
With the dependency in place, `meson compile nix-instantiate`
produces a reliable outcome.
On FreeBSD, sysctl(KERN_PROC_PATHNAME) returns a null-terminated
string with pathLen including the terminator. This causes Nix to
fail during manual generation with:
error:
… while calling the 'concatStringsSep' builtin
at /nix/var/nix/builds/nix-63232-402489527/source/doc/manual/generate-settings.nix:99:1:
98| in
99| concatStrings (attrValues (mapAttrs (showSetting prefix) settingsInfo))
| ^
100|
error: input string '/nix/store/gq89cj02b5zs67cbd85vzg5cgsgnd8mj-nix-2.31.2/bin/nix␀'
cannot be represented as Nix string because it contains null bytes
The issue occurs because generate-settings.nix reads the nix binary
path from JSON and evaluates it as a Nix string, which cannot contain
null bytes. Normal C++ string operations don't trigger this since they
handle null-terminated strings correctly.
Strip the null terminator on FreeBSD to match other platforms (Linux
uses /proc/self/exe, macOS uses _NSGetExecutablePath).
Credit: @wahjava (FreeBSD ports and Nixpkgs contributor)
Since the root cause (the lack of backpressure control) has
been fixed in the previous commit we can revert the change from
8ffea0a018 and make the default size much
smaller.
Instead of naively stalling the download thread we can instead stop the transfer.
This allows the other multiplexed connections to continue downloading (and unpacking),
if the result of the download gets piped into a GitFileSystemObjectSink.
Prior art in lix project:
- 4ae6fb5a8f
- 12156d3beb
This patch is very different from the lix one, since we are using a decompression sink
in the middle of the pipeline but the co-authored-by is there since I was motivated to
implement this by looking at the lix side of things.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Similar to the deepSeq fix, toJSON on deeply nested structures caused
an uncontrolled OS-level stack overflow.
Fix by adding call depth tracking to printValueAsJSON.
When deepSeq encounters an error while evaluating a list element, the
error trace now includes the list index, making it easier to locate
the problematic element.
builtins.deepSeq on deeply nested structures (e.g., a linked list with
100,000 elements) caused an uncontrolled OS-level stack overflow with
no Nix stack trace.
Fix by adding call depth tracking to forceValueDeep, integrating with
Nix's existing max-call-depth mechanism. Now produces a controlled
"stack overflow; max-call-depth exceeded" error with a proper stack
trace.
Closes: https://github.com/NixOS/nix/issues/7816
this is a painful change. we should really add EvalState or EvalMemory
as an argument to various functions as we need it, but because we want
to preserve the stablity API, we hack it in as a field of nix_value.
The use of sourceToSink is an unnecessary serialization bottleneck.
While we are at it, fix the copyRecursive implementation to actually copy
the whole directory. It wasn't used for anything prior, but now it has a use
and accompanying tests for flake clone.
Also do a better JSON and testing for deep and shallow NAR listings.
As documented, this for file system objects themselves, since
`MemorySourceAccessor` is an implementation detail.
`listNar` did the not-so-pretty thing of going straight to JSON. Now it
uses `MemorySourceAccessor::File`, or rather variations of it, to go to
a C++ data type first, and only JSON second.
To accomplish this we add some type parameters to the `File` data type.
Actually, we need to do two rounds of this, because shallow NAR
listings. There is `FileT` and `DirectoryT` accordingly.
This matches the "NAR Listing" JSON format, and also helps distinguish
from regular file contents.
Why we want to match that will become clear in the next comments, when
we will in fact use (variations of) this data type for NAR listings.
`nix derivation add`, and its C API counterpart, now works a bit closer
to `builtins.derivation` in that they don't require the user to fill-in
input addressed paths correctly ahead of time.
The logic for this is carefully deduplicated, between all 3 entry
points, and also between the existing `checkInvariants` function. There
are some more functional tests, and there are also many more unit tests.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: edef <edef@edef.eu>
These do a read/write test in the middles of some computation. They are
an imperative way to test intermediate values rather than functionally
testing end outputs.
We don't use the various set<string_view>s that we construct,
and all we really care about is ensuring that all outputs are
of a single, consistent type.
This is necessary to ban symlink following. It can be considered
a defense in depth against issues similar to CVE-2024-45593. By
slightly changing the API in a follow-up commit we will be able
to mitigate the symlink following issue for good.
This is an example of "Parse, don't validate" principle [1].
Before, we had a number of `StringSet`s in `DerivationOptions` that
were not *actually* allowed to be arbitrary sets of strings. Instead,
each set member had to be one of:
- a store path
- a CA "downstream placeholder"
- an output name
Only later, in the code that checks outputs, would these strings be
further parsed to match these cases. (Actually, only 2 by that point,
because the placeholders must be rewritten away by then.)
Now, we fully parse everything up front, and have an "honest" data type
that reflects these invariants:
- store paths are parsed, stored as (opaque) deriving paths
- CA "downstream placeholders" are rewritten to the output deriving
paths they denote
- output names are the only arbitrary strings left
Since the first two cases both become deriving paths, that leaves us
with a `std::variant<SingleDerivedPath, String>` data type, which we use
in our sets instead.
Getting rid of placeholders is especially nice because we are replacing
them with something much more internally-structured / transparent.
[1]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Now the error message looks something like:
error:
… during upload of 'file:///tmp/storeabc/4yxrw9flcvca7f3fs7c5igl2ica39zaw.narinfo'
error: blah blah
Also makes fail and failEx themselves noexcept, since all the operations they
do are noexcept and we don't want exceptions escaping from them.
The indentation level of the code is already high enough. We can just
wrap the whole function in a try/catch and mark it noexcept.
Partially cherry-picked from https://gerrit.lix.systems/c/lix/+/2133
Co-authored-by: eldritch horrors <pennae@lix.systems>
Makes the cross-x86_64-w64-mingw32 devshell slightly less
broken. It still needs a bit of massaging to function, but
that's much less cumbersome now that the generic machinery
with genericClosure that evaluates drvPath doesn't barf on
unavailable packages.
The test was failing because nix path-info --json now returns narHash as
a structured dictionary {"algorithm": "sha256", "format": "base64",
"hash": "..."} instead of an SRI string "sha256-...".
This change was introduced in commit 5e7ee808d. The functional test
path-info.sh was updated at that time, but this NixOS test was missed.
The fix converts the dictionary format to SRI format inline:
tarball_hash_sri = f"{narHash_obj['algorithm']}-{narHash_obj['hash']}"
Several bugs to squash:
- Apparently DELETE is an already used macro with Win32. We can avoid it
by using Camel case instead (slightly hacky but also fits the naming
convention better)
- Gets rid of the raw usage of isatty. Added an isTTY impl to abstract over
the raw API.
Replaces the usage of createAtRoot, which goes as far up the
directory tree as possible with rooted variant makeFSSourceAccessor.
The changes in this patch should be safe wrt to not asserting on relative
paths. Arguments passed to makeFSSourceAccessor here should already be using
absolute paths.
without this, there is no way to swap them out for structures using a
different allocator. This should be reverted as part of redesiging
ExprAttrs to use an ExprAttrsBuilder
And also render the docs nicely.
I would like to use a markdown AST for this, but to avoid new deps
(lowdown's AST doesn't suffice) I am just doing crude string
manipulations for now.
Since 918c1a9e58 configurePhase variable points to cmakeConfigurePhase
and runPhase configurePhase does the wrong thing.
configurePhase function on the other hand still worked correctly.
Move StorePath and Derivation declarations to their own headers in a
backwards compatible way:
- Created nix_api_store/store_path.h for StorePath operations
- Created nix_api_store/derivation.h for Derivation operations
- Main nix_api_store.h includes both headers for backwards compatibility
This reorganization improves modularity and hopefully makes the API
easier to navigate.
I found this because of a test failure on cygwin in
nix_api_expr_test.nix_eval_state_lookup_path:
'std::filesystem::__cxx11::filesystem_error'
what(): filesystem error: cannot remove all: Device or resource busy
[...]
[.../my_state/db/db.sqlite]
LocalState was never getting destroyed due to a reference leak. These
_free functions use an 'operator delete' which doesn't call the
destructor for the type.
Fixes: 309d55807c
Currently, --gtest_filter=nix_api_store_test.nix_eval_state_lookup_path
will result in:
terminating due to unexpected unrecoverable internal error: Assertion
'gcInitialised' failed in void nix::assertGCInitialized() at
../src/libexpr/eval-gc.cc:138
Changing the test fixture to _exr_test causes GC to be initialised.
Typically PosixSourceAccessor can be used from multiple threads,
but mtime is not updated atomically (i.e. with compare_exchange_weak),
so mtime gets raced. It's only needed in dumpPathAndGetMtime and mtime
tracking can be gated behind that.
Also start using getLastModified interface instead of dynamic casts.
This makes the output easier to compare with the new machine-generated
lists in #9732.
The hand-curated order did have the advantage of putting more important
attributes at the top, but I don't think it is worth preserving that
when `std::map` is so much easier to work with. The right solution to
leading the reader to the more important attributes is to call them out
in the intro texts.
This makes the proto serializer characterisation test data be
accompanied by JSON data.
This is arguably useful for a reasons:
- The JSON data is human-readable while the binary data is not, so it
provides some indication of what the test data means beyond the C++
literals.
- The JSON data is language-agnostic, and so can be used to quickly rig
up tests for implementation in other languages, without having source
code literals at all (just go back and forth between the JSON and the
binary).
- Even though we have no concrete plans to place the binary protocol 1-1
or with JSON, it is still nice to ensure that the JSON serializers and
binary protocols have (near) equal coverage over data types, to help
ensure we didn't forget a JSON (de)serializer.
Make instances for them that share code with `nix path-info`, but do a
slightly different format without store paths containing store dirs
(matching the other latest JSON formats).
Progress on #13570.
If we depend on the store dir, our JSON serializers/deserializers take
extra arguements, and that interfaces with the likes of various
frameworks for associating these with types (e.g. nlohmann in C++, Serde
in Rust, and Aeson in Haskell).
For now, `nix path-info` still uses the previous format, with store
dirs. We may yet decide to "rip of the band-aid", and just switch it
over, but that is left as a future PR.
Add support for configuring S3 storage class via the storage-class
parameter for S3BinaryCacheStore. This allows users to optimize costs
by selecting appropriate storage tiers (STANDARD, GLACIER,
INTELLIGENT_TIERING, etc.) based on access patterns.
The storage class is applied via the x-amz-storage-class header for
both regular PUT uploads and multipart upload initiation.
I would run `nix flake show` on a flake than hit:
===
├───ihaskell: package 'ihaskell-wrapper'
├───ihaskell-96: package 'ihaskell-wrapper'
├───ihaskell-96-dev: package 'ghc-shell-for-ihaskell-0.10.4.0'
error: expected a derivation
===
and it is not obvious what package is the culprit here since nix stops
rightaway.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
... via _NIX_TEST_NO_ENVIRONMENT_WARNINGS
This environment variable suppresses warnings that depend on the test
environment (such as ulimit warnings in builds on systems with lower
limits, which may well succeed if it weren't for the warning).
This prevents non-deterministic test failures in golden/characterization
tests.
Alternative considered: filtering stderr in test scripts, but that approach
is fragile with binary test output, and potentially multiple call sites.
Show the actual attempted stack size value (capped at hard limit)
separately from the desired value, making it clearer what's happening
when the hard limit is lower than requested.
2025-11-04 23:38:50 +01:00
953 changed files with 28251 additions and 11733 deletions
# 2. Store your dockerhub username as DOCKERHUB_USERNAME in "Repository secrets" of your fork repository settings (https://github.com/$githubuser/nix/settings/secrets/actions)
# 3. Create an access token in https://hub.docker.com/settings/security and store it as DOCKERHUB_TOKEN in "Repository secrets" of your fork
synopsis: Channel URLs migrated to channels.nixos.org subdomain
prs: [14518]
issues: [14517]
---
Channel URLs have been updated from `https://nixos.org/channels/` to `https://channels.nixos.org/` throughout Nix.
The subdomain provides better reliability with IPv6 support and improved CDN distribution. The old domain apex (`nixos.org/channels/`) currently redirects to the new location but may be deprecated in the future.
synopsis: "New setting `ignore-gc-delete-failure` for local stores"
prs: [15054]
---
A new local store setting [`ignore-gc-delete-failure`](@docroot@/store/types/local-store.md#store-local-store-ignore-gc-delete-failure) has been added.
When enabled, garbage collection will log warnings instead of failing when it cannot delete store paths.
This is useful when running Nix as an unprivileged user that may not have write access to all paths in the store.
This setting is experimental and requires the [`local-overlay-store`](@docroot@/development/experimental-features.md#xp-feature-local-overlay-store) experimental feature.
synopsis: New command `nix store roots-daemon` for serving GC roots
prs: [15143]
---
New command [`nix store roots-daemon`](@docroot@/command-ref/new-cli/nix3-store-roots-daemon.md) runs a daemon that serves garbage collector roots over a Unix domain socket.
It enables the garbage collector to discover runtime roots when the main Nix daemon doesn't have `CAP_SYS_PTRACE` capability and therefore cannot scan `/proc`.
The garbage collector can be configured to use this daemon via the [`use-roots-daemon`](@docroot@/store/types/local-store.md#store-experimental-option-use-roots-daemon) store setting.
This feature requires the [`local-overlay-store` experimental feature](@docroot@/development/experimental-features.md#xp-feature-local-overlay-store).
This section provides some notes on how to start hacking on Nix.
To get the latest version of Nix from GitHub:
> **Note**
>
> When checking out the repo on Windows, make sure you have the git setting `core.symlinks` enabled, before cloning, as there are symlinks in the repo.
@@ -6,16 +6,9 @@ Additionally, see [Testing Nix](./testing.md) for further instructions on how to
## Building Nix with Debug Symbols
In the development shell, set the`mesonBuildType`environment variable to `debug` before configuring the build:
In the development shell, `mesonBuildType`is set automatically to `debugoptimized`. This builds Nix with debug symbols, which are essential for effective debugging.
```console
[nix-shell]$ export mesonBuildType=debugoptimized
```
Then, proceed to build Nix as described in [Building Nix](./building.md).
This will build Nix with debug symbols, which are essential for effective debugging.
It is also possible to build without debugging for faster build:
It is also possible to build without optimization for faster build:
@@ -137,6 +137,12 @@ $ _NIX_TEST_ACCEPT=1 meson test nix-store-tests -v
will regenerate the "golden master" expected result for the `libnixstore` characterisation tests.
The characterisation tests will mark themselves "skipped" since they regenerated the expected result instead of actually testing anything.
### JSON Schema testing
In `doc/manual/source/protocols/json/` we have a number of manual pages generated from [JSON Schema](https://json-schema.org/).
That JSON schema is tested against the JSON file test data used in [characterisation tests](#characterisation-testing-unit ) for JSON (de)serialization, in `src/json-schema-checks`.
Between the JSON (de)serialization testing, and this testing of the same data against the schema, we make sure that the manual, the implementation, and a machine-readable schema are are all in sync.
### Unit test support libraries
There are headers and code which are not just used to test the library in question, but also downstream libraries.
@@ -338,7 +338,7 @@ Here is more information on the `output*` attributes, and what values they may b
This will specify the output hash of the single output of a [fixed-output derivation].
The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The "nix32" encoding is an adaptation of base-32 encoding.
The ["nix32" encoding](@docroot@/protocols/nix32.md) is Nix's variant of base-32 encoding.
@@ -74,4 +74,48 @@ in f { x = throw "error"; y = throw "error"; }
=>"ok"
```
## Evaluation order
The order in which expressions are evaluated is generally unspecified, because it does not affect successful evaluation outcomes.
This allows more freedom for the evaluator to evolve and to evaluate efficiently.
Data dependencies naturally impose some ordering constraints: a value cannot be used before it is computed.
Beyond these constraints, the evaluator is free to choose any order.
The order in which side effects such as [`builtins.trace`](@docroot@/language/builtins.md#builtins-trace) output occurs is not defined, but may be expected to follow data dependencies. <!-- we may want to be more specific about this. -->
In a lazy language, evaluation order is often opposite to expectations from strict languages.
For example, in `let wrap = x: { wrapped = x; }; in wrap (1 + 2)`, the function body produces a result (`{ wrapped = ...; }`) *before* evaluating `x`.
## Infinite recursion and stack overflow
During evaluation, two types of errors can occur when expressions reference themselves or call functions too deeply:
### Infinite recursion
This error occurs when a value depends on itself through a cycle, making it impossible to compute.
```nix
letx=x;inx
=>error:infiniterecursionencountered
```
Infinite recursion happens at the value level when evaluating an expression requires evaluating the same expression again.
Despite the name, infinite recursion is cheap to compute and does not involve a stack overflow.
The cycle is finite and fairly easy to detect.
### Stack overflow
This error occurs when the call depth exceeds the maximum allowed limit.
```nix
letf=x:f(x+1);
inf0
=>error:stackoverflow;max-call-depthexceeded
```
Stack overflow happens when too many function calls are nested without returning.
The maximum call depth is controlled by the [`max-call-depth` setting](@docroot@/command-ref/conf-file.md#conf-max-call-depth).
@@ -162,6 +162,9 @@ Update [attribute set] *attrset1* with names and values from *attrset2*.
The returned attribute set will have all of the attributes in *attrset1* and *attrset2*.
If an attribute name is present in both, the attribute value from the latter is taken.
This operator is [strict](@docroot@/language/evaluation.md#strictness) in both *attrset1* and *attrset2*.
That means that both arguments are evaluated to [weak head normal form](@docroot@/language/evaluation.md#values), so the attribute sets themselves are evaluated, but their attribute values are not evaluated.
[Update]: #update
## Comparison
@@ -185,18 +188,95 @@ All comparison operators are implemented in terms of `<`, and the following equi
## Equality
- [Attribute sets][attribute set] and [lists][list] are compared recursively, and therefore are fully evaluated.
-Comparison of [functions][function] always returns `false`.
- [Attribute sets][attribute set] are compared first by attribute names and then by items until a difference is found.
-[Lists][list] are compared first by length and then by items until a difference is found.
- Comparison of distinct [functions][function] returns `false`, but identical functions may be subject to [value identity optimization](#value-identity-optimization).
- Numbers are type-compatible, see [arithmetic] operators.
- Floating point numbers only differ up to a limited precision.
The `==` operator is [strict](@docroot@/language/evaluation.md#strictness) in both arguments; when comparing composite types ([attribute sets][attribute set] and [lists][list]), it is partially strict in their contained values: they are evaluated until a difference is found. <!-- this is woefully underspecified, affecting which expressions evaluate correctly; not just "ordering" or error messages. -->
### Value identity optimization
Nix performs equality comparisons of nested values by pointer equality or more abstractly, _identity_.
Nix semantics ideally do not assign a unique identity to values as they are created, but equality is an exception to this rule.
The disputable benefit of this is that it is more efficient, and it allows cyclical structures to be compared, e.g. `let x = { x = x; }; in x == x` evaluates to `true`.
However, as a consequence, it makes a function equal to itself when the comparison is made in a list or attribute set, in contradiction to a simple direct comparison.
[function]: ./syntax.md#functions
[Equality]: #equality
## Logical conjunction
> **Syntax**
>
> *bool1* `&&` *bool2*
Logical AND. Equivalent to `if`*bool1*`then`*bool2*`else false`.
This operator is [strict](@docroot@/language/evaluation.md#strictness) in *bool1*, but only evaluates *bool2* if *bool1* is `true`.
> **Example**
>
> ```nix
> true && false
> => false
>
> false && throw "never evaluated"
> => false
> ```
[Logical conjunction]: #logical-conjunction
## Logical disjunction
> **Syntax**
>
> *bool1* `||` *bool2*
Logical OR. Equivalent to `if`*bool1*`then true``else`*bool2*.
This operator is [strict](@docroot@/language/evaluation.md#strictness) in *bool1*, but only evaluates *bool2* if *bool1* is `false`.
> **Example**
>
> ```nix
> true || false
> => true
>
> true || throw "never evaluated"
> => true
> ```
[Logical disjunction]: #logical-disjunction
### Precedence and disjunctive normal form
The precedence of `&&` and `||` aligns with disjunctive normal form.
Without parentheses, an expression describes multiple "permissible situations" (connected by `||`), where each situation consists of multiple simultaneous conditions (connected by `&&`).
For example, `A || B && C || D && E` is parsed as `A || (B && C) || (D && E)`, describing three permissible situations: A holds, or both B and C hold, or both D and E hold.
## Logical implication
Equivalent to `!`*b1* `||`*b2* (or `if`*b1*`then`*b2*`else true`)
> **Syntax**
>
> *bool1* `->` *bool2*
Logical implication. Equivalent to `!`*bool1* `||`*bool2* (or `if`*bool1*`then`*bool2*`else true`).
This operator is [strict](@docroot@/language/evaluation.md#strictness) in *bool1*, but only evaluates *bool2* if *bool1* is `true`.
In this form, the attribute set between the `{``}` is recursive.
One of the attributes must have the special name `body`,
which is the result of the expression.
Example:
```nix
let{
foo=bar;
bar="baz";
body=foo;
}
```
This evaluates to "baz".
## Inheriting attributes
When defining an [attribute set](./types.md#type-attrs) or in a [let-expression](#let-expressions) it is often convenient to copy variables from the surrounding lexical scope (e.g., when you want to propagate attributes).
To check whether it works, try the following on the client:
To check whether it works, try fetching the [`nix-cache-info`](@docroot@/protocols/nix-cache-info.md) file on the client:
```console
$ curl http://avalon:8080/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
```
which should print something like:
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
When writing to a binary cache (e.g., with [`nix copy`](@docroot@/command-ref/new-cli/nix3-copy.md)), Nix creates [`nix-cache-info`](@docroot@/protocols/nix-cache-info.md) automatically if it doesn't exist.
On the client side, you can tell Nix to use your binary cache using
The path to the store object that resulted from building this derivation for the given output name.
dependentRealisations:
type:object
title:Underlying Base Build Trace
description:|
This is for [*derived*](@docroot@/store/build-trace.md#derived) build trace entries to ensure coherence.
Keys are derivation output IDs (same format as the main `id` field).
Values are the store paths that those dependencies resolved to.
As described in the linked section on derived build trace traces, derived build trace entries must be kept in addition and not instead of the underlying base build entries.
This is the set of base build trace entries that this derived build trace is derived from.
(The set is also a map since this miniature base build trace must be coherent, mapping each key to a single value.)
patternProperties:
"^sha256:[0-9a-f]{64}![a-zA-Z_][a-zA-Z0-9_-]*$":
$ref:"store-path-v1.yaml"
title:Dependent Store Path
description:Store path that this dependency resolved to during the build
additionalProperties:false
signatures:
type:array
title:Build Signatures
description:|
A set of cryptographic signatures attesting to the authenticity of this build trace entry.
A cryptographic hash value used throughout Nix for content addressing and integrity verification.
This schema describes the JSON representation of Nix's `Hash` type.
type:object
properties:
algorithm:
"$ref": "#/$defs/algorithm"
format:
type:string
enum:
- base64
- nix32
- base16
- sri
title:Hash format
description:|
The encoding format of the hash value.
This schema describes the JSON representation of Nix's `Hash` type as an [SRI](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) string.
Experimental JSON representation of a Nix [Store](@docroot@/store/index.md).
This schema describes the JSON serialization of a Nix store.
We use it for (de)serializing in-memory "dummy stores" used for testing, but in principle the data represented in this schema could live in any type of store.
@@ -52,4 +52,4 @@ The Nix Archive (NAR) format is also formally described using [Kaitai Struct](ht
{{#include nar.ksy}}
```
The source of the spec can be found [here](https://github.com/nixos/nix/blob/master/src/nix-manual/source/protocols/nix-archive/nar.ksy). Contributions and improvements to the spec are welcomed.
The source of the spec can be found [here](https://github.com/nixos/nix/blob/master/src/nix-manual/source/protocols/nix-archive/nar.ksy). Contributions and improvements to the spec are welcomed.
The `nix-cache-info` file is a metadata file at the root of a [binary cache](@docroot@/package-management/binary-cache-substituter.md) (e.g., `https://cache.example.com/nix-cache-info`).
MIME type: `text/x-nix-cache-info`
## Format
Line-based key-value format:
```
Key: value
```
Leading and trailing whitespace is trimmed from values.
Lines without a colon are ignored.
Unknown keys are silently ignored.
## Fields
### `StoreDir`
The Nix store directory path that this cache was built for (e.g., `/nix/store`).
If present, Nix verifies that this matches the client's store directory:
```
error: binary cache 'https://example.com' is for Nix stores with prefix '/nix/store', not '/home/user/nix/store'
```
### `WantMassQuery`
`1` or `0`. Sets the default for [`want-mass-query`](@docroot@/store/types/http-binary-cache-store.md#store-http-binary-cache-store-want-mass-query).
### `Priority`
Integer. Sets the default for [`priority`](@docroot@/store/types/http-binary-cache-store.md#store-http-binary-cache-store-priority).
## Example
```
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
```
## Caching Behavior
Nix caches `nix-cache-info` in the [cache directory](@docroot@/command-ref/env-common.md#env-NIX_CACHE_HOME) with a 7-day TTL.
Nix32 is Nix's variant of base-32 encoding, used for [store path digests](@docroot@/protocols/store-path.md), hash output via [`nix hash`](@docroot@/command-ref/new-cli/nix3-hash.md), and the [`outputHash`](@docroot@/language/advanced-attributes.md#adv-attr-outputHash) derivation attribute.
## Alphabet
The Nix32 alphabet consists of these 32 characters:
```
0 1 2 3 4 5 6 7 8 9 a b c d f g h i j k l m n p q r s v w x y z
```
The letters `e`, `o`, `u`, and `t` are omitted.
## Byte Order
Nix32 encoding processes the hash bytes from the end (last byte first), while base-16 encoding processes from the beginning (first byte first).
Consequently, the string sort order is determined primarily by the first bytes for base-16, and by the last bytes for Nix32.
We ultimately want to rectify this issue with all JSON formats to the extent allowed by our stability promises. To start with, we are changing the JSON format for derivations because the `nix derivation` commands are — in addition to being formally unstable — less widely used than other unstable commands.
See the documentation on the [JSON format for derivations](@docroot@/protocols/json/derivation.md) for further details.
See the documentation on the [JSON format for derivations](@docroot@/protocols/json/derivation/index.md) for further details.
- C API: `nix_get_attr_name_byidx`, `nix_get_attr_byidx` take a `nix_value *` instead of `const nix_value *` [#13987](https://github.com/NixOS/nix/pull/13987)
- New command `nix registry resolve` [#14595](https://github.com/NixOS/nix/pull/14595)
This command looks up a flake registry input name and returns the flakeref it resolves to.
For example, looking up Nixpkgs:
```
$ nix registry resolve nixpkgs
github:NixOS/nixpkgs/nixpkgs-unstable
```
Upstreamed from [Determinate Nix 3.14.0](https://github.com/DeterminateSystems/nix-src/pull/273).
- `nix flake clone` supports all input types [#14581](https://github.com/NixOS/nix/pull/14581)
`nix flake clone` now supports arbitrary input types. In particular, this allows you to clone tarball flakes, such as flakes on FlakeHub.
Upstreamed from [Determinate Nix 3.12.0](https://github.com/DeterminateSystems/nix-src/pull/229).
## Performance improvements
- Git fetcher computes `revCount`s using multiple threads [#14462](https://github.com/NixOS/nix/pull/14462)
When using Git repositories with a long history, calculating the `revCount` attribute can take a long time. Nix now computes `revCount` using multiple threads, making it much faster (e.g. 9.1s to 3.7s for Nixpkgs).
Note that if you don't need `revCount`, you can disable it altogether by setting the flake input attribute `shallow = true`.
Upstreamed from [Determinate Nix 3.12.2](https://github.com/DeterminateSystems/nix-src/pull/245).
- `builtins.stringLength` now runs in constant time [#14442](https://github.com/NixOS/nix/pull/14442)
The internal representation of strings has been replaced with a size-prefixed Pascal style string. Previously Nix stored strings as a NUL-terminated array of bytes, necessitating a linear scan to calculate the length.
- Uploads to `http://` and `https://` binary cache stores now run in constant memory [#14390](https://github.com/NixOS/nix/pull/14390)
Nix used to buffer the whole compressed NAR contents in memory. It now reads it in a streaming fashion.
- Channel URLs migrated to channels.nixos.org subdomain [#14517](https://github.com/NixOS/nix/issues/14517) [#14518](https://github.com/NixOS/nix/pull/14518)
Channel URLs have been updated from `https://nixos.org/channels/` to `https://channels.nixos.org/` throughout Nix. This subdomain provides better reliability with IPv6 support and improved CDN distribution. The old domain apex (`nixos.org/channels/`) currently redirects to the new location but may be deprecated in the future.
- Fix `download buffer is full; consider increasing the 'download-buffer-size' setting` warning [#11728](https://github.com/NixOS/nix/issues/11728) [#14614](https://github.com/NixOS/nix/pull/14614)
The underlying issue that led to [#11728](https://github.com/NixOS/nix/issues/11728) has been resolved by utilizing
[libcurl write pausing functionality](https://curl.se/libcurl/c/curl_easy_pause.html) to control backpressure when unpacking to slow destinations like the git-backed tarball cache. The default value of `download-buffer-size` is now 1 MiB and it's no longer recommended to increase it, since the root cause has been fixed.
This is expected to improve download performance on fast connections, since previously a single slow download consumer would stall the thread and prevent any other transfers from progressing.
Many thanks go out to the [Lix project](https://lix.systems/) for the [implementation](https://git.lix.systems/lix-project/lix/commit/4ae6fb5a8f0d456b8d2ba2aaca3712b4e49057fc) that served as inspiration for this change and for triaging libcurl [issues with pausing](https://github.com/curl/curl/issues/19334).
Nix uses a content-addressed cache backed by libgit2 for deduplicating files fetched via `fetchTarball` and `github`, `tarball` flake inputs. Its usage has been significantly optimised to reduce the amount of I/O operations that are performed. For a typical nixpkgs source tarball this results in 200 times fewer system calls on Linux. In combination with libcurl pausing this alleviates performance regressions stemming from the tarball cache.
- Already valid derivations are no longer copied to the store [#14219](https://github.com/NixOS/nix/pull/14219)
This results in a modest speedup when using the Nix daemon.
- `nix nar ls` and `nix nar cat` are significantly faster and no longer buffer the whole NAR in memory [#14273](https://github.com/NixOS/nix/pull/14273) [#14732](https://github.com/NixOS/nix/pull/14732)
S3 binary cache operations now happen via HTTP, leveraging `libcurl`'s native AWS SigV4 authentication instead of the AWS C++ SDK, providing significant improvements:
- **Reduced memory usage**: Eliminates memory buffering issues that caused segfaults with large files
- **Lighter dependencies**: Uses lightweight `aws-crt-cpp` instead of full `aws-cpp-sdk`, reducing build complexity
The new implementation requires curl >= 7.75.0 and `aws-crt-cpp` for credential management.
All existing S3 URL formats and parameters remain supported, however the store settings for configuring multipart uploads have changed:
- **`multipart-upload`** (default: `false`): Enable multipart uploads for large files. When enabled, files exceeding the multipart threshold will be uploaded in multiple parts.
- **`multipart-threshold`** (default: `100 MiB`): Minimum file size for using multipart uploads. Files smaller than this will use regular PUT requests. Only takes effect when `multipart-upload` is enabled.
- **`multipart-chunk-size`** (default: `5 MiB`): Size of each part in multipart uploads. Must be at least 5 MiB (AWS S3 requirement). Larger chunk sizes reduce the number of requests but use more memory.
- **`buffer-size`**: Has been replaced by `multipart-chunk-size` and is now an alias to it.
Note that this change also means Nix now supports S3 binary cache stores even if built without `aws-crt-cpp`, but only for public buckets which do not require authentication.
- S3 URLs now support object versioning via `versionId` parameter [#13955](https://github.com/NixOS/nix/issues/13955) [#14274](https://github.com/NixOS/nix/pull/14274)
S3 URLs now support a `versionId` query parameter to fetch specific versions
of objects from S3 buckets with versioning enabled. This allows pinning to
exact object versions for reproducibility and protection against unexpected
- S3 binary cache stores now support storage class configuration [#7015](https://github.com/NixOS/nix/issues/7015) [#14464](https://github.com/NixOS/nix/pull/14464)
S3 binary cache stores now support configuring the storage class for uploaded objects via the `storage-class` parameter. This allows users to optimize costs by selecting appropriate storage tiers based on access patterns.
The storage class applies to both regular uploads and multipart uploads. When not specified, objects use the bucket's default storage class.
See the [S3 storage classes documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) for available storage classes and their characteristics.
## Store path info JSON format changes
The JSON format emitted by `nix path-info --json` has been updated to a new version with improved structure.
To maintain compatibility, `nix path-info --json` now requires a `--json-format` flag to specify the output format version.
Using `--json` without `--json-format` is deprecated and will become an error in a future release.
For now, it defaults to version 1 with a warning, for a smoother migration.
### Version 1 (`--json-format 1`)
This is the legacy format, preserved for backwards compatibility:
- Still `null` values for input-addressed store objects
The `hash` field uses the [SRI](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) format like other hashes.
Additionally the following fields are added to both formats:
- **`version` field**:
All store path info JSON now includes `"version": <1|2>`. The `version` tracks breaking changes, and adding fields to outputted JSON is not a breaking change.
- **`storeDir` field**:
Top-level `"storeDir"` field contains the store directory path (e.g., `"/nix/store"`).
## Derivation JSON format changes
The derivation JSON format has been updated from version 3 to version 4:
- **Nested structure with top-level metadata**:
The output of `nix derivation show` is now wrapped in an object with `version` and `derivations` fields:
```json
{
"version": 4,
"derivations": { ... }
}
```
The map from derivation paths to derivation info is nested under the `derivations` field.
This matches the structure used for `nix path-info --json --json-format 2`, and likewise brings this command into compliance with the JSON guidelines.
- **Restructured inputs**:
Inputs are now nested under an `inputs` object:
- Old: `"inputSrcs": [...], "inputDrvs": {...}`
- New: `"inputs": {"srcs": [...], "drvs": {...}}`
- **Consistent content addresses**:
Fixed content-addressed outputs now use structured JSON format.
This is the same format as `ca` in store path info (after the new version).
Version 3 and earlier formats are *not* accepted when reading.
**Affected command**: `nix derivation`, namely its `show` and `add` sub-commands.
Nix used to feel "stuck" while it was cloning large repositories. Nix now shows Git's native progress indicator while fetching.
Upstreamed from [Determinate Nix 3.13.0](https://github.com/DeterminateSystems/nix-src/pull/250).
- Interrupting REPL commands works more than once [#13481](https://github.com/NixOS/nix/issues/13481)
Previously, this only worked once per REPL session; further attempts would be ignored.
This issue is now fixed, so REPL commands such as `:b` or `:p` can be canceled consistently.
This is a cherry-pick of the change from the [Lix project](https://gerrit.lix.systems/c/lix/+/1097).
- NAR unpacking code has been rewritten to make use of dirfd-based `openat` and `openat2` system calls when available [#14597](https://github.com/NixOS/nix/pull/14597)
- Dynamic size unit rendering [#14423](https://github.com/NixOS/nix/pull/14423) [#14364](https://github.com/NixOS/nix/pull/14364)
Various commands and the progress bar now use dynamically determined size units instead
of always using `MiB`. For example, the progress bar now reports download status like:
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
