`tryToBuild` threaded a single `PathLocks outputLocks` by reference
across all build phases and managed a `std::unique_ptr<Activity> actLock`
with explicit `if (!actLock)` guards and `.reset()` calls around the hook
retry loop. This commit introduces coroutine lambdas for the three phases:
`tryHookLoop` owns a `PathLocks` in a scoped block for the first attempt
and per-iteration in the retry loop, `tryBuildLocally` acquires its own
`PathLocks`, and the hook-wait `Activity` is a stack variable scoped to
the postpone block.
When `max-jobs = 0` and no remote builders are available, Nix reported
"required system or feature not available" even though the system and
features matched fine. The `canBuildLocally` lambda returned a plain
`bool`, conflating a configuration knob (`max-jobs = 0`) with actual
incompatibility (wrong platform, missing features). It also short-circuited
on the first failing check, so a user with both a platform mismatch and
missing features would only see one of the two.
This commit replaces the bool with a `LocalBuildRejection` struct whose
`WrongLocalStore` variant collects all applicable failures into
`badPlatform`, `missingFeatures`, and an orthogonal `maxJobsZero` flag.
Platform mismatch and missing features now produce separate error
paragraphs, and all applicable reasons appear in a single message.
The local-build capability check also now returns
`std::variant<LocalBuildCapability, LocalBuildRejection>`, bundling
the `LocalStore &` and optional `ExternalBuilder *` together.
This comes in two parts: a `nix store roots-daemon` command that
can run as root and list runtime roots,
and client logic to find runtime roots for a `LocalStore` by connecting
to that daemon.
This may be useful with an unprivileged nix daemon, as it would
otherwise be unable to find runtime roots from process open files
and maps.
Using nix::unreachable() in getInternalType() and type() turns
out to be quite expensive and prevents inlining. Also Value::type
got compiled to a jump table which has a high overhead from indirect
jumps. Using an explicit lookup table turns out to be more efficient.
This does mean that we lose out on nice diagnostics from nix::unreachable
calls, but this code is probably one of the hottests functions in the whole
evaluator, so I think the tradeoff is worth it. The nixUnreachableWhenHardened
boils down to nix::unreachable when UBSan is enabled so we still have good
coverage there.
This makes sure that ExprVar::eval inlines lookupVar call. In practice
this seems to reduce instruction count by ~2%, though it doesn't have
a statistically significant impact on the wall time.
This reduces the churn when changing up the order of
values in a follow-up commit. This should have been done
from the start ideally to improve readability.
The external-builders test expands `$PATH` into a heredoc without quotes,
so any `PATH` entry containing spaces causes bash to parse the line as a
command instead of an assignment, failing the test.
The tab completion handler in `completePrefix` only caught `ParseError`,
`EvalError`, `BadURL`, and `FileNotFound`. Other error types like
`JSONParseError` (which derives from `Error`, not `EvalError`) escaped
the catch block and propagated through editline's C code as undefined
behavior, crashing the REPL. This happened when tab-completing
expressions like `(builtins.fromJSON "invalid").` where evaluation
throws a non-`EvalError` exception.
This commit marks `completionCallback` and `listPossibleCallback` as
`noexcept` with function-try-blocks that catch all exceptions at the
C/C++ boundary, preventing any exception from reaching editline.
Fixes#15133.
When a URL like `github:nixos/nixpkgs/nixpkgs.git?ref=<hash>` (using
`ref` instead of `rev`) failed the github input scheme, it fell
through to `parsePathFlakeRefWithFragment` which constructed a `path:`
`ParsedURL` with an empty authority but a relative path. This violated
RFC 3986 section 3.3 (authority present requires path starting with
`/`), causing an assertion failure in `renderAuthorityAndPath` when
`PathInputScheme` tried to format the URL for an error message.
This commit only sets the authority on absolute paths. Relative paths
get `std::nullopt` for authority, which is the correct representation
per the URL spec.
Fixes#15196. Fixes#14830.
`file-descriptor.{cc,hh}` was getting too big, split out
`file-system-at.{cc,hh}` for the FD-based file system stuff,
`file-descriptor.{cc,hh}` will only be for the fundamental primitives
that are file-system agnostic and work on almost all file types.
Review with `git show --color-moved` to see that this is indeed all
moving.
Docker 28+ defaults to the containerd image store, which pushes layers
uncompressed instead of gzip. The GHA runner image updated Docker to
29.x (actions/runner-images#13633), causing the `nixos/nix:2.33.3`
image to balloon from 138 MB to 505 MB, with all 70 layers pushed as
`application/vnd.docker.image.rootfs.diff.tar` instead of `.tar.gzip`.
OCI clients that only support gzip (e.g. `go-containerregistry`, used
by Concourse CI) fail with "gzip: invalid header".
This commit disables the containerd snapshotter in the release workflow
before any Docker operations, restoring the classic storage driver that
preserves gzip compression through the `docker load` / `docker push`
pipeline.
Fixes#15246
Linux, macOS, and all 3 BSDs have it (according to man page google
search), so let's just drop this. Support for not having it was added in
d03f0d4117 in 2006, things have changed in
the last 20 years!
When ref::cast() fails, the error message was cryptic ("null pointer
cast to ref"). Now it throws a proper bad_ref_cast (a std::bad_cast
subclass) with a clear message showing the actual types involved:
ref<nix::Base> cannot be cast to ref<nix::Derived>
This also adds a demangle.hh utility.
ref<Derived> was already implicitly convertible to ref<Base>, but the
mechanism was unclear and error messages for rejected downcasts were
more cryptic than necessary. This change:
- Adds RefImplicitlyUpcastableTo concept to constrain the conversion
operator, making the intent explicit and improving error messages
- Documents .cast() and .dynamic_pointer_cast() as alternatives for
explicit downcasting
- Adds unit tests for covariance behavior
The old check rejected any relative path whose first character was a
dot, producing false negatives for valid descendants like `.ssh` or
`.config`. This commit changes the logic such that now it inspects the
first path component via `path::begin()`, only rejects `.` and `..`
rather than anything dot-prefixed. Fixes#15207.
libutil tests were crashing on Windows due to issues finding `environ`.
Replace process creation of `getEnv` with a new `getEnvOs` function that
uses native windows APIs.
Also convert a bunch of `RunOptions` fields to use `OsString` to better
reflect the underlying interfaces.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
The fake cacert didn't have subjectAltName for 127.0.0.1, so the test
was failing for a different reason. Also `tries` setting wasn't being respected.
There's no callsite specifying it in the request, so just use the one specified
in the FileTransferSettings and remove the fields from the FileTransferRequest.
This commit removes the `nixStore` member from `Settings` and instead
computes the default Nix store directory directly in
`StoreConfigBase::getDefaultNixStoreDir()` from env vars
(`NIX_STORE_DIR`, `NIX_STORE`) or the compile-time default. The method
is made public so callers that previously reached through the global
`settings.nixStore` can use it instead.
Progress on #5638
nix store gc: prints number of paths that would be freed, but not bytes
nix-collect-garbage: ditto
nix-store --gc: retains current behavior
It would be very non-trivial to also compute the bytes that would be
freed, due to hardlinking in the store.
Also adds checking for incompatible mixing of dry-run and max-freed
options.
Resolves#5704
Signed-off-by: Lisanna Dettwyler <lisanna.dettwyler@gmail.com>
This commit inlines `DerivationOptions::willBuildLocally` and
`DerivationOptions::canBuildLocally` into their sole call site in
`DerivationBuildingGoal::tryToBuild`. The `canBuildLocally` logic is now
a lambda capturing the surrounding context, and `willBuildLocally` is
replaced by `drvOptions.preferLocalBuild && canBuildLocally`.
Progress on #5638
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
This commit removes the `getMachines` free function and inlines `Machine::parseConfig({settings.thisSystem}, settings.getWorkerSettings().builders)` at its two call sites in `worker.cc` and `build-remote.cc`. The wrapper just forwarded to `Machine::parseConfig` with global settings, so inlining it removes an unnecessary layer of indirection and makes the global dependency explicit at each call site.
Progress on #5638
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
This commit moves `nixStateDir` and `useXDGBaseDirectories` into a dedicated `ProfileDirsOptions` struct and threads it through the profile directory functions (`profilesDir`, `rootProfilesDir`, `defaultChannelsDir`, `rootChannelsDir`, `getDefaultProfile`) so they no longer read from the global `Settings` object directly. This follows the same pattern as `LocalSettings`, `WorkerSettings`, and `NarInfoDiskCacheSettings`.
Progress on #5638
Co-authored-by: Amaan Qureshi <git@amaanq.com>
This commit moves `ttlNegativeNarInfoCache` and `ttlPositiveNarInfoCache` into a dedicated `NarInfoDiskCacheSettings` struct that `Settings` privately inherits from, following the same pattern as `LocalSettings`, `LogFileSettings`, and `WorkerSettings`.
`NarInfoDiskCache` now takes explicit `NarInfoDiskCacheSettings` and `SQLiteSettings` in its constructor instead of reading from the global. The singleton `getNarInfoDiskCache()` is replaced with a `NarInfoDiskCache::get()` static method that accepts these settings, though they are only used on the first call (subsequent calls return the cached instance regardless of arguments).
Progress on #5638
It was a crude hack that this one low-level function was dependent on
the high-level read-only mode setting --- all the more so because rather
than making derivation writing fail, that setting made it silently
"succeed" why not actually writing the derivation. (Also, for context,
we didn't have an such behavior for any other store-mutating operations,
just for this one function.)
I have gotten rid of the wrapper, and updated the call sites
accordingly.
- For the ones that should remain dependent on this setting, I made this
explicit, and added a comment.
- For others, surrounding operations assumed writability (e.g. we had
written something before, or were about to try to read back the
written derivation after), and so I just made those do the underlying
`Store::writeDerivation` operation.
This makes `acquireUserLock` take an explicit stateDir parameter,
since it was previously reaching into the global settings object
just to read `nixStateDir` for constructing the userpool paths.
Progress on #5638
This commit introduces a `getReadOnly` method on the store config that returns if the current store is read only or not. This is then used in subtitution, so we fail gracefully with a nice error message if only the individual store is read-only.
As a bonus, it gets us one step closer to getting rid of the global because we can use the per-store method instead.
Progress on #5638
S3 binary caches now use virtual-hosted-style URLs by default for
standard AWS endpoints. Path-style endpoints (s3.region.amazonaws.com)
only serve HTTP/1.1, preventing HTTP/2 multiplexing and causing TCP
TIME_WAIT socket exhaustion under high concurrency. Virtual-hosted-style
endpoints (bucket.s3.region.amazonaws.com) support HTTP/2, enabling
multiplexing with the existing CURLPIPE_MULTIPLEX configuration.
Add a new `addressing-style` store option (auto/path/virtual) to control
this behavior. `auto` (default) uses virtual-hosted-style for standard
AWS endpoints and path-style for custom endpoints. `path` forces
path-style for backwards compatibility. `virtual` forces virtual-hosted-
style for all endpoints including custom ones.
Fixes: https://github.com/NixOS/nix/issues/15208
Idle connections in libcurl's connection pool can be silently dropped by
the OS or intermediate firewalls/NATs before they can be reused, forcing
new TCP connections to be created. This is especially problematic for
HTTP/1.1 endpoints where multiplexing is unavailable.
Enable TCP keep-alive with a 60-second idle/interval on all curl easy
handles to prevent idle connection drops and improve connection reuse.
This commit moves `pollInterval`, `maxSubstitutionJobs`, `postBuildHook`, and `logLines` into a dedicated `WorkerSettings` struct that `Settings` privately inherits from, as they are only used by the build worker subsystem. This follows the same pattern as `LocalSettings` and `LogFileSettings`.
This expands to __GCC_DESTRUCTIVE_SIZE, which is also 64 (at least in the x86_64 stdenv).
Let the compiler decide what's the appropriate cache line size is. Also, on aarch64-darwin
the cache line size 128 bytes, so the previous fix didn't actually get rid of false sharing
reliably. Clang does this [1] [2], so it overestimates the sizes somewhat, but that's still enough
for avoiding false sharing on darwin.
[1]: a289341ded/clang/lib/Frontend/InitPreprocessor.cpp (L1331-L1339)
[2]: 6f51f8e0f9/clang/lib/Basic/Targets/AArch64.h (L262-L264)
In the https-store tests, a `TestHttpBinaryCacheStoreConfig` is constructed with a call to format to create the cache uri. This commit adds a constructor to `HttpBinaryCacheStoreConfig` to remove the need for this call, and updates the test type to leverage this so we're no longer manually calling fmt on a string to format the port.
This commit makes `FileTransfer` self-contained by giving it a reference
to `FileTransferSettings` instead of reading from the global. It also
adds an optional `FileTransfer` parameter to `HttpBinaryCacheStore` so
callers can inject their own instance.
The main motivation is test isolation. The HTTPS store tests now create
custom `FileTransferSettings` with the test CA certificate and pass it
through `makeFileTransfer()`, avoiding global state mutation entirely.
nix-nswrapper allows running nix in its own user namespace,
believing it is root and with access to build users for sandboxing
with auto-allocate-uids, while it is actually unprivileged.
It is used to wrap nix, and an example of its use has been
added to the unprivileged daemon functional tests.
Running it does not require any elevated privileges,
only uids and gids allocated in /etc/sub{uid,gid}
The `caFile`, `netrcFile`, and `downloadSpeed` settings are only used by
the file transfer subsystem but lived in the global `Settings` class.
This moves them to `FileTransferSettings` where they belong.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
`hashedMirrors` is only relevant to local builds (it is consumed by
`builtin:fetchurl` during derivation building) but lived in the global
`Settings` class. This moves it to `LocalSettings` where it belongs
and threads it through `BuiltinBuilderContext` so `fetchurl.cc` reads
it from the context instead of reaching into `settings` directly.
The underlying mechanism is now in a new `fetchBuildLog` function I put
in `libcmd`.
I am putting it in here and not in libstore because I have some doubts
about `getDefaultSubstituters`, so I would like to keep it in a more
"peripheral" part of the codebase for now.
The `storeUri`, `substituters`, and `trustedSubstituters` settings now
store typed `StoreReference` values directly instead of raw strings,
so callers work with the real types without manual parsing.
This is a reworked version of #10761.
Instead of constructing a `StoreReference` and letting `openStore()` resolve it,
this commit creates the store config directly and calls `openStore()` on it. This
avoids the indirection through the store registry.
Stores hold their config as `ref<const Config>` or `const Config &`,
so `Setting<>` fields are already immutable after store construction.
The field-level `const` is redundant and prevents pre-construction
mutation which is sometimes useful. This commit updates these settings by dropping the `const` qualifier, as it's not needed.
The CLI flags `--from`, `--to`, `--eval-store`, and substituter URIs now
parse to `StoreReference` at the argument boundary. `fetchClosure` uses
`StoreReference::parse` instead of `parseURL`. This also adds
`operator<=>` to `StoreReference`.
This commit makes `nix daemon` inherit from `StoreConfigCommand`
instead of `Command`, so that it receives a `StoreConfig` to open
and serve stores with. This cleans up a few things (removes
`openUncachedStore` helper, passes `storeConfig` through
`daemonLoop`/`runDaemon` instead of opening stores ad-hoc) and will
allow further cleanups.
It's apparently a common footgun in Linux that the death signal isn't
preserved across calls to setuid/setgid. If nix-build gets SIGKILL-ed
while a build is running that would lead to a runaway build process that
would get reparented to init/systemd.
This is pretty easy to reproduce with the following derivation:
derivation {
name = "pdeathsig-repro";
system = builtins.currentSystem;
builder = "/bin/sh";
args = [
"-c"
''
while :; do :; done
''
];
}
And the reproduction script:
sudo nix-build repro.nix &
sleep 3
BUILDER=$(pgrep -u nixbld1)
sudo kill -9 $(pgrep -f 'nix-build.*repro')
sleep 1
ps -p $BUILDER -o pid,ppid,user,comm
To address this we have to restore the death signal after all the calls
to setuid/setgid. This is done in a helper function preserveDeathSignal
that takes a callback to avoid code duplication.
See: https://github.com/golang/go/issues/9686
This is a fairly simple function, isolated from the rest of libmain
and could be useful if new programs are made that are not part of the
main nix-cli subproject.
This commit extracts the Unix domain socket server loop (`PeerInfo`,
`getPeerInfo`, and the systemd socket activation / poll / accept loop)
from `src/nix/unix/daemon.cc` into a reusable `unix::serveUnixSocket`
function in `libcmd`.
Diagnosed by UBSan in hydra [1]:
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /nix/store/m1k4nxs8r0fl0pjxqp5n37vxgms7gdlb-gcc-14.3.0/include/c++/14.3.0/bits/move.h:234:11 in
+(prefetch.sh:6) path=
++(prefetch.sh:6) onError
++(/build/source/tests/functional/common/functions.sh:241) set +x
prefetch.sh: test failed at:
main in prefetch.sh:6
[1]: https://hydra.nixos.org/build/321098757/nixlog/1
To get `bashInteractive`, `nix develop` currently
[gets a flake reference to nixpkgs](3b473c4be5/src/nix/develop.cc (L650-L658)),
either from [`inputs.nixpkgs`](3b473c4be5/src/libcmd/installable-flake.cc (L204))
or [`<nixpkgs>`](3b473c4be5/src/libcmd/include/nix/cmd/installable-flake.hh (L87)).
Currently, this is done by name for any (locked) reference.
For any `nixpkgs` reference lacking a `flake.nix`, `nix develop` thus errors:
```
error (ignored): path '.../flake.nix' does not exist
```
While the registry does not expose info to help check this,
flake inputs expose the `flake` boolean.
This means that given `inputs.<name>.flake = false;`,
we may know in advance that our reference should not be to a flake.
This change incorporates this, so that given `inputs.nixpkgs.flake = false;`,
`nix develop` will fall back `<nixpkgs>` to use its flake for `bashInteractive`.
While Nixpkgs itself of course does expose a flake,
this check is relevant in particular for dummy inputs
(e.g. inputs using `{ url = "file:/dev/null"; flake = false; }`),
which may be overridden by `--override-input` or `.follows`,
yet do not expose (flake) code themselves.
Signed-off-by: cinereal <cinereal@riseup.net>
Extract the logic for determining whether a derivation should be resolved
before building into a dedicated method. Then use that to not resolve
unnecessarily in `nix-shell`.
The recent conversion of WorkerProto::Version from unsigned int to a
struct exposed a latent issue: `.version = 16` was being interpreted
as aggregate initialization `{.major = 16, .minor = 0}` rather than
the intended wire format value.
This commit adds -Werror=c99-designator to catch this class of bugs at
compile time. (The bug itself was fixed in
7eb23c15f39ca413a5f3cc0d3ab630311b4709be.)
For background:
The hardcoded version was originally the integer 16, which in the old
wire format (major << 8 | minor) meant version 0.16. However, the
version checks only compared minor versions via GET_PROTOCOL_MINOR(),
so this worked by accident.
After the Version struct conversion, the aggregate initialization
{.major = 16, .minor = 0} happened to still work because 16 > 1 in
lexicographic comparison against {1, 16}.
The correct version is {1, 16} because:
- The worker protocol uses major version 1 (all checks are {1, x})
- Version 1.16 is when ultimate/sigs/ca fields were added
- This matches the serialization check `>= {1, 16}`
Fixes abort on :ll if nothing has been loaded yet. Also gets rid of
redundant openStore() calls that were dead code (store can be extracted
from EvalState already) and arguably openStore is a layer violation.
Also catches EPIPE in case the pager gets interrupted to avoid superfluous
error messages.
This commit embeds the negotiated `FeatureSet` directly into `WorkerProto::Version` and introduces a `Number` inner type with total ordering, so that `Version` itself (number + features) only has partial ordering. This is a follow-up to #15155, cleaning up the separate `features` fields on `ReadConn`/`WriteConn`.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
This improves the performance of parseNarListing, which is used by commands
like `nix nar ls` when the underlying source allows cheap seeks (like StringSource
or FdSource that does lseek).
For `nix nar ls` of a NAR for linux source tarball this cuts down the runtime almost
in half (from 300ms -> 175ms).
This commit replaces the `GET_PROTOCOL_MINOR(version)` macros with a proper `WorkerProto::Version` struct. As a bonus, this also fixes some version checks that were incorrectly ignoring the major version number.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
This change is required for implementing the unprivileged garbage collection daemon,
but it may also be useful to reduce code duplication and separate out OS-specific
garbage collector roots implementations in the future.
Caching NAR hashes instead of store paths makes the cache more
general, because we can always compute the store path from the NAR
hash, but not the other way around. This is useful for lazy trees,
where we want to compute the NAR hash of an input with caching.
The global `Settings` struct contained many settings that only apply to
local builds or the local store (sandbox configuration, GC settings,
build user groups, etc.). This commit extracts these into a dedicated
`LocalSettings` struct in its own header, along with `GCSettings` and
`AutoAllocateUidSettings`.
This improves code organization and prepares for eventually making these
per-store settings in the future. Settings are accessed via
`getLocalSettings()` from the global settings object or through
`LocalStoreConfig::getLocalSettings()` for store-specific access.
Useful for commands that need a `StoreConfig` but do not want to open
the store, as a `StoreCommand` would do.
At the same time, make copy commands more clear by making store choice
for `updateProfile` explicit and removing the unused `getDstStore` function
in `StoreCommand`. This was a layer violation, as `StoreCommand` does
not have a concept of a source/destination store distinction.
This is purely a fix to use CreateFileW in mingw builds. Also adds some
FIXMEs for suspicious symlink following on truncation that can probably
be tightened down without any problems (other than nix-channel), but for
now this is a no-op change other than consistently using O_CLOEXEC, which
is harmless.
We now support `LISTEN_FDS` values greater than 1, per the systemd
socket activation spec.
These changes are by @edolstra, taken from #5265. This is just that PR
*without* the TCP parts, which I gathered are the controversial parts.
Hopefully this remainder is not so controversial.
Review with indentation ignored, because some code has moved inside a
new loop but otherwise is mostly unchanged.
aws-sdk-cpp used to include a checksum for uploads (CRC64 since ~September 2025).
Content-MD5 [1] should be universally supported by all s3 compatible services, since the SDK used
to include it unconditionally too.
[1]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
This commit consolidates the four separate boolean flags
(`permanentFailure`, `timedOut`, `hashMismatch`, & `checkMismatch`) into
a single `ExitStatusFlags` struct with methods for computing exit status
codes and updating from failure status.
The explicit serializer added in
bfdd124837 is the right place to adjust
values for sake of wire protocol compat. The protocol-agnostic `Worker`
code where it was before is the wrong spot.
(That spot was originally chosen because the back compat logic predates
having an explicit serializer for this data type to use instead.)
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
This test insisted on placing profiles in NIX_STATE_DIR, but all
packages were removed from the profile immediately after so they did not
act as garbage collector roots. Switch to directly calling nix-build,
allowing the test to run in VMs without NIX_STATE_DIR.
This commit refactors the `canonicalisePathMetaData` function to take an
options struct instead of individual parameters with platform-specific
`#ifdef`s.
The struct contains a `uidRange` field (Unix only) for build user
ownership validation, and an `ignoredAcls` field for ACLs to skip when
removing extended attributes
Due to a typo in quoteRegexChars, finding runtime garbage collection roots
was failing on paths that contained a dot, or any other regex chars that would
have to be replaced.
When fixing that error, also add tests to make sure gc continues to
work.
This introduces a `PersonalityArgs` struct to pass named arguments to `setPersonality`. The `impersonateLinux26` setting is now passed from the call site rather than read from settings inside the function.
The C++ rule of five suggests that when a custom destructor is needed
then several other functions are as well. The lack of those makes
certain operations challenging
We use a different fstat on posix and windows systems,
and not all fstat users were using the correct one.
Factor out fstat to make the change easier.
See also a13de50df3 for other stat
functions
This cleans up the logic for checking if the worker's store is a valid
local store when we're not hooking it. If we have a local store, we then
pass that as an argument to `DerivationBuildingGoal::buildLocally`,
rather than checking inside the function itself.
The two settings `envKeepDerivations` and `upgradeNixStorePathUrl` were
only used in one command each, so it makes more sense to move them to
their own files. This commit moves them both into a small self-contained
settings struct and registers them with the global config.
Currently, tests fail when the host system has `commit.gpgsign` or
`tag.gpgsign` enabled at the system level (in my case, a custom path
located at `/etc/git/config`), since the signing key is unavailable in
the test sandbox.
The tests set `HOME=$TEST_HOME` to isolate themselves, which bypasses
the user-level git config (`~/.gitconfig`). However, if a user sets the
system-level config via `GIT_CONFIG_GLOBAL` or `GIT_CONFIG_SYSTEM`, it
still applies, causing commits to fail when signing is enabled there.
In this PR, I explicitly set `GIT_CONFIG_GLOBAL` and `GIT_CONFIG_SYSTEM`
to `/dev/null` so that the user's system config is never read from or
written to. I've also replaced `git config --global protocol.file.allow
always` with `GIT_CONFIG_*` environment variables to avoid writing to
`/dev/null`.
The test was checking for `$stdenv` but the `fixed` derivation doesn't
actually have stdenv, it just has `FOO`. I've updated it to check the
value of `FOO` instead.
The settings related to diff hook (`run-diff-hook` and `diff-hook`) are
a little redundant and don't need to be leaked in derivation-builder
when computing the diff hook path to execute.
Instead of directly using both `runDiffHook` and `diffHook` settings in
derivation-builder, we can just encapsulate the logic to determine
whether or not we have a diff hook executable to run in a helper
function. We also mark `handleDiffHook` as static.
- Extract destructor logic into named methods (`deletePath()`,
`unmount()`, and `remove()`) that can be called explicitly. These ones
will throw exceptions normally, unlike the destructor which must quash
them to avoid double-exceptions.
- Use `std::filesystem::path` in `AutoUnmount` (changed from `Path`)
- Remove `del` field from `AutoRemoveJail`, using `INVALID_JAIL`
sentinel value instead.
- Add move assignment operators implemented via friend `swap` functions
for all three RAII classes.
- Remove old `reset(...)` methods that took parameters. These were a bit
misleading --- do they cancel or immediately destroy? --- and doing it
explicitly with cancel and then assignment is not hard.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
The previous implementation double-quoted the _args variable by escaping
each argument individually and then wrapping them all in single quotes,
producing output like: _args=''-e' 'arg1' 'arg2''
This fix concatenates all arguments into a single string first, then
escapes that string once, producing correct output like:
_args='-e arg1 arg2'
This prevents potential command injection issues when the output is
sourced in shell scripts.
Fixes#14327
Use wrappers to make error handling easier.
On Windows we are using proper 64-bit time and size info.
We still have the problem of no `lstat` on Windows, but this will be
dealt with in future PRs.
Follows the same pattern as `GCSettings`: extract UID allocation
settings
into a dedicated struct that Settings inherits privately from.
The current settings infrastructure prevents correct data modeling that
would allow `autoAllocateUids` to be a
`std::optional<AutoAllocateUidSettings>`.
To compensate, the getter `getAutoAllocateUidSettings()` returns a
pointer -
nullptr when disabled, providing the optional-like semantics we want.
Co-authored-by: Amaan Qureshi <git@amaanq.com>
settings.buildHook.setDefault was running after nix.conf was parsed,
causing whatever value settings.buildHook had to be clobbered.
Re-arrange the logic so that the default is set before nix.conf is
parsed, so that custom build hooks can be used by specifying them in
nix.conf.
Signed-off-by: Lisanna Dettwyler <lisanna.dettwyler@gmail.com>
This is no longer necessary and produces an eval warning:
> evaluation warning: The 'pie' hardening flag has been removed in favor of enabling PIE by default in compilers and should no longer be used.
This was first introduced in 2200f315da, but
is no longer necessary since the switch to meson.
Previously AWS CRT logs went directly to stderr via ApiHandle::InitializeLogging,
causing log spam that didn't respect Nix's verbosity settings.
This implements a custom aws_logger using the aws-c-common C API that:
- Routes all AWS logs through nix::logger
- Maps AWS log levels conservatively (ERROR/WARN -> lvlInfo) since the SDK
treats expected conditions like missing IMDS as errors
- Prefixes messages with (aws) for clarity
- Respects Nix's verbosity flags (-v, -vv, etc.)
This unsures that we call the correct virtual functions when destroying a particular
DerivationBuilder.
Usually the order of destructors is in the reverse order of inheritance:
ChrootLinuxDerivationBuilder -> ChrootDerivationBuilder -> DerivationBuilderImpl
autoDelChroot was being destroyed before the DerivationBuilderImpl::killChild was
run and it would fail to clean up the chroot directory, since there were still processes
writing to it. Note that ChrootLinuxDerivationBuilder::killSandbox was never run in
the interrupted case at all, since virtual functions in destructors do not call derived class
methods.
I could reproduce the issue with the following derivation:
let
pkgs = import <nixpkgs> { };
in
pkgs.runCommand "chroot-cleanup-race" { } ''
mkdir -p $out
for i in $(seq 1 200); do
(
mkfifo $out/fifo$i
cat $out/fifo$i > /dev/null &
while true; do
: > $out/file$i
done
) &
done
sleep 0.05
echo done > $out/main
''
While interrupting it manually when it would hang.
Wrapping the unique pointer in a custom deleter function we can run all
of the necessary clean up code consistently and calling the right virtual
functions. Ideally we'd have a lint that bans the usage of virtual functions
in destructors completely.
This addresses the concerns with network isolation that have been raised
previously [1] by only running the tests by default in a network namespace.
This way all networks tests are independent of each other and do not bind
to ports in the host namespace.
This is much neater than doing these sorts of tests in functional suite.
[1]: https://github.com/NixOS/nix/pull/14266#issuecomment-3411261285
Don't add the whole store to sandbox-paths unconditionally. Exposing
the entire store defeats the purpose of sandboxing, and when the test
store is the same as the system store (NixOS VM), it causes an obscure
"Permission denied" error.
Only add sandbox-paths when not on NixOS, indicating a separate test
store that needs access to system store build tools.
All current NixOS functional VM tests have a daemon as root with the
tests running as different unprivileged users.
The new `functional_unprivileged-daemon` test runs the daemon and the
nix functional tests as separate unprivileged users.
Users may want to run an unprivileged daemon on non-NixOS systems
where the administrator does not fully trust nix, but multiple users
want to use nix for their own purposes. It could also be useful in
concert with an overlay-mount store, where the nix daemon cannot
modify the derivations used by the system, and thus a nix vulnerability
would not lead to root code execution.
When running nix as an unprivileged user it may not be able to write to
all paths in the nix store. Ignore deletion failures to fix tests that
run `nix-collect-garbage` in this configuration.
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
We are now seeing. I guess we are out with the cache. When the API responds with 418 (I'm a teapot)
it seems like the only reasonable solution is to oblige.
error: unable to download 'http://127.0.0.1:37515/7ms9f25xyxavf32pvdc3vb28nzzmkbn3.narinfo': HTTP error 418
response body:
GitHub API error: GitHub Actions Cache throttled Magic Nix Cache. Not trying to use it again on this run.
This means that `RestoreSink` can work in the TOCTOU-resilliant way on
Windows too. And it also bodes will for the upcoming OS source accessor
improvements.
A few misc little refactors around error handling and whatnot are done
along the way too. (No more attempt to support pre Windows Vista! lol.)
This cannot be realiably automatically tested until we have a newer
version of Wine, but it does build, so I am inclined to say we just try
it for now.
This PR follows the same approach as #15043 and the
[`LogFileSettings`](https://github.com/NixOS/nix/pull/15051)
extraction:
- `GCSettings` struct inherits from virtual `Config`
- `Settings` privately inherits from it
- Accessed through `getGCSettings()`
The new method on `LocalStoreConfig` anticipates on making these
settings per-store. 0b606aad46 added both
the autoGC and periodic wakeups, which is why we think they are related.
When an upload fails with a transient HTTP error (e.g., S3 rate limiting
with HTTP 503), retries would fail with "curl error: Failed to open/read
local data from file/application" because the upload source was already
exhausted from the previous attempt.
Restart the source in init() to ensure it's at the beginning for both
first attempts (no-op) and retries (necessary fix).
Fixes: #15023
Progress on #5638
Replace the SQLite constructor's mode parameter with a Settings struct
that includes both the open mode and useWAL flag. This makes the
dependency on useSQLiteWAL explicit at call sites rather than having
it read from the global settings inside the constructor.
All call sites now explicitly pass settings.useSQLiteWAL, preparing
for downstream work where stores can pass their own settings instead
of relying on the global.
Some S3-compatible services (like GCS) modify the Accept-Encoding header
in transit, which breaks AWS SigV4 signature verification since curl's
implementation signs all headers including Accept-Encoding.
Fixes: #15019
It is possible that the `nix` executable is installed but not `nix-env`
(this may be unusual but for example in Fedora we have a separate
`nix-legacy` subpackage, which includes the `nix-env` symlink).
The current error message:
```
$ nix config check --verbose
Running checks against store uri: local
[FAIL] Multiple versions of nix found in PATH:
```
when there is no nix-env in PATH is confusing.
This change makes the error message precise for the missing nix-env case.
Use
```
git show --color-moved --patience --color-moved-ws=ignore-all-space
```
to review and see that this is mostly code motion.
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
Introduce a new `Signature` struct that represents a cryptographic
signature
along with the key name that produced it. This provides:
- Structured representation instead of colon-separated strings
- Type-safe parsing with `Signature::parse()`
- Serialization with `to_string()`
- JSON serialization/deserialization
- Batch parsing with `parseMany<Container>()`
- Batch serialization with `toStrings()`
This is scaffolding for future changes that will use this type
throughout the codebase.
This will once and for all get rid of all double-quoting issues. On windows the quoting
is doubly bad because it escaped all \ to \\, which is very bad for error messages. In
order to prevent future regression std::filesystem::path formatting now must use a special
type PathFmt (like Magenta). In the future we could even change how we render filesystem paths.
Instead of the stringly typed code we should use an enum class, this is
more clear and less error-prone. Also adds settings implementations for
CompressionAlgo and std::optional<CompressionAlgo>. The first is used
for NAR compression, since we never accepted empty strings there:
error: unknown compression method ''
The other one is used for optional .narinfo, .ls, and log/ compression.
Those treated empty strings as compression being disabled. The same exact
semantics is kept.
This has the benefit of improving error messages for incorrect values:
error: option 'compression' has invalid value 'bz'
Did you mean one of br, xz or lz4?
The docs were out of date. Since 8a0c00b856 Nix
supports all compression algorithms exposed by libarchive (if it's built with
native support for them). Let's be honest about it in the docs.
This avoids the wall of text like, because ThreadPool doesn't print interrupts
on shutdowns.
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
Without this we can abort by throwing an exception in the destructor:
[24/635/2958 copied (3.8/26.0 GiB)] copying path '/nix/store/ncd2iic2nwxwhqsf4gp9sdybkwnwz20b-ruby3.3-mini_portile2-2.8.9' from 'ssh-ng://localhost:22'
Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:
Exception: nix::Interrupted: error: interrupted by the user
Stack trace:
0# 0x00000000004AFFE9 in result/bin/nix
1# 0x00007F946290A1AA in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
2# __cxa_call_terminate in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
3# __gxx_personality_v0 in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
4# 0x00007F946283FA19 in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libgcc_s.so.1
5# _Unwind_RaiseException in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libgcc_s.so.1
6# __cxa_throw in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
7# 0x00007F94635D82D0 in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
8# nix::Pid::wait() in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
9# nix::Pid::~Pid() in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
forced_unwind is thrown by Boost.Context when destroying the coroutine.
This lead to us resetting the remote connection for each narFromPath
with the ssh-ng:// store, so copying was very slow.
This makes all addToStore operations that use these source accessors
constant memory regardless of file sizes. Also make the other overload
altogether and relegate it to the base class as a non-virtual method to
avoid such mistakes.
This factors out the helper function from seekableGetNarBytes into copyFdRange
and adds some more sanity checks for offset/length truncation/wrapping at that
API boundary where we work with NAR-style offsets and convert to native off_t.
Instead of mutating the file pointer we can instead safely do
preads. That makes the local-nar-info cache once again thread safe
without the overhead of reopening the file that we used to have prior
to b9b6defca6 which broke the thread safety
by persisting the file descriptor.
Without the change the build fails for me as:
../unix/file-descriptor.cc:404:70: error: 'RESOLVE_BENEATH' was not declared in this scope
404 | dirFd, path.rel_c_str(), flags, static_cast<uint64_t>(mode), RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS);
| ^~~~~~~~~~~~~~~
This happens for 2 reasons:
1. `__NR_openat2` constant was not pulled in from the according headers
and as a result `<linux/openat2.h>` was not included.
2. `define HAVE_OPENAT2 0` build is broken: refers to missing
`RESOLVE_BENEATH` normally pulled in from `<linux/openat2.h>`
This changes fixes both.
Document the nix-cache-info file format used by binary caches, including
the StoreDir, WantMassQuery, and Priority fields, their behavior, and
links to related store options.
This is the usual conventions on windows.
See https://learn.microsoft.com/en-us/windows/win32/shell/knownfolderid and
https://github.com/adrg/xdg for examples of the mapping of XDG paths to Windows
known folders.
Additionally, on Windows, this allows us to dispense with a hard-coded
default for `nixConfDir`, which is both nice (fewer compile-time
configuration options) and necessary, because we don't know what drive
the `ProgramData` directory will live on.
Tested on wine.
Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
The test data was using invalid signature strings like "asdf" and
"qwer" which don't follow the required "name:base64signature" format.
This updates them to use properly formatted signatures with valid
base64-encoded data.
Document the Nix32 base-32 variant used for store path digests and
hash output. The new page covers:
- The 32-character alphabet (omitting e, o, u, t)
- Byte order differences from base-16 encoding
Also update references throughout the manual to link to the new page.
Change "cannot build missing derivation" to "failed to obtain derivation of"
since the path (e.g. '...drv^out') is a derivation output, not a derivation.
The message could be improved further to resolve ambiguity when multiple
outputOf links are involved, but for now we err on the side of brevity
since this message is already merged into larger error messages with
other context from the Worker and CLI.
When !keepGoing and a goal fails, other goals are cancelled and
remain with exitCode == ecBusy. These cancelled goals have a default
BuildResult::Failure{} with empty errorMsg.
Previously, buildPathsWithResults would return these cancelled goals,
and throwBuildErrors would report them as failures. When only one such
cancelled goal was present, it would throw an error with an empty
message like:
error: build of '/nix/store/...drv^*' failed:
Now we skip goals with ecBusy since their state is indeterminate.
Cancelled goals could be reported, but this keeps the output relevant.
Other indeterminate goal states were already not being reported, for
instance: derivations that weren't started for being blocked on a
concurrency limit, or blocked on a currently building dependency.
When keepGoing=false and a build fails, other goals are cancelled.
Previously, these cancelled goals were reported in the "build of ...
failed" error message alongside actual failures. This was misleading
since cancelled goals didn't actually fail - they were never tried.
Update the test to expect only the actual failure (hash mismatch) to
be reported, not the cancelled goals.
DerivationTrampolineGoal is the top-level goal whose buildResult is
returned by buildPathsWithResults. When it failed without setting
buildResult.inner, buildPathsWithResults would return failures with
empty errorMsg, producing error messages like:
error: failed to build attribute 'checks.x86_64-linux.foo',
build of '/nix/store/...drv^*' failed:
(note the empty message after "failed:")
Use the new doneFailure helper to ensure buildResult is populated
with meaningful error information.
There can be a long time between the creation of `TransferItem` and
the start of the curl download, which can lead to misleading download
durations and progress bar status. So now we create the `Activity` and
update `startTime` when curl actually starts the download.
Previously, calling queryValidPaths() with a large number (e.g. 100K)
of store paths failed because Nix immediately creates a `TransferItem`
for each .narinfo, which is then registered as a handle with
curl. However curl appears to scale poorly internally: even though
only a few downloads are actually started (up to the
connections/streams limits), it spends a lot of CPU time dealing with
the inactive handles. So the curl thread is sitting at 100% CPU, the
active downloads stall and time out, and everything grind to a halt.
So now we limit the number of curl handles to http-connections *
5. With this, fetching 100K .narinfo files from localhost succeeds in
~15 seconds.
This operation has been deprecated since
09a6321aeb (July 2012). It was used by
client versions <= 11, which is below `MINIMUM_PROTOCOL_VERSION`
(currently 18).
We can get rid of `NarMember`, because it is just `NarListing` in
disguise! The use of `std::variant` makes clear that certain stat fields
we don't care about in the non-regular-file case too.
Relative paths (e.g., "relative/repo") would crash in renderAuthorityAndPath()
because an empty authority was set, violating RFC 3986 section 3.3 which
requires paths to start with "/" when an authority is present.
Fix by only setting authority for absolute paths:
- Absolute paths: file:///path (empty authority)
- Relative paths: file:path (no authority)
Also reject SCP-like URLs without a user (e.g., "github.com:path") with a
clear error message, since proper support requires careful implementation,
which is not something I can do right now.
Relative paths like `file:./foo.tar.gz` have never worked for tarballs
because curl rejects relative file: URLs. Previously this resulted in
cryptic curl errors. Now we reject them early with a clear message
explaining that relative paths are not supported because there is no
defined base directory to resolve them against.
See https://github.com/NixOS/nix/issues/12281
When a goal with an active child process is destroyed (e.g., during
failure cascades without `--keep-going`), the child process gets killed
but `childTerminated` was never called. This left stale entries in the
worker's `children` list.
Fix this by ensuring `childTerminated` is called from destructors:
- `DerivationBuilderImpl::killChild` now calls `childTerminated` via
the `miscMethods` callback.
- `HookInstance` gains an `onKillChild` callback that is invoked from
its destructor when killing the process. `buildWithHook` sets this
callback to call `childTerminated`.
To make these calls safe from destructors (where the goal object may be
partially destroyed), add a new overload of `Worker::childTerminated`
that takes an explicit `JobCategory` parameter instead of calling the
virtual method `Goal::jobCategory`. The original overload still exists
for convenience for normal (non-destructor) call sites.
Add helpers to the base Goal class that set buildResult and call amDone,
ensuring buildResult is always populated when a goal terminates.
Derived class helpers now call the base class versions. This reorders
operations: previously buildResult was set before bookkeeping (counter
resets, worker stats), now it's set after. This is safe because the
bookkeeping code (mcExpectedBuilds.reset(), worker.doneBuilds++,
worker.updateProgress(), etc.) only accesses worker counters, not
buildResult.
It correctly models the is-a relation. This will be useful for doing a dynamic_cast in
downstream code that wants to copy from a file descriptor to a file descriptor.
Nix currently doesn't do any resource control, and Delegate=yes turns on all the controllers.
In particular, this enables using cpusets with cgroups V1 alongside the Nix daemon.
This also adds a utility for opening a file descriptor from a path in readonly mode.
Previous commit helps a bit with error handling, since now we just throw a NativeSysError.
This way each consumer of NativeSysError doesn't have to
also conditionally include the windows-error.hh, which is very cumbersome.
And we can't include windows-error.hh in error.hh because of a circular import.
Previously builtins.readDir would return an empty attribute set
instead of barfing on non-existent paths. This is a regression from
2.32 for impure eval.
Using fchmodat after a fstatat in deletePath has a slight TOCTOU
window. We can plug it by using fchmodat (the libc wrapper with
AT_SYMLINK_NOFOLLOW), but it tries fchmodat2 and falls back to the
O_PATH trick while failing when procfs isn't mounted. We can do a bit
better than that and also cache whether syscalls are unsupported to
avoid the repeated context switching that glibc would impose.
Also tests the fallback path. It's only for kernels older than 6.6 and
when procfs isn't accessible that we fall back to the racy fchmodat
without AT_SYMLINK_NOFOLLOW.
What previously used to be:
openat(AT_FDCWD, "/tmp/store-race/nix/var/nix/builds", O_RDONLY) = 11
newfstatat(11, "nix-2704212-84654554", {st_mode=S_IFDIR|000, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(11, "nix-2704212-84654554", 040700) = 0
Is now a TOCTOU-free sequence of syscalls:
openat(AT_FDCWD, "/tmp/store-race/nix/var/nix/builds", O_RDONLY) = 11
newfstatat(11, "nix-2704953-1733606057", {st_mode=S_IFDIR|000, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat2(11, "nix-2704953-1733606057", 040700, AT_SYMLINK_NOFOLLOW) = 0
Or if the fchmodat2 is not supported:
openat(11, "nix-2705443-3010460784", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 12
fstat(12, {st_mode=S_IFDIR|000, st_size=3, ...}) = 0
chmod("/proc/self/fd/12", 040700) = 0
openat(11, "nix-2705443-3010460784", O_RDONLY|O_NOFOLLOW|O_DIRECTORY) = 12
This prevents a potentially arbitrary chmod that follows symlinks,
though the race window is very small. Also in the case that fchmodat2
isn't supported we could instead open the /proc/self/fd/N path instead
of using openat, but that's pretty much equivalent. We only care
about ensuring that the thing we chmodded wasn't a symlink since
fchmodat follows symlinks and the support for AT_SYMLINK_NOFOLLOW
in libc for that is pretty spotty on Linux. E.g. glibc fails if the
AT_SYMLINK_NOFOLLOW is specified and procfs isn't available even on
regular files. The patch also includes a test that uses a user namespace
on Linux to test this exact scenario (though it's rather exotic).
This makes the logic much easier to follow. Unlike before, the use of
separate functions is not making us pass a gazillion arguments or use
the crutch of class variables.
There was a bunch of logic in there which was, effectively, using the
build hook, rather than deciding *whether* to use the build hook. We
want it to only be the latter.
Pulling in the java into the tests closure for just testing a piece of code
for the docs (and the tests actually are wrong, since a correct parser must *reject*
those NARs). This is too much of an ask to maintain for zero benefit. I already had
to disable it basically everywhere, because it works only on linux.
It can be revisited in the future, but considering that it's not exercised anywhere and
shouldn't be used anywhere other than a toy example for the docs I think it's best to drop
it.
This is no longer needed (best I can tell), since nix docker
images now get uploaded to GHCR as part of the release process too
and they contain both aarch64 and x86_64 instead of only x86_64.
I messed up and accidentally configured the S3 client to use the same
host as the nix-releases bucket, but nix-channels is us-east-1 and
nix-releases is eu-west-1.
This workflow is supposed to automate release uploads by using OIDC
for AWS setup. DockerHub still uses long-lived credentials, but that's
not fixable. In a follow-up we could set up release uploads to GHCR too.
Previously it was only Eeclo doing releases that were signed with
B541D55301270E0BCF15CA5D8170B4726D7198DE. Other linux distributions
have the expectation (rightfully so) that our tags are signed. Let's
document this.
We could do cross-signing to make tracing the chain of trust easier
for all Nix team members [1].
[1]: https://nixos.org/community/teams/nix/
This allows for testing with a local minio deployment like:
./upload-release.pl --skip-docker --skip-git --s3-endpoint http://localhost:9000 --s3-host localhost:9000 1821360
Add a test case that explicitly demonstrates NAR hashing of a directory
without using a filter. Add comments to clarify what each test case is
testing (NAR vs flat hashing).
The sha256 parameter documentation said "file at the path" but it
works with directories too (using NAR hashing). Link to the
content-address documentation instead of duplicating information.
Remove include of signals.hh from signals-impl.hh to fix
misc-header-include-cycle warning. The impl header is only included
from signals.hh which already provides the necessary declarations.
When using designated initializers, clang-tidy warns about skipped
fields. Explicitly initialize pos to {} to silence the
clang-diagnostic-missing-designated-field-initializers warning.
Runs the tests against the new daemon as well as the cli.
This more reliably shares the artifact (not relying directly on github
actions cache). We've seen github evict our caches super fast, so it would
be nice to move away from it entirely if possible.
Best reviewed with -w --color-moved. This just moves the code
into a separate workflow. This will allow us to reuse it in
the release job for github releng of releases.
When a remote SSH client disconnects during a long-running operation
like addToStore(), the nix-daemon can deadlock in a circular wait:
- Process A (SSH daemon): blocked reading from downstream store socket,
waiting for response from local daemon
- Process B (local daemon): blocked reading from upstream socket,
waiting for more NAR data from SSH daemon
The existing interrupt mechanism (ReceiveInterrupts + MonitorFdHup)
correctly detects the SSH disconnect and sets _isInterrupted, but the
daemon remains blocked in read() on the downstream store connection.
Even though SIGUSR1 causes read() to return EINTR, the circular
dependency prevents forward progress.
Fix this by adding shutdownConnections() to RemoteStore that calls
shutdown(fd, SHUT_RDWR) on all tracked connection file descriptors.
Register an interrupt callback in processConnection() that invokes
this method when the store is a RemoteStore. This causes any blocking
read() to return 0 (EOF), breaking the circular wait and allowing
both processes to exit cleanly.
The fix tracks connection FDs in a synchronized set, populated when
connections are created by the Pool factory. On interrupt, all FDs
are shut down regardless of whether they're idle or in-use.
Makes the error messages render paths correctly, also introduces
a new hierarchy of error classes for SourceAccessor related errors
that we might want to handle differently (e.g. like when doing a readFile
on a directory and such). This should make it easier to implement better
UnionSourceAccessor and AllowListSourceAccessor by catching these errors
consistently.
See #8188. Resolves issues about the error not
being actionable, but I am not marking it closing
yet because of further discussion about the naming
of these flags in the thread.
`nix build --rebuild` (and others)
will fail if the derivation has not been built
before, because it runs a check build and
confirms that the build was deterministic.
It may be unclear to users that --rebuild will fail
if the derivation has never been built before,
because the flag makes no indication that a
determinism check occurs.
The error message does
not help clear this up, or provide any actionable
steps, and at first glance seems to indicate that
the derivation being built is invalid, rather than
just not present in the store:
```
error: some outputs of '...' are not valid, so checking is not possible
```
We can suggest to the user the following (correct)
rewrites. This list of commands that may result in
the error is comprehensive.
- `nix build --rebuild` to `nix build` or `nix build --repair`
- `nix-build --check` to `nix-build` or `nix-build --repair`
- `nix-store --realise --check` to `nix-store --realise` or `nix-store --realise --repair`
Wording is based on that in the documentation:
```
(nix build)
--repair During evaluation, rewrite missing or
corrupted files in the Nix store. During
building, rebuild missing or corrupted
store paths.
(nix-build)
--repair Fix corrupted or missing store paths by
redownloading or rebuilding them. Note
that this is slow because it requires
computing a cryptographic hash of the
contents of every path in the closure
of the build. Also note the warning
under nix-store --repair-path.
(nix-store --realise)
--repair Fix corrupted or missing store paths by
redownloading or rebuilding them. (etc)
```
Make the C API error message more explicit about what went wrong and
why it's invalid. The new message explains that a zero-length path was
passed and clarifies that it would refer to the flake itself.
Updates the unit test to match the new error message.
Make the error message more explicit about what went wrong and why
it's invalid. The new message explains that a zero-length path was
passed and clarifies that it would refer to the flake itself.
An empty path refers to the flake itself, not an input. Apply the same
type safety to inputUpdates as inputOverrides.
The deprecated --update-input flag (deprecated since Nix 2.4) and the
modern 'nix flake update' command now properly reject empty paths.
Includes functional tests for both commands.
Wraps InputAttrPath with compile-time guarantee of non-emptiness.
Replaces obscure .back() calls with domain-specific inputName() method.
An empty path refers to the flake itself, making it nonsensical for
input override operations. The type system now prevents this.
- getDerivations() filters attribute names with std::regex_match, which runs the regex engine for every attribute visited during nixpkgs scanning.
- BM_GetDerivationsAttrScan/10000_mean: 3.338 ms → 1.506 ms (≈ -54.9%)
- RegexCache::get() returned std::regex by value, copying the compiled regex on every cache hit.
- Store the compiled regex behind std::shared_ptr<const std::regex> and return the shared pointer instead, so callers reuse the same compiled object.
- BM_EvalManyBuiltinsMatchSameRegex_mean improved about 8%
Remove the per-call reserve() inside printString to avoid linear-growth reallocations when called in loops (e.g. printStrings). Derivation::unparse already pre-reserves a large buffer, so this remains efficient while preserving amortized growth behavior when the initial estimate is exceeded.
Testing with 10 derivations is sufficient to verify performance
characteristics. The larger test cases (50, 200) don't provide
additional insight and slow down the benchmark unnecessarily.
The separate checkInvariants loop after addValidPath was added in 2014
(d210cdc43) to work around an assertion failure:
nix-store: derivations.cc:242: Assertion 'store.isValidPath(i->first)' failed.
At that time, hashDerivationModulo() contained assert(store.isValidPath(...))
which required input derivations to be registered as valid in the database
before computing their hash. The workaround was to:
1. Call addValidPath with checkOutputs=false
2. Add all references to the database
3. Run checkInvariants in a separate loop after paths were valid
In 2020 (bccff827d), the isValidPath assertion was removed to fix a
deadlock in IFD through the daemon (issue #4235). The fix changed
hashDerivationModulo to use readInvalidDerivation, which reads directly
from the filesystem without requiring database validity.
This made the separate checkInvariants loop unnecessary, but nobody
noticed the code could be simplified. The comment "We can't do this in
addValidPath() above, because the references might not be valid yet"
became stale.
Now we simply call addValidPath() with the default checkOutputs=true,
which runs checkInvariants internally using the already-parsed
derivation. This commit eliminates the separate loop over derivations.
- LocalStore::registerValidPaths() parsed derivations twice: once in addValidPath() and again when calling checkInvariants(), despite already having loaded the derivation.
- Plumb the parsed Derivation out of addValidPath() and reuse it for the invariant check pass, falling back to re-parsing only when a derivation wasn’t newly registered in this call.
- BM_RegisterValidPathsDerivations/200_mean runs 32% faster
rsync was only used to copy source files while following symlinks.
Replace with tar --dereference, which serves the same purpose.
Tried plain cp but couldn't get it to work reliably. tar is already
a test dependency.
Add tests/functional/derivation to fileset to include the symlink
targets.
Fixes#14776
While working on #12464, I realized this method was not correct in this
case. With the current binary cache format, it is harmless, since we
don't create arbitrary directories, but with my change, we started to.
Regardless of whether we need it or not, I think it is better if the
function just does the right thing.
For windows we should live fully in the HANDLE land instead
of converting back-n-forth (which sometimes is destructive).
Using native API is much better for this.
It doesn't track the number of bytes deleted, but since this code is
security critical also we can split unix and windows implementations.
If the need arises we can implement a smarter recursive deletion function
ourselves in the future.
Review with --color-moved.
This at least makes canonPath not consider the drive letter as a path
component. There still some issues with it on windows, but at least
this gets us through some of the libutil-tests.
Also since we don't want to change which env variables nix considers
we don't use std::filesystem::temp_directory_path and implement the
windows version directly.
- Skip packages that don't build for Windows when building for windows
- Automatically disable kaitai / json schema, fixing todo
- Skip native build of Nix for manual
Should be pretty self-explanatory. We didn't really have unit tests
for the filesystem source accessor. Now we do and this will be immensely
useful for implementing a unix-only smarter accessor that doesn't suffer
from TOCTOU on symlinks.
We now have a nice separation of concerns: `DrvOutputSubstitutionGoal`
is *just* for getting realisations, and `PathSubstitutionGoal` is just
for fetching store objects.
The fetching of store objects that this used to do is now moved to the
caller.
This progress on #11896. It introduces some issues temporarily which
will be fixed when #11928 is fixed.
The SQL tables are left in place because there is no point inducing a
migration now, when we will be immediately landing more changes after
this that also require schema changes. They will simply be ignored by in
this commit, and so all data will be preserved.
Error messages now include suggestions like:
error: unknown compression method 'bzip'
Did you mean one of bzip2, gzip, lzip, grzip or lrzip?
Also a bit of progress on making the compression code use less stringly
typed compression type, which is good because it's easy to confuse
which strings are accepted where (e.g. Content-Encoding should be able
to accept x-gzip, but it shouldn't be exposed in NAR decompression and
so on). An enum cleanly separates the concerns of parsing strings / handling
libarchive write/read filters.
The previous error message was ambiguous about which specific directory failed the check.
This commit updates checkNotWorldWritable to return the failing path so it can be included in the error message, making debugging easier.
Ref #14787
This really doesn't really fixes the problem of the symlink, but it
solves the progress of getting windows working.
TODO: find out if it's a bug from meason & make a feature request to
avoid symlinks or generate symlinks upon build and git ignore, but still
goes back to the issue of is this a bug or do we need to make a feature
requests.
Co-authored-by: John Ericson <git@JohnEricson.me>
- More concise
- Also checks error messages
- Checks more error codes
The nature of that bug is that if the first command's exit status is
correctly 101 and not 1, the rest should be correctly 101, 100, etc.
too.
Unfortunately previous tarball caches had loose objects written to
them and subsequent switch to thin packfiles. This results in possibly
broken thin packfiles when the loose objects backend is disabled. Thin
packfiles do not necessarily contain the whole closure of objects.
When packfilesOnly is true we end up with an inconsistent state where
a tree lives in a packfiles which refers to a blob in the loose objects
backend.
In the future we might want to nuke old cache directories and repack
the tarball cache.
The SSO provider was unconditionally setting profile_name_override to
the (potentially empty) profile string from the S3 URL. When profile
was empty, this prevented the AWS CRT SDK from falling back to the
AWS_PROFILE environment variable.
Only set profile_name_override when a profile is explicitly specified
in the URL, allowing the SDK's built-in AWS_PROFILE handling to work.
The default (empty) profile case was using CreateCredentialsProviderChainDefault
which didn't properly support role_arn/source_profile based role assumption via
STS because TLS context wasn't being passed to the Profile provider.
This change unifies the credential chain for all profiles (default and named),
ensuring:
- Consistent behavior between default and named profiles
- Proper TLS context is passed for STS operations
- SSO support works for both cases
Add validation for TLS context and client bootstrap initialization,
with appropriate error messages when these fail. The TLS context failure
is now a warning that gracefully disables SSO, while bootstrap failure
throws since it's required for all providers.
This enables seamless AWS SSO authentication for S3 binary caches
without requiring users to manually export credentials.
This adds SSO support by calling aws_credentials_provider_new_sso() from
the C library directly. It builds a custom credential chain: Env → SSO →
Profile → IMDS
The SSO provider requires a TLS context for HTTPS connections to SSO
endpoints, which is created once and shared across all providers.
Good to explicitly declare things to not accidentally do twice the work by
preventing that kind of misuse.
This is essentially just cppcoreguidelines-special-member-functions lint
in clang-tidy.
This is mostly theoretical, but the code was calling getenv("TZ")
twice: once to check if it's non-null, and again to get its value.
This creates a potential race condition where the environment could
change between calls.
The nix_api_store.cc tests and derivation-parser-bench.cc were using raw
getenv() calls or unsafe .value() calls on optional, which would segfault
when passed to std::filesystem::path constructor if the
_NIX_TEST_UNIT_DATA environment variable was not set.
The previous implementation called .value() on std::optional without
checking if it had a value. When _NIX_TEST_UNIT_DATA was not set, this
would throw std::bad_optional_access or cause a segfault in code that
used the raw getenv() result.
The new implementation checks the optional first and throws an Error
with a helpful message directing users to run tests via meson. The
example includes --gdb since this situation may arise when trying to
debug tests without knowing about meson's test infrastructure.
Pretty self-explanatory. More RAII is good and unclutters the already heavily overloaded
destructors from ownership logic. Not yet touching CURL *req because that would be too churny.
Fetching gcc-15.2.0.tar.gz I get a warning about UTF8 archive names. This
now mentions problematic pathnames.
warning: getting archive member 'gcc-15.2.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Äfoo.go': Pathname can't be converted from UTF-8 to current locale.
warning: getting archive member 'gcc-15.2.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Ämain.go': Pathname can't be converted from UTF-8 to current locale.
Also apparently libarchive depends on locale (yikes). Fixing reproducibility issues
that stem from this is a separate issue. At least having the warning actually mention
the pathname should be useful enough even though it's not actionable.
At least using the default locale yields something sane:
builtins.readDir "${gcc}/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir"
{
"Äfoo.go" = "regular";
"Ämain.go" = "regular";
}
Fix intermittent SIGSEGV (exit code 139) on macOS when running
nix-shell and shebang tests inside the nix sandbox.
The foo, bar, and ruby test scripts were created without shebangs,
which causes intermittent crashes when executed via command
substitution on macOS. Adding proper shebangs resolves the flakiness.
Potentially closes: #13106
This matches what we just did for `nix path-info`, and I hope will allow
us to avoiding any more breaking changes to this command for the
foreseeable future.
Previously the daemon didn't validate that it got a valid GCAction
and did a naive C-style cast to the enum. This is certainly unintentional,
albeit mostly harmless in practice.
The fact that we were introducing a conversion from the output of `nix
path-info` into the input of `builtins.fetchTree` was the deciding
factor. We want scripting outputs into inputs like that to be easy.
Since JSON strings and objects are trivially distinguishable, we still
have the option of introducing the JSON format as an alternative input
scheme in the future, should we want to. (The output format would still
be SRI in that case, presumably.)
Floating CA outputs just have a hash algorith, not a whole hash. It is
fixed ones which are a pair of a method and a hash, just like the `ca`
field of store object info.
This restores compatibility with Nix 2.18, which behaved this
way. Note that this doesn't scan for the actually visible references.
Unlike in Nix 2.18, we only do this for paths with context, i.e. it
applies to `builtins.storePath "/nix/store/bla..."` but not
`"/nix/store/bla..."`. We don't want the latter because it shouldn't
matter whether a source file happens to be in the Nix store.
This was lost after 2.32 while making the accessor lazy. We can restore the support
for it pretty easily. Also this is significant optimization for nix nar cat.
E.g. with a NAR of a linux repo this speeds up by ~3x:
Benchmark 1: nix nar cat /tmp/linux.nar README
Time (mean ± σ): 737.2 ms ± 5.6 ms [User: 298.1 ms, System: 435.7 ms]
Range (min … max): 728.6 ms … 746.9 ms 10 runs
Benchmark 2: build/src/nix/nix nar cat /tmp/linux.nar README
Time (mean ± σ): 253.5 ms ± 2.9 ms [User: 56.4 ms, System: 196.3 ms]
Range (min … max): 248.1 ms … 258.7 ms 12 runs
The whole NarAccessor -> listing -> lazy NarAccessor is very weird. Source
can now be seek-ed over when supported, so we can support it pretty easily.
Alternatively we could also make it single-pass very easily with a custom
FileSystemObjectSink. It will get removed in a follow-up commit anyway.
It's not evaling on hydra currently and is only part of the docs. Support
for this it best effort at best so we should not build this in hydra, considering
the amount of effort required to support this. I would even consider dropping these
checks and component altogether, since there doesn't seem to be much interest in maintaining
these docs from the core team anyway.
This somehow fails https://hydra.nixos.org/build/315675349/nixlog/1. I don't know the exact
details, but it seems that something goes very wrong with LTO and sanitizers that lead to the
string literal to be moved? Instead of relying on the string literal deduplication to provide
a consistent address we can use a global. That should have a single address (modulo wonky copy
relocations).
I didn't do things quite right in 496e43ec72:
- Forgot to remove the now-redundant `isAllowed` check.
- Called the non-virtual, not the superclass's impl, in
`addDependencyPrep`, causing bad recursion / UB.
Doing this fixes a crash I encountered with manual testing an Nix Ninja
--- hopefully we will get Nix Ninja or similar in a NixOS test longer
term to defend against this thing happening again.
Add tests for function equality covering both direct comparisons and
comparisons within composite types (lists and attribute sets).
Tests verify:
- Direct function comparisons always return false
- Value identity optimization in composite types allows identical
functions to compare as equal when both references point to the
same function value
Correct and clarify evaluation semantics including to help users
understand Nix language behavior without unnecessarily pinning down
the implementation.
Add `html-manual` Meson option to allow building manpages without the
HTML manual, removing the mdbook dependency for manpage-only builds.
Changes:
- Add `html-manual` Meson option (default: true)
- Make HTML manual build conditional in meson.build
- Add `buildHtmlManual` parameter to package.nix
- Conditional outputs: ["out" "man"] when enabled, ["out"] when disabled
- Make mdbook/rsync/json-schema-for-humans dependencies conditional
- Add `nix-manual-manpages-only` package variant
This allows systems that only need manpages to avoid the mdbook build
dependency while preserving full functionality for HTML manual builds.
Add configurable documentation URLs that change based on whether this is
an official release or development build:
- Nix builds:
- Development (officialRelease = false): Use /latest/ URLs
- Official releases (officialRelease = true): Use versioned URLs with
MAJOR.MINOR only (e.g., /2.33/ instead of /2.33.0/)
- Plain meson builds: Default to versioned URLs (official-release = true)
Changes:
- Add --doc-url parameter to expand-includes.py
- Add meson option 'official-release' (defaults to true for Meson builds)
- Compute doc_url in meson.build based on version and official-release
- Forward Nix officialRelease variable to Meson in package.nix
- Update render-manpage.sh to pass doc-url parameter
This allows distros (Fedora, etc.) to have stable versioned URLs by default,
while Nix development builds point to /latest/ for up-to-date documentation.
Add standalone markdown preprocessor to generate manpages without requiring
mdbook's Rust toolchain. This removes a significant build dependency for
manpage generation while keeping the HTML manual (mdbook) working unchanged.
Changes:
- Add expand-includes.py: Python 3 script that recursively expands
{{#include}} directives, resolves @docroot@ to nix.dev URLs, and handles
@generated@/ paths for build-generated files
- Update render-manpage.sh: Replace mdbook-based implementation with
standalone version that uses expand-includes.py + lowdown
- Update meson.build: All 134 manpage targets now use standalone renderer
with proper dependencies (expand-includes.py, experimental-features-shortlist)
- Fix nix-hash.md: Remove extra parenthesis in markdown link syntax
Benefits:
- No mdbook/Rust toolchain required for manpage builds
- Manpages contain nix.dev/latest URLs instead of broken relative paths
- Fixes bug where mdbook didn't expand experimental-features-shortlist.md
- 98.5% identical output to mdbook (2 files differ, both acceptable)
All 134 manpages (131 section 1, 2 section 5, 1 section 8) build successfully.
We have the machinery to make a more informative error, telling the
user what format was actually encountered, and not just that it is not
the format that was requested.
The S3BinaryCacheStoreConfig::doc() function was returning a minimal
hardcoded 3-line string instead of including the comprehensive
documentation from s3-binary-cache-store.md.
This was introduced in PR #13752 which moved the prose documentation to
the markdown file but forgot to update the doc() function to use it.
The previous commit hacked it into the output of `nix path-info --json`,
this cleans that up my making it an actual field of that data type, and
part of the canonical JSON serializers for it (and `ValidPathInfo` and
`NarInfo`).
Beyond cleaning up the JSON code, this also opens the doors to things
like:
- Binary caches that contain store objects that don't all belong in the
same store directory
- Relocatable store objects which carefully don't mention any store
directory by absolute path, and instead use relative paths for
anything. (#9549)
As discussed today at great length in the Nix meeting, we don't want to
break the format, but we also don't want to impede the improvement of
JSON formats. The solution is to add a new flag for control the output
format.
Note that prior to the release, we may want to replace `--json
--json-format N` with `--json=N`, but this is being left for a separate
PR, as we don't yet have `=` support for CLI flags.
Fix#14532.
As discussed on the call today:
1. We'll stick with `format = "base16"` and `hash = "<hash>"`, not do
`base16 = "<hash>"`, in order to be forward compatible with
supporting more versioning formats.
The motivation we discussed for someday *possibly* doing this is
making it easier to write very slap-dash lang2nix tools that create
(not consume) derivations with dynamic derivations.
2. We will remove support for non-base16 (and make that the default, not
base64) in `Hash`, so this is strictly forward contingency, *not*
yet something we support. (And also not something we have concrete
plans to start supporting.)
This leads to incredibly wasteful refreshes (see [^]) when oids are not found.
Since we are writing the pack files only once per unpacking we should not bother
with this refreshing at all.
This brings down the number of syscalls during `nix flake metadata "https://releases.nixos.org/nixos/25.05/nixos-25.05.813095.1c8ba8d3f763/nixexprs.tar.xz" --store "dummy://?read-only=false"`
Down from 576334 to just 6235 (100x less syscalls):
(Before)
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ------------------
32.98 0.625288 3 162898 getdents64
29.58 0.560686 3 163514 81917 openat
15.01 0.284509 3 81819 186 newfstatat
10.99 0.208349 2 81601 close
10.56 0.200145 2 81552 fstat
All these are coming from [2] and are totally useless.
(After)
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ------------------
76.47 0.108558 247 438 20 futex
6.55 0.009292 18 513 munmap
3.30 0.004680 7 639 492 openat
2.68 0.003803 10 359 write
2.30 0.003268 2 1146 read
2.26 0.003215 3 870 mmap
[^]: 58d9363f02/include/git2/sys/odb_backend.h (L68-L75)
[2]: 58d9363f02/src/libgit2/odb_pack.c (L517-L546)
Fixes#14628
- Remove mdbook-linkcheck dependency and configuration (was blocking
upgrades to mdbook 0.5.0+, configured with warning-policy = "ignore"
due to false positives, and redundant with lychee-based link checking)
- Update substitute.py and anchors.jq to handle 'items' (mdbook 0.5.x)
in addition to 'sections' (mdbook 0.4.x), as per mdbook 0.5.0
changelog: "Book::sections was renamed to Book::items"
https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#05-migration-guide
Instead we can just seek back in the file - duh. Also this makes use
of the anonymous temp file facility, since that is much safer (no need
window where the we don't have an open file descriptor for it).
With the addition of "delete" method we can no longer rely on
just concatenating "ing" to get the continuous form of the verb.
Also some use-cases actually need a noun instead.
queryRealisationUncached was crashing with an assertion failure when
ca-derivations experimental feature is not enabled, because the SQLite
statements for realisations are only initialized when ca-derivations
is enabled.
Return nullptr (no realisation found) when ca-derivations is disabled,
matching the behavior of other CA-related functions like registerDrvOutput
which check for the feature before proceeding.
They are flagged by Fedora CI checks as a potential security issue.
Use of such raw Right-to-Left control characters in source code is
strongly discouraged
also update release-credits-handle-to-name.json
We can precompute the exact information we need for topo sorting and
store it in `PerhapsNeedToRegister`. Depending on how `topoSort` works,
this is easy a performance improvement or just completely harmless.
Co-Authored-By: Bernardo Meurer Costa <beme@anthropic.com>
Build the inverse of `scratchOuputs` before running topoSort, avoiding
quadratic complexity when determining which outputs reference each
other. This fixes the FIXME comment about building the inverted map up
front.
Inspired by Lix commit 10c04ce84 / Change Id
Ibdd46e7b2e895bfeeebc173046d1297b41998181, but ended up being completely
different code.
Co-Authored-By: Maximilian Bosch <maximilian@mbosch.me>
Co-Authored-By: Bernardo Meurer Costa <beme@anthropic.com>
This was failing under ASAN in https://hydra.nixos.org/build/315173638/nixlog/1.
ASAN uses a bit more stack space and the default max call depth is not enough.
Not sure what's so special about this particular test.
This test is now pointless and the comment is outdated. Also the test fails
on 32 bit systems with:
../nix_api_value_internal.cc:22: Failure
Expected equality of these values:
sizeof(nix::Value)
Which is: 12
sizeof(nix_value)
Which is: 8
It just happeneded to work because Value is 16 bytes and nix_value was also 16 bytes.
Also get rid of a pointless inline in new_nix_value, since it's already static and
inline there does nothing.
When an SSH connection dies during a remote build, MonitorFdHup correctly
detects the disconnect and calls triggerInterrupt(). However, without
ReceiveInterrupts instantiated, no SIGUSR1 is sent to interrupt the
blocking read() syscall. This causes the daemon to hang indefinitely
while holding file locks, blocking subsequent builds.
The fix instantiates ReceiveInterrupts in processConnection(), which
registers a callback to send SIGUSR1 to the current thread when
triggerInterrupt() is called. This allows the blocking read() to return
with EINTR, causing checkInterrupt() to throw and the daemon to exit
cleanly.
This pattern is already used in ThreadPool::doWork() and
SubstitutionGoal for the same purpose.
Clarifies that the first positional argument is always treated as the
installable, even after --. Adds tests to prevent accidental change.
Addresses https://github.com/NixOS/nix/issues/13994
Previously inputFromAttrs() didn't do any validation. inputFromURL()
now calls inputFromAttrs(), so we only need to validate in one place.
Fixes#14655.
It's very weird it doesn't work here, but I don't mind not debugging
this now as I just added this part of the functional test --- it's
already better than it was before.
Add several new functions to the C API:
StorePath operations:
- nix_store_path_hash: Extract the hash part from a store path
- nix_store_create_from_parts: Construct a store path from hash and name
Derivation operations:
- nix_derivation_clone: Clone a derivation
- nix_derivation_to_json: Serialize a derivation to JSON
Store operations:
- nix_store_drv_from_store_path: Load a derivation from a store path
Test the new functions, and improve documentation of some existing
functions to better distinguish them, also.
Co-authored-by: Tristan Ross <tristan.ross@determinate.systems>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Before, had some funny logic with an unnecessary is CA enabled branch,
and erroneous use of the comma operator. Now, take advantage of the new
`Derivation::fillInOutputPaths` to fill in input addresses (and output
path env vars) in a much-more lightweight manner.
Also, fix `nix develop` on fixed-output derivations so that weird things
don't happen when we have that experimental feature enabled.
As a slight behavior change, if the original derivation was
content-addressing this one will be too, but I really don't think that
matters --- if anything, it is a slight improvement for users that have
already opted into content-addressing anyways.
When this function is called as a coroutine (e.g. when it's called by
`copyStorePath()`), the code after `decompressor->finish()` is never
reached because the coroutine is destroyed when the caller reaches the
end of the NAR. So put that code in a `LambdaSink` destructor.
Implements a safe no symlink following primitive operation for opening file descriptors.
This is unix-only for the time being, since windows doesn't really suffer from symlink
races, since they are admin-only.
Tested with enosys --syscall openat2 as well.
I missed this because I assumed all Exprs were recursed into by
bindVars, but ExprLet's ExprAttrs field is not really its own AST node,
so it doesn't get recursed into.
The single-string syntax '>=8.16.0 <8.17.0' only applied the lower
bound, causing curl 8.17.0 to be incorrectly rejected. Split into two
separate version_compare() calls for compatibility with Meson 1.1,
since multi-argument syntax requires Meson 1.8+.
printAmbiguous (used by nix-instantiate --eval and nix-env) had a depth
parameter, but all callers passed INT_MAX, effectively disabling the
limit. The function relied on the C++ stack to eventually overflow,
which could cause uncontrolled SIGSEGV crashes on deeply nested
pre-forced structures.
Now printAmbiguous checks depth against max-call-depth (default 10000)
and throws StackOverflowError with a proper trace, consistent with
other recursive value traversal functions.
The function signature is updated to take EvalState& to access the
settings and throw proper errors. The depth parameter now counts up
from 0 instead of down from INT_MAX.
Non-cyclic structures can be infinitely deep when values are lazily
produced (e.g., `let f = n: { inner = f (n + 1); }; in f 0`). Since f
returns immediately with a thunk, Nix call depth stays at 1, but
Printer::print recurses on the C++ stack when printing.
We check print depth against max-call-depth rather than incrementing
the callDepth counter, because accessing an attribute is not a call.
StackOverflowError is always re-thrown because stack overflow is a
serious condition that expressions should avoid, unlike say `throw`,
which can be part of legitimate expression patterns.
This partially reverts commit bc6b9ce.
This transformation is unsound and thread unsafe. Internal libgit2
structures must *never* be shared between threads. This causes
internal odb corruption with e.g.:
nix flake prefetch-inputs:
error:
… while fetching the input 'github:nixos/nixpkgs/89c2b2330e733d6cdb5eae7b899326930c2c0648?narHash=sha256-Stk9ZYRkGrnnpyJ4eqt9eQtdFWRRIvMxpNRf4sIegnw%3D'
error: adding a file to a tree builder: failed to insert entry: invalid object specified - upload-image.sh
error:
… while fetching the input 'github:NixOS/nixpkgs/a8d610af3f1a5fb71e23e08434d8d61a466fc942?narHash=sha256-v5afmLjn/uyD9EQuPBn7nZuaZVV9r%2BJerayK/4wvdWA%3D'
error: adding a file to a tree builder: failed to insert entry: invalid object specified - outline.nix
double free or corruption (!prev)
Thread 21 "nix" received signal SIGABRT, Aborted.
nix-instantiate on deeply nested structures with recurseForDerivations
(e.g., `let x = { recurseForDerivations = true; more = x; }; in x`)
caused an uncontrolled OS-level stack overflow with no Nix stack trace.
Fix by adding call depth tracking to getDerivations, integrating with
Nix's existing max-call-depth mechanism. Now produces a controlled
"stack overflow; max-call-depth exceeded" error with a proper stack
trace.
Without this dependency, e.g. `meson compile nix-instantiate`
would produce a broken symlink, or the `nix` it points to may be
stale.
With the dependency in place, `meson compile nix-instantiate`
produces a reliable outcome.
On FreeBSD, sysctl(KERN_PROC_PATHNAME) returns a null-terminated
string with pathLen including the terminator. This causes Nix to
fail during manual generation with:
error:
… while calling the 'concatStringsSep' builtin
at /nix/var/nix/builds/nix-63232-402489527/source/doc/manual/generate-settings.nix:99:1:
98| in
99| concatStrings (attrValues (mapAttrs (showSetting prefix) settingsInfo))
| ^
100|
error: input string '/nix/store/gq89cj02b5zs67cbd85vzg5cgsgnd8mj-nix-2.31.2/bin/nix␀'
cannot be represented as Nix string because it contains null bytes
The issue occurs because generate-settings.nix reads the nix binary
path from JSON and evaluates it as a Nix string, which cannot contain
null bytes. Normal C++ string operations don't trigger this since they
handle null-terminated strings correctly.
Strip the null terminator on FreeBSD to match other platforms (Linux
uses /proc/self/exe, macOS uses _NSGetExecutablePath).
Credit: @wahjava (FreeBSD ports and Nixpkgs contributor)
Since the root cause (the lack of backpressure control) has
been fixed in the previous commit we can revert the change from
8ffea0a018 and make the default size much
smaller.
Instead of naively stalling the download thread we can instead stop the transfer.
This allows the other multiplexed connections to continue downloading (and unpacking),
if the result of the download gets piped into a GitFileSystemObjectSink.
Prior art in lix project:
- 4ae6fb5a8f
- 12156d3beb
This patch is very different from the lix one, since we are using a decompression sink
in the middle of the pipeline but the co-authored-by is there since I was motivated to
implement this by looking at the lix side of things.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Similar to the deepSeq fix, toJSON on deeply nested structures caused
an uncontrolled OS-level stack overflow.
Fix by adding call depth tracking to printValueAsJSON.
When deepSeq encounters an error while evaluating a list element, the
error trace now includes the list index, making it easier to locate
the problematic element.
builtins.deepSeq on deeply nested structures (e.g., a linked list with
100,000 elements) caused an uncontrolled OS-level stack overflow with
no Nix stack trace.
Fix by adding call depth tracking to forceValueDeep, integrating with
Nix's existing max-call-depth mechanism. Now produces a controlled
"stack overflow; max-call-depth exceeded" error with a proper stack
trace.
Closes: https://github.com/NixOS/nix/issues/7816
this is a painful change. we should really add EvalState or EvalMemory
as an argument to various functions as we need it, but because we want
to preserve the stablity API, we hack it in as a field of nix_value.
The use of sourceToSink is an unnecessary serialization bottleneck.
While we are at it, fix the copyRecursive implementation to actually copy
the whole directory. It wasn't used for anything prior, but now it has a use
and accompanying tests for flake clone.
Also do a better JSON and testing for deep and shallow NAR listings.
As documented, this for file system objects themselves, since
`MemorySourceAccessor` is an implementation detail.
`listNar` did the not-so-pretty thing of going straight to JSON. Now it
uses `MemorySourceAccessor::File`, or rather variations of it, to go to
a C++ data type first, and only JSON second.
To accomplish this we add some type parameters to the `File` data type.
Actually, we need to do two rounds of this, because shallow NAR
listings. There is `FileT` and `DirectoryT` accordingly.
This matches the "NAR Listing" JSON format, and also helps distinguish
from regular file contents.
Why we want to match that will become clear in the next comments, when
we will in fact use (variations of) this data type for NAR listings.
`nix derivation add`, and its C API counterpart, now works a bit closer
to `builtins.derivation` in that they don't require the user to fill-in
input addressed paths correctly ahead of time.
The logic for this is carefully deduplicated, between all 3 entry
points, and also between the existing `checkInvariants` function. There
are some more functional tests, and there are also many more unit tests.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: edef <edef@edef.eu>
These do a read/write test in the middles of some computation. They are
an imperative way to test intermediate values rather than functionally
testing end outputs.
We don't use the various set<string_view>s that we construct,
and all we really care about is ensuring that all outputs are
of a single, consistent type.
This is necessary to ban symlink following. It can be considered
a defense in depth against issues similar to CVE-2024-45593. By
slightly changing the API in a follow-up commit we will be able
to mitigate the symlink following issue for good.
This is an example of "Parse, don't validate" principle [1].
Before, we had a number of `StringSet`s in `DerivationOptions` that
were not *actually* allowed to be arbitrary sets of strings. Instead,
each set member had to be one of:
- a store path
- a CA "downstream placeholder"
- an output name
Only later, in the code that checks outputs, would these strings be
further parsed to match these cases. (Actually, only 2 by that point,
because the placeholders must be rewritten away by then.)
Now, we fully parse everything up front, and have an "honest" data type
that reflects these invariants:
- store paths are parsed, stored as (opaque) deriving paths
- CA "downstream placeholders" are rewritten to the output deriving
paths they denote
- output names are the only arbitrary strings left
Since the first two cases both become deriving paths, that leaves us
with a `std::variant<SingleDerivedPath, String>` data type, which we use
in our sets instead.
Getting rid of placeholders is especially nice because we are replacing
them with something much more internally-structured / transparent.
[1]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Now the error message looks something like:
error:
… during upload of 'file:///tmp/storeabc/4yxrw9flcvca7f3fs7c5igl2ica39zaw.narinfo'
error: blah blah
Also makes fail and failEx themselves noexcept, since all the operations they
do are noexcept and we don't want exceptions escaping from them.
The indentation level of the code is already high enough. We can just
wrap the whole function in a try/catch and mark it noexcept.
Partially cherry-picked from https://gerrit.lix.systems/c/lix/+/2133
Co-authored-by: eldritch horrors <pennae@lix.systems>
Makes the cross-x86_64-w64-mingw32 devshell slightly less
broken. It still needs a bit of massaging to function, but
that's much less cumbersome now that the generic machinery
with genericClosure that evaluates drvPath doesn't barf on
unavailable packages.
The test was failing because nix path-info --json now returns narHash as
a structured dictionary {"algorithm": "sha256", "format": "base64",
"hash": "..."} instead of an SRI string "sha256-...".
This change was introduced in commit 5e7ee808d. The functional test
path-info.sh was updated at that time, but this NixOS test was missed.
The fix converts the dictionary format to SRI format inline:
tarball_hash_sri = f"{narHash_obj['algorithm']}-{narHash_obj['hash']}"
Several bugs to squash:
- Apparently DELETE is an already used macro with Win32. We can avoid it
by using Camel case instead (slightly hacky but also fits the naming
convention better)
- Gets rid of the raw usage of isatty. Added an isTTY impl to abstract over
the raw API.
Replaces the usage of createAtRoot, which goes as far up the
directory tree as possible with rooted variant makeFSSourceAccessor.
The changes in this patch should be safe wrt to not asserting on relative
paths. Arguments passed to makeFSSourceAccessor here should already be using
absolute paths.
without this, there is no way to swap them out for structures using a
different allocator. This should be reverted as part of redesiging
ExprAttrs to use an ExprAttrsBuilder
And also render the docs nicely.
I would like to use a markdown AST for this, but to avoid new deps
(lowdown's AST doesn't suffice) I am just doing crude string
manipulations for now.
Since 918c1a9e58 configurePhase variable points to cmakeConfigurePhase
and runPhase configurePhase does the wrong thing.
configurePhase function on the other hand still worked correctly.
Move StorePath and Derivation declarations to their own headers in a
backwards compatible way:
- Created nix_api_store/store_path.h for StorePath operations
- Created nix_api_store/derivation.h for Derivation operations
- Main nix_api_store.h includes both headers for backwards compatibility
This reorganization improves modularity and hopefully makes the API
easier to navigate.
I found this because of a test failure on cygwin in
nix_api_expr_test.nix_eval_state_lookup_path:
'std::filesystem::__cxx11::filesystem_error'
what(): filesystem error: cannot remove all: Device or resource busy
[...]
[.../my_state/db/db.sqlite]
LocalState was never getting destroyed due to a reference leak. These
_free functions use an 'operator delete' which doesn't call the
destructor for the type.
Fixes: 309d55807c
Currently, --gtest_filter=nix_api_store_test.nix_eval_state_lookup_path
will result in:
terminating due to unexpected unrecoverable internal error: Assertion
'gcInitialised' failed in void nix::assertGCInitialized() at
../src/libexpr/eval-gc.cc:138
Changing the test fixture to _exr_test causes GC to be initialised.
Typically PosixSourceAccessor can be used from multiple threads,
but mtime is not updated atomically (i.e. with compare_exchange_weak),
so mtime gets raced. It's only needed in dumpPathAndGetMtime and mtime
tracking can be gated behind that.
Also start using getLastModified interface instead of dynamic casts.
This makes the output easier to compare with the new machine-generated
lists in #9732.
The hand-curated order did have the advantage of putting more important
attributes at the top, but I don't think it is worth preserving that
when `std::map` is so much easier to work with. The right solution to
leading the reader to the more important attributes is to call them out
in the intro texts.
This makes the proto serializer characterisation test data be
accompanied by JSON data.
This is arguably useful for a reasons:
- The JSON data is human-readable while the binary data is not, so it
provides some indication of what the test data means beyond the C++
literals.
- The JSON data is language-agnostic, and so can be used to quickly rig
up tests for implementation in other languages, without having source
code literals at all (just go back and forth between the JSON and the
binary).
- Even though we have no concrete plans to place the binary protocol 1-1
or with JSON, it is still nice to ensure that the JSON serializers and
binary protocols have (near) equal coverage over data types, to help
ensure we didn't forget a JSON (de)serializer.
Make instances for them that share code with `nix path-info`, but do a
slightly different format without store paths containing store dirs
(matching the other latest JSON formats).
Progress on #13570.
If we depend on the store dir, our JSON serializers/deserializers take
extra arguements, and that interfaces with the likes of various
frameworks for associating these with types (e.g. nlohmann in C++, Serde
in Rust, and Aeson in Haskell).
For now, `nix path-info` still uses the previous format, with store
dirs. We may yet decide to "rip of the band-aid", and just switch it
over, but that is left as a future PR.
The variant has on the left-hand side the topologically sorted vector
and the right-hand side is a pair showing the path and its parent that
represent a cycle in the graph making the sort impossible.
This change prepares for enhanced cycle error messages that can provide
more context about the cycle. The variant approach allows callers to
handle cycles more flexibly, enabling better error reporting that shows
the full cycle path and which files are involved.
Adapted from Lix commit f7871fcb5.
Change-Id: I70a987f470437df8beb3b1cc203ff88701d0aa1b
Co-Authored-By: Maximilian Bosch <maximilian@mbosch.me>
Add support for configuring S3 storage class via the storage-class
parameter for S3BinaryCacheStore. This allows users to optimize costs
by selecting appropriate storage tiers (STANDARD, GLACIER,
INTELLIGENT_TIERING, etc.) based on access patterns.
The storage class is applied via the x-amz-storage-class header for
both regular PUT uploads and multipart upload initiation.
Replace the null-terminated C-style strings in Value with hybrid C /
Pascal strings, where the length is stored in the allocation before the
data, and there is still a null byte at the end for the sake of C
interopt.
Co-Authored-By: Taeer Bar-Yam <taeer@bar-yam.me>
Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
These steps are done (originally in order, but I squashed it as the end
result is still pretty small, and the churn in the code comments was a
bit annoying to keep straight).
1. Create proper struct type for string contexts on the heap
This will make it easier to change this type in the future.
2. Make `Value::StringWithContext` iterable
This make some for loops a lot more terse.
3. Encapsulate `Value::StringWithContext::Context::elems`
It turns out the iterators we just exposed are sufficient.
4. Make `StringWithContext::Context` length-prefixed instead
Rather than having a null pointer at the end, have a `size_t` at the
beginning. This is the exact same size (note that null pointer is
longer than null byte) and thus takes no more space!
Also, see the new TODO on naming. The thing we already so-named is a
builder type for string contexts, not the on-heap type. The
`fromBuilder` static method reflects what the names ought to be too.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Otherwise PosTable grows indefinitely for each reload. Since
the total input size is limited to 4GB (uint32_t for byte offset PosIdx)
it can get exhausted pretty. This ensures that we don't waste memory
on reloads as well.
Update all channel URLs from https://nixos.org/channels/ to
https://channels.nixos.org/ to use the more reliable subdomain.
The nixos.org domain apex lacks IPv6 support due to DNS hoster
limitations. Using the subdomain allows better CDN distribution
and improved reliability.
Updated files:
- Installation scripts (multi-user and tarball installers)
- Channel URL resolution in eval-settings.cc
- Documentation and examples
- Docker image default channel URL
- Release notes (added note about URL change)
Fixes#14517
I've run into this quite a few times when working with characterization test
infra. It would print an invalid command:
_NIX_TEST_ACCEPT=1 meson test main/lang
Which you'd then proceed to run and it would fail. This commit makes it
be honest about the command you need to run:
_NIX_TEST_ACCEPT=1 meson test --suite main lang
I would run `nix flake show` on a flake than hit:
===
├───ihaskell: package 'ihaskell-wrapper'
├───ihaskell-96: package 'ihaskell-wrapper'
├───ihaskell-96-dev: package 'ghc-shell-for-ihaskell-0.10.4.0'
error: expected a derivation
===
and it is not obvious what package is the culprit here since nix stops
rightaway.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
boost::concurrent_flat_map (used in libutil and libstore) includes the
C++17 <execution> header. GCC's libstdc++ implements parallel algorithms
using Intel TBB as the backend, which creates a link-time dependency on
libtbb even though we don't actually use any parallel algorithms.
Disable the TBB backend for libstdc++ by setting
_GLIBCXX_USE_TBB_PAR_BACKEND=0. This makes parallel algorithms fall back
to serial execution, which is acceptable since we don't use them anyway.
This only affects libstdc++ (GCC's standard library); other standard
libraries like libc++ (LLVM) are unaffected.
`allowedReferences` and friends can, in addition to supporting store
paths (and placeholders, but because those will be rewritten to store
paths), they also support to refering to other outputs in the derivation
by name.
We update the tests in order to cover for that.
(While we are at it, also introduce some scratch variables for paths and
placeholders to make the C++ literalsf for this test more concise.)
Since we haven't released v2 yet (2.32 has v1) we can just update this
in-place and avoid version churn.
Note that as a nice side effect of using the standard `Hash` JSON impl,
we don't neeed this `hashFormat` parameter anymore.
It turns out this code path is only used for unit tests (to ensure our
JSON formats are possible to parse by other code, elsewhere). No
user-facing functionality consumes this format.
Therefore, let's drop the old version parsing support.
We now have functional tests for these. The unit tests added negligible
value while imposing a much higher maintenance cost.
The maintenance cost is high:
- No automatic accept option
- They broke 5+ times during this session due to implementation changes (trace count, ordering)
- They require understanding ANSI escape codes, Uncolored() wrappers, trace reversal
- They test empty traces HintFmt("") from withTrace(pos, "") - pure implementation detail
- They're fragile: adding any trace anywhere breaks the exact count assertions
The additional value over functional tests is minimal:
- Functional tests already verify the error message
- Functional tests already show trace order and content (as users see it, helps review)
- Unit tests verify "exactly 3 traces, not 2 or 4" - but users don't count traces
- Unit tests verify empty traces exist - but users never see them
The white-box testing catches the wrong things:
- It catches "you added helpful context" as a failure
- It doesn't catch "the context is confusing" (which functional tests would show)
- It enforces implementation details that should be allowed to evolve
Show which element(s) are involved at each error point:
- When an element is missing the "key" attribute, show the element
- When an element is not an attribute set, show the element
- When comparing keys fails, show both elements being compared
- When calling operator fails, show which element was being processed
This provides concrete context using ValuePrinter with errorPrintOptions.
Note: errorPrintOptions uses maxDepth=10 by default, which may print
quite deeply nested structures in error messages. This could potentially
be overwhelming, but follows the existing default for error contexts.
The old string format is a holdover from the pre JSON days. It is not
friendly to users who need to get the information out of it.
Also introduce the sort of versioning we have for derivation for this
format too.
- Use canonical content address JSON format for floating content
addressed derivation outputs
This keeps it more consistent.
- Reorganize inputs into nested structure (`inputs.srcs` and
`inputs.drvs`)
This will allow for an easier to use, but less compact, alternative
where `srcs` is just a list of derived paths.
It also allows for other experiments for derivations with a different
input structure, as I suspect will be needed for secure build traces.
This was already dropped in `inputFromURL()`, but not in
`inputFromAttrs()`. Now it's done in `fixGitURL()`, which is used by
both.
In principle, `git+` shouldn't be used in the `url` attribute, since
we already know that it's a Git URL. But since it currently works, we
don't want to break it.
Fixes#14429.
Have one to that instead of one to `Derivation`. `DerivationBuilder`
doesn't need `inputDrvs`, so `BasicDerivation` suffices.
(In fact, it doesn't need `inputSrcs` either, but we don't yet hve a
type to exclude that.)
We were calling git with `--quiet` in order not to mess up Nix's
progress bar. However, `runProgram()` already suspends the progress
bar (since git may be interactive) so that's no longer an issue. So we
can just run with `--progress` instead.
Fix#14480
This method is not well-defined for arbitrary stores, which do not have
a notion of a "real path" -- it is only well-defined for local file
systems stores, which do have exactly that notion, and so it is moved to
that sub-interface instead.
Some call-sites had to be fixed up for this, but in all cases the
changes are positive. Using `getFSSourceAccessor` allows for more other
stores to work properly. `nix-channel` was straight-up wrong in the case
of redirected local stores. And the building logic with remote building
and a non-local store is also fixed, properly gating some deletions on
store type.
Co-authored-by: Robert Hensing <robert@roberthensing.nl>
The assumption that no unknown paths can be returned is incorrect. It
can happen if a derivation has outputs that are substitutable, but
that have references that cannot be substituted (i.e. an incomplete
closure in the binary cache). This can easily happen with
magic-nix-cache.
... via _NIX_TEST_NO_ENVIRONMENT_WARNINGS
This environment variable suppresses warnings that depend on the test
environment (such as ulimit warnings in builds on systems with lower
limits, which may well succeed if it weren't for the warning).
This prevents non-deterministic test failures in golden/characterization
tests.
Alternative considered: filtering stderr in test scripts, but that approach
is fragile with binary test output, and potentially multiple call sites.
Show the actual attempted stack size value (capped at hard limit)
separately from the desired value, making it clearer what's happening
when the hard limit is lower than requested.
Previously, only shared memory segments were cleaned up.
This could lead to leaked message queues and semaphore sets when builds use System V IPC, exhausting kernel IPC limits over time.
This commit extends the cleanup to all three System V IPC types:
1. Shared memory segments
2. Message queues
3. Semaphores
Additionally, we stop removing IPC objects during iteration, as it could corrupt the kernel's iterator state and cause some objects to be skipped. The new implementation uses a two-pass approach where we list first and then remove them in a separate pass.
The IPC IDs are now extracted during iteration using actual system calls (shmget, msgget, semget) rather than being looked up later, ensuring the objects exist when we capture their IDs.
In Linux, IPC objects are automatically cleaned up when the IPC namespace is destroyed.
On Darwin, since there are no IPC namespaces, the IPC objects may sometimes persist after the build user's processes are killed.
This patch modifies the cleanup logic to use sysctl calls to identify and remove left over shm segments associated with the build user.
Fixes: #12548
For repos with a lot of non-linearity in the commit graph (like
Nixpkgs), this speeds up getting the revcount a lot, e.g. `nix flake
metadata /path/to/nixpkgs?rev=9dc7035bbee85ffc740d893e02cb64460f11989f` went
from 9.1s to 3.7s.
Warning:
```
[39/483] Generating src/kaitai-struct-checks/kaitai-generated-sources with a custom command
../src/kaitai-struct-checks/nar.ksy: /types/padded_str/seq/1/encoding:
warning: use canonical encoding name `ASCII` instead of `ascii` (see https://doc.kaitai.io/ksy_style_guide.html#encoding-name)
```
This will allow us to more accurately test dropping support for
dependent realisations, by separating the tests that should not change
from the tests that should.
I do that change in PR #14247, but even if for some reasons we don't end
up doing this soon, I think it is still good to separate the test data
this way so we have the option of doing that at some point.
Progress on #13405, which asks for an explicit characterisation of the
equivalence relation like the one given here.
Also progress on #11895, because we're using the term "build trace
entry" instead of "realisation".
Mention #9259, a future work item.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
It is better to avoid null termination for performance and memory
safety, wherever possible.
These are good cleanups extracted from the Pascal String work that we
can land by themselves first, shrinking the diff in that PR.
Co-Authored-By: Aspen Smith <root@gws.fyi>
Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
Add three configuration settings to `S3BinaryCacheStoreConfig` to control
multipart upload behavior:
- `bool multipart-upload` (default `false`): Enable/disable multipart uploads
- `uint64_t multipart-chunk-size` (default 5 MiB): Size of each upload part
- `uint64_t multipart-threshold` (default 100 MiB): Minimum file size for multipart
The feature is disabled by default.
The rule of silence can be a little surprising. As a compromise to
changing the default behavior, this adds printing a success message in
verbose mode, where we don't really have a reason to be silent about
our success.
Refactor `ExprConcatStrings::eval` by inlining two only-called-once
closures into the call-site, so that the code is easier to reason about
locally (especially since the variables that were closed over were
mutated all over the place within this function).
Also use curly braces with each branch for consistency in the the
resulting code.
This is a pure refactor, but also arguably causes us to depend less on
the optimizer; now, we don't have to make sure that this closure is
inlined.
3a3c062982 introduced a buffer overflow for the
case when there are more than 65535 formal arguments. It is a perfectly reasonable
limitation, but we *must* not crash, corrupt memory or otherwise crash the process.
Add a test for the graceful behavior and switch to using an explicit uninitialized_copy_n
to further guard against buffer overflows.
Stop delegating to `HttpBinaryCacheStore::upsertFile` and instead
handle compression in the S3 store's `upsertFile` override, then call
our own `upload()` method. This separation is necessary for future
multipart upload support.
Introduce protected `upload` method overloads in `HttpBinaryCacheStore`
that handle the actual upload after compression has been applied. This
separates compression concerns (in `upsertFile`) from upload mechanics
(in `upload`).
Two overloads are provided:
1. `upload(path, RestartableSource &, sizeHint, mimeType, contentEncoding)`
2. `upload(path, CompressedSource &, mimeType)`
Introduce a `CompressedSource` class in libutil's `serialise.hh` that
compresses a `RestartableSource` and owns the compressed data. This is a
general-purpose utility that can be used anywhere compressed data needs
to be treated as a source.
We were getting this flex lexer warning during build:
```
../src/libexpr/lexer.l:333: warning, -s option given but default rule can be matched
```
The lexer uses `%option nodefault` but the `PATH_START` state only had
rules for specific patterns (`PATH_SEG` and `HPATH_START`) without a
catch-all rule to handle unexpected input.
Added a catch-all rule with `unreachable()`. This code path should never
be reached in normal operation since `PATH_START` is only entered after
matching `PATH_SEG` or `HPATH_START`, and we immediately rewind to
re-parse those same patterns. The catch-all exists solely to satisfy
flex's `%option nodefault` requirement.
Make uploads run in constant memory. Also change the callbacks to be
noexcept, since we really don't want to be unwinding the stack in the
curl thread. That will definitely corrupt that stack and make nix/curl
crash in very bad ways.
Fix a race condition where interrupting a download (via Ctrl-C) during a
retry attempt could cause a crash. When `enqueueItem()` throws because the
download thread is shutting down, the exception would propagate without
setting `done=true`, causing the `TransferItem` destructor to invoke the
callback a second time.
This triggered an assertion failure in `Callback::rethrow()` with:
`Assertion '!prev' failed` and the error message `cannot enqueue download
request because the download thread is shutting down`.
The fix catches the exception from `enqueueItem()` and calls `fail()` to
properly complete the transfer, ensuring the callback is invoked exactly
once.
Some zsh setups (including mine) do not load the
completion if `#compdef` is not on the first line.
So we move the `# shellcheck` comment to the
second line to avoid this issue.
This continues the work for formalizing our current JSON docs. Note that
in the process, a few bugs were caught:
- `closureSize` was repeated twice, forgot `closureDownloadSize`
- `file*` fields should be `download*`. They are in fact called that in
the line-oriented `.narinfo` file, but were renamed in the JSON
format.
We immediately use this in the JSON schemas for Derivation and Deriving
Path, but we cannot yet use it in Store Object Info because those paths
*do* include the store dir currently.
- Uses the more explicit `@ingroup` most of the time, to avoid problems
with nested groups, and to make group membership more explicit.
The division into headers is not great for documentation purposes,
so this helps.
- More attention for memory management details
- Various other improvements to doc comments
Per #7591, the `nix-store --gc --print-dead` command does not provide
any feedback about the amount of disk space that is used by dead store
paths. It looks like this has been the case since 7ab68961e (* Garbage
collector: added an option `--use-atime' to delete paths in...,
2008-09-17).
Update the nix-store documentation to remove the claim that this is
function that `nix-store --gc --print-dead` performs.
Implement `uploadPart()` for uploading individual parts in S3 multipart
uploads:
- Constructs URL with `?partNumber=N&uploadId=ID` query parameters
- Uploads chunk data with `application/octet-stream` mime type
- Extracts and returns `ETag` from response
Add concurrency group configuration to the CI workflow to automatically
cancel outdated runs when a PR receives new commits or is force-pushed.
This prevents wasting CI resources on superseded code.
This is a good default (the methods that allow for an arbitrary choice
of source accessor are generally preferable both to implement and to
use). And it also pays its way by allowing us to delete *both* the
`DummyStore` and `LocalStore` implementations.
Introduces `scanForReferencesDeep` to provide per-file granularity when
scanning for store path references, enabling better diagnostics for
cycle detection and `nix why-depends --precise`.
Implement `abortMultipartUpload()` for cleaning up incomplete multipart
uploads on error:
- Constructs URL with `?uploadId=ID` query parameter
- Issues `DELETE` request to abort the multipart upload
With #14314, in some places in the parser we started using C++ objects
directly rather than pointers. In those places lines like `$$ = $1` now
imply a copy when we don't need one. This commit changes those to `$$ =
std::move($1)` to avoid those copies.
Previously it used the `ThreadPool` default,
i.e. `std::thread::hardware_concurrency()`. But copying signatures is
not primarily CPU-bound so it makes more sense to use the
`http-connections` setting (since we're typically copying from/to a
binary cache).
The `showBytes()` function was redundant with `renderSize()` as the
latter automatically selects the appropriate unit (KiB, MiB, GiB, etc.)
based on the value, whereas `showBytes()` always formatted as MiB
regardless of size.
Co-authored-by: Bernardo Meurer Costa <beme@anthropic.com>
Instead of iterating over the newly built bindings we can
do a cheaper set_intersection to count duplicates or fall back
to a per-element binary search over the "base" bindings.
This speeds up `hello` evaluation by around 10ms (0.196s -> 0.187s) and
`nixos.closures.ec2.x86_64-linux` by 140ms (2.744s -> 2.609s).
This addresses a somewhat steep performance regression from 82315c3807
that reduced memory requirements of attribute set merges. With this patch
we get back around to 2.31 level of eval performance while keeping the memory
usage optimization.
Also document the optimization a bit more.
In particular
- Remove `get`, it is redundant with `valueAt` and the `get` in
`util.hh`.
- Remove `nullableValueAt`. It is morally just the function composition
`getNullable . valueAt`, not an orthogonal combinator like the others.
- `optionalValueAt` return a pointer, not `std::optional`. This also
expresses optionality, but without creating a needless copy. This
brings it in line with the other combinators which also return
references.
- Delete `valueAt` and `optionalValueAt` taking the map by value, as we
did for `get` in 408c09a120, which
prevents bugs / unnecessary copies.
`adl_serializer<DerivationOptions::OutputChecks>::from_json` was the one
use of `getNullable`. I give it a little static function for the
ultimate creation of a `std::optional` it does need to do (after
switching it to using `getNullable . valueAt`. That could go in
`json-utils.hh` eventually, but I didn't bother for now since only one
things needs it.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
S3 buckets support object versioning to prevent unexpected changes,
but Nix previously lacked the ability to fetch specific versions of
S3 objects. This adds support for a `versionId` query parameter in S3
URLs, enabling users to pin to specific object versions:
```
s3://bucket/key?region=us-east-1&versionId=abc123
```
This has already been implemented in 1e709554d5
as a side-effect of mounting the accessors in storeFS. Let's test this so it
doesn't regress.
(cherry-picked from https://github.com/NixOS/nix/pull/12915)
Move HttpBinaryCacheStore class from .cc file to header to enable
inheritance by S3BinaryCacheStore. Create S3BinaryCacheStore class that
overrides upsertFile() to implement multipart upload logic.
Add a sizeHint parameter to BinaryCacheStore::upsertFile() to enable
size-based upload decisions in implementations. This lays the groundwork
for reintroducing S3 multipart upload support.
Add support for HTTP DELETE requests to FileTransfer infrastructure:
This enables S3 multipart upload abort functionality via DELETE requests
to S3 endpoints.
This reverts commit 90d1ff4805.
The initial issue with EPIPE was solved in 9f680874c5.
Now this patch does move bad than good by eating up boost::io::format_error that are
bugs.
addToStore(): Don't parse the NAR
* StringSource: Implement skip()
This is slightly faster than doing a read() into a buffer just to
discard the data.
* LocalStore::addToStore(): Skip unnecessary NARs rather than parsing them
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
A few changes had cropped up with `_NIX_TEST_ACCEPT=1`:
1. Blake hashing test JSON had a different indentation
2. Store URI had improper non-quoted spaces
(1) was is just fixed, as we trust nlohmann JSON to parse JSON
correctly, regardless of whitespace.
For (2), the existing URL was made a read-only test, since we very much
wish to continue parsing such invalid URLs directly. And then the
original read/write test was updated to properly percent-encode the
space, as the normal form should be.
Since 2.32, nix now needs boost 1.87 or later to build,
due to using unordered::concurrent_flat_map try_emplace_and_cvisit
../src/libexpr/eval.cc: In member function ‘void nix::EvalState::evalFile(const nix::SourcePath&, nix::Value&, bool)’:
../src/libexpr/eval.cc:1096:20: error: ‘class boost::unordered::concurrent_flat_map<nix::SourcePath, nix::Value*, std::hash<nix::SourcePath>, std::equal_to<nix::SourcePath>, traceable_allocator<std::pair<const nix::SourcePath, nix::Value*> > >’ has no member named ‘try_emplace_and_cvisit’; did you mean ‘try_emplace_or_cvisit’?
1096 | fileEvalCache->try_emplace_and_cvisit(
| ^~~~~~~~~~~~~~~~~~~~~~
| try_emplace_or_cvisit
See 834580b539
The s3:ListBucket permission is required for read operations on S3
binary caches, not just for writes. Without this permission, users get
"Access Denied" errors when running nix-build.
Extract the path-based compression method determination logic into a
protected method that returns std::optional<std::string>. This allows
subclasses to reuse the logic and makes the semantics clearer (nullopt
means no compression, not empty string).
This prepares for S3BinaryCacheStore to apply the same compression
rules when implementing multipart uploads.
Fix POST requests with data to use the correct curl option for specifying
body size. Previously used CURLOPT_INFILESIZE_LARGE for both POST and PUT,
but POST requires CURLOPT_POSTFIELDSIZE_LARGE.
This caused POST request bodies to not be sent correctly, manifesting as
S3 multipart CompleteMultipartUpload requests failing with "You must
specify at least one part" even though the XML body contained valid parts.
When Nix's SQLite narinfo cache indicates a NAR exists, but the NAR
has been garbage collected from the binary cache, Nix displays error
messages even though the operation succeeds via fallback. This is
misleading because the cached narinfo is simply outdated.
This changes SubstituteGone exceptions to produce warnings instead of
errors, accurately reflecting that this is an expected cache coherency
issue, not an actual failure.
Fixes#11411🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
At least one user has probably used `file+git://` when they mean `git+file://`, maybe thinking of it as "a file-based git repository". This adds a specific error message to hint at the correct URL scheme format and may save some users from resorting to `path:///` and copying an entire repo.
Adds a comprehensive test to verify that `nix-prefetch-url` correctly
handles S3 URLs with query parameters (e.g., custom endpoints and regions).
Previously, nix-prefetch-url would fail with "invalid store
path" errors when given S3 URLs with query parameters like
`?endpoint=http://server:9000®ion=eu-west-1`, because it incorrectly
extracted the filename from the query parameters instead of the path.
Previously, `prefetchFile()` used `baseNameOf()` directly on the URL string
to extract the filename. This caused issues with URLs containing query
parameters that include slashes, such as S3 URLs with custom endpoints:
```
s3://bucket/file.txt?endpoint=http://server:9000
```
The `baseNameOf()` function naively searches for the rightmost `/` in the
entire string, which would find the `/` in `http://server:9000` and extract
`server:9000®ion=...` as the filename. This resulted in invalid store
path names containing illegal characters like `:`.
This commit fixes the issue by:
1. Adding a `VerbatimURL::lastPathSegment()` method that extracts the last
non-empty path segment from a URL, using `pathSegments(true)` to filter
empty segments
2. Changing `prefetchFile()` to accept `const VerbatimURL &` and use the new
`lastPathSegment()` method instead of manual path parsing
3. Adding early validation with `checkName()` to fail quickly on invalid
filenames
4. Maintains backward compatibility by falling back to `baseNameOf()` for
unparsable `VerbatimURL`s
Old code would do very much incorrect reentrancy crimes (trying to do an
erase inside the emplace callback). This would fail miserably with an assertion
in Boost:
terminating due to unexpected unrecoverable internal error: Assertion '(!find(px))&&("reentrancy not allowed")' failed in boost::unordered::detail::foa::entry_trace::entry_trace(const void *) at include/boost/unordered/detail/foa/reentrancy_check.hpp:33
This is trivially reproduced by using any S3 URL with a non-empty profile:
nix-prefetch-url "s3://happy/crash?profile=default"
The previous message was vague about what "deprecated" meant and why
unlocked inputs with NAR hashes "may not be reproducible". It also
used "verifiable" which was confusing.
The new message makes it clear that the NAR hash provides verification
(is checked by NAR hash) and explicitly states the failure modes:
garbage collection and sharing.
Add `test_public_bucket_operations` to validate that store operations
work correctly on public S3 buckets without requiring credentials.
Tests nix store info and nix copy operations.
Add cleanup of client store in the finally block of setup_s3 decorator.
Uses `nix store delete --ignore-liveness` to properly handle GC roots
and only attempts deletion if the path exists.
This slightly improves the logs situation by including the region/profile/endpoint
in the logs when S3 store references get printed. Instead of:
copying path '/nix/store/lxnp9cs4cfh2g9r2bs4z7gwwz9kdj2r9-test-package-c' to 's3://bucketname'...
This now includes:
copying path '/nix/store/lxnp9cs4cfh2g9r2bs4z7gwwz9kdj2r9-test-package-c' to 's3://bucketname?endpoint=http://server:9000®ion=eu-west-1'...
Nix attempts to set the stack size to 64 MB during initialization, which is
required for the repl tests to run successfully. Skip the tests on systems
where the hard stack limit is less than this value rather than failing.
We now unconditionally compile support for s3:// URLs and stores
without authentication. The whole curl version check can be greatly
simplified by the previous commit, which bumps the minimum required curl
version.
This version has been released a long time ago in 2021 and it's doubtful
that anybody actually uses it still, since it's full of vulnerabilities [^]
[^]: https://curl.se/docs/vuln-7.75.0.html
I realized that we can actually do this thing, even though it is not
what nlohmann expects at all, because the extra parameter has a default
argument so nlohmann doesn't need to care. Sneaky!
Since 3c610df550 this resulted in `getting status of`
errors on paths inside the chroot if a path was already valid. Careful inspection
of the logic shows that if buildMode != bmCheck actualPath gets reassigned to
store.toRealPath(finalDestPath). The only branch that cares about actualPath is
the buildMode == bmCheck case, which doesn't lead to optimisePath anyway.
Instead of the cryptic:
> error: Failed to resolve AWS credentials: error code 6153`
We now get more legible:
> error: AWS authentication error: 'Valid credentials could not be sourced by the IMDS provider' (6153)
This makes it so we don't need to rely on global variables and hacky destructors to
clean up another global variable. Just putting it in the correct order in the class
is more than enough.
This partially reverts commit 5e46df973f,
partially reversing changes made to
8c789db05b.
We do this because Hydra, while using the newer version of the protocol,
still uses this command, even though Nix (as a client) doesn't use it.
On that basis, we don't want to remove it (or consider it only part of
the older versions of the protocol) until Hydra no longer uses the
Legacy SSH Protocol.
This is necessary to fix nix-everything-llvm.
The problem here is that nix-cli is taken from the previous
stage that is built with libstdc++, but this derivation builds
plugins with libc++ and the plugin load fails miserably.
Realisations are conceptually key-value pairs, mapping `DrvOutputs` (the
key) to information about that derivation output.
This separate the value type, which will be useful in maps, etc., where
we don't want to denormalize by including the key twice.
This matches similar changes for existing types:
| keyed | unkeyed |
|--------------------|------------------------|
| `ValidPathInfo` | `UnkeyedValidPathInfo` |
| `KeyedBuildResult` | `BuildResult` |
| `Realisation` | `UnkeyedRealisation` |
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Turns out there's a much better API for this that doesn't have the
footguns of the previous method.
isLegalRefName is somewhat of a misnomer, since it's mainly used to
validate user inputs that can be either references, branch names,
psedorefs or tags.
The macro now accurately reflects its purpose: gating only AWS
authentication code, not all S3 functionality. S3 URL parsing, store
configuration, and public bucket access work regardless of this flag.
This rename clarifies that:
- S3 support is always available (URL parsing, store registration)
- Only AWS credential resolution requires the flag
- The flag controls AWS CRT SDK dependency, not S3 protocol support
Move S3 URL parsing, store configuration, and public bucket support
outside of NIX_WITH_S3_SUPPORT guards. Only AWS credential resolution
remains gated, allowing builds with withAWS = false to:
- Parse s3:// URLs
- Register S3 store types
- Access public S3 buckets (via HTTPS conversion)
- Use S3-compatible services without authentication
The setupForS3() function now always performs URL conversion, with
authentication code conditionally compiled based on NIX_WITH_S3_SUPPORT.
The aws-creds.cc file (only code using AWS CRT SDK) is now conditionally
compiled by meson.
This commit replaces the AWS C++ SDK with a lighter curl-based approach
for S3 binary cache operations.
- Removed dependency on the heavy aws-cpp-sdk-s3 and aws-cpp-sdk-transfer
- Added lightweight aws-crt-cpp for credential resolution only
- Leverages curl's native AWS SigV4 authentication (requires curl >= 7.75.0)
- S3BinaryCacheStore now delegates to HttpBinaryCacheStore
- Function s3ToHttpsUrl converts ParsedS3URL to ParsedURL
- Multipart uploads are no longer supported (may be reimplemented later)
- Build now requires curl >= 7.75.0 for AWS SigV4 support
Fixes: #13084, #12671, #11748, #12403, #5947
This forces the code to go through proper abstractions instead of the raw filesystem
API.
This issue is evident from this reproducer:
nix eval --expr 'builtins.fetchurl { url = "https://example.com"; sha256 = ""; }' --json --eval-store "dummy://?read-only=false"
error:
… while calling the 'fetchurl' builtin
at «string»:1:1:
1| builtins.fetchurl { url = "https://example.com"; sha256 = ""; }
| ^
error: opening file '/nix/store/r4f87yrl98f2m6v9z8ai2rbg4qwlcakq-example.com': No such file or directory
We only care about the accessor for a single store object anyway, but
the validity gets ignored. Also `pathExists(store.printStorePath(path))`
is definitely incorrect since it confuses the logical location vs physical
location in case of a chroot store.
This is a simple wrapper around getFSAccessor that throws an InvalidPath
error. This simplifies usage in callsites that only care about getting
a non-null accessor.
Wrap fmt() calls in lambdas to defer string formatting until the
feature check fails. This avoids unnecessary string formatting in
the common case where the feature is enabled.
Addresses performance concern raised by xokdvium in PR review.
This, alongside the other invariants of the CanonPath is important
to uphold. std::filesystem happily crashes on NUL bytes in the constructor,
as we've seen with `path:%00` prior to c436b7a32a.
Best to stay clear of NUL bytes when we're talking about syscalls, especially
on Unix where strings are null terminated.
Very nice to have if we decide to switch over to pascal-style strings.
The refactor in the last commit fixed the bug it was supposed to fix,
but introduced a new bug in that sometimes we tried to write a resolved
derivation to a store before all its `inputSrcs` were in that store.
The solution is to defer writing the derivation until inside
`DerivationBuildingGoal`, just before we do an actual build. At this
point, we are sure that all inputs in are the store.
This does have the side effect of meaning we don't write down the
resolved derivation in the substituting case, only the building case,
but I think that is actually fine. The store that actually does the
building should make a record of what it built by storing the resolved
derivation. Other stores that just substitute from that store don't
necessary want that derivation however. They can trust the substituter
to keep the record around, or baring that, they can attempt to re
resolve everything, if they need to be audited.
(cherry picked from commit c97b050a6c)
Resolve the derivation before creating a building goal, in a context
where we know what output(s) we want. That way we have a chance just to
download the outputs we want.
Fix#13247
(cherry picked from commit 39f6fd9b46)
Store the reason string as a field in the exception class rather than
only embedding it in the error message. This supports better structured
error handling and future JSON error reporting.
Suggested by Ericson2314 in PR review.
std::regex is a really bad tool for parsing things, since
it tends to overflow the stack pretty badly. See the build failure
under ASan in [^].
[^]: https://hydra.nixos.org/build/310077167/nixlog/5
CURL is not very strict about validation of URLs passed to it. We
should reflect this in our handling of URLs that we get from the user
in <nix/fetchurl.nix> or builtins.fetchurl. ValidURL was an attempt to
rectify this, but it turned out to be too strict. The only good way to
resolve this is to pass (in some cases) the user-provided string verbatim
to CURL. Other usages in libfetchers still benefit from using structured
ParsedURL and validation though.
nix store prefetch-file --name foo 'https://cdn.skypack.dev/big.js@^5.2.2'
error: 'https://cdn.skypack.dev/big.js@^5.2.2' is not a valid URL: leftover
Add support for pre-resolving AWS credentials in the parent process
before forking for builtin:fetchurl. This avoids recreating credential
providers in the forked child process.
The previous implementation had a check-then-create race condition where
multiple threads could simultaneously:
1. Check the cache and find no provider (line 122)
2. Create their own providers (lines 126-145)
3. Insert into cache (line 161)
This resulted in multiple credential providers being created when
downloading multiple packages in parallel, as each .narinfo download
would trigger provider creation on its own thread.
Fix by using boost::concurrent_flat_map's try_emplace_and_cvisit, which
provides atomic get-or-create semantics:
- f1 callback: Called atomically during insertion, creates the provider
- f2 callback: Called if key exists, returns cached provider
- Other threads are blocked during f1, so no nullptr is ever visible
This will reduce the load on hydra. It doesn't make sense to
build 2 slightly different variations where the difference
is only in the nix-perl-bindings and additional sanitizers.
There's some unfortunate ODR violations that get dianosed with GCC but not Clang
for static inline constexpr variables defined inside the class body:
template<typename T>
struct static_const
{
static JSON_INLINE_VARIABLE constexpr T value{};
};
This can be ignored pretty much. There is the same problem for std::piecewise_construct:
http://lists.boost.org/Archives/boost/2007/06/123353.php
==2455704==ERROR: AddressSanitizer: odr-violation (0x7efddc460e20):
[1] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/pkmljfq97a83dbanr0n64zbm8cyhna33-nix-store-2.33.0pre/lib/libnixstore.so.2.33.0
[2] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/gbjpkjj0g8vk20fzlyrwj491gwp6g1qw-nix-util-2.33.0pre/lib/libnixutil.so.2.33.0
Instead of specifying env variables all the time
we can instead embed the __asan_default_options symbol
in all executables / shared objects. This reduces code
duplication.
This change overrides __assert_fail on glibc/musl
to instead call std::terminate that we have a custom
handler for. This ensures that we have more context
to diagnose issues encountered by users in the wild.
This commit adds two key fixes to http-binary-cache-store.cc to
properly support the new curl-based S3 implementation:
1. **Consistent cache key handling**: Use `getReference().render(withParams=false)`
for disk cache keys instead of `cacheUri.to_string()`. This ensures cache
keys are consistent with the S3 implementation and don't include query
parameters, which matches the behavior expected by Store::queryPathInfo()
lookups.
2. **S3 query parameter preservation**: When generating file transfer requests
for S3 URLs, preserve query parameters from the base URL (region, endpoint,
etc.) when the relative path doesn't have its own query parameters. This
ensures S3-specific configuration is propagated to all requests.
I want to separate "policy" from "mechanism".
Now the logic to decide how to build (a policy choice, though with some
hard constraints) is all in derivation building goal, and all in the
same spot. build hook, external builder, or local builder --- the choice
between all three is made in the same spot --- pure policy.
Now, if you want to use the external deriation builder, you simply
provide the `ExternalBuilder` you wish to use, and there is no
additional checking --- pure mechanism. It is the responsibility of the
caller to choose an external builder that works for the derivation in
question.
Also, `checkSystem()` was the only thing throwing `BuildError` from
`startBuilder`. Now that that is gone, we can now remove the
`try...catch` around that.
Add a new S3BinaryCacheStore implementation that inherits from
HttpBinaryCacheStore.
The implementation is activated with NIX_WITH_CURL_S3, keeping the
existing NIX_WITH_S3_SUPPORT (AWS SDK) implementation unchanged.
This code had several issues:
1. Not going through the SourceAccessor means that we can only work
with physical paths.
2. It did not actually check that the file exists. (std::ifstream does not check
it by default).
Most of the eval cache logic is flake-independent and libexpr,
but the loading part is not.
`nix-flake` is the right component for this, as the eval cache
isn't exactly specific to the command line.
we have now merge queues for maintainance branches. We still build it
for master to have our installer beeing updated. In future this part
could go in new workflow instead.
This barfed with
error: [json.exception.type_error.302] type must be string, but is array
on `nix build github:malt3/bazel-env#bazel-env` because it has a `exportReferencesGraph` with a value like `["string",...["string"]]`.
Add a `UsernameAuth` struct and optional `usernameAuth` field to
`FileTransferRequest` to support programmatic username/password
authentication.
This uses curl's `CURLOPT_USERNAME`/`CURLOPT_PASSWORD` options, which
works with multiple protocols (HTTP, FTP, etc.) and is not specific to
any particular authentication scheme.
The primary motivation is to enable S3 authentication refactoring where
AWS credentials (access key ID and secret access key) can be passed
through this general-purpose mechanism, reducing the amount of
S3-specific code behind `#if NIX_WITH_CURL_S3` guards.
This breaks gdb pretty-printers inserted into .debug_gdb_scripts section,
because it implies --compress-debug-sections=zlib, -Wa,--compress-debug-sections.
This is very unfortunate, because then gdb can't use pretty printers for
Boost.Unordered (which are very useful, since boost::unoredred_flat_map is
impossible to debug). This seems perfectly fine to disable in the dev-shell for
the time being.
See [1-3] for further references.
With this change I'm able to use boost's pretty-printers out-of-the box:
```
p *importResolutionCache
$2 = boost::concurrent_flat_map with 1 elements = {[{accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
path = "/derivation-internal.nix"}}] = {accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
path = "/derivation-internal.nix"}}}
```
When combined with a simple `add-auto-load-safe-path ~/code` in .gdbinit
[1]: https://gerrit.lix.systems/c/lix/+/3880
[2]: https://git.lix.systems/lix-project/lix/issues/1003
[3]: https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
Firstly, this is now available on darwin where the default in llvm 19.
Secondly, this leads to very weird segfaults when building with newer nixpkgs for some reason.
(It's UB after all).
This appears when building with the following:
mesonComponentOverrides = finalAttrs: prevAttrs: {
mesonBuildType = "debugoptimized";
dontStrip = true;
doCheck = false;
separateDebugInfo = false;
preConfigure = (prevAttrs.preConfigure or "") + ''
case "$mesonBuildType" in
release|minsize|debugoptimized) appendToVar mesonFlags "-Db_lto=true" ;;
*) appendToVar mesonFlags "-Db_lto=false" ;;
esac
'';
};
And with the following nixpkgs input:
nix build ".#nix-cli" -L --override-input nixpkgs "https://releases.nixos.org/nixos/unstable/nixos-25.11pre870157.7df7ff7d8e00/nixexprs.tar.xz"
Stacktrace:
#0 0x00000000006afdc0 in ?? ()
#1 0x00007ffff71cebb6 in _Unwind_ForcedUnwind_Phase2 () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libgcc_s.so.1
#2 0x00007ffff71cf5b5 in _Unwind_Resume () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libgcc_s.so.1
#3 0x00007ffff7eac7d8 in std::basic_ios<char, std::char_traits<char> >::~basic_ios (this=<optimized out>, this=<optimized out>)
at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/basic_ios.h:286
#4 std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream (this=<optimized out>, this=<optimized out>)
at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/sstream:806
#5 nix::SimpleLogger::logEI (this=<optimized out>, ei=...) at ../logging.cc:121
#6 0x00007ffff7515794 in nix::Logger::logEI (this=0x675450, lvl=nix::lvlError, ei=...) at /nix/store/bkshji3nnxmrmgwa4n2kaxadajkwvn65-nix-util-2.32.0pre-dev/include/nix/util/logging.hh:144
#7 nix::handleExceptions (programName=..., fun=...) at ../shared.cc:336
#8 0x000000000047b76b in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/new_allocator.h:88
This broke invocations like:
NIX_SSHOPTS='-p2222 -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no' nix copy /nix/store/......-foo --to ssh-ng://root@localhost
In Nix 2.30.2, fakeSSH was enabled when the "thing I want to connect to"
was plain old "localhost". Previously, this check was written as:
, fakeSSH(host == "localhost")
Given the above invocation, `host` would have been `root@localhost`, and
thus `fakeSSH` would be `false` because `root@localhost` != `localhost`.
However, since 49ba06175e, `authority.host`
returned _just_ the host (`localhost`, no user) and erroneously enabled
`fakeSSH` in this case, causing `NIX_SSHOPTS` to be ignored (since,
when `fakeSSH` is `true`, `SSHMaster::startCommand` doesn't call
`addCommonSSHOpts`).
`authority.to_string()` accurately returns the expected `root@localhost`
format (given the above invocation), fixing this.
These are helper programs that execute derivations for specified
system types (e.g. using QEMU to emulate another system type).
To use, set `external-builders`:
external-builders = [{"systems": ["aarch64-linux"], "program": "/path/to/external-builder.py"}]
The external builder gets one command line argument, the path to a JSON file containing all necessary information about the derivation:
{
"args": [...],
"builder": "/nix/store/kwcyvgdg98n98hqapaz8sw92pc2s78x6-bash-5.2p37/bin/bash",
"env": {
"HOME": "/homeless-shelter",
...
},
"realStoreDir": "/tmp/nix/nix/store",
"storeDir": "/nix/store",
"tmpDir": "/tmp/nix-shell.dzQ2hE/nix-build-patchelf-0.14.3.drv-46/build",
"tmpDirInSandbox": "/build"
}
Co-authored-by: Cole Helbling <cole.helbling@determinate.systems>
Until these repos are potentially merged, this is good for dogfooding
alongside the experimental installer. It also uses the more official
`artifacts.nixos.org` endpoint to install stable releases now
More immediately though, we need a patch for the experimental installer
to really work in CI at all, and that hasn't landed in a tag yet. So,
this lets us use it right from `main`!
Introduce a new build option 'curl-s3-store' for the curl-based S3
implementation, separate from the existing AWS SDK-based 's3-store'.
The two options are mutually exclusive to avoid conflicts.
Users can enable the new implementation with:
-Dcurl-s3-store=enabled -Ds3-store=disabled
Add lightweight AWS credential resolution using AWS CRT (Common Runtime)
instead of the full AWS SDK. This provides credential management for the
upcoming curl-based S3 implementation.
Realisations are conceptually key-value pairs, mapping `DrvOutputs` (the
key) to information about that derivation output.
This separate the value type, which will be useful in maps, etc., where
we don't want to denormalize by including the key twice.
This matches similar changes for existing types:
| keyed | unkeyed |
|--------------------|------------------------|
| `ValidPathInfo` | `UnkeyedValidPathInfo` |
| `KeyedBuildResult` | `BuildResult` |
| `Realisation` | `UnkeyedRealisation` |
Best I can tell this was never supposed to be exposed to the user
and has been this way since 2.19.
2.18 did not expose this file to the user:
nix run nix/2.18-maintenance -- eval --expr "import <nix/derivation-internal.nix>"
error: getting status of '/__corepkgs__/derivation-internal.nix': No such file or directory
https://en.cppreference.com/w/cpp/thread.html
src/libstore/gc.cc:121:39: error: no member named 'sleep_for' in namespace 'std::this_thread'
121 | std::this_thread::sleep_for(std::chrono::milliseconds(100));
| ~~~~~~~~~~~~~~~~~~^
Move ParsedS3URL from s3.cc/.hh into dedicated s3-url.cc/.hh files.
This separates URL parsing utilities (which are protocol-agnostic) from
the AWS SDK-specific S3Helper implementation, making the code cleaner
and enabling reuse by future curl-based S3 implementation.
The refactor in the last commit fixed the bug it was supposed to fix,
but introduced a new bug in that sometimes we tried to write a resolved
derivation to a store before all its `inputSrcs` were in that store.
The solution is to defer writing the derivation until inside
`DerivationBuildingGoal`, just before we do an actual build. At this
point, we are sure that all inputs in are the store.
This does have the side effect of meaning we don't write down the
resolved derivation in the substituting case, only the building case,
but I think that is actually fine. The store that actually does the
building should make a record of what it built by storing the resolved
derivation. Other stores that just substitute from that store don't
necessary want that derivation however. They can trust the substituter
to keep the record around, or baring that, they can attempt to re
resolve everything, if they need to be audited.
Resolve the derivation before creating a building goal, in a context
where we know what output(s) we want. That way we have a chance just to
download the outputs we want.
Fix#13247
A very unfortunate interaction of current filtering with pure eval is
that the following actually leads to `lib.a = {}`. This just adds a unit
test for this broken behavior. This is really good to be done as a unit test
via the in-memory store.
{
outputs =
{ ... }:
{
lib.a = builtins.readDir /.;
};
}
Whoever first calls `quit` now empties the queue, instead of waiting for
the worker thread to do it.
(Note that in the unwinding case, the worker thread is still the first
to call `quit`, though.)
This is my SNAFU. Accidentally broken in 02c9ac445f.
There's very dubious behavior for 'builtins.readDir /.':
{
outputs =
{ ... }:
{
lib.a = builtins.readDir /.;
};
}
nix eval /tmp/test-flake#lib.a
Starting from 2.27 this now returns an empty set. This really isn't supposed
to happen, but this change in the semantics of makeEmptySourceAccessor accidentally
changed the behavior of this.
The followLinksToStore() function could hang indefinitely when encountering
symlink cycles outside the Nix store, causing 100% CPU usage and blocking
any operations that use this function.
This affects multiple commands including nix-store --query, --delete,
--verify, nix-env, and nix-copy-closure when given paths with symlink cycles.
The fix adds a maximum limit of 1024 symlink follows (matching the limit
used by canonPath) and throws an error when exceeded, preventing the
infinite loop while preserving the original semantics of stopping at
the first path inside the store.
Replace non-thread-safe ptsname() calls with a new getPtsName() helper
function that:
- Uses thread-safe ptsname_r() on Linux/BSD platforms
- Uses mutex-protected ptsname() on macOS (which lacks ptsname_r())
This turns out to be a big problem for performance of Bison
generated code, that for whatever reason cannot be made internal
to the shared library. This causes GCC to make a bunch of function
calls go through PLT. Ideally these hot functions (like move/copy ctor) could become
inline in upstream Bison. That will make sure that GCC can do interprocedular
optimizations without -fno-semantic-interposition [^]. Considering that
LLVM already does inlining and whatnot is a good motivation for this change.
I don't know of any case where Nix relies on LD_PRELOAD tricks for the shared
libraries in production use-cases.
[^]: https://maskray.me/blog/2021-05-09-fno-semantic-interposition
Since the parser is now LALR we can easily switch
over to the less ugly sketelon than the default C one.
This would allow us to switch from %union to %define api.value.type variant
in the future to avoid the need for triviall POD types.
1. Saves 24-32 bytes per string (size of std::string)
2. Saves additional bytes by not over-allocating strings (in total we
save ~1% memory)
3. Sets us up to perform a similar transformation on the other Expr
subclasses
4. Makes ExprString trivially moveable (before the string data might
move, causing the Value's pointer to become invalid). This is important
so we can put ExprStrings in an std::vector and refer to them by index
We have introduced a string copy in ParserState::stripIndentation().
This could be removed by pre-allocating the right sized string in the
arena, but this adds complexity and doesn't seem to improve performance,
so for now we've left the copy in.
This mirrors what OptionalPathSetting does. Otherwise we run into
an assertion failure for relative paths specified as the authority + path:
nix build nixpkgs#hello --store "local://a/b"
nix: ../posix-source-accessor.cc:13: nix::PosixSourceAccessor::PosixSourceAccessor(std::filesystem::__cxx11::path&&): Assertion `root.empty() || root.is_absolute()' failed.
This is now diagnosed properly:
error: not an absolute path: 'a/b'
Just as you'd specify the root via a query parameter:
nix build nixpkgs#hello --store "local?root=a/b"
Fewer macros is better!
Introduce a new `JsonChacterizationTest` mixin class to help with this.
Also, avoid some needless copies with `GetParam`.
Part of my effort shoring up the JSON formats with #13570.
These stragglers have been accidentally left out when implementing the StoreConfig::getReference.
Also HttpBinaryCacheStore::getReference now returns the actual store parameters, not the cacheUri
parameters.
In the case where the store object doesn't exist, we do correctly move
(rather than copy) the scratch data into place. In this case, the
destination store object already exists, but we still want to clean up
after ourselves.
This avoids any complications that can arise from the environment
affecting evaluation of the help pages (which don't need to be calling
out to anything external anyways)
A recent example of one of these problems is
https://github.com/NixOS/nix/issues/14085, which would break help pages
by causing them to make invalid calls to the dummy store they're
evaluated with
Fixes: https://github.com/NixOS/nix/issues/14062
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
fetchToStore() caching was broken because it uses the fingerprint of
the accessor, but now that the accessor (typically storeFS) is a
composite (like MountedSourceAccessor or AllowListSourceAccessor),
there was no fingerprint anymore. So fetchToStore now uses the new
getFingerprint() method to get the specific fingerprint for the
subpath.
This returns the fingerprint for a specific subpath. This is intended
for "composite" accessors like MountedSourceAccessor, where different
subdirectories can have different fingerprints.
Previously, Nix would not create a cache entry for substituted/cached
inputs
This led to severe slowdowns in some scenarios where a large input (like
Nixpkgs) had already been unpacked to the store but didn't exist in a
users cache, as described in https://github.com/NixOS/nix/issues/11228
Using the same method as https://github.com/NixOS/nix/pull/12911, we can
create a cache entry for the fingerprint of substituted/cached inputs
and avoid this problem entirely
These counters are extremely expensive in a multi-threaded
program. For instance, disabling them speeds up evaluation of the
NixOS/nix/2.21.2 from 32.6s to 17.8s.
With this change, the store-wide `getFSAccessor` has only one usage left
--- the evaluator. If we get rid of that (as is planned), we can then
remove that method altogether, simplifying `Store`. Hurray!
I removed the store dir by mistake from the pretty-printed (for humans)
output in eb643d034f. That change was not
supposed to change output.
This is sometimes easier / more performant to implement, and
independently it is also a more convenient interface for many callers.
The existing store-wide `getFSAccessor` is only used for
- `nix why-depends`
- the evaluator
I hope we can get rid of it for those, too, and then we have the option
of getting rid of the store-wide method.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
This makes the CI fail fast and more explicitly in case the formatting
is incorrect and provides a better error messages. This also ensures
that we don't burn CI on useless checks for code that wouldn't pass lints
anyway.
Old code is now just used for `nix build` --- there is no CLI breaking
change.
Test the new format, too.
The new format is not currently used, but will be used going forward,
for example in the C API.
Progress on #13570
This brings them in line with the other tests, and furthers my goals of
separating unit test data from code.
Doing this cleanup as part of my #13570 effort, but strictly-speaking,
this is separate as these data types' JSON never contained and store
paths or store dirs, just simple output name strings.
Tested by building with b_sanitize=thread and running:
nix flake prefetch-inputs --store "dummy://?read-only=false"
It might make sense to move this utility class out of dummy-store.cc,
but it seems fine for now.
No behavior is changed, just:
- Declare a canonical `nlohmnan::json::adl_serializer`
- Use `json-utils.hh` to shorten code without getting worse error
messages.
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
We should use proper abstractions for reading files from the store.
E.g. this caused errors when trying to download github flakes into
an in-memory store in #14023.
The docs weren't 100% clear about bounds checking, but suggested that
errors would be caught.
The bounds checks are cheap compared to the function calls they're in,
so we have no reason to omit them.
Enables builds with ASAN to catch memory corruption
bugs faster and in CI. This is an incredibly valuable
instrument that must be used as much as possible.
Somewhat based on jade's work from Lix, though there's a lot that
we have to do differently:
19ae87e5ce
Co-authored-by: Jade Lovelace <lix@jade.fyi>
This leads to ASAN errors:
==1137785==ERROR: AddressSanitizer: new-delete-type-mismatch on 0x523000001d00 in thread T0:
object passed to delete has wrong type:
size of the allocated type: 5968 bytes;
size of the deallocated type: 5968 bytes.
alignment of the allocated type: 8 bytes;
alignment of the deallocated type: default-aligned.
This has multiple dangling pointer issues that lead to segfaults in e.g.:
nix eval --expr '(builtins.getFlake "github:nixos/nixpkgs/25.05")' --impure
This reverts commit ad175727e4, reversing
changes made to d314750174.
See #13570 for details --- the idea is that included the store dir in
store paths makes systematic JSON parting with e.g. Serde, Aeson,
nlohmann, or similiar harder.
After talking to Eelco, we are changing the `Derivation` format right
away because not only is `nix derivation` technically experimental, we think it is
also less widely used in practice than, say, `nix path-info`.
Progress on #13570
Add `read-only` setting to `dummy://` store for back compat.
Test by changing an existing test to use this instead, fixing a TODO.
Co-Authored-By: HaeNoe <git@haenoe.party>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
Since `nix flake check` doesn't produce a `result` symlink, it doesn't
actually need to build/substitute derivations that are already known
to have succeeded, i.e. that are substitutable.
This can speed up CI jobs in cases where the derivations have already
been built by other jobs. For instance, a command like
nix flake check github:NixOS/hydra/aa62c7f7db31753f0cde690f8654dd1907fc0ce2
should no longer build anything because the outputs are already in
cache.nixos.org.
Based-on: https://github.com/DeterminateSystems/nix-src/pull/134
Based-on: https://gerrit.lix.systems/c/lix/+/3841
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
- Use `const K`, not `K`, otherwise we don't get auto referencing of
rvalues.
- Generalized the deleted overloads, because we don't care what the key
type is --- we want to get rid of anything that has an rvalue map
type.
A follow-up optimization will make it impossible to make a find function
that returns an iterator in an efficient manner. All consumer code can
easily use the `get` variant.
As evident from the number of tests that were holding this API completely
wrong (the end() iterator returned from find() is NEVER nullptr) we should
not have this footgun. A proper strong type guarantees that this confusion
will not happen again.
Also this will be helpful down the road when Bindings becomes something
smarter than an array of Attr.
This allows the weird network or DNS server fallback mechanism inside
glibc to work, and prevents a "Resolving timed out after 5000
milliseconds" error. Read on for details.
The DNS request stuff (dns-hosts) in glibc uses this fallback procedure
to minimize network RTT in the ideal case while dealing with
ill-behaving networks and DNS servers gracefully (see resolv.conf(5)):
- Use sendmmsg() to send UDP DNS requests for IPv4 and IPv6 in parallel
- If that times out (meaning that none or only one of the responses have
been received), send the requests one by one, waiting for the response
before sending the next request ("single-request")
- If that still times out, try to use a different socket (hence
different address) for each request ("single-request-reopen")
The default timeout inside glibc is 5 seconds. Therefore, setting
connect-timeout, and therefore CURLOPT_CONNECTTIMEOUT to 5 seconds
prevents the single-request fallback, and setting it to even 10 seconds
prevents the single-request-reopen fallback as well.
The fallback decision is saved by glibc, but only thread-locally, and
libcurl starts a new thread for getaddrinfo() for each connection.
Therefore for every connection the fallback starts from sendmmsg() all
over again. And since these are considered to have timed out by libcurl,
even though getaddrinfo() might return a successful result, it is not
cached in libcurl.
While a user could tweak these with resolv.conf(5) options (e.g. using
networking.resolvconf.extraOptions in NixOS), and indeed that is
probably needed to avoid annoying delays, it still means that the
default connect-timeout of 5 is too low. Raise it to give fallback a
chance.
../hash.cc: In function 'nix::{anonymous}::DecodeNamePair nix::baseExplicit(HashFormat)':
../hash.cc:114:1: warning: control reaches end of non-void function [-Wreturn-type]
114 | }
| ^
This has been dropped on unstable an nix no longer
compiled with overridden nixpkgs input. On 25.05 these
overrides already do nothing.
Tested with:
nix build .#packages.x86_64-darwin.nix-cli -L --override-input nixpkgs https://releases.nixos.org/nixos/unstable/nixos-25.11pre859555.ab0f3607a6c7/nixexprs.tar.xz
Default deployment target on 25.05 is 11.3, so 10.13
sdk override doesn't have to be updated at all as evident
from the fact that we didn't observe any issues with it.
This is because we need it in declarations where we should not be
including the full `nlohmann/json.hpp`.
Already can clean up by moving the experimental feature "instance".
Also, make the `std::map` instance better by allowing for other
comparison functions.
This reverts commit bdbc739d6e.
Such a change needs more thought put into it. By versioning
shared libraries we'd make a false impression that libraries
themselves are actually versioned and have some sort of stable
ABI, which is not the case.
This will be useful when C bindings become stable, but as long
as they are experimental it does not make sense to set SONAME.
Also this change should not have been backported, since it's
severely breaking.
When doing multithreaded evaluation, we want to ensure that any Nix
file is parsed and evaluated only once. The easiest way to do this is
to rely on thunks, since those ensure locking in the multithreaded
evaluator. `fileEvalCache` is now a mapping from `SourcePath` to a
`Value *`. The value is initially a thunk (pointing to a
`ExprParseFile` helper object) that can be forced to parse and
evaluate the file. So a subsequent thread requesting the same file
will see a thunk that is possibly locked and wait for it.
The parser cache is gone since it's no longer needed. However, there
is a new `importResolutionCache` that maps `SourcePath`s to
`SourcePath`s (e.g. `/foo` to `/foo/default.nix`). Previously we put
multiple entries in `fileEvalCache`, which was ugly and could result
in work duplication.
These constant Values have no business being in the EvalState in the
first place. The ultimate goal is to get rid of the ugly `getBuiltins`
and its relience (in `createBaseEnv`) on these global constants is getting in the way.
Same idea as in f017f9ddd3.
Co-authored-by: eldritch horrors <pennae@lix.systems>
This object is always constant and will never get modified.
Having it as a global (constant) static is much easier and
unclutters the EvalState.
Same idea as in f017f9ddd3.
Co-authored-by: eldritch horrors <pennae@lix.systems>
This implements a special back-compat shim to specifically allow
unbracketed IPv6 addresses in store references. This is something
that is relied upon in the wild and the old parsing logic accepted
both ways (brackets were optional). This patch restores this behavior.
As always, we didn't have any tests for this.
Addresses #13937.
`perf c2c` shows a lot of cacheline conflicts between purely read-only
Store methods (like `parseStorePath()`) and the Sync classes. So
allocate pathInfoCache separately to avoid that.
Calling `drainFD()` will hang if another process has the write side
open, since then the child won't get an EOF. This can happen if we
have multiple threads doing a build, since in that case another thread
may fork a child process that inherits the write side of the first
thread.
We could set O_CLOEXEC on the write side (using pipe2()) but it won't
help here since we don't always do an exec() in the child, e.g. in the
case of builtin builders. (We need a "close-on-fork", not a
"close-on-exec".)
Since the only construction and push_back() calls
to Bindings happen through the `BindingsBuilder` [1] we don't
need to keep `capacity` around on the heap anymore. This saves 8 bytes
(because of the member alignment padding)
per one Bindings allocation. This isn't that much, but it does
save significant memory.
This also shows that the Bindings don't necessarily have to
be mutable, which opens up opportunities for doing small bindings
optimization and storing a 1-element Bindings directly in Value.
For the following scenario:
nix-env --query --available --out-path --file ../nixpkgs --eval-system x86_64-linux
(nixpkgs revision: ddcddd7b09a417ca9a88899f4bd43a8edb72308d)
This patch results in reduction of `sets.bytes` 13115104016 -> 12653087640,
which amounts to 462 MB less bytes allocated for Bindings.
[1]: Not actually, `getBuiltins` does mutate bindings, but this is pretty
inconsequential and doesn't lead to problems.
This is relied upon (specifically the `local` store) by existing
tooling [1] and we broke this in 3e7879e6df (which
was first released in 2.31).
To lessen the scope of the breakage we should not normalize "auto" references
and explicitly specified references like "local" or "daemon". It also makes
sense to canonicalize local://,daemon:// to be more compatible with prior
behavior.
[1]: 05e1b3cba2/lib/NOM/Builds.hs (L60-L64)
Exactly why is is correct is a little subtle, because sometimes the
worker is owned by the worker. But the commit message in
e437b08250 explained the situation well
enough: I made that commit message part of the ABI docs, and now it
should be understandable to the next person.
Do this with a new `useHook` boolean we carefully make sure is set in
all cases. This change isn't really worthwhile by itself, but it allows
us to make further refactors (see later commits) which are
well-motivated.
When useMaster is true, startMaster() acquires the state lock, then
calls isMasterRunning(), which calls addCommonSSHOpts(), which tries
to acquire the state lock again, causing a deadlock.
The solution is to move tmpDir out of the state. It doesn't need to be
there in the first place because it never changes.
On macOS, poll() is fundamentally broken for HUP detection. It loses event
subscriptions when EVFILT_READ fires without matching the requested events
in the pollfd. This causes daemon processes to linger after client disconnect.
This commit replaces poll() with kqueue on macOS, which is what poll()
uses internally but without the bugs. The kqueue implementation uses
EVFILT_READ which works for both sockets and pipes, avoiding EVFILT_SOCK
which only works for sockets.
On Linux and other platforms, we continue using poll() with the standard
POSIX behavior where POLLHUP is always reported regardless of requested events.
Based on work from the Lix project (https://git.lix.systems/lix-project/lix)
commit 69ba3c92db3ecca468bcd5ff7849fa8e8e0fc6c0
Fixes: https://github.com/NixOS/nix/issues/13847
Related: https://git.lix.systems/lix-project/lix/issues/729
Apple bugs: rdar://37537852 (poll), FB17447257 (poll)
Co-authored-by: Jade Lovelace <jadel@mercury.com>
Now that Symbols are statically allocated at compile time with known IDs,
we can use switch statements instead of if-else chains for Symbol comparisons.
This provides better performance through compiler optimizations like jump tables.
Changes:
- Add public getId() method to Symbol class to access the internal ID
- Convert if-else chains comparing Symbol values to switch statements
in primops.cc's derivationStrictInternal function
- Simplify control flow by removing the 'handled' flag and moving the
default attribute handling into the switch's default case
The static and runtime Symbol IDs are guaranteed to match by the
copyIntoSymbolTable implementation which asserts this invariant.
Co-authored-by: John Ericson <git@JohnEricson.me>
In b70d22b `mkStringNoCopy()` was renamed to
`mkString()`, but this is a bit risky since in code like
vStringRegular.mkString("regular");
we want to be sure that the right overload is picked. (This is
especially problematic since the overload that takes an
`std::string_view` *does* allocate.) So let's be explicit.
(Rebased from https://github.com/NixOS/nix/pull/11551)
I (@Ericson2314) messed up. We were supposed to test the status quo
before landing any new chnages, and also there is one change that is not
quite right (relative paths).
I am reverting for now, and then backporting the test suite to the old
situation.
This reverts commit 04ad66af5f.
Git URI can also support scp style links similar to git itself.
This change augments the function fixGitURL to better handle the scp
style urls through a minimal parser rather than regex which has been
found to be brittle.
* Support for IPV6 added
* New test cases added for fixGitURL
* Clearer documentation on purpose and goal of function
* More `std::string_view` for performance
* A few more URL tests
Fixes#5958
The URL should not be normalized before handing it off to cURL, because
builtin fetchers like fetchTarball/fetchurl are expected to work with
arbitrary URLs, that might not be RFC3986 compliant. For those cases
Nix should not normalize URLs, though validation is fine. ParseURL and
cURL are supposed to match the set of acceptable URLs, since they implement
the same RFC.
This adds regression tests for fromTOML overflow/underflow behavior.
Previous versions of toml11 used to saturate, but this was never an
intended behavior (and Snix/Nix 2.3/toml11 >= 4.0 validate this).
(cherry picked from Lix [1,2])
[1]: 7ee442079d
[2]: 4de09b6b54
The motivation for this change is two-fold:
1. Commonly used Symbol values can be referred to
quite often and they can be assigned at compile-time
rather than runtime.
2. This also unclutters EvalState constructor, which was
getting very long and unreadable.
Spiritually similar to https://gerrit.lix.systems/c/lix/+/2218,
though that patch doesn't allocate the Symbol at compile time.
Co-authored-by: eldritch horrors <pennae@lix.systems>
Looking at perf:
0.21 │ push %rbp
0.99 │ mov %rsp,%rbp
│ push %r15
0.25 │ push %r14
│ push %r13
0.49 │ push %r12
0.66 │ push %rbx
1.23 │ lea -0x10000(%rsp),%r11
0.23 │ 15: sub $0x1000,%rsp
1.01 │ orq $0x0,(%rsp)
59.12 │ cmp %r11,%rsp
0.27 │ ↑ jne 15
Seems like 64K is too much to have on the stack for each invocation, considering
that only a minuscule number of allocations are actually larger than 4K.
There's actually no good reason this function should use so much stack space. Or
use small_string at all. Everything can be done in small chunks that don't require
any memory allocations and use up 2K bytes on the stack.
This patch also adds a microbenchmark for tracking the unparsing performance. Here
are the results for this change:
(Before)
BM_UnparseRealDerivationFile/hello 7275 ns 7247 ns 96093 bytes_per_second=232.136Mi/s
BM_UnparseRealDerivationFile/firefox 40538 ns 40376 ns 17327 bytes_per_second=378.534Mi/s
(After)
BM_UnparseRealDerivationFile/hello 3228 ns 3218 ns 215671 bytes_per_second=522.775Mi/s
BM_UnparseRealDerivationFile/firefox 39724 ns 39584 ns 17617 bytes_per_second=386.101Mi/s
This translates into nice evaluation performance improvements (compared to 18c3d2348f):
Benchmark 1: GC_INITIAL_HEAP_SIZE=8G old-nix/bin/nix-instantiate ../nixpkgs -A nixosTests.gnome --readonly-mode
Time (mean ± σ): 3.111 s ± 0.021 s [User: 2.513 s, System: 0.580 s]
Range (min … max): 3.083 s … 3.143 s 10 runs
Benchmark 2: GC_INITIAL_HEAP_SIZE=8G result/bin/nix-instantiate ../nixpkgs -A nixosTests.gnome --readonly-mode
Time (mean ± σ): 3.037 s ± 0.038 s [User: 2.461 s, System: 0.558 s]
Range (min … max): 2.960 s … 3.086 s 10 runs
Old versions of nix happily accepted a lot of weird flake references,
which we didn't have tests for, so this was accidentally broken in
c436b7a32a.
This patch restores previous behavior and adds a plethora of tests
to ensure we don't break this in the future.
These test cases are aligned with how 2.18/2.28 parsed flake references.
Starting from c436b7a32a
this used to lead to assertion failures like:
> std::string nix::ParsedURL::renderAuthorityAndPath() const: Assertion `path.empty() || path.front().empty()' failed.
This has the bugfix for the issue and regressions tests
so that this gets properly tested in the future.
This would print erroneous and misleading diagnostics like:
> error (ignored): error: '--arg' and '--argstr' are incompatible with flakes
When run with --expr/--file. Since this installable is used to get the
bash package it doesn't make sense to check this.
It is only done in the `force = true` case, and the only
`cleanupBuild(true)` call is right after where it used to be, so this
has the exact same behavior as before.
Calling `reset` on this `std::optional` field of `DerivationBuilderImpl`
is also what the (automatically created) destructor of
`DerivationBuilderImpl` will do. We should be making sure that the
derivation builder is cleaned up by the goal anyways, and if we do that,
then this `Finally` is no longer needed.
Before, had a very ugly `appendLogTailErrorMsg` callback. Now, we
instead have a `fixupBuilderFailureErrorMessage` that is just used by
`DerivationBuildingGoal`, and `DerivationBuilder` just returns the raw
data needed by this.
Now we have better separation of the core logic --- an integral part of
the store layer spec even --- from the goal mechanism and other
minutiae.
Co-authored-by: Jeremy Kolb <kjeremy@gmail.com>
See the new extensive doxygen in `url.hh`.
This fixes fetching gitlab: flakes.
Paths are now stored as a std::vector of individual path
segments, which can themselves contain path separators '/' (%2F).
This is necessary to make the Gitlab's /projects/ API work.
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
I think this should be fine for repairing. If anything, it is better,
because it would be weird to "mark and output good" only for it to then
fail output checks.
Sadly we cannot unexpose `DerivationBuilder::killChild` yet, because
`DerivationBuildingGoal` calls it elsewhere, but we can at least haave a
better division of labor between the two destructors.
It's hard to tell if I changed any behavior, but if I did, I think I
made it better, because now we explicitly move stuff out of the chroot
(if we were going to) before trying to delete the chroot.
Do this to match `DerivationBuilder::deleteTmpDir`, which we'll want to
combine it with next.
Also chenge one caller from `deleteTmpDir(true)` to `cleanupBuild(true)`
now that this is done, because it will not make a difference.
This should be a pure refactor with no behavioral change.
Aftet the previous simplifications, there is no reason to catch the
error and immediately return it with a `std::variant` --- just let the
caller catch it instead.
Instead of that funny business, the fixed output checks are not put in
`checkOutputs`, with the other (newer) output checks, where they also
better belong. The control flow is reworked (with comments!) so that
`checkOutputs` also runs in the `bmCheck` case.
Not only does this preserve existing behavior of `bmCheck`
double-checking fixed output hashes with less tricky code, it also makes
`bmCheck` better by also double-checking the other output checks, rather
than just assuming they pass if the derivation is deterministic.
It's fine to set these worker flags a little later in the control flow,
since we'll be sure to reach those points in the error cases. And doing
that is much nicer than having these tangled callbacks.
I originally made the callbacks to meticulously recreate the exact
behavior which I didn't quite understand. Now, thanks to cleaning up the
error handling, I do understand what is going on, so I can be confident
that this change is safe to make.
Instead of passing them around separately, or doing finicky logic in a
try-catch block to recover them, just make `BuildError` always contain a
status, and make it the thrower's responsibility to set it. This is much
more simple and explicit.
Once that change is done, split the `done` functions of `DerivationGoal`
and `DerivationBuildingGoal` into separate success and failure
functions, which ends up being easier to understand and hardly any
duplication.
Also, change the handling of failures in resolved cases to use
`BuildResult::DependencyFailed` and a new message. This is because the
underlying derivation will also get its message printed --- which is
good, because in general the resolved derivation is not unique. One dyn
drv test had to be updated, but CA (and dyn drv) is experimental, so I
do not mind.
Finally, delete `SubstError` because it is unused.
The commit says it was added for CA testing --- manual I assume, since
there is no use of this in the test suite. I don't think we need it any
more, and I am not sure whether it was ever supposed to have made it to
`master` either.
This reverts commit 2eec2f765a.
We currently just use this during the build of a derivation, but there is no
reason we wouldn't want to use it elsewhere, e.g. to check the outputs
of someone else's build after the fact.
Moreover, I like pulling things out of `DerivationBuilder` that are
simple and don't need access to all that state. While
`DerivationBuilder` is unix-only, this refactor also make the code more
portable "for free".
The header is private, at Eelco's request.
With the migration to /nix/var/nix/builds we now have failing builds
when the derivation name is too long.
This change removes the derivation name from the temporary build to have
a predictable prefix length:
Also see: https://github.com/NixOS/infra/pull/764
for context.
This allows us to replace some very hacky and not correct string
concatentation in `HttpBinaryCacheStore`. It will especially be useful
with #13752, when today's hacks started to cause problems in practice,
not just theory.
Also make `fixGitURL` returned a `ParsedURL`.
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/cd32a774ac52caaa03bcfc9e7591ac8c18617ced?narHash=sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI%3D' (2025-08-17)
→ 'github:NixOS/nixpkgs/d98ce345cdab58477ca61855540999c86577d19d?narHash=sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8%3D' (2025-08-26)
This update contains d1266642a8722f2a05e311fa151c1413d2b9653c, which
is necessary for the TOML timestamps to get tested via nixpkgsLibTests job.
I need this for some `ParseURL` improvements, but I figure this is
better to send as its own PR.
I changed the tests willy-nilly to sometimes use
`std::list<std::string_view>` instead of `Strings` (which is
`std::list<std::string>`).
Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
It is suppposed to be "post build" not "during the build" after all. Its
location now matches that for the hook case (see elsewhere in
`DerivationdBuildingGoal`).
It was in a try-catch before, and now it isn't, but I believe that it is
impossible for it to throw `BuildError`, which is sufficient for this
code motion to be correct.
Update src/libutil/windows/current-process.cc
Prefer `nullptr` over `NULL`
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Update src/libutil/unix/current-process.cc
Prefer C++ type casts
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Update src/libutil/windows/current-process.cc
Prefer C++ type casts
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Update src/libutil/unix/current-process.cc
Don't allocate exception
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
It would have been nice to use std::views::enumerate here, but
it uses a signed difference type for the value_type:
> value_type = std::tuple<difference_type, ranges::range_value_t<Base>>
zip + iota has the same semantics as the code used to have, so there's
no behavior change here.
This is a nicer separation of concerns --- `DerivationBuilder` just
mounts the extra paths you tell it too, and the outside world is
responsible for making sure those extra paths make sense.
Since the closure only depends on global settings, and not
per-derivation information, we also have the option of moving this up
further and caching it across all local builds. (I only just realized
this after having done this refactor. I am not doing that change at this
time, however.)
Now, `DerivationBuilder` only concerns itself with `finalEnv` and
`extraFiles`, in straightforward unconditional code. All the fancy
desugaring logic is consolidated in `DerivationBuildingGoal`.
We should better share the pulled-out logic with `nix-shell`/`nix
develop`, which would fill in some missing features, arguably fixing
bugs.
I think this is a better separation of concerns. `DerivationBuilder`
doesn't need to to the final, query-heavy details about how these things
are constructed. It just operates on the level of "simple, stupid" files
and environment variables.
As much as I prefer rewriting the parsed rather than unparsed JSON for
elegance, this gets in the way of the separation of concerns that I am
trying to do.
As a practical matter, any rewriting that this did will also be done by
the second round of rewriting that remains below, so removing this code
should have no effect.
This is needed to rearrange include order, but I also think it is a good
thing anyways, as we seek to reduce the use of global settings variables
over time.
This avoids problems with older versions of Nix that don't put the
caches in WAL mode. That's generally not a problem, until you do something like
nix build --print-out-paths ... | cachix
which deadlocks because cachix tries to switch the caches to truncate
mode, which requires exclusive access. But the first process cannot
make progress because the cachix process isn't reading from the pipe.
With "truncate" mode, if we try to write to the database while another
process has an active write transaction, we'll block until the other
transaction finishes. This is a problem for the evaluation cache in
particular, since it uses long-running transactions.
WAL mode does not have this issue: it just returns "busy" right away,
so Nix will print
error (ignored): SQLite database '/home/eelco/.cache/nix/eval-cache-v5/...' is busy
and stop trying to write to the evaluation cache. (This was the
intended/original behaviour, see AttrDb::doSQLite().)
Compilers in nixpkgs have caught up and major distros
should also have recent enough compilers. It would be
nice to have newer features like more full featured
ranges and deducing this.
About time we upgraded our nixpkgs flake input. Ideally
we'd have automation to do this.
Flake lock file updates:
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/adaa24fbf46737f3f1b5497bf64bae750f82942e?narHash=sha256-qhFMmDkeJX9KJwr5H32f1r7Prs7XbQWtO0h3V0a0rFY%3D' (2025-05-13)
→ 'github:NixOS/nixpkgs/cd32a774ac52caaa03bcfc9e7591ac8c18617ced?narHash=sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI%3D' (2025-08-17)
This caused RemoteStore::queryPathInfoUncached() to mark the
connection as invalid (see
RemoteStore::ConnectionHandle::~ConnectionHandle()), causing it to
disconnect and reconnect after every lookup of an invalid path. This
caused huge slowdowns in conjunction with
19f89eb684 and lazy-trees.
Period '.' is a special branch name in the gitsubmodule file which
represents the branch of the parent repository [1].
We add support for this by registering the ref of the InputAccessor to
be that of the parent input if '.' is encountered.
Fixes#13215
[1]: man gitmodules
lowdown >= 1.4.0 supports LOWDOWN_TERM_NORELLINK to render
absolute urls. This is useful, since we want to keep links to
web resources and such intact.
Fix description of `NIX_CONF_DIR`. It currently say that it defaults to `prefix/etc/nix`, which would mean `/nix/etc/nix` on default installations, and contradicts the description in `conf-file-prefix.md`.
This fix makes the description of `NIX_CONF_DIR` consistent with `conf-file-prefix.md`, assuming that the latter is correct.
Turns out we didn't have tests for some of the important behavior introduced
for flake reference fragments and url queries [1]. This is rather important
and is relied upon by existing tooling. This fixes up these exact cases before
handing off the URL to the Boost.URL parser.
To the best of my knowledge this implements the same behavior as prior regex-based
parser did [2]:
> fragmentRegex = "(?:" + pcharRegex + "|[/? \"^])*";
> queryRegex = "(?:" + pcharRegex + "|[/? \"])*";
[1]: 9c0a09f09f
[2]: https://github.com/NixOS/nix/blob/2.30.2/src/libutil/include/nix/util/url-parts.hh
It's about time we added debuggers to the dev-shell. Having it in build inputs
does some magic so pretty printers for standard library types work better.
Since this goal has no (goal-wide) notion of "wanted outputs" (we're
building the derivation, and thus making all outputs), we should have
`initialOutputs` for all outputs, and if we're missing one that's an
internal error caused by a bug in Nix.
Concretely, `DerivationBuildingGoal::gaveUpOnSubstitution` now clearly
does create `initialOutputs` for all outputs, whereas a few commits ago
that was not obvious, so I feel confident in saying that this invariant
that should be upheld, in fact is upheld.
`scatchOutputs` is initialized for every initial output, so the same
change to it follows for the same reasons.
This is just more honest, since we downcasted it to `LocalStore` in many
places. We had the downcast before because it wasn't needed in the hook
case, just the local building case, but now that `DerivationBuilder` is
separated and just does the building case, we have formalized the
boundary where the single downcast should occur.
No derivation goal type has a notion of variable wanted outputs any
more. They either want them all, or they just care about a single
output, in which case we would just store this information for the one
output in question.
We can cut out some gratuitous inhertence as follows:
- `MixStoreDirMethods` -> `StoreDirConfig`
- `StoreDirConfig` deleted because no longer needed. It is just folded
into `StoreConfig`.
- `StoreDirConfigBase` -> `StoreConfigBase` same trick still needed, but
now is for `StoreConfig` not `StoreDirConfig`
Here's how we got here:
1. I once factored out `StoreDirConfig` in #6236.
2. I factored out `MixStoreDirMethods` in #13154.
But, I didn't realize at point (2) that we didn't need `StoreDirConfig`
anymore, all uses of `StoreDirConfig` could instead be uses of
`MixStoreDirMethods`. Now I am doing that, and renaming
`MixStoreDirMethods` to just `StoreDirConfig` to reduce churn.
This leads to a use-after free, because staticOutputHashes returns a temporary
object that dies before we can do a `return *mOutputHash`.
This is most likely the cause for random failures in Hydra [1].
[1]: https://hydra.nixos.org/build/305091330/nixlog/2
Old code completely ignored query parameters and it seems ok to keep
that behavior. There's a lot of code out there that parses nix code
like nix-output-monitor and it can't parse messages like:
> copying path '/nix/store/wha2hi4yhkjmccqhivxavbfspsg1wrsj-source' from 'https://cache.nixos.org' to 'local://'...
Let's not break these tools without a good reason. This goes in line
with what other code does by ignoring parameters in logs.
The issue is just in detecting the shorthand notations for the store
reference - not in printing the url in logs.
By default the daemon opens a local store with ?path-info-cache-size=0,
so that leads to the erronenous 'local://'.
The problem with old code was that it used getUri for both the `diskCache`
as well as logging. This is really bad because it mixes the textual human
readable representation with the caching.
Also using getUri for the cache key is really problematic for the S3 store,
since it doesn't include the `endpoint` in the cache key, so it's totally broken.
This starts separating the logging / cache concerns by introducing a
`getHumanReadableURI` that should only be used for logging. The caching
logic now instead uses `getReference().render(/*withParams=*/false)` exclusively.
This would need to be fixed in follow-ups, because that's really fragile and
broken for some store types (but it was already broken before).
Move output result filtering logic and assert just into the branch where
it is not obviously a no op / meeting the assertion.
Add a comment too, while we are at it.
Now that `DerivationGoal::checkPathValidity` is legible, we can see that
it only sets `outputKnown`, and doesn't read it. Likewise, with
co-routines, we don't have tiny scopes that make local variables
difficult. Between these two things, we can simply have
`checkPathValidity` return what it finds, rather than mutate some state,
and update everyting to use local variables.
The same transformation could probably be done to the other derivation
goal types (which currently, unfortunately, contain their own
`checkPathValidity`s, though they are diverging, and we hope and believe
that they continue to diverge).
`Store::queryPartialDerivationOutputMap` is nothing but checking
statically-known output paths, and then `Store::queryRealisation`, and
we were doing both of those things already. Inline that and simplify,
again taking advantage of the fact that we only care about one output.
Rather than having store implementations return a free-form URI string,
have them return a `StoreReference`. This reflects that fact that this
method is supposed to invert `resolveStoreConfig`, which goes from a
`StoreReference` to some `StoreConfig` concrete derived class (based on
the registry).
`StoreConfig::getUri` is kept only as a convenience for the common case
that we want to immediately render the `StoreReference`.
A few tests were changed to use `local://` not `local`, since
`StoreReference` does not encode the `local` and `daemon` shorthands
(and instead desugars them to `local://` and `unix://` right away). I
think that is fine. `local` and `daemon` still work as input.
* Adds support for NIX_SSHOPTS
* Properly uses the parsed port from URL (fixes#13337)
* Don't guess the HTTP endpoint, use the response of git-lfs-authenticate
* Add an SSH Git LFS test
* Removed some unused test code
Clang refused to do a narrowing conversion in an initializer list:
```
local-keys.cc:56:90: note: insert an explicit cast to silence this issue
return name + ":" + base64::encode(std::as_bytes(std::span<const unsigned char>{sig, sigLen}));
^~~~~~
static_cast<size_type>( )
```
This addresses several changes from toml11 4.0 bump in
nixpkgs [1].
1. Added more regression tests for timestamp formats.
Special attention needs to be paid to the precision
of the subsecond range for local-time. Prior versions select the closest
(upwards) multiple of 3 with a hard cap of 9 digits.
2. Normalize local datetime and offset datetime to always
use the uppercase separator `T`. This is actually the issue
surfaced in [2]. This canonicalization is basically a requirement
by (a certain reading) of rfc3339 section 5.6 [3].
3. If using toml11 >= 4.0 also keep the old behavior wrt
to the number of digits used for subsecond part of the local-time.
Newer versions cap it at 6 digits unconditionally.
[1]: https://www.github.com/NixOS/nixpkgs/pull/331649
[2]: https://www.github.com/NixOS/nix/issues/11441
[3]: https://datatracker.ietf.org/doc/html/rfc3339
Current test suite doesn't cover the subsecond formatting at
all and toml11 is quite finicky with that. We should at the very
least test its behavior to avoid silent breakages on updates.
Previously `getUri` didn't include store query parameters,
`ssh-ng` didn't include any information at all and the local
store didn't have the path:
```
$ nix store info --store "local?root=/tmp/aaa&require-sigs=false"
Store URL: local
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh-ng://localhost?remote-program=nix-daemon"
Store URL: ssh-ng://
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh://localhost?remote-program=nix-store"
Store URL: ssh://localhost
```
This commit changes this to:
```
$ nix store info --store "local?root=/tmp/aaa&require-sigs=false"
Store URL: local?require-sigs=false&root=/tmp/aaa
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh-ng://localhost?remote-program=nix-daemon"
Store URL: ssh-ng://localhost?remote-program=nix-daemon
Version: 2.31.0
Trusted: 1
$ nix store info --store "ssh://localhost?remote-program=nix-store"
Store URL: ssh://localhost?remote-program=nix-store
```
The clang-analyzer incorrectly flags a use-after-free for GC-managed objects
when used with std::unique_ptr. Since NixRepl inherits from gc, its memory
is properly managed by Boehm GC and this is a false positive.
Added NOLINTNEXTLINE directive to suppress the warning.
Fixes usage of `#` symbol in the reference name.
This also seems to identify several deficiencies in the libgit2 refname
validation code wrt to DEL symbol and a singular `@` symbol [1].
[1]: https://git-scm.com/docs/git-check-ref-format#_description
This change was necessary when we were using `nix flake check` for CI
(see 6db6b269ed). Now this is not really
necessary, because we are running the checks in a much saner way, that
doesn't use up too much memory for evaluation.
Sometime ago we lost the coverage job in the midst of
meson migration. Until we have something like codecov
it'd be very useful to restore this job with the html
reports and historical metrics.
As a bonus we get more coverage metrics by switching to
LLVM tooling from LCOV.
The implicit dependency on refLength (which is the StorePath::HashLen)
is not good. Also the companion tests and benchmarks are already in libstore-tests.
This doesn't seem to be used anywhere at the moment.
It might be used out-of-tree, but this is a small convenience
function that is not worth keeping without in-tree usage.
With this I'm able to do a fresh config + meson test with all dependencies
correctly propagated.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
This was removed in https://github.com/NixOS/nix/pull/11152. However,
we need it for the multi-threaded evaluator, because otherwise Boehm
GC will crash while scanning the thread stack:
#0 GC_push_all_eager (bottom=<optimized out>, top=<optimized out>) at extra/../mark.c:1488
#1 0x00007ffff74691d5 in GC_push_all_stack_sections (lo=<optimized out>, hi=<optimized out>, traced_stack_sect=0x0) at extra/../mark_rts.c:704
#2 GC_push_all_stacks () at extra/../pthread_stop_world.c:876
#3 GC_default_push_other_roots () at extra/../os_dep.c:2893
#4 0x00007ffff746235c in GC_mark_some (cold_gc_frame=0x7ffee8ecaa50 "`\304G\367\377\177") at extra/../mark.c:374
#5 0x00007ffff7465a8d in GC_stopped_mark (stop_func=stop_func@entry=0x7ffff7453c80 <GC_never_stop_func>) at extra/../alloc.c:875
#6 0x00007ffff7466724 in GC_try_to_collect_inner (stop_func=0x7ffff7453c80 <GC_never_stop_func>) at extra/../alloc.c:624
#7 0x00007ffff7466a22 in GC_collect_or_expand (needed_blocks=needed_blocks@entry=1, ignore_off_page=ignore_off_page@entry=0, retry=retry@entry=0) at extra/../alloc.c:1688
#8 0x00007ffff746878f in GC_allocobj (gran=<optimized out>, kind=<optimized out>) at extra/../alloc.c:1798
#9 GC_generic_malloc_inner (lb=<optimized out>, k=k@entry=1) at extra/../malloc.c:193
#10 0x00007ffff746cd40 in GC_generic_malloc_many (lb=<optimized out>, k=<optimized out>, result=<optimized out>) at extra/../mallocx.c:477
#11 0x00007ffff746cf35 in GC_malloc_kind (bytes=120, kind=1) at extra/../thread_local_alloc.c:187
#12 0x00007ffff796ede5 in nix::allocBytes (n=<optimized out>, n=<optimized out>) at ../src/libexpr/include/nix/expr/eval-inline.hh:19
This is because it will use the stack pointer of the coroutine, so it
will scan a region of memory that doesn't exist, e.g.
Stack for thread 0x7ffea4ff96c0 is [0x7ffe80197af0w,0x7ffea4ffa000)
(where 0x7ffe80197af0w is the sp of the coroutine and 0x7ffea4ffa000
is the base of the thread stack).
We don't scan coroutine stacks, because currently they don't have GC
roots (there is no evaluation happening in coroutines). So there is
currently no need to restore the other parts of the original patch,
such as BoehmGCStackAllocator.
I had not wanted to cause unncessary churn before, but now that we've
bitten the bullet with the Big Reformat, I feel it is the right time.
Future readers will appreciate that the declarations and definitions
files are one-to-one as they should be, and `store-api.cc` is good to
shrink in any event.
I don't think there are outstanding PRs changing this code either. (I
had some for a while, but they are all merged.)
- No more private constructor that is kinda weird
- Two new static functions, `baseFromSize` and `baseFromSize`, that do
one thing, and one thing only (simple).
- Two `Hash::parse*` that previously used the private constructor now
can use these two functions directly.
- The remaining `Hash::parseAny*` methods, which are inherently more
complex, are written in terms of a `parseAnyHelper` static function
which is also complex, but keeps the complexity in one spot.
This patch allows users to specify the connection port
in the store URLS like so:
```
nix store info --store "ssh-ng://localhost:22" --json
```
Previously this failed with: `error: failed to start SSH connection to 'localhost:22'`,
because the code did not distinguish the port from the hostname. This
patch remedies that problem by introducing a ParsedURL::Authority type
for working with parsed authority components of URIs.
Now that the URL parsing code is less ad-hoc we can
add more long-awaited fixes for specifying SSH connection
ports in store URIs.
Builds upon the work from bd1d2d1041.
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Prior to this patch, mode 0444 is not updated to 0555 for directories.
That means for instance 0554 is canonicalized, but not 0444.
We don't believe this has any implications for backwards compatibility,
because directories do not have permissions in NAR format and so are
always 0555 after deserialization, and store paths with wrong
permissions can’t be copied to another host.
Co-authored-by: Robert Hensing <robert@roberthensing.nl>
Make it separate from Hash, since other things can be base-encoded too.
This isn't really needed for Nix, but it makes the code easier to read
e.g. for someone reimplementing this stuff in a different language. (Of
course, Base16/Base64 should be gotten off-the-shelf, but now the hash
code, which is more bespoke, is less cluttered with the parts that would
be from some library.)
Many reimplementations of "Nix32" and our hash type already exist, so
this cleanup is coming years too late, but I say better late than never
/ it is always good to nudge the code in the direction of being a
"living spec".
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
Having a State class in the nix namespace is asking
for ODR trouble. This class is already private to the
translation unit, let's move it into an anonymous namespace.
There have been prior concerns about reogranizing the repo, but
this seems like a trivial simplification which will not interfere
with either our packaging or the modular builds in nixpkgs.
This adds the necessary infrastructure to collect
code coverage in CI, which could be useful to look
at munually or track consistently via something like
codecov.
Co-authored-by: Jade Lovelace <lix@jade.fyi>
This keeps things fast by making the function inline, but also prevents
people from having to know about the `0xFF` implementation detail
directly, instead making one go through a `std::optional` (which could be
fused away with a sufficiently smart compiler).
Additionally, the base "nix32" implementation is moved to its own header
file pair, as it is logically distinct and prior to the `Hash` data
type. It would probably be nice to do this with all the hash format
implementations.
Some binary caches (incorrectly) use this header to indicate lack of
compression, inspired by the valid "identity" token in the
"Accept-Encoding" header.
This benchmark should provide a relatively realistic
scenario for reference scanning.
As shown by the following results, reference scanning code
is already plenty fast and is definitely not a bottleneck:
```
BM_RefScanSinkRandom/10000 1672 ns 1682 ns 413354 bytes_per_second=5.53691Gi/s
BM_RefScanSinkRandom/100000 11217 ns 11124 ns 64341 bytes_per_second=8.37231Gi/s
BM_RefScanSinkRandom/1000000 205745 ns 204421 ns 3360 bytes_per_second=4.55591Gi/s
BM_RefScanSinkRandom/5000000 1208407 ns 1201046 ns 597 bytes_per_second=3.87713Gi/s
BM_RefScanSinkRandom/10000000 2534397 ns 2523344 ns 273 bytes_per_second=3.69083Gi/s
```
(Measurements on Ryzen 5900X via `nix build --file ci/gha/tests componentTests.nix-store-tests-run -L`)
The changes include:
* Defining nix32Chars as a constexpr char[].
* Adding a constexpr std::array<unsigned char, 256> (reverseNix32Map) to map characters to their base-32 digit values at compile time.
* Replacing the slow character search loop with a direct lookup using reverseNix32Map.
* Removing std::once_flag/isBase32 logic in references.cc in favor of reverseNix32Map
Signed-off-by: Alexander V. Nikolaev <avn@avnik.info>
This changes our GHA CI and nix-store-tests packaging
to build and run the benchmarks. This does not affect
the default packaging - the overrides apply only for the
GHA CI.
GCC doesn't really benefit as much as Clang does from
using precompiled headers. Another aspect to consider is that
clangd doesn't really like GCC's PCH flags in the compilation database,
so GCC based devshells would continue to work with clangd.
This also has the slight advantage of ensuring that our includes are in
order, since we build with both Clang and GCC.
Instead of parsing a structured attrs at some later point, we parsed it
right away when parsing the A-Term format, and likewise serialize it to
`__json = <JSON dump>` when serializing a derivation to A-Term.
The JSON format can directly contain the JSON structured attrs without
so encoding it, so we just do that.
Add a new setting to warn about path literals that don't start with "." or "/". When enabled,
expressions like `foo/bar` will emit a warning suggesting to use `./foo/bar` instead.
A functional test is included.
The setting defaults to false for backward compatibility but could eventually default to true in
the future.
Closes: #13374
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
All of the existing tests only run on x86_64-linux and
the whole `nixpkgsFor` makes it hard to reuse those for
e.g. running the nixos tests with a sanitizer build of nix.
This rips off the bandaid and removes the `nixpkgsFor` parameter
in favor of a single instance of nixpkgs with a separate `nixComponents`.
This command fetches all inputs of a flake in parallel.
Example runtime for
$ chmod -R u+w /tmp/nix2; rm -rf /tmp/nix2; rm ~/.cache/nix/fetcher-cache-v3.sqlite*; rm -rf ~/.cache/nix/tarball-cache/ ~/.cache/nix/gitv3/; time nix flake prefetch-inputs --store /tmp/nix2 https://api.flakehub.com/f/pinned/informalsystems/cosmos.nix/0.3.0/018ce9ed-d0be-7ce5-81b6-a3c6e3ae1187/source.tar.gz
with http-connections = 1:
real 4m11.859s
user 2m6.931s
sys 0m25.619s
and http-connections = 25 (the default):
real 0m57.146s
user 2m49.506s
sys 0m36.008s
This moves out the checks that get run in GHA CI into ci/gha/tests
folder and splits those into `topLevel` and `componentTests` attributes.
The idea behind this is to make it easier to parametrize tests that can
be run with sanitizers in order to run those as a matrix of jobs. The same
can be said for static builds.
Existing stdenv selection infrastructure via `lib.makeComponents` would
also allow us to switch over to using `clangStdenv` to significantly speed
up pre-merge CI (though the default stdenv would still be used for non-overridable
topLevel checks, like installer artifacts).
Prior patches in 54dc5314e8
and 6db6190002 fixed the default
system double for i686 and ppc/ppc64. This also patch also covers
32 bit arm and mips. ARM cpu names are taken from host_machine.cpu()
for a lack of a better option, but host_machine.cpu_family() is
preferred, since that is supposed to be somewhat standard for cross
files. Endianness is handled correctly by looking at host_machine.endian().
This also updates the documentation to be up to date to how system cpu
is translated from the host_machine specification.
Before this change, if you were cross compiling Nix, then the nix-manual
subproject would never get built. In some situations, it makes sense to
not build the nix-manual subproject when cross compiling. For example,
if the build system is x86_64 and the host system is riscv64, then it
makes sense to not build the manual. Building the manual requires
executing certain build artifacts, and you can’t run x86_64 executables
on riscv64 systems.
That being said, there are some situations where it does make sense to
build the nix-manual subproject when cross compiling. For example, if
the build system is x86_64 and the host system is i686, then it doesn’t
make sense to not build the manual. You can run i686 executables on
x86_64 systems just fine.
This change makes it so that the nix-manual subproject will sometimes
get built when cross compiling. Specifically, the nix-manual subproject
will get built as long as the doc-gen option is enabled and the build
system is capable of running host binaries.
---
The main motivation behind this change is to fix this Nixpkgs issue [1].
Building pkgs.nixStatic counts as cross compiling Nix, and
pkgs.nixStatic is supposed to produce a man output. Building
pkgs.nixStatic currently fails because it isn’t actually producing a man
output. That issue will go away once this commit gets backported to Nix
2.28.x.
[1]: <https://github.com/NixOS/nixpkgs/issues/426410>
SHA-256 is Git's next hash algorithm. The world is still basically stuck
on SHA-1 with git, but shouldn't be. We can at least do our part to get
ready.
On the C++ implementation side, only a little bit of generalization was
needed, and that was fairly straight-forward. The tests (unit and
system) were actually bigger, and care was taken to make sure they were
all cover both algorithms equally.
For regular, non-executable files, there is no str("") between str("regular")
and str("contents"). Note that str("") is exactly 8 zero bytes, while just ""
is actual empty string (0 bytes).
libfetchers uses `git_mempack_write_thin_pack` which was introduced in libgit2-1.9.0
This avoids error like:
../src/libfetchers/git-utils.cc: In member function ‘virtual void nix::GitRepoImpl::flush()’:
../src/libfetchers/git-utils.cc:270:13: error: ‘git_mempack_write_thin_pack’ was not declared in this scope
270 | git_mempack_write_thin_pack(mempack_backend, packBuilder.get())
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
on older libgit2 (like 1.7.2 in Centos Stream 10)
If the reference for git+file is an annotated tag, the revision will
differ than when it's fetched using other fetchers such as `github:`
since Github seems to automatiacally peel to the underlying commit.
Turns out that rev-parse has the capability through it's syntax to
request the underlying commit by "peeling" using the `^{commit}` syntax.
This is safe to apply in all scenarios where the goal is to get an
underlying commit.
fixes#11266
I've missed this while reviewing 6db6190002.
I only built big endian ppc64, so that didn't occur to me.
From meson manual:
> Those porting from autotools should note that Meson does not add
> endianness to the name of the cpu_family. For example, autotools will
> call little endian PPC64 "ppc64le", Meson will not, you must also check
> the .endian() value of the machine for this information.
This code should handle that correctly.
builtins.fetchGit is not using the cached Git directory if
packed-references are used.
This is because the ref file for the fetchGit `refs/heads/master` is
used to check the mtime for whether to cache or not.
Let's at least codify this failure in a test case.
This is a follow-up to 6ec50ba736, which
also almost halves the compile times on clang for subprojects that use PCH.
`-fpch-instantiate-templates` is a clang-only option to force the instantiation
of templates once in the PCH itself, not all of the translation units that
it gets included to. This really cuts down on the overhead from nlohmann::json
and std::format code:
48244 ms: nlohmann::basic_json<>::parse<const char *> (76 times, avg 634 ms)
36193 ms: nlohmann::basic_json<>::basic_json (310 times, avg 116 ms)
28307 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (76 times, avg 372 ms)
20334 ms: nlohmann::detail::parser<nlohmann::basic_json<>, nlohmann::detail::i... (76 times, avg 267 ms)
17387 ms: nlohmann::basic_json<>::json_value::json_value (389 times, avg 44 ms)
16822 ms: std::vformat_to<std::__format::_Sink_iter<char>> (76 times, avg 221 ms)
16771 ms: std::__format::__do_vformat_to<std::__format::_Sink_iter<char>, char... (76 times, avg 220 ms)
12160 ms: std::vformat_to<std::__format::_Sink_iter<wchar_t>> (76 times, avg 160 ms)
12127 ms: std::__format::__do_vformat_to<std::__format::_Sink_iter<wchar_t>, w... (76 times, avg 159 ms)
10397 ms: nlohmann::detail::json_sax_dom_callback_parser<nlohmann::basic_json<... (76 times, avg 136 ms)
9118 ms: nlohmann::basic_json<>::data::data (76 times, avg 119 ms)
Initially done by Jade Lovelace <lix@jade.fyi> in https://gerrit.lix.systems/c/lix/+/1842.
We are doing basically the same, but unconditionally. It would be
a huge pain to add a pch option for all subprojects to just support the
usecase of using clangd in a gcc devshell.
In total, this basically halves the frontend times for nix-util and nix-store
to the point that the most expensive part of the build is linking.
(nix-store before):
```
**** Time summary:
Compilation (77 times):
Parsing (frontend): 243.4 s
Codegen & opts (backend): 140.3 s
```
(nix-store after):
```
**** Time summary:
Compilation (77 times):
Parsing (frontend): 120.2 s
Codegen & opts (backend): 141.2 s
```
This adds a GHA jobs to help analyze build times
and its regressions. It is based on `clangStdenv` with `-ftime-trace`
together with `ClangBuildAnalyzer` to prepare markdown summary for
individual components.
This also has the minor benefit of dogfooding CA and impure derivations.
Users can access these through the `lib.makeComponents` return value,
so it's helpful to briefly explain some of them.
This doesn't replace `meta.description`, but supplements it.
(TODO: improve `meta.description`)
This was carefully refactored by inlining some Nixpkgs definitions
like `generateSplicesForMkScope`, so the memoization properties
should remain the same.
Boost.URL is a significantly more RFC-compliant parser
than what libutil currently has a bundle of incomprehensible
regexes.
One aspect of this change is that RFC4007 ZoneId IPv6 literals
are represented in URIs according to RFC6874 [1].
Previously they were represented naively like so: [fe80::818c:da4d:8975:415c\%enp0s25].
This is not entirely correct, because the percent itself has to be pct-encoded:
> "%" is always treated as
an escape character in a URI, so, according to the established URI
syntax [RFC3986] any occurrences of literal "%" symbols in a URI MUST
be percent-encoded and represented in the form "%25". Thus, the
scoped address fe80::a%en1 would appear in a URI as
http://[fe80::a%25en1].
[1]: https://datatracker.ietf.org/doc/html/rfc6874
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
The myriad of hand-rolled URL parsing and validation code
is a constant source of problems. Regexes are not a great way
of writing parsers and there's a history of getting them wrong.
Boost.URL is a good library we can outsource most of the heavy
lifting to.
* It is tough to contribute to a project that doesn't use a formatter,
* It is extra hard to contribute to a project which has configured the formatter, but ignores it for some files
* Code formatting makes it harder to hide obscure / weird bugs by accident or on purpose,
Let's rip the bandaid off?
Note that PRs currently in flight should be able to be merged relatively easily by applying `clang-format` to their tip prior to merge.
Previously, `nix help shell` failed with "Nix has no subcommand 'shell'" despite `nix shell --help`
working correctly. This happened because the `shell` command is actually an alias for `env shell`,
and the help system wasn't resolving aliases when looking up documentation.
This patch modifies the `showHelp` function to check for and resolve aliases before generating the
manpage name, ensuring that shorthand commands like `shell` get proper help documentation.
Closes: #13431
The default constructor for Attr was not initializing the value pointer,
which could lead to undefined behavior when the uninitialized pointer is
accessed. This was caught by clang-tidy's UninitializedObject check.
This fixes the warning:
1 uninitialized field at the end of the constructor call
[clang-analyzer-optin.cplusplus.UninitializedObject]
The s3.hh public header was incorrectly including store-config-private.hh
instead of the public config.hh. Since NIX_WITH_S3_SUPPORT is defined in
the public config, this caused clang-tidy to report it as undefined.
Move init() call from constructor to openStore() method to avoid calling
virtual methods during object construction. This prevents undefined
behavior when virtual methods are called before the object is fully
constructed.
Move init() call from constructor to openStore() method to avoid calling
virtual methods during object construction. This prevents undefined
behavior when virtual methods are called before the object is fully
constructed.
When `file://` is used accidentally in a flake as the source it is
expected to be a tarball by default.
Add some friendlier error messages to either inform the user this is not
in fact a tarball or if it's a git directory, let them know they can use
`git+file`.
fixes#12935
Users have complained that fetchGit is flaky however the culprit is
likely that `git fetch` was unable itself to download the repository for
whatever reason (i.e. poor network etc..)
Nothing was checking the status of `git fetch` and the error message
that would eventually surface to the users were that the commit was not
found.
Add explicit error checking for status code from `git fetch` and return
a message earlier on to indicate that the failure was from that point.
fixes#10431
We’re already allowing `/tmp` anyway, so this should be harmless,
and it fixes a regression in the default configuration caused by
moving the build directories out of `temp-dir`. (For instance, that
broke the Lix `guessOrInventPath.sockets` test.)
Note that removing `/tmp` breaks quite a few builds, so although it may
be a good idea in general it would require work on the Nixpkgs side.
Fixes: 749afbbe99fd7b45f828b72628252feba9241362
Change-Id: I6a6a69645f429bc50d4cb24283feda3d3091f534
(This is a cherry-pick of commit d1db3e5fa3faa43b3d2f2e2e843e9cfc1e6e1b71)
Lix patch: https://gerrit.lix.systems/c/lix/+/3500
When AWS credentials expired, in some scenarios they led to the
nix process aborting with an error similar to ' Unable to parse ExceptionName: ExpiredToken'.
This change updates the S3 handling code such that those errors are treated like 403s or 404s.
Closes#13459
c39cc00404 has added assertions for
all Value accesses and the following case has started failing with
an `unreachable`:
(/tmp/fun.nix):
```nix
{a}: a
```
```
$ nix eval --impure --expr 'import /tmp/fun.nix {a="a";b="b";}'
```
This would crash:
```
terminating due to unexpected unrecoverable internal error: Unexpected condition in getStorage at ../include/nix/expr/value.hh:844
```
This is not a regression, but rather surfaces an existing problem, which previously
was left undiagnosed. In the case of an import `fun` is the `import` primOp, so that read is invalid
and previously this resulted in an access into an inactive union member, which is UB.
The correct thing to use is `vCur`. Identical problem also affected the case of a missing argument.
Add previously failing test cases to the functional/lang test suite.
Fixes#13448.
Fixes:
[261/394] Linking target src/libexpr/libnixexpr.so
In function ‘copy’,
inlined from ‘__ct ’ at /nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/basic_string.h:688:23,
inlined from ‘operator+’ at /nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/basic_string.h:3735:43,
inlined from ‘operator()’ at ../src/libexpr/primops/fetchClosure.cc:127:58,
inlined from ‘prim_fetchClosure’ at ../src/libexpr/primops/fetchClosure.cc:132:88:
/nix/store/24sdvjs6rfqs69d21gdn437mb3vc0svh-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/char_traits.h:427:56: warning: ‘__builtin_memcpy’ writing 74 bytes into a region of size 16 overflows the destination [-Wstringop-overflow=]
427 | return static_cast<char_type*>(__builtin_memcpy(__s1, __s2, __n));
| ^
../src/libexpr/primops/fetchClosure.cc: In function ‘prim_fetchClosure’:
../src/libexpr/primops/fetchClosure.cc:132:88: note: at offset 16 into destination object ‘<anonymous>’ of size 32
132 | fromPath = state.coerceToStorePath(attr.pos, *attr.value, context, attrHint());
| ^
-p preserves xattrs and acls which can be incompatible between
filesystems
Unfortunately keep -p on darwin because the bsd coreutils do not
support --preserve.
Fixes#13426
This failed on macOS:
nix-store-tests-run> C++ exception with description "../nix_api_store.cc:33: nix_err_code(ctx) != NIX_OK, message: error: getting status of '/nix/var/nix/daemon-socket/socket': Operation not permitted" thrown in the test body.
This changes makes nix detect a machines available cores automatically whenever build-cores is set to 0.
So far, nix simply passed NIX_BUILD_CORES=0 whenever build-cores is set to 0. (only when build-cores is unset it was detecting cores automatically)
The behavior of passing NIX_BUILD_CORES=0 leads to a performance penalty when sourcing nixpkgs' generic builder's `setup.sh`, as setup.sh has to execute `nproc`. This significantly slows down sourcing of setup.sh
The use of R"(...)" added a bunch of unnecessary whitespace, e.g.
error:
Unable to start any build;
either increase '--max-jobs' or enable remote builds.
For more information run 'man nix.conf' and search for '/machines'.
Now we get
error: Unable to start any build; either increase '--max-jobs' or enable remote builds.
For more information run 'man nix.conf' and search for '/machines'.
This shaves off a very significand amount of memory used
for evaluation as well as reduces the GC-managed heap.
Previously the union discriminator (InternalType) was
stored as a separate field in the Value, which takes up
whole 8 bytes due to padding needed for member alignment.
This effectively wasted 7 whole bytes of memory. Instead
of doing that InternalType is instead packed into pointer
alignment niches. As it turns out, there's more than enough
unused bits there for the bit packing to be effective.
See the doxygen comment in the ValueStorage specialization
for more details.
This does not add any performance overhead, even though
we now consistently assert the InternalType in all getters.
This can also be made atomic with a double width compare-and-swap
instruction on x86_64 (CMPXCHG16B instruction) for parallel evaluation.
This factors out most of the value representation into a mixin class.
`finishValue` is now gone for good and replaced with a simple template
function `setStorage` which derives the type information/disriminator from
the type of the argument. Likewise, reading of the value goes through function
template `getStorage`.
An empty type `Null` is introduced to make the bijection InternalType <-> C++ type
complete.
Previous use of symlink_status() always translated into a stat call, leading
to huge performance penalties for by-name-overlay in nixpkgs. The comment
below references the possible caching, but that seemed to be erroneous, since
the correct way to make use of the caching API is by calling a bunch of `is_*`
functions [1]. For example, here's how libstdc++ does that [2], [3].
This translates to great nixpkgs eval performance improvements:
```
Benchmark 1: GC_INITIAL_HEAP_SIZE=4G result/bin/nix-instantiate ../nixpkgs -A hello --readonly-mode
Time (mean ± σ): 186.7 ms ± 6.7 ms [User: 121.3 ms, System: 64.9 ms]
Range (min … max): 179.4 ms … 201.6 ms 16 runs
Benchmark 2: GC_INITIAL_HEAP_SIZE=4G nix-instantiate ../nixpkgs -A hello --readonly-mode
Time (mean ± σ): 230.6 ms ± 5.0 ms [User: 126.9 ms, System: 103.1 ms]
Range (min … max): 225.1 ms … 241.4 ms 13 runs
```
[1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0317r1.html
[2]: 8ea555b7b4/libstdc%2B%2B-v3/include/bits/fs_dir.h (L341-L348)
[3]: 8ea555b7b4/libstdc%2B%2B-v3/include/bits/fs_dir.h (L161-L163)
These wrappers don't seem to be used anywhere in and out of tree.
Also the declaration in the header has an incorrect function name `maybeParseFlake`.
Closes#11948
Fixes
error:
… while processing sandbox path '/private/tmp/nix-shell.0MDgyx/nix-test/ca/repl/store/nix/var/nix/builds/nix-build-simple.drv-65916-3910734210' (/private/tmp/nix-shell.0MDgyx/nix-test/ca/repl/store)
error: 'nix' is too short to be a valid store path
which happened because we were now putting the build directory
underneath the store directory.
If a build directory is accessible to other users it is possible to
smuggle data in and out of build directories. Usually this is only
a build purity problem, but in combination with other issues it can
be used to break out of a build sandbox. to prevent this we default
to using a subdirectory of nixStateDir (which is more restrictive).
(cherry picked from pennae Lix commit 55b416f6897fb0d8a9315a530a9b7f0914458ded)
(store setting done by roberth)
In the case the deletion fails, we should ensure that the temporary
directory cannot be used for nefarious purposes.
Change-Id: I498a2dd0999a74195d13642f44a5de1e69d46120
Signed-off-by: Raito Bezarius <raito@lix.systems>
When calling `_deletePath` with a parent file descriptor, `openat` is
made effective by using relative paths to the directory file descriptor.
To avoid the problem, the signature is changed to resist misuse with an
assert in the prologue of the function.
Change-Id: I6b3fc766bad2afe54dc27d47d1df3873e188de96
Signed-off-by: Raito Bezarius <raito@lix.systems>
This ensures that `passAsFile` data is created inside the expected
temporary build directory by `openat()` from the parent directory file
descriptor.
This avoids a TOCTOU which is part of the attack chain of CVE-????.
Change-Id: Ie5273446c4a19403088d0389ae8e3f473af8879a
Signed-off-by: Raito Bezarius <raito@lix.systems>
`writeFile` lose its `sync` boolean flag to make things simpler.
A new `writeFileAndSync` function is created and all call sites are
converted to it.
Change-Id: Ib871a5283a9c047db1e4fe48a241506e4aab9192
Signed-off-by: Raito Bezarius <raito@lix.systems>
We use it immediately for the build temporary directory.
Change-Id: I180193c63a2b98721f5fb8e542c4e39c099bb947
Signed-off-by: Raito Bezarius <raito@lix.systems>
We now keep around a proper AutoCloseFD around the temporary directory
which we plan to use for openat operations and avoiding the build
directory being swapped out while we are doing something else.
Change-Id: I18d387b0f123ebf2d20c6405cd47ebadc5505f2a
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is useful for certain error recovery paths (no pun intended) that
does not thread through the original path name.
Change-Id: I2d800740cb4f9912e64c923120d3f977c58ccb7e
Signed-off-by: Raito Bezarius <raito@lix.systems>
This is just a code cleanup; it should not be behavior change.
`addWantedOutputs` is removed by introducing `DerivationTrampolineGoal`.
`DerivationGoal` now only tracks a single output, and is back to
tracking a plain store path `drvPath`, not a deriving path one. Its
`addWantedOutputs` method is gone. These changes will allow subsequent
PRs to simplify it greatly.
Because the purpose of each goal is back to being immutable, we can also
once again make `Goal::buildResult` a public field, and get rid of the
`getBuildResult` method. This simplifies things also.
`DerivationTrampolineGoal` is, as the nane is supposed to indicate, a
cheap "trampoline" goal. It takes immutable sets of wanted outputs, and
just kicks of `DerivationGoal`s for them. Since now "actual work" is
done in these goals, it is not wasteful to have separate ones for
separate sets of outputs, even if those outputs (and the derivations
they are from) overlap.
This design is described in more detail in the doc comments on the goal
types, which I've now greatly expanded.
---
This separation of concerns will make it possible for future work on
issues like #11928, and to continue the path of having more goal types,
but each goal type does fewer things (issue #12628).
---
This commit in some sense reverts
f4f28cdd0e, but that one kept around
`addWantedOutputs`. I am quite sure it was having two layers of goals
with `addWantedOutputs` that caused the issues --- restarting logic like
`addWantedOutputs` has is very tempermental! In this version of the
change, we have *zero* layers of `addWantedOutputs` --- no goal type
needs it, or otherwise has a mutable objective --- and so I think this
change is safe.
Co-authored-by: Sergei Zimmerman <145775305+xokdvium@users.noreply.github.com>
Running parallel nix in nix can lead to multiple instances trying to
create the state directories and failing on the `createSymlink` step,
because the link already exists.
`replaceSymlink` is already idempotent, so let's use that.
Resolves#2706
libstdc++'s std::stable_sort and new builtins.sort implementation
special-case ranges with length less than or equal to 16 and delegate
to insertionsort.
Having a larger e2e test would allow catching sort stability issues
at functional level as well.
This prevents C++ level undefined behavior from affecting
the evaluator. Stdlib implementation details should not affect
eval, regardless of the build platform. Even erroneous usage
of `builtins.sort` should not make it possible to crash the
evaluator or produce results that depend on the host platform.
Unlike std::sort and std::stable_sort, this implementation
does not lead to out-of-bounds memory reads or other undefined
behavior when the predicate is not strict weak ordering.
This makes it possible to use this function in libexpr for
builtins.sort, where an incorrectly implemented comparator
in the user nix code currently can crash and burn the evaluator
by invoking C++ UB.
Makes the behavoral change of #13263 without the underlying refactor.
Hopefully this clearly safe from a perf and GC perspective, and will
make it easier to benchmark #13263.
This overload isn't actually necessary anywhere and
doesn't make much sense. The pointers to `Value`s are
themselves const, but the `Value`s are mutable.
A non-const member function implies that the object itself
can be modified but this doesn't make much sense considering
the return type: `Value * const * `, which is a pointer
to a constant array of pointers to mutable values.
It seems obvious that erasing any output paths from pathsInChroot needs
to happen after getPathsInSandbox(), not before.
Signed-off-by: Samuli Thomasson <samuli.thomasson@pm.me>
This adds a meson.format file that mostly mirrors the projects
meson style and a pre-commit hook to enforce this style.
Some low-diff files are formatted.
Seen in https://github.com/DeterminateSystems/nix-src/actions/runs/15590867877/job/43909540271:
nix-functional-tests> grep: repl_output: No such file or directory
nix-functional-tests> +(repl.sh:174) cat repl_output
This is because there is a small possibility that the `nix repl` child
process hasn't created `repl_output` yet. So make sure it exists.
Add helpful context when opening the Nix database lock fails due to
permission errors. Instead of just showing "Permission denied", now
provides guidance about possible causes:
- Running as non-root in a single-user Nix installation
- Nix daemon may have crashed
Invoking `:ll` will start a pager with all variables which have just
been loaded by `:lf`, `:l`, or by a flake provided to `nix repl` as an
argument.
https://github.com/NixOS/nix/issues/11404
When we run `nix repl nixpkgs` we get "Added 6 variables". This is not
useful as it doesn't tell us which variables the flake has exported to
our global repl scope.
This patch prints the name of each variable that was just loaded. We
currently cap printing to 20 variables in order to avoid excessive
prints.
https://github.com/NixOS/nix/issues/11404
`getPrimOp` function was basically identical to existing
`Value::primOpAppPrimOp` modulo some trivial differences.
Makes sense to reuse existing code for that.
There's actually no mutation happening so there's no point in using
a mutable shared_ptr. Furthermore, this makes it much more evident to
the reader that no actual mutation (especially in multithreaded case)
is happening.
Also get rid of redundant constructor that isn't actually used anywhere
other than `Pos::operator std::shared_ptr<Pos>` which just passes in &*this,
(identical to just `this`), which can't be nullptr.
Judging by the comment for `makeEmptySourceAccessor` the prefix has
to be empty:
> Return a source accessor that contains only an empty root directory.
Fixes#13295.
Since the migration to meson precompiled-headers.h isn't actually used
anymore and is just confusing. Meson can't handle shared pch across
subprojects [1] and without that there's no performances benefit of PCH
at all. Also rolling our own support for that isn't trivial. See [2] for
an example of how that would look like.
[1]: https://github.com/mesonbuild/meson/issues/4350
[2]: 22bc8b6473/plugins/meson.build
Clang doesn't like the double indent that is needed for the `if...else`
that is CPP'd away. Adding braces is fine in the `if...else...` case,
and fine as a naked block in the CPP'd away case, and properly-indented
both ways.
These tests have been collected from nixpkgs f870c6ccc8951fc48aeb293cf3e98ade6ac42668
usage of builtins.match for x86_64-linux eval system. At most 2 matching and
non-matching cases are included for each encountered regex. This should
hopefully add more confidence when possibly trying to switch the regex implementation
in the future.
This is the utility changes from #9968, which were easier to rebase
first.
I (@Ericson2314) didn't write this code; I just rebased it.
Co-Authored-By: Artemis Tosini <me@artem.ist>
Co-Authored-By: Audrey Dutcher <audrey@rhelmot.io>
Allows to copy the archive to a remote host and not get
error: cannot add path '/nix/store/01x2k4nlxcpyd85nnr0b9gm89rm8ff4x-source' because it lacks a signature by a trusted key
Before we got something like this but only inside the VM test:
vm-test-run-functional-tests-on-nixos_user> machine # fetchGit.sh: line 286: unexpected EOF while looking for matching `)'
We now try to do not too much in a single line, so that the bash parser does not get confused.
This also seems more readable and better quoted.
This makes the profiler much more useful by actually distiguishing
different derivations being evaluated. This does make the implementation
a bit more convoluted, but I think it's worth it.
Sometimes the profiler might want to do evaluation (e.g. for getting
derivation names). This is not ideal, but is really necessary
to make the profiler stack traces useful for end users.
...and also NIX_STATE_HOME in nix-profile.fish. This is directly
translated from the bash scripts and makes the fish scripts equivalent
in functionality to the bash scripts.
Note that nix-profile.fish checks for NIX_STATE_HOME and
nix-profile-daemon.fish does not, so the two scripts are no longer
identical.
This was discussed in https://github.com/NixOS/nix/issues/8034. I
personally like `PRJ_ROOT`, which hopefully avoids some ambiguity
around with subflakes.
I only implemented this for `nix fmt` because it doesn't let you point
at a flake not on your filesystem.
macOS compilation fixes
This method does *not* create a new type of goal. We instead just make
`DerivationGoal` more sophisticated, which is much easier to do now that
`DerivationBuildingGoal` has been split from it (and so many fields are
gone, or or local variables instead).
This avoids the need for a secondarily trampoline goal that interacted
poorly with `addWantedOutputs`. That, I hope, will mean the bugs from
before do not reappear.
There may in fact be a reason to introduce such a trampoline in the
future, but it would only happen in conjunction with getting rid of
`addWantedOutputs`.
Restores the functionality (and tests) that was reverted in
f4f28cdd0e.
This patch adds support for a native stack sampling
profiler to the evaluator, which saves a collapsed stack
profile information to a configurable location.
Introduced options (in `EvalSettings`):
- `eval-profile-file` - path to the collected profile file.
- `eval-profiler-frequency` - sampling frequency.
- `eval-profiler` - enumeration option for enabling the profiler.
Currently only `flamegraph` is supported, but having this an
enumeration rather than a boolean switch leaves the door open
for other profiler variants (e.g. tracy).
Profile includes the following information on best-effort basis (e.g. some lambdas might
have an undefined name). Callstack information contains:
- Call site location (where the function gets called).
- Primop/lambda name of the function being called.
- Functors/partial applications don't have a name attached to them unlike special-cased primops and lambads.
For cases where callsite location isn't available we have to resort to providing
the location where the lambda itself is defined. This removes some of the confusing
`«none»:0` locations in the profile from previous attempts.
Example usage with piping directly into zstd for compression:
```
nix eval --no-eval-cache nixpkgs#nixosTests.gnome \
--eval-profiler flamegraph \
--eval-profile-file >(zstd -of nix.profile.zstd)
```
Co-authored-by: Jörg Thalheim <joerg@thalheim.io>
Commit b36637c8f7 set
`__ETC_PROFILE_NIX_SOURCED` globally, but this is not enough to prevent
the script from being run again by child shells, because the
variable was not exported and thus not inherited by any child process.
Exporting the variable also agrees with the bash scripts.
Notably, the old behavior broke `nix develop -c fish` in some cases,
because the profile bin directory got prepended to the path, causing
binaries from the profile to override binareis from the devshell.
Try to make `DerivationGoal` care less whether we're working from an
in-memory derivation or not.
It's a clean-up in its own right, but it will also help with other
cleanups under the umbrella of #12628.
Now, each class provides the initial coroutine by value. This avoids
some sketchy virtual function stuff, and will also be further put to
good use in the next commit.
The "derivation" is one of the key concepts and captures the most distinctive aspect of Nix:
that we work with a certain type data (linked files) in a certain manner (using pure functions).
Here we finally arrange all the important pieces to show how they belong
together, while referring to the respective reference documentation for details.
This change also unbreaks downstream links to `//glossary#gloss-derivation`,
which had been broken by removing the glossary entry
2025-04-11 19:47:28 +02:00
1690 changed files with 84037 additions and 44493 deletions
# 2. Store your dockerhub username as DOCKERHUB_USERNAME in "Repository secrets" of your fork repository settings (https://github.com/$githubuser/nix/settings/secrets/actions)
# 3. Create an access token in https://hub.docker.com/settings/security and store it as DOCKERHUB_TOKEN in "Repository secrets" of your fork
check_secrets:
permissions:
contents:none
name:Check Docker secrets present for installer tests
synopsis: "New setting `ignore-gc-delete-failure` for local stores"
prs: [15054]
---
A new local store setting [`ignore-gc-delete-failure`](@docroot@/store/types/local-store.md#store-local-store-ignore-gc-delete-failure) has been added.
When enabled, garbage collection will log warnings instead of failing when it cannot delete store paths.
This is useful when running Nix as an unprivileged user that may not have write access to all paths in the store.
This setting is experimental and requires the [`local-overlay-store`](@docroot@/development/experimental-features.md#xp-feature-local-overlay-store) experimental feature.
synopsis: Revert incomplete closure mixed download and build feature
issues: [77, 12628]
prs: [13176]
---
Since Nix 1.3 (299141ecbd08bae17013226dbeae71e842b4fdd7 in 2013) Nix has attempted to mix together upstream fresh builds and downstream substitutions when remote substuters contain an "incomplete closure" (have some store objects, but not the store objects they reference).
This feature is now removed.
Worst case, removing this feature could cause more building downstream, but it should not cause outright failures, since this is not happening for opaque store objects that we don't know how to build if we decide not to substitute.
In practice, however, we doubt even the more building is very likely to happen.
Remote stores that are missing dependencies in arbitrary ways (e.g. corruption) don't seem to be very common.
On the contrary, when remote stores fail to implement the [closure property](@docroot@/store/store-object.md#closure-property), it is usually an *intentional* choice on the part of the remote store, because it wishes to serve as an "overlay" store over another store, such as `https://cache.nixos.org`.
If an "incomplete closure" is encountered in that situation, the right fix is not to do some sort of "franken-building" as this feature implemented, but instead to make sure both substituters are enabled in the settings.
(In the future, we should make it easier for remote stores to indicate this to clients, to catch settings that won't work in general before a missing dependency is actually encountered.)
synopsis: New command `nix store roots-daemon` for serving GC roots
prs: [15143]
---
New command [`nix store roots-daemon`](@docroot@/command-ref/new-cli/nix3-store-roots-daemon.md) runs a daemon that serves garbage collector roots over a Unix domain socket.
It enables the garbage collector to discover runtime roots when the main Nix daemon doesn't have `CAP_SYS_PTRACE` capability and therefore cannot scan `/proc`.
The garbage collector can be configured to use this daemon via the [`use-roots-daemon`](@docroot@/store/types/local-store.md#store-experimental-option-use-roots-daemon) store setting.
This feature requires the [`local-overlay-store` experimental feature](@docroot@/development/experimental-features.md#xp-feature-local-overlay-store).
Channels are a mechanism for referencing remote Nix expressions and conveniently retrieving their latest version.
The moving parts of channels are:
- The official channels listed at <https://nixos.org/channels>
- The official channels listed at <https://channels.nixos.org>
- The user-specific list of [subscribed channels](#subscribed-channels)
- The [downloaded channel contents](#channels)
- The [Nix expression search path](@docroot@/command-ref/conf-file.md#conf-nix-path), set with the [`-I` option](#opt-i) or the [`NIX_PATH` environment variable](#env-NIX_PATH)
- The [Nix expression search path](@docroot@/command-ref/conf-file.md#conf-nix-path), set with the [`-I` option](#opt-I) or the [`NIX_PATH` environment variable](#env-NIX_PATH)
> **Note**
>
@@ -53,6 +53,11 @@ This command has the following operations:
Download the Nix expressions of subscribed channels and create a new generation.
Update all channels if none is specified, and only those included in *names* otherwise.
> **Note**
>
> Downloaded channel contents are cached.
> Use `--tarball-ttl` or the [`tarball-ttl` configuration option](@docroot@/command-ref/conf-file.md#conf-tarball-ttl) to change the validity period of cached downloads.
-`--list-generations`
Prints a list of all the current existing generations for the
@@ -83,9 +88,9 @@ This command has the following operations:
Subscribe to the Nixpkgs channel and run `hello` from the GNU Hello package:
This guide explains how to build and run performance benchmarks in the Nix codebase.
## Overview
Nix uses the [Google Benchmark](https://github.com/google/benchmark) framework for performance testing. Benchmarks help measure and track the performance of critical operations like derivation parsing.
## Building Benchmarks
Benchmarks are disabled by default and must be explicitly enabled during the build configuration. For accurate results, use a debug-optimized release build.
### Development Environment Setup
First, enter the development shell which includes the necessary dependencies:
```bash
nix develop .#native-ccacheStdenv
```
### Configure Build with Benchmarks
From the project root, configure the build with benchmarks enabled and optimization:
This section provides some notes on how to start hacking on Nix.
To get the latest version of Nix from GitHub:
> **Note**
>
> When checking out the repo on Windows, make sure you have the git setting `core.symlinks` enabled, before cloning, as there are symlinks in the repo.
```console
$ git clone https://github.com/NixOS/nix.git
$cd nix
@@ -23,7 +27,7 @@ $ nix-shell
To get a shell with one of the other [supported compilation environments](#compilation-environments):
@@ -252,7 +260,7 @@ You can use any of the other supported environments in place of `nix-cli-ccacheS
## Editor integration
The `clangd` LSP server is installed by default on the `clang`-based `devShell`s.
See [supported compilation environments](#compilation-environments) and instructions how to set up a shell [with flakes](#nix-with-flakes) or in [classic Nix](#classic-nix).
See [supported compilation environments](#compilation-environments) and instructions how to set up a shell [with flakes](#building-nix-with-flakes) or in [classic Nix](#building-nix).
To use the LSP with your editor, you will want a `compile_commands.json` file telling `clangd` how we are compiling the code.
Meson's configure always produces this inside the build directory.
@@ -6,16 +6,9 @@ Additionally, see [Testing Nix](./testing.md) for further instructions on how to
## Building Nix with Debug Symbols
In the development shell, set the`mesonBuildType`environment variable to `debug` before configuring the build:
In the development shell, `mesonBuildType`is set automatically to `debugoptimized`. This builds Nix with debug symbols, which are essential for effective debugging.
```console
[nix-shell]$ export mesonBuildType=debugoptimized
```
Then, proceed to build Nix as described in [Building Nix](./building.md).
This will build Nix with debug symbols, which are essential for effective debugging.
It is also possible to build without debugging for faster build:
It is also possible to build without optimization for faster build:
See [functional characterisation testing](#characterisation-testing-functional) for a broader discussion of characterisation testing.
@@ -137,6 +137,12 @@ $ _NIX_TEST_ACCEPT=1 meson test nix-store-tests -v
will regenerate the "golden master" expected result for the `libnixstore` characterisation tests.
The characterisation tests will mark themselves "skipped" since they regenerated the expected result instead of actually testing anything.
### JSON Schema testing
In `doc/manual/source/protocols/json/` we have a number of manual pages generated from [JSON Schema](https://json-schema.org/).
That JSON schema is tested against the JSON file test data used in [characterisation tests](#characterisation-testing-unit ) for JSON (de)serialization, in `src/json-schema-checks`.
Between the JSON (de)serialization testing, and this testing of the same data against the schema, we make sure that the manual, the implementation, and a machine-readable schema are are all in sync.
### Unit test support libraries
There are headers and code which are not just used to test the library in question, but also downstream libraries.
The industry term for storage and retrieval systems using [content addressing](#gloss-content-address). A Nix store also has [input addressing](#gloss-input-addressed-store-object), and metadata.
- [derivation]{#gloss-derivation}
A derivation can be thought of as a [pure function](https://en.wikipedia.org/wiki/Pure_function) that produces new [store objects][store object] from existing store objects.
Derivations are implemented as [operating system processes that run in a sandbox](@docroot@/store/building.md#builder-execution).
This sandbox by default only allows reading from store objects specified as inputs, and only allows writing to designated [outputs][output] to be [captured as store objects](@docroot@/store/building.md#processing-outputs).
A derivation is typically specified as a [derivation expression] in the [Nix language], and [instantiated][instantiate] to a [store derivation].
There are multiple ways of obtaining store objects from store derivatons, collectively called [realisation][realise].
[derivation]: #gloss-derivation
- [store derivation]{#gloss-store-derivation}
A single build task.
A [derivation] represented as a [store object].
See [Store Derivation](@docroot@/store/derivation/index.md#store-derivation) for details.
See [Store Path](@docroot@/store/store-path.md) for details.
@@ -198,7 +208,7 @@
- [impure derivation]{#gloss-impure-derivation}
[An experimental feature](#@docroot@/development/experimental-features.md#xp-feature-impure-derivations) that allows derivations to be explicitly marked as impure,
[An experimental feature](@docroot@/development/experimental-features.md#xp-feature-impure-derivations) that allows derivations to be explicitly marked as impure,
so that they are always rebuilt, and their outputs not reused by subsequent calls to realise them.
- [Nix database]{#gloss-nix-database}
@@ -269,7 +279,7 @@
See [References](@docroot@/store/store-object.md#references) for details.
- [referrer]{#gloss-reference}
- [referrer]{#gloss-referrer}
A reversed edge from one [store object] to another.
@@ -357,8 +367,8 @@
Nix represents files as [file system objects][file system object], and how they belong together is encoded as [references][reference] between [store objects][store object] that contain these file system objects.
The [Nix language] allows denoting packages in terms of [attribute sets](@docroot@/language/types.md#attribute-set) containing:
- attributes that refer to the files of a package, typically in the form of [derivation outputs](#output),
The [Nix language] allows denoting packages in terms of [attribute sets](@docroot@/language/types.md#type-attrs) containing:
- attributes that refer to the files of a package, typically in the form of [derivation outputs](#gloss-output),
- attributes with metadata, such as information about how the package is supposed to be used.
The exact shape of these attribute sets is up to convention.
If a derivation has the `requiredSystemFeatures` attribute, then Nix will only build it on a machine that has the corresponding features set in its [`system-features` configuration](@docroot@/command-ref/conf-file.md#conf-system-features).
For example, setting
@@ -345,12 +333,12 @@ Here is more information on the `output*` attributes, and what values they may b
`outputHashAlgo` can only be `null` when `outputHash` follows the SRI format, because in that case the choice of hash algorithm is determined by `outputHash`.
This will specify the output hash of the single output of a [fixed-output derivation].
The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The "nix32" encoding is an adaptation of base-32 encoding.
The ["nix32" encoding](@docroot@/protocols/nix32.md) is Nix's variant of base-32 encoding.
@@ -74,4 +74,48 @@ in f { x = throw "error"; y = throw "error"; }
=>"ok"
```
## Evaluation order
The order in which expressions are evaluated is generally unspecified, because it does not affect successful evaluation outcomes.
This allows more freedom for the evaluator to evolve and to evaluate efficiently.
Data dependencies naturally impose some ordering constraints: a value cannot be used before it is computed.
Beyond these constraints, the evaluator is free to choose any order.
The order in which side effects such as [`builtins.trace`](@docroot@/language/builtins.md#builtins-trace) output occurs is not defined, but may be expected to follow data dependencies. <!-- we may want to be more specific about this. -->
In a lazy language, evaluation order is often opposite to expectations from strict languages.
For example, in `let wrap = x: { wrapped = x; }; in wrap (1 + 2)`, the function body produces a result (`{ wrapped = ...; }`) *before* evaluating `x`.
## Infinite recursion and stack overflow
During evaluation, two types of errors can occur when expressions reference themselves or call functions too deeply:
### Infinite recursion
This error occurs when a value depends on itself through a cycle, making it impossible to compute.
```nix
letx=x;inx
=>error:infiniterecursionencountered
```
Infinite recursion happens at the value level when evaluating an expression requires evaluating the same expression again.
Despite the name, infinite recursion is cheap to compute and does not involve a stack overflow.
The cycle is finite and fairly easy to detect.
### Stack overflow
This error occurs when the call depth exceeds the maximum allowed limit.
```nix
letf=x:f(x+1);
inf0
=>error:stackoverflow;max-call-depthexceeded
```
Stack overflow happens when too many function calls are nested without returning.
The maximum call depth is controlled by the [`max-call-depth` setting](@docroot@/command-ref/conf-file.md#conf-max-call-depth).
The Nix language is designed for conveniently creating and composing *derivations*– precise descriptions of how contents of existing files are used to derive new files.
The Nix language is designed for conveniently creating and composing [derivations](@docroot@/glossary.md#gloss-derivation)– precise descriptions of how contents of existing files are used to derive new files.
> **Tip**
>
@@ -137,7 +137,7 @@ This is an incomplete overview of language features, by example.
@@ -162,6 +162,9 @@ Update [attribute set] *attrset1* with names and values from *attrset2*.
The returned attribute set will have all of the attributes in *attrset1* and *attrset2*.
If an attribute name is present in both, the attribute value from the latter is taken.
This operator is [strict](@docroot@/language/evaluation.md#strictness) in both *attrset1* and *attrset2*.
That means that both arguments are evaluated to [weak head normal form](@docroot@/language/evaluation.md#values), so the attribute sets themselves are evaluated, but their attribute values are not evaluated.
[Update]: #update
## Comparison
@@ -185,18 +188,95 @@ All comparison operators are implemented in terms of `<`, and the following equi
## Equality
- [Attribute sets][attribute set] and [lists][list] are compared recursively, and therefore are fully evaluated.
-Comparison of [functions][function] always returns `false`.
- [Attribute sets][attribute set] are compared first by attribute names and then by items until a difference is found.
-[Lists][list] are compared first by length and then by items until a difference is found.
- Comparison of distinct [functions][function] returns `false`, but identical functions may be subject to [value identity optimization](#value-identity-optimization).
- Numbers are type-compatible, see [arithmetic] operators.
- Floating point numbers only differ up to a limited precision.
The `==` operator is [strict](@docroot@/language/evaluation.md#strictness) in both arguments; when comparing composite types ([attribute sets][attribute set] and [lists][list]), it is partially strict in their contained values: they are evaluated until a difference is found. <!-- this is woefully underspecified, affecting which expressions evaluate correctly; not just "ordering" or error messages. -->
### Value identity optimization
Nix performs equality comparisons of nested values by pointer equality or more abstractly, _identity_.
Nix semantics ideally do not assign a unique identity to values as they are created, but equality is an exception to this rule.
The disputable benefit of this is that it is more efficient, and it allows cyclical structures to be compared, e.g. `let x = { x = x; }; in x == x` evaluates to `true`.
However, as a consequence, it makes a function equal to itself when the comparison is made in a list or attribute set, in contradiction to a simple direct comparison.
[function]: ./syntax.md#functions
[Equality]: #equality
## Logical conjunction
> **Syntax**
>
> *bool1* `&&` *bool2*
Logical AND. Equivalent to `if`*bool1*`then`*bool2*`else false`.
This operator is [strict](@docroot@/language/evaluation.md#strictness) in *bool1*, but only evaluates *bool2* if *bool1* is `true`.
> **Example**
>
> ```nix
> true && false
> => false
>
> false && throw "never evaluated"
> => false
> ```
[Logical conjunction]: #logical-conjunction
## Logical disjunction
> **Syntax**
>
> *bool1* `||` *bool2*
Logical OR. Equivalent to `if`*bool1*`then true``else`*bool2*.
This operator is [strict](@docroot@/language/evaluation.md#strictness) in *bool1*, but only evaluates *bool2* if *bool1* is `false`.
> **Example**
>
> ```nix
> true || false
> => true
>
> true || throw "never evaluated"
> => true
> ```
[Logical disjunction]: #logical-disjunction
### Precedence and disjunctive normal form
The precedence of `&&` and `||` aligns with disjunctive normal form.
Without parentheses, an expression describes multiple "permissible situations" (connected by `||`), where each situation consists of multiple simultaneous conditions (connected by `&&`).
For example, `A || B && C || D && E` is parsed as `A || (B && C) || (D && E)`, describing three permissible situations: A holds, or both B and C hold, or both D and E hold.
## Logical implication
Equivalent to `!`*b1* `||`*b2*.
> **Syntax**
>
> *bool1* `->` *bool2*
Logical implication. Equivalent to `!`*bool1* `||`*bool2* (or `if`*bool1*`then`*bool2*`else true`).
This operator is [strict](@docroot@/language/evaluation.md#strictness) in *bool1*, but only evaluates *bool2* if *bool1* is `true`.
In this form, the attribute set between the `{``}` is recursive.
One of the attributes must have the special name `body`,
which is the result of the expression.
Example:
```nix
let{
foo=bar;
bar="baz";
body=foo;
}
```
This evaluates to "baz".
## Inheriting attributes
When defining an [attribute set](./types.md#attribute-set) or in a [let-expression](#let-expressions) it is often convenient to copy variables from the surrounding lexical scope (e.g., when you want to propagate attributes).
When defining an [attribute set](./types.md#type-attrs) or in a [let-expression](#let-expressions) it is often convenient to copy variables from the surrounding lexical scope (e.g., when you want to propagate attributes).
This can be shortened using the `inherit` keyword.
To check whether it works, try the following on the client:
To check whether it works, try fetching the [`nix-cache-info`](@docroot@/protocols/nix-cache-info.md) file on the client:
```console
$ curl http://avalon:8080/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
```
which should print something like:
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
When writing to a binary cache (e.g., with [`nix copy`](@docroot@/command-ref/new-cli/nix3-copy.md)), Nix creates [`nix-cache-info`](@docroot@/protocols/nix-cache-info.md) automatically if it doesn't exist.
On the client side, you can tell Nix to use your binary cache using
For historical reasons, [store derivations][store derivation] are stored on-disk in [ATerm](https://homepages.cwi.nl/~daybuild/daily-books/technology/aterm-guide/aterm-guide.html) format.
For historical reasons, [store derivations][store derivation] are stored on-disk in "Annotated Term" (ATerm) format
This is used when calculating the store paths of the derivation's outputs.
*`outputs`:
Information about the output paths of the derivation.
This is a JSON object with one member per output, where the key is the output name and the value is a JSON object with these fields:
*`path`:
The output path, if it is known in advanced.
Otherwise, `null`.
*`method`:
For an output which will be [content addressed], a string representing the [method](@docroot@/store/store-object/content-address.md) of content addressing that is chosen.
[JSON Schema for Derivation Options v1](schema/derivation-options-v1.json)
-->
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
