Compare commits

..
Author SHA1 Message Date
Sergei Zimmerman 2ded675e56 treewide: Make exceptions cloneable
This is needed to make it possible to store exceptions in failed values
with each new rethrow getting a fresh copy of the exception object.
2026-02-18 19:45:49 +03:00
John Ericson 663db5b48b Merge pull request #15278 from puffnfresh/windows/bar-log-format
Windows: don't use bar log format
2026-02-18 05:14:27 +00:00
Brian McKenna c486e78235 Windows: don't use bar log format
Relies on terminal features that don't always work on Windows.
2026-02-18 14:35:35 +11:00
John Ericson 4fff871383 Merge pull request #15274 from obsidiansystems/tryToBuild-raii
libstore: refactor `tryToBuild` with coroutine lambdas and RAII
2026-02-17 22:10:42 +00:00
Amaan Qureshi b9acea908e libstore: refactor tryToBuild with coroutine lambdas and RAII
`tryToBuild` threaded a single `PathLocks outputLocks` by reference
across all build phases and managed a `std::unique_ptr<Activity> actLock`
with explicit `if (!actLock)` guards and `.reset()` calls around the hook
retry loop. This commit introduces coroutine lambdas for the three phases:
`tryHookLoop` owns a `PathLocks` in a scoped block for the first attempt
and per-iteration in the retry loop, `tryBuildLocally` acquires its own
`PathLocks`, and the hook-wait `Activity` is a stack variable scoped to
the postpone block.
2026-02-17 16:23:44 -05:00
John Ericson c3f0670b4e Merge pull request #15266 from obsidiansystems/fix-maxjobs-error
libstore: structured diagnostics for local build rejection
2026-02-17 18:39:58 +00:00
Amaan Qureshi 7cd4359a8b libstore: structured diagnostics for local build rejection
When `max-jobs = 0` and no remote builders are available, Nix reported
"required system or feature not available" even though the system and
features matched fine. The `canBuildLocally` lambda returned a plain
`bool`, conflating a configuration knob (`max-jobs = 0`) with actual
incompatibility (wrong platform, missing features). It also short-circuited
on the first failing check, so a user with both a platform mismatch and
missing features would only see one of the two.

This commit replaces the bool with a `LocalBuildRejection` struct whose
`WrongLocalStore` variant collects all applicable failures into
`badPlatform`, `missingFeatures`, and an orthogonal `maxJobsZero` flag.
Platform mismatch and missing features now produce separate error
paragraphs, and all applicable reasons appear in a single message.

The local-build capability check also now returns
`std::variant<LocalBuildCapability, LocalBuildRejection>`, bundling
the `LocalStore &` and optional `ExternalBuilder *` together.
2026-02-17 12:54:24 -05:00
John Ericson 6e725093e6 Merge pull request #15143 from obsidiansystems/rootless-daemon-minimal
Support garbage collection in external daemon
2026-02-17 16:53:06 +00:00
Artemis Tosini 96fef69755 libstore: support searching for roots from an external daemon
This comes in two parts: a `nix store roots-daemon` command that
can run as root and list runtime roots,
and client logic to find runtime roots for a `LocalStore` by connecting
to that daemon.

This may be useful with an unprivileged nix daemon, as it would
otherwise be unable to find runtime roots from process open files
and maps.
2026-02-17 10:42:04 -05:00
John Ericson 16b0bb7548 Merge pull request #15270 from NixOS/inline-lookup-var
libexpr: Make sure `EvalState::lookupVar` is inlined
2026-02-17 15:12:00 +00:00
John Ericson ebcd31e434 Merge pull request #15271 from NixOS/faster-type-internal-type
libexpr: Optimise `Value::type()`, `ValueStorage::getInternalType()`
2026-02-17 15:11:23 +00:00
John Ericson f940ab5146 Merge pull request #15265 from xokdvium/libgit2-error
libfetchers/git-utils: Add GitError class for deduplicating error…
2026-02-17 15:06:31 +00:00
Sergei Zimmerman 3df91bea62 libexpr: Optimise Value::type(), ValueStorage::getInternalType()
Using nix::unreachable() in getInternalType() and type() turns
out to be quite expensive and prevents inlining. Also Value::type
got compiled to a jump table which has a high overhead from indirect
jumps. Using an explicit lookup table turns out to be more efficient.

This does mean that we lose out on nice diagnostics from nix::unreachable
calls, but this code is probably one of the hottests functions in the whole
evaluator, so I think the tradeoff is worth it. The nixUnreachableWhenHardened
boils down to nix::unreachable when UBSan is enabled so we still have good
coverage there.
2026-02-17 16:50:07 +03:00
Sergei Zimmerman aaabe82483 libexpr: Make sure EvalState::lookupVar is inlined
This makes sure that ExprVar::eval inlines lookupVar call. In practice
this seems to reduce instruction count by ~2%, though it doesn't have
a statistically significant impact on the wall time.
2026-02-17 15:32:26 +03:00
Sergei Zimmerman a81f83604b libexpr: Add marker values to InternalType enum
This reduces the churn when changing up the order of
values in a follow-up commit. This should have been done
from the start ideally to improve readability.
2026-02-17 13:32:45 +03:00
Sergei Zimmerman c1bfa30303 libfetchers/git-utils: Add GitError class for deduplicating error message printing
Consolidates all the error message formatting in one place. It was very weird
and tiring to remember to call git_error_last() in all the places.
2026-02-17 12:18:37 +03:00
John Ericson 509694d5f0 Merge pull request #15267 from obsidiansystems/fix-external-builders-path
tests: quote `PATH` in external-builders test heredoc
2026-02-17 05:53:17 +00:00
Amaan Qureshi 0b7629da08 tests: quote PATH in external-builders test heredoc
The external-builders test expands `$PATH` into a heredoc without quotes,
so any `PATH` entry containing spaces causes bash to parse the line as a
command instead of an assignment, failing the test.
2026-02-16 23:20:10 -05:00
Sergei Zimmerman e7e5eaaa37 Merge pull request #15255 from obsidiansystems/fix-repl-tab-crash
repl: catch all errors during tab completion
2026-02-16 21:58:22 +00:00
Jörg Thalheim 974545290e Merge pull request #15252 from obsidiansystems/fix-docker-compression
upload-release: disable containerd image store to preserve gzip layer compression
2026-02-16 21:26:31 +00:00
Amaan Qureshi be6e72f11b repl: prevent exceptions from escaping editline callbacks
The tab completion handler in `completePrefix` only caught `ParseError`,
`EvalError`, `BadURL`, and `FileNotFound`. Other error types like
`JSONParseError` (which derives from `Error`, not `EvalError`) escaped
the catch block and propagated through editline's C code as undefined
behavior, crashing the REPL. This happened when tab-completing
expressions like `(builtins.fromJSON "invalid").` where evaluation
throws a non-`EvalError` exception.

This commit marks `completionCallback` and `listPossibleCallback` as
`noexcept` with function-try-blocks that catch all exceptions at the
C/C++ boundary, preventing any exception from reaching editline.

Fixes #15133.
2026-02-16 16:02:37 -05:00
Sergei Zimmerman 27782fcc42 Merge pull request #15253 from obsidiansystems/fix-url-assertion
libflake: fix assertion crash when malformed URL falls through to path scheme
2026-02-16 20:49:49 +00:00
John Ericson 06d4d5779f Merge pull request #15251 from obsidiansystems/file-system-at
Split `file-system-at.{cc,hh}` from `file-descriptor.{cc,hh}`
2026-02-16 20:10:28 +00:00
Amaan Qureshi a32cd16f64 libflake: fix assertion crash when malformed URL falls through to path scheme
When a URL like `github:nixos/nixpkgs/nixpkgs.git?ref=<hash>` (using
`ref` instead of `rev`) failed the github input scheme, it fell
through to `parsePathFlakeRefWithFragment` which constructed a `path:`
`ParsedURL` with an empty authority but a relative path. This violated
RFC 3986 section 3.3 (authority present requires path starting with
`/`), causing an assertion failure in `renderAuthorityAndPath` when
`PathInputScheme` tried to format the URL for an error message.

This commit only sets the authority on absolute paths. Relative paths
get `std::nullopt` for authority, which is the correct representation
per the URL spec.

Fixes #15196. Fixes #14830.
2026-02-16 15:10:19 -05:00
Sergei Zimmerman 46a4a554ca Merge pull request #15237 from xokdvium/add-missing-temp-roots
Add missing temproots for cached sources and existing derivations
2026-02-16 19:35:15 +00:00
John Ericson cc0b489967 Merge pull request #15250 from obsidiansystems/assume-lchown
Remove suppport for not having `lchown`
2026-02-16 19:29:08 +00:00
John Ericson af7e585009 Split file-system-at.{cc,hh} from file-descriptor.{cc,hh}
`file-descriptor.{cc,hh}` was getting too big, split out
`file-system-at.{cc,hh}` for the FD-based file system stuff,
`file-descriptor.{cc,hh}` will only be for the fundamental primitives
that are file-system agnostic and work on almost all file types.

Review with `git show --color-moved` to see that this is indeed all
moving.
2026-02-16 14:21:52 -05:00
Amaan Qureshi 2ccb8a9a56 upload-release: disable containerd image store to preserve gzip layer compression
Docker 28+ defaults to the containerd image store, which pushes layers
uncompressed instead of gzip. The GHA runner image updated Docker to
29.x (actions/runner-images#13633), causing the `nixos/nix:2.33.3`
image to balloon from 138 MB to 505 MB, with all 70 layers pushed as
`application/vnd.docker.image.rootfs.diff.tar` instead of `.tar.gzip`.
OCI clients that only support gzip (e.g. `go-containerregistry`, used
by Concourse CI) fail with "gzip: invalid header".

This commit disables the containerd snapshotter in the release workflow
before any Docker operations, restoring the classic storage driver that
preserves gzip compression through the `docker load` / `docker push`
pipeline.

Fixes #15246
2026-02-16 14:08:08 -05:00
John Ericson fefa66880a Remove suppport for not having lchown
Linux, macOS, and all 3 BSDs have it (according to man page google
search), so let's just drop this. Support for not having it was added in
d03f0d4117 in 2006, things have changed in
the last 20 years!
2026-02-16 13:40:29 -05:00
John Ericson a53391fd0e Merge pull request #15247 from roberth/clarify-ref-upcasting
Better `ref` casting DX
2026-02-16 17:09:16 +00:00
Robert Hensing 771421a34e fix(ref): improve cast exception type and add demangled type names
When ref::cast() fails, the error message was cryptic ("null pointer
cast to ref"). Now it throws a proper bad_ref_cast (a std::bad_cast
subclass) with a clear message showing the actual types involved:

    ref<nix::Base> cannot be cast to ref<nix::Derived>

This also adds a demangle.hh utility.
2026-02-16 17:07:40 +01:00
Robert Hensing 5aaa0cc4a6 refactor(ref): clarify implicit conversion semantics with requires clause
ref<Derived> was already implicitly convertible to ref<Base>, but the
mechanism was unclear and error messages for rejected downcasts were
more cryptic than necessary. This change:

- Adds RefImplicitlyUpcastableTo concept to constrain the conversion
  operator, making the intent explicit and improving error messages
- Documents .cast() and .dynamic_pointer_cast() as alternatives for
  explicit downcasting
- Adds unit tests for covariance behavior
2026-02-16 16:43:08 +01:00
John Ericson 0749ec4e55 Merge pull request #15230 from obsidiansystems/new-wine
flake: Use Wine 11 for running mingw tests
2026-02-15 16:41:52 +00:00
Artemis Tosini 4cc97150df flake: Use Wine 11 for running mingw tests
Set wine_11 as the emulator for Windows.
2026-02-15 10:56:02 -05:00
John Ericson 2bbd1094a2 flake.lock: Update Nixpkgs
Flake lock file updates:

• Updated input 'nixpkgs':
    'https://releases.nixos.org/nixos/25.11/nixos-25.11.4506.078d69f03934/nixexprs.tar.xz?narHash=sha256-Xu%2B7iYcAuOvsI2wdkUcIEmkqEJbvvE6n7qR9QNjJyP4%3D' (2026-01-22)
  → 'https://releases.nixos.org/nixos/25.11/nixos-25.11.5960.3aadb7ca9eac/nixexprs.tar.xz?narHash=sha256-WoiezqWJQ3OHILah%2Bp6rzNXdJceEAmAhyDFZFZ6pZzY%3D' (2026-02-14)

This will be needed to get Wine 11.
2026-02-15 10:53:15 -05:00
John Ericson 95251a51dd Merge pull request #15241 from obsidiansystems/fix-isindir
libutil: fix `isInDir` rejecting paths starting with dot
2026-02-15 15:52:37 +00:00
John Ericson 02d9f4ecb4 Merge pull request #15239 from xokdvium/fix-warnings-no-intereference-size
meson: Only enable -Wno-interference-size with GCC
2026-02-15 15:06:54 +00:00
John Ericson 3269c71e9d Merge pull request #15240 from xokdvium/fix-mtls-redirect-test
libstore-tests: Fix mTLS test for redirect, correctly propagate tries
2026-02-15 15:04:50 +00:00
Amaan Qureshi ad0055e67c libutil: fix isInDir rejecting paths starting with dot
The old check rejected any relative path whose first character was a
dot, producing false negatives for valid descendants like `.ssh` or
`.config`. This commit changes the logic such that now it inspects the
first path component via `path::begin()`, only rejects `.` and `..`
rather than anything dot-prefixed. Fixes #15207.
2026-02-15 10:04:08 -05:00
John Ericson 7c915b371d Merge pull request #15235 from obsidiansystems/os-environ
libutil-tests: Fix crash on Windows
2026-02-15 14:58:22 +00:00
Artemis TosiniandJohn Ericson 36d0e9580f Implement Pid::kill for Windows
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2026-02-14 20:39:32 -05:00
Artemis TosiniandJohn Ericson c9abefbc30 libutil-tests: Fix crash on Windows
libutil tests were crashing on Windows due to issues finding `environ`.
Replace process creation of `getEnv` with a new `getEnvOs` function that
uses native windows APIs.

Also convert a bunch of `RunOptions` fields to use `OsString` to better
reflect the underlying interfaces.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2026-02-14 20:39:32 -05:00
Sergei Zimmerman 6cbf80a0b0 Merge pull request #15219 from obsidiansystems/writeDerivation-lighter-read-only
Get rid of the settings-dependent `writeDerivation` wrapper
2026-02-14 21:27:52 +00:00
Sergei Zimmerman d3d63a4b5b libstore-tests: Fix mTLS test for redirect, correctly propagate tries
The fake cacert didn't have subjectAltName for 127.0.0.1, so the test
was failing for a different reason. Also `tries` setting wasn't being respected.
There's no callsite specifying it in the request, so just use the one specified
in the FileTransferSettings and remove the fields from the FileTransferRequest.
2026-02-15 00:08:21 +03:00
Sergei Zimmerman 6a5ee08737 meson: Only enable -Wno-interference-size with GCC
Clang doesn't recognise this option.
2026-02-14 23:42:28 +03:00
Sergei Zimmerman ac2dd58b6f Add missing temproots for cached sources and existing derivations 2026-02-14 12:09:24 +03:00
John Ericson 8fadcceb6d Merge pull request #15233 from obsidiansystems/remove-nixstore-global
libstore: remove `Settings::nixStore` in favor of `StoreConfigBase::getDefaultNixStoreDir`
2026-02-13 20:29:11 +00:00
John Ericson 2913722781 Merge pull request #15229 from lisanna-dettwyler/fix-gc-dry-run
Emit basic dry run message for garbage collection
2026-02-13 20:19:49 +00:00
Amaan Qureshi 12f97382af libstore: remove Settings::nixStore in favor of StoreConfigBase::getDefaultNixStoreDir
This commit removes the `nixStore` member from `Settings` and instead
computes the default Nix store directory directly in
`StoreConfigBase::getDefaultNixStoreDir()` from env vars
(`NIX_STORE_DIR`, `NIX_STORE`) or the compile-time default. The method
is made public so callers that previously reached through the global
`settings.nixStore` can use it instead.

Progress on #5638
2026-02-13 14:45:49 -05:00
Lisanna Dettwyler fdfc772114 Emit basic dry run message for garbage collection
nix store gc: prints number of paths that would be freed, but not bytes
nix-collect-garbage: ditto
nix-store --gc: retains current behavior

It would be very non-trivial to also compute the bytes that would be
freed, due to hardlinking in the store.

Also adds checking for incompatible mixing of dry-run and max-freed
options.

Resolves #5704

Signed-off-by: Lisanna Dettwyler <lisanna.dettwyler@gmail.com>
2026-02-13 14:40:36 -05:00
John Ericson a4b1814d67 Merge pull request #15232 from obsidiansystems/inline-buildlocally
libstore: inline `willBuildLocally` and `canBuildLocally` into call sites
2026-02-13 19:29:15 +00:00
John Ericson 702ebdb11b Merge pull request #15231 from obsidiansystems/inline-getmachines
libstore: inline `getMachines` into call sites
2026-02-13 19:07:18 +00:00
Amaan QureshiandJohn Ericson 7106de16e6 libstore: inline willBuildLocally and canBuildLocally into call sites
This commit inlines `DerivationOptions::willBuildLocally` and
`DerivationOptions::canBuildLocally` into their sole call site in
`DerivationBuildingGoal::tryToBuild`. The `canBuildLocally` logic is now
a lambda capturing the surrounding context, and `willBuildLocally` is
replaced by `drvOptions.preferLocalBuild && canBuildLocally`.

Progress on #5638

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2026-02-13 13:45:33 -05:00
John Ericson b818594ba2 Merge pull request #15228 from obsidiansystems/profile-dirs-options
libstore: extract `ProfileDirsOptions` from `Settings`
2026-02-13 18:28:56 +00:00
Amaan QureshiandJohn Ericson 9ae12ede4c libstore: inline getMachines into call sites
This commit removes the `getMachines` free function and inlines `Machine::parseConfig({settings.thisSystem}, settings.getWorkerSettings().builders)` at its two call sites in `worker.cc` and `build-remote.cc`. The wrapper just forwarded to `Machine::parseConfig` with global settings, so inlining it removes an unnecessary layer of indirection and makes the global dependency explicit at each call site.

Progress on #5638

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2026-02-13 13:22:49 -05:00
John Ericson 20f7f33123 Merge pull request #15227 from obsidiansystems/narinfo-disk-cache-settings
libstore: extract `NarInfoDiskCacheSettings` from `Settings`
2026-02-13 17:58:33 +00:00
John EricsonandAmaan Qureshi 002cbefa9f libstore: extract ProfileDirsOptions from Settings
This commit moves `nixStateDir` and `useXDGBaseDirectories` into a dedicated `ProfileDirsOptions` struct and threads it through the profile directory functions (`profilesDir`, `rootProfilesDir`, `defaultChannelsDir`, `rootChannelsDir`, `getDefaultProfile`) so they no longer read from the global `Settings` object directly. This follows the same pattern as `LocalSettings`, `WorkerSettings`, and `NarInfoDiskCacheSettings`.

Progress on #5638

Co-authored-by: Amaan Qureshi <git@amaanq.com>
2026-02-13 12:43:53 -05:00
John Ericson dc636dde10 libstore: extract NarInfoDiskCacheSettings from Settings
This commit moves `ttlNegativeNarInfoCache` and `ttlPositiveNarInfoCache` into a dedicated `NarInfoDiskCacheSettings` struct that `Settings` privately inherits from, following the same pattern as `LocalSettings`, `LogFileSettings`, and `WorkerSettings`.

`NarInfoDiskCache` now takes explicit `NarInfoDiskCacheSettings` and `SQLiteSettings` in its constructor instead of reading from the global. The singleton `getNarInfoDiskCache()` is replaced with a `NarInfoDiskCache::get()` static method that accepts these settings, though they are only used on the first call (subsequent calls return the cached instance regardless of arguments).

Progress on #5638
2026-02-13 12:12:34 -05:00
John Ericson a06ab4871c Merge pull request #15217 from amaanq/acquire-user-lock-state-dir
libstore: pass `stateDir` to `acquireUserLock` instead of using global
2026-02-13 01:31:41 +00:00
John Ericson ed22ef2b89 Merge pull request #15218 from obsidiansystems/read-only-per-store
libstore: make substitution use the per-store `getReadOnly` method
2026-02-13 01:31:32 +00:00
John Ericson 7926a629e2 Get rid of the settings-dependent writeDerivation wrapper
It was a crude hack that this one low-level function was dependent on
the high-level read-only mode setting --- all the more so because rather
than making derivation writing fail, that setting made it silently
"succeed" why not actually writing the derivation. (Also, for context,
we didn't have an such behavior for any other store-mutating operations,
just for this one function.)

I have gotten rid of the wrapper, and updated the call sites
accordingly.

- For the ones that should remain dependent on this setting, I made this
  explicit, and added a comment.

- For others, surrounding operations assumed writability (e.g. we had
  written something before, or were about to try to read back the
  written derivation after), and so I just made those do the underlying
  `Store::writeDerivation` operation.
2026-02-12 20:26:24 -05:00
Bernardo Meurer a8f305add3 Merge pull request #15216 from NixOS/fix-s3-conn-reuse
fix: #15208
2026-02-13 00:53:58 +00:00
Amaan Qureshi cecbe9f73a libstore: pass stateDir to acquireUserLock instead of using global
This makes `acquireUserLock` take an explicit stateDir parameter,
since it was previously reaching into the global settings object
just to read `nixStateDir` for constructing the userpool paths.

Progress on #5638
2026-02-12 19:43:40 -05:00
Amaan Qureshi 9ac91e36a9 libstore: make substitution use the per-store getReadOnly method
This commit introduces a `getReadOnly` method on the store config that returns if the current store is read only or not. This is then used in subtitution, so we fail gracefully with a nice error message if only the individual store is read-only.

As a bonus, it gets us one step closer to getting rid of the global because we can use the per-store method instead.

Progress on #5638
2026-02-12 19:43:20 -05:00
Bernardo Meurer Costa 759f6c856b feat(libstore/s3): use virtual-hosted-style URLs and add addressing-style option
S3 binary caches now use virtual-hosted-style URLs by default for
standard AWS endpoints. Path-style endpoints (s3.region.amazonaws.com)
only serve HTTP/1.1, preventing HTTP/2 multiplexing and causing TCP
TIME_WAIT socket exhaustion under high concurrency. Virtual-hosted-style
endpoints (bucket.s3.region.amazonaws.com) support HTTP/2, enabling
multiplexing with the existing CURLPIPE_MULTIPLEX configuration.

Add a new `addressing-style` store option (auto/path/virtual) to control
this behavior. `auto` (default) uses virtual-hosted-style for standard
AWS endpoints and path-style for custom endpoints. `path` forces
path-style for backwards compatibility. `virtual` forces virtual-hosted-
style for all endpoints including custom ones.

Fixes: https://github.com/NixOS/nix/issues/15208
2026-02-13 00:03:50 +00:00
Bernardo Meurer Costa 736abd50ff fix(libstore/filetransfer): enable TCP keep-alive on curl handles
Idle connections in libcurl's connection pool can be silently dropped by
the OS or intermediate firewalls/NATs before they can be reused, forcing
new TCP connections to be created. This is especially problematic for
HTTP/1.1 endpoints where multiplexing is unavailable.

Enable TCP keep-alive with a 60-second idle/interval on all curl easy
handles to prevent idle connection drops and improve connection reuse.
2026-02-12 22:52:48 +00:00
John Ericson a3d51172e9 Merge pull request #15211 from obsidiansystems/worker-settings
libstore: extract `WorkerSettings` from `Settings`
2026-02-12 21:15:43 +00:00
Sergei Zimmerman eae7e0151c Merge pull request #15213 from xokdvium/unhardcode-alignas-cache-line-size
Unhardcode alignas cache line size
2026-02-12 20:53:09 +00:00
Amaan Qureshi d3388d3d81 libstore: extract WorkerSettings from Settings
This commit  moves `pollInterval`, `maxSubstitutionJobs`, `postBuildHook`, and `logLines` into a dedicated `WorkerSettings` struct that `Settings` privately inherits from, as they are only used by the build worker subsystem. This follows the same pattern as `LocalSettings` and `LogFileSettings`.
2026-02-12 15:31:08 -05:00
Sergei Zimmerman 7352205ce9 libexpr: Replace hardcoded cache line size with std::hardware_destructive_interference_size
This expands to __GCC_DESTRUCTIVE_SIZE, which is also 64 (at least in the x86_64 stdenv).
Let the compiler decide what's the appropriate cache line size is. Also, on aarch64-darwin
the cache line size 128 bytes, so the previous fix didn't actually get rid of false sharing
reliably. Clang does this [1] [2], so it overestimates the sizes somewhat, but that's still enough
for avoiding false sharing on darwin.

[1]: https://github.com/llvm/llvm-project/blob/a289341dedf4d73955faefbb0b3c13881e13dd06/clang/lib/Frontend/InitPreprocessor.cpp#L1331-L1339
[2]: https://github.com/llvm/llvm-project/blob/6f51f8e0f93e6ff2baf0ed2f60021308920a557f/clang/lib/Basic/Targets/AArch64.h#L262-L264
2026-02-12 23:04:40 +03:00
Sergei Zimmerman f3f9eac8fc Merge pull request #15209 from obsidiansystems/http-store-port-ctor
libstore: add `HttpBinaryCacheStoreConfig` constructor that takes a ` ParsedURL`
2026-02-12 18:48:42 +00:00
Sergei Zimmerman df21c81191 libexpr: Fix some typos in value.hh 2026-02-12 20:51:38 +03:00
Amaan Qureshi 52b1906995 libstore: add HttpBinaryCacheStoreConfig constructor that takes a ParsedURL
In the https-store tests, a `TestHttpBinaryCacheStoreConfig` is constructed with a call to format to create the cache uri. This commit adds a constructor to `HttpBinaryCacheStoreConfig` to remove the need for this call, and updates the test type to leverage this so we're no longer manually calling fmt on a string to format the port.
2026-02-12 11:22:29 -05:00
John Ericson c756d02948 Merge pull request #15206 from obsidiansystems/injectable-filetransfer
libstore: make `FileTransfer` injectable into `HttpBinaryCacheStore`
2026-02-12 14:58:34 +00:00
Amaan Qureshi 403e30f136 libstore: make FileTransfer injectable into HttpBinaryCacheStore
This commit makes `FileTransfer` self-contained by giving it a reference
to `FileTransferSettings` instead of reading from the global. It also
adds an optional `FileTransfer` parameter to `HttpBinaryCacheStore` so
callers can inject their own instance.

The main motivation is test isolation. The HTTPS store tests now create
custom `FileTransferSettings` with the test CA certificate and pass it
through `makeFileTransfer()`, avoiding global state mutation entirely.
2026-02-11 19:00:53 -05:00
Sergei Zimmerman 3a60a04bf8 Merge pull request #15183 from obsidiansystems/newuidmap
Support build users on unprivileged users with subuid/subgid
2026-02-11 22:35:44 +00:00
Artemis Tosini c9526e289a Add new libexec/nix-nswrapper program
nix-nswrapper allows running nix in its own user namespace,
believing it is root and with access to build users for sandboxing
with auto-allocate-uids, while it is actually unprivileged.

It is used to wrap nix, and an example of its use has been
added to the unprivileged daemon functional tests.

Running it does not require any elevated privileges,
only uids and gids allocated in /etc/sub{uid,gid}
2026-02-11 16:53:08 -05:00
Eelco Dolstra d4a0024184 Merge pull request #15205 from NixOS/bump-file-limit-upstream
Increase the open file soft limit to the hard limit
2026-02-11 21:40:39 +00:00
Sergei Zimmerman d9651b1f82 Merge pull request #15193 from NixOS/restore-death-signal
DerivationBuilder: Preserve death signal across setuid,setgid
2026-02-11 21:26:32 +00:00
John Ericson 912c6c283d Merge pull request #15202 from obsidiansystems/migrate-ca-netrc-downloadspeed-filetransfer
libstore: migrate `caFile`, `netrcFile`, and `downloadSpeed` to `FileTransferSettings`
2026-02-11 21:09:06 +00:00
Eelco Dolstra 04fd722b1b Increase the open file soft limit to the hard limit
On some platforms (macOS), the default soft limit is very low, but the
hard limit is high. So let's just raise it the maximum permitted.
2026-02-11 21:55:57 +01:00
John Ericson 1a57df3473 Merge pull request #15203 from obsidiansystems/substituter-confs
Deduplicate `nix repl` and `nix log`
2026-02-11 20:41:56 +00:00
eveeifyeveandAmaan Qureshi 04d13a96e3 libstore: migrate caFile, netrcFile, and downloadSpeed to FileTransferSettings
The `caFile`, `netrcFile`, and `downloadSpeed` settings are only used by
the file transfer subsystem but lived in the global `Settings` class.
This moves them to `FileTransferSettings` where they belong.

Co-authored-by: Amaan Qureshi <git@amaanq.com>
2026-02-11 14:58:27 -05:00
Amaan Qureshi 46eabe34c2 libstore: move hashedMirrors to LocalSettings
`hashedMirrors` is only relevant to local builds (it is consumed by
`builtin:fetchurl` during derivation building) but lived in the global
`Settings` class. This moves it to `LocalSettings` where it belongs
and threads it through `BuiltinBuilderContext` so `fetchurl.cc` reads
it from the context instead of reaching into `settings` directly.
2026-02-11 14:58:27 -05:00
John Ericson ecdcdd82e0 Deduplicate nix repl and nix log
The underlying mechanism is now in a new `fetchBuildLog` function I put
in `libcmd`.

I am putting it in here and not in libstore because I have some doubts
about `getDefaultSubstituters`, so I would like to keep it in a more
"peripheral" part of the codebase for now.
2026-02-11 14:55:31 -05:00
Jörg Thalheim a4c421da22 Merge pull request #15201 from mkenigs/install-release-notes
beta nix-installer: add release-note
2026-02-11 19:07:49 +00:00
John Ericson ae4e4d9afd Merge pull request #15192 from obsidiansystems/store-reference-types
globals: change store settings to use `StoreReference` types directly
2026-02-11 17:49:07 +00:00
Matthew KenigsbergandCole Helbling fbd837c911 beta nix-installer: add release-note
Add a release note asking for help testing
https://github.com/NixOS/nix-installer

We're hoping to start recommending the Rust-based installer after one
release cycle.

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2026-02-11 10:07:59 -07:00
Amaan Qureshi 857fd2a3a4 globals: change store settings to use StoreReference types directly
The `storeUri`, `substituters`, and `trustedSubstituters` settings now
store typed `StoreReference` values directly instead of raw strings,
so callers work with the real types without manual parsing.

This is a reworked version of #10761.
2026-02-11 12:05:50 -05:00
John Ericson 3c1ad7d978 Merge pull request #15197 from obsidiansystems/ssh-store-config-direct
nix-copy-closure: create `LegacySSHStoreConfig` directly
2026-02-11 16:15:29 +00:00
Amaan Qureshi 8020a847ab nix-copy-closure: create LegacySSHStoreConfig directly
Instead of constructing a `StoreReference` and letting `openStore()` resolve it,
this commit creates the store config directly and calls `openStore()` on it. This
avoids the indirection through the store registry.
2026-02-11 10:31:46 -05:00
John Ericson db8499e62f Merge pull request #15200 from obsidiansystems/remove-const-settings
store-config: remove unnecessary `const` from `Setting<>` fields
2026-02-11 15:29:59 +00:00
Amaan Qureshi 1add77677f store-config: remove unnecessary const from Setting<> fields
Stores hold their config as `ref<const Config>` or `const Config &`,
so `Setting<>` fields are already immutable after store construction.
The field-level `const` is redundant and prevents pre-construction
mutation which is sometimes useful. This commit updates these settings by dropping the `const` qualifier, as it's not needed.
2026-02-11 09:20:41 -05:00
John Ericson d5eda907ef Merge pull request #15195 from obsidiansystems/store-reference-args
globals: use `StoreReference` types in CLI argument handlers
2026-02-11 00:22:08 +00:00
Amaan Qureshi f9300514cd globals: use StoreReference types in CLI argument handlers
The CLI flags `--from`, `--to`, `--eval-store`, and substituter URIs now
parse to `StoreReference` at the argument boundary. `fetchClosure` uses
`StoreReference::parse` instead of `parseURL`. This also adds
`operator<=>` to `StoreReference`.
2026-02-10 18:37:34 -05:00
John Ericson 036a47be83 Merge pull request #15194 from obsidiansystems/nix-daemon-store-config
Make `nix daemon` a `StoreConfigCommand`
2026-02-10 23:13:15 +00:00
John Ericson c4e408459a Make nix daemon a StoreConfigCommand
This commit makes `nix daemon` inherit from `StoreConfigCommand`
instead of `Command`, so that it receives a `StoreConfig` to open
and serve stores with. This cleans up a few things (removes
`openUncachedStore` helper, passes `storeConfig` through
`daemonLoop`/`runDaemon` instead of opening stores ad-hoc) and will
allow further cleanups.
2026-02-10 17:34:13 -05:00
Sergei Zimmerman f0498b94d8 Merge pull request #14768 from pkpbynum/capi/copy-path
C API: Add copy_path to Store API
2026-02-10 22:04:22 +00:00
Sergei Zimmerman 34688ecf5f DerivationBuilder: Preserve death signal across setuid,setgid
It's apparently a common footgun in Linux that the death signal isn't
preserved across calls to setuid/setgid. If nix-build gets SIGKILL-ed
while a build is running that would lead to a runaway build process that
would get reparented to init/systemd.

This is pretty easy to reproduce with the following derivation:

derivation {
  name = "pdeathsig-repro";
  system = builtins.currentSystem;
  builder = "/bin/sh";
  args = [
    "-c"
    ''
      while :; do :; done
    ''
  ];
}

And the reproduction script:

sudo nix-build repro.nix &
sleep 3
BUILDER=$(pgrep -u nixbld1)
sudo kill -9 $(pgrep -f 'nix-build.*repro')
sleep 1
ps -p $BUILDER -o pid,ppid,user,comm

To address this we have to restore the death signal after all the calls
to setuid/setgid. This is done in a helper function preserveDeathSignal
that takes a callback to avoid code duplication.

See: https://github.com/golang/go/issues/9686
2026-02-11 00:56:34 +03:00
John Ericson 92d0fe000b Merge pull request #15188 from obsidiansystems/handleexceptions
Move `nix::handleExceptions` to libutil
2026-02-10 18:55:14 +00:00
John Ericson 75af0351ac Merge pull request #15089 from NixOS/sfp-global-settings
`std::filesystem::path` in some `Settings` fields
2026-02-10 18:43:55 +00:00
Artemis Tosini c79ff97c07 Move nix::handleExceptions to libutil
This is a fairly simple function, isolated from the rest of libmain
and could be useful if new programs are made that are not part of the
main nix-cli subproject.
2026-02-10 13:06:27 -05:00
John Ericson ef659136ca std::filesystem::path in some Settings fields 2026-02-10 12:50:17 -05:00
John Ericson 582e4fa0f6 Merge pull request #15187 from obsidiansystems/serve-unix-socket
Factor out `serveUnixSocket`
2026-02-10 17:03:36 +00:00
Amaan Qureshi 6674c23416 Factor out serveUnixSocket
This commit extracts the Unix domain socket server loop (`PeerInfo`,
`getPeerInfo`, and the systemd socket activation / poll / accept loop)
from `src/nix/unix/daemon.cc` into a reusable `unix::serveUnixSocket`
function in `libcmd`.
2026-02-10 11:25:57 -05:00
Eelco Dolstra b06d0f764f Merge pull request #15175 from KiaraGrouwstra/flake-ref-nixpkgs
check `isFlake` in `nixpkgsFlakeRef`
2026-02-10 14:03:37 +00:00
cinereal 30e213c948 check isFlake in nixpkgsFlakeRef
To get `bashInteractive`, `nix develop` currently
[gets a flake reference to nixpkgs](https://github.com/NixOS/nix/blob/3b473c4be5e6abc58aa43271f35ff127af806c55/src/nix/develop.cc#L650-L658),
either from [`inputs.nixpkgs`](https://github.com/NixOS/nix/blob/3b473c4be5e6abc58aa43271f35ff127af806c55/src/libcmd/installable-flake.cc#L204)
or [`<nixpkgs>`](https://github.com/NixOS/nix/blob/3b473c4be5e6abc58aa43271f35ff127af806c55/src/libcmd/include/nix/cmd/installable-flake.hh#L87).

Currently, this is done by name for any (locked) reference.
For any `nixpkgs` reference lacking a `flake.nix`, `nix develop` thus errors:

```
error (ignored): path '.../flake.nix' does not exist
```

While the registry does not expose info to help check this,
flake inputs expose the `flake` boolean.
This means that given `inputs.<name>.flake = false;`,
we may know in advance that our reference should not be to a flake.

This change incorporates this, so that given `inputs.nixpkgs.flake = false;`,
`nix develop` will fall back `<nixpkgs>` to use its flake for `bashInteractive`.

While Nixpkgs itself of course does expose a flake,
this check is relevant in particular for dummy inputs
(e.g. inputs using `{ url = "file:/dev/null"; flake = false; }`),
which may be overridden by `--override-input` or `.follows`,
yet do not expose (flake) code themselves.

Signed-off-by: cinereal <cinereal@riseup.net>
2026-02-08 13:32:20 +01:00
Peter Bynum 72ab64b612 Add nix_store_copy_path C API 2026-02-06 11:59:37 -05:00
209 changed files with 4727 additions and 2510 deletions

No files matched your search

+12 -1
View File
@@ -34,11 +34,22 @@ jobs:
# get the same uberhack that nix-shell has to support it.
echo "NIX_PATH=nixpkgs=$NIXPKGS_PATH" >> "$GITHUB_ENV"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # v6.0.0
uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708 # v5.1.1
with:
role-to-assume: "arn:aws:iam::080433136561:role/nix-release"
role-session-name: nix-release-oidc-${{ github.run_id }}
aws-region: eu-west-1
- name: Disable containerd image store
run: |
# Docker 28+ defaults to the containerd image store, which
# pushes layers uncompressed instead of gzip. OCI clients
# that only support gzip (e.g. go-containerregistry) fail
# with "gzip: invalid header". Disabling the containerd
# snapshotter restores the classic storage driver, which
# preserves gzip-compressed layers through the
# `docker load` / `docker push` pipeline.
echo '{"features":{"containerd-snapshotter":false}}' | sudo tee /etc/docker/daemon.json > /dev/null
sudo systemctl restart docker
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
with:
+29
View File
@@ -0,0 +1,29 @@
---
synopsis: "Rust nix-installer in beta"
prs: []
---
The Rust-based rewrite of the Nix installer is now in beta.
We'd love help testing it out!
To test out the new installer, run:
```
curl -sSfL https://artifacts.nixos.org/nix-installer | sh -s -- install
```
This installer can be run even when you have an existing, script-based Nix installation without any adjustments.
This new installer also comes with the ability to uninstall your Nix installation; run:
```
/nix/nix-installer uninstall
```
This will get rid of your entire Nix installation (even if you installed over an existing, script-based installation).
This installer is a modified version of the [Determinate Nix Installer](https://github.com/DeterminateSystems/nix-installer) by Determinate Systems.
Thanks to Determinate Systems for all the investment they've put into the installer.
Source for the installer is in https://github.com/NixOS/nix-installer.
Report any issues in that repo.
For CI usage, a GitHub Action to install Nix using this installer is available at https://github.com/NixOS/nix-installer-action.
@@ -6,3 +6,4 @@ prs: [14766]
The C API now includes additional methods:
- `nix_store_query_path_from_hash_part()` - Get the full store path given its hash part
- `nix_store_copy_path()` - Copy a single store path between two stores, allows repairs and configuring signature checking
+11
View File
@@ -0,0 +1,11 @@
---
synopsis: New command `nix store roots-daemon` for serving GC roots
prs: [15143]
---
New command [`nix store roots-daemon`](@docroot@/command-ref/new-cli/nix3-store-roots-daemon.md) runs a daemon that serves garbage collector roots over a Unix domain socket.
It enables the garbage collector to discover runtime roots when the main Nix daemon doesn't have `CAP_SYS_PTRACE` capability and therefore cannot scan `/proc`.
The garbage collector can be configured to use this daemon via the [`use-roots-daemon`](@docroot@/store/types/local-store.md#store-experimental-option-use-roots-daemon) store setting.
This feature requires the [`local-overlay-store` experimental feature](@docroot@/development/experimental-features.md#xp-feature-local-overlay-store).
@@ -0,0 +1,32 @@
---
synopsis: S3 binary caches now use virtual-hosted-style addressing by default
issues: [15208]
---
S3 binary caches now use virtual-hosted-style URLs
(`https://bucket.s3.region.amazonaws.com/key`) instead of path-style URLs
(`https://s3.region.amazonaws.com/bucket/key`) when connecting to standard AWS
S3 endpoints. This enables HTTP/2 multiplexing and fixes TCP connection
exhaustion (TIME_WAIT socket accumulation) under high-concurrency workloads.
A new `addressing-style` store option controls this behavior:
- `auto` (default): virtual-hosted-style for standard AWS endpoints, path-style
for custom endpoints.
- `path`: forces path-style addressing (deprecated by AWS).
- `virtual`: forces virtual-hosted-style addressing (bucket names must not
contain dots).
Bucket names containing dots (e.g., `my.bucket.name`) automatically fall back
to path-style addressing in `auto` mode, because dotted names create
multi-level subdomains that break TLS wildcard certificate validation.
Example using path-style for backwards compatibility:
```
s3://my-bucket/key?region=us-east-1&addressing-style=path
```
Additionally, TCP keep-alive is now enabled on all HTTP connections, preventing
idle connections from being silently dropped by intermediate network devices
(NATs, firewalls, load balancers).
Generated
+4 -4
View File
@@ -63,11 +63,11 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1769089682,
"narHash": "sha256-Xu+7iYcAuOvsI2wdkUcIEmkqEJbvvE6n7qR9QNjJyP4=",
"rev": "078d69f03934859a181e81ba987c2bb033eebfc5",
"lastModified": 1771043024,
"narHash": "sha256-WoiezqWJQ3OHILah+p6rzNXdJceEAmAhyDFZFZ6pZzY=",
"rev": "3aadb7ca9eac2891d52a9dec199d9580a6e2bf44",
"type": "tarball",
"url": "https://releases.nixos.org/nixos/25.11/nixos-25.11.4506.078d69f03934/nixexprs.tar.xz"
"url": "https://releases.nixos.org/nixos/25.11/nixos-25.11.5960.3aadb7ca9eac/nixexprs.tar.xz"
},
"original": {
"type": "tarball",
+5
View File
@@ -115,6 +115,9 @@
}
// lib.optionalAttrs (crossSystem == "x86_64-unknown-freebsd13") {
useLLVM = true;
}
// lib.optionalAttrs (crossSystem == "x86_64-w64-mingw32") {
emulator = pkgs: "${pkgs.buildPackages.wineWow64Packages.stable_11}/bin/wine";
};
overlays = [
(overlayFor (pkgs: pkgs.${stdenv}))
@@ -406,6 +409,8 @@
"nix-cmd" = { };
"nix-nswrapper" = { };
"nix-cli" = { };
"nix-everything" = { };
+4
View File
@@ -24,6 +24,10 @@ subproject('libcmd')
# Executables
subproject('nix')
if host_machine.system() == 'linux'
subproject('nswrapper')
endif
# Docs
if get_option('doc-gen')
subproject('internal-api-docs')
@@ -9,3 +9,9 @@ endif
if 'address' in get_option('b_sanitize')
deps_other += declare_dependency(sources : 'asan-options.cc')
endif
if 'undefined' in get_option('b_sanitize')
add_project_arguments('-DNIX_UBSAN_ENABLED=1', language : 'cpp')
else
add_project_arguments('-DNIX_UBSAN_ENABLED=0', language : 'cpp')
endif
@@ -33,6 +33,13 @@ add_project_arguments(
# GCC doesn't benefit much from precompiled headers.
do_pch = cxx.get_id() == 'clang'
if cxx.get_id() == 'gcc'
add_project_arguments(
'-Wno-interference-size', # Used for C++ ABI only. We don't provide any guarantees about different march tunings.
language : 'cpp',
)
endif
# This is a clang-only option for improving build times.
# It forces the instantiation of templates in the PCH itself and
# not every translation unit it's included in.
+2
View File
@@ -418,6 +418,8 @@ in
nix-cmd = callPackage ../src/libcmd/package.nix { };
nix-nswrapper = callPackage ../src/nswrapper/package.nix { };
/**
The Nix command line interface. Note that this does not include its tests, whereas `nix-everything` does.
*/
+5
View File
@@ -31,6 +31,8 @@
nix-cmd,
nix-nswrapper,
nix-cli,
nix-functional-tests,
@@ -171,6 +173,9 @@ stdenv.mkDerivation (finalAttrs: {
# Forwarded outputs
ln -sT ${nix-manual} $doc
ln -sT ${nix-manual.man} $man
''
+ lib.optionalString stdenv.isLinux ''
lndir ${nix-nswrapper} $out
'';
passthru = {
+1
View File
@@ -57,6 +57,7 @@ let
"nix-flake"
"nix-flake-c"
"nix-flake-tests"
"nix-nswrapper"
"nix-main"
"nix-main-c"
"nix-cmd"
+8 -8
View File
@@ -75,7 +75,7 @@ ref<StoreConfig> StoreConfigCommand::getStoreConfig()
ref<StoreConfig> StoreConfigCommand::createStoreConfig()
{
return resolveStoreConfig(StoreReference::parse(settings.storeUri.get()));
return resolveStoreConfig(StoreReference{settings.storeUri.get()});
}
void StoreConfigCommand::run()
@@ -116,28 +116,28 @@ CopyCommand::CopyCommand()
.longName = "from",
.description = "URL of the source Nix store.",
.labels = {"store-uri"},
.handler = {&srcUri},
.handler = {[this](std::string s) { srcUri = StoreReference::parse(s); }},
});
addFlag({
.longName = "to",
.description = "URL of the destination Nix store.",
.labels = {"store-uri"},
.handler = {&dstUri},
.handler = {[this](std::string s) { dstUri = StoreReference::parse(s); }},
});
}
ref<StoreConfig> CopyCommand::createStoreConfig()
{
return srcUri.empty() ? StoreCommand::createStoreConfig() : resolveStoreConfig(StoreReference::parse(srcUri));
return !srcUri ? StoreCommand::createStoreConfig() : resolveStoreConfig(StoreReference{*srcUri});
}
ref<Store> CopyCommand::getDstStore()
{
if (srcUri.empty() && dstUri.empty())
if (!srcUri && !dstUri)
throw UsageError("you must pass '--from' and/or '--to'");
return dstUri.empty() ? openStore() : openStore(dstUri);
return !dstUri ? openStore() : openStore(StoreReference{*dstUri});
}
EvalCommand::EvalCommand()
@@ -159,7 +159,7 @@ EvalCommand::~EvalCommand()
ref<Store> EvalCommand::getEvalStore()
{
if (!evalStore)
evalStore = evalStoreUrl ? openStore(*evalStoreUrl) : getStore();
evalStore = evalStoreUrl ? openStore(StoreReference{*evalStoreUrl}) : getStore();
return ref<Store>(evalStore);
}
@@ -325,7 +325,7 @@ void MixProfile::updateProfile(Store & store, const BuiltPaths & buildables)
MixDefaultProfile::MixDefaultProfile()
{
profile = getDefaultProfile().string();
profile = getDefaultProfile(settings.getProfileDirsOptions()).string();
}
static constexpr auto environmentVariablesCategory = "Options that change environment variables";
+1 -1
View File
@@ -148,7 +148,7 @@ MixEvalArgs::MixEvalArgs()
)",
.category = category,
.labels = {"store-url"},
.handler = {&evalStoreUrl},
.handler = {[this](std::string s) { evalStoreUrl = StoreReference::parse(s); }},
});
}
+31
View File
@@ -0,0 +1,31 @@
#include "nix/cmd/get-build-log.hh"
#include "nix/store/log-store.hh"
#include "nix/store/store-open.hh"
namespace nix {
std::string fetchBuildLog(ref<Store> store, const StorePath & path, std::string_view what)
{
auto subs = getDefaultSubstituters();
subs.push_front(store);
for (auto & sub : subs) {
auto * logSubP = dynamic_cast<LogStore *>(&*sub);
if (!logSubP) {
printInfo("Skipped '%s' which does not support retrieving build logs", sub->config.getHumanReadableURI());
continue;
}
auto & logSub = *logSubP;
auto log = logSub.getBuildLog(path);
if (!log)
continue;
printInfo("got build log for '%s' from '%s'", what, logSub.config.getHumanReadableURI());
return *log;
}
throw Error("build log of '%s' is not available", what);
}
} // namespace nix
+2 -1
View File
@@ -5,6 +5,7 @@
#include "nix/util/args.hh"
#include "nix/cmd/common-eval-args.hh"
#include "nix/store/path.hh"
#include "nix/store/store-reference.hh"
#include "nix/flake/lockfile.hh"
#include <optional>
@@ -92,7 +93,7 @@ private:
*/
struct CopyCommand : virtual StoreCommand
{
std::string srcUri, dstUri;
std::optional<StoreReference> srcUri, dstUri;
CopyCommand();
@@ -6,6 +6,7 @@
#include "nix/main/common-args.hh"
#include "nix/expr/search-path.hh"
#include "nix/expr/eval-settings.hh"
#include "nix/store/store-reference.hh"
#include <filesystem>
@@ -55,7 +56,7 @@ struct MixEvalArgs : virtual Args, virtual MixRepair
LookupPath lookupPath;
std::optional<std::string> evalStoreUrl;
std::optional<StoreReference> evalStoreUrl;
private:
struct AutoArgExpr
@@ -0,0 +1,23 @@
#pragma once
///@file
#include "nix/store/store-api.hh"
#include <string>
#include <string_view>
namespace nix {
/**
* Fetch the build log for a store path, searching the store and its
* substituters.
*
* @param store The store to search (and its substituters).
* @param path The store path to get the build log for.
* @param what A description of what we're fetching the log for (used in messages).
* @return The build log content.
* @throws Error if the build log is not available.
*/
std::string fetchBuildLog(ref<Store> store, const StorePath & path, std::string_view what);
} // namespace nix
+2
View File
@@ -9,6 +9,7 @@ headers = files(
'common-eval-args.hh',
'compatibility-settings.hh',
'editor-for.hh',
'get-build-log.hh',
'installable-attr-path.hh',
'installable-derived-path.hh',
'installable-flake.hh',
@@ -20,4 +21,5 @@ headers = files(
'network-proxy.hh',
'repl-interacter.hh',
'repl.hh',
'unix-socket-server.hh',
)
@@ -0,0 +1,79 @@
#pragma once
///@file
#include "nix/util/file-descriptor.hh"
#include <filesystem>
#include <functional>
#include <optional>
#include <sys/types.h>
namespace nix::unix {
/**
* Information about the identity of the peer on a Unix domain socket connection.
*/
struct PeerInfo
{
std::optional<pid_t> pid;
std::optional<uid_t> uid;
std::optional<gid_t> gid;
};
/**
* Get the identity of the caller, if possible.
*/
PeerInfo getPeerInfo(Descriptor remote);
/**
* Callback type for handling new connections.
*
* The callback receives ownership of the connection and is responsible
* for handling it (e.g., forking a child process, spawning a thread, etc.).
*
* @param socket The accepted connection file descriptor.
* @param closeListeners A callback to close the listening sockets.
* Useful in forked child processes to release the bound sockets.
*/
using UnixSocketHandler = std::function<void(AutoCloseFD socket, std::function<void()> closeListeners)>;
/**
* Options for the serve loop.
*
* Only used if no systemd socket activation is detected.
*/
struct ServeUnixSocketOptions
{
/**
* The Unix domain socket path to create and listen on.
*/
std::filesystem::path socketPath;
/**
* Mode for the created socket file.
*/
mode_t socketMode = 0666;
};
/**
* Run a server loop that accepts connections and calls the handler for each.
*
* This function handles:
* - systemd socket activation (via LISTEN_FDS environment variable)
* - Creating and binding a Unix domain socket if no activation is detected
* - Polling for incoming connections
* - Accepting connections
*
* For each accepted connection, the handler is called with the connection
* file descriptor. The handler takes ownership of the file descriptor and
* is responsible for closing it when done.
*
* This function never returns normally. It runs until interrupted
* (e.g., via SIGINT), at which point it throws `Interrupted`.
*
* @param options Configuration for the server.
* @param handler Callback invoked for each accepted connection.
*/
[[noreturn]] void serveUnixSocket(const ServeUnixSocketOptions & options, UnixSocketHandler handler);
} // namespace nix::unix
+4 -2
View File
@@ -203,8 +203,10 @@ FlakeRef InstallableFlake::nixpkgsFlakeRef() const
if (auto nixpkgsInput = lockedFlake->lockFile.findInput({"nixpkgs"})) {
if (auto lockedNode = std::dynamic_pointer_cast<const flake::LockedNode>(nixpkgsInput)) {
debug("using nixpkgs flake '%s'", lockedNode->lockedRef);
return std::move(lockedNode->lockedRef);
if (lockedNode->isFlake) {
debug("using nixpkgs flake '%s'", lockedNode->lockedRef);
return std::move(lockedNode->lockedRef);
}
}
}
+7
View File
@@ -74,6 +74,7 @@ sources = files(
'command.cc',
'common-eval-args.cc',
'editor-for.cc',
'get-build-log.cc',
'installable-attr-path.cc',
'installable-derived-path.cc',
'installable-flake.cc',
@@ -86,6 +87,12 @@ sources = files(
'repl.cc',
)
if host_machine.system() != 'windows'
sources += files(
'unix/unix-socket-server.cc',
)
endif
subdir('include/nix/cmd')
subdir('nix-meson-build-support/export-all-symbols')
+9 -4
View File
@@ -40,8 +40,8 @@ void sigintHandler(int signo)
static detail::ReplCompleterMixin * curRepl; // ugly
#if !USE_READLINE
static char * completionCallback(char * s, int * match)
{
static char * completionCallback(char * s, int * match) noexcept
try {
auto possible = curRepl->completePrefix(s);
if (possible.size() == 1) {
*match = 1;
@@ -73,10 +73,12 @@ static char * completionCallback(char * s, int * match)
*match = 0;
return nullptr;
} catch (...) {
return nullptr;
}
static int listPossibleCallback(char * s, char *** avp)
{
static int listPossibleCallback(char * s, char *** avp) noexcept
try {
auto possible = curRepl->completePrefix(s);
if (possible.size() > (std::numeric_limits<int>::max() / sizeof(char *)))
@@ -105,6 +107,9 @@ static int listPossibleCallback(char * s, char *** avp)
*avp = vp;
return ac;
} catch (...) {
*avp = nullptr;
return 0;
}
#endif
+3 -26
View File
@@ -12,9 +12,8 @@
#include "nix/expr/eval-settings.hh"
#include "nix/expr/attr-path.hh"
#include "nix/util/signals.hh"
#include "nix/store/store-open.hh"
#include "nix/store/log-store.hh"
#include "nix/cmd/common-eval-args.hh"
#include "nix/cmd/get-build-log.hh"
#include "nix/expr/get-drvs.hh"
#include "nix/store/derivations.hh"
#include "nix/store/globals.hh"
@@ -564,31 +563,9 @@ ProcessLineResult NixRepl::processLine(std::string line)
} else if (command == ":log") {
settings.readOnlyMode = true;
Finally roModeReset([&]() { settings.readOnlyMode = false; });
auto subs = getDefaultSubstituters();
subs.push_front(state->store);
bool foundLog = false;
RunPager pager;
for (auto & sub : subs) {
auto * logSubP = dynamic_cast<LogStore *>(&*sub);
if (!logSubP) {
printInfo(
"Skipped '%s' which does not support retrieving build logs", sub->config.getHumanReadableURI());
continue;
}
auto & logSub = *logSubP;
auto log = logSub.getBuildLog(drvPath);
if (log) {
printInfo("got build log for '%s' from '%s'", drvPathRaw, logSub.config.getHumanReadableURI());
logger->writeToStdout(*log);
foundLog = true;
break;
}
}
if (!foundLog)
throw Error("build log of '%s' is not available", drvPathRaw);
auto log = fetchBuildLog(state->store, drvPath, drvPathRaw);
logger->writeToStdout(log);
} else {
runNix("nix-shell", {drvPathRaw});
}
+126
View File
@@ -0,0 +1,126 @@
///@file
#include "nix/cmd/unix-socket-server.hh"
#include "nix/util/environment-variables.hh"
#include "nix/util/file-system.hh"
#include "nix/util/logging.hh"
#include "nix/util/signals.hh"
#include "nix/util/strings.hh"
#include "nix/util/unix-domain-socket.hh"
#include "nix/util/util.hh"
#include <sys/socket.h>
#include <sys/un.h>
#include <poll.h>
#if defined(__APPLE__) || defined(__FreeBSD__)
# include <sys/ucred.h>
#endif
namespace nix::unix {
PeerInfo getPeerInfo(Descriptor remote)
{
PeerInfo peer;
#if defined(SO_PEERCRED)
# if defined(__OpenBSD__)
struct sockpeercred cred;
# else
ucred cred;
# endif
socklen_t credLen = sizeof(cred);
if (getsockopt(remote, SOL_SOCKET, SO_PEERCRED, &cred, &credLen) == 0) {
peer.pid = cred.pid;
peer.uid = cred.uid;
peer.gid = cred.gid;
}
#elif defined(LOCAL_PEERCRED)
# if !defined(SOL_LOCAL)
# define SOL_LOCAL 0
# endif
xucred cred;
socklen_t credLen = sizeof(cred);
if (getsockopt(remote, SOL_LOCAL, LOCAL_PEERCRED, &cred, &credLen) == 0)
peer.uid = cred.cr_uid;
#endif
return peer;
}
[[noreturn]] void serveUnixSocket(const ServeUnixSocketOptions & options, UnixSocketHandler handler)
{
std::vector<AutoCloseFD> listeningSockets;
static constexpr int SD_LISTEN_FDS_START = 3;
// Handle socket-based activation by systemd.
auto listenFds = getEnv("LISTEN_FDS");
if (listenFds) {
if (getEnv("LISTEN_PID") != std::to_string(getpid()))
throw Error("unexpected systemd environment variables");
auto count = string2Int<unsigned int>(*listenFds);
assert(count);
for (unsigned int i = 0; i < count; ++i) {
AutoCloseFD fdSocket(SD_LISTEN_FDS_START + i);
closeOnExec(fdSocket.get());
listeningSockets.push_back(std::move(fdSocket));
}
}
// Otherwise, create and bind to a Unix domain socket.
else {
createDirs(options.socketPath.parent_path());
listeningSockets.push_back(createUnixDomainSocket(options.socketPath.string(), options.socketMode));
}
std::vector<struct pollfd> fds;
for (auto & i : listeningSockets)
fds.push_back({.fd = i.get(), .events = POLLIN});
// Loop accepting connections.
while (1) {
try {
checkInterrupt();
auto count = poll(fds.data(), fds.size(), -1);
if (count == -1) {
if (errno == EINTR)
continue;
throw SysError("polling for incoming connections");
}
for (auto & fd : fds) {
if (!fd.revents)
continue;
// Accept a connection.
struct sockaddr_un remoteAddr;
socklen_t remoteAddrLen = sizeof(remoteAddr);
AutoCloseFD remote = accept(fd.fd, (struct sockaddr *) &remoteAddr, &remoteAddrLen);
checkInterrupt();
if (!remote) {
if (errno == EINTR)
continue;
throw SysError("accepting connection");
}
handler(std::move(remote), [&]() { listeningSockets.clear(); });
}
} catch (Error & error) {
auto ei = error.info();
// FIXME: add to trace?
ei.msg = HintFmt("while processing connection: %1%", ei.msg.str());
logError(ei);
}
}
}
} // namespace nix::unix
+1 -1
View File
@@ -756,7 +756,7 @@ TEST_F(PrimOpTest, langVersion)
TEST_F(PrimOpTest, storeDir)
{
auto v = eval("builtins.storeDir");
ASSERT_THAT(v, IsStringEq(settings.nixStore));
ASSERT_THAT(v, IsStringEq(state.store->storeDir));
}
TEST_F(PrimOpTest, nixVersion)
+1 -2
View File
@@ -13,8 +13,7 @@ TEST_F(ValueTest, unsetValue)
{
Value unsetValue;
ASSERT_EQ(false, unsetValue.isValid());
ASSERT_EQ(nThunk, unsetValue.type(true));
ASSERT_DEATH(unsetValue.type(), "");
ASSERT_EQ(nThunk, unsetValue.type</*invalidIsThunk=*/true>());
}
TEST_F(ValueTest, vInt)
+1 -1
View File
@@ -11,7 +11,7 @@
namespace nix::eval_cache {
CachedEvalError::CachedEvalError(ref<AttrCursor> cursor, Symbol attr)
: EvalError(cursor->root->state, "cached failure of attribute '%s'", cursor->getAttrPathStr(attr))
: CloneableError(cursor->root->state, "cached failure of attribute '%s'", cursor->getAttrPathStr(attr))
, cursor(cursor)
, attr(attr)
{
+3 -2
View File
@@ -71,8 +71,9 @@ Strings EvalSettings::getDefaultNixPath()
};
add(std::filesystem::path{getNixDefExpr()} / "channels");
add(rootChannelsDir() / "nixpkgs", "nixpkgs");
add(rootChannelsDir());
auto profilesDirOpts = settings.getProfileDirsOptions();
add(rootChannelsDir(profilesDirOpts) / "nixpkgs", "nixpkgs");
add(rootChannelsDir(profilesDirOpts));
return res;
}
+4 -5
View File
@@ -318,7 +318,6 @@ EvalState::EvalState(
countCalls = getEnv("NIX_COUNT_CALLS").value_or("0") != "0";
static_assert(sizeof(Env) <= 16, "environment must be <= 16 bytes");
static_assert(sizeof(Counter) == 64, "counters must be 64 bytes");
/* Construct the Nix expression search path. */
assert(lookupPath.elements.empty());
@@ -467,7 +466,7 @@ void EvalState::addConstant(const std::string & name, Value * v, Constant info)
We might know the type of a thunk in advance, so be allowed
to just write it down in that case. */
if (auto gotType = v->type(true); gotType != nThunk)
if (auto gotType = v->type</*invalidIsThunk=*/true>(); gotType != nThunk)
assert(info.type == gotType);
/* Install value the base environment. */
@@ -886,7 +885,7 @@ void Value::mkPath(const SourcePath & path, EvalMemory & mem)
mkPath(&*path.accessor, StringData::make(mem, path.path.abs()));
}
inline Value * EvalState::lookupVar(Env * env, const ExprVar & var, bool noEval)
[[gnu::always_inline]] inline Value * EvalState::lookupVar(Env * env, const ExprVar & var, bool noEval)
{
for (auto l = var.level; l; --l, env = env->up)
;
@@ -904,11 +903,11 @@ inline Value * EvalState::lookupVar(Env * env, const ExprVar & var, bool noEval)
while (1) {
forceAttrs(*env->values[0], fromWith->pos, "while evaluating the first subexpression of a with expression");
if (auto j = env->values[0]->attrs()->get(var.name)) {
if (countCalls)
if (countCalls) [[unlikely]]
attrSelects[j->pos]++;
return j->value;
}
if (!fromWith->parentWith)
if (!fromWith->parentWith) [[unlikely]]
error<UndefinedVarError>("undefined variable '%1%'", symbols[var.name])
.atPos(var.pos)
.withFrame(*env, var)
+1 -1
View File
@@ -11,7 +11,7 @@ namespace nix {
* variable is set. This is to prevent contention on these counters
* when multi-threaded evaluation is enabled.
*/
struct alignas(64) Counter
struct alignas(std::hardware_destructive_interference_size) Counter
{
using value_type = uint64_t;
+1 -1
View File
@@ -14,7 +14,7 @@ namespace nix::eval_cache {
struct AttrDb;
class AttrCursor;
struct CachedEvalError : EvalError
struct CachedEvalError : CloneableError<CachedEvalError, EvalError>
{
const ref<AttrCursor> cursor;
const Symbol attr;
+7 -7
View File
@@ -18,7 +18,7 @@ class EvalErrorBuilder;
*
* Most subclasses should inherit from `EvalError` instead of this class.
*/
class EvalBaseError : public Error
class EvalBaseError : public CloneableError<EvalBaseError, Error>
{
template<class T>
friend class EvalErrorBuilder;
@@ -26,14 +26,14 @@ public:
EvalState & state;
EvalBaseError(EvalState & state, ErrorInfo && errorInfo)
: Error(errorInfo)
: CloneableError(errorInfo)
, state(state)
{
}
template<typename... Args>
explicit EvalBaseError(EvalState & state, const std::string & formatString, const Args &... formatArgs)
: Error(formatString, formatArgs...)
: CloneableError(formatString, formatArgs...)
, state(state)
{
}
@@ -60,23 +60,23 @@ MakeError(InfiniteRecursionError, EvalError);
* Inherits from EvalBaseError (not EvalError) because resource exhaustion
* should not be cached.
*/
struct StackOverflowError : public EvalBaseError
struct StackOverflowError : public CloneableError<StackOverflowError, EvalBaseError>
{
StackOverflowError(EvalState & state)
: EvalBaseError(state, "stack overflow; max-call-depth exceeded")
: CloneableError(state, "stack overflow; max-call-depth exceeded")
{
}
};
MakeError(IFDError, EvalBaseError);
struct InvalidPathError : public EvalError
struct InvalidPathError : public CloneableError<InvalidPathError, EvalError>
{
public:
Path path;
InvalidPathError(EvalState & state, const Path & path)
: EvalError(state, "path '%s' is not valid", path)
: CloneableError(state, "path '%s' is not valid", path)
{
}
};
+50 -47
View File
@@ -35,7 +35,7 @@ class BindingsBuilder;
* about how this is mapped into the alignment bits to save significant memory.
* This also restricts the number of internal types represented with distinct memory layouts.
*/
typedef enum {
enum InternalType {
tUninitialized = 0,
/* layout: Single/zero field payload */
tInt = 1,
@@ -46,16 +46,20 @@ typedef enum {
tPrimOp,
tAttrs,
/* layout: Pair of pointers payload */
tListSmall,
tFirstPairOfPointers,
tListSmall = tFirstPairOfPointers,
tPrimOpApp,
tApp,
tThunk,
tLambda,
tLastPairOfPointers = tLambda,
/* layout: Single untaggable field */
tListN,
tFirstSingleUntaggable,
tListN = tFirstSingleUntaggable,
tString,
tPath,
} InternalType;
tNumberOfInternalTypes, // Must be last
};
/**
* This type abstracts over all actual value types in the language,
@@ -134,7 +138,7 @@ public:
virtual bool operator==(const ExternalValueBase & b) const noexcept;
/**
* Print the value as JSON. Defaults to unconvertable, i.e. throws an error
* Print the value as JSON. Defaults to unconvertible, i.e. throws an error
*/
virtual nlohmann::json
printValueAsJSON(EvalState & state, bool strict, NixStringContext & context, bool copyToStore = true) const;
@@ -587,7 +591,7 @@ class alignas(16)
enum PrimaryDiscriminator : int {
pdUninitialized = 0,
pdSingleDWord, //< layout: Single/zero field payload
/* The order of these enumations must be the same as in InternalType. */
/* The order of these enumerations must be the same as in InternalType. */
pdListN, //< layout: Single untaggable field.
pdString,
pdPath,
@@ -633,7 +637,7 @@ class alignas(16)
template<InternalType type, typename T, typename U>
void setPairOfPointersPayload(T * firstPtrField, U * secondPtrField) noexcept
{
static_assert(type >= tListSmall && type <= tLambda);
static_assert(type >= tFirstPairOfPointers && type <= tLastPairOfPointers);
{
auto firstFieldPayload = std::bit_cast<PackedPointer>(firstPtrField);
assertAligned(firstFieldPayload);
@@ -642,7 +646,7 @@ class alignas(16)
{
auto secondFieldPayload = std::bit_cast<PackedPointer>(secondPtrField);
assertAligned(secondFieldPayload);
payload[1] = (type - tListSmall) | secondFieldPayload;
payload[1] = (type - tFirstPairOfPointers) | secondFieldPayload;
}
}
@@ -670,11 +674,11 @@ protected:
case pdListN:
case pdString:
case pdPath:
return static_cast<InternalType>(tListN + (pd - pdListN));
return static_cast<InternalType>(tFirstSingleUntaggable + (pd - pdListN));
case pdPairOfPointers:
return static_cast<InternalType>(tListSmall + (payload[1] & discriminatorMask));
return static_cast<InternalType>(tFirstPairOfPointers + (payload[1] & discriminatorMask));
[[unlikely]] default:
unreachable();
nixUnreachableWhenHardened();
}
}
@@ -1027,7 +1031,7 @@ private:
T getStorage() const noexcept
{
if (getInternalType() != detail::payloadTypeToInternalType<T>) [[unlikely]]
unreachable();
nixUnreachableWhenHardened();
T out;
ValueStorage::getStorage(out);
return out;
@@ -1079,45 +1083,44 @@ public:
* Returns the normal type of a Value. This only returns nThunk if
* the Value hasn't been forceValue'd
*
* @param invalidIsThunk Instead of aborting an an invalid (probably
* @param invalidIsThunk Instead of UB an an invalid (probably
* 0, so uninitialized) internal type, return `nThunk`.
*/
inline ValueType type(bool invalidIsThunk = false) const
template<bool invalidIsThunk = false>
inline ValueType type() const
{
switch (getInternalType()) {
case tUninitialized:
break;
case tInt:
return nInt;
case tBool:
return nBool;
case tString:
return nString;
case tPath:
return nPath;
case tNull:
return nNull;
case tAttrs:
return nAttrs;
case tListSmall:
case tListN:
return nList;
case tLambda:
case tPrimOp:
case tPrimOpApp:
return nFunction;
case tExternal:
return nExternal;
case tFloat:
return nFloat;
case tThunk:
case tApp:
return nThunk;
/* Explicit lookup table. switch() might compile down (and it does at least with GCC 14)
to a jump table. Let's help the compiler a bit here. */
static constexpr auto table = [] {
std::array<ValueType, tNumberOfInternalTypes> t{};
t[tUninitialized] = nThunk;
t[tInt] = nInt;
t[tBool] = nBool;
t[tNull] = nNull;
t[tFloat] = nFloat;
t[tExternal] = nExternal;
t[tAttrs] = nAttrs;
t[tPrimOp] = nFunction;
t[tLambda] = nFunction;
t[tPrimOpApp] = nFunction;
t[tApp] = nThunk;
t[tThunk] = nThunk;
t[tListSmall] = nList;
t[tListN] = nList;
t[tString] = nString;
t[tPath] = nPath;
return t;
}();
auto it = getInternalType();
if (it == tUninitialized || it >= tNumberOfInternalTypes) [[unlikely]] {
if constexpr (invalidIsThunk)
return nThunk;
else
nixUnreachableWhenHardened();
}
if (invalidIsThunk)
return nThunk;
else
unreachable();
return table[it];
}
/**
@@ -9,14 +9,14 @@
namespace nix {
class BadNixStringContextElem : public Error
class BadNixStringContextElem final : public CloneableError<BadNixStringContextElem, Error>
{
public:
std::string_view raw;
template<typename... Args>
BadNixStringContextElem(std::string_view raw_, const Args &... args)
: Error("")
: CloneableError("")
{
raw = raw_;
auto hf = HintFmt(args...);
+1 -1
View File
@@ -28,7 +28,7 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/big-objs')
# Check for each of these functions, and create a define like `#define HAVE_LCHOWN 1`.
# Check for each of these functions, and create a define like `#define HAVE_SYSCONF 1`.
check_funcs = [
'sysconf',
]
+13 -7
View File
@@ -1312,11 +1312,12 @@ static void prim_warn(EvalState & state, const PosIdx pos, Value ** args, Value
state.forceString(*args[0], pos, "while evaluating the first argument; the message passed to builtins.warn");
{
BaseError msg(std::string{msgStr});
msg.atPos(state.positions[pos]);
auto info = msg.info();
info.level = lvlWarn;
info.isFromExpr = true;
ErrorInfo info{
.level = lvlWarn,
.msg = HintFmt(std::string(msgStr)),
.pos = state.positions[pos],
.isFromExpr = true,
};
logWarning(info);
}
@@ -1814,8 +1815,13 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
drv.fillInOutputPaths(*state.store);
}
/* Write the resulting term into the Nix store directory. */
auto drvPath = writeDerivation(*state.store, drv, state.repair);
/* Write the resulting term into the Nix store directory.
Unless we are in read-only mode, that is, in which case we do not
write anything. Users commonly do this to speed up evaluation in
contexts where they don't actually want to build anything. */
auto drvPath =
settings.readOnlyMode ? computeStorePath(*state.store, drv) : state.store->writeDerivation(drv, state.repair);
auto drvPathS = state.store->printStorePath(drvPath);
printMsg(lvlChatty, "instantiated '%1%' -> '%2%'", drvName, drvPathS);
+13 -7
View File
@@ -191,19 +191,25 @@ static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value ** args
{.msg = HintFmt("attribute '%s' is missing in call to 'fetchClosure'", "fromStore"),
.pos = state.positions[pos]});
auto parsedURL = parseURL(*fromStoreUrl, /*lenient=*/true);
auto storeRef = StoreReference::parse(*fromStoreUrl);
if (parsedURL.scheme != "http" && parsedURL.scheme != "https"
&& !(getEnv("_NIX_IN_TEST").has_value() && parsedURL.scheme == "file"))
throw Error(
{.msg = HintFmt("'fetchClosure' only supports http:// and https:// stores"), .pos = state.positions[pos]});
if ([&] {
auto * specified = std::get_if<StoreReference::Specified>(&storeRef.variant);
return !specified
|| (specified->scheme != "http" && specified->scheme != "https"
&& !(getEnv("_NIX_IN_TEST").has_value() && specified->scheme == "file"));
}())
throw Error({
.msg = HintFmt("'fetchClosure' only supports http:// and https:// stores"),
.pos = state.positions[pos],
});
if (!parsedURL.query.empty())
if (!storeRef.params.empty())
throw Error(
{.msg = HintFmt("'fetchClosure' does not support URL query parameters (in '%s')", *fromStoreUrl),
.pos = state.positions[pos]});
auto fromStore = openStore(parsedURL.to_string());
auto fromStore = openStore(std::move(storeRef));
if (toPath)
runFetchClosureWithRewrite(state, pos, *fromStore, *fromPath, *toPath, v);
+9 -3
View File
@@ -34,14 +34,20 @@ struct CacheImpl : Cache
Sync<State> _state;
CacheImpl()
/**
* This is a back-reference to the `Settings` that owns us.
*/
const Settings & settings;
CacheImpl(const Settings & _settings)
: settings(_settings)
{
auto state(_state.lock());
auto dbPath = (getCacheDir() / "fetcher-cache-v4.sqlite").string();
createDirs(dirOf(dbPath));
state->db = SQLite(dbPath, {.useWAL = settings.useSQLiteWAL});
state->db = SQLite(dbPath, {.useWAL = nix::settings.useSQLiteWAL});
state->db.isCache();
state->db.exec(schema);
@@ -154,7 +160,7 @@ ref<Cache> Settings::getCache() const
{
auto cache(_cache.lock());
if (!*cache)
*cache = std::make_shared<CacheImpl>();
*cache = std::make_shared<CacheImpl>(*this);
return ref<Cache>(*cache);
}
+6
View File
@@ -47,6 +47,12 @@ std::pair<StorePath, Hash> fetchToStore2(
auto hash = Hash::parseSRI(fetchers::getStrAttr(*res, "hash"));
auto storePath =
store.makeFixedOutputPathFromCA(name, ContentAddressWithReferences::fromParts(method, hash, {}));
/* Add a temproot before the call to isValidPath to prevent accidental GC in case the
input is cached. Note that this must be done before to avoid races. */
if (mode != FetchMode::DryRun)
store.addTempRoot(storePath);
if (mode == FetchMode::DryRun || store.isValidPath(storePath)) {
debug(
"source path '%s' cache hit in '%s' (hash '%s')",
+63 -45
View File
@@ -74,6 +74,29 @@ namespace nix {
struct GitSourceAccessor;
struct GitError final : public CloneableError<GitError, Error>
{
template<typename... Ts>
GitError(const git_error & error, Ts &&... args)
: CloneableError("")
{
auto hf = HintFmt(std::forward<Ts>(args)...);
err.msg = HintFmt("%1%: %2% (libgit2 error code = %3%)", Uncolored(hf.str()), error.message, error.klass);
}
template<typename... Ts>
GitError(Ts &&... args)
: GitError(
[]() -> const git_error & {
const git_error * p = git_error_last();
assert(p && "git_error_last() is unexpectedly null");
return *p;
}(),
std::forward<Ts>(args)...)
{
}
};
typedef std::unique_ptr<git_repository, Deleter<git_repository_free>> Repository;
typedef std::unique_ptr<git_tree_entry, Deleter<git_tree_entry_free>> TreeEntry;
typedef std::unique_ptr<git_tree, Deleter<git_tree_free>> Tree;
@@ -106,7 +129,7 @@ static void initLibGit2()
static std::once_flag initialized;
std::call_once(initialized, []() {
if (git_libgit2_init() < 0)
throw Error("initialising libgit2: %s", git_error_last()->message);
throw GitError("initialising libgit2");
});
}
@@ -114,7 +137,7 @@ static git_oid hashToOID(const Hash & hash)
{
git_oid oid;
if (git_oid_fromstr(&oid, hash.gitRev().c_str()))
throw Error("cannot convert '%s' to a Git OID", hash.gitRev());
throw GitError("cannot convert '%s' to a Git OID", hash.gitRev());
return oid;
}
@@ -122,8 +145,7 @@ static Object lookupObject(git_repository * repo, const git_oid & oid, git_objec
{
Object obj;
if (git_object_lookup(Setter(obj), repo, &oid, type)) {
auto err = git_error_last();
throw Error("getting Git object '%s': %s", oid, err->message);
throw GitError("getting Git object '%s'", oid);
}
return obj;
}
@@ -133,8 +155,7 @@ static T peelObject(git_object * obj, git_object_t type)
{
T obj2;
if (git_object_peel((git_object **) (typename T::pointer *) Setter(obj2), obj, type)) {
auto err = git_error_last();
throw Error("peeling Git object '%s': %s", *git_object_id(obj), err->message);
throw Error("peeling Git object '%s'", *git_object_id(obj));
}
return obj2;
}
@@ -144,7 +165,7 @@ static T dupObject(typename T::pointer obj)
{
T obj2;
if (git_object_dup((git_object **) (typename T::pointer *) Setter(obj2), (git_object *) obj))
throw Error("duplicating object '%s': %s", *git_object_id((git_object *) obj), git_error_last()->message);
throw GitError("duplicating object '%s'", *git_object_id((git_object *) obj));
return obj2;
}
@@ -216,7 +237,7 @@ static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options op
Repository tmpRepo;
if (git_repository_init(Setter(tmpRepo), tmpDir.string().c_str(), options.bare))
throw Error("creating Git repository %s: %s", PathFmt(path), git_error_last()->message);
throw GitError("creating Git repository %s", PathFmt(path));
try {
std::filesystem::rename(tmpDir, path);
} catch (std::filesystem::filesystem_error & e) {
@@ -266,7 +287,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
initRepoAtomically(path, options);
if (git_repository_open(Setter(repo), path.string().c_str()))
throw Error("opening Git repository %s: %s", PathFmt(path), git_error_last()->message);
throw GitError("opening Git repository %s", PathFmt(path));
ObjectDb odb;
if (options.packfilesOnly) {
@@ -279,28 +300,28 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
*/
if (git_odb_new(Setter(odb)))
throw Error("creating Git object database: %s", git_error_last()->message);
throw GitError("creating Git object database");
if (git_odb_backend_pack(&packBackend, (path / "objects").string().c_str()))
throw Error("creating pack backend: %s", git_error_last()->message);
throw GitError("creating pack backend");
if (git_odb_add_backend(odb.get(), packBackend, 1))
throw Error("adding pack backend to Git object database: %s", git_error_last()->message);
throw GitError("adding pack backend to Git object database");
} else {
if (git_repository_odb(Setter(odb), repo.get()))
throw Error("getting Git object database: %s", git_error_last()->message);
throw GitError("getting Git object database");
}
// mempack_backend will be owned by the repository, so we are not expected to free it ourselves.
if (git_mempack_new(&mempackBackend))
throw Error("creating mempack backend: %s", git_error_last()->message);
throw GitError("creating mempack backend");
if (git_odb_add_backend(odb.get(), mempackBackend, 999))
throw Error("adding mempack backend to Git object database: %s", git_error_last()->message);
throw GitError("adding mempack backend to Git object database");
if (options.packfilesOnly) {
if (git_repository_set_odb(repo.get(), odb.get()))
throw Error("setting Git object database: %s", git_error_last()->message);
throw GitError("setting Git object database");
}
}
@@ -340,7 +361,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
Indexer indexer;
git_indexer_progress stats;
if (git_indexer_new(Setter(indexer), pack_dir_path.c_str(), 0, nullptr, nullptr))
throw Error("creating git packfile indexer: %s", git_error_last()->message);
throw GitError("creating git packfile indexer");
// TODO: provide index callback for checkInterrupt() termination
// though this is about an order of magnitude faster than the packbuilder
@@ -348,15 +369,15 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
constexpr size_t chunkSize = 128 * 1024;
for (size_t offset = 0; offset < buf.size; offset += chunkSize) {
if (git_indexer_append(indexer.get(), buf.ptr + offset, std::min(chunkSize, buf.size - offset), &stats))
throw Error("appending to git packfile index: %s", git_error_last()->message);
throw GitError("appending to git packfile index");
checkInterrupt();
}
if (git_indexer_commit(indexer.get(), &stats))
throw Error("committing git packfile index: %s", git_error_last()->message);
throw GitError("committing git packfile index");
if (git_mempack_reset(mempackBackend))
throw Error("resetting git mempack backend: %s", git_error_last()->message);
throw GitError("resetting git mempack backend");
checkInterrupt();
}
@@ -449,7 +470,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
void setRemote(const std::string & name, const std::string & url) override
{
if (git_remote_set_url(*this, name.c_str(), url.c_str()))
throw Error("setting remote '%s' URL to '%s': %s", name, url, git_error_last()->message);
throw GitError("setting remote '%s' URL to '%s'", name, url);
}
Hash resolveRef(std::string ref) override
@@ -462,7 +483,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
// an object_id.
std::string peeledRef = ref + "^{commit}";
if (git_revparse_single(Setter(object), *this, peeledRef.c_str()))
throw Error("resolving Git reference '%s': %s", ref, git_error_last()->message);
throw GitError("resolving Git reference '%s'", ref);
auto oid = git_object_id(object.get());
return toHash(*oid);
}
@@ -471,11 +492,11 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
{
GitConfig config;
if (git_config_open_ondisk(Setter(config), configFile.string().c_str()))
throw Error("parsing .gitmodules file: %s", git_error_last()->message);
throw GitError("parsing .gitmodules file");
ConfigIterator it;
if (git_config_iterator_glob_new(Setter(it), config.get(), "^submodule\\..*\\.(path|url|branch)$"))
throw Error("iterating over .gitmodules: %s", git_error_last()->message);
throw GitError("iterating over .gitmodules");
StringMap entries;
@@ -484,7 +505,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
if (auto err = git_config_next(&entry, it.get())) {
if (err == GIT_ITEROVER)
break;
throw Error("iterating over .gitmodules: %s", git_error_last()->message);
throw GitError("iterating over .gitmodules");
}
entries.emplace(entry->name + 10, entry->value);
}
@@ -521,7 +542,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
git_oid headRev;
if (auto err = git_reference_name_to_id(&headRev, *this, "HEAD")) {
if (err != GIT_ENOTFOUND)
throw Error("resolving HEAD: %s", git_error_last()->message);
throw GitError("resolving HEAD");
} else
info.headRev = toHash(headRev);
@@ -544,7 +565,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
options.flags |= GIT_STATUS_OPT_INCLUDE_UNMODIFIED;
options.flags |= GIT_STATUS_OPT_EXCLUDE_SUBMODULES;
if (git_status_foreach_ext(*this, &options, &statusCallbackTrampoline, &statusCallback))
throw Error("getting working directory status: %s", git_error_last()->message);
throw GitError("getting working directory status");
/* Get submodule info. */
auto modulesFile = path / ".gitmodules";
@@ -587,8 +608,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
if (auto errCode = git_object_lookup(Setter(obj), *this, &oid, GIT_OBJECT_ANY)) {
if (errCode == GIT_ENOTFOUND)
return false;
auto err = git_error_last();
throw Error("getting Git object '%s': %s", oid, err->message);
throw GitError("getting Git object '%s'", oid);
}
return true;
@@ -918,7 +938,7 @@ struct GitSourceAccessor : SourceAccessor
TreeEntry copy;
if (git_tree_entry_dup(Setter(copy), entry))
throw Error("dupping tree entry: %s", git_error_last()->message);
throw GitError("dupping tree entry");
auto entryName = std::string_view(git_tree_entry_name(entry));
@@ -948,7 +968,7 @@ struct GitSourceAccessor : SourceAccessor
Tree tree;
if (git_tree_entry_to_object((git_object **) (git_tree **) Setter(tree), *state.repo, entry))
throw Error("looking up directory '%s': %s", showPath(path), git_error_last()->message);
throw GitError("looking up directory '%s'", showPath(path));
return tree;
}
@@ -983,7 +1003,7 @@ struct GitSourceAccessor : SourceAccessor
Tree tree;
if (git_tree_entry_to_object((git_object **) (git_tree **) Setter(tree), *state.repo, entry))
throw Error("looking up directory '%s': %s", showPath(path), git_error_last()->message);
throw GitError("looking up directory '%s'", showPath(path));
return tree;
}
@@ -1016,7 +1036,7 @@ struct GitSourceAccessor : SourceAccessor
Blob blob;
if (git_tree_entry_to_object((git_object **) (git_blob **) Setter(blob), *state.repo, entry))
throw Error("looking up file '%s': %s", showPath(path), git_error_last()->message);
throw GitError("looking up file '%s'", showPath(path));
return blob;
}
@@ -1068,7 +1088,7 @@ struct GitExportIgnoreSourceAccessor : CachingFilteringSourceAccessor
if (git_error_last()->klass == GIT_ENOTFOUND)
return false;
else
throw Error("looking up '%s': %s", showPath(path), git_error_last()->message);
throw GitError("looking up '%s'", showPath(path));
} else {
// Official git will silently reject export-ignore lines that have
// values. We do the same.
@@ -1224,17 +1244,17 @@ struct GitFileSystemObjectSinkImpl : GitFileSystemObjectSink
repo.emplace(parent.repoPool.get());
if (git_blob_create_from_stream(Setter(stream), **repo, nullptr))
throw Error("creating a blob stream object: %s", git_error_last()->message);
throw GitError("creating a blob stream object");
if (stream->write(stream.get(), contents.data(), contents.size()))
throw Error("writing a blob for tarball member '%s': %s", path, git_error_last()->message);
throw GitError("writing a blob for tarball member '%s'", path);
parent.totalBufSize -= contents.size();
contents.clear();
}
} else {
if (stream->write(stream.get(), data.data(), data.size()))
throw Error("writing a blob for tarball member '%s': %s", path, git_error_last()->message);
throw GitError("writing a blob for tarball member '%s'", path);
}
}
@@ -1256,7 +1276,7 @@ struct GitFileSystemObjectSinkImpl : GitFileSystemObjectSink
acquires ownership and frees the stream. */
git_oid oid;
if (git_blob_create_from_stream_commit(&oid, crf->stream.release()))
throw Error("creating a blob object for '%s': %s", path, git_error_last()->message);
throw GitError("creating a blob object for '%s'", path);
addNode(
*_state.lock(),
crf->path,
@@ -1270,8 +1290,7 @@ struct GitFileSystemObjectSinkImpl : GitFileSystemObjectSink
git_oid oid;
if (git_blob_create_from_buffer(&oid, *repo, crf->contents.data(), crf->contents.size()))
throw Error(
"creating a blob object for '%s' from in-memory buffer: %s", crf->path, git_error_last()->message);
throw GitError("creating a blob object for '%s' from in-memory buffer", crf->path);
addNode(
*_state.lock(),
@@ -1295,8 +1314,7 @@ struct GitFileSystemObjectSinkImpl : GitFileSystemObjectSink
git_oid oid;
if (git_blob_create_from_buffer(&oid, *repo, target.c_str(), target.size()))
throw Error(
"creating a blob object for tarball symlink member '%s': %s", path, git_error_last()->message);
throw GitError("creating a blob object for tarball symlink member '%s'", path);
auto state(_state.lock());
addNode(*state, path, Child{GIT_FILEMODE_LINK, oid});
@@ -1357,19 +1375,19 @@ struct GitFileSystemObjectSinkImpl : GitFileSystemObjectSink
// Write this directory.
git_treebuilder * b;
if (git_treebuilder_new(&b, *repo, nullptr))
throw Error("creating a tree builder: %s", git_error_last()->message);
throw GitError("creating a tree builder");
TreeBuilder builder(b);
for (auto & [name, child] : node.children) {
auto oid_p = std::get_if<git_oid>(&child.file);
auto oid = oid_p ? *oid_p : std::get<Directory>(child.file).oid.value();
if (git_treebuilder_insert(nullptr, builder.get(), name.c_str(), &oid, child.mode))
throw Error("adding a file to a tree builder: %s", git_error_last()->message);
throw GitError("adding a file to a tree builder");
}
git_oid oid;
if (git_treebuilder_write(&oid, builder.get()))
throw Error("creating a tree object: %s", git_error_last()->message);
throw GitError("creating a tree object");
node.oid = oid;
}(_state.lock()->root);
+7 -7
View File
@@ -37,7 +37,7 @@ namespace {
// old version of git, which will ignore unrecognized `-c` options.
const std::string gitInitialBranch = "__nix_dummy_branch";
bool isCacheFileWithinTtl(time_t now, const PosixStat & st)
static bool isCacheFileWithinTtl(const Settings & settings, time_t now, const PosixStat & st)
{
return st.st_mtime + static_cast<time_t>(settings.tarballTtl) > now;
}
@@ -105,7 +105,7 @@ bool storeCachedHead(const std::string & actualUrl, bool shallow, const std::str
return true;
}
std::optional<std::string> readHeadCached(const std::string & actualUrl, bool shallow)
static std::optional<std::string> readHeadCached(const Settings & settings, const std::string & actualUrl, bool shallow)
{
// Create a cache path to store the branch of the HEAD ref. Append something
// in front of the URL to prevent collision with the repository itself.
@@ -117,7 +117,7 @@ std::optional<std::string> readHeadCached(const std::string & actualUrl, bool sh
std::optional<std::string> cachedRef;
if (st) {
cachedRef = readHead(cacheDir);
if (cachedRef != std::nullopt && *cachedRef != gitInitialBranch && isCacheFileWithinTtl(now, *st)) {
if (cachedRef != std::nullopt && *cachedRef != gitInitialBranch && isCacheFileWithinTtl(settings, now, *st)) {
debug("using cached HEAD ref '%s' for repo '%s'", *cachedRef, actualUrl);
return cachedRef;
}
@@ -728,12 +728,12 @@ struct GitInputScheme : InputScheme
return revCount;
}
std::string getDefaultRef(const RepoInfo & repoInfo, bool shallow) const
std::string getDefaultRef(const Settings & settings, const RepoInfo & repoInfo, bool shallow) const
{
auto head = std::visit(
overloaded{
[&](const std::filesystem::path & path) { return GitRepo::openRepo(path, {})->getWorkdirRef(); },
[&](const ParsedURL & url) { return readHeadCached(url.to_string(), shallow); }},
[&](const ParsedURL & url) { return readHeadCached(settings, url.to_string(), shallow); }},
repoInfo.location);
if (!head) {
warn("could not read HEAD ref from repo at '%s', using 'master'", repoInfo.locationToArg());
@@ -783,7 +783,7 @@ struct GitInputScheme : InputScheme
auto originalRef = input.getRef();
bool shallow = getShallowAttr(input);
auto ref = originalRef ? *originalRef : getDefaultRef(repoInfo, shallow);
auto ref = originalRef ? *originalRef : getDefaultRef(settings, repoInfo, shallow);
input.attrs.insert_or_assign("ref", ref);
std::filesystem::path repoDir;
@@ -822,7 +822,7 @@ struct GitInputScheme : InputScheme
/* If the local ref is older than 'tarball-ttl' seconds, do a
git fetch to update the local ref to the remote ref. */
auto st = maybeStat(localRefFile);
doFetch = !st || !isCacheFileWithinTtl(now, *st);
doFetch = !st || !isCacheFileWithinTtl(settings, now, *st);
}
}
@@ -129,6 +129,25 @@ struct Settings : public Config
true,
Xp::Flakes};
Setting<unsigned int> tarballTtl{
this,
60 * 60,
"tarball-ttl",
R"(
The number of seconds a downloaded tarball is considered fresh. If
the cached tarball is stale, Nix checks whether it is still up
to date using the ETag header. Nix downloads a new version if
the ETag header is unsupported, or the cached ETag doesn't match.
Setting the TTL to `0` forces Nix to always check if the tarball is
up to date.
Nix caches tarballs in `$XDG_CACHE_HOME/nix/tarballs`.
Files fetched via `NIX_PATH`, `fetchGit`, `fetchMercurial`,
`fetchTarball`, and `fetchurl` respect this TTL.
)"};
ref<Cache> getCache() const;
ref<GitRepo> getTarballCache() const;
+3 -2
View File
@@ -1,5 +1,6 @@
#include "nix/fetchers/fetchers.hh"
#include "nix/util/processes.hh"
#include "nix/util/environment-variables.hh"
#include "nix/util/users.hh"
#include "nix/fetchers/cache.hh"
#include "nix/store/globals.hh"
@@ -16,9 +17,9 @@ namespace nix::fetchers {
static RunOptions hgOptions(const Strings & args)
{
auto env = getEnv();
auto env = getEnvOs();
// Set HGPLAIN: this means we get consistent output from hg and avoids leakage from a user or system .hgrc.
env["HGPLAIN"] = "";
env[OS_STR("HGPLAIN")] = OS_STR("");
return {.program = "hg", .lookupPath = true, .args = args, .environment = env};
}
+1 -1
View File
@@ -92,7 +92,7 @@ void Registry::remove(const Input & input)
static std::filesystem::path getSystemRegistryPath()
{
return settings.nixConfDir / "registry.json";
return nixConfDir() / "registry.json";
}
static std::shared_ptr<Registry> getSystemRegistry(const Settings & settings)
+14
View File
@@ -283,4 +283,18 @@ TEST(to_string, doesntReencodeUrl)
ASSERT_EQ(unparsed, expected);
}
TEST(parseFlakeRef, malformedGithubUrlDoesNotCrash)
{
experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes);
fetchers::Settings fetchSettings;
// Using ref= instead of rev= with a github: URL should produce an
// error, not an assertion failure in renderAuthorityAndPath
// (https://github.com/NixOS/nix/issues/15196).
EXPECT_THROW(
parseFlakeRef(fetchSettings, "github:nixos/nixpkgs/nixpkgs.git?ref=aead170c1a49253ebfa5027010dfd89a77b73ca4"),
Error);
}
} // namespace nix
+1 -1
View File
@@ -205,7 +205,7 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
fetchSettings,
{
.scheme = "path",
.authority = ParsedURL::Authority{},
.authority = isAbsolute(path) ? std::optional{ParsedURL::Authority{}} : std::nullopt,
.path = splitString<std::vector<std::string>>(path, "/"),
.query = query,
.fragment = fragment,
+1 -15
View File
@@ -13,8 +13,6 @@
namespace nix {
int handleExceptions(const std::string & programName, std::function<void()> fun);
/**
* Don't forget to call initPlugins() after settings are initialized!
* @param loadConfig Whether to load configuration from `nix.conf`, `NIX_CONFIG`, etc. May be disabled for unit tests.
@@ -91,19 +89,7 @@ extern volatile ::sig_atomic_t blockInt;
struct GCResults;
struct PrintFreed
{
bool show;
const GCResults & results;
PrintFreed(bool show, const GCResults & results)
: show(show)
, results(results)
{
}
~PrintFreed();
};
void printFreed(bool dryRun, const GCResults & results);
#ifndef _WIN32
/**
+41 -37
View File
@@ -1,7 +1,9 @@
#include "nix/store/globals.hh"
#include "nix/util/current-process.hh"
#include "nix/util/executable-path.hh"
#include "nix/main/shared.hh"
#include "nix/store/store-api.hh"
#include "nix/store/store-open.hh"
#include "nix/store/gc-store.hh"
#include "nix/main/loggers.hh"
#include "nix/main/progress-bar.hh"
@@ -17,6 +19,10 @@
#include <sys/stat.h>
#include <unistd.h>
#include <signal.h>
#ifndef _WIN32
# include <sys/resource.h>
#endif
#ifdef __linux__
# include <features.h>
#endif
@@ -116,6 +122,26 @@ std::string getArg(const std::string & opt, Strings::iterator & i, const Strings
static void sigHandler(int signo) {}
#endif
/**
* Increase the open file soft limit to the hard limit. On some
* platforms (macOS), the default soft limit is very low, but the hard
* limit is high. So let's just raise it the maximum permitted.
*/
void bumpFileLimit()
{
#ifndef _WIN32
struct rlimit limit;
if (getrlimit(RLIMIT_NOFILE, &limit) != 0)
return;
if (limit.rlim_cur < limit.rlim_max) {
limit.rlim_cur = limit.rlim_max;
// Ignore errors, this is best effort.
setrlimit(RLIMIT_NOFILE, &limit);
}
#endif
}
void initNix(bool loadConfig)
{
/* Turn on buffering for cerr. */
@@ -183,6 +209,8 @@ void initNix(bool loadConfig)
now. In particular, store objects should be readable by
everybody. */
umask(0022);
bumpFileLimit();
}
LegacyArgs::LegacyArgs(
@@ -209,13 +237,13 @@ LegacyArgs::LegacyArgs(
.longName = "keep-going",
.shortName = 'k',
.description = "Keep going after a build fails.",
.handler = {&(bool &) settings.keepGoing, true},
.handler = {&(bool &) settings.getWorkerSettings().keepGoing, true},
});
addFlag({
.longName = "fallback",
.description = "Build from source if substitution fails.",
.handler = {&(bool &) settings.tryFallback, true},
.handler = {&(bool &) settings.getWorkerSettings().tryFallback, true},
});
auto intSettingAlias =
@@ -252,7 +280,7 @@ LegacyArgs::LegacyArgs(
.longName = "store",
.description = "The URL of the Nix store to use.",
.labels = {"store-uri"},
.handler = {&(std::string &) settings.storeUri},
.handler = {[](std::string s) { settings.storeUri = StoreReference::parse(s); }},
});
}
@@ -304,43 +332,16 @@ void printVersion(const std::string & programName)
std::cout << "System type: " << settings.thisSystem << "\n";
std::cout << "Additional system types: " << concatStringsSep(", ", settings.extraPlatforms.get()) << "\n";
std::cout << "Features: " << concatStringsSep(", ", cfg) << "\n";
std::cout << "System configuration file: " << (settings.nixConfDir / "nix.conf").string() << "\n";
std::cout << "User configuration files: " << concatStringsSep(":", settings.nixUserConfFiles) << "\n";
std::cout << "Store directory: " << settings.nixStore << "\n";
std::cout << "System configuration file: " << nixConfFile() << "\n";
std::cout << "User configuration files: "
<< os_string_to_string(ExecutablePath{.directories = nixUserConfFiles()}.render()) << "\n";
std::cout << "Store directory: " << resolveStoreConfig(StoreReference{settings.storeUri.get()})->storeDir
<< "\n";
std::cout << "State directory: " << settings.nixStateDir << "\n";
}
throw Exit();
}
int handleExceptions(const std::string & programName, std::function<void()> fun)
{
ReceiveInterrupts receiveInterrupts; // FIXME: need better place for this
ErrorInfo::programName = baseNameOf(programName);
std::string error = ANSI_RED "error:" ANSI_NORMAL " ";
try {
fun();
} catch (Exit & e) {
return e.status;
} catch (UsageError & e) {
logError(e.info());
printError("Try '%1% --help' for more information.", programName);
return 1;
} catch (BaseError & e) {
logError(e.info());
return e.info().status;
} catch (std::bad_alloc & e) {
printError(error + "out of memory");
return 1;
} catch (std::exception & e) {
printError(error + e.what());
return 1;
}
return 0;
}
RunPager::RunPager()
{
if (!isatty(STDOUT_FILENO))
@@ -395,9 +396,12 @@ RunPager::~RunPager()
}
}
PrintFreed::~PrintFreed()
void printFreed(bool dryRun, const GCResults & results)
{
if (show)
/* bytesFreed cannot be reliably computed without actually deleting store paths because of hardlinking. */
if (dryRun)
std::cout << fmt("%d store paths would be deleted\n", results.paths.size());
else
std::cout << fmt("%d store paths deleted, %s freed\n", results.paths.size(), renderSize(results.bytesFreed));
}
+35 -7
View File
@@ -9,6 +9,7 @@
#include "nix/store/path.hh"
#include "nix/store/store-api.hh"
#include "nix/store/store-open.hh"
#include "nix/store/store-reference.hh"
#include "nix/store/build-result.hh"
#include "nix/store/local-fs-store.hh"
#include "nix/util/base-nix-32.hh"
@@ -47,14 +48,14 @@ Store * nix_store_open(nix_c_context * context, const char * uri, const char ***
if (uri_str.empty())
return new Store{nix::openStore()};
if (!params)
return new Store{nix::openStore(uri_str)};
auto storeRef = nix::StoreReference::parse(uri_str);
nix::Store::Config::Params params_map;
for (size_t i = 0; params[i] != nullptr; i++) {
params_map[params[i][0]] = params[i][1];
if (params) {
for (size_t i = 0; params[i] != nullptr; i++) {
storeRef.params[params[i][0]] = params[i][1];
}
}
return new Store{nix::openStore(uri_str, params_map)};
return new Store{nix::openStore(std::move(storeRef))};
}
NIXC_CATCH_ERRS_NULL
}
@@ -307,7 +308,12 @@ StorePath * nix_add_derivation(nix_c_context * context, Store * store, nix_deriv
if (context)
context->last_err_code = NIX_OK;
try {
auto ret = nix::writeDerivation(*store->ptr, derivation->drv, nix::NoRepair);
/* Quite dubious that users would want this to silently suceed
without actually writing the derivation if this setting is
set, but it was that way already, so we are doing this for
back-compat for now. */
auto ret = nix::settings.readOnlyMode ? nix::computeStorePath(*store->ptr, derivation->drv)
: store->ptr->writeDerivation(derivation->drv, nix::NoRepair);
return new StorePath{ret};
}
@@ -352,4 +358,26 @@ StorePath * nix_store_query_path_from_hash_part(nix_c_context * context, Store *
NIXC_CATCH_ERRS_NULL
}
nix_err nix_store_copy_path(
nix_c_context * context, Store * srcStore, Store * dstStore, const StorePath * path, bool repair, bool checkSigs)
{
if (context)
context->last_err_code = NIX_OK;
try {
if (srcStore == nullptr)
return nix_set_err_msg(context, NIX_ERR_UNKNOWN, "Source store is null");
if (dstStore == nullptr)
return nix_set_err_msg(context, NIX_ERR_UNKNOWN, "Destination store is null");
if (path == nullptr)
return nix_set_err_msg(context, NIX_ERR_UNKNOWN, "Store path is null");
auto repairFlag = repair ? nix::RepairFlag::Repair : nix::RepairFlag::NoRepair;
auto checkSigsFlag = checkSigs ? nix::CheckSigsFlag::CheckSigs : nix::CheckSigsFlag::NoCheckSigs;
nix::copyStorePath(*srcStore->ptr, *dstStore->ptr, path->path, repairFlag, checkSigsFlag);
}
NIXC_CATCH_ERRS
}
} // extern "C"
+13
View File
@@ -270,6 +270,19 @@ nix_derivation * nix_store_drv_from_store_path(nix_c_context * context, Store *
*/
StorePath * nix_store_query_path_from_hash_part(nix_c_context * context, Store * store, const char * hash);
/**
* @brief Copy a path from one store to another.
*
* @param[out] context Optional, stores error information
* @param[in] srcStore nix source store reference
* @param[in] dstStore nix destination store reference
* @param[in] path The path to copy
* @param[in] repair Whether to repair the path
* @param[in] checkSigs Whether to check path signatures are trusted before copying
*/
nix_err nix_store_copy_path(
nix_c_context * context, Store * srcStore, Store * dstStore, const StorePath * path, bool repair, bool checkSigs);
// cffi end
#ifdef __cplusplus
}
+29 -10
View File
@@ -9,11 +9,12 @@ void TestHttpBinaryCacheStore::init()
BinaryCacheStore::init();
}
ref<TestHttpBinaryCacheStore> TestHttpBinaryCacheStoreConfig::openTestStore() const
ref<TestHttpBinaryCacheStore> TestHttpBinaryCacheStoreConfig::openTestStore(ref<FileTransfer> fileTransfer) const
{
auto store = make_ref<TestHttpBinaryCacheStore>(
ref{// FIXME we shouldn't actually need a mutable config
std::const_pointer_cast<HttpBinaryCacheStore::Config>(shared_from_this())});
std::const_pointer_cast<HttpBinaryCacheStore::Config>(shared_from_this())},
fileTransfer);
store->init();
return store;
}
@@ -49,9 +50,11 @@ void HttpsBinaryCacheStoreTest::SetUp()
// clang-format off
openssl({"ecparam", "-genkey", "-name", "prime256v1", "-out", caKey.string()});
openssl({"req", "-new", "-x509", "-days", "1", "-key", caKey.string(), "-out", caCert.string(), "-subj", "/CN=TestCA"});
auto serverExtFile = tmpDir / "server.ext";
writeFile(serverExtFile, "subjectAltName=DNS:localhost,IP:127.0.0.1");
openssl({"ecparam", "-genkey", "-name", "prime256v1", "-out", serverKey.string()});
openssl({"req", "-new", "-key", serverKey.string(), "-out", (tmpDir / "server.csr").string(), "-subj", "/CN=localhost"});
openssl({"x509", "-req", "-in", (tmpDir / "server.csr").string(), "-CA", caCert.string(), "-CAkey", caKey.string(), "-CAcreateserial", "-out", serverCert.string(), "-days", "1"});
openssl({"req", "-new", "-key", serverKey.string(), "-out", (tmpDir / "server.csr").string(), "-subj", "/CN=localhost", "-addext", "subjectAltName=DNS:localhost,IP:127.0.0.1"});
openssl({"x509", "-req", "-in", (tmpDir / "server.csr").string(), "-CA", caCert.string(), "-CAkey", caKey.string(), "-CAcreateserial", "-out", serverCert.string(), "-days", "1", "-extfile", serverExtFile.string()});
openssl({"ecparam", "-genkey", "-name", "prime256v1", "-out", clientKey.string()});
openssl({"req", "-new", "-key", clientKey.string(), "-out", (tmpDir / "client.csr").string(), "-subj", "/CN=TestClient"});
openssl({"x509", "-req", "-in", (tmpDir / "client.csr").string(), "-CA", caCert.string(), "-CAkey", caKey.string(), "-CAcreateserial", "-out", clientCert.string(), "-days", "1"});
@@ -79,16 +82,18 @@ void HttpsBinaryCacheStoreTest::SetUp()
for the port explicitly - this is enough. */
std::this_thread::sleep_for(std::chrono::milliseconds(50));
/* FIXME: Don't use global settings. Tests are not run concurrently, so this is fine for now. */
oldCaCert = settings.caFile;
settings.caFile = caCert.string();
/* Create custom FileTransferSettings with our test CA certificate.
This avoids mutating global settings. */
testFileTransferSettings = std::make_unique<FileTransferSettings>();
testFileTransferSettings->caFile = caCert;
testFileTransfer = makeFileTransfer(*testFileTransferSettings);
}
void HttpsBinaryCacheStoreTest::TearDown()
{
settings.caFile = oldCaCert;
serverPid.kill();
delTmpDir.reset();
testFileTransferSettings.reset();
}
std::vector<std::string> HttpsBinaryCacheStoreTest::serverArgs()
@@ -115,11 +120,25 @@ std::vector<std::string> HttpsBinaryCacheStoreMtlsTest::serverArgs()
return args;
}
ref<TestHttpBinaryCacheStoreConfig> HttpsBinaryCacheStoreTest::makeConfig(BinaryCacheStoreConfig::Params params)
ref<TestHttpBinaryCacheStoreConfig> HttpsBinaryCacheStoreTest::makeConfig()
{
auto res = make_ref<TestHttpBinaryCacheStoreConfig>("https", fmt("localhost:%d", port), std::move(params));
auto res = make_ref<TestHttpBinaryCacheStoreConfig>(
ParsedURL{
.scheme = "https",
.authority =
ParsedURL::Authority{
.host = "localhost",
.port = port,
},
},
TestHttpBinaryCacheStoreConfig::Params{});
res->pathInfoCacheSize = 0; /* We don't want any caching in tests. */
return res;
}
ref<TestHttpBinaryCacheStore> HttpsBinaryCacheStoreTest::openStore(ref<TestHttpBinaryCacheStoreConfig> config)
{
return config->openTestStore(ref<FileTransfer>{testFileTransfer});
}
} // namespace nix::testing
@@ -28,10 +28,10 @@ public:
TestHttpBinaryCacheStore & operator=(const TestHttpBinaryCacheStore &) = delete;
TestHttpBinaryCacheStore & operator=(TestHttpBinaryCacheStore &&) = delete;
TestHttpBinaryCacheStore(ref<HttpBinaryCacheStoreConfig> config)
TestHttpBinaryCacheStore(ref<HttpBinaryCacheStoreConfig> config, ref<FileTransfer> fileTransfer)
: Store{*config}
, BinaryCacheStore{*config}
, HttpBinaryCacheStore(config)
, HttpBinaryCacheStore(config, fileTransfer)
{
diskCache = nullptr; /* Disable caching, we'll be creating a new binary cache for each test. */
}
@@ -42,19 +42,13 @@ public:
class TestHttpBinaryCacheStoreConfig : public HttpBinaryCacheStoreConfig
{
public:
TestHttpBinaryCacheStoreConfig(
std::string_view scheme, std::string_view cacheUri, const Store::Config::Params & params)
TestHttpBinaryCacheStoreConfig(ParsedURL url, const Store::Config::Params & params)
: StoreConfig(params)
, HttpBinaryCacheStoreConfig(scheme, cacheUri, params)
, HttpBinaryCacheStoreConfig(url, params)
{
}
ref<TestHttpBinaryCacheStore> openTestStore() const;
ref<Store> openStore() const override
{
return openTestStore();
}
ref<TestHttpBinaryCacheStore> openTestStore(ref<FileTransfer> fileTransfer) const;
};
class HttpsBinaryCacheStoreTest : public virtual LibStoreNetworkTest
@@ -71,17 +65,29 @@ protected:
std::filesystem::path tmpDir, cacheDir;
std::filesystem::path caCert, caKey, serverCert, serverKey;
std::filesystem::path clientCert, clientKey;
std::string oldCaCert;
Pid serverPid;
uint16_t port = 8443;
std::shared_ptr<Store> localCacheStore;
/**
* Custom FileTransferSettings with the test CA certificate.
* This is used instead of modifying global settings.
*/
std::unique_ptr<FileTransferSettings> testFileTransferSettings;
/**
* FileTransfer instance using our test settings.
* Initialized in SetUp().
*/
std::shared_ptr<FileTransfer> testFileTransfer;
static void openssl(Strings args);
void SetUp() override;
void TearDown() override;
virtual std::vector<std::string> serverArgs();
ref<TestHttpBinaryCacheStoreConfig> makeConfig(BinaryCacheStoreConfig::Params params);
ref<TestHttpBinaryCacheStoreConfig> makeConfig();
ref<TestHttpBinaryCacheStore> openStore(ref<TestHttpBinaryCacheStoreConfig> config);
};
class HttpsBinaryCacheStoreMtlsTest : public HttpsBinaryCacheStoreTest
+2 -2
View File
@@ -15,10 +15,10 @@ int testMainForBuidingPre(int argc, char ** argv)
}
// Disable build hook. We won't be testing remote builds in these unit tests. If we do, fix the above build hook.
settings.buildHook = {};
settings.getWorkerSettings().buildHook = {};
// No substituters, unless a test specifically requests.
settings.substituters = {};
settings.getWorkerSettings().substituters = {};
#ifdef __linux__ // should match the conditional around sandboxBuildDir declaration.
@@ -5,6 +5,7 @@
#include "nix/store/derivations.hh"
#include "nix/store/derived-path.hh"
#include "nix/store/derivation-options.hh"
#include "nix/store/globals.hh"
#include "nix/store/parsed-derivations.hh"
#include "nix/util/types.hh"
#include "nix/util/json-utils.hh"
@@ -192,9 +193,7 @@ TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes_defaults)
EXPECT_EQ(options, advancedAttributes_defaults);
EXPECT_EQ(options.canBuildLocally(*this->store, got), false);
EXPECT_EQ(options.willBuildLocally(*this->store, got), false);
EXPECT_EQ(options.substitutesAllowed(), true);
EXPECT_EQ(options.substitutesAllowed(settings.getWorkerSettings()), true);
EXPECT_EQ(options.useUidRange(got), false);
});
};
@@ -242,7 +241,7 @@ TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes)
EXPECT_EQ(options, expected);
EXPECT_EQ(options.substitutesAllowed(), false);
EXPECT_EQ(options.substitutesAllowed(settings.getWorkerSettings()), false);
EXPECT_EQ(options.useUidRange(got), true);
});
};
@@ -334,9 +333,7 @@ TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes_structuredAttrs_d
EXPECT_EQ(options, advancedAttributes_structuredAttrs_defaults);
EXPECT_EQ(options.canBuildLocally(*this->store, got), false);
EXPECT_EQ(options.willBuildLocally(*this->store, got), false);
EXPECT_EQ(options.substitutesAllowed(), true);
EXPECT_EQ(options.substitutesAllowed(settings.getWorkerSettings()), true);
EXPECT_EQ(options.useUidRange(got), false);
});
};
@@ -401,9 +398,7 @@ TYPED_TEST(DerivationAdvancedAttrsBothTest, advancedAttributes_structuredAttrs)
EXPECT_EQ(options, expected);
EXPECT_EQ(options.canBuildLocally(*this->store, got), false);
EXPECT_EQ(options.willBuildLocally(*this->store, got), false);
EXPECT_EQ(options.substitutesAllowed(), false);
EXPECT_EQ(options.substitutesAllowed(settings.getWorkerSettings()), false);
EXPECT_EQ(options.useUidRange(got), true);
});
};
+1 -1
View File
@@ -51,7 +51,7 @@ protected:
depDrv.fillInOutputPaths(*store);
// Write the dependency to the store
return writeDerivation(*store, depDrv, NoRepair);
return store->writeDerivation(depDrv, NoRepair);
}
public:
+17 -31
View File
@@ -45,28 +45,18 @@ using namespace std::string_literals;
TEST_F(HttpsBinaryCacheStoreTest, queryPathInfo)
{
auto config = makeConfig({});
auto store = config->openStore();
auto store = openStore(makeConfig());
StringSource dump{"test"sv};
auto path = localCacheStore->addToStoreFromDump(dump, "test-name", FileSerialisationMethod::Flat);
EXPECT_NO_THROW(store->queryPathInfo(path));
}
auto withNoRetries()
{
auto oldTries = fileTransferSettings.tries.get();
Finally restoreTries = [=]() { fileTransferSettings.tries = oldTries; };
fileTransferSettings.tries = 1; /* FIXME: Don't use global settings. */
return restoreTries;
}
TEST_F(HttpsBinaryCacheStoreMtlsTest, queryPathInfo)
{
auto config = makeConfig({
{"tls-certificate"s, clientCert.string()},
{"tls-private-key"s, clientKey.string()},
});
auto store = config->openStore();
auto config = makeConfig();
config->tlsCert = clientCert;
config->tlsKey = clientKey;
auto store = openStore(config);
StringSource dump{"test"sv};
auto path = localCacheStore->addToStoreFromDump(dump, "test-name", FileSerialisationMethod::Flat);
EXPECT_NO_THROW(store->queryPathInfo(path));
@@ -74,9 +64,8 @@ TEST_F(HttpsBinaryCacheStoreMtlsTest, queryPathInfo)
TEST_F(HttpsBinaryCacheStoreMtlsTest, rejectsWithoutClientCert)
{
auto restoreTries = withNoRetries();
auto config = makeConfig({});
EXPECT_THROW(config->openStore(), Error);
testFileTransferSettings->tries = 1;
EXPECT_THROW(openStore(makeConfig()), Error);
}
TEST_F(HttpsBinaryCacheStoreMtlsTest, rejectsWrongClientCert)
@@ -89,12 +78,11 @@ TEST_F(HttpsBinaryCacheStoreMtlsTest, rejectsWrongClientCert)
openssl({"req", "-new", "-x509", "-days", "1", "-key", wrongKey.string(), "-out", wrongCert.string(), "-subj", "/CN=WrongClient"});
// clang-format on
auto config = makeConfig({
{"tls-certificate"s, wrongCert.string()},
{"tls-private-key"s, wrongKey.string()},
});
auto restoreTries = withNoRetries();
EXPECT_THROW(config->openStore(), Error);
auto config = makeConfig();
config->tlsCert = wrongCert;
config->tlsKey = wrongKey;
testFileTransferSettings->tries = 1;
EXPECT_THROW(openStore(config), Error);
}
TEST_F(HttpsBinaryCacheStoreMtlsTest, doesNotSendCertOnRedirectToDifferentAuthority)
@@ -109,13 +97,11 @@ TEST_F(HttpsBinaryCacheStoreMtlsTest, doesNotSendCertOnRedirectToDifferentAuthor
writeFile(entry.path(), content);
}
auto config = makeConfig({
{"tls-certificate"s, clientCert.string()},
{"tls-private-key"s, clientKey.string()},
});
auto store = config->openStore();
auto restoreTries = withNoRetries();
auto config = makeConfig();
config->tlsCert = clientCert;
config->tlsKey = clientKey;
testFileTransferSettings->tries = 1;
auto store = openStore(config);
auto info = store->queryPathInfo(path);
NullSink null;
EXPECT_THROW(store->narFromPath(path, null), Error);
+4 -2
View File
@@ -25,7 +25,8 @@ TEST(NarInfoDiskCacheImpl, create_and_read)
SQLiteStmt getIds;
{
auto cache = getTestNarInfoDiskCache(dbPath.string());
auto cache = NarInfoDiskCache::getTest(
settings.getNarInfoDiskCacheSettings(), {.useWAL = settings.useSQLiteWAL}, dbPath.string());
// Set up "background noise" and check that different caches receive different ids
{
@@ -74,7 +75,8 @@ TEST(NarInfoDiskCacheImpl, create_and_read)
{
// We can't clear the in-memory cache, so we use a new cache object. This is
// more realistic anyway.
auto cache2 = getTestNarInfoDiskCache(dbPath.string());
auto cache2 = NarInfoDiskCache::getTest(
settings.getNarInfoDiskCacheSettings(), {.useWAL = settings.useSQLiteWAL}, dbPath.string());
{
auto r = cache2->upToDateCacheExists("http://foo");
+6 -6
View File
@@ -299,7 +299,7 @@ public:
nix_api_store_test_base::SetUp();
nix::experimentalFeatureSettings.set("extra-experimental-features", "ca-derivations");
nix::settings.substituters = {};
nix::settings.getWorkerSettings().substituters = {};
store = open_local_store();
@@ -354,7 +354,7 @@ TEST_F(nix_api_store_test_base, build_from_json)
{
// FIXME get rid of these
nix::experimentalFeatureSettings.set("extra-experimental-features", "ca-derivations");
nix::settings.substituters = {};
nix::settings.getWorkerSettings().substituters = {};
auto * store = open_local_store();
@@ -401,7 +401,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_invalid_system)
{
// Test that nix_store_realise properly reports errors when the system is invalid
nix::experimentalFeatureSettings.set("extra-experimental-features", "ca-derivations");
nix::settings.substituters = {};
nix::settings.getWorkerSettings().substituters = {};
auto * store = open_local_store();
@@ -446,7 +446,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_builder_fails)
{
// Test that nix_store_realise properly reports errors when the builder fails
nix::experimentalFeatureSettings.set("extra-experimental-features", "ca-derivations");
nix::settings.substituters = {};
nix::settings.getWorkerSettings().substituters = {};
auto * store = open_local_store();
@@ -491,7 +491,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_builder_no_output)
{
// Test that nix_store_realise properly reports errors when builder succeeds but produces no output
nix::experimentalFeatureSettings.set("extra-experimental-features", "ca-derivations");
nix::settings.substituters = {};
nix::settings.getWorkerSettings().substituters = {};
auto * store = open_local_store();
@@ -687,7 +687,7 @@ TEST_F(NixApiStoreTestWithRealisedPath, nix_store_realise_output_ordering)
// This test uses a CA derivation with 10 outputs in randomized input order
// to verify that the callback order is deterministic and alphabetical.
nix::experimentalFeatureSettings.set("extra-experimental-features", "ca-derivations");
nix::settings.substituters = {};
nix::settings.getWorkerSettings().substituters = {};
auto * store = open_local_store();
+237 -15
View File
@@ -104,6 +104,33 @@ INSTANTIATE_TEST_SUITE_P(
},
},
"with_absolute_endpoint_uri",
},
ParsedS3URLTestCase{
"s3://bucket/key?addressing-style=virtual",
{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
},
"with_addressing_style_virtual",
},
ParsedS3URLTestCase{
"s3://bucket/key?addressing-style=path",
{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Path,
},
"with_addressing_style_path",
},
ParsedS3URLTestCase{
"s3://bucket/key?addressing-style=auto",
{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Auto,
},
"with_addressing_style_auto",
}),
[](const ::testing::TestParamInfo<ParsedS3URLTestCase> & info) { return info.param.description; });
@@ -138,6 +165,26 @@ INSTANTIATE_TEST_SUITE_P(
InvalidS3URLTestCase{"s3://bucket", "error: URI has a missing or invalid key", "missing_key"}),
[](const ::testing::TestParamInfo<InvalidS3URLTestCase> & info) { return info.param.description; });
TEST(ParsedS3URLTest, invalidAddressingStyleThrows)
{
ASSERT_THROW(ParsedS3URL::parse(parseURL("s3://bucket/key?addressing-style=bogus")), InvalidS3AddressingStyle);
}
TEST(ParsedS3URLTest, virtualStyleWithAuthoritylessEndpointThrows)
{
ParsedS3URL input{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint =
ParsedURL{
.scheme = "file",
.path = {"", "some", "path"},
},
};
ASSERT_THROW(input.toHttpsUrl(), nix::Error);
}
// =============================================================================
// S3 URL to HTTPS Conversion Tests
// =============================================================================
@@ -166,6 +213,7 @@ INSTANTIATE_TEST_SUITE_P(
S3ToHttpsConversion,
S3ToHttpsConversionTest,
::testing::Values(
// Default (auto) addressing style: virtual-hosted for standard AWS endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my-bucket",
@@ -173,10 +221,10 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my-bucket", "my-key.txt"},
.authority = ParsedURL::Authority{.host = "my-bucket.s3.us-east-1.amazonaws.com"},
.path = {"", "my-key.txt"},
},
"https://s3.us-east-1.amazonaws.com/my-bucket/my-key.txt",
"https://my-bucket.s3.us-east-1.amazonaws.com/my-key.txt",
"basic_s3_default_region",
},
S3ToHttpsConversionTestCase{
@@ -187,12 +235,13 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
.path = {"", "prod-cache", "nix", "store", "abc123.nar.xz"},
.authority = ParsedURL::Authority{.host = "prod-cache.s3.eu-west-1.amazonaws.com"},
.path = {"", "nix", "store", "abc123.nar.xz"},
},
"https://s3.eu-west-1.amazonaws.com/prod-cache/nix/store/abc123.nar.xz",
"https://prod-cache.s3.eu-west-1.amazonaws.com/nix/store/abc123.nar.xz",
"with_eu_west_1_region",
},
// Custom endpoint authority: path-style by default
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
@@ -208,6 +257,7 @@ INSTANTIATE_TEST_SUITE_P(
"http://custom.s3.com/bucket/key",
"custom_endpoint_authority",
},
// Custom endpoint URL: path-style by default
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
@@ -236,10 +286,10 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.ap-southeast-2.amazonaws.com"},
.path = {"", "bucket", "path", "to", "file.txt"},
.authority = ParsedURL::Authority{.host = "bucket.s3.ap-southeast-2.amazonaws.com"},
.path = {"", "path", "to", "file.txt"},
},
"https://s3.ap-southeast-2.amazonaws.com/bucket/path/to/file.txt",
"https://bucket.s3.ap-southeast-2.amazonaws.com/path/to/file.txt",
"complex_path_and_region",
},
S3ToHttpsConversionTestCase{
@@ -250,11 +300,11 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my-bucket", "my-key.txt"},
.authority = ParsedURL::Authority{.host = "my-bucket.s3.us-east-1.amazonaws.com"},
.path = {"", "my-key.txt"},
.query = {{"versionId", "abc123xyz"}},
},
"https://s3.us-east-1.amazonaws.com/my-bucket/my-key.txt?versionId=abc123xyz",
"https://my-bucket.s3.us-east-1.amazonaws.com/my-key.txt?versionId=abc123xyz",
"with_versionId",
},
S3ToHttpsConversionTestCase{
@@ -266,13 +316,185 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
.path = {"", "versioned-bucket", "path", "to", "object"},
.authority = ParsedURL::Authority{.host = "versioned-bucket.s3.eu-west-1.amazonaws.com"},
.path = {"", "path", "to", "object"},
.query = {{"versionId", "version456"}},
},
"https://s3.eu-west-1.amazonaws.com/versioned-bucket/path/to/object?versionId=version456",
"https://versioned-bucket.s3.eu-west-1.amazonaws.com/path/to/object?versionId=version456",
"with_region_and_versionId",
},
// Explicit addressing-style=path forces path-style on standard AWS endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my-bucket",
.key = {"my-key.txt"},
.region = "us-west-2",
.addressingStyle = S3AddressingStyle::Path,
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-west-2.amazonaws.com"},
.path = {"", "my-bucket", "my-key.txt"},
},
"https://s3.us-west-2.amazonaws.com/my-bucket/my-key.txt",
"explicit_path_style",
},
// Explicit addressing-style=virtual forces virtual-hosted-style on custom endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"key"},
.scheme = "http",
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "bucket.custom.s3.com"},
.path = {"", "key"},
},
"http://bucket.custom.s3.com/key",
"explicit_virtual_style_custom_endpoint",
},
// Explicit addressing-style=virtual with full endpoint URL
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint =
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
.path = {""},
},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "bucket.server", .port = 9000},
.path = {"", "key"},
},
"http://bucket.server:9000/key",
"explicit_virtual_style_full_endpoint_url",
},
// Dotted bucket names work normally with explicit path-style
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Path,
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my.bucket", "key"},
},
"https://s3.us-east-1.amazonaws.com/my.bucket/key",
"dotted_bucket_with_path_style",
},
// Dotted bucket names fall back to path-style with auto on standard AWS endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my.bucket.name",
.key = {"key"},
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my.bucket.name", "key"},
},
"https://s3.us-east-1.amazonaws.com/my.bucket.name/key",
"dotted_bucket_with_auto_style_on_aws",
},
// Dotted bucket names work with auto style on custom endpoints (auto = path-style)
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.endpoint = ParsedURL::Authority{.host = "minio.local"},
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "minio.local"},
.path = {"", "my.bucket", "key"},
},
"https://minio.local/my.bucket/key",
"dotted_bucket_with_auto_style_custom_endpoint",
}),
[](const ::testing::TestParamInfo<S3ToHttpsConversionTestCase> & info) { return info.param.description; });
// =============================================================================
// S3 URL to HTTPS Conversion Error Tests
// =============================================================================
struct S3ToHttpsConversionErrorTestCase
{
ParsedS3URL input;
std::string description;
};
class S3ToHttpsConversionErrorTest : public ::testing::WithParamInterface<S3ToHttpsConversionErrorTestCase>,
public ::testing::Test
{};
TEST_P(S3ToHttpsConversionErrorTest, ThrowsOnConversion)
{
auto & [input, description] = GetParam();
ASSERT_THROW(input.toHttpsUrl(), nix::Error);
}
INSTANTIATE_TEST_SUITE_P(
S3ToHttpsConversionErrors,
S3ToHttpsConversionErrorTest,
::testing::Values(
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint = ParsedURL::Authority{.host = ""},
},
"virtual_style_with_empty_host_authority",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
},
"dotted_bucket_with_explicit_virtual_style",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket.name",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
},
"dotted_bucket_with_explicit_virtual_style_multi_dot",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint = ParsedURL::Authority{.host = "minio.local"},
},
"dotted_bucket_with_explicit_virtual_style_custom_authority",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint =
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "minio.local", .port = 9000},
.path = {""},
},
},
"dotted_bucket_with_explicit_virtual_style_full_endpoint_url",
}),
[](const ::testing::TestParamInfo<S3ToHttpsConversionErrorTestCase> & info) { return info.param.description; });
} // namespace nix
+3 -3
View File
@@ -193,7 +193,7 @@ TEST_F(WorkerSubstitutionTest, floatingDerivationOutput)
};
// Write the derivation to the destination store
auto drvPath = writeDerivation(*dummyStore, drv);
auto drvPath = dummyStore->writeDerivation(drv);
// Snapshot the destination store before
checkpointJson("ca-drv/store-before", dummyStore);
@@ -297,7 +297,7 @@ TEST_F(WorkerSubstitutionTest, floatingDerivationOutputWithDepDrv)
};
// Write the dependency derivation to the destination store
auto depDrvPath = writeDerivation(*dummyStore, depDrv);
auto depDrvPath = dummyStore->writeDerivation(depDrv);
// Compute the hash modulo for the dependency derivation
auto depHashModulo = hashDerivationModulo(*dummyStore, depDrv, true);
@@ -348,7 +348,7 @@ TEST_F(WorkerSubstitutionTest, floatingDerivationOutputWithDepDrv)
rootDrv.inputDrvs = {.map = {{depDrvPath, {.value = {"out"}}}}};
// Write the root derivation to the destination store
auto rootDrvPath = writeDerivation(*dummyStore, rootDrv);
auto rootDrvPath = dummyStore->writeDerivation(rootDrv);
// Snapshot the destination store before
checkpointJson("issue-11928/store-before", dummyStore);
+3 -3
View File
@@ -44,12 +44,12 @@ TEST_F(WriteDerivationTest, addToStoreFromDumpCalledOnce)
{
auto drv = makeSimpleDrv();
auto path1 = writeDerivation(*store, drv, NoRepair);
auto path1 = store->writeDerivation(drv, NoRepair);
config->readOnly = true;
auto path2 = writeDerivation(*store, drv, NoRepair);
auto path2 = computeStorePath(*store, drv);
EXPECT_EQ(path1, path2);
EXPECT_THAT(
[&] { writeDerivation(*store, drv, Repair); },
[&] { store->writeDerivation(drv, Repair); },
::testing::ThrowsMessage<Error>(
testing::HasSubstrIgnoreANSIMatcher("operation 'writeDerivation' is not supported by store 'dummy://'")));
}
+1 -1
View File
@@ -28,7 +28,7 @@
namespace nix {
AwsAuthError::AwsAuthError(int errorCode)
: Error("AWS authentication error: '%s' (%d)", aws_error_str(errorCode), errorCode)
: CloneableError("AWS authentication error: '%s' (%d)", aws_error_str(errorCode), errorCode)
, errorCode(errorCode)
{
}
+278 -126
View File
@@ -5,6 +5,7 @@
# include "nix/store/build/derivation-builder.hh"
#endif
#include "nix/util/processes.hh"
#include "nix/util/environment-variables.hh"
#include "nix/util/config-global.hh"
#include "nix/store/build/worker.hh"
#include "nix/util/util.hh"
@@ -14,6 +15,7 @@
#include "nix/store/local-store.hh" // TODO remove, along with remaining downcasts
#include "nix/store/globals.hh"
#include <algorithm>
#include <fstream>
#include <sys/types.h>
#include <fcntl.h>
@@ -63,7 +65,11 @@ std::string showKnownOutputs(const StoreDirConfig & store, const Derivation & dr
}
static void runPostBuildHook(
const StoreDirConfig & store, Logger & logger, const StorePath & drvPath, const StorePathSet & outputPaths);
const WorkerSettings & workerSettings,
const StoreDirConfig & store,
Logger & logger,
const StorePath & drvPath,
const StorePathSet & outputPaths);
/* At least one of the output paths could not be
produced using a substitute. So we have to build instead. */
@@ -87,7 +93,7 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution(bool storeDerivation)
for (auto & i : drv->inputSrcs) {
if (worker.store.isValidPath(i))
continue;
if (!settings.useSubstitutes)
if (!worker.settings.useSubstitutes)
throw Error(
"dependency '%s' of '%s' does not exist, and substitution is disabled",
worker.store.printStorePath(i),
@@ -119,7 +125,7 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution(bool storeDerivation)
assert(drv->inputDrvs.map.empty());
/* Store the resolved derivation, as part of the record of
what we're actually building */
writeDerivation(worker.store, *drv);
worker.store.writeDerivation(*drv);
}
StorePathSet inputPaths;
@@ -179,6 +185,82 @@ struct LogFile
~LogFile();
};
struct LocalBuildRejection
{
bool maxJobsZero = false;
struct NoLocalStore
{};
/**
* We have a local store, but we don't have an external derivation builder (which is fine), if we did, it'd be
* fine because we would not care about platforms and features then. Since we don't, we either have the wrong
* platform, or we are missing some system features.
*/
struct WrongLocalStore
{
template<typename T>
struct Pair
{
T derivation;
T localStore;
};
std::optional<Pair<std::string>> badPlatform;
std::optional<Pair<StringSet>> missingFeatures;
};
std::variant<NoLocalStore, WrongLocalStore> rejection;
};
static BuildError reject(const LocalBuildRejection & rejection, std::string_view thingCannotBuild)
{
if (std::get_if<LocalBuildRejection::NoLocalStore>(&rejection.rejection))
return BuildError(
BuildResult::Failure::InputRejected,
"Unable to build with a primary store that isn't a local store; "
"either pass a different '--store' or enable remote builds.\n\n"
"For more information check 'man nix.conf' and search for '/machines'.");
auto & wrongStore = std::get<LocalBuildRejection::WrongLocalStore>(rejection.rejection);
std::string msg = fmt("Cannot build '%s'.", Magenta(thingCannotBuild));
if (rejection.maxJobsZero)
msg += "\nReason: " ANSI_RED "local builds are disabled" ANSI_NORMAL
" (max-jobs = 0)"
"\nHint: set 'max-jobs' to a non-zero value to enable local builds, "
"or configure remote builders via 'builders'";
if (wrongStore.badPlatform)
msg +=
fmt("\nReason: " ANSI_RED "platform mismatch" ANSI_NORMAL
"\nRequired system: '%s'"
"\nCurrent system: '%s'",
Magenta(wrongStore.badPlatform->derivation),
Magenta(wrongStore.badPlatform->localStore));
if (wrongStore.missingFeatures)
msg +=
fmt("\nReason: " ANSI_RED "missing system features" ANSI_NORMAL
"\nRequired features: {%s}"
"\nAvailable features: {%s}",
concatStringsSep(", ", wrongStore.missingFeatures->derivation),
concatStringsSep<StringSet>(", ", wrongStore.missingFeatures->localStore));
if (wrongStore.badPlatform || wrongStore.missingFeatures) {
// since aarch64-darwin has Rosetta 2, this user can actually run x86_64-darwin on their
// hardware - we should tell them to run the command to install Rosetta
if (wrongStore.badPlatform && wrongStore.badPlatform->derivation == "x86_64-darwin"
&& wrongStore.badPlatform->localStore == "aarch64-darwin")
msg +=
fmt("\nNote: run `%s` to run programs for x86_64-darwin",
Magenta("/usr/sbin/softwareupdate --install-rosetta && launchctl stop org.nixos.nix-daemon"));
}
return BuildError(BuildResult::Failure::InputRejected, std::move(msg));
}
Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
{
auto drvOptions = [&] {
@@ -240,29 +322,57 @@ Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
}
checkPathValidity(initialOutputs);
/**
* Activity that denotes waiting for a lock.
*/
std::unique_ptr<Activity> actLock;
auto localBuildResult = [&]() -> std::variant<LocalBuildCapability, LocalBuildRejection> {
bool maxJobsZero = worker.settings.maxBuildJobs.get() == 0;
/**
* Locks on (fixed) output paths.
*/
PathLocks outputLocks;
auto * localStoreP = dynamic_cast<LocalStore *>(&worker.store);
if (!localStoreP)
return LocalBuildRejection{.maxJobsZero = maxJobsZero, .rejection = LocalBuildRejection::NoLocalStore{}};
bool useHook;
/**
* Now that we've decided we can't / won't do a remote build, check
* that we can in fact build locally. First see if there is an
* external builder for a "semi-local build". If there is, prefer to
* use that. If there is not, then check if we can do a "true" local
* build.
*/
auto * ext = settings.getLocalSettings().findExternalDerivationBuilderIfSupported(*drv);
const ExternalBuilder * externalBuilder = nullptr;
if (ext)
return LocalBuildCapability{*localStoreP, ext};
while (true) {
using WrongLocalStore = LocalBuildRejection::WrongLocalStore;
WrongLocalStore wrongStore;
if (drv->platform != settings.thisSystem.get() && !settings.extraPlatforms.get().count(drv->platform)
&& !drv->isBuiltin())
wrongStore.badPlatform = WrongLocalStore::Pair<std::string>{drv->platform, settings.thisSystem.get()};
{
auto required = drvOptions.getRequiredSystemFeatures(*drv);
auto & available = worker.store.config.systemFeatures.get();
if (std::ranges::any_of(required, [&](const std::string & f) { return !available.count(f); }))
wrongStore.missingFeatures = WrongLocalStore::Pair<StringSet>{required, available};
}
if (maxJobsZero || wrongStore.badPlatform || wrongStore.missingFeatures)
return LocalBuildRejection{.maxJobsZero = maxJobsZero, .rejection = std::move(wrongStore)};
return LocalBuildCapability{*localStoreP, ext};
}();
auto acquireResources = [&](bool & done, PathLocks & outputLocks) -> Goal::Co {
trace("trying to build");
/* Obtain locks on all output paths, if the paths are known a priori.
The locks are automatically released when we exit this function or Nix
crashes. If we can't acquire the lock, then continue; hopefully some
other goal can start a build, and if not, the main loop will sleep a few
seconds and then retry this goal. */
/**
* Output paths to acquire locks on, if known a priori.
*
* The locks are automatically released when the caller's `PathLocks` goes
* out of scope, including on exception unwinding. If we can't acquire the lock, then
* continue; hopefully some other goal can start a build, and if not, the
* main loop will sleep a few seconds and then retry this goal.
*/
std::set<std::filesystem::path> lockFiles;
/* FIXME: Should lock something like the drv itself so we don't build same
CA drv concurrently */
@@ -306,7 +416,8 @@ Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
debug("skipping build of derivation '%s', someone beat us to it", worker.store.printStorePath(drvPath));
outputLocks.setDeletion(true);
outputLocks.unlock();
co_return doneSuccess(BuildResult::Success::AlreadyValid, std::move(validOutputs));
done = true;
co_return Return{};
}
/* If any of the outputs already exist but are not valid, delete
@@ -321,97 +432,133 @@ Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
}
}
/* Don't do a remote build if the derivation has the attribute
`preferLocalBuild' set. Also, check and repair modes are only
supported for local builds. */
bool buildLocally = (buildMode != bmNormal || drvOptions.willBuildLocally(worker.store, *drv))
&& settings.maxBuildJobs.get() != 0;
co_return Return{};
};
auto tryHookLoop = [&](bool & valid) -> Goal::Co {
{
PathLocks outputLocks;
co_await acquireResources(valid, outputLocks);
if (valid)
co_return doneSuccess(BuildResult::Success::AlreadyValid, checkPathValidity(initialOutputs).second);
if (buildLocally) {
useHook = false;
} else {
switch (tryBuildHook(drvOptions)) {
case rpAccept:
/* Yes, it has started doing so. Wait until we get
EOF from the hook. */
useHook = true;
break;
valid = true;
co_return buildWithHook(
std::move(inputPaths), std::move(initialOutputs), std::move(drvOptions), std::move(outputLocks));
case rpDecline:
// We should do it ourselves.
co_return Return{};
case rpPostpone:
/* Not now; wait until at least one child finishes or
the wake-up timeout expires. */
if (!actLock)
actLock = std::make_unique<Activity>(
*logger,
lvlWarn,
actBuildWaiting,
fmt("waiting for a machine to build '%s'", Magenta(worker.store.printStorePath(drvPath))));
outputLocks.unlock();
co_await waitForAWhile();
continue;
case rpDecline:
/* We should do it ourselves.
Now that we've decided we can't / won't do a remote build, check
that we can in fact build locally. First see if there is an
external builder for a "semi-local build". If there is, prefer to
use that. If there is not, then check if we can do a "true" local
build. */
externalBuilder = settings.findExternalDerivationBuilderIfSupported(*drv);
if (!externalBuilder && !drvOptions.canBuildLocally(worker.store, *drv)) {
auto msg =
fmt("Cannot build '%s'.\n"
"Reason: " ANSI_RED "required system or feature not available" ANSI_NORMAL
"\n"
"Required system: '%s' with features {%s}\n"
"Current system: '%s' with features {%s}",
Magenta(worker.store.printStorePath(drvPath)),
Magenta(drv->platform),
concatStringsSep(", ", drvOptions.getRequiredSystemFeatures(*drv)),
Magenta(settings.thisSystem),
concatStringsSep<StringSet>(", ", worker.store.Store::config.systemFeatures));
// since aarch64-darwin has Rosetta 2, this user can actually run x86_64-darwin on their hardware -
// we should tell them to run the command to install Darwin 2
if (drv->platform == "x86_64-darwin" && settings.thisSystem == "aarch64-darwin")
msg += fmt(
"\nNote: run `%s` to run programs for x86_64-darwin",
Magenta(
"/usr/sbin/softwareupdate --install-rosetta && launchctl stop org.nixos.nix-daemon"));
outputLocks.unlock();
co_return doneFailure({BuildResult::Failure::InputRejected, std::move(msg)});
}
useHook = false;
break;
}
}
break;
}
actLock.reset();
PathLocks outputLocks;
{
// First attempt was postponed. Retry in a loop with an activity
// that lives until accept or decline.
Activity act(
*logger,
lvlWarn,
actBuildWaiting,
fmt("waiting for a machine to build '%s'", Magenta(worker.store.printStorePath(drvPath))));
if (useHook) {
co_return buildWithHook(
std::move(inputPaths), std::move(initialOutputs), std::move(drvOptions), std::move(outputLocks));
} else if (auto * localStoreP = dynamic_cast<LocalStore *>(&worker.store)) {
co_return buildLocally(
*localStoreP,
std::move(inputPaths),
std::move(initialOutputs),
std::move(drvOptions),
std::move(outputLocks),
externalBuilder);
while (true) {
co_await waitForAWhile();
co_await acquireResources(valid, outputLocks);
if (valid)
break;
switch (tryBuildHook(drvOptions)) {
case rpAccept:
/* Yes, it has started doing so. Wait until we get
EOF from the hook. */
break;
case rpPostpone:
/* Not now; wait until at least one child finishes or
the wake-up timeout expires. */
outputLocks.unlock();
continue;
case rpDecline:
// We should do it ourselves.
co_return Return{};
}
break;
}
}
if (valid) {
co_return doneSuccess(BuildResult::Success::AlreadyValid, checkPathValidity(initialOutputs).second);
} else {
co_return buildWithHook(
std::move(inputPaths), std::move(initialOutputs), std::move(drvOptions), std::move(outputLocks));
}
};
auto tryBuildLocally = [&](bool & valid) -> Goal::Co {
if (auto * cap = std::get_if<LocalBuildCapability>(&localBuildResult)) {
PathLocks outputLocks;
co_await acquireResources(valid, outputLocks);
if (valid)
co_return doneSuccess(BuildResult::Success::AlreadyValid, checkPathValidity(initialOutputs).second);
valid = true;
co_return buildLocally(
*cap, std::move(inputPaths), std::move(initialOutputs), std::move(drvOptions), std::move(outputLocks));
}
co_return Return{};
};
if (buildMode != bmNormal) {
// Check and repair modes operate on the state of this store specifically,
// so they must always build locally.
bool valid = false;
co_await tryBuildLocally(valid);
if (valid)
co_return Return{};
} else if (drvOptions.preferLocalBuild) {
// Local is preferred, so try it first. If it's not available, fall back to the hook.
{
bool valid = false;
co_await tryBuildLocally(valid);
if (valid)
co_return Return{};
}
{
bool valid = false;
co_await tryHookLoop(valid);
if (valid)
co_return Return{};
}
} else {
throw Error(
R"(
Unable to build with a primary store that isn't a local store;
either pass a different '--store' or enable remote builds.
For more information check 'man nix.conf' and search for '/machines'.
)");
// Default preference is a remote build: they tend to be faster and preserve local
// resources for other tasks. Fall back to local if no remote is available.
{
bool valid = false;
co_await tryHookLoop(valid);
if (valid)
co_return Return{};
}
{
bool valid = false;
co_await tryBuildLocally(valid);
if (valid)
co_return Return{};
}
}
std::string storePath = worker.store.printStorePath(drvPath);
auto * rejection = std::get_if<LocalBuildRejection>(&localBuildResult);
assert(rejection);
co_return doneFailure(reject(*rejection, storePath));
}
Goal::Co DerivationBuildingGoal::buildWithHook(
@@ -476,7 +623,7 @@ Goal::Co DerivationBuildingGoal::buildWithHook(
msg += fmt(" on '%s'", hook->machineName);
std::unique_ptr<BuildLog> buildLog = std::make_unique<BuildLog>(
settings.logLines,
worker.settings.logLines,
std::make_unique<Activity>(
*logger,
lvlInfo,
@@ -496,7 +643,7 @@ Goal::Co DerivationBuildingGoal::buildWithHook(
auto & data = output->data;
if (fd == hook->builderOut.readSide.get()) {
logSize += data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
if (worker.settings.maxLogSize && logSize > worker.settings.maxLogSize) {
hook.reset();
co_return doneFailureLogTooLong(*buildLog);
}
@@ -605,7 +752,7 @@ Goal::Co DerivationBuildingGoal::buildWithHook(
StorePathSet outputPaths;
for (auto & [_, output] : builtOutputs)
outputPaths.insert(output.outPath);
runPostBuildHook(worker.store, *logger, drvPath, outputPaths);
runPostBuildHook(worker.settings, worker.store, *logger, drvPath, outputPaths);
/* It is now safe to delete the lock files, since all future
lockers will see that the output paths are valid; they will
@@ -619,12 +766,11 @@ Goal::Co DerivationBuildingGoal::buildWithHook(
}
Goal::Co DerivationBuildingGoal::buildLocally(
LocalStore & localStore,
LocalBuildCapability localBuildCap,
StorePathSet inputPaths,
std::map<std::string, InitialOutput> initialOutputs,
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks,
const ExternalBuilder * externalBuilder)
PathLocks outputLocks)
{
co_await yield();
@@ -647,7 +793,7 @@ Goal::Co DerivationBuildingGoal::buildLocally(
: "building '%s'",
worker.store.printStorePath(drvPath));
buildLog = std::make_unique<BuildLog>(
settings.logLines,
worker.settings.logLines,
std::make_unique<Activity>(
*logger, lvlInfo, actBuild, msg, Logger::Fields{worker.store.printStorePath(drvPath), "", 1, 1}));
mcRunningBuilds = std::make_unique<MaintainCount<uint64_t>>(worker.runningBuilds);
@@ -662,7 +808,7 @@ Goal::Co DerivationBuildingGoal::buildLocally(
while (true) {
unsigned int curBuilds = worker.getNrLocalBuilds();
if (curBuilds >= settings.maxBuildJobs) {
if (curBuilds >= worker.settings.maxBuildJobs) {
outputLocks.unlock();
co_await waitForBuildSlot();
co_return tryToBuild(std::move(inputPaths));
@@ -708,7 +854,7 @@ Goal::Co DerivationBuildingGoal::buildLocally(
};
decltype(DerivationBuilderParams::defaultPathsInChroot) defaultPathsInChroot =
localStore.config->getLocalSettings().sandboxPaths.get();
localBuildCap.localStore.config->getLocalSettings().sandboxPaths.get();
DesugaredEnv desugaredEnv;
/* Add the closure of store paths to the chroot. */
@@ -750,14 +896,14 @@ Goal::Co DerivationBuildingGoal::buildLocally(
/* If we have to wait and retry (see below), then `builder` will
already be created, so we don't need to create it again. */
builder = externalBuilder
builder = localBuildCap.externalBuilder
? makeExternalDerivationBuilder(
localStore,
localBuildCap.localStore,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, openLogFile, closeLogFile),
std::move(params),
*externalBuilder)
*localBuildCap.externalBuilder)
: makeDerivationBuilder(
localStore,
localBuildCap.localStore,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, openLogFile, closeLogFile),
std::move(params));
}
@@ -791,7 +937,7 @@ Goal::Co DerivationBuildingGoal::buildLocally(
if (auto * output = std::get_if<ChildOutput>(&event)) {
if (output->fd == builder->builderOut.get()) {
logSize += output->data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
if (worker.settings.maxLogSize && logSize > worker.settings.maxLogSize) {
builder->killChild();
co_return doneFailureLogTooLong(*buildLog);
}
@@ -838,7 +984,7 @@ Goal::Co DerivationBuildingGoal::buildLocally(
worker.markContentsGood(output.outPath);
outputPaths.insert(output.outPath);
}
runPostBuildHook(worker.store, *logger, drvPath, outputPaths);
runPostBuildHook(worker.settings, worker.store, *logger, drvPath, outputPaths);
/* It is now safe to delete the lock files, since all future
lockers will see that the output paths are valid; they will
@@ -852,9 +998,13 @@ Goal::Co DerivationBuildingGoal::buildLocally(
}
static void runPostBuildHook(
const StoreDirConfig & store, Logger & logger, const StorePath & drvPath, const StorePathSet & outputPaths)
const WorkerSettings & workerSettings,
const StoreDirConfig & store,
Logger & logger,
const StorePath & drvPath,
const StorePathSet & outputPaths)
{
auto hook = settings.postBuildHook;
auto hook = workerSettings.postBuildHook;
if (hook == "")
return;
@@ -862,14 +1012,15 @@ static void runPostBuildHook(
logger,
lvlTalkative,
actPostBuildHook,
fmt("running post-build-hook '%s'", settings.postBuildHook),
fmt("running post-build-hook '%s'", workerSettings.postBuildHook),
Logger::Fields{store.printStorePath(drvPath)});
PushActivity pact(act.id);
StringMap hookEnvironment = getEnv();
OsStringMap hookEnvironment = getEnvOs();
hookEnvironment.emplace("DRV_PATH", store.printStorePath(drvPath));
hookEnvironment.emplace("OUT_PATHS", chomp(concatStringsSep(" ", store.printStorePathSet(outputPaths))));
hookEnvironment.emplace("NIX_CONFIG", globalConfig.toKeyValue());
hookEnvironment.emplace(OS_STR("DRV_PATH"), string_to_os_string(store.printStorePath(drvPath)));
hookEnvironment.emplace(
OS_STR("OUT_PATHS"), string_to_os_string(chomp(concatStringsSep(" ", store.printStorePathSet(outputPaths)))));
hookEnvironment.emplace(OS_STR("NIX_CONFIG"), string_to_os_string(globalConfig.toKeyValue()));
struct LogSink : Sink
{
@@ -910,7 +1061,7 @@ static void runPostBuildHook(
LogSink sink(act);
runProgram2({
.program = settings.postBuildHook,
.program = workerSettings.postBuildHook,
.environment = hookEnvironment,
.standardOut = &sink,
.mergeStderrToStdout = true,
@@ -957,17 +1108,18 @@ HookReply DerivationBuildingGoal::tryBuildHook(const DerivationOptions<StorePath
#else
/* This should use `worker.evalStore`, but per #13179 the build hook
doesn't work with eval store anyways. */
if (settings.buildHook.get().empty() || !worker.tryBuildHook || !worker.store.isValidPath(drvPath))
if (worker.settings.buildHook.get().empty() || !worker.tryBuildHook || !worker.store.isValidPath(drvPath))
return rpDecline;
if (!worker.hook)
worker.hook = std::make_unique<HookInstance>();
worker.hook = std::make_unique<HookInstance>(worker.settings.buildHook);
try {
/* Send the request to the hook. */
worker.hook->sink << "try" << (worker.getNrLocalBuilds() < settings.maxBuildJobs ? 1 : 0) << drv->platform
<< worker.store.printStorePath(drvPath) << drvOptions.getRequiredSystemFeatures(*drv);
worker.hook->sink << "try" << (worker.getNrLocalBuilds() < worker.settings.maxBuildJobs ? 1 : 0)
<< drv->platform << worker.store.printStorePath(drvPath)
<< drvOptions.getRequiredSystemFeatures(*drv);
worker.hook->sink.flush();
/* Read the first line of input, which should be a word indicating
@@ -1030,7 +1182,7 @@ LogFile::LogFile(Store & store, const StorePath & drvPath, const LogFileSettings
if (auto localStore = dynamic_cast<LocalStore *>(&store))
logDir = localStore->config->logDir;
else
logDir = logSettings.nixLogDir;
logDir = logSettings.nixLogDir.string();
Path dir = fmt("%s/%s/%s/", logDir, LocalFSStore::drvsLogDir, baseName.substr(0, 2));
createDirs(dir);
@@ -1073,7 +1225,7 @@ Goal::Done DerivationBuildingGoal::doneFailureLogTooLong(BuildLog & buildLog)
BuildResult::Failure::LogLimitExceeded,
"%s killed after writing more than %d bytes of log output",
getName(),
settings.maxLogSize));
worker.settings.maxLogSize));
}
std::map<std::string, std::optional<StorePath>> DerivationBuildingGoal::queryPartialDerivationOutputMap()
+2 -2
View File
@@ -100,7 +100,7 @@ Goal::Co DerivationGoal::haveDerivation(bool storeDerivation)
/* We are first going to try to create the invalid output paths
through substitutes. If that doesn't work, we'll build
them. */
if (settings.useSubstitutes && drvOptions.substitutesAllowed()) {
if (worker.settings.useSubstitutes && drvOptions.substitutesAllowed(worker.settings)) {
if (!checkResult) {
DrvOutput id{outputHash, wantedOutput};
auto g = worker.makeDrvOutputSubstitutionGoal(id);
@@ -133,7 +133,7 @@ Goal::Co DerivationGoal::haveDerivation(bool storeDerivation)
assert(!drv->type().isImpure());
if (nrFailed > 0 && nrFailed > nrNoSubstituters && !settings.tryFallback) {
if (nrFailed > 0 && nrFailed > nrNoSubstituters && !worker.settings.tryFallback) {
co_return doneFailure(BuildError(
BuildResult::Failure::TransientFailure,
"some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ",
@@ -144,7 +144,7 @@ Goal::Co DerivationResolutionGoal::resolveDerivation()
}
assert(attempt);
auto pathResolved = writeDerivation(worker.store, *attempt, NoRepair, /*readOnly =*/true);
auto pathResolved = computeStorePath(worker.store, Derivation{*attempt});
auto msg =
fmt("resolved derivation: '%s' -> '%s'",
+3 -3
View File
@@ -1,11 +1,11 @@
#include "nix/store/build/goal.hh"
#include "nix/store/build/worker.hh"
#include "nix/store/globals.hh"
#include "nix/store/worker-settings.hh"
namespace nix {
TimedOut::TimedOut(time_t maxDuration)
: BuildError(BuildResult::Failure::TimedOut, "timed out after %1% seconds", maxDuration)
: CloneableError(BuildResult::Failure::TimedOut, "timed out after %1% seconds", maxDuration)
, maxDuration(maxDuration)
{
}
@@ -232,7 +232,7 @@ Goal::Done Goal::amDone(ExitCode result)
if (goal->waitees.empty()) {
worker.wakeUp(goal);
} else if (result == ecFailed && !settings.keepGoing) {
} else if (result == ecFailed && !worker.settings.keepGoing) {
/* If we failed and keepGoing is not set, we remove all
remaining waitees. */
for (auto & g : goal->waitees) {
+3 -3
View File
@@ -37,7 +37,7 @@ Goal::Co PathSubstitutionGoal::init()
co_return doneSuccess(BuildResult::Success{.status = BuildResult::Success::AlreadyValid});
}
if (settings.readOnlyMode)
if (worker.store.config.getReadOnly())
throw Error(
"cannot substitute path '%s' - no write access to the Nix store", worker.store.printStorePath(storePath));
@@ -145,7 +145,7 @@ Goal::Co PathSubstitutionGoal::init()
worker.updateProgress();
}
if (lastStoresException.has_value()) {
if (!settings.tryFallback) {
if (!worker.settings.tryFallback) {
throw *lastStoresException;
} else
logError(lastStoresException->info());
@@ -197,7 +197,7 @@ Goal::Co PathSubstitutionGoal::tryToRun(
/* Make sure that we are allowed to start a substitution. Note that even
if maxSubstitutionJobs == 0, we still allow a substituter to run. This
prevents infinite waiting. */
while (worker.getNrSubstitutions() >= std::max(1U, (unsigned int) settings.maxSubstitutionJobs)) {
while (worker.getNrSubstitutions() >= std::max(1U, (unsigned int) worker.settings.maxSubstitutionJobs)) {
co_await waitForBuildSlot();
}
+5 -2
View File
@@ -22,7 +22,10 @@ Worker::Worker(Store & store, Store & evalStore)
, actSubstitutions(*logger, actCopyPaths)
, store(store)
, evalStore(evalStore)
, getSubstituters{[] { return settings.useSubstitutes ? getDefaultSubstituters() : std::list<ref<Store>>{}; }}
, settings(nix::settings.getWorkerSettings())
, getSubstituters{[] {
return nix::settings.getWorkerSettings().useSubstitutes ? getDefaultSubstituters() : std::list<ref<Store>>{};
}}
{
nrLocalBuilds = 0;
nrSubstitutions = 0;
@@ -364,7 +367,7 @@ void Worker::run(const Goals & _topGoals)
if (!children.empty() || !waitingForAWhile.empty())
waitForInput();
else if (awake.empty() && 0U == settings.maxBuildJobs) {
if (getMachines().empty())
if (Machine::parseConfig({nix::settings.thisSystem}, nix::settings.getWorkerSettings().builders).empty())
throw Error(
"Unable to start any build; either increase '--max-jobs' or enable remote builds.\n"
"\n"
+5 -5
View File
@@ -14,12 +14,12 @@ static void builtinFetchurl(const BuiltinBuilderContext & ctx)
this to be stored in a file. It would be nice if we could just
pass a pointer to the data. */
if (ctx.netrcData != "") {
settings.netrcFile = "netrc";
writeFile(settings.netrcFile, ctx.netrcData, 0600);
fileTransferSettings.netrcFile = "netrc";
writeFile(fileTransferSettings.netrcFile, ctx.netrcData, 0600);
}
settings.caFile = "ca-certificates.crt";
writeFile(settings.caFile, ctx.caFileData, 0600);
fileTransferSettings.caFile = "ca-certificates.crt";
writeFile(*fileTransferSettings.caFile.get(), ctx.caFileData, 0600);
auto out = get(ctx.drv.outputs, "out");
if (!out)
@@ -73,7 +73,7 @@ static void builtinFetchurl(const BuiltinBuilderContext & ctx)
/* Try the hashed mirrors first. */
auto dof = std::get_if<DerivationOutput::CAFixed>(&out->raw);
if (dof && dof->ca.method.getFileIngestionMethod() == FileIngestionMethod::Flat)
for (auto hashedMirror : settings.hashedMirrors.get())
for (auto hashedMirror : ctx.hashedMirrors)
try {
if (!hasSuffix(hashedMirror, "/"))
hashedMirror += '/';
+33 -21
View File
@@ -233,38 +233,49 @@ struct ClientSettings
void apply(TrustedFlag trusted)
{
settings.keepFailed = keepFailed;
settings.keepGoing = keepGoing;
settings.tryFallback = tryFallback;
settings.getWorkerSettings().keepGoing = keepGoing;
settings.getWorkerSettings().tryFallback = tryFallback;
nix::verbosity = verbosity;
settings.maxBuildJobs.assign(maxBuildJobs);
settings.maxSilentTime = maxSilentTime;
settings.getWorkerSettings().maxBuildJobs.assign(maxBuildJobs);
settings.getWorkerSettings().maxSilentTime = maxSilentTime;
settings.verboseBuild = verboseBuild;
settings.buildCores = buildCores;
settings.useSubstitutes = useSubstitutes;
settings.getLocalSettings().buildCores = buildCores;
settings.getWorkerSettings().useSubstitutes = useSubstitutes;
for (auto & i : overrides) {
auto & name(i.first);
auto & value(i.second);
auto setSubstituters = [&](Setting<Strings> & res) {
auto setSubstituters = [&](Setting<std::vector<StoreReference>> & res) {
if (name != res.name && res.aliases.count(name) == 0)
return false;
StringSet trusted = settings.trustedSubstituters;
for (auto & s : settings.substituters.get())
trusted.insert(s);
Strings subs;
std::set<StoreReference> trusted = settings.trustedSubstituters;
for (auto & ref : settings.getWorkerSettings().substituters.get())
trusted.insert(ref);
std::vector<StoreReference> subs;
auto ss = tokenizeString<Strings>(value);
for (auto & s : ss)
if (trusted.count(s))
subs.push_back(s);
else if (!hasSuffix(s, "/") && trusted.count(s + "/"))
subs.push_back(s + "/");
for (auto & s : ss) {
auto ref = StoreReference::parse(s);
auto tryTrust = [&] {
if (trusted.count(ref))
return true;
if (auto * specified = std::get_if<StoreReference::Specified>(&ref.variant);
specified && !hasSuffix(specified->authority, "/")) {
specified->authority += "/";
if (trusted.count(ref))
return true;
}
return false;
};
if (tryTrust())
subs.push_back(std::move(ref));
else
warn(
"ignoring untrusted substituter '%s', you are not a trusted user.\n"
"Run `man nix.conf` for more information on the `substituters` configuration option.",
s);
res = subs;
}
res = std::move(subs);
return true;
};
@@ -282,11 +293,12 @@ struct ClientSettings
"Ignoring the client-specified plugin-files.\n"
"The client specifying plugins to the daemon never made sense, and was removed in Nix >=2.14.");
} else if (
trusted || name == settings.buildTimeout.name || name == settings.maxSilentTime.name
|| name == settings.pollInterval.name || name == "connect-timeout"
trusted || name == settings.getWorkerSettings().buildTimeout.name
|| name == settings.getWorkerSettings().maxSilentTime.name
|| name == settings.getWorkerSettings().pollInterval.name || name == "connect-timeout"
|| (name == "builders" && value == ""))
settings.set(name, value);
else if (setSubstituters(settings.substituters))
else if (setSubstituters(settings.getWorkerSettings().substituters))
;
else
warn(
@@ -635,7 +647,7 @@ static void performOp(
Derivation drv2;
static_cast<BasicDerivation &>(drv2) = drv;
drvPath = writeDerivation(*store, Derivation{drv2});
drvPath = store->writeDerivation(Derivation{drv2});
}
auto res = store->buildDerivation(drvPath, drv, buildMode);
+2 -26
View File
@@ -6,7 +6,6 @@
#include "nix/store/store-api.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
#include "nix/store/globals.hh"
#include "nix/util/variant-wrapper.hh"
#include <optional>
@@ -360,32 +359,9 @@ StringSet DerivationOptions<Input>::getRequiredSystemFeatures(const BasicDerivat
}
template<typename Input>
bool DerivationOptions<Input>::canBuildLocally(Store & localStore, const BasicDerivation & drv) const
bool DerivationOptions<Input>::substitutesAllowed(const WorkerSettings & workerSettings) const
{
if (drv.platform != settings.thisSystem.get() && !settings.extraPlatforms.get().count(drv.platform)
&& !drv.isBuiltin())
return false;
if (settings.maxBuildJobs.get() == 0 && !drv.isBuiltin())
return false;
for (auto & feature : getRequiredSystemFeatures(drv))
if (!localStore.config.systemFeatures.get().count(feature))
return false;
return true;
}
template<typename Input>
bool DerivationOptions<Input>::willBuildLocally(Store & localStore, const BasicDerivation & drv) const
{
return preferLocalBuild && canBuildLocally(localStore, drv);
}
template<typename Input>
bool DerivationOptions<Input>::substitutesAllowed() const
{
return settings.alwaysAllowSubstitutes ? true : allowSubstitutes;
return workerSettings.alwaysAllowSubstitutes ? true : allowSubstitutes;
}
template<typename Input>
+10 -7
View File
@@ -106,7 +106,7 @@ bool BasicDerivation::isBuiltin() const
return builder.substr(0, 8) == "builtin:";
}
static auto infoForDerivation(Store & store, const Derivation & drv)
static auto infoForDerivation(const StoreDirConfig & store, const Derivation & drv)
{
auto references = drv.inputSrcs;
for (auto & i : drv.inputDrvs.map)
@@ -126,19 +126,22 @@ static auto infoForDerivation(Store & store, const Derivation & drv)
};
}
StorePath writeDerivation(Store & store, const Derivation & drv, RepairFlag repair, bool readOnly)
StorePath computeStorePath(const StoreDirConfig & store, const Derivation & drv)
{
if (readOnly || settings.readOnlyMode) {
auto [_x, _y, _z, path] = infoForDerivation(store, drv);
return path;
} else
return store.writeDerivation(drv, repair);
auto [_suffix, _contents, _references, path] = infoForDerivation(store, drv);
return path;
}
StorePath Store::writeDerivation(const Derivation & drv, RepairFlag repair)
{
auto [suffix, contents, references, path] = infoForDerivation(*this, drv);
/* In case the derivation is already valid, we bail out early since that's
faster. But we need to make sure that the derivation has a corresponding
temproot. It is added by the remote in addToStoreFromDump, but we'd like
to avoid sending a lot of drv contents to the daemon. */
addTempRoot(path);
if (isValidPath(path) && !repair)
return path;
+6 -1
View File
@@ -119,6 +119,11 @@ ref<Store> DummyStoreConfig::openStore() const
return openDummyStore();
}
bool DummyStoreConfig::getReadOnly() const
{
return readOnly.get() || StoreConfig::getReadOnly();
}
struct DummyStoreImpl : DummyStore
{
using Config = DummyStoreConfig;
@@ -296,7 +301,7 @@ struct DummyStoreImpl : DummyStore
StorePath writeDerivation(const Derivation & drv, RepairFlag repair = NoRepair) override
{
auto drvPath = ::nix::writeDerivation(*this, drv, repair, /*readonly=*/true);
auto drvPath = nix::computeStorePath(*this, drv);
if (!derivations.contains(drvPath) || repair) {
if (config->readOnly)
+49 -21
View File
@@ -35,6 +35,22 @@ namespace nix {
const unsigned int RETRY_TIME_MS_DEFAULT = 250;
const unsigned int RETRY_TIME_MS_TOO_MANY_REQUESTS = 60000;
std::filesystem::path FileTransferSettings::getDefaultSSLCertFile()
{
for (auto & fn :
{"/etc/ssl/certs/ca-certificates.crt", "/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"})
if (pathAccessible(fn))
return fn;
return "";
}
FileTransferSettings::FileTransferSettings()
{
auto sslOverride = getEnv("NIX_SSL_CERT_FILE").value_or(getEnv("SSL_CERT_FILE").value_or(""));
if (sslOverride != "")
caFile = sslOverride;
}
FileTransferSettings fileTransferSettings;
static GlobalConfig::Register rFileTransferSettings(&fileTransferSettings);
@@ -44,12 +60,12 @@ namespace {
using curlSList = std::unique_ptr<::curl_slist, decltype([](::curl_slist * list) { ::curl_slist_free_all(list); })>;
using curlMulti = std::unique_ptr<::CURLM, decltype([](::CURLM * multi) { ::curl_multi_cleanup(multi); })>;
struct curlMultiError : Error
struct curlMultiError final : CloneableError<curlMultiError, Error>
{
::CURLMcode code;
curlMultiError(::CURLMcode code)
: Error{"unexpected curl multi error: %s", ::curl_multi_strerror(code)}
: CloneableError{"unexpected curl multi error: %s", ::curl_multi_strerror(code)}
{
assert(code != CURLM_OK);
}
@@ -59,6 +75,8 @@ struct curlMultiError : Error
struct curlFileTransfer : public FileTransfer
{
const FileTransferSettings & settings;
curlMulti curlm;
std::random_device rd;
@@ -464,10 +482,11 @@ struct curlFileTransfer : public FileTransfer
req,
CURLOPT_USERAGENT,
("curl/" LIBCURL_VERSION " Nix/" + nixVersion
+ (fileTransferSettings.userAgentSuffix != "" ? " " + fileTransferSettings.userAgentSuffix.get() : ""))
+ (fileTransfer.settings.userAgentSuffix != "" ? " " + fileTransfer.settings.userAgentSuffix.get()
: ""))
.c_str());
curl_easy_setopt(req, CURLOPT_PIPEWAIT, 1);
if (fileTransferSettings.enableHttp2)
if (fileTransfer.settings.enableHttp2)
curl_easy_setopt(req, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2TLS);
else
curl_easy_setopt(req, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
@@ -482,8 +501,9 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_HTTPHEADER, requestHeaders.get());
if (settings.downloadSpeed.get() > 0)
curl_easy_setopt(req, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t) (settings.downloadSpeed.get() * 1024));
if (fileTransfer.settings.downloadSpeed.get() > 0)
curl_easy_setopt(
req, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t) (fileTransfer.settings.downloadSpeed.get() * 1024));
if (request.method == HttpMethod::Head)
curl_easy_setopt(req, CURLOPT_NOBODY, 1);
@@ -512,20 +532,26 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_SEEKDATA, this);
}
if (settings.caFile != "")
curl_easy_setopt(req, CURLOPT_CAINFO, settings.caFile.get().c_str());
if (auto & caFile = fileTransfer.settings.caFile.get())
curl_easy_setopt(req, CURLOPT_CAINFO, caFile->c_str());
#if !defined(_WIN32)
curl_easy_setopt(req, CURLOPT_SOCKOPTFUNCTION, cloexec_callback);
#endif
curl_easy_setopt(req, CURLOPT_CONNECTTIMEOUT, fileTransferSettings.connectTimeout.get());
curl_easy_setopt(req, CURLOPT_CONNECTTIMEOUT, fileTransfer.settings.connectTimeout.get());
// Enable TCP keep-alive so that idle connections in curl's reuse pool
// are not silently dropped by NATs, firewalls, or load balancers.
curl_easy_setopt(req, CURLOPT_TCP_KEEPALIVE, 1L);
curl_easy_setopt(req, CURLOPT_TCP_KEEPIDLE, 60L);
curl_easy_setopt(req, CURLOPT_TCP_KEEPINTVL, 60L);
curl_easy_setopt(req, CURLOPT_LOW_SPEED_LIMIT, 1L);
curl_easy_setopt(req, CURLOPT_LOW_SPEED_TIME, fileTransferSettings.stalledDownloadTimeout.get());
curl_easy_setopt(req, CURLOPT_LOW_SPEED_TIME, fileTransfer.settings.stalledDownloadTimeout.get());
/* If no file exist in the specified path, curl continues to work
anyway as if netrc support was disabled. */
curl_easy_setopt(req, CURLOPT_NETRC_FILE, settings.netrcFile.get().c_str());
curl_easy_setopt(req, CURLOPT_NETRC_FILE, fileTransfer.settings.netrcFile.get().c_str());
curl_easy_setopt(req, CURLOPT_NETRC, CURL_NETRC_OPTIONAL);
if (writtenToSink)
@@ -711,7 +737,7 @@ struct curlFileTransfer : public FileTransfer
download after a while. If we're writing to a
sink, we can only retry if the server supports
ranged requests. */
if (err == Transient && attempt < request.tries
if (err == Transient && attempt < fileTransfer.settings.tries
&& (!this->request.dataCallback || writtenToSink == 0 || (acceptRanges && encoding.empty()))) {
int ms = retryTimeMs
* std::pow(
@@ -771,10 +797,12 @@ struct curlFileTransfer : public FileTransfer
std::thread workerThread;
const size_t maxQueueSize = fileTransferSettings.httpConnections.get() * 5;
const size_t maxQueueSize;
curlFileTransfer()
: mt19937(rd())
curlFileTransfer(const FileTransferSettings & settings)
: settings(settings)
, mt19937(rd())
, maxQueueSize(settings.httpConnections.get() * 5)
{
static std::once_flag globalInit;
std::call_once(globalInit, curl_global_init, CURL_GLOBAL_ALL);
@@ -782,7 +810,7 @@ struct curlFileTransfer : public FileTransfer
curlm = curlMulti(curl_multi_init());
curl_multi_setopt(curlm.get(), CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
curl_multi_setopt(curlm.get(), CURLMOPT_MAX_TOTAL_CONNECTIONS, fileTransferSettings.httpConnections.get());
curl_multi_setopt(curlm.get(), CURLMOPT_MAX_TOTAL_CONNECTIONS, settings.httpConnections.get());
workerThread = std::thread([&]() { workerThreadEntry(); });
}
@@ -983,9 +1011,9 @@ struct curlFileTransfer : public FileTransfer
}
};
ref<curlFileTransfer> makeCurlFileTransfer()
ref<curlFileTransfer> makeCurlFileTransfer(const FileTransferSettings & settings = fileTransferSettings)
{
return make_ref<curlFileTransfer>();
return make_ref<curlFileTransfer>(settings);
}
ref<FileTransfer> getFileTransfer()
@@ -998,9 +1026,9 @@ ref<FileTransfer> getFileTransfer()
return fileTransfer;
}
ref<FileTransfer> makeFileTransfer()
ref<FileTransfer> makeFileTransfer(const FileTransferSettings & settings)
{
return makeCurlFileTransfer();
return makeCurlFileTransfer(settings);
}
void FileTransferRequest::setupForS3()
@@ -1184,7 +1212,7 @@ void FileTransfer::download(
template<typename... Args>
FileTransferError::FileTransferError(
FileTransfer::Error error, std::optional<std::string> response, const Args &... args)
: Error(args...)
: CloneableError(args...)
, error(error)
, response(response)
{
+30 -3
View File
@@ -3,6 +3,7 @@
#include "nix/store/local-gc.hh"
#include "nix/store/local-store.hh"
#include "nix/store/path.hh"
#include "nix/util/configuration.hh"
#include "nix/util/finally.hh"
#include "nix/util/unix-domain-socket.hh"
#include "nix/util/signals.hh"
@@ -306,9 +307,33 @@ Roots LocalStore::findRoots(bool censor)
return roots;
}
static Roots requestRuntimeRoots(const LocalStoreConfig & config, const std::filesystem::path & socketPath)
{
Roots roots;
auto socket = connect(socketPath);
auto socketSource = FdSource(socket.get());
while (1) {
auto line = socketSource.readLine(true, '\0');
if (line == "")
break;
roots[config.parseStorePath(line)].insert(censored);
};
return roots;
}
void LocalStore::findRuntimeRoots(Roots & roots, bool censor)
{
auto unchecked = findRuntimeRootsUnchecked(*config);
Roots unchecked;
if (config->useRootsDaemon) {
experimentalFeatureSettings.require(Xp::LocalOverlayStore);
unchecked = requestRuntimeRoots(*config, config->getRootsSocketPath());
} else {
unchecked = findRuntimeRootsUnchecked(*config);
}
for (auto & [path, links] : unchecked) {
if (!isValidPath(path))
@@ -762,12 +787,14 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
accounting. */
}
auto st = stat(linksDir);
int64_t overhead =
#ifdef _WIN32
0
#else
st.st_blocks * 512ULL
[&] {
auto st = stat(linksDir);
return st.st_blocks * 512ULL;
}()
#endif
;
+133 -49
View File
@@ -1,11 +1,16 @@
#include "nix/store/globals.hh"
#include "nix/store/profiles.hh"
#include "nix/util/config-impl.hh"
#include "nix/util/config-global.hh"
#include "nix/util/current-process.hh"
#include "nix/util/executable-path.hh"
#include "nix/util/archive.hh"
#include "nix/util/args.hh"
#include "nix/util/abstract-setting-to-json.hh"
#include "nix/util/compute-levels.hh"
#include "nix/util/executable-path.hh"
#include "nix/util/signals.hh"
#include "nix/store/filetransfer.hh"
#include <algorithm>
#include <map>
@@ -48,7 +53,7 @@ namespace nix {
Nix daemon by setting the mode/ownership of the directory
appropriately. (This wouldn't work on the socket itself since it
must be deleted and recreated on startup.) */
#define DEFAULT_SOCKET_PATH "/daemon-socket/socket"
#define DEFAULT_SOCKET_PATH "daemon-socket/socket"
/**
* Helper to resolve the NIX_CONF_DIR at runtime on Windows.
@@ -79,30 +84,16 @@ Settings settings;
static GlobalConfig::Register rSettings(&settings);
Settings::Settings()
: nixStore(
#ifndef _WIN32
// On Windows `/nix/store` is not a canonical path, but we dont'
// want to deal with that yet.
canonPath
#endif
(getEnvNonEmpty("NIX_STORE_DIR").value_or(getEnvNonEmpty("NIX_STORE").value_or(NIX_STORE_DIR))))
, nixStateDir(canonPath(getEnvNonEmpty("NIX_STATE_DIR").value_or(NIX_STATE_DIR)))
, nixConfDir(canonPath(getEnvOsNonEmpty(OS_STR("NIX_CONF_DIR"))
.transform([](auto && s) { return std::filesystem::path(s); })
.value_or(resolveNixConfDir())))
, nixUserConfFiles(getUserConfigFiles())
, nixDaemonSocketFile(
canonPath(getEnvNonEmpty("NIX_DAEMON_SOCKET_PATH").value_or(nixStateDir + DEFAULT_SOCKET_PATH)))
: nixStateDir(canonPath(getEnvNonEmpty("NIX_STATE_DIR").value_or(NIX_STATE_DIR)))
, nixDaemonSocketFile(canonPath(getEnvOsNonEmpty(OS_STR("NIX_DAEMON_SOCKET_PATH"))
.transform([](auto && s) { return std::filesystem::path(s); })
.value_or(nixStateDir / DEFAULT_SOCKET_PATH)))
{
#ifndef _WIN32
buildUsersGroup = isRootUser() ? "nixbld" : "";
#endif
allowSymlinkedStore = getEnv("NIX_IGNORE_SYMLINK_STORE") == "1";
auto sslOverride = getEnv("NIX_SSL_CERT_FILE").value_or(getEnv("SSL_CERT_FILE").value_or(""));
if (sslOverride != "")
caFile = sslOverride;
/* Backwards compatibility. */
auto s = getEnv("NIX_REMOTE_SYSTEMS");
if (s) {
@@ -135,23 +126,23 @@ Settings::Settings()
void loadConfFile(AbstractConfig & config)
{
auto applyConfigFile = [&](const Path & path) {
auto applyConfigFile = [&](const std::filesystem::path & path) {
try {
std::string contents = readFile(path);
config.applyConfig(contents, path);
config.applyConfig(contents, path.string());
} catch (SystemError &) {
}
};
applyConfigFile((settings.nixConfDir / "nix.conf").string());
applyConfigFile(nixConfFile());
/* We only want to send overrides to the daemon, i.e. stuff from
~/.nix/nix.conf or the command line. */
config.resetOverridden();
auto files = settings.nixUserConfFiles;
auto files = nixUserConfFiles();
for (auto file = files.rbegin(); file != files.rend(); file++) {
applyConfigFile(*file);
applyConfigFile(file->string());
}
auto nixConfEnv = getEnv("NIX_CONFIG");
@@ -160,20 +151,32 @@ void loadConfFile(AbstractConfig & config)
}
}
std::vector<Path> getUserConfigFiles()
const std::filesystem::path & nixConfDir()
{
// Use the paths specified in NIX_USER_CONF_FILES if it has been defined
auto nixConfFiles = getEnv("NIX_USER_CONF_FILES");
if (nixConfFiles.has_value()) {
return tokenizeString<std::vector<std::string>>(nixConfFiles.value(), ":");
}
static const std::filesystem::path dir =
canonPath(getEnvOsNonEmpty(OS_STR("NIX_CONF_DIR"))
.transform([](auto && s) { return std::filesystem::path(s); })
.value_or(resolveNixConfDir()));
return dir;
}
// Use the paths specified by the XDG spec
std::vector<Path> files;
auto dirs = getConfigDirs();
for (auto & dir : dirs) {
files.insert(files.end(), (dir / "nix.conf").string());
}
const std::vector<std::filesystem::path> & nixUserConfFiles()
{
static const std::vector<std::filesystem::path> files = [] {
// Use the paths specified in NIX_USER_CONF_FILES if it has been defined
auto nixConfFiles = getEnvOs(OS_STR("NIX_USER_CONF_FILES"));
if (nixConfFiles.has_value()) {
return ExecutablePath::parse(*nixConfFiles).directories;
}
// Use the paths specified by the XDG spec
std::vector<std::filesystem::path> files;
auto dirs = getConfigDirs();
for (auto & dir : dirs) {
files.insert(files.end(), dir / "nix.conf");
}
return files;
}();
return files;
}
@@ -276,16 +279,7 @@ bool Settings::isWSL1()
#endif
}
Path Settings::getDefaultSSLCertFile()
{
for (auto & fn :
{"/etc/ssl/certs/ca-certificates.crt", "/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"})
if (pathAccessible(fn))
return fn;
return "";
}
const ExternalBuilder * Settings::findExternalDerivationBuilderIfSupported(const Derivation & drv)
const ExternalBuilder * LocalSettings::findExternalDerivationBuilderIfSupported(const Derivation & drv)
{
if (auto it = std::ranges::find_if(
externalBuilders.get(), [&](const auto & handler) { return handler.systems.contains(drv.platform); });
@@ -294,6 +288,14 @@ const ExternalBuilder * Settings::findExternalDerivationBuilderIfSupported(const
return nullptr;
}
ProfileDirsOptions Settings::getProfileDirsOptions() const
{
return {
.nixStateDir = nixStateDir,
.useXDGBaseDirectories = useXDGBaseDirectories,
};
}
std::string nixVersion = PACKAGE_VERSION;
NLOHMANN_JSON_SERIALIZE_ENUM(
@@ -427,17 +429,17 @@ unsigned int MaxBuildJobsSetting::parse(const std::string & str) const
}
template<>
Settings::ExternalBuilders BaseSetting<Settings::ExternalBuilders>::parse(const std::string & str) const
LocalSettings::ExternalBuilders BaseSetting<LocalSettings::ExternalBuilders>::parse(const std::string & str) const
{
try {
return nlohmann::json::parse(str).template get<Settings::ExternalBuilders>();
return nlohmann::json::parse(str).template get<LocalSettings::ExternalBuilders>();
} catch (std::exception & e) {
throw UsageError("parsing setting '%s': %s", name, e.what());
}
}
template<>
std::string BaseSetting<Settings::ExternalBuilders>::to_string() const
std::string BaseSetting<LocalSettings::ExternalBuilders>::to_string() const
{
return nlohmann::json(value).dump();
}
@@ -450,6 +452,88 @@ void BaseSetting<PathsInChroot>::appendOrSet(PathsInChroot newValue, bool append
value.insert(std::make_move_iterator(newValue.begin()), std::make_move_iterator(newValue.end()));
}
template<>
struct BaseSetting<std::vector<StoreReference>>::trait
{
static constexpr bool appendable = true;
};
template<>
struct BaseSetting<std::set<StoreReference>>::trait
{
static constexpr bool appendable = true;
};
template<>
StoreReference BaseSetting<StoreReference>::parse(const std::string & str) const
{
return StoreReference::parse(str);
}
template<>
std::string BaseSetting<StoreReference>::to_string() const
{
return value.render();
}
template<>
std::vector<StoreReference> BaseSetting<std::vector<StoreReference>>::parse(const std::string & str) const
{
std::vector<StoreReference> res;
for (const auto & s : tokenizeString<Strings>(str))
res.push_back(StoreReference::parse(s));
return res;
}
template<>
std::string BaseSetting<std::vector<StoreReference>>::to_string() const
{
Strings ss;
for (const auto & ref : value)
ss.push_back(ref.render());
return concatStringsSep(" ", ss);
}
template<>
void BaseSetting<std::vector<StoreReference>>::appendOrSet(std::vector<StoreReference> newValue, bool append)
{
if (append)
value.insert(value.end(), std::make_move_iterator(newValue.begin()), std::make_move_iterator(newValue.end()));
else
value = std::move(newValue);
}
template<>
std::set<StoreReference> BaseSetting<std::set<StoreReference>>::parse(const std::string & str) const
{
std::set<StoreReference> res;
for (const auto & s : tokenizeString<Strings>(str))
res.insert(StoreReference::parse(s));
return res;
}
template<>
std::string BaseSetting<std::set<StoreReference>>::to_string() const
{
Strings ss;
for (const auto & ref : value)
ss.push_back(ref.render());
return concatStringsSep(" ", ss);
}
template<>
void BaseSetting<std::set<StoreReference>>::appendOrSet(std::set<StoreReference> newValue, bool append)
{
if (append)
value.insert(std::make_move_iterator(newValue.begin()), std::make_move_iterator(newValue.end()));
else
value = std::move(newValue);
}
template class BaseSetting<StoreReference>;
template class BaseSetting<std::vector<StoreReference>>;
template class BaseSetting<std::set<StoreReference>>;
static void preloadNSS()
{
/* builtin:fetchurl can trigger a DNS lookup, which with glibc can trigger a dynamic library load of
+43 -26
View File
@@ -2,8 +2,10 @@
#include "nix/store/filetransfer.hh"
#include "nix/store/globals.hh"
#include "nix/store/nar-info-disk-cache.hh"
#include "nix/store/sqlite.hh"
#include "nix/util/callback.hh"
#include "nix/store/store-registration.hh"
#include "nix/store/globals.hh"
namespace nix {
@@ -20,12 +22,20 @@ StringSet HttpBinaryCacheStoreConfig::uriSchemes()
HttpBinaryCacheStoreConfig::HttpBinaryCacheStoreConfig(
std::string_view scheme, std::string_view _cacheUri, const Params & params)
: HttpBinaryCacheStoreConfig(
parseURL(
std::string{scheme} + "://"
+ (!_cacheUri.empty()
? _cacheUri
: throw UsageError("`%s` Store requires a non-empty authority in Store URL", scheme))),
params)
{
}
HttpBinaryCacheStoreConfig::HttpBinaryCacheStoreConfig(ParsedURL _cacheUri, const Params & params)
: StoreConfig(params)
, BinaryCacheStoreConfig(params)
, cacheUri(parseURL(
std::string{scheme} + "://"
+ (!_cacheUri.empty() ? _cacheUri
: throw UsageError("`%s` Store requires a non-empty authority in Store URL", scheme))))
, cacheUri(std::move(_cacheUri))
{
while (!cacheUri.path.empty() && cacheUri.path.back() == "")
cacheUri.path.pop_back();
@@ -50,12 +60,13 @@ std::string HttpBinaryCacheStoreConfig::doc()
;
}
HttpBinaryCacheStore::HttpBinaryCacheStore(ref<Config> config)
HttpBinaryCacheStore::HttpBinaryCacheStore(ref<Config> config, ref<FileTransfer> fileTransfer)
: Store{*config} // TODO it will actually mutate the configuration
, BinaryCacheStore{*config}
, fileTransfer{fileTransfer}
, config{config}
{
diskCache = getNarInfoDiskCache();
diskCache = NarInfoDiskCache::get(settings.getNarInfoDiskCacheSettings(), {.useWAL = settings.useSQLiteWAL});
}
void HttpBinaryCacheStore::init()
@@ -93,7 +104,7 @@ std::optional<CompressionAlgo> HttpBinaryCacheStore::getCompressionMethod(const
void HttpBinaryCacheStore::maybeDisable()
{
auto state(_state.lock());
if (state->enabled && settings.tryFallback) {
if (state->enabled && settings.getWorkerSettings().tryFallback) {
int t = 60;
printError("disabling binary cache '%s' for %s seconds", config->getHumanReadableURI(), t);
state->enabled = false;
@@ -121,7 +132,7 @@ bool HttpBinaryCacheStore::fileExists(const std::string & path)
try {
FileTransferRequest request(makeRequest(path));
request.method = HttpMethod::Head;
getFileTransfer()->download(request);
fileTransfer->download(request);
return true;
} catch (FileTransferError & e) {
/* S3 buckets return 403 if a file doesn't exist and the
@@ -151,7 +162,7 @@ void HttpBinaryCacheStore::upload(
req.data = {sizeHint, source};
req.mimeType = mimeType;
getFileTransfer()->upload(req);
fileTransfer->upload(req);
}
void HttpBinaryCacheStore::upsertFile(
@@ -219,7 +230,7 @@ void HttpBinaryCacheStore::getFile(const std::string & path, Sink & sink)
checkEnabled();
auto request(makeRequest(path));
try {
getFileTransfer()->download(std::move(request), sink);
fileTransfer->download(std::move(request), sink);
} catch (FileTransferError & e) {
if (e.error == FileTransfer::NotFound || e.error == FileTransfer::Forbidden)
throw NoSuchBinaryCacheFile(
@@ -238,19 +249,19 @@ void HttpBinaryCacheStore::getFile(const std::string & path, Callback<std::optio
auto request(makeRequest(path));
getFileTransfer()->enqueueFileTransfer(request, {[callbackPtr, this](std::future<FileTransferResult> result) {
try {
(*callbackPtr)(std::move(result.get().data));
} catch (FileTransferError & e) {
if (e.error == FileTransfer::NotFound
|| e.error == FileTransfer::Forbidden)
return (*callbackPtr)({});
maybeDisable();
callbackPtr->rethrow();
} catch (...) {
callbackPtr->rethrow();
}
}});
fileTransfer->enqueueFileTransfer(request, {[callbackPtr, this](std::future<FileTransferResult> result) {
try {
(*callbackPtr)(std::move(result.get().data));
} catch (FileTransferError & e) {
if (e.error == FileTransfer::NotFound
|| e.error == FileTransfer::Forbidden)
return (*callbackPtr)({});
maybeDisable();
callbackPtr->rethrow();
} catch (...) {
callbackPtr->rethrow();
}
}});
} catch (...) {
callbackPtr->rethrow();
@@ -261,7 +272,7 @@ void HttpBinaryCacheStore::getFile(const std::string & path, Callback<std::optio
std::optional<std::string> HttpBinaryCacheStore::getNixCacheInfo()
{
try {
auto result = getFileTransfer()->download(makeRequest(cacheInfoFile));
auto result = fileTransfer->download(makeRequest(cacheInfoFile));
return result.data;
} catch (FileTransferError & e) {
if (e.error == FileTransfer::NotFound)
@@ -284,11 +295,17 @@ std::optional<TrustedFlag> HttpBinaryCacheStore::isTrustedClient()
return std::nullopt;
}
ref<Store> HttpBinaryCacheStore::Config::openStore() const
ref<Store> HttpBinaryCacheStore::Config::openStore(ref<FileTransfer> fileTransfer) const
{
return make_ref<HttpBinaryCacheStore>(
ref{// FIXME we shouldn't actually need a mutable config
std::const_pointer_cast<HttpBinaryCacheStore::Config>(shared_from_this())});
std::const_pointer_cast<HttpBinaryCacheStore::Config>(shared_from_this())},
fileTransfer);
}
ref<Store> HttpBinaryCacheStoreConfig::openStore() const
{
return openStore(getFileTransfer());
}
static RegisterStoreImplementation<HttpBinaryCacheStore::Config> regHttpBinaryCacheStore;
+3 -2
View File
@@ -34,12 +34,13 @@ struct AwsCredentials
}
};
class AwsAuthError : public Error
class AwsAuthError final : public CloneableError<AwsAuthError, Error>
{
std::optional<int> errorCode;
public:
using Error::Error;
using CloneableError::CloneableError;
AwsAuthError(int errorCode);
std::optional<int> getErrorCode() const
@@ -18,7 +18,7 @@ struct BinaryCacheStoreConfig : virtual StoreConfig
{
using StoreConfig::StoreConfig;
const Setting<CompressionAlgo> compression{
Setting<CompressionAlgo> compression{
this,
CompressionAlgo::xz,
"compression",
@@ -27,10 +27,10 @@ struct BinaryCacheStoreConfig : virtual StoreConfig
To use a particular compression method Nix has to be built with a version of libarchive that natively supports that compression algorithm.
)"};
const Setting<bool> writeNARListing{
Setting<bool> writeNARListing{
this, false, "write-nar-listing", "Whether to write a JSON file that lists the files in each NAR."};
const Setting<bool> writeDebugInfo{
Setting<bool> writeDebugInfo{
this,
false,
"index-debug-info",
@@ -39,24 +39,24 @@ struct BinaryCacheStoreConfig : virtual StoreConfig
fetch debug info on demand
)"};
const Setting<Path> secretKeyFile{this, "", "secret-key", "Path to the secret key used to sign the binary cache."};
Setting<Path> secretKeyFile{this, "", "secret-key", "Path to the secret key used to sign the binary cache."};
const Setting<std::string> secretKeyFiles{
Setting<std::string> secretKeyFiles{
this, "", "secret-keys", "List of comma-separated paths to the secret keys used to sign the binary cache."};
const Setting<std::optional<std::filesystem::path>> localNarCache{
Setting<std::optional<std::filesystem::path>> localNarCache{
this,
std::nullopt,
"local-nar-cache",
"Path to a local cache of NARs fetched from this binary cache, used by commands such as `nix store cat`."};
const Setting<bool> parallelCompression{
Setting<bool> parallelCompression{
this,
false,
"parallel-compression",
"Enable multi-threaded compression of NARs. This is currently only available for `xz` and `zstd`."};
const Setting<int> compressionLevel{
Setting<int> compressionLevel{
this,
-1,
"compression-level",
@@ -58,7 +58,7 @@ enum struct BuildResultFailureStatus : uint8_t {
* This is both an exception type (inherits from Error) and serves as
* the failure variant in BuildResult::inner.
*/
struct BuildError : public Error
struct BuildError : public CloneableError<BuildError, Error>
{
using Status = BuildResultFailureStatus;
using enum Status;
@@ -80,7 +80,7 @@ public:
*/
template<typename... Args>
BuildError(Status status, const Args &... args)
: Error(args...)
: CloneableError(args...)
, status{status}
{
}
@@ -97,7 +97,7 @@ public:
* Also used for deserialization.
*/
BuildError(Args args)
: Error(std::move(args.msg))
: CloneableError(std::move(args.msg))
, status{args.status}
, isNonDeterministic{args.isNonDeterministic}
@@ -108,7 +108,7 @@ public:
* Default constructor for deserialization.
*/
BuildError()
: Error("")
: CloneableError("")
{
}
@@ -20,14 +20,14 @@ namespace nix {
* Denotes a build failure that stemmed from the builder exiting with a
* failing exist status.
*/
struct BuilderFailureError : BuildError
struct BuilderFailureError final : CloneableError<BuilderFailureError, BuildError>
{
int builderStatus;
std::string extraMsgAfter;
BuilderFailureError(BuildResult::Failure::Status status, int builderStatus, std::string extraMsgAfter)
: BuildError{
: CloneableError{
status,
/* No message for now, because the caller will make for
us, with extra context */
@@ -183,6 +183,19 @@ struct DerivationBuilder : RestrictionContext
virtual bool killChild() = 0;
};
/**
* Run a callback that may change process credentials (setuid, setgid, etc.)
* while preserving the parent-death signal.
*
* The parent-death signal setting is cleared by the Linux kernel upon changes
* to EUID, EGID.
*
* @note Does nothing on non-Linux systems.
* @see man PR_SET_PDEATHSIG
* @see https://github.com/golang/go/issues/9686
*/
void preserveDeathSignal(std::function<void()> fn);
struct ExternalBuilder
{
StringSet systems;
@@ -64,6 +64,12 @@ private:
std::string key() override;
struct LocalBuildCapability
{
LocalStore & localStore;
const ExternalBuilder * externalBuilder;
};
/**
* The states.
*/
@@ -75,12 +81,11 @@ private:
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks);
Co buildLocally(
LocalStore & localStore,
LocalBuildCapability localBuildCap,
StorePathSet inputPaths,
std::map<std::string, InitialOutput> initialOutputs,
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks,
const ExternalBuilder * externalBuilder);
PathLocks outputLocks);
/**
* Is the build hook willing to perform the build?
+1 -1
View File
@@ -10,7 +10,7 @@
namespace nix {
struct TimedOut : BuildError
struct TimedOut final : CloneableError<TimedOut, BuildError>
{
time_t maxDuration;
@@ -16,6 +16,7 @@
namespace nix {
/* Forward definition. */
struct WorkerSettings;
struct DerivationTrampolineGoal;
struct DerivationGoal;
struct DerivationResolutionGoal;
@@ -156,6 +157,7 @@ public:
Store & store;
Store & evalStore;
const WorkerSettings & settings;
/**
* Function to get the substituters to use for path substitution.
@@ -16,6 +16,7 @@ struct BuiltinBuilderContext
std::map<std::string, Path> outputs;
std::string netrcData;
std::string caFileData;
Strings hashedMirrors;
Path tmpDirInSandbox;
#if NIX_WITH_AWS_AUTH
@@ -22,7 +22,7 @@ struct Package
}
};
class BuildEnvFileConflictError : public Error
class BuildEnvFileConflictError final : public CloneableError<BuildEnvFileConflictError, Error>
{
public:
const Path fileA;
@@ -30,7 +30,7 @@ public:
int priority;
BuildEnvFileConflictError(const Path fileA, const Path fileB, int priority)
: Error(
: CloneableError(
"Unable to build profile. There is a conflict for the following files:\n"
"\n"
" %1%\n"
@@ -15,15 +15,15 @@ struct CommonSSHStoreConfig : virtual StoreConfig
CommonSSHStoreConfig(std::string_view scheme, const ParsedURL::Authority & authority, const Params & params);
CommonSSHStoreConfig(std::string_view scheme, std::string_view authority, const Params & params);
const Setting<Path> sshKey{
Setting<Path> sshKey{
this, "", "ssh-key", "Path to the SSH private key used to authenticate to the remote machine."};
const Setting<std::string> sshPublicHostKey{
Setting<std::string> sshPublicHostKey{
this, "", "base64-ssh-public-host-key", "The public host key of the remote machine."};
const Setting<bool> compress{this, false, "compress", "Whether to enable SSH compression."};
Setting<bool> compress{this, false, "compress", "Whether to enable SSH compression."};
const Setting<std::string> remoteStore{
Setting<std::string> remoteStore{
this,
"",
"remote-store",
@@ -10,10 +10,10 @@
#include "nix/util/json-impls.hh"
#include "nix/store/store-dir-config.hh"
#include "nix/store/downstream-placeholder.hh"
#include "nix/store/worker-settings.hh"
namespace nix {
class Store;
struct StoreDirConfig;
struct BasicDerivation;
struct StructuredAttrs;
@@ -183,17 +183,7 @@ struct DerivationOptions
*/
StringSet getRequiredSystemFeatures(const BasicDerivation & drv) const;
/**
* @param drv See note on `getRequiredSystemFeatures`
*/
bool canBuildLocally(Store & localStore, const BasicDerivation & drv) const;
/**
* @param drv See note on `getRequiredSystemFeatures`
*/
bool willBuildLocally(Store & localStore, const BasicDerivation & drv) const;
bool substitutesAllowed() const;
bool substitutesAllowed(const WorkerSettings & workerSettings) const;
/**
* @param drv See note on `getRequiredSystemFeatures`
@@ -466,9 +466,11 @@ struct Derivation : BasicDerivation
class Store;
/**
* Write a derivation to the Nix store, and return its path.
* Compute the store path that would be used for a derivation without writing it.
*
* This is a pure computation based on the derivation content and store directory.
*/
StorePath writeDerivation(Store & store, const Derivation & drv, RepairFlag repair = NoRepair, bool readOnly = false);
StorePath computeStorePath(const StoreDirConfig & store, const Derivation & drv);
/**
* Read a derivation from a file.
@@ -35,6 +35,8 @@ struct DummyStoreConfig : public std::enable_shared_from_this<DummyStoreConfig>,
No additional memory will be used, because no information needs to be stored.
)"};
bool getReadOnly() const override;
static const std::string name()
{
return "Dummy Store";
+68 -3
View File
@@ -19,8 +19,16 @@
namespace nix {
const std::filesystem::path & nixConfDir();
struct FileTransferSettings : Config
{
private:
static std::filesystem::path getDefaultSSLCertFile();
public:
FileTransferSettings();
Setting<bool> enableHttp2{this, true, "http2", "Whether to enable HTTP/2 support."};
Setting<std::string> userAgentSuffix{
@@ -77,6 +85,64 @@ struct FileTransferSettings : Config
not processed quickly enough to exceed the size of this buffer, downloads may stall.
The default is 1048576 (1 MiB).
)"};
Setting<unsigned int> downloadSpeed{
this,
0,
"download-speed",
R"(
Specify the maximum transfer rate in kilobytes per second you want
Nix to use for downloads.
)"};
Setting<std::filesystem::path> netrcFile{
this,
nixConfDir() / "netrc",
"netrc-file",
R"(
If set to an absolute path to a `netrc` file, Nix uses the HTTP
authentication credentials in this file when trying to download from
a remote host through HTTP or HTTPS. Defaults to
`$NIX_CONF_DIR/netrc`.
The `netrc` file consists of a list of accounts in the following
format:
machine my-machine
login my-username
password my-password
For the exact syntax, see [the `curl`
documentation](https://ec.haxx.se/usingcurl-netrc.html).
> **Note**
>
> This must be an absolute path, and `~` is not resolved. For
> example, `~/.netrc` won't resolve to your home directory's
> `.netrc`.
)"};
Setting<std::optional<std::filesystem::path>> caFile{
this,
getDefaultSSLCertFile(),
"ssl-cert-file",
R"(
The path of a file containing CA certificates used to
authenticate `https://` downloads. Nix by default uses
the first of the following files that exists:
1. `/etc/ssl/certs/ca-certificates.crt`
2. `/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt`
The path can be overridden by the following environment
variables, in order of precedence:
1. `NIX_SSL_CERT_FILE`
2. `SSL_CERT_FILE`
)",
{},
// Don't document the machine-specific default value
false};
};
extern FileTransferSettings fileTransferSettings;
@@ -116,7 +182,6 @@ struct FileTransferRequest
Headers headers;
std::string expectedETag;
HttpMethod method = HttpMethod::Get;
size_t tries = fileTransferSettings.tries;
unsigned int baseRetryTimeMs = RETRY_TIME_MS_DEFAULT;
ActivityId parentAct;
bool decompress = true;
@@ -336,9 +401,9 @@ ref<FileTransfer> getFileTransfer();
*
* Prefer getFileTransfer() to this; see its docs for why.
*/
ref<FileTransfer> makeFileTransfer();
ref<FileTransfer> makeFileTransfer(const FileTransferSettings & settings = fileTransferSettings);
class FileTransferError : public Error
class FileTransferError final : public CloneableError<FileTransferError, Error>
{
public:
FileTransfer::Error error;
+1 -2
View File
@@ -69,8 +69,7 @@ struct GCResults
PathSet paths;
/**
* For `gcReturnDead`, `gcDeleteDead` and `gcDeleteSpecific`, the
* number of bytes that would be or was freed.
* For `gcDeleteDead` and `gcDeleteSpecific`, the number of bytes that were freed.
*/
uint64_t bytesFreed = 0;
};
@@ -0,0 +1,33 @@
#pragma once
///@file
#include <filesystem>
#include <vector>
namespace nix {
/**
* The directory where system configuration files are stored.
*
* This is needed very early during initialization, before a main
* `Settings` object can be constructed.
*/
const std::filesystem::path & nixConfDir();
/**
* The path to the system configuration file (`nix.conf`).
*/
static inline std::filesystem::path nixConfFile()
{
return nixConfDir() / "nix.conf";
}
/**
* A list of user configuration files to load.
*
* This is needed very early during initialization, before a main
* `Settings` object can be constructed.
*/
const std::vector<std::filesystem::path> & nixUserConfFiles();
} // namespace nix
+81 -536
View File
@@ -8,33 +8,21 @@
#include "nix/util/environment-variables.hh"
#include "nix/store/build/derivation-builder.hh"
#include "nix/store/local-settings.hh"
#include "nix/store/store-reference.hh"
#include "nix/store/worker-settings.hh"
#include "nix/store/config.hh"
namespace nix {
struct MaxBuildJobsSetting : public BaseSetting<unsigned int>
{
MaxBuildJobsSetting(
Config * options,
unsigned int def,
const std::string & name,
const std::string & description,
const StringSet & aliases = {})
: BaseSetting<unsigned int>(def, true, name, description, aliases)
{
options->addSetting(this);
}
unsigned int parse(const std::string & str) const override;
};
struct ProfileDirsOptions;
struct LogFileSettings : public virtual Config
{
/**
* The directory where we log various operations.
*/
const Path nixLogDir;
std::filesystem::path nixLogDir;
protected:
LogFileSettings();
@@ -64,7 +52,47 @@ public:
{"build-compress-log"}};
};
class Settings : public virtual Config, private LocalSettings, private LogFileSettings
struct NarInfoDiskCacheSettings : public virtual Config
{
Setting<unsigned int> ttlNegative{
this,
3600,
"narinfo-cache-negative-ttl",
R"(
The TTL in seconds for negative lookups.
If a store path is queried from a [substituter](#conf-substituters) but was not found, a negative lookup is cached in the local disk cache database for the specified duration.
Set to `0` to force updating the lookup cache.
To wipe the lookup cache completely:
```shell-session
$ rm $HOME/.cache/nix/binary-cache-v*.sqlite*
# rm /root/.cache/nix/binary-cache-v*.sqlite*
```
)"};
Setting<unsigned int> ttlPositive{
this,
30 * 24 * 3600,
"narinfo-cache-positive-ttl",
R"(
The TTL in seconds for positive lookups. If a store path is queried
from a substituter, the result of the query is cached in the
local disk cache database including some of the NAR metadata. The
default TTL is a month, setting a shorter TTL for positive lookups
can be useful for binary caches that have frequent garbage
collection, in which case having a more frequent cache invalidation
would prevent trying to pull the path again and failing with a hash
mismatch if the build isn't reproducible.
)"};
};
class Settings : public virtual Config,
private LocalSettings,
private LogFileSettings,
private WorkerSettings,
private NarInfoDiskCacheSettings
{
StringSet getDefaultSystemFeatures();
@@ -72,14 +100,10 @@ class Settings : public virtual Config, private LocalSettings, private LogFileSe
bool isWSL1();
Path getDefaultSSLCertFile();
public:
Settings();
using ExternalBuilders = std::vector<ExternalBuilder>;
/**
* Get the local store settings.
*/
@@ -106,36 +130,47 @@ public:
return *this;
}
static unsigned int getDefaultCores();
/**
* Get the worker settings.
*/
WorkerSettings & getWorkerSettings()
{
return *this;
}
const WorkerSettings & getWorkerSettings() const
{
return *this;
}
/**
* The directory where we store sources and derived files.
* Get the NAR info disk cache settings.
*/
Path nixStore;
NarInfoDiskCacheSettings & getNarInfoDiskCacheSettings()
{
return *this;
}
const NarInfoDiskCacheSettings & getNarInfoDiskCacheSettings() const
{
return *this;
}
static unsigned int getDefaultCores();
/**
* The directory where state is stored.
*/
Path nixStateDir;
/**
* The directory where system configuration files are stored.
*/
std::filesystem::path nixConfDir;
/**
* A list of user configuration files to load.
*/
std::vector<Path> nixUserConfFiles;
std::filesystem::path nixStateDir;
/**
* File name of the socket the daemon listens to.
*/
Path nixDaemonSocketFile;
std::filesystem::path nixDaemonSocketFile;
Setting<std::string> storeUri{
Setting<StoreReference> storeUri{
this,
getEnv("NIX_REMOTE").value_or("auto"),
StoreReference::parse(getEnv("NIX_REMOTE").value_or("auto")),
"store",
R"(
The [URL of the Nix store](@docroot@/store/types/index.md#store-url-format)
@@ -149,86 +184,11 @@ public:
Setting<bool> keepFailed{this, false, "keep-failed", "Whether to keep temporary directories of failed builds."};
Setting<bool> keepGoing{
this, false, "keep-going", "Whether to keep building derivations when another build fails."};
Setting<bool> tryFallback{
this,
false,
"fallback",
R"(
If set to `true`, Nix falls back to building from source if a
binary substitute fails. This is equivalent to the `--fallback`
flag. The default is `false`.
)",
{"build-fallback"}};
/**
* Whether to show build log output in real time.
*/
bool verboseBuild = true;
Setting<size_t> logLines{
this,
25,
"log-lines",
"The number of lines of the tail of "
"the log to show if a build fails."};
MaxBuildJobsSetting maxBuildJobs{
this,
1,
"max-jobs",
R"(
Maximum number of jobs that Nix tries to build locally in parallel.
The special value `auto` causes Nix to use the number of CPUs in your system.
Use `0` to disable local builds and directly use the remote machines specified in [`builders`](#conf-builders).
This doesn't affect derivations that have [`preferLocalBuild = true`](@docroot@/language/advanced-attributes.md#adv-attr-preferLocalBuild), which are always built locally.
> **Note**
>
> The number of CPU cores to use for each build job is independently determined by the [`cores`](#conf-cores) setting.
<!-- TODO(@fricklerhandwerk): would be good to have those shorthands for common options as part of the specification -->
The setting can be overridden using the `--max-jobs` (`-j`) command line switch.
)",
{"build-max-jobs"}};
Setting<unsigned int> maxSubstitutionJobs{
this,
16,
"max-substitution-jobs",
R"(
This option defines the maximum number of substitution jobs that Nix
tries to run in parallel. The default is `16`. The minimum value
one can choose is `1` and lower values are interpreted as `1`.
)",
{"substitution-max-jobs"}};
Setting<unsigned int> buildCores{
this,
0,
"cores",
R"(
Sets the value of the `NIX_BUILD_CORES` environment variable in the [invocation of the `builder` executable](@docroot@/store/building.md#builder-execution) of a derivation.
The `builder` executable can use this variable to control its own maximum amount of parallelism.
<!--
FIXME(@fricklerhandwerk): I don't think this should even be mentioned here.
A very generic example using `derivation` and `xargs` may be more appropriate to explain the mechanism.
Using `mkDerivation` as an example requires being aware of that there are multiple independent layers that are completely opaque here.
-->
For instance, in Nixpkgs, if the attribute `enableParallelBuilding` for the `mkDerivation` build helper is set to `true`, it passes the `-j${NIX_BUILD_CORES}` flag to GNU Make.
If set to `0`, nix will detect the number of CPU cores and pass this number via `NIX_BUILD_CORES`.
> **Note**
>
> The number of parallel local Nix build jobs is independently controlled with the [`max-jobs`](#conf-max-jobs) setting.
)",
{"build-cores"}};
/**
* Read-only mode. Don't copy stuff to the store, don't change
* the database.
@@ -264,219 +224,6 @@ public:
configuration option is set as the empty string.
)"};
Setting<time_t> maxSilentTime{
this,
0,
"max-silent-time",
R"(
This option defines the maximum number of seconds that a builder can
go without producing any data on standard output or standard error.
This is useful (for instance in an automated build system) to catch
builds that are stuck in an infinite loop, or to catch remote builds
that are hanging due to network problems. It can be overridden using
the `--max-silent-time` command line switch.
The value `0` means that there is no timeout. This is also the
default.
)",
{"build-max-silent-time"}};
Setting<time_t> buildTimeout{
this,
0,
"timeout",
R"(
This option defines the maximum number of seconds that a builder can
run. This is useful (for instance in an automated build system) to
catch builds that are stuck in an infinite loop but keep writing to
their standard output or standard error. It can be overridden using
the `--timeout` command line switch.
The value `0` means that there is no timeout. This is also the
default.
)",
{"build-timeout"}};
Setting<Strings> buildHook{
this,
{"nix", "__build-remote"},
"build-hook",
R"(
The path to the helper program that executes remote builds.
Nix communicates with the build hook over `stdio` using a custom protocol to request builds that cannot be performed directly by the Nix daemon.
The default value is the internal Nix binary that implements remote building.
> **Important**
>
> Change this setting only if you really know what you’re doing.
)"};
Setting<std::string> builders{
this,
"@" + nixConfDir.string() + "/machines",
"builders",
R"(
A semicolon- or newline-separated list of build machines.
In addition to the [usual ways of setting configuration options](@docroot@/command-ref/conf-file.md), the value can be read from a file by prefixing its absolute path with `@`.
> **Example**
>
> This is the default setting:
>
> ```
> builders = @/etc/nix/machines
> ```
Each machine specification consists of the following elements, separated by spaces.
Only the first element is required.
To leave a field at its default, set it to `-`.
1. The URI of the remote store in the format `ssh://[username@]hostname[:port]`.
> **Example**
>
> `ssh://nix@mac`
For backward compatibility, `ssh://` may be omitted.
The hostname may be an alias defined in `~/.ssh/config`.
2. A comma-separated list of [Nix system types](@docroot@/development/building.md#system-type).
If omitted, this defaults to the local platform type.
> **Example**
>
> `aarch64-darwin`
It is possible for a machine to support multiple platform types.
> **Example**
>
> `i686-linux,x86_64-linux`
3. The SSH identity file to be used to log in to the remote machine.
If omitted, SSH uses its regular identities.
> **Example**
>
> `/home/user/.ssh/id_mac`
4. The maximum number of builds that Nix executes in parallel on the machine.
Typically this should be equal to the number of CPU cores.
5. The “speed factor”, indicating the relative speed of the machine as a positive integer.
If there are multiple machines of the right type, Nix prefers the fastest, taking load into account.
6. A comma-separated list of supported [system features](#conf-system-features).
A machine is only used to build a derivation if all the features in the derivation's [`requiredSystemFeatures`](@docroot@/language/advanced-attributes.html#adv-attr-requiredSystemFeatures) attribute are supported by that machine.
7. A comma-separated list of required [system features](#conf-system-features).
A machine is only used to build a derivation if all of the machine’s required features appear in the derivation’s [`requiredSystemFeatures`](@docroot@/language/advanced-attributes.html#adv-attr-requiredSystemFeatures) attribute.
8. The (base64-encoded) public host key of the remote machine.
If omitted, SSH uses its regular `known_hosts` file.
The value for this field can be obtained via `base64 -w0`.
> **Example**
>
> Multiple builders specified on the command line:
>
> ```console
> --builders 'ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd'
> ```
> **Example**
>
> This specifies several machines that can perform `i686-linux` builds:
>
> ```
> nix@scratchy.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy 8 1 kvm
> nix@itchy.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy 8 2
> nix@poochie.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy 1 2 kvm benchmark
> ```
>
> However, `poochie` only builds derivations that have the attribute
>
> ```nix
> requiredSystemFeatures = [ "benchmark" ];
> ```
>
> or
>
> ```nix
> requiredSystemFeatures = [ "benchmark" "kvm" ];
> ```
>
> `itchy` cannot do builds that require `kvm`, but `scratchy` does support such builds.
> For regular builds, `itchy` is preferred over `scratchy` because it has a higher speed factor.
For Nix to use substituters, the calling user must be in the [`trusted-users`](#conf-trusted-users) list.
> **Note**
>
> A build machine must be accessible via SSH and have Nix installed.
> `nix` must be available in `$PATH` for the user connecting over SSH.
> **Warning**
>
> If you are building via the Nix daemon (default), the Nix daemon user account on the local machine (that is, `root`) requires access to a user account on the remote machine (not necessarily `root`).
>
> If you can’t or don’t want to configure `root` to be able to access the remote machine, set [`store`](#conf-store) to any [local store](@docroot@/store/types/local-store.html), e.g. by passing `--store /tmp` to the command on the local machine.
To build only on remote machines and disable local builds, set [`max-jobs`](#conf-max-jobs) to 0.
If you want the remote machines to use substituters, set [`builders-use-substitutes`](#conf-builders-use-substitutes) to `true`.
)",
{},
false};
Setting<bool> alwaysAllowSubstitutes{
this,
false,
"always-allow-substitutes",
R"(
If set to `true`, Nix ignores the [`allowSubstitutes`](@docroot@/language/advanced-attributes.md) attribute in derivations and always attempt to use [available substituters](#conf-substituters).
)"};
Setting<bool> buildersUseSubstitutes{
this,
false,
"builders-use-substitutes",
R"(
If set to `true`, Nix instructs [remote build machines](#conf-builders) to use their own [`substituters`](#conf-substituters) if available.
It means that remote build hosts fetch as many dependencies as possible from their own substituters (e.g, from `cache.nixos.org`) instead of waiting for the local machine to upload them all.
This can drastically reduce build times if the network connection between the local machine and the remote build host is slow.
)"};
Setting<bool> useSubstitutes{
this,
true,
"substitute",
R"(
If set to `true` (default), Nix uses binary substitutes if
available. This option can be disabled to force building from
source.
)",
{"build-use-substitutes"}};
Setting<unsigned long> maxLogSize{
this,
0,
"max-build-log-size",
R"(
This option defines the maximum number of bytes that a builder can
write to its stdout/stderr. If the builder exceeds this limit, it’s
killed. A value of `0` (the default) means that there is no limit.
)",
{"build-max-log-size"}};
Setting<unsigned int> pollInterval{this, 5, "build-poll-interval", "How often (in seconds) to poll for locks."};
Setting<Strings> trustedPublicKeys{
this,
{"cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="},
@@ -507,25 +254,6 @@ public:
can add it to `trusted-public-keys` in their `nix.conf`.
)"};
Setting<unsigned int> tarballTtl{
this,
60 * 60,
"tarball-ttl",
R"(
The number of seconds a downloaded tarball is considered fresh. If
the cached tarball is stale, Nix checks whether it is still up
to date using the ETag header. Nix downloads a new version if
the ETag header is unsupported, or the cached ETag doesn't match.
Setting the TTL to `0` forces Nix to always check if the tarball is
up to date.
Nix caches tarballs in `$XDG_CACHE_HOME/nix/tarballs`.
Files fetched via `NIX_PATH`, `fetchGit`, `fetchMercurial`,
`fetchTarball`, and `fetchurl` respect this TTL.
)"};
Setting<bool> requireSigs{
this,
true,
@@ -616,28 +344,10 @@ public:
// Don't document the machine-specific default value
false};
Setting<Strings> substituters{
this,
Strings{"https://cache.nixos.org/"},
"substituters",
R"(
A list of [URLs of Nix stores](@docroot@/store/types/index.md#store-url-format) to be used as substituters, separated by whitespace.
A substituter is an additional [store](@docroot@/glossary.md#gloss-store) from which Nix can obtain [store objects](@docroot@/store/store-object.md) instead of building them.
Substituters are tried based on their priority value, which each substituter can set independently.
Lower value means higher priority.
The default is `https://cache.nixos.org`, which has a priority of 40.
At least one of the following conditions must be met for Nix to use a substituter:
- The substituter is in the [`trusted-substituters`](#conf-trusted-substituters) list
- The user calling Nix is in the [`trusted-users`](#conf-trusted-users) list
In addition, each store path should be trusted as described in [`trusted-public-keys`](#conf-trusted-public-keys)
)",
{"binary-caches"}};
Setting<StringSet> trustedSubstituters{
// move to daemonsettings in another pass
//
// we'll add a another parameter to processConnection to thread it through
Setting<std::set<StoreReference>> trustedSubstituters{
this,
{},
"trusted-substituters",
@@ -649,171 +359,10 @@ public:
)",
{"trusted-binary-caches"}};
Setting<unsigned int> ttlNegativeNarInfoCache{
this,
3600,
"narinfo-cache-negative-ttl",
R"(
The TTL in seconds for negative lookups.
If a store path is queried from a [substituter](#conf-substituters) but was not found, a negative lookup is cached in the local disk cache database for the specified duration.
Set to `0` to force updating the lookup cache.
To wipe the lookup cache completely:
```shell-session
$ rm $HOME/.cache/nix/binary-cache-v*.sqlite*
# rm /root/.cache/nix/binary-cache-v*.sqlite*
```
)"};
Setting<unsigned int> ttlPositiveNarInfoCache{
this,
30 * 24 * 3600,
"narinfo-cache-positive-ttl",
R"(
The TTL in seconds for positive lookups. If a store path is queried
from a substituter, the result of the query is cached in the
local disk cache database including some of the NAR metadata. The
default TTL is a month, setting a shorter TTL for positive lookups
can be useful for binary caches that have frequent garbage
collection, in which case having a more frequent cache invalidation
would prevent trying to pull the path again and failing with a hash
mismatch if the build isn't reproducible.
)"};
// move it out in the 2nd pass
Setting<bool> printMissing{
this, true, "print-missing", "Whether to print what paths need to be built or downloaded."};
Setting<std::string> postBuildHook{
this,
"",
"post-build-hook",
R"(
Optional. The path to a program to execute after each build.
This option is only settable in the global `nix.conf`, or on the
command line by trusted users.
When using the nix-daemon, the daemon executes the hook as `root`.
If the nix-daemon is not involved, the hook runs as the user
executing the nix-build.
- The hook executes after an evaluation-time build.
- The hook does not execute on substituted paths.
- The hook's output always goes to the user's terminal.
- If the hook fails, the build succeeds but no further builds
execute.
- The hook executes synchronously, and blocks other builds from
progressing while it runs.
The program executes with no arguments. The program's environment
contains the following environment variables:
- `DRV_PATH`
The derivation for the built paths.
Example:
`/nix/store/5nihn1a7pa8b25l9zafqaqibznlvvp3f-bash-4.4-p23.drv`
- `OUT_PATHS`
Output paths of the built derivation, separated by a space
character.
Example:
`/nix/store/l88brggg9hpy96ijds34dlq4n8fan63g-bash-4.4-p23-dev
/nix/store/vch71bhyi5akr5zs40k8h2wqxx69j80l-bash-4.4-p23-doc
/nix/store/c5cxjywi66iwn9dcx5yvwjkvl559ay6p-bash-4.4-p23-info
/nix/store/scz72lskj03ihkcn42ias5mlp4i4gr1k-bash-4.4-p23-man
/nix/store/a724znygmd1cac856j3gfsyvih3lw07j-bash-4.4-p23`.
)"};
Setting<unsigned int> downloadSpeed{
this,
0,
"download-speed",
R"(
Specify the maximum transfer rate in kilobytes per second you want
Nix to use for downloads.
)"};
Setting<std::string> netrcFile{
this,
(nixConfDir / "netrc").string(),
"netrc-file",
R"(
If set to an absolute path to a `netrc` file, Nix uses the HTTP
authentication credentials in this file when trying to download from
a remote host through HTTP or HTTPS. Defaults to
`$NIX_CONF_DIR/netrc`.
The `netrc` file consists of a list of accounts in the following
format:
machine my-machine
login my-username
password my-password
For the exact syntax, see [the `curl`
documentation](https://ec.haxx.se/usingcurl-netrc.html).
> **Note**
>
> This must be an absolute path, and `~` is not resolved. For
> example, `~/.netrc` won't resolve to your home directory's
> `.netrc`.
)"};
Setting<Path> caFile{
this,
getDefaultSSLCertFile(),
"ssl-cert-file",
R"(
The path of a file containing CA certificates used to
authenticate `https://` downloads. Nix by default uses
the first of the following files that exists:
1. `/etc/ssl/certs/ca-certificates.crt`
2. `/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt`
The path can be overridden by the following environment
variables, in order of precedence:
1. `NIX_SSL_CERT_FILE`
2. `SSL_CERT_FILE`
)",
{},
// Don't document the machine-specific default value
false};
Setting<Strings> hashedMirrors{
this,
{},
"hashed-mirrors",
R"(
A list of web servers used by `builtins.fetchurl` to obtain files by
hash. Given a hash algorithm *ha* and a base-16 hash *h*, Nix tries to
download the file from *hashed-mirror*/*ha*/*h*. This allows files to
be downloaded even if they have disappeared from their original URI.
For example, given an example mirror `http://tarballs.nixos.org/`,
when building the derivation
```nix
builtins.fetchurl {
url = "https://example.org/foo-1.2.3.tar.xz";
sha256 = "2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae";
}
```
Nix will attempt to download this file from
`http://tarballs.nixos.org/sha256/2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae`
first. If it is not available there, it tries the original URI.
)"};
Setting<bool> useXDGBaseDirectories{
this,
false,
@@ -860,14 +409,13 @@ public:
)"};
/**
* Finds the first external derivation builder that supports this
* derivation, or else returns a null pointer.
* Get the options needed for profile directory functions.
*/
const ExternalBuilder * findExternalDerivationBuilderIfSupported(const Derivation & drv);
ProfileDirsOptions getProfileDirsOptions() const;
};
// FIXME: don't use a global variable.
extern Settings settings;
extern nix::Settings settings;
/**
* Load the configuration (from `nix.conf`, `NIX_CONFIG`, etc.) into the
@@ -877,9 +425,6 @@ extern Settings settings;
*/
void loadConfFile(AbstractConfig & config);
// Used by the Settings constructor
std::vector<Path> getUserConfigFiles();
/**
* The version of Nix itself.
*
@@ -19,15 +19,17 @@ struct HttpBinaryCacheStoreConfig : std::enable_shared_from_this<HttpBinaryCache
HttpBinaryCacheStoreConfig(
std::string_view scheme, std::string_view cacheUri, const Store::Config::Params & params);
HttpBinaryCacheStoreConfig(ParsedURL cacheUri, const Store::Config::Params & params);
ParsedURL cacheUri;
const Setting<std::optional<CompressionAlgo>> narinfoCompression{
Setting<std::optional<CompressionAlgo>> narinfoCompression{
this, std::nullopt, "narinfo-compression", "Compression method for `.narinfo` files."};
const Setting<std::optional<CompressionAlgo>> lsCompression{
Setting<std::optional<CompressionAlgo>> lsCompression{
this, std::nullopt, "ls-compression", "Compression method for `.ls` files."};
const Setting<std::optional<CompressionAlgo>> logCompression{
Setting<std::optional<CompressionAlgo>> logCompression{
this,
std::nullopt,
"log-compression",
@@ -37,10 +39,10 @@ struct HttpBinaryCacheStoreConfig : std::enable_shared_from_this<HttpBinaryCache
(e.g. `brotli`).
)"};
const Setting<std::optional<std::filesystem::path>> tlsCert{
Setting<std::optional<std::filesystem::path>> tlsCert{
this, std::nullopt, "tls-certificate", "Path to an optional TLS client certificate in PEM format."};
const Setting<std::optional<std::filesystem::path>> tlsKey{
Setting<std::optional<std::filesystem::path>> tlsKey{
this, std::nullopt, "tls-private-key", "Path to an optional TLS client certificate private key in PEM format."};
static const std::string name()
@@ -52,6 +54,8 @@ struct HttpBinaryCacheStoreConfig : std::enable_shared_from_this<HttpBinaryCache
static std::string doc();
ref<Store> openStore(ref<FileTransfer> fileTransfer) const;
ref<Store> openStore() const override;
StoreReference getReference() const override;
@@ -67,13 +71,17 @@ class HttpBinaryCacheStore : public virtual BinaryCacheStore
Sync<State> _state;
protected:
ref<FileTransfer> fileTransfer;
public:
using Config = HttpBinaryCacheStoreConfig;
ref<Config> config;
HttpBinaryCacheStore(ref<Config> config);
HttpBinaryCacheStore(ref<Config> config, ref<FileTransfer> fileTransfer = getFileTransfer());
void init() override;
@@ -20,15 +20,15 @@ struct LegacySSHStoreConfig : std::enable_shared_from_this<LegacySSHStoreConfig>
// Hack for getting remote build log output.
// Intentionally not in `LegacySSHStoreConfig` so that it doesn't appear in
// the documentation
const Setting<int> logFD{this, INVALID_DESCRIPTOR, "log-fd", "file descriptor to which SSH's stderr is connected"};
Setting<int> logFD{this, INVALID_DESCRIPTOR, "log-fd", "file descriptor to which SSH's stderr is connected"};
#else
Descriptor logFD = INVALID_DESCRIPTOR;
#endif
const Setting<Strings> remoteProgram{
Setting<Strings> remoteProgram{
this, {"nix-store"}, "remote-program", "Path to the `nix-store` executable on the remote machine."};
const Setting<int> maxConnections{this, 1, "max-connections", "Maximum number of concurrent SSH connections."};
Setting<int> maxConnections{this, 1, "max-connections", "Maximum number of concurrent SSH connections."};
/**
* Hack for hydra
@@ -19,7 +19,7 @@ struct LocalOverlayStoreConfig : virtual LocalStoreConfig
{
}
const Setting<std::string> lowerStoreUri{
Setting<std::string> lowerStoreUri{
(StoreConfig *) this,
"",
"lower-store",
@@ -31,7 +31,7 @@ struct LocalOverlayStoreConfig : virtual LocalStoreConfig
Must be used as OverlayFS lower layer for this store's store dir.
)"};
const PathSetting upperLayer{
PathSetting upperLayer{
(StoreConfig *) this,
"",
"upper-layer",
@@ -53,7 +53,7 @@ struct LocalOverlayStoreConfig : virtual LocalStoreConfig
default, but can be disabled if needed.
)"};
const PathSetting remountHook{
PathSetting remountHook{
(StoreConfig *) this,
"",
"remount-hook",
@@ -185,6 +185,29 @@ struct LocalSettings : public virtual Config, public GCSettings, public AutoAllo
return autoAllocateUids ? this : nullptr;
}
Setting<unsigned int> buildCores{
this,
0,
"cores",
R"(
Sets the value of the `NIX_BUILD_CORES` environment variable in the [invocation of the `builder` executable](@docroot@/store/building.md#builder-execution) of a derivation.
The `builder` executable can use this variable to control its own maximum amount of parallelism.
<!--
FIXME(@fricklerhandwerk): I don't think this should even be mentioned here.
A very generic example using `derivation` and `xargs` may be more appropriate to explain the mechanism.
Using `mkDerivation` as an example requires being aware of that there are multiple independent layers that are completely opaque here.
-->
For instance, in Nixpkgs, if the attribute `enableParallelBuilding` for the `mkDerivation` build helper is set to `true`, it passes the `-j${NIX_BUILD_CORES}` flag to GNU Make.
If set to `0`, nix will detect the number of CPU cores and pass this number via `NIX_BUILD_CORES`.
> **Note**
>
> The number of parallel local Nix build jobs is independently controlled with the [`max-jobs`](#conf-max-jobs) setting.
)",
{"build-cores"}};
Setting<bool> fsyncMetadata{
this,
true,
@@ -416,7 +439,7 @@ struct LocalSettings : public virtual Config, public GCSettings, public AutoAllo
#endif
#if defined(__linux__) || defined(__FreeBSD__)
Setting<Path> sandboxBuildDir{
Setting<std::filesystem::path> sandboxBuildDir{
this,
"/build",
"sandbox-build-dir",
@@ -429,7 +452,7 @@ struct LocalSettings : public virtual Config, public GCSettings, public AutoAllo
)"};
#endif
Setting<std::optional<Path>> buildDir{
Setting<std::optional<std::filesystem::path>> buildDir{
this,
std::nullopt,
"build-dir",
@@ -467,7 +490,7 @@ struct LocalSettings : public virtual Config, public GCSettings, public AutoAllo
private:
OptionalPathSetting diffHook{
Setting<std::optional<std::filesystem::path>> diffHook{
this,
std::nullopt,
"diff-hook",
@@ -505,7 +528,7 @@ public:
* Get the diff hook path if run-diff-hook is enabled.
* @return Pointer to path if enabled, nullptr otherwise.
*/
const Path * getDiffHook() const
const std::filesystem::path * getDiffHook() const
{
if (!runDiffHook.get()) {
return nullptr;
@@ -597,6 +620,30 @@ public:
true, // document default
Xp::ConfigurableImpureEnv};
Setting<Strings> hashedMirrors{
this,
{},
"hashed-mirrors",
R"(
A list of web servers used by `builtins.fetchurl` to obtain files by
hash. Given a hash algorithm *ha* and a base-16 hash *h*, Nix tries to
download the file from *hashed-mirror*/*ha*/*h*. This allows files to
be downloaded even if they have disappeared from their original URI.
For example, given an example mirror `http://tarballs.nixos.org/`,
when building the derivation
```nix
builtins.fetchurl {
url = "https://example.org/foo-1.2.3.tar.xz";
sha256 = "2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae";
}
```
Nix will attempt to download this file from
`http://tarballs.nixos.org/sha256/2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae`
first. If it is not available there, it tries the original URI.
)"};
using ExternalBuilders = std::vector<ExternalBuilder>;
Setting<ExternalBuilders> externalBuilders{
@@ -660,6 +707,12 @@ public:
// Current system: 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}
// Xp::ExternalBuilders
};
/**
* Finds the first external derivation builder that supports this
* derivation, or else returns a null pointer.
*/
const ExternalBuilder * findExternalDerivationBuilderIfSupported(const Derivation & drv);
};
} // namespace nix
Loaded 100 of 209 files, more files were not shown because too many files have changed in this diff. Show more