Compare commits

...
Author SHA1 Message Date
John Ericson 2933eb3684 WIP get rid of global Settings in libstore
This is analogous to 52bfccf8d8 for
`EvalSettings`, and 3fc77f281e for
`FlakeSettings` and `fetchers::Settings`.
2026-02-15 10:50:25 -05:00
John Ericson 7c915b371d Merge pull request #15235 from obsidiansystems/os-environ
libutil-tests: Fix crash on Windows
2026-02-15 14:58:22 +00:00
Artemis TosiniandJohn Ericson 36d0e9580f Implement Pid::kill for Windows
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2026-02-14 20:39:32 -05:00
Artemis TosiniandJohn Ericson c9abefbc30 libutil-tests: Fix crash on Windows
libutil tests were crashing on Windows due to issues finding `environ`.
Replace process creation of `getEnv` with a new `getEnvOs` function that
uses native windows APIs.

Also convert a bunch of `RunOptions` fields to use `OsString` to better
reflect the underlying interfaces.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2026-02-14 20:39:32 -05:00
Sergei Zimmerman 6cbf80a0b0 Merge pull request #15219 from obsidiansystems/writeDerivation-lighter-read-only
Get rid of the settings-dependent `writeDerivation` wrapper
2026-02-14 21:27:52 +00:00
John Ericson 8fadcceb6d Merge pull request #15233 from obsidiansystems/remove-nixstore-global
libstore: remove `Settings::nixStore` in favor of `StoreConfigBase::getDefaultNixStoreDir`
2026-02-13 20:29:11 +00:00
John Ericson 2913722781 Merge pull request #15229 from lisanna-dettwyler/fix-gc-dry-run
Emit basic dry run message for garbage collection
2026-02-13 20:19:49 +00:00
Amaan Qureshi 12f97382af libstore: remove Settings::nixStore in favor of StoreConfigBase::getDefaultNixStoreDir
This commit removes the `nixStore` member from `Settings` and instead
computes the default Nix store directory directly in
`StoreConfigBase::getDefaultNixStoreDir()` from env vars
(`NIX_STORE_DIR`, `NIX_STORE`) or the compile-time default. The method
is made public so callers that previously reached through the global
`settings.nixStore` can use it instead.

Progress on #5638
2026-02-13 14:45:49 -05:00
Lisanna Dettwyler fdfc772114 Emit basic dry run message for garbage collection
nix store gc: prints number of paths that would be freed, but not bytes
nix-collect-garbage: ditto
nix-store --gc: retains current behavior

It would be very non-trivial to also compute the bytes that would be
freed, due to hardlinking in the store.

Also adds checking for incompatible mixing of dry-run and max-freed
options.

Resolves #5704

Signed-off-by: Lisanna Dettwyler <lisanna.dettwyler@gmail.com>
2026-02-13 14:40:36 -05:00
John Ericson a4b1814d67 Merge pull request #15232 from obsidiansystems/inline-buildlocally
libstore: inline `willBuildLocally` and `canBuildLocally` into call sites
2026-02-13 19:29:15 +00:00
John Ericson 702ebdb11b Merge pull request #15231 from obsidiansystems/inline-getmachines
libstore: inline `getMachines` into call sites
2026-02-13 19:07:18 +00:00
Amaan QureshiandJohn Ericson 7106de16e6 libstore: inline willBuildLocally and canBuildLocally into call sites
This commit inlines `DerivationOptions::willBuildLocally` and
`DerivationOptions::canBuildLocally` into their sole call site in
`DerivationBuildingGoal::tryToBuild`. The `canBuildLocally` logic is now
a lambda capturing the surrounding context, and `willBuildLocally` is
replaced by `drvOptions.preferLocalBuild && canBuildLocally`.

Progress on #5638

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2026-02-13 13:45:33 -05:00
John Ericson b818594ba2 Merge pull request #15228 from obsidiansystems/profile-dirs-options
libstore: extract `ProfileDirsOptions` from `Settings`
2026-02-13 18:28:56 +00:00
Amaan QureshiandJohn Ericson 9ae12ede4c libstore: inline getMachines into call sites
This commit removes the `getMachines` free function and inlines `Machine::parseConfig({settings.thisSystem}, settings.getWorkerSettings().builders)` at its two call sites in `worker.cc` and `build-remote.cc`. The wrapper just forwarded to `Machine::parseConfig` with global settings, so inlining it removes an unnecessary layer of indirection and makes the global dependency explicit at each call site.

Progress on #5638

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2026-02-13 13:22:49 -05:00
John Ericson 20f7f33123 Merge pull request #15227 from obsidiansystems/narinfo-disk-cache-settings
libstore: extract `NarInfoDiskCacheSettings` from `Settings`
2026-02-13 17:58:33 +00:00
John EricsonandAmaan Qureshi 002cbefa9f libstore: extract ProfileDirsOptions from Settings
This commit moves `nixStateDir` and `useXDGBaseDirectories` into a dedicated `ProfileDirsOptions` struct and threads it through the profile directory functions (`profilesDir`, `rootProfilesDir`, `defaultChannelsDir`, `rootChannelsDir`, `getDefaultProfile`) so they no longer read from the global `Settings` object directly. This follows the same pattern as `LocalSettings`, `WorkerSettings`, and `NarInfoDiskCacheSettings`.

Progress on #5638

Co-authored-by: Amaan Qureshi <git@amaanq.com>
2026-02-13 12:43:53 -05:00
John Ericson dc636dde10 libstore: extract NarInfoDiskCacheSettings from Settings
This commit moves `ttlNegativeNarInfoCache` and `ttlPositiveNarInfoCache` into a dedicated `NarInfoDiskCacheSettings` struct that `Settings` privately inherits from, following the same pattern as `LocalSettings`, `LogFileSettings`, and `WorkerSettings`.

`NarInfoDiskCache` now takes explicit `NarInfoDiskCacheSettings` and `SQLiteSettings` in its constructor instead of reading from the global. The singleton `getNarInfoDiskCache()` is replaced with a `NarInfoDiskCache::get()` static method that accepts these settings, though they are only used on the first call (subsequent calls return the cached instance regardless of arguments).

Progress on #5638
2026-02-13 12:12:34 -05:00
John Ericson a06ab4871c Merge pull request #15217 from amaanq/acquire-user-lock-state-dir
libstore: pass `stateDir` to `acquireUserLock` instead of using global
2026-02-13 01:31:41 +00:00
John Ericson ed22ef2b89 Merge pull request #15218 from obsidiansystems/read-only-per-store
libstore: make substitution use the per-store `getReadOnly` method
2026-02-13 01:31:32 +00:00
John Ericson 7926a629e2 Get rid of the settings-dependent writeDerivation wrapper
It was a crude hack that this one low-level function was dependent on
the high-level read-only mode setting --- all the more so because rather
than making derivation writing fail, that setting made it silently
"succeed" why not actually writing the derivation. (Also, for context,
we didn't have an such behavior for any other store-mutating operations,
just for this one function.)

I have gotten rid of the wrapper, and updated the call sites
accordingly.

- For the ones that should remain dependent on this setting, I made this
  explicit, and added a comment.

- For others, surrounding operations assumed writability (e.g. we had
  written something before, or were about to try to read back the
  written derivation after), and so I just made those do the underlying
  `Store::writeDerivation` operation.
2026-02-12 20:26:24 -05:00
Bernardo Meurer a8f305add3 Merge pull request #15216 from NixOS/fix-s3-conn-reuse
fix: #15208
2026-02-13 00:53:58 +00:00
Amaan Qureshi cecbe9f73a libstore: pass stateDir to acquireUserLock instead of using global
This makes `acquireUserLock` take an explicit stateDir parameter,
since it was previously reaching into the global settings object
just to read `nixStateDir` for constructing the userpool paths.

Progress on #5638
2026-02-12 19:43:40 -05:00
Amaan Qureshi 9ac91e36a9 libstore: make substitution use the per-store getReadOnly method
This commit introduces a `getReadOnly` method on the store config that returns if the current store is read only or not. This is then used in subtitution, so we fail gracefully with a nice error message if only the individual store is read-only.

As a bonus, it gets us one step closer to getting rid of the global because we can use the per-store method instead.

Progress on #5638
2026-02-12 19:43:20 -05:00
Bernardo Meurer Costa 759f6c856b feat(libstore/s3): use virtual-hosted-style URLs and add addressing-style option
S3 binary caches now use virtual-hosted-style URLs by default for
standard AWS endpoints. Path-style endpoints (s3.region.amazonaws.com)
only serve HTTP/1.1, preventing HTTP/2 multiplexing and causing TCP
TIME_WAIT socket exhaustion under high concurrency. Virtual-hosted-style
endpoints (bucket.s3.region.amazonaws.com) support HTTP/2, enabling
multiplexing with the existing CURLPIPE_MULTIPLEX configuration.

Add a new `addressing-style` store option (auto/path/virtual) to control
this behavior. `auto` (default) uses virtual-hosted-style for standard
AWS endpoints and path-style for custom endpoints. `path` forces
path-style for backwards compatibility. `virtual` forces virtual-hosted-
style for all endpoints including custom ones.

Fixes: https://github.com/NixOS/nix/issues/15208
2026-02-13 00:03:50 +00:00
Bernardo Meurer Costa 736abd50ff fix(libstore/filetransfer): enable TCP keep-alive on curl handles
Idle connections in libcurl's connection pool can be silently dropped by
the OS or intermediate firewalls/NATs before they can be reused, forcing
new TCP connections to be created. This is especially problematic for
HTTP/1.1 endpoints where multiplexing is unavailable.

Enable TCP keep-alive with a 60-second idle/interval on all curl easy
handles to prevent idle connection drops and improve connection reuse.
2026-02-12 22:52:48 +00:00
John Ericson a3d51172e9 Merge pull request #15211 from obsidiansystems/worker-settings
libstore: extract `WorkerSettings` from `Settings`
2026-02-12 21:15:43 +00:00
Sergei Zimmerman eae7e0151c Merge pull request #15213 from xokdvium/unhardcode-alignas-cache-line-size
Unhardcode alignas cache line size
2026-02-12 20:53:09 +00:00
Amaan Qureshi d3388d3d81 libstore: extract WorkerSettings from Settings
This commit  moves `pollInterval`, `maxSubstitutionJobs`, `postBuildHook`, and `logLines` into a dedicated `WorkerSettings` struct that `Settings` privately inherits from, as they are only used by the build worker subsystem. This follows the same pattern as `LocalSettings` and `LogFileSettings`.
2026-02-12 15:31:08 -05:00
Sergei Zimmerman 7352205ce9 libexpr: Replace hardcoded cache line size with std::hardware_destructive_interference_size
This expands to __GCC_DESTRUCTIVE_SIZE, which is also 64 (at least in the x86_64 stdenv).
Let the compiler decide what's the appropriate cache line size is. Also, on aarch64-darwin
the cache line size 128 bytes, so the previous fix didn't actually get rid of false sharing
reliably. Clang does this [1] [2], so it overestimates the sizes somewhat, but that's still enough
for avoiding false sharing on darwin.

[1]: https://github.com/llvm/llvm-project/blob/a289341dedf4d73955faefbb0b3c13881e13dd06/clang/lib/Frontend/InitPreprocessor.cpp#L1331-L1339
[2]: https://github.com/llvm/llvm-project/blob/6f51f8e0f93e6ff2baf0ed2f60021308920a557f/clang/lib/Basic/Targets/AArch64.h#L262-L264
2026-02-12 23:04:40 +03:00
Sergei Zimmerman f3f9eac8fc Merge pull request #15209 from obsidiansystems/http-store-port-ctor
libstore: add `HttpBinaryCacheStoreConfig` constructor that takes a ` ParsedURL`
2026-02-12 18:48:42 +00:00
Sergei Zimmerman df21c81191 libexpr: Fix some typos in value.hh 2026-02-12 20:51:38 +03:00
Amaan Qureshi 52b1906995 libstore: add HttpBinaryCacheStoreConfig constructor that takes a ParsedURL
In the https-store tests, a `TestHttpBinaryCacheStoreConfig` is constructed with a call to format to create the cache uri. This commit adds a constructor to `HttpBinaryCacheStoreConfig` to remove the need for this call, and updates the test type to leverage this so we're no longer manually calling fmt on a string to format the port.
2026-02-12 11:22:29 -05:00
John Ericson c756d02948 Merge pull request #15206 from obsidiansystems/injectable-filetransfer
libstore: make `FileTransfer` injectable into `HttpBinaryCacheStore`
2026-02-12 14:58:34 +00:00
Amaan Qureshi 403e30f136 libstore: make FileTransfer injectable into HttpBinaryCacheStore
This commit makes `FileTransfer` self-contained by giving it a reference
to `FileTransferSettings` instead of reading from the global. It also
adds an optional `FileTransfer` parameter to `HttpBinaryCacheStore` so
callers can inject their own instance.

The main motivation is test isolation. The HTTPS store tests now create
custom `FileTransferSettings` with the test CA certificate and pass it
through `makeFileTransfer()`, avoiding global state mutation entirely.
2026-02-11 19:00:53 -05:00
Sergei Zimmerman 3a60a04bf8 Merge pull request #15183 from obsidiansystems/newuidmap
Support build users on unprivileged users with subuid/subgid
2026-02-11 22:35:44 +00:00
Artemis Tosini c9526e289a Add new libexec/nix-nswrapper program
nix-nswrapper allows running nix in its own user namespace,
believing it is root and with access to build users for sandboxing
with auto-allocate-uids, while it is actually unprivileged.

It is used to wrap nix, and an example of its use has been
added to the unprivileged daemon functional tests.

Running it does not require any elevated privileges,
only uids and gids allocated in /etc/sub{uid,gid}
2026-02-11 16:53:08 -05:00
Eelco Dolstra d4a0024184 Merge pull request #15205 from NixOS/bump-file-limit-upstream
Increase the open file soft limit to the hard limit
2026-02-11 21:40:39 +00:00
Sergei Zimmerman d9651b1f82 Merge pull request #15193 from NixOS/restore-death-signal
DerivationBuilder: Preserve death signal across setuid,setgid
2026-02-11 21:26:32 +00:00
John Ericson 912c6c283d Merge pull request #15202 from obsidiansystems/migrate-ca-netrc-downloadspeed-filetransfer
libstore: migrate `caFile`, `netrcFile`, and `downloadSpeed` to `FileTransferSettings`
2026-02-11 21:09:06 +00:00
Eelco Dolstra 04fd722b1b Increase the open file soft limit to the hard limit
On some platforms (macOS), the default soft limit is very low, but the
hard limit is high. So let's just raise it the maximum permitted.
2026-02-11 21:55:57 +01:00
John Ericson 1a57df3473 Merge pull request #15203 from obsidiansystems/substituter-confs
Deduplicate `nix repl` and `nix log`
2026-02-11 20:41:56 +00:00
eveeifyeveandAmaan Qureshi 04d13a96e3 libstore: migrate caFile, netrcFile, and downloadSpeed to FileTransferSettings
The `caFile`, `netrcFile`, and `downloadSpeed` settings are only used by
the file transfer subsystem but lived in the global `Settings` class.
This moves them to `FileTransferSettings` where they belong.

Co-authored-by: Amaan Qureshi <git@amaanq.com>
2026-02-11 14:58:27 -05:00
Amaan Qureshi 46eabe34c2 libstore: move hashedMirrors to LocalSettings
`hashedMirrors` is only relevant to local builds (it is consumed by
`builtin:fetchurl` during derivation building) but lived in the global
`Settings` class. This moves it to `LocalSettings` where it belongs
and threads it through `BuiltinBuilderContext` so `fetchurl.cc` reads
it from the context instead of reaching into `settings` directly.
2026-02-11 14:58:27 -05:00
John Ericson ecdcdd82e0 Deduplicate nix repl and nix log
The underlying mechanism is now in a new `fetchBuildLog` function I put
in `libcmd`.

I am putting it in here and not in libstore because I have some doubts
about `getDefaultSubstituters`, so I would like to keep it in a more
"peripheral" part of the codebase for now.
2026-02-11 14:55:31 -05:00
Jörg Thalheim a4c421da22 Merge pull request #15201 from mkenigs/install-release-notes
beta nix-installer: add release-note
2026-02-11 19:07:49 +00:00
John Ericson ae4e4d9afd Merge pull request #15192 from obsidiansystems/store-reference-types
globals: change store settings to use `StoreReference` types directly
2026-02-11 17:49:07 +00:00
Matthew KenigsbergandCole Helbling fbd837c911 beta nix-installer: add release-note
Add a release note asking for help testing
https://github.com/NixOS/nix-installer

We're hoping to start recommending the Rust-based installer after one
release cycle.

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2026-02-11 10:07:59 -07:00
Amaan Qureshi 857fd2a3a4 globals: change store settings to use StoreReference types directly
The `storeUri`, `substituters`, and `trustedSubstituters` settings now
store typed `StoreReference` values directly instead of raw strings,
so callers work with the real types without manual parsing.

This is a reworked version of #10761.
2026-02-11 12:05:50 -05:00
John Ericson 3c1ad7d978 Merge pull request #15197 from obsidiansystems/ssh-store-config-direct
nix-copy-closure: create `LegacySSHStoreConfig` directly
2026-02-11 16:15:29 +00:00
Amaan Qureshi 8020a847ab nix-copy-closure: create LegacySSHStoreConfig directly
Instead of constructing a `StoreReference` and letting `openStore()` resolve it,
this commit creates the store config directly and calls `openStore()` on it. This
avoids the indirection through the store registry.
2026-02-11 10:31:46 -05:00
John Ericson db8499e62f Merge pull request #15200 from obsidiansystems/remove-const-settings
store-config: remove unnecessary `const` from `Setting<>` fields
2026-02-11 15:29:59 +00:00
Amaan Qureshi 1add77677f store-config: remove unnecessary const from Setting<> fields
Stores hold their config as `ref<const Config>` or `const Config &`,
so `Setting<>` fields are already immutable after store construction.
The field-level `const` is redundant and prevents pre-construction
mutation which is sometimes useful. This commit updates these settings by dropping the `const` qualifier, as it's not needed.
2026-02-11 09:20:41 -05:00
John Ericson d5eda907ef Merge pull request #15195 from obsidiansystems/store-reference-args
globals: use `StoreReference` types in CLI argument handlers
2026-02-11 00:22:08 +00:00
Amaan Qureshi f9300514cd globals: use StoreReference types in CLI argument handlers
The CLI flags `--from`, `--to`, `--eval-store`, and substituter URIs now
parse to `StoreReference` at the argument boundary. `fetchClosure` uses
`StoreReference::parse` instead of `parseURL`. This also adds
`operator<=>` to `StoreReference`.
2026-02-10 18:37:34 -05:00
John Ericson 036a47be83 Merge pull request #15194 from obsidiansystems/nix-daemon-store-config
Make `nix daemon` a `StoreConfigCommand`
2026-02-10 23:13:15 +00:00
John Ericson c4e408459a Make nix daemon a StoreConfigCommand
This commit makes `nix daemon` inherit from `StoreConfigCommand`
instead of `Command`, so that it receives a `StoreConfig` to open
and serve stores with. This cleans up a few things (removes
`openUncachedStore` helper, passes `storeConfig` through
`daemonLoop`/`runDaemon` instead of opening stores ad-hoc) and will
allow further cleanups.
2026-02-10 17:34:13 -05:00
Sergei Zimmerman f0498b94d8 Merge pull request #14768 from pkpbynum/capi/copy-path
C API: Add copy_path to Store API
2026-02-10 22:04:22 +00:00
Sergei Zimmerman 34688ecf5f DerivationBuilder: Preserve death signal across setuid,setgid
It's apparently a common footgun in Linux that the death signal isn't
preserved across calls to setuid/setgid. If nix-build gets SIGKILL-ed
while a build is running that would lead to a runaway build process that
would get reparented to init/systemd.

This is pretty easy to reproduce with the following derivation:

derivation {
  name = "pdeathsig-repro";
  system = builtins.currentSystem;
  builder = "/bin/sh";
  args = [
    "-c"
    ''
      while :; do :; done
    ''
  ];
}

And the reproduction script:

sudo nix-build repro.nix &
sleep 3
BUILDER=$(pgrep -u nixbld1)
sudo kill -9 $(pgrep -f 'nix-build.*repro')
sleep 1
ps -p $BUILDER -o pid,ppid,user,comm

To address this we have to restore the death signal after all the calls
to setuid/setgid. This is done in a helper function preserveDeathSignal
that takes a callback to avoid code duplication.

See: https://github.com/golang/go/issues/9686
2026-02-11 00:56:34 +03:00
John Ericson 92d0fe000b Merge pull request #15188 from obsidiansystems/handleexceptions
Move `nix::handleExceptions` to libutil
2026-02-10 18:55:14 +00:00
John Ericson 75af0351ac Merge pull request #15089 from NixOS/sfp-global-settings
`std::filesystem::path` in some `Settings` fields
2026-02-10 18:43:55 +00:00
Artemis Tosini c79ff97c07 Move nix::handleExceptions to libutil
This is a fairly simple function, isolated from the rest of libmain
and could be useful if new programs are made that are not part of the
main nix-cli subproject.
2026-02-10 13:06:27 -05:00
John Ericson ef659136ca std::filesystem::path in some Settings fields 2026-02-10 12:50:17 -05:00
John Ericson 582e4fa0f6 Merge pull request #15187 from obsidiansystems/serve-unix-socket
Factor out `serveUnixSocket`
2026-02-10 17:03:36 +00:00
Amaan Qureshi 6674c23416 Factor out serveUnixSocket
This commit extracts the Unix domain socket server loop (`PeerInfo`,
`getPeerInfo`, and the systemd socket activation / poll / accept loop)
from `src/nix/unix/daemon.cc` into a reusable `unix::serveUnixSocket`
function in `libcmd`.
2026-02-10 11:25:57 -05:00
Eelco Dolstra b06d0f764f Merge pull request #15175 from KiaraGrouwstra/flake-ref-nixpkgs
check `isFlake` in `nixpkgsFlakeRef`
2026-02-10 14:03:37 +00:00
John Ericson 845d951682 Merge pull request #15180 from xokdvium/more-werror
meson: Add -Werror=return-type and -Werror=non-virtual-dtor flags
2026-02-09 21:48:15 +00:00
Sergei Zimmerman a900bf1548 meson: Add -Werror=return-type and -Werror=non-virtual-dtor flags
Some easy compile-time safety features to catch mistakes earlier.
Fixes some missing virtual destructors.
2026-02-10 00:02:00 +03:00
Sergei Zimmerman 36ad2962ca Merge pull request #14944 from iljah/patch-2
Use unreachable in nix::listNarImpl()
2026-02-09 20:12:33 +00:00
Sergei Zimmerman e4ce788f9d Merge pull request #15172 from NixOS/misc-fixes
libutil: Assorted collection of fixes, address UBSan failure in AutoDelete
2026-02-08 22:56:42 +00:00
Sergei Zimmerman 3cd840d7f1 libutil: Fix error message in readLinkAt
More correctly describes the error, since we are always reading a relative path.
2026-02-09 00:57:13 +03:00
Sergei Zimmerman 6b90755cad libutil: Fix uninitialised variable in AutoDelete
Diagnosed by UBSan in hydra [1]:

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /nix/store/m1k4nxs8r0fl0pjxqp5n37vxgms7gdlb-gcc-14.3.0/include/c++/14.3.0/bits/move.h:234:11 in
+(prefetch.sh:6) path=
++(prefetch.sh:6) onError
++(/build/source/tests/functional/common/functions.sh:241) set +x
prefetch.sh: test failed at:
  main in prefetch.sh:6

[1]: https://hydra.nixos.org/build/321098757/nixlog/1
2026-02-09 00:57:07 +03:00
cinereal 30e213c948 check isFlake in nixpkgsFlakeRef
To get `bashInteractive`, `nix develop` currently
[gets a flake reference to nixpkgs](https://github.com/NixOS/nix/blob/3b473c4be5e6abc58aa43271f35ff127af806c55/src/nix/develop.cc#L650-L658),
either from [`inputs.nixpkgs`](https://github.com/NixOS/nix/blob/3b473c4be5e6abc58aa43271f35ff127af806c55/src/libcmd/installable-flake.cc#L204)
or [`<nixpkgs>`](https://github.com/NixOS/nix/blob/3b473c4be5e6abc58aa43271f35ff127af806c55/src/libcmd/include/nix/cmd/installable-flake.hh#L87).

Currently, this is done by name for any (locked) reference.
For any `nixpkgs` reference lacking a `flake.nix`, `nix develop` thus errors:

```
error (ignored): path '.../flake.nix' does not exist
```

While the registry does not expose info to help check this,
flake inputs expose the `flake` boolean.
This means that given `inputs.<name>.flake = false;`,
we may know in advance that our reference should not be to a flake.

This change incorporates this, so that given `inputs.nixpkgs.flake = false;`,
`nix develop` will fall back `<nixpkgs>` to use its flake for `bashInteractive`.

While Nixpkgs itself of course does expose a flake,
this check is relevant in particular for dummy inputs
(e.g. inputs using `{ url = "file:/dev/null"; flake = false; }`),
which may be overridden by `--override-input` or `.follows`,
yet do not expose (flake) code themselves.

Signed-off-by: cinereal <cinereal@riseup.net>
2026-02-08 13:32:20 +01:00
Sergei Zimmerman 3b473c4be5 Merge pull request #14952 from koberbe/koberbe/master/SW-100752-nix-commands-help-is-missing-unformatted
Install nix-manual in default user profile
2026-02-07 15:37:54 +00:00
Ilja 23ddb0bfc7 Throw on unsupported type in nix::listNarImpl()
Prevent
```
../src/libutil/nar-accessor.cc: In function ”nix::ListNarResult<deep> nix::listNarImpl(SourceAccessor&, const CanonPath&) [with bool deep = true]”:
../src/libutil/nar-accessor.cc:335:1: varoitus: ei-void-tyyppisen funktion loppu saavutettu [-Wreturn-type]
  335 | }
      | ^
../src/libutil/nar-accessor.cc: In function ”nix::ListNarResult<deep> nix::listNarImpl(SourceAccessor&, const CanonPath&) [with bool deep = false]”:
../src/libutil/nar-accessor.cc:335:1: varoitus: ei-void-tyyppisen funktion loppu saavutettu [-Wreturn-type]
```
2026-02-07 17:31:45 +03:00
John Ericson e29bb23cf9 Merge pull request #15170 from obsidiansystems/should-resolve
Add `Derivation::shouldResolve()` method, use in `nix-shell`
2026-02-07 00:09:31 +00:00
John Ericson 08da3311b3 Add Derivation::shouldResolve() method, use in nix-shell
Extract the logic for determining whether a derivation should be resolved
before building into a dedicated method. Then use that to not resolve
unnecessarily in `nix-shell`.
2026-02-06 18:20:33 -05:00
John Ericson ba3dc07bf1 Merge pull request #15168 from roberth/fix-protocol-addToStore-version-comparisons
Add -Werror=c99-designator and fix brace elision warnings
2026-02-06 22:11:36 +00:00
Robert Hensing e5278ac66b Add -Werror=c99-designator and fix brace elision warnings
The recent conversion of WorkerProto::Version from unsigned int to a
struct exposed a latent issue: `.version = 16` was being interpreted
as aggregate initialization `{.major = 16, .minor = 0}` rather than
the intended wire format value.

This commit adds -Werror=c99-designator to catch this class of bugs at
compile time. (The bug itself was fixed in
7eb23c15f39ca413a5f3cc0d3ab630311b4709be.)

For background:

The hardcoded version was originally the integer 16, which in the old
wire format (major << 8 | minor) meant version 0.16. However, the
version checks only compared minor versions via GET_PROTOCOL_MINOR(),
so this worked by accident.

After the Version struct conversion, the aggregate initialization
{.major = 16, .minor = 0} happened to still work because 16 > 1 in
lexicographic comparison against {1, 16}.

The correct version is {1, 16} because:
- The worker protocol uses major version 1 (all checks are {1, x})
- Version 1.16 is when ultimate/sigs/ca fields were added
- This matches the serialization check `>= {1, 16}`
2026-02-06 16:23:16 -05:00
Sergei Zimmerman 2f49b730cf Merge pull request #15167 from NixOS/repl-last-loaded-fix
libcmd/repl: Fix issues with :ll before anything is loaded, get rid o…
2026-02-06 20:17:40 +00:00
John Ericson ca8e6cae91 Merge pull request #15161 from obsidiansystems/version-number-subtype
worker-protocol: embed features in `Version` and add `Number` inner type
2026-02-06 19:56:37 +00:00
Sergei Zimmerman bcc63908ba libcmd/repl: Fix issues with :ll before anything is loaded, get rid of store parameters to constructors
Fixes abort on :ll if nothing has been loaded yet. Also gets rid of
redundant openStore() calls that were dead code (store can be extracted
from EvalState already) and arguably openStore is a layer violation.

Also catches EPIPE in case the pager gets interrupted to avoid superfluous
error messages.
2026-02-06 22:29:20 +03:00
John EricsonandAmaan Qureshi 7eb23c15f3 worker-protocol: embed features in Version and add Number inner type
This commit embeds the negotiated `FeatureSet` directly into `WorkerProto::Version` and introduces a `Number` inner type with total ordering, so that `Version` itself (number + features) only has partial ordering. This is a follow-up to #15155, cleaning up the separate `features` fields on `ReadConn`/`WriteConn`.

Co-authored-by: Amaan Qureshi <git@amaanq.com>
2026-02-06 14:14:13 -05:00
John Ericson 103f912c40 Merge pull request #15163 from xokdvium/faster-nar-listing
NarIndexer: Implement skip
2026-02-06 18:48:47 +00:00
Eelco Dolstra 34cbfffa11 Merge pull request #15160 from NixOS/fix-flakeRefToString
builtins.flakeRefToString: Evaluate attributes
2026-02-06 18:21:20 +00:00
Sergei Zimmerman 499ffaf940 NarIndexer: Implement skip
This improves the performance of parseNarListing, which is used by commands
like `nix nar ls` when the underlying source allows cheap seeks (like StringSource
or FdSource that does lseek).

For `nix nar ls` of a NAR for linux source tarball this cuts down the runtime almost
in half (from 300ms -> 175ms).
2026-02-06 20:55:04 +03:00
Peter Bynum 72ab64b612 Add nix_store_copy_path C API 2026-02-06 11:59:37 -05:00
Eelco Dolstra 2989a23fca builtins.flakeRefToString: Evaluate attributes
Fixes "attribute 'x' is a thunk".
2026-02-06 16:30:19 +01:00
Eelco Dolstra bbb4b009ec Merge pull request #15157 from NixOS/respect-noexcept
BinaryCacheStore::queryPathInfoUncached(): Ensure noexcept
2026-02-06 08:49:59 +00:00
John Ericson 91c706852b Merge pull request #15155 from obsidiansystems/protocol-version-structs
worker-protocol: replace version bit-shifting with structs
2026-02-05 18:09:53 +00:00
John Ericson 80b944a3f6 Merge pull request #15156 from obsidiansystems/findroots-refactor
libstore: move logic of `findRuntimeRoots` to new file
2026-02-05 17:04:47 +00:00
John EricsonandAmaan Qureshi cccc9440d7 worker-protocol: replace version bit-shifting with structs
This commit replaces the `GET_PROTOCOL_MINOR(version)` macros with a proper `WorkerProto::Version` struct. As a bonus, this also fixes some version checks that were incorrectly ignoring the major version number.

Co-authored-by: Amaan Qureshi <git@amaanq.com>
2026-02-05 11:50:07 -05:00
Eelco Dolstra c21820db07 BinaryCacheStore::queryPathInfoUncached(): Ensure noexcept
Make sure we don't throw an exception, since that will terminate the
program.
2026-02-05 17:35:15 +01:00
John Ericson 4496a7eead Merge pull request #15101 from obsidiansystems/split-local-store-settings
libstore: split out local build and store related settings under `LocalSettings`
2026-02-05 16:23:59 +00:00
John Ericson d1ad4b183a Merge pull request #14769 from NixOS/nar-hash-cache
Replace fetchToStore cache by sourcePathToHash cache
2026-02-05 16:23:52 +00:00
Artemis Tosini 766316223c libstore: move logic of findRuntimeRoots to new file
This change is required for implementing the unprivileged garbage collection daemon,
but it may also be useful to reduce code duplication and separate out OS-specific
garbage collector roots implementations in the future.
2026-02-05 11:18:23 -05:00
Eelco Dolstra de6b5f60cd Replace fetchToStore cache by sourcePathToHash cache
Caching NAR hashes instead of store paths makes the cache more
general, because we can always compute the store path from the NAR
hash, but not the other way around. This is useful for lazy trees,
where we want to compute the NAR hash of an input with caching.
2026-02-05 16:45:50 +01:00
Amaan Qureshi afd40adc90 libstore: split out local build and store related settings under LocalSettings
The global `Settings` struct contained many settings that only apply to
local builds or the local store (sandbox configuration, GC settings,
build user groups, etc.). This commit extracts these into a dedicated
`LocalSettings` struct in its own header, along with `GCSettings` and
`AutoAllocateUidSettings`.

This improves code organization and prepares for eventually making these
per-store settings in the future. Settings are accessed via
`getLocalSettings()` from the global settings object or through
`LocalStoreConfig::getLocalSettings()` for store-specific access.
2026-02-05 10:38:08 -05:00
Jörg Thalheim dcc71da7e8 Merge pull request #15148 from Mic92/fix-tests
tests: fix URL literals in functional tests
2026-02-05 01:04:32 +00:00
Sergei Zimmerman ecda8c2329 Merge pull request #15149 from NixOS/align-up-robustness
libutil: Add overflow check to alignUp
2026-02-05 00:51:20 +00:00
Jörg Thalheim 0da728b1f5 tests: fix URL literals in functional tests 2026-02-05 00:52:02 +01:00
Sergei Zimmerman ea53914e47 Merge pull request #15147 from NixOS/fix-mingw-read-line
libutil: Fix mingw build
2026-02-04 23:14:52 +00:00
Sergei Zimmerman d77c131df3 libutil: Add overflow check to alignUp
Old code with size + (size % 8 ? 8 - (size % 8) : 0) also suffered from this.
2026-02-05 02:04:33 +03:00
Sergei Zimmerman 72c2954625 libutil: Fix mingw build
This was broken by b038500b47.
2026-02-05 00:53:23 +03:00
Eelco Dolstra 27d5cc39c8 Merge pull request #14957 from NixOS/invalidate-lstat-cache
SourceAccessor: Allow the lstat cache to be invalidated
2026-02-04 21:44:04 +00:00
John Ericson 25ab7f5850 Merge pull request #15038 from obsidiansystems/drainfd-improvements
More IO portability cleanups
2026-02-04 21:36:18 +00:00
Eelco Dolstra 139d05af6f SourceAccessor: Allow cached information to be invalidated
After lockFlake() creates flake.lock in a PosixSourceAccessor, it
needs to be able to invalidate the lstat cache.
2026-02-04 21:56:17 +01:00
John Ericson b489c8ea15 More IO portability cleanups
- options structs for `drainFD` (both versions)
- portable `read` wrapper
- portable `GetFileSize` wrapper
- dedup `readFile` and `drainFD` using the above
- Use `drainFD` in `PosixSourceAccessor`, avoiding manual IO
- Remove `fromDescriptorReadOnly` entirely!
2026-02-04 15:49:21 -05:00
John Ericson a4c0295822 Merge pull request #15060 from NixOS/read-link-at
Support `readLinkAt` and `openFileEnsureBeneathNoSymlinks` on Windows too
2026-02-04 20:40:08 +00:00
Eelco Dolstra 8336e71c19 Merge pull request #15135 from NixOS/dependabot/github_actions/docker/login-action-3.7.0
build(deps): bump docker/login-action from 3.6.0 to 3.7.0
2026-02-04 20:34:18 +00:00
John Ericson 037a19441a Merge pull request #15145 from obsidiansystems/storeconfigcommand
libcmd: add new `StoreConfigCommand` class
2026-02-04 20:09:48 +00:00
Artemis Tosini 124605dffc libcmd: add new StoreConfigCommand class
Useful for commands that need a `StoreConfig` but do not want to open
the store, as a `StoreCommand` would do.

At the same time, make copy commands more clear by making store choice
for `updateProfile` explicit and removing the unused `getDstStore` function
in `StoreCommand`. This was a layer violation, as `StoreCommand` does
not have a concept of a source/destination store distinction.
2026-02-04 14:31:51 -05:00
John Ericson 936f6c6c7d Merge pull request #15144 from obsidiansystems/readline-terminator
libutil: add terminator option to readLine
2026-02-04 17:48:28 +00:00
Artemis Tosini b038500b47 libutil: add terminator option to readLine
Some APIs use "lines" that end in `\0` instead of `\n`.
2026-02-04 11:58:34 -05:00
John Ericson a357d77492 Merge pull request #15141 from NixOS/open-new-file-for-write-helper
libutil: Add openNewFileForWrite helper function, wrap callsites
2026-02-04 03:23:40 +00:00
Sergei Zimmerman 47f261cc19 libutil: Add openNewFileForWrite helper function, wrap callsites
This is purely a fix to use CreateFileW in mingw builds. Also adds some
FIXMEs for suspicious symlink following on truncation that can probably
be tightened down without any problems (other than nix-channel), but for
now this is a no-op change other than consistently using O_CLOEXEC, which
is harmless.
2026-02-04 02:10:12 +03:00
Sergei Zimmerman 27435e0036 Merge pull request #15134 from pkpbynum/pb/fix-query-path-info-daemon
Fix: `QueryPathInfo` throws on invalid path error in daemon
2026-02-03 21:34:55 +00:00
John Ericson 0e2fc2a2f1 Merge pull request #15119 from obsidiansystems/canonicalize-pmd-options
libstore: introduce `CanonicalizePathMetadataOptions` for `canonicalisePathMetaData`
2026-02-03 20:00:27 +00:00
John Ericson 6d6cbf78cc Merge pull request #15137 from NixOS/pragma-once
input-cache.hh: Add missing `#pragma once`
2026-02-03 12:17:33 +00:00
Eelco Dolstra 39a9a004e2 input-cache.hh: Add missing #pragma once 2026-02-03 11:51:04 +01:00
dependabot[bot] 5f9483519a build(deps): bump docker/login-action from 3.6.0 to 3.7.0
Bumps [docker/login-action](https://github.com/docker/login-action) from 3.6.0 to 3.7.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/5e57cd118135c172c3672efd75eb46360885c0ef...c94ce9fb468520275223c153574b00df6fe4bcc9)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 3.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-02-03 03:28:21 +00:00
Peter Bynum b9c77ecafc Fix QueryPathInfo in daemon 2026-02-02 20:52:43 -05:00
Eelco Dolstra d5ce1a79ec Merge pull request #15107 from bouk/fsync-key-generation
nix-store: fsync generated key files
2026-02-02 16:12:38 +00:00
John Ericson 663d27c9df Merge pull request #7892 from obsidiansystems/systemd-multi-socket
Support systemd socket activation with multiple sockets
2026-02-02 16:09:32 +00:00
John Ericson 4da0b36f83 Merge pull request #15127 from NixOS/s3-binary-cache-store-md5
libstore/s3-binary-cache-store: Add Content-MD5 header as message int…
2026-02-01 02:58:28 +00:00
John Ericson a2de09c9fa Support systemd socket activation with multiple sockets
We now support `LISTEN_FDS` values greater than 1, per the systemd
socket activation spec.

These changes are by @edolstra, taken from #5265. This is just that PR
*without* the TCP parts, which I gathered are the controversial parts.
Hopefully this remainder is not so controversial.

Review with indentation ignored, because some code has moved inside a
new loop but otherwise is mostly unchanged.
2026-01-31 21:52:37 -05:00
Sergei Zimmerman 2ba65f1f26 libstore/s3-binary-cache-store: Add Content-MD5 header as message integrity check
aws-sdk-cpp used to include a checksum for uploads (CRC64 since ~September 2025).
Content-MD5 [1] should be universally supported by all s3 compatible services, since the SDK used
to include it unconditionally too.

[1]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
2026-02-01 00:26:19 +03:00
John Ericson 77b6b01b72 Merge pull request #15118 from obsidiansystems/exit-status-flags
libstore: introduce `ExitStatusFlags` for exit status computation
2026-01-30 18:37:47 +00:00
Amaan Qureshi 5e7195e1a4 libstore: introduce ExitStatusFlags for exit status computation
This commit consolidates the four separate boolean flags
(`permanentFailure`, `timedOut`, `hashMismatch`, & `checkMismatch`) into
a single `ExitStatusFlags` struct with methods for computing exit status
codes and updating from failure status.
2026-01-30 12:52:51 -05:00
Amaan QureshiandJohn Ericson 78e8896d22 Move HashMismatch wire protocol back compat logic to better spot
The explicit serializer added in
bfdd124837 is the right place to adjust
values for sake of wire protocol compat. The protocol-agnostic `Worker`
code where it was before is the wrong spot.

(That spot was originally chosen because the back compat logic predates
having an explicit serializer for this data type to use instead.)

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2026-01-30 12:52:51 -05:00
John Ericson d84624d23d Merge pull request #15123 from obsidiansystems/gc-test-vm
Support gc-runtime functional tests in VMs
2026-01-30 17:42:19 +00:00
Artemis Tosini 9e8cf9055a tests/gc-functional: fix running on NixOS
This test insisted on placing profiles in NIX_STATE_DIR, but all
packages were removed from the profile immediately after so they did not
act as garbage collector roots. Switch to directly calling nix-build,
allowing the test to run in VMs without NIX_STATE_DIR.
2026-01-30 11:52:34 -05:00
John Ericson 22372d7889 Merge pull request #15122 from obsidiansystems/gc-test
libstore: fix runtime gc on non-standard store paths
2026-01-30 16:48:20 +00:00
Amaan Qureshi d09f03d742 libstore: introduce CanonicalizePathMetadataOptions for canonicalisePathMetaData
This commit refactors the `canonicalisePathMetaData` function to take an
options struct instead of individual parameters with platform-specific
`#ifdef`s.

The struct contains a `uidRange` field (Unix only) for build user
ownership validation, and an `ignoredAcls` field for ACLs to skip when
removing extended attributes
2026-01-30 11:06:01 -05:00
Artemis Tosini b026649c62 libstore: fix runtime gc roots on non-standard store paths
Due to a typo in quoteRegexChars, finding runtime garbage collection roots
was failing on paths that contained a dot, or any other regex chars that would
have to be replaced.

When fixing that error, also add tests to make sure gc continues to
work.
2026-01-30 11:05:10 -05:00
John Ericson d69ca7bf35 Remove obsolete CPP for Windows in nix-store
`LocalStore` and `canonicalisePathMetaData` are defined on Windows by
now, so we don't have to gate their usage like this.
2026-01-29 19:08:46 -05:00
John Ericson ee6cb7890a Merge pull request #15117 from obsidiansystems/move-impersonate-linux
libstore: add `PersonalityArgs` struct for `setPersonality`
2026-01-29 16:34:12 +00:00
Amaan Qureshi 351b8dd768 libstore: add PersonalityArgs struct for setPersonality
This introduces a `PersonalityArgs` struct to pass named arguments to `setPersonality`. The `impersonateLinux26` setting is now passed from the call site rather than read from settings inside the function.
2026-01-29 10:50:07 -05:00
John Ericson 1713f4c976 Merge pull request #15070 from amaanq/build-result-error-cleanup
build-result: Make `Failure` an alias for `BuildError`
2026-01-29 01:53:14 +00:00
Amaan Qureshi de88141cdf build-result: Make Failure an alias for BuildError and remove exception parameters from goal 2026-01-28 19:32:41 -05:00
John Ericson da9426b8fc Merge pull request #15095 from NixOS/small-fixes
Some small fixes
2026-01-28 21:27:06 +00:00
John Ericson bcbc8ae4e3 Merge pull request #15110 from obsidiansystems/pid-cleanup
libutil: add useful functions to Pid
2026-01-28 20:34:00 +00:00
Artemis Tosini 538e82aa0b libutil: add useful functions to Pid
The C++ rule of five suggests that when a custom destructor is needed
then several other functions are as well. The lack of those makes
certain operations challenging
2026-01-28 13:47:09 -05:00
John Ericson 711e6b3476 Merge pull request #15111 from obsidiansystems/fstat-windows
Refactor fstat into a wrapper in libutil
2026-01-28 18:45:15 +00:00
Artemis Tosini 0fc20e3e20 Refactor fstat into a wrapper in libutil
We use a different fstat on posix and windows systems,
and not all fstat users were using the correct one.
Factor out fstat to make the change easier.

See also a13de50df3 for other stat
functions
2026-01-28 12:53:54 -05:00
Bouke van der Bijl 97f71909d7 nix-store: fsync generated key files
Fixes #15106
2026-01-28 11:59:46 +01:00
John Ericson d5e4b0b4b8 Merge pull request #15104 from obsidiansystems/refactor-buildlocally
libstore: decide how to build in one spot
2026-01-28 00:19:22 +00:00
Amaan Qureshi 00d0e6dff3 libstore: decide how to build in one spot
This cleans up the logic for checking if the worker's store is a valid
local store when we're not hooking it. If we have a local store, we then
pass that as an argument to `DerivationBuildingGoal::buildLocally`,
rather than checking inside the function itself.
2026-01-27 18:35:05 -05:00
John Ericson f326f02813 Merge pull request #15099 from obsidiansystems/split-command-specific-settings
libstore: move command-specific settings to their own files
2026-01-27 19:01:32 +00:00
John Ericson 9e9b6d44f8 Merge pull request #15094 from amaanq/git-signing-isolation
tests/functional: isolate git tests from host signing config
2026-01-27 18:11:25 +00:00
Amaan Qureshi 11f6f07598 libstore: move command-specific settings to their own files
The two settings `envKeepDerivations` and `upgradeNixStorePathUrl` were
only used in one command each, so it makes more sense to move them to
their own files. This commit moves them both into a small self-contained
settings struct and registers them with the global config.
2026-01-27 12:36:31 -05:00
Amaan Qureshi ac9682c52f tests/functional: isolate git tests from host signing config
Currently, tests fail when the host system has `commit.gpgsign` or
`tag.gpgsign` enabled at the system level (in my case, a custom path
located at `/etc/git/config`), since the signing key is unavailable in
the test sandbox.

The tests set `HOME=$TEST_HOME` to isolate themselves, which bypasses
the user-level git config (`~/.gitconfig`). However, if a user sets the
system-level config via `GIT_CONFIG_GLOBAL` or `GIT_CONFIG_SYSTEM`, it
still applies, causing commits to fail when signing is enabled there.

In this PR, I explicitly set `GIT_CONFIG_GLOBAL` and `GIT_CONFIG_SYSTEM`
to `/dev/null` so that the user's system config is never read from or
written to. I've also replaced `git config --global protocol.file.allow
always` with `GIT_CONFIG_*` environment variables to avoid writing to
`/dev/null`.
2026-01-27 12:32:41 -05:00
John Ericson c7f1036bcb Merge pull request #15098 from amaanq/fix-nix-shell-test
tests/functional: fix nix-shell fixed-output derivation test
2026-01-27 17:26:11 +00:00
Amaan Qureshi d1348a2477 tests/functional: fix nix-shell fixed-output derivation test
The test was checking for `$stdenv` but the `fixed` derivation doesn't
actually have stdenv, it just has `FOO`. I've updated it to check the
value of `FOO` instead.
2026-01-27 10:35:06 -05:00
John Ericson c0ab135860 Some small fixes
I think this has to do with the 25.11 bump.
2026-01-27 00:06:59 -05:00
John Ericson e5536c8935 Merge pull request #15091 from obsidiansystems/split-diff-hook-settings
globals: refactor `diffHook` settings
2026-01-27 00:24:17 +00:00
John Ericson 929022c8f8 Merge pull request #15092 from NixOS/improve-error-messages
libexpr/parser: Use readable tokens in error messages instead of inte…
2026-01-27 00:01:21 +00:00
John Ericson bad1a005ed Merge pull request #15079 from NixOS/auto-cleanup-cleanups
Clean up `AutoRemoveJail`, `AutoDelete`, and `AutoUnmount`
2026-01-26 23:46:59 +00:00
Amaan Qureshi 692102f0dc globals: refactor diffHook settings
The settings related to diff hook (`run-diff-hook` and `diff-hook`) are
a little redundant and don't need to be leaked in derivation-builder
when computing the diff hook path to execute.

Instead of directly using both `runDiffHook` and `diffHook` settings in
derivation-builder, we can just encapsulate the logic to determine
whether or not we have a diff hook executable to run in a helper
function. We also mark `handleDiffHook` as static.
2026-01-26 18:37:13 -05:00
Sergei Zimmerman 68cf0a7f8a libexpr/parser: Use readable tokens in error messages instead of internal token names
Very low-hanging fruit for improving parser error messages.
2026-01-27 02:11:40 +03:00
John EricsonandSergei Zimmerman 5dfd81cbc0 Clean up AutoRemoveJail, AutoDelete, and AutoUnmount
- Extract destructor logic into named methods (`deletePath()`,
  `unmount()`, and `remove()`) that can be called explicitly. These ones
  will throw exceptions normally, unlike the destructor which must quash
  them to avoid double-exceptions.

- Use `std::filesystem::path` in `AutoUnmount` (changed from `Path`)

- Remove `del` field from `AutoRemoveJail`, using `INVALID_JAIL`
  sentinel value instead.

- Add move assignment operators implemented via friend `swap` functions
  for all three RAII classes.

- Remove old `reset(...)` methods that took parameters. These were a bit
  misleading --- do they cancel or immediately destroy? --- and doing it
  explicitly with cancel and then assignment is not hard.

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2026-01-26 17:26:59 -05:00
John Ericson 395eef30f1 Merge pull request #14688 from NixOS/nixpkgs-25.11
flake: Bump nixpkgs to 25.11
2026-01-26 22:21:25 +00:00
John Ericson c7098ec8da Merge pull request #14329 from Mic92/nix-store-print-env
nix-store --print-env: fix shell quoting on _args output
2026-01-26 20:55:40 +00:00
John Ericson e8d1cb0668 Merge pull request #15086 from JustAGuyTryingHisBest/darwin-pathfmt
libutil: unix filesystem add more PathFmt
2026-01-26 20:49:51 +00:00
John Ericson a7c043b95d Merge pull request #14749 from obsidiansystems/build-status-serializer
Create proper serializer for BuildResult status
2026-01-26 20:37:57 +00:00
Jörg Thalheim ffe97db4f9 nix-store --print-env: fix shell quoting on _args output
The previous implementation double-quoted the _args variable by escaping
each argument individually and then wrapping them all in single quotes,
producing output like: _args=''-e' 'arg1' 'arg2''

This fix concatenates all arguments into a single string first, then
escapes that string once, producing correct output like:
_args='-e arg1 arg2'

This prevents potential command injection issues when the output is
sourced in shell scripts.

Fixes #14327
2026-01-26 15:08:08 -05:00
John Ericson 252e4ee5ca Merge pull request #15077 from NixOS/stat-wrapper
Cleanup stat usage
2026-01-26 19:55:00 +00:00
John Ericson 623360d07f Merge pull request #15083 from lisanna-dettwyler/fix-build-hook-setting
Fix build-hook setting being clobbered
2026-01-26 19:50:38 +00:00
Some Guy 7f95112fac libutil: unix filesystem add more PathFmt 2026-01-26 11:39:34 -08:00
John Ericson d3116dc764 Merge pull request #15043 from obsidiansystems/settings-split-0
Split out `AutoAllocateUidSettings`
2026-01-26 19:25:51 +00:00
John Ericson b190548c83 Merge pull request #15040 from NixOS/factor-out-nar-cache-0
Factor out `NarCache` from `RemoteFSAccessor`
2026-01-26 19:19:39 +00:00
John Ericson a13de50df3 Cleanup stat usage
Use wrappers to make error handling easier.

On Windows we are using proper 64-bit time and size info.

We still have the problem of no `lstat` on Windows, but this will be
dealt with in future PRs.
2026-01-26 14:00:19 -05:00
John EricsonandAmaan Qureshi ab56ac49e3 libstore: split out AutoAllocateUidSettings
Follows the same pattern as `GCSettings`: extract UID allocation
settings
into a dedicated struct that Settings inherits privately from.

The current settings infrastructure prevents correct data modeling that
would allow `autoAllocateUids` to be a
`std::optional<AutoAllocateUidSettings>`.
To compensate, the getter `getAutoAllocateUidSettings()` returns a
pointer -
nullptr when disabled, providing the optional-like semantics we want.

Co-authored-by: Amaan Qureshi <git@amaanq.com>
2026-01-26 13:43:41 -05:00
Lisanna Dettwyler 0e3a620374 Fix build-hook setting being clobbered
settings.buildHook.setDefault was running after nix.conf was parsed,
causing whatever value settings.buildHook had to be clobbered.
Re-arrange the logic so that the default is set before nix.conf is
parsed, so that custom build hooks can be used by specifying them in
nix.conf.

Signed-off-by: Lisanna Dettwyler <lisanna.dettwyler@gmail.com>
2026-01-26 12:13:20 -05:00
Sergei Zimmerman 00f67ee5d5 tests/functional: Require newer daemon version for empty error message bugfix 2026-01-25 23:56:44 +03:00
Sergei Zimmerman d69001600b tests/nixos/functional/unpriviledged-daemon: Use nixStoreMountOpts instead of readOnlyNixStore
This option is not available in 25.11:

> Please use the `boot.nixStoreMountOpts' option to define mount options for the Nix store, including 'ro'
2026-01-25 22:24:34 +03:00
Taeer Bar-Yam c1ab73f921 tests: Update version requirements on tests 2026-01-25 22:19:29 +03:00
Taeer Bar-Yam 3cb27988fb update error message of new daemon 2026-01-25 22:19:22 +03:00
Sergei Zimmerman f43566f4d7 packaging/components: Drop hardeningDisable
This is no longer necessary and produces an eval warning:

> evaluation warning: The 'pie' hardening flag has been removed in favor of enabling PIE by default in compilers and should no longer be used.

This was first introduced in 2200f315da, but
is no longer necessary since the switch to meson.
2026-01-25 22:12:32 +03:00
Taeer Bar-Yam fb6274b312 fix nix-serve with hacky workaround 2026-01-25 22:09:13 +03:00
Taeer Bar-Yam e72a8bebb8 update .gitignore
new version of meson creates some state file
2026-01-25 22:08:40 +03:00
Taeer Bar-Yam dad793fcfd fix perl dependencies error 2026-01-25 22:08:33 +03:00
Taeer Bar-Yam 7985873f73 inputDerivation is fixed upstream
fixed in nixpkgs PR #469652
2026-01-25 22:08:21 +03:00
Taeer Bar-Yam db576d599c fix infinite recursion 2026-01-25 22:07:18 +03:00
Taeer Bar-Yam d5544919e4 tests: minio: mc config host add -> mc alias set
`mc config host add` has been removed
SEE: https://github.com/minio/mc/issues/5206
2026-01-25 22:07:11 +03:00
Taeer Bar-Yam 8928cb4fb8 separateDebugInfo implies __structuredAttrs 2026-01-25 22:07:00 +03:00
Sergei Zimmerman 0dd38bc8b6 packaging/dependencies: Override fixes
- nghttp3 is not supported on mingw
- onetbb doesn't build on mingw
- lowdown override is no longer needed, same for toml11
2026-01-25 22:05:53 +03:00
Sergei Zimmerman d45004f5ec treewide: Apply formatter diffs
Also disable some churny formatters on some specific files.
2026-01-25 22:03:16 +03:00
Sergei Zimmerman 50050b5ef1 flake: Bump nixpkgs to 25.11 2026-01-25 21:59:36 +03:00
Sergei Zimmerman d0c194efc1 maintainers/flake-module: Pin clang-format to 21
We don't want too much unnecessary formatting churn.
2026-01-25 21:57:18 +03:00
Sergei Zimmerman ed9d8af93d Merge pull request #15059 from lovesegfault/fix-aws-logs
feat(libstore/aws-creds): route AWS CRT logs through Nix logger
2026-01-25 18:06:03 +00:00
Sergei Zimmerman e3b788b4ca tests/nixos/s3-binary-cache-store: Drop superfluous prints
As requested in review.
2026-01-25 19:40:30 +03:00
Bernardo Meurer Costa 3b8b764e29 feat(libstore/aws-creds): route AWS CRT logs through Nix logger
Previously AWS CRT logs went directly to stderr via ApiHandle::InitializeLogging,
causing log spam that didn't respect Nix's verbosity settings.

This implements a custom aws_logger using the aws-c-common C API that:
- Routes all AWS logs through nix::logger
- Maps AWS log levels conservatively (ERROR/WARN -> lvlInfo) since the SDK
  treats expected conditions like missing IMDS as errors
- Prefixes messages with (aws) for clarity
- Respects Nix's verbosity flags (-v, -vv, etc.)
2026-01-25 19:40:29 +03:00
Jörg Thalheim 2eb19a6353 Merge pull request #13030 from vlaci/mtls-auth
libstore/filetransfer: add support for MTLS authentication
2026-01-25 13:58:12 +00:00
John Ericson e8e3c30dfc Merge pull request #15076 from NixOS/prepare-for-25.11
Prepare for nixpkgs 25.11, enable S3 support in static builds
2026-01-24 23:11:18 +00:00
John Ericson a3f2d2b3e9 Merge pull request #15075 from NixOS/chmod-wrapper
Share the exception-using `chmod` wrapper with more code
2026-01-24 22:51:42 +00:00
Sergei Zimmerman 64458acde2 packaging: Fix static builds with S3 support, enable by default
aws-crt-cpp doesn't provide pkg-config files and has a bunch of transitive
deps, so switch to cmake for resolving the dependency.
2026-01-25 01:26:23 +03:00
John Ericson 6e2e53a8d2 Share the exception-using chmod wrapper with more code
It is not just useful to `DerivationBuilder`.
2026-01-24 17:03:48 -05:00
Sergei Zimmerman dcaaf2c65f dev-shell: Use stdenv.hostPlatform instead of hostPlatform
This is now a warning in 25.11:

> evaluation warning: 'hostPlatform' has been renamed to/replaced by 'stdenv.hostPlatform'
2026-01-25 00:50:56 +03:00
Sergei Zimmerman c4c0aee4f1 tests/nixos: Drop otherNixes.nix_2_3, replace with 2_18
Since [1] there's no way to run 2.3 anymore and overrides wouldn't be very
helpful. Let's instead use 2.18, which is the baseline for nixpkgs.
2026-01-25 00:50:55 +03:00
Sergei Zimmerman 0f22d60c7e tests/nixos: Specify -f argument to mount
Otherwise we barf on btrfs:

machine # [   17.027621] EXT4-fs error (device vdb): ext4_lookup:1819: inode #2476: comm nix: iget: checksum invalid
machine # error: getting status of '/mnt/nix/store/j8645yndikbrvn292zgvyv64xrrmwdcb-bash-5.3p3': Bad message
machine # checking '/nix/store/m3954qff15v7z1l6lpyqf8v2h47c7hv2-mailcap-2.1.54'...
machine # checking '/nix/store/xh1ff9c9c0yv1wxrwa5gnfp092yagh7v-tzdata-2025b'...
machine # [   17.172793] EXT4-fs error (device vdb): ext4_lookup:1819: inode #1777: comm nix: iget: checksum invalid
machine # error: getting status of '/mnt/nix/store/xh1ff9c9c0yv1wxrwa5gnfp092yagh7v-tzdata-2025b/share/zoneinfo/leap-seconds.list': Bad message
2026-01-25 00:50:54 +03:00
John Ericson 943c18f9fe Merge pull request #15072 from NixOS/fix-interrupted-linux-derivation-builder
Fix destruction of DerivationBuilder implementations
2026-01-24 21:16:34 +00:00
Sergei Zimmerman b752c5cb64 Fix destruction of DerivationBuilder implementations
This unsures that we call the correct virtual functions when destroying a particular
DerivationBuilder.

Usually the order of destructors is in the reverse order of inheritance:

ChrootLinuxDerivationBuilder -> ChrootDerivationBuilder -> DerivationBuilderImpl

autoDelChroot was being destroyed before the DerivationBuilderImpl::killChild was
run and it would fail to clean up the chroot directory, since there were still processes
writing to it. Note that ChrootLinuxDerivationBuilder::killSandbox was never run in
the interrupted case at all, since virtual functions in destructors do not call derived class
methods.

I could reproduce the issue with the following derivation:

let
  pkgs = import <nixpkgs> { };
in
pkgs.runCommand "chroot-cleanup-race" { } ''
  mkdir -p $out

  for i in $(seq 1 200); do
    (
      mkfifo $out/fifo$i
      cat $out/fifo$i > /dev/null &

      while true; do
        : > $out/file$i
      done
    ) &
  done

  sleep 0.05
  echo done > $out/main
''

While interrupting it manually when it would hang.

Wrapping the unique pointer in a custom deleter function we can run all
of the necessary clean up code consistently and calling the right virtual
functions. Ideally we'd have a lint that bans the usage of virtual functions
in destructors completely.
2026-01-24 23:31:11 +03:00
John Ericson b7d07e42dc Merge pull request #15071 from roberth/fix-concurrent-failure-bug
tests: fix sandbox-paths in cancelled-builds test
2026-01-24 19:48:25 +00:00
Sergei Zimmerman 0f17a1f655 libutil-tests: Add unit tests for https binary cache stores with mTLS
This addresses the concerns with network isolation that have been raised
previously [1] by only running the tests by default in a network namespace.
This way all networks tests are independent of each other and do not bind
to ports in the host namespace.

This is much neater than doing these sorts of tests in functional suite.

[1]: https://github.com/NixOS/nix/pull/14266#issuecomment-3411261285
2026-01-24 21:59:59 +03:00
Damien Diederen 36b0bebe25 http-binary-cache-store: Add 'tls-certificate' and 'tls-private-key' settings
Those are set via the store's URI, e.g.:

    https://substituter.invalid?tls-certificate=/path/to/cert.pem&tls-private-key=/path/to/key.pem
2026-01-24 21:59:58 +03:00
Robert Hensing 7b4444f174 tests: fix sandbox-paths in cancelled-builds test
Don't add the whole store to sandbox-paths unconditionally. Exposing
the entire store defeats the purpose of sandboxing, and when the test
store is the same as the system store (NixOS VM), it causes an obscure
"Permission denied" error.

Only add sandbox-paths when not on NixOS, indicating a separate test
store that needs access to system store build tools.
2026-01-24 19:55:50 +01:00
John Ericson bfdd124837 Create proper serializer for BuildResult status
The casts were not safe with respect to unknown values, but these are.
2026-01-23 16:27:26 -05:00
John Ericson aa17b75601 Merge pull request #15054 from obsidiansystems/unprivileged-test
Add new VM test with unprivileged daemon user
2026-01-23 19:49:46 +00:00
Jörg Thalheim 18176d2678 ignore-gc-delete-failure: add release note 2026-01-23 14:08:05 -05:00
John Ericson 1a17c9d02b Merge pull request #15051 from amaanq/split-build-log-settings
libstore: split out `LogFileSettings`
2026-01-23 18:58:51 +00:00
Artemis Tosini 94907eb37a Add new VM test with unprivileged daemon user
All current NixOS functional VM tests have a daemon as root with the
tests running as different unprivileged users.
The new `functional_unprivileged-daemon` test runs the daemon and the
nix functional tests as separate unprivileged users.
Users may want to run an unprivileged daemon on non-NixOS systems
where the administrator does not fully trust nix, but multiple users
want to use nix for their own purposes. It could also be useful in
concert with an overlay-mount store, where the nix daemon cannot
modify the derivations used by the system, and thus a nix vulnerability
would not lead to root code execution.
2026-01-23 13:31:16 -05:00
Artemis TosiniandJohn Ericson 2f1ce8900b Ignore delete failures during garbage collection
When running nix as an unprivileged user it may not be able to write to
all paths in the nix store. Ignore deletion failures to fix tests that
run `nix-collect-garbage` in this configuration.

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2026-01-23 13:31:07 -05:00
Amaan Qureshi 98178e24d0 libstore: split out LogFileSettings 2026-01-23 13:17:15 -05:00
Eelco Dolstra 4c6ad728d0 Merge pull request #15058 from amaanq/split-gc-settings
libstore: split out `GCSettings`
2026-01-23 16:18:30 +00:00
John Ericson 83360cd7b7 Merge pull request #14972 from roberth/fix-concurrent-failure-bug
Fix concurrent builder failure empty message bugs
2026-01-23 00:50:47 +00:00
John Ericson 2a21bd6d0a Merge pull request #15056 from NixOS/setns-explicit-arguments
linux-derivation-builder: Explicitly specify nstype for setns calls i…
2026-01-23 00:44:46 +00:00
Sergei Zimmerman 5570c31b30 Merge pull request #15062 from NixOS/drop-magic-nix-cache
ci: Drop magic-nix-cache
2026-01-22 23:52:53 +00:00
John Ericson eead36de18 Merge pull request #15061 from amaanq/build-result-throw
build-result: throw better
2026-01-22 23:23:28 +00:00
Sergei Zimmerman dae41e06e8 ci: Drop magic-nix-cache
We are now seeing. I guess we are out with the cache. When the API responds with 418 (I'm a teapot)
it seems like the only reasonable solution is to oblige.

error: unable to download 'http://127.0.0.1:37515/7ms9f25xyxavf32pvdc3vb28nzzmkbn3.narinfo': HTTP error 418
       response body:
       GitHub API error: GitHub Actions Cache throttled Magic Nix Cache. Not trying to use it again on this run.
2026-01-23 02:02:10 +03:00
Amaan Qureshi daba5f6386 build-result: throw better 2026-01-22 17:40:01 -05:00
John Ericson 1100c9dc23 Support readLinkAt and openFileEnsureBeneathNoSymlinks on Windows too
This means that `RestoreSink` can work in the TOCTOU-resilliant way on
Windows too. And it also bodes will for the upcoming OS source accessor
improvements.

A few misc little refactors around error handling and whatnot are done
along the way too. (No more attempt to support pre Windows Vista! lol.)

This cannot be realiably automatically tested until we have a newer
version of Wine, but it does build, so I am inclined to say we just try
it for now.
2026-01-22 17:35:28 -05:00
Amaan Qureshi 6f0fe5636d libstore: split out GCSettings
This PR follows the same approach as #15043 and the
[`LogFileSettings`](https://github.com/NixOS/nix/pull/15051)
extraction:

- `GCSettings` struct inherits from virtual `Config`
- `Settings` privately inherits from it
- Accessed through `getGCSettings()`

The new method on `LocalStoreConfig` anticipates on making these
settings per-store. 0b606aad46 added both
the autoGC and periodic wakeups, which is why we think they are related.
2026-01-22 17:06:08 -05:00
John Ericson 73a727f3d2 Merge pull request #15047 from lovesegfault/fix-15023
fix(libstore/filetransfer): restart source before upload retries
2026-01-22 19:45:48 +00:00
Sergei Zimmerman cc8f4912f5 linux-derivation-builder: Explicitly specify nstype for setns calls in addDependencyImpl
We already use file descriptors for this, so 0 is perfectly fine here, but this still
serves as a sanity check and slightly more self-documenting.
2026-01-22 22:29:22 +03:00
John Ericson 6276642164 Merge pull request #15050 from amaanq/sqlite-settings-struct
libstore: add `SQLite::Settings` struct for explicit configuration
2026-01-22 18:27:53 +00:00
Bernardo Meurer Costa fbd787b910 fix(libstore/filetransfer): restart source before upload retries
When an upload fails with a transient HTTP error (e.g., S3 rate limiting
with HTTP 503), retries would fail with "curl error: Failed to open/read
local data from file/application" because the upload source was already
exhausted from the previous attempt.

Restart the source in init() to ensure it's at the beginning for both
first attempts (no-op) and retries (necessary fix).

Fixes: #15023
2026-01-22 18:25:36 +00:00
Amaan Qureshi 751a0f40bc libstore: add SQLite::Settings struct for explicit configuration
Progress on #5638

Replace the SQLite constructor's mode parameter with a Settings struct
that includes both the open mode and useWAL flag. This makes the
dependency on useSQLiteWAL explicit at call sites rather than having
it read from the global settings inside the constructor.

All call sites now explicitly pass settings.useSQLiteWAL, preparing
for downstream work where stores can pass their own settings instead
of relying on the global.
2026-01-22 12:06:26 -05:00
Eelco Dolstra 857a2053ad Merge pull request #15048 from lovesegfault/fix-15019
fix(libstore/filetransfer): skip Accept-Encoding header for S3 SigV4 requests
2026-01-22 16:31:10 +00:00
John Ericson 087b6c4dc2 Merge pull request #15044 from obsidiansystems/remove-nixPrefix
Remove `Settings::nixPrefix`
2026-01-22 16:25:39 +00:00
John Ericson 3b8c408108 Merge pull request #15045 from obsidiansystems/remove-nixDataDir
Remove `nixDataDir`, `NIX_DATA_DIR`
2026-01-22 16:23:44 +00:00
John Ericson b7ddbb8e2d Merge pull request #15039 from NixOS/read-link-at
libutil: Add unix::readLinkAt function
2026-01-22 15:11:54 +00:00
Bernardo Meurer Costa fcfa1dc8ab fix(libstore/filetransfer): skip Accept-Encoding header for S3 SigV4 requests
Some S3-compatible services (like GCS) modify the Accept-Encoding header
in transit, which breaks AWS SigV4 signature verification since curl's
implementation signs all headers including Accept-Encoding.

Fixes: #15019
2026-01-22 06:35:01 +00:00
John Ericson ba219cb047 Merge pull request #15046 from juhp/patch-3
nix config check: improve error when no nix-env in PATH
2026-01-22 05:31:05 +00:00
Jens Petersen 9fa69276c4 nix config check: improve error when no nix-env
It is possible that the `nix` executable is installed but not `nix-env`
(this may be unusual but for example in Fedora we have a separate
`nix-legacy` subpackage, which includes the `nix-env` symlink).

The current error message:
```
$ nix config check --verbose
Running checks against store uri: local
[FAIL] Multiple versions of nix found in PATH:

```
when there is no nix-env in PATH is confusing.

This change makes the error message precise for the missing nix-env case.
2026-01-22 12:49:17 +08:00
John Ericson 98f6881d11 Remove nixDataDir, NIX_DATA_DIR
Since 25300c0ecd it is dead code.
2026-01-21 23:23:23 -05:00
John Ericson 56c9d5f04e Remove Settings::nixPrefix
It has been dead code since c9f51e8705
2026-01-21 23:16:27 -05:00
John Ericson 07a3171fb9 Merge pull request #15028 from NixOS/copyFdRange-improvements
Deduplicate `copyFdRange` with new `readOffset`
2026-01-22 00:28:01 +00:00
John Ericson fe8f574471 Clean up NarAccessor construction
We had a minor combinatorial explosion of ways to do things. We can get
rid of those by just having the caller call `parseNarListing` intead.
2026-01-21 19:25:52 -05:00
Eelco DolstraandJohn Ericson 0b2dffefea Factor out NarCache from RemoteFSAccessor
Use

```
git show --color-moved --patience --color-moved-ws=ignore-all-space
```

to review and see that this is mostly code motion.

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2026-01-21 19:15:51 -05:00
Sergei Zimmerman 52100c6ee1 libutil: Add unix::readLinkAt function
This will be used for TOCTOU-free NAR serialisation and recursive copying.
2026-01-22 02:50:39 +03:00
John Ericson fa53a9cec8 Deduplicate copyFdRange with new readOffset 2026-01-21 18:16:28 -05:00
Eelco Dolstra 44dce7a3d1 Merge pull request #15029 from amaanq/signature-type-core
libutil: add `Signature` struct for typed signatures
2026-01-21 21:17:01 +00:00
Amaan Qureshi 12ef043655 libutil: add Signature struct for typed signatures
Introduce a new `Signature` struct that represents a cryptographic
signature
along with the key name that produced it. This provides:

- Structured representation instead of colon-separated strings
- Type-safe parsing with `Signature::parse()`
- Serialization with `to_string()`
- JSON serialization/deserialization
- Batch parsing with `parseMany<Container>()`
- Batch serialization with `toStrings()`

This is scaffolding for future changes that will use this type
throughout the codebase.
2026-01-21 11:51:46 -05:00
Eelco Dolstra 7a40df3510 Merge pull request #15037 from NixOS/realise-path
realisePath(): Move into EvalState
2026-01-21 16:24:42 +00:00
Eelco Dolstra 4f733f736e realisePath(): Move into EvalState
This allows it to be used by primops defined outside of primops.cc.
2026-01-21 14:43:58 +01:00
John Ericson c6d07ec0aa Merge pull request #15030 from xokdvium/path-fmt-squash-double-quotes
libutil: Add PathFmt wrapped type for formatting fs::path, fix all double-quoting issues
2026-01-21 04:28:30 +00:00
John Ericson e2cd5679eb Merge pull request #15031 from NixOS/bump-magic-nix-cache
ci: Bump magic-nix-cache to disable on 429
2026-01-21 04:11:31 +00:00
Sergei Zimmerman 1555677cd5 ci: Bump magic-nix-cache to disable on 429 2026-01-21 06:14:34 +03:00
Sergei Zimmerman 6dd89b5a2a libutil: Add PathFmt wrapped type for formatting fs::path, fix all double-quoting issues
This will once and for all get rid of all double-quoting issues. On windows the quoting
is doubly bad because it escaped all \ to \\, which is very bad for error messages. In
order to prevent future regression std::filesystem::path formatting now must use a special
type PathFmt (like Magenta). In the future we could even change how we render filesystem paths.
2026-01-21 06:06:19 +03:00
Sergei Zimmerman 73beff89cb libutil: Fix mingw build 2026-01-21 04:54:54 +03:00
tomberek f429d8d4aa Merge pull request #14766 from pkpbynum/capi/query-path-from-hash-part
C API: Add query_from_hash_part to Store API
2026-01-20 16:39:05 +00:00
John Ericson 017a247e63 Merge pull request #15025 from cole-h/cole-h/push-qmyswwomnsnl
libutil: add missing tracking URLs for external-builders and blake3-h…
2026-01-20 16:17:30 +00:00
Eelco Dolstra 4a267f720e Merge pull request #14998 from NixOS/fix-remote-store-nar-from-path
libstore: Do not mark connections as bad when RemoteStore::narFromPath is called as a coroutine
2026-01-20 16:08:07 +00:00
John Ericson 490f6eeba5 Merge pull request #15021 from OPNA2608/fix/libstore-ppc64
libstore: make withAWS depend on aws-c-common availability
2026-01-20 15:44:35 +00:00
Cole Helbling c398dd7cbd libutil: add missing tracking URLs for external-builders and blake3-hashes 2026-01-20 07:32:26 -08:00
OPNA2608 37834c5e58 libstore: make withAWS depend on platform's aws-c-common availability 2026-01-20 14:39:57 +01:00
John Ericson 67a99db5be Merge pull request #15020 from xokdvium/more-enum-compression
Use CompressionAlgo enum throughout
2026-01-20 04:50:56 +00:00
John Ericson a59bc630aa Merge pull request #15015 from NixOS/catch-system-error
Catch `SystemError` in portable code
2026-01-20 04:41:29 +00:00
Sergei Zimmerman 6ba067831a Use CompressionAlgo throughout
Instead of the stringly typed code we should use an enum class, this is
more clear and less error-prone. Also adds settings implementations for
CompressionAlgo and std::optional<CompressionAlgo>. The first is used
for NAR compression, since we never accepted empty strings there:

error: unknown compression method ''

The other one is used for optional .narinfo, .ls, and log/ compression.
Those treated empty strings as compression being disabled. The same exact
semantics is kept.

This has the benefit of improving error messages for incorrect values:

error: option 'compression' has invalid value 'bz'
       Did you mean one of br, xz or lz4?
2026-01-20 04:35:16 +03:00
Sergei Zimmerman b24df97a11 binary-cache-store: Update compression setting documentation to match reality
The docs were out of date. Since 8a0c00b856 Nix
supports all compression algorithms exposed by libarchive (if it's built with
native support for them). Let's be honest about it in the docs.
2026-01-20 03:31:30 +03:00
Sergei Zimmerman 556974f33b libutil: Remove unused overload of unpackTarfile 2026-01-20 01:32:43 +03:00
Sergei Zimmerman b40b786839 libstore/remote-store: Add checkInterrupt in openConnectionWrapper
This avoids the wall of text like, because ThreadPool doesn't print interrupts
on shutdowns.

error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
2026-01-19 22:28:38 +03:00
Sergei Zimmerman 9e496f9af2 libutil: Make Pid destructor more robust
Without this we can abort by throwing an exception in the destructor:

[24/635/2958 copied (3.8/26.0 GiB)] copying path '/nix/store/ncd2iic2nwxwhqsf4gp9sdybkwnwz20b-ruby3.3-mini_portile2-2.8.9' from 'ssh-ng://localhost:22'

Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:

Exception: nix::Interrupted: error: interrupted by the user
Stack trace:
 0# 0x00000000004AFFE9 in result/bin/nix
 1# 0x00007F946290A1AA in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
 2# __cxa_call_terminate in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
 3# __gxx_personality_v0 in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
 4# 0x00007F946283FA19 in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libgcc_s.so.1
 5# _Unwind_RaiseException in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libgcc_s.so.1
 6# __cxa_throw in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
 7# 0x00007F94635D82D0 in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
 8# nix::Pid::wait() in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
 9# nix::Pid::~Pid() in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
2026-01-19 22:28:37 +03:00
Sergei Zimmerman 726e924bd7 libstore: Do not mark connections as bad when RemoteStore::narFromPath is called as a coroutine
forced_unwind is thrown by Boost.Context when destroying the coroutine.
This lead to us resetting the remote connection for each narFromPath
with the ssh-ng:// store, so copying was very slow.
2026-01-19 22:28:34 +03:00
Eelco Dolstra 3a421388dd Merge pull request #15006 from roberth/doc-nix-cache-info
doc: add nix-cache-info format documentation
2026-01-19 15:16:20 +00:00
John Ericson a32c139379 Catch SystemError in portable code
This will ensure this catching works on Windows too, not just Unix.
2026-01-18 18:53:45 -05:00
John Ericson 63344d3a3b Merge pull request #15013 from NixOS/seekable-nar-bytes-thread-safe-pread
nar-accessor: Fix thread safety of `seekableGetNarBytes`, use `Sink`, reduce memory usage of `nix store cat`
2026-01-18 23:26:08 +00:00
Sergei Zimmerman 4db68c28c1 treewide: Add missing overrides of streaming readFile, make readFile non-virtual
This makes all addToStore operations that use these source accessors
constant memory regardless of file sizes. Also make the other overload
altogether and relegate it to the base class as a non-virtual method to
avoid such mistakes.
2026-01-19 01:15:26 +03:00
Sergei Zimmerman 6ba468805b libutil: Factor out copyFdRange
This factors out the helper function from seekableGetNarBytes into copyFdRange
and adds some more sanity checks for offset/length truncation/wrapping at that
API boundary where we work with NAR-style offsets and convert to native off_t.
2026-01-19 01:15:23 +03:00
Sergei Zimmerman d25ab60d66 nix/cat: Use streaming version of readFile
This reduces the memory overhead of nix store cat down to constant memory
with a local-nar-cache.
2026-01-19 00:52:05 +03:00
Sergei Zimmerman 656e1fc659 nar-accessor: Fix thread safety of seekableGetNarBytes, use Sink
Instead of mutating the file pointer we can instead safely do
preads. That makes the local-nar-info cache once again thread safe
without the overhead of reopening the file that we used to have prior
to b9b6defca6 which broke the thread safety
by persisting the file descriptor.
2026-01-19 00:51:57 +03:00
Sergei Zimmerman 054de385d8 Merge pull request #15011 from trofi/lixnu-openat-sysno
libutil: fix `linux` build on fresh `glibc` and `gcc`
2026-01-17 17:16:43 +00:00
Sergei Trofimovich 3256aba6a2 libutil: fix linux build on fresh glibc and gcc
Without the change the build fails for me as:

    ../unix/file-descriptor.cc:404:70: error: 'RESOLVE_BENEATH' was not declared in this scope
      404 |         dirFd, path.rel_c_str(), flags, static_cast<uint64_t>(mode), RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS);
          |                                                                      ^~~~~~~~~~~~~~~

This happens for 2 reasons:
1. `__NR_openat2` constant was not pulled in from the according headers
   and as a result `<linux/openat2.h>` was not included.
2. `define HAVE_OPENAT2 0` build is broken: refers to missing
   `RESOLVE_BENEATH` normally pulled in from `<linux/openat2.h>`

This changes fixes both.
2026-01-17 13:01:22 +00:00
Robert Hensing b3df7f8a3d doc: add nix-cache-info format documentation
Document the nix-cache-info file format used by binary caches, including
the StoreDir, WantMassQuery, and Priority fields, their behavior, and
links to related store options.
2026-01-17 13:31:32 +01:00
Sergei Zimmerman af7c7b6723 Merge pull request #14973 from NixOS/windows-known-folders
Use known folders for nix data on windows
2026-01-16 23:58:17 +00:00
John Ericson d6fa3e3b50 Merge pull request #15010 from amaanq/fix-test-signature-format
tests: use valid signature format in protocol characterization tests
2026-01-16 23:20:41 +00:00
Sergei ZimmermanandJohn Ericson 5d2938520c Use known folders for nix data on windows
This is the usual conventions on windows.

See https://learn.microsoft.com/en-us/windows/win32/shell/knownfolderid and
https://github.com/adrg/xdg for examples of the mapping of XDG paths to Windows
known folders.

Additionally, on Windows, this allows us to dispense with a hard-coded
default for `nixConfDir`, which is both nice (fewer compile-time
configuration options) and necessary, because we don't know what drive
the `ProgramData` directory will live on.

Tested on wine.

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2026-01-17 02:12:41 +03:00
Amaan Qureshi 51ca872c60 tests: use valid signature format in realisation protocol tests
The test data was using invalid signature strings like "asdf" and
"qwer" which don't follow the required "name:base64signature" format.
This updates them to use properly formatted signatures with valid
base64-encoded data.
2026-01-16 17:49:46 -05:00
John Ericson 1e1d9f28ba Merge pull request #14754 from NixOS/structured-attrs-cleanup
Clean up structured attrs parsing using JSON utils
2026-01-16 22:13:18 +00:00
John Ericson 54c62782f5 Move environmentVariablesCategory to libcmd
It does not belong in libutil.
2026-01-16 16:01:04 -05:00
Eelco Dolstra 162b0072a7 Merge pull request #15008 from NixOS/typo
Fix typo spotted by coderabbit
2026-01-16 20:22:28 +00:00
John Ericson eb653e5928 Clean up structured attrs parsing using JSON utils 2026-01-16 14:55:18 -05:00
Eelco Dolstra 33ce87276f Fix typo spotted by coderabbit 2026-01-16 20:08:56 +01:00
Eelco Dolstra f1d468e840 Merge pull request #14965 from shlevy/controllerless-delegation
systemd: Delegate cgroup management without turning on controllers.
2026-01-16 16:16:27 +00:00
Eelco Dolstra 3fb0fb00ae Merge pull request #15004 from roberth/docs
Add Nix32 encoding documentation
2026-01-16 15:36:26 +00:00
Robert Hensing 3374fdc04a Add Nix32 encoding documentation
Document the Nix32 base-32 variant used for store path digests and
hash output. The new page covers:
- The 32-character alphabet (omitting e, o, u, t)
- Byte order differences from base-16 encoding

Also update references throughout the manual to link to the new page.
2026-01-16 11:28:55 +01:00
John Ericson a9b1a52623 Merge pull request #14949 from iljah/patch-3
Add example commands to source installation
2026-01-15 17:02:45 +00:00
Eelco Dolstra f162bb62f7 Merge pull request #14994 from NixOS/curl-scalability
resolverCallbackWrapper(): Catch exceptions
2026-01-14 22:35:50 +00:00
Eelco Dolstra 292156c336 Merge pull request #14995 from NixOS/daemon-proto-cleanup
Remove more obsolete daemon operations
2026-01-14 21:26:22 +00:00
Eelco Dolstra 3cb42b581a Remove WorkerProto::Op::{QueryFailedPaths,ClearFailedPaths}
Support for failed paths was removed in
8cffec8485 (April 2016).
2026-01-14 21:30:24 +01:00
Eelco Dolstra 17295066e8 Remove WorkerProto::Op::QueryReferences
This has been obsolete since e0204f8d46
(April 2016).
2026-01-14 21:25:33 +01:00
Eelco Dolstra 036738be11 Remove WorkerProto::Op::QueryPathHash
This has been obsolete since e0204f8d46
(April 2016).
2026-01-14 21:22:11 +01:00
Robert Hensing 3c3ceb18e9 DerivationTrampolineGoal: improve error message wording
Change "cannot build missing derivation" to "failed to obtain derivation of"
since the path (e.g. '...drv^out') is a derivation output, not a derivation.

The message could be improved further to resolve ambiguity when multiple
outputOf links are involved, but for now we err on the side of brevity
since this message is already merged into larger error messages with
other context from the Worker and CLI.
2026-01-14 20:42:20 +01:00
Robert Hensing 68f549def4 buildPathsWithResults: don't report cancelled goals as failures
When !keepGoing and a goal fails, other goals are cancelled and
remain with exitCode == ecBusy. These cancelled goals have a default
BuildResult::Failure{} with empty errorMsg.

Previously, buildPathsWithResults would return these cancelled goals,
and throwBuildErrors would report them as failures. When only one such
cancelled goal was present, it would throw an error with an empty
message like:

    error: build of '/nix/store/...drv^*' failed:

Now we skip goals with ecBusy since their state is indeterminate.
Cancelled goals could be reported, but this keeps the output relevant.
Other indeterminate goal states were already not being reported, for
instance: derivations that weren't started for being blocked on a
concurrency limit, or blocked on a currently building dependency.
2026-01-14 20:42:20 +01:00
Robert Hensing 3fd85c7d64 tests: don't expect cancelled goals to be reported as failures
When keepGoing=false and a build fails, other goals are cancelled.
Previously, these cancelled goals were reported in the "build of ...
failed" error message alongside actual failures. This was misleading
since cancelled goals didn't actually fail - they were never tried.

Update the test to expect only the actual failure (hash mismatch) to
be reported, not the cancelled goals.
2026-01-14 20:42:20 +01:00
Robert Hensing 25eb07a91b DerivationTrampolineGoal: use doneFailure to set buildResult
DerivationTrampolineGoal is the top-level goal whose buildResult is
returned by buildPathsWithResults. When it failed without setting
buildResult.inner, buildPathsWithResults would return failures with
empty errorMsg, producing error messages like:

  error: failed to build attribute 'checks.x86_64-linux.foo',
  build of '/nix/store/...drv^*' failed:

(note the empty message after "failed:")

Use the new doneFailure helper to ensure buildResult is populated
with meaningful error information.
2026-01-14 20:42:20 +01:00
Eelco Dolstra 0a632cbc3a Merge pull request #14989 from NixOS/querySubstitutablePaths
Move LocalStore::querySubstitutablePaths() to Store
2026-01-14 19:38:28 +00:00
Eelco Dolstra 8c2021989e Merge pull request #14993 from NixOS/curl-scalability
Limit the number of active curl handles
2026-01-14 19:02:21 +00:00
Eelco Dolstra 92344a31fa LegacySSHStore: Override querySubstitutablePaths() 2026-01-14 19:48:14 +01:00
Eelco Dolstra 77fa94d8d4 resolverCallbackWrapper(): Catch exceptions 2026-01-14 19:40:53 +01:00
Eelco Dolstra ee64ffcd75 curlFileTransfer: Lazily create activity and set startTime
There can be a long time between the creation of `TransferItem` and
the start of the curl download, which can lead to misleading download
durations and progress bar status. So now we create the `Activity` and
update `startTime` when curl actually starts the download.
2026-01-14 19:19:01 +01:00
Eelco Dolstra 8012b584c3 Limit the number of active curl handles
Previously, calling queryValidPaths() with a large number (e.g. 100K)
of store paths failed because Nix immediately creates a `TransferItem`
for each .narinfo, which is then registered as a handle with
curl. However curl appears to scale poorly internally: even though
only a few downloads are actually started (up to the
connections/streams limits), it spends a lot of CPU time dealing with
the inactive handles. So the curl thread is sitting at 100% CPU, the
active downloads stall and time out, and everything grind to a halt.

So now we limit the number of curl handles to http-connections *
5. With this, fetching 100K .narinfo files from localhost succeeds in
~15 seconds.
2026-01-14 18:38:59 +01:00
John Ericson 609a4e999f Merge pull request #14988 from NixOS/remove-has-substitutes
Remove WorkerProto::Op::HasSubstitutes
2026-01-14 17:17:39 +00:00
Eelco Dolstra 6fd36553b9 Merge pull request #14990 from NixOS/nar-cache-cleanup
More NAR accessor / listing cleanup
2026-01-14 12:53:36 +00:00
John Ericson ce28cb32e9 BinaryCacheStore: Avoid recreating NAR listing
We already have it, so let's just use it!
2026-01-13 14:01:25 -05:00
John Ericson 3d18d73003 Remove redundant NarAccessor::nar 2026-01-13 13:32:29 -05:00
John Ericson 8089af3bb0 Better type for NarMemberConstructor::regular
This makes some invariants clearer and more local.
2026-01-13 13:12:15 -05:00
John Ericson b49ea8b246 Factor out parseNarListing, move to nar-listing.{cc,hh}
Now we have less of a maze of implementation structs.

Review with
```
git show --color-moved --patience --color-moved-ws=ignore-all-space
```
2026-01-13 13:10:04 -05:00
John Ericson af821ba647 Split out nar-listing.{cc,hh}
I like the separation of concerns from NAR accessing.
2026-01-13 12:10:54 -05:00
Eelco Dolstra 1b1c949d0c Merge pull request #14981 from roberth/fix-git-relative-url-crash
`fixGitURL`: fix crash for "relative" `file:` paths and reject unsupported SCP URLs
2026-01-13 12:59:59 +00:00
Eelco Dolstra fb70a45b9e Move LocalStore::querySubstitutablePaths() to Store
Nothing in this method depends on LocalStore, so we can use it as the
default implementation. (It's only overriden in RemoteStore.)
2026-01-13 13:55:08 +01:00
Eelco Dolstra 53e942bfcc Merge pull request #14983 from roberth/fetchTree-relative-file
fetchTree: reject relative `file:` paths for tarballs
2026-01-13 12:46:08 +00:00
Eelco Dolstra 4f5172ba21 Remove WorkerProto::Op::HasSubstitutes
This operation has been deprecated since
09a6321aeb (July 2012). It was used by
client versions <= 11, which is below `MINIMUM_PROTOCOL_VERSION`
(currently 18).
2026-01-13 12:26:10 +01:00
Jörg Thalheim 1bddbff3a6 Merge pull request #14986 from NixOS/nar-cache-cleanup
Two NAR accessor / listing cleanups
2026-01-13 07:31:29 +00:00
John Ericson 4991defc44 Make reading in a nar listing all well typed
We can get rid of `NarMember`, because it is just `NarListing` in
disguise! The use of `std::variant` makes clear that certain stat fields
we don't care about in the non-regular-file case too.
2026-01-13 01:33:53 -05:00
John Ericson 8f829e478f Inline RemoteFSAccessor::makeCacheFile into lambda
It's just easier to avoid headers.
2026-01-13 01:33:53 -05:00
John Ericson 7ac5a61208 Merge pull request #14948 from NixOS/ca-nar-cache
RemoteFSAccessor: Make the local NAR cache content-addressed
2026-01-13 04:56:31 +00:00
John Ericson 2af5792cc2 Inline RemoteFSAccessor::addToCache
It was not pulling its weight. (Only used once, optional paths are
confusing, we already have an `if` / branch fit-for-purpose.)
2026-01-12 23:09:45 -05:00
Eelco DolstraandJohn Ericson e251ffdd46 RemoteFSAccessor: Make the local NAR cache content-addressed
Use double-indirection for better NAR accessor caching

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2026-01-12 23:00:20 -05:00
John Ericson a8087ebf52 Merge pull request #14984 from NixOS/more-windows-fixes
More windows fixes
2026-01-12 23:19:38 +00:00
Sergei Zimmerman 66fefcd795 libutil: Better implementation of createAnonymousTempFile on windows 2026-01-13 01:28:17 +03:00
Sergei Zimmerman 1ea3a841bb libstore/local-binary-cache-store: Use portable error_code in getFile 2026-01-13 00:56:03 +03:00
Sergei Zimmerman fd0bcd97e8 libutil: Include std::error_code in the base class SystemError 2026-01-13 00:50:42 +03:00
John Ericson aaccb73916 ptrToOwned should be in the nix namespace 2026-01-12 16:14:21 -05:00
Robert Hensing b19bfc6373 fetchTree: improve relative path error wording
Avoid implying that relative paths could work if a base directory
were defined. The file: scheme fundamentally does not support them.
2026-01-12 20:16:26 +01:00
Robert Hensing 3b028edbf7 fixGitURL: fix crash for relative paths and reject unsupported SCP URLs
Relative paths (e.g., "relative/repo") would crash in renderAuthorityAndPath()
because an empty authority was set, violating RFC 3986 section 3.3 which
requires paths to start with "/" when an authority is present.

Fix by only setting authority for absolute paths:
- Absolute paths: file:///path (empty authority)
- Relative paths: file:path (no authority)

Also reject SCP-like URLs without a user (e.g., "github.com:path") with a
clear error message, since proper support requires careful implementation,
which is not something I can do right now.
2026-01-12 13:03:57 +01:00
Robert Hensing 23a7178eb4 fetchTree: reject relative file: paths for tarballs
Relative paths like `file:./foo.tar.gz` have never worked for tarballs
because curl rejects relative file: URLs. Previously this resulted in
cryptic curl errors. Now we reject them early with a clear message
explaining that relative paths are not supported because there is no
defined base directory to resolve them against.

See https://github.com/NixOS/nix/issues/12281
2026-01-12 03:31:42 +01:00
John Ericson 252aff5c8f Merge pull request #14971 from obsidiansystems/misc-builder-fixes
Misc builder fixes
2026-01-11 21:02:14 +00:00
Sergei Zimmerman 920c5ceb0c Merge pull request #14961 from NixOS/readdir-nonexistent-fix
libutil/union-source-accessor: Barf on non-existent directories
2026-01-11 18:15:22 +00:00
John Ericson de76cb681d Make sure we reliably call Worker::childTerminated
When a goal with an active child process is destroyed (e.g., during
failure cascades without `--keep-going`), the child process gets killed
but `childTerminated` was never called. This left stale entries in the
worker's `children` list.

Fix this by ensuring `childTerminated` is called from destructors:

- `DerivationBuilderImpl::killChild` now calls `childTerminated` via
  the `miscMethods` callback.

- `HookInstance` gains an `onKillChild` callback that is invoked from
  its destructor when killing the process. `buildWithHook` sets this
  callback to call `childTerminated`.

To make these calls safe from destructors (where the goal object may be
partially destroyed), add a new overload of `Worker::childTerminated`
that takes an explicit `JobCategory` parameter instead of calling the
virtual method `Goal::jobCategory`. The original overload still exists
for convenience for normal (non-destructor) call sites.
2026-01-10 20:24:05 -05:00
John Ericson d19bfb0045 Avoid short circuiting in Worker::removeGoal
We have to call into both branches no matter what.
2026-01-10 20:23:50 -05:00
John Ericson 160822858a Merge pull request #14788 from NixOS/coroutine-child-output
Way more RAII for `DerivationBuildingGoal`
2026-01-11 00:09:53 +00:00
John Ericson 25998fcd1e Merge pull request #14970 from NixOS/more-openfile-readonly
Use openFileReadonly in more places
2026-01-11 00:07:15 +00:00
John Ericson 7ba09399cc Merge pull request #14968 from NixOS/fs-sink-restore-regular-file-fd
libutil: RestoreRegularFile is an FdSink
2026-01-11 00:05:20 +00:00
Sergei Zimmerman f8a92564f7 More std::filesystem::path for nix {cat,ls}
Also fixes a double quoting issue I accidentally introduced ccdd1f1c65
in seekableGetNarBytes.
2026-01-11 01:44:16 +03:00
Sergei Zimmerman 2ae4121c1d libutil/file-system: Use openFileReadonly in more places 2026-01-11 01:44:15 +03:00
John Ericson 38c755f168 Merge pull request #14966 from drupol/push-ozvunuqvxrvw
chore: replace `edolstra/flake-compat` with `NixOS/flake-compat`
2026-01-10 15:59:07 +00:00
Robert Hensing cb2ade20d4 Goal: add doneSuccess/doneFailure helpers to base class
Add helpers to the base Goal class that set buildResult and call amDone,
ensuring buildResult is always populated when a goal terminates.

Derived class helpers now call the base class versions. This reorders
operations: previously buildResult was set before bookkeeping (counter
resets, worker stats), now it's set after. This is safe because the
bookkeeping code (mcExpectedBuilds.reset(), worker.doneBuilds++,
worker.updateProgress(), etc.) only accesses worker counters, not
buildResult.
2026-01-10 16:15:48 +01:00
Sergei Zimmerman 08887caa1a libutil: RestoreRegularFile is an FdSink
It correctly models the is-a relation. This will be useful for doing a dynamic_cast in
downstream code that wants to copy from a file descriptor to a file descriptor.
2026-01-10 16:31:05 +03:00
Pol Dellaiera 8d588ad471 chore: replace edolstra/flake-compat with NixOS/flake-compat 2026-01-10 10:10:55 +01:00
Shea Levy d51ac82dd2 systemd: Delegate cgroup management without turning on controllers.
Nix currently doesn't do any resource control, and Delegate=yes turns on all the controllers.

In particular, this enables using cpusets with cgroups V1 alongside the Nix daemon.
2026-01-09 23:22:15 -05:00
Sergei Zimmerman 6970efe2e1 Merge pull request #14962 from NixOS/fix-mingw
Fix mingw build (once again), add openFileReadonly and clean up error.hh to include WinError
2026-01-10 00:39:46 +00:00
Sergei Zimmerman 3b95b7c9aa Merge pull request #14963 from corngood/test-leak
libflake-tests: fix leak in nix_api_store_test.nix_api_load_flake_with_flags
2026-01-09 23:44:30 +00:00
David McFarland 4289e2f9e6 libflake-tests: fix leak in nix_api_store_test.nix_api_load_flake_with_flags 2026-01-09 16:51:15 -04:00
Sergei Zimmerman ccdd1f1c65 libstore: Fix mingw build
This also adds a utility for opening a file descriptor from a path in readonly mode.
Previous commit helps a bit with error handling, since now we just throw a NativeSysError.
2026-01-09 21:06:34 +03:00
Sergei Zimmerman b7fd471f84 libutil: Inline windows-error.hh into error.hh
This way each consumer of NativeSysError doesn't have to
also conditionally include the windows-error.hh, which is very cumbersome.
And we can't include windows-error.hh in error.hh because of a circular import.
2026-01-09 20:59:35 +03:00
Sergei Zimmerman 4ab2cdacfc libutil/union-source-accessor: Barf on non-existent directories
Previously builtins.readDir would return an empty attribute set
instead of barfing on non-existent paths. This is a regression from
2.32 for impure eval.
2026-01-09 20:19:32 +03:00
Kai Oberbeckmann cd6eb7e473 Install nix-manual in default user profile
This makes man pages available in the default profile after using nix
installer.

Relates to: https://github.com/NixOS/nix/issues/12382
2026-01-09 16:16:44 +01:00
Ilja 75571ec0a0 Add example commands to source installation 2026-01-08 12:46:24 +02:00
John Ericson 1807dc78d6 Split out separate coroutine functions for with/without hook
This makes the logic much easier to follow. Unlike before, the use of
separate functions is not making us pass a gazillion arguments or use
the crutch of class variables.
2026-01-05 14:38:25 -05:00
John Ericson cf5644ce99 DerivationBuildingGoal Make some fields constant
These are immutable parameters, not state, set in the constructor.
2026-01-05 14:38:25 -05:00
John Ericson 47ccf201fb Make DerivationBuildingGoal::inputPaths local variable 2026-01-05 14:38:25 -05:00
John Ericson ec65132ab9 Shrink DerivationBuildingGoal::tryBuildHook
There was a bunch of logic in there which was, effectively, using the
build hook, rather than deciding *whether* to use the build hook. We
want it to only be the latter.
2026-01-05 14:38:25 -05:00
John Ericson bde0b22dc8 Factor out LogFile, do more RAII
It is weird that `LogFile` and `BuildLog` are basically unrelated, but
the does currently reflect the logic that exists.
2026-01-05 14:38:24 -05:00
John Ericson a8e8614fd3 BuildLog owns activities
Now it does more, using the logging.hh functionality as needed.
2026-01-05 14:29:42 -05:00
John Ericson 6a32b754a9 Inline DerivationBuildingGoal::logSize
Separate state for each loop.
2026-01-05 14:29:42 -05:00
John Ericson 43a472631b Factor out BuildLog
This gets some trickier pure logic out of `DerivationBuilderGoal`.
2026-01-05 14:29:42 -05:00
John Ericson eff403b5ab Inline DerivationBuildingGoal::builder 2026-01-05 14:29:42 -05:00
John Ericson 6e6dda9f67 Inline DerivationBuildingGoal::hook
Also remove uneeded arguments from `processChildOutput`.
2026-01-05 14:29:42 -05:00
John Ericson ee1383f75f Get cleanup logic out of DerivationBuildingGoal::processChildOutput
This will allow for making `hook` and `builder` local variables.
2026-01-05 14:29:42 -05:00
John Ericson 5be07abf6d Inline DerivationBuildingGoal::killChild 2026-01-05 14:29:42 -05:00
John Ericson 469212bd38 Inline DerivationBuildingGoal::currentHookLine
That's easy to do now.
2026-01-05 14:29:42 -05:00
John Ericson 4a1a562461 Inline the hook-only part of processChildOutput to that caller 2026-01-05 14:29:42 -05:00
Peter Bynum eaf474bf24 add query_from_hash_part c API 2026-01-05 11:39:15 -05:00
512 changed files with 11787 additions and 6130 deletions

No files matched your search

@@ -24,8 +24,8 @@ inputs:
description: "Github token"
required: true
use_cache:
description: "Whether to setup magic-nix-cache"
default: true
description: "Whether to setup github actions cache (not implemented currently)"
default: false
required: false
runs:
using: "composite"
@@ -122,10 +122,3 @@ runs:
source-url: ${{ inputs.experimental-installer-version != 'latest' && 'https://artifacts.nixos.org/experimental-installer/tag/${{ inputs.experimental-installer-version }}/${{ env.EXPERIMENTAL_INSTALLER_ARTIFACT }}' || '' }}
nix-package-url: ${{ inputs.dogfood == 'true' && steps.download-nix-installer.outputs.tarball-path || (inputs.tarball_url || '') }}
extra-conf: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 # v13
if: ${{ inputs.use_cache == 'true' }}
with:
diagnostic-endpoint: ''
use-flakehub: false
use-gha-cache: true
source-revision: 92d9581367be2233c2d5714a2640e1339f4087d8 # main
+2 -2
View File
@@ -40,12 +40,12 @@ jobs:
role-session-name: nix-release-oidc-${{ github.run_id }}
aws-region: eu-west-1
- name: Login to Docker Hub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
with:
registry: ghcr.io
username: ${{ github.actor }}
+2
View File
@@ -1,5 +1,7 @@
# Default meson build dir
/build
# Meson creates this file too
src/.wraplock
# /tests/functional/
/tests/functional/common/subst-vars.sh
+29
View File
@@ -0,0 +1,29 @@
---
synopsis: "Rust nix-installer in beta"
prs: []
---
The Rust-based rewrite of the Nix installer is now in beta.
We'd love help testing it out!
To test out the new installer, run:
```
curl -sSfL https://artifacts.nixos.org/nix-installer | sh -s -- install
```
This installer can be run even when you have an existing, script-based Nix installation without any adjustments.
This new installer also comes with the ability to uninstall your Nix installation; run:
```
/nix/nix-installer uninstall
```
This will get rid of your entire Nix installation (even if you installed over an existing, script-based installation).
This installer is a modified version of the [Determinate Nix Installer](https://github.com/DeterminateSystems/nix-installer) by Determinate Systems.
Thanks to Determinate Systems for all the investment they've put into the installer.
Source for the installer is in https://github.com/NixOS/nix-installer.
Report any issues in that repo.
For CI usage, a GitHub Action to install Nix using this installer is available at https://github.com/NixOS/nix-installer-action.
@@ -0,0 +1,9 @@
---
synopsis: "C API: New store API methods"
prs: [14766]
---
The C API now includes additional methods:
- `nix_store_query_path_from_hash_part()` - Get the full store path given its hash part
- `nix_store_copy_path()` - Copy a single store path between two stores, allows repairs and configuring signature checking
@@ -0,0 +1,10 @@
---
synopsis: "New setting `ignore-gc-delete-failure` for local stores"
prs: [15054]
---
A new local store setting [`ignore-gc-delete-failure`](@docroot@/store/types/local-store.md#store-local-store-ignore-gc-delete-failure) has been added.
When enabled, garbage collection will log warnings instead of failing when it cannot delete store paths.
This is useful when running Nix as an unprivileged user that may not have write access to all paths in the store.
This setting is experimental and requires the [`local-overlay-store`](@docroot@/development/experimental-features.md#xp-feature-local-overlay-store) experimental feature.
+15
View File
@@ -0,0 +1,15 @@
---
synopsis: Support HTTPS binary caches using mTLS (client certificate) authentication
issues: [13002]
prs: [13030]
---
Added support for `tls-certificate` and `tls-private-key` options in substituter URLs.
Example:
```
https://substituter.invalid?tls-certificate=/path/to/cert.pem&tls-private-key=/path/to/key.pem
```
When these options are configured, Nix will use this certificate/private key pair to authenticate to the server.
@@ -0,0 +1,32 @@
---
synopsis: S3 binary caches now use virtual-hosted-style addressing by default
issues: [15208]
---
S3 binary caches now use virtual-hosted-style URLs
(`https://bucket.s3.region.amazonaws.com/key`) instead of path-style URLs
(`https://s3.region.amazonaws.com/bucket/key`) when connecting to standard AWS
S3 endpoints. This enables HTTP/2 multiplexing and fixes TCP connection
exhaustion (TIME_WAIT socket accumulation) under high-concurrency workloads.
A new `addressing-style` store option controls this behavior:
- `auto` (default): virtual-hosted-style for standard AWS endpoints, path-style
for custom endpoints.
- `path`: forces path-style addressing (deprecated by AWS).
- `virtual`: forces virtual-hosted-style addressing (bucket names must not
contain dots).
Bucket names containing dots (e.g., `my.bucket.name`) automatically fall back
to path-style addressing in `auto` mode, because dotted names create
multi-level subdomains that break TLS wildcard certificate validation.
Example using path-style for backwards compatibility:
```
s3://my-bucket/key?region=us-east-1&addressing-style=path
```
Additionally, TCP keep-alive is now enabled on all HTTP connections, preventing
idle connections from being silently dropped by intermediate network devices
(NATs, firewalls, load balancers).
+2
View File
@@ -135,7 +135,9 @@
- [Serving Tarball Flakes](protocols/tarball-fetcher.md)
- [Store Path Specification](protocols/store-path.md)
- [Nix Archive (NAR) Format](protocols/nix-archive/index.md)
- [Nix Cache Info Format](protocols/nix-cache-info.md)
- [Derivation "ATerm" file format](protocols/derivation-aterm.md)
- [Nix32 Encoding](protocols/nix32.md)
- [C API](c-api.md)
- [Glossary](glossary.md)
- [Development](development/index.md)
@@ -57,11 +57,6 @@ Most Nix commands interpret the following environment variables:
Overrides the location of the Nix store (default `prefix/store`).
- <span id="env-NIX_DATA_DIR">[`NIX_DATA_DIR`](#env-NIX_DATA_DIR)</span>
Overrides the location of the Nix static data directory (default
`prefix/share`).
- <span id="env-NIX_LOG_DIR">[`NIX_LOG_DIR`](#env-NIX_LOG_DIR)</span>
Overrides the location of the Nix log directory (default
@@ -6,14 +6,23 @@ It is broken up into multiple Meson packages, which are optionally combined in a
There are no mandatory extra steps to the building process:
generic Meson installation instructions like [this](https://mesonbuild.com/Quick-guide.html#using-meson-as-a-distro-packager) should work.
The installation path can be specified by passing the `-Dprefix=prefix`
to `configure`. The default installation directory is `/usr/local`. You
```bash
git clone https://github.com/NixOS/nix.git
cd nix
meson setup build
cd build
ninja
(sudo) ninja install
```
The installation path can be specified by passing `-Dprefix=prefix`
to `meson setup build`. The default installation directory is `/usr/local`. You
can change this to any location you like. You must have write permission
to the *prefix* path.
Nix keeps its *store* (the place where packages are stored) in
`/nix/store` by default. This can be changed using
`-Dstore-dir=path`.
`-Dlibstore:store-dir=path`.
> **Warning**
>
@@ -338,7 +338,7 @@ Here is more information on the `output*` attributes, and what values they may b
This will specify the output hash of the single output of a [fixed-output derivation].
The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The "nix32" encoding is an adaptation of base-32 encoding.
The ["nix32" encoding](@docroot@/protocols/nix32.md) is Nix's variant of base-32 encoding.
> **Note**
>
@@ -19,17 +19,16 @@ whatever port you like:
$ nix-serve -p 8080
```
To check whether it works, try the following on the client:
To check whether it works, try fetching the [`nix-cache-info`](@docroot@/protocols/nix-cache-info.md) file on the client:
```console
$ curl http://avalon:8080/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
```
which should print something like:
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
When writing to a binary cache (e.g., with [`nix copy`](@docroot@/command-ref/new-cli/nix3-copy.md)), Nix creates [`nix-cache-info`](@docroot@/protocols/nix-cache-info.md) automatically if it doesn't exist.
On the client side, you can tell Nix to use your binary cache using
`--substituters`, e.g.:
@@ -24,7 +24,7 @@ description: |
The format follows this pattern: `${digest}-${name}`
- **hash**: Digest rendered in a custom variant of [Base32](https://en.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters)
- **hash**: Digest rendered in [Nix32](@docroot@/protocols/nix32.md), a variant of base-32 (20 hash bytes become 32 ASCII characters)
- **name**: The package name and optional version/suffix information
type: string
@@ -0,0 +1,55 @@
# Nix Cache Info Format
The `nix-cache-info` file is a metadata file at the root of a [binary cache](@docroot@/package-management/binary-cache-substituter.md) (e.g., `https://cache.example.com/nix-cache-info`).
MIME type: `text/x-nix-cache-info`
## Format
Line-based key-value format:
```
Key: value
```
Leading and trailing whitespace is trimmed from values.
Lines without a colon are ignored.
Unknown keys are silently ignored.
## Fields
### `StoreDir`
The Nix store directory path that this cache was built for (e.g., `/nix/store`).
If present, Nix verifies that this matches the client's store directory:
```
error: binary cache 'https://example.com' is for Nix stores with prefix '/nix/store', not '/home/user/nix/store'
```
### `WantMassQuery`
`1` or `0`. Sets the default for [`want-mass-query`](@docroot@/store/types/http-binary-cache-store.md#store-http-binary-cache-store-want-mass-query).
### `Priority`
Integer. Sets the default for [`priority`](@docroot@/store/types/http-binary-cache-store.md#store-http-binary-cache-store-priority).
## Example
```
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
```
## Caching Behavior
Nix caches `nix-cache-info` in the [cache directory](@docroot@/command-ref/env-common.md#env-NIX_CACHE_HOME) with a 7-day TTL.
## See Also
- [HTTP Binary Cache Store](@docroot@/store/types/http-binary-cache-store.md)
- [Serving a Nix store via HTTP](@docroot@/package-management/binary-cache-substituter.md)
- [`substituters`](@docroot@/command-ref/conf-file.md#conf-substituters)
+19
View File
@@ -0,0 +1,19 @@
# Nix32 Encoding
Nix32 is Nix's variant of base-32 encoding, used for [store path digests](@docroot@/protocols/store-path.md), hash output via [`nix hash`](@docroot@/command-ref/new-cli/nix3-hash.md), and the [`outputHash`](@docroot@/language/advanced-attributes.md#adv-attr-outputHash) derivation attribute.
## Alphabet
The Nix32 alphabet consists of these 32 characters:
```
0 1 2 3 4 5 6 7 8 9 a b c d f g h i j k l m n p q r s v w x y z
```
The letters `e`, `o`, `u`, and `t` are omitted.
## Byte Order
Nix32 encoding processes the hash bytes from the end (last byte first), while base-16 encoding processes from the beginning (first byte first).
Consequently, the string sort order is determined primarily by the first bytes for base-16, and by the last bytes for Nix32.
+4 -5
View File
@@ -20,12 +20,11 @@ where
- `store-dir` = the [store directory](@docroot@/store/store-path.md#store-directory)
- `digest` = base-32 representation of the compressed to 160 bits [SHA-256] hash of `fingerprint`
- `digest` = base-32 representation of the compressed to 160 bits [SHA-256] hash of `fingerprint`.
For the definition of the hash compression algorithm, please refer to the section 5.1 of
the [Nix thesis](https://edolstra.github.io/pubs/phd-thesis.pdf), which also defines the
specifics of base-32 encoding. Note that base-32 encoding processes the hash bytestring from
the end, while base-16 processes in from the beginning.
Nix uses a custom base-32 encoding called [Nix32](@docroot@/protocols/nix32.md).
For the definition of the hash compression algorithm, please refer to section 5.1 of the [Nix thesis](https://edolstra.github.io/pubs/phd-thesis.pdf).
## Fingerprint
+1 -1
View File
@@ -31,7 +31,7 @@ A store path is rendered to a file system path as the concatenation of
- [Store directory](#store-directory) (typically `/nix/store`)
- Path separator (`/`)
- Digest rendered in a custom variant of [Base32](https://en.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters)
- Digest rendered in [Nix32](@docroot@/protocols/nix32.md), a variant of base-32 (20 hash bytes become 32 ASCII characters)
- Hyphen (`-`)
- Name
Generated
+10 -10
View File
@@ -3,15 +3,15 @@
"flake-compat": {
"flake": false,
"locked": {
"lastModified": 1733328505,
"narHash": "sha256-NeCCThCEP3eCl2l/+27kNNK7QrwZB1IJCrXfrbv5oqU=",
"owner": "edolstra",
"lastModified": 1767039857,
"narHash": "sha256-vNpUSpF5Nuw8xvDLj2KCwwksIbjua2LZCqhV1LNRDns=",
"owner": "NixOS",
"repo": "flake-compat",
"rev": "ff81ac966bb2cae68946d5ed5fc4994f96d0ffec",
"rev": "5edf11c44bc78a0d334f6334cdaf7d60d732daab",
"type": "github"
},
"original": {
"owner": "edolstra",
"owner": "NixOS",
"repo": "flake-compat",
"type": "github"
}
@@ -63,15 +63,15 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1763948260,
"narHash": "sha256-zZk7fn2ARAqmLwaYTpxBJmj81KIdz11NiWt7ydHHD/M=",
"rev": "1c8ba8d3f7634acac4a2094eef7c32ad9106532c",
"lastModified": 1769089682,
"narHash": "sha256-Xu+7iYcAuOvsI2wdkUcIEmkqEJbvvE6n7qR9QNjJyP4=",
"rev": "078d69f03934859a181e81ba987c2bb033eebfc5",
"type": "tarball",
"url": "https://releases.nixos.org/nixos/25.05/nixos-25.05.813095.1c8ba8d3f763/nixexprs.tar.xz"
"url": "https://releases.nixos.org/nixos/25.11/nixos-25.11.4506.078d69f03934/nixexprs.tar.xz"
},
"original": {
"type": "tarball",
"url": "https://channels.nixos.org/nixos-25.05/nixexprs.tar.xz"
"url": "https://channels.nixos.org/nixos-25.11/nixexprs.tar.xz"
}
},
"nixpkgs-23-11": {
+4 -2
View File
@@ -1,12 +1,12 @@
{
description = "The purely functional package manager";
inputs.nixpkgs.url = "https://channels.nixos.org/nixos-25.05/nixexprs.tar.xz";
inputs.nixpkgs.url = "https://channels.nixos.org/nixos-25.11/nixexprs.tar.xz";
inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.nixpkgs-23-11.url = "github:NixOS/nixpkgs/a62e6edd6d5e1fa0329b8653c801147986f8d446";
inputs.flake-compat = {
url = "github:edolstra/flake-compat";
url = "github:NixOS/flake-compat";
flake = false;
};
@@ -406,6 +406,8 @@
"nix-cmd" = { };
"nix-nswrapper" = { };
"nix-cli" = { };
"nix-everything" = { };
+9 -2
View File
@@ -88,16 +88,23 @@
''^tests/functional/lang/eval-fail-path-slash\.nix$''
''^tests/functional/lang/eval-fail-toJSON-non-utf-8\.nix$''
''^tests/functional/lang/eval-fail-set\.nix$''
# Language tests, don't churn the formatting of strings
''^tests/functional/lang/eval-fail-fromTOML-overflow\.nix$''
''^tests/functional/lang/eval-fail-fromTOML-underflow\.nix$''
''^tests/functional/lang/eval-fail-bad-string-interpolation-3\.nix$''
''^tests/functional/lang/eval-fail-bad-string-interpolation-4\.nix$''
''^tests/functional/lang/eval-okay-regex-match2\.nix$''
];
};
clang-format = {
enable = true;
# https://github.com/cachix/git-hooks.nix/pull/532
package = pkgs.llvmPackages_latest.clang-tools;
package = pkgs.llvmPackages_21.clang-tools;
excludes = [
# We don't want to format test data
# ''tests/(?!nixos/).*\.nix''
''^src/[^/]*-tests/data/.*$''
"^src/[^/]*-tests/data/.*$"
# Don't format vendored code
''^doc/manual/redirects\.js$''
+4
View File
@@ -24,6 +24,10 @@ subproject('libcmd')
# Executables
subproject('nix')
if host_machine.system() == 'linux'
subproject('nswrapper')
endif
# Docs
if get_option('doc-gen')
subproject('internal-api-docs')
+1 -1
View File
@@ -11,7 +11,7 @@ ExecStart=@@bindir@/nix-daemon nix-daemon --daemon
KillMode=process
LimitNOFILE=1048576
TasksMax=1048576
Delegate=yes
Delegate=
[Install]
WantedBy=multi-user.target
@@ -1,7 +1,7 @@
# Clang gets grumpy about missing libasan symbols if -shared-libasan is not
# passed when building shared libs, at least on Linux
if cxx.get_id() == 'clang' and ('address' in get_option('b_sanitize') or 'undefined' in get_option(
'b_sanitize',
'b_sanitize',
))
add_project_link_arguments('-shared-libasan', language : 'cpp')
endif
@@ -22,9 +22,12 @@ add_project_arguments(
'-Werror=undef',
'-Werror=unused-result',
'-Werror=sign-compare',
'-Werror=return-type',
'-Werror=non-virtual-dtor',
'-Wignored-qualifiers',
'-Wimplicit-fallthrough',
'-Wno-deprecated-declarations',
'-Wno-interference-size', # Used for C++ ABI only. We don't provide any guarantees about different march tunings.
language : 'cpp',
)
@@ -40,6 +43,11 @@ do_pch = cxx.get_id() == 'clang'
# instantiations in libutil and libstore.
if cxx.get_id() == 'clang'
add_project_arguments('-fpch-instantiate-templates', language : 'cpp')
# Catch brace elision bugs: when WorkerProto::Version changed from `unsigned int`
# to `struct { unsigned int major; uint8_t minor; }`, `.version = 16` silently
# became `.version = {16, 0}` instead of failing, breaking protocol compatibility
# in a subtle way
add_project_arguments('-Werror=c99-designator', language : 'cpp')
endif
# Detect if we're using libstdc++ (GCC's standard library)
+3
View File
@@ -4,6 +4,7 @@
buildPackages,
cacert,
nix,
nixComponents2,
}:
let
@@ -11,6 +12,7 @@ let
installerClosureInfo = buildPackages.closureInfo {
rootPaths = [
nix
nixComponents2.nix-manual.man
cacert
];
};
@@ -42,6 +44,7 @@ runCommand "nix-binary-tarball-${version}" env ''
--subst-var-by cacert ${cacert}
substitute ${../scripts/install-multi-user.sh} $TMPDIR/install-multi-user \
--subst-var-by nix ${nix} \
--subst-var-by nix-manual ${nixComponents2.nix-manual.man} \
--subst-var-by cacert ${cacert}
if type -p shellcheck; then
+6 -2
View File
@@ -155,12 +155,14 @@ let
];
};
mesonBuildLayer = finalAttrs: prevAttrs: {
mesonBuildLayer = finalAttrs: prevAttrs: rec {
nativeBuildInputs = prevAttrs.nativeBuildInputs or [ ] ++ [
pkg-config
];
separateDebugInfo = !stdenv.hostPlatform.isStatic;
hardeningDisable = lib.optional stdenv.hostPlatform.isStatic "pie";
# needed by separateDebugInfo
# SEE: https://github.com/NixOS/nixpkgs/pull/394674/commits/a4d355342976e9e9823fb94f133bc43ebec9da5b
__structuredAttrs = separateDebugInfo;
};
mesonLibraryLayer = finalAttrs: prevAttrs: {
@@ -416,6 +418,8 @@ in
nix-cmd = callPackage ../src/libcmd/package.nix { };
nix-nswrapper = callPackage ../src/nswrapper/package.nix { };
/**
The Nix command line interface. Note that this does not include its tests, whereas `nix-everything` does.
*/
+6 -26
View File
@@ -30,33 +30,13 @@ scope: {
NIX_CFLAGS_COMPILE = "-DINITIAL_MARK_STACK_SIZE=1048576";
});
lowdown = pkgs.lowdown.overrideAttrs (prevAttrs: rec {
version = "2.0.2";
src = pkgs.fetchurl {
url = "https://kristaps.bsd.lv/lowdown/snapshots/lowdown-${version}.tar.gz";
hash = "sha512-cfzhuF4EnGmLJf5EGSIbWqJItY3npbRSALm+GarZ7SMU7Hr1xw0gtBFMpOdi5PBar4TgtvbnG4oRPh+COINGlA==";
};
nativeBuildInputs = prevAttrs.nativeBuildInputs ++ [ pkgs.buildPackages.bmake ];
postInstall =
lib.replaceStrings [ "lowdown.so.1" "lowdown.1.dylib" ] [ "lowdown.so.2" "lowdown.2.dylib" ]
(prevAttrs.postInstall or "");
patches = [ ];
});
curl = pkgs.curl.override {
http3Support = !pkgs.stdenv.hostPlatform.isWindows;
};
# TODO: Remove this when https://github.com/NixOS/nixpkgs/pull/442682 is included in a stable release
toml11 =
if lib.versionAtLeast pkgs.toml11.version "4.4.0" then
pkgs.toml11
else
pkgs.toml11.overrideAttrs rec {
version = "4.4.0";
src = pkgs.fetchFromGitHub {
owner = "ToruNiina";
repo = "toml11";
tag = "v${version}";
hash = "sha256-sgWKYxNT22nw376ttGsTdg0AMzOwp8QH3E8mx0BZJTQ=";
};
};
libblake3 = pkgs.libblake3.override {
useTBB = !(stdenv.hostPlatform.isWindows || stdenv.hostPlatform.isStatic);
};
# TODO Hack until https://github.com/NixOS/nixpkgs/issues/45462 is fixed.
boost =
+4 -4
View File
@@ -131,7 +131,7 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
ignoreCrossFile = flags: builtins.filter (flag: !(lib.strings.hasInfix "cross-file" flag)) flags;
availableComponents = lib.filterAttrs (
k: v: lib.meta.availableOn pkgs.hostPlatform v
k: v: lib.meta.availableOn pkgs.stdenv.hostPlatform v
) allComponents;
activeComponents = buildInputsClosureCond isInternal (
@@ -300,7 +300,7 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
lib.filter (x: !isInternal x) (
lib.lists.concatMap (
# Nix manual has a build-time dependency on nix, but we
# don't want to do a native build just to enter the ross
# don't want to do a native build just to enter the cross
# dev shell.
#
# TODO: think of a more principled fix for this.
@@ -323,7 +323,7 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
pkgs.buildPackages.shellcheck
pkgs.buildPackages.include-what-you-use
]
++ lib.optional pkgs.hostPlatform.isUnix pkgs.buildPackages.gdb
++ lib.optional stdenv.hostPlatform.isUnix pkgs.buildPackages.gdb
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
)
@@ -341,7 +341,7 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
buildInputs =
# TODO change Nixpkgs to mark gbenchmark as building on Windows
lib.optional pkgs.hostPlatform.isUnix pkgs.gbenchmark
lib.optional stdenv.hostPlatform.isUnix pkgs.gbenchmark
++ dedupByString (v: "${v}") (
lib.filter (x: !isInternal x) (lib.lists.concatMap (c: c.buildInputs) activeComponents)
)
+5
View File
@@ -31,6 +31,8 @@
nix-cmd,
nix-nswrapper,
nix-cli,
nix-functional-tests,
@@ -171,6 +173,9 @@ stdenv.mkDerivation (finalAttrs: {
# Forwarded outputs
ln -sT ${nix-manual} $doc
ln -sT ${nix-manual.man} $man
''
+ lib.optionalString stdenv.isLinux ''
lndir ${nix-nswrapper} $out
'';
passthru = {
+1
View File
@@ -57,6 +57,7 @@ let
"nix-flake"
"nix-flake-c"
"nix-flake-tests"
"nix-nswrapper"
"nix-main"
"nix-main-c"
"nix-cmd"
+3
View File
@@ -52,6 +52,7 @@ readonly PROFILE_FISH_PREFIXES=(
readonly PROFILE_NIX_FILE_FISH="$NIX_ROOT/var/nix/profiles/default/etc/profile.d/nix-daemon.fish"
readonly NIX_INSTALLED_NIX="@nix@"
readonly NIX_INSTALLED_NIX_MAN="@nix-manual@"
readonly NIX_INSTALLED_CACERT="@cacert@"
#readonly NIX_INSTALLED_NIX="/nix/store/byi37zv50wnfrpp4d81z3spswd5zva37-nix-2.3.6"
#readonly NIX_INSTALLED_CACERT="/nix/store/7pi45g541xa8ahwgpbpy7ggsl0xj1jj6-nss-cacert-3.49.2"
@@ -969,6 +970,8 @@ setup_default_profile() {
task "Setting up the default profile"
_sudo "to install a bootstrapping Nix in to the default profile" \
HOME="$ROOT_HOME" "$NIX_INSTALLED_NIX/bin/nix-env" -i "$NIX_INSTALLED_NIX"
_sudo "to install Nix man pages in to the default profile" \
HOME="$ROOT_HOME" "$NIX_INSTALLED_NIX/bin/nix-env" -i "$NIX_INSTALLED_NIX_MAN"
if [ -z "${NIX_SSL_CERT_FILE:-}" ] || ! [ -f "${NIX_SSL_CERT_FILE:-}" ] || cert_in_store; then
_sudo "to install a bootstrapping SSL certificate just for Nix in to the default profile" \
+1
View File
@@ -38,6 +38,7 @@ escape_systemd_env() {
create_systemd_proxy_env() {
vars="http_proxy https_proxy ftp_proxy all_proxy no_proxy HTTP_PROXY HTTPS_PROXY FTP_PROXY ALL_PROXY NO_PROXY"
for v in $vars; do
# shellcheck disable=SC2268
if [ "x${!v:-}" != "x" ]; then
echo "Environment=${v}=$(escape_systemd_env "${!v}")"
fi
+46 -15
View File
@@ -4,6 +4,8 @@
#include "nix/cmd/command.hh"
#include "nix/cmd/legacy.hh"
#include "nix/cmd/markdown.hh"
#include "nix/main/shared.hh"
#include "nix/store/globals.hh"
#include "nix/store/store-open.hh"
#include "nix/store/local-fs-store.hh"
#include "nix/store/derivations.hh"
@@ -63,6 +65,25 @@ void NixMultiCommand::run()
command->second->run();
}
StoreConfigCommand::StoreConfigCommand() {}
ref<StoreConfig> StoreConfigCommand::getStoreConfig()
{
if (!_storeConfig)
_storeConfig = createStoreConfig();
return ref<StoreConfig>(_storeConfig);
}
ref<StoreConfig> StoreConfigCommand::createStoreConfig()
{
return resolveStoreConfig(settings, StoreReference{settings.storeUri.get()});
}
void StoreConfigCommand::run()
{
run(getStoreConfig());
}
StoreCommand::StoreCommand() {}
ref<Store> StoreCommand::getStore()
@@ -74,12 +95,20 @@ ref<Store> StoreCommand::getStore()
ref<Store> StoreCommand::createStore()
{
return openStore();
auto store = getStoreConfig()->openStore();
store->init();
return store;
}
void StoreCommand::run()
void StoreCommand::run(ref<StoreConfig> storeConfig)
{
run(getStore());
// We can either efficiently implement getStore/createStore with memoization,
// or use the StoreConfig passed in run.
// It's more efficient to memoize, especially since there are some direct users
// of getStore. The StoreConfig in both cases should be the same, though.
auto store = getStore();
assert(&*storeConfig == &store->config);
run(std::move(store));
}
CopyCommand::CopyCommand()
@@ -88,28 +117,28 @@ CopyCommand::CopyCommand()
.longName = "from",
.description = "URL of the source Nix store.",
.labels = {"store-uri"},
.handler = {&srcUri},
.handler = {[this](std::string s) { srcUri = StoreReference::parse(s); }},
});
addFlag({
.longName = "to",
.description = "URL of the destination Nix store.",
.labels = {"store-uri"},
.handler = {&dstUri},
.handler = {[this](std::string s) { dstUri = StoreReference::parse(s); }},
});
}
ref<Store> CopyCommand::createStore()
ref<StoreConfig> CopyCommand::createStoreConfig()
{
return srcUri.empty() ? StoreCommand::createStore() : openStore(srcUri);
return !srcUri ? StoreCommand::createStoreConfig() : resolveStoreConfig(settings, StoreReference{*srcUri});
}
ref<Store> CopyCommand::getDstStore()
{
if (srcUri.empty() && dstUri.empty())
if (!srcUri && !dstUri)
throw UsageError("you must pass '--from' and/or '--to'");
return dstUri.empty() ? openStore() : openStore(dstUri);
return !dstUri ? openStore(settings) : openStore(settings, StoreReference{*dstUri});
}
EvalCommand::EvalCommand()
@@ -131,7 +160,7 @@ EvalCommand::~EvalCommand()
ref<Store> EvalCommand::getEvalStore()
{
if (!evalStore)
evalStore = evalStoreUrl ? openStore(*evalStoreUrl) : getStore();
evalStore = evalStoreUrl ? openStore(settings, StoreReference{*evalStoreUrl}) : getStore();
return ref<Store>(evalStore);
}
@@ -257,18 +286,18 @@ MixProfile::MixProfile()
});
}
void MixProfile::updateProfile(const StorePath & storePath)
void MixProfile::updateProfile(Store & store_, const StorePath & storePath)
{
if (!profile)
return;
auto store = getDstStore().dynamic_pointer_cast<LocalFSStore>();
auto * store = dynamic_cast<LocalFSStore *>(&store_);
if (!store)
throw Error("'--profile' is not supported for this Nix store");
auto profile2 = absPath(*profile);
switchLink(profile2, createGeneration(*store, profile2, storePath));
}
void MixProfile::updateProfile(const BuiltPaths & buildables)
void MixProfile::updateProfile(Store & store, const BuiltPaths & buildables)
{
if (!profile)
return;
@@ -292,14 +321,16 @@ void MixProfile::updateProfile(const BuiltPaths & buildables)
throw UsageError(
"'--profile' requires that the arguments produce a single store path, but there are %d", result.size());
updateProfile(result[0]);
updateProfile(store, result[0]);
}
MixDefaultProfile::MixDefaultProfile()
{
profile = getDefaultProfile().string();
profile = getDefaultProfile(settings.getProfileDirsOptions()).string();
}
static constexpr auto environmentVariablesCategory = "Options that change environment variables";
MixEnvironment::MixEnvironment()
: ignoreEnvironment(false)
{
+4 -4
View File
@@ -19,12 +19,12 @@
namespace nix {
fetchers::Settings fetchSettings;
fetchers::Settings fetchSettings{settings};
static GlobalConfig::Register rFetchSettings(&fetchSettings);
EvalSettings evalSettings{
settings.readOnlyMode,
settings,
{
{
"flake",
@@ -135,7 +135,7 @@ MixEvalArgs::MixEvalArgs()
fetchers::overrideRegistry(from.input, to.input, extraAttrs);
}},
.completer = {[&](AddCompletions & completions, size_t, std::string_view prefix) {
completeFlakeRef(completions, openStore(), prefix);
completeFlakeRef(completions, openStore(settings), prefix);
}},
});
@@ -148,7 +148,7 @@ MixEvalArgs::MixEvalArgs()
)",
.category = category,
.labels = {"store-url"},
.handler = {&evalStoreUrl},
.handler = {[this](std::string s) { evalStoreUrl = StoreReference::parse(s); }},
});
}
+31
View File
@@ -0,0 +1,31 @@
#include "nix/cmd/get-build-log.hh"
#include "nix/store/log-store.hh"
#include "nix/store/store-open.hh"
namespace nix {
std::string fetchBuildLog(Settings & settings, ref<Store> store, const StorePath & path, std::string_view what)
{
auto subs = getDefaultSubstituters(settings);
subs.push_front(store);
for (auto & sub : subs) {
auto * logSubP = dynamic_cast<LogStore *>(&*sub);
if (!logSubP) {
printInfo("Skipped '%s' which does not support retrieving build logs", sub->config.getHumanReadableURI());
continue;
}
auto & logSub = *logSubP;
auto log = logSub.getBuildLog(path);
if (!log)
continue;
printInfo("got build log for '%s' from '%s'", what, logSub.config.getHumanReadableURI());
return *log;
}
throw Error("build log of '%s' is not available", what);
}
} // namespace nix
+32 -16
View File
@@ -5,6 +5,7 @@
#include "nix/util/args.hh"
#include "nix/cmd/common-eval-args.hh"
#include "nix/store/path.hh"
#include "nix/store/store-reference.hh"
#include "nix/flake/lockfile.hh"
#include <optional>
@@ -40,28 +41,43 @@ struct NixMultiCommand : MultiCommand, virtual Command
// For the overloaded run methods
#pragma GCC diagnostic ignored "-Woverloaded-virtual"
/**
* A command that requires a \ref StoreConfig store configuration.
*/
struct StoreConfigCommand : virtual Command
{
StoreConfigCommand();
void run() override;
/**
* Return the default Nix store configuration.
*/
ref<StoreConfig> getStoreConfig();
virtual ref<StoreConfig> createStoreConfig();
/**
* Main entry point, with a `StoreConfig` provided
*/
virtual void run(ref<StoreConfig>) = 0;
private:
std::shared_ptr<StoreConfig> _storeConfig;
};
/**
* A command that requires a \ref Store "Nix store".
*/
struct StoreCommand : virtual Command
struct StoreCommand : virtual StoreConfigCommand
{
StoreCommand();
void run() override;
void run(ref<StoreConfig>) override;
/**
* Return the default Nix store.
*/
ref<Store> getStore();
/**
* Return the destination Nix store.
*/
virtual ref<Store> getDstStore()
{
return getStore();
}
virtual ref<Store> createStore();
ref<Store> createStore();
/**
* Main entry point, with a `Store` provided
*/
@@ -77,13 +93,13 @@ private:
*/
struct CopyCommand : virtual StoreCommand
{
std::string srcUri, dstUri;
std::optional<StoreReference> srcUri, dstUri;
CopyCommand();
ref<Store> createStore() override;
ref<StoreConfig> createStoreConfig() override;
ref<Store> getDstStore() override;
ref<Store> getDstStore();
};
/**
@@ -315,11 +331,11 @@ struct MixProfile : virtual StoreCommand
MixProfile();
/* If 'profile' is set, make it point at 'storePath'. */
void updateProfile(const StorePath & storePath);
void updateProfile(Store & store, const StorePath & storePath);
/* If 'profile' is set, make it point at the store path produced
by 'buildables'. */
void updateProfile(const BuiltPaths & buildables);
void updateProfile(Store & store, const BuiltPaths & buildables);
};
struct MixDefaultProfile : MixProfile
@@ -6,6 +6,7 @@
#include "nix/main/common-args.hh"
#include "nix/expr/search-path.hh"
#include "nix/expr/eval-settings.hh"
#include "nix/store/store-reference.hh"
#include <filesystem>
@@ -55,7 +56,7 @@ struct MixEvalArgs : virtual Args, virtual MixRepair
LookupPath lookupPath;
std::optional<std::string> evalStoreUrl;
std::optional<StoreReference> evalStoreUrl;
private:
struct AutoArgExpr
@@ -0,0 +1,25 @@
#pragma once
///@file
#include "nix/store/store-api.hh"
#include <string>
#include <string_view>
namespace nix {
class Settings;
/**
* Fetch the build log for a store path, searching the store and its
* substituters.
*
* @param store The store to search (and its substituters).
* @param path The store path to get the build log for.
* @param what A description of what we're fetching the log for (used in messages).
* @return The build log content.
* @throws Error if the build log is not available.
*/
std::string fetchBuildLog(Settings & settings, ref<Store> store, const StorePath & path, std::string_view what);
} // namespace nix
@@ -1,7 +1,6 @@
#pragma once
///@file
#include "nix/store/globals.hh"
#include "nix/cmd/installable-value.hh"
#include "nix/store/outputs-spec.hh"
#include "nix/cmd/command.hh"
+2
View File
@@ -9,6 +9,7 @@ headers = files(
'common-eval-args.hh',
'compatibility-settings.hh',
'editor-for.hh',
'get-build-log.hh',
'installable-attr-path.hh',
'installable-derived-path.hh',
'installable-flake.hh',
@@ -20,4 +21,5 @@ headers = files(
'network-proxy.hh',
'repl-interacter.hh',
'repl.hh',
'unix-socket-server.hh',
)
@@ -14,6 +14,7 @@ namespace detail {
struct ReplCompleterMixin
{
virtual StringSet completePrefix(const std::string & prefix) = 0;
virtual ~ReplCompleterMixin() = default;
};
}; // namespace detail
+1 -2
View File
@@ -25,7 +25,7 @@ struct AbstractNixRepl
* @todo this is a layer violation
*
* @param programName Name of the command, e.g. `nix` or `nix-env`.
* @param args aguments to the command.
* @param args arguments to the command.
*/
using RunNix =
void(const std::string & programName, const Strings & args, const std::optional<std::string> & input);
@@ -37,7 +37,6 @@ struct AbstractNixRepl
*/
static std::unique_ptr<AbstractNixRepl> create(
const LookupPath & lookupPath,
nix::ref<Store> store,
ref<EvalState> state,
std::function<AnnotatedValues()> getValues,
RunNix * runNix = nullptr);
@@ -0,0 +1,79 @@
#pragma once
///@file
#include "nix/util/file-descriptor.hh"
#include <filesystem>
#include <functional>
#include <optional>
#include <sys/types.h>
namespace nix::unix {
/**
* Information about the identity of the peer on a Unix domain socket connection.
*/
struct PeerInfo
{
std::optional<pid_t> pid;
std::optional<uid_t> uid;
std::optional<gid_t> gid;
};
/**
* Get the identity of the caller, if possible.
*/
PeerInfo getPeerInfo(Descriptor remote);
/**
* Callback type for handling new connections.
*
* The callback receives ownership of the connection and is responsible
* for handling it (e.g., forking a child process, spawning a thread, etc.).
*
* @param socket The accepted connection file descriptor.
* @param closeListeners A callback to close the listening sockets.
* Useful in forked child processes to release the bound sockets.
*/
using UnixSocketHandler = std::function<void(AutoCloseFD socket, std::function<void()> closeListeners)>;
/**
* Options for the serve loop.
*
* Only used if no systemd socket activation is detected.
*/
struct ServeUnixSocketOptions
{
/**
* The Unix domain socket path to create and listen on.
*/
std::filesystem::path socketPath;
/**
* Mode for the created socket file.
*/
mode_t socketMode = 0666;
};
/**
* Run a server loop that accepts connections and calls the handler for each.
*
* This function handles:
* - systemd socket activation (via LISTEN_FDS environment variable)
* - Creating and binding a Unix domain socket if no activation is detected
* - Polling for incoming connections
* - Accepting connections
*
* For each accepted connection, the handler is called with the connection
* file descriptor. The handler takes ownership of the file descriptor and
* is responsible for closing it when done.
*
* This function never returns normally. It runs until interrupted
* (e.g., via SIGINT), at which point it throws `Interrupted`.
*
* @param options Configuration for the server.
* @param handler Callback invoked for each accepted connection.
*/
[[noreturn]] void serveUnixSocket(const ServeUnixSocketOptions & options, UnixSocketHandler handler);
} // namespace nix::unix
+4 -2
View File
@@ -203,8 +203,10 @@ FlakeRef InstallableFlake::nixpkgsFlakeRef() const
if (auto nixpkgsInput = lockedFlake->lockFile.findInput({"nixpkgs"})) {
if (auto lockedNode = std::dynamic_pointer_cast<const flake::LockedNode>(nixpkgsInput)) {
debug("using nixpkgs flake '%s'", lockedNode->lockedRef);
return std::move(lockedNode->lockedRef);
if (lockedNode->isFlake) {
debug("using nixpkgs flake '%s'", lockedNode->lockedRef);
return std::move(lockedNode->lockedRef);
}
}
}
+3 -7
View File
@@ -583,16 +583,12 @@ static void throwBuildErrors(std::vector<KeyedBuildResult> & buildResults, const
auto failedResult = failed.begin();
if (failedResult != failed.end()) {
if (failed.size() == 1) {
failedResult->second->rethrow();
throw *failedResult->second;
} else {
StringSet failedPaths;
for (; failedResult != failed.end(); failedResult++) {
if (!failedResult->second->errorMsg.empty()) {
logError(
ErrorInfo{
.level = lvlError,
.msg = failedResult->second->errorMsg,
});
if (!failedResult->second->message().empty()) {
logError(failedResult->second->info());
}
failedPaths.insert(failedResult->first->path.to_string(store));
}
+7
View File
@@ -74,6 +74,7 @@ sources = files(
'command.cc',
'common-eval-args.cc',
'editor-for.cc',
'get-build-log.cc',
'installable-attr-path.cc',
'installable-derived-path.cc',
'installable-flake.cc',
@@ -86,6 +87,12 @@ sources = files(
'repl.cc',
)
if host_machine.system() != 'windows'
sources += files(
'unix/unix-socket-server.cc',
)
endif
subdir('include/nix/cmd')
subdir('nix-meson-build-support/export-all-symbols')
+19 -41
View File
@@ -12,9 +12,8 @@
#include "nix/expr/eval-settings.hh"
#include "nix/expr/attr-path.hh"
#include "nix/util/signals.hh"
#include "nix/store/store-open.hh"
#include "nix/store/log-store.hh"
#include "nix/cmd/common-eval-args.hh"
#include "nix/cmd/get-build-log.hh"
#include "nix/expr/get-drvs.hh"
#include "nix/store/derivations.hh"
#include "nix/store/globals.hh"
@@ -65,7 +64,7 @@ struct NixRepl : AbstractNixRepl, detail::ReplCompleterMixin, gc
const static int envSize = 32768;
std::shared_ptr<StaticEnv> staticEnv;
Value lastLoaded;
std::optional<Value> lastLoaded;
Env * env;
int displ;
StringSet varNames;
@@ -78,7 +77,6 @@ struct NixRepl : AbstractNixRepl, detail::ReplCompleterMixin, gc
NixRepl(
const LookupPath & lookupPath,
nix::ref<Store> store,
ref<EvalState> state,
std::function<AnnotatedValues()> getValues,
RunNix * runNix);
@@ -133,7 +131,6 @@ std::string removeWhitespace(std::string s)
NixRepl::NixRepl(
const LookupPath & lookupPath,
nix::ref<Store> store,
ref<EvalState> state,
std::function<NixRepl::AnnotatedValues()> getValues,
RunNix * runNix)
@@ -566,31 +563,9 @@ ProcessLineResult NixRepl::processLine(std::string line)
} else if (command == ":log") {
settings.readOnlyMode = true;
Finally roModeReset([&]() { settings.readOnlyMode = false; });
auto subs = getDefaultSubstituters();
subs.push_front(state->store);
bool foundLog = false;
RunPager pager;
for (auto & sub : subs) {
auto * logSubP = dynamic_cast<LogStore *>(&*sub);
if (!logSubP) {
printInfo(
"Skipped '%s' which does not support retrieving build logs", sub->config.getHumanReadableURI());
continue;
}
auto & logSub = *logSubP;
auto log = logSub.getBuildLog(drvPath);
if (log) {
printInfo("got build log for '%s' from '%s'", drvPathRaw, logSub.config.getHumanReadableURI());
logger->writeToStdout(*log);
foundLog = true;
break;
}
}
if (!foundLog)
throw Error("build log of '%s' is not available", drvPathRaw);
auto log = fetchBuildLog(settings, state->store, drvPath, drvPathRaw);
logger->writeToStdout(log);
} else {
runNix("nix-shell", {drvPathRaw});
}
@@ -773,11 +748,19 @@ void NixRepl::initEnv()
void NixRepl::showLastLoaded()
{
RunPager pager;
if (!lastLoaded)
throw Error("nothing has been loaded yet");
for (auto & i : *lastLoaded.attrs()) {
std::string_view name = state->symbols[i.name];
logger->cout(name);
RunPager pager;
try {
for (auto & i : *lastLoaded->attrs()) {
std::string_view name = state->symbols[i.name];
logger->cout(name);
}
} catch (SystemError & e) {
/* Ignore broken pipes when the pager gets interrupted. */
if (!e.is(std::errc::broken_pipe))
throw;
}
}
@@ -795,7 +778,7 @@ void NixRepl::loadFiles()
loadedFiles.clear();
for (auto & i : old) {
notice("Loading '%1%'...", i);
notice("Loading %1%...", PathFmt(i));
loadFile(i);
}
@@ -900,13 +883,9 @@ void NixRepl::runNix(const std::string & program, const Strings & args, const st
}
std::unique_ptr<AbstractNixRepl> AbstractNixRepl::create(
const LookupPath & lookupPath,
nix::ref<Store> store,
ref<EvalState> state,
std::function<AnnotatedValues()> getValues,
RunNix * runNix)
const LookupPath & lookupPath, ref<EvalState> state, std::function<AnnotatedValues()> getValues, RunNix * runNix)
{
return std::make_unique<NixRepl>(lookupPath, std::move(store), state, getValues, runNix);
return std::make_unique<NixRepl>(lookupPath, state, getValues, runNix);
}
ReplExitStatus AbstractNixRepl::runSimple(ref<EvalState> evalState, const ValMap & extraEnv)
@@ -919,7 +898,6 @@ ReplExitStatus AbstractNixRepl::runSimple(ref<EvalState> evalState, const ValMap
// NOLINTNEXTLINE(clang-analyzer-cplusplus.NewDelete)
auto repl = std::make_unique<NixRepl>(
lookupPath,
openStore(),
evalState,
getValues,
/*runNix=*/nullptr);
+126
View File
@@ -0,0 +1,126 @@
///@file
#include "nix/cmd/unix-socket-server.hh"
#include "nix/util/environment-variables.hh"
#include "nix/util/file-system.hh"
#include "nix/util/logging.hh"
#include "nix/util/signals.hh"
#include "nix/util/strings.hh"
#include "nix/util/unix-domain-socket.hh"
#include "nix/util/util.hh"
#include <sys/socket.h>
#include <sys/un.h>
#include <poll.h>
#if defined(__APPLE__) || defined(__FreeBSD__)
# include <sys/ucred.h>
#endif
namespace nix::unix {
PeerInfo getPeerInfo(Descriptor remote)
{
PeerInfo peer;
#if defined(SO_PEERCRED)
# if defined(__OpenBSD__)
struct sockpeercred cred;
# else
ucred cred;
# endif
socklen_t credLen = sizeof(cred);
if (getsockopt(remote, SOL_SOCKET, SO_PEERCRED, &cred, &credLen) == 0) {
peer.pid = cred.pid;
peer.uid = cred.uid;
peer.gid = cred.gid;
}
#elif defined(LOCAL_PEERCRED)
# if !defined(SOL_LOCAL)
# define SOL_LOCAL 0
# endif
xucred cred;
socklen_t credLen = sizeof(cred);
if (getsockopt(remote, SOL_LOCAL, LOCAL_PEERCRED, &cred, &credLen) == 0)
peer.uid = cred.cr_uid;
#endif
return peer;
}
[[noreturn]] void serveUnixSocket(const ServeUnixSocketOptions & options, UnixSocketHandler handler)
{
std::vector<AutoCloseFD> listeningSockets;
static constexpr int SD_LISTEN_FDS_START = 3;
// Handle socket-based activation by systemd.
auto listenFds = getEnv("LISTEN_FDS");
if (listenFds) {
if (getEnv("LISTEN_PID") != std::to_string(getpid()))
throw Error("unexpected systemd environment variables");
auto count = string2Int<unsigned int>(*listenFds);
assert(count);
for (unsigned int i = 0; i < count; ++i) {
AutoCloseFD fdSocket(SD_LISTEN_FDS_START + i);
closeOnExec(fdSocket.get());
listeningSockets.push_back(std::move(fdSocket));
}
}
// Otherwise, create and bind to a Unix domain socket.
else {
createDirs(options.socketPath.parent_path());
listeningSockets.push_back(createUnixDomainSocket(options.socketPath.string(), options.socketMode));
}
std::vector<struct pollfd> fds;
for (auto & i : listeningSockets)
fds.push_back({.fd = i.get(), .events = POLLIN});
// Loop accepting connections.
while (1) {
try {
checkInterrupt();
auto count = poll(fds.data(), fds.size(), -1);
if (count == -1) {
if (errno == EINTR)
continue;
throw SysError("polling for incoming connections");
}
for (auto & fd : fds) {
if (!fd.revents)
continue;
// Accept a connection.
struct sockaddr_un remoteAddr;
socklen_t remoteAddrLen = sizeof(remoteAddr);
AutoCloseFD remote = accept(fd.fd, (struct sockaddr *) &remoteAddr, &remoteAddrLen);
checkInterrupt();
if (!remote) {
if (errno == EINTR)
continue;
throw SysError("accepting connection");
}
handler(std::move(remote), [&]() { listeningSockets.clear(); });
}
} catch (Error & error) {
auto ei = error.info();
// FIXME: add to trace?
ei.msg = HintFmt("while processing connection: %1%", ei.msg.str());
logError(ei);
}
}
}
} // namespace nix::unix
+3 -5
View File
@@ -132,9 +132,8 @@ nix_eval_state_builder * nix_eval_state_builder_new(nix_c_context * context, Sto
return unsafe_new_with_self<nix_eval_state_builder>([&](auto * self) {
return nix_eval_state_builder{
.store = nix::ref<nix::Store>(store->ptr),
.settings = nix::EvalSettings{/* &bool */ self->readOnlyMode},
.fetchSettings = nix::fetchers::Settings{},
.readOnlyMode = true,
.settings = nix::EvalSettings{cStoreSettings},
.fetchSettings = nix::fetchers::Settings{cStoreSettings},
};
});
}
@@ -153,8 +152,7 @@ nix_err nix_eval_state_builder_load(nix_c_context * context, nix_eval_state_buil
if (context)
context->last_err_code = NIX_OK;
try {
// TODO: load in one go?
builder->settings.readOnlyMode = nix::settings.readOnlyMode;
loadConfFile(cStoreSettings);
loadConfFile(builder->settings);
loadConfFile(builder->fetchSettings);
}
-2
View File
@@ -16,8 +16,6 @@ struct nix_eval_state_builder
nix::EvalSettings settings;
nix::fetchers::Settings fetchSettings;
nix::LookupPath lookupPath;
// TODO: make an EvalSettings setting own this instead?
bool readOnlyMode;
};
struct EvalState
@@ -26,18 +26,18 @@ public:
}
protected:
LibExprTest(ref<Store> store, auto && makeEvalSettings)
: LibStoreTest()
, evalSettings(makeEvalSettings(readOnlyMode))
LibExprTest(auto && makeEvalSettings, auto &&... args)
: LibStoreTest(args...)
, evalSettings(makeEvalSettings(settings))
, state({}, store, fetchSettings, evalSettings, nullptr)
{
}
LibExprTest()
: LibExprTest(openStore("dummy://"), [](bool & readOnlyMode) {
EvalSettings settings{readOnlyMode};
settings.nixPath = {};
return settings;
: LibExprTest([](Settings & settings) {
EvalSettings evalSettings{settings};
evalSettings.nixPath = {};
return evalSettings;
})
{
}
@@ -66,8 +66,8 @@ protected:
}
bool readOnlyMode = true;
fetchers::Settings fetchSettings{};
EvalSettings evalSettings{readOnlyMode};
fetchers::Settings fetchSettings{settings};
EvalSettings evalSettings{settings};
EvalState state;
};
+9 -6
View File
@@ -179,12 +179,15 @@ class PureEvalTest : public LibExprTest
{
public:
PureEvalTest()
: LibExprTest(openStore("dummy://", {{"read-only", "false"}}), [](bool & readOnlyMode) {
EvalSettings settings{readOnlyMode};
settings.pureEval = true;
settings.restrictEval = true;
return settings;
})
: LibExprTest{
[](auto & settings) {
EvalSettings evalSettings{settings};
evalSettings.pureEval = true;
evalSettings.restrictEval = true;
return evalSettings;
},
[](auto & settings) { return openStore(settings, "dummy://", {{"read-only", "false"}}); },
}
{
}
};
+1 -1
View File
@@ -756,7 +756,7 @@ TEST_F(PrimOpTest, langVersion)
TEST_F(PrimOpTest, storeDir)
{
auto v = eval("builtins.storeDir");
ASSERT_THAT(v, IsStringEq(settings.nixStore));
ASSERT_THAT(v, IsStringEq(state.store->storeDir));
}
TEST_F(PrimOpTest, nixVersion)
+8 -6
View File
@@ -63,7 +63,7 @@ struct AttrDb
SymbolTable & symbols;
AttrDb(const StoreDirConfig & cfg, const Hash & fingerprint, SymbolTable & symbols)
AttrDb(bool useSQLiteWAL, const StoreDirConfig & cfg, const Hash & fingerprint, SymbolTable & symbols)
: cfg(cfg)
, _state(std::make_unique<Sync<State>>())
, symbols(symbols)
@@ -75,7 +75,7 @@ struct AttrDb
auto dbPath = cacheDir / (fingerprint.to_string(HashFormat::Base16, false) + ".sqlite");
state->db = SQLite(dbPath);
state->db = SQLite(dbPath, {.useWAL = useSQLiteWAL});
state->db.isCache();
state->db.exec(schema);
@@ -287,10 +287,11 @@ struct AttrDb
}
};
static std::shared_ptr<AttrDb> makeAttrDb(const StoreDirConfig & cfg, const Hash & fingerprint, SymbolTable & symbols)
static std::shared_ptr<AttrDb>
makeAttrDb(bool useSQLiteWAL, const StoreDirConfig & cfg, const Hash & fingerprint, SymbolTable & symbols)
{
try {
return std::make_shared<AttrDb>(cfg, fingerprint, symbols);
return std::make_shared<AttrDb>(useSQLiteWAL, cfg, fingerprint, symbols);
} catch (SQLiteError &) {
ignoreExceptionExceptInterrupt();
return nullptr;
@@ -299,7 +300,8 @@ static std::shared_ptr<AttrDb> makeAttrDb(const StoreDirConfig & cfg, const Hash
EvalCache::EvalCache(
std::optional<std::reference_wrapper<const Hash>> useCache, EvalState & state, RootLoader rootLoader)
: db(useCache ? makeAttrDb(*state.store, *useCache, state.symbols) : nullptr)
: db(useCache ? makeAttrDb(state.store->config.settings.useSQLiteWAL, *state.store, *useCache, state.symbols)
: nullptr)
, state(state)
, rootLoader(rootLoader)
{
@@ -707,7 +709,7 @@ StorePath AttrCursor::forceDerivation()
auto aDrvPath = getAttr(root->state.s.drvPath);
auto drvPath = root->state.store->parseStorePath(aDrvPath->getString());
drvPath.requireDerivation();
if (!root->state.store->isValidPath(drvPath) && !settings.readOnlyMode) {
if (!root->state.store->isValidPath(drvPath) && !root->state.store->config.settings.readOnlyMode) {
/* The eval cache contains 'drvPath', but the actual path has
been garbage-collected. So force it to be regenerated. */
aDrvPath->forceValue();
+9 -3
View File
@@ -133,13 +133,19 @@ class SampleStack : public EvalProfiler
FrameInfo getPrimOpFrameInfo(const PrimOp & primOp, std::span<Value *> args, PosIdx pos);
public:
SampleStack(EvalState & state, std::filesystem::path profileFile, std::chrono::nanoseconds period)
SampleStack(EvalState & state, const std::filesystem::path & profileFile, std::chrono::nanoseconds period)
: state(state)
, sampleInterval(period)
, profileFd([&]() {
AutoCloseFD fd = toDescriptor(open(profileFile.string().c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0660));
AutoCloseFD fd = openNewFileForWrite(
profileFile,
0660,
{
.truncateExisting = true,
.followSymlinksOnTruncate = true, /* FIXME: Probably shouldn't follow symlinks. */
});
if (!fd)
throw SysError("opening file %s", profileFile);
throw SysError("opening file %s", PathFmt(profileFile));
return fd;
}())
, posCache(state)
+9 -8
View File
@@ -48,8 +48,8 @@ Strings EvalSettings::parseNixPath(const std::string & s)
return res;
}
EvalSettings::EvalSettings(bool & readOnlyMode, EvalSettings::LookupPathHooks lookupPathHooks)
: readOnlyMode{readOnlyMode}
EvalSettings::EvalSettings(nix::Settings & storeSettings, EvalSettings::LookupPathHooks lookupPathHooks)
: storeSettings{storeSettings}
, lookupPathHooks{lookupPathHooks}
{
auto var = getEnv("NIX_ABORT_ON_WARN");
@@ -57,7 +57,7 @@ EvalSettings::EvalSettings(bool & readOnlyMode, EvalSettings::LookupPathHooks lo
builtinsAbortOnWarn = true;
}
Strings EvalSettings::getDefaultNixPath()
Strings EvalSettings::getDefaultNixPath(nix::Settings & settings)
{
Strings res;
auto add = [&](const std::filesystem::path & p, const std::string & s = std::string()) {
@@ -70,9 +70,10 @@ Strings EvalSettings::getDefaultNixPath()
}
};
add(std::filesystem::path{getNixDefExpr()} / "channels");
add(rootChannelsDir() / "nixpkgs", "nixpkgs");
add(rootChannelsDir());
add(std::filesystem::path{getNixDefExpr(settings)} / "channels");
auto profilesDirOpts = settings.getProfileDirsOptions();
add(rootChannelsDir(profilesDirOpts) / "nixpkgs", "nixpkgs");
add(rootChannelsDir(profilesDirOpts));
return res;
}
@@ -100,10 +101,10 @@ std::string EvalSettings::resolvePseudoUrl(std::string_view url)
const std::string & EvalSettings::getCurrentSystem() const
{
const auto & evalSystem = currentSystem.get();
return evalSystem != "" ? evalSystem : settings.thisSystem.get();
return evalSystem != "" ? evalSystem : storeSettings.thisSystem.get();
}
std::filesystem::path getNixDefExpr()
std::filesystem::path getNixDefExpr(const Settings & settings)
{
return settings.useXDGBaseDirectories ? getStateDir() / "defexpr" : getHome() / ".nix-defexpr";
}
+3 -3
View File
@@ -15,6 +15,7 @@
#include "nix/store/filetransfer.hh"
#include "nix/expr/function-trace.hh"
#include "nix/store/profiles.hh"
#include "nix/store/globals.hh"
#include "nix/expr/print.hh"
#include "nix/fetchers/filtering-source-accessor.hh"
#include "nix/util/memory-source-accessor.hh"
@@ -318,7 +319,6 @@ EvalState::EvalState(
countCalls = getEnv("NIX_COUNT_CALLS").value_or("0") != "0";
static_assert(sizeof(Env) <= 16, "environment must be <= 16 bytes");
static_assert(sizeof(Counter) == 64, "counters must be 64 bytes");
/* Construct the Nix expression search path. */
assert(lookupPath.elements.empty());
@@ -331,7 +331,7 @@ EvalState::EvalState(
lookupPath.elements.emplace_back(LookupPath::Elem::parse(i));
}
if (!settings.restrictEval) {
for (auto & i : EvalSettings::getDefaultNixPath()) {
for (auto & i : EvalSettings::getDefaultNixPath(store->config.settings)) {
lookupPath.elements.emplace_back(LookupPath::Elem::parse(i));
}
}
@@ -2509,7 +2509,7 @@ StorePath EvalState::copyPathToStore(NixStringContext & context, const SourcePat
fetchSettings,
*store,
path.resolveSymlinks(SymlinkResolution::Ancestors),
settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
settings.storeSettings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
path.baseName(),
ContentAddressMethod::Raw::NixArchive,
nullptr,
+1 -1
View File
@@ -11,7 +11,7 @@ namespace nix {
* variable is set. This is to prevent contention on these counters
* when multi-threaded evaluation is enabled.
*/
struct alignas(64) Counter
struct alignas(std::hardware_destructive_interference_size) Counter
{
using value_type = uint64_t;
@@ -9,9 +9,15 @@ namespace nix {
class EvalState;
struct PrimOp;
class Settings;
struct EvalSettings : Config
{
/**
* Reference to the "parent" store-layer settings.
*/
nix::Settings & storeSettings;
/**
* Function used to interpret look path entries of a given scheme.
*
@@ -38,11 +44,9 @@ struct EvalSettings : Config
*/
using LookupPathHooks = std::map<std::string, std::function<LookupPathHook>>;
EvalSettings(bool & readOnlyMode, LookupPathHooks lookupPathHooks = {});
EvalSettings(nix::Settings & settings, LookupPathHooks lookupPathHooks = {});
bool & readOnlyMode;
static Strings getDefaultNixPath();
static Strings getDefaultNixPath(nix::Settings & settings);
static bool isPseudoUrl(std::string_view s);
@@ -366,6 +370,6 @@ struct EvalSettings : Config
/**
* Conventionally part of the default nix path in impure mode.
*/
std::filesystem::path getNixDefExpr();
std::filesystem::path getNixDefExpr(const Settings & settings);
} // namespace nix
+6
View File
@@ -1002,6 +1002,12 @@ public:
[[nodiscard]] StringMap
realiseContext(const NixStringContext & context, StorePathSet * maybePaths = nullptr, bool isIFD = true);
/**
* Coerce `v` to a path and realise it, i.e. build anything in the value's string context using `realiseContext()`.
*/
SourcePath realisePath(
const PosIdx pos, Value & v, std::optional<SymlinkResolution> resolveSymlinks = SymlinkResolution::Full);
/**
* Realise the given string with context, and return the string with outputs instead of downstream output
* placeholders.
+4 -4
View File
@@ -134,7 +134,7 @@ public:
virtual bool operator==(const ExternalValueBase & b) const noexcept;
/**
* Print the value as JSON. Defaults to unconvertable, i.e. throws an error
* Print the value as JSON. Defaults to unconvertible, i.e. throws an error
*/
virtual nlohmann::json
printValueAsJSON(EvalState & state, bool strict, NixStringContext & context, bool copyToStore = true) const;
@@ -533,8 +533,8 @@ inline constexpr bool useBitPackedValueStorage = (ptrSize == 8) && (__STDCPP_DEF
* Packs discriminator bits into the pointer alignment niches.
*/
template<std::size_t ptrSize>
class alignas(16) ValueStorage<ptrSize, std::enable_if_t<detail::useBitPackedValueStorage<ptrSize>>>
: public detail::ValueBase
class alignas(16)
ValueStorage<ptrSize, std::enable_if_t<detail::useBitPackedValueStorage<ptrSize>>> : public detail::ValueBase
{
/* Needs a dependent type name in order for member functions (and
* potentially ill-formed bit casts) to be SFINAE'd out.
@@ -587,7 +587,7 @@ class alignas(16) ValueStorage<ptrSize, std::enable_if_t<detail::useBitPackedVal
enum PrimaryDiscriminator : int {
pdUninitialized = 0,
pdSingleDWord, //< layout: Single/zero field payload
/* The order of these enumations must be the same as in InternalType. */
/* The order of these enumerations must be the same as in InternalType. */
pdListN, //< layout: Single untaggable field.
pdString,
pdPath,
+36 -13
View File
@@ -3,7 +3,7 @@
%define api.namespace { ::nix::parser }
%define api.parser.class { BisonParser }
%locations
%define parse.error verbose
%define parse.error detailed
%defines
/* %no-lines */
%parse-param { void * scanner }
@@ -140,18 +140,41 @@ static Expr * makeCall(Exprs & exprs, PosIdx pos, Expr * fn, Expr * arg) {
%type <Expr *> path_start
%type <ToBeStringyExpr> string_parts string_attr
%type <StringToken> attr
%token <StringToken> ID
%token <StringToken> STR IND_STR
%token <NixInt> INT_LIT
%token <NixFloat> FLOAT_LIT
%token <StringToken> PATH HPATH SPATH PATH_END
%token <StringToken> URI
%token IF THEN ELSE ASSERT WITH LET IN_KW REC INHERIT EQ NEQ AND OR IMPL OR_KW
%token PIPE_FROM PIPE_INTO /* <| and |> */
%token DOLLAR_CURLY /* == ${ */
%token IND_STRING_OPEN IND_STRING_CLOSE
%token ELLIPSIS
%token <StringToken> ID "identifier"
%token <StringToken> STR "string"
%token <StringToken> IND_STR "indented string"
%token <NixInt> INT_LIT "integer"
%token <NixFloat> FLOAT_LIT "floating-point literal"
%token <StringToken> PATH "path"
%token <StringToken> HPATH "'~/…' path"
%token <StringToken> SPATH "'<…>' path"
%token <StringToken> PATH_END "end of path"
%token <StringToken> URI "URI"
%token IF "'if'"
%token THEN "'then'"
%token ELSE "'else'"
%token ASSERT "'assert'"
%token WITH "'with'"
%token LET "'let'"
%token IN_KW "'in'"
%token REC "'rec'"
%token INHERIT "'inherit'"
%token EQ "'=='"
%token NEQ "'!='"
%token LEQ "'<='"
%token GEQ "'>='"
%token UPDATE "'//'"
%token CONCAT "'++'"
%token AND "'&&'"
%token OR "'||'"
%token IMPL "'->'"
%token OR_KW "'or'"
%token PIPE_FROM "'<|'"
%token PIPE_INTO "'|>'"
%token DOLLAR_CURLY "'${'"
%token IND_STRING_OPEN "start of an indented string"
%token IND_STRING_CLOSE "end of an indented string"
%token ELLIPSIS "'...'"
%right IMPL
%left OR
+1 -2
View File
@@ -23,13 +23,12 @@ SourcePath EvalState::storePath(const StorePath & path)
StorePath
EvalState::mountInput(fetchers::Input & input, const fetchers::Input & originalInput, ref<SourceAccessor> accessor)
{
auto storePath = fetchToStore(fetchSettings, *store, accessor, FetchMode::Copy, input.getName());
auto [storePath, narHash] = fetchToStore2(fetchSettings, *store, accessor, FetchMode::Copy, input.getName());
allowPath(storePath); // FIXME: should just whitelist the entire virtual store
storeFS->mount(CanonPath(store->printStorePath(storePath)), accessor);
auto narHash = store->queryPathInfo(storePath)->narHash;
input.attrs.insert_or_assign("narHash", narHash.to_string(HashFormat::SRI, true));
if (originalInput.getNarHash() && narHash != *originalInput.getNarHash())
+56 -53
View File
@@ -157,24 +157,20 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
return res;
}
static SourcePath realisePath(
EvalState & state,
const PosIdx pos,
Value & v,
std::optional<SymlinkResolution> resolveSymlinks = SymlinkResolution::Full)
SourcePath EvalState::realisePath(const PosIdx pos, Value & v, std::optional<SymlinkResolution> resolveSymlinks)
{
NixStringContext context;
auto path = state.coerceToPath(noPos, v, context, "while realising the context of a path");
auto path = coerceToPath(noPos, v, context, "while realising the context of a path");
try {
if (!context.empty() && path.accessor == state.rootFS) {
auto rewrites = state.realiseContext(context);
if (!context.empty() && path.accessor == rootFS) {
auto rewrites = realiseContext(context);
path = {path.accessor, CanonPath(rewriteStrings(path.path.abs(), rewrites))};
}
return resolveSymlinks ? path.resolveSymlinks(*resolveSymlinks) : path;
} catch (Error & e) {
e.addTrace(state.positions[pos], "while realising the context of path '%s'", path);
e.addTrace(positions[pos], "while realising the context of path '%s'", path);
throw;
}
}
@@ -294,7 +290,7 @@ static void scopedImport(EvalState & state, const PosIdx pos, SourcePath & path,
argument. */
static void import(EvalState & state, const PosIdx pos, Value & vPath, Value * vScope, Value & v)
{
auto path = realisePath(state, pos, vPath, std::nullopt);
auto path = state.realisePath(pos, vPath, std::nullopt);
auto path2 = path.path.abs();
// FIXME
@@ -447,7 +443,7 @@ extern "C" typedef void (*ValueInitializer)(EvalState & state, Value & v);
/* Load a ValueInitializer from a DSO and return whatever it initializes */
void prim_importNative(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0]);
auto path = state.realisePath(pos, *args[0]);
std::string sym(
state.forceStringNoCtx(*args[1], pos, "while evaluating the second argument passed to builtins.importNative"));
@@ -1818,8 +1814,13 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
drv.fillInOutputPaths(*state.store);
}
/* Write the resulting term into the Nix store directory. */
auto drvPath = writeDerivation(*state.store, drv, state.repair);
/* Write the resulting term into the Nix store directory.
Unless we are in read-only mode, that is, in which case we do not
write anything. Users commonly do this to speed up evaluation in
contexts where they don't actually want to build anything. */
auto drvPath = state.settings.storeSettings.readOnlyMode ? computeStorePath(*state.store, drv)
: state.store->writeDerivation(drv, state.repair);
auto drvPathS = state.store->printStorePath(drvPath);
printMsg(lvlChatty, "instantiated '%1%' -> '%2%'", drvName, drvPathS);
@@ -1934,7 +1935,7 @@ static void prim_storePath(EvalState & state, const PosIdx pos, Value ** args, V
if (!state.store->isInStore(path.abs()))
state.error<EvalError>("path '%1%' is not in the Nix store", path).atPos(pos).debugThrow();
auto path2 = state.store->toStorePath(path.abs()).first;
if (!settings.readOnlyMode)
if (!state.settings.storeSettings.readOnlyMode)
state.store->ensurePath(path2);
context.insert(NixStringContextElem::Opaque{.path = path2});
v.mkString(path.abs(), context, state.mem);
@@ -1971,7 +1972,7 @@ static void prim_pathExists(EvalState & state, const PosIdx pos, Value ** args,
arg.type() == nString && (arg.string_view().ends_with("/") || arg.string_view().ends_with("/."));
auto symlinkResolution = mustBeDir ? SymlinkResolution::Full : SymlinkResolution::Ancestors;
auto path = realisePath(state, pos, arg, symlinkResolution);
auto path = state.realisePath(pos, arg, symlinkResolution);
auto st = path.maybeLstat();
auto exists = st && (!mustBeDir || st->type == SourceAccessor::tDirectory);
@@ -2078,7 +2079,7 @@ static RegisterPrimOp primop_dirOf({
/* Return the contents of a file as a string. */
static void prim_readFile(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0]);
auto path = state.realisePath(pos, *args[0]);
auto s = path.readFile();
if (s.find((char) 0) != std::string::npos)
state.error<EvalError>("the contents of the file '%1%' cannot be represented as a Nix string", path)
@@ -2313,7 +2314,7 @@ static void prim_hashFile(EvalState & state, const PosIdx pos, Value ** args, Va
if (!ha)
state.error<EvalError>("unknown hash algorithm '%1%'", algo).atPos(pos).debugThrow();
auto path = realisePath(state, pos, *args[1]);
auto path = state.realisePath(pos, *args[1]);
v.mkString(hashString(*ha, path.readFile()).to_string(HashFormat::Base16, false), state.mem);
}
@@ -2365,7 +2366,7 @@ static const Value & fileTypeToString(EvalState & state, SourceAccessor::Type ty
static void prim_readFileType(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0], std::nullopt);
auto path = state.realisePath(pos, *args[0], std::nullopt);
/* Retrieve the directory entry type and stringize it. */
v = fileTypeToString(state, path.lstat().type);
}
@@ -2383,7 +2384,7 @@ static RegisterPrimOp primop_readFileType({
/* Read a directory (without . or ..) */
static void prim_readDir(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0]);
auto path = state.realisePath(pos, *args[0]);
// Retrieve directory entries for all nodes in a directory.
// This is similar to `getFileType` but is optimized to reduce system calls
@@ -2675,23 +2676,24 @@ static void prim_toFile(EvalState & state, const PosIdx pos, Value ** args, Valu
.debugThrow();
}
auto storePath = settings.readOnlyMode ? state.store->makeFixedOutputPathFromCA(
name,
TextInfo{
.hash = hashString(HashAlgorithm::SHA256, contents),
.references = std::move(refs),
})
: ({
StringSource s{contents};
state.store->addToStoreFromDump(
s,
name,
FileSerialisationMethod::Flat,
ContentAddressMethod::Raw::Text,
HashAlgorithm::SHA256,
refs,
state.repair);
});
auto storePath = state.settings.storeSettings.readOnlyMode
? state.store->makeFixedOutputPathFromCA(
name,
TextInfo{
.hash = hashString(HashAlgorithm::SHA256, contents),
.references = std::move(refs),
})
: ({
StringSource s{contents};
state.store->addToStoreFromDump(
s,
name,
FileSerialisationMethod::Flat,
ContentAddressMethod::Raw::Text,
HashAlgorithm::SHA256,
refs,
state.repair);
});
/* Note: we don't need to add `context' to the context of the
result, since `storePath' itself has references to the paths
@@ -2835,23 +2837,24 @@ static void addPath(
if (!expectedHash || !state.store->isValidPath(*expectedStorePath)) {
// FIXME: support refs in fetchToStore()?
auto dstPath = refs.empty() ? fetchToStore(
state.fetchSettings,
*state.store,
path.resolveSymlinks(),
settings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
name,
method,
filter.get(),
state.repair)
: state.store->addToStore(
name,
path.resolveSymlinks(),
method,
HashAlgorithm::SHA256,
refs,
filter ? *filter.get() : defaultPathFilter,
state.repair);
auto dstPath = refs.empty()
? fetchToStore(
state.fetchSettings,
*state.store,
path.resolveSymlinks(),
state.settings.storeSettings.readOnlyMode ? FetchMode::DryRun : FetchMode::Copy,
name,
method,
filter.get(),
state.repair)
: state.store->addToStore(
name,
path.resolveSymlinks(),
method,
HashAlgorithm::SHA256,
refs,
filter ? *filter.get() : defaultPathFilter,
state.repair);
if (expectedHash && expectedStorePath != dstPath)
state.error<EvalError>("store path mismatch in (possibly filtered) path added from '%s'", path)
.atPos(pos)
+1 -1
View File
@@ -271,7 +271,7 @@ static void prim_appendContext(EvalState & state, const PosIdx pos, Value ** arg
if (!state.store->isStorePath(name))
state.error<EvalError>("context key '%s' is not a store path", name).atPos(i.pos).debugThrow();
auto namePath = state.store->parseStorePath(name);
if (!settings.readOnlyMode)
if (!state.settings.storeSettings.readOnlyMode)
state.store->ensurePath(namePath);
state.forceAttrs(*i.value, i.pos, "while evaluating the value of a string context");
+13 -7
View File
@@ -191,19 +191,25 @@ static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value ** args
{.msg = HintFmt("attribute '%s' is missing in call to 'fetchClosure'", "fromStore"),
.pos = state.positions[pos]});
auto parsedURL = parseURL(*fromStoreUrl, /*lenient=*/true);
auto storeRef = StoreReference::parse(*fromStoreUrl);
if (parsedURL.scheme != "http" && parsedURL.scheme != "https"
&& !(getEnv("_NIX_IN_TEST").has_value() && parsedURL.scheme == "file"))
throw Error(
{.msg = HintFmt("'fetchClosure' only supports http:// and https:// stores"), .pos = state.positions[pos]});
if ([&] {
auto * specified = std::get_if<StoreReference::Specified>(&storeRef.variant);
return !specified
|| (specified->scheme != "http" && specified->scheme != "https"
&& !(getEnv("_NIX_IN_TEST").has_value() && specified->scheme == "file"));
}())
throw Error({
.msg = HintFmt("'fetchClosure' only supports http:// and https:// stores"),
.pos = state.positions[pos],
});
if (!parsedURL.query.empty())
if (!storeRef.params.empty())
throw Error(
{.msg = HintFmt("'fetchClosure' does not support URL query parameters (in '%s')", *fromStoreUrl),
.pos = state.positions[pos]});
auto fromStore = openStore(parsedURL.to_string());
auto fromStore = openStore(state.settings.storeSettings, std::move(storeRef));
if (toPath)
runFetchClosureWithRewrite(state, pos, *fromStore, *fromPath, *toPath, v);
+2 -1
View File
@@ -1,13 +1,14 @@
#include "nix_api_fetchers.h"
#include "nix_api_fetchers_internal.hh"
#include "nix_api_util_internal.h"
#include "nix_api_store_internal.h"
extern "C" {
nix_fetchers_settings * nix_fetchers_settings_new(nix_c_context * context)
{
try {
auto fetchersSettings = nix::make_ref<nix::fetchers::Settings>(nix::fetchers::Settings{});
auto fetchersSettings = nix::make_ref<nix::fetchers::Settings>(nix::fetchers::Settings{cStoreSettings});
return new nix_fetchers_settings{
.settings = fetchersSettings,
};
+16 -7
View File
@@ -1,9 +1,12 @@
#include <nlohmann/json.hpp>
#include <gtest/gtest.h>
#include "nix/util/json-utils.hh"
#include "nix/store/globals.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/fetchers/fetch-settings.hh"
#include "nix/util/json-utils.hh"
#include "nix/store/tests/test-main.hh"
#include "nix/util/tests/characterization.hh"
namespace nix::fetchers {
@@ -25,7 +28,8 @@ public:
TEST_F(AccessKeysTest, singleOrgGitHub)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
auto settings = getTestSettings();
fetchers::Settings fetchSettings = fetchers::Settings{settings};
fetchSettings.accessTokens.get().insert({"github.com/a", "token"});
auto i = Input::fromURL(fetchSettings, "github:a/b");
@@ -35,7 +39,8 @@ TEST_F(AccessKeysTest, singleOrgGitHub)
TEST_F(AccessKeysTest, nonMatches)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
auto settings = getTestSettings();
fetchers::Settings fetchSettings = fetchers::Settings{settings};
fetchSettings.accessTokens.get().insert({"github.com", "token"});
auto i = Input::fromURL(fetchSettings, "gitlab:github.com/evil");
@@ -45,7 +50,8 @@ TEST_F(AccessKeysTest, nonMatches)
TEST_F(AccessKeysTest, noPartialMatches)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
auto settings = getTestSettings();
fetchers::Settings fetchSettings = fetchers::Settings{settings};
fetchSettings.accessTokens.get().insert({"github.com/partial", "token"});
auto i = Input::fromURL(fetchSettings, "github:partial-match/repo");
@@ -55,7 +61,8 @@ TEST_F(AccessKeysTest, noPartialMatches)
TEST_F(AccessKeysTest, repoGitHub)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
auto settings = getTestSettings();
fetchers::Settings fetchSettings = fetchers::Settings{settings};
fetchSettings.accessTokens.get().insert({"github.com", "token"});
fetchSettings.accessTokens.get().insert({"github.com/a/b", "another_token"});
fetchSettings.accessTokens.get().insert({"github.com/a/c", "yet_another_token"});
@@ -73,7 +80,8 @@ TEST_F(AccessKeysTest, repoGitHub)
TEST_F(AccessKeysTest, multipleGitLab)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
auto settings = getTestSettings();
fetchers::Settings fetchSettings = fetchers::Settings{settings};
fetchSettings.accessTokens.get().insert({"gitlab.com", "token"});
fetchSettings.accessTokens.get().insert({"gitlab.com/a/b", "another_token"});
auto i = Input::fromURL(fetchSettings, "gitlab:a/b");
@@ -87,7 +95,8 @@ TEST_F(AccessKeysTest, multipleGitLab)
TEST_F(AccessKeysTest, multipleSourceHut)
{
fetchers::Settings fetchSettings = fetchers::Settings{};
auto settings = getTestSettings();
fetchers::Settings fetchSettings = fetchers::Settings{settings};
fetchSettings.accessTokens.get().insert({"git.sr.ht", "token"});
fetchSettings.accessTokens.get().insert({"git.sr.ht/~a/b", "another_token"});
auto i = Input::fromURL(fetchSettings, "sourcehut:a/b");
+8 -4
View File
@@ -5,6 +5,8 @@
#include "nix/fetchers/fetchers.hh"
#include "nix/fetchers/git-utils.hh"
#include "nix/store/tests/test-main.hh"
#include <git2.h>
#include <gtest/gtest.h>
@@ -180,13 +182,15 @@ TEST_F(GitTest, submodulePeriodSupport)
// 6) Commit the addition in super
commitAll(super.get(), "Add submodule with branch='.'");
auto store = [] {
auto cfg = make_ref<DummyStoreConfig>(StoreReference::Params{});
auto settings = getTestSettings();
auto store = [&] {
auto cfg = make_ref<DummyStoreConfig>(settings, StoreReference::Params{});
cfg->readOnly = false;
return cfg->openStore();
}();
auto settings = fetchers::Settings{};
auto fetchSettings = fetchers::Settings{settings};
auto input = fetchers::Input::fromAttrs(
settings,
{
@@ -196,7 +200,7 @@ TEST_F(GitTest, submodulePeriodSupport)
{"ref", "main"},
});
auto [accessor, i] = input.getAccessor(settings, *store);
auto [accessor, i] = input.getAccessor(fetchSettings, *store);
ASSERT_EQ(accessor->readFile(CanonPath("deps/sub/lib.txt")), "hello from submodule\n");
}
+6 -1
View File
@@ -1,7 +1,11 @@
#include "nix/store/globals.hh"
#include "nix/fetchers/fetch-settings.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/store/tests/test-main.hh"
#include <gtest/gtest.h>
#include <string>
@@ -23,7 +27,8 @@ class InputFromAttrsTest : public ::testing::WithParamInterface<InputFromAttrsTe
TEST_P(InputFromAttrsTest, attrsAreCorrectAndRoundTrips)
{
fetchers::Settings fetchSettings;
auto settings = getTestSettings();
fetchers::Settings fetchSettings{settings};
const auto & testCase = GetParam();
+9 -3
View File
@@ -34,14 +34,20 @@ struct CacheImpl : Cache
Sync<State> _state;
CacheImpl()
/**
* This is a back-reference to the `Settings` that owns us.
*/
const Settings & settings;
CacheImpl(const Settings & _settings)
: settings(_settings)
{
auto state(_state.lock());
auto dbPath = (getCacheDir() / "fetcher-cache-v4.sqlite").string();
createDirs(dirOf(dbPath));
state->db = SQLite(dbPath);
state->db = SQLite(dbPath, {.useWAL = settings.storeSettings.useSQLiteWAL});
state->db.isCache();
state->db.exec(schema);
@@ -154,7 +160,7 @@ ref<Cache> Settings::getCache() const
{
auto cache(_cache.lock());
if (!*cache)
*cache = std::make_shared<CacheImpl>();
*cache = std::make_shared<CacheImpl>(*this);
return ref<Cache>(*cache);
}
+4 -1
View File
@@ -2,6 +2,9 @@
namespace nix::fetchers {
Settings::Settings() {}
Settings::Settings(const nix::Settings & storeSettings)
: storeSettings(storeSettings)
{
}
} // namespace nix::fetchers
+63 -18
View File
@@ -5,12 +5,12 @@
namespace nix {
fetchers::Cache::Key makeFetchToStoreCacheKey(
const std::string & name, const std::string & fingerprint, ContentAddressMethod method, const std::string & path)
fetchers::Cache::Key
makeSourcePathToHashCacheKey(std::string_view fingerprint, ContentAddressMethod method, const CanonPath & path)
{
return fetchers::Cache::Key{
"fetchToStore",
{{"name", name}, {"fingerprint", fingerprint}, {"method", std::string{method.render()}}, {"path", path}}};
"sourcePathToHash",
{{"fingerprint", std::string(fingerprint)}, {"method", std::string{method.render()}}, {"path", path.abs()}}};
}
StorePath fetchToStore(
@@ -23,19 +23,39 @@ StorePath fetchToStore(
PathFilter * filter,
RepairFlag repair)
{
// FIXME: add an optimisation for the case where the accessor is
// a `PosixSourceAccessor` pointing to a store path.
return fetchToStore2(settings, store, path, mode, name, method, filter, repair).first;
}
std::pair<StorePath, Hash> fetchToStore2(
const fetchers::Settings & settings,
Store & store,
const SourcePath & path,
FetchMode mode,
std::string_view name,
ContentAddressMethod method,
PathFilter * filter,
RepairFlag repair)
{
std::optional<fetchers::Cache::Key> cacheKey;
auto [subpath, fingerprint] = filter ? std::pair<CanonPath, std::optional<std::string>>{path.path, std::nullopt}
: path.accessor->getFingerprint(path.path);
if (fingerprint) {
cacheKey = makeFetchToStoreCacheKey(std::string{name}, *fingerprint, method, subpath.abs());
if (auto res = settings.getCache()->lookupStorePath(*cacheKey, store)) {
debug("store path cache hit for '%s'", path);
return res->storePath;
cacheKey = makeSourcePathToHashCacheKey(*fingerprint, method, subpath);
if (auto res = settings.getCache()->lookup(*cacheKey)) {
auto hash = Hash::parseSRI(fetchers::getStrAttr(*res, "hash"));
auto storePath =
store.makeFixedOutputPathFromCA(name, ContentAddressWithReferences::fromParts(method, hash, {}));
if (mode == FetchMode::DryRun || store.isValidPath(storePath)) {
debug(
"source path '%s' cache hit in '%s' (hash '%s')",
path,
store.printStorePath(storePath),
hash.to_string(HashFormat::SRI, true));
return {storePath, hash};
}
debug("source path '%s' not in store", path);
}
} else {
static auto barf = getEnv("_NIX_TEST_BARF_ON_UNCACHEABLE").value_or("") == "1";
@@ -53,16 +73,41 @@ StorePath fetchToStore(
auto filter2 = filter ? *filter : defaultPathFilter;
auto storePath = mode == FetchMode::DryRun
? store.computeStorePath(name, path, method, HashAlgorithm::SHA256, {}, filter2).first
: store.addToStore(name, path, method, HashAlgorithm::SHA256, {}, filter2, repair);
auto [storePath, hash] =
mode == FetchMode::DryRun
? [&]() {
auto [storePath, hash] = store.computeStorePath(
store.config.settings, name, path, method, HashAlgorithm::SHA256, {}, filter2);
debug(
"hashed '%s' to '%s' (hash '%s')",
path,
store.printStorePath(storePath),
hash.to_string(HashFormat::SRI, true));
return std::make_pair(storePath, hash);
}()
: [&]() {
// FIXME: ideally addToStore() would return the hash
// right away (like computeStorePath()).
auto storePath = store.addToStore(name, path, method, HashAlgorithm::SHA256, {}, filter2, repair);
auto info = store.queryPathInfo(storePath);
assert(info->references.empty());
auto hash = method == ContentAddressMethod::Raw::NixArchive ? info->narHash : ({
if (!info->ca || info->ca->method != method)
throw Error("path '%s' lacks a CA field", store.printStorePath(storePath));
info->ca->hash;
});
debug(
"copied '%s' to '%s' (hash '%s')",
path,
store.printStorePath(storePath),
hash.to_string(HashFormat::SRI, true));
return std::make_pair(storePath, hash);
}();
debug(mode == FetchMode::DryRun ? "hashed '%s'" : "copied '%s' to '%s'", path, store.printStorePath(storePath));
if (cacheKey)
settings.getCache()->upsert(*cacheKey, {{"hash", hash.to_string(HashFormat::SRI, true)}});
if (cacheKey && mode == FetchMode::Copy)
settings.getCache()->upsert(*cacheKey, store, {}, storePath);
return storePath;
return {storePath, hash};
}
} // namespace nix
+6 -5
View File
@@ -339,9 +339,10 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(const Settings
// can reuse the existing nar instead of copying the unpacked
// input back into the store on every evaluation.
if (accessor->fingerprint) {
ContentAddressMethod method = ContentAddressMethod::Raw::NixArchive;
auto cacheKey = makeFetchToStoreCacheKey(getName(), *accessor->fingerprint, method, "/");
settings.getCache()->upsert(cacheKey, store, {}, storePath);
settings.getCache()->upsert(
makeSourcePathToHashCacheKey(
*accessor->fingerprint, ContentAddressMethod::Raw::NixArchive, CanonPath::root),
{{"hash", store.queryPathInfo(storePath)->narHash.to_string(HashFormat::SRI, true)}});
}
accessor->setPathDisplay("«" + to_string() + "»");
@@ -489,11 +490,11 @@ void InputScheme::clone(
const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir) const
{
if (std::filesystem::exists(destDir))
throw Error("cannot clone into existing path %s", destDir);
throw Error("cannot clone into existing path %s", PathFmt(destDir));
auto [accessor, input2] = getAccessor(settings, store, input);
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying '%s' to %s...", input2.to_string(), destDir));
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying '%s' to %s...", input2.to_string(), PathFmt(destDir)));
RestoreSink sink(/*startFsync=*/false);
sink.dstPath = destDir;
+5 -6
View File
@@ -10,12 +10,6 @@ std::optional<std::filesystem::path> FilteringSourceAccessor::getPhysicalPath(co
return next->getPhysicalPath(prefix / path);
}
std::string FilteringSourceAccessor::readFile(const CanonPath & path)
{
checkAccess(path);
return next->readFile(prefix / path);
}
void FilteringSourceAccessor::readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback)
{
checkAccess(path);
@@ -67,6 +61,11 @@ std::pair<CanonPath, std::optional<std::string>> FilteringSourceAccessor::getFin
return next->getFingerprint(prefix / path);
}
void FilteringSourceAccessor::invalidateCache(const CanonPath & path)
{
next->invalidateCache(prefix / path);
}
void FilteringSourceAccessor::checkAccess(const CanonPath & path)
{
if (!isAllowed(path))
+2 -2
View File
@@ -273,7 +273,7 @@ void Fetch::fetch(
+ "/" + pointer->oid;
std::filesystem::path cachePath = cacheDir / key;
if (pathExists(cachePath)) {
debug("using cache entry %s -> %s", key, cachePath);
debug("using cache entry %s -> %s", key, PathFmt(cachePath));
sink(readFile(cachePath));
return;
}
@@ -301,7 +301,7 @@ void Fetch::fetch(
sizeCallback(size);
downloadToSink(ourl, authHeader, sink, sha256, size);
debug("creating cache entry %s -> %s", key, cachePath);
debug("creating cache entry %s -> %s", key, PathFmt(cachePath));
if (!pathExists(cachePath.parent_path()))
createDirs(cachePath.parent_path());
writeFile(cachePath, sink.s);
+19 -11
View File
@@ -209,14 +209,14 @@ static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options op
return;
if (!options.create)
throw Error("Git repository %s does not exist.", path);
throw Error("Git repository %s does not exist.", PathFmt(path));
std::filesystem::path tmpDir = createTempDir(path.parent_path());
AutoDelete delTmpDir(tmpDir, true);
Repository tmpRepo;
if (git_repository_init(Setter(tmpRepo), tmpDir.string().c_str(), options.bare))
throw Error("creating Git repository %s: %s", path, git_error_last()->message);
throw Error("creating Git repository %s: %s", PathFmt(path), git_error_last()->message);
try {
std::filesystem::rename(tmpDir, path);
} catch (std::filesystem::filesystem_error & e) {
@@ -226,7 +226,7 @@ static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options op
|| e.code() == std::errc::directory_not_empty) {
return;
} else
throw SysError("moving temporary git repository from %s to %s", tmpDir, path);
throw SysError("moving temporary git repository from %s to %s", PathFmt(tmpDir), PathFmt(path));
}
// we successfully moved the repository, so the temporary directory no longer exists.
delTmpDir.cancel();
@@ -266,7 +266,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
initRepoAtomically(path, options);
if (git_repository_open(Setter(repo), path.string().c_str()))
throw Error("opening Git repository %s: %s", path, git_error_last()->message);
throw Error("opening Git repository %s: %s", PathFmt(path), git_error_last()->message);
ObjectDb odb;
if (options.packfilesOnly) {
@@ -766,7 +766,7 @@ struct GitSourceAccessor : SourceAccessor
{
}
std::string readBlob(const CanonPath & path, bool symlink)
void readBlob(const CanonPath & path, bool symlink, Sink & sink, std::function<void(uint64_t)> sizeCallback)
{
auto state(state_.lock());
@@ -785,16 +785,22 @@ struct GitSourceAccessor : SourceAccessor
e.addTrace({}, "while smudging git-lfs file '%s'", path);
throw;
}
return s.s;
sizeCallback(s.s.size());
StringSource source{s.s};
source.drainInto(sink);
return;
}
}
return std::string((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
auto view = std::string_view((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
sizeCallback(view.size());
StringSource source{view};
source.drainInto(sink);
}
std::string readFile(const CanonPath & path) override
void readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback) override
{
return readBlob(path, false);
return readBlob(path, false, sink, sizeCallback);
}
bool pathExists(const CanonPath & path) override
@@ -861,7 +867,9 @@ struct GitSourceAccessor : SourceAccessor
std::string readLink(const CanonPath & path) override
{
return readBlob(path, true);
StringSink s;
readBlob(path, true, s, [&](uint64_t size) { s.s.reserve(size); });
return std::move(s.s);
}
/**
@@ -1425,7 +1433,7 @@ std::vector<std::tuple<GitRepoImpl::Submodule, Hash>> GitRepoImpl::getSubmodules
auto [fdTemp, pathTemp] = createTempFile("nix-git-submodules");
try {
writeFull(fdTemp.get(), configS);
} catch (SysError & e) {
} catch (SystemError & e) {
e.addTrace({}, "while writing .gitmodules file to temporary file");
throw;
}
+17 -16
View File
@@ -37,7 +37,7 @@ namespace {
// old version of git, which will ignore unrecognized `-c` options.
const std::string gitInitialBranch = "__nix_dummy_branch";
bool isCacheFileWithinTtl(time_t now, const struct stat & st)
static bool isCacheFileWithinTtl(const Settings & settings, time_t now, const PosixStat & st)
{
return st.st_mtime + static_cast<time_t>(settings.tarballTtl) > now;
}
@@ -72,10 +72,10 @@ std::optional<std::string> readHead(const std::filesystem::path & path)
if (const auto parseResult = git::parseLsRemoteLine(line); parseResult && parseResult->reference == "HEAD") {
switch (parseResult->kind) {
case git::LsRemoteRefLine::Kind::Symbolic:
debug("resolved HEAD ref '%s' for repo '%s'", parseResult->target, path);
debug("resolved HEAD ref '%s' for repo %s", parseResult->target, PathFmt(path));
break;
case git::LsRemoteRefLine::Kind::Object:
debug("resolved HEAD rev '%s' for repo '%s'", parseResult->target, path);
debug("resolved HEAD rev '%s' for repo %s", parseResult->target, PathFmt(path));
break;
}
return parseResult->target;
@@ -105,7 +105,7 @@ bool storeCachedHead(const std::string & actualUrl, bool shallow, const std::str
return true;
}
std::optional<std::string> readHeadCached(const std::string & actualUrl, bool shallow)
static std::optional<std::string> readHeadCached(const Settings & settings, const std::string & actualUrl, bool shallow)
{
// Create a cache path to store the branch of the HEAD ref. Append something
// in front of the URL to prevent collision with the repository itself.
@@ -113,11 +113,11 @@ std::optional<std::string> readHeadCached(const std::string & actualUrl, bool sh
std::filesystem::path headRefFile = cacheDir / "HEAD";
time_t now = time(0);
struct stat st;
auto st = maybeStat(headRefFile);
std::optional<std::string> cachedRef;
if (stat(headRefFile.string().c_str(), &st) == 0) {
if (st) {
cachedRef = readHead(cacheDir);
if (cachedRef != std::nullopt && *cachedRef != gitInitialBranch && isCacheFileWithinTtl(now, st)) {
if (cachedRef != std::nullopt && *cachedRef != gitInitialBranch && isCacheFileWithinTtl(settings, now, *st)) {
debug("using cached HEAD ref '%s' for repo '%s'", *cachedRef, actualUrl);
return cachedRef;
}
@@ -728,12 +728,12 @@ struct GitInputScheme : InputScheme
return revCount;
}
std::string getDefaultRef(const RepoInfo & repoInfo, bool shallow) const
std::string getDefaultRef(const Settings & settings, const RepoInfo & repoInfo, bool shallow) const
{
auto head = std::visit(
overloaded{
[&](const std::filesystem::path & path) { return GitRepo::openRepo(path, {})->getWorkdirRef(); },
[&](const ParsedURL & url) { return readHeadCached(url.to_string(), shallow); }},
[&](const ParsedURL & url) { return readHeadCached(settings, url.to_string(), shallow); }},
repoInfo.location);
if (!head) {
warn("could not read HEAD ref from repo at '%s', using 'master'", repoInfo.locationToArg());
@@ -753,9 +753,10 @@ struct GitInputScheme : InputScheme
"\n"
"git -C %2% add \"%1%\"",
path.rel(),
repoPath);
PathFmt(repoPath));
else
return RestrictedPathError("Path '%s' does not exist in Git repository %s.", path.rel(), repoPath);
return RestrictedPathError(
"Path '%s' does not exist in Git repository %s.", path.rel(), PathFmt(repoPath));
};
}
@@ -782,7 +783,7 @@ struct GitInputScheme : InputScheme
auto originalRef = input.getRef();
bool shallow = getShallowAttr(input);
auto ref = originalRef ? *originalRef : getDefaultRef(repoInfo, shallow);
auto ref = originalRef ? *originalRef : getDefaultRef(settings, repoInfo, shallow);
input.attrs.insert_or_assign("ref", ref);
std::filesystem::path repoDir;
@@ -818,10 +819,10 @@ struct GitInputScheme : InputScheme
if (getAllRefsAttr(input)) {
doFetch = true;
} else {
/* If the local ref is older than ‘tarball-ttl’ seconds, do a
/* If the local ref is older than 'tarball-ttl' seconds, do a
git fetch to update the local ref to the remote ref. */
struct stat st;
doFetch = stat(localRefFile.string().c_str(), &st) != 0 || !isCacheFileWithinTtl(now, st);
auto st = maybeStat(localRefFile);
doFetch = !st || !isCacheFileWithinTtl(settings, now, *st);
}
}
@@ -848,7 +849,7 @@ struct GitInputScheme : InputScheme
if (!input.getRev())
setWriteTime(localRefFile, now, now);
} catch (Error & e) {
warn("could not update mtime for file %s: %s", localRefFile, e.info().msg);
warn("could not update mtime for file %s: %s", PathFmt(localRefFile), e.info().msg);
}
if (!originalRef && !storeCachedHead(repoUrl.to_string(), shallow, ref))
warn("could not update cached head '%s' for '%s'", ref, repoInfo.locationToArg());
@@ -13,9 +13,10 @@
namespace nix {
class Settings;
struct GitRepo;
}
} // namespace nix
namespace nix::fetchers {
@@ -23,7 +24,12 @@ struct Cache;
struct Settings : public Config
{
Settings();
/**
* Reference to the "parent" store-layer settings.
*/
const nix::Settings & storeSettings;
Settings(const nix::Settings & storeSettings);
Setting<StringMap> accessTokens{
this,
@@ -129,6 +135,25 @@ struct Settings : public Config
true,
Xp::Flakes};
Setting<unsigned int> tarballTtl{
this,
60 * 60,
"tarball-ttl",
R"(
The number of seconds a downloaded tarball is considered fresh. If
the cached tarball is stale, Nix checks whether it is still up
to date using the ETag header. Nix downloads a new version if
the ETag header is unsupported, or the cached ETag doesn't match.
Setting the TTL to `0` forces Nix to always check if the tarball is
up to date.
Nix caches tarballs in `$XDG_CACHE_HOME/nix/tarballs`.
Files fetched via `NIX_PATH`, `fetchGit`, `fetchMercurial`,
`fetchTarball`, and `fetchurl` respect this TTL.
)"};
ref<Cache> getCache() const;
ref<GitRepo> getTarballCache() const;
@@ -24,7 +24,17 @@ StorePath fetchToStore(
PathFilter * filter = nullptr,
RepairFlag repair = NoRepair);
fetchers::Cache::Key makeFetchToStoreCacheKey(
const std::string & name, const std::string & fingerprint, ContentAddressMethod method, const std::string & path);
std::pair<StorePath, Hash> fetchToStore2(
const fetchers::Settings & settings,
Store & store,
const SourcePath & path,
FetchMode mode,
std::string_view name = "source",
ContentAddressMethod method = ContentAddressMethod::Raw::NixArchive,
PathFilter * filter = nullptr,
RepairFlag repair = NoRepair);
fetchers::Cache::Key
makeSourcePathToHashCacheKey(std::string_view fingerprint, ContentAddressMethod method, const CanonPath & path);
} // namespace nix
@@ -34,7 +34,7 @@ struct FilteringSourceAccessor : SourceAccessor
std::optional<std::filesystem::path> getPhysicalPath(const CanonPath & path) override;
std::string readFile(const CanonPath & path) override;
using SourceAccessor::readFile;
void readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback) override;
@@ -52,6 +52,8 @@ struct FilteringSourceAccessor : SourceAccessor
std::pair<CanonPath, std::optional<std::string>> getFingerprint(const CanonPath & path) override;
void invalidateCache(const CanonPath & path) override;
/**
* Call `makeNotAllowedError` to throw a `RestrictedPathError`
* exception if `isAllowed()` returns `false` for `path`.
@@ -1,3 +1,5 @@
#pragma once
#include "nix/fetchers/fetchers.hh"
namespace nix::fetchers {
+3 -2
View File
@@ -1,5 +1,6 @@
#include "nix/fetchers/fetchers.hh"
#include "nix/util/processes.hh"
#include "nix/util/environment-variables.hh"
#include "nix/util/users.hh"
#include "nix/fetchers/cache.hh"
#include "nix/store/globals.hh"
@@ -16,9 +17,9 @@ namespace nix::fetchers {
static RunOptions hgOptions(const Strings & args)
{
auto env = getEnv();
auto env = getEnvOs();
// Set HGPLAIN: this means we get consistent output from hg and avoids leakage from a user or system .hgrc.
env["HGPLAIN"] = "";
env[OS_STR("HGPLAIN")] = OS_STR("");
return {.program = "hg", .lookupPath = true, .args = args, .environment = env};
}
+6 -8
View File
@@ -154,7 +154,7 @@ struct PathInputScheme : InputScheme
time_t mtime = 0;
if (!storePath || storePath->name() != "source" || !store.isValidPath(*storePath)) {
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying %s to the store", absPath));
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying %s to the store", PathFmt(absPath)));
// FIXME: try to substitute storePath.
auto src = sinkToSource(
[&](Sink & sink) { mtime = dumpPathAndGetMtime(absPath.string(), sink, defaultPathFilter); });
@@ -165,14 +165,12 @@ struct PathInputScheme : InputScheme
// To prevent `fetchToStore()` copying the path again to Nix
// store, pre-create an entry in the fetcher cache.
accessor->fingerprint =
fmt("path:%s", store.queryPathInfo(*storePath)->narHash.to_string(HashFormat::SRI, true));
auto narHash = store.queryPathInfo(*storePath)->narHash.to_string(HashFormat::SRI, true);
accessor->fingerprint = fmt("path:%s", narHash);
settings.getCache()->upsert(
makeFetchToStoreCacheKey(
input.getName(), *accessor->fingerprint, ContentAddressMethod::Raw::NixArchive, "/"),
store,
{},
*storePath);
makeSourcePathToHashCacheKey(
*accessor->fingerprint, ContentAddressMethod::Raw::NixArchive, CanonPath::root),
{{"hash", narHash}});
/* Trust the lastModified value supplied by the user, if
any. It's not a "secure" attribute so we don't care. */
+1 -1
View File
@@ -92,7 +92,7 @@ void Registry::remove(const Input & input)
static std::filesystem::path getSystemRegistryPath()
{
return settings.nixConfDir / "registry.json";
return nixConfDir() / "registry.json";
}
static std::shared_ptr<Registry> getSystemRegistry(const Settings & settings)
+12 -4
View File
@@ -114,15 +114,22 @@ static DownloadTarballResult downloadTarball_(
// it is not in fact a tarball.
if (url.scheme == "file") {
std::filesystem::path localPath = renderUrlPathEnsureLegal(url.path);
if (!localPath.is_absolute()) {
throw Error(
"tarball '%s' must use an absolute path. "
"The 'file' scheme does not support relative paths.",
url);
}
if (!exists(localPath)) {
throw Error("tarball '%s' does not exist.", localPath);
throw Error("tarball %s does not exist.", PathFmt(localPath));
}
if (is_directory(localPath)) {
if (exists(localPath / ".git")) {
throw Error(
"tarball '%s' is a git repository, not a tarball. Please use `git+file` as the scheme.", localPath);
"tarball %s is a git repository, not a tarball. Please use `git+file` as the scheme.",
PathFmt(localPath));
}
throw Error("tarball '%s' is a directory, not a file.", localPath);
throw Error("tarball %s is a directory, not a file.", PathFmt(localPath));
}
}
@@ -170,7 +177,8 @@ static DownloadTarballResult downloadTarball_(
the entire file to disk so libarchive can access it
in random-access mode. */
auto [fdTemp, path] = createTempFile("nix-zipfile");
cleanupTemp.reset(path);
cleanupTemp.cancel();
cleanupTemp = {path};
debug("downloading '%s' into '%s'...", url, path);
{
FdSink sink(fdTemp.get());
+9 -4
View File
@@ -3,6 +3,7 @@
#include <string>
#include <utility>
#include "nix/store/globals.hh"
#include "nix/fetchers/fetch-settings.hh"
#include "nix/flake/flakeref.hh"
#include "nix/fetchers/attrs.hh"
@@ -19,7 +20,8 @@ TEST(parseFlakeRef, path)
{
experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes);
fetchers::Settings fetchSettings;
Settings settings;
fetchers::Settings fetchSettings{settings};
{
auto s = "/foo/bar";
@@ -69,7 +71,8 @@ TEST(parseFlakeRef, GitArchiveInput)
{
experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes);
fetchers::Settings fetchSettings;
Settings settings;
fetchers::Settings fetchSettings{settings};
{
auto s = "github:foo/bar/branch%23"; // branch name with `#`
@@ -112,7 +115,8 @@ class InputFromURLTest : public ::testing::WithParamInterface<InputFromURLTestCa
TEST_P(InputFromURLTest, attrsAreCorrectAndRoundTrips)
{
experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes);
fetchers::Settings fetchSettings;
Settings settings;
fetchers::Settings fetchSettings{settings};
const auto & testCase = GetParam();
@@ -274,7 +278,8 @@ INSTANTIATE_TEST_SUITE_P(
TEST(to_string, doesntReencodeUrl)
{
fetchers::Settings fetchSettings;
Settings settings;
fetchers::Settings fetchSettings{settings};
auto s = "http://localhost:8181/test/+3d.tar.gz";
auto flakeref = parseFlakeRef(fetchSettings, s);
auto unparsed = flakeref.to_string();
+1
View File
@@ -377,6 +377,7 @@ TEST_F(nix_api_store_test, nix_api_load_flake_with_flags)
assert_ctx_ok();
ASSERT_EQ("Claire", helloStr);
nix_locked_flake_free(lockedFlake);
nix_flake_reference_parse_flags_free(parseFlags);
nix_flake_lock_flags_free(lockFlags);
nix_flake_reference_free(flakeReference);
+1
View File
@@ -128,6 +128,7 @@ static void prim_flakeRefToString(EvalState & state, const PosIdx pos, Value **
state.forceAttrs(*args[0], noPos, "while evaluating the argument passed to builtins.flakeRefToString");
fetchers::Attrs attrs;
for (const auto & attr : *args[0]->attrs()) {
state.forceValue(*attr.value, attr.pos);
auto t = attr.value->type();
if (t == nInt) {
auto intValue = attr.value->integer().value;
+5 -3
View File
@@ -850,11 +850,11 @@ lockFlake(const Settings & settings, EvalState & state, const FlakeRef & topRef,
auto s = chomp(diff);
if (lockFileExists) {
if (s.empty())
warn("updating lock file %s", outputLockFilePath);
warn("updating lock file %s", PathFmt(outputLockFilePath));
else
warn("updating lock file %s:\n%s", outputLockFilePath, s);
warn("updating lock file %s:\n%s", PathFmt(outputLockFilePath), s);
} else
warn("creating lock file %s: \n%s", outputLockFilePath, s);
warn("creating lock file %s: \n%s", PathFmt(outputLockFilePath), s);
std::optional<std::string> commitMessage = std::nullopt;
@@ -876,6 +876,8 @@ lockFlake(const Settings & settings, EvalState & state, const FlakeRef & topRef,
CanonPath((topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock"),
newLockFileS,
commitMessage);
flake.lockFilePath().invalidateCache();
}
/* Rewriting the lockfile changed the top-level
+1
View File
@@ -8,6 +8,7 @@
#include "nix/main/loggers.hh"
#include "nix/util/util.hh"
#include "nix/main/plugin.hh"
#include "nix/main/shared.hh"
namespace nix {
+6 -13
View File
@@ -13,6 +13,11 @@
namespace nix {
class Settings;
// FIXME: don't use a global variable.
extern Settings settings;
int handleExceptions(const std::string & programName, std::function<void()> fun);
/**
@@ -91,19 +96,7 @@ extern volatile ::sig_atomic_t blockInt;
struct GCResults;
struct PrintFreed
{
bool show;
const GCResults & results;
PrintFreed(bool show, const GCResults & results)
: show(show)
, results(results)
{
}
~PrintFreed();
};
void printFreed(bool dryRun, const GCResults & results);
#ifndef _WIN32
/**
+4 -4
View File
@@ -83,8 +83,8 @@ void initPlugins()
checkInterrupt();
pluginFiles.emplace_back(ent.path());
}
} catch (SysError & e) {
if (e.errNo != ENOTDIR)
} catch (SystemError & e) {
if (!e.is(std::errc::not_a_directory))
throw;
pluginFiles.emplace_back(pluginFile);
}
@@ -95,7 +95,7 @@ void initPlugins()
#ifndef _WIN32 // TODO implement via DLL loading on Windows
void * handle = dlopen(file.c_str(), RTLD_LAZY | RTLD_LOCAL);
if (!handle)
throw Error("could not dynamically open plugin file '%s': %s", file, dlerror());
throw Error("could not dynamically open plugin file %s: %s", PathFmt(file), dlerror());
/* Older plugins use a statically initialized object to run their code.
Newer plugins can also export nix_plugin_entry() */
@@ -103,7 +103,7 @@ void initPlugins()
if (nix_plugin_entry)
nix_plugin_entry();
#else
throw Error("could not dynamically open plugin file '%s'", file);
throw Error("could not dynamically open plugin file %s", PathFmt(file));
#endif
}
}
+46 -38
View File
@@ -1,7 +1,9 @@
#include "nix/store/globals.hh"
#include "nix/util/current-process.hh"
#include "nix/util/executable-path.hh"
#include "nix/main/shared.hh"
#include "nix/store/store-api.hh"
#include "nix/store/store-open.hh"
#include "nix/store/gc-store.hh"
#include "nix/main/loggers.hh"
#include "nix/main/progress-bar.hh"
@@ -17,6 +19,10 @@
#include <sys/stat.h>
#include <unistd.h>
#include <signal.h>
#ifndef _WIN32
# include <sys/resource.h>
#endif
#ifdef __linux__
# include <features.h>
#endif
@@ -25,12 +31,17 @@
#include "nix/util/exit.hh"
#include "nix/util/strings.hh"
#include "nix/util/config-global.hh"
#include "main-config-private.hh"
#include "nix/expr/config.hh"
namespace nix {
Settings settings;
static GlobalConfig::Register rSettings(&settings);
char ** savedArgv;
static bool gcWarning = true;
@@ -116,6 +127,26 @@ std::string getArg(const std::string & opt, Strings::iterator & i, const Strings
static void sigHandler(int signo) {}
#endif
/**
* Increase the open file soft limit to the hard limit. On some
* platforms (macOS), the default soft limit is very low, but the hard
* limit is high. So let's just raise it the maximum permitted.
*/
void bumpFileLimit()
{
#ifndef _WIN32
struct rlimit limit;
if (getrlimit(RLIMIT_NOFILE, &limit) != 0)
return;
if (limit.rlim_cur < limit.rlim_max) {
limit.rlim_cur = limit.rlim_max;
// Ignore errors, this is best effort.
setrlimit(RLIMIT_NOFILE, &limit);
}
#endif
}
void initNix(bool loadConfig)
{
/* Turn on buffering for cerr. */
@@ -183,6 +214,8 @@ void initNix(bool loadConfig)
now. In particular, store objects should be readable by
everybody. */
umask(0022);
bumpFileLimit();
}
LegacyArgs::LegacyArgs(
@@ -209,13 +242,13 @@ LegacyArgs::LegacyArgs(
.longName = "keep-going",
.shortName = 'k',
.description = "Keep going after a build fails.",
.handler = {&(bool &) settings.keepGoing, true},
.handler = {&(bool &) settings.getWorkerSettings().keepGoing, true},
});
addFlag({
.longName = "fallback",
.description = "Build from source if substitution fails.",
.handler = {&(bool &) settings.tryFallback, true},
.handler = {&(bool &) settings.getWorkerSettings().tryFallback, true},
});
auto intSettingAlias =
@@ -252,7 +285,7 @@ LegacyArgs::LegacyArgs(
.longName = "store",
.description = "The URL of the Nix store to use.",
.labels = {"store-uri"},
.handler = {&(std::string &) settings.storeUri},
.handler = {[](std::string s) { settings.storeUri = StoreReference::parse(s); }},
});
}
@@ -304,44 +337,16 @@ void printVersion(const std::string & programName)
std::cout << "System type: " << settings.thisSystem << "\n";
std::cout << "Additional system types: " << concatStringsSep(", ", settings.extraPlatforms.get()) << "\n";
std::cout << "Features: " << concatStringsSep(", ", cfg) << "\n";
std::cout << "System configuration file: " << (settings.nixConfDir / "nix.conf") << "\n";
std::cout << "User configuration files: " << concatStringsSep(":", settings.nixUserConfFiles) << "\n";
std::cout << "Store directory: " << settings.nixStore << "\n";
std::cout << "System configuration file: " << nixConfFile() << "\n";
std::cout << "User configuration files: "
<< os_string_to_string(ExecutablePath{.directories = nixUserConfFiles()}.render()) << "\n";
std::cout << "Store directory: "
<< resolveStoreConfig(settings, StoreReference{settings.storeUri.get()})->storeDir << "\n";
std::cout << "State directory: " << settings.nixStateDir << "\n";
std::cout << "Data directory: " << settings.nixDataDir << "\n";
}
throw Exit();
}
int handleExceptions(const std::string & programName, std::function<void()> fun)
{
ReceiveInterrupts receiveInterrupts; // FIXME: need better place for this
ErrorInfo::programName = baseNameOf(programName);
std::string error = ANSI_RED "error:" ANSI_NORMAL " ";
try {
fun();
} catch (Exit & e) {
return e.status;
} catch (UsageError & e) {
logError(e.info());
printError("Try '%1% --help' for more information.", programName);
return 1;
} catch (BaseError & e) {
logError(e.info());
return e.info().status;
} catch (std::bad_alloc & e) {
printError(error + "out of memory");
return 1;
} catch (std::exception & e) {
printError(error + e.what());
return 1;
}
return 0;
}
RunPager::RunPager()
{
if (!isatty(STDOUT_FILENO))
@@ -396,9 +401,12 @@ RunPager::~RunPager()
}
}
PrintFreed::~PrintFreed()
void printFreed(bool dryRun, const GCResults & results)
{
if (show)
/* bytesFreed cannot be reliably computed without actually deleting store paths because of hardlinking. */
if (dryRun)
std::cout << fmt("%d store paths would be deleted\n", results.paths.size());
else
std::cout << fmt("%d store paths deleted, %s freed\n", results.paths.size(), renderSize(results.bytesFreed));
}
+57 -12
View File
@@ -9,6 +9,7 @@
#include "nix/store/path.hh"
#include "nix/store/store-api.hh"
#include "nix/store/store-open.hh"
#include "nix/store/store-reference.hh"
#include "nix/store/build-result.hh"
#include "nix/store/local-fs-store.hh"
#include "nix/util/base-nix-32.hh"
@@ -17,6 +18,9 @@
extern "C" {
// FIXME: This is going to conflict with the one in libmain.
nix::Settings cStoreSettings;
nix_err nix_libstore_init(nix_c_context * context)
{
if (context)
@@ -45,16 +49,16 @@ Store * nix_store_open(nix_c_context * context, const char * uri, const char ***
std::string uri_str = uri ? uri : "";
if (uri_str.empty())
return new Store{nix::openStore()};
return new Store{nix::openStore(cStoreSettings)};
if (!params)
return new Store{nix::openStore(uri_str)};
auto storeRef = nix::StoreReference::parse(uri_str);
nix::Store::Config::Params params_map;
for (size_t i = 0; params[i] != nullptr; i++) {
params_map[params[i][0]] = params[i][1];
if (params) {
for (size_t i = 0; params[i] != nullptr; i++) {
storeRef.params[params[i][0]] = params[i][1];
}
}
return new Store{nix::openStore(uri_str, params_map)};
return new Store{nix::openStore(cStoreSettings, std::move(storeRef))};
}
NIXC_CATCH_ERRS_NULL
}
@@ -182,10 +186,8 @@ nix_err nix_store_realise(
assert(results.size() == 1);
// Check if any builds failed
for (auto & result : results) {
if (auto * failureP = result.tryGetFailure())
failureP->rethrow();
}
for (auto & result : results)
result.tryThrowBuildError();
if (callback) {
for (const auto & result : results) {
@@ -309,7 +311,12 @@ StorePath * nix_add_derivation(nix_c_context * context, Store * store, nix_deriv
if (context)
context->last_err_code = NIX_OK;
try {
auto ret = nix::writeDerivation(*store->ptr, derivation->drv, nix::NoRepair);
/* Quite dubious that users would want this to silently suceed
without actually writing the derivation if this setting is
set, but it was that way already, so we are doing this for
back-compat for now. */
auto ret = cStoreSettings.readOnlyMode ? nix::computeStorePath(*store->ptr, derivation->drv)
: store->ptr->writeDerivation(derivation->drv, nix::NoRepair);
return new StorePath{ret};
}
@@ -338,4 +345,42 @@ nix_derivation * nix_store_drv_from_store_path(nix_c_context * context, Store *
NIXC_CATCH_ERRS_NULL
}
StorePath * nix_store_query_path_from_hash_part(nix_c_context * context, Store * store, const char * hash)
{
if (context)
context->last_err_code = NIX_OK;
try {
std::optional<nix::StorePath> s = store->ptr->queryPathFromHashPart(hash);
if (!s.has_value()) {
return nullptr;
}
return new StorePath{std::move(s.value())};
}
NIXC_CATCH_ERRS_NULL
}
nix_err nix_store_copy_path(
nix_c_context * context, Store * srcStore, Store * dstStore, const StorePath * path, bool repair, bool checkSigs)
{
if (context)
context->last_err_code = NIX_OK;
try {
if (srcStore == nullptr)
return nix_set_err_msg(context, NIX_ERR_UNKNOWN, "Source store is null");
if (dstStore == nullptr)
return nix_set_err_msg(context, NIX_ERR_UNKNOWN, "Destination store is null");
if (path == nullptr)
return nix_set_err_msg(context, NIX_ERR_UNKNOWN, "Store path is null");
auto repairFlag = repair ? nix::RepairFlag::Repair : nix::RepairFlag::NoRepair;
auto checkSigsFlag = checkSigs ? nix::CheckSigsFlag::CheckSigs : nix::CheckSigsFlag::NoCheckSigs;
nix::copyStorePath(*srcStore->ptr, *dstStore->ptr, path->path, repairFlag, checkSigsFlag);
}
NIXC_CATCH_ERRS
}
} // extern "C"
+24
View File
@@ -259,6 +259,30 @@ nix_err nix_store_get_fs_closure(
*/
nix_derivation * nix_store_drv_from_store_path(nix_c_context * context, Store * store, const StorePath * path);
/**
* @brief Query the full store path given the hash part of a valid store
* path, or empty if no matching path is found.
*
* @param[out] context Optional, stores error information
* @param[in] store nix store reference
* @param[in] hash Hash part of path as a string
* @return Store path reference, NULL if no matching path is found.
*/
StorePath * nix_store_query_path_from_hash_part(nix_c_context * context, Store * store, const char * hash);
/**
* @brief Copy a path from one store to another.
*
* @param[out] context Optional, stores error information
* @param[in] srcStore nix source store reference
* @param[in] dstStore nix destination store reference
* @param[in] path The path to copy
* @param[in] repair Whether to repair the path
* @param[in] checkSigs Whether to check path signatures are trusted before copying
*/
nix_err nix_store_copy_path(
nix_c_context * context, Store * srcStore, Store * dstStore, const StorePath * path, bool repair, bool checkSigs);
// cffi end
#ifdef __cplusplus
}
+2
View File
@@ -5,6 +5,8 @@
extern "C" {
extern nix::Settings cStoreSettings;
struct Store
{
nix::ref<nix::Store> ptr;
+144
View File
@@ -0,0 +1,144 @@
#include "nix/store/tests/https-store.hh"
#include "nix/store/tests/test-main.hh"
#include <thread>
namespace nix::testing {
void TestHttpBinaryCacheStore::init()
{
BinaryCacheStore::init();
}
ref<TestHttpBinaryCacheStore> TestHttpBinaryCacheStoreConfig::openTestStore(ref<FileTransfer> fileTransfer) const
{
auto store = make_ref<TestHttpBinaryCacheStore>(
ref{// FIXME we shouldn't actually need a mutable config
std::const_pointer_cast<HttpBinaryCacheStore::Config>(shared_from_this())},
fileTransfer);
store->init();
return store;
}
void HttpsBinaryCacheStoreTest::openssl(Strings args)
{
runProgram("openssl", /*lookupPath=*/true, args);
}
void HttpsBinaryCacheStoreTest::SetUp()
{
LibStoreNetworkTest::SetUp();
#ifdef _WIN32
GTEST_SKIP() << "HTTPS store tests are not supported on Windows";
#endif
tmpDir = createTempDir();
cacheDir = tmpDir / "cache";
delTmpDir = std::make_unique<AutoDelete>(tmpDir);
localCacheStore = make_ref<LocalBinaryCacheStoreConfig>(
settings, "file", cacheDir.string(), LocalBinaryCacheStoreConfig::Params{})
->openStore();
caCert = tmpDir / "ca.crt";
caKey = tmpDir / "ca.key";
serverCert = tmpDir / "server.crt";
serverKey = tmpDir / "server.key";
clientCert = tmpDir / "client.crt";
clientKey = tmpDir / "client.key";
// clang-format off
openssl({"ecparam", "-genkey", "-name", "prime256v1", "-out", caKey.string()});
openssl({"req", "-new", "-x509", "-days", "1", "-key", caKey.string(), "-out", caCert.string(), "-subj", "/CN=TestCA"});
openssl({"ecparam", "-genkey", "-name", "prime256v1", "-out", serverKey.string()});
openssl({"req", "-new", "-key", serverKey.string(), "-out", (tmpDir / "server.csr").string(), "-subj", "/CN=localhost"});
openssl({"x509", "-req", "-in", (tmpDir / "server.csr").string(), "-CA", caCert.string(), "-CAkey", caKey.string(), "-CAcreateserial", "-out", serverCert.string(), "-days", "1"});
openssl({"ecparam", "-genkey", "-name", "prime256v1", "-out", clientKey.string()});
openssl({"req", "-new", "-key", clientKey.string(), "-out", (tmpDir / "client.csr").string(), "-subj", "/CN=TestClient"});
openssl({"x509", "-req", "-in", (tmpDir / "client.csr").string(), "-CA", caCert.string(), "-CAkey", caKey.string(), "-CAcreateserial", "-out", clientCert.string(), "-days", "1"});
// clang-format on
#ifndef _WIN32 /* FIXME: Can't yet start processes on windows */
auto args = serverArgs();
serverPid = startProcess(
[&] {
if (chdir(cacheDir.c_str()) == -1)
_exit(1);
std::vector<char *> argv;
argv.push_back(const_cast<char *>("openssl"));
for (auto & a : args)
argv.push_back(const_cast<char *>(a.c_str()));
argv.push_back(nullptr);
execvp("openssl", argv.data());
_exit(1);
},
{.dieWithParent = true});
#endif
/* As an optimization, sleep for a bit to allow the server to come up to avoid retrying when connecting.
This won't make the tests fail, but does make them run faster. We don't need to overcomplicate by waiting
for the port explicitly - this is enough. */
std::this_thread::sleep_for(std::chrono::milliseconds(50));
/* Create custom FileTransferSettings with our test CA certificate.
This avoids mutating global settings. */
testFileTransferSettings = std::make_unique<FileTransferSettings>();
testFileTransferSettings->caFile = caCert;
testFileTransfer = makeFileTransfer(*testFileTransferSettings);
}
void HttpsBinaryCacheStoreTest::TearDown()
{
serverPid.kill();
delTmpDir.reset();
testFileTransferSettings.reset();
}
std::vector<std::string> HttpsBinaryCacheStoreTest::serverArgs()
{
return {
"s_server",
"-accept",
std::to_string(port),
"-cert",
serverCert.string(),
"-key",
serverKey.string(),
"-WWW", /* Serve from current directory. */
"-quiet",
};
}
std::vector<std::string> HttpsBinaryCacheStoreMtlsTest::serverArgs()
{
auto args = HttpsBinaryCacheStoreTest::serverArgs();
/* With the -Verify option the client must supply a certificate or an error occurs, which is not the
case with -verify. */
args.insert(args.end(), {"-CAfile", caCert.string(), "-Verify", "1", "-verify_return_error"});
return args;
}
ref<TestHttpBinaryCacheStoreConfig> HttpsBinaryCacheStoreTest::makeConfig()
{
auto res = make_ref<TestHttpBinaryCacheStoreConfig>(
settings,
ParsedURL{
.scheme = "https",
.authority =
ParsedURL::Authority{
.host = "localhost",
.port = port,
},
},
TestHttpBinaryCacheStoreConfig::Params{});
res->pathInfoCacheSize = 0; /* We don't want any caching in tests. */
return res;
}
ref<TestHttpBinaryCacheStore> HttpsBinaryCacheStoreTest::openStore(ref<TestHttpBinaryCacheStoreConfig> config)
{
return config->openTestStore(ref<FileTransfer>{testFileTransfer});
}
} // namespace nix::testing
@@ -0,0 +1,103 @@
#pragma once
///@file
#include <gtest/gtest.h>
#include <gmock/gmock.h>
#include "nix/store/tests/libstore-network.hh"
#include "nix/store/http-binary-cache-store.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
#include "nix/store/local-binary-cache-store.hh"
#include "nix/util/file-system.hh"
#include "nix/util/processes.hh"
#include "nix/store/tests/test-main.hh"
namespace nix::testing {
class TestHttpBinaryCacheStoreConfig;
/**
* Test shim for testing. We don't want to use the on-disk narinfo cache in unit
* tests.
*/
class TestHttpBinaryCacheStore : public HttpBinaryCacheStore
{
public:
TestHttpBinaryCacheStore(const TestHttpBinaryCacheStore &) = delete;
TestHttpBinaryCacheStore(TestHttpBinaryCacheStore &&) = delete;
TestHttpBinaryCacheStore & operator=(const TestHttpBinaryCacheStore &) = delete;
TestHttpBinaryCacheStore & operator=(TestHttpBinaryCacheStore &&) = delete;
TestHttpBinaryCacheStore(ref<HttpBinaryCacheStoreConfig> config, ref<FileTransfer> fileTransfer)
: Store{*config}
, BinaryCacheStore{*config}
, HttpBinaryCacheStore(config, fileTransfer)
{
diskCache = nullptr; /* Disable caching, we'll be creating a new binary cache for each test. */
}
void init() override;
};
class TestHttpBinaryCacheStoreConfig : public HttpBinaryCacheStoreConfig
{
public:
TestHttpBinaryCacheStoreConfig(nix::Settings & settings, ParsedURL url, const Store::Config::Params & params)
: StoreConfig(settings, params)
, HttpBinaryCacheStoreConfig(settings, url, params)
{
}
ref<TestHttpBinaryCacheStore> openTestStore(ref<FileTransfer> fileTransfer) const;
};
class HttpsBinaryCacheStoreTest : public virtual LibStoreNetworkTest
{
std::unique_ptr<AutoDelete> delTmpDir;
public:
Settings settings = getTestSettings();
static void SetUpTestSuite()
{
initLibStore(/*loadConfig=*/false);
}
protected:
std::filesystem::path tmpDir, cacheDir;
std::filesystem::path caCert, caKey, serverCert, serverKey;
std::filesystem::path clientCert, clientKey;
Pid serverPid;
uint16_t port = 8443;
std::shared_ptr<Store> localCacheStore;
/**
* Custom FileTransferSettings with the test CA certificate.
* This is used instead of modifying global settings.
*/
std::unique_ptr<FileTransferSettings> testFileTransferSettings;
/**
* FileTransfer instance using our test settings.
* Initialized in SetUp().
*/
std::shared_ptr<FileTransfer> testFileTransfer;
static void openssl(Strings args);
void SetUp() override;
void TearDown() override;
virtual std::vector<std::string> serverArgs();
ref<TestHttpBinaryCacheStoreConfig> makeConfig();
ref<TestHttpBinaryCacheStore> openStore(ref<TestHttpBinaryCacheStoreConfig> config);
};
class HttpsBinaryCacheStoreMtlsTest : public HttpsBinaryCacheStoreTest
{
protected:
std::vector<std::string> serverArgs() override;
};
} // namespace nix::testing
Loaded 100 of 512 files, more files were not shown because too many files have changed in this diff. Show more