Compare commits

...
Author SHA1 Message Date
John Ericson bec4da9e26 NarCache improvements
- Separate implementation, abstract implements

- `NarCache` work in constant memory

  This is using `RestoreSink` to dedup some write-side IO, but I am not
  so sure that is a good idea. We'll continue reworking it.

- Use pathlocks to avoid download storms

- Use `Descriptor`-based logic, not `pathExists` to avoid symlink races
  if anything is deleted.
2026-01-26 15:30:09 -05:00
John Ericson d3116dc764 Merge pull request #15043 from obsidiansystems/settings-split-0
Split out `AutoAllocateUidSettings`
2026-01-26 19:25:51 +00:00
John Ericson b190548c83 Merge pull request #15040 from NixOS/factor-out-nar-cache-0
Factor out `NarCache` from `RemoteFSAccessor`
2026-01-26 19:19:39 +00:00
John EricsonandAmaan Qureshi ab56ac49e3 libstore: split out AutoAllocateUidSettings
Follows the same pattern as `GCSettings`: extract UID allocation
settings
into a dedicated struct that Settings inherits privately from.

The current settings infrastructure prevents correct data modeling that
would allow `autoAllocateUids` to be a
`std::optional<AutoAllocateUidSettings>`.
To compensate, the getter `getAutoAllocateUidSettings()` returns a
pointer -
nullptr when disabled, providing the optional-like semantics we want.

Co-authored-by: Amaan Qureshi <git@amaanq.com>
2026-01-26 13:43:41 -05:00
Sergei Zimmerman ed9d8af93d Merge pull request #15059 from lovesegfault/fix-aws-logs
feat(libstore/aws-creds): route AWS CRT logs through Nix logger
2026-01-25 18:06:03 +00:00
Sergei Zimmerman e3b788b4ca tests/nixos/s3-binary-cache-store: Drop superfluous prints
As requested in review.
2026-01-25 19:40:30 +03:00
Bernardo Meurer Costa 3b8b764e29 feat(libstore/aws-creds): route AWS CRT logs through Nix logger
Previously AWS CRT logs went directly to stderr via ApiHandle::InitializeLogging,
causing log spam that didn't respect Nix's verbosity settings.

This implements a custom aws_logger using the aws-c-common C API that:
- Routes all AWS logs through nix::logger
- Maps AWS log levels conservatively (ERROR/WARN -> lvlInfo) since the SDK
  treats expected conditions like missing IMDS as errors
- Prefixes messages with (aws) for clarity
- Respects Nix's verbosity flags (-v, -vv, etc.)
2026-01-25 19:40:29 +03:00
Jörg Thalheim 2eb19a6353 Merge pull request #13030 from vlaci/mtls-auth
libstore/filetransfer: add support for MTLS authentication
2026-01-25 13:58:12 +00:00
John Ericson e8e3c30dfc Merge pull request #15076 from NixOS/prepare-for-25.11
Prepare for nixpkgs 25.11, enable S3 support in static builds
2026-01-24 23:11:18 +00:00
John Ericson a3f2d2b3e9 Merge pull request #15075 from NixOS/chmod-wrapper
Share the exception-using `chmod` wrapper with more code
2026-01-24 22:51:42 +00:00
Sergei Zimmerman 64458acde2 packaging: Fix static builds with S3 support, enable by default
aws-crt-cpp doesn't provide pkg-config files and has a bunch of transitive
deps, so switch to cmake for resolving the dependency.
2026-01-25 01:26:23 +03:00
John Ericson 6e2e53a8d2 Share the exception-using chmod wrapper with more code
It is not just useful to `DerivationBuilder`.
2026-01-24 17:03:48 -05:00
Sergei Zimmerman dcaaf2c65f dev-shell: Use stdenv.hostPlatform instead of hostPlatform
This is now a warning in 25.11:

> evaluation warning: 'hostPlatform' has been renamed to/replaced by 'stdenv.hostPlatform'
2026-01-25 00:50:56 +03:00
Sergei Zimmerman c4c0aee4f1 tests/nixos: Drop otherNixes.nix_2_3, replace with 2_18
Since [1] there's no way to run 2.3 anymore and overrides wouldn't be very
helpful. Let's instead use 2.18, which is the baseline for nixpkgs.
2026-01-25 00:50:55 +03:00
Sergei Zimmerman 0f22d60c7e tests/nixos: Specify -f argument to mount
Otherwise we barf on btrfs:

machine # [   17.027621] EXT4-fs error (device vdb): ext4_lookup:1819: inode #2476: comm nix: iget: checksum invalid
machine # error: getting status of '/mnt/nix/store/j8645yndikbrvn292zgvyv64xrrmwdcb-bash-5.3p3': Bad message
machine # checking '/nix/store/m3954qff15v7z1l6lpyqf8v2h47c7hv2-mailcap-2.1.54'...
machine # checking '/nix/store/xh1ff9c9c0yv1wxrwa5gnfp092yagh7v-tzdata-2025b'...
machine # [   17.172793] EXT4-fs error (device vdb): ext4_lookup:1819: inode #1777: comm nix: iget: checksum invalid
machine # error: getting status of '/mnt/nix/store/xh1ff9c9c0yv1wxrwa5gnfp092yagh7v-tzdata-2025b/share/zoneinfo/leap-seconds.list': Bad message
2026-01-25 00:50:54 +03:00
John Ericson 943c18f9fe Merge pull request #15072 from NixOS/fix-interrupted-linux-derivation-builder
Fix destruction of DerivationBuilder implementations
2026-01-24 21:16:34 +00:00
Sergei Zimmerman b752c5cb64 Fix destruction of DerivationBuilder implementations
This unsures that we call the correct virtual functions when destroying a particular
DerivationBuilder.

Usually the order of destructors is in the reverse order of inheritance:

ChrootLinuxDerivationBuilder -> ChrootDerivationBuilder -> DerivationBuilderImpl

autoDelChroot was being destroyed before the DerivationBuilderImpl::killChild was
run and it would fail to clean up the chroot directory, since there were still processes
writing to it. Note that ChrootLinuxDerivationBuilder::killSandbox was never run in
the interrupted case at all, since virtual functions in destructors do not call derived class
methods.

I could reproduce the issue with the following derivation:

let
  pkgs = import <nixpkgs> { };
in
pkgs.runCommand "chroot-cleanup-race" { } ''
  mkdir -p $out

  for i in $(seq 1 200); do
    (
      mkfifo $out/fifo$i
      cat $out/fifo$i > /dev/null &

      while true; do
        : > $out/file$i
      done
    ) &
  done

  sleep 0.05
  echo done > $out/main
''

While interrupting it manually when it would hang.

Wrapping the unique pointer in a custom deleter function we can run all
of the necessary clean up code consistently and calling the right virtual
functions. Ideally we'd have a lint that bans the usage of virtual functions
in destructors completely.
2026-01-24 23:31:11 +03:00
John Ericson b7d07e42dc Merge pull request #15071 from roberth/fix-concurrent-failure-bug
tests: fix sandbox-paths in cancelled-builds test
2026-01-24 19:48:25 +00:00
Sergei Zimmerman 0f17a1f655 libutil-tests: Add unit tests for https binary cache stores with mTLS
This addresses the concerns with network isolation that have been raised
previously [1] by only running the tests by default in a network namespace.
This way all networks tests are independent of each other and do not bind
to ports in the host namespace.

This is much neater than doing these sorts of tests in functional suite.

[1]: https://github.com/NixOS/nix/pull/14266#issuecomment-3411261285
2026-01-24 21:59:59 +03:00
Damien Diederen 36b0bebe25 http-binary-cache-store: Add 'tls-certificate' and 'tls-private-key' settings
Those are set via the store's URI, e.g.:

    https://substituter.invalid?tls-certificate=/path/to/cert.pem&tls-private-key=/path/to/key.pem
2026-01-24 21:59:58 +03:00
Robert Hensing 7b4444f174 tests: fix sandbox-paths in cancelled-builds test
Don't add the whole store to sandbox-paths unconditionally. Exposing
the entire store defeats the purpose of sandboxing, and when the test
store is the same as the system store (NixOS VM), it causes an obscure
"Permission denied" error.

Only add sandbox-paths when not on NixOS, indicating a separate test
store that needs access to system store build tools.
2026-01-24 19:55:50 +01:00
John Ericson aa17b75601 Merge pull request #15054 from obsidiansystems/unprivileged-test
Add new VM test with unprivileged daemon user
2026-01-23 19:49:46 +00:00
Jörg Thalheim 18176d2678 ignore-gc-delete-failure: add release note 2026-01-23 14:08:05 -05:00
John Ericson 1a17c9d02b Merge pull request #15051 from amaanq/split-build-log-settings
libstore: split out `LogFileSettings`
2026-01-23 18:58:51 +00:00
Artemis Tosini 94907eb37a Add new VM test with unprivileged daemon user
All current NixOS functional VM tests have a daemon as root with the
tests running as different unprivileged users.
The new `functional_unprivileged-daemon` test runs the daemon and the
nix functional tests as separate unprivileged users.
Users may want to run an unprivileged daemon on non-NixOS systems
where the administrator does not fully trust nix, but multiple users
want to use nix for their own purposes. It could also be useful in
concert with an overlay-mount store, where the nix daemon cannot
modify the derivations used by the system, and thus a nix vulnerability
would not lead to root code execution.
2026-01-23 13:31:16 -05:00
Artemis TosiniandJohn Ericson 2f1ce8900b Ignore delete failures during garbage collection
When running nix as an unprivileged user it may not be able to write to
all paths in the nix store. Ignore deletion failures to fix tests that
run `nix-collect-garbage` in this configuration.

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2026-01-23 13:31:07 -05:00
Amaan Qureshi 98178e24d0 libstore: split out LogFileSettings 2026-01-23 13:17:15 -05:00
Eelco Dolstra 4c6ad728d0 Merge pull request #15058 from amaanq/split-gc-settings
libstore: split out `GCSettings`
2026-01-23 16:18:30 +00:00
John Ericson 83360cd7b7 Merge pull request #14972 from roberth/fix-concurrent-failure-bug
Fix concurrent builder failure empty message bugs
2026-01-23 00:50:47 +00:00
John Ericson 2a21bd6d0a Merge pull request #15056 from NixOS/setns-explicit-arguments
linux-derivation-builder: Explicitly specify nstype for setns calls i…
2026-01-23 00:44:46 +00:00
Sergei Zimmerman 5570c31b30 Merge pull request #15062 from NixOS/drop-magic-nix-cache
ci: Drop magic-nix-cache
2026-01-22 23:52:53 +00:00
John Ericson eead36de18 Merge pull request #15061 from amaanq/build-result-throw
build-result: throw better
2026-01-22 23:23:28 +00:00
Sergei Zimmerman dae41e06e8 ci: Drop magic-nix-cache
We are now seeing. I guess we are out with the cache. When the API responds with 418 (I'm a teapot)
it seems like the only reasonable solution is to oblige.

error: unable to download 'http://127.0.0.1:37515/7ms9f25xyxavf32pvdc3vb28nzzmkbn3.narinfo': HTTP error 418
       response body:
       GitHub API error: GitHub Actions Cache throttled Magic Nix Cache. Not trying to use it again on this run.
2026-01-23 02:02:10 +03:00
Amaan Qureshi daba5f6386 build-result: throw better 2026-01-22 17:40:01 -05:00
Amaan Qureshi 6f0fe5636d libstore: split out GCSettings
This PR follows the same approach as #15043 and the
[`LogFileSettings`](https://github.com/NixOS/nix/pull/15051)
extraction:

- `GCSettings` struct inherits from virtual `Config`
- `Settings` privately inherits from it
- Accessed through `getGCSettings()`

The new method on `LocalStoreConfig` anticipates on making these
settings per-store. 0b606aad46 added both
the autoGC and periodic wakeups, which is why we think they are related.
2026-01-22 17:06:08 -05:00
John Ericson 73a727f3d2 Merge pull request #15047 from lovesegfault/fix-15023
fix(libstore/filetransfer): restart source before upload retries
2026-01-22 19:45:48 +00:00
Sergei Zimmerman cc8f4912f5 linux-derivation-builder: Explicitly specify nstype for setns calls in addDependencyImpl
We already use file descriptors for this, so 0 is perfectly fine here, but this still
serves as a sanity check and slightly more self-documenting.
2026-01-22 22:29:22 +03:00
John Ericson 6276642164 Merge pull request #15050 from amaanq/sqlite-settings-struct
libstore: add `SQLite::Settings` struct for explicit configuration
2026-01-22 18:27:53 +00:00
Bernardo Meurer Costa fbd787b910 fix(libstore/filetransfer): restart source before upload retries
When an upload fails with a transient HTTP error (e.g., S3 rate limiting
with HTTP 503), retries would fail with "curl error: Failed to open/read
local data from file/application" because the upload source was already
exhausted from the previous attempt.

Restart the source in init() to ensure it's at the beginning for both
first attempts (no-op) and retries (necessary fix).

Fixes: #15023
2026-01-22 18:25:36 +00:00
Amaan Qureshi 751a0f40bc libstore: add SQLite::Settings struct for explicit configuration
Progress on #5638

Replace the SQLite constructor's mode parameter with a Settings struct
that includes both the open mode and useWAL flag. This makes the
dependency on useSQLiteWAL explicit at call sites rather than having
it read from the global settings inside the constructor.

All call sites now explicitly pass settings.useSQLiteWAL, preparing
for downstream work where stores can pass their own settings instead
of relying on the global.
2026-01-22 12:06:26 -05:00
Eelco Dolstra 857a2053ad Merge pull request #15048 from lovesegfault/fix-15019
fix(libstore/filetransfer): skip Accept-Encoding header for S3 SigV4 requests
2026-01-22 16:31:10 +00:00
John Ericson 087b6c4dc2 Merge pull request #15044 from obsidiansystems/remove-nixPrefix
Remove `Settings::nixPrefix`
2026-01-22 16:25:39 +00:00
John Ericson 3b8c408108 Merge pull request #15045 from obsidiansystems/remove-nixDataDir
Remove `nixDataDir`, `NIX_DATA_DIR`
2026-01-22 16:23:44 +00:00
John Ericson b7ddbb8e2d Merge pull request #15039 from NixOS/read-link-at
libutil: Add unix::readLinkAt function
2026-01-22 15:11:54 +00:00
Bernardo Meurer Costa fcfa1dc8ab fix(libstore/filetransfer): skip Accept-Encoding header for S3 SigV4 requests
Some S3-compatible services (like GCS) modify the Accept-Encoding header
in transit, which breaks AWS SigV4 signature verification since curl's
implementation signs all headers including Accept-Encoding.

Fixes: #15019
2026-01-22 06:35:01 +00:00
John Ericson ba219cb047 Merge pull request #15046 from juhp/patch-3
nix config check: improve error when no nix-env in PATH
2026-01-22 05:31:05 +00:00
Jens Petersen 9fa69276c4 nix config check: improve error when no nix-env
It is possible that the `nix` executable is installed but not `nix-env`
(this may be unusual but for example in Fedora we have a separate
`nix-legacy` subpackage, which includes the `nix-env` symlink).

The current error message:
```
$ nix config check --verbose
Running checks against store uri: local
[FAIL] Multiple versions of nix found in PATH:

```
when there is no nix-env in PATH is confusing.

This change makes the error message precise for the missing nix-env case.
2026-01-22 12:49:17 +08:00
John Ericson 98f6881d11 Remove nixDataDir, NIX_DATA_DIR
Since 25300c0ecd it is dead code.
2026-01-21 23:23:23 -05:00
John Ericson 56c9d5f04e Remove Settings::nixPrefix
It has been dead code since c9f51e8705
2026-01-21 23:16:27 -05:00
John Ericson 07a3171fb9 Merge pull request #15028 from NixOS/copyFdRange-improvements
Deduplicate `copyFdRange` with new `readOffset`
2026-01-22 00:28:01 +00:00
John Ericson fe8f574471 Clean up NarAccessor construction
We had a minor combinatorial explosion of ways to do things. We can get
rid of those by just having the caller call `parseNarListing` intead.
2026-01-21 19:25:52 -05:00
Eelco DolstraandJohn Ericson 0b2dffefea Factor out NarCache from RemoteFSAccessor
Use

```
git show --color-moved --patience --color-moved-ws=ignore-all-space
```

to review and see that this is mostly code motion.

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2026-01-21 19:15:51 -05:00
Sergei Zimmerman 52100c6ee1 libutil: Add unix::readLinkAt function
This will be used for TOCTOU-free NAR serialisation and recursive copying.
2026-01-22 02:50:39 +03:00
John Ericson fa53a9cec8 Deduplicate copyFdRange with new readOffset 2026-01-21 18:16:28 -05:00
Eelco Dolstra 44dce7a3d1 Merge pull request #15029 from amaanq/signature-type-core
libutil: add `Signature` struct for typed signatures
2026-01-21 21:17:01 +00:00
Amaan Qureshi 12ef043655 libutil: add Signature struct for typed signatures
Introduce a new `Signature` struct that represents a cryptographic
signature
along with the key name that produced it. This provides:

- Structured representation instead of colon-separated strings
- Type-safe parsing with `Signature::parse()`
- Serialization with `to_string()`
- JSON serialization/deserialization
- Batch parsing with `parseMany<Container>()`
- Batch serialization with `toStrings()`

This is scaffolding for future changes that will use this type
throughout the codebase.
2026-01-21 11:51:46 -05:00
Eelco Dolstra 7a40df3510 Merge pull request #15037 from NixOS/realise-path
realisePath(): Move into EvalState
2026-01-21 16:24:42 +00:00
Eelco Dolstra 4f733f736e realisePath(): Move into EvalState
This allows it to be used by primops defined outside of primops.cc.
2026-01-21 14:43:58 +01:00
John Ericson c6d07ec0aa Merge pull request #15030 from xokdvium/path-fmt-squash-double-quotes
libutil: Add PathFmt wrapped type for formatting fs::path, fix all double-quoting issues
2026-01-21 04:28:30 +00:00
John Ericson e2cd5679eb Merge pull request #15031 from NixOS/bump-magic-nix-cache
ci: Bump magic-nix-cache to disable on 429
2026-01-21 04:11:31 +00:00
Sergei Zimmerman 1555677cd5 ci: Bump magic-nix-cache to disable on 429 2026-01-21 06:14:34 +03:00
Sergei Zimmerman 6dd89b5a2a libutil: Add PathFmt wrapped type for formatting fs::path, fix all double-quoting issues
This will once and for all get rid of all double-quoting issues. On windows the quoting
is doubly bad because it escaped all \ to \\, which is very bad for error messages. In
order to prevent future regression std::filesystem::path formatting now must use a special
type PathFmt (like Magenta). In the future we could even change how we render filesystem paths.
2026-01-21 06:06:19 +03:00
Sergei Zimmerman 73beff89cb libutil: Fix mingw build 2026-01-21 04:54:54 +03:00
tomberek f429d8d4aa Merge pull request #14766 from pkpbynum/capi/query-path-from-hash-part
C API: Add query_from_hash_part to Store API
2026-01-20 16:39:05 +00:00
John Ericson 017a247e63 Merge pull request #15025 from cole-h/cole-h/push-qmyswwomnsnl
libutil: add missing tracking URLs for external-builders and blake3-h…
2026-01-20 16:17:30 +00:00
Eelco Dolstra 4a267f720e Merge pull request #14998 from NixOS/fix-remote-store-nar-from-path
libstore: Do not mark connections as bad when RemoteStore::narFromPath is called as a coroutine
2026-01-20 16:08:07 +00:00
John Ericson 490f6eeba5 Merge pull request #15021 from OPNA2608/fix/libstore-ppc64
libstore: make withAWS depend on aws-c-common availability
2026-01-20 15:44:35 +00:00
Cole Helbling c398dd7cbd libutil: add missing tracking URLs for external-builders and blake3-hashes 2026-01-20 07:32:26 -08:00
OPNA2608 37834c5e58 libstore: make withAWS depend on platform's aws-c-common availability 2026-01-20 14:39:57 +01:00
John Ericson 67a99db5be Merge pull request #15020 from xokdvium/more-enum-compression
Use CompressionAlgo enum throughout
2026-01-20 04:50:56 +00:00
John Ericson a59bc630aa Merge pull request #15015 from NixOS/catch-system-error
Catch `SystemError` in portable code
2026-01-20 04:41:29 +00:00
Sergei Zimmerman 6ba067831a Use CompressionAlgo throughout
Instead of the stringly typed code we should use an enum class, this is
more clear and less error-prone. Also adds settings implementations for
CompressionAlgo and std::optional<CompressionAlgo>. The first is used
for NAR compression, since we never accepted empty strings there:

error: unknown compression method ''

The other one is used for optional .narinfo, .ls, and log/ compression.
Those treated empty strings as compression being disabled. The same exact
semantics is kept.

This has the benefit of improving error messages for incorrect values:

error: option 'compression' has invalid value 'bz'
       Did you mean one of br, xz or lz4?
2026-01-20 04:35:16 +03:00
Sergei Zimmerman b24df97a11 binary-cache-store: Update compression setting documentation to match reality
The docs were out of date. Since 8a0c00b856 Nix
supports all compression algorithms exposed by libarchive (if it's built with
native support for them). Let's be honest about it in the docs.
2026-01-20 03:31:30 +03:00
Sergei Zimmerman 556974f33b libutil: Remove unused overload of unpackTarfile 2026-01-20 01:32:43 +03:00
Sergei Zimmerman b40b786839 libstore/remote-store: Add checkInterrupt in openConnectionWrapper
This avoids the wall of text like, because ThreadPool doesn't print interrupts
on shutdowns.

error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
error (ignored): opening a connection to remote store 'ssh-ng://127.0.0.1' previously failed
2026-01-19 22:28:38 +03:00
Sergei Zimmerman 9e496f9af2 libutil: Make Pid destructor more robust
Without this we can abort by throwing an exception in the destructor:

[24/635/2958 copied (3.8/26.0 GiB)] copying path '/nix/store/ncd2iic2nwxwhqsf4gp9sdybkwnwz20b-ruby3.3-mini_portile2-2.8.9' from 'ssh-ng://localhost:22'

Nix crashed. This is a bug. Please report this at https://github.com/NixOS/nix/issues with the following information included:

Exception: nix::Interrupted: error: interrupted by the user
Stack trace:
 0# 0x00000000004AFFE9 in result/bin/nix
 1# 0x00007F946290A1AA in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
 2# __cxa_call_terminate in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
 3# __gxx_personality_v0 in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
 4# 0x00007F946283FA19 in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libgcc_s.so.1
 5# _Unwind_RaiseException in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libgcc_s.so.1
 6# __cxa_throw in /nix/store/cf1a53iqg6ncnygl698c4v0l8qam5a2q-gcc-14.3.0-lib/lib/libstdc++.so.6
 7# 0x00007F94635D82D0 in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
 8# nix::Pid::wait() in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
 9# nix::Pid::~Pid() in /nix/store/9wrnk0nizdwba4sy9lg3h0xd30pg1x5a-nix-util-2.34.0pre/lib/libnixutil.so.2.34.0
2026-01-19 22:28:37 +03:00
Sergei Zimmerman 726e924bd7 libstore: Do not mark connections as bad when RemoteStore::narFromPath is called as a coroutine
forced_unwind is thrown by Boost.Context when destroying the coroutine.
This lead to us resetting the remote connection for each narFromPath
with the ssh-ng:// store, so copying was very slow.
2026-01-19 22:28:34 +03:00
Eelco Dolstra 3a421388dd Merge pull request #15006 from roberth/doc-nix-cache-info
doc: add nix-cache-info format documentation
2026-01-19 15:16:20 +00:00
John Ericson a32c139379 Catch SystemError in portable code
This will ensure this catching works on Windows too, not just Unix.
2026-01-18 18:53:45 -05:00
John Ericson 63344d3a3b Merge pull request #15013 from NixOS/seekable-nar-bytes-thread-safe-pread
nar-accessor: Fix thread safety of `seekableGetNarBytes`, use `Sink`, reduce memory usage of `nix store cat`
2026-01-18 23:26:08 +00:00
Sergei Zimmerman 4db68c28c1 treewide: Add missing overrides of streaming readFile, make readFile non-virtual
This makes all addToStore operations that use these source accessors
constant memory regardless of file sizes. Also make the other overload
altogether and relegate it to the base class as a non-virtual method to
avoid such mistakes.
2026-01-19 01:15:26 +03:00
Sergei Zimmerman 6ba468805b libutil: Factor out copyFdRange
This factors out the helper function from seekableGetNarBytes into copyFdRange
and adds some more sanity checks for offset/length truncation/wrapping at that
API boundary where we work with NAR-style offsets and convert to native off_t.
2026-01-19 01:15:23 +03:00
Sergei Zimmerman d25ab60d66 nix/cat: Use streaming version of readFile
This reduces the memory overhead of nix store cat down to constant memory
with a local-nar-cache.
2026-01-19 00:52:05 +03:00
Sergei Zimmerman 656e1fc659 nar-accessor: Fix thread safety of seekableGetNarBytes, use Sink
Instead of mutating the file pointer we can instead safely do
preads. That makes the local-nar-info cache once again thread safe
without the overhead of reopening the file that we used to have prior
to b9b6defca6 which broke the thread safety
by persisting the file descriptor.
2026-01-19 00:51:57 +03:00
Sergei Zimmerman 054de385d8 Merge pull request #15011 from trofi/lixnu-openat-sysno
libutil: fix `linux` build on fresh `glibc` and `gcc`
2026-01-17 17:16:43 +00:00
Sergei Trofimovich 3256aba6a2 libutil: fix linux build on fresh glibc and gcc
Without the change the build fails for me as:

    ../unix/file-descriptor.cc:404:70: error: 'RESOLVE_BENEATH' was not declared in this scope
      404 |         dirFd, path.rel_c_str(), flags, static_cast<uint64_t>(mode), RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS);
          |                                                                      ^~~~~~~~~~~~~~~

This happens for 2 reasons:
1. `__NR_openat2` constant was not pulled in from the according headers
   and as a result `<linux/openat2.h>` was not included.
2. `define HAVE_OPENAT2 0` build is broken: refers to missing
   `RESOLVE_BENEATH` normally pulled in from `<linux/openat2.h>`

This changes fixes both.
2026-01-17 13:01:22 +00:00
Robert Hensing b3df7f8a3d doc: add nix-cache-info format documentation
Document the nix-cache-info file format used by binary caches, including
the StoreDir, WantMassQuery, and Priority fields, their behavior, and
links to related store options.
2026-01-17 13:31:32 +01:00
Sergei Zimmerman af7c7b6723 Merge pull request #14973 from NixOS/windows-known-folders
Use known folders for nix data on windows
2026-01-16 23:58:17 +00:00
John Ericson d6fa3e3b50 Merge pull request #15010 from amaanq/fix-test-signature-format
tests: use valid signature format in protocol characterization tests
2026-01-16 23:20:41 +00:00
Sergei ZimmermanandJohn Ericson 5d2938520c Use known folders for nix data on windows
This is the usual conventions on windows.

See https://learn.microsoft.com/en-us/windows/win32/shell/knownfolderid and
https://github.com/adrg/xdg for examples of the mapping of XDG paths to Windows
known folders.

Additionally, on Windows, this allows us to dispense with a hard-coded
default for `nixConfDir`, which is both nice (fewer compile-time
configuration options) and necessary, because we don't know what drive
the `ProgramData` directory will live on.

Tested on wine.

Co-Authored-By: John Ericson <John.Ericson@Obsidian.Systems>
2026-01-17 02:12:41 +03:00
Amaan Qureshi 51ca872c60 tests: use valid signature format in realisation protocol tests
The test data was using invalid signature strings like "asdf" and
"qwer" which don't follow the required "name:base64signature" format.
This updates them to use properly formatted signatures with valid
base64-encoded data.
2026-01-16 17:49:46 -05:00
John Ericson 1e1d9f28ba Merge pull request #14754 from NixOS/structured-attrs-cleanup
Clean up structured attrs parsing using JSON utils
2026-01-16 22:13:18 +00:00
John Ericson 54c62782f5 Move environmentVariablesCategory to libcmd
It does not belong in libutil.
2026-01-16 16:01:04 -05:00
Eelco Dolstra 162b0072a7 Merge pull request #15008 from NixOS/typo
Fix typo spotted by coderabbit
2026-01-16 20:22:28 +00:00
John Ericson eb653e5928 Clean up structured attrs parsing using JSON utils 2026-01-16 14:55:18 -05:00
Eelco Dolstra 33ce87276f Fix typo spotted by coderabbit 2026-01-16 20:08:56 +01:00
Eelco Dolstra f1d468e840 Merge pull request #14965 from shlevy/controllerless-delegation
systemd: Delegate cgroup management without turning on controllers.
2026-01-16 16:16:27 +00:00
Eelco Dolstra 3fb0fb00ae Merge pull request #15004 from roberth/docs
Add Nix32 encoding documentation
2026-01-16 15:36:26 +00:00
Robert Hensing 3374fdc04a Add Nix32 encoding documentation
Document the Nix32 base-32 variant used for store path digests and
hash output. The new page covers:
- The 32-character alphabet (omitting e, o, u, t)
- Byte order differences from base-16 encoding

Also update references throughout the manual to link to the new page.
2026-01-16 11:28:55 +01:00
John Ericson a9b1a52623 Merge pull request #14949 from iljah/patch-3
Add example commands to source installation
2026-01-15 17:02:45 +00:00
Eelco Dolstra f162bb62f7 Merge pull request #14994 from NixOS/curl-scalability
resolverCallbackWrapper(): Catch exceptions
2026-01-14 22:35:50 +00:00
Eelco Dolstra 292156c336 Merge pull request #14995 from NixOS/daemon-proto-cleanup
Remove more obsolete daemon operations
2026-01-14 21:26:22 +00:00
Eelco Dolstra 3cb42b581a Remove WorkerProto::Op::{QueryFailedPaths,ClearFailedPaths}
Support for failed paths was removed in
8cffec8485 (April 2016).
2026-01-14 21:30:24 +01:00
Eelco Dolstra 17295066e8 Remove WorkerProto::Op::QueryReferences
This has been obsolete since e0204f8d46
(April 2016).
2026-01-14 21:25:33 +01:00
Eelco Dolstra 036738be11 Remove WorkerProto::Op::QueryPathHash
This has been obsolete since e0204f8d46
(April 2016).
2026-01-14 21:22:11 +01:00
Robert Hensing 3c3ceb18e9 DerivationTrampolineGoal: improve error message wording
Change "cannot build missing derivation" to "failed to obtain derivation of"
since the path (e.g. '...drv^out') is a derivation output, not a derivation.

The message could be improved further to resolve ambiguity when multiple
outputOf links are involved, but for now we err on the side of brevity
since this message is already merged into larger error messages with
other context from the Worker and CLI.
2026-01-14 20:42:20 +01:00
Robert Hensing 68f549def4 buildPathsWithResults: don't report cancelled goals as failures
When !keepGoing and a goal fails, other goals are cancelled and
remain with exitCode == ecBusy. These cancelled goals have a default
BuildResult::Failure{} with empty errorMsg.

Previously, buildPathsWithResults would return these cancelled goals,
and throwBuildErrors would report them as failures. When only one such
cancelled goal was present, it would throw an error with an empty
message like:

    error: build of '/nix/store/...drv^*' failed:

Now we skip goals with ecBusy since their state is indeterminate.
Cancelled goals could be reported, but this keeps the output relevant.
Other indeterminate goal states were already not being reported, for
instance: derivations that weren't started for being blocked on a
concurrency limit, or blocked on a currently building dependency.
2026-01-14 20:42:20 +01:00
Robert Hensing 3fd85c7d64 tests: don't expect cancelled goals to be reported as failures
When keepGoing=false and a build fails, other goals are cancelled.
Previously, these cancelled goals were reported in the "build of ...
failed" error message alongside actual failures. This was misleading
since cancelled goals didn't actually fail - they were never tried.

Update the test to expect only the actual failure (hash mismatch) to
be reported, not the cancelled goals.
2026-01-14 20:42:20 +01:00
Robert Hensing 25eb07a91b DerivationTrampolineGoal: use doneFailure to set buildResult
DerivationTrampolineGoal is the top-level goal whose buildResult is
returned by buildPathsWithResults. When it failed without setting
buildResult.inner, buildPathsWithResults would return failures with
empty errorMsg, producing error messages like:

  error: failed to build attribute 'checks.x86_64-linux.foo',
  build of '/nix/store/...drv^*' failed:

(note the empty message after "failed:")

Use the new doneFailure helper to ensure buildResult is populated
with meaningful error information.
2026-01-14 20:42:20 +01:00
Eelco Dolstra 0a632cbc3a Merge pull request #14989 from NixOS/querySubstitutablePaths
Move LocalStore::querySubstitutablePaths() to Store
2026-01-14 19:38:28 +00:00
Eelco Dolstra 8c2021989e Merge pull request #14993 from NixOS/curl-scalability
Limit the number of active curl handles
2026-01-14 19:02:21 +00:00
Eelco Dolstra 92344a31fa LegacySSHStore: Override querySubstitutablePaths() 2026-01-14 19:48:14 +01:00
Eelco Dolstra 77fa94d8d4 resolverCallbackWrapper(): Catch exceptions 2026-01-14 19:40:53 +01:00
Eelco Dolstra ee64ffcd75 curlFileTransfer: Lazily create activity and set startTime
There can be a long time between the creation of `TransferItem` and
the start of the curl download, which can lead to misleading download
durations and progress bar status. So now we create the `Activity` and
update `startTime` when curl actually starts the download.
2026-01-14 19:19:01 +01:00
Eelco Dolstra 8012b584c3 Limit the number of active curl handles
Previously, calling queryValidPaths() with a large number (e.g. 100K)
of store paths failed because Nix immediately creates a `TransferItem`
for each .narinfo, which is then registered as a handle with
curl. However curl appears to scale poorly internally: even though
only a few downloads are actually started (up to the
connections/streams limits), it spends a lot of CPU time dealing with
the inactive handles. So the curl thread is sitting at 100% CPU, the
active downloads stall and time out, and everything grind to a halt.

So now we limit the number of curl handles to http-connections *
5. With this, fetching 100K .narinfo files from localhost succeeds in
~15 seconds.
2026-01-14 18:38:59 +01:00
John Ericson 609a4e999f Merge pull request #14988 from NixOS/remove-has-substitutes
Remove WorkerProto::Op::HasSubstitutes
2026-01-14 17:17:39 +00:00
Eelco Dolstra 6fd36553b9 Merge pull request #14990 from NixOS/nar-cache-cleanup
More NAR accessor / listing cleanup
2026-01-14 12:53:36 +00:00
John Ericson ce28cb32e9 BinaryCacheStore: Avoid recreating NAR listing
We already have it, so let's just use it!
2026-01-13 14:01:25 -05:00
John Ericson 3d18d73003 Remove redundant NarAccessor::nar 2026-01-13 13:32:29 -05:00
John Ericson 8089af3bb0 Better type for NarMemberConstructor::regular
This makes some invariants clearer and more local.
2026-01-13 13:12:15 -05:00
John Ericson b49ea8b246 Factor out parseNarListing, move to nar-listing.{cc,hh}
Now we have less of a maze of implementation structs.

Review with
```
git show --color-moved --patience --color-moved-ws=ignore-all-space
```
2026-01-13 13:10:04 -05:00
John Ericson af821ba647 Split out nar-listing.{cc,hh}
I like the separation of concerns from NAR accessing.
2026-01-13 12:10:54 -05:00
Eelco Dolstra 1b1c949d0c Merge pull request #14981 from roberth/fix-git-relative-url-crash
`fixGitURL`: fix crash for "relative" `file:` paths and reject unsupported SCP URLs
2026-01-13 12:59:59 +00:00
Eelco Dolstra fb70a45b9e Move LocalStore::querySubstitutablePaths() to Store
Nothing in this method depends on LocalStore, so we can use it as the
default implementation. (It's only overriden in RemoteStore.)
2026-01-13 13:55:08 +01:00
Eelco Dolstra 53e942bfcc Merge pull request #14983 from roberth/fetchTree-relative-file
fetchTree: reject relative `file:` paths for tarballs
2026-01-13 12:46:08 +00:00
Eelco Dolstra 4f5172ba21 Remove WorkerProto::Op::HasSubstitutes
This operation has been deprecated since
09a6321aeb (July 2012). It was used by
client versions <= 11, which is below `MINIMUM_PROTOCOL_VERSION`
(currently 18).
2026-01-13 12:26:10 +01:00
Jörg Thalheim 1bddbff3a6 Merge pull request #14986 from NixOS/nar-cache-cleanup
Two NAR accessor / listing cleanups
2026-01-13 07:31:29 +00:00
John Ericson 4991defc44 Make reading in a nar listing all well typed
We can get rid of `NarMember`, because it is just `NarListing` in
disguise! The use of `std::variant` makes clear that certain stat fields
we don't care about in the non-regular-file case too.
2026-01-13 01:33:53 -05:00
John Ericson 8f829e478f Inline RemoteFSAccessor::makeCacheFile into lambda
It's just easier to avoid headers.
2026-01-13 01:33:53 -05:00
John Ericson 7ac5a61208 Merge pull request #14948 from NixOS/ca-nar-cache
RemoteFSAccessor: Make the local NAR cache content-addressed
2026-01-13 04:56:31 +00:00
John Ericson 2af5792cc2 Inline RemoteFSAccessor::addToCache
It was not pulling its weight. (Only used once, optional paths are
confusing, we already have an `if` / branch fit-for-purpose.)
2026-01-12 23:09:45 -05:00
Eelco DolstraandJohn Ericson e251ffdd46 RemoteFSAccessor: Make the local NAR cache content-addressed
Use double-indirection for better NAR accessor caching

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2026-01-12 23:00:20 -05:00
John Ericson a8087ebf52 Merge pull request #14984 from NixOS/more-windows-fixes
More windows fixes
2026-01-12 23:19:38 +00:00
Sergei Zimmerman 66fefcd795 libutil: Better implementation of createAnonymousTempFile on windows 2026-01-13 01:28:17 +03:00
Sergei Zimmerman 1ea3a841bb libstore/local-binary-cache-store: Use portable error_code in getFile 2026-01-13 00:56:03 +03:00
Sergei Zimmerman fd0bcd97e8 libutil: Include std::error_code in the base class SystemError 2026-01-13 00:50:42 +03:00
John Ericson aaccb73916 ptrToOwned should be in the nix namespace 2026-01-12 16:14:21 -05:00
Robert Hensing b19bfc6373 fetchTree: improve relative path error wording
Avoid implying that relative paths could work if a base directory
were defined. The file: scheme fundamentally does not support them.
2026-01-12 20:16:26 +01:00
Robert Hensing 3b028edbf7 fixGitURL: fix crash for relative paths and reject unsupported SCP URLs
Relative paths (e.g., "relative/repo") would crash in renderAuthorityAndPath()
because an empty authority was set, violating RFC 3986 section 3.3 which
requires paths to start with "/" when an authority is present.

Fix by only setting authority for absolute paths:
- Absolute paths: file:///path (empty authority)
- Relative paths: file:path (no authority)

Also reject SCP-like URLs without a user (e.g., "github.com:path") with a
clear error message, since proper support requires careful implementation,
which is not something I can do right now.
2026-01-12 13:03:57 +01:00
Robert Hensing 23a7178eb4 fetchTree: reject relative file: paths for tarballs
Relative paths like `file:./foo.tar.gz` have never worked for tarballs
because curl rejects relative file: URLs. Previously this resulted in
cryptic curl errors. Now we reject them early with a clear message
explaining that relative paths are not supported because there is no
defined base directory to resolve them against.

See https://github.com/NixOS/nix/issues/12281
2026-01-12 03:31:42 +01:00
John Ericson 252aff5c8f Merge pull request #14971 from obsidiansystems/misc-builder-fixes
Misc builder fixes
2026-01-11 21:02:14 +00:00
Sergei Zimmerman 920c5ceb0c Merge pull request #14961 from NixOS/readdir-nonexistent-fix
libutil/union-source-accessor: Barf on non-existent directories
2026-01-11 18:15:22 +00:00
John Ericson de76cb681d Make sure we reliably call Worker::childTerminated
When a goal with an active child process is destroyed (e.g., during
failure cascades without `--keep-going`), the child process gets killed
but `childTerminated` was never called. This left stale entries in the
worker's `children` list.

Fix this by ensuring `childTerminated` is called from destructors:

- `DerivationBuilderImpl::killChild` now calls `childTerminated` via
  the `miscMethods` callback.

- `HookInstance` gains an `onKillChild` callback that is invoked from
  its destructor when killing the process. `buildWithHook` sets this
  callback to call `childTerminated`.

To make these calls safe from destructors (where the goal object may be
partially destroyed), add a new overload of `Worker::childTerminated`
that takes an explicit `JobCategory` parameter instead of calling the
virtual method `Goal::jobCategory`. The original overload still exists
for convenience for normal (non-destructor) call sites.
2026-01-10 20:24:05 -05:00
John Ericson d19bfb0045 Avoid short circuiting in Worker::removeGoal
We have to call into both branches no matter what.
2026-01-10 20:23:50 -05:00
John Ericson 160822858a Merge pull request #14788 from NixOS/coroutine-child-output
Way more RAII for `DerivationBuildingGoal`
2026-01-11 00:09:53 +00:00
John Ericson 25998fcd1e Merge pull request #14970 from NixOS/more-openfile-readonly
Use openFileReadonly in more places
2026-01-11 00:07:15 +00:00
John Ericson 7ba09399cc Merge pull request #14968 from NixOS/fs-sink-restore-regular-file-fd
libutil: RestoreRegularFile is an FdSink
2026-01-11 00:05:20 +00:00
Sergei Zimmerman f8a92564f7 More std::filesystem::path for nix {cat,ls}
Also fixes a double quoting issue I accidentally introduced ccdd1f1c65
in seekableGetNarBytes.
2026-01-11 01:44:16 +03:00
Sergei Zimmerman 2ae4121c1d libutil/file-system: Use openFileReadonly in more places 2026-01-11 01:44:15 +03:00
John Ericson 38c755f168 Merge pull request #14966 from drupol/push-ozvunuqvxrvw
chore: replace `edolstra/flake-compat` with `NixOS/flake-compat`
2026-01-10 15:59:07 +00:00
Robert Hensing cb2ade20d4 Goal: add doneSuccess/doneFailure helpers to base class
Add helpers to the base Goal class that set buildResult and call amDone,
ensuring buildResult is always populated when a goal terminates.

Derived class helpers now call the base class versions. This reorders
operations: previously buildResult was set before bookkeeping (counter
resets, worker stats), now it's set after. This is safe because the
bookkeeping code (mcExpectedBuilds.reset(), worker.doneBuilds++,
worker.updateProgress(), etc.) only accesses worker counters, not
buildResult.
2026-01-10 16:15:48 +01:00
Sergei Zimmerman 08887caa1a libutil: RestoreRegularFile is an FdSink
It correctly models the is-a relation. This will be useful for doing a dynamic_cast in
downstream code that wants to copy from a file descriptor to a file descriptor.
2026-01-10 16:31:05 +03:00
Pol Dellaiera 8d588ad471 chore: replace edolstra/flake-compat with NixOS/flake-compat 2026-01-10 10:10:55 +01:00
Shea Levy d51ac82dd2 systemd: Delegate cgroup management without turning on controllers.
Nix currently doesn't do any resource control, and Delegate=yes turns on all the controllers.

In particular, this enables using cpusets with cgroups V1 alongside the Nix daemon.
2026-01-09 23:22:15 -05:00
Sergei Zimmerman 6970efe2e1 Merge pull request #14962 from NixOS/fix-mingw
Fix mingw build (once again), add openFileReadonly and clean up error.hh to include WinError
2026-01-10 00:39:46 +00:00
Sergei Zimmerman 3b95b7c9aa Merge pull request #14963 from corngood/test-leak
libflake-tests: fix leak in nix_api_store_test.nix_api_load_flake_with_flags
2026-01-09 23:44:30 +00:00
David McFarland 4289e2f9e6 libflake-tests: fix leak in nix_api_store_test.nix_api_load_flake_with_flags 2026-01-09 16:51:15 -04:00
Sergei Zimmerman ccdd1f1c65 libstore: Fix mingw build
This also adds a utility for opening a file descriptor from a path in readonly mode.
Previous commit helps a bit with error handling, since now we just throw a NativeSysError.
2026-01-09 21:06:34 +03:00
Sergei Zimmerman b7fd471f84 libutil: Inline windows-error.hh into error.hh
This way each consumer of NativeSysError doesn't have to
also conditionally include the windows-error.hh, which is very cumbersome.
And we can't include windows-error.hh in error.hh because of a circular import.
2026-01-09 20:59:35 +03:00
Sergei Zimmerman 4ab2cdacfc libutil/union-source-accessor: Barf on non-existent directories
Previously builtins.readDir would return an empty attribute set
instead of barfing on non-existent paths. This is a regression from
2.32 for impure eval.
2026-01-09 20:19:32 +03:00
Ilja 75571ec0a0 Add example commands to source installation 2026-01-08 12:46:24 +02:00
John Ericson 1807dc78d6 Split out separate coroutine functions for with/without hook
This makes the logic much easier to follow. Unlike before, the use of
separate functions is not making us pass a gazillion arguments or use
the crutch of class variables.
2026-01-05 14:38:25 -05:00
John Ericson cf5644ce99 DerivationBuildingGoal Make some fields constant
These are immutable parameters, not state, set in the constructor.
2026-01-05 14:38:25 -05:00
John Ericson 47ccf201fb Make DerivationBuildingGoal::inputPaths local variable 2026-01-05 14:38:25 -05:00
John Ericson ec65132ab9 Shrink DerivationBuildingGoal::tryBuildHook
There was a bunch of logic in there which was, effectively, using the
build hook, rather than deciding *whether* to use the build hook. We
want it to only be the latter.
2026-01-05 14:38:25 -05:00
John Ericson bde0b22dc8 Factor out LogFile, do more RAII
It is weird that `LogFile` and `BuildLog` are basically unrelated, but
the does currently reflect the logic that exists.
2026-01-05 14:38:24 -05:00
John Ericson a8e8614fd3 BuildLog owns activities
Now it does more, using the logging.hh functionality as needed.
2026-01-05 14:29:42 -05:00
John Ericson 6a32b754a9 Inline DerivationBuildingGoal::logSize
Separate state for each loop.
2026-01-05 14:29:42 -05:00
John Ericson 43a472631b Factor out BuildLog
This gets some trickier pure logic out of `DerivationBuilderGoal`.
2026-01-05 14:29:42 -05:00
John Ericson eff403b5ab Inline DerivationBuildingGoal::builder 2026-01-05 14:29:42 -05:00
John Ericson 6e6dda9f67 Inline DerivationBuildingGoal::hook
Also remove uneeded arguments from `processChildOutput`.
2026-01-05 14:29:42 -05:00
John Ericson ee1383f75f Get cleanup logic out of DerivationBuildingGoal::processChildOutput
This will allow for making `hook` and `builder` local variables.
2026-01-05 14:29:42 -05:00
John Ericson 5be07abf6d Inline DerivationBuildingGoal::killChild 2026-01-05 14:29:42 -05:00
John Ericson 469212bd38 Inline DerivationBuildingGoal::currentHookLine
That's easy to do now.
2026-01-05 14:29:42 -05:00
John Ericson 4a1a562461 Inline the hook-only part of processChildOutput to that caller 2026-01-05 14:29:42 -05:00
Peter Bynum eaf474bf24 add query_from_hash_part c API 2026-01-05 11:39:15 -05:00
263 changed files with 4435 additions and 2179 deletions

No files matched your search

@@ -24,8 +24,8 @@ inputs:
description: "Github token"
required: true
use_cache:
description: "Whether to setup magic-nix-cache"
default: true
description: "Whether to setup github actions cache (not implemented currently)"
default: false
required: false
runs:
using: "composite"
@@ -122,10 +122,3 @@ runs:
source-url: ${{ inputs.experimental-installer-version != 'latest' && 'https://artifacts.nixos.org/experimental-installer/tag/${{ inputs.experimental-installer-version }}/${{ env.EXPERIMENTAL_INSTALLER_ARTIFACT }}' || '' }}
nix-package-url: ${{ inputs.dogfood == 'true' && steps.download-nix-installer.outputs.tarball-path || (inputs.tarball_url || '') }}
extra-conf: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 # v13
if: ${{ inputs.use_cache == 'true' }}
with:
diagnostic-endpoint: ''
use-flakehub: false
use-gha-cache: true
source-revision: 92d9581367be2233c2d5714a2640e1339f4087d8 # main
@@ -0,0 +1,8 @@
---
synopsis: "C API: New store API methods"
prs: [14766]
---
The C API now includes additional methods:
- `nix_store_query_path_from_hash_part()` - Get the full store path given its hash part
@@ -0,0 +1,10 @@
---
synopsis: "New setting `ignore-gc-delete-failure` for local stores"
prs: [15054]
---
A new local store setting [`ignore-gc-delete-failure`](@docroot@/store/types/local-store.md#store-local-store-ignore-gc-delete-failure) has been added.
When enabled, garbage collection will log warnings instead of failing when it cannot delete store paths.
This is useful when running Nix as an unprivileged user that may not have write access to all paths in the store.
This setting is experimental and requires the [`local-overlay-store`](@docroot@/development/experimental-features.md#xp-feature-local-overlay-store) experimental feature.
+15
View File
@@ -0,0 +1,15 @@
---
synopsis: Support HTTPS binary caches using mTLS (client certificate) authentication
issues: [13002]
prs: [13030]
---
Added support for `tls-certificate` and `tls-private-key` options in substituter URLs.
Example:
```
https://substituter.invalid?tls-certificate=/path/to/cert.pem&tls-private-key=/path/to/key.pem
```
When these options are configured, Nix will use this certificate/private key pair to authenticate to the server.
+2
View File
@@ -135,7 +135,9 @@
- [Serving Tarball Flakes](protocols/tarball-fetcher.md)
- [Store Path Specification](protocols/store-path.md)
- [Nix Archive (NAR) Format](protocols/nix-archive/index.md)
- [Nix Cache Info Format](protocols/nix-cache-info.md)
- [Derivation "ATerm" file format](protocols/derivation-aterm.md)
- [Nix32 Encoding](protocols/nix32.md)
- [C API](c-api.md)
- [Glossary](glossary.md)
- [Development](development/index.md)
@@ -57,11 +57,6 @@ Most Nix commands interpret the following environment variables:
Overrides the location of the Nix store (default `prefix/store`).
- <span id="env-NIX_DATA_DIR">[`NIX_DATA_DIR`](#env-NIX_DATA_DIR)</span>
Overrides the location of the Nix static data directory (default
`prefix/share`).
- <span id="env-NIX_LOG_DIR">[`NIX_LOG_DIR`](#env-NIX_LOG_DIR)</span>
Overrides the location of the Nix log directory (default
@@ -6,14 +6,23 @@ It is broken up into multiple Meson packages, which are optionally combined in a
There are no mandatory extra steps to the building process:
generic Meson installation instructions like [this](https://mesonbuild.com/Quick-guide.html#using-meson-as-a-distro-packager) should work.
The installation path can be specified by passing the `-Dprefix=prefix`
to `configure`. The default installation directory is `/usr/local`. You
```bash
git clone https://github.com/NixOS/nix.git
cd nix
meson setup build
cd build
ninja
(sudo) ninja install
```
The installation path can be specified by passing `-Dprefix=prefix`
to `meson setup build`. The default installation directory is `/usr/local`. You
can change this to any location you like. You must have write permission
to the *prefix* path.
Nix keeps its *store* (the place where packages are stored) in
`/nix/store` by default. This can be changed using
`-Dstore-dir=path`.
`-Dlibstore:store-dir=path`.
> **Warning**
>
@@ -338,7 +338,7 @@ Here is more information on the `output*` attributes, and what values they may b
This will specify the output hash of the single output of a [fixed-output derivation].
The `outputHash` attribute must be a string containing the hash in either hexadecimal or "nix32" encoding, or following the format for integrity metadata as defined by [SRI](https://www.w3.org/TR/SRI/).
The "nix32" encoding is an adaptation of base-32 encoding.
The ["nix32" encoding](@docroot@/protocols/nix32.md) is Nix's variant of base-32 encoding.
> **Note**
>
@@ -19,17 +19,16 @@ whatever port you like:
$ nix-serve -p 8080
```
To check whether it works, try the following on the client:
To check whether it works, try fetching the [`nix-cache-info`](@docroot@/protocols/nix-cache-info.md) file on the client:
```console
$ curl http://avalon:8080/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
```
which should print something like:
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
When writing to a binary cache (e.g., with [`nix copy`](@docroot@/command-ref/new-cli/nix3-copy.md)), Nix creates [`nix-cache-info`](@docroot@/protocols/nix-cache-info.md) automatically if it doesn't exist.
On the client side, you can tell Nix to use your binary cache using
`--substituters`, e.g.:
@@ -24,7 +24,7 @@ description: |
The format follows this pattern: `${digest}-${name}`
- **hash**: Digest rendered in a custom variant of [Base32](https://en.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters)
- **hash**: Digest rendered in [Nix32](@docroot@/protocols/nix32.md), a variant of base-32 (20 hash bytes become 32 ASCII characters)
- **name**: The package name and optional version/suffix information
type: string
@@ -0,0 +1,55 @@
# Nix Cache Info Format
The `nix-cache-info` file is a metadata file at the root of a [binary cache](@docroot@/package-management/binary-cache-substituter.md) (e.g., `https://cache.example.com/nix-cache-info`).
MIME type: `text/x-nix-cache-info`
## Format
Line-based key-value format:
```
Key: value
```
Leading and trailing whitespace is trimmed from values.
Lines without a colon are ignored.
Unknown keys are silently ignored.
## Fields
### `StoreDir`
The Nix store directory path that this cache was built for (e.g., `/nix/store`).
If present, Nix verifies that this matches the client's store directory:
```
error: binary cache 'https://example.com' is for Nix stores with prefix '/nix/store', not '/home/user/nix/store'
```
### `WantMassQuery`
`1` or `0`. Sets the default for [`want-mass-query`](@docroot@/store/types/http-binary-cache-store.md#store-http-binary-cache-store-want-mass-query).
### `Priority`
Integer. Sets the default for [`priority`](@docroot@/store/types/http-binary-cache-store.md#store-http-binary-cache-store-priority).
## Example
```
StoreDir: /nix/store
WantMassQuery: 1
Priority: 30
```
## Caching Behavior
Nix caches `nix-cache-info` in the [cache directory](@docroot@/command-ref/env-common.md#env-NIX_CACHE_HOME) with a 7-day TTL.
## See Also
- [HTTP Binary Cache Store](@docroot@/store/types/http-binary-cache-store.md)
- [Serving a Nix store via HTTP](@docroot@/package-management/binary-cache-substituter.md)
- [`substituters`](@docroot@/command-ref/conf-file.md#conf-substituters)
+19
View File
@@ -0,0 +1,19 @@
# Nix32 Encoding
Nix32 is Nix's variant of base-32 encoding, used for [store path digests](@docroot@/protocols/store-path.md), hash output via [`nix hash`](@docroot@/command-ref/new-cli/nix3-hash.md), and the [`outputHash`](@docroot@/language/advanced-attributes.md#adv-attr-outputHash) derivation attribute.
## Alphabet
The Nix32 alphabet consists of these 32 characters:
```
0 1 2 3 4 5 6 7 8 9 a b c d f g h i j k l m n p q r s v w x y z
```
The letters `e`, `o`, `u`, and `t` are omitted.
## Byte Order
Nix32 encoding processes the hash bytes from the end (last byte first), while base-16 encoding processes from the beginning (first byte first).
Consequently, the string sort order is determined primarily by the first bytes for base-16, and by the last bytes for Nix32.
+4 -5
View File
@@ -20,12 +20,11 @@ where
- `store-dir` = the [store directory](@docroot@/store/store-path.md#store-directory)
- `digest` = base-32 representation of the compressed to 160 bits [SHA-256] hash of `fingerprint`
- `digest` = base-32 representation of the compressed to 160 bits [SHA-256] hash of `fingerprint`.
For the definition of the hash compression algorithm, please refer to the section 5.1 of
the [Nix thesis](https://edolstra.github.io/pubs/phd-thesis.pdf), which also defines the
specifics of base-32 encoding. Note that base-32 encoding processes the hash bytestring from
the end, while base-16 processes in from the beginning.
Nix uses a custom base-32 encoding called [Nix32](@docroot@/protocols/nix32.md).
For the definition of the hash compression algorithm, please refer to section 5.1 of the [Nix thesis](https://edolstra.github.io/pubs/phd-thesis.pdf).
## Fingerprint
+1 -1
View File
@@ -31,7 +31,7 @@ A store path is rendered to a file system path as the concatenation of
- [Store directory](#store-directory) (typically `/nix/store`)
- Path separator (`/`)
- Digest rendered in a custom variant of [Base32](https://en.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters)
- Digest rendered in [Nix32](@docroot@/protocols/nix32.md), a variant of base-32 (20 hash bytes become 32 ASCII characters)
- Hyphen (`-`)
- Name
Generated
+5 -5
View File
@@ -3,15 +3,15 @@
"flake-compat": {
"flake": false,
"locked": {
"lastModified": 1733328505,
"narHash": "sha256-NeCCThCEP3eCl2l/+27kNNK7QrwZB1IJCrXfrbv5oqU=",
"owner": "edolstra",
"lastModified": 1767039857,
"narHash": "sha256-vNpUSpF5Nuw8xvDLj2KCwwksIbjua2LZCqhV1LNRDns=",
"owner": "NixOS",
"repo": "flake-compat",
"rev": "ff81ac966bb2cae68946d5ed5fc4994f96d0ffec",
"rev": "5edf11c44bc78a0d334f6334cdaf7d60d732daab",
"type": "github"
},
"original": {
"owner": "edolstra",
"owner": "NixOS",
"repo": "flake-compat",
"type": "github"
}
+1 -1
View File
@@ -6,7 +6,7 @@
inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.nixpkgs-23-11.url = "github:NixOS/nixpkgs/a62e6edd6d5e1fa0329b8653c801147986f8d446";
inputs.flake-compat = {
url = "github:edolstra/flake-compat";
url = "github:NixOS/flake-compat";
flake = false;
};
+1 -1
View File
@@ -11,7 +11,7 @@ ExecStart=@@bindir@/nix-daemon nix-daemon --daemon
KillMode=process
LimitNOFILE=1048576
TasksMax=1048576
Delegate=yes
Delegate=
[Install]
WantedBy=multi-user.target
+4 -4
View File
@@ -131,7 +131,7 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
ignoreCrossFile = flags: builtins.filter (flag: !(lib.strings.hasInfix "cross-file" flag)) flags;
availableComponents = lib.filterAttrs (
k: v: lib.meta.availableOn pkgs.hostPlatform v
k: v: lib.meta.availableOn pkgs.stdenv.hostPlatform v
) allComponents;
activeComponents = buildInputsClosureCond isInternal (
@@ -300,7 +300,7 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
lib.filter (x: !isInternal x) (
lib.lists.concatMap (
# Nix manual has a build-time dependency on nix, but we
# don't want to do a native build just to enter the ross
# don't want to do a native build just to enter the cross
# dev shell.
#
# TODO: think of a more principled fix for this.
@@ -323,7 +323,7 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
pkgs.buildPackages.shellcheck
pkgs.buildPackages.include-what-you-use
]
++ lib.optional pkgs.hostPlatform.isUnix pkgs.buildPackages.gdb
++ lib.optional stdenv.hostPlatform.isUnix pkgs.buildPackages.gdb
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
)
@@ -341,7 +341,7 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
buildInputs =
# TODO change Nixpkgs to mark gbenchmark as building on Windows
lib.optional pkgs.hostPlatform.isUnix pkgs.gbenchmark
lib.optional stdenv.hostPlatform.isUnix pkgs.gbenchmark
++ dedupByString (v: "${v}") (
lib.filter (x: !isInternal x) (lib.lists.concatMap (c: c.buildInputs) activeComponents)
)
+2
View File
@@ -300,6 +300,8 @@ MixDefaultProfile::MixDefaultProfile()
profile = getDefaultProfile().string();
}
static constexpr auto environmentVariablesCategory = "Options that change environment variables";
MixEnvironment::MixEnvironment()
: ignoreEnvironment(false)
{
+1 -1
View File
@@ -795,7 +795,7 @@ void NixRepl::loadFiles()
loadedFiles.clear();
for (auto & i : old) {
notice("Loading '%1%'...", i);
notice("Loading %1%...", PathFmt(i));
loadFile(i);
}
+1 -1
View File
@@ -75,7 +75,7 @@ struct AttrDb
auto dbPath = cacheDir / (fingerprint.to_string(HashFormat::Base16, false) + ".sqlite");
state->db = SQLite(dbPath);
state->db = SQLite(dbPath, {.useWAL = settings.useSQLiteWAL});
state->db.isCache();
state->db.exec(schema);
+1 -1
View File
@@ -139,7 +139,7 @@ public:
, profileFd([&]() {
AutoCloseFD fd = toDescriptor(open(profileFile.string().c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0660));
if (!fd)
throw SysError("opening file %s", profileFile);
throw SysError("opening file %s", PathFmt(profileFile));
return fd;
}())
, posCache(state)
+6
View File
@@ -1002,6 +1002,12 @@ public:
[[nodiscard]] StringMap
realiseContext(const NixStringContext & context, StorePathSet * maybePaths = nullptr, bool isIFD = true);
/**
* Coerce `v` to a path and realise it, i.e. build anything in the value's string context using `realiseContext()`.
*/
SourcePath realisePath(
const PosIdx pos, Value & v, std::optional<SymlinkResolution> resolveSymlinks = SymlinkResolution::Full);
/**
* Realise the given string with context, and return the string with outputs instead of downstream output
* placeholders.
+12 -16
View File
@@ -157,24 +157,20 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
return res;
}
static SourcePath realisePath(
EvalState & state,
const PosIdx pos,
Value & v,
std::optional<SymlinkResolution> resolveSymlinks = SymlinkResolution::Full)
SourcePath EvalState::realisePath(const PosIdx pos, Value & v, std::optional<SymlinkResolution> resolveSymlinks)
{
NixStringContext context;
auto path = state.coerceToPath(noPos, v, context, "while realising the context of a path");
auto path = coerceToPath(noPos, v, context, "while realising the context of a path");
try {
if (!context.empty() && path.accessor == state.rootFS) {
auto rewrites = state.realiseContext(context);
if (!context.empty() && path.accessor == rootFS) {
auto rewrites = realiseContext(context);
path = {path.accessor, CanonPath(rewriteStrings(path.path.abs(), rewrites))};
}
return resolveSymlinks ? path.resolveSymlinks(*resolveSymlinks) : path;
} catch (Error & e) {
e.addTrace(state.positions[pos], "while realising the context of path '%s'", path);
e.addTrace(positions[pos], "while realising the context of path '%s'", path);
throw;
}
}
@@ -294,7 +290,7 @@ static void scopedImport(EvalState & state, const PosIdx pos, SourcePath & path,
argument. */
static void import(EvalState & state, const PosIdx pos, Value & vPath, Value * vScope, Value & v)
{
auto path = realisePath(state, pos, vPath, std::nullopt);
auto path = state.realisePath(pos, vPath, std::nullopt);
auto path2 = path.path.abs();
// FIXME
@@ -447,7 +443,7 @@ extern "C" typedef void (*ValueInitializer)(EvalState & state, Value & v);
/* Load a ValueInitializer from a DSO and return whatever it initializes */
void prim_importNative(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0]);
auto path = state.realisePath(pos, *args[0]);
std::string sym(
state.forceStringNoCtx(*args[1], pos, "while evaluating the second argument passed to builtins.importNative"));
@@ -1971,7 +1967,7 @@ static void prim_pathExists(EvalState & state, const PosIdx pos, Value ** args,
arg.type() == nString && (arg.string_view().ends_with("/") || arg.string_view().ends_with("/."));
auto symlinkResolution = mustBeDir ? SymlinkResolution::Full : SymlinkResolution::Ancestors;
auto path = realisePath(state, pos, arg, symlinkResolution);
auto path = state.realisePath(pos, arg, symlinkResolution);
auto st = path.maybeLstat();
auto exists = st && (!mustBeDir || st->type == SourceAccessor::tDirectory);
@@ -2078,7 +2074,7 @@ static RegisterPrimOp primop_dirOf({
/* Return the contents of a file as a string. */
static void prim_readFile(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0]);
auto path = state.realisePath(pos, *args[0]);
auto s = path.readFile();
if (s.find((char) 0) != std::string::npos)
state.error<EvalError>("the contents of the file '%1%' cannot be represented as a Nix string", path)
@@ -2313,7 +2309,7 @@ static void prim_hashFile(EvalState & state, const PosIdx pos, Value ** args, Va
if (!ha)
state.error<EvalError>("unknown hash algorithm '%1%'", algo).atPos(pos).debugThrow();
auto path = realisePath(state, pos, *args[1]);
auto path = state.realisePath(pos, *args[1]);
v.mkString(hashString(*ha, path.readFile()).to_string(HashFormat::Base16, false), state.mem);
}
@@ -2365,7 +2361,7 @@ static const Value & fileTypeToString(EvalState & state, SourceAccessor::Type ty
static void prim_readFileType(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0], std::nullopt);
auto path = state.realisePath(pos, *args[0], std::nullopt);
/* Retrieve the directory entry type and stringize it. */
v = fileTypeToString(state, path.lstat().type);
}
@@ -2383,7 +2379,7 @@ static RegisterPrimOp primop_readFileType({
/* Read a directory (without . or ..) */
static void prim_readDir(EvalState & state, const PosIdx pos, Value ** args, Value & v)
{
auto path = realisePath(state, pos, *args[0]);
auto path = state.realisePath(pos, *args[0]);
// Retrieve directory entries for all nodes in a directory.
// This is similar to `getFileType` but is optimized to reduce system calls
+1 -1
View File
@@ -41,7 +41,7 @@ struct CacheImpl : Cache
auto dbPath = (getCacheDir() / "fetcher-cache-v4.sqlite").string();
createDirs(dirOf(dbPath));
state->db = SQLite(dbPath);
state->db = SQLite(dbPath, {.useWAL = settings.useSQLiteWAL});
state->db.isCache();
state->db.exec(schema);
+2 -2
View File
@@ -489,11 +489,11 @@ void InputScheme::clone(
const Settings & settings, Store & store, const Input & input, const std::filesystem::path & destDir) const
{
if (std::filesystem::exists(destDir))
throw Error("cannot clone into existing path %s", destDir);
throw Error("cannot clone into existing path %s", PathFmt(destDir));
auto [accessor, input2] = getAccessor(settings, store, input);
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying '%s' to %s...", input2.to_string(), destDir));
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying '%s' to %s...", input2.to_string(), PathFmt(destDir)));
RestoreSink sink(/*startFsync=*/false);
sink.dstPath = destDir;
@@ -10,12 +10,6 @@ std::optional<std::filesystem::path> FilteringSourceAccessor::getPhysicalPath(co
return next->getPhysicalPath(prefix / path);
}
std::string FilteringSourceAccessor::readFile(const CanonPath & path)
{
checkAccess(path);
return next->readFile(prefix / path);
}
void FilteringSourceAccessor::readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback)
{
checkAccess(path);
+2 -2
View File
@@ -273,7 +273,7 @@ void Fetch::fetch(
+ "/" + pointer->oid;
std::filesystem::path cachePath = cacheDir / key;
if (pathExists(cachePath)) {
debug("using cache entry %s -> %s", key, cachePath);
debug("using cache entry %s -> %s", key, PathFmt(cachePath));
sink(readFile(cachePath));
return;
}
@@ -301,7 +301,7 @@ void Fetch::fetch(
sizeCallback(size);
downloadToSink(ourl, authHeader, sink, sha256, size);
debug("creating cache entry %s -> %s", key, cachePath);
debug("creating cache entry %s -> %s", key, PathFmt(cachePath));
if (!pathExists(cachePath.parent_path()))
createDirs(cachePath.parent_path());
writeFile(cachePath, sink.s);
+19 -11
View File
@@ -209,14 +209,14 @@ static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options op
return;
if (!options.create)
throw Error("Git repository %s does not exist.", path);
throw Error("Git repository %s does not exist.", PathFmt(path));
std::filesystem::path tmpDir = createTempDir(path.parent_path());
AutoDelete delTmpDir(tmpDir, true);
Repository tmpRepo;
if (git_repository_init(Setter(tmpRepo), tmpDir.string().c_str(), options.bare))
throw Error("creating Git repository %s: %s", path, git_error_last()->message);
throw Error("creating Git repository %s: %s", PathFmt(path), git_error_last()->message);
try {
std::filesystem::rename(tmpDir, path);
} catch (std::filesystem::filesystem_error & e) {
@@ -226,7 +226,7 @@ static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options op
|| e.code() == std::errc::directory_not_empty) {
return;
} else
throw SysError("moving temporary git repository from %s to %s", tmpDir, path);
throw SysError("moving temporary git repository from %s to %s", PathFmt(tmpDir), PathFmt(path));
}
// we successfully moved the repository, so the temporary directory no longer exists.
delTmpDir.cancel();
@@ -266,7 +266,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
initRepoAtomically(path, options);
if (git_repository_open(Setter(repo), path.string().c_str()))
throw Error("opening Git repository %s: %s", path, git_error_last()->message);
throw Error("opening Git repository %s: %s", PathFmt(path), git_error_last()->message);
ObjectDb odb;
if (options.packfilesOnly) {
@@ -766,7 +766,7 @@ struct GitSourceAccessor : SourceAccessor
{
}
std::string readBlob(const CanonPath & path, bool symlink)
void readBlob(const CanonPath & path, bool symlink, Sink & sink, std::function<void(uint64_t)> sizeCallback)
{
auto state(state_.lock());
@@ -785,16 +785,22 @@ struct GitSourceAccessor : SourceAccessor
e.addTrace({}, "while smudging git-lfs file '%s'", path);
throw;
}
return s.s;
sizeCallback(s.s.size());
StringSource source{s.s};
source.drainInto(sink);
return;
}
}
return std::string((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
auto view = std::string_view((const char *) git_blob_rawcontent(blob.get()), git_blob_rawsize(blob.get()));
sizeCallback(view.size());
StringSource source{view};
source.drainInto(sink);
}
std::string readFile(const CanonPath & path) override
void readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback) override
{
return readBlob(path, false);
return readBlob(path, false, sink, sizeCallback);
}
bool pathExists(const CanonPath & path) override
@@ -861,7 +867,9 @@ struct GitSourceAccessor : SourceAccessor
std::string readLink(const CanonPath & path) override
{
return readBlob(path, true);
StringSink s;
readBlob(path, true, s, [&](uint64_t size) { s.s.reserve(size); });
return std::move(s.s);
}
/**
@@ -1425,7 +1433,7 @@ std::vector<std::tuple<GitRepoImpl::Submodule, Hash>> GitRepoImpl::getSubmodules
auto [fdTemp, pathTemp] = createTempFile("nix-git-submodules");
try {
writeFull(fdTemp.get(), configS);
} catch (SysError & e) {
} catch (SystemError & e) {
e.addTrace({}, "while writing .gitmodules file to temporary file");
throw;
}
+6 -5
View File
@@ -72,10 +72,10 @@ std::optional<std::string> readHead(const std::filesystem::path & path)
if (const auto parseResult = git::parseLsRemoteLine(line); parseResult && parseResult->reference == "HEAD") {
switch (parseResult->kind) {
case git::LsRemoteRefLine::Kind::Symbolic:
debug("resolved HEAD ref '%s' for repo '%s'", parseResult->target, path);
debug("resolved HEAD ref '%s' for repo %s", parseResult->target, PathFmt(path));
break;
case git::LsRemoteRefLine::Kind::Object:
debug("resolved HEAD rev '%s' for repo '%s'", parseResult->target, path);
debug("resolved HEAD rev '%s' for repo %s", parseResult->target, PathFmt(path));
break;
}
return parseResult->target;
@@ -753,9 +753,10 @@ struct GitInputScheme : InputScheme
"\n"
"git -C %2% add \"%1%\"",
path.rel(),
repoPath);
PathFmt(repoPath));
else
return RestrictedPathError("Path '%s' does not exist in Git repository %s.", path.rel(), repoPath);
return RestrictedPathError(
"Path '%s' does not exist in Git repository %s.", path.rel(), PathFmt(repoPath));
};
}
@@ -848,7 +849,7 @@ struct GitInputScheme : InputScheme
if (!input.getRev())
setWriteTime(localRefFile, now, now);
} catch (Error & e) {
warn("could not update mtime for file %s: %s", localRefFile, e.info().msg);
warn("could not update mtime for file %s: %s", PathFmt(localRefFile), e.info().msg);
}
if (!originalRef && !storeCachedHead(repoUrl.to_string(), shallow, ref))
warn("could not update cached head '%s' for '%s'", ref, repoInfo.locationToArg());
@@ -34,7 +34,7 @@ struct FilteringSourceAccessor : SourceAccessor
std::optional<std::filesystem::path> getPhysicalPath(const CanonPath & path) override;
std::string readFile(const CanonPath & path) override;
using SourceAccessor::readFile;
void readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback) override;
+1 -1
View File
@@ -154,7 +154,7 @@ struct PathInputScheme : InputScheme
time_t mtime = 0;
if (!storePath || storePath->name() != "source" || !store.isValidPath(*storePath)) {
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying %s to the store", absPath));
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying %s to the store", PathFmt(absPath)));
// FIXME: try to substitute storePath.
auto src = sinkToSource(
[&](Sink & sink) { mtime = dumpPathAndGetMtime(absPath.string(), sink, defaultPathFilter); });
+10 -3
View File
@@ -114,15 +114,22 @@ static DownloadTarballResult downloadTarball_(
// it is not in fact a tarball.
if (url.scheme == "file") {
std::filesystem::path localPath = renderUrlPathEnsureLegal(url.path);
if (!localPath.is_absolute()) {
throw Error(
"tarball '%s' must use an absolute path. "
"The 'file' scheme does not support relative paths.",
url);
}
if (!exists(localPath)) {
throw Error("tarball '%s' does not exist.", localPath);
throw Error("tarball %s does not exist.", PathFmt(localPath));
}
if (is_directory(localPath)) {
if (exists(localPath / ".git")) {
throw Error(
"tarball '%s' is a git repository, not a tarball. Please use `git+file` as the scheme.", localPath);
"tarball %s is a git repository, not a tarball. Please use `git+file` as the scheme.",
PathFmt(localPath));
}
throw Error("tarball '%s' is a directory, not a file.", localPath);
throw Error("tarball %s is a directory, not a file.", PathFmt(localPath));
}
}
+1
View File
@@ -377,6 +377,7 @@ TEST_F(nix_api_store_test, nix_api_load_flake_with_flags)
assert_ctx_ok();
ASSERT_EQ("Claire", helloStr);
nix_locked_flake_free(lockedFlake);
nix_flake_reference_parse_flags_free(parseFlags);
nix_flake_lock_flags_free(lockFlags);
nix_flake_reference_free(flakeReference);
+3 -3
View File
@@ -850,11 +850,11 @@ lockFlake(const Settings & settings, EvalState & state, const FlakeRef & topRef,
auto s = chomp(diff);
if (lockFileExists) {
if (s.empty())
warn("updating lock file %s", outputLockFilePath);
warn("updating lock file %s", PathFmt(outputLockFilePath));
else
warn("updating lock file %s:\n%s", outputLockFilePath, s);
warn("updating lock file %s:\n%s", PathFmt(outputLockFilePath), s);
} else
warn("creating lock file %s: \n%s", outputLockFilePath, s);
warn("creating lock file %s: \n%s", PathFmt(outputLockFilePath), s);
std::optional<std::string> commitMessage = std::nullopt;
+4 -4
View File
@@ -83,8 +83,8 @@ void initPlugins()
checkInterrupt();
pluginFiles.emplace_back(ent.path());
}
} catch (SysError & e) {
if (e.errNo != ENOTDIR)
} catch (SystemError & e) {
if (!e.is(std::errc::not_a_directory))
throw;
pluginFiles.emplace_back(pluginFile);
}
@@ -95,7 +95,7 @@ void initPlugins()
#ifndef _WIN32 // TODO implement via DLL loading on Windows
void * handle = dlopen(file.c_str(), RTLD_LAZY | RTLD_LOCAL);
if (!handle)
throw Error("could not dynamically open plugin file '%s': %s", file, dlerror());
throw Error("could not dynamically open plugin file %s: %s", PathFmt(file), dlerror());
/* Older plugins use a statically initialized object to run their code.
Newer plugins can also export nix_plugin_entry() */
@@ -103,7 +103,7 @@ void initPlugins()
if (nix_plugin_entry)
nix_plugin_entry();
#else
throw Error("could not dynamically open plugin file '%s'", file);
throw Error("could not dynamically open plugin file %s", PathFmt(file));
#endif
}
}
+1 -2
View File
@@ -304,11 +304,10 @@ void printVersion(const std::string & programName)
std::cout << "System type: " << settings.thisSystem << "\n";
std::cout << "Additional system types: " << concatStringsSep(", ", settings.extraPlatforms.get()) << "\n";
std::cout << "Features: " << concatStringsSep(", ", cfg) << "\n";
std::cout << "System configuration file: " << (settings.nixConfDir / "nix.conf") << "\n";
std::cout << "System configuration file: " << (settings.nixConfDir / "nix.conf").string() << "\n";
std::cout << "User configuration files: " << concatStringsSep(":", settings.nixUserConfFiles) << "\n";
std::cout << "Store directory: " << settings.nixStore << "\n";
std::cout << "State directory: " << settings.nixStateDir << "\n";
std::cout << "Data directory: " << settings.nixDataDir << "\n";
}
throw Exit();
}
+16
View File
@@ -338,4 +338,20 @@ nix_derivation * nix_store_drv_from_store_path(nix_c_context * context, Store *
NIXC_CATCH_ERRS_NULL
}
StorePath * nix_store_query_path_from_hash_part(nix_c_context * context, Store * store, const char * hash)
{
if (context)
context->last_err_code = NIX_OK;
try {
std::optional<nix::StorePath> s = store->ptr->queryPathFromHashPart(hash);
if (!s.has_value()) {
return nullptr;
}
return new StorePath{std::move(s.value())};
}
NIXC_CATCH_ERRS_NULL
}
} // extern "C"
+11
View File
@@ -259,6 +259,17 @@ nix_err nix_store_get_fs_closure(
*/
nix_derivation * nix_store_drv_from_store_path(nix_c_context * context, Store * store, const StorePath * path);
/**
* @brief Query the full store path given the hash part of a valid store
* path, or empty if no matching path is found.
*
* @param[out] context Optional, stores error information
* @param[in] store nix store reference
* @param[in] hash Hash part of path as a string
* @return Store path reference, NULL if no matching path is found.
*/
StorePath * nix_store_query_path_from_hash_part(nix_c_context * context, Store * store, const char * hash);
// cffi end
#ifdef __cplusplus
}
+125
View File
@@ -0,0 +1,125 @@
#include "nix/store/tests/https-store.hh"
#include <thread>
namespace nix::testing {
void TestHttpBinaryCacheStore::init()
{
BinaryCacheStore::init();
}
ref<TestHttpBinaryCacheStore> TestHttpBinaryCacheStoreConfig::openTestStore() const
{
auto store = make_ref<TestHttpBinaryCacheStore>(
ref{// FIXME we shouldn't actually need a mutable config
std::const_pointer_cast<HttpBinaryCacheStore::Config>(shared_from_this())});
store->init();
return store;
}
void HttpsBinaryCacheStoreTest::openssl(Strings args)
{
runProgram("openssl", /*lookupPath=*/true, args);
}
void HttpsBinaryCacheStoreTest::SetUp()
{
LibStoreNetworkTest::SetUp();
#ifdef _WIN32
GTEST_SKIP() << "HTTPS store tests are not supported on Windows";
#endif
tmpDir = createTempDir();
cacheDir = tmpDir / "cache";
delTmpDir = std::make_unique<AutoDelete>(tmpDir);
localCacheStore =
make_ref<LocalBinaryCacheStoreConfig>("file", cacheDir.string(), LocalBinaryCacheStoreConfig::Params{})
->openStore();
caCert = tmpDir / "ca.crt";
caKey = tmpDir / "ca.key";
serverCert = tmpDir / "server.crt";
serverKey = tmpDir / "server.key";
clientCert = tmpDir / "client.crt";
clientKey = tmpDir / "client.key";
// clang-format off
openssl({"ecparam", "-genkey", "-name", "prime256v1", "-out", caKey.string()});
openssl({"req", "-new", "-x509", "-days", "1", "-key", caKey.string(), "-out", caCert.string(), "-subj", "/CN=TestCA"});
openssl({"ecparam", "-genkey", "-name", "prime256v1", "-out", serverKey.string()});
openssl({"req", "-new", "-key", serverKey.string(), "-out", (tmpDir / "server.csr").string(), "-subj", "/CN=localhost"});
openssl({"x509", "-req", "-in", (tmpDir / "server.csr").string(), "-CA", caCert.string(), "-CAkey", caKey.string(), "-CAcreateserial", "-out", serverCert.string(), "-days", "1"});
openssl({"ecparam", "-genkey", "-name", "prime256v1", "-out", clientKey.string()});
openssl({"req", "-new", "-key", clientKey.string(), "-out", (tmpDir / "client.csr").string(), "-subj", "/CN=TestClient"});
openssl({"x509", "-req", "-in", (tmpDir / "client.csr").string(), "-CA", caCert.string(), "-CAkey", caKey.string(), "-CAcreateserial", "-out", clientCert.string(), "-days", "1"});
// clang-format on
#ifndef _WIN32 /* FIXME: Can't yet start processes on windows */
auto args = serverArgs();
serverPid = startProcess(
[&] {
if (chdir(cacheDir.c_str()) == -1)
_exit(1);
std::vector<char *> argv;
argv.push_back(const_cast<char *>("openssl"));
for (auto & a : args)
argv.push_back(const_cast<char *>(a.c_str()));
argv.push_back(nullptr);
execvp("openssl", argv.data());
_exit(1);
},
{.dieWithParent = true});
#endif
/* As an optimization, sleep for a bit to allow the server to come up to avoid retrying when connecting.
This won't make the tests fail, but does make them run faster. We don't need to overcomplicate by waiting
for the port explicitly - this is enough. */
std::this_thread::sleep_for(std::chrono::milliseconds(50));
/* FIXME: Don't use global settings. Tests are not run concurrently, so this is fine for now. */
oldCaCert = settings.caFile;
settings.caFile = caCert.string();
}
void HttpsBinaryCacheStoreTest::TearDown()
{
settings.caFile = oldCaCert;
serverPid.kill();
delTmpDir.reset();
}
std::vector<std::string> HttpsBinaryCacheStoreTest::serverArgs()
{
return {
"s_server",
"-accept",
std::to_string(port),
"-cert",
serverCert.string(),
"-key",
serverKey.string(),
"-WWW", /* Serve from current directory. */
"-quiet",
};
}
std::vector<std::string> HttpsBinaryCacheStoreMtlsTest::serverArgs()
{
auto args = HttpsBinaryCacheStoreTest::serverArgs();
/* With the -Verify option the client must supply a certificate or an error occurs, which is not the
case with -verify. */
args.insert(args.end(), {"-CAfile", caCert.string(), "-Verify", "1", "-verify_return_error"});
return args;
}
ref<TestHttpBinaryCacheStoreConfig> HttpsBinaryCacheStoreTest::makeConfig(BinaryCacheStoreConfig::Params params)
{
auto res = make_ref<TestHttpBinaryCacheStoreConfig>("https", fmt("localhost:%d", port), std::move(params));
res->pathInfoCacheSize = 0; /* We don't want any caching in tests. */
return res;
}
} // namespace nix::testing
@@ -0,0 +1,93 @@
#pragma once
///@file
#include <gtest/gtest.h>
#include <gmock/gmock.h>
#include "nix/store/tests/libstore-network.hh"
#include "nix/store/http-binary-cache-store.hh"
#include "nix/store/store-api.hh"
#include "nix/store/globals.hh"
#include "nix/store/local-binary-cache-store.hh"
#include "nix/util/file-system.hh"
#include "nix/util/processes.hh"
namespace nix::testing {
class TestHttpBinaryCacheStoreConfig;
/**
* Test shim for testing. We don't want to use the on-disk narinfo cache in unit
* tests.
*/
class TestHttpBinaryCacheStore : public HttpBinaryCacheStore
{
public:
TestHttpBinaryCacheStore(const TestHttpBinaryCacheStore &) = delete;
TestHttpBinaryCacheStore(TestHttpBinaryCacheStore &&) = delete;
TestHttpBinaryCacheStore & operator=(const TestHttpBinaryCacheStore &) = delete;
TestHttpBinaryCacheStore & operator=(TestHttpBinaryCacheStore &&) = delete;
TestHttpBinaryCacheStore(ref<HttpBinaryCacheStoreConfig> config)
: Store{*config}
, BinaryCacheStore{*config}
, HttpBinaryCacheStore(config)
{
diskCache = nullptr; /* Disable caching, we'll be creating a new binary cache for each test. */
}
void init() override;
};
class TestHttpBinaryCacheStoreConfig : public HttpBinaryCacheStoreConfig
{
public:
TestHttpBinaryCacheStoreConfig(
std::string_view scheme, std::string_view cacheUri, const Store::Config::Params & params)
: StoreConfig(params)
, HttpBinaryCacheStoreConfig(scheme, cacheUri, params)
{
}
ref<TestHttpBinaryCacheStore> openTestStore() const;
ref<Store> openStore() const override
{
return openTestStore();
}
};
class HttpsBinaryCacheStoreTest : public virtual LibStoreNetworkTest
{
std::unique_ptr<AutoDelete> delTmpDir;
public:
static void SetUpTestSuite()
{
initLibStore(/*loadConfig=*/false);
}
protected:
std::filesystem::path tmpDir, cacheDir;
std::filesystem::path caCert, caKey, serverCert, serverKey;
std::filesystem::path clientCert, clientKey;
std::string oldCaCert;
Pid serverPid;
uint16_t port = 8443;
std::shared_ptr<Store> localCacheStore;
static void openssl(Strings args);
void SetUp() override;
void TearDown() override;
virtual std::vector<std::string> serverArgs();
ref<TestHttpBinaryCacheStoreConfig> makeConfig(BinaryCacheStoreConfig::Params params);
};
class HttpsBinaryCacheStoreMtlsTest : public HttpsBinaryCacheStoreTest
{
protected:
std::vector<std::string> serverArgs() override;
};
} // namespace nix::testing
@@ -0,0 +1,39 @@
#pragma once
/// @file
#include <gtest/gtest.h>
namespace nix::testing {
/**
* Whether to run network tests. This is global so that the test harness can
* enable this by default if we can run tests in isolation.
*/
extern bool networkTestsAvailable;
/**
* Set up network tests and, if on linux, create a new network namespace for
* tests with a loopback interface. This is to avoid binding to ports in the
* host's namespace.
*/
void setupNetworkTests();
class LibStoreNetworkTest : public virtual ::testing::Test
{
protected:
void SetUp() override
{
if (networkTestsAvailable)
return;
static bool warned = false;
if (!warned) {
warned = true;
GTEST_SKIP()
<< "Network tests not enabled by default without user namespaces, use NIX_TEST_FORCE_NETWORK_TESTS=1 to override";
} else {
GTEST_SKIP();
}
}
};
} // namespace nix::testing
@@ -4,6 +4,8 @@ include_dirs = [ include_directories('../../..') ]
headers = files(
'derived-path.hh',
'https-store.hh',
'libstore-network.hh',
'libstore.hh',
'nix_api_store.hh',
'outputs-spec.hh',
@@ -0,0 +1,60 @@
#include "nix/store/tests/libstore-network.hh"
#include "nix/util/error.hh"
#include "nix/util/environment-variables.hh"
#ifdef __linux__
# include "nix/util/file-system.hh"
# include "nix/util/linux-namespaces.hh"
# include <sched.h>
# include <sys/ioctl.h>
# include <net/if.h>
# include <netinet/in.h>
#endif
namespace nix::testing {
bool networkTestsAvailable = false;
#ifdef __linux__
static void enterNetworkNamespace()
{
auto uid = ::getuid();
auto gid = ::getgid();
if (::unshare(CLONE_NEWUSER | CLONE_NEWNET) == -1)
throw SysError("setting up a private network namespace for tests");
std::filesystem::path procSelf = "/proc/self";
writeFile(procSelf / "setgroups", "deny");
writeFile(procSelf / "uid_map", fmt("%d %d 1", uid, uid));
writeFile(procSelf / "gid_map", fmt("%d %d 1", gid, gid));
AutoCloseFD fd(::socket(PF_INET, SOCK_DGRAM, IPPROTO_IP));
if (!fd)
throw SysError("cannot open IP socket for loopback interface");
struct ::ifreq ifr = {};
strcpy(ifr.ifr_name, "lo");
ifr.ifr_flags = IFF_UP | IFF_LOOPBACK | IFF_RUNNING;
if (::ioctl(fd.get(), SIOCSIFFLAGS, &ifr) == -1)
throw SysError("cannot set loopback interface flags");
}
#endif
void setupNetworkTests()
try {
networkTestsAvailable = getEnvOs(OS_STR("NIX_TEST_FORCE_NETWORK_TESTS")).has_value();
#ifdef __linux__
if (!networkTestsAvailable && userNamespacesSupported()) {
enterNetworkNamespace();
networkTestsAvailable = true;
}
#endif
} catch (SystemError & e) {
/* Ignore any set up errors. */
}
} // namespace nix::testing
+5
View File
@@ -28,10 +28,15 @@ subdir('nix-meson-build-support/subprojects')
rapidcheck = dependency('rapidcheck')
deps_public += rapidcheck
gtest = dependency('gtest')
deps_public += gtest
subdir('nix-meson-build-support/common')
sources = files(
'derived-path.cc',
'https-store.cc',
'libstore-network.cc',
'outputs-spec.cc',
'path.cc',
'test-main.cc',
+2
View File
@@ -7,6 +7,7 @@
nix-store-c,
rapidcheck,
gtest,
# Configuration Options
@@ -39,6 +40,7 @@ mkMesonLibrary (finalAttrs: {
nix-store
nix-store-c
rapidcheck
gtest
];
mesonFlags = [
+10 -2
View File
@@ -138,7 +138,11 @@ CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -154,7 +158,11 @@ READ_CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -6,8 +6,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf",
"qwer"
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
]
}
]
@@ -10,8 +10,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf",
"qwer"
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
]
}
]
@@ -12,8 +12,8 @@
],
"registrationTime": 23423,
"signatures": [
"asdf",
"qwer"
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
],
"storeDir": "/nix/store",
"ultimate": true,
@@ -15,8 +15,8 @@
],
"registrationTime": 23423,
"signatures": [
"asdf",
"qwer"
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
],
"storeDir": "/nix/store",
"ultimate": true,
@@ -9,8 +9,8 @@
],
"registrationTime": 23423,
"signatures": [
"asdf",
"qwer"
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
],
"storeDir": "/nix/store",
"ultimate": true,
@@ -12,8 +12,8 @@
],
"registrationTime": 23423,
"signatures": [
"asdf",
"qwer"
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
],
"storeDir": "/nix/store",
"ultimate": true,
@@ -3,6 +3,6 @@
"id": "sha256:ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad!foo",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv",
"signatures": [
"asdfasdfasdf"
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
]
}
@@ -6,8 +6,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf",
"qwer"
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
]
}
]
@@ -10,8 +10,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf",
"qwer"
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
]
}
]
@@ -27,8 +27,8 @@
],
"registrationTime": null,
"signatures": [
"fake-sig-1",
"fake-sig-2"
"fake-sig-1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"fake-sig-2:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
],
"storeDir": "/nix/store",
"ultimate": false,
@@ -6,8 +6,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf",
"qwer"
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
]
}
]
@@ -10,8 +10,8 @@
"id": "sha256:15e3c560894cbb27085cf65b5a2ecb18488c999497f4531b6907a7581ce6d527!baz",
"outPath": "g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo",
"signatures": [
"asdf",
"qwer"
"asdf:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"qwer:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
]
}
]
@@ -24,8 +24,8 @@
],
"registrationTime": 23423,
"signatures": [
"fake-sig-1",
"fake-sig-2"
"fake-sig-1:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
"fake-sig-2:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
],
"storeDir": "/nix/store",
"ultimate": false,
@@ -1,6 +1,9 @@
#include <gtest/gtest.h>
#include <regex>
#include "nix/store/http-binary-cache-store.hh"
#include "nix/store/tests/https-store.hh"
#include "nix/util/fs-sink.hh"
namespace nix {
@@ -34,4 +37,88 @@ TEST(HttpBinaryCacheStore, constructConfigWithParamsAndUrlWithParams)
EXPECT_EQ(config.getReference().params, params);
}
using testing::HttpsBinaryCacheStoreMtlsTest;
using testing::HttpsBinaryCacheStoreTest;
using namespace std::string_view_literals;
using namespace std::string_literals;
TEST_F(HttpsBinaryCacheStoreTest, queryPathInfo)
{
auto config = makeConfig({});
auto store = config->openStore();
StringSource dump{"test"sv};
auto path = localCacheStore->addToStoreFromDump(dump, "test-name", FileSerialisationMethod::Flat);
EXPECT_NO_THROW(store->queryPathInfo(path));
}
auto withNoRetries()
{
auto oldTries = fileTransferSettings.tries.get();
Finally restoreTries = [=]() { fileTransferSettings.tries = oldTries; };
fileTransferSettings.tries = 1; /* FIXME: Don't use global settings. */
return restoreTries;
}
TEST_F(HttpsBinaryCacheStoreMtlsTest, queryPathInfo)
{
auto config = makeConfig({
{"tls-certificate"s, clientCert.string()},
{"tls-private-key"s, clientKey.string()},
});
auto store = config->openStore();
StringSource dump{"test"sv};
auto path = localCacheStore->addToStoreFromDump(dump, "test-name", FileSerialisationMethod::Flat);
EXPECT_NO_THROW(store->queryPathInfo(path));
}
TEST_F(HttpsBinaryCacheStoreMtlsTest, rejectsWithoutClientCert)
{
auto restoreTries = withNoRetries();
auto config = makeConfig({});
EXPECT_THROW(config->openStore(), Error);
}
TEST_F(HttpsBinaryCacheStoreMtlsTest, rejectsWrongClientCert)
{
auto wrongKey = tmpDir / "wrong.key";
auto wrongCert = tmpDir / "wrong.crt";
// clang-format off
openssl({"ecparam", "-genkey", "-name", "prime256v1", "-out", wrongKey.string()});
openssl({"req", "-new", "-x509", "-days", "1", "-key", wrongKey.string(), "-out", wrongCert.string(), "-subj", "/CN=WrongClient"});
// clang-format on
auto config = makeConfig({
{"tls-certificate"s, wrongCert.string()},
{"tls-private-key"s, wrongKey.string()},
});
auto restoreTries = withNoRetries();
EXPECT_THROW(config->openStore(), Error);
}
TEST_F(HttpsBinaryCacheStoreMtlsTest, doesNotSendCertOnRedirectToDifferentAuthority)
{
StringSource dump{"test"sv};
auto path = localCacheStore->addToStoreFromDump(dump, "test-name", FileSerialisationMethod::Flat);
for (auto & entry : DirectoryIterator{cacheDir})
if (entry.path().extension() == ".narinfo") {
auto content = readFile(entry.path());
content = std::regex_replace(content, std::regex("URL: nar/"), fmt("URL: https://127.0.0.1:%d/nar/", port));
writeFile(entry.path(), content);
}
auto config = makeConfig({
{"tls-certificate"s, clientCert.string()},
{"tls-private-key"s, clientKey.string()},
});
auto store = config->openStore();
auto restoreTries = withNoRetries();
auto info = store->queryPathInfo(path);
NullSink null;
EXPECT_THROW(store->narFromPath(path, null), Error);
}
} // namespace nix
+2
View File
@@ -1,6 +1,7 @@
#include <gtest/gtest.h>
#include "nix/store/tests/test-main.hh"
#include "nix/store/tests/libstore-network.hh"
using namespace nix;
@@ -10,6 +11,7 @@ int main(int argc, char ** argv)
if (res)
return res;
nix::testing::setupNetworkTests();
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
+2 -1
View File
@@ -2,6 +2,7 @@
#include <gtest/gtest.h>
#include <rapidcheck/gtest.h>
#include "nix/store/globals.hh"
#include "nix/store/sqlite.hh"
#include <sqlite3.h>
@@ -48,7 +49,7 @@ TEST(NarInfoDiskCacheImpl, create_and_read)
// We're going to pay special attention to the id field because we had a bug
// that changed it.
db = SQLite(dbPath);
db = SQLite(dbPath, {.useWAL = settings.useSQLiteWAL});
getIds.create(db, "select id from BinaryCaches where url = 'http://foo'");
{
+4 -1
View File
@@ -59,7 +59,10 @@ static NarInfo makeNarInfo(const Store & store, bool includeImpureInfo)
};
info.registrationTime = 23423;
info.ultimate = true;
info.sigs = {"asdf", "qwer"};
info.sigs = {
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
};
info.url = "nar/1w1fff338fvdw53sqgamddn1b2xgds473pv6y13gizdbqjv4i5p3.nar.xz";
info.compression = "xz";
+5 -1
View File
@@ -9,6 +9,7 @@
nix-store-c,
nix-store-test-support,
sqlite,
openssl,
rapidcheck,
gtest,
@@ -75,7 +76,10 @@ mkMesonExecutable (finalAttrs: {
runCommand "${finalAttrs.pname}-run"
{
meta.broken = !stdenv.hostPlatform.emulatorAvailable buildPackages;
buildInputs = [ writableTmpDirAsHomeHook ];
nativeBuildInputs = [
writableTmpDirAsHomeHook
openssl
];
}
(
''
+4 -1
View File
@@ -66,7 +66,10 @@ static ValidPathInfo makeFullKeyed(const Store & store, bool includeImpureInfo)
};
info.registrationTime = 23423;
info.ultimate = true;
info.sigs = {"asdf", "qwer"};
info.sigs = {
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
};
}
return info;
}
+3 -1
View File
@@ -70,7 +70,9 @@ INSTANTIATE_TEST_SUITE_P(
[&] {
auto r = simple;
// FIXME actually sign properly
r.signatures = {"asdfasdfasdf"};
r.signatures = {
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
};
return r;
}(),
}));
+12 -4
View File
@@ -109,7 +109,11 @@ VERSIONED_CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -127,7 +131,11 @@ VERSIONED_READ_CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -321,8 +329,8 @@ VERSIONED_CHARACTERIZATION_TEST(
info.narSize = 34878;
info.sigs =
{
"fake-sig-1",
"fake-sig-2",
Signature{.keyName = "fake-sig-1", .sig = std::string(64, '\0')},
Signature{.keyName = "fake-sig-2", .sig = std::string(64, '\0')},
},
static_cast<UnkeyedValidPathInfo>(std::move(info));
}),
+12 -4
View File
@@ -162,7 +162,11 @@ VERSIONED_CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -180,7 +184,11 @@ VERSIONED_READ_CHARACTERIZATION_TEST(
Realisation{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.signatures =
{
Signature{.keyName = "asdf", .sig = std::string(64, '\0')},
Signature{.keyName = "qwer", .sig = std::string(64, '\0')},
},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
@@ -541,8 +549,8 @@ VERSIONED_CHARACTERIZATION_TEST(
info.narSize = 34878;
info.sigs =
{
"fake-sig-1",
"fake-sig-2",
Signature{.keyName = "fake-sig-1", .sig = std::string(64, '\0')},
Signature{.keyName = "fake-sig-2", .sig = std::string(64, '\0')},
},
info;
}),
+103 -12
View File
@@ -13,6 +13,11 @@
// C library headers for SSO provider support
# include <aws/auth/credentials.h>
// C library headers for custom logging
# include <aws/common/logging.h>
# include <cstdarg>
# include <boost/unordered/concurrent_flat_map.hpp>
# include <chrono>
@@ -30,6 +35,101 @@ AwsAuthError::AwsAuthError(int errorCode)
namespace {
/**
* Map AWS log level to Nix verbosity.
* AWS levels: AWS_LL_NONE=0, AWS_LL_FATAL=1, AWS_LL_ERROR=2, AWS_LL_WARN=3,
* AWS_LL_INFO=4, AWS_LL_DEBUG=5, AWS_LL_TRACE=6
*
* We map very conservatively because the AWS SDK is extremely noisy. What AWS
* considers "info" includes low-level details like "Initializing epoll" and
* "Starting event-loop thread". What it considers "errors" includes expected
* conditions like missing ~/.aws/config or IMDS being unavailable on non-EC2.
*
* To avoid spamming users, we only show FATAL at default verbosity. Everything
* else requires -vvvvv (lvlDebug) or higher to see.
*/
static Verbosity awsLogLevelToVerbosity(enum aws_log_level level)
{
switch (level) {
case AWS_LL_FATAL:
return lvlError;
case AWS_LL_ERROR:
case AWS_LL_WARN:
case AWS_LL_INFO:
return lvlDebug;
case AWS_LL_DEBUG:
case AWS_LL_TRACE:
return lvlVomit;
// AWS_LL_NONE and AWS_LL_COUNT are enum sentinels, not real log levels
case AWS_LL_NONE:
case AWS_LL_COUNT:
return lvlDebug;
}
unreachable();
}
/**
* Custom AWS logger that routes logs through Nix's logging infrastructure.
*
* The AWS CRT C++ wrapper (ApiHandle::InitializeLogging) only supports FILE*
* or filename-based logging. The underlying C library supports custom loggers
* via aws_logger struct with a vtable containing callback functions.
*/
static int nixAwsLoggerLog(
struct aws_logger * logger, enum aws_log_level logLevel, aws_log_subject_t subject, const char * format, ...)
{
Verbosity nixLevel = awsLogLevelToVerbosity(logLevel);
if (nixLevel > verbosity)
return AWS_OP_SUCCESS; /* Bail out early to avoid formatting the message unnecessarily. */
va_list args;
va_start(args, format);
std::array<char, 4096> buffer{};
auto res = vsnprintf(buffer.data(), buffer.size(), format, args);
va_end(args);
if (res < 0) /* Skip garbage debug messages in case the SDK is busted. */
return AWS_OP_SUCCESS;
const char * subjectName = aws_log_subject_name(subject);
printMsgUsing(nix::logger, nixLevel, "(aws:%s) %s", subjectName ? subjectName : "unknown", chomp(buffer.data()));
return AWS_OP_SUCCESS;
}
/**
* Get current log level for a subject - determines which messages will be logged.
* Must be consistent with awsLogLevelToVerbosity mapping.
*/
static aws_log_level nixAwsLoggerGetLevel(struct aws_logger * logger, aws_log_subject_t subject)
{
// Map Nix verbosity back to AWS log level (inverse of awsLogLevelToVerbosity)
if (verbosity >= lvlVomit)
return AWS_LL_TRACE;
if (verbosity >= lvlDebug)
return AWS_LL_INFO; // error/warn/info are all mapped to lvlDebug
return AWS_LL_FATAL;
}
static void initialiseAwsLogger()
{
static std::once_flag initialised; /* aws_logger_set must only be called once */
std::call_once(initialised, []() {
static aws_logger_vtable nixAwsLoggerVtable = {
.log = nixAwsLoggerLog,
.get_log_level = nixAwsLoggerGetLevel,
.clean_up = [](struct aws_logger *) {}, // No resources to clean up
.set_log_level = nullptr,
};
static aws_logger nixAwsLogger = {
.vtable = &nixAwsLoggerVtable,
.allocator = nullptr,
.p_impl = nullptr,
};
aws_logger_set(&nixAwsLogger);
});
}
/**
* Helper function to wrap a C credentials provider in the C++ interface.
* This replicates the static s_CreateWrappedProvider from aws-crt-cpp.
@@ -119,18 +219,9 @@ class AwsCredentialProviderImpl : public AwsCredentialProvider
public:
AwsCredentialProviderImpl()
{
// Map Nix's verbosity to AWS CRT log level
Aws::Crt::LogLevel logLevel;
if (verbosity >= lvlVomit) {
logLevel = Aws::Crt::LogLevel::Trace;
} else if (verbosity >= lvlDebug) {
logLevel = Aws::Crt::LogLevel::Debug;
} else if (verbosity >= lvlChatty) {
logLevel = Aws::Crt::LogLevel::Info;
} else {
logLevel = Aws::Crt::LogLevel::Warn;
}
apiHandle.InitializeLogging(logLevel, stderr);
// Install custom logger that routes AWS CRT logs through Nix's logging infrastructure.
// This ensures AWS logs respect Nix's verbosity settings and are formatted consistently.
initialiseAwsLogger();
// Create a shared TLS context for SSO (required for HTTPS connections)
auto allocator = Aws::Crt::ApiAllocator();
+13 -13
View File
@@ -147,7 +147,7 @@ ref<const ValidPathInfo> BinaryCacheStore::addToStoreCommon(
write the compressed NAR to disk), into a HashSink (to get the
NAR hash), and into a NarAccessor (to get the NAR listing). */
HashSink fileHashSink{HashAlgorithm::SHA256};
std::shared_ptr<SourceAccessor> narAccessor;
std::shared_ptr<NarAccessor> narAccessor;
HashSink narHashSink{HashAlgorithm::SHA256};
{
FdSink fileSink(fdTemp.get());
@@ -156,7 +156,7 @@ ref<const ValidPathInfo> BinaryCacheStore::addToStoreCommon(
config.compression, teeSinkCompressed, config.parallelCompression, config.compressionLevel);
TeeSink teeSinkUncompressed{*compressionSink, narHashSink};
TeeSource teeSource{narSource, teeSinkUncompressed};
narAccessor = makeNarAccessor(teeSource);
narAccessor = makeNarAccessor(parseNarListing(teeSource));
compressionSink->finish();
fileSink.flush();
}
@@ -165,18 +165,18 @@ ref<const ValidPathInfo> BinaryCacheStore::addToStoreCommon(
auto info = mkInfo(narHashSink.finish());
auto narInfo = make_ref<NarInfo>(info);
narInfo->compression = config.compression;
narInfo->compression = config.compression.to_string(); // FIXME: Make NarInfo use CompressionAlgo
auto [fileHash, fileSize] = fileHashSink.finish();
narInfo->fileHash = fileHash;
narInfo->fileSize = fileSize;
narInfo->url = "nar/" + narInfo->fileHash->to_string(HashFormat::Nix32, false) + ".nar"
+ (config.compression == "xz" ? ".xz"
: config.compression == "bzip2" ? ".bz2"
: config.compression == "zstd" ? ".zst"
: config.compression == "lzip" ? ".lzip"
: config.compression == "lz4" ? ".lz4"
: config.compression == "br" ? ".br"
: "");
+ (config.compression == CompressionAlgo::xz ? ".xz"
: config.compression == CompressionAlgo::bzip2 ? ".bz2"
: config.compression == CompressionAlgo::zstd ? ".zst"
: config.compression == CompressionAlgo::lzip ? ".lzip"
: config.compression == CompressionAlgo::lz4 ? ".lz4"
: config.compression == CompressionAlgo::brotli ? ".br"
: "");
auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(now2 - now1).count();
printMsg(
@@ -205,7 +205,7 @@ ref<const ValidPathInfo> BinaryCacheStore::addToStoreCommon(
if (config.writeNARListing) {
nlohmann::json j = {
{"version", 1},
{"root", listNarDeep(*narAccessor, CanonPath::root)},
{"root", narAccessor->getListing()},
};
upsertFile(std::string(info.path.hashPart()) + ".ls", j.dump(), "application/json");
@@ -562,10 +562,10 @@ ref<SourceAccessor> BinaryCacheStore::getFSAccessor(bool requireValidPath)
std::shared_ptr<SourceAccessor> BinaryCacheStore::getFSAccessor(const StorePath & storePath, bool requireValidPath)
{
return getRemoteFSAccessor(requireValidPath)->accessObject(storePath);
return static_cast<ref<SourceAccessor>>(getRemoteFSAccessor(requireValidPath)->accessObject(storePath));
}
void BinaryCacheStore::addSignatures(const StorePath & storePath, const StringSet & sigs)
void BinaryCacheStore::addSignatures(const StorePath & storePath, const std::set<Signature> & sigs)
{
/* Note: this is inherently racy since there is no locking on
binary caches. In particular, with S3 this unreliable, even
+5
View File
@@ -75,6 +75,11 @@ static BuildResult::Failure::Status failureStatusFromString(std::string_view str
throw Error("unknown built result failure status '%s'", str);
}
[[noreturn]] void BuildResult::Failure::rethrow() const
{
throw BuildError(status, "%s", errorMsg);
}
} // namespace nix
namespace nlohmann {
+48
View File
@@ -0,0 +1,48 @@
#include "nix/store/build/build-log.hh"
namespace nix {
BuildLog::BuildLog(size_t maxTailLines, std::unique_ptr<Activity> act)
: maxTailLines(maxTailLines)
, act(std::move(act))
{
}
void BuildLog::operator()(std::string_view data)
{
for (auto c : data)
if (c == '\r')
currentLogLinePos = 0;
else if (c == '\n')
flushLine();
else {
if (currentLogLinePos >= currentLogLine.size())
currentLogLine.resize(currentLogLinePos + 1);
currentLogLine[currentLogLinePos++] = c;
}
}
void BuildLog::flush()
{
if (!currentLogLine.empty())
flushLine();
}
void BuildLog::flushLine()
{
// Truncate to actual content (currentLogLinePos may be less than size due to \r)
currentLogLine.resize(currentLogLinePos);
if (!handleJSONLogMessage(currentLogLine, *act, builderActivities, "the derivation builder", false)) {
// Line was not handled as JSON, emit and add to tail
act->result(resBuildLogLine, currentLogLine);
logTail.push_back(currentLogLine);
if (logTail.size() > maxTailLines)
logTail.pop_front();
}
currentLogLine.clear();
currentLogLinePos = 0;
}
} // namespace nix
+343 -343
View File
@@ -40,37 +40,13 @@ DerivationBuildingGoal::DerivationBuildingGoal(
worker.store.addTempRoot(this->drvPath);
}
DerivationBuildingGoal::~DerivationBuildingGoal()
{
/* Careful: we should never ever throw an exception from a
destructor. */
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
if (builder)
builder.reset();
#endif
try {
closeLogFile();
} catch (...) {
ignoreExceptionInDestructor();
}
}
DerivationBuildingGoal::~DerivationBuildingGoal() = default;
std::string DerivationBuildingGoal::key()
{
return "dd$" + std::string(drvPath.name()) + "$" + worker.store.printStorePath(drvPath);
}
void DerivationBuildingGoal::killChild()
{
#ifndef _WIN32 // TODO enable build hook on Windows
hook.reset();
#endif
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
if (builder && builder->killChild())
worker.childTerminated(this);
#endif
}
std::string showKnownOutputs(const StoreDirConfig & store, const Derivation & drv)
{
std::string msg;
@@ -146,6 +122,8 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution(bool storeDerivation)
writeDerivation(worker.store, *drv);
}
StorePathSet inputPaths;
{
/* If we get this far, we know no dynamic drvs inputs */
@@ -185,10 +163,23 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution(bool storeDerivation)
slot to become available, since we don't need one if there is a
build hook. */
co_await yield();
co_return tryToBuild();
co_return tryToBuild(std::move(inputPaths));
}
Goal::Co DerivationBuildingGoal::tryToBuild()
/**
* RAII wrapper for build log file.
* Constructor opens the log file, destructor closes it.
*/
struct LogFile
{
AutoCloseFD fd;
std::shared_ptr<BufferedSink> fileSink, sink;
LogFile(Store & store, const StorePath & drvPath, const LogFileSettings & logSettings);
~LogFile();
};
Goal::Co DerivationBuildingGoal::tryToBuild(StorePathSet inputPaths)
{
auto drvOptions = [&] {
DerivationOptions<SingleDerivedPath> temp;
@@ -249,33 +240,6 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
}
checkPathValidity(initialOutputs);
auto started = [&]() {
auto msg =
fmt(buildMode == bmRepair ? "repairing outputs of '%s'"
: buildMode == bmCheck ? "checking outputs of '%s'"
: "building '%s'",
worker.store.printStorePath(drvPath));
#ifndef _WIN32 // TODO enable build hook on Windows
if (hook)
msg += fmt(" on '%s'", hook->machineName);
#endif
act = std::make_unique<Activity>(
*logger,
lvlInfo,
actBuild,
msg,
Logger::Fields{
worker.store.printStorePath(drvPath),
#ifndef _WIN32 // TODO enable build hook on Windows
hook ? hook->machineName :
#endif
"",
1,
1});
mcRunningBuilds = std::make_unique<MaintainCount<uint64_t>>(worker.runningBuilds);
worker.updateProgress();
};
/**
* Activity that denotes waiting for a lock.
*/
@@ -366,7 +330,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
if (buildLocally) {
useHook = false;
} else {
switch (tryBuildHook(initialOutputs, drvOptions)) {
switch (tryBuildHook(drvOptions)) {
case rpAccept:
/* Yes, it has started doing so. Wait until we get
EOF from the hook. */
@@ -416,9 +380,6 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
Magenta(
"/usr/sbin/softwareupdate --install-rosetta && launchctl stop org.nixos.nix-daemon"));
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
builder.reset();
#endif
outputLocks.unlock();
worker.permanentFailure = true;
co_return doneFailure({BuildResult::Failure::InputRejected, std::move(msg)});
@@ -433,103 +394,229 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
actLock.reset();
if (useHook) {
buildResult.startTime = time(0); // inexact
started();
co_return buildWithHook(
std::move(inputPaths), std::move(initialOutputs), std::move(drvOptions), std::move(outputLocks));
} else {
co_return buildLocally(
std::move(inputPaths),
std::move(initialOutputs),
std::move(drvOptions),
std::move(outputLocks),
externalBuilder);
}
}
while (true) {
auto event = co_await WaitForChildEvent{};
if (auto * output = std::get_if<ChildOutput>(&event)) {
co_await processChildOutput(output->fd, output->data);
} else if (std::get_if<ChildEOF>(&event)) {
if (!currentLogLine.empty())
flushLine();
break;
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
killChild();
co_return doneFailure(std::move(*timeout));
}
}
#ifndef _WIN32
assert(hook);
#endif
trace("hook build done");
/* Since we got an EOF on the logger pipe, the builder is presumed
to have terminated. In fact, the builder could also have
simply have closed its end of the pipe, so just to be sure,
kill it. */
int status =
#ifndef _WIN32 // TODO enable build hook on Windows
hook->pid.kill();
Goal::Co DerivationBuildingGoal::buildWithHook(
StorePathSet inputPaths,
std::map<std::string, InitialOutput> initialOutputs,
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks)
{
#ifdef _WIN32 // TODO enable build hook on Windows
unreachable();
#else
0;
#endif
std::unique_ptr<HookInstance> hook = std::move(worker.hook);
debug("build hook for '%s' finished", worker.store.printStorePath(drvPath));
/* Set up callback so childTerminated is called if the hook is
destroyed (e.g., during failure cascades). */
hook->onKillChild = [this]() { worker.childTerminated(this, JobCategory::Build); };
buildResult.timesBuilt++;
buildResult.stopTime = time(0);
/* So the child is gone now. */
worker.childTerminated(this);
/* Close the read side of the logger pipe. */
#ifndef _WIN32 // TODO enable build hook on Windows
hook->builderOut.readSide.close();
hook->fromHook.readSide.close();
#endif
/* Close the log file. */
closeLogFile();
/* Check the exit status. */
if (!statusOk(status)) {
auto e = fixupBuilderFailureErrorMessage({BuildResult::Failure::MiscFailure, status, ""});
outputLocks.unlock();
/* TODO (once again) support fine-grained error codes, see issue #12641. */
co_return doneFailure(std::move(e));
}
/* Compute the FS closure of the outputs and register them as
being valid. */
auto builtOutputs =
/* When using a build hook, the build hook can register the output
as valid (by doing `nix-store --import'). If so we don't have
to do anything here.
We can only early return when the outputs are known a priori. For
floating content-addressing derivations this isn't the case.
Aborts if any output is not valid or corrupt, and otherwise
returns a 'SingleDrvOutputs' structure containing all outputs.
*/
[&] {
auto [allValid, validOutputs] = checkPathValidity(initialOutputs);
if (!allValid)
throw Error("some outputs are unexpectedly invalid");
return validOutputs;
}();
StorePathSet outputPaths;
for (auto & [_, output] : builtOutputs)
outputPaths.insert(output.outPath);
runPostBuildHook(worker.store, *logger, drvPath, outputPaths);
/* It is now safe to delete the lock files, since all future
lockers will see that the output paths are valid; they will
not create new lock files with the same names as the old
(unlinked) lock files. */
outputLocks.setDeletion(true);
outputLocks.unlock();
co_return doneSuccess(BuildResult::Success::Built, std::move(builtOutputs));
try {
hook->machineName = readLine(hook->fromHook.readSide.get());
} catch (Error & e) {
e.addTrace({}, "while reading the machine name from the build hook");
throw;
}
CommonProto::WriteConn conn{hook->sink};
/* Tell the hook all the inputs that have to be copied to the
remote system. */
CommonProto::write(worker.store, conn, inputPaths);
/* Tell the hooks the missing outputs that have to be copied back
from the remote system. */
{
StringSet missingOutputs;
for (auto & [outputName, status] : initialOutputs) {
// XXX: Does this include known CA outputs?
if (buildMode != bmCheck && status.known && status.known->isValid())
continue;
missingOutputs.insert(outputName);
}
CommonProto::write(worker.store, conn, missingOutputs);
}
hook->sink = FdSink();
hook->toHook.writeSide.close();
/* Create the log file and pipe. */
std::unique_ptr<LogFile> logFile = std::make_unique<LogFile>(worker.store, drvPath, settings.getLogFileSettings());
std::set<MuxablePipePollState::CommChannel> fds;
fds.insert(hook->fromHook.readSide.get());
fds.insert(hook->builderOut.readSide.get());
worker.childStarted(shared_from_this(), fds, false, false);
buildResult.startTime = time(0); // inexact
auto msg =
fmt(buildMode == bmRepair ? "repairing outputs of '%s'"
: buildMode == bmCheck ? "checking outputs of '%s'"
: "building '%s'",
worker.store.printStorePath(drvPath));
msg += fmt(" on '%s'", hook->machineName);
std::unique_ptr<BuildLog> buildLog = std::make_unique<BuildLog>(
settings.logLines,
std::make_unique<Activity>(
*logger,
lvlInfo,
actBuild,
msg,
Logger::Fields{worker.store.printStorePath(drvPath), hook->machineName, 1, 1}));
mcRunningBuilds = std::make_unique<MaintainCount<uint64_t>>(worker.runningBuilds);
worker.updateProgress();
std::string currentHookLine;
uint64_t logSize = 0;
while (true) {
auto event = co_await WaitForChildEvent{};
if (auto * output = std::get_if<ChildOutput>(&event)) {
auto & fd = output->fd;
auto & data = output->data;
if (fd == hook->builderOut.readSide.get()) {
logSize += data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
hook.reset();
co_return doneFailureLogTooLong(*buildLog);
}
(*buildLog)(data);
if (logFile->sink)
(*logFile->sink)(data);
} else if (fd == hook->fromHook.readSide.get()) {
for (auto c : data)
if (c == '\n') {
auto json = parseJSONMessage(currentHookLine, "the derivation builder");
if (json) {
auto s = handleJSONLogMessage(
*json, worker.act, hook->activities, "the derivation builder", true);
// ensure that logs from a builder using `ssh-ng://` as protocol
// are also available to `nix log`.
if (s && logFile->sink) {
const auto type = (*json)["type"];
const auto fields = (*json)["fields"];
if (type == resBuildLogLine) {
(*logFile->sink)((fields.size() > 0 ? fields[0].get<std::string>() : "") + "\n");
} else if (type == resSetPhase && !fields.is_null()) {
const auto phase = fields[0];
if (!phase.is_null()) {
// nixpkgs' stdenv produces lines in the log to signal
// phase changes.
// We want to get the same lines in case of remote builds.
// The format is:
// @nix { "action": "setPhase", "phase": "$curPhase" }
const auto logLine =
nlohmann::json::object({{"action", "setPhase"}, {"phase", phase}});
(*logFile->sink)(
"@nix "
+ logLine.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace)
+ "\n");
}
}
}
}
currentHookLine.clear();
} else
currentHookLine += c;
}
} else if (std::get_if<ChildEOF>(&event)) {
buildLog->flush();
break;
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
hook.reset();
co_return doneFailure(std::move(*timeout));
}
}
trace("hook build done");
/* Since we got an EOF on the logger pipe, the builder is presumed
to have terminated. In fact, the builder could also have
simply have closed its end of the pipe, so just to be sure,
kill it. */
int status = hook->pid.kill();
debug("build hook for '%s' finished", worker.store.printStorePath(drvPath));
buildResult.timesBuilt++;
buildResult.stopTime = time(0);
/* So the child is gone now. */
worker.childTerminated(this);
/* Close the read side of the logger pipe. */
hook->builderOut.readSide.close();
hook->fromHook.readSide.close();
/* Close the log file. */
logFile.reset();
/* Check the exit status. */
if (!statusOk(status)) {
auto e = fixupBuilderFailureErrorMessage({BuildResult::Failure::MiscFailure, status, ""}, *buildLog);
outputLocks.unlock();
/* TODO (once again) support fine-grained error codes, see issue #12641. */
co_return doneFailure(std::move(e));
}
/* Compute the FS closure of the outputs and register them as
being valid. */
auto builtOutputs =
/* When using a build hook, the build hook can register the output
as valid (by doing `nix-store --import'). If so we don't have
to do anything here.
We can only early return when the outputs are known a priori. For
floating content-addressing derivations this isn't the case.
Aborts if any output is not valid or corrupt, and otherwise
returns a 'SingleDrvOutputs' structure containing all outputs.
*/
[&] {
auto [allValid, validOutputs] = checkPathValidity(initialOutputs);
if (!allValid)
throw Error("some outputs are unexpectedly invalid");
return validOutputs;
}();
StorePathSet outputPaths;
for (auto & [_, output] : builtOutputs)
outputPaths.insert(output.outPath);
runPostBuildHook(worker.store, *logger, drvPath, outputPaths);
/* It is now safe to delete the lock files, since all future
lockers will see that the output paths are valid; they will
not create new lock files with the same names as the old
(unlinked) lock files. */
outputLocks.setDeletion(true);
outputLocks.unlock();
co_return doneSuccess(BuildResult::Success::Built, std::move(builtOutputs));
#endif
}
Goal::Co DerivationBuildingGoal::buildLocally(
StorePathSet inputPaths,
std::map<std::string, InitialOutput> initialOutputs,
DerivationOptions<StorePath> drvOptions,
PathLocks outputLocks,
const ExternalBuilder * externalBuilder)
{
co_await yield();
if (!dynamic_cast<LocalStore *>(&worker.store)) {
@@ -545,8 +632,31 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
#ifdef _WIN32 // TODO enable `DerivationBuilder` on Windows
throw UnimplementedError("building derivations is not yet implemented on Windows");
#else
assert(!hook);
std::unique_ptr<BuildLog> buildLog;
std::unique_ptr<LogFile> logFile;
auto openLogFile = [&]() {
logFile = std::make_unique<LogFile>(worker.store, drvPath, settings.getLogFileSettings());
};
auto closeLogFile = [&]() { logFile.reset(); };
auto started = [&]() {
auto msg =
fmt(buildMode == bmRepair ? "repairing outputs of '%s'"
: buildMode == bmCheck ? "checking outputs of '%s'"
: "building '%s'",
worker.store.printStorePath(drvPath));
buildLog = std::make_unique<BuildLog>(
settings.logLines,
std::make_unique<Activity>(
*logger, lvlInfo, actBuild, msg, Logger::Fields{worker.store.printStorePath(drvPath), "", 1, 1}));
mcRunningBuilds = std::make_unique<MaintainCount<uint64_t>>(worker.runningBuilds);
worker.updateProgress();
};
std::unique_ptr<Activity> actLock;
DerivationBuilderUnique builder;
Descriptor builderOut;
// Will continue here while waiting for a build user below
@@ -556,7 +666,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
if (curBuilds >= settings.maxBuildJobs) {
outputLocks.unlock();
co_await waitForBuildSlot();
co_return tryToBuild();
co_return tryToBuild(std::move(inputPaths));
}
if (!builder) {
@@ -567,10 +677,16 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
struct DerivationBuildingGoalCallbacks : DerivationBuilderCallbacks
{
DerivationBuildingGoal & goal;
std::function<void()> openLogFileFn;
std::function<void()> closeLogFileFn;
DerivationBuildingGoalCallbacks(
DerivationBuildingGoal & goal, std::unique_ptr<DerivationBuilder> & builder)
DerivationBuildingGoal & goal,
std::function<void()> openLogFileFn,
std::function<void()> closeLogFileFn)
: goal{goal}
, openLogFileFn{std::move(openLogFileFn)}
, closeLogFileFn{std::move(closeLogFileFn)}
{
}
@@ -578,17 +694,17 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
void childTerminated() override
{
goal.worker.childTerminated(&goal);
goal.worker.childTerminated(&goal, JobCategory::Build);
}
std::filesystem::path openLogFile() override
void openLogFile() override
{
return goal.openLogFile();
openLogFileFn();
}
void closeLogFile() override
{
goal.closeLogFile();
closeLogFileFn();
}
};
@@ -607,7 +723,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
worker.store.toStorePath(i.second.source.string()).first, closure);
} catch (InvalidPath & e) {
} catch (Error & e) {
e.addTrace({}, "while processing sandbox path '%s'", i.second.source);
e.addTrace({}, "while processing sandbox path %s", PathFmt(i.second.source));
throw;
}
for (auto & i : closure) {
@@ -638,15 +754,16 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
/* If we have to wait and retry (see below), then `builder` will
already be created, so we don't need to create it again. */
builder = externalBuilder ? makeExternalDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
std::move(params),
*externalBuilder)
: makeDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
std::move(params));
builder = externalBuilder
? makeExternalDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, openLogFile, closeLogFile),
std::move(params),
*externalBuilder)
: makeDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, openLogFile, closeLogFile),
std::move(params));
}
if (auto builderOutOpt = builder->startBuild()) {
@@ -671,16 +788,26 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
started();
uint64_t logSize = 0;
while (true) {
auto event = co_await WaitForChildEvent{};
if (auto * output = std::get_if<ChildOutput>(&event)) {
co_await processChildOutput(output->fd, output->data);
if (output->fd == builder->builderOut.get()) {
logSize += output->data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
builder->killChild();
co_return doneFailureLogTooLong(*buildLog);
}
(*buildLog)(output->data);
if (logFile->sink)
(*logFile->sink)(output->data);
}
} else if (std::get_if<ChildEOF>(&event)) {
if (!currentLogLine.empty())
flushLine();
buildLog->flush();
break;
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
killChild();
builder->killChild();
co_return doneFailure(std::move(*timeout));
}
}
@@ -693,7 +820,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
} catch (BuilderFailureError & e) {
builder.reset();
outputLocks.unlock();
co_return doneFailure(fixupBuilderFailureErrorMessage(std::move(e)));
co_return doneFailure(fixupBuilderFailureErrorMessage(std::move(e), *buildLog));
} catch (BuildError & e) {
builder.reset();
outputLocks.unlock();
@@ -814,7 +941,7 @@ static void runPostBuildHook(
});
}
BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailureError e)
BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailureError e, BuildLog & buildLog)
{
auto msg =
fmt("Cannot build '%s'.\n"
@@ -824,6 +951,7 @@ BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailur
msg += showKnownOutputs(worker.store, *drv);
auto & logTail = buildLog.getTail();
if (!logger->isVerbose() && !logTail.empty()) {
msg += fmt("\nLast %d log lines:\n", logTail.size());
for (auto & line : logTail) {
@@ -846,8 +974,7 @@ BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailur
return BuildError{e.status, msg};
}
HookReply DerivationBuildingGoal::tryBuildHook(
const std::map<std::string, InitialOutput> & initialOutputs, const DerivationOptions<StorePath> & drvOptions)
HookReply DerivationBuildingGoal::tryBuildHook(const DerivationOptions<StorePath> & drvOptions)
{
#ifdef _WIN32 // TODO enable build hook on Windows
return rpDecline;
@@ -903,8 +1030,8 @@ HookReply DerivationBuildingGoal::tryBuildHook(
else if (reply != "accept")
throw Error("bad hook reply '%s'", reply);
} catch (SysError & e) {
if (e.errNo == EPIPE) {
} catch (SystemError & e) {
if (e.is(std::errc::broken_pipe)) {
printError("build hook died unexpectedly: %s", chomp(drainFD(worker.hook->fromHook.readSide.get())));
worker.hook = 0;
return rpDecline;
@@ -912,70 +1039,28 @@ HookReply DerivationBuildingGoal::tryBuildHook(
throw;
}
hook = std::move(worker.hook);
try {
hook->machineName = readLine(hook->fromHook.readSide.get());
} catch (Error & e) {
e.addTrace({}, "while reading the machine name from the build hook");
throw;
}
CommonProto::WriteConn conn{hook->sink};
/* Tell the hook all the inputs that have to be copied to the
remote system. */
CommonProto::write(worker.store, conn, inputPaths);
/* Tell the hooks the missing outputs that have to be copied back
from the remote system. */
{
StringSet missingOutputs;
for (auto & [outputName, status] : initialOutputs) {
// XXX: Does this include known CA outputs?
if (buildMode != bmCheck && status.known && status.known->isValid())
continue;
missingOutputs.insert(outputName);
}
CommonProto::write(worker.store, conn, missingOutputs);
}
hook->sink = FdSink();
hook->toHook.writeSide.close();
/* Create the log file and pipe. */
[[maybe_unused]] Path logFile = openLogFile();
std::set<MuxablePipePollState::CommChannel> fds;
fds.insert(hook->fromHook.readSide.get());
fds.insert(hook->builderOut.readSide.get());
worker.childStarted(shared_from_this(), fds, false, false);
return rpAccept;
#endif
}
Path DerivationBuildingGoal::openLogFile()
LogFile::LogFile(Store & store, const StorePath & drvPath, const LogFileSettings & logSettings)
{
logSize = 0;
if (!logSettings.keepLog)
return;
if (!settings.keepLog)
return "";
auto baseName = std::string(baseNameOf(store.printStorePath(drvPath)));
auto baseName = std::string(baseNameOf(worker.store.printStorePath(drvPath)));
/* Create a log file. */
Path logDir;
if (auto localStore = dynamic_cast<LocalStore *>(&worker.store))
if (auto localStore = dynamic_cast<LocalStore *>(&store))
logDir = localStore->config->logDir;
else
logDir = settings.nixLogDir;
logDir = logSettings.nixLogDir;
Path dir = fmt("%s/%s/%s/", logDir, LocalFSStore::drvsLogDir, baseName.substr(0, 2));
createDirs(dir);
Path logFileName = fmt("%s/%s%s", dir, baseName.substr(2), settings.compressLog ? ".bz2" : "");
Path logFileName = fmt("%s/%s%s", dir, baseName.substr(2), logSettings.compressLog ? ".bz2" : "");
fdLogFile = toDescriptor(open(
fd = toDescriptor(open(
logFileName.c_str(),
O_CREAT | O_WRONLY | O_TRUNC
#ifndef _WIN32
@@ -983,122 +1068,37 @@ Path DerivationBuildingGoal::openLogFile()
#endif
,
0666));
if (!fdLogFile)
if (!fd)
throw SysError("creating log file '%1%'", logFileName);
logFileSink = std::make_shared<FdSink>(fdLogFile.get());
fileSink = std::make_shared<FdSink>(fd.get());
if (settings.compressLog)
logSink = std::shared_ptr<CompressionSink>(makeCompressionSink(CompressionAlgo::bzip2, *logFileSink));
if (logSettings.compressLog)
sink = std::shared_ptr<CompressionSink>(makeCompressionSink(CompressionAlgo::bzip2, *fileSink));
else
logSink = logFileSink;
return logFileName;
sink = fileSink;
}
void DerivationBuildingGoal::closeLogFile()
LogFile::~LogFile()
{
auto logSink2 = std::dynamic_pointer_cast<CompressionSink>(logSink);
if (logSink2)
logSink2->finish();
if (logFileSink)
logFileSink->flush();
logSink = logFileSink = 0;
fdLogFile.close();
}
bool DerivationBuildingGoal::isReadDesc(Descriptor fd)
{
#ifdef _WIN32 // TODO enable build hook on Windows
return false;
#else
return (hook && fd == hook->builderOut.readSide.get()) || (builder && fd == builder->builderOut.get());
#endif
}
Goal::Co DerivationBuildingGoal::processChildOutput(Descriptor fd, std::string_view data)
{
// local & `ssh://`-builds are dealt with here.
auto isWrittenToLog = isReadDesc(fd);
if (isWrittenToLog) {
logSize += data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
killChild();
co_return doneFailure(BuildError(
BuildResult::Failure::LogLimitExceeded,
"%s killed after writing more than %d bytes of log output",
getName(),
settings.maxLogSize));
}
for (auto c : data)
if (c == '\r')
currentLogLinePos = 0;
else if (c == '\n')
flushLine();
else {
if (currentLogLinePos >= currentLogLine.size())
currentLogLine.resize(currentLogLinePos + 1);
currentLogLine[currentLogLinePos++] = c;
}
if (logSink)
(*logSink)(data);
try {
auto sink2 = std::dynamic_pointer_cast<CompressionSink>(sink);
if (sink2)
sink2->finish();
if (fileSink)
fileSink->flush();
} catch (...) {
ignoreExceptionInDestructor();
}
#ifndef _WIN32 // TODO enable build hook on Windows
if (hook && fd == hook->fromHook.readSide.get()) {
for (auto c : data)
if (c == '\n') {
auto json = parseJSONMessage(currentHookLine, "the derivation builder");
if (json) {
auto s = handleJSONLogMessage(*json, worker.act, hook->activities, "the derivation builder", true);
// ensure that logs from a builder using `ssh-ng://` as protocol
// are also available to `nix log`.
if (s && !isWrittenToLog && logSink) {
const auto type = (*json)["type"];
const auto fields = (*json)["fields"];
if (type == resBuildLogLine) {
(*logSink)((fields.size() > 0 ? fields[0].get<std::string>() : "") + "\n");
} else if (type == resSetPhase && !fields.is_null()) {
const auto phase = fields[0];
if (!phase.is_null()) {
// nixpkgs' stdenv produces lines in the log to signal
// phase changes.
// We want to get the same lines in case of remote builds.
// The format is:
// @nix { "action": "setPhase", "phase": "$curPhase" }
const auto logLine = nlohmann::json::object({{"action", "setPhase"}, {"phase", phase}});
(*logSink)(
"@nix " + logLine.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace)
+ "\n");
}
}
}
}
currentHookLine.clear();
} else
currentHookLine += c;
}
#endif
co_return Return{};
}
void DerivationBuildingGoal::flushLine()
Goal::Done DerivationBuildingGoal::doneFailureLogTooLong(BuildLog & buildLog)
{
if (handleJSONLogMessage(currentLogLine, *act, builderActivities, "the derivation builder", false))
;
else {
logTail.push_back(currentLogLine);
if (logTail.size() > settings.logLines)
logTail.pop_front();
act->result(resBuildLogLine, currentLogLine);
}
currentLogLine = "";
currentLogLinePos = 0;
return doneFailure(BuildError(
BuildResult::Failure::LogLimitExceeded,
"%s killed after writing more than %d bytes of log output",
getName(),
settings.maxLogSize));
}
std::map<std::string, std::optional<StorePath>> DerivationBuildingGoal::queryPartialDerivationOutputMap()
@@ -1186,11 +1186,6 @@ DerivationBuildingGoal::checkPathValidity(std::map<std::string, InitialOutput> &
Goal::Done DerivationBuildingGoal::doneSuccess(BuildResult::Success::Status status, SingleDrvOutputs builtOutputs)
{
buildResult.inner = BuildResult::Success{
.status = status,
.builtOutputs = std::move(builtOutputs),
};
mcRunningBuilds.reset();
if (status == BuildResult::Success::Built)
@@ -1198,16 +1193,15 @@ Goal::Done DerivationBuildingGoal::doneSuccess(BuildResult::Success::Status stat
worker.updateProgress();
return amDone(ecSuccess, std::nullopt);
return Goal::doneSuccess(
BuildResult::Success{
.status = status,
.builtOutputs = std::move(builtOutputs),
});
}
Goal::Done DerivationBuildingGoal::doneFailure(BuildError ex)
{
buildResult.inner = BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
};
mcRunningBuilds.reset();
if (ex.status == BuildResult::Failure::TimedOut)
@@ -1219,7 +1213,13 @@ Goal::Done DerivationBuildingGoal::doneFailure(BuildError ex)
worker.updateProgress();
return amDone(ecFailed, {std::move(ex)});
return Goal::doneFailure(
ecFailed,
BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
},
std::move(ex));
}
} // namespace nix
+21 -21
View File
@@ -465,20 +465,6 @@ UnkeyedRealisation DerivationGoal::assertPathValidity()
Goal::Done DerivationGoal::doneSuccess(BuildResult::Success::Status status, UnkeyedRealisation builtOutput)
{
buildResult.inner = BuildResult::Success{
.status = status,
.builtOutputs = {{
wantedOutput,
{
std::move(builtOutput),
DrvOutput{
.drvHash = outputHash,
.outputName = wantedOutput,
},
},
}},
};
mcExpectedBuilds.reset();
if (status == BuildResult::Success::Built)
@@ -486,16 +472,24 @@ Goal::Done DerivationGoal::doneSuccess(BuildResult::Success::Status status, Unke
worker.updateProgress();
return amDone(ecSuccess, std::nullopt);
return Goal::doneSuccess(
BuildResult::Success{
.status = status,
.builtOutputs = {{
wantedOutput,
{
std::move(builtOutput),
DrvOutput{
.drvHash = outputHash,
.outputName = wantedOutput,
},
},
}},
});
}
Goal::Done DerivationGoal::doneFailure(BuildError ex)
{
buildResult.inner = BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
};
mcExpectedBuilds.reset();
if (ex.status == BuildResult::Failure::TimedOut)
@@ -507,7 +501,13 @@ Goal::Done DerivationGoal::doneFailure(BuildError ex)
worker.updateProgress();
return amDone(ecFailed, {std::move(ex)});
return Goal::doneFailure(
ecFailed,
BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
},
std::move(ex));
}
} // namespace nix
@@ -98,7 +98,13 @@ Goal::Co DerivationTrampolineGoal::init()
trace("outer load and build derivation");
if (nrFailed != 0) {
co_return amDone(ecFailed, Error("cannot build missing derivation '%s'", drvReq->to_string(worker.store)));
co_return doneFailure(
ecFailed,
BuildResult::Failure{
.status = BuildResult::Failure::DependencyFailed,
.errorMsg = fmt("failed to obtain derivation of '%s'", drvReq->to_string(worker.store)),
},
Error("failed to obtain derivation of '%s'", drvReq->to_string(worker.store)));
}
StorePath drvPath = resolveDerivedPath(worker.store, *drvReq);
+7 -1
View File
@@ -63,12 +63,18 @@ std::vector<KeyedBuildResult> Store::buildPathsWithResults(
std::vector<KeyedBuildResult> results;
results.reserve(state.size());
for (auto & [req, goalPtr] : state)
for (auto & [req, goalPtr] : state) {
/* Goals that were never started or were cancelled have exitCode
ecBusy and a default buildResult with empty errorMsg. Skip them
to avoid reporting spurious failures with empty messages. */
if (goalPtr->exitCode == Goal::ecBusy)
continue;
results.emplace_back(
KeyedBuildResult{
goalPtr->buildResult,
/* .path = */ req,
});
}
return results;
}
+13
View File
@@ -183,6 +183,19 @@ Co Goal::await(Goals new_waitees)
co_return Return{};
}
Goal::Done Goal::doneSuccess(BuildResult::Success success)
{
buildResult.inner = std::move(success);
return amDone(ecSuccess);
}
Goal::Done Goal::doneFailure(ExitCode result, BuildResult::Failure failure, std::optional<Error> ex)
{
assert(result == ecFailed || result == ecNoSubstituters);
buildResult.inner = std::move(failure);
return amDone(result, std::move(ex));
}
Goal::Done Goal::amDone(ExitCode result, std::optional<Error> ex)
{
trace("done");
+11 -10
View File
@@ -28,20 +28,21 @@ PathSubstitutionGoal::~PathSubstitutionGoal()
Goal::Done PathSubstitutionGoal::doneSuccess(BuildResult::Success::Status status)
{
buildResult.inner = BuildResult::Success{
.status = status,
};
return amDone(ecSuccess);
return Goal::doneSuccess(
BuildResult::Success{
.status = status,
});
}
Goal::Done PathSubstitutionGoal::doneFailure(ExitCode result, BuildResult::Failure::Status status, std::string errorMsg)
{
debug(errorMsg);
buildResult.inner = BuildResult::Failure{
.status = status,
.errorMsg = std::move(errorMsg),
};
return amDone(result);
return Goal::doneFailure(
result,
BuildResult::Failure{
.status = status,
.errorMsg = std::move(errorMsg),
});
}
Goal::Co PathSubstitutionGoal::init()
@@ -324,7 +325,7 @@ void PathSubstitutionGoal::cleanup()
if (thr.joinable()) {
// FIXME: signal worker thread to quit.
thr.join();
worker.childTerminated(this);
worker.childTerminated(this, JobCategory::Substitution);
}
outPipe.close();
+15 -4
View File
@@ -162,7 +162,9 @@ template<typename G>
static bool
removeGoal(std::shared_ptr<G> goal, typename DerivedPathMap<std::map<OutputsSpec, std::weak_ptr<G>>>::ChildNode & node)
{
return removeGoal(goal, node.value) || removeGoal(goal, node.childMap);
bool valueKeep = removeGoal(goal, node.value);
bool childMapKeep = removeGoal(goal, node.childMap);
return valueKeep || childMapKeep;
}
void Worker::removeGoal(GoalPtr goal)
@@ -245,13 +247,18 @@ void Worker::childStarted(
}
void Worker::childTerminated(Goal * goal, bool wakeSleepers)
{
childTerminated(goal, goal->jobCategory(), wakeSleepers);
}
void Worker::childTerminated(Goal * goal, JobCategory jobCategory, bool wakeSleepers)
{
auto i = std::find_if(children.begin(), children.end(), [&](const Child & child) { return child.goal2 == goal; });
if (i == children.end())
return;
if (i->inBuildSlot) {
switch (goal->jobCategory()) {
switch (jobCategory) {
case JobCategory::Substitution:
assert(nrSubstitutions > 0);
nrSubstitutions--;
@@ -401,8 +408,12 @@ void Worker::waitForInput()
is a build timeout, then wait for input until the first
deadline for any child. */
auto nearest = steady_time_point::max(); // nearest deadline
if (settings.minFree.get() != 0)
// Periodicallty wake up to see if we need to run the garbage collector.
auto localStore = dynamic_cast<LocalStore *>(&store);
if (localStore && localStore->config->getGCSettings().minFree.get() != 0)
// If we have a local store (and thus are capable of automatically collecting garbage) and configured to do so,
// periodically wake up to see if we need to run the garbage collector. (See the `autoGC` call site above in
// this file, also gated on having a local store. when we wake up, we intended to reach that call site.)
nearest = before + std::chrono::seconds(10);
for (auto & i : children) {
if (!i.respectTimeouts)
+6 -6
View File
@@ -33,8 +33,8 @@ static void createLinks(State & state, const Path & srcDir, const Path & dstDir,
try {
srcFiles = DirectoryIterator{srcDir};
} catch (SysError & e) {
if (e.errNo == ENOTDIR) {
} catch (SystemError & e) {
if (e.is(std::errc::not_a_directory)) {
warn("not including '%s' in the user environment because it's not a directory", srcDir);
return;
}
@@ -54,8 +54,8 @@ static void createLinks(State & state, const Path & srcDir, const Path & dstDir,
try {
if (stat(srcFile.c_str(), &srcSt) == -1)
throw SysError("getting status of '%1%'", srcFile);
} catch (SysError & e) {
if (e.errNo == ENOENT || e.errNo == ENOTDIR) {
} catch (SystemError & e) {
if (e.is(std::errc::no_such_file_or_directory) || e.is(std::errc::not_a_directory)) {
warn("skipping dangling symlink '%s'", dstFile);
continue;
}
@@ -141,8 +141,8 @@ void buildProfile(const Path & out, Packages && pkgs)
readFile(pkgDir + "/nix-support/propagated-user-env-packages"), " \n"))
if (!done.count(p))
postponed.insert(p);
} catch (SysError & e) {
if (e.errNo != ENOENT && e.errNo != ENOTDIR)
} catch (SystemError & e) {
if (!e.is(std::errc::no_such_file_or_directory) && !e.is(std::errc::not_a_directory))
throw;
}
};
+2 -2
View File
@@ -4,6 +4,7 @@
#include "nix/store/globals.hh"
#include "nix/util/archive.hh"
#include "nix/util/compression.hh"
#include "nix/util/file-system.hh"
namespace nix {
@@ -65,8 +66,7 @@ static void builtinFetchurl(const BuiltinBuilderContext & ctx)
auto executable = ctx.drv.env.find("executable");
if (executable != ctx.drv.env.end() && executable->second == "1") {
if (chmod(storePath.c_str(), 0755) == -1)
throw SysError("making '%1%' executable", storePath);
chmod(storePath, 0755);
}
};
+12
View File
@@ -6,6 +6,7 @@
#include "nix/store/common-protocol-impl.hh"
#include "nix/util/archive.hh"
#include "nix/store/derivations.hh"
#include "nix/util/signature/local-keys.hh"
#include <nlohmann/json.hpp>
@@ -99,4 +100,15 @@ void CommonProto::Serialise<std::optional<ContentAddress>>::write(
conn.to << (caOpt ? renderContentAddress(*caOpt) : "");
}
Signature CommonProto::Serialise<Signature>::read(const StoreDirConfig & store, CommonProto::ReadConn conn)
{
return Signature::parse(readString(conn.from));
}
void CommonProto::Serialise<Signature>::write(
const StoreDirConfig & store, CommonProto::WriteConn conn, const Signature & sig)
{
conn.to << sig.to_string();
}
} // namespace nix
+3 -31
View File
@@ -339,17 +339,6 @@ static void performOp(
break;
}
case WorkerProto::Op::HasSubstitutes: {
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
StorePathSet paths; // FIXME
paths.insert(path);
auto res = store->querySubstitutablePaths(paths);
logger->stopWork();
conn.to << (res.count(path) != 0);
break;
}
case WorkerProto::Op::QuerySubstitutablePaths: {
auto paths = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
logger->startWork();
@@ -359,26 +348,13 @@ static void performOp(
break;
}
case WorkerProto::Op::QueryPathHash: {
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
auto hash = store->queryPathInfo(path)->narHash;
logger->stopWork();
conn.to << hash.to_string(HashFormat::Base16, false);
break;
}
case WorkerProto::Op::QueryReferences:
case WorkerProto::Op::QueryReferrers:
case WorkerProto::Op::QueryValidDerivers:
case WorkerProto::Op::QueryDerivationOutputs: {
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
StorePathSet paths;
if (op == WorkerProto::Op::QueryReferences)
for (auto & i : store->queryPathInfo(path)->references)
paths.insert(i);
else if (op == WorkerProto::Op::QueryReferrers)
if (op == WorkerProto::Op::QueryReferrers)
store->queryReferrers(path, paths);
else if (op == WorkerProto::Op::QueryValidDerivers)
paths = store->queryValidDerivers(path);
@@ -885,7 +861,7 @@ static void performOp(
case WorkerProto::Op::AddSignatures: {
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
StringSet sigs = readStrings<StringSet>(conn.from);
auto sigs = WorkerProto::Serialise<std::set<Signature>>::read(*store, rconn);
logger->startWork();
store->addSignatures(path, sigs);
logger->stopWork();
@@ -910,7 +886,7 @@ static void performOp(
info.deriver = std::move(deriver);
info.references = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
conn.from >> info.registrationTime >> info.narSize >> info.ultimate;
info.sigs = readStrings<StringSet>(conn.from);
info.sigs = WorkerProto::Serialise<std::set<Signature>>::read(*store, rconn);
info.ca = ContentAddress::parseOpt(readString(conn.from));
conn.from >> repair >> dontCheckSigs;
if (!trusted && dontCheckSigs)
@@ -1012,10 +988,6 @@ static void performOp(
break;
}
case WorkerProto::Op::QueryFailedPaths:
case WorkerProto::Op::ClearFailedPaths:
throw Error("Removed operation %1%", op);
default:
throw Error("invalid operation %1%", op);
}
+54 -92
View File
@@ -21,76 +21,53 @@ static std::optional<std::string>
getStringAttr(const StringMap & env, const StructuredAttrs * parsed, const std::string & name)
{
if (parsed) {
auto i = parsed->structuredAttrs.find(name);
if (i == parsed->structuredAttrs.end())
return {};
else {
if (!i->second.is_string())
throw Error("attribute '%s' of must be a string", name);
return i->second.get<std::string>();
}
if (auto * i = get(parsed->structuredAttrs, name))
try {
return getString(*i);
} catch (Error & e) {
e.addTrace({}, "while parsing attribute \"%s\"", name);
throw;
}
} else {
auto i = env.find(name);
if (i == env.end())
return {};
else
return i->second;
if (auto * i = get(env, name))
return *i;
}
return {};
}
static bool getBoolAttr(const StringMap & env, const StructuredAttrs * parsed, const std::string & name, bool def)
{
if (parsed) {
auto i = parsed->structuredAttrs.find(name);
if (i == parsed->structuredAttrs.end())
return def;
else {
if (!i->second.is_boolean())
throw Error("attribute '%s' must be a Boolean", name);
return i->second.get<bool>();
}
} else {
auto i = env.find(name);
if (i == env.end())
return def;
else
return i->second == "1";
}
}
static std::optional<Strings>
getStringsAttr(const StringMap & env, const StructuredAttrs * parsed, const std::string & name)
{
if (parsed) {
auto i = parsed->structuredAttrs.find(name);
if (i == parsed->structuredAttrs.end())
return {};
else {
if (!i->second.is_array())
throw Error("attribute '%s' must be a list of strings", name);
auto & a = getArray(i->second);
Strings res;
for (auto j = a.begin(); j != a.end(); ++j) {
if (!j->is_string())
throw Error("attribute '%s' must be a list of strings", name);
res.push_back(j->get<std::string>());
if (auto * i = get(parsed->structuredAttrs, name))
try {
return getBoolean(*i);
} catch (Error & e) {
e.addTrace({}, "while parsing attribute \"%s\"", name);
throw;
}
return res;
}
} else {
auto i = env.find(name);
if (i == env.end())
return {};
else
return tokenizeString<Strings>(i->second);
if (auto * i = get(env, name))
return *i == "1";
}
return def;
}
static std::optional<StringSet>
getStringSetAttr(const StringMap & env, const StructuredAttrs * parsed, const std::string & name)
{
auto ss = getStringsAttr(env, parsed, name);
return ss ? (std::optional{StringSet{ss->begin(), ss->end()}}) : (std::optional<StringSet>{});
if (parsed) {
if (auto * i = get(parsed->structuredAttrs, name))
try {
return getStringSet(*i);
} catch (Error & e) {
e.addTrace({}, "while parsing attribute \"%s\"", name);
throw;
}
} else {
if (auto * i = get(env, name))
return tokenizeString<StringSet>(*i);
}
return {};
}
template<typename Inputs>
@@ -233,26 +210,21 @@ DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
std::map<std::string, OutputChecks<SingleDerivedPath>> res;
if (auto * outputChecks = get(structuredAttrs, "outputChecks")) {
for (auto & [outputName, output_] : getObject(*outputChecks)) {
OutputChecks<SingleDerivedPath> checks;
auto & output = getObject(output_);
if (auto maxSize = get(output, "maxSize"))
checks.maxSize = maxSize->get<uint64_t>();
if (auto maxClosureSize = get(output, "maxClosureSize"))
checks.maxClosureSize = maxClosureSize->get<uint64_t>();
auto get_ =
[&](const std::string & name) -> std::optional<std::set<DrvRef<SingleDerivedPath>>> {
if (auto i = get(output, name)) {
std::set<DrvRef<SingleDerivedPath>> res;
for (auto j = i->begin(); j != i->end(); ++j) {
if (!j->is_string())
throw Error("attribute '%s' must be a list of strings", name);
res.insert(parseRef(j->get<std::string>()));
if (auto * i = get(output, name)) {
try {
std::set<DrvRef<SingleDerivedPath>> res;
for (auto & s : getStringList(*i))
res.insert(parseRef(s));
return res;
} catch (Error & e) {
e.addTrace(
{}, "while parsing attribute 'outputChecks.\"%s\".%s'", outputName, name);
throw;
}
return res;
}
return {};
};
@@ -260,18 +232,8 @@ DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
res.insert_or_assign(
outputName,
OutputChecks<SingleDerivedPath>{
.maxSize = [&]() -> std::optional<uint64_t> {
if (auto maxSize = get(output, "maxSize"))
return maxSize->get<uint64_t>();
else
return std::nullopt;
}(),
.maxClosureSize = [&]() -> std::optional<uint64_t> {
if (auto maxClosureSize = get(output, "maxClosureSize"))
return maxClosureSize->get<uint64_t>();
else
return std::nullopt;
}(),
.maxSize = ptrToOwned<uint64_t>(get(output, "maxSize")),
.maxClosureSize = ptrToOwned<uint64_t>(get(output, "maxClosureSize")),
.allowedReferences = get_("allowedReferences"),
.disallowedReferences =
get_("disallowedReferences").value_or(std::set<DrvRef<SingleDerivedPath>>{}),
@@ -307,13 +269,13 @@ DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
std::map<std::string, bool> res;
if (parsed) {
auto & structuredAttrs = parsed->structuredAttrs;
if (auto * udr = get(structuredAttrs, "unsafeDiscardReferences")) {
for (auto & [outputName, output] : getObject(*udr)) {
if (!output.is_boolean())
throw Error("attribute 'unsafeDiscardReferences.\"%s\"' must be a Boolean", outputName);
res.insert_or_assign(outputName, output.get<bool>());
if (auto * udr = get(parsed->structuredAttrs, "unsafeDiscardReferences")) {
try {
for (auto & [outputName, output] : getObject(*udr))
res.insert_or_assign(outputName, getBoolean(output));
} catch (Error & e) {
e.addTrace({}, "while parsing attribute 'unsafeDiscardReferences'");
throw;
}
}
}
@@ -340,7 +302,7 @@ DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
std::map<std::string, std::set<SingleDerivedPath>> ret;
if (parsed) {
auto * e = optionalValueAt(parsed->structuredAttrs, "exportReferencesGraph");
auto * e = get(parsed->structuredAttrs, "exportReferencesGraph");
if (!e)
return ret;
if (!e->is_object()) {
@@ -595,8 +557,8 @@ DerivationOptions<SingleDerivedPath> adl_serializer<DerivationOptions<SingleDeri
.outputChecks = [&]() -> OutputChecksVariant<SingleDerivedPath> {
auto outputChecks = getObject(valueAt(json, "outputChecks"));
auto forAllOutputsOpt = optionalValueAt(outputChecks, "forAllOutputs");
auto perOutputOpt = optionalValueAt(outputChecks, "perOutput");
auto forAllOutputsOpt = get(outputChecks, "forAllOutputs");
auto perOutputOpt = get(outputChecks, "perOutput");
if (forAllOutputsOpt && !perOutputOpt) {
return static_cast<OutputChecks<SingleDerivedPath>>(*forAllOutputsOpt);
-6
View File
@@ -80,12 +80,6 @@ public:
subdirs.emplace(baseName, std::move(accessor));
}
std::string readFile(const CanonPath & path) override
{
return callWithAccessorForPath(
path, [](SourceAccessor & accessor, const CanonPath & path) { return accessor.readFile(path); });
}
void readFile(const CanonPath & path, Sink & sink, std::function<void(uint64_t)> sizeCallback) override
{
return callWithAccessorForPath(path, [&](SourceAccessor & accessor, const CanonPath & path) {
+75 -10
View File
@@ -69,7 +69,7 @@ struct curlFileTransfer : public FileTransfer
curlFileTransfer & fileTransfer;
FileTransferRequest request;
FileTransferResult result;
Activity act;
std::unique_ptr<Activity> _act;
bool done = false; // whether either the success or failure function has been called
Callback<FileTransferResult> callback;
CURL * req = 0;
@@ -124,12 +124,6 @@ struct curlFileTransfer : public FileTransfer
Callback<FileTransferResult> && callback)
: fileTransfer(fileTransfer)
, request(request)
, act(*logger,
lvlTalkative,
actFileTransfer,
fmt("%s '%s'", request.verb(/*continuous=*/true), request.uri),
{request.uri.to_string()},
request.parentAct)
, callback(std::move(callback))
, finalSink([this](std::string_view data) {
if (errorSink) {
@@ -157,7 +151,15 @@ struct curlFileTransfer : public FileTransfer
{
result.urls.push_back(request.uri.to_string());
appendHeaders("Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
/* Don't set Accept-Encoding for S3 requests that use AWS SigV4 signing.
curl's SigV4 implementation signs all headers including Accept-Encoding,
but some S3-compatible services (like GCS) modify this header in transit,
causing signature verification to fail.
See https://github.com/NixOS/nix/issues/15019 */
#if NIX_WITH_AWS_AUTH
if (!request.awsSigV4Provider)
#endif
appendHeaders("Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
if (!request.expectedETag.empty())
appendHeaders("If-None-Match: " + request.expectedETag);
if (!request.mimeType.empty())
@@ -325,9 +327,29 @@ struct curlFileTransfer : public FileTransfer
return ((TransferItem *) userp)->headerCallback(contents, size, nmemb);
}
/**
* Lazily start an `Activity`. We don't do this in the `TransferItem` constructor to avoid showing downloads
* that are only enqueued but not actually started.
*/
Activity & act()
{
if (!_act) {
_act = std::make_unique<Activity>(
*logger,
lvlTalkative,
actFileTransfer,
fmt("%s '%s'", request.verb(/*continuous=*/true), request.uri),
Logger::Fields{request.uri.to_string()},
request.parentAct);
// Reset the start time to when we actually started the download.
startTime = std::chrono::steady_clock::now();
}
return *_act;
}
int progressCallback(curl_off_t dltotal, curl_off_t dlnow) noexcept
try {
act.progress(dlnow, dltotal);
act().progress(dlnow, dltotal);
return getInterrupted();
} catch (nix::Interrupted &) {
assert(getInterrupted());
@@ -404,6 +426,15 @@ struct curlFileTransfer : public FileTransfer
return ((TransferItem *) clientp)->seekCallback(offset, origin);
}
static int resolverCallbackWrapper(void *, void *, void * clientp) noexcept
try {
// Create the `Activity` associated with this download.
((TransferItem *) clientp)->act();
return 0;
} catch (...) {
return 1;
}
void unpause()
{
/* Unpausing an already unpaused transfer is a no-op. */
@@ -461,6 +492,11 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_CUSTOMREQUEST, "DELETE");
if (request.data) {
// Restart the source to ensure it's at the beginning.
// This is necessary for retries, where the source was
// already consumed by a previous attempt.
request.data->source->restart();
if (request.method == HttpMethod::Post) {
curl_easy_setopt(req, CURLOPT_POST, 1L);
curl_easy_setopt(req, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t) request.data->sizeHint);
@@ -495,6 +531,21 @@ struct curlFileTransfer : public FileTransfer
if (writtenToSink)
curl_easy_setopt(req, CURLOPT_RESUME_FROM_LARGE, writtenToSink);
/* Note that the underlying strings get copied by libcurl, so the path -> string conversion is ok:
> The application does not have to keep the string around after setting this option.
https://curl.se/libcurl/c/CURLOPT_SSLKEY.html
https://curl.se/libcurl/c/CURLOPT_SSLCERT.html */
if (request.tlsCert) {
curl_easy_setopt(req, CURLOPT_SSLCERTTYPE, "PEM");
curl_easy_setopt(req, CURLOPT_SSLCERT, request.tlsCert->string().c_str());
}
if (request.tlsKey) {
curl_easy_setopt(req, CURLOPT_SSLKEYTYPE, "PEM");
curl_easy_setopt(req, CURLOPT_SSLKEY, request.tlsKey->string().c_str());
}
curl_easy_setopt(req, CURLOPT_ERRORBUFFER, errbuf);
errbuf[0] = 0;
@@ -516,6 +567,11 @@ struct curlFileTransfer : public FileTransfer
}
#endif
// This seems to be the earliest libcurl callback that signals that the download is happening, so we can
// call act().
curl_easy_setopt(req, CURLOPT_RESOLVER_START_FUNCTION, resolverCallbackWrapper);
curl_easy_setopt(req, CURLOPT_RESOLVER_START_DATA, this);
result.data.clear();
result.bodySize = 0;
}
@@ -564,7 +620,7 @@ struct curlFileTransfer : public FileTransfer
if (httpStatus == 304 && result.etag == "")
result.etag = request.expectedETag;
act.progress(result.bodySize, result.bodySize);
act().progress(result.bodySize, result.bodySize);
done = true;
callback(std::move(result));
}
@@ -715,6 +771,8 @@ struct curlFileTransfer : public FileTransfer
std::thread workerThread;
const size_t maxQueueSize = fileTransferSettings.httpConnections.get() * 5;
curlFileTransfer()
: mt19937(rd())
{
@@ -820,6 +878,13 @@ struct curlFileTransfer : public FileTransfer
{
auto state(state_.lock());
while (!state->incoming.empty()) {
/* Limit the number of active curl handles, since curl doesn't scale well. */
if (items.size() + incoming.size() >= maxQueueSize) {
auto t = now + std::chrono::milliseconds(100);
if (nextWakeup == std::chrono::steady_clock::time_point() || t < nextWakeup)
nextWakeup = t;
break;
}
auto item = state->incoming.top();
if (item->embargo <= now) {
incoming.push_back(item);
+29 -24
View File
@@ -115,10 +115,10 @@ restart:
*fdRootsSocket = createUnixDomainSocket();
try {
nix::connect(toSocket(fdRootsSocket->get()), socketPath);
} catch (SysError & e) {
} catch (SystemError & e) {
/* The garbage collector may have exited or not
created the socket yet, so we need to restart. */
if (e.errNo == ECONNREFUSED || e.errNo == ENOENT) {
if (e.is(std::errc::connection_refused) || e.is(std::errc::no_such_file_or_directory)) {
debug("GC socket connection refused: %s", e.msg());
fdRootsSocket->close();
std::this_thread::sleep_for(std::chrono::milliseconds(100));
@@ -135,10 +135,10 @@ restart:
readFull(fdRootsSocket->get(), &c, 1);
assert(c == '1');
debug("got ack for GC root '%s'", printStorePath(path));
} catch (SysError & e) {
} catch (SystemError & e) {
/* The garbage collector may have exited, so we need to
restart. */
if (e.errNo == EPIPE || e.errNo == ECONNRESET) {
if (e.is(std::errc::broken_pipe) || e.is(std::errc::connection_reset)) {
debug("GC socket disconnected");
fdRootsSocket->close();
goto restart;
@@ -280,9 +280,10 @@ void LocalStore::findRoots(const Path & path, std::filesystem::file_type type, R
throw;
}
catch (SysError & e) {
catch (SystemError & e) {
/* We only ignore permanent failures. */
if (e.errNo == EACCES || e.errNo == ENOENT || e.errNo == ENOTDIR)
if (e.is(std::errc::permission_denied) || e.is(std::errc::no_such_file_or_directory)
|| e.is(std::errc::not_a_directory))
printInfo("cannot read potential root '%1%'", path);
else
throw;
@@ -347,8 +348,8 @@ static void readFileRoots(const std::filesystem::path & path, UncheckedRoots & r
{
try {
roots[readFile(path)].emplace(path.string());
} catch (SysError & e) {
if (e.errNo != ENOENT && e.errNo != EACCES)
} catch (SystemError & e) {
if (!e.is(std::errc::no_such_file_or_directory) && !e.is(std::errc::permission_denied))
throw;
}
}
@@ -464,9 +465,11 @@ struct GCLimitReached
void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
{
const auto & gcSettings = config->getGCSettings();
bool shouldDelete = options.action == GCOptions::gcDeleteDead || options.action == GCOptions::gcDeleteSpecific;
bool gcKeepOutputs = settings.gcKeepOutputs;
bool gcKeepDerivations = settings.gcKeepDerivations;
bool keepOutputs = gcSettings.keepOutputs;
bool keepDerivations = gcSettings.keepDerivations;
boost::unordered_flat_set<StorePath, std::hash<StorePath>> roots, dead, alive;
@@ -490,8 +493,8 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
(the garbage collector will recurse into deleting the outputs
or derivers, respectively). So disable them. */
if (options.action == GCOptions::gcDeleteSpecific && options.ignoreLiveness) {
gcKeepOutputs = false;
gcKeepDerivations = false;
keepOutputs = false;
keepDerivations = false;
}
if (shouldDelete)
@@ -648,7 +651,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
/* Helper function that deletes a path from the store and throws
GCLimitReached if we've deleted enough garbage. */
auto deleteFromStore = [&](std::string_view baseName) {
auto deleteFromStore = [&](std::string_view baseName, bool isKnownPath) {
Path path = storeDir + "/" + std::string(baseName);
Path realPath = config->realStoreDir + "/" + std::string(baseName);
@@ -668,7 +671,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
results.paths.insert(path);
uint64_t bytesFreed;
deleteStorePath(realPath, bytesFreed);
deleteStorePath(realPath, bytesFreed, isKnownPath);
results.bytesFreed += bytesFreed;
@@ -727,8 +730,8 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
*path,
closure,
/* flipDirection */ false,
gcKeepOutputs,
gcKeepDerivations);
keepOutputs,
keepDerivations);
for (auto & p : closure)
alive.insert(p);
} catch (InvalidPath &) {
@@ -769,7 +772,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
/* If keep-derivations is set and this is a
derivation, then visit the derivation outputs. */
if (gcKeepDerivations && path->isDerivation()) {
if (keepDerivations && path->isDerivation()) {
for (auto & [name, maybeOutPath] : queryPartialDerivationOutputMap(*path))
if (maybeOutPath && isValidPath(*maybeOutPath)
&& queryPathInfo(*maybeOutPath)->deriver == *path)
@@ -777,7 +780,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
}
/* If keep-outputs is set, then visit the derivers. */
if (gcKeepOutputs) {
if (keepOutputs) {
auto derivers = queryValidDerivers(*path);
for (auto & i : derivers)
enqueue(i);
@@ -790,7 +793,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
if (shouldDelete) {
try {
invalidatePathChecked(path);
deleteFromStore(path.to_string());
deleteFromStore(path.to_string(), true);
referrersCache.erase(path);
} catch (PathInUse & e) {
// If we end up here, it's likely a new occurrence
@@ -841,7 +844,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
if (auto storePath = maybeParseStorePath(storeDir + "/" + name))
deleteReferrersClosure(*storePath);
else
deleteFromStore(name);
deleteFromStore(name, false);
}
} catch (GCLimitReached & e) {
}
@@ -919,6 +922,8 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
void LocalStore::autoGC(bool sync)
{
#if HAVE_STATVFS
const auto & gcSettings = config->getGCSettings();
static auto fakeFreeSpaceFile = getEnv("_NIX_TEST_FREE_SPACE_FILE");
auto getAvail = [this]() -> uint64_t {
@@ -945,14 +950,14 @@ void LocalStore::autoGC(bool sync)
auto now = std::chrono::steady_clock::now();
if (now < state->lastGCCheck + std::chrono::seconds(settings.minFreeCheckInterval))
if (now < state->lastGCCheck + std::chrono::seconds(gcSettings.minFreeCheckInterval))
return;
auto avail = getAvail();
state->lastGCCheck = now;
if (avail >= settings.minFree || avail >= settings.maxFree)
if (avail >= gcSettings.minFree || avail >= gcSettings.maxFree)
return;
if (avail > state->availAfterGC * 0.97)
@@ -963,7 +968,7 @@ void LocalStore::autoGC(bool sync)
std::promise<void> promise;
future = state->gcFuture = promise.get_future().share();
std::thread([promise{std::move(promise)}, this, avail, getAvail]() mutable {
std::thread([promise{std::move(promise)}, this, avail, getAvail, &gcSettings]() mutable {
try {
/* Wake up any threads waiting for the auto-GC to finish. */
@@ -975,7 +980,7 @@ void LocalStore::autoGC(bool sync)
});
GCOptions options;
options.maxFreed = settings.maxFree - avail;
options.maxFreed = gcSettings.maxFree - avail;
printInfo("running auto-GC to free %d bytes", options.maxFreed);
+32 -5
View File
@@ -35,6 +35,10 @@
# include <sys/sysctl.h>
#endif
#ifdef _WIN32
# include "nix/util/windows-known-folders.hh"
#endif
#include "store-config-private.hh"
namespace nix {
@@ -46,23 +50,46 @@ namespace nix {
must be deleted and recreated on startup.) */
#define DEFAULT_SOCKET_PATH "/daemon-socket/socket"
/**
* Helper to resolve the NIX_CONF_DIR at runtime on Windows.
* On Windows, NIX_CONF_DIR is not defined at compile time, so we determine
* the path at runtime using the Windows known folders API (FOLDERID_ProgramData).
* This allows Nix to work correctly regardless of which drive Windows is installed on.
*/
static std::filesystem::path resolveNixConfDir()
{
#ifdef _WIN32
# ifdef NIX_CONF_DIR
// On Windows, NIX_CONF_DIR should not be defined at compile time
# error "NIX_CONF_DIR should not be defined on Windows"
# endif
return windows::known_folders::getProgramData() / "nix";
#else
return NIX_CONF_DIR;
#endif
}
LogFileSettings::LogFileSettings()
: nixLogDir(canonPath(getEnvNonEmpty("NIX_LOG_DIR").value_or(NIX_LOG_DIR)))
{
}
Settings settings;
static GlobalConfig::Register rSettings(&settings);
Settings::Settings()
: nixPrefix(NIX_PREFIX)
, nixStore(
: nixStore(
#ifndef _WIN32
// On Windows `/nix/store` is not a canonical path, but we dont'
// want to deal with that yet.
canonPath
#endif
(getEnvNonEmpty("NIX_STORE_DIR").value_or(getEnvNonEmpty("NIX_STORE").value_or(NIX_STORE_DIR))))
, nixDataDir(canonPath(getEnvNonEmpty("NIX_DATA_DIR").value_or(NIX_DATA_DIR)))
, nixLogDir(canonPath(getEnvNonEmpty("NIX_LOG_DIR").value_or(NIX_LOG_DIR)))
, nixStateDir(canonPath(getEnvNonEmpty("NIX_STATE_DIR").value_or(NIX_STATE_DIR)))
, nixConfDir(canonPath(getEnvNonEmpty("NIX_CONF_DIR").value_or(NIX_CONF_DIR)))
, nixConfDir(canonPath(getEnvOsNonEmpty(OS_STR("NIX_CONF_DIR"))
.transform([](auto && s) { return std::filesystem::path(s); })
.value_or(resolveNixConfDir())))
, nixUserConfFiles(getUserConfigFiles())
, nixDaemonSocketFile(
canonPath(getEnvNonEmpty("NIX_DAEMON_SOCKET_PATH").value_or(nixStateDir + DEFAULT_SOCKET_PATH)))
+24 -6
View File
@@ -78,13 +78,13 @@ void HttpBinaryCacheStore::init()
}
}
std::optional<std::string> HttpBinaryCacheStore::getCompressionMethod(const std::string & path)
std::optional<CompressionAlgo> HttpBinaryCacheStore::getCompressionMethod(const std::string & path)
{
if (hasSuffix(path, ".narinfo") && !config->narinfoCompression.get().empty())
if (hasSuffix(path, ".narinfo") && config->narinfoCompression.get())
return config->narinfoCompression;
else if (hasSuffix(path, ".ls") && !config->lsCompression.get().empty())
else if (hasSuffix(path, ".ls") && config->lsCompression.get())
return config->lsCompression;
else if (hasPrefix(path, "log/") && !config->logCompression.get().empty())
else if (hasPrefix(path, "log/") && config->logCompression.get())
return config->logCompression;
else
return std::nullopt;
@@ -160,7 +160,9 @@ void HttpBinaryCacheStore::upsertFile(
try {
if (auto compressionMethod = getCompressionMethod(path)) {
CompressedSource compressed(source, *compressionMethod);
Headers headers = {{"Content-Encoding", *compressionMethod}};
/* TODO: Validate that this is a valid content encoding. We probably shouldn't set non-standard values here.
*/
Headers headers = {{"Content-Encoding", showCompressionAlgo(*compressionMethod)}};
upload(path, compressed, compressed.size(), mimeType, std::move(headers));
} else {
upload(path, source, sizeHint, mimeType, std::nullopt);
@@ -193,7 +195,23 @@ FileTransferRequest HttpBinaryCacheStore::makeRequest(std::string_view path)
result.query = config->cacheUri.query;
}
return FileTransferRequest(result);
FileTransferRequest request(result);
/* Only use the specified SSL certificate and private key if the resolved URL names the same
authority and uses the same protocol. */
if (result.scheme == config->cacheUri.scheme && result.authority == config->cacheUri.authority) {
if (const auto & cert = config->tlsCert.get()) {
debug("using TLS client certificate %s for '%s'", PathFmt(*cert), request.uri);
request.tlsCert = *cert;
}
if (const auto & key = config->tlsKey.get()) {
debug("using TLS client key '%s' for '%s'", PathFmt(*key), request.uri);
request.tlsKey = *key;
}
}
return request;
}
void HttpBinaryCacheStore::getFile(const std::string & path, Sink & sink)
@@ -1,7 +1,7 @@
#pragma once
///@file
#include "nix/util/signature/local-keys.hh"
#include "nix/util/compression-settings.hh"
#include "nix/store/store-api.hh"
#include "nix/store/log-store.hh"
@@ -18,8 +18,14 @@ struct BinaryCacheStoreConfig : virtual StoreConfig
{
using StoreConfig::StoreConfig;
const Setting<std::string> compression{
this, "xz", "compression", "NAR compression method (`xz`, `bzip2`, `gzip`, `zstd`, or `none`)."};
const Setting<CompressionAlgo> compression{
this,
CompressionAlgo::xz,
"compression",
R"(
NAR compression method. One of: `xz`, `bzip2`, `gzip`, `zstd`, `none`, `br`, `compress`, `grzip`, `lrzip`, `lz4`, `lzip`, `lzma` or `lzop`.
To use a particular compression method Nix has to be built with a version of libarchive that natively supports that compression algorithm.
)"};
const Setting<bool> writeNARListing{
this, false, "write-nar-listing", "Whether to write a JSON file that lists the files in each NAR."};
@@ -205,7 +211,7 @@ public:
std::shared_ptr<SourceAccessor> getFSAccessor(const StorePath &, bool requireValidPath = true) override;
void addSignatures(const StorePath & storePath, const StringSet & sigs) override;
void addSignatures(const StorePath & storePath, const std::set<Signature> & sigs) override;
std::optional<std::string> getBuildLogExact(const StorePath & path) override;
@@ -95,10 +95,7 @@ struct BuildResult
bool operator==(const BuildResult::Failure &) const noexcept;
std::strong_ordering operator<=>(const BuildResult::Failure &) const noexcept;
[[noreturn]] void rethrow() const
{
throw Error("%s", errorMsg);
}
[[noreturn]] void rethrow() const;
};
std::variant<Success, Failure> inner = Failure{};
Loaded 100 of 263 files, more files were not shown because too many files have changed in this diff. Show more