Compare commits

...
Author SHA1 Message Date
Sergei Zimmerman b9ecd329ae treewide: Replace a lot of std::function with recursive lambdas via 'deducing this'
This is much simpler and has less overhead than a std::function, which does
type erasure.
2025-10-17 02:58:10 +03:00
Farid Zakaria 64c55961eb Merge pull request #14273 from fzakaria/fzakaria/issue-13944
Make `nix nar [cat|ls]` lazy
2025-10-17 02:16:54 +03:00
John Ericson 27767a6094 Merge pull request #14276 from NixOS/fix-14193
libstore/registerOutputs: Don't try to optimize a non-existent actual…
2025-10-16 21:06:43 +00:00
Sergei Zimmerman 4cbcaad435 libstore/registerOutputs: Don't try to optimize a non-existent actualPath
Since 3c610df550 this resulted in `getting status of`
errors on paths inside the chroot if a path was already valid. Careful inspection
of the logic shows that if buildMode != bmCheck actualPath gets reassigned to
store.toRealPath(finalDestPath). The only branch that cares about actualPath is
the buildMode == bmCheck case, which doesn't lead to optimisePath anyway.
2025-10-16 23:08:30 +03:00
John Ericson d87a06af7a Merge pull request #14275 from NixOS/s3-cleanup
libstore: Miscellaneous S3 store cleanups
2025-10-16 19:36:59 +00:00
Eelco Dolstra 2dc9f2a2b7 Merge pull request #14272 from NixOS/use-store-path-serializer
Daemon protocol: Use the WorkerProto serializer for store paths
2025-10-16 19:35:25 +00:00
Eelco Dolstra a7991d55cc Merge pull request #14270 from NixOS/use-optional-storepath-serializer
Use serializer for std::optional<StorePath>
2025-10-16 19:07:07 +00:00
Sergei Zimmerman e7047fde25 libstore: Remove the unnecessary 'error: ' prefix in warning message 2025-10-16 21:49:38 +03:00
Sergei Zimmerman 33e94fe19f libstore: Make AwsAuthError more legible
Instead of the cryptic:

> error: Failed to resolve AWS credentials: error code 6153`

We now get more legible:

> error: AWS authentication error: 'Valid credentials could not be sourced by the IMDS provider' (6153)
2025-10-16 21:49:37 +03:00
Sergei Zimmerman dc03c6a812 libstore: Put all the AWS credentials logic behind interface class AwsCredentialProvider
This makes it so we don't need to rely on global variables and hacky destructors to
clean up another global variable. Just putting it in the correct order in the class
is more than enough.
2025-10-16 21:49:36 +03:00
Sergei Zimmerman b1d067c9bb tests/nixos: Rename back S3 store nixos test 2025-10-16 21:49:35 +03:00
Eelco Dolstra d782c5e586 Daemon protocol: Use the WorkerProto serializer for store paths 2025-10-16 17:34:33 +02:00
Eelco Dolstra f84b33644c Merge pull request #14271 from NixOS/no-check-sigs
Factor out `--no-check-sigs` into its own class
2025-10-16 15:07:29 +00:00
Eelco Dolstra 3bd2b76f6e nix store sign: Use required attribute 2025-10-16 16:35:13 +02:00
Eelco Dolstra 139df77440 Factor out --no-check-sigs 2025-10-16 16:35:09 +02:00
Eelco Dolstra a48a737517 Use serializer for std::optional<StorePath> 2025-10-16 16:32:18 +02:00
John Ericson 0503a862ef Merge pull request #14268 from roberth/dev-doc-manual
doc/dev/doc: Update local build instructions for manual
2025-10-16 13:50:58 +00:00
Robert Hensing 61cb9c4832 doc/dev/doc: Update local build instructions for manual 2025-10-16 13:22:22 +02:00
John Ericson 721f5572a6 Merge pull request #14263 from NixOS/hydra-import-paths
Restore `ServeProto::Command::ImportPaths`
2025-10-15 22:57:50 +00:00
John Ericson 5a6864c027 Merge pull request #14264 from xokdvium/fix-splicing-test
tests: Fix splicing in functional tests for nix-cli
2025-10-15 22:32:45 +00:00
Sergei Zimmerman 0deb492b3d Restore ServeProto::Command::ImportPaths
This partially reverts commit 5e46df973f,
partially reversing changes made to
8c789db05b.

We do this because Hydra, while using the newer version of the protocol,
still uses this command, even though Nix (as a client) doesn't use it.
On that basis, we don't want to remove it (or consider it only part of
the older versions of the protocol) until Hydra no longer uses the
Legacy SSH Protocol.
2025-10-15 18:18:59 -04:00
Sergei Zimmerman 17b7fb383f tests: Fix splicing in functional tests for nix-cli
This is necessary to fix nix-everything-llvm.
The problem here is that nix-cli is taken from the previous
stage that is built with libstdc++, but this derivation builds
plugins with libc++ and the plugin load fails miserably.
2025-10-16 01:04:50 +03:00
John Ericson 94cfba7e84 Merge pull request #14226 from obsidiansystems/unkeyed-realisation
Reapply #14097
2025-10-15 21:27:13 +00:00
Sergei Zimmerman 0f1cfa4d60 Merge pull request #14262 from lovesegfault/ci-cleanup-s3
ci: cleanup s3 tests
2025-10-15 21:19:20 +00:00
Bernardo Meurer Costa fa0d00e668 ci: cleanup s3 tests
This cleans up the work done in 8c2828387. Now that #13752 has landed,
there's no need to test configurations without AWS auth in CI.
2025-10-15 23:51:08 +03:00
Sergei Zimmerman d2b6499154 Merge pull request #13752 from lovesegfault/curl-based-s3-v2
feat(libstore): curl-based s3
2025-10-15 20:01:26 +00:00
Sergei Zimmerman e3232af558 Merge pull request #14253 from NixOS/libgit2-refname-wa
libfetchers/git-utils: Be more correct about validating refnames
2025-10-15 19:30:53 +00:00
Bernardo Meurer Costa e069c9892e docs(rl-next): add notes for curl-based s3 2025-10-15 19:26:53 +00:00
John Ericson 266fbebe66 Implement realisation operations on dummy store 2025-10-15 14:59:08 -04:00
John EricsonandSergei Zimmerman 6995d325ef Split out UnkeyedRealisation from Realisation
Realisations are conceptually key-value pairs, mapping `DrvOutputs` (the
key) to information about that derivation output.

This separate the value type, which will be useful in maps, etc., where
we don't want to denormalize by including the key twice.

This matches similar changes for existing types:

| keyed              | unkeyed                |
|--------------------|------------------------|
| `ValidPathInfo`    | `UnkeyedValidPathInfo` |
| `KeyedBuildResult` | `BuildResult`          |
| `Realisation`      | `UnkeyedRealisation`   |

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-15 14:59:04 -04:00
Sergei Zimmerman 5d1178b817 libfetchers/git-utils: Be more correct about validating refnames
Turns out there's a much better API for this that doesn't have the
footguns of the previous method.

isLegalRefName is somewhat of a misnomer, since it's mainly used to
validate user inputs that can be either references, branch names,
psedorefs or tags.
2025-10-15 21:54:09 +03:00
Bernardo Meurer Costa 3224636ab0 refactor(libstore): rename NIX_WITH_S3_SUPPORT to NIX_WITH_AWS_AUTH
The macro now accurately reflects its purpose: gating only AWS
authentication code, not all S3 functionality. S3 URL parsing, store
configuration, and public bucket access work regardless of this flag.

This rename clarifies that:
- S3 support is always available (URL parsing, store registration)
- Only AWS credential resolution requires the flag
- The flag controls AWS CRT SDK dependency, not S3 protocol support
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa bb1f22a8df refactor(libstore): minimize NIX_WITH_S3_SUPPORT scope to auth only
Move S3 URL parsing, store configuration, and public bucket support
outside of NIX_WITH_S3_SUPPORT guards. Only AWS credential resolution
remains gated, allowing builds with withAWS = false to:

- Parse s3:// URLs
- Register S3 store types
- Access public S3 buckets (via HTTPS conversion)
- Use S3-compatible services without authentication

The setupForS3() function now always performs URL conversion, with
authentication code conditionally compiled based on NIX_WITH_S3_SUPPORT.
The aws-creds.cc file (only code using AWS CRT SDK) is now conditionally
compiled by meson.
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa 1f710300c9 refactor(libstore): withCurlS3 -> withAWS
Now that the legacy S3 implementation is gone, we can go back to calling
things `NIX_WITH_S3_SUPPORT`.
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa 9295c14a35 refactor(libstore): replace AWS SDK with curl-based S3 implementation
This commit replaces the AWS C++ SDK with a lighter curl-based approach
for S3 binary cache operations.

- Removed dependency on the heavy aws-cpp-sdk-s3 and aws-cpp-sdk-transfer
- Added lightweight aws-crt-cpp for credential resolution only
- Leverages curl's native AWS SigV4 authentication (requires curl >= 7.75.0)
- S3BinaryCacheStore now delegates to HttpBinaryCacheStore
- Function s3ToHttpsUrl converts ParsedS3URL to ParsedURL
- Multipart uploads are no longer supported (may be reimplemented later)
- Build now requires curl >= 7.75.0 for AWS SigV4 support

Fixes: #13084, #12671, #11748, #12403, #5947
2025-10-15 18:23:55 +00:00
John Ericson a543519ca9 Merge pull request #14257 from obsidiansystems/misc-builder-tech-debt
Cleanup misc builder tech debt
2025-10-15 17:28:27 +00:00
John Ericson 632ccfb8c0 Remove dead outputPaths variable. 2025-10-15 12:16:53 -04:00
John Ericson 46357468a4 Remove unused parameters to DrvOutputSubstitutionGoal 2025-10-15 12:16:52 -04:00
John Ericson b20cebf993 Remove unused typedef and field 2025-10-15 12:15:23 -04:00
Eelco Dolstra 5700112127 Merge pull request #14205 from GrahamDennis/gdennis/improve-dir-url-backcompat
Improved backwards compatibility hack for git URLs using dir=...
2025-10-15 15:20:02 +00:00
John Ericson 606eb1dfb5 Merge pull request #14250 from fzakaria/patch-1
Remove duplicate shellcheck in dev-shell.nix
2025-10-15 05:03:19 +00:00
John Ericson e07754d888 Merge pull request #14251 from fzakaria/fzakaria/iwyu-libflake
Clean-up libflake headers
2025-10-15 04:27:07 +00:00
Farid Zakaria 01a8499d2f Format cpp files 2025-10-14 23:51:40 -04:00
Farid Zakaria e8b126fa90 Remove unecessary includes 2025-10-14 23:48:45 -04:00
Farid Zakaria 902faf4fe5 More fixes for iwyu 2025-10-14 23:20:35 -04:00
Farid Zakaria 7bc3d9b9a9 First attempt at uwyu for libflake 2025-10-14 22:53:13 -04:00
Farid Zakaria 092639709f Remove duplicate shellcheck in dev-shell.nix 2025-10-14 19:25:06 -07:00
John Ericson 620091bc8b Merge pull request #14223 from lovesegfault/curl-based-s3-tests
test(nixos): add comprehensive curl-based S3 VM tests
2025-10-14 23:08:55 +00:00
John Ericson 6dcc468253 Merge pull request #14249 from NixOS/more-to-real-path-cleanups
More toRealPath cleanups
2025-10-14 22:46:15 +00:00
Sergei Zimmerman 0347958dd2 nix/develop: Remove usage of toRealPath, replace with SourceAccessor 2025-10-15 00:52:13 +03:00
Sergei Zimmerman 918a3cebaa libexpr: Use Store::requireStoreObjectAccessor instead or toRealPath in fetch
This forces the code to go through proper abstractions instead of the raw filesystem
API.

This issue is evident from this reproducer:

nix eval --expr 'builtins.fetchurl { url = "https://example.com"; sha256 = ""; }' --json --eval-store "dummy://?read-only=false"

error:
       … while calling the 'fetchurl' builtin
         at «string»:1:1:
            1| builtins.fetchurl { url = "https://example.com"; sha256 = ""; }
             | ^

       error: opening file '/nix/store/r4f87yrl98f2m6v9z8ai2rbg4qwlcakq-example.com': No such file or directory
2025-10-15 00:27:41 +03:00
Sergei Zimmerman 69c005e805 libstore: Use getFSAccessor for store object in Worker::pathContentsGood
We only care about the accessor for a single store object anyway, but
the validity gets ignored. Also `pathExists(store.printStorePath(path))`
is definitely incorrect since it confuses the logical location vs physical
location in case of a chroot store.
2025-10-15 00:15:50 +03:00
Sergei Zimmerman 0c32fb3fa2 treewide: Add Store::requireStoreObjectAccessor, simplify uses of getFSAccessor
This is a simple wrapper around getFSAccessor that throws an InvalidPath
error. This simplifies usage in callsites that only care about getting
a non-null accessor.
2025-10-14 23:58:20 +03:00
Bernardo Meurer Costa d18f959d4f test(nixos): add comprehensive curl-based S3 VM tests
Add `curl-s3-binary-cache-store.nix` with comprehensive test coverage
for the curl-based S3 implementation.

Depends-On: #14206, #14222
2025-10-14 20:55:14 +00:00
Sergei Zimmerman 4041bfdb40 Merge pull request #14206 from lovesegfault/curl-based-s3-pieces
feat(libstore): add builtin fetchurl S3 credential pre-resolution
2025-10-14 20:10:41 +00:00
John Ericson 1fb4ff8c0e Merge pull request #14232 from roberth/dyndrv-messages
Better dyndrv messages
2025-10-14 15:40:27 +00:00
Robert Hensing 1b96a704d3 Add lazy evaluation for experimental feature reasons
Wrap fmt() calls in lambdas to defer string formatting until the
feature check fails. This avoids unnecessary string formatting in
the common case where the feature is enabled.

Addresses performance concern raised by xokdvium in PR review.
2025-10-14 16:49:59 +02:00
John Ericson 959c244a12 Merge pull request #14243 from NixOS/canon-path-nul-bytes
libutil: Ensure that CanonPath does not contain NUL bytes
2025-10-14 14:30:24 +00:00
Eelco Dolstra c44d2d5913 Merge pull request #14241 from NixOS/dependabot/github_actions/actions/create-github-app-token-2
build(deps): bump actions/create-github-app-token from 1 to 2
2025-10-14 11:55:43 +00:00
Eelco Dolstra dd590eca74 Merge pull request #14242 from NixOS/dependabot/github_actions/actions/checkout-5
build(deps): bump actions/checkout from 4 to 5
2025-10-14 11:55:25 +00:00
Sergei Zimmerman 1633ceaff2 libutil: Ensure that CanonPath does not contain NUL bytes
This, alongside the other invariants of the CanonPath is important
to uphold. std::filesystem happily crashes on NUL bytes in the constructor,
as we've seen with `path:%00` prior to c436b7a32a.
Best to stay clear of NUL bytes when we're talking about syscalls, especially
on Unix where strings are null terminated.

Very nice to have if we decide to switch over to pascal-style strings.
2025-10-14 02:33:42 +03:00
John Ericson 16e946bfb1 Merge pull request #14225 from obsidiansystems/derivation-resolution-goal-2
Reapply the rest of #14022
2025-10-13 23:26:29 +00:00
Sergei Zimmerman edf9163c22 libutil: Make CanonPath::root const
By all means CanonPath::root must be immutable. Let's enforce this with
in the code.
2025-10-14 02:24:40 +03:00
John Ericson ad893acf46 Fix ca/eval-store.sh test
The refactor in the last commit fixed the bug it was supposed to fix,
but introduced a new bug in that sometimes we tried to write a resolved
derivation to a store before all its `inputSrcs` were in that store.

The solution is to defer writing the derivation until inside
`DerivationBuildingGoal`, just before we do an actual build. At this
point, we are sure that all inputs in are the store.

This does have the side effect of meaning we don't write down the
resolved derivation in the substituting case, only the building case,
but I think that is actually fine. The store that actually does the
building should make a record of what it built by storing the resolved
derivation. Other stores that just substitute from that store don't
necessary want that derivation however. They can trust the substituter
to keep the record around, or baring that, they can attempt to re
resolve everything, if they need to be audited.

(cherry picked from commit c97b050a6c)
2025-10-13 18:41:59 -04:00
John Ericson 06bb1c2f93 Remove some buildMode default parameters
Force the internals to be more explicit.
2025-10-13 18:40:10 -04:00
John Ericson 2ee41976c2 Fix #13247
Resolve the derivation before creating a building goal, in a context
where we know what output(s) we want. That way we have a chance just to
download the outputs we want.

Fix #13247

(cherry picked from commit 39f6fd9b46)
2025-10-13 18:37:14 -04:00
dependabot[bot] b846f27682 build(deps): bump actions/checkout from 4 to 5
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-10-13 22:00:59 +00:00
dependabot[bot] 962862e9e0 build(deps): bump actions/create-github-app-token from 1 to 2
Bumps [actions/create-github-app-token](https://github.com/actions/create-github-app-token) from 1 to 2.
- [Release notes](https://github.com/actions/create-github-app-token/releases)
- [Commits](https://github.com/actions/create-github-app-token/compare/v1...v2)

---
updated-dependencies:
- dependency-name: actions/create-github-app-token
  dependency-version: '2'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-10-13 22:00:55 +00:00
Robert Hensing 39c4665488 Store reason as a field in MissingExperimentalFeature
Store the reason string as a field in the exception class rather than
only embedding it in the error message. This supports better structured
error handling and future JSON error reporting.

Suggested by Ericson2314 in PR review.
2025-10-13 23:49:20 +02:00
Robert Hensing 71aa9a4798 Add reasons to dyndrv xp messages 2025-10-13 23:49:20 +02:00
Robert Hensing 0fd890a8d6 Add reason string support to MissingExperimentalFeature 2025-10-13 23:49:20 +02:00
John Ericson 6642ffb506 Merge pull request #14239 from NixOS/asan-stack-overflow
libstore/outputs-spec: Drop usage of std::regex
2025-10-13 21:44:49 +00:00
Sergei Zimmerman 3ba221025f libstore/outputs-spec: Drop usage of std::regex
std::regex is a really bad tool for parsing things, since
it tends to overflow the stack pretty badly. See the build failure
under ASan in [^].

[^]: https://hydra.nixos.org/build/310077167/nixlog/5
2025-10-13 23:58:20 +03:00
Eelco Dolstra b56cc1808d Merge pull request #14237 from NixOS/url-parser-regression
Remove validation of URLs passed to FileTransferRequest verbatim
2025-10-13 20:01:01 +00:00
Sergei Zimmerman 47f427a172 Remove validation of URLs passed to FileTransferRequest verbatim
CURL is not very strict about validation of URLs passed to it. We
should reflect this in our handling of URLs that we get from the user
in <nix/fetchurl.nix> or builtins.fetchurl. ValidURL was an attempt to
rectify this, but it turned out to be too strict. The only good way to
resolve this is to pass (in some cases) the user-provided string verbatim
to CURL. Other usages in libfetchers still benefit from using structured
ParsedURL and validation though.

nix store prefetch-file --name foo 'https://cdn.skypack.dev/big.js@^5.2.2'
error: 'https://cdn.skypack.dev/big.js@^5.2.2' is not a valid URL: leftover
2025-10-13 22:23:26 +03:00
John Ericson 0f85ef3677 Merge pull request #14219 from lovesegfault/eval-copy-less
libstore: Avoid copying derivations to the store if they are already valid
2025-10-13 16:36:40 +00:00
Eelco Dolstra be2c9ef44c Merge pull request #14229 from NixOS/reduce-hydra-load
packaging/hydra: buildNoGC is the same as buildWithSanitizers
2025-10-13 16:22:30 +00:00
John Ericson d2c0c0607c Merge branch 'master' into eval-copy-less 2025-10-13 11:52:42 -04:00
John Ericson 480ce19011 Merge pull request #14233 from neuralsorcerer/fix-typo
Fix typos
2025-10-13 15:30:05 +00:00
John Ericson 3f876bcb61 Merge pull request #14231 from roberth/code-docs
Code docs
2025-10-13 15:05:30 +00:00
Soumyadip Sarkar 998f93f267 Fix typos 2025-10-13 18:15:52 +05:30
Robert Hensing 583f5e37fc Refactor: use optionalBracket in nix search 2025-10-13 14:02:59 +02:00
Robert Hensing 5dcfddf997 strings: Add optionalBracket helper 2025-10-13 13:59:39 +02:00
Robert Hensing 48a5e2dde2 EvalState: add doc comment 2025-10-13 13:14:05 +02:00
Robert Hensing 6db86389ce util/error: Document addTrace params
... and rename e -> pos. That was weird.
2025-10-13 12:57:22 +02:00
Bernardo Meurer Costa 000e6f6282 feat(libstore): add builtin fetchurl S3 credential pre-resolution
Add support for pre-resolving AWS credentials in the parent process
before forking for builtin:fetchurl. This avoids recreating credential
providers in the forked child process.
2025-10-12 23:01:13 +00:00
Bernardo Meurer CostaandSergei Zimmerman 18ec3d1094 libstore: Avoid copying derivations to the store if they are already valid
This avoids the quite costly copying of derivations to the daemon over the
wire in case it already exists in the eval store.

For a fresh instantiatation (after running nix-collect-garbage) this doesn't
significantly slow down eval:

taskset -c 2,3 hyperfine --reference "result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket" --prepare "nix-collect-garbage --store /tmp/store1111 --no-keep-derivations" "result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket"
Benchmark 1: result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     388.7 ms ±  10.5 ms    [User: 157.0 ms, System: 61.3 ms]
  Range (min … max):   379.4 ms … 415.9 ms    10 runs

Benchmark 2: result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     389.2 ms ±   4.8 ms    [User: 158.5 ms, System: 60.7 ms]
  Range (min … max):   381.2 ms … 397.6 ms    10 runs

But if the derivations are already instantiated this shows a pretty neat speedup:

taskset -c 2,3 hyperfine --reference "result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket" "result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket"
Benchmark 1: result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     240.4 ms ±   3.1 ms    [User: 148.1 ms, System: 57.0 ms]
  Range (min … max):   233.8 ms … 245.0 ms    12 runs

Benchmark 2: result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     226.5 ms ±   4.5 ms    [User: 147.8 ms, System: 55.2 ms]
  Range (min … max):   214.0 ms … 231.2 ms    13 runs

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-13 01:59:38 +03:00
John Ericson f77094715f Merge pull request #14222 from lovesegfault/curl-based-s3-fix-race
fix(libstore): fix race condition in AWS credential provider caching
2025-10-12 22:07:10 +00:00
Bernardo Meurer Costa f0e1f65260 fix(libstore): fix race condition in AWS credential provider caching
The previous implementation had a check-then-create race condition where
multiple threads could simultaneously:
1. Check the cache and find no provider (line 122)
2. Create their own providers (lines 126-145)
3. Insert into cache (line 161)

This resulted in multiple credential providers being created when
downloading multiple packages in parallel, as each .narinfo download
would trigger provider creation on its own thread.

Fix by using boost::concurrent_flat_map's try_emplace_and_cvisit, which
provides atomic get-or-create semantics:
- f1 callback: Called atomically during insertion, creates the provider
- f2 callback: Called if key exists, returns cached provider
- Other threads are blocked during f1, so no nullptr is ever visible
2025-10-12 20:16:02 +00:00
Sergei Zimmerman 89b35ec0dc packaging/hydra: buildNoGC is the same as buildWithSanitizers
This will reduce the load on hydra. It doesn't make sense to
build 2 slightly different variations where the difference
is only in the nix-perl-bindings and additional sanitizers.
2025-10-12 22:10:35 +03:00
Sergei Zimmerman 4f585dedbe Merge pull request #14228 from obsidiansystems/fix-windows-build
Fix windows build
2025-10-12 18:07:50 +00:00
John Ericson a01df8de21 Merge pull request #14227 from NixOS/asan-hydra
packaging: Add buildWithSanitizers to hydraJobs
2025-10-12 17:43:54 +00:00
John Ericson 10223fae86 Fix windows build
I forget to add some CPP in b57caaa1a2.

Hopefully, as we relyon RAII more, these explicit resets become
unneeded.
2025-10-12 13:22:35 -04:00
John Ericson 9150ccb89e Fix Windows dev shell (mostly)
gbenchmark still has too-narrow supported systems, however. That needs
to be fixed in Nixpkgs.
2025-10-12 13:16:50 -04:00
Sergei Zimmerman de75a180cb packaging: Add buildWithSanitizers to hydraJobs 2025-10-12 19:38:01 +03:00
Sergei Zimmerman a491173369 packaging: Add withASan,withUBSan options to the scope 2025-10-12 19:16:08 +03:00
Sergei Zimmerman 199b6ff3fb Disable detect_odr_violation for ASan
There's some unfortunate ODR violations that get dianosed with GCC but not Clang
for static inline constexpr variables defined inside the class body:

template<typename T>
struct static_const
{
    static JSON_INLINE_VARIABLE constexpr T value{};
};

This can be ignored pretty much. There is the same problem for std::piecewise_construct:

http://lists.boost.org/Archives/boost/2007/06/123353.php

==2455704==ERROR: AddressSanitizer: odr-violation (0x7efddc460e20):
  [1] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/pkmljfq97a83dbanr0n64zbm8cyhna33-nix-store-2.33.0pre/lib/libnixstore.so.2.33.0
  [2] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/gbjpkjj0g8vk20fzlyrwj491gwp6g1qw-nix-util-2.33.0pre/lib/libnixutil.so.2.33.0
2025-10-12 19:16:07 +03:00
Sergei Zimmerman 711e738bf9 meson: Simplify asan-options handling even more
Instead of specifying env variables all the time
we can instead embed the __asan_default_options symbol
in all executables / shared objects. This reduces code
duplication.
2025-10-12 19:16:06 +03:00
Sergei Zimmerman d9cabddd17 Merge pull request #14214 from obsidiansystems/derivation-resolution-goal
Split out `DerivationResolutionGoal`
2025-10-12 11:31:54 +00:00
John Ericson 0da430be35 Split out DerivationResolutionGoal
This prepares the way for fixing a few issues.

Take 2: was landed before in 8f4a739d0f.
2025-10-11 19:52:37 -04:00
John Ericson 07df87652c Make keys of Derivation*Goal more legible
The property that substitution goals come first is still preserved
2025-10-11 19:50:45 -04:00
John Ericson a629ce3dec Use member initializer list for Derivation*Goal::drv 2025-10-11 18:37:04 -04:00
John Ericson ba7bbcd1da Cleanup Derivation*Goal names 2025-10-11 18:30:47 -04:00
Sergei Zimmerman 97e770ad01 Merge pull request #14209 from lovesegfault/curl-based-s3-pieces-fix
fix(libstore): improve http-binary-cache-store S3 compatibility
2025-10-11 21:19:18 +00:00
John Ericson 682cf0b266 Merge pull request #14216 from NixOS/simplify-asan-meson
meson: Move asan-options to common
2025-10-11 18:21:24 +00:00
John Ericson f1cb837888 Merge pull request #14217 from NixOS/remove-libgit2-patches
packaging: Remove no longer necessary libgit2 patches
2025-10-11 18:20:36 +00:00
Sergei Zimmerman 47705139c9 packaging: Remove no longer necessary libgit2 patches
25.05 already has 1.9.0 and we don't support older nixpkgs versions.
2025-10-11 16:30:55 +03:00
Sergei Zimmerman d26a337c09 meson: Move asan-options to common
This way we don't have to duplicate the subdir everywhere.
Less copy-pasta is good.
2025-10-11 16:08:35 +03:00
John Ericson 8064c75694 Merge pull request #14194 from xokdvium/stacktrace-assertion
libutil: Print stack trace on assertion failure
2025-10-10 23:25:16 +00:00
John Ericson d75614a315 Merge pull request #14208 from obsidiansystems/consolidate-builder-dispatch
Consolidate logic choosing where we can/should build a bit
2025-10-10 23:05:40 +00:00
Sergei Zimmerman 46382ade74 libutil: Print stack trace on assertion failure
This change overrides __assert_fail on glibc/musl
to instead call std::terminate that we have a custom
handler for. This ensures that we have more context
to diagnose issues encountered by users in the wild.
2025-10-11 01:35:58 +03:00
John Ericson 9cb686f816 Merge pull request #14210 from NixOS/double-quotes
libstore: Fix double-quoting of paths in logs
2025-10-10 22:10:36 +00:00
Bernardo Meurer Costa f02218873e fix(libstore): improve http-binary-cache-store S3 compatibility
This commit adds two key fixes to http-binary-cache-store.cc to
properly support the new curl-based S3 implementation:

1. **Consistent cache key handling**: Use `getReference().render(withParams=false)`
   for disk cache keys instead of `cacheUri.to_string()`. This ensures cache
   keys are consistent with the S3 implementation and don't include query
   parameters, which matches the behavior expected by Store::queryPathInfo()
   lookups.

2. **S3 query parameter preservation**: When generating file transfer requests
   for S3 URLs, preserve query parameters from the base URL (region, endpoint,
   etc.) when the relative path doesn't have its own query parameters. This
   ensures S3-specific configuration is propagated to all requests.
2025-10-10 22:05:57 +00:00
John Ericson b57caaa1a2 Consolidate logic choosing where we can/should build a bit
I want to separate "policy" from "mechanism".

Now the logic to decide how to build (a policy choice, though with some
hard constraints) is all in derivation building goal, and all in the
same spot. build hook, external builder, or local builder --- the choice
between all three is made in the same spot --- pure policy.

Now, if you want to use the external deriation builder, you simply
provide the `ExternalBuilder` you wish to use, and there is no
additional checking --- pure mechanism. It is the responsibility of the
caller to choose an external builder that works for the derivation in
question.

Also, `checkSystem()` was the only thing throwing `BuildError` from
`startBuilder`. Now that that is gone, we can now remove the
`try...catch` around that.
2025-10-10 17:28:57 -04:00
John EricsonandSergei Zimmerman 2ff59ec3e0 Use std::ranges::find_if for finding external builders
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-10 17:27:41 -04:00
Sergei Zimmerman f30cb8667b libstore: Fix double-quoting of paths in logs
std::filesystem::path is already quoted by boost::format with double quotes (").
2025-10-11 00:25:51 +03:00
John Ericson b56dd21c31 Settings::ExternalBuilder::systems make set
Nothing cares about the order, actually.
2025-10-10 17:23:45 -04:00
Sergei Zimmerman 2308aaf192 Merge pull request #14198 from lovesegfault/curl-based-s3-pieces
feat(libstore): add curl-based S3 store implementation
2025-10-10 07:55:44 +00:00
Graham Dennis 8d9e9bc400 Improve comment 2025-10-10 15:00:10 +11:00
Graham Dennis 43b01b6790 Improved backwards compatibility hack for git URLs using dir=... attribute 2025-10-10 14:54:47 +11:00
Bernardo Meurer Costa 0855b715a9 feat(libstore): add curl-based S3 store implementation
Add a new S3BinaryCacheStore implementation that inherits from
HttpBinaryCacheStore.

The implementation is activated with NIX_WITH_CURL_S3, keeping the
existing NIX_WITH_S3_SUPPORT (AWS SDK) implementation unchanged.
2025-10-10 00:13:29 +00:00
John Ericson 2fc88ec114 Merge pull request #14204 from NixOS/kill-to-real-path-fetchers
libfetchers: Remove toRealPath in SourceHutInputScheme::getRevFromRef
2025-10-09 22:01:03 +00:00
Sergei Zimmerman 9c8480becb Merge pull request #14203 from roberth/move-eval-cache-open-to-libflake
Move eval cache open to libflake
2025-10-09 21:42:06 +00:00
Sergei Zimmerman c58acff42a libfetchers: Remove toRealPath in SourceHutInputScheme::getRevFromRef
This code had several issues:

1. Not going through the SourceAccessor means that we can only work
   with physical paths.

2. It did not actually check that the file exists. (std::ifstream does not check
   it by default).
2025-10-10 00:26:57 +03:00
Robert Hensing abcceafbce Use const for lock in openEvalCache 2025-10-09 21:25:40 +02:00
Robert Hensing 42c9cbf9ca Use ref<LockedFlake> where non-null 2025-10-09 21:25:40 +02:00
Robert Hensing 0387b7d6db Move openEvalCache to libflake
Most of the eval cache logic is flake-independent and libexpr,
but the loading part is not.
`nix-flake` is the right component for this, as the eval cache
isn't exactly specific to the command line.
2025-10-09 20:40:40 +02:00
Sergei Zimmerman dfafd8bc38 Merge pull request #14199 from getchoo-contrib/getchoo/toml11-fix
packaging: only override `toml11` when necessary
2025-10-09 17:20:35 +00:00
Sergei Zimmerman 44486871e9 Merge pull request #14200 from Mic92/less-ci-builds
only build on push to master
2025-10-09 15:07:06 +00:00
Jörg Thalheim 118acc84ba only build on push to master
we have now merge queues for maintainance branches. We still build it
for master to have our installer beeing updated. In future this part
could go in new workflow instead.
2025-10-09 14:16:14 +01:00
Seth Flynn 0f016f9bf5 packaging: only override toml11 when necessary
v4.4.0 hit Nixpkgs in https://github.com/NixOS/nixpkgs/pull/442682.
Ideally we'd just use that, but this keeps the fallback behavior until
it's more widespread
2025-10-09 03:24:57 -04:00
John Ericson ce38b46e06 Merge pull request #14170 from lovesegfault/curl-based-s3-pieces
feat(libstore/filetransfer): add S3 signing support
2025-10-08 22:35:32 +00:00
Sergei Zimmerman 33d9270109 Merge pull request #14191 from NixOS/fix-14188
libutil: Fix renderAuthorityAndPath unreachable for path:/ URLs
2025-10-08 21:34:28 +00:00
Bernardo Meurer Costa 00c2a57666 feat(libstore/filetransfer): add S3 signing support 2025-10-08 21:31:34 +00:00
Eelco Dolstra d591f17ecb Merge pull request #14189 from NixOS/fix-exportReferencesGraph
exportReferencesGraph: Handle heterogeneous arrays
2025-10-08 21:19:30 +00:00
John Ericson bb1945a090 Merge pull request #14182 from NixOS/simplify-archive-tests
tests: Move invalid nar tests from tests/functional to libutil-tests
2025-10-08 20:46:37 +00:00
Eelco Dolstra 94f410b628 exportReferencesGraph: Handle heterogeneous arrays
This barfed with

   error: [json.exception.type_error.302] type must be string, but is array

on `nix build github:malt3/bazel-env#bazel-env` because it has a `exportReferencesGraph` with a value like `["string",...["string"]]`.
2025-10-08 22:15:33 +02:00
Bernardo Meurer Costa 3c1e2e56ea feat(libstore/filetransfer): add username/password authentication support
Add a `UsernameAuth` struct and optional `usernameAuth` field to
`FileTransferRequest` to support programmatic username/password
authentication.

This uses curl's `CURLOPT_USERNAME`/`CURLOPT_PASSWORD` options, which
works with multiple protocols (HTTP, FTP, etc.) and is not specific to
any particular authentication scheme.

The primary motivation is to enable S3 authentication refactoring where
AWS credentials (access key ID and secret access key) can be passed
through this general-purpose mechanism, reducing the amount of
S3-specific code behind `#if NIX_WITH_CURL_S3` guards.
2025-10-08 20:10:53 +00:00
Eelco Dolstra 925f10d5ea Merge pull request #14087 from NixOS/required-args
Args::Flag: Add `required` attribute
2025-10-08 19:33:22 +00:00
Sergei Zimmerman 1d8dd77e1d libutil: Fix renderAuthorityAndPath unreachable for path:/ URLs
This was mistakenly triggered by path:/ URL, since the `//` would
correspond to 3 empty segments.
2025-10-08 22:14:49 +03:00
Jörg Thalheim 090f7fb05e Merge pull request #14002 from getchoo-contrib/getchoo/dogfood-experimental-installer
Add experimental installer to installer tests
2025-10-08 04:05:46 +00:00
Jörg Thalheim 2ce343716d Merge pull request #14169 from NixOS/dev-shell-no-separate-debug-info
dev-shell: Disable separateDebugInfo
2025-10-08 04:03:43 +00:00
Jörg Thalheim e69e621578 Merge pull request #14180 from NixOS/release-process-drop-mergify
maintainers: Remove mergify note from release-process.md
2025-10-08 03:34:05 +00:00
Jörg Thalheim d2a0a11a8e Merge pull request #14184 from lovesegfault/nix-better-ci
ci: integrate vm_tests into main tests job
2025-10-08 05:23:53 +02:00
Bernardo Meurer Costa a400ea4257 ci: integrate vm_tests into main tests job
This consolidates the separate vm_tests job into the main tests job,
simplifying the CI workflow. VM tests now run as part of the regular
test matrix.
2025-10-08 02:46:56 +00:00
Sergei Zimmerman fc8b784924 Merge pull request #14147 from lovesegfault/nix-multi-ci
ci: test without s3 and with curl-based-s3
2025-10-08 02:38:03 +03:00
Sergei Zimmerman 0619351326 tests: Move invalid nar tests from tests/functional to libutil-tests
Since 242f362567 we have better infrastructure
for this kind of tests.
2025-10-08 02:02:33 +03:00
Sergei Zimmerman c5b88c22fa dev-shell: Disable separateDebugInfo
This breaks gdb pretty-printers inserted into .debug_gdb_scripts section,
because it implies --compress-debug-sections=zlib, -Wa,--compress-debug-sections.
This is very unfortunate, because then gdb can't use pretty printers for
Boost.Unordered (which are very useful, since boost::unoredred_flat_map is
impossible to debug). This seems perfectly fine to disable in the dev-shell for
the time being.

See [1-3] for further references.

With this change I'm able to use boost's pretty-printers out-of-the box:

```
p *importResolutionCache
$2 = boost::concurrent_flat_map with 1 elements = {[{accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
        get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
      path = "/derivation-internal.nix"}}] = {accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
        get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
      path = "/derivation-internal.nix"}}}
```

When combined with a simple `add-auto-load-safe-path ~/code` in .gdbinit

[1]: https://gerrit.lix.systems/c/lix/+/3880
[2]: https://git.lix.systems/lix-project/lix/issues/1003
[3]: https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
2025-10-08 00:57:34 +03:00
Sergei Zimmerman 75b18a6e47 maintainers: Remove mergify note from release-process.md 2025-10-08 00:51:50 +03:00
Sergei Zimmerman fb117e0cac Merge pull request #14155 from Mic92/backport-action
ci: Switch away from mergify to backport action
2025-10-08 00:32:00 +03:00
Jörg Thalheim 63e8b5f94a ci: Switch away from mergify to backport action
We want to use github native queues.
2025-10-07 23:43:03 +03:00
Eelco Dolstra 559061bf5a Merge pull request #14173 from NixOS/bump-2.33.0
Bump version
2025-10-07 20:42:25 +02:00
Eelco Dolstra b0f567e18b Update mergify.yml 2025-10-07 17:16:57 +02:00
Eelco Dolstra 875f7ec25f Bump version 2025-10-07 17:15:28 +02:00
Bernardo Meurer Costa 8c28283876 ci: test without s3 and with curl-based-s3 2025-10-06 16:24:21 +00:00
Bernardo Meurer Costa 7f22a40e3b build(libstore): assert withAWS xor withCurlS3 2025-10-06 16:22:01 +00:00
Seth Flynn 92d7381826 ci: allow for using the latest build of the experimental installer
Until these repos are potentially merged, this is good for dogfooding
alongside the experimental installer. It also uses the more official
`artifacts.nixos.org` endpoint to install stable releases now

More immediately though, we need a patch for the experimental installer
to really work in CI at all, and that hasn't landed in a tag yet. So,
this lets us use it right from `main`!
2025-10-03 03:37:59 -04:00
Seth Flynn d2293fb458 ci: enable experimental installer tests 2025-10-03 01:26:55 -04:00
Seth Flynn 2cbbb63628 ci: enable use of the experimental installer 2025-10-03 01:26:52 -04:00
Eelco Dolstra eec4dece33 Args::Flag: Add required attribute 2025-09-26 16:01:59 +02:00
211 changed files with 3402 additions and 3727 deletions

No files matched your search

@@ -4,12 +4,22 @@ inputs:
dogfood:
description: "Whether to use Nix installed from the latest artifact from master branch"
required: true # Be explicit about the fact that we are using unreleased artifacts
experimental-installer:
description: "Whether to use the experimental installer to install Nix"
default: false
experimental-installer-version:
description: "Version of the experimental installer to use. If `latest`, the newest artifact from the default branch is used."
# TODO: This should probably be pinned to a release after https://github.com/NixOS/experimental-nix-installer/pull/49 lands in one
default: "latest"
extra_nix_config:
description: "Gets appended to `/etc/nix/nix.conf` if passed."
install_url:
description: "URL of the Nix installer"
required: false
default: "https://releases.nixos.org/nix/nix-2.30.2/install"
tarball_url:
description: "URL of the Nix tarball to use with the experimental installer"
required: false
github_token:
description: "Github token"
required: true
@@ -37,14 +47,74 @@ runs:
gh run download "$RUN_ID" --repo "$DOGFOOD_REPO" -n "$INSTALLER_ARTIFACT" -D "$INSTALLER_DOWNLOAD_DIR"
echo "installer-path=file://$INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$INSTALLER_DOWNLOAD_DIR" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
echo "::notice ::Dogfooding Nix installer from master (https://github.com/$DOGFOOD_REPO/actions/runs/$RUN_ID)"
env:
GH_TOKEN: ${{ inputs.github_token }}
DOGFOOD_REPO: "NixOS/nix"
- name: "Gather system info for experimental installer"
shell: bash
if: ${{ inputs.experimental-installer == 'true' }}
run: |
echo "::notice Using experimental installer from $EXPERIMENTAL_INSTALLER_REPO (https://github.com/$EXPERIMENTAL_INSTALLER_REPO)"
if [ "$RUNNER_OS" == "Linux" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="linux"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
elif [ "$RUNNER_OS" == "macOS" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="darwin"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_OS: $RUNNER_OS"
exit 1
fi
if [ "$RUNNER_ARCH" == "X64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=x86_64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
elif [ "$RUNNER_ARCH" == "ARM64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=aarch64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_ARCH: $RUNNER_ARCH"
exit 1
fi
echo "EXPERIMENTAL_INSTALLER_ARTIFACT=nix-installer-$EXPERIMENTAL_INSTALLER_ARCH-$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
env:
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- name: "Download latest experimental installer"
shell: bash
id: download-latest-experimental-installer
if: ${{ inputs.experimental-installer == 'true' && inputs.experimental-installer-version == 'latest' }}
run: |
RUN_ID=$(gh run list --repo "$EXPERIMENTAL_INSTALLER_REPO" --workflow ci.yml --branch main --status success --json databaseId --jq ".[0].databaseId")
EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR="$GITHUB_WORKSPACE/$EXPERIMENTAL_INSTALLER_ARTIFACT"
mkdir -p "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
gh run download "$RUN_ID" --repo "$EXPERIMENTAL_INSTALLER_REPO" -n "$EXPERIMENTAL_INSTALLER_ARTIFACT" -D "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
# Executable permissions are lost in artifacts
find $EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR -type f -exec chmod +x {} +
echo "installer-path=$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
env:
GH_TOKEN: ${{ inputs.github_token }}
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- uses: cachix/install-nix-action@c134e4c9e34bac6cab09cf239815f9339aaaf84e # v31.5.1
if: ${{ inputs.experimental-installer != 'true' }}
with:
# Ternary operator in GHA: https://www.github.com/actions/runner/issues/409#issuecomment-752775072
install_url: ${{ inputs.dogfood == 'true' && format('{0}/install', steps.download-nix-installer.outputs.installer-path) || inputs.install_url }}
install_options: ${{ inputs.dogfood == 'true' && format('--tarball-url-prefix {0}', steps.download-nix-installer.outputs.installer-path) || '' }}
extra_nix_config: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/nix-installer-action@786fff0690178f1234e4e1fe9b536e94f5433196 # v20
if: ${{ inputs.experimental-installer == 'true' }}
with:
diagnostic-endpoint: ""
# TODO: It'd be nice to use `artifacts.nixos.org` for both of these, maybe through an `/experimental-installer/latest` endpoint? or `/commit/<hash>`?
local-root: ${{ inputs.experimental-installer-version == 'latest' && steps.download-latest-experimental-installer.outputs.installer-path || '' }}
source-url: ${{ inputs.experimental-installer-version != 'latest' && 'https://artifacts.nixos.org/experimental-installer/tag/${{ inputs.experimental-installer-version }}/${{ env.EXPERIMENTAL_INSTALLER_ARTIFACT }}' || '' }}
nix-package-url: ${{ inputs.dogfood == 'true' && steps.download-nix-installer.outputs.tarball-path || (inputs.tarball_url || '') }}
extra-conf: ${{ inputs.extra_nix_config }}
+37
View File
@@ -0,0 +1,37 @@
name: Backport
on:
pull_request_target:
types: [closed, labeled]
permissions:
contents: read
jobs:
backport:
name: Backport Pull Request
permissions:
# for korthout/backport-action
contents: write
pull-requests: write
if: github.repository_owner == 'NixOS' && github.event.pull_request.merged == true && (github.event_name != 'labeled' || startsWith('backport', github.event.label.name))
runs-on: ubuntu-24.04-arm
steps:
- name: Generate GitHub App token
id: generate-token
uses: actions/create-github-app-token@v2
with:
app-id: ${{ vars.CI_APP_ID }}
private-key: ${{ secrets.CI_APP_PRIVATE_KEY }}
- uses: actions/checkout@v5
with:
ref: ${{ github.event.pull_request.head.sha }}
# required to find all branches
fetch-depth: 0
- name: Create backport PRs
uses: korthout/backport-action@d07416681cab29bf2661702f925f020aaa962997 # v3.4.1
id: backport
with:
# Config README: https://github.com/korthout/backport-action#backport-action
github_token: ${{ steps.generate-token.outputs.token }}
github_workspace: ${{ github.workspace }}
auto_merge_enabled: true
pull_description: |-
Automatic backport to `${target_branch}`, triggered by a label in #${pull_number}.
+32 -23
View File
@@ -4,6 +4,8 @@ on:
pull_request:
merge_group:
push:
branches:
- master
workflow_dispatch:
inputs:
dogfood:
@@ -100,6 +102,12 @@ jobs:
nix build --file ci/gha/tests/wrapper.nix componentTests -L \
--arg withInstrumentation ${{ matrix.instrumented }} \
--argstr stdenv "${{ matrix.stdenv }}"
- name: Run VM tests
run: |
nix build --file ci/gha/tests/wrapper.nix vmTests -L \
--arg withInstrumentation ${{ matrix.instrumented }} \
--argstr stdenv "${{ matrix.stdenv }}"
if: ${{ matrix.os == 'linux' }}
- name: Run flake checks and prepare the installer tarball
run: |
ci/gha/tests/build-checks
@@ -135,9 +143,19 @@ jobs:
- scenario: on ubuntu
runs-on: ubuntu-24.04
os: linux
experimental-installer: false
- scenario: on macos
runs-on: macos-14
os: darwin
experimental-installer: false
- scenario: on ubuntu (experimental)
runs-on: ubuntu-24.04
os: linux
experimental-installer: true
- scenario: on macos (experimental)
runs-on: macos-14
os: darwin
experimental-installer: true
name: installer test ${{ matrix.scenario }}
runs-on: ${{ matrix.runs-on }}
steps:
@@ -149,11 +167,22 @@ jobs:
path: out
- name: Looking up the installer tarball URL
id: installer-tarball-url
run: echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
run: |
echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$GITHUB_WORKSPACE/out" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
- uses: cachix/install-nix-action@v31
if: ${{ !matrix.experimental-installer }}
with:
install_url: ${{ format('{0}/install', steps.installer-tarball-url.outputs.installer-url) }}
install_options: ${{ format('--tarball-url-prefix {0}', steps.installer-tarball-url.outputs.installer-url) }}
- uses: ./.github/actions/install-nix-action
if: ${{ matrix.experimental-installer }}
with:
dogfood: false
experimental-installer: true
tarball_url: ${{ steps.installer-tarball-url.outputs.tarball-path }}
github_token: ${{ secrets.GITHUB_TOKEN }}
- run: sudo apt install fish zsh
if: matrix.os == 'linux'
- run: brew install fish
@@ -185,7 +214,7 @@ jobs:
echo "docker=${{ env._DOCKER_SECRETS != '' }}" >> $GITHUB_OUTPUT
docker_push_image:
needs: [tests, vm_tests, check_secrets]
needs: [tests, check_secrets]
permissions:
contents: read
packages: write
@@ -238,28 +267,8 @@ jobs:
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
vm_tests:
needs: basic-checks
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
- uses: ./.github/actions/install-nix-action
with:
dogfood: ${{ github.event_name == 'workflow_dispatch' && inputs.dogfood || github.event_name != 'workflow_dispatch' }}
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: |
nix build -L \
.#hydraJobs.tests.functional_user \
.#hydraJobs.tests.githubFlakes \
.#hydraJobs.tests.nix-docker \
.#hydraJobs.tests.tarballFlakes \
;
flake_regressions:
needs: vm_tests
needs: tests
runs-on: ubuntu-24.04
steps:
- name: Checkout nix
-174
View File
@@ -1,174 +0,0 @@
queue_rules:
- name: default
# all required tests need to go here
merge_conditions:
- check-success=tests on macos
- check-success=tests on ubuntu
- check-success=installer test on macos
- check-success=installer test on ubuntu
- check-success=vm_tests
batch_size: 5
pull_request_rules:
- name: merge using the merge queue
conditions:
- base~=master|.+-maintenance
- label~=merge-queue|dependencies
actions:
queue: {}
# The rules below will first create backport pull requests and put those in a merge queue.
- name: backport patches to 2.18
conditions:
- label=backport 2.18-maintenance
actions:
backport:
branches:
- 2.18-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.19
conditions:
- label=backport 2.19-maintenance
actions:
backport:
branches:
- 2.19-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.20
conditions:
- label=backport 2.20-maintenance
actions:
backport:
branches:
- 2.20-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.21
conditions:
- label=backport 2.21-maintenance
actions:
backport:
branches:
- 2.21-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.22
conditions:
- label=backport 2.22-maintenance
actions:
backport:
branches:
- 2.22-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.23
conditions:
- label=backport 2.23-maintenance
actions:
backport:
branches:
- 2.23-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.24
conditions:
- label=backport 2.24-maintenance
actions:
backport:
branches:
- "2.24-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.25
conditions:
- label=backport 2.25-maintenance
actions:
backport:
branches:
- "2.25-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.26
conditions:
- label=backport 2.26-maintenance
actions:
backport:
branches:
- "2.26-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.27
conditions:
- label=backport 2.27-maintenance
actions:
backport:
branches:
- "2.27-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.28
conditions:
- label=backport 2.28-maintenance
actions:
backport:
branches:
- "2.28-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.29
conditions:
- label=backport 2.29-maintenance
actions:
backport:
branches:
- "2.29-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.30
conditions:
- label=backport 2.30-maintenance
actions:
backport:
branches:
- "2.30-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.31
conditions:
- label=backport 2.31-maintenance
actions:
backport:
branches:
- "2.31-maintenance"
labels:
- automatic backport
- merge-queue
+1 -1
View File
@@ -1 +1 @@
2.32.0
2.33.0
+28 -13
View File
@@ -21,16 +21,6 @@ let
packages' = nixFlake.packages.${system};
stdenv = (getStdenv pkgs);
enableSanitizersLayer = finalAttrs: prevAttrs: {
mesonFlags =
(prevAttrs.mesonFlags or [ ])
++ [ (lib.mesonOption "b_sanitize" "address,undefined") ]
++ (lib.optionals stdenv.cc.isClang [
# https://www.github.com/mesonbuild/meson/issues/764
(lib.mesonBool "b_lundef" false)
]);
};
collectCoverageLayer = finalAttrs: prevAttrs: {
env =
let
@@ -53,14 +43,15 @@ let
'';
};
componentOverrides =
(lib.optional withSanitizers enableSanitizersLayer)
++ (lib.optional withCoverage collectCoverageLayer);
componentOverrides = (lib.optional withCoverage collectCoverageLayer);
in
rec {
nixComponentsInstrumented = nixComponents.overrideScope (
final: prev: {
withASan = withSanitizers;
withUBSan = withSanitizers;
nix-store-tests = prev.nix-store-tests.override { withBenchmarks = true; };
# Boehm is incompatible with ASAN.
nix-expr = prev.nix-expr.override { enableGC = !withSanitizers; };
@@ -71,6 +62,14 @@ rec {
}
);
# Import NixOS tests using the instrumented components
nixosTests = import ../../../tests/nixos {
inherit lib pkgs;
nixComponents = nixComponentsInstrumented;
nixpkgs = nixFlake.inputs.nixpkgs;
inherit (nixFlake.inputs) nixpkgs-23-11;
};
/**
Top-level tests for the flake outputs, as they would be built by hydra.
These tests generally can't be overridden to run with sanitizers.
@@ -221,4 +220,20 @@ rec {
{
inherit coverageProfileDrvs mergedProfdata coverageReports;
};
vmTests = {
inherit (nixosTests) s3-binary-cache-store;
}
// lib.optionalAttrs (!withSanitizers && !withCoverage) {
# evalNixpkgs uses non-instrumented components from hydraJobs, so only run it
# when not testing with sanitizers to avoid rebuilding nix
inherit (hydraJobs.tests) evalNixpkgs;
# FIXME: CI times out when building vm tests instrumented
inherit (nixosTests)
functional_user
githubFlakes
nix-docker
tarballFlakes
;
};
}
-1
View File
@@ -15,7 +15,6 @@ pymod = import('python')
python = pymod.find_installation('python3')
nix_env_for_docs = {
'ASAN_OPTIONS' : 'abort_on_error=1:print_summary=1:detect_leaks=0',
'HOME' : '/dummy',
'NIX_CONF_DIR' : '/dummy',
'NIX_SSL_CERT_FILE' : '/dummy/no-ca-bundle.crt',
@@ -0,0 +1,26 @@
---
synopsis: "Improved S3 binary cache support via HTTP"
prs: [13823, 14026, 14120, 14131, 14135, 14144, 14170, 14190, 14198, 14206, 14209, 14222, 14223, 13752]
issues: [13084, 12671, 11748, 12403]
---
S3 binary cache operations now happen via HTTP, leveraging `libcurl`'s native
AWS SigV4 authentication instead of the AWS C++ SDK, providing significant
improvements:
- **Reduced memory usage**: Eliminates memory buffering issues that caused
segfaults with large files
- **Fixed upload reliability**: Resolves AWS SDK chunking errors
(`InvalidChunkSizeError`)
- **Lighter dependencies**: Uses lightweight `aws-crt-cpp` instead of full
`aws-cpp-sdk`, reducing build complexity
The new implementation requires curl >= 7.75.0 and `aws-crt-cpp` for credential
management.
All existing S3 URL formats and parameters remain supported, with the notable
exception of multi-part uploads, which are no longer supported.
Note that this change also means Nix now supports S3 binary cache stores even
if build without `aws-crt-cpp`, but only for public buckets which do not
require auth.
+17 -6
View File
@@ -25,20 +25,31 @@ nix build .#nix-manual
and open `./result/share/doc/nix/manual/index.html`.
To build the manual incrementally, [enter the development shell](./building.md) and run:
To build the manual incrementally, [enter the development shell](./building.md) and configure with `doc-gen` enabled:
**If using interactive `nix develop`:**
```console
make manual-html-open -j $NIX_BUILD_CORES
$ nix develop
$ mesonFlags="$mesonFlags -Ddoc-gen=true" mesonConfigurePhase
```
In order to reflect changes to the [Makefile for the manual], clear all generated files before re-building:
[Makefile for the manual]: https://github.com/NixOS/nix/blob/master/doc/manual/local.mk
**If using direnv:**
```console
rm $(git ls-files doc/manual/ -o | grep -F '.md') && rmdir doc/manual/source/command-ref/new-cli && make manual-html -j $NIX_BUILD_CORES
$ direnv allow
$ bash -c 'source $stdenv/setup && mesonFlags="$mesonFlags -Ddoc-gen=true" mesonConfigurePhase'
```
Then build the manual:
```console
$ cd build
$ meson compile manual
```
The HTML manual will be generated at `build/src/nix-manual/manual/index.html`.
## Style guide
The goal of this style guide is to make it such that
@@ -25,7 +25,7 @@ is a JSON object with the following fields:
- Version 2: Separate `method` and `hashAlgo` fields in output specs
- Verison 3: Drop store dir from store paths, just include base name.
- Version 3: Drop store dir from store paths, just include base name.
Note that while this format is experimental, the maintenance of versions is best-effort, and not promised to identify every change.
@@ -116,5 +116,5 @@ is a JSON object with the following fields:
The environment passed to the `builder`.
* `structuredAttrs`:
[Strucutured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
[Structured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
Structured attributes are JSON, and thus embedded as-is.
+1 -1
View File
@@ -106,7 +106,7 @@ The system type on which the [`builder`](#attr-builder) executable is meant to b
A necessary condition for Nix to schedule a given derivation on some [Nix instance] is for the "system" of that derivation to match that instance's [`system` configuration option] or [`extra-platforms` configuration option].
By putting the `system` in each derivation, Nix allows *heterogenous* build plans, where not all steps can be run on the same machine or same sort of machine.
By putting the `system` in each derivation, Nix allows *heterogeneous* build plans, where not all steps can be run on the same machine or same sort of machine.
Nix can schedule builds such that it automatically builds on other platforms by [forwarding build requests](@docroot@/advanced-topics/distributed-builds.md) to other Nix instances.
[`system` configuration option]: @docroot@/command-ref/conf-file.md#conf-system
-1
View File
@@ -142,7 +142,6 @@ release:
$ git pull
$ NEW_VERSION=2.13.0
$ echo $NEW_VERSION > .version
$ ... edit .mergify.yml to add the previous version ...
$ git checkout -b bump-$NEW_VERSION
$ git commit -a -m 'Bump version'
$ git push --set-upstream origin bump-$NEW_VERSION
@@ -0,0 +1,6 @@
extern "C" [[gnu::retain, gnu::weak]] const char * __asan_default_options()
{
// We leak a bunch of memory knowingly on purpose. It's not worthwhile to
// diagnose that memory being leaked for now.
return "abort_on_error=1:print_summary=1:detect_leaks=0:detect_odr_violation=0";
}
@@ -1,7 +1,3 @@
asan_test_options_env = {
'ASAN_OPTIONS' : 'abort_on_error=1:print_summary=1:detect_leaks=0',
}
# Clang gets grumpy about missing libasan symbols if -shared-libasan is not
# passed when building shared libs, at least on Linux
if cxx.get_id() == 'clang' and ('address' in get_option('b_sanitize') or 'undefined' in get_option(
@@ -10,3 +6,6 @@ if cxx.get_id() == 'clang' and ('address' in get_option('b_sanitize') or 'undefi
add_project_link_arguments('-shared-libasan', language : 'cpp')
endif
if 'address' in get_option('b_sanitize')
deps_other += declare_dependency(sources : 'asan-options.cc')
endif
@@ -0,0 +1,32 @@
can_wrap_assert_fail_test_code = '''
#include <cstdlib>
#include <cassert>
int main()
{
assert(0);
}
extern "C" void * __real___assert_fail(const char *, const char *, unsigned int, const char *);
extern "C" void *
__wrap___assert_fail(const char *, const char *, unsigned int, const char *)
{
return __real___assert_fail(nullptr, nullptr, 0, nullptr);
}
'''
wrap_assert_fail_args = [ '-Wl,--wrap=__assert_fail' ]
can_wrap_assert_fail = cxx.links(
can_wrap_assert_fail_test_code,
args : wrap_assert_fail_args,
name : 'linker can wrap __assert_fail',
)
if can_wrap_assert_fail
deps_other += declare_dependency(
sources : 'wrap-assert-fail.cc',
link_args : wrap_assert_fail_args,
)
endif
@@ -0,0 +1,17 @@
#include "nix/util/error.hh"
#include <cstdio>
#include <cstdlib>
#include <cinttypes>
#include <string_view>
extern "C" [[noreturn]] void __attribute__((weak))
__wrap___assert_fail(const char * assertion, const char * file, unsigned int line, const char * function)
{
char buf[512];
int n =
snprintf(buf, sizeof(buf), "Assertion '%s' failed in %s at %s:%" PRIuLEAST32, assertion, function, file, line);
if (n < 0)
nix::panic("Assertion failed and could not format error message");
nix::panic(std::string_view(buf, std::min(static_cast<int>(sizeof(buf)), n)));
}
@@ -44,3 +44,6 @@ endif
# Darwin ld doesn't like "X.Y.Zpre"
nix_soversion = meson.project_version().split('pre')[0]
subdir('assert-fail')
subdir('asan-options')
+33 -1
View File
@@ -204,6 +204,25 @@ let
mesonFlags = [ (lib.mesonBool "b_asneeded" false) ] ++ prevAttrs.mesonFlags or [ ];
};
enableSanitizersLayer =
finalAttrs: prevAttrs:
let
sanitizers = lib.optional scope.withASan "address" ++ lib.optional scope.withUBSan "undefined";
in
{
mesonFlags =
(prevAttrs.mesonFlags or [ ])
++ lib.optionals (lib.length sanitizers > 0) (
[
(lib.mesonOption "b_sanitize" (lib.concatStringsSep "," sanitizers))
]
++ (lib.optionals stdenv.cc.isClang [
# https://www.github.com/mesonbuild/meson/issues/764
(lib.mesonBool "b_lundef" false)
])
);
};
nixDefaultsLayer = finalAttrs: prevAttrs: {
strictDeps = prevAttrs.strictDeps or true;
enableParallelBuilding = true;
@@ -246,6 +265,16 @@ in
inherit filesetToSource;
/**
Whether meson components are built with [AddressSanitizer](https://clang.llvm.org/docs/AddressSanitizer.html).
*/
withASan = false;
/**
Whether meson components are built with [UndefinedBehaviorSanitizer](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html).
*/
withUBSan = false;
/**
A user-provided extension function to apply to each component derivation.
*/
@@ -332,6 +361,7 @@ in
setVersionLayer
mesonLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
mkMesonExecutable = mkPackageBuilder [
@@ -342,6 +372,7 @@ in
mesonLayer
mesonBuildLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
mkMesonLibrary = mkPackageBuilder [
@@ -353,6 +384,7 @@ in
mesonBuildLayer
mesonLibraryLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
@@ -458,7 +490,7 @@ in
Example:
```
overrideScope (finalScope: prevScope: { aws-sdk-cpp = null; })
overrideScope (finalScope: prevScope: { aws-crt-cpp = null; })
```
*/
overrideScope = f: (scope.overrideScope f).nix-everything;
+14 -58
View File
@@ -16,21 +16,6 @@ in
scope: {
inherit stdenv;
aws-sdk-cpp =
(pkgs.aws-sdk-cpp.override {
apis = [
"identity-management"
"s3"
"transfer"
];
customMemoryManagement = false;
}).overrideAttrs
{
# only a stripped down version is built, which takes a lot less resources
# to build, so we don't need a "big-parallel" machine.
requiredSystemFeatures = [ ];
};
boehmgc =
(pkgs.boehmgc.override {
enableLargeConfig = true;
@@ -57,15 +42,20 @@ scope: {
prevAttrs.postInstall;
});
toml11 = pkgs.toml11.overrideAttrs rec {
version = "4.4.0";
src = pkgs.fetchFromGitHub {
owner = "ToruNiina";
repo = "toml11";
tag = "v${version}";
hash = "sha256-sgWKYxNT22nw376ttGsTdg0AMzOwp8QH3E8mx0BZJTQ=";
};
};
# TODO: Remove this when https://github.com/NixOS/nixpkgs/pull/442682 is included in a stable release
toml11 =
if lib.versionAtLeast pkgs.toml11.version "4.4.0" then
pkgs.toml11
else
pkgs.toml11.overrideAttrs rec {
version = "4.4.0";
src = pkgs.fetchFromGitHub {
owner = "ToruNiina";
repo = "toml11";
tag = "v${version}";
hash = "sha256-sgWKYxNT22nw376ttGsTdg0AMzOwp8QH3E8mx0BZJTQ=";
};
};
# TODO Hack until https://github.com/NixOS/nixpkgs/issues/45462 is fixed.
boost =
@@ -84,38 +74,4 @@ scope: {
buildPhase = lib.replaceStrings [ "--without-python" ] [ "" ] old.buildPhase;
installPhase = lib.replaceStrings [ "--without-python" ] [ "" ] old.installPhase;
});
libgit2 =
if lib.versionAtLeast pkgs.libgit2.version "1.9.0" then
pkgs.libgit2
else
pkgs.libgit2.overrideAttrs (attrs: {
# libgit2: Nixpkgs 24.11 has < 1.9.0, which needs our patches
nativeBuildInputs =
attrs.nativeBuildInputs or [ ]
# gitMinimal does not build on Windows. See packbuilder patch.
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# Needed for `git apply`; see `prePatch`
pkgs.buildPackages.gitMinimal
];
# Only `git apply` can handle git binary patches
prePatch =
attrs.prePatch or ""
+ lib.optionalString (!stdenv.hostPlatform.isWindows) ''
patch() {
git apply
}
'';
patches =
attrs.patches or [ ]
++ [
./patches/libgit2-mempack-thin-packfile.patch
]
# gitMinimal does not build on Windows, but fortunately this patch only
# impacts interruptibility
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# binary patch; see `prePatch`
./patches/libgit2-packbuilder-callback-interruptible.patch
];
});
}
+41 -32
View File
@@ -70,6 +70,9 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
# We use this shell with the local checkout, not unpackPhase.
src = null;
# Workaround https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
# Remove when gdb fix is rolled out everywhere.
separateDebugInfo = false;
env = {
# For `make format`, to work without installing pre-commit
@@ -93,38 +96,44 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
++ map (transformFlag "libcmd") (ignoreCrossFile pkgs.nixComponents2.nix-cmd.mesonFlags);
nativeBuildInputs =
attrs.nativeBuildInputs or [ ]
++ pkgs.nixComponents2.nix-util.nativeBuildInputs
++ pkgs.nixComponents2.nix-store.nativeBuildInputs
++ pkgs.nixComponents2.nix-fetchers.nativeBuildInputs
++ pkgs.nixComponents2.nix-expr.nativeBuildInputs
++ lib.optionals havePerl pkgs.nixComponents2.nix-perl-bindings.nativeBuildInputs
++ lib.optionals buildCanExecuteHost pkgs.nixComponents2.nix-manual.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-internal-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-external-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-functional-tests.externalNativeBuildInputs
++ lib.optional (
!buildCanExecuteHost
# Hack around https://github.com/nixos/nixpkgs/commit/bf7ad8cfbfa102a90463433e2c5027573b462479
&& !(stdenv.hostPlatform.isWindows && stdenv.buildPlatform.isDarwin)
&& stdenv.hostPlatform.emulatorAvailable pkgs.buildPackages
&& lib.meta.availableOn stdenv.buildPlatform (stdenv.hostPlatform.emulator pkgs.buildPackages)
) pkgs.buildPackages.mesonEmulatorHook
++ [
pkgs.buildPackages.cmake
pkgs.buildPackages.gnused
pkgs.buildPackages.shellcheck
pkgs.buildPackages.changelog-d
modular.pre-commit.settings.package
(pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript)
pkgs.buildPackages.nixfmt-rfc-style
pkgs.buildPackages.shellcheck
pkgs.buildPackages.gdb
]
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
)
++ lib.optional stdenv.hostPlatform.isLinux pkgs.buildPackages.mold-wrapped;
let
inputs =
attrs.nativeBuildInputs or [ ]
++ pkgs.nixComponents2.nix-util.nativeBuildInputs
++ pkgs.nixComponents2.nix-store.nativeBuildInputs
++ pkgs.nixComponents2.nix-fetchers.nativeBuildInputs
++ pkgs.nixComponents2.nix-expr.nativeBuildInputs
++ lib.optionals havePerl pkgs.nixComponents2.nix-perl-bindings.nativeBuildInputs
++ lib.optionals buildCanExecuteHost pkgs.nixComponents2.nix-manual.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-internal-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-external-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-functional-tests.externalNativeBuildInputs
++ lib.optional (
!buildCanExecuteHost
# Hack around https://github.com/nixos/nixpkgs/commit/bf7ad8cfbfa102a90463433e2c5027573b462479
&& !(stdenv.hostPlatform.isWindows && stdenv.buildPlatform.isDarwin)
&& stdenv.hostPlatform.emulatorAvailable pkgs.buildPackages
&& lib.meta.availableOn stdenv.buildPlatform (stdenv.hostPlatform.emulator pkgs.buildPackages)
) pkgs.buildPackages.mesonEmulatorHook
++ [
pkgs.buildPackages.cmake
pkgs.buildPackages.gnused
pkgs.buildPackages.changelog-d
modular.pre-commit.settings.package
(pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript)
pkgs.buildPackages.nixfmt-rfc-style
pkgs.buildPackages.shellcheck
pkgs.buildPackages.include-what-you-use
pkgs.buildPackages.gdb
]
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
)
++ lib.optional stdenv.hostPlatform.isLinux pkgs.buildPackages.mold-wrapped;
in
# FIXME: separateDebugInfo = false doesn't actually prevent -Wa,--compress-debug-sections
# from making its way into NIX_CFLAGS_COMPILE.
lib.filter (p: !lib.hasInfix "separate-debug-info" p) inputs;
buildInputs = [
pkgs.gbenchmark
+15 -3
View File
@@ -73,7 +73,7 @@ let
]
);
in
{
rec {
/**
An internal check to make sure our package listing is complete.
*/
@@ -145,13 +145,25 @@ in
)
);
buildNoGc =
# Builds with sanitizers already have GC disabled, so this buildNoGc can just
# point to buildWithSanitizers in order to reduce the load on hydra.
buildNoGc = buildWithSanitizers;
buildWithSanitizers =
let
components = forAllSystems (
system:
nixpkgsFor.${system}.native.nixComponents2.overrideScope (
let
pkgs = nixpkgsFor.${system}.native;
in
pkgs.nixComponents2.overrideScope (
self: super: {
# Boost coroutines fail with ASAN on darwin.
withASan = !pkgs.stdenv.buildPlatform.isDarwin;
withUBSan = true;
nix-expr = super.nix-expr.override { enableGC = false; };
# Unclear how to make Perl bindings work with a dynamically linked ASAN.
nix-perl-bindings = null;
}
)
);
@@ -1,282 +0,0 @@
commit 9bacade4a3ef4b6b26e2c02f549eef0e9eb9eaa2
Author: Robert Hensing <robert@roberthensing.nl>
Date: Sun Aug 18 20:20:36 2024 +0200
Add unoptimized git_mempack_write_thin_pack
diff --git a/include/git2/sys/mempack.h b/include/git2/sys/mempack.h
index 17da590a3..3688bdd50 100644
--- a/include/git2/sys/mempack.h
+++ b/include/git2/sys/mempack.h
@@ -44,6 +44,29 @@ GIT_BEGIN_DECL
*/
GIT_EXTERN(int) git_mempack_new(git_odb_backend **out);
+/**
+ * Write a thin packfile with the objects in the memory store.
+ *
+ * A thin packfile is a packfile that does not contain its transitive closure of
+ * references. This is useful for efficiently distributing additions to a
+ * repository over the network, but also finds use in the efficient bulk
+ * addition of objects to a repository, locally.
+ *
+ * This operation performs the (shallow) insert operations into the
+ * `git_packbuilder`, but does not write the packfile to disk;
+ * see `git_packbuilder_write_buf`.
+ *
+ * It also does not reset the memory store; see `git_mempack_reset`.
+ *
+ * @note This function may or may not write trees and blobs that are not
+ * referenced by commits. Currently everything is written, but this
+ * behavior may change in the future as the packer is optimized.
+ *
+ * @param backend The mempack backend
+ * @param pb The packbuilder to use to write the packfile
+ */
+GIT_EXTERN(int) git_mempack_write_thin_pack(git_odb_backend *backend, git_packbuilder *pb);
+
/**
* Dump all the queued in-memory writes to a packfile.
*
diff --git a/src/libgit2/odb_mempack.c b/src/libgit2/odb_mempack.c
index 6f27f45f8..0b61e2b66 100644
--- a/src/libgit2/odb_mempack.c
+++ b/src/libgit2/odb_mempack.c
@@ -132,6 +132,35 @@ cleanup:
return err;
}
+int git_mempack_write_thin_pack(git_odb_backend *backend, git_packbuilder *pb)
+{
+ struct memory_packer_db *db = (struct memory_packer_db *)backend;
+ const git_oid *oid;
+ size_t iter = 0;
+ int err = -1;
+
+ /* TODO: Implement the recency heuristics.
+ For this it probably makes sense to only write what's referenced
+ through commits, an option I've carved out for you in the docs.
+ wrt heuristics: ask your favorite LLM to translate https://git-scm.com/docs/pack-heuristics/en
+ to actual normal reference documentation. */
+ while (true) {
+ err = git_oidmap_iterate(NULL, db->objects, &iter, &oid);
+ if (err == GIT_ITEROVER) {
+ err = 0;
+ break;
+ }
+ if (err != 0)
+ return err;
+
+ err = git_packbuilder_insert(pb, oid, NULL);
+ if (err != 0)
+ return err;
+ }
+
+ return 0;
+}
+
int git_mempack_dump(
git_buf *pack,
git_repository *repo,
diff --git a/tests/libgit2/mempack/thinpack.c b/tests/libgit2/mempack/thinpack.c
new file mode 100644
index 000000000..604a4dda2
--- /dev/null
+++ b/tests/libgit2/mempack/thinpack.c
@@ -0,0 +1,196 @@
+#include "clar_libgit2.h"
+#include "git2/indexer.h"
+#include "git2/odb_backend.h"
+#include "git2/tree.h"
+#include "git2/types.h"
+#include "git2/sys/mempack.h"
+#include "git2/sys/odb_backend.h"
+#include "util.h"
+
+static git_repository *_repo;
+static git_odb_backend * _mempack_backend;
+
+void test_mempack_thinpack__initialize(void)
+{
+ git_odb *odb;
+
+ _repo = cl_git_sandbox_init_new("mempack_thinpack_repo");
+
+ cl_git_pass(git_mempack_new(&_mempack_backend));
+ cl_git_pass(git_repository_odb(&odb, _repo));
+ cl_git_pass(git_odb_add_backend(odb, _mempack_backend, 999));
+ git_odb_free(odb);
+}
+
+void _mempack_thinpack__cleanup(void)
+{
+ cl_git_sandbox_cleanup();
+}
+
+/*
+ Generating a packfile for an unchanged repo works and produces an empty packfile.
+ Even if we allow this scenario to be detected, it shouldn't misbehave if the
+ application is unaware of it.
+*/
+void test_mempack_thinpack__empty(void)
+{
+ git_packbuilder *pb;
+ int version;
+ int n;
+ git_buf buf = GIT_BUF_INIT;
+
+ git_packbuilder_new(&pb, _repo);
+
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_assert_in_range(12, buf.size, 1024 /* empty packfile is >0 bytes, but certainly not that big */);
+ cl_assert(buf.ptr[0] == 'P');
+ cl_assert(buf.ptr[1] == 'A');
+ cl_assert(buf.ptr[2] == 'C');
+ cl_assert(buf.ptr[3] == 'K');
+ version = (buf.ptr[4] << 24) | (buf.ptr[5] << 16) | (buf.ptr[6] << 8) | buf.ptr[7];
+ /* Subject to change. https://git-scm.com/docs/pack-format: Git currently accepts version number 2 or 3 but generates version 2 only.*/
+ cl_assert_equal_i(2, version);
+ n = (buf.ptr[8] << 24) | (buf.ptr[9] << 16) | (buf.ptr[10] << 8) | buf.ptr[11];
+ cl_assert_equal_i(0, n);
+ git_buf_dispose(&buf);
+
+ git_packbuilder_free(pb);
+}
+
+#define LIT_LEN(x) x, sizeof(x) - 1
+
+/*
+ Check that git_mempack_write_thin_pack produces a thin packfile.
+*/
+void test_mempack_thinpack__thin(void)
+{
+ /* Outline:
+ - Create tree 1
+ - Flush to packfile A
+ - Create tree 2
+ - Flush to packfile B
+
+ Tree 2 has a new blob and a reference to a blob from tree 1.
+
+ Expectation:
+ - Packfile B is thin and does not contain the objects from packfile A
+ */
+
+
+ git_oid oid_blob_1;
+ git_oid oid_blob_2;
+ git_oid oid_blob_3;
+ git_oid oid_tree_1;
+ git_oid oid_tree_2;
+ git_treebuilder *tb;
+
+ git_packbuilder *pb;
+ git_buf buf = GIT_BUF_INIT;
+ git_indexer *indexer;
+ git_indexer_progress stats;
+ char pack_dir_path[1024];
+
+ char sbuf[1024];
+ const char * repo_path;
+ const char * pack_name_1;
+ const char * pack_name_2;
+ git_str pack_path_1 = GIT_STR_INIT;
+ git_str pack_path_2 = GIT_STR_INIT;
+ git_odb_backend * pack_odb_backend_1;
+ git_odb_backend * pack_odb_backend_2;
+
+
+ cl_assert_in_range(0, snprintf(pack_dir_path, sizeof(pack_dir_path), "%s/objects/pack", git_repository_path(_repo)), sizeof(pack_dir_path));
+
+ /* Create tree 1 */
+
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_1, _repo, LIT_LEN("thinpack blob 1")));
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_2, _repo, LIT_LEN("thinpack blob 2")));
+
+
+ cl_git_pass(git_treebuilder_new(&tb, _repo, NULL));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob1", &oid_blob_1, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob2", &oid_blob_2, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_write(&oid_tree_1, tb));
+
+ /* Flush */
+
+ cl_git_pass(git_packbuilder_new(&pb, _repo));
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_git_pass(git_indexer_new(&indexer, pack_dir_path, 0, NULL, NULL));
+ cl_git_pass(git_indexer_append(indexer, buf.ptr, buf.size, &stats));
+ cl_git_pass(git_indexer_commit(indexer, &stats));
+ pack_name_1 = strdup(git_indexer_name(indexer));
+ cl_assert(pack_name_1);
+ git_buf_dispose(&buf);
+ git_mempack_reset(_mempack_backend);
+ git_indexer_free(indexer);
+ git_packbuilder_free(pb);
+
+ /* Create tree 2 */
+
+ cl_git_pass(git_treebuilder_clear(tb));
+ /* blob 1 won't be used, but we add it anyway to test that just "declaring" an object doesn't
+ necessarily cause its inclusion in the next thin packfile. It must only be included if new. */
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_1, _repo, LIT_LEN("thinpack blob 1")));
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_3, _repo, LIT_LEN("thinpack blob 3")));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob1", &oid_blob_1, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob3", &oid_blob_3, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_write(&oid_tree_2, tb));
+
+ /* Flush */
+
+ cl_git_pass(git_packbuilder_new(&pb, _repo));
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_git_pass(git_indexer_new(&indexer, pack_dir_path, 0, NULL, NULL));
+ cl_git_pass(git_indexer_append(indexer, buf.ptr, buf.size, &stats));
+ cl_git_pass(git_indexer_commit(indexer, &stats));
+ pack_name_2 = strdup(git_indexer_name(indexer));
+ cl_assert(pack_name_2);
+ git_buf_dispose(&buf);
+ git_mempack_reset(_mempack_backend);
+ git_indexer_free(indexer);
+ git_packbuilder_free(pb);
+ git_treebuilder_free(tb);
+
+ /* Assertions */
+
+ assert(pack_name_1);
+ assert(pack_name_2);
+
+ repo_path = git_repository_path(_repo);
+
+ snprintf(sbuf, sizeof(sbuf), "objects/pack/pack-%s.pack", pack_name_1);
+ git_str_joinpath(&pack_path_1, repo_path, sbuf);
+ snprintf(sbuf, sizeof(sbuf), "objects/pack/pack-%s.pack", pack_name_2);
+ git_str_joinpath(&pack_path_2, repo_path, sbuf);
+
+ /* If they're the same, something definitely went wrong. */
+ cl_assert(strcmp(pack_name_1, pack_name_2) != 0);
+
+ cl_git_pass(git_odb_backend_one_pack(&pack_odb_backend_1, pack_path_1.ptr));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_1));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_2));
+ cl_assert(!pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_3));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_tree_1));
+ cl_assert(!pack_odb_backend_1->exists(pack_odb_backend_1, &oid_tree_2));
+
+ cl_git_pass(git_odb_backend_one_pack(&pack_odb_backend_2, pack_path_2.ptr));
+ /* blob 1 is already in the packfile 1, so packfile 2 must not include it, in order to be _thin_. */
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_1));
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_2));
+ cl_assert(pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_3));
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_tree_1));
+ cl_assert(pack_odb_backend_2->exists(pack_odb_backend_2, &oid_tree_2));
+
+ pack_odb_backend_1->free(pack_odb_backend_1);
+ pack_odb_backend_2->free(pack_odb_backend_2);
+ free((void *)pack_name_1);
+ free((void *)pack_name_2);
+ git_str_dispose(&pack_path_1);
+ git_str_dispose(&pack_path_2);
+
+}
@@ -1,930 +0,0 @@
commit e9823c5da4fa977c46bcb97167fbdd0d70adb5ff
Author: Robert Hensing <robert@roberthensing.nl>
Date: Mon Aug 26 20:07:04 2024 +0200
Make packbuilder interruptible using progress callback
Forward errors from packbuilder->progress_cb
This allows the callback to terminate long-running operations when
the application is interrupted.
diff --git a/include/git2/pack.h b/include/git2/pack.h
index 0f6bd2ab9..bee72a6c0 100644
--- a/include/git2/pack.h
+++ b/include/git2/pack.h
@@ -247,6 +247,9 @@ typedef int GIT_CALLBACK(git_packbuilder_progress)(
* @param progress_cb Function to call with progress information during
* pack building. Be aware that this is called inline with pack building
* operations, so performance may be affected.
+ * When progress_cb returns an error, the pack building process will be
+ * aborted and the error will be returned from the invoked function.
+ * `pb` must then be freed.
* @param progress_cb_payload Payload for progress callback.
* @return 0 or an error code
*/
diff --git a/src/libgit2/pack-objects.c b/src/libgit2/pack-objects.c
index b2d80cba9..7c331c2d5 100644
--- a/src/libgit2/pack-objects.c
+++ b/src/libgit2/pack-objects.c
@@ -932,6 +932,9 @@ static int report_delta_progress(
{
int ret;
+ if (pb->failure)
+ return pb->failure;
+
if (pb->progress_cb) {
uint64_t current_time = git_time_monotonic();
uint64_t elapsed = current_time - pb->last_progress_report_time;
@@ -943,8 +946,10 @@ static int report_delta_progress(
GIT_PACKBUILDER_DELTAFICATION,
count, pb->nr_objects, pb->progress_cb_payload);
- if (ret)
+ if (ret) {
+ pb->failure = ret;
return git_error_set_after_callback(ret);
+ }
}
}
@@ -976,7 +981,10 @@ static int find_deltas(git_packbuilder *pb, git_pobject **list,
}
pb->nr_deltified += 1;
- report_delta_progress(pb, pb->nr_deltified, false);
+ if ((error = report_delta_progress(pb, pb->nr_deltified, false)) < 0) {
+ GIT_ASSERT(git_packbuilder__progress_unlock(pb) == 0);
+ goto on_error;
+ }
po = *list++;
(*list_size)--;
@@ -1124,6 +1132,10 @@ struct thread_params {
size_t depth;
size_t working;
size_t data_ready;
+
+ /* A pb->progress_cb can stop the packing process by returning an error.
+ When that happens, all threads observe the error and stop voluntarily. */
+ bool stopped;
};
static void *threaded_find_deltas(void *arg)
@@ -1133,7 +1145,12 @@ static void *threaded_find_deltas(void *arg)
while (me->remaining) {
if (find_deltas(me->pb, me->list, &me->remaining,
me->window, me->depth) < 0) {
- ; /* TODO */
+ me->stopped = true;
+ GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_lock(me->pb) == 0, NULL);
+ me->working = false;
+ git_cond_signal(&me->pb->progress_cond);
+ GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_unlock(me->pb) == 0, NULL);
+ return NULL;
}
GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_lock(me->pb) == 0, NULL);
@@ -1175,8 +1192,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
pb->nr_threads = git__online_cpus();
if (pb->nr_threads <= 1) {
- find_deltas(pb, list, &list_size, window, depth);
- return 0;
+ return find_deltas(pb, list, &list_size, window, depth);
}
p = git__mallocarray(pb->nr_threads, sizeof(*p));
@@ -1195,6 +1211,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
p[i].depth = depth;
p[i].working = 1;
p[i].data_ready = 0;
+ p[i].stopped = 0;
/* try to split chunks on "path" boundaries */
while (sub_size && sub_size < list_size &&
@@ -1262,7 +1279,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
(!victim || victim->remaining < p[i].remaining))
victim = &p[i];
- if (victim) {
+ if (victim && !target->stopped) {
sub_size = victim->remaining / 2;
list = victim->list + victim->list_size - sub_size;
while (sub_size && list[0]->hash &&
@@ -1286,7 +1303,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
}
target->list_size = sub_size;
target->remaining = sub_size;
- target->working = 1;
+ target->working = 1; /* even when target->stopped, so that we don't process this thread again */
GIT_ASSERT(git_packbuilder__progress_unlock(pb) == 0);
if (git_mutex_lock(&target->mutex)) {
@@ -1299,7 +1316,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
git_cond_signal(&target->cond);
git_mutex_unlock(&target->mutex);
- if (!sub_size) {
+ if (target->stopped || !sub_size) {
git_thread_join(&target->thread, NULL);
git_cond_free(&target->cond);
git_mutex_free(&target->mutex);
@@ -1308,7 +1325,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
}
git__free(p);
- return 0;
+ return pb->failure;
}
#else
@@ -1319,6 +1336,7 @@ int git_packbuilder__prepare(git_packbuilder *pb)
{
git_pobject **delta_list;
size_t i, n = 0;
+ int error;
if (pb->nr_objects == 0 || pb->done)
return 0; /* nothing to do */
@@ -1327,8 +1345,10 @@ int git_packbuilder__prepare(git_packbuilder *pb)
* Although we do not report progress during deltafication, we
* at least report that we are in the deltafication stage
*/
- if (pb->progress_cb)
- pb->progress_cb(GIT_PACKBUILDER_DELTAFICATION, 0, pb->nr_objects, pb->progress_cb_payload);
+ if (pb->progress_cb) {
+ if ((error = pb->progress_cb(GIT_PACKBUILDER_DELTAFICATION, 0, pb->nr_objects, pb->progress_cb_payload)) < 0)
+ return git_error_set_after_callback(error);
+ }
delta_list = git__mallocarray(pb->nr_objects, sizeof(*delta_list));
GIT_ERROR_CHECK_ALLOC(delta_list);
@@ -1345,31 +1365,33 @@ int git_packbuilder__prepare(git_packbuilder *pb)
if (n > 1) {
git__tsort((void **)delta_list, n, type_size_sort);
- if (ll_find_deltas(pb, delta_list, n,
+ if ((error = ll_find_deltas(pb, delta_list, n,
GIT_PACK_WINDOW + 1,
- GIT_PACK_DEPTH) < 0) {
+ GIT_PACK_DEPTH)) < 0) {
git__free(delta_list);
- return -1;
+ return error;
}
}
- report_delta_progress(pb, pb->nr_objects, true);
+ error = report_delta_progress(pb, pb->nr_objects, true);
pb->done = true;
git__free(delta_list);
- return 0;
+ return error;
}
-#define PREPARE_PACK if (git_packbuilder__prepare(pb) < 0) { return -1; }
+#define PREPARE_PACK error = git_packbuilder__prepare(pb); if (error < 0) { return error; }
int git_packbuilder_foreach(git_packbuilder *pb, int (*cb)(void *buf, size_t size, void *payload), void *payload)
{
+ int error;
PREPARE_PACK;
return write_pack(pb, cb, payload);
}
int git_packbuilder__write_buf(git_str *buf, git_packbuilder *pb)
{
+ int error;
PREPARE_PACK;
return write_pack(pb, &write_pack_buf, buf);
diff --git a/src/libgit2/pack-objects.h b/src/libgit2/pack-objects.h
index bbc8b9430..380a28ebe 100644
--- a/src/libgit2/pack-objects.h
+++ b/src/libgit2/pack-objects.h
@@ -100,6 +100,10 @@ struct git_packbuilder {
uint64_t last_progress_report_time;
bool done;
+
+ /* A non-zero error code in failure causes all threads to shut themselves
+ down. Some functions will return this error code. */
+ volatile int failure;
};
int git_packbuilder__write_buf(git_str *buf, git_packbuilder *pb);
diff --git a/tests/libgit2/pack/cancel.c b/tests/libgit2/pack/cancel.c
new file mode 100644
index 000000000..a0aa9716a
--- /dev/null
+++ b/tests/libgit2/pack/cancel.c
@@ -0,0 +1,240 @@
+#include "clar_libgit2.h"
+#include "futils.h"
+#include "pack.h"
+#include "hash.h"
+#include "iterator.h"
+#include "vector.h"
+#include "posix.h"
+#include "hash.h"
+#include "pack-objects.h"
+
+static git_repository *_repo;
+static git_revwalk *_revwalker;
+static git_packbuilder *_packbuilder;
+static git_indexer *_indexer;
+static git_vector _commits;
+static int _commits_is_initialized;
+static git_indexer_progress _stats;
+
+extern bool git_disable_pack_keep_file_checks;
+
+static void pack_packbuilder_init(const char *sandbox) {
+ _repo = cl_git_sandbox_init(sandbox);
+ /* cl_git_pass(p_chdir(sandbox)); */
+ cl_git_pass(git_revwalk_new(&_revwalker, _repo));
+ cl_git_pass(git_packbuilder_new(&_packbuilder, _repo));
+ cl_git_pass(git_vector_init(&_commits, 0, NULL));
+ _commits_is_initialized = 1;
+ memset(&_stats, 0, sizeof(_stats));
+ p_fsync__cnt = 0;
+}
+
+void test_pack_cancel__initialize(void)
+{
+ pack_packbuilder_init("small.git");
+}
+
+void test_pack_cancel__cleanup(void)
+{
+ git_oid *o;
+ unsigned int i;
+
+ cl_git_pass(git_libgit2_opts(GIT_OPT_ENABLE_FSYNC_GITDIR, 0));
+ cl_git_pass(git_libgit2_opts(GIT_OPT_DISABLE_PACK_KEEP_FILE_CHECKS, false));
+
+ if (_commits_is_initialized) {
+ _commits_is_initialized = 0;
+ git_vector_foreach(&_commits, i, o) {
+ git__free(o);
+ }
+ git_vector_free(&_commits);
+ }
+
+ git_packbuilder_free(_packbuilder);
+ _packbuilder = NULL;
+
+ git_revwalk_free(_revwalker);
+ _revwalker = NULL;
+
+ git_indexer_free(_indexer);
+ _indexer = NULL;
+
+ /* cl_git_pass(p_chdir("..")); */
+ cl_git_sandbox_cleanup();
+ _repo = NULL;
+}
+
+static int seed_packbuilder(void)
+{
+ int error;
+ git_oid oid, *o;
+ unsigned int i;
+
+ git_revwalk_sorting(_revwalker, GIT_SORT_TIME);
+ cl_git_pass(git_revwalk_push_ref(_revwalker, "HEAD"));
+
+ while (git_revwalk_next(&oid, _revwalker) == 0) {
+ o = git__malloc(sizeof(git_oid));
+ cl_assert(o != NULL);
+ git_oid_cpy(o, &oid);
+ cl_git_pass(git_vector_insert(&_commits, o));
+ }
+
+ git_vector_foreach(&_commits, i, o) {
+ if((error = git_packbuilder_insert(_packbuilder, o, NULL)) < 0)
+ return error;
+ }
+
+ git_vector_foreach(&_commits, i, o) {
+ git_object *obj;
+ cl_git_pass(git_object_lookup(&obj, _repo, o, GIT_OBJECT_COMMIT));
+ error = git_packbuilder_insert_tree(_packbuilder,
+ git_commit_tree_id((git_commit *)obj));
+ git_object_free(obj);
+ if (error < 0)
+ return error;
+ }
+
+ return 0;
+}
+
+static int fail_stage;
+
+static int packbuilder_cancel_after_n_calls_cb(int stage, uint32_t current, uint32_t total, void *payload)
+{
+
+ /* Force the callback to run again on the next opportunity regardless
+ of how fast we're running. */
+ _packbuilder->last_progress_report_time = 0;
+
+ if (stage == fail_stage) {
+ int *calls = (int *)payload;
+ int n = *calls;
+ /* Always decrement, including past zero. This way the error is only
+ triggered once, making sure it is picked up immediately. */
+ --*calls;
+ if (n == 0)
+ return GIT_EUSER;
+ }
+
+ return 0;
+}
+
+static void test_cancel(int n)
+{
+
+ int calls_remaining = n;
+ int err;
+ git_buf buf = GIT_BUF_INIT;
+
+ /* Switch to a small repository, so that `packbuilder_cancel_after_n_calls_cb`
+ can hack the time to call the callback on every opportunity. */
+
+ cl_git_pass(git_packbuilder_set_callbacks(_packbuilder, &packbuilder_cancel_after_n_calls_cb, &calls_remaining));
+ err = seed_packbuilder();
+ if (!err)
+ err = git_packbuilder_write_buf(&buf, _packbuilder);
+
+ cl_assert_equal_i(GIT_EUSER, err);
+}
+void test_pack_cancel__cancel_after_add_0(void)
+{
+ fail_stage = GIT_PACKBUILDER_ADDING_OBJECTS;
+ test_cancel(0);
+}
+
+void test_pack_cancel__cancel_after_add_1(void)
+{
+ cl_skip();
+ fail_stage = GIT_PACKBUILDER_ADDING_OBJECTS;
+ test_cancel(1);
+}
+
+void test_pack_cancel__cancel_after_delta_0(void)
+{
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(0);
+}
+
+void test_pack_cancel__cancel_after_delta_1(void)
+{
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(1);
+}
+
+void test_pack_cancel__cancel_after_delta_0_threaded(void)
+{
+#ifdef GIT_THREADS
+ git_packbuilder_set_threads(_packbuilder, 8);
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(0);
+#else
+ cl_skip();
+#endif
+}
+
+void test_pack_cancel__cancel_after_delta_1_threaded(void)
+{
+#ifdef GIT_THREADS
+ git_packbuilder_set_threads(_packbuilder, 8);
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(1);
+#else
+ cl_skip();
+#endif
+}
+
+static int foreach_cb(void *buf, size_t len, void *payload)
+{
+ git_indexer *idx = (git_indexer *) payload;
+ cl_git_pass(git_indexer_append(idx, buf, len, &_stats));
+ return 0;
+}
+
+void test_pack_cancel__foreach(void)
+{
+ git_indexer *idx;
+
+ seed_packbuilder();
+
+#ifdef GIT_EXPERIMENTAL_SHA256
+ cl_git_pass(git_indexer_new(&idx, ".", GIT_OID_SHA1, NULL));
+#else
+ cl_git_pass(git_indexer_new(&idx, ".", 0, NULL, NULL));
+#endif
+
+ cl_git_pass(git_packbuilder_foreach(_packbuilder, foreach_cb, idx));
+ cl_git_pass(git_indexer_commit(idx, &_stats));
+ git_indexer_free(idx);
+}
+
+static int foreach_cancel_cb(void *buf, size_t len, void *payload)
+{
+ git_indexer *idx = (git_indexer *)payload;
+ cl_git_pass(git_indexer_append(idx, buf, len, &_stats));
+ return (_stats.total_objects > 2) ? -1111 : 0;
+}
+
+void test_pack_cancel__foreach_with_cancel(void)
+{
+ git_indexer *idx;
+
+ seed_packbuilder();
+
+#ifdef GIT_EXPERIMENTAL_SHA256
+ cl_git_pass(git_indexer_new(&idx, ".", GIT_OID_SHA1, NULL));
+#else
+ cl_git_pass(git_indexer_new(&idx, ".", 0, NULL, NULL));
+#endif
+
+ cl_git_fail_with(
+ git_packbuilder_foreach(_packbuilder, foreach_cancel_cb, idx), -1111);
+ git_indexer_free(idx);
+}
+
+void test_pack_cancel__keep_file_check(void)
+{
+ assert(!git_disable_pack_keep_file_checks);
+ cl_git_pass(git_libgit2_opts(GIT_OPT_DISABLE_PACK_KEEP_FILE_CHECKS, true));
+ assert(git_disable_pack_keep_file_checks);
+}
diff --git a/tests/resources/small.git/HEAD b/tests/resources/small.git/HEAD
new file mode 100644
index 0000000000000000000000000000000000000000..cb089cd89a7d7686d284d8761201649346b5aa1c
GIT binary patch
literal 23
ecmXR)O|w!cN=+-)&qz&7Db~+TEG|hc;sO9;xClW2
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/config b/tests/resources/small.git/config
new file mode 100644
index 0000000000000000000000000000000000000000..07d359d07cf1ed0c0074fdad71ffff5942f0adfa
GIT binary patch
literal 66
zcmaz}&M!)h<>D+#Eyyp<EXgmbOv^9IO)M!(Eh^5;&r`5fFyP`$%gjm5%}+@M@=A(I
MQ@J>k5{uv*03B5png9R*
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/description b/tests/resources/small.git/description
new file mode 100644
index 0000000000000000000000000000000000000000..498b267a8c7812490d6479839c5577eaaec79d62
GIT binary patch
literal 73
zcmWH|%S+5nO;IRHEyyp<EXgmbv{pz>$t+PQ$;d2LNXyJgRZve!Elw`VEGWs$&r??@
Q$yWgB0LrH#Y0~2Y0PnOK(EtDd
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/applypatch-msg.sample b/tests/resources/small.git/hooks/applypatch-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..dcbf8167fa503f96ff6a39c68409007eadc9b1f3
GIT binary patch
literal 535
zcmY+AX;Q;542A#a6e8^~FyI8r&I~hf2QJ{GO6(?HuvEG*+#R{4EI%zhfA8r{j%sh$
zHE~E-UtQd8{bq4@*S%jq3@bmxwQDXGv#o!N`o3AHMw3xD)hy0#>&E&zzl%vRffo<B
z)-H|+CWHZ~O*S%cfYx9;02_ohIA<Bg(1SxF-6OCb&_lBkf{t<AM9r;%E(Hf#h{|a@
z9>mqo=v6>_2NRa#TwDdYvTVQyueO*15Nlo%=#DXgC0bhF3vTa`LQGaO9;jeD$OP?~
za$G4Q{z+Q_{5V?5h;a-noM$P{<>Q~j4o7u%#P6^o^16{y*jU=-K8GYD_dUtdj4FSx
zSC0C!DvAnv%S!4d<Yg@O<;m`;oSw)=Fz+hrL<mY{rBr8j4pi^88FX3}jKrYUP)>gk
XB^)11aoGMJPCqWs%IS0YSv(eBT&%T6
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/commit-msg.sample b/tests/resources/small.git/hooks/commit-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..f3780f92349638ebe32f6baf24c7c3027675d7c9
GIT binary patch
literal 953
zcmaJ<TW`}a6n<`g#Ya}rZCaYOzy?UGG&Tf#VG`?}7@eHtB=MTqneFUS%75oLS;atz
zm&iUo=ey->y@-{3h^^Cx;#d0zEA@DDc$nY4ez&|=%jTg@_HU*ub=!!y$xW09TSjlj
z(`I@QCsM`!9&80$I98wsQ8yK#)Orb<8re8FjkKh630D$QUDwi~(gkX=RunYm$rDjk
zlp%RUSnzA#6yjdG5?T?2DcYKp+v_lts0ljn&bh3J0bD5@N@1UKZ190O6ZeWr-BuZ^
zWRebCX%(%=Xoj#(xYk1Cjtr!=tyBesf@m6}8zY6Ijbz9i9ziI_jG9Mv<Cz(ymp*>R
zDH*e>^ga9IR?2wrSrAVm;eButj4<aWB@zzNl|1Wp@4;}1O?MUF>Y>7(E2?b~jsu>&
zRKCJ7bp#19sqYh627wD%D9R$8=Ml$TNlumDypl~$jBu*G>5fIR^FB0h0Ex&TGZNr>
zL5hs1_K>taRb!|ThN9ns7^@4MXKP+6aGI_UK)T-M#rcP$;kN(Vcf#P)+5GzWa{l@J
z>-E{`$1iiNVYxq27}<DnwLRXQUG0o_hw&da-s5T#H=`Y9D_8=eTZ?cpWatp#a1vs@
z2BjrO)z@aTuI#g#`)oJcnhM7oYLT@~CHX@CNXv4>j;uo%;)r3kJI2xCFF~Ux;$Q%)
wjbk6JlDCM`jU&P+UVOvg`|iYl<7~9k>HHB4I;pdlQ=I-^$DrHaN$@lH1?P!0U;qFB
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/fsmonitor-watchman.sample b/tests/resources/small.git/hooks/fsmonitor-watchman.sample
new file mode 100755
index 0000000000000000000000000000000000000000..41184ebc318c159f51cd1ebe2290559805df89d8
GIT binary patch
literal 4777
zcmbtYYi}F368$Xwipg4lq(BeHMvzvH-4;n7DGJBPqq#tw3aed8+IU5-m)yvL>;Cqh
z8FFRGj$`9CA8ao<GaSz2oMCnz4Rv-gw9b@j_$0GQT1?XYi|;S??Y`Z6`Z;}C6#27N
z8ShS>J?j^$%==FV``-=rhLcPW`McSytRm~mEO7_&_cAVZrf1fFy*ha@8oe%*-aBYE
zcjzZg>LOkgxuUr-XJnHyD;zmPnRaSc#!k_P*d_BttRdc+J6G7za5#+<HG#rlmbrN~
z8DwU-3}VABEwM=0VLP@^Dy6ERR5_J6cmg|GEh*M1EliqCGwe^ZT-iZ$2Yc`4!I#WZ
z5nGGhn7*jeW=2ydsmfAmm#=8AD<<;TI+#z{Q)kW;yE!%GB6f~7EtEMLdM47Qaz*8=
zIObA(VVj-VG{Ax|66d*hi`+bRG>^Y1nkc2Oowk`ya47uUR3Feu?B<phm31&PQB<lt
zb{W(W4wf#Bab%|Q_tKPS?3^o=5)Z8^Vh(#slNI}pO(f^|{U0GZhLnycSaNd&h?CaC
z0XklU6^<ky6rES9T=na$M8P<_)aKMAMo+UDewAu4wF{#&6diFshiudixAoh|&0<BJ
zR>(w;S{(VYzxh}q-=#zP@uxSx{wbyPUMFU;K(06)$o{07&3yI?q{GqMcQ1c_^M<0<
zF4acAV)Il-V(rCTC1(;bsZ*}bl8dmejAk~yb`B}!^0;g^(o9kGUfZfDOvyp@x4OQt
zSgWh6T|3eq;9MFs8-#z+FDM1h(IjRUP|``PxupgJ7CUHOH90gbgl^2~97`?_X{P))
zB*$r1cDlF-%azKND}?Gv`2K8-9v5e`gQoft=j?T<&a13c^!wY_$D`5z-X1g?ty&6-
zQN50{8?bUk9AI->^W@~~nkOghHIC2YN+<JiT_ob7ttND1oh`HH28Y+OV~HedG&uB`
zy}rA*r_xT#bR`Q7-*)3t*(!Hf?jKzyxk=8hdi3L^d<p<uU9q_<4k&xEr4@YWv_vsW
zp(#32bYtA5u|s#1+}h`0kwpz4kUD&+>AXkLQG_2-{Pq3%{`3KUMeG$iIn%%^6*NYb
zn|_BdV#C)n4565Vcc<EWC-nglECZGy!j9I4&;hUCzr(?6TftH=0^@!mI^E@y5HZw8
ztH&kaSNyg=O6riqR^MPOX6oD__Jz@K;*icS)?m$!p{S$n;*EwlO<L!d7;utu(W9D!
zaF!B~xV^2i?wH0s?Lw%Q)(`aPkajs1XojlPv@Y-d5#YFg#KG+!r7A(dJLnkiJMs`p
zV|_=d!upN{TsxH1?sZNdzxeHsmtzTV`1{pykJ_~+^*>X;uT8&z3vSi!HXGbUj2B!R
zdz~&#<?<tHJql=W&((YpzS06y-Z6Cn^H!*9qH?pOrD~(GV=JJ~z{tpLnGK|xp&C1`
zsbn7O86xjF<~G*S1l*;;Bh%6><Up=oKy99?62P^?M&22U6QFRBXLb&u%=Ur<74wRy
zMRG!f{AvZ>fk#L-&k$fLwo$4?>12g@AXOKFekuo#6EHB%gmpD?1eyh%N8s{2wGoTu
z*@6cEZ^ZW!FAF_|JL`NkV7k}0ow|-2jHwbgH0;c@Dq*o?@&c*HnGdyx6^su8Qk%2{
z*ye(dxO*6-&>qn1+zw}tc6;=sOX{4WB=VqjTS^))y1jlX2Q;=e!qMmFA5lC$#;BxC
z=Y%tRpWxb+_uQAvAw7Q{HGV#R$xb&udLCzZ+HN?kTyB};1EJ8UlQ5!>5eGW@)RX0n
zkjj>EF!3=0Gl^8dzv$B^NMGRxJoqN4A`xq-@wCbrx*u2NmIJ1<fUDQ=*^)h6`vzco
z3F+ro$F!F6pc<B;<;isobIgbVGKUBByoQ4#CO({S7g?<Dh0^!7uJ3gxS=6F;+^gQc
zeKi4`4`Fm3p|BU2v{M|-u!#QGCXiM=k=%np0<ZOPQqEjP_nneyOdgEuG9IH&YYPtp
zKB_dvcYCcyhyT#<uhUEL$o~!TUz;cb&|`uSM{Dkv%&W2lcpYL&kv)tUvVv?&>xZ%H
zh;{|4T3(!E9sY#Ni(wUJYs1MmIc9bl)(4Nl3_wD_BWB>i<1S(LX7m*{Q7PU$muMS*
zM!%0EZx-Vw=Zey;erC?SNxF;pY@^A%-krqzfLV2meBp1vWdyArFYn`DD19T)Hw(?n
z)}{NP(Lk(o*?gl#B@pP7^*r|=;PIDT4|F#{2Hzh-AL0Rv$6uT;<CP7qxbYms@WU7}
z%}TsHJ!V_56ZFF{BfI=8jTBxMEATG376pS6a;u1@c?{~sL<N52`U6fuLkz4P@Mb^b
z2`Q48y&!C0&A+WRCJAdmo3u2#?eI=si9Vm47$|`m**x<wKkM=QR&g?C63@P5X@xP8
zi5QG2b-Fdz%S0%1{kKvL%^RTdpK|SU^VJ7JCmKdwp1`;HRoGM7ef^k_s_}2An=cql
za|{IaxQWpdq<ae&x3BOJq+M5QNIk#C{Nv@u>n|WzE4=slK?on@(fZeGhRgQCu56qB
z{+n81Az96qnQjMY*-*r-KV*7;Z#4Q<xfjbcBx_6JAN-HP@bq+eI%HhAB9&vLyOap{
bw<Ywj(b#kdcCk7dCBY;|DBNpPjIa1F6*dXN
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/post-update.sample b/tests/resources/small.git/hooks/post-update.sample
new file mode 100755
index 0000000000000000000000000000000000000000..9f95e41a39cf974958277c8881ac6cce405ebb20
GIT binary patch
literal 246
zcmXZVO?HDY3<Ti4Poaiwjq^yGw9DKf7mz^&Ly=V5q$H=W^Rt|p_r9s#9Ea7VERo!9
zyT9>uJRJJV$M^KdldiMhj?ImK6~FvwJ*L5a){QoM=L5TYHkGO1$UrO3`a>{?Opw|b
zG(#59NQ#jFL9v~vgOVkM@^^(^A}onOE))yWEwhIlk&{ZyseZ^O0b=w8&O=BK{k<5B
k^Q-B@eG}LeHrquz%(SVEp_N)VhYZikCW__82JXfD17`J9Qvd(}
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-applypatch.sample b/tests/resources/small.git/hooks/pre-applypatch.sample
new file mode 100755
index 0000000000000000000000000000000000000000..625837e25f91421b8809a097f4a3103dd387ef31
GIT binary patch
literal 481
zcmY+ATTa6;5Jms9iouO45IBJXEg&Jm9@v1LPHMM_ZR|;#6tQ<EeSrA%_2`_rGr1_8
z?aM?yVtIc%-@9SGSk&8x=grP-Lf`7!^=$7xgL=|ysZ}!av6zL~ywui}<2##V6L@!k
zy=p^)V7%Wzs-g`9<Y9}^)&uN}BCrXR_T3@Z2$gSJON2`X=mAs+%@7n-2I}ZrP|TFA
zvJJGDl3HPLP<@!Q!}zXQvey#qEE#a#$vs97i4=A0stF@YQ)k_ZajaoS^dVYBc&37_
zVI(L=X<V335r9~7T<;|HfKF+yM}}LB9d96V)Si;sj(;9Rh$#P>h$71hSXq*MxP;V&
zj0cY7SCL=x4`a46sF)C>94Gk%=3q$W2s;j6iHtB2$R0%gix4oK@&T~=ALd_o*CKxt
I-`Pv{1Bpzc>;M1&
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-commit.sample b/tests/resources/small.git/hooks/pre-commit.sample
new file mode 100755
index 0000000000000000000000000000000000000000..10b39b2e26981b8f87ea424e735ef87359066dbb
GIT binary patch
literal 1706
zcmZuxU2ohr5PY_N#pZ0-F<{-<Zkx1j&cMNSQ3FN`GzR*R1O+9oPV`BnOj7q@1pV!u
zrF0Hl^i33(yR)-1d-!H%&2|=|^E~_R{N1zNJ-&Zmt-t?iwXv&i+ZN}Km(TX8Q$H4u
zd7(m`|1iDmF5k@xV`p;C4zojASmLc}yN0QDZbhN=ri&CEt=XGuN1IwjGJ#a#`t-kG
zDqY)}7+Ft|;YKwLYbtg$S(-TBO=x3cP1cd}%f4kB!<6Wu-dCwz-)KDMEuM^_Hh*UC
zC`1)|)T<(U6b`+yOH!6p*Ll}@qastwA*dyjsgOf5C=?LTprfORG6O{5L%@S0wyHpj
zu|_A-=NWnYYR5m7kvm6|&T~GzoJ_OKR3sgFUgw?ifho^NQhvK#{6g0=&Fh)%n}#m0
zk1sNmwb_AMDq};OOGw5|;OyX#?yQMMH6yAk(x$3tjFjHE?c$E2XC_xXav8tnIeIG?
zYMI|~MLEVGkuT*>v&v-X^RA+u>k}E$4d&uD7=g_fA8+pNNV=4s0|iD3p<=DTXClTS
zXV23tJ;ECmN@M0j@zUAKEYW@3bv!SeYZ8ZH`YQNTApFVNc;F|9r5p4TqGs=>8E?6y
zi|gY{iM#PG1nL?UE9YCnWTk72kgZPG*Usqw!~Qd3c?~@w2?%eg@~)+VlSs6N5Yf2^
zz;ow<fjf3n`imj7u5lnzt||q9QFM(H@<3EJCK|l5P!$yDfn~U-(5Vu7s+GqxNKyeI
zP=-Oa(KH&gK`NhUa`cLj3B8%qL};DR7dk!`A^h&3-hFB6p($5Ufy^tGir)3et}qK4
zpkPKYWzC+?=&gw-L6S)S=<lfq)%uLUAa%~81Jf9hk)w~U!DItnoSy`m^}#38M}K-o
z!PqisQkV!&z4E*V01ro~qlVK^0BI`pLk6D+)f~*y!hCvwHq8zY9BGh<2s$@b^A<8G
zRaqk}&qZyyv&|0QDFPA%h4TgF&vdlc|JUq*=>F#K#r^&KMq1A`oqVGFpD&-!Pv|Rc
zO3KSqA@h9nSc%bm`0)Amk6*J}@14J*1-219l%%7D!Pl}UK>|lVi0Dfgu2jN3WC!uL
z0ej??b2iSehVgdnWHmZV4kUo*QL#aiIp}U=9x)IXk}JJ7VQ;CI9Rtn5e0VcjbY<bp
zv{}WlG6L;H!EzFKdY>cVt+`x5D+svCGD<sXw4|)E|JX43I1_3P(sI4{wj87bPSrNG
w!SIr>;Z5hm*<gY+X;)Ryx4=nzaab9m`bwE*^s(%u*E3HbUuOR@+&s_z1=MCi2LJ#7
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-merge-commit.sample b/tests/resources/small.git/hooks/pre-merge-commit.sample
new file mode 100755
index 0000000000000000000000000000000000000000..d8e8ad4bcf8337f23c78a98537ecab4831301202
GIT binary patch
literal 473
zcmaJ-O;f@!5WV+TJd4B0whSt$H%Dh2t`2u6q1!glCNbGU;n%yxdNsG~zR#WA6xIwy
zWEZHoU#u?nykD=Y<HPgeWDkDm^kTof*l(|%^gh!nHrZpo^vhMDjV;E1GD~K7wV*+D
zz9lry9T0cHcm_KhDVXYvQ==FrLTT4u=bEr{U1yl7%thf%wJnv<XQZ`ZbQEezaWdS%
zI;c?h9a)Y!ux<WK8rQd_aA^?61hv_Pf<t7*z1USuL40FxYz<|hybsO?qnN}aMpcuf
z6phFw1%Xx=wUk(m>E$jSEQZ%SQ(}oLgslTvrKK@9Qf#b!hajVFnp9@oIix;NcI9Wk
xjnh0ya!AWet{I7YpD;y6HXyzI*lfSvH=o6*7mJZPkuaYpm>vzZ`wyGEBtOQPo|pgt
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-push.sample b/tests/resources/small.git/hooks/pre-push.sample
new file mode 100755
index 0000000000000000000000000000000000000000..02cbd80c287f959fe33975bb66c56293e3f5b396
GIT binary patch
literal 1431
zcmaJ>U60!~5PUX&#a1@z9B{IIZkjLT0t5kq9#8~D(I5{+8&J~9;#ndUk~-ZT`r|uG
z$#K$$J{TsK<m~Lsu9iP+t-0TZ=sa(K+C6);54X>s*LP1}9!GoZ@4I4myMMG_di|of
z%?llx{O8TS-#^<H#%^V=)RNv>;(OioEmPy%kwWQBA1OMzV{hsQ8XFzS1k!~YQoLa5
zhtP1fA$q6VmMbbAC_9)4I628k*O5J$NR19uHe4QYDK<==I~SQk)Nu%xQ~<Hy8U>KH
z53w=!ke(FGb_PpnZfd*+hnXDTn;2*`u^~;?+5C~cn?bRka7NR%06%e6O91{MAgN6J
zmlO8{Biw4&wr&&(z4p3eln`E}XR9m9bNYZ7Ibrg(4yZIXrfgD7N*AFD7L3YSM#j}%
zo__rOS5fr;@8UM<6cl+cv_$YB$PQ&9dv($eM*))g!_cu!QcSh-mqE9i#QDZT)=o#`
z?8!RtE?w6p?GkGZ-6yt_p~5~4ecu|Sf^)6096%h*q-eNiEA1;Xwg)p~Q&iGSG7-IQ
z9aII&`ps$WOojFA`*bjG<mBv1n0hcYZWN0~(X01-hx(ExqWqaXgSk*@-GMp|K_3!5
z9|O21N3%~izh(4fbp9wzd+!b&7cVwSP5H00)m5ej-(s=Pl#(90UOhn@OA9u+D{i@r
za4*CP0I#<d-)-#xq5q-iY5nIef2s5OuQjcA>kFk|E@sHHuD}W^d`7YJ3YE^zrQnqR
zGoq?;YGKe)93o|_=^f%3U1KYZGPOXRRxK7w`UUbMMa3<86OmVH!EKP$8RCrn9mWX+
zC?9yF!fRVLmud3hF<}x;;sR}f(*r}6Gap3fR6zLHR~kbMgD{98N`L+r&?3p~*0+FX
zcAL%j=(SO}xTJUTvA`&Lf`2mv4koPG9&|<CA~EHbWHMoFPwT(&U=7t0`RoFZPO9Kq
zwwe$i=T|AXY#hD$aZlNMH`wZ%gwilGJ(zeYpOn*F3cy0XKXio^Sj#WXx=PWV`WGaA
B&~*R+
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-rebase.sample b/tests/resources/small.git/hooks/pre-rebase.sample
new file mode 100755
index 0000000000000000000000000000000000000000..193ee3f127387f469894675f0d621b5bb9c275da
GIT binary patch
literal 5006
zcmb7IX>;2+68$XxiXKL@ma;l5d2^5Ba_rPh_DHI-u1#&_upttZXp;no03$20|NFiM
zK#D#xQ>!Z3JkX8T-LDVm!B5j7y_{;JDmmTTef+K1oIiPzeEr+Ai*<2PUgnG4^ZB>p
z_fkAvoR1emuf~ri^K$-px=4#D-v<wZ2Xv&$O_eTJh6d4)=DWL(NBs9G{k<+yMMw0T
z$VH*-+LM)}u&m^`l8~1nt(3Z;R8v(KbY5#i3z+~8h0D}Xvq&3J8BMWDizPNpaeb~9
zBN9bSkthfXzskapf%Zt{*e#}{QaNiaAVZ4{$;;I6<vKNhO@%7P-(;l-x=pPoExHC!
zB(hA#cDdD?s4P=!)=-K{<kHAWKetl-8I8wwO<ihJNs-$dEvr;&S_@6E=mNSJ5;mg#
zyb)MbqKH<one{qrV;ZQ6WL}yLtyi*ekNLf|uC6M!)Cmq7*l?g0d6`MlE49|}>Y9w&
z`bCv#<YfTKtb`!}CyNYd;|(C?vRVQmWOfR9X?FZ#=f$No)^#4>2zVn=YnJyeNey(Y
zRh`9vtLw~A+5zsjp|W0Nsa|29Rm!B>OoG5a+vi;ari8O>KkU!KAWg_fa3btK2x*_@
z0bEc7J;Ubghm}n9bOi(Sv_B66nQ7U)J7f0fO}<cB8i8vG{r39s_>8Wuf*uorcIgEG
zOHc|-V6+HlRhOP}?Cn?@5iwSl43abmBA^2lyL$+cpabCGVES+v^j^FO_}?FIp<qP?
z#_?*YMHIkyZxJxS;h@A)WDJ0bN&+F-#_lFjAf^g_Pb#5YXqYe|dZUpa^zI)VOBXQQ
zAMhT>%En%Ll?Z*7*}TwrZyg5OSZ9rY-`aU~Mc-jjv{Ll)FLMgtB4ujktfQ`Xhqrka
zT=P!A;9w^;Z?PqpLwOLu=cj3L>TdUKw2;DMu)`oVkj}<z_EjO_2uWYuvKG==%Zu?h
zJiMVR^c30RbpW}<+z;jjoNC}YI4f6QC7d-08*4mCB1>#bcDx4tYg=j%D`+i{W~fVM
zVmZ>W9VMyin9c-0KzI_;iZ-g|OyzuG`Yq%(%dvl;ifnVr0;jWE&S`z|rQu=!yHBBO
zx`OJ;oOQ(KKM<$(bC38o>pD0%|HA(E0TRw7qj$fJ_pRN+7Nm>dS<q{AcOz#-;d7_2
zL$z(_nhH{vOzT(}>C(gLg{(`t+5Z=?o+}wXU4tHy+&%F&aRhFebeEhR2R5|<c6J);
zEY(q5F5<n*XP0|=PtPBn$B)V~d$Os-?&8UlaVe_|jdkzoWNsTP-_uyq4$R6o<h`&@
z{loXa{^#TF=NJBYu9qB?hs}x=It_O}ZjX(_wy9@X(lmkRpNi0{8T{O_b_j*JC^_SM
zz3G?1$KCNWF-|`Jbx2cQ-y5LW?Z2eikngTZmsx5C(@({8<l)Ue+gIp##Mosfa~iZN
zZ|NLN9uE6Xaqpwk+@D+f?$tg2JRCY`pwZwbR9OvEn*zYm`ffKIzl4{r{ZgjfpbuX)
z_r0=0y3)T-j$gljPyEJO*6Y<pj7G72aLoqaTpc=#u)*xJcVUm0;k(n;r)^DQNa6Oj
zWvlHEGg~lz`Q_8`yQ9<BZ;ylE1Z}bD<8}<uB9Y5lRB=;JUD0h?_)4H&EhJjvwzJx?
zr<o_vk&75j_5^d$8Z$_Oc1=R-I_DlNZ2@~81oV*J6%o3RuiCz}^VEW>$#Ycbp^w@t
zTl%=f1t=w+WpJzF<|CE@?SCNAz)%9?w33lQ8vrHJqPfH9@}qs*QXOG71W=ylx;wOB
zcx!Bj^)Yy6WX$a^vBkBJ5Cob<oubBW+a#9bX{0bkMTX_2sIrs`HMk@$q{dK5h2-g}
z({`~#gm#G?c#>qlaDx_B0c<3b+8)f84LCrt;e;qxc+7>VbwVK{skNv!wvBiTa^9Iu
zkwP;VK)jH$WJ{`MRwAA9fal!y0dtV;FWg8PTkWU>CwnqD>1ZX2B@;$DlX%C5MI+}{
z9xQVnffR*~v2KAUj*hCd<gRCjR7~6Q(vF%VT7j97iv3j4Z0p%mU`7t061~l7b$!#t
z3*Pveii}aQ?QD9aiX>gul~`bk#mk`o>zk9)<2Uc8?hUZAEvd!`9em)~$Z)zev>w^8
zyAgCP_$&Y)7HSQ84`xG}OeTavaEswwF|8Xpi5iZzZa@hCiv(J-%bfFC&)HLlO+Rhw
zG6g?9eL5&A!SuJnQ6}LxG%tU+@vZ`i+!+Rz6iYvsTdhnPo7lW{m-}{hya@viX4)XZ
zngaw+j;gloB#|UwI@8sOmQpc`h+bicQJnQIB5eifIMQNgD2+oai33m!34~xU|0Azj
zhu$8z+T5^;Pxx@d{N)pzOJLSa^e;aDf$W%N5XcOf!mGC9l9j$Ev2h6N+6ZQC+CJzl
zaM7?S!SrFLS2DASjj(h6y1WN3N?|bmqmyzm!&nLoE|`rKBOc_yDF$a#FsUn!IQf(t
zdC&Us(kQz*7mv<H12p@UD8XaLXdK{>H^j*^MC@>wTDb}g%~sx*ng#>{@lR=XG-Z5_
z#<9*Oh0joMzt;nS)ObAp)347`D=}r-;nV!TbIq&xrGRGsF6fZg+!VkfUei@_&l-M&
zPqQ+Dw)RV}+)I8RuqAxa`Pv8e&!_gXS=e2-un>=Ktn}-;%lLZxaVn?Q>yZCb2R3Wk
z77zr%;Rq&h|2ncqyKYmFI0148JVY7Q$V5p=dWj<MQ<8r+@QLq+UROk&%quICq^O2C
zp(17882jXI)_GaqzAY4AjoB_nh8k*r1mJ>+Qqpu%i|xp2<qF`Tw6&3`h654ceBjZ7
z4>C=WaOb2Wudn^h0EcD%$p9YVU1fnoRV9`(cy(vv6K>FXS!2jY>1GnU--7)4usH&K
zao*&P^@9~YmUe|ZdLW@C>H;!*<TIU}-!Tw#3oqZA*99}b3&uHiGO<{I6!pMnAiQdS
P!fA@Yk4s_DjDP<F98V`a
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-receive.sample b/tests/resources/small.git/hooks/pre-receive.sample
new file mode 100755
index 0000000000000000000000000000000000000000..32b597f8c32399017c3d7d51134d1f6490ad9575
GIT binary patch
literal 601
zcmZ`#+iu%141Kn~f>Vt3>Nw4M*;=?j(TBD#O@XCv0|MEhA;z}kTFRv@`tPHhp=&Yh
zg%Zhg4i7o_k{a5i&f5;tZ==%}^Sn4aD_6%qs<o-wO_Prn;}`SPs_*$C$(7T|$#C3`
zPt%-C8gelZ1GqAP8`ZQmg0{8-S9H{R@D>_XAuJt&EumdH4Yu`UjT<s+s_~uXh}qA8
zg|_HG)%7Pdc&$7*uR0HF@)~vmFjqyD?XZwCbLen^h5t)sm9<90Oa!@Y%8!~rF8G?W
zkzmCF8kMtuuelMHIAlqqnm?72LeGM1J4`w(kX9&%LQn}ForlDLjBoCyvxmo@x3kH^
z^loxLyPiDWPo-cBMnsg2M6}kuPGHEGBqVkC{D&9Kt%xFAsTw4QC1$_=fwBl=3dI+e
zaSxI}JS}=Z(Ec80eF`!Zq3mqapXI|UN!a)t;@4hcu%Eq2xcoW}%!><-+XHTuHss+b
YOmM2;hq8Egm*4=7_P9T{21QBYH*F=mfB*mh
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/prepare-commit-msg.sample b/tests/resources/small.git/hooks/prepare-commit-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..b1970da1b6d3f42f00069fd17c325de72cda812e
GIT binary patch
literal 1702
zcmb_cTW{Mo6n>t6#i?x6xmZ$SFLf{QfG*3r0L?Pg?px55l8$UTGO3bO;spKi{V3XX
z))weX0X>M9bNMcZ-6yG%>(n}JI2|25dr<ew@wmMG{l(3lx~bQz>}WZBP@ih?JX^+@
zu#5O48P>yRX(m<b*PU*sORp92TCD1dX`%HE+1$w5k<(Ngu7zQ83#MGJR?<<W=d@yL
z#heqwo{FmCg0g#x<~R+PBD#}q(MBn;V$x;%UrJPP3*l%XtlvTWChI2SfJ$9e`YvSj
zRSOQ?NUgSMLI`3vL48YBHzwzVXoe7v0ef|0YHgV$N@?N(-R)rPqRDrK$n(%+OF$`P
zWdjC5N~`#RjV9}aYwQ4_yF5O-$h2`>fDIhYP)doc1&TADZa@ZGpusJ$6G+e$ZMcmC
zoOosDQPS}l{H?YPsq(4;0SGkATa9eeqAaDcj<jNAU+LTSmM1jo(ti~T0B7acJnnTX
zTarYy;Husdxal0!S<ba8=uu&a*SNYtr7|d7$g-q3_JHER2*oBsVW~i~XXdMx%LW~0
zT*960LF<qZ6K&FZ;vGmtyywBU3^Q>q8n2wALbFwU@2i@FAaRV!=uw-nwx1gKn2SvY
z>Ff>;2sg!+Hxfkwv1lsiii=p6WenF=5)6LZc<a$zDCD$GRuwvG4Fr+B#h?#9gTbio
zk#MdxC@WY%zSGN#i}Ts_#q`bf-{)`7CcWeB*7WlIyHjioJJWw&A5VItPUq3^9!r}S
zbykelFV-VFvcr>QaZ=aS_}+-4Y&?!@HWh|<^gJ21!|T@+%On#w6azxPHV}XsRbe*w
zR_TZ2XEsQa1lPK~biYqg@0-RW@5J1@=<87cFzEUABdCoFH2CZo?}l(Z*!OFqUxo>K
z_d`l#4d9|H6;VPT{X?^{VJ>oL|D7K{BJwwqB>`YcPoGk+9hbvHnoQ{EM|kPgD_`wk
zKm4#2xu;-y`RAm!=L_BnLvJ8$AZm8@?)v<%vwvsw8AF2x6!mTT;c72A_~U9nIq0ST
zv)N0!I!^1p=g8-RQfx5)E_Mb_4I2vtQpI30XZ&t<!9D6nI|;V7YR3)l6=S~QhuwLQ
g$e&^kTY-M99*<-Iw@(78*M5W!4}U}|8YyMx3->-9h5!Hn
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/push-to-checkout.sample b/tests/resources/small.git/hooks/push-to-checkout.sample
new file mode 100755
index 0000000000000000000000000000000000000000..a80611e18896f212c390d845e49a3f6d5693b41d
GIT binary patch
literal 2840
zcmai0U31$u5PXh)#YOS7cE^-rw@uolNhe9&aUS|HtvhX>G$45tVUYj>fRdF?|9kfU
zNR~aG=E)WbEbeyq7JTw}ZuHIE2kUtL<<n;$&G!2F^Je|kx2ug=4L5!H^!ogx`7o$&
z%Il(3zAe6<oe$^F=A|}s`8}CDp*M#3M)gC-)LOeDUpYMl3YNy9R)I-T)pE7sy09aj
zJ7%&5PnSB-F#2{jc><WLR{I2izuK%VHc+{hRfXe<^_q)8RjcE(6WX+F2)iAtDtI{x
ztAHVBq)eSp_E^xcV^i_5KLIHA$g{zEjh?rsacu+(Eyvve2~Kmw%;n3g(kWB56j~Js
z<yE5tYUsAR&PY0wgRvM8x!zgLX8SI!eVY&}YZ|>AoeCNptd-NM1aZLhESzC;I`+Ns
zfmNNjdAp^W8#Q*}l>CT7RB9F5(BbI8ly2l~+E};JW|>&d1)=epZ-8vm8ppkbEVn#R
zt30a5A-c(YQR8eM5%;|UAnO>rt!&@x@G@yp+92%w-}%(5P_+P&Wf_zb$f-Qrl5(7z
z2ah(bkE;!DK(&aAMuQ%1TS>ai?wSXCOCSj=_}8x4IbCx^$}9q)<W{Y<9o<WqZo^oV
z3bR;Qa%VT-M~kU@2n{=+=)C!MD`12;@<F*EtPfV3K#g^1%&ggH?4{m<R$WEKcUI4%
zl6{ik6Bo46pmNjdXg5@?hn;SjG$}rr3Gy#-BGgVD$8oD)OcK(oqca)L_kk)UBZ_&6
z*ourb#Yc8l3J+uSda_Y$GY--5Zp3QK9w^?P%E0xb57?fY+Q#+KU4)+R>whwv)SBt|
zg#MX4;;Oau`m=MI9(^&zPbueY@~>3*ixX%mvR5m_1&nAg@ZKvY1E$O}&EtLiG;mhV
z1xhMIm~fGjmf_#{62f`y;09?I7M1W2tWQvz<}i9lR>OpQyUJi45_&*pQus&EkwY<>
zI|ZAx=*3i9a-)g)hXkvO7>UJ5MNgL(Z+-wpXVcgbSgpmFmbf1~DPA(OVGI&FNLeIE
zNH!_aiH$vsif$_j7=T2{cS(!DOI`~bn@)vSd-0d7xL=DF;UNP|tW}4i<qWTSNCe|y
zg9kV&L9g;FhC@tvsVu#V-brqShHy2qZpA!gg{ZTY>h>DvHtu9tY_pbJ6x(6E*hxgC
zzNDao%qlr-IE%YGbS4hF!n!on7#W3$bX-_hbZAaws^nHu#)Dx=WzdbJ>AKzAy@T$x
zSWE^x9+|TEHVEPyaPYa0DOChp?AeHSBBDbZNokQpAY{lE!7geZI=jV)G^2@<iI(N4
zJLJjy@Y<VI$yq;3bVpJICW3D?YDMDl4Oe5pDf{d`i1_3Qx%4uX`$dOz<9e}jm2B-u
z8-?%!7XuiNF2O&MAdl-iw{drG+$F^zT2Z7WRV#c6;IRZEf>l)&91Zb1+`T+oq9wWF
zRV~kGTGce0O~p^6mj{kT5kL(pv>r;Lvd7VDX*P>A^Th`$3cWO<svk?_?FeP@458*2
zA1PqUOdd%VP5&4~ocLK16{1GLll64c=mU81RMFstpnMoLuI7hNIE4N)U%h*#<Fz{C
z9#>0<l7lRrls|W(8=W1O80P~6+U7<4BqC}N4-4GR3w#{O7YmH?JxwJfru2d?e){$5
z@5R+`7Z;1)FW;OkE-(HPisCS;5F4O^Q>L81p4Ysdo3ZP1(SrR-peEdTo;-@bkB((G
zPHYQXUL!@Q$e(OQ;R9r%@Afz+50I7>*^^c&&|E*r-jN)LH=pM4AqMwWxSv|nqjddE
Z4{_hwv8!W(<d3>T<BC%y-6O5i(|^OS%`gA}
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/sendemail-validate.sample b/tests/resources/small.git/hooks/sendemail-validate.sample
new file mode 100755
index 0000000000000000000000000000000000000000..effa6f54f2a5c1f0cebe258bf4805904be4b4852
GIT binary patch
literal 2365
zcmb_dU2oe)5PUX&#g-Nv2{5JDX_MA~3%Iq?8g*kpRgpYZIFU=~BI=I5J6e{T{`bz^
zk+$48h#x9Ika!=nv$M0y{clD}-j1x(hDWbnzP?l2k8j?TH{brS+Nf21yPm)Nczma>
zYw`X3V>TCdnSD1ru8&`j=2DIPbCT@SnIgUw>$+lEYP}+x8(BMYnr=iT3*ndq)xzaV
z>I+qjv}vC#8_9M+b1p#uNS0M0)q<p>8!3p_LRQ0MA3M`!2foxzRUjbFY@}O~(ki=S
zqscnq8cU*dY)D$$cqE}n)V0yIk>CNKHCrndOtSP*HbOb;nbwAHSb;R+gs^?^Dve%)
zoW}t(*D}$>O3ab0TS^-;J|u&sb-PkZzo#kn*#xYt(;<xzKW(YtQZ$u23?yV#kyh1~
z@+Idh;EG5jXx8@%<;Y_W8SA=|T;MPQ)TB!!<QcbU)YR4)79eeeg4|vp-8jm%Dl3^I
zRS7+i3>FGuwzSb^g&RDiGcOz9TB;Hu`nJh)$W=C=XCSm2AY=$w3G3P-V#Oo+N*;#2
z4ijJ-pBZ=;T(RTgp_HYrD!uW-dTMfkuqY5jwOy)~gM;#=P^i{!l7`pXTS^s(&^{RU
zydaw}OpS#^D1cXM8?FW+fh`t7D(g;yr6|}fdaNtZBx3hlK~IpkTu3!Qq%R+zAo#<L
zU&m+XIFB0>t}Bs8^3$vHD+-TGT@`F>H1Cc#WAVW;&$S6%fE2d6@kLS0g&ihIM{}0z
z8#XhD>b>3{(BH|Px7}&lJ4%y1v<t$W+!F|W@<gaU4;MqSHKR(wdML<XnCv;zaPrSi
zxVCvek26-bf#Q!H+uAgy_{e_1UZCq>(CihZJx@8MPoGdl*BJGD;usf*iS7%;{Joe;
zNFuBa>*~o&qETDPo~u&~$FxE1xb^x&(CbE`Y3GfsibL2rl+L;>P6j&Y3U>K$mkp*6
zd`Q{<^+^&;GskGjwD-%!boR&i-TC<Uvy02w+l$Nb?B}aL-%ZDpluqd=K_@}z*fyuV
zzAs1Hf?3v$k!X7GTc8Q=r`WJ_Uu=>A9UOR|@=GYb5x#<f&WSMH%mCJU<#=7=qFdL6
zG?Wz&6z&J<@I(B>+dhd7fkaVIR^pol`Mv+rUbmZ43dVL6^S7g3{NsPiG$iy$5EDB%
z6KIgnb$H(n&t3e4E6d4V7w^B?JS}JkG)PM6+X3Co`SQs($O*AA+MG~{S7RJ=cy-l&
z>~%3y`tjfx2>uOu<lDGWewcb=oL@}B@B6FCZ?oxSJWldrm%UfOduahk%C0H>tB_^s
ziwG=e=ch|FQ0IkN91US7rhdQkXhwwt$gU0WEVDjo=IPb+?6PC=s8}J*ua(Ms))`UL
fi$|vMHn?H<rred|1&u#kOoJ`%vx)-*g-ZSfgGvdP
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/update.sample b/tests/resources/small.git/hooks/update.sample
new file mode 100755
index 0000000000000000000000000000000000000000..c3430f6fe27e572a1cebdb2f2bec70add0642a60
GIT binary patch
literal 3707
zcmb_fU2oeq6n(aS#jRq*br9K3y0z;^QgkUc^kKlTrB6js&@yE)mPu8l<0$L?`wmIT
zrsXba))@gJi@e|G+<SfSXe`CeSQ}OG@sr8ZTUlQ{dzM}Q@O-hBi}GeUom`#X%FiYH
zX?m4Rna-0RN2lfK)A3ZuvHcz$L<jUn62D=~vfz{}wIH2VqBLX_O$(JSXeF7H$}q!c
zWY}C&R;eX%X?P{%d;|>_tSE3ettp-hLlsZCxaLX8(nU;bVRB;Ce6@s#eu2|WvLz>-
zvy(&>Gyfp@+BtKnpqWkKi^+v{4jn_pNw_zeuxE<mRXKx8G3;9pl+45&4~hHW!G@wo
za7?X(0B}HbX*ExkDmas*xzV)FxygC8AL?2Z1x-0QJqS@qn8sD7r{bm30@<%eL_gOw
z;~85O$Xw2AS}Qp)5ViRUe3|ir8;&&I<B7Y6^!kkNyYXF4EY(b8_5DsTYn_&?wkdEz
z0y$tADo<&}nGs5kg2-J=0KlEGPb(%<An(pXY{K`qIZCuwiT|2{8JD&5o_~`o6<;dD
zi@J#zCE4={8j%<uy|iutvHu1QKo5Tno-BAF2FwD%%O#UDDum=w!;!PNe-cOFNX4)5
zd>TifiGO|)w}OANj2n2D^K=o3j6P6uOL70#cbA{uzWXDlk1wr9GV1X(2W{RuTvjXV
zCmd<W?kH^?PXjkbF`XZtwu1B+%4@ZvHIwGpJ*8@8`MWAhq^B|Hj1lzj3R8bVuMpNb
zz4Gzk!3T3bY;WEGIww&kq9BYW6EP*q$K|EB-@TH(Fjtz*`HMTO?i+3E;5tdSah&xZ
z+t!x4K7)dpy5wiJhlJz~8qF|r8a&-SV7^I3C@_q=P`yt@_x_F-;PQR)fzP<zNN>8u
zH%V`94=q3)Dk)PHNrnFC(T1)Om6f{Usj;u1R->&XoCYVK2V3ZlgZuF?N}1+33<P7e
z<0yVF?Qoa{l#7q(3&jv=Ab)gpM8A7`p%3InNzSxy)ZER2U0C#9zKpnLY0I?>OER*x
z*9Z=L=zI8CN>A_^jYjt0F$psO$sL=38q5q|SG)qCN6{^>RFh5E&l5GZ$pEahnF&d+
z5c>64t}uJPkf~_!VUj#&N%nC-gUMj%=@B=!V>&}xtj2%@-mOm#rQUSJ3(ccmc+fza
znZ#uxF>N?QN5UrIEd!5RgHEf<eGg}P45aAs(Xs6u!XW9r1I*E6XJ^1movX@xYLuPz
z|7xBN4z@b}#x>W#;(nKYF+D<*rdshJ$X-z2OZ2X;)nn@KSVdVhaA?}@3;6gZxb4<W
z`9sa`0lR_azMX|=t_(FvG@%w2);9P}SG0u&K1(*oX3};~=<<!N*F$UTSxD{V&6mgL
ztw9Ntc2eOF@c!OJytNC4T^#)Ien7-`dI{6s#co~0f^E3J<0Ty)l3xq2u@Y8DXTK>v
zozoWSr{{+!h}zGpumG3H`=AvWpm^9kW;J$Jp^Xl*?8ckr`fqN%c|Z;VC0|cM4vSrk
zH_O8Yvh85nvJp^;``wo8=z0f`FWg?`>gO#y1hjX1{}rTlg9rwIKia8eyGexA3GnuR
z`Rg~XZoW;0pA)vI8=p5!+6sIn#C^FCvR>ffv39h6SCNi9v);%WD;WZ`of_MgwyRWy
z-yY%n*Y>X8<Sf+RyB|Sr2YG@1w~%U$o`(5EDkJ|3$u=eMZA&_wxEsy<Xxh2k^tP?4
RGx&ZHQs^8zuIpu!=pQhfp8Eg*
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/info/exclude b/tests/resources/small.git/info/exclude
new file mode 100644
index 0000000000000000000000000000000000000000..a5196d1be8fb59edf8062bef36d3a602e0812139
GIT binary patch
literal 240
zcmXYqK?=e!6h!x)VxWtv*mf_t5?px$aS_{}Hj?C*<cHXeXE&AZhT*-L3ZoI&*l1%Z
zqG?zr3TvQGZ__}H4(u*%p*rI=cU!%ya5ugfGATh66$IJHgu1Gs0-<N;$V+SsdE)?u
zIq;i$f#WE4f$_MWicZjMEob9LWKMR#iwZq54~QgST^6=i%u0lUkJu-_J**QBMq}ZG
Wth_)NDbl|`oQr&HAFQ5h`0jqAIZ*BZ
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/88/f9a6eaa2cf008b9bc92847178621f21fa99f3e b/tests/resources/small.git/objects/88/f9a6eaa2cf008b9bc92847178621f21fa99f3e
new file mode 100644
index 0000000000000000000000000000000000000000..7d80b8d78e0dc55669d831a6638f48ec9fed0982
GIT binary patch
literal 50
zcmV-20L}k+0V^p=O;s>9W-v4`Ff%bx$Vkn}$!Ay}rnY6F$m-Kg*KD_+;Lx#g4|^&N
I02NaX#p`nv=Kufz
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/af/5626b4a114abcb82d63db7c8082c3c4756e51b b/tests/resources/small.git/objects/af/5626b4a114abcb82d63db7c8082c3c4756e51b
new file mode 100644
index 0000000000000000000000000000000000000000..822bc151862ec3763cf2d3fa2372b93bbd3a4b65
GIT binary patch
literal 30
mcmb<m^geacKghr&@q@?NlP9kSYMj?U<r(;diNWtH+YSKNt_|)0
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/c6/97d4f7a6eac8d7b131673c340bd3cc5bac14d4 b/tests/resources/small.git/objects/c6/97d4f7a6eac8d7b131673c340bd3cc5bac14d4
new file mode 100644
index 0000000000000000000000000000000000000000..5adfa88cc6b00bb19f2031b8ab61f6d07f9ccdb8
GIT binary patch
literal 130
zcmV-|0Db>>0i}&W3IZ_@1U=^!a~EV1casc=c+{&un1qQN*i9hD|0|m(2n|iwp*q%W
z%N;b$hu%cM`$TMo*~EnC1BFP&Pfj~;jZVKXQ96s_PhV<-XAROi+@-v8dBLUa`!;GB
k^i<X>XlEv8$>R)1G>9th&t3j;s7J{?^9n<zzF|~BaA?ar-~a#s
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/refs/heads/master b/tests/resources/small.git/refs/heads/master
new file mode 100644
index 0000000000000000000000000000000000000000..4eb36d22298f060fd324155ab854d9d6486fc498
GIT binary patch
literal 41
ucmV~$!4Uu;2m`Rc)5uXl$AMP&AHjriQg~T$i(A>|7U^`%mXoWC24Q^m!3%@{
literal 0
HcmV?d00001
+3 -2
View File
@@ -117,10 +117,11 @@ RealisedPath::Set BuiltPath::toRealisedPaths(Store & store) const
"the derivation '%s' has unrealised output '%s' (derived-path.cc/toRealisedPaths)",
store.printStorePath(p.drvPath->outPath()),
outputName);
auto thisRealisation = store.queryRealisation(DrvOutput{*drvOutput, outputName});
DrvOutput key{*drvOutput, outputName};
auto thisRealisation = store.queryRealisation(key);
assert(thisRealisation); // We’ve built it, so we must
// have the realisation
res.insert(*thisRealisation);
res.insert(Realisation{*thisRealisation, std::move(key)});
} else {
res.insert(outputPath);
}
+14
View File
@@ -350,6 +350,20 @@ struct MixEnvironment : virtual Args
void setEnviron();
};
struct MixNoCheckSigs : virtual Args
{
CheckSigsFlag checkSigs = CheckSigs;
MixNoCheckSigs()
{
addFlag({
.longName = "no-check-sigs",
.description = "Do not require that paths are signed by trusted keys.",
.handler = {&checkSigs, NoCheckSigs},
});
}
};
void completeFlakeInputAttrPath(
AddCompletions & completions,
ref<EvalState> evalState,
@@ -69,7 +69,7 @@ struct InstallableFlake : InstallableValue
*/
std::vector<ref<eval_cache::AttrCursor>> getCursors(EvalState & state) override;
std::shared_ptr<flake::LockedFlake> getLockedFlake() const;
ref<flake::LockedFlake> getLockedFlake() const;
FlakeRef nixpkgsFlakeRef() const;
};
@@ -87,6 +87,4 @@ static inline FlakeRef defaultNixpkgsFlakeRef()
return FlakeRef::fromAttrs(fetchSettings, {{"type", "indirect"}, {"id", "nixpkgs"}});
}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, std::shared_ptr<flake::LockedFlake> lockedFlake);
} // namespace nix
+4 -4
View File
@@ -185,16 +185,16 @@ std::vector<ref<eval_cache::AttrCursor>> InstallableFlake::getCursors(EvalState
return res;
}
std::shared_ptr<flake::LockedFlake> InstallableFlake::getLockedFlake() const
ref<flake::LockedFlake> InstallableFlake::getLockedFlake() const
{
if (!_lockedFlake) {
flake::LockFlags lockFlagsApplyConfig = lockFlags;
// FIXME why this side effect?
lockFlagsApplyConfig.applyNixConfig = true;
_lockedFlake =
std::make_shared<flake::LockedFlake>(lockFlake(flakeSettings, *state, flakeRef, lockFlagsApplyConfig));
_lockedFlake = make_ref<flake::LockedFlake>(lockFlake(flakeSettings, *state, flakeRef, lockFlagsApplyConfig));
}
return _lockedFlake;
// _lockedFlake is now non-null but still just a shared_ptr
return ref<flake::LockedFlake>(_lockedFlake);
}
FlakeRef InstallableFlake::nixpkgsFlakeRef() const
+1 -38
View File
@@ -342,8 +342,7 @@ void completeFlakeRefWithFragment(
parseFlakeRef(fetchSettings, expandTilde(flakeRefS), std::filesystem::current_path().string());
auto evalCache = openEvalCache(
*evalState,
std::make_shared<flake::LockedFlake>(lockFlake(flakeSettings, *evalState, flakeRef, lockFlags)));
*evalState, make_ref<flake::LockedFlake>(lockFlake(flakeSettings, *evalState, flakeRef, lockFlags)));
auto root = evalCache->getRoot();
@@ -443,42 +442,6 @@ static StorePath getDeriver(ref<Store> store, const Installable & i, const Store
return *derivers.begin();
}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, std::shared_ptr<flake::LockedFlake> lockedFlake)
{
auto fingerprint = evalSettings.useEvalCache && evalSettings.pureEval
? lockedFlake->getFingerprint(state.store, state.fetchSettings)
: std::nullopt;
auto rootLoader = [&state, lockedFlake]() {
/* For testing whether the evaluation cache is
complete. */
if (getEnv("NIX_ALLOW_EVAL").value_or("1") == "0")
throw Error("not everything is cached, but evaluation is not allowed");
auto vFlake = state.allocValue();
flake::callFlake(state, *lockedFlake, *vFlake);
state.forceAttrs(*vFlake, noPos, "while parsing cached flake data");
auto aOutputs = vFlake->attrs()->get(state.symbols.create("outputs"));
assert(aOutputs);
return aOutputs->value;
};
if (fingerprint) {
auto search = state.evalCaches.find(fingerprint.value());
if (search == state.evalCaches.end()) {
search =
state.evalCaches
.emplace(fingerprint.value(), make_ref<nix::eval_cache::EvalCache>(fingerprint, state, rootLoader))
.first;
}
return search->second;
} else {
return make_ref<nix::eval_cache::EvalCache>(std::nullopt, state, rootLoader);
}
}
Installables SourceExprCommand::parseInstallables(ref<Store> store, std::vector<std::string> ss)
{
Installables result;
-1
View File
@@ -67,7 +67,6 @@ config_priv_h = configure_file(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'built-path.cc',
-1
View File
@@ -28,7 +28,6 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_expr.cc',
-1
View File
@@ -31,7 +31,6 @@ rapidcheck = dependency('rapidcheck')
deps_public += rapidcheck
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'tests/value/context.cc',
+1 -2
View File
@@ -45,7 +45,6 @@ config_priv_h = configure_file(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'derived-path.cc',
@@ -83,7 +82,7 @@ this_exe = executable(
test(
meson.project_name(),
this_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
},
protocol : 'gtest',
-1
View File
@@ -62,7 +62,6 @@ mkMesonExecutable (finalAttrs: {
mkdir -p "$HOME"
''
+ ''
export ASAN_OPTIONS=abort_on_error=1:print_summary=1:detect_leaks=0
export _NIX_TEST_UNIT_DATA=${resolvePath ./data}
${stdenv.hostPlatform.emulator buildPackages} ${lib.getExe finalAttrs.finalPackage}
touch $out
+1 -3
View File
@@ -2160,9 +2160,7 @@ void EvalState::forceValueDeep(Value & v)
{
std::set<const Value *> seen;
std::function<void(Value & v)> recurse;
recurse = [&](Value & v) {
auto recurse = [&](this auto & recurse, Value & v) {
if (!seen.insert(&v).second)
return;
+8 -1
View File
@@ -508,8 +508,15 @@ private:
public:
/**
* @param lookupPath Only used during construction.
* @param store The store to use for instantiation
* @param fetchSettings Must outlive the lifetime of this EvalState!
* @param settings Must outlive the lifetime of this EvalState!
* @param buildStore The store to use for builds ("import from derivation", C API `nix_string_realise`)
*/
EvalState(
const LookupPath & _lookupPath,
const LookupPath & lookupPath,
ref<Store> store,
const fetchers::Settings & fetchSettings,
const EvalSettings & settings,
-1
View File
@@ -97,7 +97,6 @@ config_priv_h = configure_file(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
parser_tab = custom_target(
input : 'parser.y',
+2 -1
View File
@@ -1420,7 +1420,8 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
.debugThrow();
}
if (ingestionMethod == ContentAddressMethod::Raw::Text)
experimentalFeatureSettings.require(Xp::DynamicDerivations);
experimentalFeatureSettings.require(
Xp::DynamicDerivations, fmt("text-hashed derivation '%s', outputHashMode = \"text\"", drvName));
if (ingestionMethod == ContentAddressMethod::Raw::Git)
experimentalFeatureSettings.require(Xp::GitHashing);
};
+5 -1
View File
@@ -588,7 +588,11 @@ static void fetch(
if (expectedHash) {
auto hash = unpack ? state.store->queryPathInfo(storePath)->narHash
: hashFile(HashAlgorithm::SHA256, state.store->toRealPath(storePath));
: hashPath(
{state.store->requireStoreObjectAccessor(storePath)},
FileSerialisationMethod::Flat,
HashAlgorithm::SHA256)
.hash;
if (hash != *expectedHash) {
state
.error<EvalError>(
+3 -4
View File
@@ -92,7 +92,7 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Va
std::istringstream tomlStream(std::string{toml});
auto visit = [&](auto & self, Value & v, toml::value t) -> void {
auto visit = [&](this auto & visit, Value & v, toml::value t) -> void {
switch (t.type()) {
case toml::value_t::table: {
auto table = toml::get<toml::table>(t);
@@ -100,7 +100,7 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Va
for (auto & elem : table) {
forceNoNullByte(elem.first);
self(self, attrs.alloc(elem.first), elem.second);
visit(attrs.alloc(elem.first), elem.second);
}
v.mkAttrs(attrs);
@@ -110,7 +110,7 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Va
auto list = state.buildList(array.size());
for (const auto & [n, v] : enumerate(list))
self(self, *(v = state.allocValue()), array[n]);
visit(*(v = state.allocValue()), array[n]);
v.mkList(list);
} break;
case toml::value_t::boolean:
@@ -155,7 +155,6 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Va
try {
visit(
visit,
val,
toml::parse(
tomlStream,
-1
View File
@@ -32,7 +32,6 @@ add_project_arguments(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_fetchers.cc',
+6
View File
@@ -175,6 +175,12 @@ TEST_F(GitUtilsTest, peel_reference)
TEST(GitUtils, isLegalRefName)
{
ASSERT_TRUE(isLegalRefName("A/b"));
ASSERT_TRUE(isLegalRefName("AaA/b"));
ASSERT_TRUE(isLegalRefName("FOO/BAR/BAZ"));
ASSERT_TRUE(isLegalRefName("HEAD"));
ASSERT_TRUE(isLegalRefName("refs/tags/1.2.3"));
ASSERT_TRUE(isLegalRefName("refs/heads/master"));
ASSERT_TRUE(isLegalRefName("foox"));
ASSERT_TRUE(isLegalRefName("1337"));
ASSERT_TRUE(isLegalRefName("foo.baz"));
+1 -2
View File
@@ -37,7 +37,6 @@ libgit2 = dependency('libgit2')
deps_private += libgit2
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'access-tokens.cc',
@@ -64,7 +63,7 @@ this_exe = executable(
test(
meson.project_name(),
this_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
},
protocol : 'gtest',
-1
View File
@@ -61,7 +61,6 @@ mkMesonExecutable (finalAttrs: {
buildInputs = [ writableTmpDirAsHomeHook ];
}
''
export ASAN_OPTIONS=abort_on_error=1:print_summary=1:detect_leaks=0
export _NIX_TEST_UNIT_DATA=${resolvePath ./data}
${stdenv.hostPlatform.emulator buildPackages} ${lib.getExe finalAttrs.finalPackage}
touch $out
+1 -2
View File
@@ -332,8 +332,7 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(ref<Store> sto
debug("using substituted/cached input '%s' in '%s'", to_string(), store->printStorePath(storePath));
// We just ensured the store object was there
auto accessor = ref{store->getFSAccessor(storePath)};
auto accessor = store->requireStoreObjectAccessor(storePath);
accessor->fingerprint = getFingerprint(store);
+15 -43
View File
@@ -12,6 +12,7 @@
#include <git2/attr.h>
#include <git2/blob.h>
#include <git2/branch.h>
#include <git2/commit.h>
#include <git2/config.h>
#include <git2/describe.h>
@@ -28,6 +29,7 @@
#include <git2/submodule.h>
#include <git2/sys/odb_backend.h>
#include <git2/sys/mempack.h>
#include <git2/tag.h>
#include <git2/tree.h>
#include <boost/unordered/unordered_flat_map.hpp>
@@ -1323,63 +1325,33 @@ GitRepo::WorkdirInfo GitRepo::getCachedWorkdirInfo(const std::filesystem::path &
return workdirInfo;
}
/**
* Checks that the git reference is valid and normalizes slash '/' sequences.
*
* Accepts shorthand references (one-level refnames are allowed).
*/
bool isValidRefNameAllowNormalizations(const std::string & refName)
{
/* Unfortunately libgit2 doesn't expose the limit in headers, but its internal
limit is also 1024. */
std::array<char, 1024> normalizedRefBuffer;
/* It would be nice to have a better API like git_reference_name_is_valid, but
* with GIT_REFERENCE_FORMAT_REFSPEC_SHORTHAND flag. libgit2 uses it internally
* but doesn't expose it in public headers [1].
* [1]:
* https://github.com/libgit2/libgit2/blob/9d5f1bacc23594c2ba324c8f0d41b88bf0e9ef04/src/libgit2/refs.c#L1362-L1365
*/
auto res = git_reference_normalize_name(
normalizedRefBuffer.data(),
normalizedRefBuffer.size(),
refName.c_str(),
GIT_REFERENCE_FORMAT_ALLOW_ONELEVEL | GIT_REFERENCE_FORMAT_REFSPEC_SHORTHAND);
return res == 0;
}
bool isLegalRefName(const std::string & refName)
{
initLibGit2();
/* Since `git_reference_normalize_name` is the best API libgit2 has for verifying
* reference names with shorthands (see comment in normalizeRefName), we need to
* ensure that exceptions to the validity checks imposed by normalization [1] are checked
* explicitly.
* [1]: https://git-scm.com/docs/git-check-ref-format#Documentation/git-check-ref-format.txt---normalize
*/
/* Check for cases that don't get rejected by libgit2.
* FIXME: libgit2 should reject this. */
if (refName == "@")
return false;
/* Leading slashes and consecutive slashes are stripped during normalizatiton. */
if (refName.starts_with('/') || refName.find("//") != refName.npos)
return false;
/* Refer to libgit2. */
if (!isValidRefNameAllowNormalizations(refName))
return false;
/* libgit2 doesn't barf on DEL symbol.
* FIXME: libgit2 should reject this. */
if (refName.find('\177') != refName.npos)
return false;
return true;
for (auto * func : {
git_reference_name_is_valid,
git_branch_name_is_valid,
git_tag_name_is_valid,
}) {
int valid = 0;
if (func(&valid, refName.c_str()))
throw Error("checking git reference '%s': %s", refName, git_error_last()->message);
if (valid)
return true;
}
return false;
}
} // namespace nix
+30
View File
@@ -496,6 +496,36 @@ struct GitInputScheme : InputScheme
Git interprets them as part of the file name. So get
rid of them. */
url.query.clear();
/* Backward compatibility hack: In old versions of Nix, if you had
a flake input like
inputs.foo.url = "git+https://foo/bar?dir=subdir";
it would result in a lock file entry like
"original": {
"dir": "subdir",
"type": "git",
"url": "https://foo/bar?dir=subdir"
}
New versions of Nix remove `?dir=subdir` from the `url` field,
since the subdirectory is intended for `FlakeRef`, not the
fetcher (and specifically the remote server), that is, the
flakeref is parsed into
"original": {
"dir": "subdir",
"type": "git",
"url": "https://foo/bar"
}
However, new versions of nix parsing old flake.lock files would pass the dir=
query parameter in the "url" attribute to git, which will then complain.
For this reason, we are filtering the `dir` query parameter from the URL
before passing it to git. */
url.query.erase("dir");
repoInfo.location = url;
}
+11 -11
View File
@@ -399,7 +399,8 @@ struct GitHubInputScheme : GitArchiveInputScheme
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
auto downloadResult = downloadFile(store, *input.settings, url, "source", headers);
auto json = nlohmann::json::parse(store->getFSAccessor(downloadResult.storePath)->readFile(CanonPath::root));
auto json = nlohmann::json::parse(
store->requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
return RefInfo{
.rev = Hash::parseAny(std::string{json["sha"]}, HashAlgorithm::SHA1),
@@ -473,7 +474,8 @@ struct GitLabInputScheme : GitArchiveInputScheme
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
auto downloadResult = downloadFile(store, *input.settings, url, "source", headers);
auto json = nlohmann::json::parse(store->getFSAccessor(downloadResult.storePath)->readFile(CanonPath::root));
auto json = nlohmann::json::parse(
store->requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
if (json.is_array() && json.size() >= 1 && json[0]["id"] != nullptr) {
return RefInfo{.rev = Hash::parseAny(std::string(json[0]["id"]), HashAlgorithm::SHA1)};
@@ -548,13 +550,10 @@ struct SourceHutInputScheme : GitArchiveInputScheme
std::string refUri;
if (ref == "HEAD") {
auto file = store->toRealPath(
downloadFile(store, *input.settings, fmt("%s/HEAD", base_url), "source", headers).storePath);
std::ifstream is(file);
std::string line;
getline(is, line);
auto downloadFileResult = downloadFile(store, *input.settings, fmt("%s/HEAD", base_url), "source", headers);
auto contents = store->requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
auto remoteLine = git::parseLsRemoteLine(line);
auto remoteLine = git::parseLsRemoteLine(getLine(contents).first);
if (!remoteLine) {
throw BadURL("in '%d', couldn't resolve HEAD ref '%d'", input.to_string(), ref);
}
@@ -564,9 +563,10 @@ struct SourceHutInputScheme : GitArchiveInputScheme
}
std::regex refRegex(refUri);
auto file = store->toRealPath(
downloadFile(store, *input.settings, fmt("%s/info/refs", base_url), "source", headers).storePath);
std::ifstream is(file);
auto downloadFileResult =
downloadFile(store, *input.settings, fmt("%s/info/refs", base_url), "source", headers);
auto contents = store->requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
std::istringstream is(contents);
std::string line;
std::optional<std::string> id;
@@ -158,9 +158,12 @@ struct Setter
};
/**
* Checks that the git reference is valid and normalized.
* Checks that the string can be a valid git reference, branch or tag name.
* Accepts shorthand references (one-level refnames are allowed), pseudorefs
* like `HEAD`.
*
* Accepts shorthand references (one-level refnames are allowed).
* @note This is a coarse test to make sure that the refname is at least something
* that Git can make sense of.
*/
bool isLegalRefName(const std::string & refName);
+1 -3
View File
@@ -329,9 +329,7 @@ struct MercurialInputScheme : InputScheme
Input input(_input);
auto storePath = fetchToStore(store, input);
// We just added it, it should be there.
auto accessor = ref{store->getFSAccessor(storePath)};
auto accessor = store->requireStoreObjectAccessor(storePath);
accessor->setPathDisplay("«" + input.to_string() + "»");
-1
View File
@@ -32,7 +32,6 @@ libgit2 = dependency('libgit2', version : '>= 1.9')
deps_private += libgit2
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'attrs.cc',
+1 -1
View File
@@ -138,7 +138,7 @@ struct PathInputScheme : InputScheme
storePath = store->addToStoreFromDump(*src, "source");
}
auto accessor = ref{store->getFSAccessor(*storePath)};
auto accessor = store->requireStoreObjectAccessor(*storePath);
// To prevent `fetchToStore()` copying the path again to Nix
// store, pre-create an entry in the fetcher cache.
+6 -6
View File
@@ -42,7 +42,7 @@ DownloadFileResult downloadFile(
if (cached && !cached->expired)
return useCached();
FileTransferRequest request(ValidURL{url});
FileTransferRequest request(VerbatimURL{url});
request.headers = headers;
if (cached)
request.expectedETag = getStrAttr(cached->value, "etag");
@@ -107,13 +107,13 @@ DownloadFileResult downloadFile(
static DownloadTarballResult downloadTarball_(
const Settings & settings, const std::string & urlS, const Headers & headers, const std::string & displayPrefix)
{
ValidURL url = urlS;
ParsedURL url = parseURL(urlS);
// Some friendly error messages for common mistakes.
// Namely lets catch when the url is a local file path, but
// it is not in fact a tarball.
if (url.scheme() == "file") {
std::filesystem::path localPath = renderUrlPathEnsureLegal(url.path());
if (url.scheme == "file") {
std::filesystem::path localPath = renderUrlPathEnsureLegal(url.path);
if (!exists(localPath)) {
throw Error("tarball '%s' does not exist.", localPath);
}
@@ -164,7 +164,7 @@ static DownloadTarballResult downloadTarball_(
/* Note: if the download is cached, `importTarball()` will receive
no data, which causes it to import an empty tarball. */
auto archive = !url.path().empty() && hasSuffix(toLower(url.path().back()), ".zip") ? ({
auto archive = !url.path.empty() && hasSuffix(toLower(url.path.back()), ".zip") ? ({
/* In streaming mode, libarchive doesn't handle
symlinks in zip files correctly (#10649). So write
the entire file to disk so libarchive can access it
@@ -178,7 +178,7 @@ static DownloadTarballResult downloadTarball_(
}
TarArchive{path};
})
: TarArchive{*source};
: TarArchive{*source};
auto tarballCache = getTarballCache();
auto parseSink = tarballCache->getFileSystemObjectSink();
auto lastModified = unpackTarfileToSink(archive, *parseSink);
-1
View File
@@ -32,7 +32,6 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_flake.cc',
+7
View File
@@ -1,8 +1,15 @@
#include <gtest/gtest.h>
#include <set>
#include <string>
#include <utility>
#include "nix/fetchers/fetch-settings.hh"
#include "nix/flake/flakeref.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/util/configuration.hh"
#include "nix/util/error.hh"
#include "nix/util/experimental-features.hh"
namespace nix {
+1 -2
View File
@@ -34,7 +34,6 @@ gtest = dependency('gtest', main : true)
deps_private += gtest
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'flakeref.cc',
@@ -59,7 +58,7 @@ this_exe = executable(
test(
meson.project_name(),
this_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
'NIX_CONFIG' : 'extra-experimental-features = flakes',
'HOME' : meson.current_build_dir() / 'test-home',
+7 -5
View File
@@ -1,15 +1,17 @@
#include <gtest/gtest.h>
#include <filesystem>
#include <string>
#include "nix/util/file-system.hh"
#include "nix_api_store.h"
#include "nix_api_util.h"
#include "nix_api_expr.h"
#include "nix_api_value.h"
#include "nix_api_flake.h"
#include "nix/expr/tests/nix_api_expr.hh"
#include "nix/util/tests/string_callback.hh"
#include <gmock/gmock.h>
#include <gtest/gtest.h>
#include "nix/store/tests/nix_api_store.hh"
#include "nix/util/tests/nix_api_util.hh"
#include "nix_api_fetchers.h"
namespace nixC {
-1
View File
@@ -59,7 +59,6 @@ mkMesonExecutable (finalAttrs: {
buildInputs = [ writableTmpDirAsHomeHook ];
}
(''
export ASAN_OPTIONS=abort_on_error=1:print_summary=1:detect_leaks=0
export _NIX_TEST_UNIT_DATA=${resolvePath ./data}
export NIX_CONFIG="extra-experimental-features = flakes"
${stdenv.hostPlatform.emulator buildPackages} ${lib.getExe finalAttrs.finalPackage}
+3 -1
View File
@@ -1,6 +1,8 @@
#include "nix/flake/url-name.hh"
#include <gtest/gtest.h>
#include "nix/flake/url-name.hh"
#include "nix/util/url.hh"
namespace nix {
/* ----------- tests for url-name.hh --------------------------------------------------*/
+23 -2
View File
@@ -1,9 +1,30 @@
#include <nlohmann/json.hpp>
#include <assert.h>
#include <stdint.h>
#include <nlohmann/json_fwd.hpp>
#include <cctype>
#include <filesystem>
#include <map>
#include <memory>
#include <optional>
#include <string>
#include <utility>
#include <variant>
#include <vector>
#include <format>
#include "nix/util/users.hh"
#include "nix/util/config-global.hh"
#include "nix/flake/settings.hh"
#include "nix/flake/flake.hh"
#include <nlohmann/json.hpp>
#include "nix/util/ansicolor.hh"
#include "nix/util/configuration.hh"
#include "nix/util/file-system.hh"
#include "nix/util/fmt.hh"
#include "nix/util/logging.hh"
#include "nix/util/strings.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
namespace nix::flake {
+26
View File
@@ -1,8 +1,34 @@
#include <stdint.h>
#include <functional>
#include <map>
#include <optional>
#include <set>
#include <string>
#include <utility>
#include <variant>
#include <vector>
#include "nix/flake/flake-primops.hh"
#include "nix/expr/eval.hh"
#include "nix/flake/flake.hh"
#include "nix/flake/flakeref.hh"
#include "nix/flake/settings.hh"
#include "nix/expr/attr-set.hh"
#include "nix/expr/eval-error.hh"
#include "nix/expr/eval-inline.hh"
#include "nix/expr/eval-settings.hh"
#include "nix/expr/symbol-table.hh"
#include "nix/expr/value.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/util/configuration.hh"
#include "nix/util/error.hh"
#include "nix/util/experimental-features.hh"
#include "nix/util/pos-idx.hh"
#include "nix/util/pos-table.hh"
#include "nix/util/source-path.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
namespace nix::flake::primops {
+108 -38
View File
@@ -1,9 +1,32 @@
#include <nlohmann/json.hpp>
#include <assert.h>
#include <stdint.h>
#include <boost/container/detail/std_fwd.hpp>
#include <boost/core/pointer_traits.hpp>
#include <boost/unordered/detail/foa/table.hpp>
#include <algorithm>
#include <filesystem>
#include <functional>
#include <map>
#include <memory>
#include <optional>
#include <set>
#include <span>
#include <string>
#include <tuple>
#include <utility>
#include <variant>
#include <vector>
#include <format>
#include "nix/util/terminal.hh"
#include "nix/util/ref.hh"
#include "nix/util/environment-variables.hh"
#include "nix/flake/flake.hh"
#include "nix/expr/eval.hh"
#include "nix/expr/eval-cache.hh"
#include "nix/expr/eval-settings.hh"
#include "nix/flake/lockfile.hh"
#include "nix/expr/primops.hh"
#include "nix/expr/eval-inline.hh"
#include "nix/store/store-api.hh"
#include "nix/fetchers/fetchers.hh"
@@ -11,14 +34,36 @@
#include "nix/fetchers/fetch-settings.hh"
#include "nix/flake/settings.hh"
#include "nix/expr/value-to-json.hh"
#include "nix/store/local-fs-store.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/util/memory-source-accessor.hh"
#include "nix/fetchers/input-cache.hh"
#include <nlohmann/json.hpp>
#include "nix/expr/attr-set.hh"
#include "nix/expr/eval-error.hh"
#include "nix/expr/nixexpr.hh"
#include "nix/expr/symbol-table.hh"
#include "nix/expr/value.hh"
#include "nix/expr/value/context.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/registry.hh"
#include "nix/flake/flakeref.hh"
#include "nix/store/path.hh"
#include "nix/util/canon-path.hh"
#include "nix/util/configuration.hh"
#include "nix/util/error.hh"
#include "nix/util/experimental-features.hh"
#include "nix/util/file-system.hh"
#include "nix/util/fmt.hh"
#include "nix/util/hash.hh"
#include "nix/util/logging.hh"
#include "nix/util/pos-idx.hh"
#include "nix/util/pos-table.hh"
#include "nix/util/position.hh"
#include "nix/util/source-path.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
namespace nix {
struct SourceAccessor;
using namespace flake;
@@ -424,41 +469,31 @@ lockFlake(const Settings & settings, EvalState & state, const FlakeRef & topRef,
std::vector<FlakeRef> parents;
std::function<void(
const FlakeInputs & flakeInputs,
ref<Node> node,
const InputAttrPath & inputAttrPathPrefix,
std::shared_ptr<const Node> oldNode,
const InputAttrPath & followsPrefix,
const SourcePath & sourcePath,
bool trustLock)>
computeLocks;
computeLocks = [&](
/* The inputs of this node, either from flake.nix or
flake.lock. */
const FlakeInputs & flakeInputs,
/* The node whose locks are to be updated.*/
ref<Node> node,
/* The path to this node in the lock file graph. */
const InputAttrPath & inputAttrPathPrefix,
/* The old node, if any, from which locks can be
copied. */
std::shared_ptr<const Node> oldNode,
/* The prefix relative to which 'follows' should be
interpreted. When a node is initially locked, it's
relative to the node's flake; when it's already locked,
it's relative to the root of the lock file. */
const InputAttrPath & followsPrefix,
/* The source path of this node's flake. */
const SourcePath & sourcePath,
bool trustLock) {
auto computeLocks = [&](this auto & computeLocks,
/* The inputs of this node, either from flake.nix or
flake.lock. */
const FlakeInputs & flakeInputs,
/* The node whose locks are to be updated.*/
ref<Node> node,
/* The path to this node in the lock file graph. */
const InputAttrPath & inputAttrPathPrefix,
/* The old node, if any, from which locks can be
copied. */
std::shared_ptr<const Node> oldNode,
/* The prefix relative to which 'follows' should be
interpreted. When a node is initially locked, it's
relative to the node's flake; when it's already locked,
it's relative to the root of the lock file. */
const InputAttrPath & followsPrefix,
/* The source path of this node's flake. */
const SourcePath & sourcePath,
bool trustLock) -> void {
debug("computing lock file node '%s'", printInputAttrPath(inputAttrPathPrefix));
/* Get the overrides (i.e. attributes of the form
'inputs.nixops.inputs.nixpkgs.url = ...'). */
std::function<void(const FlakeInput & input, const InputAttrPath & prefix)> addOverrides;
addOverrides = [&](const FlakeInput & input, const InputAttrPath & prefix) {
auto addOverrides =
[&](this auto & addOverrides, const FlakeInput & input, const InputAttrPath & prefix) -> void {
for (auto & [idOverride, inputOverride] : input.overrides) {
auto inputAttrPath(prefix);
inputAttrPath.push_back(idOverride);
@@ -862,7 +897,7 @@ static ref<SourceAccessor> makeInternalFS()
internalFS->setPathDisplay("«flakes-internal»", "");
internalFS->addFile(
CanonPath("call-flake.nix"),
#include "call-flake.nix.gen.hh"
#include "call-flake.nix.gen.hh" // IWYU pragma: keep
);
return internalFS;
}
@@ -924,8 +959,6 @@ void callFlake(EvalState & state, const LockedFlake & lockedFlake, Value & vRes)
state.callFunction(*vCallFlake, args, vRes, noPos);
}
} // namespace flake
std::optional<Fingerprint> LockedFlake::getFingerprint(ref<Store> store, const fetchers::Settings & fetchSettings) const
{
if (lockFile.isUnlocked(fetchSettings))
@@ -953,4 +986,41 @@ std::optional<Fingerprint> LockedFlake::getFingerprint(ref<Store> store, const f
Flake::~Flake() {}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, ref<const LockedFlake> lockedFlake)
{
auto fingerprint = state.settings.useEvalCache && state.settings.pureEval
? lockedFlake->getFingerprint(state.store, state.fetchSettings)
: std::nullopt;
auto rootLoader = [&state, lockedFlake]() {
/* For testing whether the evaluation cache is
complete. */
if (getEnv("NIX_ALLOW_EVAL").value_or("1") == "0")
throw Error("not everything is cached, but evaluation is not allowed");
auto vFlake = state.allocValue();
callFlake(state, *lockedFlake, *vFlake);
state.forceAttrs(*vFlake, noPos, "while parsing cached flake data");
auto aOutputs = vFlake->attrs()->get(state.symbols.create("outputs"));
assert(aOutputs);
return aOutputs->value;
};
if (fingerprint) {
auto search = state.evalCaches.find(fingerprint.value());
if (search == state.evalCaches.end()) {
search = state.evalCaches
.emplace(fingerprint.value(), make_ref<eval_cache::EvalCache>(fingerprint, state, rootLoader))
.first;
}
return search->second;
} else {
return make_ref<eval_cache::EvalCache>(std::nullopt, state, rootLoader);
}
}
} // namespace flake
} // namespace nix
+30 -1
View File
@@ -1,10 +1,39 @@
#include <assert.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <filesystem>
#include <ostream>
#include <string_view>
#include <vector>
#include <optional>
#include <regex>
#include <string>
#include <tuple>
#include <utility>
#include "nix/flake/flakeref.hh"
#include "nix/store/store-api.hh"
#include "nix/util/url.hh"
#include "nix/util/url-parts.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/util/error.hh"
#include "nix/util/file-system.hh"
#include "nix/util/fmt.hh"
#include "nix/util/logging.hh"
#include "nix/util/strings.hh"
#include "nix/util/util.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/registry.hh"
#include "nix/store/outputs-spec.hh"
#include "nix/util/ref.hh"
#include "nix/util/types.hh"
namespace nix {
class Store;
struct SourceAccessor;
namespace fetchers {
struct Settings;
} // namespace fetchers
#if 0
// 'dir' path elements cannot start with a '.'. We also reject
@@ -1,7 +1,12 @@
#pragma once
#include "nix/expr/eval.hh"
#include "nix/flake/settings.hh"
namespace nix {
namespace flake {
struct Settings;
} // namespace flake
} // namespace nix
namespace nix::flake::primops {
+6
View File
@@ -5,6 +5,7 @@
#include "nix/flake/flakeref.hh"
#include "nix/flake/lockfile.hh"
#include "nix/expr/value.hh"
#include "nix/expr/eval-cache.hh"
namespace nix {
@@ -218,6 +219,11 @@ lockFlake(const Settings & settings, EvalState & state, const FlakeRef & flakeRe
void callFlake(EvalState & state, const LockedFlake & lockedFlake, Value & v);
/**
* Open an evaluation cache for a flake.
*/
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, ref<const LockedFlake> lockedFlake);
} // namespace flake
void emitTreeAttrs(
+8 -2
View File
@@ -2,9 +2,11 @@
///@file
#include <regex>
#include <iosfwd>
#include <string>
#include <tuple>
#include <utility>
#include "nix/util/types.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/store/outputs-spec.hh"
#include "nix/fetchers/registry.hh"
@@ -12,6 +14,10 @@ namespace nix {
class Store;
namespace fetchers {
struct Settings;
} // namespace fetchers
typedef std::string FlakeId;
/**
+3 -2
View File
@@ -1,9 +1,10 @@
#pragma once
///@file
#include "nix/util/configuration.hh"
#include <sys/types.h>
#include <string>
#include "nix/util/configuration.hh"
namespace nix {
// Forward declarations
+3 -4
View File
@@ -1,9 +1,8 @@
#include "nix/util/url.hh"
#include "nix/util/url-parts.hh"
#include "nix/util/util.hh"
#include "nix/util/split.hh"
#include <optional>
#include <string>
namespace nix {
struct ParsedURL;
/**
* Try to extract a reasonably unique and meaningful, human-readable
+46 -18
View File
@@ -1,15 +1,49 @@
#include "nix/fetchers/fetch-settings.hh"
#include "nix/flake/settings.hh"
#include "nix/flake/lockfile.hh"
#include "nix/store/store-api.hh"
#include "nix/util/strings.hh"
#include <boost/unordered/unordered_flat_set.hpp>
#include <nlohmann/json.hpp>
#include <assert.h>
#include <boost/unordered/unordered_flat_set_fwd.hpp>
#include <nlohmann/detail/iterators/iter_impl.hpp>
#include <nlohmann/detail/iterators/iteration_proxy.hpp>
#include <nlohmann/json_fwd.hpp>
#include <algorithm>
#include <iomanip>
#include <boost/unordered/unordered_flat_set.hpp>
#include <iterator>
#include <nlohmann/json.hpp>
#include <compare>
#include <ctime>
#include <format>
#include <functional>
#include <map>
#include <memory>
#include <optional>
#include <ostream>
#include <regex>
#include <set>
#include <string>
#include <string_view>
#include <utility>
#include <variant>
#include <vector>
#include "nix/fetchers/fetch-settings.hh"
#include "nix/flake/lockfile.hh"
#include "nix/util/strings.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/flake/flakeref.hh"
#include "nix/store/path.hh"
#include "nix/util/ansicolor.hh"
#include "nix/util/configuration.hh"
#include "nix/util/error.hh"
#include "nix/util/fmt.hh"
#include "nix/util/hash.hh"
#include "nix/util/logging.hh"
#include "nix/util/ref.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
namespace nix {
class Store;
} // namespace nix
namespace nix::flake {
@@ -115,9 +149,7 @@ LockFile::LockFile(const fetchers::Settings & fetchSettings, std::string_view co
std::map<std::string, ref<Node>> nodeMap;
std::function<void(Node & node, const nlohmann::json & jsonNode)> getInputs;
getInputs = [&](Node & node, const nlohmann::json & jsonNode) {
auto getInputs = [&](this auto & getInputs, Node & node, const nlohmann::json & jsonNode) -> void {
if (jsonNode.find("inputs") == jsonNode.end())
return;
for (auto & i : jsonNode["inputs"].items()) {
@@ -242,9 +274,7 @@ std::optional<FlakeRef> LockFile::isUnlocked(const fetchers::Settings & fetchSet
{
std::set<ref<const Node>> nodes;
std::function<void(ref<const Node> node)> visit;
visit = [&](ref<const Node> node) {
auto visit = [&](this auto & visit, ref<const Node> node) -> void {
if (!nodes.insert(node).second)
return;
for (auto & i : node->inputs)
@@ -298,9 +328,7 @@ std::map<InputAttrPath, Node::Edge> LockFile::getAllInputs() const
std::set<ref<Node>> done;
std::map<InputAttrPath, Node::Edge> res;
std::function<void(const InputAttrPath & prefix, ref<Node> node)> recurse;
recurse = [&](const InputAttrPath & prefix, ref<Node> node) {
auto recurse = [&](this auto & recurse, const InputAttrPath & prefix, ref<Node> node) -> void {
if (!done.insert(node).second)
return;
-1
View File
@@ -29,7 +29,6 @@ nlohmann_json = dependency('nlohmann_json', version : '>= 3.9')
deps_public += nlohmann_json
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
subdir('nix-meson-build-support/generate-header')
+4
View File
@@ -1,5 +1,9 @@
#include <vector>
#include "nix/flake/settings.hh"
#include "nix/flake/flake-primops.hh"
#include "nix/expr/eval-settings.hh"
#include "nix/expr/eval.hh"
namespace nix::flake {
+6 -2
View File
@@ -1,6 +1,10 @@
#include "nix/flake/url-name.hh"
#include <regex>
#include <iostream>
#include <optional>
#include <string>
#include "nix/flake/url-name.hh"
#include "nix/util/strings.hh"
#include "nix/util/url.hh"
namespace nix {
-1
View File
@@ -28,7 +28,6 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_main.cc',
-1
View File
@@ -53,7 +53,6 @@ config_priv_h = configure_file(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'common-args.cc',
-1
View File
@@ -26,7 +26,6 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_store.cc',
-1
View File
@@ -29,7 +29,6 @@ rapidcheck = dependency('rapidcheck')
deps_public += rapidcheck
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'derived-path.cc',
+23 -21
View File
@@ -112,32 +112,34 @@ CHARACTERIZATION_TEST(
"realisation",
(std::tuple<Realisation, Realisation>{
Realisation{
.id =
DrvOutput{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
},
DrvOutput{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
},
Realisation{
.id =
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.dependentRealisations =
{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.dependentRealisations =
{
DrvOutput{
.drvHash = Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "quux",
{
DrvOutput{
.drvHash = Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "quux",
},
StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
},
}))
+13 -2
View File
@@ -1,6 +1,6 @@
#include <gtest/gtest.h>
#include "nix/store/dummy-store.hh"
#include "nix/store/dummy-store-impl.hh"
#include "nix/store/globals.hh"
#include "nix/store/realisation.hh"
@@ -13,7 +13,7 @@ TEST(DummyStore, realisation_read)
auto store = [] {
auto cfg = make_ref<DummyStoreConfig>(StoreReference::Params{});
cfg->readOnly = false;
return cfg->openStore();
return cfg->openDummyStore();
}();
auto drvHash = Hash::parseExplicitFormatUnprefixed(
@@ -22,6 +22,17 @@ TEST(DummyStore, realisation_read)
auto outputName = "foo";
EXPECT_EQ(store->queryRealisation({drvHash, outputName}), nullptr);
UnkeyedRealisation value{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv"},
};
store->buildTrace.insert({drvHash, {{outputName, make_ref<UnkeyedRealisation>(value)}}});
auto value2 = store->queryRealisation({drvHash, outputName});
ASSERT_TRUE(value2);
EXPECT_EQ(*value2, value);
}
} // namespace nix
+3 -3
View File
@@ -52,7 +52,6 @@ gtest = dependency('gmock')
deps_private += gtest
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'common-protocol.cc',
@@ -84,6 +83,7 @@ sources = files(
'store-reference.cc',
'uds-remote-store.cc',
'worker-protocol.cc',
'write-derivation.cc',
)
include_dirs = [ include_directories('.') ]
@@ -105,7 +105,7 @@ this_exe = executable(
test(
meson.project_name(),
this_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
'HOME' : meson.current_build_dir() / 'test-home',
'NIX_REMOTE' : meson.current_build_dir() / 'test-home' / 'store',
@@ -139,7 +139,7 @@ if get_option('benchmarks')
benchmark(
'nix-store-benchmarks',
benchmark_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
},
)
-1
View File
@@ -83,7 +83,6 @@ mkMesonExecutable (finalAttrs: {
}
(
''
export ASAN_OPTIONS=abort_on_error=1:print_summary=1:detect_leaks=0
export _NIX_TEST_UNIT_DATA=${data + "/src/libstore-tests/data"}
export NIX_REMOTE=$HOME/store
${stdenv.hostPlatform.emulator buildPackages} ${lib.getExe finalAttrs.finalPackage}
+10 -10
View File
@@ -49,16 +49,16 @@ INSTANTIATE_TEST_SUITE_P(
RealisationJsonTest,
([] {
Realisation simple{
.id =
{
.drvHash = Hash::parseExplicitFormatUnprefixed(
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
HashAlgorithm::SHA256,
HashFormat::Base16),
.outputName = "foo",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv"},
},
{
.drvHash = Hash::parseExplicitFormatUnprefixed(
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
HashAlgorithm::SHA256,
HashFormat::Base16),
.outputName = "foo",
},
};
return ::testing::Values(
std::pair{
+113 -6
View File
@@ -1,8 +1,9 @@
#include "nix/store/s3-binary-cache-store.hh"
#include "nix/store/http-binary-cache-store.hh"
#include "nix/store/filetransfer.hh"
#include "nix/store/s3-url.hh"
#if NIX_WITH_S3_SUPPORT
# include <gtest/gtest.h>
#include <gtest/gtest.h>
namespace nix {
@@ -10,9 +11,115 @@ TEST(S3BinaryCacheStore, constructConfig)
{
S3BinaryCacheStoreConfig config{"s3", "foobar", {}};
EXPECT_EQ(config.bucketName, "foobar");
// The bucket name is stored as the host part of the authority in cacheUri
EXPECT_EQ(
config.cacheUri,
(ParsedURL{
.scheme = "s3",
.authority = ParsedURL::Authority{.host = "foobar"},
}));
}
TEST(S3BinaryCacheStore, constructConfigWithRegion)
{
Store::Config::Params params{{"region", "eu-west-1"}};
S3BinaryCacheStoreConfig config{"s3", "my-bucket", params};
EXPECT_EQ(
config.cacheUri,
(ParsedURL{
.scheme = "s3",
.authority = ParsedURL::Authority{.host = "my-bucket"},
.query = (StringMap) {{"region", "eu-west-1"}},
}));
EXPECT_EQ(config.region.get(), "eu-west-1");
}
TEST(S3BinaryCacheStore, defaultSettings)
{
S3BinaryCacheStoreConfig config{"s3", "test-bucket", {}};
EXPECT_EQ(
config.cacheUri,
(ParsedURL{
.scheme = "s3",
.authority = ParsedURL::Authority{.host = "test-bucket"},
}));
// Check default values
EXPECT_EQ(config.region.get(), "us-east-1");
EXPECT_EQ(config.profile.get(), "default");
EXPECT_EQ(config.scheme.get(), "https");
EXPECT_EQ(config.endpoint.get(), "");
}
/**
* Test that S3BinaryCacheStore properly preserves S3-specific parameters
*/
TEST(S3BinaryCacheStore, s3StoreConfigPreservesParameters)
{
StringMap params;
params["region"] = "eu-west-1";
params["endpoint"] = "custom.s3.com";
S3BinaryCacheStoreConfig config("s3", "test-bucket", params);
// The config should preserve S3-specific parameters
EXPECT_EQ(
config.cacheUri,
(ParsedURL{
.scheme = "s3",
.authority = ParsedURL::Authority{.host = "test-bucket"},
.query = (StringMap) {{"region", "eu-west-1"}, {"endpoint", "custom.s3.com"}},
}));
}
/**
* Test that S3 store scheme is properly registered
*/
TEST(S3BinaryCacheStore, s3SchemeRegistration)
{
auto schemes = S3BinaryCacheStoreConfig::uriSchemes();
EXPECT_TRUE(schemes.count("s3") > 0) << "S3 scheme should be supported";
// Verify HttpBinaryCacheStoreConfig doesn't directly list S3
auto httpSchemes = HttpBinaryCacheStoreConfig::uriSchemes();
EXPECT_FALSE(httpSchemes.count("s3") > 0) << "HTTP store shouldn't directly list S3 scheme";
}
/**
* Test that only S3-specific parameters are preserved in cacheUri,
* while non-S3 store parameters are not propagated to the URL
*/
TEST(S3BinaryCacheStore, parameterFiltering)
{
StringMap params;
params["region"] = "eu-west-1";
params["endpoint"] = "minio.local";
params["want-mass-query"] = "true"; // Non-S3 store parameter
params["priority"] = "10"; // Non-S3 store parameter
S3BinaryCacheStoreConfig config("s3", "test-bucket", params);
// Only S3-specific params should be in cacheUri.query
EXPECT_EQ(
config.cacheUri,
(ParsedURL{
.scheme = "s3",
.authority = ParsedURL::Authority{.host = "test-bucket"},
.query = (StringMap) {{"region", "eu-west-1"}, {"endpoint", "minio.local"}},
}));
// But the non-S3 params should still be set on the config
EXPECT_EQ(config.wantMassQuery.get(), true);
EXPECT_EQ(config.priority.get(), 10);
// And all params (S3 and non-S3) should be returned by getReference()
auto ref = config.getReference();
EXPECT_EQ(ref.params["region"], "eu-west-1");
EXPECT_EQ(ref.params["endpoint"], "minio.local");
EXPECT_EQ(ref.params["want-mass-query"], "true");
EXPECT_EQ(ref.params["priority"], "10");
}
} // namespace nix
#endif
+2 -6
View File
@@ -1,10 +1,8 @@
#include "nix/store/s3-url.hh"
#include "nix/util/tests/gmock-matchers.hh"
#if NIX_WITH_S3_SUPPORT || NIX_WITH_CURL_S3
# include <gtest/gtest.h>
# include <gmock/gmock.h>
#include <gtest/gtest.h>
#include <gmock/gmock.h>
namespace nix {
@@ -228,5 +226,3 @@ INSTANTIATE_TEST_SUITE_P(
[](const ::testing::TestParamInfo<S3ToHttpsConversionTestCase> & info) { return info.param.description; });
} // namespace nix
#endif
+39 -35
View File
@@ -95,32 +95,34 @@ VERSIONED_CHARACTERIZATION_TEST(
defaultVersion,
(std::tuple<Realisation, Realisation>{
Realisation{
.id =
DrvOutput{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
},
Realisation{
.id =
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.dependentRealisations =
{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.dependentRealisations =
{
DrvOutput{
.drvHash = Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "quux",
{
DrvOutput{
.drvHash = Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "quux",
},
StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
},
}))
@@ -196,25 +198,27 @@ VERSIONED_CHARACTERIZATION_TEST(
{
"foo",
{
.id =
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "foo",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "foo",
},
},
},
{
"bar",
{
.id =
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "bar",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar"},
},
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "bar",
},
},
},
},
+69 -63
View File
@@ -148,32 +148,34 @@ VERSIONED_CHARACTERIZATION_TEST(
defaultVersion,
(std::tuple<Realisation, Realisation>{
Realisation{
.id =
DrvOutput{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
},
DrvOutput{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
},
Realisation{
.id =
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.dependentRealisations =
{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.dependentRealisations =
{
DrvOutput{
.drvHash = Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "quux",
{
DrvOutput{
.drvHash = Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "quux",
},
StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
},
},
DrvOutput{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
},
}))
@@ -214,25 +216,25 @@ VERSIONED_CHARACTERIZATION_TEST(
{
"foo",
{
.id =
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "foo",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
DrvOutput{
.drvHash = Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "foo",
},
},
},
{
"bar",
{
.id =
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "bar",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar"},
},
DrvOutput{
.drvHash = Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "bar",
},
},
},
},
@@ -267,25 +269,27 @@ VERSIONED_CHARACTERIZATION_TEST(
{
"foo",
{
.id =
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "foo",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "foo",
},
},
},
{
"bar",
{
.id =
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "bar",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar"},
},
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "bar",
},
},
},
},
@@ -324,25 +328,27 @@ VERSIONED_CHARACTERIZATION_TEST(
{
"foo",
{
.id =
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "foo",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "foo",
},
},
},
{
"bar",
{
.id =
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "bar",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-bar"},
},
DrvOutput{
.drvHash =
Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "bar",
},
},
},
},
+57
View File
@@ -0,0 +1,57 @@
#include <gtest/gtest.h>
#include <gmock/gmock.h>
#include "nix/util/tests/gmock-matchers.hh"
#include "nix/store/derivations.hh"
#include "nix/store/dummy-store-impl.hh"
#include "nix/store/tests/libstore.hh"
namespace nix {
namespace {
class WriteDerivationTest : public LibStoreTest
{
protected:
WriteDerivationTest(ref<DummyStoreConfig> config_)
: LibStoreTest(config_->openDummyStore())
, config(std::move(config_))
{
config->readOnly = false;
}
WriteDerivationTest()
: WriteDerivationTest(make_ref<DummyStoreConfig>(DummyStoreConfig::Params{}))
{
}
ref<DummyStoreConfig> config;
};
static Derivation makeSimpleDrv()
{
Derivation drv;
drv.name = "simple-derivation";
drv.platform = "system";
drv.builder = "foo";
drv.args = {"bar", "baz"};
drv.env = StringPairs{{"BIG_BAD", "WOLF"}};
return drv;
}
} // namespace
TEST_F(WriteDerivationTest, addToStoreFromDumpCalledOnce)
{
auto drv = makeSimpleDrv();
auto path1 = writeDerivation(*store, drv, NoRepair);
config->readOnly = true;
auto path2 = writeDerivation(*store, drv, NoRepair);
EXPECT_EQ(path1, path2);
EXPECT_THAT(
[&] { writeDerivation(*store, drv, Repair); },
::testing::ThrowsMessage<Error>(testing::HasSubstrIgnoreANSIMatcher(
"operation 'addToStoreFromDump' is not supported by store 'dummy://'")));
}
} // namespace nix
+86 -88
View File
@@ -1,6 +1,6 @@
#include "nix/store/aws-creds.hh"
#if NIX_WITH_CURL_S3
#if NIX_WITH_AWS_AUTH
# include <aws/crt/Types.h>
# include "nix/store/s3-url.hh"
@@ -22,36 +22,14 @@
namespace nix {
namespace {
static void initAwsCrt()
AwsAuthError::AwsAuthError(int errorCode)
: Error("AWS authentication error: '%s' (%d)", aws_error_str(errorCode), errorCode)
, errorCode(errorCode)
{
struct CrtWrapper
{
Aws::Crt::ApiHandle apiHandle;
CrtWrapper()
{
apiHandle.InitializeLogging(Aws::Crt::LogLevel::Warn, static_cast<FILE *>(nullptr));
}
~CrtWrapper()
{
try {
// CRITICAL: Clear credential provider cache BEFORE AWS CRT shuts down
// This ensures all providers (which hold references to ClientBootstrap)
// are destroyed while AWS CRT is still valid
clearAwsCredentialsCache();
// Now it's safe for ApiHandle destructor to run
} catch (...) {
ignoreExceptionInDestructor();
}
}
};
static CrtWrapper crt;
}
namespace {
static AwsCredentials getCredentialsFromProvider(std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider> provider)
{
if (!provider || !provider->IsValid()) {
@@ -63,8 +41,7 @@ static AwsCredentials getCredentialsFromProvider(std::shared_ptr<Aws::Crt::Auth:
provider->GetCredentials([prom](std::shared_ptr<Aws::Crt::Auth::Credentials> credentials, int errorCode) {
if (errorCode != 0 || !credentials) {
prom->set_exception(
std::make_exception_ptr(AwsAuthError("Failed to resolve AWS credentials: error code %d", errorCode)));
prom->set_exception(std::make_exception_ptr(AwsAuthError(errorCode)));
} else {
auto accessKeyId = Aws::Crt::ByteCursorToStringView(credentials->GetAccessKeyId());
auto secretAccessKey = Aws::Crt::ByteCursorToStringView(credentials->GetSecretAccessKey());
@@ -95,82 +72,103 @@ static AwsCredentials getCredentialsFromProvider(std::shared_ptr<Aws::Crt::Auth:
return fut.get(); // This will throw if set_exception was called
}
// Global credential provider cache using boost's concurrent map
// Key: profile name (empty string for default profile)
using CredentialProviderCache =
boost::concurrent_flat_map<std::string, std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider>>;
static CredentialProviderCache credentialProviderCache;
} // anonymous namespace
AwsCredentials getAwsCredentials(const std::string & profile)
class AwsCredentialProviderImpl : public AwsCredentialProvider
{
public:
AwsCredentialProviderImpl()
{
apiHandle.InitializeLogging(Aws::Crt::LogLevel::Warn, static_cast<FILE *>(nullptr));
}
AwsCredentials getCredentialsRaw(const std::string & profile);
AwsCredentials getCredentials(const ParsedS3URL & url) override
{
auto profile = url.profile.value_or("");
try {
return getCredentialsRaw(profile);
} catch (AwsAuthError & e) {
warn("AWS authentication failed for S3 request %s: %s", url.toHttpsUrl(), e.message());
credentialProviderCache.erase(profile);
throw;
}
}
private:
Aws::Crt::ApiHandle apiHandle;
boost::concurrent_flat_map<std::string, std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider>>
credentialProviderCache;
};
AwsCredentials AwsCredentialProviderImpl::getCredentialsRaw(const std::string & profile)
{
// Get or create credential provider with caching
std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider> provider;
// Try to find existing provider
credentialProviderCache.visit(profile, [&](const auto & pair) { provider = pair.second; });
// Use try_emplace_and_cvisit for atomic get-or-create
// This prevents race conditions where multiple threads create providers
credentialProviderCache.try_emplace_and_cvisit(
profile,
nullptr, // Placeholder - will be replaced in f1 before any thread can see it
[&](auto & kv) {
// f1: Called atomically during insertion with non-const reference
// Other threads are blocked until we finish, so nullptr is never visible
debug(
"[pid=%d] creating new AWS credential provider for profile '%s'",
getpid(),
profile.empty() ? "(default)" : profile.c_str());
if (!provider) {
// Create new provider if not found
debug(
"[pid=%d] creating new AWS credential provider for profile '%s'",
getpid(),
profile.empty() ? "(default)" : profile.c_str());
try {
if (profile.empty()) {
Aws::Crt::Auth::CredentialsProviderChainDefaultConfig config;
config.Bootstrap = Aws::Crt::ApiHandle::GetOrCreateStaticDefaultClientBootstrap();
kv.second = Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderChainDefault(config);
} else {
Aws::Crt::Auth::CredentialsProviderProfileConfig config;
config.Bootstrap = Aws::Crt::ApiHandle::GetOrCreateStaticDefaultClientBootstrap();
// This is safe because the underlying C library will copy this string
// c.f. https://github.com/awslabs/aws-c-auth/blob/main/source/credentials_provider_profile.c#L220
config.ProfileNameOverride = Aws::Crt::ByteCursorFromCString(profile.c_str());
kv.second = Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderProfile(config);
}
try {
initAwsCrt();
if (!kv.second) {
throw AwsAuthError(
"Failed to create AWS credentials provider for %s",
profile.empty() ? "default profile" : fmt("profile '%s'", profile));
}
if (profile.empty()) {
Aws::Crt::Auth::CredentialsProviderChainDefaultConfig config;
config.Bootstrap = Aws::Crt::ApiHandle::GetOrCreateStaticDefaultClientBootstrap();
provider = Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderChainDefault(config);
} else {
Aws::Crt::Auth::CredentialsProviderProfileConfig config;
config.Bootstrap = Aws::Crt::ApiHandle::GetOrCreateStaticDefaultClientBootstrap();
// This is safe because the underlying C library will copy this string
// c.f. https://github.com/awslabs/aws-c-auth/blob/main/source/credentials_provider_profile.c#L220
config.ProfileNameOverride = Aws::Crt::ByteCursorFromCString(profile.c_str());
provider = Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderProfile(config);
provider = kv.second;
} catch (Error & e) {
// Exception during creation - remove the entry to allow retry
credentialProviderCache.erase(profile);
e.addTrace({}, "for AWS profile: %s", profile.empty() ? "(default)" : profile);
throw;
} catch (...) {
// Non-Error exception - still need to clean up
credentialProviderCache.erase(profile);
throw;
}
} catch (Error & e) {
e.addTrace(
{},
"while creating AWS credentials provider for %s",
profile.empty() ? "default profile" : fmt("profile '%s'", profile));
throw;
}
if (!provider) {
throw AwsAuthError(
"Failed to create AWS credentials provider for %s",
profile.empty() ? "default profile" : fmt("profile '%s'", profile));
}
// Insert into cache (try_emplace is thread-safe and won't overwrite if another thread added it)
credentialProviderCache.try_emplace(profile, provider);
}
},
[&](const auto & kv) {
// f2: Called if key already exists (const reference)
provider = kv.second;
});
return getCredentialsFromProvider(provider);
}
void invalidateAwsCredentials(const std::string & profile)
ref<AwsCredentialProvider> makeAwsCredentialsProvider()
{
credentialProviderCache.erase(profile);
return make_ref<AwsCredentialProviderImpl>();
}
void clearAwsCredentialsCache()
ref<AwsCredentialProvider> getAwsCredentialsProvider()
{
credentialProviderCache.clear();
}
AwsCredentials preResolveAwsCredentials(const ParsedS3URL & s3Url)
{
std::string profile = s3Url.profile.value_or("");
// Get credentials (automatically cached)
return getAwsCredentials(profile);
static auto instance = makeAwsCredentialsProvider();
return instance;
}
} // namespace nix
+13 -8
View File
@@ -502,10 +502,15 @@ StorePath BinaryCacheStore::addToStore(
->path;
}
void BinaryCacheStore::queryRealisationUncached(
const DrvOutput & id, Callback<std::shared_ptr<const Realisation>> callback) noexcept
std::string BinaryCacheStore::makeRealisationPath(const DrvOutput & id)
{
auto outputInfoFilePath = realisationsPrefix + "/" + id.to_string() + ".doi";
return realisationsPrefix + "/" + id.to_string() + ".doi";
}
void BinaryCacheStore::queryRealisationUncached(
const DrvOutput & id, Callback<std::shared_ptr<const UnkeyedRealisation>> callback) noexcept
{
auto outputInfoFilePath = makeRealisationPath(id);
auto callbackPtr = std::make_shared<decltype(callback)>(std::move(callback));
@@ -515,11 +520,12 @@ void BinaryCacheStore::queryRealisationUncached(
if (!data)
return (*callbackPtr)({});
std::shared_ptr<const Realisation> realisation;
std::shared_ptr<const UnkeyedRealisation> realisation;
try {
realisation = std::make_shared<const Realisation>(nlohmann::json::parse(*data));
realisation = std::make_shared<const UnkeyedRealisation>(nlohmann::json::parse(*data));
} catch (Error & e) {
e.addTrace({}, "while parsing file '%s' as a realisation", outputInfoFilePath);
e.addTrace(
{}, "while parsing file '%s' as a realisation for key '%s'", outputInfoFilePath, id.to_string());
throw;
}
return (*callbackPtr)(std::move(realisation));
@@ -535,8 +541,7 @@ void BinaryCacheStore::registerDrvOutput(const Realisation & info)
{
if (diskCache)
diskCache->upsertRealisation(config.getReference().render(/*FIXME withParams=*/false), info);
auto filePath = realisationsPrefix + "/" + info.id.to_string() + ".doi";
upsertFile(filePath, static_cast<nlohmann::json>(info).dump(), "application/json");
upsertFile(makeRealisationPath(info.id), static_cast<nlohmann::json>(info).dump(), "application/json");
}
ref<RemoteFSAccessor> BinaryCacheStore::getRemoteFSAccessor(bool requireValidPath)
+27
View File
@@ -0,0 +1,27 @@
#include "nix/util/json-utils.hh"
#include "nix/store/build/derivation-builder.hh"
namespace nlohmann {
using namespace nix;
ExternalBuilder adl_serializer<ExternalBuilder>::from_json(const json & json)
{
auto obj = getObject(json);
return {
.systems = valueAt(obj, "systems"),
.program = valueAt(obj, "program"),
.args = valueAt(obj, "args"),
};
}
void adl_serializer<ExternalBuilder>::to_json(json & json, const ExternalBuilder & eb)
{
json = {
{"systems", eb.systems},
{"program", eb.program},
{"args", eb.args},
};
}
} // namespace nlohmann
+92 -265
View File
@@ -1,6 +1,5 @@
#include "nix/store/build/derivation-building-goal.hh"
#include "nix/store/build/derivation-env-desugar.hh"
#include "nix/store/build/derivation-trampoline-goal.hh"
#ifndef _WIN32 // TODO enable build hook on Windows
# include "nix/store/build/hook-instance.hh"
# include "nix/store/build/derivation-builder.hh"
@@ -27,22 +26,21 @@
namespace nix {
DerivationBuildingGoal::DerivationBuildingGoal(
const StorePath & drvPath, const Derivation & drv_, Worker & worker, BuildMode buildMode)
: Goal(worker, gaveUpOnSubstitution())
const StorePath & drvPath, const Derivation & drv, Worker & worker, BuildMode buildMode, bool storeDerivation)
: Goal(worker, gaveUpOnSubstitution(storeDerivation))
, drvPath(drvPath)
, drv{std::make_unique<Derivation>(drv)}
, buildMode(buildMode)
{
drv = std::make_unique<Derivation>(drv_);
try {
drvOptions =
std::make_unique<DerivationOptions>(DerivationOptions::fromStructuredAttrs(drv->env, drv->structuredAttrs));
std::make_unique<DerivationOptions>(DerivationOptions::fromStructuredAttrs(drv.env, drv.structuredAttrs));
} catch (Error & e) {
e.addTrace({}, "while parsing derivation '%s'", worker.store.printStorePath(drvPath));
throw;
}
name = fmt("building of '%s' from in-memory derivation", worker.store.printStorePath(drvPath));
name = fmt("building derivation '%s'", worker.store.printStorePath(drvPath));
trace("created");
/* Prevent the .chroot directory from being
@@ -67,11 +65,7 @@ DerivationBuildingGoal::~DerivationBuildingGoal()
std::string DerivationBuildingGoal::key()
{
/* Ensure that derivations get built in order of their name,
i.e. a derivation named "aardvark" always comes before
"baboon". And substitution goals always happen before
derivation goals (due to "bd$"). */
return "bd$" + std::string(drvPath.name()) + "$" + worker.store.printStorePath(drvPath);
return "dd$" + std::string(drvPath.name()) + "$" + worker.store.printStorePath(drvPath);
}
void DerivationBuildingGoal::killChild()
@@ -93,18 +87,6 @@ void DerivationBuildingGoal::timedOut(Error && ex)
[[maybe_unused]] Done _ = doneFailure({BuildResult::Failure::TimedOut, std::move(ex)});
}
/**
* Used for `inputGoals` local variable below
*/
struct value_comparison
{
template<typename T>
bool operator()(const ref<T> & lhs, const ref<T> & rhs) const
{
return *lhs < *rhs;
}
};
std::string showKnownOutputs(const StoreDirConfig & store, const Derivation & drv)
{
std::string msg;
@@ -125,50 +107,10 @@ static void runPostBuildHook(
/* At least one of the output paths could not be
produced using a substitute. So we have to build instead. */
Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution()
Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution(bool storeDerivation)
{
Goals waitees;
std::map<ref<const SingleDerivedPath>, GoalPtr, value_comparison> inputGoals;
{
std::function<void(ref<const SingleDerivedPath>, const DerivedPathMap<StringSet>::ChildNode &)>
addWaiteeDerivedPath;
addWaiteeDerivedPath = [&](ref<const SingleDerivedPath> inputDrv,
const DerivedPathMap<StringSet>::ChildNode & inputNode) {
if (!inputNode.value.empty()) {
auto g = worker.makeGoal(
DerivedPath::Built{
.drvPath = inputDrv,
.outputs = inputNode.value,
},
buildMode == bmRepair ? bmRepair : bmNormal);
inputGoals.insert_or_assign(inputDrv, g);
waitees.insert(std::move(g));
}
for (const auto & [outputName, childNode] : inputNode.childMap)
addWaiteeDerivedPath(
make_ref<SingleDerivedPath>(SingleDerivedPath::Built{inputDrv, outputName}), childNode);
};
for (const auto & [inputDrvPath, inputNode] : drv->inputDrvs.map) {
/* Ensure that pure, non-fixed-output derivations don't
depend on impure derivations. */
if (experimentalFeatureSettings.isEnabled(Xp::ImpureDerivations) && !drv->type().isImpure()
&& !drv->type().isFixed()) {
auto inputDrv = worker.evalStore.readDerivation(inputDrvPath);
if (inputDrv.type().isImpure())
throw Error(
"pure derivation '%s' depends on impure derivation '%s'",
worker.store.printStorePath(drvPath),
worker.store.printStorePath(inputDrvPath));
}
addWaiteeDerivedPath(makeConstantStorePathRef(inputDrvPath), inputNode);
}
}
/* Copy the input sources from the eval store to the build
store.
@@ -213,177 +155,17 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution()
/* Determine the full set of input paths. */
/* First, the input derivations. */
if (storeDerivation) {
assert(drv->inputDrvs.map.empty());
/* Store the resolved derivation, as part of the record of
what we're actually building */
writeDerivation(worker.store, *drv);
}
{
auto & fullDrv = *drv;
auto drvType = fullDrv.type();
bool resolveDrv =
std::visit(
overloaded{
[&](const DerivationType::InputAddressed & ia) {
/* must resolve if deferred. */
return ia.deferred;
},
[&](const DerivationType::ContentAddressed & ca) {
return !fullDrv.inputDrvs.map.empty()
&& (ca.fixed
/* Can optionally resolve if fixed, which is good
for avoiding unnecessary rebuilds. */
? experimentalFeatureSettings.isEnabled(Xp::CaDerivations)
/* Must resolve if floating and there are any inputs
drvs. */
: true);
},
[&](const DerivationType::Impure &) { return true; }},
drvType.raw)
/* no inputs are outputs of dynamic derivations */
|| std::ranges::any_of(fullDrv.inputDrvs.map.begin(), fullDrv.inputDrvs.map.end(), [](auto & pair) {
return !pair.second.childMap.empty();
});
if (resolveDrv && !fullDrv.inputDrvs.map.empty()) {
experimentalFeatureSettings.require(Xp::CaDerivations);
/* We are be able to resolve this derivation based on the
now-known results of dependencies. If so, we become a
stub goal aliasing that resolved derivation goal. */
std::optional attempt = fullDrv.tryResolve(
worker.store,
[&](ref<const SingleDerivedPath> drvPath, const std::string & outputName) -> std::optional<StorePath> {
auto mEntry = get(inputGoals, drvPath);
if (!mEntry)
return std::nullopt;
auto & buildResult = (*mEntry)->buildResult;
return std::visit(
overloaded{
[](const BuildResult::Failure &) -> std::optional<StorePath> { return std::nullopt; },
[&](const BuildResult::Success & success) -> std::optional<StorePath> {
auto i = get(success.builtOutputs, outputName);
if (!i)
return std::nullopt;
return i->outPath;
},
},
buildResult.inner);
});
if (!attempt) {
/* TODO (impure derivations-induced tech debt) (see below):
The above attempt should have found it, but because we manage
inputDrvOutputs statefully, sometimes it gets out of sync with
the real source of truth (store). So we query the store
directly if there's a problem. */
attempt = fullDrv.tryResolve(worker.store, &worker.evalStore);
}
assert(attempt);
Derivation drvResolved{std::move(*attempt)};
auto pathResolved = writeDerivation(worker.store, drvResolved);
auto msg =
fmt("resolved derivation: '%s' -> '%s'",
worker.store.printStorePath(drvPath),
worker.store.printStorePath(pathResolved));
act = std::make_unique<Activity>(
*logger,
lvlInfo,
actBuildWaiting,
msg,
Logger::Fields{
worker.store.printStorePath(drvPath),
worker.store.printStorePath(pathResolved),
});
/* TODO https://github.com/NixOS/nix/issues/13247 we should
let the calling goal do this, so it has a change to pass
just the output(s) it cares about. */
auto resolvedDrvGoal =
worker.makeDerivationTrampolineGoal(pathResolved, OutputsSpec::All{}, drvResolved, buildMode);
{
Goals waitees{resolvedDrvGoal};
co_await await(std::move(waitees));
}
trace("resolved derivation finished");
auto resolvedResult = resolvedDrvGoal->buildResult;
// No `std::visit` for coroutines yet
if (auto * successP = resolvedResult.tryGetSuccess()) {
auto & success = *successP;
SingleDrvOutputs builtOutputs;
auto outputHashes = staticOutputHashes(worker.evalStore, *drv);
auto resolvedHashes = staticOutputHashes(worker.store, drvResolved);
StorePathSet outputPaths;
for (auto & outputName : drvResolved.outputNames()) {
auto outputHash = get(outputHashes, outputName);
auto resolvedHash = get(resolvedHashes, outputName);
if ((!outputHash) || (!resolvedHash))
throw Error(
"derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/resolve)",
worker.store.printStorePath(drvPath),
outputName);
auto realisation = [&] {
auto take1 = get(success.builtOutputs, outputName);
if (take1)
return *take1;
/* The above `get` should work. But stateful tracking of
outputs in resolvedResult, this can get out of sync with the
store, which is our actual source of truth. For now we just
check the store directly if it fails. */
auto take2 = worker.evalStore.queryRealisation(DrvOutput{*resolvedHash, outputName});
if (take2)
return *take2;
throw Error(
"derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/realisation)",
worker.store.printStorePath(pathResolved),
outputName);
}();
if (!drv->type().isImpure()) {
auto newRealisation = realisation;
newRealisation.id = DrvOutput{*outputHash, outputName};
newRealisation.signatures.clear();
if (!drv->type().isFixed()) {
auto & drvStore = worker.evalStore.isValidPath(drvPath) ? worker.evalStore : worker.store;
newRealisation.dependentRealisations =
drvOutputReferences(worker.store, *drv, realisation.outPath, &drvStore);
}
worker.store.signRealisation(newRealisation);
worker.store.registerDrvOutput(newRealisation);
}
outputPaths.insert(realisation.outPath);
builtOutputs.emplace(outputName, realisation);
}
runPostBuildHook(worker.store, *logger, drvPath, outputPaths);
auto status = success.status;
if (status == BuildResult::Success::AlreadyValid)
status = BuildResult::Success::ResolvesToAlreadyValid;
co_return doneSuccess(success.status, std::move(builtOutputs));
} else if (resolvedResult.tryGetFailure()) {
co_return doneFailure({
BuildResult::Failure::DependencyFailed,
"build of resolved derivation '%s' failed",
worker.store.printStorePath(pathResolved),
});
} else
assert(false);
}
/* If we get this far, we know no dynamic drvs inputs */
for (auto & [depDrvPath, depNode] : fullDrv.inputDrvs.map) {
for (auto & [depDrvPath, depNode] : drv->inputDrvs.map) {
for (auto & outputName : depNode.value) {
/* Don't need to worry about `inputGoals`, because
impure derivations are always resolved above. Can
@@ -491,6 +273,8 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
bool useHook;
const ExternalBuilder * externalBuilder = nullptr;
while (true) {
trace("trying to build");
@@ -584,7 +368,44 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
co_await waitForAWhile();
continue;
case rpDecline:
/* We should do it ourselves. */
/* We should do it ourselves.
Now that we've decided we can't / won't do a remote build, check
that we can in fact build locally. First see if there is an
external builder for a "semi-local build". If there is, prefer to
use that. If there is not, then check if we can do a "true" local
build. */
externalBuilder = settings.findExternalDerivationBuilderIfSupported(*drv);
if (!externalBuilder && !drvOptions->canBuildLocally(worker.store, *drv)) {
auto msg =
fmt("Cannot build '%s'.\n"
"Reason: " ANSI_RED "required system or feature not available" ANSI_NORMAL
"\n"
"Required system: '%s' with features {%s}\n"
"Current system: '%s' with features {%s}",
Magenta(worker.store.printStorePath(drvPath)),
Magenta(drv->platform),
concatStringsSep(", ", drvOptions->getRequiredSystemFeatures(*drv)),
Magenta(settings.thisSystem),
concatStringsSep<StringSet>(", ", worker.store.Store::config.systemFeatures));
// since aarch64-darwin has Rosetta 2, this user can actually run x86_64-darwin on their hardware -
// we should tell them to run the command to install Darwin 2
if (drv->platform == "x86_64-darwin" && settings.thisSystem == "aarch64-darwin")
msg += fmt(
"\nNote: run `%s` to run programs for x86_64-darwin",
Magenta(
"/usr/sbin/softwareupdate --install-rosetta && launchctl stop org.nixos.nix-daemon"));
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
builder.reset();
#endif
outputLocks.unlock();
worker.permanentFailure = true;
co_return doneFailure({BuildResult::Failure::InputRejected, std::move(msg)});
}
useHook = false;
break;
}
@@ -771,36 +592,35 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
co_return doneFailure(std::move(e));
}
DerivationBuilderParams params{
.drvPath = drvPath,
.buildResult = buildResult,
.drv = *drv,
.drvOptions = *drvOptions,
.inputPaths = inputPaths,
.initialOutputs = initialOutputs,
.buildMode = buildMode,
.defaultPathsInChroot = std::move(defaultPathsInChroot),
.systemFeatures = worker.store.config.systemFeatures.get(),
.desugaredEnv = std::move(desugaredEnv),
};
/* If we have to wait and retry (see below), then `builder` will
already be created, so we don't need to create it again. */
builder = makeDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
DerivationBuilderParams{
.drvPath = drvPath,
.buildResult = buildResult,
.drv = *drv,
.drvOptions = *drvOptions,
.inputPaths = inputPaths,
.initialOutputs = initialOutputs,
.buildMode = buildMode,
.defaultPathsInChroot = std::move(defaultPathsInChroot),
.systemFeatures = worker.store.config.systemFeatures.get(),
.desugaredEnv = std::move(desugaredEnv),
});
builder = externalBuilder ? makeExternalDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
std::move(params),
*externalBuilder)
: makeDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
std::move(params));
}
std::optional<Descriptor> builderOutOpt;
try {
/* Okay, we have to build. */
builderOutOpt = builder->startBuild();
} catch (BuildError & e) {
builder.reset();
outputLocks.unlock();
worker.permanentFailure = true;
co_return doneFailure(std::move(e)); // InputRejected
}
if (!builderOutOpt) {
if (auto builderOutOpt = builder->startBuild()) {
builderOut = *std::move(builderOutOpt);
} else {
if (!actLock)
actLock = std::make_unique<Activity>(
*logger,
@@ -809,9 +629,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
fmt("waiting for a free build user ID for '%s'", Magenta(worker.store.printStorePath(drvPath))));
co_await waitForAWhile();
continue;
} else {
builderOut = *std::move(builderOutOpt);
};
}
break;
}
@@ -1293,13 +1111,22 @@ DerivationBuildingGoal::checkPathValidity(std::map<std::string, InitialOutput> &
// without the `ca-derivations` experimental flag).
worker.store.registerDrvOutput(
Realisation{
{
.outPath = info.known->path,
},
drvOutput,
info.known->path,
});
}
}
if (info.known && info.known->isValid())
validOutputs.emplace(i.first, Realisation{drvOutput, info.known->path});
validOutputs.emplace(
i.first,
Realisation{
{
.outPath = info.known->path,
},
drvOutput,
});
}
bool allValid = true;
+137 -35
View File
@@ -1,5 +1,6 @@
#include "nix/store/build/derivation-goal.hh"
#include "nix/store/build/derivation-building-goal.hh"
#include "nix/store/build/derivation-resolution-goal.hh"
#ifndef _WIN32 // TODO enable build hook on Windows
# include "nix/store/build/hook-instance.hh"
# include "nix/store/build/derivation-builder.hh"
@@ -29,10 +30,12 @@ DerivationGoal::DerivationGoal(
const Derivation & drv,
const OutputName & wantedOutput,
Worker & worker,
BuildMode buildMode)
: Goal(worker, haveDerivation())
BuildMode buildMode,
bool storeDerivation)
: Goal(worker, haveDerivation(storeDerivation))
, drvPath(drvPath)
, wantedOutput(wantedOutput)
, drv{std::make_unique<Derivation>(drv)}
, outputHash{[&] {
auto outputHashes = staticOutputHashes(worker.evalStore, drv);
if (auto * mOutputHash = get(outputHashes, wantedOutput))
@@ -41,11 +44,8 @@ DerivationGoal::DerivationGoal(
}()}
, buildMode(buildMode)
{
this->drv = std::make_unique<Derivation>(drv);
name =
fmt("building of '%s' from in-memory derivation",
DerivedPath::Built{makeConstantStorePathRef(drvPath), drv.outputNames()}.to_string(worker.store));
name = fmt("getting output '%s' from derivation '%s'", wantedOutput, worker.store.printStorePath(drvPath));
trace("created");
mcExpectedBuilds = std::make_unique<MaintainCount<uint64_t>>(worker.expectedBuilds);
@@ -54,17 +54,13 @@ DerivationGoal::DerivationGoal(
std::string DerivationGoal::key()
{
/* Ensure that derivations get built in order of their name,
i.e. a derivation named "aardvark" always comes before
"baboon". And substitution goals always happen before
derivation goals (due to "b$"). */
return "b$" + std::string(drvPath.name()) + "$" + SingleDerivedPath::Built{
return "db$" + std::string(drvPath.name()) + "$" + SingleDerivedPath::Built{
.drvPath = makeConstantStorePathRef(drvPath),
.output = wantedOutput,
}.to_string(worker.store);
}
Goal::Co DerivationGoal::haveDerivation()
Goal::Co DerivationGoal::haveDerivation(bool storeDerivation)
{
trace("have derivation");
@@ -104,8 +100,7 @@ Goal::Co DerivationGoal::haveDerivation()
them. */
if (settings.useSubstitutes && drvOptions.substitutesAllowed()) {
if (!checkResult)
waitees.insert(upcast_goal(worker.makeDrvOutputSubstitutionGoal(
DrvOutput{outputHash, wantedOutput}, buildMode == bmRepair ? Repair : NoRepair)));
waitees.insert(upcast_goal(worker.makeDrvOutputSubstitutionGoal(DrvOutput{outputHash, wantedOutput})));
else {
auto * cap = getDerivationCA(*drv);
waitees.insert(upcast_goal(worker.makePathSubstitutionGoal(
@@ -146,9 +141,101 @@ Goal::Co DerivationGoal::haveDerivation()
worker.store.printStorePath(drvPath));
}
auto resolutionGoal = worker.makeDerivationResolutionGoal(drvPath, *drv, buildMode);
{
Goals waitees{resolutionGoal};
co_await await(std::move(waitees));
}
if (nrFailed != 0) {
co_return doneFailure({BuildResult::Failure::DependencyFailed, "resolution failed"});
}
if (resolutionGoal->resolvedDrv) {
auto & [pathResolved, drvResolved] = *resolutionGoal->resolvedDrv;
auto resolvedDrvGoal =
worker.makeDerivationGoal(pathResolved, drvResolved, wantedOutput, buildMode, /*storeDerivation=*/true);
{
Goals waitees{resolvedDrvGoal};
co_await await(std::move(waitees));
}
trace("resolved derivation finished");
auto resolvedResult = resolvedDrvGoal->buildResult;
// No `std::visit` for coroutines yet
if (auto * successP = resolvedResult.tryGetSuccess()) {
auto & success = *successP;
auto outputHashes = staticOutputHashes(worker.evalStore, *drv);
auto resolvedHashes = staticOutputHashes(worker.store, drvResolved);
auto outputHash = get(outputHashes, wantedOutput);
auto resolvedHash = get(resolvedHashes, wantedOutput);
if ((!outputHash) || (!resolvedHash))
throw Error(
"derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/resolve)",
worker.store.printStorePath(drvPath),
wantedOutput);
auto realisation = [&] {
auto take1 = get(success.builtOutputs, wantedOutput);
if (take1)
return static_cast<UnkeyedRealisation>(*take1);
/* The above `get` should work. But stateful tracking of
outputs in resolvedResult, this can get out of sync with the
store, which is our actual source of truth. For now we just
check the store directly if it fails. */
auto take2 = worker.evalStore.queryRealisation(
DrvOutput{
.drvHash = *resolvedHash,
.outputName = wantedOutput,
});
if (take2)
return *take2;
throw Error(
"derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/realisation)",
worker.store.printStorePath(pathResolved),
wantedOutput);
}();
if (!drv->type().isImpure()) {
Realisation newRealisation{
realisation,
{
.drvHash = *outputHash,
.outputName = wantedOutput,
}};
newRealisation.signatures.clear();
if (!drv->type().isFixed()) {
auto & drvStore = worker.evalStore.isValidPath(drvPath) ? worker.evalStore : worker.store;
newRealisation.dependentRealisations =
drvOutputReferences(worker.store, *drv, realisation.outPath, &drvStore);
}
worker.store.signRealisation(newRealisation);
worker.store.registerDrvOutput(newRealisation);
}
auto status = success.status;
if (status == BuildResult::Success::AlreadyValid)
status = BuildResult::Success::ResolvesToAlreadyValid;
co_return doneSuccess(status, std::move(realisation));
} else if (resolvedResult.tryGetFailure()) {
co_return doneFailure({
BuildResult::Failure::DependencyFailed,
"build of resolved derivation '%s' failed",
worker.store.printStorePath(pathResolved),
});
} else
assert(false);
}
/* Give up on substitution for the output we want, actually build this derivation */
auto g = worker.makeDerivationBuildingGoal(drvPath, *drv, buildMode);
auto g = worker.makeDerivationBuildingGoal(drvPath, *drv, buildMode, storeDerivation);
/* We will finish with it ourselves, as if we were the derivational goal. */
g->preserveException = true;
@@ -169,7 +256,16 @@ Goal::Co DerivationGoal::haveDerivation()
/* In checking mode, the builder will not register any outputs.
So we want to make sure the ones that we wanted to check are
properly there. */
success.builtOutputs = {{wantedOutput, assertPathValidity()}};
success.builtOutputs = {{
wantedOutput,
{
assertPathValidity(),
{
.drvHash = outputHash,
.outputName = wantedOutput,
},
},
}};
} else {
/* Otherwise the builder will give us info for out output, but
also for other outputs. Filter down to just our output so as
@@ -189,18 +285,6 @@ Goal::Co DerivationGoal::haveDerivation()
co_return amDone(g->exitCode, g->ex);
}
/**
* Used for `inputGoals` local variable below
*/
struct value_comparison
{
template<typename T>
bool operator()(const ref<T> & lhs, const ref<T> & rhs) const
{
return *lhs < *rhs;
}
};
Goal::Co DerivationGoal::repairClosure()
{
assert(!drv->type().isImpure());
@@ -285,18 +369,20 @@ Goal::Co DerivationGoal::repairClosure()
co_return doneSuccess(BuildResult::Success::AlreadyValid, assertPathValidity());
}
std::optional<std::pair<Realisation, PathStatus>> DerivationGoal::checkPathValidity()
std::optional<std::pair<UnkeyedRealisation, PathStatus>> DerivationGoal::checkPathValidity()
{
if (drv->type().isImpure())
return std::nullopt;
auto drvOutput = DrvOutput{outputHash, wantedOutput};
std::optional<Realisation> mRealisation;
std::optional<UnkeyedRealisation> mRealisation;
if (auto * mOutput = get(drv->outputs, wantedOutput)) {
if (auto mPath = mOutput->path(worker.store, drv->name, wantedOutput)) {
mRealisation = Realisation{drvOutput, std::move(*mPath)};
mRealisation = UnkeyedRealisation{
.outPath = std::move(*mPath),
};
}
} else {
throw Error(
@@ -324,7 +410,14 @@ std::optional<std::pair<Realisation, PathStatus>> DerivationGoal::checkPathValid
// derivation, and the output path is valid, but we don't have
// its realisation stored (probably because it has been built
// without the `ca-derivations` experimental flag).
worker.store.registerDrvOutput(*mRealisation);
worker.store.registerDrvOutput(
Realisation{
*mRealisation,
{
.drvHash = outputHash,
.outputName = wantedOutput,
},
});
}
return {{*mRealisation, status}};
@@ -332,7 +425,7 @@ std::optional<std::pair<Realisation, PathStatus>> DerivationGoal::checkPathValid
return std::nullopt;
}
Realisation DerivationGoal::assertPathValidity()
UnkeyedRealisation DerivationGoal::assertPathValidity()
{
auto checkResult = checkPathValidity();
if (!(checkResult && checkResult->second == PathStatus::Valid))
@@ -340,11 +433,20 @@ Realisation DerivationGoal::assertPathValidity()
return checkResult->first;
}
Goal::Done DerivationGoal::doneSuccess(BuildResult::Success::Status status, Realisation builtOutput)
Goal::Done DerivationGoal::doneSuccess(BuildResult::Success::Status status, UnkeyedRealisation builtOutput)
{
buildResult.inner = BuildResult::Success{
.status = status,
.builtOutputs = {{wantedOutput, std::move(builtOutput)}},
.builtOutputs = {{
wantedOutput,
{
std::move(builtOutput),
DrvOutput{
.drvHash = outputHash,
.outputName = wantedOutput,
},
},
}},
};
mcExpectedBuilds.reset();
@@ -0,0 +1,191 @@
#include "nix/store/build/derivation-resolution-goal.hh"
#include "nix/store/build/worker.hh"
#include "nix/util/util.hh"
#include <nlohmann/json.hpp>
namespace nix {
DerivationResolutionGoal::DerivationResolutionGoal(
const StorePath & drvPath, const Derivation & drv, Worker & worker, BuildMode buildMode)
: Goal(worker, resolveDerivation())
, drvPath(drvPath)
, drv{std::make_unique<Derivation>(drv)}
, buildMode{buildMode}
{
name = fmt("resolving derivation '%s'", worker.store.printStorePath(drvPath));
trace("created");
}
std::string DerivationResolutionGoal::key()
{
return "dc$" + std::string(drvPath.name()) + "$" + worker.store.printStorePath(drvPath);
}
/**
* Used for `inputGoals` local variable below
*/
struct value_comparison
{
template<typename T>
bool operator()(const ref<T> & lhs, const ref<T> & rhs) const
{
return *lhs < *rhs;
}
};
Goal::Co DerivationResolutionGoal::resolveDerivation()
{
Goals waitees;
std::map<ref<const SingleDerivedPath>, GoalPtr, value_comparison> inputGoals;
{
std::function<void(ref<const SingleDerivedPath>, const DerivedPathMap<StringSet>::ChildNode &)>
addWaiteeDerivedPath;
addWaiteeDerivedPath = [&](ref<const SingleDerivedPath> inputDrv,
const DerivedPathMap<StringSet>::ChildNode & inputNode) {
if (!inputNode.value.empty()) {
auto g = worker.makeGoal(
DerivedPath::Built{
.drvPath = inputDrv,
.outputs = inputNode.value,
},
buildMode == bmRepair ? bmRepair : bmNormal);
inputGoals.insert_or_assign(inputDrv, g);
waitees.insert(std::move(g));
}
for (const auto & [outputName, childNode] : inputNode.childMap)
addWaiteeDerivedPath(
make_ref<SingleDerivedPath>(SingleDerivedPath::Built{inputDrv, outputName}), childNode);
};
for (const auto & [inputDrvPath, inputNode] : drv->inputDrvs.map) {
/* Ensure that pure, non-fixed-output derivations don't
depend on impure derivations. */
if (experimentalFeatureSettings.isEnabled(Xp::ImpureDerivations) && !drv->type().isImpure()
&& !drv->type().isFixed()) {
auto inputDrv = worker.evalStore.readDerivation(inputDrvPath);
if (inputDrv.type().isImpure())
throw Error(
"pure derivation '%s' depends on impure derivation '%s'",
worker.store.printStorePath(drvPath),
worker.store.printStorePath(inputDrvPath));
}
addWaiteeDerivedPath(makeConstantStorePathRef(inputDrvPath), inputNode);
}
}
co_await await(std::move(waitees));
trace("all inputs realised");
if (nrFailed != 0) {
auto msg =
fmt("Cannot build '%s'.\n"
"Reason: " ANSI_RED "%d %s failed" ANSI_NORMAL ".",
Magenta(worker.store.printStorePath(drvPath)),
nrFailed,
nrFailed == 1 ? "dependency" : "dependencies");
msg += showKnownOutputs(worker.store, *drv);
co_return amDone(ecFailed, {BuildError(BuildResult::Failure::DependencyFailed, msg)});
}
/* Gather information necessary for computing the closure and/or
running the build hook. */
/* Determine the full set of input paths. */
/* First, the input derivations. */
{
auto & fullDrv = *drv;
auto drvType = fullDrv.type();
bool resolveDrv =
std::visit(
overloaded{
[&](const DerivationType::InputAddressed & ia) {
/* must resolve if deferred. */
return ia.deferred;
},
[&](const DerivationType::ContentAddressed & ca) {
return !fullDrv.inputDrvs.map.empty()
&& (ca.fixed
/* Can optionally resolve if fixed, which is good
for avoiding unnecessary rebuilds. */
? experimentalFeatureSettings.isEnabled(Xp::CaDerivations)
/* Must resolve if floating and there are any inputs
drvs. */
: true);
},
[&](const DerivationType::Impure &) { return true; }},
drvType.raw)
/* no inputs are outputs of dynamic derivations */
|| std::ranges::any_of(fullDrv.inputDrvs.map.begin(), fullDrv.inputDrvs.map.end(), [](auto & pair) {
return !pair.second.childMap.empty();
});
if (resolveDrv && !fullDrv.inputDrvs.map.empty()) {
experimentalFeatureSettings.require(Xp::CaDerivations);
/* We are be able to resolve this derivation based on the
now-known results of dependencies. If so, we become a
stub goal aliasing that resolved derivation goal. */
std::optional attempt = fullDrv.tryResolve(
worker.store,
[&](ref<const SingleDerivedPath> drvPath, const std::string & outputName) -> std::optional<StorePath> {
auto mEntry = get(inputGoals, drvPath);
if (!mEntry)
return std::nullopt;
auto & buildResult = (*mEntry)->buildResult;
return std::visit(
overloaded{
[](const BuildResult::Failure &) -> std::optional<StorePath> { return std::nullopt; },
[&](const BuildResult::Success & success) -> std::optional<StorePath> {
auto i = get(success.builtOutputs, outputName);
if (!i)
return std::nullopt;
return i->outPath;
},
},
buildResult.inner);
});
if (!attempt) {
/* TODO (impure derivations-induced tech debt) (see below):
The above attempt should have found it, but because we manage
inputDrvOutputs statefully, sometimes it gets out of sync with
the real source of truth (store). So we query the store
directly if there's a problem. */
attempt = fullDrv.tryResolve(worker.store, &worker.evalStore);
}
assert(attempt);
auto pathResolved = writeDerivation(worker.store, *attempt, NoRepair, /*readOnly =*/true);
auto msg =
fmt("resolved derivation: '%s' -> '%s'",
worker.store.printStorePath(drvPath),
worker.store.printStorePath(pathResolved));
act = std::make_unique<Activity>(
*logger,
lvlInfo,
actBuildWaiting,
msg,
Logger::Fields{
worker.store.printStorePath(drvPath),
worker.store.printStorePath(pathResolved),
});
resolvedDrv =
std::make_unique<std::pair<StorePath, BasicDerivation>>(std::move(pathResolved), *std::move(attempt));
}
}
co_return amDone(ecSuccess, std::nullopt);
}
} // namespace nix
@@ -31,7 +31,7 @@ DerivationTrampolineGoal::DerivationTrampolineGoal(
void DerivationTrampolineGoal::commonInit()
{
name =
fmt("outer obtaining drv from '%s' and then building outputs %s",
fmt("obtaining derivation from '%s' and then building outputs %s",
drvReq->to_string(worker.store),
std::visit(
overloaded{
@@ -58,18 +58,12 @@ static StorePath pathPartOfReq(const SingleDerivedPath & req)
std::string DerivationTrampolineGoal::key()
{
/* Ensure that derivations get built in order of their name,
i.e. a derivation named "aardvark" always comes before "baboon". And
substitution goals, derivation goals, and derivation building goals always happen before
derivation goals (due to "bt$"). */
return "bt$" + std::string(pathPartOfReq(*drvReq).name()) + "$" + DerivedPath::Built{
return "da$" + std::string(pathPartOfReq(*drvReq).name()) + "$" + DerivedPath::Built{
.drvPath = drvReq,
.outputs = wantedOutputs,
}.to_string(worker.store);
}
void DerivationTrampolineGoal::timedOut(Error && ex) {}
Goal::Co DerivationTrampolineGoal::init()
{
trace("need to load derivation from file");
@@ -151,7 +145,7 @@ Goal::Co DerivationTrampolineGoal::haveDerivation(StorePath drvPath, Derivation
/* Build this step! */
for (auto & output : resolvedWantedOutputs) {
auto g = upcast_goal(worker.makeDerivationGoal(drvPath, drv, output, buildMode));
auto g = upcast_goal(worker.makeDerivationGoal(drvPath, drv, output, buildMode, false));
g->preserveException = true;
/* We will finish with it ourselves, as if we were the derivational goal. */
concreteDrvGoals.insert(std::move(g));
@@ -8,8 +8,7 @@
namespace nix {
DrvOutputSubstitutionGoal::DrvOutputSubstitutionGoal(
const DrvOutput & id, Worker & worker, RepairFlag repair, std::optional<ContentAddress> ca)
DrvOutputSubstitutionGoal::DrvOutputSubstitutionGoal(const DrvOutput & id, Worker & worker)
: Goal(worker, init())
, id(id)
{
@@ -43,10 +42,10 @@ Goal::Co DrvOutputSubstitutionGoal::init()
outPipe->createAsyncPipe(worker.ioport.get());
#endif
auto promise = std::make_shared<std::promise<std::shared_ptr<const Realisation>>>();
auto promise = std::make_shared<std::promise<std::shared_ptr<const UnkeyedRealisation>>>();
sub->queryRealisation(
id, {[outPipe(outPipe), promise(promise)](std::future<std::shared_ptr<const Realisation>> res) {
id, {[outPipe(outPipe), promise(promise)](std::future<std::shared_ptr<const UnkeyedRealisation>> res) {
try {
Finally updateStats([&]() { outPipe->writeSide.close(); });
promise->set_value(res.get());
@@ -75,7 +74,7 @@ Goal::Co DrvOutputSubstitutionGoal::init()
* The realisation corresponding to the given output id.
* Will be filled once we can get it.
*/
std::shared_ptr<const Realisation> outputInfo;
std::shared_ptr<const UnkeyedRealisation> outputInfo;
try {
outputInfo = promise->get_future().get();
@@ -132,7 +131,7 @@ Goal::Co DrvOutputSubstitutionGoal::init()
}
Goal::Co DrvOutputSubstitutionGoal::realisationFetched(
Goals waitees, std::shared_ptr<const Realisation> outputInfo, nix::ref<nix::Store> sub)
Goals waitees, std::shared_ptr<const UnkeyedRealisation> outputInfo, nix::ref<nix::Store> sub)
{
waitees.insert(worker.makePathSubstitutionGoal(outputInfo->outPath));
@@ -145,7 +144,7 @@ Goal::Co DrvOutputSubstitutionGoal::realisationFetched(
co_return amDone(nrNoSubstituters > 0 ? ecNoSubstituters : ecFailed);
}
worker.store.registerDrvOutput(*outputInfo);
worker.store.registerDrvOutput({*outputInfo, id});
trace("finished");
co_return amDone(ecSuccess);
@@ -153,8 +152,6 @@ Goal::Co DrvOutputSubstitutionGoal::realisationFetched(
std::string DrvOutputSubstitutionGoal::key()
{
/* "a$" ensures substitution goals happen before derivation
goals. */
return "a$" + std::string(id.to_string());
}
+24 -17
View File
@@ -4,6 +4,7 @@
#include "nix/store/build/substitution-goal.hh"
#include "nix/store/build/drv-output-substitution-goal.hh"
#include "nix/store/build/derivation-goal.hh"
#include "nix/store/build/derivation-resolution-goal.hh"
#include "nix/store/build/derivation-building-goal.hh"
#include "nix/store/build/derivation-trampoline-goal.hh"
#ifndef _WIN32 // TODO Enable building on Windows
@@ -75,15 +76,26 @@ std::shared_ptr<DerivationTrampolineGoal> Worker::makeDerivationTrampolineGoal(
}
std::shared_ptr<DerivationGoal> Worker::makeDerivationGoal(
const StorePath & drvPath, const Derivation & drv, const OutputName & wantedOutput, BuildMode buildMode)
const StorePath & drvPath,
const Derivation & drv,
const OutputName & wantedOutput,
BuildMode buildMode,
bool storeDerivation)
{
return initGoalIfNeeded(derivationGoals[drvPath][wantedOutput], drvPath, drv, wantedOutput, *this, buildMode);
return initGoalIfNeeded(
derivationGoals[drvPath][wantedOutput], drvPath, drv, wantedOutput, *this, buildMode, storeDerivation);
}
std::shared_ptr<DerivationBuildingGoal>
Worker::makeDerivationBuildingGoal(const StorePath & drvPath, const Derivation & drv, BuildMode buildMode)
std::shared_ptr<DerivationResolutionGoal>
Worker::makeDerivationResolutionGoal(const StorePath & drvPath, const Derivation & drv, BuildMode buildMode)
{
return initGoalIfNeeded(derivationBuildingGoals[drvPath], drvPath, drv, *this, buildMode);
return initGoalIfNeeded(derivationResolutionGoals[drvPath], drvPath, drv, *this, buildMode);
}
std::shared_ptr<DerivationBuildingGoal> Worker::makeDerivationBuildingGoal(
const StorePath & drvPath, const Derivation & drv, BuildMode buildMode, bool storeDerivation)
{
return initGoalIfNeeded(derivationBuildingGoals[drvPath], drvPath, drv, *this, buildMode, storeDerivation);
}
std::shared_ptr<PathSubstitutionGoal>
@@ -92,10 +104,9 @@ Worker::makePathSubstitutionGoal(const StorePath & path, RepairFlag repair, std:
return initGoalIfNeeded(substitutionGoals[path], path, *this, repair, ca);
}
std::shared_ptr<DrvOutputSubstitutionGoal>
Worker::makeDrvOutputSubstitutionGoal(const DrvOutput & id, RepairFlag repair, std::optional<ContentAddress> ca)
std::shared_ptr<DrvOutputSubstitutionGoal> Worker::makeDrvOutputSubstitutionGoal(const DrvOutput & id)
{
return initGoalIfNeeded(drvOutputSubstitutionGoals[id], id, *this, repair, ca);
return initGoalIfNeeded(drvOutputSubstitutionGoals[id], id, *this);
}
GoalPtr Worker::makeGoal(const DerivedPath & req, BuildMode buildMode)
@@ -158,6 +169,8 @@ void Worker::removeGoal(GoalPtr goal)
nix::removeGoal(drvGoal, derivationTrampolineGoals.map);
else if (auto drvGoal = std::dynamic_pointer_cast<DerivationGoal>(goal))
nix::removeGoal(drvGoal, derivationGoals);
else if (auto drvResolutionGoal = std::dynamic_pointer_cast<DerivationResolutionGoal>(goal))
nix::removeGoal(drvResolutionGoal, derivationResolutionGoals);
else if (auto drvBuildingGoal = std::dynamic_pointer_cast<DerivationBuildingGoal>(goal))
nix::removeGoal(drvBuildingGoal, derivationBuildingGoals);
else if (auto subGoal = std::dynamic_pointer_cast<PathSubstitutionGoal>(goal))
@@ -515,15 +528,9 @@ bool Worker::pathContentsGood(const StorePath & path)
return i->second;
printInfo("checking path '%s'...", store.printStorePath(path));
auto info = store.queryPathInfo(path);
bool res;
if (!pathExists(store.printStorePath(path)))
res = false;
else {
auto current = hashPath(
{store.getFSAccessor(), CanonPath(path.to_string())},
FileIngestionMethod::NixArchive,
info->narHash.algo)
.first;
bool res = false;
if (auto accessor = store.getFSAccessor(path, /*requireValidPath=*/false)) {
auto current = hashPath({ref{accessor}}, FileIngestionMethod::NixArchive, info->narHash.algo).first;
Hash nullHash(HashAlgorithm::SHA256);
res = info->narHash == nullHash || info->narHash == current;
}
+14 -1
View File
@@ -33,13 +33,26 @@ static void builtinFetchurl(const BuiltinBuilderContext & ctx)
/* Note: have to use a fresh fileTransfer here because we're in
a forked process. */
debug("[pid=%d] builtin:fetchurl creating fresh FileTransfer instance", getpid());
auto fileTransfer = makeFileTransfer();
auto fetch = [&](const std::string & url) {
auto source = sinkToSource([&](Sink & sink) {
FileTransferRequest request(ValidURL{url});
FileTransferRequest request(VerbatimURL{url});
request.decompress = false;
#if NIX_WITH_AWS_AUTH
// Use pre-resolved credentials if available
if (ctx.awsCredentials && request.uri.scheme() == "s3") {
debug("[pid=%d] Using pre-resolved AWS credentials from parent process", getpid());
request.usernameAuth = UsernameAuth{
.username = ctx.awsCredentials->accessKeyId,
.password = ctx.awsCredentials->secretAccessKey,
};
request.preResolvedAwsSessionToken = ctx.awsCredentials->sessionToken;
}
#endif
auto decompressor = makeDecompressionSink(unpack && hasSuffix(mainUrl, ".xz") ? "xz" : "none", sink);
fileTransfer->download(std::move(request), *decompressor);
decompressor->finish();
+31 -29
View File
@@ -312,7 +312,7 @@ static void performOp(
switch (op) {
case WorkerProto::Op::IsValidPath: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
bool result = store->isValidPath(path);
logger->stopWork();
@@ -339,7 +339,7 @@ static void performOp(
}
case WorkerProto::Op::HasSubstitutes: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
StorePathSet paths; // FIXME
paths.insert(path);
@@ -359,7 +359,7 @@ static void performOp(
}
case WorkerProto::Op::QueryPathHash: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
auto hash = store->queryPathInfo(path)->narHash;
logger->stopWork();
@@ -371,7 +371,7 @@ static void performOp(
case WorkerProto::Op::QueryReferrers:
case WorkerProto::Op::QueryValidDerivers:
case WorkerProto::Op::QueryDerivationOutputs: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
StorePathSet paths;
if (op == WorkerProto::Op::QueryReferences)
@@ -389,7 +389,7 @@ static void performOp(
}
case WorkerProto::Op::QueryDerivationOutputNames: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
auto names = store->readDerivation(path).outputNames();
logger->stopWork();
@@ -398,7 +398,7 @@ static void performOp(
}
case WorkerProto::Op::QueryDerivationOutputMap: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
auto outputs = store->queryPartialDerivationOutputMap(path);
logger->stopWork();
@@ -407,11 +407,11 @@ static void performOp(
}
case WorkerProto::Op::QueryDeriver: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
auto info = store->queryPathInfo(path);
logger->stopWork();
conn.to << (info->deriver ? store->printStorePath(*info->deriver) : "");
WorkerProto::write(*store, conn, info->deriver);
break;
}
@@ -420,7 +420,7 @@ static void performOp(
logger->startWork();
auto path = store->queryPathFromHashPart(hashPart);
logger->stopWork();
conn.to << (path ? store->printStorePath(*path) : "");
WorkerProto::write(*store, conn, path);
break;
}
@@ -505,7 +505,7 @@ static void performOp(
store->addToStoreFromDump(*dumpSource, baseName, FileSerialisationMethod::NixArchive, method, hashAlgo);
logger->stopWork();
conn.to << store->printStorePath(path);
WorkerProto::write(*store, wconn, path);
}
break;
}
@@ -542,7 +542,7 @@ static void performOp(
NoRepair);
});
logger->stopWork();
conn.to << store->printStorePath(path);
WorkerProto::write(*store, wconn, path);
break;
}
@@ -591,7 +591,7 @@ static void performOp(
}
case WorkerProto::Op::BuildDerivation: {
auto drvPath = store->parseStorePath(readString(conn.from));
auto drvPath = WorkerProto::Serialise<StorePath>::read(*store, rconn);
BasicDerivation drv;
/*
* Note: unlike wopEnsurePath, this operation reads a
@@ -668,7 +668,7 @@ static void performOp(
}
case WorkerProto::Op::EnsurePath: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
store->ensurePath(path);
logger->stopWork();
@@ -677,7 +677,7 @@ static void performOp(
}
case WorkerProto::Op::AddTempRoot: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
store->addTempRoot(path);
logger->stopWork();
@@ -733,8 +733,10 @@ static void performOp(
conn.to << size;
for (auto & [target, links] : roots)
for (auto & link : links)
conn.to << link << store->printStorePath(target);
for (auto & link : links) {
conn.to << link;
WorkerProto::write(*store, wconn, target);
}
break;
}
@@ -799,7 +801,7 @@ static void performOp(
}
case WorkerProto::Op::QuerySubstitutablePathInfo: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
SubstitutablePathInfos infos;
store->querySubstitutablePathInfos({{path, std::nullopt}}, infos);
@@ -808,7 +810,8 @@ static void performOp(
if (i == infos.end())
conn.to << 0;
else {
conn.to << 1 << (i->second.deriver ? store->printStorePath(*i->second.deriver) : "");
conn.to << 1;
WorkerProto::write(*store, wconn, i->second.deriver);
WorkerProto::write(*store, wconn, i->second.references);
conn.to << i->second.downloadSize << i->second.narSize;
}
@@ -829,8 +832,8 @@ static void performOp(
logger->stopWork();
conn.to << infos.size();
for (auto & i : infos) {
conn.to << store->printStorePath(i.first)
<< (i.second.deriver ? store->printStorePath(*i.second.deriver) : "");
WorkerProto::write(*store, wconn, i.first);
WorkerProto::write(*store, wconn, i.second.deriver);
WorkerProto::write(*store, wconn, i.second.references);
conn.to << i.second.downloadSize << i.second.narSize;
}
@@ -846,7 +849,7 @@ static void performOp(
}
case WorkerProto::Op::QueryPathInfo: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
std::shared_ptr<const ValidPathInfo> info;
logger->startWork();
info = store->queryPathInfo(path);
@@ -880,7 +883,7 @@ static void performOp(
}
case WorkerProto::Op::AddSignatures: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
StringSet sigs = readStrings<StringSet>(conn.from);
logger->startWork();
store->addSignatures(path, sigs);
@@ -890,7 +893,7 @@ static void performOp(
}
case WorkerProto::Op::NarFromPath: {
auto path = store->parseStorePath(readString(conn.from));
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
logger->startWork();
logger->stopWork();
dumpPath(store->toRealPath(path), conn.to);
@@ -899,12 +902,11 @@ static void performOp(
case WorkerProto::Op::AddToStoreNar: {
bool repair, dontCheckSigs;
auto path = store->parseStorePath(readString(conn.from));
auto deriver = readString(conn.from);
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
auto deriver = WorkerProto::Serialise<std::optional<StorePath>>::read(*store, rconn);
auto narHash = Hash::parseAny(readString(conn.from), HashAlgorithm::SHA256);
ValidPathInfo info{path, narHash};
if (deriver != "")
info.deriver = store->parseStorePath(deriver);
info.deriver = std::move(deriver);
info.references = WorkerProto::Serialise<StorePathSet>::read(*store, rconn);
conn.from >> info.registrationTime >> info.narSize >> info.ultimate;
info.sigs = readStrings<StringSet>(conn.from);
@@ -964,7 +966,7 @@ static void performOp(
if (GET_PROTOCOL_MINOR(conn.protoVersion) < 31) {
auto outputId = DrvOutput::parse(readString(conn.from));
auto outputPath = StorePath(readString(conn.from));
store->registerDrvOutput(Realisation{.id = outputId, .outPath = outputPath});
store->registerDrvOutput(Realisation{{.outPath = outputPath}, outputId});
} else {
auto realisation = WorkerProto::Serialise<Realisation>::read(*store, rconn);
store->registerDrvOutput(realisation);
@@ -986,7 +988,7 @@ static void performOp(
} else {
std::set<Realisation> realisations;
if (info)
realisations.insert(*info);
realisations.insert({*info, outputId});
WorkerProto::write(*store, wconn, realisations);
}
break;
+14 -6
View File
@@ -99,6 +99,17 @@ DerivationOptions DerivationOptions::fromStructuredAttrs(
return fromStructuredAttrs(env, parsed ? &*parsed : nullptr);
}
static void flatten(const nlohmann::json & value, StringSet & res)
{
if (value.is_array())
for (auto & v : value)
flatten(v, res);
else if (value.is_string())
res.insert(value);
else
throw Error("'exportReferencesGraph' value is not an array or a string");
}
DerivationOptions
DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAttrs * parsed, bool shouldWarn)
{
@@ -219,12 +230,9 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
if (!e || !e->is_object())
return ret;
for (auto & [key, value] : getObject(*e)) {
if (value.is_array())
ret.insert_or_assign(key, value);
else if (value.is_string())
ret.insert_or_assign(key, StringSet{value});
else
throw Error("'exportReferencesGraph' value is not an array or a string");
StringSet ss;
flatten(value, ss);
ret.insert_or_assign(key, std::move(ss));
}
} else {
auto s = getOr(env, "exportReferencesGraph", "");
+26 -21
View File
@@ -115,23 +115,25 @@ StorePath writeDerivation(Store & store, const Derivation & drv, RepairFlag repa
held during a garbage collection). */
auto suffix = std::string(drv.name) + drvExtension;
auto contents = drv.unparse(store, false);
return readOnly || settings.readOnlyMode ? store.makeFixedOutputPathFromCA(
suffix,
TextInfo{
.hash = hashString(HashAlgorithm::SHA256, contents),
.references = std::move(references),
})
: ({
StringSource s{contents};
store.addToStoreFromDump(
s,
suffix,
FileSerialisationMethod::Flat,
ContentAddressMethod::Raw::Text,
HashAlgorithm::SHA256,
references,
repair);
});
auto hash = hashString(HashAlgorithm::SHA256, contents);
auto ca = TextInfo{.hash = hash, .references = references};
auto path = store.makeFixedOutputPathFromCA(suffix, ca);
if (readOnly || settings.readOnlyMode || (store.isValidPath(path) && !repair))
return path;
StringSource s{contents};
auto path2 = store.addToStoreFromDump(
s,
suffix,
FileSerialisationMethod::Flat,
ContentAddressMethod::Raw::Text,
HashAlgorithm::SHA256,
references,
repair);
assert(path2 == path);
return path;
}
namespace {
@@ -288,7 +290,7 @@ static DerivationOutput parseDerivationOutput(
if (!hashAlgoStr.empty()) {
ContentAddressMethod method = ContentAddressMethod::parsePrefix(hashAlgoStr);
if (method == ContentAddressMethod::Raw::Text)
xpSettings.require(Xp::DynamicDerivations);
xpSettings.require(Xp::DynamicDerivations, "text-hashed derivation output");
const auto hashAlgo = parseHashAlgo(hashAlgoStr);
if (hashS == "impure"sv) {
xpSettings.require(Xp::ImpureDerivations);
@@ -426,7 +428,9 @@ Derivation parseDerivation(
if (*versionS == "xp-dyn-drv"sv) {
// Only version we have so far
version = DerivationATermVersion::DynamicDerivations;
xpSettings.require(Xp::DynamicDerivations);
xpSettings.require(Xp::DynamicDerivations, [&] {
return fmt("derivation '%s', ATerm format version 'xp-dyn-drv'", name);
});
} else {
throw FormatError("Unknown derivation ATerm format version '%s'", *versionS);
}
@@ -1301,7 +1305,7 @@ DerivationOutput::fromJSON(const nlohmann::json & _json, const ExperimentalFeatu
auto methodAlgo = [&]() -> std::pair<ContentAddressMethod, HashAlgorithm> {
ContentAddressMethod method = ContentAddressMethod::parse(getString(valueAt(json, "method")));
if (method == ContentAddressMethod::Raw::Text)
xpSettings.require(Xp::DynamicDerivations);
xpSettings.require(Xp::DynamicDerivations, "text-hashed derivation output in JSON");
auto hashAlgo = parseHashAlgo(getString(valueAt(json, "hashAlgo")));
return {std::move(method), std::move(hashAlgo)};
@@ -1454,7 +1458,8 @@ Derivation Derivation::fromJSON(const nlohmann::json & _json, const Experimental
node.value = getStringSet(valueAt(json, "outputs"));
auto drvs = getObject(valueAt(json, "dynamicOutputs"));
for (auto & [outputId, childNode] : drvs) {
xpSettings.require(Xp::DynamicDerivations);
xpSettings.require(
Xp::DynamicDerivations, [&] { return fmt("dynamic output '%s' in JSON", outputId); });
node.childMap[outputId] = doInput(childNode);
}
return node;
+6 -8
View File
@@ -6,15 +6,14 @@ namespace nix {
template<typename V>
typename DerivedPathMap<V>::ChildNode & DerivedPathMap<V>::ensureSlot(const SingleDerivedPath & k)
{
std::function<ChildNode &(const SingleDerivedPath &)> initIter;
initIter = [&](const auto & k) -> auto & {
auto initIter = [&](this auto & initIter, const auto & k) -> ChildNode & {
return std::visit(
overloaded{
[&](const SingleDerivedPath::Opaque & bo) -> auto & {
[&](const SingleDerivedPath::Opaque & bo) -> ChildNode & {
// will not overwrite if already there
return map[bo.path];
},
[&](const SingleDerivedPath::Built & bfd) -> auto & {
[&](const SingleDerivedPath::Built & bfd) -> ChildNode & {
auto & n = initIter(*bfd.drvPath);
return n.childMap[bfd.output];
},
@@ -27,15 +26,14 @@ typename DerivedPathMap<V>::ChildNode & DerivedPathMap<V>::ensureSlot(const Sing
template<typename V>
typename DerivedPathMap<V>::ChildNode * DerivedPathMap<V>::findSlot(const SingleDerivedPath & k)
{
std::function<ChildNode *(const SingleDerivedPath &)> initIter;
initIter = [&](const auto & k) {
auto initIter = [&](this auto & initIter, const auto & k) -> ChildNode * {
return std::visit(
overloaded{
[&](const SingleDerivedPath::Opaque & bo) {
[&](const SingleDerivedPath::Opaque & bo) -> ChildNode * {
auto it = map.find(bo.path);
return it != map.end() ? &it->second : nullptr;
},
[&](const SingleDerivedPath::Built & bfd) {
[&](const SingleDerivedPath::Built & bfd) -> ChildNode * {
auto * n = initIter(*bfd.drvPath);
if (!n)
return (ChildNode *) nullptr;
+5 -1
View File
@@ -85,7 +85,11 @@ void drvRequireExperiment(const SingleDerivedPath & drv, const ExperimentalFeatu
[&](const SingleDerivedPath::Opaque &) {
// plain drv path; no experimental features required.
},
[&](const SingleDerivedPath::Built &) { xpSettings.require(Xp::DynamicDerivations); },
[&](const SingleDerivedPath::Built & b) {
xpSettings.require(Xp::DynamicDerivations, [&] {
return fmt("building output '%s' of '%s'", b.output, b.drvPath->getBaseStorePath().to_string());
});
},
},
drv.raw());
}
Loaded 100 of 211 files, more files were not shown because too many files have changed in this diff. Show more