Compare commits

..
211 Commits
Author SHA1 Message Date
John Ericson a4b62a646a nlohmann::json instance and JSON Schema for MemorySourceAccessor
As documented, this for file system objects themselves, since
`MemorySourceAccessor` is an implementation detail.
2025-10-21 13:26:16 -04:00
John Ericson ef8218f2e3 Merge pull request #14307 from NixOS/json-schema-hash
`nlohmann::json` instance and JSON Schema for `Hash`
2025-10-21 06:03:20 +00:00
John Ericson ada008a795 Merge pull request #14310 from obsidiansystems/inline-drv-output-subst-goal
Inline `realisationFetched`
2025-10-21 06:03:06 +00:00
John Ericson 2a2bb8330d Merge pull request #14312 from corngood/develop-structuredAttrs-fix
tests/functional/flakes/develop.sh: Add test for outputChecks stripping
2025-10-20 22:48:39 +00:00
David McFarland 645794b458 tests/functional/flakes/develop.sh: Add test for outputChecks stripping 2025-10-20 19:16:20 -03:00
John Ericson 1121f0d8ec Inline realisationFetched
Now that we are using coroutines, we don't need this to be a separate
method of `DrvOutputSubstitutionGoal`.
2025-10-20 16:45:41 -04:00
Sergei Zimmerman 6420879728 Merge pull request #14296 from lovesegfault/nix-s3-more-tests
fix(nix-prefetch-url): correctly extract filename from URLs with query parameters
2025-10-20 19:42:22 +00:00
Sergei Zimmerman 67f5cb97a3 Merge pull request #14306 from corngood/develop-structuredAttrs-fix
nix/develop: Strip outputChecks when structuredAttrs is enabled
2025-10-20 19:38:19 +00:00
John EricsonandRobert Hensing 5e7ee808de nlohmann::json instance and JSON Schema for Hash
Improving and codifying our experimental JSON interfacing.

Co-Authored-By: Robert Hensing <robert@roberthensing.nl>
2025-10-20 15:21:07 -04:00
John Ericson 270f20a505 Merge pull request #14305 from NixOS/alignment-utils
libutil: Add alignUp helper function, use in archive.cc
2025-10-20 19:08:20 +00:00
Bernardo Meurer Costa 1b1d7e3047 test(nixos): add nix-prefetch-url test for S3 URLs with query parameters
Adds a comprehensive test to verify that `nix-prefetch-url` correctly
handles S3 URLs with query parameters (e.g., custom endpoints and regions).

Previously, nix-prefetch-url would fail with "invalid store
path" errors when given S3 URLs with query parameters like
`?endpoint=http://server:9000&region=eu-west-1`, because it incorrectly
extracted the filename from the query parameters instead of the path.
2025-10-20 21:45:37 +03:00
David McFarland 0f28c76a44 nix/develop: Strip outputChecks when structuredAttrs is enabled 2025-10-20 15:40:05 -03:00
Bernardo Meurer Costa e3b3f05e5d fix(nix-prefetch-url): correctly extract filename from URLs with query parameters
Previously, `prefetchFile()` used `baseNameOf()` directly on the URL string
to extract the filename. This caused issues with URLs containing query
parameters that include slashes, such as S3 URLs with custom endpoints:

```
s3://bucket/file.txt?endpoint=http://server:9000
```

The `baseNameOf()` function naively searches for the rightmost `/` in the
entire string, which would find the `/` in `http://server:9000` and extract
`server:9000&region=...` as the filename. This resulted in invalid store
path names containing illegal characters like `:`.

This commit fixes the issue by:

1. Adding a `VerbatimURL::lastPathSegment()` method that extracts the last
   non-empty path segment from a URL, using `pathSegments(true)` to filter
   empty segments
2. Changing `prefetchFile()` to accept `const VerbatimURL &` and use the new
   `lastPathSegment()` method instead of manual path parsing
3. Adding early validation with `checkName()` to fail quickly on invalid
   filenames
4. Maintains backward compatibility by falling back to `baseNameOf()` for
   unparsable `VerbatimURL`s
2025-10-20 21:40:03 +03:00
John Ericson f05d240222 Merge pull request #14278 from obsidiansystems/adl-serializer-xp
Cleanup and JSON serializer and XP feature interations
2025-10-20 18:22:21 +00:00
Sergei Zimmerman 22c73868c3 libutil/archive: Use alignUp
With this change it's much more apparent what's going on.
2025-10-20 21:15:11 +03:00
Sergei Zimmerman a91b787524 libutil: Add alignUp helper function 2025-10-20 21:11:00 +03:00
Eelco Dolstra ddf7de0a76 Merge pull request #14291 from NixOS/skip-source
Add skip() method to Source interface to allow efficient seeks
2025-10-20 15:04:36 +00:00
Sergei Zimmerman 1fabed18b6 Merge pull request #14301 from NixOS/s3-terminate-unknown-profile
libstore: Fix reentrancy in AwsCredentialProviderImpl::getCredentialsRaw
2025-10-20 14:28:16 +00:00
Eelco Dolstra 6c9083db2c Use a smaller buffer 2025-10-20 13:40:19 +02:00
Sergei Zimmerman c663f7ec79 libstore: Fix reentrancy in AwsCredentialProviderImpl::getCredentialsRaw
Old code would do very much incorrect reentrancy crimes (trying to do an
erase inside the emplace callback). This would fail miserably with an assertion
in Boost:

terminating due to unexpected unrecoverable internal error: Assertion '(!find(px))&&("reentrancy not allowed")' failed in boost::unordered::detail::foa::entry_trace::entry_trace(const void *) at include/boost/unordered/detail/foa/reentrancy_check.hpp:33

This is trivially reproduced by using any S3 URL with a non-empty profile:

nix-prefetch-url "s3://happy/crash?profile=default"
2025-10-19 21:03:13 +03:00
Sergei Zimmerman d0fb03c35d Merge pull request #14282 from NixOS/s3-cleanup
Simplify meson for S3 support via aws-crt-cpp
2025-10-19 17:00:46 +00:00
Sergei Zimmerman c847cd87f1 Merge pull request #14297 from lovesegfault/nix-s3-test-public
test(nixos/s3-binary-cache-store): misc improvements
2025-10-19 16:53:40 +00:00
tomberek dbbdae926b Merge pull request #14299 from roberth/unlocked-msg
Clarify unlocked input warning message
2025-10-19 16:50:16 +00:00
Eelco Dolstra 3c03050cd6 Merge pull request #14290 from NixOS/dont-write-nar-to-tty
nix store dump-path: Refuse to write NARs to the terminal
2025-10-19 12:41:55 +00:00
Robert Hensing e33cd5aa38 Clarify unlocked input warning message
The previous message was vague about what "deprecated" meant and why
unlocked inputs with NAR hashes "may not be reproducible". It also
used "verifiable" which was confusing.

The new message makes it clear that the NAR hash provides verification
(is checked by NAR hash) and explicitly states the failure modes:
garbage collection and sharing.
2025-10-19 14:08:34 +02:00
Bernardo Meurer Costa d9c808f8a7 refactor(tests/nixos/s3-binary-cache-store): add verify_packages_in_store helper 2025-10-19 00:21:54 +00:00
Bernardo Meurer Costa 55ea3d3476 test(tests/nixos/s3-binary-cache-store): test public bucket operations
Add `test_public_bucket_operations` to validate that store operations
work correctly on public S3 buckets without requiring credentials.
Tests nix store info and nix copy operations.
2025-10-19 00:04:33 +00:00
Bernardo Meurer Costa 7d0c06f921 feat(tests/nixos/s3-binary-cache-store): add public parameter to setup_s3
Add optional 'public' parameter to setup_s3 decorator. When set to True,
the bucket will be made publicly accessible using mc anonymous set.
2025-10-18 23:57:51 +00:00
Bernardo Meurer Costa 5b4bd5bcb8 refactor(tests/nixos/s3-binary-cache-store): inline make_http_url fn
Remove make_http_url helper function and inline its single usage.
2025-10-18 23:51:44 +00:00
Bernardo Meurer Costa 4ae6c65bc5 test(tests/nixos/s3-binary-cache-store): verify credential caching in concurrent fetches
Add assertion to test_concurrent_fetches to verify that only one
credential provider is created even with 5 concurrent fetches.
2025-10-18 23:48:55 +00:00
Bernardo Meurer Costa 4f19e63a8f refactor(tests/nixos/s3-binary-cache-store): add --no-link to nix build commands
Prevent creation of result symlinks in all nix build commands by
adding the --no-link flag.
2025-10-18 23:44:13 +00:00
Bernardo Meurer Costa f88c3055f8 refactor(tests/nixos/s3-binary-cache-store): clean client store in setup_s3
Add cleanup of client store in the finally block of setup_s3 decorator.
Uses `nix store delete --ignore-liveness` to properly handle GC roots
and only attempts deletion if the path exists.
2025-10-18 23:36:48 +00:00
Bernardo Meurer Costa 9058d90ab2 refactor(tests/nixos/s3-binary-cache-store): rename populate_with to populate_bucket 2025-10-18 23:27:03 +00:00
Bernardo Meurer Costa c1a15d1a26 refactor(tests/nixos/s3-binary-cache-store): rename with_test_bucket to setup_s3 2025-10-18 23:24:30 +00:00
Bernardo Meurer Costa 22f4cccc71 refactor(tests/nixos/s3-binary-cache-store): use a PKGS dict
Replace individual PKG_A, PKG_B, and PKG_C variables with a PKGS
dictionary. This will enable `@with_clean_client_store` in the future.
2025-10-18 23:23:50 +00:00
John Ericson b56e456b0d Merge pull request #14269 from roberth/json-schema
Add a JSON Schema for `Derivation`
2025-10-18 18:50:39 +00:00
Robert Hensing c92ba4b9b7 Add titles in JSON schemas
This way, the description isn't rendered in the tables of contents,
leading to no more formatting errors.
2025-10-17 21:53:29 +02:00
Eelco Dolstra 67bffa19a5 NullFileSystemObjectSink: Skip over file contents 2025-10-17 20:44:02 +02:00
Eelco Dolstra daa7e0d2e9 Source: Add skip() method
This allows FdSource to efficiently skip data we don't care about.
2025-10-17 20:41:33 +02:00
Eelco Dolstra 109f6449cc nix store dump-path: Refuse to write NARs to the terminal 2025-10-17 20:27:10 +02:00
John Ericson ad2360c59f Merge pull request #14288 from lovesegfault/repl-skip-stack
fix(tests/functional/repl): skip test if stack size limit is insufficient
2025-10-17 17:35:52 +00:00
Bernardo Meurer Costa 20c7c551bf fix(tests/functional/repl): skip test if stack size limit is insufficient
Nix attempts to set the stack size to 64 MB during initialization, which is
required for the repl tests to run successfully. Skip the tests on systems
where the hard stack limit is less than this value rather than failing.
2025-10-17 17:05:12 +00:00
John Ericson e78e6ca4f4 Merge pull request #14281 from NixOS/dead-code
libutil: Drop unused SubdirSourceAccessor
2025-10-17 03:01:17 +00:00
John Ericson e34063cf21 Merge pull request #14283 from NixOS/nar-check
nix {cat,ls}: Add back missing checks for file descriptors
2025-10-17 02:58:23 +00:00
Sergei Zimmerman e457ea7688 nix {cat,ls}: Add back missing checks for file descriptors
I didn't catch this during the review of https://github.com/NixOS/nix/pull/14273.
This fixes that mistake.
2025-10-17 02:26:24 +03:00
Farid Zakaria 64c55961eb Merge pull request #14273 from fzakaria/fzakaria/issue-13944
Make `nix nar [cat|ls]` lazy
2025-10-17 02:16:54 +03:00
Sergei Zimmerman ffbc33fec6 libstore/meson: Rename curl-s3-store to s3-aws-auth
We now unconditionally compile support for s3:// URLs and stores
without authentication. The whole curl version check can be greatly
simplified by the previous commit, which bumps the minimum required curl
version.
2025-10-17 01:18:46 +03:00
Sergei Zimmerman a80fc252e8 libstore/meson: Require curl >= 7.75.0
This version has been released a long time ago in 2021 and it's doubtful
that anybody actually uses it still, since it's full of vulnerabilities [^]

[^]: https://curl.se/docs/vuln-7.75.0.html
2025-10-17 01:18:14 +03:00
Sergei Zimmerman bcd5a9d05c libutil: Drop unused SubdirSourceAccessor 2025-10-17 00:56:53 +03:00
Robert Hensing 01b001d5ba Add JSON Schema infrastructure, use for Derivation
For manual, and testing formats
2025-10-16 17:24:18 -04:00
John Ericson 27767a6094 Merge pull request #14276 from NixOS/fix-14193
libstore/registerOutputs: Don't try to optimize a non-existent actual…
2025-10-16 21:06:43 +00:00
John Ericson 1177d65094 Properly check xp features when deserializing deriving paths 2025-10-16 16:45:22 -04:00
John Ericson a2c6f38e1f Remove now-redundant methods for JSON on Derivation 2025-10-16 16:45:22 -04:00
John Ericson 1c02dd5b9c Allow for standard nlohmann JSON serializers to take separate XP features
I realized that we can actually do this thing, even though it is not
what nlohmann expects at all, because the extra parameter has a default
argument so nlohmann doesn't need to care. Sneaky!
2025-10-16 16:45:22 -04:00
Sergei Zimmerman 4cbcaad435 libstore/registerOutputs: Don't try to optimize a non-existent actualPath
Since 3c610df550 this resulted in `getting status of`
errors on paths inside the chroot if a path was already valid. Careful inspection
of the logic shows that if buildMode != bmCheck actualPath gets reassigned to
store.toRealPath(finalDestPath). The only branch that cares about actualPath is
the buildMode == bmCheck case, which doesn't lead to optimisePath anyway.
2025-10-16 23:08:30 +03:00
John Ericson d87a06af7a Merge pull request #14275 from NixOS/s3-cleanup
libstore: Miscellaneous S3 store cleanups
2025-10-16 19:36:59 +00:00
Eelco Dolstra 2dc9f2a2b7 Merge pull request #14272 from NixOS/use-store-path-serializer
Daemon protocol: Use the WorkerProto serializer for store paths
2025-10-16 19:35:25 +00:00
Eelco Dolstra a7991d55cc Merge pull request #14270 from NixOS/use-optional-storepath-serializer
Use serializer for std::optional<StorePath>
2025-10-16 19:07:07 +00:00
Sergei Zimmerman e7047fde25 libstore: Remove the unnecessary 'error: ' prefix in warning message 2025-10-16 21:49:38 +03:00
Sergei Zimmerman 33e94fe19f libstore: Make AwsAuthError more legible
Instead of the cryptic:

> error: Failed to resolve AWS credentials: error code 6153`

We now get more legible:

> error: AWS authentication error: 'Valid credentials could not be sourced by the IMDS provider' (6153)
2025-10-16 21:49:37 +03:00
Sergei Zimmerman dc03c6a812 libstore: Put all the AWS credentials logic behind interface class AwsCredentialProvider
This makes it so we don't need to rely on global variables and hacky destructors to
clean up another global variable. Just putting it in the correct order in the class
is more than enough.
2025-10-16 21:49:36 +03:00
Sergei Zimmerman b1d067c9bb tests/nixos: Rename back S3 store nixos test 2025-10-16 21:49:35 +03:00
Eelco Dolstra d782c5e586 Daemon protocol: Use the WorkerProto serializer for store paths 2025-10-16 17:34:33 +02:00
Eelco Dolstra f84b33644c Merge pull request #14271 from NixOS/no-check-sigs
Factor out `--no-check-sigs` into its own class
2025-10-16 15:07:29 +00:00
Eelco Dolstra 3bd2b76f6e nix store sign: Use required attribute 2025-10-16 16:35:13 +02:00
Eelco Dolstra 139df77440 Factor out --no-check-sigs 2025-10-16 16:35:09 +02:00
Eelco Dolstra a48a737517 Use serializer for std::optional<StorePath> 2025-10-16 16:32:18 +02:00
John Ericson 0503a862ef Merge pull request #14268 from roberth/dev-doc-manual
doc/dev/doc: Update local build instructions for manual
2025-10-16 13:50:58 +00:00
Robert Hensing 61cb9c4832 doc/dev/doc: Update local build instructions for manual 2025-10-16 13:22:22 +02:00
John Ericson 721f5572a6 Merge pull request #14263 from NixOS/hydra-import-paths
Restore `ServeProto::Command::ImportPaths`
2025-10-15 22:57:50 +00:00
John Ericson 5a6864c027 Merge pull request #14264 from xokdvium/fix-splicing-test
tests: Fix splicing in functional tests for nix-cli
2025-10-15 22:32:45 +00:00
Sergei Zimmerman 0deb492b3d Restore ServeProto::Command::ImportPaths
This partially reverts commit 5e46df973f,
partially reversing changes made to
8c789db05b.

We do this because Hydra, while using the newer version of the protocol,
still uses this command, even though Nix (as a client) doesn't use it.
On that basis, we don't want to remove it (or consider it only part of
the older versions of the protocol) until Hydra no longer uses the
Legacy SSH Protocol.
2025-10-15 18:18:59 -04:00
Sergei Zimmerman 17b7fb383f tests: Fix splicing in functional tests for nix-cli
This is necessary to fix nix-everything-llvm.
The problem here is that nix-cli is taken from the previous
stage that is built with libstdc++, but this derivation builds
plugins with libc++ and the plugin load fails miserably.
2025-10-16 01:04:50 +03:00
John Ericson 94cfba7e84 Merge pull request #14226 from obsidiansystems/unkeyed-realisation
Reapply #14097
2025-10-15 21:27:13 +00:00
Sergei Zimmerman 0f1cfa4d60 Merge pull request #14262 from lovesegfault/ci-cleanup-s3
ci: cleanup s3 tests
2025-10-15 21:19:20 +00:00
Bernardo Meurer Costa fa0d00e668 ci: cleanup s3 tests
This cleans up the work done in 8c2828387. Now that #13752 has landed,
there's no need to test configurations without AWS auth in CI.
2025-10-15 23:51:08 +03:00
Sergei Zimmerman d2b6499154 Merge pull request #13752 from lovesegfault/curl-based-s3-v2
feat(libstore): curl-based s3
2025-10-15 20:01:26 +00:00
Sergei Zimmerman e3232af558 Merge pull request #14253 from NixOS/libgit2-refname-wa
libfetchers/git-utils: Be more correct about validating refnames
2025-10-15 19:30:53 +00:00
Bernardo Meurer Costa e069c9892e docs(rl-next): add notes for curl-based s3 2025-10-15 19:26:53 +00:00
John Ericson 266fbebe66 Implement realisation operations on dummy store 2025-10-15 14:59:08 -04:00
John EricsonandSergei Zimmerman 6995d325ef Split out UnkeyedRealisation from Realisation
Realisations are conceptually key-value pairs, mapping `DrvOutputs` (the
key) to information about that derivation output.

This separate the value type, which will be useful in maps, etc., where
we don't want to denormalize by including the key twice.

This matches similar changes for existing types:

| keyed              | unkeyed                |
|--------------------|------------------------|
| `ValidPathInfo`    | `UnkeyedValidPathInfo` |
| `KeyedBuildResult` | `BuildResult`          |
| `Realisation`      | `UnkeyedRealisation`   |

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-15 14:59:04 -04:00
Sergei Zimmerman 5d1178b817 libfetchers/git-utils: Be more correct about validating refnames
Turns out there's a much better API for this that doesn't have the
footguns of the previous method.

isLegalRefName is somewhat of a misnomer, since it's mainly used to
validate user inputs that can be either references, branch names,
psedorefs or tags.
2025-10-15 21:54:09 +03:00
Bernardo Meurer Costa 3224636ab0 refactor(libstore): rename NIX_WITH_S3_SUPPORT to NIX_WITH_AWS_AUTH
The macro now accurately reflects its purpose: gating only AWS
authentication code, not all S3 functionality. S3 URL parsing, store
configuration, and public bucket access work regardless of this flag.

This rename clarifies that:
- S3 support is always available (URL parsing, store registration)
- Only AWS credential resolution requires the flag
- The flag controls AWS CRT SDK dependency, not S3 protocol support
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa bb1f22a8df refactor(libstore): minimize NIX_WITH_S3_SUPPORT scope to auth only
Move S3 URL parsing, store configuration, and public bucket support
outside of NIX_WITH_S3_SUPPORT guards. Only AWS credential resolution
remains gated, allowing builds with withAWS = false to:

- Parse s3:// URLs
- Register S3 store types
- Access public S3 buckets (via HTTPS conversion)
- Use S3-compatible services without authentication

The setupForS3() function now always performs URL conversion, with
authentication code conditionally compiled based on NIX_WITH_S3_SUPPORT.
The aws-creds.cc file (only code using AWS CRT SDK) is now conditionally
compiled by meson.
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa 1f710300c9 refactor(libstore): withCurlS3 -> withAWS
Now that the legacy S3 implementation is gone, we can go back to calling
things `NIX_WITH_S3_SUPPORT`.
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa 9295c14a35 refactor(libstore): replace AWS SDK with curl-based S3 implementation
This commit replaces the AWS C++ SDK with a lighter curl-based approach
for S3 binary cache operations.

- Removed dependency on the heavy aws-cpp-sdk-s3 and aws-cpp-sdk-transfer
- Added lightweight aws-crt-cpp for credential resolution only
- Leverages curl's native AWS SigV4 authentication (requires curl >= 7.75.0)
- S3BinaryCacheStore now delegates to HttpBinaryCacheStore
- Function s3ToHttpsUrl converts ParsedS3URL to ParsedURL
- Multipart uploads are no longer supported (may be reimplemented later)
- Build now requires curl >= 7.75.0 for AWS SigV4 support

Fixes: #13084, #12671, #11748, #12403, #5947
2025-10-15 18:23:55 +00:00
John Ericson a543519ca9 Merge pull request #14257 from obsidiansystems/misc-builder-tech-debt
Cleanup misc builder tech debt
2025-10-15 17:28:27 +00:00
John Ericson 632ccfb8c0 Remove dead outputPaths variable. 2025-10-15 12:16:53 -04:00
John Ericson 46357468a4 Remove unused parameters to DrvOutputSubstitutionGoal 2025-10-15 12:16:52 -04:00
John Ericson b20cebf993 Remove unused typedef and field 2025-10-15 12:15:23 -04:00
Eelco Dolstra 5700112127 Merge pull request #14205 from GrahamDennis/gdennis/improve-dir-url-backcompat
Improved backwards compatibility hack for git URLs using dir=...
2025-10-15 15:20:02 +00:00
John Ericson 606eb1dfb5 Merge pull request #14250 from fzakaria/patch-1
Remove duplicate shellcheck in dev-shell.nix
2025-10-15 05:03:19 +00:00
John Ericson e07754d888 Merge pull request #14251 from fzakaria/fzakaria/iwyu-libflake
Clean-up libflake headers
2025-10-15 04:27:07 +00:00
Farid Zakaria 01a8499d2f Format cpp files 2025-10-14 23:51:40 -04:00
Farid Zakaria e8b126fa90 Remove unecessary includes 2025-10-14 23:48:45 -04:00
Farid Zakaria 902faf4fe5 More fixes for iwyu 2025-10-14 23:20:35 -04:00
Farid Zakaria 7bc3d9b9a9 First attempt at uwyu for libflake 2025-10-14 22:53:13 -04:00
Farid Zakaria 092639709f Remove duplicate shellcheck in dev-shell.nix 2025-10-14 19:25:06 -07:00
John Ericson 620091bc8b Merge pull request #14223 from lovesegfault/curl-based-s3-tests
test(nixos): add comprehensive curl-based S3 VM tests
2025-10-14 23:08:55 +00:00
John Ericson 6dcc468253 Merge pull request #14249 from NixOS/more-to-real-path-cleanups
More toRealPath cleanups
2025-10-14 22:46:15 +00:00
Sergei Zimmerman 0347958dd2 nix/develop: Remove usage of toRealPath, replace with SourceAccessor 2025-10-15 00:52:13 +03:00
Sergei Zimmerman 918a3cebaa libexpr: Use Store::requireStoreObjectAccessor instead or toRealPath in fetch
This forces the code to go through proper abstractions instead of the raw filesystem
API.

This issue is evident from this reproducer:

nix eval --expr 'builtins.fetchurl { url = "https://example.com"; sha256 = ""; }' --json --eval-store "dummy://?read-only=false"

error:
       … while calling the 'fetchurl' builtin
         at «string»:1:1:
            1| builtins.fetchurl { url = "https://example.com"; sha256 = ""; }
             | ^

       error: opening file '/nix/store/r4f87yrl98f2m6v9z8ai2rbg4qwlcakq-example.com': No such file or directory
2025-10-15 00:27:41 +03:00
Sergei Zimmerman 69c005e805 libstore: Use getFSAccessor for store object in Worker::pathContentsGood
We only care about the accessor for a single store object anyway, but
the validity gets ignored. Also `pathExists(store.printStorePath(path))`
is definitely incorrect since it confuses the logical location vs physical
location in case of a chroot store.
2025-10-15 00:15:50 +03:00
Sergei Zimmerman 0c32fb3fa2 treewide: Add Store::requireStoreObjectAccessor, simplify uses of getFSAccessor
This is a simple wrapper around getFSAccessor that throws an InvalidPath
error. This simplifies usage in callsites that only care about getting
a non-null accessor.
2025-10-14 23:58:20 +03:00
Bernardo Meurer Costa d18f959d4f test(nixos): add comprehensive curl-based S3 VM tests
Add `curl-s3-binary-cache-store.nix` with comprehensive test coverage
for the curl-based S3 implementation.

Depends-On: #14206, #14222
2025-10-14 20:55:14 +00:00
Sergei Zimmerman 4041bfdb40 Merge pull request #14206 from lovesegfault/curl-based-s3-pieces
feat(libstore): add builtin fetchurl S3 credential pre-resolution
2025-10-14 20:10:41 +00:00
John Ericson 1fb4ff8c0e Merge pull request #14232 from roberth/dyndrv-messages
Better dyndrv messages
2025-10-14 15:40:27 +00:00
Robert Hensing 1b96a704d3 Add lazy evaluation for experimental feature reasons
Wrap fmt() calls in lambdas to defer string formatting until the
feature check fails. This avoids unnecessary string formatting in
the common case where the feature is enabled.

Addresses performance concern raised by xokdvium in PR review.
2025-10-14 16:49:59 +02:00
John Ericson 959c244a12 Merge pull request #14243 from NixOS/canon-path-nul-bytes
libutil: Ensure that CanonPath does not contain NUL bytes
2025-10-14 14:30:24 +00:00
Eelco Dolstra c44d2d5913 Merge pull request #14241 from NixOS/dependabot/github_actions/actions/create-github-app-token-2
build(deps): bump actions/create-github-app-token from 1 to 2
2025-10-14 11:55:43 +00:00
Eelco Dolstra dd590eca74 Merge pull request #14242 from NixOS/dependabot/github_actions/actions/checkout-5
build(deps): bump actions/checkout from 4 to 5
2025-10-14 11:55:25 +00:00
Sergei Zimmerman 1633ceaff2 libutil: Ensure that CanonPath does not contain NUL bytes
This, alongside the other invariants of the CanonPath is important
to uphold. std::filesystem happily crashes on NUL bytes in the constructor,
as we've seen with `path:%00` prior to c436b7a32a.
Best to stay clear of NUL bytes when we're talking about syscalls, especially
on Unix where strings are null terminated.

Very nice to have if we decide to switch over to pascal-style strings.
2025-10-14 02:33:42 +03:00
John Ericson 16e946bfb1 Merge pull request #14225 from obsidiansystems/derivation-resolution-goal-2
Reapply the rest of #14022
2025-10-13 23:26:29 +00:00
Sergei Zimmerman edf9163c22 libutil: Make CanonPath::root const
By all means CanonPath::root must be immutable. Let's enforce this with
in the code.
2025-10-14 02:24:40 +03:00
John Ericson ad893acf46 Fix ca/eval-store.sh test
The refactor in the last commit fixed the bug it was supposed to fix,
but introduced a new bug in that sometimes we tried to write a resolved
derivation to a store before all its `inputSrcs` were in that store.

The solution is to defer writing the derivation until inside
`DerivationBuildingGoal`, just before we do an actual build. At this
point, we are sure that all inputs in are the store.

This does have the side effect of meaning we don't write down the
resolved derivation in the substituting case, only the building case,
but I think that is actually fine. The store that actually does the
building should make a record of what it built by storing the resolved
derivation. Other stores that just substitute from that store don't
necessary want that derivation however. They can trust the substituter
to keep the record around, or baring that, they can attempt to re
resolve everything, if they need to be audited.

(cherry picked from commit c97b050a6c)
2025-10-13 18:41:59 -04:00
John Ericson 06bb1c2f93 Remove some buildMode default parameters
Force the internals to be more explicit.
2025-10-13 18:40:10 -04:00
John Ericson 2ee41976c2 Fix #13247
Resolve the derivation before creating a building goal, in a context
where we know what output(s) we want. That way we have a chance just to
download the outputs we want.

Fix #13247

(cherry picked from commit 39f6fd9b46)
2025-10-13 18:37:14 -04:00
dependabot[bot] b846f27682 build(deps): bump actions/checkout from 4 to 5
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-10-13 22:00:59 +00:00
dependabot[bot] 962862e9e0 build(deps): bump actions/create-github-app-token from 1 to 2
Bumps [actions/create-github-app-token](https://github.com/actions/create-github-app-token) from 1 to 2.
- [Release notes](https://github.com/actions/create-github-app-token/releases)
- [Commits](https://github.com/actions/create-github-app-token/compare/v1...v2)

---
updated-dependencies:
- dependency-name: actions/create-github-app-token
  dependency-version: '2'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-10-13 22:00:55 +00:00
Robert Hensing 39c4665488 Store reason as a field in MissingExperimentalFeature
Store the reason string as a field in the exception class rather than
only embedding it in the error message. This supports better structured
error handling and future JSON error reporting.

Suggested by Ericson2314 in PR review.
2025-10-13 23:49:20 +02:00
Robert Hensing 71aa9a4798 Add reasons to dyndrv xp messages 2025-10-13 23:49:20 +02:00
Robert Hensing 0fd890a8d6 Add reason string support to MissingExperimentalFeature 2025-10-13 23:49:20 +02:00
John Ericson 6642ffb506 Merge pull request #14239 from NixOS/asan-stack-overflow
libstore/outputs-spec: Drop usage of std::regex
2025-10-13 21:44:49 +00:00
Sergei Zimmerman 3ba221025f libstore/outputs-spec: Drop usage of std::regex
std::regex is a really bad tool for parsing things, since
it tends to overflow the stack pretty badly. See the build failure
under ASan in [^].

[^]: https://hydra.nixos.org/build/310077167/nixlog/5
2025-10-13 23:58:20 +03:00
Eelco Dolstra b56cc1808d Merge pull request #14237 from NixOS/url-parser-regression
Remove validation of URLs passed to FileTransferRequest verbatim
2025-10-13 20:01:01 +00:00
Sergei Zimmerman 47f427a172 Remove validation of URLs passed to FileTransferRequest verbatim
CURL is not very strict about validation of URLs passed to it. We
should reflect this in our handling of URLs that we get from the user
in <nix/fetchurl.nix> or builtins.fetchurl. ValidURL was an attempt to
rectify this, but it turned out to be too strict. The only good way to
resolve this is to pass (in some cases) the user-provided string verbatim
to CURL. Other usages in libfetchers still benefit from using structured
ParsedURL and validation though.

nix store prefetch-file --name foo 'https://cdn.skypack.dev/big.js@^5.2.2'
error: 'https://cdn.skypack.dev/big.js@^5.2.2' is not a valid URL: leftover
2025-10-13 22:23:26 +03:00
John Ericson 0f85ef3677 Merge pull request #14219 from lovesegfault/eval-copy-less
libstore: Avoid copying derivations to the store if they are already valid
2025-10-13 16:36:40 +00:00
Eelco Dolstra be2c9ef44c Merge pull request #14229 from NixOS/reduce-hydra-load
packaging/hydra: buildNoGC is the same as buildWithSanitizers
2025-10-13 16:22:30 +00:00
John Ericson d2c0c0607c Merge branch 'master' into eval-copy-less 2025-10-13 11:52:42 -04:00
John Ericson 480ce19011 Merge pull request #14233 from neuralsorcerer/fix-typo
Fix typos
2025-10-13 15:30:05 +00:00
John Ericson 3f876bcb61 Merge pull request #14231 from roberth/code-docs
Code docs
2025-10-13 15:05:30 +00:00
Soumyadip Sarkar 998f93f267 Fix typos 2025-10-13 18:15:52 +05:30
Robert Hensing 583f5e37fc Refactor: use optionalBracket in nix search 2025-10-13 14:02:59 +02:00
Robert Hensing 5dcfddf997 strings: Add optionalBracket helper 2025-10-13 13:59:39 +02:00
Robert Hensing 48a5e2dde2 EvalState: add doc comment 2025-10-13 13:14:05 +02:00
Robert Hensing 6db86389ce util/error: Document addTrace params
... and rename e -> pos. That was weird.
2025-10-13 12:57:22 +02:00
Bernardo Meurer Costa 000e6f6282 feat(libstore): add builtin fetchurl S3 credential pre-resolution
Add support for pre-resolving AWS credentials in the parent process
before forking for builtin:fetchurl. This avoids recreating credential
providers in the forked child process.
2025-10-12 23:01:13 +00:00
Bernardo Meurer CostaandSergei Zimmerman 18ec3d1094 libstore: Avoid copying derivations to the store if they are already valid
This avoids the quite costly copying of derivations to the daemon over the
wire in case it already exists in the eval store.

For a fresh instantiatation (after running nix-collect-garbage) this doesn't
significantly slow down eval:

taskset -c 2,3 hyperfine --reference "result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket" --prepare "nix-collect-garbage --store /tmp/store1111 --no-keep-derivations" "result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket"
Benchmark 1: result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     388.7 ms ±  10.5 ms    [User: 157.0 ms, System: 61.3 ms]
  Range (min … max):   379.4 ms … 415.9 ms    10 runs

Benchmark 2: result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     389.2 ms ±   4.8 ms    [User: 158.5 ms, System: 60.7 ms]
  Range (min … max):   381.2 ms … 397.6 ms    10 runs

But if the derivations are already instantiated this shows a pretty neat speedup:

taskset -c 2,3 hyperfine --reference "result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket" "result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket"
Benchmark 1: result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     240.4 ms ±   3.1 ms    [User: 148.1 ms, System: 57.0 ms]
  Range (min … max):   233.8 ms … 245.0 ms    12 runs

Benchmark 2: result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     226.5 ms ±   4.5 ms    [User: 147.8 ms, System: 55.2 ms]
  Range (min … max):   214.0 ms … 231.2 ms    13 runs

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-13 01:59:38 +03:00
John Ericson f77094715f Merge pull request #14222 from lovesegfault/curl-based-s3-fix-race
fix(libstore): fix race condition in AWS credential provider caching
2025-10-12 22:07:10 +00:00
Bernardo Meurer Costa f0e1f65260 fix(libstore): fix race condition in AWS credential provider caching
The previous implementation had a check-then-create race condition where
multiple threads could simultaneously:
1. Check the cache and find no provider (line 122)
2. Create their own providers (lines 126-145)
3. Insert into cache (line 161)

This resulted in multiple credential providers being created when
downloading multiple packages in parallel, as each .narinfo download
would trigger provider creation on its own thread.

Fix by using boost::concurrent_flat_map's try_emplace_and_cvisit, which
provides atomic get-or-create semantics:
- f1 callback: Called atomically during insertion, creates the provider
- f2 callback: Called if key exists, returns cached provider
- Other threads are blocked during f1, so no nullptr is ever visible
2025-10-12 20:16:02 +00:00
Sergei Zimmerman 89b35ec0dc packaging/hydra: buildNoGC is the same as buildWithSanitizers
This will reduce the load on hydra. It doesn't make sense to
build 2 slightly different variations where the difference
is only in the nix-perl-bindings and additional sanitizers.
2025-10-12 22:10:35 +03:00
Sergei Zimmerman 4f585dedbe Merge pull request #14228 from obsidiansystems/fix-windows-build
Fix windows build
2025-10-12 18:07:50 +00:00
John Ericson a01df8de21 Merge pull request #14227 from NixOS/asan-hydra
packaging: Add buildWithSanitizers to hydraJobs
2025-10-12 17:43:54 +00:00
John Ericson 10223fae86 Fix windows build
I forget to add some CPP in b57caaa1a2.

Hopefully, as we relyon RAII more, these explicit resets become
unneeded.
2025-10-12 13:22:35 -04:00
John Ericson 9150ccb89e Fix Windows dev shell (mostly)
gbenchmark still has too-narrow supported systems, however. That needs
to be fixed in Nixpkgs.
2025-10-12 13:16:50 -04:00
Sergei Zimmerman de75a180cb packaging: Add buildWithSanitizers to hydraJobs 2025-10-12 19:38:01 +03:00
Sergei Zimmerman a491173369 packaging: Add withASan,withUBSan options to the scope 2025-10-12 19:16:08 +03:00
Sergei Zimmerman 199b6ff3fb Disable detect_odr_violation for ASan
There's some unfortunate ODR violations that get dianosed with GCC but not Clang
for static inline constexpr variables defined inside the class body:

template<typename T>
struct static_const
{
    static JSON_INLINE_VARIABLE constexpr T value{};
};

This can be ignored pretty much. There is the same problem for std::piecewise_construct:

http://lists.boost.org/Archives/boost/2007/06/123353.php

==2455704==ERROR: AddressSanitizer: odr-violation (0x7efddc460e20):
  [1] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/pkmljfq97a83dbanr0n64zbm8cyhna33-nix-store-2.33.0pre/lib/libnixstore.so.2.33.0
  [2] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/gbjpkjj0g8vk20fzlyrwj491gwp6g1qw-nix-util-2.33.0pre/lib/libnixutil.so.2.33.0
2025-10-12 19:16:07 +03:00
Sergei Zimmerman 711e738bf9 meson: Simplify asan-options handling even more
Instead of specifying env variables all the time
we can instead embed the __asan_default_options symbol
in all executables / shared objects. This reduces code
duplication.
2025-10-12 19:16:06 +03:00
Sergei Zimmerman d9cabddd17 Merge pull request #14214 from obsidiansystems/derivation-resolution-goal
Split out `DerivationResolutionGoal`
2025-10-12 11:31:54 +00:00
John Ericson 0da430be35 Split out DerivationResolutionGoal
This prepares the way for fixing a few issues.

Take 2: was landed before in 8f4a739d0f.
2025-10-11 19:52:37 -04:00
John Ericson 07df87652c Make keys of Derivation*Goal more legible
The property that substitution goals come first is still preserved
2025-10-11 19:50:45 -04:00
John Ericson a629ce3dec Use member initializer list for Derivation*Goal::drv 2025-10-11 18:37:04 -04:00
John Ericson ba7bbcd1da Cleanup Derivation*Goal names 2025-10-11 18:30:47 -04:00
Sergei Zimmerman 97e770ad01 Merge pull request #14209 from lovesegfault/curl-based-s3-pieces-fix
fix(libstore): improve http-binary-cache-store S3 compatibility
2025-10-11 21:19:18 +00:00
John Ericson 682cf0b266 Merge pull request #14216 from NixOS/simplify-asan-meson
meson: Move asan-options to common
2025-10-11 18:21:24 +00:00
John Ericson f1cb837888 Merge pull request #14217 from NixOS/remove-libgit2-patches
packaging: Remove no longer necessary libgit2 patches
2025-10-11 18:20:36 +00:00
Sergei Zimmerman 47705139c9 packaging: Remove no longer necessary libgit2 patches
25.05 already has 1.9.0 and we don't support older nixpkgs versions.
2025-10-11 16:30:55 +03:00
Sergei Zimmerman d26a337c09 meson: Move asan-options to common
This way we don't have to duplicate the subdir everywhere.
Less copy-pasta is good.
2025-10-11 16:08:35 +03:00
John Ericson 8064c75694 Merge pull request #14194 from xokdvium/stacktrace-assertion
libutil: Print stack trace on assertion failure
2025-10-10 23:25:16 +00:00
John Ericson d75614a315 Merge pull request #14208 from obsidiansystems/consolidate-builder-dispatch
Consolidate logic choosing where we can/should build a bit
2025-10-10 23:05:40 +00:00
Sergei Zimmerman 46382ade74 libutil: Print stack trace on assertion failure
This change overrides __assert_fail on glibc/musl
to instead call std::terminate that we have a custom
handler for. This ensures that we have more context
to diagnose issues encountered by users in the wild.
2025-10-11 01:35:58 +03:00
John Ericson 9cb686f816 Merge pull request #14210 from NixOS/double-quotes
libstore: Fix double-quoting of paths in logs
2025-10-10 22:10:36 +00:00
Bernardo Meurer Costa f02218873e fix(libstore): improve http-binary-cache-store S3 compatibility
This commit adds two key fixes to http-binary-cache-store.cc to
properly support the new curl-based S3 implementation:

1. **Consistent cache key handling**: Use `getReference().render(withParams=false)`
   for disk cache keys instead of `cacheUri.to_string()`. This ensures cache
   keys are consistent with the S3 implementation and don't include query
   parameters, which matches the behavior expected by Store::queryPathInfo()
   lookups.

2. **S3 query parameter preservation**: When generating file transfer requests
   for S3 URLs, preserve query parameters from the base URL (region, endpoint,
   etc.) when the relative path doesn't have its own query parameters. This
   ensures S3-specific configuration is propagated to all requests.
2025-10-10 22:05:57 +00:00
John Ericson b57caaa1a2 Consolidate logic choosing where we can/should build a bit
I want to separate "policy" from "mechanism".

Now the logic to decide how to build (a policy choice, though with some
hard constraints) is all in derivation building goal, and all in the
same spot. build hook, external builder, or local builder --- the choice
between all three is made in the same spot --- pure policy.

Now, if you want to use the external deriation builder, you simply
provide the `ExternalBuilder` you wish to use, and there is no
additional checking --- pure mechanism. It is the responsibility of the
caller to choose an external builder that works for the derivation in
question.

Also, `checkSystem()` was the only thing throwing `BuildError` from
`startBuilder`. Now that that is gone, we can now remove the
`try...catch` around that.
2025-10-10 17:28:57 -04:00
John EricsonandSergei Zimmerman 2ff59ec3e0 Use std::ranges::find_if for finding external builders
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-10 17:27:41 -04:00
Sergei Zimmerman f30cb8667b libstore: Fix double-quoting of paths in logs
std::filesystem::path is already quoted by boost::format with double quotes (").
2025-10-11 00:25:51 +03:00
John Ericson b56dd21c31 Settings::ExternalBuilder::systems make set
Nothing cares about the order, actually.
2025-10-10 17:23:45 -04:00
Sergei Zimmerman 2308aaf192 Merge pull request #14198 from lovesegfault/curl-based-s3-pieces
feat(libstore): add curl-based S3 store implementation
2025-10-10 07:55:44 +00:00
Graham Dennis 8d9e9bc400 Improve comment 2025-10-10 15:00:10 +11:00
Graham Dennis 43b01b6790 Improved backwards compatibility hack for git URLs using dir=... attribute 2025-10-10 14:54:47 +11:00
Bernardo Meurer Costa 0855b715a9 feat(libstore): add curl-based S3 store implementation
Add a new S3BinaryCacheStore implementation that inherits from
HttpBinaryCacheStore.

The implementation is activated with NIX_WITH_CURL_S3, keeping the
existing NIX_WITH_S3_SUPPORT (AWS SDK) implementation unchanged.
2025-10-10 00:13:29 +00:00
John Ericson 2fc88ec114 Merge pull request #14204 from NixOS/kill-to-real-path-fetchers
libfetchers: Remove toRealPath in SourceHutInputScheme::getRevFromRef
2025-10-09 22:01:03 +00:00
Sergei Zimmerman 9c8480becb Merge pull request #14203 from roberth/move-eval-cache-open-to-libflake
Move eval cache open to libflake
2025-10-09 21:42:06 +00:00
Sergei Zimmerman c58acff42a libfetchers: Remove toRealPath in SourceHutInputScheme::getRevFromRef
This code had several issues:

1. Not going through the SourceAccessor means that we can only work
   with physical paths.

2. It did not actually check that the file exists. (std::ifstream does not check
   it by default).
2025-10-10 00:26:57 +03:00
Robert Hensing abcceafbce Use const for lock in openEvalCache 2025-10-09 21:25:40 +02:00
Robert Hensing 42c9cbf9ca Use ref<LockedFlake> where non-null 2025-10-09 21:25:40 +02:00
Robert Hensing 0387b7d6db Move openEvalCache to libflake
Most of the eval cache logic is flake-independent and libexpr,
but the loading part is not.
`nix-flake` is the right component for this, as the eval cache
isn't exactly specific to the command line.
2025-10-09 20:40:40 +02:00
Sergei Zimmerman dfafd8bc38 Merge pull request #14199 from getchoo-contrib/getchoo/toml11-fix
packaging: only override `toml11` when necessary
2025-10-09 17:20:35 +00:00
Sergei Zimmerman 44486871e9 Merge pull request #14200 from Mic92/less-ci-builds
only build on push to master
2025-10-09 15:07:06 +00:00
Jörg Thalheim 118acc84ba only build on push to master
we have now merge queues for maintainance branches. We still build it
for master to have our installer beeing updated. In future this part
could go in new workflow instead.
2025-10-09 14:16:14 +01:00
Seth Flynn 0f016f9bf5 packaging: only override toml11 when necessary
v4.4.0 hit Nixpkgs in https://github.com/NixOS/nixpkgs/pull/442682.
Ideally we'd just use that, but this keeps the fallback behavior until
it's more widespread
2025-10-09 03:24:57 -04:00
John Ericson ce38b46e06 Merge pull request #14170 from lovesegfault/curl-based-s3-pieces
feat(libstore/filetransfer): add S3 signing support
2025-10-08 22:35:32 +00:00
Sergei Zimmerman 33d9270109 Merge pull request #14191 from NixOS/fix-14188
libutil: Fix renderAuthorityAndPath unreachable for path:/ URLs
2025-10-08 21:34:28 +00:00
Bernardo Meurer Costa 00c2a57666 feat(libstore/filetransfer): add S3 signing support 2025-10-08 21:31:34 +00:00
Eelco Dolstra d591f17ecb Merge pull request #14189 from NixOS/fix-exportReferencesGraph
exportReferencesGraph: Handle heterogeneous arrays
2025-10-08 21:19:30 +00:00
John Ericson bb1945a090 Merge pull request #14182 from NixOS/simplify-archive-tests
tests: Move invalid nar tests from tests/functional to libutil-tests
2025-10-08 20:46:37 +00:00
Eelco Dolstra 94f410b628 exportReferencesGraph: Handle heterogeneous arrays
This barfed with

   error: [json.exception.type_error.302] type must be string, but is array

on `nix build github:malt3/bazel-env#bazel-env` because it has a `exportReferencesGraph` with a value like `["string",...["string"]]`.
2025-10-08 22:15:33 +02:00
Bernardo Meurer Costa 3c1e2e56ea feat(libstore/filetransfer): add username/password authentication support
Add a `UsernameAuth` struct and optional `usernameAuth` field to
`FileTransferRequest` to support programmatic username/password
authentication.

This uses curl's `CURLOPT_USERNAME`/`CURLOPT_PASSWORD` options, which
works with multiple protocols (HTTP, FTP, etc.) and is not specific to
any particular authentication scheme.

The primary motivation is to enable S3 authentication refactoring where
AWS credentials (access key ID and secret access key) can be passed
through this general-purpose mechanism, reducing the amount of
S3-specific code behind `#if NIX_WITH_CURL_S3` guards.
2025-10-08 20:10:53 +00:00
Eelco Dolstra 925f10d5ea Merge pull request #14087 from NixOS/required-args
Args::Flag: Add `required` attribute
2025-10-08 19:33:22 +00:00
Sergei Zimmerman 1d8dd77e1d libutil: Fix renderAuthorityAndPath unreachable for path:/ URLs
This was mistakenly triggered by path:/ URL, since the `//` would
correspond to 3 empty segments.
2025-10-08 22:14:49 +03:00
Jörg Thalheim 090f7fb05e Merge pull request #14002 from getchoo-contrib/getchoo/dogfood-experimental-installer
Add experimental installer to installer tests
2025-10-08 04:05:46 +00:00
Jörg Thalheim 2ce343716d Merge pull request #14169 from NixOS/dev-shell-no-separate-debug-info
dev-shell: Disable separateDebugInfo
2025-10-08 04:03:43 +00:00
Jörg Thalheim e69e621578 Merge pull request #14180 from NixOS/release-process-drop-mergify
maintainers: Remove mergify note from release-process.md
2025-10-08 03:34:05 +00:00
Jörg Thalheim d2a0a11a8e Merge pull request #14184 from lovesegfault/nix-better-ci
ci: integrate vm_tests into main tests job
2025-10-08 05:23:53 +02:00
Bernardo Meurer Costa a400ea4257 ci: integrate vm_tests into main tests job
This consolidates the separate vm_tests job into the main tests job,
simplifying the CI workflow. VM tests now run as part of the regular
test matrix.
2025-10-08 02:46:56 +00:00
Sergei Zimmerman fc8b784924 Merge pull request #14147 from lovesegfault/nix-multi-ci
ci: test without s3 and with curl-based-s3
2025-10-08 02:38:03 +03:00
Sergei Zimmerman 0619351326 tests: Move invalid nar tests from tests/functional to libutil-tests
Since 242f362567 we have better infrastructure
for this kind of tests.
2025-10-08 02:02:33 +03:00
Sergei Zimmerman c5b88c22fa dev-shell: Disable separateDebugInfo
This breaks gdb pretty-printers inserted into .debug_gdb_scripts section,
because it implies --compress-debug-sections=zlib, -Wa,--compress-debug-sections.
This is very unfortunate, because then gdb can't use pretty printers for
Boost.Unordered (which are very useful, since boost::unoredred_flat_map is
impossible to debug). This seems perfectly fine to disable in the dev-shell for
the time being.

See [1-3] for further references.

With this change I'm able to use boost's pretty-printers out-of-the box:

```
p *importResolutionCache
$2 = boost::concurrent_flat_map with 1 elements = {[{accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
        get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
      path = "/derivation-internal.nix"}}] = {accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
        get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
      path = "/derivation-internal.nix"}}}
```

When combined with a simple `add-auto-load-safe-path ~/code` in .gdbinit

[1]: https://gerrit.lix.systems/c/lix/+/3880
[2]: https://git.lix.systems/lix-project/lix/issues/1003
[3]: https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
2025-10-08 00:57:34 +03:00
Sergei Zimmerman 75b18a6e47 maintainers: Remove mergify note from release-process.md 2025-10-08 00:51:50 +03:00
Sergei Zimmerman fb117e0cac Merge pull request #14155 from Mic92/backport-action
ci: Switch away from mergify to backport action
2025-10-08 00:32:00 +03:00
Jörg Thalheim 63e8b5f94a ci: Switch away from mergify to backport action
We want to use github native queues.
2025-10-07 23:43:03 +03:00
Eelco Dolstra 559061bf5a Merge pull request #14173 from NixOS/bump-2.33.0
Bump version
2025-10-07 20:42:25 +02:00
Eelco Dolstra b0f567e18b Update mergify.yml 2025-10-07 17:16:57 +02:00
Eelco Dolstra 875f7ec25f Bump version 2025-10-07 17:15:28 +02:00
Bernardo Meurer Costa 8c28283876 ci: test without s3 and with curl-based-s3 2025-10-06 16:24:21 +00:00
Bernardo Meurer Costa 7f22a40e3b build(libstore): assert withAWS xor withCurlS3 2025-10-06 16:22:01 +00:00
Seth Flynn 92d7381826 ci: allow for using the latest build of the experimental installer
Until these repos are potentially merged, this is good for dogfooding
alongside the experimental installer. It also uses the more official
`artifacts.nixos.org` endpoint to install stable releases now

More immediately though, we need a patch for the experimental installer
to really work in CI at all, and that hasn't landed in a tag yet. So,
this lets us use it right from `main`!
2025-10-03 03:37:59 -04:00
Seth Flynn d2293fb458 ci: enable experimental installer tests 2025-10-03 01:26:55 -04:00
Seth Flynn 2cbbb63628 ci: enable use of the experimental installer 2025-10-03 01:26:52 -04:00
Eelco Dolstra eec4dece33 Args::Flag: Add required attribute 2025-09-26 16:01:59 +02:00
250 changed files with 4815 additions and 4027 deletions

No files matched your search

@@ -4,12 +4,22 @@ inputs:
dogfood:
description: "Whether to use Nix installed from the latest artifact from master branch"
required: true # Be explicit about the fact that we are using unreleased artifacts
experimental-installer:
description: "Whether to use the experimental installer to install Nix"
default: false
experimental-installer-version:
description: "Version of the experimental installer to use. If `latest`, the newest artifact from the default branch is used."
# TODO: This should probably be pinned to a release after https://github.com/NixOS/experimental-nix-installer/pull/49 lands in one
default: "latest"
extra_nix_config:
description: "Gets appended to `/etc/nix/nix.conf` if passed."
install_url:
description: "URL of the Nix installer"
required: false
default: "https://releases.nixos.org/nix/nix-2.30.2/install"
tarball_url:
description: "URL of the Nix tarball to use with the experimental installer"
required: false
github_token:
description: "Github token"
required: true
@@ -37,14 +47,74 @@ runs:
gh run download "$RUN_ID" --repo "$DOGFOOD_REPO" -n "$INSTALLER_ARTIFACT" -D "$INSTALLER_DOWNLOAD_DIR"
echo "installer-path=file://$INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$INSTALLER_DOWNLOAD_DIR" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
echo "::notice ::Dogfooding Nix installer from master (https://github.com/$DOGFOOD_REPO/actions/runs/$RUN_ID)"
env:
GH_TOKEN: ${{ inputs.github_token }}
DOGFOOD_REPO: "NixOS/nix"
- name: "Gather system info for experimental installer"
shell: bash
if: ${{ inputs.experimental-installer == 'true' }}
run: |
echo "::notice Using experimental installer from $EXPERIMENTAL_INSTALLER_REPO (https://github.com/$EXPERIMENTAL_INSTALLER_REPO)"
if [ "$RUNNER_OS" == "Linux" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="linux"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
elif [ "$RUNNER_OS" == "macOS" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="darwin"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_OS: $RUNNER_OS"
exit 1
fi
if [ "$RUNNER_ARCH" == "X64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=x86_64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
elif [ "$RUNNER_ARCH" == "ARM64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=aarch64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_ARCH: $RUNNER_ARCH"
exit 1
fi
echo "EXPERIMENTAL_INSTALLER_ARTIFACT=nix-installer-$EXPERIMENTAL_INSTALLER_ARCH-$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
env:
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- name: "Download latest experimental installer"
shell: bash
id: download-latest-experimental-installer
if: ${{ inputs.experimental-installer == 'true' && inputs.experimental-installer-version == 'latest' }}
run: |
RUN_ID=$(gh run list --repo "$EXPERIMENTAL_INSTALLER_REPO" --workflow ci.yml --branch main --status success --json databaseId --jq ".[0].databaseId")
EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR="$GITHUB_WORKSPACE/$EXPERIMENTAL_INSTALLER_ARTIFACT"
mkdir -p "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
gh run download "$RUN_ID" --repo "$EXPERIMENTAL_INSTALLER_REPO" -n "$EXPERIMENTAL_INSTALLER_ARTIFACT" -D "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
# Executable permissions are lost in artifacts
find $EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR -type f -exec chmod +x {} +
echo "installer-path=$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
env:
GH_TOKEN: ${{ inputs.github_token }}
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- uses: cachix/install-nix-action@c134e4c9e34bac6cab09cf239815f9339aaaf84e # v31.5.1
if: ${{ inputs.experimental-installer != 'true' }}
with:
# Ternary operator in GHA: https://www.github.com/actions/runner/issues/409#issuecomment-752775072
install_url: ${{ inputs.dogfood == 'true' && format('{0}/install', steps.download-nix-installer.outputs.installer-path) || inputs.install_url }}
install_options: ${{ inputs.dogfood == 'true' && format('--tarball-url-prefix {0}', steps.download-nix-installer.outputs.installer-path) || '' }}
extra_nix_config: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/nix-installer-action@786fff0690178f1234e4e1fe9b536e94f5433196 # v20
if: ${{ inputs.experimental-installer == 'true' }}
with:
diagnostic-endpoint: ""
# TODO: It'd be nice to use `artifacts.nixos.org` for both of these, maybe through an `/experimental-installer/latest` endpoint? or `/commit/<hash>`?
local-root: ${{ inputs.experimental-installer-version == 'latest' && steps.download-latest-experimental-installer.outputs.installer-path || '' }}
source-url: ${{ inputs.experimental-installer-version != 'latest' && 'https://artifacts.nixos.org/experimental-installer/tag/${{ inputs.experimental-installer-version }}/${{ env.EXPERIMENTAL_INSTALLER_ARTIFACT }}' || '' }}
nix-package-url: ${{ inputs.dogfood == 'true' && steps.download-nix-installer.outputs.tarball-path || (inputs.tarball_url || '') }}
extra-conf: ${{ inputs.extra_nix_config }}
+37
View File
@@ -0,0 +1,37 @@
name: Backport
on:
pull_request_target:
types: [closed, labeled]
permissions:
contents: read
jobs:
backport:
name: Backport Pull Request
permissions:
# for korthout/backport-action
contents: write
pull-requests: write
if: github.repository_owner == 'NixOS' && github.event.pull_request.merged == true && (github.event_name != 'labeled' || startsWith('backport', github.event.label.name))
runs-on: ubuntu-24.04-arm
steps:
- name: Generate GitHub App token
id: generate-token
uses: actions/create-github-app-token@v2
with:
app-id: ${{ vars.CI_APP_ID }}
private-key: ${{ secrets.CI_APP_PRIVATE_KEY }}
- uses: actions/checkout@v5
with:
ref: ${{ github.event.pull_request.head.sha }}
# required to find all branches
fetch-depth: 0
- name: Create backport PRs
uses: korthout/backport-action@d07416681cab29bf2661702f925f020aaa962997 # v3.4.1
id: backport
with:
# Config README: https://github.com/korthout/backport-action#backport-action
github_token: ${{ steps.generate-token.outputs.token }}
github_workspace: ${{ github.workspace }}
auto_merge_enabled: true
pull_description: |-
Automatic backport to `${target_branch}`, triggered by a label in #${pull_number}.
+32 -23
View File
@@ -4,6 +4,8 @@ on:
pull_request:
merge_group:
push:
branches:
- master
workflow_dispatch:
inputs:
dogfood:
@@ -100,6 +102,12 @@ jobs:
nix build --file ci/gha/tests/wrapper.nix componentTests -L \
--arg withInstrumentation ${{ matrix.instrumented }} \
--argstr stdenv "${{ matrix.stdenv }}"
- name: Run VM tests
run: |
nix build --file ci/gha/tests/wrapper.nix vmTests -L \
--arg withInstrumentation ${{ matrix.instrumented }} \
--argstr stdenv "${{ matrix.stdenv }}"
if: ${{ matrix.os == 'linux' }}
- name: Run flake checks and prepare the installer tarball
run: |
ci/gha/tests/build-checks
@@ -135,9 +143,19 @@ jobs:
- scenario: on ubuntu
runs-on: ubuntu-24.04
os: linux
experimental-installer: false
- scenario: on macos
runs-on: macos-14
os: darwin
experimental-installer: false
- scenario: on ubuntu (experimental)
runs-on: ubuntu-24.04
os: linux
experimental-installer: true
- scenario: on macos (experimental)
runs-on: macos-14
os: darwin
experimental-installer: true
name: installer test ${{ matrix.scenario }}
runs-on: ${{ matrix.runs-on }}
steps:
@@ -149,11 +167,22 @@ jobs:
path: out
- name: Looking up the installer tarball URL
id: installer-tarball-url
run: echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
run: |
echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$GITHUB_WORKSPACE/out" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
- uses: cachix/install-nix-action@v31
if: ${{ !matrix.experimental-installer }}
with:
install_url: ${{ format('{0}/install', steps.installer-tarball-url.outputs.installer-url) }}
install_options: ${{ format('--tarball-url-prefix {0}', steps.installer-tarball-url.outputs.installer-url) }}
- uses: ./.github/actions/install-nix-action
if: ${{ matrix.experimental-installer }}
with:
dogfood: false
experimental-installer: true
tarball_url: ${{ steps.installer-tarball-url.outputs.tarball-path }}
github_token: ${{ secrets.GITHUB_TOKEN }}
- run: sudo apt install fish zsh
if: matrix.os == 'linux'
- run: brew install fish
@@ -185,7 +214,7 @@ jobs:
echo "docker=${{ env._DOCKER_SECRETS != '' }}" >> $GITHUB_OUTPUT
docker_push_image:
needs: [tests, vm_tests, check_secrets]
needs: [tests, check_secrets]
permissions:
contents: read
packages: write
@@ -238,28 +267,8 @@ jobs:
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
vm_tests:
needs: basic-checks
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
- uses: ./.github/actions/install-nix-action
with:
dogfood: ${{ github.event_name == 'workflow_dispatch' && inputs.dogfood || github.event_name != 'workflow_dispatch' }}
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: |
nix build -L \
.#hydraJobs.tests.functional_user \
.#hydraJobs.tests.githubFlakes \
.#hydraJobs.tests.nix-docker \
.#hydraJobs.tests.tarballFlakes \
;
flake_regressions:
needs: vm_tests
needs: tests
runs-on: ubuntu-24.04
steps:
- name: Checkout nix
-174
View File
@@ -1,174 +0,0 @@
queue_rules:
- name: default
# all required tests need to go here
merge_conditions:
- check-success=tests on macos
- check-success=tests on ubuntu
- check-success=installer test on macos
- check-success=installer test on ubuntu
- check-success=vm_tests
batch_size: 5
pull_request_rules:
- name: merge using the merge queue
conditions:
- base~=master|.+-maintenance
- label~=merge-queue|dependencies
actions:
queue: {}
# The rules below will first create backport pull requests and put those in a merge queue.
- name: backport patches to 2.18
conditions:
- label=backport 2.18-maintenance
actions:
backport:
branches:
- 2.18-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.19
conditions:
- label=backport 2.19-maintenance
actions:
backport:
branches:
- 2.19-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.20
conditions:
- label=backport 2.20-maintenance
actions:
backport:
branches:
- 2.20-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.21
conditions:
- label=backport 2.21-maintenance
actions:
backport:
branches:
- 2.21-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.22
conditions:
- label=backport 2.22-maintenance
actions:
backport:
branches:
- 2.22-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.23
conditions:
- label=backport 2.23-maintenance
actions:
backport:
branches:
- 2.23-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.24
conditions:
- label=backport 2.24-maintenance
actions:
backport:
branches:
- "2.24-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.25
conditions:
- label=backport 2.25-maintenance
actions:
backport:
branches:
- "2.25-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.26
conditions:
- label=backport 2.26-maintenance
actions:
backport:
branches:
- "2.26-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.27
conditions:
- label=backport 2.27-maintenance
actions:
backport:
branches:
- "2.27-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.28
conditions:
- label=backport 2.28-maintenance
actions:
backport:
branches:
- "2.28-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.29
conditions:
- label=backport 2.29-maintenance
actions:
backport:
branches:
- "2.29-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.30
conditions:
- label=backport 2.30-maintenance
actions:
backport:
branches:
- "2.30-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.31
conditions:
- label=backport 2.31-maintenance
actions:
backport:
branches:
- "2.31-maintenance"
labels:
- automatic backport
- merge-queue
+1 -1
View File
@@ -1 +1 @@
2.32.2
2.33.0
+29 -13
View File
@@ -21,16 +21,6 @@ let
packages' = nixFlake.packages.${system};
stdenv = (getStdenv pkgs);
enableSanitizersLayer = finalAttrs: prevAttrs: {
mesonFlags =
(prevAttrs.mesonFlags or [ ])
++ [ (lib.mesonOption "b_sanitize" "address,undefined") ]
++ (lib.optionals stdenv.cc.isClang [
# https://www.github.com/mesonbuild/meson/issues/764
(lib.mesonBool "b_lundef" false)
]);
};
collectCoverageLayer = finalAttrs: prevAttrs: {
env =
let
@@ -53,14 +43,15 @@ let
'';
};
componentOverrides =
(lib.optional withSanitizers enableSanitizersLayer)
++ (lib.optional withCoverage collectCoverageLayer);
componentOverrides = (lib.optional withCoverage collectCoverageLayer);
in
rec {
nixComponentsInstrumented = nixComponents.overrideScope (
final: prev: {
withASan = withSanitizers;
withUBSan = withSanitizers;
nix-store-tests = prev.nix-store-tests.override { withBenchmarks = true; };
# Boehm is incompatible with ASAN.
nix-expr = prev.nix-expr.override { enableGC = !withSanitizers; };
@@ -71,6 +62,14 @@ rec {
}
);
# Import NixOS tests using the instrumented components
nixosTests = import ../../../tests/nixos {
inherit lib pkgs;
nixComponents = nixComponentsInstrumented;
nixpkgs = nixFlake.inputs.nixpkgs;
inherit (nixFlake.inputs) nixpkgs-23-11;
};
/**
Top-level tests for the flake outputs, as they would be built by hydra.
These tests generally can't be overridden to run with sanitizers.
@@ -117,6 +116,7 @@ rec {
) nixComponentsInstrumented)
// lib.optionalAttrs (pkgs.stdenv.hostPlatform == pkgs.stdenv.buildPlatform) {
"${componentTestsPrefix}nix-functional-tests" = nixComponentsInstrumented.nix-functional-tests;
"${componentTestsPrefix}nix-json-schema-checks" = nixComponentsInstrumented.nix-json-schema-checks;
};
codeCoverage =
@@ -221,4 +221,20 @@ rec {
{
inherit coverageProfileDrvs mergedProfdata coverageReports;
};
vmTests = {
inherit (nixosTests) s3-binary-cache-store;
}
// lib.optionalAttrs (!withSanitizers && !withCoverage) {
# evalNixpkgs uses non-instrumented components from hydraJobs, so only run it
# when not testing with sanitizers to avoid rebuilding nix
inherit (hydraJobs.tests) evalNixpkgs;
# FIXME: CI times out when building vm tests instrumented
inherit (nixosTests)
functional_user
githubFlakes
nix-docker
tarballFlakes
;
};
}
+2 -2
View File
@@ -15,7 +15,6 @@ pymod = import('python')
python = pymod.find_installation('python3')
nix_env_for_docs = {
'ASAN_OPTIONS' : 'abort_on_error=1:print_summary=1:detect_leaks=0',
'HOME' : '/dummy',
'NIX_CONF_DIR' : '/dummy',
'NIX_SSL_CERT_FILE' : '/dummy/no-ca-bundle.crt',
@@ -89,7 +88,7 @@ manual = custom_target(
@0@ @INPUT0@ @CURRENT_SOURCE_DIR@ > @DEPFILE@
@0@ @INPUT1@ summary @2@ < @CURRENT_SOURCE_DIR@/source/SUMMARY.md.in > @2@/source/SUMMARY.md
sed -e 's|@version@|@3@|g' < @INPUT2@ > @2@/book.toml
@4@ -r --include='*.md' @CURRENT_SOURCE_DIR@/ @2@/
@4@ -r -L --include='*.md' @CURRENT_SOURCE_DIR@/ @2@/
(cd @2@; RUST_LOG=warn @1@ build -d @2@ 3>&2 2>&1 1>&3) | { grep -Fv "because fragment resolution isn't implemented" || :; } 3>&2 2>&1 1>&3
rm -rf @2@/manual
mv @2@/html @2@/manual
@@ -116,6 +115,7 @@ manual = custom_target(
builtins_md,
rl_next_generated,
summary_rl_next,
json_schema_generated_files,
nix_input,
],
output : [
+5
View File
@@ -12,6 +12,7 @@
rsync,
nix-cli,
changelog-d,
json-schema-for-humans,
officialRelease,
# Configuration Options
@@ -32,6 +33,9 @@ mkMesonDerivation (finalAttrs: {
fileset.difference
(fileset.unions [
../../.version
# For example JSON
../../src/libutil-tests/data/memory-source-accessor
../../src/libutil-tests/data/hash
# Too many different types of files to filter for now
../../doc/manual
./.
@@ -55,6 +59,7 @@ mkMesonDerivation (finalAttrs: {
jq
python3
rsync
json-schema-for-humans
changelog-d
]
++ lib.optionals (!officialRelease) [
@@ -0,0 +1,26 @@
---
synopsis: "Improved S3 binary cache support via HTTP"
prs: [13823, 14026, 14120, 14131, 14135, 14144, 14170, 14190, 14198, 14206, 14209, 14222, 14223, 13752]
issues: [13084, 12671, 11748, 12403]
---
S3 binary cache operations now happen via HTTP, leveraging `libcurl`'s native
AWS SigV4 authentication instead of the AWS C++ SDK, providing significant
improvements:
- **Reduced memory usage**: Eliminates memory buffering issues that caused
segfaults with large files
- **Fixed upload reliability**: Resolves AWS SDK chunking errors
(`InvalidChunkSizeError`)
- **Lighter dependencies**: Uses lightweight `aws-crt-cpp` instead of full
`aws-cpp-sdk`, reducing build complexity
The new implementation requires curl >= 7.75.0 and `aws-crt-cpp` for credential
management.
All existing S3 URL formats and parameters remain supported, with the notable
exception of multi-part uploads, which are no longer supported.
Note that this change also means Nix now supports S3 binary cache stores even
if build without `aws-crt-cpp`, but only for public buckets which do not
require auth.
+2
View File
@@ -117,6 +117,8 @@
- [Architecture and Design](architecture/architecture.md)
- [Formats and Protocols](protocols/index.md)
- [JSON Formats](protocols/json/index.md)
- [File System Object](protocols/json/file-system-object.md)
- [Hash](protocols/json/hash.md)
- [Store Object Info](protocols/json/store-object-info.md)
- [Derivation](protocols/json/derivation.md)
- [Serving Tarball Flakes](protocols/tarball-fetcher.md)
+17 -6
View File
@@ -25,20 +25,31 @@ nix build .#nix-manual
and open `./result/share/doc/nix/manual/index.html`.
To build the manual incrementally, [enter the development shell](./building.md) and run:
To build the manual incrementally, [enter the development shell](./building.md) and configure with `doc-gen` enabled:
**If using interactive `nix develop`:**
```console
make manual-html-open -j $NIX_BUILD_CORES
$ nix develop
$ mesonFlags="$mesonFlags -Ddoc-gen=true" mesonConfigurePhase
```
In order to reflect changes to the [Makefile for the manual], clear all generated files before re-building:
[Makefile for the manual]: https://github.com/NixOS/nix/blob/master/doc/manual/local.mk
**If using direnv:**
```console
rm $(git ls-files doc/manual/ -o | grep -F '.md') && rmdir doc/manual/source/command-ref/new-cli && make manual-html -j $NIX_BUILD_CORES
$ direnv allow
$ bash -c 'source $stdenv/setup && mesonFlags="$mesonFlags -Ddoc-gen=true" mesonConfigurePhase'
```
Then build the manual:
```console
$ cd build
$ meson compile manual
```
The HTML manual will be generated at `build/src/nix-manual/manual/index.html`.
## Style guide
The goal of this style guide is to make it such that
+3
View File
@@ -1,3 +1,6 @@
# Process JSON schema documentation
subdir('protocols')
summary_rl_next = custom_target(
command : [
bash,
+5 -118
View File
@@ -1,120 +1,7 @@
# Derivation JSON Format
{{#include derivation-v3-fixed.md}}
> **Warning**
>
> This JSON format is currently
> [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
> and subject to change.
<!--
## Raw Schema
The JSON serialization of a
[derivations](@docroot@/glossary.md#gloss-store-derivation)
is a JSON object with the following fields:
* `name`:
The name of the derivation.
This is used when calculating the store paths of the derivation's outputs.
* `version`:
Must be `3`.
This is a guard that allows us to continue evolving this format.
The choice of `3` is fairly arbitrary, but corresponds to this informal version:
- Version 0: A-Term format
- Version 1: Original JSON format, with ugly `"r:sha256"` inherited from A-Term format.
- Version 2: Separate `method` and `hashAlgo` fields in output specs
- Verison 3: Drop store dir from store paths, just include base name.
Note that while this format is experimental, the maintenance of versions is best-effort, and not promised to identify every change.
* `outputs`:
Information about the output paths of the derivation.
This is a JSON object with one member per output, where the key is the output name and the value is a JSON object with these fields:
* `path`:
The output path, if it is known in advanced.
Otherwise, `null`.
* `method`:
For an output which will be [content addressed], a string representing the [method](@docroot@/store/store-object/content-address.md) of content addressing that is chosen.
Valid method strings are:
- [`flat`](@docroot@/store/store-object/content-address.md#method-flat)
- [`nar`](@docroot@/store/store-object/content-address.md#method-nix-archive)
- [`text`](@docroot@/store/store-object/content-address.md#method-text)
- [`git`](@docroot@/store/store-object/content-address.md#method-git)
Otherwise, `null`.
* `hashAlgo`:
For an output which will be [content addressed], the name of the hash algorithm used.
Valid algorithm strings are:
- `blake3`
- `md5`
- `sha1`
- `sha256`
- `sha512`
* `hash`:
For fixed-output derivations, the expected content hash in base-16.
> **Example**
>
> ```json
> "outputs": {
> "out": {
> "method": "nar",
> "hashAlgo": "sha256",
> "hash": "6fc80dcc62179dbc12fc0b5881275898f93444833d21b89dfe5f7fbcbb1d0d62"
> }
> }
> ```
* `inputSrcs`:
A list of store paths on which this derivation depends.
> **Example**
>
> ```json
> "inputSrcs": [
> "47y241wqdhac3jm5l7nv0x4975mb1975-separate-debug-info.sh",
> "56d0w71pjj9bdr363ym3wj1zkwyqq97j-fix-pop-var-context-error.patch"
> ]
> ```
* `inputDrvs`:
A JSON object specifying the derivations on which this derivation depends, and what outputs of those derivations.
> **Example**
>
> ```json
> "inputDrvs": {
> "6lkh5yi7nlb7l6dr8fljlli5zfd9hq58-curl-7.73.0.drv": ["dev"],
> "fn3kgnfzl5dzym26j8g907gq3kbm8bfh-unzip-6.0.drv": ["out"]
> }
> ```
specifies that this derivation depends on the `dev` output of `curl`, and the `out` output of `unzip`.
* `system`:
The system type on which this derivation is to be built
(e.g. `x86_64-linux`).
* `builder`:
The absolute path of the program to be executed to run the build.
Typically this is the `bash` shell
(e.g. `/nix/store/r3j288vpmczbl500w6zz89gyfa4nr0b1-bash-4.4-p23/bin/bash`).
* `args`:
The command-line arguments passed to the `builder`.
* `env`:
The environment passed to the `builder`.
* `structuredAttrs`:
[Strucutured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
Structured attributes are JSON, and thus embedded as-is.
[JSON Schema for Derivation v3](schema/derivation-v3.json)
-->
@@ -0,0 +1,21 @@
{{#include file-system-object-v1-fixed.md}}
## Examples
### Simple
```json
{{#include schema/file-system-object-v1/simple.json}}
```
### Complex
```json
{{#include schema/file-system-object-v1/complex.json}}
```
<!--
## Raw Schema
[JSON Schema for File System Object v1](schema/file-system-object-v1.json)
-->
@@ -0,0 +1,15 @@
# For some reason, backticks in the JSON schema are being escaped rather
# than being kept as intentional code spans. This removes all backtick
# escaping, which is an ugly solution, but one that is fine, because we
# are not using backticks for any other purpose.
s/\\`/`/g
# The way that semi-external references are rendered (i.e. ones to
# sibling schema files, as opposed to separate website ones, is not nice
# for humans. Replace it with a nice relative link within the manual
# instead.
#
# As we have more such relative links, more replacements of this nature
# should appear below.
s^\(./hash-v1.yaml\)\?#/$defs/algorithm^[JSON format for `Hash`](./hash.html#algorithm)^g
s^#/\$defs/\(regular\|symlink\|directory\)^In this schema^g
+33
View File
@@ -0,0 +1,33 @@
{{#include hash-v1-fixed.md}}
## Examples
### SHA-256 with Base64 encoding
```json
{{#include schema/hash-v1/sha256-base64.json}}
```
### SHA-256 with Base16 (hexadecimal) encoding
```json
{{#include schema/hash-v1/sha256-base16.json}}
```
### SHA-256 with Nix32 encoding
```json
{{#include schema/hash-v1/sha256-nix32.json}}
```
### BLAKE3 with Base64 encoding
```json
{{#include schema/hash-v1/blake3-base64.json}}
```
<!--
## Raw Schema
[JSON Schema for Hash v1](schema/hash-v1.json)
-->
@@ -0,0 +1,17 @@
# Configuration file for json-schema-for-humans
#
# https://github.com/coveooss/json-schema-for-humans/blob/main/docs/examples/examples_md_default/Configuration.md
template_name: md
show_toc: true
# impure timestamp and distracting
with_footer: false
recursive_detection_depth: 3
show_breadcrumbs: false
description_is_markdown: true
template_md_options:
properties_table_columns:
- Property
- Type
- Pattern
- Title/Description
@@ -0,0 +1,75 @@
# Tests in: ../../../../src/json-schema-checks
fs = import('fs')
# Find json-schema-for-humans if available
json_schema_for_humans = find_program('generate-schema-doc', required : false)
# Configuration for json-schema-for-humans
json_schema_config = files('json-schema-for-humans-config.yaml')
schemas = [
'file-system-object-v1',
'hash-v1',
'derivation-v3',
]
schema_files = files()
foreach schema_name : schemas
schema_files += files('schema' / schema_name + '.yaml')
endforeach
schema_outputs = []
foreach schema_name : schemas
schema_outputs += schema_name + '.md'
endforeach
json_schema_generated_files = []
# Generate markdown documentation from JSON schema
# Note: output must be just a filename, not a path
gen_file = custom_target(
schema_name + '-schema-docs.tmp',
command : [
json_schema_for_humans,
'--config-file',
json_schema_config,
meson.current_source_dir() / 'schema',
meson.current_build_dir(),
],
input : schema_files + [
json_schema_config,
],
output : schema_outputs,
capture : false,
build_by_default : true,
)
idx = 0
if json_schema_for_humans.found()
foreach schema_name : schemas
#schema_file = 'schema' / schema_name + '.yaml'
# There is one so-so hack, and one horrible hack being done here.
sedded_file = custom_target(
schema_name + '-schema-docs',
command : [
'sed',
'-f',
# Out of line to avoid https://github.com/mesonbuild/meson/issues/1564
files('fixup-json-schema-generated-doc.sed'),
'@INPUT@',
],
capture : true,
input : gen_file[idx],
output : schema_name + '-fixed.md',
)
idx += 1
json_schema_generated_files += [ sedded_file ]
endforeach
else
warning(
'json-schema-for-humans not found, skipping JSON schema documentation generation',
)
endif
@@ -0,0 +1,178 @@
"$schema": http://json-schema.org/draft-04/schema#
"$id": https://nix.dev/manual/nix/latest/protocols/json/schema/derivation-v3.json
title: Derivation
description: |
Experimental JSON representation of a Nix derivation (version 3).
This schema describes the JSON representation of Nix's `Derivation` type.
> **Warning**
>
> This JSON format is currently
> [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
> and subject to change.
type: object
required:
- name
- version
- outputs
- inputSrcs
- inputDrvs
- system
- builder
- args
- env
properties:
name:
type: string
title: Derivation name
description: |
The name of the derivation.
Used when calculating store paths for the derivation’s outputs.
version:
const: 3
title: Format version (must be 3)
description: |
Must be `3`.
This is a guard that allows us to continue evolving this format.
The choice of `3` is fairly arbitrary, but corresponds to this informal version:
- Version 0: A-Term format
- Version 1: Original JSON format, with ugly `"r:sha256"` inherited from A-Term format.
- Version 2: Separate `method` and `hashAlgo` fields in output specs
- Version 3: Drop store dir from store paths, just include base name.
Note that while this format is experimental, the maintenance of versions is best-effort, and not promised to identify every change.
outputs:
type: object
title: Output specifications
description: |
Information about the output paths of the derivation.
This is a JSON object with one member per output, where the key is the output name and the value is a JSON object as described.
> **Example**
>
> ```json
> "outputs": {
> "out": {
> "method": "nar",
> "hashAlgo": "sha256",
> "hash": "6fc80dcc62179dbc12fc0b5881275898f93444833d21b89dfe5f7fbcbb1d0d62"
> }
> }
> ```
additionalProperties:
"$ref": "#/$defs/output"
inputSrcs:
type: array
title: Input source paths
description: |
List of store paths on which this derivation depends.
> **Example**
>
> ```json
> "inputSrcs": [
> "47y241wqdhac3jm5l7nv0x4975mb1975-separate-debug-info.sh",
> "56d0w71pjj9bdr363ym3wj1zkwyqq97j-fix-pop-var-context-error.patch"
> ]
> ```
items:
type: string
inputDrvs:
type: object
title: Input derivations
description: |
Mapping of derivation paths to lists of output names they provide.
> **Example**
>
> ```json
> "inputDrvs": {
> "6lkh5yi7nlb7l6dr8fljlli5zfd9hq58-curl-7.73.0.drv": ["dev"],
> "fn3kgnfzl5dzym26j8g907gq3kbm8bfh-unzip-6.0.drv": ["out"]
> }
> ```
>
> specifies that this derivation depends on the `dev` output of `curl`, and the `out` output of `unzip`.
system:
type: string
title: Build system type
description: |
The system type on which this derivation is to be built
(e.g. `x86_64-linux`).
builder:
type: string
title: Build program path
description: |
Absolute path of the program used to perform the build.
Typically this is the `bash` shell
(e.g. `/nix/store/r3j288vpmczbl500w6zz89gyfa4nr0b1-bash-4.4-p23/bin/bash`).
args:
type: array
title: Builder arguments
description: |
Command-line arguments passed to the `builder`.
items:
type: string
env:
type: object
title: Environment variables
description: |
Environment variables passed to the `builder`.
additionalProperties:
type: string
structuredAttrs:
title: Structured attributes
description: |
[Structured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
Structured attributes are JSON, and thus embedded as-is.
type: object
additionalProperties: true
"$defs":
output:
type: object
properties:
path:
type: string
title: Output path
description: |
The output path, if known in advance.
method:
type: string
title: Content addressing method
enum: [flat, nar, text, git]
description: |
For an output which will be [content addressed](@docroot@/store/derivation/outputs/content-address.md), a string representing the [method](@docroot@/store/store-object/content-address.md) of content addressing that is chosen.
Valid method strings are:
- [`flat`](@docroot@/store/store-object/content-address.md#method-flat)
- [`nar`](@docroot@/store/store-object/content-address.md#method-nix-archive)
- [`text`](@docroot@/store/store-object/content-address.md#method-text)
- [`git`](@docroot@/store/store-object/content-address.md#method-git)
hashAlgo:
title: Hash algorithm
"$ref": "./hash-v1.yaml#/$defs/algorithm"
hash:
type: string
title: Expected hash value
description: |
For fixed-output derivations, the expected content hash in base-16.
@@ -0,0 +1 @@
../../../../../../src/libutil-tests/data/memory-source-accessor
@@ -0,0 +1,65 @@
"$schema": http://json-schema.org/draft-04/schema#
"$id": https://nix.dev/manual/nix/latest/protocols/json/schema/file-system-object-v1.json
title: File System Object
description: |
This schema describes the JSON representation of Nix's [File System Object](@docroot@/store/file-system-object.md).
The schema is recursive because file system objects contain other file system objects.
type: object
required: ["type"]
properties:
type:
type: string
enum: ["regular", "symlink", "directory"]
# Enforce conditional structure based on `type`
anyOf:
- $ref: "#/$defs/regular"
required: ["type", "contents"]
- $ref: "#/$defs/symlink"
required: ["type", "target"]
- $ref: "#/$defs/directory"
required: ["type", "contents"]
"$defs":
regular:
title: Regular File
required: ["contents"]
properties:
type:
const: "regular"
contents:
type: string
description: Base64-encoded file contents
executable:
type: boolean
description: Whether the file is executable.
default: false
additionalProperties: false
symlink:
title: Symbolic Link
required: ["target"]
properties:
type:
const: "symlink"
target:
type: string
description: Target path of the symlink.
additionalProperties: false
directory:
title: Directory
required: ["contents"]
properties:
type:
const: "directory"
contents:
type: object
description: |
Map of names to nested file system objects (for type=directory)
additionalProperties:
$ref: "#"
additionalProperties: false
+1
View File
@@ -0,0 +1 @@
../../../../../../src/libutil-tests/data/hash/
@@ -0,0 +1,54 @@
"$schema": http://json-schema.org/draft-04/schema#
"$id": https://nix.dev/manual/nix/latest/protocols/json/schema/hash-v1.json
title: Hash
description: |
A cryptographic hash value used throughout Nix for content addressing and integrity verification.
This schema describes the JSON representation of Nix's `Hash` type.
type: object
properties:
algorithm:
"$ref": "#/$defs/algorithm"
format:
type: string
enum:
- base64
- nix32
- base16
- sri
title: Hash format
description: |
The encoding format of the hash value.
- `base64` uses standard Base64 encoding [RFC 4648, section 4](https://datatracker.ietf.org/doc/html/rfc4648#section-4)
- `nix32` is Nix-specific base-32 encoding
- `base16` is lowercase hexadecimal
- `sri` is the [Subresource Integrity format](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity).
hash:
type: string
title: Hash
description: |
The encoded hash value, itself.
It is specified in the format specified by the `format` field.
It must be the right length for the hash algorithm specified in the `algorithm` field, also.
The hash value does not include any algorithm prefix.
required:
- algorithm
- format
- hash
additionalProperties: false
"$defs":
algorithm:
type: string
enum:
- blake3
- md5
- sha1
- sha256
- sha512
title: Hash algorithm
description: |
The hash algorithm used to compute the hash value.
`blake3` is currently experimental and requires the [`blake-hashing`](@docroot@/development/experimental-features.md#xp-feature-blake-hashing) experimental feature.
+2
View File
@@ -0,0 +1,2 @@
# Process JSON schema documentation
subdir('json')
+1 -1
View File
@@ -106,7 +106,7 @@ The system type on which the [`builder`](#attr-builder) executable is meant to b
A necessary condition for Nix to schedule a given derivation on some [Nix instance] is for the "system" of that derivation to match that instance's [`system` configuration option] or [`extra-platforms` configuration option].
By putting the `system` in each derivation, Nix allows *heterogenous* build plans, where not all steps can be run on the same machine or same sort of machine.
By putting the `system` in each derivation, Nix allows *heterogeneous* build plans, where not all steps can be run on the same machine or same sort of machine.
Nix can schedule builds such that it automatically builds on other platforms by [forwarding build requests](@docroot@/advanced-topics/distributed-builds.md) to other Nix instances.
[`system` configuration option]: @docroot@/command-ref/conf-file.md#conf-system
+5 -1
View File
@@ -32,7 +32,7 @@
let
inherit (nixpkgs) lib;
officialRelease = true;
officialRelease = false;
linux32BitSystems = [ "i686-linux" ];
linux64BitSystems = [
@@ -413,6 +413,10 @@
supportsCross = false;
};
"nix-json-schema-checks" = {
supportsCross = false;
};
"nix-perl-bindings" = {
supportsCross = false;
};
-1
View File
@@ -142,7 +142,6 @@ release:
$ git pull
$ NEW_VERSION=2.13.0
$ echo $NEW_VERSION > .version
$ ... edit .mergify.yml to add the previous version ...
$ git checkout -b bump-$NEW_VERSION
$ git commit -a -m 'Bump version'
$ git push --set-upstream origin bump-$NEW_VERSION
+1
View File
@@ -60,3 +60,4 @@ if get_option('unit-tests')
subproject('libflake-tests')
endif
subproject('nix-functional-tests')
subproject('json-schema-checks')
@@ -0,0 +1,6 @@
extern "C" [[gnu::retain, gnu::weak]] const char * __asan_default_options()
{
// We leak a bunch of memory knowingly on purpose. It's not worthwhile to
// diagnose that memory being leaked for now.
return "abort_on_error=1:print_summary=1:detect_leaks=0:detect_odr_violation=0";
}
@@ -1,7 +1,3 @@
asan_test_options_env = {
'ASAN_OPTIONS' : 'abort_on_error=1:print_summary=1:detect_leaks=0',
}
# Clang gets grumpy about missing libasan symbols if -shared-libasan is not
# passed when building shared libs, at least on Linux
if cxx.get_id() == 'clang' and ('address' in get_option('b_sanitize') or 'undefined' in get_option(
@@ -10,3 +6,6 @@ if cxx.get_id() == 'clang' and ('address' in get_option('b_sanitize') or 'undefi
add_project_link_arguments('-shared-libasan', language : 'cpp')
endif
if 'address' in get_option('b_sanitize')
deps_other += declare_dependency(sources : 'asan-options.cc')
endif
@@ -46,3 +46,4 @@ endif
nix_soversion = meson.project_version().split('pre')[0]
subdir('assert-fail')
subdir('asan-options')
+38 -1
View File
@@ -204,6 +204,25 @@ let
mesonFlags = [ (lib.mesonBool "b_asneeded" false) ] ++ prevAttrs.mesonFlags or [ ];
};
enableSanitizersLayer =
finalAttrs: prevAttrs:
let
sanitizers = lib.optional scope.withASan "address" ++ lib.optional scope.withUBSan "undefined";
in
{
mesonFlags =
(prevAttrs.mesonFlags or [ ])
++ lib.optionals (lib.length sanitizers > 0) (
[
(lib.mesonOption "b_sanitize" (lib.concatStringsSep "," sanitizers))
]
++ (lib.optionals stdenv.cc.isClang [
# https://www.github.com/mesonbuild/meson/issues/764
(lib.mesonBool "b_lundef" false)
])
);
};
nixDefaultsLayer = finalAttrs: prevAttrs: {
strictDeps = prevAttrs.strictDeps or true;
enableParallelBuilding = true;
@@ -246,6 +265,16 @@ in
inherit filesetToSource;
/**
Whether meson components are built with [AddressSanitizer](https://clang.llvm.org/docs/AddressSanitizer.html).
*/
withASan = false;
/**
Whether meson components are built with [UndefinedBehaviorSanitizer](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html).
*/
withUBSan = false;
/**
A user-provided extension function to apply to each component derivation.
*/
@@ -332,6 +361,7 @@ in
setVersionLayer
mesonLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
mkMesonExecutable = mkPackageBuilder [
@@ -342,6 +372,7 @@ in
mesonLayer
mesonBuildLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
mkMesonLibrary = mkPackageBuilder [
@@ -353,6 +384,7 @@ in
mesonBuildLayer
mesonLibraryLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
@@ -406,6 +438,11 @@ in
*/
nix-external-api-docs = callPackage ../src/external-api-docs/package.nix { version = fineVersion; };
/**
JSON schema validation checks
*/
nix-json-schema-checks = callPackage ../src/json-schema-checks/package.nix { };
nix-perl-bindings = callPackage ../src/perl/package.nix { };
/**
@@ -458,7 +495,7 @@ in
Example:
```
overrideScope (finalScope: prevScope: { aws-sdk-cpp = null; })
overrideScope (finalScope: prevScope: { aws-crt-cpp = null; })
```
*/
overrideScope = f: (scope.overrideScope f).nix-everything;
-49
View File
@@ -16,21 +16,6 @@ in
scope: {
inherit stdenv;
aws-sdk-cpp =
(pkgs.aws-sdk-cpp.override {
apis = [
"identity-management"
"s3"
"transfer"
];
customMemoryManagement = false;
}).overrideAttrs
{
# only a stripped down version is built, which takes a lot less resources
# to build, so we don't need a "big-parallel" machine.
requiredSystemFeatures = [ ];
};
boehmgc =
(pkgs.boehmgc.override {
enableLargeConfig = true;
@@ -89,38 +74,4 @@ scope: {
buildPhase = lib.replaceStrings [ "--without-python" ] [ "" ] old.buildPhase;
installPhase = lib.replaceStrings [ "--without-python" ] [ "" ] old.installPhase;
});
libgit2 =
if lib.versionAtLeast pkgs.libgit2.version "1.9.0" then
pkgs.libgit2
else
pkgs.libgit2.overrideAttrs (attrs: {
# libgit2: Nixpkgs 24.11 has < 1.9.0, which needs our patches
nativeBuildInputs =
attrs.nativeBuildInputs or [ ]
# gitMinimal does not build on Windows. See packbuilder patch.
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# Needed for `git apply`; see `prePatch`
pkgs.buildPackages.gitMinimal
];
# Only `git apply` can handle git binary patches
prePatch =
attrs.prePatch or ""
+ lib.optionalString (!stdenv.hostPlatform.isWindows) ''
patch() {
git apply
}
'';
patches =
attrs.patches or [ ]
++ [
./patches/libgit2-mempack-thin-packfile.patch
]
# gitMinimal does not build on Windows, but fortunately this patch only
# impacts interruptibility
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# binary patch; see `prePatch`
./patches/libgit2-packbuilder-callback-interruptible.patch
];
});
}
+42 -32
View File
@@ -70,6 +70,9 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
# We use this shell with the local checkout, not unpackPhase.
src = null;
# Workaround https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
# Remove when gdb fix is rolled out everywhere.
separateDebugInfo = false;
env = {
# For `make format`, to work without installing pre-commit
@@ -93,38 +96,45 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
++ map (transformFlag "libcmd") (ignoreCrossFile pkgs.nixComponents2.nix-cmd.mesonFlags);
nativeBuildInputs =
attrs.nativeBuildInputs or [ ]
++ pkgs.nixComponents2.nix-util.nativeBuildInputs
++ pkgs.nixComponents2.nix-store.nativeBuildInputs
++ pkgs.nixComponents2.nix-fetchers.nativeBuildInputs
++ pkgs.nixComponents2.nix-expr.nativeBuildInputs
++ lib.optionals havePerl pkgs.nixComponents2.nix-perl-bindings.nativeBuildInputs
++ lib.optionals buildCanExecuteHost pkgs.nixComponents2.nix-manual.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-internal-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-external-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-functional-tests.externalNativeBuildInputs
++ lib.optional (
!buildCanExecuteHost
# Hack around https://github.com/nixos/nixpkgs/commit/bf7ad8cfbfa102a90463433e2c5027573b462479
&& !(stdenv.hostPlatform.isWindows && stdenv.buildPlatform.isDarwin)
&& stdenv.hostPlatform.emulatorAvailable pkgs.buildPackages
&& lib.meta.availableOn stdenv.buildPlatform (stdenv.hostPlatform.emulator pkgs.buildPackages)
) pkgs.buildPackages.mesonEmulatorHook
++ [
pkgs.buildPackages.cmake
pkgs.buildPackages.gnused
pkgs.buildPackages.shellcheck
pkgs.buildPackages.changelog-d
modular.pre-commit.settings.package
(pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript)
pkgs.buildPackages.nixfmt-rfc-style
pkgs.buildPackages.shellcheck
pkgs.buildPackages.gdb
]
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
)
++ lib.optional stdenv.hostPlatform.isLinux pkgs.buildPackages.mold-wrapped;
let
inputs =
attrs.nativeBuildInputs or [ ]
++ pkgs.nixComponents2.nix-util.nativeBuildInputs
++ pkgs.nixComponents2.nix-store.nativeBuildInputs
++ pkgs.nixComponents2.nix-fetchers.nativeBuildInputs
++ pkgs.nixComponents2.nix-expr.nativeBuildInputs
++ lib.optionals havePerl pkgs.nixComponents2.nix-perl-bindings.nativeBuildInputs
++ lib.optionals buildCanExecuteHost pkgs.nixComponents2.nix-manual.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-internal-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-external-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-functional-tests.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-json-schema-checks.externalNativeBuildInputs
++ lib.optional (
!buildCanExecuteHost
# Hack around https://github.com/nixos/nixpkgs/commit/bf7ad8cfbfa102a90463433e2c5027573b462479
&& !(stdenv.hostPlatform.isWindows && stdenv.buildPlatform.isDarwin)
&& stdenv.hostPlatform.emulatorAvailable pkgs.buildPackages
&& lib.meta.availableOn stdenv.buildPlatform (stdenv.hostPlatform.emulator pkgs.buildPackages)
) pkgs.buildPackages.mesonEmulatorHook
++ [
pkgs.buildPackages.cmake
pkgs.buildPackages.gnused
pkgs.buildPackages.changelog-d
modular.pre-commit.settings.package
(pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript)
pkgs.buildPackages.nixfmt-rfc-style
pkgs.buildPackages.shellcheck
pkgs.buildPackages.include-what-you-use
pkgs.buildPackages.gdb
]
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
)
++ lib.optional stdenv.hostPlatform.isLinux pkgs.buildPackages.mold-wrapped;
in
# FIXME: separateDebugInfo = false doesn't actually prevent -Wa,--compress-debug-sections
# from making its way into NIX_CFLAGS_COMPILE.
lib.filter (p: !lib.hasInfix "separate-debug-info" p) inputs;
buildInputs = [
pkgs.gbenchmark
+16 -3
View File
@@ -62,6 +62,7 @@ let
"nix-cmd"
"nix-cli"
"nix-functional-tests"
"nix-json-schema-checks"
]
++ lib.optionals enableBindings [
"nix-perl-bindings"
@@ -73,7 +74,7 @@ let
]
);
in
{
rec {
/**
An internal check to make sure our package listing is complete.
*/
@@ -145,13 +146,25 @@ in
)
);
buildNoGc =
# Builds with sanitizers already have GC disabled, so this buildNoGc can just
# point to buildWithSanitizers in order to reduce the load on hydra.
buildNoGc = buildWithSanitizers;
buildWithSanitizers =
let
components = forAllSystems (
system:
nixpkgsFor.${system}.native.nixComponents2.overrideScope (
let
pkgs = nixpkgsFor.${system}.native;
in
pkgs.nixComponents2.overrideScope (
self: super: {
# Boost coroutines fail with ASAN on darwin.
withASan = !pkgs.stdenv.buildPlatform.isDarwin;
withUBSan = true;
nix-expr = super.nix-expr.override { enableGC = false; };
# Unclear how to make Perl bindings work with a dynamically linked ASAN.
nix-perl-bindings = null;
}
)
);
@@ -1,282 +0,0 @@
commit 9bacade4a3ef4b6b26e2c02f549eef0e9eb9eaa2
Author: Robert Hensing <robert@roberthensing.nl>
Date: Sun Aug 18 20:20:36 2024 +0200
Add unoptimized git_mempack_write_thin_pack
diff --git a/include/git2/sys/mempack.h b/include/git2/sys/mempack.h
index 17da590a3..3688bdd50 100644
--- a/include/git2/sys/mempack.h
+++ b/include/git2/sys/mempack.h
@@ -44,6 +44,29 @@ GIT_BEGIN_DECL
*/
GIT_EXTERN(int) git_mempack_new(git_odb_backend **out);
+/**
+ * Write a thin packfile with the objects in the memory store.
+ *
+ * A thin packfile is a packfile that does not contain its transitive closure of
+ * references. This is useful for efficiently distributing additions to a
+ * repository over the network, but also finds use in the efficient bulk
+ * addition of objects to a repository, locally.
+ *
+ * This operation performs the (shallow) insert operations into the
+ * `git_packbuilder`, but does not write the packfile to disk;
+ * see `git_packbuilder_write_buf`.
+ *
+ * It also does not reset the memory store; see `git_mempack_reset`.
+ *
+ * @note This function may or may not write trees and blobs that are not
+ * referenced by commits. Currently everything is written, but this
+ * behavior may change in the future as the packer is optimized.
+ *
+ * @param backend The mempack backend
+ * @param pb The packbuilder to use to write the packfile
+ */
+GIT_EXTERN(int) git_mempack_write_thin_pack(git_odb_backend *backend, git_packbuilder *pb);
+
/**
* Dump all the queued in-memory writes to a packfile.
*
diff --git a/src/libgit2/odb_mempack.c b/src/libgit2/odb_mempack.c
index 6f27f45f8..0b61e2b66 100644
--- a/src/libgit2/odb_mempack.c
+++ b/src/libgit2/odb_mempack.c
@@ -132,6 +132,35 @@ cleanup:
return err;
}
+int git_mempack_write_thin_pack(git_odb_backend *backend, git_packbuilder *pb)
+{
+ struct memory_packer_db *db = (struct memory_packer_db *)backend;
+ const git_oid *oid;
+ size_t iter = 0;
+ int err = -1;
+
+ /* TODO: Implement the recency heuristics.
+ For this it probably makes sense to only write what's referenced
+ through commits, an option I've carved out for you in the docs.
+ wrt heuristics: ask your favorite LLM to translate https://git-scm.com/docs/pack-heuristics/en
+ to actual normal reference documentation. */
+ while (true) {
+ err = git_oidmap_iterate(NULL, db->objects, &iter, &oid);
+ if (err == GIT_ITEROVER) {
+ err = 0;
+ break;
+ }
+ if (err != 0)
+ return err;
+
+ err = git_packbuilder_insert(pb, oid, NULL);
+ if (err != 0)
+ return err;
+ }
+
+ return 0;
+}
+
int git_mempack_dump(
git_buf *pack,
git_repository *repo,
diff --git a/tests/libgit2/mempack/thinpack.c b/tests/libgit2/mempack/thinpack.c
new file mode 100644
index 000000000..604a4dda2
--- /dev/null
+++ b/tests/libgit2/mempack/thinpack.c
@@ -0,0 +1,196 @@
+#include "clar_libgit2.h"
+#include "git2/indexer.h"
+#include "git2/odb_backend.h"
+#include "git2/tree.h"
+#include "git2/types.h"
+#include "git2/sys/mempack.h"
+#include "git2/sys/odb_backend.h"
+#include "util.h"
+
+static git_repository *_repo;
+static git_odb_backend * _mempack_backend;
+
+void test_mempack_thinpack__initialize(void)
+{
+ git_odb *odb;
+
+ _repo = cl_git_sandbox_init_new("mempack_thinpack_repo");
+
+ cl_git_pass(git_mempack_new(&_mempack_backend));
+ cl_git_pass(git_repository_odb(&odb, _repo));
+ cl_git_pass(git_odb_add_backend(odb, _mempack_backend, 999));
+ git_odb_free(odb);
+}
+
+void _mempack_thinpack__cleanup(void)
+{
+ cl_git_sandbox_cleanup();
+}
+
+/*
+ Generating a packfile for an unchanged repo works and produces an empty packfile.
+ Even if we allow this scenario to be detected, it shouldn't misbehave if the
+ application is unaware of it.
+*/
+void test_mempack_thinpack__empty(void)
+{
+ git_packbuilder *pb;
+ int version;
+ int n;
+ git_buf buf = GIT_BUF_INIT;
+
+ git_packbuilder_new(&pb, _repo);
+
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_assert_in_range(12, buf.size, 1024 /* empty packfile is >0 bytes, but certainly not that big */);
+ cl_assert(buf.ptr[0] == 'P');
+ cl_assert(buf.ptr[1] == 'A');
+ cl_assert(buf.ptr[2] == 'C');
+ cl_assert(buf.ptr[3] == 'K');
+ version = (buf.ptr[4] << 24) | (buf.ptr[5] << 16) | (buf.ptr[6] << 8) | buf.ptr[7];
+ /* Subject to change. https://git-scm.com/docs/pack-format: Git currently accepts version number 2 or 3 but generates version 2 only.*/
+ cl_assert_equal_i(2, version);
+ n = (buf.ptr[8] << 24) | (buf.ptr[9] << 16) | (buf.ptr[10] << 8) | buf.ptr[11];
+ cl_assert_equal_i(0, n);
+ git_buf_dispose(&buf);
+
+ git_packbuilder_free(pb);
+}
+
+#define LIT_LEN(x) x, sizeof(x) - 1
+
+/*
+ Check that git_mempack_write_thin_pack produces a thin packfile.
+*/
+void test_mempack_thinpack__thin(void)
+{
+ /* Outline:
+ - Create tree 1
+ - Flush to packfile A
+ - Create tree 2
+ - Flush to packfile B
+
+ Tree 2 has a new blob and a reference to a blob from tree 1.
+
+ Expectation:
+ - Packfile B is thin and does not contain the objects from packfile A
+ */
+
+
+ git_oid oid_blob_1;
+ git_oid oid_blob_2;
+ git_oid oid_blob_3;
+ git_oid oid_tree_1;
+ git_oid oid_tree_2;
+ git_treebuilder *tb;
+
+ git_packbuilder *pb;
+ git_buf buf = GIT_BUF_INIT;
+ git_indexer *indexer;
+ git_indexer_progress stats;
+ char pack_dir_path[1024];
+
+ char sbuf[1024];
+ const char * repo_path;
+ const char * pack_name_1;
+ const char * pack_name_2;
+ git_str pack_path_1 = GIT_STR_INIT;
+ git_str pack_path_2 = GIT_STR_INIT;
+ git_odb_backend * pack_odb_backend_1;
+ git_odb_backend * pack_odb_backend_2;
+
+
+ cl_assert_in_range(0, snprintf(pack_dir_path, sizeof(pack_dir_path), "%s/objects/pack", git_repository_path(_repo)), sizeof(pack_dir_path));
+
+ /* Create tree 1 */
+
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_1, _repo, LIT_LEN("thinpack blob 1")));
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_2, _repo, LIT_LEN("thinpack blob 2")));
+
+
+ cl_git_pass(git_treebuilder_new(&tb, _repo, NULL));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob1", &oid_blob_1, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob2", &oid_blob_2, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_write(&oid_tree_1, tb));
+
+ /* Flush */
+
+ cl_git_pass(git_packbuilder_new(&pb, _repo));
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_git_pass(git_indexer_new(&indexer, pack_dir_path, 0, NULL, NULL));
+ cl_git_pass(git_indexer_append(indexer, buf.ptr, buf.size, &stats));
+ cl_git_pass(git_indexer_commit(indexer, &stats));
+ pack_name_1 = strdup(git_indexer_name(indexer));
+ cl_assert(pack_name_1);
+ git_buf_dispose(&buf);
+ git_mempack_reset(_mempack_backend);
+ git_indexer_free(indexer);
+ git_packbuilder_free(pb);
+
+ /* Create tree 2 */
+
+ cl_git_pass(git_treebuilder_clear(tb));
+ /* blob 1 won't be used, but we add it anyway to test that just "declaring" an object doesn't
+ necessarily cause its inclusion in the next thin packfile. It must only be included if new. */
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_1, _repo, LIT_LEN("thinpack blob 1")));
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_3, _repo, LIT_LEN("thinpack blob 3")));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob1", &oid_blob_1, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob3", &oid_blob_3, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_write(&oid_tree_2, tb));
+
+ /* Flush */
+
+ cl_git_pass(git_packbuilder_new(&pb, _repo));
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_git_pass(git_indexer_new(&indexer, pack_dir_path, 0, NULL, NULL));
+ cl_git_pass(git_indexer_append(indexer, buf.ptr, buf.size, &stats));
+ cl_git_pass(git_indexer_commit(indexer, &stats));
+ pack_name_2 = strdup(git_indexer_name(indexer));
+ cl_assert(pack_name_2);
+ git_buf_dispose(&buf);
+ git_mempack_reset(_mempack_backend);
+ git_indexer_free(indexer);
+ git_packbuilder_free(pb);
+ git_treebuilder_free(tb);
+
+ /* Assertions */
+
+ assert(pack_name_1);
+ assert(pack_name_2);
+
+ repo_path = git_repository_path(_repo);
+
+ snprintf(sbuf, sizeof(sbuf), "objects/pack/pack-%s.pack", pack_name_1);
+ git_str_joinpath(&pack_path_1, repo_path, sbuf);
+ snprintf(sbuf, sizeof(sbuf), "objects/pack/pack-%s.pack", pack_name_2);
+ git_str_joinpath(&pack_path_2, repo_path, sbuf);
+
+ /* If they're the same, something definitely went wrong. */
+ cl_assert(strcmp(pack_name_1, pack_name_2) != 0);
+
+ cl_git_pass(git_odb_backend_one_pack(&pack_odb_backend_1, pack_path_1.ptr));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_1));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_2));
+ cl_assert(!pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_3));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_tree_1));
+ cl_assert(!pack_odb_backend_1->exists(pack_odb_backend_1, &oid_tree_2));
+
+ cl_git_pass(git_odb_backend_one_pack(&pack_odb_backend_2, pack_path_2.ptr));
+ /* blob 1 is already in the packfile 1, so packfile 2 must not include it, in order to be _thin_. */
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_1));
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_2));
+ cl_assert(pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_3));
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_tree_1));
+ cl_assert(pack_odb_backend_2->exists(pack_odb_backend_2, &oid_tree_2));
+
+ pack_odb_backend_1->free(pack_odb_backend_1);
+ pack_odb_backend_2->free(pack_odb_backend_2);
+ free((void *)pack_name_1);
+ free((void *)pack_name_2);
+ git_str_dispose(&pack_path_1);
+ git_str_dispose(&pack_path_2);
+
+}
@@ -1,930 +0,0 @@
commit e9823c5da4fa977c46bcb97167fbdd0d70adb5ff
Author: Robert Hensing <robert@roberthensing.nl>
Date: Mon Aug 26 20:07:04 2024 +0200
Make packbuilder interruptible using progress callback
Forward errors from packbuilder->progress_cb
This allows the callback to terminate long-running operations when
the application is interrupted.
diff --git a/include/git2/pack.h b/include/git2/pack.h
index 0f6bd2ab9..bee72a6c0 100644
--- a/include/git2/pack.h
+++ b/include/git2/pack.h
@@ -247,6 +247,9 @@ typedef int GIT_CALLBACK(git_packbuilder_progress)(
* @param progress_cb Function to call with progress information during
* pack building. Be aware that this is called inline with pack building
* operations, so performance may be affected.
+ * When progress_cb returns an error, the pack building process will be
+ * aborted and the error will be returned from the invoked function.
+ * `pb` must then be freed.
* @param progress_cb_payload Payload for progress callback.
* @return 0 or an error code
*/
diff --git a/src/libgit2/pack-objects.c b/src/libgit2/pack-objects.c
index b2d80cba9..7c331c2d5 100644
--- a/src/libgit2/pack-objects.c
+++ b/src/libgit2/pack-objects.c
@@ -932,6 +932,9 @@ static int report_delta_progress(
{
int ret;
+ if (pb->failure)
+ return pb->failure;
+
if (pb->progress_cb) {
uint64_t current_time = git_time_monotonic();
uint64_t elapsed = current_time - pb->last_progress_report_time;
@@ -943,8 +946,10 @@ static int report_delta_progress(
GIT_PACKBUILDER_DELTAFICATION,
count, pb->nr_objects, pb->progress_cb_payload);
- if (ret)
+ if (ret) {
+ pb->failure = ret;
return git_error_set_after_callback(ret);
+ }
}
}
@@ -976,7 +981,10 @@ static int find_deltas(git_packbuilder *pb, git_pobject **list,
}
pb->nr_deltified += 1;
- report_delta_progress(pb, pb->nr_deltified, false);
+ if ((error = report_delta_progress(pb, pb->nr_deltified, false)) < 0) {
+ GIT_ASSERT(git_packbuilder__progress_unlock(pb) == 0);
+ goto on_error;
+ }
po = *list++;
(*list_size)--;
@@ -1124,6 +1132,10 @@ struct thread_params {
size_t depth;
size_t working;
size_t data_ready;
+
+ /* A pb->progress_cb can stop the packing process by returning an error.
+ When that happens, all threads observe the error and stop voluntarily. */
+ bool stopped;
};
static void *threaded_find_deltas(void *arg)
@@ -1133,7 +1145,12 @@ static void *threaded_find_deltas(void *arg)
while (me->remaining) {
if (find_deltas(me->pb, me->list, &me->remaining,
me->window, me->depth) < 0) {
- ; /* TODO */
+ me->stopped = true;
+ GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_lock(me->pb) == 0, NULL);
+ me->working = false;
+ git_cond_signal(&me->pb->progress_cond);
+ GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_unlock(me->pb) == 0, NULL);
+ return NULL;
}
GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_lock(me->pb) == 0, NULL);
@@ -1175,8 +1192,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
pb->nr_threads = git__online_cpus();
if (pb->nr_threads <= 1) {
- find_deltas(pb, list, &list_size, window, depth);
- return 0;
+ return find_deltas(pb, list, &list_size, window, depth);
}
p = git__mallocarray(pb->nr_threads, sizeof(*p));
@@ -1195,6 +1211,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
p[i].depth = depth;
p[i].working = 1;
p[i].data_ready = 0;
+ p[i].stopped = 0;
/* try to split chunks on "path" boundaries */
while (sub_size && sub_size < list_size &&
@@ -1262,7 +1279,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
(!victim || victim->remaining < p[i].remaining))
victim = &p[i];
- if (victim) {
+ if (victim && !target->stopped) {
sub_size = victim->remaining / 2;
list = victim->list + victim->list_size - sub_size;
while (sub_size && list[0]->hash &&
@@ -1286,7 +1303,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
}
target->list_size = sub_size;
target->remaining = sub_size;
- target->working = 1;
+ target->working = 1; /* even when target->stopped, so that we don't process this thread again */
GIT_ASSERT(git_packbuilder__progress_unlock(pb) == 0);
if (git_mutex_lock(&target->mutex)) {
@@ -1299,7 +1316,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
git_cond_signal(&target->cond);
git_mutex_unlock(&target->mutex);
- if (!sub_size) {
+ if (target->stopped || !sub_size) {
git_thread_join(&target->thread, NULL);
git_cond_free(&target->cond);
git_mutex_free(&target->mutex);
@@ -1308,7 +1325,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
}
git__free(p);
- return 0;
+ return pb->failure;
}
#else
@@ -1319,6 +1336,7 @@ int git_packbuilder__prepare(git_packbuilder *pb)
{
git_pobject **delta_list;
size_t i, n = 0;
+ int error;
if (pb->nr_objects == 0 || pb->done)
return 0; /* nothing to do */
@@ -1327,8 +1345,10 @@ int git_packbuilder__prepare(git_packbuilder *pb)
* Although we do not report progress during deltafication, we
* at least report that we are in the deltafication stage
*/
- if (pb->progress_cb)
- pb->progress_cb(GIT_PACKBUILDER_DELTAFICATION, 0, pb->nr_objects, pb->progress_cb_payload);
+ if (pb->progress_cb) {
+ if ((error = pb->progress_cb(GIT_PACKBUILDER_DELTAFICATION, 0, pb->nr_objects, pb->progress_cb_payload)) < 0)
+ return git_error_set_after_callback(error);
+ }
delta_list = git__mallocarray(pb->nr_objects, sizeof(*delta_list));
GIT_ERROR_CHECK_ALLOC(delta_list);
@@ -1345,31 +1365,33 @@ int git_packbuilder__prepare(git_packbuilder *pb)
if (n > 1) {
git__tsort((void **)delta_list, n, type_size_sort);
- if (ll_find_deltas(pb, delta_list, n,
+ if ((error = ll_find_deltas(pb, delta_list, n,
GIT_PACK_WINDOW + 1,
- GIT_PACK_DEPTH) < 0) {
+ GIT_PACK_DEPTH)) < 0) {
git__free(delta_list);
- return -1;
+ return error;
}
}
- report_delta_progress(pb, pb->nr_objects, true);
+ error = report_delta_progress(pb, pb->nr_objects, true);
pb->done = true;
git__free(delta_list);
- return 0;
+ return error;
}
-#define PREPARE_PACK if (git_packbuilder__prepare(pb) < 0) { return -1; }
+#define PREPARE_PACK error = git_packbuilder__prepare(pb); if (error < 0) { return error; }
int git_packbuilder_foreach(git_packbuilder *pb, int (*cb)(void *buf, size_t size, void *payload), void *payload)
{
+ int error;
PREPARE_PACK;
return write_pack(pb, cb, payload);
}
int git_packbuilder__write_buf(git_str *buf, git_packbuilder *pb)
{
+ int error;
PREPARE_PACK;
return write_pack(pb, &write_pack_buf, buf);
diff --git a/src/libgit2/pack-objects.h b/src/libgit2/pack-objects.h
index bbc8b9430..380a28ebe 100644
--- a/src/libgit2/pack-objects.h
+++ b/src/libgit2/pack-objects.h
@@ -100,6 +100,10 @@ struct git_packbuilder {
uint64_t last_progress_report_time;
bool done;
+
+ /* A non-zero error code in failure causes all threads to shut themselves
+ down. Some functions will return this error code. */
+ volatile int failure;
};
int git_packbuilder__write_buf(git_str *buf, git_packbuilder *pb);
diff --git a/tests/libgit2/pack/cancel.c b/tests/libgit2/pack/cancel.c
new file mode 100644
index 000000000..a0aa9716a
--- /dev/null
+++ b/tests/libgit2/pack/cancel.c
@@ -0,0 +1,240 @@
+#include "clar_libgit2.h"
+#include "futils.h"
+#include "pack.h"
+#include "hash.h"
+#include "iterator.h"
+#include "vector.h"
+#include "posix.h"
+#include "hash.h"
+#include "pack-objects.h"
+
+static git_repository *_repo;
+static git_revwalk *_revwalker;
+static git_packbuilder *_packbuilder;
+static git_indexer *_indexer;
+static git_vector _commits;
+static int _commits_is_initialized;
+static git_indexer_progress _stats;
+
+extern bool git_disable_pack_keep_file_checks;
+
+static void pack_packbuilder_init(const char *sandbox) {
+ _repo = cl_git_sandbox_init(sandbox);
+ /* cl_git_pass(p_chdir(sandbox)); */
+ cl_git_pass(git_revwalk_new(&_revwalker, _repo));
+ cl_git_pass(git_packbuilder_new(&_packbuilder, _repo));
+ cl_git_pass(git_vector_init(&_commits, 0, NULL));
+ _commits_is_initialized = 1;
+ memset(&_stats, 0, sizeof(_stats));
+ p_fsync__cnt = 0;
+}
+
+void test_pack_cancel__initialize(void)
+{
+ pack_packbuilder_init("small.git");
+}
+
+void test_pack_cancel__cleanup(void)
+{
+ git_oid *o;
+ unsigned int i;
+
+ cl_git_pass(git_libgit2_opts(GIT_OPT_ENABLE_FSYNC_GITDIR, 0));
+ cl_git_pass(git_libgit2_opts(GIT_OPT_DISABLE_PACK_KEEP_FILE_CHECKS, false));
+
+ if (_commits_is_initialized) {
+ _commits_is_initialized = 0;
+ git_vector_foreach(&_commits, i, o) {
+ git__free(o);
+ }
+ git_vector_free(&_commits);
+ }
+
+ git_packbuilder_free(_packbuilder);
+ _packbuilder = NULL;
+
+ git_revwalk_free(_revwalker);
+ _revwalker = NULL;
+
+ git_indexer_free(_indexer);
+ _indexer = NULL;
+
+ /* cl_git_pass(p_chdir("..")); */
+ cl_git_sandbox_cleanup();
+ _repo = NULL;
+}
+
+static int seed_packbuilder(void)
+{
+ int error;
+ git_oid oid, *o;
+ unsigned int i;
+
+ git_revwalk_sorting(_revwalker, GIT_SORT_TIME);
+ cl_git_pass(git_revwalk_push_ref(_revwalker, "HEAD"));
+
+ while (git_revwalk_next(&oid, _revwalker) == 0) {
+ o = git__malloc(sizeof(git_oid));
+ cl_assert(o != NULL);
+ git_oid_cpy(o, &oid);
+ cl_git_pass(git_vector_insert(&_commits, o));
+ }
+
+ git_vector_foreach(&_commits, i, o) {
+ if((error = git_packbuilder_insert(_packbuilder, o, NULL)) < 0)
+ return error;
+ }
+
+ git_vector_foreach(&_commits, i, o) {
+ git_object *obj;
+ cl_git_pass(git_object_lookup(&obj, _repo, o, GIT_OBJECT_COMMIT));
+ error = git_packbuilder_insert_tree(_packbuilder,
+ git_commit_tree_id((git_commit *)obj));
+ git_object_free(obj);
+ if (error < 0)
+ return error;
+ }
+
+ return 0;
+}
+
+static int fail_stage;
+
+static int packbuilder_cancel_after_n_calls_cb(int stage, uint32_t current, uint32_t total, void *payload)
+{
+
+ /* Force the callback to run again on the next opportunity regardless
+ of how fast we're running. */
+ _packbuilder->last_progress_report_time = 0;
+
+ if (stage == fail_stage) {
+ int *calls = (int *)payload;
+ int n = *calls;
+ /* Always decrement, including past zero. This way the error is only
+ triggered once, making sure it is picked up immediately. */
+ --*calls;
+ if (n == 0)
+ return GIT_EUSER;
+ }
+
+ return 0;
+}
+
+static void test_cancel(int n)
+{
+
+ int calls_remaining = n;
+ int err;
+ git_buf buf = GIT_BUF_INIT;
+
+ /* Switch to a small repository, so that `packbuilder_cancel_after_n_calls_cb`
+ can hack the time to call the callback on every opportunity. */
+
+ cl_git_pass(git_packbuilder_set_callbacks(_packbuilder, &packbuilder_cancel_after_n_calls_cb, &calls_remaining));
+ err = seed_packbuilder();
+ if (!err)
+ err = git_packbuilder_write_buf(&buf, _packbuilder);
+
+ cl_assert_equal_i(GIT_EUSER, err);
+}
+void test_pack_cancel__cancel_after_add_0(void)
+{
+ fail_stage = GIT_PACKBUILDER_ADDING_OBJECTS;
+ test_cancel(0);
+}
+
+void test_pack_cancel__cancel_after_add_1(void)
+{
+ cl_skip();
+ fail_stage = GIT_PACKBUILDER_ADDING_OBJECTS;
+ test_cancel(1);
+}
+
+void test_pack_cancel__cancel_after_delta_0(void)
+{
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(0);
+}
+
+void test_pack_cancel__cancel_after_delta_1(void)
+{
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(1);
+}
+
+void test_pack_cancel__cancel_after_delta_0_threaded(void)
+{
+#ifdef GIT_THREADS
+ git_packbuilder_set_threads(_packbuilder, 8);
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(0);
+#else
+ cl_skip();
+#endif
+}
+
+void test_pack_cancel__cancel_after_delta_1_threaded(void)
+{
+#ifdef GIT_THREADS
+ git_packbuilder_set_threads(_packbuilder, 8);
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(1);
+#else
+ cl_skip();
+#endif
+}
+
+static int foreach_cb(void *buf, size_t len, void *payload)
+{
+ git_indexer *idx = (git_indexer *) payload;
+ cl_git_pass(git_indexer_append(idx, buf, len, &_stats));
+ return 0;
+}
+
+void test_pack_cancel__foreach(void)
+{
+ git_indexer *idx;
+
+ seed_packbuilder();
+
+#ifdef GIT_EXPERIMENTAL_SHA256
+ cl_git_pass(git_indexer_new(&idx, ".", GIT_OID_SHA1, NULL));
+#else
+ cl_git_pass(git_indexer_new(&idx, ".", 0, NULL, NULL));
+#endif
+
+ cl_git_pass(git_packbuilder_foreach(_packbuilder, foreach_cb, idx));
+ cl_git_pass(git_indexer_commit(idx, &_stats));
+ git_indexer_free(idx);
+}
+
+static int foreach_cancel_cb(void *buf, size_t len, void *payload)
+{
+ git_indexer *idx = (git_indexer *)payload;
+ cl_git_pass(git_indexer_append(idx, buf, len, &_stats));
+ return (_stats.total_objects > 2) ? -1111 : 0;
+}
+
+void test_pack_cancel__foreach_with_cancel(void)
+{
+ git_indexer *idx;
+
+ seed_packbuilder();
+
+#ifdef GIT_EXPERIMENTAL_SHA256
+ cl_git_pass(git_indexer_new(&idx, ".", GIT_OID_SHA1, NULL));
+#else
+ cl_git_pass(git_indexer_new(&idx, ".", 0, NULL, NULL));
+#endif
+
+ cl_git_fail_with(
+ git_packbuilder_foreach(_packbuilder, foreach_cancel_cb, idx), -1111);
+ git_indexer_free(idx);
+}
+
+void test_pack_cancel__keep_file_check(void)
+{
+ assert(!git_disable_pack_keep_file_checks);
+ cl_git_pass(git_libgit2_opts(GIT_OPT_DISABLE_PACK_KEEP_FILE_CHECKS, true));
+ assert(git_disable_pack_keep_file_checks);
+}
diff --git a/tests/resources/small.git/HEAD b/tests/resources/small.git/HEAD
new file mode 100644
index 0000000000000000000000000000000000000000..cb089cd89a7d7686d284d8761201649346b5aa1c
GIT binary patch
literal 23
ecmXR)O|w!cN=+-)&qz&7Db~+TEG|hc;sO9;xClW2
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/config b/tests/resources/small.git/config
new file mode 100644
index 0000000000000000000000000000000000000000..07d359d07cf1ed0c0074fdad71ffff5942f0adfa
GIT binary patch
literal 66
zcmaz}&M!)h<>D+#Eyyp<EXgmbOv^9IO)M!(Eh^5;&r`5fFyP`$%gjm5%}+@M@=A(I
MQ@J>k5{uv*03B5png9R*
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/description b/tests/resources/small.git/description
new file mode 100644
index 0000000000000000000000000000000000000000..498b267a8c7812490d6479839c5577eaaec79d62
GIT binary patch
literal 73
zcmWH|%S+5nO;IRHEyyp<EXgmbv{pz>$t+PQ$;d2LNXyJgRZve!Elw`VEGWs$&r??@
Q$yWgB0LrH#Y0~2Y0PnOK(EtDd
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/applypatch-msg.sample b/tests/resources/small.git/hooks/applypatch-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..dcbf8167fa503f96ff6a39c68409007eadc9b1f3
GIT binary patch
literal 535
zcmY+AX;Q;542A#a6e8^~FyI8r&I~hf2QJ{GO6(?HuvEG*+#R{4EI%zhfA8r{j%sh$
zHE~E-UtQd8{bq4@*S%jq3@bmxwQDXGv#o!N`o3AHMw3xD)hy0#>&E&zzl%vRffo<B
z)-H|+CWHZ~O*S%cfYx9;02_ohIA<Bg(1SxF-6OCb&_lBkf{t<AM9r;%E(Hf#h{|a@
z9>mqo=v6>_2NRa#TwDdYvTVQyueO*15Nlo%=#DXgC0bhF3vTa`LQGaO9;jeD$OP?~
za$G4Q{z+Q_{5V?5h;a-noM$P{<>Q~j4o7u%#P6^o^16{y*jU=-K8GYD_dUtdj4FSx
zSC0C!DvAnv%S!4d<Yg@O<;m`;oSw)=Fz+hrL<mY{rBr8j4pi^88FX3}jKrYUP)>gk
XB^)11aoGMJPCqWs%IS0YSv(eBT&%T6
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/commit-msg.sample b/tests/resources/small.git/hooks/commit-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..f3780f92349638ebe32f6baf24c7c3027675d7c9
GIT binary patch
literal 953
zcmaJ<TW`}a6n<`g#Ya}rZCaYOzy?UGG&Tf#VG`?}7@eHtB=MTqneFUS%75oLS;atz
zm&iUo=ey->y@-{3h^^Cx;#d0zEA@DDc$nY4ez&|=%jTg@_HU*ub=!!y$xW09TSjlj
z(`I@QCsM`!9&80$I98wsQ8yK#)Orb<8re8FjkKh630D$QUDwi~(gkX=RunYm$rDjk
zlp%RUSnzA#6yjdG5?T?2DcYKp+v_lts0ljn&bh3J0bD5@N@1UKZ190O6ZeWr-BuZ^
zWRebCX%(%=Xoj#(xYk1Cjtr!=tyBesf@m6}8zY6Ijbz9i9ziI_jG9Mv<Cz(ymp*>R
zDH*e>^ga9IR?2wrSrAVm;eButj4<aWB@zzNl|1Wp@4;}1O?MUF>Y>7(E2?b~jsu>&
zRKCJ7bp#19sqYh627wD%D9R$8=Ml$TNlumDypl~$jBu*G>5fIR^FB0h0Ex&TGZNr>
zL5hs1_K>taRb!|ThN9ns7^@4MXKP+6aGI_UK)T-M#rcP$;kN(Vcf#P)+5GzWa{l@J
z>-E{`$1iiNVYxq27}<DnwLRXQUG0o_hw&da-s5T#H=`Y9D_8=eTZ?cpWatp#a1vs@
z2BjrO)z@aTuI#g#`)oJcnhM7oYLT@~CHX@CNXv4>j;uo%;)r3kJI2xCFF~Ux;$Q%)
wjbk6JlDCM`jU&P+UVOvg`|iYl<7~9k>HHB4I;pdlQ=I-^$DrHaN$@lH1?P!0U;qFB
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/fsmonitor-watchman.sample b/tests/resources/small.git/hooks/fsmonitor-watchman.sample
new file mode 100755
index 0000000000000000000000000000000000000000..41184ebc318c159f51cd1ebe2290559805df89d8
GIT binary patch
literal 4777
zcmbtYYi}F368$Xwipg4lq(BeHMvzvH-4;n7DGJBPqq#tw3aed8+IU5-m)yvL>;Cqh
z8FFRGj$`9CA8ao<GaSz2oMCnz4Rv-gw9b@j_$0GQT1?XYi|;S??Y`Z6`Z;}C6#27N
z8ShS>J?j^$%==FV``-=rhLcPW`McSytRm~mEO7_&_cAVZrf1fFy*ha@8oe%*-aBYE
zcjzZg>LOkgxuUr-XJnHyD;zmPnRaSc#!k_P*d_BttRdc+J6G7za5#+<HG#rlmbrN~
z8DwU-3}VABEwM=0VLP@^Dy6ERR5_J6cmg|GEh*M1EliqCGwe^ZT-iZ$2Yc`4!I#WZ
z5nGGhn7*jeW=2ydsmfAmm#=8AD<<;TI+#z{Q)kW;yE!%GB6f~7EtEMLdM47Qaz*8=
zIObA(VVj-VG{Ax|66d*hi`+bRG>^Y1nkc2Oowk`ya47uUR3Feu?B<phm31&PQB<lt
zb{W(W4wf#Bab%|Q_tKPS?3^o=5)Z8^Vh(#slNI}pO(f^|{U0GZhLnycSaNd&h?CaC
z0XklU6^<ky6rES9T=na$M8P<_)aKMAMo+UDewAu4wF{#&6diFshiudixAoh|&0<BJ
zR>(w;S{(VYzxh}q-=#zP@uxSx{wbyPUMFU;K(06)$o{07&3yI?q{GqMcQ1c_^M<0<
zF4acAV)Il-V(rCTC1(;bsZ*}bl8dmejAk~yb`B}!^0;g^(o9kGUfZfDOvyp@x4OQt
zSgWh6T|3eq;9MFs8-#z+FDM1h(IjRUP|``PxupgJ7CUHOH90gbgl^2~97`?_X{P))
zB*$r1cDlF-%azKND}?Gv`2K8-9v5e`gQoft=j?T<&a13c^!wY_$D`5z-X1g?ty&6-
zQN50{8?bUk9AI->^W@~~nkOghHIC2YN+<JiT_ob7ttND1oh`HH28Y+OV~HedG&uB`
zy}rA*r_xT#bR`Q7-*)3t*(!Hf?jKzyxk=8hdi3L^d<p<uU9q_<4k&xEr4@YWv_vsW
zp(#32bYtA5u|s#1+}h`0kwpz4kUD&+>AXkLQG_2-{Pq3%{`3KUMeG$iIn%%^6*NYb
zn|_BdV#C)n4565Vcc<EWC-nglECZGy!j9I4&;hUCzr(?6TftH=0^@!mI^E@y5HZw8
ztH&kaSNyg=O6riqR^MPOX6oD__Jz@K;*icS)?m$!p{S$n;*EwlO<L!d7;utu(W9D!
zaF!B~xV^2i?wH0s?Lw%Q)(`aPkajs1XojlPv@Y-d5#YFg#KG+!r7A(dJLnkiJMs`p
zV|_=d!upN{TsxH1?sZNdzxeHsmtzTV`1{pykJ_~+^*>X;uT8&z3vSi!HXGbUj2B!R
zdz~&#<?<tHJql=W&((YpzS06y-Z6Cn^H!*9qH?pOrD~(GV=JJ~z{tpLnGK|xp&C1`
zsbn7O86xjF<~G*S1l*;;Bh%6><Up=oKy99?62P^?M&22U6QFRBXLb&u%=Ur<74wRy
zMRG!f{AvZ>fk#L-&k$fLwo$4?>12g@AXOKFekuo#6EHB%gmpD?1eyh%N8s{2wGoTu
z*@6cEZ^ZW!FAF_|JL`NkV7k}0ow|-2jHwbgH0;c@Dq*o?@&c*HnGdyx6^su8Qk%2{
z*ye(dxO*6-&>qn1+zw}tc6;=sOX{4WB=VqjTS^))y1jlX2Q;=e!qMmFA5lC$#;BxC
z=Y%tRpWxb+_uQAvAw7Q{HGV#R$xb&udLCzZ+HN?kTyB};1EJ8UlQ5!>5eGW@)RX0n
zkjj>EF!3=0Gl^8dzv$B^NMGRxJoqN4A`xq-@wCbrx*u2NmIJ1<fUDQ=*^)h6`vzco
z3F+ro$F!F6pc<B;<;isobIgbVGKUBByoQ4#CO({S7g?<Dh0^!7uJ3gxS=6F;+^gQc
zeKi4`4`Fm3p|BU2v{M|-u!#QGCXiM=k=%np0<ZOPQqEjP_nneyOdgEuG9IH&YYPtp
zKB_dvcYCcyhyT#<uhUEL$o~!TUz;cb&|`uSM{Dkv%&W2lcpYL&kv)tUvVv?&>xZ%H
zh;{|4T3(!E9sY#Ni(wUJYs1MmIc9bl)(4Nl3_wD_BWB>i<1S(LX7m*{Q7PU$muMS*
zM!%0EZx-Vw=Zey;erC?SNxF;pY@^A%-krqzfLV2meBp1vWdyArFYn`DD19T)Hw(?n
z)}{NP(Lk(o*?gl#B@pP7^*r|=;PIDT4|F#{2Hzh-AL0Rv$6uT;<CP7qxbYms@WU7}
z%}TsHJ!V_56ZFF{BfI=8jTBxMEATG376pS6a;u1@c?{~sL<N52`U6fuLkz4P@Mb^b
z2`Q48y&!C0&A+WRCJAdmo3u2#?eI=si9Vm47$|`m**x<wKkM=QR&g?C63@P5X@xP8
zi5QG2b-Fdz%S0%1{kKvL%^RTdpK|SU^VJ7JCmKdwp1`;HRoGM7ef^k_s_}2An=cql
za|{IaxQWpdq<ae&x3BOJq+M5QNIk#C{Nv@u>n|WzE4=slK?on@(fZeGhRgQCu56qB
z{+n81Az96qnQjMY*-*r-KV*7;Z#4Q<xfjbcBx_6JAN-HP@bq+eI%HhAB9&vLyOap{
bw<Ywj(b#kdcCk7dCBY;|DBNpPjIa1F6*dXN
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/post-update.sample b/tests/resources/small.git/hooks/post-update.sample
new file mode 100755
index 0000000000000000000000000000000000000000..9f95e41a39cf974958277c8881ac6cce405ebb20
GIT binary patch
literal 246
zcmXZVO?HDY3<Ti4Poaiwjq^yGw9DKf7mz^&Ly=V5q$H=W^Rt|p_r9s#9Ea7VERo!9
zyT9>uJRJJV$M^KdldiMhj?ImK6~FvwJ*L5a){QoM=L5TYHkGO1$UrO3`a>{?Opw|b
zG(#59NQ#jFL9v~vgOVkM@^^(^A}onOE))yWEwhIlk&{ZyseZ^O0b=w8&O=BK{k<5B
k^Q-B@eG}LeHrquz%(SVEp_N)VhYZikCW__82JXfD17`J9Qvd(}
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-applypatch.sample b/tests/resources/small.git/hooks/pre-applypatch.sample
new file mode 100755
index 0000000000000000000000000000000000000000..625837e25f91421b8809a097f4a3103dd387ef31
GIT binary patch
literal 481
zcmY+ATTa6;5Jms9iouO45IBJXEg&Jm9@v1LPHMM_ZR|;#6tQ<EeSrA%_2`_rGr1_8
z?aM?yVtIc%-@9SGSk&8x=grP-Lf`7!^=$7xgL=|ysZ}!av6zL~ywui}<2##V6L@!k
zy=p^)V7%Wzs-g`9<Y9}^)&uN}BCrXR_T3@Z2$gSJON2`X=mAs+%@7n-2I}ZrP|TFA
zvJJGDl3HPLP<@!Q!}zXQvey#qEE#a#$vs97i4=A0stF@YQ)k_ZajaoS^dVYBc&37_
zVI(L=X<V335r9~7T<;|HfKF+yM}}LB9d96V)Si;sj(;9Rh$#P>h$71hSXq*MxP;V&
zj0cY7SCL=x4`a46sF)C>94Gk%=3q$W2s;j6iHtB2$R0%gix4oK@&T~=ALd_o*CKxt
I-`Pv{1Bpzc>;M1&
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-commit.sample b/tests/resources/small.git/hooks/pre-commit.sample
new file mode 100755
index 0000000000000000000000000000000000000000..10b39b2e26981b8f87ea424e735ef87359066dbb
GIT binary patch
literal 1706
zcmZuxU2ohr5PY_N#pZ0-F<{-<Zkx1j&cMNSQ3FN`GzR*R1O+9oPV`BnOj7q@1pV!u
zrF0Hl^i33(yR)-1d-!H%&2|=|^E~_R{N1zNJ-&Zmt-t?iwXv&i+ZN}Km(TX8Q$H4u
zd7(m`|1iDmF5k@xV`p;C4zojASmLc}yN0QDZbhN=ri&CEt=XGuN1IwjGJ#a#`t-kG
zDqY)}7+Ft|;YKwLYbtg$S(-TBO=x3cP1cd}%f4kB!<6Wu-dCwz-)KDMEuM^_Hh*UC
zC`1)|)T<(U6b`+yOH!6p*Ll}@qastwA*dyjsgOf5C=?LTprfORG6O{5L%@S0wyHpj
zu|_A-=NWnYYR5m7kvm6|&T~GzoJ_OKR3sgFUgw?ifho^NQhvK#{6g0=&Fh)%n}#m0
zk1sNmwb_AMDq};OOGw5|;OyX#?yQMMH6yAk(x$3tjFjHE?c$E2XC_xXav8tnIeIG?
zYMI|~MLEVGkuT*>v&v-X^RA+u>k}E$4d&uD7=g_fA8+pNNV=4s0|iD3p<=DTXClTS
zXV23tJ;ECmN@M0j@zUAKEYW@3bv!SeYZ8ZH`YQNTApFVNc;F|9r5p4TqGs=>8E?6y
zi|gY{iM#PG1nL?UE9YCnWTk72kgZPG*Usqw!~Qd3c?~@w2?%eg@~)+VlSs6N5Yf2^
zz;ow<fjf3n`imj7u5lnzt||q9QFM(H@<3EJCK|l5P!$yDfn~U-(5Vu7s+GqxNKyeI
zP=-Oa(KH&gK`NhUa`cLj3B8%qL};DR7dk!`A^h&3-hFB6p($5Ufy^tGir)3et}qK4
zpkPKYWzC+?=&gw-L6S)S=<lfq)%uLUAa%~81Jf9hk)w~U!DItnoSy`m^}#38M}K-o
z!PqisQkV!&z4E*V01ro~qlVK^0BI`pLk6D+)f~*y!hCvwHq8zY9BGh<2s$@b^A<8G
zRaqk}&qZyyv&|0QDFPA%h4TgF&vdlc|JUq*=>F#K#r^&KMq1A`oqVGFpD&-!Pv|Rc
zO3KSqA@h9nSc%bm`0)Amk6*J}@14J*1-219l%%7D!Pl}UK>|lVi0Dfgu2jN3WC!uL
z0ej??b2iSehVgdnWHmZV4kUo*QL#aiIp}U=9x)IXk}JJ7VQ;CI9Rtn5e0VcjbY<bp
zv{}WlG6L;H!EzFKdY>cVt+`x5D+svCGD<sXw4|)E|JX43I1_3P(sI4{wj87bPSrNG
w!SIr>;Z5hm*<gY+X;)Ryx4=nzaab9m`bwE*^s(%u*E3HbUuOR@+&s_z1=MCi2LJ#7
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-merge-commit.sample b/tests/resources/small.git/hooks/pre-merge-commit.sample
new file mode 100755
index 0000000000000000000000000000000000000000..d8e8ad4bcf8337f23c78a98537ecab4831301202
GIT binary patch
literal 473
zcmaJ-O;f@!5WV+TJd4B0whSt$H%Dh2t`2u6q1!glCNbGU;n%yxdNsG~zR#WA6xIwy
zWEZHoU#u?nykD=Y<HPgeWDkDm^kTof*l(|%^gh!nHrZpo^vhMDjV;E1GD~K7wV*+D
zz9lry9T0cHcm_KhDVXYvQ==FrLTT4u=bEr{U1yl7%thf%wJnv<XQZ`ZbQEezaWdS%
zI;c?h9a)Y!ux<WK8rQd_aA^?61hv_Pf<t7*z1USuL40FxYz<|hybsO?qnN}aMpcuf
z6phFw1%Xx=wUk(m>E$jSEQZ%SQ(}oLgslTvrKK@9Qf#b!hajVFnp9@oIix;NcI9Wk
xjnh0ya!AWet{I7YpD;y6HXyzI*lfSvH=o6*7mJZPkuaYpm>vzZ`wyGEBtOQPo|pgt
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-push.sample b/tests/resources/small.git/hooks/pre-push.sample
new file mode 100755
index 0000000000000000000000000000000000000000..02cbd80c287f959fe33975bb66c56293e3f5b396
GIT binary patch
literal 1431
zcmaJ>U60!~5PUX&#a1@z9B{IIZkjLT0t5kq9#8~D(I5{+8&J~9;#ndUk~-ZT`r|uG
z$#K$$J{TsK<m~Lsu9iP+t-0TZ=sa(K+C6);54X>s*LP1}9!GoZ@4I4myMMG_di|of
z%?llx{O8TS-#^<H#%^V=)RNv>;(OioEmPy%kwWQBA1OMzV{hsQ8XFzS1k!~YQoLa5
zhtP1fA$q6VmMbbAC_9)4I628k*O5J$NR19uHe4QYDK<==I~SQk)Nu%xQ~<Hy8U>KH
z53w=!ke(FGb_PpnZfd*+hnXDTn;2*`u^~;?+5C~cn?bRka7NR%06%e6O91{MAgN6J
zmlO8{Biw4&wr&&(z4p3eln`E}XR9m9bNYZ7Ibrg(4yZIXrfgD7N*AFD7L3YSM#j}%
zo__rOS5fr;@8UM<6cl+cv_$YB$PQ&9dv($eM*))g!_cu!QcSh-mqE9i#QDZT)=o#`
z?8!RtE?w6p?GkGZ-6yt_p~5~4ecu|Sf^)6096%h*q-eNiEA1;Xwg)p~Q&iGSG7-IQ
z9aII&`ps$WOojFA`*bjG<mBv1n0hcYZWN0~(X01-hx(ExqWqaXgSk*@-GMp|K_3!5
z9|O21N3%~izh(4fbp9wzd+!b&7cVwSP5H00)m5ej-(s=Pl#(90UOhn@OA9u+D{i@r
za4*CP0I#<d-)-#xq5q-iY5nIef2s5OuQjcA>kFk|E@sHHuD}W^d`7YJ3YE^zrQnqR
zGoq?;YGKe)93o|_=^f%3U1KYZGPOXRRxK7w`UUbMMa3<86OmVH!EKP$8RCrn9mWX+
zC?9yF!fRVLmud3hF<}x;;sR}f(*r}6Gap3fR6zLHR~kbMgD{98N`L+r&?3p~*0+FX
zcAL%j=(SO}xTJUTvA`&Lf`2mv4koPG9&|<CA~EHbWHMoFPwT(&U=7t0`RoFZPO9Kq
zwwe$i=T|AXY#hD$aZlNMH`wZ%gwilGJ(zeYpOn*F3cy0XKXio^Sj#WXx=PWV`WGaA
B&~*R+
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-rebase.sample b/tests/resources/small.git/hooks/pre-rebase.sample
new file mode 100755
index 0000000000000000000000000000000000000000..193ee3f127387f469894675f0d621b5bb9c275da
GIT binary patch
literal 5006
zcmb7IX>;2+68$XxiXKL@ma;l5d2^5Ba_rPh_DHI-u1#&_upttZXp;no03$20|NFiM
zK#D#xQ>!Z3JkX8T-LDVm!B5j7y_{;JDmmTTef+K1oIiPzeEr+Ai*<2PUgnG4^ZB>p
z_fkAvoR1emuf~ri^K$-px=4#D-v<wZ2Xv&$O_eTJh6d4)=DWL(NBs9G{k<+yMMw0T
z$VH*-+LM)}u&m^`l8~1nt(3Z;R8v(KbY5#i3z+~8h0D}Xvq&3J8BMWDizPNpaeb~9
zBN9bSkthfXzskapf%Zt{*e#}{QaNiaAVZ4{$;;I6<vKNhO@%7P-(;l-x=pPoExHC!
zB(hA#cDdD?s4P=!)=-K{<kHAWKetl-8I8wwO<ihJNs-$dEvr;&S_@6E=mNSJ5;mg#
zyb)MbqKH<one{qrV;ZQ6WL}yLtyi*ekNLf|uC6M!)Cmq7*l?g0d6`MlE49|}>Y9w&
z`bCv#<YfTKtb`!}CyNYd;|(C?vRVQmWOfR9X?FZ#=f$No)^#4>2zVn=YnJyeNey(Y
zRh`9vtLw~A+5zsjp|W0Nsa|29Rm!B>OoG5a+vi;ari8O>KkU!KAWg_fa3btK2x*_@
z0bEc7J;Ubghm}n9bOi(Sv_B66nQ7U)J7f0fO}<cB8i8vG{r39s_>8Wuf*uorcIgEG
zOHc|-V6+HlRhOP}?Cn?@5iwSl43abmBA^2lyL$+cpabCGVES+v^j^FO_}?FIp<qP?
z#_?*YMHIkyZxJxS;h@A)WDJ0bN&+F-#_lFjAf^g_Pb#5YXqYe|dZUpa^zI)VOBXQQ
zAMhT>%En%Ll?Z*7*}TwrZyg5OSZ9rY-`aU~Mc-jjv{Ll)FLMgtB4ujktfQ`Xhqrka
zT=P!A;9w^;Z?PqpLwOLu=cj3L>TdUKw2;DMu)`oVkj}<z_EjO_2uWYuvKG==%Zu?h
zJiMVR^c30RbpW}<+z;jjoNC}YI4f6QC7d-08*4mCB1>#bcDx4tYg=j%D`+i{W~fVM
zVmZ>W9VMyin9c-0KzI_;iZ-g|OyzuG`Yq%(%dvl;ifnVr0;jWE&S`z|rQu=!yHBBO
zx`OJ;oOQ(KKM<$(bC38o>pD0%|HA(E0TRw7qj$fJ_pRN+7Nm>dS<q{AcOz#-;d7_2
zL$z(_nhH{vOzT(}>C(gLg{(`t+5Z=?o+}wXU4tHy+&%F&aRhFebeEhR2R5|<c6J);
zEY(q5F5<n*XP0|=PtPBn$B)V~d$Os-?&8UlaVe_|jdkzoWNsTP-_uyq4$R6o<h`&@
z{loXa{^#TF=NJBYu9qB?hs}x=It_O}ZjX(_wy9@X(lmkRpNi0{8T{O_b_j*JC^_SM
zz3G?1$KCNWF-|`Jbx2cQ-y5LW?Z2eikngTZmsx5C(@({8<l)Ue+gIp##Mosfa~iZN
zZ|NLN9uE6Xaqpwk+@D+f?$tg2JRCY`pwZwbR9OvEn*zYm`ffKIzl4{r{ZgjfpbuX)
z_r0=0y3)T-j$gljPyEJO*6Y<pj7G72aLoqaTpc=#u)*xJcVUm0;k(n;r)^DQNa6Oj
zWvlHEGg~lz`Q_8`yQ9<BZ;ylE1Z}bD<8}<uB9Y5lRB=;JUD0h?_)4H&EhJjvwzJx?
zr<o_vk&75j_5^d$8Z$_Oc1=R-I_DlNZ2@~81oV*J6%o3RuiCz}^VEW>$#Ycbp^w@t
zTl%=f1t=w+WpJzF<|CE@?SCNAz)%9?w33lQ8vrHJqPfH9@}qs*QXOG71W=ylx;wOB
zcx!Bj^)Yy6WX$a^vBkBJ5Cob<oubBW+a#9bX{0bkMTX_2sIrs`HMk@$q{dK5h2-g}
z({`~#gm#G?c#>qlaDx_B0c<3b+8)f84LCrt;e;qxc+7>VbwVK{skNv!wvBiTa^9Iu
zkwP;VK)jH$WJ{`MRwAA9fal!y0dtV;FWg8PTkWU>CwnqD>1ZX2B@;$DlX%C5MI+}{
z9xQVnffR*~v2KAUj*hCd<gRCjR7~6Q(vF%VT7j97iv3j4Z0p%mU`7t061~l7b$!#t
z3*Pveii}aQ?QD9aiX>gul~`bk#mk`o>zk9)<2Uc8?hUZAEvd!`9em)~$Z)zev>w^8
zyAgCP_$&Y)7HSQ84`xG}OeTavaEswwF|8Xpi5iZzZa@hCiv(J-%bfFC&)HLlO+Rhw
zG6g?9eL5&A!SuJnQ6}LxG%tU+@vZ`i+!+Rz6iYvsTdhnPo7lW{m-}{hya@viX4)XZ
zngaw+j;gloB#|UwI@8sOmQpc`h+bicQJnQIB5eifIMQNgD2+oai33m!34~xU|0Azj
zhu$8z+T5^;Pxx@d{N)pzOJLSa^e;aDf$W%N5XcOf!mGC9l9j$Ev2h6N+6ZQC+CJzl
zaM7?S!SrFLS2DASjj(h6y1WN3N?|bmqmyzm!&nLoE|`rKBOc_yDF$a#FsUn!IQf(t
zdC&Us(kQz*7mv<H12p@UD8XaLXdK{>H^j*^MC@>wTDb}g%~sx*ng#>{@lR=XG-Z5_
z#<9*Oh0joMzt;nS)ObAp)347`D=}r-;nV!TbIq&xrGRGsF6fZg+!VkfUei@_&l-M&
zPqQ+Dw)RV}+)I8RuqAxa`Pv8e&!_gXS=e2-un>=Ktn}-;%lLZxaVn?Q>yZCb2R3Wk
z77zr%;Rq&h|2ncqyKYmFI0148JVY7Q$V5p=dWj<MQ<8r+@QLq+UROk&%quICq^O2C
zp(17882jXI)_GaqzAY4AjoB_nh8k*r1mJ>+Qqpu%i|xp2<qF`Tw6&3`h654ceBjZ7
z4>C=WaOb2Wudn^h0EcD%$p9YVU1fnoRV9`(cy(vv6K>FXS!2jY>1GnU--7)4usH&K
zao*&P^@9~YmUe|ZdLW@C>H;!*<TIU}-!Tw#3oqZA*99}b3&uHiGO<{I6!pMnAiQdS
P!fA@Yk4s_DjDP<F98V`a
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-receive.sample b/tests/resources/small.git/hooks/pre-receive.sample
new file mode 100755
index 0000000000000000000000000000000000000000..32b597f8c32399017c3d7d51134d1f6490ad9575
GIT binary patch
literal 601
zcmZ`#+iu%141Kn~f>Vt3>Nw4M*;=?j(TBD#O@XCv0|MEhA;z}kTFRv@`tPHhp=&Yh
zg%Zhg4i7o_k{a5i&f5;tZ==%}^Sn4aD_6%qs<o-wO_Prn;}`SPs_*$C$(7T|$#C3`
zPt%-C8gelZ1GqAP8`ZQmg0{8-S9H{R@D>_XAuJt&EumdH4Yu`UjT<s+s_~uXh}qA8
zg|_HG)%7Pdc&$7*uR0HF@)~vmFjqyD?XZwCbLen^h5t)sm9<90Oa!@Y%8!~rF8G?W
zkzmCF8kMtuuelMHIAlqqnm?72LeGM1J4`w(kX9&%LQn}ForlDLjBoCyvxmo@x3kH^
z^loxLyPiDWPo-cBMnsg2M6}kuPGHEGBqVkC{D&9Kt%xFAsTw4QC1$_=fwBl=3dI+e
zaSxI}JS}=Z(Ec80eF`!Zq3mqapXI|UN!a)t;@4hcu%Eq2xcoW}%!><-+XHTuHss+b
YOmM2;hq8Egm*4=7_P9T{21QBYH*F=mfB*mh
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/prepare-commit-msg.sample b/tests/resources/small.git/hooks/prepare-commit-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..b1970da1b6d3f42f00069fd17c325de72cda812e
GIT binary patch
literal 1702
zcmb_cTW{Mo6n>t6#i?x6xmZ$SFLf{QfG*3r0L?Pg?px55l8$UTGO3bO;spKi{V3XX
z))weX0X>M9bNMcZ-6yG%>(n}JI2|25dr<ew@wmMG{l(3lx~bQz>}WZBP@ih?JX^+@
zu#5O48P>yRX(m<b*PU*sORp92TCD1dX`%HE+1$w5k<(Ngu7zQ83#MGJR?<<W=d@yL
z#heqwo{FmCg0g#x<~R+PBD#}q(MBn;V$x;%UrJPP3*l%XtlvTWChI2SfJ$9e`YvSj
zRSOQ?NUgSMLI`3vL48YBHzwzVXoe7v0ef|0YHgV$N@?N(-R)rPqRDrK$n(%+OF$`P
zWdjC5N~`#RjV9}aYwQ4_yF5O-$h2`>fDIhYP)doc1&TADZa@ZGpusJ$6G+e$ZMcmC
zoOosDQPS}l{H?YPsq(4;0SGkATa9eeqAaDcj<jNAU+LTSmM1jo(ti~T0B7acJnnTX
zTarYy;Husdxal0!S<ba8=uu&a*SNYtr7|d7$g-q3_JHER2*oBsVW~i~XXdMx%LW~0
zT*960LF<qZ6K&FZ;vGmtyywBU3^Q>q8n2wALbFwU@2i@FAaRV!=uw-nwx1gKn2SvY
z>Ff>;2sg!+Hxfkwv1lsiii=p6WenF=5)6LZc<a$zDCD$GRuwvG4Fr+B#h?#9gTbio
zk#MdxC@WY%zSGN#i}Ts_#q`bf-{)`7CcWeB*7WlIyHjioJJWw&A5VItPUq3^9!r}S
zbykelFV-VFvcr>QaZ=aS_}+-4Y&?!@HWh|<^gJ21!|T@+%On#w6azxPHV}XsRbe*w
zR_TZ2XEsQa1lPK~biYqg@0-RW@5J1@=<87cFzEUABdCoFH2CZo?}l(Z*!OFqUxo>K
z_d`l#4d9|H6;VPT{X?^{VJ>oL|D7K{BJwwqB>`YcPoGk+9hbvHnoQ{EM|kPgD_`wk
zKm4#2xu;-y`RAm!=L_BnLvJ8$AZm8@?)v<%vwvsw8AF2x6!mTT;c72A_~U9nIq0ST
zv)N0!I!^1p=g8-RQfx5)E_Mb_4I2vtQpI30XZ&t<!9D6nI|;V7YR3)l6=S~QhuwLQ
g$e&^kTY-M99*<-Iw@(78*M5W!4}U}|8YyMx3->-9h5!Hn
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/push-to-checkout.sample b/tests/resources/small.git/hooks/push-to-checkout.sample
new file mode 100755
index 0000000000000000000000000000000000000000..a80611e18896f212c390d845e49a3f6d5693b41d
GIT binary patch
literal 2840
zcmai0U31$u5PXh)#YOS7cE^-rw@uolNhe9&aUS|HtvhX>G$45tVUYj>fRdF?|9kfU
zNR~aG=E)WbEbeyq7JTw}ZuHIE2kUtL<<n;$&G!2F^Je|kx2ug=4L5!H^!ogx`7o$&
z%Il(3zAe6<oe$^F=A|}s`8}CDp*M#3M)gC-)LOeDUpYMl3YNy9R)I-T)pE7sy09aj
zJ7%&5PnSB-F#2{jc><WLR{I2izuK%VHc+{hRfXe<^_q)8RjcE(6WX+F2)iAtDtI{x
ztAHVBq)eSp_E^xcV^i_5KLIHA$g{zEjh?rsacu+(Eyvve2~Kmw%;n3g(kWB56j~Js
z<yE5tYUsAR&PY0wgRvM8x!zgLX8SI!eVY&}YZ|>AoeCNptd-NM1aZLhESzC;I`+Ns
zfmNNjdAp^W8#Q*}l>CT7RB9F5(BbI8ly2l~+E};JW|>&d1)=epZ-8vm8ppkbEVn#R
zt30a5A-c(YQR8eM5%;|UAnO>rt!&@x@G@yp+92%w-}%(5P_+P&Wf_zb$f-Qrl5(7z
z2ah(bkE;!DK(&aAMuQ%1TS>ai?wSXCOCSj=_}8x4IbCx^$}9q)<W{Y<9o<WqZo^oV
z3bR;Qa%VT-M~kU@2n{=+=)C!MD`12;@<F*EtPfV3K#g^1%&ggH?4{m<R$WEKcUI4%
zl6{ik6Bo46pmNjdXg5@?hn;SjG$}rr3Gy#-BGgVD$8oD)OcK(oqca)L_kk)UBZ_&6
z*ourb#Yc8l3J+uSda_Y$GY--5Zp3QK9w^?P%E0xb57?fY+Q#+KU4)+R>whwv)SBt|
zg#MX4;;Oau`m=MI9(^&zPbueY@~>3*ixX%mvR5m_1&nAg@ZKvY1E$O}&EtLiG;mhV
z1xhMIm~fGjmf_#{62f`y;09?I7M1W2tWQvz<}i9lR>OpQyUJi45_&*pQus&EkwY<>
zI|ZAx=*3i9a-)g)hXkvO7>UJ5MNgL(Z+-wpXVcgbSgpmFmbf1~DPA(OVGI&FNLeIE
zNH!_aiH$vsif$_j7=T2{cS(!DOI`~bn@)vSd-0d7xL=DF;UNP|tW}4i<qWTSNCe|y
zg9kV&L9g;FhC@tvsVu#V-brqShHy2qZpA!gg{ZTY>h>DvHtu9tY_pbJ6x(6E*hxgC
zzNDao%qlr-IE%YGbS4hF!n!on7#W3$bX-_hbZAaws^nHu#)Dx=WzdbJ>AKzAy@T$x
zSWE^x9+|TEHVEPyaPYa0DOChp?AeHSBBDbZNokQpAY{lE!7geZI=jV)G^2@<iI(N4
zJLJjy@Y<VI$yq;3bVpJICW3D?YDMDl4Oe5pDf{d`i1_3Qx%4uX`$dOz<9e}jm2B-u
z8-?%!7XuiNF2O&MAdl-iw{drG+$F^zT2Z7WRV#c6;IRZEf>l)&91Zb1+`T+oq9wWF
zRV~kGTGce0O~p^6mj{kT5kL(pv>r;Lvd7VDX*P>A^Th`$3cWO<svk?_?FeP@458*2
zA1PqUOdd%VP5&4~ocLK16{1GLll64c=mU81RMFstpnMoLuI7hNIE4N)U%h*#<Fz{C
z9#>0<l7lRrls|W(8=W1O80P~6+U7<4BqC}N4-4GR3w#{O7YmH?JxwJfru2d?e){$5
z@5R+`7Z;1)FW;OkE-(HPisCS;5F4O^Q>L81p4Ysdo3ZP1(SrR-peEdTo;-@bkB((G
zPHYQXUL!@Q$e(OQ;R9r%@Afz+50I7>*^^c&&|E*r-jN)LH=pM4AqMwWxSv|nqjddE
Z4{_hwv8!W(<d3>T<BC%y-6O5i(|^OS%`gA}
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/sendemail-validate.sample b/tests/resources/small.git/hooks/sendemail-validate.sample
new file mode 100755
index 0000000000000000000000000000000000000000..effa6f54f2a5c1f0cebe258bf4805904be4b4852
GIT binary patch
literal 2365
zcmb_dU2oe)5PUX&#g-Nv2{5JDX_MA~3%Iq?8g*kpRgpYZIFU=~BI=I5J6e{T{`bz^
zk+$48h#x9Ika!=nv$M0y{clD}-j1x(hDWbnzP?l2k8j?TH{brS+Nf21yPm)Nczma>
zYw`X3V>TCdnSD1ru8&`j=2DIPbCT@SnIgUw>$+lEYP}+x8(BMYnr=iT3*ndq)xzaV
z>I+qjv}vC#8_9M+b1p#uNS0M0)q<p>8!3p_LRQ0MA3M`!2foxzRUjbFY@}O~(ki=S
zqscnq8cU*dY)D$$cqE}n)V0yIk>CNKHCrndOtSP*HbOb;nbwAHSb;R+gs^?^Dve%)
zoW}t(*D}$>O3ab0TS^-;J|u&sb-PkZzo#kn*#xYt(;<xzKW(YtQZ$u23?yV#kyh1~
z@+Idh;EG5jXx8@%<;Y_W8SA=|T;MPQ)TB!!<QcbU)YR4)79eeeg4|vp-8jm%Dl3^I
zRS7+i3>FGuwzSb^g&RDiGcOz9TB;Hu`nJh)$W=C=XCSm2AY=$w3G3P-V#Oo+N*;#2
z4ijJ-pBZ=;T(RTgp_HYrD!uW-dTMfkuqY5jwOy)~gM;#=P^i{!l7`pXTS^s(&^{RU
zydaw}OpS#^D1cXM8?FW+fh`t7D(g;yr6|}fdaNtZBx3hlK~IpkTu3!Qq%R+zAo#<L
zU&m+XIFB0>t}Bs8^3$vHD+-TGT@`F>H1Cc#WAVW;&$S6%fE2d6@kLS0g&ihIM{}0z
z8#XhD>b>3{(BH|Px7}&lJ4%y1v<t$W+!F|W@<gaU4;MqSHKR(wdML<XnCv;zaPrSi
zxVCvek26-bf#Q!H+uAgy_{e_1UZCq>(CihZJx@8MPoGdl*BJGD;usf*iS7%;{Joe;
zNFuBa>*~o&qETDPo~u&~$FxE1xb^x&(CbE`Y3GfsibL2rl+L;>P6j&Y3U>K$mkp*6
zd`Q{<^+^&;GskGjwD-%!boR&i-TC<Uvy02w+l$Nb?B}aL-%ZDpluqd=K_@}z*fyuV
zzAs1Hf?3v$k!X7GTc8Q=r`WJ_Uu=>A9UOR|@=GYb5x#<f&WSMH%mCJU<#=7=qFdL6
zG?Wz&6z&J<@I(B>+dhd7fkaVIR^pol`Mv+rUbmZ43dVL6^S7g3{NsPiG$iy$5EDB%
z6KIgnb$H(n&t3e4E6d4V7w^B?JS}JkG)PM6+X3Co`SQs($O*AA+MG~{S7RJ=cy-l&
z>~%3y`tjfx2>uOu<lDGWewcb=oL@}B@B6FCZ?oxSJWldrm%UfOduahk%C0H>tB_^s
ziwG=e=ch|FQ0IkN91US7rhdQkXhwwt$gU0WEVDjo=IPb+?6PC=s8}J*ua(Ms))`UL
fi$|vMHn?H<rred|1&u#kOoJ`%vx)-*g-ZSfgGvdP
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/update.sample b/tests/resources/small.git/hooks/update.sample
new file mode 100755
index 0000000000000000000000000000000000000000..c3430f6fe27e572a1cebdb2f2bec70add0642a60
GIT binary patch
literal 3707
zcmb_fU2oeq6n(aS#jRq*br9K3y0z;^QgkUc^kKlTrB6js&@yE)mPu8l<0$L?`wmIT
zrsXba))@gJi@e|G+<SfSXe`CeSQ}OG@sr8ZTUlQ{dzM}Q@O-hBi}GeUom`#X%FiYH
zX?m4Rna-0RN2lfK)A3ZuvHcz$L<jUn62D=~vfz{}wIH2VqBLX_O$(JSXeF7H$}q!c
zWY}C&R;eX%X?P{%d;|>_tSE3ettp-hLlsZCxaLX8(nU;bVRB;Ce6@s#eu2|WvLz>-
zvy(&>Gyfp@+BtKnpqWkKi^+v{4jn_pNw_zeuxE<mRXKx8G3;9pl+45&4~hHW!G@wo
za7?X(0B}HbX*ExkDmas*xzV)FxygC8AL?2Z1x-0QJqS@qn8sD7r{bm30@<%eL_gOw
z;~85O$Xw2AS}Qp)5ViRUe3|ir8;&&I<B7Y6^!kkNyYXF4EY(b8_5DsTYn_&?wkdEz
z0y$tADo<&}nGs5kg2-J=0KlEGPb(%<An(pXY{K`qIZCuwiT|2{8JD&5o_~`o6<;dD
zi@J#zCE4={8j%<uy|iutvHu1QKo5Tno-BAF2FwD%%O#UDDum=w!;!PNe-cOFNX4)5
zd>TifiGO|)w}OANj2n2D^K=o3j6P6uOL70#cbA{uzWXDlk1wr9GV1X(2W{RuTvjXV
zCmd<W?kH^?PXjkbF`XZtwu1B+%4@ZvHIwGpJ*8@8`MWAhq^B|Hj1lzj3R8bVuMpNb
zz4Gzk!3T3bY;WEGIww&kq9BYW6EP*q$K|EB-@TH(Fjtz*`HMTO?i+3E;5tdSah&xZ
z+t!x4K7)dpy5wiJhlJz~8qF|r8a&-SV7^I3C@_q=P`yt@_x_F-;PQR)fzP<zNN>8u
zH%V`94=q3)Dk)PHNrnFC(T1)Om6f{Usj;u1R->&XoCYVK2V3ZlgZuF?N}1+33<P7e
z<0yVF?Qoa{l#7q(3&jv=Ab)gpM8A7`p%3InNzSxy)ZER2U0C#9zKpnLY0I?>OER*x
z*9Z=L=zI8CN>A_^jYjt0F$psO$sL=38q5q|SG)qCN6{^>RFh5E&l5GZ$pEahnF&d+
z5c>64t}uJPkf~_!VUj#&N%nC-gUMj%=@B=!V>&}xtj2%@-mOm#rQUSJ3(ccmc+fza
znZ#uxF>N?QN5UrIEd!5RgHEf<eGg}P45aAs(Xs6u!XW9r1I*E6XJ^1movX@xYLuPz
z|7xBN4z@b}#x>W#;(nKYF+D<*rdshJ$X-z2OZ2X;)nn@KSVdVhaA?}@3;6gZxb4<W
z`9sa`0lR_azMX|=t_(FvG@%w2);9P}SG0u&K1(*oX3};~=<<!N*F$UTSxD{V&6mgL
ztw9Ntc2eOF@c!OJytNC4T^#)Ien7-`dI{6s#co~0f^E3J<0Ty)l3xq2u@Y8DXTK>v
zozoWSr{{+!h}zGpumG3H`=AvWpm^9kW;J$Jp^Xl*?8ckr`fqN%c|Z;VC0|cM4vSrk
zH_O8Yvh85nvJp^;``wo8=z0f`FWg?`>gO#y1hjX1{}rTlg9rwIKia8eyGexA3GnuR
z`Rg~XZoW;0pA)vI8=p5!+6sIn#C^FCvR>ffv39h6SCNi9v);%WD;WZ`of_MgwyRWy
z-yY%n*Y>X8<Sf+RyB|Sr2YG@1w~%U$o`(5EDkJ|3$u=eMZA&_wxEsy<Xxh2k^tP?4
RGx&ZHQs^8zuIpu!=pQhfp8Eg*
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/info/exclude b/tests/resources/small.git/info/exclude
new file mode 100644
index 0000000000000000000000000000000000000000..a5196d1be8fb59edf8062bef36d3a602e0812139
GIT binary patch
literal 240
zcmXYqK?=e!6h!x)VxWtv*mf_t5?px$aS_{}Hj?C*<cHXeXE&AZhT*-L3ZoI&*l1%Z
zqG?zr3TvQGZ__}H4(u*%p*rI=cU!%ya5ugfGATh66$IJHgu1Gs0-<N;$V+SsdE)?u
zIq;i$f#WE4f$_MWicZjMEob9LWKMR#iwZq54~QgST^6=i%u0lUkJu-_J**QBMq}ZG
Wth_)NDbl|`oQr&HAFQ5h`0jqAIZ*BZ
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/88/f9a6eaa2cf008b9bc92847178621f21fa99f3e b/tests/resources/small.git/objects/88/f9a6eaa2cf008b9bc92847178621f21fa99f3e
new file mode 100644
index 0000000000000000000000000000000000000000..7d80b8d78e0dc55669d831a6638f48ec9fed0982
GIT binary patch
literal 50
zcmV-20L}k+0V^p=O;s>9W-v4`Ff%bx$Vkn}$!Ay}rnY6F$m-Kg*KD_+;Lx#g4|^&N
I02NaX#p`nv=Kufz
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/af/5626b4a114abcb82d63db7c8082c3c4756e51b b/tests/resources/small.git/objects/af/5626b4a114abcb82d63db7c8082c3c4756e51b
new file mode 100644
index 0000000000000000000000000000000000000000..822bc151862ec3763cf2d3fa2372b93bbd3a4b65
GIT binary patch
literal 30
mcmb<m^geacKghr&@q@?NlP9kSYMj?U<r(;diNWtH+YSKNt_|)0
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/c6/97d4f7a6eac8d7b131673c340bd3cc5bac14d4 b/tests/resources/small.git/objects/c6/97d4f7a6eac8d7b131673c340bd3cc5bac14d4
new file mode 100644
index 0000000000000000000000000000000000000000..5adfa88cc6b00bb19f2031b8ab61f6d07f9ccdb8
GIT binary patch
literal 130
zcmV-|0Db>>0i}&W3IZ_@1U=^!a~EV1casc=c+{&un1qQN*i9hD|0|m(2n|iwp*q%W
z%N;b$hu%cM`$TMo*~EnC1BFP&Pfj~;jZVKXQ96s_PhV<-XAROi+@-v8dBLUa`!;GB
k^i<X>XlEv8$>R)1G>9th&t3j;s7J{?^9n<zzF|~BaA?ar-~a#s
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/refs/heads/master b/tests/resources/small.git/refs/heads/master
new file mode 100644
index 0000000000000000000000000000000000000000..4eb36d22298f060fd324155ab854d9d6486fc498
GIT binary patch
literal 41
ucmV~$!4Uu;2m`Rc)5uXl$AMP&AHjriQg~T$i(A>|7U^`%mXoWC24Q^m!3%@{
literal 0
HcmV?d00001
+1
View File
@@ -0,0 +1 @@
../../.version
+1
View File
@@ -0,0 +1 @@
../../src/libstore-tests/data/derivation
+1
View File
@@ -0,0 +1 @@
../../src/libutil-tests/data/memory-source-accessor
+1
View File
@@ -0,0 +1 @@
../../src/libutil-tests/data/hash
+92
View File
@@ -0,0 +1,92 @@
# Run with:
# meson test --suite json-schema
# Run with: (without shell / configure)
# nix build .#nix-json-schema-checks
project(
'nix-json-schema-checks',
version : files('.version'),
meson_version : '>= 1.1',
license : 'LGPL-2.1-or-later',
)
fs = import('fs')
# Note: The 'jsonschema' package provides the 'jv' command
jv = find_program('jv', required : true)
# The schema directory is a committed symlink to the actual schema location
schema_dir = meson.current_source_dir() / 'schema'
# Get all example files
schemas = [
{
'stem' : 'file-system-object',
'schema' : schema_dir / 'file-system-object-v1.yaml',
'files' : [
'simple.json',
'complex.json',
],
},
{
'stem' : 'hash',
'schema' : schema_dir / 'hash-v1.yaml',
'files' : [
'sha256-base64.json',
'sha256-base16.json',
'sha256-nix32.json',
'blake3-base64.json',
],
},
{
'stem' : 'derivation',
'schema' : schema_dir / 'derivation-v3.yaml',
'files' : [
'dyn-dep-derivation.json',
'simple-derivation.json',
],
},
# # Not sure how to make subschema work
# {
# 'stem': 'derivation',
# 'schema': schema_dir / 'derivation-v3.yaml#output',
# 'files' : [
# 'output-caFixedFlat.json',
# 'output-caFixedNAR.json',
# 'output-caFixedText.json',
# 'output-caFloating.json',
# 'output-deferred.json',
# 'output-impure.json',
# 'output-inputAddressed.json',
# ],
# },
]
# Validate each example against the schema
foreach schema : schemas
stem = schema['stem']
schema_file = schema['schema']
if '#' not in schema_file
# Validate the schema itself against JSON Schema Draft 04
test(
stem + '-schema-valid',
jv,
args : [
'http://json-schema.org/draft-04/schema',
schema_file,
],
suite : 'json-schema',
)
endif
foreach example : schema['files']
test(
stem + '-example-' + fs.stem(example),
jv,
args : [
schema_file,
files(stem / example),
],
suite : 'json-schema',
)
endforeach
endforeach
+52
View File
@@ -0,0 +1,52 @@
# Run with: nix build .#nix-json-schema-checks
{
lib,
mkMesonDerivation,
meson,
ninja,
jsonschema,
# Configuration Options
version,
}:
mkMesonDerivation (finalAttrs: {
pname = "nix-json-schema-checks";
inherit version;
workDir = ./.;
fileset = lib.fileset.unions [
../../.version
../../doc/manual/source/protocols/json/schema
../../src/libutil-tests/data/memory-source-accessor
../../src/libutil-tests/data/hash
../../src/libstore-tests/data/derivation
./.
];
outputs = [ "out" ];
passthru.externalNativeBuildInputs = [
jsonschema
];
nativeBuildInputs = [
meson
ninja
]
++ finalAttrs.passthru.externalNativeBuildInputs;
doCheck = true;
mesonCheckFlags = [ "--print-errorlogs" ];
postInstall = ''
touch $out
'';
meta = {
platforms = lib.platforms.all;
};
})
+1
View File
@@ -0,0 +1 @@
../../doc/manual/source/protocols/json/schema
+3 -2
View File
@@ -117,10 +117,11 @@ RealisedPath::Set BuiltPath::toRealisedPaths(Store & store) const
"the derivation '%s' has unrealised output '%s' (derived-path.cc/toRealisedPaths)",
store.printStorePath(p.drvPath->outPath()),
outputName);
auto thisRealisation = store.queryRealisation(DrvOutput{*drvOutput, outputName});
DrvOutput key{*drvOutput, outputName};
auto thisRealisation = store.queryRealisation(key);
assert(thisRealisation); // We’ve built it, so we must
// have the realisation
res.insert(*thisRealisation);
res.insert(Realisation{*thisRealisation, std::move(key)});
} else {
res.insert(outputPath);
}
+14
View File
@@ -350,6 +350,20 @@ struct MixEnvironment : virtual Args
void setEnviron();
};
struct MixNoCheckSigs : virtual Args
{
CheckSigsFlag checkSigs = CheckSigs;
MixNoCheckSigs()
{
addFlag({
.longName = "no-check-sigs",
.description = "Do not require that paths are signed by trusted keys.",
.handler = {&checkSigs, NoCheckSigs},
});
}
};
void completeFlakeInputAttrPath(
AddCompletions & completions,
ref<EvalState> evalState,
@@ -69,7 +69,7 @@ struct InstallableFlake : InstallableValue
*/
std::vector<ref<eval_cache::AttrCursor>> getCursors(EvalState & state) override;
std::shared_ptr<flake::LockedFlake> getLockedFlake() const;
ref<flake::LockedFlake> getLockedFlake() const;
FlakeRef nixpkgsFlakeRef() const;
};
@@ -87,6 +87,4 @@ static inline FlakeRef defaultNixpkgsFlakeRef()
return FlakeRef::fromAttrs(fetchSettings, {{"type", "indirect"}, {"id", "nixpkgs"}});
}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, std::shared_ptr<flake::LockedFlake> lockedFlake);
} // namespace nix
+4 -4
View File
@@ -185,16 +185,16 @@ std::vector<ref<eval_cache::AttrCursor>> InstallableFlake::getCursors(EvalState
return res;
}
std::shared_ptr<flake::LockedFlake> InstallableFlake::getLockedFlake() const
ref<flake::LockedFlake> InstallableFlake::getLockedFlake() const
{
if (!_lockedFlake) {
flake::LockFlags lockFlagsApplyConfig = lockFlags;
// FIXME why this side effect?
lockFlagsApplyConfig.applyNixConfig = true;
_lockedFlake =
std::make_shared<flake::LockedFlake>(lockFlake(flakeSettings, *state, flakeRef, lockFlagsApplyConfig));
_lockedFlake = make_ref<flake::LockedFlake>(lockFlake(flakeSettings, *state, flakeRef, lockFlagsApplyConfig));
}
return _lockedFlake;
// _lockedFlake is now non-null but still just a shared_ptr
return ref<flake::LockedFlake>(_lockedFlake);
}
FlakeRef InstallableFlake::nixpkgsFlakeRef() const
+1 -38
View File
@@ -342,8 +342,7 @@ void completeFlakeRefWithFragment(
parseFlakeRef(fetchSettings, expandTilde(flakeRefS), std::filesystem::current_path().string());
auto evalCache = openEvalCache(
*evalState,
std::make_shared<flake::LockedFlake>(lockFlake(flakeSettings, *evalState, flakeRef, lockFlags)));
*evalState, make_ref<flake::LockedFlake>(lockFlake(flakeSettings, *evalState, flakeRef, lockFlags)));
auto root = evalCache->getRoot();
@@ -443,42 +442,6 @@ static StorePath getDeriver(ref<Store> store, const Installable & i, const Store
return *derivers.begin();
}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, std::shared_ptr<flake::LockedFlake> lockedFlake)
{
auto fingerprint = evalSettings.useEvalCache && evalSettings.pureEval
? lockedFlake->getFingerprint(state.store, state.fetchSettings)
: std::nullopt;
auto rootLoader = [&state, lockedFlake]() {
/* For testing whether the evaluation cache is
complete. */
if (getEnv("NIX_ALLOW_EVAL").value_or("1") == "0")
throw Error("not everything is cached, but evaluation is not allowed");
auto vFlake = state.allocValue();
flake::callFlake(state, *lockedFlake, *vFlake);
state.forceAttrs(*vFlake, noPos, "while parsing cached flake data");
auto aOutputs = vFlake->attrs()->get(state.symbols.create("outputs"));
assert(aOutputs);
return aOutputs->value;
};
if (fingerprint) {
auto search = state.evalCaches.find(fingerprint.value());
if (search == state.evalCaches.end()) {
search =
state.evalCaches
.emplace(fingerprint.value(), make_ref<nix::eval_cache::EvalCache>(fingerprint, state, rootLoader))
.first;
}
return search->second;
} else {
return make_ref<nix::eval_cache::EvalCache>(std::nullopt, state, rootLoader);
}
}
Installables SourceExprCommand::parseInstallables(ref<Store> store, std::vector<std::string> ss)
{
Installables result;
-1
View File
@@ -67,7 +67,6 @@ config_priv_h = configure_file(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'built-path.cc',
-1
View File
@@ -28,7 +28,6 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_expr.cc',
-1
View File
@@ -31,7 +31,6 @@ rapidcheck = dependency('rapidcheck')
deps_public += rapidcheck
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'tests/value/context.cc',
+1 -2
View File
@@ -45,7 +45,6 @@ config_priv_h = configure_file(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'derived-path.cc',
@@ -83,7 +82,7 @@ this_exe = executable(
test(
meson.project_name(),
this_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
},
protocol : 'gtest',
-1
View File
@@ -62,7 +62,6 @@ mkMesonExecutable (finalAttrs: {
mkdir -p "$HOME"
''
+ ''
export ASAN_OPTIONS=abort_on_error=1:print_summary=1:detect_leaks=0
export _NIX_TEST_UNIT_DATA=${resolvePath ./data}
${stdenv.hostPlatform.emulator buildPackages} ${lib.getExe finalAttrs.finalPackage}
touch $out
+8 -1
View File
@@ -508,8 +508,15 @@ private:
public:
/**
* @param lookupPath Only used during construction.
* @param store The store to use for instantiation
* @param fetchSettings Must outlive the lifetime of this EvalState!
* @param settings Must outlive the lifetime of this EvalState!
* @param buildStore The store to use for builds ("import from derivation", C API `nix_string_realise`)
*/
EvalState(
const LookupPath & _lookupPath,
const LookupPath & lookupPath,
ref<Store> store,
const fetchers::Settings & fetchSettings,
const EvalSettings & settings,
-1
View File
@@ -97,7 +97,6 @@ config_priv_h = configure_file(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
parser_tab = custom_target(
input : 'parser.y',
+2 -1
View File
@@ -1420,7 +1420,8 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
.debugThrow();
}
if (ingestionMethod == ContentAddressMethod::Raw::Text)
experimentalFeatureSettings.require(Xp::DynamicDerivations);
experimentalFeatureSettings.require(
Xp::DynamicDerivations, fmt("text-hashed derivation '%s', outputHashMode = \"text\"", drvName));
if (ingestionMethod == ContentAddressMethod::Raw::Git)
experimentalFeatureSettings.require(Xp::GitHashing);
};
+7 -3
View File
@@ -199,8 +199,8 @@ static void fetchTree(
if (state.settings.pureEval && !input.isLocked()) {
if (input.getNarHash())
warn(
"Input '%s' is unlocked (e.g. lacks a Git revision) but does have a NAR hash. "
"This is deprecated since such inputs are verifiable but may not be reproducible.",
"Input '%s' is unlocked (e.g. lacks a Git revision) but is checked by NAR hash. "
"This is not reproducible and will break after garbage collection or when shared.",
input.to_string());
else
state
@@ -588,7 +588,11 @@ static void fetch(
if (expectedHash) {
auto hash = unpack ? state.store->queryPathInfo(storePath)->narHash
: hashFile(HashAlgorithm::SHA256, state.store->toRealPath(storePath));
: hashPath(
{state.store->requireStoreObjectAccessor(storePath)},
FileSerialisationMethod::Flat,
HashAlgorithm::SHA256)
.hash;
if (hash != *expectedHash) {
state
.error<EvalError>(
-1
View File
@@ -32,7 +32,6 @@ add_project_arguments(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_fetchers.cc',
+1 -2
View File
@@ -37,7 +37,6 @@ libgit2 = dependency('libgit2')
deps_private += libgit2
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'access-tokens.cc',
@@ -64,7 +63,7 @@ this_exe = executable(
test(
meson.project_name(),
this_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
},
protocol : 'gtest',
-1
View File
@@ -61,7 +61,6 @@ mkMesonExecutable (finalAttrs: {
buildInputs = [ writableTmpDirAsHomeHook ];
}
''
export ASAN_OPTIONS=abort_on_error=1:print_summary=1:detect_leaks=0
export _NIX_TEST_UNIT_DATA=${resolvePath ./data}
${stdenv.hostPlatform.emulator buildPackages} ${lib.getExe finalAttrs.finalPackage}
touch $out
+1 -2
View File
@@ -332,8 +332,7 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(ref<Store> sto
debug("using substituted/cached input '%s' in '%s'", to_string(), store->printStorePath(storePath));
// We just ensured the store object was there
auto accessor = ref{store->getFSAccessor(storePath)};
auto accessor = store->requireStoreObjectAccessor(storePath);
accessor->fingerprint = getFingerprint(store);
+11 -11
View File
@@ -399,7 +399,8 @@ struct GitHubInputScheme : GitArchiveInputScheme
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
auto downloadResult = downloadFile(store, *input.settings, url, "source", headers);
auto json = nlohmann::json::parse(store->getFSAccessor(downloadResult.storePath)->readFile(CanonPath::root));
auto json = nlohmann::json::parse(
store->requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
return RefInfo{
.rev = Hash::parseAny(std::string{json["sha"]}, HashAlgorithm::SHA1),
@@ -473,7 +474,8 @@ struct GitLabInputScheme : GitArchiveInputScheme
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
auto downloadResult = downloadFile(store, *input.settings, url, "source", headers);
auto json = nlohmann::json::parse(store->getFSAccessor(downloadResult.storePath)->readFile(CanonPath::root));
auto json = nlohmann::json::parse(
store->requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
if (json.is_array() && json.size() >= 1 && json[0]["id"] != nullptr) {
return RefInfo{.rev = Hash::parseAny(std::string(json[0]["id"]), HashAlgorithm::SHA1)};
@@ -548,13 +550,10 @@ struct SourceHutInputScheme : GitArchiveInputScheme
std::string refUri;
if (ref == "HEAD") {
auto file = store->toRealPath(
downloadFile(store, *input.settings, fmt("%s/HEAD", base_url), "source", headers).storePath);
std::ifstream is(file);
std::string line;
getline(is, line);
auto downloadFileResult = downloadFile(store, *input.settings, fmt("%s/HEAD", base_url), "source", headers);
auto contents = store->requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
auto remoteLine = git::parseLsRemoteLine(line);
auto remoteLine = git::parseLsRemoteLine(getLine(contents).first);
if (!remoteLine) {
throw BadURL("in '%d', couldn't resolve HEAD ref '%d'", input.to_string(), ref);
}
@@ -564,9 +563,10 @@ struct SourceHutInputScheme : GitArchiveInputScheme
}
std::regex refRegex(refUri);
auto file = store->toRealPath(
downloadFile(store, *input.settings, fmt("%s/info/refs", base_url), "source", headers).storePath);
std::ifstream is(file);
auto downloadFileResult =
downloadFile(store, *input.settings, fmt("%s/info/refs", base_url), "source", headers);
auto contents = store->requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
std::istringstream is(contents);
std::string line;
std::optional<std::string> id;
+1 -3
View File
@@ -329,9 +329,7 @@ struct MercurialInputScheme : InputScheme
Input input(_input);
auto storePath = fetchToStore(store, input);
// We just added it, it should be there.
auto accessor = ref{store->getFSAccessor(storePath)};
auto accessor = store->requireStoreObjectAccessor(storePath);
accessor->setPathDisplay("«" + input.to_string() + "»");
-1
View File
@@ -32,7 +32,6 @@ libgit2 = dependency('libgit2', version : '>= 1.9')
deps_private += libgit2
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'attrs.cc',
+1 -1
View File
@@ -138,7 +138,7 @@ struct PathInputScheme : InputScheme
storePath = store->addToStoreFromDump(*src, "source");
}
auto accessor = ref{store->getFSAccessor(*storePath)};
auto accessor = store->requireStoreObjectAccessor(*storePath);
// To prevent `fetchToStore()` copying the path again to Nix
// store, pre-create an entry in the fetcher cache.
-1
View File
@@ -32,7 +32,6 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_flake.cc',
+7
View File
@@ -1,8 +1,15 @@
#include <gtest/gtest.h>
#include <set>
#include <string>
#include <utility>
#include "nix/fetchers/fetch-settings.hh"
#include "nix/flake/flakeref.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/util/configuration.hh"
#include "nix/util/error.hh"
#include "nix/util/experimental-features.hh"
namespace nix {
+1 -2
View File
@@ -34,7 +34,6 @@ gtest = dependency('gtest', main : true)
deps_private += gtest
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'flakeref.cc',
@@ -59,7 +58,7 @@ this_exe = executable(
test(
meson.project_name(),
this_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
'NIX_CONFIG' : 'extra-experimental-features = flakes',
'HOME' : meson.current_build_dir() / 'test-home',
+7 -5
View File
@@ -1,15 +1,17 @@
#include <gtest/gtest.h>
#include <filesystem>
#include <string>
#include "nix/util/file-system.hh"
#include "nix_api_store.h"
#include "nix_api_util.h"
#include "nix_api_expr.h"
#include "nix_api_value.h"
#include "nix_api_flake.h"
#include "nix/expr/tests/nix_api_expr.hh"
#include "nix/util/tests/string_callback.hh"
#include <gmock/gmock.h>
#include <gtest/gtest.h>
#include "nix/store/tests/nix_api_store.hh"
#include "nix/util/tests/nix_api_util.hh"
#include "nix_api_fetchers.h"
namespace nixC {
-1
View File
@@ -59,7 +59,6 @@ mkMesonExecutable (finalAttrs: {
buildInputs = [ writableTmpDirAsHomeHook ];
}
(''
export ASAN_OPTIONS=abort_on_error=1:print_summary=1:detect_leaks=0
export _NIX_TEST_UNIT_DATA=${resolvePath ./data}
export NIX_CONFIG="extra-experimental-features = flakes"
${stdenv.hostPlatform.emulator buildPackages} ${lib.getExe finalAttrs.finalPackage}
+3 -1
View File
@@ -1,6 +1,8 @@
#include "nix/flake/url-name.hh"
#include <gtest/gtest.h>
#include "nix/flake/url-name.hh"
#include "nix/util/url.hh"
namespace nix {
/* ----------- tests for url-name.hh --------------------------------------------------*/
+23 -2
View File
@@ -1,9 +1,30 @@
#include <nlohmann/json.hpp>
#include <assert.h>
#include <stdint.h>
#include <nlohmann/json_fwd.hpp>
#include <cctype>
#include <filesystem>
#include <map>
#include <memory>
#include <optional>
#include <string>
#include <utility>
#include <variant>
#include <vector>
#include <format>
#include "nix/util/users.hh"
#include "nix/util/config-global.hh"
#include "nix/flake/settings.hh"
#include "nix/flake/flake.hh"
#include <nlohmann/json.hpp>
#include "nix/util/ansicolor.hh"
#include "nix/util/configuration.hh"
#include "nix/util/file-system.hh"
#include "nix/util/fmt.hh"
#include "nix/util/logging.hh"
#include "nix/util/strings.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
namespace nix::flake {
+26
View File
@@ -1,8 +1,34 @@
#include <stdint.h>
#include <functional>
#include <map>
#include <optional>
#include <set>
#include <string>
#include <utility>
#include <variant>
#include <vector>
#include "nix/flake/flake-primops.hh"
#include "nix/expr/eval.hh"
#include "nix/flake/flake.hh"
#include "nix/flake/flakeref.hh"
#include "nix/flake/settings.hh"
#include "nix/expr/attr-set.hh"
#include "nix/expr/eval-error.hh"
#include "nix/expr/eval-inline.hh"
#include "nix/expr/eval-settings.hh"
#include "nix/expr/symbol-table.hh"
#include "nix/expr/value.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/util/configuration.hh"
#include "nix/util/error.hh"
#include "nix/util/experimental-features.hh"
#include "nix/util/pos-idx.hh"
#include "nix/util/pos-table.hh"
#include "nix/util/source-path.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
namespace nix::flake::primops {
+87 -7
View File
@@ -1,9 +1,32 @@
#include <nlohmann/json.hpp>
#include <assert.h>
#include <stdint.h>
#include <boost/container/detail/std_fwd.hpp>
#include <boost/core/pointer_traits.hpp>
#include <boost/unordered/detail/foa/table.hpp>
#include <algorithm>
#include <filesystem>
#include <functional>
#include <map>
#include <memory>
#include <optional>
#include <set>
#include <span>
#include <string>
#include <tuple>
#include <utility>
#include <variant>
#include <vector>
#include <format>
#include "nix/util/terminal.hh"
#include "nix/util/ref.hh"
#include "nix/util/environment-variables.hh"
#include "nix/flake/flake.hh"
#include "nix/expr/eval.hh"
#include "nix/expr/eval-cache.hh"
#include "nix/expr/eval-settings.hh"
#include "nix/flake/lockfile.hh"
#include "nix/expr/primops.hh"
#include "nix/expr/eval-inline.hh"
#include "nix/store/store-api.hh"
#include "nix/fetchers/fetchers.hh"
@@ -11,14 +34,36 @@
#include "nix/fetchers/fetch-settings.hh"
#include "nix/flake/settings.hh"
#include "nix/expr/value-to-json.hh"
#include "nix/store/local-fs-store.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/util/memory-source-accessor.hh"
#include "nix/fetchers/input-cache.hh"
#include <nlohmann/json.hpp>
#include "nix/expr/attr-set.hh"
#include "nix/expr/eval-error.hh"
#include "nix/expr/nixexpr.hh"
#include "nix/expr/symbol-table.hh"
#include "nix/expr/value.hh"
#include "nix/expr/value/context.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/registry.hh"
#include "nix/flake/flakeref.hh"
#include "nix/store/path.hh"
#include "nix/util/canon-path.hh"
#include "nix/util/configuration.hh"
#include "nix/util/error.hh"
#include "nix/util/experimental-features.hh"
#include "nix/util/file-system.hh"
#include "nix/util/fmt.hh"
#include "nix/util/hash.hh"
#include "nix/util/logging.hh"
#include "nix/util/pos-idx.hh"
#include "nix/util/pos-table.hh"
#include "nix/util/position.hh"
#include "nix/util/source-path.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
namespace nix {
struct SourceAccessor;
using namespace flake;
@@ -862,7 +907,7 @@ static ref<SourceAccessor> makeInternalFS()
internalFS->setPathDisplay("«flakes-internal»", "");
internalFS->addFile(
CanonPath("call-flake.nix"),
#include "call-flake.nix.gen.hh"
#include "call-flake.nix.gen.hh" // IWYU pragma: keep
);
return internalFS;
}
@@ -924,8 +969,6 @@ void callFlake(EvalState & state, const LockedFlake & lockedFlake, Value & vRes)
state.callFunction(*vCallFlake, args, vRes, noPos);
}
} // namespace flake
std::optional<Fingerprint> LockedFlake::getFingerprint(ref<Store> store, const fetchers::Settings & fetchSettings) const
{
if (lockFile.isUnlocked(fetchSettings))
@@ -953,4 +996,41 @@ std::optional<Fingerprint> LockedFlake::getFingerprint(ref<Store> store, const f
Flake::~Flake() {}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, ref<const LockedFlake> lockedFlake)
{
auto fingerprint = state.settings.useEvalCache && state.settings.pureEval
? lockedFlake->getFingerprint(state.store, state.fetchSettings)
: std::nullopt;
auto rootLoader = [&state, lockedFlake]() {
/* For testing whether the evaluation cache is
complete. */
if (getEnv("NIX_ALLOW_EVAL").value_or("1") == "0")
throw Error("not everything is cached, but evaluation is not allowed");
auto vFlake = state.allocValue();
callFlake(state, *lockedFlake, *vFlake);
state.forceAttrs(*vFlake, noPos, "while parsing cached flake data");
auto aOutputs = vFlake->attrs()->get(state.symbols.create("outputs"));
assert(aOutputs);
return aOutputs->value;
};
if (fingerprint) {
auto search = state.evalCaches.find(fingerprint.value());
if (search == state.evalCaches.end()) {
search = state.evalCaches
.emplace(fingerprint.value(), make_ref<eval_cache::EvalCache>(fingerprint, state, rootLoader))
.first;
}
return search->second;
} else {
return make_ref<eval_cache::EvalCache>(std::nullopt, state, rootLoader);
}
}
} // namespace flake
} // namespace nix
+30 -1
View File
@@ -1,10 +1,39 @@
#include <assert.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <filesystem>
#include <ostream>
#include <string_view>
#include <vector>
#include <optional>
#include <regex>
#include <string>
#include <tuple>
#include <utility>
#include "nix/flake/flakeref.hh"
#include "nix/store/store-api.hh"
#include "nix/util/url.hh"
#include "nix/util/url-parts.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/util/error.hh"
#include "nix/util/file-system.hh"
#include "nix/util/fmt.hh"
#include "nix/util/logging.hh"
#include "nix/util/strings.hh"
#include "nix/util/util.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/registry.hh"
#include "nix/store/outputs-spec.hh"
#include "nix/util/ref.hh"
#include "nix/util/types.hh"
namespace nix {
class Store;
struct SourceAccessor;
namespace fetchers {
struct Settings;
} // namespace fetchers
#if 0
// 'dir' path elements cannot start with a '.'. We also reject
@@ -1,7 +1,12 @@
#pragma once
#include "nix/expr/eval.hh"
#include "nix/flake/settings.hh"
namespace nix {
namespace flake {
struct Settings;
} // namespace flake
} // namespace nix
namespace nix::flake::primops {
+6
View File
@@ -5,6 +5,7 @@
#include "nix/flake/flakeref.hh"
#include "nix/flake/lockfile.hh"
#include "nix/expr/value.hh"
#include "nix/expr/eval-cache.hh"
namespace nix {
@@ -218,6 +219,11 @@ lockFlake(const Settings & settings, EvalState & state, const FlakeRef & flakeRe
void callFlake(EvalState & state, const LockedFlake & lockedFlake, Value & v);
/**
* Open an evaluation cache for a flake.
*/
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, ref<const LockedFlake> lockedFlake);
} // namespace flake
void emitTreeAttrs(
+8 -2
View File
@@ -2,9 +2,11 @@
///@file
#include <regex>
#include <iosfwd>
#include <string>
#include <tuple>
#include <utility>
#include "nix/util/types.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/store/outputs-spec.hh"
#include "nix/fetchers/registry.hh"
@@ -12,6 +14,10 @@ namespace nix {
class Store;
namespace fetchers {
struct Settings;
} // namespace fetchers
typedef std::string FlakeId;
/**
+3 -2
View File
@@ -1,9 +1,10 @@
#pragma once
///@file
#include "nix/util/configuration.hh"
#include <sys/types.h>
#include <string>
#include "nix/util/configuration.hh"
namespace nix {
// Forward declarations
+3 -4
View File
@@ -1,9 +1,8 @@
#include "nix/util/url.hh"
#include "nix/util/url-parts.hh"
#include "nix/util/util.hh"
#include "nix/util/split.hh"
#include <optional>
#include <string>
namespace nix {
struct ParsedURL;
/**
* Try to extract a reasonably unique and meaningful, human-readable
+45 -11
View File
@@ -1,15 +1,49 @@
#include "nix/fetchers/fetch-settings.hh"
#include "nix/flake/settings.hh"
#include "nix/flake/lockfile.hh"
#include "nix/store/store-api.hh"
#include "nix/util/strings.hh"
#include <boost/unordered/unordered_flat_set.hpp>
#include <nlohmann/json.hpp>
#include <assert.h>
#include <boost/unordered/unordered_flat_set_fwd.hpp>
#include <nlohmann/detail/iterators/iter_impl.hpp>
#include <nlohmann/detail/iterators/iteration_proxy.hpp>
#include <nlohmann/json_fwd.hpp>
#include <algorithm>
#include <iomanip>
#include <boost/unordered/unordered_flat_set.hpp>
#include <iterator>
#include <nlohmann/json.hpp>
#include <compare>
#include <ctime>
#include <format>
#include <functional>
#include <map>
#include <memory>
#include <optional>
#include <ostream>
#include <regex>
#include <set>
#include <string>
#include <string_view>
#include <utility>
#include <variant>
#include <vector>
#include "nix/fetchers/fetch-settings.hh"
#include "nix/flake/lockfile.hh"
#include "nix/util/strings.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/flake/flakeref.hh"
#include "nix/store/path.hh"
#include "nix/util/ansicolor.hh"
#include "nix/util/configuration.hh"
#include "nix/util/error.hh"
#include "nix/util/fmt.hh"
#include "nix/util/hash.hh"
#include "nix/util/logging.hh"
#include "nix/util/ref.hh"
#include "nix/util/types.hh"
#include "nix/util/util.hh"
namespace nix {
class Store;
} // namespace nix
namespace nix::flake {
@@ -43,8 +77,8 @@ LockedNode::LockedNode(const fetchers::Settings & fetchSettings, const nlohmann:
if (!lockedRef.input.isLocked() && !lockedRef.input.isRelative()) {
if (lockedRef.input.getNarHash())
warn(
"Lock file entry '%s' is unlocked (e.g. lacks a Git revision) but does have a NAR hash. "
"This is deprecated since such inputs are verifiable but may not be reproducible.",
"Lock file entry '%s' is unlocked (e.g. lacks a Git revision) but is checked by NAR hash. "
"This is not reproducible and will break after garbage collection or when shared.",
lockedRef.to_string());
else
throw Error(
-1
View File
@@ -29,7 +29,6 @@ nlohmann_json = dependency('nlohmann_json', version : '>= 3.9')
deps_public += nlohmann_json
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
subdir('nix-meson-build-support/generate-header')
+4
View File
@@ -1,5 +1,9 @@
#include <vector>
#include "nix/flake/settings.hh"
#include "nix/flake/flake-primops.hh"
#include "nix/expr/eval-settings.hh"
#include "nix/expr/eval.hh"
namespace nix::flake {
+6 -2
View File
@@ -1,6 +1,10 @@
#include "nix/flake/url-name.hh"
#include <regex>
#include <iostream>
#include <optional>
#include <string>
#include "nix/flake/url-name.hh"
#include "nix/util/strings.hh"
#include "nix/util/url.hh"
namespace nix {
-1
View File
@@ -28,7 +28,6 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_main.cc',
-1
View File
@@ -53,7 +53,6 @@ config_priv_h = configure_file(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'common-args.cc',
+27 -22
View File
@@ -320,29 +320,34 @@ int handleExceptions(const std::string & programName, std::function<void()> fun)
std::string error = ANSI_RED "error:" ANSI_NORMAL " ";
try {
try {
fun();
} catch (...) {
/* Subtle: we have to make sure that any `interrupted'
condition is discharged before we reach printMsg()
below, since otherwise it will throw an (uncaught)
exception. */
setInterruptThrown();
throw;
try {
fun();
} catch (...) {
/* Subtle: we have to make sure that any `interrupted'
condition is discharged before we reach printMsg()
below, since otherwise it will throw an (uncaught)
exception. */
setInterruptThrown();
throw;
}
} catch (Exit & e) {
return e.status;
} catch (UsageError & e) {
logError(e.info());
printError("Try '%1% --help' for more information.", programName);
return 1;
} catch (BaseError & e) {
logError(e.info());
return e.info().status;
} catch (std::bad_alloc & e) {
printError(error + "out of memory");
return 1;
} catch (std::exception & e) {
printError(error + e.what());
return 1;
}
} catch (Exit & e) {
return e.status;
} catch (UsageError & e) {
logError(e.info());
printError("Try '%1% --help' for more information.", programName);
return 1;
} catch (BaseError & e) {
logError(e.info());
return e.info().status;
} catch (std::bad_alloc & e) {
printError(error + "out of memory");
return 1;
} catch (std::exception & e) {
printError(error + e.what());
} catch (...) {
/* In case logger also throws just give up. */
return 1;
}
-1
View File
@@ -26,7 +26,6 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_store.cc',
-1
View File
@@ -29,7 +29,6 @@ rapidcheck = dependency('rapidcheck')
deps_public += rapidcheck
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'derived-path.cc',
+23 -21
View File
@@ -112,32 +112,34 @@ CHARACTERIZATION_TEST(
"realisation",
(std::tuple<Realisation, Realisation>{
Realisation{
.id =
DrvOutput{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
},
DrvOutput{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
},
Realisation{
.id =
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.dependentRealisations =
{
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
.signatures = {"asdf", "qwer"},
.dependentRealisations =
{
DrvOutput{
.drvHash = Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "quux",
{
DrvOutput{
.drvHash = Hash::parseSRI("sha256-b4afnqKCO9oWXgYHb9DeQ2berSwOjS27rSd9TxXDc/U="),
.outputName = "quux",
},
StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo"},
},
},
},
{
.drvHash = Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
.outputName = "baz",
},
},
}))
+39 -39
View File
@@ -14,7 +14,7 @@
namespace nix {
using nlohmann::json;
using namespace nlohmann;
class DerivationAdvancedAttrsTest : public CharacterizationTest, public LibStoreTest
{
@@ -51,44 +51,44 @@ using BothFixtures = ::testing::Types<DerivationAdvancedAttrsTest, CaDerivationA
TYPED_TEST_SUITE(DerivationAdvancedAttrsBothTest, BothFixtures);
#define TEST_ATERM_JSON(STEM, NAME) \
TYPED_TEST(DerivationAdvancedAttrsBothTest, Derivation_##STEM##_from_json) \
{ \
this->readTest(NAME ".json", [&](const auto & encoded_) { \
auto encoded = json::parse(encoded_); \
/* Use DRV file instead of C++ literal as source of truth. */ \
auto aterm = readFile(this->goldenMaster(NAME ".drv")); \
auto expected = parseDerivation(*this->store, std::move(aterm), NAME, this->mockXpSettings); \
Derivation got = Derivation::fromJSON(encoded, this->mockXpSettings); \
EXPECT_EQ(got, expected); \
}); \
} \
\
TYPED_TEST(DerivationAdvancedAttrsBothTest, Derivation_##STEM##_to_json) \
{ \
this->writeTest( \
NAME ".json", \
[&]() -> json { \
/* Use DRV file instead of C++ literal as source of truth. */ \
auto aterm = readFile(this->goldenMaster(NAME ".drv")); \
return parseDerivation(*this->store, std::move(aterm), NAME, this->mockXpSettings).toJSON(); \
}, \
[](const auto & file) { return json::parse(readFile(file)); }, \
[](const auto & file, const auto & got) { return writeFile(file, got.dump(2) + "\n"); }); \
} \
\
TYPED_TEST(DerivationAdvancedAttrsBothTest, Derivation_##STEM##_from_aterm) \
{ \
this->readTest(NAME ".drv", [&](auto encoded) { \
/* Use JSON file instead of C++ literal as source of truth. */ \
auto json = json::parse(readFile(this->goldenMaster(NAME ".json"))); \
auto expected = Derivation::fromJSON(json, this->mockXpSettings); \
auto got = parseDerivation(*this->store, std::move(encoded), NAME, this->mockXpSettings); \
EXPECT_EQ(got.toJSON(), expected.toJSON()); \
EXPECT_EQ(got, expected); \
}); \
} \
\
#define TEST_ATERM_JSON(STEM, NAME) \
TYPED_TEST(DerivationAdvancedAttrsBothTest, Derivation_##STEM##_from_json) \
{ \
this->readTest(NAME ".json", [&](const auto & encoded_) { \
auto encoded = json::parse(encoded_); \
/* Use DRV file instead of C++ literal as source of truth. */ \
auto aterm = readFile(this->goldenMaster(NAME ".drv")); \
auto expected = parseDerivation(*this->store, std::move(aterm), NAME, this->mockXpSettings); \
Derivation got = adl_serializer<Derivation>::from_json(encoded, this->mockXpSettings); \
EXPECT_EQ(got, expected); \
}); \
} \
\
TYPED_TEST(DerivationAdvancedAttrsBothTest, Derivation_##STEM##_to_json) \
{ \
this->writeTest( \
NAME ".json", \
[&]() -> json { \
/* Use DRV file instead of C++ literal as source of truth. */ \
auto aterm = readFile(this->goldenMaster(NAME ".drv")); \
return parseDerivation(*this->store, std::move(aterm), NAME, this->mockXpSettings); \
}, \
[](const auto & file) { return json::parse(readFile(file)); }, \
[](const auto & file, const auto & got) { return writeFile(file, got.dump(2) + "\n"); }); \
} \
\
TYPED_TEST(DerivationAdvancedAttrsBothTest, Derivation_##STEM##_from_aterm) \
{ \
this->readTest(NAME ".drv", [&](auto encoded) { \
/* Use JSON file instead of C++ literal as source of truth. */ \
auto j = json::parse(readFile(this->goldenMaster(NAME ".json"))); \
auto expected = adl_serializer<Derivation>::from_json(j, this->mockXpSettings); \
auto got = parseDerivation(*this->store, std::move(encoded), NAME, this->mockXpSettings); \
EXPECT_EQ(static_cast<json>(got), static_cast<json>(expected)); \
EXPECT_EQ(got, expected); \
}); \
} \
\
/* No corresponding write test, because we need to read the drv to write the json file */
TEST_ATERM_JSON(advancedAttributes, "advanced-attributes-defaults");
+14 -25
View File
@@ -66,23 +66,17 @@ TEST_F(DynDerivationTest, BadATerm_oldVersionDynDeps)
FormatError);
}
#define MAKE_OUTPUT_JSON_TEST_P(FIXTURE) \
TEST_P(FIXTURE, from_json) \
{ \
const auto & [name, expected] = GetParam(); \
/* Don't use readJsonTest because we want to check experimental \
features. */ \
readTest(Path{"output-"} + name + ".json", [&](const auto & encoded_) { \
json j = json::parse(encoded_); \
DerivationOutput got = DerivationOutput::fromJSON(j, mockXpSettings); \
ASSERT_EQ(got, expected); \
}); \
} \
\
TEST_P(FIXTURE, to_json) \
{ \
const auto & [name, value] = GetParam(); \
writeJsonTest("output-" + name, value); \
#define MAKE_OUTPUT_JSON_TEST_P(FIXTURE) \
TEST_P(FIXTURE, from_json) \
{ \
const auto & [name, expected] = GetParam(); \
readJsonTest(Path{"output-"} + name, expected, mockXpSettings); \
} \
\
TEST_P(FIXTURE, to_json) \
{ \
const auto & [name, value] = GetParam(); \
writeJsonTest("output-" + name, value); \
}
struct DerivationOutputJsonTest : DerivationTest,
@@ -193,13 +187,7 @@ INSTANTIATE_TEST_SUITE_P(
TEST_P(FIXTURE, from_json) \
{ \
const auto & drv = GetParam(); \
/* Don't use readJsonTest because we want to check experimental \
features. */ \
readTest(drv.name + ".json", [&](const auto & encoded_) { \
auto encoded = json::parse(encoded_); \
Derivation got = Derivation::fromJSON(encoded, mockXpSettings); \
ASSERT_EQ(got, drv); \
}); \
readJsonTest(drv.name, drv, mockXpSettings); \
} \
\
TEST_P(FIXTURE, to_json) \
@@ -213,7 +201,8 @@ INSTANTIATE_TEST_SUITE_P(
const auto & drv = GetParam(); \
readTest(drv.name + ".drv", [&](auto encoded) { \
auto got = parseDerivation(*store, std::move(encoded), drv.name, mockXpSettings); \
ASSERT_EQ(got.toJSON(), drv.toJSON()); \
using nlohmann::json; \
ASSERT_EQ(static_cast<json>(got), static_cast<json>(drv)); \
ASSERT_EQ(got, drv); \
}); \
} \
+50 -24
View File
@@ -3,13 +3,13 @@
#include <gtest/gtest.h>
#include <rapidcheck/gtest.h>
#include "nix/util/tests/characterization.hh"
#include "nix/util/tests/json-characterization.hh"
#include "nix/store/tests/derived-path.hh"
#include "nix/store/tests/libstore.hh"
namespace nix {
class DerivedPathTest : public CharacterizationTest, public LibStoreTest
class DerivedPathTest : public virtual CharacterizationTest, public LibStoreTest
{
std::filesystem::path unitTestData = getUnitTestData() / "derived-path";
@@ -123,25 +123,51 @@ RC_GTEST_FIXTURE_PROP(DerivedPathTest, prop_round_rip, (const DerivedPath & o))
using nlohmann::json;
#define TEST_JSON(TYPE, NAME, VAL) \
static const TYPE NAME = VAL; \
\
TEST_F(DerivedPathTest, NAME##_from_json) \
{ \
readTest(#NAME ".json", [&](const auto & encoded_) { \
auto encoded = json::parse(encoded_); \
TYPE got = static_cast<TYPE>(encoded); \
ASSERT_EQ(got, NAME); \
}); \
} \
\
TEST_F(DerivedPathTest, NAME##_to_json) \
{ \
writeTest( \
#NAME ".json", \
[&]() -> json { return static_cast<json>(NAME); }, \
[](const auto & file) { return json::parse(readFile(file)); }, \
[](const auto & file, const auto & got) { return writeFile(file, got.dump(2) + "\n"); }); \
struct SingleDerivedPathJsonTest : DerivedPathTest,
JsonCharacterizationTest<SingleDerivedPath>,
::testing::WithParamInterface<SingleDerivedPath>
{};
struct DerivedPathJsonTest : DerivedPathTest,
JsonCharacterizationTest<DerivedPath>,
::testing::WithParamInterface<DerivedPath>
{};
#define TEST_JSON(TYPE, NAME, VAL) \
static const TYPE NAME = VAL; \
\
TEST_F(TYPE##JsonTest, NAME##_from_json) \
{ \
readJsonTest(#NAME, NAME); \
} \
\
TEST_F(TYPE##JsonTest, NAME##_to_json) \
{ \
writeJsonTest(#NAME, NAME); \
}
#define TEST_JSON_XP_DYN(TYPE, NAME, VAL) \
static const TYPE NAME = VAL; \
\
TEST_F(TYPE##JsonTest, NAME##_from_json_throws_without_xp) \
{ \
std::optional<json> ret; \
readTest(#NAME ".json", [&](const auto & encoded_) { ret = json::parse(encoded_); }); \
if (ret) { \
EXPECT_THROW(nlohmann::adl_serializer<TYPE>::from_json(*ret), MissingExperimentalFeature); \
} \
} \
\
TEST_F(TYPE##JsonTest, NAME##_from_json) \
{ \
ExperimentalFeatureSettings xpSettings; \
xpSettings.set("experimental-features", "dynamic-derivations"); \
readJsonTest(#NAME, NAME, xpSettings); \
} \
\
TEST_F(TYPE##JsonTest, NAME##_to_json) \
{ \
writeJsonTest(#NAME, NAME); \
}
TEST_JSON(
@@ -156,7 +182,7 @@ TEST_JSON(
.output = "bar",
}));
TEST_JSON(
TEST_JSON_XP_DYN(
SingleDerivedPath,
single_built_built,
(SingleDerivedPath::Built{
@@ -179,7 +205,7 @@ TEST_JSON(
.outputs = OutputsSpec::Names{"bar", "baz"},
}));
TEST_JSON(
TEST_JSON_XP_DYN(
DerivedPath,
multi_built_built,
(DerivedPath::Built{
@@ -191,7 +217,7 @@ TEST_JSON(
.outputs = OutputsSpec::Names{"baz", "quux"},
}));
TEST_JSON(
TEST_JSON_XP_DYN(
DerivedPath,
multi_built_built_wildcard,
(DerivedPath::Built{
+13 -2
View File
@@ -1,6 +1,6 @@
#include <gtest/gtest.h>
#include "nix/store/dummy-store.hh"
#include "nix/store/dummy-store-impl.hh"
#include "nix/store/globals.hh"
#include "nix/store/realisation.hh"
@@ -13,7 +13,7 @@ TEST(DummyStore, realisation_read)
auto store = [] {
auto cfg = make_ref<DummyStoreConfig>(StoreReference::Params{});
cfg->readOnly = false;
return cfg->openStore();
return cfg->openDummyStore();
}();
auto drvHash = Hash::parseExplicitFormatUnprefixed(
@@ -22,6 +22,17 @@ TEST(DummyStore, realisation_read)
auto outputName = "foo";
EXPECT_EQ(store->queryRealisation({drvHash, outputName}), nullptr);
UnkeyedRealisation value{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv"},
};
store->buildTrace.insert({drvHash, {{outputName, make_ref<UnkeyedRealisation>(value)}}});
auto value2 = store->queryRealisation({drvHash, outputName});
ASSERT_TRUE(value2);
EXPECT_EQ(*value2, value);
}
} // namespace nix
+3 -3
View File
@@ -52,7 +52,6 @@ gtest = dependency('gmock')
deps_private += gtest
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'common-protocol.cc',
@@ -84,6 +83,7 @@ sources = files(
'store-reference.cc',
'uds-remote-store.cc',
'worker-protocol.cc',
'write-derivation.cc',
)
include_dirs = [ include_directories('.') ]
@@ -105,7 +105,7 @@ this_exe = executable(
test(
meson.project_name(),
this_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
'HOME' : meson.current_build_dir() / 'test-home',
'NIX_REMOTE' : meson.current_build_dir() / 'test-home' / 'store',
@@ -139,7 +139,7 @@ if get_option('benchmarks')
benchmark(
'nix-store-benchmarks',
benchmark_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
},
)
-1
View File
@@ -83,7 +83,6 @@ mkMesonExecutable (finalAttrs: {
}
(
''
export ASAN_OPTIONS=abort_on_error=1:print_summary=1:detect_leaks=0
export _NIX_TEST_UNIT_DATA=${data + "/src/libstore-tests/data"}
export NIX_REMOTE=$HOME/store
${stdenv.hostPlatform.emulator buildPackages} ${lib.getExe finalAttrs.finalPackage}
+10 -10
View File
@@ -49,16 +49,16 @@ INSTANTIATE_TEST_SUITE_P(
RealisationJsonTest,
([] {
Realisation simple{
.id =
{
.drvHash = Hash::parseExplicitFormatUnprefixed(
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
HashAlgorithm::SHA256,
HashFormat::Base16),
.outputName = "foo",
},
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv"},
{
.outPath = StorePath{"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv"},
},
{
.drvHash = Hash::parseExplicitFormatUnprefixed(
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
HashAlgorithm::SHA256,
HashFormat::Base16),
.outputName = "foo",
},
};
return ::testing::Values(
std::pair{
+113 -6
View File
@@ -1,8 +1,9 @@
#include "nix/store/s3-binary-cache-store.hh"
#include "nix/store/http-binary-cache-store.hh"
#include "nix/store/filetransfer.hh"
#include "nix/store/s3-url.hh"
#if NIX_WITH_S3_SUPPORT
# include <gtest/gtest.h>
#include <gtest/gtest.h>
namespace nix {
@@ -10,9 +11,115 @@ TEST(S3BinaryCacheStore, constructConfig)
{
S3BinaryCacheStoreConfig config{"s3", "foobar", {}};
EXPECT_EQ(config.bucketName, "foobar");
// The bucket name is stored as the host part of the authority in cacheUri
EXPECT_EQ(
config.cacheUri,
(ParsedURL{
.scheme = "s3",
.authority = ParsedURL::Authority{.host = "foobar"},
}));
}
TEST(S3BinaryCacheStore, constructConfigWithRegion)
{
Store::Config::Params params{{"region", "eu-west-1"}};
S3BinaryCacheStoreConfig config{"s3", "my-bucket", params};
EXPECT_EQ(
config.cacheUri,
(ParsedURL{
.scheme = "s3",
.authority = ParsedURL::Authority{.host = "my-bucket"},
.query = (StringMap) {{"region", "eu-west-1"}},
}));
EXPECT_EQ(config.region.get(), "eu-west-1");
}
TEST(S3BinaryCacheStore, defaultSettings)
{
S3BinaryCacheStoreConfig config{"s3", "test-bucket", {}};
EXPECT_EQ(
config.cacheUri,
(ParsedURL{
.scheme = "s3",
.authority = ParsedURL::Authority{.host = "test-bucket"},
}));
// Check default values
EXPECT_EQ(config.region.get(), "us-east-1");
EXPECT_EQ(config.profile.get(), "default");
EXPECT_EQ(config.scheme.get(), "https");
EXPECT_EQ(config.endpoint.get(), "");
}
/**
* Test that S3BinaryCacheStore properly preserves S3-specific parameters
*/
TEST(S3BinaryCacheStore, s3StoreConfigPreservesParameters)
{
StringMap params;
params["region"] = "eu-west-1";
params["endpoint"] = "custom.s3.com";
S3BinaryCacheStoreConfig config("s3", "test-bucket", params);
// The config should preserve S3-specific parameters
EXPECT_EQ(
config.cacheUri,
(ParsedURL{
.scheme = "s3",
.authority = ParsedURL::Authority{.host = "test-bucket"},
.query = (StringMap) {{"region", "eu-west-1"}, {"endpoint", "custom.s3.com"}},
}));
}
/**
* Test that S3 store scheme is properly registered
*/
TEST(S3BinaryCacheStore, s3SchemeRegistration)
{
auto schemes = S3BinaryCacheStoreConfig::uriSchemes();
EXPECT_TRUE(schemes.count("s3") > 0) << "S3 scheme should be supported";
// Verify HttpBinaryCacheStoreConfig doesn't directly list S3
auto httpSchemes = HttpBinaryCacheStoreConfig::uriSchemes();
EXPECT_FALSE(httpSchemes.count("s3") > 0) << "HTTP store shouldn't directly list S3 scheme";
}
/**
* Test that only S3-specific parameters are preserved in cacheUri,
* while non-S3 store parameters are not propagated to the URL
*/
TEST(S3BinaryCacheStore, parameterFiltering)
{
StringMap params;
params["region"] = "eu-west-1";
params["endpoint"] = "minio.local";
params["want-mass-query"] = "true"; // Non-S3 store parameter
params["priority"] = "10"; // Non-S3 store parameter
S3BinaryCacheStoreConfig config("s3", "test-bucket", params);
// Only S3-specific params should be in cacheUri.query
EXPECT_EQ(
config.cacheUri,
(ParsedURL{
.scheme = "s3",
.authority = ParsedURL::Authority{.host = "test-bucket"},
.query = (StringMap) {{"region", "eu-west-1"}, {"endpoint", "minio.local"}},
}));
// But the non-S3 params should still be set on the config
EXPECT_EQ(config.wantMassQuery.get(), true);
EXPECT_EQ(config.priority.get(), 10);
// And all params (S3 and non-S3) should be returned by getReference()
auto ref = config.getReference();
EXPECT_EQ(ref.params["region"], "eu-west-1");
EXPECT_EQ(ref.params["endpoint"], "minio.local");
EXPECT_EQ(ref.params["want-mass-query"], "true");
EXPECT_EQ(ref.params["priority"], "10");
}
} // namespace nix
#endif
+2 -6
View File
@@ -1,10 +1,8 @@
#include "nix/store/s3-url.hh"
#include "nix/util/tests/gmock-matchers.hh"
#if NIX_WITH_S3_SUPPORT || NIX_WITH_CURL_S3
# include <gtest/gtest.h>
# include <gmock/gmock.h>
#include <gtest/gtest.h>
#include <gmock/gmock.h>
namespace nix {
@@ -228,5 +226,3 @@ INSTANTIATE_TEST_SUITE_P(
[](const ::testing::TestParamInfo<S3ToHttpsConversionTestCase> & info) { return info.param.description; });
} // namespace nix
#endif
Loaded 100 of 250 files, more files were not shown because too many files have changed in this diff. Show more