Compare commits

..
Author SHA1 Message Date
John Ericson a4b62a646a nlohmann::json instance and JSON Schema for MemorySourceAccessor
As documented, this for file system objects themselves, since
`MemorySourceAccessor` is an implementation detail.
2025-10-21 13:26:16 -04:00
John Ericson ef8218f2e3 Merge pull request #14307 from NixOS/json-schema-hash
`nlohmann::json` instance and JSON Schema for `Hash`
2025-10-21 06:03:20 +00:00
John Ericson ada008a795 Merge pull request #14310 from obsidiansystems/inline-drv-output-subst-goal
Inline `realisationFetched`
2025-10-21 06:03:06 +00:00
John Ericson 2a2bb8330d Merge pull request #14312 from corngood/develop-structuredAttrs-fix
tests/functional/flakes/develop.sh: Add test for outputChecks stripping
2025-10-20 22:48:39 +00:00
David McFarland 645794b458 tests/functional/flakes/develop.sh: Add test for outputChecks stripping 2025-10-20 19:16:20 -03:00
John Ericson 1121f0d8ec Inline realisationFetched
Now that we are using coroutines, we don't need this to be a separate
method of `DrvOutputSubstitutionGoal`.
2025-10-20 16:45:41 -04:00
Sergei Zimmerman 6420879728 Merge pull request #14296 from lovesegfault/nix-s3-more-tests
fix(nix-prefetch-url): correctly extract filename from URLs with query parameters
2025-10-20 19:42:22 +00:00
Sergei Zimmerman 67f5cb97a3 Merge pull request #14306 from corngood/develop-structuredAttrs-fix
nix/develop: Strip outputChecks when structuredAttrs is enabled
2025-10-20 19:38:19 +00:00
John EricsonandRobert Hensing 5e7ee808de nlohmann::json instance and JSON Schema for Hash
Improving and codifying our experimental JSON interfacing.

Co-Authored-By: Robert Hensing <robert@roberthensing.nl>
2025-10-20 15:21:07 -04:00
John Ericson 270f20a505 Merge pull request #14305 from NixOS/alignment-utils
libutil: Add alignUp helper function, use in archive.cc
2025-10-20 19:08:20 +00:00
Bernardo Meurer Costa 1b1d7e3047 test(nixos): add nix-prefetch-url test for S3 URLs with query parameters
Adds a comprehensive test to verify that `nix-prefetch-url` correctly
handles S3 URLs with query parameters (e.g., custom endpoints and regions).

Previously, nix-prefetch-url would fail with "invalid store
path" errors when given S3 URLs with query parameters like
`?endpoint=http://server:9000&region=eu-west-1`, because it incorrectly
extracted the filename from the query parameters instead of the path.
2025-10-20 21:45:37 +03:00
David McFarland 0f28c76a44 nix/develop: Strip outputChecks when structuredAttrs is enabled 2025-10-20 15:40:05 -03:00
Bernardo Meurer Costa e3b3f05e5d fix(nix-prefetch-url): correctly extract filename from URLs with query parameters
Previously, `prefetchFile()` used `baseNameOf()` directly on the URL string
to extract the filename. This caused issues with URLs containing query
parameters that include slashes, such as S3 URLs with custom endpoints:

```
s3://bucket/file.txt?endpoint=http://server:9000
```

The `baseNameOf()` function naively searches for the rightmost `/` in the
entire string, which would find the `/` in `http://server:9000` and extract
`server:9000&region=...` as the filename. This resulted in invalid store
path names containing illegal characters like `:`.

This commit fixes the issue by:

1. Adding a `VerbatimURL::lastPathSegment()` method that extracts the last
   non-empty path segment from a URL, using `pathSegments(true)` to filter
   empty segments
2. Changing `prefetchFile()` to accept `const VerbatimURL &` and use the new
   `lastPathSegment()` method instead of manual path parsing
3. Adding early validation with `checkName()` to fail quickly on invalid
   filenames
4. Maintains backward compatibility by falling back to `baseNameOf()` for
   unparsable `VerbatimURL`s
2025-10-20 21:40:03 +03:00
John Ericson f05d240222 Merge pull request #14278 from obsidiansystems/adl-serializer-xp
Cleanup and JSON serializer and XP feature interations
2025-10-20 18:22:21 +00:00
Sergei Zimmerman 22c73868c3 libutil/archive: Use alignUp
With this change it's much more apparent what's going on.
2025-10-20 21:15:11 +03:00
Sergei Zimmerman a91b787524 libutil: Add alignUp helper function 2025-10-20 21:11:00 +03:00
Eelco Dolstra ddf7de0a76 Merge pull request #14291 from NixOS/skip-source
Add skip() method to Source interface to allow efficient seeks
2025-10-20 15:04:36 +00:00
Sergei Zimmerman 1fabed18b6 Merge pull request #14301 from NixOS/s3-terminate-unknown-profile
libstore: Fix reentrancy in AwsCredentialProviderImpl::getCredentialsRaw
2025-10-20 14:28:16 +00:00
Eelco Dolstra 6c9083db2c Use a smaller buffer 2025-10-20 13:40:19 +02:00
Sergei Zimmerman c663f7ec79 libstore: Fix reentrancy in AwsCredentialProviderImpl::getCredentialsRaw
Old code would do very much incorrect reentrancy crimes (trying to do an
erase inside the emplace callback). This would fail miserably with an assertion
in Boost:

terminating due to unexpected unrecoverable internal error: Assertion '(!find(px))&&("reentrancy not allowed")' failed in boost::unordered::detail::foa::entry_trace::entry_trace(const void *) at include/boost/unordered/detail/foa/reentrancy_check.hpp:33

This is trivially reproduced by using any S3 URL with a non-empty profile:

nix-prefetch-url "s3://happy/crash?profile=default"
2025-10-19 21:03:13 +03:00
Sergei Zimmerman d0fb03c35d Merge pull request #14282 from NixOS/s3-cleanup
Simplify meson for S3 support via aws-crt-cpp
2025-10-19 17:00:46 +00:00
Sergei Zimmerman c847cd87f1 Merge pull request #14297 from lovesegfault/nix-s3-test-public
test(nixos/s3-binary-cache-store): misc improvements
2025-10-19 16:53:40 +00:00
tomberek dbbdae926b Merge pull request #14299 from roberth/unlocked-msg
Clarify unlocked input warning message
2025-10-19 16:50:16 +00:00
Eelco Dolstra 3c03050cd6 Merge pull request #14290 from NixOS/dont-write-nar-to-tty
nix store dump-path: Refuse to write NARs to the terminal
2025-10-19 12:41:55 +00:00
Robert Hensing e33cd5aa38 Clarify unlocked input warning message
The previous message was vague about what "deprecated" meant and why
unlocked inputs with NAR hashes "may not be reproducible". It also
used "verifiable" which was confusing.

The new message makes it clear that the NAR hash provides verification
(is checked by NAR hash) and explicitly states the failure modes:
garbage collection and sharing.
2025-10-19 14:08:34 +02:00
Bernardo Meurer Costa d9c808f8a7 refactor(tests/nixos/s3-binary-cache-store): add verify_packages_in_store helper 2025-10-19 00:21:54 +00:00
Bernardo Meurer Costa 55ea3d3476 test(tests/nixos/s3-binary-cache-store): test public bucket operations
Add `test_public_bucket_operations` to validate that store operations
work correctly on public S3 buckets without requiring credentials.
Tests nix store info and nix copy operations.
2025-10-19 00:04:33 +00:00
Bernardo Meurer Costa 7d0c06f921 feat(tests/nixos/s3-binary-cache-store): add public parameter to setup_s3
Add optional 'public' parameter to setup_s3 decorator. When set to True,
the bucket will be made publicly accessible using mc anonymous set.
2025-10-18 23:57:51 +00:00
Bernardo Meurer Costa 5b4bd5bcb8 refactor(tests/nixos/s3-binary-cache-store): inline make_http_url fn
Remove make_http_url helper function and inline its single usage.
2025-10-18 23:51:44 +00:00
Bernardo Meurer Costa 4ae6c65bc5 test(tests/nixos/s3-binary-cache-store): verify credential caching in concurrent fetches
Add assertion to test_concurrent_fetches to verify that only one
credential provider is created even with 5 concurrent fetches.
2025-10-18 23:48:55 +00:00
Bernardo Meurer Costa 4f19e63a8f refactor(tests/nixos/s3-binary-cache-store): add --no-link to nix build commands
Prevent creation of result symlinks in all nix build commands by
adding the --no-link flag.
2025-10-18 23:44:13 +00:00
Bernardo Meurer Costa f88c3055f8 refactor(tests/nixos/s3-binary-cache-store): clean client store in setup_s3
Add cleanup of client store in the finally block of setup_s3 decorator.
Uses `nix store delete --ignore-liveness` to properly handle GC roots
and only attempts deletion if the path exists.
2025-10-18 23:36:48 +00:00
Bernardo Meurer Costa 9058d90ab2 refactor(tests/nixos/s3-binary-cache-store): rename populate_with to populate_bucket 2025-10-18 23:27:03 +00:00
Bernardo Meurer Costa c1a15d1a26 refactor(tests/nixos/s3-binary-cache-store): rename with_test_bucket to setup_s3 2025-10-18 23:24:30 +00:00
Bernardo Meurer Costa 22f4cccc71 refactor(tests/nixos/s3-binary-cache-store): use a PKGS dict
Replace individual PKG_A, PKG_B, and PKG_C variables with a PKGS
dictionary. This will enable `@with_clean_client_store` in the future.
2025-10-18 23:23:50 +00:00
John Ericson b56e456b0d Merge pull request #14269 from roberth/json-schema
Add a JSON Schema for `Derivation`
2025-10-18 18:50:39 +00:00
Robert Hensing c92ba4b9b7 Add titles in JSON schemas
This way, the description isn't rendered in the tables of contents,
leading to no more formatting errors.
2025-10-17 21:53:29 +02:00
Eelco Dolstra 67bffa19a5 NullFileSystemObjectSink: Skip over file contents 2025-10-17 20:44:02 +02:00
Eelco Dolstra daa7e0d2e9 Source: Add skip() method
This allows FdSource to efficiently skip data we don't care about.
2025-10-17 20:41:33 +02:00
Eelco Dolstra 109f6449cc nix store dump-path: Refuse to write NARs to the terminal 2025-10-17 20:27:10 +02:00
John Ericson ad2360c59f Merge pull request #14288 from lovesegfault/repl-skip-stack
fix(tests/functional/repl): skip test if stack size limit is insufficient
2025-10-17 17:35:52 +00:00
Bernardo Meurer Costa 20c7c551bf fix(tests/functional/repl): skip test if stack size limit is insufficient
Nix attempts to set the stack size to 64 MB during initialization, which is
required for the repl tests to run successfully. Skip the tests on systems
where the hard stack limit is less than this value rather than failing.
2025-10-17 17:05:12 +00:00
John Ericson e78e6ca4f4 Merge pull request #14281 from NixOS/dead-code
libutil: Drop unused SubdirSourceAccessor
2025-10-17 03:01:17 +00:00
John Ericson e34063cf21 Merge pull request #14283 from NixOS/nar-check
nix {cat,ls}: Add back missing checks for file descriptors
2025-10-17 02:58:23 +00:00
Sergei Zimmerman e457ea7688 nix {cat,ls}: Add back missing checks for file descriptors
I didn't catch this during the review of https://github.com/NixOS/nix/pull/14273.
This fixes that mistake.
2025-10-17 02:26:24 +03:00
Farid Zakaria 64c55961eb Merge pull request #14273 from fzakaria/fzakaria/issue-13944
Make `nix nar [cat|ls]` lazy
2025-10-17 02:16:54 +03:00
Sergei Zimmerman ffbc33fec6 libstore/meson: Rename curl-s3-store to s3-aws-auth
We now unconditionally compile support for s3:// URLs and stores
without authentication. The whole curl version check can be greatly
simplified by the previous commit, which bumps the minimum required curl
version.
2025-10-17 01:18:46 +03:00
Sergei Zimmerman a80fc252e8 libstore/meson: Require curl >= 7.75.0
This version has been released a long time ago in 2021 and it's doubtful
that anybody actually uses it still, since it's full of vulnerabilities [^]

[^]: https://curl.se/docs/vuln-7.75.0.html
2025-10-17 01:18:14 +03:00
Sergei Zimmerman bcd5a9d05c libutil: Drop unused SubdirSourceAccessor 2025-10-17 00:56:53 +03:00
Robert Hensing 01b001d5ba Add JSON Schema infrastructure, use for Derivation
For manual, and testing formats
2025-10-16 17:24:18 -04:00
John Ericson 27767a6094 Merge pull request #14276 from NixOS/fix-14193
libstore/registerOutputs: Don't try to optimize a non-existent actual…
2025-10-16 21:06:43 +00:00
John Ericson 1177d65094 Properly check xp features when deserializing deriving paths 2025-10-16 16:45:22 -04:00
John Ericson a2c6f38e1f Remove now-redundant methods for JSON on Derivation 2025-10-16 16:45:22 -04:00
John Ericson 1c02dd5b9c Allow for standard nlohmann JSON serializers to take separate XP features
I realized that we can actually do this thing, even though it is not
what nlohmann expects at all, because the extra parameter has a default
argument so nlohmann doesn't need to care. Sneaky!
2025-10-16 16:45:22 -04:00
Sergei Zimmerman 4cbcaad435 libstore/registerOutputs: Don't try to optimize a non-existent actualPath
Since 3c610df550 this resulted in `getting status of`
errors on paths inside the chroot if a path was already valid. Careful inspection
of the logic shows that if buildMode != bmCheck actualPath gets reassigned to
store.toRealPath(finalDestPath). The only branch that cares about actualPath is
the buildMode == bmCheck case, which doesn't lead to optimisePath anyway.
2025-10-16 23:08:30 +03:00
John Ericson d87a06af7a Merge pull request #14275 from NixOS/s3-cleanup
libstore: Miscellaneous S3 store cleanups
2025-10-16 19:36:59 +00:00
Eelco Dolstra 2dc9f2a2b7 Merge pull request #14272 from NixOS/use-store-path-serializer
Daemon protocol: Use the WorkerProto serializer for store paths
2025-10-16 19:35:25 +00:00
Eelco Dolstra a7991d55cc Merge pull request #14270 from NixOS/use-optional-storepath-serializer
Use serializer for std::optional<StorePath>
2025-10-16 19:07:07 +00:00
Sergei Zimmerman e7047fde25 libstore: Remove the unnecessary 'error: ' prefix in warning message 2025-10-16 21:49:38 +03:00
Sergei Zimmerman 33e94fe19f libstore: Make AwsAuthError more legible
Instead of the cryptic:

> error: Failed to resolve AWS credentials: error code 6153`

We now get more legible:

> error: AWS authentication error: 'Valid credentials could not be sourced by the IMDS provider' (6153)
2025-10-16 21:49:37 +03:00
Sergei Zimmerman dc03c6a812 libstore: Put all the AWS credentials logic behind interface class AwsCredentialProvider
This makes it so we don't need to rely on global variables and hacky destructors to
clean up another global variable. Just putting it in the correct order in the class
is more than enough.
2025-10-16 21:49:36 +03:00
Sergei Zimmerman b1d067c9bb tests/nixos: Rename back S3 store nixos test 2025-10-16 21:49:35 +03:00
Eelco Dolstra d782c5e586 Daemon protocol: Use the WorkerProto serializer for store paths 2025-10-16 17:34:33 +02:00
Eelco Dolstra f84b33644c Merge pull request #14271 from NixOS/no-check-sigs
Factor out `--no-check-sigs` into its own class
2025-10-16 15:07:29 +00:00
Eelco Dolstra 3bd2b76f6e nix store sign: Use required attribute 2025-10-16 16:35:13 +02:00
Eelco Dolstra 139df77440 Factor out --no-check-sigs 2025-10-16 16:35:09 +02:00
Eelco Dolstra a48a737517 Use serializer for std::optional<StorePath> 2025-10-16 16:32:18 +02:00
John Ericson 0503a862ef Merge pull request #14268 from roberth/dev-doc-manual
doc/dev/doc: Update local build instructions for manual
2025-10-16 13:50:58 +00:00
Robert Hensing 61cb9c4832 doc/dev/doc: Update local build instructions for manual 2025-10-16 13:22:22 +02:00
John Ericson 721f5572a6 Merge pull request #14263 from NixOS/hydra-import-paths
Restore `ServeProto::Command::ImportPaths`
2025-10-15 22:57:50 +00:00
John Ericson 5a6864c027 Merge pull request #14264 from xokdvium/fix-splicing-test
tests: Fix splicing in functional tests for nix-cli
2025-10-15 22:32:45 +00:00
Sergei Zimmerman 0deb492b3d Restore ServeProto::Command::ImportPaths
This partially reverts commit 5e46df973f,
partially reversing changes made to
8c789db05b.

We do this because Hydra, while using the newer version of the protocol,
still uses this command, even though Nix (as a client) doesn't use it.
On that basis, we don't want to remove it (or consider it only part of
the older versions of the protocol) until Hydra no longer uses the
Legacy SSH Protocol.
2025-10-15 18:18:59 -04:00
Sergei Zimmerman 17b7fb383f tests: Fix splicing in functional tests for nix-cli
This is necessary to fix nix-everything-llvm.
The problem here is that nix-cli is taken from the previous
stage that is built with libstdc++, but this derivation builds
plugins with libc++ and the plugin load fails miserably.
2025-10-16 01:04:50 +03:00
John Ericson 94cfba7e84 Merge pull request #14226 from obsidiansystems/unkeyed-realisation
Reapply #14097
2025-10-15 21:27:13 +00:00
Sergei Zimmerman 0f1cfa4d60 Merge pull request #14262 from lovesegfault/ci-cleanup-s3
ci: cleanup s3 tests
2025-10-15 21:19:20 +00:00
Bernardo Meurer Costa fa0d00e668 ci: cleanup s3 tests
This cleans up the work done in 8c2828387. Now that #13752 has landed,
there's no need to test configurations without AWS auth in CI.
2025-10-15 23:51:08 +03:00
Sergei Zimmerman d2b6499154 Merge pull request #13752 from lovesegfault/curl-based-s3-v2
feat(libstore): curl-based s3
2025-10-15 20:01:26 +00:00
Sergei Zimmerman e3232af558 Merge pull request #14253 from NixOS/libgit2-refname-wa
libfetchers/git-utils: Be more correct about validating refnames
2025-10-15 19:30:53 +00:00
Bernardo Meurer Costa e069c9892e docs(rl-next): add notes for curl-based s3 2025-10-15 19:26:53 +00:00
John Ericson 266fbebe66 Implement realisation operations on dummy store 2025-10-15 14:59:08 -04:00
John EricsonandSergei Zimmerman 6995d325ef Split out UnkeyedRealisation from Realisation
Realisations are conceptually key-value pairs, mapping `DrvOutputs` (the
key) to information about that derivation output.

This separate the value type, which will be useful in maps, etc., where
we don't want to denormalize by including the key twice.

This matches similar changes for existing types:

| keyed              | unkeyed                |
|--------------------|------------------------|
| `ValidPathInfo`    | `UnkeyedValidPathInfo` |
| `KeyedBuildResult` | `BuildResult`          |
| `Realisation`      | `UnkeyedRealisation`   |

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-15 14:59:04 -04:00
Sergei Zimmerman 5d1178b817 libfetchers/git-utils: Be more correct about validating refnames
Turns out there's a much better API for this that doesn't have the
footguns of the previous method.

isLegalRefName is somewhat of a misnomer, since it's mainly used to
validate user inputs that can be either references, branch names,
psedorefs or tags.
2025-10-15 21:54:09 +03:00
Bernardo Meurer Costa 3224636ab0 refactor(libstore): rename NIX_WITH_S3_SUPPORT to NIX_WITH_AWS_AUTH
The macro now accurately reflects its purpose: gating only AWS
authentication code, not all S3 functionality. S3 URL parsing, store
configuration, and public bucket access work regardless of this flag.

This rename clarifies that:
- S3 support is always available (URL parsing, store registration)
- Only AWS credential resolution requires the flag
- The flag controls AWS CRT SDK dependency, not S3 protocol support
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa bb1f22a8df refactor(libstore): minimize NIX_WITH_S3_SUPPORT scope to auth only
Move S3 URL parsing, store configuration, and public bucket support
outside of NIX_WITH_S3_SUPPORT guards. Only AWS credential resolution
remains gated, allowing builds with withAWS = false to:

- Parse s3:// URLs
- Register S3 store types
- Access public S3 buckets (via HTTPS conversion)
- Use S3-compatible services without authentication

The setupForS3() function now always performs URL conversion, with
authentication code conditionally compiled based on NIX_WITH_S3_SUPPORT.
The aws-creds.cc file (only code using AWS CRT SDK) is now conditionally
compiled by meson.
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa 1f710300c9 refactor(libstore): withCurlS3 -> withAWS
Now that the legacy S3 implementation is gone, we can go back to calling
things `NIX_WITH_S3_SUPPORT`.
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa 9295c14a35 refactor(libstore): replace AWS SDK with curl-based S3 implementation
This commit replaces the AWS C++ SDK with a lighter curl-based approach
for S3 binary cache operations.

- Removed dependency on the heavy aws-cpp-sdk-s3 and aws-cpp-sdk-transfer
- Added lightweight aws-crt-cpp for credential resolution only
- Leverages curl's native AWS SigV4 authentication (requires curl >= 7.75.0)
- S3BinaryCacheStore now delegates to HttpBinaryCacheStore
- Function s3ToHttpsUrl converts ParsedS3URL to ParsedURL
- Multipart uploads are no longer supported (may be reimplemented later)
- Build now requires curl >= 7.75.0 for AWS SigV4 support

Fixes: #13084, #12671, #11748, #12403, #5947
2025-10-15 18:23:55 +00:00
John Ericson a543519ca9 Merge pull request #14257 from obsidiansystems/misc-builder-tech-debt
Cleanup misc builder tech debt
2025-10-15 17:28:27 +00:00
John Ericson 632ccfb8c0 Remove dead outputPaths variable. 2025-10-15 12:16:53 -04:00
John Ericson 46357468a4 Remove unused parameters to DrvOutputSubstitutionGoal 2025-10-15 12:16:52 -04:00
John Ericson b20cebf993 Remove unused typedef and field 2025-10-15 12:15:23 -04:00
Eelco Dolstra 5700112127 Merge pull request #14205 from GrahamDennis/gdennis/improve-dir-url-backcompat
Improved backwards compatibility hack for git URLs using dir=...
2025-10-15 15:20:02 +00:00
John Ericson 606eb1dfb5 Merge pull request #14250 from fzakaria/patch-1
Remove duplicate shellcheck in dev-shell.nix
2025-10-15 05:03:19 +00:00
John Ericson e07754d888 Merge pull request #14251 from fzakaria/fzakaria/iwyu-libflake
Clean-up libflake headers
2025-10-15 04:27:07 +00:00
Farid Zakaria 01a8499d2f Format cpp files 2025-10-14 23:51:40 -04:00
Farid Zakaria e8b126fa90 Remove unecessary includes 2025-10-14 23:48:45 -04:00
Farid Zakaria 902faf4fe5 More fixes for iwyu 2025-10-14 23:20:35 -04:00
Farid Zakaria 7bc3d9b9a9 First attempt at uwyu for libflake 2025-10-14 22:53:13 -04:00
Farid Zakaria 092639709f Remove duplicate shellcheck in dev-shell.nix 2025-10-14 19:25:06 -07:00
John Ericson 620091bc8b Merge pull request #14223 from lovesegfault/curl-based-s3-tests
test(nixos): add comprehensive curl-based S3 VM tests
2025-10-14 23:08:55 +00:00
John Ericson 6dcc468253 Merge pull request #14249 from NixOS/more-to-real-path-cleanups
More toRealPath cleanups
2025-10-14 22:46:15 +00:00
Sergei Zimmerman 0347958dd2 nix/develop: Remove usage of toRealPath, replace with SourceAccessor 2025-10-15 00:52:13 +03:00
Sergei Zimmerman 918a3cebaa libexpr: Use Store::requireStoreObjectAccessor instead or toRealPath in fetch
This forces the code to go through proper abstractions instead of the raw filesystem
API.

This issue is evident from this reproducer:

nix eval --expr 'builtins.fetchurl { url = "https://example.com"; sha256 = ""; }' --json --eval-store "dummy://?read-only=false"

error:
       … while calling the 'fetchurl' builtin
         at «string»:1:1:
            1| builtins.fetchurl { url = "https://example.com"; sha256 = ""; }
             | ^

       error: opening file '/nix/store/r4f87yrl98f2m6v9z8ai2rbg4qwlcakq-example.com': No such file or directory
2025-10-15 00:27:41 +03:00
Sergei Zimmerman 69c005e805 libstore: Use getFSAccessor for store object in Worker::pathContentsGood
We only care about the accessor for a single store object anyway, but
the validity gets ignored. Also `pathExists(store.printStorePath(path))`
is definitely incorrect since it confuses the logical location vs physical
location in case of a chroot store.
2025-10-15 00:15:50 +03:00
Sergei Zimmerman 0c32fb3fa2 treewide: Add Store::requireStoreObjectAccessor, simplify uses of getFSAccessor
This is a simple wrapper around getFSAccessor that throws an InvalidPath
error. This simplifies usage in callsites that only care about getting
a non-null accessor.
2025-10-14 23:58:20 +03:00
Bernardo Meurer Costa d18f959d4f test(nixos): add comprehensive curl-based S3 VM tests
Add `curl-s3-binary-cache-store.nix` with comprehensive test coverage
for the curl-based S3 implementation.

Depends-On: #14206, #14222
2025-10-14 20:55:14 +00:00
Sergei Zimmerman 4041bfdb40 Merge pull request #14206 from lovesegfault/curl-based-s3-pieces
feat(libstore): add builtin fetchurl S3 credential pre-resolution
2025-10-14 20:10:41 +00:00
John Ericson 1fb4ff8c0e Merge pull request #14232 from roberth/dyndrv-messages
Better dyndrv messages
2025-10-14 15:40:27 +00:00
Robert Hensing 1b96a704d3 Add lazy evaluation for experimental feature reasons
Wrap fmt() calls in lambdas to defer string formatting until the
feature check fails. This avoids unnecessary string formatting in
the common case where the feature is enabled.

Addresses performance concern raised by xokdvium in PR review.
2025-10-14 16:49:59 +02:00
John Ericson 959c244a12 Merge pull request #14243 from NixOS/canon-path-nul-bytes
libutil: Ensure that CanonPath does not contain NUL bytes
2025-10-14 14:30:24 +00:00
Eelco Dolstra c44d2d5913 Merge pull request #14241 from NixOS/dependabot/github_actions/actions/create-github-app-token-2
build(deps): bump actions/create-github-app-token from 1 to 2
2025-10-14 11:55:43 +00:00
Eelco Dolstra dd590eca74 Merge pull request #14242 from NixOS/dependabot/github_actions/actions/checkout-5
build(deps): bump actions/checkout from 4 to 5
2025-10-14 11:55:25 +00:00
Sergei Zimmerman 1633ceaff2 libutil: Ensure that CanonPath does not contain NUL bytes
This, alongside the other invariants of the CanonPath is important
to uphold. std::filesystem happily crashes on NUL bytes in the constructor,
as we've seen with `path:%00` prior to c436b7a32a.
Best to stay clear of NUL bytes when we're talking about syscalls, especially
on Unix where strings are null terminated.

Very nice to have if we decide to switch over to pascal-style strings.
2025-10-14 02:33:42 +03:00
John Ericson 16e946bfb1 Merge pull request #14225 from obsidiansystems/derivation-resolution-goal-2
Reapply the rest of #14022
2025-10-13 23:26:29 +00:00
Sergei Zimmerman edf9163c22 libutil: Make CanonPath::root const
By all means CanonPath::root must be immutable. Let's enforce this with
in the code.
2025-10-14 02:24:40 +03:00
John Ericson ad893acf46 Fix ca/eval-store.sh test
The refactor in the last commit fixed the bug it was supposed to fix,
but introduced a new bug in that sometimes we tried to write a resolved
derivation to a store before all its `inputSrcs` were in that store.

The solution is to defer writing the derivation until inside
`DerivationBuildingGoal`, just before we do an actual build. At this
point, we are sure that all inputs in are the store.

This does have the side effect of meaning we don't write down the
resolved derivation in the substituting case, only the building case,
but I think that is actually fine. The store that actually does the
building should make a record of what it built by storing the resolved
derivation. Other stores that just substitute from that store don't
necessary want that derivation however. They can trust the substituter
to keep the record around, or baring that, they can attempt to re
resolve everything, if they need to be audited.

(cherry picked from commit c97b050a6c)
2025-10-13 18:41:59 -04:00
John Ericson 06bb1c2f93 Remove some buildMode default parameters
Force the internals to be more explicit.
2025-10-13 18:40:10 -04:00
John Ericson 2ee41976c2 Fix #13247
Resolve the derivation before creating a building goal, in a context
where we know what output(s) we want. That way we have a chance just to
download the outputs we want.

Fix #13247

(cherry picked from commit 39f6fd9b46)
2025-10-13 18:37:14 -04:00
dependabot[bot] b846f27682 build(deps): bump actions/checkout from 4 to 5
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-10-13 22:00:59 +00:00
dependabot[bot] 962862e9e0 build(deps): bump actions/create-github-app-token from 1 to 2
Bumps [actions/create-github-app-token](https://github.com/actions/create-github-app-token) from 1 to 2.
- [Release notes](https://github.com/actions/create-github-app-token/releases)
- [Commits](https://github.com/actions/create-github-app-token/compare/v1...v2)

---
updated-dependencies:
- dependency-name: actions/create-github-app-token
  dependency-version: '2'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-10-13 22:00:55 +00:00
Robert Hensing 39c4665488 Store reason as a field in MissingExperimentalFeature
Store the reason string as a field in the exception class rather than
only embedding it in the error message. This supports better structured
error handling and future JSON error reporting.

Suggested by Ericson2314 in PR review.
2025-10-13 23:49:20 +02:00
Robert Hensing 71aa9a4798 Add reasons to dyndrv xp messages 2025-10-13 23:49:20 +02:00
Robert Hensing 0fd890a8d6 Add reason string support to MissingExperimentalFeature 2025-10-13 23:49:20 +02:00
John Ericson 6642ffb506 Merge pull request #14239 from NixOS/asan-stack-overflow
libstore/outputs-spec: Drop usage of std::regex
2025-10-13 21:44:49 +00:00
Sergei Zimmerman 3ba221025f libstore/outputs-spec: Drop usage of std::regex
std::regex is a really bad tool for parsing things, since
it tends to overflow the stack pretty badly. See the build failure
under ASan in [^].

[^]: https://hydra.nixos.org/build/310077167/nixlog/5
2025-10-13 23:58:20 +03:00
Eelco Dolstra b56cc1808d Merge pull request #14237 from NixOS/url-parser-regression
Remove validation of URLs passed to FileTransferRequest verbatim
2025-10-13 20:01:01 +00:00
Sergei Zimmerman 47f427a172 Remove validation of URLs passed to FileTransferRequest verbatim
CURL is not very strict about validation of URLs passed to it. We
should reflect this in our handling of URLs that we get from the user
in <nix/fetchurl.nix> or builtins.fetchurl. ValidURL was an attempt to
rectify this, but it turned out to be too strict. The only good way to
resolve this is to pass (in some cases) the user-provided string verbatim
to CURL. Other usages in libfetchers still benefit from using structured
ParsedURL and validation though.

nix store prefetch-file --name foo 'https://cdn.skypack.dev/big.js@^5.2.2'
error: 'https://cdn.skypack.dev/big.js@^5.2.2' is not a valid URL: leftover
2025-10-13 22:23:26 +03:00
John Ericson 0f85ef3677 Merge pull request #14219 from lovesegfault/eval-copy-less
libstore: Avoid copying derivations to the store if they are already valid
2025-10-13 16:36:40 +00:00
Eelco Dolstra be2c9ef44c Merge pull request #14229 from NixOS/reduce-hydra-load
packaging/hydra: buildNoGC is the same as buildWithSanitizers
2025-10-13 16:22:30 +00:00
John Ericson d2c0c0607c Merge branch 'master' into eval-copy-less 2025-10-13 11:52:42 -04:00
John Ericson 480ce19011 Merge pull request #14233 from neuralsorcerer/fix-typo
Fix typos
2025-10-13 15:30:05 +00:00
John Ericson 3f876bcb61 Merge pull request #14231 from roberth/code-docs
Code docs
2025-10-13 15:05:30 +00:00
Soumyadip Sarkar 998f93f267 Fix typos 2025-10-13 18:15:52 +05:30
Robert Hensing 583f5e37fc Refactor: use optionalBracket in nix search 2025-10-13 14:02:59 +02:00
Robert Hensing 5dcfddf997 strings: Add optionalBracket helper 2025-10-13 13:59:39 +02:00
Robert Hensing 48a5e2dde2 EvalState: add doc comment 2025-10-13 13:14:05 +02:00
Robert Hensing 6db86389ce util/error: Document addTrace params
... and rename e -> pos. That was weird.
2025-10-13 12:57:22 +02:00
Bernardo Meurer Costa 000e6f6282 feat(libstore): add builtin fetchurl S3 credential pre-resolution
Add support for pre-resolving AWS credentials in the parent process
before forking for builtin:fetchurl. This avoids recreating credential
providers in the forked child process.
2025-10-12 23:01:13 +00:00
Bernardo Meurer CostaandSergei Zimmerman 18ec3d1094 libstore: Avoid copying derivations to the store if they are already valid
This avoids the quite costly copying of derivations to the daemon over the
wire in case it already exists in the eval store.

For a fresh instantiatation (after running nix-collect-garbage) this doesn't
significantly slow down eval:

taskset -c 2,3 hyperfine --reference "result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket" --prepare "nix-collect-garbage --store /tmp/store1111 --no-keep-derivations" "result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket"
Benchmark 1: result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     388.7 ms ±  10.5 ms    [User: 157.0 ms, System: 61.3 ms]
  Range (min … max):   379.4 ms … 415.9 ms    10 runs

Benchmark 2: result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     389.2 ms ±   4.8 ms    [User: 158.5 ms, System: 60.7 ms]
  Range (min … max):   381.2 ms … 397.6 ms    10 runs

But if the derivations are already instantiated this shows a pretty neat speedup:

taskset -c 2,3 hyperfine --reference "result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket" "result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket"
Benchmark 1: result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     240.4 ms ±   3.1 ms    [User: 148.1 ms, System: 57.0 ms]
  Range (min … max):   233.8 ms … 245.0 ms    12 runs

Benchmark 2: result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     226.5 ms ±   4.5 ms    [User: 147.8 ms, System: 55.2 ms]
  Range (min … max):   214.0 ms … 231.2 ms    13 runs

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-13 01:59:38 +03:00
John Ericson f77094715f Merge pull request #14222 from lovesegfault/curl-based-s3-fix-race
fix(libstore): fix race condition in AWS credential provider caching
2025-10-12 22:07:10 +00:00
Bernardo Meurer Costa f0e1f65260 fix(libstore): fix race condition in AWS credential provider caching
The previous implementation had a check-then-create race condition where
multiple threads could simultaneously:
1. Check the cache and find no provider (line 122)
2. Create their own providers (lines 126-145)
3. Insert into cache (line 161)

This resulted in multiple credential providers being created when
downloading multiple packages in parallel, as each .narinfo download
would trigger provider creation on its own thread.

Fix by using boost::concurrent_flat_map's try_emplace_and_cvisit, which
provides atomic get-or-create semantics:
- f1 callback: Called atomically during insertion, creates the provider
- f2 callback: Called if key exists, returns cached provider
- Other threads are blocked during f1, so no nullptr is ever visible
2025-10-12 20:16:02 +00:00
Sergei Zimmerman 89b35ec0dc packaging/hydra: buildNoGC is the same as buildWithSanitizers
This will reduce the load on hydra. It doesn't make sense to
build 2 slightly different variations where the difference
is only in the nix-perl-bindings and additional sanitizers.
2025-10-12 22:10:35 +03:00
Sergei Zimmerman 4f585dedbe Merge pull request #14228 from obsidiansystems/fix-windows-build
Fix windows build
2025-10-12 18:07:50 +00:00
John Ericson a01df8de21 Merge pull request #14227 from NixOS/asan-hydra
packaging: Add buildWithSanitizers to hydraJobs
2025-10-12 17:43:54 +00:00
John Ericson 10223fae86 Fix windows build
I forget to add some CPP in b57caaa1a2.

Hopefully, as we relyon RAII more, these explicit resets become
unneeded.
2025-10-12 13:22:35 -04:00
John Ericson 9150ccb89e Fix Windows dev shell (mostly)
gbenchmark still has too-narrow supported systems, however. That needs
to be fixed in Nixpkgs.
2025-10-12 13:16:50 -04:00
Sergei Zimmerman de75a180cb packaging: Add buildWithSanitizers to hydraJobs 2025-10-12 19:38:01 +03:00
Sergei Zimmerman a491173369 packaging: Add withASan,withUBSan options to the scope 2025-10-12 19:16:08 +03:00
Sergei Zimmerman 199b6ff3fb Disable detect_odr_violation for ASan
There's some unfortunate ODR violations that get dianosed with GCC but not Clang
for static inline constexpr variables defined inside the class body:

template<typename T>
struct static_const
{
    static JSON_INLINE_VARIABLE constexpr T value{};
};

This can be ignored pretty much. There is the same problem for std::piecewise_construct:

http://lists.boost.org/Archives/boost/2007/06/123353.php

==2455704==ERROR: AddressSanitizer: odr-violation (0x7efddc460e20):
  [1] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/pkmljfq97a83dbanr0n64zbm8cyhna33-nix-store-2.33.0pre/lib/libnixstore.so.2.33.0
  [2] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/gbjpkjj0g8vk20fzlyrwj491gwp6g1qw-nix-util-2.33.0pre/lib/libnixutil.so.2.33.0
2025-10-12 19:16:07 +03:00
Sergei Zimmerman 711e738bf9 meson: Simplify asan-options handling even more
Instead of specifying env variables all the time
we can instead embed the __asan_default_options symbol
in all executables / shared objects. This reduces code
duplication.
2025-10-12 19:16:06 +03:00
Sergei Zimmerman d9cabddd17 Merge pull request #14214 from obsidiansystems/derivation-resolution-goal
Split out `DerivationResolutionGoal`
2025-10-12 11:31:54 +00:00
John Ericson 0da430be35 Split out DerivationResolutionGoal
This prepares the way for fixing a few issues.

Take 2: was landed before in 8f4a739d0f.
2025-10-11 19:52:37 -04:00
John Ericson 07df87652c Make keys of Derivation*Goal more legible
The property that substitution goals come first is still preserved
2025-10-11 19:50:45 -04:00
John Ericson a629ce3dec Use member initializer list for Derivation*Goal::drv 2025-10-11 18:37:04 -04:00
John Ericson ba7bbcd1da Cleanup Derivation*Goal names 2025-10-11 18:30:47 -04:00
Sergei Zimmerman 97e770ad01 Merge pull request #14209 from lovesegfault/curl-based-s3-pieces-fix
fix(libstore): improve http-binary-cache-store S3 compatibility
2025-10-11 21:19:18 +00:00
John Ericson 682cf0b266 Merge pull request #14216 from NixOS/simplify-asan-meson
meson: Move asan-options to common
2025-10-11 18:21:24 +00:00
John Ericson f1cb837888 Merge pull request #14217 from NixOS/remove-libgit2-patches
packaging: Remove no longer necessary libgit2 patches
2025-10-11 18:20:36 +00:00
Sergei Zimmerman 47705139c9 packaging: Remove no longer necessary libgit2 patches
25.05 already has 1.9.0 and we don't support older nixpkgs versions.
2025-10-11 16:30:55 +03:00
Sergei Zimmerman d26a337c09 meson: Move asan-options to common
This way we don't have to duplicate the subdir everywhere.
Less copy-pasta is good.
2025-10-11 16:08:35 +03:00
John Ericson 8064c75694 Merge pull request #14194 from xokdvium/stacktrace-assertion
libutil: Print stack trace on assertion failure
2025-10-10 23:25:16 +00:00
John Ericson d75614a315 Merge pull request #14208 from obsidiansystems/consolidate-builder-dispatch
Consolidate logic choosing where we can/should build a bit
2025-10-10 23:05:40 +00:00
Sergei Zimmerman 46382ade74 libutil: Print stack trace on assertion failure
This change overrides __assert_fail on glibc/musl
to instead call std::terminate that we have a custom
handler for. This ensures that we have more context
to diagnose issues encountered by users in the wild.
2025-10-11 01:35:58 +03:00
John Ericson 9cb686f816 Merge pull request #14210 from NixOS/double-quotes
libstore: Fix double-quoting of paths in logs
2025-10-10 22:10:36 +00:00
Bernardo Meurer Costa f02218873e fix(libstore): improve http-binary-cache-store S3 compatibility
This commit adds two key fixes to http-binary-cache-store.cc to
properly support the new curl-based S3 implementation:

1. **Consistent cache key handling**: Use `getReference().render(withParams=false)`
   for disk cache keys instead of `cacheUri.to_string()`. This ensures cache
   keys are consistent with the S3 implementation and don't include query
   parameters, which matches the behavior expected by Store::queryPathInfo()
   lookups.

2. **S3 query parameter preservation**: When generating file transfer requests
   for S3 URLs, preserve query parameters from the base URL (region, endpoint,
   etc.) when the relative path doesn't have its own query parameters. This
   ensures S3-specific configuration is propagated to all requests.
2025-10-10 22:05:57 +00:00
John Ericson b57caaa1a2 Consolidate logic choosing where we can/should build a bit
I want to separate "policy" from "mechanism".

Now the logic to decide how to build (a policy choice, though with some
hard constraints) is all in derivation building goal, and all in the
same spot. build hook, external builder, or local builder --- the choice
between all three is made in the same spot --- pure policy.

Now, if you want to use the external deriation builder, you simply
provide the `ExternalBuilder` you wish to use, and there is no
additional checking --- pure mechanism. It is the responsibility of the
caller to choose an external builder that works for the derivation in
question.

Also, `checkSystem()` was the only thing throwing `BuildError` from
`startBuilder`. Now that that is gone, we can now remove the
`try...catch` around that.
2025-10-10 17:28:57 -04:00
John EricsonandSergei Zimmerman 2ff59ec3e0 Use std::ranges::find_if for finding external builders
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-10 17:27:41 -04:00
Sergei Zimmerman f30cb8667b libstore: Fix double-quoting of paths in logs
std::filesystem::path is already quoted by boost::format with double quotes (").
2025-10-11 00:25:51 +03:00
John Ericson b56dd21c31 Settings::ExternalBuilder::systems make set
Nothing cares about the order, actually.
2025-10-10 17:23:45 -04:00
Sergei Zimmerman 2308aaf192 Merge pull request #14198 from lovesegfault/curl-based-s3-pieces
feat(libstore): add curl-based S3 store implementation
2025-10-10 07:55:44 +00:00
Graham Dennis 8d9e9bc400 Improve comment 2025-10-10 15:00:10 +11:00
Graham Dennis 43b01b6790 Improved backwards compatibility hack for git URLs using dir=... attribute 2025-10-10 14:54:47 +11:00
Bernardo Meurer Costa 0855b715a9 feat(libstore): add curl-based S3 store implementation
Add a new S3BinaryCacheStore implementation that inherits from
HttpBinaryCacheStore.

The implementation is activated with NIX_WITH_CURL_S3, keeping the
existing NIX_WITH_S3_SUPPORT (AWS SDK) implementation unchanged.
2025-10-10 00:13:29 +00:00
John Ericson 2fc88ec114 Merge pull request #14204 from NixOS/kill-to-real-path-fetchers
libfetchers: Remove toRealPath in SourceHutInputScheme::getRevFromRef
2025-10-09 22:01:03 +00:00
Sergei Zimmerman 9c8480becb Merge pull request #14203 from roberth/move-eval-cache-open-to-libflake
Move eval cache open to libflake
2025-10-09 21:42:06 +00:00
Sergei Zimmerman c58acff42a libfetchers: Remove toRealPath in SourceHutInputScheme::getRevFromRef
This code had several issues:

1. Not going through the SourceAccessor means that we can only work
   with physical paths.

2. It did not actually check that the file exists. (std::ifstream does not check
   it by default).
2025-10-10 00:26:57 +03:00
Robert Hensing abcceafbce Use const for lock in openEvalCache 2025-10-09 21:25:40 +02:00
Robert Hensing 42c9cbf9ca Use ref<LockedFlake> where non-null 2025-10-09 21:25:40 +02:00
Robert Hensing 0387b7d6db Move openEvalCache to libflake
Most of the eval cache logic is flake-independent and libexpr,
but the loading part is not.
`nix-flake` is the right component for this, as the eval cache
isn't exactly specific to the command line.
2025-10-09 20:40:40 +02:00
Sergei Zimmerman dfafd8bc38 Merge pull request #14199 from getchoo-contrib/getchoo/toml11-fix
packaging: only override `toml11` when necessary
2025-10-09 17:20:35 +00:00
Sergei Zimmerman 44486871e9 Merge pull request #14200 from Mic92/less-ci-builds
only build on push to master
2025-10-09 15:07:06 +00:00
Jörg Thalheim 118acc84ba only build on push to master
we have now merge queues for maintainance branches. We still build it
for master to have our installer beeing updated. In future this part
could go in new workflow instead.
2025-10-09 14:16:14 +01:00
Seth Flynn 0f016f9bf5 packaging: only override toml11 when necessary
v4.4.0 hit Nixpkgs in https://github.com/NixOS/nixpkgs/pull/442682.
Ideally we'd just use that, but this keeps the fallback behavior until
it's more widespread
2025-10-09 03:24:57 -04:00
John Ericson ce38b46e06 Merge pull request #14170 from lovesegfault/curl-based-s3-pieces
feat(libstore/filetransfer): add S3 signing support
2025-10-08 22:35:32 +00:00
Sergei Zimmerman 33d9270109 Merge pull request #14191 from NixOS/fix-14188
libutil: Fix renderAuthorityAndPath unreachable for path:/ URLs
2025-10-08 21:34:28 +00:00
Bernardo Meurer Costa 00c2a57666 feat(libstore/filetransfer): add S3 signing support 2025-10-08 21:31:34 +00:00
Eelco Dolstra d591f17ecb Merge pull request #14189 from NixOS/fix-exportReferencesGraph
exportReferencesGraph: Handle heterogeneous arrays
2025-10-08 21:19:30 +00:00
John Ericson bb1945a090 Merge pull request #14182 from NixOS/simplify-archive-tests
tests: Move invalid nar tests from tests/functional to libutil-tests
2025-10-08 20:46:37 +00:00
Eelco Dolstra 94f410b628 exportReferencesGraph: Handle heterogeneous arrays
This barfed with

   error: [json.exception.type_error.302] type must be string, but is array

on `nix build github:malt3/bazel-env#bazel-env` because it has a `exportReferencesGraph` with a value like `["string",...["string"]]`.
2025-10-08 22:15:33 +02:00
Bernardo Meurer Costa 3c1e2e56ea feat(libstore/filetransfer): add username/password authentication support
Add a `UsernameAuth` struct and optional `usernameAuth` field to
`FileTransferRequest` to support programmatic username/password
authentication.

This uses curl's `CURLOPT_USERNAME`/`CURLOPT_PASSWORD` options, which
works with multiple protocols (HTTP, FTP, etc.) and is not specific to
any particular authentication scheme.

The primary motivation is to enable S3 authentication refactoring where
AWS credentials (access key ID and secret access key) can be passed
through this general-purpose mechanism, reducing the amount of
S3-specific code behind `#if NIX_WITH_CURL_S3` guards.
2025-10-08 20:10:53 +00:00
Eelco Dolstra 925f10d5ea Merge pull request #14087 from NixOS/required-args
Args::Flag: Add `required` attribute
2025-10-08 19:33:22 +00:00
Sergei Zimmerman 1d8dd77e1d libutil: Fix renderAuthorityAndPath unreachable for path:/ URLs
This was mistakenly triggered by path:/ URL, since the `//` would
correspond to 3 empty segments.
2025-10-08 22:14:49 +03:00
Jörg Thalheim 090f7fb05e Merge pull request #14002 from getchoo-contrib/getchoo/dogfood-experimental-installer
Add experimental installer to installer tests
2025-10-08 04:05:46 +00:00
Jörg Thalheim 2ce343716d Merge pull request #14169 from NixOS/dev-shell-no-separate-debug-info
dev-shell: Disable separateDebugInfo
2025-10-08 04:03:43 +00:00
Jörg Thalheim e69e621578 Merge pull request #14180 from NixOS/release-process-drop-mergify
maintainers: Remove mergify note from release-process.md
2025-10-08 03:34:05 +00:00
Jörg Thalheim d2a0a11a8e Merge pull request #14184 from lovesegfault/nix-better-ci
ci: integrate vm_tests into main tests job
2025-10-08 05:23:53 +02:00
Bernardo Meurer Costa a400ea4257 ci: integrate vm_tests into main tests job
This consolidates the separate vm_tests job into the main tests job,
simplifying the CI workflow. VM tests now run as part of the regular
test matrix.
2025-10-08 02:46:56 +00:00
Sergei Zimmerman fc8b784924 Merge pull request #14147 from lovesegfault/nix-multi-ci
ci: test without s3 and with curl-based-s3
2025-10-08 02:38:03 +03:00
Sergei Zimmerman 0619351326 tests: Move invalid nar tests from tests/functional to libutil-tests
Since 242f362567 we have better infrastructure
for this kind of tests.
2025-10-08 02:02:33 +03:00
Sergei Zimmerman c5b88c22fa dev-shell: Disable separateDebugInfo
This breaks gdb pretty-printers inserted into .debug_gdb_scripts section,
because it implies --compress-debug-sections=zlib, -Wa,--compress-debug-sections.
This is very unfortunate, because then gdb can't use pretty printers for
Boost.Unordered (which are very useful, since boost::unoredred_flat_map is
impossible to debug). This seems perfectly fine to disable in the dev-shell for
the time being.

See [1-3] for further references.

With this change I'm able to use boost's pretty-printers out-of-the box:

```
p *importResolutionCache
$2 = boost::concurrent_flat_map with 1 elements = {[{accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
        get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
      path = "/derivation-internal.nix"}}] = {accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
        get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
      path = "/derivation-internal.nix"}}}
```

When combined with a simple `add-auto-load-safe-path ~/code` in .gdbinit

[1]: https://gerrit.lix.systems/c/lix/+/3880
[2]: https://git.lix.systems/lix-project/lix/issues/1003
[3]: https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
2025-10-08 00:57:34 +03:00
Sergei Zimmerman 75b18a6e47 maintainers: Remove mergify note from release-process.md 2025-10-08 00:51:50 +03:00
Sergei Zimmerman fb117e0cac Merge pull request #14155 from Mic92/backport-action
ci: Switch away from mergify to backport action
2025-10-08 00:32:00 +03:00
Jörg Thalheim 63e8b5f94a ci: Switch away from mergify to backport action
We want to use github native queues.
2025-10-07 23:43:03 +03:00
Eelco Dolstra 559061bf5a Merge pull request #14173 from NixOS/bump-2.33.0
Bump version
2025-10-07 20:42:25 +02:00
Eelco Dolstra b0f567e18b Update mergify.yml 2025-10-07 17:16:57 +02:00
Eelco Dolstra 875f7ec25f Bump version 2025-10-07 17:15:28 +02:00
Eelco Dolstra 6c21c810b9 Merge pull request #14164 from NixOS/release-notes
Release notes for Nix 2.32
2025-10-07 12:48:53 +02:00
John Ericson eea6d75783 Merge pull request #14168 from xokdvium/nar-require-contents
libutil: Throw if `str("contents")` not found
2025-10-06 19:39:26 -04:00
Samuel Connelly 242f362567 libutil: Throw if str("contents") not found
This was broken in 7aa3e7e3a5 (since 2.25).
2025-10-07 01:04:49 +03:00
Sergei Zimmerman 0068ee6ca7 Release note for attrset optimization 2025-10-06 22:16:21 +03:00
Eelco Dolstra 1e709554d5 Merge pull request #14050 from NixOS/fix-fetch-to-store-caching
Fix fetchToStore caching
2025-10-06 19:39:41 +02:00
Eelco DolstraandTaeer Bar-Yam 8f71ef7ede Update doc/manual/source/release-notes/rl-2.32.md
Co-authored-by: Taeer Bar-Yam <Radvendii@users.noreply.github.com>
2025-10-06 19:27:30 +02:00
Eelco Dolstra 8142989e67 Merge pull request #14166 from lovesegfault/nix-rl-notes
docs(release-notes): note fix for fetchTarball/fetchurl substitution
2025-10-06 19:11:18 +02:00
Bernardo Meurer Costa 776038f842 docs(release-notes): note fix for fetchTarball/fetchurl substitution 2025-10-06 17:09:34 +00:00
Bernardo Meurer Costa 8c28283876 ci: test without s3 and with curl-based-s3 2025-10-06 16:24:21 +00:00
Bernardo Meurer Costa 7f22a40e3b build(libstore): assert withAWS xor withCurlS3 2025-10-06 16:22:01 +00:00
Eelco Dolstra f4e44040d4 Release note for external derivation builders 2025-10-06 16:26:29 +02:00
Eelco Dolstra 0376112a51 Organize release notes 2025-10-06 16:11:24 +02:00
Eelco Dolstra c1761b867b Contributors 2025-10-06 16:11:15 +02:00
Eelco Dolstra 9f6ed70429 release notes: 2.32.0 2025-10-06 16:04:58 +02:00
Eelco Dolstra 35b3557fe4 Merge pull request #14162 from NixOS/fix-windows-build
Don't build getPtsName() on Windows
2025-10-06 14:55:39 +02:00
Eelco Dolstra 8aa0acb9e8 Don't build getPtsName() on Windows
It's not needed.

https://hydra.nixos.org/build/309215536
2025-10-06 13:26:36 +02:00
Eelco Dolstra 5e65584edf Merge pull request #14145 from NixOS/external-derivation-builder
External derivation builders
2025-10-06 12:13:01 +02:00
Eelco Dolstra e7e2ac97f8 Merge remote-tracking branch 'origin/master' into external-derivation-builder 2025-10-06 11:40:56 +02:00
Eelco Dolstra e9c5d721d8 ExternalDerivationBuilder: Emit a version field 2025-10-06 11:36:26 +02:00
Eelco Dolstra 68bd2e40f4 ExternalDerivationBuilder: Pass the (scratch) outputs 2025-10-06 11:33:29 +02:00
Eelco Dolstra 6c0d67769d ExternalDerivationBuilder: Pass inputPaths 2025-10-06 11:29:15 +02:00
Eelco Dolstra e5ae81c21a Merge pull request #14158 from fzakaria/fzakaria/small-clang-tidy-fix
clang-tidy fix for src/libstore/build/derivation-check.cc
2025-10-06 11:11:48 +02:00
Sergei Zimmerman d5fa131cfb Merge pull request #14161 from Radvendii/exprophasattr-alloc
libexpr: fixup ExprOpHasAttr() to take allocator reference
2025-10-05 22:49:17 +00:00
Taeer Bar-Yam 14b119c948 libexpr: fixup ExprOpHasAttr() to take allocator reference 2025-10-05 18:28:47 -04:00
Sergei Zimmerman 29f3da1305 Merge branch 'master' into fzakaria/small-clang-tidy-fix 2025-10-05 21:27:39 +00:00
John Ericson 34bca9212a Merge pull request #14156 from NixOS/fix-ub
treewide: Squash some user-after-free bugs
2025-10-05 16:49:22 -04:00
John Ericson cd9c208baf Merge pull request #14159 from NixOS/revert-resolution-goal
Revert #14097, #14022
2025-10-05 16:28:33 -04:00
Sergei Zimmerman ce749454dc Revert "Merge pull request #14022 from obsidiansystems/derivation-resolution-goal"
This reverts commit d02dca099f, reversing
changes made to 9bd09155ac.
2025-10-05 21:54:59 +03:00
Sergei Zimmerman 7e39ab4dc7 Revert "Merge pull request #14097 from obsidiansystems/light-realisation-improvements"
This reverts commit dc8c1461da, reversing
changes made to 28adcfda32.
2025-10-05 21:54:32 +03:00
Farid Zakaria 06a82da6f5 clang-tidy fix for src/libstore/build/derivation-check.cc 2025-10-05 11:18:30 -07:00
Sergei Zimmerman be1ade7373 libexpr: Use use-after-move in SampleStack::saveProfile() 2025-10-05 16:57:13 +03:00
Sergei Zimmerman 452ec09fe0 libstore: Fix use-after-move in DerivationGoal::repairClosure 2025-10-05 16:55:41 +03:00
Jörg Thalheim 7ba3ef21a6 Merge pull request #14154 from NixOS/fix-segfault-toView
treewide: Remove toView() because it leads to segfaults when compiled…
2025-10-05 13:38:40 +02:00
Sergei Zimmerman dce1a893d0 treewide: Remove toView() because it leads to segfaults when compiled with newer nixpkgs
Firstly, this is now available on darwin where the default in llvm 19.
Secondly, this leads to very weird segfaults when building with newer nixpkgs for some reason.
(It's UB after all).

This appears when building with the following:

mesonComponentOverrides = finalAttrs: prevAttrs: {
mesonBuildType = "debugoptimized";
dontStrip = true;
doCheck = false;
separateDebugInfo = false;
preConfigure = (prevAttrs.preConfigure or "") + ''
  case "$mesonBuildType" in
  release|minsize|debugoptimized) appendToVar mesonFlags "-Db_lto=true"  ;;
  *)                              appendToVar mesonFlags "-Db_lto=false" ;;
  esac
'';
};

And with the following nixpkgs input:

nix build ".#nix-cli" -L --override-input nixpkgs "https://releases.nixos.org/nixos/unstable/nixos-25.11pre870157.7df7ff7d8e00/nixexprs.tar.xz"

Stacktrace:

 #0  0x00000000006afdc0 in ?? ()
 #1  0x00007ffff71cebb6 in _Unwind_ForcedUnwind_Phase2 () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libgcc_s.so.1
 #2  0x00007ffff71cf5b5 in _Unwind_Resume () from /nix/store/41ym1jm1b7j3rhglk82gwg9jml26z1km-gcc-14.3.0-lib/lib/libgcc_s.so.1
 #3  0x00007ffff7eac7d8 in std::basic_ios<char, std::char_traits<char> >::~basic_ios (this=<optimized out>, this=<optimized out>)
     at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/basic_ios.h:286
 #4  std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream (this=<optimized out>, this=<optimized out>)
     at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/sstream:806
 #5  nix::SimpleLogger::logEI (this=<optimized out>, ei=...) at ../logging.cc:121
 #6  0x00007ffff7515794 in nix::Logger::logEI (this=0x675450, lvl=nix::lvlError, ei=...) at /nix/store/bkshji3nnxmrmgwa4n2kaxadajkwvn65-nix-util-2.32.0pre-dev/include/nix/util/logging.hh:144
 #7  nix::handleExceptions (programName=..., fun=...) at ../shared.cc:336
 #8  0x000000000047b76b in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/82kmz7r96navanrc2fgckh2bamiqrgsw-gcc-14.3.0/include/c++/14.3.0/bits/new_allocator.h:88
2025-10-05 02:30:21 +03:00
Sergei Zimmerman 35d7719f46 Merge pull request #14149 from Radvendii/exprophasattr-alloc
libexpr: allocate ExprOpHasAttr's AttrPath in Exprs::alloc
2025-10-03 23:50:46 +03:00
Taeer Bar-Yam 39109c05be libexpr: allocate ExprOpHasAttr's AttrPath in Exprs::alloc 2025-10-03 23:26:41 +03:00
Sergei Zimmerman 75826824d0 Merge pull request #14124 from Radvendii/exprselect-alloc
libexpr: allocate ExprSelect's AttrPath in Expr::alloc
2025-10-03 23:25:02 +03:00
Taeer Bar-Yam 76a92985d7 libexpr: allocate ExprSelect's AttrName vector in Expr::alloc 2025-10-03 22:51:23 +03:00
Sergei Zimmerman 862c816498 Merge pull request #14150 from cole-h/fixup-fakessh-check
libstore: fixup fakeSSH check
2025-10-03 22:46:40 +03:00
Cole Helbling 7ec1427fc3 libstore: fixup fakeSSH check
This broke invocations like:

    NIX_SSHOPTS='-p2222 -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no' nix copy /nix/store/......-foo --to ssh-ng://root@localhost

In Nix 2.30.2, fakeSSH was enabled when the "thing I want to connect to"
was plain old "localhost". Previously, this check was written as:

         , fakeSSH(host == "localhost")

Given the above invocation, `host` would have been `root@localhost`, and
thus `fakeSSH` would be `false` because `root@localhost` != `localhost`.

However, since 49ba06175e, `authority.host`
returned _just_ the host (`localhost`, no user) and erroneously enabled
`fakeSSH` in this case, causing `NIX_SSHOPTS` to be ignored (since,
when `fakeSSH` is `true`, `SSHMaster::startCommand` doesn't call
`addCommonSSHOpts`).

`authority.to_string()` accurately returns the expected `root@localhost`
format (given the above invocation), fixing this.
2025-10-03 12:17:17 -07:00
Eelco Dolstra 73e4c40e64 Add test for external-builders 2025-10-03 16:32:32 +02:00
Eelco Dolstra d5d7ca01b3 Merge pull request #14138 from lovesegfault/nix-fix-4313
fix(libfetchers): substitute fetchTarball and fetchurl
2025-10-03 15:13:06 +02:00
Eelco DolstraandCole Helbling 584ef0ffd3 Add external builders
These are helper programs that execute derivations for specified
system types (e.g. using QEMU to emulate another system type).

To use, set `external-builders`:

  external-builders = [{"systems": ["aarch64-linux"], "program": "/path/to/external-builder.py"}]

The external builder gets one command line argument, the path to a JSON file containing all necessary information about the derivation:

  {
    "args": [...],
    "builder": "/nix/store/kwcyvgdg98n98hqapaz8sw92pc2s78x6-bash-5.2p37/bin/bash",
    "env": {
      "HOME": "/homeless-shelter",
      ...
    },
    "realStoreDir": "/tmp/nix/nix/store",
    "storeDir": "/nix/store",
    "tmpDir": "/tmp/nix-shell.dzQ2hE/nix-build-patchelf-0.14.3.drv-46/build",
    "tmpDirInSandbox": "/build"
  }

Co-authored-by: Cole Helbling <cole.helbling@determinate.systems>
2025-10-03 14:34:13 +02:00
Seth Flynn 92d7381826 ci: allow for using the latest build of the experimental installer
Until these repos are potentially merged, this is good for dogfooding
alongside the experimental installer. It also uses the more official
`artifacts.nixos.org` endpoint to install stable releases now

More immediately though, we need a patch for the experimental installer
to really work in CI at all, and that hasn't landed in a tag yet. So,
this lets us use it right from `main`!
2025-10-03 03:37:59 -04:00
Jörg Thalheim 76ac3758d7 Merge pull request #14144 from lovesegfault/curl-based-s3-pieces
build(libstore): add NIX_WITH_CURL_S3 build option
2025-10-03 09:25:22 +02:00
Seth Flynn d2293fb458 ci: enable experimental installer tests 2025-10-03 01:26:55 -04:00
Seth Flynn 2cbbb63628 ci: enable use of the experimental installer 2025-10-03 01:26:52 -04:00
Bernardo Meurer Costa 27f6417128 build(libstore): add NIX_WITH_CURL_S3 build option
Introduce a new build option 'curl-s3-store' for the curl-based S3
implementation, separate from the existing AWS SDK-based 's3-store'.
The two options are mutually exclusive to avoid conflicts.

Users can enable the new implementation with:
  -Dcurl-s3-store=enabled -Ds3-store=disabled
2025-10-03 03:34:57 +00:00
Sergei Zimmerman 8a8a0c2a4b Merge pull request #14135 from lovesegfault/curl-based-s3-pieces
feat(libstore): add AWS CRT-based credential infrastructure
2025-10-02 22:58:36 +00:00
Sergei Zimmerman eb67b0df5a Merge pull request #14142 from NixOS/move-settings-http-store
libstore: Move {narinfo,ls,log}-compression settings from BinaryCache…
2025-10-02 21:04:22 +00:00
Sergei Zimmerman ea14933915 Merge pull request #14139 from osbm/master
docs: Update documentation regarding the flake outputs
2025-10-02 20:17:42 +00:00
Sergei Zimmerman d2017e0e1a libstore: Move {narinfo,ls,log}-compression settings from BinaryCacheStoreConfig to HttpBinaryCacheStoreConfig
These settings are only implemented for the http store and should not be
there for the file:// stores.
2025-10-02 23:11:16 +03:00
Bernardo Meurer Costa 1e92b61750 fix(libfetchers): substitute fetchTarball and fetchurl
Fixes #4313 by enabling builtins.fetchurl, builtins.fetchTarball to use
binary cache substituters before attempting to download from the
original URL.
2025-10-02 19:33:02 +00:00
osbm 7f3f0f2a0b docs: Update documentation regarding the flake outputs 2025-10-02 10:44:30 +03:00
Bernardo Meurer Costa a4e792cba7 feat(libstore): add AWS CRT-based credential infrastructure
Add lightweight AWS credential resolution using AWS CRT (Common Runtime)
instead of the full AWS SDK. This provides credential management for the
upcoming curl-based S3 implementation.
2025-10-01 21:53:55 +00:00
John Ericson dc8c1461da Merge pull request #14097 from obsidiansystems/light-realisation-improvements
Realisation improvements, dummy store support, unit testing
2025-10-01 17:28:26 -04:00
John Ericson 5592bb717b Implement realisation operations on dummy store 2025-10-01 17:05:06 -04:00
John Ericson e06968ec25 Split out UnkeyedRealisation from Realisation
Realisations are conceptually key-value pairs, mapping `DrvOutputs` (the
key) to information about that derivation output.

This separate the value type, which will be useful in maps, etc., where
we don't want to denormalize by including the key twice.

This matches similar changes for existing types:

| keyed              | unkeyed                |
|--------------------|------------------------|
| `ValidPathInfo`    | `UnkeyedValidPathInfo` |
| `KeyedBuildResult` | `BuildResult`          |
| `Realisation`      | `UnkeyedRealisation`   |
2025-10-01 17:01:26 -04:00
Sergei Zimmerman 28adcfda32 Merge pull request #14119 from NixOS/hide-derivation-internal
libexpr: Move derivation-internal.nix from corepkgsFS to internalFS
2025-10-01 20:58:17 +00:00
Jörg Thalheim d02dca099f Merge pull request #14022 from obsidiansystems/derivation-resolution-goal
Introduce `DerivationResolutionGoal`, fix substituting a single CA drv output
2025-10-01 22:53:58 +02:00
Jörg Thalheim 9bd09155ac Merge pull request #14136 from Mic92/jitsi
link to jitsi meeting in the PR docs
2025-10-01 22:39:13 +02:00
Sergei Zimmerman 2774e67c60 Merge pull request #14128 from obsidiansystems/expose-dummy-store-for-tests-somewhat
Expose some core implementation details and write a basic unit test for the dummy store
2025-10-01 20:27:37 +00:00
Jörg Thalheim 85d6c8af4d link to jitsi meeting in the PR docs 2025-10-01 22:23:31 +02:00
Sergei Zimmerman 2a0fddc7d5 libexpr: Move derivation-internal.nix from corepkgsFS to internalFS
Best I can tell this was never supposed to be exposed to the user
and has been this way since 2.19.

2.18 did not expose this file to the user:

nix run nix/2.18-maintenance -- eval --expr "import <nix/derivation-internal.nix>"

error: getting status of '/__corepkgs__/derivation-internal.nix': No such file or directory
2025-10-01 23:13:11 +03:00
Sergei Zimmerman d0c017def5 Merge pull request #14134 from JamiKettunen/fix-libc++-build
libstore: Include missing header to fix compile with libc++ 20
2025-10-01 20:07:03 +00:00
Sergei Zimmerman 30a6cbe90b Merge pull request #14131 from lovesegfault/curl-based-s3-pieces
refactor(libstore): extract S3 URL parsing into separate files
2025-10-01 19:39:46 +00:00
Jami Kettunen 140b08ae3e libstore: Include missing header to fix compile with libc++ 20
https://en.cppreference.com/w/cpp/thread.html

src/libstore/gc.cc:121:39: error: no member named 'sleep_for' in namespace 'std::this_thread'
  121 |                     std::this_thread::sleep_for(std::chrono::milliseconds(100));
      |                     ~~~~~~~~~~~~~~~~~~^
2025-10-01 22:19:08 +03:00
Bernardo Meurer Costa b72898b2aa refactor(libstore): extract S3 URL parsing into separate files
Move ParsedS3URL from s3.cc/.hh into dedicated s3-url.cc/.hh files.
This separates URL parsing utilities (which are protocol-agnostic) from
the AWS SDK-specific S3Helper implementation, making the code cleaner
and enabling reuse by future curl-based S3 implementation.
2025-10-01 16:11:38 +00:00
John Ericson 251479bdda Merge pull request #14127 from obsidiansystems/registerDrvOutput-no-blanket-unsupported
`Store::registerDrvOutput` make pure virtual
2025-10-01 08:51:18 -04:00
John Ericson 772a38069e Merge pull request #14129 from fzakaria/fzakaria/shellcheck-multiple-6
Remove all shellcheck exclusions
2025-09-30 23:54:53 -04:00
Farid Zakaria 015b639cea shellcheck fix: tests/functional/why-depends.sh 2025-09-30 20:27:51 -07:00
Farid Zakaria c8ef6dfa5a shellcheck fix: tests/functional/user-envs.sh 2025-09-30 20:27:50 -07:00
Farid Zakaria 13eac5295d shellcheck fix: tests/functional/user-envs.builder.sh 2025-09-30 20:23:02 -07:00
Farid Zakaria 049c4c7546 shellcheck fix: tests/functional/user-envs-test-case.sh 2025-09-30 20:22:11 -07:00
Farid Zakaria 359e73a6db shellcheck fix: tests/functional/user-envs-migration.sh 2025-09-30 20:20:13 -07:00
Farid Zakaria 799cd62ec8 shellcheck fix: tests/functional/toString-path.sh 2025-09-30 20:19:47 -07:00
Farid Zakaria b349783830 shellcheck fix: tests/functional/supplementary-groups.sh 2025-09-30 20:19:27 -07:00
Farid Zakaria 8c9bfb6e12 shellcheck fix: tests/functional/simple.builder.sh 2025-09-30 20:18:48 -07:00
Farid Zakaria 7266a51412 shellcheck fix: tests/functional/selfref-gc.sh 2025-09-30 20:17:55 -07:00
Farid Zakaria b8f1a8a0c1 shellcheck fix: tests/functional/selfref-gc.sh 2025-09-30 20:17:55 -07:00
Farid Zakaria 1a5ccbeafc shellcheck fix: tests/functional/secure-drv-outputs.sh 2025-09-30 20:17:55 -07:00
Farid Zakaria 64d828b8c4 shellcheck fix: tests/functional/search.sh 2025-09-30 20:17:55 -07:00
Farid Zakaria b42ed6a74d shellcheck fix: tests/functional/restricted.sh 2025-09-30 20:17:55 -07:00
Farid Zakaria d35d86da89 shellcheck fix: tests/functional/repair.sh 2025-09-30 20:17:54 -07:00
Farid Zakaria 06f21e101f shellcheck fix: tests/functional/remote-store.sh 2025-09-30 20:15:34 -07:00
Farid Zakaria 7ed4011990 shellcheck fix: tests/functional/referrers.sh 2025-09-30 20:15:34 -07:00
Farid Zakaria 5d1333bf4b shellcheck fix: tests/functional/recursive.sh 2025-09-30 20:15:34 -07:00
Farid Zakaria 8a36cf4422 shellcheck fix: tests/functional/readfile-context.sh 2025-09-30 20:15:34 -07:00
Farid Zakaria c8a7719614 shellcheck fix: tests/functional/read-only-store.sh 2025-09-30 20:15:34 -07:00
Farid Zakaria 1492c1bc5d shellcheck fix: tests/functional/push-to-store.sh 2025-09-30 20:15:34 -07:00
Farid Zakaria a11195d6ce shellcheck fix: tests/functional/push-to-store-old.sh 2025-09-30 20:15:34 -07:00
Farid Zakaria b951e6e1ed shellcheck fix: tests/functional/pure-eval.sh 2025-09-30 20:15:34 -07:00
Farid Zakaria bcd8311ec6 shellcheck fix: tests/functional/post-hook.sh 2025-09-30 20:15:33 -07:00
Farid Zakaria 1aaa3dafee shellcheck fix: tests/functional/placeholders.sh 2025-09-30 19:54:29 -07:00
Farid Zakaria c82aa04a3d shellcheck fix: tests/functional/path-info.sh 2025-09-30 19:53:54 -07:00
Farid Zakaria 112c9d8f54 shellcheck fix: tests/functional/path-from-hash-part.sh 2025-09-30 19:53:33 -07:00
Farid Zakaria 32cbf5f55a shellcheck fix: tests/functional/pass-as-file.sh 2025-09-30 19:52:44 -07:00
John Ericson 9ac306c4df Expose some core implementation details and write a basic unit test for the dummy store
This test currently doesn't use the new-exposed functionality, but with
future changes the tests will be expanded and they will be used.
2025-09-30 14:52:32 -04:00
John Ericson 88bd0c25f2 Store::registerDrvOutput make pure virtual
It should be the responsibility of implementations that don't implement
it to say so.

See also PR #9799, and issue #5729
2025-09-30 14:13:04 -04:00
John Ericson c97b050a6c Fix ca/eval-store.sh test
The refactor in the last commit fixed the bug it was supposed to fix,
but introduced a new bug in that sometimes we tried to write a resolved
derivation to a store before all its `inputSrcs` were in that store.

The solution is to defer writing the derivation until inside
`DerivationBuildingGoal`, just before we do an actual build. At this
point, we are sure that all inputs in are the store.

This does have the side effect of meaning we don't write down the
resolved derivation in the substituting case, only the building case,
but I think that is actually fine. The store that actually does the
building should make a record of what it built by storing the resolved
derivation. Other stores that just substitute from that store don't
necessary want that derivation however. They can trust the substituter
to keep the record around, or baring that, they can attempt to re
resolve everything, if they need to be audited.
2025-09-30 11:29:21 -04:00
John Ericson 39f6fd9b46 Fix #13247
Resolve the derivation before creating a building goal, in a context
where we know what output(s) we want. That way we have a chance just to
download the outputs we want.

Fix #13247
2025-09-30 11:29:19 -04:00
John Ericson 8f4a739d0f Split out DerivationResolutionGoal
This prepares the way for fixing a few issues.
2025-09-30 11:25:52 -04:00
John Ericson d76dc2406f Merge pull request #14060 from obsidiansystems/build-result-variant
Use `std::variant` to enforce `BuildResult` invariants
2025-09-30 11:02:13 -04:00
Jörg Thalheim bc66e131f8 Merge pull request #14120 from lovesegfault/http-binary-cache-compression
feat(libstore/http-binary-cache-store): narinfo/ls/log compression
2025-09-30 12:50:10 +02:00
Jörg Thalheim 6e6f88ac45 add changelog for http binary cache compression 2025-09-30 11:05:20 +02:00
Jörg Thalheim 3fcd33079c add http binary cache test for compression options 2025-09-30 10:35:46 +02:00
Jörg Thalheim a5facbd2d1 Merge pull request #14121 from obsidiansystems/file-transfer-quit
Some Curl file transfer cleanups
2025-09-30 09:12:08 +02:00
Jörg Thalheim a5b35ec129 Merge pull request #14106 from Radvendii/exprpath-alloc
libexpr: allocate ExprPath strings in the allocator
2025-09-30 09:04:24 +02:00
John Ericson e52e801421 Merge pull request #14123 from NixOS/path-tests-pure-eval
libexpr-tests: Add unit tests for broken readDir /. for pure eval
2025-09-30 00:33:51 -04:00
Sergei Zimmerman a8670e8a7d libexpr-tests: Add unit tests for broken readDir /. for pure eval
A very unfortunate interaction of current filtering with pure eval is
that the following actually leads to `lib.a = {}`. This just adds a unit
test for this broken behavior. This is really good to be done as a unit test
via the in-memory store.

{
  outputs =
    { ... }:
    {
      lib.a = builtins.readDir /.;
    };
}
2025-09-30 03:16:35 +03:00
John Ericson 86fb5b24a9 curlFileTransfer::workerThreadEntry Only call quit if we need to. 2025-09-29 18:10:34 -04:00
John Ericson 1f65b08d94 curlFileTransfer::State:quit emptys the queue
Whoever first calls `quit` now empties the queue, instead of waiting for
the worker thread to do it.

(Note that in the unwinding case, the worker thread is still the first
to call `quit`, though.)
2025-09-29 18:10:34 -04:00
John Ericson d5402b8527 Encapsulate curlFileTransfer::State:quit
It is allowed to read it, and to set it to `false`, but not to set it
to `true`.
2025-09-29 18:10:34 -04:00
Bernardo Meurer Costa 689fa81dc9 feat(libstore/http-binary-cache-store): narinfo/ls/log compression 2025-09-29 21:53:40 +00:00
Sergei Zimmerman 823c0d1140 Merge pull request #14118 from xokdvium/fix-make-empty-source-accessor
libutil: Create empty directory at the root for makeEmptySourceAccessor
2025-09-29 21:19:02 +00:00
Taeer Bar-Yam f70b0b599c libexpr: allocate ExprPath strings in the allocator 2025-09-29 17:02:05 -04:00
Sergei Zimmerman 1830f5f967 libutil: Create empty directory at the root for makeEmptySourceAccessor
This is my SNAFU. Accidentally broken in 02c9ac445f.

There's very dubious behavior for 'builtins.readDir /.':

{
  outputs =
    { ... }:
    {
      lib.a = builtins.readDir /.;
    };
}

nix eval /tmp/test-flake#lib.a

Starting from 2.27 this now returns an empty set. This really isn't supposed
to happen, but this change in the semantics of makeEmptySourceAccessor accidentally
changed the behavior of this.
2025-09-29 23:16:28 +03:00
Jörg Thalheim 13a236ba29 Merge pull request #14114 from fzakaria/fzakaria/shellcheck-multiple-4
shellcheck fixes continued
2025-09-29 21:32:19 +02:00
Farid Zakaria ef17baf50d shellcheck fix: tests/functional/parallel.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 4dc5dbaba2 shellcheck fix: tests/functional/parallel.builder.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria c09cf33a3a shellcheck fix: tests/functional/output-normalization.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 32818483a5 shellcheck fix: tests/functional/optimise-store.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 375529c7e5 shellcheck fix: tests/functional/nix_path.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria fe4e476d13 shellcheck fix: tests/functional/nix-shell.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 78833ca8d0 shellcheck fix: tests/functional/nix-profile.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria cf206ef61e shellcheck fix: tests/functional/nix-daemon-untrusting.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 8c2664ed15 shellcheck fix: tests/functional/nix-copy-ssh.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria ca7414cd18 shellcheck fix: tests/functional/nix-copy-ssh-ng.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria c9fd721be9 shellcheck fix: tests/functional/nix-copy-ssh-common.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 83e203fe45 shellcheck fix: tests/functional/nix-collect-garbage-d.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 2b1a0963f9 shellcheck fix: tests/functional/nix-channel.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 2bfc9019fa shellcheck fix: tests/functional/nix-build.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 794723142b shellcheck fix: tests/functional/nested-sandboxing/command.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 1a71c1ef9f shellcheck fix: tests/functional/nested-sandboxing.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria e26b0c66b0 shellcheck fix: tests/functional/multiple-outputs.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria f2eef5b0a4 shellcheck fix: tests/functional/misc.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 5a13f9fc91 shellcheck fix: tests/functional/logging.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria c4da98c8f4 shellcheck fix: tests/functional/linux-sandbox.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 5341d82428 shellcheck fix: tests/functional/legacy-ssh-store.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria f702101224 shellcheck fix: tests/functional/install-darwin.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 78d9a8d92b shellcheck fix: tests/functional/impure-eval.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria 1cd96f22c0 shellcheck fix: tests/functional/impure-derivations.sh 2025-09-29 10:52:17 -07:00
Farid Zakaria f3a2876c3a shellcheck fix: tests/functional/hash-convert.sh 2025-09-29 10:52:16 -07:00
Robert Hensing 8a968c599d Merge pull request #14116 from fzakaria/fzakaria/shellcheck-multiple-5
shellcheck fixes CA functional tests
2025-09-29 19:48:05 +02:00
Farid Zakaria 4232cb045a Remaining functional/ca tests for shellcheck 2025-09-29 10:25:49 -07:00
Farid Zakaria 5846d9d4dc shellcheck fix: tests/functional/ca/build-dry.sh 2025-09-29 10:12:04 -07:00
Farid Zakaria 745d1f9519 shellcheck fix: tests/functional/ca/build-delete.sh 2025-09-29 10:11:29 -07:00
Jörg Thalheim fca6d8f1cc Merge pull request #14112 from EphraimSiegfried/make-content-addressed-doc-fix
docs: fix build command in make-content-addressed.md
2025-09-29 18:45:14 +02:00
Farid Zakaria 52b9fb38e0 shellcheck fix: tests/functional/gc-non-blocking.sh 2025-09-29 09:23:41 -07:00
Farid Zakaria 2e5952fb6a shellcheck fix: tests/functional/gc-concurrent2.builder.sh 2025-09-29 09:22:45 -07:00
Farid Zakaria 75df03204b shellcheck fix: tests/functional/gc-concurrent.sh 2025-09-29 09:21:47 -07:00
Farid Zakaria 613bd67574 shellcheck fix: tests/functional/gc-concurrent.builder.sh 2025-09-29 09:20:02 -07:00
Farid Zakaria 4192ca9131 shellcheck fix: tests/functional/gc-auto.sh 2025-09-29 09:18:50 -07:00
Farid Zakaria 08a82f4682 shellcheck fix: tests/functional/formatter.simple.sh 2025-09-29 09:17:24 -07:00
Farid Zakaria f596c9b8c3 shellcheck fix: tests/functional/flakes/show.sh 2025-09-29 09:16:29 -07:00
Farid Zakaria cb22518754 shellcheck fix: tests/functional/flakes/run.sh 2025-09-29 09:15:11 -07:00
Farid Zakaria 020f67a653 shellcheck fix: tests/functional/flakes/prefetch.sh 2025-09-29 09:14:41 -07:00
Ephraim Siegfried 121dda0f1f docs: fix build command in make-content-addressed.md 2025-09-29 14:07:26 +02:00
Jörg Thalheim b6f4788a8f Merge pull request #14110 from Mic92/ptsname
Fix thread-safety issue with ptsname() usage
2025-09-29 13:49:58 +02:00
Jörg Thalheim f816b9bcb8 Merge pull request #14111 from Mic92/symlinks
Prevent infinite symlink loop in followLinksToStore()
2025-09-29 13:49:19 +02:00
Jörg Thalheim 5ec9138179 Prevent infinite symlink loop in followLinksToStore()
The followLinksToStore() function could hang indefinitely when encountering
symlink cycles outside the Nix store, causing 100% CPU usage and blocking
any operations that use this function.

This affects multiple commands including nix-store --query, --delete,
--verify, nix-env, and nix-copy-closure when given paths with symlink cycles.

The fix adds a maximum limit of 1024 symlink follows (matching the limit
used by canonPath) and throws an error when exceeded, preventing the
infinite loop while preserving the original semantics of stopping at
the first path inside the store.
2025-09-29 12:22:43 +02:00
Jörg Thalheim a9ffa42dda Fix thread-safety issue with ptsname() usage
Replace non-thread-safe ptsname() calls with a new getPtsName() helper
function that:
- Uses thread-safe ptsname_r() on Linux/BSD platforms
- Uses mutex-protected ptsname() on macOS (which lacks ptsname_r())
2025-09-29 12:01:49 +02:00
Eelco Dolstra b0431a76f5 Merge pull request #14058 from DeterminateSystems/upstream-RossComputerGuy/feat/expose-logfmt
C API: add log format and verbosity functions
2025-09-29 11:21:43 +02:00
Jörg Thalheim 5e65fa7069 Merge pull request #14109 from Mic92/mingw
fix mingw build
2025-09-29 10:57:43 +02:00
Jörg Thalheim 69eae7770a fix mingw build 2025-09-29 10:29:37 +02:00
Jörg Thalheim 738d141dd9 Merge pull request #14108 from fzakaria/fzakaria/shellcheck-multiple-3
shellcheck fixes for tests/functional/local-overlay-store
2025-09-29 09:58:07 +02:00
Jörg Thalheim 7cbc0f97e7 Merge pull request #14090 from Radvendii/expr-slim
nixexpr: introduce arena to hold ExprString strings
2025-09-29 08:33:10 +02:00
Jörg Thalheim c5b3567512 Merge pull request #14105 from xokdvium/cpp-bison
libexpr: Switch parser.y to %skeleton lalr1.cc
2025-09-29 08:23:03 +02:00
Farid Zakaria f394390492 shellcheck fix: tests/functional/local-overlay-store/add-lower-inner.sh 2025-09-28 20:40:08 -07:00
Farid Zakaria 8f0d9412ba shellcheck fix: tests/functional/local-overlay-store/bad-uris.sh 2025-09-28 20:39:40 -07:00
Farid Zakaria 8f14301533 shellcheck fix: tests/functional/local-overlay-store/build-inner.sh 2025-09-28 20:38:46 -07:00
Farid Zakaria 6cae8da29d shellcheck fix: tests/functional/local-overlay-store/check-post-init.sh 2025-09-28 20:37:23 -07:00
Farid Zakaria bb97f4b07a shellcheck fix: tests/functional/local-overlay-store/common.sh 2025-09-28 20:35:47 -07:00
Farid Zakaria 20665e1c3d shellcheck fix: tests/functional/local-overlay-store/delete-duplicate-inner.sh 2025-09-28 20:32:49 -07:00
Farid Zakaria 326d626ad7 shellcheck fix: tests/functional/local-overlay-store/delete-refs-inner.sh 2025-09-28 20:32:12 -07:00
Farid Zakaria 62b36eba11 shellcheck fix: tests/functional/local-overlay-store/gc-inner.sh 2025-09-28 20:31:09 -07:00
Farid Zakaria 675179a510 shellcheck fix: tests/functional/local-overlay-store/gc.sh 2025-09-28 20:30:02 -07:00
Farid Zakaria 283a9c4c5a shellcheck fix: tests/functional/local-overlay-store/optimise-inner.sh 2025-09-28 20:29:35 -07:00
Farid Zakaria dbb53de9d3 shellcheck fix: tests/functional/local-overlay-store/redundant-add.sh 2025-09-28 20:28:18 -07:00
Farid Zakaria 0c50d5b25a shellcheck fix: tests/functional/local-overlay-store/redundant-add-inner.sh 2025-09-28 20:27:21 -07:00
Farid Zakaria 1bee4d0988 shellcheck fix: tests/functional/local-overlay-store/redundant-add.sh 2025-09-28 20:25:42 -07:00
Farid Zakaria 4ef4e96788 shellcheck fix: tests/functional/local-overlay-store/remount.sh 2025-09-28 20:25:09 -07:00
Farid Zakaria c4c95f3d39 shellcheck fix: tests/functional/local-overlay-store/stale-file-handle-inner.sh 2025-09-28 20:24:38 -07:00
Farid Zakaria e896bf1cb1 shellcheck fix: tests/functional/local-overlay-store/stale-file-handle.sh 2025-09-28 20:24:09 -07:00
Farid Zakaria 3a1ba8e41e shellcheck fix: tests/functional/local-overlay-store/verify-inner.sh 2025-09-28 20:23:19 -07:00
Farid Zakaria 76c9d3885c shellcheck fix: tests/functional/local-overlay-store/verify.sh 2025-09-28 20:22:55 -07:00
John Ericson 676e885f8d Merge pull request #14084 from obsidiansystems/issue-13247-test
Create test for Issue 13247
2025-09-28 19:12:44 -04:00
Sergei Zimmerman c1f805b856 packaging: Build without symbolic interposition on GCC
This turns out to be a big problem for performance of Bison
generated code, that for whatever reason cannot be made internal
to the shared library. This causes GCC to make a bunch of function
calls go through PLT. Ideally these hot functions (like move/copy ctor) could become
inline in upstream Bison. That will make sure that GCC can do interprocedular
optimizations without -fno-semantic-interposition [^]. Considering that
LLVM already does inlining and whatnot is a good motivation for this change.
I don't know of any case where Nix relies on LD_PRELOAD tricks for the shared
libraries in production use-cases.

[^]: https://maskray.me/blog/2021-05-09-fno-semantic-interposition
2025-09-29 01:46:40 +03:00
Sergei Zimmerman a8715a2d6e libexpr: Switch parser.y to %skeleton lalr1.cc
Since the parser is now LALR we can easily switch
over to the less ugly sketelon than the default C one.
This would allow us to switch from %union to %define api.value.type variant
in the future to avoid the need for triviall POD types.
2025-09-29 00:58:41 +03:00
John Ericson 241c7cd1f9 Merge pull request #14104 from xokdvium/dead-code-location
libexpr: Remove unused members from ParserLocation
2025-09-28 16:46:50 -04:00
Sergei Zimmerman 0f08feaa58 libexpr: Remove unused members from ParserLocation 2025-09-28 22:57:11 +03:00
Taeer Bar-Yam eab467ecfb libexpr: introduce arena to hold ExprString strings
1. Saves 24-32 bytes per string (size of std::string)
2. Saves additional bytes by not over-allocating strings (in total we
save ~1% memory)
3. Sets us up to perform a similar transformation on the other Expr
subclasses
4. Makes ExprString trivially moveable (before the string data might
move, causing the Value's pointer to become invalid). This is important
so we can put ExprStrings in an std::vector and refer to them by index

We have introduced a string copy in ParserState::stripIndentation().
This could be removed by pre-allocating the right sized string in the
arena, but this adds complexity and doesn't seem to improve performance,
so for now we've left the copy in.
2025-09-28 14:23:13 -04:00
Sergei Zimmerman c43ea09b9b Merge pull request #14100 from obsidiansystems/add-missing-pragma-once
Add `#pragma once` to `dummy-store.hh`
2025-09-28 16:41:28 +00:00
John Ericson 582d3ee611 Add #pragma once to dummy-store.hh
We should have a lint for this.

In later (yet to be merged at this time) commits, this started causing
problems that only the sanitzer caught.
2025-09-28 12:12:24 -04:00
John Ericson f7d35dc1dc Merge pull request #14099 from xokdvium/fix-assert-failure
libstore: Call canonPath for constructing LocalFSStoreConfig::rootDir
2025-09-28 12:06:02 -04:00
Sergei ZimmermanandJohn Ericson 0866ba0b4a libstore: Deduplicate LocalFSStoreConfig::rootDir initializers
Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
2025-09-28 18:38:57 +03:00
John Ericson a01d52e57b Merge pull request #14063 from obsidiansystems/test-less-macro
Minimize the use of C Macros for characterization tests
2025-09-28 11:14:29 -04:00
John Ericson 87fc579bb7 Merge pull request #14098 from xokdvium/store-references-fixes
libstore: Make all StoreConfig::getReference implementations return s…
2025-09-28 11:11:49 -04:00
Sergei Zimmerman 3a64d3c0da libstore: Call canonPath for constructing LocalFSStoreConfig::rootDir
This mirrors what OptionalPathSetting does. Otherwise we run into
an assertion failure for relative paths specified as the authority + path:

nix build nixpkgs#hello --store "local://a/b"
nix: ../posix-source-accessor.cc:13: nix::PosixSourceAccessor::PosixSourceAccessor(std::filesystem::__cxx11::path&&): Assertion `root.empty() || root.is_absolute()' failed.

This is now diagnosed properly:

error: not an absolute path: 'a/b'

Just as you'd specify the root via a query parameter:

nix build nixpkgs#hello --store "local?root=a/b"
2025-09-28 17:42:19 +03:00
John Ericson 01b2037bc0 Minimize the use of C Macros for characterization tests
Fewer macros is better!

Introduce a new `JsonChacterizationTest` mixin class to help with this.

Also, avoid some needless copies with `GetParam`.

Part of my effort shoring up the JSON formats with #13570.
2025-09-28 09:54:46 -04:00
Sergei Zimmerman 426a72c9cf libstore: Make all StoreConfig::getReference implementations return store parameters
These stragglers have been accidentally left out when implementing the StoreConfig::getReference.
Also HttpBinaryCacheStore::getReference now returns the actual store parameters, not the cacheUri
parameters.
2025-09-28 16:29:12 +03:00
John Ericson e35abb1102 Create test for issue 13247
This test ends up being skipped, since the bug has not yet been fixed. A
future commit will fix the bug.

Progress on #13247, naturally.
2025-09-27 21:53:14 -04:00
John Ericson e731c43eae Use std::variant to enforce BuildResult invariants
There is now a clean separation between successful and failing build
results.
2025-09-27 15:56:06 -04:00
John Ericson 43550e8edb Lock down BuildResult::Status enum values
This allows refactoring without changing wire protocol by mistake.
2025-09-27 15:15:20 -04:00
John Ericson 3c610df550 Delete scratch data for CA derivation that produced already-extant output
In the case where the store object doesn't exist, we do correctly move
(rather than copy) the scratch data into place. In this case, the
destination store object already exists, but we still want to clean up
after ourselves.
2025-09-27 15:14:33 -04:00
John Ericson 3bf1268ac6 Merge pull request #14096 from Mic92/concurrency-bugs-2
document thread-unsafe mutation in PosixSourceAccessor
2025-09-27 11:18:56 -04:00
Sergei Zimmerman b9571b6e52 Merge pull request #13508 from CertainLach/push-oyyysvytlnpr
fix: wait on incomplete assignment in REPL
2025-09-27 10:43:41 +00:00
Yaroslav Bolyukin 0a3eb22360 fix: wait on incomplete assignment in REPL
Fixes: https://github.com/NixOS/nix/issues/13507
2025-09-27 13:25:27 +03:00
Jörg Thalheim 866c9179a0 document thread-unsafe mutation in PosixSourceAccessor 2025-09-26 23:31:56 +02:00
Jörg Thalheim 7817239644 Merge pull request #14092 from rszyma/fix-devshell-attrpath-in-docs
doc: Fix invalid devshell attrpath
2025-09-26 21:01:02 +02:00
Jörg Thalheim 6721ef5feb Merge branch 'master' into fix-fetch-to-store-caching 2025-09-26 20:44:39 +02:00
Jörg Thalheim be92b18add Merge pull request #14083 from fzakaria/fzakaria/shellcheck-multiple-2
shellcheck fixes
2025-09-26 20:42:43 +02:00
rszyma 7bd67cd8dc doc: Fix invalid devshell attrpath
`native-clangStdenvPackages` devshell attrpath was being mentioned in
development docs, but doesn't work anymore (since 69fde530).
2025-09-26 19:49:36 +02:00
Jörg Thalheim b5f765b7eb Merge pull request #14047 from Radvendii/eval-memory
libexpr: move eval memory allocation to own struct
2025-09-26 19:24:30 +02:00
Farid Zakaria 2a6724299a shellcheck fix: tests/functional/flakes/follow-paths.sh 2025-09-26 08:41:30 -07:00
Farid Zakaria 6fc8f04ecb shellcheck fix: tests/functional/flakes/flakes.sh 2025-09-26 08:41:30 -07:00
Farid Zakaria ac5615dd91 shellcheck fix: tests/functional/flakes/config.sh 2025-09-26 08:41:30 -07:00
Farid Zakaria 26a10453c3 shellcheck fix: tests/functional/flakes/check.sh 2025-09-26 08:41:30 -07:00
Farid Zakaria 9bf8e7b730 shellcheck fix: tests/functional/flakes/absolute-paths.sh 2025-09-26 08:41:30 -07:00
Farid Zakaria 8839bab84d shellcheck fix: completion files 2025-09-26 08:41:30 -07:00
Farid Zakaria f8e351cd94 shellcheck fix: tests/functional/fixed 2025-09-26 08:41:30 -07:00
Farid Zakaria 4cec876319 shellcheck fix: tests/functional/fetchMercurial.sh 2025-09-26 08:41:30 -07:00
Farid Zakaria c4c3524318 shellcheck fix: tests/functional/fetchGitVerification.sh 2025-09-26 08:41:29 -07:00
Tristan Ross bb6a4dccdf libutil-c: add nix_set_verbosity function 2025-09-26 08:31:23 -07:00
Eelco Dolstra eec4dece33 Args::Flag: Add required attribute 2025-09-26 16:01:59 +02:00
Eelco Dolstra 8aa4669328 Merge pull request #14086 from getchoo-contrib/getchoo/help-pure-eval
nix-cli: use pure/restricted eval for help pages
2025-09-26 11:32:14 +02:00
Seth FlynnandSergei Zimmerman ff82de86da nix-cli: use pure/restricted eval for help pages
This avoids any complications that can arise from the environment
affecting evaluation of the help pages (which don't need to be calling
out to anything external anyways)

A recent example of one of these problems is
https://github.com/NixOS/nix/issues/14085, which would break help pages
by causing them to make invalid calls to the dummy store they're
evaluated with

Fixes: https://github.com/NixOS/nix/issues/14062
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-09-26 02:05:58 -04:00
7b3c193bd3 libexpr: move eval memory allocation to own struct
Co-authored-by: eldritch horrors <pennae@lix.systems>
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>

See original commit on lix:
https://git.lix.systems/lix-project/lix/commit/f5754dc90ae9b1207656d0e29ad2704d3ef1e554
2025-09-26 00:40:43 +03:00
Farid Zakaria 53ad2433b4 shellcheck fix: tests/functional/fetchGitSubmodules.sh 2025-09-25 13:09:36 -07:00
Farid Zakaria ea035ae165 shellcheck fix: tests/tests/functional/dump-db.sh 2025-09-25 13:07:41 -07:00
John Ericson 89141f1d67 Merge pull request #14082 from fzakaria/fzakaria/shellcheck-multiple
shellcheck: multiple file fixes
2025-09-25 15:59:43 -04:00
Farid Zakaria 1619409bf2 shellcheck fix: tests/functional/fetchGitRefs.sh 2025-09-25 12:45:43 -07:00
Farid Zakaria 32e1b5209b shellcheck fix: tests/functional/fetchGit.sh 2025-09-25 12:45:43 -07:00
Farid Zakaria d07dd92db3 shellcheck fix: tests/functional/fetchClosure.sh 2025-09-25 12:45:43 -07:00
Farid Zakaria 230da1cbe7 shellcheck fix: tests/functional/export.sh 2025-09-25 12:45:40 -07:00
Tristan Ross cf595b81d5 libmain-c: add nix_set_log_format function 2025-09-25 10:57:48 -07:00
Farid Zakaria 9e3c502521 shellcheck fix: tests/functional/extra-sandbox-profile.sh 2025-09-25 10:35:26 -07:00
Farid Zakaria a209748ec0 shellcheck fix: tests/functional/export-graph.sh 2025-09-25 10:35:01 -07:00
Farid Zakaria b8c24cdaef shellcheck fix: tests/functional/eval-store.sh 2025-09-25 10:33:40 -07:00
Jörg Thalheim e3d62f35ea Merge pull request #14075 from fzakaria/fzakaria/shellcheck-functional-dump-db
shellcheck fix functional/dump-db.sh
2025-09-25 19:31:49 +02:00
Farid Zakaria dc69e2e520 shellcheck fix: tests/functional/dyn-drv/recursive-mod-json.sh 2025-09-25 10:31:06 -07:00
Farid Zakaria 119489f253 shellcheck fix: tests/functional/dyn-drv/old-daemon-error-hack.sh 2025-09-25 10:30:03 -07:00
Farid Zakaria 412e51215f shellcheck fix: functional/dyn-drv/eval-outputOf.sh 2025-09-25 10:29:27 -07:00
Eelco Dolstra 4b9735b761 Test against uncacheable paths
This is to test the non-functional property that most paths should be
cacheable. We've had frequent cases where caching broken but we didn't
notice.
2025-09-25 11:30:11 -04:00
Eelco Dolstra 1d130492d7 Mount inputs on storeFS to restore fetchToStore() caching
fetchToStore() caching was broken because it uses the fingerprint of
the accessor, but now that the accessor (typically storeFS) is a
composite (like MountedSourceAccessor or AllowListSourceAccessor),
there was no fingerprint anymore. So fetchToStore now uses the new
getFingerprint() method to get the specific fingerprint for the
subpath.
2025-09-25 11:30:11 -04:00
Eelco Dolstra ec6d5c7de3 Path fetcher: Simplify fingerprint computation 2025-09-25 11:21:14 -04:00
Eelco Dolstra 3450a72ba0 Git fetcher: Make dirty repos with no commits cacheable 2025-09-25 11:20:00 -04:00
Eelco Dolstra 28d11c5bcc Add SourceAccessor::getFingerprint()
This returns the fingerprint for a specific subpath. This is intended
for "composite" accessors like MountedSourceAccessor, where different
subdirectories can have different fingerprints.
2025-09-25 11:20:00 -04:00
Eelco Dolstra 55c7ef9d40 SourceAccessor: Make lstat() virtual
With FilteringSourceAccessor, lstat() needs to throw a different
exception if the path is inaccessible than if it doesn't exist.
2025-09-25 11:20:00 -04:00
John Ericson 46095284f1 Merge pull request #14080 from NixOS/storeFS-prep
Some `storeFS` and similar cleanup
2025-09-25 10:50:25 -04:00
Jörg Thalheim 099a74e9f4 Merge pull request #14041 from getchoo-contrib/getchoo/cache-substituted-inputs
libfetchers: avoid re-copying substituted inputs
2025-09-25 13:27:02 +02:00
Seth Flynn 74305d5260 libfetchers: avoid re-copying substituted inputs
Previously, Nix would not create a cache entry for substituted/cached
inputs

This led to severe slowdowns in some scenarios where a large input (like
Nixpkgs) had already been unpacked to the store but didn't exist in a
users cache, as described in https://github.com/NixOS/nix/issues/11228

Using the same method as https://github.com/NixOS/nix/pull/12911, we can
create a cache entry for the fingerprint of substituted/cached inputs
and avoid this problem entirely
2025-09-25 04:04:57 -04:00
Jörg Thalheim 534b29068a Merge pull request #14071 from fzakaria/fzakaria/shellcheck-functional-db-migration
shellcheck fix functional/db-migration.sh
2025-09-25 09:44:09 +02:00
Jörg Thalheim 697c704f28 Merge branch 'master' into fzakaria/shellcheck-functional-db-migration 2025-09-25 09:26:32 +02:00
Farid Zakaria 6e2c11e296 shellcheck fix functional/dump-db.sh
Add back the path variable
2025-09-25 09:24:39 +02:00
Jörg Thalheim bdd3eb400a Merge pull request #14030 from roberth/c-api-item-access
C API: Various improvements to attribute/item access
2025-09-25 09:06:11 +02:00
John Ericson ae896bebdf Merge pull request #14067 from fzakaria/fzakaria/shellcheck-systemd-multi-user
shellcheck fix scipts/install-systemd-multi-user.sh
2025-09-25 02:50:40 -04:00
Jörg Thalheim eb04a6d3e3 Merge pull request #14079 from obsidiansystems/clean-up-why-depends
Clean up `nix why-depends` store accessor usage, and put back store dir in output
2025-09-25 08:49:12 +02:00
Jörg Thalheim a08ae1d024 doc: Add release notes for C API lazy accessors 2025-09-25 08:45:32 +02:00
Jörg Thalheim 1877c477fc Merge branch 'master' into fzakaria/shellcheck-functional-db-migration 2025-09-25 08:35:45 +02:00
Jörg Thalheim 6dcfce0450 Merge branch 'master' into fzakaria/shellcheck-systemd-multi-user 2025-09-25 08:35:10 +02:00
Jörg Thalheim ed3f847225 Merge pull request #14051 from NixOS/atomic-counters
Atomic statistics counters
2025-09-25 08:24:37 +02:00
John Ericson 2898dbe2d9 Merge pull request #14073 from fzakaria/fzakaria/shellcheck-functional-dependencies.builder0
shellcheck fix functional/dependencies.builder0.sh
2025-09-25 02:10:14 -04:00
Eelco Dolstra e8f951289f EvalState: Don't maintain stats by default
These counters are extremely expensive in a multi-threaded
program. For instance, disabling them speeds up evaluation of the
NixOS/nix/2.21.2 from 32.6s to 17.8s.
2025-09-25 08:03:24 +02:00
Eelco Dolstra 8d257f5510 EvalState: Make the counters atomic 2025-09-25 08:03:24 +02:00
Jörg Thalheim a6fcca3888 Merge branch 'master' into fzakaria/shellcheck-functional-db-migration 2025-09-25 07:54:43 +02:00
Farid Zakaria e15c44d46b shellcheck fix functional/db-migration.sh 2025-09-25 07:53:29 +02:00
Jörg Thalheim f3416913d4 Merge pull request #14069 from fzakaria/fzakaria/shellcheck-compute-levels
shellcheck fix for functional/compute-levels.sh
2025-09-25 07:52:56 +02:00
John Ericson 7f9867b548 Merge pull request #14076 from fzakaria/fzakaria/shellcheck-functional-dyn-drv-build-built-drv
shellcheck fix functional/dyn-drv/build-built-drv.sh
2025-09-25 01:51:21 -04:00
Jörg Thalheim 8ff5aeb367 Merge pull request #14065 from obsidiansystems/realisation-json-unit-test
Add JSON tests for `Realisation`
2025-09-25 07:42:15 +02:00
Jörg Thalheim 3076cc9e84 Merge branch 'master' into fzakaria/shellcheck-systemd-multi-user 2025-09-25 07:40:27 +02:00
Jörg Thalheim 31695f3690 Merge branch 'master' into fzakaria/shellcheck-compute-levels 2025-09-25 07:39:25 +02:00
Jörg Thalheim a534a27571 Merge branch 'master' into fzakaria/shellcheck-functional-dependencies.builder0 2025-09-25 07:37:30 +02:00
Jörg Thalheim 5d6af1243a Merge branch 'master' into fzakaria/shellcheck-functional-dyn-drv-build-built-drv 2025-09-25 07:35:10 +02:00
John Ericson 9b2f282af5 Simplify the definition of rootFS
It was getting very hard to follow.
2025-09-25 00:14:14 -04:00
Eelco Dolstra 35189c0ae0 Expose the fact that storeFS is a MountedSourceAccessor
This will become useful.
2025-09-25 00:14:14 -04:00
John Ericson 8ef70ef522 Rename one overload to allowPathLegacy
Makes it easier to tell when it is isued.
2025-09-25 00:14:14 -04:00
Eelco Dolstra 339338e166 MountedSourceAccessor: Move into a separate header, add mount method 2025-09-25 00:14:14 -04:00
Farid Zakaria bc13130497 shellcheck fix tests/functional/dyn-drv/dep-built-drv.sh (#14078) 2025-09-25 03:28:16 +00:00
John Ericson d26dee20b2 Clean up nix why-depends store accessor usage, and put back store dir in output
With this change, the store-wide `getFSAccessor` has only one usage left
--- the evaluator. If we get rid of that (as is planned), we can then
remove that method altogether, simplifying `Store`. Hurray!

I removed the store dir by mistake from the pretty-printed (for humans)
output in eb643d034f. That change was not
supposed to change output.
2025-09-24 23:06:03 -04:00
John Ericson d1958e1b2c Merge pull request #14066 from fzakaria/fzakaria/shellcheck-install-multi-user
shellcheck fix scripts/install-multi-user.sh
2025-09-24 23:00:16 -04:00
John Ericson d0e2babee1 Merge pull request #14072 from fzakaria/fzakaria/shellcheck-functional-debugger
shellcheck fix: functional/debugger.sh
2025-09-24 22:58:32 -04:00
John Ericson d2595130c6 Merge pull request #14074 from fzakaria/fzakaria/shellcheck-functional-dependencies
shellcheck fix functional/dependencies.sh
2025-09-24 22:57:09 -04:00
John Ericson db25195bb2 Merge pull request #14077 from fzakaria/fzakaria/shellcheck-dyn-drv-common
shellcheck fix functional/dyn-drv/common.sh
2025-09-24 22:51:55 -04:00
John Ericson 5842aa2cac Merge pull request #14070 from fzakaria/fzakaria/shellcheck-functional-config.sh
shellcheck fix functional/config.sh
2025-09-24 22:47:21 -04:00
John Ericson 1e1c6d1042 Merge pull request #14068 from fzakaria/fzakaria/shellcheck-functional-completions.sh
shellcheck fix tests/functional/completions.sh
2025-09-24 22:44:32 -04:00
Farid Zakaria 614ef6cfb1 shellcheck fix functional/dyn-drv/common.sh 2025-09-24 19:18:30 -07:00
Farid Zakaria 59791082fa shellcheck fix functional/dyn-drv/build-built-drv.sh 2025-09-24 19:17:12 -07:00
Farid Zakaria 98f716f78c Revert change for SC2059 for nix_user_for_core 2025-09-24 19:13:46 -07:00
Farid Zakaria c7c74fec67 shellcheck fix functional/dependencies.sh 2025-09-24 19:11:00 -07:00
Farid Zakaria 121a8ab3ec shellcheck fix functional/dependencies.builder0.sh 2025-09-24 19:09:21 -07:00
Farid Zakaria 67d43f3b12 shellcheck fix: functional/debugger.sh 2025-09-24 19:06:23 -07:00
Farid Zakaria 832100f543 shellcheck fix functional/config.sh 2025-09-24 18:59:41 -07:00
Farid Zakaria 2732812524 Enable shellcheck for functional/compute-levels.sh 2025-09-24 18:57:30 -07:00
Farid Zakaria 92f8f87dd1 shellcheck fix tests/functional/completions.sh 2025-09-24 18:56:00 -07:00
Farid Zakaria 76b9565414 shellcheck fix scipts/install-systemd-multi-user.sh 2025-09-24 18:52:46 -07:00
Farid Zakaria c77b15a178 shellcheck fix scripts/install-multi-user.sh 2025-09-24 18:49:53 -07:00
John Ericson 30691c38c2 Add JSON tests for Realisation 2025-09-24 18:09:24 -04:00
John Ericson 9f26c20ebd Merge pull request #14057 from obsidiansystems/derived-path-json
Modernize and test derived path JSON
2025-09-24 18:08:06 -04:00
John Ericson f2cd6235a7 Merge pull request #14056 from obsidiansystems/better-output-spec-json-tests
Convert `{Extended,}OutputsSpec` JSON tests to characterization tests
2025-09-24 17:41:25 -04:00
Sergei Zimmerman 05279f2ba0 Merge pull request #14061 from xokdvium/k-way-update
libexpr: Preparation for more k-way updates of attribute sets (NFC)
2025-09-25 00:29:54 +03:00
John Ericson 19fa132a8c Merge pull request #14049 from NixOS/per-object-fs-accessor
Create a second `Store::getFSAccessor` for a single store object
2025-09-24 17:29:10 -04:00
John EricsonandSergei Zimmerman a97d6d89d8 Create a second Store::getFSAccessor for a single store object
This is sometimes easier / more performant to implement, and
independently it is also a more convenient interface for many callers.

The existing store-wide `getFSAccessor` is only used for

 - `nix why-depends`
 - the evaluator

I hope we can get rid of it for those, too, and then we have the option
of getting rid of the store-wide method.

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-09-24 15:49:14 -04:00
Sergei Zimmerman 97ce7759d0 libexpr: Use same naive iterative merging but with evalForUpdate 2025-09-24 21:47:59 +03:00
Jörg Thalheim 0175f7e836 Merge pull request #14059 from xokdvium/formatting-ci
ci: Split formatting check into a separate job, gate other jobs
2025-09-24 13:29:22 +02:00
Jörg ThalheimandSeth Flynn 00775ad83c Apply suggestion from @getchoo
Co-authored-by: Seth Flynn <getchoo@tuta.io>
2025-09-24 13:14:00 +02:00
Sergei Zimmerman 9789019a50 libexpr: Move *StackReservation constants to gc-small-vector.hh
There are other places where it's useful to use these constants
(notably in eval.hh).
2025-09-24 01:05:18 +03:00
Sergei Zimmerman b7c6cf900f libexpr: Explicitly define ExprOpUpdate 2025-09-24 01:04:26 +03:00
Sergei Zimmerman e282175f48 libexpr: Split out MakeBinOpMembers from MakeBinOp 2025-09-24 01:04:23 +03:00
Sergei Zimmerman 35d8ffe01d ci: Split formatting check into a separate job, gate other jobs
This makes the CI fail fast and more explicitly in case the formatting
is incorrect and provides a better error messages. This also ensures
that we don't burn CI on useless checks for code that wouldn't pass lints
anyway.
2025-09-24 00:34:35 +03:00
John Ericson d23e59bb6b Modernize and test derived path JSON
Old code is now just used for `nix build` --- there is no CLI breaking
change.

Test the new format, too.

The new format is not currently used, but will be used going forward,
for example in the C API.

Progress on #13570
2025-09-23 15:05:56 -04:00
Eelco Dolstra 73d3ab05b6 Merge pull request #14054 from obsidiansystems/to-json-no-copy
Fix `JSON_IMPL` macro to avoid extraneous copies
2025-09-23 20:58:12 +02:00
John Ericson f24e00710e Convert {Extended,}OutputsSpec JSON tests to characterization tests
This brings them in line with the other tests, and furthers my goals of
separating unit test data from code.

Doing this cleanup as part of my #13570 effort, but strictly-speaking,
this is separate as these data types' JSON never contained and store
paths or store dirs, just simple output name strings.
2025-09-23 14:47:00 -04:00
Eelco Dolstra 03440946cd Merge pull request #14055 from obsidiansystems/rm-pointless-std-visit
Remove some pointless `std::visit`
2025-09-23 20:14:33 +02:00
John Ericson 1c71cb4005 Remove some pointless std::visit
These are not needed, because the `toJSON` methods are already
implemented for the variant wrapper too.
2025-09-23 13:59:27 -04:00
John Ericson af71a9dbd9 Fix JSON_IMPL macro to avoid extraneous copies
Should take the thing we're serializing by reference.
2025-09-23 13:05:12 -04:00
John Ericson d9de675357 Merge pull request #14027 from obsidiansystems/more-ca-tests
More floating content-addressing derivation tests
2025-09-22 17:09:39 -04:00
John Ericson 7ea31c6e56 Run multiple outputs and build-delete test for CA drvs also 2025-09-22 16:54:30 -04:00
Jörg Thalheim af82c847a7 Merge pull request #14048 from roberth/shellcheck
Shellcheck
2025-09-22 21:57:42 +02:00
Robert Hensing 8a9d9bb0e9 pre-commit: Remove exclusion for removed file 2025-09-22 21:06:26 +02:00
Robert Hensing 4183308ee2 tests/func*/characterisation-test-infra: Fix shellcheck 2025-09-22 21:06:26 +02:00
Robert Hensing 993ea14f52 pre-commit: Remove exclude that passes 2025-09-22 21:06:26 +02:00
Robert Hensing 926287d813 tests/func*/ca/common: Fix shellcheck 2025-09-22 21:06:26 +02:00
Robert Hensing 8c31e07cce tests/func*/ca/build-with-garbage-path: Fix shellcheck 2025-09-22 21:06:26 +02:00
John Ericson 5292b0e49e Merge pull request #14038 from NixOS/thread-safe-dummy
libstore: Make writable dummy store thread-safe
2025-09-22 14:26:35 -04:00
Robert Hensing 01357d0808 Merge pull request #14012 from obsidiansystems/drv-realisation-issue-13570
Convert Realisation JSON logic to standard style
2025-09-22 20:25:06 +02:00
Robert Hensing 8b97d14c08 pre-commit: Give reason for ca test wrappers exclusion 2025-09-22 19:57:28 +02:00
Robert Hensing 5af644492b nix develop: Apply shellcheck 2025-09-22 19:31:22 +02:00
Sergei Zimmerman 5915fe3190 Revert "Use shared pointers in the memory source accessor"
This is no longer necessary.

This reverts commit 4df60e639b.
2025-09-22 20:22:17 +03:00
Sergei Zimmerman c4c92c4c61 libstore: Make writable dummy store thread-safe
Tested by building with b_sanitize=thread and running:

nix flake prefetch-inputs --store "dummy://?read-only=false"

It might make sense to move this utility class out of dummy-store.cc,
but it seems fine for now.
2025-09-22 20:22:16 +03:00
Robert Hensing 43ec36cddf pre-commit: Remove exclude that passes 2025-09-22 19:21:06 +02:00
Robert Hensing 033f13fb1a pre-commit: Remove exclude that passes 2025-09-22 19:19:39 +02:00
Robert Hensing 34e9caaf9b pre-commit: Move zsh exclude 2025-09-22 19:18:52 +02:00
Robert Hensing 6195dfff3a pre-commit: Move fish exclude 2025-09-22 19:17:58 +02:00
Sergei Zimmerman c71f80b6eb libstore: Implement boost::hash for StorePath 2025-09-22 20:16:11 +03:00
Robert Hensing 1878e788ce misc/bash/completion.sh: Fix shellcheck 2025-09-22 19:15:44 +02:00
Robert Hensing c12187b15a pre-commit: Drop exclude config/install-sh
This file was part of the make-based build, which has been removed.
2025-09-22 19:12:33 +02:00
Robert Hensing df23f2b3c1 packaging/dev-shell: Add shellcheck
It was already in the closure for the pre-commit hook installation
script.
2025-09-22 19:09:35 +02:00
John EricsonandRobert Hensing 91593a237f Convert Realisation JSON logic to standard style
No behavior is changed, just:

- Declare a canonical `nlohmnan::json::adl_serializer`

- Use `json-utils.hh` to shorten code without getting worse error
  messages.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2025-09-22 12:59:37 -04:00
John Ericson 6389f65d63 Rework derivation format release note slightly 2025-09-22 12:59:37 -04:00
Robert Hensing 169a368459 Merge pull request #14040 from NixOS/import-thunk
Ensure that files are parsed/evaluated only once (2nd attempt)
2025-09-22 18:33:00 +02:00
Eelco Dolstra a0103fc302 Merge pull request #13852 from lovesegfault/warn-no-kvm
feat(libstore): warn when kvm is enabled but /dev/kvm isn't available
2025-09-22 18:31:57 +02:00
Jörg Thalheim dcaf001d52 Merge pull request #14021 from jfroche/fix/installer-shell-path
installer: prepend nix paths to shell config files instead of appending
2025-09-22 18:27:03 +02:00
John Ericson 74b31d10db Merge pull request #14042 from corngood/cygwin-cross
fix cross build for cygwin
2025-09-22 11:55:40 -04:00
David McFarland 32d4ea8140 fix cross-build for cygwin 2025-09-22 12:27:04 -03:00
Eelco Dolstra d32d77f4d4 Allocate ExprParseFile on the heap for now
https://github.com/NixOS/nix/pull/14013#issuecomment-3308085755
2025-09-22 12:06:51 +02:00
Robert Hensing 28c0089268 Merge pull request #14039 from NixOS/document-expr-methods
libexpr: Document {eval,maybeThunk} methods
2025-09-22 11:59:55 +02:00
Eelco Dolstra 5f60602875 Reapply "Merge pull request #13938 from NixOS/import-thunk"
This reverts commit fd034814dc.
2025-09-22 11:48:58 +02:00
Robert Hensing 71b27774f0 libexpr: Document {eval,maybeThunk} methods 2025-09-22 01:41:02 +03:00
Robert Hensing c121c65640 C API: Clarify valid use of bindings ordering 2025-09-21 21:16:46 +02:00
Robert Hensing ab7feb3898 Merge pull request #14023 from NixOS/dummy-store-path-info
Make dummy store also store path info
2025-09-21 16:58:18 +02:00
Sergei Zimmerman a453a49043 tests: Tests for writeable dummy in-memory store 2025-09-21 13:36:31 +03:00
Sergei Zimmerman b66c357b58 libstore: Implement DummyStore::narFromPath 2025-09-21 13:19:50 +03:00
Sergei Zimmerman 3a4c618483 libstore: Fix typo in description of dummy store 2025-09-21 13:19:49 +03:00
Sergei Zimmerman ed9b377928 libstore: Disable path info cache for dummy store 2025-09-21 13:19:48 +03:00
Sergei Zimmerman 341878ce0f libstore: Make dummy store also store path info 2025-09-21 13:19:47 +03:00
Sergei Zimmerman 02c9ac445f libutil: Improve handling of non-directory root in MemorySourceAccessor 2025-09-21 13:19:45 +03:00
John Ericson 4df60e639b Use shared pointers in the memory source accessor
This allows aliasing, like hard links.
2025-09-21 13:19:44 +03:00
John Ericson f66b56ad3f Merge pull request #14035 from NixOS/github-fetcher-accessor
libfetchers/github: Use getFSAccessor for downloadFile result
2025-09-20 21:42:54 -04:00
Sergei Zimmerman e04381edbd libfetchers/github: Use getFSAccessor for downloadFile result
We should use proper abstractions for reading files from the store.
E.g. this caused errors when trying to download github flakes into
an in-memory store in #14023.
2025-09-21 01:12:42 +03:00
Robert Hensing d8d1fb0e34 Merge pull request #14029 from roberth/enable-libexpr-tests
libexpr-tests: Enable when test setup for building succeeds
2025-09-20 00:57:15 +02:00
Robert Hensing d0b1caf53a C API: Document and verify NIX_ERR_KEY behavior 2025-09-20 00:13:50 +02:00
Robert Hensing 2d1b412e5b libexpr-tests: Enable when test setup for building succeeds
Accidentally disabled by 9bc218ca3f
2025-09-19 23:39:00 +02:00
Robert Hensing 3d777eb37f C API: Add lazy attribute value and list item accessors 2025-09-19 23:39:00 +02:00
Robert Hensing 7c553a30a9 C API: Improve nix_get_attr_name_byidx() doc 2025-09-19 23:39:00 +02:00
Robert Hensing 0e74b25f62 C API: Fix bounds checking in _byidx functions
The docs weren't 100% clear about bounds checking, but suggested that
errors would be caught.
The bounds checks are cheap compared to the function calls they're in,
so we have no reason to omit them.
2025-09-19 23:39:00 +02:00
John Ericson 773dd61d1c Merge pull request #14026 from lovesegfault/s3-url-tests
test(libstore): additional ParsedS3Url tests
2025-09-19 17:08:36 -04:00
Bernardo Meurer Costa b63d9fbc87 test(libstore): additional ParsedS3Url tests
Extracted from the work in #13752
2025-09-19 20:52:44 +00:00
Jean-François Roche a408bc3e30 installer: prepend nix paths to shell config files instead of appending
Some distribution will stop evaluating the shell config file if
they are not running in an interactive shell. As we append the nix
paths to the end of the file, they will not be evaluated.

We better prepend the nix paths to the shell config files to be sure
that once nix is installed, nix path will be available in any shell.

Note that this is already the case for the detsys installer script for
a while: https://github.com/DeterminateSystems/nix-installer/pull/148

Possibly related errors:

https://github.com/NixOS/nix/issues/8061
https://github.com/NixOS/nix/pull/6628
https://github.com/NixOS/nix/issues/2587
2025-09-19 14:06:06 +02:00
Jörg Thalheim 07b96c1d14 Merge pull request #14018 from NixOS/fix-soname-for-real
meson: Fix SONAME for unstable versions
2025-09-19 13:01:04 +02:00
John Ericson 936334638d Merge pull request #14020 from xokdvium/dummy-store-fixes
libstore: Set display prefix for dummy store
2025-09-19 00:00:00 -04:00
John Ericson 8719c2e6b7 Merge pull request #14016 from xokdvium/asan
treewide: Support builds with ASAN, enable in CI
2025-09-18 23:48:23 -04:00
Sergei Zimmerman 72e2b0efea libstore: Set display prefix for dummy store
Otherwise the prefix is «unknown».
2025-09-19 02:28:59 +03:00
Sergei Zimmerman d5e84383d1 doc: Document building with sanitizers 2025-09-19 01:37:15 +03:00
Sergei ZimmermanandJade Lovelace 94d37e62fc treewide: Support builds with ASAN, enable in CI
Enables builds with ASAN to catch memory corruption
bugs faster and in CI. This is an incredibly valuable
instrument that must be used as much as possible.

Somewhat based on jade's work from Lix, though there's a lot that
we have to do differently:

https://git.lix.systems/lix-project/lix/commit/19ae87e5cec71912c7e7ecec5dc8ff18d18c60ee

Co-authored-by: Jade Lovelace <lix@jade.fyi>
2025-09-19 01:33:57 +03:00
Sergei Zimmerman 7dbfe9f576 meson: Fix SONAME for unstable versions
Replacing the string is not enough [^] for e.g. nixpkgs precise versions:

`2.31pre20250712_b1245123`

[^]: https://github.com/NixOS/nixpkgs/pull/444089
2025-09-18 23:34:27 +03:00
Jörg Thalheim 3a687eeb4f Merge pull request #14015 from eclairevoyant/doc-global-builtins
doc: document global builtins
2025-09-18 08:11:18 +02:00
Jörg Thalheim 4808b0e24f Merge pull request #14014 from xokdvium/fish delete
libexpr-c: Fix mismatched new/delete
2025-09-18 08:08:39 +02:00
éclairevoyant 40b47b9395 doc: document global builtins 2025-09-18 00:04:31 -04:00
Sergei Zimmerman 309d55807c libexpr-c: Fix mismatched new/delete
This leads to ASAN errors:

==1137785==ERROR: AddressSanitizer: new-delete-type-mismatch on 0x523000001d00 in thread T0:
  object passed to delete has wrong type:
  size of the allocated type:   5968 bytes;
  size of the deallocated type: 5968 bytes.
  alignment of the allocated type:   8 bytes;
  alignment of the deallocated type: default-aligned.
2025-09-18 02:58:32 +03:00
Jörg Thalheim 9d8c6a6646 Merge pull request #14013 from xokdvium/revert-13938
Revert "Merge pull request #13938 from NixOS/import-thunk"
2025-09-18 01:23:14 +02:00
Sergei Zimmerman fd034814dc Revert "Merge pull request #13938 from NixOS/import-thunk"
This has multiple dangling pointer issues that lead to segfaults in e.g.:

nix eval --expr '(builtins.getFlake "github:nixos/nixpkgs/25.05")' --impure

This reverts commit ad175727e4, reversing
changes made to d314750174.
2025-09-18 01:52:46 +03:00
Robert Hensing b4fcb64276 Merge pull request #13980 from obsidiansystems/drv-json-issue-13570
Make the JSON format for derivation use basename store paths
2025-09-17 23:31:41 +02:00
Robert Hensing 82315c3807 Merge pull request #13987 from xokdvium/bindings-bananza
libexpr: Structural sharing of attrsets
2025-09-17 23:15:05 +02:00
Sergei Zimmerman 0ccb00bb81 libexpr: Add release note for c-api-byidx change 2025-09-17 23:54:46 +03:00
Sergei Zimmerman 6138bc3de3 libexpr: Structural sharing of attrsets
This changes the implementation of Bindings to allow
for a more space-efficient implementation of attribute
set merges. This is accomplished by "layering" over the "base" Bindings.
The top "layer" is naturally the right-hand-side of the update operator //.

Such an implementation leads to significantly better memory usage on
something like nixpkgs:

nix-env --query --available --out-path --file ../nixpkgs --eval-system x86_64-linux > /dev/null

Comparison against 2b0fd88324 for x86_64-linux on nixpkgs f06c7c3b6f5074dbffcf02542fb86af3a5526afa:

| metric                 | mean_before     | mean_after      | mean_diff       | mean_%_change | p_value | t_stat  |
| -                      | -               | -               | -               | -             | -       | -       |
| cpuTime                | 21.1520         | 21.3414         | 0.1894          | 0.7784        | 0.3190  | 1.0219  |
| envs.bytes             | 461451951.6190  | 461451951.6190  | -               | -             | -       | -       |
| envs.elements          | 34344544.8571   | 34344544.8571   | -               | -             | -       | -       |
| envs.number            | 23336949.0952   | 23336949.0952   | -               | -             | -       | -       |
| gc.cycles              | 7.5238          | 7.2857          | -0.2381         | -4.6825       | 0.0565  | -2.0244 |
| gc.heapSize            | 1777848124.9524 | 1252162023.6190 | -525686101.3333 | -29.9472      | 0.0000  | -8.7041 |
| gc.totalBytes          | 3102787383.6190 | 2498431578.6667 | -604355804.9524 | -19.7704      | 0.0000  | -9.3502 |
| list.bytes             | 59928225.9048   | 59928225.9048   | -               | -             | -       | -       |
| list.concats           | 1240028.2857    | 1240028.2857    | -               | -             | -       | -       |
| list.elements          | 7491028.2381    | 7491028.2381    | -               | -             | -       | -       |
| nrAvoided              | 28165342.2381   | 28165342.2381   | -               | -             | -       | -       |
| nrExprs                | 1577412.9524    | 1577412.9524    | -               | -             | -       | -       |
| nrFunctionCalls        | 20970743.4286   | 20970743.4286   | -               | -             | -       | -       |
| nrLookups              | 10867306.0952   | 10867306.0952   | -               | -             | -       | -       |
| nrOpUpdateValuesCopied | 61206062.0000   | 25748169.5238   | -35457892.4762  | -58.8145      | 0.0000  | -8.9189 |
| nrOpUpdates            | 2167097.4286    | 2167097.4286    | -               | -             | -       | -       |
| nrPrimOpCalls          | 12337423.4286   | 12337423.4286   | -               | -             | -       | -       |
| nrThunks               | 29361806.7619   | 29361806.7619   | -               | -             | -       | -       |
| sets.bytes             | 1393822818.6667 | 897587655.2381  | -496235163.4286 | -36.7168      | 0.0000  | -9.1115 |
| sets.elements          | 84504465.3333   | 48270845.9524   | -36233619.3810  | -43.8698      | 0.0000  | -8.9181 |
| sets.number            | 5218921.6667    | 5218921.6667    | -               | -             | -       | -       |
| sizes.Attr             | 16.0000         | 16.0000         | -               | -             | -       | -       |
| sizes.Bindings         | 8.0000          | 24.0000         | 16.0000         | 200.0000      | -       | inf     |
| sizes.Env              | 8.0000          | 8.0000          | -               | -             | -       | -       |
| sizes.Value            | 16.0000         | 16.0000         | -               | -             | -       | -       |
| symbols.bytes          | 1368494.0952    | 1368494.0952    | -               | -             | -       | -       |
| symbols.number         | 109147.1905     | 109147.1905     | -               | -             | -       | -       |
| time.cpu               | 21.1520         | 21.3414         | 0.1894          | 0.7784        | 0.3190  | 1.0219  |
| time.gc                | 1.6011          | 0.8508          | -0.7503         | -37.1507      | 0.0017  | -3.6328 |
| time.gcFraction        | 0.0849          | 0.0399          | -0.0450         | -37.4504      | 0.0035  | -3.3116 |
| values.bytes           | 615968144.7619  | 615968144.7619  | -               | -             | -       | -       |
| values.number          | 38498009.0476   | 38498009.0476   | -               | -             | -       | -       |

Overall this does slow down the evaluator slightly (no more than ~10% in most cases),
but this seems like a very decent tradeoff for shaving off 33% of memory usage.
2025-09-17 23:54:45 +03:00
John Ericson 9d7229a2a4 Make the JSON format for derivation use basename store paths
See #13570 for details --- the idea is that included the store dir in
store paths makes systematic JSON parting with e.g. Serde, Aeson,
nlohmann, or similiar harder.

After talking to Eelco, we are changing the `Derivation` format right
away because not only is `nix derivation` technically experimental, we think it is
also less widely used in practice than, say, `nix path-info`.

Progress on #13570
2025-09-17 16:38:17 -04:00
Robert Hensing 3eb223f4bb Merge pull request #10915 from NixOS/dummy-memory
Use `MemorySourceAccessor` in `DummyStore` so writes work
2025-09-17 22:33:17 +02:00
d5ce8c3caa Use MemorySourceAccessor in DummyStore
Add `read-only` setting to `dummy://` store for back compat.

Test by changing an existing test to use this instead, fixing a TODO.

Co-Authored-By: HaeNoe <git@haenoe.party>
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-09-17 16:09:53 -04:00
Eelco Dolstra 7ff61576f2 Merge pull request #14009 from xokdvium/fix-authorization
Revert "tests/nixos: Fix daemon store reference in authorization test"
2025-09-17 21:58:44 +02:00
John Ericson 168c24b605 Declare DummyStoreConfig in a header
This will useful for unit tests.
2025-09-17 15:54:30 -04:00
John Ericson 613de9d9cc Add missing #pragma once 2025-09-17 15:21:21 -04:00
Sergei Zimmerman 86ad8d49f9 Revert "tests/nixos: Fix daemon store reference in authorization test"
This reverts commit 695f3bc7e3.
2025-09-17 22:05:26 +03:00
Jörg Thalheim 187520ce88 Merge pull request #14008 from juhp/update-COPYING
COPYING: update to latest lgpl-2.1.txt
2025-09-17 20:15:01 +02:00
Jens Petersen a66ba324d7 COPYING: update to latest lgpl-2.1.txt (fixes #13758)
$ curl -I https://www.gnu.org/licenses/old-licenses/lgpl-2.1.txt
:
Last-Modified: Wed, 18 Sep 2024 14:34:04 GMT
ETag: "6733-62265b29fd1ee"
:
2025-09-18 00:45:01 +08:00
Jörg Thalheim 7822bd5692 Merge pull request #14005 from juhp/nix_soversion
nix-meson-build-support/common nix_soversion: fixup removal of 'pre'
2025-09-17 12:19:57 +02:00
Robert Hensing 0b19468368 Merge pull request #13907 from obsidiansystems/store-building-unit-test
C API for building
2025-09-16 19:45:51 +02:00
John Ericson 9bc218ca3f Add new C API for working with derivations
Also test the APIs we just added.
2025-09-16 13:25:36 -04:00
Jens Petersen ca23c819e0 nix-meson-build-support/common nix_soversion: fixup removal of 'pre'
.strip() removes individual chars whereas .replace() affects whole substring

Thanks @keszybz
2025-09-16 18:31:40 +08:00
Jörg Thalheim 1a69fc6ab5 Merge pull request #14001 from juhp/common-soversion
meson: refactor nix_soversion into nix-meson-build-support/common
2025-09-16 08:50:05 +02:00
Jens Petersen 86bb7c958a meson: refactor nix_soversion into nix-meson-build-support/common
This is a follow-on to #13995 which added soversion to the libraries
2025-09-16 12:54:30 +08:00
Jörg Thalheim 5e17a3f81c Merge pull request #13998 from Mic92/fast-flake-check
nix flake check: Skip substitutable derivations
2025-09-15 21:44:11 +02:00
Seth FlynnandEelco Dolstra ecdda5798c nix flake check: Skip substitutable derivations
Since `nix flake check` doesn't produce a `result` symlink, it doesn't
actually need to build/substitute derivations that are already known
to have succeeded, i.e. that are substitutable.

This can speed up CI jobs in cases where the derivations have already
been built by other jobs. For instance, a command like

  nix flake check github:NixOS/hydra/aa62c7f7db31753f0cde690f8654dd1907fc0ce2

should no longer build anything because the outputs are already in
cache.nixos.org.

Based-on: https://github.com/DeterminateSystems/nix-src/pull/134
Based-on: https://gerrit.lix.systems/c/lix/+/3841
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-09-15 21:31:03 +02:00
Jörg Thalheim 0b401e2199 Merge pull request #13995 from juhp/soversion
meson: add soversion with nix version to give SONAME to libs
2025-09-15 20:36:31 +02:00
Eelco Dolstra ad175727e4 Merge pull request #13938 from NixOS/import-thunk
Ensure that files are parsed/evaluated only once
2025-09-15 19:03:18 +02:00
John Ericson d314750174 Merge pull request #13999 from obsidiansystems/more-get-improvements
More `get` / `getOr` improvements
2025-09-15 12:55:40 -04:00
John Ericson f3e3f75838 More get / getOr improvements
- Use `const K`, not `K`, otherwise we don't get auto referencing of
  rvalues.

- Generalized the deleted overloads, because we don't care what the key
  type is --- we want to get rid of anything that has an rvalue map
  type.
2025-09-15 12:25:37 -04:00
Jens Petersen dd1a554aba meson: add soversion with nix version to give SONAME to libs (#13960, #13979)
remove 'pre' version suffix for non-releases (chokes Darwin ld)
2025-09-15 22:08:26 +08:00
Sergei Zimmerman 2b0fd88324 Merge pull request #13991 from xokdvium/bindings-remove-find
libexpr: Remove Bindings::find
2025-09-14 21:32:31 +00:00
John Ericson ffc14ac91b Merge pull request #13983 from xokdvium/bindings-fixes
libexpr: Make Bindings::iterator a proper strong type instead of pointer
2025-09-14 17:31:48 -04:00
Sergei Zimmerman d830840433 libexpr: Remove Bindings::find
A follow-up optimization will make it impossible to make a find function
that returns an iterator in an efficient manner. All consumer code can
easily use the `get` variant.
2025-09-14 23:29:44 +03:00
Sergei Zimmerman ddabd94f82 libexpr: Make Bindings::iterator a proper strong type instead of pointer
As evident from the number of tests that were holding this API completely
wrong (the end() iterator returned from find() is NEVER nullptr) we should
not have this footgun. A proper strong type guarantees that this confusion
will not happen again.

Also this will be helpful down the road when Bindings becomes something
smarter than an array of Attr.
2025-09-14 22:52:37 +03:00
Sergei Zimmerman b974b7dc1e Merge pull request #13985 from dramforever/master 2025-09-14 11:19:10 +00:00
dramforever 7295034362 libstore: Raise default connect-timeout to 15 secs
This allows the weird network or DNS server fallback mechanism inside
glibc to work, and prevents a "Resolving timed out after 5000
milliseconds" error. Read on for details.

The DNS request stuff (dns-hosts) in glibc uses this fallback procedure
to minimize network RTT in the ideal case while dealing with
ill-behaving networks and DNS servers gracefully (see resolv.conf(5)):

- Use sendmmsg() to send UDP DNS requests for IPv4 and IPv6 in parallel
- If that times out (meaning that none or only one of the responses have
  been received), send the requests one by one, waiting for the response
  before sending the next request ("single-request")
- If that still times out, try to use a different socket (hence
  different address) for each request ("single-request-reopen")

The default timeout inside glibc is 5 seconds. Therefore, setting
connect-timeout, and therefore CURLOPT_CONNECTTIMEOUT to 5 seconds
prevents the single-request fallback, and setting it to even 10 seconds
prevents the single-request-reopen fallback as well.

The fallback decision is saved by glibc, but only thread-locally, and
libcurl starts a new thread for getaddrinfo() for each connection.
Therefore for every connection the fallback starts from sendmmsg() all
over again. And since these are considered to have timed out by libcurl,
even though getaddrinfo() might return a successful result, it is not
cached in libcurl.

While a user could tweak these with resolv.conf(5) options (e.g. using
networking.resolvconf.extraOptions in NixOS), and indeed that is
probably needed to avoid annoying delays, it still means that the
default connect-timeout of 5 is too low. Raise it to give fallback a
chance.
2025-09-14 08:48:29 +08:00
John Ericson 5bc96798b1 Merge pull request #13982 from obsidiansystems/path-info-static-function
`ValidPathInfo`, `NarInfo`, turn funky constructor into static method
2025-09-13 20:10:45 -04:00
Sergei Zimmerman e75501da3e libexpr: Remove non-const iterators of Bindings 2025-09-13 23:21:24 +03:00
John Ericson 74be28820c ValidPathInfo, NarInfo, turn funky constructor into static method
This is more flexible, and needed for me to be able to reshuffle the
inheritance bureaucracy to make the JSON instances more precise.
2025-09-13 13:17:14 -04:00
Jörg Thalheim 465d627f7f Merge pull request #13978 from xokdvium/fix-warn
libutil: Fix missing return warning
2025-09-13 10:30:32 +02:00
Sergei Zimmerman 298ea97c12 libutil: Fix missing return warning
../hash.cc: In function 'nix::{anonymous}::DecodeNamePair nix::baseExplicit(HashFormat)':
../hash.cc:114:1: warning: control reaches end of non-void function [-Wreturn-type]
  114 | }
      | ^
2025-09-13 09:19:07 +03:00
John Ericson 1907a3300f Merge pull request #13811 from hgl/patch-2
doc: Rephrase store-object.md
2025-09-12 23:34:52 -04:00
John Ericson 1710fd09f3 Merge pull request #13975 from obsidiansystems/prep-json-0
Misc prep changes for #13942
2025-09-12 23:32:20 -04:00
Glen Huang a0b633dd2b doc: Rephrase store-object.md 2025-09-13 10:17:12 +08:00
John Ericson f78062d2fb Merge pull request #13976 from xokdvium/darwin-packaging
packaging: Drop legacy apple sdk pattern
2025-09-12 18:37:36 -04:00
Sergei Zimmerman 20b532eab0 packaging: Drop legacy apple sdk pattern
This has been dropped on unstable an nix no longer
compiled with overridden nixpkgs input. On 25.05 these
overrides already do nothing.

Tested with:

nix build .#packages.x86_64-darwin.nix-cli -L --override-input nixpkgs https://releases.nixos.org/nixos/unstable/nixos-25.11pre859555.ab0f3607a6c7/nixexprs.tar.xz

Default deployment target on 25.05 is 11.3, so 10.13
sdk override doesn't have to be updated at all as evident
from the fact that we didn't observe any issues with it.
2025-09-13 01:07:42 +03:00
John Ericson 095ac66d4c Introduce Hash::parseExplicitFormatUnprefixed 2025-09-12 18:04:29 -04:00
John Ericson c6d06ce486 Fix hash error message
Wrong number of arguments was causing a format assertion.
2025-09-12 18:04:29 -04:00
John Ericson c242706319 Move json_avoids_null to its own header
This is because we need it in declarations where we should not be
including the full `nlohmann/json.hpp`.

Already can clean up by moving the experimental feature "instance".

Also, make the `std::map` instance better by allowing for other
comparison functions.
2025-09-12 18:04:29 -04:00
Philip Wilk aef431fbd1 bugfix/3514: do not throw on substituter errors if other substituters are still enabled (#13301)
## Motivation

Nix currently hard fails if a substituter is inaccessible, even when they are other substituters available, unless `fallback = true`. 
This breaks nix build, run, shell et al entirely. 
This would modify the default behaviour so that nix would actually use the other available substituters and not hard error.

Here is an example before vs after when using dotenv where I have manually stopped my own cache to trigger this issue, before and after the patch. The initial error is really frustrating because there is other caches available.
![image](https://github.com/user-attachments/assets/b4aec474-52d1-497d-b4e8-6f5737d6acc7)
![image](https://github.com/user-attachments/assets/ee91fcd4-4a1a-4c33-bf88-3aee67ad3cc9)

## Context

https://github.com/NixOS/nix/issues/3514#issuecomment-2905056198 is the earliest issue I could find, but there are many duplicates.

There is an initial PR at https://github.com/NixOS/nix/pull/7188, but this appears to have been abandoned - over 2 years with no activity, then a no comment review in jan. There was a subsequent PR at https://github.com/NixOS/nix/pull/8983 but this was closed without merge - over a year without activity.
<!-- Non-trivial change: Briefly outline the implementation strategy. -->
I have visualised the current and proposed flows. I believe my logic flows line up with what is suggested in https://github.com/NixOS/nix/pull/7188#issuecomment-1375652870 but correct me if I am wrong.
Current behaviour:
![current](https://github.com/user-attachments/assets/d9501b34-274c-4eb3-88c3-9021a482e364)
Proposed behaviour:
![proposed](https://github.com/user-attachments/assets/8236e4f4-21ef-45d7-87e1-6c8d416e8c1c)

[Charts in lucid](https://lucid.app/lucidchart/1b51b08d-6c4f-40e0-bf54-480df322cccf/view)
<!-- Invasive change: Discuss alternative designs or approaches you considered. -->

Possible issues to think about:
- I could not figure out where the curl error is created... I can't figure out how to swallow it and turn it into a warn or better yet, a debug log.
- Unfortunately, in contrast with the previous point, I'm not sure how verbose we want to warns/traces to be - personally I think that the warn that a substituter has been disabled (when it happens) is sufficient, and that the next one is being used, but this is personal preference.
2025-09-12 17:29:34 -04:00
Sergei Zimmerman 92df96543c Merge pull request #13972 from xokdvium/no-string-values-in-evalstate 2025-09-12 21:19:09 +00:00
Jörg Thalheim 5772c207d8 Merge pull request #13970 from xokdvium/no-soname
Revert "meson: add soversion to libraries (#13960)"
2025-09-12 23:17:17 +02:00
Sergei Zimmerman f4c38278ca libexpr: Remove vString* Values from EvalState
EvalState is too big and cluttered. These strings
can be private constant statics.
2025-09-12 23:44:52 +03:00
Sergei Zimmerman 0db2b8c8fe Revert "meson: add soversion to libraries (#13960)"
This reverts commit bdbc739d6e.

Such a change needs more thought put into it. By versioning
shared libraries we'd make a false impression that libraries
themselves are actually versioned and have some sort of stable
ABI, which is not the case.

This will be useful when C bindings become stable, but as long
as they are experimental it does not make sense to set SONAME.

Also this change should not have been backported, since it's
severely breaking.
2025-09-12 20:43:34 +03:00
Eelco Dolstra 8d8f49cb5a Use concurrent_flat_map_fwd.hpp 2025-09-12 17:49:15 +02:00
Eelco Dolstra 4b63ff63a4 Remove some unnecessary hash template arguments 2025-09-12 17:26:29 +02:00
Eelco Dolstra ad6eb22368 Ensure that files are parsed/evaluated only once
When doing multithreaded evaluation, we want to ensure that any Nix
file is parsed and evaluated only once. The easiest way to do this is
to rely on thunks, since those ensure locking in the multithreaded
evaluator. `fileEvalCache` is now a mapping from `SourcePath` to a
`Value *`. The value is initially a thunk (pointing to a
`ExprParseFile` helper object) that can be forced to parse and
evaluate the file. So a subsequent thread requesting the same file
will see a thunk that is possibly locked and wait for it.

The parser cache is gone since it's no longer needed. However, there
is a new `importResolutionCache` that maps `SourcePath`s to
`SourcePath`s (e.g. `/foo` to `/foo/default.nix`). Previously we put
multiple entries in `fileEvalCache`, which was ugly and could result
in work duplication.
2025-09-12 17:05:21 +02:00
Eelco Dolstra 47c16fc4bd SourcePath: Implement boost::hash 2025-09-12 17:05:21 +02:00
Eelco Dolstra 8fbf4b9427 CanonPath: Implement boost::hash 2025-09-12 17:05:21 +02:00
Eelco Dolstra 7f9b5226af Add getConcurrent helper function 2025-09-12 16:49:25 +02:00
Jörg Thalheim 377b60ee9b Merge pull request #13926 from NaN-git/opt_boost-unordered
replace all occurences of std::unordered_* by equivalents from boost
2025-09-12 11:46:42 +02:00
Jörg Thalheim 429812cdb8 Merge pull request #13966 from juhp/patch-1
meson: add soversion to libraries (#13960)
2025-09-12 08:25:12 +02:00
Jörg Thalheim f6802a8ccf Merge pull request #13963 from xokdvium/fix-no-gc
libexpr: Fix build without Boehm
2025-09-12 08:23:19 +02:00
Jens Petersen bdbc739d6e meson: add soversion to libraries (#13960) 2025-09-12 14:06:39 +08:00
Sergei Zimmerman c0b35c71cd libexpr: Fix build without Boehm
This should have been placed under the ifdef.
2025-09-12 04:02:07 +03:00
Sergei Zimmerman ef5fedbc0d Merge pull request #13936 from xokdvium/empty-list-bindings
libexpr: Make constant Values global constants, move out of EvalState
2025-09-10 23:12:20 +00:00
Sergei Zimmermanandeldritch horrors 5db4b0699c libexpr: Make constant Values global constants, move out of EvalState
These constant Values have no business being in the EvalState in the
first place. The ultimate goal is to get rid of the ugly `getBuiltins`
and its relience (in `createBaseEnv`) on these global constants is getting in the way.

Same idea as in https://git.lix.systems/lix-project/lix/commit/f017f9ddd336e32a5ed1ee835f1c6c7e73a052ae.

Co-authored-by: eldritch horrors <pennae@lix.systems>
2025-09-11 01:53:41 +03:00
Sergei Zimmerman 462b9ac49c libexpr: Make Value::isa and Value::getStorage private methods
This was always intended to be the case, but accidentally left
in the public interface.
2025-09-11 01:52:17 +03:00
Sergei Zimmermanandeldritch horrors 4df1a3ca76 libexpr: Make emptyBindings a global constant
This object is always constant and will never get modified.
Having it as a global (constant) static is much easier and
unclutters the EvalState.

Same idea as in https://git.lix.systems/lix-project/lix/commit/f017f9ddd336e32a5ed1ee835f1c6c7e73a052ae.

Co-authored-by: eldritch horrors <pennae@lix.systems>
2025-09-11 01:51:48 +03:00
Philipp Otterbein 9dbc2cae4f hashmaps with string keys: add transparent lookups 2025-09-10 23:04:44 +02:00
Philipp Otterbein 9f2b6a1b94 replace more std::unordered_* types by faster boost hash maps 2025-09-10 23:04:44 +02:00
Philipp Otterbein 4f8c50fb77 libexpr: replace std::unordered_* types by faster boost hash maps 2025-09-10 23:04:44 +02:00
John Ericson c0fd9146d6 Move exportPaths() / importPaths() out of the Store class (#13959) 2025-09-10 15:39:20 -04:00
Eelco Dolstra 3898a7343a Merge pull request #13961 from NyCodeGHG/push-prlsssvmxwvl
meson: link to libatomic on powerpc-linux
2025-09-10 19:22:54 +02:00
Marie Ramlow 37eec84bc1 meson: link to libatomic on powerpc-linux
Like 32-bit Arm, 32-bit PowerPC also needs linking against libatomic
because it doesn't support some atomic instructions in hardware.
2025-09-10 18:50:35 +02:00
Eelco Dolstra fe5b669534 Move exportPaths() / importPaths() out of the Store class 2025-09-10 14:22:46 +02:00
Jörg Thalheim 5e46df973f Merge pull request #13957 from NixOS/drop-old-serve-protocol
Remove support for serve protocol version < 5
2025-09-10 13:50:12 +02:00
Eelco Dolstra 9df99e0658 Remove ServeProto::Command::ExportPaths
This seems to have been unused since the build-remote.pl removal in February 2017 (27dc76c1a5).
2025-09-10 10:57:15 +02:00
Eelco Dolstra fa048e4383 Remove support for serve protocol < 5
This was introduced in August 2018 (2825e05d21).
2025-09-10 10:57:15 +02:00
Jörg Thalheim 8c789db05b Merge pull request #13956 from NixOS/drop-unused-addMultipleToStoreLegacy
Drop unused LegacySSHStore::addMultipleToStoreLegacy()
2025-09-10 10:52:05 +02:00
Eelco Dolstra f8b15bfc7f Merge pull request #13951 from NixOS/drop-old-daemon-protocol
Remove support for worker protocol version < 18
2025-09-10 10:43:48 +02:00
Eelco Dolstra 5013b38df4 Drop unused LegacySSHStore::addMultipleToStoreLegacy() 2025-09-10 10:27:07 +02:00
Eelco Dolstra 247d16a530 Merge pull request #13954 from xokdvium/empty-list-elems-fix
libexpr: Fix Value::mkList for empty lists
2025-09-10 10:23:49 +02:00
Eelco Dolstra d1d3ed6241 Add release note 2025-09-10 10:20:37 +02:00
Jörg Thalheim 7d26bf8cc7 Merge pull request #13906 from obsidiansystems/derivation-builder-simpler
More `DerivationBuilder` simplifications
2025-09-10 09:58:07 +02:00
Jörg Thalheim 9c186c35fa Merge pull request #13932 from NixOS/move-pathInfoCache
Reduce false sharing between pathInfoCache and Store
2025-09-10 09:56:46 +02:00
Sergei Zimmerman 2ed2c79721 libexpr: Fix Value::mkList for empty lists
This code used to save the pointer to a small
list allocated on the stack to the Value, which
is unintended.
2025-09-10 01:37:39 +03:00
Sergei Zimmerman 3c331b7ef3 Merge pull request #13953 from xokdvium/value-alignment
libexpr: Overalign Value to 16 bytes
2025-09-09 20:44:43 +00:00
Sergei Zimmerman 4524235af4 libexpr: Overalign Value to 16 bytes
This is necessary to make use of 128 bit atomics on x86_64 [1],
since MOVAPD, MOVAPS, and MOVDQA need memory operands to be 16-byte
aligned. We are not losing anything here, because Value is already 16-byte
wide and Boehm allocates memory in granules that are 16 bytes by default
on 64 bit systems [2].

[1]: https://patchwork.sourceware.org/project/gcc/patch/YhxkfzGEEQ9KHbBC@tucnak/
[2]: https://github.com/bdwgc/bdwgc/blob/54ac18ccbc5a833dd7edaff94a10ab9b65044d61/include/gc/gc_tiny_fl.h#L31-L33
2025-09-09 22:18:52 +03:00
Eelco Dolstra 86d19956f2 Remove WorkerProto::Op::ImportPaths
This was obsoleted in May 2016 (538a64e8c3).
2025-09-09 15:41:17 +02:00
Eelco Dolstra 4fb61bc5af Remove WorkerProto::Op::ExportPath
This was obsoleted in May 2016 (538a64e8c3).
2025-09-09 15:41:12 +02:00
Eelco Dolstra 137a55122c Remove support for daemon protocol version < 18
Version 18 was introduced in November 2016 (4b8f1b0ec0).
2025-09-09 15:41:01 +02:00
Eelco Dolstra 7658f00bb1 Merge pull request #13941 from NixOS/dependabot/github_actions/actions/labeler-6
build(deps): bump actions/labeler from 5 to 6
2025-09-09 09:37:08 +02:00
Eelco Dolstra a97c5df47c Merge pull request #13939 from DeterminateSystems/fix-inputs-from-ignoring-dir-param-upstreaming
Pass `dir` in extraAttrs when overriding the registry
2025-09-09 09:36:33 +02:00
Sergei Zimmerman 371623bf0c Merge pull request #13940 from xokdvium/unbracketed-ipv6
libstore: Reallow unbracketed IPv6 addresses in store references
2025-09-08 23:21:34 +00:00
dependabot[bot] 7128abd217 build(deps): bump actions/labeler from 5 to 6
Bumps [actions/labeler](https://github.com/actions/labeler) from 5 to 6.
- [Release notes](https://github.com/actions/labeler/releases)
- [Commits](https://github.com/actions/labeler/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/labeler
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-08 22:00:58 +00:00
Sergei Zimmerman 7cc654afa9 libstore: Reallow unbracketed IPv6 addresses in store references
This implements a special back-compat shim to specifically allow
unbracketed IPv6 addresses in store references. This is something
that is relied upon in the wild and the old parsing logic accepted
both ways (brackets were optional). This patch restores this behavior.
As always, we didn't have any tests for this.

Addresses #13937.
2025-09-09 00:41:03 +03:00
Cole Helbling 38663fb434 Pass dir in extraAttrs when overriding the registry
This is handled similarly in the handler for `--override-flake` in
`MixEvalArgs`.
2025-09-08 09:00:59 +02:00
Cole Helbling ed6ef7cdf4 Test that using --inputs-from with a flakeref that has a dir works
Will not pass until the next commit.
2025-09-08 09:00:59 +02:00
Eelco Dolstra 12db0726e9 Merge pull request #13934 from DeterminateSystems/fix-flake-registry-ignoring-dir-param-upstreaming
Fix flake registry ignoring `dir` parameter
2025-09-08 07:50:54 +02:00
Eelco Dolstra 525245181a Merge pull request #13933 from NixOS/local-store-state
LocalStore::State: Put behind a ref to reduce false sharing
2025-09-08 06:22:04 +02:00
Eelco Dolstra 9302ec5e0e Add comment 2025-09-08 05:57:02 +02:00
Cole Helbling 9c832a08b0 fixup: cached case
I couldn't come up with a test that failed before this, but my existing
test still passes so 🤷
2025-09-07 19:40:24 +02:00
Cole Helbling bccdb95a86 Fix flake registry ignoring dir parameter
This broke in e3042f10af.
2025-09-07 19:40:24 +02:00
Cole Helbling 258d41bfb6 Test that dir is propagated from registry entry 2025-09-07 19:40:23 +02:00
Tom Westerhout dbc235cc62 Generalize recognized git url schemas (#13925)
Use `parseUrlScheme` instead of manually parsing `url.scheme`.
2025-09-07 15:22:20 +02:00
Eelco Dolstra df9b3bfba8 Merge pull request #13845 from NixOS/nix-flake-check-build-test
Add a test for `nix flake check` building checks
2025-09-07 14:53:19 +02:00
Eelco Dolstra 14c001d613 Add a test for nix flake check building checks 2025-09-07 14:41:40 +02:00
Eelco Dolstra e791ede495 LocalStore::State: Put behind a ref to reduce false sharing 2025-09-07 14:32:24 +02:00
Eelco Dolstra a73cf447ac Reduce false sharing between pathInfoCache and Store
`perf c2c` shows a lot of cacheline conflicts between purely read-only
Store methods (like `parseStorePath()`) and the Sync classes. So
allocate pathInfoCache separately to avoid that.
2025-09-07 14:27:38 +02:00
Eelco Dolstra 9ff427d7ba Merge pull request #13911 from xokdvium/store-uri-daemon-local
libstore: Do not normalize daemon -> unix://, local -> local://
2025-09-07 14:10:45 +02:00
Eelco Dolstra 4dd27a292c Merge pull request #13929 from NixOS/remove-unused
Remove unused function setChildSignalMask()
2025-09-07 13:30:09 +02:00
Jörg Thalheim 5ae1b5f88b Merge pull request #13916 from sinanmohd/fix/develop-interactive-shell
nix/develop: pass down the interactive shell to subshells
2025-09-07 10:14:25 +02:00
Eelco Dolstra a7c6a42344 Merge pull request #13923 from NixOS/fix-multithreaded-chroot-hang
Fix hang in enterChroot() draining userNamespaceSync
2025-09-07 09:13:54 +02:00
Eelco Dolstra f363d958a7 Fix hang in enterChroot() draining userNamespaceSync
Calling `drainFD()` will hang if another process has the write side
open, since then the child won't get an EOF. This can happen if we
have multiple threads doing a build, since in that case another thread
may fork a child process that inherits the write side of the first
thread.

We could set O_CLOEXEC on the write side (using pipe2()) but it won't
help here since we don't always do an exec() in the child, e.g. in the
case of builtin builders. (We need a "close-on-fork", not a
"close-on-exec".)
2025-09-07 01:12:44 +02:00
Eelco Dolstra a44dcbff13 Remove unused function setChildSignalMask() 2025-09-06 23:02:57 +02:00
John Ericson 12b6d8d208 Merge pull request #13924 from xokdvium/dead-code
libexpr: Remove decl for undefined overload of Value::mkPath
2025-09-06 10:47:26 -04:00
Sergei Zimmerman bbdabe4973 libexpr: Remove decl for undefined overload of Value::mkPath 2025-09-06 16:36:16 +03:00
Jörg Thalheim 1d62ccdb3d Merge pull request #13767 from ethanavatar/master
libutil, libexpr: #10542 abstract over getrusage for getting cpuTime stat and implement windows version
2025-09-06 09:26:13 +02:00
Jörg Thalheim 533c6d38aa Merge pull request #13901 from Mic92/fix-macos-hup-detection
Fix macOS HUP detection using kqueue instead of poll
2025-09-06 09:19:51 +02:00
Jörg Thalheim dbc8d0ab64 Merge pull request #13919 from xokdvium/smaller-bindings
libexpr: Slim down Bindings to 8 bytes (on 64 bit systems)
2025-09-06 09:11:48 +02:00
Sergei Zimmerman 738924b705 libexpr: Slim down Bindings to 8 bytes (on 64 bit systems)
Since the only construction and push_back() calls
to Bindings happen through the `BindingsBuilder` [1] we don't
need to keep `capacity` around on the heap anymore. This saves 8 bytes
(because of the member alignment padding)
per one Bindings allocation. This isn't that much, but it does
save significant memory.

This also shows that the Bindings don't necessarily have to
be mutable, which opens up opportunities for doing small bindings
optimization and storing a 1-element Bindings directly in Value.

For the following scenario:

nix-env --query --available --out-path --file ../nixpkgs --eval-system x86_64-linux

(nixpkgs revision: ddcddd7b09a417ca9a88899f4bd43a8edb72308d)

This patch results in reduction of `sets.bytes` 13115104016 -> 12653087640,
which amounts to 462 MB less bytes allocated for Bindings.

[1]: Not actually, `getBuiltins` does mutate bindings, but this is pretty
     inconsequential and doesn't lead to problems.
2025-09-06 00:23:54 +03:00
sinanmohd 211cbe4abf nix/develop: pass down the interactive shell to subshells 2025-09-05 20:18:25 +05:30
Sergei Zimmerman 3513ab13dc libstore: Do not normalize daemon -> unix://, local -> local://
This is relied upon (specifically the `local` store) by existing
tooling [1] and we broke this in 3e7879e6df (which
was first released in 2.31).

To lessen the scope of the breakage we should not normalize "auto" references
and explicitly specified references like "local" or "daemon". It also makes
sense to canonicalize local://,daemon:// to be more compatible with prior
behavior.

[1]: https://github.com/maralorn/nix-output-monitor/blob/05e1b3cba2fa328a1781390a4e4515e9c432229e/lib/NOM/Builds.hs#L60-L64
2025-09-05 04:14:36 +03:00
John Ericson 49e9c14e2f Merge pull request #13900 from NixOS/fix-mingw-windows-build
Fix downstream MinGW build by not looking for Boost Regex
2025-09-04 21:05:12 -04:00
John Ericson 25d3c197b8 Merge pull request #13902 from NixOS/ssh-master-deadlock
Fix deadlock in SSHMaster::addCommonSSHOpts()
2025-09-03 21:44:06 -04:00
John Ericson 2acb9559d5 Combine DerivationBuilder::{prepareBuild,startBuilder}
After many other cleanups, it turns out there is no reason for these to
be separate methods. We can combine them to simplify things.
2025-09-03 17:58:50 -04:00
John Ericson 14c206f05a DerivationBuilder no more callback soup for logging
`startBuilder` just returns the descriptor for the pipe now.
2025-09-03 17:34:45 -04:00
John Ericson 7f3314a68c DerivationBuilder::initialOutputs make const
At one point I remember it did mutatate `initialOutputs`, but not
anymore!
2025-09-03 17:34:45 -04:00
John Ericson b69576e2b3 Merge pull request #13905 from obsidiansystems/derivation-building-goal-simplify-0
Derivation building goal simplify -- no `goto`
2025-09-03 17:34:27 -04:00
John Ericson 7b22cd5105 Merge pull request #13839 from Mic92/infra
don't include derivation name in temporary build directories
2025-09-03 17:15:03 -04:00
Jörg Thalheim 1732b4a61b Merge pull request #13885 from netadr/fix-ssh-key-ids
libfetchers: Fix SSH key types for sk type keys
2025-09-03 23:13:48 +02:00
John Ericson 819bf13607 Merge pull request #13880 from Mic92/static-alloc-symbol-ids
libexpr: Convert Symbol comparisons to switch statements
2025-09-03 17:13:12 -04:00
Jörg Thalheim 81e068ab8a Merge pull request #13904 from NixOS/c-ffi-improvements
C ffi improvements
2025-09-03 23:10:27 +02:00
John Ericson a30bf96349 DerivationBuildingGoal::initialOutputs make local variable
Also inline `assertPathValidity` in the process.
2025-09-03 17:04:13 -04:00
John Ericson c0c2a89f05 DerivationBuildingGoal::initialOutputs move initialization down to tryToBuild
Will help us make this a local variable.
2025-09-03 17:04:08 -04:00
John Ericson 450633aa8c Move machineName from DerivationBuildingGoal to HookInstance
Exactly why is is correct is a little subtle, because sometimes the
worker is owned by the worker. But the commit message in
e437b08250 explained the situation well
enough: I made that commit message part of the ABI docs, and now it
should be understandable to the next person.
2025-09-03 17:03:56 -04:00
netadr 671c21db9f libfetchers: Fix SSH key identifiers for sk type keys
libfetchers: Mark ssh-ecdsa-sk key type mapping as a TODO for now
2025-09-03 22:56:33 +02:00
John Ericson 8089102164 Separate internal from non-internal unit tests of the C API
This helps us make sure that the external C API is sufficient for the
tasks that we think it is sufficient for.
2025-09-03 22:50:42 +02:00
John Ericson f6bc47bc50 nix_store_realise: Improve typing of store path
Use `StorePath *` not `const char *`.
2025-09-03 22:50:42 +02:00
John Ericson fa76b6e215 nix store benchmarks: Only get unit test dir from env var 2025-09-03 22:50:42 +02:00
John Ericson 44d096f68d nix_store_is_valid_path param path should be const 2025-09-03 22:50:42 +02:00
John Ericson 7e4608a3f8 More extern "C" for FFI
This allows us to catch the header and file getting out of sync, because
we are not doing overloading by mistake.
2025-09-03 22:50:42 +02:00
John Ericson eb56b181ae DerivationBuildingGoal: Make almost everything private 2025-09-03 16:25:12 -04:00
John Ericson c6ba120000 DerivationBuildingGoal::started make local (lambda) variable 2025-09-03 16:19:35 -04:00
John Ericson 3b9c510ab1 DerivationBuildingGoal::outputLocks make local variable 2025-09-03 16:19:35 -04:00
John Ericson a63ac8d98b Inline DerivationBuildingGoal::hookDone 2025-09-03 16:19:35 -04:00
John Ericson 51dadaded4 Move up assert(!hook);
We don't need to keep doing this every loop iteration, hook stuff it is only set
above.
2025-09-03 16:19:35 -04:00
John Ericson 7c1e5b3345 In DerivationBuildingGoal Demote actLock to local variable
It doesn't need to be a field any more, because we just use it with two
loops.
2025-09-03 16:19:35 -04:00
John Ericson 4c44a213a3 Get rid of a tryToBuild tail recursive call with loop
This will make it easier to convert somethings to RAII.
2025-09-03 16:19:35 -04:00
John Ericson 95c5779880 DerivationBuildingGoal::tryToBuild pull hook waiting out of switch
Do this with a new `useHook` boolean we carefully make sure is set in
all cases. This change isn't really worthwhile by itself, but it allows
us to make further refactors (see later commits) which are
well-motivated.
2025-09-03 16:19:35 -04:00
Eelco Dolstra c7603c61c8 Mark tmpDir as const 2025-09-03 20:17:42 +02:00
Eelco Dolstra 2fe629c5d4 Fix deadlock in SSHMaster::addCommonSSHOpts()
When useMaster is true, startMaster() acquires the state lock, then
calls isMasterRunning(), which calls addCommonSSHOpts(), which tries
to acquire the state lock again, causing a deadlock.

The solution is to move tmpDir out of the state. It doesn't need to be
there in the first place because it never changes.
2025-09-03 17:49:24 +02:00
Jörg ThalheimandJade Lovelace 1286d5db78 Fix macOS HUP detection using kqueue instead of poll
On macOS, poll() is fundamentally broken for HUP detection. It loses event
subscriptions when EVFILT_READ fires without matching the requested events
in the pollfd. This causes daemon processes to linger after client disconnect.

This commit replaces poll() with kqueue on macOS, which is what poll()
uses internally but without the bugs. The kqueue implementation uses
EVFILT_READ which works for both sockets and pipes, avoiding EVFILT_SOCK
which only works for sockets.

On Linux and other platforms, we continue using poll() with the standard
POSIX behavior where POLLHUP is always reported regardless of requested events.

Based on work from the Lix project (https://git.lix.systems/lix-project/lix)
commit 69ba3c92db3ecca468bcd5ff7849fa8e8e0fc6c0

Fixes: https://github.com/NixOS/nix/issues/13847
Related: https://git.lix.systems/lix-project/lix/issues/729
Apple bugs: rdar://37537852 (poll), FB17447257 (poll)

Co-authored-by: Jade Lovelace <jadel@mercury.com>
2025-09-03 11:33:23 +02:00
Jörg ThalheimandJohn Ericson cbcb434cb3 libexpr: Convert Symbol comparisons to switch statements
Now that Symbols are statically allocated at compile time with known IDs,
we can use switch statements instead of if-else chains for Symbol comparisons.
This provides better performance through compiler optimizations like jump tables.

Changes:
- Add public getId() method to Symbol class to access the internal ID
- Convert if-else chains comparing Symbol values to switch statements
  in primops.cc's derivationStrictInternal function
- Simplify control flow by removing the 'handled' flag and moving the
  default attribute handling into the switch's default case

The static and runtime Symbol IDs are guaranteed to match by the
copyIntoSymbolTable implementation which asserts this invariant.

Co-authored-by: John Ericson <git@JohnEricson.me>
2025-09-03 10:13:12 +02:00
Sergei Zimmerman 1935c19705 Merge pull request #13890 from xokdvium/mkstring-no-copy
Re-introduce mkStringNoCopy (revised)
2025-09-02 17:13:17 +00:00
John Ericson 6bdb5e8e09 Fix downstream MinGW build by not looking for Boost Regex 2025-09-02 10:41:39 -04:00
John Ericson b806440808 Merge pull request #13894 from NixOS/more-url-testing
More URL testing
2025-09-02 00:10:13 -04:00
John Ericson 7f91e91876 More URL testing
More parameterized tests, we can have more coverage.
2025-09-01 18:26:21 -04:00
John Ericson ab095c029c Merge pull request #13891 from NixOS/another-url-test
Add another `fixGitURL` test
2025-09-01 17:36:18 -04:00
John Ericson 7195250fc4 Add another fixGitURL test
Also improve a similar `parseURL` test.
2025-09-01 17:19:26 -04:00
Sergei Zimmerman 34181afc6a libexpr: Use mkStringNoCopy in prim_typeOf
This would lead to an unnecessary allocation. Not
a significant issue by any means, but it doesn't
have to allocate for most cases.
2025-09-02 00:16:11 +03:00
Eelco Dolstra d62cfc1c97 Re-introduce mkStringNoCopy (revised)
In b70d22b `mkStringNoCopy()` was renamed to
`mkString()`, but this is a bit risky since in code like

    vStringRegular.mkString("regular");

we want to be sure that the right overload is picked. (This is
especially problematic since the overload that takes an
`std::string_view` *does* allocate.)  So let's be explicit.

(Rebased from https://github.com/NixOS/nix/pull/11551)
2025-09-02 00:16:06 +03:00
John Ericson 3a19ea96d9 Merge pull request #13888 from NixOS/old-busted-git-url-with-tests
Old busted git url with tests
2025-09-01 16:46:01 -04:00
Farid Zakaria 2b310aee13 A few more URL tests
Adapted from commit 04ad66af5f
2025-09-01 16:31:45 -04:00
John Ericson d2f1860ee5 Revert "Improve Git URI handling"
I (@Ericson2314) messed up. We were supposed to test the status quo
before landing any new chnages, and also there is one change that is not
quite right (relative paths).

I am reverting for now, and then backporting the test suite to the old
situation.

This reverts commit 04ad66af5f.
2025-09-01 16:13:32 -04:00
Jörg Thalheim a0ce514769 Merge pull request #13866 from obsidiansystems/more-derivation-builder-cleanup
Even more `DerivationBuilder` cleanup
2025-09-01 20:35:16 +02:00
Jörg Thalheim 0d300112fa Merge pull request #13862 from obsidiansystems/build-failure-content-vs-presentation
Properly separater builder failure content and presentation
2025-09-01 20:25:50 +02:00
Jörg Thalheim de7f137f31 Merge pull request #13860 from obsidiansystems/derivation-building-resources-code-cleanup
Derivation building resources code cleanup
2025-09-01 20:22:30 +02:00
John Ericson 7fde4f7d6f Merge pull request #13821 from fzakaria/fzakaria/improve-fixgiturl
Improve Git URI handling
2025-09-01 14:15:17 -04:00
Jörg Thalheim dc29cdf66d Merge pull request #13858 from obsidiansystems/no-more-defered-exception
Get rid of `delayedException` in `DerivationBuilder`
2025-09-01 20:11:51 +02:00
Jörg Thalheim 3e0fb3f8d2 Merge pull request #13881 from xokdvium/pass-url-verbatim
lib{store,fetchers}: Pass URLs specified directly verbatim to FileTra…
2025-09-01 20:06:50 +02:00
Farid Zakaria 04ad66af5f Improve Git URI handling
Git URI can also support scp style links similar to git itself.

This change augments the function fixGitURL to better handle the scp
style urls through a minimal parser rather than regex which has been
found to be brittle.

* Support for IPV6 added
* New test cases added for fixGitURL
* Clearer documentation on purpose and goal of function
* More `std::string_view` for performance
* A few more URL tests

Fixes #5958
2025-09-01 14:04:04 -04:00
Jörg Thalheim fea4a29c0a Merge pull request #13883 from xokdvium/toml-timestamps-reapply
Reapply "Merge pull request #13741 from xokdvium/toml-timestamps"
2025-09-01 09:12:33 +02:00
Sergei Zimmerman e548700010 lib{store,fetchers}: Pass URLs specified directly verbatim to FileTransferRequest
The URL should not be normalized before handing it off to cURL, because
builtin fetchers like fetchTarball/fetchurl are expected to work with
arbitrary URLs, that might not be RFC3986 compliant. For those cases
Nix should not normalize URLs, though validation is fine. ParseURL and
cURL are supposed to match the set of acceptable URLs, since they implement
the same RFC.
2025-09-01 02:22:23 +03:00
Emily acd627fa46 tests/functional/lang: Add tests for builtins.fromTOML overflow
This adds regression tests for fromTOML overflow/underflow behavior.
Previous versions of toml11 used to saturate, but this was never an
intended behavior (and Snix/Nix 2.3/toml11 >= 4.0 validate this).

(cherry picked from Lix [1,2])

[1]: https://git.lix.systems/lix-project/lix/commit/7ee442079dfbae4fe2d2fbb91a7226b87251cd65
[2]: https://git.lix.systems/lix-project/lix/commit/4de09b6b5493db4bd7f6348255a1fdcb38b9ed2f
2025-09-01 01:49:15 +03:00
Sergei Zimmerman 8251305aff Reapply "Merge pull request #13741 from xokdvium/toml-timestamps"
This reverts commit 75740fbd75.
2025-09-01 01:26:14 +03:00
Jörg Thalheim 73cdfe7066 Merge pull request #13878 from urbas/hacking-instructions
hacking.md: set installation outputs as well
2025-08-31 13:58:36 +02:00
Jörg Thalheim 1f7d43e5bd Merge pull request #13879 from xokdvium/static-alloc-symbol-ids
libexpr: Statically allocate commonly used symbols
2025-08-31 13:40:55 +02:00
Sergei Zimmermanandeldritch horrors 363620dd24 libexpr: Statically allocate commonly used symbols
The motivation for this change is two-fold:

1. Commonly used Symbol values can be referred to
   quite often and they can be assigned at compile-time
   rather than runtime.

2. This also unclutters EvalState constructor, which was
   getting very long and unreadable.

Spiritually similar to https://gerrit.lix.systems/c/lix/+/2218,
though that patch doesn't allocate the Symbol at compile time.

Co-authored-by: eldritch horrors <pennae@lix.systems>
2025-08-31 13:24:06 +02:00
Matej Urbas 112f311c50 hacking.md: set installation outputs as well 2025-08-31 09:53:14 +01:00
John Ericson 2746985d90 Merge pull request #13877 from xokdvium/opt-print-string
libstore: Get rid of allocations in printString, allocate 2K bytes on the stack
2025-08-31 00:42:18 -04:00
Sergei Zimmerman e1c9bc0ef6 libstore: Get rid of allocations in printString, allocate 2K bytes on the stack
Looking at perf:

   0.21 │       push   %rbp
   0.99 │       mov    %rsp,%rbp
        │       push   %r15
   0.25 │       push   %r14
        │       push   %r13
   0.49 │       push   %r12
   0.66 │       push   %rbx
   1.23 │       lea    -0x10000(%rsp),%r11
   0.23 │ 15:   sub    $0x1000,%rsp
   1.01 │       orq    $0x0,(%rsp)
  59.12 │       cmp    %r11,%rsp
   0.27 │     ↑ jne    15

Seems like 64K is too much to have on the stack for each invocation, considering
that only a minuscule number of allocations are actually larger than 4K.

There's actually no good reason this function should use so much stack space. Or
use small_string at all. Everything can be done in small chunks that don't require
any memory allocations and use up 2K bytes on the stack.

This patch also adds a microbenchmark for tracking the unparsing performance. Here
are the results for this change:

(Before)

BM_UnparseRealDerivationFile/hello         7275 ns         7247 ns        96093 bytes_per_second=232.136Mi/s
BM_UnparseRealDerivationFile/firefox      40538 ns        40376 ns        17327 bytes_per_second=378.534Mi/s

(After)

BM_UnparseRealDerivationFile/hello         3228 ns         3218 ns       215671 bytes_per_second=522.775Mi/s
BM_UnparseRealDerivationFile/firefox      39724 ns        39584 ns        17617 bytes_per_second=386.101Mi/s

This translates into nice evaluation performance improvements (compared to 18c3d2348f):

Benchmark 1: GC_INITIAL_HEAP_SIZE=8G old-nix/bin/nix-instantiate ../nixpkgs -A nixosTests.gnome --readonly-mode
  Time (mean ± σ):      3.111 s ±  0.021 s    [User: 2.513 s, System: 0.580 s]
  Range (min … max):    3.083 s …  3.143 s    10 runs

Benchmark 2: GC_INITIAL_HEAP_SIZE=8G result/bin/nix-instantiate ../nixpkgs -A nixosTests.gnome --readonly-mode
  Time (mean ± σ):      3.037 s ±  0.038 s    [User: 2.461 s, System: 0.558 s]
  Range (min … max):    2.960 s …  3.086 s    10 runs
2025-08-31 00:48:37 +03:00
John Ericson 18c3d2348f Merge pull request #13875 from xokdvium/restore-weird-flakeref-path
libfetchers: Restore path separator ignoring behavior for indirect an…
2025-08-30 11:17:42 -04:00
Sergei Zimmerman a38ebdd511 libfetchers: Restore path separator ignoring behavior for indirect and git-archive flakerefs
Old versions of nix happily accepted a lot of weird flake references,
which we didn't have tests for, so this was accidentally broken in
c436b7a32a.

This patch restores previous behavior and adds a plethora of tests
to ensure we don't break this in the future.

These test cases are aligned with how 2.18/2.28 parsed flake references.
2025-08-30 14:40:56 +03:00
John Ericson 401e7fe3ad Merge pull request #13873 from xokdvium/fix-mingw
libfetchers: Fix mingw build
2025-08-29 19:56:20 -04:00
Sergei Zimmerman b88a22504f libfetchers: Fix mingw build 2025-08-30 02:36:16 +03:00
John Ericson 511d885d60 Merge pull request #13872 from xokdvium/fix-indirect-flake-refs
libflake: Fix flake id flake refs with revisions
2025-08-29 18:51:25 -04:00
Sergei Zimmerman 3ef3f525c3 libflake: Fix flake id flake refs with revisions
Starting from c436b7a32a
this used to lead to assertion failures like:

> std::string nix::ParsedURL::renderAuthorityAndPath() const: Assertion `path.empty() || path.front().empty()' failed.

This has the bugfix for the issue and regressions tests
so that this gets properly tested in the future.
2025-08-30 01:26:51 +03:00
John Ericson 53a7d87b93 Merge pull request #13871 from obsidiansystems/fix-refactor-bug
`DerivationBuildingGoal::done*` restore `outputLocks.unlock()`
2025-08-29 18:25:43 -04:00
John Ericson a8c4cfae26 DerivationBuildingGoal::done* restore outputLocks.unlock()
This was accidentally removed in
169033001d.
2025-08-29 17:49:11 -04:00
John Ericson d50d4b01c7 Merge pull request #13867 from xokdvium/fix-13482
nix/develop: Fix misleading ignored error when run with --arg/--argstr
2025-08-29 17:17:25 -04:00
Sergei Zimmerman b6f98b52a4 nix/develop: Fix misleading ignored error when run with --arg/--argstr
This would print erroneous and misleading diagnostics like:

> error (ignored): error: '--arg' and '--argstr' are incompatible with flakes

When run with --expr/--file. Since this installable is used to get the
bash package it doesn't make sense to check this.
2025-08-30 00:03:54 +03:00
John Ericson d7ed86ceb1 Move deleting redirected outputs in to cleanupBuild
It is only done in the `force = true` case, and the only
`cleanupBuild(true)` call is right after where it used to be, so this
has the exact same behavior as before.
2025-08-29 16:10:25 -04:00
John Ericson 76125f8eb1 Get rid of Finally in DerivationBuilderImpl::unprepareBuild
Calling `reset` on this `std::optional` field of `DerivationBuilderImpl`
is also what the (automatically created) destructor of
`DerivationBuilderImpl` will do. We should be making sure that the
derivation builder is cleaned up by the goal anyways, and if we do that,
then this `Finally` is no longer needed.
2025-08-29 13:22:36 -04:00
Jörg Thalheim 0d006aedd6 Merge pull request #13854 from obsidiansystems/register-outputs-slight-simplify
Simplify handling of statuses for build errors
2025-08-29 07:20:55 +02:00
Jörg Thalheim 04d2122de2 Merge pull request #13861 from xokdvium/terminate-for-unreachable
libutil: Try to call std::terminate for panic, use C++20 std::source_location
2025-08-29 07:15:49 +02:00
John Ericson 8825bfa7fe Properly separater builer failure content and presentation
Before, had a very ugly `appendLogTailErrorMsg` callback. Now, we
instead have a `fixupBuilderFailureErrorMessage` that is just used by
`DerivationBuildingGoal`, and `DerivationBuilder` just returns the raw
data needed by this.
2025-08-28 22:17:15 -04:00
Sergei Zimmerman d59b959c87 libutil: Use std::source_location for unreachable
Make unreachable a function instead of a macro, since
C++20 provides a convenience class as a replacement for
older __FILE__, __LINE__ macros.
2025-08-29 00:21:07 +03:00
John Ericson 47cae1f72b Merge pull request #13850 from obsidiansystems/factor-out-drv-env-desugar
Factor out a new `DesugaredEnv` from `DerivationBuildingGoal`
2025-08-28 17:10:48 -04:00
Sergei Zimmerman 1f607b5def libutil: Try to call std::terminate for panic
We now have a terminate handler that prints a
stack trace, which is useful to have when encountering
an unreachable.
2025-08-29 00:02:13 +03:00
John EricsonandJeremy Kolb 53c31c8b29 Factor out a new DesugaredEnv from DerivationBuildingGoal
Now we have better separation of the core logic --- an integral part of
the store layer spec even --- from the goal mechanism and other
minutiae.

Co-authored-by: Jeremy Kolb <kjeremy@gmail.com>
2025-08-28 16:45:45 -04:00
Sergei Zimmerman 731349639f Merge pull request #13524 from gmarti/fix_cacertificate
Add /etc/ssl/certs/ca-certificates.crt in docker.nix
2025-08-28 23:28:53 +03:00
John Ericson f019f1b75a Merge pull request #13838 from NixOS/parse-url-path
Fix `ParsedURL` handling of `%2F` in URL paths
2025-08-28 16:07:25 -04:00
c436b7a32a Fix ParsedURL handling of %2F in URL paths
See the new extensive doxygen in `url.hh`.
This fixes fetching gitlab: flakes.

Paths are now stored as a std::vector of individual path
segments, which can themselves contain path separators '/' (%2F).
This is necessary to make the Gitlab's /projects/ API work.

Co-authored-by: John Ericson <John.Ericson@Obsidian.Systems>
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-08-28 22:20:04 +03:00
Sergei Zimmerman 6839f3de55 libutil-tests: Add more URL tests 2025-08-28 14:58:17 -04:00
John Ericson 3e0b1705c1 Move markContentsGood to after DerivationBuilder finishes
I think this should be fine for repairing. If anything, it is better,
because it would be weird to "mark and output good" only for it to then
fail output checks.
2025-08-28 14:54:11 -04:00
Bernardo Meurer Costa f193bca595 feat(libstore): warn when kvm is enabled but /dev/kvm isn't available 2025-08-28 18:44:28 +00:00
Sergei Zimmerman c2782d7b84 Merge pull request #13853 from obsidiansystems/no-old-debugging-aid
Revert "Add a crude tracing mechansim for the build results"
2025-08-28 21:18:10 +03:00
John Ericson bde745cb3f Move killChild call from ~DerivationBuildingGoal to ~DerivationBuilder
Sadly we cannot unexpose `DerivationBuilder::killChild` yet, because
`DerivationBuildingGoal` calls it elsewhere, but we can at least haave a
better division of labor between the two destructors.
2025-08-28 14:01:24 -04:00
John Ericson c632c823ce Take DerivationBuilder::pid private 2025-08-28 14:01:20 -04:00
John Ericson 4388e3dcb5 Create DerivationBuilder::killChild
Then the derivation building goal doesn't need to snoop around as much.
2025-08-28 14:01:17 -04:00
John Ericson 49da508f46 Write a destructor for DerivationBuilderImpl
This allows `DerivationBuildingGoal` to know less.
2025-08-28 14:01:14 -04:00
John Ericson 557bbe969e Combine cleanupBuild and deleteTmpDir
It's hard to tell if I changed any behavior, but if I did, I think I
made it better, because now we explicitly move stuff out of the chroot
(if we were going to) before trying to delete the chroot.
2025-08-28 14:01:11 -04:00
John Ericson 4db6bf96b7 Give DerivationBuilderImpl::cleanupBuild bool arg
Do this to match `DerivationBuilder::deleteTmpDir`, which we'll want to
combine it with next.

Also chenge one caller from `deleteTmpDir(true)` to `cleanupBuild(true)`
now that this is done, because it will not make a difference.

This should be a pure refactor with no behavioral change.
2025-08-28 14:01:08 -04:00
John Ericson 8dd289099c Simplify DerivationGoal::unprepareBuild::diskFull
We only need it defined in the narrower scope
2025-08-28 14:01:05 -04:00
John Ericson 374f8e79a1 DerivationBuilderImpl::unprepareBuild Just throw error
Aftet the previous simplifications, there is no reason to catch the
error and immediately return it with a `std::variant` --- just let the
caller catch it instead.
2025-08-28 14:00:35 -04:00
John Ericson 0b85b023d8 Get rid of delayedException in DerivationBuilder
Instead of that funny business, the fixed output checks are not put in
`checkOutputs`, with the other (newer) output checks, where they also
better belong. The control flow is reworked (with comments!) so that
`checkOutputs` also runs in the `bmCheck` case.

Not only does this preserve existing behavior of `bmCheck`
double-checking fixed output hashes with less tricky code, it also makes
`bmCheck` better by also double-checking the other output checks, rather
than just assuming they pass if the derivation is deterministic.
2025-08-28 11:44:18 -04:00
John Ericson ff961fd9e2 Get rid of DerivationBuilder::note*Mismatch
It's fine to set these worker flags a little later in the control flow,
since we'll be sure to reach those points in the error cases. And doing
that is much nicer than having these tangled callbacks.

I originally made the callbacks to meticulously recreate the exact
behavior which I didn't quite understand. Now, thanks to cleaning up the
error handling, I do understand what is going on, so I can be confident
that this change is safe to make.
2025-08-28 11:44:18 -04:00
Sergei Zimmerman 2eacb3c36f Merge pull request #13851 from lovesegfault/http-binary-cache-store-once
refactor(libstore/http-binary-cache-store): pragma once
2025-08-28 03:44:17 +03:00
John Ericson 169033001d Simplify handling of statuses for build errors
Instead of passing them around separately, or doing finicky logic in a
try-catch block to recover them, just make `BuildError` always contain a
status, and make it the thrower's responsibility to set it. This is much
more simple and explicit.

Once that change is done, split the `done` functions of `DerivationGoal`
and `DerivationBuildingGoal` into separate success and failure
functions, which ends up being easier to understand and hardly any
duplication.

Also, change the handling of failures in resolved cases to use
`BuildResult::DependencyFailed` and a new message. This is because the
underlying derivation will also get its message printed --- which is
good, because in general the resolved derivation is not unique. One dyn
drv test had to be updated, but CA (and dyn drv) is experimental, so I
do not mind.

Finally, delete `SubstError` because it is unused.
2025-08-27 20:05:06 -04:00
John Ericson 0590b13156 Revert "Add a crude tracing mechansim for the build results"
The commit says it was added for CA testing --- manual I assume, since
there is no use of this in the test suite. I don't think we need it any
more, and I am not sure whether it was ever supposed to have made it to
`master` either.

This reverts commit 2eec2f765a.
2025-08-27 19:36:02 -04:00
Bernardo Meurer Costa 241abcca86 refactor(libstore/http-binary-cache-store): pragma once 2025-08-27 21:13:59 +00:00
John Ericson 35978ca47b Merge pull request #13848 from obsidiansystems/factor-out-drv-check
Factor out `checkOutputs`
2025-08-27 16:50:38 -04:00
John Ericson d1bdaef04e Factor out checkOutputs
We currently just use this during the build of a derivation, but there is no
reason we wouldn't want to use it elsewhere, e.g. to check the outputs
of someone else's build after the fact.

Moreover, I like pulling things out of `DerivationBuilder` that are
simple and don't need access to all that state. While
`DerivationBuilder` is unix-only, this refactor also make the code more
portable "for free".

The header is private, at Eelco's request.
2025-08-27 16:25:46 -04:00
John Ericson 6c8f5ef9f7 Merge pull request #13802 from obsidiansystems/post-build-hook-later
Move `runPostBuildHook` out of `DerivationBuilder`
2025-08-27 15:48:05 -04:00
John Ericson 193ad73ce2 Merge pull request #13808 from obsidiansystems/derivation-builder-kvm
Create `StringSet DerivationBuilderParams::systemFeatures`
2025-08-27 15:19:06 -04:00
John Ericson f4a0161cb1 Create StringSet DerivationBuilderParams::systemFeatures
Do this to avoid checking "system features" from the store config
directly, because we rather not have `DerivationBuilder` depend on
`Store`.
2025-08-27 12:38:15 -04:00
John Ericson 79211b6110 Merge pull request #13846 from obsidiansystems/derivation-builder-params-aggregate-initialize
No more `DerivationBuilderParams:` constructor!
2025-08-27 12:30:49 -04:00
John Ericson f5f9e32f54 No more DerivationBuilderParams: constructor!
I am not sure how/why this started working. C++23?
2025-08-27 11:40:02 -04:00
Jörg Thalheim 725a2f379f don't include derivation name in temporary build directories
With the migration to /nix/var/nix/builds we now have failing builds
when the derivation name is too long.
This change removes the derivation name from the temporary build to have
a predictable prefix length:

Also see: https://github.com/NixOS/infra/pull/764
for context.
2025-08-27 09:48:31 +02:00
Jörg Thalheim 564593bcb9 Merge pull request #13837 from xokdvium/bump-nixpkgs
flake: Update nixpkgs
2025-08-27 09:33:04 +02:00
Sergei Zimmerman 8ee74792fe Merge pull request #13819 from obsidiansystems/relative-url
Implement `parseURLRelative`, use in `HttpBinaryCacheStore`
2025-08-27 03:34:57 +03:00
John Ericson e82210b3b2 Implement parseURLRelative, use in HttpBinaryCacheStore
This allows us to replace some very hacky and not correct string
concatentation in `HttpBinaryCacheStore`. It will especially be useful
with #13752, when today's hacks started to cause problems in practice,
not just theory.

Also make `fixGitURL` returned a `ParsedURL`.
2025-08-26 19:45:10 -04:00
Sergei Zimmerman 625477a7df flake: Update nixpkgs
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/cd32a774ac52caaa03bcfc9e7591ac8c18617ced?narHash=sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI%3D' (2025-08-17)
  → 'github:NixOS/nixpkgs/d98ce345cdab58477ca61855540999c86577d19d?narHash=sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8%3D' (2025-08-26)

This update contains d1266642a8722f2a05e311fa151c1413d2b9653c, which
is necessary for the TOML timestamps to get tested via nixpkgsLibTests job.
2025-08-27 02:23:05 +03:00
Sergei Zimmerman 231f3af535 Merge pull request #13835 from obsidiansystems/better-string-split
Better `stringSplit`
2025-08-27 01:20:46 +03:00
John EricsonandSergei Zimmerman cc4aa70e6e Better stringSplit
I need this for some `ParseURL` improvements, but I figure this is
better to send as its own PR.

I changed the tests willy-nilly to sometimes use
`std::list<std::string_view>` instead of `Strings` (which is
`std::list<std::string>`).

Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
2025-08-26 18:03:23 -04:00
John Ericson 0bd9d6a28e Merge pull request #13832 from kip93/fix/empty-ports
Handle empty ports with new URL parsing
2025-08-26 13:55:59 -04:00
Leandro Reina 7989e3192d Handle empty ports 2025-08-26 17:41:27 +02:00
Robert Hensing 1e16a54ee5 Merge pull request #13828 from NixOS/readme-meeting-times
Update work meeting time in README
2025-08-26 01:37:07 +02:00
Robert Hensing afade27123 Update work meeting time in README 2025-08-26 00:50:12 +02:00
John Ericson 0250d50df3 Move runPostBuildHook out of DerivationBuilder
It is suppposed to be "post build" not "during the build" after all. Its
location now matches that for the hook case (see elsewhere in
`DerivationdBuildingGoal`).

It was in a try-catch before, and now it isn't, but I believe that it is
impossible for it to throw `BuildError`, which is sufficient for this
code motion to be correct.
2025-08-25 18:29:24 -04:00
Robert Hensing c1e2396d58 Merge pull request #13826 from xokdvium/sqlite-zfs-hack
SQLite: fsync db.sqlite-shm before opening the database
2025-08-26 00:05:40 +02:00
John Ericson ca94905593 Merge pull request #13825 from obsidiansystems/slight-optimize-s3ToHttpsURL
`ParsedS3URL::toHttpsUrl` Slight optimize
2025-08-25 17:51:09 -04:00
Eelco Dolstra e492c64c8e SQLite: fsync db.sqlite-shm before opening the database
This is a workaround for https://github.com/NixOS/nix/issues/13515
(opening the SQLite DB randomly taking a couple of seconds on ZFS).

(cherry picked from commit a7fceb5eec)
2025-08-26 00:42:18 +03:00
John Ericson e4e8a615fa ParsedS3URL::toHttpsUrl Slight optimize
I didn't want to block that PR on further code review while I figured
out these new (to us) C++23 goodies.
2025-08-25 16:53:39 -04:00
John Ericson fac34ad20f Merge pull request #13824 from xokdvium/fix-formatting
libexpr: Fix weird formatting after treewide reformat
2025-08-25 15:26:17 -04:00
John Ericson 024d3954af Merge pull request #13823 from lovesegfault/extract-s3ToHttpsURL
feat(libstore/s3): add toHttpsUrl
2025-08-25 15:11:38 -04:00
Sergei Zimmerman f0e4af4365 libexpr: Fix weird formatting after treewide reformat 2025-08-25 22:09:18 +03:00
Bernardo Meurer Costa 5985d67906 feat(libstore/s3): add toHttpsUrl
This is extracted from the work in #13752
2025-08-25 18:48:19 +00:00
Eelco Dolstra 9bee0fa6ac Merge pull request #13822 from NixOS/bump-2.32.0
Bump version to 2.32
2025-08-25 17:22:57 +02:00
Eelco Dolstra adec28bf85 Update release-process.md 2025-08-25 10:30:21 +02:00
Eelco Dolstra f5e09d9b58 Update mergify.yml 2025-08-25 10:28:47 +02:00
Eelco Dolstra f67daa4a87 Bump version 2025-08-25 10:27:46 +02:00
Ethan EvansandSergei Zimmerman 7b8ceb5d2d libutil, libexpr: #10542 abstract over getrusage for getting cpuTime stat and implement windows version
Update src/libutil/windows/current-process.cc

Prefer `nullptr` over `NULL`

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>

Update src/libutil/unix/current-process.cc

Prefer C++ type casts

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>

Update src/libutil/windows/current-process.cc

Prefer C++ type casts

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>

Update src/libutil/unix/current-process.cc

Don't allocate exception

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-08-24 18:45:33 -07:00
John Ericson c9211b0b2d Merge pull request #13803 from obsidiansystems/more-parsed-urls
Make more URLs parsed, most notably `FileTransferRequest::url`
2025-08-23 10:54:39 -04:00
John Ericson 2fa2c0b09f Merge pull request #13812 from obsidiansystems/url-parse-leniency
Limit to lenient parsing of non-standard URLs only where needed
2025-08-23 10:53:59 -04:00
Jörg Thalheim ebf1cf5227 Merge pull request #13807 from roberth/release-notes-todo
maintainers: Add script for release notes todo list
2025-08-23 08:52:22 +02:00
John Ericson 3e86d75c9d Make more URLs parsed, most notably FileTransferRequest::url
Trying to gradually replace the use of strings with better types in ways
that makes sense.
2025-08-22 12:42:48 -04:00
John Ericson 72a548ed6a Limit to lenient parsing of non-standard URLs only where needed
This allows us to put `parseURL` in more spots without furthering
technical debt.
2025-08-22 12:37:37 -04:00
John Ericson 4083eff0c0 decodeQuery Take std::string_view not string ref 2025-08-22 12:26:48 -04:00
Eelco Dolstra 9c6019dd1a Merge pull request #13804 from NixOS/release-notes
Release notes for Nix 2.31
2025-08-22 17:18:15 +02:00
Eelco Dolstra 82d3662f09 Tweak 2025-08-22 11:03:05 +02:00
Sergei Zimmerman 01691e1180 Merge pull request #13810 from hgl/patch-1 2025-08-22 11:49:41 +03:00
Glen Huang c0246460a0 doc: Fix typo 2025-08-22 15:39:18 +08:00
John Ericson 0c46c2c37f Mention experimental SHA-256 git hashing in release notes 2025-08-21 23:17:12 -04:00
John Ericson 58278974b6 Merge pull request #13809 from xokdvium/enumerate
libutil: Replace hand-rolled enumerate with std::views::{zip,iota}
2025-08-21 22:58:19 -04:00
Sergei Zimmerman b853994e7a libutil: Replace hand-rolled enumerate with std::views::{zip,iota}
It would have been nice to use std::views::enumerate here, but
it uses a signed difference type for the value_type:

> value_type = std::tuple<difference_type, ranges::range_value_t<Base>>

zip + iota has the same semantics as the code used to have, so there's
no behavior change here.
2025-08-22 03:10:33 +03:00
Robert Hensing a1b3934a78 maintainers: Add script for release notes todo list 2025-08-21 14:19:22 +02:00
Eelco Dolstra ae09483498 Add more release notes 2025-08-21 13:37:38 +02:00
Eelco Dolstra 4dcfb36c1e Cleanup 2025-08-21 13:37:38 +02:00
Eelco Dolstra 2e7bb61a83 Document that you need to set GITHUB_TOKEN 2025-08-21 13:37:38 +02:00
Eelco Dolstra 0eaed891f4 Update release credits 2025-08-21 13:37:38 +02:00
Eelco Dolstra 6c391b7446 release notes: 2.31.0 2025-08-21 13:37:38 +02:00
Eelco Dolstra d7612f350f Merge pull request #13806 from NixOS/fix-meson-format
Fix meson formatting error
2025-08-21 13:31:50 +02:00
Eelco Dolstra 4fe700b7cb Fix meson-format 2025-08-21 13:17:06 +02:00
Eelco Dolstra 3826d51a65 Merge pull request #13565 from NixOS/prefetch-inputs
Add `nix flake prefetch-inputs` command
2025-08-21 12:30:17 +02:00
Eelco Dolstra 615b10cb44 Merge pull request #13800 from NixOS/concurrent-eval-cache
Use WAL mode for SQLite cache databases
2025-08-21 11:42:15 +02:00
John Ericson e2b984704a Merge pull request #13801 from obsidiansystems/move-sandbox-path-closure-code
Make sure `settings.sandboxedPaths` is closed outside `DerivationBuilder`
2025-08-20 19:15:16 -04:00
John Ericson 4c76db8e7c Make sure settings.sandboxedPaths is closed outside DerivationBuilder
This is a nicer separation of concerns --- `DerivationBuilder` just
mounts the extra paths you tell it too, and the outside world is
responsible for making sure those extra paths make sense.

Since the closure only depends on global settings, and not
per-derivation information, we also have the option of moving this up
further and caching it across all local builds. (I only just realized
this after having done this refactor. I am not doing that change at this
time, however.)
2025-08-20 18:49:11 -04:00
John Ericson 08e42e20fa Merge pull request #13769 from obsidiansystems/simplify-derivation-building-goal
Handle structured attrs, "export references graph" outside of `DerivationBuilder`
2025-08-20 18:05:04 -04:00
John Ericson 1d3ddb21fa Further consolidate environment variable processing outside DerivationBuilder
Now, `DerivationBuilder` only concerns itself with `finalEnv` and
`extraFiles`, in straightforward unconditional code. All the fancy
desugaring logic is consolidated in `DerivationBuildingGoal`.

We should better share the pulled-out logic with `nix-shell`/`nix
develop`, which would fill in some missing features, arguably fixing
bugs.
2025-08-20 16:54:17 -04:00
John Ericson e3c74f5a13 Desugar structured attrs, "export reference graph" outside DerivationBuilder
I think this is a better separation of concerns. `DerivationBuilder`
doesn't need to to the final, query-heavy details about how these things
are constructed. It just operates on the level of "simple, stupid" files
and environment variables.
2025-08-20 16:54:17 -04:00
John Ericson 92b10cf3f5 DerivationBuilderImpl::writeStructuredAttrs remove a rewrite
As much as I prefer rewriting the parsed rather than unparsed JSON for
elegance, this gets in the way of the separation of concerns that I am
trying to do.

As a practical matter, any rewriting that this did will also be done by
the second round of rewriting that remains below, so removing this code
should have no effect.
2025-08-20 16:54:17 -04:00
John Ericson 2767ae35d9 Deduplicate "export reference graph" logic a bit
The first part on `drvOptions.exportReferencesGraph` is the same in both
cases. It is just how the information is finally rendered that is
different.
2025-08-20 16:54:17 -04:00
John Ericson ca86d34077 Merge pull request #13799 from obsidiansystems/typed-sandbox-paths
Make `sandbox-settings` better typed, get `globals.hh` out of other headers
2025-08-20 16:50:34 -04:00
John Ericson a712445a7a Make Settings::sandboxPaths well-typed
Parsing logic is moved from `DerivationBuilder`, where is doesn't
belong, to `Settings` itself, where it does.
2025-08-20 16:24:37 -04:00
John Ericson 52212635db No more globals.hh in headers
This is needed to rearrange include order, but I also think it is a good
thing anyways, as we seek to reduce the use of global settings variables
over time.
2025-08-20 16:24:37 -04:00
John Ericson bce29ab2cf Move header outside Unix-only portion
We'll neeed some definitions elsewhere
2025-08-20 16:24:37 -04:00
John Ericson 8463fef161 Expose ChrootPath, PathsInChroot
Will want these for settings in a moment.
2025-08-20 16:24:37 -04:00
John Ericson d53c7b816b Push #include down to .cc file
That is where it should be.
2025-08-20 16:24:37 -04:00
John Ericson 4ab579b469 Remove constructor from ChrootPath
I rather use designated initializers.
2025-08-20 16:24:36 -04:00
Eelco Dolstra 0df147b145 Bump the version of the SQLite caches
This avoids problems with older versions of Nix that don't put the
caches in WAL mode. That's generally not a problem, until you do something like

  nix build --print-out-paths ... | cachix

which deadlocks because cachix tries to switch the caches to truncate
mode, which requires exclusive access. But the first process cannot
make progress because the cachix process isn't reading from the pipe.
2025-08-20 20:44:58 +02:00
Eelco Dolstra 4ab8ff5b4c SQLite: Use std::filesystem::path 2025-08-20 20:44:21 +02:00
Eelco Dolstra 349d2c58e5 Use WAL mode for SQLite cache databases
With "truncate" mode, if we try to write to the database while another
process has an active write transaction, we'll block until the other
transaction finishes. This is a problem for the evaluation cache in
particular, since it uses long-running transactions.

WAL mode does not have this issue: it just returns "busy" right away,
so Nix will print

  error (ignored): SQLite database '/home/eelco/.cache/nix/eval-cache-v5/...' is busy

and stop trying to write to the evaluation cache. (This was the
intended/original behaviour, see AttrDb::doSQLite().)
2025-08-20 20:34:43 +02:00
Eelco Dolstra 5fe6c53703 nix flake prefetch-inputs: Add
This command fetches all inputs of a flake in parallel.

Example runtime for

  $ chmod -R u+w /tmp/nix2; rm -rf /tmp/nix2; rm ~/.cache/nix/fetcher-cache-v3.sqlite*; rm -rf ~/.cache/nix/tarball-cache/ ~/.cache/nix/gitv3/; time nix flake prefetch-inputs --store /tmp/nix2 https://api.flakehub.com/f/pinned/informalsystems/cosmos.nix/0.3.0/018ce9ed-d0be-7ce5-81b6-a3c6e3ae1187/source.tar.gz

with http-connections = 1:

real    4m11.859s
user    2m6.931s
sys     0m25.619s

and http-connections = 25 (the default):

real    0m57.146s
user    2m49.506s
sys     0m36.008s
2025-07-28 18:53:49 +02:00
Grégory marti f0c7fbcdab Add /etc/ssl/certs/ca-certificates.crt in docker.nix 2025-07-22 17:39:29 +02:00
674 changed files with 17630 additions and 9686 deletions

No files matched your search

+4
View File
@@ -15,6 +15,10 @@ so you understand the process and the expectations.
- volunteering contributions effectively
- how to get help and our review process.
PR stuck in review? We have two Nix team meetings per week online that are open for everyone in a jitsi conference:
- https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com
-->
## Motivation
@@ -4,12 +4,22 @@ inputs:
dogfood:
description: "Whether to use Nix installed from the latest artifact from master branch"
required: true # Be explicit about the fact that we are using unreleased artifacts
experimental-installer:
description: "Whether to use the experimental installer to install Nix"
default: false
experimental-installer-version:
description: "Version of the experimental installer to use. If `latest`, the newest artifact from the default branch is used."
# TODO: This should probably be pinned to a release after https://github.com/NixOS/experimental-nix-installer/pull/49 lands in one
default: "latest"
extra_nix_config:
description: "Gets appended to `/etc/nix/nix.conf` if passed."
install_url:
description: "URL of the Nix installer"
required: false
default: "https://releases.nixos.org/nix/nix-2.30.2/install"
tarball_url:
description: "URL of the Nix tarball to use with the experimental installer"
required: false
github_token:
description: "Github token"
required: true
@@ -37,14 +47,74 @@ runs:
gh run download "$RUN_ID" --repo "$DOGFOOD_REPO" -n "$INSTALLER_ARTIFACT" -D "$INSTALLER_DOWNLOAD_DIR"
echo "installer-path=file://$INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$INSTALLER_DOWNLOAD_DIR" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
echo "::notice ::Dogfooding Nix installer from master (https://github.com/$DOGFOOD_REPO/actions/runs/$RUN_ID)"
env:
GH_TOKEN: ${{ inputs.github_token }}
DOGFOOD_REPO: "NixOS/nix"
- name: "Gather system info for experimental installer"
shell: bash
if: ${{ inputs.experimental-installer == 'true' }}
run: |
echo "::notice Using experimental installer from $EXPERIMENTAL_INSTALLER_REPO (https://github.com/$EXPERIMENTAL_INSTALLER_REPO)"
if [ "$RUNNER_OS" == "Linux" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="linux"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
elif [ "$RUNNER_OS" == "macOS" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="darwin"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_OS: $RUNNER_OS"
exit 1
fi
if [ "$RUNNER_ARCH" == "X64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=x86_64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
elif [ "$RUNNER_ARCH" == "ARM64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=aarch64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_ARCH: $RUNNER_ARCH"
exit 1
fi
echo "EXPERIMENTAL_INSTALLER_ARTIFACT=nix-installer-$EXPERIMENTAL_INSTALLER_ARCH-$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
env:
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- name: "Download latest experimental installer"
shell: bash
id: download-latest-experimental-installer
if: ${{ inputs.experimental-installer == 'true' && inputs.experimental-installer-version == 'latest' }}
run: |
RUN_ID=$(gh run list --repo "$EXPERIMENTAL_INSTALLER_REPO" --workflow ci.yml --branch main --status success --json databaseId --jq ".[0].databaseId")
EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR="$GITHUB_WORKSPACE/$EXPERIMENTAL_INSTALLER_ARTIFACT"
mkdir -p "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
gh run download "$RUN_ID" --repo "$EXPERIMENTAL_INSTALLER_REPO" -n "$EXPERIMENTAL_INSTALLER_ARTIFACT" -D "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
# Executable permissions are lost in artifacts
find $EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR -type f -exec chmod +x {} +
echo "installer-path=$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
env:
GH_TOKEN: ${{ inputs.github_token }}
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- uses: cachix/install-nix-action@c134e4c9e34bac6cab09cf239815f9339aaaf84e # v31.5.1
if: ${{ inputs.experimental-installer != 'true' }}
with:
# Ternary operator in GHA: https://www.github.com/actions/runner/issues/409#issuecomment-752775072
install_url: ${{ inputs.dogfood == 'true' && format('{0}/install', steps.download-nix-installer.outputs.installer-path) || inputs.install_url }}
install_options: ${{ inputs.dogfood == 'true' && format('--tarball-url-prefix {0}', steps.download-nix-installer.outputs.installer-path) || '' }}
extra_nix_config: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/nix-installer-action@786fff0690178f1234e4e1fe9b536e94f5433196 # v20
if: ${{ inputs.experimental-installer == 'true' }}
with:
diagnostic-endpoint: ""
# TODO: It'd be nice to use `artifacts.nixos.org` for both of these, maybe through an `/experimental-installer/latest` endpoint? or `/commit/<hash>`?
local-root: ${{ inputs.experimental-installer-version == 'latest' && steps.download-latest-experimental-installer.outputs.installer-path || '' }}
source-url: ${{ inputs.experimental-installer-version != 'latest' && 'https://artifacts.nixos.org/experimental-installer/tag/${{ inputs.experimental-installer-version }}/${{ env.EXPERIMENTAL_INSTALLER_ARTIFACT }}' || '' }}
nix-package-url: ${{ inputs.dogfood == 'true' && steps.download-nix-installer.outputs.tarball-path || (inputs.tarball_url || '') }}
extra-conf: ${{ inputs.extra_nix_config }}
+37
View File
@@ -0,0 +1,37 @@
name: Backport
on:
pull_request_target:
types: [closed, labeled]
permissions:
contents: read
jobs:
backport:
name: Backport Pull Request
permissions:
# for korthout/backport-action
contents: write
pull-requests: write
if: github.repository_owner == 'NixOS' && github.event.pull_request.merged == true && (github.event_name != 'labeled' || startsWith('backport', github.event.label.name))
runs-on: ubuntu-24.04-arm
steps:
- name: Generate GitHub App token
id: generate-token
uses: actions/create-github-app-token@v2
with:
app-id: ${{ vars.CI_APP_ID }}
private-key: ${{ secrets.CI_APP_PRIVATE_KEY }}
- uses: actions/checkout@v5
with:
ref: ${{ github.event.pull_request.head.sha }}
# required to find all branches
fetch-depth: 0
- name: Create backport PRs
uses: korthout/backport-action@d07416681cab29bf2661702f925f020aaa962997 # v3.4.1
id: backport
with:
# Config README: https://github.com/korthout/backport-action#backport-action
github_token: ${{ steps.generate-token.outputs.token }}
github_workspace: ${{ github.workspace }}
auto_merge_enabled: true
pull_description: |-
Automatic backport to `${target_branch}`, triggered by a label in #${pull_number}.
+57 -22
View File
@@ -4,6 +4,8 @@ on:
pull_request:
merge_group:
push:
branches:
- master
workflow_dispatch:
inputs:
dogfood:
@@ -29,7 +31,32 @@ jobs:
github_token: ${{ secrets.GITHUB_TOKEN }}
- run: nix flake show --all-systems --json
pre-commit-checks:
name: pre-commit checks
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
- uses: ./.github/actions/install-nix-action
with:
dogfood: ${{ github.event_name == 'workflow_dispatch' && inputs.dogfood || github.event_name != 'workflow_dispatch' }}
extra_nix_config: experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: ./ci/gha/tests/pre-commit-checks
basic-checks:
name: aggregate basic checks
if: ${{ always() }}
runs-on: ubuntu-24.04
needs: [pre-commit-checks, eval]
steps:
- name: Exit with any errors
if: ${{ contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') }}
run: |
exit 1
tests:
needs: basic-checks
strategy:
fail-fast: false
matrix:
@@ -75,6 +102,12 @@ jobs:
nix build --file ci/gha/tests/wrapper.nix componentTests -L \
--arg withInstrumentation ${{ matrix.instrumented }} \
--argstr stdenv "${{ matrix.stdenv }}"
- name: Run VM tests
run: |
nix build --file ci/gha/tests/wrapper.nix vmTests -L \
--arg withInstrumentation ${{ matrix.instrumented }} \
--argstr stdenv "${{ matrix.stdenv }}"
if: ${{ matrix.os == 'linux' }}
- name: Run flake checks and prepare the installer tarball
run: |
ci/gha/tests/build-checks
@@ -110,9 +143,19 @@ jobs:
- scenario: on ubuntu
runs-on: ubuntu-24.04
os: linux
experimental-installer: false
- scenario: on macos
runs-on: macos-14
os: darwin
experimental-installer: false
- scenario: on ubuntu (experimental)
runs-on: ubuntu-24.04
os: linux
experimental-installer: true
- scenario: on macos (experimental)
runs-on: macos-14
os: darwin
experimental-installer: true
name: installer test ${{ matrix.scenario }}
runs-on: ${{ matrix.runs-on }}
steps:
@@ -124,11 +167,22 @@ jobs:
path: out
- name: Looking up the installer tarball URL
id: installer-tarball-url
run: echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
run: |
echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$GITHUB_WORKSPACE/out" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
- uses: cachix/install-nix-action@v31
if: ${{ !matrix.experimental-installer }}
with:
install_url: ${{ format('{0}/install', steps.installer-tarball-url.outputs.installer-url) }}
install_options: ${{ format('--tarball-url-prefix {0}', steps.installer-tarball-url.outputs.installer-url) }}
- uses: ./.github/actions/install-nix-action
if: ${{ matrix.experimental-installer }}
with:
dogfood: false
experimental-installer: true
tarball_url: ${{ steps.installer-tarball-url.outputs.tarball-path }}
github_token: ${{ secrets.GITHUB_TOKEN }}
- run: sudo apt install fish zsh
if: matrix.os == 'linux'
- run: brew install fish
@@ -160,7 +214,7 @@ jobs:
echo "docker=${{ env._DOCKER_SECRETS != '' }}" >> $GITHUB_OUTPUT
docker_push_image:
needs: [tests, vm_tests, check_secrets]
needs: [tests, check_secrets]
permissions:
contents: read
packages: write
@@ -213,27 +267,8 @@ jobs:
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
vm_tests:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
- uses: ./.github/actions/install-nix-action
with:
dogfood: ${{ github.event_name == 'workflow_dispatch' && inputs.dogfood || github.event_name != 'workflow_dispatch' }}
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: |
nix build -L \
.#hydraJobs.tests.functional_user \
.#hydraJobs.tests.githubFlakes \
.#hydraJobs.tests.nix-docker \
.#hydraJobs.tests.tarballFlakes \
;
flake_regressions:
needs: vm_tests
needs: tests
runs-on: ubuntu-24.04
steps:
- name: Checkout nix
+1 -1
View File
@@ -18,7 +18,7 @@ jobs:
runs-on: ubuntu-24.04
if: github.repository_owner == 'NixOS'
steps:
- uses: actions/labeler@v5
- uses: actions/labeler@v6
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
sync-labels: false
-163
View File
@@ -1,163 +0,0 @@
queue_rules:
- name: default
# all required tests need to go here
merge_conditions:
- check-success=tests on macos
- check-success=tests on ubuntu
- check-success=installer test on macos
- check-success=installer test on ubuntu
- check-success=vm_tests
batch_size: 5
pull_request_rules:
- name: merge using the merge queue
conditions:
- base~=master|.+-maintenance
- label~=merge-queue|dependencies
actions:
queue: {}
# The rules below will first create backport pull requests and put those in a merge queue.
- name: backport patches to 2.18
conditions:
- label=backport 2.18-maintenance
actions:
backport:
branches:
- 2.18-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.19
conditions:
- label=backport 2.19-maintenance
actions:
backport:
branches:
- 2.19-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.20
conditions:
- label=backport 2.20-maintenance
actions:
backport:
branches:
- 2.20-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.21
conditions:
- label=backport 2.21-maintenance
actions:
backport:
branches:
- 2.21-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.22
conditions:
- label=backport 2.22-maintenance
actions:
backport:
branches:
- 2.22-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.23
conditions:
- label=backport 2.23-maintenance
actions:
backport:
branches:
- 2.23-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.24
conditions:
- label=backport 2.24-maintenance
actions:
backport:
branches:
- "2.24-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.25
conditions:
- label=backport 2.25-maintenance
actions:
backport:
branches:
- "2.25-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.26
conditions:
- label=backport 2.26-maintenance
actions:
backport:
branches:
- "2.26-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.27
conditions:
- label=backport 2.27-maintenance
actions:
backport:
branches:
- "2.27-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.28
conditions:
- label=backport 2.28-maintenance
actions:
backport:
branches:
- "2.28-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.29
conditions:
- label=backport 2.29-maintenance
actions:
backport:
branches:
- "2.29-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.30
conditions:
- label=backport 2.30-maintenance
actions:
backport:
branches:
- "2.30-maintenance"
labels:
- automatic backport
- merge-queue
+1 -1
View File
@@ -1 +1 @@
2.31.0
2.33.0
+11 -14
View File
@@ -1,8 +1,8 @@
GNU LESSER GENERAL PUBLIC LICENSE
Version 2.1, February 1999
GNU LESSER GENERAL PUBLIC LICENSE
Version 2.1, February 1999
Copyright (C) 1991, 1999 Free Software Foundation, Inc.
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
<https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
@@ -10,7 +10,7 @@
as the successor of the GNU Library Public License, version 2, hence
the version number 2.1.]
Preamble
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
@@ -112,7 +112,7 @@ modification follow. Pay close attention to the difference between a
former contains code derived from the library, whereas the latter must
be combined with the library in order to run.
GNU LESSER GENERAL PUBLIC LICENSE
GNU LESSER GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License Agreement applies to any software library or other
@@ -146,7 +146,7 @@ such a program is covered only if its contents constitute a work based
on the Library (independent of the use of the Library in a tool for
writing it). Whether that is true depends on what the Library does
and what the program that uses the Library does.
1. You may copy and distribute verbatim copies of the Library's
complete source code as you receive it, in any medium, provided that
you conspicuously and appropriately publish on each copy an
@@ -432,7 +432,7 @@ decision will be guided by the two goals of preserving the free status
of all derivatives of our free software and of promoting the sharing
and reuse of software generally.
NO WARRANTY
NO WARRANTY
15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
@@ -455,7 +455,7 @@ FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
DAMAGES.
END OF TERMS AND CONDITIONS
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Libraries
@@ -484,8 +484,7 @@ convey the exclusion of warranty; and each file should have at least the
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
License along with this library; if not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
@@ -496,9 +495,7 @@ necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the
library `Frob' (a library for tweaking knobs) written by James Random Hacker.
<signature of Ty Coon>, 1 April 1990
Ty Coon, President of Vice
<signature of Moe Ghoul>, 1 April 1990
Moe Ghoul, President of Vice
That's all there is to it!
+33 -22
View File
@@ -21,25 +21,6 @@ let
packages' = nixFlake.packages.${system};
stdenv = (getStdenv pkgs);
enableSanitizersLayer = finalAttrs: prevAttrs: {
mesonFlags =
(prevAttrs.mesonFlags or [ ])
++ [
# Run all tests with UBSAN enabled. Running both with ubsan and
# without doesn't seem to have much immediate benefit for doubling
# the GHA CI workaround.
#
# TODO: Work toward enabling "address,undefined" if it seems feasible.
# This would maybe require dropping Boost coroutines and ignoring intentional
# memory leaks with detect_leaks=0.
(lib.mesonOption "b_sanitize" "undefined")
]
++ (lib.optionals stdenv.cc.isClang [
# https://www.github.com/mesonbuild/meson/issues/764
(lib.mesonBool "b_lundef" false)
]);
};
collectCoverageLayer = finalAttrs: prevAttrs: {
env =
let
@@ -62,20 +43,33 @@ let
'';
};
componentOverrides =
(lib.optional withSanitizers enableSanitizersLayer)
++ (lib.optional withCoverage collectCoverageLayer);
componentOverrides = (lib.optional withCoverage collectCoverageLayer);
in
rec {
nixComponentsInstrumented = nixComponents.overrideScope (
final: prev: {
withASan = withSanitizers;
withUBSan = withSanitizers;
nix-store-tests = prev.nix-store-tests.override { withBenchmarks = true; };
# Boehm is incompatible with ASAN.
nix-expr = prev.nix-expr.override { enableGC = !withSanitizers; };
mesonComponentOverrides = lib.composeManyExtensions componentOverrides;
# Unclear how to make Perl bindings work with a dynamically linked ASAN.
nix-perl-bindings = if withSanitizers then null else prev.nix-perl-bindings;
}
);
# Import NixOS tests using the instrumented components
nixosTests = import ../../../tests/nixos {
inherit lib pkgs;
nixComponents = nixComponentsInstrumented;
nixpkgs = nixFlake.inputs.nixpkgs;
inherit (nixFlake.inputs) nixpkgs-23-11;
};
/**
Top-level tests for the flake outputs, as they would be built by hydra.
These tests generally can't be overridden to run with sanitizers.
@@ -122,6 +116,7 @@ rec {
) nixComponentsInstrumented)
// lib.optionalAttrs (pkgs.stdenv.hostPlatform == pkgs.stdenv.buildPlatform) {
"${componentTestsPrefix}nix-functional-tests" = nixComponentsInstrumented.nix-functional-tests;
"${componentTestsPrefix}nix-json-schema-checks" = nixComponentsInstrumented.nix-json-schema-checks;
};
codeCoverage =
@@ -226,4 +221,20 @@ rec {
{
inherit coverageProfileDrvs mergedProfdata coverageReports;
};
vmTests = {
inherit (nixosTests) s3-binary-cache-store;
}
// lib.optionalAttrs (!withSanitizers && !withCoverage) {
# evalNixpkgs uses non-instrumented components from hydraJobs, so only run it
# when not testing with sanitizers to avoid rebuilding nix
inherit (hydraJobs.tests) evalNixpkgs;
# FIXME: CI times out when building vm tests instrumented
inherit (nixosTests)
functional_user
githubFlakes
nix-docker
tarballFlakes
;
};
}
+24
View File
@@ -0,0 +1,24 @@
#!/usr/bin/env bash
set -euo pipefail
system=$(nix eval --raw --impure --expr builtins.currentSystem)
echo "::group::Running pre-commit checks"
if nix build ".#checks.$system.pre-commit" -L; then
echo "::endgroup::"
exit 0
fi
echo "::error ::Changes do not pass pre-commit checks"
cat <<EOF
The code isn't formatted or doesn't pass lints. You can run pre-commit locally with:
nix develop -c ./maintainers/format.sh
EOF
echo "::endgroup::"
exit 1
+2 -1
View File
@@ -88,7 +88,7 @@ manual = custom_target(
@0@ @INPUT0@ @CURRENT_SOURCE_DIR@ > @DEPFILE@
@0@ @INPUT1@ summary @2@ < @CURRENT_SOURCE_DIR@/source/SUMMARY.md.in > @2@/source/SUMMARY.md
sed -e 's|@version@|@3@|g' < @INPUT2@ > @2@/book.toml
@4@ -r --include='*.md' @CURRENT_SOURCE_DIR@/ @2@/
@4@ -r -L --include='*.md' @CURRENT_SOURCE_DIR@/ @2@/
(cd @2@; RUST_LOG=warn @1@ build -d @2@ 3>&2 2>&1 1>&3) | { grep -Fv "because fragment resolution isn't implemented" || :; } 3>&2 2>&1 1>&3
rm -rf @2@/manual
mv @2@/html @2@/manual
@@ -115,6 +115,7 @@ manual = custom_target(
builtins_md,
rl_next_generated,
summary_rl_next,
json_schema_generated_files,
nix_input,
],
output : [
+5
View File
@@ -12,6 +12,7 @@
rsync,
nix-cli,
changelog-d,
json-schema-for-humans,
officialRelease,
# Configuration Options
@@ -32,6 +33,9 @@ mkMesonDerivation (finalAttrs: {
fileset.difference
(fileset.unions [
../../.version
# For example JSON
../../src/libutil-tests/data/memory-source-accessor
../../src/libutil-tests/data/hash
# Too many different types of files to filter for now
../../doc/manual
./.
@@ -55,6 +59,7 @@ mkMesonDerivation (finalAttrs: {
jq
python3
rsync
json-schema-for-humans
changelog-d
]
++ lib.optionals (!officialRelease) [
@@ -1,6 +0,0 @@
---
synopsis: "`build-cores = 0` now auto-detects CPU cores"
prs: [13402]
---
When `build-cores` is set to `0`, nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
-11
View File
@@ -1,11 +0,0 @@
---
synopsis: "Fix Git LFS SSH issues"
prs: [13743]
issues: [13337]
---
Fixed some outstanding issues with Git LFS and SSH.
* Added support for `NIX_SSHOPTS`.
* Properly use the parsed port from URL.
* Better use of the response of `git-lfs-authenticate` to determine API endpoint when the API is not exposed on port 443.
-13
View File
@@ -1,13 +0,0 @@
---
synopsis: "Add support for user@address:port syntax in store URIs"
prs: [3425]
issues: [7044]
---
It's now possible to specify the port used for the SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable. This has now been fixed.
This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
- `ssh://127.0.0.1:2222`
- `ssh://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
- `ssh-ng://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
@@ -1,6 +0,0 @@
---
synopsis: "Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874"
prs: [13445]
---
Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.
@@ -0,0 +1,26 @@
---
synopsis: "Improved S3 binary cache support via HTTP"
prs: [13823, 14026, 14120, 14131, 14135, 14144, 14170, 14190, 14198, 14206, 14209, 14222, 14223, 13752]
issues: [13084, 12671, 11748, 12403]
---
S3 binary cache operations now happen via HTTP, leveraging `libcurl`'s native
AWS SigV4 authentication instead of the AWS C++ SDK, providing significant
improvements:
- **Reduced memory usage**: Eliminates memory buffering issues that caused
segfaults with large files
- **Fixed upload reliability**: Resolves AWS SDK chunking errors
(`InvalidChunkSizeError`)
- **Lighter dependencies**: Uses lightweight `aws-crt-cpp` instead of full
`aws-cpp-sdk`, reducing build complexity
The new implementation requires curl >= 7.75.0 and `aws-crt-cpp` for credential
management.
All existing S3 URL formats and parameters remain supported, with the notable
exception of multi-part uploads, which are no longer supported.
Note that this change also means Nix now supports S3 binary cache stores even
if build without `aws-crt-cpp`, but only for public buckets which do not
require auth.
+4
View File
@@ -117,6 +117,8 @@
- [Architecture and Design](architecture/architecture.md)
- [Formats and Protocols](protocols/index.md)
- [JSON Formats](protocols/json/index.md)
- [File System Object](protocols/json/file-system-object.md)
- [Hash](protocols/json/hash.md)
- [Store Object Info](protocols/json/store-object-info.md)
- [Derivation](protocols/json/derivation.md)
- [Serving Tarball Flakes](protocols/tarball-fetcher.md)
@@ -138,6 +140,8 @@
- [Contributing](development/contributing.md)
- [Releases](release-notes/index.md)
{{#include ./SUMMARY-rl-next.md}}
- [Release 2.32 (2025-10-06)](release-notes/rl-2.32.md)
- [Release 2.31 (2025-08-21)](release-notes/rl-2.31.md)
- [Release 2.30 (2025-07-07)](release-notes/rl-2.30.md)
- [Release 2.29 (2025-05-14)](release-notes/rl-2.29.md)
- [Release 2.28 (2025-04-02)](release-notes/rl-2.28.md)
@@ -2,6 +2,7 @@ xp_features_json = custom_target(
command : [ nix, '__dump-xp-features' ],
capture : true,
output : 'xp-features.json',
env : nix_env_for_docs,
)
experimental_features_shortlist_md = custom_target(
+2 -2
View File
@@ -23,7 +23,7 @@ $ nix-shell
To get a shell with one of the other [supported compilation environments](#compilation-environments):
```console
$ nix-shell --attr devShells.x86_64-linux.native-clangStdenvPackages
$ nix-shell --attr devShells.x86_64-linux.native-clangStdenv
```
> **Note**
@@ -34,7 +34,7 @@ $ nix-shell --attr devShells.x86_64-linux.native-clangStdenvPackages
To build Nix itself in this shell:
```console
[nix-shell]$ mesonFlags+=" --prefix=$(pwd)/outputs/out"
[nix-shell]$ out="$(pwd)/outputs/out" dev=$out debug=$out mesonFlags+=" --prefix=${out}"
[nix-shell]$ dontAddPrefix=1 configurePhase
[nix-shell]$ buildPhase
```
@@ -24,6 +24,19 @@ It is also possible to build without debugging for faster build:
(The first line is needed because `fortify` hardening requires at least some optimization.)
## Building Nix with sanitizers
Nix can be built with [Address](https://clang.llvm.org/docs/AddressSanitizer.html) and
[UB](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) sanitizers using LLVM
or GCC. This is useful when debugging memory corruption issues.
```console
[nix-shell]$ export mesonBuildType=debugoptimized
[nix-shell]$ appendToVar mesonFlags "-Dlibexpr:gc=disabled" # Disable Boehm
[nix-shell]$ appendToVar mesonFlags "-Dbindings=false" # Disable nix-perl
[nix-shell]$ appendToVar mesonFlags "-Db_sanitize=address,undefined"
```
## Debugging the Nix Binary
Obtain your preferred debugger within the development shell:
+17 -6
View File
@@ -25,20 +25,31 @@ nix build .#nix-manual
and open `./result/share/doc/nix/manual/index.html`.
To build the manual incrementally, [enter the development shell](./building.md) and run:
To build the manual incrementally, [enter the development shell](./building.md) and configure with `doc-gen` enabled:
**If using interactive `nix develop`:**
```console
make manual-html-open -j $NIX_BUILD_CORES
$ nix develop
$ mesonFlags="$mesonFlags -Ddoc-gen=true" mesonConfigurePhase
```
In order to reflect changes to the [Makefile for the manual], clear all generated files before re-building:
[Makefile for the manual]: https://github.com/NixOS/nix/blob/master/doc/manual/local.mk
**If using direnv:**
```console
rm $(git ls-files doc/manual/ -o | grep -F '.md') && rmdir doc/manual/source/command-ref/new-cli && make manual-html -j $NIX_BUILD_CORES
$ direnv allow
$ bash -c 'source $stdenv/setup && mesonFlags="$mesonFlags -Ddoc-gen=true" mesonConfigurePhase'
```
Then build the manual:
```console
$ cd build
$ meson compile manual
```
The HTML manual will be generated at `build/src/nix-manual/manual/index.html`.
## Style guide
The goal of this style guide is to make it such that
@@ -7,5 +7,6 @@ experimental_feature_descriptions_md = custom_target(
xp_features_json,
],
capture : true,
env : nix_env_for_docs,
output : 'experimental-feature-descriptions.md',
)
+19 -3
View File
@@ -5,12 +5,28 @@ All built-ins are available through the global [`builtins`](#builtins-builtins)
Some built-ins are also exposed directly in the global scope:
<!-- TODO(@rhendric, #10970): this list is incomplete -->
- [`derivation`](#builtins-derivation)
- [`import`](#builtins-import)
- `derivationStrict`
- [`abort`](#builtins-abort)
- [`baseNameOf`](#builtins-baseNameOf)
- [`break`](#builtins-break)
- [`dirOf`](#builtins-dirOf)
- [`false`](#builtins-false)
- [`fetchGit`](#builtins-fetchGit)
- `fetchMercurial`
- [`fetchTarball`](#builtins-fetchTarball)
- [`fetchTree`](#builtins-fetchTree)
- [`fromTOML`](#builtins-fromTOML)
- [`import`](#builtins-import)
- [`isNull`](#builtins-isNull)
- [`map`](#builtins-map)
- [`null`](#builtins-null)
- [`placeholder`](#builtins-placeholder)
- [`removeAttrs`](#builtins-removeAttrs)
- `scopedImport`
- [`throw`](#builtins-throw)
- [`toString`](#builtins-toString)
- [`true`](#builtins-true)
<dl>
<dt id="builtins-derivation"><a href="#builtins-derivation"><code>derivation <var>attrs</var></code></a></dt>
+3
View File
@@ -1,3 +1,6 @@
# Process JSON schema documentation
subdir('protocols')
summary_rl_next = custom_target(
command : [
bash,
+5 -95
View File
@@ -1,97 +1,7 @@
# Derivation JSON Format
{{#include derivation-v3-fixed.md}}
> **Warning**
>
> This JSON format is currently
> [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
> and subject to change.
<!--
## Raw Schema
The JSON serialization of a
[derivations](@docroot@/glossary.md#gloss-store-derivation)
is a JSON object with the following fields:
* `name`:
The name of the derivation.
This is used when calculating the store paths of the derivation's outputs.
* `outputs`:
Information about the output paths of the derivation.
This is a JSON object with one member per output, where the key is the output name and the value is a JSON object with these fields:
* `path`:
The output path, if it is known in advanced.
Otherwise, `null`.
* `method`:
For an output which will be [content addressed], a string representing the [method](@docroot@/store/store-object/content-address.md) of content addressing that is chosen.
Valid method strings are:
- [`flat`](@docroot@/store/store-object/content-address.md#method-flat)
- [`nar`](@docroot@/store/store-object/content-address.md#method-nix-archive)
- [`text`](@docroot@/store/store-object/content-address.md#method-text)
- [`git`](@docroot@/store/store-object/content-address.md#method-git)
Otherwise, `null`.
* `hashAlgo`:
For an output which will be [content addressed], the name of the hash algorithm used.
Valid algorithm strings are:
- `blake3`
- `md5`
- `sha1`
- `sha256`
- `sha512`
* `hash`:
For fixed-output derivations, the expected content hash in base-16.
> **Example**
>
> ```json
> "outputs": {
> "out": {
> "path": "/nix/store/2543j7c6jn75blc3drf4g5vhb1rhdq29-source",
> "method": "nar",
> "hashAlgo": "sha256",
> "hash": "6fc80dcc62179dbc12fc0b5881275898f93444833d21b89dfe5f7fbcbb1d0d62"
> }
> }
> ```
* `inputSrcs`:
A list of store paths on which this derivation depends.
* `inputDrvs`:
A JSON object specifying the derivations on which this derivation depends, and what outputs of those derivations.
> **Example**
>
> ```json
> "inputDrvs": {
> "/nix/store/6lkh5yi7nlb7l6dr8fljlli5zfd9hq58-curl-7.73.0.drv": ["dev"],
> "/nix/store/fn3kgnfzl5dzym26j8g907gq3kbm8bfh-unzip-6.0.drv": ["out"]
> }
> ```
specifies that this derivation depends on the `dev` output of `curl`, and the `out` output of `unzip`.
* `system`:
The system type on which this derivation is to be built
(e.g. `x86_64-linux`).
* `builder`:
The absolute path of the program to be executed to run the build.
Typically this is the `bash` shell
(e.g. `/nix/store/r3j288vpmczbl500w6zz89gyfa4nr0b1-bash-4.4-p23/bin/bash`).
* `args`:
The command-line arguments passed to the `builder`.
* `env`:
The environment passed to the `builder`.
* `structuredAttrs`:
[Strucutured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
Structured attributes are JSON, and thus embedded as-is.
[JSON Schema for Derivation v3](schema/derivation-v3.json)
-->
@@ -0,0 +1,21 @@
{{#include file-system-object-v1-fixed.md}}
## Examples
### Simple
```json
{{#include schema/file-system-object-v1/simple.json}}
```
### Complex
```json
{{#include schema/file-system-object-v1/complex.json}}
```
<!--
## Raw Schema
[JSON Schema for File System Object v1](schema/file-system-object-v1.json)
-->
@@ -0,0 +1,15 @@
# For some reason, backticks in the JSON schema are being escaped rather
# than being kept as intentional code spans. This removes all backtick
# escaping, which is an ugly solution, but one that is fine, because we
# are not using backticks for any other purpose.
s/\\`/`/g
# The way that semi-external references are rendered (i.e. ones to
# sibling schema files, as opposed to separate website ones, is not nice
# for humans. Replace it with a nice relative link within the manual
# instead.
#
# As we have more such relative links, more replacements of this nature
# should appear below.
s^\(./hash-v1.yaml\)\?#/$defs/algorithm^[JSON format for `Hash`](./hash.html#algorithm)^g
s^#/\$defs/\(regular\|symlink\|directory\)^In this schema^g
+33
View File
@@ -0,0 +1,33 @@
{{#include hash-v1-fixed.md}}
## Examples
### SHA-256 with Base64 encoding
```json
{{#include schema/hash-v1/sha256-base64.json}}
```
### SHA-256 with Base16 (hexadecimal) encoding
```json
{{#include schema/hash-v1/sha256-base16.json}}
```
### SHA-256 with Nix32 encoding
```json
{{#include schema/hash-v1/sha256-nix32.json}}
```
### BLAKE3 with Base64 encoding
```json
{{#include schema/hash-v1/blake3-base64.json}}
```
<!--
## Raw Schema
[JSON Schema for Hash v1](schema/hash-v1.json)
-->
@@ -0,0 +1,17 @@
# Configuration file for json-schema-for-humans
#
# https://github.com/coveooss/json-schema-for-humans/blob/main/docs/examples/examples_md_default/Configuration.md
template_name: md
show_toc: true
# impure timestamp and distracting
with_footer: false
recursive_detection_depth: 3
show_breadcrumbs: false
description_is_markdown: true
template_md_options:
properties_table_columns:
- Property
- Type
- Pattern
- Title/Description
@@ -0,0 +1,75 @@
# Tests in: ../../../../src/json-schema-checks
fs = import('fs')
# Find json-schema-for-humans if available
json_schema_for_humans = find_program('generate-schema-doc', required : false)
# Configuration for json-schema-for-humans
json_schema_config = files('json-schema-for-humans-config.yaml')
schemas = [
'file-system-object-v1',
'hash-v1',
'derivation-v3',
]
schema_files = files()
foreach schema_name : schemas
schema_files += files('schema' / schema_name + '.yaml')
endforeach
schema_outputs = []
foreach schema_name : schemas
schema_outputs += schema_name + '.md'
endforeach
json_schema_generated_files = []
# Generate markdown documentation from JSON schema
# Note: output must be just a filename, not a path
gen_file = custom_target(
schema_name + '-schema-docs.tmp',
command : [
json_schema_for_humans,
'--config-file',
json_schema_config,
meson.current_source_dir() / 'schema',
meson.current_build_dir(),
],
input : schema_files + [
json_schema_config,
],
output : schema_outputs,
capture : false,
build_by_default : true,
)
idx = 0
if json_schema_for_humans.found()
foreach schema_name : schemas
#schema_file = 'schema' / schema_name + '.yaml'
# There is one so-so hack, and one horrible hack being done here.
sedded_file = custom_target(
schema_name + '-schema-docs',
command : [
'sed',
'-f',
# Out of line to avoid https://github.com/mesonbuild/meson/issues/1564
files('fixup-json-schema-generated-doc.sed'),
'@INPUT@',
],
capture : true,
input : gen_file[idx],
output : schema_name + '-fixed.md',
)
idx += 1
json_schema_generated_files += [ sedded_file ]
endforeach
else
warning(
'json-schema-for-humans not found, skipping JSON schema documentation generation',
)
endif
@@ -0,0 +1,178 @@
"$schema": http://json-schema.org/draft-04/schema#
"$id": https://nix.dev/manual/nix/latest/protocols/json/schema/derivation-v3.json
title: Derivation
description: |
Experimental JSON representation of a Nix derivation (version 3).
This schema describes the JSON representation of Nix's `Derivation` type.
> **Warning**
>
> This JSON format is currently
> [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
> and subject to change.
type: object
required:
- name
- version
- outputs
- inputSrcs
- inputDrvs
- system
- builder
- args
- env
properties:
name:
type: string
title: Derivation name
description: |
The name of the derivation.
Used when calculating store paths for the derivation’s outputs.
version:
const: 3
title: Format version (must be 3)
description: |
Must be `3`.
This is a guard that allows us to continue evolving this format.
The choice of `3` is fairly arbitrary, but corresponds to this informal version:
- Version 0: A-Term format
- Version 1: Original JSON format, with ugly `"r:sha256"` inherited from A-Term format.
- Version 2: Separate `method` and `hashAlgo` fields in output specs
- Version 3: Drop store dir from store paths, just include base name.
Note that while this format is experimental, the maintenance of versions is best-effort, and not promised to identify every change.
outputs:
type: object
title: Output specifications
description: |
Information about the output paths of the derivation.
This is a JSON object with one member per output, where the key is the output name and the value is a JSON object as described.
> **Example**
>
> ```json
> "outputs": {
> "out": {
> "method": "nar",
> "hashAlgo": "sha256",
> "hash": "6fc80dcc62179dbc12fc0b5881275898f93444833d21b89dfe5f7fbcbb1d0d62"
> }
> }
> ```
additionalProperties:
"$ref": "#/$defs/output"
inputSrcs:
type: array
title: Input source paths
description: |
List of store paths on which this derivation depends.
> **Example**
>
> ```json
> "inputSrcs": [
> "47y241wqdhac3jm5l7nv0x4975mb1975-separate-debug-info.sh",
> "56d0w71pjj9bdr363ym3wj1zkwyqq97j-fix-pop-var-context-error.patch"
> ]
> ```
items:
type: string
inputDrvs:
type: object
title: Input derivations
description: |
Mapping of derivation paths to lists of output names they provide.
> **Example**
>
> ```json
> "inputDrvs": {
> "6lkh5yi7nlb7l6dr8fljlli5zfd9hq58-curl-7.73.0.drv": ["dev"],
> "fn3kgnfzl5dzym26j8g907gq3kbm8bfh-unzip-6.0.drv": ["out"]
> }
> ```
>
> specifies that this derivation depends on the `dev` output of `curl`, and the `out` output of `unzip`.
system:
type: string
title: Build system type
description: |
The system type on which this derivation is to be built
(e.g. `x86_64-linux`).
builder:
type: string
title: Build program path
description: |
Absolute path of the program used to perform the build.
Typically this is the `bash` shell
(e.g. `/nix/store/r3j288vpmczbl500w6zz89gyfa4nr0b1-bash-4.4-p23/bin/bash`).
args:
type: array
title: Builder arguments
description: |
Command-line arguments passed to the `builder`.
items:
type: string
env:
type: object
title: Environment variables
description: |
Environment variables passed to the `builder`.
additionalProperties:
type: string
structuredAttrs:
title: Structured attributes
description: |
[Structured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
Structured attributes are JSON, and thus embedded as-is.
type: object
additionalProperties: true
"$defs":
output:
type: object
properties:
path:
type: string
title: Output path
description: |
The output path, if known in advance.
method:
type: string
title: Content addressing method
enum: [flat, nar, text, git]
description: |
For an output which will be [content addressed](@docroot@/store/derivation/outputs/content-address.md), a string representing the [method](@docroot@/store/store-object/content-address.md) of content addressing that is chosen.
Valid method strings are:
- [`flat`](@docroot@/store/store-object/content-address.md#method-flat)
- [`nar`](@docroot@/store/store-object/content-address.md#method-nix-archive)
- [`text`](@docroot@/store/store-object/content-address.md#method-text)
- [`git`](@docroot@/store/store-object/content-address.md#method-git)
hashAlgo:
title: Hash algorithm
"$ref": "./hash-v1.yaml#/$defs/algorithm"
hash:
type: string
title: Expected hash value
description: |
For fixed-output derivations, the expected content hash in base-16.
@@ -0,0 +1 @@
../../../../../../src/libutil-tests/data/memory-source-accessor
@@ -0,0 +1,65 @@
"$schema": http://json-schema.org/draft-04/schema#
"$id": https://nix.dev/manual/nix/latest/protocols/json/schema/file-system-object-v1.json
title: File System Object
description: |
This schema describes the JSON representation of Nix's [File System Object](@docroot@/store/file-system-object.md).
The schema is recursive because file system objects contain other file system objects.
type: object
required: ["type"]
properties:
type:
type: string
enum: ["regular", "symlink", "directory"]
# Enforce conditional structure based on `type`
anyOf:
- $ref: "#/$defs/regular"
required: ["type", "contents"]
- $ref: "#/$defs/symlink"
required: ["type", "target"]
- $ref: "#/$defs/directory"
required: ["type", "contents"]
"$defs":
regular:
title: Regular File
required: ["contents"]
properties:
type:
const: "regular"
contents:
type: string
description: Base64-encoded file contents
executable:
type: boolean
description: Whether the file is executable.
default: false
additionalProperties: false
symlink:
title: Symbolic Link
required: ["target"]
properties:
type:
const: "symlink"
target:
type: string
description: Target path of the symlink.
additionalProperties: false
directory:
title: Directory
required: ["contents"]
properties:
type:
const: "directory"
contents:
type: object
description: |
Map of names to nested file system objects (for type=directory)
additionalProperties:
$ref: "#"
additionalProperties: false
+1
View File
@@ -0,0 +1 @@
../../../../../../src/libutil-tests/data/hash/
@@ -0,0 +1,54 @@
"$schema": http://json-schema.org/draft-04/schema#
"$id": https://nix.dev/manual/nix/latest/protocols/json/schema/hash-v1.json
title: Hash
description: |
A cryptographic hash value used throughout Nix for content addressing and integrity verification.
This schema describes the JSON representation of Nix's `Hash` type.
type: object
properties:
algorithm:
"$ref": "#/$defs/algorithm"
format:
type: string
enum:
- base64
- nix32
- base16
- sri
title: Hash format
description: |
The encoding format of the hash value.
- `base64` uses standard Base64 encoding [RFC 4648, section 4](https://datatracker.ietf.org/doc/html/rfc4648#section-4)
- `nix32` is Nix-specific base-32 encoding
- `base16` is lowercase hexadecimal
- `sri` is the [Subresource Integrity format](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity).
hash:
type: string
title: Hash
description: |
The encoded hash value, itself.
It is specified in the format specified by the `format` field.
It must be the right length for the hash algorithm specified in the `algorithm` field, also.
The hash value does not include any algorithm prefix.
required:
- algorithm
- format
- hash
additionalProperties: false
"$defs":
algorithm:
type: string
enum:
- blake3
- md5
- sha1
- sha256
- sha512
title: Hash algorithm
description: |
The hash algorithm used to compute the hash value.
`blake3` is currently experimental and requires the [`blake-hashing`](@docroot@/development/experimental-features.md#xp-feature-blake-hashing) experimental feature.
+2
View File
@@ -0,0 +1,2 @@
# Process JSON schema documentation
subdir('json')
@@ -0,0 +1,96 @@
# Release 2.31.0 (2025-08-21)
- `build-cores = 0` now auto-detects CPU cores [#13402](https://github.com/NixOS/nix/pull/13402)
When `build-cores` is set to `0`, Nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
- Fix Git LFS SSH issues [#13337](https://github.com/NixOS/nix/issues/13337) [#13743](https://github.com/NixOS/nix/pull/13743)
Fixed some outstanding issues with Git LFS and SSH.
* Added support for `NIX_SSHOPTS`.
* Properly use the parsed port from URL.
* Better use of the response of `git-lfs-authenticate` to determine API endpoint when the API is not exposed on port 443.
- Add support for `user@address:port` syntax in store URIs [#7044](https://github.com/NixOS/nix/issues/7044) [#3425](https://github.com/NixOS/nix/pull/3425)
It's now possible to specify the port used for SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or the `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable.
This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
- `ssh://127.0.0.1:2222`
- `ssh://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
- `ssh-ng://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
- Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874 [#13445](https://github.com/NixOS/nix/pull/13445)
Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.
- Use WAL mode for SQLite cache databases [#13800](https://github.com/NixOS/nix/pull/13800)
Previously, Nix used SQLite's "truncate" mode for caches. However, this could cause a Nix process to block if another process was updating the cache. This was a problem for the flake evaluation cache in particular, since it uses long-running transactions. Thus, concurrent Nix commands operating on the same flake could be blocked for an unbounded amount of time. WAL mode avoids this problem.
This change required updating the versions of the SQLite caches. For instance, `eval-cache-v5.sqlite` is now `eval-cache-v6.sqlite`.
- Enable parallel marking in bdwgc [#13708](https://github.com/NixOS/nix/pull/13708)
Previously marking was done by only one thread, which takes a long time if the heap gets big. Enabling parallel marking speeds up evaluation a lot, for example (on a Ryzen 9 5900X 12-Core):
* `nix search nixpkgs` from 24.3s to 18.9s.
* Evaluating the `NixOS/nix/2.21.2` flake regression test from 86.1s to 71.2s.
- New command `nix flake prefetch-inputs` [#13565](https://github.com/NixOS/nix/pull/13565)
This command fetches all inputs of a flake in parallel. This can be a lot faster than the serialized on-demand fetching during regular flake evaluation. The downside is that it may fetch inputs that aren't normally used.
- Add `warn-short-path-literals` setting [#13489](https://github.com/NixOS/nix/pull/13489)
This setting, when enabled, causes Nix to emit warnings when encountering relative path literals that don't start with `.` or `/`, for instance suggesting that `foo/bar` should be rewritten to `./foo/bar`.
- When updating a lock, respect the input's lock file [#13437](https://github.com/NixOS/nix/pull/13437)
For example, if a flake has a lock for `a` and `a/b`, and we change the flakeref for `a`, previously Nix would fetch the latest version of `b` rather than using the lock for `b` from `a`.
- Implement support for Git hashing with SHA-256 [#13543](https://github.com/NixOS/nix/pull/13543)
The experimental support for [Git-hashing](@docroot@/development/experimental-features.md#xp-feature-git-hashing) store objects now also includes support for SHA-256, not just SHA-1, in line with upstream Git.
## Contributors
This release was made possible by the following 34 contributors:
- John Soo [**(@jsoo1)**](https://github.com/jsoo1)
- Alan Urmancheev [**(@alurm)**](https://github.com/alurm)
- Manse [**(@PedroManse)**](https://github.com/PedroManse)
- Pol Dellaiera [**(@drupol)**](https://github.com/drupol)
- DavHau [**(@DavHau)**](https://github.com/DavHau)
- Leandro Emmanuel Reina Kiperman [**(@kip93)**](https://github.com/kip93)
- h0nIg [**(@h0nIg)**](https://github.com/h0nIg)
- Philip Taron [**(@philiptaron)**](https://github.com/philiptaron)
- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
- Connor Baker [**(@ConnorBaker)**](https://github.com/ConnorBaker)
- kenji [**(@a-kenji)**](https://github.com/a-kenji)
- Oleksandr Knyshuk [**(@k1gen)**](https://github.com/k1gen)
- Maciej Krüger [**(@mkg20001)**](https://github.com/mkg20001)
- Justin Bailey [**(@jgbailey-well)**](https://github.com/jgbailey-well)
- Emily [**(@emilazy)**](https://github.com/emilazy)
- Volker Diels-Grabsch [**(@vog)**](https://github.com/vog)
- gustavderdrache [**(@gustavderdrache)**](https://github.com/gustavderdrache)
- Elliot Cameron [**(@de11n)**](https://github.com/de11n)
- Alexander V. Nikolaev [**(@avnik)**](https://github.com/avnik)
- tomberek [**(@tomberek)**](https://github.com/tomberek)
- Matthew Kenigsberg [**(@mkenigs)**](https://github.com/mkenigs)
- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
- Cosima Neidahl [**(@OPNA2608)**](https://github.com/OPNA2608)
- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
- m4dc4p [**(@m4dc4p)**](https://github.com/m4dc4p)
- Graham Christensen [**(@grahamc)**](https://github.com/grahamc)
- Jason Yundt [**(@Jayman2000)**](https://github.com/Jayman2000)
- Jens Petersen [**(@juhp)**](https://github.com/juhp)
- the-sun-will-rise-tomorrow [**(@the-sun-will-rise-tomorrow)**](https://github.com/the-sun-will-rise-tomorrow)
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
- AGawas [**(@aln730)**](https://github.com/aln730)
- Robert Hensing [**(@roberth)**](https://github.com/roberth)
- Dmitry Bogatov [**(@KAction)**](https://github.com/KAction)
- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
- Philipp Otterbein
+130
View File
@@ -0,0 +1,130 @@
# Release 2.32.0 (2025-10-06)
## Incompatible changes
- Removed support for daemons and clients older than Nix 2.0 [#13951](https://github.com/NixOS/nix/pull/13951)
We have dropped support in the daemon worker protocol for daemons and clients that don't speak at least version 18 of the protocol. This first Nix release that supports this version is Nix 2.0, released in February 2018.
- Derivation JSON format now uses store path basenames only [#13570](https://github.com/NixOS/nix/issues/13570) [#13980](https://github.com/NixOS/nix/pull/13980)
Experience with many JSON frameworks (e.g. nlohmann/json in C++, Serde in Rust, and Aeson in Haskell) has shown that the use of the store directory in JSON formats is an impediment to systematic JSON formats, because it requires the serializer/deserializer to take an extra paramater (the store directory).
We ultimately want to rectify this issue with all JSON formats to the extent allowed by our stability promises. To start with, we are changing the JSON format for derivations because the `nix derivation` commands are — in addition to being formally unstable — less widely used than other unstable commands.
See the documentation on the [JSON format for derivations](@docroot@/protocols/json/derivation.md) for further details.
- C API: `nix_get_attr_name_byidx`, `nix_get_attr_byidx` take a `nix_value *` instead of `const nix_value *` [#13987](https://github.com/NixOS/nix/pull/13987)
In order to accommodate a more optimized internal representation of attribute set merges these functions require
a mutable `nix_value *` that might be modified on access. This does *not* break the ABI of these functions.
## New features
- C API: Add lazy attribute and list item accessors [#14030](https://github.com/NixOS/nix/pull/14030)
The C API now includes lazy accessor functions for retrieving values from lists and attribute sets without forcing evaluation:
- `nix_get_list_byidx_lazy()` - Get a list element without forcing its evaluation
- `nix_get_attr_byname_lazy()` - Get an attribute value by name without forcing evaluation
- `nix_get_attr_byidx_lazy()` - Get an attribute by index without forcing evaluation
These functions are useful when forwarding unevaluated sub-values to other lists, attribute sets, or function calls. They allow more efficient handling of Nix values by deferring evaluation until actually needed.
Additionally, bounds checking has been improved for all `_byidx` functions to properly validate indices before access, preventing potential out-of-bounds errors.
The documentation for `NIX_ERR_KEY` error handling has also been clarified to specify when this error code is returned.
- HTTP binary caches now support transparent compression for metadata
HTTP binary cache stores can now compress `.narinfo`, `.ls`, and build log files before uploading them,
reducing bandwidth usage and storage requirements. The compression is applied transparently using the
`Content-Encoding` header, allowing compatible clients to automatically decompress the files.
Three new configuration options control this behavior:
- `narinfo-compression`: Compression method for `.narinfo` files
- `ls-compression`: Compression method for `.ls` files
- `log-compression`: Compression method for build logs in `log/` directory
Example usage:
```
nix copy --to 'http://cache.example.com?narinfo-compression=gzip&ls-compression=gzip' /nix/store/...
nix store copy-log --to 'http://cache.example.com?log-compression=br' /nix/store/...
```
- Temporary build directories no longer include derivation names [#13839](https://github.com/NixOS/nix/pull/13839)
Temporary build directories created during derivation builds no longer include the derivation name in their path to avoid build failures when the derivation name is too long. This change ensures predictable prefix lengths for build directories under `/nix/var/nix/builds`.
- External derivation builders [#14145](https://github.com/NixOS/nix/pull/14145)
These are helper programs that Nix calls to perform derivations for specified system types, e.g. by using QEMU to emulate a different type of platform. For more information, see the [`external-builders` setting](../command-ref/conf-file.md#conf-external-builders).
This is currently an experimental feature.
## Performance improvements
- Optimize memory usage of attribute set merges [#13987](https://github.com/NixOS/nix/pull/13987)
[Attribute set update operations](@docroot@/language/operators.md#update) have been optimized to
reduce reallocations in cases when the second operand is small.
For typical evaluations of nixpkgs this optimization leads to ~20% less memory allocated in total
without significantly affecting evaluation performance.
See [eval-attrset-update-layer-rhs-threshold](@docroot@/command-ref/conf-file.md#conf-eval-attrset-update-layer-rhs-threshold)
- Substituted flake inputs are no longer re-copied to the store [#14041](https://github.com/NixOS/nix/pull/14041)
Since 2.25, Nix would fail to store a cache entry for substituted flake inputs, which in turn would cause them to be re-copied to the store on initial evaluation. Caching these inputs results in a near doubling of performance in some cases — especially on I/O-bound machines and when using commands that fetch many inputs, like `nix flake [archive|prefetch-inputs]`.
- `nix flake check` now skips derivations that can be substituted [#13574](https://github.com/NixOS/nix/pull/13574)
Previously, `nix flake check` would evaluate and build/substitute all
derivations. Now, it will skip downloading derivations that can be substituted.
This can drastically decrease the time invocations take in environments where
checks may already be cached (like in CI).
- `fetchTarball` and `fetchurl` now correctly substitute (#14138)
At some point we stopped substituting calls to `fetchTarball` and `fetchurl` with a set `narHash` to avoid incorrectly substituting things in `fetchTree`, even though it would be safe to substitute when calling the legacy `fetch{Tarball,url}`. This fixes that regression where it is safe.
- Started moving AST allocations into a bump allocator [#14088](https://github.com/NixOS/nix/issues/14088)
This leaves smaller, immutable structures in the AST. So far this saves about 2% memory on a NixOS config evaluation.
## Contributors
This release was made possible by the following 32 contributors:
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
- dram [**(@dramforever)**](https://github.com/dramforever)
- Ephraim Siegfried [**(@EphraimSiegfried)**](https://github.com/EphraimSiegfried)
- Robert Hensing [**(@roberth)**](https://github.com/roberth)
- Taeer Bar-Yam [**(@Radvendii)**](https://github.com/Radvendii)
- Emily [**(@emilazy)**](https://github.com/emilazy)
- Jens Petersen [**(@juhp)**](https://github.com/juhp)
- Bernardo Meurer [**(@lovesegfault)**](https://github.com/lovesegfault)
- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
- Leandro Emmanuel Reina Kiperman [**(@kip93)**](https://github.com/kip93)
- Marie [**(@NyCodeGHG)**](https://github.com/NyCodeGHG)
- Ethan Evans [**(@ethanavatar)**](https://github.com/ethanavatar)
- Yaroslav Bolyukin [**(@CertainLach)**](https://github.com/CertainLach)
- Matej Urbas [**(@urbas)**](https://github.com/urbas)
- Jami Kettunen [**(@JamiKettunen)**](https://github.com/JamiKettunen)
- Clayton [**(@netadr)**](https://github.com/netadr)
- Grégory Marti [**(@gmarti)**](https://github.com/gmarti)
- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
- rszyma [**(@rszyma)**](https://github.com/rszyma)
- Philip Wilk [**(@philipwilk)**](https://github.com/philipwilk)
- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
- Tom Westerhout [**(@twesterhout)**](https://github.com/twesterhout)
- Tristan Ross [**(@RossComputerGuy)**](https://github.com/RossComputerGuy)
- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
- Jean-François Roche [**(@jfroche)**](https://github.com/jfroche)
- Seth Flynn [**(@getchoo)**](https://github.com/getchoo)
- éclairevoyant [**(@eclairevoyant)**](https://github.com/eclairevoyant)
- Glen Huang [**(@hgl)**](https://github.com/hgl)
- osman - オスマン [**(@osbm)**](https://github.com/osbm)
- David McFarland [**(@corngood)**](https://github.com/corngood)
- Cole Helbling [**(@cole-h)**](https://github.com/cole-h)
- Sinan Mohd [**(@sinanmohd)**](https://github.com/sinanmohd)
- Philipp Otterbein
+2 -2
View File
@@ -9,7 +9,7 @@ This is where Nix distinguishes itself.
## Store Derivation {#store-derivation}
A derivation is a specification for running an executable on precisely defined input to produce on more [store objects][store object].
A derivation is a specification for running an executable on precisely defined input to produce one or more [store objects][store object].
These store objects are known as the derivation's *outputs*.
Derivations are *built*, in which case the process is spawned according to the spec, and when it exits, required to leave behind files which will (after post-processing) become the outputs of the derivation.
@@ -106,7 +106,7 @@ The system type on which the [`builder`](#attr-builder) executable is meant to b
A necessary condition for Nix to schedule a given derivation on some [Nix instance] is for the "system" of that derivation to match that instance's [`system` configuration option] or [`extra-platforms` configuration option].
By putting the `system` in each derivation, Nix allows *heterogenous* build plans, where not all steps can be run on the same machine or same sort of machine.
By putting the `system` in each derivation, Nix allows *heterogeneous* build plans, where not all steps can be run on the same machine or same sort of machine.
Nix can schedule builds such that it automatically builds on other platforms by [forwarding build requests](@docroot@/advanced-topics/distributed-builds.md) to other Nix instances.
[`system` configuration option]: @docroot@/command-ref/conf-file.md#conf-system
+2 -1
View File
@@ -20,7 +20,8 @@ The graph of references excluding self-references thus forms a [directed acyclic
[directed acyclic graph]: @docroot@/glossary.md#gloss-directed-acyclic-graph
We can take the [transitive closure] of the references graph, which any pair of store objects have an edge not if there is a single reference from the first to the second, but a path of one or more references from the first to the second.
We can take the [transitive closure] of the references graph, in which any pair of store objects have an edge if a *path* of one or more references exists from the first to the second object.
(A single reference always forms a path which is one reference long, but longer paths may connect objects which have no direct reference between them.)
The *requisites* of a store object are all store objects reachable by paths of references which start with given store object's references.
[transitive closure]: https://en.wikipedia.org/wiki/Transitive_closure
+3
View File
@@ -281,7 +281,10 @@ let
# may get replaced by pkgs.dockerTools.caCertificates
mkdir -p $out/etc/ssl/certs
# Old NixOS compatibility.
ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt $out/etc/ssl/certs
# NixOS canonical location
ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt $out/etc/ssl/certs/ca-certificates.crt
cat $passwdContentsPath > $out/etc/passwd
echo "" >> $out/etc/passwd
Generated
+3 -3
View File
@@ -63,11 +63,11 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1755442223,
"narHash": "sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI=",
"lastModified": 1756178832,
"narHash": "sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "cd32a774ac52caaa03bcfc9e7591ac8c18617ced",
"rev": "d98ce345cdab58477ca61855540999c86577d19d",
"type": "github"
},
"original": {
+4
View File
@@ -413,6 +413,10 @@
supportsCross = false;
};
"nix-json-schema-checks" = {
supportsCross = false;
};
"nix-perl-bindings" = {
supportsCross = false;
};
+1 -1
View File
@@ -46,7 +46,7 @@ The team meets twice a week (times are denoted in the [Europe/Amsterdam](https:/
- mark it as draft if it is blocked on the contributor
- escalate it back to the team by moving it to To discuss, and leaving a comment as to why the issue needs to be discussed again.
- Work meeting: Mondays 14:00-16:00 Europe/Amsterdam see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
- Work meeting: Mondays 18:00-20:00 Europe/Amsterdam; see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
1. Code review on pull requests from [In review](#in-review).
2. Other chores and tasks.
@@ -185,5 +185,44 @@
"gwenn.lebihan7@gmail.com": "gwennlbh",
"hey@ewen.works": "gwennlbh",
"matt@sturgeon.me.uk": "MattSturgeon",
"pbsds@hotmail.com": "pbsds"
"pbsds@hotmail.com": "pbsds",
"sergei@zimmerman.foo": "xokdvium",
"v@njh.eu": "vog",
"pedro.manse@dmk3.com.br": "PedroManse",
"arnavgawas707@gmail.com": "aln730",
"mkg20001@gmail.com": "mkg20001",
"avn@avnik.info": "avnik",
"olk@disr.it": "k1gen",
"108410815+alurm@users.noreply.github.com": "alurm",
"kaction.cc@gmail.com": "KAction",
"juhpetersen@gmail.com": "juhp",
"opna2608@protonmail.com": "OPNA2608",
"jgbailey@gmail.com": "m4dc4p",
"justin.bailey@well.co": "jgbailey-well",
"130508846+de11n@users.noreply.github.com": "de11n",
"ConnorBaker01@Gmail.com": "ConnorBaker",
"jsoo1@asu.edu": "jsoo1",
"hsngrmpf+github@gmail.com": "DavHau",
"matthew@floxdev.com": "mkenigs",
"taeer@bar-yam.me": "Radvendii",
"beme@anthropic.com": "lovesegfault",
"osbm@osbm.dev": "osbm",
"jami.kettunen@protonmail.com": "JamiKettunen",
"ephraim.siegfried@hotmail.com": "EphraimSiegfried",
"rszyma.dev@gmail.com": "rszyma",
"tristan.ross@determinate.systems": "RossComputerGuy",
"corngood@gmail.com": "corngood",
"jfroche@pyxel.be": "jfroche",
"848000+eclairevoyant@users.noreply.github.com": "eclairevoyant",
"petersen@redhat.com": "juhp",
"dramforever@live.com": "dramforever",
"me@glenhuang.com": "hgl",
"philip.wilk@fivium.co.uk": "philipwilk",
"me@nycode.dev": "NyCodeGHG",
"14264576+twesterhout@users.noreply.github.com": "twesterhout",
"sinan@sinanmohd.com": "sinanmohd",
"42688647+netadr@users.noreply.github.com": "netadr",
"matej.urbas@gmail.com": "urbas",
"ethanalexevans@gmail.com": "ethanavatar",
"greg.marti@gmail.com": "gmarti"
}
@@ -162,5 +162,39 @@
"pbsds": "Peder Bergebakken Sundt",
"egorkonovalov": "Egor Konovalov",
"jayeshv": "jayeshv",
"vcunat": "Vladim\u00edr \u010cun\u00e1t"
"vcunat": "Vladim\u00edr \u010cun\u00e1t",
"mkenigs": "Matthew Kenigsberg",
"alurm": "Alan Urmancheev",
"jgbailey-well": "Justin Bailey",
"k1gen": "Oleksandr Knyshuk",
"juhp": "Jens Petersen",
"de11n": "Elliot Cameron",
"jsoo1": "John Soo",
"m4dc4p": null,
"PedroManse": "Manse",
"OPNA2608": "Cosima Neidahl",
"mkg20001": "Maciej Kr\u00fcger",
"avnik": "Alexander V. Nikolaev",
"DavHau": null,
"aln730": "AGawas",
"vog": "Volker Diels-Grabsch",
"corngood": "David McFarland",
"twesterhout": "Tom Westerhout",
"JamiKettunen": "Jami Kettunen",
"dramforever": "dram",
"philipwilk": "Philip Wilk",
"netadr": "Clayton",
"NyCodeGHG": "Marie",
"jfroche": "Jean-Fran\u00e7ois Roche",
"urbas": "Matej Urbas",
"osbm": "osman - \u30aa\u30b9\u30de\u30f3",
"rszyma": null,
"eclairevoyant": "\u00e9clairevoyant",
"Radvendii": "Taeer Bar-Yam",
"sinanmohd": "Sinan Mohd",
"ethanavatar": "Ethan Evans",
"gmarti": "Gr\u00e9gory Marti",
"lovesegfault": "Bernardo Meurer",
"EphraimSiegfried": "Ephraim Siegfried",
"hgl": "Glen Huang"
}
-145
View File
@@ -104,151 +104,6 @@
};
shellcheck = {
enable = true;
excludes = [
# We haven't linted these files yet
''^config/install-sh$''
''^misc/bash/completion\.sh$''
''^misc/fish/completion\.fish$''
''^misc/zsh/completion\.zsh$''
''^scripts/create-darwin-volume\.sh$''
''^scripts/install-darwin-multi-user\.sh$''
''^scripts/install-multi-user\.sh$''
''^scripts/install-systemd-multi-user\.sh$''
''^src/nix/get-env\.sh$''
''^tests/functional/ca/build-dry\.sh$''
''^tests/functional/ca/build-with-garbage-path\.sh$''
''^tests/functional/ca/common\.sh$''
''^tests/functional/ca/concurrent-builds\.sh$''
''^tests/functional/ca/eval-store\.sh$''
''^tests/functional/ca/gc\.sh$''
''^tests/functional/ca/import-from-derivation\.sh$''
''^tests/functional/ca/new-build-cmd\.sh$''
''^tests/functional/ca/nix-shell\.sh$''
''^tests/functional/ca/post-hook\.sh$''
''^tests/functional/ca/recursive\.sh$''
''^tests/functional/ca/repl\.sh$''
''^tests/functional/ca/selfref-gc\.sh$''
''^tests/functional/ca/why-depends\.sh$''
''^tests/functional/characterisation-test-infra\.sh$''
''^tests/functional/common/vars-and-functions\.sh$''
''^tests/functional/completions\.sh$''
''^tests/functional/compute-levels\.sh$''
''^tests/functional/config\.sh$''
''^tests/functional/db-migration\.sh$''
''^tests/functional/debugger\.sh$''
''^tests/functional/dependencies\.builder0\.sh$''
''^tests/functional/dependencies\.sh$''
''^tests/functional/dump-db\.sh$''
''^tests/functional/dyn-drv/build-built-drv\.sh$''
''^tests/functional/dyn-drv/common\.sh$''
''^tests/functional/dyn-drv/dep-built-drv\.sh$''
''^tests/functional/dyn-drv/eval-outputOf\.sh$''
''^tests/functional/dyn-drv/old-daemon-error-hack\.sh$''
''^tests/functional/dyn-drv/recursive-mod-json\.sh$''
''^tests/functional/eval-store\.sh$''
''^tests/functional/export-graph\.sh$''
''^tests/functional/export\.sh$''
''^tests/functional/extra-sandbox-profile\.sh$''
''^tests/functional/fetchClosure\.sh$''
''^tests/functional/fetchGit\.sh$''
''^tests/functional/fetchGitRefs\.sh$''
''^tests/functional/fetchGitSubmodules\.sh$''
''^tests/functional/fetchGitVerification\.sh$''
''^tests/functional/fetchMercurial\.sh$''
''^tests/functional/fixed\.builder1\.sh$''
''^tests/functional/fixed\.builder2\.sh$''
''^tests/functional/fixed\.sh$''
''^tests/functional/flakes/absolute-paths\.sh$''
''^tests/functional/flakes/check\.sh$''
''^tests/functional/flakes/config\.sh$''
''^tests/functional/flakes/flakes\.sh$''
''^tests/functional/flakes/follow-paths\.sh$''
''^tests/functional/flakes/prefetch\.sh$''
''^tests/functional/flakes/run\.sh$''
''^tests/functional/flakes/show\.sh$''
''^tests/functional/formatter\.sh$''
''^tests/functional/formatter\.simple\.sh$''
''^tests/functional/gc-auto\.sh$''
''^tests/functional/gc-concurrent\.builder\.sh$''
''^tests/functional/gc-concurrent\.sh$''
''^tests/functional/gc-concurrent2\.builder\.sh$''
''^tests/functional/gc-non-blocking\.sh$''
''^tests/functional/hash-convert\.sh$''
''^tests/functional/impure-derivations\.sh$''
''^tests/functional/impure-eval\.sh$''
''^tests/functional/install-darwin\.sh$''
''^tests/functional/legacy-ssh-store\.sh$''
''^tests/functional/linux-sandbox\.sh$''
''^tests/functional/local-overlay-store/add-lower-inner\.sh$''
''^tests/functional/local-overlay-store/add-lower\.sh$''
''^tests/functional/local-overlay-store/bad-uris\.sh$''
''^tests/functional/local-overlay-store/build-inner\.sh$''
''^tests/functional/local-overlay-store/build\.sh$''
''^tests/functional/local-overlay-store/check-post-init-inner\.sh$''
''^tests/functional/local-overlay-store/check-post-init\.sh$''
''^tests/functional/local-overlay-store/common\.sh$''
''^tests/functional/local-overlay-store/delete-duplicate-inner\.sh$''
''^tests/functional/local-overlay-store/delete-duplicate\.sh$''
''^tests/functional/local-overlay-store/delete-refs-inner\.sh$''
''^tests/functional/local-overlay-store/delete-refs\.sh$''
''^tests/functional/local-overlay-store/gc-inner\.sh$''
''^tests/functional/local-overlay-store/gc\.sh$''
''^tests/functional/local-overlay-store/optimise-inner\.sh$''
''^tests/functional/local-overlay-store/optimise\.sh$''
''^tests/functional/local-overlay-store/redundant-add-inner\.sh$''
''^tests/functional/local-overlay-store/redundant-add\.sh$''
''^tests/functional/local-overlay-store/remount\.sh$''
''^tests/functional/local-overlay-store/stale-file-handle-inner\.sh$''
''^tests/functional/local-overlay-store/stale-file-handle\.sh$''
''^tests/functional/local-overlay-store/verify-inner\.sh$''
''^tests/functional/local-overlay-store/verify\.sh$''
''^tests/functional/logging\.sh$''
''^tests/functional/misc\.sh$''
''^tests/functional/multiple-outputs\.sh$''
''^tests/functional/nested-sandboxing\.sh$''
''^tests/functional/nested-sandboxing/command\.sh$''
''^tests/functional/nix-build\.sh$''
''^tests/functional/nix-channel\.sh$''
''^tests/functional/nix-collect-garbage-d\.sh$''
''^tests/functional/nix-copy-ssh-common\.sh$''
''^tests/functional/nix-copy-ssh-ng\.sh$''
''^tests/functional/nix-copy-ssh\.sh$''
''^tests/functional/nix-daemon-untrusting\.sh$''
''^tests/functional/nix-profile\.sh$''
''^tests/functional/nix-shell\.sh$''
''^tests/functional/nix_path\.sh$''
''^tests/functional/optimise-store\.sh$''
''^tests/functional/output-normalization\.sh$''
''^tests/functional/parallel\.builder\.sh$''
''^tests/functional/parallel\.sh$''
''^tests/functional/pass-as-file\.sh$''
''^tests/functional/path-from-hash-part\.sh$''
''^tests/functional/path-info\.sh$''
''^tests/functional/placeholders\.sh$''
''^tests/functional/post-hook\.sh$''
''^tests/functional/pure-eval\.sh$''
''^tests/functional/push-to-store-old\.sh$''
''^tests/functional/push-to-store\.sh$''
''^tests/functional/read-only-store\.sh$''
''^tests/functional/readfile-context\.sh$''
''^tests/functional/recursive\.sh$''
''^tests/functional/referrers\.sh$''
''^tests/functional/remote-store\.sh$''
''^tests/functional/repair\.sh$''
''^tests/functional/restricted\.sh$''
''^tests/functional/search\.sh$''
''^tests/functional/secure-drv-outputs\.sh$''
''^tests/functional/selfref-gc\.sh$''
''^tests/functional/shell\.shebang\.sh$''
''^tests/functional/simple\.builder\.sh$''
''^tests/functional/supplementary-groups\.sh$''
''^tests/functional/toString-path\.sh$''
''^tests/functional/user-envs-migration\.sh$''
''^tests/functional/user-envs-test-case\.sh$''
''^tests/functional/user-envs\.builder\.sh$''
''^tests/functional/user-envs\.sh$''
''^tests/functional/why-depends\.sh$''
];
};
};
};
+58
View File
@@ -0,0 +1,58 @@
#!/usr/bin/env bash
set -euo pipefail
# debug:
# set -x
START_REF="${1}"
END_REF="${2:-upstream/master}"
# Get the merge base
MERGE_BASE=$(git merge-base "$START_REF" "$END_REF")
unset START_REF
# Get date range
START_DATE=$(git show -s --format=%cI "$MERGE_BASE")
END_DATE=$(git show -s --format=%cI "$END_REF")
echo "Checking PRs merged between $START_DATE and $END_DATE" >&2
# Get all commits between merge base and HEAD
COMMITS=$(git rev-list "$MERGE_BASE..$END_REF")
# Convert to set for fast lookup
declare -A commit_set
for commit in $COMMITS; do
commit_set["$commit"]=1
done
# Get the current changelog
LOG_DONE="$(changelog-d doc/manual/rl-next)"
is_done(){
local nr="$1"
echo "$LOG_DONE" | grep -E "^- .*/pull/$nr)"
}
# Query merged PRs in date range
gh pr list \
--repo NixOS/nix \
--state merged \
--limit 1000 \
--json number,title,author,mergeCommit \
--search "merged:$START_DATE..$END_DATE" | \
jq -r '.[] | [.number, .mergeCommit.oid, .title, .author.login] | @tsv' | \
while IFS=$'\t' read -r pr_num merge_commit _title author; do
# Check if this PR's merge commit is in our branch
if [[ -n "${commit_set[$merge_commit]:-}" ]]; then
# Full detail, not suitable for comment due to mass ping and duplicate title
# echo "- #$pr_num $_title (@$author)"
echo "- #$pr_num ($author)"
if is_done "$pr_num"
then
echo " - [x] has note"
else
echo " - [ ] has note"
fi
echo " - [ ] skip"
fi
done
+9 -4
View File
@@ -24,11 +24,18 @@ release:
* In a checkout of the Nix repo, make sure you're on `master` and run
`git pull`.
* Compile a release notes to-do list by running
```console
$ ./maintainers/release-notes-todo PREV_RELEASE HEAD
```
* Compile the release notes by running
```console
$ export VERSION=X.YY
$ git checkout -b release-notes
$ export GITHUB_TOKEN=...
$ ./maintainers/release-notes
```
@@ -126,6 +133,8 @@ release:
Commit and push this to the maintenance branch.
* Create a backport label.
* Bump the version of `master`:
```console
@@ -140,10 +149,6 @@ release:
Make a pull request and auto-merge it.
* Create a backport label.
* Add the new backport label to `.mergify.yml`.
* Post an [announcement on Discourse](https://discourse.nixos.org/c/announcements/8), including the contents of
`rl-$VERSION.md`.
+4 -1
View File
@@ -41,8 +41,10 @@ subproject('libexpr-c')
subproject('libflake-c')
subproject('libmain-c')
asan_enabled = 'address' in get_option('b_sanitize')
# Language Bindings
if get_option('bindings') and not meson.is_cross_build()
if get_option('bindings') and not meson.is_cross_build() and not asan_enabled
subproject('perl')
endif
@@ -58,3 +60,4 @@ if get_option('unit-tests')
subproject('libflake-tests')
endif
subproject('nix-functional-tests')
subproject('json-schema-checks')
+1
View File
@@ -1,3 +1,4 @@
# shellcheck shell=bash
function _complete_nix {
local -a words
local cword cur
+1
View File
@@ -1,3 +1,4 @@
# shellcheck disable=all
function _nix_complete
# Get the current command up to a cursor.
# - Behaves correctly even with pipes and nested in commands like env.
+1
View File
@@ -1,3 +1,4 @@
# shellcheck disable=all
#compdef nix
function _nix() {
@@ -0,0 +1,6 @@
extern "C" [[gnu::retain, gnu::weak]] const char * __asan_default_options()
{
// We leak a bunch of memory knowingly on purpose. It's not worthwhile to
// diagnose that memory being leaked for now.
return "abort_on_error=1:print_summary=1:detect_leaks=0:detect_odr_violation=0";
}
@@ -0,0 +1,11 @@
# Clang gets grumpy about missing libasan symbols if -shared-libasan is not
# passed when building shared libs, at least on Linux
if cxx.get_id() == 'clang' and ('address' in get_option('b_sanitize') or 'undefined' in get_option(
'b_sanitize',
))
add_project_link_arguments('-shared-libasan', language : 'cpp')
endif
if 'address' in get_option('b_sanitize')
deps_other += declare_dependency(sources : 'asan-options.cc')
endif
@@ -0,0 +1,32 @@
can_wrap_assert_fail_test_code = '''
#include <cstdlib>
#include <cassert>
int main()
{
assert(0);
}
extern "C" void * __real___assert_fail(const char *, const char *, unsigned int, const char *);
extern "C" void *
__wrap___assert_fail(const char *, const char *, unsigned int, const char *)
{
return __real___assert_fail(nullptr, nullptr, 0, nullptr);
}
'''
wrap_assert_fail_args = [ '-Wl,--wrap=__assert_fail' ]
can_wrap_assert_fail = cxx.links(
can_wrap_assert_fail_test_code,
args : wrap_assert_fail_args,
name : 'linker can wrap __assert_fail',
)
if can_wrap_assert_fail
deps_other += declare_dependency(
sources : 'wrap-assert-fail.cc',
link_args : wrap_assert_fail_args,
)
endif
@@ -0,0 +1,17 @@
#include "nix/util/error.hh"
#include <cstdio>
#include <cstdlib>
#include <cinttypes>
#include <string_view>
extern "C" [[noreturn]] void __attribute__((weak))
__wrap___assert_fail(const char * assertion, const char * file, unsigned int line, const char * function)
{
char buf[512];
int n =
snprintf(buf, sizeof(buf), "Assertion '%s' failed in %s at %s:%" PRIuLEAST32, assertion, function, file, line);
if (n < 0)
nix::panic("Assertion failed and could not format error message");
nix::panic(std::string_view(buf, std::min(static_cast<int>(sizeof(buf)), n)));
}
+14 -7
View File
@@ -5,6 +5,15 @@ if not (host_machine.system() == 'windows' and cxx.get_id() == 'gcc')
deps_private += dependency('threads')
endif
if host_machine.system() == 'cygwin'
# -std=gnu on cygwin defines 'unix', which conflicts with the namespace
add_project_arguments(
'-D_POSIX_C_SOURCE=200809L',
'-D_GNU_SOURCE',
language : 'cpp',
)
endif
add_project_arguments(
'-Wdeprecated-copy',
'-Werror=suggest-override',
@@ -33,10 +42,8 @@ if cxx.get_id() == 'clang'
add_project_arguments('-fpch-instantiate-templates', language : 'cpp')
endif
# Clang gets grumpy about missing libasan symbols if -shared-libasan is not
# passed when building shared libs, at least on Linux
if cxx.get_id() == 'clang' and ('address' in get_option('b_sanitize') or 'undefined' in get_option(
'b_sanitize',
))
add_project_link_arguments('-shared-libasan', language : 'cpp')
endif
# Darwin ld doesn't like "X.Y.Zpre"
nix_soversion = meson.project_version().split('pre')[0]
subdir('assert-fail')
subdir('asan-options')
@@ -3,6 +3,6 @@
# This is needed for std::atomic on some platforms
# We did not manage to test this reliably on all platforms, so we hardcode
# it for now.
if host_machine.cpu_family() == 'arm'
if host_machine.cpu_family() in [ 'arm', 'ppc' ]
deps_other += cxx.find_library('atomic')
endif
+56 -1
View File
@@ -164,6 +164,24 @@ let
};
mesonLibraryLayer = finalAttrs: prevAttrs: {
preConfigure =
let
interpositionFlags = [
"-fno-semantic-interposition"
"-Wl,-Bsymbolic-functions"
];
in
# NOTE: By default GCC disables interprocedular optimizations (in particular inlining) for
# position-independent code and thus shared libraries.
# Since LD_PRELOAD tricks aren't worth losing out on optimizations, we disable it for good.
# This is not the case for Clang, where inlining is done by default even without -fno-semantic-interposition.
# https://reviews.llvm.org/D102453
# https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup
prevAttrs.preConfigure or ""
+ lib.optionalString stdenv.cc.isGNU ''
export CFLAGS="''${CFLAGS:-} ${toString interpositionFlags}"
export CXXFLAGS="''${CXXFLAGS:-} ${toString interpositionFlags}"
'';
outputs = prevAttrs.outputs or [ "out" ] ++ [ "dev" ];
};
@@ -186,6 +204,25 @@ let
mesonFlags = [ (lib.mesonBool "b_asneeded" false) ] ++ prevAttrs.mesonFlags or [ ];
};
enableSanitizersLayer =
finalAttrs: prevAttrs:
let
sanitizers = lib.optional scope.withASan "address" ++ lib.optional scope.withUBSan "undefined";
in
{
mesonFlags =
(prevAttrs.mesonFlags or [ ])
++ lib.optionals (lib.length sanitizers > 0) (
[
(lib.mesonOption "b_sanitize" (lib.concatStringsSep "," sanitizers))
]
++ (lib.optionals stdenv.cc.isClang [
# https://www.github.com/mesonbuild/meson/issues/764
(lib.mesonBool "b_lundef" false)
])
);
};
nixDefaultsLayer = finalAttrs: prevAttrs: {
strictDeps = prevAttrs.strictDeps or true;
enableParallelBuilding = true;
@@ -228,6 +265,16 @@ in
inherit filesetToSource;
/**
Whether meson components are built with [AddressSanitizer](https://clang.llvm.org/docs/AddressSanitizer.html).
*/
withASan = false;
/**
Whether meson components are built with [UndefinedBehaviorSanitizer](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html).
*/
withUBSan = false;
/**
A user-provided extension function to apply to each component derivation.
*/
@@ -314,6 +361,7 @@ in
setVersionLayer
mesonLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
mkMesonExecutable = mkPackageBuilder [
@@ -324,6 +372,7 @@ in
mesonLayer
mesonBuildLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
mkMesonLibrary = mkPackageBuilder [
@@ -335,6 +384,7 @@ in
mesonBuildLayer
mesonLibraryLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
@@ -388,6 +438,11 @@ in
*/
nix-external-api-docs = callPackage ../src/external-api-docs/package.nix { version = fineVersion; };
/**
JSON schema validation checks
*/
nix-json-schema-checks = callPackage ../src/json-schema-checks/package.nix { };
nix-perl-bindings = callPackage ../src/perl/package.nix { };
/**
@@ -440,7 +495,7 @@ in
Example:
```
overrideScope (finalScope: prevScope: { aws-sdk-cpp = null; })
overrideScope (finalScope: prevScope: { aws-crt-cpp = null; })
```
*/
overrideScope = f: (scope.overrideScope f).nix-everything;
+15 -68
View File
@@ -10,46 +10,12 @@
stdenv,
}:
let
prevStdenv = stdenv;
in
let
inherit (pkgs) lib;
stdenv = if prevStdenv.isDarwin && prevStdenv.isx86_64 then darwinStdenv else prevStdenv;
# Fix the following error with the default x86_64-darwin SDK:
#
# error: aligned allocation function of type 'void *(std::size_t, std::align_val_t)' is only available on macOS 10.13 or newer
#
# Despite the use of the 10.13 deployment target here, the aligned
# allocation function Clang uses with this setting actually works
# all the way back to 10.6.
# NOTE: this is not just a version constraint, but a request to make Darwin
# provide this version level of support. Removing this minimum version
# request will regress the above error.
darwinStdenv = pkgs.overrideSDK prevStdenv { darwinMinVersion = "10.13"; };
in
scope: {
inherit stdenv;
aws-sdk-cpp =
(pkgs.aws-sdk-cpp.override {
apis = [
"identity-management"
"s3"
"transfer"
];
customMemoryManagement = false;
}).overrideAttrs
{
# only a stripped down version is built, which takes a lot less resources
# to build, so we don't need a "big-parallel" machine.
requiredSystemFeatures = [ ];
};
boehmgc =
(pkgs.boehmgc.override {
enableLargeConfig = true;
@@ -76,6 +42,21 @@ scope: {
prevAttrs.postInstall;
});
# TODO: Remove this when https://github.com/NixOS/nixpkgs/pull/442682 is included in a stable release
toml11 =
if lib.versionAtLeast pkgs.toml11.version "4.4.0" then
pkgs.toml11
else
pkgs.toml11.overrideAttrs rec {
version = "4.4.0";
src = pkgs.fetchFromGitHub {
owner = "ToruNiina";
repo = "toml11";
tag = "v${version}";
hash = "sha256-sgWKYxNT22nw376ttGsTdg0AMzOwp8QH3E8mx0BZJTQ=";
};
};
# TODO Hack until https://github.com/NixOS/nixpkgs/issues/45462 is fixed.
boost =
(pkgs.boost.override {
@@ -93,38 +74,4 @@ scope: {
buildPhase = lib.replaceStrings [ "--without-python" ] [ "" ] old.buildPhase;
installPhase = lib.replaceStrings [ "--without-python" ] [ "" ] old.installPhase;
});
libgit2 =
if lib.versionAtLeast pkgs.libgit2.version "1.9.0" then
pkgs.libgit2
else
pkgs.libgit2.overrideAttrs (attrs: {
# libgit2: Nixpkgs 24.11 has < 1.9.0, which needs our patches
nativeBuildInputs =
attrs.nativeBuildInputs or [ ]
# gitMinimal does not build on Windows. See packbuilder patch.
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# Needed for `git apply`; see `prePatch`
pkgs.buildPackages.gitMinimal
];
# Only `git apply` can handle git binary patches
prePatch =
attrs.prePatch or ""
+ lib.optionalString (!stdenv.hostPlatform.isWindows) ''
patch() {
git apply
}
'';
patches =
attrs.patches or [ ]
++ [
./patches/libgit2-mempack-thin-packfile.patch
]
# gitMinimal does not build on Windows, but fortunately this patch only
# impacts interruptibility
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# binary patch; see `prePatch`
./patches/libgit2-packbuilder-callback-interruptible.patch
];
});
}
+42 -31
View File
@@ -70,6 +70,9 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
# We use this shell with the local checkout, not unpackPhase.
src = null;
# Workaround https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
# Remove when gdb fix is rolled out everywhere.
separateDebugInfo = false;
env = {
# For `make format`, to work without installing pre-commit
@@ -93,37 +96,45 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
++ map (transformFlag "libcmd") (ignoreCrossFile pkgs.nixComponents2.nix-cmd.mesonFlags);
nativeBuildInputs =
attrs.nativeBuildInputs or [ ]
++ pkgs.nixComponents2.nix-util.nativeBuildInputs
++ pkgs.nixComponents2.nix-store.nativeBuildInputs
++ pkgs.nixComponents2.nix-fetchers.nativeBuildInputs
++ pkgs.nixComponents2.nix-expr.nativeBuildInputs
++ lib.optionals havePerl pkgs.nixComponents2.nix-perl-bindings.nativeBuildInputs
++ lib.optionals buildCanExecuteHost pkgs.nixComponents2.nix-manual.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-internal-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-external-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-functional-tests.externalNativeBuildInputs
++ lib.optional (
!buildCanExecuteHost
# Hack around https://github.com/nixos/nixpkgs/commit/bf7ad8cfbfa102a90463433e2c5027573b462479
&& !(stdenv.hostPlatform.isWindows && stdenv.buildPlatform.isDarwin)
&& stdenv.hostPlatform.emulatorAvailable pkgs.buildPackages
&& lib.meta.availableOn stdenv.buildPlatform (stdenv.hostPlatform.emulator pkgs.buildPackages)
) pkgs.buildPackages.mesonEmulatorHook
++ [
pkgs.buildPackages.cmake
pkgs.buildPackages.gnused
pkgs.buildPackages.shellcheck
pkgs.buildPackages.changelog-d
modular.pre-commit.settings.package
(pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript)
pkgs.buildPackages.nixfmt-rfc-style
pkgs.buildPackages.gdb
]
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
)
++ lib.optional stdenv.hostPlatform.isLinux pkgs.buildPackages.mold-wrapped;
let
inputs =
attrs.nativeBuildInputs or [ ]
++ pkgs.nixComponents2.nix-util.nativeBuildInputs
++ pkgs.nixComponents2.nix-store.nativeBuildInputs
++ pkgs.nixComponents2.nix-fetchers.nativeBuildInputs
++ pkgs.nixComponents2.nix-expr.nativeBuildInputs
++ lib.optionals havePerl pkgs.nixComponents2.nix-perl-bindings.nativeBuildInputs
++ lib.optionals buildCanExecuteHost pkgs.nixComponents2.nix-manual.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-internal-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-external-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-functional-tests.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-json-schema-checks.externalNativeBuildInputs
++ lib.optional (
!buildCanExecuteHost
# Hack around https://github.com/nixos/nixpkgs/commit/bf7ad8cfbfa102a90463433e2c5027573b462479
&& !(stdenv.hostPlatform.isWindows && stdenv.buildPlatform.isDarwin)
&& stdenv.hostPlatform.emulatorAvailable pkgs.buildPackages
&& lib.meta.availableOn stdenv.buildPlatform (stdenv.hostPlatform.emulator pkgs.buildPackages)
) pkgs.buildPackages.mesonEmulatorHook
++ [
pkgs.buildPackages.cmake
pkgs.buildPackages.gnused
pkgs.buildPackages.changelog-d
modular.pre-commit.settings.package
(pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript)
pkgs.buildPackages.nixfmt-rfc-style
pkgs.buildPackages.shellcheck
pkgs.buildPackages.include-what-you-use
pkgs.buildPackages.gdb
]
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
)
++ lib.optional stdenv.hostPlatform.isLinux pkgs.buildPackages.mold-wrapped;
in
# FIXME: separateDebugInfo = false doesn't actually prevent -Wa,--compress-debug-sections
# from making its way into NIX_CFLAGS_COMPILE.
lib.filter (p: !lib.hasInfix "separate-debug-info" p) inputs;
buildInputs = [
pkgs.gbenchmark
+16 -3
View File
@@ -62,6 +62,7 @@ let
"nix-cmd"
"nix-cli"
"nix-functional-tests"
"nix-json-schema-checks"
]
++ lib.optionals enableBindings [
"nix-perl-bindings"
@@ -73,7 +74,7 @@ let
]
);
in
{
rec {
/**
An internal check to make sure our package listing is complete.
*/
@@ -145,13 +146,25 @@ in
)
);
buildNoGc =
# Builds with sanitizers already have GC disabled, so this buildNoGc can just
# point to buildWithSanitizers in order to reduce the load on hydra.
buildNoGc = buildWithSanitizers;
buildWithSanitizers =
let
components = forAllSystems (
system:
nixpkgsFor.${system}.native.nixComponents2.overrideScope (
let
pkgs = nixpkgsFor.${system}.native;
in
pkgs.nixComponents2.overrideScope (
self: super: {
# Boost coroutines fail with ASAN on darwin.
withASan = !pkgs.stdenv.buildPlatform.isDarwin;
withUBSan = true;
nix-expr = super.nix-expr.override { enableGC = false; };
# Unclear how to make Perl bindings work with a dynamically linked ASAN.
nix-perl-bindings = null;
}
)
);
@@ -1,282 +0,0 @@
commit 9bacade4a3ef4b6b26e2c02f549eef0e9eb9eaa2
Author: Robert Hensing <robert@roberthensing.nl>
Date: Sun Aug 18 20:20:36 2024 +0200
Add unoptimized git_mempack_write_thin_pack
diff --git a/include/git2/sys/mempack.h b/include/git2/sys/mempack.h
index 17da590a3..3688bdd50 100644
--- a/include/git2/sys/mempack.h
+++ b/include/git2/sys/mempack.h
@@ -44,6 +44,29 @@ GIT_BEGIN_DECL
*/
GIT_EXTERN(int) git_mempack_new(git_odb_backend **out);
+/**
+ * Write a thin packfile with the objects in the memory store.
+ *
+ * A thin packfile is a packfile that does not contain its transitive closure of
+ * references. This is useful for efficiently distributing additions to a
+ * repository over the network, but also finds use in the efficient bulk
+ * addition of objects to a repository, locally.
+ *
+ * This operation performs the (shallow) insert operations into the
+ * `git_packbuilder`, but does not write the packfile to disk;
+ * see `git_packbuilder_write_buf`.
+ *
+ * It also does not reset the memory store; see `git_mempack_reset`.
+ *
+ * @note This function may or may not write trees and blobs that are not
+ * referenced by commits. Currently everything is written, but this
+ * behavior may change in the future as the packer is optimized.
+ *
+ * @param backend The mempack backend
+ * @param pb The packbuilder to use to write the packfile
+ */
+GIT_EXTERN(int) git_mempack_write_thin_pack(git_odb_backend *backend, git_packbuilder *pb);
+
/**
* Dump all the queued in-memory writes to a packfile.
*
diff --git a/src/libgit2/odb_mempack.c b/src/libgit2/odb_mempack.c
index 6f27f45f8..0b61e2b66 100644
--- a/src/libgit2/odb_mempack.c
+++ b/src/libgit2/odb_mempack.c
@@ -132,6 +132,35 @@ cleanup:
return err;
}
+int git_mempack_write_thin_pack(git_odb_backend *backend, git_packbuilder *pb)
+{
+ struct memory_packer_db *db = (struct memory_packer_db *)backend;
+ const git_oid *oid;
+ size_t iter = 0;
+ int err = -1;
+
+ /* TODO: Implement the recency heuristics.
+ For this it probably makes sense to only write what's referenced
+ through commits, an option I've carved out for you in the docs.
+ wrt heuristics: ask your favorite LLM to translate https://git-scm.com/docs/pack-heuristics/en
+ to actual normal reference documentation. */
+ while (true) {
+ err = git_oidmap_iterate(NULL, db->objects, &iter, &oid);
+ if (err == GIT_ITEROVER) {
+ err = 0;
+ break;
+ }
+ if (err != 0)
+ return err;
+
+ err = git_packbuilder_insert(pb, oid, NULL);
+ if (err != 0)
+ return err;
+ }
+
+ return 0;
+}
+
int git_mempack_dump(
git_buf *pack,
git_repository *repo,
diff --git a/tests/libgit2/mempack/thinpack.c b/tests/libgit2/mempack/thinpack.c
new file mode 100644
index 000000000..604a4dda2
--- /dev/null
+++ b/tests/libgit2/mempack/thinpack.c
@@ -0,0 +1,196 @@
+#include "clar_libgit2.h"
+#include "git2/indexer.h"
+#include "git2/odb_backend.h"
+#include "git2/tree.h"
+#include "git2/types.h"
+#include "git2/sys/mempack.h"
+#include "git2/sys/odb_backend.h"
+#include "util.h"
+
+static git_repository *_repo;
+static git_odb_backend * _mempack_backend;
+
+void test_mempack_thinpack__initialize(void)
+{
+ git_odb *odb;
+
+ _repo = cl_git_sandbox_init_new("mempack_thinpack_repo");
+
+ cl_git_pass(git_mempack_new(&_mempack_backend));
+ cl_git_pass(git_repository_odb(&odb, _repo));
+ cl_git_pass(git_odb_add_backend(odb, _mempack_backend, 999));
+ git_odb_free(odb);
+}
+
+void _mempack_thinpack__cleanup(void)
+{
+ cl_git_sandbox_cleanup();
+}
+
+/*
+ Generating a packfile for an unchanged repo works and produces an empty packfile.
+ Even if we allow this scenario to be detected, it shouldn't misbehave if the
+ application is unaware of it.
+*/
+void test_mempack_thinpack__empty(void)
+{
+ git_packbuilder *pb;
+ int version;
+ int n;
+ git_buf buf = GIT_BUF_INIT;
+
+ git_packbuilder_new(&pb, _repo);
+
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_assert_in_range(12, buf.size, 1024 /* empty packfile is >0 bytes, but certainly not that big */);
+ cl_assert(buf.ptr[0] == 'P');
+ cl_assert(buf.ptr[1] == 'A');
+ cl_assert(buf.ptr[2] == 'C');
+ cl_assert(buf.ptr[3] == 'K');
+ version = (buf.ptr[4] << 24) | (buf.ptr[5] << 16) | (buf.ptr[6] << 8) | buf.ptr[7];
+ /* Subject to change. https://git-scm.com/docs/pack-format: Git currently accepts version number 2 or 3 but generates version 2 only.*/
+ cl_assert_equal_i(2, version);
+ n = (buf.ptr[8] << 24) | (buf.ptr[9] << 16) | (buf.ptr[10] << 8) | buf.ptr[11];
+ cl_assert_equal_i(0, n);
+ git_buf_dispose(&buf);
+
+ git_packbuilder_free(pb);
+}
+
+#define LIT_LEN(x) x, sizeof(x) - 1
+
+/*
+ Check that git_mempack_write_thin_pack produces a thin packfile.
+*/
+void test_mempack_thinpack__thin(void)
+{
+ /* Outline:
+ - Create tree 1
+ - Flush to packfile A
+ - Create tree 2
+ - Flush to packfile B
+
+ Tree 2 has a new blob and a reference to a blob from tree 1.
+
+ Expectation:
+ - Packfile B is thin and does not contain the objects from packfile A
+ */
+
+
+ git_oid oid_blob_1;
+ git_oid oid_blob_2;
+ git_oid oid_blob_3;
+ git_oid oid_tree_1;
+ git_oid oid_tree_2;
+ git_treebuilder *tb;
+
+ git_packbuilder *pb;
+ git_buf buf = GIT_BUF_INIT;
+ git_indexer *indexer;
+ git_indexer_progress stats;
+ char pack_dir_path[1024];
+
+ char sbuf[1024];
+ const char * repo_path;
+ const char * pack_name_1;
+ const char * pack_name_2;
+ git_str pack_path_1 = GIT_STR_INIT;
+ git_str pack_path_2 = GIT_STR_INIT;
+ git_odb_backend * pack_odb_backend_1;
+ git_odb_backend * pack_odb_backend_2;
+
+
+ cl_assert_in_range(0, snprintf(pack_dir_path, sizeof(pack_dir_path), "%s/objects/pack", git_repository_path(_repo)), sizeof(pack_dir_path));
+
+ /* Create tree 1 */
+
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_1, _repo, LIT_LEN("thinpack blob 1")));
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_2, _repo, LIT_LEN("thinpack blob 2")));
+
+
+ cl_git_pass(git_treebuilder_new(&tb, _repo, NULL));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob1", &oid_blob_1, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob2", &oid_blob_2, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_write(&oid_tree_1, tb));
+
+ /* Flush */
+
+ cl_git_pass(git_packbuilder_new(&pb, _repo));
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_git_pass(git_indexer_new(&indexer, pack_dir_path, 0, NULL, NULL));
+ cl_git_pass(git_indexer_append(indexer, buf.ptr, buf.size, &stats));
+ cl_git_pass(git_indexer_commit(indexer, &stats));
+ pack_name_1 = strdup(git_indexer_name(indexer));
+ cl_assert(pack_name_1);
+ git_buf_dispose(&buf);
+ git_mempack_reset(_mempack_backend);
+ git_indexer_free(indexer);
+ git_packbuilder_free(pb);
+
+ /* Create tree 2 */
+
+ cl_git_pass(git_treebuilder_clear(tb));
+ /* blob 1 won't be used, but we add it anyway to test that just "declaring" an object doesn't
+ necessarily cause its inclusion in the next thin packfile. It must only be included if new. */
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_1, _repo, LIT_LEN("thinpack blob 1")));
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_3, _repo, LIT_LEN("thinpack blob 3")));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob1", &oid_blob_1, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob3", &oid_blob_3, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_write(&oid_tree_2, tb));
+
+ /* Flush */
+
+ cl_git_pass(git_packbuilder_new(&pb, _repo));
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_git_pass(git_indexer_new(&indexer, pack_dir_path, 0, NULL, NULL));
+ cl_git_pass(git_indexer_append(indexer, buf.ptr, buf.size, &stats));
+ cl_git_pass(git_indexer_commit(indexer, &stats));
+ pack_name_2 = strdup(git_indexer_name(indexer));
+ cl_assert(pack_name_2);
+ git_buf_dispose(&buf);
+ git_mempack_reset(_mempack_backend);
+ git_indexer_free(indexer);
+ git_packbuilder_free(pb);
+ git_treebuilder_free(tb);
+
+ /* Assertions */
+
+ assert(pack_name_1);
+ assert(pack_name_2);
+
+ repo_path = git_repository_path(_repo);
+
+ snprintf(sbuf, sizeof(sbuf), "objects/pack/pack-%s.pack", pack_name_1);
+ git_str_joinpath(&pack_path_1, repo_path, sbuf);
+ snprintf(sbuf, sizeof(sbuf), "objects/pack/pack-%s.pack", pack_name_2);
+ git_str_joinpath(&pack_path_2, repo_path, sbuf);
+
+ /* If they're the same, something definitely went wrong. */
+ cl_assert(strcmp(pack_name_1, pack_name_2) != 0);
+
+ cl_git_pass(git_odb_backend_one_pack(&pack_odb_backend_1, pack_path_1.ptr));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_1));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_2));
+ cl_assert(!pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_3));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_tree_1));
+ cl_assert(!pack_odb_backend_1->exists(pack_odb_backend_1, &oid_tree_2));
+
+ cl_git_pass(git_odb_backend_one_pack(&pack_odb_backend_2, pack_path_2.ptr));
+ /* blob 1 is already in the packfile 1, so packfile 2 must not include it, in order to be _thin_. */
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_1));
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_2));
+ cl_assert(pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_3));
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_tree_1));
+ cl_assert(pack_odb_backend_2->exists(pack_odb_backend_2, &oid_tree_2));
+
+ pack_odb_backend_1->free(pack_odb_backend_1);
+ pack_odb_backend_2->free(pack_odb_backend_2);
+ free((void *)pack_name_1);
+ free((void *)pack_name_2);
+ git_str_dispose(&pack_path_1);
+ git_str_dispose(&pack_path_2);
+
+}
@@ -1,930 +0,0 @@
commit e9823c5da4fa977c46bcb97167fbdd0d70adb5ff
Author: Robert Hensing <robert@roberthensing.nl>
Date: Mon Aug 26 20:07:04 2024 +0200
Make packbuilder interruptible using progress callback
Forward errors from packbuilder->progress_cb
This allows the callback to terminate long-running operations when
the application is interrupted.
diff --git a/include/git2/pack.h b/include/git2/pack.h
index 0f6bd2ab9..bee72a6c0 100644
--- a/include/git2/pack.h
+++ b/include/git2/pack.h
@@ -247,6 +247,9 @@ typedef int GIT_CALLBACK(git_packbuilder_progress)(
* @param progress_cb Function to call with progress information during
* pack building. Be aware that this is called inline with pack building
* operations, so performance may be affected.
+ * When progress_cb returns an error, the pack building process will be
+ * aborted and the error will be returned from the invoked function.
+ * `pb` must then be freed.
* @param progress_cb_payload Payload for progress callback.
* @return 0 or an error code
*/
diff --git a/src/libgit2/pack-objects.c b/src/libgit2/pack-objects.c
index b2d80cba9..7c331c2d5 100644
--- a/src/libgit2/pack-objects.c
+++ b/src/libgit2/pack-objects.c
@@ -932,6 +932,9 @@ static int report_delta_progress(
{
int ret;
+ if (pb->failure)
+ return pb->failure;
+
if (pb->progress_cb) {
uint64_t current_time = git_time_monotonic();
uint64_t elapsed = current_time - pb->last_progress_report_time;
@@ -943,8 +946,10 @@ static int report_delta_progress(
GIT_PACKBUILDER_DELTAFICATION,
count, pb->nr_objects, pb->progress_cb_payload);
- if (ret)
+ if (ret) {
+ pb->failure = ret;
return git_error_set_after_callback(ret);
+ }
}
}
@@ -976,7 +981,10 @@ static int find_deltas(git_packbuilder *pb, git_pobject **list,
}
pb->nr_deltified += 1;
- report_delta_progress(pb, pb->nr_deltified, false);
+ if ((error = report_delta_progress(pb, pb->nr_deltified, false)) < 0) {
+ GIT_ASSERT(git_packbuilder__progress_unlock(pb) == 0);
+ goto on_error;
+ }
po = *list++;
(*list_size)--;
@@ -1124,6 +1132,10 @@ struct thread_params {
size_t depth;
size_t working;
size_t data_ready;
+
+ /* A pb->progress_cb can stop the packing process by returning an error.
+ When that happens, all threads observe the error and stop voluntarily. */
+ bool stopped;
};
static void *threaded_find_deltas(void *arg)
@@ -1133,7 +1145,12 @@ static void *threaded_find_deltas(void *arg)
while (me->remaining) {
if (find_deltas(me->pb, me->list, &me->remaining,
me->window, me->depth) < 0) {
- ; /* TODO */
+ me->stopped = true;
+ GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_lock(me->pb) == 0, NULL);
+ me->working = false;
+ git_cond_signal(&me->pb->progress_cond);
+ GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_unlock(me->pb) == 0, NULL);
+ return NULL;
}
GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_lock(me->pb) == 0, NULL);
@@ -1175,8 +1192,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
pb->nr_threads = git__online_cpus();
if (pb->nr_threads <= 1) {
- find_deltas(pb, list, &list_size, window, depth);
- return 0;
+ return find_deltas(pb, list, &list_size, window, depth);
}
p = git__mallocarray(pb->nr_threads, sizeof(*p));
@@ -1195,6 +1211,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
p[i].depth = depth;
p[i].working = 1;
p[i].data_ready = 0;
+ p[i].stopped = 0;
/* try to split chunks on "path" boundaries */
while (sub_size && sub_size < list_size &&
@@ -1262,7 +1279,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
(!victim || victim->remaining < p[i].remaining))
victim = &p[i];
- if (victim) {
+ if (victim && !target->stopped) {
sub_size = victim->remaining / 2;
list = victim->list + victim->list_size - sub_size;
while (sub_size && list[0]->hash &&
@@ -1286,7 +1303,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
}
target->list_size = sub_size;
target->remaining = sub_size;
- target->working = 1;
+ target->working = 1; /* even when target->stopped, so that we don't process this thread again */
GIT_ASSERT(git_packbuilder__progress_unlock(pb) == 0);
if (git_mutex_lock(&target->mutex)) {
@@ -1299,7 +1316,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
git_cond_signal(&target->cond);
git_mutex_unlock(&target->mutex);
- if (!sub_size) {
+ if (target->stopped || !sub_size) {
git_thread_join(&target->thread, NULL);
git_cond_free(&target->cond);
git_mutex_free(&target->mutex);
@@ -1308,7 +1325,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
}
git__free(p);
- return 0;
+ return pb->failure;
}
#else
@@ -1319,6 +1336,7 @@ int git_packbuilder__prepare(git_packbuilder *pb)
{
git_pobject **delta_list;
size_t i, n = 0;
+ int error;
if (pb->nr_objects == 0 || pb->done)
return 0; /* nothing to do */
@@ -1327,8 +1345,10 @@ int git_packbuilder__prepare(git_packbuilder *pb)
* Although we do not report progress during deltafication, we
* at least report that we are in the deltafication stage
*/
- if (pb->progress_cb)
- pb->progress_cb(GIT_PACKBUILDER_DELTAFICATION, 0, pb->nr_objects, pb->progress_cb_payload);
+ if (pb->progress_cb) {
+ if ((error = pb->progress_cb(GIT_PACKBUILDER_DELTAFICATION, 0, pb->nr_objects, pb->progress_cb_payload)) < 0)
+ return git_error_set_after_callback(error);
+ }
delta_list = git__mallocarray(pb->nr_objects, sizeof(*delta_list));
GIT_ERROR_CHECK_ALLOC(delta_list);
@@ -1345,31 +1365,33 @@ int git_packbuilder__prepare(git_packbuilder *pb)
if (n > 1) {
git__tsort((void **)delta_list, n, type_size_sort);
- if (ll_find_deltas(pb, delta_list, n,
+ if ((error = ll_find_deltas(pb, delta_list, n,
GIT_PACK_WINDOW + 1,
- GIT_PACK_DEPTH) < 0) {
+ GIT_PACK_DEPTH)) < 0) {
git__free(delta_list);
- return -1;
+ return error;
}
}
- report_delta_progress(pb, pb->nr_objects, true);
+ error = report_delta_progress(pb, pb->nr_objects, true);
pb->done = true;
git__free(delta_list);
- return 0;
+ return error;
}
-#define PREPARE_PACK if (git_packbuilder__prepare(pb) < 0) { return -1; }
+#define PREPARE_PACK error = git_packbuilder__prepare(pb); if (error < 0) { return error; }
int git_packbuilder_foreach(git_packbuilder *pb, int (*cb)(void *buf, size_t size, void *payload), void *payload)
{
+ int error;
PREPARE_PACK;
return write_pack(pb, cb, payload);
}
int git_packbuilder__write_buf(git_str *buf, git_packbuilder *pb)
{
+ int error;
PREPARE_PACK;
return write_pack(pb, &write_pack_buf, buf);
diff --git a/src/libgit2/pack-objects.h b/src/libgit2/pack-objects.h
index bbc8b9430..380a28ebe 100644
--- a/src/libgit2/pack-objects.h
+++ b/src/libgit2/pack-objects.h
@@ -100,6 +100,10 @@ struct git_packbuilder {
uint64_t last_progress_report_time;
bool done;
+
+ /* A non-zero error code in failure causes all threads to shut themselves
+ down. Some functions will return this error code. */
+ volatile int failure;
};
int git_packbuilder__write_buf(git_str *buf, git_packbuilder *pb);
diff --git a/tests/libgit2/pack/cancel.c b/tests/libgit2/pack/cancel.c
new file mode 100644
index 000000000..a0aa9716a
--- /dev/null
+++ b/tests/libgit2/pack/cancel.c
@@ -0,0 +1,240 @@
+#include "clar_libgit2.h"
+#include "futils.h"
+#include "pack.h"
+#include "hash.h"
+#include "iterator.h"
+#include "vector.h"
+#include "posix.h"
+#include "hash.h"
+#include "pack-objects.h"
+
+static git_repository *_repo;
+static git_revwalk *_revwalker;
+static git_packbuilder *_packbuilder;
+static git_indexer *_indexer;
+static git_vector _commits;
+static int _commits_is_initialized;
+static git_indexer_progress _stats;
+
+extern bool git_disable_pack_keep_file_checks;
+
+static void pack_packbuilder_init(const char *sandbox) {
+ _repo = cl_git_sandbox_init(sandbox);
+ /* cl_git_pass(p_chdir(sandbox)); */
+ cl_git_pass(git_revwalk_new(&_revwalker, _repo));
+ cl_git_pass(git_packbuilder_new(&_packbuilder, _repo));
+ cl_git_pass(git_vector_init(&_commits, 0, NULL));
+ _commits_is_initialized = 1;
+ memset(&_stats, 0, sizeof(_stats));
+ p_fsync__cnt = 0;
+}
+
+void test_pack_cancel__initialize(void)
+{
+ pack_packbuilder_init("small.git");
+}
+
+void test_pack_cancel__cleanup(void)
+{
+ git_oid *o;
+ unsigned int i;
+
+ cl_git_pass(git_libgit2_opts(GIT_OPT_ENABLE_FSYNC_GITDIR, 0));
+ cl_git_pass(git_libgit2_opts(GIT_OPT_DISABLE_PACK_KEEP_FILE_CHECKS, false));
+
+ if (_commits_is_initialized) {
+ _commits_is_initialized = 0;
+ git_vector_foreach(&_commits, i, o) {
+ git__free(o);
+ }
+ git_vector_free(&_commits);
+ }
+
+ git_packbuilder_free(_packbuilder);
+ _packbuilder = NULL;
+
+ git_revwalk_free(_revwalker);
+ _revwalker = NULL;
+
+ git_indexer_free(_indexer);
+ _indexer = NULL;
+
+ /* cl_git_pass(p_chdir("..")); */
+ cl_git_sandbox_cleanup();
+ _repo = NULL;
+}
+
+static int seed_packbuilder(void)
+{
+ int error;
+ git_oid oid, *o;
+ unsigned int i;
+
+ git_revwalk_sorting(_revwalker, GIT_SORT_TIME);
+ cl_git_pass(git_revwalk_push_ref(_revwalker, "HEAD"));
+
+ while (git_revwalk_next(&oid, _revwalker) == 0) {
+ o = git__malloc(sizeof(git_oid));
+ cl_assert(o != NULL);
+ git_oid_cpy(o, &oid);
+ cl_git_pass(git_vector_insert(&_commits, o));
+ }
+
+ git_vector_foreach(&_commits, i, o) {
+ if((error = git_packbuilder_insert(_packbuilder, o, NULL)) < 0)
+ return error;
+ }
+
+ git_vector_foreach(&_commits, i, o) {
+ git_object *obj;
+ cl_git_pass(git_object_lookup(&obj, _repo, o, GIT_OBJECT_COMMIT));
+ error = git_packbuilder_insert_tree(_packbuilder,
+ git_commit_tree_id((git_commit *)obj));
+ git_object_free(obj);
+ if (error < 0)
+ return error;
+ }
+
+ return 0;
+}
+
+static int fail_stage;
+
+static int packbuilder_cancel_after_n_calls_cb(int stage, uint32_t current, uint32_t total, void *payload)
+{
+
+ /* Force the callback to run again on the next opportunity regardless
+ of how fast we're running. */
+ _packbuilder->last_progress_report_time = 0;
+
+ if (stage == fail_stage) {
+ int *calls = (int *)payload;
+ int n = *calls;
+ /* Always decrement, including past zero. This way the error is only
+ triggered once, making sure it is picked up immediately. */
+ --*calls;
+ if (n == 0)
+ return GIT_EUSER;
+ }
+
+ return 0;
+}
+
+static void test_cancel(int n)
+{
+
+ int calls_remaining = n;
+ int err;
+ git_buf buf = GIT_BUF_INIT;
+
+ /* Switch to a small repository, so that `packbuilder_cancel_after_n_calls_cb`
+ can hack the time to call the callback on every opportunity. */
+
+ cl_git_pass(git_packbuilder_set_callbacks(_packbuilder, &packbuilder_cancel_after_n_calls_cb, &calls_remaining));
+ err = seed_packbuilder();
+ if (!err)
+ err = git_packbuilder_write_buf(&buf, _packbuilder);
+
+ cl_assert_equal_i(GIT_EUSER, err);
+}
+void test_pack_cancel__cancel_after_add_0(void)
+{
+ fail_stage = GIT_PACKBUILDER_ADDING_OBJECTS;
+ test_cancel(0);
+}
+
+void test_pack_cancel__cancel_after_add_1(void)
+{
+ cl_skip();
+ fail_stage = GIT_PACKBUILDER_ADDING_OBJECTS;
+ test_cancel(1);
+}
+
+void test_pack_cancel__cancel_after_delta_0(void)
+{
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(0);
+}
+
+void test_pack_cancel__cancel_after_delta_1(void)
+{
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(1);
+}
+
+void test_pack_cancel__cancel_after_delta_0_threaded(void)
+{
+#ifdef GIT_THREADS
+ git_packbuilder_set_threads(_packbuilder, 8);
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(0);
+#else
+ cl_skip();
+#endif
+}
+
+void test_pack_cancel__cancel_after_delta_1_threaded(void)
+{
+#ifdef GIT_THREADS
+ git_packbuilder_set_threads(_packbuilder, 8);
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(1);
+#else
+ cl_skip();
+#endif
+}
+
+static int foreach_cb(void *buf, size_t len, void *payload)
+{
+ git_indexer *idx = (git_indexer *) payload;
+ cl_git_pass(git_indexer_append(idx, buf, len, &_stats));
+ return 0;
+}
+
+void test_pack_cancel__foreach(void)
+{
+ git_indexer *idx;
+
+ seed_packbuilder();
+
+#ifdef GIT_EXPERIMENTAL_SHA256
+ cl_git_pass(git_indexer_new(&idx, ".", GIT_OID_SHA1, NULL));
+#else
+ cl_git_pass(git_indexer_new(&idx, ".", 0, NULL, NULL));
+#endif
+
+ cl_git_pass(git_packbuilder_foreach(_packbuilder, foreach_cb, idx));
+ cl_git_pass(git_indexer_commit(idx, &_stats));
+ git_indexer_free(idx);
+}
+
+static int foreach_cancel_cb(void *buf, size_t len, void *payload)
+{
+ git_indexer *idx = (git_indexer *)payload;
+ cl_git_pass(git_indexer_append(idx, buf, len, &_stats));
+ return (_stats.total_objects > 2) ? -1111 : 0;
+}
+
+void test_pack_cancel__foreach_with_cancel(void)
+{
+ git_indexer *idx;
+
+ seed_packbuilder();
+
+#ifdef GIT_EXPERIMENTAL_SHA256
+ cl_git_pass(git_indexer_new(&idx, ".", GIT_OID_SHA1, NULL));
+#else
+ cl_git_pass(git_indexer_new(&idx, ".", 0, NULL, NULL));
+#endif
+
+ cl_git_fail_with(
+ git_packbuilder_foreach(_packbuilder, foreach_cancel_cb, idx), -1111);
+ git_indexer_free(idx);
+}
+
+void test_pack_cancel__keep_file_check(void)
+{
+ assert(!git_disable_pack_keep_file_checks);
+ cl_git_pass(git_libgit2_opts(GIT_OPT_DISABLE_PACK_KEEP_FILE_CHECKS, true));
+ assert(git_disable_pack_keep_file_checks);
+}
diff --git a/tests/resources/small.git/HEAD b/tests/resources/small.git/HEAD
new file mode 100644
index 0000000000000000000000000000000000000000..cb089cd89a7d7686d284d8761201649346b5aa1c
GIT binary patch
literal 23
ecmXR)O|w!cN=+-)&qz&7Db~+TEG|hc;sO9;xClW2
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/config b/tests/resources/small.git/config
new file mode 100644
index 0000000000000000000000000000000000000000..07d359d07cf1ed0c0074fdad71ffff5942f0adfa
GIT binary patch
literal 66
zcmaz}&M!)h<>D+#Eyyp<EXgmbOv^9IO)M!(Eh^5;&r`5fFyP`$%gjm5%}+@M@=A(I
MQ@J>k5{uv*03B5png9R*
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/description b/tests/resources/small.git/description
new file mode 100644
index 0000000000000000000000000000000000000000..498b267a8c7812490d6479839c5577eaaec79d62
GIT binary patch
literal 73
zcmWH|%S+5nO;IRHEyyp<EXgmbv{pz>$t+PQ$;d2LNXyJgRZve!Elw`VEGWs$&r??@
Q$yWgB0LrH#Y0~2Y0PnOK(EtDd
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/applypatch-msg.sample b/tests/resources/small.git/hooks/applypatch-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..dcbf8167fa503f96ff6a39c68409007eadc9b1f3
GIT binary patch
literal 535
zcmY+AX;Q;542A#a6e8^~FyI8r&I~hf2QJ{GO6(?HuvEG*+#R{4EI%zhfA8r{j%sh$
zHE~E-UtQd8{bq4@*S%jq3@bmxwQDXGv#o!N`o3AHMw3xD)hy0#>&E&zzl%vRffo<B
z)-H|+CWHZ~O*S%cfYx9;02_ohIA<Bg(1SxF-6OCb&_lBkf{t<AM9r;%E(Hf#h{|a@
z9>mqo=v6>_2NRa#TwDdYvTVQyueO*15Nlo%=#DXgC0bhF3vTa`LQGaO9;jeD$OP?~
za$G4Q{z+Q_{5V?5h;a-noM$P{<>Q~j4o7u%#P6^o^16{y*jU=-K8GYD_dUtdj4FSx
zSC0C!DvAnv%S!4d<Yg@O<;m`;oSw)=Fz+hrL<mY{rBr8j4pi^88FX3}jKrYUP)>gk
XB^)11aoGMJPCqWs%IS0YSv(eBT&%T6
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/commit-msg.sample b/tests/resources/small.git/hooks/commit-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..f3780f92349638ebe32f6baf24c7c3027675d7c9
GIT binary patch
literal 953
zcmaJ<TW`}a6n<`g#Ya}rZCaYOzy?UGG&Tf#VG`?}7@eHtB=MTqneFUS%75oLS;atz
zm&iUo=ey->y@-{3h^^Cx;#d0zEA@DDc$nY4ez&|=%jTg@_HU*ub=!!y$xW09TSjlj
z(`I@QCsM`!9&80$I98wsQ8yK#)Orb<8re8FjkKh630D$QUDwi~(gkX=RunYm$rDjk
zlp%RUSnzA#6yjdG5?T?2DcYKp+v_lts0ljn&bh3J0bD5@N@1UKZ190O6ZeWr-BuZ^
zWRebCX%(%=Xoj#(xYk1Cjtr!=tyBesf@m6}8zY6Ijbz9i9ziI_jG9Mv<Cz(ymp*>R
zDH*e>^ga9IR?2wrSrAVm;eButj4<aWB@zzNl|1Wp@4;}1O?MUF>Y>7(E2?b~jsu>&
zRKCJ7bp#19sqYh627wD%D9R$8=Ml$TNlumDypl~$jBu*G>5fIR^FB0h0Ex&TGZNr>
zL5hs1_K>taRb!|ThN9ns7^@4MXKP+6aGI_UK)T-M#rcP$;kN(Vcf#P)+5GzWa{l@J
z>-E{`$1iiNVYxq27}<DnwLRXQUG0o_hw&da-s5T#H=`Y9D_8=eTZ?cpWatp#a1vs@
z2BjrO)z@aTuI#g#`)oJcnhM7oYLT@~CHX@CNXv4>j;uo%;)r3kJI2xCFF~Ux;$Q%)
wjbk6JlDCM`jU&P+UVOvg`|iYl<7~9k>HHB4I;pdlQ=I-^$DrHaN$@lH1?P!0U;qFB
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/fsmonitor-watchman.sample b/tests/resources/small.git/hooks/fsmonitor-watchman.sample
new file mode 100755
index 0000000000000000000000000000000000000000..41184ebc318c159f51cd1ebe2290559805df89d8
GIT binary patch
literal 4777
zcmbtYYi}F368$Xwipg4lq(BeHMvzvH-4;n7DGJBPqq#tw3aed8+IU5-m)yvL>;Cqh
z8FFRGj$`9CA8ao<GaSz2oMCnz4Rv-gw9b@j_$0GQT1?XYi|;S??Y`Z6`Z;}C6#27N
z8ShS>J?j^$%==FV``-=rhLcPW`McSytRm~mEO7_&_cAVZrf1fFy*ha@8oe%*-aBYE
zcjzZg>LOkgxuUr-XJnHyD;zmPnRaSc#!k_P*d_BttRdc+J6G7za5#+<HG#rlmbrN~
z8DwU-3}VABEwM=0VLP@^Dy6ERR5_J6cmg|GEh*M1EliqCGwe^ZT-iZ$2Yc`4!I#WZ
z5nGGhn7*jeW=2ydsmfAmm#=8AD<<;TI+#z{Q)kW;yE!%GB6f~7EtEMLdM47Qaz*8=
zIObA(VVj-VG{Ax|66d*hi`+bRG>^Y1nkc2Oowk`ya47uUR3Feu?B<phm31&PQB<lt
zb{W(W4wf#Bab%|Q_tKPS?3^o=5)Z8^Vh(#slNI}pO(f^|{U0GZhLnycSaNd&h?CaC
z0XklU6^<ky6rES9T=na$M8P<_)aKMAMo+UDewAu4wF{#&6diFshiudixAoh|&0<BJ
zR>(w;S{(VYzxh}q-=#zP@uxSx{wbyPUMFU;K(06)$o{07&3yI?q{GqMcQ1c_^M<0<
zF4acAV)Il-V(rCTC1(;bsZ*}bl8dmejAk~yb`B}!^0;g^(o9kGUfZfDOvyp@x4OQt
zSgWh6T|3eq;9MFs8-#z+FDM1h(IjRUP|``PxupgJ7CUHOH90gbgl^2~97`?_X{P))
zB*$r1cDlF-%azKND}?Gv`2K8-9v5e`gQoft=j?T<&a13c^!wY_$D`5z-X1g?ty&6-
zQN50{8?bUk9AI->^W@~~nkOghHIC2YN+<JiT_ob7ttND1oh`HH28Y+OV~HedG&uB`
zy}rA*r_xT#bR`Q7-*)3t*(!Hf?jKzyxk=8hdi3L^d<p<uU9q_<4k&xEr4@YWv_vsW
zp(#32bYtA5u|s#1+}h`0kwpz4kUD&+>AXkLQG_2-{Pq3%{`3KUMeG$iIn%%^6*NYb
zn|_BdV#C)n4565Vcc<EWC-nglECZGy!j9I4&;hUCzr(?6TftH=0^@!mI^E@y5HZw8
ztH&kaSNyg=O6riqR^MPOX6oD__Jz@K;*icS)?m$!p{S$n;*EwlO<L!d7;utu(W9D!
zaF!B~xV^2i?wH0s?Lw%Q)(`aPkajs1XojlPv@Y-d5#YFg#KG+!r7A(dJLnkiJMs`p
zV|_=d!upN{TsxH1?sZNdzxeHsmtzTV`1{pykJ_~+^*>X;uT8&z3vSi!HXGbUj2B!R
zdz~&#<?<tHJql=W&((YpzS06y-Z6Cn^H!*9qH?pOrD~(GV=JJ~z{tpLnGK|xp&C1`
zsbn7O86xjF<~G*S1l*;;Bh%6><Up=oKy99?62P^?M&22U6QFRBXLb&u%=Ur<74wRy
zMRG!f{AvZ>fk#L-&k$fLwo$4?>12g@AXOKFekuo#6EHB%gmpD?1eyh%N8s{2wGoTu
z*@6cEZ^ZW!FAF_|JL`NkV7k}0ow|-2jHwbgH0;c@Dq*o?@&c*HnGdyx6^su8Qk%2{
z*ye(dxO*6-&>qn1+zw}tc6;=sOX{4WB=VqjTS^))y1jlX2Q;=e!qMmFA5lC$#;BxC
z=Y%tRpWxb+_uQAvAw7Q{HGV#R$xb&udLCzZ+HN?kTyB};1EJ8UlQ5!>5eGW@)RX0n
zkjj>EF!3=0Gl^8dzv$B^NMGRxJoqN4A`xq-@wCbrx*u2NmIJ1<fUDQ=*^)h6`vzco
z3F+ro$F!F6pc<B;<;isobIgbVGKUBByoQ4#CO({S7g?<Dh0^!7uJ3gxS=6F;+^gQc
zeKi4`4`Fm3p|BU2v{M|-u!#QGCXiM=k=%np0<ZOPQqEjP_nneyOdgEuG9IH&YYPtp
zKB_dvcYCcyhyT#<uhUEL$o~!TUz;cb&|`uSM{Dkv%&W2lcpYL&kv)tUvVv?&>xZ%H
zh;{|4T3(!E9sY#Ni(wUJYs1MmIc9bl)(4Nl3_wD_BWB>i<1S(LX7m*{Q7PU$muMS*
zM!%0EZx-Vw=Zey;erC?SNxF;pY@^A%-krqzfLV2meBp1vWdyArFYn`DD19T)Hw(?n
z)}{NP(Lk(o*?gl#B@pP7^*r|=;PIDT4|F#{2Hzh-AL0Rv$6uT;<CP7qxbYms@WU7}
z%}TsHJ!V_56ZFF{BfI=8jTBxMEATG376pS6a;u1@c?{~sL<N52`U6fuLkz4P@Mb^b
z2`Q48y&!C0&A+WRCJAdmo3u2#?eI=si9Vm47$|`m**x<wKkM=QR&g?C63@P5X@xP8
zi5QG2b-Fdz%S0%1{kKvL%^RTdpK|SU^VJ7JCmKdwp1`;HRoGM7ef^k_s_}2An=cql
za|{IaxQWpdq<ae&x3BOJq+M5QNIk#C{Nv@u>n|WzE4=slK?on@(fZeGhRgQCu56qB
z{+n81Az96qnQjMY*-*r-KV*7;Z#4Q<xfjbcBx_6JAN-HP@bq+eI%HhAB9&vLyOap{
bw<Ywj(b#kdcCk7dCBY;|DBNpPjIa1F6*dXN
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/post-update.sample b/tests/resources/small.git/hooks/post-update.sample
new file mode 100755
index 0000000000000000000000000000000000000000..9f95e41a39cf974958277c8881ac6cce405ebb20
GIT binary patch
literal 246
zcmXZVO?HDY3<Ti4Poaiwjq^yGw9DKf7mz^&Ly=V5q$H=W^Rt|p_r9s#9Ea7VERo!9
zyT9>uJRJJV$M^KdldiMhj?ImK6~FvwJ*L5a){QoM=L5TYHkGO1$UrO3`a>{?Opw|b
zG(#59NQ#jFL9v~vgOVkM@^^(^A}onOE))yWEwhIlk&{ZyseZ^O0b=w8&O=BK{k<5B
k^Q-B@eG}LeHrquz%(SVEp_N)VhYZikCW__82JXfD17`J9Qvd(}
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-applypatch.sample b/tests/resources/small.git/hooks/pre-applypatch.sample
new file mode 100755
index 0000000000000000000000000000000000000000..625837e25f91421b8809a097f4a3103dd387ef31
GIT binary patch
literal 481
zcmY+ATTa6;5Jms9iouO45IBJXEg&Jm9@v1LPHMM_ZR|;#6tQ<EeSrA%_2`_rGr1_8
z?aM?yVtIc%-@9SGSk&8x=grP-Lf`7!^=$7xgL=|ysZ}!av6zL~ywui}<2##V6L@!k
zy=p^)V7%Wzs-g`9<Y9}^)&uN}BCrXR_T3@Z2$gSJON2`X=mAs+%@7n-2I}ZrP|TFA
zvJJGDl3HPLP<@!Q!}zXQvey#qEE#a#$vs97i4=A0stF@YQ)k_ZajaoS^dVYBc&37_
zVI(L=X<V335r9~7T<;|HfKF+yM}}LB9d96V)Si;sj(;9Rh$#P>h$71hSXq*MxP;V&
zj0cY7SCL=x4`a46sF)C>94Gk%=3q$W2s;j6iHtB2$R0%gix4oK@&T~=ALd_o*CKxt
I-`Pv{1Bpzc>;M1&
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-commit.sample b/tests/resources/small.git/hooks/pre-commit.sample
new file mode 100755
index 0000000000000000000000000000000000000000..10b39b2e26981b8f87ea424e735ef87359066dbb
GIT binary patch
literal 1706
zcmZuxU2ohr5PY_N#pZ0-F<{-<Zkx1j&cMNSQ3FN`GzR*R1O+9oPV`BnOj7q@1pV!u
zrF0Hl^i33(yR)-1d-!H%&2|=|^E~_R{N1zNJ-&Zmt-t?iwXv&i+ZN}Km(TX8Q$H4u
zd7(m`|1iDmF5k@xV`p;C4zojASmLc}yN0QDZbhN=ri&CEt=XGuN1IwjGJ#a#`t-kG
zDqY)}7+Ft|;YKwLYbtg$S(-TBO=x3cP1cd}%f4kB!<6Wu-dCwz-)KDMEuM^_Hh*UC
zC`1)|)T<(U6b`+yOH!6p*Ll}@qastwA*dyjsgOf5C=?LTprfORG6O{5L%@S0wyHpj
zu|_A-=NWnYYR5m7kvm6|&T~GzoJ_OKR3sgFUgw?ifho^NQhvK#{6g0=&Fh)%n}#m0
zk1sNmwb_AMDq};OOGw5|;OyX#?yQMMH6yAk(x$3tjFjHE?c$E2XC_xXav8tnIeIG?
zYMI|~MLEVGkuT*>v&v-X^RA+u>k}E$4d&uD7=g_fA8+pNNV=4s0|iD3p<=DTXClTS
zXV23tJ;ECmN@M0j@zUAKEYW@3bv!SeYZ8ZH`YQNTApFVNc;F|9r5p4TqGs=>8E?6y
zi|gY{iM#PG1nL?UE9YCnWTk72kgZPG*Usqw!~Qd3c?~@w2?%eg@~)+VlSs6N5Yf2^
zz;ow<fjf3n`imj7u5lnzt||q9QFM(H@<3EJCK|l5P!$yDfn~U-(5Vu7s+GqxNKyeI
zP=-Oa(KH&gK`NhUa`cLj3B8%qL};DR7dk!`A^h&3-hFB6p($5Ufy^tGir)3et}qK4
zpkPKYWzC+?=&gw-L6S)S=<lfq)%uLUAa%~81Jf9hk)w~U!DItnoSy`m^}#38M}K-o
z!PqisQkV!&z4E*V01ro~qlVK^0BI`pLk6D+)f~*y!hCvwHq8zY9BGh<2s$@b^A<8G
zRaqk}&qZyyv&|0QDFPA%h4TgF&vdlc|JUq*=>F#K#r^&KMq1A`oqVGFpD&-!Pv|Rc
zO3KSqA@h9nSc%bm`0)Amk6*J}@14J*1-219l%%7D!Pl}UK>|lVi0Dfgu2jN3WC!uL
z0ej??b2iSehVgdnWHmZV4kUo*QL#aiIp}U=9x)IXk}JJ7VQ;CI9Rtn5e0VcjbY<bp
zv{}WlG6L;H!EzFKdY>cVt+`x5D+svCGD<sXw4|)E|JX43I1_3P(sI4{wj87bPSrNG
w!SIr>;Z5hm*<gY+X;)Ryx4=nzaab9m`bwE*^s(%u*E3HbUuOR@+&s_z1=MCi2LJ#7
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-merge-commit.sample b/tests/resources/small.git/hooks/pre-merge-commit.sample
new file mode 100755
index 0000000000000000000000000000000000000000..d8e8ad4bcf8337f23c78a98537ecab4831301202
GIT binary patch
literal 473
zcmaJ-O;f@!5WV+TJd4B0whSt$H%Dh2t`2u6q1!glCNbGU;n%yxdNsG~zR#WA6xIwy
zWEZHoU#u?nykD=Y<HPgeWDkDm^kTof*l(|%^gh!nHrZpo^vhMDjV;E1GD~K7wV*+D
zz9lry9T0cHcm_KhDVXYvQ==FrLTT4u=bEr{U1yl7%thf%wJnv<XQZ`ZbQEezaWdS%
zI;c?h9a)Y!ux<WK8rQd_aA^?61hv_Pf<t7*z1USuL40FxYz<|hybsO?qnN}aMpcuf
z6phFw1%Xx=wUk(m>E$jSEQZ%SQ(}oLgslTvrKK@9Qf#b!hajVFnp9@oIix;NcI9Wk
xjnh0ya!AWet{I7YpD;y6HXyzI*lfSvH=o6*7mJZPkuaYpm>vzZ`wyGEBtOQPo|pgt
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-push.sample b/tests/resources/small.git/hooks/pre-push.sample
new file mode 100755
index 0000000000000000000000000000000000000000..02cbd80c287f959fe33975bb66c56293e3f5b396
GIT binary patch
literal 1431
zcmaJ>U60!~5PUX&#a1@z9B{IIZkjLT0t5kq9#8~D(I5{+8&J~9;#ndUk~-ZT`r|uG
z$#K$$J{TsK<m~Lsu9iP+t-0TZ=sa(K+C6);54X>s*LP1}9!GoZ@4I4myMMG_di|of
z%?llx{O8TS-#^<H#%^V=)RNv>;(OioEmPy%kwWQBA1OMzV{hsQ8XFzS1k!~YQoLa5
zhtP1fA$q6VmMbbAC_9)4I628k*O5J$NR19uHe4QYDK<==I~SQk)Nu%xQ~<Hy8U>KH
z53w=!ke(FGb_PpnZfd*+hnXDTn;2*`u^~;?+5C~cn?bRka7NR%06%e6O91{MAgN6J
zmlO8{Biw4&wr&&(z4p3eln`E}XR9m9bNYZ7Ibrg(4yZIXrfgD7N*AFD7L3YSM#j}%
zo__rOS5fr;@8UM<6cl+cv_$YB$PQ&9dv($eM*))g!_cu!QcSh-mqE9i#QDZT)=o#`
z?8!RtE?w6p?GkGZ-6yt_p~5~4ecu|Sf^)6096%h*q-eNiEA1;Xwg)p~Q&iGSG7-IQ
z9aII&`ps$WOojFA`*bjG<mBv1n0hcYZWN0~(X01-hx(ExqWqaXgSk*@-GMp|K_3!5
z9|O21N3%~izh(4fbp9wzd+!b&7cVwSP5H00)m5ej-(s=Pl#(90UOhn@OA9u+D{i@r
za4*CP0I#<d-)-#xq5q-iY5nIef2s5OuQjcA>kFk|E@sHHuD}W^d`7YJ3YE^zrQnqR
zGoq?;YGKe)93o|_=^f%3U1KYZGPOXRRxK7w`UUbMMa3<86OmVH!EKP$8RCrn9mWX+
zC?9yF!fRVLmud3hF<}x;;sR}f(*r}6Gap3fR6zLHR~kbMgD{98N`L+r&?3p~*0+FX
zcAL%j=(SO}xTJUTvA`&Lf`2mv4koPG9&|<CA~EHbWHMoFPwT(&U=7t0`RoFZPO9Kq
zwwe$i=T|AXY#hD$aZlNMH`wZ%gwilGJ(zeYpOn*F3cy0XKXio^Sj#WXx=PWV`WGaA
B&~*R+
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-rebase.sample b/tests/resources/small.git/hooks/pre-rebase.sample
new file mode 100755
index 0000000000000000000000000000000000000000..193ee3f127387f469894675f0d621b5bb9c275da
GIT binary patch
literal 5006
zcmb7IX>;2+68$XxiXKL@ma;l5d2^5Ba_rPh_DHI-u1#&_upttZXp;no03$20|NFiM
zK#D#xQ>!Z3JkX8T-LDVm!B5j7y_{;JDmmTTef+K1oIiPzeEr+Ai*<2PUgnG4^ZB>p
z_fkAvoR1emuf~ri^K$-px=4#D-v<wZ2Xv&$O_eTJh6d4)=DWL(NBs9G{k<+yMMw0T
z$VH*-+LM)}u&m^`l8~1nt(3Z;R8v(KbY5#i3z+~8h0D}Xvq&3J8BMWDizPNpaeb~9
zBN9bSkthfXzskapf%Zt{*e#}{QaNiaAVZ4{$;;I6<vKNhO@%7P-(;l-x=pPoExHC!
zB(hA#cDdD?s4P=!)=-K{<kHAWKetl-8I8wwO<ihJNs-$dEvr;&S_@6E=mNSJ5;mg#
zyb)MbqKH<one{qrV;ZQ6WL}yLtyi*ekNLf|uC6M!)Cmq7*l?g0d6`MlE49|}>Y9w&
z`bCv#<YfTKtb`!}CyNYd;|(C?vRVQmWOfR9X?FZ#=f$No)^#4>2zVn=YnJyeNey(Y
zRh`9vtLw~A+5zsjp|W0Nsa|29Rm!B>OoG5a+vi;ari8O>KkU!KAWg_fa3btK2x*_@
z0bEc7J;Ubghm}n9bOi(Sv_B66nQ7U)J7f0fO}<cB8i8vG{r39s_>8Wuf*uorcIgEG
zOHc|-V6+HlRhOP}?Cn?@5iwSl43abmBA^2lyL$+cpabCGVES+v^j^FO_}?FIp<qP?
z#_?*YMHIkyZxJxS;h@A)WDJ0bN&+F-#_lFjAf^g_Pb#5YXqYe|dZUpa^zI)VOBXQQ
zAMhT>%En%Ll?Z*7*}TwrZyg5OSZ9rY-`aU~Mc-jjv{Ll)FLMgtB4ujktfQ`Xhqrka
zT=P!A;9w^;Z?PqpLwOLu=cj3L>TdUKw2;DMu)`oVkj}<z_EjO_2uWYuvKG==%Zu?h
zJiMVR^c30RbpW}<+z;jjoNC}YI4f6QC7d-08*4mCB1>#bcDx4tYg=j%D`+i{W~fVM
zVmZ>W9VMyin9c-0KzI_;iZ-g|OyzuG`Yq%(%dvl;ifnVr0;jWE&S`z|rQu=!yHBBO
zx`OJ;oOQ(KKM<$(bC38o>pD0%|HA(E0TRw7qj$fJ_pRN+7Nm>dS<q{AcOz#-;d7_2
zL$z(_nhH{vOzT(}>C(gLg{(`t+5Z=?o+}wXU4tHy+&%F&aRhFebeEhR2R5|<c6J);
zEY(q5F5<n*XP0|=PtPBn$B)V~d$Os-?&8UlaVe_|jdkzoWNsTP-_uyq4$R6o<h`&@
z{loXa{^#TF=NJBYu9qB?hs}x=It_O}ZjX(_wy9@X(lmkRpNi0{8T{O_b_j*JC^_SM
zz3G?1$KCNWF-|`Jbx2cQ-y5LW?Z2eikngTZmsx5C(@({8<l)Ue+gIp##Mosfa~iZN
zZ|NLN9uE6Xaqpwk+@D+f?$tg2JRCY`pwZwbR9OvEn*zYm`ffKIzl4{r{ZgjfpbuX)
z_r0=0y3)T-j$gljPyEJO*6Y<pj7G72aLoqaTpc=#u)*xJcVUm0;k(n;r)^DQNa6Oj
zWvlHEGg~lz`Q_8`yQ9<BZ;ylE1Z}bD<8}<uB9Y5lRB=;JUD0h?_)4H&EhJjvwzJx?
zr<o_vk&75j_5^d$8Z$_Oc1=R-I_DlNZ2@~81oV*J6%o3RuiCz}^VEW>$#Ycbp^w@t
zTl%=f1t=w+WpJzF<|CE@?SCNAz)%9?w33lQ8vrHJqPfH9@}qs*QXOG71W=ylx;wOB
zcx!Bj^)Yy6WX$a^vBkBJ5Cob<oubBW+a#9bX{0bkMTX_2sIrs`HMk@$q{dK5h2-g}
z({`~#gm#G?c#>qlaDx_B0c<3b+8)f84LCrt;e;qxc+7>VbwVK{skNv!wvBiTa^9Iu
zkwP;VK)jH$WJ{`MRwAA9fal!y0dtV;FWg8PTkWU>CwnqD>1ZX2B@;$DlX%C5MI+}{
z9xQVnffR*~v2KAUj*hCd<gRCjR7~6Q(vF%VT7j97iv3j4Z0p%mU`7t061~l7b$!#t
z3*Pveii}aQ?QD9aiX>gul~`bk#mk`o>zk9)<2Uc8?hUZAEvd!`9em)~$Z)zev>w^8
zyAgCP_$&Y)7HSQ84`xG}OeTavaEswwF|8Xpi5iZzZa@hCiv(J-%bfFC&)HLlO+Rhw
zG6g?9eL5&A!SuJnQ6}LxG%tU+@vZ`i+!+Rz6iYvsTdhnPo7lW{m-}{hya@viX4)XZ
zngaw+j;gloB#|UwI@8sOmQpc`h+bicQJnQIB5eifIMQNgD2+oai33m!34~xU|0Azj
zhu$8z+T5^;Pxx@d{N)pzOJLSa^e;aDf$W%N5XcOf!mGC9l9j$Ev2h6N+6ZQC+CJzl
zaM7?S!SrFLS2DASjj(h6y1WN3N?|bmqmyzm!&nLoE|`rKBOc_yDF$a#FsUn!IQf(t
zdC&Us(kQz*7mv<H12p@UD8XaLXdK{>H^j*^MC@>wTDb}g%~sx*ng#>{@lR=XG-Z5_
z#<9*Oh0joMzt;nS)ObAp)347`D=}r-;nV!TbIq&xrGRGsF6fZg+!VkfUei@_&l-M&
zPqQ+Dw)RV}+)I8RuqAxa`Pv8e&!_gXS=e2-un>=Ktn}-;%lLZxaVn?Q>yZCb2R3Wk
z77zr%;Rq&h|2ncqyKYmFI0148JVY7Q$V5p=dWj<MQ<8r+@QLq+UROk&%quICq^O2C
zp(17882jXI)_GaqzAY4AjoB_nh8k*r1mJ>+Qqpu%i|xp2<qF`Tw6&3`h654ceBjZ7
z4>C=WaOb2Wudn^h0EcD%$p9YVU1fnoRV9`(cy(vv6K>FXS!2jY>1GnU--7)4usH&K
zao*&P^@9~YmUe|ZdLW@C>H;!*<TIU}-!Tw#3oqZA*99}b3&uHiGO<{I6!pMnAiQdS
P!fA@Yk4s_DjDP<F98V`a
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-receive.sample b/tests/resources/small.git/hooks/pre-receive.sample
new file mode 100755
index 0000000000000000000000000000000000000000..32b597f8c32399017c3d7d51134d1f6490ad9575
GIT binary patch
literal 601
zcmZ`#+iu%141Kn~f>Vt3>Nw4M*;=?j(TBD#O@XCv0|MEhA;z}kTFRv@`tPHhp=&Yh
zg%Zhg4i7o_k{a5i&f5;tZ==%}^Sn4aD_6%qs<o-wO_Prn;}`SPs_*$C$(7T|$#C3`
zPt%-C8gelZ1GqAP8`ZQmg0{8-S9H{R@D>_XAuJt&EumdH4Yu`UjT<s+s_~uXh}qA8
zg|_HG)%7Pdc&$7*uR0HF@)~vmFjqyD?XZwCbLen^h5t)sm9<90Oa!@Y%8!~rF8G?W
zkzmCF8kMtuuelMHIAlqqnm?72LeGM1J4`w(kX9&%LQn}ForlDLjBoCyvxmo@x3kH^
z^loxLyPiDWPo-cBMnsg2M6}kuPGHEGBqVkC{D&9Kt%xFAsTw4QC1$_=fwBl=3dI+e
zaSxI}JS}=Z(Ec80eF`!Zq3mqapXI|UN!a)t;@4hcu%Eq2xcoW}%!><-+XHTuHss+b
YOmM2;hq8Egm*4=7_P9T{21QBYH*F=mfB*mh
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/prepare-commit-msg.sample b/tests/resources/small.git/hooks/prepare-commit-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..b1970da1b6d3f42f00069fd17c325de72cda812e
GIT binary patch
literal 1702
zcmb_cTW{Mo6n>t6#i?x6xmZ$SFLf{QfG*3r0L?Pg?px55l8$UTGO3bO;spKi{V3XX
z))weX0X>M9bNMcZ-6yG%>(n}JI2|25dr<ew@wmMG{l(3lx~bQz>}WZBP@ih?JX^+@
zu#5O48P>yRX(m<b*PU*sORp92TCD1dX`%HE+1$w5k<(Ngu7zQ83#MGJR?<<W=d@yL
z#heqwo{FmCg0g#x<~R+PBD#}q(MBn;V$x;%UrJPP3*l%XtlvTWChI2SfJ$9e`YvSj
zRSOQ?NUgSMLI`3vL48YBHzwzVXoe7v0ef|0YHgV$N@?N(-R)rPqRDrK$n(%+OF$`P
zWdjC5N~`#RjV9}aYwQ4_yF5O-$h2`>fDIhYP)doc1&TADZa@ZGpusJ$6G+e$ZMcmC
zoOosDQPS}l{H?YPsq(4;0SGkATa9eeqAaDcj<jNAU+LTSmM1jo(ti~T0B7acJnnTX
zTarYy;Husdxal0!S<ba8=uu&a*SNYtr7|d7$g-q3_JHER2*oBsVW~i~XXdMx%LW~0
zT*960LF<qZ6K&FZ;vGmtyywBU3^Q>q8n2wALbFwU@2i@FAaRV!=uw-nwx1gKn2SvY
z>Ff>;2sg!+Hxfkwv1lsiii=p6WenF=5)6LZc<a$zDCD$GRuwvG4Fr+B#h?#9gTbio
zk#MdxC@WY%zSGN#i}Ts_#q`bf-{)`7CcWeB*7WlIyHjioJJWw&A5VItPUq3^9!r}S
zbykelFV-VFvcr>QaZ=aS_}+-4Y&?!@HWh|<^gJ21!|T@+%On#w6azxPHV}XsRbe*w
zR_TZ2XEsQa1lPK~biYqg@0-RW@5J1@=<87cFzEUABdCoFH2CZo?}l(Z*!OFqUxo>K
z_d`l#4d9|H6;VPT{X?^{VJ>oL|D7K{BJwwqB>`YcPoGk+9hbvHnoQ{EM|kPgD_`wk
zKm4#2xu;-y`RAm!=L_BnLvJ8$AZm8@?)v<%vwvsw8AF2x6!mTT;c72A_~U9nIq0ST
zv)N0!I!^1p=g8-RQfx5)E_Mb_4I2vtQpI30XZ&t<!9D6nI|;V7YR3)l6=S~QhuwLQ
g$e&^kTY-M99*<-Iw@(78*M5W!4}U}|8YyMx3->-9h5!Hn
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/push-to-checkout.sample b/tests/resources/small.git/hooks/push-to-checkout.sample
new file mode 100755
index 0000000000000000000000000000000000000000..a80611e18896f212c390d845e49a3f6d5693b41d
GIT binary patch
literal 2840
zcmai0U31$u5PXh)#YOS7cE^-rw@uolNhe9&aUS|HtvhX>G$45tVUYj>fRdF?|9kfU
zNR~aG=E)WbEbeyq7JTw}ZuHIE2kUtL<<n;$&G!2F^Je|kx2ug=4L5!H^!ogx`7o$&
z%Il(3zAe6<oe$^F=A|}s`8}CDp*M#3M)gC-)LOeDUpYMl3YNy9R)I-T)pE7sy09aj
zJ7%&5PnSB-F#2{jc><WLR{I2izuK%VHc+{hRfXe<^_q)8RjcE(6WX+F2)iAtDtI{x
ztAHVBq)eSp_E^xcV^i_5KLIHA$g{zEjh?rsacu+(Eyvve2~Kmw%;n3g(kWB56j~Js
z<yE5tYUsAR&PY0wgRvM8x!zgLX8SI!eVY&}YZ|>AoeCNptd-NM1aZLhESzC;I`+Ns
zfmNNjdAp^W8#Q*}l>CT7RB9F5(BbI8ly2l~+E};JW|>&d1)=epZ-8vm8ppkbEVn#R
zt30a5A-c(YQR8eM5%;|UAnO>rt!&@x@G@yp+92%w-}%(5P_+P&Wf_zb$f-Qrl5(7z
z2ah(bkE;!DK(&aAMuQ%1TS>ai?wSXCOCSj=_}8x4IbCx^$}9q)<W{Y<9o<WqZo^oV
z3bR;Qa%VT-M~kU@2n{=+=)C!MD`12;@<F*EtPfV3K#g^1%&ggH?4{m<R$WEKcUI4%
zl6{ik6Bo46pmNjdXg5@?hn;SjG$}rr3Gy#-BGgVD$8oD)OcK(oqca)L_kk)UBZ_&6
z*ourb#Yc8l3J+uSda_Y$GY--5Zp3QK9w^?P%E0xb57?fY+Q#+KU4)+R>whwv)SBt|
zg#MX4;;Oau`m=MI9(^&zPbueY@~>3*ixX%mvR5m_1&nAg@ZKvY1E$O}&EtLiG;mhV
z1xhMIm~fGjmf_#{62f`y;09?I7M1W2tWQvz<}i9lR>OpQyUJi45_&*pQus&EkwY<>
zI|ZAx=*3i9a-)g)hXkvO7>UJ5MNgL(Z+-wpXVcgbSgpmFmbf1~DPA(OVGI&FNLeIE
zNH!_aiH$vsif$_j7=T2{cS(!DOI`~bn@)vSd-0d7xL=DF;UNP|tW}4i<qWTSNCe|y
zg9kV&L9g;FhC@tvsVu#V-brqShHy2qZpA!gg{ZTY>h>DvHtu9tY_pbJ6x(6E*hxgC
zzNDao%qlr-IE%YGbS4hF!n!on7#W3$bX-_hbZAaws^nHu#)Dx=WzdbJ>AKzAy@T$x
zSWE^x9+|TEHVEPyaPYa0DOChp?AeHSBBDbZNokQpAY{lE!7geZI=jV)G^2@<iI(N4
zJLJjy@Y<VI$yq;3bVpJICW3D?YDMDl4Oe5pDf{d`i1_3Qx%4uX`$dOz<9e}jm2B-u
z8-?%!7XuiNF2O&MAdl-iw{drG+$F^zT2Z7WRV#c6;IRZEf>l)&91Zb1+`T+oq9wWF
zRV~kGTGce0O~p^6mj{kT5kL(pv>r;Lvd7VDX*P>A^Th`$3cWO<svk?_?FeP@458*2
zA1PqUOdd%VP5&4~ocLK16{1GLll64c=mU81RMFstpnMoLuI7hNIE4N)U%h*#<Fz{C
z9#>0<l7lRrls|W(8=W1O80P~6+U7<4BqC}N4-4GR3w#{O7YmH?JxwJfru2d?e){$5
z@5R+`7Z;1)FW;OkE-(HPisCS;5F4O^Q>L81p4Ysdo3ZP1(SrR-peEdTo;-@bkB((G
zPHYQXUL!@Q$e(OQ;R9r%@Afz+50I7>*^^c&&|E*r-jN)LH=pM4AqMwWxSv|nqjddE
Z4{_hwv8!W(<d3>T<BC%y-6O5i(|^OS%`gA}
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/sendemail-validate.sample b/tests/resources/small.git/hooks/sendemail-validate.sample
new file mode 100755
index 0000000000000000000000000000000000000000..effa6f54f2a5c1f0cebe258bf4805904be4b4852
GIT binary patch
literal 2365
zcmb_dU2oe)5PUX&#g-Nv2{5JDX_MA~3%Iq?8g*kpRgpYZIFU=~BI=I5J6e{T{`bz^
zk+$48h#x9Ika!=nv$M0y{clD}-j1x(hDWbnzP?l2k8j?TH{brS+Nf21yPm)Nczma>
zYw`X3V>TCdnSD1ru8&`j=2DIPbCT@SnIgUw>$+lEYP}+x8(BMYnr=iT3*ndq)xzaV
z>I+qjv}vC#8_9M+b1p#uNS0M0)q<p>8!3p_LRQ0MA3M`!2foxzRUjbFY@}O~(ki=S
zqscnq8cU*dY)D$$cqE}n)V0yIk>CNKHCrndOtSP*HbOb;nbwAHSb;R+gs^?^Dve%)
zoW}t(*D}$>O3ab0TS^-;J|u&sb-PkZzo#kn*#xYt(;<xzKW(YtQZ$u23?yV#kyh1~
z@+Idh;EG5jXx8@%<;Y_W8SA=|T;MPQ)TB!!<QcbU)YR4)79eeeg4|vp-8jm%Dl3^I
zRS7+i3>FGuwzSb^g&RDiGcOz9TB;Hu`nJh)$W=C=XCSm2AY=$w3G3P-V#Oo+N*;#2
z4ijJ-pBZ=;T(RTgp_HYrD!uW-dTMfkuqY5jwOy)~gM;#=P^i{!l7`pXTS^s(&^{RU
zydaw}OpS#^D1cXM8?FW+fh`t7D(g;yr6|}fdaNtZBx3hlK~IpkTu3!Qq%R+zAo#<L
zU&m+XIFB0>t}Bs8^3$vHD+-TGT@`F>H1Cc#WAVW;&$S6%fE2d6@kLS0g&ihIM{}0z
z8#XhD>b>3{(BH|Px7}&lJ4%y1v<t$W+!F|W@<gaU4;MqSHKR(wdML<XnCv;zaPrSi
zxVCvek26-bf#Q!H+uAgy_{e_1UZCq>(CihZJx@8MPoGdl*BJGD;usf*iS7%;{Joe;
zNFuBa>*~o&qETDPo~u&~$FxE1xb^x&(CbE`Y3GfsibL2rl+L;>P6j&Y3U>K$mkp*6
zd`Q{<^+^&;GskGjwD-%!boR&i-TC<Uvy02w+l$Nb?B}aL-%ZDpluqd=K_@}z*fyuV
zzAs1Hf?3v$k!X7GTc8Q=r`WJ_Uu=>A9UOR|@=GYb5x#<f&WSMH%mCJU<#=7=qFdL6
zG?Wz&6z&J<@I(B>+dhd7fkaVIR^pol`Mv+rUbmZ43dVL6^S7g3{NsPiG$iy$5EDB%
z6KIgnb$H(n&t3e4E6d4V7w^B?JS}JkG)PM6+X3Co`SQs($O*AA+MG~{S7RJ=cy-l&
z>~%3y`tjfx2>uOu<lDGWewcb=oL@}B@B6FCZ?oxSJWldrm%UfOduahk%C0H>tB_^s
ziwG=e=ch|FQ0IkN91US7rhdQkXhwwt$gU0WEVDjo=IPb+?6PC=s8}J*ua(Ms))`UL
fi$|vMHn?H<rred|1&u#kOoJ`%vx)-*g-ZSfgGvdP
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/update.sample b/tests/resources/small.git/hooks/update.sample
new file mode 100755
index 0000000000000000000000000000000000000000..c3430f6fe27e572a1cebdb2f2bec70add0642a60
GIT binary patch
literal 3707
zcmb_fU2oeq6n(aS#jRq*br9K3y0z;^QgkUc^kKlTrB6js&@yE)mPu8l<0$L?`wmIT
zrsXba))@gJi@e|G+<SfSXe`CeSQ}OG@sr8ZTUlQ{dzM}Q@O-hBi}GeUom`#X%FiYH
zX?m4Rna-0RN2lfK)A3ZuvHcz$L<jUn62D=~vfz{}wIH2VqBLX_O$(JSXeF7H$}q!c
zWY}C&R;eX%X?P{%d;|>_tSE3ettp-hLlsZCxaLX8(nU;bVRB;Ce6@s#eu2|WvLz>-
zvy(&>Gyfp@+BtKnpqWkKi^+v{4jn_pNw_zeuxE<mRXKx8G3;9pl+45&4~hHW!G@wo
za7?X(0B}HbX*ExkDmas*xzV)FxygC8AL?2Z1x-0QJqS@qn8sD7r{bm30@<%eL_gOw
z;~85O$Xw2AS}Qp)5ViRUe3|ir8;&&I<B7Y6^!kkNyYXF4EY(b8_5DsTYn_&?wkdEz
z0y$tADo<&}nGs5kg2-J=0KlEGPb(%<An(pXY{K`qIZCuwiT|2{8JD&5o_~`o6<;dD
zi@J#zCE4={8j%<uy|iutvHu1QKo5Tno-BAF2FwD%%O#UDDum=w!;!PNe-cOFNX4)5
zd>TifiGO|)w}OANj2n2D^K=o3j6P6uOL70#cbA{uzWXDlk1wr9GV1X(2W{RuTvjXV
zCmd<W?kH^?PXjkbF`XZtwu1B+%4@ZvHIwGpJ*8@8`MWAhq^B|Hj1lzj3R8bVuMpNb
zz4Gzk!3T3bY;WEGIww&kq9BYW6EP*q$K|EB-@TH(Fjtz*`HMTO?i+3E;5tdSah&xZ
z+t!x4K7)dpy5wiJhlJz~8qF|r8a&-SV7^I3C@_q=P`yt@_x_F-;PQR)fzP<zNN>8u
zH%V`94=q3)Dk)PHNrnFC(T1)Om6f{Usj;u1R->&XoCYVK2V3ZlgZuF?N}1+33<P7e
z<0yVF?Qoa{l#7q(3&jv=Ab)gpM8A7`p%3InNzSxy)ZER2U0C#9zKpnLY0I?>OER*x
z*9Z=L=zI8CN>A_^jYjt0F$psO$sL=38q5q|SG)qCN6{^>RFh5E&l5GZ$pEahnF&d+
z5c>64t}uJPkf~_!VUj#&N%nC-gUMj%=@B=!V>&}xtj2%@-mOm#rQUSJ3(ccmc+fza
znZ#uxF>N?QN5UrIEd!5RgHEf<eGg}P45aAs(Xs6u!XW9r1I*E6XJ^1movX@xYLuPz
z|7xBN4z@b}#x>W#;(nKYF+D<*rdshJ$X-z2OZ2X;)nn@KSVdVhaA?}@3;6gZxb4<W
z`9sa`0lR_azMX|=t_(FvG@%w2);9P}SG0u&K1(*oX3};~=<<!N*F$UTSxD{V&6mgL
ztw9Ntc2eOF@c!OJytNC4T^#)Ien7-`dI{6s#co~0f^E3J<0Ty)l3xq2u@Y8DXTK>v
zozoWSr{{+!h}zGpumG3H`=AvWpm^9kW;J$Jp^Xl*?8ckr`fqN%c|Z;VC0|cM4vSrk
zH_O8Yvh85nvJp^;``wo8=z0f`FWg?`>gO#y1hjX1{}rTlg9rwIKia8eyGexA3GnuR
z`Rg~XZoW;0pA)vI8=p5!+6sIn#C^FCvR>ffv39h6SCNi9v);%WD;WZ`of_MgwyRWy
z-yY%n*Y>X8<Sf+RyB|Sr2YG@1w~%U$o`(5EDkJ|3$u=eMZA&_wxEsy<Xxh2k^tP?4
RGx&ZHQs^8zuIpu!=pQhfp8Eg*
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/info/exclude b/tests/resources/small.git/info/exclude
new file mode 100644
index 0000000000000000000000000000000000000000..a5196d1be8fb59edf8062bef36d3a602e0812139
GIT binary patch
literal 240
zcmXYqK?=e!6h!x)VxWtv*mf_t5?px$aS_{}Hj?C*<cHXeXE&AZhT*-L3ZoI&*l1%Z
zqG?zr3TvQGZ__}H4(u*%p*rI=cU!%ya5ugfGATh66$IJHgu1Gs0-<N;$V+SsdE)?u
zIq;i$f#WE4f$_MWicZjMEob9LWKMR#iwZq54~QgST^6=i%u0lUkJu-_J**QBMq}ZG
Wth_)NDbl|`oQr&HAFQ5h`0jqAIZ*BZ
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/88/f9a6eaa2cf008b9bc92847178621f21fa99f3e b/tests/resources/small.git/objects/88/f9a6eaa2cf008b9bc92847178621f21fa99f3e
new file mode 100644
index 0000000000000000000000000000000000000000..7d80b8d78e0dc55669d831a6638f48ec9fed0982
GIT binary patch
literal 50
zcmV-20L}k+0V^p=O;s>9W-v4`Ff%bx$Vkn}$!Ay}rnY6F$m-Kg*KD_+;Lx#g4|^&N
I02NaX#p`nv=Kufz
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/af/5626b4a114abcb82d63db7c8082c3c4756e51b b/tests/resources/small.git/objects/af/5626b4a114abcb82d63db7c8082c3c4756e51b
new file mode 100644
index 0000000000000000000000000000000000000000..822bc151862ec3763cf2d3fa2372b93bbd3a4b65
GIT binary patch
literal 30
mcmb<m^geacKghr&@q@?NlP9kSYMj?U<r(;diNWtH+YSKNt_|)0
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/c6/97d4f7a6eac8d7b131673c340bd3cc5bac14d4 b/tests/resources/small.git/objects/c6/97d4f7a6eac8d7b131673c340bd3cc5bac14d4
new file mode 100644
index 0000000000000000000000000000000000000000..5adfa88cc6b00bb19f2031b8ab61f6d07f9ccdb8
GIT binary patch
literal 130
zcmV-|0Db>>0i}&W3IZ_@1U=^!a~EV1casc=c+{&un1qQN*i9hD|0|m(2n|iwp*q%W
z%N;b$hu%cM`$TMo*~EnC1BFP&Pfj~;jZVKXQ96s_PhV<-XAROi+@-v8dBLUa`!;GB
k^i<X>XlEv8$>R)1G>9th&t3j;s7J{?^9n<zzF|~BaA?ar-~a#s
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/refs/heads/master b/tests/resources/small.git/refs/heads/master
new file mode 100644
index 0000000000000000000000000000000000000000..4eb36d22298f060fd324155ab854d9d6486fc498
GIT binary patch
literal 41
ucmV~$!4Uu;2m`Rc)5uXl$AMP&AHjriQg~T$i(A>|7U^`%mXoWC24Q^m!3%@{
literal 0
HcmV?d00001
+21 -10
View File
@@ -55,18 +55,22 @@ readonly NIX_INSTALLED_NIX="@nix@"
readonly NIX_INSTALLED_CACERT="@cacert@"
#readonly NIX_INSTALLED_NIX="/nix/store/j8dbv5w6jl34caywh2ygdy88knx1mdf7-nix-2.3.6"
#readonly NIX_INSTALLED_CACERT="/nix/store/7dxhzymvy330i28ii676fl1pqwcahv2f-nss-cacert-3.49.2"
readonly EXTRACTED_NIX_PATH="$(dirname "$0")"
EXTRACTED_NIX_PATH="$(dirname "$0")"
readonly EXTRACTED_NIX_PATH
# allow to override identity change command
readonly NIX_BECOME=${NIX_BECOME:-sudo}
NIX_BECOME=${NIX_BECOME:-sudo}
readonly NIX_BECOME
readonly ROOT_HOME=~root
ROOT_HOME=~root
readonly ROOT_HOME
if [ -t 0 ] && [ -z "${NIX_INSTALLER_YES:-}" ]; then
readonly IS_HEADLESS='no'
IS_HEADLESS='no'
else
readonly IS_HEADLESS='yes'
IS_HEADLESS='yes'
fi
readonly IS_HEADLESS
headless() {
if [ "$IS_HEADLESS" = "yes" ]; then
@@ -156,6 +160,7 @@ EOF
}
nix_user_for_core() {
# shellcheck disable=SC2059
printf "$NIX_BUILD_USER_NAME_TEMPLATE" "$1"
}
@@ -381,10 +386,12 @@ _sudo() {
# Ensure that $TMPDIR exists if defined.
if [[ -n "${TMPDIR:-}" ]] && [[ ! -d "${TMPDIR:-}" ]]; then
# shellcheck disable=SC2174
mkdir -m 0700 -p "${TMPDIR:-}"
fi
readonly SCRATCH=$(mktemp -d)
SCRATCH=$(mktemp -d)
readonly SCRATCH
finish_cleanup() {
rm -rf "$SCRATCH"
}
@@ -677,7 +684,8 @@ create_directories() {
# hiding behind || true, and the general state
# should be one the user can repair once they
# figure out where chown is...
local get_chr_own="$(PATH="$(getconf PATH 2>/dev/null)" command -vp chown)"
local get_chr_own
get_chr_own="$(PATH="$(getconf PATH 2>/dev/null)" command -vp chown)"
if [[ -z "$get_chr_own" ]]; then
get_chr_own="$(command -v chown)"
fi
@@ -915,9 +923,11 @@ configure_shell_profile() {
fi
if [ -e "$profile_target" ]; then
shell_source_lines \
| _sudo "extend your $profile_target with nix-daemon settings" \
tee -a "$profile_target"
{
shell_source_lines
cat "$profile_target"
} | _sudo "extend your $profile_target with nix-daemon settings" \
tee "$profile_target"
fi
done
@@ -1013,6 +1023,7 @@ main() {
# Set profile targets after OS-specific scripts are loaded
if command -v poly_configure_default_profile_targets > /dev/null 2>&1; then
# shellcheck disable=SC2207
PROFILE_TARGETS=($(poly_configure_default_profile_targets))
else
PROFILE_TARGETS=("/etc/bashrc" "/etc/profile.d/nix.sh" "/etc/zshrc" "/etc/bash.bashrc" "/etc/zsh/zshrc")
+1 -1
View File
@@ -39,7 +39,7 @@ create_systemd_proxy_env() {
vars="http_proxy https_proxy ftp_proxy all_proxy no_proxy HTTP_PROXY HTTPS_PROXY FTP_PROXY ALL_PROXY NO_PROXY"
for v in $vars; do
if [ "x${!v:-}" != "x" ]; then
echo "Environment=${v}=$(escape_systemd_env ${!v})"
echo "Environment=${v}=$(escape_systemd_env "${!v}")"
fi
done
}
+1
View File
@@ -0,0 +1 @@
../../.version
+1
View File
@@ -0,0 +1 @@
../../src/libstore-tests/data/derivation
+1
View File
@@ -0,0 +1 @@
../../src/libutil-tests/data/memory-source-accessor
+1
View File
@@ -0,0 +1 @@
../../src/libutil-tests/data/hash
+92
View File
@@ -0,0 +1,92 @@
# Run with:
# meson test --suite json-schema
# Run with: (without shell / configure)
# nix build .#nix-json-schema-checks
project(
'nix-json-schema-checks',
version : files('.version'),
meson_version : '>= 1.1',
license : 'LGPL-2.1-or-later',
)
fs = import('fs')
# Note: The 'jsonschema' package provides the 'jv' command
jv = find_program('jv', required : true)
# The schema directory is a committed symlink to the actual schema location
schema_dir = meson.current_source_dir() / 'schema'
# Get all example files
schemas = [
{
'stem' : 'file-system-object',
'schema' : schema_dir / 'file-system-object-v1.yaml',
'files' : [
'simple.json',
'complex.json',
],
},
{
'stem' : 'hash',
'schema' : schema_dir / 'hash-v1.yaml',
'files' : [
'sha256-base64.json',
'sha256-base16.json',
'sha256-nix32.json',
'blake3-base64.json',
],
},
{
'stem' : 'derivation',
'schema' : schema_dir / 'derivation-v3.yaml',
'files' : [
'dyn-dep-derivation.json',
'simple-derivation.json',
],
},
# # Not sure how to make subschema work
# {
# 'stem': 'derivation',
# 'schema': schema_dir / 'derivation-v3.yaml#output',
# 'files' : [
# 'output-caFixedFlat.json',
# 'output-caFixedNAR.json',
# 'output-caFixedText.json',
# 'output-caFloating.json',
# 'output-deferred.json',
# 'output-impure.json',
# 'output-inputAddressed.json',
# ],
# },
]
# Validate each example against the schema
foreach schema : schemas
stem = schema['stem']
schema_file = schema['schema']
if '#' not in schema_file
# Validate the schema itself against JSON Schema Draft 04
test(
stem + '-schema-valid',
jv,
args : [
'http://json-schema.org/draft-04/schema',
schema_file,
],
suite : 'json-schema',
)
endif
foreach example : schema['files']
test(
stem + '-example-' + fs.stem(example),
jv,
args : [
schema_file,
files(stem / example),
],
suite : 'json-schema',
)
endforeach
endforeach
+52
View File
@@ -0,0 +1,52 @@
# Run with: nix build .#nix-json-schema-checks
{
lib,
mkMesonDerivation,
meson,
ninja,
jsonschema,
# Configuration Options
version,
}:
mkMesonDerivation (finalAttrs: {
pname = "nix-json-schema-checks";
inherit version;
workDir = ./.;
fileset = lib.fileset.unions [
../../.version
../../doc/manual/source/protocols/json/schema
../../src/libutil-tests/data/memory-source-accessor
../../src/libutil-tests/data/hash
../../src/libstore-tests/data/derivation
./.
];
outputs = [ "out" ];
passthru.externalNativeBuildInputs = [
jsonschema
];
nativeBuildInputs = [
meson
ninja
]
++ finalAttrs.passthru.externalNativeBuildInputs;
doCheck = true;
mesonCheckFlags = [ "--print-errorlogs" ];
postInstall = ''
touch $out
'';
meta = {
platforms = lib.platforms.all;
};
})
+1
View File
@@ -0,0 +1 @@
../../doc/manual/source/protocols/json/schema
+15 -4
View File
@@ -83,12 +83,22 @@ nlohmann::json SingleBuiltPath::Built::toJSON(const StoreDirConfig & store) cons
nlohmann::json SingleBuiltPath::toJSON(const StoreDirConfig & store) const
{
return std::visit([&](const auto & buildable) { return buildable.toJSON(store); }, raw());
return std::visit(
overloaded{
[&](const SingleBuiltPath::Opaque & o) -> nlohmann::json { return store.printStorePath(o.path); },
[&](const SingleBuiltPath::Built & b) { return b.toJSON(store); },
},
raw());
}
nlohmann::json BuiltPath::toJSON(const StoreDirConfig & store) const
{
return std::visit([&](const auto & buildable) { return buildable.toJSON(store); }, raw());
return std::visit(
overloaded{
[&](const BuiltPath::Opaque & o) -> nlohmann::json { return store.printStorePath(o.path); },
[&](const BuiltPath::Built & b) { return b.toJSON(store); },
},
raw());
}
RealisedPath::Set BuiltPath::toRealisedPaths(Store & store) const
@@ -107,10 +117,11 @@ RealisedPath::Set BuiltPath::toRealisedPaths(Store & store) const
"the derivation '%s' has unrealised output '%s' (derived-path.cc/toRealisedPaths)",
store.printStorePath(p.drvPath->outPath()),
outputName);
auto thisRealisation = store.queryRealisation(DrvOutput{*drvOutput, outputName});
DrvOutput key{*drvOutput, outputName};
auto thisRealisation = store.queryRealisation(key);
assert(thisRealisation); // We’ve built it, so we must
// have the realisation
res.insert(*thisRealisation);
res.insert(Realisation{*thisRealisation, std::move(key)});
} else {
res.insert(outputPath);
}
+1
View File
@@ -15,6 +15,7 @@
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/cmd/compatibility-settings.hh"
#include "nix/expr/eval-settings.hh"
#include "nix/store/globals.hh"
namespace nix {
+14
View File
@@ -350,6 +350,20 @@ struct MixEnvironment : virtual Args
void setEnviron();
};
struct MixNoCheckSigs : virtual Args
{
CheckSigsFlag checkSigs = CheckSigs;
MixNoCheckSigs()
{
addFlag({
.longName = "no-check-sigs",
.description = "Do not require that paths are signed by trusted keys.",
.handler = {&checkSigs, NoCheckSigs},
});
}
};
void completeFlakeInputAttrPath(
AddCompletions & completions,
ref<EvalState> evalState,
@@ -69,7 +69,7 @@ struct InstallableFlake : InstallableValue
*/
std::vector<ref<eval_cache::AttrCursor>> getCursors(EvalState & state) override;
std::shared_ptr<flake::LockedFlake> getLockedFlake() const;
ref<flake::LockedFlake> getLockedFlake() const;
FlakeRef nixpkgsFlakeRef() const;
};
@@ -87,6 +87,4 @@ static inline FlakeRef defaultNixpkgsFlakeRef()
return FlakeRef::fromAttrs(fetchSettings, {{"type", "indirect"}, {"id", "nixpkgs"}});
}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, std::shared_ptr<flake::LockedFlake> lockedFlake);
} // namespace nix
+1 -3
View File
@@ -89,8 +89,7 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths()
}
DerivedPathsWithInfo res;
for (auto & [drvPath, outputs] : byDrvPath) {
state->waitForPath(drvPath);
for (auto & [drvPath, outputs] : byDrvPath)
res.push_back({
.path =
DerivedPath::Built{
@@ -103,7 +102,6 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths()
so we can fill in this info. */
}),
});
}
return res;
}
+8 -9
View File
@@ -102,12 +102,11 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths()
}
auto drvPath = attr->forceDerivation();
state->waitForPath(drvPath);
std::optional<NixInt::Inner> priority;
if (attr->maybeGetAttr(state->sOutputSpecified)) {
} else if (auto aMeta = attr->maybeGetAttr(state->sMeta)) {
if (attr->maybeGetAttr(state->s.outputSpecified)) {
} else if (auto aMeta = attr->maybeGetAttr(state->s.meta)) {
if (auto aPriority = aMeta->maybeGetAttr("priority"))
priority = aPriority->getInt().value;
}
@@ -120,12 +119,12 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths()
overloaded{
[&](const ExtendedOutputsSpec::Default & d) -> OutputsSpec {
StringSet outputsToInstall;
if (auto aOutputSpecified = attr->maybeGetAttr(state->sOutputSpecified)) {
if (auto aOutputSpecified = attr->maybeGetAttr(state->s.outputSpecified)) {
if (aOutputSpecified->getBool()) {
if (auto aOutputName = attr->maybeGetAttr("outputName"))
outputsToInstall = {aOutputName->getString()};
}
} else if (auto aMeta = attr->maybeGetAttr(state->sMeta)) {
} else if (auto aMeta = attr->maybeGetAttr(state->s.meta)) {
if (auto aOutputsToInstall = aMeta->maybeGetAttr("outputsToInstall"))
for (auto & s : aOutputsToInstall->getListOfStrings())
outputsToInstall.insert(s);
@@ -186,16 +185,16 @@ std::vector<ref<eval_cache::AttrCursor>> InstallableFlake::getCursors(EvalState
return res;
}
std::shared_ptr<flake::LockedFlake> InstallableFlake::getLockedFlake() const
ref<flake::LockedFlake> InstallableFlake::getLockedFlake() const
{
if (!_lockedFlake) {
flake::LockFlags lockFlagsApplyConfig = lockFlags;
// FIXME why this side effect?
lockFlagsApplyConfig.applyNixConfig = true;
_lockedFlake =
std::make_shared<flake::LockedFlake>(lockFlake(flakeSettings, *state, flakeRef, lockFlagsApplyConfig));
_lockedFlake = make_ref<flake::LockedFlake>(lockFlake(flakeSettings, *state, flakeRef, lockFlagsApplyConfig));
}
return _lockedFlake;
// _lockedFlake is now non-null but still just a shared_ptr
return ref<flake::LockedFlake>(_lockedFlake);
}
FlakeRef InstallableFlake::nixpkgsFlakeRef() const
+18 -47
View File
@@ -178,10 +178,16 @@ MixFlakeOptions::MixFlakeOptions()
for (auto & [inputName, input] : flake.lockFile.root->inputs) {
auto input2 = flake.lockFile.findInput({inputName}); // resolve 'follows' nodes
if (auto input3 = std::dynamic_pointer_cast<const flake::LockedNode>(input2)) {
fetchers::Attrs extraAttrs;
if (!input3->lockedRef.subdir.empty()) {
extraAttrs["dir"] = input3->lockedRef.subdir;
}
overrideRegistry(
fetchers::Input::fromAttrs(fetchSettings, {{"type", "indirect"}, {"id", inputName}}),
input3->lockedRef.input,
{});
extraAttrs);
}
}
}},
@@ -336,8 +342,7 @@ void completeFlakeRefWithFragment(
parseFlakeRef(fetchSettings, expandTilde(flakeRefS), std::filesystem::current_path().string());
auto evalCache = openEvalCache(
*evalState,
std::make_shared<flake::LockedFlake>(lockFlake(flakeSettings, *evalState, flakeRef, lockFlags)));
*evalState, make_ref<flake::LockedFlake>(lockFlake(flakeSettings, *evalState, flakeRef, lockFlags)));
auto root = evalCache->getRoot();
@@ -437,42 +442,6 @@ static StorePath getDeriver(ref<Store> store, const Installable & i, const Store
return *derivers.begin();
}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, std::shared_ptr<flake::LockedFlake> lockedFlake)
{
auto fingerprint = evalSettings.useEvalCache && evalSettings.pureEval
? lockedFlake->getFingerprint(state.store, state.fetchSettings)
: std::nullopt;
auto rootLoader = [&state, lockedFlake]() {
/* For testing whether the evaluation cache is
complete. */
if (getEnv("NIX_ALLOW_EVAL").value_or("1") == "0")
throw Error("not everything is cached, but evaluation is not allowed");
auto vFlake = state.allocValue();
flake::callFlake(state, *lockedFlake, *vFlake);
state.forceAttrs(*vFlake, noPos, "while parsing cached flake data");
auto aOutputs = vFlake->attrs()->get(state.symbols.create("outputs"));
assert(aOutputs);
return aOutputs->value;
};
if (fingerprint) {
auto search = state.evalCaches.find(fingerprint.value());
if (search == state.evalCaches.end()) {
search =
state.evalCaches
.emplace(fingerprint.value(), make_ref<nix::eval_cache::EvalCache>(fingerprint, state, rootLoader))
.first;
}
return search->second;
} else {
return make_ref<nix::eval_cache::EvalCache>(std::nullopt, state, rootLoader);
}
}
Installables SourceExprCommand::parseInstallables(ref<Store> store, std::vector<std::string> ss)
{
Installables result;
@@ -598,28 +567,28 @@ std::vector<BuiltPathWithResult> Installable::build(
static void throwBuildErrors(std::vector<KeyedBuildResult> & buildResults, const Store & store)
{
std::vector<KeyedBuildResult> failed;
std::vector<std::pair<const KeyedBuildResult *, const KeyedBuildResult::Failure *>> failed;
for (auto & buildResult : buildResults) {
if (!buildResult.success()) {
failed.push_back(buildResult);
if (auto * failure = buildResult.tryGetFailure()) {
failed.push_back({&buildResult, failure});
}
}
auto failedResult = failed.begin();
if (failedResult != failed.end()) {
if (failed.size() == 1) {
failedResult->rethrow();
failedResult->second->rethrow();
} else {
StringSet failedPaths;
for (; failedResult != failed.end(); failedResult++) {
if (!failedResult->errorMsg.empty()) {
if (!failedResult->second->errorMsg.empty()) {
logError(
ErrorInfo{
.level = lvlError,
.msg = failedResult->errorMsg,
.msg = failedResult->second->errorMsg,
});
}
failedPaths.insert(failedResult->path.to_string(store));
failedPaths.insert(failedResult->first->path.to_string(store));
}
throw Error("build of %s failed", concatStringsSep(", ", quoteStrings(failedPaths)));
}
@@ -689,12 +658,14 @@ std::vector<std::pair<ref<Installable>, BuiltPathWithResult>> Installable::build
auto buildResults = store->buildPathsWithResults(pathsToBuild, bMode, evalStore);
throwBuildErrors(buildResults, *store);
for (auto & buildResult : buildResults) {
// If we didn't throw, they must all be sucesses
auto & success = std::get<nix::BuildResult::Success>(buildResult.inner);
for (auto & aux : backmap[buildResult.path]) {
std::visit(
overloaded{
[&](const DerivedPath::Built & bfd) {
std::map<std::string, StorePath> outputs;
for (auto & [outputName, realisation] : buildResult.builtOutputs)
for (auto & [outputName, realisation] : success.builtOutputs)
outputs.emplace(outputName, realisation.outPath);
res.push_back(
{aux.installable,
+1
View File
@@ -95,6 +95,7 @@ this_library = library(
'nixcmd',
sources,
config_priv_h,
soversion : nix_soversion,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
link_args : linker_export_flags,
+8 -10
View File
@@ -333,7 +333,6 @@ StorePath NixRepl::getDerivationPath(Value & v)
auto drvPath = packageInfo->queryDrvPath();
if (!drvPath)
throw Error("expression did not evaluate to a valid derivation (no 'drvPath' attribute)");
state->waitForPath(*drvPath);
if (!state->store->isValidPath(*drvPath))
throw Error("expression evaluated to invalid derivation '%s'", state->store->printStorePath(*drvPath));
return *drvPath;
@@ -670,7 +669,7 @@ ProcessLineResult NixRepl::processLine(std::string line)
ss << "No documentation found.\n\n";
}
auto markdown = toView(ss);
auto markdown = ss.view();
logger->cout(trim(renderMarkdownToTerminal(markdown)));
} else
@@ -761,7 +760,7 @@ void NixRepl::loadFlake(const std::string & flakeRefS)
void NixRepl::initEnv()
{
env = &state->allocEnv(envSize);
env = &state->mem.allocEnv(envSize);
env->up = &state->baseEnv;
displ = 0;
staticEnv->vars.clear();
@@ -870,14 +869,8 @@ void NixRepl::addVarToScope(const Symbol name, Value & v)
Expr * NixRepl::parseString(std::string s)
{
return state->parseExprFromString(std::move(s), state->rootPath("."), staticEnv);
}
void NixRepl::evalString(std::string s, Value & v)
{
Expr * e;
try {
e = parseString(s);
return state->parseExprFromString(std::move(s), state->rootPath("."), staticEnv);
} catch (ParseError & e) {
if (e.msg().find("unexpected end of file") != std::string::npos)
// For parse errors on incomplete input, we continue waiting for the next line of
@@ -886,6 +879,11 @@ void NixRepl::evalString(std::string s, Value & v)
else
throw;
}
}
void NixRepl::evalString(std::string s, Value & v)
{
Expr * e = parseString(s);
e->eval(*state, *env, v);
state->forceValue(v, v.determinePos(noPos));
}
+1
View File
@@ -50,6 +50,7 @@ subdir('nix-meson-build-support/windows-version')
this_library = library(
'nixexprc',
sources,
soversion : nix_soversion,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
link_args : linker_export_flags,
+16 -25
View File
@@ -16,7 +16,7 @@
#include "nix_api_util_internal.h"
#if NIX_USE_BOEHMGC
# include <mutex>
# include <boost/unordered/concurrent_flat_map.hpp>
#endif
/**
@@ -40,6 +40,8 @@ static T * unsafe_new_with_self(F && init)
return new (p) T(init(static_cast<T *>(p)));
}
extern "C" {
nix_err nix_libexpr_init(nix_c_context * context)
{
if (context)
@@ -69,7 +71,6 @@ nix_err nix_expr_eval_from_string(
nix::Expr * parsedExpr = state->state.parseExprFromString(expr, state->state.rootPath(nix::CanonPath(path)));
state->state.eval(parsedExpr, value->value);
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -81,7 +82,6 @@ nix_err nix_value_call(nix_c_context * context, EvalState * state, Value * fn, n
try {
state->state.callFunction(fn->value, arg->value, value->value, nix::noPos);
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -94,7 +94,6 @@ nix_err nix_value_call_multi(
try {
state->state.callFunction(fn->value, {(nix::Value **) args, nargs}, value->value, nix::noPos);
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -105,7 +104,6 @@ nix_err nix_value_force(nix_c_context * context, EvalState * state, nix_value *
context->last_err_code = NIX_OK;
try {
state->state.forceValue(value->value, nix::noPos);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -116,7 +114,6 @@ nix_err nix_value_force_deep(nix_c_context * context, EvalState * state, nix_val
context->last_err_code = NIX_OK;
try {
state->state.forceValueDeep(value->value);
state->state.waitForAllPaths();
}
NIXC_CATCH_ERRS
}
@@ -140,7 +137,7 @@ nix_eval_state_builder * nix_eval_state_builder_new(nix_c_context * context, Sto
void nix_eval_state_builder_free(nix_eval_state_builder * builder)
{
delete builder;
operator delete(builder, static_cast<std::align_val_t>(alignof(nix_eval_state_builder)));
}
nix_err nix_eval_state_builder_load(nix_c_context * context, nix_eval_state_builder * builder)
@@ -206,32 +203,24 @@ EvalState * nix_state_create(nix_c_context * context, const char ** lookupPath_c
void nix_state_free(EvalState * state)
{
delete state;
operator delete(state, static_cast<std::align_val_t>(alignof(EvalState)));
}
#if NIX_USE_BOEHMGC
std::unordered_map<
boost::concurrent_flat_map<
const void *,
unsigned int,
std::hash<const void *>,
std::equal_to<const void *>,
traceable_allocator<std::pair<const void * const, unsigned int>>>
nix_refcounts;
std::mutex nix_refcount_lock;
nix_refcounts{};
nix_err nix_gc_incref(nix_c_context * context, const void * p)
{
if (context)
context->last_err_code = NIX_OK;
try {
std::scoped_lock lock(nix_refcount_lock);
auto f = nix_refcounts.find(p);
if (f != nix_refcounts.end()) {
f->second++;
} else {
nix_refcounts[p] = 1;
}
nix_refcounts.insert_or_visit({p, 1}, [](auto & kv) { kv.second++; });
}
NIXC_CATCH_ERRS
}
@@ -242,12 +231,12 @@ nix_err nix_gc_decref(nix_c_context * context, const void * p)
if (context)
context->last_err_code = NIX_OK;
try {
std::scoped_lock lock(nix_refcount_lock);
auto f = nix_refcounts.find(p);
if (f != nix_refcounts.end()) {
if (--f->second == 0)
nix_refcounts.erase(f);
} else
bool fail = true;
nix_refcounts.erase_if(p, [&](auto & kv) {
fail = false;
return !--kv.second;
});
if (fail)
throw std::runtime_error("nix_gc_decref: object was not referenced");
}
NIXC_CATCH_ERRS
@@ -292,3 +281,5 @@ void nix_gc_register_finalizer(void * obj, void * cd, void (*finalizer)(void * o
GC_REGISTER_FINALIZER(obj, finalizer, cd, 0, 0);
#endif
}
} // extern "C"
+4
View File
@@ -8,6 +8,8 @@
#include "nix_api_value.h"
#include "nix/expr/search-path.hh"
extern "C" {
struct nix_eval_state_builder
{
nix::ref<nix::Store> store;
@@ -61,4 +63,6 @@ struct nix_realised_string
std::vector<StorePath> storePaths;
};
} // extern "C"
#endif // NIX_API_EXPR_INTERNAL_H
+8
View File
@@ -14,6 +14,8 @@
#include <nlohmann/json.hpp>
extern "C" {
void nix_set_string_return(nix_string_return * str, const char * c)
{
str->str = c;
@@ -40,6 +42,8 @@ nix_err nix_external_add_string_context(nix_c_context * context, nix_string_cont
NIXC_CATCH_ERRS
}
} // extern "C"
class NixCExternalValue : public nix::ExternalValueBase
{
NixCExternalValueDesc & desc;
@@ -170,6 +174,8 @@ public:
virtual ~NixCExternalValue() override {};
};
extern "C" {
ExternalValue * nix_create_external_value(nix_c_context * context, NixCExternalValueDesc * desc, void * v)
{
if (context)
@@ -198,3 +204,5 @@ void * nix_get_external_value_content(nix_c_context * context, ExternalValue * b
}
NIXC_CATCH_ERRS_NULL
}
} // extern "C"
+94 -6
View File
@@ -111,6 +111,8 @@ static void nix_c_primop_wrapper(
v = vTmp;
}
extern "C" {
PrimOp * nix_alloc_primop(
nix_c_context * context,
PrimOpFun fun,
@@ -324,6 +326,10 @@ nix_value * nix_get_list_byidx(nix_c_context * context, const nix_value * value,
try {
auto & v = check_value_in(value);
assert(v.type() == nix::nList);
if (ix >= v.listSize()) {
nix_set_err_msg(context, NIX_ERR_KEY, "list index out of bounds");
return nullptr;
}
auto * p = v.listView()[ix];
nix_gc_incref(nullptr, p);
if (p != nullptr)
@@ -333,6 +339,26 @@ nix_value * nix_get_list_byidx(nix_c_context * context, const nix_value * value,
NIXC_CATCH_ERRS_NULL
}
nix_value *
nix_get_list_byidx_lazy(nix_c_context * context, const nix_value * value, EvalState * state, unsigned int ix)
{
if (context)
context->last_err_code = NIX_OK;
try {
auto & v = check_value_in(value);
assert(v.type() == nix::nList);
if (ix >= v.listSize()) {
nix_set_err_msg(context, NIX_ERR_KEY, "list index out of bounds");
return nullptr;
}
auto * p = v.listView()[ix];
nix_gc_incref(nullptr, p);
// Note: intentionally NOT calling forceValue() to keep the element lazy
return as_nix_value_ptr(p);
}
NIXC_CATCH_ERRS_NULL
}
nix_value * nix_get_attr_byname(nix_c_context * context, const nix_value * value, EvalState * state, const char * name)
{
if (context)
@@ -345,7 +371,27 @@ nix_value * nix_get_attr_byname(nix_c_context * context, const nix_value * value
if (attr) {
nix_gc_incref(nullptr, attr->value);
state->state.forceValue(*attr->value, nix::noPos);
state->state.waitForAllPaths();
return as_nix_value_ptr(attr->value);
}
nix_set_err_msg(context, NIX_ERR_KEY, "missing attribute");
return nullptr;
}
NIXC_CATCH_ERRS_NULL
}
nix_value *
nix_get_attr_byname_lazy(nix_c_context * context, const nix_value * value, EvalState * state, const char * name)
{
if (context)
context->last_err_code = NIX_OK;
try {
auto & v = check_value_in(value);
assert(v.type() == nix::nAttrs);
nix::Symbol s = state->state.symbols.create(name);
auto attr = v.attrs()->get(s);
if (attr) {
nix_gc_incref(nullptr, attr->value);
// Note: intentionally NOT calling forceValue() to keep the attribute lazy
return as_nix_value_ptr(attr->value);
}
nix_set_err_msg(context, NIX_ERR_KEY, "missing attribute");
@@ -370,13 +416,28 @@ bool nix_has_attr_byname(nix_c_context * context, const nix_value * value, EvalS
NIXC_CATCH_ERRS_RES(false);
}
nix_value * nix_get_attr_byidx(
nix_c_context * context, const nix_value * value, EvalState * state, unsigned int i, const char ** name)
static void collapse_attrset_layer_chain_if_needed(nix::Value & v, EvalState * state)
{
auto & attrs = *v.attrs();
if (attrs.isLayered()) {
auto bindings = state->state.buildBindings(attrs.size());
std::ranges::copy(attrs, std::back_inserter(bindings));
v.mkAttrs(bindings);
}
}
nix_value *
nix_get_attr_byidx(nix_c_context * context, nix_value * value, EvalState * state, unsigned int i, const char ** name)
{
if (context)
context->last_err_code = NIX_OK;
try {
auto & v = check_value_in(value);
collapse_attrset_layer_chain_if_needed(v, state);
if (i >= v.attrs()->size()) {
nix_set_err_msg(context, NIX_ERR_KEY, "attribute index out of bounds");
return nullptr;
}
const nix::Attr & a = (*v.attrs())[i];
*name = state->state.symbols[a.name].c_str();
nix_gc_incref(nullptr, a.value);
@@ -386,13 +447,38 @@ nix_value * nix_get_attr_byidx(
NIXC_CATCH_ERRS_NULL
}
const char *
nix_get_attr_name_byidx(nix_c_context * context, const nix_value * value, EvalState * state, unsigned int i)
nix_value * nix_get_attr_byidx_lazy(
nix_c_context * context, nix_value * value, EvalState * state, unsigned int i, const char ** name)
{
if (context)
context->last_err_code = NIX_OK;
try {
auto & v = check_value_in(value);
collapse_attrset_layer_chain_if_needed(v, state);
if (i >= v.attrs()->size()) {
nix_set_err_msg(context, NIX_ERR_KEY, "attribute index out of bounds (Nix C API contract violation)");
return nullptr;
}
const nix::Attr & a = (*v.attrs())[i];
*name = state->state.symbols[a.name].c_str();
nix_gc_incref(nullptr, a.value);
// Note: intentionally NOT calling forceValue() to keep the attribute lazy
return as_nix_value_ptr(a.value);
}
NIXC_CATCH_ERRS_NULL
}
const char * nix_get_attr_name_byidx(nix_c_context * context, nix_value * value, EvalState * state, unsigned int i)
{
if (context)
context->last_err_code = NIX_OK;
try {
auto & v = check_value_in(value);
collapse_attrset_layer_chain_if_needed(v, state);
if (i >= v.attrs()->size()) {
nix_set_err_msg(context, NIX_ERR_KEY, "attribute index out of bounds (Nix C API contract violation)");
return nullptr;
}
const nix::Attr & a = (*v.attrs())[i];
return state->state.symbols[a.name].c_str();
}
@@ -593,7 +679,7 @@ nix_err nix_bindings_builder_insert(nix_c_context * context, BindingsBuilder * b
context->last_err_code = NIX_OK;
try {
auto & v = check_value_not_null(value);
nix::Symbol s = bb->builder.state.symbols.create(name);
nix::Symbol s = bb->builder.symbols.get().create(name);
bb->builder.insert(s, &v);
}
NIXC_CATCH_ERRS
@@ -652,3 +738,5 @@ const StorePath * nix_realised_string_get_store_path(nix_realised_string * s, si
{
return &s->storePaths[i];
}
} // extern "C"
+86 -9
View File
@@ -265,10 +265,25 @@ ExternalValue * nix_get_external(nix_c_context * context, nix_value * value);
*/
nix_value * nix_get_list_byidx(nix_c_context * context, const nix_value * value, EvalState * state, unsigned int ix);
/** @brief Get an attr by name
/** @brief Get the ix'th element of a list without forcing evaluation of the element
*
* Returns the list element without forcing its evaluation, allowing access to lazy values.
* The list value itself must already be evaluated.
*
* Owned by the GC. Use nix_gc_decref when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect (must be an evaluated list)
* @param[in] state nix evaluator state
* @param[in] ix list element to get
* @return value, NULL in case of errors
*/
nix_value *
nix_get_list_byidx_lazy(nix_c_context * context, const nix_value * value, EvalState * state, unsigned int ix);
/** @brief Get an attr by name
*
* Use nix_gc_decref when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @param[in] state nix evaluator state
* @param[in] name attribute name
@@ -276,6 +291,21 @@ nix_value * nix_get_list_byidx(nix_c_context * context, const nix_value * value,
*/
nix_value * nix_get_attr_byname(nix_c_context * context, const nix_value * value, EvalState * state, const char * name);
/** @brief Get an attribute value by attribute name, without forcing evaluation of the attribute's value
*
* Returns the attribute value without forcing its evaluation, allowing access to lazy values.
* The attribute set value itself must already be evaluated.
*
* Use nix_gc_decref when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect (must be an evaluated attribute set)
* @param[in] state nix evaluator state
* @param[in] name attribute name
* @return value, NULL in case of errors
*/
nix_value *
nix_get_attr_byname_lazy(nix_c_context * context, const nix_value * value, EvalState * state, const char * name);
/** @brief Check if an attribute name exists on a value
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
@@ -285,11 +315,21 @@ nix_value * nix_get_attr_byname(nix_c_context * context, const nix_value * value
*/
bool nix_has_attr_byname(nix_c_context * context, const nix_value * value, EvalState * state, const char * name);
/** @brief Get an attribute by index in the sorted bindings
/** @brief Get an attribute by index
*
* Also gives you the name.
*
* Owned by the GC. Use nix_gc_decref when you're done with the pointer
* Attributes are returned in an unspecified order which is NOT suitable for
* reproducible operations. In Nix's domain, reproducibility is paramount. The caller
* is responsible for sorting the attributes or storing them in an ordered map to
* ensure deterministic behavior in your application.
*
* @note When Nix does sort attributes, which it does for virtually all intermediate
* operations and outputs, it uses byte-wise lexicographic order (equivalent to
* lexicographic order by Unicode scalar value for valid UTF-8). We recommend
* applying this same ordering for consistency.
*
* Use nix_gc_decref when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect
* @param[in] state nix evaluator state
@@ -297,12 +337,50 @@ bool nix_has_attr_byname(nix_c_context * context, const nix_value * value, EvalS
* @param[out] name will store a pointer to the attribute name
* @return value, NULL in case of errors
*/
nix_value * nix_get_attr_byidx(
nix_c_context * context, const nix_value * value, EvalState * state, unsigned int i, const char ** name);
nix_value *
nix_get_attr_byidx(nix_c_context * context, nix_value * value, EvalState * state, unsigned int i, const char ** name);
/** @brief Get an attribute name by index in the sorted bindings
/** @brief Get an attribute by index, without forcing evaluation of the attribute's value
*
* Useful when you want the name but want to avoid evaluation.
* Also gives you the name.
*
* Returns the attribute value without forcing its evaluation, allowing access to lazy values.
* The attribute set value itself must already have been evaluated.
*
* Attributes are returned in an unspecified order which is NOT suitable for
* reproducible operations. In Nix's domain, reproducibility is paramount. The caller
* is responsible for sorting the attributes or storing them in an ordered map to
* ensure deterministic behavior in your application.
*
* @note When Nix does sort attributes, which it does for virtually all intermediate
* operations and outputs, it uses byte-wise lexicographic order (equivalent to
* lexicographic order by Unicode scalar value for valid UTF-8). We recommend
* applying this same ordering for consistency.
*
* Use nix_gc_decref when you're done with the pointer
* @param[out] context Optional, stores error information
* @param[in] value Nix value to inspect (must be an evaluated attribute set)
* @param[in] state nix evaluator state
* @param[in] i attribute index
* @param[out] name will store a pointer to the attribute name
* @return value, NULL in case of errors
*/
nix_value * nix_get_attr_byidx_lazy(
nix_c_context * context, nix_value * value, EvalState * state, unsigned int i, const char ** name);
/** @brief Get an attribute name by index
*
* Returns the attribute name without forcing evaluation of the attribute's value.
*
* Attributes are returned in an unspecified order which is NOT suitable for
* reproducible operations. In Nix's domain, reproducibility is paramount. The caller
* is responsible for sorting the attributes or storing them in an ordered map to
* ensure deterministic behavior in your application.
*
* @note When Nix does sort attributes, which it does for virtually all intermediate
* operations and outputs, it uses byte-wise lexicographic order (equivalent to
* lexicographic order by Unicode scalar value for valid UTF-8). We recommend
* applying this same ordering for consistency.
*
* Owned by the nix EvalState
* @param[out] context Optional, stores error information
@@ -311,8 +389,7 @@ nix_value * nix_get_attr_byidx(
* @param[in] i attribute index
* @return name, NULL in case of errors
*/
const char *
nix_get_attr_name_byidx(nix_c_context * context, const nix_value * value, EvalState * state, unsigned int i);
const char * nix_get_attr_name_byidx(nix_c_context * context, nix_value * value, EvalState * state, unsigned int i);
/**@}*/
/** @name Initializers
@@ -26,11 +26,20 @@ public:
}
protected:
LibExprTest()
LibExprTest(ref<Store> store, auto && makeEvalSettings)
: LibStoreTest()
, evalSettings(makeEvalSettings(readOnlyMode))
, state({}, store, fetchSettings, evalSettings, nullptr)
{
evalSettings.nixPath = {};
}
LibExprTest()
: LibExprTest(openStore("dummy://"), [](bool & readOnlyMode) {
EvalSettings settings{readOnlyMode};
settings.nixPath = {};
return settings;
})
{
}
Value eval(std::string input, bool forceValue = true)
+1
View File
@@ -44,6 +44,7 @@ subdir('nix-meson-build-support/windows-version')
this_library = library(
'nix-expr-test-support',
sources,
soversion : nix_soversion,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
# TODO: Remove `-lrapidcheck` when https://github.com/emil-e/rapidcheck/pull/326
+38
View File
@@ -3,6 +3,7 @@
#include "nix/expr/eval.hh"
#include "nix/expr/tests/libexpr.hh"
#include "nix/util/memory-source-accessor.hh"
namespace nix {
@@ -174,4 +175,41 @@ TEST_F(EvalStateTest, getBuiltin_fail)
ASSERT_THROW(state.getBuiltin("nonexistent"), EvalError);
}
class PureEvalTest : public LibExprTest
{
public:
PureEvalTest()
: LibExprTest(openStore("dummy://", {{"read-only", "false"}}), [](bool & readOnlyMode) {
EvalSettings settings{readOnlyMode};
settings.pureEval = true;
settings.restrictEval = true;
return settings;
})
{
}
};
TEST_F(PureEvalTest, pathExists)
{
ASSERT_THAT(eval("builtins.pathExists /."), IsFalse());
ASSERT_THAT(eval("builtins.pathExists /nix"), IsFalse());
ASSERT_THAT(eval("builtins.pathExists /nix/store"), IsFalse());
{
std::string contents = "Lorem ipsum";
StringSource s{contents};
auto path = state.store->addToStoreFromDump(
s, "source", FileSerialisationMethod::Flat, ContentAddressMethod::Raw::Text, HashAlgorithm::SHA256);
auto printed = store->printStorePath(path);
ASSERT_THROW(eval(fmt("builtins.readFile %s", printed)), RestrictedPathError);
ASSERT_THAT(eval(fmt("builtins.pathExists %s", printed)), IsFalse());
ASSERT_THROW(eval("builtins.readDir /."), RestrictedPathError);
state.allowPath(path); // FIXME: This shouldn't behave this way.
ASSERT_THAT(eval("builtins.readDir /."), IsAttrsOfSize(0));
}
}
} // namespace nix
+2 -2
View File
@@ -54,7 +54,7 @@ TEST_F(JSONValueTest, IntNegative)
TEST_F(JSONValueTest, String)
{
Value v;
v.mkString("test");
v.mkStringNoCopy("test");
ASSERT_EQ(getJSONValue(v), "\"test\"");
}
@@ -62,7 +62,7 @@ TEST_F(JSONValueTest, StringQuotes)
{
Value v;
v.mkString("test\"");
v.mkStringNoCopy("test\"");
ASSERT_EQ(getJSONValue(v), "\"test\\\"\"");
}
+6 -31
View File
@@ -1,40 +1,15 @@
#include <gtest/gtest.h>
#include <cstdlib>
#include "nix/store/globals.hh"
#include "nix/util/logging.hh"
#include "nix/store/tests/test-main.hh"
#include "nix/util/config-global.hh"
using namespace nix;
int main(int argc, char ** argv)
{
if (argc > 1 && std::string_view(argv[1]) == "__build-remote") {
printError("test-build-remote: not supported in libexpr unit tests");
return 1;
}
// Disable build hook. We won't be testing remote builds in these unit tests. If we do, fix the above build hook.
settings.buildHook = {};
#ifdef __linux__ // should match the conditional around sandboxBuildDir declaration.
// When building and testing nix within the host's Nix sandbox, our store dir will be located in the host's
// sandboxBuildDir, e.g.: Host
// storeDir = /nix/store
// sandboxBuildDir = /build
// This process
// storeDir = /build/foo/bar/store
// sandboxBuildDir = /build
// However, we have a rule that the store dir must not be inside the storeDir, so we need to pick a different
// sandboxBuildDir.
settings.sandboxBuildDir = "/test-build-dir-instead-of-usual-build-dir";
#endif
#ifdef __APPLE__
// Avoid this error, when already running in a sandbox:
// sandbox-exec: sandbox_apply: Operation not permitted
settings.sandboxMode = smDisabled;
setEnv("_NIX_TEST_NO_SANDBOX", "1");
#endif
auto res = testMainForBuidingPre(argc, argv);
if (res)
return res;
// For pipe operator tests in trivial.cc
experimentalFeatureSettings.set("experimental-features", "pipe-operators");
+1
View File
@@ -55,6 +55,7 @@ sources = files(
'nix_api_expr.cc',
'nix_api_external.cc',
'nix_api_value.cc',
'nix_api_value_internal.cc',
'primops.cc',
'search-path.cc',
'trivial.cc',
+88 -16
View File
@@ -1,7 +1,5 @@
#include "nix_api_store.h"
#include "nix_api_store_internal.h"
#include "nix_api_util.h"
#include "nix_api_util_internal.h"
#include "nix_api_expr.h"
#include "nix_api_value.h"
@@ -151,8 +149,8 @@ TEST_F(nix_api_expr_test, nix_expr_realise_context_bad_value)
assert_ctx_ok();
auto r = nix_string_realise(ctx, state, value, false);
ASSERT_EQ(nullptr, r);
ASSERT_EQ(ctx->last_err_code, NIX_ERR_NIX_ERROR);
ASSERT_THAT(ctx->last_err, testing::Optional(testing::HasSubstr("cannot coerce")));
ASSERT_EQ(nix_err_code(ctx), NIX_ERR_NIX_ERROR);
ASSERT_THAT(nix_err_msg(nullptr, ctx, nullptr), testing::HasSubstr("cannot coerce"));
}
TEST_F(nix_api_expr_test, nix_expr_realise_context_bad_build)
@@ -168,8 +166,8 @@ TEST_F(nix_api_expr_test, nix_expr_realise_context_bad_build)
assert_ctx_ok();
auto r = nix_string_realise(ctx, state, value, false);
ASSERT_EQ(nullptr, r);
ASSERT_EQ(ctx->last_err_code, NIX_ERR_NIX_ERROR);
ASSERT_THAT(ctx->last_err, testing::Optional(testing::HasSubstr("failed with exit code 1")));
ASSERT_EQ(nix_err_code(ctx), NIX_ERR_NIX_ERROR);
ASSERT_THAT(nix_err_msg(nullptr, ctx, nullptr), testing::HasSubstr("failed with exit code 1"));
}
TEST_F(nix_api_expr_test, nix_expr_realise_context)
@@ -381,12 +379,11 @@ TEST_F(nix_api_expr_test, nix_expr_primop_bad_no_return)
nix_value * result = nix_alloc_value(ctx, state);
assert_ctx_ok();
nix_value_call(ctx, state, primopValue, three, result);
ASSERT_EQ(ctx->last_err_code, NIX_ERR_NIX_ERROR);
ASSERT_EQ(nix_err_code(ctx), NIX_ERR_NIX_ERROR);
ASSERT_THAT(
ctx->last_err,
testing::Optional(
testing::HasSubstr("Implementation error in custom function: return value was not initialized")));
ASSERT_THAT(ctx->last_err, testing::Optional(testing::HasSubstr("badNoReturn")));
nix_err_msg(nullptr, ctx, nullptr),
testing::HasSubstr("Implementation error in custom function: return value was not initialized"));
ASSERT_THAT(nix_err_msg(nullptr, ctx, nullptr), testing::HasSubstr("badNoReturn"));
}
static void primop_bad_return_thunk(
@@ -419,12 +416,60 @@ TEST_F(nix_api_expr_test, nix_expr_primop_bad_return_thunk)
assert_ctx_ok();
NIX_VALUE_CALL(ctx, state, result, primopValue, toString, four);
ASSERT_EQ(ctx->last_err_code, NIX_ERR_NIX_ERROR);
ASSERT_EQ(nix_err_code(ctx), NIX_ERR_NIX_ERROR);
ASSERT_THAT(
ctx->last_err,
testing::Optional(
testing::HasSubstr("Implementation error in custom function: return value must not be a thunk")));
ASSERT_THAT(ctx->last_err, testing::Optional(testing::HasSubstr("badReturnThunk")));
nix_err_msg(nullptr, ctx, nullptr),
testing::HasSubstr("Implementation error in custom function: return value must not be a thunk"));
ASSERT_THAT(nix_err_msg(nullptr, ctx, nullptr), testing::HasSubstr("badReturnThunk"));
}
static void primop_with_nix_err_key(
void * user_data, nix_c_context * context, EvalState * state, nix_value ** args, nix_value * ret)
{
nix_set_err_msg(context, NIX_ERR_KEY, "Test error from primop");
}
TEST_F(nix_api_expr_test, nix_expr_primop_nix_err_key_conversion)
{
// Test that NIX_ERR_KEY from a custom primop gets converted to a generic EvalError
//
// RATIONALE: NIX_ERR_KEY must not be propagated from custom primops because it would
// create semantic confusion. NIX_ERR_KEY indicates missing keys/indices in C API functions
// (like nix_get_attr_byname, nix_get_list_byidx). If custom primops could return NIX_ERR_KEY,
// an evaluation error would be indistinguishable from an actual missing attribute.
//
// For example, if nix_get_attr_byname returned NIX_ERR_KEY when the attribute is present
// but the value evaluation fails, callers expecting NIX_ERR_KEY to mean "missing attribute"
// would incorrectly handle evaluation failures as missing attributes. In places where
// missing attributes are tolerated (like optional attributes), this would cause the
// program to continue after swallowing the error, leading to silent failures.
PrimOp * primop = nix_alloc_primop(
ctx, primop_with_nix_err_key, 1, "testErrorPrimop", nullptr, "a test primop that sets NIX_ERR_KEY", nullptr);
assert_ctx_ok();
nix_value * primopValue = nix_alloc_value(ctx, state);
assert_ctx_ok();
nix_init_primop(ctx, primopValue, primop);
assert_ctx_ok();
nix_value * arg = nix_alloc_value(ctx, state);
assert_ctx_ok();
nix_init_int(ctx, arg, 42);
assert_ctx_ok();
nix_value * result = nix_alloc_value(ctx, state);
assert_ctx_ok();
nix_value_call(ctx, state, primopValue, arg, result);
// Verify that NIX_ERR_KEY gets converted to NIX_ERR_NIX_ERROR (generic evaluation error)
ASSERT_EQ(nix_err_code(ctx), NIX_ERR_NIX_ERROR);
ASSERT_THAT(nix_err_msg(nullptr, ctx, nullptr), testing::HasSubstr("Error from custom function"));
ASSERT_THAT(nix_err_msg(nullptr, ctx, nullptr), testing::HasSubstr("Test error from primop"));
ASSERT_THAT(nix_err_msg(nullptr, ctx, nullptr), testing::HasSubstr("testErrorPrimop"));
// Clean up
nix_gc_decref(ctx, primopValue);
nix_gc_decref(ctx, arg);
nix_gc_decref(ctx, result);
}
TEST_F(nix_api_expr_test, nix_value_call_multi_no_args)
@@ -441,4 +486,31 @@ TEST_F(nix_api_expr_test, nix_value_call_multi_no_args)
assert_ctx_ok();
ASSERT_EQ(3, rInt);
}
TEST_F(nix_api_expr_test, nix_expr_attrset_update)
{
nix_expr_eval_from_string(ctx, state, "{ a = 0; b = 2; } // { a = 1; b = 3; } // { a = 2; }", ".", value);
assert_ctx_ok();
ASSERT_EQ(nix_get_attrs_size(ctx, value), 2);
assert_ctx_ok();
std::array<std::pair<std::string_view, nix_value *>, 2> values;
for (unsigned int i = 0; i < 2; ++i) {
const char * name;
values[i].second = nix_get_attr_byidx(ctx, value, state, i, &name);
assert_ctx_ok();
values[i].first = name;
}
std::sort(values.begin(), values.end(), [](const auto & lhs, const auto & rhs) { return lhs.first < rhs.first; });
nix_value * a = values[0].second;
ASSERT_EQ("a", values[0].first);
ASSERT_EQ(nix_get_int(ctx, a), 2);
assert_ctx_ok();
nix_value * b = values[1].second;
ASSERT_EQ("b", values[1].first);
ASSERT_EQ(nix_get_int(ctx, b), 3);
assert_ctx_ok();
}
} // namespace nixC
+1 -4
View File
@@ -1,9 +1,6 @@
#include "nix_api_store.h"
#include "nix_api_store_internal.h"
#include "nix_api_util.h"
#include "nix_api_util_internal.h"
#include "nix_api_expr.h"
#include "nix_api_expr_internal.h"
#include "nix_api_value.h"
#include "nix_api_external.h"
@@ -39,7 +36,7 @@ private:
std::string type_string = "nix-external<MyExternalValueDesc( ";
type_string += std::to_string(obj->_x);
type_string += " )>";
res->str = &*type_string.begin();
nix_set_string_return(res, &*type_string.begin());
}
};
+334 -14
View File
@@ -1,10 +1,7 @@
#include "nix_api_store.h"
#include "nix_api_store_internal.h"
#include "nix_api_util.h"
#include "nix_api_util_internal.h"
#include "nix_api_expr.h"
#include "nix_api_value.h"
#include "nix_api_expr_internal.h"
#include "nix/expr/tests/nix_api_expr.hh"
#include "nix/util/tests/string_callback.hh"
@@ -16,14 +13,6 @@
namespace nixC {
TEST_F(nix_api_expr_test, as_nix_value_ptr)
{
// nix_alloc_value casts nix::Value to nix_value
// It should be obvious from the decl that that works, but if it doesn't,
// the whole implementation would be utterly broken.
ASSERT_EQ(sizeof(nix::Value), sizeof(nix_value));
}
TEST_F(nix_api_expr_test, nix_value_get_int_invalid)
{
ASSERT_EQ(0, nix_get_int(ctx, nullptr));
@@ -173,6 +162,114 @@ TEST_F(nix_api_expr_test, nix_build_and_init_list)
nix_gc_decref(ctx, intValue);
}
TEST_F(nix_api_expr_test, nix_get_list_byidx_large_indices)
{
// Create a small list to test extremely large out-of-bounds access
ListBuilder * builder = nix_make_list_builder(ctx, state, 2);
nix_value * intValue = nix_alloc_value(ctx, state);
nix_init_int(ctx, intValue, 42);
nix_list_builder_insert(ctx, builder, 0, intValue);
nix_list_builder_insert(ctx, builder, 1, intValue);
nix_make_list(ctx, builder, value);
nix_list_builder_free(builder);
// Test extremely large indices that would definitely crash without bounds checking
ASSERT_EQ(nullptr, nix_get_list_byidx(ctx, value, state, 1000000));
ASSERT_EQ(NIX_ERR_KEY, nix_err_code(ctx));
ASSERT_EQ(nullptr, nix_get_list_byidx(ctx, value, state, UINT_MAX / 2));
ASSERT_EQ(NIX_ERR_KEY, nix_err_code(ctx));
ASSERT_EQ(nullptr, nix_get_list_byidx(ctx, value, state, UINT_MAX / 2 + 1000000));
ASSERT_EQ(NIX_ERR_KEY, nix_err_code(ctx));
// Clean up
nix_gc_decref(ctx, intValue);
}
TEST_F(nix_api_expr_test, nix_get_list_byidx_lazy)
{
// Create a list with a throwing lazy element, an already-evaluated int, and a lazy function call
// 1. Throwing lazy element - create a function application thunk that will throw when forced
nix_value * throwingFn = nix_alloc_value(ctx, state);
nix_value * throwingValue = nix_alloc_value(ctx, state);
nix_expr_eval_from_string(
ctx,
state,
R"(
_: throw "This should not be evaluated by the lazy accessor"
)",
"<test>",
throwingFn);
assert_ctx_ok();
nix_init_apply(ctx, throwingValue, throwingFn, throwingFn);
assert_ctx_ok();
// 2. Already evaluated int (not lazy)
nix_value * intValue = nix_alloc_value(ctx, state);
nix_init_int(ctx, intValue, 42);
assert_ctx_ok();
// 3. Lazy function application that would compute increment 5 = 6
nix_value * lazyApply = nix_alloc_value(ctx, state);
nix_value * incrementFn = nix_alloc_value(ctx, state);
nix_value * argFive = nix_alloc_value(ctx, state);
nix_expr_eval_from_string(ctx, state, "x: x + 1", "<test>", incrementFn);
assert_ctx_ok();
nix_init_int(ctx, argFive, 5);
// Create a lazy application: (x: x + 1) 5
nix_init_apply(ctx, lazyApply, incrementFn, argFive);
assert_ctx_ok();
ListBuilder * builder = nix_make_list_builder(ctx, state, 3);
nix_list_builder_insert(ctx, builder, 0, throwingValue);
nix_list_builder_insert(ctx, builder, 1, intValue);
nix_list_builder_insert(ctx, builder, 2, lazyApply);
nix_make_list(ctx, builder, value);
nix_list_builder_free(builder);
// Test 1: Lazy accessor should return the throwing element without forcing evaluation
nix_value * lazyThrowingElement = nix_get_list_byidx_lazy(ctx, value, state, 0);
assert_ctx_ok();
ASSERT_NE(nullptr, lazyThrowingElement);
// Verify the element is still lazy by checking that forcing it throws
nix_value_force(ctx, state, lazyThrowingElement);
assert_ctx_err();
ASSERT_THAT(
nix_err_msg(nullptr, ctx, nullptr), testing::HasSubstr("This should not be evaluated by the lazy accessor"));
// Test 2: Lazy accessor should return the already-evaluated int
nix_value * intElement = nix_get_list_byidx_lazy(ctx, value, state, 1);
assert_ctx_ok();
ASSERT_NE(nullptr, intElement);
ASSERT_EQ(42, nix_get_int(ctx, intElement));
// Test 3: Lazy accessor should return the lazy function application without forcing
nix_value * lazyFunctionElement = nix_get_list_byidx_lazy(ctx, value, state, 2);
assert_ctx_ok();
ASSERT_NE(nullptr, lazyFunctionElement);
// Force the lazy function application - should compute 5 + 1 = 6
nix_value_force(ctx, state, lazyFunctionElement);
assert_ctx_ok();
ASSERT_EQ(6, nix_get_int(ctx, lazyFunctionElement));
// Clean up
nix_gc_decref(ctx, throwingFn);
nix_gc_decref(ctx, throwingValue);
nix_gc_decref(ctx, intValue);
nix_gc_decref(ctx, lazyApply);
nix_gc_decref(ctx, incrementFn);
nix_gc_decref(ctx, argFive);
nix_gc_decref(ctx, lazyThrowingElement);
nix_gc_decref(ctx, intElement);
nix_gc_decref(ctx, lazyFunctionElement);
}
TEST_F(nix_api_expr_test, nix_build_and_init_attr_invalid)
{
ASSERT_EQ(nullptr, nix_get_attr_byname(ctx, nullptr, state, 0));
@@ -255,6 +352,225 @@ TEST_F(nix_api_expr_test, nix_build_and_init_attr)
free(out_name);
}
TEST_F(nix_api_expr_test, nix_get_attr_byidx_large_indices)
{
// Create a small attribute set to test extremely large out-of-bounds access
const char ** out_name = (const char **) malloc(sizeof(char *));
BindingsBuilder * builder = nix_make_bindings_builder(ctx, state, 2);
nix_value * intValue = nix_alloc_value(ctx, state);
nix_init_int(ctx, intValue, 42);
nix_bindings_builder_insert(ctx, builder, "test", intValue);
nix_make_attrs(ctx, value, builder);
nix_bindings_builder_free(builder);
// Test extremely large indices that would definitely crash without bounds checking
ASSERT_EQ(nullptr, nix_get_attr_byidx(ctx, value, state, 1000000, out_name));
ASSERT_EQ(NIX_ERR_KEY, nix_err_code(ctx));
ASSERT_EQ(nullptr, nix_get_attr_byidx(ctx, value, state, UINT_MAX / 2, out_name));
ASSERT_EQ(NIX_ERR_KEY, nix_err_code(ctx));
ASSERT_EQ(nullptr, nix_get_attr_byidx(ctx, value, state, UINT_MAX / 2 + 1000000, out_name));
ASSERT_EQ(NIX_ERR_KEY, nix_err_code(ctx));
// Test nix_get_attr_name_byidx with large indices too
ASSERT_EQ(nullptr, nix_get_attr_name_byidx(ctx, value, state, 1000000));
ASSERT_EQ(NIX_ERR_KEY, nix_err_code(ctx));
ASSERT_EQ(nullptr, nix_get_attr_name_byidx(ctx, value, state, UINT_MAX / 2));
ASSERT_EQ(NIX_ERR_KEY, nix_err_code(ctx));
ASSERT_EQ(nullptr, nix_get_attr_name_byidx(ctx, value, state, UINT_MAX / 2 + 1000000));
ASSERT_EQ(NIX_ERR_KEY, nix_err_code(ctx));
// Clean up
nix_gc_decref(ctx, intValue);
free(out_name);
}
TEST_F(nix_api_expr_test, nix_get_attr_byname_lazy)
{
// Create an attribute set with a throwing lazy attribute, an already-evaluated int, and a lazy function call
// 1. Throwing lazy element - create a function application thunk that will throw when forced
nix_value * throwingFn = nix_alloc_value(ctx, state);
nix_value * throwingValue = nix_alloc_value(ctx, state);
nix_expr_eval_from_string(
ctx,
state,
R"(
_: throw "This should not be evaluated by the lazy accessor"
)",
"<test>",
throwingFn);
assert_ctx_ok();
nix_init_apply(ctx, throwingValue, throwingFn, throwingFn);
assert_ctx_ok();
// 2. Already evaluated int (not lazy)
nix_value * intValue = nix_alloc_value(ctx, state);
nix_init_int(ctx, intValue, 42);
assert_ctx_ok();
// 3. Lazy function application that would compute increment 7 = 8
nix_value * lazyApply = nix_alloc_value(ctx, state);
nix_value * incrementFn = nix_alloc_value(ctx, state);
nix_value * argSeven = nix_alloc_value(ctx, state);
nix_expr_eval_from_string(ctx, state, "x: x + 1", "<test>", incrementFn);
assert_ctx_ok();
nix_init_int(ctx, argSeven, 7);
// Create a lazy application: (x: x + 1) 7
nix_init_apply(ctx, lazyApply, incrementFn, argSeven);
assert_ctx_ok();
BindingsBuilder * builder = nix_make_bindings_builder(ctx, state, 3);
nix_bindings_builder_insert(ctx, builder, "throwing", throwingValue);
nix_bindings_builder_insert(ctx, builder, "normal", intValue);
nix_bindings_builder_insert(ctx, builder, "lazy", lazyApply);
nix_make_attrs(ctx, value, builder);
nix_bindings_builder_free(builder);
// Test 1: Lazy accessor should return the throwing attribute without forcing evaluation
nix_value * lazyThrowingAttr = nix_get_attr_byname_lazy(ctx, value, state, "throwing");
assert_ctx_ok();
ASSERT_NE(nullptr, lazyThrowingAttr);
// Verify the attribute is still lazy by checking that forcing it throws
nix_value_force(ctx, state, lazyThrowingAttr);
assert_ctx_err();
ASSERT_THAT(
nix_err_msg(nullptr, ctx, nullptr), testing::HasSubstr("This should not be evaluated by the lazy accessor"));
// Test 2: Lazy accessor should return the already-evaluated int
nix_value * intAttr = nix_get_attr_byname_lazy(ctx, value, state, "normal");
assert_ctx_ok();
ASSERT_NE(nullptr, intAttr);
ASSERT_EQ(42, nix_get_int(ctx, intAttr));
// Test 3: Lazy accessor should return the lazy function application without forcing
nix_value * lazyFunctionAttr = nix_get_attr_byname_lazy(ctx, value, state, "lazy");
assert_ctx_ok();
ASSERT_NE(nullptr, lazyFunctionAttr);
// Force the lazy function application - should compute 7 + 1 = 8
nix_value_force(ctx, state, lazyFunctionAttr);
assert_ctx_ok();
ASSERT_EQ(8, nix_get_int(ctx, lazyFunctionAttr));
// Test 4: Missing attribute should return NULL with NIX_ERR_KEY
nix_value * missingAttr = nix_get_attr_byname_lazy(ctx, value, state, "nonexistent");
ASSERT_EQ(nullptr, missingAttr);
ASSERT_EQ(NIX_ERR_KEY, nix_err_code(ctx));
// Clean up
nix_gc_decref(ctx, throwingFn);
nix_gc_decref(ctx, throwingValue);
nix_gc_decref(ctx, intValue);
nix_gc_decref(ctx, lazyApply);
nix_gc_decref(ctx, incrementFn);
nix_gc_decref(ctx, argSeven);
nix_gc_decref(ctx, lazyThrowingAttr);
nix_gc_decref(ctx, intAttr);
nix_gc_decref(ctx, lazyFunctionAttr);
}
TEST_F(nix_api_expr_test, nix_get_attr_byidx_lazy)
{
// Create an attribute set with a throwing lazy attribute, an already-evaluated int, and a lazy function call
// 1. Throwing lazy element - create a function application thunk that will throw when forced
nix_value * throwingFn = nix_alloc_value(ctx, state);
nix_value * throwingValue = nix_alloc_value(ctx, state);
nix_expr_eval_from_string(
ctx,
state,
R"(
_: throw "This should not be evaluated by the lazy accessor"
)",
"<test>",
throwingFn);
assert_ctx_ok();
nix_init_apply(ctx, throwingValue, throwingFn, throwingFn);
assert_ctx_ok();
// 2. Already evaluated int (not lazy)
nix_value * intValue = nix_alloc_value(ctx, state);
nix_init_int(ctx, intValue, 99);
assert_ctx_ok();
// 3. Lazy function application that would compute increment 10 = 11
nix_value * lazyApply = nix_alloc_value(ctx, state);
nix_value * incrementFn = nix_alloc_value(ctx, state);
nix_value * argTen = nix_alloc_value(ctx, state);
nix_expr_eval_from_string(ctx, state, "x: x + 1", "<test>", incrementFn);
assert_ctx_ok();
nix_init_int(ctx, argTen, 10);
// Create a lazy application: (x: x + 1) 10
nix_init_apply(ctx, lazyApply, incrementFn, argTen);
assert_ctx_ok();
BindingsBuilder * builder = nix_make_bindings_builder(ctx, state, 3);
nix_bindings_builder_insert(ctx, builder, "a_throwing", throwingValue);
nix_bindings_builder_insert(ctx, builder, "b_normal", intValue);
nix_bindings_builder_insert(ctx, builder, "c_lazy", lazyApply);
nix_make_attrs(ctx, value, builder);
nix_bindings_builder_free(builder);
// Proper usage: first get the size and gather all attributes into a map
unsigned int attrCount = nix_get_attrs_size(ctx, value);
assert_ctx_ok();
ASSERT_EQ(3u, attrCount);
// Gather all attributes into a map (proper contract usage)
std::map<std::string, nix_value *> attrMap;
const char * name;
for (unsigned int i = 0; i < attrCount; i++) {
nix_value * attr = nix_get_attr_byidx_lazy(ctx, value, state, i, &name);
assert_ctx_ok();
ASSERT_NE(nullptr, attr);
attrMap[std::string(name)] = attr;
}
// Now test the gathered attributes
ASSERT_EQ(3u, attrMap.size());
ASSERT_TRUE(attrMap.count("a_throwing"));
ASSERT_TRUE(attrMap.count("b_normal"));
ASSERT_TRUE(attrMap.count("c_lazy"));
// Test 1: Throwing attribute should be lazy
nix_value * throwingAttr = attrMap["a_throwing"];
nix_value_force(ctx, state, throwingAttr);
assert_ctx_err();
ASSERT_THAT(
nix_err_msg(nullptr, ctx, nullptr), testing::HasSubstr("This should not be evaluated by the lazy accessor"));
// Test 2: Normal attribute should be already evaluated
nix_value * normalAttr = attrMap["b_normal"];
ASSERT_EQ(99, nix_get_int(ctx, normalAttr));
// Test 3: Lazy function should compute when forced
nix_value * lazyAttr = attrMap["c_lazy"];
nix_value_force(ctx, state, lazyAttr);
assert_ctx_ok();
ASSERT_EQ(11, nix_get_int(ctx, lazyAttr));
// Clean up
nix_gc_decref(ctx, throwingFn);
nix_gc_decref(ctx, throwingValue);
nix_gc_decref(ctx, intValue);
nix_gc_decref(ctx, lazyApply);
nix_gc_decref(ctx, incrementFn);
nix_gc_decref(ctx, argTen);
for (auto & pair : attrMap) {
nix_gc_decref(ctx, pair.second);
}
}
TEST_F(nix_api_expr_test, nix_value_init)
{
// Setup
@@ -320,8 +636,10 @@ TEST_F(nix_api_expr_test, nix_value_init_apply_error)
// Evaluate it
nix_value_force(ctx, state, v);
ASSERT_EQ(ctx->last_err_code, NIX_ERR_NIX_ERROR);
ASSERT_THAT(ctx->last_err.value(), testing::HasSubstr("attempt to call something which is not a function but"));
ASSERT_EQ(nix_err_code(ctx), NIX_ERR_NIX_ERROR);
ASSERT_THAT(
nix_err_msg(nullptr, ctx, nullptr),
testing::HasSubstr("attempt to call something which is not a function but"));
// Clean up
nix_gc_decref(ctx, some_string);
@@ -380,7 +698,9 @@ TEST_F(nix_api_expr_test, nix_value_init_apply_lazy_arg)
// nix_get_attr_byname isn't lazy (it could have been) so it will throw the exception
nix_value * foo = nix_get_attr_byname(ctx, r, state, "foo");
ASSERT_EQ(nullptr, foo);
ASSERT_THAT(ctx->last_err.value(), testing::HasSubstr("error message for test case nix_value_init_apply_lazy_arg"));
ASSERT_THAT(
nix_err_msg(nullptr, ctx, nullptr),
testing::HasSubstr("error message for test case nix_value_init_apply_lazy_arg"));
// Clean up
nix_gc_decref(ctx, f);
@@ -0,0 +1,25 @@
#include "nix_api_store.h"
#include "nix_api_util.h"
#include "nix_api_expr.h"
#include "nix_api_value.h"
#include "nix_api_expr_internal.h"
#include "nix/expr/tests/nix_api_expr.hh"
#include "nix/util/tests/string_callback.hh"
#include <gmock/gmock.h>
#include <cstddef>
#include <cstdlib>
#include <gtest/gtest.h>
namespace nixC {
TEST_F(nix_api_expr_test, as_nix_value_ptr)
{
// nix_alloc_value casts nix::Value to nix_value
// It should be obvious from the decl that that works, but if it doesn't,
// the whole implementation would be utterly broken.
ASSERT_EQ(sizeof(nix::Value), sizeof(nix_value));
}
} // namespace nixC
Loaded 100 of 674 files, more files were not shown because too many files have changed in this diff. Show more