Compare commits

...
351 Commits
Author SHA1 Message Date
Bernardo Meurer Costa a565e173bf ci: Add Coverity Scan workflow for static analysis 2025-10-30 13:23:11 +01:00
John Ericson 00f4a860e7 Merge pull request #14400 from obsidiansystems/json-schema-derivation-output
Enable JSON schema testing for derivation outputs
2025-10-29 19:40:29 +00:00
Sergei Zimmerman da637a05da Merge pull request #14410 from bryango/patch-1
zsh/completion: put compdef on first line
2025-10-29 15:24:02 +00:00
bryango 956fffdd6f zsh/completion: put compdef on first line
Some zsh setups (including mine) do not load the
completion if `#compdef` is not on the first line.

So we move the `# shellcheck` comment to the
second line to avoid this issue.
2025-10-29 18:09:42 +08:00
John Ericson bac41d6989 Merge pull request #14289 from obsidiansystems/fix-14287
Fix issue #14287
2025-10-29 07:17:59 +00:00
John EricsonandEelco Dolstra de192794c9 Fix issue #14287
The test added in the previous commit now passes.

Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2025-10-29 02:15:46 -04:00
John EricsonandSergei Zimmerman 246dbe1c05 Regression test for issue #14287
This will currently fail, until the bug is fixed.

Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-29 02:15:41 -04:00
Sergei Zimmerman 194c21fc82 Merge pull request #14407 from NixOS/fix-upload-put-http
libstore/filetransfer: Add HttpMethod::PUT
2025-10-29 03:24:10 +00:00
Sergei Zimmerman e08853a67c Merge pull request #14406 from NixOS/better-error-message
libstore/http-binary-cache-store: Improve error messages in HttpBinar…
2025-10-29 03:23:49 +00:00
Sergei Zimmerman ae49074548 libstore/filetransfer: Add HttpMethod::PUT
This got lost in f1968ea38e and
now we had incorrect logs that confused "downloading" when we were
in fact "uploading" things.
2025-10-29 02:48:26 +03:00
Sergei Zimmerman f1d4fab1e5 Merge pull request #14405 from obsidiansystems/json-schema-store-path
Create JSON Schema for Store Paths
2025-10-28 23:24:05 +00:00
Sergei Zimmerman c874e7071b libstore/http-binary-cache-store: Improve error messages in HttpBinaryCacheStore::upsertFile
Now the error message doesn't cram everything into a single line and we now instead get:

error:
       … while uploading to HTTP binary cache at 's3://my-cache?endpoint=http://localhost:9000?compression%3Dzstd&region=eu-west-1'

       error: unable to download 'http://localhost:9000/my-cache/nar/1125zqba8cx8wbfa632vy458a3j3xja0qpcqafsfdildyl9dqa7x.nar.xz': Operation was aborted by an application callback (42)
2025-10-29 02:05:14 +03:00
John Ericson c67966418f Create JSON Schema for Store Paths
We immediately use this in the JSON schemas for Derivation and Deriving
Path, but we cannot yet use it in Store Object Info because those paths
*do* include the store dir currently.
2025-10-28 17:22:51 -04:00
John Ericson be2572ed8d Make inputDrvs JSON schema more precise
It now captures the stable non-recursive format (just an output set) and
the unstable recursive form for dynamic derivations.
2025-10-28 17:22:30 -04:00
Sergei Zimmerman be99a1c6bb Merge pull request #14404 from Mic92/test-settings
coderabbit: disable high_level_summary/poem/github status
2025-10-28 21:10:56 +00:00
Jörg Thalheim fe8cdbc3e4 coderabbit: disable high_level_summary/poem/github status/sequence_diagrams 2025-10-28 22:09:05 +01:00
Sergei Zimmerman 70176ed317 Merge pull request #14402 from Mic92/test-settings
coderabbit: don't show review status
2025-10-28 20:46:31 +00:00
Jörg Thalheim 84a5bee424 coderabbit: disable reporting review status 2025-10-28 21:45:30 +01:00
John Ericson e3246301a6 Enable JSON schema testing for derivation outputs
I figured out what the problem was: the fragment needs to start with a
`/`.
2025-10-28 16:07:44 -04:00
Sergei Zimmerman d4c69c7b8f Merge pull request #14398 from roberth/quiet-coderabbit 2025-10-28 18:41:43 +00:00
Robert Hensing f5aafbd6ed .coderabbit.yaml: Disable auto-review 2025-10-28 19:39:04 +01:00
Eelco Dolstra 1a4ad0706b Merge pull request #14394 from me-and/no-print-dead-space-usage
docs: remove incorrect claim re gc --print-dead
2025-10-28 15:10:54 +00:00
Adam Dinwoodie 972915cabd docs: remove incorrect claim re gc --print-dead
Per #7591, the `nix-store --gc --print-dead` command does not provide
any feedback about the amount of disk space that is used by dead store
paths.  It looks like this has been the case since 7ab68961e (* Garbage
collector: added an option `--use-atime' to delete paths in...,
2008-09-17).

Update the nix-store documentation to remove the claim that this is
function that `nix-store --gc --print-dead` performs.
2025-10-28 09:47:25 +00:00
Jörg Thalheim dd0d006517 Merge pull request #14375 from lovesegfault/nix-s3-upload-part
feat(libstore/s3-binary-cache-store): implement `uploadPart()`
2025-10-27 22:40:00 +00:00
John Ericson 1d3f0ca22e Merge pull request #14383 from obsidiansystems/misc-cleanups
Two misc cleanups
2025-10-27 22:16:37 +00:00
Bernardo Meurer Costa c592090fff feat(libstore/s3-binary-cache-store): implement uploadPart()
Implement `uploadPart()` for uploading individual parts in S3 multipart
uploads:

- Constructs URL with `?partNumber=N&uploadId=ID` query parameters
- Uploads chunk data with `application/octet-stream` mime type
- Extracts and returns `ETag` from response
2025-10-27 21:09:39 +00:00
Bernardo Meurer Costa 4b6d07d642 feat(libstore/s3-binary-cache-store): implement createMultipartUpload()
POST to key with `?uploads` query parameter, optionally set
`Content-Encoding` header, parse `uploadId` from XML response using
regex
2025-10-27 21:07:29 +00:00
John Ericson e177f42536 Merge pull request #14379 from Radvendii/exprlist-alloc
libexpr: store ExprList data in Exprs::alloc
2025-10-27 21:04:45 +00:00
John Ericson ac8b1efcf9 Merge pull request #14379 from Radvendii/exprlist-alloc
libexpr: store ExprList data in Exprs::alloc
2025-10-27 21:04:45 +00:00
John Ericson aa4106fd68 Merge pull request #14360 from lovesegfault/scan-for-references-detailed
feat(libstore): add scanForReferencesDeep and use it for why-depends
2025-10-27 20:38:10 +00:00
John Ericson 7f1d92793e Merge pull request #14360 from lovesegfault/scan-for-references-detailed
feat(libstore): add scanForReferencesDeep and use it for why-depends
2025-10-27 20:38:10 +00:00
John Ericson 234f029940 Add consuming ref <-> std::share_ptr methods/ctrs
This can help churning ref counts when we don't need to.
2025-10-27 16:23:43 -04:00
John Ericson dd716dc9be Create default Store::narFromPath implementation in terms of getFSAccessor
This is a good default (the methods that allow for an arbitrary choice
of source accessor are generally preferable both to implement and to
use). And it also pays its way by allowing us to delete *both* the
`DummyStore` and `LocalStore` implementations.
2025-10-27 15:57:26 -04:00
John Ericson ea17cc1b57 Merge pull request #14376 from lovesegfault/nix-s3-abort-multipart
feat(libstore/s3-binary-cache-store): implement `abortMultipartUpload()`
2025-10-27 19:52:36 +00:00
John Ericson 0c1be3aabe Merge pull request #14309 from obsidiansystems/json-schema-content-address
` nlohmann::json` instance and JSON Schema for `ContentAddress`
2025-10-27 19:52:19 +00:00
John Ericson 1c41e07b46 Merge pull request #14385 from lovesegfault/ci-concurrency-group
ci: cancel previous workflow runs on PR updates
2025-10-27 17:35:50 -04:00
Bernardo Meurer Costa ad664ce64e ci: cancel previous workflow runs on PR updates
Add concurrency group configuration to the CI workflow to automatically
cancel outdated runs when a PR receives new commits or is force-pushed.
This prevents wasting CI resources on superseded code.
2025-10-27 20:56:56 +00:00
John Ericson 6ca3434cac Merge pull request #14309 from obsidiansystems/json-schema-content-address
` nlohmann::json` instance and JSON Schema for `ContentAddress`
2025-10-27 19:52:19 +00:00
Bernardo Meurer Costa 6129aee988 refactor(nix/why-depends): use scanForReferencesDeep for --precise mode
Replaces manual tree-walking and reference scanning with the new
scanForReferencesDeep function.
2025-10-27 19:14:49 +00:00
Bernardo Meurer Costa 5e220271e2 feat(libstore): add scanForReferencesDeep for per-file reference tracking
Introduces `scanForReferencesDeep` to provide per-file granularity when
scanning for store path references, enabling better diagnostics for
cycle detection and `nix why-depends --precise`.
2025-10-27 19:14:49 +00:00
John Ericson 8e6b69de54 Merge pull request #14378 from Radvendii/parser-improvements
parser.y: remove some unnecessary copies
2025-10-27 19:11:11 +00:00
Bernardo Meurer Costa 3915b3a111 feat(libstore/s3-binary-cache-store): implement abortMultipartUpload()
Implement `abortMultipartUpload()` for cleaning up incomplete multipart
uploads on error:

- Constructs URL with `?uploadId=ID` query parameter
- Issues `DELETE` request to abort the multipart upload
2025-10-27 18:56:52 +00:00
Jörg Thalheim c5515bb22e Merge pull request #14364 from MarcelCoding/human-sizes
diff-closures: print sizes with dynamic unit
2025-10-27 18:49:23 +00:00
John EricsonandRobert Hensing 91b69e9e70 nlohmann::json instance and JSON Schema for ContentAddress
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2025-10-27 14:47:50 -04:00
Taeer Bar-Yam 9e9dfe36df libexpr: store ExprList data in Exprs::alloc 2025-10-27 19:38:03 +01:00
Taeer Bar-Yam 50e8d17f3c parser.y: use emplace_back() for vector<AttrName> 2025-10-27 19:30:32 +01:00
Taeer Bar-Yam ef8dd58d9b parser.y: use std::move() to avoid unnecessary copies
With #14314, in some places in the parser we started using C++ objects
directly rather than pointers. In those places lines like `$$ = $1` now
imply a copy when we don't need one. This commit changes those to `$$ =
std::move($1)` to avoid those copies.
2025-10-27 19:30:32 +01:00
John Ericson 91ed3701fe Merge pull request #14377 from lovesegfault/makerequest-stringview
refactor(libstore): use string_view in HttpBinaryCacheStore::makeRequest
2025-10-27 17:58:53 +00:00
Jörg Thalheim b8e5d1f290 Merge pull request #14369 from NixOS/copy-sigs-docs
nix store copy-sigs: Add docs
2025-10-27 17:55:56 +00:00
Jörg Thalheim d44b33562f Merge pull request #14373 from NixOS/copy-sigs-parallel
nix store copy-sigs: Use http-connections setting to control parallelism
2025-10-27 17:53:23 +00:00
Jörg Thalheim d46504a136 Merge pull request #14359 from obsidiansystems/structured-attrs-always-object
Use types to show that structured attrs are always JSON objects
2025-10-27 17:51:33 +00:00
Eelco Dolstra 126f30deb2 Merge pull request #14366 from NixOS/const-fields
EvalState: Make some more fields const
2025-10-27 17:47:21 +00:00
Bernardo Meurer Costa 5dcfa86910 refactor(libstore): use string_view in HttpBinaryCacheStore::makeRequest 2025-10-27 17:13:48 +00:00
Eelco Dolstra 6b6ceddf72 nix store copy-sigs: Use http-connections setting to control parallelism
Previously it used the `ThreadPool` default,
i.e. `std::thread::hardware_concurrency()`. But copying signatures is
not primarily CPU-bound so it makes more sense to use the
`http-connections` setting (since we're typically copying from/to a
binary cache).
2025-10-27 16:43:25 +01:00
Eelco Dolstra 60f9489b83 Merge pull request #14370 from NixOS/misc-cleanups
Miscellaneous cleanups
2025-10-27 15:04:53 +00:00
Marcel 584a8e8a00 treewide: replace manual MiB calculations with renderSize 2025-10-27 16:04:19 +01:00
MarcelandBernardo Meurer Costa f234633e27 refactor(libutil): remove showBytes() in favor of renderSize()
The `showBytes()` function was redundant with `renderSize()` as the
latter automatically selects the appropriate unit (KiB, MiB, GiB, etc.)
based on the value, whereas `showBytes()` always formatted as MiB
regardless of size.

Co-authored-by: Bernardo Meurer Costa <beme@anthropic.com>
2025-10-27 16:04:08 +01:00
Sergei Zimmerman 6417863ce9 Merge pull request #14357 from lovesegfault/s3-setup-pub
refactor(libstore/filetransfer): make setupForS3 public
2025-10-27 14:36:41 +00:00
Eelco Dolstra 91cd42511e Introduce MINIMUM_PROTOCOL_VERSION constant 2025-10-27 15:11:20 +01:00
Eelco Dolstra 1af5a98955 Document removed WorkerProto ops 2025-10-27 15:09:03 +01:00
Eelco Dolstra 17777e3b70 Settings typos 2025-10-27 15:07:56 +01:00
Eelco Dolstra 9321669353 Make getDefaultCores() static 2025-10-27 15:07:01 +01:00
Eelco Dolstra 3742ae061e Typo 2025-10-27 15:04:56 +01:00
Eelco Dolstra a91115bf22 Remove unnecessary virtual 2025-10-27 15:04:13 +01:00
Eelco Dolstra 8c8b706f6b Fix an update to a finished value 2025-10-27 15:01:46 +01:00
Eelco Dolstra fb26285458 Fix #include 2025-10-27 14:53:46 +01:00
Eelco Dolstra bbfaaf3a20 showHelp(): Use one callFunction 2025-10-27 14:52:18 +01:00
Sergei Zimmerman f9b73185e4 Merge pull request #14362 from NixOS/k-way-merge-speedup
libexpr: Speed up BindingsBuilder::finishSizeIfNecessary
2025-10-27 13:45:29 +00:00
Eelco Dolstra 27e3d28ed8 Merge pull request #14340 from juhp/patch-1
nix-2.32 needs boost-1.87+ for `try_emplace_and_cvisit`
2025-10-27 13:44:37 +00:00
Eelco Dolstra 3994e5627f nix store copy-sigs: Add docs 2025-10-27 14:42:22 +01:00
Sergei Zimmerman ec2fd2dc23 libexpr: Speed up BindingsBuilder::finishSizeIfNecessary
Instead of iterating over the newly built bindings we can
do a cheaper set_intersection to count duplicates or fall back
to a per-element binary search over the "base" bindings.

This speeds up `hello` evaluation by around 10ms (0.196s -> 0.187s) and
`nixos.closures.ec2.x86_64-linux` by 140ms (2.744s -> 2.609s).

This addresses a somewhat steep performance regression from 82315c3807
that reduced memory requirements of attribute set merges. With this patch
we get back around to 2.31 level of eval performance while keeping the memory
usage optimization.

Also document the optimization a bit more.
2025-10-27 16:14:19 +03:00
Eelco Dolstra fdc5600fa7 makeRegexCache(): Return a ref 2025-10-27 14:11:59 +01:00
Eelco Dolstra 1f6ac88efc Mark some fields in EvalState as const 2025-10-27 14:10:34 +01:00
Marcel 9d4d10954a diff-closures: print sizes with dynamic unit 2025-10-27 02:05:03 +01:00
John Ericson 7e53afd8b9 Use types to show that structured attrs are always JSON objects
Before we just had partial code accessing it. Now, we use
`nlohmann::json::object_t`, which is a `std::map`, to enforce this by
construction.
2025-10-26 12:53:58 -04:00
John Ericson bef3c37cb2 Merge pull request #14351 from obsidiansystems/json-project-reference
Clean up JSON utils in a few ways
2025-10-25 19:32:30 +00:00
John EricsonandSergei Zimmerman 0f0d9255c6 Clean up JSON utils in a few ways
In particular

- Remove `get`, it is redundant with `valueAt` and the `get` in
  `util.hh`.

- Remove `nullableValueAt`. It is morally just the function composition
  `getNullable . valueAt`, not an orthogonal combinator like the others.

- `optionalValueAt` return a pointer, not `std::optional`. This also
  expresses optionality, but without creating a needless copy. This
  brings it in line with the other combinators which also return
  references.

- Delete `valueAt` and `optionalValueAt` taking the map by value, as we
  did for `get` in 408c09a120, which
  prevents bugs / unnecessary copies.

`adl_serializer<DerivationOptions::OutputChecks>::from_json` was the one
use of `getNullable`. I give it a little static function for the
ultimate creation of a `std::optional` it does need to do (after
switching it to using `getNullable . valueAt`. That could go in
`json-utils.hh` eventually, but I didn't bother for now since only one
things needs it.

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-25 14:49:51 -04:00
Sergei Zimmerman f0b95b6d5b Merge pull request #14274 from lovesegfault/nix-s3-versioned
feat(libstore): support S3 object versioning via versionId parameter
2025-10-25 08:39:12 +00:00
Bernardo Meurer Costa e38128b90d feat(libstore): support S3 object versioning via versionId parameter
S3 buckets support object versioning to prevent unexpected changes,
but Nix previously lacked the ability to fetch specific versions of
S3 objects. This adds support for a `versionId` query parameter in S3
URLs, enabling users to pin to specific object versions:

```
s3://bucket/key?region=us-east-1&versionId=abc123
```
2025-10-25 07:57:58 +00:00
Bernardo Meurer Costa 78e98691d6 refactor(libstore/filetransfer): make setupForS3 public 2025-10-25 03:45:30 +00:00
John Ericson e213fd64b6 Merge pull request #14352 from NixOS/source-paths-tests
tests/functional: Add source-paths tests
2025-10-24 23:50:07 +00:00
Eelco Dolstra 1cd8458c28 tests/functional: Add source-paths tests
This has already been implemented in 1e709554d5
as a side-effect of mounting the accessors in storeFS. Let's test this so it
doesn't regress.

(cherry-picked from https://github.com/NixOS/nix/pull/12915)
2025-10-25 02:13:30 +03:00
John Ericson ecaf9470b9 Merge pull request #14344 from obsidiansystems/json-schema-deriving-path
JSON Schema for `DerivedPath`
2025-10-24 23:09:08 +00:00
Sergei Zimmerman 8b7e03f0f9 Merge pull request #14350 from lovesegfault/s3-binary-cache-store
refactor(libstore): expose HttpBinaryCacheStore and add S3BinaryCacheStore
2025-10-24 22:59:02 +00:00
Sergei Zimmerman 04606d50d1 Merge pull request #14343 from NixOS/epipe-graceful
Revert "libmain: Catch logger exceptions in `handleExceptions`"
2025-10-24 22:52:29 +00:00
Bernardo Meurer Costa 476c21d5ef refactor(libstore): expose HttpBinaryCacheStore and add S3BinaryCacheStore
Move HttpBinaryCacheStore class from .cc file to header to enable
inheritance by S3BinaryCacheStore. Create S3BinaryCacheStore class that
overrides upsertFile() to implement multipart upload logic.
2025-10-24 21:54:13 +00:00
John Ericson 1a9ba0d6fe Merge pull request #14333 from lovesegfault/upsert-size-hint
refactor(libstore): add sizeHint parameter to upsertFile()
2025-10-24 19:29:06 +00:00
John Ericson 648714cd44 Merge pull request #14336 from lovesegfault/filetransfer-delete
feat(libstore): add DELETE method support to FileTransfer
2025-10-24 18:50:53 +00:00
Bernardo Meurer Costa 6b7223b6b7 refactor(libstore): add sizeHint parameter to upsertFile()
Add a sizeHint parameter to BinaryCacheStore::upsertFile() to enable
size-based upload decisions in implementations. This lays the groundwork
for reintroducing S3 multipart upload support.
2025-10-24 18:49:28 +00:00
Bernardo Meurer Costa afe5ed879f feat(libstore): add DELETE method support to FileTransfer
Add support for HTTP DELETE requests to FileTransfer infrastructure:

This enables S3 multipart upload abort functionality via DELETE requests
to S3 endpoints.
2025-10-24 18:03:14 +00:00
Bernardo Meurer Costa d924374bf2 docs(libstore): document verb() method returns verb root for gerund form
Add documentation to FileTransferRequest::verb() explaining that it returns
a verb root intended to be concatenated with "ing" to form the gerund.
2025-10-24 18:03:13 +00:00
Bernardo Meurer Costa f1968ea38e refactor(libstore): replace HTTP method boolean flags with enum
Replace the individual boolean flags (head, post) with a unified
HttpMethod enum struct in FileTransferRequest.
2025-10-24 18:03:12 +00:00
John Ericson 8d338c9234 JSON Schema for DerivedPath
Note that this is "deriving path" in the manual -- the great sed of the
code base to bring it in sync has yet to happen yet.
2025-10-24 12:08:00 -04:00
John Ericson 9a695f9067 Merge pull request #14348 from NixOS/fetchClosure-access
Allow access to the result of fetchClosure
2025-10-24 15:44:31 +00:00
Sergei Zimmerman 925c0fa4a2 Merge pull request #14346 from NixOS/remove-verify-tls
libstore/filetransfer: Remove verifyTLS from FileTransferRequest, sin…
2025-10-24 10:48:43 +00:00
Eelco Dolstra 7308fde0bc Allow access to the result of fetchClosure 2025-10-24 11:11:03 +02:00
Sergei Zimmerman 324bfd82dc Merge pull request #14337 from lovesegfault/fix-post-large
fix(libstore): use CURLOPT_POSTFIELDSIZE_LARGE for POST requests
2025-10-23 22:00:08 +00:00
Sergei Zimmerman 8e01e4ad5c Merge pull request #14347 from NixOS/mahic-nix-cache-hook-fix
ci: Bump magic-nix-cache with post-build-hook fix
2025-10-23 22:43:46 +00:00
Sergei Zimmerman 4c4eb5d07f ci: Bump magic-nix-cache with post-build-hook fix
No tagged release with the fix for [^].

[^]: https://github.com/DeterminateSystems/magic-nix-cache/commit/578f01e1473129cc289d579ba6dee9cdac40aeab
2025-10-24 01:34:09 +03:00
Sergei Zimmerman b5ae3e10c2 libstore/filetransfer: Remove verifyTLS from FileTransferRequest, since it's always true
This variable is always true, so there's no use-case for it anymore.
2025-10-24 00:29:10 +03:00
Sergei Zimmerman 4f5af471fb Revert "libmain: Catch logger exceptions in handleExceptions"
This reverts commit 90d1ff4805.

The initial issue with EPIPE was solved in 9f680874c5.
Now this patch does move bad than good by eating up boost::io::format_error that are
bugs.
2025-10-23 23:49:41 +03:00
Sergei Zimmerman b9af19cedf Merge pull request #14295 from NixOS/s3-store-human-readable-uri
libstore: Implement getHumanReadableURI for S3BinaryCacheStoreConfig
2025-10-23 19:33:49 +00:00
Eelco Dolstraandcoderabbitai[bot] d6f1e2de21 Merge pull request #14323 from NixOS/skip-nar-parse
addToStore(): Don't parse the NAR

* StringSource: Implement skip()

This is slightly faster than doing a read() into a buffer just to
discard the data.

* LocalStore::addToStore(): Skip unnecessary NARs rather than parsing them

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-10-23 22:17:09 +03:00
John Ericson 5d365cd61f Merge pull request #14341 from obsidiansystems/fix-golden-tests
Fix some characterization tests
2025-10-23 19:08:43 +00:00
John Ericson c87f29a0b6 Fix some characterization tests
A few changes had cropped up with `_NIX_TEST_ACCEPT=1`:

1. Blake hashing test JSON had a different indentation

2. Store URI had improper non-quoted spaces

(1) was is just fixed, as we trust nlohmann JSON to parse JSON
correctly, regardless of whitespace.

For (2), the existing URL was made a read-only test, since we very much
wish to continue parsing such invalid URLs directly. And then the
original read/write test was updated to properly percent-encode the
space, as the normal form should be.
2025-10-23 14:03:21 -04:00
Jens Petersen f594a8e11e libexpr needs boost-1.87+ for try_emplace_and_cvisit
Since 2.32, nix now needs boost 1.87 or later to build,
due to using unordered::concurrent_flat_map try_emplace_and_cvisit

../src/libexpr/eval.cc: In member function ‘void nix::EvalState::evalFile(const nix::SourcePath&, nix::Value&, bool)’:
../src/libexpr/eval.cc:1096:20: error: ‘class boost::unordered::concurrent_flat_map<nix::SourcePath, nix::Value*, std::hash<nix::SourcePath>, std::equal_to<nix::SourcePath>, traceable_allocator<std::pair<const nix::SourcePath, nix::Value*> > >’ has no member named ‘try_emplace_and_cvisit’; did you mean ‘try_emplace_or_cvisit’?
 1096 |     fileEvalCache->try_emplace_and_cvisit(
      |                    ^~~~~~~~~~~~~~~~~~~~~~
      |                    try_emplace_or_cvisit

See https://github.com/boostorg/unordered/commit/834580b53948eec553c232dda40beefc68b3e8f9
2025-10-24 01:24:04 +08:00
Eelco Dolstra 0a74b4905c Merge pull request #14332 from NixOS/cleanup-ci
ci: Assorted collection of cleanups
2025-10-23 16:50:11 +00:00
Eelco Dolstra d74177dccc Merge pull request #14328 from cachix/nar-substitutiongone
Fix misleading error messages for missing NARs due to stale cache
2025-10-23 16:48:31 +00:00
Sergei Zimmerman 36ee38efd1 Merge pull request #14338 from lovesegfault/s3-docs-listbucket
docs: add s3:ListBucket to S3 read permissions
2025-10-23 08:43:01 +00:00
Sergei Zimmerman 5d7912eb18 Merge pull request #14335 from lovesegfault/extract-getcompressionmethod
refactor(libstore): extract getCompressionMethod() in HttpBinaryCacheStore
2025-10-23 08:30:08 +00:00
Bernardo Meurer Costa 78888ec8a8 docs: add s3:ListBucket to S3 read permissions
The s3:ListBucket permission is required for read operations on S3
binary caches, not just for writes. Without this permission, users get
"Access Denied" errors when running nix-build.
2025-10-23 06:03:00 +00:00
Bernardo Meurer Costa b047cecf5c refactor(libstore): extract getCompressionMethod() in HttpBinaryCacheStore
Extract the path-based compression method determination logic into a
protected method that returns std::optional<std::string>. This allows
subclasses to reuse the logic and makes the semantics clearer (nullopt
means no compression, not empty string).

This prepares for S3BinaryCacheStore to apply the same compression
rules when implementing multipart uploads.
2025-10-23 05:03:02 +00:00
John Ericson d0217ec180 Merge pull request #14331 from NixOS/debug-build-fix
meson: Only enable b_lto for nixexpr-parser when b_lto is enabled glo…
2025-10-23 04:52:55 +00:00
Bernardo Meurer Costa 953929f899 fix(libstore): use CURLOPT_POSTFIELDSIZE_LARGE for POST requests
Fix POST requests with data to use the correct curl option for specifying
body size. Previously used CURLOPT_INFILESIZE_LARGE for both POST and PUT,
but POST requires CURLOPT_POSTFIELDSIZE_LARGE.

This caused POST request bodies to not be sent correctly, manifesting as
S3 multipart CompleteMultipartUpload requests failing with "You must
specify at least one part" even though the XML body contained valid parts.
2025-10-23 02:26:45 +00:00
Sergei Zimmerman 3c83856494 ci: Update pinned install_url 2.30.2 -> 2.32.1 2025-10-23 02:17:12 +03:00
Sergei Zimmerman f3d8d1f719 ci: Reuse composite install-nix-action for docker_push_image job 2025-10-23 02:17:11 +03:00
Sergei Zimmerman c8a15bf70d ci: Pin cachix action 2025-10-23 02:17:10 +03:00
Sergei Zimmerman ad5c6a53b9 ci: Move magic-nix-cache-action into install-nix-action composite
This reduces duplication and pins the underlying version of magic-nix-cache,
as we already do with other actions.
2025-10-23 02:17:09 +03:00
Sergei Zimmerman 350d602832 meson: Only enable b_lto for nixexpr-parser when b_lto is enabled globally 2025-10-23 01:49:31 +03:00
John Ericson 115dea10b2 Merge pull request #14320 from roberth/open-manual-app
flake.nix: Add nix run .#open-manual
2025-10-22 21:37:21 +00:00
Eelco Dolstra ddb8830c97 Merge pull request #14326 from adeci/githint
fetchers: add helpful hint for file+git URL scheme error
2025-10-22 20:39:16 +00:00
Domen KožarandClaude 459f9e0185 Fix misleading error messages for missing NARs due to stale cache
When Nix's SQLite narinfo cache indicates a NAR exists, but the NAR
has been garbage collected from the binary cache, Nix displays error
messages even though the operation succeeds via fallback. This is
misleading because the cached narinfo is simply outdated.

This changes SubstituteGone exceptions to produce warnings instead of
errors, accurately reflecting that this is an expected cache coherency
issue, not an actual failure.

Fixes #11411

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-22 15:07:42 -05:00
Sergei Zimmerman 5390bba920 Merge pull request #14314 from Radvendii/parser-cpp-variant
libexpr: parser.y: use api.value.type variant
2025-10-22 18:49:14 +00:00
adeci 387eceff45 fetchers: Add helpful hint for file+git URL scheme error
At least one user has probably used `file+git://` when they mean `git+file://`, maybe thinking of it as "a file-based git repository". This adds a specific error message to hint at the correct URL scheme format and may save some users from resorting to `path:///` and copying an entire repo.
2025-10-22 13:57:51 -04:00
Sergei Zimmerman 96c8cc550f libexpr/meson: Rice the compiler inlining heuristics to improve perf of the bison generated parser
Turns out both GCC and Clang need a bit of hand-holding to optimize the bison generated
code well, otherwise parser performance tanks.

(Comparisons against baseline in 7e8db2eb59):

For GCC:

Benchmark 1 (15 runs): result/bin/nix-instantiate --parse ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           335ms ± 2.89ms     332ms …  342ms          0 ( 0%)        0%

Benchmark 2 (16 runs): result-old/bin/nix-instantiate --parse ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           330ms ± 2.87ms     326ms …  337ms          0 ( 0%)          -  1.4% ±  0.6%

For Clang:

Benchmark 1 (15 runs): result-clang/bin/nix-instantiate --parse ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           340ms ± 1.43ms     338ms …  343ms          0 ( 0%)        0%

Benchmark 2 (15 runs): result-old-clang/bin/nix-instantiate --parse ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           334ms ± 1.61ms     332ms …  338ms          0 ( 0%)        ⚡-  1.7% ±  0.3%
2025-10-22 02:25:11 +02:00
Taeer Bar-Yam 32b286e5d6 libexpr: parser.y: api.value.type variant 2025-10-22 02:25:11 +02:00
Robert Hensing b558dac7a9 flake.nix: Add nix run .#open-manual
Great for reviewing the rendered manual
2025-10-22 00:42:18 +02:00
Sergei Zimmerman 7e8db2eb59 Merge pull request #14318 from cole-h/remove-useless-fmt
libstore: remove useless fmt
2025-10-21 17:50:56 +00:00
John Ericson 6ca2efc7d4 Merge pull request #14254 from roberth/upstream-RossComputerGuy/feat/expose-computefsclosure
libstore-c: add nix_store_get_fs_closure #14025 with tests and realise error fix
2025-10-21 17:41:29 +00:00
Cole Helbling 62247af363 libstore: remove useless fmt 2025-10-21 10:06:35 -07:00
John Ericson a4a49a9dae Merge pull request #14316 from NixOS/fix-computeStorePath-arg
Fix computeStorePath() default argument
2025-10-21 14:48:56 +00:00
Eelco Dolstra 606c258c6f Fix computeStorePath() default argument 2025-10-21 15:58:44 +02:00
John Ericson ef8218f2e3 Merge pull request #14307 from NixOS/json-schema-hash
`nlohmann::json` instance and JSON Schema for `Hash`
2025-10-21 06:03:20 +00:00
John Ericson ada008a795 Merge pull request #14310 from obsidiansystems/inline-drv-output-subst-goal
Inline `realisationFetched`
2025-10-21 06:03:06 +00:00
John Ericson 2a2bb8330d Merge pull request #14312 from corngood/develop-structuredAttrs-fix
tests/functional/flakes/develop.sh: Add test for outputChecks stripping
2025-10-20 22:48:39 +00:00
David McFarland 645794b458 tests/functional/flakes/develop.sh: Add test for outputChecks stripping 2025-10-20 19:16:20 -03:00
John Ericson 1121f0d8ec Inline realisationFetched
Now that we are using coroutines, we don't need this to be a separate
method of `DrvOutputSubstitutionGoal`.
2025-10-20 16:45:41 -04:00
Sergei Zimmerman 6420879728 Merge pull request #14296 from lovesegfault/nix-s3-more-tests
fix(nix-prefetch-url): correctly extract filename from URLs with query parameters
2025-10-20 19:42:22 +00:00
Sergei Zimmerman 67f5cb97a3 Merge pull request #14306 from corngood/develop-structuredAttrs-fix
nix/develop: Strip outputChecks when structuredAttrs is enabled
2025-10-20 19:38:19 +00:00
John EricsonandRobert Hensing 5e7ee808de nlohmann::json instance and JSON Schema for Hash
Improving and codifying our experimental JSON interfacing.

Co-Authored-By: Robert Hensing <robert@roberthensing.nl>
2025-10-20 15:21:07 -04:00
John Ericson 270f20a505 Merge pull request #14305 from NixOS/alignment-utils
libutil: Add alignUp helper function, use in archive.cc
2025-10-20 19:08:20 +00:00
Bernardo Meurer Costa 1b1d7e3047 test(nixos): add nix-prefetch-url test for S3 URLs with query parameters
Adds a comprehensive test to verify that `nix-prefetch-url` correctly
handles S3 URLs with query parameters (e.g., custom endpoints and regions).

Previously, nix-prefetch-url would fail with "invalid store
path" errors when given S3 URLs with query parameters like
`?endpoint=http://server:9000&region=eu-west-1`, because it incorrectly
extracted the filename from the query parameters instead of the path.
2025-10-20 21:45:37 +03:00
David McFarland 0f28c76a44 nix/develop: Strip outputChecks when structuredAttrs is enabled 2025-10-20 15:40:05 -03:00
Bernardo Meurer Costa e3b3f05e5d fix(nix-prefetch-url): correctly extract filename from URLs with query parameters
Previously, `prefetchFile()` used `baseNameOf()` directly on the URL string
to extract the filename. This caused issues with URLs containing query
parameters that include slashes, such as S3 URLs with custom endpoints:

```
s3://bucket/file.txt?endpoint=http://server:9000
```

The `baseNameOf()` function naively searches for the rightmost `/` in the
entire string, which would find the `/` in `http://server:9000` and extract
`server:9000&region=...` as the filename. This resulted in invalid store
path names containing illegal characters like `:`.

This commit fixes the issue by:

1. Adding a `VerbatimURL::lastPathSegment()` method that extracts the last
   non-empty path segment from a URL, using `pathSegments(true)` to filter
   empty segments
2. Changing `prefetchFile()` to accept `const VerbatimURL &` and use the new
   `lastPathSegment()` method instead of manual path parsing
3. Adding early validation with `checkName()` to fail quickly on invalid
   filenames
4. Maintains backward compatibility by falling back to `baseNameOf()` for
   unparsable `VerbatimURL`s
2025-10-20 21:40:03 +03:00
John Ericson f05d240222 Merge pull request #14278 from obsidiansystems/adl-serializer-xp
Cleanup and JSON serializer and XP feature interations
2025-10-20 18:22:21 +00:00
Sergei Zimmerman 22c73868c3 libutil/archive: Use alignUp
With this change it's much more apparent what's going on.
2025-10-20 21:15:11 +03:00
Sergei Zimmerman a91b787524 libutil: Add alignUp helper function 2025-10-20 21:11:00 +03:00
Eelco Dolstra ddf7de0a76 Merge pull request #14291 from NixOS/skip-source
Add skip() method to Source interface to allow efficient seeks
2025-10-20 15:04:36 +00:00
Sergei Zimmerman 1fabed18b6 Merge pull request #14301 from NixOS/s3-terminate-unknown-profile
libstore: Fix reentrancy in AwsCredentialProviderImpl::getCredentialsRaw
2025-10-20 14:28:16 +00:00
Eelco Dolstra 6c9083db2c Use a smaller buffer 2025-10-20 13:40:19 +02:00
Sergei Zimmerman c663f7ec79 libstore: Fix reentrancy in AwsCredentialProviderImpl::getCredentialsRaw
Old code would do very much incorrect reentrancy crimes (trying to do an
erase inside the emplace callback). This would fail miserably with an assertion
in Boost:

terminating due to unexpected unrecoverable internal error: Assertion '(!find(px))&&("reentrancy not allowed")' failed in boost::unordered::detail::foa::entry_trace::entry_trace(const void *) at include/boost/unordered/detail/foa/reentrancy_check.hpp:33

This is trivially reproduced by using any S3 URL with a non-empty profile:

nix-prefetch-url "s3://happy/crash?profile=default"
2025-10-19 21:03:13 +03:00
Sergei Zimmerman d0fb03c35d Merge pull request #14282 from NixOS/s3-cleanup
Simplify meson for S3 support via aws-crt-cpp
2025-10-19 17:00:46 +00:00
Sergei Zimmerman c847cd87f1 Merge pull request #14297 from lovesegfault/nix-s3-test-public
test(nixos/s3-binary-cache-store): misc improvements
2025-10-19 16:53:40 +00:00
tomberek dbbdae926b Merge pull request #14299 from roberth/unlocked-msg
Clarify unlocked input warning message
2025-10-19 16:50:16 +00:00
Eelco Dolstra 3c03050cd6 Merge pull request #14290 from NixOS/dont-write-nar-to-tty
nix store dump-path: Refuse to write NARs to the terminal
2025-10-19 12:41:55 +00:00
Robert Hensing e33cd5aa38 Clarify unlocked input warning message
The previous message was vague about what "deprecated" meant and why
unlocked inputs with NAR hashes "may not be reproducible". It also
used "verifiable" which was confusing.

The new message makes it clear that the NAR hash provides verification
(is checked by NAR hash) and explicitly states the failure modes:
garbage collection and sharing.
2025-10-19 14:08:34 +02:00
Bernardo Meurer Costa d9c808f8a7 refactor(tests/nixos/s3-binary-cache-store): add verify_packages_in_store helper 2025-10-19 00:21:54 +00:00
Bernardo Meurer Costa 55ea3d3476 test(tests/nixos/s3-binary-cache-store): test public bucket operations
Add `test_public_bucket_operations` to validate that store operations
work correctly on public S3 buckets without requiring credentials.
Tests nix store info and nix copy operations.
2025-10-19 00:04:33 +00:00
Bernardo Meurer Costa 7d0c06f921 feat(tests/nixos/s3-binary-cache-store): add public parameter to setup_s3
Add optional 'public' parameter to setup_s3 decorator. When set to True,
the bucket will be made publicly accessible using mc anonymous set.
2025-10-18 23:57:51 +00:00
Bernardo Meurer Costa 5b4bd5bcb8 refactor(tests/nixos/s3-binary-cache-store): inline make_http_url fn
Remove make_http_url helper function and inline its single usage.
2025-10-18 23:51:44 +00:00
Bernardo Meurer Costa 4ae6c65bc5 test(tests/nixos/s3-binary-cache-store): verify credential caching in concurrent fetches
Add assertion to test_concurrent_fetches to verify that only one
credential provider is created even with 5 concurrent fetches.
2025-10-18 23:48:55 +00:00
Bernardo Meurer Costa 4f19e63a8f refactor(tests/nixos/s3-binary-cache-store): add --no-link to nix build commands
Prevent creation of result symlinks in all nix build commands by
adding the --no-link flag.
2025-10-18 23:44:13 +00:00
Bernardo Meurer Costa f88c3055f8 refactor(tests/nixos/s3-binary-cache-store): clean client store in setup_s3
Add cleanup of client store in the finally block of setup_s3 decorator.
Uses `nix store delete --ignore-liveness` to properly handle GC roots
and only attempts deletion if the path exists.
2025-10-18 23:36:48 +00:00
Bernardo Meurer Costa 9058d90ab2 refactor(tests/nixos/s3-binary-cache-store): rename populate_with to populate_bucket 2025-10-18 23:27:03 +00:00
Bernardo Meurer Costa c1a15d1a26 refactor(tests/nixos/s3-binary-cache-store): rename with_test_bucket to setup_s3 2025-10-18 23:24:30 +00:00
Bernardo Meurer Costa 22f4cccc71 refactor(tests/nixos/s3-binary-cache-store): use a PKGS dict
Replace individual PKG_A, PKG_B, and PKG_C variables with a PKGS
dictionary. This will enable `@with_clean_client_store` in the future.
2025-10-18 23:23:50 +00:00
John Ericson b56e456b0d Merge pull request #14269 from roberth/json-schema
Add a JSON Schema for `Derivation`
2025-10-18 18:50:39 +00:00
Sergei Zimmerman 3d147c04a5 libstore: Implement getHumanReadableURI for S3BinaryCacheStoreConfig
This slightly improves the logs situation by including the region/profile/endpoint
in the logs when S3 store references get printed. Instead of:

copying path '/nix/store/lxnp9cs4cfh2g9r2bs4z7gwwz9kdj2r9-test-package-c' to 's3://bucketname'...

This now includes:

copying path '/nix/store/lxnp9cs4cfh2g9r2bs4z7gwwz9kdj2r9-test-package-c' to 's3://bucketname?endpoint=http://server:9000&region=eu-west-1'...
2025-10-18 19:11:39 +03:00
Sergei Zimmerman 61fbef42a6 libstore: Simplify check for S3-specific URI query parameters
Instead of hardcoding strings we should instead use the setting
objects to determine the query names that should be preserved.
2025-10-18 18:47:27 +03:00
Robert Hensing c92ba4b9b7 Add titles in JSON schemas
This way, the description isn't rendered in the tables of contents,
leading to no more formatting errors.
2025-10-17 21:53:29 +02:00
Eelco Dolstra 67bffa19a5 NullFileSystemObjectSink: Skip over file contents 2025-10-17 20:44:02 +02:00
Eelco Dolstra daa7e0d2e9 Source: Add skip() method
This allows FdSource to efficiently skip data we don't care about.
2025-10-17 20:41:33 +02:00
Eelco Dolstra 109f6449cc nix store dump-path: Refuse to write NARs to the terminal 2025-10-17 20:27:10 +02:00
John Ericson ad2360c59f Merge pull request #14288 from lovesegfault/repl-skip-stack
fix(tests/functional/repl): skip test if stack size limit is insufficient
2025-10-17 17:35:52 +00:00
Bernardo Meurer Costa 20c7c551bf fix(tests/functional/repl): skip test if stack size limit is insufficient
Nix attempts to set the stack size to 64 MB during initialization, which is
required for the repl tests to run successfully. Skip the tests on systems
where the hard stack limit is less than this value rather than failing.
2025-10-17 17:05:12 +00:00
John Ericson e78e6ca4f4 Merge pull request #14281 from NixOS/dead-code
libutil: Drop unused SubdirSourceAccessor
2025-10-17 03:01:17 +00:00
John Ericson e34063cf21 Merge pull request #14283 from NixOS/nar-check
nix {cat,ls}: Add back missing checks for file descriptors
2025-10-17 02:58:23 +00:00
Sergei Zimmerman e457ea7688 nix {cat,ls}: Add back missing checks for file descriptors
I didn't catch this during the review of https://github.com/NixOS/nix/pull/14273.
This fixes that mistake.
2025-10-17 02:26:24 +03:00
Farid Zakaria 64c55961eb Merge pull request #14273 from fzakaria/fzakaria/issue-13944
Make `nix nar [cat|ls]` lazy
2025-10-17 02:16:54 +03:00
Sergei Zimmerman ffbc33fec6 libstore/meson: Rename curl-s3-store to s3-aws-auth
We now unconditionally compile support for s3:// URLs and stores
without authentication. The whole curl version check can be greatly
simplified by the previous commit, which bumps the minimum required curl
version.
2025-10-17 01:18:46 +03:00
Sergei Zimmerman a80fc252e8 libstore/meson: Require curl >= 7.75.0
This version has been released a long time ago in 2021 and it's doubtful
that anybody actually uses it still, since it's full of vulnerabilities [^]

[^]: https://curl.se/docs/vuln-7.75.0.html
2025-10-17 01:18:14 +03:00
Sergei Zimmerman bcd5a9d05c libutil: Drop unused SubdirSourceAccessor 2025-10-17 00:56:53 +03:00
Robert Hensing 01b001d5ba Add JSON Schema infrastructure, use for Derivation
For manual, and testing formats
2025-10-16 17:24:18 -04:00
John Ericson 27767a6094 Merge pull request #14276 from NixOS/fix-14193
libstore/registerOutputs: Don't try to optimize a non-existent actual…
2025-10-16 21:06:43 +00:00
John Ericson 1177d65094 Properly check xp features when deserializing deriving paths 2025-10-16 16:45:22 -04:00
John Ericson a2c6f38e1f Remove now-redundant methods for JSON on Derivation 2025-10-16 16:45:22 -04:00
John Ericson 1c02dd5b9c Allow for standard nlohmann JSON serializers to take separate XP features
I realized that we can actually do this thing, even though it is not
what nlohmann expects at all, because the extra parameter has a default
argument so nlohmann doesn't need to care. Sneaky!
2025-10-16 16:45:22 -04:00
Sergei Zimmerman 4cbcaad435 libstore/registerOutputs: Don't try to optimize a non-existent actualPath
Since 3c610df550 this resulted in `getting status of`
errors on paths inside the chroot if a path was already valid. Careful inspection
of the logic shows that if buildMode != bmCheck actualPath gets reassigned to
store.toRealPath(finalDestPath). The only branch that cares about actualPath is
the buildMode == bmCheck case, which doesn't lead to optimisePath anyway.
2025-10-16 23:08:30 +03:00
John Ericson d87a06af7a Merge pull request #14275 from NixOS/s3-cleanup
libstore: Miscellaneous S3 store cleanups
2025-10-16 19:36:59 +00:00
Eelco Dolstra 2dc9f2a2b7 Merge pull request #14272 from NixOS/use-store-path-serializer
Daemon protocol: Use the WorkerProto serializer for store paths
2025-10-16 19:35:25 +00:00
Eelco Dolstra a7991d55cc Merge pull request #14270 from NixOS/use-optional-storepath-serializer
Use serializer for std::optional<StorePath>
2025-10-16 19:07:07 +00:00
Sergei Zimmerman e7047fde25 libstore: Remove the unnecessary 'error: ' prefix in warning message 2025-10-16 21:49:38 +03:00
Sergei Zimmerman 33e94fe19f libstore: Make AwsAuthError more legible
Instead of the cryptic:

> error: Failed to resolve AWS credentials: error code 6153`

We now get more legible:

> error: AWS authentication error: 'Valid credentials could not be sourced by the IMDS provider' (6153)
2025-10-16 21:49:37 +03:00
Sergei Zimmerman dc03c6a812 libstore: Put all the AWS credentials logic behind interface class AwsCredentialProvider
This makes it so we don't need to rely on global variables and hacky destructors to
clean up another global variable. Just putting it in the correct order in the class
is more than enough.
2025-10-16 21:49:36 +03:00
Sergei Zimmerman b1d067c9bb tests/nixos: Rename back S3 store nixos test 2025-10-16 21:49:35 +03:00
Eelco Dolstra d782c5e586 Daemon protocol: Use the WorkerProto serializer for store paths 2025-10-16 17:34:33 +02:00
Eelco Dolstra f84b33644c Merge pull request #14271 from NixOS/no-check-sigs
Factor out `--no-check-sigs` into its own class
2025-10-16 15:07:29 +00:00
Eelco Dolstra 3bd2b76f6e nix store sign: Use required attribute 2025-10-16 16:35:13 +02:00
Eelco Dolstra 139df77440 Factor out --no-check-sigs 2025-10-16 16:35:09 +02:00
Eelco Dolstra a48a737517 Use serializer for std::optional<StorePath> 2025-10-16 16:32:18 +02:00
John Ericson 0503a862ef Merge pull request #14268 from roberth/dev-doc-manual
doc/dev/doc: Update local build instructions for manual
2025-10-16 13:50:58 +00:00
Robert Hensing 61cb9c4832 doc/dev/doc: Update local build instructions for manual 2025-10-16 13:22:22 +02:00
John Ericson 721f5572a6 Merge pull request #14263 from NixOS/hydra-import-paths
Restore `ServeProto::Command::ImportPaths`
2025-10-15 22:57:50 +00:00
John Ericson 5a6864c027 Merge pull request #14264 from xokdvium/fix-splicing-test
tests: Fix splicing in functional tests for nix-cli
2025-10-15 22:32:45 +00:00
Sergei Zimmerman 0deb492b3d Restore ServeProto::Command::ImportPaths
This partially reverts commit 5e46df973f,
partially reversing changes made to
8c789db05b.

We do this because Hydra, while using the newer version of the protocol,
still uses this command, even though Nix (as a client) doesn't use it.
On that basis, we don't want to remove it (or consider it only part of
the older versions of the protocol) until Hydra no longer uses the
Legacy SSH Protocol.
2025-10-15 18:18:59 -04:00
Sergei Zimmerman 17b7fb383f tests: Fix splicing in functional tests for nix-cli
This is necessary to fix nix-everything-llvm.
The problem here is that nix-cli is taken from the previous
stage that is built with libstdc++, but this derivation builds
plugins with libc++ and the plugin load fails miserably.
2025-10-16 01:04:50 +03:00
John Ericson 94cfba7e84 Merge pull request #14226 from obsidiansystems/unkeyed-realisation
Reapply #14097
2025-10-15 21:27:13 +00:00
Sergei Zimmerman 0f1cfa4d60 Merge pull request #14262 from lovesegfault/ci-cleanup-s3
ci: cleanup s3 tests
2025-10-15 21:19:20 +00:00
Bernardo Meurer Costa fa0d00e668 ci: cleanup s3 tests
This cleans up the work done in 8c2828387. Now that #13752 has landed,
there's no need to test configurations without AWS auth in CI.
2025-10-15 23:51:08 +03:00
Robert Hensing 6036aaf798 C API: Check output callback order 2025-10-15 22:04:21 +02:00
Sergei Zimmerman d2b6499154 Merge pull request #13752 from lovesegfault/curl-based-s3-v2
feat(libstore): curl-based s3
2025-10-15 20:01:26 +00:00
Sergei Zimmerman e3232af558 Merge pull request #14253 from NixOS/libgit2-refname-wa
libfetchers/git-utils: Be more correct about validating refnames
2025-10-15 19:30:53 +00:00
Bernardo Meurer Costa e069c9892e docs(rl-next): add notes for curl-based s3 2025-10-15 19:26:53 +00:00
John Ericson 266fbebe66 Implement realisation operations on dummy store 2025-10-15 14:59:08 -04:00
John EricsonandSergei Zimmerman 6995d325ef Split out UnkeyedRealisation from Realisation
Realisations are conceptually key-value pairs, mapping `DrvOutputs` (the
key) to information about that derivation output.

This separate the value type, which will be useful in maps, etc., where
we don't want to denormalize by including the key twice.

This matches similar changes for existing types:

| keyed              | unkeyed                |
|--------------------|------------------------|
| `ValidPathInfo`    | `UnkeyedValidPathInfo` |
| `KeyedBuildResult` | `BuildResult`          |
| `Realisation`      | `UnkeyedRealisation`   |

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-15 14:59:04 -04:00
Sergei Zimmerman 5d1178b817 libfetchers/git-utils: Be more correct about validating refnames
Turns out there's a much better API for this that doesn't have the
footguns of the previous method.

isLegalRefName is somewhat of a misnomer, since it's mainly used to
validate user inputs that can be either references, branch names,
psedorefs or tags.
2025-10-15 21:54:09 +03:00
Bernardo Meurer Costa 3224636ab0 refactor(libstore): rename NIX_WITH_S3_SUPPORT to NIX_WITH_AWS_AUTH
The macro now accurately reflects its purpose: gating only AWS
authentication code, not all S3 functionality. S3 URL parsing, store
configuration, and public bucket access work regardless of this flag.

This rename clarifies that:
- S3 support is always available (URL parsing, store registration)
- Only AWS credential resolution requires the flag
- The flag controls AWS CRT SDK dependency, not S3 protocol support
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa bb1f22a8df refactor(libstore): minimize NIX_WITH_S3_SUPPORT scope to auth only
Move S3 URL parsing, store configuration, and public bucket support
outside of NIX_WITH_S3_SUPPORT guards. Only AWS credential resolution
remains gated, allowing builds with withAWS = false to:

- Parse s3:// URLs
- Register S3 store types
- Access public S3 buckets (via HTTPS conversion)
- Use S3-compatible services without authentication

The setupForS3() function now always performs URL conversion, with
authentication code conditionally compiled based on NIX_WITH_S3_SUPPORT.
The aws-creds.cc file (only code using AWS CRT SDK) is now conditionally
compiled by meson.
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa 1f710300c9 refactor(libstore): withCurlS3 -> withAWS
Now that the legacy S3 implementation is gone, we can go back to calling
things `NIX_WITH_S3_SUPPORT`.
2025-10-15 18:23:56 +00:00
Bernardo Meurer Costa 9295c14a35 refactor(libstore): replace AWS SDK with curl-based S3 implementation
This commit replaces the AWS C++ SDK with a lighter curl-based approach
for S3 binary cache operations.

- Removed dependency on the heavy aws-cpp-sdk-s3 and aws-cpp-sdk-transfer
- Added lightweight aws-crt-cpp for credential resolution only
- Leverages curl's native AWS SigV4 authentication (requires curl >= 7.75.0)
- S3BinaryCacheStore now delegates to HttpBinaryCacheStore
- Function s3ToHttpsUrl converts ParsedS3URL to ParsedURL
- Multipart uploads are no longer supported (may be reimplemented later)
- Build now requires curl >= 7.75.0 for AWS SigV4 support

Fixes: #13084, #12671, #11748, #12403, #5947
2025-10-15 18:23:55 +00:00
John Ericson a543519ca9 Merge pull request #14257 from obsidiansystems/misc-builder-tech-debt
Cleanup misc builder tech debt
2025-10-15 17:28:27 +00:00
John Ericson 632ccfb8c0 Remove dead outputPaths variable. 2025-10-15 12:16:53 -04:00
John Ericson 46357468a4 Remove unused parameters to DrvOutputSubstitutionGoal 2025-10-15 12:16:52 -04:00
John Ericson b20cebf993 Remove unused typedef and field 2025-10-15 12:15:23 -04:00
Eelco Dolstra 5700112127 Merge pull request #14205 from GrahamDennis/gdennis/improve-dir-url-backcompat
Improved backwards compatibility hack for git URLs using dir=...
2025-10-15 15:20:02 +00:00
Robert Hensing a9d9b50b72 Merge remote-tracking branch 'upstream/master' into upstream-RossComputerGuy/feat/expose-computefsclosure 2025-10-15 15:40:10 +02:00
Robert Hensing 6fa03765ed C API: Propagate nix_store_realise build errors 2025-10-15 15:20:24 +02:00
Robert Hensing 12293a8b11 C API: Document nix_store_copy_closure flags 2025-10-15 15:05:50 +02:00
Robert Hensing 3fb943d130 C API: Make store realise tests multi-platform
... and improve assertions.
2025-10-15 14:55:28 +02:00
Robert Hensing aace1fb5d6 C API: test nix_store_get_fs_closure 2025-10-15 13:27:09 +02:00
John Ericson 606eb1dfb5 Merge pull request #14250 from fzakaria/patch-1
Remove duplicate shellcheck in dev-shell.nix
2025-10-15 05:03:19 +00:00
John Ericson e07754d888 Merge pull request #14251 from fzakaria/fzakaria/iwyu-libflake
Clean-up libflake headers
2025-10-15 04:27:07 +00:00
Farid Zakaria 01a8499d2f Format cpp files 2025-10-14 23:51:40 -04:00
Farid Zakaria e8b126fa90 Remove unecessary includes 2025-10-14 23:48:45 -04:00
Farid Zakaria 902faf4fe5 More fixes for iwyu 2025-10-14 23:20:35 -04:00
Farid Zakaria 7bc3d9b9a9 First attempt at uwyu for libflake 2025-10-14 22:53:13 -04:00
Farid Zakaria 092639709f Remove duplicate shellcheck in dev-shell.nix 2025-10-14 19:25:06 -07:00
John Ericson 620091bc8b Merge pull request #14223 from lovesegfault/curl-based-s3-tests
test(nixos): add comprehensive curl-based S3 VM tests
2025-10-14 23:08:55 +00:00
John Ericson 6dcc468253 Merge pull request #14249 from NixOS/more-to-real-path-cleanups
More toRealPath cleanups
2025-10-14 22:46:15 +00:00
Sergei Zimmerman 0347958dd2 nix/develop: Remove usage of toRealPath, replace with SourceAccessor 2025-10-15 00:52:13 +03:00
Sergei Zimmerman 918a3cebaa libexpr: Use Store::requireStoreObjectAccessor instead or toRealPath in fetch
This forces the code to go through proper abstractions instead of the raw filesystem
API.

This issue is evident from this reproducer:

nix eval --expr 'builtins.fetchurl { url = "https://example.com"; sha256 = ""; }' --json --eval-store "dummy://?read-only=false"

error:
       … while calling the 'fetchurl' builtin
         at «string»:1:1:
            1| builtins.fetchurl { url = "https://example.com"; sha256 = ""; }
             | ^

       error: opening file '/nix/store/r4f87yrl98f2m6v9z8ai2rbg4qwlcakq-example.com': No such file or directory
2025-10-15 00:27:41 +03:00
Sergei Zimmerman 69c005e805 libstore: Use getFSAccessor for store object in Worker::pathContentsGood
We only care about the accessor for a single store object anyway, but
the validity gets ignored. Also `pathExists(store.printStorePath(path))`
is definitely incorrect since it confuses the logical location vs physical
location in case of a chroot store.
2025-10-15 00:15:50 +03:00
Sergei Zimmerman 0c32fb3fa2 treewide: Add Store::requireStoreObjectAccessor, simplify uses of getFSAccessor
This is a simple wrapper around getFSAccessor that throws an InvalidPath
error. This simplifies usage in callsites that only care about getting
a non-null accessor.
2025-10-14 23:58:20 +03:00
Bernardo Meurer Costa d18f959d4f test(nixos): add comprehensive curl-based S3 VM tests
Add `curl-s3-binary-cache-store.nix` with comprehensive test coverage
for the curl-based S3 implementation.

Depends-On: #14206, #14222
2025-10-14 20:55:14 +00:00
Sergei Zimmerman 4041bfdb40 Merge pull request #14206 from lovesegfault/curl-based-s3-pieces
feat(libstore): add builtin fetchurl S3 credential pre-resolution
2025-10-14 20:10:41 +00:00
John Ericson 1fb4ff8c0e Merge pull request #14232 from roberth/dyndrv-messages
Better dyndrv messages
2025-10-14 15:40:27 +00:00
Robert Hensing 1b96a704d3 Add lazy evaluation for experimental feature reasons
Wrap fmt() calls in lambdas to defer string formatting until the
feature check fails. This avoids unnecessary string formatting in
the common case where the feature is enabled.

Addresses performance concern raised by xokdvium in PR review.
2025-10-14 16:49:59 +02:00
John Ericson 959c244a12 Merge pull request #14243 from NixOS/canon-path-nul-bytes
libutil: Ensure that CanonPath does not contain NUL bytes
2025-10-14 14:30:24 +00:00
Eelco Dolstra c44d2d5913 Merge pull request #14241 from NixOS/dependabot/github_actions/actions/create-github-app-token-2
build(deps): bump actions/create-github-app-token from 1 to 2
2025-10-14 11:55:43 +00:00
Eelco Dolstra dd590eca74 Merge pull request #14242 from NixOS/dependabot/github_actions/actions/checkout-5
build(deps): bump actions/checkout from 4 to 5
2025-10-14 11:55:25 +00:00
Sergei Zimmerman 1633ceaff2 libutil: Ensure that CanonPath does not contain NUL bytes
This, alongside the other invariants of the CanonPath is important
to uphold. std::filesystem happily crashes on NUL bytes in the constructor,
as we've seen with `path:%00` prior to c436b7a32a.
Best to stay clear of NUL bytes when we're talking about syscalls, especially
on Unix where strings are null terminated.

Very nice to have if we decide to switch over to pascal-style strings.
2025-10-14 02:33:42 +03:00
John Ericson 16e946bfb1 Merge pull request #14225 from obsidiansystems/derivation-resolution-goal-2
Reapply the rest of #14022
2025-10-13 23:26:29 +00:00
Sergei Zimmerman edf9163c22 libutil: Make CanonPath::root const
By all means CanonPath::root must be immutable. Let's enforce this with
in the code.
2025-10-14 02:24:40 +03:00
John Ericson ad893acf46 Fix ca/eval-store.sh test
The refactor in the last commit fixed the bug it was supposed to fix,
but introduced a new bug in that sometimes we tried to write a resolved
derivation to a store before all its `inputSrcs` were in that store.

The solution is to defer writing the derivation until inside
`DerivationBuildingGoal`, just before we do an actual build. At this
point, we are sure that all inputs in are the store.

This does have the side effect of meaning we don't write down the
resolved derivation in the substituting case, only the building case,
but I think that is actually fine. The store that actually does the
building should make a record of what it built by storing the resolved
derivation. Other stores that just substitute from that store don't
necessary want that derivation however. They can trust the substituter
to keep the record around, or baring that, they can attempt to re
resolve everything, if they need to be audited.

(cherry picked from commit c97b050a6c)
2025-10-13 18:41:59 -04:00
John Ericson 06bb1c2f93 Remove some buildMode default parameters
Force the internals to be more explicit.
2025-10-13 18:40:10 -04:00
John Ericson 2ee41976c2 Fix #13247
Resolve the derivation before creating a building goal, in a context
where we know what output(s) we want. That way we have a chance just to
download the outputs we want.

Fix #13247

(cherry picked from commit 39f6fd9b46)
2025-10-13 18:37:14 -04:00
dependabot[bot] b846f27682 build(deps): bump actions/checkout from 4 to 5
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-10-13 22:00:59 +00:00
dependabot[bot] 962862e9e0 build(deps): bump actions/create-github-app-token from 1 to 2
Bumps [actions/create-github-app-token](https://github.com/actions/create-github-app-token) from 1 to 2.
- [Release notes](https://github.com/actions/create-github-app-token/releases)
- [Commits](https://github.com/actions/create-github-app-token/compare/v1...v2)

---
updated-dependencies:
- dependency-name: actions/create-github-app-token
  dependency-version: '2'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-10-13 22:00:55 +00:00
Robert Hensing 39c4665488 Store reason as a field in MissingExperimentalFeature
Store the reason string as a field in the exception class rather than
only embedding it in the error message. This supports better structured
error handling and future JSON error reporting.

Suggested by Ericson2314 in PR review.
2025-10-13 23:49:20 +02:00
Robert Hensing 71aa9a4798 Add reasons to dyndrv xp messages 2025-10-13 23:49:20 +02:00
Robert Hensing 0fd890a8d6 Add reason string support to MissingExperimentalFeature 2025-10-13 23:49:20 +02:00
John Ericson 6642ffb506 Merge pull request #14239 from NixOS/asan-stack-overflow
libstore/outputs-spec: Drop usage of std::regex
2025-10-13 21:44:49 +00:00
Sergei Zimmerman 3ba221025f libstore/outputs-spec: Drop usage of std::regex
std::regex is a really bad tool for parsing things, since
it tends to overflow the stack pretty badly. See the build failure
under ASan in [^].

[^]: https://hydra.nixos.org/build/310077167/nixlog/5
2025-10-13 23:58:20 +03:00
Eelco Dolstra b56cc1808d Merge pull request #14237 from NixOS/url-parser-regression
Remove validation of URLs passed to FileTransferRequest verbatim
2025-10-13 20:01:01 +00:00
Sergei Zimmerman 47f427a172 Remove validation of URLs passed to FileTransferRequest verbatim
CURL is not very strict about validation of URLs passed to it. We
should reflect this in our handling of URLs that we get from the user
in <nix/fetchurl.nix> or builtins.fetchurl. ValidURL was an attempt to
rectify this, but it turned out to be too strict. The only good way to
resolve this is to pass (in some cases) the user-provided string verbatim
to CURL. Other usages in libfetchers still benefit from using structured
ParsedURL and validation though.

nix store prefetch-file --name foo 'https://cdn.skypack.dev/big.js@^5.2.2'
error: 'https://cdn.skypack.dev/big.js@^5.2.2' is not a valid URL: leftover
2025-10-13 22:23:26 +03:00
John Ericson 0f85ef3677 Merge pull request #14219 from lovesegfault/eval-copy-less
libstore: Avoid copying derivations to the store if they are already valid
2025-10-13 16:36:40 +00:00
Eelco Dolstra be2c9ef44c Merge pull request #14229 from NixOS/reduce-hydra-load
packaging/hydra: buildNoGC is the same as buildWithSanitizers
2025-10-13 16:22:30 +00:00
John Ericson d2c0c0607c Merge branch 'master' into eval-copy-less 2025-10-13 11:52:42 -04:00
John Ericson 480ce19011 Merge pull request #14233 from neuralsorcerer/fix-typo
Fix typos
2025-10-13 15:30:05 +00:00
John Ericson 3f876bcb61 Merge pull request #14231 from roberth/code-docs
Code docs
2025-10-13 15:05:30 +00:00
Soumyadip Sarkar 998f93f267 Fix typos 2025-10-13 18:15:52 +05:30
Robert Hensing 583f5e37fc Refactor: use optionalBracket in nix search 2025-10-13 14:02:59 +02:00
Robert Hensing 5dcfddf997 strings: Add optionalBracket helper 2025-10-13 13:59:39 +02:00
Robert Hensing 48a5e2dde2 EvalState: add doc comment 2025-10-13 13:14:05 +02:00
Robert Hensing 6db86389ce util/error: Document addTrace params
... and rename e -> pos. That was weird.
2025-10-13 12:57:22 +02:00
Bernardo Meurer Costa 000e6f6282 feat(libstore): add builtin fetchurl S3 credential pre-resolution
Add support for pre-resolving AWS credentials in the parent process
before forking for builtin:fetchurl. This avoids recreating credential
providers in the forked child process.
2025-10-12 23:01:13 +00:00
Bernardo Meurer CostaandSergei Zimmerman 18ec3d1094 libstore: Avoid copying derivations to the store if they are already valid
This avoids the quite costly copying of derivations to the daemon over the
wire in case it already exists in the eval store.

For a fresh instantiatation (after running nix-collect-garbage) this doesn't
significantly slow down eval:

taskset -c 2,3 hyperfine --reference "result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket" --prepare "nix-collect-garbage --store /tmp/store1111 --no-keep-derivations" "result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket"
Benchmark 1: result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     388.7 ms ±  10.5 ms    [User: 157.0 ms, System: 61.3 ms]
  Range (min … max):   379.4 ms … 415.9 ms    10 runs

Benchmark 2: result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     389.2 ms ±   4.8 ms    [User: 158.5 ms, System: 60.7 ms]
  Range (min … max):   381.2 ms … 397.6 ms    10 runs

But if the derivations are already instantiated this shows a pretty neat speedup:

taskset -c 2,3 hyperfine --reference "result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket" "result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket"
Benchmark 1: result-old/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     240.4 ms ±   3.1 ms    [User: 148.1 ms, System: 57.0 ms]
  Range (min … max):   233.8 ms … 245.0 ms    12 runs

Benchmark 2: result/bin/nix eval -f ../nixpkgs hello --store unix:///tmp/nix_socket
  Time (mean ± σ):     226.5 ms ±   4.5 ms    [User: 147.8 ms, System: 55.2 ms]
  Range (min … max):   214.0 ms … 231.2 ms    13 runs

Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-13 01:59:38 +03:00
John Ericson f77094715f Merge pull request #14222 from lovesegfault/curl-based-s3-fix-race
fix(libstore): fix race condition in AWS credential provider caching
2025-10-12 22:07:10 +00:00
Bernardo Meurer Costa f0e1f65260 fix(libstore): fix race condition in AWS credential provider caching
The previous implementation had a check-then-create race condition where
multiple threads could simultaneously:
1. Check the cache and find no provider (line 122)
2. Create their own providers (lines 126-145)
3. Insert into cache (line 161)

This resulted in multiple credential providers being created when
downloading multiple packages in parallel, as each .narinfo download
would trigger provider creation on its own thread.

Fix by using boost::concurrent_flat_map's try_emplace_and_cvisit, which
provides atomic get-or-create semantics:
- f1 callback: Called atomically during insertion, creates the provider
- f2 callback: Called if key exists, returns cached provider
- Other threads are blocked during f1, so no nullptr is ever visible
2025-10-12 20:16:02 +00:00
Sergei Zimmerman 89b35ec0dc packaging/hydra: buildNoGC is the same as buildWithSanitizers
This will reduce the load on hydra. It doesn't make sense to
build 2 slightly different variations where the difference
is only in the nix-perl-bindings and additional sanitizers.
2025-10-12 22:10:35 +03:00
Sergei Zimmerman 4f585dedbe Merge pull request #14228 from obsidiansystems/fix-windows-build
Fix windows build
2025-10-12 18:07:50 +00:00
John Ericson a01df8de21 Merge pull request #14227 from NixOS/asan-hydra
packaging: Add buildWithSanitizers to hydraJobs
2025-10-12 17:43:54 +00:00
John Ericson 10223fae86 Fix windows build
I forget to add some CPP in b57caaa1a2.

Hopefully, as we relyon RAII more, these explicit resets become
unneeded.
2025-10-12 13:22:35 -04:00
John Ericson 9150ccb89e Fix Windows dev shell (mostly)
gbenchmark still has too-narrow supported systems, however. That needs
to be fixed in Nixpkgs.
2025-10-12 13:16:50 -04:00
Sergei Zimmerman de75a180cb packaging: Add buildWithSanitizers to hydraJobs 2025-10-12 19:38:01 +03:00
Sergei Zimmerman a491173369 packaging: Add withASan,withUBSan options to the scope 2025-10-12 19:16:08 +03:00
Sergei Zimmerman 199b6ff3fb Disable detect_odr_violation for ASan
There's some unfortunate ODR violations that get dianosed with GCC but not Clang
for static inline constexpr variables defined inside the class body:

template<typename T>
struct static_const
{
    static JSON_INLINE_VARIABLE constexpr T value{};
};

This can be ignored pretty much. There is the same problem for std::piecewise_construct:

http://lists.boost.org/Archives/boost/2007/06/123353.php

==2455704==ERROR: AddressSanitizer: odr-violation (0x7efddc460e20):
  [1] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/pkmljfq97a83dbanr0n64zbm8cyhna33-nix-store-2.33.0pre/lib/libnixstore.so.2.33.0
  [2] size=1 'value' /nix/store/235hvgzcbl06fxy53515q8sr6lljvf68-nlohmann_json-3.11.3/include/nlohmann/detail/meta/cpp_future.hpp:156:45 in /nix/store/gbjpkjj0g8vk20fzlyrwj491gwp6g1qw-nix-util-2.33.0pre/lib/libnixutil.so.2.33.0
2025-10-12 19:16:07 +03:00
Sergei Zimmerman 711e738bf9 meson: Simplify asan-options handling even more
Instead of specifying env variables all the time
we can instead embed the __asan_default_options symbol
in all executables / shared objects. This reduces code
duplication.
2025-10-12 19:16:06 +03:00
Sergei Zimmerman d9cabddd17 Merge pull request #14214 from obsidiansystems/derivation-resolution-goal
Split out `DerivationResolutionGoal`
2025-10-12 11:31:54 +00:00
John Ericson 0da430be35 Split out DerivationResolutionGoal
This prepares the way for fixing a few issues.

Take 2: was landed before in 8f4a739d0f.
2025-10-11 19:52:37 -04:00
John Ericson 07df87652c Make keys of Derivation*Goal more legible
The property that substitution goals come first is still preserved
2025-10-11 19:50:45 -04:00
John Ericson a629ce3dec Use member initializer list for Derivation*Goal::drv 2025-10-11 18:37:04 -04:00
John Ericson ba7bbcd1da Cleanup Derivation*Goal names 2025-10-11 18:30:47 -04:00
Sergei Zimmerman 97e770ad01 Merge pull request #14209 from lovesegfault/curl-based-s3-pieces-fix
fix(libstore): improve http-binary-cache-store S3 compatibility
2025-10-11 21:19:18 +00:00
John Ericson 682cf0b266 Merge pull request #14216 from NixOS/simplify-asan-meson
meson: Move asan-options to common
2025-10-11 18:21:24 +00:00
John Ericson f1cb837888 Merge pull request #14217 from NixOS/remove-libgit2-patches
packaging: Remove no longer necessary libgit2 patches
2025-10-11 18:20:36 +00:00
Sergei Zimmerman 47705139c9 packaging: Remove no longer necessary libgit2 patches
25.05 already has 1.9.0 and we don't support older nixpkgs versions.
2025-10-11 16:30:55 +03:00
Sergei Zimmerman d26a337c09 meson: Move asan-options to common
This way we don't have to duplicate the subdir everywhere.
Less copy-pasta is good.
2025-10-11 16:08:35 +03:00
John Ericson 8064c75694 Merge pull request #14194 from xokdvium/stacktrace-assertion
libutil: Print stack trace on assertion failure
2025-10-10 23:25:16 +00:00
John Ericson d75614a315 Merge pull request #14208 from obsidiansystems/consolidate-builder-dispatch
Consolidate logic choosing where we can/should build a bit
2025-10-10 23:05:40 +00:00
Sergei Zimmerman 46382ade74 libutil: Print stack trace on assertion failure
This change overrides __assert_fail on glibc/musl
to instead call std::terminate that we have a custom
handler for. This ensures that we have more context
to diagnose issues encountered by users in the wild.
2025-10-11 01:35:58 +03:00
John Ericson 9cb686f816 Merge pull request #14210 from NixOS/double-quotes
libstore: Fix double-quoting of paths in logs
2025-10-10 22:10:36 +00:00
Bernardo Meurer Costa f02218873e fix(libstore): improve http-binary-cache-store S3 compatibility
This commit adds two key fixes to http-binary-cache-store.cc to
properly support the new curl-based S3 implementation:

1. **Consistent cache key handling**: Use `getReference().render(withParams=false)`
   for disk cache keys instead of `cacheUri.to_string()`. This ensures cache
   keys are consistent with the S3 implementation and don't include query
   parameters, which matches the behavior expected by Store::queryPathInfo()
   lookups.

2. **S3 query parameter preservation**: When generating file transfer requests
   for S3 URLs, preserve query parameters from the base URL (region, endpoint,
   etc.) when the relative path doesn't have its own query parameters. This
   ensures S3-specific configuration is propagated to all requests.
2025-10-10 22:05:57 +00:00
John Ericson b57caaa1a2 Consolidate logic choosing where we can/should build a bit
I want to separate "policy" from "mechanism".

Now the logic to decide how to build (a policy choice, though with some
hard constraints) is all in derivation building goal, and all in the
same spot. build hook, external builder, or local builder --- the choice
between all three is made in the same spot --- pure policy.

Now, if you want to use the external deriation builder, you simply
provide the `ExternalBuilder` you wish to use, and there is no
additional checking --- pure mechanism. It is the responsibility of the
caller to choose an external builder that works for the derivation in
question.

Also, `checkSystem()` was the only thing throwing `BuildError` from
`startBuilder`. Now that that is gone, we can now remove the
`try...catch` around that.
2025-10-10 17:28:57 -04:00
John EricsonandSergei Zimmerman 2ff59ec3e0 Use std::ranges::find_if for finding external builders
Co-authored-by: Sergei Zimmerman <sergei@zimmerman.foo>
2025-10-10 17:27:41 -04:00
Sergei Zimmerman f30cb8667b libstore: Fix double-quoting of paths in logs
std::filesystem::path is already quoted by boost::format with double quotes (").
2025-10-11 00:25:51 +03:00
John Ericson b56dd21c31 Settings::ExternalBuilder::systems make set
Nothing cares about the order, actually.
2025-10-10 17:23:45 -04:00
Sergei Zimmerman 2308aaf192 Merge pull request #14198 from lovesegfault/curl-based-s3-pieces
feat(libstore): add curl-based S3 store implementation
2025-10-10 07:55:44 +00:00
Graham Dennis 8d9e9bc400 Improve comment 2025-10-10 15:00:10 +11:00
Graham Dennis 43b01b6790 Improved backwards compatibility hack for git URLs using dir=... attribute 2025-10-10 14:54:47 +11:00
Bernardo Meurer Costa 0855b715a9 feat(libstore): add curl-based S3 store implementation
Add a new S3BinaryCacheStore implementation that inherits from
HttpBinaryCacheStore.

The implementation is activated with NIX_WITH_CURL_S3, keeping the
existing NIX_WITH_S3_SUPPORT (AWS SDK) implementation unchanged.
2025-10-10 00:13:29 +00:00
John Ericson 2fc88ec114 Merge pull request #14204 from NixOS/kill-to-real-path-fetchers
libfetchers: Remove toRealPath in SourceHutInputScheme::getRevFromRef
2025-10-09 22:01:03 +00:00
Sergei Zimmerman 9c8480becb Merge pull request #14203 from roberth/move-eval-cache-open-to-libflake
Move eval cache open to libflake
2025-10-09 21:42:06 +00:00
Sergei Zimmerman c58acff42a libfetchers: Remove toRealPath in SourceHutInputScheme::getRevFromRef
This code had several issues:

1. Not going through the SourceAccessor means that we can only work
   with physical paths.

2. It did not actually check that the file exists. (std::ifstream does not check
   it by default).
2025-10-10 00:26:57 +03:00
Robert Hensing abcceafbce Use const for lock in openEvalCache 2025-10-09 21:25:40 +02:00
Robert Hensing 42c9cbf9ca Use ref<LockedFlake> where non-null 2025-10-09 21:25:40 +02:00
Robert Hensing 0387b7d6db Move openEvalCache to libflake
Most of the eval cache logic is flake-independent and libexpr,
but the loading part is not.
`nix-flake` is the right component for this, as the eval cache
isn't exactly specific to the command line.
2025-10-09 20:40:40 +02:00
Sergei Zimmerman dfafd8bc38 Merge pull request #14199 from getchoo-contrib/getchoo/toml11-fix
packaging: only override `toml11` when necessary
2025-10-09 17:20:35 +00:00
Sergei Zimmerman 44486871e9 Merge pull request #14200 from Mic92/less-ci-builds
only build on push to master
2025-10-09 15:07:06 +00:00
Jörg Thalheim 118acc84ba only build on push to master
we have now merge queues for maintainance branches. We still build it
for master to have our installer beeing updated. In future this part
could go in new workflow instead.
2025-10-09 14:16:14 +01:00
Seth Flynn 0f016f9bf5 packaging: only override toml11 when necessary
v4.4.0 hit Nixpkgs in https://github.com/NixOS/nixpkgs/pull/442682.
Ideally we'd just use that, but this keeps the fallback behavior until
it's more widespread
2025-10-09 03:24:57 -04:00
John Ericson ce38b46e06 Merge pull request #14170 from lovesegfault/curl-based-s3-pieces
feat(libstore/filetransfer): add S3 signing support
2025-10-08 22:35:32 +00:00
Sergei Zimmerman 33d9270109 Merge pull request #14191 from NixOS/fix-14188
libutil: Fix renderAuthorityAndPath unreachable for path:/ URLs
2025-10-08 21:34:28 +00:00
Bernardo Meurer Costa 00c2a57666 feat(libstore/filetransfer): add S3 signing support 2025-10-08 21:31:34 +00:00
Eelco Dolstra d591f17ecb Merge pull request #14189 from NixOS/fix-exportReferencesGraph
exportReferencesGraph: Handle heterogeneous arrays
2025-10-08 21:19:30 +00:00
John Ericson bb1945a090 Merge pull request #14182 from NixOS/simplify-archive-tests
tests: Move invalid nar tests from tests/functional to libutil-tests
2025-10-08 20:46:37 +00:00
Eelco Dolstra 94f410b628 exportReferencesGraph: Handle heterogeneous arrays
This barfed with

   error: [json.exception.type_error.302] type must be string, but is array

on `nix build github:malt3/bazel-env#bazel-env` because it has a `exportReferencesGraph` with a value like `["string",...["string"]]`.
2025-10-08 22:15:33 +02:00
Bernardo Meurer Costa 3c1e2e56ea feat(libstore/filetransfer): add username/password authentication support
Add a `UsernameAuth` struct and optional `usernameAuth` field to
`FileTransferRequest` to support programmatic username/password
authentication.

This uses curl's `CURLOPT_USERNAME`/`CURLOPT_PASSWORD` options, which
works with multiple protocols (HTTP, FTP, etc.) and is not specific to
any particular authentication scheme.

The primary motivation is to enable S3 authentication refactoring where
AWS credentials (access key ID and secret access key) can be passed
through this general-purpose mechanism, reducing the amount of
S3-specific code behind `#if NIX_WITH_CURL_S3` guards.
2025-10-08 20:10:53 +00:00
Eelco Dolstra 925f10d5ea Merge pull request #14087 from NixOS/required-args
Args::Flag: Add `required` attribute
2025-10-08 19:33:22 +00:00
Sergei Zimmerman 1d8dd77e1d libutil: Fix renderAuthorityAndPath unreachable for path:/ URLs
This was mistakenly triggered by path:/ URL, since the `//` would
correspond to 3 empty segments.
2025-10-08 22:14:49 +03:00
Jörg Thalheim 090f7fb05e Merge pull request #14002 from getchoo-contrib/getchoo/dogfood-experimental-installer
Add experimental installer to installer tests
2025-10-08 04:05:46 +00:00
Jörg Thalheim 2ce343716d Merge pull request #14169 from NixOS/dev-shell-no-separate-debug-info
dev-shell: Disable separateDebugInfo
2025-10-08 04:03:43 +00:00
Jörg Thalheim e69e621578 Merge pull request #14180 from NixOS/release-process-drop-mergify
maintainers: Remove mergify note from release-process.md
2025-10-08 03:34:05 +00:00
Jörg Thalheim d2a0a11a8e Merge pull request #14184 from lovesegfault/nix-better-ci
ci: integrate vm_tests into main tests job
2025-10-08 05:23:53 +02:00
Bernardo Meurer Costa a400ea4257 ci: integrate vm_tests into main tests job
This consolidates the separate vm_tests job into the main tests job,
simplifying the CI workflow. VM tests now run as part of the regular
test matrix.
2025-10-08 02:46:56 +00:00
Sergei Zimmerman fc8b784924 Merge pull request #14147 from lovesegfault/nix-multi-ci
ci: test without s3 and with curl-based-s3
2025-10-08 02:38:03 +03:00
Sergei Zimmerman 0619351326 tests: Move invalid nar tests from tests/functional to libutil-tests
Since 242f362567 we have better infrastructure
for this kind of tests.
2025-10-08 02:02:33 +03:00
Sergei Zimmerman c5b88c22fa dev-shell: Disable separateDebugInfo
This breaks gdb pretty-printers inserted into .debug_gdb_scripts section,
because it implies --compress-debug-sections=zlib, -Wa,--compress-debug-sections.
This is very unfortunate, because then gdb can't use pretty printers for
Boost.Unordered (which are very useful, since boost::unoredred_flat_map is
impossible to debug). This seems perfectly fine to disable in the dev-shell for
the time being.

See [1-3] for further references.

With this change I'm able to use boost's pretty-printers out-of-the box:

```
p *importResolutionCache
$2 = boost::concurrent_flat_map with 1 elements = {[{accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
        get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
      path = "/derivation-internal.nix"}}] = {accessor = {p = std::shared_ptr<nix::SourceAccessor> (use count 5, weak count 1) = {
        get() = 0x555555d830a8}}, path = {static root = {static root = <same as static member of an already seen type>, path = "/"},
      path = "/derivation-internal.nix"}}}
```

When combined with a simple `add-auto-load-safe-path ~/code` in .gdbinit

[1]: https://gerrit.lix.systems/c/lix/+/3880
[2]: https://git.lix.systems/lix-project/lix/issues/1003
[3]: https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
2025-10-08 00:57:34 +03:00
Sergei Zimmerman 75b18a6e47 maintainers: Remove mergify note from release-process.md 2025-10-08 00:51:50 +03:00
Sergei Zimmerman fb117e0cac Merge pull request #14155 from Mic92/backport-action
ci: Switch away from mergify to backport action
2025-10-08 00:32:00 +03:00
Jörg Thalheim 63e8b5f94a ci: Switch away from mergify to backport action
We want to use github native queues.
2025-10-07 23:43:03 +03:00
Eelco Dolstra 559061bf5a Merge pull request #14173 from NixOS/bump-2.33.0
Bump version
2025-10-07 20:42:25 +02:00
Eelco Dolstra b0f567e18b Update mergify.yml 2025-10-07 17:16:57 +02:00
Eelco Dolstra 875f7ec25f Bump version 2025-10-07 17:15:28 +02:00
Bernardo Meurer Costa 8c28283876 ci: test without s3 and with curl-based-s3 2025-10-06 16:24:21 +00:00
Bernardo Meurer Costa 7f22a40e3b build(libstore): assert withAWS xor withCurlS3 2025-10-06 16:22:01 +00:00
Tristan Ross 9abcc68ad1 libstore-c: add nix_store_get_fs_closure 2025-10-06 09:12:02 -07:00
Seth Flynn 92d7381826 ci: allow for using the latest build of the experimental installer
Until these repos are potentially merged, this is good for dogfooding
alongside the experimental installer. It also uses the more official
`artifacts.nixos.org` endpoint to install stable releases now

More immediately though, we need a patch for the experimental installer
to really work in CI at all, and that hasn't landed in a tag yet. So,
this lets us use it right from `main`!
2025-10-03 03:37:59 -04:00
Seth Flynn d2293fb458 ci: enable experimental installer tests 2025-10-03 01:26:55 -04:00
Seth Flynn 2cbbb63628 ci: enable use of the experimental installer 2025-10-03 01:26:52 -04:00
Eelco Dolstra eec4dece33 Args::Flag: Add required attribute 2025-09-26 16:01:59 +02:00
332 changed files with 7174 additions and 4745 deletions

No files matched your search

+14
View File
@@ -0,0 +1,14 @@
# Disable CodeRabbit auto-review to prevent verbose comments on PRs.
# When enabled: false, CodeRabbit won't attempt reviews and won't post
# "Review skipped" or other automated comments.
reviews:
auto_review:
enabled: false
review_status: false
high_level_summary: false
poem: false
sequence_diagrams: false
changed_files_summary: false
tools:
github-checks:
enabled: false
+82 -1
View File
@@ -4,15 +4,29 @@ inputs:
dogfood:
description: "Whether to use Nix installed from the latest artifact from master branch"
required: true # Be explicit about the fact that we are using unreleased artifacts
experimental-installer:
description: "Whether to use the experimental installer to install Nix"
default: false
experimental-installer-version:
description: "Version of the experimental installer to use. If `latest`, the newest artifact from the default branch is used."
# TODO: This should probably be pinned to a release after https://github.com/NixOS/experimental-nix-installer/pull/49 lands in one
default: "latest"
extra_nix_config:
description: "Gets appended to `/etc/nix/nix.conf` if passed."
install_url:
description: "URL of the Nix installer"
required: false
default: "https://releases.nixos.org/nix/nix-2.30.2/install"
default: "https://releases.nixos.org/nix/nix-2.32.1/install"
tarball_url:
description: "URL of the Nix tarball to use with the experimental installer"
required: false
github_token:
description: "Github token"
required: true
use_cache:
description: "Whether to setup magic-nix-cache"
default: true
required: false
runs:
using: "composite"
steps:
@@ -37,14 +51,81 @@ runs:
gh run download "$RUN_ID" --repo "$DOGFOOD_REPO" -n "$INSTALLER_ARTIFACT" -D "$INSTALLER_DOWNLOAD_DIR"
echo "installer-path=file://$INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$INSTALLER_DOWNLOAD_DIR" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
echo "::notice ::Dogfooding Nix installer from master (https://github.com/$DOGFOOD_REPO/actions/runs/$RUN_ID)"
env:
GH_TOKEN: ${{ inputs.github_token }}
DOGFOOD_REPO: "NixOS/nix"
- name: "Gather system info for experimental installer"
shell: bash
if: ${{ inputs.experimental-installer == 'true' }}
run: |
echo "::notice Using experimental installer from $EXPERIMENTAL_INSTALLER_REPO (https://github.com/$EXPERIMENTAL_INSTALLER_REPO)"
if [ "$RUNNER_OS" == "Linux" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="linux"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
elif [ "$RUNNER_OS" == "macOS" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="darwin"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_OS: $RUNNER_OS"
exit 1
fi
if [ "$RUNNER_ARCH" == "X64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=x86_64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
elif [ "$RUNNER_ARCH" == "ARM64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=aarch64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_ARCH: $RUNNER_ARCH"
exit 1
fi
echo "EXPERIMENTAL_INSTALLER_ARTIFACT=nix-installer-$EXPERIMENTAL_INSTALLER_ARCH-$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
env:
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- name: "Download latest experimental installer"
shell: bash
id: download-latest-experimental-installer
if: ${{ inputs.experimental-installer == 'true' && inputs.experimental-installer-version == 'latest' }}
run: |
RUN_ID=$(gh run list --repo "$EXPERIMENTAL_INSTALLER_REPO" --workflow ci.yml --branch main --status success --json databaseId --jq ".[0].databaseId")
EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR="$GITHUB_WORKSPACE/$EXPERIMENTAL_INSTALLER_ARTIFACT"
mkdir -p "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
gh run download "$RUN_ID" --repo "$EXPERIMENTAL_INSTALLER_REPO" -n "$EXPERIMENTAL_INSTALLER_ARTIFACT" -D "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
# Executable permissions are lost in artifacts
find $EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR -type f -exec chmod +x {} +
echo "installer-path=$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
env:
GH_TOKEN: ${{ inputs.github_token }}
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- uses: cachix/install-nix-action@c134e4c9e34bac6cab09cf239815f9339aaaf84e # v31.5.1
if: ${{ inputs.experimental-installer != 'true' }}
with:
# Ternary operator in GHA: https://www.github.com/actions/runner/issues/409#issuecomment-752775072
install_url: ${{ inputs.dogfood == 'true' && format('{0}/install', steps.download-nix-installer.outputs.installer-path) || inputs.install_url }}
install_options: ${{ inputs.dogfood == 'true' && format('--tarball-url-prefix {0}', steps.download-nix-installer.outputs.installer-path) || '' }}
extra_nix_config: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/nix-installer-action@786fff0690178f1234e4e1fe9b536e94f5433196 # v20
if: ${{ inputs.experimental-installer == 'true' }}
with:
diagnostic-endpoint: ""
# TODO: It'd be nice to use `artifacts.nixos.org` for both of these, maybe through an `/experimental-installer/latest` endpoint? or `/commit/<hash>`?
local-root: ${{ inputs.experimental-installer-version == 'latest' && steps.download-latest-experimental-installer.outputs.installer-path || '' }}
source-url: ${{ inputs.experimental-installer-version != 'latest' && 'https://artifacts.nixos.org/experimental-installer/tag/${{ inputs.experimental-installer-version }}/${{ env.EXPERIMENTAL_INSTALLER_ARTIFACT }}' || '' }}
nix-package-url: ${{ inputs.dogfood == 'true' && steps.download-nix-installer.outputs.tarball-path || (inputs.tarball_url || '') }}
extra-conf: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 # v13
if: ${{ inputs.use_cache == 'true' }}
with:
diagnostic-endpoint: ''
use-flakehub: false
use-gha-cache: true
source-revision: 92d9581367be2233c2d5714a2640e1339f4087d8 # main
+37
View File
@@ -0,0 +1,37 @@
name: Backport
on:
pull_request_target:
types: [closed, labeled]
permissions:
contents: read
jobs:
backport:
name: Backport Pull Request
permissions:
# for korthout/backport-action
contents: write
pull-requests: write
if: github.repository_owner == 'NixOS' && github.event.pull_request.merged == true && (github.event_name != 'labeled' || startsWith('backport', github.event.label.name))
runs-on: ubuntu-24.04-arm
steps:
- name: Generate GitHub App token
id: generate-token
uses: actions/create-github-app-token@v2
with:
app-id: ${{ vars.CI_APP_ID }}
private-key: ${{ secrets.CI_APP_PRIVATE_KEY }}
- uses: actions/checkout@v5
with:
ref: ${{ github.event.pull_request.head.sha }}
# required to find all branches
fetch-depth: 0
- name: Create backport PRs
uses: korthout/backport-action@d07416681cab29bf2661702f925f020aaa962997 # v3.4.1
id: backport
with:
# Config README: https://github.com/korthout/backport-action#backport-action
github_token: ${{ steps.generate-token.outputs.token }}
github_workspace: ${{ github.workspace }}
auto_merge_enabled: true
pull_description: |-
Automatic backport to `${target_branch}`, triggered by a label in #${pull_number}.
+44 -33
View File
@@ -4,6 +4,8 @@ on:
pull_request:
merge_group:
push:
branches:
- master
workflow_dispatch:
inputs:
dogfood:
@@ -12,6 +14,10 @@ on:
default: true
type: boolean
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions: read-all
jobs:
@@ -27,6 +33,7 @@ jobs:
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
use_cache: false
- run: nix flake show --all-systems --json
pre-commit-checks:
@@ -39,7 +46,6 @@ jobs:
dogfood: ${{ github.event_name == 'workflow_dispatch' && inputs.dogfood || github.event_name != 'workflow_dispatch' }}
extra_nix_config: experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: ./ci/gha/tests/pre-commit-checks
basic-checks:
@@ -90,7 +96,6 @@ jobs:
dogfood: ${{ github.event_name == 'workflow_dispatch' && inputs.dogfood || github.event_name != 'workflow_dispatch' }}
# The sandbox would otherwise be disabled by default on Darwin
extra_nix_config: "sandbox = true"
- uses: DeterminateSystems/magic-nix-cache-action@main
# Since ubuntu 22.30, unprivileged usernamespaces are no longer allowed to map to the root user:
# https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
- run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
@@ -100,6 +105,12 @@ jobs:
nix build --file ci/gha/tests/wrapper.nix componentTests -L \
--arg withInstrumentation ${{ matrix.instrumented }} \
--argstr stdenv "${{ matrix.stdenv }}"
- name: Run VM tests
run: |
nix build --file ci/gha/tests/wrapper.nix vmTests -L \
--arg withInstrumentation ${{ matrix.instrumented }} \
--argstr stdenv "${{ matrix.stdenv }}"
if: ${{ matrix.os == 'linux' }}
- name: Run flake checks and prepare the installer tarball
run: |
ci/gha/tests/build-checks
@@ -135,9 +146,19 @@ jobs:
- scenario: on ubuntu
runs-on: ubuntu-24.04
os: linux
experimental-installer: false
- scenario: on macos
runs-on: macos-14
os: darwin
experimental-installer: false
- scenario: on ubuntu (experimental)
runs-on: ubuntu-24.04
os: linux
experimental-installer: true
- scenario: on macos (experimental)
runs-on: macos-14
os: darwin
experimental-installer: true
name: installer test ${{ matrix.scenario }}
runs-on: ${{ matrix.runs-on }}
steps:
@@ -149,11 +170,22 @@ jobs:
path: out
- name: Looking up the installer tarball URL
id: installer-tarball-url
run: echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
- uses: cachix/install-nix-action@v31
run: |
echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$GITHUB_WORKSPACE/out" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
- uses: cachix/install-nix-action@c134e4c9e34bac6cab09cf239815f9339aaaf84e # v31.5.1
if: ${{ !matrix.experimental-installer }}
with:
install_url: ${{ format('{0}/install', steps.installer-tarball-url.outputs.installer-url) }}
install_options: ${{ format('--tarball-url-prefix {0}', steps.installer-tarball-url.outputs.installer-url) }}
- uses: ./.github/actions/install-nix-action
if: ${{ matrix.experimental-installer }}
with:
dogfood: false
experimental-installer: true
tarball_url: ${{ steps.installer-tarball-url.outputs.tarball-path }}
github_token: ${{ secrets.GITHUB_TOKEN }}
- run: sudo apt install fish zsh
if: matrix.os == 'linux'
- run: brew install fish
@@ -185,7 +217,7 @@ jobs:
echo "docker=${{ env._DOCKER_SECRETS != '' }}" >> $GITHUB_OUTPUT
docker_push_image:
needs: [tests, vm_tests, check_secrets]
needs: [tests, check_secrets]
permissions:
contents: read
packages: write
@@ -198,12 +230,13 @@ jobs:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v31
- uses: ./.github/actions/install-nix-action
with:
install_url: https://releases.nixos.org/nix/nix-2.20.3/install
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: echo NIX_VERSION="$(nix --experimental-features 'nix-command flakes' eval .\#nix.version | tr -d \")" >> $GITHUB_ENV
- run: nix --experimental-features 'nix-command flakes' build .#dockerImage -L
dogfood: false
extra_nix_config: |
experimental-features = flakes nix-command
- run: echo NIX_VERSION="$(nix eval .\#nix.version | tr -d \")" >> $GITHUB_ENV
- run: nix build .#dockerImage -L
- run: docker load -i ./result/image.tar.gz
- run: docker tag nix:$NIX_VERSION ${{ secrets.DOCKERHUB_USERNAME }}/nix:$NIX_VERSION
- run: docker tag nix:$NIX_VERSION ${{ secrets.DOCKERHUB_USERNAME }}/nix:master
@@ -238,28 +271,8 @@ jobs:
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
vm_tests:
needs: basic-checks
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
- uses: ./.github/actions/install-nix-action
with:
dogfood: ${{ github.event_name == 'workflow_dispatch' && inputs.dogfood || github.event_name != 'workflow_dispatch' }}
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: |
nix build -L \
.#hydraJobs.tests.functional_user \
.#hydraJobs.tests.githubFlakes \
.#hydraJobs.tests.nix-docker \
.#hydraJobs.tests.tarballFlakes \
;
flake_regressions:
needs: vm_tests
needs: tests
runs-on: ubuntu-24.04
steps:
- name: Checkout nix
@@ -280,7 +293,6 @@ jobs:
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: nix build -L --out-link ./new-nix && PATH=$(pwd)/new-nix/bin:$PATH MAX_FLAKES=25 flake-regressions/eval-all.sh
profile_build:
@@ -301,7 +313,6 @@ jobs:
extra_nix_config: |
experimental-features = flakes nix-command ca-derivations impure-derivations
max-jobs = 1
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: |
nix build -L --file ./ci/gha/profile-build buildTimeReport --out-link build-time-report.md
cat build-time-report.md >> $GITHUB_STEP_SUMMARY
+35
View File
@@ -0,0 +1,35 @@
name: "Coverity Scan"
on:
# FIXME: This is here for testing, should be removed before merging
pull_request:
push:
branches:
- master
- coverity-test
permissions: read-all
jobs:
coverity:
name: Coverity static analysis
runs-on: ubuntu-24.04
timeout-minutes: 120
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
extra_nix_config: experimental-features = nix-command flakes
- name: configure
run: nix develop --configure
- uses: vapier/coverity-scan-action@v1
with:
project: NixOS/nix
email: ${{ secrets.COVERITY_SCAN_EMAIL }}
token: ${{ secrets.COVERITY_SCAN_TOKEN }}
build_language: cxx
command: nix develop --build
version: ${{ github.sha }}
-174
View File
@@ -1,174 +0,0 @@
queue_rules:
- name: default
# all required tests need to go here
merge_conditions:
- check-success=tests on macos
- check-success=tests on ubuntu
- check-success=installer test on macos
- check-success=installer test on ubuntu
- check-success=vm_tests
batch_size: 5
pull_request_rules:
- name: merge using the merge queue
conditions:
- base~=master|.+-maintenance
- label~=merge-queue|dependencies
actions:
queue: {}
# The rules below will first create backport pull requests and put those in a merge queue.
- name: backport patches to 2.18
conditions:
- label=backport 2.18-maintenance
actions:
backport:
branches:
- 2.18-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.19
conditions:
- label=backport 2.19-maintenance
actions:
backport:
branches:
- 2.19-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.20
conditions:
- label=backport 2.20-maintenance
actions:
backport:
branches:
- 2.20-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.21
conditions:
- label=backport 2.21-maintenance
actions:
backport:
branches:
- 2.21-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.22
conditions:
- label=backport 2.22-maintenance
actions:
backport:
branches:
- 2.22-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.23
conditions:
- label=backport 2.23-maintenance
actions:
backport:
branches:
- 2.23-maintenance
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.24
conditions:
- label=backport 2.24-maintenance
actions:
backport:
branches:
- "2.24-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.25
conditions:
- label=backport 2.25-maintenance
actions:
backport:
branches:
- "2.25-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.26
conditions:
- label=backport 2.26-maintenance
actions:
backport:
branches:
- "2.26-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.27
conditions:
- label=backport 2.27-maintenance
actions:
backport:
branches:
- "2.27-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.28
conditions:
- label=backport 2.28-maintenance
actions:
backport:
branches:
- "2.28-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.29
conditions:
- label=backport 2.29-maintenance
actions:
backport:
branches:
- "2.29-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.30
conditions:
- label=backport 2.30-maintenance
actions:
backport:
branches:
- "2.30-maintenance"
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.31
conditions:
- label=backport 2.31-maintenance
actions:
backport:
branches:
- "2.31-maintenance"
labels:
- automatic backport
- merge-queue
+1 -1
View File
@@ -1 +1 @@
2.32.0
2.33.0
+29 -13
View File
@@ -21,16 +21,6 @@ let
packages' = nixFlake.packages.${system};
stdenv = (getStdenv pkgs);
enableSanitizersLayer = finalAttrs: prevAttrs: {
mesonFlags =
(prevAttrs.mesonFlags or [ ])
++ [ (lib.mesonOption "b_sanitize" "address,undefined") ]
++ (lib.optionals stdenv.cc.isClang [
# https://www.github.com/mesonbuild/meson/issues/764
(lib.mesonBool "b_lundef" false)
]);
};
collectCoverageLayer = finalAttrs: prevAttrs: {
env =
let
@@ -53,14 +43,15 @@ let
'';
};
componentOverrides =
(lib.optional withSanitizers enableSanitizersLayer)
++ (lib.optional withCoverage collectCoverageLayer);
componentOverrides = (lib.optional withCoverage collectCoverageLayer);
in
rec {
nixComponentsInstrumented = nixComponents.overrideScope (
final: prev: {
withASan = withSanitizers;
withUBSan = withSanitizers;
nix-store-tests = prev.nix-store-tests.override { withBenchmarks = true; };
# Boehm is incompatible with ASAN.
nix-expr = prev.nix-expr.override { enableGC = !withSanitizers; };
@@ -71,6 +62,14 @@ rec {
}
);
# Import NixOS tests using the instrumented components
nixosTests = import ../../../tests/nixos {
inherit lib pkgs;
nixComponents = nixComponentsInstrumented;
nixpkgs = nixFlake.inputs.nixpkgs;
inherit (nixFlake.inputs) nixpkgs-23-11;
};
/**
Top-level tests for the flake outputs, as they would be built by hydra.
These tests generally can't be overridden to run with sanitizers.
@@ -117,6 +116,7 @@ rec {
) nixComponentsInstrumented)
// lib.optionalAttrs (pkgs.stdenv.hostPlatform == pkgs.stdenv.buildPlatform) {
"${componentTestsPrefix}nix-functional-tests" = nixComponentsInstrumented.nix-functional-tests;
"${componentTestsPrefix}nix-json-schema-checks" = nixComponentsInstrumented.nix-json-schema-checks;
};
codeCoverage =
@@ -221,4 +221,20 @@ rec {
{
inherit coverageProfileDrvs mergedProfdata coverageReports;
};
vmTests = {
inherit (nixosTests) s3-binary-cache-store;
}
// lib.optionalAttrs (!withSanitizers && !withCoverage) {
# evalNixpkgs uses non-instrumented components from hydraJobs, so only run it
# when not testing with sanitizers to avoid rebuilding nix
inherit (hydraJobs.tests) evalNixpkgs;
# FIXME: CI times out when building vm tests instrumented
inherit (nixosTests)
functional_user
githubFlakes
nix-docker
tarballFlakes
;
};
}
+2 -2
View File
@@ -15,7 +15,6 @@ pymod = import('python')
python = pymod.find_installation('python3')
nix_env_for_docs = {
'ASAN_OPTIONS' : 'abort_on_error=1:print_summary=1:detect_leaks=0',
'HOME' : '/dummy',
'NIX_CONF_DIR' : '/dummy',
'NIX_SSL_CERT_FILE' : '/dummy/no-ca-bundle.crt',
@@ -89,7 +88,7 @@ manual = custom_target(
@0@ @INPUT0@ @CURRENT_SOURCE_DIR@ > @DEPFILE@
@0@ @INPUT1@ summary @2@ < @CURRENT_SOURCE_DIR@/source/SUMMARY.md.in > @2@/source/SUMMARY.md
sed -e 's|@version@|@3@|g' < @INPUT2@ > @2@/book.toml
@4@ -r --include='*.md' @CURRENT_SOURCE_DIR@/ @2@/
@4@ -r -L --include='*.md' @CURRENT_SOURCE_DIR@/ @2@/
(cd @2@; RUST_LOG=warn @1@ build -d @2@ 3>&2 2>&1 1>&3) | { grep -Fv "because fragment resolution isn't implemented" || :; } 3>&2 2>&1 1>&3
rm -rf @2@/manual
mv @2@/html @2@/manual
@@ -116,6 +115,7 @@ manual = custom_target(
builtins_md,
rl_next_generated,
summary_rl_next,
json_schema_generated_files,
nix_input,
],
output : [
+7
View File
@@ -12,6 +12,7 @@
rsync,
nix-cli,
changelog-d,
json-schema-for-humans,
officialRelease,
# Configuration Options
@@ -32,6 +33,11 @@ mkMesonDerivation (finalAttrs: {
fileset.difference
(fileset.unions [
../../.version
# For example JSON
../../src/libutil-tests/data/hash
../../src/libstore-tests/data/content-address
../../src/libstore-tests/data/store-path
../../src/libstore-tests/data/derived-path
# Too many different types of files to filter for now
../../doc/manual
./.
@@ -55,6 +61,7 @@ mkMesonDerivation (finalAttrs: {
jq
python3
rsync
json-schema-for-humans
changelog-d
]
++ lib.optionals (!officialRelease) [
@@ -0,0 +1,26 @@
---
synopsis: "Improved S3 binary cache support via HTTP"
prs: [13823, 14026, 14120, 14131, 14135, 14144, 14170, 14190, 14198, 14206, 14209, 14222, 14223, 13752]
issues: [13084, 12671, 11748, 12403]
---
S3 binary cache operations now happen via HTTP, leveraging `libcurl`'s native
AWS SigV4 authentication instead of the AWS C++ SDK, providing significant
improvements:
- **Reduced memory usage**: Eliminates memory buffering issues that caused
segfaults with large files
- **Fixed upload reliability**: Resolves AWS SDK chunking errors
(`InvalidChunkSizeError`)
- **Lighter dependencies**: Uses lightweight `aws-crt-cpp` instead of full
`aws-cpp-sdk`, reducing build complexity
The new implementation requires curl >= 7.75.0 and `aws-crt-cpp` for credential
management.
All existing S3 URL formats and parameters remain supported, with the notable
exception of multi-part uploads, which are no longer supported.
Note that this change also means Nix now supports S3 binary cache stores even
if build without `aws-crt-cpp`, but only for public buckets which do not
require auth.
@@ -0,0 +1,14 @@
---
synopsis: "S3 URLs now support object versioning via versionId parameter"
prs: [14274]
issues: [13955]
---
S3 URLs now support a `versionId` query parameter to fetch specific versions
of objects from S3 buckets with versioning enabled. This allows pinning to
exact object versions for reproducibility and protection against unexpected
changes:
```
s3://bucket/key?region=us-east-1&versionId=abc123def456
```
+4
View File
@@ -117,8 +117,12 @@
- [Architecture and Design](architecture/architecture.md)
- [Formats and Protocols](protocols/index.md)
- [JSON Formats](protocols/json/index.md)
- [Hash](protocols/json/hash.md)
- [Content Address](protocols/json/content-address.md)
- [Store Path](protocols/json/store-path.md)
- [Store Object Info](protocols/json/store-object-info.md)
- [Derivation](protocols/json/derivation.md)
- [Deriving Path](protocols/json/deriving-path.md)
- [Serving Tarball Flakes](protocols/tarball-fetcher.md)
- [Store Path Specification](protocols/store-path.md)
- [Nix Archive (NAR) Format](protocols/nix-archive.md)
@@ -48,8 +48,7 @@ The behaviour of the collector is also influenced by the
configuration file.
By default, the collector prints the total number of freed bytes when it
finishes (or when it is interrupted). With `--print-dead`, it prints the
number of bytes that would be freed.
finishes (or when it is interrupted).
{{#include ./opt-common.md}}
+17 -6
View File
@@ -25,20 +25,31 @@ nix build .#nix-manual
and open `./result/share/doc/nix/manual/index.html`.
To build the manual incrementally, [enter the development shell](./building.md) and run:
To build the manual incrementally, [enter the development shell](./building.md) and configure with `doc-gen` enabled:
**If using interactive `nix develop`:**
```console
make manual-html-open -j $NIX_BUILD_CORES
$ nix develop
$ mesonFlags="$mesonFlags -Ddoc-gen=true" mesonConfigurePhase
```
In order to reflect changes to the [Makefile for the manual], clear all generated files before re-building:
[Makefile for the manual]: https://github.com/NixOS/nix/blob/master/doc/manual/local.mk
**If using direnv:**
```console
rm $(git ls-files doc/manual/ -o | grep -F '.md') && rmdir doc/manual/source/command-ref/new-cli && make manual-html -j $NIX_BUILD_CORES
$ direnv allow
$ bash -c 'source $stdenv/setup && mesonFlags="$mesonFlags -Ddoc-gen=true" mesonConfigurePhase'
```
Then build the manual:
```console
$ cd build
$ meson compile manual
```
The HTML manual will be generated at `build/src/nix-manual/manual/index.html`.
## Style guide
The goal of this style guide is to make it such that
+3
View File
@@ -1,3 +1,6 @@
# Process JSON schema documentation
subdir('protocols')
summary_rl_next = custom_target(
command : [
bash,
@@ -0,0 +1,21 @@
{{#include content-address-v1-fixed.md}}
## Examples
### [Text](@docroot@/store/store-object/content-address.html#method-text) method
```json
{{#include schema/content-address-v1/text.json}}
```
### [Nix Archive](@docroot@/store/store-object/content-address.html#method-nix-archive) method
```json
{{#include schema/content-address-v1/nar.json}}
```
<!-- need to convert YAML to JSON first
## Raw Schema
[JSON Schema for Hash v1](schema/content-address-v1.json)
-->
+5 -118
View File
@@ -1,120 +1,7 @@
# Derivation JSON Format
{{#include derivation-v3-fixed.md}}
> **Warning**
>
> This JSON format is currently
> [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
> and subject to change.
<!--
## Raw Schema
The JSON serialization of a
[derivations](@docroot@/glossary.md#gloss-store-derivation)
is a JSON object with the following fields:
* `name`:
The name of the derivation.
This is used when calculating the store paths of the derivation's outputs.
* `version`:
Must be `3`.
This is a guard that allows us to continue evolving this format.
The choice of `3` is fairly arbitrary, but corresponds to this informal version:
- Version 0: A-Term format
- Version 1: Original JSON format, with ugly `"r:sha256"` inherited from A-Term format.
- Version 2: Separate `method` and `hashAlgo` fields in output specs
- Verison 3: Drop store dir from store paths, just include base name.
Note that while this format is experimental, the maintenance of versions is best-effort, and not promised to identify every change.
* `outputs`:
Information about the output paths of the derivation.
This is a JSON object with one member per output, where the key is the output name and the value is a JSON object with these fields:
* `path`:
The output path, if it is known in advanced.
Otherwise, `null`.
* `method`:
For an output which will be [content addressed], a string representing the [method](@docroot@/store/store-object/content-address.md) of content addressing that is chosen.
Valid method strings are:
- [`flat`](@docroot@/store/store-object/content-address.md#method-flat)
- [`nar`](@docroot@/store/store-object/content-address.md#method-nix-archive)
- [`text`](@docroot@/store/store-object/content-address.md#method-text)
- [`git`](@docroot@/store/store-object/content-address.md#method-git)
Otherwise, `null`.
* `hashAlgo`:
For an output which will be [content addressed], the name of the hash algorithm used.
Valid algorithm strings are:
- `blake3`
- `md5`
- `sha1`
- `sha256`
- `sha512`
* `hash`:
For fixed-output derivations, the expected content hash in base-16.
> **Example**
>
> ```json
> "outputs": {
> "out": {
> "method": "nar",
> "hashAlgo": "sha256",
> "hash": "6fc80dcc62179dbc12fc0b5881275898f93444833d21b89dfe5f7fbcbb1d0d62"
> }
> }
> ```
* `inputSrcs`:
A list of store paths on which this derivation depends.
> **Example**
>
> ```json
> "inputSrcs": [
> "47y241wqdhac3jm5l7nv0x4975mb1975-separate-debug-info.sh",
> "56d0w71pjj9bdr363ym3wj1zkwyqq97j-fix-pop-var-context-error.patch"
> ]
> ```
* `inputDrvs`:
A JSON object specifying the derivations on which this derivation depends, and what outputs of those derivations.
> **Example**
>
> ```json
> "inputDrvs": {
> "6lkh5yi7nlb7l6dr8fljlli5zfd9hq58-curl-7.73.0.drv": ["dev"],
> "fn3kgnfzl5dzym26j8g907gq3kbm8bfh-unzip-6.0.drv": ["out"]
> }
> ```
specifies that this derivation depends on the `dev` output of `curl`, and the `out` output of `unzip`.
* `system`:
The system type on which this derivation is to be built
(e.g. `x86_64-linux`).
* `builder`:
The absolute path of the program to be executed to run the build.
Typically this is the `bash` shell
(e.g. `/nix/store/r3j288vpmczbl500w6zz89gyfa4nr0b1-bash-4.4-p23/bin/bash`).
* `args`:
The command-line arguments passed to the `builder`.
* `env`:
The environment passed to the `builder`.
* `structuredAttrs`:
[Strucutured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
Structured attributes are JSON, and thus embedded as-is.
[JSON Schema for Derivation v3](schema/derivation-v3.json)
-->
@@ -0,0 +1,21 @@
{{#include deriving-path-v1-fixed.md}}
## Examples
### Constant
```json
{{#include schema/deriving-path-v1/single_opaque.json}}
```
### Output of static derivation
```json
{{#include schema/deriving-path-v1/single_built.json}}
```
### Output of dynamic derivation
```json
{{#include schema/deriving-path-v1/single_built_built.json}}
```
@@ -0,0 +1,17 @@
# For some reason, backticks in the JSON schema are being escaped rather
# than being kept as intentional code spans. This removes all backtick
# escaping, which is an ugly solution, but one that is fine, because we
# are not using backticks for any other purpose.
s/\\`/`/g
# The way that semi-external references are rendered (i.e. ones to
# sibling schema files, as opposed to separate website ones, is not nice
# for humans. Replace it with a nice relative link within the manual
# instead.
#
# As we have more such relative links, more replacements of this nature
# should appear below.
s^\(./hash-v1.yaml\)\?#/$defs/algorithm^[JSON format for `Hash`](./hash.html#algorithm)^g
s^\(./hash-v1.yaml\)^[JSON format for `Hash`](./hash.html)^g
s^\(./content-address-v1.yaml\)\?#/$defs/method^[JSON format for `ContentAddress`](./content-address.html#method)^g
s^\(./content-address-v1.yaml\)^[JSON format for `ContentAddress`](./content-address.html)^g
+33
View File
@@ -0,0 +1,33 @@
{{#include hash-v1-fixed.md}}
## Examples
### SHA-256 with Base64 encoding
```json
{{#include schema/hash-v1/sha256-base64.json}}
```
### SHA-256 with Base16 (hexadecimal) encoding
```json
{{#include schema/hash-v1/sha256-base16.json}}
```
### SHA-256 with Nix32 encoding
```json
{{#include schema/hash-v1/sha256-nix32.json}}
```
### BLAKE3 with Base64 encoding
```json
{{#include schema/hash-v1/blake3-base64.json}}
```
<!--
## Raw Schema
[JSON Schema for Hash v1](schema/hash-v1.json)
-->
@@ -0,0 +1,17 @@
# Configuration file for json-schema-for-humans
#
# https://github.com/coveooss/json-schema-for-humans/blob/main/docs/examples/examples_md_default/Configuration.md
template_name: md
show_toc: true
# impure timestamp and distracting
with_footer: false
recursive_detection_depth: 3
show_breadcrumbs: false
description_is_markdown: true
template_md_options:
properties_table_columns:
- Property
- Type
- Pattern
- Title/Description
@@ -0,0 +1,77 @@
# Tests in: ../../../../src/json-schema-checks
fs = import('fs')
# Find json-schema-for-humans if available
json_schema_for_humans = find_program('generate-schema-doc', required : false)
# Configuration for json-schema-for-humans
json_schema_config = files('json-schema-for-humans-config.yaml')
schemas = [
'hash-v1',
'content-address-v1',
'store-path-v1',
'derivation-v3',
'deriving-path-v1',
]
schema_files = files()
foreach schema_name : schemas
schema_files += files('schema' / schema_name + '.yaml')
endforeach
schema_outputs = []
foreach schema_name : schemas
schema_outputs += schema_name + '.md'
endforeach
json_schema_generated_files = []
# Generate markdown documentation from JSON schema
# Note: output must be just a filename, not a path
gen_file = custom_target(
schema_name + '-schema-docs.tmp',
command : [
json_schema_for_humans,
'--config-file',
json_schema_config,
meson.current_source_dir() / 'schema',
meson.current_build_dir(),
],
input : schema_files + [
json_schema_config,
],
output : schema_outputs,
capture : false,
build_by_default : true,
)
idx = 0
if json_schema_for_humans.found()
foreach schema_name : schemas
#schema_file = 'schema' / schema_name + '.yaml'
# There is one so-so hack, and one horrible hack being done here.
sedded_file = custom_target(
schema_name + '-schema-docs',
command : [
'sed',
'-f',
# Out of line to avoid https://github.com/mesonbuild/meson/issues/1564
files('fixup-json-schema-generated-doc.sed'),
'@INPUT@',
],
capture : true,
input : gen_file[idx],
output : schema_name + '-fixed.md',
)
idx += 1
json_schema_generated_files += [ sedded_file ]
endforeach
else
warning(
'json-schema-for-humans not found, skipping JSON schema documentation generation',
)
endif
@@ -0,0 +1 @@
../../../../../../src/libstore-tests/data/content-address
@@ -0,0 +1,55 @@
"$schema": "http://json-schema.org/draft-04/schema"
"$id": "https://nix.dev/manual/nix/latest/protocols/json/schema/content-address-v1.json"
title: Content Address
description: |
This schema describes the JSON representation of Nix's `ContentAddress` type, which conveys information about [content-addressing store objects](@docroot@/store/store-object/content-address.md).
> **Note**
>
> For current methods of content addressing, this data type is a bit suspicious, because it is neither simply a content address of a file system object (the `method` is richer), nor simply a content address of a store object (the `hash` doesn't account for the references).
> It should thus only be used in contexts where the references are also known / otherwise made tamper-resistant.
<!--
TODO currently `ContentAddress` is used in both of these, and so same rationale applies, but actually in both cases the JSON is currently ad-hoc.
That will be fixed, and as each is fixed, the example (along with a more precise link to the field in question) should be become part of the above note, so what is is saying is more clear.
> For example:
> - Fixed outputs of derivations are not allowed to have any references, so an empty reference set is statically known by assumption.
> - [Store object info](./store-object-info.md) includes the set of references along side the (optional) content address.
> This data type is thus safely used in both of these contexts.
-->
type: object
properties:
method:
"$ref": "#/$defs/method"
hash:
title: Content Address
description: |
This would be the content-address itself.
For all current methods, this is just a content address of the file system object of the store object, [as described in the store chapter](@docroot@/store/file-system-object/content-address.md), and not of the store object as a whole.
In particular, the references of the store object are *not* taken into account with this hash (and currently-supported methods).
"$ref": "./hash-v1.yaml"
required:
- method
- hash
additionalProperties: false
"$defs":
method:
type: string
enum: [flat, nar, text, git]
title: Content-Addressing Method
description: |
A string representing the [method](@docroot@/store/store-object/content-address.md) of content addressing that is chosen.
Valid method strings are:
- [`flat`](@docroot@/store/store-object/content-address.md#method-flat) (provided the contents are a single file)
- [`nar`](@docroot@/store/store-object/content-address.md#method-nix-archive)
- [`text`](@docroot@/store/store-object/content-address.md#method-text)
- [`git`](@docroot@/store/store-object/content-address.md#method-git)
@@ -0,0 +1,203 @@
"$schema": "http://json-schema.org/draft-04/schema"
"$id": "https://nix.dev/manual/nix/latest/protocols/json/schema/derivation-v3.json"
title: Derivation
description: |
Experimental JSON representation of a Nix derivation (version 3).
This schema describes the JSON representation of Nix's `Derivation` type.
> **Warning**
>
> This JSON format is currently
> [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
> and subject to change.
type: object
required:
- name
- version
- outputs
- inputSrcs
- inputDrvs
- system
- builder
- args
- env
properties:
name:
type: string
title: Derivation name
description: |
The name of the derivation.
Used when calculating store paths for the derivation’s outputs.
version:
const: 3
title: Format version (must be 3)
description: |
Must be `3`.
This is a guard that allows us to continue evolving this format.
The choice of `3` is fairly arbitrary, but corresponds to this informal version:
- Version 0: A-Term format
- Version 1: Original JSON format, with ugly `"r:sha256"` inherited from A-Term format.
- Version 2: Separate `method` and `hashAlgo` fields in output specs
- Version 3: Drop store dir from store paths, just include base name.
Note that while this format is experimental, the maintenance of versions is best-effort, and not promised to identify every change.
outputs:
type: object
title: Output specifications
description: |
Information about the output paths of the derivation.
This is a JSON object with one member per output, where the key is the output name and the value is a JSON object as described.
> **Example**
>
> ```json
> "outputs": {
> "out": {
> "method": "nar",
> "hashAlgo": "sha256",
> "hash": "6fc80dcc62179dbc12fc0b5881275898f93444833d21b89dfe5f7fbcbb1d0d62"
> }
> }
> ```
additionalProperties:
"$ref": "#/$defs/output"
inputSrcs:
type: array
title: Input source paths
description: |
List of store paths on which this derivation depends.
> **Example**
>
> ```json
> "inputSrcs": [
> "47y241wqdhac3jm5l7nv0x4975mb1975-separate-debug-info.sh",
> "56d0w71pjj9bdr363ym3wj1zkwyqq97j-fix-pop-var-context-error.patch"
> ]
> ```
items:
$ref: "store-path-v1.yaml"
inputDrvs:
type: object
title: Input derivations
description: |
Mapping of derivation paths to lists of output names they provide.
> **Example**
>
> ```json
> "inputDrvs": {
> "6lkh5yi7nlb7l6dr8fljlli5zfd9hq58-curl-7.73.0.drv": ["dev"],
> "fn3kgnfzl5dzym26j8g907gq3kbm8bfh-unzip-6.0.drv": ["out"]
> }
> ```
>
> specifies that this derivation depends on the `dev` output of `curl`, and the `out` output of `unzip`.
patternProperties:
"^[0123456789abcdfghijklmnpqrsvwxyz]{32}-.+\\.drv$":
title: Store Path
description: |
A store path to a derivation, mapped to the outputs of that derivation.
oneOf:
- "$ref": "#/$defs/outputNames"
- "$ref": "#/$defs/dynamicOutputs"
additionalProperties: false
system:
type: string
title: Build system type
description: |
The system type on which this derivation is to be built
(e.g. `x86_64-linux`).
builder:
type: string
title: Build program path
description: |
Absolute path of the program used to perform the build.
Typically this is the `bash` shell
(e.g. `/nix/store/r3j288vpmczbl500w6zz89gyfa4nr0b1-bash-4.4-p23/bin/bash`).
args:
type: array
title: Builder arguments
description: |
Command-line arguments passed to the `builder`.
items:
type: string
env:
type: object
title: Environment variables
description: |
Environment variables passed to the `builder`.
additionalProperties:
type: string
structuredAttrs:
title: Structured attributes
description: |
[Structured Attributes](@docroot@/store/derivation/index.md#structured-attrs), only defined if the derivation contains them.
Structured attributes are JSON, and thus embedded as-is.
type: object
additionalProperties: true
"$defs":
output:
type: object
properties:
path:
$ref: "store-path-v1.yaml"
title: Output path
description: |
The output path, if known in advance.
method:
"$ref": "./content-address-v1.yaml#/$defs/method"
description: |
For an output which will be [content addressed](@docroot@/store/derivation/outputs/content-address.md), a string representing the [method](@docroot@/store/store-object/content-address.md) of content addressing that is chosen.
See the linked original definition for further details.
hashAlgo:
title: Hash algorithm
"$ref": "./hash-v1.yaml#/$defs/algorithm"
hash:
type: string
title: Expected hash value
description: |
For fixed-output derivations, the expected content hash in base-16.
outputName:
type: string
title: Output name
description: Name of the derivation output to depend on
outputNames:
type: array
title: Output Names
description: Set of names of derivation outputs to depend on
items:
"$ref": "#/$defs/outputName"
dynamicOutputs:
type: object
title: Dynamic Outputs
description: |
**Experimental feature**: [`dynamic-derivations`](@docroot@/development/experimental-features.md#xp-feature-dynamic-derivations)
This recursive data type allows for depending on outputs of outputs.
properties:
outputs:
"$ref": "#/$defs/outputNames"
dynamicOutputs:
"$ref": "#/$defs/dynamicOutputs"
@@ -0,0 +1 @@
../../../../../../src/libstore-tests/data/derived-path
@@ -0,0 +1,27 @@
"$schema": "http://json-schema.org/draft-04/schema"
"$id": "https://nix.dev/manual/nix/latest/protocols/json/schema/deriving-path-v1.json"
title: Deriving Path
description: |
This schema describes the JSON representation of Nix's [Deriving Path](@docroot@/store/derivation/index.md#deriving-path).
oneOf:
- title: Constant
description: |
See [Constant](@docroot@/store/derivation/index.md#deriving-path-constant) deriving path.
$ref: "store-path-v1.yaml"
- title: Output
description: |
See [Output](@docroot@/store/derivation/index.md#deriving-path-output) deriving path.
type: object
properties:
drvPath:
"$ref": "#"
description: |
A deriving path to a [Derivation](@docroot@/store/derivation/index.md#store-derivation), whose output is being referred to.
output:
type: string
description: |
The name of an output produced by that derivation (e.g. "out", "doc", etc.).
required:
- drvPath
- output
additionalProperties: false
+1
View File
@@ -0,0 +1 @@
../../../../../../src/libutil-tests/data/hash/
@@ -0,0 +1,54 @@
"$schema": "http://json-schema.org/draft-04/schema"
"$id": "https://nix.dev/manual/nix/latest/protocols/json/schema/hash-v1.json"
title: Hash
description: |
A cryptographic hash value used throughout Nix for content addressing and integrity verification.
This schema describes the JSON representation of Nix's `Hash` type.
type: object
properties:
algorithm:
"$ref": "#/$defs/algorithm"
format:
type: string
enum:
- base64
- nix32
- base16
- sri
title: Hash format
description: |
The encoding format of the hash value.
- `base64` uses standard Base64 encoding [RFC 4648, section 4](https://datatracker.ietf.org/doc/html/rfc4648#section-4)
- `nix32` is Nix-specific base-32 encoding
- `base16` is lowercase hexadecimal
- `sri` is the [Subresource Integrity format](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity).
hash:
type: string
title: Hash
description: |
The encoded hash value, itself.
It is specified in the format specified by the `format` field.
It must be the right length for the hash algorithm specified in the `algorithm` field, also.
The hash value does not include any algorithm prefix.
required:
- algorithm
- format
- hash
additionalProperties: false
"$defs":
algorithm:
type: string
enum:
- blake3
- md5
- sha1
- sha256
- sha512
title: Hash algorithm
description: |
The hash algorithm used to compute the hash value.
`blake3` is currently experimental and requires the [`blake-hashing`](@docroot@/development/experimental-features.md#xp-feature-blake-hashing) experimental feature.
@@ -0,0 +1 @@
../../../../../../src/libstore-tests/data/store-path
@@ -0,0 +1,32 @@
"$schema": "http://json-schema.org/draft-07/schema"
"$id": "https://nix.dev/manual/nix/latest/protocols/json/schema/store-path-v1.json"
title: Store Path
description: |
A [store path](@docroot@/store/store-path.md) identifying a store object.
This schema describes the JSON representation of store paths as used in various Nix JSON APIs.
> **Warning**
>
> This JSON format is currently
> [**experimental**](@docroot@/development/experimental-features.md#xp-feature-nix-command)
> and subject to change.
## Format
Store paths in JSON are represented as strings containing just the hash and name portion, without the store directory prefix.
For example: `"g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv"`
(If the store dir is `/nix/store`, then this corresponds to the path `/nix/store/g1w7hy3qg1w7hy3qg1w7hy3qg1w7hy3q-foo.drv`.)
## Structure
The format follows this pattern: `${digest}-${name}`
- **hash**: Digest rendered in a custom variant of [Base32](https://en.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters)
- **name**: The package name and optional version/suffix information
type: string
pattern: "^[0123456789abcdfghijklmnpqrsvwxyz]{32}-.+$"
minLength: 34
@@ -0,0 +1,15 @@
{{#include store-path-v1-fixed.md}}
## Examples
### Simple store path
```json
{{#include schema/store-path-v1/simple.json}}
```
<!--
## Raw Schema
[JSON Schema for Store Path v1](schema/store-path-v1.json)
-->
+2
View File
@@ -0,0 +1,2 @@
# Process JSON schema documentation
subdir('json')
+1 -1
View File
@@ -106,7 +106,7 @@ The system type on which the [`builder`](#attr-builder) executable is meant to b
A necessary condition for Nix to schedule a given derivation on some [Nix instance] is for the "system" of that derivation to match that instance's [`system` configuration option] or [`extra-platforms` configuration option].
By putting the `system` in each derivation, Nix allows *heterogenous* build plans, where not all steps can be run on the same machine or same sort of machine.
By putting the `system` in each derivation, Nix allows *heterogeneous* build plans, where not all steps can be run on the same machine or same sort of machine.
Nix can schedule builds such that it automatically builds on other platforms by [forwarding build requests](@docroot@/advanced-topics/distributed-builds.md) to other Nix instances.
[`system` configuration option]: @docroot@/command-ref/conf-file.md#conf-system
+25
View File
@@ -413,6 +413,10 @@
supportsCross = false;
};
"nix-json-schema-checks" = {
supportsCross = false;
};
"nix-perl-bindings" = {
supportsCross = false;
};
@@ -467,6 +471,27 @@
}
);
apps = forAllSystems (
system:
let
pkgs = nixpkgsFor.${system}.native;
opener = if pkgs.stdenv.isDarwin then "open" else "xdg-open";
in
{
open-manual = {
type = "app";
program = "${pkgs.writeShellScript "open-nix-manual" ''
manual_path="${self.packages.${system}.nix-manual}/share/doc/nix/manual/index.html"
if ! ${opener} "$manual_path"; then
echo "Failed to open manual with ${opener}. Manual is located at:"
echo "$manual_path"
fi
''}";
meta.description = "Open the Nix manual in your browser";
};
}
);
devShells =
let
makeShell = import ./packaging/dev-shell.nix { inherit lib devFlake; };
-1
View File
@@ -142,7 +142,6 @@ release:
$ git pull
$ NEW_VERSION=2.13.0
$ echo $NEW_VERSION > .version
$ ... edit .mergify.yml to add the previous version ...
$ git checkout -b bump-$NEW_VERSION
$ git commit -a -m 'Bump version'
$ git push --set-upstream origin bump-$NEW_VERSION
+1
View File
@@ -60,3 +60,4 @@ if get_option('unit-tests')
subproject('libflake-tests')
endif
subproject('nix-functional-tests')
subproject('json-schema-checks')
+1 -1
View File
@@ -1,5 +1,5 @@
# shellcheck disable=all
#compdef nix
# shellcheck disable=all
function _nix() {
local ifs_bk="$IFS"
@@ -0,0 +1,6 @@
extern "C" [[gnu::retain, gnu::weak]] const char * __asan_default_options()
{
// We leak a bunch of memory knowingly on purpose. It's not worthwhile to
// diagnose that memory being leaked for now.
return "abort_on_error=1:print_summary=1:detect_leaks=0:detect_odr_violation=0";
}
@@ -1,7 +1,3 @@
asan_test_options_env = {
'ASAN_OPTIONS' : 'abort_on_error=1:print_summary=1:detect_leaks=0',
}
# Clang gets grumpy about missing libasan symbols if -shared-libasan is not
# passed when building shared libs, at least on Linux
if cxx.get_id() == 'clang' and ('address' in get_option('b_sanitize') or 'undefined' in get_option(
@@ -10,3 +6,6 @@ if cxx.get_id() == 'clang' and ('address' in get_option('b_sanitize') or 'undefi
add_project_link_arguments('-shared-libasan', language : 'cpp')
endif
if 'address' in get_option('b_sanitize')
deps_other += declare_dependency(sources : 'asan-options.cc')
endif
@@ -0,0 +1,32 @@
can_wrap_assert_fail_test_code = '''
#include <cstdlib>
#include <cassert>
int main()
{
assert(0);
}
extern "C" void * __real___assert_fail(const char *, const char *, unsigned int, const char *);
extern "C" void *
__wrap___assert_fail(const char *, const char *, unsigned int, const char *)
{
return __real___assert_fail(nullptr, nullptr, 0, nullptr);
}
'''
wrap_assert_fail_args = [ '-Wl,--wrap=__assert_fail' ]
can_wrap_assert_fail = cxx.links(
can_wrap_assert_fail_test_code,
args : wrap_assert_fail_args,
name : 'linker can wrap __assert_fail',
)
if can_wrap_assert_fail
deps_other += declare_dependency(
sources : 'wrap-assert-fail.cc',
link_args : wrap_assert_fail_args,
)
endif
@@ -0,0 +1,17 @@
#include "nix/util/error.hh"
#include <cstdio>
#include <cstdlib>
#include <cinttypes>
#include <string_view>
extern "C" [[noreturn]] void __attribute__((weak))
__wrap___assert_fail(const char * assertion, const char * file, unsigned int line, const char * function)
{
char buf[512];
int n =
snprintf(buf, sizeof(buf), "Assertion '%s' failed in %s at %s:%" PRIuLEAST32, assertion, function, file, line);
if (n < 0)
nix::panic("Assertion failed and could not format error message");
nix::panic(std::string_view(buf, std::min(static_cast<int>(sizeof(buf)), n)));
}
@@ -44,3 +44,6 @@ endif
# Darwin ld doesn't like "X.Y.Zpre"
nix_soversion = meson.project_version().split('pre')[0]
subdir('assert-fail')
subdir('asan-options')
+38 -1
View File
@@ -204,6 +204,25 @@ let
mesonFlags = [ (lib.mesonBool "b_asneeded" false) ] ++ prevAttrs.mesonFlags or [ ];
};
enableSanitizersLayer =
finalAttrs: prevAttrs:
let
sanitizers = lib.optional scope.withASan "address" ++ lib.optional scope.withUBSan "undefined";
in
{
mesonFlags =
(prevAttrs.mesonFlags or [ ])
++ lib.optionals (lib.length sanitizers > 0) (
[
(lib.mesonOption "b_sanitize" (lib.concatStringsSep "," sanitizers))
]
++ (lib.optionals stdenv.cc.isClang [
# https://www.github.com/mesonbuild/meson/issues/764
(lib.mesonBool "b_lundef" false)
])
);
};
nixDefaultsLayer = finalAttrs: prevAttrs: {
strictDeps = prevAttrs.strictDeps or true;
enableParallelBuilding = true;
@@ -246,6 +265,16 @@ in
inherit filesetToSource;
/**
Whether meson components are built with [AddressSanitizer](https://clang.llvm.org/docs/AddressSanitizer.html).
*/
withASan = false;
/**
Whether meson components are built with [UndefinedBehaviorSanitizer](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html).
*/
withUBSan = false;
/**
A user-provided extension function to apply to each component derivation.
*/
@@ -332,6 +361,7 @@ in
setVersionLayer
mesonLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
mkMesonExecutable = mkPackageBuilder [
@@ -342,6 +372,7 @@ in
mesonLayer
mesonBuildLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
mkMesonLibrary = mkPackageBuilder [
@@ -353,6 +384,7 @@ in
mesonBuildLayer
mesonLibraryLayer
fixupStaticLayer
enableSanitizersLayer
scope.mesonComponentOverrides
];
@@ -406,6 +438,11 @@ in
*/
nix-external-api-docs = callPackage ../src/external-api-docs/package.nix { version = fineVersion; };
/**
JSON schema validation checks
*/
nix-json-schema-checks = callPackage ../src/json-schema-checks/package.nix { };
nix-perl-bindings = callPackage ../src/perl/package.nix { };
/**
@@ -458,7 +495,7 @@ in
Example:
```
overrideScope (finalScope: prevScope: { aws-sdk-cpp = null; })
overrideScope (finalScope: prevScope: { aws-crt-cpp = null; })
```
*/
overrideScope = f: (scope.overrideScope f).nix-everything;
+14 -58
View File
@@ -16,21 +16,6 @@ in
scope: {
inherit stdenv;
aws-sdk-cpp =
(pkgs.aws-sdk-cpp.override {
apis = [
"identity-management"
"s3"
"transfer"
];
customMemoryManagement = false;
}).overrideAttrs
{
# only a stripped down version is built, which takes a lot less resources
# to build, so we don't need a "big-parallel" machine.
requiredSystemFeatures = [ ];
};
boehmgc =
(pkgs.boehmgc.override {
enableLargeConfig = true;
@@ -57,15 +42,20 @@ scope: {
prevAttrs.postInstall;
});
toml11 = pkgs.toml11.overrideAttrs rec {
version = "4.4.0";
src = pkgs.fetchFromGitHub {
owner = "ToruNiina";
repo = "toml11";
tag = "v${version}";
hash = "sha256-sgWKYxNT22nw376ttGsTdg0AMzOwp8QH3E8mx0BZJTQ=";
};
};
# TODO: Remove this when https://github.com/NixOS/nixpkgs/pull/442682 is included in a stable release
toml11 =
if lib.versionAtLeast pkgs.toml11.version "4.4.0" then
pkgs.toml11
else
pkgs.toml11.overrideAttrs rec {
version = "4.4.0";
src = pkgs.fetchFromGitHub {
owner = "ToruNiina";
repo = "toml11";
tag = "v${version}";
hash = "sha256-sgWKYxNT22nw376ttGsTdg0AMzOwp8QH3E8mx0BZJTQ=";
};
};
# TODO Hack until https://github.com/NixOS/nixpkgs/issues/45462 is fixed.
boost =
@@ -84,38 +74,4 @@ scope: {
buildPhase = lib.replaceStrings [ "--without-python" ] [ "" ] old.buildPhase;
installPhase = lib.replaceStrings [ "--without-python" ] [ "" ] old.installPhase;
});
libgit2 =
if lib.versionAtLeast pkgs.libgit2.version "1.9.0" then
pkgs.libgit2
else
pkgs.libgit2.overrideAttrs (attrs: {
# libgit2: Nixpkgs 24.11 has < 1.9.0, which needs our patches
nativeBuildInputs =
attrs.nativeBuildInputs or [ ]
# gitMinimal does not build on Windows. See packbuilder patch.
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# Needed for `git apply`; see `prePatch`
pkgs.buildPackages.gitMinimal
];
# Only `git apply` can handle git binary patches
prePatch =
attrs.prePatch or ""
+ lib.optionalString (!stdenv.hostPlatform.isWindows) ''
patch() {
git apply
}
'';
patches =
attrs.patches or [ ]
++ [
./patches/libgit2-mempack-thin-packfile.patch
]
# gitMinimal does not build on Windows, but fortunately this patch only
# impacts interruptibility
++ lib.optionals (!stdenv.hostPlatform.isWindows) [
# binary patch; see `prePatch`
./patches/libgit2-packbuilder-callback-interruptible.patch
];
});
}
+42 -32
View File
@@ -70,6 +70,9 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
# We use this shell with the local checkout, not unpackPhase.
src = null;
# Workaround https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
# Remove when gdb fix is rolled out everywhere.
separateDebugInfo = false;
env = {
# For `make format`, to work without installing pre-commit
@@ -93,38 +96,45 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
++ map (transformFlag "libcmd") (ignoreCrossFile pkgs.nixComponents2.nix-cmd.mesonFlags);
nativeBuildInputs =
attrs.nativeBuildInputs or [ ]
++ pkgs.nixComponents2.nix-util.nativeBuildInputs
++ pkgs.nixComponents2.nix-store.nativeBuildInputs
++ pkgs.nixComponents2.nix-fetchers.nativeBuildInputs
++ pkgs.nixComponents2.nix-expr.nativeBuildInputs
++ lib.optionals havePerl pkgs.nixComponents2.nix-perl-bindings.nativeBuildInputs
++ lib.optionals buildCanExecuteHost pkgs.nixComponents2.nix-manual.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-internal-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-external-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-functional-tests.externalNativeBuildInputs
++ lib.optional (
!buildCanExecuteHost
# Hack around https://github.com/nixos/nixpkgs/commit/bf7ad8cfbfa102a90463433e2c5027573b462479
&& !(stdenv.hostPlatform.isWindows && stdenv.buildPlatform.isDarwin)
&& stdenv.hostPlatform.emulatorAvailable pkgs.buildPackages
&& lib.meta.availableOn stdenv.buildPlatform (stdenv.hostPlatform.emulator pkgs.buildPackages)
) pkgs.buildPackages.mesonEmulatorHook
++ [
pkgs.buildPackages.cmake
pkgs.buildPackages.gnused
pkgs.buildPackages.shellcheck
pkgs.buildPackages.changelog-d
modular.pre-commit.settings.package
(pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript)
pkgs.buildPackages.nixfmt-rfc-style
pkgs.buildPackages.shellcheck
pkgs.buildPackages.gdb
]
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
)
++ lib.optional stdenv.hostPlatform.isLinux pkgs.buildPackages.mold-wrapped;
let
inputs =
attrs.nativeBuildInputs or [ ]
++ pkgs.nixComponents2.nix-util.nativeBuildInputs
++ pkgs.nixComponents2.nix-store.nativeBuildInputs
++ pkgs.nixComponents2.nix-fetchers.nativeBuildInputs
++ pkgs.nixComponents2.nix-expr.nativeBuildInputs
++ lib.optionals havePerl pkgs.nixComponents2.nix-perl-bindings.nativeBuildInputs
++ lib.optionals buildCanExecuteHost pkgs.nixComponents2.nix-manual.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-internal-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-external-api-docs.nativeBuildInputs
++ pkgs.nixComponents2.nix-functional-tests.externalNativeBuildInputs
++ pkgs.nixComponents2.nix-json-schema-checks.externalNativeBuildInputs
++ lib.optional (
!buildCanExecuteHost
# Hack around https://github.com/nixos/nixpkgs/commit/bf7ad8cfbfa102a90463433e2c5027573b462479
&& !(stdenv.hostPlatform.isWindows && stdenv.buildPlatform.isDarwin)
&& stdenv.hostPlatform.emulatorAvailable pkgs.buildPackages
&& lib.meta.availableOn stdenv.buildPlatform (stdenv.hostPlatform.emulator pkgs.buildPackages)
) pkgs.buildPackages.mesonEmulatorHook
++ [
pkgs.buildPackages.cmake
pkgs.buildPackages.gnused
pkgs.buildPackages.changelog-d
modular.pre-commit.settings.package
(pkgs.writeScriptBin "pre-commit-hooks-install" modular.pre-commit.settings.installationScript)
pkgs.buildPackages.nixfmt-rfc-style
pkgs.buildPackages.shellcheck
pkgs.buildPackages.include-what-you-use
pkgs.buildPackages.gdb
]
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
)
++ lib.optional stdenv.hostPlatform.isLinux pkgs.buildPackages.mold-wrapped;
in
# FIXME: separateDebugInfo = false doesn't actually prevent -Wa,--compress-debug-sections
# from making its way into NIX_CFLAGS_COMPILE.
lib.filter (p: !lib.hasInfix "separate-debug-info" p) inputs;
buildInputs = [
pkgs.gbenchmark
+16 -3
View File
@@ -62,6 +62,7 @@ let
"nix-cmd"
"nix-cli"
"nix-functional-tests"
"nix-json-schema-checks"
]
++ lib.optionals enableBindings [
"nix-perl-bindings"
@@ -73,7 +74,7 @@ let
]
);
in
{
rec {
/**
An internal check to make sure our package listing is complete.
*/
@@ -145,13 +146,25 @@ in
)
);
buildNoGc =
# Builds with sanitizers already have GC disabled, so this buildNoGc can just
# point to buildWithSanitizers in order to reduce the load on hydra.
buildNoGc = buildWithSanitizers;
buildWithSanitizers =
let
components = forAllSystems (
system:
nixpkgsFor.${system}.native.nixComponents2.overrideScope (
let
pkgs = nixpkgsFor.${system}.native;
in
pkgs.nixComponents2.overrideScope (
self: super: {
# Boost coroutines fail with ASAN on darwin.
withASan = !pkgs.stdenv.buildPlatform.isDarwin;
withUBSan = true;
nix-expr = super.nix-expr.override { enableGC = false; };
# Unclear how to make Perl bindings work with a dynamically linked ASAN.
nix-perl-bindings = null;
}
)
);
@@ -1,282 +0,0 @@
commit 9bacade4a3ef4b6b26e2c02f549eef0e9eb9eaa2
Author: Robert Hensing <robert@roberthensing.nl>
Date: Sun Aug 18 20:20:36 2024 +0200
Add unoptimized git_mempack_write_thin_pack
diff --git a/include/git2/sys/mempack.h b/include/git2/sys/mempack.h
index 17da590a3..3688bdd50 100644
--- a/include/git2/sys/mempack.h
+++ b/include/git2/sys/mempack.h
@@ -44,6 +44,29 @@ GIT_BEGIN_DECL
*/
GIT_EXTERN(int) git_mempack_new(git_odb_backend **out);
+/**
+ * Write a thin packfile with the objects in the memory store.
+ *
+ * A thin packfile is a packfile that does not contain its transitive closure of
+ * references. This is useful for efficiently distributing additions to a
+ * repository over the network, but also finds use in the efficient bulk
+ * addition of objects to a repository, locally.
+ *
+ * This operation performs the (shallow) insert operations into the
+ * `git_packbuilder`, but does not write the packfile to disk;
+ * see `git_packbuilder_write_buf`.
+ *
+ * It also does not reset the memory store; see `git_mempack_reset`.
+ *
+ * @note This function may or may not write trees and blobs that are not
+ * referenced by commits. Currently everything is written, but this
+ * behavior may change in the future as the packer is optimized.
+ *
+ * @param backend The mempack backend
+ * @param pb The packbuilder to use to write the packfile
+ */
+GIT_EXTERN(int) git_mempack_write_thin_pack(git_odb_backend *backend, git_packbuilder *pb);
+
/**
* Dump all the queued in-memory writes to a packfile.
*
diff --git a/src/libgit2/odb_mempack.c b/src/libgit2/odb_mempack.c
index 6f27f45f8..0b61e2b66 100644
--- a/src/libgit2/odb_mempack.c
+++ b/src/libgit2/odb_mempack.c
@@ -132,6 +132,35 @@ cleanup:
return err;
}
+int git_mempack_write_thin_pack(git_odb_backend *backend, git_packbuilder *pb)
+{
+ struct memory_packer_db *db = (struct memory_packer_db *)backend;
+ const git_oid *oid;
+ size_t iter = 0;
+ int err = -1;
+
+ /* TODO: Implement the recency heuristics.
+ For this it probably makes sense to only write what's referenced
+ through commits, an option I've carved out for you in the docs.
+ wrt heuristics: ask your favorite LLM to translate https://git-scm.com/docs/pack-heuristics/en
+ to actual normal reference documentation. */
+ while (true) {
+ err = git_oidmap_iterate(NULL, db->objects, &iter, &oid);
+ if (err == GIT_ITEROVER) {
+ err = 0;
+ break;
+ }
+ if (err != 0)
+ return err;
+
+ err = git_packbuilder_insert(pb, oid, NULL);
+ if (err != 0)
+ return err;
+ }
+
+ return 0;
+}
+
int git_mempack_dump(
git_buf *pack,
git_repository *repo,
diff --git a/tests/libgit2/mempack/thinpack.c b/tests/libgit2/mempack/thinpack.c
new file mode 100644
index 000000000..604a4dda2
--- /dev/null
+++ b/tests/libgit2/mempack/thinpack.c
@@ -0,0 +1,196 @@
+#include "clar_libgit2.h"
+#include "git2/indexer.h"
+#include "git2/odb_backend.h"
+#include "git2/tree.h"
+#include "git2/types.h"
+#include "git2/sys/mempack.h"
+#include "git2/sys/odb_backend.h"
+#include "util.h"
+
+static git_repository *_repo;
+static git_odb_backend * _mempack_backend;
+
+void test_mempack_thinpack__initialize(void)
+{
+ git_odb *odb;
+
+ _repo = cl_git_sandbox_init_new("mempack_thinpack_repo");
+
+ cl_git_pass(git_mempack_new(&_mempack_backend));
+ cl_git_pass(git_repository_odb(&odb, _repo));
+ cl_git_pass(git_odb_add_backend(odb, _mempack_backend, 999));
+ git_odb_free(odb);
+}
+
+void _mempack_thinpack__cleanup(void)
+{
+ cl_git_sandbox_cleanup();
+}
+
+/*
+ Generating a packfile for an unchanged repo works and produces an empty packfile.
+ Even if we allow this scenario to be detected, it shouldn't misbehave if the
+ application is unaware of it.
+*/
+void test_mempack_thinpack__empty(void)
+{
+ git_packbuilder *pb;
+ int version;
+ int n;
+ git_buf buf = GIT_BUF_INIT;
+
+ git_packbuilder_new(&pb, _repo);
+
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_assert_in_range(12, buf.size, 1024 /* empty packfile is >0 bytes, but certainly not that big */);
+ cl_assert(buf.ptr[0] == 'P');
+ cl_assert(buf.ptr[1] == 'A');
+ cl_assert(buf.ptr[2] == 'C');
+ cl_assert(buf.ptr[3] == 'K');
+ version = (buf.ptr[4] << 24) | (buf.ptr[5] << 16) | (buf.ptr[6] << 8) | buf.ptr[7];
+ /* Subject to change. https://git-scm.com/docs/pack-format: Git currently accepts version number 2 or 3 but generates version 2 only.*/
+ cl_assert_equal_i(2, version);
+ n = (buf.ptr[8] << 24) | (buf.ptr[9] << 16) | (buf.ptr[10] << 8) | buf.ptr[11];
+ cl_assert_equal_i(0, n);
+ git_buf_dispose(&buf);
+
+ git_packbuilder_free(pb);
+}
+
+#define LIT_LEN(x) x, sizeof(x) - 1
+
+/*
+ Check that git_mempack_write_thin_pack produces a thin packfile.
+*/
+void test_mempack_thinpack__thin(void)
+{
+ /* Outline:
+ - Create tree 1
+ - Flush to packfile A
+ - Create tree 2
+ - Flush to packfile B
+
+ Tree 2 has a new blob and a reference to a blob from tree 1.
+
+ Expectation:
+ - Packfile B is thin and does not contain the objects from packfile A
+ */
+
+
+ git_oid oid_blob_1;
+ git_oid oid_blob_2;
+ git_oid oid_blob_3;
+ git_oid oid_tree_1;
+ git_oid oid_tree_2;
+ git_treebuilder *tb;
+
+ git_packbuilder *pb;
+ git_buf buf = GIT_BUF_INIT;
+ git_indexer *indexer;
+ git_indexer_progress stats;
+ char pack_dir_path[1024];
+
+ char sbuf[1024];
+ const char * repo_path;
+ const char * pack_name_1;
+ const char * pack_name_2;
+ git_str pack_path_1 = GIT_STR_INIT;
+ git_str pack_path_2 = GIT_STR_INIT;
+ git_odb_backend * pack_odb_backend_1;
+ git_odb_backend * pack_odb_backend_2;
+
+
+ cl_assert_in_range(0, snprintf(pack_dir_path, sizeof(pack_dir_path), "%s/objects/pack", git_repository_path(_repo)), sizeof(pack_dir_path));
+
+ /* Create tree 1 */
+
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_1, _repo, LIT_LEN("thinpack blob 1")));
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_2, _repo, LIT_LEN("thinpack blob 2")));
+
+
+ cl_git_pass(git_treebuilder_new(&tb, _repo, NULL));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob1", &oid_blob_1, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob2", &oid_blob_2, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_write(&oid_tree_1, tb));
+
+ /* Flush */
+
+ cl_git_pass(git_packbuilder_new(&pb, _repo));
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_git_pass(git_indexer_new(&indexer, pack_dir_path, 0, NULL, NULL));
+ cl_git_pass(git_indexer_append(indexer, buf.ptr, buf.size, &stats));
+ cl_git_pass(git_indexer_commit(indexer, &stats));
+ pack_name_1 = strdup(git_indexer_name(indexer));
+ cl_assert(pack_name_1);
+ git_buf_dispose(&buf);
+ git_mempack_reset(_mempack_backend);
+ git_indexer_free(indexer);
+ git_packbuilder_free(pb);
+
+ /* Create tree 2 */
+
+ cl_git_pass(git_treebuilder_clear(tb));
+ /* blob 1 won't be used, but we add it anyway to test that just "declaring" an object doesn't
+ necessarily cause its inclusion in the next thin packfile. It must only be included if new. */
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_1, _repo, LIT_LEN("thinpack blob 1")));
+ cl_git_pass(git_blob_create_from_buffer(&oid_blob_3, _repo, LIT_LEN("thinpack blob 3")));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob1", &oid_blob_1, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_insert(NULL, tb, "blob3", &oid_blob_3, GIT_FILEMODE_BLOB));
+ cl_git_pass(git_treebuilder_write(&oid_tree_2, tb));
+
+ /* Flush */
+
+ cl_git_pass(git_packbuilder_new(&pb, _repo));
+ cl_git_pass(git_mempack_write_thin_pack(_mempack_backend, pb));
+ cl_git_pass(git_packbuilder_write_buf(&buf, pb));
+ cl_git_pass(git_indexer_new(&indexer, pack_dir_path, 0, NULL, NULL));
+ cl_git_pass(git_indexer_append(indexer, buf.ptr, buf.size, &stats));
+ cl_git_pass(git_indexer_commit(indexer, &stats));
+ pack_name_2 = strdup(git_indexer_name(indexer));
+ cl_assert(pack_name_2);
+ git_buf_dispose(&buf);
+ git_mempack_reset(_mempack_backend);
+ git_indexer_free(indexer);
+ git_packbuilder_free(pb);
+ git_treebuilder_free(tb);
+
+ /* Assertions */
+
+ assert(pack_name_1);
+ assert(pack_name_2);
+
+ repo_path = git_repository_path(_repo);
+
+ snprintf(sbuf, sizeof(sbuf), "objects/pack/pack-%s.pack", pack_name_1);
+ git_str_joinpath(&pack_path_1, repo_path, sbuf);
+ snprintf(sbuf, sizeof(sbuf), "objects/pack/pack-%s.pack", pack_name_2);
+ git_str_joinpath(&pack_path_2, repo_path, sbuf);
+
+ /* If they're the same, something definitely went wrong. */
+ cl_assert(strcmp(pack_name_1, pack_name_2) != 0);
+
+ cl_git_pass(git_odb_backend_one_pack(&pack_odb_backend_1, pack_path_1.ptr));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_1));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_2));
+ cl_assert(!pack_odb_backend_1->exists(pack_odb_backend_1, &oid_blob_3));
+ cl_assert(pack_odb_backend_1->exists(pack_odb_backend_1, &oid_tree_1));
+ cl_assert(!pack_odb_backend_1->exists(pack_odb_backend_1, &oid_tree_2));
+
+ cl_git_pass(git_odb_backend_one_pack(&pack_odb_backend_2, pack_path_2.ptr));
+ /* blob 1 is already in the packfile 1, so packfile 2 must not include it, in order to be _thin_. */
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_1));
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_2));
+ cl_assert(pack_odb_backend_2->exists(pack_odb_backend_2, &oid_blob_3));
+ cl_assert(!pack_odb_backend_2->exists(pack_odb_backend_2, &oid_tree_1));
+ cl_assert(pack_odb_backend_2->exists(pack_odb_backend_2, &oid_tree_2));
+
+ pack_odb_backend_1->free(pack_odb_backend_1);
+ pack_odb_backend_2->free(pack_odb_backend_2);
+ free((void *)pack_name_1);
+ free((void *)pack_name_2);
+ git_str_dispose(&pack_path_1);
+ git_str_dispose(&pack_path_2);
+
+}
@@ -1,930 +0,0 @@
commit e9823c5da4fa977c46bcb97167fbdd0d70adb5ff
Author: Robert Hensing <robert@roberthensing.nl>
Date: Mon Aug 26 20:07:04 2024 +0200
Make packbuilder interruptible using progress callback
Forward errors from packbuilder->progress_cb
This allows the callback to terminate long-running operations when
the application is interrupted.
diff --git a/include/git2/pack.h b/include/git2/pack.h
index 0f6bd2ab9..bee72a6c0 100644
--- a/include/git2/pack.h
+++ b/include/git2/pack.h
@@ -247,6 +247,9 @@ typedef int GIT_CALLBACK(git_packbuilder_progress)(
* @param progress_cb Function to call with progress information during
* pack building. Be aware that this is called inline with pack building
* operations, so performance may be affected.
+ * When progress_cb returns an error, the pack building process will be
+ * aborted and the error will be returned from the invoked function.
+ * `pb` must then be freed.
* @param progress_cb_payload Payload for progress callback.
* @return 0 or an error code
*/
diff --git a/src/libgit2/pack-objects.c b/src/libgit2/pack-objects.c
index b2d80cba9..7c331c2d5 100644
--- a/src/libgit2/pack-objects.c
+++ b/src/libgit2/pack-objects.c
@@ -932,6 +932,9 @@ static int report_delta_progress(
{
int ret;
+ if (pb->failure)
+ return pb->failure;
+
if (pb->progress_cb) {
uint64_t current_time = git_time_monotonic();
uint64_t elapsed = current_time - pb->last_progress_report_time;
@@ -943,8 +946,10 @@ static int report_delta_progress(
GIT_PACKBUILDER_DELTAFICATION,
count, pb->nr_objects, pb->progress_cb_payload);
- if (ret)
+ if (ret) {
+ pb->failure = ret;
return git_error_set_after_callback(ret);
+ }
}
}
@@ -976,7 +981,10 @@ static int find_deltas(git_packbuilder *pb, git_pobject **list,
}
pb->nr_deltified += 1;
- report_delta_progress(pb, pb->nr_deltified, false);
+ if ((error = report_delta_progress(pb, pb->nr_deltified, false)) < 0) {
+ GIT_ASSERT(git_packbuilder__progress_unlock(pb) == 0);
+ goto on_error;
+ }
po = *list++;
(*list_size)--;
@@ -1124,6 +1132,10 @@ struct thread_params {
size_t depth;
size_t working;
size_t data_ready;
+
+ /* A pb->progress_cb can stop the packing process by returning an error.
+ When that happens, all threads observe the error and stop voluntarily. */
+ bool stopped;
};
static void *threaded_find_deltas(void *arg)
@@ -1133,7 +1145,12 @@ static void *threaded_find_deltas(void *arg)
while (me->remaining) {
if (find_deltas(me->pb, me->list, &me->remaining,
me->window, me->depth) < 0) {
- ; /* TODO */
+ me->stopped = true;
+ GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_lock(me->pb) == 0, NULL);
+ me->working = false;
+ git_cond_signal(&me->pb->progress_cond);
+ GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_unlock(me->pb) == 0, NULL);
+ return NULL;
}
GIT_ASSERT_WITH_RETVAL(git_packbuilder__progress_lock(me->pb) == 0, NULL);
@@ -1175,8 +1192,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
pb->nr_threads = git__online_cpus();
if (pb->nr_threads <= 1) {
- find_deltas(pb, list, &list_size, window, depth);
- return 0;
+ return find_deltas(pb, list, &list_size, window, depth);
}
p = git__mallocarray(pb->nr_threads, sizeof(*p));
@@ -1195,6 +1211,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
p[i].depth = depth;
p[i].working = 1;
p[i].data_ready = 0;
+ p[i].stopped = 0;
/* try to split chunks on "path" boundaries */
while (sub_size && sub_size < list_size &&
@@ -1262,7 +1279,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
(!victim || victim->remaining < p[i].remaining))
victim = &p[i];
- if (victim) {
+ if (victim && !target->stopped) {
sub_size = victim->remaining / 2;
list = victim->list + victim->list_size - sub_size;
while (sub_size && list[0]->hash &&
@@ -1286,7 +1303,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
}
target->list_size = sub_size;
target->remaining = sub_size;
- target->working = 1;
+ target->working = 1; /* even when target->stopped, so that we don't process this thread again */
GIT_ASSERT(git_packbuilder__progress_unlock(pb) == 0);
if (git_mutex_lock(&target->mutex)) {
@@ -1299,7 +1316,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
git_cond_signal(&target->cond);
git_mutex_unlock(&target->mutex);
- if (!sub_size) {
+ if (target->stopped || !sub_size) {
git_thread_join(&target->thread, NULL);
git_cond_free(&target->cond);
git_mutex_free(&target->mutex);
@@ -1308,7 +1325,7 @@ static int ll_find_deltas(git_packbuilder *pb, git_pobject **list,
}
git__free(p);
- return 0;
+ return pb->failure;
}
#else
@@ -1319,6 +1336,7 @@ int git_packbuilder__prepare(git_packbuilder *pb)
{
git_pobject **delta_list;
size_t i, n = 0;
+ int error;
if (pb->nr_objects == 0 || pb->done)
return 0; /* nothing to do */
@@ -1327,8 +1345,10 @@ int git_packbuilder__prepare(git_packbuilder *pb)
* Although we do not report progress during deltafication, we
* at least report that we are in the deltafication stage
*/
- if (pb->progress_cb)
- pb->progress_cb(GIT_PACKBUILDER_DELTAFICATION, 0, pb->nr_objects, pb->progress_cb_payload);
+ if (pb->progress_cb) {
+ if ((error = pb->progress_cb(GIT_PACKBUILDER_DELTAFICATION, 0, pb->nr_objects, pb->progress_cb_payload)) < 0)
+ return git_error_set_after_callback(error);
+ }
delta_list = git__mallocarray(pb->nr_objects, sizeof(*delta_list));
GIT_ERROR_CHECK_ALLOC(delta_list);
@@ -1345,31 +1365,33 @@ int git_packbuilder__prepare(git_packbuilder *pb)
if (n > 1) {
git__tsort((void **)delta_list, n, type_size_sort);
- if (ll_find_deltas(pb, delta_list, n,
+ if ((error = ll_find_deltas(pb, delta_list, n,
GIT_PACK_WINDOW + 1,
- GIT_PACK_DEPTH) < 0) {
+ GIT_PACK_DEPTH)) < 0) {
git__free(delta_list);
- return -1;
+ return error;
}
}
- report_delta_progress(pb, pb->nr_objects, true);
+ error = report_delta_progress(pb, pb->nr_objects, true);
pb->done = true;
git__free(delta_list);
- return 0;
+ return error;
}
-#define PREPARE_PACK if (git_packbuilder__prepare(pb) < 0) { return -1; }
+#define PREPARE_PACK error = git_packbuilder__prepare(pb); if (error < 0) { return error; }
int git_packbuilder_foreach(git_packbuilder *pb, int (*cb)(void *buf, size_t size, void *payload), void *payload)
{
+ int error;
PREPARE_PACK;
return write_pack(pb, cb, payload);
}
int git_packbuilder__write_buf(git_str *buf, git_packbuilder *pb)
{
+ int error;
PREPARE_PACK;
return write_pack(pb, &write_pack_buf, buf);
diff --git a/src/libgit2/pack-objects.h b/src/libgit2/pack-objects.h
index bbc8b9430..380a28ebe 100644
--- a/src/libgit2/pack-objects.h
+++ b/src/libgit2/pack-objects.h
@@ -100,6 +100,10 @@ struct git_packbuilder {
uint64_t last_progress_report_time;
bool done;
+
+ /* A non-zero error code in failure causes all threads to shut themselves
+ down. Some functions will return this error code. */
+ volatile int failure;
};
int git_packbuilder__write_buf(git_str *buf, git_packbuilder *pb);
diff --git a/tests/libgit2/pack/cancel.c b/tests/libgit2/pack/cancel.c
new file mode 100644
index 000000000..a0aa9716a
--- /dev/null
+++ b/tests/libgit2/pack/cancel.c
@@ -0,0 +1,240 @@
+#include "clar_libgit2.h"
+#include "futils.h"
+#include "pack.h"
+#include "hash.h"
+#include "iterator.h"
+#include "vector.h"
+#include "posix.h"
+#include "hash.h"
+#include "pack-objects.h"
+
+static git_repository *_repo;
+static git_revwalk *_revwalker;
+static git_packbuilder *_packbuilder;
+static git_indexer *_indexer;
+static git_vector _commits;
+static int _commits_is_initialized;
+static git_indexer_progress _stats;
+
+extern bool git_disable_pack_keep_file_checks;
+
+static void pack_packbuilder_init(const char *sandbox) {
+ _repo = cl_git_sandbox_init(sandbox);
+ /* cl_git_pass(p_chdir(sandbox)); */
+ cl_git_pass(git_revwalk_new(&_revwalker, _repo));
+ cl_git_pass(git_packbuilder_new(&_packbuilder, _repo));
+ cl_git_pass(git_vector_init(&_commits, 0, NULL));
+ _commits_is_initialized = 1;
+ memset(&_stats, 0, sizeof(_stats));
+ p_fsync__cnt = 0;
+}
+
+void test_pack_cancel__initialize(void)
+{
+ pack_packbuilder_init("small.git");
+}
+
+void test_pack_cancel__cleanup(void)
+{
+ git_oid *o;
+ unsigned int i;
+
+ cl_git_pass(git_libgit2_opts(GIT_OPT_ENABLE_FSYNC_GITDIR, 0));
+ cl_git_pass(git_libgit2_opts(GIT_OPT_DISABLE_PACK_KEEP_FILE_CHECKS, false));
+
+ if (_commits_is_initialized) {
+ _commits_is_initialized = 0;
+ git_vector_foreach(&_commits, i, o) {
+ git__free(o);
+ }
+ git_vector_free(&_commits);
+ }
+
+ git_packbuilder_free(_packbuilder);
+ _packbuilder = NULL;
+
+ git_revwalk_free(_revwalker);
+ _revwalker = NULL;
+
+ git_indexer_free(_indexer);
+ _indexer = NULL;
+
+ /* cl_git_pass(p_chdir("..")); */
+ cl_git_sandbox_cleanup();
+ _repo = NULL;
+}
+
+static int seed_packbuilder(void)
+{
+ int error;
+ git_oid oid, *o;
+ unsigned int i;
+
+ git_revwalk_sorting(_revwalker, GIT_SORT_TIME);
+ cl_git_pass(git_revwalk_push_ref(_revwalker, "HEAD"));
+
+ while (git_revwalk_next(&oid, _revwalker) == 0) {
+ o = git__malloc(sizeof(git_oid));
+ cl_assert(o != NULL);
+ git_oid_cpy(o, &oid);
+ cl_git_pass(git_vector_insert(&_commits, o));
+ }
+
+ git_vector_foreach(&_commits, i, o) {
+ if((error = git_packbuilder_insert(_packbuilder, o, NULL)) < 0)
+ return error;
+ }
+
+ git_vector_foreach(&_commits, i, o) {
+ git_object *obj;
+ cl_git_pass(git_object_lookup(&obj, _repo, o, GIT_OBJECT_COMMIT));
+ error = git_packbuilder_insert_tree(_packbuilder,
+ git_commit_tree_id((git_commit *)obj));
+ git_object_free(obj);
+ if (error < 0)
+ return error;
+ }
+
+ return 0;
+}
+
+static int fail_stage;
+
+static int packbuilder_cancel_after_n_calls_cb(int stage, uint32_t current, uint32_t total, void *payload)
+{
+
+ /* Force the callback to run again on the next opportunity regardless
+ of how fast we're running. */
+ _packbuilder->last_progress_report_time = 0;
+
+ if (stage == fail_stage) {
+ int *calls = (int *)payload;
+ int n = *calls;
+ /* Always decrement, including past zero. This way the error is only
+ triggered once, making sure it is picked up immediately. */
+ --*calls;
+ if (n == 0)
+ return GIT_EUSER;
+ }
+
+ return 0;
+}
+
+static void test_cancel(int n)
+{
+
+ int calls_remaining = n;
+ int err;
+ git_buf buf = GIT_BUF_INIT;
+
+ /* Switch to a small repository, so that `packbuilder_cancel_after_n_calls_cb`
+ can hack the time to call the callback on every opportunity. */
+
+ cl_git_pass(git_packbuilder_set_callbacks(_packbuilder, &packbuilder_cancel_after_n_calls_cb, &calls_remaining));
+ err = seed_packbuilder();
+ if (!err)
+ err = git_packbuilder_write_buf(&buf, _packbuilder);
+
+ cl_assert_equal_i(GIT_EUSER, err);
+}
+void test_pack_cancel__cancel_after_add_0(void)
+{
+ fail_stage = GIT_PACKBUILDER_ADDING_OBJECTS;
+ test_cancel(0);
+}
+
+void test_pack_cancel__cancel_after_add_1(void)
+{
+ cl_skip();
+ fail_stage = GIT_PACKBUILDER_ADDING_OBJECTS;
+ test_cancel(1);
+}
+
+void test_pack_cancel__cancel_after_delta_0(void)
+{
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(0);
+}
+
+void test_pack_cancel__cancel_after_delta_1(void)
+{
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(1);
+}
+
+void test_pack_cancel__cancel_after_delta_0_threaded(void)
+{
+#ifdef GIT_THREADS
+ git_packbuilder_set_threads(_packbuilder, 8);
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(0);
+#else
+ cl_skip();
+#endif
+}
+
+void test_pack_cancel__cancel_after_delta_1_threaded(void)
+{
+#ifdef GIT_THREADS
+ git_packbuilder_set_threads(_packbuilder, 8);
+ fail_stage = GIT_PACKBUILDER_DELTAFICATION;
+ test_cancel(1);
+#else
+ cl_skip();
+#endif
+}
+
+static int foreach_cb(void *buf, size_t len, void *payload)
+{
+ git_indexer *idx = (git_indexer *) payload;
+ cl_git_pass(git_indexer_append(idx, buf, len, &_stats));
+ return 0;
+}
+
+void test_pack_cancel__foreach(void)
+{
+ git_indexer *idx;
+
+ seed_packbuilder();
+
+#ifdef GIT_EXPERIMENTAL_SHA256
+ cl_git_pass(git_indexer_new(&idx, ".", GIT_OID_SHA1, NULL));
+#else
+ cl_git_pass(git_indexer_new(&idx, ".", 0, NULL, NULL));
+#endif
+
+ cl_git_pass(git_packbuilder_foreach(_packbuilder, foreach_cb, idx));
+ cl_git_pass(git_indexer_commit(idx, &_stats));
+ git_indexer_free(idx);
+}
+
+static int foreach_cancel_cb(void *buf, size_t len, void *payload)
+{
+ git_indexer *idx = (git_indexer *)payload;
+ cl_git_pass(git_indexer_append(idx, buf, len, &_stats));
+ return (_stats.total_objects > 2) ? -1111 : 0;
+}
+
+void test_pack_cancel__foreach_with_cancel(void)
+{
+ git_indexer *idx;
+
+ seed_packbuilder();
+
+#ifdef GIT_EXPERIMENTAL_SHA256
+ cl_git_pass(git_indexer_new(&idx, ".", GIT_OID_SHA1, NULL));
+#else
+ cl_git_pass(git_indexer_new(&idx, ".", 0, NULL, NULL));
+#endif
+
+ cl_git_fail_with(
+ git_packbuilder_foreach(_packbuilder, foreach_cancel_cb, idx), -1111);
+ git_indexer_free(idx);
+}
+
+void test_pack_cancel__keep_file_check(void)
+{
+ assert(!git_disable_pack_keep_file_checks);
+ cl_git_pass(git_libgit2_opts(GIT_OPT_DISABLE_PACK_KEEP_FILE_CHECKS, true));
+ assert(git_disable_pack_keep_file_checks);
+}
diff --git a/tests/resources/small.git/HEAD b/tests/resources/small.git/HEAD
new file mode 100644
index 0000000000000000000000000000000000000000..cb089cd89a7d7686d284d8761201649346b5aa1c
GIT binary patch
literal 23
ecmXR)O|w!cN=+-)&qz&7Db~+TEG|hc;sO9;xClW2
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/config b/tests/resources/small.git/config
new file mode 100644
index 0000000000000000000000000000000000000000..07d359d07cf1ed0c0074fdad71ffff5942f0adfa
GIT binary patch
literal 66
zcmaz}&M!)h<>D+#Eyyp<EXgmbOv^9IO)M!(Eh^5;&r`5fFyP`$%gjm5%}+@M@=A(I
MQ@J>k5{uv*03B5png9R*
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/description b/tests/resources/small.git/description
new file mode 100644
index 0000000000000000000000000000000000000000..498b267a8c7812490d6479839c5577eaaec79d62
GIT binary patch
literal 73
zcmWH|%S+5nO;IRHEyyp<EXgmbv{pz>$t+PQ$;d2LNXyJgRZve!Elw`VEGWs$&r??@
Q$yWgB0LrH#Y0~2Y0PnOK(EtDd
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/applypatch-msg.sample b/tests/resources/small.git/hooks/applypatch-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..dcbf8167fa503f96ff6a39c68409007eadc9b1f3
GIT binary patch
literal 535
zcmY+AX;Q;542A#a6e8^~FyI8r&I~hf2QJ{GO6(?HuvEG*+#R{4EI%zhfA8r{j%sh$
zHE~E-UtQd8{bq4@*S%jq3@bmxwQDXGv#o!N`o3AHMw3xD)hy0#>&E&zzl%vRffo<B
z)-H|+CWHZ~O*S%cfYx9;02_ohIA<Bg(1SxF-6OCb&_lBkf{t<AM9r;%E(Hf#h{|a@
z9>mqo=v6>_2NRa#TwDdYvTVQyueO*15Nlo%=#DXgC0bhF3vTa`LQGaO9;jeD$OP?~
za$G4Q{z+Q_{5V?5h;a-noM$P{<>Q~j4o7u%#P6^o^16{y*jU=-K8GYD_dUtdj4FSx
zSC0C!DvAnv%S!4d<Yg@O<;m`;oSw)=Fz+hrL<mY{rBr8j4pi^88FX3}jKrYUP)>gk
XB^)11aoGMJPCqWs%IS0YSv(eBT&%T6
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/commit-msg.sample b/tests/resources/small.git/hooks/commit-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..f3780f92349638ebe32f6baf24c7c3027675d7c9
GIT binary patch
literal 953
zcmaJ<TW`}a6n<`g#Ya}rZCaYOzy?UGG&Tf#VG`?}7@eHtB=MTqneFUS%75oLS;atz
zm&iUo=ey->y@-{3h^^Cx;#d0zEA@DDc$nY4ez&|=%jTg@_HU*ub=!!y$xW09TSjlj
z(`I@QCsM`!9&80$I98wsQ8yK#)Orb<8re8FjkKh630D$QUDwi~(gkX=RunYm$rDjk
zlp%RUSnzA#6yjdG5?T?2DcYKp+v_lts0ljn&bh3J0bD5@N@1UKZ190O6ZeWr-BuZ^
zWRebCX%(%=Xoj#(xYk1Cjtr!=tyBesf@m6}8zY6Ijbz9i9ziI_jG9Mv<Cz(ymp*>R
zDH*e>^ga9IR?2wrSrAVm;eButj4<aWB@zzNl|1Wp@4;}1O?MUF>Y>7(E2?b~jsu>&
zRKCJ7bp#19sqYh627wD%D9R$8=Ml$TNlumDypl~$jBu*G>5fIR^FB0h0Ex&TGZNr>
zL5hs1_K>taRb!|ThN9ns7^@4MXKP+6aGI_UK)T-M#rcP$;kN(Vcf#P)+5GzWa{l@J
z>-E{`$1iiNVYxq27}<DnwLRXQUG0o_hw&da-s5T#H=`Y9D_8=eTZ?cpWatp#a1vs@
z2BjrO)z@aTuI#g#`)oJcnhM7oYLT@~CHX@CNXv4>j;uo%;)r3kJI2xCFF~Ux;$Q%)
wjbk6JlDCM`jU&P+UVOvg`|iYl<7~9k>HHB4I;pdlQ=I-^$DrHaN$@lH1?P!0U;qFB
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/fsmonitor-watchman.sample b/tests/resources/small.git/hooks/fsmonitor-watchman.sample
new file mode 100755
index 0000000000000000000000000000000000000000..41184ebc318c159f51cd1ebe2290559805df89d8
GIT binary patch
literal 4777
zcmbtYYi}F368$Xwipg4lq(BeHMvzvH-4;n7DGJBPqq#tw3aed8+IU5-m)yvL>;Cqh
z8FFRGj$`9CA8ao<GaSz2oMCnz4Rv-gw9b@j_$0GQT1?XYi|;S??Y`Z6`Z;}C6#27N
z8ShS>J?j^$%==FV``-=rhLcPW`McSytRm~mEO7_&_cAVZrf1fFy*ha@8oe%*-aBYE
zcjzZg>LOkgxuUr-XJnHyD;zmPnRaSc#!k_P*d_BttRdc+J6G7za5#+<HG#rlmbrN~
z8DwU-3}VABEwM=0VLP@^Dy6ERR5_J6cmg|GEh*M1EliqCGwe^ZT-iZ$2Yc`4!I#WZ
z5nGGhn7*jeW=2ydsmfAmm#=8AD<<;TI+#z{Q)kW;yE!%GB6f~7EtEMLdM47Qaz*8=
zIObA(VVj-VG{Ax|66d*hi`+bRG>^Y1nkc2Oowk`ya47uUR3Feu?B<phm31&PQB<lt
zb{W(W4wf#Bab%|Q_tKPS?3^o=5)Z8^Vh(#slNI}pO(f^|{U0GZhLnycSaNd&h?CaC
z0XklU6^<ky6rES9T=na$M8P<_)aKMAMo+UDewAu4wF{#&6diFshiudixAoh|&0<BJ
zR>(w;S{(VYzxh}q-=#zP@uxSx{wbyPUMFU;K(06)$o{07&3yI?q{GqMcQ1c_^M<0<
zF4acAV)Il-V(rCTC1(;bsZ*}bl8dmejAk~yb`B}!^0;g^(o9kGUfZfDOvyp@x4OQt
zSgWh6T|3eq;9MFs8-#z+FDM1h(IjRUP|``PxupgJ7CUHOH90gbgl^2~97`?_X{P))
zB*$r1cDlF-%azKND}?Gv`2K8-9v5e`gQoft=j?T<&a13c^!wY_$D`5z-X1g?ty&6-
zQN50{8?bUk9AI->^W@~~nkOghHIC2YN+<JiT_ob7ttND1oh`HH28Y+OV~HedG&uB`
zy}rA*r_xT#bR`Q7-*)3t*(!Hf?jKzyxk=8hdi3L^d<p<uU9q_<4k&xEr4@YWv_vsW
zp(#32bYtA5u|s#1+}h`0kwpz4kUD&+>AXkLQG_2-{Pq3%{`3KUMeG$iIn%%^6*NYb
zn|_BdV#C)n4565Vcc<EWC-nglECZGy!j9I4&;hUCzr(?6TftH=0^@!mI^E@y5HZw8
ztH&kaSNyg=O6riqR^MPOX6oD__Jz@K;*icS)?m$!p{S$n;*EwlO<L!d7;utu(W9D!
zaF!B~xV^2i?wH0s?Lw%Q)(`aPkajs1XojlPv@Y-d5#YFg#KG+!r7A(dJLnkiJMs`p
zV|_=d!upN{TsxH1?sZNdzxeHsmtzTV`1{pykJ_~+^*>X;uT8&z3vSi!HXGbUj2B!R
zdz~&#<?<tHJql=W&((YpzS06y-Z6Cn^H!*9qH?pOrD~(GV=JJ~z{tpLnGK|xp&C1`
zsbn7O86xjF<~G*S1l*;;Bh%6><Up=oKy99?62P^?M&22U6QFRBXLb&u%=Ur<74wRy
zMRG!f{AvZ>fk#L-&k$fLwo$4?>12g@AXOKFekuo#6EHB%gmpD?1eyh%N8s{2wGoTu
z*@6cEZ^ZW!FAF_|JL`NkV7k}0ow|-2jHwbgH0;c@Dq*o?@&c*HnGdyx6^su8Qk%2{
z*ye(dxO*6-&>qn1+zw}tc6;=sOX{4WB=VqjTS^))y1jlX2Q;=e!qMmFA5lC$#;BxC
z=Y%tRpWxb+_uQAvAw7Q{HGV#R$xb&udLCzZ+HN?kTyB};1EJ8UlQ5!>5eGW@)RX0n
zkjj>EF!3=0Gl^8dzv$B^NMGRxJoqN4A`xq-@wCbrx*u2NmIJ1<fUDQ=*^)h6`vzco
z3F+ro$F!F6pc<B;<;isobIgbVGKUBByoQ4#CO({S7g?<Dh0^!7uJ3gxS=6F;+^gQc
zeKi4`4`Fm3p|BU2v{M|-u!#QGCXiM=k=%np0<ZOPQqEjP_nneyOdgEuG9IH&YYPtp
zKB_dvcYCcyhyT#<uhUEL$o~!TUz;cb&|`uSM{Dkv%&W2lcpYL&kv)tUvVv?&>xZ%H
zh;{|4T3(!E9sY#Ni(wUJYs1MmIc9bl)(4Nl3_wD_BWB>i<1S(LX7m*{Q7PU$muMS*
zM!%0EZx-Vw=Zey;erC?SNxF;pY@^A%-krqzfLV2meBp1vWdyArFYn`DD19T)Hw(?n
z)}{NP(Lk(o*?gl#B@pP7^*r|=;PIDT4|F#{2Hzh-AL0Rv$6uT;<CP7qxbYms@WU7}
z%}TsHJ!V_56ZFF{BfI=8jTBxMEATG376pS6a;u1@c?{~sL<N52`U6fuLkz4P@Mb^b
z2`Q48y&!C0&A+WRCJAdmo3u2#?eI=si9Vm47$|`m**x<wKkM=QR&g?C63@P5X@xP8
zi5QG2b-Fdz%S0%1{kKvL%^RTdpK|SU^VJ7JCmKdwp1`;HRoGM7ef^k_s_}2An=cql
za|{IaxQWpdq<ae&x3BOJq+M5QNIk#C{Nv@u>n|WzE4=slK?on@(fZeGhRgQCu56qB
z{+n81Az96qnQjMY*-*r-KV*7;Z#4Q<xfjbcBx_6JAN-HP@bq+eI%HhAB9&vLyOap{
bw<Ywj(b#kdcCk7dCBY;|DBNpPjIa1F6*dXN
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/post-update.sample b/tests/resources/small.git/hooks/post-update.sample
new file mode 100755
index 0000000000000000000000000000000000000000..9f95e41a39cf974958277c8881ac6cce405ebb20
GIT binary patch
literal 246
zcmXZVO?HDY3<Ti4Poaiwjq^yGw9DKf7mz^&Ly=V5q$H=W^Rt|p_r9s#9Ea7VERo!9
zyT9>uJRJJV$M^KdldiMhj?ImK6~FvwJ*L5a){QoM=L5TYHkGO1$UrO3`a>{?Opw|b
zG(#59NQ#jFL9v~vgOVkM@^^(^A}onOE))yWEwhIlk&{ZyseZ^O0b=w8&O=BK{k<5B
k^Q-B@eG}LeHrquz%(SVEp_N)VhYZikCW__82JXfD17`J9Qvd(}
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-applypatch.sample b/tests/resources/small.git/hooks/pre-applypatch.sample
new file mode 100755
index 0000000000000000000000000000000000000000..625837e25f91421b8809a097f4a3103dd387ef31
GIT binary patch
literal 481
zcmY+ATTa6;5Jms9iouO45IBJXEg&Jm9@v1LPHMM_ZR|;#6tQ<EeSrA%_2`_rGr1_8
z?aM?yVtIc%-@9SGSk&8x=grP-Lf`7!^=$7xgL=|ysZ}!av6zL~ywui}<2##V6L@!k
zy=p^)V7%Wzs-g`9<Y9}^)&uN}BCrXR_T3@Z2$gSJON2`X=mAs+%@7n-2I}ZrP|TFA
zvJJGDl3HPLP<@!Q!}zXQvey#qEE#a#$vs97i4=A0stF@YQ)k_ZajaoS^dVYBc&37_
zVI(L=X<V335r9~7T<;|HfKF+yM}}LB9d96V)Si;sj(;9Rh$#P>h$71hSXq*MxP;V&
zj0cY7SCL=x4`a46sF)C>94Gk%=3q$W2s;j6iHtB2$R0%gix4oK@&T~=ALd_o*CKxt
I-`Pv{1Bpzc>;M1&
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-commit.sample b/tests/resources/small.git/hooks/pre-commit.sample
new file mode 100755
index 0000000000000000000000000000000000000000..10b39b2e26981b8f87ea424e735ef87359066dbb
GIT binary patch
literal 1706
zcmZuxU2ohr5PY_N#pZ0-F<{-<Zkx1j&cMNSQ3FN`GzR*R1O+9oPV`BnOj7q@1pV!u
zrF0Hl^i33(yR)-1d-!H%&2|=|^E~_R{N1zNJ-&Zmt-t?iwXv&i+ZN}Km(TX8Q$H4u
zd7(m`|1iDmF5k@xV`p;C4zojASmLc}yN0QDZbhN=ri&CEt=XGuN1IwjGJ#a#`t-kG
zDqY)}7+Ft|;YKwLYbtg$S(-TBO=x3cP1cd}%f4kB!<6Wu-dCwz-)KDMEuM^_Hh*UC
zC`1)|)T<(U6b`+yOH!6p*Ll}@qastwA*dyjsgOf5C=?LTprfORG6O{5L%@S0wyHpj
zu|_A-=NWnYYR5m7kvm6|&T~GzoJ_OKR3sgFUgw?ifho^NQhvK#{6g0=&Fh)%n}#m0
zk1sNmwb_AMDq};OOGw5|;OyX#?yQMMH6yAk(x$3tjFjHE?c$E2XC_xXav8tnIeIG?
zYMI|~MLEVGkuT*>v&v-X^RA+u>k}E$4d&uD7=g_fA8+pNNV=4s0|iD3p<=DTXClTS
zXV23tJ;ECmN@M0j@zUAKEYW@3bv!SeYZ8ZH`YQNTApFVNc;F|9r5p4TqGs=>8E?6y
zi|gY{iM#PG1nL?UE9YCnWTk72kgZPG*Usqw!~Qd3c?~@w2?%eg@~)+VlSs6N5Yf2^
zz;ow<fjf3n`imj7u5lnzt||q9QFM(H@<3EJCK|l5P!$yDfn~U-(5Vu7s+GqxNKyeI
zP=-Oa(KH&gK`NhUa`cLj3B8%qL};DR7dk!`A^h&3-hFB6p($5Ufy^tGir)3et}qK4
zpkPKYWzC+?=&gw-L6S)S=<lfq)%uLUAa%~81Jf9hk)w~U!DItnoSy`m^}#38M}K-o
z!PqisQkV!&z4E*V01ro~qlVK^0BI`pLk6D+)f~*y!hCvwHq8zY9BGh<2s$@b^A<8G
zRaqk}&qZyyv&|0QDFPA%h4TgF&vdlc|JUq*=>F#K#r^&KMq1A`oqVGFpD&-!Pv|Rc
zO3KSqA@h9nSc%bm`0)Amk6*J}@14J*1-219l%%7D!Pl}UK>|lVi0Dfgu2jN3WC!uL
z0ej??b2iSehVgdnWHmZV4kUo*QL#aiIp}U=9x)IXk}JJ7VQ;CI9Rtn5e0VcjbY<bp
zv{}WlG6L;H!EzFKdY>cVt+`x5D+svCGD<sXw4|)E|JX43I1_3P(sI4{wj87bPSrNG
w!SIr>;Z5hm*<gY+X;)Ryx4=nzaab9m`bwE*^s(%u*E3HbUuOR@+&s_z1=MCi2LJ#7
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-merge-commit.sample b/tests/resources/small.git/hooks/pre-merge-commit.sample
new file mode 100755
index 0000000000000000000000000000000000000000..d8e8ad4bcf8337f23c78a98537ecab4831301202
GIT binary patch
literal 473
zcmaJ-O;f@!5WV+TJd4B0whSt$H%Dh2t`2u6q1!glCNbGU;n%yxdNsG~zR#WA6xIwy
zWEZHoU#u?nykD=Y<HPgeWDkDm^kTof*l(|%^gh!nHrZpo^vhMDjV;E1GD~K7wV*+D
zz9lry9T0cHcm_KhDVXYvQ==FrLTT4u=bEr{U1yl7%thf%wJnv<XQZ`ZbQEezaWdS%
zI;c?h9a)Y!ux<WK8rQd_aA^?61hv_Pf<t7*z1USuL40FxYz<|hybsO?qnN}aMpcuf
z6phFw1%Xx=wUk(m>E$jSEQZ%SQ(}oLgslTvrKK@9Qf#b!hajVFnp9@oIix;NcI9Wk
xjnh0ya!AWet{I7YpD;y6HXyzI*lfSvH=o6*7mJZPkuaYpm>vzZ`wyGEBtOQPo|pgt
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-push.sample b/tests/resources/small.git/hooks/pre-push.sample
new file mode 100755
index 0000000000000000000000000000000000000000..02cbd80c287f959fe33975bb66c56293e3f5b396
GIT binary patch
literal 1431
zcmaJ>U60!~5PUX&#a1@z9B{IIZkjLT0t5kq9#8~D(I5{+8&J~9;#ndUk~-ZT`r|uG
z$#K$$J{TsK<m~Lsu9iP+t-0TZ=sa(K+C6);54X>s*LP1}9!GoZ@4I4myMMG_di|of
z%?llx{O8TS-#^<H#%^V=)RNv>;(OioEmPy%kwWQBA1OMzV{hsQ8XFzS1k!~YQoLa5
zhtP1fA$q6VmMbbAC_9)4I628k*O5J$NR19uHe4QYDK<==I~SQk)Nu%xQ~<Hy8U>KH
z53w=!ke(FGb_PpnZfd*+hnXDTn;2*`u^~;?+5C~cn?bRka7NR%06%e6O91{MAgN6J
zmlO8{Biw4&wr&&(z4p3eln`E}XR9m9bNYZ7Ibrg(4yZIXrfgD7N*AFD7L3YSM#j}%
zo__rOS5fr;@8UM<6cl+cv_$YB$PQ&9dv($eM*))g!_cu!QcSh-mqE9i#QDZT)=o#`
z?8!RtE?w6p?GkGZ-6yt_p~5~4ecu|Sf^)6096%h*q-eNiEA1;Xwg)p~Q&iGSG7-IQ
z9aII&`ps$WOojFA`*bjG<mBv1n0hcYZWN0~(X01-hx(ExqWqaXgSk*@-GMp|K_3!5
z9|O21N3%~izh(4fbp9wzd+!b&7cVwSP5H00)m5ej-(s=Pl#(90UOhn@OA9u+D{i@r
za4*CP0I#<d-)-#xq5q-iY5nIef2s5OuQjcA>kFk|E@sHHuD}W^d`7YJ3YE^zrQnqR
zGoq?;YGKe)93o|_=^f%3U1KYZGPOXRRxK7w`UUbMMa3<86OmVH!EKP$8RCrn9mWX+
zC?9yF!fRVLmud3hF<}x;;sR}f(*r}6Gap3fR6zLHR~kbMgD{98N`L+r&?3p~*0+FX
zcAL%j=(SO}xTJUTvA`&Lf`2mv4koPG9&|<CA~EHbWHMoFPwT(&U=7t0`RoFZPO9Kq
zwwe$i=T|AXY#hD$aZlNMH`wZ%gwilGJ(zeYpOn*F3cy0XKXio^Sj#WXx=PWV`WGaA
B&~*R+
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-rebase.sample b/tests/resources/small.git/hooks/pre-rebase.sample
new file mode 100755
index 0000000000000000000000000000000000000000..193ee3f127387f469894675f0d621b5bb9c275da
GIT binary patch
literal 5006
zcmb7IX>;2+68$XxiXKL@ma;l5d2^5Ba_rPh_DHI-u1#&_upttZXp;no03$20|NFiM
zK#D#xQ>!Z3JkX8T-LDVm!B5j7y_{;JDmmTTef+K1oIiPzeEr+Ai*<2PUgnG4^ZB>p
z_fkAvoR1emuf~ri^K$-px=4#D-v<wZ2Xv&$O_eTJh6d4)=DWL(NBs9G{k<+yMMw0T
z$VH*-+LM)}u&m^`l8~1nt(3Z;R8v(KbY5#i3z+~8h0D}Xvq&3J8BMWDizPNpaeb~9
zBN9bSkthfXzskapf%Zt{*e#}{QaNiaAVZ4{$;;I6<vKNhO@%7P-(;l-x=pPoExHC!
zB(hA#cDdD?s4P=!)=-K{<kHAWKetl-8I8wwO<ihJNs-$dEvr;&S_@6E=mNSJ5;mg#
zyb)MbqKH<one{qrV;ZQ6WL}yLtyi*ekNLf|uC6M!)Cmq7*l?g0d6`MlE49|}>Y9w&
z`bCv#<YfTKtb`!}CyNYd;|(C?vRVQmWOfR9X?FZ#=f$No)^#4>2zVn=YnJyeNey(Y
zRh`9vtLw~A+5zsjp|W0Nsa|29Rm!B>OoG5a+vi;ari8O>KkU!KAWg_fa3btK2x*_@
z0bEc7J;Ubghm}n9bOi(Sv_B66nQ7U)J7f0fO}<cB8i8vG{r39s_>8Wuf*uorcIgEG
zOHc|-V6+HlRhOP}?Cn?@5iwSl43abmBA^2lyL$+cpabCGVES+v^j^FO_}?FIp<qP?
z#_?*YMHIkyZxJxS;h@A)WDJ0bN&+F-#_lFjAf^g_Pb#5YXqYe|dZUpa^zI)VOBXQQ
zAMhT>%En%Ll?Z*7*}TwrZyg5OSZ9rY-`aU~Mc-jjv{Ll)FLMgtB4ujktfQ`Xhqrka
zT=P!A;9w^;Z?PqpLwOLu=cj3L>TdUKw2;DMu)`oVkj}<z_EjO_2uWYuvKG==%Zu?h
zJiMVR^c30RbpW}<+z;jjoNC}YI4f6QC7d-08*4mCB1>#bcDx4tYg=j%D`+i{W~fVM
zVmZ>W9VMyin9c-0KzI_;iZ-g|OyzuG`Yq%(%dvl;ifnVr0;jWE&S`z|rQu=!yHBBO
zx`OJ;oOQ(KKM<$(bC38o>pD0%|HA(E0TRw7qj$fJ_pRN+7Nm>dS<q{AcOz#-;d7_2
zL$z(_nhH{vOzT(}>C(gLg{(`t+5Z=?o+}wXU4tHy+&%F&aRhFebeEhR2R5|<c6J);
zEY(q5F5<n*XP0|=PtPBn$B)V~d$Os-?&8UlaVe_|jdkzoWNsTP-_uyq4$R6o<h`&@
z{loXa{^#TF=NJBYu9qB?hs}x=It_O}ZjX(_wy9@X(lmkRpNi0{8T{O_b_j*JC^_SM
zz3G?1$KCNWF-|`Jbx2cQ-y5LW?Z2eikngTZmsx5C(@({8<l)Ue+gIp##Mosfa~iZN
zZ|NLN9uE6Xaqpwk+@D+f?$tg2JRCY`pwZwbR9OvEn*zYm`ffKIzl4{r{ZgjfpbuX)
z_r0=0y3)T-j$gljPyEJO*6Y<pj7G72aLoqaTpc=#u)*xJcVUm0;k(n;r)^DQNa6Oj
zWvlHEGg~lz`Q_8`yQ9<BZ;ylE1Z}bD<8}<uB9Y5lRB=;JUD0h?_)4H&EhJjvwzJx?
zr<o_vk&75j_5^d$8Z$_Oc1=R-I_DlNZ2@~81oV*J6%o3RuiCz}^VEW>$#Ycbp^w@t
zTl%=f1t=w+WpJzF<|CE@?SCNAz)%9?w33lQ8vrHJqPfH9@}qs*QXOG71W=ylx;wOB
zcx!Bj^)Yy6WX$a^vBkBJ5Cob<oubBW+a#9bX{0bkMTX_2sIrs`HMk@$q{dK5h2-g}
z({`~#gm#G?c#>qlaDx_B0c<3b+8)f84LCrt;e;qxc+7>VbwVK{skNv!wvBiTa^9Iu
zkwP;VK)jH$WJ{`MRwAA9fal!y0dtV;FWg8PTkWU>CwnqD>1ZX2B@;$DlX%C5MI+}{
z9xQVnffR*~v2KAUj*hCd<gRCjR7~6Q(vF%VT7j97iv3j4Z0p%mU`7t061~l7b$!#t
z3*Pveii}aQ?QD9aiX>gul~`bk#mk`o>zk9)<2Uc8?hUZAEvd!`9em)~$Z)zev>w^8
zyAgCP_$&Y)7HSQ84`xG}OeTavaEswwF|8Xpi5iZzZa@hCiv(J-%bfFC&)HLlO+Rhw
zG6g?9eL5&A!SuJnQ6}LxG%tU+@vZ`i+!+Rz6iYvsTdhnPo7lW{m-}{hya@viX4)XZ
zngaw+j;gloB#|UwI@8sOmQpc`h+bicQJnQIB5eifIMQNgD2+oai33m!34~xU|0Azj
zhu$8z+T5^;Pxx@d{N)pzOJLSa^e;aDf$W%N5XcOf!mGC9l9j$Ev2h6N+6ZQC+CJzl
zaM7?S!SrFLS2DASjj(h6y1WN3N?|bmqmyzm!&nLoE|`rKBOc_yDF$a#FsUn!IQf(t
zdC&Us(kQz*7mv<H12p@UD8XaLXdK{>H^j*^MC@>wTDb}g%~sx*ng#>{@lR=XG-Z5_
z#<9*Oh0joMzt;nS)ObAp)347`D=}r-;nV!TbIq&xrGRGsF6fZg+!VkfUei@_&l-M&
zPqQ+Dw)RV}+)I8RuqAxa`Pv8e&!_gXS=e2-un>=Ktn}-;%lLZxaVn?Q>yZCb2R3Wk
z77zr%;Rq&h|2ncqyKYmFI0148JVY7Q$V5p=dWj<MQ<8r+@QLq+UROk&%quICq^O2C
zp(17882jXI)_GaqzAY4AjoB_nh8k*r1mJ>+Qqpu%i|xp2<qF`Tw6&3`h654ceBjZ7
z4>C=WaOb2Wudn^h0EcD%$p9YVU1fnoRV9`(cy(vv6K>FXS!2jY>1GnU--7)4usH&K
zao*&P^@9~YmUe|ZdLW@C>H;!*<TIU}-!Tw#3oqZA*99}b3&uHiGO<{I6!pMnAiQdS
P!fA@Yk4s_DjDP<F98V`a
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/pre-receive.sample b/tests/resources/small.git/hooks/pre-receive.sample
new file mode 100755
index 0000000000000000000000000000000000000000..32b597f8c32399017c3d7d51134d1f6490ad9575
GIT binary patch
literal 601
zcmZ`#+iu%141Kn~f>Vt3>Nw4M*;=?j(TBD#O@XCv0|MEhA;z}kTFRv@`tPHhp=&Yh
zg%Zhg4i7o_k{a5i&f5;tZ==%}^Sn4aD_6%qs<o-wO_Prn;}`SPs_*$C$(7T|$#C3`
zPt%-C8gelZ1GqAP8`ZQmg0{8-S9H{R@D>_XAuJt&EumdH4Yu`UjT<s+s_~uXh}qA8
zg|_HG)%7Pdc&$7*uR0HF@)~vmFjqyD?XZwCbLen^h5t)sm9<90Oa!@Y%8!~rF8G?W
zkzmCF8kMtuuelMHIAlqqnm?72LeGM1J4`w(kX9&%LQn}ForlDLjBoCyvxmo@x3kH^
z^loxLyPiDWPo-cBMnsg2M6}kuPGHEGBqVkC{D&9Kt%xFAsTw4QC1$_=fwBl=3dI+e
zaSxI}JS}=Z(Ec80eF`!Zq3mqapXI|UN!a)t;@4hcu%Eq2xcoW}%!><-+XHTuHss+b
YOmM2;hq8Egm*4=7_P9T{21QBYH*F=mfB*mh
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/prepare-commit-msg.sample b/tests/resources/small.git/hooks/prepare-commit-msg.sample
new file mode 100755
index 0000000000000000000000000000000000000000..b1970da1b6d3f42f00069fd17c325de72cda812e
GIT binary patch
literal 1702
zcmb_cTW{Mo6n>t6#i?x6xmZ$SFLf{QfG*3r0L?Pg?px55l8$UTGO3bO;spKi{V3XX
z))weX0X>M9bNMcZ-6yG%>(n}JI2|25dr<ew@wmMG{l(3lx~bQz>}WZBP@ih?JX^+@
zu#5O48P>yRX(m<b*PU*sORp92TCD1dX`%HE+1$w5k<(Ngu7zQ83#MGJR?<<W=d@yL
z#heqwo{FmCg0g#x<~R+PBD#}q(MBn;V$x;%UrJPP3*l%XtlvTWChI2SfJ$9e`YvSj
zRSOQ?NUgSMLI`3vL48YBHzwzVXoe7v0ef|0YHgV$N@?N(-R)rPqRDrK$n(%+OF$`P
zWdjC5N~`#RjV9}aYwQ4_yF5O-$h2`>fDIhYP)doc1&TADZa@ZGpusJ$6G+e$ZMcmC
zoOosDQPS}l{H?YPsq(4;0SGkATa9eeqAaDcj<jNAU+LTSmM1jo(ti~T0B7acJnnTX
zTarYy;Husdxal0!S<ba8=uu&a*SNYtr7|d7$g-q3_JHER2*oBsVW~i~XXdMx%LW~0
zT*960LF<qZ6K&FZ;vGmtyywBU3^Q>q8n2wALbFwU@2i@FAaRV!=uw-nwx1gKn2SvY
z>Ff>;2sg!+Hxfkwv1lsiii=p6WenF=5)6LZc<a$zDCD$GRuwvG4Fr+B#h?#9gTbio
zk#MdxC@WY%zSGN#i}Ts_#q`bf-{)`7CcWeB*7WlIyHjioJJWw&A5VItPUq3^9!r}S
zbykelFV-VFvcr>QaZ=aS_}+-4Y&?!@HWh|<^gJ21!|T@+%On#w6azxPHV}XsRbe*w
zR_TZ2XEsQa1lPK~biYqg@0-RW@5J1@=<87cFzEUABdCoFH2CZo?}l(Z*!OFqUxo>K
z_d`l#4d9|H6;VPT{X?^{VJ>oL|D7K{BJwwqB>`YcPoGk+9hbvHnoQ{EM|kPgD_`wk
zKm4#2xu;-y`RAm!=L_BnLvJ8$AZm8@?)v<%vwvsw8AF2x6!mTT;c72A_~U9nIq0ST
zv)N0!I!^1p=g8-RQfx5)E_Mb_4I2vtQpI30XZ&t<!9D6nI|;V7YR3)l6=S~QhuwLQ
g$e&^kTY-M99*<-Iw@(78*M5W!4}U}|8YyMx3->-9h5!Hn
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/push-to-checkout.sample b/tests/resources/small.git/hooks/push-to-checkout.sample
new file mode 100755
index 0000000000000000000000000000000000000000..a80611e18896f212c390d845e49a3f6d5693b41d
GIT binary patch
literal 2840
zcmai0U31$u5PXh)#YOS7cE^-rw@uolNhe9&aUS|HtvhX>G$45tVUYj>fRdF?|9kfU
zNR~aG=E)WbEbeyq7JTw}ZuHIE2kUtL<<n;$&G!2F^Je|kx2ug=4L5!H^!ogx`7o$&
z%Il(3zAe6<oe$^F=A|}s`8}CDp*M#3M)gC-)LOeDUpYMl3YNy9R)I-T)pE7sy09aj
zJ7%&5PnSB-F#2{jc><WLR{I2izuK%VHc+{hRfXe<^_q)8RjcE(6WX+F2)iAtDtI{x
ztAHVBq)eSp_E^xcV^i_5KLIHA$g{zEjh?rsacu+(Eyvve2~Kmw%;n3g(kWB56j~Js
z<yE5tYUsAR&PY0wgRvM8x!zgLX8SI!eVY&}YZ|>AoeCNptd-NM1aZLhESzC;I`+Ns
zfmNNjdAp^W8#Q*}l>CT7RB9F5(BbI8ly2l~+E};JW|>&d1)=epZ-8vm8ppkbEVn#R
zt30a5A-c(YQR8eM5%;|UAnO>rt!&@x@G@yp+92%w-}%(5P_+P&Wf_zb$f-Qrl5(7z
z2ah(bkE;!DK(&aAMuQ%1TS>ai?wSXCOCSj=_}8x4IbCx^$}9q)<W{Y<9o<WqZo^oV
z3bR;Qa%VT-M~kU@2n{=+=)C!MD`12;@<F*EtPfV3K#g^1%&ggH?4{m<R$WEKcUI4%
zl6{ik6Bo46pmNjdXg5@?hn;SjG$}rr3Gy#-BGgVD$8oD)OcK(oqca)L_kk)UBZ_&6
z*ourb#Yc8l3J+uSda_Y$GY--5Zp3QK9w^?P%E0xb57?fY+Q#+KU4)+R>whwv)SBt|
zg#MX4;;Oau`m=MI9(^&zPbueY@~>3*ixX%mvR5m_1&nAg@ZKvY1E$O}&EtLiG;mhV
z1xhMIm~fGjmf_#{62f`y;09?I7M1W2tWQvz<}i9lR>OpQyUJi45_&*pQus&EkwY<>
zI|ZAx=*3i9a-)g)hXkvO7>UJ5MNgL(Z+-wpXVcgbSgpmFmbf1~DPA(OVGI&FNLeIE
zNH!_aiH$vsif$_j7=T2{cS(!DOI`~bn@)vSd-0d7xL=DF;UNP|tW}4i<qWTSNCe|y
zg9kV&L9g;FhC@tvsVu#V-brqShHy2qZpA!gg{ZTY>h>DvHtu9tY_pbJ6x(6E*hxgC
zzNDao%qlr-IE%YGbS4hF!n!on7#W3$bX-_hbZAaws^nHu#)Dx=WzdbJ>AKzAy@T$x
zSWE^x9+|TEHVEPyaPYa0DOChp?AeHSBBDbZNokQpAY{lE!7geZI=jV)G^2@<iI(N4
zJLJjy@Y<VI$yq;3bVpJICW3D?YDMDl4Oe5pDf{d`i1_3Qx%4uX`$dOz<9e}jm2B-u
z8-?%!7XuiNF2O&MAdl-iw{drG+$F^zT2Z7WRV#c6;IRZEf>l)&91Zb1+`T+oq9wWF
zRV~kGTGce0O~p^6mj{kT5kL(pv>r;Lvd7VDX*P>A^Th`$3cWO<svk?_?FeP@458*2
zA1PqUOdd%VP5&4~ocLK16{1GLll64c=mU81RMFstpnMoLuI7hNIE4N)U%h*#<Fz{C
z9#>0<l7lRrls|W(8=W1O80P~6+U7<4BqC}N4-4GR3w#{O7YmH?JxwJfru2d?e){$5
z@5R+`7Z;1)FW;OkE-(HPisCS;5F4O^Q>L81p4Ysdo3ZP1(SrR-peEdTo;-@bkB((G
zPHYQXUL!@Q$e(OQ;R9r%@Afz+50I7>*^^c&&|E*r-jN)LH=pM4AqMwWxSv|nqjddE
Z4{_hwv8!W(<d3>T<BC%y-6O5i(|^OS%`gA}
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/sendemail-validate.sample b/tests/resources/small.git/hooks/sendemail-validate.sample
new file mode 100755
index 0000000000000000000000000000000000000000..effa6f54f2a5c1f0cebe258bf4805904be4b4852
GIT binary patch
literal 2365
zcmb_dU2oe)5PUX&#g-Nv2{5JDX_MA~3%Iq?8g*kpRgpYZIFU=~BI=I5J6e{T{`bz^
zk+$48h#x9Ika!=nv$M0y{clD}-j1x(hDWbnzP?l2k8j?TH{brS+Nf21yPm)Nczma>
zYw`X3V>TCdnSD1ru8&`j=2DIPbCT@SnIgUw>$+lEYP}+x8(BMYnr=iT3*ndq)xzaV
z>I+qjv}vC#8_9M+b1p#uNS0M0)q<p>8!3p_LRQ0MA3M`!2foxzRUjbFY@}O~(ki=S
zqscnq8cU*dY)D$$cqE}n)V0yIk>CNKHCrndOtSP*HbOb;nbwAHSb;R+gs^?^Dve%)
zoW}t(*D}$>O3ab0TS^-;J|u&sb-PkZzo#kn*#xYt(;<xzKW(YtQZ$u23?yV#kyh1~
z@+Idh;EG5jXx8@%<;Y_W8SA=|T;MPQ)TB!!<QcbU)YR4)79eeeg4|vp-8jm%Dl3^I
zRS7+i3>FGuwzSb^g&RDiGcOz9TB;Hu`nJh)$W=C=XCSm2AY=$w3G3P-V#Oo+N*;#2
z4ijJ-pBZ=;T(RTgp_HYrD!uW-dTMfkuqY5jwOy)~gM;#=P^i{!l7`pXTS^s(&^{RU
zydaw}OpS#^D1cXM8?FW+fh`t7D(g;yr6|}fdaNtZBx3hlK~IpkTu3!Qq%R+zAo#<L
zU&m+XIFB0>t}Bs8^3$vHD+-TGT@`F>H1Cc#WAVW;&$S6%fE2d6@kLS0g&ihIM{}0z
z8#XhD>b>3{(BH|Px7}&lJ4%y1v<t$W+!F|W@<gaU4;MqSHKR(wdML<XnCv;zaPrSi
zxVCvek26-bf#Q!H+uAgy_{e_1UZCq>(CihZJx@8MPoGdl*BJGD;usf*iS7%;{Joe;
zNFuBa>*~o&qETDPo~u&~$FxE1xb^x&(CbE`Y3GfsibL2rl+L;>P6j&Y3U>K$mkp*6
zd`Q{<^+^&;GskGjwD-%!boR&i-TC<Uvy02w+l$Nb?B}aL-%ZDpluqd=K_@}z*fyuV
zzAs1Hf?3v$k!X7GTc8Q=r`WJ_Uu=>A9UOR|@=GYb5x#<f&WSMH%mCJU<#=7=qFdL6
zG?Wz&6z&J<@I(B>+dhd7fkaVIR^pol`Mv+rUbmZ43dVL6^S7g3{NsPiG$iy$5EDB%
z6KIgnb$H(n&t3e4E6d4V7w^B?JS}JkG)PM6+X3Co`SQs($O*AA+MG~{S7RJ=cy-l&
z>~%3y`tjfx2>uOu<lDGWewcb=oL@}B@B6FCZ?oxSJWldrm%UfOduahk%C0H>tB_^s
ziwG=e=ch|FQ0IkN91US7rhdQkXhwwt$gU0WEVDjo=IPb+?6PC=s8}J*ua(Ms))`UL
fi$|vMHn?H<rred|1&u#kOoJ`%vx)-*g-ZSfgGvdP
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/hooks/update.sample b/tests/resources/small.git/hooks/update.sample
new file mode 100755
index 0000000000000000000000000000000000000000..c3430f6fe27e572a1cebdb2f2bec70add0642a60
GIT binary patch
literal 3707
zcmb_fU2oeq6n(aS#jRq*br9K3y0z;^QgkUc^kKlTrB6js&@yE)mPu8l<0$L?`wmIT
zrsXba))@gJi@e|G+<SfSXe`CeSQ}OG@sr8ZTUlQ{dzM}Q@O-hBi}GeUom`#X%FiYH
zX?m4Rna-0RN2lfK)A3ZuvHcz$L<jUn62D=~vfz{}wIH2VqBLX_O$(JSXeF7H$}q!c
zWY}C&R;eX%X?P{%d;|>_tSE3ettp-hLlsZCxaLX8(nU;bVRB;Ce6@s#eu2|WvLz>-
zvy(&>Gyfp@+BtKnpqWkKi^+v{4jn_pNw_zeuxE<mRXKx8G3;9pl+45&4~hHW!G@wo
za7?X(0B}HbX*ExkDmas*xzV)FxygC8AL?2Z1x-0QJqS@qn8sD7r{bm30@<%eL_gOw
z;~85O$Xw2AS}Qp)5ViRUe3|ir8;&&I<B7Y6^!kkNyYXF4EY(b8_5DsTYn_&?wkdEz
z0y$tADo<&}nGs5kg2-J=0KlEGPb(%<An(pXY{K`qIZCuwiT|2{8JD&5o_~`o6<;dD
zi@J#zCE4={8j%<uy|iutvHu1QKo5Tno-BAF2FwD%%O#UDDum=w!;!PNe-cOFNX4)5
zd>TifiGO|)w}OANj2n2D^K=o3j6P6uOL70#cbA{uzWXDlk1wr9GV1X(2W{RuTvjXV
zCmd<W?kH^?PXjkbF`XZtwu1B+%4@ZvHIwGpJ*8@8`MWAhq^B|Hj1lzj3R8bVuMpNb
zz4Gzk!3T3bY;WEGIww&kq9BYW6EP*q$K|EB-@TH(Fjtz*`HMTO?i+3E;5tdSah&xZ
z+t!x4K7)dpy5wiJhlJz~8qF|r8a&-SV7^I3C@_q=P`yt@_x_F-;PQR)fzP<zNN>8u
zH%V`94=q3)Dk)PHNrnFC(T1)Om6f{Usj;u1R->&XoCYVK2V3ZlgZuF?N}1+33<P7e
z<0yVF?Qoa{l#7q(3&jv=Ab)gpM8A7`p%3InNzSxy)ZER2U0C#9zKpnLY0I?>OER*x
z*9Z=L=zI8CN>A_^jYjt0F$psO$sL=38q5q|SG)qCN6{^>RFh5E&l5GZ$pEahnF&d+
z5c>64t}uJPkf~_!VUj#&N%nC-gUMj%=@B=!V>&}xtj2%@-mOm#rQUSJ3(ccmc+fza
znZ#uxF>N?QN5UrIEd!5RgHEf<eGg}P45aAs(Xs6u!XW9r1I*E6XJ^1movX@xYLuPz
z|7xBN4z@b}#x>W#;(nKYF+D<*rdshJ$X-z2OZ2X;)nn@KSVdVhaA?}@3;6gZxb4<W
z`9sa`0lR_azMX|=t_(FvG@%w2);9P}SG0u&K1(*oX3};~=<<!N*F$UTSxD{V&6mgL
ztw9Ntc2eOF@c!OJytNC4T^#)Ien7-`dI{6s#co~0f^E3J<0Ty)l3xq2u@Y8DXTK>v
zozoWSr{{+!h}zGpumG3H`=AvWpm^9kW;J$Jp^Xl*?8ckr`fqN%c|Z;VC0|cM4vSrk
zH_O8Yvh85nvJp^;``wo8=z0f`FWg?`>gO#y1hjX1{}rTlg9rwIKia8eyGexA3GnuR
z`Rg~XZoW;0pA)vI8=p5!+6sIn#C^FCvR>ffv39h6SCNi9v);%WD;WZ`of_MgwyRWy
z-yY%n*Y>X8<Sf+RyB|Sr2YG@1w~%U$o`(5EDkJ|3$u=eMZA&_wxEsy<Xxh2k^tP?4
RGx&ZHQs^8zuIpu!=pQhfp8Eg*
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/info/exclude b/tests/resources/small.git/info/exclude
new file mode 100644
index 0000000000000000000000000000000000000000..a5196d1be8fb59edf8062bef36d3a602e0812139
GIT binary patch
literal 240
zcmXYqK?=e!6h!x)VxWtv*mf_t5?px$aS_{}Hj?C*<cHXeXE&AZhT*-L3ZoI&*l1%Z
zqG?zr3TvQGZ__}H4(u*%p*rI=cU!%ya5ugfGATh66$IJHgu1Gs0-<N;$V+SsdE)?u
zIq;i$f#WE4f$_MWicZjMEob9LWKMR#iwZq54~QgST^6=i%u0lUkJu-_J**QBMq}ZG
Wth_)NDbl|`oQr&HAFQ5h`0jqAIZ*BZ
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/88/f9a6eaa2cf008b9bc92847178621f21fa99f3e b/tests/resources/small.git/objects/88/f9a6eaa2cf008b9bc92847178621f21fa99f3e
new file mode 100644
index 0000000000000000000000000000000000000000..7d80b8d78e0dc55669d831a6638f48ec9fed0982
GIT binary patch
literal 50
zcmV-20L}k+0V^p=O;s>9W-v4`Ff%bx$Vkn}$!Ay}rnY6F$m-Kg*KD_+;Lx#g4|^&N
I02NaX#p`nv=Kufz
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/af/5626b4a114abcb82d63db7c8082c3c4756e51b b/tests/resources/small.git/objects/af/5626b4a114abcb82d63db7c8082c3c4756e51b
new file mode 100644
index 0000000000000000000000000000000000000000..822bc151862ec3763cf2d3fa2372b93bbd3a4b65
GIT binary patch
literal 30
mcmb<m^geacKghr&@q@?NlP9kSYMj?U<r(;diNWtH+YSKNt_|)0
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/objects/c6/97d4f7a6eac8d7b131673c340bd3cc5bac14d4 b/tests/resources/small.git/objects/c6/97d4f7a6eac8d7b131673c340bd3cc5bac14d4
new file mode 100644
index 0000000000000000000000000000000000000000..5adfa88cc6b00bb19f2031b8ab61f6d07f9ccdb8
GIT binary patch
literal 130
zcmV-|0Db>>0i}&W3IZ_@1U=^!a~EV1casc=c+{&un1qQN*i9hD|0|m(2n|iwp*q%W
z%N;b$hu%cM`$TMo*~EnC1BFP&Pfj~;jZVKXQ96s_PhV<-XAROi+@-v8dBLUa`!;GB
k^i<X>XlEv8$>R)1G>9th&t3j;s7J{?^9n<zzF|~BaA?ar-~a#s
literal 0
HcmV?d00001
diff --git a/tests/resources/small.git/refs/heads/master b/tests/resources/small.git/refs/heads/master
new file mode 100644
index 0000000000000000000000000000000000000000..4eb36d22298f060fd324155ab854d9d6486fc498
GIT binary patch
literal 41
ucmV~$!4Uu;2m`Rc)5uXl$AMP&AHjriQg~T$i(A>|7U^`%mXoWC24Q^m!3%@{
literal 0
HcmV?d00001
+1
View File
@@ -0,0 +1 @@
../../.version
+1
View File
@@ -0,0 +1 @@
../../src/libstore-tests/data/content-address
+1
View File
@@ -0,0 +1 @@
../../src/libstore-tests/data/derivation
+1
View File
@@ -0,0 +1 @@
../../src/libstore-tests/data/derived-path
+1
View File
@@ -0,0 +1 @@
../../src/libutil-tests/data/hash
+107
View File
@@ -0,0 +1,107 @@
# Run with:
# meson test --suite json-schema
# Run with: (without shell / configure)
# nix build .#nix-json-schema-checks
project(
'nix-json-schema-checks',
version : files('.version'),
meson_version : '>= 1.1',
license : 'LGPL-2.1-or-later',
)
fs = import('fs')
# Note: The 'jsonschema' package provides the 'jv' command
jv = find_program('jv', required : true)
# The schema directory is a committed symlink to the actual schema location
schema_dir = meson.current_source_dir() / 'schema'
# Get all example files
schemas = [
{
'stem' : 'hash',
'schema' : schema_dir / 'hash-v1.yaml',
'files' : [
'sha256-base64.json',
'sha256-base16.json',
'sha256-nix32.json',
'blake3-base64.json',
],
},
{
'stem' : 'content-address',
'schema' : schema_dir / 'content-address-v1.yaml',
'files' : [
'text.json',
'nar.json',
],
},
{
'stem' : 'store-path',
'schema' : schema_dir / 'store-path-v1.yaml',
'files' : [
'simple.json',
],
},
{
'stem' : 'derivation',
'schema' : schema_dir / 'derivation-v3.yaml',
'files' : [
'dyn-dep-derivation.json',
'simple-derivation.json',
],
},
{
'stem' : 'derivation',
'schema' : schema_dir / 'derivation-v3.yaml#/$defs/output',
'files' : [
'output-caFixedFlat.json',
'output-caFixedNAR.json',
'output-caFixedText.json',
'output-caFloating.json',
'output-deferred.json',
'output-impure.json',
'output-inputAddressed.json',
],
},
{
'stem' : 'deriving-path',
'schema' : schema_dir / 'deriving-path-v1.yaml',
'files' : [
'single_opaque.json',
'single_built.json',
'single_built_built.json',
],
},
]
# Validate each example against the schema
foreach schema : schemas
stem = schema['stem']
schema_file = schema['schema']
if '#' not in schema_file
# Validate the schema itself against JSON Schema Draft 04
test(
stem + '-schema-valid',
jv,
args : [
'http://json-schema.org/draft-04/schema',
schema_file,
],
suite : 'json-schema',
)
endif
foreach example : schema['files']
test(
stem + '-example-' + fs.stem(example),
jv,
args : [
schema_file,
files(stem / example),
],
suite : 'json-schema',
)
endforeach
endforeach
+54
View File
@@ -0,0 +1,54 @@
# Run with: nix build .#nix-json-schema-checks
{
lib,
mkMesonDerivation,
meson,
ninja,
jsonschema,
# Configuration Options
version,
}:
mkMesonDerivation (finalAttrs: {
pname = "nix-json-schema-checks";
inherit version;
workDir = ./.;
fileset = lib.fileset.unions [
../../.version
../../doc/manual/source/protocols/json/schema
../../src/libutil-tests/data/hash
../../src/libstore-tests/data/content-address
../../src/libstore-tests/data/store-path
../../src/libstore-tests/data/derivation
../../src/libstore-tests/data/derived-path
./.
];
outputs = [ "out" ];
passthru.externalNativeBuildInputs = [
jsonschema
];
nativeBuildInputs = [
meson
ninja
]
++ finalAttrs.passthru.externalNativeBuildInputs;
doCheck = true;
mesonCheckFlags = [ "--print-errorlogs" ];
postInstall = ''
touch $out
'';
meta = {
platforms = lib.platforms.all;
};
})
+1
View File
@@ -0,0 +1 @@
../../doc/manual/source/protocols/json/schema
+1
View File
@@ -0,0 +1 @@
../../src/libstore-tests/data/store-path
+3 -2
View File
@@ -117,10 +117,11 @@ RealisedPath::Set BuiltPath::toRealisedPaths(Store & store) const
"the derivation '%s' has unrealised output '%s' (derived-path.cc/toRealisedPaths)",
store.printStorePath(p.drvPath->outPath()),
outputName);
auto thisRealisation = store.queryRealisation(DrvOutput{*drvOutput, outputName});
DrvOutput key{*drvOutput, outputName};
auto thisRealisation = store.queryRealisation(key);
assert(thisRealisation); // We’ve built it, so we must
// have the realisation
res.insert(*thisRealisation);
res.insert(Realisation{*thisRealisation, std::move(key)});
} else {
res.insert(outputPath);
}
+14
View File
@@ -350,6 +350,20 @@ struct MixEnvironment : virtual Args
void setEnviron();
};
struct MixNoCheckSigs : virtual Args
{
CheckSigsFlag checkSigs = CheckSigs;
MixNoCheckSigs()
{
addFlag({
.longName = "no-check-sigs",
.description = "Do not require that paths are signed by trusted keys.",
.handler = {&checkSigs, NoCheckSigs},
});
}
};
void completeFlakeInputAttrPath(
AddCompletions & completions,
ref<EvalState> evalState,
@@ -69,7 +69,7 @@ struct InstallableFlake : InstallableValue
*/
std::vector<ref<eval_cache::AttrCursor>> getCursors(EvalState & state) override;
std::shared_ptr<flake::LockedFlake> getLockedFlake() const;
ref<flake::LockedFlake> getLockedFlake() const;
FlakeRef nixpkgsFlakeRef() const;
};
@@ -87,6 +87,4 @@ static inline FlakeRef defaultNixpkgsFlakeRef()
return FlakeRef::fromAttrs(fetchSettings, {{"type", "indirect"}, {"id", "nixpkgs"}});
}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, std::shared_ptr<flake::LockedFlake> lockedFlake);
} // namespace nix
+4 -4
View File
@@ -185,16 +185,16 @@ std::vector<ref<eval_cache::AttrCursor>> InstallableFlake::getCursors(EvalState
return res;
}
std::shared_ptr<flake::LockedFlake> InstallableFlake::getLockedFlake() const
ref<flake::LockedFlake> InstallableFlake::getLockedFlake() const
{
if (!_lockedFlake) {
flake::LockFlags lockFlagsApplyConfig = lockFlags;
// FIXME why this side effect?
lockFlagsApplyConfig.applyNixConfig = true;
_lockedFlake =
std::make_shared<flake::LockedFlake>(lockFlake(flakeSettings, *state, flakeRef, lockFlagsApplyConfig));
_lockedFlake = make_ref<flake::LockedFlake>(lockFlake(flakeSettings, *state, flakeRef, lockFlagsApplyConfig));
}
return _lockedFlake;
// _lockedFlake is now non-null but still just a shared_ptr
return ref<flake::LockedFlake>(_lockedFlake);
}
FlakeRef InstallableFlake::nixpkgsFlakeRef() const
+1 -38
View File
@@ -342,8 +342,7 @@ void completeFlakeRefWithFragment(
parseFlakeRef(fetchSettings, expandTilde(flakeRefS), std::filesystem::current_path().string());
auto evalCache = openEvalCache(
*evalState,
std::make_shared<flake::LockedFlake>(lockFlake(flakeSettings, *evalState, flakeRef, lockFlags)));
*evalState, make_ref<flake::LockedFlake>(lockFlake(flakeSettings, *evalState, flakeRef, lockFlags)));
auto root = evalCache->getRoot();
@@ -443,42 +442,6 @@ static StorePath getDeriver(ref<Store> store, const Installable & i, const Store
return *derivers.begin();
}
ref<eval_cache::EvalCache> openEvalCache(EvalState & state, std::shared_ptr<flake::LockedFlake> lockedFlake)
{
auto fingerprint = evalSettings.useEvalCache && evalSettings.pureEval
? lockedFlake->getFingerprint(state.store, state.fetchSettings)
: std::nullopt;
auto rootLoader = [&state, lockedFlake]() {
/* For testing whether the evaluation cache is
complete. */
if (getEnv("NIX_ALLOW_EVAL").value_or("1") == "0")
throw Error("not everything is cached, but evaluation is not allowed");
auto vFlake = state.allocValue();
flake::callFlake(state, *lockedFlake, *vFlake);
state.forceAttrs(*vFlake, noPos, "while parsing cached flake data");
auto aOutputs = vFlake->attrs()->get(state.symbols.create("outputs"));
assert(aOutputs);
return aOutputs->value;
};
if (fingerprint) {
auto search = state.evalCaches.find(fingerprint.value());
if (search == state.evalCaches.end()) {
search =
state.evalCaches
.emplace(fingerprint.value(), make_ref<nix::eval_cache::EvalCache>(fingerprint, state, rootLoader))
.first;
}
return search->second;
} else {
return make_ref<nix::eval_cache::EvalCache>(std::nullopt, state, rootLoader);
}
}
Installables SourceExprCommand::parseInstallables(ref<Store> store, std::vector<std::string> ss)
{
Installables result;
-1
View File
@@ -67,7 +67,6 @@ config_priv_h = configure_file(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'built-path.cc',
-1
View File
@@ -28,7 +28,6 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_expr.cc',
-1
View File
@@ -31,7 +31,6 @@ rapidcheck = dependency('rapidcheck')
deps_public += rapidcheck
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'tests/value/context.cc',
+1 -2
View File
@@ -45,7 +45,6 @@ config_priv_h = configure_file(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'derived-path.cc',
@@ -83,7 +82,7 @@ this_exe = executable(
test(
meson.project_name(),
this_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
},
protocol : 'gtest',
-1
View File
@@ -62,7 +62,6 @@ mkMesonExecutable (finalAttrs: {
mkdir -p "$HOME"
''
+ ''
export ASAN_OPTIONS=abort_on_error=1:print_summary=1:detect_leaks=0
export _NIX_TEST_UNIT_DATA=${resolvePath ./data}
${stdenv.hostPlatform.emulator buildPackages} ${lib.getExe finalAttrs.finalPackage}
touch $out
+4 -3
View File
@@ -10,7 +10,7 @@ using namespace testing;
struct ValuePrintingTests : LibExprTest
{
template<class... A>
void test(Value v, std::string_view expected, A... args)
void test(Value & v, std::string_view expected, A... args)
{
std::stringstream out;
v.print(state, out, args...);
@@ -625,10 +625,11 @@ TEST_F(ValuePrintingTests, ansiColorsAttrsElided)
vThree.mkInt(3);
builder.insert(state.symbols.create("three"), &vThree);
vAttrs.mkAttrs(builder.finish());
Value vAttrs2;
vAttrs2.mkAttrs(builder.finish());
test(
vAttrs,
vAttrs2,
"{ one = " ANSI_CYAN "1" ANSI_NORMAL "; " ANSI_FAINT "«2 attributes elided»" ANSI_NORMAL " }",
PrintOptions{.ansiColors = true, .maxAttrs = 1});
}
+11 -7
View File
@@ -2050,10 +2050,10 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
};
// List of returned strings. References to these Values must NOT be persisted.
SmallTemporaryValueVector<conservativeStackReservation> values(es->size());
SmallTemporaryValueVector<conservativeStackReservation> values(es.size());
Value * vTmpP = values.data();
for (auto & [i_pos, i] : *es) {
for (auto & [i_pos, i] : es) {
Value & vTmp = *vTmpP++;
i->eval(state, env, vTmp);
@@ -2097,7 +2097,7 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
.debugThrow();
} else {
if (s.empty())
s.reserve(es->size());
s.reserve(es.size());
/* skip canonization of first path, which would only be not
canonized in the first place if it's coming from a ./${foo} type
path */
@@ -3067,7 +3067,7 @@ Expr * EvalState::parseExprFromFile(const SourcePath & path)
return parseExprFromFile(path, staticBaseEnv);
}
Expr * EvalState::parseExprFromFile(const SourcePath & path, std::shared_ptr<StaticEnv> & staticEnv)
Expr * EvalState::parseExprFromFile(const SourcePath & path, const std::shared_ptr<StaticEnv> & staticEnv)
{
auto buffer = path.resolveSymlinks().readFile();
// readFile hopefully have left some extra space for terminators
@@ -3075,8 +3075,8 @@ Expr * EvalState::parseExprFromFile(const SourcePath & path, std::shared_ptr<Sta
return parse(buffer.data(), buffer.size(), Pos::Origin(path), path.parent(), staticEnv);
}
Expr *
EvalState::parseExprFromString(std::string s_, const SourcePath & basePath, std::shared_ptr<StaticEnv> & staticEnv)
Expr * EvalState::parseExprFromString(
std::string s_, const SourcePath & basePath, const std::shared_ptr<StaticEnv> & staticEnv)
{
// NOTE this method (and parseStdin) must take care to *fully copy* their input
// into their respective Pos::Origin until the parser stops overwriting its input
@@ -3210,7 +3210,11 @@ std::optional<SourcePath> EvalState::resolveLookupPathPath(const LookupPath::Pat
}
Expr * EvalState::parse(
char * text, size_t length, Pos::Origin origin, const SourcePath & basePath, std::shared_ptr<StaticEnv> & staticEnv)
char * text,
size_t length,
Pos::Origin origin,
const SourcePath & basePath,
const std::shared_ptr<StaticEnv> & staticEnv)
{
DocCommentMap tmpDocComments; // Only used when not origin is not a SourcePath
DocCommentMap * docComments = &tmpDocComments;
+41 -4
View File
@@ -5,6 +5,7 @@
#include "nix/expr/symbol-table.hh"
#include <boost/container/static_vector.hpp>
#include <boost/iterator/function_output_iterator.hpp>
#include <algorithm>
#include <functional>
@@ -463,12 +464,48 @@ private:
return bindings->baseLayer;
}
/**
* If the bindings gets "layered" on top of another we need to recalculate
* the number of unique attributes in the chain.
*
* This is done by either iterating over the base "layer" and the newly added
* attributes and counting duplicates. If the base "layer" is big this approach
* is inefficient and we fall back to doing per-element binary search in the base
* "layer".
*/
void finishSizeIfNecessary()
{
if (hasBaseLayer())
/* NOTE: Do not use std::ranges::distance, since Bindings is a sized
range, but we are calculating this size here. */
bindings->numAttrsInChain = std::distance(bindings->begin(), bindings->end());
if (!hasBaseLayer())
return;
auto & base = *bindings->baseLayer;
auto attrs = std::span(bindings->attrs, bindings->numAttrs);
Bindings::size_type duplicates = 0;
/* If the base bindings is smaller than the newly added attributes
iterate using std::set_intersection to run in O(|base| + |attrs|) =
O(|attrs|). Otherwise use an O(|attrs| * log(|base|)) per-attr binary
search to check for duplicates. Note that if we are in this code path then
|attrs| <= bindingsUpdateLayerRhsSizeThreshold, which 16 by default. We are
optimizing for the case when a small attribute set gets "layered" on top of
a much larger one. When attrsets are already small it's fine to do a linear
scan, but we should avoid expensive iterations over large "base" attrsets. */
if (attrs.size() > base.size()) {
std::set_intersection(
base.begin(),
base.end(),
attrs.begin(),
attrs.end(),
boost::make_function_output_iterator([&]([[maybe_unused]] auto && _) { ++duplicates; }));
} else {
for (const auto & attr : attrs) {
if (base.get(attr.name))
++duplicates;
}
}
bindings->numAttrsInChain = base.numAttrsInChain + attrs.size() - duplicates;
}
public:
+21 -12
View File
@@ -191,7 +191,7 @@ std::ostream & operator<<(std::ostream & os, const ValueType t);
struct RegexCache;
std::shared_ptr<RegexCache> makeRegexCache();
ref<RegexCache> makeRegexCache();
struct DebugTrace
{
@@ -372,6 +372,7 @@ public:
const fetchers::Settings & fetchSettings;
const EvalSettings & settings;
SymbolTable symbols;
PosTable positions;
@@ -418,7 +419,7 @@ public:
RootValue vImportedDrvToDerivation = nullptr;
ref<fetchers::InputCache> inputCache;
const ref<fetchers::InputCache> inputCache;
/**
* Debugger
@@ -471,18 +472,18 @@ private:
/* Cache for calls to addToStore(); maps source paths to the store
paths. */
ref<boost::concurrent_flat_map<SourcePath, StorePath>> srcToStore;
const ref<boost::concurrent_flat_map<SourcePath, StorePath>> srcToStore;
/**
* A cache that maps paths to "resolved" paths for importing Nix
* expressions, i.e. `/foo` to `/foo/default.nix`.
*/
ref<boost::concurrent_flat_map<SourcePath, SourcePath>> importResolutionCache;
const ref<boost::concurrent_flat_map<SourcePath, SourcePath>> importResolutionCache;
/**
* A cache from resolved paths to values.
*/
ref<boost::concurrent_flat_map<
const ref<boost::concurrent_flat_map<
SourcePath,
Value *,
std::hash<SourcePath>,
@@ -504,12 +505,19 @@ private:
/**
* Cache used by prim_match().
*/
std::shared_ptr<RegexCache> regexCache;
const ref<RegexCache> regexCache;
public:
/**
* @param lookupPath Only used during construction.
* @param store The store to use for instantiation
* @param fetchSettings Must outlive the lifetime of this EvalState!
* @param settings Must outlive the lifetime of this EvalState!
* @param buildStore The store to use for builds ("import from derivation", C API `nix_string_realise`)
*/
EvalState(
const LookupPath & _lookupPath,
const LookupPath & lookupPath,
ref<Store> store,
const fetchers::Settings & fetchSettings,
const EvalSettings & settings,
@@ -585,12 +593,13 @@ public:
* Parse a Nix expression from the specified file.
*/
Expr * parseExprFromFile(const SourcePath & path);
Expr * parseExprFromFile(const SourcePath & path, std::shared_ptr<StaticEnv> & staticEnv);
Expr * parseExprFromFile(const SourcePath & path, const std::shared_ptr<StaticEnv> & staticEnv);
/**
* Parse a Nix expression from the specified string.
*/
Expr * parseExprFromString(std::string s, const SourcePath & basePath, std::shared_ptr<StaticEnv> & staticEnv);
Expr *
parseExprFromString(std::string s, const SourcePath & basePath, const std::shared_ptr<StaticEnv> & staticEnv);
Expr * parseExprFromString(std::string s, const SourcePath & basePath);
Expr * parseStdin();
@@ -759,7 +768,7 @@ public:
#if NIX_USE_BOEHMGC
/** A GC root for the baseEnv reference. */
std::shared_ptr<Env *> baseEnvP;
const std::shared_ptr<Env *> baseEnvP;
#endif
public:
@@ -773,7 +782,7 @@ public:
/**
* The same, but used during parsing to resolve variables.
*/
std::shared_ptr<StaticEnv> staticBaseEnv; // !!! should be private
const std::shared_ptr<StaticEnv> staticBaseEnv; // !!! should be private
/**
* Internal primops not exposed to the user.
@@ -855,7 +864,7 @@ private:
size_t length,
Pos::Origin origin,
const SourcePath & basePath,
std::shared_ptr<StaticEnv> & staticEnv);
const std::shared_ptr<StaticEnv> & staticEnv);
/**
* Current Nix call stack depth, used with `max-call-depth` setting to throw stack overflow hopefully before we run
+11 -5
View File
@@ -442,8 +442,14 @@ struct ExprAttrs : Expr
struct ExprList : Expr
{
std::vector<Expr *> elems;
ExprList() {};
std::span<Expr *> elems;
ExprList(std::pmr::polymorphic_allocator<char> & alloc, std::vector<Expr *> exprs)
: elems({alloc.allocate_object<Expr *>(exprs.size()), exprs.size()})
{
std::ranges::copy(exprs, elems.begin());
};
COMMON_METHODS
Value * maybeThunk(EvalState & state, Env & env) override;
@@ -695,11 +701,11 @@ struct ExprConcatStrings : Expr
{
PosIdx pos;
bool forceString;
std::vector<std::pair<PosIdx, Expr *>> * es;
ExprConcatStrings(const PosIdx & pos, bool forceString, std::vector<std::pair<PosIdx, Expr *>> * es)
std::vector<std::pair<PosIdx, Expr *>> es;
ExprConcatStrings(const PosIdx & pos, bool forceString, std::vector<std::pair<PosIdx, Expr *>> && es)
: pos(pos)
, forceString(forceString)
, es(es) {};
, es(std::move(es)) {};
PosIdx getPos() const override
{
+7 -9
View File
@@ -282,7 +282,7 @@ ParserState::stripIndentation(const PosIdx pos, std::vector<std::pair<PosIdx, st
}
/* Strip spaces from each line. */
auto * es2 = new std::vector<std::pair<PosIdx, Expr *>>;
std::vector<std::pair<PosIdx, Expr *>> es2{};
atStartOfLine = true;
size_t curDropped = 0;
size_t n = es.size();
@@ -290,7 +290,7 @@ ParserState::stripIndentation(const PosIdx pos, std::vector<std::pair<PosIdx, st
const auto trimExpr = [&](Expr * e) {
atStartOfLine = false;
curDropped = 0;
es2->emplace_back(i->first, e);
es2.emplace_back(i->first, e);
};
const auto trimString = [&](const StringToken & t) {
std::string s2;
@@ -324,7 +324,7 @@ ParserState::stripIndentation(const PosIdx pos, std::vector<std::pair<PosIdx, st
// Ignore empty strings for a minor optimisation and AST simplification
if (s2 != "") {
es2->emplace_back(i->first, new ExprString(alloc, s2));
es2.emplace_back(i->first, new ExprString(alloc, s2));
}
};
for (; i != es.end(); ++i, --n) {
@@ -333,19 +333,17 @@ ParserState::stripIndentation(const PosIdx pos, std::vector<std::pair<PosIdx, st
// If there is nothing at all, return the empty string directly.
// This also ensures that equivalent empty strings result in the same ast, which is helpful when testing formatters.
if (es2->size() == 0) {
if (es2.size() == 0) {
auto * const result = new ExprString("");
delete es2;
return result;
}
/* If this is a single string, then don't do a concatenation. */
if (es2->size() == 1 && dynamic_cast<ExprString *>((*es2)[0].second)) {
auto * const result = (*es2)[0].second;
delete es2;
if (es2.size() == 1 && dynamic_cast<ExprString *>((es2)[0].second)) {
auto * const result = (es2)[0].second;
return result;
}
return new ExprConcatStrings(pos, true, es2);
return new ExprConcatStrings(pos, true, std::move(es2));
}
inline PosIdx LexerState::at(const ParserLocation & loc)
+17 -17
View File
@@ -142,11 +142,11 @@ or { return OR_KW; }
return PIPE_INTO;
}
{ID} { yylval->id = {yytext, (size_t) yyleng}; return ID; }
{ID} { yylval->emplace<StringToken>(yytext, (size_t) yyleng); return ID; }
{INT} { errno = 0;
std::optional<int64_t> numMay = string2Int<int64_t>(yytext);
if (numMay.has_value()) {
yylval->n = NixInt{*numMay};
yylval->emplace<NixInt>(*numMay);
} else {
throw ParseError(ErrorInfo{
.msg = HintFmt("invalid integer '%1%'", yytext),
@@ -156,7 +156,7 @@ or { return OR_KW; }
return INT_LIT;
}
{FLOAT} { errno = 0;
yylval->nf = strtod(yytext, 0);
yylval->emplace<NixFloat>(strtod(yytext, 0));
if (errno != 0)
throw ParseError(ErrorInfo{
.msg = HintFmt("invalid float '%1%'", yytext),
@@ -183,7 +183,7 @@ or { return OR_KW; }
/* It is impossible to match strings ending with '$' with one
regex because trailing contexts are only valid at the end
of a rule. (A sane but undocumented limitation.) */
yylval->str = unescapeStr(yytext, yyleng, [&]() { return state->positions[CUR_POS]; });
yylval->emplace<StringToken>(unescapeStr(yytext, yyleng, [&]() { return state->positions[CUR_POS]; }));
return STR;
}
<STRING>\$\{ { PUSH_STATE(DEFAULT); return DOLLAR_CURLY; }
@@ -198,27 +198,27 @@ or { return OR_KW; }
\'\'(\ *\n)? { PUSH_STATE(IND_STRING); return IND_STRING_OPEN; }
<IND_STRING>([^\$\']|\$[^\{\']|\'[^\'\$])+ {
yylval->str = {yytext, (size_t) yyleng, true};
forceNoNullByte(yylval->str, [&]() { return state->positions[CUR_POS]; });
yylval->emplace<StringToken>(yytext, (size_t) yyleng, true);
forceNoNullByte(yylval->as<StringToken>(), [&]() { return state->positions[CUR_POS]; });
return IND_STR;
}
<IND_STRING>\'\'\$ |
<IND_STRING>\$ {
yylval->str = {"$", 1};
yylval->emplace<StringToken>("$", 1);
return IND_STR;
}
<IND_STRING>\'\'\' {
yylval->str = {"''", 2};
yylval->emplace<StringToken>("''", 2);
return IND_STR;
}
<IND_STRING>\'\'\\{ANY} {
yylval->str = unescapeStr(yytext + 2, yyleng - 2, [&]() { return state->positions[CUR_POS]; });
yylval->emplace<StringToken>(unescapeStr(yytext + 2, yyleng - 2, [&]() { return state->positions[CUR_POS]; }));
return IND_STR;
}
<IND_STRING>\$\{ { PUSH_STATE(DEFAULT); return DOLLAR_CURLY; }
<IND_STRING>\'\' { POP_STATE(); return IND_STRING_CLOSE; }
<IND_STRING>\' {
yylval->str = {"'", 1};
yylval->emplace<StringToken>("'", 1);
return IND_STR;
}
@@ -232,14 +232,14 @@ or { return OR_KW; }
<PATH_START>{PATH_SEG} {
POP_STATE();
PUSH_STATE(INPATH_SLASH);
yylval->path = {yytext, (size_t) yyleng};
yylval->emplace<StringToken>(yytext, (size_t) yyleng);
return PATH;
}
<PATH_START>{HPATH_START} {
POP_STATE();
PUSH_STATE(INPATH_SLASH);
yylval->path = {yytext, (size_t) yyleng};
yylval->emplace<StringToken>(yytext, (size_t) yyleng);
return HPATH;
}
@@ -248,7 +248,7 @@ or { return OR_KW; }
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
yylval->path = {yytext, (size_t) yyleng};
yylval->emplace<StringToken>(yytext, (size_t) yyleng);
return PATH;
}
{HPATH} {
@@ -256,7 +256,7 @@ or { return OR_KW; }
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
yylval->path = {yytext, (size_t) yyleng};
yylval->emplace<StringToken>(yytext, (size_t) yyleng);
return HPATH;
}
@@ -272,7 +272,7 @@ or { return OR_KW; }
PUSH_STATE(INPATH_SLASH);
else
PUSH_STATE(INPATH);
yylval->str = {yytext, (size_t) yyleng};
yylval->emplace<StringToken>(yytext, (size_t) yyleng);
return STR;
}
<INPATH>{ANY} |
@@ -294,8 +294,8 @@ or { return OR_KW; }
});
}
{SPATH} { yylval->path = {yytext, (size_t) yyleng}; return SPATH; }
{URI} { yylval->uri = {yytext, (size_t) yyleng}; return URI; }
{SPATH} { yylval->emplace<StringToken>(yytext, (size_t) yyleng); return SPATH; }
{URI} { yylval->emplace<StringToken>(yytext, (size_t) yyleng); return URI; }
%{
// Doc comment rule
+47 -3
View File
@@ -97,7 +97,6 @@ config_priv_h = configure_file(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
parser_tab = custom_target(
input : 'parser.y',
@@ -184,17 +183,62 @@ subdir('primops')
subdir('nix-meson-build-support/export-all-symbols')
subdir('nix-meson-build-support/windows-version')
# Turns out that Bison/Flex are particularly sensitive to compilers
# failing to inline functions. For that reason we crank up the inlining
# threshold manually for optimized builds. Yes, this can be considered 'ricing'
# the compiler, but it does pay off.
#
# NOTE: missed inlining can be spotted (for Clang) using -Rpass-missed=inline
# and -fdump-ipa-inline-missed (for GCC).
parser_library_cpp_args = []
if not get_option('debug')
if cxx.get_id() == 'clang'
# The default as of LLVM 21 is 225:
# llc --help-hidden | grep inline-threshold
parser_library_cpp_args += [
'-mllvm',
'-inline-threshold=5000',
]
elif cxx.get_id() == 'gcc'
parser_library_cpp_args += [
'--param=max-inline-insns-single=1000',
'--param=max-inline-insns-auto=1000',
'--param=inline-unit-growth=400',
]
endif
endif
# Working around https://github.com/mesonbuild/meson/issues/1367.
parser_library = static_library(
'nixexpr-parser',
parser_tab,
lexer_tab,
cpp_args : parser_library_cpp_args,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
# 1. Stdlib and regular assertions regress parser performance significantly, so build without
# them for this one library when building in a release configuration.
# 2. Disable LTO for GCC because then inlining flags won't apply, since LTO in GCC is done
# by plonking down GIMPLE in the archive.
override_options : [
'b_ndebug=@0@'.format(not get_option('debug')),
'b_lto=@0@'.format(get_option('b_lto') and cxx.get_id() != 'gcc'),
],
)
this_library = library(
'nixexpr',
sources,
config_priv_h,
parser_tab,
lexer_tab,
parser_tab[1],
lexer_tab[1],
generated_headers,
soversion : nix_soversion,
dependencies : deps_public + deps_private + deps_other,
include_directories : include_dirs,
link_args : linker_export_flags,
link_whole : [ parser_library ],
prelink : true, # For C++ static initializers
install : true,
cpp_pch : do_pch ? [ 'pch/precompiled-headers.hh' ] : [],
+2 -2
View File
@@ -246,7 +246,7 @@ void ExprConcatStrings::show(const SymbolTable & symbols, std::ostream & str) co
{
bool first = true;
str << "(";
for (auto & i : *es) {
for (auto & i : es) {
if (first)
first = false;
else
@@ -564,7 +564,7 @@ void ExprConcatStrings::bindVars(EvalState & es, const std::shared_ptr<const Sta
if (es.debugRepl)
es.exprEnvs.insert(std::make_pair(this, env));
for (auto & i : *this->es)
for (auto & i : this->es)
i.second->bindVars(es, env);
}
+74 -98
View File
@@ -14,6 +14,10 @@
%code requires {
// bison adds a bunch of switch statements with default:
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wswitch-enum"
#ifndef BISON_HEADER
#define BISON_HEADER
@@ -120,46 +124,28 @@ static Expr * makeCall(PosIdx pos, Expr * fn, Expr * arg) {
%}
%union {
// !!! We're probably leaking stuff here.
nix::Expr * e;
nix::ExprList * list;
nix::ExprAttrs * attrs;
nix::Formals * formals;
nix::Formal * formal;
nix::NixInt n;
nix::NixFloat nf;
nix::StringToken id; // !!! -> Symbol
nix::StringToken path;
nix::StringToken uri;
nix::StringToken str;
std::vector<nix::AttrName> * attrNames;
std::vector<std::pair<nix::AttrName, nix::PosIdx>> * inheritAttrs;
std::vector<std::pair<nix::PosIdx, nix::Expr *>> * string_parts;
std::variant<nix::Expr *, std::string_view> * to_be_string;
std::vector<std::pair<nix::PosIdx, std::variant<nix::Expr *, nix::StringToken>>> * ind_string_parts;
}
%define api.value.type variant
%type <e> start expr expr_function expr_if expr_op
%type <e> expr_select expr_simple expr_app
%type <e> expr_pipe_from expr_pipe_into
%type <list> expr_list
%type <attrs> binds binds1
%type <formals> formals formal_set
%type <formal> formal
%type <attrNames> attrpath
%type <inheritAttrs> attrs
%type <string_parts> string_parts_interpolated
%type <ind_string_parts> ind_string_parts
%type <e> path_start
%type <to_be_string> string_parts string_attr
%type <id> attr
%token <id> ID
%token <str> STR IND_STR
%token <n> INT_LIT
%token <nf> FLOAT_LIT
%token <path> PATH HPATH SPATH PATH_END
%token <uri> URI
%type <nix::Expr *> start expr expr_function expr_if expr_op
%type <nix::Expr *> expr_select expr_simple expr_app
%type <nix::Expr *> expr_pipe_from expr_pipe_into
%type <std::vector<Expr *>> list
%type <nix::ExprAttrs *> binds binds1
%type <nix::Formals *> formals formal_set
%type <nix::Formal> formal
%type <std::vector<nix::AttrName>> attrpath
%type <std::vector<std::pair<nix::AttrName, nix::PosIdx>>> attrs
%type <std::vector<std::pair<nix::PosIdx, nix::Expr *>>> string_parts_interpolated
%type <std::vector<std::pair<nix::PosIdx, std::variant<nix::Expr *, nix::StringToken>>>> ind_string_parts
%type <nix::Expr *> path_start
%type <std::variant<nix::Expr *, std::string_view>> string_parts string_attr
%type <nix::StringToken> attr
%token <nix::StringToken> ID
%token <nix::StringToken> STR IND_STR
%token <nix::NixInt> INT_LIT
%token <nix::NixFloat> FLOAT_LIT
%token <nix::StringToken> PATH HPATH SPATH PATH_END
%token <nix::StringToken> URI
%token IF THEN ELSE ASSERT WITH LET IN_KW REC INHERIT EQ NEQ AND OR IMPL OR_KW
%token PIPE_FROM PIPE_INTO /* <| and |> */
%token DOLLAR_CURLY /* == ${ */
@@ -261,9 +247,9 @@ expr_op
| expr_op OR expr_op { $$ = new ExprOpOr(state->at(@2), $1, $3); }
| expr_op IMPL expr_op { $$ = new ExprOpImpl(state->at(@2), $1, $3); }
| expr_op UPDATE expr_op { $$ = new ExprOpUpdate(state->at(@2), $1, $3); }
| expr_op '?' attrpath { $$ = new ExprOpHasAttr(state->alloc, $1, std::move(*$3)); delete $3; }
| expr_op '?' attrpath { $$ = new ExprOpHasAttr(state->alloc, $1, std::move($3)); }
| expr_op '+' expr_op
{ $$ = new ExprConcatStrings(state->at(@2), false, new std::vector<std::pair<PosIdx, Expr *> >({{state->at(@1), $1}, {state->at(@3), $3}})); }
{ $$ = new ExprConcatStrings(state->at(@2), false, {{state->at(@1), $1}, {state->at(@3), $3}}); }
| expr_op '-' expr_op { $$ = new ExprCall(state->at(@2), new ExprVar(state->s.sub), {$1, $3}); }
| expr_op '*' expr_op { $$ = new ExprCall(state->at(@2), new ExprVar(state->s.mul), {$1, $3}); }
| expr_op '/' expr_op { $$ = new ExprCall(state->at(@2), new ExprVar(state->s.div), {$1, $3}); }
@@ -282,9 +268,9 @@ expr_app
expr_select
: expr_simple '.' attrpath
{ $$ = new ExprSelect(state->alloc, CUR_POS, $1, std::move(*$3), nullptr); delete $3; }
{ $$ = new ExprSelect(state->alloc, CUR_POS, $1, std::move($3), nullptr); }
| expr_simple '.' attrpath OR_KW expr_select
{ $$ = new ExprSelect(state->alloc, CUR_POS, $1, std::move(*$3), $5); delete $3; $5->warnIfCursedOr(state->symbols, state->positions); }
{ $$ = new ExprSelect(state->alloc, CUR_POS, $1, std::move($3), $5); $5->warnIfCursedOr(state->symbols, state->positions); }
| /* Backwards compatibility: because Nixpkgs has a function named ‘or’,
allow stuff like ‘map or [...]’. This production is problematic (see
https://github.com/NixOS/nix/issues/11118) and will be refactored in the
@@ -311,17 +297,15 @@ expr_simple
std::visit(overloaded{
[&](std::string_view str) { $$ = new ExprString(state->alloc, str); },
[&](Expr * expr) { $$ = expr; }},
*$2);
delete $2;
$2);
}
| IND_STRING_OPEN ind_string_parts IND_STRING_CLOSE {
$$ = state->stripIndentation(CUR_POS, std::move(*$2));
delete $2;
$$ = state->stripIndentation(CUR_POS, std::move($2));
}
| path_start PATH_END
| path_start string_parts_interpolated PATH_END {
$2->insert($2->begin(), {state->at(@1), $1});
$$ = new ExprConcatStrings(CUR_POS, false, $2);
$2.insert($2.begin(), {state->at(@1), $1});
$$ = new ExprConcatStrings(CUR_POS, false, std::move($2));
}
| SPATH {
std::string_view path($1.p + 1, $1.l - 2);
@@ -350,24 +334,23 @@ expr_simple
{ $2->pos = CUR_POS; $$ = $2; }
| '{' '}'
{ $$ = new ExprAttrs(CUR_POS); }
| '[' expr_list ']' { $$ = $2; }
| '[' list ']' { $$ = new ExprList(state->alloc, std::move($2)); }
;
string_parts
: STR { $$ = new std::variant<Expr *, std::string_view>($1); }
| string_parts_interpolated { $$ = new std::variant<Expr *, std::string_view>(new ExprConcatStrings(CUR_POS, true, $1)); }
| { $$ = new std::variant<Expr *, std::string_view>(std::string_view()); }
: STR { $$ = $1; }
| string_parts_interpolated { $$ = new ExprConcatStrings(CUR_POS, true, std::move($1)); }
| { $$ = std::string_view(); }
;
string_parts_interpolated
: string_parts_interpolated STR
{ $$ = $1; $1->emplace_back(state->at(@2), new ExprString(state->alloc, $2)); }
| string_parts_interpolated DOLLAR_CURLY expr '}' { $$ = $1; $1->emplace_back(state->at(@2), $3); }
| DOLLAR_CURLY expr '}' { $$ = new std::vector<std::pair<PosIdx, Expr *>>; $$->emplace_back(state->at(@1), $2); }
{ $$ = std::move($1); $$.emplace_back(state->at(@2), new ExprString(state->alloc, $2)); }
| string_parts_interpolated DOLLAR_CURLY expr '}' { $$ = std::move($1); $$.emplace_back(state->at(@2), $3); }
| DOLLAR_CURLY expr '}' { $$.emplace_back(state->at(@1), $2); }
| STR DOLLAR_CURLY expr '}' {
$$ = new std::vector<std::pair<PosIdx, Expr *>>;
$$->emplace_back(state->at(@1), new ExprString(state->alloc, $1));
$$->emplace_back(state->at(@2), $3);
$$.emplace_back(state->at(@1), new ExprString(state->alloc, $1));
$$.emplace_back(state->at(@2), $3);
}
;
@@ -408,9 +391,9 @@ path_start
;
ind_string_parts
: ind_string_parts IND_STR { $$ = $1; $1->emplace_back(state->at(@2), $2); }
| ind_string_parts DOLLAR_CURLY expr '}' { $$ = $1; $1->emplace_back(state->at(@2), $3); }
| { $$ = new std::vector<std::pair<PosIdx, std::variant<Expr *, StringToken>>>; }
: ind_string_parts IND_STR { $$ = std::move($1); $$.emplace_back(state->at(@2), $2); }
| ind_string_parts DOLLAR_CURLY expr '}' { $$ = std::move($1); $$.emplace_back(state->at(@2), $3); }
| { }
;
binds
@@ -421,19 +404,17 @@ binds
binds1
: binds1[accum] attrpath '=' expr ';'
{ $$ = $accum;
state->addAttr($$, std::move(*$attrpath), @attrpath, $expr, @expr);
delete $attrpath;
state->addAttr($$, std::move($attrpath), @attrpath, $expr, @expr);
}
| binds[accum] INHERIT attrs ';'
{ $$ = $accum;
for (auto & [i, iPos] : *$attrs) {
for (auto & [i, iPos] : $attrs) {
if ($accum->attrs.find(i.symbol) != $accum->attrs.end())
state->dupAttr(i.symbol, iPos, $accum->attrs[i.symbol].pos);
$accum->attrs.emplace(
i.symbol,
ExprAttrs::AttrDef(new ExprVar(iPos, i.symbol), iPos, ExprAttrs::AttrDef::Kind::Inherited));
}
delete $attrs;
}
| binds[accum] INHERIT '(' expr ')' attrs ';'
{ $$ = $accum;
@@ -441,7 +422,7 @@ binds1
$accum->inheritFromExprs = std::make_unique<std::vector<Expr *>>();
$accum->inheritFromExprs->push_back($expr);
auto from = new nix::ExprInheritFrom(state->at(@expr), $accum->inheritFromExprs->size() - 1);
for (auto & [i, iPos] : *$attrs) {
for (auto & [i, iPos] : $attrs) {
if ($accum->attrs.find(i.symbol) != $accum->attrs.end())
state->dupAttr(i.symbol, iPos, $accum->attrs[i.symbol].pos);
$accum->attrs.emplace(
@@ -451,51 +432,45 @@ binds1
iPos,
ExprAttrs::AttrDef::Kind::InheritedFrom));
}
delete $attrs;
}
| attrpath '=' expr ';'
{ $$ = new ExprAttrs;
state->addAttr($$, std::move(*$attrpath), @attrpath, $expr, @expr);
delete $attrpath;
state->addAttr($$, std::move($attrpath), @attrpath, $expr, @expr);
}
;
attrs
: attrs attr { $$ = $1; $1->emplace_back(AttrName(state->symbols.create($2)), state->at(@2)); }
: attrs attr { $$ = std::move($1); $$.emplace_back(state->symbols.create($2), state->at(@2)); }
| attrs string_attr
{ $$ = $1;
{ $$ = std::move($1);
std::visit(overloaded {
[&](std::string_view str) { $$->emplace_back(AttrName(state->symbols.create(str)), state->at(@2)); },
[&](std::string_view str) { $$.emplace_back(state->symbols.create(str), state->at(@2)); },
[&](Expr * expr) {
throw ParseError({
.msg = HintFmt("dynamic attributes not allowed in inherit"),
.pos = state->positions[state->at(@2)]
});
}
}, *$2);
delete $2;
}, $2);
}
| { $$ = new std::vector<std::pair<AttrName, PosIdx>>; }
| { }
;
attrpath
: attrpath '.' attr { $$ = $1; $1->push_back(AttrName(state->symbols.create($3))); }
: attrpath '.' attr { $$ = std::move($1); $$.emplace_back(state->symbols.create($3)); }
| attrpath '.' string_attr
{ $$ = $1;
{ $$ = std::move($1);
std::visit(overloaded {
[&](std::string_view str) { $$->push_back(AttrName(state->symbols.create(str))); },
[&](Expr * expr) { $$->push_back(AttrName(expr)); }
}, *$3);
delete $3;
[&](std::string_view str) { $$.emplace_back(state->symbols.create(str)); },
[&](Expr * expr) { $$.emplace_back(expr); }
}, std::move($3));
}
| attr { $$ = new std::vector<AttrName>; $$->push_back(AttrName(state->symbols.create($1))); }
| attr { $$.emplace_back(state->symbols.create($1)); }
| string_attr
{ $$ = new std::vector<AttrName>;
std::visit(overloaded {
[&](std::string_view str) { $$->push_back(AttrName(state->symbols.create(str))); },
[&](Expr * expr) { $$->push_back(AttrName(expr)); }
}, *$1);
delete $1;
{ std::visit(overloaded {
[&](std::string_view str) { $$.emplace_back(state->symbols.create(str)); },
[&](Expr * expr) { $$.emplace_back(expr); }
}, std::move($1));
}
;
@@ -505,13 +480,13 @@ attr
;
string_attr
: '"' string_parts '"' { $$ = $2; }
| DOLLAR_CURLY expr '}' { $$ = new std::variant<Expr *, std::string_view>($2); }
: '"' string_parts '"' { $$ = std::move($2); }
| DOLLAR_CURLY expr '}' { $$ = $2; }
;
expr_list
: expr_list expr_select { $$ = $1; $1->elems.push_back($2); /* !!! dangerous */; $2->warnIfCursedOr(state->symbols, state->positions); }
| { $$ = new ExprList; }
list
: list expr_select { $$ = std::move($1); $$.push_back($2); /* !!! dangerous */; $2->warnIfCursedOr(state->symbols, state->positions); }
| { }
;
formal_set
@@ -524,14 +499,14 @@ formal_set
formals
: formals[accum] ',' formal
{ $$ = $accum; $$->formals.emplace_back(*$formal); delete $formal; }
{ $$ = $accum; $$->formals.emplace_back(std::move($formal)); }
| formal
{ $$ = new Formals; $$->formals.emplace_back(*$formal); delete $formal; }
{ $$ = new Formals; $$->formals.emplace_back(std::move($formal)); }
;
formal
: ID { $$ = new Formal{CUR_POS, state->symbols.create($1), 0}; }
| ID '?' expr { $$ = new Formal{CUR_POS, state->symbols.create($1), $3}; }
: ID { $$ = Formal{CUR_POS, state->symbols.create($1), 0}; }
| ID '?' expr { $$ = Formal{CUR_POS, state->symbols.create($1), $3}; }
;
%%
@@ -582,3 +557,4 @@ Expr * parseExprFromBuf(
}
#pragma GCC diagnostic pop // end ignored "-Wswitch-enum"
+5 -4
View File
@@ -1374,7 +1374,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
pos,
"while evaluating the `__structuredAttrs` "
"attribute passed to builtins.derivationStrict"))
jsonObject = StructuredAttrs{.structuredAttrs = json::object()};
jsonObject = StructuredAttrs{};
/* Check whether null attributes should be ignored. */
bool ignoreNulls = false;
@@ -1420,7 +1420,8 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
.debugThrow();
}
if (ingestionMethod == ContentAddressMethod::Raw::Text)
experimentalFeatureSettings.require(Xp::DynamicDerivations);
experimentalFeatureSettings.require(
Xp::DynamicDerivations, fmt("text-hashed derivation '%s', outputHashMode = \"text\"", drvName));
if (ingestionMethod == ContentAddressMethod::Raw::Git)
experimentalFeatureSettings.require(Xp::GitHashing);
};
@@ -4610,9 +4611,9 @@ struct RegexCache
}
};
std::shared_ptr<RegexCache> makeRegexCache()
ref<RegexCache> makeRegexCache()
{
return std::make_shared<RegexCache>();
return make_ref<RegexCache>();
}
void prim_match(EvalState & state, const PosIdx pos, Value ** args, Value & v)
+6
View File
@@ -64,6 +64,8 @@ static void runFetchClosureWithRewrite(
.pos = state.positions[pos]});
}
state.allowClosure(toPath);
state.mkStorePathString(toPath, v);
}
@@ -91,6 +93,8 @@ static void runFetchClosureWithContentAddressedPath(
.pos = state.positions[pos]});
}
state.allowClosure(fromPath);
state.mkStorePathString(fromPath, v);
}
@@ -115,6 +119,8 @@ static void runFetchClosureWithInputAddressedPath(
.pos = state.positions[pos]});
}
state.allowClosure(fromPath);
state.mkStorePathString(fromPath, v);
}
+7 -3
View File
@@ -199,8 +199,8 @@ static void fetchTree(
if (state.settings.pureEval && !input.isLocked()) {
if (input.getNarHash())
warn(
"Input '%s' is unlocked (e.g. lacks a Git revision) but does have a NAR hash. "
"This is deprecated since such inputs are verifiable but may not be reproducible.",
"Input '%s' is unlocked (e.g. lacks a Git revision) but is checked by NAR hash. "
"This is not reproducible and will break after garbage collection or when shared.",
input.to_string());
else
state
@@ -588,7 +588,11 @@ static void fetch(
if (expectedHash) {
auto hash = unpack ? state.store->queryPathInfo(storePath)->narHash
: hashFile(HashAlgorithm::SHA256, state.store->toRealPath(storePath));
: hashPath(
{state.store->requireStoreObjectAccessor(storePath)},
FileSerialisationMethod::Flat,
HashAlgorithm::SHA256)
.hash;
if (hash != *expectedHash) {
state
.error<EvalError>(
-1
View File
@@ -32,7 +32,6 @@ add_project_arguments(
)
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_fetchers.cc',
+6
View File
@@ -175,6 +175,12 @@ TEST_F(GitUtilsTest, peel_reference)
TEST(GitUtils, isLegalRefName)
{
ASSERT_TRUE(isLegalRefName("A/b"));
ASSERT_TRUE(isLegalRefName("AaA/b"));
ASSERT_TRUE(isLegalRefName("FOO/BAR/BAZ"));
ASSERT_TRUE(isLegalRefName("HEAD"));
ASSERT_TRUE(isLegalRefName("refs/tags/1.2.3"));
ASSERT_TRUE(isLegalRefName("refs/heads/master"));
ASSERT_TRUE(isLegalRefName("foox"));
ASSERT_TRUE(isLegalRefName("1337"));
ASSERT_TRUE(isLegalRefName("foo.baz"));
+1 -2
View File
@@ -37,7 +37,6 @@ libgit2 = dependency('libgit2')
deps_private += libgit2
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'access-tokens.cc',
@@ -64,7 +63,7 @@ this_exe = executable(
test(
meson.project_name(),
this_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
},
protocol : 'gtest',
-1
View File
@@ -61,7 +61,6 @@ mkMesonExecutable (finalAttrs: {
buildInputs = [ writableTmpDirAsHomeHook ];
}
''
export ASAN_OPTIONS=abort_on_error=1:print_summary=1:detect_leaks=0
export _NIX_TEST_UNIT_DATA=${resolvePath ./data}
${stdenv.hostPlatform.emulator buildPackages} ${lib.getExe finalAttrs.finalPackage}
touch $out
+11 -4
View File
@@ -5,6 +5,7 @@
#include "nix/util/json-utils.hh"
#include "nix/fetchers/fetch-settings.hh"
#include "nix/fetchers/fetch-to-store.hh"
#include "nix/util/url.hh"
#include <nlohmann/json.hpp>
@@ -65,6 +66,12 @@ Input Input::fromURL(const Settings & settings, const ParsedURL & url, bool requ
}
}
// Provide a helpful hint when user tries file+git instead of git+file
auto parsedScheme = parseUrlScheme(url.scheme);
if (parsedScheme.application == "file" && parsedScheme.transport == "git") {
throw Error("input '%s' is unsupported; did you mean 'git+file' instead of 'file+git'?", url);
}
throw Error("input '%s' is unsupported", url);
}
@@ -332,8 +339,7 @@ std::pair<ref<SourceAccessor>, Input> Input::getAccessorUnchecked(ref<Store> sto
debug("using substituted/cached input '%s' in '%s'", to_string(), store->printStorePath(storePath));
// We just ensured the store object was there
auto accessor = ref{store->getFSAccessor(storePath)};
auto accessor = store->requireStoreObjectAccessor(storePath);
accessor->fingerprint = getFingerprint(store);
@@ -513,10 +519,11 @@ using namespace nix;
fetchers::PublicKey adl_serializer<fetchers::PublicKey>::from_json(const json & json)
{
fetchers::PublicKey res = {};
if (auto type = optionalValueAt(json, "type"))
auto & obj = getObject(json);
if (auto * type = optionalValueAt(obj, "type"))
res.type = getString(*type);
res.key = getString(valueAt(json, "key"));
res.key = getString(valueAt(obj, "key"));
return res;
}
+1 -1
View File
@@ -209,7 +209,7 @@ std::vector<nlohmann::json> Fetch::fetchUrls(const std::vector<Pointer> & pointe
auto url = api.endpoint + "/objects/batch";
const auto & authHeader = api.authHeader;
FileTransferRequest request(parseURL(url));
request.post = true;
request.method = HttpMethod::POST;
Headers headers;
if (authHeader.has_value())
headers.push_back({"Authorization", *authHeader});
+18 -45
View File
@@ -9,9 +9,11 @@
#include "nix/util/users.hh"
#include "nix/util/fs-sink.hh"
#include "nix/util/sync.hh"
#include "nix/util/util.hh"
#include <git2/attr.h>
#include <git2/blob.h>
#include <git2/branch.h>
#include <git2/commit.h>
#include <git2/config.h>
#include <git2/describe.h>
@@ -28,6 +30,7 @@
#include <git2/submodule.h>
#include <git2/sys/odb_backend.h>
#include <git2/sys/mempack.h>
#include <git2/tag.h>
#include <git2/tree.h>
#include <boost/unordered/unordered_flat_map.hpp>
@@ -528,12 +531,12 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
auto act = (Activity *) payload;
act->result(
resFetchStatus,
fmt("%d/%d objects received, %d/%d deltas indexed, %.1f MiB",
fmt("%d/%d objects received, %d/%d deltas indexed, %s",
stats->received_objects,
stats->total_objects,
stats->indexed_deltas,
stats->total_deltas,
stats->received_bytes / (1024.0 * 1024.0)));
renderSize(stats->received_bytes)));
return getInterrupted() ? -1 : 0;
}
@@ -1323,63 +1326,33 @@ GitRepo::WorkdirInfo GitRepo::getCachedWorkdirInfo(const std::filesystem::path &
return workdirInfo;
}
/**
* Checks that the git reference is valid and normalizes slash '/' sequences.
*
* Accepts shorthand references (one-level refnames are allowed).
*/
bool isValidRefNameAllowNormalizations(const std::string & refName)
{
/* Unfortunately libgit2 doesn't expose the limit in headers, but its internal
limit is also 1024. */
std::array<char, 1024> normalizedRefBuffer;
/* It would be nice to have a better API like git_reference_name_is_valid, but
* with GIT_REFERENCE_FORMAT_REFSPEC_SHORTHAND flag. libgit2 uses it internally
* but doesn't expose it in public headers [1].
* [1]:
* https://github.com/libgit2/libgit2/blob/9d5f1bacc23594c2ba324c8f0d41b88bf0e9ef04/src/libgit2/refs.c#L1362-L1365
*/
auto res = git_reference_normalize_name(
normalizedRefBuffer.data(),
normalizedRefBuffer.size(),
refName.c_str(),
GIT_REFERENCE_FORMAT_ALLOW_ONELEVEL | GIT_REFERENCE_FORMAT_REFSPEC_SHORTHAND);
return res == 0;
}
bool isLegalRefName(const std::string & refName)
{
initLibGit2();
/* Since `git_reference_normalize_name` is the best API libgit2 has for verifying
* reference names with shorthands (see comment in normalizeRefName), we need to
* ensure that exceptions to the validity checks imposed by normalization [1] are checked
* explicitly.
* [1]: https://git-scm.com/docs/git-check-ref-format#Documentation/git-check-ref-format.txt---normalize
*/
/* Check for cases that don't get rejected by libgit2.
* FIXME: libgit2 should reject this. */
if (refName == "@")
return false;
/* Leading slashes and consecutive slashes are stripped during normalizatiton. */
if (refName.starts_with('/') || refName.find("//") != refName.npos)
return false;
/* Refer to libgit2. */
if (!isValidRefNameAllowNormalizations(refName))
return false;
/* libgit2 doesn't barf on DEL symbol.
* FIXME: libgit2 should reject this. */
if (refName.find('\177') != refName.npos)
return false;
return true;
for (auto * func : {
git_reference_name_is_valid,
git_branch_name_is_valid,
git_tag_name_is_valid,
}) {
int valid = 0;
if (func(&valid, refName.c_str()))
throw Error("checking git reference '%s': %s", refName, git_error_last()->message);
if (valid)
return true;
}
return false;
}
} // namespace nix
+30
View File
@@ -496,6 +496,36 @@ struct GitInputScheme : InputScheme
Git interprets them as part of the file name. So get
rid of them. */
url.query.clear();
/* Backward compatibility hack: In old versions of Nix, if you had
a flake input like
inputs.foo.url = "git+https://foo/bar?dir=subdir";
it would result in a lock file entry like
"original": {
"dir": "subdir",
"type": "git",
"url": "https://foo/bar?dir=subdir"
}
New versions of Nix remove `?dir=subdir` from the `url` field,
since the subdirectory is intended for `FlakeRef`, not the
fetcher (and specifically the remote server), that is, the
flakeref is parsed into
"original": {
"dir": "subdir",
"type": "git",
"url": "https://foo/bar"
}
However, new versions of nix parsing old flake.lock files would pass the dir=
query parameter in the "url" attribute to git, which will then complain.
For this reason, we are filtering the `dir` query parameter from the URL
before passing it to git. */
url.query.erase("dir");
repoInfo.location = url;
}
+11 -11
View File
@@ -399,7 +399,8 @@ struct GitHubInputScheme : GitArchiveInputScheme
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
auto downloadResult = downloadFile(store, *input.settings, url, "source", headers);
auto json = nlohmann::json::parse(store->getFSAccessor(downloadResult.storePath)->readFile(CanonPath::root));
auto json = nlohmann::json::parse(
store->requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
return RefInfo{
.rev = Hash::parseAny(std::string{json["sha"]}, HashAlgorithm::SHA1),
@@ -473,7 +474,8 @@ struct GitLabInputScheme : GitArchiveInputScheme
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
auto downloadResult = downloadFile(store, *input.settings, url, "source", headers);
auto json = nlohmann::json::parse(store->getFSAccessor(downloadResult.storePath)->readFile(CanonPath::root));
auto json = nlohmann::json::parse(
store->requireStoreObjectAccessor(downloadResult.storePath)->readFile(CanonPath::root));
if (json.is_array() && json.size() >= 1 && json[0]["id"] != nullptr) {
return RefInfo{.rev = Hash::parseAny(std::string(json[0]["id"]), HashAlgorithm::SHA1)};
@@ -548,13 +550,10 @@ struct SourceHutInputScheme : GitArchiveInputScheme
std::string refUri;
if (ref == "HEAD") {
auto file = store->toRealPath(
downloadFile(store, *input.settings, fmt("%s/HEAD", base_url), "source", headers).storePath);
std::ifstream is(file);
std::string line;
getline(is, line);
auto downloadFileResult = downloadFile(store, *input.settings, fmt("%s/HEAD", base_url), "source", headers);
auto contents = store->requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
auto remoteLine = git::parseLsRemoteLine(line);
auto remoteLine = git::parseLsRemoteLine(getLine(contents).first);
if (!remoteLine) {
throw BadURL("in '%d', couldn't resolve HEAD ref '%d'", input.to_string(), ref);
}
@@ -564,9 +563,10 @@ struct SourceHutInputScheme : GitArchiveInputScheme
}
std::regex refRegex(refUri);
auto file = store->toRealPath(
downloadFile(store, *input.settings, fmt("%s/info/refs", base_url), "source", headers).storePath);
std::ifstream is(file);
auto downloadFileResult =
downloadFile(store, *input.settings, fmt("%s/info/refs", base_url), "source", headers);
auto contents = store->requireStoreObjectAccessor(downloadFileResult.storePath)->readFile(CanonPath::root);
std::istringstream is(contents);
std::string line;
std::optional<std::string> id;
@@ -158,9 +158,12 @@ struct Setter
};
/**
* Checks that the git reference is valid and normalized.
* Checks that the string can be a valid git reference, branch or tag name.
* Accepts shorthand references (one-level refnames are allowed), pseudorefs
* like `HEAD`.
*
* Accepts shorthand references (one-level refnames are allowed).
* @note This is a coarse test to make sure that the refname is at least something
* that Git can make sense of.
*/
bool isLegalRefName(const std::string & refName);
+1 -3
View File
@@ -329,9 +329,7 @@ struct MercurialInputScheme : InputScheme
Input input(_input);
auto storePath = fetchToStore(store, input);
// We just added it, it should be there.
auto accessor = ref{store->getFSAccessor(storePath)};
auto accessor = store->requireStoreObjectAccessor(storePath);
accessor->setPathDisplay("«" + input.to_string() + "»");
-1
View File
@@ -32,7 +32,6 @@ libgit2 = dependency('libgit2', version : '>= 1.9')
deps_private += libgit2
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'attrs.cc',
+1 -1
View File
@@ -138,7 +138,7 @@ struct PathInputScheme : InputScheme
storePath = store->addToStoreFromDump(*src, "source");
}
auto accessor = ref{store->getFSAccessor(*storePath)};
auto accessor = store->requireStoreObjectAccessor(*storePath);
// To prevent `fetchToStore()` copying the path again to Nix
// store, pre-create an entry in the fetcher cache.
+6 -6
View File
@@ -42,7 +42,7 @@ DownloadFileResult downloadFile(
if (cached && !cached->expired)
return useCached();
FileTransferRequest request(ValidURL{url});
FileTransferRequest request(VerbatimURL{url});
request.headers = headers;
if (cached)
request.expectedETag = getStrAttr(cached->value, "etag");
@@ -107,13 +107,13 @@ DownloadFileResult downloadFile(
static DownloadTarballResult downloadTarball_(
const Settings & settings, const std::string & urlS, const Headers & headers, const std::string & displayPrefix)
{
ValidURL url = urlS;
ParsedURL url = parseURL(urlS);
// Some friendly error messages for common mistakes.
// Namely lets catch when the url is a local file path, but
// it is not in fact a tarball.
if (url.scheme() == "file") {
std::filesystem::path localPath = renderUrlPathEnsureLegal(url.path());
if (url.scheme == "file") {
std::filesystem::path localPath = renderUrlPathEnsureLegal(url.path);
if (!exists(localPath)) {
throw Error("tarball '%s' does not exist.", localPath);
}
@@ -164,7 +164,7 @@ static DownloadTarballResult downloadTarball_(
/* Note: if the download is cached, `importTarball()` will receive
no data, which causes it to import an empty tarball. */
auto archive = !url.path().empty() && hasSuffix(toLower(url.path().back()), ".zip") ? ({
auto archive = !url.path.empty() && hasSuffix(toLower(url.path.back()), ".zip") ? ({
/* In streaming mode, libarchive doesn't handle
symlinks in zip files correctly (#10649). So write
the entire file to disk so libarchive can access it
@@ -178,7 +178,7 @@ static DownloadTarballResult downloadTarball_(
}
TarArchive{path};
})
: TarArchive{*source};
: TarArchive{*source};
auto tarballCache = getTarballCache();
auto parseSink = tarballCache->getFileSystemObjectSink();
auto lastModified = unpackTarfileToSink(archive, *parseSink);
-1
View File
@@ -32,7 +32,6 @@ deps_public_maybe_subproject = [
subdir('nix-meson-build-support/subprojects')
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'nix_api_flake.cc',
+7
View File
@@ -1,8 +1,15 @@
#include <gtest/gtest.h>
#include <set>
#include <string>
#include <utility>
#include "nix/fetchers/fetch-settings.hh"
#include "nix/flake/flakeref.hh"
#include "nix/fetchers/attrs.hh"
#include "nix/fetchers/fetchers.hh"
#include "nix/util/configuration.hh"
#include "nix/util/error.hh"
#include "nix/util/experimental-features.hh"
namespace nix {
+1 -2
View File
@@ -34,7 +34,6 @@ gtest = dependency('gtest', main : true)
deps_private += gtest
subdir('nix-meson-build-support/common')
subdir('nix-meson-build-support/asan-options')
sources = files(
'flakeref.cc',
@@ -59,7 +58,7 @@ this_exe = executable(
test(
meson.project_name(),
this_exe,
env : asan_test_options_env + {
env : {
'_NIX_TEST_UNIT_DATA' : meson.current_source_dir() / 'data',
'NIX_CONFIG' : 'extra-experimental-features = flakes',
'HOME' : meson.current_build_dir() / 'test-home',
+7 -5
View File
@@ -1,15 +1,17 @@
#include <gtest/gtest.h>
#include <filesystem>
#include <string>
#include "nix/util/file-system.hh"
#include "nix_api_store.h"
#include "nix_api_util.h"
#include "nix_api_expr.h"
#include "nix_api_value.h"
#include "nix_api_flake.h"
#include "nix/expr/tests/nix_api_expr.hh"
#include "nix/util/tests/string_callback.hh"
#include <gmock/gmock.h>
#include <gtest/gtest.h>
#include "nix/store/tests/nix_api_store.hh"
#include "nix/util/tests/nix_api_util.hh"
#include "nix_api_fetchers.h"
namespace nixC {
Loaded 100 of 332 files, more files were not shown because too many files have changed in this diff. Show more