Compare commits

..

1 Commits

Author SHA1 Message Date
John Ericson
da9ade20e3 Add warning for non-JSON-object exportReferencesGraph 2025-12-09 13:02:46 -05:00
65 changed files with 557 additions and 1195 deletions

View File

@@ -1 +1 @@
2.34.0
2.33.0

View File

@@ -0,0 +1,9 @@
---
synopsis: Channel URLs migrated to channels.nixos.org subdomain
prs: [14518]
issues: [14517]
---
Channel URLs have been updated from `https://nixos.org/channels/` to `https://channels.nixos.org/` throughout Nix.
The subdomain provides better reliability with IPv6 support and improved CDN distribution. The old domain apex (`nixos.org/channels/`) currently redirects to the new location but may be deprecated in the future.

View File

@@ -0,0 +1,88 @@
---
synopsis: "JSON format changes for store path info and derivations"
prs: []
issues: []
---
JSON formats for store path info and derivations have been updated with new versions and structured fields.
## Store Path Info JSON
`nix path-info --json` now requires a `--json-format` flag to specify the output format version.
Using `--json` without `--json-format` is deprecated and will become an error in a future release.
For now, it defaults to version 1 with a warning, for a smoother migration.
### Version 1 (`--json-format 1`)
This is the legacy format, preserved for backwards compatibility:
- String-based hash values (e.g., `"narHash": "sha256:FePFYIlM..."`)
- String-based content addresses (e.g., `"ca": "fixed:r:sha256:1abc..."`)
- Full store paths for map keys and references (e.g., `"/nix/store/abc...-foo"`)
- Now includes `"storeDir"` field at the top level
### Version 2 (`--json-format 2`)
The new structured format follows the [JSON guidelines](@docroot@/development/json-guideline.md) with the following changes:
- **Nested structure with top-level metadata**:
The output is now wrapped in an object with `version`, `storeDir`, and `info` fields:
```json
{
"version": 2,
"storeDir": "/nix/store",
"info": { ... }
}
```
The map from store bath base names to store object info is nested under the `info` field.
- **Store path base names instead of full paths**:
Map keys and references use store path base names (e.g., `"abc...-foo"`) instead of full absolute store paths.
Combined with `storeDir`, the full path can be reconstructed.
- **Structured `ca` field**:
Content address is now a structured JSON object instead of a string:
- Old: `"ca": "fixed:r:sha256:1abc..."`
- New: `"ca": {"method": "nar", "hash": "sha256-ungWv48Bz+pBQUDeXa4iI7ADYaOWF3qctBD/YfIAFa0="}`
- Still `null` values for input-addressed store objects
The `hash` field uses the [SRI](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) format like other hashes.
Nix currently only produces, and doesn't consume this format.
Additionally the following field is added to both formats.
(The `version` tracks breaking changes, and adding fields to outputted JSON is not a breaking change.)
- **`version` field**:
All store path info JSON now includes `"version": <1|2>`.
- **`storeDir` field**:
Top-level `"storeDir"` field contains the store directory path (e.g., `"/nix/store"`).
## Derivation JSON (Version 4)
The derivation JSON format has been updated from version 3 to version 4:
- **Restructured inputs**:
Inputs are now nested under an `inputs` object:
- Old: `"inputSrcs": [...], "inputDrvs": {...}`
- New: `"inputs": {"srcs": [...], "drvs": {...}}`
- **Consistent content addresses**:
Fixed content-addressed outputs now use structured JSON format.
This is the same format as `ca` in store path info (after the new version).
Version 3 and earlier formats are *not* accepted when reading.
**Affected command**: `nix derivation`, namely its `show` and `add` sub-commands.

View File

@@ -0,0 +1,12 @@
---
synopsis: Fix "download buffer is full; consider increasing the 'download-buffer-size' setting" warning
prs: [14614]
issues: [11728]
---
The underlying issue that led to [#11728](https://github.com/NixOS/nix/issues/11728) has been resolved by utilizing
[libcurl write pausing functionality](https://curl.se/libcurl/c/curl_easy_pause.html) to control backpressure when unpacking to slow destinations like the git-backed tarball cache. The default value of `download-buffer-size` is now 1 MiB and it's no longer recommended to increase it, since the root cause has been fixed.
This is expected to improve download performance on fast connections, since previously a single slow download consumer would stall the thread and prevent any other transfers from progressing.
Many thanks go out to the [Lix project](https://lix.systems/) for the [implementation](https://git.lix.systems/lix-project/lix/commit/4ae6fb5a8f0d456b8d2ba2aaca3712b4e49057fc) that served as inspiration for this change and for triaging libcurl [issues with pausing](https://github.com/curl/curl/issues/19334).

View File

@@ -0,0 +1,8 @@
---
synopsis: Interrupting REPL commands works more than once
issues: [13481]
---
Previously, this only worked once per REPL session; further attempts would be ignored.
This issue is now fixed, so REPL commands such as `:b` or `:p` can be canceled consistently.
This is a cherry-pick of the change from the [Lix project](https://gerrit.lix.systems/c/lix/+/1097).

View File

@@ -0,0 +1,40 @@
---
synopsis: "Improved S3 binary cache support via HTTP"
prs: [13752, 13823, 14026, 14120, 14131, 14135, 14144, 14170, 14190, 14198, 14206, 14209, 14222, 14223, 14330, 14333, 14335, 14336, 14337, 14350, 14356, 14357, 14374, 14375, 14376, 14377, 14391, 14393, 14420, 14421]
issues: [13084, 12671, 11748, 12403]
---
S3 binary cache operations now happen via HTTP, leveraging `libcurl`'s native
AWS SigV4 authentication instead of the AWS C++ SDK, providing significant
improvements:
- **Reduced memory usage**: Eliminates memory buffering issues that caused
segfaults with large files
- **Fixed upload reliability**: Resolves AWS SDK chunking errors
(`InvalidChunkSizeError`)
- **Lighter dependencies**: Uses lightweight `aws-crt-cpp` instead of full
`aws-cpp-sdk`, reducing build complexity
The new implementation requires curl >= 7.75.0 and `aws-crt-cpp` for credential
management.
All existing S3 URL formats and parameters remain supported, however the store
settings for configuring multipart uploads have changed:
- **`multipart-upload`** (default: `false`): Enable multipart uploads for large
files. When enabled, files exceeding the multipart threshold will be uploaded
in multiple parts.
- **`multipart-threshold`** (default: `100 MiB`): Minimum file size for using
multipart uploads. Files smaller than this will use regular PUT requests.
Only takes effect when `multipart-upload` is enabled.
- **`multipart-chunk-size`** (default: `5 MiB`): Size of each part in multipart
uploads. Must be at least 5 MiB (AWS S3 requirement). Larger chunk sizes
reduce the number of requests but use more memory.
- **`buffer-size`**: Has been replaced by `multipart-chunk-size` and is now an alias to it.
Note that this change also means Nix now supports S3 binary cache stores even
if built without `aws-crt-cpp`, but only for public buckets which do not
require authentication.

View File

@@ -0,0 +1,14 @@
---
synopsis: "S3 URLs now support object versioning via versionId parameter"
prs: [14274]
issues: [13955]
---
S3 URLs now support a `versionId` query parameter to fetch specific versions
of objects from S3 buckets with versioning enabled. This allows pinning to
exact object versions for reproducibility and protection against unexpected
changes:
```
s3://bucket/key?region=us-east-1&versionId=abc123def456
```

View File

@@ -0,0 +1,21 @@
---
synopsis: "S3 binary cache stores now support storage class configuration"
prs: [14464]
issues: [7015]
---
S3 binary cache stores now support configuring the storage class for uploaded objects via the `storage-class` parameter. This allows users to optimize costs by selecting appropriate storage tiers based on access patterns.
Example usage:
```bash
# Use Glacier storage for long-term archival
nix copy --to 's3://my-bucket?storage-class=GLACIER' /nix/store/...
# Use Intelligent Tiering for automatic cost optimization
nix copy --to 's3://my-bucket?storage-class=INTELLIGENT_TIERING' /nix/store/...
```
The storage class applies to both regular uploads and multipart uploads. When not specified, objects use the bucket's default storage class.
See the [S3 storage classes documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) for available storage classes and their characteristics.

View File

@@ -151,7 +151,6 @@
- [Contributing](development/contributing.md)
- [Releases](release-notes/index.md)
{{#include ./SUMMARY-rl-next.md}}
- [Release 2.33 (2025-12-09)](release-notes/rl-2.33.md)
- [Release 2.32 (2025-10-06)](release-notes/rl-2.32.md)
- [Release 2.31 (2025-08-21)](release-notes/rl-2.31.md)
- [Release 2.30 (2025-07-07)](release-notes/rl-2.30.md)

View File

@@ -6,7 +6,14 @@ Additionally, see [Testing Nix](./testing.md) for further instructions on how to
## Building Nix with Debug Symbols
In the development shell, `mesonBuildType` is set automatically to `debugoptimized`. This builds Nix with debug symbols, which are essential for effective debugging.
In the development shell, set the `mesonBuildType` environment variable to `debug` before configuring the build:
```console
[nix-shell]$ export mesonBuildType=debugoptimized
```
Then, proceed to build Nix as described in [Building Nix](./building.md).
This will build Nix with debug symbols, which are essential for effective debugging.
It is also possible to build without optimization for faster build:

View File

@@ -1,281 +0,0 @@
# Release 2.33.0 (2025-12-09)
## New features
- New command `nix registry resolve` [#14595](https://github.com/NixOS/nix/pull/14595)
This command looks up a flake registry input name and returns the flakeref it resolves to.
For example, looking up Nixpkgs:
```
$ nix registry resolve nixpkgs
github:NixOS/nixpkgs/nixpkgs-unstable
```
Upstreamed from [Determinate Nix 3.14.0](https://github.com/DeterminateSystems/nix-src/pull/273).
- `nix flake clone` supports all input types [#14581](https://github.com/NixOS/nix/pull/14581)
`nix flake clone` now supports arbitrary input types. In particular, this allows you to clone tarball flakes, such as flakes on FlakeHub.
Upstreamed from [Determinate Nix 3.12.0](https://github.com/DeterminateSystems/nix-src/pull/229).
## Performance improvements
- Git fetcher computes `revCount`s using multiple threads [#14462](https://github.com/NixOS/nix/pull/14462)
When using Git repositories with a long history, calculating the `revCount` attribute can take a long time. Nix now computes `revCount` using multiple threads, making it much faster (e.g. 9.1s to 3.7s for Nixpkgs).
Note that if you don't need `revCount`, you can disable it altogether by setting the flake input attribute `shallow = true`.
Upstreamed from [Determinate Nix 3.12.2](https://github.com/DeterminateSystems/nix-src/pull/245).
- `builtins.stringLength` now runs in constant time [#14442](https://github.com/NixOS/nix/pull/14442)
The internal representation of strings has been replaced with a size-prefixed Pascal style string. Previously Nix stored strings as a NUL-terminated array of bytes, necessitating a linear scan to calculate the length.
- Uploads to `http://` and `https://` binary cache stores now run in constant memory [#14390](https://github.com/NixOS/nix/pull/14390)
Nix used to buffer the whole compressed NAR contents in memory. It now reads it in a streaming fashion.
- Channel URLs migrated to channels.nixos.org subdomain [#14517](https://github.com/NixOS/nix/issues/14517) [#14518](https://github.com/NixOS/nix/pull/14518)
Channel URLs have been updated from `https://nixos.org/channels/` to `https://channels.nixos.org/` throughout Nix. This subdomain provides better reliability with IPv6 support and improved CDN distribution. The old domain apex (`nixos.org/channels/`) currently redirects to the new location but may be deprecated in the future.
- Fix `download buffer is full; consider increasing the 'download-buffer-size' setting` warning [#11728](https://github.com/NixOS/nix/issues/11728) [#14614](https://github.com/NixOS/nix/pull/14614)
The underlying issue that led to [#11728](https://github.com/NixOS/nix/issues/11728) has been resolved by utilizing
[libcurl write pausing functionality](https://curl.se/libcurl/c/curl_easy_pause.html) to control backpressure when unpacking to slow destinations like the git-backed tarball cache. The default value of `download-buffer-size` is now 1 MiB and it's no longer recommended to increase it, since the root cause has been fixed.
This is expected to improve download performance on fast connections, since previously a single slow download consumer would stall the thread and prevent any other transfers from progressing.
Many thanks go out to the [Lix project](https://lix.systems/) for the [implementation](https://git.lix.systems/lix-project/lix/commit/4ae6fb5a8f0d456b8d2ba2aaca3712b4e49057fc) that served as inspiration for this change and for triaging libcurl [issues with pausing](https://github.com/curl/curl/issues/19334).
- Significantly improve tarball unpacking performance [#14689](https://github.com/NixOS/nix/pull/14689) [#14696](https://github.com/NixOS/nix/pull/14696) [#10683](https://github.com/NixOS/nix/issues/10683) [#11098](https://github.com/NixOS/nix/issues/11098)
Nix uses a content-addressed cache backed by libgit2 for deduplicating files fetched via `fetchTarball` and `github`, `tarball` flake inputs. Its usage has been significantly optimised to reduce the amount of I/O operations that are performed. For a typical nixpkgs source tarball this results in 200 times fewer system calls on Linux. In combination with libcurl pausing this alleviates performance regressions stemming from the tarball cache.
- Already valid derivations are no longer copied to the store [#14219](https://github.com/NixOS/nix/pull/14219)
This results in a modest speedup when using the Nix daemon.
- `nix nar ls` and `nix nar cat` are significantly faster and no longer buffer the whole NAR in memory [#14273](https://github.com/NixOS/nix/pull/14273) [#14732](https://github.com/NixOS/nix/pull/14732)
## S3 improvements
- Improved S3 binary cache support via HTTP [#11748](https://github.com/NixOS/nix/issues/11748) [#12403](https://github.com/NixOS/nix/issues/12403) [#12671](https://github.com/NixOS/nix/issues/12671) [#13084](https://github.com/NixOS/nix/issues/13084) [#13752](https://github.com/NixOS/nix/pull/13752) [#13823](https://github.com/NixOS/nix/pull/13823) [#14026](https://github.com/NixOS/nix/pull/14026) [#14120](https://github.com/NixOS/nix/pull/14120) [#14131](https://github.com/NixOS/nix/pull/14131) [#14135](https://github.com/NixOS/nix/pull/14135) [#14144](https://github.com/NixOS/nix/pull/14144) [#14170](https://github.com/NixOS/nix/pull/14170) [#14190](https://github.com/NixOS/nix/pull/14190) [#14198](https://github.com/NixOS/nix/pull/14198) [#14206](https://github.com/NixOS/nix/pull/14206) [#14209](https://github.com/NixOS/nix/pull/14209) [#14222](https://github.com/NixOS/nix/pull/14222) [#14223](https://github.com/NixOS/nix/pull/14223) [#14330](https://github.com/NixOS/nix/pull/14330) [#14333](https://github.com/NixOS/nix/pull/14333) [#14335](https://github.com/NixOS/nix/pull/14335) [#14336](https://github.com/NixOS/nix/pull/14336) [#14337](https://github.com/NixOS/nix/pull/14337) [#14350](https://github.com/NixOS/nix/pull/14350) [#14356](https://github.com/NixOS/nix/pull/14356) [#14357](https://github.com/NixOS/nix/pull/14357) [#14374](https://github.com/NixOS/nix/pull/14374) [#14375](https://github.com/NixOS/nix/pull/14375) [#14376](https://github.com/NixOS/nix/pull/14376) [#14377](https://github.com/NixOS/nix/pull/14377) [#14391](https://github.com/NixOS/nix/pull/14391) [#14393](https://github.com/NixOS/nix/pull/14393) [#14420](https://github.com/NixOS/nix/pull/14420) [#14421](https://github.com/NixOS/nix/pull/14421)
S3 binary cache operations now happen via HTTP, leveraging `libcurl`'s native AWS SigV4 authentication instead of the AWS C++ SDK, providing significant improvements:
- **Reduced memory usage**: Eliminates memory buffering issues that caused segfaults with large files
- **Fixed upload reliability**: Resolves AWS SDK chunking errors (`InvalidChunkSizeError`)
- **Lighter dependencies**: Uses lightweight `aws-crt-cpp` instead of full `aws-cpp-sdk`, reducing build complexity
The new implementation requires curl >= 7.75.0 and `aws-crt-cpp` for credential management.
All existing S3 URL formats and parameters remain supported, however the store settings for configuring multipart uploads have changed:
- **`multipart-upload`** (default: `false`): Enable multipart uploads for large files. When enabled, files exceeding the multipart threshold will be uploaded in multiple parts.
- **`multipart-threshold`** (default: `100 MiB`): Minimum file size for using multipart uploads. Files smaller than this will use regular PUT requests. Only takes effect when `multipart-upload` is enabled.
- **`multipart-chunk-size`** (default: `5 MiB`): Size of each part in multipart uploads. Must be at least 5 MiB (AWS S3 requirement). Larger chunk sizes reduce the number of requests but use more memory.
- **`buffer-size`**: Has been replaced by `multipart-chunk-size` and is now an alias to it.
Note that this change also means Nix now supports S3 binary cache stores even if built without `aws-crt-cpp`, but only for public buckets which do not require authentication.
- S3 URLs now support object versioning via `versionId` parameter [#13955](https://github.com/NixOS/nix/issues/13955) [#14274](https://github.com/NixOS/nix/pull/14274)
S3 URLs now support a `versionId` query parameter to fetch specific versions
of objects from S3 buckets with versioning enabled. This allows pinning to
exact object versions for reproducibility and protection against unexpected
changes:
```
s3://bucket/key?region=us-east-1&versionId=abc123def456
```
- S3 binary cache stores now support storage class configuration [#7015](https://github.com/NixOS/nix/issues/7015) [#14464](https://github.com/NixOS/nix/pull/14464)
S3 binary cache stores now support configuring the storage class for uploaded objects via the `storage-class` parameter. This allows users to optimize costs by selecting appropriate storage tiers based on access patterns.
Example usage:
```bash
# Use Glacier storage for long-term archival
nix copy --to 's3://my-bucket?storage-class=GLACIER' /nix/store/...
# Use Intelligent Tiering for automatic cost optimization
nix copy --to 's3://my-bucket?storage-class=INTELLIGENT_TIERING' /nix/store/...
```
The storage class applies to both regular uploads and multipart uploads. When not specified, objects use the bucket's default storage class.
See the [S3 storage classes documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) for available storage classes and their characteristics.
## Store path info JSON format changes
The JSON format emitted by `nix path-info --json` has been updated to a new version with improved structure.
To maintain compatibility, `nix path-info --json` now requires a `--json-format` flag to specify the output format version.
Using `--json` without `--json-format` is deprecated and will become an error in a future release.
For now, it defaults to version 1 with a warning, for a smoother migration.
### Version 1 (`--json-format 1`)
This is the legacy format, preserved for backwards compatibility:
- String-based hash values (e.g., `"narHash": "sha256:FePFYIlM..."`)
- String-based content addresses (e.g., `"ca": "fixed:r:sha256:1abc..."`)
- Full store paths for map keys and references (e.g., `"/nix/store/abc...-foo"`)
- Now includes `"storeDir"` field at the top level
### Version 2 (`--json-format 2`)
The new structured format follows the [JSON guidelines](@docroot@/development/json-guideline.md) with the following changes:
- **Nested structure with top-level metadata**:
The output is now wrapped in an object with `version`, `storeDir`, and `info` fields:
```json
{
"version": 2,
"storeDir": "/nix/store",
"info": { ... }
}
```
The map from store path base names to store object info is nested under the `info` field.
- **Store path base names instead of full paths**:
Map keys and references use store path base names (e.g., `"abc...-foo"`) instead of full absolute store paths.
Combined with `storeDir`, the full path can be reconstructed.
- **Structured `ca` field**:
Content address is now a structured JSON object instead of a string:
- Old: `"ca": "fixed:r:sha256:1abc..."`
- New: `"ca": {"method": "nar", "hash": "sha256-ungWv48Bz+pBQUDeXa4iI7ADYaOWF3qctBD/YfIAFa0="}`
- Still `null` values for input-addressed store objects
The `hash` field uses the [SRI](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) format like other hashes.
Additionally the following fields are added to both formats:
- **`version` field**:
All store path info JSON now includes `"version": <1|2>`. The `version` tracks breaking changes, and adding fields to outputted JSON is not a breaking change.
- **`storeDir` field**:
Top-level `"storeDir"` field contains the store directory path (e.g., `"/nix/store"`).
## Derivation JSON format changes
The derivation JSON format has been updated from version 3 to version 4:
- **Nested structure with top-level metadata**:
The output of `nix derivation show` is now wrapped in an object with `version` and `derivations` fields:
```json
{
"version": 4,
"derivations": { ... }
}
```
The map from derivation paths to derivation info is nested under the `derivations` field.
This matches the structure used for `nix path-info --json --json-format 2`, and likewise brings this command into compliance with the JSON guidelines.
- **Restructured inputs**:
Inputs are now nested under an `inputs` object:
- Old: `"inputSrcs": [...], "inputDrvs": {...}`
- New: `"inputs": {"srcs": [...], "drvs": {...}}`
- **Consistent content addresses**:
Fixed content-addressed outputs now use structured JSON format.
This is the same format as `ca` in store path info (after the new version).
Version 3 and earlier formats are *not* accepted when reading.
**Affected command**: `nix derivation`, namely its `show` and `add` sub-commands.
## Miscellaneous changes
- Git fetcher: Restore progress indication [#14487](https://github.com/NixOS/nix/pull/14487)
Nix used to feel "stuck" while it was cloning large repositories. Nix now shows Git's native progress indicator while fetching.
Upstreamed from [Determinate Nix 3.13.0](https://github.com/DeterminateSystems/nix-src/pull/250).
- Interrupting REPL commands works more than once [#13481](https://github.com/NixOS/nix/issues/13481)
Previously, this only worked once per REPL session; further attempts would be ignored.
This issue is now fixed, so REPL commands such as `:b` or `:p` can be canceled consistently.
This is a cherry-pick of the change from the [Lix project](https://gerrit.lix.systems/c/lix/+/1097).
- NAR unpacking code has been rewritten to make use of dirfd-based `openat` and `openat2` system calls when available [#14597](https://github.com/NixOS/nix/pull/14597)
- Dynamic size unit rendering [#14423](https://github.com/NixOS/nix/pull/14423) [#14364](https://github.com/NixOS/nix/pull/14364)
Various commands and the progress bar now use dynamically determined size units instead
of always using `MiB`. For example, the progress bar now reports download status like:
```
[1/196/197 copied (773.7 MiB/2.1 GiB), 172.4/421.5 MiB DL]
```
Instead of:
```
[1/196/197 copied (773.7/2147.3 MiB), 172.4/421.5 MiB DL]
```
## Contributors
This release was made possible by the following 33 contributors:
- Adam Dinwoodie [**(@me-and)**](https://github.com/me-and)
- jonhermansen [**(@jonhermansen)**](https://github.com/jonhermansen)
- Arnout Engelen [**(@raboof)**](https://github.com/raboof)
- Jean-François Roche [**(@jfroche)**](https://github.com/jfroche)
- tomberek [**(@tomberek)**](https://github.com/tomberek)
- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
- Marcel [**(@MarcelCoding)**](https://github.com/MarcelCoding)
- David McFarland [**(@corngood)**](https://github.com/corngood)
- Soumyadip Sarkar [**(@neuralsorcerer)**](https://github.com/neuralsorcerer)
- Cole Helbling [**(@cole-h)**](https://github.com/cole-h)
- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
- Tristan Ross [**(@RossComputerGuy)**](https://github.com/RossComputerGuy)
- Alex Auvolat [**(@Alexis211)**](https://github.com/Alexis211)
- edef [**(@edef1c)**](https://github.com/edef1c)
- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
- Vinayak Goyal [**(@vinayakankugoyal)**](https://github.com/vinayakankugoyal)
- Graham Dennis [**(@GrahamDennis)**](https://github.com/GrahamDennis)
- Aspen Smith [**(@glittershark)**](https://github.com/glittershark)
- Jens Petersen [**(@juhp)**](https://github.com/juhp)
- Bernardo Meurer [**(@lovesegfault)**](https://github.com/lovesegfault)
- Peter Bynum [**(@pkpbynum)**](https://github.com/pkpbynum)
- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
- Alex Decious [**(@adeci)**](https://github.com/adeci)
- Matthieu Coudron [**(@teto)**](https://github.com/teto)
- Domen Kožar [**(@domenkozar)**](https://github.com/domenkozar)
- Taeer Bar-Yam [**(@Radvendii)**](https://github.com/Radvendii)
- Seth Flynn [**(@getchoo)**](https://github.com/getchoo)
- Robert Hensing [**(@roberth)**](https://github.com/roberth)
- Vladimir Panteleev [**(@CyberShadow)**](https://github.com/CyberShadow)
- bryango [**(@bryango)**](https://github.com/bryango)
- Henry [**(@cootshk)**](https://github.com/cootshk)
- Martin Joerg [**(@mjoerg)**](https://github.com/mjoerg)
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)

View File

@@ -224,25 +224,5 @@
"42688647+netadr@users.noreply.github.com": "netadr",
"matej.urbas@gmail.com": "urbas",
"ethanalexevans@gmail.com": "ethanavatar",
"greg.marti@gmail.com": "gmarti",
"arnout@bzzt.net": "raboof",
"vinayakankugoyal@gmail.com": "vinayakankugoyal",
"Radvendii@users.noreply.github.com": "Radvendii",
"jon@jh86.org": "jonhermansen",
"edef@edef.eu": "edef1c",
"pkpbynum@gmail.com": "pkpbynum",
"886074+teto@users.noreply.github.com": "teto",
"alex@adnab.me": "Alexis211",
"root@gws.fyi": "glittershark",
"me@m4rc3l.de": "MarcelCoding",
"taeer.bar-yam@bevuta.com": "Radvendii",
"martin.joerg@gmail.com": "mjoerg",
"git@cy.md": "CyberShadow",
"cootshk@duck.com": "cootshk",
"adam@dinwoodie.org": "me-and",
"domen@cachix.org": "domenkozar",
"alex.decious@gmail.com": "adeci",
"soumya.papanvk18@gmail.com": "neuralsorcerer",
"gdennis@anduril.com": null,
"graham.dennis@gmail.com": "GrahamDennis"
"greg.marti@gmail.com": "gmarti"
}

View File

@@ -196,21 +196,5 @@
"gmarti": "Gr\u00e9gory Marti",
"lovesegfault": "Bernardo Meurer",
"EphraimSiegfried": "Ephraim Siegfried",
"hgl": "Glen Huang",
"mjoerg": "Martin Joerg",
"Alexis211": "Alex Auvolat",
"domenkozar": "Domen Ko\u017ear",
"edef1c": "edef",
"cootshk": "Henry",
"raboof": "Arnout Engelen",
"pkpbynum": "Peter Bynum",
"glittershark": "Aspen Smith",
"MarcelCoding": "Marcel",
"teto": "Matthieu Coudron",
"jonhermansen": null,
"neuralsorcerer": "Soumyadip Sarkar",
"adeci": "Alex Decious",
"vinayakankugoyal": "Vinayak Goyal",
"me-and": "Adam Dinwoodie",
"GrahamDennis": "Graham Dennis"
"hgl": "Glen Huang"
}

View File

@@ -258,13 +258,10 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
# We use this shell with the local checkout, not unpackPhase.
src = null;
# Workaround https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
# Remove when gdb fix is rolled out everywhere.
separateDebugInfo = false;
mesonBuildType = "debugoptimized";
env = {
# For `make format`, to work without installing pre-commit
_NIX_PRE_COMMIT_HOOKS_CONFIG = "${(pkgs.formats.yaml { }).generate "pre-commit-config.yaml"

View File

@@ -48,7 +48,7 @@ public:
ref<GitRepo> openRepo()
{
return GitRepo::openRepo(tmpDir, {.create = true});
return GitRepo::openRepo(tmpDir, true, false);
}
std::string getRepoName() const

View File

@@ -203,19 +203,16 @@ static git_packbuilder_progress PACKBUILDER_PROGRESS_CHECK_INTERRUPT = &packBuil
} // extern "C"
static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options options)
static void initRepoAtomically(std::filesystem::path & path, bool bare)
{
if (pathExists(path.string()))
return;
if (!options.create)
throw Error("Git repository %s does not exist.", path);
std::filesystem::path tmpDir = createTempDir(path.parent_path());
AutoDelete delTmpDir(tmpDir, true);
Repository tmpRepo;
if (git_repository_init(Setter(tmpRepo), tmpDir.string().c_str(), options.bare))
if (git_repository_init(Setter(tmpRepo), tmpDir.string().c_str(), bare))
throw Error("creating Git repository %s: %s", path, git_error_last()->message);
try {
std::filesystem::rename(tmpDir, path);
@@ -237,7 +234,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
/** Location of the repository on disk. */
std::filesystem::path path;
Options options;
bool bare;
/**
* libgit2 repository. Note that new objects are not written to disk,
@@ -258,18 +255,18 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
*/
git_odb_backend * packBackend = nullptr;
GitRepoImpl(std::filesystem::path _path, Options _options)
GitRepoImpl(std::filesystem::path _path, bool create, bool bare, bool packfilesOnly = false)
: path(std::move(_path))
, options(_options)
, bare(bare)
{
initLibGit2();
initRepoAtomically(path, options);
initRepoAtomically(path, bare);
if (git_repository_open(Setter(repo), path.string().c_str()))
throw Error("opening Git repository %s: %s", path, git_error_last()->message);
ObjectDb odb;
if (options.packfilesOnly) {
if (packfilesOnly) {
/* Create a fresh object database because by default the repo also
loose object backends. We are not using any of those for the
tarball cache, but libgit2 still does a bunch of unnecessary
@@ -298,7 +295,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
if (git_odb_add_backend(odb.get(), mempackBackend, 999))
throw Error("adding mempack backend to Git object database: %s", git_error_last()->message);
if (options.packfilesOnly) {
if (packfilesOnly) {
if (git_repository_set_odb(repo.get(), odb.get()))
throw Error("setting Git object database: %s", git_error_last()->message);
}
@@ -369,7 +366,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
{
// TODO: as an optimization, it would be nice to include `this` in the pool.
return Pool<GitRepoImpl>(std::numeric_limits<size_t>::max(), [this]() -> ref<GitRepoImpl> {
return make_ref<GitRepoImpl>(path, options);
return make_ref<GitRepoImpl>(path, false, bare);
});
}
@@ -715,9 +712,9 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
}
};
ref<GitRepo> GitRepo::openRepo(const std::filesystem::path & path, GitRepo::Options options)
ref<GitRepo> GitRepo::openRepo(const std::filesystem::path & path, bool create, bool bare, bool packfilesOnly)
{
return make_ref<GitRepoImpl>(path, options);
return make_ref<GitRepoImpl>(path, create, bare, packfilesOnly);
}
/**
@@ -1430,12 +1427,8 @@ namespace fetchers {
ref<GitRepo> Settings::getTarballCache() const
{
/* v1: Had either only loose objects or thin packfiles referring to loose objects
* v2: Must have only packfiles with no loose objects. Should get repacked periodically
* for optimal packfiles.
*/
static auto repoDir = std::filesystem::path(getCacheDir()) / "tarball-cache-v2";
return GitRepo::openRepo(repoDir, {.create = true, .bare = true, .packfilesOnly = true});
static auto repoDir = std::filesystem::path(getCacheDir()) / "tarball-cache";
return GitRepo::openRepo(repoDir, /*create=*/true, /*bare=*/true, /*packfilesOnly=*/true);
}
} // namespace fetchers
@@ -1449,7 +1442,7 @@ GitRepo::WorkdirInfo GitRepo::getCachedWorkdirInfo(const std::filesystem::path &
if (i != cache->end())
return i->second;
}
auto workdirInfo = GitRepo::openRepo(path, {})->getWorkdirInfo();
auto workdirInfo = GitRepo::openRepo(path)->getWorkdirInfo();
_cache.lock()->emplace(path, workdirInfo);
return workdirInfo;
}

View File

@@ -637,6 +637,11 @@ struct GitInputScheme : InputScheme
url);
}
// If we don't check here for the path existence, then we can give libgit2 any directory
// and it will initialize them as git directories.
if (!pathExists(path)) {
throw Error("The path '%s' does not exist.", path);
}
repoInfo.location = std::filesystem::absolute(path);
} else {
if (url.scheme == "file")
@@ -698,7 +703,7 @@ struct GitInputScheme : InputScheme
if (auto res = cache->lookup(key))
return getIntAttr(*res, "lastModified");
auto lastModified = GitRepo::openRepo(repoDir, {})->getLastModified(rev);
auto lastModified = GitRepo::openRepo(repoDir)->getLastModified(rev);
cache->upsert(key, {{"lastModified", lastModified}});
@@ -721,7 +726,7 @@ struct GitInputScheme : InputScheme
Activity act(
*logger, lvlChatty, actUnknown, fmt("getting Git revision count of '%s'", repoInfo.locationToArg()));
auto revCount = GitRepo::openRepo(repoDir, {})->getRevCount(rev);
auto revCount = GitRepo::openRepo(repoDir)->getRevCount(rev);
cache->upsert(key, Attrs{{"revCount", revCount}});
@@ -732,7 +737,7 @@ struct GitInputScheme : InputScheme
{
auto head = std::visit(
overloaded{
[&](const std::filesystem::path & path) { return GitRepo::openRepo(path, {})->getWorkdirRef(); },
[&](const std::filesystem::path & path) { return GitRepo::openRepo(path)->getWorkdirRef(); },
[&](const ParsedURL & url) { return readHeadCached(url.to_string(), shallow); }},
repoInfo.location);
if (!head) {
@@ -790,7 +795,7 @@ struct GitInputScheme : InputScheme
if (auto repoPath = repoInfo.getPath()) {
repoDir = *repoPath;
if (!input.getRev())
input.attrs.insert_or_assign("rev", GitRepo::openRepo(repoDir, {})->resolveRef(ref).gitRev());
input.attrs.insert_or_assign("rev", GitRepo::openRepo(repoDir)->resolveRef(ref).gitRev());
} else {
auto repoUrl = std::get<ParsedURL>(repoInfo.location);
std::filesystem::path cacheDir = getCachePath(repoUrl.to_string(), shallow);
@@ -800,7 +805,7 @@ struct GitInputScheme : InputScheme
std::filesystem::create_directories(cacheDir.parent_path());
PathLocks cacheDirLock({cacheDir.string()});
auto repo = GitRepo::openRepo(cacheDir, {.create = true, .bare = true});
auto repo = GitRepo::openRepo(cacheDir, true, true);
// We need to set the origin so resolving submodule URLs works
repo->setRemote("origin", repoUrl.to_string());
@@ -871,7 +876,7 @@ struct GitInputScheme : InputScheme
// the remainder
}
auto repo = GitRepo::openRepo(repoDir, {});
auto repo = GitRepo::openRepo(repoDir);
auto isShallow = repo->isShallow();
@@ -958,7 +963,7 @@ struct GitInputScheme : InputScheme
for (auto & submodule : repoInfo.workdirInfo.submodules)
repoInfo.workdirInfo.files.insert(submodule.path);
auto repo = GitRepo::openRepo(repoPath, {});
auto repo = GitRepo::openRepo(repoPath, false, false);
auto exportIgnore = getExportIgnoreAttr(input);
@@ -998,7 +1003,7 @@ struct GitInputScheme : InputScheme
}
if (!repoInfo.workdirInfo.isDirty) {
auto repo = GitRepo::openRepo(repoPath, {});
auto repo = GitRepo::openRepo(repoPath);
if (auto ref = repo->getWorkdirRef())
input.attrs.insert_or_assign("ref", *ref);

View File

@@ -32,14 +32,8 @@ struct GitRepo
{
virtual ~GitRepo() {}
struct Options
{
bool create = false;
bool bare = false;
bool packfilesOnly = false;
};
static ref<GitRepo> openRepo(const std::filesystem::path & path, Options options);
static ref<GitRepo>
openRepo(const std::filesystem::path & path, bool create = false, bool bare = false, bool packfilesOnly = false);
virtual uint64_t getRevCount(const Hash & rev) = 0;

View File

@@ -2,7 +2,7 @@
#include "nix/store/derivations.hh"
#include "nix/store/store-api.hh"
#include "nix/util/experimental-features.hh"
#include "nix/util/tests/test-data.hh"
#include "nix/util/environment-variables.hh"
#include "nix/store/store-open.hh"
#include <fstream>
#include <sstream>
@@ -50,7 +50,11 @@ static void BM_UnparseRealDerivationFile(benchmark::State & state, const std::st
}
// Register benchmarks for actual test derivation files if they exist
BENCHMARK_CAPTURE(BM_ParseRealDerivationFile, hello, (getUnitTestData() / "derivation/hello.drv").string());
BENCHMARK_CAPTURE(BM_ParseRealDerivationFile, firefox, (getUnitTestData() / "derivation/firefox.drv").string());
BENCHMARK_CAPTURE(BM_UnparseRealDerivationFile, hello, (getUnitTestData() / "derivation/hello.drv").string());
BENCHMARK_CAPTURE(BM_UnparseRealDerivationFile, firefox, (getUnitTestData() / "derivation/firefox.drv").string());
BENCHMARK_CAPTURE(
BM_ParseRealDerivationFile, hello, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/hello.drv");
BENCHMARK_CAPTURE(
BM_ParseRealDerivationFile, firefox, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/firefox.drv");
BENCHMARK_CAPTURE(
BM_UnparseRealDerivationFile, hello, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/hello.drv");
BENCHMARK_CAPTURE(
BM_UnparseRealDerivationFile, firefox, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/firefox.drv");

View File

@@ -8,7 +8,6 @@
#include "nix/store/tests/nix_api_store.hh"
#include "nix/store/globals.hh"
#include "nix/util/tests/string_callback.hh"
#include "nix/util/tests/test-data.hh"
#include "nix/util/url.hh"
#include "store-tests-config.hh"
@@ -303,7 +302,7 @@ public:
store = open_local_store();
std::filesystem::path unitTestData = nix::getUnitTestData();
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
std::stringstream buffer;
buffer << t.rdbuf();
@@ -358,7 +357,7 @@ TEST_F(nix_api_store_test_base, build_from_json)
auto * store = open_local_store();
std::filesystem::path unitTestData = nix::getUnitTestData();
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
std::stringstream buffer;
@@ -405,7 +404,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_invalid_system)
auto * store = open_local_store();
std::filesystem::path unitTestData = nix::getUnitTestData();
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
std::stringstream buffer;
buffer << t.rdbuf();
@@ -450,7 +449,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_builder_fails)
auto * store = open_local_store();
std::filesystem::path unitTestData = nix::getUnitTestData();
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
std::stringstream buffer;
buffer << t.rdbuf();
@@ -495,7 +494,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_builder_no_output)
auto * store = open_local_store();
std::filesystem::path unitTestData = nix::getUnitTestData();
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
std::stringstream buffer;
buffer << t.rdbuf();
@@ -871,7 +870,7 @@ TEST_F(NixApiStoreTestWithRealisedPath, nix_store_get_fs_closure_error_propagati
*/
static std::string load_json_from_test_data(const char * filename)
{
std::filesystem::path unitTestData = nix::getUnitTestData();
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::ifstream t{unitTestData / filename};
std::stringstream buffer;
buffer << t.rdbuf();

View File

@@ -4,15 +4,15 @@
# include <aws/crt/Types.h>
# include "nix/store/s3-url.hh"
# include "nix/util/finally.hh"
# include "nix/util/logging.hh"
# include "nix/util/url.hh"
# include "nix/util/util.hh"
# include <aws/crt/Api.h>
# include <aws/crt/auth/Credentials.h>
# include <aws/crt/io/Bootstrap.h>
// C library headers for SSO provider support
# include <aws/auth/credentials.h>
# include <boost/unordered/concurrent_flat_map.hpp>
# include <chrono>
@@ -30,46 +30,6 @@ AwsAuthError::AwsAuthError(int errorCode)
namespace {
/**
* Helper function to wrap a C credentials provider in the C++ interface.
* This replicates the static s_CreateWrappedProvider from aws-crt-cpp.
*/
static std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider> createWrappedProvider(
aws_credentials_provider * rawProvider, Aws::Crt::Allocator * allocator = Aws::Crt::ApiAllocator())
{
if (rawProvider == nullptr) {
return nullptr;
}
auto provider = Aws::Crt::MakeShared<Aws::Crt::Auth::CredentialsProvider>(allocator, rawProvider, allocator);
return std::static_pointer_cast<Aws::Crt::Auth::ICredentialsProvider>(provider);
}
/**
* Create an SSO credentials provider using the C library directly.
* The C++ wrapper doesn't expose SSO, so we call the C library and wrap the result.
* Returns nullptr if SSO provider creation fails (e.g., profile doesn't have SSO config).
*/
static std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider> createSSOProvider(
const std::string & profileName,
Aws::Crt::Io::ClientBootstrap * bootstrap,
Aws::Crt::Io::TlsContext * tlsContext,
Aws::Crt::Allocator * allocator = Aws::Crt::ApiAllocator())
{
aws_credentials_provider_sso_options options;
AWS_ZERO_STRUCT(options);
options.bootstrap = bootstrap->GetUnderlyingHandle();
options.tls_ctx = tlsContext ? tlsContext->GetUnderlyingHandle() : nullptr;
if (!profileName.empty()) {
options.profile_name_override = aws_byte_cursor_from_c_str(profileName.c_str());
}
// Create the SSO provider - will return nullptr if SSO isn't configured for this profile
// createWrappedProvider handles nullptr gracefully
return createWrappedProvider(aws_credentials_provider_new_sso(allocator, &options), allocator);
}
static AwsCredentials getCredentialsFromProvider(std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider> provider)
{
if (!provider || !provider->IsValid()) {
@@ -131,22 +91,6 @@ public:
logLevel = Aws::Crt::LogLevel::Warn;
}
apiHandle.InitializeLogging(logLevel, stderr);
// Create a shared TLS context for SSO (required for HTTPS connections)
auto allocator = Aws::Crt::ApiAllocator();
auto tlsCtxOptions = Aws::Crt::Io::TlsContextOptions::InitDefaultClient(allocator);
tlsContext =
std::make_shared<Aws::Crt::Io::TlsContext>(tlsCtxOptions, Aws::Crt::Io::TlsMode::CLIENT, allocator);
if (!tlsContext || !*tlsContext) {
warn("failed to create TLS context for AWS SSO; SSO authentication will be unavailable");
tlsContext = nullptr;
}
// Get bootstrap (lives as long as apiHandle)
bootstrap = Aws::Crt::ApiHandle::GetOrCreateStaticDefaultClientBootstrap();
if (!bootstrap) {
throw AwsAuthError("failed to create AWS client bootstrap");
}
}
AwsCredentials getCredentialsRaw(const std::string & profile);
@@ -167,8 +111,6 @@ public:
private:
Aws::Crt::ApiHandle apiHandle;
std::shared_ptr<Aws::Crt::Io::TlsContext> tlsContext;
Aws::Crt::Io::ClientBootstrap * bootstrap;
boost::concurrent_flat_map<std::string, std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider>>
credentialProviderCache;
};
@@ -176,58 +118,23 @@ private:
std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider>
AwsCredentialProviderImpl::createProviderForProfile(const std::string & profile)
{
// profileDisplayName is only used for debug logging - SDK uses its default profile
// when ProfileNameOverride is not set
const char * profileDisplayName = profile.empty() ? "(default)" : profile.c_str();
debug(
"[pid=%d] creating new AWS credential provider for profile '%s'",
getpid(),
profile.empty() ? "(default)" : profile.c_str());
debug("[pid=%d] creating new AWS credential provider for profile '%s'", getpid(), profileDisplayName);
// Build a custom credential chain: Environment → SSO → Profile → IMDS
// This works for both default and named profiles, ensuring consistent behavior
// including SSO support and proper TLS context for STS-based role assumption.
Aws::Crt::Auth::CredentialsProviderChainConfig chainConfig;
auto allocator = Aws::Crt::ApiAllocator();
auto addProviderToChain = [&](std::string_view name, auto createProvider) {
if (auto provider = createProvider()) {
chainConfig.Providers.push_back(provider);
debug("Added AWS %s Credential Provider to chain for profile '%s'", name, profileDisplayName);
} else {
debug("Skipped AWS %s Credential Provider for profile '%s'", name, profileDisplayName);
}
};
// 1. Environment variables (highest priority)
addProviderToChain("Environment", [&]() {
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderEnvironment(allocator);
});
// 2. SSO provider (try it, will fail gracefully if not configured)
if (tlsContext) {
addProviderToChain("SSO", [&]() { return createSSOProvider(profile, bootstrap, tlsContext.get(), allocator); });
} else {
debug("Skipped AWS SSO Credential Provider for profile '%s': TLS context unavailable", profileDisplayName);
if (profile.empty()) {
Aws::Crt::Auth::CredentialsProviderChainDefaultConfig config;
config.Bootstrap = Aws::Crt::ApiHandle::GetOrCreateStaticDefaultClientBootstrap();
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderChainDefault(config);
}
// 3. Profile provider (for static credentials and role_arn/source_profile with STS)
addProviderToChain("Profile", [&]() {
Aws::Crt::Auth::CredentialsProviderProfileConfig profileConfig;
profileConfig.Bootstrap = bootstrap;
profileConfig.TlsContext = tlsContext.get();
if (!profile.empty()) {
profileConfig.ProfileNameOverride = Aws::Crt::ByteCursorFromCString(profile.c_str());
}
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderProfile(profileConfig, allocator);
});
// 4. IMDS provider (for EC2 instances, lowest priority)
addProviderToChain("IMDS", [&]() {
Aws::Crt::Auth::CredentialsProviderImdsConfig imdsConfig;
imdsConfig.Bootstrap = bootstrap;
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderImds(imdsConfig, allocator);
});
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderChain(chainConfig, allocator);
Aws::Crt::Auth::CredentialsProviderProfileConfig config;
config.Bootstrap = Aws::Crt::ApiHandle::GetOrCreateStaticDefaultClientBootstrap();
// This is safe because the underlying C library will copy this string
// c.f. https://github.com/awslabs/aws-c-auth/blob/main/source/credentials_provider_profile.c#L220
config.ProfileNameOverride = Aws::Crt::ByteCursorFromCString(profile.c_str());
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderProfile(config);
}
AwsCredentials AwsCredentialProviderImpl::getCredentialsRaw(const std::string & profile)

View File

@@ -71,6 +71,14 @@ void DerivationBuildingGoal::killChild()
#endif
}
void DerivationBuildingGoal::timedOut(Error && ex)
{
killChild();
// We're not inside a coroutine, hence we can't use co_return here.
// Thus we ignore the return value.
[[maybe_unused]] Done _ = doneFailure({BuildResult::Failure::TimedOut, std::move(ex)});
}
std::string showKnownOutputs(const StoreDirConfig & store, const Derivation & drv)
{
std::string msg;
@@ -435,20 +443,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
if (useHook) {
buildResult.startTime = time(0); // inexact
started();
while (true) {
auto event = co_await WaitForChildEvent{};
if (auto * output = std::get_if<ChildOutput>(&event)) {
co_await processChildOutput(output->fd, output->data);
} else if (std::get_if<ChildEOF>(&event)) {
if (!currentLogLine.empty())
flushLine();
break;
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
killChild();
co_return doneFailure(std::move(*timeout));
}
}
co_await Suspend{};
#ifndef _WIN32
assert(hook);
@@ -669,20 +664,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
worker.childStarted(shared_from_this(), {builderOut}, true, true);
started();
while (true) {
auto event = co_await WaitForChildEvent{};
if (auto * output = std::get_if<ChildOutput>(&event)) {
co_await processChildOutput(output->fd, output->data);
} else if (std::get_if<ChildEOF>(&event)) {
if (!currentLogLine.empty())
flushLine();
break;
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
killChild();
co_return doneFailure(std::move(*timeout));
}
}
co_await Suspend{};
trace("build done");
@@ -1015,7 +997,7 @@ bool DerivationBuildingGoal::isReadDesc(Descriptor fd)
#endif
}
Goal::Co DerivationBuildingGoal::processChildOutput(Descriptor fd, std::string_view data)
void DerivationBuildingGoal::handleChildOutput(Descriptor fd, std::string_view data)
{
// local & `ssh://`-builds are dealt with here.
auto isWrittenToLog = isReadDesc(fd);
@@ -1023,11 +1005,14 @@ Goal::Co DerivationBuildingGoal::processChildOutput(Descriptor fd, std::string_v
logSize += data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
killChild();
co_return doneFailure(BuildError(
// We're not inside a coroutine, hence we can't use co_return here.
// Thus we ignore the return value.
[[maybe_unused]] Done _ = doneFailure(BuildError(
BuildResult::Failure::LogLimitExceeded,
"%s killed after writing more than %d bytes of log output",
getName(),
settings.maxLogSize));
return;
}
for (auto c : data)
@@ -1080,7 +1065,13 @@ Goal::Co DerivationBuildingGoal::processChildOutput(Descriptor fd, std::string_v
currentHookLine += c;
}
#endif
co_return Return{};
}
void DerivationBuildingGoal::handleEOF(Descriptor fd)
{
if (!currentLogLine.empty())
flushLine();
worker.wakeUp(shared_from_this());
}
void DerivationBuildingGoal::flushLine()

View File

@@ -66,16 +66,7 @@ Goal::Co DrvOutputSubstitutionGoal::init()
true,
false);
while (true) {
auto event = co_await WaitForChildEvent{};
if (std::get_if<ChildOutput>(&event)) {
// Doesn't process child output
} else if (std::get_if<ChildEOF>(&event)) {
break;
} else if (std::get_if<TimedOut>(&event)) {
unreachable();
}
}
co_await Suspend{};
worker.childTerminated(this);
@@ -158,4 +149,9 @@ std::string DrvOutputSubstitutionGoal::key()
return "a$" + std::string(id.to_string());
}
void DrvOutputSubstitutionGoal::handleEOF(Descriptor fd)
{
worker.wakeUp(shared_from_this());
}
} // namespace nix

View File

@@ -4,58 +4,8 @@
namespace nix {
TimedOut::TimedOut(time_t maxDuration)
: BuildError(BuildResult::Failure::TimedOut, "timed out after %1% seconds", maxDuration)
, maxDuration(maxDuration)
{
}
using Co = nix::Goal::Co;
using promise_type = nix::Goal::promise_type;
using ChildEvents = decltype(promise_type::childEvents);
void ChildEvents::pushChildEvent(ChildOutput event)
{
if (childTimeout)
return; // Already timed out, ignore
childOutputs.push(std::move(event));
}
void ChildEvents::pushChildEvent(ChildEOF event)
{
if (childTimeout)
return; // Already timed out, ignore
assert(!childEOF);
childEOF = std::move(event);
}
void ChildEvents::pushChildEvent(TimedOut event)
{
// Timeout is immediate - flush pending events
childOutputs = {};
childEOF.reset();
childTimeout = std::move(event);
}
bool ChildEvents::hasChildEvent() const
{
return !childOutputs.empty() || childEOF || childTimeout;
}
Goal::ChildEvent ChildEvents::popChildEvent()
{
if (!childOutputs.empty()) {
auto event = std::move(childOutputs.front());
childOutputs.pop();
return event;
}
if (childEOF)
return *std::exchange(childEOF, std::nullopt);
if (childTimeout)
return *std::exchange(childTimeout, std::nullopt);
unreachable();
}
using handle_type = nix::Goal::handle_type;
using Suspend = nix::Goal::Suspend;
@@ -256,27 +206,6 @@ void Goal::work()
assert(top_co || exitCode != ecBusy);
}
void Goal::handleChildOutput(Descriptor fd, std::string_view data)
{
assert(top_co);
top_co->handle.promise().childEvents.pushChildEvent(ChildOutput{fd, std::string{data}});
worker.wakeUp(shared_from_this());
}
void Goal::handleEOF(Descriptor fd)
{
assert(top_co);
top_co->handle.promise().childEvents.pushChildEvent(ChildEOF{fd});
worker.wakeUp(shared_from_this());
}
void Goal::timedOut(TimedOut && ex)
{
assert(top_co);
top_co->handle.promise().childEvents.pushChildEvent(std::move(ex));
worker.wakeUp(shared_from_this());
}
Goal::Co Goal::yield()
{
worker.wakeUp(shared_from_this());

View File

@@ -258,16 +258,7 @@ Goal::Co PathSubstitutionGoal::tryToRun(
true,
false);
while (true) {
auto event = co_await WaitForChildEvent{};
if (std::get_if<ChildOutput>(&event)) {
// Substitution doesn't process child output
} else if (std::get_if<ChildEOF>(&event)) {
break;
} else if (std::get_if<TimedOut>(&event)) {
unreachable(); // Substitution doesn't use timeouts
}
}
co_await Suspend{};
trace("substitute finished");
@@ -319,6 +310,11 @@ Goal::Co PathSubstitutionGoal::tryToRun(
co_return doneSuccess(BuildResult::Success::Substituted);
}
void PathSubstitutionGoal::handleEOF(Descriptor fd)
{
worker.wakeUp(shared_from_this());
}
void PathSubstitutionGoal::cleanup()
{
try {

View File

@@ -479,13 +479,14 @@ void Worker::waitForInput()
if (goal->exitCode == Goal::ecBusy && 0 != settings.maxSilentTime && j->respectTimeouts
&& after - j->lastOutput >= std::chrono::seconds(settings.maxSilentTime)) {
goal->timedOut(TimedOut(settings.maxSilentTime));
goal->timedOut(
Error("%1% timed out after %2% seconds of silence", goal->getName(), settings.maxSilentTime));
}
else if (
goal->exitCode == Goal::ecBusy && 0 != settings.buildTimeout && j->respectTimeouts
&& after - j->timeStarted >= std::chrono::seconds(settings.buildTimeout)) {
goal->timedOut(TimedOut(settings.buildTimeout));
goal->timedOut(Error("%1% timed out after %2% seconds", goal->getName(), settings.buildTimeout));
}
}

View File

@@ -341,8 +341,12 @@ DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
if (parsed) {
auto * e = optionalValueAt(parsed->structuredAttrs, "exportReferencesGraph");
if (!e || !e->is_object())
if (!e)
return ret;
if (!e->is_object()) {
warn("'exportReferencesGraph' in structured attrs is not a JSON object, ignoring");
return ret;
}
for (auto & [key, storePathsJson] : getObject(*e)) {
StringSet ss;
flatten(storePathsJson, ss);

View File

@@ -1489,6 +1489,8 @@ adl_serializer<DerivationOutput>::from_json(const json & _json, const Experiment
}
}
static unsigned constexpr expectedJsonVersionDerivation = 4;
void adl_serializer<Derivation>::to_json(json & res, const Derivation & d)
{
res = nlohmann::json::object();

View File

@@ -41,27 +41,9 @@ FileTransferSettings fileTransferSettings;
static GlobalConfig::Register rFileTransferSettings(&fileTransferSettings);
namespace {
using curlSList = std::unique_ptr<::curl_slist, decltype([](::curl_slist * list) { ::curl_slist_free_all(list); })>;
using curlMulti = std::unique_ptr<::CURLM, decltype([](::CURLM * multi) { ::curl_multi_cleanup(multi); })>;
struct curlMultiError : Error
{
::CURLMcode code;
curlMultiError(::CURLMcode code)
: Error{"unexpected curl multi error: %s", ::curl_multi_strerror(code)}
{
assert(code != CURLM_OK);
}
};
} // namespace
struct curlFileTransfer : public FileTransfer
{
curlMulti curlm;
CURLM * curlm = 0;
std::random_device rd;
std::mt19937 mt19937;
@@ -87,7 +69,7 @@ struct curlFileTransfer : public FileTransfer
has been reached. */
std::chrono::steady_clock::time_point embargo;
curlSList requestHeaders;
struct curl_slist * requestHeaders = 0;
std::string encoding;
@@ -110,15 +92,6 @@ struct curlFileTransfer : public FileTransfer
return httpStatus;
}
void appendHeaders(const std::string & header)
{
curlSList tmpSList = curlSList(::curl_slist_append(requestHeaders.get(), requireCString(header)));
if (!tmpSList)
throw std::bad_alloc();
requestHeaders.release();
requestHeaders = std::move(tmpSList);
}
TransferItem(
curlFileTransfer & fileTransfer,
const FileTransferRequest & request,
@@ -158,13 +131,13 @@ struct curlFileTransfer : public FileTransfer
{
result.urls.push_back(request.uri.to_string());
appendHeaders("Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
requestHeaders = curl_slist_append(requestHeaders, "Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
if (!request.expectedETag.empty())
appendHeaders("If-None-Match: " + request.expectedETag);
requestHeaders = curl_slist_append(requestHeaders, ("If-None-Match: " + request.expectedETag).c_str());
if (!request.mimeType.empty())
appendHeaders("Content-Type: " + request.mimeType);
requestHeaders = curl_slist_append(requestHeaders, ("Content-Type: " + request.mimeType).c_str());
for (auto it = request.headers.begin(); it != request.headers.end(); ++it) {
appendHeaders(fmt("%s: %s", it->first, it->second));
requestHeaders = curl_slist_append(requestHeaders, fmt("%s: %s", it->first, it->second).c_str());
}
}
@@ -172,9 +145,11 @@ struct curlFileTransfer : public FileTransfer
{
if (req) {
if (active)
curl_multi_remove_handle(fileTransfer.curlm.get(), req);
curl_multi_remove_handle(fileTransfer.curlm, req);
curl_easy_cleanup(req);
}
if (requestHeaders)
curl_slist_free_all(requestHeaders);
try {
if (!done)
fail(FileTransferError(
@@ -373,7 +348,7 @@ struct curlFileTransfer : public FileTransfer
return ((TransferItem *) userp)->readCallback(buffer, size, nitems);
}
#if !defined(_WIN32)
#if !defined(_WIN32) && LIBCURL_VERSION_NUM >= 0x071000
static int cloexec_callback(void *, curl_socket_t curlfd, curlsocktype purpose)
{
unix::closeOnExec(curlfd);
@@ -436,11 +411,15 @@ struct curlFileTransfer : public FileTransfer
("curl/" LIBCURL_VERSION " Nix/" + nixVersion
+ (fileTransferSettings.userAgentSuffix != "" ? " " + fileTransferSettings.userAgentSuffix.get() : ""))
.c_str());
#if LIBCURL_VERSION_NUM >= 0x072b00
curl_easy_setopt(req, CURLOPT_PIPEWAIT, 1);
#endif
#if LIBCURL_VERSION_NUM >= 0x072f00
if (fileTransferSettings.enableHttp2)
curl_easy_setopt(req, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2TLS);
else
curl_easy_setopt(req, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
#endif
curl_easy_setopt(req, CURLOPT_WRITEFUNCTION, TransferItem::writeCallbackWrapper);
curl_easy_setopt(req, CURLOPT_WRITEDATA, this);
curl_easy_setopt(req, CURLOPT_HEADERFUNCTION, TransferItem::headerCallbackWrapper);
@@ -450,7 +429,7 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_XFERINFODATA, this);
curl_easy_setopt(req, CURLOPT_NOPROGRESS, 0);
curl_easy_setopt(req, CURLOPT_HTTPHEADER, requestHeaders.get());
curl_easy_setopt(req, CURLOPT_HTTPHEADER, requestHeaders);
if (settings.downloadSpeed.get() > 0)
curl_easy_setopt(req, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t) (settings.downloadSpeed.get() * 1024));
@@ -480,9 +459,10 @@ struct curlFileTransfer : public FileTransfer
if (settings.caFile != "")
curl_easy_setopt(req, CURLOPT_CAINFO, settings.caFile.get().c_str());
#if !defined(_WIN32)
#if !defined(_WIN32) && LIBCURL_VERSION_NUM >= 0x071000
curl_easy_setopt(req, CURLOPT_SOCKOPTFUNCTION, cloexec_callback);
#endif
curl_easy_setopt(req, CURLOPT_CONNECTTIMEOUT, fileTransferSettings.connectTimeout.get());
curl_easy_setopt(req, CURLOPT_LOW_SPEED_LIMIT, 1L);
@@ -714,6 +694,13 @@ struct curlFileTransfer : public FileTransfer
Sync<State> state_;
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
/* We can't use a std::condition_variable to wake up the curl
thread, because it only monitors file descriptors. So use a
pipe instead. */
Pipe wakeupPipe;
#endif
std::thread workerThread;
curlFileTransfer()
@@ -722,35 +709,43 @@ struct curlFileTransfer : public FileTransfer
static std::once_flag globalInit;
std::call_once(globalInit, curl_global_init, CURL_GLOBAL_ALL);
curlm = curlMulti(curl_multi_init());
curlm = curl_multi_init();
curl_multi_setopt(curlm.get(), CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
curl_multi_setopt(curlm.get(), CURLMOPT_MAX_TOTAL_CONNECTIONS, fileTransferSettings.httpConnections.get());
#if LIBCURL_VERSION_NUM >= 0x072b00 // Multiplex requires >= 7.43.0
curl_multi_setopt(curlm, CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
#endif
#if LIBCURL_VERSION_NUM >= 0x071e00 // Max connections requires >= 7.30.0
curl_multi_setopt(curlm, CURLMOPT_MAX_TOTAL_CONNECTIONS, fileTransferSettings.httpConnections.get());
#endif
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
wakeupPipe.create();
fcntl(wakeupPipe.readSide.get(), F_SETFL, O_NONBLOCK);
#endif
workerThread = std::thread([&]() { workerThreadEntry(); });
}
~curlFileTransfer()
{
try {
stopWorkerThread();
} catch (...) {
ignoreExceptionInDestructor();
}
stopWorkerThread();
workerThread.join();
if (curlm)
curl_multi_cleanup(curlm);
}
void stopWorkerThread()
{
/* Signal the worker thread to exit. */
state_.lock()->quit();
wakeupMulti();
}
void wakeupMulti()
{
if (auto ec = ::curl_multi_wakeup(curlm.get()))
throw curlMultiError(ec);
{
auto state(state_.lock());
state->quit();
}
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
writeFull(wakeupPipe.writeSide.get(), " ", false);
#endif
}
void workerThreadMain()
@@ -780,25 +775,32 @@ struct curlFileTransfer : public FileTransfer
/* Let curl do its thing. */
int running;
CURLMcode mc = curl_multi_perform(curlm.get(), &running);
CURLMcode mc = curl_multi_perform(curlm, &running);
if (mc != CURLM_OK)
throw nix::Error("unexpected error from curl_multi_perform(): %s", curl_multi_strerror(mc));
/* Set the promises of any finished requests. */
CURLMsg * msg;
int left;
while ((msg = curl_multi_info_read(curlm.get(), &left))) {
while ((msg = curl_multi_info_read(curlm, &left))) {
if (msg->msg == CURLMSG_DONE) {
auto i = items.find(msg->easy_handle);
assert(i != items.end());
i->second->finish(msg->data.result);
curl_multi_remove_handle(curlm.get(), i->second->req);
curl_multi_remove_handle(curlm, i->second->req);
i->second->active = false;
items.erase(i);
}
}
/* Wait for activity, including wakeup events. */
int numfds = 0;
struct curl_waitfd extraFDs[1];
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
extraFDs[0].fd = wakeupPipe.readSide.get();
extraFDs[0].events = CURL_WAIT_POLLIN;
extraFDs[0].revents = 0;
#endif
long maxSleepTimeMs = items.empty() ? 10000 : 100;
auto sleepTimeMs = nextWakeup != std::chrono::steady_clock::time_point()
? std::max(
@@ -807,14 +809,23 @@ struct curlFileTransfer : public FileTransfer
nextWakeup - std::chrono::steady_clock::now())
.count())
: maxSleepTimeMs;
int numfds = 0;
mc = curl_multi_poll(curlm.get(), nullptr, 0, sleepTimeMs, &numfds);
vomit("download thread waiting for %d ms", sleepTimeMs);
mc = curl_multi_wait(curlm, extraFDs, 1, sleepTimeMs, &numfds);
if (mc != CURLM_OK)
throw curlMultiError(mc);
throw nix::Error("unexpected error from curl_multi_wait(): %s", curl_multi_strerror(mc));
nextWakeup = std::chrono::steady_clock::time_point();
/* Add new curl requests from the incoming requests queue,
except for requests that are embargoed (waiting for a
retry timeout to expire). */
if (extraFDs[0].revents & CURL_WAIT_POLLIN) {
char buf[1024];
auto res = read(extraFDs[0].fd, buf, sizeof(buf));
if (res == -1 && errno != EINTR)
throw SysError("reading curl wakeup socket");
}
std::vector<std::shared_ptr<TransferItem>> incoming;
auto now = std::chrono::steady_clock::now();
@@ -837,7 +848,7 @@ struct curlFileTransfer : public FileTransfer
for (auto & item : incoming) {
debug("starting %s of %s", item->request.noun(), item->request.uri);
item->init();
curl_multi_add_handle(curlm.get(), item->req);
curl_multi_add_handle(curlm, item->req);
item->active = true;
items[item->req] = item;
}
@@ -888,8 +899,10 @@ struct curlFileTransfer : public FileTransfer
throw nix::Error("cannot enqueue download request because the download thread is shutting down");
state->incoming.push(item);
}
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
writeFull(wakeupPipe.writeSide.get(), " ");
#endif
wakeupMulti();
return ItemHandle(static_cast<Item &>(*item));
}
@@ -909,7 +922,9 @@ struct curlFileTransfer : public FileTransfer
{
auto state(state_.lock());
state->unpause.push_back(std::move(item));
wakeupMulti();
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
writeFull(wakeupPipe.writeSide.get(), " ");
#endif
}
void unpauseTransfer(ItemHandle handle) override

View File

@@ -79,7 +79,7 @@ struct DerivationBuilderParams
*/
const StorePathSet & inputPaths;
const std::map<std::string, InitialOutput> initialOutputs;
const std::map<std::string, InitialOutput> & initialOutputs;
const BuildMode & buildMode;

View File

@@ -100,6 +100,8 @@ private:
std::map<ActivityId, Activity> builderActivities;
void timedOut(Error && ex) override;
std::string key() override;
/**
@@ -127,10 +129,10 @@ private:
bool isReadDesc(Descriptor fd);
/**
* Process output from a child process.
* Callback used by the worker to write to the log.
*/
Co processChildOutput(Descriptor fd, std::string_view data);
void handleChildOutput(Descriptor fd, std::string_view data) override;
void handleEOF(Descriptor fd) override;
void flushLine();
/**

View File

@@ -52,6 +52,11 @@ struct DerivationGoal : public Goal
bool storeDerivation);
~DerivationGoal() = default;
void timedOut(Error && ex) override
{
unreachable();
};
std::string key() override;
JobCategory jobCategory() const override

View File

@@ -43,6 +43,8 @@ struct DerivationResolutionGoal : public Goal
*/
std::unique_ptr<std::pair<StorePath, BasicDerivation>> resolvedDrv;
void timedOut(Error && ex) override {}
private:
/**

View File

@@ -109,6 +109,8 @@ struct DerivationTrampolineGoal : public Goal
virtual ~DerivationTrampolineGoal();
void timedOut(Error && ex) override {}
std::string key() override;
JobCategory jobCategory() const override

View File

@@ -33,8 +33,15 @@ public:
Co init();
void timedOut(Error && ex) override
{
unreachable();
};
std::string key() override;
void handleEOF(Descriptor fd) override;
JobCategory jobCategory() const override
{
return JobCategory::Substitution;

View File

@@ -5,18 +5,9 @@
#include "nix/store/build-result.hh"
#include <coroutine>
#include <queue>
#include <variant>
namespace nix {
struct TimedOut : BuildError
{
time_t maxDuration;
TimedOut(time_t maxDuration);
};
/**
* Forward definition.
*/
@@ -147,29 +138,6 @@ public:
friend Goal;
};
/**
* Event types for child process communication, delivered via coroutines.
*/
struct ChildOutput
{
Descriptor fd;
std::string data;
};
struct ChildEOF
{
Descriptor fd;
};
using ChildEvent = std::variant<ChildOutput, ChildEOF, TimedOut>;
/**
* Tag type for `co_await`-ing child events.
* Returns a `ChildEvent` when resumed.
*/
struct WaitForChildEvent
{};
// forward declaration of promise_type, see below
struct promise_type;
@@ -308,28 +276,6 @@ public:
*/
bool alive = true;
class
{
/**
* Structured queue of child events:
* - outputs: stream of data from child
* - eof: optional end-of-stream marker
* - timeout: optional timeout that flushes/overrides other events
*/
std::queue<ChildOutput> childOutputs;
std::optional<ChildEOF> childEOF;
std::optional<TimedOut> childTimeout;
public:
void pushChildEvent(ChildOutput event);
void pushChildEvent(ChildEOF event);
void pushChildEvent(TimedOut event);
bool hasChildEvent() const;
ChildEvent popChildEvent();
} childEvents;
/**
* The awaiter used by @ref final_suspend.
*/
@@ -423,66 +369,13 @@ public:
return static_cast<Co &&>(co);
}
/**
* Awaiter for @ref Suspend. Always suspends, but asserts
* there are no pending child events (those should be
* consumed first via @ref WaitForChildEvent).
*/
struct SuspendAwaiter
{
promise_type & promise;
bool await_ready()
{
assert(!promise.childEvents.hasChildEvent());
return false;
}
void await_suspend(handle_type) {}
void await_resume() {}
};
/**
* Allows awaiting a @ref Suspend.
* Always suspends.
*/
SuspendAwaiter await_transform(Suspend)
std::suspend_always await_transform(Suspend)
{
return SuspendAwaiter{*this};
};
/**
* Awaiter for child events. Suspends and returns the
* pending child event when resumed.
*/
struct ChildEventAwaiter
{
handle_type handle;
bool await_ready()
{
return handle && handle.promise().childEvents.hasChildEvent();
}
void await_suspend(handle_type h)
{
handle = h;
}
ChildEvent await_resume()
{
assert(handle);
return handle.promise().childEvents.popChildEvent();
}
};
/**
* Allows awaiting child events (output, EOF, timeout).
*/
ChildEventAwaiter await_transform(WaitForChildEvent)
{
return ChildEventAwaiter{handle_type::from_promise(*this)};
return {};
};
};
@@ -539,23 +432,15 @@ public:
void work();
/**
* Called by the worker when data is received from a child process.
* Stores the event and resumes the coroutine.
*/
void handleChildOutput(Descriptor fd, std::string_view data);
virtual void handleChildOutput(Descriptor fd, std::string_view data)
{
unreachable();
}
/**
* Called by the worker when EOF is received from a child process.
* Stores the event and resumes the coroutine.
*/
void handleEOF(Descriptor fd);
/**
* Called by the worker when a build times out.
* Stores the event and resumes the coroutine.
*/
void timedOut(TimedOut && ex);
virtual void handleEOF(Descriptor fd)
{
unreachable();
}
void trace(std::string_view s);
@@ -564,6 +449,13 @@ public:
return name;
}
/**
* Callback in case of a timeout. It should wake up its waiters,
* get rid of any running child processes that are being monitored
* by the worker (important!), etc.
*/
virtual void timedOut(Error && ex) = 0;
/**
* Used for comparisons. The order matters a bit for scheduling. We
* want:

View File

@@ -53,6 +53,11 @@ public:
std::optional<ContentAddress> ca = std::nullopt);
~PathSubstitutionGoal();
void timedOut(Error && ex) override
{
unreachable();
};
std::string key() override
{
return "a$" + std::string(storePath.name()) + "$" + worker.store.printStorePath(storePath);
@@ -67,6 +72,12 @@ public:
StorePath subPath, nix::ref<Store> sub, std::shared_ptr<const ValidPathInfo> info, bool & substituterFailed);
Co finished();
/**
* Callback used by the worker to write to the log.
*/
void handleChildOutput(Descriptor fd, std::string_view data) override {};
void handleEOF(Descriptor fd) override;
/* Called by destructor, can't be overridden */
void cleanup() override final;

View File

@@ -586,12 +586,6 @@ void writeDerivation(Sink & out, const StoreDirConfig & store, const BasicDeriva
*/
std::string hashPlaceholder(const OutputNameView outputName);
/**
* The expected JSON version for derivation serialization.
* Used by `nix derivation show` and `nix derivation add`.
*/
constexpr unsigned expectedJsonVersionDerivation = 4;
} // namespace nix
JSON_IMPL_WITH_XP_FEATURES(nix::DerivationOutput)

View File

@@ -1143,7 +1143,7 @@ public:
Setting<std::string> netrcFile{
this,
(nixConfDir / "netrc").string(),
fmt("%s/%s", nixConfDir, "netrc"),
"netrc-file",
R"(
If set to an absolute path to a `netrc` file, Nix uses the HTTP

View File

@@ -39,32 +39,28 @@ deps_public_maybe_subproject = [
]
subdir('nix-meson-build-support/subprojects')
can_link_symlink = false
native_ln = find_program('ln', required : false, native : true)
if native_ln.found()
run_command(
native_ln,
'-s',
meson.project_build_root() / '__nothing_link_target',
meson.project_build_root() / '__nothing_symlink',
# native doesn't allow dangling symlinks, which the tests require
env : {'MSYS' : 'winsymlinks:lnk'},
check : true,
)
can_link_symlink = run_command(
native_ln,
meson.project_build_root() / '__nothing_symlink',
meson.project_build_root() / '__nothing_hardlink',
check : false,
).returncode() == 0
run_command(
'rm',
'-f',
meson.project_build_root() / '__nothing_symlink',
meson.project_build_root() / '__nothing_hardlink',
check : true,
)
endif
run_command(
'ln',
'-s',
meson.project_build_root() / '__nothing_link_target',
meson.project_build_root() / '__nothing_symlink',
# native doesn't allow dangling symlinks, which the tests require
env : {'MSYS' : 'winsymlinks:lnk'},
check : true,
)
can_link_symlink = run_command(
'ln',
meson.project_build_root() / '__nothing_symlink',
meson.project_build_root() / '__nothing_hardlink',
check : false,
).returncode() == 0
run_command(
'rm',
'-f',
meson.project_build_root() / '__nothing_symlink',
meson.project_build_root() / '__nothing_hardlink',
check : true,
)
summary('can hardlink to symlink', can_link_symlink, bool_yn : true)
configdata_priv.set('CAN_LINK_SYMLINK', can_link_symlink.to_int())
@@ -164,8 +160,6 @@ if s3_aws_auth.enabled()
deps_other += aws_crt_cpp
aws_c_common = cxx.find_library('aws-c-common', required : true)
deps_other += aws_c_common
aws_c_auth = cxx.find_library('aws-c-auth', required : true)
deps_other += aws_c_auth
endif
configdata_pub.set('NIX_WITH_AWS_AUTH', s3_aws_auth.enabled().to_int())

View File

@@ -4,11 +4,20 @@
#include <gtest/gtest.h>
#include "nix/util/types.hh"
#include "nix/util/environment-variables.hh"
#include "nix/util/file-system.hh"
#include "nix/util/tests/test-data.hh"
namespace nix {
/**
* The path to the unit test data directory. See the contributing guide
* in the manual for further details.
*/
static inline std::filesystem::path getUnitTestData()
{
return getEnv("_NIX_TEST_UNIT_DATA").value();
}
/**
* Whether we should update "golden masters" instead of running tests
* against them. See the contributing guide in the manual for further

View File

@@ -10,5 +10,4 @@ headers = files(
'json-characterization.hh',
'nix_api_util.hh',
'string_callback.hh',
'test-data.hh',
)

View File

@@ -1,24 +0,0 @@
#pragma once
///@file
#include <filesystem>
#include "nix/util/environment-variables.hh"
#include "nix/util/error.hh"
namespace nix {
/**
* The path to the unit test data directory. See the contributing guide
* in the manual for further details.
*/
static inline std::filesystem::path getUnitTestData()
{
auto data = getEnv("_NIX_TEST_UNIT_DATA");
if (!data)
throw Error(
"_NIX_TEST_UNIT_DATA environment variable is not set. "
"Recommendation: use meson, example: 'meson test -C build --gdb'");
return std::filesystem::path(*data);
}
} // namespace nix

View File

@@ -33,9 +33,6 @@ deps_private += rapidcheck
gtest = dependency('gtest', main : true)
deps_private += gtest
gmock = dependency('gmock')
deps_private += gmock
configdata = configuration_data()
configdata.set_quoted('PACKAGE_VERSION', meson.project_version())
@@ -75,7 +72,6 @@ sources = files(
'position.cc',
'processes.cc',
'sort.cc',
'source-accessor.cc',
'spawn.cc',
'strings.cc',
'suggestions.cc',

View File

@@ -1,138 +0,0 @@
#include "nix/util/fs-sink.hh"
#include "nix/util/file-system.hh"
#include "nix/util/processes.hh"
#include <gtest/gtest.h>
#include <gmock/gmock.h>
#include <rapidcheck/gtest.h>
namespace nix {
MATCHER_P2(HasContents, path, expected, "")
{
auto stat = arg->maybeLstat(path);
if (!stat) {
*result_listener << arg->showPath(path) << " does not exist";
return false;
}
if (stat->type != SourceAccessor::tRegular) {
*result_listener << arg->showPath(path) << " is not a regular file";
return false;
}
auto actual = arg->readFile(path);
if (actual != expected) {
*result_listener << arg->showPath(path) << " has contents " << ::testing::PrintToString(actual);
return false;
}
return true;
}
MATCHER_P2(HasSymlink, path, target, "")
{
auto stat = arg->maybeLstat(path);
if (!stat) {
*result_listener << arg->showPath(path) << " does not exist";
return false;
}
if (stat->type != SourceAccessor::tSymlink) {
*result_listener << arg->showPath(path) << " is not a symlink";
return false;
}
auto actual = arg->readLink(path);
if (actual != target) {
*result_listener << arg->showPath(path) << " points to " << ::testing::PrintToString(actual);
return false;
}
return true;
}
MATCHER_P2(HasDirectory, path, dirents, "")
{
auto stat = arg->maybeLstat(path);
if (!stat) {
*result_listener << arg->showPath(path) << " does not exist";
return false;
}
if (stat->type != SourceAccessor::tDirectory) {
*result_listener << arg->showPath(path) << " is not a directory";
return false;
}
auto actual = arg->readDirectory(path);
std::set<std::string> actualKeys, expectedKeys(dirents.begin(), dirents.end());
for (auto & [k, _] : actual)
actualKeys.insert(k);
if (actualKeys != expectedKeys) {
*result_listener << arg->showPath(path) << " has entries " << ::testing::PrintToString(actualKeys);
return false;
}
return true;
}
class FSSourceAccessorTest : public ::testing::Test
{
protected:
std::filesystem::path tmpDir;
std::unique_ptr<nix::AutoDelete> delTmpDir;
void SetUp() override
{
tmpDir = nix::createTempDir();
delTmpDir = std::make_unique<nix::AutoDelete>(tmpDir, true);
}
void TearDown() override
{
delTmpDir.release();
}
};
TEST_F(FSSourceAccessorTest, works)
{
{
RestoreSink sink(false);
sink.dstPath = tmpDir;
sink.dirFd = openDirectory(tmpDir);
sink.createDirectory(CanonPath("subdir"));
sink.createRegularFile(CanonPath("file1"), [](CreateRegularFileSink & crf) { crf("content1"); });
sink.createRegularFile(CanonPath("subdir/file2"), [](CreateRegularFileSink & crf) { crf("content2"); });
sink.createSymlink(CanonPath("rootlink"), "target");
sink.createDirectory(CanonPath("a"));
sink.createSymlink(CanonPath("a/dirlink"), "../subdir");
}
EXPECT_THAT(makeFSSourceAccessor(tmpDir / "file1"), HasContents(CanonPath::root, "content1"));
EXPECT_THAT(makeFSSourceAccessor(tmpDir / "rootlink"), HasSymlink(CanonPath::root, "target"));
EXPECT_THAT(
makeFSSourceAccessor(tmpDir),
HasDirectory(CanonPath::root, std::set<std::string>{"file1", "subdir", "rootlink", "a"}));
EXPECT_THAT(makeFSSourceAccessor(tmpDir / "subdir"), HasDirectory(CanonPath::root, std::set<std::string>{"file2"}));
{
auto accessor = makeFSSourceAccessor(tmpDir);
EXPECT_THAT(accessor, HasContents(CanonPath("file1"), "content1"));
EXPECT_THAT(accessor, HasContents(CanonPath("subdir/file2"), "content2"));
EXPECT_TRUE(accessor->pathExists(CanonPath("file1")));
EXPECT_FALSE(accessor->pathExists(CanonPath("nonexistent")));
EXPECT_THROW(accessor->readFile(CanonPath("a/dirlink/file2")), SymlinkNotAllowed);
EXPECT_THROW(accessor->maybeLstat(CanonPath("a/dirlink/file2")), SymlinkNotAllowed);
EXPECT_THROW(accessor->readDirectory(CanonPath("a/dirlink")), SymlinkNotAllowed);
EXPECT_THROW(accessor->pathExists(CanonPath("a/dirlink/file2")), SymlinkNotAllowed);
}
{
auto accessor = makeFSSourceAccessor(tmpDir / "nonexistent");
EXPECT_FALSE(accessor->maybeLstat(CanonPath::root));
EXPECT_THROW(accessor->readFile(CanonPath::root), SystemError);
}
{
auto accessor = makeFSSourceAccessor(tmpDir, true);
EXPECT_EQ(accessor->getLastModified(), 0);
accessor->maybeLstat(CanonPath("file1"));
EXPECT_GT(accessor->getLastModified(), 0);
}
}
} // namespace nix

View File

@@ -103,8 +103,7 @@ void SourceAccessor::dumpPath(const CanonPath & path, Sink & sink, PathFilter &
time_t dumpPathAndGetMtime(const Path & path, Sink & sink, PathFilter & filter)
{
SourcePath path2 = {
makeFSSourceAccessor(std::filesystem::path{}, /*trackLastModified=*/true), CanonPath(absPath(path))};
auto path2 = PosixSourceAccessor::createAtRoot(path, /*trackLastModified=*/true);
path2.dumpPath(sink, filter);
return path2.accessor->getLastModified().value();
}

View File

@@ -2,6 +2,7 @@
///@file
#include "nix/util/canon-path.hh"
#include "nix/util/types.hh"
#include "nix/util/error.hh"
#ifdef _WIN32
@@ -235,6 +236,18 @@ std::wstring handleToFileName(Descriptor handle);
#ifndef _WIN32
namespace unix {
struct SymlinkNotAllowed : public Error
{
CanonPath path;
SymlinkNotAllowed(CanonPath path)
/* Can't provide better error message, since the parent directory is only known to the caller. */
: Error("relative path '%s' points to a symlink, which is not allowed", path.rel())
, path(std::move(path))
{
}
};
/**
* Safe(r) function to open \param path file relative to \param dirFd, while
* disallowing escaping from a directory and resolving any symlinks in the

View File

@@ -43,6 +43,36 @@ public:
std::optional<std::filesystem::path> getPhysicalPath(const CanonPath & path) override;
/**
* Create a `PosixSourceAccessor` and `SourcePath` corresponding to
* some native path.
*
* @param Whether the accessor should return a non-null getLastModified.
* When true the accessor must be used only by a single thread.
*
* The `PosixSourceAccessor` is rooted as far up the tree as
* possible, (e.g. on Windows it could scoped to a drive like
* `C:\`). This allows more `..` parent accessing to work.
*
* @note When `path` is trusted user input, canonicalize it using
* `std::filesystem::canonical`, `makeParentCanonical`, `std::filesystem::weakly_canonical`, etc,
* as appropriate for the use case. At least weak canonicalization is
* required for the `SourcePath` to do anything useful at the location it
* points to.
*
* @note A canonicalizing behavior is not built in `createAtRoot` so that
* callers do not accidentally introduce symlink-related security vulnerabilities.
* Furthermore, `createAtRoot` does not know whether the file pointed to by
* `path` should be resolved if it is itself a symlink. In other words,
* `createAtRoot` can not decide between aforementioned `canonical`, `makeParentCanonical`, etc. for its callers.
*
* See
* [`std::filesystem::path::root_path`](https://en.cppreference.com/w/cpp/filesystem/path/root_path)
* and
* [`std::filesystem::path::relative_path`](https://en.cppreference.com/w/cpp/filesystem/path/relative_path).
*/
static SourcePath createAtRoot(const std::filesystem::path & path, bool trackLastModified = false);
std::optional<std::time_t> getLastModified() override
{
return trackLastModified ? std::optional{mtime} : std::nullopt;

View File

@@ -222,24 +222,6 @@ ref<SourceAccessor> makeEmptySourceAccessor();
*/
MakeError(RestrictedPathError, Error);
struct SymlinkNotAllowed : public Error
{
CanonPath path;
SymlinkNotAllowed(CanonPath path)
: Error("relative path '%s' points to a symlink, which is not allowed", path.rel())
, path(std::move(path))
{
}
template<typename... Args>
SymlinkNotAllowed(CanonPath path, const std::string & fs, Args &&... args)
: Error(fs, std::forward<Args>(args)...)
, path(std::move(path))
{
}
};
/**
* Return an accessor for the root filesystem.
*/
@@ -251,7 +233,7 @@ ref<SourceAccessor> getFSSourceAccessor();
* elements, and that absolute symlinks are resolved relative to
* `root`.
*/
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root, bool trackLastModified = false);
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root);
/**
* Construct an accessor that presents a "union" view of a vector of

View File

@@ -1,5 +1,7 @@
#include "nix/util/posix-source-accessor.hh"
#include "nix/util/source-path.hh"
#include "nix/util/signals.hh"
#include "nix/util/sync.hh"
#include <boost/unordered/concurrent_flat_map.hpp>
@@ -18,6 +20,15 @@ PosixSourceAccessor::PosixSourceAccessor()
{
}
SourcePath PosixSourceAccessor::createAtRoot(const std::filesystem::path & path, bool trackLastModified)
{
std::filesystem::path path2 = absPath(path);
return {
make_ref<PosixSourceAccessor>(path2.root_path(), trackLastModified),
CanonPath{path2.relative_path().string()},
};
}
std::filesystem::path PosixSourceAccessor::makeAbsPath(const CanonPath & path)
{
return root.empty() ? (std::filesystem::path{path.abs()})
@@ -197,7 +208,7 @@ void PosixSourceAccessor::assertNoSymlinks(CanonPath path)
while (!path.isRoot()) {
auto st = cachedLstat(path);
if (st && S_ISLNK(st->st_mode))
throw SymlinkNotAllowed(path, "path '%s' is a symlink", showPath(path));
throw Error("path '%s' is a symlink", showPath(path));
path.pop();
}
}
@@ -208,8 +219,8 @@ ref<SourceAccessor> getFSSourceAccessor()
return rootFS;
}
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root, bool trackLastModified)
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root)
{
return make_ref<PosixSourceAccessor>(std::move(root), trackLastModified);
return make_ref<PosixSourceAccessor>(std::move(root));
}
} // namespace nix

View File

@@ -136,7 +136,7 @@ static void extract_archive(TarArchive & archive, const std::filesystem::path &
if (!name)
throw Error("cannot get archive member name: %s", archive_error_string(archive.archive));
if (r == ARCHIVE_WARN)
warn("getting archive member '%1%': %2%", name, archive_error_string(archive.archive));
warn(archive_error_string(archive.archive));
else
archive.check(r);
@@ -193,7 +193,7 @@ time_t unpackTarfileToSink(TarArchive & archive, ExtendedFileSystemObjectSink &
throw Error("cannot get archive member name: %s", archive_error_string(archive.archive));
auto cpath = CanonPath{path};
if (r == ARCHIVE_WARN)
warn("getting archive member '%1%': %2%", path, archive_error_string(archive.archive));
warn(archive_error_string(archive.archive));
else
archive.check(r);

View File

@@ -3,7 +3,6 @@
#include "nix/util/signals.hh"
#include "nix/util/finally.hh"
#include "nix/util/serialise.hh"
#include "nix/util/source-accessor.hh"
#include <fcntl.h>
#include <unistd.h>
@@ -302,10 +301,10 @@ openFileEnsureBeneathNoSymlinksIterative(Descriptor dirFd, const CanonPath & pat
if (errno == ENOTDIR) /* Path component might be a symlink. */ {
struct ::stat st;
if (::fstatat(getParentFd(), component.c_str(), &st, AT_SYMLINK_NOFOLLOW) == 0 && S_ISLNK(st.st_mode))
throw SymlinkNotAllowed(path2);
throw unix::SymlinkNotAllowed(path2);
errno = ENOTDIR; /* Restore the errno. */
} else if (errno == ELOOP) {
throw SymlinkNotAllowed(path2);
throw unix::SymlinkNotAllowed(path2);
}
return INVALID_DESCRIPTOR;
@@ -316,7 +315,7 @@ openFileEnsureBeneathNoSymlinksIterative(Descriptor dirFd, const CanonPath & pat
auto res = ::openat(getParentFd(), std::string(path.baseName().value()).c_str(), flags | O_NOFOLLOW, mode);
if (res < 0 && errno == ELOOP)
throw SymlinkNotAllowed(path);
throw unix::SymlinkNotAllowed(path);
return res;
}
@@ -329,7 +328,7 @@ Descriptor unix::openFileEnsureBeneathNoSymlinks(Descriptor dirFd, const CanonPa
dirFd, path.rel_c_str(), flags, static_cast<uint64_t>(mode), RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS);
if (maybeFd) {
if (*maybeFd < 0 && errno == ELOOP)
throw SymlinkNotAllowed(path);
throw unix::SymlinkNotAllowed(path);
return *maybeFd;
}
#endif

View File

@@ -38,7 +38,7 @@ struct CmdAddToStore : MixDryRun, StoreCommand
if (!namePart)
namePart = baseNameOf(path);
SourcePath sourcePath = makeFSSourceAccessor(makeParentCanonical(path));
auto sourcePath = PosixSourceAccessor::createAtRoot(makeParentCanonical(path));
auto storePath = dryRun ? store->computeStorePath(*namePart, sourcePath, caMethod, hashAlgo, {}).first
: store->addToStoreSlow(*namePart, sourcePath, caMethod, hashAlgo, {}).path;

View File

@@ -60,11 +60,7 @@ struct CmdShowDerivation : InstallablesCommand, MixPrintJSON
jsonRoot[drvPath.to_string()] = store->readDerivation(drvPath);
}
printJSON(
nlohmann::json{
{"version", expectedJsonVersionDerivation},
{"derivations", std::move(jsonRoot)},
});
printJSON(jsonRoot);
}
};

View File

@@ -85,7 +85,9 @@ struct CmdHashBase : Command
return std::make_unique<HashSink>(hashAlgo);
};
auto makeSourcePath = [&]() -> SourcePath { return makeFSSourceAccessor(makeParentCanonical(path)); };
auto makeSourcePath = [&]() -> SourcePath {
return PosixSourceAccessor::createAtRoot(makeParentCanonical(path));
};
Hash h{HashAlgorithm::SHA256}; // throwaway def to appease C++
switch (mode) {

View File

@@ -613,8 +613,6 @@ static void main_nix_build(int argc, char ** argv)
environment variables and shell functions. Also don't
lose the current $PATH directories. */
auto rcfile = (tmpDir.path() / "rc").string();
auto tz = getEnv("TZ");
auto tzExport = tz ? "export TZ=" + escapeShellArgAlways(*tz) + "; " : "";
std::string rc = fmt(
(R"(_nix_shell_clean_tmpdir() { command rm -rf %1%; };)"s
"trap _nix_shell_clean_tmpdir EXIT; "
@@ -648,7 +646,7 @@ static void main_nix_build(int argc, char ** argv)
(pure ? "" : "PATH=$PATH:$p; unset p; "),
escapeShellArgAlways(dirOf(*shell)),
escapeShellArgAlways(*shell),
tzExport,
(getenv("TZ") ? (std::string("export TZ=") + escapeShellArgAlways(getenv("TZ")) + "; ") : ""),
envCommand);
vomit("Sourcing nix-shell with file %s and contents:\n%s", rcfile, rc);
writeFile(rcfile, rc);

View File

@@ -191,7 +191,7 @@ static void opAdd(Strings opFlags, Strings opArgs)
throw UsageError("unknown flag");
for (auto & i : opArgs) {
SourcePath sourcePath = makeFSSourceAccessor(makeParentCanonical(i));
auto sourcePath = PosixSourceAccessor::createAtRoot(makeParentCanonical(i));
cout << fmt("%s\n", store->printStorePath(store->addToStore(std::string(baseNameOf(i)), sourcePath)));
}
}
@@ -215,7 +215,7 @@ static void opAddFixed(Strings opFlags, Strings opArgs)
opArgs.pop_front();
for (auto & i : opArgs) {
SourcePath sourcePath = makeFSSourceAccessor(makeParentCanonical(i));
auto sourcePath = PosixSourceAccessor::createAtRoot(makeParentCanonical(i));
std::cout << fmt(
"%s\n", store->printStorePath(store->addToStoreSlow(baseNameOf(i), sourcePath, method, hashAlgo).path));
}

View File

@@ -256,7 +256,7 @@ hashPath(char * algo, int base32, char * path)
PPCODE:
try {
Hash h = hashPath(
SourcePath{getFSSourceAccessor(), CanonPath(absPath(Path(path)))},
PosixSourceAccessor::createAtRoot(path),
FileIngestionMethod::NixArchive, parseHashAlgo(algo)).first;
auto s = h.to_string(base32 ? HashFormat::Nix32 : HashFormat::Base16, false);
XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
@@ -336,7 +336,7 @@ StoreWrapper::addToStore(char * srcPath, int recursive, char * algo)
auto method = recursive ? ContentAddressMethod::Raw::NixArchive : ContentAddressMethod::Raw::Flat;
auto path = THIS->store->addToStore(
std::string(baseNameOf(srcPath)),
SourcePath{getFSSourceAccessor(), CanonPath(absPath(Path(srcPath)))},
PosixSourceAccessor::createAtRoot(srcPath),
method, parseHashAlgo(algo));
XPUSHs(sv_2mortal(newSVpv(THIS->store->printStorePath(path).c_str(), 0)));
} catch (Error & e) {

View File

@@ -6,7 +6,7 @@ export NIX_TESTS_CA_BY_DEFAULT=1
drvPath=$(nix-instantiate ../simple.nix)
nix derivation show "$drvPath" | jq '.derivations[]' > "$TEST_HOME"/simple.json
nix derivation show "$drvPath" | jq .[] > "$TEST_HOME"/simple.json
drvPath2=$(nix derivation add < "$TEST_HOME"/simple.json)
@@ -27,5 +27,5 @@ drvPath4=$(nix derivation add < "$TEST_HOME"/foo.json)
[[ -e "$drvPath3" ]]
# The modified derivation read back as JSON matches
nix derivation show "$drvPath3" | jq '.derivations[]' > "$TEST_HOME"/foo-read.json
nix derivation show "$drvPath3" | jq .[] > "$TEST_HOME"/foo-read.json
diff "$TEST_HOME"/foo.json "$TEST_HOME"/foo-read.json

View File

@@ -4,7 +4,7 @@ source common.sh
drvPath=$(nix-instantiate simple.nix)
nix derivation show "$drvPath" | jq '.derivations[]' > "$TEST_HOME/simple.json"
nix derivation show "$drvPath" | jq '.[]' > "$TEST_HOME/simple.json"
# Round tripping to JSON works
drvPath2=$(nix derivation add < "$TEST_HOME/simple.json")

View File

@@ -18,7 +18,7 @@ mkDerivation rec {
PATH=${builtins.getEnv "EXTRA_PATH"}:$PATH
# JSON of pre-existing drv
nix derivation show $drv | jq '.derivations[]' > drv0.json
nix derivation show $drv | jq .[] > drv0.json
# Fix name
jq < drv0.json '.name = "${innerName}"' > drv1.json

View File

@@ -16,7 +16,7 @@ printf 0 > "$TEST_ROOT"/counter
# `nix derivation add` with impure derivations work
drvPath=$(nix-instantiate ./impure-derivations.nix -A impure)
nix derivation show "$drvPath" | jq '.derivations[]' > "$TEST_HOME"/impure-drv.json
nix derivation show "$drvPath" | jq .[] > "$TEST_HOME"/impure-drv.json
drvPath2=$(nix derivation add < "$TEST_HOME"/impure-drv.json)
[[ "$drvPath" = "$drvPath2" ]]
@@ -50,8 +50,8 @@ path4=$(nix build -L --no-link --json --file ./impure-derivations.nix impureOnIm
(! nix build -L --no-link --json --file ./impure-derivations.nix inputAddressed 2>&1) | grep 'depends on impure derivation'
drvPath=$(nix eval --json --file ./impure-derivations.nix impure.drvPath | jq -r .)
[[ $(nix derivation show "$drvPath" | jq ".derivations[\"$(basename "$drvPath")\"].outputs.out.impure") = true ]]
[[ $(nix derivation show "$drvPath" | jq ".derivations[\"$(basename "$drvPath")\"].outputs.stuff.impure") = true ]]
[[ $(nix derivation show "$drvPath" | jq ".[\"$(basename "$drvPath")\"].outputs.out.impure") = true ]]
[[ $(nix derivation show "$drvPath" | jq ".[\"$(basename "$drvPath")\"].outputs.stuff.impure") = true ]]
# Fixed-output derivations *can* depend on impure derivations.
path5=$(nix build -L --no-link --json --file ./impure-derivations.nix contentAddressed | jq -r .[].outputs.out)

View File

@@ -106,16 +106,14 @@ let
foo = runCommand "foo" { } ''
mkdir -p $out/bin
echo '#!${shell}' > $out/bin/foo
echo 'echo ${fooContents}' >> $out/bin/foo
echo 'echo ${fooContents}' > $out/bin/foo
chmod a+rx $out/bin/foo
ln -s ${shell} $out/bin/bash
'';
bar = runCommand "bar" { } ''
mkdir -p $out/bin
echo '#!${shell}' > $out/bin/bar
echo 'echo bar' >> $out/bin/bar
echo 'echo bar' > $out/bin/bar
chmod a+rx $out/bin/bar
'';
@@ -128,8 +126,7 @@ let
# ruby "interpreter" that outputs "$@"
ruby = runCommand "ruby" { } ''
mkdir -p $out/bin
echo '#!${shell}' > $out/bin/ruby
echo 'printf %s "$*"' >> $out/bin/ruby
echo 'printf %s "$*"' > $out/bin/ruby
chmod a+rx $out/bin/ruby
'';

View File

@@ -49,4 +49,4 @@ expectStderr 0 nix-instantiate --expr "$hackyExpr" --eval --strict | grepQuiet "
# Check it works with the expected structured attrs
hacky=$(nix-instantiate --expr "$hackyExpr")
nix derivation show "$hacky" | jq --exit-status '.derivations."'"$(basename "$hacky")"'".structuredAttrs | . == {"a": 1}'
nix derivation show "$hacky" | jq --exit-status '."'"$(basename "$hacky")"'".structuredAttrs | . == {"a": 1}'

View File

@@ -147,7 +147,7 @@ in
else:
machine.fail(f"nix path-info {pkg}")
def setup_s3(populate_bucket=[], public=False, versioned=False, profiles=None):
def setup_s3(populate_bucket=[], public=False, versioned=False):
"""
Decorator that creates/destroys a unique bucket for each test.
Optionally pre-populates bucket with specified packages.
@@ -157,22 +157,9 @@ in
populate_bucket: List of packages to upload before test runs
public: If True, make the bucket publicly accessible
versioned: If True, enable versioning on the bucket before populating
profiles: Dict of AWS profiles to create, e.g.:
{"valid": {"access_key": "...", "secret_key": "..."},
"invalid": {"access_key": "WRONG", "secret_key": "WRONG"}}
Profiles are created on the client machine at /root/.aws/credentials
"""
def decorator(test_func):
def wrapper():
# Restart nix-daemon on both machines to clear the credential provider cache.
# The AwsCredentialProviderImpl singleton persists in the daemon process,
# and its cache can cause credentials from previous tests to be reused.
# We reset-failed first to avoid systemd's start rate limiting.
server.succeed("systemctl reset-failed nix-daemon.service nix-daemon.socket")
server.succeed("systemctl restart nix-daemon")
client.succeed("systemctl reset-failed nix-daemon.service nix-daemon.socket")
client.succeed("systemctl restart nix-daemon")
bucket = str(uuid.uuid4())
server.succeed(f"mc mb minio/{bucket}")
try:
@@ -180,15 +167,6 @@ in
server.succeed(f"mc anonymous set download minio/{bucket}")
if versioned:
server.succeed(f"mc version enable minio/{bucket}")
if profiles:
# Build credentials file content
creds_content = ""
for name, creds in profiles.items():
creds_content += f"[{name}]\n"
creds_content += f"aws_access_key_id = {creds['access_key']}\n"
creds_content += f"aws_secret_access_key = {creds['secret_key']}\n\n"
client.succeed("mkdir -p /root/.aws")
client.succeed(f"cat > /root/.aws/credentials << 'AWSCREDS'\n{creds_content}AWSCREDS")
if populate_bucket:
store_url = make_s3_url(bucket)
for pkg in populate_bucket:
@@ -196,9 +174,6 @@ in
test_func(bucket)
finally:
server.succeed(f"mc rb --force minio/{bucket}")
# Clean up AWS profiles if created
if profiles:
client.succeed("rm -rf /root/.aws")
# Clean up client store - only delete if path exists
for pkg in PKGS.values():
client.succeed(f"[ ! -e {pkg} ] || nix store delete --ignore-liveness {pkg}")
@@ -789,108 +764,6 @@ in
print(" Compressed log uploaded with multipart")
@setup_s3(
populate_bucket=[PKGS['A']],
profiles={
"valid": {"access_key": ACCESS_KEY, "secret_key": SECRET_KEY},
"invalid": {"access_key": "INVALIDKEY", "secret_key": "INVALIDSECRET"},
}
)
def test_profile_credentials(bucket):
"""Test that profile-based credentials work without environment variables"""
print("\n=== Testing Profile-Based Credentials ===")
store_url = make_s3_url(bucket, profile="valid")
# Verify store info works with profile credentials (no env vars)
client.succeed(f"HOME=/root nix store info --store '{store_url}' >&2")
print(" nix store info works with profile credentials")
# Verify we can copy from the store using profile
verify_packages_in_store(client, PKGS['A'], should_exist=False)
client.succeed(f"HOME=/root nix copy --no-check-sigs --from '{store_url}' {PKGS['A']}")
verify_packages_in_store(client, PKGS['A'])
print(" nix copy works with profile credentials")
# Clean up the package we just copied so we can test invalid profile
client.succeed(f"nix store delete --ignore-liveness {PKGS['A']}")
verify_packages_in_store(client, PKGS['A'], should_exist=False)
# Verify invalid profile fails when trying to copy
invalid_url = make_s3_url(bucket, profile="invalid")
client.fail(f"HOME=/root nix copy --no-check-sigs --from '{invalid_url}' {PKGS['A']} 2>&1")
print(" Invalid profile credentials correctly rejected")
@setup_s3(
populate_bucket=[PKGS['A']],
profiles={
"wrong": {"access_key": "WRONGKEY", "secret_key": "WRONGSECRET"},
}
)
def test_env_vars_precedence(bucket):
"""Test that environment variables take precedence over profile credentials"""
print("\n=== Testing Environment Variables Precedence ===")
# Use profile with wrong credentials, but provide correct creds via env vars
store_url = make_s3_url(bucket, profile="wrong")
# Ensure package is not in client store
verify_packages_in_store(client, PKGS['A'], should_exist=False)
# This should succeed because env vars (correct) override profile (wrong)
output = client.succeed(
f"HOME=/root {ENV_WITH_CREDS} nix copy --no-check-sigs --debug --from '{store_url}' {PKGS['A']} 2>&1"
)
print(" nix copy succeeded with env vars overriding wrong profile")
# Verify the credential chain shows Environment provider was added
if "Added AWS Environment Credential Provider" not in output:
print("Debug output:")
print(output)
raise Exception("Expected Environment provider to be added to chain")
print(" Environment provider added to credential chain")
# Clean up the package so we can test again without env vars
client.succeed(f"nix store delete --ignore-liveness {PKGS['A']}")
verify_packages_in_store(client, PKGS['A'], should_exist=False)
# Without env vars, same URL should fail (proving profile creds are actually wrong)
client.fail(f"HOME=/root nix copy --no-check-sigs --from '{store_url}' {PKGS['A']} 2>&1")
print(" Without env vars, wrong profile credentials correctly fail")
@setup_s3(
populate_bucket=[PKGS['A']],
profiles={
"testprofile": {"access_key": ACCESS_KEY, "secret_key": SECRET_KEY},
}
)
def test_credential_provider_chain(bucket):
"""Test that debug logging shows which providers are added to the chain"""
print("\n=== Testing Credential Provider Chain Logging ===")
store_url = make_s3_url(bucket, profile="testprofile")
output = client.succeed(
f"HOME=/root nix store info --debug --store '{store_url}' 2>&1"
)
# For a named profile, we expect to see these providers in the chain
expected_providers = ["Environment", "Profile", "IMDS"]
for provider in expected_providers:
msg = f"Added AWS {provider} Credential Provider to chain for profile 'testprofile'"
if msg not in output:
print("Debug output:")
print(output)
raise Exception(f"Expected to find: {msg}")
print(f" {provider} provider added to chain")
# SSO should be skipped (no SSO config for this profile)
if "Skipped AWS SSO Credential Provider for profile 'testprofile'" not in output:
print("Debug output:")
print(output)
raise Exception("Expected SSO provider to be skipped")
print(" SSO provider correctly skipped (not configured)")
# ============================================================================
# Main Test Execution
# ============================================================================
@@ -924,9 +797,6 @@ in
test_multipart_upload_basic()
test_multipart_threshold()
test_multipart_with_log_compression()
test_profile_credentials()
test_env_vars_precedence()
test_credential_provider_chain()
print("\n" + "="*80)
print(" All S3 Binary Cache Store Tests Passed!")