Compare commits
1 Commits
makefs-sou
...
warn-non-o
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
da9ade20e3 |
9
doc/manual/rl-next/channels-subdomain.md
Normal file
9
doc/manual/rl-next/channels-subdomain.md
Normal file
@@ -0,0 +1,9 @@
|
||||
---
|
||||
synopsis: Channel URLs migrated to channels.nixos.org subdomain
|
||||
prs: [14518]
|
||||
issues: [14517]
|
||||
---
|
||||
|
||||
Channel URLs have been updated from `https://nixos.org/channels/` to `https://channels.nixos.org/` throughout Nix.
|
||||
|
||||
The subdomain provides better reliability with IPv6 support and improved CDN distribution. The old domain apex (`nixos.org/channels/`) currently redirects to the new location but may be deprecated in the future.
|
||||
88
doc/manual/rl-next/json-format-changes.md
Normal file
88
doc/manual/rl-next/json-format-changes.md
Normal file
@@ -0,0 +1,88 @@
|
||||
---
|
||||
synopsis: "JSON format changes for store path info and derivations"
|
||||
prs: []
|
||||
issues: []
|
||||
---
|
||||
|
||||
JSON formats for store path info and derivations have been updated with new versions and structured fields.
|
||||
|
||||
## Store Path Info JSON
|
||||
|
||||
`nix path-info --json` now requires a `--json-format` flag to specify the output format version.
|
||||
Using `--json` without `--json-format` is deprecated and will become an error in a future release.
|
||||
For now, it defaults to version 1 with a warning, for a smoother migration.
|
||||
|
||||
### Version 1 (`--json-format 1`)
|
||||
|
||||
This is the legacy format, preserved for backwards compatibility:
|
||||
|
||||
- String-based hash values (e.g., `"narHash": "sha256:FePFYIlM..."`)
|
||||
- String-based content addresses (e.g., `"ca": "fixed:r:sha256:1abc..."`)
|
||||
- Full store paths for map keys and references (e.g., `"/nix/store/abc...-foo"`)
|
||||
- Now includes `"storeDir"` field at the top level
|
||||
|
||||
### Version 2 (`--json-format 2`)
|
||||
|
||||
The new structured format follows the [JSON guidelines](@docroot@/development/json-guideline.md) with the following changes:
|
||||
|
||||
- **Nested structure with top-level metadata**:
|
||||
|
||||
The output is now wrapped in an object with `version`, `storeDir`, and `info` fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 2,
|
||||
"storeDir": "/nix/store",
|
||||
"info": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
The map from store bath base names to store object info is nested under the `info` field.
|
||||
|
||||
- **Store path base names instead of full paths**:
|
||||
|
||||
Map keys and references use store path base names (e.g., `"abc...-foo"`) instead of full absolute store paths.
|
||||
Combined with `storeDir`, the full path can be reconstructed.
|
||||
|
||||
- **Structured `ca` field**:
|
||||
|
||||
Content address is now a structured JSON object instead of a string:
|
||||
|
||||
- Old: `"ca": "fixed:r:sha256:1abc..."`
|
||||
- New: `"ca": {"method": "nar", "hash": "sha256-ungWv48Bz+pBQUDeXa4iI7ADYaOWF3qctBD/YfIAFa0="}`
|
||||
- Still `null` values for input-addressed store objects
|
||||
|
||||
The `hash` field uses the [SRI](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) format like other hashes.
|
||||
|
||||
Nix currently only produces, and doesn't consume this format.
|
||||
|
||||
Additionally the following field is added to both formats.
|
||||
(The `version` tracks breaking changes, and adding fields to outputted JSON is not a breaking change.)
|
||||
|
||||
- **`version` field**:
|
||||
|
||||
All store path info JSON now includes `"version": <1|2>`.
|
||||
|
||||
- **`storeDir` field**:
|
||||
|
||||
Top-level `"storeDir"` field contains the store directory path (e.g., `"/nix/store"`).
|
||||
|
||||
## Derivation JSON (Version 4)
|
||||
|
||||
The derivation JSON format has been updated from version 3 to version 4:
|
||||
|
||||
- **Restructured inputs**:
|
||||
|
||||
Inputs are now nested under an `inputs` object:
|
||||
|
||||
- Old: `"inputSrcs": [...], "inputDrvs": {...}`
|
||||
- New: `"inputs": {"srcs": [...], "drvs": {...}}`
|
||||
|
||||
- **Consistent content addresses**:
|
||||
|
||||
Fixed content-addressed outputs now use structured JSON format.
|
||||
This is the same format as `ca` in store path info (after the new version).
|
||||
|
||||
Version 3 and earlier formats are *not* accepted when reading.
|
||||
|
||||
**Affected command**: `nix derivation`, namely its `show` and `add` sub-commands.
|
||||
12
doc/manual/rl-next/libcurl-pausing.md
Normal file
12
doc/manual/rl-next/libcurl-pausing.md
Normal file
@@ -0,0 +1,12 @@
|
||||
---
|
||||
synopsis: Fix "download buffer is full; consider increasing the 'download-buffer-size' setting" warning
|
||||
prs: [14614]
|
||||
issues: [11728]
|
||||
---
|
||||
|
||||
The underlying issue that led to [#11728](https://github.com/NixOS/nix/issues/11728) has been resolved by utilizing
|
||||
[libcurl write pausing functionality](https://curl.se/libcurl/c/curl_easy_pause.html) to control backpressure when unpacking to slow destinations like the git-backed tarball cache. The default value of `download-buffer-size` is now 1 MiB and it's no longer recommended to increase it, since the root cause has been fixed.
|
||||
|
||||
This is expected to improve download performance on fast connections, since previously a single slow download consumer would stall the thread and prevent any other transfers from progressing.
|
||||
|
||||
Many thanks go out to the [Lix project](https://lix.systems/) for the [implementation](https://git.lix.systems/lix-project/lix/commit/4ae6fb5a8f0d456b8d2ba2aaca3712b4e49057fc) that served as inspiration for this change and for triaging libcurl [issues with pausing](https://github.com/curl/curl/issues/19334).
|
||||
8
doc/manual/rl-next/repl-interrupt.md
Normal file
8
doc/manual/rl-next/repl-interrupt.md
Normal file
@@ -0,0 +1,8 @@
|
||||
---
|
||||
synopsis: Interrupting REPL commands works more than once
|
||||
issues: [13481]
|
||||
---
|
||||
|
||||
Previously, this only worked once per REPL session; further attempts would be ignored.
|
||||
This issue is now fixed, so REPL commands such as `:b` or `:p` can be canceled consistently.
|
||||
This is a cherry-pick of the change from the [Lix project](https://gerrit.lix.systems/c/lix/+/1097).
|
||||
40
doc/manual/rl-next/s3-curl-implementation.md
Normal file
40
doc/manual/rl-next/s3-curl-implementation.md
Normal file
@@ -0,0 +1,40 @@
|
||||
---
|
||||
synopsis: "Improved S3 binary cache support via HTTP"
|
||||
prs: [13752, 13823, 14026, 14120, 14131, 14135, 14144, 14170, 14190, 14198, 14206, 14209, 14222, 14223, 14330, 14333, 14335, 14336, 14337, 14350, 14356, 14357, 14374, 14375, 14376, 14377, 14391, 14393, 14420, 14421]
|
||||
issues: [13084, 12671, 11748, 12403]
|
||||
---
|
||||
|
||||
S3 binary cache operations now happen via HTTP, leveraging `libcurl`'s native
|
||||
AWS SigV4 authentication instead of the AWS C++ SDK, providing significant
|
||||
improvements:
|
||||
|
||||
- **Reduced memory usage**: Eliminates memory buffering issues that caused
|
||||
segfaults with large files
|
||||
- **Fixed upload reliability**: Resolves AWS SDK chunking errors
|
||||
(`InvalidChunkSizeError`)
|
||||
- **Lighter dependencies**: Uses lightweight `aws-crt-cpp` instead of full
|
||||
`aws-cpp-sdk`, reducing build complexity
|
||||
|
||||
The new implementation requires curl >= 7.75.0 and `aws-crt-cpp` for credential
|
||||
management.
|
||||
|
||||
All existing S3 URL formats and parameters remain supported, however the store
|
||||
settings for configuring multipart uploads have changed:
|
||||
|
||||
- **`multipart-upload`** (default: `false`): Enable multipart uploads for large
|
||||
files. When enabled, files exceeding the multipart threshold will be uploaded
|
||||
in multiple parts.
|
||||
|
||||
- **`multipart-threshold`** (default: `100 MiB`): Minimum file size for using
|
||||
multipart uploads. Files smaller than this will use regular PUT requests.
|
||||
Only takes effect when `multipart-upload` is enabled.
|
||||
|
||||
- **`multipart-chunk-size`** (default: `5 MiB`): Size of each part in multipart
|
||||
uploads. Must be at least 5 MiB (AWS S3 requirement). Larger chunk sizes
|
||||
reduce the number of requests but use more memory.
|
||||
|
||||
- **`buffer-size`**: Has been replaced by `multipart-chunk-size` and is now an alias to it.
|
||||
|
||||
Note that this change also means Nix now supports S3 binary cache stores even
|
||||
if built without `aws-crt-cpp`, but only for public buckets which do not
|
||||
require authentication.
|
||||
14
doc/manual/rl-next/s3-object-versioning.md
Normal file
14
doc/manual/rl-next/s3-object-versioning.md
Normal file
@@ -0,0 +1,14 @@
|
||||
---
|
||||
synopsis: "S3 URLs now support object versioning via versionId parameter"
|
||||
prs: [14274]
|
||||
issues: [13955]
|
||||
---
|
||||
|
||||
S3 URLs now support a `versionId` query parameter to fetch specific versions
|
||||
of objects from S3 buckets with versioning enabled. This allows pinning to
|
||||
exact object versions for reproducibility and protection against unexpected
|
||||
changes:
|
||||
|
||||
```
|
||||
s3://bucket/key?region=us-east-1&versionId=abc123def456
|
||||
```
|
||||
21
doc/manual/rl-next/s3-storage-class.md
Normal file
21
doc/manual/rl-next/s3-storage-class.md
Normal file
@@ -0,0 +1,21 @@
|
||||
---
|
||||
synopsis: "S3 binary cache stores now support storage class configuration"
|
||||
prs: [14464]
|
||||
issues: [7015]
|
||||
---
|
||||
|
||||
S3 binary cache stores now support configuring the storage class for uploaded objects via the `storage-class` parameter. This allows users to optimize costs by selecting appropriate storage tiers based on access patterns.
|
||||
|
||||
Example usage:
|
||||
|
||||
```bash
|
||||
# Use Glacier storage for long-term archival
|
||||
nix copy --to 's3://my-bucket?storage-class=GLACIER' /nix/store/...
|
||||
|
||||
# Use Intelligent Tiering for automatic cost optimization
|
||||
nix copy --to 's3://my-bucket?storage-class=INTELLIGENT_TIERING' /nix/store/...
|
||||
```
|
||||
|
||||
The storage class applies to both regular uploads and multipart uploads. When not specified, objects use the bucket's default storage class.
|
||||
|
||||
See the [S3 storage classes documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) for available storage classes and their characteristics.
|
||||
@@ -151,7 +151,6 @@
|
||||
- [Contributing](development/contributing.md)
|
||||
- [Releases](release-notes/index.md)
|
||||
{{#include ./SUMMARY-rl-next.md}}
|
||||
- [Release 2.33 (2025-12-09)](release-notes/rl-2.33.md)
|
||||
- [Release 2.32 (2025-10-06)](release-notes/rl-2.32.md)
|
||||
- [Release 2.31 (2025-08-21)](release-notes/rl-2.31.md)
|
||||
- [Release 2.30 (2025-07-07)](release-notes/rl-2.30.md)
|
||||
|
||||
@@ -6,7 +6,14 @@ Additionally, see [Testing Nix](./testing.md) for further instructions on how to
|
||||
|
||||
## Building Nix with Debug Symbols
|
||||
|
||||
In the development shell, `mesonBuildType` is set automatically to `debugoptimized`. This builds Nix with debug symbols, which are essential for effective debugging.
|
||||
In the development shell, set the `mesonBuildType` environment variable to `debug` before configuring the build:
|
||||
|
||||
```console
|
||||
[nix-shell]$ export mesonBuildType=debugoptimized
|
||||
```
|
||||
|
||||
Then, proceed to build Nix as described in [Building Nix](./building.md).
|
||||
This will build Nix with debug symbols, which are essential for effective debugging.
|
||||
|
||||
It is also possible to build without optimization for faster build:
|
||||
|
||||
|
||||
@@ -1,281 +0,0 @@
|
||||
# Release 2.33.0 (2025-12-09)
|
||||
|
||||
## New features
|
||||
|
||||
- New command `nix registry resolve` [#14595](https://github.com/NixOS/nix/pull/14595)
|
||||
|
||||
This command looks up a flake registry input name and returns the flakeref it resolves to.
|
||||
|
||||
For example, looking up Nixpkgs:
|
||||
|
||||
```
|
||||
$ nix registry resolve nixpkgs
|
||||
github:NixOS/nixpkgs/nixpkgs-unstable
|
||||
```
|
||||
|
||||
Upstreamed from [Determinate Nix 3.14.0](https://github.com/DeterminateSystems/nix-src/pull/273).
|
||||
|
||||
- `nix flake clone` supports all input types [#14581](https://github.com/NixOS/nix/pull/14581)
|
||||
|
||||
`nix flake clone` now supports arbitrary input types. In particular, this allows you to clone tarball flakes, such as flakes on FlakeHub.
|
||||
|
||||
Upstreamed from [Determinate Nix 3.12.0](https://github.com/DeterminateSystems/nix-src/pull/229).
|
||||
|
||||
## Performance improvements
|
||||
|
||||
- Git fetcher computes `revCount`s using multiple threads [#14462](https://github.com/NixOS/nix/pull/14462)
|
||||
|
||||
When using Git repositories with a long history, calculating the `revCount` attribute can take a long time. Nix now computes `revCount` using multiple threads, making it much faster (e.g. 9.1s to 3.7s for Nixpkgs).
|
||||
|
||||
Note that if you don't need `revCount`, you can disable it altogether by setting the flake input attribute `shallow = true`.
|
||||
|
||||
Upstreamed from [Determinate Nix 3.12.2](https://github.com/DeterminateSystems/nix-src/pull/245).
|
||||
|
||||
- `builtins.stringLength` now runs in constant time [#14442](https://github.com/NixOS/nix/pull/14442)
|
||||
|
||||
The internal representation of strings has been replaced with a size-prefixed Pascal style string. Previously Nix stored strings as a NUL-terminated array of bytes, necessitating a linear scan to calculate the length.
|
||||
|
||||
- Uploads to `http://` and `https://` binary cache stores now run in constant memory [#14390](https://github.com/NixOS/nix/pull/14390)
|
||||
|
||||
Nix used to buffer the whole compressed NAR contents in memory. It now reads it in a streaming fashion.
|
||||
|
||||
- Channel URLs migrated to channels.nixos.org subdomain [#14517](https://github.com/NixOS/nix/issues/14517) [#14518](https://github.com/NixOS/nix/pull/14518)
|
||||
|
||||
Channel URLs have been updated from `https://nixos.org/channels/` to `https://channels.nixos.org/` throughout Nix. This subdomain provides better reliability with IPv6 support and improved CDN distribution. The old domain apex (`nixos.org/channels/`) currently redirects to the new location but may be deprecated in the future.
|
||||
|
||||
- Fix `download buffer is full; consider increasing the 'download-buffer-size' setting` warning [#11728](https://github.com/NixOS/nix/issues/11728) [#14614](https://github.com/NixOS/nix/pull/14614)
|
||||
|
||||
The underlying issue that led to [#11728](https://github.com/NixOS/nix/issues/11728) has been resolved by utilizing
|
||||
[libcurl write pausing functionality](https://curl.se/libcurl/c/curl_easy_pause.html) to control backpressure when unpacking to slow destinations like the git-backed tarball cache. The default value of `download-buffer-size` is now 1 MiB and it's no longer recommended to increase it, since the root cause has been fixed.
|
||||
|
||||
This is expected to improve download performance on fast connections, since previously a single slow download consumer would stall the thread and prevent any other transfers from progressing.
|
||||
|
||||
Many thanks go out to the [Lix project](https://lix.systems/) for the [implementation](https://git.lix.systems/lix-project/lix/commit/4ae6fb5a8f0d456b8d2ba2aaca3712b4e49057fc) that served as inspiration for this change and for triaging libcurl [issues with pausing](https://github.com/curl/curl/issues/19334).
|
||||
|
||||
- Significantly improve tarball unpacking performance [#14689](https://github.com/NixOS/nix/pull/14689) [#14696](https://github.com/NixOS/nix/pull/14696) [#10683](https://github.com/NixOS/nix/issues/10683) [#11098](https://github.com/NixOS/nix/issues/11098)
|
||||
|
||||
Nix uses a content-addressed cache backed by libgit2 for deduplicating files fetched via `fetchTarball` and `github`, `tarball` flake inputs. Its usage has been significantly optimised to reduce the amount of I/O operations that are performed. For a typical nixpkgs source tarball this results in 200 times fewer system calls on Linux. In combination with libcurl pausing this alleviates performance regressions stemming from the tarball cache.
|
||||
|
||||
- Already valid derivations are no longer copied to the store [#14219](https://github.com/NixOS/nix/pull/14219)
|
||||
|
||||
This results in a modest speedup when using the Nix daemon.
|
||||
|
||||
- `nix nar ls` and `nix nar cat` are significantly faster and no longer buffer the whole NAR in memory [#14273](https://github.com/NixOS/nix/pull/14273) [#14732](https://github.com/NixOS/nix/pull/14732)
|
||||
|
||||
## S3 improvements
|
||||
|
||||
- Improved S3 binary cache support via HTTP [#11748](https://github.com/NixOS/nix/issues/11748) [#12403](https://github.com/NixOS/nix/issues/12403) [#12671](https://github.com/NixOS/nix/issues/12671) [#13084](https://github.com/NixOS/nix/issues/13084) [#13752](https://github.com/NixOS/nix/pull/13752) [#13823](https://github.com/NixOS/nix/pull/13823) [#14026](https://github.com/NixOS/nix/pull/14026) [#14120](https://github.com/NixOS/nix/pull/14120) [#14131](https://github.com/NixOS/nix/pull/14131) [#14135](https://github.com/NixOS/nix/pull/14135) [#14144](https://github.com/NixOS/nix/pull/14144) [#14170](https://github.com/NixOS/nix/pull/14170) [#14190](https://github.com/NixOS/nix/pull/14190) [#14198](https://github.com/NixOS/nix/pull/14198) [#14206](https://github.com/NixOS/nix/pull/14206) [#14209](https://github.com/NixOS/nix/pull/14209) [#14222](https://github.com/NixOS/nix/pull/14222) [#14223](https://github.com/NixOS/nix/pull/14223) [#14330](https://github.com/NixOS/nix/pull/14330) [#14333](https://github.com/NixOS/nix/pull/14333) [#14335](https://github.com/NixOS/nix/pull/14335) [#14336](https://github.com/NixOS/nix/pull/14336) [#14337](https://github.com/NixOS/nix/pull/14337) [#14350](https://github.com/NixOS/nix/pull/14350) [#14356](https://github.com/NixOS/nix/pull/14356) [#14357](https://github.com/NixOS/nix/pull/14357) [#14374](https://github.com/NixOS/nix/pull/14374) [#14375](https://github.com/NixOS/nix/pull/14375) [#14376](https://github.com/NixOS/nix/pull/14376) [#14377](https://github.com/NixOS/nix/pull/14377) [#14391](https://github.com/NixOS/nix/pull/14391) [#14393](https://github.com/NixOS/nix/pull/14393) [#14420](https://github.com/NixOS/nix/pull/14420) [#14421](https://github.com/NixOS/nix/pull/14421)
|
||||
|
||||
S3 binary cache operations now happen via HTTP, leveraging `libcurl`'s native AWS SigV4 authentication instead of the AWS C++ SDK, providing significant improvements:
|
||||
|
||||
- **Reduced memory usage**: Eliminates memory buffering issues that caused segfaults with large files
|
||||
- **Fixed upload reliability**: Resolves AWS SDK chunking errors (`InvalidChunkSizeError`)
|
||||
- **Lighter dependencies**: Uses lightweight `aws-crt-cpp` instead of full `aws-cpp-sdk`, reducing build complexity
|
||||
|
||||
The new implementation requires curl >= 7.75.0 and `aws-crt-cpp` for credential management.
|
||||
|
||||
All existing S3 URL formats and parameters remain supported, however the store settings for configuring multipart uploads have changed:
|
||||
|
||||
- **`multipart-upload`** (default: `false`): Enable multipart uploads for large files. When enabled, files exceeding the multipart threshold will be uploaded in multiple parts.
|
||||
|
||||
- **`multipart-threshold`** (default: `100 MiB`): Minimum file size for using multipart uploads. Files smaller than this will use regular PUT requests. Only takes effect when `multipart-upload` is enabled.
|
||||
|
||||
- **`multipart-chunk-size`** (default: `5 MiB`): Size of each part in multipart uploads. Must be at least 5 MiB (AWS S3 requirement). Larger chunk sizes reduce the number of requests but use more memory.
|
||||
|
||||
- **`buffer-size`**: Has been replaced by `multipart-chunk-size` and is now an alias to it.
|
||||
|
||||
Note that this change also means Nix now supports S3 binary cache stores even if built without `aws-crt-cpp`, but only for public buckets which do not require authentication.
|
||||
|
||||
- S3 URLs now support object versioning via `versionId` parameter [#13955](https://github.com/NixOS/nix/issues/13955) [#14274](https://github.com/NixOS/nix/pull/14274)
|
||||
|
||||
S3 URLs now support a `versionId` query parameter to fetch specific versions
|
||||
of objects from S3 buckets with versioning enabled. This allows pinning to
|
||||
exact object versions for reproducibility and protection against unexpected
|
||||
changes:
|
||||
|
||||
```
|
||||
s3://bucket/key?region=us-east-1&versionId=abc123def456
|
||||
```
|
||||
|
||||
- S3 binary cache stores now support storage class configuration [#7015](https://github.com/NixOS/nix/issues/7015) [#14464](https://github.com/NixOS/nix/pull/14464)
|
||||
|
||||
S3 binary cache stores now support configuring the storage class for uploaded objects via the `storage-class` parameter. This allows users to optimize costs by selecting appropriate storage tiers based on access patterns.
|
||||
|
||||
Example usage:
|
||||
|
||||
```bash
|
||||
# Use Glacier storage for long-term archival
|
||||
nix copy --to 's3://my-bucket?storage-class=GLACIER' /nix/store/...
|
||||
|
||||
# Use Intelligent Tiering for automatic cost optimization
|
||||
nix copy --to 's3://my-bucket?storage-class=INTELLIGENT_TIERING' /nix/store/...
|
||||
```
|
||||
|
||||
The storage class applies to both regular uploads and multipart uploads. When not specified, objects use the bucket's default storage class.
|
||||
|
||||
See the [S3 storage classes documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) for available storage classes and their characteristics.
|
||||
|
||||
|
||||
## Store path info JSON format changes
|
||||
|
||||
The JSON format emitted by `nix path-info --json` has been updated to a new version with improved structure.
|
||||
|
||||
To maintain compatibility, `nix path-info --json` now requires a `--json-format` flag to specify the output format version.
|
||||
Using `--json` without `--json-format` is deprecated and will become an error in a future release.
|
||||
For now, it defaults to version 1 with a warning, for a smoother migration.
|
||||
|
||||
### Version 1 (`--json-format 1`)
|
||||
|
||||
This is the legacy format, preserved for backwards compatibility:
|
||||
|
||||
- String-based hash values (e.g., `"narHash": "sha256:FePFYIlM..."`)
|
||||
- String-based content addresses (e.g., `"ca": "fixed:r:sha256:1abc..."`)
|
||||
- Full store paths for map keys and references (e.g., `"/nix/store/abc...-foo"`)
|
||||
- Now includes `"storeDir"` field at the top level
|
||||
|
||||
### Version 2 (`--json-format 2`)
|
||||
|
||||
The new structured format follows the [JSON guidelines](@docroot@/development/json-guideline.md) with the following changes:
|
||||
|
||||
- **Nested structure with top-level metadata**:
|
||||
|
||||
The output is now wrapped in an object with `version`, `storeDir`, and `info` fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 2,
|
||||
"storeDir": "/nix/store",
|
||||
"info": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
The map from store path base names to store object info is nested under the `info` field.
|
||||
|
||||
- **Store path base names instead of full paths**:
|
||||
|
||||
Map keys and references use store path base names (e.g., `"abc...-foo"`) instead of full absolute store paths.
|
||||
Combined with `storeDir`, the full path can be reconstructed.
|
||||
|
||||
- **Structured `ca` field**:
|
||||
|
||||
Content address is now a structured JSON object instead of a string:
|
||||
|
||||
- Old: `"ca": "fixed:r:sha256:1abc..."`
|
||||
- New: `"ca": {"method": "nar", "hash": "sha256-ungWv48Bz+pBQUDeXa4iI7ADYaOWF3qctBD/YfIAFa0="}`
|
||||
- Still `null` values for input-addressed store objects
|
||||
|
||||
The `hash` field uses the [SRI](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) format like other hashes.
|
||||
|
||||
Additionally the following fields are added to both formats:
|
||||
|
||||
- **`version` field**:
|
||||
|
||||
All store path info JSON now includes `"version": <1|2>`. The `version` tracks breaking changes, and adding fields to outputted JSON is not a breaking change.
|
||||
|
||||
- **`storeDir` field**:
|
||||
|
||||
Top-level `"storeDir"` field contains the store directory path (e.g., `"/nix/store"`).
|
||||
|
||||
## Derivation JSON format changes
|
||||
|
||||
The derivation JSON format has been updated from version 3 to version 4:
|
||||
|
||||
- **Nested structure with top-level metadata**:
|
||||
|
||||
The output of `nix derivation show` is now wrapped in an object with `version` and `derivations` fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 4,
|
||||
"derivations": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
The map from derivation paths to derivation info is nested under the `derivations` field.
|
||||
|
||||
This matches the structure used for `nix path-info --json --json-format 2`, and likewise brings this command into compliance with the JSON guidelines.
|
||||
|
||||
- **Restructured inputs**:
|
||||
|
||||
Inputs are now nested under an `inputs` object:
|
||||
|
||||
- Old: `"inputSrcs": [...], "inputDrvs": {...}`
|
||||
- New: `"inputs": {"srcs": [...], "drvs": {...}}`
|
||||
|
||||
- **Consistent content addresses**:
|
||||
|
||||
Fixed content-addressed outputs now use structured JSON format.
|
||||
This is the same format as `ca` in store path info (after the new version).
|
||||
|
||||
Version 3 and earlier formats are *not* accepted when reading.
|
||||
|
||||
**Affected command**: `nix derivation`, namely its `show` and `add` sub-commands.
|
||||
|
||||
## Miscellaneous changes
|
||||
|
||||
- Git fetcher: Restore progress indication [#14487](https://github.com/NixOS/nix/pull/14487)
|
||||
|
||||
Nix used to feel "stuck" while it was cloning large repositories. Nix now shows Git's native progress indicator while fetching.
|
||||
|
||||
Upstreamed from [Determinate Nix 3.13.0](https://github.com/DeterminateSystems/nix-src/pull/250).
|
||||
|
||||
- Interrupting REPL commands works more than once [#13481](https://github.com/NixOS/nix/issues/13481)
|
||||
|
||||
Previously, this only worked once per REPL session; further attempts would be ignored.
|
||||
This issue is now fixed, so REPL commands such as `:b` or `:p` can be canceled consistently.
|
||||
This is a cherry-pick of the change from the [Lix project](https://gerrit.lix.systems/c/lix/+/1097).
|
||||
|
||||
- NAR unpacking code has been rewritten to make use of dirfd-based `openat` and `openat2` system calls when available [#14597](https://github.com/NixOS/nix/pull/14597)
|
||||
|
||||
- Dynamic size unit rendering [#14423](https://github.com/NixOS/nix/pull/14423) [#14364](https://github.com/NixOS/nix/pull/14364)
|
||||
|
||||
Various commands and the progress bar now use dynamically determined size units instead
|
||||
of always using `MiB`. For example, the progress bar now reports download status like:
|
||||
|
||||
```
|
||||
[1/196/197 copied (773.7 MiB/2.1 GiB), 172.4/421.5 MiB DL]
|
||||
```
|
||||
|
||||
Instead of:
|
||||
|
||||
```
|
||||
[1/196/197 copied (773.7/2147.3 MiB), 172.4/421.5 MiB DL]
|
||||
```
|
||||
|
||||
## Contributors
|
||||
|
||||
This release was made possible by the following 33 contributors:
|
||||
|
||||
- Adam Dinwoodie [**(@me-and)**](https://github.com/me-and)
|
||||
- jonhermansen [**(@jonhermansen)**](https://github.com/jonhermansen)
|
||||
- Arnout Engelen [**(@raboof)**](https://github.com/raboof)
|
||||
- Jean-François Roche [**(@jfroche)**](https://github.com/jfroche)
|
||||
- tomberek [**(@tomberek)**](https://github.com/tomberek)
|
||||
- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
|
||||
- Marcel [**(@MarcelCoding)**](https://github.com/MarcelCoding)
|
||||
- David McFarland [**(@corngood)**](https://github.com/corngood)
|
||||
- Soumyadip Sarkar [**(@neuralsorcerer)**](https://github.com/neuralsorcerer)
|
||||
- Cole Helbling [**(@cole-h)**](https://github.com/cole-h)
|
||||
- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
|
||||
- Tristan Ross [**(@RossComputerGuy)**](https://github.com/RossComputerGuy)
|
||||
- Alex Auvolat [**(@Alexis211)**](https://github.com/Alexis211)
|
||||
- edef [**(@edef1c)**](https://github.com/edef1c)
|
||||
- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
|
||||
- Vinayak Goyal [**(@vinayakankugoyal)**](https://github.com/vinayakankugoyal)
|
||||
- Graham Dennis [**(@GrahamDennis)**](https://github.com/GrahamDennis)
|
||||
- Aspen Smith [**(@glittershark)**](https://github.com/glittershark)
|
||||
- Jens Petersen [**(@juhp)**](https://github.com/juhp)
|
||||
- Bernardo Meurer [**(@lovesegfault)**](https://github.com/lovesegfault)
|
||||
- Peter Bynum [**(@pkpbynum)**](https://github.com/pkpbynum)
|
||||
- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
|
||||
- Alex Decious [**(@adeci)**](https://github.com/adeci)
|
||||
- Matthieu Coudron [**(@teto)**](https://github.com/teto)
|
||||
- Domen Kožar [**(@domenkozar)**](https://github.com/domenkozar)
|
||||
- Taeer Bar-Yam [**(@Radvendii)**](https://github.com/Radvendii)
|
||||
- Seth Flynn [**(@getchoo)**](https://github.com/getchoo)
|
||||
- Robert Hensing [**(@roberth)**](https://github.com/roberth)
|
||||
- Vladimir Panteleev [**(@CyberShadow)**](https://github.com/CyberShadow)
|
||||
- bryango [**(@bryango)**](https://github.com/bryango)
|
||||
- Henry [**(@cootshk)**](https://github.com/cootshk)
|
||||
- Martin Joerg [**(@mjoerg)**](https://github.com/mjoerg)
|
||||
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
|
||||
@@ -224,25 +224,5 @@
|
||||
"42688647+netadr@users.noreply.github.com": "netadr",
|
||||
"matej.urbas@gmail.com": "urbas",
|
||||
"ethanalexevans@gmail.com": "ethanavatar",
|
||||
"greg.marti@gmail.com": "gmarti",
|
||||
"arnout@bzzt.net": "raboof",
|
||||
"vinayakankugoyal@gmail.com": "vinayakankugoyal",
|
||||
"Radvendii@users.noreply.github.com": "Radvendii",
|
||||
"jon@jh86.org": "jonhermansen",
|
||||
"edef@edef.eu": "edef1c",
|
||||
"pkpbynum@gmail.com": "pkpbynum",
|
||||
"886074+teto@users.noreply.github.com": "teto",
|
||||
"alex@adnab.me": "Alexis211",
|
||||
"root@gws.fyi": "glittershark",
|
||||
"me@m4rc3l.de": "MarcelCoding",
|
||||
"taeer.bar-yam@bevuta.com": "Radvendii",
|
||||
"martin.joerg@gmail.com": "mjoerg",
|
||||
"git@cy.md": "CyberShadow",
|
||||
"cootshk@duck.com": "cootshk",
|
||||
"adam@dinwoodie.org": "me-and",
|
||||
"domen@cachix.org": "domenkozar",
|
||||
"alex.decious@gmail.com": "adeci",
|
||||
"soumya.papanvk18@gmail.com": "neuralsorcerer",
|
||||
"gdennis@anduril.com": null,
|
||||
"graham.dennis@gmail.com": "GrahamDennis"
|
||||
"greg.marti@gmail.com": "gmarti"
|
||||
}
|
||||
@@ -196,21 +196,5 @@
|
||||
"gmarti": "Gr\u00e9gory Marti",
|
||||
"lovesegfault": "Bernardo Meurer",
|
||||
"EphraimSiegfried": "Ephraim Siegfried",
|
||||
"hgl": "Glen Huang",
|
||||
"mjoerg": "Martin Joerg",
|
||||
"Alexis211": "Alex Auvolat",
|
||||
"domenkozar": "Domen Ko\u017ear",
|
||||
"edef1c": "edef",
|
||||
"cootshk": "Henry",
|
||||
"raboof": "Arnout Engelen",
|
||||
"pkpbynum": "Peter Bynum",
|
||||
"glittershark": "Aspen Smith",
|
||||
"MarcelCoding": "Marcel",
|
||||
"teto": "Matthieu Coudron",
|
||||
"jonhermansen": null,
|
||||
"neuralsorcerer": "Soumyadip Sarkar",
|
||||
"adeci": "Alex Decious",
|
||||
"vinayakankugoyal": "Vinayak Goyal",
|
||||
"me-and": "Adam Dinwoodie",
|
||||
"GrahamDennis": "Graham Dennis"
|
||||
"hgl": "Glen Huang"
|
||||
}
|
||||
@@ -258,13 +258,10 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
|
||||
|
||||
# We use this shell with the local checkout, not unpackPhase.
|
||||
src = null;
|
||||
|
||||
# Workaround https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
|
||||
# Remove when gdb fix is rolled out everywhere.
|
||||
separateDebugInfo = false;
|
||||
|
||||
mesonBuildType = "debugoptimized";
|
||||
|
||||
env = {
|
||||
# For `make format`, to work without installing pre-commit
|
||||
_NIX_PRE_COMMIT_HOOKS_CONFIG = "${(pkgs.formats.yaml { }).generate "pre-commit-config.yaml"
|
||||
|
||||
@@ -48,7 +48,7 @@ public:
|
||||
|
||||
ref<GitRepo> openRepo()
|
||||
{
|
||||
return GitRepo::openRepo(tmpDir, {.create = true});
|
||||
return GitRepo::openRepo(tmpDir, true, false);
|
||||
}
|
||||
|
||||
std::string getRepoName() const
|
||||
|
||||
@@ -203,19 +203,16 @@ static git_packbuilder_progress PACKBUILDER_PROGRESS_CHECK_INTERRUPT = &packBuil
|
||||
|
||||
} // extern "C"
|
||||
|
||||
static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options options)
|
||||
static void initRepoAtomically(std::filesystem::path & path, bool bare)
|
||||
{
|
||||
if (pathExists(path.string()))
|
||||
return;
|
||||
|
||||
if (!options.create)
|
||||
throw Error("Git repository %s does not exist.", path);
|
||||
|
||||
std::filesystem::path tmpDir = createTempDir(path.parent_path());
|
||||
AutoDelete delTmpDir(tmpDir, true);
|
||||
Repository tmpRepo;
|
||||
|
||||
if (git_repository_init(Setter(tmpRepo), tmpDir.string().c_str(), options.bare))
|
||||
if (git_repository_init(Setter(tmpRepo), tmpDir.string().c_str(), bare))
|
||||
throw Error("creating Git repository %s: %s", path, git_error_last()->message);
|
||||
try {
|
||||
std::filesystem::rename(tmpDir, path);
|
||||
@@ -237,7 +234,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
|
||||
/** Location of the repository on disk. */
|
||||
std::filesystem::path path;
|
||||
|
||||
Options options;
|
||||
bool bare;
|
||||
|
||||
/**
|
||||
* libgit2 repository. Note that new objects are not written to disk,
|
||||
@@ -258,18 +255,18 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
|
||||
*/
|
||||
git_odb_backend * packBackend = nullptr;
|
||||
|
||||
GitRepoImpl(std::filesystem::path _path, Options _options)
|
||||
GitRepoImpl(std::filesystem::path _path, bool create, bool bare, bool packfilesOnly = false)
|
||||
: path(std::move(_path))
|
||||
, options(_options)
|
||||
, bare(bare)
|
||||
{
|
||||
initLibGit2();
|
||||
|
||||
initRepoAtomically(path, options);
|
||||
initRepoAtomically(path, bare);
|
||||
if (git_repository_open(Setter(repo), path.string().c_str()))
|
||||
throw Error("opening Git repository %s: %s", path, git_error_last()->message);
|
||||
|
||||
ObjectDb odb;
|
||||
if (options.packfilesOnly) {
|
||||
if (packfilesOnly) {
|
||||
/* Create a fresh object database because by default the repo also
|
||||
loose object backends. We are not using any of those for the
|
||||
tarball cache, but libgit2 still does a bunch of unnecessary
|
||||
@@ -298,7 +295,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
|
||||
if (git_odb_add_backend(odb.get(), mempackBackend, 999))
|
||||
throw Error("adding mempack backend to Git object database: %s", git_error_last()->message);
|
||||
|
||||
if (options.packfilesOnly) {
|
||||
if (packfilesOnly) {
|
||||
if (git_repository_set_odb(repo.get(), odb.get()))
|
||||
throw Error("setting Git object database: %s", git_error_last()->message);
|
||||
}
|
||||
@@ -369,7 +366,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
|
||||
{
|
||||
// TODO: as an optimization, it would be nice to include `this` in the pool.
|
||||
return Pool<GitRepoImpl>(std::numeric_limits<size_t>::max(), [this]() -> ref<GitRepoImpl> {
|
||||
return make_ref<GitRepoImpl>(path, options);
|
||||
return make_ref<GitRepoImpl>(path, false, bare);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -715,9 +712,9 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
|
||||
}
|
||||
};
|
||||
|
||||
ref<GitRepo> GitRepo::openRepo(const std::filesystem::path & path, GitRepo::Options options)
|
||||
ref<GitRepo> GitRepo::openRepo(const std::filesystem::path & path, bool create, bool bare, bool packfilesOnly)
|
||||
{
|
||||
return make_ref<GitRepoImpl>(path, options);
|
||||
return make_ref<GitRepoImpl>(path, create, bare, packfilesOnly);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1430,12 +1427,8 @@ namespace fetchers {
|
||||
|
||||
ref<GitRepo> Settings::getTarballCache() const
|
||||
{
|
||||
/* v1: Had either only loose objects or thin packfiles referring to loose objects
|
||||
* v2: Must have only packfiles with no loose objects. Should get repacked periodically
|
||||
* for optimal packfiles.
|
||||
*/
|
||||
static auto repoDir = std::filesystem::path(getCacheDir()) / "tarball-cache-v2";
|
||||
return GitRepo::openRepo(repoDir, {.create = true, .bare = true, .packfilesOnly = true});
|
||||
static auto repoDir = std::filesystem::path(getCacheDir()) / "tarball-cache";
|
||||
return GitRepo::openRepo(repoDir, /*create=*/true, /*bare=*/true, /*packfilesOnly=*/true);
|
||||
}
|
||||
|
||||
} // namespace fetchers
|
||||
@@ -1449,7 +1442,7 @@ GitRepo::WorkdirInfo GitRepo::getCachedWorkdirInfo(const std::filesystem::path &
|
||||
if (i != cache->end())
|
||||
return i->second;
|
||||
}
|
||||
auto workdirInfo = GitRepo::openRepo(path, {})->getWorkdirInfo();
|
||||
auto workdirInfo = GitRepo::openRepo(path)->getWorkdirInfo();
|
||||
_cache.lock()->emplace(path, workdirInfo);
|
||||
return workdirInfo;
|
||||
}
|
||||
|
||||
@@ -637,6 +637,11 @@ struct GitInputScheme : InputScheme
|
||||
url);
|
||||
}
|
||||
|
||||
// If we don't check here for the path existence, then we can give libgit2 any directory
|
||||
// and it will initialize them as git directories.
|
||||
if (!pathExists(path)) {
|
||||
throw Error("The path '%s' does not exist.", path);
|
||||
}
|
||||
repoInfo.location = std::filesystem::absolute(path);
|
||||
} else {
|
||||
if (url.scheme == "file")
|
||||
@@ -698,7 +703,7 @@ struct GitInputScheme : InputScheme
|
||||
if (auto res = cache->lookup(key))
|
||||
return getIntAttr(*res, "lastModified");
|
||||
|
||||
auto lastModified = GitRepo::openRepo(repoDir, {})->getLastModified(rev);
|
||||
auto lastModified = GitRepo::openRepo(repoDir)->getLastModified(rev);
|
||||
|
||||
cache->upsert(key, {{"lastModified", lastModified}});
|
||||
|
||||
@@ -721,7 +726,7 @@ struct GitInputScheme : InputScheme
|
||||
Activity act(
|
||||
*logger, lvlChatty, actUnknown, fmt("getting Git revision count of '%s'", repoInfo.locationToArg()));
|
||||
|
||||
auto revCount = GitRepo::openRepo(repoDir, {})->getRevCount(rev);
|
||||
auto revCount = GitRepo::openRepo(repoDir)->getRevCount(rev);
|
||||
|
||||
cache->upsert(key, Attrs{{"revCount", revCount}});
|
||||
|
||||
@@ -732,7 +737,7 @@ struct GitInputScheme : InputScheme
|
||||
{
|
||||
auto head = std::visit(
|
||||
overloaded{
|
||||
[&](const std::filesystem::path & path) { return GitRepo::openRepo(path, {})->getWorkdirRef(); },
|
||||
[&](const std::filesystem::path & path) { return GitRepo::openRepo(path)->getWorkdirRef(); },
|
||||
[&](const ParsedURL & url) { return readHeadCached(url.to_string(), shallow); }},
|
||||
repoInfo.location);
|
||||
if (!head) {
|
||||
@@ -790,7 +795,7 @@ struct GitInputScheme : InputScheme
|
||||
if (auto repoPath = repoInfo.getPath()) {
|
||||
repoDir = *repoPath;
|
||||
if (!input.getRev())
|
||||
input.attrs.insert_or_assign("rev", GitRepo::openRepo(repoDir, {})->resolveRef(ref).gitRev());
|
||||
input.attrs.insert_or_assign("rev", GitRepo::openRepo(repoDir)->resolveRef(ref).gitRev());
|
||||
} else {
|
||||
auto repoUrl = std::get<ParsedURL>(repoInfo.location);
|
||||
std::filesystem::path cacheDir = getCachePath(repoUrl.to_string(), shallow);
|
||||
@@ -800,7 +805,7 @@ struct GitInputScheme : InputScheme
|
||||
std::filesystem::create_directories(cacheDir.parent_path());
|
||||
PathLocks cacheDirLock({cacheDir.string()});
|
||||
|
||||
auto repo = GitRepo::openRepo(cacheDir, {.create = true, .bare = true});
|
||||
auto repo = GitRepo::openRepo(cacheDir, true, true);
|
||||
|
||||
// We need to set the origin so resolving submodule URLs works
|
||||
repo->setRemote("origin", repoUrl.to_string());
|
||||
@@ -871,7 +876,7 @@ struct GitInputScheme : InputScheme
|
||||
// the remainder
|
||||
}
|
||||
|
||||
auto repo = GitRepo::openRepo(repoDir, {});
|
||||
auto repo = GitRepo::openRepo(repoDir);
|
||||
|
||||
auto isShallow = repo->isShallow();
|
||||
|
||||
@@ -958,7 +963,7 @@ struct GitInputScheme : InputScheme
|
||||
for (auto & submodule : repoInfo.workdirInfo.submodules)
|
||||
repoInfo.workdirInfo.files.insert(submodule.path);
|
||||
|
||||
auto repo = GitRepo::openRepo(repoPath, {});
|
||||
auto repo = GitRepo::openRepo(repoPath, false, false);
|
||||
|
||||
auto exportIgnore = getExportIgnoreAttr(input);
|
||||
|
||||
@@ -998,7 +1003,7 @@ struct GitInputScheme : InputScheme
|
||||
}
|
||||
|
||||
if (!repoInfo.workdirInfo.isDirty) {
|
||||
auto repo = GitRepo::openRepo(repoPath, {});
|
||||
auto repo = GitRepo::openRepo(repoPath);
|
||||
|
||||
if (auto ref = repo->getWorkdirRef())
|
||||
input.attrs.insert_or_assign("ref", *ref);
|
||||
|
||||
@@ -32,14 +32,8 @@ struct GitRepo
|
||||
{
|
||||
virtual ~GitRepo() {}
|
||||
|
||||
struct Options
|
||||
{
|
||||
bool create = false;
|
||||
bool bare = false;
|
||||
bool packfilesOnly = false;
|
||||
};
|
||||
|
||||
static ref<GitRepo> openRepo(const std::filesystem::path & path, Options options);
|
||||
static ref<GitRepo>
|
||||
openRepo(const std::filesystem::path & path, bool create = false, bool bare = false, bool packfilesOnly = false);
|
||||
|
||||
virtual uint64_t getRevCount(const Hash & rev) = 0;
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
#include "nix/store/derivations.hh"
|
||||
#include "nix/store/store-api.hh"
|
||||
#include "nix/util/experimental-features.hh"
|
||||
#include "nix/util/tests/test-data.hh"
|
||||
#include "nix/util/environment-variables.hh"
|
||||
#include "nix/store/store-open.hh"
|
||||
#include <fstream>
|
||||
#include <sstream>
|
||||
@@ -50,7 +50,11 @@ static void BM_UnparseRealDerivationFile(benchmark::State & state, const std::st
|
||||
}
|
||||
|
||||
// Register benchmarks for actual test derivation files if they exist
|
||||
BENCHMARK_CAPTURE(BM_ParseRealDerivationFile, hello, (getUnitTestData() / "derivation/hello.drv").string());
|
||||
BENCHMARK_CAPTURE(BM_ParseRealDerivationFile, firefox, (getUnitTestData() / "derivation/firefox.drv").string());
|
||||
BENCHMARK_CAPTURE(BM_UnparseRealDerivationFile, hello, (getUnitTestData() / "derivation/hello.drv").string());
|
||||
BENCHMARK_CAPTURE(BM_UnparseRealDerivationFile, firefox, (getUnitTestData() / "derivation/firefox.drv").string());
|
||||
BENCHMARK_CAPTURE(
|
||||
BM_ParseRealDerivationFile, hello, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/hello.drv");
|
||||
BENCHMARK_CAPTURE(
|
||||
BM_ParseRealDerivationFile, firefox, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/firefox.drv");
|
||||
BENCHMARK_CAPTURE(
|
||||
BM_UnparseRealDerivationFile, hello, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/hello.drv");
|
||||
BENCHMARK_CAPTURE(
|
||||
BM_UnparseRealDerivationFile, firefox, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/firefox.drv");
|
||||
|
||||
@@ -8,7 +8,6 @@
|
||||
#include "nix/store/tests/nix_api_store.hh"
|
||||
#include "nix/store/globals.hh"
|
||||
#include "nix/util/tests/string_callback.hh"
|
||||
#include "nix/util/tests/test-data.hh"
|
||||
#include "nix/util/url.hh"
|
||||
|
||||
#include "store-tests-config.hh"
|
||||
@@ -303,7 +302,7 @@ public:
|
||||
|
||||
store = open_local_store();
|
||||
|
||||
std::filesystem::path unitTestData = nix::getUnitTestData();
|
||||
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
|
||||
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
|
||||
std::stringstream buffer;
|
||||
buffer << t.rdbuf();
|
||||
@@ -358,7 +357,7 @@ TEST_F(nix_api_store_test_base, build_from_json)
|
||||
|
||||
auto * store = open_local_store();
|
||||
|
||||
std::filesystem::path unitTestData = nix::getUnitTestData();
|
||||
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
|
||||
|
||||
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
|
||||
std::stringstream buffer;
|
||||
@@ -405,7 +404,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_invalid_system)
|
||||
|
||||
auto * store = open_local_store();
|
||||
|
||||
std::filesystem::path unitTestData = nix::getUnitTestData();
|
||||
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
|
||||
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
|
||||
std::stringstream buffer;
|
||||
buffer << t.rdbuf();
|
||||
@@ -450,7 +449,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_builder_fails)
|
||||
|
||||
auto * store = open_local_store();
|
||||
|
||||
std::filesystem::path unitTestData = nix::getUnitTestData();
|
||||
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
|
||||
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
|
||||
std::stringstream buffer;
|
||||
buffer << t.rdbuf();
|
||||
@@ -495,7 +494,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_builder_no_output)
|
||||
|
||||
auto * store = open_local_store();
|
||||
|
||||
std::filesystem::path unitTestData = nix::getUnitTestData();
|
||||
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
|
||||
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
|
||||
std::stringstream buffer;
|
||||
buffer << t.rdbuf();
|
||||
@@ -871,7 +870,7 @@ TEST_F(NixApiStoreTestWithRealisedPath, nix_store_get_fs_closure_error_propagati
|
||||
*/
|
||||
static std::string load_json_from_test_data(const char * filename)
|
||||
{
|
||||
std::filesystem::path unitTestData = nix::getUnitTestData();
|
||||
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
|
||||
std::ifstream t{unitTestData / filename};
|
||||
std::stringstream buffer;
|
||||
buffer << t.rdbuf();
|
||||
|
||||
@@ -4,15 +4,15 @@
|
||||
|
||||
# include <aws/crt/Types.h>
|
||||
# include "nix/store/s3-url.hh"
|
||||
# include "nix/util/finally.hh"
|
||||
# include "nix/util/logging.hh"
|
||||
# include "nix/util/url.hh"
|
||||
# include "nix/util/util.hh"
|
||||
|
||||
# include <aws/crt/Api.h>
|
||||
# include <aws/crt/auth/Credentials.h>
|
||||
# include <aws/crt/io/Bootstrap.h>
|
||||
|
||||
// C library headers for SSO provider support
|
||||
# include <aws/auth/credentials.h>
|
||||
|
||||
# include <boost/unordered/concurrent_flat_map.hpp>
|
||||
|
||||
# include <chrono>
|
||||
@@ -30,46 +30,6 @@ AwsAuthError::AwsAuthError(int errorCode)
|
||||
|
||||
namespace {
|
||||
|
||||
/**
|
||||
* Helper function to wrap a C credentials provider in the C++ interface.
|
||||
* This replicates the static s_CreateWrappedProvider from aws-crt-cpp.
|
||||
*/
|
||||
static std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider> createWrappedProvider(
|
||||
aws_credentials_provider * rawProvider, Aws::Crt::Allocator * allocator = Aws::Crt::ApiAllocator())
|
||||
{
|
||||
if (rawProvider == nullptr) {
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
auto provider = Aws::Crt::MakeShared<Aws::Crt::Auth::CredentialsProvider>(allocator, rawProvider, allocator);
|
||||
return std::static_pointer_cast<Aws::Crt::Auth::ICredentialsProvider>(provider);
|
||||
}
|
||||
|
||||
/**
|
||||
* Create an SSO credentials provider using the C library directly.
|
||||
* The C++ wrapper doesn't expose SSO, so we call the C library and wrap the result.
|
||||
* Returns nullptr if SSO provider creation fails (e.g., profile doesn't have SSO config).
|
||||
*/
|
||||
static std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider> createSSOProvider(
|
||||
const std::string & profileName,
|
||||
Aws::Crt::Io::ClientBootstrap * bootstrap,
|
||||
Aws::Crt::Io::TlsContext * tlsContext,
|
||||
Aws::Crt::Allocator * allocator = Aws::Crt::ApiAllocator())
|
||||
{
|
||||
aws_credentials_provider_sso_options options;
|
||||
AWS_ZERO_STRUCT(options);
|
||||
|
||||
options.bootstrap = bootstrap->GetUnderlyingHandle();
|
||||
options.tls_ctx = tlsContext ? tlsContext->GetUnderlyingHandle() : nullptr;
|
||||
if (!profileName.empty()) {
|
||||
options.profile_name_override = aws_byte_cursor_from_c_str(profileName.c_str());
|
||||
}
|
||||
|
||||
// Create the SSO provider - will return nullptr if SSO isn't configured for this profile
|
||||
// createWrappedProvider handles nullptr gracefully
|
||||
return createWrappedProvider(aws_credentials_provider_new_sso(allocator, &options), allocator);
|
||||
}
|
||||
|
||||
static AwsCredentials getCredentialsFromProvider(std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider> provider)
|
||||
{
|
||||
if (!provider || !provider->IsValid()) {
|
||||
@@ -131,22 +91,6 @@ public:
|
||||
logLevel = Aws::Crt::LogLevel::Warn;
|
||||
}
|
||||
apiHandle.InitializeLogging(logLevel, stderr);
|
||||
|
||||
// Create a shared TLS context for SSO (required for HTTPS connections)
|
||||
auto allocator = Aws::Crt::ApiAllocator();
|
||||
auto tlsCtxOptions = Aws::Crt::Io::TlsContextOptions::InitDefaultClient(allocator);
|
||||
tlsContext =
|
||||
std::make_shared<Aws::Crt::Io::TlsContext>(tlsCtxOptions, Aws::Crt::Io::TlsMode::CLIENT, allocator);
|
||||
if (!tlsContext || !*tlsContext) {
|
||||
warn("failed to create TLS context for AWS SSO; SSO authentication will be unavailable");
|
||||
tlsContext = nullptr;
|
||||
}
|
||||
|
||||
// Get bootstrap (lives as long as apiHandle)
|
||||
bootstrap = Aws::Crt::ApiHandle::GetOrCreateStaticDefaultClientBootstrap();
|
||||
if (!bootstrap) {
|
||||
throw AwsAuthError("failed to create AWS client bootstrap");
|
||||
}
|
||||
}
|
||||
|
||||
AwsCredentials getCredentialsRaw(const std::string & profile);
|
||||
@@ -167,8 +111,6 @@ public:
|
||||
|
||||
private:
|
||||
Aws::Crt::ApiHandle apiHandle;
|
||||
std::shared_ptr<Aws::Crt::Io::TlsContext> tlsContext;
|
||||
Aws::Crt::Io::ClientBootstrap * bootstrap;
|
||||
boost::concurrent_flat_map<std::string, std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider>>
|
||||
credentialProviderCache;
|
||||
};
|
||||
@@ -176,58 +118,23 @@ private:
|
||||
std::shared_ptr<Aws::Crt::Auth::ICredentialsProvider>
|
||||
AwsCredentialProviderImpl::createProviderForProfile(const std::string & profile)
|
||||
{
|
||||
// profileDisplayName is only used for debug logging - SDK uses its default profile
|
||||
// when ProfileNameOverride is not set
|
||||
const char * profileDisplayName = profile.empty() ? "(default)" : profile.c_str();
|
||||
debug(
|
||||
"[pid=%d] creating new AWS credential provider for profile '%s'",
|
||||
getpid(),
|
||||
profile.empty() ? "(default)" : profile.c_str());
|
||||
|
||||
debug("[pid=%d] creating new AWS credential provider for profile '%s'", getpid(), profileDisplayName);
|
||||
|
||||
// Build a custom credential chain: Environment → SSO → Profile → IMDS
|
||||
// This works for both default and named profiles, ensuring consistent behavior
|
||||
// including SSO support and proper TLS context for STS-based role assumption.
|
||||
Aws::Crt::Auth::CredentialsProviderChainConfig chainConfig;
|
||||
auto allocator = Aws::Crt::ApiAllocator();
|
||||
|
||||
auto addProviderToChain = [&](std::string_view name, auto createProvider) {
|
||||
if (auto provider = createProvider()) {
|
||||
chainConfig.Providers.push_back(provider);
|
||||
debug("Added AWS %s Credential Provider to chain for profile '%s'", name, profileDisplayName);
|
||||
} else {
|
||||
debug("Skipped AWS %s Credential Provider for profile '%s'", name, profileDisplayName);
|
||||
}
|
||||
};
|
||||
|
||||
// 1. Environment variables (highest priority)
|
||||
addProviderToChain("Environment", [&]() {
|
||||
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderEnvironment(allocator);
|
||||
});
|
||||
|
||||
// 2. SSO provider (try it, will fail gracefully if not configured)
|
||||
if (tlsContext) {
|
||||
addProviderToChain("SSO", [&]() { return createSSOProvider(profile, bootstrap, tlsContext.get(), allocator); });
|
||||
} else {
|
||||
debug("Skipped AWS SSO Credential Provider for profile '%s': TLS context unavailable", profileDisplayName);
|
||||
if (profile.empty()) {
|
||||
Aws::Crt::Auth::CredentialsProviderChainDefaultConfig config;
|
||||
config.Bootstrap = Aws::Crt::ApiHandle::GetOrCreateStaticDefaultClientBootstrap();
|
||||
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderChainDefault(config);
|
||||
}
|
||||
|
||||
// 3. Profile provider (for static credentials and role_arn/source_profile with STS)
|
||||
addProviderToChain("Profile", [&]() {
|
||||
Aws::Crt::Auth::CredentialsProviderProfileConfig profileConfig;
|
||||
profileConfig.Bootstrap = bootstrap;
|
||||
profileConfig.TlsContext = tlsContext.get();
|
||||
if (!profile.empty()) {
|
||||
profileConfig.ProfileNameOverride = Aws::Crt::ByteCursorFromCString(profile.c_str());
|
||||
}
|
||||
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderProfile(profileConfig, allocator);
|
||||
});
|
||||
|
||||
// 4. IMDS provider (for EC2 instances, lowest priority)
|
||||
addProviderToChain("IMDS", [&]() {
|
||||
Aws::Crt::Auth::CredentialsProviderImdsConfig imdsConfig;
|
||||
imdsConfig.Bootstrap = bootstrap;
|
||||
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderImds(imdsConfig, allocator);
|
||||
});
|
||||
|
||||
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderChain(chainConfig, allocator);
|
||||
Aws::Crt::Auth::CredentialsProviderProfileConfig config;
|
||||
config.Bootstrap = Aws::Crt::ApiHandle::GetOrCreateStaticDefaultClientBootstrap();
|
||||
// This is safe because the underlying C library will copy this string
|
||||
// c.f. https://github.com/awslabs/aws-c-auth/blob/main/source/credentials_provider_profile.c#L220
|
||||
config.ProfileNameOverride = Aws::Crt::ByteCursorFromCString(profile.c_str());
|
||||
return Aws::Crt::Auth::CredentialsProvider::CreateCredentialsProviderProfile(config);
|
||||
}
|
||||
|
||||
AwsCredentials AwsCredentialProviderImpl::getCredentialsRaw(const std::string & profile)
|
||||
|
||||
@@ -71,6 +71,14 @@ void DerivationBuildingGoal::killChild()
|
||||
#endif
|
||||
}
|
||||
|
||||
void DerivationBuildingGoal::timedOut(Error && ex)
|
||||
{
|
||||
killChild();
|
||||
// We're not inside a coroutine, hence we can't use co_return here.
|
||||
// Thus we ignore the return value.
|
||||
[[maybe_unused]] Done _ = doneFailure({BuildResult::Failure::TimedOut, std::move(ex)});
|
||||
}
|
||||
|
||||
std::string showKnownOutputs(const StoreDirConfig & store, const Derivation & drv)
|
||||
{
|
||||
std::string msg;
|
||||
@@ -435,20 +443,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
|
||||
if (useHook) {
|
||||
buildResult.startTime = time(0); // inexact
|
||||
started();
|
||||
|
||||
while (true) {
|
||||
auto event = co_await WaitForChildEvent{};
|
||||
if (auto * output = std::get_if<ChildOutput>(&event)) {
|
||||
co_await processChildOutput(output->fd, output->data);
|
||||
} else if (std::get_if<ChildEOF>(&event)) {
|
||||
if (!currentLogLine.empty())
|
||||
flushLine();
|
||||
break;
|
||||
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
|
||||
killChild();
|
||||
co_return doneFailure(std::move(*timeout));
|
||||
}
|
||||
}
|
||||
co_await Suspend{};
|
||||
|
||||
#ifndef _WIN32
|
||||
assert(hook);
|
||||
@@ -669,20 +664,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
|
||||
worker.childStarted(shared_from_this(), {builderOut}, true, true);
|
||||
|
||||
started();
|
||||
|
||||
while (true) {
|
||||
auto event = co_await WaitForChildEvent{};
|
||||
if (auto * output = std::get_if<ChildOutput>(&event)) {
|
||||
co_await processChildOutput(output->fd, output->data);
|
||||
} else if (std::get_if<ChildEOF>(&event)) {
|
||||
if (!currentLogLine.empty())
|
||||
flushLine();
|
||||
break;
|
||||
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
|
||||
killChild();
|
||||
co_return doneFailure(std::move(*timeout));
|
||||
}
|
||||
}
|
||||
co_await Suspend{};
|
||||
|
||||
trace("build done");
|
||||
|
||||
@@ -1015,7 +997,7 @@ bool DerivationBuildingGoal::isReadDesc(Descriptor fd)
|
||||
#endif
|
||||
}
|
||||
|
||||
Goal::Co DerivationBuildingGoal::processChildOutput(Descriptor fd, std::string_view data)
|
||||
void DerivationBuildingGoal::handleChildOutput(Descriptor fd, std::string_view data)
|
||||
{
|
||||
// local & `ssh://`-builds are dealt with here.
|
||||
auto isWrittenToLog = isReadDesc(fd);
|
||||
@@ -1023,11 +1005,14 @@ Goal::Co DerivationBuildingGoal::processChildOutput(Descriptor fd, std::string_v
|
||||
logSize += data.size();
|
||||
if (settings.maxLogSize && logSize > settings.maxLogSize) {
|
||||
killChild();
|
||||
co_return doneFailure(BuildError(
|
||||
// We're not inside a coroutine, hence we can't use co_return here.
|
||||
// Thus we ignore the return value.
|
||||
[[maybe_unused]] Done _ = doneFailure(BuildError(
|
||||
BuildResult::Failure::LogLimitExceeded,
|
||||
"%s killed after writing more than %d bytes of log output",
|
||||
getName(),
|
||||
settings.maxLogSize));
|
||||
return;
|
||||
}
|
||||
|
||||
for (auto c : data)
|
||||
@@ -1080,7 +1065,13 @@ Goal::Co DerivationBuildingGoal::processChildOutput(Descriptor fd, std::string_v
|
||||
currentHookLine += c;
|
||||
}
|
||||
#endif
|
||||
co_return Return{};
|
||||
}
|
||||
|
||||
void DerivationBuildingGoal::handleEOF(Descriptor fd)
|
||||
{
|
||||
if (!currentLogLine.empty())
|
||||
flushLine();
|
||||
worker.wakeUp(shared_from_this());
|
||||
}
|
||||
|
||||
void DerivationBuildingGoal::flushLine()
|
||||
|
||||
@@ -66,16 +66,7 @@ Goal::Co DrvOutputSubstitutionGoal::init()
|
||||
true,
|
||||
false);
|
||||
|
||||
while (true) {
|
||||
auto event = co_await WaitForChildEvent{};
|
||||
if (std::get_if<ChildOutput>(&event)) {
|
||||
// Doesn't process child output
|
||||
} else if (std::get_if<ChildEOF>(&event)) {
|
||||
break;
|
||||
} else if (std::get_if<TimedOut>(&event)) {
|
||||
unreachable();
|
||||
}
|
||||
}
|
||||
co_await Suspend{};
|
||||
|
||||
worker.childTerminated(this);
|
||||
|
||||
@@ -158,4 +149,9 @@ std::string DrvOutputSubstitutionGoal::key()
|
||||
return "a$" + std::string(id.to_string());
|
||||
}
|
||||
|
||||
void DrvOutputSubstitutionGoal::handleEOF(Descriptor fd)
|
||||
{
|
||||
worker.wakeUp(shared_from_this());
|
||||
}
|
||||
|
||||
} // namespace nix
|
||||
|
||||
@@ -4,58 +4,8 @@
|
||||
|
||||
namespace nix {
|
||||
|
||||
TimedOut::TimedOut(time_t maxDuration)
|
||||
: BuildError(BuildResult::Failure::TimedOut, "timed out after %1% seconds", maxDuration)
|
||||
, maxDuration(maxDuration)
|
||||
{
|
||||
}
|
||||
|
||||
using Co = nix::Goal::Co;
|
||||
using promise_type = nix::Goal::promise_type;
|
||||
using ChildEvents = decltype(promise_type::childEvents);
|
||||
|
||||
void ChildEvents::pushChildEvent(ChildOutput event)
|
||||
{
|
||||
if (childTimeout)
|
||||
return; // Already timed out, ignore
|
||||
childOutputs.push(std::move(event));
|
||||
}
|
||||
|
||||
void ChildEvents::pushChildEvent(ChildEOF event)
|
||||
{
|
||||
if (childTimeout)
|
||||
return; // Already timed out, ignore
|
||||
assert(!childEOF);
|
||||
childEOF = std::move(event);
|
||||
}
|
||||
|
||||
void ChildEvents::pushChildEvent(TimedOut event)
|
||||
{
|
||||
// Timeout is immediate - flush pending events
|
||||
childOutputs = {};
|
||||
childEOF.reset();
|
||||
childTimeout = std::move(event);
|
||||
}
|
||||
|
||||
bool ChildEvents::hasChildEvent() const
|
||||
{
|
||||
return !childOutputs.empty() || childEOF || childTimeout;
|
||||
}
|
||||
|
||||
Goal::ChildEvent ChildEvents::popChildEvent()
|
||||
{
|
||||
if (!childOutputs.empty()) {
|
||||
auto event = std::move(childOutputs.front());
|
||||
childOutputs.pop();
|
||||
return event;
|
||||
}
|
||||
if (childEOF)
|
||||
return *std::exchange(childEOF, std::nullopt);
|
||||
if (childTimeout)
|
||||
return *std::exchange(childTimeout, std::nullopt);
|
||||
unreachable();
|
||||
}
|
||||
|
||||
using handle_type = nix::Goal::handle_type;
|
||||
using Suspend = nix::Goal::Suspend;
|
||||
|
||||
@@ -256,27 +206,6 @@ void Goal::work()
|
||||
assert(top_co || exitCode != ecBusy);
|
||||
}
|
||||
|
||||
void Goal::handleChildOutput(Descriptor fd, std::string_view data)
|
||||
{
|
||||
assert(top_co);
|
||||
top_co->handle.promise().childEvents.pushChildEvent(ChildOutput{fd, std::string{data}});
|
||||
worker.wakeUp(shared_from_this());
|
||||
}
|
||||
|
||||
void Goal::handleEOF(Descriptor fd)
|
||||
{
|
||||
assert(top_co);
|
||||
top_co->handle.promise().childEvents.pushChildEvent(ChildEOF{fd});
|
||||
worker.wakeUp(shared_from_this());
|
||||
}
|
||||
|
||||
void Goal::timedOut(TimedOut && ex)
|
||||
{
|
||||
assert(top_co);
|
||||
top_co->handle.promise().childEvents.pushChildEvent(std::move(ex));
|
||||
worker.wakeUp(shared_from_this());
|
||||
}
|
||||
|
||||
Goal::Co Goal::yield()
|
||||
{
|
||||
worker.wakeUp(shared_from_this());
|
||||
|
||||
@@ -258,16 +258,7 @@ Goal::Co PathSubstitutionGoal::tryToRun(
|
||||
true,
|
||||
false);
|
||||
|
||||
while (true) {
|
||||
auto event = co_await WaitForChildEvent{};
|
||||
if (std::get_if<ChildOutput>(&event)) {
|
||||
// Substitution doesn't process child output
|
||||
} else if (std::get_if<ChildEOF>(&event)) {
|
||||
break;
|
||||
} else if (std::get_if<TimedOut>(&event)) {
|
||||
unreachable(); // Substitution doesn't use timeouts
|
||||
}
|
||||
}
|
||||
co_await Suspend{};
|
||||
|
||||
trace("substitute finished");
|
||||
|
||||
@@ -319,6 +310,11 @@ Goal::Co PathSubstitutionGoal::tryToRun(
|
||||
co_return doneSuccess(BuildResult::Success::Substituted);
|
||||
}
|
||||
|
||||
void PathSubstitutionGoal::handleEOF(Descriptor fd)
|
||||
{
|
||||
worker.wakeUp(shared_from_this());
|
||||
}
|
||||
|
||||
void PathSubstitutionGoal::cleanup()
|
||||
{
|
||||
try {
|
||||
|
||||
@@ -479,13 +479,14 @@ void Worker::waitForInput()
|
||||
|
||||
if (goal->exitCode == Goal::ecBusy && 0 != settings.maxSilentTime && j->respectTimeouts
|
||||
&& after - j->lastOutput >= std::chrono::seconds(settings.maxSilentTime)) {
|
||||
goal->timedOut(TimedOut(settings.maxSilentTime));
|
||||
goal->timedOut(
|
||||
Error("%1% timed out after %2% seconds of silence", goal->getName(), settings.maxSilentTime));
|
||||
}
|
||||
|
||||
else if (
|
||||
goal->exitCode == Goal::ecBusy && 0 != settings.buildTimeout && j->respectTimeouts
|
||||
&& after - j->timeStarted >= std::chrono::seconds(settings.buildTimeout)) {
|
||||
goal->timedOut(TimedOut(settings.buildTimeout));
|
||||
goal->timedOut(Error("%1% timed out after %2% seconds", goal->getName(), settings.buildTimeout));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -341,8 +341,12 @@ DerivationOptions<SingleDerivedPath> derivationOptionsFromStructuredAttrs(
|
||||
|
||||
if (parsed) {
|
||||
auto * e = optionalValueAt(parsed->structuredAttrs, "exportReferencesGraph");
|
||||
if (!e || !e->is_object())
|
||||
if (!e)
|
||||
return ret;
|
||||
if (!e->is_object()) {
|
||||
warn("'exportReferencesGraph' in structured attrs is not a JSON object, ignoring");
|
||||
return ret;
|
||||
}
|
||||
for (auto & [key, storePathsJson] : getObject(*e)) {
|
||||
StringSet ss;
|
||||
flatten(storePathsJson, ss);
|
||||
|
||||
@@ -1489,6 +1489,8 @@ adl_serializer<DerivationOutput>::from_json(const json & _json, const Experiment
|
||||
}
|
||||
}
|
||||
|
||||
static unsigned constexpr expectedJsonVersionDerivation = 4;
|
||||
|
||||
void adl_serializer<Derivation>::to_json(json & res, const Derivation & d)
|
||||
{
|
||||
res = nlohmann::json::object();
|
||||
|
||||
@@ -41,27 +41,9 @@ FileTransferSettings fileTransferSettings;
|
||||
|
||||
static GlobalConfig::Register rFileTransferSettings(&fileTransferSettings);
|
||||
|
||||
namespace {
|
||||
|
||||
using curlSList = std::unique_ptr<::curl_slist, decltype([](::curl_slist * list) { ::curl_slist_free_all(list); })>;
|
||||
using curlMulti = std::unique_ptr<::CURLM, decltype([](::CURLM * multi) { ::curl_multi_cleanup(multi); })>;
|
||||
|
||||
struct curlMultiError : Error
|
||||
{
|
||||
::CURLMcode code;
|
||||
|
||||
curlMultiError(::CURLMcode code)
|
||||
: Error{"unexpected curl multi error: %s", ::curl_multi_strerror(code)}
|
||||
{
|
||||
assert(code != CURLM_OK);
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
struct curlFileTransfer : public FileTransfer
|
||||
{
|
||||
curlMulti curlm;
|
||||
CURLM * curlm = 0;
|
||||
|
||||
std::random_device rd;
|
||||
std::mt19937 mt19937;
|
||||
@@ -87,7 +69,7 @@ struct curlFileTransfer : public FileTransfer
|
||||
has been reached. */
|
||||
std::chrono::steady_clock::time_point embargo;
|
||||
|
||||
curlSList requestHeaders;
|
||||
struct curl_slist * requestHeaders = 0;
|
||||
|
||||
std::string encoding;
|
||||
|
||||
@@ -110,15 +92,6 @@ struct curlFileTransfer : public FileTransfer
|
||||
return httpStatus;
|
||||
}
|
||||
|
||||
void appendHeaders(const std::string & header)
|
||||
{
|
||||
curlSList tmpSList = curlSList(::curl_slist_append(requestHeaders.get(), requireCString(header)));
|
||||
if (!tmpSList)
|
||||
throw std::bad_alloc();
|
||||
requestHeaders.release();
|
||||
requestHeaders = std::move(tmpSList);
|
||||
}
|
||||
|
||||
TransferItem(
|
||||
curlFileTransfer & fileTransfer,
|
||||
const FileTransferRequest & request,
|
||||
@@ -158,13 +131,13 @@ struct curlFileTransfer : public FileTransfer
|
||||
{
|
||||
result.urls.push_back(request.uri.to_string());
|
||||
|
||||
appendHeaders("Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
|
||||
requestHeaders = curl_slist_append(requestHeaders, "Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
|
||||
if (!request.expectedETag.empty())
|
||||
appendHeaders("If-None-Match: " + request.expectedETag);
|
||||
requestHeaders = curl_slist_append(requestHeaders, ("If-None-Match: " + request.expectedETag).c_str());
|
||||
if (!request.mimeType.empty())
|
||||
appendHeaders("Content-Type: " + request.mimeType);
|
||||
requestHeaders = curl_slist_append(requestHeaders, ("Content-Type: " + request.mimeType).c_str());
|
||||
for (auto it = request.headers.begin(); it != request.headers.end(); ++it) {
|
||||
appendHeaders(fmt("%s: %s", it->first, it->second));
|
||||
requestHeaders = curl_slist_append(requestHeaders, fmt("%s: %s", it->first, it->second).c_str());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -172,9 +145,11 @@ struct curlFileTransfer : public FileTransfer
|
||||
{
|
||||
if (req) {
|
||||
if (active)
|
||||
curl_multi_remove_handle(fileTransfer.curlm.get(), req);
|
||||
curl_multi_remove_handle(fileTransfer.curlm, req);
|
||||
curl_easy_cleanup(req);
|
||||
}
|
||||
if (requestHeaders)
|
||||
curl_slist_free_all(requestHeaders);
|
||||
try {
|
||||
if (!done)
|
||||
fail(FileTransferError(
|
||||
@@ -373,7 +348,7 @@ struct curlFileTransfer : public FileTransfer
|
||||
return ((TransferItem *) userp)->readCallback(buffer, size, nitems);
|
||||
}
|
||||
|
||||
#if !defined(_WIN32)
|
||||
#if !defined(_WIN32) && LIBCURL_VERSION_NUM >= 0x071000
|
||||
static int cloexec_callback(void *, curl_socket_t curlfd, curlsocktype purpose)
|
||||
{
|
||||
unix::closeOnExec(curlfd);
|
||||
@@ -436,11 +411,15 @@ struct curlFileTransfer : public FileTransfer
|
||||
("curl/" LIBCURL_VERSION " Nix/" + nixVersion
|
||||
+ (fileTransferSettings.userAgentSuffix != "" ? " " + fileTransferSettings.userAgentSuffix.get() : ""))
|
||||
.c_str());
|
||||
#if LIBCURL_VERSION_NUM >= 0x072b00
|
||||
curl_easy_setopt(req, CURLOPT_PIPEWAIT, 1);
|
||||
#endif
|
||||
#if LIBCURL_VERSION_NUM >= 0x072f00
|
||||
if (fileTransferSettings.enableHttp2)
|
||||
curl_easy_setopt(req, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2TLS);
|
||||
else
|
||||
curl_easy_setopt(req, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
|
||||
#endif
|
||||
curl_easy_setopt(req, CURLOPT_WRITEFUNCTION, TransferItem::writeCallbackWrapper);
|
||||
curl_easy_setopt(req, CURLOPT_WRITEDATA, this);
|
||||
curl_easy_setopt(req, CURLOPT_HEADERFUNCTION, TransferItem::headerCallbackWrapper);
|
||||
@@ -450,7 +429,7 @@ struct curlFileTransfer : public FileTransfer
|
||||
curl_easy_setopt(req, CURLOPT_XFERINFODATA, this);
|
||||
curl_easy_setopt(req, CURLOPT_NOPROGRESS, 0);
|
||||
|
||||
curl_easy_setopt(req, CURLOPT_HTTPHEADER, requestHeaders.get());
|
||||
curl_easy_setopt(req, CURLOPT_HTTPHEADER, requestHeaders);
|
||||
|
||||
if (settings.downloadSpeed.get() > 0)
|
||||
curl_easy_setopt(req, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t) (settings.downloadSpeed.get() * 1024));
|
||||
@@ -480,9 +459,10 @@ struct curlFileTransfer : public FileTransfer
|
||||
if (settings.caFile != "")
|
||||
curl_easy_setopt(req, CURLOPT_CAINFO, settings.caFile.get().c_str());
|
||||
|
||||
#if !defined(_WIN32)
|
||||
#if !defined(_WIN32) && LIBCURL_VERSION_NUM >= 0x071000
|
||||
curl_easy_setopt(req, CURLOPT_SOCKOPTFUNCTION, cloexec_callback);
|
||||
#endif
|
||||
|
||||
curl_easy_setopt(req, CURLOPT_CONNECTTIMEOUT, fileTransferSettings.connectTimeout.get());
|
||||
|
||||
curl_easy_setopt(req, CURLOPT_LOW_SPEED_LIMIT, 1L);
|
||||
@@ -714,6 +694,13 @@ struct curlFileTransfer : public FileTransfer
|
||||
|
||||
Sync<State> state_;
|
||||
|
||||
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
|
||||
/* We can't use a std::condition_variable to wake up the curl
|
||||
thread, because it only monitors file descriptors. So use a
|
||||
pipe instead. */
|
||||
Pipe wakeupPipe;
|
||||
#endif
|
||||
|
||||
std::thread workerThread;
|
||||
|
||||
curlFileTransfer()
|
||||
@@ -722,35 +709,43 @@ struct curlFileTransfer : public FileTransfer
|
||||
static std::once_flag globalInit;
|
||||
std::call_once(globalInit, curl_global_init, CURL_GLOBAL_ALL);
|
||||
|
||||
curlm = curlMulti(curl_multi_init());
|
||||
curlm = curl_multi_init();
|
||||
|
||||
curl_multi_setopt(curlm.get(), CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
|
||||
curl_multi_setopt(curlm.get(), CURLMOPT_MAX_TOTAL_CONNECTIONS, fileTransferSettings.httpConnections.get());
|
||||
#if LIBCURL_VERSION_NUM >= 0x072b00 // Multiplex requires >= 7.43.0
|
||||
curl_multi_setopt(curlm, CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
|
||||
#endif
|
||||
#if LIBCURL_VERSION_NUM >= 0x071e00 // Max connections requires >= 7.30.0
|
||||
curl_multi_setopt(curlm, CURLMOPT_MAX_TOTAL_CONNECTIONS, fileTransferSettings.httpConnections.get());
|
||||
#endif
|
||||
|
||||
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
|
||||
wakeupPipe.create();
|
||||
fcntl(wakeupPipe.readSide.get(), F_SETFL, O_NONBLOCK);
|
||||
#endif
|
||||
|
||||
workerThread = std::thread([&]() { workerThreadEntry(); });
|
||||
}
|
||||
|
||||
~curlFileTransfer()
|
||||
{
|
||||
try {
|
||||
stopWorkerThread();
|
||||
} catch (...) {
|
||||
ignoreExceptionInDestructor();
|
||||
}
|
||||
stopWorkerThread();
|
||||
|
||||
workerThread.join();
|
||||
|
||||
if (curlm)
|
||||
curl_multi_cleanup(curlm);
|
||||
}
|
||||
|
||||
void stopWorkerThread()
|
||||
{
|
||||
/* Signal the worker thread to exit. */
|
||||
state_.lock()->quit();
|
||||
wakeupMulti();
|
||||
}
|
||||
|
||||
void wakeupMulti()
|
||||
{
|
||||
if (auto ec = ::curl_multi_wakeup(curlm.get()))
|
||||
throw curlMultiError(ec);
|
||||
{
|
||||
auto state(state_.lock());
|
||||
state->quit();
|
||||
}
|
||||
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
|
||||
writeFull(wakeupPipe.writeSide.get(), " ", false);
|
||||
#endif
|
||||
}
|
||||
|
||||
void workerThreadMain()
|
||||
@@ -780,25 +775,32 @@ struct curlFileTransfer : public FileTransfer
|
||||
|
||||
/* Let curl do its thing. */
|
||||
int running;
|
||||
CURLMcode mc = curl_multi_perform(curlm.get(), &running);
|
||||
CURLMcode mc = curl_multi_perform(curlm, &running);
|
||||
if (mc != CURLM_OK)
|
||||
throw nix::Error("unexpected error from curl_multi_perform(): %s", curl_multi_strerror(mc));
|
||||
|
||||
/* Set the promises of any finished requests. */
|
||||
CURLMsg * msg;
|
||||
int left;
|
||||
while ((msg = curl_multi_info_read(curlm.get(), &left))) {
|
||||
while ((msg = curl_multi_info_read(curlm, &left))) {
|
||||
if (msg->msg == CURLMSG_DONE) {
|
||||
auto i = items.find(msg->easy_handle);
|
||||
assert(i != items.end());
|
||||
i->second->finish(msg->data.result);
|
||||
curl_multi_remove_handle(curlm.get(), i->second->req);
|
||||
curl_multi_remove_handle(curlm, i->second->req);
|
||||
i->second->active = false;
|
||||
items.erase(i);
|
||||
}
|
||||
}
|
||||
|
||||
/* Wait for activity, including wakeup events. */
|
||||
int numfds = 0;
|
||||
struct curl_waitfd extraFDs[1];
|
||||
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
|
||||
extraFDs[0].fd = wakeupPipe.readSide.get();
|
||||
extraFDs[0].events = CURL_WAIT_POLLIN;
|
||||
extraFDs[0].revents = 0;
|
||||
#endif
|
||||
long maxSleepTimeMs = items.empty() ? 10000 : 100;
|
||||
auto sleepTimeMs = nextWakeup != std::chrono::steady_clock::time_point()
|
||||
? std::max(
|
||||
@@ -807,14 +809,23 @@ struct curlFileTransfer : public FileTransfer
|
||||
nextWakeup - std::chrono::steady_clock::now())
|
||||
.count())
|
||||
: maxSleepTimeMs;
|
||||
|
||||
int numfds = 0;
|
||||
mc = curl_multi_poll(curlm.get(), nullptr, 0, sleepTimeMs, &numfds);
|
||||
vomit("download thread waiting for %d ms", sleepTimeMs);
|
||||
mc = curl_multi_wait(curlm, extraFDs, 1, sleepTimeMs, &numfds);
|
||||
if (mc != CURLM_OK)
|
||||
throw curlMultiError(mc);
|
||||
throw nix::Error("unexpected error from curl_multi_wait(): %s", curl_multi_strerror(mc));
|
||||
|
||||
nextWakeup = std::chrono::steady_clock::time_point();
|
||||
|
||||
/* Add new curl requests from the incoming requests queue,
|
||||
except for requests that are embargoed (waiting for a
|
||||
retry timeout to expire). */
|
||||
if (extraFDs[0].revents & CURL_WAIT_POLLIN) {
|
||||
char buf[1024];
|
||||
auto res = read(extraFDs[0].fd, buf, sizeof(buf));
|
||||
if (res == -1 && errno != EINTR)
|
||||
throw SysError("reading curl wakeup socket");
|
||||
}
|
||||
|
||||
std::vector<std::shared_ptr<TransferItem>> incoming;
|
||||
auto now = std::chrono::steady_clock::now();
|
||||
|
||||
@@ -837,7 +848,7 @@ struct curlFileTransfer : public FileTransfer
|
||||
for (auto & item : incoming) {
|
||||
debug("starting %s of %s", item->request.noun(), item->request.uri);
|
||||
item->init();
|
||||
curl_multi_add_handle(curlm.get(), item->req);
|
||||
curl_multi_add_handle(curlm, item->req);
|
||||
item->active = true;
|
||||
items[item->req] = item;
|
||||
}
|
||||
@@ -888,8 +899,10 @@ struct curlFileTransfer : public FileTransfer
|
||||
throw nix::Error("cannot enqueue download request because the download thread is shutting down");
|
||||
state->incoming.push(item);
|
||||
}
|
||||
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
|
||||
writeFull(wakeupPipe.writeSide.get(), " ");
|
||||
#endif
|
||||
|
||||
wakeupMulti();
|
||||
return ItemHandle(static_cast<Item &>(*item));
|
||||
}
|
||||
|
||||
@@ -909,7 +922,9 @@ struct curlFileTransfer : public FileTransfer
|
||||
{
|
||||
auto state(state_.lock());
|
||||
state->unpause.push_back(std::move(item));
|
||||
wakeupMulti();
|
||||
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
|
||||
writeFull(wakeupPipe.writeSide.get(), " ");
|
||||
#endif
|
||||
}
|
||||
|
||||
void unpauseTransfer(ItemHandle handle) override
|
||||
|
||||
@@ -79,7 +79,7 @@ struct DerivationBuilderParams
|
||||
*/
|
||||
const StorePathSet & inputPaths;
|
||||
|
||||
const std::map<std::string, InitialOutput> initialOutputs;
|
||||
const std::map<std::string, InitialOutput> & initialOutputs;
|
||||
|
||||
const BuildMode & buildMode;
|
||||
|
||||
|
||||
@@ -100,6 +100,8 @@ private:
|
||||
|
||||
std::map<ActivityId, Activity> builderActivities;
|
||||
|
||||
void timedOut(Error && ex) override;
|
||||
|
||||
std::string key() override;
|
||||
|
||||
/**
|
||||
@@ -127,10 +129,10 @@ private:
|
||||
bool isReadDesc(Descriptor fd);
|
||||
|
||||
/**
|
||||
* Process output from a child process.
|
||||
* Callback used by the worker to write to the log.
|
||||
*/
|
||||
Co processChildOutput(Descriptor fd, std::string_view data);
|
||||
|
||||
void handleChildOutput(Descriptor fd, std::string_view data) override;
|
||||
void handleEOF(Descriptor fd) override;
|
||||
void flushLine();
|
||||
|
||||
/**
|
||||
|
||||
@@ -52,6 +52,11 @@ struct DerivationGoal : public Goal
|
||||
bool storeDerivation);
|
||||
~DerivationGoal() = default;
|
||||
|
||||
void timedOut(Error && ex) override
|
||||
{
|
||||
unreachable();
|
||||
};
|
||||
|
||||
std::string key() override;
|
||||
|
||||
JobCategory jobCategory() const override
|
||||
|
||||
@@ -43,6 +43,8 @@ struct DerivationResolutionGoal : public Goal
|
||||
*/
|
||||
std::unique_ptr<std::pair<StorePath, BasicDerivation>> resolvedDrv;
|
||||
|
||||
void timedOut(Error && ex) override {}
|
||||
|
||||
private:
|
||||
|
||||
/**
|
||||
|
||||
@@ -109,6 +109,8 @@ struct DerivationTrampolineGoal : public Goal
|
||||
|
||||
virtual ~DerivationTrampolineGoal();
|
||||
|
||||
void timedOut(Error && ex) override {}
|
||||
|
||||
std::string key() override;
|
||||
|
||||
JobCategory jobCategory() const override
|
||||
|
||||
@@ -33,8 +33,15 @@ public:
|
||||
|
||||
Co init();
|
||||
|
||||
void timedOut(Error && ex) override
|
||||
{
|
||||
unreachable();
|
||||
};
|
||||
|
||||
std::string key() override;
|
||||
|
||||
void handleEOF(Descriptor fd) override;
|
||||
|
||||
JobCategory jobCategory() const override
|
||||
{
|
||||
return JobCategory::Substitution;
|
||||
|
||||
@@ -5,18 +5,9 @@
|
||||
#include "nix/store/build-result.hh"
|
||||
|
||||
#include <coroutine>
|
||||
#include <queue>
|
||||
#include <variant>
|
||||
|
||||
namespace nix {
|
||||
|
||||
struct TimedOut : BuildError
|
||||
{
|
||||
time_t maxDuration;
|
||||
|
||||
TimedOut(time_t maxDuration);
|
||||
};
|
||||
|
||||
/**
|
||||
* Forward definition.
|
||||
*/
|
||||
@@ -147,29 +138,6 @@ public:
|
||||
friend Goal;
|
||||
};
|
||||
|
||||
/**
|
||||
* Event types for child process communication, delivered via coroutines.
|
||||
*/
|
||||
struct ChildOutput
|
||||
{
|
||||
Descriptor fd;
|
||||
std::string data;
|
||||
};
|
||||
|
||||
struct ChildEOF
|
||||
{
|
||||
Descriptor fd;
|
||||
};
|
||||
|
||||
using ChildEvent = std::variant<ChildOutput, ChildEOF, TimedOut>;
|
||||
|
||||
/**
|
||||
* Tag type for `co_await`-ing child events.
|
||||
* Returns a `ChildEvent` when resumed.
|
||||
*/
|
||||
struct WaitForChildEvent
|
||||
{};
|
||||
|
||||
// forward declaration of promise_type, see below
|
||||
struct promise_type;
|
||||
|
||||
@@ -308,28 +276,6 @@ public:
|
||||
*/
|
||||
bool alive = true;
|
||||
|
||||
class
|
||||
{
|
||||
/**
|
||||
* Structured queue of child events:
|
||||
* - outputs: stream of data from child
|
||||
* - eof: optional end-of-stream marker
|
||||
* - timeout: optional timeout that flushes/overrides other events
|
||||
*/
|
||||
std::queue<ChildOutput> childOutputs;
|
||||
std::optional<ChildEOF> childEOF;
|
||||
std::optional<TimedOut> childTimeout;
|
||||
|
||||
public:
|
||||
|
||||
void pushChildEvent(ChildOutput event);
|
||||
void pushChildEvent(ChildEOF event);
|
||||
void pushChildEvent(TimedOut event);
|
||||
bool hasChildEvent() const;
|
||||
ChildEvent popChildEvent();
|
||||
|
||||
} childEvents;
|
||||
|
||||
/**
|
||||
* The awaiter used by @ref final_suspend.
|
||||
*/
|
||||
@@ -423,66 +369,13 @@ public:
|
||||
return static_cast<Co &&>(co);
|
||||
}
|
||||
|
||||
/**
|
||||
* Awaiter for @ref Suspend. Always suspends, but asserts
|
||||
* there are no pending child events (those should be
|
||||
* consumed first via @ref WaitForChildEvent).
|
||||
*/
|
||||
struct SuspendAwaiter
|
||||
{
|
||||
promise_type & promise;
|
||||
|
||||
bool await_ready()
|
||||
{
|
||||
assert(!promise.childEvents.hasChildEvent());
|
||||
return false;
|
||||
}
|
||||
|
||||
void await_suspend(handle_type) {}
|
||||
|
||||
void await_resume() {}
|
||||
};
|
||||
|
||||
/**
|
||||
* Allows awaiting a @ref Suspend.
|
||||
* Always suspends.
|
||||
*/
|
||||
SuspendAwaiter await_transform(Suspend)
|
||||
std::suspend_always await_transform(Suspend)
|
||||
{
|
||||
return SuspendAwaiter{*this};
|
||||
};
|
||||
|
||||
/**
|
||||
* Awaiter for child events. Suspends and returns the
|
||||
* pending child event when resumed.
|
||||
*/
|
||||
struct ChildEventAwaiter
|
||||
{
|
||||
handle_type handle;
|
||||
|
||||
bool await_ready()
|
||||
{
|
||||
return handle && handle.promise().childEvents.hasChildEvent();
|
||||
}
|
||||
|
||||
void await_suspend(handle_type h)
|
||||
{
|
||||
handle = h;
|
||||
}
|
||||
|
||||
ChildEvent await_resume()
|
||||
{
|
||||
assert(handle);
|
||||
return handle.promise().childEvents.popChildEvent();
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Allows awaiting child events (output, EOF, timeout).
|
||||
*/
|
||||
ChildEventAwaiter await_transform(WaitForChildEvent)
|
||||
{
|
||||
return ChildEventAwaiter{handle_type::from_promise(*this)};
|
||||
return {};
|
||||
};
|
||||
};
|
||||
|
||||
@@ -539,23 +432,15 @@ public:
|
||||
|
||||
void work();
|
||||
|
||||
/**
|
||||
* Called by the worker when data is received from a child process.
|
||||
* Stores the event and resumes the coroutine.
|
||||
*/
|
||||
void handleChildOutput(Descriptor fd, std::string_view data);
|
||||
virtual void handleChildOutput(Descriptor fd, std::string_view data)
|
||||
{
|
||||
unreachable();
|
||||
}
|
||||
|
||||
/**
|
||||
* Called by the worker when EOF is received from a child process.
|
||||
* Stores the event and resumes the coroutine.
|
||||
*/
|
||||
void handleEOF(Descriptor fd);
|
||||
|
||||
/**
|
||||
* Called by the worker when a build times out.
|
||||
* Stores the event and resumes the coroutine.
|
||||
*/
|
||||
void timedOut(TimedOut && ex);
|
||||
virtual void handleEOF(Descriptor fd)
|
||||
{
|
||||
unreachable();
|
||||
}
|
||||
|
||||
void trace(std::string_view s);
|
||||
|
||||
@@ -564,6 +449,13 @@ public:
|
||||
return name;
|
||||
}
|
||||
|
||||
/**
|
||||
* Callback in case of a timeout. It should wake up its waiters,
|
||||
* get rid of any running child processes that are being monitored
|
||||
* by the worker (important!), etc.
|
||||
*/
|
||||
virtual void timedOut(Error && ex) = 0;
|
||||
|
||||
/**
|
||||
* Used for comparisons. The order matters a bit for scheduling. We
|
||||
* want:
|
||||
|
||||
@@ -53,6 +53,11 @@ public:
|
||||
std::optional<ContentAddress> ca = std::nullopt);
|
||||
~PathSubstitutionGoal();
|
||||
|
||||
void timedOut(Error && ex) override
|
||||
{
|
||||
unreachable();
|
||||
};
|
||||
|
||||
std::string key() override
|
||||
{
|
||||
return "a$" + std::string(storePath.name()) + "$" + worker.store.printStorePath(storePath);
|
||||
@@ -67,6 +72,12 @@ public:
|
||||
StorePath subPath, nix::ref<Store> sub, std::shared_ptr<const ValidPathInfo> info, bool & substituterFailed);
|
||||
Co finished();
|
||||
|
||||
/**
|
||||
* Callback used by the worker to write to the log.
|
||||
*/
|
||||
void handleChildOutput(Descriptor fd, std::string_view data) override {};
|
||||
void handleEOF(Descriptor fd) override;
|
||||
|
||||
/* Called by destructor, can't be overridden */
|
||||
void cleanup() override final;
|
||||
|
||||
|
||||
@@ -586,12 +586,6 @@ void writeDerivation(Sink & out, const StoreDirConfig & store, const BasicDeriva
|
||||
*/
|
||||
std::string hashPlaceholder(const OutputNameView outputName);
|
||||
|
||||
/**
|
||||
* The expected JSON version for derivation serialization.
|
||||
* Used by `nix derivation show` and `nix derivation add`.
|
||||
*/
|
||||
constexpr unsigned expectedJsonVersionDerivation = 4;
|
||||
|
||||
} // namespace nix
|
||||
|
||||
JSON_IMPL_WITH_XP_FEATURES(nix::DerivationOutput)
|
||||
|
||||
@@ -1143,7 +1143,7 @@ public:
|
||||
|
||||
Setting<std::string> netrcFile{
|
||||
this,
|
||||
(nixConfDir / "netrc").string(),
|
||||
fmt("%s/%s", nixConfDir, "netrc"),
|
||||
"netrc-file",
|
||||
R"(
|
||||
If set to an absolute path to a `netrc` file, Nix uses the HTTP
|
||||
|
||||
@@ -39,32 +39,28 @@ deps_public_maybe_subproject = [
|
||||
]
|
||||
subdir('nix-meson-build-support/subprojects')
|
||||
|
||||
can_link_symlink = false
|
||||
native_ln = find_program('ln', required : false, native : true)
|
||||
if native_ln.found()
|
||||
run_command(
|
||||
native_ln,
|
||||
'-s',
|
||||
meson.project_build_root() / '__nothing_link_target',
|
||||
meson.project_build_root() / '__nothing_symlink',
|
||||
# native doesn't allow dangling symlinks, which the tests require
|
||||
env : {'MSYS' : 'winsymlinks:lnk'},
|
||||
check : true,
|
||||
)
|
||||
can_link_symlink = run_command(
|
||||
native_ln,
|
||||
meson.project_build_root() / '__nothing_symlink',
|
||||
meson.project_build_root() / '__nothing_hardlink',
|
||||
check : false,
|
||||
).returncode() == 0
|
||||
run_command(
|
||||
'rm',
|
||||
'-f',
|
||||
meson.project_build_root() / '__nothing_symlink',
|
||||
meson.project_build_root() / '__nothing_hardlink',
|
||||
check : true,
|
||||
)
|
||||
endif
|
||||
run_command(
|
||||
'ln',
|
||||
'-s',
|
||||
meson.project_build_root() / '__nothing_link_target',
|
||||
meson.project_build_root() / '__nothing_symlink',
|
||||
# native doesn't allow dangling symlinks, which the tests require
|
||||
env : {'MSYS' : 'winsymlinks:lnk'},
|
||||
check : true,
|
||||
)
|
||||
can_link_symlink = run_command(
|
||||
'ln',
|
||||
meson.project_build_root() / '__nothing_symlink',
|
||||
meson.project_build_root() / '__nothing_hardlink',
|
||||
check : false,
|
||||
).returncode() == 0
|
||||
run_command(
|
||||
'rm',
|
||||
'-f',
|
||||
meson.project_build_root() / '__nothing_symlink',
|
||||
meson.project_build_root() / '__nothing_hardlink',
|
||||
check : true,
|
||||
)
|
||||
summary('can hardlink to symlink', can_link_symlink, bool_yn : true)
|
||||
configdata_priv.set('CAN_LINK_SYMLINK', can_link_symlink.to_int())
|
||||
|
||||
@@ -164,8 +160,6 @@ if s3_aws_auth.enabled()
|
||||
deps_other += aws_crt_cpp
|
||||
aws_c_common = cxx.find_library('aws-c-common', required : true)
|
||||
deps_other += aws_c_common
|
||||
aws_c_auth = cxx.find_library('aws-c-auth', required : true)
|
||||
deps_other += aws_c_auth
|
||||
endif
|
||||
|
||||
configdata_pub.set('NIX_WITH_AWS_AUTH', s3_aws_auth.enabled().to_int())
|
||||
|
||||
@@ -4,11 +4,20 @@
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include "nix/util/types.hh"
|
||||
#include "nix/util/environment-variables.hh"
|
||||
#include "nix/util/file-system.hh"
|
||||
#include "nix/util/tests/test-data.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
/**
|
||||
* The path to the unit test data directory. See the contributing guide
|
||||
* in the manual for further details.
|
||||
*/
|
||||
static inline std::filesystem::path getUnitTestData()
|
||||
{
|
||||
return getEnv("_NIX_TEST_UNIT_DATA").value();
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether we should update "golden masters" instead of running tests
|
||||
* against them. See the contributing guide in the manual for further
|
||||
|
||||
@@ -10,5 +10,4 @@ headers = files(
|
||||
'json-characterization.hh',
|
||||
'nix_api_util.hh',
|
||||
'string_callback.hh',
|
||||
'test-data.hh',
|
||||
)
|
||||
|
||||
@@ -1,24 +0,0 @@
|
||||
#pragma once
|
||||
///@file
|
||||
|
||||
#include <filesystem>
|
||||
#include "nix/util/environment-variables.hh"
|
||||
#include "nix/util/error.hh"
|
||||
|
||||
namespace nix {
|
||||
|
||||
/**
|
||||
* The path to the unit test data directory. See the contributing guide
|
||||
* in the manual for further details.
|
||||
*/
|
||||
static inline std::filesystem::path getUnitTestData()
|
||||
{
|
||||
auto data = getEnv("_NIX_TEST_UNIT_DATA");
|
||||
if (!data)
|
||||
throw Error(
|
||||
"_NIX_TEST_UNIT_DATA environment variable is not set. "
|
||||
"Recommendation: use meson, example: 'meson test -C build --gdb'");
|
||||
return std::filesystem::path(*data);
|
||||
}
|
||||
|
||||
} // namespace nix
|
||||
@@ -33,9 +33,6 @@ deps_private += rapidcheck
|
||||
gtest = dependency('gtest', main : true)
|
||||
deps_private += gtest
|
||||
|
||||
gmock = dependency('gmock')
|
||||
deps_private += gmock
|
||||
|
||||
configdata = configuration_data()
|
||||
configdata.set_quoted('PACKAGE_VERSION', meson.project_version())
|
||||
|
||||
@@ -75,7 +72,6 @@ sources = files(
|
||||
'position.cc',
|
||||
'processes.cc',
|
||||
'sort.cc',
|
||||
'source-accessor.cc',
|
||||
'spawn.cc',
|
||||
'strings.cc',
|
||||
'suggestions.cc',
|
||||
|
||||
@@ -1,138 +0,0 @@
|
||||
#include "nix/util/fs-sink.hh"
|
||||
#include "nix/util/file-system.hh"
|
||||
#include "nix/util/processes.hh"
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
#include <gmock/gmock.h>
|
||||
#include <rapidcheck/gtest.h>
|
||||
|
||||
namespace nix {
|
||||
|
||||
MATCHER_P2(HasContents, path, expected, "")
|
||||
{
|
||||
auto stat = arg->maybeLstat(path);
|
||||
if (!stat) {
|
||||
*result_listener << arg->showPath(path) << " does not exist";
|
||||
return false;
|
||||
}
|
||||
if (stat->type != SourceAccessor::tRegular) {
|
||||
*result_listener << arg->showPath(path) << " is not a regular file";
|
||||
return false;
|
||||
}
|
||||
auto actual = arg->readFile(path);
|
||||
if (actual != expected) {
|
||||
*result_listener << arg->showPath(path) << " has contents " << ::testing::PrintToString(actual);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
MATCHER_P2(HasSymlink, path, target, "")
|
||||
{
|
||||
auto stat = arg->maybeLstat(path);
|
||||
if (!stat) {
|
||||
*result_listener << arg->showPath(path) << " does not exist";
|
||||
return false;
|
||||
}
|
||||
if (stat->type != SourceAccessor::tSymlink) {
|
||||
*result_listener << arg->showPath(path) << " is not a symlink";
|
||||
return false;
|
||||
}
|
||||
auto actual = arg->readLink(path);
|
||||
if (actual != target) {
|
||||
*result_listener << arg->showPath(path) << " points to " << ::testing::PrintToString(actual);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
MATCHER_P2(HasDirectory, path, dirents, "")
|
||||
{
|
||||
auto stat = arg->maybeLstat(path);
|
||||
if (!stat) {
|
||||
*result_listener << arg->showPath(path) << " does not exist";
|
||||
return false;
|
||||
}
|
||||
if (stat->type != SourceAccessor::tDirectory) {
|
||||
*result_listener << arg->showPath(path) << " is not a directory";
|
||||
return false;
|
||||
}
|
||||
auto actual = arg->readDirectory(path);
|
||||
std::set<std::string> actualKeys, expectedKeys(dirents.begin(), dirents.end());
|
||||
for (auto & [k, _] : actual)
|
||||
actualKeys.insert(k);
|
||||
if (actualKeys != expectedKeys) {
|
||||
*result_listener << arg->showPath(path) << " has entries " << ::testing::PrintToString(actualKeys);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
class FSSourceAccessorTest : public ::testing::Test
|
||||
{
|
||||
protected:
|
||||
std::filesystem::path tmpDir;
|
||||
std::unique_ptr<nix::AutoDelete> delTmpDir;
|
||||
|
||||
void SetUp() override
|
||||
{
|
||||
tmpDir = nix::createTempDir();
|
||||
delTmpDir = std::make_unique<nix::AutoDelete>(tmpDir, true);
|
||||
}
|
||||
|
||||
void TearDown() override
|
||||
{
|
||||
delTmpDir.release();
|
||||
}
|
||||
};
|
||||
|
||||
TEST_F(FSSourceAccessorTest, works)
|
||||
{
|
||||
{
|
||||
RestoreSink sink(false);
|
||||
sink.dstPath = tmpDir;
|
||||
sink.dirFd = openDirectory(tmpDir);
|
||||
sink.createDirectory(CanonPath("subdir"));
|
||||
sink.createRegularFile(CanonPath("file1"), [](CreateRegularFileSink & crf) { crf("content1"); });
|
||||
sink.createRegularFile(CanonPath("subdir/file2"), [](CreateRegularFileSink & crf) { crf("content2"); });
|
||||
sink.createSymlink(CanonPath("rootlink"), "target");
|
||||
sink.createDirectory(CanonPath("a"));
|
||||
sink.createSymlink(CanonPath("a/dirlink"), "../subdir");
|
||||
}
|
||||
|
||||
EXPECT_THAT(makeFSSourceAccessor(tmpDir / "file1"), HasContents(CanonPath::root, "content1"));
|
||||
EXPECT_THAT(makeFSSourceAccessor(tmpDir / "rootlink"), HasSymlink(CanonPath::root, "target"));
|
||||
EXPECT_THAT(
|
||||
makeFSSourceAccessor(tmpDir),
|
||||
HasDirectory(CanonPath::root, std::set<std::string>{"file1", "subdir", "rootlink", "a"}));
|
||||
EXPECT_THAT(makeFSSourceAccessor(tmpDir / "subdir"), HasDirectory(CanonPath::root, std::set<std::string>{"file2"}));
|
||||
|
||||
{
|
||||
auto accessor = makeFSSourceAccessor(tmpDir);
|
||||
EXPECT_THAT(accessor, HasContents(CanonPath("file1"), "content1"));
|
||||
EXPECT_THAT(accessor, HasContents(CanonPath("subdir/file2"), "content2"));
|
||||
|
||||
EXPECT_TRUE(accessor->pathExists(CanonPath("file1")));
|
||||
EXPECT_FALSE(accessor->pathExists(CanonPath("nonexistent")));
|
||||
|
||||
EXPECT_THROW(accessor->readFile(CanonPath("a/dirlink/file2")), SymlinkNotAllowed);
|
||||
EXPECT_THROW(accessor->maybeLstat(CanonPath("a/dirlink/file2")), SymlinkNotAllowed);
|
||||
EXPECT_THROW(accessor->readDirectory(CanonPath("a/dirlink")), SymlinkNotAllowed);
|
||||
EXPECT_THROW(accessor->pathExists(CanonPath("a/dirlink/file2")), SymlinkNotAllowed);
|
||||
}
|
||||
|
||||
{
|
||||
auto accessor = makeFSSourceAccessor(tmpDir / "nonexistent");
|
||||
EXPECT_FALSE(accessor->maybeLstat(CanonPath::root));
|
||||
EXPECT_THROW(accessor->readFile(CanonPath::root), SystemError);
|
||||
}
|
||||
|
||||
{
|
||||
auto accessor = makeFSSourceAccessor(tmpDir, true);
|
||||
EXPECT_EQ(accessor->getLastModified(), 0);
|
||||
accessor->maybeLstat(CanonPath("file1"));
|
||||
EXPECT_GT(accessor->getLastModified(), 0);
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace nix
|
||||
@@ -103,8 +103,7 @@ void SourceAccessor::dumpPath(const CanonPath & path, Sink & sink, PathFilter &
|
||||
|
||||
time_t dumpPathAndGetMtime(const Path & path, Sink & sink, PathFilter & filter)
|
||||
{
|
||||
SourcePath path2 = {
|
||||
makeFSSourceAccessor(std::filesystem::path{}, /*trackLastModified=*/true), CanonPath(absPath(path))};
|
||||
auto path2 = PosixSourceAccessor::createAtRoot(path, /*trackLastModified=*/true);
|
||||
path2.dumpPath(sink, filter);
|
||||
return path2.accessor->getLastModified().value();
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
///@file
|
||||
|
||||
#include "nix/util/canon-path.hh"
|
||||
#include "nix/util/types.hh"
|
||||
#include "nix/util/error.hh"
|
||||
|
||||
#ifdef _WIN32
|
||||
@@ -235,6 +236,18 @@ std::wstring handleToFileName(Descriptor handle);
|
||||
#ifndef _WIN32
|
||||
namespace unix {
|
||||
|
||||
struct SymlinkNotAllowed : public Error
|
||||
{
|
||||
CanonPath path;
|
||||
|
||||
SymlinkNotAllowed(CanonPath path)
|
||||
/* Can't provide better error message, since the parent directory is only known to the caller. */
|
||||
: Error("relative path '%s' points to a symlink, which is not allowed", path.rel())
|
||||
, path(std::move(path))
|
||||
{
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Safe(r) function to open \param path file relative to \param dirFd, while
|
||||
* disallowing escaping from a directory and resolving any symlinks in the
|
||||
|
||||
@@ -43,6 +43,36 @@ public:
|
||||
|
||||
std::optional<std::filesystem::path> getPhysicalPath(const CanonPath & path) override;
|
||||
|
||||
/**
|
||||
* Create a `PosixSourceAccessor` and `SourcePath` corresponding to
|
||||
* some native path.
|
||||
*
|
||||
* @param Whether the accessor should return a non-null getLastModified.
|
||||
* When true the accessor must be used only by a single thread.
|
||||
*
|
||||
* The `PosixSourceAccessor` is rooted as far up the tree as
|
||||
* possible, (e.g. on Windows it could scoped to a drive like
|
||||
* `C:\`). This allows more `..` parent accessing to work.
|
||||
*
|
||||
* @note When `path` is trusted user input, canonicalize it using
|
||||
* `std::filesystem::canonical`, `makeParentCanonical`, `std::filesystem::weakly_canonical`, etc,
|
||||
* as appropriate for the use case. At least weak canonicalization is
|
||||
* required for the `SourcePath` to do anything useful at the location it
|
||||
* points to.
|
||||
*
|
||||
* @note A canonicalizing behavior is not built in `createAtRoot` so that
|
||||
* callers do not accidentally introduce symlink-related security vulnerabilities.
|
||||
* Furthermore, `createAtRoot` does not know whether the file pointed to by
|
||||
* `path` should be resolved if it is itself a symlink. In other words,
|
||||
* `createAtRoot` can not decide between aforementioned `canonical`, `makeParentCanonical`, etc. for its callers.
|
||||
*
|
||||
* See
|
||||
* [`std::filesystem::path::root_path`](https://en.cppreference.com/w/cpp/filesystem/path/root_path)
|
||||
* and
|
||||
* [`std::filesystem::path::relative_path`](https://en.cppreference.com/w/cpp/filesystem/path/relative_path).
|
||||
*/
|
||||
static SourcePath createAtRoot(const std::filesystem::path & path, bool trackLastModified = false);
|
||||
|
||||
std::optional<std::time_t> getLastModified() override
|
||||
{
|
||||
return trackLastModified ? std::optional{mtime} : std::nullopt;
|
||||
|
||||
@@ -222,24 +222,6 @@ ref<SourceAccessor> makeEmptySourceAccessor();
|
||||
*/
|
||||
MakeError(RestrictedPathError, Error);
|
||||
|
||||
struct SymlinkNotAllowed : public Error
|
||||
{
|
||||
CanonPath path;
|
||||
|
||||
SymlinkNotAllowed(CanonPath path)
|
||||
: Error("relative path '%s' points to a symlink, which is not allowed", path.rel())
|
||||
, path(std::move(path))
|
||||
{
|
||||
}
|
||||
|
||||
template<typename... Args>
|
||||
SymlinkNotAllowed(CanonPath path, const std::string & fs, Args &&... args)
|
||||
: Error(fs, std::forward<Args>(args)...)
|
||||
, path(std::move(path))
|
||||
{
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Return an accessor for the root filesystem.
|
||||
*/
|
||||
@@ -251,7 +233,7 @@ ref<SourceAccessor> getFSSourceAccessor();
|
||||
* elements, and that absolute symlinks are resolved relative to
|
||||
* `root`.
|
||||
*/
|
||||
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root, bool trackLastModified = false);
|
||||
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root);
|
||||
|
||||
/**
|
||||
* Construct an accessor that presents a "union" view of a vector of
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
#include "nix/util/posix-source-accessor.hh"
|
||||
#include "nix/util/source-path.hh"
|
||||
#include "nix/util/signals.hh"
|
||||
#include "nix/util/sync.hh"
|
||||
|
||||
#include <boost/unordered/concurrent_flat_map.hpp>
|
||||
|
||||
@@ -18,6 +20,15 @@ PosixSourceAccessor::PosixSourceAccessor()
|
||||
{
|
||||
}
|
||||
|
||||
SourcePath PosixSourceAccessor::createAtRoot(const std::filesystem::path & path, bool trackLastModified)
|
||||
{
|
||||
std::filesystem::path path2 = absPath(path);
|
||||
return {
|
||||
make_ref<PosixSourceAccessor>(path2.root_path(), trackLastModified),
|
||||
CanonPath{path2.relative_path().string()},
|
||||
};
|
||||
}
|
||||
|
||||
std::filesystem::path PosixSourceAccessor::makeAbsPath(const CanonPath & path)
|
||||
{
|
||||
return root.empty() ? (std::filesystem::path{path.abs()})
|
||||
@@ -197,7 +208,7 @@ void PosixSourceAccessor::assertNoSymlinks(CanonPath path)
|
||||
while (!path.isRoot()) {
|
||||
auto st = cachedLstat(path);
|
||||
if (st && S_ISLNK(st->st_mode))
|
||||
throw SymlinkNotAllowed(path, "path '%s' is a symlink", showPath(path));
|
||||
throw Error("path '%s' is a symlink", showPath(path));
|
||||
path.pop();
|
||||
}
|
||||
}
|
||||
@@ -208,8 +219,8 @@ ref<SourceAccessor> getFSSourceAccessor()
|
||||
return rootFS;
|
||||
}
|
||||
|
||||
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root, bool trackLastModified)
|
||||
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root)
|
||||
{
|
||||
return make_ref<PosixSourceAccessor>(std::move(root), trackLastModified);
|
||||
return make_ref<PosixSourceAccessor>(std::move(root));
|
||||
}
|
||||
} // namespace nix
|
||||
|
||||
@@ -136,7 +136,7 @@ static void extract_archive(TarArchive & archive, const std::filesystem::path &
|
||||
if (!name)
|
||||
throw Error("cannot get archive member name: %s", archive_error_string(archive.archive));
|
||||
if (r == ARCHIVE_WARN)
|
||||
warn("getting archive member '%1%': %2%", name, archive_error_string(archive.archive));
|
||||
warn(archive_error_string(archive.archive));
|
||||
else
|
||||
archive.check(r);
|
||||
|
||||
@@ -193,7 +193,7 @@ time_t unpackTarfileToSink(TarArchive & archive, ExtendedFileSystemObjectSink &
|
||||
throw Error("cannot get archive member name: %s", archive_error_string(archive.archive));
|
||||
auto cpath = CanonPath{path};
|
||||
if (r == ARCHIVE_WARN)
|
||||
warn("getting archive member '%1%': %2%", path, archive_error_string(archive.archive));
|
||||
warn(archive_error_string(archive.archive));
|
||||
else
|
||||
archive.check(r);
|
||||
|
||||
|
||||
@@ -3,7 +3,6 @@
|
||||
#include "nix/util/signals.hh"
|
||||
#include "nix/util/finally.hh"
|
||||
#include "nix/util/serialise.hh"
|
||||
#include "nix/util/source-accessor.hh"
|
||||
|
||||
#include <fcntl.h>
|
||||
#include <unistd.h>
|
||||
@@ -302,10 +301,10 @@ openFileEnsureBeneathNoSymlinksIterative(Descriptor dirFd, const CanonPath & pat
|
||||
if (errno == ENOTDIR) /* Path component might be a symlink. */ {
|
||||
struct ::stat st;
|
||||
if (::fstatat(getParentFd(), component.c_str(), &st, AT_SYMLINK_NOFOLLOW) == 0 && S_ISLNK(st.st_mode))
|
||||
throw SymlinkNotAllowed(path2);
|
||||
throw unix::SymlinkNotAllowed(path2);
|
||||
errno = ENOTDIR; /* Restore the errno. */
|
||||
} else if (errno == ELOOP) {
|
||||
throw SymlinkNotAllowed(path2);
|
||||
throw unix::SymlinkNotAllowed(path2);
|
||||
}
|
||||
|
||||
return INVALID_DESCRIPTOR;
|
||||
@@ -316,7 +315,7 @@ openFileEnsureBeneathNoSymlinksIterative(Descriptor dirFd, const CanonPath & pat
|
||||
|
||||
auto res = ::openat(getParentFd(), std::string(path.baseName().value()).c_str(), flags | O_NOFOLLOW, mode);
|
||||
if (res < 0 && errno == ELOOP)
|
||||
throw SymlinkNotAllowed(path);
|
||||
throw unix::SymlinkNotAllowed(path);
|
||||
return res;
|
||||
}
|
||||
|
||||
@@ -329,7 +328,7 @@ Descriptor unix::openFileEnsureBeneathNoSymlinks(Descriptor dirFd, const CanonPa
|
||||
dirFd, path.rel_c_str(), flags, static_cast<uint64_t>(mode), RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS);
|
||||
if (maybeFd) {
|
||||
if (*maybeFd < 0 && errno == ELOOP)
|
||||
throw SymlinkNotAllowed(path);
|
||||
throw unix::SymlinkNotAllowed(path);
|
||||
return *maybeFd;
|
||||
}
|
||||
#endif
|
||||
|
||||
@@ -38,7 +38,7 @@ struct CmdAddToStore : MixDryRun, StoreCommand
|
||||
if (!namePart)
|
||||
namePart = baseNameOf(path);
|
||||
|
||||
SourcePath sourcePath = makeFSSourceAccessor(makeParentCanonical(path));
|
||||
auto sourcePath = PosixSourceAccessor::createAtRoot(makeParentCanonical(path));
|
||||
|
||||
auto storePath = dryRun ? store->computeStorePath(*namePart, sourcePath, caMethod, hashAlgo, {}).first
|
||||
: store->addToStoreSlow(*namePart, sourcePath, caMethod, hashAlgo, {}).path;
|
||||
|
||||
@@ -60,11 +60,7 @@ struct CmdShowDerivation : InstallablesCommand, MixPrintJSON
|
||||
|
||||
jsonRoot[drvPath.to_string()] = store->readDerivation(drvPath);
|
||||
}
|
||||
printJSON(
|
||||
nlohmann::json{
|
||||
{"version", expectedJsonVersionDerivation},
|
||||
{"derivations", std::move(jsonRoot)},
|
||||
});
|
||||
printJSON(jsonRoot);
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@@ -85,7 +85,9 @@ struct CmdHashBase : Command
|
||||
return std::make_unique<HashSink>(hashAlgo);
|
||||
};
|
||||
|
||||
auto makeSourcePath = [&]() -> SourcePath { return makeFSSourceAccessor(makeParentCanonical(path)); };
|
||||
auto makeSourcePath = [&]() -> SourcePath {
|
||||
return PosixSourceAccessor::createAtRoot(makeParentCanonical(path));
|
||||
};
|
||||
|
||||
Hash h{HashAlgorithm::SHA256}; // throwaway def to appease C++
|
||||
switch (mode) {
|
||||
|
||||
@@ -613,8 +613,6 @@ static void main_nix_build(int argc, char ** argv)
|
||||
environment variables and shell functions. Also don't
|
||||
lose the current $PATH directories. */
|
||||
auto rcfile = (tmpDir.path() / "rc").string();
|
||||
auto tz = getEnv("TZ");
|
||||
auto tzExport = tz ? "export TZ=" + escapeShellArgAlways(*tz) + "; " : "";
|
||||
std::string rc = fmt(
|
||||
(R"(_nix_shell_clean_tmpdir() { command rm -rf %1%; };)"s
|
||||
"trap _nix_shell_clean_tmpdir EXIT; "
|
||||
@@ -648,7 +646,7 @@ static void main_nix_build(int argc, char ** argv)
|
||||
(pure ? "" : "PATH=$PATH:$p; unset p; "),
|
||||
escapeShellArgAlways(dirOf(*shell)),
|
||||
escapeShellArgAlways(*shell),
|
||||
tzExport,
|
||||
(getenv("TZ") ? (std::string("export TZ=") + escapeShellArgAlways(getenv("TZ")) + "; ") : ""),
|
||||
envCommand);
|
||||
vomit("Sourcing nix-shell with file %s and contents:\n%s", rcfile, rc);
|
||||
writeFile(rcfile, rc);
|
||||
|
||||
@@ -191,7 +191,7 @@ static void opAdd(Strings opFlags, Strings opArgs)
|
||||
throw UsageError("unknown flag");
|
||||
|
||||
for (auto & i : opArgs) {
|
||||
SourcePath sourcePath = makeFSSourceAccessor(makeParentCanonical(i));
|
||||
auto sourcePath = PosixSourceAccessor::createAtRoot(makeParentCanonical(i));
|
||||
cout << fmt("%s\n", store->printStorePath(store->addToStore(std::string(baseNameOf(i)), sourcePath)));
|
||||
}
|
||||
}
|
||||
@@ -215,7 +215,7 @@ static void opAddFixed(Strings opFlags, Strings opArgs)
|
||||
opArgs.pop_front();
|
||||
|
||||
for (auto & i : opArgs) {
|
||||
SourcePath sourcePath = makeFSSourceAccessor(makeParentCanonical(i));
|
||||
auto sourcePath = PosixSourceAccessor::createAtRoot(makeParentCanonical(i));
|
||||
std::cout << fmt(
|
||||
"%s\n", store->printStorePath(store->addToStoreSlow(baseNameOf(i), sourcePath, method, hashAlgo).path));
|
||||
}
|
||||
|
||||
@@ -256,7 +256,7 @@ hashPath(char * algo, int base32, char * path)
|
||||
PPCODE:
|
||||
try {
|
||||
Hash h = hashPath(
|
||||
SourcePath{getFSSourceAccessor(), CanonPath(absPath(Path(path)))},
|
||||
PosixSourceAccessor::createAtRoot(path),
|
||||
FileIngestionMethod::NixArchive, parseHashAlgo(algo)).first;
|
||||
auto s = h.to_string(base32 ? HashFormat::Nix32 : HashFormat::Base16, false);
|
||||
XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
|
||||
@@ -336,7 +336,7 @@ StoreWrapper::addToStore(char * srcPath, int recursive, char * algo)
|
||||
auto method = recursive ? ContentAddressMethod::Raw::NixArchive : ContentAddressMethod::Raw::Flat;
|
||||
auto path = THIS->store->addToStore(
|
||||
std::string(baseNameOf(srcPath)),
|
||||
SourcePath{getFSSourceAccessor(), CanonPath(absPath(Path(srcPath)))},
|
||||
PosixSourceAccessor::createAtRoot(srcPath),
|
||||
method, parseHashAlgo(algo));
|
||||
XPUSHs(sv_2mortal(newSVpv(THIS->store->printStorePath(path).c_str(), 0)));
|
||||
} catch (Error & e) {
|
||||
|
||||
@@ -6,7 +6,7 @@ export NIX_TESTS_CA_BY_DEFAULT=1
|
||||
|
||||
drvPath=$(nix-instantiate ../simple.nix)
|
||||
|
||||
nix derivation show "$drvPath" | jq '.derivations[]' > "$TEST_HOME"/simple.json
|
||||
nix derivation show "$drvPath" | jq .[] > "$TEST_HOME"/simple.json
|
||||
|
||||
drvPath2=$(nix derivation add < "$TEST_HOME"/simple.json)
|
||||
|
||||
@@ -27,5 +27,5 @@ drvPath4=$(nix derivation add < "$TEST_HOME"/foo.json)
|
||||
[[ -e "$drvPath3" ]]
|
||||
|
||||
# The modified derivation read back as JSON matches
|
||||
nix derivation show "$drvPath3" | jq '.derivations[]' > "$TEST_HOME"/foo-read.json
|
||||
nix derivation show "$drvPath3" | jq .[] > "$TEST_HOME"/foo-read.json
|
||||
diff "$TEST_HOME"/foo.json "$TEST_HOME"/foo-read.json
|
||||
|
||||
@@ -4,7 +4,7 @@ source common.sh
|
||||
|
||||
drvPath=$(nix-instantiate simple.nix)
|
||||
|
||||
nix derivation show "$drvPath" | jq '.derivations[]' > "$TEST_HOME/simple.json"
|
||||
nix derivation show "$drvPath" | jq '.[]' > "$TEST_HOME/simple.json"
|
||||
|
||||
# Round tripping to JSON works
|
||||
drvPath2=$(nix derivation add < "$TEST_HOME/simple.json")
|
||||
|
||||
@@ -18,7 +18,7 @@ mkDerivation rec {
|
||||
PATH=${builtins.getEnv "EXTRA_PATH"}:$PATH
|
||||
|
||||
# JSON of pre-existing drv
|
||||
nix derivation show $drv | jq '.derivations[]' > drv0.json
|
||||
nix derivation show $drv | jq .[] > drv0.json
|
||||
|
||||
# Fix name
|
||||
jq < drv0.json '.name = "${innerName}"' > drv1.json
|
||||
|
||||
@@ -16,7 +16,7 @@ printf 0 > "$TEST_ROOT"/counter
|
||||
|
||||
# `nix derivation add` with impure derivations work
|
||||
drvPath=$(nix-instantiate ./impure-derivations.nix -A impure)
|
||||
nix derivation show "$drvPath" | jq '.derivations[]' > "$TEST_HOME"/impure-drv.json
|
||||
nix derivation show "$drvPath" | jq .[] > "$TEST_HOME"/impure-drv.json
|
||||
drvPath2=$(nix derivation add < "$TEST_HOME"/impure-drv.json)
|
||||
[[ "$drvPath" = "$drvPath2" ]]
|
||||
|
||||
@@ -50,8 +50,8 @@ path4=$(nix build -L --no-link --json --file ./impure-derivations.nix impureOnIm
|
||||
(! nix build -L --no-link --json --file ./impure-derivations.nix inputAddressed 2>&1) | grep 'depends on impure derivation'
|
||||
|
||||
drvPath=$(nix eval --json --file ./impure-derivations.nix impure.drvPath | jq -r .)
|
||||
[[ $(nix derivation show "$drvPath" | jq ".derivations[\"$(basename "$drvPath")\"].outputs.out.impure") = true ]]
|
||||
[[ $(nix derivation show "$drvPath" | jq ".derivations[\"$(basename "$drvPath")\"].outputs.stuff.impure") = true ]]
|
||||
[[ $(nix derivation show "$drvPath" | jq ".[\"$(basename "$drvPath")\"].outputs.out.impure") = true ]]
|
||||
[[ $(nix derivation show "$drvPath" | jq ".[\"$(basename "$drvPath")\"].outputs.stuff.impure") = true ]]
|
||||
|
||||
# Fixed-output derivations *can* depend on impure derivations.
|
||||
path5=$(nix build -L --no-link --json --file ./impure-derivations.nix contentAddressed | jq -r .[].outputs.out)
|
||||
|
||||
@@ -106,16 +106,14 @@ let
|
||||
|
||||
foo = runCommand "foo" { } ''
|
||||
mkdir -p $out/bin
|
||||
echo '#!${shell}' > $out/bin/foo
|
||||
echo 'echo ${fooContents}' >> $out/bin/foo
|
||||
echo 'echo ${fooContents}' > $out/bin/foo
|
||||
chmod a+rx $out/bin/foo
|
||||
ln -s ${shell} $out/bin/bash
|
||||
'';
|
||||
|
||||
bar = runCommand "bar" { } ''
|
||||
mkdir -p $out/bin
|
||||
echo '#!${shell}' > $out/bin/bar
|
||||
echo 'echo bar' >> $out/bin/bar
|
||||
echo 'echo bar' > $out/bin/bar
|
||||
chmod a+rx $out/bin/bar
|
||||
'';
|
||||
|
||||
@@ -128,8 +126,7 @@ let
|
||||
# ruby "interpreter" that outputs "$@"
|
||||
ruby = runCommand "ruby" { } ''
|
||||
mkdir -p $out/bin
|
||||
echo '#!${shell}' > $out/bin/ruby
|
||||
echo 'printf %s "$*"' >> $out/bin/ruby
|
||||
echo 'printf %s "$*"' > $out/bin/ruby
|
||||
chmod a+rx $out/bin/ruby
|
||||
'';
|
||||
|
||||
|
||||
@@ -49,4 +49,4 @@ expectStderr 0 nix-instantiate --expr "$hackyExpr" --eval --strict | grepQuiet "
|
||||
|
||||
# Check it works with the expected structured attrs
|
||||
hacky=$(nix-instantiate --expr "$hackyExpr")
|
||||
nix derivation show "$hacky" | jq --exit-status '.derivations."'"$(basename "$hacky")"'".structuredAttrs | . == {"a": 1}'
|
||||
nix derivation show "$hacky" | jq --exit-status '."'"$(basename "$hacky")"'".structuredAttrs | . == {"a": 1}'
|
||||
|
||||
@@ -147,7 +147,7 @@ in
|
||||
else:
|
||||
machine.fail(f"nix path-info {pkg}")
|
||||
|
||||
def setup_s3(populate_bucket=[], public=False, versioned=False, profiles=None):
|
||||
def setup_s3(populate_bucket=[], public=False, versioned=False):
|
||||
"""
|
||||
Decorator that creates/destroys a unique bucket for each test.
|
||||
Optionally pre-populates bucket with specified packages.
|
||||
@@ -157,22 +157,9 @@ in
|
||||
populate_bucket: List of packages to upload before test runs
|
||||
public: If True, make the bucket publicly accessible
|
||||
versioned: If True, enable versioning on the bucket before populating
|
||||
profiles: Dict of AWS profiles to create, e.g.:
|
||||
{"valid": {"access_key": "...", "secret_key": "..."},
|
||||
"invalid": {"access_key": "WRONG", "secret_key": "WRONG"}}
|
||||
Profiles are created on the client machine at /root/.aws/credentials
|
||||
"""
|
||||
def decorator(test_func):
|
||||
def wrapper():
|
||||
# Restart nix-daemon on both machines to clear the credential provider cache.
|
||||
# The AwsCredentialProviderImpl singleton persists in the daemon process,
|
||||
# and its cache can cause credentials from previous tests to be reused.
|
||||
# We reset-failed first to avoid systemd's start rate limiting.
|
||||
server.succeed("systemctl reset-failed nix-daemon.service nix-daemon.socket")
|
||||
server.succeed("systemctl restart nix-daemon")
|
||||
client.succeed("systemctl reset-failed nix-daemon.service nix-daemon.socket")
|
||||
client.succeed("systemctl restart nix-daemon")
|
||||
|
||||
bucket = str(uuid.uuid4())
|
||||
server.succeed(f"mc mb minio/{bucket}")
|
||||
try:
|
||||
@@ -180,15 +167,6 @@ in
|
||||
server.succeed(f"mc anonymous set download minio/{bucket}")
|
||||
if versioned:
|
||||
server.succeed(f"mc version enable minio/{bucket}")
|
||||
if profiles:
|
||||
# Build credentials file content
|
||||
creds_content = ""
|
||||
for name, creds in profiles.items():
|
||||
creds_content += f"[{name}]\n"
|
||||
creds_content += f"aws_access_key_id = {creds['access_key']}\n"
|
||||
creds_content += f"aws_secret_access_key = {creds['secret_key']}\n\n"
|
||||
client.succeed("mkdir -p /root/.aws")
|
||||
client.succeed(f"cat > /root/.aws/credentials << 'AWSCREDS'\n{creds_content}AWSCREDS")
|
||||
if populate_bucket:
|
||||
store_url = make_s3_url(bucket)
|
||||
for pkg in populate_bucket:
|
||||
@@ -196,9 +174,6 @@ in
|
||||
test_func(bucket)
|
||||
finally:
|
||||
server.succeed(f"mc rb --force minio/{bucket}")
|
||||
# Clean up AWS profiles if created
|
||||
if profiles:
|
||||
client.succeed("rm -rf /root/.aws")
|
||||
# Clean up client store - only delete if path exists
|
||||
for pkg in PKGS.values():
|
||||
client.succeed(f"[ ! -e {pkg} ] || nix store delete --ignore-liveness {pkg}")
|
||||
@@ -789,108 +764,6 @@ in
|
||||
|
||||
print(" ✓ Compressed log uploaded with multipart")
|
||||
|
||||
@setup_s3(
|
||||
populate_bucket=[PKGS['A']],
|
||||
profiles={
|
||||
"valid": {"access_key": ACCESS_KEY, "secret_key": SECRET_KEY},
|
||||
"invalid": {"access_key": "INVALIDKEY", "secret_key": "INVALIDSECRET"},
|
||||
}
|
||||
)
|
||||
def test_profile_credentials(bucket):
|
||||
"""Test that profile-based credentials work without environment variables"""
|
||||
print("\n=== Testing Profile-Based Credentials ===")
|
||||
|
||||
store_url = make_s3_url(bucket, profile="valid")
|
||||
|
||||
# Verify store info works with profile credentials (no env vars)
|
||||
client.succeed(f"HOME=/root nix store info --store '{store_url}' >&2")
|
||||
print(" ✓ nix store info works with profile credentials")
|
||||
|
||||
# Verify we can copy from the store using profile
|
||||
verify_packages_in_store(client, PKGS['A'], should_exist=False)
|
||||
client.succeed(f"HOME=/root nix copy --no-check-sigs --from '{store_url}' {PKGS['A']}")
|
||||
verify_packages_in_store(client, PKGS['A'])
|
||||
print(" ✓ nix copy works with profile credentials")
|
||||
|
||||
# Clean up the package we just copied so we can test invalid profile
|
||||
client.succeed(f"nix store delete --ignore-liveness {PKGS['A']}")
|
||||
verify_packages_in_store(client, PKGS['A'], should_exist=False)
|
||||
|
||||
# Verify invalid profile fails when trying to copy
|
||||
invalid_url = make_s3_url(bucket, profile="invalid")
|
||||
client.fail(f"HOME=/root nix copy --no-check-sigs --from '{invalid_url}' {PKGS['A']} 2>&1")
|
||||
print(" ✓ Invalid profile credentials correctly rejected")
|
||||
|
||||
@setup_s3(
|
||||
populate_bucket=[PKGS['A']],
|
||||
profiles={
|
||||
"wrong": {"access_key": "WRONGKEY", "secret_key": "WRONGSECRET"},
|
||||
}
|
||||
)
|
||||
def test_env_vars_precedence(bucket):
|
||||
"""Test that environment variables take precedence over profile credentials"""
|
||||
print("\n=== Testing Environment Variables Precedence ===")
|
||||
|
||||
# Use profile with wrong credentials, but provide correct creds via env vars
|
||||
store_url = make_s3_url(bucket, profile="wrong")
|
||||
|
||||
# Ensure package is not in client store
|
||||
verify_packages_in_store(client, PKGS['A'], should_exist=False)
|
||||
|
||||
# This should succeed because env vars (correct) override profile (wrong)
|
||||
output = client.succeed(
|
||||
f"HOME=/root {ENV_WITH_CREDS} nix copy --no-check-sigs --debug --from '{store_url}' {PKGS['A']} 2>&1"
|
||||
)
|
||||
print(" ✓ nix copy succeeded with env vars overriding wrong profile")
|
||||
|
||||
# Verify the credential chain shows Environment provider was added
|
||||
if "Added AWS Environment Credential Provider" not in output:
|
||||
print("Debug output:")
|
||||
print(output)
|
||||
raise Exception("Expected Environment provider to be added to chain")
|
||||
print(" ✓ Environment provider added to credential chain")
|
||||
|
||||
# Clean up the package so we can test again without env vars
|
||||
client.succeed(f"nix store delete --ignore-liveness {PKGS['A']}")
|
||||
verify_packages_in_store(client, PKGS['A'], should_exist=False)
|
||||
|
||||
# Without env vars, same URL should fail (proving profile creds are actually wrong)
|
||||
client.fail(f"HOME=/root nix copy --no-check-sigs --from '{store_url}' {PKGS['A']} 2>&1")
|
||||
print(" ✓ Without env vars, wrong profile credentials correctly fail")
|
||||
|
||||
@setup_s3(
|
||||
populate_bucket=[PKGS['A']],
|
||||
profiles={
|
||||
"testprofile": {"access_key": ACCESS_KEY, "secret_key": SECRET_KEY},
|
||||
}
|
||||
)
|
||||
def test_credential_provider_chain(bucket):
|
||||
"""Test that debug logging shows which providers are added to the chain"""
|
||||
print("\n=== Testing Credential Provider Chain Logging ===")
|
||||
|
||||
store_url = make_s3_url(bucket, profile="testprofile")
|
||||
|
||||
output = client.succeed(
|
||||
f"HOME=/root nix store info --debug --store '{store_url}' 2>&1"
|
||||
)
|
||||
|
||||
# For a named profile, we expect to see these providers in the chain
|
||||
expected_providers = ["Environment", "Profile", "IMDS"]
|
||||
for provider in expected_providers:
|
||||
msg = f"Added AWS {provider} Credential Provider to chain for profile 'testprofile'"
|
||||
if msg not in output:
|
||||
print("Debug output:")
|
||||
print(output)
|
||||
raise Exception(f"Expected to find: {msg}")
|
||||
print(f" ✓ {provider} provider added to chain")
|
||||
|
||||
# SSO should be skipped (no SSO config for this profile)
|
||||
if "Skipped AWS SSO Credential Provider for profile 'testprofile'" not in output:
|
||||
print("Debug output:")
|
||||
print(output)
|
||||
raise Exception("Expected SSO provider to be skipped")
|
||||
print(" ✓ SSO provider correctly skipped (not configured)")
|
||||
|
||||
# ============================================================================
|
||||
# Main Test Execution
|
||||
# ============================================================================
|
||||
@@ -924,9 +797,6 @@ in
|
||||
test_multipart_upload_basic()
|
||||
test_multipart_threshold()
|
||||
test_multipart_with_log_compression()
|
||||
test_profile_credentials()
|
||||
test_env_vars_precedence()
|
||||
test_credential_provider_chain()
|
||||
|
||||
print("\n" + "="*80)
|
||||
print("✓ All S3 Binary Cache Store Tests Passed!")
|
||||
|
||||
Reference in New Issue
Block a user