Compare commits

..
Author SHA1 Message Date
Sergei Zimmerman bb5d2e9676 libutil-tests: Add tests for makeFSSourceAccessor
Should be pretty self-explanatory. We didn't really have unit tests
for the filesystem source accessor. Now we do and this will be immensely
useful for implementing a unix-only smarter accessor that doesn't suffer
from TOCTOU on symlinks.
2025-12-16 03:04:12 +03:00
Sergei Zimmerman 8b0026312a treewide: Get rid of PosixSourceAccessor::createAtRoot
We'd like to split out the implementation into Unix/Windows-specific
parts to more easily iterate on improving UNIX accessors to make use
of dirfd-based operations (or even openat2). This should be hidden behind
the appropriate interface and not exposed as a static member function of
the PosixSourceAccessor.
2025-12-16 03:00:24 +03:00
John Ericson 85bbfd4493 Merge pull request #14803 from Eveeifyeve/windows-work
nix: don't require ln to build libstore
2025-12-15 22:27:45 +00:00
eveeifyeve 832b81761e nix: don't require ln to build libstore 2025-12-16 08:33:20 +11:00
John Ericson df7542247e Merge pull request #14801 from NixOS/coroutine-child-output-0
Use coroutines for worker child I/O
2025-12-15 20:25:17 +00:00
Jörg Thalheim 49f666c64d Merge pull request #14799 from NixOS/tarball-cache-v2
libfetchers: Bump tarball-cache version to v2
2025-12-15 19:58:28 +00:00
Jörg Thalheim 11f5a3124b Merge pull request #14645 from lovesegfault/s3-sts
feat(libstore): add AWS SSO support for S3 authentication
2025-12-15 19:44:26 +00:00
John Ericson 92e698426b Use coroutines for worker child I/O
This will enable way more RAII going forward.
2025-12-15 14:28:07 -05:00
John Ericson 906334686c Make worker timeouts a bit more strongly typed
This tidies things up in general, but also prepares the way for the next
commit in particular.
2025-12-15 14:27:21 -05:00
Sergei Zimmerman 0ffe83aa14 libfetchers: Bump tarball-cache version to v2
Unfortunately previous tarball caches had loose objects written to
them and subsequent switch to thin packfiles. This results in possibly
broken thin packfiles when the loose objects backend is disabled. Thin
packfiles do not necessarily contain the whole closure of objects.
When packfilesOnly is true we end up with an inconsistent state where
a tree lives in a packfiles which refers to a blob in the loose objects
backend.

In the future we might want to nuke old cache directories and repack
the tarball cache.
2025-12-15 22:12:08 +03:00
John Ericson 8e044f1ed0 Merge pull request #14798 from NixOS/devshell-debug
dev-shell: Set mesonBuildType to debugoptimized
2025-12-15 19:01:45 +00:00
Jörg Thalheim 453dbab1e8 fix(libstore/aws-creds): respect AWS_PROFILE environment variable
The SSO provider was unconditionally setting profile_name_override to
the (potentially empty) profile string from the S3 URL. When profile
was empty, this prevented the AWS CRT SDK from falling back to the
AWS_PROFILE environment variable.

Only set profile_name_override when a profile is explicitly specified
in the URL, allowing the SDK's built-in AWS_PROFILE handling to work.
2025-12-15 19:40:34 +01:00
Eelco Dolstra fc81840a8e dev-shell: Set mesonBuildType to debugoptimized
Previously, we got debug symbols implicitly because we were using
`separateDebugInfo = true`, which adds `-ggdb` to the compiler flags.
2025-12-15 19:09:37 +01:00
Bernardo Meurer 71bdb33a36 test(s3-binary-cache-store): test profiles and provider chain 2025-12-15 19:05:16 +01:00
Bernardo Meurer 0595c5f7ee test(s3-binary-cache-store): clear credential cache between tests 2025-12-15 19:05:16 +01:00
Bernardo Meurer 11f108d898 test(s3-binary-cache-store): add profile support for setup_for_s3 2025-12-15 19:05:16 +01:00
Bernardo Meurer 128b2b5c56 chore(libstore/aws-creds): remove unused includes 2025-12-15 19:05:16 +01:00
Bernardo Meurer 508d4463e5 fix(libstore/aws-creds): add STS support for default profile
The default (empty) profile case was using CreateCredentialsProviderChainDefault
which didn't properly support role_arn/source_profile based role assumption via
STS because TLS context wasn't being passed to the Profile provider.

This change unifies the credential chain for all profiles (default and named),
ensuring:
- Consistent behavior between default and named profiles
- Proper TLS context is passed for STS operations
- SSO support works for both cases
2025-12-15 19:05:16 +01:00
Bernardo Meurer 3c8e45c061 refactor(libstore/aws-creds): improve error handling and logging
Add validation for TLS context and client bootstrap initialization,
with appropriate error messages when these fail. The TLS context failure
is now a warning that gracefully disables SSO, while bootstrap failure
throws since it's required for all providers.
2025-12-15 19:05:16 +01:00
Jörg Thalheim ec91479076 libstore: add AWS SSO support for S3 authentication
This enables seamless AWS SSO authentication for S3 binary caches
without requiring users to manually export credentials.

This adds SSO support by calling aws_credentials_provider_new_sso() from
the C library directly. It builds a custom credential chain: Env → SSO →
Profile → IMDS

The SSO provider requires a TLS context for HTTPS connections to SSO
endpoints, which is created once and shared across all providers.
2025-12-15 19:05:16 +01:00
Sergei Zimmerman b398c14045 Merge pull request #14795 from NixOS/git-repo-options
Add GitRepo::Options type
2025-12-15 17:38:44 +00:00
Eelco Dolstra 9a6f1e6266 GitRepo: Implement create flag
This was ignored for some reason.
2025-12-15 14:36:04 +01:00
Eelco Dolstra 1c728ce0de Add GitRepo::Options type
This makes a bunch of bool parameters more explicit.
2025-12-15 14:35:19 +01:00
Jörg Thalheim a6eb2e91b7 Merge pull request #14774 from roberth/fix-getenv-segfault
Fix getenv segfault
2025-12-13 08:09:54 +00:00
Robert Hensing 76c09bf3d4 Fix nix-build.cc double getenv("TZ") race condition
This is mostly theoretical, but the code was calling getenv("TZ")
twice: once to check if it's non-null, and again to get its value.
This creates a potential race condition where the environment could
change between calls.
2025-12-13 08:34:27 +01:00
Robert Hensing de6fdb7da5 Extract getUnitTestData() to test-data.hh and fix unsafe getenv calls
The nix_api_store.cc tests and derivation-parser-bench.cc were using raw
getenv() calls or unsafe .value() calls on optional, which would segfault
when passed to std::filesystem::path constructor if the
_NIX_TEST_UNIT_DATA environment variable was not set.
2025-12-13 08:34:27 +01:00
Robert Hensing b54dfb66dd Fix segfault in getUnitTestData() when env var not set
The previous implementation called .value() on std::optional without
checking if it had a value. When _NIX_TEST_UNIT_DATA was not set, this
would throw std::bad_optional_access or cause a segfault in code that
used the raw getenv() result.

The new implementation checks the optional first and throws an Error
with a helpful message directing users to run tests via meson. The
example includes --gdb since this situation may arise when trying to
debug tests without knowing about meson's test infrastructure.
2025-12-13 08:34:27 +01:00
Sergei Zimmerman bb718d20a2 Merge pull request #14778 from agucova/fix-macos-shebang-flakiness
test: add shebangs to shell.nix test scripts
2025-12-13 03:41:30 +00:00
John Ericson 3b3bd018a5 Merge pull request #14781 from NixOS/curl-cleanup
libstore: Clean up cruft from filetransfer
2025-12-13 03:40:01 +00:00
tomberek 26b86a02db Merge pull request #14780 from NixOS/tarfile-warning
libutil/tarfile: Mention pathname in warning
2025-12-13 03:06:09 +00:00
Sergei Zimmerman 8358409fd0 Merge pull request #14772 from GrahamDennis/gdennis/fix-heap-use-after-free
[libstore]: Fix a heap-use-after-free bug
2025-12-13 00:25:11 +00:00
Sergei Zimmerman 46670a7f46 libstore/filetransfer: Replace curl_multi_wait with curl_multi_poll and get rid of CPP
Since 7.68 libcurl already provides curl_multi_wakeup, so we can drop the hacky
pipe setup (libcurl does this internally).
2025-12-13 03:00:58 +03:00
Sergei Zimmerman ea96e6d07c libstore/filetransfer: Factor out appendHeaders, use std::unique_ptr to simplify ownership
Pretty self-explanatory. More RAII is good and unclutters the already heavily overloaded
destructors from ownership logic. Not yet touching CURL *req because that would be too churny.
2025-12-13 02:59:18 +03:00
Sergei Zimmerman 7e3de5361a libutil/tarfile: Mention pathname in warning
Fetching gcc-15.2.0.tar.gz I get a warning about UTF8 archive names. This
now mentions problematic pathnames.

warning: getting archive member 'gcc-15.2.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Äfoo.go': Pathname can't be converted from UTF-8 to current locale.
warning: getting archive member 'gcc-15.2.0/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir/Ämain.go': Pathname can't be converted from UTF-8 to current locale.

Also apparently libarchive depends on locale (yikes). Fixing reproducibility issues
that stem from this is a separate issue. At least having the warning actually mention
the pathname should be useful enough even though it's not actionable.

At least using the default locale yields something sane:

builtins.readDir "${gcc}/gcc/testsuite/go.test/test/fixedbugs/issue27836.dir"
{
  "Äfoo.go" = "regular";
  "Ämain.go" = "regular";
}
2025-12-13 01:54:14 +03:00
Agustín Covarrubias 7b3d7eb634 test: add shebangs to shell.nix test scripts
Fix intermittent SIGSEGV (exit code 139) on macOS when running
  nix-shell and shebang tests inside the nix sandbox.

  The foo, bar, and ruby test scripts were created without shebangs,
  which causes intermittent crashes when executed via command
  substitution on macOS. Adding proper shebangs resolves the flakiness.

  Potentially closes: #13106
2025-12-12 18:04:37 -03:00
Graham Dennis 819a61acae [libstore]: Fix a heap-use-after-free bug 2025-12-12 08:42:23 +11:00
John Ericson ccba158780 Merge pull request #14767 from NixOS/bump-2.34.0
Bump version
2025-12-10 21:14:12 +00:00
John Ericson 4945c38b88 Merge pull request #14770 from NixOS/derivation-show-json-guidlines
Bring `nix derivation show` in compliance with JSON guidelines
2025-12-10 21:13:27 +00:00
John Ericson 0f18076f3a Bring nix derivation show in compliance with JSON guidelines
This matches what we just did for `nix path-info`, and I hope will allow
us to avoiding any more breaking changes to this command for the
foreseeable future.
2025-12-10 15:30:12 -05:00
Eelco Dolstra c6ddc5cf1d Bump version 2025-12-10 17:35:28 +01:00
93 changed files with 1100 additions and 2166 deletions

No files matched your search

@@ -24,8 +24,8 @@ inputs:
description: "Github token"
required: true
use_cache:
description: "Whether to setup github actions cache (not implemented currently)"
default: false
description: "Whether to setup magic-nix-cache"
default: true
required: false
runs:
using: "composite"
@@ -122,3 +122,10 @@ runs:
source-url: ${{ inputs.experimental-installer-version != 'latest' && 'https://artifacts.nixos.org/experimental-installer/tag/${{ inputs.experimental-installer-version }}/${{ env.EXPERIMENTAL_INSTALLER_ARTIFACT }}' || '' }}
nix-package-url: ${{ inputs.dogfood == 'true' && steps.download-nix-installer.outputs.tarball-path || (inputs.tarball_url || '') }}
extra-conf: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 # v13
if: ${{ inputs.use_cache == 'true' }}
with:
diagnostic-endpoint: ''
use-flakehub: false
use-gha-cache: true
source-revision: 92d9581367be2233c2d5714a2640e1339f4087d8 # main
+1 -1
View File
@@ -26,7 +26,7 @@ jobs:
# required to find all branches
fetch-depth: 0
- name: Create backport PRs
uses: korthout/backport-action@c656f5d5851037b2b38fb5db2691a03fa229e3b2 # v4.0.1
uses: korthout/backport-action@d07416681cab29bf2661702f925f020aaa962997 # v3.4.1
id: backport
with:
# Config README: https://github.com/korthout/backport-action#backport-action
+76 -25
View File
@@ -164,7 +164,7 @@ jobs:
steps:
- uses: actions/checkout@v6
- name: Download installer tarball
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
uses: actions/download-artifact@v6
with:
name: installer-${{matrix.os}}
path: out
@@ -197,20 +197,79 @@ jobs:
- run: exec bash -c "nix-channel --add https://releases.nixos.org/nixos/unstable/nixos-23.05pre466020.60c1d71f2ba nixpkgs"
- run: exec bash -c "nix-channel --update && nix-env -iA nixpkgs.hello && hello"
# Steps to test CI automation in your own fork.
# 1. Sign-up for https://hub.docker.com/
# 2. Store your dockerhub username as DOCKERHUB_USERNAME in "Repository secrets" of your fork repository settings (https://github.com/$githubuser/nix/settings/secrets/actions)
# 3. Create an access token in https://hub.docker.com/settings/security and store it as DOCKERHUB_TOKEN in "Repository secrets" of your fork
check_secrets:
permissions:
contents: none
name: Check presence of secrets
runs-on: ubuntu-24.04
outputs:
docker: ${{ steps.secret.outputs.docker }}
steps:
- name: Check for DockerHub secrets
id: secret
env:
_DOCKER_SECRETS: ${{ secrets.DOCKERHUB_USERNAME }}${{ secrets.DOCKERHUB_TOKEN }}
run: |
echo "docker=${{ env._DOCKER_SECRETS != '' }}" >> $GITHUB_OUTPUT
docker_push_image:
name: Push docker image to DockerHub and GHCR
needs: [flake_regressions, installer_test]
if: github.event_name == 'push' && github.ref_name == 'master'
uses: ./.github/workflows/docker-push.yml
with:
ref: ${{ github.sha }}
is_master: true
needs: [tests, check_secrets]
permissions:
contents: read
packages: write
secrets:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
if: >-
needs.check_secrets.outputs.docker == 'true' &&
github.event_name == 'push' &&
github.ref_name == 'master'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
extra_nix_config: |
experimental-features = flakes nix-command
- run: echo NIX_VERSION="$(nix eval .\#nix.version | tr -d \")" >> $GITHUB_ENV
- run: nix build .#dockerImage -L
- run: docker load -i ./result/image.tar.gz
- run: docker tag nix:$NIX_VERSION ${{ secrets.DOCKERHUB_USERNAME }}/nix:$NIX_VERSION
- run: docker tag nix:$NIX_VERSION ${{ secrets.DOCKERHUB_USERNAME }}/nix:master
# We'll deploy the newly built image to both Docker Hub and Github Container Registry.
#
# Push to Docker Hub first
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- run: docker push ${{ secrets.DOCKERHUB_USERNAME }}/nix:$NIX_VERSION
- run: docker push ${{ secrets.DOCKERHUB_USERNAME }}/nix:master
# Push to GitHub Container Registry as well
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Push image
run: |
IMAGE_ID=ghcr.io/${{ github.repository_owner }}/nix
# Change all uppercase to lowercase
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
docker tag nix:$NIX_VERSION $IMAGE_ID:$NIX_VERSION
docker tag nix:$NIX_VERSION $IMAGE_ID:latest
docker push $IMAGE_ID:$NIX_VERSION
docker push $IMAGE_ID:latest
# deprecated 2024-02-24
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
flake_regressions:
needs: tests
@@ -228,21 +287,13 @@ jobs:
with:
repository: NixOS/flake-regressions-data
path: flake-regressions/tests
- name: Download installer tarball
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7.0.0
- uses: ./.github/actions/install-nix-action
with:
name: installer-linux
path: out
- name: Looking up the installer tarball URL
id: installer-tarball-url
run: |
echo "installer-url=file://$GITHUB_WORKSPACE/out" >> "$GITHUB_OUTPUT"
- uses: cachix/install-nix-action@4e002c8ec80594ecd40e759629461e26c8abed15 # v31.9.0
with:
install_url: ${{ format('{0}/install', steps.installer-tarball-url.outputs.installer-url) }}
install_options: ${{ format('--tarball-url-prefix {0}', steps.installer-tarball-url.outputs.installer-url) }}
- name: Run flake regressions tests
run: MAX_FLAKES=25 flake-regressions/eval-all.sh
dogfood: ${{ github.event_name == 'workflow_dispatch' && inputs.dogfood || github.event_name != 'workflow_dispatch' }}
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
- run: nix build -L --out-link ./new-nix && PATH=$(pwd)/new-nix/bin:$PATH MAX_FLAKES=25 flake-regressions/eval-all.sh
profile_build:
needs: tests
-101
View File
@@ -1,101 +0,0 @@
name: "Push Docker Image"
on:
workflow_call:
inputs:
ref:
description: "Git ref to build the docker image from"
required: true
type: string
is_master:
description: "Whether run from master branch"
required: true
type: boolean
secrets:
DOCKERHUB_USERNAME:
required: true
DOCKERHUB_TOKEN:
required: true
permissions: {}
jobs:
# Steps to test CI automation in your own fork.
# 1. Sign-up for https://hub.docker.com/
# 2. Store your dockerhub username as DOCKERHUB_USERNAME in "Repository secrets" of your fork repository settings (https://github.com/$githubuser/nix/settings/secrets/actions)
# 3. Create an access token in https://hub.docker.com/settings/security and store it as DOCKERHUB_TOKEN in "Repository secrets" of your fork
check_secrets:
permissions:
contents: none
name: Check presence of secrets
runs-on: ubuntu-24.04
outputs:
docker: ${{ steps.secret.outputs.docker }}
steps:
- name: Check for DockerHub secrets
id: secret
env:
_DOCKER_SECRETS: ${{ secrets.DOCKERHUB_USERNAME }}${{ secrets.DOCKERHUB_TOKEN }}
run: |
echo "docker=${{ env._DOCKER_SECRETS != '' }}" >> $GITHUB_OUTPUT
push:
name: Push docker image to DockerHub and GHCR
needs: [check_secrets]
permissions:
contents: read
packages: write
if: needs.check_secrets.outputs.docker == 'true'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
fetch-depth: 0
ref: ${{ inputs.ref }}
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
extra_nix_config: |
experimental-features = flakes nix-command
- run: echo NIX_VERSION="$(nix eval .\#nix.version | tr -d \")" >> $GITHUB_ENV
- run: nix build .#dockerImage -L
- run: docker load -i ./result/image.tar.gz
# We'll deploy the newly built image to both Docker Hub and Github Container Registry.
#
# Push to Docker Hub first
- name: Login to Docker Hub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Push to Docker Hub
env:
IS_MASTER: ${{ inputs.is_master }}
DOCKERHUB_REPO: ${{ secrets.DOCKERHUB_USERNAME }}/nix
run: |
docker tag nix:$NIX_VERSION $DOCKERHUB_REPO:$NIX_VERSION
docker push $DOCKERHUB_REPO:$NIX_VERSION
if [ "$IS_MASTER" = "true" ]; then
docker tag nix:$NIX_VERSION $DOCKERHUB_REPO:master
docker push $DOCKERHUB_REPO:master
fi
# Push to GitHub Container Registry as well
- name: Login to GitHub Container Registry
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Push to GHCR
env:
IS_MASTER: ${{ inputs.is_master }}
run: |
IMAGE_ID=ghcr.io/${{ github.repository_owner }}/nix
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
docker tag nix:$NIX_VERSION $IMAGE_ID:$NIX_VERSION
docker push $IMAGE_ID:$NIX_VERSION
if [ "$IS_MASTER" = "true" ]; then
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
fi
-80
View File
@@ -1,80 +0,0 @@
name: Upload Release
on:
workflow_dispatch:
inputs:
eval_id:
description: "Hydra evaluation ID"
required: true
type: number
is_latest:
description: "Mark as latest release"
required: false
type: boolean
default: false
permissions:
contents: read
id-token: write
packages: write
jobs:
release:
runs-on: ubuntu-24.04
environment: releases
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- uses: ./.github/actions/install-nix-action
with:
dogfood: false # Use stable version
use_cache: false # Don't want any cache injection shenanigans
extra_nix_config: |
experimental-features = nix-command flakes
- name: Set NIX_PATH from flake input
run: |
NIXPKGS_PATH=$(nix build --inputs-from .# nixpkgs#path --print-out-paths --no-link)
# Shebangs with perl have issues. Pin nixpkgs this way. nix shell should maybe
# get the same uberhack that nix-shell has to support it.
echo "NIX_PATH=nixpkgs=$NIXPKGS_PATH" >> "$GITHUB_ENV"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708 # v5.1.1
with:
role-to-assume: "arn:aws:iam::080433136561:role/nix-release"
role-session-name: nix-release-oidc-${{ github.run_id }}
aws-region: eu-west-1
- name: Disable containerd image store
run: |
# Docker 28+ defaults to the containerd image store, which
# pushes layers uncompressed instead of gzip. OCI clients
# that only support gzip (e.g. go-containerregistry) fail
# with "gzip: invalid header". Disabling the containerd
# snapshotter restores the classic storage driver, which
# preserves gzip-compressed layers through the
# `docker load` / `docker push` pipeline.
echo '{"features":{"containerd-snapshotter":false}}' | sudo tee /etc/docker/daemon.json > /dev/null
sudo systemctl restart docker
- name: Login to Docker Hub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Upload release
run: |
./maintainers/upload-release.pl \
${{ inputs.eval_id }} \
--skip-git
env:
IS_LATEST: ${{ inputs.is_latest && '1' || '' }}
- name: Push to GHCR
run: |
DOCKER_OWNER="ghcr.io/$(echo '${{ github.repository_owner }}' | tr '[A-Z]' '[a-z]')/nix"
./maintainers/upload-release.pl \
${{ inputs.eval_id }} \
--skip-git \
--skip-s3 \
--docker-owner "$DOCKER_OWNER"
env:
IS_LATEST: ${{ inputs.is_latest && '1' || '' }}
+1 -1
View File
@@ -1 +1 @@
2.33.4
2.34.0
+1 -8
View File
@@ -6,14 +6,7 @@ Additionally, see [Testing Nix](./testing.md) for further instructions on how to
## Building Nix with Debug Symbols
In the development shell, set the `mesonBuildType` environment variable to `debug` before configuring the build:
```console
[nix-shell]$ export mesonBuildType=debugoptimized
```
Then, proceed to build Nix as described in [Building Nix](./building.md).
This will build Nix with debug symbols, which are essential for effective debugging.
In the development shell, `mesonBuildType` is set automatically to `debugoptimized`. This builds Nix with debug symbols, which are essential for effective debugging.
It is also possible to build without optimization for faster build:
@@ -279,35 +279,3 @@ This release was made possible by the following 33 contributors:
- Henry [**(@cootshk)**](https://github.com/cootshk)
- Martin Joerg [**(@mjoerg)**](https://github.com/mjoerg)
- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
# Release 2.33.3 (2026-02-13)
- S3 binary caches now use virtual-hosted-style addressing by default [#15208](https://github.com/NixOS/nix/issues/15208)
S3 binary caches now use virtual-hosted-style URLs
(`https://bucket.s3.region.amazonaws.com/key`) instead of path-style URLs
(`https://s3.region.amazonaws.com/bucket/key`) when connecting to standard AWS
S3 endpoints. This enables HTTP/2 multiplexing and fixes TCP connection
exhaustion (TIME_WAIT socket accumulation) under high-concurrency workloads.
A new `addressing-style` store option controls this behavior:
- `auto` (default): virtual-hosted-style for standard AWS endpoints, path-style
for custom endpoints.
- `path`: forces path-style addressing (deprecated by AWS).
- `virtual`: forces virtual-hosted-style addressing (bucket names must not
contain dots).
Bucket names containing dots (e.g., `my.bucket.name`) automatically fall back
to path-style addressing in `auto` mode, because dotted names create
multi-level subdomains that break TLS wildcard certificate validation.
Example using path-style for backwards compatibility:
```
s3://my-bucket/key?region=us-east-1&addressing-style=path
```
Additionally, TCP keep-alive is now enabled on all HTTP connections, preventing
idle connections from being silently dropped by intermediate network devices
(NATs, firewalls, load balancers).
+1 -1
View File
@@ -32,7 +32,7 @@
let
inherit (nixpkgs) lib;
officialRelease = true;
officialRelease = false;
linux32BitSystems = [ "i686-linux" ];
linux64BitSystems = [
@@ -1,110 +0,0 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGPtMiwBEAC0sFZW2QW/OaDjKm5zGRpDvHXDsMIUtlHfoi5ce8pocC63W05o
FSXbUZjZ1VfYO8lT8DFANCzTkiXYaZx0cPRG2pVY4AOQZDNFt5XrAyvw496XCAIM
DTYGFLjCqgjPt9RUFEy4MyHPJTEpB0x3rXgT4ILNu9vsj9Q0vttps7SpbZ3Ldq5H
o/BBbLW77q/vNjpYzCbBIXF7ycUGpnNv9Go/WuiDnrBMcyxh+8kjjIHB5cxZSnjJ
DUv681+m83v+gLZQGX/jexQrrf5JpS0X9qEnhGLrNUDhtyv5ud3Je4EfamkjLVVC
RlNLofgflOCsl/tP80i+K7S1QdKhUALxuJ6H0prYUflGBDxDyC8XYuJ62TT0OUpa
vJvgwVlCq8/jq+ykYQXlbuBVOzi5wAuI4l3+HqreSQYPSiwe+6N590Zbafdv1fvN
WFtZKCTGMqfyaaAnppioH9/+NWkI2AQxaYVasYM/JEYvY9pJgA7alh51jHW4JglP
ErypKfBKPKJID0QENqYoa3bDDCihuNWhgQf9dxzPlj2ckd35Zb6w4DfuSmtjaa9D
o0jZVY1JbFuxBqP09+saVPrxLHgmPxjcdzPGQQtAqdO2vyJXNEGLFMoVEZPNaLo3
QmcIJnT7oSck+4vGfOYtWUHXQynu/Tnwsv2XkA/uyw8HNe+RRMqv/apnzQARAQAB
tCdTZXJnZWkgWmltbWVybWFuIDxzZXJnZWlAemltbWVybWFuLmZvbz6JAlEEEwEK
ADsCGwEFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQQVim9TDqIC5fZRYRMU+upj
RI4d+QUCaUbsaAIZAQAKCRAU+upjRI4d+VdDD/492HRaJ/8V7R7VUzkafmb2Hb28
SLf7oiB8Uq9I7SukiDEaIT1fUhquYWQ9KWpPRNR1TX6ApXnIeuJRMGFoDVIRnmnr
cKnYYXfqqc81VxIyKvaumB7KWbS7G4Nbor8AH1ouOOOMMS50OTJOWQA4A26inIuG
n+7L8MeS5aT+3uNKDoTKsidC47vnaxNMcke1taPfbfo7vn69PsRCM/g9/7TQYU8b
6xp+pM9Ao9nJneRk2YCpsGYRrWTpaik0DFKnfpPKJM/yunhtGLF2IYAp3l1mvHPK
nnzo92zjpQuZwazEIK+23V1vRT4IjM2BewbJPAzf2/UuxEjjgNQm0tOtH2JhFNeB
VM0BVrGxWrrwrsmv6lWghTtBc6zRWyHrj/rpjtVQNmeKYrHWJeXwVz1rqgGPmB2N
k0MZD1UjHHhEs1Cntn7yLmxTPztRJCtR+euRu81Uo2NAvrMJ4xsDjaM0LeLTnzjV
9AsPjD188dOFyz7VExZum4+XaaEJ41FIPLEqU3U0GAa6stEy0ylSlIN4x9aiXXVW
xfzHHchS5jK6QAjuZxN8t01GRactNylINRf7uoTECFZTtXNqfeuk0HQxBH0LuKVE
0PxJbcNI4mVWw1KgTJ8PUVC1IXP3sEpqPdJOYiRXgnpcS26fWOBu0aQ4mhxiaJhr
/zBfrkLEqp20TdNDZLkCDQRj7TM1ARAAt73xO24curnHTTgXkkVMMRzcMLx3Mb1a
2FuddxC5hzTpEpw01L91UBrXVJEg9K2KAwP5CtCLgPCqXr47Tm7krvHxWwBksgY/
6aHRsoPQfCFUZHc0aiO+C4NCzR+aEeGKn66Oc1Hq9oUTpDgiBWhsuEPiyA1OSGF0
4L0jeTCqfm68kWp4PIK9yuugkdDsoyj6TonuMsb3V5ctHLqop9KH+eHSkUTPo+Lk
+bxaeAOJ1UfbohgbRbrYKAfsaghhOMDH3R1w2pvtUJz+sDbuQsiPFTqbxsXDTFws
H4N/AQCYnnvOhqEek2sOEZ19bJXt5UrAr10mX4PGmAkWqE1JWBxpOKG3BXSGOTu1
3dFhQfPMK+PmvUrs0kcWQr53K/aRUdKKhIfTcMfkqYTGPK5HclHph24WjXj3QFFA
SjksQTdm6486ZmLZK4CTbAFOPfTF/aWg8gu9v4ihdq6lqHNNXxv2xBAChcd59H7p
D5zy9z8SpwWR9V5JDmlF6HWIIau1c6lSsQq1xHvYM8EuPe03vJvor+2u/cn5zYF1
5ZxAuPI2i5vtavg1s8ZGAAogJ9dVcP36LdJfL9quXWvmovkd//qHIepBB+l/zQio
ZRDZlIcfV3Xycaqsb5OqHGARHE0097koipMt5y/iXlqG4Ruue6Idb8bW96EKpaWj
kKy/iNfQfQMAEQEAAYkEcgQYAQoAJgIbAhYhBBWKb1MOogLl9lFhExT66mNEjh35
BQJpRuz3BQkJHCDCAkDBdCAEGQEKAB0WIQRK3hK0WyJ4BicGpcmpsLVXymMjJQUC
Y+0zNQAKCRCpsLVXymMjJbdSD/9+f1FOOeGDAJI6Duo5fsWnf4xJJdtQtDbz6d2A
SeDapxeJ3zWfKBD0wu5sISEa0uiWsYSmLtsa2SqVAKHlEaMGRR+tkBMPQ+rvgI4c
62YjGTgm+IPd+NFIn+ixFU1hpinTh+KhUEoeOwWCvKs9nZfSG9vkienfiG0bBxo2
zrvBzXA50x5hbUL+ghKu/AVfN9qZDwh30O4KZTwk4g4cM9SeaQa4YvHYIS3IEhDZ
hGybwrrqV9cs92ln4IJw9WCy9QReBNrdeFgC4+3ziUp1QsG3RvqrtuMttwBVC1Z3
bj5QjLLOREhhodfvk98t9yVkragObb4rGrLo1mWuF0c4mJGvXwnrqhCMvzv4M+0T
Zdrmw6YpGkGOaOPghVuwoTtqSAkl+zFWIJS89jidvkYG3EqKAkgLKog/TQReCq13
HWrF8cMck+Rf2K8k26q/RNZaA9ZUKjLExzz8lsWmd2C7rvkGLrlxnzxz0gGyNR3Q
KK74vcPhqeABt2GSkHtEXZFFA9IVVzwlRWK3e0S+mVQnZVjNL+cBPn3/hZHMLesB
CucyYZv+DxvT+JkYXBkGSw4s3hpABqGym7gdPUIa0q4rbBFG6xP5sLLBG4yru8vV
2dyCMmFqRuxpT49uNfyQ6Vj+dobN6qHnP/9NwfzOixXYBHXR6LBqb/M+iCiJaaIn
uiRLHwkQFPrqY0SOHfkQpA//W51vj8meuz7snRO+vZFcjLneFFzqfh1Jdz8IqDpO
CkI5pBJmi8e0oSe6r68MkahiQLlYPwm7d+sjHvJhPWipNKWq/uwCgBs+Ac1lpPXR
MwLbrZukcLMYlLmb2MrCKmjcMt0BZsZKBNYL3a3X9nHgwXdeqFYS4WQDMCCc09lz
9YqfdoEsqRO4qN7D0hFqnwjOzb34ixZ6UO8a8ekY9QKxAgWc9fJWGMg6Pjdg4qsK
nqymOIAdGVOJdoRM46wKGVBvbsF2gNfQU4XyzgJo5vHGFwJm6EoSnODlL5e2wsQh
uN1oqBt/8ef/plloMEqVBweUBATqSqjRF6IhhYJvWVuQHQL1p1vnV9FebiVj34ir
Z8ID+o0AnTJcclbUcDwannGJ0cuDcPhk/v/ahVuoMERCi12qnMBo5B/e6Omyh1yB
4pbf4GATGGQipDQG75eC/kP2GQEqJP5WYN0Ar8Le/AA/2xyL7upW0yIByyXCwGEb
JRwEgU3+bPyu58bFt8Pftit6J7rA3oBVVMOPrYH5eZwRaj5m2RptwKGL6BfHnhNv
ZqmCq9EBGX6L1NI0xHMjEFfXJ8jU01XdfG8nCqkwqsHwslXLhqjJphfHcx89YwbV
/15GCuURAv1cKe/7277sOhcvP/QpQqSWgvYExHw8PeFJcTYtF2NrRgNwcQsWS1Rj
gXa5Ag0EY+0zcAEQANC5N6kSfezuucAgi+X3BD+MT37mxQyvICSggEJf1LDSmy0+
bnvD7setL8CP9etTA2fcVNYKI1oboMyhoCnsRP2jDdv1iXOI/hZg4wSb/D1yUkae
fUpxv3Wuci2QKavH2MfraDD7BFMbsQeMcHtn4Rk216T6jndZHnzT1Ih7iX0XeQPb
li5fojOiZssgWAVT4HPXFCJB6lI35Hjp35oRYwrtMmu5INinZ79n9h1igGtt1ItZ
b7rQKNd772Jxcn4UU71ovORSL/xT5i5sxZ+evQOxkpqUAokMOFaoHcOXLmA1NsFv
yryXHK4Ioq9ap2jKlLTWkJWjua9JZ4AmKhbvT8X4ELxIKSCAdJKAWP8ZHbXNu5MD
aznyzZQLxSO7uFvu356De75mI5iohZNj5wB5Wju71pBiorTKVj4+iJ4e+xVIzFdG
hFC0DehNcl2t9w/y8qHwIQ1yUAjXHLXq0/2jsVeH6bU5q/MsgvUP1jcFe0eyOpxy
CDvyFdzZFbI57TnB/fvcZTRZ5ewXMFpH8gzuoFzAjUAP95UjYKgaGdrNPNIy28Ii
4zhvdghei2+n9jgiMfcGQg8lyfH5yF0vWWWynX0KcJsRwEZoL2EauVdwq4PcYOoU
pQFhpcreCjD4LdZ4yRU4InbhcUogXjrQ9Dz01TbPmQD5b5iso21bCEFBXrhzABEB
AAGJAjwEGAEKACYCGwwWIQQVim9TDqIC5fZRYRMU+upjRI4d+QUCaUbs9wUJCRwg
hwAKCRAU+upjRI4d+X/XD/sH5xvHPfTJq52v8weFmB52up+DzqG2lyhGdoUQ1Muw
dRDLTLXLJrFdfpoOo7/j4Scr0rdc7/dpCn0DLcPuCoPxu+SkjEnVehFmZrGSv7Ga
x9dHr3DBh42fdlX/U/EnDuyosY0JU1gNF2/6FIA+bTTOFE3RxfN906RjslYQDjMZ
UAlSeLYHOZofdltI0YIr32vrxgdWQGZXPxU4XusDUc0z163OO+TGg7iUNWFZP5Qj
ubM7e0YbDX0NPIshk8us99YJmrWnhaix1/W5ryO3DXiGaQ7XFi9u7QofRqvRIctg
QXavdepkzJow9V9qpMECAJePIuICq7rm+xy+njjbuF436W7390bfVBwRr+FPADsl
jgQP4KvY5rykss30kheom8wNEbveWkhH5oTfH9b7O4KXJfpfJzrlgOWp2BD9JL8t
/M4HvFXTr2a75H/QbHK5OFrZeGATuv9OTxv7EZvnrPXU+DYTFldpu7TrNNqKCoj3
ZyXmc3Hhg5kskDhfHJppaeOayuhMOpT3ud1MFzROY5SLVIH8rBR12KUgsCUYQcGs
Iy0+0QvEGkjb4cAH1NK3VlbqVNsy1RmqRt2B28R2ueewDfTOoqkzt4MmzLqTdnAx
mTqmHmkEKhEf3K4MRNUPO2yieUg2COk5l6x9HhAnoxxeOZrTmcMsPY/UViG2HEPm
ybkCDQRj7TRDARAA9DZuKdfKq4Bs2+NwxC0aplljWOl8VIsEVg+Q8agD7/HU6/b6
Dry0njtWybn2x6Axf/nUdeOC01Fi1lmht/fpj6mRkgAvd/V6P10xnsUoykPSDSTh
P25MFFGW3JAA82bwdJ4AJpEQvTZG2nTb3237vlBiI1qHQrac8GYkju2O4UfySRN6
7cyi7bMf2pjWBBOEhaNy4b6CMDsb32P/N5J7sTE/TXgrS+u4ITIgjzSrkUkh5Z+B
8QVRa7xPIDZJdvZWTEXWu5fgRPZvxbr154GIkWJkFzlDoB1UcO56/uzRUuKhEV6o
HW3LMUuWdPMjpHpq8hrL0G2rDniJFUtbDFzHdZK1LUU3T2BJM8rjI3D/euph+IDT
27vl5qo72zCYE/iKzx4FMLZcQvx1kUAxkPX8l+dzZEwKeRIIpFDxQvatRtl+z0bM
jbkpDb+Yjv66sC4dYRpgTTGX6rok0PWHR3IxDNzyf2j8zQ4LFJ+rVBM1GjGSt6mG
j9TeL8CVeiSp4SuJ7I/FJVPHsKb50m+BDzeB31qTydNqh2kKr0DVAUa+TUsCr7e0
OYr8WE2adJcRXIW0qw50xXF+W7/05GqSCVD0dpeOUdBTQTsSkQmM3/0hcj9aVo9e
UDCM9RF0WRqiDAoHzJFfg+ztamkQI5HO6CklC4Ok22qrHRf6HDNYSuT6QFkAEQEA
AYkCPQQYAQoAJwMbIAQWIQQVim9TDqIC5fZRYRMU+upjRI4d+QUCaUbs9wUJCRwf
tAAKCRAU+upjRI4d+Y+cD/9yllG6uo934pcHNsVppZBfREFwSc8ywlbosCuSVpay
PjSqgrWwDrnqrsk0F2kUdC6rR3BIcXbn+lA9KqylH+cCXAJCkh8EDq6TlQ7Lt5EV
w1U0MAMXOyxPwDymQ/BO+iDyjXWkRRYgbF5XiFhCfGeuKyhkhACisAgNZ1uA1P5k
0SJYc14YfEhQkB46Y20SpfVHRsQ46FyNB6GHbmTmfoO8La8VTh++7GBdh85HfvkG
VNQ3wpi5oXsOLN9+MJOezc0XsW2LQsKQj1/J7QKzGh+lxN5cemsA5aqPzh8dyxeT
0lYRFp4AHkimqGUomVpRkbegMIPxXqOE+ZAmsddErw0UtmrKxcmMptOJwNgYzEgu
++2vtqerL/NYp+wsdcWaBjCz2F3NiwHgNli7NSB/FPwucZZ5gN5C4SnmeFzrGdHg
Oy+tQUN6ayQKljHeBO7CjMlsFNo/dcVrEMa1ShxBMqlj/6ivoEhktLz0Nru4FwNU
xE5SJYDYfpjD7Ws8y4LoXgWXjFHrMO6N9GzqLN/e8LT7I+w4ps2MrgJ8QSrelmQ3
rjkxp3uWp5v2lqy4rLfpi9iB6zIAeoN2eU1yOM9joxOYMxKYaYeYyP1Mm90wFol8
LcTSaN+tVniPddBiL6zvsGBEMbCR9XN3EQ+mErbuw5ovWBOCrr+dvN3FxvD11y4J
7w==
=mXYP
-----END PGP PUBLIC KEY BLOCK-----
@@ -1,51 +0,0 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQENBFZu2zwBCADfatenjH3cvhlU6AeInvp4R0JmPBG942aghFj1Qh57smRcO5Bv
y9mqrX3UDdmVvu58V3k1k9/GzPnAG1t+c7ohdymv/AMuNY4pE2sfxx7bX+mncTHX
5wthipn8kTNm4WjREjCJM1Bm5sozzEZetED3+0/dWlnHl8b38evnLsD+WbSrDPVp
o6M6Eg9IfMwTfcXzdmLmSnGolBWDQ9i1a0x0r3o+sDW5UTnr7jVP+zILcnOZ1Ewl
Rn9OJ4Qg3ULM7WTMDYpKH4BO7RLR3aJgmsFAHp17vgUnzzFBZ10MCS3UOyUNoyph
xo3belf7Q9nrHcSNbqSeQuBnW/vafAZUreAlABEBAAG0IkVlbGNvIERvbHN0cmEg
PGVkb2xzdHJhQGdtYWlsLmNvbT6JATwEEwEIACYCGyMHCwkIBwMCAQYVCAIJCgsE
FgIDAQIeAQIXgAUCVm7etAIZAQAKCRCBcLRybXGY3q51B/96qt41tmcDSzrj/UTl
O6rErfW5zFvVsJTZ95Duwu87t/DVhw5lKBQcjALqVddufw1nMzyN/tSOMVDW8xe4
wMEdcU4+QAMzNX80enuyinsw1glxfLcK0+VbTvqNIfw0sG3MjPqNs6cK2VRfMHK4
paJjytBVICszNX9TfjLyIpKKoSSo1vqnT47LDZ5GIMy7l9Cs2sO/rqQHSPcR79yz
8m8tbHpDDEMZmJeklckKP2QoiqnHiIvlisDxLclYnUmNaPdaN/f++qZz5Yqvu1n+
sNUBA5eLaZH64Uy2SwtABxO3JPJ8nQ2+SFZ7ocFm4Gcdv4aM+Ura9S6fvM91tEJp
yAQOiQE5BBMBCAAjBQJWbts8AhsjBwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AA
CgkQgXC0cm1xmN6sIAgAielxO8zJREqEkA2xudg/o4e9ZlNZ3X1NvY8OzJH/qlB2
SmwKqwifhtbC1K0uavXA7eaxdtd2zrI+Yq7IooUyv7juMjHTZhLcFbR5iVkQ4Mfp
JmeHXJ/ChYKxD5mMj/C3WbCZ91oCSNZ6Iyi5fvQj/691OC4q+y/2NEUcOI8D8cw8
XKHbKtceFYc+nZmdOv3ZZrNTSN/kszGViNNLKgnpPdDVPtLp+vjXtbmitiFG2HL/
WfbJ+3Gh2Yr1Vy3O9dWKH++e1AmIv7WWqmUjRFVpqC/wr7/BLaScWT8WKF5vkshU
gq8Ez1/cuizsgs3wQIZWgXKQK5njvwnbKg+Zmh/uGbQmRWVsY28gRG9sc3RyYSA8
ZWVsY28uZG9sc3RyYUB0d2VhZy5pbz6JAU4EEwEIADgWIQS1QdVTAScOC88Vyl2B
cLRybXGY3gUCXELt4gIbIwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRCBcLRy
bXGY3ujFCADfS5D1xHU8KH6TpqgssSggYVq62Wwn/Ga+4XPPetM+ajcXagyH6SwB
mxlHICcnv9xC93ryiTI10P1ADJl+aBsI66wEdHBU+ty4RTDy4JZNUPtmRCk9LhSc
mtUO3ry/wtWkRLdJxP49hg7BbQvWoU0M6WODp7SJjPKPWNX64mzHBeOuy+DqGCbM
lpGNCvW8ahU/ewbm7+xwWmzqLDoWzXjHsdF4QdzMVM/vkAgWEP4y0wEqFASzIYaR
GNEkBWU4OQVq5Bdm9+wWWAgsbM0FJAQl0GDqnz4QxWzxxCAAXdbh9F5ffafWYsA9
bise4ZQLkvYo6iUnrcFm4dtZbT8iL3gptCtFZWxjbyBEb2xzdHJhIDxlZWxjby5k
b2xzdHJhQGxvZ2ljYmxveC5jb20+iQE5BBMBCAAjBQJWbt6nAhsjBwsJCAcDAgEG
FQgCCQoLBBYCAwECHgECF4AACgkQgXC0cm1xmN4b/wf8DApMV/jSPEpibekrUPQu
Ye3Z8cxBQuRm/nOPowtPEH/ShAevrCdRiob2nuEZWNoqZ2e5/+6ud07Hs9bslvco
cDv1jeY1dof1idxfKhH3kfSpuD2XJhuzQBxBqOrIlCS/rdnW+Y9wOGD7+bs9QpcA
IyAeQGLLkfggAxaGYQ2Aev8pS7i3a/+lOWbFhcTe02I49KemCOJqBorG5FfILLNr
DjO3EoutNGpuz6rZvc/BlymphWBoAdUmxgoObr7NYWgw9pI8WeE6C7bbSOO7p5aQ
spWXU7Hm17DkzsVDpaJlyClllqK+DdKza5oWlBMe/P02jD3Y+0P/2rCCyQQwmH3D
RbkBDQRWbts8AQgA0g556xc08dH5YNEjbCwEt1j+XoRnV4+GfbSJIXOl9joIgzRC
4IaijvL8+4biWvX7HiybfvBKto0XB1AWLZRC3jWKX5p74I77UAcrD+VQ/roWQqlJ
BKbiQMlRYEsj/5Xnf72G90IP4DAFKvNl+rLChe+jUySA91BCtrYoP75Sw1BE9Cyz
xEtm4WUzKAJdXI+ZTBttA2Nbqy+GSuzBs7fSKDwREJaZmVrosvmns+pQVG4WPWf4
0l4mPguDQmZ9wSWZvBDkpG7AgHYDRYRGkMbAGsVfc6cScN2VsSTa6cbeeAEowKxM
qx9RbY3WOq6aKAm0qDvow1nl7WwXwe8K0wQxfQARAQABiQEfBBgBCAAJBQJWbts8
AhsMAAoJEIFwtHJtcZjeuAAH/0YNz2Qe1IAEO5oqEZNFOccL4KxVPrBhWUen83/b
C6PjOnOqv6q5ztAcms88WIKxBlfzIfq+dzJcbKVS/H7TEXgcaC+7EYW8sJVEsipN
BtEZ3LQNJ5coDjm7WZygniah1lfXNuiritAXduK5FWNNndqGArEaeZ8Shzdo/Uyi
b9lOsBIL6xc2ZcnX5f+rTu02LCEtEb0FwCycZLEWYf8hG4k8uttIOZOC+CLk/k8d
kBmPikMwUVTTV0CdT1cemQKdTaoAaK+kurF6FYXwcnjhRlHrisSt/tVMEwTw4LUM
3MYf6qfjjvE4HlDwZal8th7ccoQp/flfJIuRv85xCcKK+PI=
=u5cX
-----END PGP PUBLIC KEY BLOCK-----
-13
View File
@@ -1,13 +0,0 @@
# Maintainer GPG Keys
Release tags are signed by members of the [Nix maintainer team](https://nixos.org/community/teams/nix/) as part of the [release process](../release-process.md). This directory contains the public GPG keys used for signing.
## Keys
- **Eelco Dolstra**
GPG Fingerprint: `B541 D553 0127 0E0B CF15 CA5D 8170 B472 6D71 98DE`
- **Sergei Zimmerman**
GPG Fingerprint: [`158A 6F53 0EA2 02E5 F651 6113 14FA EA63 448E 1DF9`](https://keys.openpgp.org/vks/v1/by-fingerprint/158A6F530EA202E5F651611314FAEA63448E1DF9)
<!-- TODO: Add keys for other Nix team members -->
+11 -9
View File
@@ -5,11 +5,11 @@
The release process is intended to create the following for each
release:
* A signed Git tag (public keys in `maintainers/keys/`)
* A Git tag
* Binary tarballs in https://releases.nixos.org/?prefix=nix/
* Docker images (arm64 and amd64 variants, uploaded to DockerHub and GHCR)
* Docker images
* Closures in https://cache.nixos.org
@@ -104,17 +104,21 @@ release:
evaluation ID (e.g. `1780832` in
`https://hydra.nixos.org/eval/1780832`).
* Tag the release:
* Tag the release and upload the release artifacts to
[`releases.nixos.org`](https://releases.nixos.org/) and [Docker Hub](https://hub.docker.com/):
```console
$ IS_LATEST=1 ./maintainers/upload-release.pl --skip-docker --skip-s3 --project-root $PWD <EVAL-ID>
$ IS_LATEST=1 ./maintainers/upload-release.pl <EVAL-ID>
```
Note: `IS_LATEST=1` causes the `latest-release` branch to be
force-updated. This is used by the `nixos.org` website to get the
[latest Nix manual](https://nixos.org/manual/nixpkgs/unstable/).
* Trigger the [`upload-release.yml` workflow](https://github.com/NixOS/nix/actions/workflows/upload-release.yml) via `workflow_dispatch` trigger. At the top click `Run workflow` -> select the current release branch from `Use workflow from` -> fill in `Hydra evaluation ID` with `<EVAL-ID>` value from previous steps -> click `Run workflow`. Wait for the run to be approved by `NixOS/nix-team` (or bypass checks if warranted). Wait for the workflow to succeed.
TODO: This script requires the right AWS credentials. Document.
TODO: This script currently requires a
`/home/eelco/Dev/nix-pristine`.
TODO: trigger nixos.org netlify: https://docs.netlify.com/configure-builds/build-hooks/
@@ -177,18 +181,16 @@ release:
* Wait for the desired evaluation of the maintenance jobset to finish
building.
* Tag the release
* Run
```console
$ IS_LATEST=1 ./maintainers/upload-release.pl --skip-docker --skip-s3 --project-root $PWD <EVAL-ID>
$ IS_LATEST=1 ./maintainers/upload-release.pl <EVAL-ID>
```
Omit `IS_LATEST=1` when creating a point release that is not on the
most recent stable branch. This prevents `nixos.org` to going back
to an older release.
* Trigger the [`upload-release.yml` workflow](https://github.com/NixOS/nix/actions/workflows/upload-release.yml) via `workflow_dispatch` trigger. At the top click `Run workflow` -> select the current release branch from `Use workflow from` -> fill in `Hydra evaluation ID` with `<EVAL-ID>` value from previous steps -> click `Run workflow`. Wait for the run to be approved by `NixOS/nix-team` (or bypass checks if warranted). Wait for the workflow to succeed.
* Bump the version number of the release branch as above (e.g. to
`2.12.2`).
+82 -128
View File
@@ -1,8 +1,7 @@
#! /usr/bin/env nix-shell
#! nix-shell -i perl -p awscli2 perl perlPackages.LWPUserAgent perlPackages.LWPProtocolHttps perlPackages.FileSlurp perlPackages.NetAmazonS3 perlPackages.GetoptLongDescriptive gnupg1
#! nix-shell -i perl -p perl perlPackages.LWPUserAgent perlPackages.LWPProtocolHttps perlPackages.FileSlurp perlPackages.NetAmazonS3 gnupg1
use strict;
use Getopt::Long::Descriptive;
use Data::Dumper;
use File::Basename;
use File::Path;
@@ -14,30 +13,7 @@ use Net::Amazon::S3;
delete $ENV{'shell'}; # shut up a LWP::UserAgent.pm warning
my ($opt, $usage) = describe_options(
'%c %o <eval-id>',
[ 'skip-docker', 'Skip Docker image upload' ],
[ 'skip-git', 'Skip Git tagging' ],
[ 'skip-s3', 'Skip S3 upload' ],
[ 'docker-owner=s', 'Docker image owner', { default => 'nixos/nix' } ],
[ 'project-root=s', 'Pristine git repository path' ],
[ 's3-endpoint=s', 'Custom S3 endpoint' ],
[ 's3-host=s', 'S3 host', { default => 's3-eu-west-1.amazonaws.com' } ],
[],
[ 'help|h', 'Show this help message', { shortcircuit => 1 } ],
[],
[ 'Environment variables:' ],
[ 'AWS_ACCESS_KEY_ID' ],
[ 'AWS_SECRET_ACCESS_KEY' ],
[ 'AWS_SESSION_TOKEN For OIDC' ],
[ 'IS_LATEST Set to "1" to mark as latest release' ],
);
print($usage->text), exit if $opt->help;
my $evalId = $ARGV[0] or do { print STDERR $usage->text; exit 1 };
die "--project-root is required unless --skip-git is specified\n" unless $opt->skip_git || $opt->project_root;
my $evalId = $ARGV[0] or die "Usage: $0 EVAL-ID\n";
my $releasesBucketName = "nix-releases";
my $channelsBucketName = "nix-channels";
@@ -86,38 +62,25 @@ File::Path::make_path($narCache);
my $binaryCache = "https://cache.nixos.org/?local-nar-cache=$narCache";
# S3 setup.
my $aws_access_key_id = $ENV{'AWS_ACCESS_KEY_ID'};
my $aws_secret_access_key = $ENV{'AWS_SECRET_ACCESS_KEY'};
my $aws_session_token = $ENV{'AWS_SESSION_TOKEN'};
my $aws_access_key_id = $ENV{'AWS_ACCESS_KEY_ID'} or die "No AWS_ACCESS_KEY_ID given.";
my $aws_secret_access_key = $ENV{'AWS_SECRET_ACCESS_KEY'} or die "No AWS_SECRET_ACCESS_KEY given.";
my ($s3, $releasesBucket, $s3_channels, $channelsBucket);
my $s3 = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
retry => 1,
host => "s3-eu-west-1.amazonaws.com",
});
unless ($opt->skip_s3) {
$aws_access_key_id or die "No AWS_ACCESS_KEY_ID given.";
$aws_secret_access_key or die "No AWS_SECRET_ACCESS_KEY given.";
my $releasesBucket = $s3->bucket($releasesBucketName) or die;
$s3 = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
$aws_session_token ? (aws_session_token => $aws_session_token) : (),
retry => 1,
host => $opt->s3_host,
secure => ($opt->s3_endpoint && $opt->s3_endpoint =~ /^http:/) ? 0 : 1,
});
my $s3_us = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
retry => 1,
});
$releasesBucket = $s3->bucket($releasesBucketName) or die;
$s3_channels = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
$aws_session_token ? (aws_session_token => $aws_session_token) : (),
retry => 1,
$opt->s3_endpoint ? (host => $opt->s3_host) : (),
$opt->s3_endpoint ? (secure => ($opt->s3_endpoint =~ /^http:/) ? 0 : 1) : (),
});
$channelsBucket = $s3_channels->bucket($channelsBucketName) or die;
}
my $channelsBucket = $s3_us->bucket($channelsBucketName) or die;
sub getStorePath {
my ($jobName, $output) = @_;
@@ -152,12 +115,11 @@ sub copyManual {
File::Path::remove_tree("$tmpDir/manual.tmp", {safe => 1});
}
my $awsEndpoint = $opt->s3_endpoint ? "--endpoint-url " . $opt->s3_endpoint : "";
system("aws $awsEndpoint s3 sync '$tmpDir/manual' s3://$releasesBucketName/$releaseDir/manual") == 0
system("aws s3 sync '$tmpDir/manual' s3://$releasesBucketName/$releaseDir/manual") == 0
or die "syncing manual to S3\n";
}
copyManual unless $opt->skip_s3;
copyManual;
sub downloadFile {
my ($jobName, $productNr, $dstName) = @_;
@@ -196,12 +158,30 @@ sub downloadFile {
return $sha256_expected;
}
# Upload docker images.
downloadFile("binaryTarball.i686-linux", "1");
downloadFile("binaryTarball.x86_64-linux", "1");
downloadFile("binaryTarball.aarch64-linux", "1");
downloadFile("binaryTarball.x86_64-darwin", "1");
downloadFile("binaryTarball.aarch64-darwin", "1");
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv6l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv7l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.riscv64-unknown-linux-gnu", "1");
};
warn "$@" if $@;
downloadFile("installerScript", "1");
# Upload docker images to dockerhub.
my $dockerManifest = "";
my $dockerManifestLatest = "";
my $haveDocker = 0;
unless ($opt->skip_docker) {
for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
my $system = $platforms->[0];
my $dockerPlatform = $platforms->[1];
@@ -215,8 +195,8 @@ for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
print STDERR "loading docker image for $dockerPlatform...\n";
system("docker load -i $tmpDir/$fn") == 0 or die;
my $tag = $opt->docker_owner . ":$version-$dockerPlatform";
my $latestTag = $opt->docker_owner . ":latest-$dockerPlatform";
my $tag = "nixos/nix:$version-$dockerPlatform";
my $latestTag = "nixos/nix:latest-$dockerPlatform";
print STDERR "tagging $version docker image for $dockerPlatform...\n";
system("docker tag nix:$version $tag") == 0 or die;
@@ -239,94 +219,68 @@ for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
}
if ($haveDocker) {
my $dockerOwner = $opt->docker_owner;
print STDERR "creating multi-platform docker manifest...\n";
system("docker manifest rm $dockerOwner:$version");
system("docker manifest create $dockerOwner:$version $dockerManifest") == 0 or die;
system("docker manifest rm nixos/nix:$version");
system("docker manifest create nixos/nix:$version $dockerManifest") == 0 or die;
if ($isLatest) {
print STDERR "creating latest multi-platform docker manifest...\n";
system("docker manifest rm $dockerOwner:latest");
system("docker manifest create $dockerOwner:latest $dockerManifestLatest") == 0 or die;
system("docker manifest rm nixos/nix:latest");
system("docker manifest create nixos/nix:latest $dockerManifestLatest") == 0 or die;
}
print STDERR "pushing multi-platform docker manifest...\n";
system("docker manifest push $dockerOwner:$version") == 0 or die;
system("docker manifest push nixos/nix:$version") == 0 or die;
if ($isLatest) {
print STDERR "pushing latest multi-platform docker manifest...\n";
system("docker manifest push $dockerOwner:latest") == 0 or die;
system("docker manifest push nixos/nix:latest") == 0 or die;
}
}
}
# Upload nix-fallback-paths.nix.
write_file("$tmpDir/fallback-paths.nix",
"{\n" .
" x86_64-linux = \"" . getStorePath("build.nix-everything.x86_64-linux") . "\";\n" .
" i686-linux = \"" . getStorePath("build.nix-everything.i686-linux") . "\";\n" .
" aarch64-linux = \"" . getStorePath("build.nix-everything.aarch64-linux") . "\";\n" .
" riscv64-linux = \"" . getStorePath("buildCross.nix-everything.riscv64-unknown-linux-gnu.x86_64-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.nix-everything.x86_64-darwin") . "\";\n" .
" aarch64-darwin = \"" . getStorePath("build.nix-everything.aarch64-darwin") . "\";\n" .
"}\n");
# Upload release files to S3.
unless ($opt->skip_s3) {
downloadFile("binaryTarball.i686-linux", "1");
downloadFile("binaryTarball.x86_64-linux", "1");
downloadFile("binaryTarball.aarch64-linux", "1");
downloadFile("binaryTarball.x86_64-darwin", "1");
downloadFile("binaryTarball.aarch64-darwin", "1");
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv6l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv7l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.riscv64-unknown-linux-gnu", "1");
};
warn "$@" if $@;
downloadFile("installerScript", "1");
for my $fn (glob "$tmpDir/*") {
my $name = basename($fn);
next if $name eq "manual";
my $dstKey = "$releaseDir/" . $name;
unless (defined $releasesBucket->head_key($dstKey)) {
print STDERR "uploading $fn to s3://$releasesBucketName/$dstKey...\n";
# Upload nix-fallback-paths.nix.
write_file("$tmpDir/fallback-paths.nix",
"{\n" .
" x86_64-linux = \"" . getStorePath("build.nix-everything.x86_64-linux") . "\";\n" .
" i686-linux = \"" . getStorePath("build.nix-everything.i686-linux") . "\";\n" .
" aarch64-linux = \"" . getStorePath("build.nix-everything.aarch64-linux") . "\";\n" .
" riscv64-linux = \"" . getStorePath("buildCross.nix-everything.riscv64-unknown-linux-gnu.x86_64-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.nix-everything.x86_64-darwin") . "\";\n" .
" aarch64-darwin = \"" . getStorePath("build.nix-everything.aarch64-darwin") . "\";\n" .
"}\n");
my $configuration = ();
$configuration->{content_type} = "application/octet-stream";
for my $fn (glob "$tmpDir/*") {
my $name = basename($fn);
next if $name eq "manual";
my $dstKey = "$releaseDir/" . $name;
unless (defined $releasesBucket->head_key($dstKey)) {
print STDERR "uploading $fn to s3://$releasesBucketName/$dstKey...\n";
my $configuration = ();
$configuration->{content_type} = "application/octet-stream";
if ($fn =~ /.sha256|install|\.nix$/) {
$configuration->{content_type} = "text/plain";
}
$releasesBucket->add_key_filename($dstKey, $fn, $configuration)
or die $releasesBucket->err . ": " . $releasesBucket->errstr;
if ($fn =~ /.sha256|install|\.nix$/) {
$configuration->{content_type} = "text/plain";
}
}
# Update the "latest" symlink.
$channelsBucket->add_key(
"nix-latest/install", "",
{ "x-amz-website-redirect-location" => "https://releases.nixos.org/$releaseDir/install" })
or die $channelsBucket->err . ": " . $channelsBucket->errstr
if $isLatest;
$releasesBucket->add_key_filename($dstKey, $fn, $configuration)
or die $releasesBucket->err . ": " . $releasesBucket->errstr;
}
}
# Update the "latest" symlink.
$channelsBucket->add_key(
"nix-latest/install", "",
{ "x-amz-website-redirect-location" => "https://releases.nixos.org/$releaseDir/install" })
or die $channelsBucket->err . ": " . $channelsBucket->errstr
if $isLatest;
# Tag the release in Git.
unless ($opt->skip_git) {
chdir($opt->project_root) or die "Cannot chdir to " . $opt->project_root . ": $!";
system("git remote update origin") == 0 or die;
system("git tag --force --sign $version $nixRev -m 'Tagging release $version'") == 0 or die;
system("git push origin refs/tags/$version") == 0 or die;
system("git push --force-with-lease origin $nixRev:refs/heads/latest-release") == 0 or die if $isLatest;
}
chdir("/home/eelco/Dev/nix-pristine") or die;
system("git remote update origin") == 0 or die;
system("git tag --force --sign $version $nixRev -m 'Tagging release $version'") == 0 or die;
system("git push --tags") == 0 or die;
system("git push --force-with-lease origin $nixRev:refs/heads/latest-release") == 0 or die if $isLatest;
File::Path::remove_tree($narCache, {safe => 1});
File::Path::remove_tree($tmpDir, {safe => 1});
+42 -52
View File
@@ -148,15 +148,6 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
isInternal =
dep: internalDrvs ? ${builtins.unsafeDiscardStringContext dep.drvPath or "_non-existent_"};
activeComponentNames = lib.listToAttrs (
map (c: {
name = c.pname or c.name;
value = null;
}) activeComponents
);
isActiveComponent = name: activeComponentNames ? ${name};
in
{
pname = "shell-for-nix";
@@ -199,19 +190,27 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
}
);
small = finalAttrs.finalPackage.withActiveComponents (
c:
lib.intersectAttrs (lib.genAttrs [
"nix-cli"
"nix-util-tests"
"nix-store-tests"
"nix-expr-tests"
"nix-fetchers-tests"
"nix-flake-tests"
"nix-functional-tests"
"nix-perl-bindings"
] (_: null)) c
);
small =
(finalAttrs.finalPackage.withActiveComponents (
c:
lib.intersectAttrs (lib.genAttrs [
"nix-cli"
"nix-util-tests"
"nix-store-tests"
"nix-expr-tests"
"nix-fetchers-tests"
"nix-flake-tests"
"nix-functional-tests"
"nix-perl-bindings"
] (_: null)) c
)).overrideAttrs
(o: {
mesonFlags = o.mesonFlags ++ [
# TODO: infer from activeComponents or vice versa
"-Dkaitai-struct-checks=false"
"-Djson-schema-checks=false"
];
});
};
# Remove the version suffix to avoid unnecessary attempts to substitute in nix develop
@@ -259,10 +258,13 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
# We use this shell with the local checkout, not unpackPhase.
src = null;
# Workaround https://sourceware.org/pipermail/gdb-patches/2025-October/221398.html
# Remove when gdb fix is rolled out everywhere.
separateDebugInfo = false;
mesonBuildType = "debugoptimized";
env = {
# For `make format`, to work without installing pre-commit
_NIX_PRE_COMMIT_HOOKS_CONFIG = "${(pkgs.formats.yaml { }).generate "pre-commit-config.yaml"
@@ -276,33 +278,21 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
dontUseCmakeConfigure = true;
mesonFlags = [
(lib.mesonBool "kaitai-struct-checks" (isActiveComponent "nix-kaitai-struct-checks"))
(lib.mesonBool "json-schema-checks" (isActiveComponent "nix-json-schema-checks"))
]
++ map (transformFlag "libutil") (ignoreCrossFile pkgs.nixComponents2.nix-util.mesonFlags)
++ map (transformFlag "libstore") (ignoreCrossFile pkgs.nixComponents2.nix-store.mesonFlags)
++ map (transformFlag "libfetchers") (ignoreCrossFile pkgs.nixComponents2.nix-fetchers.mesonFlags)
++ lib.optionals havePerl (
map (transformFlag "perl") (ignoreCrossFile pkgs.nixComponents2.nix-perl-bindings.mesonFlags)
)
++ map (transformFlag "libexpr") (ignoreCrossFile pkgs.nixComponents2.nix-expr.mesonFlags)
++ map (transformFlag "libcmd") (ignoreCrossFile pkgs.nixComponents2.nix-cmd.mesonFlags);
mesonFlags =
map (transformFlag "libutil") (ignoreCrossFile pkgs.nixComponents2.nix-util.mesonFlags)
++ map (transformFlag "libstore") (ignoreCrossFile pkgs.nixComponents2.nix-store.mesonFlags)
++ map (transformFlag "libfetchers") (ignoreCrossFile pkgs.nixComponents2.nix-fetchers.mesonFlags)
++ lib.optionals havePerl (
map (transformFlag "perl") (ignoreCrossFile pkgs.nixComponents2.nix-perl-bindings.mesonFlags)
)
++ map (transformFlag "libexpr") (ignoreCrossFile pkgs.nixComponents2.nix-expr.mesonFlags)
++ map (transformFlag "libcmd") (ignoreCrossFile pkgs.nixComponents2.nix-cmd.mesonFlags);
nativeBuildInputs =
let
inputs =
dedupByString (v: "${v}") (
lib.filter (x: !isInternal x) (
lib.lists.concatMap (
# Nix manual has a build-time dependency on nix, but we
# don't want to do a native build just to enter the ross
# dev shell.
#
# TODO: think of a more principled fix for this.
c: lib.filter (f: f.pname or null != "nix") c.nativeBuildInputs
) activeComponents
)
lib.filter (x: !isInternal x) (lib.lists.concatMap (c: c.nativeBuildInputs) activeComponents)
)
++ lib.optional (
!buildCanExecuteHost
@@ -318,8 +308,8 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
pkgs.buildPackages.nixfmt-rfc-style
pkgs.buildPackages.shellcheck
pkgs.buildPackages.include-what-you-use
pkgs.buildPackages.gdb
]
++ lib.optional pkgs.hostPlatform.isUnix pkgs.buildPackages.gdb
++ lib.optional (stdenv.cc.isClang && stdenv.hostPlatform == stdenv.buildPlatform) (
lib.hiPrio pkgs.buildPackages.clang-tools
)
@@ -335,13 +325,13 @@ pkgs.nixComponents2.nix-util.overrideAttrs (
)
);
buildInputs =
# TODO change Nixpkgs to mark gbenchmark as building on Windows
lib.optional pkgs.hostPlatform.isUnix pkgs.gbenchmark
++ dedupByString (v: "${v}") (
lib.filter (x: !isInternal x) (lib.lists.concatMap (c: c.buildInputs) activeComponents)
)
++ lib.optional havePerl pkgs.perl;
buildInputs = [
pkgs.gbenchmark
]
++ dedupByString (v: "${v}") (
lib.filter (x: !isInternal x) (lib.lists.concatMap (c: c.buildInputs) activeComponents)
)
++ lib.optional havePerl pkgs.perl;
propagatedBuildInputs = dedupByString (v: "${v}") (
lib.filter (x: !isInternal x) (lib.lists.concatMap (c: c.propagatedBuildInputs) activeComponents)
+1 -1
View File
@@ -48,7 +48,7 @@ public:
ref<GitRepo> openRepo()
{
return GitRepo::openRepo(tmpDir, true, false);
return GitRepo::openRepo(tmpDir, {.create = true});
}
std::string getRepoName() const
-3
View File
@@ -236,9 +236,6 @@ void Input::checkLocks(Input specified, Input & result)
if (auto prevNarHash = specified.getNarHash())
specified.attrs.insert_or_assign("narHash", prevNarHash->to_string(HashFormat::SRI, true));
if (auto narHash = result.getNarHash())
result.attrs.insert_or_assign("narHash", narHash->to_string(HashFormat::SRI, true));
for (auto & field : specified.attrs) {
auto field2 = result.attrs.find(field.first);
if (field2 != result.attrs.end() && field.second != field2->second)
+16 -13
View File
@@ -203,16 +203,19 @@ static git_packbuilder_progress PACKBUILDER_PROGRESS_CHECK_INTERRUPT = &packBuil
} // extern "C"
static void initRepoAtomically(std::filesystem::path & path, bool bare)
static void initRepoAtomically(std::filesystem::path & path, GitRepo::Options options)
{
if (pathExists(path.string()))
return;
if (!options.create)
throw Error("Git repository %s does not exist.", path);
std::filesystem::path tmpDir = createTempDir(path.parent_path());
AutoDelete delTmpDir(tmpDir, true);
Repository tmpRepo;
if (git_repository_init(Setter(tmpRepo), tmpDir.string().c_str(), bare))
if (git_repository_init(Setter(tmpRepo), tmpDir.string().c_str(), options.bare))
throw Error("creating Git repository %s: %s", path, git_error_last()->message);
try {
std::filesystem::rename(tmpDir, path);
@@ -234,7 +237,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
/** Location of the repository on disk. */
std::filesystem::path path;
bool bare;
Options options;
/**
* libgit2 repository. Note that new objects are not written to disk,
@@ -255,18 +258,18 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
*/
git_odb_backend * packBackend = nullptr;
GitRepoImpl(std::filesystem::path _path, bool create, bool bare, bool packfilesOnly = false)
GitRepoImpl(std::filesystem::path _path, Options _options)
: path(std::move(_path))
, bare(bare)
, options(_options)
{
initLibGit2();
initRepoAtomically(path, bare);
initRepoAtomically(path, options);
if (git_repository_open(Setter(repo), path.string().c_str()))
throw Error("opening Git repository %s: %s", path, git_error_last()->message);
ObjectDb odb;
if (packfilesOnly) {
if (options.packfilesOnly) {
/* Create a fresh object database because by default the repo also
loose object backends. We are not using any of those for the
tarball cache, but libgit2 still does a bunch of unnecessary
@@ -295,7 +298,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
if (git_odb_add_backend(odb.get(), mempackBackend, 999))
throw Error("adding mempack backend to Git object database: %s", git_error_last()->message);
if (packfilesOnly) {
if (options.packfilesOnly) {
if (git_repository_set_odb(repo.get(), odb.get()))
throw Error("setting Git object database: %s", git_error_last()->message);
}
@@ -366,7 +369,7 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
{
// TODO: as an optimization, it would be nice to include `this` in the pool.
return Pool<GitRepoImpl>(std::numeric_limits<size_t>::max(), [this]() -> ref<GitRepoImpl> {
return make_ref<GitRepoImpl>(path, false, bare);
return make_ref<GitRepoImpl>(path, options);
});
}
@@ -712,9 +715,9 @@ struct GitRepoImpl : GitRepo, std::enable_shared_from_this<GitRepoImpl>
}
};
ref<GitRepo> GitRepo::openRepo(const std::filesystem::path & path, bool create, bool bare, bool packfilesOnly)
ref<GitRepo> GitRepo::openRepo(const std::filesystem::path & path, GitRepo::Options options)
{
return make_ref<GitRepoImpl>(path, create, bare, packfilesOnly);
return make_ref<GitRepoImpl>(path, options);
}
/**
@@ -1432,7 +1435,7 @@ ref<GitRepo> Settings::getTarballCache() const
* for optimal packfiles.
*/
static auto repoDir = std::filesystem::path(getCacheDir()) / "tarball-cache-v2";
return GitRepo::openRepo(repoDir, /*create=*/true, /*bare=*/true, /*packfilesOnly=*/true);
return GitRepo::openRepo(repoDir, {.create = true, .bare = true, .packfilesOnly = true});
}
} // namespace fetchers
@@ -1446,7 +1449,7 @@ GitRepo::WorkdirInfo GitRepo::getCachedWorkdirInfo(const std::filesystem::path &
if (i != cache->end())
return i->second;
}
auto workdirInfo = GitRepo::openRepo(path)->getWorkdirInfo();
auto workdirInfo = GitRepo::openRepo(path, {})->getWorkdirInfo();
_cache.lock()->emplace(path, workdirInfo);
return workdirInfo;
}
+8 -13
View File
@@ -637,11 +637,6 @@ struct GitInputScheme : InputScheme
url);
}
// If we don't check here for the path existence, then we can give libgit2 any directory
// and it will initialize them as git directories.
if (!pathExists(path)) {
throw Error("The path '%s' does not exist.", path);
}
repoInfo.location = std::filesystem::absolute(path);
} else {
if (url.scheme == "file")
@@ -703,7 +698,7 @@ struct GitInputScheme : InputScheme
if (auto res = cache->lookup(key))
return getIntAttr(*res, "lastModified");
auto lastModified = GitRepo::openRepo(repoDir)->getLastModified(rev);
auto lastModified = GitRepo::openRepo(repoDir, {})->getLastModified(rev);
cache->upsert(key, {{"lastModified", lastModified}});
@@ -726,7 +721,7 @@ struct GitInputScheme : InputScheme
Activity act(
*logger, lvlChatty, actUnknown, fmt("getting Git revision count of '%s'", repoInfo.locationToArg()));
auto revCount = GitRepo::openRepo(repoDir)->getRevCount(rev);
auto revCount = GitRepo::openRepo(repoDir, {})->getRevCount(rev);
cache->upsert(key, Attrs{{"revCount", revCount}});
@@ -737,7 +732,7 @@ struct GitInputScheme : InputScheme
{
auto head = std::visit(
overloaded{
[&](const std::filesystem::path & path) { return GitRepo::openRepo(path)->getWorkdirRef(); },
[&](const std::filesystem::path & path) { return GitRepo::openRepo(path, {})->getWorkdirRef(); },
[&](const ParsedURL & url) { return readHeadCached(url.to_string(), shallow); }},
repoInfo.location);
if (!head) {
@@ -795,7 +790,7 @@ struct GitInputScheme : InputScheme
if (auto repoPath = repoInfo.getPath()) {
repoDir = *repoPath;
if (!input.getRev())
input.attrs.insert_or_assign("rev", GitRepo::openRepo(repoDir)->resolveRef(ref).gitRev());
input.attrs.insert_or_assign("rev", GitRepo::openRepo(repoDir, {})->resolveRef(ref).gitRev());
} else {
auto repoUrl = std::get<ParsedURL>(repoInfo.location);
std::filesystem::path cacheDir = getCachePath(repoUrl.to_string(), shallow);
@@ -805,7 +800,7 @@ struct GitInputScheme : InputScheme
std::filesystem::create_directories(cacheDir.parent_path());
PathLocks cacheDirLock({cacheDir.string()});
auto repo = GitRepo::openRepo(cacheDir, true, true);
auto repo = GitRepo::openRepo(cacheDir, {.create = true, .bare = true});
// We need to set the origin so resolving submodule URLs works
repo->setRemote("origin", repoUrl.to_string());
@@ -876,7 +871,7 @@ struct GitInputScheme : InputScheme
// the remainder
}
auto repo = GitRepo::openRepo(repoDir);
auto repo = GitRepo::openRepo(repoDir, {});
auto isShallow = repo->isShallow();
@@ -963,7 +958,7 @@ struct GitInputScheme : InputScheme
for (auto & submodule : repoInfo.workdirInfo.submodules)
repoInfo.workdirInfo.files.insert(submodule.path);
auto repo = GitRepo::openRepo(repoPath, false, false);
auto repo = GitRepo::openRepo(repoPath, {});
auto exportIgnore = getExportIgnoreAttr(input);
@@ -1003,7 +998,7 @@ struct GitInputScheme : InputScheme
}
if (!repoInfo.workdirInfo.isDirty) {
auto repo = GitRepo::openRepo(repoPath);
auto repo = GitRepo::openRepo(repoPath, {});
if (auto ref = repo->getWorkdirRef())
input.attrs.insert_or_assign("ref", *ref);
@@ -32,8 +32,14 @@ struct GitRepo
{
virtual ~GitRepo() {}
static ref<GitRepo>
openRepo(const std::filesystem::path & path, bool create = false, bool bare = false, bool packfilesOnly = false);
struct Options
{
bool create = false;
bool bare = false;
bool packfilesOnly = false;
};
static ref<GitRepo> openRepo(const std::filesystem::path & path, Options options);
virtual uint64_t getRevCount(const Hash & rev) = 0;
-1
View File
@@ -128,7 +128,6 @@ static void prim_flakeRefToString(EvalState & state, const PosIdx pos, Value **
state.forceAttrs(*args[0], noPos, "while evaluating the argument passed to builtins.flakeRefToString");
fetchers::Attrs attrs;
for (const auto & attr : *args[0]->attrs()) {
state.forceValue(*attr.value, attr.pos);
auto t = attr.value->type();
if (t == nInt) {
auto intValue = attr.value->integer().value;
@@ -2,7 +2,7 @@
#include "nix/store/derivations.hh"
#include "nix/store/store-api.hh"
#include "nix/util/experimental-features.hh"
#include "nix/util/environment-variables.hh"
#include "nix/util/tests/test-data.hh"
#include "nix/store/store-open.hh"
#include <fstream>
#include <sstream>
@@ -50,11 +50,7 @@ static void BM_UnparseRealDerivationFile(benchmark::State & state, const std::st
}
// Register benchmarks for actual test derivation files if they exist
BENCHMARK_CAPTURE(
BM_ParseRealDerivationFile, hello, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/hello.drv");
BENCHMARK_CAPTURE(
BM_ParseRealDerivationFile, firefox, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/firefox.drv");
BENCHMARK_CAPTURE(
BM_UnparseRealDerivationFile, hello, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/hello.drv");
BENCHMARK_CAPTURE(
BM_UnparseRealDerivationFile, firefox, getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value() + "/derivation/firefox.drv");
BENCHMARK_CAPTURE(BM_ParseRealDerivationFile, hello, (getUnitTestData() / "derivation/hello.drv").string());
BENCHMARK_CAPTURE(BM_ParseRealDerivationFile, firefox, (getUnitTestData() / "derivation/firefox.drv").string());
BENCHMARK_CAPTURE(BM_UnparseRealDerivationFile, hello, (getUnitTestData() / "derivation/hello.drv").string());
BENCHMARK_CAPTURE(BM_UnparseRealDerivationFile, firefox, (getUnitTestData() / "derivation/firefox.drv").string());
+7 -6
View File
@@ -8,6 +8,7 @@
#include "nix/store/tests/nix_api_store.hh"
#include "nix/store/globals.hh"
#include "nix/util/tests/string_callback.hh"
#include "nix/util/tests/test-data.hh"
#include "nix/util/url.hh"
#include "store-tests-config.hh"
@@ -302,7 +303,7 @@ public:
store = open_local_store();
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::filesystem::path unitTestData = nix::getUnitTestData();
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
std::stringstream buffer;
buffer << t.rdbuf();
@@ -357,7 +358,7 @@ TEST_F(nix_api_store_test_base, build_from_json)
auto * store = open_local_store();
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::filesystem::path unitTestData = nix::getUnitTestData();
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
std::stringstream buffer;
@@ -404,7 +405,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_invalid_system)
auto * store = open_local_store();
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::filesystem::path unitTestData = nix::getUnitTestData();
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
std::stringstream buffer;
buffer << t.rdbuf();
@@ -449,7 +450,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_builder_fails)
auto * store = open_local_store();
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::filesystem::path unitTestData = nix::getUnitTestData();
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
std::stringstream buffer;
buffer << t.rdbuf();
@@ -494,7 +495,7 @@ TEST_F(nix_api_store_test_base, nix_store_realise_builder_no_output)
auto * store = open_local_store();
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::filesystem::path unitTestData = nix::getUnitTestData();
std::ifstream t{unitTestData / "derivation/ca/self-contained.json"};
std::stringstream buffer;
buffer << t.rdbuf();
@@ -870,7 +871,7 @@ TEST_F(NixApiStoreTestWithRealisedPath, nix_store_get_fs_closure_error_propagati
*/
static std::string load_json_from_test_data(const char * filename)
{
std::filesystem::path unitTestData{getenv("_NIX_TEST_UNIT_DATA")};
std::filesystem::path unitTestData = nix::getUnitTestData();
std::ifstream t{unitTestData / filename};
std::stringstream buffer;
buffer << t.rdbuf();
+15 -237
View File
@@ -104,33 +104,6 @@ INSTANTIATE_TEST_SUITE_P(
},
},
"with_absolute_endpoint_uri",
},
ParsedS3URLTestCase{
"s3://bucket/key?addressing-style=virtual",
{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
},
"with_addressing_style_virtual",
},
ParsedS3URLTestCase{
"s3://bucket/key?addressing-style=path",
{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Path,
},
"with_addressing_style_path",
},
ParsedS3URLTestCase{
"s3://bucket/key?addressing-style=auto",
{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Auto,
},
"with_addressing_style_auto",
}),
[](const ::testing::TestParamInfo<ParsedS3URLTestCase> & info) { return info.param.description; });
@@ -165,26 +138,6 @@ INSTANTIATE_TEST_SUITE_P(
InvalidS3URLTestCase{"s3://bucket", "error: URI has a missing or invalid key", "missing_key"}),
[](const ::testing::TestParamInfo<InvalidS3URLTestCase> & info) { return info.param.description; });
TEST(ParsedS3URLTest, invalidAddressingStyleThrows)
{
ASSERT_THROW(ParsedS3URL::parse(parseURL("s3://bucket/key?addressing-style=bogus")), InvalidS3AddressingStyle);
}
TEST(ParsedS3URLTest, virtualStyleWithAuthoritylessEndpointThrows)
{
ParsedS3URL input{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint =
ParsedURL{
.scheme = "file",
.path = {"", "some", "path"},
},
};
ASSERT_THROW(input.toHttpsUrl(), nix::Error);
}
// =============================================================================
// S3 URL to HTTPS Conversion Tests
// =============================================================================
@@ -213,7 +166,6 @@ INSTANTIATE_TEST_SUITE_P(
S3ToHttpsConversion,
S3ToHttpsConversionTest,
::testing::Values(
// Default (auto) addressing style: virtual-hosted for standard AWS endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my-bucket",
@@ -221,10 +173,10 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "my-bucket.s3.us-east-1.amazonaws.com"},
.path = {"", "my-key.txt"},
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my-bucket", "my-key.txt"},
},
"https://my-bucket.s3.us-east-1.amazonaws.com/my-key.txt",
"https://s3.us-east-1.amazonaws.com/my-bucket/my-key.txt",
"basic_s3_default_region",
},
S3ToHttpsConversionTestCase{
@@ -235,13 +187,12 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "prod-cache.s3.eu-west-1.amazonaws.com"},
.path = {"", "nix", "store", "abc123.nar.xz"},
.authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
.path = {"", "prod-cache", "nix", "store", "abc123.nar.xz"},
},
"https://prod-cache.s3.eu-west-1.amazonaws.com/nix/store/abc123.nar.xz",
"https://s3.eu-west-1.amazonaws.com/prod-cache/nix/store/abc123.nar.xz",
"with_eu_west_1_region",
},
// Custom endpoint authority: path-style by default
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
@@ -257,7 +208,6 @@ INSTANTIATE_TEST_SUITE_P(
"http://custom.s3.com/bucket/key",
"custom_endpoint_authority",
},
// Custom endpoint URL: path-style by default
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
@@ -286,10 +236,10 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "bucket.s3.ap-southeast-2.amazonaws.com"},
.path = {"", "path", "to", "file.txt"},
.authority = ParsedURL::Authority{.host = "s3.ap-southeast-2.amazonaws.com"},
.path = {"", "bucket", "path", "to", "file.txt"},
},
"https://bucket.s3.ap-southeast-2.amazonaws.com/path/to/file.txt",
"https://s3.ap-southeast-2.amazonaws.com/bucket/path/to/file.txt",
"complex_path_and_region",
},
S3ToHttpsConversionTestCase{
@@ -300,11 +250,11 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "my-bucket.s3.us-east-1.amazonaws.com"},
.path = {"", "my-key.txt"},
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my-bucket", "my-key.txt"},
.query = {{"versionId", "abc123xyz"}},
},
"https://my-bucket.s3.us-east-1.amazonaws.com/my-key.txt?versionId=abc123xyz",
"https://s3.us-east-1.amazonaws.com/my-bucket/my-key.txt?versionId=abc123xyz",
"with_versionId",
},
S3ToHttpsConversionTestCase{
@@ -316,185 +266,13 @@ INSTANTIATE_TEST_SUITE_P(
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "versioned-bucket.s3.eu-west-1.amazonaws.com"},
.path = {"", "path", "to", "object"},
.authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
.path = {"", "versioned-bucket", "path", "to", "object"},
.query = {{"versionId", "version456"}},
},
"https://versioned-bucket.s3.eu-west-1.amazonaws.com/path/to/object?versionId=version456",
"https://s3.eu-west-1.amazonaws.com/versioned-bucket/path/to/object?versionId=version456",
"with_region_and_versionId",
},
// Explicit addressing-style=path forces path-style on standard AWS endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my-bucket",
.key = {"my-key.txt"},
.region = "us-west-2",
.addressingStyle = S3AddressingStyle::Path,
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-west-2.amazonaws.com"},
.path = {"", "my-bucket", "my-key.txt"},
},
"https://s3.us-west-2.amazonaws.com/my-bucket/my-key.txt",
"explicit_path_style",
},
// Explicit addressing-style=virtual forces virtual-hosted-style on custom endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"key"},
.scheme = "http",
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "bucket.custom.s3.com"},
.path = {"", "key"},
},
"http://bucket.custom.s3.com/key",
"explicit_virtual_style_custom_endpoint",
},
// Explicit addressing-style=virtual with full endpoint URL
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint =
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
.path = {""},
},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "bucket.server", .port = 9000},
.path = {"", "key"},
},
"http://bucket.server:9000/key",
"explicit_virtual_style_full_endpoint_url",
},
// Dotted bucket names work normally with explicit path-style
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Path,
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my.bucket", "key"},
},
"https://s3.us-east-1.amazonaws.com/my.bucket/key",
"dotted_bucket_with_path_style",
},
// Dotted bucket names fall back to path-style with auto on standard AWS endpoints
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my.bucket.name",
.key = {"key"},
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = {"", "my.bucket.name", "key"},
},
"https://s3.us-east-1.amazonaws.com/my.bucket.name/key",
"dotted_bucket_with_auto_style_on_aws",
},
// Dotted bucket names work with auto style on custom endpoints (auto = path-style)
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.endpoint = ParsedURL::Authority{.host = "minio.local"},
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "minio.local"},
.path = {"", "my.bucket", "key"},
},
"https://minio.local/my.bucket/key",
"dotted_bucket_with_auto_style_custom_endpoint",
}),
[](const ::testing::TestParamInfo<S3ToHttpsConversionTestCase> & info) { return info.param.description; });
// =============================================================================
// S3 URL to HTTPS Conversion Error Tests
// =============================================================================
struct S3ToHttpsConversionErrorTestCase
{
ParsedS3URL input;
std::string description;
};
class S3ToHttpsConversionErrorTest : public ::testing::WithParamInterface<S3ToHttpsConversionErrorTestCase>,
public ::testing::Test
{};
TEST_P(S3ToHttpsConversionErrorTest, ThrowsOnConversion)
{
auto & [input, description] = GetParam();
ASSERT_THROW(input.toHttpsUrl(), nix::Error);
}
INSTANTIATE_TEST_SUITE_P(
S3ToHttpsConversionErrors,
S3ToHttpsConversionErrorTest,
::testing::Values(
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint = ParsedURL::Authority{.host = ""},
},
"virtual_style_with_empty_host_authority",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
},
"dotted_bucket_with_explicit_virtual_style",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket.name",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
},
"dotted_bucket_with_explicit_virtual_style_multi_dot",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint = ParsedURL::Authority{.host = "minio.local"},
},
"dotted_bucket_with_explicit_virtual_style_custom_authority",
},
S3ToHttpsConversionErrorTestCase{
ParsedS3URL{
.bucket = "my.bucket",
.key = {"key"},
.addressingStyle = S3AddressingStyle::Virtual,
.endpoint =
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "minio.local", .port = 9000},
.path = {""},
},
},
"dotted_bucket_with_explicit_virtual_style_full_endpoint_url",
}),
[](const ::testing::TestParamInfo<S3ToHttpsConversionErrorTestCase> & info) { return info.param.description; });
} // namespace nix
+12 -103
View File
@@ -13,11 +13,6 @@
// C library headers for SSO provider support
# include <aws/auth/credentials.h>
// C library headers for custom logging
# include <aws/common/logging.h>
# include <cstdarg>
# include <boost/unordered/concurrent_flat_map.hpp>
# include <chrono>
@@ -35,101 +30,6 @@ AwsAuthError::AwsAuthError(int errorCode)
namespace {
/**
* Map AWS log level to Nix verbosity.
* AWS levels: AWS_LL_NONE=0, AWS_LL_FATAL=1, AWS_LL_ERROR=2, AWS_LL_WARN=3,
* AWS_LL_INFO=4, AWS_LL_DEBUG=5, AWS_LL_TRACE=6
*
* We map very conservatively because the AWS SDK is extremely noisy. What AWS
* considers "info" includes low-level details like "Initializing epoll" and
* "Starting event-loop thread". What it considers "errors" includes expected
* conditions like missing ~/.aws/config or IMDS being unavailable on non-EC2.
*
* To avoid spamming users, we only show FATAL at default verbosity. Everything
* else requires -vvvvv (lvlDebug) or higher to see.
*/
static Verbosity awsLogLevelToVerbosity(enum aws_log_level level)
{
switch (level) {
case AWS_LL_FATAL:
return lvlError;
case AWS_LL_ERROR:
case AWS_LL_WARN:
case AWS_LL_INFO:
return lvlDebug;
case AWS_LL_DEBUG:
case AWS_LL_TRACE:
return lvlVomit;
// AWS_LL_NONE and AWS_LL_COUNT are enum sentinels, not real log levels
case AWS_LL_NONE:
case AWS_LL_COUNT:
return lvlDebug;
}
unreachable();
}
/**
* Custom AWS logger that routes logs through Nix's logging infrastructure.
*
* The AWS CRT C++ wrapper (ApiHandle::InitializeLogging) only supports FILE*
* or filename-based logging. The underlying C library supports custom loggers
* via aws_logger struct with a vtable containing callback functions.
*/
static int nixAwsLoggerLog(
struct aws_logger * logger, enum aws_log_level logLevel, aws_log_subject_t subject, const char * format, ...)
{
Verbosity nixLevel = awsLogLevelToVerbosity(logLevel);
if (nixLevel > verbosity)
return AWS_OP_SUCCESS; /* Bail out early to avoid formatting the message unnecessarily. */
va_list args;
va_start(args, format);
std::array<char, 4096> buffer{};
auto res = vsnprintf(buffer.data(), buffer.size(), format, args);
va_end(args);
if (res < 0) /* Skip garbage debug messages in case the SDK is busted. */
return AWS_OP_SUCCESS;
const char * subjectName = aws_log_subject_name(subject);
printMsgUsing(nix::logger, nixLevel, "(aws:%s) %s", subjectName ? subjectName : "unknown", chomp(buffer.data()));
return AWS_OP_SUCCESS;
}
/**
* Get current log level for a subject - determines which messages will be logged.
* Must be consistent with awsLogLevelToVerbosity mapping.
*/
static aws_log_level nixAwsLoggerGetLevel(struct aws_logger * logger, aws_log_subject_t subject)
{
// Map Nix verbosity back to AWS log level (inverse of awsLogLevelToVerbosity)
if (verbosity >= lvlVomit)
return AWS_LL_TRACE;
if (verbosity >= lvlDebug)
return AWS_LL_INFO; // error/warn/info are all mapped to lvlDebug
return AWS_LL_FATAL;
}
static void initialiseAwsLogger()
{
static std::once_flag initialised; /* aws_logger_set must only be called once */
std::call_once(initialised, []() {
static aws_logger_vtable nixAwsLoggerVtable = {
.log = nixAwsLoggerLog,
.get_log_level = nixAwsLoggerGetLevel,
.clean_up = [](struct aws_logger *) {}, // No resources to clean up
.set_log_level = nullptr,
};
static aws_logger nixAwsLogger = {
.vtable = &nixAwsLoggerVtable,
.allocator = nullptr,
.p_impl = nullptr,
};
aws_logger_set(&nixAwsLogger);
});
}
/**
* Helper function to wrap a C credentials provider in the C++ interface.
* This replicates the static s_CreateWrappedProvider from aws-crt-cpp.
@@ -219,9 +119,18 @@ class AwsCredentialProviderImpl : public AwsCredentialProvider
public:
AwsCredentialProviderImpl()
{
// Install custom logger that routes AWS CRT logs through Nix's logging infrastructure.
// This ensures AWS logs respect Nix's verbosity settings and are formatted consistently.
initialiseAwsLogger();
// Map Nix's verbosity to AWS CRT log level
Aws::Crt::LogLevel logLevel;
if (verbosity >= lvlVomit) {
logLevel = Aws::Crt::LogLevel::Trace;
} else if (verbosity >= lvlDebug) {
logLevel = Aws::Crt::LogLevel::Debug;
} else if (verbosity >= lvlChatty) {
logLevel = Aws::Crt::LogLevel::Info;
} else {
logLevel = Aws::Crt::LogLevel::Warn;
}
apiHandle.InitializeLogging(logLevel, stderr);
// Create a shared TLS context for SSO (required for HTTPS connections)
auto allocator = Aws::Crt::ApiAllocator();
+54 -44
View File
@@ -71,14 +71,6 @@ void DerivationBuildingGoal::killChild()
#endif
}
void DerivationBuildingGoal::timedOut(Error && ex)
{
killChild();
// We're not inside a coroutine, hence we can't use co_return here.
// Thus we ignore the return value.
[[maybe_unused]] Done _ = doneFailure({BuildResult::Failure::TimedOut, std::move(ex)});
}
std::string showKnownOutputs(const StoreDirConfig & store, const Derivation & drv)
{
std::string msg;
@@ -443,7 +435,20 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
if (useHook) {
buildResult.startTime = time(0); // inexact
started();
co_await Suspend{};
while (true) {
auto event = co_await WaitForChildEvent{};
if (auto * output = std::get_if<ChildOutput>(&event)) {
co_await processChildOutput(output->fd, output->data);
} else if (std::get_if<ChildEOF>(&event)) {
if (!currentLogLine.empty())
flushLine();
break;
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
killChild();
co_return doneFailure(std::move(*timeout));
}
}
#ifndef _WIN32
assert(hook);
@@ -563,7 +568,8 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
{
DerivationBuildingGoal & goal;
DerivationBuildingGoalCallbacks(DerivationBuildingGoal & goal)
DerivationBuildingGoalCallbacks(
DerivationBuildingGoal & goal, std::unique_ptr<DerivationBuilder> & builder)
: goal{goal}
{
}
@@ -631,15 +637,15 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
/* If we have to wait and retry (see below), then `builder` will
already be created, so we don't need to create it again. */
builder =
externalBuilder
? makeExternalDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this),
std::move(params),
*externalBuilder)
: makeDerivationBuilder(
*localStoreP, std::make_unique<DerivationBuildingGoalCallbacks>(*this), std::move(params));
builder = externalBuilder ? makeExternalDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
std::move(params),
*externalBuilder)
: makeDerivationBuilder(
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
std::move(params));
}
if (auto builderOutOpt = builder->startBuild()) {
@@ -663,7 +669,20 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
worker.childStarted(shared_from_this(), {builderOut}, true, true);
started();
co_await Suspend{};
while (true) {
auto event = co_await WaitForChildEvent{};
if (auto * output = std::get_if<ChildOutput>(&event)) {
co_await processChildOutput(output->fd, output->data);
} else if (std::get_if<ChildEOF>(&event)) {
if (!currentLogLine.empty())
flushLine();
break;
} else if (auto * timeout = std::get_if<TimedOut>(&event)) {
killChild();
co_return doneFailure(std::move(*timeout));
}
}
trace("build done");
@@ -996,7 +1015,7 @@ bool DerivationBuildingGoal::isReadDesc(Descriptor fd)
#endif
}
void DerivationBuildingGoal::handleChildOutput(Descriptor fd, std::string_view data)
Goal::Co DerivationBuildingGoal::processChildOutput(Descriptor fd, std::string_view data)
{
// local & `ssh://`-builds are dealt with here.
auto isWrittenToLog = isReadDesc(fd);
@@ -1004,14 +1023,11 @@ void DerivationBuildingGoal::handleChildOutput(Descriptor fd, std::string_view d
logSize += data.size();
if (settings.maxLogSize && logSize > settings.maxLogSize) {
killChild();
// We're not inside a coroutine, hence we can't use co_return here.
// Thus we ignore the return value.
[[maybe_unused]] Done _ = doneFailure(BuildError(
co_return doneFailure(BuildError(
BuildResult::Failure::LogLimitExceeded,
"%s killed after writing more than %d bytes of log output",
getName(),
settings.maxLogSize));
return;
}
for (auto c : data)
@@ -1064,13 +1080,7 @@ void DerivationBuildingGoal::handleChildOutput(Descriptor fd, std::string_view d
currentHookLine += c;
}
#endif
}
void DerivationBuildingGoal::handleEOF(Descriptor fd)
{
if (!currentLogLine.empty())
flushLine();
worker.wakeUp(shared_from_this());
co_return Return{};
}
void DerivationBuildingGoal::flushLine()
@@ -1175,6 +1185,11 @@ DerivationBuildingGoal::checkPathValidity(std::map<std::string, InitialOutput> &
Goal::Done DerivationBuildingGoal::doneSuccess(BuildResult::Success::Status status, SingleDrvOutputs builtOutputs)
{
buildResult.inner = BuildResult::Success{
.status = status,
.builtOutputs = std::move(builtOutputs),
};
mcRunningBuilds.reset();
if (status == BuildResult::Success::Built)
@@ -1182,15 +1197,16 @@ Goal::Done DerivationBuildingGoal::doneSuccess(BuildResult::Success::Status stat
worker.updateProgress();
return Goal::doneSuccess(
BuildResult::Success{
.status = status,
.builtOutputs = std::move(builtOutputs),
});
return amDone(ecSuccess, std::nullopt);
}
Goal::Done DerivationBuildingGoal::doneFailure(BuildError ex)
{
buildResult.inner = BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
};
mcRunningBuilds.reset();
if (ex.status == BuildResult::Failure::TimedOut)
@@ -1202,13 +1218,7 @@ Goal::Done DerivationBuildingGoal::doneFailure(BuildError ex)
worker.updateProgress();
return Goal::doneFailure(
ecFailed,
BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
},
std::move(ex));
return amDone(ecFailed, {std::move(ex)});
}
} // namespace nix
+21 -21
View File
@@ -452,6 +452,20 @@ UnkeyedRealisation DerivationGoal::assertPathValidity()
Goal::Done DerivationGoal::doneSuccess(BuildResult::Success::Status status, UnkeyedRealisation builtOutput)
{
buildResult.inner = BuildResult::Success{
.status = status,
.builtOutputs = {{
wantedOutput,
{
std::move(builtOutput),
DrvOutput{
.drvHash = outputHash,
.outputName = wantedOutput,
},
},
}},
};
mcExpectedBuilds.reset();
if (status == BuildResult::Success::Built)
@@ -459,24 +473,16 @@ Goal::Done DerivationGoal::doneSuccess(BuildResult::Success::Status status, Unke
worker.updateProgress();
return Goal::doneSuccess(
BuildResult::Success{
.status = status,
.builtOutputs = {{
wantedOutput,
{
std::move(builtOutput),
DrvOutput{
.drvHash = outputHash,
.outputName = wantedOutput,
},
},
}},
});
return amDone(ecSuccess, std::nullopt);
}
Goal::Done DerivationGoal::doneFailure(BuildError ex)
{
buildResult.inner = BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
};
mcExpectedBuilds.reset();
if (ex.status == BuildResult::Failure::TimedOut)
@@ -488,13 +494,7 @@ Goal::Done DerivationGoal::doneFailure(BuildError ex)
worker.updateProgress();
return Goal::doneFailure(
ecFailed,
BuildResult::Failure{
.status = ex.status,
.errorMsg = fmt("%s", Uncolored(ex.info().msg)),
},
std::move(ex));
return amDone(ecFailed, {std::move(ex)});
}
} // namespace nix
@@ -98,13 +98,7 @@ Goal::Co DerivationTrampolineGoal::init()
trace("outer load and build derivation");
if (nrFailed != 0) {
co_return doneFailure(
ecFailed,
BuildResult::Failure{
.status = BuildResult::Failure::DependencyFailed,
.errorMsg = fmt("failed to obtain derivation of '%s'", drvReq->to_string(worker.store)),
},
Error("failed to obtain derivation of '%s'", drvReq->to_string(worker.store)));
co_return amDone(ecFailed, Error("cannot build missing derivation '%s'", drvReq->to_string(worker.store)));
}
StorePath drvPath = resolveDerivedPath(worker.store, *drvReq);
@@ -66,7 +66,16 @@ Goal::Co DrvOutputSubstitutionGoal::init()
true,
false);
co_await Suspend{};
while (true) {
auto event = co_await WaitForChildEvent{};
if (std::get_if<ChildOutput>(&event)) {
// Doesn't process child output
} else if (std::get_if<ChildEOF>(&event)) {
break;
} else if (std::get_if<TimedOut>(&event)) {
unreachable();
}
}
worker.childTerminated(this);
@@ -149,9 +158,4 @@ std::string DrvOutputSubstitutionGoal::key()
return "a$" + std::string(id.to_string());
}
void DrvOutputSubstitutionGoal::handleEOF(Descriptor fd)
{
worker.wakeUp(shared_from_this());
}
} // namespace nix
+1 -7
View File
@@ -63,18 +63,12 @@ std::vector<KeyedBuildResult> Store::buildPathsWithResults(
std::vector<KeyedBuildResult> results;
results.reserve(state.size());
for (auto & [req, goalPtr] : state) {
/* Goals that were never started or were cancelled have exitCode
ecBusy and a default buildResult with empty errorMsg. Skip them
to avoid reporting spurious failures with empty messages. */
if (goalPtr->exitCode == Goal::ecBusy)
continue;
for (auto & [req, goalPtr] : state)
results.emplace_back(
KeyedBuildResult{
goalPtr->buildResult,
/* .path = */ req,
});
}
return results;
}
+71 -13
View File
@@ -4,8 +4,58 @@
namespace nix {
TimedOut::TimedOut(time_t maxDuration)
: BuildError(BuildResult::Failure::TimedOut, "timed out after %1% seconds", maxDuration)
, maxDuration(maxDuration)
{
}
using Co = nix::Goal::Co;
using promise_type = nix::Goal::promise_type;
using ChildEvents = decltype(promise_type::childEvents);
void ChildEvents::pushChildEvent(ChildOutput event)
{
if (childTimeout)
return; // Already timed out, ignore
childOutputs.push(std::move(event));
}
void ChildEvents::pushChildEvent(ChildEOF event)
{
if (childTimeout)
return; // Already timed out, ignore
assert(!childEOF);
childEOF = std::move(event);
}
void ChildEvents::pushChildEvent(TimedOut event)
{
// Timeout is immediate - flush pending events
childOutputs = {};
childEOF.reset();
childTimeout = std::move(event);
}
bool ChildEvents::hasChildEvent() const
{
return !childOutputs.empty() || childEOF || childTimeout;
}
Goal::ChildEvent ChildEvents::popChildEvent()
{
if (!childOutputs.empty()) {
auto event = std::move(childOutputs.front());
childOutputs.pop();
return event;
}
if (childEOF)
return *std::exchange(childEOF, std::nullopt);
if (childTimeout)
return *std::exchange(childTimeout, std::nullopt);
unreachable();
}
using handle_type = nix::Goal::handle_type;
using Suspend = nix::Goal::Suspend;
@@ -133,19 +183,6 @@ Co Goal::await(Goals new_waitees)
co_return Return{};
}
Goal::Done Goal::doneSuccess(BuildResult::Success success)
{
buildResult.inner = std::move(success);
return amDone(ecSuccess);
}
Goal::Done Goal::doneFailure(ExitCode result, BuildResult::Failure failure, std::optional<Error> ex)
{
assert(result == ecFailed || result == ecNoSubstituters);
buildResult.inner = std::move(failure);
return amDone(result, std::move(ex));
}
Goal::Done Goal::amDone(ExitCode result, std::optional<Error> ex)
{
trace("done");
@@ -219,6 +256,27 @@ void Goal::work()
assert(top_co || exitCode != ecBusy);
}
void Goal::handleChildOutput(Descriptor fd, std::string_view data)
{
assert(top_co);
top_co->handle.promise().childEvents.pushChildEvent(ChildOutput{fd, std::string{data}});
worker.wakeUp(shared_from_this());
}
void Goal::handleEOF(Descriptor fd)
{
assert(top_co);
top_co->handle.promise().childEvents.pushChildEvent(ChildEOF{fd});
worker.wakeUp(shared_from_this());
}
void Goal::timedOut(TimedOut && ex)
{
assert(top_co);
top_co->handle.promise().childEvents.pushChildEvent(std::move(ex));
worker.wakeUp(shared_from_this());
}
Goal::Co Goal::yield()
{
worker.wakeUp(shared_from_this());
+19 -16
View File
@@ -29,21 +29,20 @@ PathSubstitutionGoal::~PathSubstitutionGoal()
Goal::Done PathSubstitutionGoal::doneSuccess(BuildResult::Success::Status status)
{
return Goal::doneSuccess(
BuildResult::Success{
.status = status,
});
buildResult.inner = BuildResult::Success{
.status = status,
};
return amDone(ecSuccess);
}
Goal::Done PathSubstitutionGoal::doneFailure(ExitCode result, BuildResult::Failure::Status status, std::string errorMsg)
{
debug(errorMsg);
return Goal::doneFailure(
result,
BuildResult::Failure{
.status = status,
.errorMsg = std::move(errorMsg),
});
buildResult.inner = BuildResult::Failure{
.status = status,
.errorMsg = std::move(errorMsg),
};
return amDone(result);
}
Goal::Co PathSubstitutionGoal::init()
@@ -259,7 +258,16 @@ Goal::Co PathSubstitutionGoal::tryToRun(
true,
false);
co_await Suspend{};
while (true) {
auto event = co_await WaitForChildEvent{};
if (std::get_if<ChildOutput>(&event)) {
// Substitution doesn't process child output
} else if (std::get_if<ChildEOF>(&event)) {
break;
} else if (std::get_if<TimedOut>(&event)) {
unreachable(); // Substitution doesn't use timeouts
}
}
trace("substitute finished");
@@ -311,11 +319,6 @@ Goal::Co PathSubstitutionGoal::tryToRun(
co_return doneSuccess(BuildResult::Success::Substituted);
}
void PathSubstitutionGoal::handleEOF(Descriptor fd)
{
worker.wakeUp(shared_from_this());
}
void PathSubstitutionGoal::cleanup()
{
try {
+2 -3
View File
@@ -479,14 +479,13 @@ void Worker::waitForInput()
if (goal->exitCode == Goal::ecBusy && 0 != settings.maxSilentTime && j->respectTimeouts
&& after - j->lastOutput >= std::chrono::seconds(settings.maxSilentTime)) {
goal->timedOut(
Error("%1% timed out after %2% seconds of silence", goal->getName(), settings.maxSilentTime));
goal->timedOut(TimedOut(settings.maxSilentTime));
}
else if (
goal->exitCode == Goal::ecBusy && 0 != settings.buildTimeout && j->respectTimeouts
&& after - j->timeStarted >= std::chrono::seconds(settings.buildTimeout)) {
goal->timedOut(Error("%1% timed out after %2% seconds", goal->getName(), settings.buildTimeout));
goal->timedOut(TimedOut(settings.buildTimeout));
}
}
+1 -4
View File
@@ -852,10 +852,7 @@ static void performOp(
auto path = WorkerProto::Serialise<StorePath>::read(*store, rconn);
std::shared_ptr<const ValidPathInfo> info;
logger->startWork();
try {
info = store->queryPathInfo(path);
} catch (InvalidPath &) {
}
info = store->queryPathInfo(path);
logger->stopWork();
if (info) {
conn.to << 1;
+63 -98
View File
@@ -41,9 +41,27 @@ FileTransferSettings fileTransferSettings;
static GlobalConfig::Register rFileTransferSettings(&fileTransferSettings);
namespace {
using curlSList = std::unique_ptr<::curl_slist, decltype([](::curl_slist * list) { ::curl_slist_free_all(list); })>;
using curlMulti = std::unique_ptr<::CURLM, decltype([](::CURLM * multi) { ::curl_multi_cleanup(multi); })>;
struct curlMultiError : Error
{
::CURLMcode code;
curlMultiError(::CURLMcode code)
: Error{"unexpected curl multi error: %s", ::curl_multi_strerror(code)}
{
assert(code != CURLM_OK);
}
};
} // namespace
struct curlFileTransfer : public FileTransfer
{
CURLM * curlm = 0;
curlMulti curlm;
std::random_device rd;
std::mt19937 mt19937;
@@ -69,7 +87,7 @@ struct curlFileTransfer : public FileTransfer
has been reached. */
std::chrono::steady_clock::time_point embargo;
struct curl_slist * requestHeaders = 0;
curlSList requestHeaders;
std::string encoding;
@@ -92,6 +110,15 @@ struct curlFileTransfer : public FileTransfer
return httpStatus;
}
void appendHeaders(const std::string & header)
{
curlSList tmpSList = curlSList(::curl_slist_append(requestHeaders.get(), requireCString(header)));
if (!tmpSList)
throw std::bad_alloc();
requestHeaders.release();
requestHeaders = std::move(tmpSList);
}
TransferItem(
curlFileTransfer & fileTransfer,
const FileTransferRequest & request,
@@ -131,22 +158,13 @@ struct curlFileTransfer : public FileTransfer
{
result.urls.push_back(request.uri.to_string());
/* Don't set Accept-Encoding for S3 requests that use AWS SigV4 signing.
curl's SigV4 implementation signs all headers including Accept-Encoding,
but some S3-compatible services (like GCS) modify this header in transit,
causing signature verification to fail.
See https://github.com/NixOS/nix/issues/15019 */
#if NIX_WITH_AWS_AUTH
if (!request.awsSigV4Provider)
#endif
requestHeaders =
curl_slist_append(requestHeaders, "Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
appendHeaders("Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
if (!request.expectedETag.empty())
requestHeaders = curl_slist_append(requestHeaders, ("If-None-Match: " + request.expectedETag).c_str());
appendHeaders("If-None-Match: " + request.expectedETag);
if (!request.mimeType.empty())
requestHeaders = curl_slist_append(requestHeaders, ("Content-Type: " + request.mimeType).c_str());
appendHeaders("Content-Type: " + request.mimeType);
for (auto it = request.headers.begin(); it != request.headers.end(); ++it) {
requestHeaders = curl_slist_append(requestHeaders, fmt("%s: %s", it->first, it->second).c_str());
appendHeaders(fmt("%s: %s", it->first, it->second));
}
}
@@ -154,11 +172,9 @@ struct curlFileTransfer : public FileTransfer
{
if (req) {
if (active)
curl_multi_remove_handle(fileTransfer.curlm, req);
curl_multi_remove_handle(fileTransfer.curlm.get(), req);
curl_easy_cleanup(req);
}
if (requestHeaders)
curl_slist_free_all(requestHeaders);
try {
if (!done)
fail(FileTransferError(
@@ -357,7 +373,7 @@ struct curlFileTransfer : public FileTransfer
return ((TransferItem *) userp)->readCallback(buffer, size, nitems);
}
#if !defined(_WIN32) && LIBCURL_VERSION_NUM >= 0x071000
#if !defined(_WIN32)
static int cloexec_callback(void *, curl_socket_t curlfd, curlsocktype purpose)
{
unix::closeOnExec(curlfd);
@@ -420,15 +436,11 @@ struct curlFileTransfer : public FileTransfer
("curl/" LIBCURL_VERSION " Nix/" + nixVersion
+ (fileTransferSettings.userAgentSuffix != "" ? " " + fileTransferSettings.userAgentSuffix.get() : ""))
.c_str());
#if LIBCURL_VERSION_NUM >= 0x072b00
curl_easy_setopt(req, CURLOPT_PIPEWAIT, 1);
#endif
#if LIBCURL_VERSION_NUM >= 0x072f00
if (fileTransferSettings.enableHttp2)
curl_easy_setopt(req, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2TLS);
else
curl_easy_setopt(req, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
#endif
curl_easy_setopt(req, CURLOPT_WRITEFUNCTION, TransferItem::writeCallbackWrapper);
curl_easy_setopt(req, CURLOPT_WRITEDATA, this);
curl_easy_setopt(req, CURLOPT_HEADERFUNCTION, TransferItem::headerCallbackWrapper);
@@ -438,7 +450,7 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_XFERINFODATA, this);
curl_easy_setopt(req, CURLOPT_NOPROGRESS, 0);
curl_easy_setopt(req, CURLOPT_HTTPHEADER, requestHeaders);
curl_easy_setopt(req, CURLOPT_HTTPHEADER, requestHeaders.get());
if (settings.downloadSpeed.get() > 0)
curl_easy_setopt(req, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t) (settings.downloadSpeed.get() * 1024));
@@ -450,11 +462,6 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_CUSTOMREQUEST, "DELETE");
if (request.data) {
// Restart the source to ensure it's at the beginning.
// This is necessary for retries, where the source was
// already consumed by a previous attempt.
request.data->source->restart();
if (request.method == HttpMethod::Post) {
curl_easy_setopt(req, CURLOPT_POST, 1L);
curl_easy_setopt(req, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t) request.data->sizeHint);
@@ -473,18 +480,11 @@ struct curlFileTransfer : public FileTransfer
if (settings.caFile != "")
curl_easy_setopt(req, CURLOPT_CAINFO, settings.caFile.get().c_str());
#if !defined(_WIN32) && LIBCURL_VERSION_NUM >= 0x071000
#if !defined(_WIN32)
curl_easy_setopt(req, CURLOPT_SOCKOPTFUNCTION, cloexec_callback);
#endif
curl_easy_setopt(req, CURLOPT_CONNECTTIMEOUT, fileTransferSettings.connectTimeout.get());
// Enable TCP keep-alive so that idle connections in curl's reuse pool
// are not silently dropped by NATs, firewalls, or load balancers.
curl_easy_setopt(req, CURLOPT_TCP_KEEPALIVE, 1L);
curl_easy_setopt(req, CURLOPT_TCP_KEEPIDLE, 60L);
curl_easy_setopt(req, CURLOPT_TCP_KEEPINTVL, 60L);
curl_easy_setopt(req, CURLOPT_LOW_SPEED_LIMIT, 1L);
curl_easy_setopt(req, CURLOPT_LOW_SPEED_TIME, fileTransferSettings.stalledDownloadTimeout.get());
@@ -714,13 +714,6 @@ struct curlFileTransfer : public FileTransfer
Sync<State> state_;
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
/* We can't use a std::condition_variable to wake up the curl
thread, because it only monitors file descriptors. So use a
pipe instead. */
Pipe wakeupPipe;
#endif
std::thread workerThread;
curlFileTransfer()
@@ -729,43 +722,35 @@ struct curlFileTransfer : public FileTransfer
static std::once_flag globalInit;
std::call_once(globalInit, curl_global_init, CURL_GLOBAL_ALL);
curlm = curl_multi_init();
curlm = curlMulti(curl_multi_init());
#if LIBCURL_VERSION_NUM >= 0x072b00 // Multiplex requires >= 7.43.0
curl_multi_setopt(curlm, CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
#endif
#if LIBCURL_VERSION_NUM >= 0x071e00 // Max connections requires >= 7.30.0
curl_multi_setopt(curlm, CURLMOPT_MAX_TOTAL_CONNECTIONS, fileTransferSettings.httpConnections.get());
#endif
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
wakeupPipe.create();
fcntl(wakeupPipe.readSide.get(), F_SETFL, O_NONBLOCK);
#endif
curl_multi_setopt(curlm.get(), CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
curl_multi_setopt(curlm.get(), CURLMOPT_MAX_TOTAL_CONNECTIONS, fileTransferSettings.httpConnections.get());
workerThread = std::thread([&]() { workerThreadEntry(); });
}
~curlFileTransfer()
{
stopWorkerThread();
try {
stopWorkerThread();
} catch (...) {
ignoreExceptionInDestructor();
}
workerThread.join();
if (curlm)
curl_multi_cleanup(curlm);
}
void stopWorkerThread()
{
/* Signal the worker thread to exit. */
{
auto state(state_.lock());
state->quit();
}
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
writeFull(wakeupPipe.writeSide.get(), " ", false);
#endif
state_.lock()->quit();
wakeupMulti();
}
void wakeupMulti()
{
if (auto ec = ::curl_multi_wakeup(curlm.get()))
throw curlMultiError(ec);
}
void workerThreadMain()
@@ -795,32 +780,25 @@ struct curlFileTransfer : public FileTransfer
/* Let curl do its thing. */
int running;
CURLMcode mc = curl_multi_perform(curlm, &running);
CURLMcode mc = curl_multi_perform(curlm.get(), &running);
if (mc != CURLM_OK)
throw nix::Error("unexpected error from curl_multi_perform(): %s", curl_multi_strerror(mc));
/* Set the promises of any finished requests. */
CURLMsg * msg;
int left;
while ((msg = curl_multi_info_read(curlm, &left))) {
while ((msg = curl_multi_info_read(curlm.get(), &left))) {
if (msg->msg == CURLMSG_DONE) {
auto i = items.find(msg->easy_handle);
assert(i != items.end());
i->second->finish(msg->data.result);
curl_multi_remove_handle(curlm, i->second->req);
curl_multi_remove_handle(curlm.get(), i->second->req);
i->second->active = false;
items.erase(i);
}
}
/* Wait for activity, including wakeup events. */
int numfds = 0;
struct curl_waitfd extraFDs[1];
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
extraFDs[0].fd = wakeupPipe.readSide.get();
extraFDs[0].events = CURL_WAIT_POLLIN;
extraFDs[0].revents = 0;
#endif
long maxSleepTimeMs = items.empty() ? 10000 : 100;
auto sleepTimeMs = nextWakeup != std::chrono::steady_clock::time_point()
? std::max(
@@ -829,23 +807,14 @@ struct curlFileTransfer : public FileTransfer
nextWakeup - std::chrono::steady_clock::now())
.count())
: maxSleepTimeMs;
vomit("download thread waiting for %d ms", sleepTimeMs);
mc = curl_multi_wait(curlm, extraFDs, 1, sleepTimeMs, &numfds);
int numfds = 0;
mc = curl_multi_poll(curlm.get(), nullptr, 0, sleepTimeMs, &numfds);
if (mc != CURLM_OK)
throw nix::Error("unexpected error from curl_multi_wait(): %s", curl_multi_strerror(mc));
throw curlMultiError(mc);
nextWakeup = std::chrono::steady_clock::time_point();
/* Add new curl requests from the incoming requests queue,
except for requests that are embargoed (waiting for a
retry timeout to expire). */
if (extraFDs[0].revents & CURL_WAIT_POLLIN) {
char buf[1024];
auto res = read(extraFDs[0].fd, buf, sizeof(buf));
if (res == -1 && errno != EINTR)
throw SysError("reading curl wakeup socket");
}
std::vector<std::shared_ptr<TransferItem>> incoming;
auto now = std::chrono::steady_clock::now();
@@ -868,7 +837,7 @@ struct curlFileTransfer : public FileTransfer
for (auto & item : incoming) {
debug("starting %s of %s", item->request.noun(), item->request.uri);
item->init();
curl_multi_add_handle(curlm, item->req);
curl_multi_add_handle(curlm.get(), item->req);
item->active = true;
items[item->req] = item;
}
@@ -919,10 +888,8 @@ struct curlFileTransfer : public FileTransfer
throw nix::Error("cannot enqueue download request because the download thread is shutting down");
state->incoming.push(item);
}
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
writeFull(wakeupPipe.writeSide.get(), " ");
#endif
wakeupMulti();
return ItemHandle(static_cast<Item &>(*item));
}
@@ -942,9 +909,7 @@ struct curlFileTransfer : public FileTransfer
{
auto state(state_.lock());
state->unpause.push_back(std::move(item));
#ifndef _WIN32 // TODO need graceful async exit support on Windows?
writeFull(wakeupPipe.writeSide.get(), " ");
#endif
wakeupMulti();
}
void unpauseTransfer(ItemHandle handle) override
@@ -189,22 +189,15 @@ struct ExternalBuilder
std::vector<std::string> args;
};
struct DerivationBuilderDeleter
{
void operator()(DerivationBuilder * builder) noexcept;
};
using DerivationBuilderUnique = std::unique_ptr<DerivationBuilder, DerivationBuilderDeleter>;
#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
DerivationBuilderUnique makeDerivationBuilder(
std::unique_ptr<DerivationBuilder> makeDerivationBuilder(
LocalStore & store, std::unique_ptr<DerivationBuilderCallbacks> miscMethods, DerivationBuilderParams params);
/**
* @param handler Must be chosen such that it supports the given
* derivation.
*/
DerivationBuilderUnique makeExternalDerivationBuilder(
std::unique_ptr<DerivationBuilder> makeExternalDerivationBuilder(
LocalStore & store,
std::unique_ptr<DerivationBuilderCallbacks> miscMethods,
DerivationBuilderParams params,
@@ -5,7 +5,6 @@
#include "nix/store/parsed-derivations.hh"
#include "nix/store/derivation-options.hh"
#include "nix/store/build/derivation-building-misc.hh"
#include "nix/store/build/derivation-builder.hh"
#include "nix/store/outputs-spec.hh"
#include "nix/store/store-api.hh"
#include "nix/store/pathlocks.hh"
@@ -90,7 +89,7 @@ private:
*/
std::unique_ptr<HookInstance> hook;
DerivationBuilderUnique builder;
std::unique_ptr<DerivationBuilder> builder;
#endif
BuildMode buildMode;
@@ -101,8 +100,6 @@ private:
std::map<ActivityId, Activity> builderActivities;
void timedOut(Error && ex) override;
std::string key() override;
/**
@@ -130,10 +127,10 @@ private:
bool isReadDesc(Descriptor fd);
/**
* Callback used by the worker to write to the log.
* Process output from a child process.
*/
void handleChildOutput(Descriptor fd, std::string_view data) override;
void handleEOF(Descriptor fd) override;
Co processChildOutput(Descriptor fd, std::string_view data);
void flushLine();
/**
@@ -52,11 +52,6 @@ struct DerivationGoal : public Goal
bool storeDerivation);
~DerivationGoal() = default;
void timedOut(Error && ex) override
{
unreachable();
};
std::string key() override;
JobCategory jobCategory() const override
@@ -43,8 +43,6 @@ struct DerivationResolutionGoal : public Goal
*/
std::unique_ptr<std::pair<StorePath, BasicDerivation>> resolvedDrv;
void timedOut(Error && ex) override {}
private:
/**
@@ -109,8 +109,6 @@ struct DerivationTrampolineGoal : public Goal
virtual ~DerivationTrampolineGoal();
void timedOut(Error && ex) override {}
std::string key() override;
JobCategory jobCategory() const override
@@ -33,15 +33,8 @@ public:
Co init();
void timedOut(Error && ex) override
{
unreachable();
};
std::string key() override;
void handleEOF(Descriptor fd) override;
JobCategory jobCategory() const override
{
return JobCategory::Substitution;
+125 -36
View File
@@ -5,9 +5,18 @@
#include "nix/store/build-result.hh"
#include <coroutine>
#include <queue>
#include <variant>
namespace nix {
struct TimedOut : BuildError
{
time_t maxDuration;
TimedOut(time_t maxDuration);
};
/**
* Forward definition.
*/
@@ -138,6 +147,29 @@ public:
friend Goal;
};
/**
* Event types for child process communication, delivered via coroutines.
*/
struct ChildOutput
{
Descriptor fd;
std::string data;
};
struct ChildEOF
{
Descriptor fd;
};
using ChildEvent = std::variant<ChildOutput, ChildEOF, TimedOut>;
/**
* Tag type for `co_await`-ing child events.
* Returns a `ChildEvent` when resumed.
*/
struct WaitForChildEvent
{};
// forward declaration of promise_type, see below
struct promise_type;
@@ -276,6 +308,28 @@ public:
*/
bool alive = true;
class
{
/**
* Structured queue of child events:
* - outputs: stream of data from child
* - eof: optional end-of-stream marker
* - timeout: optional timeout that flushes/overrides other events
*/
std::queue<ChildOutput> childOutputs;
std::optional<ChildEOF> childEOF;
std::optional<TimedOut> childTimeout;
public:
void pushChildEvent(ChildOutput event);
void pushChildEvent(ChildEOF event);
void pushChildEvent(TimedOut event);
bool hasChildEvent() const;
ChildEvent popChildEvent();
} childEvents;
/**
* The awaiter used by @ref final_suspend.
*/
@@ -369,13 +423,66 @@ public:
return static_cast<Co &&>(co);
}
/**
* Awaiter for @ref Suspend. Always suspends, but asserts
* there are no pending child events (those should be
* consumed first via @ref WaitForChildEvent).
*/
struct SuspendAwaiter
{
promise_type & promise;
bool await_ready()
{
assert(!promise.childEvents.hasChildEvent());
return false;
}
void await_suspend(handle_type) {}
void await_resume() {}
};
/**
* Allows awaiting a @ref Suspend.
* Always suspends.
*/
std::suspend_always await_transform(Suspend)
SuspendAwaiter await_transform(Suspend)
{
return {};
return SuspendAwaiter{*this};
};
/**
* Awaiter for child events. Suspends and returns the
* pending child event when resumed.
*/
struct ChildEventAwaiter
{
handle_type handle;
bool await_ready()
{
return handle && handle.promise().childEvents.hasChildEvent();
}
void await_suspend(handle_type h)
{
handle = h;
}
ChildEvent await_resume()
{
assert(handle);
return handle.promise().childEvents.popChildEvent();
}
};
/**
* Allows awaiting child events (output, EOF, timeout).
*/
ChildEventAwaiter await_transform(WaitForChildEvent)
{
return ChildEventAwaiter{handle_type::from_promise(*this)};
};
};
@@ -393,28 +500,9 @@ protected:
* Signals that the goal is done.
* `co_return` the result. If you're not inside a coroutine, you can ignore
* the return value safely.
*
* Prefer using `doneSuccess` or `doneFailure` instead, which ensure
* `buildResult` is set correctly.
*/
Done amDone(ExitCode result, std::optional<Error> ex = {});
/**
* Signals successful completion of the goal.
* Sets `buildResult` and calls `amDone`.
*/
Done doneSuccess(BuildResult::Success success);
/**
* Signals failed completion of the goal.
* Sets `buildResult` and calls `amDone`.
*
* @param result The exit code (ecFailed or ecNoSubstituters)
* @param failure The failure details including status and error message
* @param ex Optional exception to store/log
*/
Done doneFailure(ExitCode result, BuildResult::Failure failure, std::optional<Error> ex = {});
public:
virtual void cleanup() {}
@@ -451,15 +539,23 @@ public:
void work();
virtual void handleChildOutput(Descriptor fd, std::string_view data)
{
unreachable();
}
/**
* Called by the worker when data is received from a child process.
* Stores the event and resumes the coroutine.
*/
void handleChildOutput(Descriptor fd, std::string_view data);
virtual void handleEOF(Descriptor fd)
{
unreachable();
}
/**
* Called by the worker when EOF is received from a child process.
* Stores the event and resumes the coroutine.
*/
void handleEOF(Descriptor fd);
/**
* Called by the worker when a build times out.
* Stores the event and resumes the coroutine.
*/
void timedOut(TimedOut && ex);
void trace(std::string_view s);
@@ -468,13 +564,6 @@ public:
return name;
}
/**
* Callback in case of a timeout. It should wake up its waiters,
* get rid of any running child processes that are being monitored
* by the worker (important!), etc.
*/
virtual void timedOut(Error && ex) = 0;
/**
* Used for comparisons. The order matters a bit for scheduling. We
* want:
@@ -53,11 +53,6 @@ public:
std::optional<ContentAddress> ca = std::nullopt);
~PathSubstitutionGoal();
void timedOut(Error && ex) override
{
unreachable();
};
std::string key() override
{
return "a$" + std::string(storePath.name()) + "$" + worker.store.printStorePath(storePath);
@@ -72,12 +67,6 @@ public:
StorePath subPath, nix::ref<Store> sub, std::shared_ptr<const ValidPathInfo> info, bool & substituterFailed);
Co finished();
/**
* Callback used by the worker to write to the log.
*/
void handleChildOutput(Descriptor fd, std::string_view data) override {};
void handleEOF(Descriptor fd) override;
/* Called by destructor, can't be overridden */
void cleanup() override final;
@@ -3,7 +3,6 @@
#include "nix/store/config.hh"
#include "nix/store/http-binary-cache-store.hh"
#include "nix/store/s3-url.hh"
namespace nix {
@@ -53,22 +52,13 @@ struct S3BinaryCacheStoreConfig : HttpBinaryCacheStoreConfig
"endpoint",
R"(
The S3 endpoint to use. When empty (default), uses AWS S3 with
region-specific endpoints. For S3-compatible services such as
MinIO, set this to your service's endpoint.
)"};
region-specific endpoints (e.g., s3.us-east-1.amazonaws.com).
For S3-compatible services such as MinIO, set this to your service's endpoint.
Setting<S3AddressingStyle> addressingStyle{
this,
S3AddressingStyle::Auto,
"addressing-style",
R"(
The S3 addressing style to use. `auto` (default) uses
virtual-hosted-style for standard AWS endpoints and path-style
for custom endpoints; bucket names containing dots automatically
fall back to path-style to avoid TLS certificate errors. `path`
forces path-style addressing (deprecated by AWS). `virtual`
forces virtual-hosted-style addressing (bucket names must not
contain dots).
> **Note**
>
> Custom endpoints must support HTTPS and use path-based
> addressing instead of virtual host based addressing.
)"};
const Setting<bool> multipartUpload{
@@ -127,7 +117,7 @@ struct S3BinaryCacheStoreConfig : HttpBinaryCacheStoreConfig
* Set of settings that are part of the S3 URI itself.
* These are needed for region specification and other S3-specific settings.
*/
const std::set<const AbstractSetting *> s3UriSettings = {&profile, &region, &scheme, &endpoint, &addressingStyle};
const std::set<const AbstractSetting *> s3UriSettings = {&profile, &region, &scheme, &endpoint};
static const std::string name()
{
+1 -28
View File
@@ -1,41 +1,16 @@
#pragma once
///@file
#include "nix/store/config.hh"
#include "nix/util/error.hh"
#include "nix/util/url.hh"
#include "nix/util/util.hh"
#include <optional>
#include <string>
#include <string_view>
#include <variant>
#include <vector>
namespace nix {
/**
* S3 addressing style for bucket access.
* - Auto: virtual-hosted-style for standard AWS endpoints, path-style for custom endpoints.
* - Path: always use path-style (bucket in URL path).
* - Virtual: always use virtual-hosted-style (bucket as hostname prefix; bucket name must not contain dots).
*/
enum class S3AddressingStyle {
Auto,
Path,
Virtual,
};
MakeError(InvalidS3AddressingStyle, Error);
S3AddressingStyle parseS3AddressingStyle(std::string_view style);
std::string_view showS3AddressingStyle(S3AddressingStyle style);
template<>
S3AddressingStyle BaseSetting<S3AddressingStyle>::parse(const std::string & str) const;
template<>
std::string BaseSetting<S3AddressingStyle>::to_string() const;
/**
* Parsed S3 URL.
*/
@@ -52,7 +27,6 @@ struct ParsedS3URL
std::optional<std::string> region;
std::optional<std::string> scheme;
std::optional<std::string> versionId;
std::optional<S3AddressingStyle> addressingStyle;
/**
* The endpoint can be either missing, be an absolute URI (with a scheme like `http:`)
* or an authority (so an IP address or a registered name).
@@ -72,8 +46,7 @@ struct ParsedS3URL
static ParsedS3URL parse(const ParsedURL & uri);
/**
* Convert this ParsedS3URL to an HTTP(S) ParsedURL for use with curl's AWS SigV4 authentication.
* The scheme defaults to HTTPS but respects the 'scheme' setting and custom endpoint schemes.
* Convert this ParsedS3URL to HTTPS ParsedURL for use with curl's AWS SigV4 authentication
*/
ParsedURL toHttpsUrl() const;
+26 -22
View File
@@ -39,28 +39,32 @@ deps_public_maybe_subproject = [
]
subdir('nix-meson-build-support/subprojects')
run_command(
'ln',
'-s',
meson.project_build_root() / '__nothing_link_target',
meson.project_build_root() / '__nothing_symlink',
# native doesn't allow dangling symlinks, which the tests require
env : {'MSYS' : 'winsymlinks:lnk'},
check : true,
)
can_link_symlink = run_command(
'ln',
meson.project_build_root() / '__nothing_symlink',
meson.project_build_root() / '__nothing_hardlink',
check : false,
).returncode() == 0
run_command(
'rm',
'-f',
meson.project_build_root() / '__nothing_symlink',
meson.project_build_root() / '__nothing_hardlink',
check : true,
)
can_link_symlink = false
native_ln = find_program('ln', required : false, native : true)
if native_ln.found()
run_command(
native_ln,
'-s',
meson.project_build_root() / '__nothing_link_target',
meson.project_build_root() / '__nothing_symlink',
# native doesn't allow dangling symlinks, which the tests require
env : {'MSYS' : 'winsymlinks:lnk'},
check : true,
)
can_link_symlink = run_command(
native_ln,
meson.project_build_root() / '__nothing_symlink',
meson.project_build_root() / '__nothing_hardlink',
check : false,
).returncode() == 0
run_command(
'rm',
'-f',
meson.project_build_root() / '__nothing_symlink',
meson.project_build_root() / '__nothing_hardlink',
check : true,
)
endif
summary('can hardlink to symlink', can_link_symlink, bool_yn : true)
configdata_priv.set('CAN_LINK_SYMLINK', can_link_symlink.to_int())
+9 -137
View File
@@ -1,14 +1,8 @@
#include "nix/store/s3-url.hh"
#include "nix/util/abstract-setting-to-json.hh"
#include "nix/util/config-impl.hh"
#include "nix/util/error.hh"
#include "nix/util/logging.hh"
#include "nix/util/json-impls.hh"
#include "nix/util/split.hh"
#include "nix/util/strings-inline.hh"
#include <atomic>
#include <nlohmann/json.hpp>
#include <ranges>
#include <string_view>
@@ -16,30 +10,6 @@ using namespace std::string_view_literals;
namespace nix {
S3AddressingStyle parseS3AddressingStyle(std::string_view style)
{
if (style == "auto")
return S3AddressingStyle::Auto;
if (style == "path")
return S3AddressingStyle::Path;
if (style == "virtual")
return S3AddressingStyle::Virtual;
throw InvalidS3AddressingStyle("unknown S3 addressing style '%s', expected 'auto', 'path', or 'virtual'", style);
}
std::string_view showS3AddressingStyle(S3AddressingStyle style)
{
switch (style) {
case S3AddressingStyle::Auto:
return "auto";
case S3AddressingStyle::Path:
return "path";
case S3AddressingStyle::Virtual:
return "virtual";
}
unreachable();
}
ParsedS3URL ParsedS3URL::parse(const ParsedURL & parsed)
try {
if (parsed.scheme != "s3"sv)
@@ -79,9 +49,6 @@ try {
.region = getOptionalParam("region"),
.scheme = getOptionalParam("scheme"),
.versionId = getOptionalParam("versionId"),
.addressingStyle = getOptionalParam("addressing-style").transform([](const std::string & s) {
return parseS3AddressingStyle(s);
}),
.endpoint = [&]() -> decltype(ParsedS3URL::endpoint) {
if (!endpoint)
return std::monostate();
@@ -98,9 +65,6 @@ try {
} catch (BadURL & e) {
e.addTrace({}, "while parsing S3 URI: '%s'", parsed.to_string());
throw;
} catch (InvalidS3AddressingStyle & e) {
e.addTrace({}, "while parsing S3 URI: '%s'", parsed.to_string());
throw;
}
ParsedURL ParsedS3URL::toHttpsUrl() const
@@ -116,95 +80,41 @@ ParsedURL ParsedS3URL::toHttpsUrl() const
queryParams["versionId"] = *versionId;
}
auto style = addressingStyle.value_or(S3AddressingStyle::Auto);
// Virtual-hosted-style prepends the bucket name to the hostname, so bucket
// names containing dots produce multi-level subdomains (e.g.
// my.bucket.s3.amazonaws.com) that break TLS wildcard certificate validation.
// In auto mode, fall back to path-style; only error on explicit virtual.
auto hasDottedBucket = bucket.find('.') != std::string::npos;
auto useVirtualForEndpoint = [&](bool defaultVirtual) {
auto useVirtual = defaultVirtual ? style != S3AddressingStyle::Path : style == S3AddressingStyle::Virtual;
if (useVirtual && hasDottedBucket) {
if (style == S3AddressingStyle::Virtual)
throw Error(
"bucket name '%s' contains a dot, which is incompatible with "
"virtual-hosted-style addressing (causes TLS certificate errors); "
"use 'addressing-style=path' or 'addressing-style=auto' instead",
bucket);
static std::atomic<bool> warnedDottedBucket{false};
warnOnce(
warnedDottedBucket,
"bucket name '%s' contains a dot; falling back to path-style addressing "
"(virtual-hosted-style requires non-dotted bucket names for TLS certificate validity); "
"set 'addressing-style=path' to silence this warning",
bucket);
return false;
}
return useVirtual;
};
// Handle endpoint configuration using std::visit
return std::visit(
overloaded{
[&](const std::monostate &) {
// No custom endpoint: use virtual-hosted-style by default (auto),
// path-style when explicitly requested or for dotted bucket names.
auto useVirtual = useVirtualForEndpoint(/* defaultVirtual = */ true);
// No custom endpoint, use standard AWS S3 endpoint
std::vector<std::string> path{""};
if (!useVirtual)
path.push_back(bucket);
path.push_back(bucket);
path.insert(path.end(), key.begin(), key.end());
return ParsedURL{
.scheme = std::string{schemeStr},
.authority =
ParsedURL::Authority{
.host = useVirtual ? bucket + ".s3." + regionStr + ".amazonaws.com"
: "s3." + regionStr + ".amazonaws.com"},
.authority = ParsedURL::Authority{.host = "s3." + regionStr + ".amazonaws.com"},
.path = std::move(path),
.query = std::move(queryParams),
};
},
[&](const ParsedURL::Authority & auth) {
// Custom endpoint authority: use path-style by default (auto),
// virtual-hosted-style only when explicitly requested (not for dotted buckets).
auto useVirtual = useVirtualForEndpoint(/* defaultVirtual = */ false);
if (useVirtual && auth.host.empty())
throw Error(
"cannot use virtual-hosted-style addressing with endpoint '%s' "
"because it has no hostname; use 'addressing-style=path' instead",
auth.to_string());
// Endpoint is just an authority (hostname/port)
std::vector<std::string> path{""};
if (!useVirtual)
path.push_back(bucket);
path.push_back(bucket);
path.insert(path.end(), key.begin(), key.end());
return ParsedURL{
.scheme = std::string{schemeStr},
.authority =
useVirtual ? ParsedURL::Authority{.host = bucket + "." + auth.host, .port = auth.port} : auth,
.authority = auth,
.path = std::move(path),
.query = std::move(queryParams),
};
},
[&](const ParsedURL & endpointUrl) {
// Full endpoint URL: use path-style by default (auto),
// virtual-hosted-style only when explicitly requested (not for dotted buckets).
auto useVirtual = useVirtualForEndpoint(/* defaultVirtual = */ false);
if (useVirtual && (!endpointUrl.authority || endpointUrl.authority->host.empty()))
throw Error(
"cannot use virtual-hosted-style addressing with endpoint '%s' "
"because it has no authority (hostname)",
endpointUrl.to_string());
// Endpoint is already a ParsedURL (e.g., http://server:9000)
auto path = endpointUrl.path;
if (!useVirtual)
path.push_back(bucket);
path.push_back(bucket);
path.insert(path.end(), key.begin(), key.end());
return ParsedURL{
.scheme = endpointUrl.scheme,
.authority = useVirtual ? std::optional{ParsedURL::Authority{
.host = bucket + "." + endpointUrl.authority->host,
.port = endpointUrl.authority->port}}
: endpointUrl.authority,
.authority = endpointUrl.authority,
.path = std::move(path),
.query = std::move(queryParams),
};
@@ -213,42 +123,4 @@ ParsedURL ParsedS3URL::toHttpsUrl() const
endpoint);
}
void to_json(nlohmann::json & j, const S3AddressingStyle & e)
{
j = std::string{showS3AddressingStyle(e)};
}
void from_json(const nlohmann::json & j, S3AddressingStyle & e)
{
e = parseS3AddressingStyle(j.get<std::string>());
}
template<>
struct json_avoids_null<S3AddressingStyle> : std::true_type
{};
template<>
S3AddressingStyle BaseSetting<S3AddressingStyle>::parse(const std::string & str) const
{
try {
return parseS3AddressingStyle(str);
} catch (InvalidS3AddressingStyle &) {
throw UsageError("option '%s' has invalid value '%s', expected 'auto', 'path', or 'virtual'", name, str);
}
}
template<>
std::string BaseSetting<S3AddressingStyle>::to_string() const
{
return std::string{showS3AddressingStyle(value)};
}
template<>
struct BaseSetting<S3AddressingStyle>::trait
{
static constexpr bool appendable = false;
};
template class BaseSetting<S3AddressingStyle>;
} // namespace nix
-2
View File
@@ -454,8 +454,6 @@ void Store::querySubstitutablePathInfos(const StorePathCAMap & paths, Substituta
.downloadSize = narInfo ? narInfo->fileSize : 0,
.narSize = info->narSize,
});
break; /* We are done. */
} catch (InvalidPath &) {
} catch (SubstituterDisabled &) {
} catch (Error & e) {
+13 -30
View File
@@ -98,13 +98,10 @@ public:
{
}
/**
* Cleanup code to run when destroying any DerivationBuilderImpl implementation.
*/
void cleanupOnDestruction() noexcept
~DerivationBuilderImpl()
{
/* Careful: we should never ever throw an exception from a
noexcept function. */
destructor. */
try {
killChild();
} catch (...) {
@@ -681,17 +678,17 @@ static void handleChildException(bool sendException)
}
}
static void checkNotWorldWritable(std::filesystem::path path)
static bool checkNotWorldWritable(std::filesystem::path path)
{
while (true) {
auto st = lstat(path);
if (st.st_mode & S_IWOTH)
throw Error("Path %s is world-writable or a symlink. That's not allowed for security.", path);
return false;
if (path == path.parent_path())
break;
path = path.parent_path();
}
return;
return true;
}
std::optional<Descriptor> DerivationBuilderImpl::startBuild()
@@ -713,8 +710,9 @@ std::optional<Descriptor> DerivationBuilderImpl::startBuild()
createDirs(buildDir);
if (buildUser)
checkNotWorldWritable(buildDir);
if (buildUser && !checkNotWorldWritable(buildDir))
throw Error(
"Path %s or a parent directory is world-writable or a symlink. That's not allowed for security.", buildDir);
/* Create a temporary directory where the build will take
place. */
@@ -1965,20 +1963,7 @@ StorePath DerivationBuilderImpl::makeFallbackPath(const StorePath & path)
namespace nix {
void DerivationBuilderDeleter::operator()(DerivationBuilder * builder) noexcept
{
if (!builder) /* Idempotent and handles nullptr as any deleter must. */
return;
if (auto builderImpl = dynamic_cast<DerivationBuilderImpl *>(builder))
/* Note that this might call into virtual functions, which we can't do in a destructor of
the DerivationBuilderImpl itself. */
builderImpl->cleanupOnDestruction();
delete builder;
}
std::unique_ptr<DerivationBuilder, DerivationBuilderDeleter> makeDerivationBuilder(
std::unique_ptr<DerivationBuilder> makeDerivationBuilder(
LocalStore & store, std::unique_ptr<DerivationBuilderCallbacks> miscMethods, DerivationBuilderParams params)
{
bool useSandbox = false;
@@ -2029,19 +2014,17 @@ std::unique_ptr<DerivationBuilder, DerivationBuilderDeleter> makeDerivationBuild
throw Error("feature 'uid-range' is only supported in sandboxed builds");
#ifdef __APPLE__
return DerivationBuilderUnique(
new DarwinDerivationBuilder(store, std::move(miscMethods), std::move(params), useSandbox));
return std::make_unique<DarwinDerivationBuilder>(store, std::move(miscMethods), std::move(params), useSandbox);
#elif defined(__linux__)
if (useSandbox)
return DerivationBuilderUnique(
new ChrootLinuxDerivationBuilder(store, std::move(miscMethods), std::move(params)));
return std::make_unique<ChrootLinuxDerivationBuilder>(store, std::move(miscMethods), std::move(params));
return DerivationBuilderUnique(new LinuxDerivationBuilder(store, std::move(miscMethods), std::move(params)));
return std::make_unique<LinuxDerivationBuilder>(store, std::move(miscMethods), std::move(params));
#else
if (useSandbox)
throw Error("sandboxing builds is not supported on this platform");
return DerivationBuilderUnique(new DerivationBuilderImpl(store, std::move(miscMethods), std::move(params)));
return std::make_unique<DerivationBuilderImpl>(store, std::move(miscMethods), std::move(params));
#endif
}
@@ -106,14 +106,13 @@ struct ExternalDerivationBuilder : DerivationBuilderImpl
}
};
DerivationBuilderUnique makeExternalDerivationBuilder(
std::unique_ptr<DerivationBuilder> makeExternalDerivationBuilder(
LocalStore & store,
std::unique_ptr<DerivationBuilderCallbacks> miscMethods,
DerivationBuilderParams params,
const ExternalBuilder & handler)
{
return DerivationBuilderUnique(
new ExternalDerivationBuilder(store, std::move(miscMethods), std::move(params), handler));
return std::make_unique<ExternalDerivationBuilder>(store, std::move(miscMethods), std::move(params), handler);
}
} // namespace nix
@@ -16,7 +16,6 @@ R""(
; Allow DNS lookups.
(allow network-outbound (remote unix-socket (path-literal "/private/var/run/mDNSResponder")))
(allow mach-lookup (global-name "com.apple.SystemConfiguration.DNSConfiguration"))
; Allow access to trustd.
(allow mach-lookup (global-name "com.apple.trustd"))
@@ -4,20 +4,11 @@
#include <gtest/gtest.h>
#include "nix/util/types.hh"
#include "nix/util/environment-variables.hh"
#include "nix/util/file-system.hh"
#include "nix/util/tests/test-data.hh"
namespace nix {
/**
* The path to the unit test data directory. See the contributing guide
* in the manual for further details.
*/
static inline std::filesystem::path getUnitTestData()
{
return getEnv("_NIX_TEST_UNIT_DATA").value();
}
/**
* Whether we should update "golden masters" instead of running tests
* against them. See the contributing guide in the manual for further
@@ -10,4 +10,5 @@ headers = files(
'json-characterization.hh',
'nix_api_util.hh',
'string_callback.hh',
'test-data.hh',
)
@@ -0,0 +1,24 @@
#pragma once
///@file
#include <filesystem>
#include "nix/util/environment-variables.hh"
#include "nix/util/error.hh"
namespace nix {
/**
* The path to the unit test data directory. See the contributing guide
* in the manual for further details.
*/
static inline std::filesystem::path getUnitTestData()
{
auto data = getEnv("_NIX_TEST_UNIT_DATA");
if (!data)
throw Error(
"_NIX_TEST_UNIT_DATA environment variable is not set. "
"Recommendation: use meson, example: 'meson test -C build --gdb'");
return std::filesystem::path(*data);
}
} // namespace nix
-28
View File
@@ -15,32 +15,4 @@ TEST(alignUp, notAPowerOf2)
ASSERT_DEATH({ alignUp(1u, 42); }, "alignment must be a power of 2");
}
template<typename T>
class alignUpOverflowTest : public ::testing::Test
{};
using UnsignedTypes = ::testing::Types<uint8_t, uint16_t, uint32_t, uint64_t>;
TYPED_TEST_SUITE(alignUpOverflowTest, UnsignedTypes);
TYPED_TEST(alignUpOverflowTest, lastSafeValue)
{
constexpr auto max = std::numeric_limits<TypeParam>::max();
ASSERT_EQ(alignUp<TypeParam>(max - 15, 16), (max - 15) & ~TypeParam{15});
ASSERT_NO_THROW(alignUp<TypeParam>(max - 15, 16));
}
TYPED_TEST(alignUpOverflowTest, overflowThrows)
{
constexpr auto max = std::numeric_limits<TypeParam>::max();
ASSERT_THROW(alignUp<TypeParam>(max - 14, 16), Error);
ASSERT_THROW(alignUp<TypeParam>(max, 16), Error);
ASSERT_THROW(alignUp<TypeParam>(max, 2), Error);
}
TYPED_TEST(alignUpOverflowTest, alignmentOneNeverOverflows)
{
constexpr auto max = std::numeric_limits<TypeParam>::max();
ASSERT_EQ(alignUp<TypeParam>(max, 1), max);
}
} // namespace nix
+4
View File
@@ -33,6 +33,9 @@ deps_private += rapidcheck
gtest = dependency('gtest', main : true)
deps_private += gtest
gmock = dependency('gmock')
deps_private += gmock
configdata = configuration_data()
configdata.set_quoted('PACKAGE_VERSION', meson.project_version())
@@ -72,6 +75,7 @@ sources = files(
'position.cc',
'processes.cc',
'sort.cc',
'source-accessor.cc',
'spawn.cc',
'strings.cc',
'suggestions.cc',
+138
View File
@@ -0,0 +1,138 @@
#include "nix/util/fs-sink.hh"
#include "nix/util/file-system.hh"
#include "nix/util/processes.hh"
#include <gtest/gtest.h>
#include <gmock/gmock.h>
#include <rapidcheck/gtest.h>
namespace nix {
MATCHER_P2(HasContents, path, expected, "")
{
auto stat = arg->maybeLstat(path);
if (!stat) {
*result_listener << arg->showPath(path) << " does not exist";
return false;
}
if (stat->type != SourceAccessor::tRegular) {
*result_listener << arg->showPath(path) << " is not a regular file";
return false;
}
auto actual = arg->readFile(path);
if (actual != expected) {
*result_listener << arg->showPath(path) << " has contents " << ::testing::PrintToString(actual);
return false;
}
return true;
}
MATCHER_P2(HasSymlink, path, target, "")
{
auto stat = arg->maybeLstat(path);
if (!stat) {
*result_listener << arg->showPath(path) << " does not exist";
return false;
}
if (stat->type != SourceAccessor::tSymlink) {
*result_listener << arg->showPath(path) << " is not a symlink";
return false;
}
auto actual = arg->readLink(path);
if (actual != target) {
*result_listener << arg->showPath(path) << " points to " << ::testing::PrintToString(actual);
return false;
}
return true;
}
MATCHER_P2(HasDirectory, path, dirents, "")
{
auto stat = arg->maybeLstat(path);
if (!stat) {
*result_listener << arg->showPath(path) << " does not exist";
return false;
}
if (stat->type != SourceAccessor::tDirectory) {
*result_listener << arg->showPath(path) << " is not a directory";
return false;
}
auto actual = arg->readDirectory(path);
std::set<std::string> actualKeys, expectedKeys(dirents.begin(), dirents.end());
for (auto & [k, _] : actual)
actualKeys.insert(k);
if (actualKeys != expectedKeys) {
*result_listener << arg->showPath(path) << " has entries " << ::testing::PrintToString(actualKeys);
return false;
}
return true;
}
class FSSourceAccessorTest : public ::testing::Test
{
protected:
std::filesystem::path tmpDir;
std::unique_ptr<nix::AutoDelete> delTmpDir;
void SetUp() override
{
tmpDir = nix::createTempDir();
delTmpDir = std::make_unique<nix::AutoDelete>(tmpDir, true);
}
void TearDown() override
{
delTmpDir.release();
}
};
TEST_F(FSSourceAccessorTest, works)
{
{
RestoreSink sink(false);
sink.dstPath = tmpDir;
sink.dirFd = openDirectory(tmpDir);
sink.createDirectory(CanonPath("subdir"));
sink.createRegularFile(CanonPath("file1"), [](CreateRegularFileSink & crf) { crf("content1"); });
sink.createRegularFile(CanonPath("subdir/file2"), [](CreateRegularFileSink & crf) { crf("content2"); });
sink.createSymlink(CanonPath("rootlink"), "target");
sink.createDirectory(CanonPath("a"));
sink.createSymlink(CanonPath("a/dirlink"), "../subdir");
}
EXPECT_THAT(makeFSSourceAccessor(tmpDir / "file1"), HasContents(CanonPath::root, "content1"));
EXPECT_THAT(makeFSSourceAccessor(tmpDir / "rootlink"), HasSymlink(CanonPath::root, "target"));
EXPECT_THAT(
makeFSSourceAccessor(tmpDir),
HasDirectory(CanonPath::root, std::set<std::string>{"file1", "subdir", "rootlink", "a"}));
EXPECT_THAT(makeFSSourceAccessor(tmpDir / "subdir"), HasDirectory(CanonPath::root, std::set<std::string>{"file2"}));
{
auto accessor = makeFSSourceAccessor(tmpDir);
EXPECT_THAT(accessor, HasContents(CanonPath("file1"), "content1"));
EXPECT_THAT(accessor, HasContents(CanonPath("subdir/file2"), "content2"));
EXPECT_TRUE(accessor->pathExists(CanonPath("file1")));
EXPECT_FALSE(accessor->pathExists(CanonPath("nonexistent")));
EXPECT_THROW(accessor->readFile(CanonPath("a/dirlink/file2")), SymlinkNotAllowed);
EXPECT_THROW(accessor->maybeLstat(CanonPath("a/dirlink/file2")), SymlinkNotAllowed);
EXPECT_THROW(accessor->readDirectory(CanonPath("a/dirlink")), SymlinkNotAllowed);
EXPECT_THROW(accessor->pathExists(CanonPath("a/dirlink/file2")), SymlinkNotAllowed);
}
{
auto accessor = makeFSSourceAccessor(tmpDir / "nonexistent");
EXPECT_FALSE(accessor->maybeLstat(CanonPath::root));
EXPECT_THROW(accessor->readFile(CanonPath::root), SystemError);
}
{
auto accessor = makeFSSourceAccessor(tmpDir, true);
EXPECT_EQ(accessor->getLastModified(), 0);
accessor->maybeLstat(CanonPath("file1"));
EXPECT_GT(accessor->getLastModified(), 0);
}
}
} // namespace nix
+2 -1
View File
@@ -103,7 +103,8 @@ void SourceAccessor::dumpPath(const CanonPath & path, Sink & sink, PathFilter &
time_t dumpPathAndGetMtime(const Path & path, Sink & sink, PathFilter & filter)
{
auto path2 = PosixSourceAccessor::createAtRoot(path, /*trackLastModified=*/true);
SourcePath path2 = {
makeFSSourceAccessor(std::filesystem::path{}, /*trackLastModified=*/true), CanonPath(absPath(path))};
path2.dumpPath(sink, filter);
return path2.accessor->getLastModified().value();
}
+6 -16
View File
@@ -713,27 +713,17 @@ AutoCloseFD createAnonymousTempFile()
{
AutoCloseFD fd;
#ifdef O_TMPFILE
static std::atomic_flag tmpfileUnsupported{};
if (!tmpfileUnsupported.test()) /* Try with O_TMPFILE first. */ {
/* Use O_EXCL, because the file is never supposed to be linked into filesystem. */
fd = ::open(defaultTempDir().c_str(), O_TMPFILE | O_CLOEXEC | O_RDWR | O_EXCL, S_IWUSR | S_IRUSR);
if (!fd) {
/* Not supported by the filesystem or the kernel. */
if (errno == EOPNOTSUPP || errno == EISDIR)
tmpfileUnsupported.test_and_set(); /* Set flag and fall through to createTempFile. */
else
throw SysError("creating anonymous temporary file");
} else {
return fd; /* Successfully created. */
}
}
#endif
fd = ::open(defaultTempDir().c_str(), O_TMPFILE | O_CLOEXEC | O_RDWR, S_IWUSR | S_IRUSR);
if (!fd)
throw SysError("creating anonymous temporary file");
#else
auto [fd2, path] = createTempFile("nix-anonymous");
if (!fd2)
throw SysError("creating temporary file '%s'", path);
fd = std::move(fd2);
#ifndef _WIN32
# ifndef _WIN32
unlink(requireCString(path)); /* We only care about the file descriptor. */
# endif
#endif
return fd;
}
+2 -7
View File
@@ -1,11 +1,9 @@
#pragma once
///@file
#include "nix/util/error.hh"
#include <cassert>
#include <limits>
#include <type_traits>
#include <cstdint>
#include <bit>
namespace nix {
@@ -18,10 +16,7 @@ template<typename T>
constexpr T alignUp(T val, unsigned alignment)
{
assert(std::has_single_bit(alignment) && "alignment must be a power of 2");
assert(alignment <= std::numeric_limits<T>::max());
T mask = ~(static_cast<T>(alignment) - 1u);
if (val > std::numeric_limits<T>::max() - (alignment - 1)) /* Overflow check. */
throw Error("can't align %d to %d: value is too large", val, alignment);
T mask = ~(T{alignment} - 1u);
return (val + alignment - 1) & mask;
}
@@ -2,7 +2,6 @@
///@file
#include "nix/util/canon-path.hh"
#include "nix/util/types.hh"
#include "nix/util/error.hh"
#ifdef _WIN32
@@ -236,18 +235,6 @@ std::wstring handleToFileName(Descriptor handle);
#ifndef _WIN32
namespace unix {
struct SymlinkNotAllowed : public Error
{
CanonPath path;
SymlinkNotAllowed(CanonPath path)
/* Can't provide better error message, since the parent directory is only known to the caller. */
: Error("relative path '%s' points to a symlink, which is not allowed", path.rel())
, path(std::move(path))
{
}
};
/**
* Safe(r) function to open \param path file relative to \param dirFd, while
* disallowing escaping from a directory and resolving any symlinks in the
-1
View File
@@ -65,7 +65,6 @@ headers = files(
'signals.hh',
'signature/local-keys.hh',
'signature/signer.hh',
'socket.hh',
'sort.hh',
'source-accessor.hh',
'source-path.hh',
@@ -43,36 +43,6 @@ public:
std::optional<std::filesystem::path> getPhysicalPath(const CanonPath & path) override;
/**
* Create a `PosixSourceAccessor` and `SourcePath` corresponding to
* some native path.
*
* @param Whether the accessor should return a non-null getLastModified.
* When true the accessor must be used only by a single thread.
*
* The `PosixSourceAccessor` is rooted as far up the tree as
* possible, (e.g. on Windows it could scoped to a drive like
* `C:\`). This allows more `..` parent accessing to work.
*
* @note When `path` is trusted user input, canonicalize it using
* `std::filesystem::canonical`, `makeParentCanonical`, `std::filesystem::weakly_canonical`, etc,
* as appropriate for the use case. At least weak canonicalization is
* required for the `SourcePath` to do anything useful at the location it
* points to.
*
* @note A canonicalizing behavior is not built in `createAtRoot` so that
* callers do not accidentally introduce symlink-related security vulnerabilities.
* Furthermore, `createAtRoot` does not know whether the file pointed to by
* `path` should be resolved if it is itself a symlink. In other words,
* `createAtRoot` can not decide between aforementioned `canonical`, `makeParentCanonical`, etc. for its callers.
*
* See
* [`std::filesystem::path::root_path`](https://en.cppreference.com/w/cpp/filesystem/path/root_path)
* and
* [`std::filesystem::path::relative_path`](https://en.cppreference.com/w/cpp/filesystem/path/relative_path).
*/
static SourcePath createAtRoot(const std::filesystem::path & path, bool trackLastModified = false);
std::optional<std::time_t> getLastModified() override
{
return trackLastModified ? std::optional{mtime} : std::nullopt;
-61
View File
@@ -1,61 +0,0 @@
#pragma once
///@file
#include "nix/util/file-descriptor.hh"
#ifdef _WIN32
# include <winsock2.h>
#endif
namespace nix {
/**
* Often we want to use `Descriptor`, but Windows makes a slightly
* stronger file descriptor vs socket distinction, at least at the level
* of C types.
*/
using Socket =
#ifdef _WIN32
SOCKET
#else
int
#endif
;
#ifdef _WIN32
/**
* Windows gives this a different name
*/
# define SHUT_WR SD_SEND
# define SHUT_RDWR SD_BOTH
#endif
/**
* Convert a `Descriptor` to a `Socket`
*
* This is a no-op except on Windows.
*/
static inline Socket toSocket(Descriptor fd)
{
#ifdef _WIN32
return reinterpret_cast<Socket>(fd);
#else
return fd;
#endif
}
/**
* Convert a `Socket` to a `Descriptor`
*
* This is a no-op except on Windows.
*/
static inline Descriptor fromSocket(Socket fd)
{
#ifdef _WIN32
return reinterpret_cast<Descriptor>(fd);
#else
return fd;
#endif
}
} // namespace nix
@@ -222,6 +222,24 @@ ref<SourceAccessor> makeEmptySourceAccessor();
*/
MakeError(RestrictedPathError, Error);
struct SymlinkNotAllowed : public Error
{
CanonPath path;
SymlinkNotAllowed(CanonPath path)
: Error("relative path '%s' points to a symlink, which is not allowed", path.rel())
, path(std::move(path))
{
}
template<typename... Args>
SymlinkNotAllowed(CanonPath path, const std::string & fs, Args &&... args)
: Error(fs, std::forward<Args>(args)...)
, path(std::move(path))
{
}
};
/**
* Return an accessor for the root filesystem.
*/
@@ -233,7 +251,7 @@ ref<SourceAccessor> getFSSourceAccessor();
* elements, and that absolute symlinks are resolved relative to
* `root`.
*/
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root);
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root, bool trackLastModified = false);
/**
* Construct an accessor that presents a "union" view of a vector of
@@ -3,8 +3,10 @@
#include "nix/util/types.hh"
#include "nix/util/file-descriptor.hh"
#include "nix/util/socket.hh"
#ifdef _WIN32
# include <winsock2.h>
#endif
#include <unistd.h>
#include <filesystem>
@@ -21,6 +23,55 @@ AutoCloseFD createUnixDomainSocket();
*/
AutoCloseFD createUnixDomainSocket(const Path & path, mode_t mode);
/**
* Often we want to use `Descriptor`, but Windows makes a slightly
* stronger file descriptor vs socket distinction, at least at the level
* of C types.
*/
using Socket =
#ifdef _WIN32
SOCKET
#else
int
#endif
;
#ifdef _WIN32
/**
* Windows gives this a different name
*/
# define SHUT_WR SD_SEND
# define SHUT_RDWR SD_BOTH
#endif
/**
* Convert a `Socket` to a `Descriptor`
*
* This is a no-op except on Windows.
*/
static inline Socket toSocket(Descriptor fd)
{
#ifdef _WIN32
return reinterpret_cast<Socket>(fd);
#else
return fd;
#endif
}
/**
* Convert a `Socket` to a `Descriptor`
*
* This is a no-op except on Windows.
*/
static inline Descriptor fromSocket(Socket fd)
{
#ifdef _WIN32
return reinterpret_cast<Descriptor>(fd);
#else
return fd;
#endif
}
/**
* Bind a Unix domain socket to a path.
*/
+11 -16
View File
@@ -150,21 +150,16 @@ public:
Verbosity verbosity = lvlInfo;
static void writeFullLogging(Descriptor fd, std::string_view s)
{
try {
writeFull(fd, s, false);
} catch (SystemError & e) {
/* Ignore failing logging writes. We need to ignore write
errors to ensure that cleanup code that writes logs runs
to completion if the other side of the logging fd has
been closed unexpectedly. */
}
}
void writeToStderr(std::string_view s)
{
writeFullLogging(getStandardError(), s);
try {
writeFull(getStandardError(), s, false);
} catch (SystemError & e) {
/* Ignore failing writes to stderr. We need to ignore write
errors to ensure that cleanup code that logs to stderr runs
to completion if the other side of stderr has been closed
unexpectedly. */
}
}
std::unique_ptr<Logger> makeSimpleLogger(bool printBuildLogs)
@@ -250,15 +245,15 @@ struct JSONLogger : Logger
void write(const nlohmann::json & json)
{
auto line = (includeNixPrefix ? "@nix " : "")
+ json.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace) + "\n";
auto line =
(includeNixPrefix ? "@nix " : "") + json.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace);
/* Acquire a lock to prevent log messages from clobbering each
other. */
try {
auto state(_state.lock());
if (state->enabled)
writeFullLogging(fd, line);
writeLine(fd, line);
} catch (...) {
bool enabled = false;
std::swap(_state.lock()->enabled, enabled);
+3 -14
View File
@@ -1,7 +1,5 @@
#include "nix/util/posix-source-accessor.hh"
#include "nix/util/source-path.hh"
#include "nix/util/signals.hh"
#include "nix/util/sync.hh"
#include <boost/unordered/concurrent_flat_map.hpp>
@@ -20,15 +18,6 @@ PosixSourceAccessor::PosixSourceAccessor()
{
}
SourcePath PosixSourceAccessor::createAtRoot(const std::filesystem::path & path, bool trackLastModified)
{
std::filesystem::path path2 = absPath(path);
return {
make_ref<PosixSourceAccessor>(path2.root_path(), trackLastModified),
CanonPath{path2.relative_path().string()},
};
}
std::filesystem::path PosixSourceAccessor::makeAbsPath(const CanonPath & path)
{
return root.empty() ? (std::filesystem::path{path.abs()})
@@ -208,7 +197,7 @@ void PosixSourceAccessor::assertNoSymlinks(CanonPath path)
while (!path.isRoot()) {
auto st = cachedLstat(path);
if (st && S_ISLNK(st->st_mode))
throw Error("path '%s' is a symlink", showPath(path));
throw SymlinkNotAllowed(path, "path '%s' is a symlink", showPath(path));
path.pop();
}
}
@@ -219,8 +208,8 @@ ref<SourceAccessor> getFSSourceAccessor()
return rootFS;
}
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root)
ref<SourceAccessor> makeFSSourceAccessor(std::filesystem::path root, bool trackLastModified)
{
return make_ref<PosixSourceAccessor>(std::move(root));
return make_ref<PosixSourceAccessor>(std::move(root), trackLastModified);
}
} // namespace nix
+5 -5
View File
@@ -1,7 +1,6 @@
#include "nix/util/serialise.hh"
#include "nix/util/compression.hh"
#include "nix/util/signals.hh"
#include "nix/util/socket.hh"
#include "nix/util/util.hh"
#include <cstring>
@@ -12,6 +11,7 @@
#ifdef _WIN32
# include <fileapi.h>
# include <winsock2.h>
# include "nix/util/windows-error.hh"
#else
# include <poll.h>
@@ -184,20 +184,20 @@ bool FdSource::hasData()
while (true) {
fd_set fds;
FD_ZERO(&fds);
Socket sock = toSocket(fd);
FD_SET(sock, &fds);
int fd_ = fromDescriptorReadOnly(fd);
FD_SET(fd_, &fds);
struct timeval timeout;
timeout.tv_sec = 0;
timeout.tv_usec = 0;
auto n = select(sock + 1, &fds, nullptr, nullptr, &timeout);
auto n = select(fd_ + 1, &fds, nullptr, nullptr, &timeout);
if (n < 0) {
if (errno == EINTR)
continue;
throw SysError("polling file descriptor");
}
return FD_ISSET(sock, &fds);
return FD_ISSET(fd, &fds);
}
}
+2 -2
View File
@@ -136,7 +136,7 @@ static void extract_archive(TarArchive & archive, const std::filesystem::path &
if (!name)
throw Error("cannot get archive member name: %s", archive_error_string(archive.archive));
if (r == ARCHIVE_WARN)
warn(archive_error_string(archive.archive));
warn("getting archive member '%1%': %2%", name, archive_error_string(archive.archive));
else
archive.check(r);
@@ -193,7 +193,7 @@ time_t unpackTarfileToSink(TarArchive & archive, ExtendedFileSystemObjectSink &
throw Error("cannot get archive member name: %s", archive_error_string(archive.archive));
auto cpath = CanonPath{path};
if (r == ARCHIVE_WARN)
warn(archive_error_string(archive.archive));
warn("getting archive member '%1%': %2%", path, archive_error_string(archive.archive));
else
archive.check(r);
-4
View File
@@ -35,18 +35,14 @@ struct UnionSourceAccessor : SourceAccessor
DirEntries readDirectory(const CanonPath & path) override
{
DirEntries result;
bool exists = false;
for (auto & accessor : accessors) {
auto st = accessor->maybeLstat(path);
if (!st)
continue;
exists = true;
for (auto & entry : accessor->readDirectory(path))
// Don't override entries from previous accessors.
result.insert(entry);
}
if (!exists)
throw FileNotFound("path '%s' does not exist", showPath(path));
return result;
}
+7 -9
View File
@@ -3,17 +3,15 @@
#include "nix/util/signals.hh"
#include "nix/util/finally.hh"
#include "nix/util/serialise.hh"
#include "nix/util/source-accessor.hh"
#include <fcntl.h>
#include <unistd.h>
#include <poll.h>
#if defined(__linux__)
# include <sys/syscall.h> /* pull __NR_* definitions */
#endif
#if defined(__linux__) && defined(__NR_openat2)
# define HAVE_OPENAT2 1
# include <sys/syscall.h>
# include <linux/openat2.h>
#else
# define HAVE_OPENAT2 0
@@ -304,10 +302,10 @@ openFileEnsureBeneathNoSymlinksIterative(Descriptor dirFd, const CanonPath & pat
if (errno == ENOTDIR) /* Path component might be a symlink. */ {
struct ::stat st;
if (::fstatat(getParentFd(), component.c_str(), &st, AT_SYMLINK_NOFOLLOW) == 0 && S_ISLNK(st.st_mode))
throw unix::SymlinkNotAllowed(path2);
throw SymlinkNotAllowed(path2);
errno = ENOTDIR; /* Restore the errno. */
} else if (errno == ELOOP) {
throw unix::SymlinkNotAllowed(path2);
throw SymlinkNotAllowed(path2);
}
return INVALID_DESCRIPTOR;
@@ -318,7 +316,7 @@ openFileEnsureBeneathNoSymlinksIterative(Descriptor dirFd, const CanonPath & pat
auto res = ::openat(getParentFd(), std::string(path.baseName().value()).c_str(), flags | O_NOFOLLOW, mode);
if (res < 0 && errno == ELOOP)
throw unix::SymlinkNotAllowed(path);
throw SymlinkNotAllowed(path);
return res;
}
@@ -326,12 +324,12 @@ Descriptor unix::openFileEnsureBeneathNoSymlinks(Descriptor dirFd, const CanonPa
{
assert(!path.rel().starts_with('/')); /* Just in case the invariant is somehow broken. */
assert(!path.isRoot());
#if HAVE_OPENAT2
#ifdef __linux__
auto maybeFd = linux::openat2(
dirFd, path.rel_c_str(), flags, static_cast<uint64_t>(mode), RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS);
if (maybeFd) {
if (*maybeFd < 0 && errno == ELOOP)
throw unix::SymlinkNotAllowed(path);
throw SymlinkNotAllowed(path);
return *maybeFd;
}
#endif
+1 -1
View File
@@ -38,7 +38,7 @@ struct CmdAddToStore : MixDryRun, StoreCommand
if (!namePart)
namePart = baseNameOf(path);
auto sourcePath = PosixSourceAccessor::createAtRoot(makeParentCanonical(path));
SourcePath sourcePath = makeFSSourceAccessor(makeParentCanonical(path));
auto storePath = dryRun ? store->computeStorePath(*namePart, sourcePath, caMethod, hashAlgo, {}).first
: store->addToStoreSlow(*namePart, sourcePath, caMethod, hashAlgo, {}).path;
+1 -3
View File
@@ -85,9 +85,7 @@ struct CmdHashBase : Command
return std::make_unique<HashSink>(hashAlgo);
};
auto makeSourcePath = [&]() -> SourcePath {
return PosixSourceAccessor::createAtRoot(makeParentCanonical(path));
};
auto makeSourcePath = [&]() -> SourcePath { return makeFSSourceAccessor(makeParentCanonical(path)); };
Hash h{HashAlgorithm::SHA256}; // throwaway def to appease C++
switch (mode) {
+3 -1
View File
@@ -613,6 +613,8 @@ static void main_nix_build(int argc, char ** argv)
environment variables and shell functions. Also don't
lose the current $PATH directories. */
auto rcfile = (tmpDir.path() / "rc").string();
auto tz = getEnv("TZ");
auto tzExport = tz ? "export TZ=" + escapeShellArgAlways(*tz) + "; " : "";
std::string rc = fmt(
(R"(_nix_shell_clean_tmpdir() { command rm -rf %1%; };)"s
"trap _nix_shell_clean_tmpdir EXIT; "
@@ -646,7 +648,7 @@ static void main_nix_build(int argc, char ** argv)
(pure ? "" : "PATH=$PATH:$p; unset p; "),
escapeShellArgAlways(dirOf(*shell)),
escapeShellArgAlways(*shell),
(getenv("TZ") ? (std::string("export TZ=") + escapeShellArgAlways(getenv("TZ")) + "; ") : ""),
tzExport,
envCommand);
vomit("Sourcing nix-shell with file %s and contents:\n%s", rcfile, rc);
writeFile(rcfile, rc);
+2 -2
View File
@@ -191,7 +191,7 @@ static void opAdd(Strings opFlags, Strings opArgs)
throw UsageError("unknown flag");
for (auto & i : opArgs) {
auto sourcePath = PosixSourceAccessor::createAtRoot(makeParentCanonical(i));
SourcePath sourcePath = makeFSSourceAccessor(makeParentCanonical(i));
cout << fmt("%s\n", store->printStorePath(store->addToStore(std::string(baseNameOf(i)), sourcePath)));
}
}
@@ -215,7 +215,7 @@ static void opAddFixed(Strings opFlags, Strings opArgs)
opArgs.pop_front();
for (auto & i : opArgs) {
auto sourcePath = PosixSourceAccessor::createAtRoot(makeParentCanonical(i));
SourcePath sourcePath = makeFSSourceAccessor(makeParentCanonical(i));
std::cout << fmt(
"%s\n", store->printStorePath(store->addToStoreSlow(baseNameOf(i), sourcePath, method, hashAlgo).path));
}
+5 -6
View File
@@ -437,23 +437,22 @@ static void forwardStdioConnection(RemoteStore & store)
int from = conn->from.fd;
int to = conn->to.fd;
Socket fromSock = toSocket(from), stdinSock = toSocket(getStandardInput());
auto nfds = std::max(fromSock, stdinSock) + 1;
auto nfds = std::max(from, STDIN_FILENO) + 1;
while (true) {
fd_set fds;
FD_ZERO(&fds);
FD_SET(fromSock, &fds);
FD_SET(stdinSock, &fds);
FD_SET(from, &fds);
FD_SET(STDIN_FILENO, &fds);
if (select(nfds, &fds, nullptr, nullptr, nullptr) == -1)
throw SysError("waiting for data from client or server");
if (FD_ISSET(fromSock, &fds)) {
if (FD_ISSET(from, &fds)) {
auto res = splice(from, nullptr, STDOUT_FILENO, nullptr, SSIZE_MAX, SPLICE_F_MOVE);
if (res == -1)
throw SysError("splicing data from daemon socket to stdout");
else if (res == 0)
throw EndOfFile("unexpected EOF from daemon socket");
}
if (FD_ISSET(stdinSock, &fds)) {
if (FD_ISSET(STDIN_FILENO, &fds)) {
auto res = splice(STDIN_FILENO, nullptr, to, nullptr, SSIZE_MAX, SPLICE_F_MOVE);
if (res == -1)
throw SysError("splicing data from stdin to daemon socket");
+2 -2
View File
@@ -256,7 +256,7 @@ hashPath(char * algo, int base32, char * path)
PPCODE:
try {
Hash h = hashPath(
PosixSourceAccessor::createAtRoot(path),
SourcePath{getFSSourceAccessor(), CanonPath(absPath(Path(path)))},
FileIngestionMethod::NixArchive, parseHashAlgo(algo)).first;
auto s = h.to_string(base32 ? HashFormat::Nix32 : HashFormat::Base16, false);
XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
@@ -336,7 +336,7 @@ StoreWrapper::addToStore(char * srcPath, int recursive, char * algo)
auto method = recursive ? ContentAddressMethod::Raw::NixArchive : ContentAddressMethod::Raw::Flat;
auto path = THIS->store->addToStore(
std::string(baseNameOf(srcPath)),
PosixSourceAccessor::createAtRoot(srcPath),
SourcePath{getFSSourceAccessor(), CanonPath(absPath(Path(srcPath)))},
method, parseHashAlgo(algo));
XPUSHs(sv_2mortal(newSVpv(THIS->store->printStorePath(path).c_str(), 0)));
} catch (Error & e) {
+5 -81
View File
@@ -158,24 +158,14 @@ printf "" | nix build --no-link --stdin --json | jq --exit-status '. == []'
printf "%s\n" "$drv^*" | nix build --no-link --stdin --json | jq --exit-status '.[0]|has("drvPath")'
# --keep-going and FOD
if isDaemonNewer "2.34pre"; then
# With the fix, cancelled goals are not reported as failures.
# Use -j1 so only x1 starts and fails; x2, x3, x4 are cancelled.
out="$(nix build -f fod-failing.nix -j1 -L 2>&1)" && status=0 || status=$?
test "$status" = 1
# Only the hash mismatch error for x1. Cancelled goals not reported.
test "$(<<<"$out" grep -cE '^error:')" = 1
# Regression test: error messages should not be empty (end with just "failed:")
<<<"$out" grepQuietInverse -E "^error:.*failed: *$"
else
out="$(nix build -f fod-failing.nix -L 2>&1)" && status=0 || status=$?
test "$status" = 1
# At minimum, check that x1 is reported as failing
<<<"$out" grepQuiet -E "error:.*-x1"
fi
out="$(nix build -f fod-failing.nix -L 2>&1)" && status=0 || status=$?
test "$status" = 1
# one "hash mismatch" error, one "build of ... failed"
test "$(<<<"$out" grep -cE '^error:')" = 2
<<<"$out" grepQuiet -E "hash mismatch in fixed-output derivation '.*-x1\\.drv'"
<<<"$out" grepQuiet -vE "hash mismatch in fixed-output derivation '.*-x3\\.drv'"
<<<"$out" grepQuiet -vE "hash mismatch in fixed-output derivation '.*-x2\\.drv'"
<<<"$out" grepQuiet -E "error: build of '.*-x[1-4]\\.drv\\^out', '.*-x[1-4]\\.drv\\^out', '.*-x[1-4]\\.drv\\^out', '.*-x[1-4]\\.drv\\^out' failed"
out="$(nix build -f fod-failing.nix -L x1 x2 x3 --keep-going 2>&1)" && status=0 || status=$?
test "$status" = 1
@@ -213,69 +203,3 @@ else
fi
<<<"$out" grepQuiet -vE "hash mismatch in fixed-output derivation '.*-x3\\.drv'"
<<<"$out" grepQuiet -vE "hash mismatch in fixed-output derivation '.*-x2\\.drv'"
# Regression test: cancelled builds should not be reported as failures
# When fast-fail fails, slow and depends-on-slow are cancelled (not failed).
# Only fast-fail should be reported as a failure.
# Uses fifo for synchronization to ensure deterministic behavior.
# Requires -j2 so slow and fast-fail run concurrently (fifo deadlocks if serialized).
if isDaemonNewer "2.34pre" && canUseSandbox; then
fifoDir="$TEST_ROOT/cancelled-builds-fifo"
mkdir -p "$fifoDir"
mkfifo "$fifoDir/fifo"
chmod a+rw "$fifoDir/fifo"
# When using a separate test store, we need sandbox-paths to access
# the system store (where bash/coreutils live). On NixOS, the test
# uses the system store directly, so this isn't needed (and would
# conflict with input paths).
sandboxPathsArg=()
if ! isTestOnNixOS; then
sandboxPathsArg=(--option sandbox-paths "/nix/store")
fi
out="$(nix flake check ./cancelled-builds --impure -L -j2 \
--option sandbox true \
"${sandboxPathsArg[@]}" \
--option sandbox-build-dir /build-tmp \
--option extra-sandbox-paths "/cancelled-builds-fifo=$fifoDir" \
2>&1)" && status=0 || status=$?
rm -rf "$fifoDir"
test "$status" = 1
# The error should be for fast-fail, not for cancelled goals
<<<"$out" grepQuiet -E "Cannot build.*fast-fail"
# Cancelled goals should NOT appear in error messages (but may appear in "will be built" list)
<<<"$out" grepQuietInverse -E "^error:.*slow"
<<<"$out" grepQuietInverse -E "^error:.*depends-on-slow"
<<<"$out" grepQuietInverse -E "^error:.*depends-on-fail"
# Error messages should not be empty (end with just "failed:")
<<<"$out" grepQuietInverse -E "^error:.*failed: *$"
# Test that nix build follows the same rules (uses a slightly different code path)
mkdir -p "$fifoDir"
mkfifo "$fifoDir/fifo"
chmod a+rw "$fifoDir/fifo"
sandboxPathsArg=()
if ! isTestOnNixOS; then
sandboxPathsArg=(--option sandbox-paths "/nix/store")
fi
system=$(nix eval --raw --impure --expr builtins.currentSystem)
out="$(nix build --impure -L -j2 \
--option sandbox true \
"${sandboxPathsArg[@]}" \
--option sandbox-build-dir /build-tmp \
--option extra-sandbox-paths "/cancelled-builds-fifo=$fifoDir" \
"./cancelled-builds#checks.$system.slow" \
"./cancelled-builds#checks.$system.depends-on-slow" \
"./cancelled-builds#checks.$system.fast-fail" \
"./cancelled-builds#checks.$system.depends-on-fail" \
2>&1)" && status=0 || status=$?
rm -rf "$fifoDir"
test "$status" = 1
# The error should be for fast-fail, not for cancelled goals
<<<"$out" grepQuiet -E "Cannot build.*fast-fail"
# Cancelled goals should NOT appear in error messages
<<<"$out" grepQuietInverse -E "^error:.*slow"
<<<"$out" grepQuietInverse -E "^error:.*depends-on-slow"
<<<"$out" grepQuietInverse -E "^error:.*depends-on-fail"
# Error messages should not be empty (end with just "failed:")
<<<"$out" grepQuietInverse -E "^error:.*failed: *$"
fi
@@ -1,64 +0,0 @@
# Regression test for cancelled builds not being reported as failures.
#
# Scenario: When a build fails while other builds are running, those other
# builds (and their dependents) get cancelled. Previously, cancelled builds
# were incorrectly reported as failures with empty error messages.
#
# Uses a fifo for synchronization: fast-fail waits for slow to start before
# failing, ensuring slow is actually running when it gets cancelled.
#
# See: tests/functional/build.sh (search for "cancelled-builds")
{
outputs =
{ self }:
let
config = import "${builtins.getEnv "_NIX_TEST_BUILD_DIR"}/config.nix";
in
with config;
{
checks.${system} = {
# A derivation that signals it started via fifo, then waits
slow = mkDerivation {
name = "slow";
buildCommand = ''
echo "slow: started, signaling via fifo"
echo started > /cancelled-builds-fifo/fifo
echo "slow: waiting..."
sleep 10
touch $out
'';
};
# Depends on slow - will be cancelled when fast-fail fails
depends-on-slow = mkDerivation {
name = "depends-on-slow";
slow = self.checks.${system}.slow;
buildCommand = ''
echo "depends-on-slow: slow finished at $slow"
touch $out
'';
};
# Waits for slow to start via fifo, then fails
fast-fail = mkDerivation {
name = "fast-fail";
buildCommand = ''
echo "fast-fail: waiting for slow to start..."
read line < /cancelled-builds-fifo/fifo
echo "fast-fail: slow started, now failing" >&2
exit 1
'';
};
# Depends on fast-fail - will fail with DependencyFailed
depends-on-fail = mkDerivation {
name = "depends-on-fail";
fast-fail = self.checks.${system}.fast-fail;
buildCommand = ''
echo "depends-on-fail: fast-fail finished (should never get here)"
touch $out
'';
};
};
};
}
-33
View File
@@ -10,36 +10,3 @@ if [[ -v NIX_DAEMON_PACKAGE ]]; then expected=1; fi # work around the daemon not
expectStderr "$expected" nix-build ./text-hashed-output.nix -A failingWrapper --no-out-link \
| grepQuiet "build of resolved derivation '.*use-dynamic-drv-in-non-dynamic-drv-wrong.drv' failed"
# Test that error messages are not empty when a producer derivation fails.
# This exercises the nrFailed path in DerivationTrampolineGoal::init().
#
# Using `nix build --expr` with builtins.outputOf creates a top-level
# DerivationTrampolineGoal that goes through buildPathsWithResults.
# When the producer fails, the nrFailed path must use doneFailure (not amDone)
# to set buildResult.inner with a proper error message.
#
# Without the fix, the error message would be empty because amDone doesn't
# set buildResult.inner, so rethrow() throws Error("") - an empty message.
out=$(nix build --impure --no-link --expr '
let
config = import (builtins.getEnv "_NIX_TEST_BUILD_DIR" + "/config.nix");
inherit (config) mkDerivation;
# A CA derivation that fails before producing a .drv
failingProducer = mkDerivation {
name = "failing-producer";
buildCommand = "echo This producer fails; exit 1";
__contentAddressed = true;
outputHashMode = "text";
outputHashAlgo = "sha256";
};
in
# Build the dynamic derivation output directly - this creates a top-level
# DerivationTrampolineGoal, not a nested one inside a DerivationGoal
builtins.outputOf failingProducer.outPath "out"
' 2>&1) || true
# The error message must NOT be empty - it should mention the failed derivation
echo "$out" | grepQuiet "failed to obtain derivation of"
@@ -1,16 +0,0 @@
error:
… while evaluating the attribute 'absolutePath'
at /pwd/lang/eval-fail-readDir-nonexistent-1.nix:2:3:
1| {
2| absolutePath = builtins.readDir /this/path/really/should/not/exist;
| ^
3| }
… while calling the 'readDir' builtin
at /pwd/lang/eval-fail-readDir-nonexistent-1.nix:2:18:
1| {
2| absolutePath = builtins.readDir /this/path/really/should/not/exist;
| ^
3| }
error: path '/this/path/really/should/not/exist' does not exist
@@ -1,3 +0,0 @@
{
absolutePath = builtins.readDir /this/path/really/should/not/exist;
}
@@ -1,16 +0,0 @@
error:
… while evaluating the attribute 'relativePath'
at /pwd/lang/eval-fail-readDir-nonexistent-2.nix:2:3:
1| {
2| relativePath = builtins.readDir ./this/path/really/should/not/exist;
| ^
3| }
… while calling the 'readDir' builtin
at /pwd/lang/eval-fail-readDir-nonexistent-2.nix:2:18:
1| {
2| relativePath = builtins.readDir ./this/path/really/should/not/exist;
| ^
3| }
error: path '/pwd/lang/this/path/really/should/not/exist' does not exist
@@ -1,3 +0,0 @@
{
relativePath = builtins.readDir ./this/path/really/should/not/exist;
}
@@ -1,16 +0,0 @@
error:
… while evaluating the attribute 'regularFile'
at /pwd/lang/eval-fail-readDir-not-a-directory-1.nix:2:3:
1| {
2| regularFile = builtins.readDir ./readDir/bar;
| ^
3| }
… while calling the 'readDir' builtin
at /pwd/lang/eval-fail-readDir-not-a-directory-1.nix:2:17:
1| {
2| regularFile = builtins.readDir ./readDir/bar;
| ^
3| }
error: cannot read directory "/pwd/lang/readDir/bar": Not a directory
@@ -1,3 +0,0 @@
{
regularFile = builtins.readDir ./readDir/bar;
}
@@ -1,16 +0,0 @@
error:
… while evaluating the attribute 'symlinkedRegularFile'
at /pwd/lang/eval-fail-readDir-not-a-directory-2.nix:2:3:
1| {
2| symlinkedRegularFile = builtins.readDir ./readDir/linked;
| ^
3| }
… while calling the 'readDir' builtin
at /pwd/lang/eval-fail-readDir-not-a-directory-2.nix:2:26:
1| {
2| symlinkedRegularFile = builtins.readDir ./readDir/linked;
| ^
3| }
error: cannot read directory "/pwd/lang/readDir/foo/git-hates-directories": Not a directory
@@ -1,3 +0,0 @@
{
symlinkedRegularFile = builtins.readDir ./readDir/linked;
}
@@ -1 +0,0 @@
{ git-hates-directories = "regular"; }
@@ -1 +0,0 @@
builtins.readDir ./readDir/ldir
+6 -3
View File
@@ -106,14 +106,16 @@ let
foo = runCommand "foo" { } ''
mkdir -p $out/bin
echo 'echo ${fooContents}' > $out/bin/foo
echo '#!${shell}' > $out/bin/foo
echo 'echo ${fooContents}' >> $out/bin/foo
chmod a+rx $out/bin/foo
ln -s ${shell} $out/bin/bash
'';
bar = runCommand "bar" { } ''
mkdir -p $out/bin
echo 'echo bar' > $out/bin/bar
echo '#!${shell}' > $out/bin/bar
echo 'echo bar' >> $out/bin/bar
chmod a+rx $out/bin/bar
'';
@@ -126,7 +128,8 @@ let
# ruby "interpreter" that outputs "$@"
ruby = runCommand "ruby" { } ''
mkdir -p $out/bin
echo 'printf %s "$*"' > $out/bin/ruby
echo '#!${shell}' > $out/bin/ruby
echo 'printf %s "$*"' >> $out/bin/ruby
chmod a+rx $out/bin/ruby
'';
+70 -133
View File
@@ -48,14 +48,9 @@ in
rootCredentialsFile = pkgs.writeText "minio-credentials-full" ''
MINIO_ROOT_USER=${accessKey}
MINIO_ROOT_PASSWORD=${secretKey}
MINIO_DOMAIN=minio.local
'';
};
networking.firewall.allowedTCPPorts = [ 9000 ];
# Static hosts for virtual-hosted-style S3 tests.
# MinIO with MINIO_DOMAIN=minio.local accepts virtual-hosted requests
# where the bucket name is a hostname prefix.
networking.extraHosts = "127.0.0.1 vhost-test.minio.local minio.local";
};
client =
@@ -67,7 +62,6 @@ in
experimental-features = nix-command
substituters =
'';
networking.extraHosts = "192.168.1.2 vhost-test.minio.local minio.local";
};
};
@@ -89,10 +83,6 @@ in
ENDPOINT = 'http://server:9000'
REGION = 'eu-west-1'
# Virtual-hosted-style configuration (requires MINIO_DOMAIN and static host entries)
VHOST_DOMAIN = 'minio.local'
VHOST_ENDPOINT = f'http://{VHOST_DOMAIN}:9000'
PKGS = {
'A': '${pkgA}',
'B': '${pkgB}',
@@ -237,40 +227,7 @@ in
"Credential provider caching failed"
)
@setup_s3()
def test_aws_log_integration(bucket):
"""Test that AWS SDK logs are properly routed through Nix logger"""
print("\n=== Testing AWS Log Integration ===")
store_url = make_s3_url(bucket)
# With default verbosity, AWS noise should NOT appear
# All AWS messages are demoted to lvlDebug or lvlVomit
output_default = server.succeed(
f"{ENV_WITH_CREDS} nix copy --to '{store_url}' {PKGS['A']} 2>&1"
)
if "(aws:" in output_default:
print("Output at default verbosity:")
print(output_default)
raise Exception("Found AWS noise at default verbosity")
# With --debug (lvlDebug), we should see AWS messages with (aws:subject) prefix
output_debug = server.succeed(
f"{ENV_WITH_CREDS} nix copy --debug --to '{store_url}' {PKGS['B']} 2>&1"
)
# Check for the (aws:subject) prefix format
if "(aws:" not in output_debug:
print("Output at --debug verbosity:")
print(output_debug)
raise Exception("Expected to see (aws:subject) prefix in debug output")
# Should also see Nix's own credential provider creation message
if "creating new AWS credential provider" not in output_debug:
print("Debug output:")
print(output_debug)
raise Exception("Expected to see credential provider creation at debug level")
print("✓ Credential provider created once and cached")
@setup_s3(populate_bucket=[PKGS['A']])
def test_fetchurl_basic(bucket):
@@ -286,6 +243,8 @@ in
f"'builtins.fetchurl {{ name = \"foo\"; url = \"{cache_info_url}\"; }}'"
)
print("✓ builtins.fetchurl works with s3:// URLs")
@setup_s3()
def test_error_message_formatting(bucket):
"""Verify error messages display URLs correctly"""
@@ -304,6 +263,8 @@ in
print(error_msg)
raise Exception("Error message formatting failed - should show actual URL, not %s placeholder")
print("✓ Error messages format URLs correctly")
@setup_s3(populate_bucket=[PKGS['A']])
def test_fork_credential_preresolution(bucket):
"""Test credential pre-resolution in forked processes"""
@@ -343,6 +304,8 @@ in
print(output)
raise Exception("Expected to find FileTransfer creation in forked process")
print(" ✓ Forked process creates fresh FileTransfer")
# Verify pre-resolution in parent
required_messages = [
"Pre-resolving AWS credentials for S3 URL in builtin:fetchurl",
@@ -355,6 +318,8 @@ in
print(output)
raise Exception(f"Missing expected message: {msg}")
print(" ✓ Parent pre-resolves credentials")
# Verify child uses pre-resolved credentials
if "Using pre-resolved AWS credentials from parent process" not in output:
print("Debug output:")
@@ -378,6 +343,8 @@ in
print(output)
raise Exception(f"Child process (pid={child_pid}) should NOT create new credential providers")
print(" ✓ Child uses pre-resolved credentials (no new providers)")
@setup_s3(populate_bucket=[PKGS['A'], PKGS['B'], PKGS['C']])
def test_store_operations(bucket):
"""Test nix store info and copy operations"""
@@ -395,6 +362,8 @@ in
if not store_info.get("url"):
raise Exception("Store should have a URL")
print(f" ✓ Store URL: {store_info['url']}")
# Test copy from store
verify_packages_in_store(client, PKGS['A'], should_exist=False)
@@ -412,6 +381,9 @@ in
verify_packages_in_store(client, [PKGS['A'], PKGS['B'], PKGS['C']])
print(" ✓ nix copy works")
print(" ✓ Credentials cached on client")
@setup_s3(populate_bucket=[PKGS['A'], PKGS['B']], public=True)
def test_public_bucket_operations(bucket):
"""Test store operations on public bucket without credentials"""
@@ -421,6 +393,7 @@ in
# Verify store info works without credentials
client.succeed(f"nix store info --store '{store_url}' >&2")
print(" ✓ nix store info works without credentials")
# Get and validate store info JSON
info_json = client.succeed(f"nix store info --json --store '{store_url}'")
@@ -429,6 +402,8 @@ in
if not store_info.get("url"):
raise Exception("Store should have a URL")
print(f" ✓ Store URL: {store_info['url']}")
# Verify packages are not yet in client store
verify_packages_in_store(client, [PKGS['A'], PKGS['B']], should_exist=False)
@@ -441,6 +416,8 @@ in
# Verify packages were copied successfully
verify_packages_in_store(client, [PKGS['A'], PKGS['B']])
print(" ✓ nix copy from public bucket works without credentials")
@setup_s3(populate_bucket=[PKGS['A']])
def test_url_format_variations(bucket):
"""Test different S3 URL parameter combinations"""
@@ -449,10 +426,12 @@ in
# Test parameter order variation (region before endpoint)
url1 = f"s3://{bucket}?region={REGION}&endpoint={ENDPOINT}"
client.succeed(f"{ENV_WITH_CREDS} nix store info --store '{url1}' >&2")
print(" ✓ Parameter order: region before endpoint works")
# Test parameter order variation (endpoint before region)
url2 = f"s3://{bucket}?endpoint={ENDPOINT}&region={REGION}"
client.succeed(f"{ENV_WITH_CREDS} nix store info --store '{url2}' >&2")
print(" ✓ Parameter order: endpoint before region works")
@setup_s3(populate_bucket=[PKGS['A']])
def test_concurrent_fetches(bucket):
@@ -506,6 +485,9 @@ in
providers_created = output.count("creating new AWS credential provider")
transfers_created = output.count("builtin:fetchurl creating fresh FileTransfer instance")
print(f" ✓ {providers_created} credential providers created")
print(f" ✓ {transfers_created} FileTransfer instances created")
if transfers_created != 5:
print("Debug output:")
print(output)
@@ -531,10 +513,14 @@ in
pkg_hash = get_package_hash(PKGS['B'])
verify_content_encoding(server, bucket, f"{pkg_hash}.narinfo", "gzip")
print(" ✓ .narinfo has Content-Encoding: gzip")
# Verify client can download and decompress
client.succeed(f"{ENV_WITH_CREDS} nix copy --from '{store_url}' --no-check-sigs {PKGS['B']}")
verify_packages_in_store(client, PKGS['B'])
print(" ✓ Client decompressed .narinfo successfully")
@setup_s3()
def test_compression_mixed(bucket):
"""Test mixed compression (narinfo=xz, ls=gzip)"""
@@ -551,14 +537,18 @@ in
# Verify .narinfo has xz compression
verify_content_encoding(server, bucket, f"{pkg_hash}.narinfo", "xz")
print(" ✓ .narinfo has Content-Encoding: xz")
# Verify .ls has gzip compression
verify_content_encoding(server, bucket, f"{pkg_hash}.ls", "gzip")
print(" ✓ .ls has Content-Encoding: gzip")
# Verify client can download with mixed compression
client.succeed(f"{ENV_WITH_CREDS} nix copy --from '{store_url}' --no-check-sigs {PKGS['C']}")
verify_packages_in_store(client, PKGS['C'])
print(" ✓ Client downloaded package with mixed compression")
@setup_s3()
def test_compression_disabled(bucket):
"""Verify no compression by default"""
@@ -570,6 +560,8 @@ in
pkg_hash = get_package_hash(PKGS['A'])
verify_no_compression(server, bucket, f"{pkg_hash}.narinfo")
print(" ✓ No compression applied by default")
@setup_s3()
def test_nix_prefetch_url(bucket):
"""Test that nix-prefetch-url retrieves actual file content from S3, not empty files (issue #8862)"""
@@ -589,6 +581,8 @@ in
"nix hash file --type sha256 --base32 /tmp/test-file.txt"
).strip()
print(f" ✓ Uploaded test file to S3 ({test_file_size} bytes)")
# Use nix-prefetch-url to download from S3
s3_url = make_s3_url(bucket, path="/test-file.txt")
@@ -608,6 +602,8 @@ in
f"Hash mismatch: expected {expected_hash}, got {prefetch_hash}"
)
print(" ✓ nix-prefetch-url completed with correct hash")
# Verify the downloaded file is NOT empty (the bug in #8862)
file_size = int(client.succeed(f"stat -c %s {store_path}").strip())
@@ -619,12 +615,16 @@ in
f"File size mismatch: expected {test_file_size}, got {file_size}"
)
print(f" ✓ File has correct size ({file_size} bytes, not empty)")
# Verify actual content matches by comparing hashes instead of printing entire file
downloaded_hash = client.succeed(f"nix hash file --type sha256 --base32 {store_path}").strip()
if downloaded_hash != expected_hash:
raise Exception(f"Content hash mismatch: expected {expected_hash}, got {downloaded_hash}")
print(" ✓ File content verified correct (hash matches)")
@setup_s3(populate_bucket=[PKGS['A']], versioned=True)
def test_versioned_urls(bucket):
"""Test that versionId parameter is accepted in S3 URLs"""
@@ -638,6 +638,7 @@ in
f"{ENV_WITH_CREDS} nix eval --impure --expr "
f"'builtins.fetchurl {{ name = \"cache-info\"; url = \"{cache_info_url}\"; }}'"
)
print(" ✓ Fetch without versionId works")
# List versions to get a version ID
# MinIO output format: [timestamp] size tier versionId versionNumber method filename
@@ -651,6 +652,8 @@ in
raise Exception("Could not extract version ID from MinIO output")
version_id = version_match.group(1)
print(f" ✓ Found version ID: {version_id}")
# Version ID should not be "null" since versioning was enabled before upload
if version_id == "null":
raise Exception("Version ID is 'null' - versioning may not be working correctly")
@@ -661,6 +664,7 @@ in
f"{ENV_WITH_CREDS} nix eval --impure --expr "
f"'builtins.fetchurl {{ name = \"cache-info-versioned\"; url = \"{versioned_url}\"; }}'"
)
print(" ✓ Fetch with versionId parameter works")
@setup_s3()
def test_multipart_upload_basic(bucket):
@@ -696,9 +700,13 @@ in
print(output)
raise Exception(f"Expected '{expected_msg}' in output")
print(f" ✓ Multipart upload used with {expected_parts} parts")
client.succeed(f"{ENV_WITH_CREDS} nix copy --from '{store_url}' {large_pkg} --no-check-sigs")
verify_packages_in_store(client, large_pkg, should_exist=True)
print(" ✓ Large file downloaded and verified")
@setup_s3()
def test_multipart_threshold(bucket):
"""Test that files below threshold use regular upload"""
@@ -721,9 +729,13 @@ in
if "using S3 regular upload" not in output:
raise Exception("Expected regular upload to be used")
print(" ✓ Regular upload used for file below threshold")
client.succeed(f"{ENV_WITH_CREDS} nix copy --no-check-sigs --from '{store_url}' {PKGS['A']}")
verify_packages_in_store(client, PKGS['A'], should_exist=True)
print(" ✓ Small file uploaded and verified")
@setup_s3()
def test_multipart_with_log_compression(bucket):
"""Test multipart upload with compressed build logs"""
@@ -775,6 +787,8 @@ in
print(output)
raise Exception("Expected multipart completion message")
print(" ✓ Compressed log uploaded with multipart")
@setup_s3(
populate_bucket=[PKGS['A']],
profiles={
@@ -790,11 +804,13 @@ in
# Verify store info works with profile credentials (no env vars)
client.succeed(f"HOME=/root nix store info --store '{store_url}' >&2")
print(" ✓ nix store info works with profile credentials")
# Verify we can copy from the store using profile
verify_packages_in_store(client, PKGS['A'], should_exist=False)
client.succeed(f"HOME=/root nix copy --no-check-sigs --from '{store_url}' {PKGS['A']}")
verify_packages_in_store(client, PKGS['A'])
print(" ✓ nix copy works with profile credentials")
# Clean up the package we just copied so we can test invalid profile
client.succeed(f"nix store delete --ignore-liveness {PKGS['A']}")
@@ -803,6 +819,7 @@ in
# Verify invalid profile fails when trying to copy
invalid_url = make_s3_url(bucket, profile="invalid")
client.fail(f"HOME=/root nix copy --no-check-sigs --from '{invalid_url}' {PKGS['A']} 2>&1")
print(" ✓ Invalid profile credentials correctly rejected")
@setup_s3(
populate_bucket=[PKGS['A']],
@@ -824,12 +841,14 @@ in
output = client.succeed(
f"HOME=/root {ENV_WITH_CREDS} nix copy --no-check-sigs --debug --from '{store_url}' {PKGS['A']} 2>&1"
)
print(" ✓ nix copy succeeded with env vars overriding wrong profile")
# Verify the credential chain shows Environment provider was added
if "Added AWS Environment Credential Provider" not in output:
print("Debug output:")
print(output)
raise Exception("Expected Environment provider to be added to chain")
print(" ✓ Environment provider added to credential chain")
# Clean up the package so we can test again without env vars
client.succeed(f"nix store delete --ignore-liveness {PKGS['A']}")
@@ -837,6 +856,7 @@ in
# Without env vars, same URL should fail (proving profile creds are actually wrong)
client.fail(f"HOME=/root nix copy --no-check-sigs --from '{store_url}' {PKGS['A']} 2>&1")
print(" ✓ Without env vars, wrong profile credentials correctly fail")
@setup_s3(
populate_bucket=[PKGS['A']],
@@ -862,98 +882,14 @@ in
print("Debug output:")
print(output)
raise Exception(f"Expected to find: {msg}")
print(f" ✓ {provider} provider added to chain")
# SSO should be skipped (no SSO config for this profile)
if "Skipped AWS SSO Credential Provider for profile 'testprofile'" not in output:
print("Debug output:")
print(output)
raise Exception("Expected SSO provider to be skipped")
def test_virtual_hosted_copy():
"""Test nix copy with virtual-hosted-style addressing on custom endpoint"""
print("\n=== Testing Virtual-Hosted-Style Addressing ===")
# Use a fixed bucket name matching the static /etc/hosts entries
bucket = 'vhost-test'
server.succeed(f"mc mb minio/{bucket}")
try:
store_url = make_s3_url(
bucket,
endpoint=VHOST_ENDPOINT,
**{'addressing-style': 'virtual'}
)
# Upload with virtual-hosted-style, capture debug output
output = server.succeed(
f"{ENV_WITH_CREDS} nix copy --debug --to '{store_url}' {PKGS['A']} 2>&1"
)
# Verify virtual-hosted-style URL was used (bucket in hostname)
vhost_url_prefix = f"http://{bucket}.{VHOST_DOMAIN}:9000/"
if vhost_url_prefix not in output:
print("Debug output:")
print(output)
raise Exception(
f"Expected virtual-hosted-style URL containing '{vhost_url_prefix}'"
)
# Verify path-style URL was NOT used (bucket should not be in the path)
path_style_pattern = f"{VHOST_ENDPOINT}/{bucket}/"
if path_style_pattern in output:
print("Debug output:")
print(output)
raise Exception("Found path-style URL when virtual-hosted-style was expected")
# Download with virtual-hosted-style
verify_packages_in_store(client, PKGS['A'], should_exist=False)
output = client.succeed(
f"{ENV_WITH_CREDS} nix copy --debug --no-check-sigs "
f"--from '{store_url}' {PKGS['A']} 2>&1"
)
if vhost_url_prefix not in output:
print("Debug output:")
print(output)
raise Exception(
f"Expected virtual-hosted-style URL in download containing '{vhost_url_prefix}'"
)
verify_packages_in_store(client, PKGS['A'])
finally:
server.succeed(f"mc rb --force minio/{bucket}")
for pkg in PKGS.values():
client.succeed(f"[ ! -e {pkg} ] || nix store delete --ignore-liveness {pkg}")
@setup_s3()
def test_explicit_path_style(bucket):
"""Test that addressing-style=path works as backwards-compatible fallback"""
print("\n=== Testing Explicit Path-Style Addressing ===")
store_url = make_s3_url(
bucket,
**{'addressing-style': 'path'}
)
# Upload with explicit path-style
output = server.succeed(
f"{ENV_WITH_CREDS} nix copy --debug --to '{store_url}' {PKGS['A']} 2>&1"
)
# Verify path-style URL was used (bucket in path, not hostname)
path_style_pattern = f"{ENDPOINT}/{bucket}/"
if path_style_pattern not in output:
print("Debug output:")
print(output)
raise Exception(
f"Expected path-style URL containing '{path_style_pattern}'"
)
# Download
verify_packages_in_store(client, PKGS['A'], should_exist=False)
client.succeed(
f"{ENV_WITH_CREDS} nix copy --no-check-sigs --from '{store_url}' {PKGS['A']}"
)
verify_packages_in_store(client, PKGS['A'])
print(" ✓ SSO provider correctly skipped (not configured)")
# ============================================================================
# Main Test Execution
@@ -973,7 +909,6 @@ in
# Run tests (each gets isolated bucket via decorator)
test_credential_caching()
test_aws_log_integration()
test_fetchurl_basic()
test_error_message_formatting()
test_fork_credential_preresolution()
@@ -992,7 +927,9 @@ in
test_profile_credentials()
test_env_vars_precedence()
test_credential_provider_chain()
test_virtual_hosted_copy()
test_explicit_path_style()
print("\n" + "="*80)
print("✓ All S3 Binary Cache Store Tests Passed!")
print("="*80)
'';
}