Commit Graph
22331 Commits
Author SHA1 Message Date
internal-nix-ci[bot] 9bf642b88d Merge pull request #15097 from NixOS/backport-15059-to-2.33-maintenance
[Backport 2.33-maintenance] feat(libstore/aws-creds): route AWS CRT logs through Nix logger
2.33.2
2026-01-27 14:43:13 +00:00
Sergei Zimmerman e96a3f7a73 tests/nixos/s3-binary-cache-store: Drop superfluous prints
As requested in review.

(cherry picked from commit e3b788b4ca)
2026-01-27 14:01:44 +00:00
Bernardo Meurer Costa d297aeca2c feat(libstore/aws-creds): route AWS CRT logs through Nix logger
Previously AWS CRT logs went directly to stderr via ApiHandle::InitializeLogging,
causing log spam that didn't respect Nix's verbosity settings.

This implements a custom aws_logger using the aws-c-common C API that:
- Routes all AWS logs through nix::logger
- Maps AWS log levels conservatively (ERROR/WARN -> lvlInfo) since the SDK
  treats expected conditions like missing IMDS as errors
- Prefixes messages with (aws) for clarity
- Respects Nix's verbosity flags (-v, -vv, etc.)

(cherry picked from commit 3b8b764e29)
2026-01-27 14:01:44 +00:00
internal-nix-ci[bot] 534de0df4e Merge pull request #15084 from NixOS/backport-14645-to-2.33-maintenance
[Backport 2.33-maintenance] feat(libstore): add AWS SSO support for S3 authentication
2026-01-26 18:44:56 +00:00
Jörg Thalheim 65c7ec71ce fix(libstore/aws-creds): respect AWS_PROFILE environment variable
The SSO provider was unconditionally setting profile_name_override to
the (potentially empty) profile string from the S3 URL. When profile
was empty, this prevented the AWS CRT SDK from falling back to the
AWS_PROFILE environment variable.

Only set profile_name_override when a profile is explicitly specified
in the URL, allowing the SDK's built-in AWS_PROFILE handling to work.

(cherry picked from commit 453dbab1e8)
2026-01-26 18:03:26 +00:00
Bernardo Meurer 50aecffbf9 test(s3-binary-cache-store): test profiles and provider chain
(cherry picked from commit 71bdb33a36)
2026-01-26 18:03:26 +00:00
Bernardo Meurer 2d85a6ba32 test(s3-binary-cache-store): clear credential cache between tests
(cherry picked from commit 0595c5f7ee)
2026-01-26 18:03:26 +00:00
Bernardo Meurer 73a0311ef8 test(s3-binary-cache-store): add profile support for setup_for_s3
(cherry picked from commit 11f108d898)
2026-01-26 18:03:26 +00:00
Bernardo Meurer cb1c413db8 chore(libstore/aws-creds): remove unused includes
(cherry picked from commit 128b2b5c56)
2026-01-26 18:03:26 +00:00
Bernardo Meurer 8a4d8aabb2 fix(libstore/aws-creds): add STS support for default profile
The default (empty) profile case was using CreateCredentialsProviderChainDefault
which didn't properly support role_arn/source_profile based role assumption via
STS because TLS context wasn't being passed to the Profile provider.

This change unifies the credential chain for all profiles (default and named),
ensuring:
- Consistent behavior between default and named profiles
- Proper TLS context is passed for STS operations
- SSO support works for both cases

(cherry picked from commit 508d4463e5)
2026-01-26 18:03:26 +00:00
Bernardo Meurer 344a0eaed1 refactor(libstore/aws-creds): improve error handling and logging
Add validation for TLS context and client bootstrap initialization,
with appropriate error messages when these fail. The TLS context failure
is now a warning that gracefully disables SSO, while bootstrap failure
throws since it's required for all providers.

(cherry picked from commit 3c8e45c061)
2026-01-26 18:03:26 +00:00
Jörg Thalheim 3034589047 libstore: add AWS SSO support for S3 authentication
This enables seamless AWS SSO authentication for S3 binary caches
without requiring users to manually export credentials.

This adds SSO support by calling aws_credentials_provider_new_sso() from
the C library directly. It builds a custom credential chain: Env → SSO →
Profile → IMDS

The SSO provider requires a TLS context for HTTPS connections to SSO
endpoints, which is created once and shared across all providers.

(cherry picked from commit ec91479076)
2026-01-26 18:03:26 +00:00
John Ericson a2b044dc8d Merge pull request #15078 from xokdvium/backport-15072-to-2.33-maintenance
[Backport 2.33-maintenance] Fix destruction of DerivationBuilder implementations
2026-01-24 19:07:48 -05:00
Sergei Zimmerman 70ecd8c8a9 Fix destruction of DerivationBuilder implementations
This unsures that we call the correct virtual functions when destroying a particular
DerivationBuilder.

Usually the order of destructors is in the reverse order of inheritance:

ChrootLinuxDerivationBuilder -> ChrootDerivationBuilder -> DerivationBuilderImpl

autoDelChroot was being destroyed before the DerivationBuilderImpl::killChild was
run and it would fail to clean up the chroot directory, since there were still processes
writing to it. Note that ChrootLinuxDerivationBuilder::killSandbox was never run in
the interrupted case at all, since virtual functions in destructors do not call derived class
methods.

I could reproduce the issue with the following derivation:

let
  pkgs = import <nixpkgs> { };
in
pkgs.runCommand "chroot-cleanup-race" { } ''
  mkdir -p $out

  for i in $(seq 1 200); do
    (
      mkfifo $out/fifo$i
      cat $out/fifo$i > /dev/null &

      while true; do
        : > $out/file$i
      done
    ) &
  done

  sleep 0.05
  echo done > $out/main
''

While interrupting it manually when it would hang.

Wrapping the unique pointer in a custom deleter function we can run all
of the necessary clean up code consistently and calling the right virtual
functions. Ideally we'd have a lint that bans the usage of virtual functions
in destructors completely.

(cherry picked from commit b752c5cb64)
2026-01-25 02:16:40 +03:00
internal-nix-ci[bot] a7276a24b9 Merge pull request #15073 from NixOS/backport-15071-to-2.33-maintenance
[Backport 2.33-maintenance] tests: fix sandbox-paths in cancelled-builds test
2026-01-24 21:03:42 +00:00
Robert Hensing 48bf9a8e50 tests: fix sandbox-paths in cancelled-builds test
Don't add the whole store to sandbox-paths unconditionally. Exposing
the entire store defeats the purpose of sandboxing, and when the test
store is the same as the system store (NixOS VM), it causes an obscure
"Permission denied" error.

Only add sandbox-paths when not on NixOS, indicating a separate test
store that needs access to system store build tools.

(cherry picked from commit 7b4444f174)
2026-01-24 20:26:44 +00:00
internal-nix-ci[bot] 2d879efee0 Merge pull request #15068 from NixOS/backport-14972-to-2.33-maintenance
[Backport 2.33-maintenance] Fix concurrent builder failure empty message bugs
2026-01-23 02:16:00 +00:00
Robert Hensing 2ec62f1974 DerivationTrampolineGoal: improve error message wording
Change "cannot build missing derivation" to "failed to obtain derivation of"
since the path (e.g. '...drv^out') is a derivation output, not a derivation.

The message could be improved further to resolve ambiguity when multiple
outputOf links are involved, but for now we err on the side of brevity
since this message is already merged into larger error messages with
other context from the Worker and CLI.

(cherry picked from commit 3c3ceb18e9)
2026-01-23 01:34:46 +00:00
Robert Hensing 09884e2c1a buildPathsWithResults: don't report cancelled goals as failures
When !keepGoing and a goal fails, other goals are cancelled and
remain with exitCode == ecBusy. These cancelled goals have a default
BuildResult::Failure{} with empty errorMsg.

Previously, buildPathsWithResults would return these cancelled goals,
and throwBuildErrors would report them as failures. When only one such
cancelled goal was present, it would throw an error with an empty
message like:

    error: build of '/nix/store/...drv^*' failed:

Now we skip goals with ecBusy since their state is indeterminate.
Cancelled goals could be reported, but this keeps the output relevant.
Other indeterminate goal states were already not being reported, for
instance: derivations that weren't started for being blocked on a
concurrency limit, or blocked on a currently building dependency.

(cherry picked from commit 68f549def4)
2026-01-23 01:34:46 +00:00
Robert Hensing 753bf479f9 tests: don't expect cancelled goals to be reported as failures
When keepGoing=false and a build fails, other goals are cancelled.
Previously, these cancelled goals were reported in the "build of ...
failed" error message alongside actual failures. This was misleading
since cancelled goals didn't actually fail - they were never tried.

Update the test to expect only the actual failure (hash mismatch) to
be reported, not the cancelled goals.

(cherry picked from commit 3fd85c7d64)
2026-01-23 01:34:46 +00:00
Robert Hensing a6c201e039 DerivationTrampolineGoal: use doneFailure to set buildResult
DerivationTrampolineGoal is the top-level goal whose buildResult is
returned by buildPathsWithResults. When it failed without setting
buildResult.inner, buildPathsWithResults would return failures with
empty errorMsg, producing error messages like:

  error: failed to build attribute 'checks.x86_64-linux.foo',
  build of '/nix/store/...drv^*' failed:

(note the empty message after "failed:")

Use the new doneFailure helper to ensure buildResult is populated
with meaningful error information.

(cherry picked from commit 25eb07a91b)
2026-01-23 01:34:46 +00:00
Robert Hensing 1d39afac38 Goal: add doneSuccess/doneFailure helpers to base class
Add helpers to the base Goal class that set buildResult and call amDone,
ensuring buildResult is always populated when a goal terminates.

Derived class helpers now call the base class versions. This reorders
operations: previously buildResult was set before bookkeeping (counter
resets, worker stats), now it's set after. This is safe because the
bookkeeping code (mcExpectedBuilds.reset(), worker.doneBuilds++,
worker.updateProgress(), etc.) only accesses worker counters, not
buildResult.

(cherry picked from commit cb2ade20d4)
2026-01-23 01:34:46 +00:00
internal-nix-ci[bot] ba42159b63 Merge pull request #15067 from NixOS/backport-15062-to-2.33-maintenance
[Backport 2.33-maintenance] ci: Drop magic-nix-cache
2026-01-23 01:10:11 +00:00
Sergei Zimmerman f42de47a40 ci: Drop magic-nix-cache
We are now seeing. I guess we are out with the cache. When the API responds with 418 (I'm a teapot)
it seems like the only reasonable solution is to oblige.

error: unable to download 'http://127.0.0.1:37515/7ms9f25xyxavf32pvdc3vb28nzzmkbn3.narinfo': HTTP error 418
       response body:
       GitHub API error: GitHub Actions Cache throttled Magic Nix Cache. Not trying to use it again on this run.
(cherry picked from commit dae41e06e8)
2026-01-23 00:35:25 +00:00
internal-nix-ci[bot] a569ebca7e Merge pull request #15057 from NixOS/backport-15047-to-2.33-maintenance
[Backport 2.33-maintenance] fix(libstore/filetransfer): restart source before upload retries
2026-01-22 21:07:05 +00:00
Bernardo Meurer Costa 7808b682bb fix(libstore/filetransfer): restart source before upload retries
When an upload fails with a transient HTTP error (e.g., S3 rate limiting
with HTTP 503), retries would fail with "curl error: Failed to open/read
local data from file/application" because the upload source was already
exhausted from the previous attempt.

Restart the source in init() to ensure it's at the beginning for both
first attempts (no-op) and retries (necessary fix).

Fixes: #15023
(cherry picked from commit fbd787b910)
2026-01-22 20:28:22 +00:00
John Ericson 7672220083 Merge pull request #15052 from lovesegfault/backport-15048-to-2.33-maintenance
[Backport 2.33-maintenance] fix(libstore/filetransfer): skip Accept-Encoding header for S3 SigV4 requests
2026-01-22 14:03:07 -05:00
Bernardo Meurer Costa ed28ceb12f fix(libstore/filetransfer): skip Accept-Encoding header for S3 SigV4 requests
Some S3-compatible services (like GCS) modify the Accept-Encoding header
in transit, which breaks AWS SigV4 signature verification since curl's
implementation signs all headers including Accept-Encoding.

Fixes: #15019
(cherry picked from commit fcfa1dc8ab)
2026-01-22 18:22:16 +00:00
internal-nix-ci[bot] f99073b646 Merge pull request #15036 from NixOS/backport-15031-to-2.33-maintenance
[Backport 2.33-maintenance] ci: Bump magic-nix-cache to disable on 429
2026-01-21 05:28:36 +00:00
Sergei Zimmerman fbffd5683e ci: Bump magic-nix-cache to disable on 429
(cherry picked from commit 1555677cd5)
2026-01-21 04:50:18 +00:00
internal-nix-ci[bot] ad5cd0b9f3 Merge pull request #15012 from NixOS/backport-15011-to-2.33-maintenance
[Backport 2.33-maintenance] libutil: fix `linux` build on fresh `glibc` and `gcc`
2026-01-17 18:47:08 +00:00
Sergei Trofimovich c0c13d7323 libutil: fix linux build on fresh glibc and gcc
Without the change the build fails for me as:

    ../unix/file-descriptor.cc:404:70: error: 'RESOLVE_BENEATH' was not declared in this scope
      404 |         dirFd, path.rel_c_str(), flags, static_cast<uint64_t>(mode), RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS);
          |                                                                      ^~~~~~~~~~~~~~~

This happens for 2 reasons:
1. `__NR_openat2` constant was not pulled in from the according headers
   and as a result `<linux/openat2.h>` was not included.
2. `define HAVE_OPENAT2 0` build is broken: refers to missing
   `RESOLVE_BENEATH` normally pulled in from `<linux/openat2.h>`

This changes fixes both.

(cherry picked from commit 3256aba6a2)
2026-01-17 18:01:49 +00:00
Sergei Zimmerman c272697224 Bump version 2026-01-15 23:37:30 +03:00
internal-nix-ci[bot] fb562abba9 Merge pull request #14977 from NixOS/backport-14961-to-2.33-maintenance
[Backport 2.33-maintenance] libutil/union-source-accessor: Barf on non-existent directories
2.33.1
2026-01-11 19:46:37 +00:00
Sergei Zimmerman a77d7b5251 libutil/union-source-accessor: Barf on non-existent directories
Previously builtins.readDir would return an empty attribute set
instead of barfing on non-existent paths. This is a regression from
2.32 for impure eval.

(cherry picked from commit 4ab2cdacfc)
2026-01-11 19:02:58 +00:00
internal-nix-ci[bot] e12aca79fd Merge pull request #14908 from NixOS/backport-14903-to-2.33-maintenance
[Backport 2.33-maintenance] upload-release.pl: Fix up nix-channels bucket location, use awscli2
2026-01-01 22:05:36 +00:00
Sergei Zimmerman 0ea6142757 upload-release.pl: Fix up nix-channels bucket location, use awscli2
I messed up and accidentally configured the S3 client to use the same
host as the nix-releases bucket, but nix-channels is us-east-1 and
nix-releases is eu-west-1.

(cherry picked from commit 0900638f1d)
2026-01-01 21:17:51 +00:00
internal-nix-ci[bot] d6d867582e Merge pull request #14902 from NixOS/backport-14888-to-2.33-maintenance
[Backport 2.33-maintenance] ci: GitHub releng for release automation
2026-01-01 15:22:38 +00:00
Sergei Zimmerman 6e098682bd release-process: Document usage of upload-release.yml workflow
(cherry picked from commit 84ff2ef347)
2026-01-01 14:45:35 +00:00
Sergei Zimmerman 9b49b5c050 upload-release: Only upload the newly created tag
(cherry picked from commit 3933e45d52)
2026-01-01 14:45:35 +00:00
Sergei Zimmerman 1e6dad7e2f upload-release: Also push to GHCR as part of the release process
(cherry picked from commit a1569458cc)
2026-01-01 14:45:35 +00:00
Sergei Zimmerman e999426f05 ci: Add upload-release.yml
This workflow is supposed to automate release uploads by using OIDC
for AWS setup. DockerHub still uses long-lived credentials, but that's
not fixable. In a follow-up we could set up release uploads to GHCR too.

(cherry picked from commit 4599daa10e)
2026-01-01 14:45:35 +00:00
Sergei Zimmerman 32635e4449 maintainers: Document git tag signing
Previously it was only Eeclo doing releases that were signed with
B541D55301270E0BCF15CA5D8170B4726D7198DE. Other linux distributions
have the expectation (rightfully so) that our tags are signed. Let's
document this.

We could do cross-signing to make tracing the chain of trust easier
for all Nix team members [1].

[1]: https://nixos.org/community/teams/nix/

(cherry picked from commit 6cb8b58a47)
2026-01-01 14:45:35 +00:00
Sergei Zimmerman bb07a0a222 maintainers/upload-release.pl: Make more configurable
This allows for testing with a local minio deployment like:

./upload-release.pl --skip-docker --skip-git --s3-endpoint http://localhost:9000 --s3-host localhost:9000 1821360

(cherry picked from commit d19b8d5f99)
2026-01-01 14:45:35 +00:00
internal-nix-ci[bot] 4c6a9cf2f7 Merge pull request #14886 from NixOS/backport-14872-to-2.33-maintenance
[Backport 2.33-maintenance] ci: Move docker_push_image into a separate workflow
2025-12-28 23:26:56 +00:00
Sergei Zimmerman d042065a6d ci: Make docker-push workflow more configurable
This should allow reusing this workflow (with more tweaks)
in the releng workflow.

(cherry picked from commit c867ed6726)
2025-12-28 22:39:58 +00:00
Sergei Zimmerman a6c7082103 ci: Pin actions in docker-push reusable workflow
(cherry picked from commit fb05f6de0d)
2025-12-28 22:39:58 +00:00
Sergei Zimmerman 6e837f6554 ci: Move docker_push_image into a separate workflow
Best reviewed with -w --color-moved. This just moves the code
into a separate workflow. This will allow us to reuse it in
the release job for github releng of releases.

(cherry picked from commit 745983dfc0)
2025-12-28 22:39:58 +00:00
Sergei Zimmerman b89f9c77cb Merge pull request #14878 from NixOS/backport-14874-to-2.33-maintenance
[Backport 2.33-maintenance] ci: Run flake-regressions also with the newly built daemon
2025-12-29 00:05:08 +03:00
Sergei Zimmerman c9ec76276d ci: Pin download-artifact actions sha
Also bumps download-artifact to v7.0.0.

(cherry picked from commit c54af23b41)
2025-12-28 18:19:42 +03:00