Compare commits

..
Author SHA1 Message Date
Eelco Dolstra 4816b640a2 Fix HUP detection on macOS
As of macOS 15.4, passing 0 or POLLHUP doesn't seem to work at all for
sockets any more (though it does work for `notifyPipe`). As a
workaround, also pass POLLIN. That does cause us to receive a bunch of
POLLIN events we don't care about, so we sleep for a bit when receiving
POLLIN.
2025-08-27 21:11:05 +02:00
John Ericson 79211b6110 Merge pull request #13846 from obsidiansystems/derivation-builder-params-aggregate-initialize
No more `DerivationBuilderParams:` constructor!
2025-08-27 12:30:49 -04:00
John Ericson f5f9e32f54 No more DerivationBuilderParams: constructor!
I am not sure how/why this started working. C++23?
2025-08-27 11:40:02 -04:00
Jörg Thalheim 564593bcb9 Merge pull request #13837 from xokdvium/bump-nixpkgs
flake: Update nixpkgs
2025-08-27 09:33:04 +02:00
Sergei Zimmerman 8ee74792fe Merge pull request #13819 from obsidiansystems/relative-url
Implement `parseURLRelative`, use in `HttpBinaryCacheStore`
2025-08-27 03:34:57 +03:00
John Ericson e82210b3b2 Implement parseURLRelative, use in HttpBinaryCacheStore
This allows us to replace some very hacky and not correct string
concatentation in `HttpBinaryCacheStore`. It will especially be useful
with #13752, when today's hacks started to cause problems in practice,
not just theory.

Also make `fixGitURL` returned a `ParsedURL`.
2025-08-26 19:45:10 -04:00
Sergei Zimmerman 625477a7df flake: Update nixpkgs
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/cd32a774ac52caaa03bcfc9e7591ac8c18617ced?narHash=sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI%3D' (2025-08-17)
  → 'github:NixOS/nixpkgs/d98ce345cdab58477ca61855540999c86577d19d?narHash=sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8%3D' (2025-08-26)

This update contains d1266642a8722f2a05e311fa151c1413d2b9653c, which
is necessary for the TOML timestamps to get tested via nixpkgsLibTests job.
2025-08-27 02:23:05 +03:00
Sergei Zimmerman 231f3af535 Merge pull request #13835 from obsidiansystems/better-string-split
Better `stringSplit`
2025-08-27 01:20:46 +03:00
John EricsonandSergei Zimmerman cc4aa70e6e Better stringSplit
I need this for some `ParseURL` improvements, but I figure this is
better to send as its own PR.

I changed the tests willy-nilly to sometimes use
`std::list<std::string_view>` instead of `Strings` (which is
`std::list<std::string>`).

Co-Authored-By: Sergei Zimmerman <sergei@zimmerman.foo>
2025-08-26 18:03:23 -04:00
John Ericson 0bd9d6a28e Merge pull request #13832 from kip93/fix/empty-ports
Handle empty ports with new URL parsing
2025-08-26 13:55:59 -04:00
Leandro Reina 7989e3192d Handle empty ports 2025-08-26 17:41:27 +02:00
Robert Hensing 1e16a54ee5 Merge pull request #13828 from NixOS/readme-meeting-times
Update work meeting time in README
2025-08-26 01:37:07 +02:00
Robert Hensing afade27123 Update work meeting time in README 2025-08-26 00:50:12 +02:00
Robert Hensing c1e2396d58 Merge pull request #13826 from xokdvium/sqlite-zfs-hack
SQLite: fsync db.sqlite-shm before opening the database
2025-08-26 00:05:40 +02:00
John Ericson ca94905593 Merge pull request #13825 from obsidiansystems/slight-optimize-s3ToHttpsURL
`ParsedS3URL::toHttpsUrl` Slight optimize
2025-08-25 17:51:09 -04:00
Eelco Dolstra e492c64c8e SQLite: fsync db.sqlite-shm before opening the database
This is a workaround for https://github.com/NixOS/nix/issues/13515
(opening the SQLite DB randomly taking a couple of seconds on ZFS).

(cherry picked from commit a7fceb5eec)
2025-08-26 00:42:18 +03:00
John Ericson e4e8a615fa ParsedS3URL::toHttpsUrl Slight optimize
I didn't want to block that PR on further code review while I figured
out these new (to us) C++23 goodies.
2025-08-25 16:53:39 -04:00
John Ericson fac34ad20f Merge pull request #13824 from xokdvium/fix-formatting
libexpr: Fix weird formatting after treewide reformat
2025-08-25 15:26:17 -04:00
John Ericson 024d3954af Merge pull request #13823 from lovesegfault/extract-s3ToHttpsURL
feat(libstore/s3): add toHttpsUrl
2025-08-25 15:11:38 -04:00
Sergei Zimmerman f0e4af4365 libexpr: Fix weird formatting after treewide reformat 2025-08-25 22:09:18 +03:00
Bernardo Meurer Costa 5985d67906 feat(libstore/s3): add toHttpsUrl
This is extracted from the work in #13752
2025-08-25 18:48:19 +00:00
Eelco Dolstra 9bee0fa6ac Merge pull request #13822 from NixOS/bump-2.32.0
Bump version to 2.32
2025-08-25 17:22:57 +02:00
Eelco Dolstra adec28bf85 Update release-process.md 2025-08-25 10:30:21 +02:00
Eelco Dolstra f5e09d9b58 Update mergify.yml 2025-08-25 10:28:47 +02:00
Eelco Dolstra f67daa4a87 Bump version 2025-08-25 10:27:46 +02:00
John Ericson c9211b0b2d Merge pull request #13803 from obsidiansystems/more-parsed-urls
Make more URLs parsed, most notably `FileTransferRequest::url`
2025-08-23 10:54:39 -04:00
John Ericson 2fa2c0b09f Merge pull request #13812 from obsidiansystems/url-parse-leniency
Limit to lenient parsing of non-standard URLs only where needed
2025-08-23 10:53:59 -04:00
Jörg Thalheim ebf1cf5227 Merge pull request #13807 from roberth/release-notes-todo
maintainers: Add script for release notes todo list
2025-08-23 08:52:22 +02:00
John Ericson 3e86d75c9d Make more URLs parsed, most notably FileTransferRequest::url
Trying to gradually replace the use of strings with better types in ways
that makes sense.
2025-08-22 12:42:48 -04:00
John Ericson 72a548ed6a Limit to lenient parsing of non-standard URLs only where needed
This allows us to put `parseURL` in more spots without furthering
technical debt.
2025-08-22 12:37:37 -04:00
John Ericson 4083eff0c0 decodeQuery Take std::string_view not string ref 2025-08-22 12:26:48 -04:00
Robert Hensing a1b3934a78 maintainers: Add script for release notes todo list 2025-08-21 14:19:22 +02:00
130 changed files with 1232 additions and 1630 deletions

No files matched your search

@@ -4,29 +4,15 @@ inputs:
dogfood:
description: "Whether to use Nix installed from the latest artifact from master branch"
required: true # Be explicit about the fact that we are using unreleased artifacts
experimental-installer:
description: "Whether to use the experimental installer to install Nix"
default: false
experimental-installer-version:
description: "Version of the experimental installer to use. If `latest`, the newest artifact from the default branch is used."
# TODO: This should probably be pinned to a release after https://github.com/NixOS/experimental-nix-installer/pull/49 lands in one
default: "latest"
extra_nix_config:
description: "Gets appended to `/etc/nix/nix.conf` if passed."
install_url:
description: "URL of the Nix installer"
required: false
default: "https://releases.nixos.org/nix/nix-2.30.2/install"
tarball_url:
description: "URL of the Nix tarball to use with the experimental installer"
required: false
github_token:
description: "Github token"
required: true
use_cache:
description: "Whether to setup magic-nix-cache"
default: true
required: false
runs:
using: "composite"
steps:
@@ -51,81 +37,14 @@ runs:
gh run download "$RUN_ID" --repo "$DOGFOOD_REPO" -n "$INSTALLER_ARTIFACT" -D "$INSTALLER_DOWNLOAD_DIR"
echo "installer-path=file://$INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
TARBALL_PATH="$(find "$INSTALLER_DOWNLOAD_DIR" -name 'nix*.tar.xz' -print | head -n 1)"
echo "tarball-path=file://$TARBALL_PATH" >> "$GITHUB_OUTPUT"
echo "::notice ::Dogfooding Nix installer from master (https://github.com/$DOGFOOD_REPO/actions/runs/$RUN_ID)"
env:
GH_TOKEN: ${{ inputs.github_token }}
DOGFOOD_REPO: "NixOS/nix"
- name: "Gather system info for experimental installer"
shell: bash
if: ${{ inputs.experimental-installer == 'true' }}
run: |
echo "::notice Using experimental installer from $EXPERIMENTAL_INSTALLER_REPO (https://github.com/$EXPERIMENTAL_INSTALLER_REPO)"
if [ "$RUNNER_OS" == "Linux" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="linux"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
elif [ "$RUNNER_OS" == "macOS" ]; then
EXPERIMENTAL_INSTALLER_SYSTEM="darwin"
echo "EXPERIMENTAL_INSTALLER_SYSTEM=$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_OS: $RUNNER_OS"
exit 1
fi
if [ "$RUNNER_ARCH" == "X64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=x86_64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
elif [ "$RUNNER_ARCH" == "ARM64" ]; then
EXPERIMENTAL_INSTALLER_ARCH=aarch64
echo "EXPERIMENTAL_INSTALLER_ARCH=$EXPERIMENTAL_INSTALLER_ARCH" >> "$GITHUB_ENV"
else
echo "::error ::Unsupported RUNNER_ARCH: $RUNNER_ARCH"
exit 1
fi
echo "EXPERIMENTAL_INSTALLER_ARTIFACT=nix-installer-$EXPERIMENTAL_INSTALLER_ARCH-$EXPERIMENTAL_INSTALLER_SYSTEM" >> "$GITHUB_ENV"
env:
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- name: "Download latest experimental installer"
shell: bash
id: download-latest-experimental-installer
if: ${{ inputs.experimental-installer == 'true' && inputs.experimental-installer-version == 'latest' }}
run: |
RUN_ID=$(gh run list --repo "$EXPERIMENTAL_INSTALLER_REPO" --workflow ci.yml --branch main --status success --json databaseId --jq ".[0].databaseId")
EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR="$GITHUB_WORKSPACE/$EXPERIMENTAL_INSTALLER_ARTIFACT"
mkdir -p "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
gh run download "$RUN_ID" --repo "$EXPERIMENTAL_INSTALLER_REPO" -n "$EXPERIMENTAL_INSTALLER_ARTIFACT" -D "$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR"
# Executable permissions are lost in artifacts
find $EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR -type f -exec chmod +x {} +
echo "installer-path=$EXPERIMENTAL_INSTALLER_DOWNLOAD_DIR" >> "$GITHUB_OUTPUT"
env:
GH_TOKEN: ${{ inputs.github_token }}
EXPERIMENTAL_INSTALLER_REPO: "NixOS/experimental-nix-installer"
- uses: cachix/install-nix-action@c134e4c9e34bac6cab09cf239815f9339aaaf84e # v31.5.1
if: ${{ inputs.experimental-installer != 'true' }}
with:
# Ternary operator in GHA: https://www.github.com/actions/runner/issues/409#issuecomment-752775072
install_url: ${{ inputs.dogfood == 'true' && format('{0}/install', steps.download-nix-installer.outputs.installer-path) || inputs.install_url }}
install_options: ${{ inputs.dogfood == 'true' && format('--tarball-url-prefix {0}', steps.download-nix-installer.outputs.installer-path) || '' }}
extra_nix_config: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/nix-installer-action@786fff0690178f1234e4e1fe9b536e94f5433196 # v20
if: ${{ inputs.experimental-installer == 'true' }}
with:
diagnostic-endpoint: ""
# TODO: It'd be nice to use `artifacts.nixos.org` for both of these, maybe through an `/experimental-installer/latest` endpoint? or `/commit/<hash>`?
local-root: ${{ inputs.experimental-installer-version == 'latest' && steps.download-latest-experimental-installer.outputs.installer-path || '' }}
source-url: ${{ inputs.experimental-installer-version != 'latest' && 'https://artifacts.nixos.org/experimental-installer/tag/${{ inputs.experimental-installer-version }}/${{ env.EXPERIMENTAL_INSTALLER_ARTIFACT }}' || '' }}
nix-package-url: ${{ inputs.dogfood == 'true' && steps.download-nix-installer.outputs.tarball-path || (inputs.tarball_url || '') }}
extra-conf: ${{ inputs.extra_nix_config }}
- uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 # v13
if: ${{ inputs.use_cache == 'true' }}
with:
diagnostic-endpoint: ''
use-flakehub: false
use-gha-cache: true
source-revision: 92d9581367be2233c2d5714a2640e1339f4087d8 # main
+71 -11
View File
@@ -27,7 +27,6 @@ jobs:
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
use_cache: false
- run: nix flake show --all-systems --json
tests:
@@ -66,6 +65,7 @@ jobs:
dogfood: ${{ github.event_name == 'workflow_dispatch' && inputs.dogfood || github.event_name != 'workflow_dispatch' }}
# The sandbox would otherwise be disabled by default on Darwin
extra_nix_config: "sandbox = true"
- uses: DeterminateSystems/magic-nix-cache-action@main
# Since ubuntu 22.30, unprivileged usernamespaces are no longer allowed to map to the root user:
# https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
- run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
@@ -140,20 +140,78 @@ jobs:
- run: exec bash -c "nix-channel --add https://releases.nixos.org/nixos/unstable/nixos-23.05pre466020.60c1d71f2ba nixpkgs"
- run: exec bash -c "nix-channel --update && nix-env -iA nixpkgs.hello && hello"
# Steps to test CI automation in your own fork.
# 1. Sign-up for https://hub.docker.com/
# 2. Store your dockerhub username as DOCKERHUB_USERNAME in "Repository secrets" of your fork repository settings (https://github.com/$githubuser/nix/settings/secrets/actions)
# 3. Create an access token in https://hub.docker.com/settings/security and store it as DOCKERHUB_TOKEN in "Repository secrets" of your fork
check_secrets:
permissions:
contents: none
name: Check presence of secrets
runs-on: ubuntu-24.04
outputs:
docker: ${{ steps.secret.outputs.docker }}
steps:
- name: Check for DockerHub secrets
id: secret
env:
_DOCKER_SECRETS: ${{ secrets.DOCKERHUB_USERNAME }}${{ secrets.DOCKERHUB_TOKEN }}
run: |
echo "docker=${{ env._DOCKER_SECRETS != '' }}" >> $GITHUB_OUTPUT
docker_push_image:
name: Push docker image to DockerHub and GHCR
needs: [tests, vm_tests]
if: github.event_name == 'push' && github.ref_name == 'master'
uses: ./.github/workflows/docker-push.yml
with:
ref: ${{ github.sha }}
is_master: true
needs: [tests, vm_tests, check_secrets]
permissions:
contents: read
packages: write
secrets:
DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
if: >-
needs.check_secrets.outputs.docker == 'true' &&
github.event_name == 'push' &&
github.ref_name == 'master'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- uses: cachix/install-nix-action@v31
with:
install_url: https://releases.nixos.org/nix/nix-2.20.3/install
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: echo NIX_VERSION="$(nix --experimental-features 'nix-command flakes' eval .\#nix.version | tr -d \")" >> $GITHUB_ENV
- run: nix --experimental-features 'nix-command flakes' build .#dockerImage -L
- run: docker load -i ./result/image.tar.gz
- run: docker tag nix:$NIX_VERSION ${{ secrets.DOCKERHUB_USERNAME }}/nix:$NIX_VERSION
- run: docker tag nix:$NIX_VERSION ${{ secrets.DOCKERHUB_USERNAME }}/nix:master
# We'll deploy the newly built image to both Docker Hub and Github Container Registry.
#
# Push to Docker Hub first
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- run: docker push ${{ secrets.DOCKERHUB_USERNAME }}/nix:$NIX_VERSION
- run: docker push ${{ secrets.DOCKERHUB_USERNAME }}/nix:master
# Push to GitHub Container Registry as well
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Push image
run: |
IMAGE_ID=ghcr.io/${{ github.repository_owner }}/nix
# Change all uppercase to lowercase
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
docker tag nix:$NIX_VERSION $IMAGE_ID:$NIX_VERSION
docker tag nix:$NIX_VERSION $IMAGE_ID:latest
docker push $IMAGE_ID:$NIX_VERSION
docker push $IMAGE_ID:latest
# deprecated 2024-02-24
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
vm_tests:
runs-on: ubuntu-24.04
@@ -196,6 +254,7 @@ jobs:
extra_nix_config:
experimental-features = nix-command flakes
github_token: ${{ secrets.GITHUB_TOKEN }}
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: nix build -L --out-link ./new-nix && PATH=$(pwd)/new-nix/bin:$PATH MAX_FLAKES=25 flake-regressions/eval-all.sh
profile_build:
@@ -216,6 +275,7 @@ jobs:
extra_nix_config: |
experimental-features = flakes nix-command ca-derivations impure-derivations
max-jobs = 1
- uses: DeterminateSystems/magic-nix-cache-action@main
- run: |
nix build -L --file ./ci/gha/profile-build buildTimeReport --out-link build-time-report.md
cat build-time-report.md >> $GITHUB_STEP_SUMMARY
-101
View File
@@ -1,101 +0,0 @@
name: "Push Docker Image"
on:
workflow_call:
inputs:
ref:
description: "Git ref to build the docker image from"
required: true
type: string
is_master:
description: "Whether run from master branch"
required: true
type: boolean
secrets:
DOCKERHUB_USERNAME:
required: true
DOCKERHUB_TOKEN:
required: true
permissions: {}
jobs:
# Steps to test CI automation in your own fork.
# 1. Sign-up for https://hub.docker.com/
# 2. Store your dockerhub username as DOCKERHUB_USERNAME in "Repository secrets" of your fork repository settings (https://github.com/$githubuser/nix/settings/secrets/actions)
# 3. Create an access token in https://hub.docker.com/settings/security and store it as DOCKERHUB_TOKEN in "Repository secrets" of your fork
check_secrets:
permissions:
contents: none
name: Check presence of secrets
runs-on: ubuntu-24.04
outputs:
docker: ${{ steps.secret.outputs.docker }}
steps:
- name: Check for DockerHub secrets
id: secret
env:
_DOCKER_SECRETS: ${{ secrets.DOCKERHUB_USERNAME }}${{ secrets.DOCKERHUB_TOKEN }}
run: |
echo "docker=${{ env._DOCKER_SECRETS != '' }}" >> $GITHUB_OUTPUT
push:
name: Push docker image to DockerHub and GHCR
needs: [check_secrets]
permissions:
contents: read
packages: write
if: needs.check_secrets.outputs.docker == 'true'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
fetch-depth: 0
ref: ${{ inputs.ref }}
- uses: ./.github/actions/install-nix-action
with:
dogfood: false
extra_nix_config: |
experimental-features = flakes nix-command
- run: echo NIX_VERSION="$(nix eval .\#nix.version | tr -d \")" >> $GITHUB_ENV
- run: nix build .#dockerImage -L
- run: docker load -i ./result/image.tar.gz
# We'll deploy the newly built image to both Docker Hub and Github Container Registry.
#
# Push to Docker Hub first
- name: Login to Docker Hub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Push to Docker Hub
env:
IS_MASTER: ${{ inputs.is_master }}
DOCKERHUB_REPO: ${{ secrets.DOCKERHUB_USERNAME }}/nix
run: |
docker tag nix:$NIX_VERSION $DOCKERHUB_REPO:$NIX_VERSION
docker push $DOCKERHUB_REPO:$NIX_VERSION
if [ "$IS_MASTER" = "true" ]; then
docker tag nix:$NIX_VERSION $DOCKERHUB_REPO:master
docker push $DOCKERHUB_REPO:master
fi
# Push to GitHub Container Registry as well
- name: Login to GitHub Container Registry
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Push to GHCR
env:
IS_MASTER: ${{ inputs.is_master }}
run: |
IMAGE_ID=ghcr.io/${{ github.repository_owner }}/nix
IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
docker tag nix:$NIX_VERSION $IMAGE_ID:$NIX_VERSION
docker push $IMAGE_ID:$NIX_VERSION
if [ "$IS_MASTER" = "true" ]; then
docker tag nix:$NIX_VERSION $IMAGE_ID:master
docker push $IMAGE_ID:master
fi
-69
View File
@@ -1,69 +0,0 @@
name: Upload Release
on:
workflow_dispatch:
inputs:
eval_id:
description: "Hydra evaluation ID"
required: true
type: number
is_latest:
description: "Mark as latest release"
required: false
type: boolean
default: false
permissions:
contents: read
id-token: write
packages: write
jobs:
release:
runs-on: ubuntu-24.04
environment: releases
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- uses: ./.github/actions/install-nix-action
with:
dogfood: false # Use stable version
use_cache: false # Don't want any cache injection shenanigans
extra_nix_config: |
experimental-features = nix-command flakes
- name: Set NIX_PATH from flake input
run: |
NIXPKGS_PATH=$(nix build --inputs-from .# nixpkgs#path --print-out-paths --no-link)
# Shebangs with perl have issues. Pin nixpkgs this way. nix shell should maybe
# get the same uberhack that nix-shell has to support it.
echo "NIX_PATH=nixpkgs=$NIXPKGS_PATH" >> "$GITHUB_ENV"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708 # v5.1.1
with:
role-to-assume: "arn:aws:iam::080433136561:role/nix-release"
role-session-name: nix-release-oidc-${{ github.run_id }}
aws-region: eu-west-1
- name: Login to Docker Hub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Upload release
run: |
./maintainers/upload-release.pl \
${{ inputs.eval_id }} \
--skip-git
env:
IS_LATEST: ${{ inputs.is_latest && '1' || '' }}
- name: Push to GHCR
run: |
DOCKER_OWNER="ghcr.io/$(echo '${{ github.repository_owner }}' | tr '[A-Z]' '[a-z]')/nix"
./maintainers/upload-release.pl \
${{ inputs.eval_id }} \
--skip-git \
--skip-s3 \
--docker-owner "$DOCKER_OWNER"
env:
IS_LATEST: ${{ inputs.is_latest && '1' || '' }}
+11
View File
@@ -161,3 +161,14 @@ pull_request_rules:
labels:
- automatic backport
- merge-queue
- name: backport patches to 2.31
conditions:
- label=backport 2.31-maintenance
actions:
backport:
branches:
- "2.31-maintenance"
labels:
- automatic backport
- merge-queue
+1 -1
View File
@@ -1 +1 @@
2.31.3
2.32.0
+3 -10
View File
@@ -24,15 +24,8 @@ def map_contents_recursively(transformer):
def process_command:
.[0] as $context |
.[1] as $body |
# mdbook 0.5.x uses 'items' instead of 'sections'
if $body.items then
$body + {
items: $body.items | map(map_contents_recursively(if $context.renderer == "html" then transform_anchors_html else transform_anchors_strip end)),
}
else
$body + {
sections: $body.sections | map(map_contents_recursively(if $context.renderer == "html" then transform_anchors_html else transform_anchors_strip end)),
}
end;
$body + {
sections: $body.sections | map(map_contents_recursively(if $context.renderer == "html" then transform_anchors_html else transform_anchors_strip end)),
};
process_command
+9
View File
@@ -23,3 +23,12 @@ renderers = ["html"]
command = "jq --from-file ./anchors.jq"
[output.markdown]
[output.linkcheck]
# no Internet during the build (in the sandbox)
follow-web-links = false
# mdbook-linkcheck does not understand [foo]{#bar} style links, resulting in
# excessive "Potential incomplete link" warnings. No other kind of warning was
# produced at the time of writing.
warning-policy = "ignore"
+3 -3
View File
@@ -24,9 +24,9 @@ let
in
concatStringsSep "\n" (map showEntry storesList);
"index.md" = replaceStrings [ "@store-types@" ] [ index ] (
readFile ./source/store/types/index.md.in
);
"index.md" =
replaceStrings [ "@store-types@" ] [ index ]
(readFile ./source/store/types/index.md.in);
tableOfContents =
let
+2
View File
@@ -6,6 +6,7 @@
ninja,
lowdown-unsandboxed,
mdbook,
mdbook-linkcheck,
jq,
python3,
rsync,
@@ -50,6 +51,7 @@ mkMesonDerivation (finalAttrs: {
ninja
(lib.getBin lowdown-unsandboxed)
mdbook
mdbook-linkcheck
jq
python3
rsync
@@ -1,6 +0,0 @@
---
synopsis: "Temporary build directories no longer include derivation names"
prs: [13839]
---
Temporary build directories created during derivation builds no longer include the derivation name in their path to avoid build failures when the derivation name is too long. This change ensures predictable prefix lengths for build directories under `/nix/var/nix/builds`.
+2 -3
View File
@@ -12,11 +12,10 @@
The [`builder`](./derivation/index.md#builder) is executed as follows:
- A temporary directory is created where the build will take place. The
- A temporary directory is created under the directory specified by
`TMPDIR` (default `/tmp`) where the build will take place. The
current directory is changed to this directory.
See the per-store [`build-dir`](@docroot@/store/types/local-store.md#store-local-store-build-dir) setting for more information.
- The environment is cleared and set to the derivation attributes, as
specified above.
-4
View File
@@ -41,10 +41,6 @@ def recursive_replace(data: dict[str, t.Any], book_root: Path, search_path: Path
return data | dict(
sections = [recursive_replace(section, book_root, search_path) for section in sections],
)
case {'items': items}:
return data | dict(
items = [recursive_replace(item, book_root, search_path) for item in items],
)
case {'Chapter': chapter}:
path_to_chapter = Path(chapter['path'])
chapter_content = chapter['content']
Generated
+4 -4
View File
@@ -63,16 +63,16 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1761597516,
"narHash": "sha256-wxX7u6D2rpkJLWkZ2E932SIvDJW8+ON/0Yy8+a5vsDU=",
"lastModified": 1756178832,
"narHash": "sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "daf6dc47aa4b44791372d6139ab7b25269184d55",
"rev": "d98ce345cdab58477ca61855540999c86577d19d",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-25.05",
"ref": "nixos-25.05-small",
"repo": "nixpkgs",
"type": "github"
}
+2 -2
View File
@@ -1,7 +1,7 @@
{
description = "The purely functional package manager";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05-small";
inputs.nixpkgs-regression.url = "github:NixOS/nixpkgs/215d4d0fd80ca5163643b03a33fde804a29cc1e2";
inputs.nixpkgs-23-11.url = "github:NixOS/nixpkgs/a62e6edd6d5e1fa0329b8653c801147986f8d446";
@@ -32,7 +32,7 @@
let
inherit (nixpkgs) lib;
officialRelease = true;
officialRelease = false;
linux32BitSystems = [ "i686-linux" ];
linux64BitSystems = [
+1 -1
View File
@@ -46,7 +46,7 @@ The team meets twice a week (times are denoted in the [Europe/Amsterdam](https:/
- mark it as draft if it is blocked on the contributor
- escalate it back to the team by moving it to To discuss, and leaving a comment as to why the issue needs to be discussed again.
- Work meeting: Mondays 14:00-16:00 Europe/Amsterdam see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
- Work meeting: Mondays 18:00-20:00 Europe/Amsterdam; see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
1. Code review on pull requests from [In review](#in-review).
2. Other chores and tasks.
@@ -1,110 +0,0 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGPtMiwBEAC0sFZW2QW/OaDjKm5zGRpDvHXDsMIUtlHfoi5ce8pocC63W05o
FSXbUZjZ1VfYO8lT8DFANCzTkiXYaZx0cPRG2pVY4AOQZDNFt5XrAyvw496XCAIM
DTYGFLjCqgjPt9RUFEy4MyHPJTEpB0x3rXgT4ILNu9vsj9Q0vttps7SpbZ3Ldq5H
o/BBbLW77q/vNjpYzCbBIXF7ycUGpnNv9Go/WuiDnrBMcyxh+8kjjIHB5cxZSnjJ
DUv681+m83v+gLZQGX/jexQrrf5JpS0X9qEnhGLrNUDhtyv5ud3Je4EfamkjLVVC
RlNLofgflOCsl/tP80i+K7S1QdKhUALxuJ6H0prYUflGBDxDyC8XYuJ62TT0OUpa
vJvgwVlCq8/jq+ykYQXlbuBVOzi5wAuI4l3+HqreSQYPSiwe+6N590Zbafdv1fvN
WFtZKCTGMqfyaaAnppioH9/+NWkI2AQxaYVasYM/JEYvY9pJgA7alh51jHW4JglP
ErypKfBKPKJID0QENqYoa3bDDCihuNWhgQf9dxzPlj2ckd35Zb6w4DfuSmtjaa9D
o0jZVY1JbFuxBqP09+saVPrxLHgmPxjcdzPGQQtAqdO2vyJXNEGLFMoVEZPNaLo3
QmcIJnT7oSck+4vGfOYtWUHXQynu/Tnwsv2XkA/uyw8HNe+RRMqv/apnzQARAQAB
tCdTZXJnZWkgWmltbWVybWFuIDxzZXJnZWlAemltbWVybWFuLmZvbz6JAlEEEwEK
ADsCGwEFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQQVim9TDqIC5fZRYRMU+upj
RI4d+QUCaUbsaAIZAQAKCRAU+upjRI4d+VdDD/492HRaJ/8V7R7VUzkafmb2Hb28
SLf7oiB8Uq9I7SukiDEaIT1fUhquYWQ9KWpPRNR1TX6ApXnIeuJRMGFoDVIRnmnr
cKnYYXfqqc81VxIyKvaumB7KWbS7G4Nbor8AH1ouOOOMMS50OTJOWQA4A26inIuG
n+7L8MeS5aT+3uNKDoTKsidC47vnaxNMcke1taPfbfo7vn69PsRCM/g9/7TQYU8b
6xp+pM9Ao9nJneRk2YCpsGYRrWTpaik0DFKnfpPKJM/yunhtGLF2IYAp3l1mvHPK
nnzo92zjpQuZwazEIK+23V1vRT4IjM2BewbJPAzf2/UuxEjjgNQm0tOtH2JhFNeB
VM0BVrGxWrrwrsmv6lWghTtBc6zRWyHrj/rpjtVQNmeKYrHWJeXwVz1rqgGPmB2N
k0MZD1UjHHhEs1Cntn7yLmxTPztRJCtR+euRu81Uo2NAvrMJ4xsDjaM0LeLTnzjV
9AsPjD188dOFyz7VExZum4+XaaEJ41FIPLEqU3U0GAa6stEy0ylSlIN4x9aiXXVW
xfzHHchS5jK6QAjuZxN8t01GRactNylINRf7uoTECFZTtXNqfeuk0HQxBH0LuKVE
0PxJbcNI4mVWw1KgTJ8PUVC1IXP3sEpqPdJOYiRXgnpcS26fWOBu0aQ4mhxiaJhr
/zBfrkLEqp20TdNDZLkCDQRj7TM1ARAAt73xO24curnHTTgXkkVMMRzcMLx3Mb1a
2FuddxC5hzTpEpw01L91UBrXVJEg9K2KAwP5CtCLgPCqXr47Tm7krvHxWwBksgY/
6aHRsoPQfCFUZHc0aiO+C4NCzR+aEeGKn66Oc1Hq9oUTpDgiBWhsuEPiyA1OSGF0
4L0jeTCqfm68kWp4PIK9yuugkdDsoyj6TonuMsb3V5ctHLqop9KH+eHSkUTPo+Lk
+bxaeAOJ1UfbohgbRbrYKAfsaghhOMDH3R1w2pvtUJz+sDbuQsiPFTqbxsXDTFws
H4N/AQCYnnvOhqEek2sOEZ19bJXt5UrAr10mX4PGmAkWqE1JWBxpOKG3BXSGOTu1
3dFhQfPMK+PmvUrs0kcWQr53K/aRUdKKhIfTcMfkqYTGPK5HclHph24WjXj3QFFA
SjksQTdm6486ZmLZK4CTbAFOPfTF/aWg8gu9v4ihdq6lqHNNXxv2xBAChcd59H7p
D5zy9z8SpwWR9V5JDmlF6HWIIau1c6lSsQq1xHvYM8EuPe03vJvor+2u/cn5zYF1
5ZxAuPI2i5vtavg1s8ZGAAogJ9dVcP36LdJfL9quXWvmovkd//qHIepBB+l/zQio
ZRDZlIcfV3Xycaqsb5OqHGARHE0097koipMt5y/iXlqG4Ruue6Idb8bW96EKpaWj
kKy/iNfQfQMAEQEAAYkEcgQYAQoAJgIbAhYhBBWKb1MOogLl9lFhExT66mNEjh35
BQJpRuz3BQkJHCDCAkDBdCAEGQEKAB0WIQRK3hK0WyJ4BicGpcmpsLVXymMjJQUC
Y+0zNQAKCRCpsLVXymMjJbdSD/9+f1FOOeGDAJI6Duo5fsWnf4xJJdtQtDbz6d2A
SeDapxeJ3zWfKBD0wu5sISEa0uiWsYSmLtsa2SqVAKHlEaMGRR+tkBMPQ+rvgI4c
62YjGTgm+IPd+NFIn+ixFU1hpinTh+KhUEoeOwWCvKs9nZfSG9vkienfiG0bBxo2
zrvBzXA50x5hbUL+ghKu/AVfN9qZDwh30O4KZTwk4g4cM9SeaQa4YvHYIS3IEhDZ
hGybwrrqV9cs92ln4IJw9WCy9QReBNrdeFgC4+3ziUp1QsG3RvqrtuMttwBVC1Z3
bj5QjLLOREhhodfvk98t9yVkragObb4rGrLo1mWuF0c4mJGvXwnrqhCMvzv4M+0T
Zdrmw6YpGkGOaOPghVuwoTtqSAkl+zFWIJS89jidvkYG3EqKAkgLKog/TQReCq13
HWrF8cMck+Rf2K8k26q/RNZaA9ZUKjLExzz8lsWmd2C7rvkGLrlxnzxz0gGyNR3Q
KK74vcPhqeABt2GSkHtEXZFFA9IVVzwlRWK3e0S+mVQnZVjNL+cBPn3/hZHMLesB
CucyYZv+DxvT+JkYXBkGSw4s3hpABqGym7gdPUIa0q4rbBFG6xP5sLLBG4yru8vV
2dyCMmFqRuxpT49uNfyQ6Vj+dobN6qHnP/9NwfzOixXYBHXR6LBqb/M+iCiJaaIn
uiRLHwkQFPrqY0SOHfkQpA//W51vj8meuz7snRO+vZFcjLneFFzqfh1Jdz8IqDpO
CkI5pBJmi8e0oSe6r68MkahiQLlYPwm7d+sjHvJhPWipNKWq/uwCgBs+Ac1lpPXR
MwLbrZukcLMYlLmb2MrCKmjcMt0BZsZKBNYL3a3X9nHgwXdeqFYS4WQDMCCc09lz
9YqfdoEsqRO4qN7D0hFqnwjOzb34ixZ6UO8a8ekY9QKxAgWc9fJWGMg6Pjdg4qsK
nqymOIAdGVOJdoRM46wKGVBvbsF2gNfQU4XyzgJo5vHGFwJm6EoSnODlL5e2wsQh
uN1oqBt/8ef/plloMEqVBweUBATqSqjRF6IhhYJvWVuQHQL1p1vnV9FebiVj34ir
Z8ID+o0AnTJcclbUcDwannGJ0cuDcPhk/v/ahVuoMERCi12qnMBo5B/e6Omyh1yB
4pbf4GATGGQipDQG75eC/kP2GQEqJP5WYN0Ar8Le/AA/2xyL7upW0yIByyXCwGEb
JRwEgU3+bPyu58bFt8Pftit6J7rA3oBVVMOPrYH5eZwRaj5m2RptwKGL6BfHnhNv
ZqmCq9EBGX6L1NI0xHMjEFfXJ8jU01XdfG8nCqkwqsHwslXLhqjJphfHcx89YwbV
/15GCuURAv1cKe/7277sOhcvP/QpQqSWgvYExHw8PeFJcTYtF2NrRgNwcQsWS1Rj
gXa5Ag0EY+0zcAEQANC5N6kSfezuucAgi+X3BD+MT37mxQyvICSggEJf1LDSmy0+
bnvD7setL8CP9etTA2fcVNYKI1oboMyhoCnsRP2jDdv1iXOI/hZg4wSb/D1yUkae
fUpxv3Wuci2QKavH2MfraDD7BFMbsQeMcHtn4Rk216T6jndZHnzT1Ih7iX0XeQPb
li5fojOiZssgWAVT4HPXFCJB6lI35Hjp35oRYwrtMmu5INinZ79n9h1igGtt1ItZ
b7rQKNd772Jxcn4UU71ovORSL/xT5i5sxZ+evQOxkpqUAokMOFaoHcOXLmA1NsFv
yryXHK4Ioq9ap2jKlLTWkJWjua9JZ4AmKhbvT8X4ELxIKSCAdJKAWP8ZHbXNu5MD
aznyzZQLxSO7uFvu356De75mI5iohZNj5wB5Wju71pBiorTKVj4+iJ4e+xVIzFdG
hFC0DehNcl2t9w/y8qHwIQ1yUAjXHLXq0/2jsVeH6bU5q/MsgvUP1jcFe0eyOpxy
CDvyFdzZFbI57TnB/fvcZTRZ5ewXMFpH8gzuoFzAjUAP95UjYKgaGdrNPNIy28Ii
4zhvdghei2+n9jgiMfcGQg8lyfH5yF0vWWWynX0KcJsRwEZoL2EauVdwq4PcYOoU
pQFhpcreCjD4LdZ4yRU4InbhcUogXjrQ9Dz01TbPmQD5b5iso21bCEFBXrhzABEB
AAGJAjwEGAEKACYCGwwWIQQVim9TDqIC5fZRYRMU+upjRI4d+QUCaUbs9wUJCRwg
hwAKCRAU+upjRI4d+X/XD/sH5xvHPfTJq52v8weFmB52up+DzqG2lyhGdoUQ1Muw
dRDLTLXLJrFdfpoOo7/j4Scr0rdc7/dpCn0DLcPuCoPxu+SkjEnVehFmZrGSv7Ga
x9dHr3DBh42fdlX/U/EnDuyosY0JU1gNF2/6FIA+bTTOFE3RxfN906RjslYQDjMZ
UAlSeLYHOZofdltI0YIr32vrxgdWQGZXPxU4XusDUc0z163OO+TGg7iUNWFZP5Qj
ubM7e0YbDX0NPIshk8us99YJmrWnhaix1/W5ryO3DXiGaQ7XFi9u7QofRqvRIctg
QXavdepkzJow9V9qpMECAJePIuICq7rm+xy+njjbuF436W7390bfVBwRr+FPADsl
jgQP4KvY5rykss30kheom8wNEbveWkhH5oTfH9b7O4KXJfpfJzrlgOWp2BD9JL8t
/M4HvFXTr2a75H/QbHK5OFrZeGATuv9OTxv7EZvnrPXU+DYTFldpu7TrNNqKCoj3
ZyXmc3Hhg5kskDhfHJppaeOayuhMOpT3ud1MFzROY5SLVIH8rBR12KUgsCUYQcGs
Iy0+0QvEGkjb4cAH1NK3VlbqVNsy1RmqRt2B28R2ueewDfTOoqkzt4MmzLqTdnAx
mTqmHmkEKhEf3K4MRNUPO2yieUg2COk5l6x9HhAnoxxeOZrTmcMsPY/UViG2HEPm
ybkCDQRj7TRDARAA9DZuKdfKq4Bs2+NwxC0aplljWOl8VIsEVg+Q8agD7/HU6/b6
Dry0njtWybn2x6Axf/nUdeOC01Fi1lmht/fpj6mRkgAvd/V6P10xnsUoykPSDSTh
P25MFFGW3JAA82bwdJ4AJpEQvTZG2nTb3237vlBiI1qHQrac8GYkju2O4UfySRN6
7cyi7bMf2pjWBBOEhaNy4b6CMDsb32P/N5J7sTE/TXgrS+u4ITIgjzSrkUkh5Z+B
8QVRa7xPIDZJdvZWTEXWu5fgRPZvxbr154GIkWJkFzlDoB1UcO56/uzRUuKhEV6o
HW3LMUuWdPMjpHpq8hrL0G2rDniJFUtbDFzHdZK1LUU3T2BJM8rjI3D/euph+IDT
27vl5qo72zCYE/iKzx4FMLZcQvx1kUAxkPX8l+dzZEwKeRIIpFDxQvatRtl+z0bM
jbkpDb+Yjv66sC4dYRpgTTGX6rok0PWHR3IxDNzyf2j8zQ4LFJ+rVBM1GjGSt6mG
j9TeL8CVeiSp4SuJ7I/FJVPHsKb50m+BDzeB31qTydNqh2kKr0DVAUa+TUsCr7e0
OYr8WE2adJcRXIW0qw50xXF+W7/05GqSCVD0dpeOUdBTQTsSkQmM3/0hcj9aVo9e
UDCM9RF0WRqiDAoHzJFfg+ztamkQI5HO6CklC4Ok22qrHRf6HDNYSuT6QFkAEQEA
AYkCPQQYAQoAJwMbIAQWIQQVim9TDqIC5fZRYRMU+upjRI4d+QUCaUbs9wUJCRwf
tAAKCRAU+upjRI4d+Y+cD/9yllG6uo934pcHNsVppZBfREFwSc8ywlbosCuSVpay
PjSqgrWwDrnqrsk0F2kUdC6rR3BIcXbn+lA9KqylH+cCXAJCkh8EDq6TlQ7Lt5EV
w1U0MAMXOyxPwDymQ/BO+iDyjXWkRRYgbF5XiFhCfGeuKyhkhACisAgNZ1uA1P5k
0SJYc14YfEhQkB46Y20SpfVHRsQ46FyNB6GHbmTmfoO8La8VTh++7GBdh85HfvkG
VNQ3wpi5oXsOLN9+MJOezc0XsW2LQsKQj1/J7QKzGh+lxN5cemsA5aqPzh8dyxeT
0lYRFp4AHkimqGUomVpRkbegMIPxXqOE+ZAmsddErw0UtmrKxcmMptOJwNgYzEgu
++2vtqerL/NYp+wsdcWaBjCz2F3NiwHgNli7NSB/FPwucZZ5gN5C4SnmeFzrGdHg
Oy+tQUN6ayQKljHeBO7CjMlsFNo/dcVrEMa1ShxBMqlj/6ivoEhktLz0Nru4FwNU
xE5SJYDYfpjD7Ws8y4LoXgWXjFHrMO6N9GzqLN/e8LT7I+w4ps2MrgJ8QSrelmQ3
rjkxp3uWp5v2lqy4rLfpi9iB6zIAeoN2eU1yOM9joxOYMxKYaYeYyP1Mm90wFol8
LcTSaN+tVniPddBiL6zvsGBEMbCR9XN3EQ+mErbuw5ovWBOCrr+dvN3FxvD11y4J
7w==
=mXYP
-----END PGP PUBLIC KEY BLOCK-----
@@ -1,51 +0,0 @@
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQENBFZu2zwBCADfatenjH3cvhlU6AeInvp4R0JmPBG942aghFj1Qh57smRcO5Bv
y9mqrX3UDdmVvu58V3k1k9/GzPnAG1t+c7ohdymv/AMuNY4pE2sfxx7bX+mncTHX
5wthipn8kTNm4WjREjCJM1Bm5sozzEZetED3+0/dWlnHl8b38evnLsD+WbSrDPVp
o6M6Eg9IfMwTfcXzdmLmSnGolBWDQ9i1a0x0r3o+sDW5UTnr7jVP+zILcnOZ1Ewl
Rn9OJ4Qg3ULM7WTMDYpKH4BO7RLR3aJgmsFAHp17vgUnzzFBZ10MCS3UOyUNoyph
xo3belf7Q9nrHcSNbqSeQuBnW/vafAZUreAlABEBAAG0IkVlbGNvIERvbHN0cmEg
PGVkb2xzdHJhQGdtYWlsLmNvbT6JATwEEwEIACYCGyMHCwkIBwMCAQYVCAIJCgsE
FgIDAQIeAQIXgAUCVm7etAIZAQAKCRCBcLRybXGY3q51B/96qt41tmcDSzrj/UTl
O6rErfW5zFvVsJTZ95Duwu87t/DVhw5lKBQcjALqVddufw1nMzyN/tSOMVDW8xe4
wMEdcU4+QAMzNX80enuyinsw1glxfLcK0+VbTvqNIfw0sG3MjPqNs6cK2VRfMHK4
paJjytBVICszNX9TfjLyIpKKoSSo1vqnT47LDZ5GIMy7l9Cs2sO/rqQHSPcR79yz
8m8tbHpDDEMZmJeklckKP2QoiqnHiIvlisDxLclYnUmNaPdaN/f++qZz5Yqvu1n+
sNUBA5eLaZH64Uy2SwtABxO3JPJ8nQ2+SFZ7ocFm4Gcdv4aM+Ura9S6fvM91tEJp
yAQOiQE5BBMBCAAjBQJWbts8AhsjBwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AA
CgkQgXC0cm1xmN6sIAgAielxO8zJREqEkA2xudg/o4e9ZlNZ3X1NvY8OzJH/qlB2
SmwKqwifhtbC1K0uavXA7eaxdtd2zrI+Yq7IooUyv7juMjHTZhLcFbR5iVkQ4Mfp
JmeHXJ/ChYKxD5mMj/C3WbCZ91oCSNZ6Iyi5fvQj/691OC4q+y/2NEUcOI8D8cw8
XKHbKtceFYc+nZmdOv3ZZrNTSN/kszGViNNLKgnpPdDVPtLp+vjXtbmitiFG2HL/
WfbJ+3Gh2Yr1Vy3O9dWKH++e1AmIv7WWqmUjRFVpqC/wr7/BLaScWT8WKF5vkshU
gq8Ez1/cuizsgs3wQIZWgXKQK5njvwnbKg+Zmh/uGbQmRWVsY28gRG9sc3RyYSA8
ZWVsY28uZG9sc3RyYUB0d2VhZy5pbz6JAU4EEwEIADgWIQS1QdVTAScOC88Vyl2B
cLRybXGY3gUCXELt4gIbIwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRCBcLRy
bXGY3ujFCADfS5D1xHU8KH6TpqgssSggYVq62Wwn/Ga+4XPPetM+ajcXagyH6SwB
mxlHICcnv9xC93ryiTI10P1ADJl+aBsI66wEdHBU+ty4RTDy4JZNUPtmRCk9LhSc
mtUO3ry/wtWkRLdJxP49hg7BbQvWoU0M6WODp7SJjPKPWNX64mzHBeOuy+DqGCbM
lpGNCvW8ahU/ewbm7+xwWmzqLDoWzXjHsdF4QdzMVM/vkAgWEP4y0wEqFASzIYaR
GNEkBWU4OQVq5Bdm9+wWWAgsbM0FJAQl0GDqnz4QxWzxxCAAXdbh9F5ffafWYsA9
bise4ZQLkvYo6iUnrcFm4dtZbT8iL3gptCtFZWxjbyBEb2xzdHJhIDxlZWxjby5k
b2xzdHJhQGxvZ2ljYmxveC5jb20+iQE5BBMBCAAjBQJWbt6nAhsjBwsJCAcDAgEG
FQgCCQoLBBYCAwECHgECF4AACgkQgXC0cm1xmN4b/wf8DApMV/jSPEpibekrUPQu
Ye3Z8cxBQuRm/nOPowtPEH/ShAevrCdRiob2nuEZWNoqZ2e5/+6ud07Hs9bslvco
cDv1jeY1dof1idxfKhH3kfSpuD2XJhuzQBxBqOrIlCS/rdnW+Y9wOGD7+bs9QpcA
IyAeQGLLkfggAxaGYQ2Aev8pS7i3a/+lOWbFhcTe02I49KemCOJqBorG5FfILLNr
DjO3EoutNGpuz6rZvc/BlymphWBoAdUmxgoObr7NYWgw9pI8WeE6C7bbSOO7p5aQ
spWXU7Hm17DkzsVDpaJlyClllqK+DdKza5oWlBMe/P02jD3Y+0P/2rCCyQQwmH3D
RbkBDQRWbts8AQgA0g556xc08dH5YNEjbCwEt1j+XoRnV4+GfbSJIXOl9joIgzRC
4IaijvL8+4biWvX7HiybfvBKto0XB1AWLZRC3jWKX5p74I77UAcrD+VQ/roWQqlJ
BKbiQMlRYEsj/5Xnf72G90IP4DAFKvNl+rLChe+jUySA91BCtrYoP75Sw1BE9Cyz
xEtm4WUzKAJdXI+ZTBttA2Nbqy+GSuzBs7fSKDwREJaZmVrosvmns+pQVG4WPWf4
0l4mPguDQmZ9wSWZvBDkpG7AgHYDRYRGkMbAGsVfc6cScN2VsSTa6cbeeAEowKxM
qx9RbY3WOq6aKAm0qDvow1nl7WwXwe8K0wQxfQARAQABiQEfBBgBCAAJBQJWbts8
AhsMAAoJEIFwtHJtcZjeuAAH/0YNz2Qe1IAEO5oqEZNFOccL4KxVPrBhWUen83/b
C6PjOnOqv6q5ztAcms88WIKxBlfzIfq+dzJcbKVS/H7TEXgcaC+7EYW8sJVEsipN
BtEZ3LQNJ5coDjm7WZygniah1lfXNuiritAXduK5FWNNndqGArEaeZ8Shzdo/Uyi
b9lOsBIL6xc2ZcnX5f+rTu02LCEtEb0FwCycZLEWYf8hG4k8uttIOZOC+CLk/k8d
kBmPikMwUVTTV0CdT1cemQKdTaoAaK+kurF6FYXwcnjhRlHrisSt/tVMEwTw4LUM
3MYf6qfjjvE4HlDwZal8th7ccoQp/flfJIuRv85xCcKK+PI=
=u5cX
-----END PGP PUBLIC KEY BLOCK-----
-13
View File
@@ -1,13 +0,0 @@
# Maintainer GPG Keys
Release tags are signed by members of the [Nix maintainer team](https://nixos.org/community/teams/nix/) as part of the [release process](../release-process.md). This directory contains the public GPG keys used for signing.
## Keys
- **Eelco Dolstra**
GPG Fingerprint: `B541 D553 0127 0E0B CF15 CA5D 8170 B472 6D71 98DE`
- **Sergei Zimmerman**
GPG Fingerprint: [`158A 6F53 0EA2 02E5 F651 6113 14FA EA63 448E 1DF9`](https://keys.openpgp.org/vks/v1/by-fingerprint/158A6F530EA202E5F651611314FAEA63448E1DF9)
<!-- TODO: Add keys for other Nix team members -->
+58
View File
@@ -0,0 +1,58 @@
#!/usr/bin/env bash
set -euo pipefail
# debug:
# set -x
START_REF="${1}"
END_REF="${2:-upstream/master}"
# Get the merge base
MERGE_BASE=$(git merge-base "$START_REF" "$END_REF")
unset START_REF
# Get date range
START_DATE=$(git show -s --format=%cI "$MERGE_BASE")
END_DATE=$(git show -s --format=%cI "$END_REF")
echo "Checking PRs merged between $START_DATE and $END_DATE" >&2
# Get all commits between merge base and HEAD
COMMITS=$(git rev-list "$MERGE_BASE..$END_REF")
# Convert to set for fast lookup
declare -A commit_set
for commit in $COMMITS; do
commit_set["$commit"]=1
done
# Get the current changelog
LOG_DONE="$(changelog-d doc/manual/rl-next)"
is_done(){
local nr="$1"
echo "$LOG_DONE" | grep -E "^- .*/pull/$nr)"
}
# Query merged PRs in date range
gh pr list \
--repo NixOS/nix \
--state merged \
--limit 1000 \
--json number,title,author,mergeCommit \
--search "merged:$START_DATE..$END_DATE" | \
jq -r '.[] | [.number, .mergeCommit.oid, .title, .author.login] | @tsv' | \
while IFS=$'\t' read -r pr_num merge_commit _title author; do
# Check if this PR's merge commit is in our branch
if [[ -n "${commit_set[$merge_commit]:-}" ]]; then
# Full detail, not suitable for comment due to mass ping and duplicate title
# echo "- #$pr_num $_title (@$author)"
echo "- #$pr_num ($author)"
if is_done "$pr_num"
then
echo " - [x] has note"
else
echo " - [ ] has note"
fi
echo " - [ ] skip"
fi
done
+20 -13
View File
@@ -5,11 +5,11 @@
The release process is intended to create the following for each
release:
* A signed Git tag (public keys in `maintainers/keys/`)
* A Git tag
* Binary tarballs in https://releases.nixos.org/?prefix=nix/
* Docker images (arm64 and amd64 variants, uploaded to DockerHub and GHCR)
* Docker images
* Closures in https://cache.nixos.org
@@ -24,6 +24,12 @@ release:
* In a checkout of the Nix repo, make sure you're on `master` and run
`git pull`.
* Compile a release notes to-do list by running
```console
$ ./maintainers/release-notes-todo PREV_RELEASE HEAD
```
* Compile the release notes by running
```console
@@ -98,17 +104,21 @@ release:
evaluation ID (e.g. `1780832` in
`https://hydra.nixos.org/eval/1780832`).
* Tag the release:
* Tag the release and upload the release artifacts to
[`releases.nixos.org`](https://releases.nixos.org/) and [Docker Hub](https://hub.docker.com/):
```console
$ IS_LATEST=1 ./maintainers/upload-release.pl --skip-docker --skip-s3 --project-root $PWD <EVAL-ID>
$ IS_LATEST=1 ./maintainers/upload-release.pl <EVAL-ID>
```
Note: `IS_LATEST=1` causes the `latest-release` branch to be
force-updated. This is used by the `nixos.org` website to get the
[latest Nix manual](https://nixos.org/manual/nixpkgs/unstable/).
* Trigger the [`upload-release.yml` workflow](https://github.com/NixOS/nix/actions/workflows/upload-release.yml) via `workflow_dispatch` trigger. At the top click `Run workflow` -> select the current release branch from `Use workflow from` -> fill in `Hydra evaluation ID` with `<EVAL-ID>` value from previous steps -> click `Run workflow`. Wait for the run to be approved by `NixOS/nix-team` (or bypass checks if warranted). Wait for the workflow to succeed.
TODO: This script requires the right AWS credentials. Document.
TODO: This script currently requires a
`/home/eelco/Dev/nix-pristine`.
TODO: trigger nixos.org netlify: https://docs.netlify.com/configure-builds/build-hooks/
@@ -123,6 +133,8 @@ release:
Commit and push this to the maintenance branch.
* Create a backport label.
* Bump the version of `master`:
```console
@@ -130,6 +142,7 @@ release:
$ git pull
$ NEW_VERSION=2.13.0
$ echo $NEW_VERSION > .version
$ ... edit .mergify.yml to add the previous version ...
$ git checkout -b bump-$NEW_VERSION
$ git commit -a -m 'Bump version'
$ git push --set-upstream origin bump-$NEW_VERSION
@@ -137,10 +150,6 @@ release:
Make a pull request and auto-merge it.
* Create a backport label.
* Add the new backport label to `.mergify.yml`.
* Post an [announcement on Discourse](https://discourse.nixos.org/c/announcements/8), including the contents of
`rl-$VERSION.md`.
@@ -173,18 +182,16 @@ release:
* Wait for the desired evaluation of the maintenance jobset to finish
building.
* Tag the release
* Run
```console
$ IS_LATEST=1 ./maintainers/upload-release.pl --skip-docker --skip-s3 --project-root $PWD <EVAL-ID>
$ IS_LATEST=1 ./maintainers/upload-release.pl <EVAL-ID>
```
Omit `IS_LATEST=1` when creating a point release that is not on the
most recent stable branch. This prevents `nixos.org` to going back
to an older release.
* Trigger the [`upload-release.yml` workflow](https://github.com/NixOS/nix/actions/workflows/upload-release.yml) via `workflow_dispatch` trigger. At the top click `Run workflow` -> select the current release branch from `Use workflow from` -> fill in `Hydra evaluation ID` with `<EVAL-ID>` value from previous steps -> click `Run workflow`. Wait for the run to be approved by `NixOS/nix-team` (or bypass checks if warranted). Wait for the workflow to succeed.
* Bump the version number of the release branch as above (e.g. to
`2.12.2`).
+82 -128
View File
@@ -1,8 +1,7 @@
#! /usr/bin/env nix-shell
#! nix-shell -i perl -p awscli2 perl perlPackages.LWPUserAgent perlPackages.LWPProtocolHttps perlPackages.FileSlurp perlPackages.NetAmazonS3 perlPackages.GetoptLongDescriptive gnupg1
#! nix-shell -i perl -p perl perlPackages.LWPUserAgent perlPackages.LWPProtocolHttps perlPackages.FileSlurp perlPackages.NetAmazonS3 gnupg1
use strict;
use Getopt::Long::Descriptive;
use Data::Dumper;
use File::Basename;
use File::Path;
@@ -14,30 +13,7 @@ use Net::Amazon::S3;
delete $ENV{'shell'}; # shut up a LWP::UserAgent.pm warning
my ($opt, $usage) = describe_options(
'%c %o <eval-id>',
[ 'skip-docker', 'Skip Docker image upload' ],
[ 'skip-git', 'Skip Git tagging' ],
[ 'skip-s3', 'Skip S3 upload' ],
[ 'docker-owner=s', 'Docker image owner', { default => 'nixos/nix' } ],
[ 'project-root=s', 'Pristine git repository path' ],
[ 's3-endpoint=s', 'Custom S3 endpoint' ],
[ 's3-host=s', 'S3 host', { default => 's3-eu-west-1.amazonaws.com' } ],
[],
[ 'help|h', 'Show this help message', { shortcircuit => 1 } ],
[],
[ 'Environment variables:' ],
[ 'AWS_ACCESS_KEY_ID' ],
[ 'AWS_SECRET_ACCESS_KEY' ],
[ 'AWS_SESSION_TOKEN For OIDC' ],
[ 'IS_LATEST Set to "1" to mark as latest release' ],
);
print($usage->text), exit if $opt->help;
my $evalId = $ARGV[0] or do { print STDERR $usage->text; exit 1 };
die "--project-root is required unless --skip-git is specified\n" unless $opt->skip_git || $opt->project_root;
my $evalId = $ARGV[0] or die "Usage: $0 EVAL-ID\n";
my $releasesBucketName = "nix-releases";
my $channelsBucketName = "nix-channels";
@@ -86,38 +62,25 @@ File::Path::make_path($narCache);
my $binaryCache = "https://cache.nixos.org/?local-nar-cache=$narCache";
# S3 setup.
my $aws_access_key_id = $ENV{'AWS_ACCESS_KEY_ID'};
my $aws_secret_access_key = $ENV{'AWS_SECRET_ACCESS_KEY'};
my $aws_session_token = $ENV{'AWS_SESSION_TOKEN'};
my $aws_access_key_id = $ENV{'AWS_ACCESS_KEY_ID'} or die "No AWS_ACCESS_KEY_ID given.";
my $aws_secret_access_key = $ENV{'AWS_SECRET_ACCESS_KEY'} or die "No AWS_SECRET_ACCESS_KEY given.";
my ($s3, $releasesBucket, $s3_channels, $channelsBucket);
my $s3 = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
retry => 1,
host => "s3-eu-west-1.amazonaws.com",
});
unless ($opt->skip_s3) {
$aws_access_key_id or die "No AWS_ACCESS_KEY_ID given.";
$aws_secret_access_key or die "No AWS_SECRET_ACCESS_KEY given.";
my $releasesBucket = $s3->bucket($releasesBucketName) or die;
$s3 = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
$aws_session_token ? (aws_session_token => $aws_session_token) : (),
retry => 1,
host => $opt->s3_host,
secure => ($opt->s3_endpoint && $opt->s3_endpoint =~ /^http:/) ? 0 : 1,
});
my $s3_us = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
retry => 1,
});
$releasesBucket = $s3->bucket($releasesBucketName) or die;
$s3_channels = Net::Amazon::S3->new(
{ aws_access_key_id => $aws_access_key_id,
aws_secret_access_key => $aws_secret_access_key,
$aws_session_token ? (aws_session_token => $aws_session_token) : (),
retry => 1,
$opt->s3_endpoint ? (host => $opt->s3_host) : (),
$opt->s3_endpoint ? (secure => ($opt->s3_endpoint =~ /^http:/) ? 0 : 1) : (),
});
$channelsBucket = $s3_channels->bucket($channelsBucketName) or die;
}
my $channelsBucket = $s3_us->bucket($channelsBucketName) or die;
sub getStorePath {
my ($jobName, $output) = @_;
@@ -152,12 +115,11 @@ sub copyManual {
File::Path::remove_tree("$tmpDir/manual.tmp", {safe => 1});
}
my $awsEndpoint = $opt->s3_endpoint ? "--endpoint-url " . $opt->s3_endpoint : "";
system("aws $awsEndpoint s3 sync '$tmpDir/manual' s3://$releasesBucketName/$releaseDir/manual") == 0
system("aws s3 sync '$tmpDir/manual' s3://$releasesBucketName/$releaseDir/manual") == 0
or die "syncing manual to S3\n";
}
copyManual unless $opt->skip_s3;
copyManual;
sub downloadFile {
my ($jobName, $productNr, $dstName) = @_;
@@ -196,12 +158,30 @@ sub downloadFile {
return $sha256_expected;
}
# Upload docker images.
downloadFile("binaryTarball.i686-linux", "1");
downloadFile("binaryTarball.x86_64-linux", "1");
downloadFile("binaryTarball.aarch64-linux", "1");
downloadFile("binaryTarball.x86_64-darwin", "1");
downloadFile("binaryTarball.aarch64-darwin", "1");
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv6l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv7l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.riscv64-unknown-linux-gnu", "1");
};
warn "$@" if $@;
downloadFile("installerScript", "1");
# Upload docker images to dockerhub.
my $dockerManifest = "";
my $dockerManifestLatest = "";
my $haveDocker = 0;
unless ($opt->skip_docker) {
for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
my $system = $platforms->[0];
my $dockerPlatform = $platforms->[1];
@@ -215,8 +195,8 @@ for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
print STDERR "loading docker image for $dockerPlatform...\n";
system("docker load -i $tmpDir/$fn") == 0 or die;
my $tag = $opt->docker_owner . ":$version-$dockerPlatform";
my $latestTag = $opt->docker_owner . ":latest-$dockerPlatform";
my $tag = "nixos/nix:$version-$dockerPlatform";
my $latestTag = "nixos/nix:latest-$dockerPlatform";
print STDERR "tagging $version docker image for $dockerPlatform...\n";
system("docker tag nix:$version $tag") == 0 or die;
@@ -239,94 +219,68 @@ for my $platforms (["x86_64-linux", "amd64"], ["aarch64-linux", "arm64"]) {
}
if ($haveDocker) {
my $dockerOwner = $opt->docker_owner;
print STDERR "creating multi-platform docker manifest...\n";
system("docker manifest rm $dockerOwner:$version");
system("docker manifest create $dockerOwner:$version $dockerManifest") == 0 or die;
system("docker manifest rm nixos/nix:$version");
system("docker manifest create nixos/nix:$version $dockerManifest") == 0 or die;
if ($isLatest) {
print STDERR "creating latest multi-platform docker manifest...\n";
system("docker manifest rm $dockerOwner:latest");
system("docker manifest create $dockerOwner:latest $dockerManifestLatest") == 0 or die;
system("docker manifest rm nixos/nix:latest");
system("docker manifest create nixos/nix:latest $dockerManifestLatest") == 0 or die;
}
print STDERR "pushing multi-platform docker manifest...\n";
system("docker manifest push $dockerOwner:$version") == 0 or die;
system("docker manifest push nixos/nix:$version") == 0 or die;
if ($isLatest) {
print STDERR "pushing latest multi-platform docker manifest...\n";
system("docker manifest push $dockerOwner:latest") == 0 or die;
system("docker manifest push nixos/nix:latest") == 0 or die;
}
}
}
# Upload nix-fallback-paths.nix.
write_file("$tmpDir/fallback-paths.nix",
"{\n" .
" x86_64-linux = \"" . getStorePath("build.nix-everything.x86_64-linux") . "\";\n" .
" i686-linux = \"" . getStorePath("build.nix-everything.i686-linux") . "\";\n" .
" aarch64-linux = \"" . getStorePath("build.nix-everything.aarch64-linux") . "\";\n" .
" riscv64-linux = \"" . getStorePath("buildCross.nix-everything.riscv64-unknown-linux-gnu.x86_64-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.nix-everything.x86_64-darwin") . "\";\n" .
" aarch64-darwin = \"" . getStorePath("build.nix-everything.aarch64-darwin") . "\";\n" .
"}\n");
# Upload release files to S3.
unless ($opt->skip_s3) {
downloadFile("binaryTarball.i686-linux", "1");
downloadFile("binaryTarball.x86_64-linux", "1");
downloadFile("binaryTarball.aarch64-linux", "1");
downloadFile("binaryTarball.x86_64-darwin", "1");
downloadFile("binaryTarball.aarch64-darwin", "1");
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv6l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.armv7l-unknown-linux-gnueabihf", "1");
};
warn "$@" if $@;
eval {
downloadFile("binaryTarballCross.x86_64-linux.riscv64-unknown-linux-gnu", "1");
};
warn "$@" if $@;
downloadFile("installerScript", "1");
for my $fn (glob "$tmpDir/*") {
my $name = basename($fn);
next if $name eq "manual";
my $dstKey = "$releaseDir/" . $name;
unless (defined $releasesBucket->head_key($dstKey)) {
print STDERR "uploading $fn to s3://$releasesBucketName/$dstKey...\n";
# Upload nix-fallback-paths.nix.
write_file("$tmpDir/fallback-paths.nix",
"{\n" .
" x86_64-linux = \"" . getStorePath("build.nix-everything.x86_64-linux") . "\";\n" .
" i686-linux = \"" . getStorePath("build.nix-everything.i686-linux") . "\";\n" .
" aarch64-linux = \"" . getStorePath("build.nix-everything.aarch64-linux") . "\";\n" .
" riscv64-linux = \"" . getStorePath("buildCross.nix-everything.riscv64-unknown-linux-gnu.x86_64-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.nix-everything.x86_64-darwin") . "\";\n" .
" aarch64-darwin = \"" . getStorePath("build.nix-everything.aarch64-darwin") . "\";\n" .
"}\n");
my $configuration = ();
$configuration->{content_type} = "application/octet-stream";
for my $fn (glob "$tmpDir/*") {
my $name = basename($fn);
next if $name eq "manual";
my $dstKey = "$releaseDir/" . $name;
unless (defined $releasesBucket->head_key($dstKey)) {
print STDERR "uploading $fn to s3://$releasesBucketName/$dstKey...\n";
my $configuration = ();
$configuration->{content_type} = "application/octet-stream";
if ($fn =~ /.sha256|install|\.nix$/) {
$configuration->{content_type} = "text/plain";
}
$releasesBucket->add_key_filename($dstKey, $fn, $configuration)
or die $releasesBucket->err . ": " . $releasesBucket->errstr;
if ($fn =~ /.sha256|install|\.nix$/) {
$configuration->{content_type} = "text/plain";
}
}
# Update the "latest" symlink.
$channelsBucket->add_key(
"nix-latest/install", "",
{ "x-amz-website-redirect-location" => "https://releases.nixos.org/$releaseDir/install" })
or die $channelsBucket->err . ": " . $channelsBucket->errstr
if $isLatest;
$releasesBucket->add_key_filename($dstKey, $fn, $configuration)
or die $releasesBucket->err . ": " . $releasesBucket->errstr;
}
}
# Update the "latest" symlink.
$channelsBucket->add_key(
"nix-latest/install", "",
{ "x-amz-website-redirect-location" => "https://releases.nixos.org/$releaseDir/install" })
or die $channelsBucket->err . ": " . $channelsBucket->errstr
if $isLatest;
# Tag the release in Git.
unless ($opt->skip_git) {
chdir($opt->project_root) or die "Cannot chdir to " . $opt->project_root . ": $!";
system("git remote update origin") == 0 or die;
system("git tag --force --sign $version $nixRev -m 'Tagging release $version'") == 0 or die;
system("git push origin refs/tags/$version") == 0 or die;
system("git push --force-with-lease origin $nixRev:refs/heads/latest-release") == 0 or die if $isLatest;
}
chdir("/home/eelco/Dev/nix-pristine") or die;
system("git remote update origin") == 0 or die;
system("git tag --force --sign $version $nixRev -m 'Tagging release $version'") == 0 or die;
system("git push --tags") == 0 or die;
system("git push --force-with-lease origin $nixRev:refs/heads/latest-release") == 0 or die if $isLatest;
File::Path::remove_tree($narCache, {safe => 1});
File::Path::remove_tree($tmpDir, {safe => 1});
+20 -1
View File
@@ -10,8 +10,27 @@
stdenv,
}:
let
prevStdenv = stdenv;
in
let
inherit (pkgs) lib;
stdenv = if prevStdenv.isDarwin && prevStdenv.isx86_64 then darwinStdenv else prevStdenv;
# Fix the following error with the default x86_64-darwin SDK:
#
# error: aligned allocation function of type 'void *(std::size_t, std::align_val_t)' is only available on macOS 10.13 or newer
#
# Despite the use of the 10.13 deployment target here, the aligned
# allocation function Clang uses with this setting actually works
# all the way back to 10.6.
# NOTE: this is not just a version constraint, but a request to make Darwin
# provide this version level of support. Removing this minimum version
# request will regress the above error.
darwinStdenv = pkgs.overrideSDK prevStdenv { darwinMinVersion = "10.13"; };
in
scope: {
inherit stdenv;
@@ -54,7 +73,7 @@ scope: {
nativeBuildInputs = prevAttrs.nativeBuildInputs ++ [ pkgs.buildPackages.bmake ];
postInstall =
lib.replaceStrings [ "lowdown.so.1" "lowdown.1.dylib" ] [ "lowdown.so.2" "lowdown.2.dylib" ]
(prevAttrs.postInstall or "");
prevAttrs.postInstall;
});
# TODO Hack until https://github.com/NixOS/nixpkgs/issues/45462 is fixed.
+1 -7
View File
@@ -178,16 +178,10 @@ MixFlakeOptions::MixFlakeOptions()
for (auto & [inputName, input] : flake.lockFile.root->inputs) {
auto input2 = flake.lockFile.findInput({inputName}); // resolve 'follows' nodes
if (auto input3 = std::dynamic_pointer_cast<const flake::LockedNode>(input2)) {
fetchers::Attrs extraAttrs;
if (!input3->lockedRef.subdir.empty()) {
extraAttrs["dir"] = input3->lockedRef.subdir;
}
overrideRegistry(
fetchers::Input::fromAttrs(fetchSettings, {{"type", "indirect"}, {"id", inputName}}),
input3->lockedRef.input,
extraAttrs);
{});
}
}
}},
+1 -1
View File
@@ -669,7 +669,7 @@ ProcessLineResult NixRepl::processLine(std::string line)
ss << "No documentation found.\n\n";
}
auto markdown = ss.view();
auto markdown = toView(ss);
logger->cout(trim(renderMarkdownToTerminal(markdown)));
} else
+1 -1
View File
@@ -324,7 +324,7 @@ void SampleStack::saveProfile()
std::visit([&](auto && info) { info.symbolize(state, os, posCache); }, pos);
}
os << " " << count;
writeLine(profileFd.get(), os.str());
writeLine(profileFd.get(), std::move(os).str());
/* Clear ostringstream. */
os.str("");
os.clear();
+2 -2
View File
@@ -649,7 +649,7 @@ std::optional<EvalState::Doc> EvalState::getDoc(Value & v)
.name = name,
.arity = 0, // FIXME: figure out how deep by syntax only? It's not semantically useful though...
.args = {},
.doc = makeImmutableString(s.view()), // NOTE: memory leak when compiled without GC
.doc = makeImmutableString(toView(s)), // NOTE: memory leak when compiled without GC
};
}
if (isFunctor(v)) {
@@ -1845,7 +1845,7 @@ void ExprAssert::eval(EvalState & state, Env & env, Value & v)
if (!state.evalBool(env, cond, pos, "in the condition of the assert statement")) {
std::ostringstream out;
cond->show(state.symbols, out);
auto exprStr = out.view();
auto exprStr = toView(out);
if (auto eq = dynamic_cast<ExprOpEq *>(cond)) {
try {
+9 -4
View File
@@ -595,12 +595,17 @@ struct ExprOpNot : Expr
{ \
return pos; \
} \
};
}
MakeBinOp(ExprOpEq, "==") MakeBinOp(ExprOpNEq, "!=") MakeBinOp(ExprOpAnd, "&&") MakeBinOp(ExprOpOr, "||")
MakeBinOp(ExprOpImpl, "->") MakeBinOp(ExprOpUpdate, "//") MakeBinOp(ExprOpConcatLists, "++")
MakeBinOp(ExprOpEq, "==");
MakeBinOp(ExprOpNEq, "!=");
MakeBinOp(ExprOpAnd, "&&");
MakeBinOp(ExprOpOr, "||");
MakeBinOp(ExprOpImpl, "->");
MakeBinOp(ExprOpUpdate, "//");
MakeBinOp(ExprOpConcatLists, "++");
struct ExprConcatStrings : Expr
struct ExprConcatStrings : Expr
{
PosIdx pos;
bool forceString;
+1 -10
View File
@@ -40,10 +40,7 @@ endforeach
boost = dependency(
'boost',
modules : [
'container',
'context',
],
modules : [ 'container', 'context' ],
include_type : 'system',
)
# boost is a public dependency, but not a pkg-config dependency unfortunately, so we
@@ -74,12 +71,6 @@ toml11 = dependency(
method : 'cmake',
include_type : 'system',
)
configdata_priv.set(
'HAVE_TOML11_4',
toml11.version().version_compare('>= 4.0.0').to_int(),
)
deps_other += toml11
config_priv_h = configure_file(
+4 -9
View File
@@ -1939,13 +1939,8 @@ static void prim_dirOf(EvalState & state, const PosIdx pos, Value ** args, Value
NixStringContext context;
auto path = state.coerceToString(
pos, *args[0], context, "while evaluating the first argument passed to 'builtins.dirOf'", false, false);
auto pos = path->rfind('/');
if (pos == path->npos)
v.mkStringMove(".", context);
else if (pos == 0)
v.mkStringMove("/", context);
else
v.mkString(path->substr(0, pos), context);
auto dir = dirOf(*path);
v.mkString(dir, context);
}
}
@@ -2355,7 +2350,7 @@ static void prim_toXML(EvalState & state, const PosIdx pos, Value ** args, Value
std::ostringstream out;
NixStringContext context;
printValueAsXML(state, true, false, *args[0], out, context, pos);
v.mkString(out.view(), context);
v.mkString(toView(out), context);
}
static RegisterPrimOp primop_toXML({
@@ -2463,7 +2458,7 @@ static void prim_toJSON(EvalState & state, const PosIdx pos, Value ** args, Valu
std::ostringstream out;
NixStringContext context;
printValueAsJSON(state, true, *args[0], pos, out, context);
v.mkString(out.view(), context);
v.mkString(toView(out), context);
}
static RegisterPrimOp primop_toJSON({
+2 -2
View File
@@ -113,7 +113,7 @@ static void fetchTree(
auto s = state.coerceToString(attr.pos, *attr.value, context, "", false, false).toOwned();
attrs.emplace(
state.symbols[attr.name],
params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s) : s);
params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s).to_string() : s);
} else if (attr.value->type() == nBool)
attrs.emplace(state.symbols[attr.name], Explicit<bool>{attr.value->boolean()});
else if (attr.value->type() == nInt) {
@@ -175,7 +175,7 @@ static void fetchTree(
if (params.isFetchGit) {
fetchers::Attrs attrs;
attrs.emplace("type", "git");
attrs.emplace("url", fixGitURL(url));
attrs.emplace("url", fixGitURL(url).to_string());
if (!attrs.contains("exportIgnore")
&& (!attrs.contains("submodules") || !*fetchers::maybeGetBoolAttr(attrs, "submodules"))) {
attrs.emplace("exportIgnore", Explicit<bool>{true});
+23 -98
View File
@@ -1,145 +1,78 @@
#include "nix/expr/primops.hh"
#include "nix/expr/eval-inline.hh"
#include "expr-config-private.hh"
#include <sstream>
#include <toml.hpp>
namespace nix {
#if HAVE_TOML11_4
/**
* This is what toml11 < 4.0 did when choosing the subsecond precision.
* TOML 1.0.0 spec doesn't define how sub-millisecond ranges should be handled and calls it
* implementation defined behavior. For a lack of a better choice we stick with what older versions
* of toml11 did [1].
*
* [1]: https://github.com/ToruNiina/toml11/blob/dcfe39a783a94e8d52c885e5883a6fbb21529019/toml/datetime.hpp#L282
*/
static size_t normalizeSubsecondPrecision(toml::local_time lt)
{
auto millis = lt.millisecond;
auto micros = lt.microsecond;
auto nanos = lt.nanosecond;
if (millis != 0 || micros != 0 || nanos != 0) {
if (micros != 0 || nanos != 0) {
if (nanos != 0)
return 9;
return 6;
}
return 3;
}
return 0;
}
/**
* Normalize date/time formats to serialize to the same strings as versions prior to toml11 4.0.
*
* Several things to consider:
*
* 1. Sub-millisecond range is represented the same way as in toml11 versions prior to 4.0. Precisioun is rounded
* towards the next multiple of 3 or capped at 9 digits.
* 2. Seconds must be specified. This may become optional in (yet unreleased) TOML 1.1.0, but 1.0.0 defined local time
* in terms of RFC3339 [1].
* 3. date-time separator (`t`, `T` or space ` `) is canonicalized to an upper T. This is compliant with RFC3339
* [1] 5.6:
* > Applications that generate this format SHOULD use upper case letters.
*
* [1]: https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
*/
static void normalizeDatetimeFormat(toml::value & t)
{
if (t.is_local_datetime()) {
auto & ldt = t.as_local_datetime();
t.as_local_datetime_fmt() = {
.delimiter = toml::datetime_delimiter_kind::upper_T,
// https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
.has_seconds = true, // Mandated by TOML 1.0.0
.subsecond_precision = normalizeSubsecondPrecision(ldt.time),
};
return;
}
if (t.is_offset_datetime()) {
auto & odt = t.as_offset_datetime();
t.as_offset_datetime_fmt() = {
.delimiter = toml::datetime_delimiter_kind::upper_T,
// https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
.has_seconds = true, // Mandated by TOML 1.0.0
.subsecond_precision = normalizeSubsecondPrecision(odt.time),
};
return;
}
if (t.is_local_time()) {
auto & lt = t.as_local_time();
t.as_local_time_fmt() = {
.has_seconds = true, // Mandated by TOML 1.0.0
.subsecond_precision = normalizeSubsecondPrecision(lt),
};
return;
}
}
#endif
static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Value & val)
{
auto toml = state.forceStringNoCtx(*args[0], pos, "while evaluating the argument passed to builtins.fromTOML");
std::istringstream tomlStream(std::string{toml});
auto visit = [&](auto & self, Value & v, toml::value t) -> void {
std::function<void(Value &, toml::value)> visit;
visit = [&](Value & v, toml::value t) {
switch (t.type()) {
case toml::value_t::table: {
auto table = toml::get<toml::table>(t);
auto attrs = state.buildBindings(table.size());
size_t size = 0;
for (auto & i : table) {
(void) i;
size++;
}
auto attrs = state.buildBindings(size);
for (auto & elem : table) {
forceNoNullByte(elem.first);
self(self, attrs.alloc(elem.first), elem.second);
visit(attrs.alloc(elem.first), elem.second);
}
v.mkAttrs(attrs);
} break;
;
case toml::value_t::array: {
auto array = toml::get<std::vector<toml::value>>(t);
auto list = state.buildList(array.size());
for (const auto & [n, v] : enumerate(list))
self(self, *(v = state.allocValue()), array[n]);
visit(*(v = state.allocValue()), array[n]);
v.mkList(list);
} break;
;
case toml::value_t::boolean:
v.mkBool(toml::get<bool>(t));
break;
;
case toml::value_t::integer:
v.mkInt(toml::get<int64_t>(t));
break;
;
case toml::value_t::floating:
v.mkFloat(toml::get<NixFloat>(t));
break;
;
case toml::value_t::string: {
auto s = toml::get<std::string_view>(t);
forceNoNullByte(s);
v.mkString(s);
} break;
;
case toml::value_t::local_datetime:
case toml::value_t::offset_datetime:
case toml::value_t::local_date:
case toml::value_t::local_time: {
if (experimentalFeatureSettings.isEnabled(Xp::ParseTomlTimestamps)) {
#if HAVE_TOML11_4
normalizeDatetimeFormat(t);
#endif
auto attrs = state.buildBindings(2);
attrs.alloc("_type").mkString("timestamp");
std::ostringstream s;
s << t;
auto str = s.view();
auto str = toView(s);
forceNoNullByte(str);
attrs.alloc("value").mkString(str);
v.mkAttrs(attrs);
@@ -147,24 +80,16 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Va
throw std::runtime_error("Dates and times are not supported");
}
} break;
;
case toml::value_t::empty:
v.mkNull();
break;
;
}
};
try {
visit(
visit,
val,
toml::parse(
tomlStream,
"fromTOML" /* the "filename" */
#if HAVE_TOML11_4
,
toml::spec::v(1, 0, 0) // Be explicit that we are parsing TOML 1.0.0 without extensions
#endif
));
visit(val, toml::parse(tomlStream, "fromTOML" /* the "filename" */));
} catch (std::exception & e) { // TODO: toml::syntax_error
state.error<EvalError>("while parsing TOML: %s", e.what()).atPos(pos).debugThrow();
}
+1 -1
View File
@@ -460,7 +460,7 @@ private:
std::ostringstream s;
s << state.positions[v.lambda().fun->pos];
output << " @ " << filterANSIEscapes(s.view());
output << " @ " << filterANSIEscapes(toView(s));
}
} else if (v.isPrimOp()) {
if (v.primOp())
-6
View File
@@ -175,12 +175,6 @@ TEST_F(GitUtilsTest, peel_reference)
TEST(GitUtils, isLegalRefName)
{
ASSERT_TRUE(isLegalRefName("A/b"));
ASSERT_TRUE(isLegalRefName("AaA/b"));
ASSERT_TRUE(isLegalRefName("FOO/BAR/BAZ"));
ASSERT_TRUE(isLegalRefName("HEAD"));
ASSERT_TRUE(isLegalRefName("refs/tags/1.2.3"));
ASSERT_TRUE(isLegalRefName("refs/heads/master"));
ASSERT_TRUE(isLegalRefName("foox"));
ASSERT_TRUE(isLegalRefName("1337"));
ASSERT_TRUE(isLegalRefName("foo.baz"));
+3 -3
View File
@@ -25,7 +25,7 @@ static void downloadToSink(
std::string sha256Expected,
size_t sizeExpected)
{
FileTransferRequest request(url);
FileTransferRequest request(parseURL(url));
Headers headers;
if (authHeader.has_value())
headers.push_back({"Authorization", *authHeader});
@@ -179,7 +179,7 @@ Fetch::Fetch(git_repository * repo, git_oid rev)
const auto remoteUrl = lfs::getLfsEndpointUrl(repo);
this->url = nix::parseURL(nix::fixGitURL(remoteUrl)).canonicalise();
this->url = nix::fixGitURL(remoteUrl).canonicalise();
}
bool Fetch::shouldFetch(const CanonPath & path) const
@@ -207,7 +207,7 @@ std::vector<nlohmann::json> Fetch::fetchUrls(const std::vector<Pointer> & pointe
auto api = lfs::getLfsApi(this->url);
auto url = api.endpoint + "/objects/batch";
const auto & authHeader = api.authHeader;
FileTransferRequest request(url);
FileTransferRequest request(parseURL(url));
request.post = true;
Headers headers;
if (authHeader.has_value())
+43 -15
View File
@@ -12,7 +12,6 @@
#include <git2/attr.h>
#include <git2/blob.h>
#include <git2/branch.h>
#include <git2/commit.h>
#include <git2/config.h>
#include <git2/describe.h>
@@ -29,7 +28,6 @@
#include <git2/submodule.h>
#include <git2/sys/odb_backend.h>
#include <git2/sys/mempack.h>
#include <git2/tag.h>
#include <git2/tree.h>
#include <iostream>
@@ -1313,33 +1311,63 @@ GitRepo::WorkdirInfo GitRepo::getCachedWorkdirInfo(const std::filesystem::path &
return workdirInfo;
}
/**
* Checks that the git reference is valid and normalizes slash '/' sequences.
*
* Accepts shorthand references (one-level refnames are allowed).
*/
bool isValidRefNameAllowNormalizations(const std::string & refName)
{
/* Unfortunately libgit2 doesn't expose the limit in headers, but its internal
limit is also 1024. */
std::array<char, 1024> normalizedRefBuffer;
/* It would be nice to have a better API like git_reference_name_is_valid, but
* with GIT_REFERENCE_FORMAT_REFSPEC_SHORTHAND flag. libgit2 uses it internally
* but doesn't expose it in public headers [1].
* [1]:
* https://github.com/libgit2/libgit2/blob/9d5f1bacc23594c2ba324c8f0d41b88bf0e9ef04/src/libgit2/refs.c#L1362-L1365
*/
auto res = git_reference_normalize_name(
normalizedRefBuffer.data(),
normalizedRefBuffer.size(),
refName.c_str(),
GIT_REFERENCE_FORMAT_ALLOW_ONELEVEL | GIT_REFERENCE_FORMAT_REFSPEC_SHORTHAND);
return res == 0;
}
bool isLegalRefName(const std::string & refName)
{
initLibGit2();
/* Since `git_reference_normalize_name` is the best API libgit2 has for verifying
* reference names with shorthands (see comment in normalizeRefName), we need to
* ensure that exceptions to the validity checks imposed by normalization [1] are checked
* explicitly.
* [1]: https://git-scm.com/docs/git-check-ref-format#Documentation/git-check-ref-format.txt---normalize
*/
/* Check for cases that don't get rejected by libgit2.
* FIXME: libgit2 should reject this. */
if (refName == "@")
return false;
/* Leading slashes and consecutive slashes are stripped during normalizatiton. */
if (refName.starts_with('/') || refName.find("//") != refName.npos)
return false;
/* Refer to libgit2. */
if (!isValidRefNameAllowNormalizations(refName))
return false;
/* libgit2 doesn't barf on DEL symbol.
* FIXME: libgit2 should reject this. */
if (refName.find('\177') != refName.npos)
return false;
for (auto * func : {
git_reference_name_is_valid,
git_branch_name_is_valid,
git_tag_name_is_valid,
}) {
int valid = 0;
if (func(&valid, refName.c_str()))
throw Error("checking git reference '%s': %s", refName, git_error_last()->message);
if (valid)
return true;
}
return false;
return true;
}
} // namespace nix
+1 -3
View File
@@ -233,9 +233,7 @@ struct GitInputScheme : InputScheme
Input input{settings};
input.attrs = attrs;
auto url = fixGitURL(getStrAttr(attrs, "url"));
parseURL(url);
input.attrs["url"] = url;
input.attrs["url"] = fixGitURL(getStrAttr(attrs, "url")).to_string();
getShallowAttr(input);
getSubmodulesAttr(input);
getAllRefsAttr(input);
+4 -4
View File
@@ -19,7 +19,7 @@ namespace nix::fetchers {
struct DownloadUrl
{
std::string url;
ParsedURL url;
Headers headers;
};
@@ -420,7 +420,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
const auto url =
fmt(urlFmt, host, getOwner(input), getRepo(input), input.getRev()->to_string(HashFormat::Base16, false));
return DownloadUrl{url, headers};
return DownloadUrl{parseURL(url), headers};
}
void clone(const Input & input, const Path & destDir) const override
@@ -500,7 +500,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
return DownloadUrl{url, headers};
return DownloadUrl{parseURL(url), headers};
}
void clone(const Input & input, const Path & destDir) const override
@@ -592,7 +592,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
input.getRev()->to_string(HashFormat::Base16, false));
Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
return DownloadUrl{url, headers};
return DownloadUrl{parseURL(url), headers};
}
void clone(const Input & input, const Path & destDir) const override
@@ -158,12 +158,9 @@ struct Setter
};
/**
* Checks that the string can be a valid git reference, branch or tag name.
* Accepts shorthand references (one-level refnames are allowed), pseudorefs
* like `HEAD`.
* Checks that the git reference is valid and normalized.
*
* @note This is a coarse test to make sure that the refname is at least something
* that Git can make sense of.
* Accepts shorthand references (one-level refnames are allowed).
*/
bool isLegalRefName(const std::string & refName);
@@ -11,7 +11,6 @@ struct InputCache
ref<SourceAccessor> accessor;
Input resolvedInput;
Input lockedInput;
Attrs extraAttrs;
};
CachedResult getAccessor(ref<Store> store, const Input & originalInput, UseRegistries useRegistries);
@@ -20,7 +19,6 @@ struct InputCache
{
Input lockedInput;
ref<SourceAccessor> accessor;
Attrs extraAttrs;
};
virtual std::optional<CachedInput> lookup(const Input & originalInput) const = 0;
+2 -3
View File
@@ -22,8 +22,7 @@ InputCache::getAccessor(ref<Store> store, const Input & originalInput, UseRegist
fetched = lookup(resolvedInput);
if (!fetched) {
auto [accessor, lockedInput] = resolvedInput.getAccessor(store);
fetched.emplace(
CachedInput{.lockedInput = lockedInput, .accessor = accessor, .extraAttrs = extraAttrs});
fetched.emplace(CachedInput{.lockedInput = lockedInput, .accessor = accessor});
}
upsert(resolvedInput, *fetched);
} else {
@@ -37,7 +36,7 @@ InputCache::getAccessor(ref<Store> store, const Input & originalInput, UseRegist
debug("got tree '%s' from '%s'", fetched->accessor, fetched->lockedInput.to_string());
return {fetched->accessor, resolvedInput, fetched->lockedInput, fetched->extraAttrs};
return {fetched->accessor, resolvedInput, fetched->lockedInput};
}
struct InputCacheImpl : InputCache
+3 -3
View File
@@ -43,7 +43,7 @@ DownloadFileResult downloadFile(
if (cached && !cached->expired)
return useCached();
FileTransferRequest request(url);
FileTransferRequest request(parseURL(url));
request.headers = headers;
if (cached)
request.expectedETag = getStrAttr(cached->value, "etag");
@@ -115,7 +115,7 @@ static DownloadTarballResult downloadTarball_(
// it is not in fact a tarball.
if (url.rfind("file://", 0) == 0) {
// Remove "file://" prefix to get the local file path
std::string localPath = percentDecode(url.substr(7));
std::string localPath = url.substr(7);
if (!std::filesystem::exists(localPath)) {
throw Error("tarball '%s' does not exist.", localPath);
}
@@ -153,7 +153,7 @@ static DownloadTarballResult downloadTarball_(
auto _res = std::make_shared<Sync<FileTransferResult>>();
auto source = sinkToSource([&](Sink & sink) {
FileTransferRequest req(url);
FileTransferRequest req(parseURL(url));
req.expectedETag = cached ? getStrAttr(cached->value, "etag") : "";
getFileTransfer()->download(std::move(req), sink, [_res](FileTransferResult r) { *_res->lock() = r; });
});
+2 -3
View File
@@ -340,9 +340,8 @@ static Flake getFlake(
// Fetch a lazy tree first.
auto cachedInput = state.inputCache->getAccessor(state.store, originalRef.input, useRegistries);
auto subdir = fetchers::maybeGetStrAttr(cachedInput.extraAttrs, "dir").value_or(originalRef.subdir);
auto resolvedRef = FlakeRef(std::move(cachedInput.resolvedInput), subdir);
auto lockedRef = FlakeRef(std::move(cachedInput.lockedInput), subdir);
auto resolvedRef = FlakeRef(std::move(cachedInput.resolvedInput), originalRef.subdir);
auto lockedRef = FlakeRef(std::move(cachedInput.lockedInput), originalRef.subdir);
// Parse/eval flake.nix to get at the input.self attributes.
auto flake = readFlake(state, originalRef, resolvedRef, lockedRef, {cachedInput.accessor}, lockRootAttrPath);
+1 -2
View File
@@ -80,8 +80,7 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
std::smatch match;
auto succeeds = std::regex_match(url, match, pathFlakeRegex);
if (!succeeds)
throw Error("invalid flakeref '%s'", url);
assert(succeeds);
auto path = match[1].str();
auto query = decodeQuery(match[3].str(), /*lenient=*/true);
auto fragment = percentDecode(match[5].str());
+1 -1
View File
@@ -183,7 +183,7 @@ public:
std::ostringstream oss;
showErrorInfo(oss, ei, loggerSettings.showTrace.get());
log(*state, ei.level, oss.view());
log(*state, ei.level, toView(oss));
}
void log(State & state, Verbosity lvl, std::string_view s)
+27 -22
View File
@@ -320,29 +320,34 @@ int handleExceptions(const std::string & programName, std::function<void()> fun)
std::string error = ANSI_RED "error:" ANSI_NORMAL " ";
try {
try {
fun();
} catch (...) {
/* Subtle: we have to make sure that any `interrupted'
condition is discharged before we reach printMsg()
below, since otherwise it will throw an (uncaught)
exception. */
setInterruptThrown();
throw;
try {
fun();
} catch (...) {
/* Subtle: we have to make sure that any `interrupted'
condition is discharged before we reach printMsg()
below, since otherwise it will throw an (uncaught)
exception. */
setInterruptThrown();
throw;
}
} catch (Exit & e) {
return e.status;
} catch (UsageError & e) {
logError(e.info());
printError("Try '%1% --help' for more information.", programName);
return 1;
} catch (BaseError & e) {
logError(e.info());
return e.info().status;
} catch (std::bad_alloc & e) {
printError(error + "out of memory");
return 1;
} catch (std::exception & e) {
printError(error + e.what());
return 1;
}
} catch (Exit & e) {
return e.status;
} catch (UsageError & e) {
logError(e.info());
printError("Try '%1% --help' for more information.", programName);
return 1;
} catch (BaseError & e) {
logError(e.info());
return e.info().status;
} catch (std::bad_alloc & e) {
printError(error + "out of memory");
return 1;
} catch (std::exception & e) {
printError(error + e.what());
} catch (...) {
/* In case logger also throws just give up. */
return 1;
}
@@ -1 +0,0 @@
daemon
@@ -1 +0,0 @@
local
@@ -1 +0,0 @@
ssh://::1
@@ -1 +0,0 @@
ssh://userinfo@fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e
@@ -1 +0,0 @@
ssh://userinfo@fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e?a=b&c=d
@@ -1 +0,0 @@
ssh://userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%eth0]?a=b&c=d
@@ -1 +0,0 @@
ssh://userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%25eth0]?a=b&c=d
@@ -1 +0,0 @@
ssh://userinfo@fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%25?a=b&c=d
@@ -1 +0,0 @@
ssh://userinfo@fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%eth0?a=b&c=d
@@ -1 +0,0 @@
ssh://fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%eth0?a=b&c=d
@@ -1 +0,0 @@
ssh://fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%eth0
-6
View File
@@ -33,10 +33,4 @@ TEST(LocalStore, constructConfig_rootPath)
EXPECT_EQ(config.rootDir.get(), std::optional{"/foo/bar"});
}
TEST(LocalStore, constructConfig_to_string)
{
LocalStoreConfig config{"local", "", {}};
EXPECT_EQ(config.getReference().render(), "local");
}
} // namespace nix
+108 -8
View File
@@ -21,7 +21,7 @@ class ParsedS3URLTest : public ::testing::WithParamInterface<ParsedS3URLTestCase
TEST_P(ParsedS3URLTest, parseS3URLSuccessfully)
{
const auto & testCase = GetParam();
auto parsed = ParsedS3URL::parse(testCase.url);
auto parsed = ParsedS3URL::parse(parseURL(testCase.url));
ASSERT_EQ(parsed, testCase.expected);
}
@@ -35,7 +35,8 @@ INSTANTIATE_TEST_SUITE_P(
.bucket = "my-bucket",
.key = "my-key.txt",
},
"basic_s3_bucket"},
"basic_s3_bucket",
},
ParsedS3URLTestCase{
"s3://prod-cache/nix/store/abc123.nar.xz?region=eu-west-1",
{
@@ -43,7 +44,8 @@ INSTANTIATE_TEST_SUITE_P(
.key = "nix/store/abc123.nar.xz",
.region = "eu-west-1",
},
"with_region"},
"with_region",
},
ParsedS3URLTestCase{
"s3://bucket/key?region=us-west-2&profile=prod&endpoint=custom.s3.com&scheme=https&region=us-east-1",
{
@@ -54,7 +56,8 @@ INSTANTIATE_TEST_SUITE_P(
.scheme = "https",
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
},
"complex"},
"complex",
},
ParsedS3URLTestCase{
"s3://cache/file.txt?profile=production&region=ap-southeast-2",
{
@@ -63,7 +66,8 @@ INSTANTIATE_TEST_SUITE_P(
.profile = "production",
.region = "ap-southeast-2",
},
"with_profile_and_region"},
"with_profile_and_region",
},
ParsedS3URLTestCase{
"s3://bucket/key?endpoint=https://minio.local&scheme=http",
{
@@ -77,7 +81,8 @@ INSTANTIATE_TEST_SUITE_P(
.authority = ParsedURL::Authority{.host = "minio.local"},
},
},
"with_absolute_endpoint_uri"}),
"with_absolute_endpoint_uri",
}),
[](const ::testing::TestParamInfo<ParsedS3URLTestCase> & info) { return info.param.description; });
TEST(InvalidParsedS3URLTest, parseS3URLErrors)
@@ -86,11 +91,106 @@ TEST(InvalidParsedS3URLTest, parseS3URLErrors)
testing::HasSubstrIgnoreANSIMatcher("error: URI has a missing or invalid bucket name"));
/* Empty bucket (authority) */
ASSERT_THAT([]() { ParsedS3URL::parse("s3:///key"); }, invalidBucketMatcher);
ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3:///key")); }, invalidBucketMatcher);
/* Invalid bucket name */
ASSERT_THAT([]() { ParsedS3URL::parse("s3://127.0.0.1"); }, invalidBucketMatcher);
ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3://127.0.0.1")); }, invalidBucketMatcher);
}
// Parameterized test for s3ToHttpsUrl conversion
struct S3ToHttpsConversionTestCase
{
ParsedS3URL input;
ParsedURL expected;
std::string description;
};
class S3ToHttpsConversionTest : public ::testing::WithParamInterface<S3ToHttpsConversionTestCase>,
public ::testing::Test
{};
TEST_P(S3ToHttpsConversionTest, ConvertsCorrectly)
{
const auto & testCase = GetParam();
auto result = testCase.input.toHttpsUrl();
EXPECT_EQ(result, testCase.expected) << "Failed for: " << testCase.description;
}
INSTANTIATE_TEST_SUITE_P(
S3ToHttpsConversion,
S3ToHttpsConversionTest,
::testing::Values(
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "my-bucket",
.key = "my-key.txt",
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
.path = "/my-bucket/my-key.txt",
},
"basic_s3_default_region",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "prod-cache",
.key = "nix/store/abc123.nar.xz",
.region = "eu-west-1",
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
.path = "/prod-cache/nix/store/abc123.nar.xz",
},
"with_eu_west_1_region",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = "key",
.scheme = "http",
.endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "custom.s3.com"},
.path = "/bucket/key",
},
"custom_endpoint_authority",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = "key",
.endpoint =
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
},
},
ParsedURL{
.scheme = "http",
.authority = ParsedURL::Authority{.host = "server", .port = 9000},
.path = "/bucket/key",
},
"custom_endpoint_with_port",
},
S3ToHttpsConversionTestCase{
ParsedS3URL{
.bucket = "bucket",
.key = "path/to/file.txt",
.region = "ap-southeast-2",
.scheme = "https",
},
ParsedURL{
.scheme = "https",
.authority = ParsedURL::Authority{.host = "s3.ap-southeast-2.amazonaws.com"},
.path = "/bucket/path/to/file.txt",
},
"complex_path_and_region",
}),
[](const ::testing::TestParamInfo<S3ToHttpsConversionTestCase> & info) { return info.param.description; });
} // namespace nix
#endif
-109
View File
@@ -107,13 +107,6 @@ URI_TEST_READ(local_shorthand_1, localExample_1)
URI_TEST_READ(local_shorthand_2, localExample_2)
URI_TEST(
local_shorthand_3,
(StoreReference{
.variant = StoreReference::Local{},
.params = {},
}))
static StoreReference unixExample{
.variant =
StoreReference::Specified{
@@ -141,106 +134,4 @@ URI_TEST(
.params = {},
}))
URI_TEST(
daemon_shorthand,
(StoreReference{
.variant = StoreReference::Daemon{},
.params = {},
}))
static StoreReference sshLoopbackIPv6{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "[::1]",
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_1, sshLoopbackIPv6)
static StoreReference sshIPv6AuthorityWithUserinfo{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e]",
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_2, sshIPv6AuthorityWithUserinfo)
static StoreReference sshIPv6AuthorityWithUserinfoAndParams{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e]",
},
.params =
{
{"a", "b"},
{"c", "d"},
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_3, sshIPv6AuthorityWithUserinfoAndParams)
static const StoreReference sshIPv6AuthorityWithUserinfoAndParamsAndZoneId{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%25eth0]",
},
.params =
{
{"a", "b"},
{"c", "d"},
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_4, sshIPv6AuthorityWithUserinfoAndParamsAndZoneId)
URI_TEST_READ(ssh_unbracketed_ipv6_5, sshIPv6AuthorityWithUserinfoAndParamsAndZoneId)
static const StoreReference sshIPv6AuthorityWithUserinfoAndParamsAndZoneIdTricky{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "userinfo@[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%2525]",
},
.params =
{
{"a", "b"},
{"c", "d"},
},
};
// Non-standard syntax where the IPv6 literal appears without brackets. In
// this case don't considering %25 to be a pct-encoded % and just take it as a
// literal value. 25 is a perfectly legal ZoneId value in theory.
URI_TEST_READ(ssh_unbracketed_ipv6_6, sshIPv6AuthorityWithUserinfoAndParamsAndZoneIdTricky)
URI_TEST_READ(ssh_unbracketed_ipv6_7, sshIPv6AuthorityWithUserinfoAndParamsAndZoneId)
static const StoreReference sshIPv6AuthorityWithParamsAndZoneId{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%25eth0]",
},
.params =
{
{"a", "b"},
{"c", "d"},
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_8, sshIPv6AuthorityWithParamsAndZoneId)
static const StoreReference sshIPv6AuthorityWithZoneId{
.variant =
StoreReference::Specified{
.scheme = "ssh",
.authority = "[fea5:23e1:3916:fc24:cb52:2837:2ecb:ea8e%25eth0]",
},
};
URI_TEST_READ(ssh_unbracketed_ipv6_9, sshIPv6AuthorityWithZoneId)
} // namespace nix
-6
View File
@@ -16,10 +16,4 @@ TEST(UDSRemoteStore, constructConfigWrongScheme)
EXPECT_THROW(UDSRemoteStoreConfig("http", "/tmp/socket", {}), UsageError);
}
TEST(UDSRemoteStore, constructConfig_to_string)
{
UDSRemoteStoreConfig config{"unix", "", {}};
EXPECT_EQ(config.getReference().render(), "daemon");
}
} // namespace nix
+10 -10
View File
@@ -758,16 +758,16 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
*localStoreP,
std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
DerivationBuilderParams{
drvPath,
buildMode,
buildResult,
*drv,
*drvOptions,
inputPaths,
initialOutputs,
std::move(defaultPathsInChroot),
std::move(finalEnv),
std::move(extraFiles),
.drvPath = drvPath,
.buildResult = buildResult,
.drv = *drv,
.drvOptions = *drvOptions,
.inputPaths = inputPaths,
.initialOutputs = initialOutputs,
.buildMode = buildMode,
.defaultPathsInChroot = std::move(defaultPathsInChroot),
.finalEnv = std::move(finalEnv),
.extraFiles = std::move(extraFiles),
});
}
+3 -12
View File
@@ -182,16 +182,8 @@ Goal::Co DerivationGoal::haveDerivation()
}
}
if (buildResult.success()) {
/* If the wanted output is not in builtOutputs (e.g., because it
was already valid and therefore not re-registered), we need to
add it ourselves to ensure we return the correct information. */
if (buildResult.builtOutputs.count(wantedOutput) == 0) {
debug(
"BUG! wanted output '%s' not in builtOutputs, working around by adding it manually", wantedOutput);
buildResult.builtOutputs = {{wantedOutput, assertPathValidity()}};
}
}
if (buildResult.success())
assert(buildResult.builtOutputs.count(wantedOutput) > 0);
}
co_return amDone(g->exitCode, g->ex);
@@ -280,10 +272,9 @@ Goal::Co DerivationGoal::repairClosure()
bmRepair));
}
bool haveWaitees = !waitees.empty();
co_await await(std::move(waitees));
if (haveWaitees) {
if (!waitees.empty()) {
trace("closure repaired");
if (nrFailed > 0)
throw Error(
+1 -1
View File
@@ -37,7 +37,7 @@ static void builtinFetchurl(const BuiltinBuilderContext & ctx)
auto fetch = [&](const std::string & url) {
auto source = sinkToSource([&](Sink & sink) {
FileTransferRequest request(url);
FileTransferRequest request(parseURL(url));
request.decompress = false;
auto decompressor = makeDecompressionSink(unpack && hasSuffix(mainUrl, ".xz") ? "xz" : "none", sink);
+1 -1
View File
@@ -102,7 +102,7 @@ struct TunnelLogger : public Logger
showErrorInfo(oss, ei, false);
StringSink buf;
buf << STDERR_NEXT << oss.view();
buf << STDERR_NEXT << toView(oss);
enqueueMsg(buf.s);
}
+6 -14
View File
@@ -99,17 +99,6 @@ DerivationOptions DerivationOptions::fromStructuredAttrs(
return fromStructuredAttrs(env, parsed ? &*parsed : nullptr);
}
static void flatten(const nlohmann::json & value, StringSet & res)
{
if (value.is_array())
for (auto & v : value)
flatten(v, res);
else if (value.is_string())
res.insert(value);
else
throw Error("'exportReferencesGraph' value is not an array or a string");
}
DerivationOptions
DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAttrs * parsed, bool shouldWarn)
{
@@ -230,9 +219,12 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
if (!e || !e->is_object())
return ret;
for (auto & [key, value] : getObject(*e)) {
StringSet ss;
flatten(value, ss);
ret.insert_or_assign(key, std::move(ss));
if (value.is_array())
ret.insert_or_assign(key, value);
else if (value.is_string())
ret.insert_or_assign(key, StringSet{value});
else
throw Error("'exportReferencesGraph' value is not an array or a string");
}
} else {
auto s = getOr(env, "exportReferencesGraph", "");
+9 -8
View File
@@ -100,7 +100,7 @@ struct curlFileTransfer : public FileTransfer
lvlTalkative,
actFileTransfer,
fmt("%sing '%s'", request.verb(), request.uri),
{request.uri},
{request.uri.to_string()},
request.parentAct)
, callback(std::move(callback))
, finalSink([this](std::string_view data) {
@@ -121,7 +121,7 @@ struct curlFileTransfer : public FileTransfer
this->result.data.append(data);
})
{
result.urls.push_back(request.uri);
result.urls.push_back(request.uri.to_string());
requestHeaders = curl_slist_append(requestHeaders, "Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
if (!request.expectedETag.empty())
@@ -350,7 +350,7 @@ struct curlFileTransfer : public FileTransfer
curl_easy_setopt(req, CURLOPT_DEBUGFUNCTION, TransferItem::debugCallback);
}
curl_easy_setopt(req, CURLOPT_URL, request.uri.c_str());
curl_easy_setopt(req, CURLOPT_URL, request.uri.to_string().c_str());
curl_easy_setopt(req, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(req, CURLOPT_MAXREDIRS, 10);
curl_easy_setopt(req, CURLOPT_NOSIGNAL, 1);
@@ -784,8 +784,8 @@ struct curlFileTransfer : public FileTransfer
void enqueueItem(std::shared_ptr<TransferItem> item)
{
if (item->request.data && !hasPrefix(item->request.uri, "http://") && !hasPrefix(item->request.uri, "https://"))
throw nix::Error("uploading to '%s' is not supported", item->request.uri);
if (item->request.data && item->request.uri.scheme != "http" && item->request.uri.scheme != "https")
throw nix::Error("uploading to '%s' is not supported", item->request.uri.to_string());
{
auto state(state_.lock());
@@ -801,7 +801,7 @@ struct curlFileTransfer : public FileTransfer
void enqueueFileTransfer(const FileTransferRequest & request, Callback<FileTransferResult> callback) override
{
/* Ugly hack to support s3:// URIs. */
if (hasPrefix(request.uri, "s3://")) {
if (request.uri.scheme == "s3") {
// FIXME: do this on a worker thread
try {
#if NIX_WITH_S3_SUPPORT
@@ -820,10 +820,11 @@ struct curlFileTransfer : public FileTransfer
if (!s3Res.data)
throw FileTransferError(NotFound, {}, "S3 object '%s' does not exist", request.uri);
res.data = std::move(*s3Res.data);
res.urls.push_back(request.uri);
res.urls.push_back(request.uri.to_string());
callback(std::move(res));
#else
throw nix::Error("cannot download '%s' because Nix is not built with S3 support", request.uri);
throw nix::Error(
"cannot download '%s' because Nix is not built with S3 support", request.uri.to_string());
#endif
} catch (...) {
callback.rethrow();
-1
View File
@@ -21,7 +21,6 @@
#include <random>
#include <climits>
#include <thread>
#include <errno.h>
#include <fcntl.h>
#include <sys/stat.h>
+8 -13
View File
@@ -154,22 +154,17 @@ protected:
FileTransferRequest makeRequest(const std::string & path)
{
/* FIXME path is not a path, but a full relative or absolute
/* Otherwise the last path fragment will get discarded. */
auto cacheUriWithTrailingSlash = config->cacheUri;
if (!cacheUriWithTrailingSlash.path.empty())
cacheUriWithTrailingSlash.path += "/";
/* path is not a path, but a full relative or absolute
URL, e.g. we've seen in the wild NARINFO files have a URL
field which is
`nar/15f99rdaf26k39knmzry4xd0d97wp6yfpnfk1z9avakis7ipb9yg.nar?hash=zphkqn2wg8mnvbkixnl2aadkbn0rcnfj`
(note the query param) and that gets passed here.
What should actually happen is that we have two parsed URLs
(if we support relative URLs), and then we combined them with
a URL `operator/` which would be like
`std::filesystem::path`'s equivalent operator, which properly
combines the the URLs, whether the right is relative or
absolute. */
return FileTransferRequest(
hasPrefix(path, "https://") || hasPrefix(path, "http://") || hasPrefix(path, "file://")
? path
: config->cacheUri.to_string() + "/" + path);
(note the query param) and that gets passed here. */
return FileTransferRequest(parseURLRelative(path, cacheUriWithTrailingSlash));
}
void getFile(const std::string & path, Sink & sink) override
@@ -93,32 +93,6 @@ struct DerivationBuilderParams
* `EnvEntry::nameOfPassAsFile` above.
*/
StringMap extraFiles;
DerivationBuilderParams(
const StorePath & drvPath,
const BuildMode & buildMode,
BuildResult & buildResult,
const Derivation & drv,
const DerivationOptions & drvOptions,
const StorePathSet & inputPaths,
std::map<std::string, InitialOutput> & initialOutputs,
PathsInChroot defaultPathsInChroot,
std::map<std::string, EnvEntry, std::less<>> finalEnv,
StringMap extraFiles)
: drvPath{drvPath}
, buildResult{buildResult}
, drv{drv}
, drvOptions{drvOptions}
, inputPaths{inputPaths}
, initialOutputs{initialOutputs}
, buildMode{buildMode}
, defaultPathsInChroot{std::move(defaultPathsInChroot)}
, finalEnv{std::move(finalEnv)}
, extraFiles{std::move(extraFiles)}
{
}
DerivationBuilderParams(DerivationBuilderParams &&) = default;
};
/**
+7 -11
View File
@@ -9,6 +9,7 @@
#include "nix/util/ref.hh"
#include "nix/util/configuration.hh"
#include "nix/util/serialise.hh"
#include "nix/util/url.hh"
namespace nix {
@@ -30,17 +31,9 @@ struct FileTransferSettings : Config
)",
{"binary-caches-parallel-connections"}};
/* Do not set this too low. On glibc, getaddrinfo() contains fallback code
paths that deal with ill-behaved DNS servers. Setting this too low
prevents some fallbacks from occurring.
See description of options timeout, single-request, single-request-reopen
in resolv.conf(5). Also see https://github.com/NixOS/nix/pull/13985 for
details on the interaction between getaddrinfo(3) behavior and libcurl
CURLOPT_CONNECTTIMEOUT. */
Setting<unsigned long> connectTimeout{
this,
15,
5,
"connect-timeout",
R"(
The timeout (in seconds) for establishing connections in the
@@ -78,7 +71,7 @@ extern const unsigned int RETRY_TIME_MS_DEFAULT;
struct FileTransferRequest
{
std::string uri;
ParsedURL uri;
Headers headers;
std::string expectedETag;
bool verifyTLS = true;
@@ -92,7 +85,7 @@ struct FileTransferRequest
std::string mimeType;
std::function<void(std::string_view data)> dataCallback;
FileTransferRequest(std::string_view uri)
FileTransferRequest(ParsedURL uri)
: uri(uri)
, parentAct(getCurActivity())
{
@@ -119,6 +112,9 @@ struct FileTransferResult
/**
* All URLs visited in the redirect chain.
*
* @note Intentionally strings and not `ParsedURL`s so we faithfully
* return what cURL gave us.
*/
std::vector<std::string> urls;
@@ -794,8 +794,6 @@ public:
"build-dir",
R"(
Override the `build-dir` store setting for all stores that have this setting.
See also the per-store [`build-dir`](@docroot@/store/types/local-store.md#store-local-store-build-dir) setting.
)"};
Setting<PathSet> allowedImpureHostPrefixes{
@@ -61,8 +61,6 @@ private:
> `build-dir` must not be set to a world-writable directory.
> Placing temporary build directories in a world-writable place allows other users to access or modify build data that is currently in use.
> This alone is merely an impurity, but combined with another factor this has allowed malicious derivations to escape the build sandbox.
See also the global [`build-dir`](@docroot@/command-ref/conf-file.md#conf-build-dir) setting.
)"};
public:
Path getBuildDir() const;
@@ -52,21 +52,7 @@ struct RestrictionContext
* Add 'path' to the set of paths that may be referenced by the
* outputs, and make it appear in the sandbox.
*/
void addDependency(const StorePath & path)
{
if (isAllowed(path))
return;
addDependencyImpl(path);
}
protected:
/**
* This is the underlying implementation to be defined. The caller
* will ensure that this is only called on newly added dependencies,
* and that idempotent calls are a no-op.
*/
virtual void addDependencyImpl(const StorePath & path) = 0;
virtual void addDependency(const StorePath & path) = 0;
};
/**
+7 -1
View File
@@ -74,7 +74,13 @@ struct ParsedS3URL
endpoint);
}
static ParsedS3URL parse(std::string_view uri);
static ParsedS3URL parse(const ParsedURL & uri);
/**
* Convert this ParsedS3URL to HTTPS ParsedURL for use with curl's AWS SigV4 authentication
*/
ParsedURL toHttpsUrl() const;
auto operator<=>(const ParsedS3URL & other) const = default;
};
@@ -64,29 +64,7 @@ struct StoreReference
auto operator<=>(const Specified & rhs) const = default;
};
/**
* Special case for `daemon` to avoid normalization.
*/
struct Daemon : Specified
{
Daemon()
: Specified({.scheme = "unix"})
{
}
};
/**
* Special case for `local` to avoid normalization.
*/
struct Local : Specified
{
Local()
: Specified({.scheme = "local"})
{
}
};
typedef std::variant<Auto, Specified, Daemon, Local> Variant;
typedef std::variant<Auto, Specified> Variant;
Variant variant;
+4 -9
View File
@@ -456,17 +456,12 @@ LocalStore::~LocalStore()
StoreReference LocalStoreConfig::getReference() const
{
auto params = getQueryParams();
/* Back-compatibility kludge. Tools like nix-output-monitor expect 'local'
and can't parse 'local://'. */
if (params.empty())
return {.variant = StoreReference::Local{}};
return {
.variant =
StoreReference::Specified{
.scheme = *uriSchemes().begin(),
},
.params = std::move(params),
.params = getQueryParams(),
};
}
@@ -1393,7 +1388,7 @@ bool LocalStore::verifyStore(bool checkContents, RepairFlag repair)
for (auto & link : DirectoryIterator{linksDir}) {
checkInterrupt();
auto name = link.path().filename();
printMsg(lvlTalkative, "checking contents of %s", name);
printMsg(lvlTalkative, "checking contents of '%s'", name);
PosixSourceAccessor accessor;
std::string hash = hashPath(
PosixSourceAccessor::createAtRoot(link.path()),
@@ -1401,10 +1396,10 @@ bool LocalStore::verifyStore(bool checkContents, RepairFlag repair)
HashAlgorithm::SHA256)
.first.to_string(HashFormat::Nix32, false);
if (hash != name.string()) {
printError("link %s was modified! expected hash %s, got '%s'", link.path(), name, hash);
printError("link '%s' was modified! expected hash '%s', got '%s'", link.path(), name, hash);
if (repair) {
std::filesystem::remove(link.path());
printInfo("removed link %s", link.path());
printInfo("removed link '%s'", link.path());
} else {
errors = true;
}
+1 -6
View File
@@ -101,12 +101,7 @@ subdir('nix-meson-build-support/libatomic')
boost = dependency(
'boost',
modules : [
'container',
# Shouldn't list, because can header-only, and Meson currently looks for libs
#'regex',
'url',
],
modules : [ 'container', 'regex' ],
include_type : 'system',
)
# boost is a public dependency, but not a pkg-config dependency unfortunately, so we
+8 -12
View File
@@ -202,7 +202,7 @@ void LocalStore::optimisePath_(
full. When that happens, it's fine to ignore it: we
just effectively disable deduplication of this
file. */
printInfo("cannot link %s to '%s': %s", linkPath, path, strerror(errno));
printInfo("cannot link '%s' to '%s': %s", linkPath, path, strerror(errno));
return;
}
@@ -216,11 +216,11 @@ void LocalStore::optimisePath_(
auto stLink = lstat(linkPath.string());
if (st.st_ino == stLink.st_ino) {
debug("'%1%' is already linked to %2%", path, linkPath);
debug("'%1%' is already linked to '%2%'", path, linkPath);
return;
}
printMsg(lvlTalkative, "linking '%1%' to %2%", path, linkPath);
printMsg(lvlTalkative, "linking '%1%' to '%2%'", path, linkPath);
/* Make the containing directory writable, but only if it's not
the store itself (we don't want or need to mess with its
@@ -234,7 +234,7 @@ void LocalStore::optimisePath_(
its timestamp back to 0. */
MakeReadOnly makeReadOnly(mustToggle ? dirOfPath : "");
std::filesystem::path tempLink = makeTempPath(config->realStoreDir.get(), ".tmp-link");
std::filesystem::path tempLink = fmt("%1%/.tmp-link-%2%-%3%", config->realStoreDir, getpid(), rand());
try {
std::filesystem::create_hard_link(linkPath, tempLink);
@@ -245,7 +245,7 @@ void LocalStore::optimisePath_(
systems). This is likely to happen with empty files.
Just shrug and ignore. */
if (st.st_size)
printInfo("%1% has maximum number of links", linkPath);
printInfo("'%1%' has maximum number of links", linkPath);
return;
}
throw;
@@ -255,18 +255,14 @@ void LocalStore::optimisePath_(
try {
std::filesystem::rename(tempLink, path);
} catch (std::filesystem::filesystem_error & e) {
{
std::error_code ec;
remove(tempLink, ec); /* Clean up after ourselves. */
if (ec)
printError("unable to unlink %1%: %2%", tempLink, ec.message());
}
std::filesystem::remove(tempLink);
printError("unable to unlink '%1%'", tempLink);
if (e.code() == std::errc::too_many_links) {
/* Some filesystems generate too many links on the rename,
rather than on the original link. (Probably it
temporarily increases the st_nlink field before
decreasing it again.) */
debug("%s has reached maximum number of links", linkPath);
debug("'%s' has reached maximum number of links", linkPath);
return;
}
throw;
+2
View File
@@ -64,6 +64,8 @@ mkMesonLibrary (finalAttrs: {
sqlite
]
++ lib.optional stdenv.hostPlatform.isLinux libseccomp
# There have been issues building these dependencies
++ lib.optional stdenv.hostPlatform.isDarwin darwin.apple_sdk.libs.sandbox
++ lib.optional withAWS aws-sdk-cpp;
propagatedBuildInputs = [
+43 -5
View File
@@ -1,6 +1,8 @@
#include "nix/store/s3.hh"
#include "nix/util/split.hh"
#include "nix/util/url.hh"
#include "nix/util/util.hh"
#include "nix/util/canon-path.hh"
namespace nix {
@@ -8,10 +10,8 @@ using namespace std::string_view_literals;
#if NIX_WITH_S3_SUPPORT
ParsedS3URL ParsedS3URL::parse(std::string_view uri)
ParsedS3URL ParsedS3URL::parse(const ParsedURL & parsed)
try {
auto parsed = parseURL(uri);
if (parsed.scheme != "s3"sv)
throw BadURL("URI scheme '%s' is not 's3'", parsed.scheme);
@@ -43,7 +43,7 @@ try {
auto endpoint = getOptionalParam("endpoint");
return ParsedS3URL{
.bucket = std::move(parsed.authority->host),
.bucket = parsed.authority->host,
.key = std::string{key},
.profile = getOptionalParam("profile"),
.region = getOptionalParam("region"),
@@ -62,10 +62,48 @@ try {
}(),
};
} catch (BadURL & e) {
e.addTrace({}, "while parsing S3 URI: '%s'", uri);
e.addTrace({}, "while parsing S3 URI: '%s'", parsed.to_string());
throw;
}
ParsedURL ParsedS3URL::toHttpsUrl() const
{
auto toView = [](const auto & x) { return std::string_view{x}; };
auto regionStr = region.transform(toView).value_or("us-east-1");
auto schemeStr = scheme.transform(toView).value_or("https");
// Handle endpoint configuration using std::visit
return std::visit(
overloaded{
[&](const std::monostate &) {
// No custom endpoint, use standard AWS S3 endpoint
return ParsedURL{
.scheme = std::string{schemeStr},
.authority = ParsedURL::Authority{.host = "s3." + regionStr + ".amazonaws.com"},
.path = (CanonPath::root / bucket / CanonPath(key)).abs(),
};
},
[&](const ParsedURL::Authority & auth) {
// Endpoint is just an authority (hostname/port)
return ParsedURL{
.scheme = std::string{schemeStr},
.authority = auth,
.path = (CanonPath::root / bucket / CanonPath(key)).abs(),
};
},
[&](const ParsedURL & endpointUrl) {
// Endpoint is already a ParsedURL (e.g., http://server:9000)
return ParsedURL{
.scheme = endpointUrl.scheme,
.authority = endpointUrl.authority,
.path = (CanonPath(endpointUrl.path) / bucket / CanonPath(key)).abs(),
};
},
},
endpoint);
}
#endif
} // namespace nix
+1 -1
View File
@@ -78,7 +78,7 @@ SSHMaster::SSHMaster(
oss << authority.host;
return std::move(oss).str();
}())
, fakeSSH(authority.to_string() == "localhost")
, fakeSSH(authority.host == "localhost")
, keyFile(keyFile)
, sshPublicHostKey(parsePublicHostKey(authority.host, sshPublicHostKey))
, useMaster(useMaster && !fakeSSH)
+1 -7
View File
@@ -817,13 +817,7 @@ makeCopyPathMessage(const StoreConfig & srcCfg, const StoreConfig & dstCfg, std:
auto isShorthand = [](const StoreReference & ref) {
/* At this point StoreReference **must** be resolved. */
const auto & specified = std::visit(
overloaded{
[](const StoreReference::Auto &) -> const StoreReference::Specified & { unreachable(); },
[](const StoreReference::Specified & specified) -> const StoreReference::Specified & {
return specified;
}},
ref.variant);
const auto & specified = std::get<StoreReference::Specified>(ref.variant);
const auto & scheme = specified.scheme;
return (scheme == "local" || scheme == "unix") && specified.authority.empty();
};
+12 -84
View File
@@ -1,12 +1,11 @@
#include <regex>
#include "nix/util/error.hh"
#include "nix/util/split.hh"
#include "nix/util/url.hh"
#include "nix/store/store-reference.hh"
#include "nix/util/file-system.hh"
#include "nix/util/util.hh"
#include <boost/url/ipv6_address.hpp>
namespace nix {
static bool isNonUriPath(const std::string & spec)
@@ -26,8 +25,6 @@ std::string StoreReference::render(bool withParams) const
std::visit(
overloaded{
[&](const StoreReference::Auto &) { res = "auto"; },
[&](const StoreReference::Daemon &) { res = "daemon"; },
[&](const StoreReference::Local &) { res = "local"; },
[&](const StoreReference::Specified & g) {
res = g.scheme;
res += "://";
@@ -44,29 +41,6 @@ std::string StoreReference::render(bool withParams) const
return res;
}
namespace {
struct SchemeAndAuthorityWithPath
{
std::string_view scheme;
std::string_view authority;
};
} // namespace
/**
* Return the 'scheme' and remove the '://' or ':' separator.
*/
static std::optional<SchemeAndAuthorityWithPath> splitSchemePrefixTo(std::string_view string)
{
auto scheme = splitPrefixTo(string, ':');
if (!scheme)
return std::nullopt;
splitPrefix(string, "//");
return SchemeAndAuthorityWithPath{.scheme = *scheme, .authority = string};
}
StoreReference StoreReference::parse(const std::string & uri, const StoreReference::Params & extraParams)
{
auto params = extraParams;
@@ -94,17 +68,21 @@ StoreReference StoreReference::parse(const std::string & uri, const StoreReferen
.params = std::move(params),
};
} else if (baseURI == "daemon") {
if (params.empty())
return {.variant = Daemon{}};
return {
.variant = Specified{.scheme = "unix", .authority = ""},
.variant =
Specified{
.scheme = "unix",
.authority = "",
},
.params = std::move(params),
};
} else if (baseURI == "local") {
if (params.empty())
return {.variant = Local{}};
return {
.variant = Specified{.scheme = "local", .authority = ""},
.variant =
Specified{
.scheme = "local",
.authority = "",
},
.params = std::move(params),
};
} else if (isNonUriPath(baseURI)) {
@@ -116,56 +94,6 @@ StoreReference StoreReference::parse(const std::string & uri, const StoreReferen
},
.params = std::move(params),
};
} else if (auto schemeAndAuthority = splitSchemePrefixTo(baseURI)) {
/* Back-compatibility shim to accept unbracketed IPv6 addresses after the scheme.
* Old versions of nix allowed that. Note that this is ambiguous and does not allow
* specifying the port number. For that the address must be bracketed, otherwise it's
* greedily assumed to be the part of the host address. */
auto authorityString = schemeAndAuthority->authority;
auto userinfo = splitPrefixTo(authorityString, '@');
/* Back-compat shim for ZoneId specifiers. Technically this isn't
* standard, but the expectation is this works with the old syntax
* for ZoneID specifiers. For the full story behind the fiasco that
* is ZoneID in URLs look at [^].
* [^]: https://datatracker.ietf.org/doc/html/draft-schinazi-httpbis-link-local-uri-bcp-03
*/
/* Fish out the internals from inside square brackets. It might be that the pct-sign is unencoded and that's
* why we failed to parse it previously. */
if (authorityString.starts_with('[') && authorityString.ends_with(']')) {
authorityString.remove_prefix(1);
authorityString.remove_suffix(1);
}
auto maybeBeforePct = splitPrefixTo(authorityString, '%');
bool hasZoneId = maybeBeforePct.has_value();
auto maybeZoneId = hasZoneId ? std::optional{authorityString} : std::nullopt;
std::string_view maybeIpv6S = maybeBeforePct.value_or(authorityString);
auto maybeIpv6 = boost::urls::parse_ipv6_address(maybeIpv6S);
if (maybeIpv6) {
std::string fixedAuthority;
if (userinfo) {
fixedAuthority += *userinfo;
fixedAuthority += '@';
}
fixedAuthority += '[';
fixedAuthority += maybeIpv6S;
if (maybeZoneId) {
fixedAuthority += "%25"; // pct-encoded percent character
fixedAuthority += *maybeZoneId;
}
fixedAuthority += ']';
return {
.variant =
Specified{
.scheme = std::string(schemeAndAuthority->scheme),
.authority = fixedAuthority,
},
.params = std::move(params),
};
}
}
}
+5 -6
View File
@@ -57,16 +57,15 @@ UDSRemoteStore::UDSRemoteStore(ref<const Config> config)
StoreReference UDSRemoteStoreConfig::getReference() const
{
/* We specifically return "daemon" here instead of "unix://" or "unix://${path}"
* to be more compatible with older versions of nix. Some tooling out there
* tries hard to parse store references and it might not be able to handle "unix://". */
if (path == settings.nixDaemonSocketFile)
return {.variant = StoreReference::Daemon{}};
return {
.variant =
StoreReference::Specified{
.scheme = *uriSchemes().begin(),
.authority = path,
// We return the empty string when the path looks like the
// default path, but we could also just return the path
// verbatim always, to be robust to overall config changes
// at the cost of some verbosity.
.authority = path == settings.nixDaemonSocketFile ? "" : path,
},
};
}
@@ -186,7 +186,7 @@ struct ChrootDerivationBuilder : virtual DerivationBuilderImpl
std::pair<Path, Path> addDependencyPrep(const StorePath & path)
{
DerivationBuilderImpl::addDependencyImpl(path);
DerivationBuilderImpl::addDependency(path);
debug("materialising '%s' in the sandbox", store.printStorePath(path));
+12 -8
View File
@@ -284,7 +284,7 @@ public:
protected:
void addDependencyImpl(const StorePath & path) override;
void addDependency(const StorePath & path) override;
/**
* Make a file owned by the builder.
@@ -652,17 +652,17 @@ static void handleChildException(bool sendException)
}
}
static void checkNotWorldWritable(std::filesystem::path path)
static bool checkNotWorldWritable(std::filesystem::path path)
{
while (true) {
auto st = lstat(path);
if (st.st_mode & S_IWOTH)
throw Error("Path %s is world-writable or a symlink. That's not allowed for security.", path);
return false;
if (path == path.parent_path())
break;
path = path.parent_path();
}
return;
return true;
}
void DerivationBuilderImpl::startBuilder()
@@ -700,12 +700,13 @@ void DerivationBuilderImpl::startBuilder()
createDirs(buildDir);
if (buildUser)
checkNotWorldWritable(buildDir);
if (buildUser && !checkNotWorldWritable(buildDir))
throw Error(
"Path %s or a parent directory is world-writable or a symlink. That's not allowed for security.", buildDir);
/* Create a temporary directory where the build will take
place. */
topTmpDir = createTempDir(buildDir, "nix", 0700);
topTmpDir = createTempDir(buildDir, "nix-build-" + std::string(drvPath.name()), 0700);
setBuildTmpDir();
assert(!tmpDir.empty());
@@ -1166,8 +1167,11 @@ void DerivationBuilderImpl::stopDaemon()
daemonSocket.close();
}
void DerivationBuilderImpl::addDependencyImpl(const StorePath & path)
void DerivationBuilderImpl::addDependency(const StorePath & path)
{
if (isAllowed(path))
return;
addedPaths.insert(path);
}
@@ -681,7 +681,7 @@ struct ChrootLinuxDerivationBuilder : ChrootDerivationBuilder, LinuxDerivationBu
DerivationBuilderImpl::killSandbox(getStats);
}
void addDependencyImpl(const StorePath & path) override
void addDependency(const StorePath & path) override
{
auto [source, target] = ChrootDerivationBuilder::addDependencyPrep(path);
@@ -16,7 +16,6 @@ R""(
; Allow DNS lookups.
(allow network-outbound (remote unix-socket (path-literal "/private/var/run/mDNSResponder")))
(allow mach-lookup (global-name "com.apple.SystemConfiguration.DNSConfiguration"))
; Allow access to trustd.
(allow mach-lookup (global-name "com.apple.trustd"))
-47
View File
@@ -1,47 +0,0 @@
#include "nix/util/archive.hh"
#include "nix/util/tests/characterization.hh"
#include "nix/util/tests/gmock-matchers.hh"
#include <gtest/gtest.h>
namespace nix {
namespace {
class NarTest : public CharacterizationTest
{
std::filesystem::path unitTestData = getUnitTestData() / "nars";
public:
std::filesystem::path goldenMaster(std::string_view testStem) const override
{
return unitTestData / (std::string(testStem) + ".nar");
}
};
class InvalidNarTest : public NarTest, public ::testing::WithParamInterface<std::tuple<std::string, std::string>>
{};
} // namespace
TEST_P(InvalidNarTest, throwsErrorMessage)
{
const auto & [name, message] = GetParam();
readTest(name, [&](const std::string & narContents) {
ASSERT_THAT(
[&]() {
StringSource source{narContents};
NullFileSystemObjectSink sink;
parseDump(sink, source);
},
::testing::ThrowsMessage<SerialisationError>(testing::HasSubstrIgnoreANSIMatcher(message)));
});
}
INSTANTIATE_TEST_SUITE_P(
NarTest,
InvalidNarTest,
::testing::Values(
std::pair{"invalid-tag-instead-of-contents", "bad archive: expected tag 'contents', got 'AAAAAAAA'"}));
} // namespace nix
-1
View File
@@ -44,7 +44,6 @@ config_priv_h = configure_file(
subdir('nix-meson-build-support/common')
sources = files(
'archive.cc',
'args.cc',
'base-n.cc',
'canon-path.cc',
+52 -42
View File
@@ -2,6 +2,7 @@
#include <rapidcheck/gtest.h>
#include "nix/util/strings.hh"
#include "nix/util/strings-inline.hh"
#include "nix/util/error.hh"
namespace nix {
@@ -271,113 +272,122 @@ TEST(tokenizeString, tokenizeSepEmpty)
* splitString
* --------------------------------------------------------------------------*/
TEST(splitString, empty)
{
Strings expected = {""};
using SplitStringTestContainerTypes = ::testing::
Types<std::vector<std::string>, std::vector<std::string_view>, std::list<std::string>, std::list<std::string_view>>;
ASSERT_EQ(splitString<Strings>("", " \t\n\r"), expected);
template<typename T>
class splitStringTest : public ::testing::Test
{};
TYPED_TEST_SUITE(splitStringTest, SplitStringTestContainerTypes);
TYPED_TEST(splitStringTest, empty)
{
TypeParam expected = {""};
EXPECT_EQ(splitString<TypeParam>("", " \t\n\r"), expected);
}
TEST(splitString, oneSep)
TYPED_TEST(splitStringTest, oneSep)
{
Strings expected = {"", ""};
TypeParam expected = {"", ""};
ASSERT_EQ(splitString<Strings>(" ", " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(" ", " \t\n\r"), expected);
}
TEST(splitString, twoSep)
TYPED_TEST(splitStringTest, twoSep)
{
Strings expected = {"", "", ""};
TypeParam expected = {"", "", ""};
ASSERT_EQ(splitString<Strings>(" \n", " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(" \n", " \t\n\r"), expected);
}
TEST(splitString, tokenizeSpacesWithSpaces)
TYPED_TEST(splitStringTest, tokenizeSpacesWithSpaces)
{
auto s = "foo bar baz";
Strings expected = {"foo", "bar", "baz"};
TypeParam expected = {"foo", "bar", "baz"};
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
}
TEST(splitString, tokenizeTabsWithDefaults)
TYPED_TEST(splitStringTest, tokenizeTabsWithDefaults)
{
auto s = "foo\tbar\tbaz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
Strings expected = {"foo", "bar", "baz"};
TypeParam expected = {"foo", "bar", "baz"};
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
}
TEST(splitString, tokenizeTabsSpacesWithDefaults)
TYPED_TEST(splitStringTest, tokenizeTabsSpacesWithDefaults)
{
auto s = "foo\t bar\t baz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
Strings expected = {"foo", "", "bar", "", "baz"};
TypeParam expected = {"foo", "", "bar", "", "baz"};
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
}
TEST(splitString, tokenizeTabsSpacesNewlineWithDefaults)
TYPED_TEST(splitStringTest, tokenizeTabsSpacesNewlineWithDefaults)
{
auto s = "foo\t\n bar\t\n baz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
Strings expected = {"foo", "", "", "bar", "", "", "baz"};
TypeParam expected = {"foo", "", "", "bar", "", "", "baz"};
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
}
TEST(splitString, tokenizeTabsSpacesNewlineRetWithDefaults)
TYPED_TEST(splitStringTest, tokenizeTabsSpacesNewlineRetWithDefaults)
{
auto s = "foo\t\n\r bar\t\n\r baz";
// Using it like this is weird, but shows the difference with tokenizeString, which also has this test
Strings expected = {"foo", "", "", "", "bar", "", "", "", "baz"};
TypeParam expected = {"foo", "", "", "", "bar", "", "", "", "baz"};
ASSERT_EQ(splitString<Strings>(s, " \t\n\r"), expected);
EXPECT_EQ(splitString<TypeParam>(s, " \t\n\r"), expected);
auto s2 = "foo \t\n\r bar \t\n\r baz";
Strings expected2 = {"foo", "", "", "", "", "bar", "", "", "", "", "baz"};
TypeParam expected2 = {"foo", "", "", "", "", "bar", "", "", "", "", "baz"};
ASSERT_EQ(splitString<Strings>(s2, " \t\n\r"), expected2);
EXPECT_EQ(splitString<TypeParam>(s2, " \t\n\r"), expected2);
}
TEST(splitString, tokenizeWithCustomSep)
TYPED_TEST(splitStringTest, tokenizeWithCustomSep)
{
auto s = "foo\n,bar\n,baz\n";
Strings expected = {"foo\n", "bar\n", "baz\n"};
TypeParam expected = {"foo\n", "bar\n", "baz\n"};
ASSERT_EQ(splitString<Strings>(s, ","), expected);
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
}
TEST(splitString, tokenizeSepAtStart)
TYPED_TEST(splitStringTest, tokenizeSepAtStart)
{
auto s = ",foo,bar,baz";
Strings expected = {"", "foo", "bar", "baz"};
TypeParam expected = {"", "foo", "bar", "baz"};
ASSERT_EQ(splitString<Strings>(s, ","), expected);
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
}
TEST(splitString, tokenizeSepAtEnd)
TYPED_TEST(splitStringTest, tokenizeSepAtEnd)
{
auto s = "foo,bar,baz,";
Strings expected = {"foo", "bar", "baz", ""};
TypeParam expected = {"foo", "bar", "baz", ""};
ASSERT_EQ(splitString<Strings>(s, ","), expected);
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
}
TEST(splitString, tokenizeSepEmpty)
TYPED_TEST(splitStringTest, tokenizeSepEmpty)
{
auto s = "foo,,baz";
Strings expected = {"foo", "", "baz"};
TypeParam expected = {"foo", "", "baz"};
ASSERT_EQ(splitString<Strings>(s, ","), expected);
EXPECT_EQ(splitString<TypeParam>(s, ","), expected);
}
// concatStringsSep sep . splitString sep = id if sep is 1 char
RC_GTEST_PROP(splitString, recoveredByConcatStringsSep, (const std::string & s))
RC_GTEST_TYPED_FIXTURE_PROP(splitStringTest, recoveredByConcatStringsSep, (const std::string & s))
{
RC_ASSERT(concatStringsSep("/", splitString<Strings>(s, "/")) == s);
RC_ASSERT(concatStringsSep("a", splitString<Strings>(s, "a")) == s);
RC_ASSERT(concatStringsSep("/", splitString<TypeParam>(s, "/")) == s);
RC_ASSERT(concatStringsSep("a", splitString<TypeParam>(s, "a")) == s);
}
/* ----------------------------------------------------------------------------
+188
View File
@@ -290,6 +290,194 @@ TEST(parseURL, parsesHttpUrlWithEmptyPort)
ASSERT_EQ("http://www.example.org/file.tar.gz?foo=bar", parsed.to_string());
}
/* ----------------------------------------------------------------------------
* parseURLRelative
* --------------------------------------------------------------------------*/
TEST(parseURLRelative, resolvesRelativePath)
{
ParsedURL base = parseURL("http://example.org/dir/page.html");
auto parsed = parseURLRelative("subdir/file.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
.path = "/dir/subdir/file.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, baseUrlIpv6AddressWithoutZoneId)
{
ParsedURL base = parseURL("http://[fe80::818c:da4d:8975:415c]/dir/page.html");
auto parsed = parseURLRelative("subdir/file.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::IPv6, .host = "fe80::818c:da4d:8975:415c"},
.path = "/dir/subdir/file.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, resolvesRelativePathIpv6AddressWithZoneId)
{
ParsedURL base = parseURL("http://[fe80::818c:da4d:8975:415c\%25enp0s25]:8080/dir/page.html");
auto parsed = parseURLRelative("subdir/file2.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = Authority{.hostType = HostType::IPv6, .host = "fe80::818c:da4d:8975:415c\%enp0s25", .port = 8080},
.path = "/dir/subdir/file2.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, resolvesRelativePathWithDot)
{
ParsedURL base = parseURL("http://example.org/dir/page.html");
auto parsed = parseURLRelative("./subdir/file.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
.path = "/dir/subdir/file.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, resolvesParentDirectory)
{
ParsedURL base = parseURL("http://example.org:234/dir/page.html");
auto parsed = parseURLRelative("../up.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org", .port = 234},
.path = "/up.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, replacesPathWithAbsoluteRelative)
{
ParsedURL base = parseURL("http://example.org/dir/page.html");
auto parsed = parseURLRelative("/rooted.txt", base);
ParsedURL expected{
.scheme = "http",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "example.org"},
.path = "/rooted.txt",
.query = {},
.fragment = "",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, keepsQueryAndFragmentFromRelative)
{
// But discard query params on base URL
ParsedURL base = parseURL("https://www.example.org/path/index.html?z=3");
auto parsed = parseURLRelative("other.html?x=1&y=2#frag", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/other.html",
.query = {{"x", "1"}, {"y", "2"}},
.fragment = "frag",
};
ASSERT_EQ(parsed, expected);
}
TEST(parseURLRelative, absOverride)
{
ParsedURL base = parseURL("http://example.org/path/page.html");
std::string_view abs = "https://127.0.0.1.org/secure";
auto parsed = parseURLRelative(abs, base);
auto parsedAbs = parseURL(abs);
ASSERT_EQ(parsed, parsedAbs);
}
TEST(parseURLRelative, absOverrideWithZoneId)
{
ParsedURL base = parseURL("http://example.org/path/page.html");
std::string_view abs = "https://[fe80::818c:da4d:8975:415c\%25enp0s25]/secure?foo=bar";
auto parsed = parseURLRelative(abs, base);
auto parsedAbs = parseURL(abs);
ASSERT_EQ(parsed, parsedAbs);
}
TEST(parseURLRelative, bothWithoutAuthority)
{
ParsedURL base = parseURL("mailto:mail-base@bar.baz?bcc=alice@asdf.com");
std::string_view over = "mailto:mail-override@foo.bar?subject=url-testing";
auto parsed = parseURLRelative(over, base);
auto parsedOverride = parseURL(over);
ASSERT_EQ(parsed, parsedOverride);
}
TEST(parseURLRelative, emptyRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"a b", "5 6"}, {"x y", "34"}},
.fragment = "",
};
EXPECT_EQ(base.fragment, "frag");
EXPECT_EQ(parsed, expected);
}
TEST(parseURLRelative, fragmentRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("#frag2", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"a b", "5 6"}, {"x y", "34"}},
.fragment = "frag2",
};
EXPECT_EQ(parsed, expected);
}
TEST(parseURLRelative, queryRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("?asdf\%20qwer=1\%202\%203", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"asdf qwer", "1 2 3"}},
.fragment = "",
};
EXPECT_EQ(parsed, expected);
}
TEST(parseURLRelative, queryFragmentRelative)
{
ParsedURL base = parseURL("https://www.example.org/path/index.html?a\%20b=5\%206&x\%20y=34#frag");
auto parsed = parseURLRelative("?asdf\%20qwer=1\%202\%203#frag2", base);
ParsedURL expected{
.scheme = "https",
.authority = ParsedURL::Authority{.hostType = HostType::Name, .host = "www.example.org"},
.path = "/path/index.html",
.query = {{"asdf qwer", "1 2 3"}},
.fragment = "frag2",
};
EXPECT_EQ(parsed, expected);
}
/* ----------------------------------------------------------------------------
* decodeQuery
* --------------------------------------------------------------------------*/
+2 -4
View File
@@ -187,10 +187,8 @@ static void parse(FileSystemObjectSink & sink, Source & source, const CanonPath
tag = getString();
}
if (tag != "contents")
throw badArchive("expected tag 'contents', got '%s'", tag);
parseContents(crf, source);
if (tag == "contents")
parseContents(crf, source);
expectTag(")");
});
+3 -10
View File
@@ -6,7 +6,6 @@
#include "nix/util/terminal.hh"
#include "nix/util/position.hh"
#include <cinttypes>
#include <iostream>
#include <optional>
#include "nix/util/serialise.hh"
@@ -437,19 +436,13 @@ void panic(std::string_view msg)
writeErr("\n\n" ANSI_RED "terminating due to unexpected unrecoverable internal error: " ANSI_NORMAL);
writeErr(msg);
writeErr("\n");
std::terminate();
abort();
}
void unreachable(std::source_location loc)
void panic(const char * file, int line, const char * func)
{
char buf[512];
int n = snprintf(
buf,
sizeof(buf),
"Unexpected condition in %s at %s:%" PRIuLEAST32,
loc.function_name(),
loc.file_name(),
loc.line());
int n = snprintf(buf, sizeof(buf), "Unexpected condition in %s at %s:%d", func, file, line);
if (n < 0)
panic("Unexpected condition and could not format error message");
panic(std::string_view(buf, std::min(static_cast<int>(sizeof(buf)), n)));
+5 -4
View File
@@ -730,10 +730,11 @@ Path makeTempPath(const Path & root, const Path & suffix)
void createSymlink(const Path & target, const Path & link)
{
std::error_code ec;
std::filesystem::create_symlink(target, link, ec);
if (ec)
throw SysError(ec.value(), "creating symlink '%1%' -> '%2%'", link, target);
try {
std::filesystem::create_symlink(target, link);
} catch (std::filesystem::filesystem_error & e) {
throw SysError("creating symlink '%1%' -> '%2%'", link, target);
}
}
void replaceSymlink(const std::filesystem::path & target, const std::filesystem::path & link)
+1 -2
View File
@@ -135,8 +135,7 @@ static Hash parseLowLevel(std::string_view rest, HashAlgorithm algo, DecodeNameP
e.addTrace({}, "While decoding hash '%s'", rest);
}
if (d.size() != res.hashSize)
throw BadHash(
"invalid %s hash '%s', length %d != expected length %d", pair.encodingName, rest, d.size(), res.hashSize);
throw BadHash("invalid %s hash '%s' %d %d", pair.encodingName, rest);
assert(res.hashSize);
memcpy(res.hash, d.data(), res.hashSize);
+10 -4
View File
@@ -22,7 +22,6 @@
#include <list>
#include <memory>
#include <optional>
#include <utility>
#include <sys/types.h>
#include <sys/stat.h>
@@ -300,16 +299,23 @@ using NativeSysError =
void throwExceptionSelfCheck();
/**
* Print a message and std::terminate().
* Print a message and abort().
*/
[[noreturn]]
void panic(std::string_view msg);
/**
* Print a basic error message with source position and std::terminate().
* Print a basic error message with source position and abort().
* Use the unreachable() macro to call this.
*/
[[noreturn]]
void panic(const char * file, int line, const char * func);
/**
* Print a basic error message with source position and abort().
*
* @note: This assumes that the logger is operational
*/
[[gnu::noinline, gnu::cold, noreturn]] void unreachable(std::source_location loc = std::source_location::current());
#define unreachable() (::nix::panic(__FILE__, __LINE__, __func__))
} // namespace nix
@@ -287,10 +287,6 @@ class AutoDelete
public:
AutoDelete();
AutoDelete(const std::filesystem::path & p, bool recursive = true);
AutoDelete(AutoDelete &&) = delete;
AutoDelete(const AutoDelete &) = delete;
AutoDelete & operator=(AutoDelete &&) = delete;
AutoDelete & operator=(const AutoDelete &) = delete;
~AutoDelete();
void cancel();
Loaded 100 of 130 files, more files were not shown because too many files have changed in this diff. Show more