Improve Git URI handling

Git URI can also support scp style links similar to git itself.

This change augments the function fixGitURL to better handle the scp
style urls through a minimal parser rather than regex which has been
found to be brittle.

* Support for IPV6 added
* New test cases added for fixGitURL
* Clearer documentation on purpose and goal of function
* More `std::string_view` for performance
* Update URL tests

Fixes #5958

Mostly undoes revert 4757487110599bbe9a287ead75741bba5436d52f
Adapted from commit 04ad66af5f

.mergify.yml

@@ -161,3 +161,14 @@ pull_request_rules:
         labels:
           - automatic backport
           - merge-queue
+
+  - name: backport patches to 2.31
+    conditions:
+      - label=backport 2.31-maintenance
+    actions:
+      backport:
+        branches:
+          - "2.31-maintenance"
+        labels:
+          - automatic backport
+          - merge-queue

.version

@@ -1 +1 @@
-2.31.0
+2.32.0

doc/manual/rl-next/build-cores-auto-detect.md

@@ -1,6 +0,0 @@
----
-synopsis: "`build-cores = 0` now auto-detects CPU cores"
-prs: [13402]
----
-
-When `build-cores` is set to `0`, nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.

doc/manual/rl-next/git-lfs-ssh.md

@@ -1,11 +0,0 @@
----
-synopsis: "Fix Git LFS SSH issues"
-prs: [13743]
-issues: [13337]
----
-
-Fixed some outstanding issues with Git LFS and SSH.
-
-* Added support for `NIX_SSHOPTS`.
-* Properly use the parsed port from URL.
-* Better use of the response of `git-lfs-authenticate` to determine API endpoint when the API is not exposed on port 443.

doc/manual/rl-next/port-in-store-uris.md

@@ -1,13 +0,0 @@
----
-synopsis: "Add support for user@address:port syntax in store URIs"
-prs: [3425]
-issues: [7044]
----
-
-It's now possible to specify the port used for the SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable. This has now been fixed.
-
-This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
-
-- `ssh://127.0.0.1:2222`
-- `ssh://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
-- `ssh-ng://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`

doc/manual/rl-next/rfc4007-zone-id-in-uri-rfc6874.md

@@ -1,6 +0,0 @@
----
-synopsis: "Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874"
-prs: [13445]
----
-
-Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.

doc/manual/source/SUMMARY.md.in

@@ -138,6 +138,7 @@
   - [Contributing](development/contributing.md)
 - [Releases](release-notes/index.md)
 {{#include ./SUMMARY-rl-next.md}}
+  - [Release 2.31 (2025-08-21)](release-notes/rl-2.31.md)
   - [Release 2.30 (2025-07-07)](release-notes/rl-2.30.md)
   - [Release 2.29 (2025-05-14)](release-notes/rl-2.29.md)
   - [Release 2.28 (2025-04-02)](release-notes/rl-2.28.md)

doc/manual/source/development/building.md

@@ -34,7 +34,7 @@ $ nix-shell --attr devShells.x86_64-linux.native-clangStdenvPackages
 To build Nix itself in this shell:
 
 ```console
-[nix-shell]$ mesonFlags+=" --prefix=$(pwd)/outputs/out"
+[nix-shell]$ out="$(pwd)/outputs/out" dev=$out debug=$out mesonFlags+=" --prefix=${out}"
 [nix-shell]$ dontAddPrefix=1 configurePhase
 [nix-shell]$ buildPhase
 ```

doc/manual/source/release-notes/rl-2.31.md

@@ -0,0 +1,96 @@
+# Release 2.31.0 (2025-08-21)
+
+- `build-cores = 0` now auto-detects CPU cores [#13402](https://github.com/NixOS/nix/pull/13402)
+
+  When `build-cores` is set to `0`, Nix now automatically detects the number of available CPU cores and passes this value via `NIX_BUILD_CORES`, instead of passing `0` directly. This matches the behavior when `build-cores` is unset. This prevents the builder from having to detect the number of cores.
+
+- Fix Git LFS SSH issues [#13337](https://github.com/NixOS/nix/issues/13337) [#13743](https://github.com/NixOS/nix/pull/13743)
+
+  Fixed some outstanding issues with Git LFS and SSH.
+
+  * Added support for `NIX_SSHOPTS`.
+  * Properly use the parsed port from URL.
+  * Better use of the response of `git-lfs-authenticate` to determine API endpoint when the API is not exposed on port 443.
+
+- Add support for `user@address:port` syntax in store URIs [#7044](https://github.com/NixOS/nix/issues/7044) [#3425](https://github.com/NixOS/nix/pull/3425)
+
+  It's now possible to specify the port used for SSH stores directly in the store URL in accordance with [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986). Previously the only way to specify custom ports was via `ssh_config` or the `NIX_SSHOPTS` environment variable, because Nix incorrectly passed the port number together with the host name to the SSH executable.
+
+  This change affects [store references](@docroot@/store/types/index.md#store-url-format) passed via the `--store` and similar flags in CLI as well as in the configuration for [remote builders](@docroot@/command-ref/conf-file.md#conf-builders). For example, the following store URIs now work:
+
+  - `ssh://127.0.0.1:2222`
+  - `ssh://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
+  - `ssh-ng://[b573:6a48:e224:840b:6007:6275:f8f7:ebf3]:22`
+
+- Represent IPv6 RFC4007 ZoneId literals in conformance with RFC6874 [#13445](https://github.com/NixOS/nix/pull/13445)
+
+  Prior versions of Nix since [#4646](https://github.com/NixOS/nix/pull/4646) accepted [IPv6 scoped addresses](https://datatracker.ietf.org/doc/html/rfc4007) in URIs like [store references](@docroot@/store/types/index.md#store-url-format) in the textual representation with a literal percent character: `[fe80::1%18]`. This was ambiguous, because the the percent literal `%` is reserved by [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986), since it's used to indicate percent encoding. Nix now requires that the percent `%` symbol is percent-encoded as `%25`. This implements [RFC6874](https://datatracker.ietf.org/doc/html/rfc6874), which defines the representation of zone identifiers in URIs. The example from above now has to be specified as `[fe80::1%2518]`.
+
+- Use WAL mode for SQLite cache databases [#13800](https://github.com/NixOS/nix/pull/13800)
+
+  Previously, Nix used SQLite's "truncate" mode for caches. However, this could cause a Nix process to block if another process was updating the cache. This was a problem for the flake evaluation cache in particular, since it uses long-running transactions. Thus, concurrent Nix commands operating on the same flake could be blocked for an unbounded amount of time. WAL mode avoids this problem.
+
+  This change required updating the versions of the SQLite caches. For instance, `eval-cache-v5.sqlite` is now `eval-cache-v6.sqlite`.
+
+- Enable parallel marking in bdwgc [#13708](https://github.com/NixOS/nix/pull/13708)
+
+  Previously marking was done by only one thread, which takes a long time if the heap gets big. Enabling parallel marking speeds up evaluation a lot, for example (on a Ryzen 9 5900X 12-Core):
+
+  * `nix search nixpkgs` from 24.3s to 18.9s.
+  * Evaluating the `NixOS/nix/2.21.2` flake regression test from 86.1s to 71.2s.
+
+- New command `nix flake prefetch-inputs` [#13565](https://github.com/NixOS/nix/pull/13565)
+
+  This command fetches all inputs of a flake in parallel. This can be a lot faster than the serialized on-demand fetching during regular flake evaluation. The downside is that it may fetch inputs that aren't normally used.
+
+- Add `warn-short-path-literals` setting [#13489](https://github.com/NixOS/nix/pull/13489)
+
+  This setting, when enabled, causes Nix to emit warnings when encountering relative path literals that don't start with `.` or `/`, for instance suggesting that `foo/bar` should be rewritten to `./foo/bar`.
+
+- When updating a lock, respect the input's lock file [#13437](https://github.com/NixOS/nix/pull/13437)
+
+  For example, if a flake has a lock for `a` and `a/b`, and we change the flakeref for `a`, previously Nix would fetch the latest version of `b` rather than using the lock for `b` from `a`.
+
+- Implement support for Git hashing with SHA-256 [#13543](https://github.com/NixOS/nix/pull/13543)
+
+  The experimental support for [Git-hashing](@docroot@/development/experimental-features.md#xp-feature-git-hashing) store objects now also includes support for SHA-256, not just SHA-1, in line with upstream Git.
+
+## Contributors
+
+This release was made possible by the following 34 contributors:
+
+- John Soo [**(@jsoo1)**](https://github.com/jsoo1)
+- Alan Urmancheev [**(@alurm)**](https://github.com/alurm)
+- Manse [**(@PedroManse)**](https://github.com/PedroManse)
+- Pol Dellaiera [**(@drupol)**](https://github.com/drupol)
+- DavHau [**(@DavHau)**](https://github.com/DavHau)
+- Leandro Emmanuel Reina Kiperman [**(@kip93)**](https://github.com/kip93)
+- h0nIg [**(@h0nIg)**](https://github.com/h0nIg)
+- Philip Taron [**(@philiptaron)**](https://github.com/philiptaron)
+- Eelco Dolstra [**(@edolstra)**](https://github.com/edolstra)
+- Connor Baker [**(@ConnorBaker)**](https://github.com/ConnorBaker)
+- kenji [**(@a-kenji)**](https://github.com/a-kenji)
+- Oleksandr Knyshuk [**(@k1gen)**](https://github.com/k1gen)
+- Maciej Krüger [**(@mkg20001)**](https://github.com/mkg20001)
+- Justin Bailey [**(@jgbailey-well)**](https://github.com/jgbailey-well)
+- Emily [**(@emilazy)**](https://github.com/emilazy)
+- Volker Diels-Grabsch [**(@vog)**](https://github.com/vog)
+- gustavderdrache [**(@gustavderdrache)**](https://github.com/gustavderdrache)
+- Elliot Cameron [**(@de11n)**](https://github.com/de11n)
+- Alexander V. Nikolaev [**(@avnik)**](https://github.com/avnik)
+- tomberek [**(@tomberek)**](https://github.com/tomberek)
+- Matthew Kenigsberg [**(@mkenigs)**](https://github.com/mkenigs)
+- Sergei Zimmerman [**(@xokdvium)**](https://github.com/xokdvium)
+- Cosima Neidahl [**(@OPNA2608)**](https://github.com/OPNA2608)
+- John Ericson [**(@Ericson2314)**](https://github.com/Ericson2314)
+- m4dc4p [**(@m4dc4p)**](https://github.com/m4dc4p)
+- Graham Christensen [**(@grahamc)**](https://github.com/grahamc)
+- Jason Yundt [**(@Jayman2000)**](https://github.com/Jayman2000)
+- Jens Petersen [**(@juhp)**](https://github.com/juhp)
+- the-sun-will-rise-tomorrow [**(@the-sun-will-rise-tomorrow)**](https://github.com/the-sun-will-rise-tomorrow)
+- Farid Zakaria [**(@fzakaria)**](https://github.com/fzakaria)
+- AGawas [**(@aln730)**](https://github.com/aln730)
+- Robert Hensing [**(@roberth)**](https://github.com/roberth)
+- Dmitry Bogatov [**(@KAction)**](https://github.com/KAction)
+- Jörg Thalheim [**(@Mic92)**](https://github.com/Mic92)
+- Philipp Otterbein

doc/manual/source/store/derivation/index.md

@@ -9,7 +9,7 @@ This is where Nix distinguishes itself.
 
 ## Store Derivation {#store-derivation}
 
-A derivation is a specification for running an executable on precisely defined input to produce on more [store objects][store object].
+A derivation is a specification for running an executable on precisely defined input to produce one or more [store objects][store object].
 These store objects are known as the derivation's *outputs*.
 
 Derivations are *built*, in which case the process is spawned according to the spec, and when it exits, required to leave behind files which will (after post-processing) become the outputs of the derivation.

docker.nix

@@ -281,7 +281,10 @@ let
 
           # may get replaced by pkgs.dockerTools.caCertificates
           mkdir -p $out/etc/ssl/certs
+          # Old NixOS compatibility.
           ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt $out/etc/ssl/certs
+          # NixOS canonical location
+          ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt $out/etc/ssl/certs/ca-certificates.crt
 
           cat $passwdContentsPath > $out/etc/passwd
           echo "" >> $out/etc/passwd

flake.lock

@@ -63,11 +63,11 @@
     },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1755442223,
-        "narHash": "sha256-VtMQg02B3kt1oejwwrGn50U9Xbjgzfbb5TV5Wtx8dKI=",
+        "lastModified": 1756178832,
+        "narHash": "sha256-O2CIn7HjZwEGqBrwu9EU76zlmA5dbmna7jL1XUmAId8=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "cd32a774ac52caaa03bcfc9e7591ac8c18617ced",
+        "rev": "d98ce345cdab58477ca61855540999c86577d19d",
         "type": "github"
       },
       "original": {

maintainers/README.md

@@ -46,7 +46,7 @@ The team meets twice a week (times are denoted in the [Europe/Amsterdam](https:/
        - mark it as draft if it is blocked on the contributor
        - escalate it back to the team by moving it to To discuss, and leaving a comment as to why the issue needs to be discussed again.
 
-- Work meeting: Mondays 14:00-16:00 Europe/Amsterdam see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
+- Work meeting: Mondays 18:00-20:00 Europe/Amsterdam; see [calendar](https://calendar.google.com/calendar/u/0/embed?src=b9o52fobqjak8oq8lfkhg3t0qg@group.calendar.google.com).
 
   1. Code review on pull requests from [In review](#in-review).
   2. Other chores and tasks.

maintainers/data/release-credits-email-to-handle.json

@@ -185,5 +185,23 @@
     "gwenn.lebihan7@gmail.com": "gwennlbh",
     "hey@ewen.works": "gwennlbh",
     "matt@sturgeon.me.uk": "MattSturgeon",
-    "pbsds@hotmail.com": "pbsds"
+    "pbsds@hotmail.com": "pbsds",
+    "sergei@zimmerman.foo": "xokdvium",
+    "v@njh.eu": "vog",
+    "pedro.manse@dmk3.com.br": "PedroManse",
+    "arnavgawas707@gmail.com": "aln730",
+    "mkg20001@gmail.com": "mkg20001",
+    "avn@avnik.info": "avnik",
+    "olk@disr.it": "k1gen",
+    "108410815+alurm@users.noreply.github.com": "alurm",
+    "kaction.cc@gmail.com": "KAction",
+    "juhpetersen@gmail.com": "juhp",
+    "opna2608@protonmail.com": "OPNA2608",
+    "jgbailey@gmail.com": "m4dc4p",
+    "justin.bailey@well.co": "jgbailey-well",
+    "130508846+de11n@users.noreply.github.com": "de11n",
+    "ConnorBaker01@Gmail.com": "ConnorBaker",
+    "jsoo1@asu.edu": "jsoo1",
+    "hsngrmpf+github@gmail.com": "DavHau",
+    "matthew@floxdev.com": "mkenigs"
 }

maintainers/data/release-credits-handle-to-name.json

@@ -162,5 +162,20 @@
     "pbsds": "Peder Bergebakken Sundt",
     "egorkonovalov": "Egor Konovalov",
     "jayeshv": "jayeshv",
-    "vcunat": "Vladim\u00edr \u010cun\u00e1t"
+    "vcunat": "Vladim\u00edr \u010cun\u00e1t",
+    "mkenigs": "Matthew Kenigsberg",
+    "alurm": "Alan Urmancheev",
+    "jgbailey-well": "Justin Bailey",
+    "k1gen": "Oleksandr Knyshuk",
+    "juhp": "Jens Petersen",
+    "de11n": "Elliot Cameron",
+    "jsoo1": "John Soo",
+    "m4dc4p": null,
+    "PedroManse": "Manse",
+    "OPNA2608": "Cosima Neidahl",
+    "mkg20001": "Maciej Kr\u00fcger",
+    "avnik": "Alexander V. Nikolaev",
+    "DavHau": null,
+    "aln730": "AGawas",
+    "vog": "Volker Diels-Grabsch"
 }

maintainers/release-notes-todo

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+# debug:
+# set -x
+
+START_REF="${1}"
+END_REF="${2:-upstream/master}"
+
+# Get the merge base
+MERGE_BASE=$(git merge-base "$START_REF" "$END_REF")
+unset START_REF
+
+# Get date range
+START_DATE=$(git show -s --format=%cI "$MERGE_BASE")
+END_DATE=$(git show -s --format=%cI "$END_REF")
+
+echo "Checking PRs merged between $START_DATE and $END_DATE" >&2
+
+# Get all commits between merge base and HEAD
+COMMITS=$(git rev-list "$MERGE_BASE..$END_REF")
+
+# Convert to set for fast lookup
+declare -A commit_set
+for commit in $COMMITS; do
+    commit_set["$commit"]=1
+done
+
+# Get the current changelog
+LOG_DONE="$(changelog-d doc/manual/rl-next)"
+is_done(){
+    local nr="$1"
+    echo "$LOG_DONE" | grep -E "^- .*/pull/$nr)"
+}
+
+# Query merged PRs in date range
+gh pr list \
+    --repo NixOS/nix \
+    --state merged \
+    --limit 1000 \
+    --json number,title,author,mergeCommit \
+    --search "merged:$START_DATE..$END_DATE" | \
+jq -r '.[] | [.number, .mergeCommit.oid, .title, .author.login] | @tsv' | \
+while IFS=$'\t' read -r pr_num merge_commit _title author; do
+    # Check if this PR's merge commit is in our branch
+    if [[ -n "${commit_set[$merge_commit]:-}" ]]; then
+        # Full detail, not suitable for comment due to mass ping and duplicate title
+        # echo "- #$pr_num $_title (@$author)"
+        echo "- #$pr_num ($author)"
+        if is_done "$pr_num"
+        then
+            echo "  - [x] has note"
+        else
+            echo "  - [ ] has note"
+        fi
+        echo "  - [ ] skip"
+    fi
+done

maintainers/release-process.md

@@ -24,11 +24,18 @@ release:
 * In a checkout of the Nix repo, make sure you're on `master` and run
   `git pull`.
 
+* Compile a release notes to-do list by running
+
+  ```console
+  $ ./maintainers/release-notes-todo PREV_RELEASE HEAD
+  ```
+
 * Compile the release notes by running
 
   ```console
   $ export VERSION=X.YY
   $ git checkout -b release-notes
+  $ export GITHUB_TOKEN=...
   $ ./maintainers/release-notes
   ```
 
@@ -126,6 +133,8 @@ release:
 
   Commit and push this to the maintenance branch.
 
+* Create a backport label.
+
 * Bump the version of `master`:
 
   ```console
@@ -133,6 +142,7 @@ release:
   $ git pull
   $ NEW_VERSION=2.13.0
   $ echo $NEW_VERSION > .version
+  $ ... edit .mergify.yml to add the previous version ...
   $ git checkout -b bump-$NEW_VERSION
   $ git commit -a -m 'Bump version'
   $ git push --set-upstream origin bump-$NEW_VERSION
@@ -140,10 +150,6 @@ release:
 
   Make a pull request and auto-merge it.
 
-* Create a backport label.
-
-* Add the new backport label to `.mergify.yml`.
-
 * Post an [announcement on Discourse](https://discourse.nixos.org/c/announcements/8), including the contents of
   `rl-$VERSION.md`.
 

packaging/dependencies.nix

@@ -76,6 +76,16 @@ scope: {
         prevAttrs.postInstall;
   });
 
+  toml11 = pkgs.toml11.overrideAttrs rec {
+    version = "4.4.0";
+    src = pkgs.fetchFromGitHub {
+      owner = "ToruNiina";
+      repo = "toml11";
+      tag = "v${version}";
+      hash = "sha256-sgWKYxNT22nw376ttGsTdg0AMzOwp8QH3E8mx0BZJTQ=";
+    };
+  };
+
   # TODO Hack until https://github.com/NixOS/nixpkgs/issues/45462 is fixed.
   boost =
     (pkgs.boost.override {

src/libcmd/common-eval-args.cc

@@ -15,6 +15,7 @@
 #include "nix/fetchers/fetch-to-store.hh"
 #include "nix/cmd/compatibility-settings.hh"
 #include "nix/expr/eval-settings.hh"
+#include "nix/store/globals.hh"
 
 namespace nix {
 

src/libcmd/installable-attr-path.cc

@@ -89,8 +89,7 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths()
     }
 
     DerivedPathsWithInfo res;
-    for (auto & [drvPath, outputs] : byDrvPath) {
-        state->waitForPath(drvPath);
+    for (auto & [drvPath, outputs] : byDrvPath)
         res.push_back({
             .path =
                 DerivedPath::Built{
@@ -103,7 +102,6 @@ DerivedPathsWithInfo InstallableAttrPath::toDerivedPaths()
                    so we can fill in this info. */
             }),
         });
-    }
 
     return res;
 }

src/libcmd/installable-flake.cc

@@ -102,12 +102,11 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths()
     }
 
     auto drvPath = attr->forceDerivation();
-    state->waitForPath(drvPath);
 
     std::optional<NixInt::Inner> priority;
 
-    if (attr->maybeGetAttr(state->sOutputSpecified)) {
-    } else if (auto aMeta = attr->maybeGetAttr(state->sMeta)) {
+    if (attr->maybeGetAttr(state->s.outputSpecified)) {
+    } else if (auto aMeta = attr->maybeGetAttr(state->s.meta)) {
         if (auto aPriority = aMeta->maybeGetAttr("priority"))
             priority = aPriority->getInt().value;
     }
@@ -120,12 +119,12 @@ DerivedPathsWithInfo InstallableFlake::toDerivedPaths()
                     overloaded{
                         [&](const ExtendedOutputsSpec::Default & d) -> OutputsSpec {
                             StringSet outputsToInstall;
-                            if (auto aOutputSpecified = attr->maybeGetAttr(state->sOutputSpecified)) {
+                            if (auto aOutputSpecified = attr->maybeGetAttr(state->s.outputSpecified)) {
                                 if (aOutputSpecified->getBool()) {
                                     if (auto aOutputName = attr->maybeGetAttr("outputName"))
                                         outputsToInstall = {aOutputName->getString()};
                                 }
-                            } else if (auto aMeta = attr->maybeGetAttr(state->sMeta)) {
+                            } else if (auto aMeta = attr->maybeGetAttr(state->s.meta)) {
                                 if (auto aOutputsToInstall = aMeta->maybeGetAttr("outputsToInstall"))
                                     for (auto & s : aOutputsToInstall->getListOfStrings())
                                         outputsToInstall.insert(s);

src/libcmd/repl.cc

@@ -333,7 +333,6 @@ StorePath NixRepl::getDerivationPath(Value & v)
     auto drvPath = packageInfo->queryDrvPath();
     if (!drvPath)
         throw Error("expression did not evaluate to a valid derivation (no 'drvPath' attribute)");
-    state->waitForPath(*drvPath);
     if (!state->store->isValidPath(*drvPath))
         throw Error("expression evaluated to invalid derivation '%s'", state->store->printStorePath(*drvPath));
     return *drvPath;

src/libexpr-c/nix_api_expr.cc

@@ -69,7 +69,6 @@ nix_err nix_expr_eval_from_string(
         nix::Expr * parsedExpr = state->state.parseExprFromString(expr, state->state.rootPath(nix::CanonPath(path)));
         state->state.eval(parsedExpr, value->value);
         state->state.forceValue(value->value, nix::noPos);
-        state->state.waitForAllPaths();
     }
     NIXC_CATCH_ERRS
 }
@@ -81,7 +80,6 @@ nix_err nix_value_call(nix_c_context * context, EvalState * state, Value * fn, n
     try {
         state->state.callFunction(fn->value, arg->value, value->value, nix::noPos);
         state->state.forceValue(value->value, nix::noPos);
-        state->state.waitForAllPaths();
     }
     NIXC_CATCH_ERRS
 }
@@ -94,7 +92,6 @@ nix_err nix_value_call_multi(
     try {
         state->state.callFunction(fn->value, {(nix::Value **) args, nargs}, value->value, nix::noPos);
         state->state.forceValue(value->value, nix::noPos);
-        state->state.waitForAllPaths();
     }
     NIXC_CATCH_ERRS
 }
@@ -105,7 +102,6 @@ nix_err nix_value_force(nix_c_context * context, EvalState * state, nix_value *
         context->last_err_code = NIX_OK;
     try {
         state->state.forceValue(value->value, nix::noPos);
-        state->state.waitForAllPaths();
     }
     NIXC_CATCH_ERRS
 }
@@ -116,7 +112,6 @@ nix_err nix_value_force_deep(nix_c_context * context, EvalState * state, nix_val
         context->last_err_code = NIX_OK;
     try {
         state->state.forceValueDeep(value->value);
-        state->state.waitForAllPaths();
     }
     NIXC_CATCH_ERRS
 }

src/libexpr-c/nix_api_value.cc

@@ -345,7 +345,6 @@ nix_value * nix_get_attr_byname(nix_c_context * context, const nix_value * value
         if (attr) {
             nix_gc_incref(nullptr, attr->value);
             state->state.forceValue(*attr->value, nix::noPos);
-            state->state.waitForAllPaths();
             return as_nix_value_ptr(attr->value);
         }
         nix_set_err_msg(context, NIX_ERR_KEY, "missing attribute");

src/libexpr-tests/value/print.cc

@@ -393,7 +393,7 @@ TEST_F(ValuePrintingTests, ansiColorsDerivation)
     vDerivation.mkString("derivation");
 
     BindingsBuilder builder(state, state.allocBindings(10));
-    builder.insert(state.sType, &vDerivation);
+    builder.insert(state.s.type, &vDerivation);
 
     Value vAttrs;
     vAttrs.mkAttrs(builder.finish());
@@ -438,8 +438,8 @@ TEST_F(ValuePrintingTests, ansiColorsDerivationError)
     vDerivation.mkString("derivation");
 
     BindingsBuilder builder(state, state.allocBindings(10));
-    builder.insert(state.sType, &vDerivation);
-    builder.insert(state.sDrvPath, &vError);
+    builder.insert(state.s.type, &vDerivation);
+    builder.insert(state.s.drvPath, &vError);
 
     Value vAttrs;
     vAttrs.mkAttrs(builder.finish());

src/libexpr/eval-cache.cc

@@ -4,6 +4,7 @@
 #include "nix/expr/eval.hh"
 #include "nix/expr/eval-inline.hh"
 #include "nix/store/store-api.hh"
+#include "nix/store/globals.hh"
 // Need specialization involving `SymbolStr` just in this one module.
 #include "nix/util/strings-inline.hh"
 
@@ -69,10 +70,10 @@ struct AttrDb
     {
         auto state(_state->lock());
 
-        Path cacheDir = getCacheDir() + "/eval-cache-v5";
+        auto cacheDir = std::filesystem::path(getCacheDir()) / "eval-cache-v6";
         createDirs(cacheDir);
 
-        Path dbPath = cacheDir + "/" + fingerprint.to_string(HashFormat::Base16, false) + ".sqlite";
+        auto dbPath = cacheDir / (fingerprint.to_string(HashFormat::Base16, false) + ".sqlite");
 
         state->db = SQLite(dbPath);
         state->db.isCache();
@@ -329,7 +330,7 @@ AttrCursor::AttrCursor(
 AttrKey AttrCursor::getKey()
 {
     if (!parent)
-        return {0, root->state.sEpsilon};
+        return {0, root->state.s.epsilon};
     if (!parent->first->cachedValue) {
         parent->first->cachedValue = root->db->getAttr(parent->first->getKey());
         assert(parent->first->cachedValue);
@@ -701,14 +702,13 @@ bool AttrCursor::isDerivation()
 
 StorePath AttrCursor::forceDerivation()
 {
-    auto aDrvPath = getAttr(root->state.sDrvPath);
+    auto aDrvPath = getAttr(root->state.s.drvPath);
     auto drvPath = root->state.store->parseStorePath(aDrvPath->getString());
     drvPath.requireDerivation();
     if (!root->state.store->isValidPath(drvPath) && !settings.readOnlyMode) {
         /* The eval cache contains 'drvPath', but the actual path has
            been garbage-collected. So force it to be regenerated. */
         aDrvPath->forceValue();
-        root->state.waitForPath(drvPath);
         if (!root->state.store->isValidPath(drvPath))
             throw Error(
                 "don't know how to recreate store derivation '%s'!", root->state.store->printStorePath(drvPath));

src/libexpr/eval-profiler.cc

@@ -185,7 +185,7 @@ FrameInfo SampleStack::getPrimOpFrameInfo(const PrimOp & primOp, std::span<Value
                 /* Error context strings don't actually matter, since we ignore all eval errors. */
                 state.forceAttrs(*args[0], pos, "");
                 auto attrs = args[0]->attrs();
-                auto nameAttr = state.getAttr(state.sName, attrs, "");
+                auto nameAttr = state.getAttr(state.s.name, attrs, "");
                 auto drvName = std::string(state.forceStringNoCtx(*nameAttr->value, pos, ""));
                 return DerivationStrictFrameInfo{.callPos = pos, .drvName = std::move(drvName)};
             } catch (...) {
@@ -211,7 +211,7 @@ FrameInfo SampleStack::getFrameInfoFromValueAndPos(const Value & v, std::span<Va
         /* Resolve primOp eagerly. Must not hold on to a reference to a Value. */
         return PrimOpFrameInfo{.expr = v.primOpAppPrimOp(), .callPos = pos};
     else if (state.isFunctor(v)) {
-        const auto functor = v.attrs()->get(state.sFunctor);
+        const auto functor = v.attrs()->get(state.s.functor);
         if (auto pos_ = posCache.lookup(pos); std::holds_alternative<std::monostate>(pos_.origin))
             /* HACK: In case callsite position is unresolved. */
             return FunctorFrameInfo{.pos = functor->pos};

src/libexpr/eval.cc

@@ -6,6 +6,7 @@
 #include "nix/util/exit.hh"
 #include "nix/util/types.hh"
 #include "nix/util/util.hh"
+#include "nix/util/environment-variables.hh"
 #include "nix/store/store-api.hh"
 #include "nix/store/derivations.hh"
 #include "nix/store/downstream-placeholder.hh"
@@ -21,7 +22,6 @@
 #include "nix/fetchers/fetch-to-store.hh"
 #include "nix/fetchers/tarball.hh"
 #include "nix/fetchers/input-cache.hh"
-#include "nix/store/async-path-writer.hh"
 
 #include "parser-tab.hh"
 
@@ -203,131 +203,71 @@ EvalState::EvalState(
     std::shared_ptr<Store> buildStore)
     : fetchSettings{fetchSettings}
     , settings{settings}
-    , sWith(symbols.create("<with>"))
-    , sOutPath(symbols.create("outPath"))
-    , sDrvPath(symbols.create("drvPath"))
-    , sType(symbols.create("type"))
-    , sMeta(symbols.create("meta"))
-    , sName(symbols.create("name"))
-    , sValue(symbols.create("value"))
-    , sSystem(symbols.create("system"))
-    , sOverrides(symbols.create("__overrides"))
-    , sOutputs(symbols.create("outputs"))
-    , sOutputName(symbols.create("outputName"))
-    , sIgnoreNulls(symbols.create("__ignoreNulls"))
-    , sFile(symbols.create("file"))
-    , sLine(symbols.create("line"))
-    , sColumn(symbols.create("column"))
-    , sFunctor(symbols.create("__functor"))
-    , sToString(symbols.create("__toString"))
-    , sRight(symbols.create("right"))
-    , sWrong(symbols.create("wrong"))
-    , sStructuredAttrs(symbols.create("__structuredAttrs"))
-    , sJson(symbols.create("__json"))
-    , sAllowedReferences(symbols.create("allowedReferences"))
-    , sAllowedRequisites(symbols.create("allowedRequisites"))
-    , sDisallowedReferences(symbols.create("disallowedReferences"))
-    , sDisallowedRequisites(symbols.create("disallowedRequisites"))
-    , sMaxSize(symbols.create("maxSize"))
-    , sMaxClosureSize(symbols.create("maxClosureSize"))
-    , sBuilder(symbols.create("builder"))
-    , sArgs(symbols.create("args"))
-    , sContentAddressed(symbols.create("__contentAddressed"))
-    , sImpure(symbols.create("__impure"))
-    , sOutputHash(symbols.create("outputHash"))
-    , sOutputHashAlgo(symbols.create("outputHashAlgo"))
-    , sOutputHashMode(symbols.create("outputHashMode"))
-    , sRecurseForDerivations(symbols.create("recurseForDerivations"))
-    , sDescription(symbols.create("description"))
-    , sSelf(symbols.create("self"))
-    , sEpsilon(symbols.create(""))
-    , sStartSet(symbols.create("startSet"))
-    , sOperator(symbols.create("operator"))
-    , sKey(symbols.create("key"))
-    , sPath(symbols.create("path"))
-    , sPrefix(symbols.create("prefix"))
-    , sOutputSpecified(symbols.create("outputSpecified"))
-    , exprSymbols{
-        .sub = symbols.create("__sub"),
-        .lessThan = symbols.create("__lessThan"),
-        .mul = symbols.create("__mul"),
-        .div = symbols.create("__div"),
-        .or_ = symbols.create("or"),
-        .findFile = symbols.create("__findFile"),
-        .nixPath = symbols.create("__nixPath"),
-        .body = symbols.create("body"),
-    }
+    , symbols(StaticEvalSymbols::staticSymbolTable())
     , repair(NoRepair)
     , emptyBindings(0)
-    , storeFS(
-        makeMountedSourceAccessor(
-            {
-                {CanonPath::root, makeEmptySourceAccessor()},
-                /* In the pure eval case, we can simply require
-                   valid paths. However, in the *impure* eval
-                   case this gets in the way of the union
-                   mechanism, because an invalid access in the
-                   upper layer will *not* be caught by the union
-                   source accessor, but instead abort the entire
-                   lookup.
+    , storeFS(makeMountedSourceAccessor({
+          {CanonPath::root, makeEmptySourceAccessor()},
+          /* In the pure eval case, we can simply require
+             valid paths. However, in the *impure* eval
+             case this gets in the way of the union
+             mechanism, because an invalid access in the
+             upper layer will *not* be caught by the union
+             source accessor, but instead abort the entire
+             lookup.
 
-                   This happens when the store dir in the
-                   ambient file system has a path (e.g. because
-                   another Nix store there), but the relocated
-                   store does not.
+             This happens when the store dir in the
+             ambient file system has a path (e.g. because
+             another Nix store there), but the relocated
+             store does not.
 
-                   TODO make the various source accessors doing
-                   access control all throw the same type of
-                   exception, and make union source accessor
-                   catch it, so we don't need to do this hack.
-                 */
-                {CanonPath(store->storeDir), store->getFSAccessor(settings.pureEval)},
-            }))
-    , rootFS(
-        ({
-            /* In pure eval mode, we provide a filesystem that only
-               contains the Nix store.
+             TODO make the various source accessors doing
+             access control all throw the same type of
+             exception, and make union source accessor
+             catch it, so we don't need to do this hack.
+           */
+          {CanonPath(store->storeDir), store->getFSAccessor(settings.pureEval)},
+      }))
+    , rootFS(({
+        /* In pure eval mode, we provide a filesystem that only
+           contains the Nix store.
 
-               If we have a chroot store and pure eval is not enabled,
-               use a union accessor to make the chroot store available
-               at its logical location while still having the
-               underlying directory available. This is necessary for
-               instance if we're evaluating a file from the physical
-               /nix/store while using a chroot store. */
-            auto accessor = getFSSourceAccessor();
+           If we have a chroot store and pure eval is not enabled,
+           use a union accessor to make the chroot store available
+           at its logical location while still having the
+           underlying directory available. This is necessary for
+           instance if we're evaluating a file from the physical
+           /nix/store while using a chroot store. */
+        auto accessor = getFSSourceAccessor();
 
-            auto realStoreDir = dirOf(store->toRealPath(StorePath::dummy));
-            if (settings.pureEval || store->storeDir != realStoreDir) {
-                accessor = settings.pureEval
-                    ? storeFS
-                    : makeUnionSourceAccessor({accessor, storeFS});
-            }
+        auto realStoreDir = dirOf(store->toRealPath(StorePath::dummy));
+        if (settings.pureEval || store->storeDir != realStoreDir) {
+            accessor = settings.pureEval ? storeFS : makeUnionSourceAccessor({accessor, storeFS});
+        }
 
-            /* Apply access control if needed. */
-            if (settings.restrictEval || settings.pureEval)
-                accessor = AllowListSourceAccessor::create(accessor, {}, {},
-                    [&settings](const CanonPath & path) -> RestrictedPathError {
-                        auto modeInformation = settings.pureEval
-                            ? "in pure evaluation mode (use '--impure' to override)"
-                            : "in restricted mode";
-                        throw RestrictedPathError("access to absolute path '%1%' is forbidden %2%", path, modeInformation);
-                    });
+        /* Apply access control if needed. */
+        if (settings.restrictEval || settings.pureEval)
+            accessor = AllowListSourceAccessor::create(
+                accessor, {}, {}, [&settings](const CanonPath & path) -> RestrictedPathError {
+                    auto modeInformation = settings.pureEval ? "in pure evaluation mode (use '--impure' to override)"
+                                                             : "in restricted mode";
+                    throw RestrictedPathError("access to absolute path '%1%' is forbidden %2%", path, modeInformation);
+                });
 
-            accessor;
-        }))
+        accessor;
+    }))
     , corepkgsFS(make_ref<MemorySourceAccessor>())
     , internalFS(make_ref<MemorySourceAccessor>())
     , derivationInternal{corepkgsFS->addFile(
-        CanonPath("derivation-internal.nix"),
+          CanonPath("derivation-internal.nix"),
 #include "primops/derivation.nix.gen.hh"
-    )}
+          )}
     , store(store)
     , buildStore(buildStore ? buildStore : store)
     , inputCache(fetchers::InputCache::create())
     , debugRepl(nullptr)
     , debugStop(false)
     , trylevel(0)
-    , asyncPathWriter(AsyncPathWriter::make(store))
     , regexCache(makeRegexCache())
 #if NIX_USE_BOEHMGC
     , valueAllocCache(std::allocate_shared<void *>(traceable_allocator<void *>(), nullptr))
@@ -655,7 +595,7 @@ std::optional<EvalState::Doc> EvalState::getDoc(Value & v)
     }
     if (isFunctor(v)) {
         try {
-            Value & functor = *v.attrs()->find(sFunctor)->value;
+            Value & functor = *v.attrs()->find(s.functor)->value;
             Value * vp[] = {&v};
             Value partiallyApplied;
             // The first parameter is not user-provided, and may be
@@ -979,8 +919,8 @@ void EvalState::mkPos(Value & v, PosIdx p)
     auto origin = positions.originOf(p);
     if (auto path = std::get_if<SourcePath>(&origin)) {
         auto attrs = buildBindings(3);
-        attrs.alloc(sFile).mkString(path->path.abs());
-        makePositionThunks(*this, p, attrs.alloc(sLine), attrs.alloc(sColumn));
+        attrs.alloc(s.file).mkString(path->path.abs());
+        makePositionThunks(*this, p, attrs.alloc(s.line), attrs.alloc(s.column));
         v.mkAttrs(attrs);
     } else
         v.mkNull();
@@ -1026,7 +966,6 @@ std::string EvalState::mkSingleDerivedPathStringRaw(const SingleDerivedPath & p)
                 auto optStaticOutputPath = std::visit(
                     overloaded{
                         [&](const SingleDerivedPath::Opaque & o) {
-                            waitForPath(o.path);
                             auto drv = store->readDerivation(o.path);
                             auto i = drv.outputs.find(b.output);
                             if (i == drv.outputs.end())
@@ -1247,7 +1186,7 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
         dynamicEnv = &env2;
         Env * inheritEnv = inheritFromExprs ? buildInheritFromEnv(state, env2) : nullptr;
 
-        AttrDefs::iterator overrides = attrs.find(state.sOverrides);
+        AttrDefs::iterator overrides = attrs.find(state.s.overrides);
         bool hasOverrides = overrides != attrs.end();
 
         /* The recursive attributes are evaluated in the new
@@ -1719,7 +1658,7 @@ void EvalState::callFunction(Value & fun, std::span<Value *> args, Value & vRes,
             }
         }
 
-        else if (vCur.type() == nAttrs && (functor = vCur.attrs()->get(sFunctor))) {
+        else if (vCur.type() == nAttrs && (functor = vCur.attrs()->get(s.functor))) {
             /* 'vCur' may be allocated on the stack of the calling
                function, but for functors we may keep a reference, so
                heap-allocate a copy and use that instead. */
@@ -1781,7 +1720,7 @@ void EvalState::autoCallFunction(const Bindings & args, Value & fun, Value & res
     forceValue(fun, pos);
 
     if (fun.type() == nAttrs) {
-        auto found = fun.attrs()->find(sFunctor);
+        auto found = fun.attrs()->find(s.functor);
         if (found != fun.attrs()->end()) {
             Value * v = allocValue();
             callFunction(*found->value, fun, *v, pos);
@@ -2243,7 +2182,7 @@ Bindings::const_iterator EvalState::getAttr(Symbol attrSym, const Bindings * att
 
 bool EvalState::isFunctor(const Value & fun) const
 {
-    return fun.type() == nAttrs && fun.attrs()->find(sFunctor) != fun.attrs()->end();
+    return fun.type() == nAttrs && fun.attrs()->find(s.functor) != fun.attrs()->end();
 }
 
 void EvalState::forceFunction(Value & v, const PosIdx pos, std::string_view errorCtx)
@@ -2312,7 +2251,7 @@ bool EvalState::isDerivation(Value & v)
 {
     if (v.type() != nAttrs)
         return false;
-    auto i = v.attrs()->get(sType);
+    auto i = v.attrs()->get(s.type);
     if (!i)
         return false;
     forceValue(*i->value, i->pos);
@@ -2324,7 +2263,7 @@ bool EvalState::isDerivation(Value & v)
 std::optional<std::string>
 EvalState::tryAttrsToString(const PosIdx pos, Value & v, NixStringContext & context, bool coerceMore, bool copyToStore)
 {
-    auto i = v.attrs()->find(sToString);
+    auto i = v.attrs()->find(s.toString);
     if (i != v.attrs()->end()) {
         Value v1;
         callFunction(*i->value, v, v1, pos);
@@ -2370,7 +2309,7 @@ BackedStringView EvalState::coerceToString(
         auto maybeString = tryAttrsToString(pos, v, context, coerceMore, copyToStore);
         if (maybeString)
             return std::move(*maybeString);
-        auto i = v.attrs()->find(sOutPath);
+        auto i = v.attrs()->find(s.outPath);
         if (i == v.attrs()->end()) {
             error<TypeError>(
                 "cannot coerce %1% to a string: %2%", showType(v), ValuePrinter(*this, v, errorPrintOptions))
@@ -2477,7 +2416,7 @@ SourcePath EvalState::coerceToPath(const PosIdx pos, Value & v, NixStringContext
     /* Similarly, handle __toString where the result may be a path
        value. */
     if (v.type() == nAttrs) {
-        auto i = v.attrs()->find(sToString);
+        auto i = v.attrs()->find(s.toString);
         if (i != v.attrs()->end()) {
             Value v1;
             callFunction(*i->value, v, v1, pos);
@@ -2667,8 +2606,8 @@ void EvalState::assertEqValues(Value & v1, Value & v2, const PosIdx pos, std::st
 
     case nAttrs: {
         if (isDerivation(v1) && isDerivation(v2)) {
-            auto i = v1.attrs()->get(sOutPath);
-            auto j = v2.attrs()->get(sOutPath);
+            auto i = v1.attrs()->get(s.outPath);
+            auto j = v2.attrs()->get(s.outPath);
             if (i && j) {
                 try {
                     assertEqValues(*i->value, *j->value, pos, errorCtx);
@@ -2821,8 +2760,8 @@ bool EvalState::eqValues(Value & v1, Value & v2, const PosIdx pos, std::string_v
         /* If both sets denote a derivation (type = "derivation"),
            then compare their outPaths. */
         if (isDerivation(v1) && isDerivation(v2)) {
-            auto i = v1.attrs()->get(sOutPath);
-            auto j = v2.attrs()->get(sOutPath);
+            auto i = v1.attrs()->get(s.outPath);
+            auto j = v2.attrs()->get(s.outPath);
             if (i && j)
                 return eqValues(*i->value, *j->value, pos, errorCtx);
         }
@@ -3198,8 +3137,7 @@ Expr * EvalState::parse(
         docComments = &it->second;
     }
 
-    auto result = parseExprFromBuf(
-        text, length, origin, basePath, symbols, settings, positions, *docComments, rootFS, exprSymbols);
+    auto result = parseExprFromBuf(text, length, origin, basePath, symbols, settings, positions, *docComments, rootFS);
 
     result->bindVars(*this, staticEnv);
 
@@ -3252,24 +3190,4 @@ void forceNoNullByte(std::string_view s, std::function<Pos()> pos)
     }
 }
 
-void EvalState::waitForPath(const StorePath & path)
-{
-    asyncPathWriter->waitForPath(path);
-}
-
-void EvalState::waitForPath(const SingleDerivedPath & path)
-{
-    std::visit(
-        overloaded{
-            [&](const DerivedPathOpaque & p) { waitForPath(p.path); },
-            [&](const SingleDerivedPathBuilt & p) { waitForPath(*p.drvPath); },
-        },
-        path.raw());
-}
-
-void EvalState::waitForAllPaths()
-{
-    asyncPathWriter->waitForAllPaths();
-}
-
 } // namespace nix

src/libexpr/get-drvs.cc

@@ -45,7 +45,7 @@ PackageInfo::PackageInfo(EvalState & state, ref<Store> store, const std::string
 std::string PackageInfo::queryName() const
 {
     if (name == "" && attrs) {
-        auto i = attrs->find(state->sName);
+        auto i = attrs->find(state->s.name);
         if (i == attrs->end())
             state->error<TypeError>("derivation name missing").debugThrow();
         name = state->forceStringNoCtx(*i->value, noPos, "while evaluating the 'name' attribute of a derivation");
@@ -56,7 +56,7 @@ std::string PackageInfo::queryName() const
 std::string PackageInfo::querySystem() const
 {
     if (system == "" && attrs) {
-        auto i = attrs->find(state->sSystem);
+        auto i = attrs->find(state->s.system);
         system =
             i == attrs->end()
                 ? "unknown"
@@ -68,7 +68,7 @@ std::string PackageInfo::querySystem() const
 std::optional<StorePath> PackageInfo::queryDrvPath() const
 {
     if (!drvPath && attrs) {
-        if (auto i = attrs->get(state->sDrvPath)) {
+        if (auto i = attrs->get(state->s.drvPath)) {
             NixStringContext context;
             auto found = state->coerceToStorePath(
                 i->pos, *i->value, context, "while evaluating the 'drvPath' attribute of a derivation");
@@ -95,7 +95,7 @@ StorePath PackageInfo::requireDrvPath() const
 StorePath PackageInfo::queryOutPath() const
 {
     if (!outPath && attrs) {
-        auto i = attrs->find(state->sOutPath);
+        auto i = attrs->find(state->s.outPath);
         NixStringContext context;
         if (i != attrs->end())
             outPath = state->coerceToStorePath(
@@ -111,7 +111,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
     if (outputs.empty()) {
         /* Get the ‘outputs’ list. */
         const Attr * i;
-        if (attrs && (i = attrs->get(state->sOutputs))) {
+        if (attrs && (i = attrs->get(state->s.outputs))) {
             state->forceList(*i->value, i->pos, "while evaluating the 'outputs' attribute of a derivation");
 
             /* For each output... */
@@ -127,7 +127,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
                     state->forceAttrs(*out->value, i->pos, "while evaluating an output of a derivation");
 
                     /* And evaluate its ‘outPath’ attribute. */
-                    auto outPath = out->value->attrs()->get(state->sOutPath);
+                    auto outPath = out->value->attrs()->get(state->s.outPath);
                     if (!outPath)
                         continue; // FIXME: throw error?
                     NixStringContext context;
@@ -146,7 +146,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
         return outputs;
 
     const Attr * i;
-    if (attrs && (i = attrs->get(state->sOutputSpecified))
+    if (attrs && (i = attrs->get(state->s.outputSpecified))
         && state->forceBool(*i->value, i->pos, "while evaluating the 'outputSpecified' attribute of a derivation")) {
         Outputs result;
         auto out = outputs.find(queryOutputName());
@@ -181,7 +181,7 @@ PackageInfo::Outputs PackageInfo::queryOutputs(bool withPaths, bool onlyOutputsT
 std::string PackageInfo::queryOutputName() const
 {
     if (outputName == "" && attrs) {
-        auto i = attrs->get(state->sOutputName);
+        auto i = attrs->get(state->s.outputName);
         outputName =
             i ? state->forceStringNoCtx(*i->value, noPos, "while evaluating the output name of a derivation") : "";
     }
@@ -194,7 +194,7 @@ const Bindings * PackageInfo::getMeta()
         return meta;
     if (!attrs)
         return 0;
-    auto a = attrs->get(state->sMeta);
+    auto a = attrs->get(state->s.meta);
     if (!a)
         return 0;
     state->forceAttrs(*a->value, a->pos, "while evaluating the 'meta' attribute of a derivation");
@@ -221,7 +221,7 @@ bool PackageInfo::checkMeta(Value & v)
                 return false;
         return true;
     } else if (v.type() == nAttrs) {
-        if (v.attrs()->get(state->sOutPath))
+        if (v.attrs()->get(state->s.outPath))
             return false;
         for (auto & i : *v.attrs())
             if (!checkMeta(*i.value))
@@ -411,7 +411,7 @@ static void getDerivations(
                     should we recurse into it?  => Only if it has a
                     `recurseForDerivations = true' attribute. */
                     if (i->value->type() == nAttrs) {
-                        auto j = i->value->attrs()->get(state.sRecurseForDerivations);
+                        auto j = i->value->attrs()->get(state.s.recurseForDerivations);
                         if (j
                             && state.forceBool(
                                 *j->value, j->pos, "while evaluating the attribute `recurseForDerivations`"))

src/libexpr/include/nix/expr/eval.hh

@@ -45,7 +45,6 @@ class StorePath;
 struct SingleDerivedPath;
 enum RepairFlag : bool;
 struct MemorySourceAccessor;
-struct AsyncPathWriter;
 
 namespace eval_cache {
 class EvalCache;
@@ -214,23 +213,100 @@ struct DebugTrace
     }
 };
 
+struct StaticEvalSymbols
+{
+    Symbol with, outPath, drvPath, type, meta, name, value, system, overrides, outputs, outputName, ignoreNulls, file,
+        line, column, functor, toString, right, wrong, structuredAttrs, json, allowedReferences, allowedRequisites,
+        disallowedReferences, disallowedRequisites, maxSize, maxClosureSize, builder, args, contentAddressed, impure,
+        outputHash, outputHashAlgo, outputHashMode, recurseForDerivations, description, self, epsilon, startSet,
+        operator_, key, path, prefix, outputSpecified;
+
+    Expr::AstSymbols exprSymbols;
+
+    static constexpr auto preallocate()
+    {
+        StaticSymbolTable alloc;
+
+        StaticEvalSymbols staticSymbols = {
+            .with = alloc.create("<with>"),
+            .outPath = alloc.create("outPath"),
+            .drvPath = alloc.create("drvPath"),
+            .type = alloc.create("type"),
+            .meta = alloc.create("meta"),
+            .name = alloc.create("name"),
+            .value = alloc.create("value"),
+            .system = alloc.create("system"),
+            .overrides = alloc.create("__overrides"),
+            .outputs = alloc.create("outputs"),
+            .outputName = alloc.create("outputName"),
+            .ignoreNulls = alloc.create("__ignoreNulls"),
+            .file = alloc.create("file"),
+            .line = alloc.create("line"),
+            .column = alloc.create("column"),
+            .functor = alloc.create("__functor"),
+            .toString = alloc.create("__toString"),
+            .right = alloc.create("right"),
+            .wrong = alloc.create("wrong"),
+            .structuredAttrs = alloc.create("__structuredAttrs"),
+            .json = alloc.create("__json"),
+            .allowedReferences = alloc.create("allowedReferences"),
+            .allowedRequisites = alloc.create("allowedRequisites"),
+            .disallowedReferences = alloc.create("disallowedReferences"),
+            .disallowedRequisites = alloc.create("disallowedRequisites"),
+            .maxSize = alloc.create("maxSize"),
+            .maxClosureSize = alloc.create("maxClosureSize"),
+            .builder = alloc.create("builder"),
+            .args = alloc.create("args"),
+            .contentAddressed = alloc.create("__contentAddressed"),
+            .impure = alloc.create("__impure"),
+            .outputHash = alloc.create("outputHash"),
+            .outputHashAlgo = alloc.create("outputHashAlgo"),
+            .outputHashMode = alloc.create("outputHashMode"),
+            .recurseForDerivations = alloc.create("recurseForDerivations"),
+            .description = alloc.create("description"),
+            .self = alloc.create("self"),
+            .epsilon = alloc.create(""),
+            .startSet = alloc.create("startSet"),
+            .operator_ = alloc.create("operator"),
+            .key = alloc.create("key"),
+            .path = alloc.create("path"),
+            .prefix = alloc.create("prefix"),
+            .outputSpecified = alloc.create("outputSpecified"),
+            .exprSymbols = {
+                .sub = alloc.create("__sub"),
+                .lessThan = alloc.create("__lessThan"),
+                .mul = alloc.create("__mul"),
+                .div = alloc.create("__div"),
+                .or_ = alloc.create("or"),
+                .findFile = alloc.create("__findFile"),
+                .nixPath = alloc.create("__nixPath"),
+                .body = alloc.create("body"),
+            }};
+
+        return std::pair{staticSymbols, alloc};
+    }
+
+    static consteval StaticEvalSymbols create()
+    {
+        return preallocate().first;
+    }
+
+    static constexpr StaticSymbolTable staticSymbolTable()
+    {
+        return preallocate().second;
+    }
+};
+
 class EvalState : public std::enable_shared_from_this<EvalState>
 {
 public:
+    static constexpr StaticEvalSymbols s = StaticEvalSymbols::create();
+
     const fetchers::Settings & fetchSettings;
     const EvalSettings & settings;
     SymbolTable symbols;
     PosTable positions;
 
-    const Symbol sWith, sOutPath, sDrvPath, sType, sMeta, sName, sValue, sSystem, sOverrides, sOutputs, sOutputName,
-        sIgnoreNulls, sFile, sLine, sColumn, sFunctor, sToString, sRight, sWrong, sStructuredAttrs, sJson,
-        sAllowedReferences, sAllowedRequisites, sDisallowedReferences, sDisallowedRequisites, sMaxSize, sMaxClosureSize,
-        sBuilder, sArgs, sContentAddressed, sImpure, sOutputHash, sOutputHashAlgo, sOutputHashMode,
-        sRecurseForDerivations, sDescription, sSelf, sEpsilon, sStartSet, sOperator, sKey, sPath, sPrefix,
-        sOutputSpecified;
-
-    const Expr::AstSymbols exprSymbols;
-
     /**
      * If set, force copying files to the Nix store even if they
      * already exist there.
@@ -321,8 +397,6 @@ public:
     std::list<DebugTrace> debugTraces;
     std::map<const Expr *, const std::shared_ptr<const StaticEnv>> exprEnvs;
 
-    ref<AsyncPathWriter> asyncPathWriter;
-
     const std::shared_ptr<const StaticEnv> getStaticEnv(const Expr & expr) const
     {
         auto i = exprEnvs.find(&expr);
@@ -910,10 +984,6 @@ public:
 
     DocComment getDocCommentForPos(PosIdx pos);
 
-    void waitForPath(const StorePath & path);
-    void waitForPath(const SingleDerivedPath & path);
-    void waitForAllPaths();
-
 private:
 
     /**

src/libexpr/include/nix/expr/nixexpr.hh

@@ -595,12 +595,17 @@ struct ExprOpNot : Expr
         {                                                                                    \
             return pos;                                                                      \
         }                                                                                    \
-    };
+    }
 
-MakeBinOp(ExprOpEq, "==") MakeBinOp(ExprOpNEq, "!=") MakeBinOp(ExprOpAnd, "&&") MakeBinOp(ExprOpOr, "||")
-    MakeBinOp(ExprOpImpl, "->") MakeBinOp(ExprOpUpdate, "//") MakeBinOp(ExprOpConcatLists, "++")
+MakeBinOp(ExprOpEq, "==");
+MakeBinOp(ExprOpNEq, "!=");
+MakeBinOp(ExprOpAnd, "&&");
+MakeBinOp(ExprOpOr, "||");
+MakeBinOp(ExprOpImpl, "->");
+MakeBinOp(ExprOpUpdate, "//");
+MakeBinOp(ExprOpConcatLists, "++");
 
-        struct ExprConcatStrings : Expr
+struct ExprConcatStrings : Expr
 {
     PosIdx pos;
     bool forceString;

src/libexpr/include/nix/expr/parser-state.hh

@@ -88,7 +88,7 @@ struct ParserState
     SourcePath basePath;
     PosTable::Origin origin;
     const ref<SourceAccessor> rootFS;
-    const Expr::AstSymbols & s;
+    static constexpr Expr::AstSymbols s = StaticEvalSymbols::create().exprSymbols;
     const EvalSettings & settings;
 
     void dupAttr(const AttrPath & attrPath, const PosIdx pos, const PosIdx prevPos);

src/libexpr/include/nix/expr/symbol-table.hh

@@ -28,6 +28,8 @@ public:
     }
 };
 
+class StaticSymbolTable;
+
 /**
  * Symbols have the property that they can be compared efficiently
  * (using an equality test), because the symbol table stores only one
@@ -37,36 +39,29 @@ class Symbol
 {
     friend class SymbolStr;
     friend class SymbolTable;
+    friend class StaticSymbolTable;
 
 private:
     uint32_t id;
 
-    explicit Symbol(uint32_t id) noexcept
+    explicit constexpr Symbol(uint32_t id) noexcept
         : id(id)
     {
     }
 
 public:
-    Symbol() noexcept
+    constexpr Symbol() noexcept
         : id(0)
     {
     }
 
     [[gnu::always_inline]]
-    explicit operator bool() const noexcept
+    constexpr explicit operator bool() const noexcept
     {
         return id > 0;
     }
 
-    auto operator<=>(const Symbol other) const noexcept
-    {
-        return id <=> other.id;
-    }
-
-    bool operator==(const Symbol other) const noexcept
-    {
-        return id == other.id;
-    }
+    constexpr auto operator<=>(const Symbol & other) const noexcept = default;
 
     friend class std::hash<Symbol>;
 };
@@ -210,6 +205,39 @@ public:
     };
 };
 
+class SymbolTable;
+
+/**
+ * Convenience class to statically assign symbol identifiers at compile-time.
+ */
+class StaticSymbolTable
+{
+    static constexpr std::size_t maxSize = 1024;
+
+    struct StaticSymbolInfo
+    {
+        std::string_view str;
+        Symbol sym;
+    };
+
+    std::array<StaticSymbolInfo, maxSize> symbols;
+    std::size_t size = 0;
+
+public:
+    constexpr StaticSymbolTable() = default;
+
+    constexpr Symbol create(std::string_view str)
+    {
+        /* No need to check bounds because out of bounds access is
+           a compilation error. */
+        auto sym = Symbol(size + 1); //< +1 because Symbol with id = 0 is reserved
+        symbols[size++] = {str, sym};
+        return sym;
+    }
+
+    void copyIntoSymbolTable(SymbolTable & symtab) const;
+};
+
 /**
  * Symbol table used by the parser and evaluator to represent and look
  * up identifiers and attributes efficiently.
@@ -232,6 +260,10 @@ private:
     boost::unordered_flat_set<SymbolStr, SymbolStr::Hash, SymbolStr::Equal> symbols{SymbolStr::chunkSize};
 
 public:
+    SymbolTable(const StaticSymbolTable & staticSymtab)
+    {
+        staticSymtab.copyIntoSymbolTable(*this);
+    }
 
     /**
      * Converts a string into a symbol.
@@ -276,6 +308,16 @@ public:
     }
 };
 
+inline void StaticSymbolTable::copyIntoSymbolTable(SymbolTable & symtab) const
+{
+    for (std::size_t i = 0; i < size; ++i) {
+        auto [str, staticSym] = symbols[i];
+        auto sym = symtab.create(str);
+        if (sym != staticSym) [[unlikely]]
+            unreachable();
+    }
+}
+
 } // namespace nix
 
 template<>

src/libexpr/meson.build

@@ -71,6 +71,12 @@ toml11 = dependency(
   method : 'cmake',
   include_type : 'system',
 )
+
+configdata_priv.set(
+  'HAVE_TOML11_4',
+  toml11.version().version_compare('>= 4.0.0').to_int(),
+)
+
 deps_other += toml11
 
 config_priv_h = configure_file(

src/libexpr/parser.y

@@ -68,8 +68,7 @@ Expr * parseExprFromBuf(
     const EvalSettings & settings,
     PosTable & positions,
     DocCommentMap & docComments,
-    const ref<SourceAccessor> rootFS,
-    const Expr::AstSymbols & astSymbols);
+    const ref<SourceAccessor> rootFS);
 
 }
 
@@ -542,8 +541,7 @@ Expr * parseExprFromBuf(
     const EvalSettings & settings,
     PosTable & positions,
     DocCommentMap & docComments,
-    const ref<SourceAccessor> rootFS,
-    const Expr::AstSymbols & astSymbols)
+    const ref<SourceAccessor> rootFS)
 {
     yyscan_t scanner;
     LexerState lexerState {
@@ -558,7 +556,6 @@ Expr * parseExprFromBuf(
         .basePath = basePath,
         .origin = lexerState.origin,
         .rootFS = rootFS,
-        .s = astSymbols,
         .settings = settings,
     };
 

src/libexpr/primops.cc

@@ -5,6 +5,7 @@
 #include "nix/expr/eval-settings.hh"
 #include "nix/expr/gc-small-vector.hh"
 #include "nix/expr/json-to-value.hh"
+#include "nix/store/globals.hh"
 #include "nix/store/names.hh"
 #include "nix/store/path-references.hh"
 #include "nix/store/store-api.hh"
@@ -63,7 +64,6 @@ StringMap EvalState::realiseContext(const NixStringContext & context, StorePathS
 
     for (auto & c : context) {
         auto ensureValid = [&](const StorePath & p) {
-            waitForPath(p);
             if (!store->isValidPath(p))
                 error<InvalidPathError>(store->printStorePath(p)).debugThrow();
         };
@@ -214,20 +214,20 @@ void derivationToValue(
     auto path2 = path.path.abs();
     Derivation drv = state.store->readDerivation(storePath);
     auto attrs = state.buildBindings(3 + drv.outputs.size());
-    attrs.alloc(state.sDrvPath)
+    attrs.alloc(state.s.drvPath)
         .mkString(
             path2,
             {
                 NixStringContextElem::DrvDeep{.drvPath = storePath},
             });
-    attrs.alloc(state.sName).mkString(drv.env["name"]);
+    attrs.alloc(state.s.name).mkString(drv.env["name"]);
 
     auto list = state.buildList(drv.outputs.size());
     for (const auto & [i, o] : enumerate(drv.outputs)) {
         mkOutputString(state, attrs, storePath, o);
         (list[i] = state.allocValue())->mkString(o.first);
     }
-    attrs.alloc(state.sOutputs).mkList(list);
+    attrs.alloc(state.s.outputs).mkList(list);
 
     auto w = state.allocValue();
     w->mkAttrs(attrs);
@@ -292,7 +292,6 @@ static void import(EvalState & state, const PosIdx pos, Value & vPath, Value * v
         if (!state.store->isStorePath(path2))
             return std::nullopt;
         auto storePath = state.store->parseStorePath(path2);
-        state.waitForPath(storePath);
         if (!(state.store->isValidPath(storePath) && isDerivation(path2)))
             return std::nullopt;
         return storePath;
@@ -732,7 +731,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar
 
     /* Get the start set. */
     auto startSet = state.getAttr(
-        state.sStartSet, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
+        state.s.startSet, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
 
     state.forceList(
         *startSet->value,
@@ -750,7 +749,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar
 
     /* Get the operator. */
     auto op = state.getAttr(
-        state.sOperator, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
+        state.s.operator_, args[0]->attrs(), "in the attrset passed as argument to builtins.genericClosure");
     state.forceFunction(
         *op->value, noPos, "while evaluating the 'operator' attribute passed as argument to builtins.genericClosure");
 
@@ -772,7 +771,7 @@ static void prim_genericClosure(EvalState & state, const PosIdx pos, Value ** ar
             "while evaluating one of the elements generated by (or initially passed to) builtins.genericClosure");
 
         auto key = state.getAttr(
-            state.sKey,
+            state.s.key,
             e->attrs(),
             "in one of the attrsets generated by (or initially passed to) builtins.genericClosure");
         state.forceValue(*key->value, noPos);
@@ -1077,11 +1076,11 @@ static void prim_tryEval(EvalState & state, const PosIdx pos, Value ** args, Val
 
     try {
         state.forceValue(*args[0], pos);
-        attrs.insert(state.sValue, args[0]);
+        attrs.insert(state.s.value, args[0]);
         attrs.insert(state.symbols.create("success"), &state.vTrue);
     } catch (AssertionError & e) {
         // `value = false;` is unfortunate but removing it is a breaking change.
-        attrs.insert(state.sValue, &state.vFalse);
+        attrs.insert(state.s.value, &state.vFalse);
         attrs.insert(state.symbols.create("success"), &state.vFalse);
     }
 
@@ -1293,7 +1292,8 @@ static void prim_derivationStrict(EvalState & state, const PosIdx pos, Value **
     auto attrs = args[0]->attrs();
 
     /* Figure out the name first (for stack backtraces). */
-    auto nameAttr = state.getAttr(state.sName, attrs, "in the attrset passed as argument to builtins.derivationStrict");
+    auto nameAttr =
+        state.getAttr(state.s.name, attrs, "in the attrset passed as argument to builtins.derivationStrict");
 
     std::string_view drvName;
     try {
@@ -1367,7 +1367,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
     using nlohmann::json;
     std::optional<StructuredAttrs> jsonObject;
     auto pos = v.determinePos(noPos);
-    auto attr = attrs->find(state.sStructuredAttrs);
+    auto attr = attrs->find(state.s.structuredAttrs);
     if (attr != attrs->end()
         && state.forceBool(
             *attr->value,
@@ -1378,7 +1378,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
 
     /* Check whether null attributes should be ignored. */
     bool ignoreNulls = false;
-    attr = attrs->find(state.sIgnoreNulls);
+    attr = attrs->find(state.s.ignoreNulls);
     if (attr != attrs->end())
         ignoreNulls = state.forceBool(
             *attr->value,
@@ -1402,7 +1402,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
     outputs.insert("out");
 
     for (auto & i : attrs->lexicographicOrder(state.symbols)) {
-        if (i->name == state.sIgnoreNulls)
+        if (i->name == state.s.ignoreNulls)
             continue;
         auto key = state.symbols[i->name];
         vomit("processing attribute '%1%'", key);
@@ -1454,19 +1454,19 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
                     continue;
             }
 
-            if (i->name == state.sContentAddressed && state.forceBool(*i->value, pos, context_below)) {
+            if (i->name == state.s.contentAddressed && state.forceBool(*i->value, pos, context_below)) {
                 contentAddressed = true;
                 experimentalFeatureSettings.require(Xp::CaDerivations);
             }
 
-            else if (i->name == state.sImpure && state.forceBool(*i->value, pos, context_below)) {
+            else if (i->name == state.s.impure && state.forceBool(*i->value, pos, context_below)) {
                 isImpure = true;
                 experimentalFeatureSettings.require(Xp::ImpureDerivations);
             }
 
             /* The `args' attribute is special: it supplies the
                command-line arguments to the builder. */
-            else if (i->name == state.sArgs) {
+            else if (i->name == state.s.args) {
                 state.forceList(*i->value, pos, context_below);
                 for (auto elem : i->value->listView()) {
                     auto s = state
@@ -1483,22 +1483,22 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
 
                 if (jsonObject) {
 
-                    if (i->name == state.sStructuredAttrs)
+                    if (i->name == state.s.structuredAttrs)
                         continue;
 
                     jsonObject->structuredAttrs.emplace(key, printValueAsJSON(state, true, *i->value, pos, context));
 
-                    if (i->name == state.sBuilder)
+                    if (i->name == state.s.builder)
                         drv.builder = state.forceString(*i->value, context, pos, context_below);
-                    else if (i->name == state.sSystem)
+                    else if (i->name == state.s.system)
                         drv.platform = state.forceStringNoCtx(*i->value, pos, context_below);
-                    else if (i->name == state.sOutputHash)
+                    else if (i->name == state.s.outputHash)
                         outputHash = state.forceStringNoCtx(*i->value, pos, context_below);
-                    else if (i->name == state.sOutputHashAlgo)
+                    else if (i->name == state.s.outputHashAlgo)
                         outputHashAlgo = parseHashAlgoOpt(state.forceStringNoCtx(*i->value, pos, context_below));
-                    else if (i->name == state.sOutputHashMode)
+                    else if (i->name == state.s.outputHashMode)
                         handleHashMode(state.forceStringNoCtx(*i->value, pos, context_below));
-                    else if (i->name == state.sOutputs) {
+                    else if (i->name == state.s.outputs) {
                         /* Require ‘outputs’ to be a list of strings. */
                         state.forceList(*i->value, pos, context_below);
                         Strings ss;
@@ -1507,51 +1507,51 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
                         handleOutputs(ss);
                     }
 
-                    if (i->name == state.sAllowedReferences)
+                    if (i->name == state.s.allowedReferences)
                         warn(
                             "In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'allowedReferences'; use 'outputChecks.<output>.allowedReferences' instead",
                             drvName);
-                    if (i->name == state.sAllowedRequisites)
+                    if (i->name == state.s.allowedRequisites)
                         warn(
                             "In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'allowedRequisites'; use 'outputChecks.<output>.allowedRequisites' instead",
                             drvName);
-                    if (i->name == state.sDisallowedReferences)
+                    if (i->name == state.s.disallowedReferences)
                         warn(
                             "In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'disallowedReferences'; use 'outputChecks.<output>.disallowedReferences' instead",
                             drvName);
-                    if (i->name == state.sDisallowedRequisites)
+                    if (i->name == state.s.disallowedRequisites)
                         warn(
                             "In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'disallowedRequisites'; use 'outputChecks.<output>.disallowedRequisites' instead",
                             drvName);
-                    if (i->name == state.sMaxSize)
+                    if (i->name == state.s.maxSize)
                         warn(
                             "In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'maxSize'; use 'outputChecks.<output>.maxSize' instead",
                             drvName);
-                    if (i->name == state.sMaxClosureSize)
+                    if (i->name == state.s.maxClosureSize)
                         warn(
                             "In a derivation named '%s', 'structuredAttrs' disables the effect of the derivation attribute 'maxClosureSize'; use 'outputChecks.<output>.maxClosureSize' instead",
                             drvName);
 
                 } else {
                     auto s = state.coerceToString(pos, *i->value, context, context_below, true).toOwned();
-                    if (i->name == state.sJson) {
+                    if (i->name == state.s.json) {
                         warn(
                             "In derivation '%s': setting structured attributes via '__json' is deprecated, and may be disallowed in future versions of Nix. Set '__structuredAttrs = true' instead.",
                             drvName);
                         drv.structuredAttrs = StructuredAttrs::parse(s);
                     } else {
                         drv.env.emplace(key, s);
-                        if (i->name == state.sBuilder)
+                        if (i->name == state.s.builder)
                             drv.builder = std::move(s);
-                        else if (i->name == state.sSystem)
+                        else if (i->name == state.s.system)
                             drv.platform = std::move(s);
-                        else if (i->name == state.sOutputHash)
+                        else if (i->name == state.s.outputHash)
                             outputHash = std::move(s);
-                        else if (i->name == state.sOutputHashAlgo)
+                        else if (i->name == state.s.outputHashAlgo)
                             outputHashAlgo = parseHashAlgoOpt(s);
-                        else if (i->name == state.sOutputHashMode)
+                        else if (i->name == state.s.outputHashMode)
                             handleHashMode(s);
-                        else if (i->name == state.sOutputs)
+                        else if (i->name == state.s.outputs)
                             handleOutputs(tokenizeString<Strings>(s));
                     }
                 }
@@ -1585,8 +1585,6 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
                 [&](const NixStringContextElem::DrvDeep & d) {
                     /* !!! This doesn't work if readOnlyMode is set. */
                     StorePathSet refs;
-                    // FIXME: don't need to wait, we only need the references.
-                    state.waitForPath(d.drvPath);
                     state.store->computeFSClosure(d.drvPath, refs);
                     for (auto & j : refs) {
                         drv.inputSrcs.insert(j);
@@ -1711,7 +1709,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
     }
 
     /* Write the resulting term into the Nix store directory. */
-    auto drvPath = writeDerivation(*state.store, *state.asyncPathWriter, drv, state.repair);
+    auto drvPath = writeDerivation(*state.store, drv, state.repair);
     auto drvPathS = state.store->printStorePath(drvPath);
 
     printMsg(lvlChatty, "instantiated '%1%' -> '%2%'", drvName, drvPathS);
@@ -1725,7 +1723,7 @@ static void derivationStrictInternal(EvalState & state, std::string_view drvName
     }
 
     auto result = state.buildBindings(1 + drv.outputs.size());
-    result.alloc(state.sDrvPath)
+    result.alloc(state.s.drvPath)
         .mkString(
             drvPathS,
             {
@@ -2009,14 +2007,14 @@ static void prim_findFile(EvalState & state, const PosIdx pos, Value ** args, Va
         state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.findFile");
 
         std::string prefix;
-        auto i = v2->attrs()->find(state.sPrefix);
+        auto i = v2->attrs()->find(state.s.prefix);
         if (i != v2->attrs()->end())
             prefix = state.forceStringNoCtx(
                 *i->value,
                 pos,
                 "while evaluating the `prefix` attribute of an element of the list passed to builtins.findFile");
 
-        i = state.getAttr(state.sPath, v2->attrs(), "in an element of the __nixPath");
+        i = state.getAttr(state.s.path, v2->attrs(), "in an element of the __nixPath");
 
         NixStringContext context;
         auto path =
@@ -2789,7 +2787,7 @@ static void prim_path(EvalState & state, const PosIdx pos, Value ** args, Value
         if (n == "path")
             path.emplace(state.coerceToPath(
                 attr.pos, *attr.value, context, "while evaluating the 'path' attribute passed to 'builtins.path'"));
-        else if (attr.name == state.sName)
+        else if (attr.name == state.s.name)
             name = state.forceStringNoCtx(
                 *attr.value, attr.pos, "while evaluating the `name` attribute passed to builtins.path");
         else if (n == "filter")
@@ -3108,7 +3106,7 @@ static void prim_listToAttrs(EvalState & state, const PosIdx pos, Value ** args,
     for (const auto & [n, v2] : enumerate(listView)) {
         state.forceAttrs(*v2, pos, "while evaluating an element of the list passed to builtins.listToAttrs");
 
-        auto j = state.getAttr(state.sName, v2->attrs(), "in a {name=...; value=...;} pair");
+        auto j = state.getAttr(state.s.name, v2->attrs(), "in a {name=...; value=...;} pair");
 
         auto name = state.forceStringNoCtx(
             *j->value,
@@ -3135,7 +3133,7 @@ static void prim_listToAttrs(EvalState & state, const PosIdx pos, Value ** args,
         // Note that .value is actually a Value * *; see earlier comments
         Value * v2 = *std::bit_cast<ElemPtr>(attr.value);
 
-        auto j = state.getAttr(state.sValue, v2->attrs(), "in a {name=...; value=...;} pair");
+        auto j = state.getAttr(state.s.value, v2->attrs(), "in a {name=...; value=...;} pair");
         prev = attr.name;
         bindings.push_back({prev, j->value, j->pos});
     }
@@ -3951,13 +3949,13 @@ static void prim_partition(EvalState & state, const PosIdx pos, Value ** args, V
     auto rlist = state.buildList(rsize);
     if (rsize)
         memcpy(rlist.elems, right.data(), sizeof(Value *) * rsize);
-    attrs.alloc(state.sRight).mkList(rlist);
+    attrs.alloc(state.s.right).mkList(rlist);
 
     auto wsize = wrong.size();
     auto wlist = state.buildList(wsize);
     if (wsize)
         memcpy(wlist.elems, wrong.data(), sizeof(Value *) * wsize);
-    attrs.alloc(state.sWrong).mkList(wlist);
+    attrs.alloc(state.s.wrong).mkList(wlist);
 
     v.mkAttrs(attrs);
 }
@@ -4876,7 +4874,7 @@ static void prim_parseDrvName(EvalState & state, const PosIdx pos, Value ** args
         state.forceStringNoCtx(*args[0], pos, "while evaluating the first argument passed to builtins.parseDrvName");
     DrvName parsed(name);
     auto attrs = state.buildBindings(2);
-    attrs.alloc(state.sName).mkString(parsed.name);
+    attrs.alloc(state.s.name).mkString(parsed.name);
     attrs.alloc("version").mkString(parsed.version);
     v.mkAttrs(attrs);
 }

src/libexpr/primops/context.cc

@@ -2,6 +2,7 @@
 #include "nix/expr/eval-inline.hh"
 #include "nix/store/derivations.hh"
 #include "nix/store/store-api.hh"
+#include "nix/store/globals.hh"
 
 namespace nix {
 
@@ -61,7 +62,6 @@ static void prim_unsafeDiscardOutputDependency(EvalState & state, const PosIdx p
     NixStringContext context2;
     for (auto && c : context) {
         if (auto * ptr = std::get_if<NixStringContextElem::DrvDeep>(&c.raw)) {
-            state.waitForPath(ptr->drvPath); // FIXME: why?
             context2.emplace(NixStringContextElem::Opaque{.path = ptr->drvPath});
         } else {
             /* Can reuse original item */
@@ -219,7 +219,7 @@ static void prim_getContext(EvalState & state, const PosIdx pos, Value ** args,
             auto list = state.buildList(info.second.outputs.size());
             for (const auto & [i, output] : enumerate(info.second.outputs))
                 (list[i] = state.allocValue())->mkString(output);
-            infoAttrs.alloc(state.sOutputs).mkList(list);
+            infoAttrs.alloc(state.s.outputs).mkList(list);
         }
         attrs.alloc(state.store->printStorePath(info.first)).mkAttrs(infoAttrs);
     }
@@ -300,7 +300,7 @@ static void prim_appendContext(EvalState & state, const PosIdx pos, Value ** arg
             }
         }
 
-        if (auto attr = i.value->attrs()->get(state.sOutputs)) {
+        if (auto attr = i.value->attrs()->get(state.s.outputs)) {
             state.forceList(*attr->value, attr->pos, "while evaluating the `outputs` attribute of a string context");
             if (attr->value->listSize() && !isDerivation(name)) {
                 state

src/libexpr/primops/fetchClosure.cc

@@ -3,6 +3,7 @@
 #include "nix/store/realisation.hh"
 #include "nix/store/make-content-addressed.hh"
 #include "nix/util/url.hh"
+#include "nix/util/environment-variables.hh"
 
 namespace nix {
 
@@ -184,7 +185,7 @@ static void prim_fetchClosure(EvalState & state, const PosIdx pos, Value ** args
             {.msg = HintFmt("attribute '%s' is missing in call to 'fetchClosure'", "fromStore"),
              .pos = state.positions[pos]});
 
-    auto parsedURL = parseURL(*fromStoreUrl);
+    auto parsedURL = parseURL(*fromStoreUrl, /*lenient=*/true);
 
     if (parsedURL.scheme != "http" && parsedURL.scheme != "https"
         && !(getEnv("_NIX_IN_TEST").has_value() && parsedURL.scheme == "file"))

src/libexpr/primops/fetchMercurial.cc

@@ -84,7 +84,7 @@ static void prim_fetchMercurial(EvalState & state, const PosIdx pos, Value ** ar
     auto [storePath, input2] = input.fetchToStore(state.store);
 
     auto attrs2 = state.buildBindings(8);
-    state.mkStorePathString(storePath, attrs2.alloc(state.sOutPath));
+    state.mkStorePathString(storePath, attrs2.alloc(state.s.outPath));
     if (input2.getRef())
         attrs2.alloc("branch").mkString(*input2.getRef());
     // Backward compatibility: set 'rev' to

src/libexpr/primops/fetchTree.cc

@@ -29,7 +29,7 @@ void emitTreeAttrs(
 {
     auto attrs = state.buildBindings(100);
 
-    state.mkStorePathString(storePath, attrs.alloc(state.sOutPath));
+    state.mkStorePathString(storePath, attrs.alloc(state.s.outPath));
 
     // FIXME: support arbitrary input attributes.
 
@@ -95,7 +95,7 @@ static void fetchTree(
 
         fetchers::Attrs attrs;
 
-        if (auto aType = args[0]->attrs()->get(state.sType)) {
+        if (auto aType = args[0]->attrs()->get(state.s.type)) {
             if (type)
                 state.error<EvalError>("unexpected argument 'type'").atPos(pos).debugThrow();
             type = state.forceStringNoCtx(
@@ -106,14 +106,14 @@ static void fetchTree(
         attrs.emplace("type", type.value());
 
         for (auto & attr : *args[0]->attrs()) {
-            if (attr.name == state.sType)
+            if (attr.name == state.s.type)
                 continue;
             state.forceValue(*attr.value, attr.pos);
             if (attr.value->type() == nPath || attr.value->type() == nString) {
                 auto s = state.coerceToString(attr.pos, *attr.value, context, "", false, false).toOwned();
                 attrs.emplace(
                     state.symbols[attr.name],
-                    params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s) : s);
+                    params.isFetchGit && state.symbols[attr.name] == "url" ? fixGitURL(s).to_string() : s);
             } else if (attr.value->type() == nBool)
                 attrs.emplace(state.symbols[attr.name], Explicit<bool>{attr.value->boolean()});
             else if (attr.value->type() == nInt) {
@@ -175,7 +175,7 @@ static void fetchTree(
         if (params.isFetchGit) {
             fetchers::Attrs attrs;
             attrs.emplace("type", "git");
-            attrs.emplace("url", fixGitURL(url));
+            attrs.emplace("url", fixGitURL(url).to_string());
             if (!attrs.contains("exportIgnore")
                 && (!attrs.contains("submodules") || !*fetchers::maybeGetBoolAttr(attrs, "submodules"))) {
                 attrs.emplace("exportIgnore", Explicit<bool>{true});

src/libexpr/primops/fromTOML.cc

@@ -1,73 +1,140 @@
 #include "nix/expr/primops.hh"
 #include "nix/expr/eval-inline.hh"
 
+#include "expr-config-private.hh"
+
 #include <sstream>
 
 #include <toml.hpp>
 
 namespace nix {
 
+#if HAVE_TOML11_4
+
+/**
+ * This is what toml11 < 4.0 did when choosing the subsecond precision.
+ * TOML 1.0.0 spec doesn't define how sub-millisecond ranges should be handled and calls it
+ * implementation defined behavior. For a lack of a better choice we stick with what older versions
+ * of toml11 did [1].
+ *
+ * [1]: https://github.com/ToruNiina/toml11/blob/dcfe39a783a94e8d52c885e5883a6fbb21529019/toml/datetime.hpp#L282
+ */
+static size_t normalizeSubsecondPrecision(toml::local_time lt)
+{
+    auto millis = lt.millisecond;
+    auto micros = lt.microsecond;
+    auto nanos = lt.nanosecond;
+    if (millis != 0 || micros != 0 || nanos != 0) {
+        if (micros != 0 || nanos != 0) {
+            if (nanos != 0)
+                return 9;
+            return 6;
+        }
+        return 3;
+    }
+    return 0;
+}
+
+/**
+ * Normalize date/time formats to serialize to the same strings as versions prior to toml11 4.0.
+ *
+ * Several things to consider:
+ *
+ * 1. Sub-millisecond range is represented the same way as in toml11 versions prior to 4.0. Precisioun is rounded
+ *    towards the next multiple of 3 or capped at 9 digits.
+ * 2. Seconds must be specified. This may become optional in (yet unreleased) TOML 1.1.0, but 1.0.0 defined local time
+ *    in terms of RFC3339 [1].
+ * 3. date-time separator (`t`, `T` or space ` `) is canonicalized to an upper T. This is compliant with RFC3339
+ *    [1] 5.6:
+ *    > Applications that generate this format SHOULD use upper case letters.
+ *
+ * [1]: https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
+ */
+static void normalizeDatetimeFormat(toml::value & t)
+{
+    if (t.is_local_datetime()) {
+        auto & ldt = t.as_local_datetime();
+        t.as_local_datetime_fmt() = {
+            .delimiter = toml::datetime_delimiter_kind::upper_T,
+            // https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
+            .has_seconds = true, // Mandated by TOML 1.0.0
+            .subsecond_precision = normalizeSubsecondPrecision(ldt.time),
+        };
+        return;
+    }
+
+    if (t.is_offset_datetime()) {
+        auto & odt = t.as_offset_datetime();
+        t.as_offset_datetime_fmt() = {
+            .delimiter = toml::datetime_delimiter_kind::upper_T,
+            // https://datatracker.ietf.org/doc/html/rfc3339#section-5.6
+            .has_seconds = true, // Mandated by TOML 1.0.0
+            .subsecond_precision = normalizeSubsecondPrecision(odt.time),
+        };
+        return;
+    }
+
+    if (t.is_local_time()) {
+        auto & lt = t.as_local_time();
+        t.as_local_time_fmt() = {
+            .has_seconds = true, // Mandated by TOML 1.0.0
+            .subsecond_precision = normalizeSubsecondPrecision(lt),
+        };
+        return;
+    }
+}
+
+#endif
+
 static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Value & val)
 {
     auto toml = state.forceStringNoCtx(*args[0], pos, "while evaluating the argument passed to builtins.fromTOML");
 
     std::istringstream tomlStream(std::string{toml});
 
-    std::function<void(Value &, toml::value)> visit;
-
-    visit = [&](Value & v, toml::value t) {
+    auto visit = [&](auto & self, Value & v, toml::value t) -> void {
         switch (t.type()) {
         case toml::value_t::table: {
             auto table = toml::get<toml::table>(t);
-
-            size_t size = 0;
-            for (auto & i : table) {
-                (void) i;
-                size++;
-            }
-
-            auto attrs = state.buildBindings(size);
+            auto attrs = state.buildBindings(table.size());
 
             for (auto & elem : table) {
                 forceNoNullByte(elem.first);
-                visit(attrs.alloc(elem.first), elem.second);
+                self(self, attrs.alloc(elem.first), elem.second);
             }
 
             v.mkAttrs(attrs);
         } break;
-            ;
         case toml::value_t::array: {
             auto array = toml::get<std::vector<toml::value>>(t);
 
             auto list = state.buildList(array.size());
             for (const auto & [n, v] : enumerate(list))
-                visit(*(v = state.allocValue()), array[n]);
+                self(self, *(v = state.allocValue()), array[n]);
             v.mkList(list);
         } break;
-            ;
         case toml::value_t::boolean:
             v.mkBool(toml::get<bool>(t));
             break;
-            ;
         case toml::value_t::integer:
             v.mkInt(toml::get<int64_t>(t));
             break;
-            ;
         case toml::value_t::floating:
             v.mkFloat(toml::get<NixFloat>(t));
             break;
-            ;
         case toml::value_t::string: {
             auto s = toml::get<std::string_view>(t);
             forceNoNullByte(s);
             v.mkString(s);
         } break;
-            ;
         case toml::value_t::local_datetime:
         case toml::value_t::offset_datetime:
         case toml::value_t::local_date:
         case toml::value_t::local_time: {
             if (experimentalFeatureSettings.isEnabled(Xp::ParseTomlTimestamps)) {
+#if HAVE_TOML11_4
+                normalizeDatetimeFormat(t);
+#endif
                 auto attrs = state.buildBindings(2);
                 attrs.alloc("_type").mkString("timestamp");
                 std::ostringstream s;
@@ -80,16 +147,24 @@ static void prim_fromTOML(EvalState & state, const PosIdx pos, Value ** args, Va
                 throw std::runtime_error("Dates and times are not supported");
             }
         } break;
-            ;
         case toml::value_t::empty:
             v.mkNull();
             break;
-            ;
         }
     };
 
     try {
-        visit(val, toml::parse(tomlStream, "fromTOML" /* the "filename" */));
+        visit(
+            visit,
+            val,
+            toml::parse(
+                tomlStream,
+                "fromTOML" /* the "filename" */
+#if HAVE_TOML11_4
+                ,
+                toml::spec::v(1, 0, 0) // Be explicit that we are parsing TOML 1.0.0 without extensions
+#endif
+                ));
     } catch (std::exception & e) { // TODO: toml::syntax_error
         state.error<EvalError>("while parsing TOML: %s", e.what()).atPos(pos).debugThrow();
     }

src/libexpr/print.cc

@@ -272,7 +272,7 @@ private:
     void printDerivation(Value & v)
     {
         std::optional<StorePath> storePath;
-        if (auto i = v.attrs()->get(state.sDrvPath)) {
+        if (auto i = v.attrs()->get(state.s.drvPath)) {
             NixStringContext context;
             storePath =
                 state.coerceToStorePath(i->pos, *i->value, context, "while evaluating the drvPath of a derivation");

src/libexpr/value-to-json.cc

@@ -53,7 +53,7 @@ json printValueAsJSON(
             out = *maybeString;
             break;
         }
-        if (auto i = v.attrs()->get(state.sOutPath))
+        if (auto i = v.attrs()->get(state.s.outPath))
             return printValueAsJSON(state, strict, *i->value, i->pos, context, copyToStore);
         else {
             out = json::object();

src/libexpr/value-to-xml.cc

@@ -98,14 +98,14 @@ static void printValueAsXML(
             XMLAttrs xmlAttrs;
 
             Path drvPath;
-            if (auto a = v.attrs()->get(state.sDrvPath)) {
+            if (auto a = v.attrs()->get(state.s.drvPath)) {
                 if (strict)
                     state.forceValue(*a->value, a->pos);
                 if (a->value->type() == nString)
                     xmlAttrs["drvPath"] = drvPath = a->value->c_str();
             }
 
-            if (auto a = v.attrs()->get(state.sOutPath)) {
+            if (auto a = v.attrs()->get(state.s.outPath)) {
                 if (strict)
                     state.forceValue(*a->value, a->pos);
                 if (a->value->type() == nString)

src/libfetchers-tests/git.cc

@@ -1,4 +1,5 @@
 #include "nix/store/store-open.hh"
+#include "nix/store/globals.hh"
 #include "nix/fetchers/fetch-settings.hh"
 #include "nix/fetchers/fetchers.hh"
 #include "nix/fetchers/git-utils.hh"

src/libfetchers/cache.cc

@@ -4,6 +4,7 @@
 #include "nix/store/sqlite.hh"
 #include "nix/util/sync.hh"
 #include "nix/store/store-api.hh"
+#include "nix/store/globals.hh"
 
 #include <nlohmann/json.hpp>
 
@@ -37,7 +38,7 @@ struct CacheImpl : Cache
     {
         auto state(_state.lock());
 
-        auto dbPath = getCacheDir() + "/fetcher-cache-v3.sqlite";
+        auto dbPath = getCacheDir() + "/fetcher-cache-v4.sqlite";
         createDirs(dirOf(dbPath));
 
         state->db = SQLite(dbPath);

src/libfetchers/git-lfs-fetch.cc

@@ -25,7 +25,7 @@ static void downloadToSink(
     std::string sha256Expected,
     size_t sizeExpected)
 {
-    FileTransferRequest request(url);
+    FileTransferRequest request(parseURL(url));
     Headers headers;
     if (authHeader.has_value())
         headers.push_back({"Authorization", *authHeader});
@@ -69,7 +69,8 @@ static LfsApiInfo getLfsApi(const ParsedURL & url)
 
         args.push_back("--");
         args.push_back("git-lfs-authenticate");
-        args.push_back(url.path);
+        // FIXME %2F encode slashes? Does this command take/accept percent encoding?
+        args.push_back(url.renderPath(/*encode=*/false));
         args.push_back("download");
 
         auto [status, output] = runProgram({.program = "ssh", .args = args});
@@ -179,7 +180,7 @@ Fetch::Fetch(git_repository * repo, git_oid rev)
 
     const auto remoteUrl = lfs::getLfsEndpointUrl(repo);
 
-    this->url = nix::parseURL(nix::fixGitURL(remoteUrl)).canonicalise();
+    this->url = nix::fixGitURL(remoteUrl).canonicalise();
 }
 
 bool Fetch::shouldFetch(const CanonPath & path) const
@@ -207,7 +208,7 @@ std::vector<nlohmann::json> Fetch::fetchUrls(const std::vector<Pointer> & pointe
     auto api = lfs::getLfsApi(this->url);
     auto url = api.endpoint + "/objects/batch";
     const auto & authHeader = api.authHeader;
-    FileTransferRequest request(url);
+    FileTransferRequest request(parseURL(url));
     request.post = true;
     Headers headers;
     if (authHeader.has_value())

src/libfetchers/git.cc

@@ -233,9 +233,7 @@ struct GitInputScheme : InputScheme
 
         Input input{settings};
         input.attrs = attrs;
-        auto url = fixGitURL(getStrAttr(attrs, "url"));
-        parseURL(url);
-        input.attrs["url"] = url;
+        input.attrs["url"] = fixGitURL(getStrAttr(attrs, "url")).to_string();
         getShallowAttr(input);
         getSubmodulesAttr(input);
         getAllRefsAttr(input);
@@ -464,8 +462,8 @@ struct GitInputScheme : InputScheme
 
         // Why are we checking for bare repository?
         // well if it's a bare repository we want to force a git fetch rather than copying the folder
-        bool isBareRepository = url.scheme == "file" && pathExists(url.path) && !pathExists(url.path + "/.git");
-        //
+        auto isBareRepository = [](PathView path) { return pathExists(path) && !pathExists(path + "/.git"); };
+
         // FIXME: here we turn a possibly relative path into an absolute path.
         // This allows relative git flake inputs to be resolved against the
         // **current working directory** (as in POSIX), which tends to work out
@@ -474,8 +472,10 @@ struct GitInputScheme : InputScheme
         //
         // See: https://discourse.nixos.org/t/57783 and #9708
         //
-        if (url.scheme == "file" && !forceHttp && !isBareRepository) {
-            if (!isAbsolute(url.path)) {
+        if (url.scheme == "file" && !forceHttp && !isBareRepository(renderUrlPathEnsureLegal(url.path))) {
+            auto path = renderUrlPathEnsureLegal(url.path);
+
+            if (!isAbsolute(path)) {
                 warn(
                     "Fetching Git repository '%s', which uses a path relative to the current directory. "
                     "This is not supported and will stop working in a future release. "
@@ -485,10 +485,10 @@ struct GitInputScheme : InputScheme
 
             // If we don't check here for the path existence, then we can give libgit2 any directory
             // and it will initialize them as git directories.
-            if (!pathExists(url.path)) {
-                throw Error("The path '%s' does not exist.", url.path);
+            if (!pathExists(path)) {
+                throw Error("The path '%s' does not exist.", path);
             }
-            repoInfo.location = std::filesystem::absolute(url.path);
+            repoInfo.location = std::filesystem::absolute(path);
         } else {
             if (url.scheme == "file")
                 /* Query parameters are meaningless for file://, but

src/libfetchers/github.cc

@@ -19,7 +19,7 @@ namespace nix::fetchers {
 
 struct DownloadUrl
 {
-    std::string url;
+    ParsedURL url;
     Headers headers;
 };
 
@@ -38,7 +38,8 @@ struct GitArchiveInputScheme : InputScheme
         if (url.scheme != schemeName())
             return {};
 
-        auto path = tokenizeString<std::vector<std::string>>(url.path, "/");
+        /* This ignores empty path segments for back-compat. Older versions used a tokenizeString here. */
+        auto path = url.pathSegments(/*skipEmpty=*/true) | std::ranges::to<std::vector<std::string>>();
 
         std::optional<Hash> rev;
         std::optional<std::string> ref;
@@ -139,12 +140,12 @@ struct GitArchiveInputScheme : InputScheme
         auto repo = getStrAttr(input.attrs, "repo");
         auto ref = input.getRef();
         auto rev = input.getRev();
-        auto path = owner + "/" + repo;
+        std::vector<std::string> path{owner, repo};
         assert(!(ref && rev));
         if (ref)
-            path += "/" + *ref;
+            path.push_back(*ref);
         if (rev)
-            path += "/" + rev->to_string(HashFormat::Base16, false);
+            path.push_back(rev->to_string(HashFormat::Base16, false));
         auto url = ParsedURL{
             .scheme = std::string{schemeName()},
             .path = path,
@@ -420,7 +421,7 @@ struct GitHubInputScheme : GitArchiveInputScheme
         const auto url =
             fmt(urlFmt, host, getOwner(input), getRepo(input), input.getRev()->to_string(HashFormat::Base16, false));
 
-        return DownloadUrl{url, headers};
+        return DownloadUrl{parseURL(url), headers};
     }
 
     void clone(const Input & input, const Path & destDir) const override
@@ -500,7 +501,7 @@ struct GitLabInputScheme : GitArchiveInputScheme
                 input.getRev()->to_string(HashFormat::Base16, false));
 
         Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
-        return DownloadUrl{url, headers};
+        return DownloadUrl{parseURL(url), headers};
     }
 
     void clone(const Input & input, const Path & destDir) const override
@@ -592,7 +593,7 @@ struct SourceHutInputScheme : GitArchiveInputScheme
                 input.getRev()->to_string(HashFormat::Base16, false));
 
         Headers headers = makeHeadersWithAuthTokens(*input.settings, host, input);
-        return DownloadUrl{url, headers};
+        return DownloadUrl{parseURL(url), headers};
     }
 
     void clone(const Input & input, const Path & destDir) const override

src/libfetchers/indirect.cc

@@ -14,7 +14,8 @@ struct IndirectInputScheme : InputScheme
         if (url.scheme != "flake")
             return {};
 
-        auto path = tokenizeString<std::vector<std::string>>(url.path, "/");
+        /* This ignores empty path segments for back-compat. Older versions used a tokenizeString here. */
+        auto path = url.pathSegments(/*skipEmpty=*/true) | std::ranges::to<std::vector<std::string>>();
 
         std::optional<Hash> rev;
         std::optional<std::string> ref;
@@ -82,16 +83,15 @@ struct IndirectInputScheme : InputScheme
 
     ParsedURL toURL(const Input & input) const override
     {
-        ParsedURL url;
-        url.scheme = "flake";
-        url.path = getStrAttr(input.attrs, "id");
+        ParsedURL url{
+            .scheme = "flake",
+            .path = {getStrAttr(input.attrs, "id")},
+        };
         if (auto ref = input.getRef()) {
-            url.path += '/';
-            url.path += *ref;
+            url.path.push_back(*ref);
         };
         if (auto rev = input.getRev()) {
-            url.path += '/';
-            url.path += rev->gitRev();
+            url.path.push_back(rev->gitRev());
         };
         return url;
     }

src/libfetchers/mercurial.cc

@@ -120,7 +120,7 @@ struct MercurialInputScheme : InputScheme
     {
         auto url = parseURL(getStrAttr(input.attrs, "url"));
         if (url.scheme == "file" && !input.getRef() && !input.getRev())
-            return url.path;
+            return renderUrlPathEnsureLegal(url.path);
         return {};
     }
 
@@ -152,7 +152,7 @@ struct MercurialInputScheme : InputScheme
     {
         auto url = parseURL(getStrAttr(input.attrs, "url"));
         bool isLocal = url.scheme == "file";
-        return {isLocal, isLocal ? url.path : url.to_string()};
+        return {isLocal, isLocal ? renderUrlPathEnsureLegal(url.path) : url.to_string()};
     }
 
     StorePath fetchToStore(ref<Store> store, Input & input) const

src/libfetchers/path.cc

@@ -20,7 +20,7 @@ struct PathInputScheme : InputScheme
 
         Input input{settings};
         input.attrs.insert_or_assign("type", "path");
-        input.attrs.insert_or_assign("path", url.path);
+        input.attrs.insert_or_assign("path", renderUrlPathEnsureLegal(url.path));
 
         for (auto & [name, value] : url.query)
             if (name == "rev" || name == "narHash")
@@ -74,7 +74,7 @@ struct PathInputScheme : InputScheme
         query.erase("__final");
         return ParsedURL{
             .scheme = "path",
-            .path = getStrAttr(input.attrs, "path"),
+            .path = splitString<std::vector<std::string>>(getStrAttr(input.attrs, "path"), "/"),
             .query = query,
         };
     }

src/libfetchers/tarball.cc

@@ -43,7 +43,7 @@ DownloadFileResult downloadFile(
     if (cached && !cached->expired)
         return useCached();
 
-    FileTransferRequest request(url);
+    FileTransferRequest request(ValidURL{url});
     request.headers = headers;
     if (cached)
         request.expectedETag = getStrAttr(cached->value, "etag");
@@ -107,20 +107,20 @@ DownloadFileResult downloadFile(
 }
 
 static DownloadTarballResult downloadTarball_(
-    const Settings & settings, const std::string & url, const Headers & headers, const std::string & displayPrefix)
+    const Settings & settings, const std::string & urlS, const Headers & headers, const std::string & displayPrefix)
 {
+    ValidURL url = urlS;
 
     // Some friendly error messages for common mistakes.
     // Namely lets catch when the url is a local file path, but
     // it is not in fact a tarball.
-    if (url.rfind("file://", 0) == 0) {
-        // Remove "file://" prefix to get the local file path
-        std::string localPath = url.substr(7);
-        if (!std::filesystem::exists(localPath)) {
+    if (url.scheme() == "file") {
+        std::filesystem::path localPath = renderUrlPathEnsureLegal(url.path());
+        if (!exists(localPath)) {
             throw Error("tarball '%s' does not exist.", localPath);
         }
-        if (std::filesystem::is_directory(localPath)) {
-            if (std::filesystem::exists(localPath + "/.git")) {
+        if (is_directory(localPath)) {
+            if (exists(localPath / ".git")) {
                 throw Error(
                     "tarball '%s' is a git repository, not a tarball. Please use `git+file` as the scheme.", localPath);
             }
@@ -128,7 +128,7 @@ static DownloadTarballResult downloadTarball_(
         }
     }
 
-    Cache::Key cacheKey{"tarball", {{"url", url}}};
+    Cache::Key cacheKey{"tarball", {{"url", urlS}}};
 
     auto cached = settings.getCache()->lookupExpired(cacheKey);
 
@@ -166,7 +166,7 @@ static DownloadTarballResult downloadTarball_(
 
     /* Note: if the download is cached, `importTarball()` will receive
        no data, which causes it to import an empty tarball. */
-    auto archive = hasSuffix(toLower(parseURL(url).path), ".zip") ? ({
+    auto archive = !url.path().empty() && hasSuffix(toLower(url.path().back()), ".zip") ? ({
         /* In streaming mode, libarchive doesn't handle
            symlinks in zip files correctly (#10649). So write
            the entire file to disk so libarchive can access it
@@ -180,7 +180,7 @@ static DownloadTarballResult downloadTarball_(
         }
         TarArchive{path};
     })
-                                                                  : TarArchive{*source};
+                                                                                        : TarArchive{*source};
     auto tarballCache = getTarballCache();
     auto parseSink = tarballCache->getFileSystemObjectSink();
     auto lastModified = unpackTarfileToSink(archive, *parseSink);
@@ -234,8 +234,11 @@ struct CurlInputScheme : InputScheme
 {
     const StringSet transportUrlSchemes = {"file", "http", "https"};
 
-    bool hasTarballExtension(std::string_view path) const
+    bool hasTarballExtension(const ParsedURL & url) const
     {
+        if (url.path.empty())
+            return false;
+        const auto & path = url.path.back();
         return hasSuffix(path, ".zip") || hasSuffix(path, ".tar") || hasSuffix(path, ".tgz")
                || hasSuffix(path, ".tar.gz") || hasSuffix(path, ".tar.xz") || hasSuffix(path, ".tar.bz2")
                || hasSuffix(path, ".tar.zst");
@@ -336,7 +339,7 @@ struct FileInputScheme : CurlInputScheme
         auto parsedUrlScheme = parseUrlScheme(url.scheme);
         return transportUrlSchemes.count(std::string(parsedUrlScheme.transport))
                && (parsedUrlScheme.application ? parsedUrlScheme.application.value() == schemeName()
-                                               : (!requireTree && !hasTarballExtension(url.path)));
+                                               : (!requireTree && !hasTarballExtension(url)));
     }
 
     std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & _input) const override
@@ -373,7 +376,7 @@ struct TarballInputScheme : CurlInputScheme
 
         return transportUrlSchemes.count(std::string(parsedUrlScheme.transport))
                && (parsedUrlScheme.application ? parsedUrlScheme.application.value() == schemeName()
-                                               : (requireTree || hasTarballExtension(url.path)));
+                                               : (requireTree || hasTarballExtension(url)));
     }
 
     std::pair<ref<SourceAccessor>, Input> getAccessor(ref<Store> store, const Input & _input) const override

src/libflake-tests/flakeref.cc

@@ -2,6 +2,7 @@
 
 #include "nix/fetchers/fetch-settings.hh"
 #include "nix/flake/flakeref.hh"
+#include "nix/fetchers/attrs.hh"
 
 namespace nix {
 
@@ -90,6 +91,158 @@ TEST(parseFlakeRef, GitArchiveInput)
     }
 }
 
+struct InputFromURLTestCase
+{
+    std::string url;
+    fetchers::Attrs attrs;
+    std::string description;
+    std::string expectedUrl = url;
+};
+
+class InputFromURLTest : public ::testing::WithParamInterface<InputFromURLTestCase>, public ::testing::Test
+{};
+
+TEST_P(InputFromURLTest, attrsAreCorrectAndRoundTrips)
+{
+    experimentalFeatureSettings.experimentalFeatures.get().insert(Xp::Flakes);
+    fetchers::Settings fetchSettings;
+
+    const auto & testCase = GetParam();
+
+    auto flakeref = parseFlakeRef(fetchSettings, testCase.url);
+
+    EXPECT_EQ(flakeref.toAttrs(), testCase.attrs);
+    EXPECT_EQ(flakeref.to_string(), testCase.expectedUrl);
+
+    auto input = fetchers::Input::fromURL(fetchSettings, flakeref.to_string());
+
+    EXPECT_EQ(input.toURLString(), testCase.expectedUrl);
+    EXPECT_EQ(input.toAttrs(), testCase.attrs);
+
+    // Round-trip check.
+    auto input2 = fetchers::Input::fromURL(fetchSettings, input.toURLString());
+    EXPECT_EQ(input, input2);
+    EXPECT_EQ(input.toURLString(), input2.toURLString());
+}
+
+using fetchers::Attr;
+
+INSTANTIATE_TEST_SUITE_P(
+    InputFromURL,
+    InputFromURLTest,
+    ::testing::Values(
+        InputFromURLTestCase{
+            .url = "flake:nixpkgs",
+            .attrs =
+                {
+                    {"id", Attr("nixpkgs")},
+                    {"type", Attr("indirect")},
+                },
+            .description = "basic_indirect",
+        },
+        InputFromURLTestCase{
+            .url = "flake:nixpkgs/branch",
+            .attrs =
+                {
+                    {"id", Attr("nixpkgs")},
+                    {"type", Attr("indirect")},
+                    {"ref", Attr("branch")},
+                },
+            .description = "basic_indirect_branch",
+        },
+        InputFromURLTestCase{
+            .url = "nixpkgs/branch",
+            .attrs =
+                {
+                    {"id", Attr("nixpkgs")},
+                    {"type", Attr("indirect")},
+                    {"ref", Attr("branch")},
+                },
+            .description = "flake_id_ref_branch",
+            .expectedUrl = "flake:nixpkgs/branch",
+        },
+        InputFromURLTestCase{
+            .url = "nixpkgs/branch/2aae6c35c94fcfb415dbe95f408b9ce91ee846ed",
+            .attrs =
+                {
+                    {"id", Attr("nixpkgs")},
+                    {"type", Attr("indirect")},
+                    {"ref", Attr("branch")},
+                    {"rev", Attr("2aae6c35c94fcfb415dbe95f408b9ce91ee846ed")},
+                },
+            .description = "flake_id_ref_branch_trailing_slash",
+            .expectedUrl = "flake:nixpkgs/branch/2aae6c35c94fcfb415dbe95f408b9ce91ee846ed",
+        },
+        // The following tests are for back-compat with lax parsers in older versions
+        // that used `tokenizeString` for splitting path segments, which ignores empty
+        // strings.
+        InputFromURLTestCase{
+            .url = "nixpkgs/branch////",
+            .attrs =
+                {
+                    {"id", Attr("nixpkgs")},
+                    {"type", Attr("indirect")},
+                    {"ref", Attr("branch")},
+                },
+            .description = "flake_id_ref_branch_ignore_empty_trailing_segments",
+            .expectedUrl = "flake:nixpkgs/branch",
+        },
+        InputFromURLTestCase{
+            .url = "nixpkgs/branch///2aae6c35c94fcfb415dbe95f408b9ce91ee846ed///",
+            .attrs =
+                {
+                    {"id", Attr("nixpkgs")},
+                    {"type", Attr("indirect")},
+                    {"ref", Attr("branch")},
+                    {"rev", Attr("2aae6c35c94fcfb415dbe95f408b9ce91ee846ed")},
+                },
+            .description = "flake_id_ref_branch_ignore_empty_segments_ref_rev",
+            .expectedUrl = "flake:nixpkgs/branch/2aae6c35c94fcfb415dbe95f408b9ce91ee846ed",
+        },
+        InputFromURLTestCase{
+            // Note that this is different from above because the "flake id" shorthand
+            // doesn't allow this.
+            .url = "flake:/nixpkgs///branch////",
+            .attrs =
+                {
+                    {"id", Attr("nixpkgs")},
+                    {"type", Attr("indirect")},
+                    {"ref", Attr("branch")},
+                },
+            .description = "indirect_branch_empty_segments_everywhere",
+            .expectedUrl = "flake:nixpkgs/branch",
+        },
+        InputFromURLTestCase{
+            // TODO: Technically this has an empty authority, but it's ignored
+            // for now. Yes, this is what all versions going back to at least
+            // 2.18 did and yes, this should not be allowed.
+            .url = "github://////owner%42/////repo%41///branch%43////",
+            .attrs =
+                {
+                    {"type", Attr("github")},
+                    {"owner", Attr("ownerB")},
+                    {"repo", Attr("repoA")},
+                    {"ref", Attr("branchC")},
+                },
+            .description = "github_ref_slashes_in_path_everywhere",
+            .expectedUrl = "github:ownerB/repoA/branchC",
+        },
+        InputFromURLTestCase{
+            // FIXME: Subgroups in gitlab URLs are busted. This double-encoding
+            // behavior exists since 2.18. See issue #9161 and PR #8845.
+            .url = "gitlab:/owner%252Fsubgroup/////repo%41///branch%43////",
+            .attrs =
+                {
+                    {"type", Attr("gitlab")},
+                    {"owner", Attr("owner%2Fsubgroup")},
+                    {"repo", Attr("repoA")},
+                    {"ref", Attr("branchC")},
+                },
+            .description = "gitlab_ref_slashes_in_path_everywhere_with_pct_encoding",
+            .expectedUrl = "gitlab:owner%252Fsubgroup/repoA/branchC",
+        }),
+    [](const ::testing::TestParamInfo<InputFromURLTestCase> & info) { return info.param.description; });
+
 TEST(to_string, doesntReencodeUrl)
 {
     fetchers::Settings fetchSettings;

src/libflake-tests/url-name.cc

@@ -13,8 +13,9 @@ TEST(getNameFromURL, getNameFromURL)
     ASSERT_EQ(getNameFromURL(parseURL("path:~/repos/nixpkgs#packages.x86_64-linux.Hello")), "Hello");
     ASSERT_EQ(getNameFromURL(parseURL("path:.#nonStandardAttr.mylaptop")), "mylaptop");
     ASSERT_EQ(getNameFromURL(parseURL("path:./repos/myflake#nonStandardAttr.mylaptop")), "mylaptop");
-    ASSERT_EQ(getNameFromURL(parseURL("path:./nixpkgs#packages.x86_64-linux.complex^bin,man")), "complex");
-    ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#packages.x86_64-linux.default^*")), "myproj");
+    ASSERT_EQ(
+        getNameFromURL(parseURL("path:./nixpkgs#packages.x86_64-linux.complex^bin,man", /*lenient=*/true)), "complex");
+    ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#packages.x86_64-linux.default^*", /*lenient=*/true)), "myproj");
     ASSERT_EQ(getNameFromURL(parseURL("path:./myproj#defaultPackage.x86_64-linux")), "myproj");
 
     ASSERT_EQ(getNameFromURL(parseURL("github:NixOS/nixpkgs#packages.x86_64-linux.hello")), "hello");
@@ -80,6 +81,6 @@ TEST(getNameFromURL, getNameFromURL)
     ASSERT_EQ(getNameFromURL(parseURL("path:.")), std::nullopt);
     ASSERT_EQ(getNameFromURL(parseURL("file:.#")), std::nullopt);
     ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default")), std::nullopt);
-    ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default^*")), std::nullopt);
+    ASSERT_EQ(getNameFromURL(parseURL("path:.#packages.x86_64-linux.default^*", /*lenient=*/true)), std::nullopt);
 }
 } // namespace nix

src/libflake/flake.cc

@@ -232,7 +232,7 @@ static Flake readFlake(
         .path = flakePath,
     };
 
-    if (auto description = vInfo.attrs()->get(state.sDescription)) {
+    if (auto description = vInfo.attrs()->get(state.s.description)) {
         expectType(state, nString, *description->value, description->pos);
         flake.description = description->value->c_str();
     }
@@ -253,7 +253,7 @@ static Flake readFlake(
 
         if (outputs->value->isLambda() && outputs->value->lambda().fun->hasFormals()) {
             for (auto & formal : outputs->value->lambda().fun->formals->formals) {
-                if (formal.name != state.sSelf)
+                if (formal.name != state.s.self)
                     flake.inputs.emplace(
                         state.symbols[formal.name],
                         FlakeInput{.ref = parseFlakeRef(state.fetchSettings, std::string(state.symbols[formal.name]))});
@@ -305,7 +305,8 @@ static Flake readFlake(
     }
 
     for (auto & attr : *vInfo.attrs()) {
-        if (attr.name != state.sDescription && attr.name != sInputs && attr.name != sOutputs && attr.name != sNixConfig)
+        if (attr.name != state.s.description && attr.name != sInputs && attr.name != sOutputs
+            && attr.name != sNixConfig)
             throw Error(
                 "flake '%s' has an unsupported attribute '%s', at %s",
                 resolvedRef,

src/libflake/flakeref.cc

@@ -82,7 +82,7 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
     auto succeeds = std::regex_match(url, match, pathFlakeRegex);
     assert(succeeds);
     auto path = match[1].str();
-    auto query = decodeQuery(match[3]);
+    auto query = decodeQuery(match[3].str(), /*lenient=*/true);
     auto fragment = percentDecode(match[5].str());
 
     if (baseDir) {
@@ -143,7 +143,7 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
                     auto parsedURL = ParsedURL{
                         .scheme = "git+file",
                         .authority = ParsedURL::Authority{},
-                        .path = flakeRoot,
+                        .path = splitString<std::vector<std::string>>(flakeRoot, "/"),
                         .query = query,
                         .fragment = fragment,
                     };
@@ -172,7 +172,13 @@ std::pair<FlakeRef, std::string> parsePathFlakeRefWithFragment(
 
     return fromParsedURL(
         fetchSettings,
-        {.scheme = "path", .authority = ParsedURL::Authority{}, .path = path, .query = query, .fragment = fragment},
+        {
+            .scheme = "path",
+            .authority = ParsedURL::Authority{},
+            .path = splitString<std::vector<std::string>>(path, "/"),
+            .query = query,
+            .fragment = fragment,
+        },
         isFlake);
 }
 
@@ -192,8 +198,8 @@ parseFlakeIdRef(const fetchers::Settings & fetchSettings, const std::string & ur
     if (std::regex_match(url, match, flakeRegex)) {
         auto parsedURL = ParsedURL{
             .scheme = "flake",
-            .authority = ParsedURL::Authority{},
-            .path = match[1],
+            .authority = std::nullopt,
+            .path = splitString<std::vector<std::string>>(match[1].str(), "/"),
         };
 
         return std::make_pair(
@@ -210,9 +216,13 @@ std::optional<std::pair<FlakeRef, std::string>> parseURLFlakeRef(
     bool isFlake)
 {
     try {
-        auto parsed = parseURL(url);
-        if (baseDir && (parsed.scheme == "path" || parsed.scheme == "git+file") && !isAbsolute(parsed.path))
-            parsed.path = absPath(parsed.path, *baseDir);
+        auto parsed = parseURL(url, /*lenient=*/true);
+        if (baseDir && (parsed.scheme == "path" || parsed.scheme == "git+file")) {
+            /* Here we know that the path must not contain encoded '/' or NUL bytes. */
+            auto path = renderUrlPathEnsureLegal(parsed.path);
+            if (!isAbsolute(path))
+                parsed.path = splitString<std::vector<std::string>>(absPath(path, *baseDir), "/");
+        }
         return fromParsedURL(fetchSettings, std::move(parsed), isFlake);
     } catch (BadURL &) {
         return std::nullopt;
@@ -289,7 +299,7 @@ FlakeRef FlakeRef::canonicalize() const
        filtering the `dir` query parameter from the URL. */
     if (auto url = fetchers::maybeGetStrAttr(flakeRef.input.attrs, "url")) {
         try {
-            auto parsed = parseURL(*url);
+            auto parsed = parseURL(*url, /*lenient=*/true);
             if (auto dir2 = get(parsed.query, "dir")) {
                 if (flakeRef.subdir != "" && flakeRef.subdir == *dir2)
                     parsed.query.erase("dir");

src/libflake/url-name.cc

@@ -27,16 +27,21 @@ std::optional<std::string> getNameFromURL(const ParsedURL & url)
         return match.str(2);
     }
 
+    /* This is not right, because special chars like slashes within the
+       path fragments should be percent encoded, but I don't think any
+       of the regexes above care. */
+    auto path = concatStringsSep("/", url.path);
+
     /* If this is a github/gitlab/sourcehut flake, use the repo name */
-    if (std::regex_match(url.scheme, gitProviderRegex) && std::regex_match(url.path, match, secondPathSegmentRegex))
+    if (std::regex_match(url.scheme, gitProviderRegex) && std::regex_match(path, match, secondPathSegmentRegex))
         return match.str(1);
 
     /* If it is a regular git flake, use the directory name */
-    if (std::regex_match(url.scheme, gitSchemeRegex) && std::regex_match(url.path, match, lastPathSegmentRegex))
+    if (std::regex_match(url.scheme, gitSchemeRegex) && std::regex_match(path, match, lastPathSegmentRegex))
         return match.str(1);
 
     /* If there is no fragment, take the last element of the path */
-    if (std::regex_match(url.path, match, lastPathSegmentRegex))
+    if (std::regex_match(path, match, lastPathSegmentRegex))
         return match.str(1);
 
     /* If even that didn't work, the URL does not contain enough info to determine a useful name */

src/libstore-test-support/include/nix/store/tests/libstore.hh

@@ -6,6 +6,7 @@
 
 #include "nix/store/store-api.hh"
 #include "nix/store/store-open.hh"
+#include "nix/store/globals.hh"
 
 namespace nix {
 

src/libstore-tests/derivation-parser-bench.cc

@@ -28,6 +28,27 @@ static void BM_ParseRealDerivationFile(benchmark::State & state, const std::stri
     state.SetBytesProcessed(state.iterations() * content.size());
 }
 
+// Benchmark unparsing real derivation files
+static void BM_UnparseRealDerivationFile(benchmark::State & state, const std::string & filename)
+{
+    // Read the file once
+    std::ifstream file(filename);
+    std::stringstream buffer;
+    buffer << file.rdbuf();
+    std::string content = buffer.str();
+
+    auto store = openStore("dummy://");
+    ExperimentalFeatureSettings xpSettings;
+    auto drv = parseDerivation(*store, std::string(content), "test", xpSettings);
+
+    for (auto _ : state) {
+        auto unparsed = drv.unparse(*store, /*maskOutputs=*/false);
+        benchmark::DoNotOptimize(unparsed);
+        assert(unparsed.size() == content.size());
+    }
+    state.SetBytesProcessed(state.iterations() * content.size());
+}
+
 // Register benchmarks for actual test derivation files if they exist
 BENCHMARK_CAPTURE(
     BM_ParseRealDerivationFile,
@@ -37,3 +58,11 @@ BENCHMARK_CAPTURE(
     BM_ParseRealDerivationFile,
     firefox,
     getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value_or(NIX_UNIT_TEST_DATA) + "/derivation/firefox.drv");
+BENCHMARK_CAPTURE(
+    BM_UnparseRealDerivationFile,
+    hello,
+    getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value_or(NIX_UNIT_TEST_DATA) + "/derivation/hello.drv");
+BENCHMARK_CAPTURE(
+    BM_UnparseRealDerivationFile,
+    firefox,
+    getEnvNonEmpty("_NIX_TEST_UNIT_DATA").value_or(NIX_UNIT_TEST_DATA) + "/derivation/firefox.drv");

src/libstore-tests/s3.cc

@@ -21,7 +21,7 @@ class ParsedS3URLTest : public ::testing::WithParamInterface<ParsedS3URLTestCase
 TEST_P(ParsedS3URLTest, parseS3URLSuccessfully)
 {
     const auto & testCase = GetParam();
-    auto parsed = ParsedS3URL::parse(testCase.url);
+    auto parsed = ParsedS3URL::parse(parseURL(testCase.url));
     ASSERT_EQ(parsed, testCase.expected);
 }
 
@@ -33,51 +33,57 @@ INSTANTIATE_TEST_SUITE_P(
             "s3://my-bucket/my-key.txt",
             {
                 .bucket = "my-bucket",
-                .key = "my-key.txt",
+                .key = {"my-key.txt"},
             },
-            "basic_s3_bucket"},
+            "basic_s3_bucket",
+        },
         ParsedS3URLTestCase{
             "s3://prod-cache/nix/store/abc123.nar.xz?region=eu-west-1",
             {
                 .bucket = "prod-cache",
-                .key = "nix/store/abc123.nar.xz",
+                .key = {"nix", "store", "abc123.nar.xz"},
                 .region = "eu-west-1",
             },
-            "with_region"},
+            "with_region",
+        },
         ParsedS3URLTestCase{
             "s3://bucket/key?region=us-west-2&profile=prod&endpoint=custom.s3.com&scheme=https&region=us-east-1",
             {
                 .bucket = "bucket",
-                .key = "key",
+                .key = {"key"},
                 .profile = "prod",
                 .region = "us-west-2", //< using the first parameter (decodeQuery ignores dupicates)
                 .scheme = "https",
                 .endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
             },
-            "complex"},
+            "complex",
+        },
         ParsedS3URLTestCase{
             "s3://cache/file.txt?profile=production&region=ap-southeast-2",
             {
                 .bucket = "cache",
-                .key = "file.txt",
+                .key = {"file.txt"},
                 .profile = "production",
                 .region = "ap-southeast-2",
             },
-            "with_profile_and_region"},
+            "with_profile_and_region",
+        },
         ParsedS3URLTestCase{
             "s3://bucket/key?endpoint=https://minio.local&scheme=http",
             {
                 .bucket = "bucket",
-                .key = "key",
+                .key = {"key"},
                 /* TODO: Figure out what AWS SDK is doing when both endpointOverride and scheme are set. */
                 .scheme = "http",
                 .endpoint =
                     ParsedURL{
                         .scheme = "https",
                         .authority = ParsedURL::Authority{.host = "minio.local"},
+                        .path = {""},
                     },
             },
-            "with_absolute_endpoint_uri"}),
+            "with_absolute_endpoint_uri",
+        }),
     [](const ::testing::TestParamInfo<ParsedS3URLTestCase> & info) { return info.param.description; });
 
 TEST(InvalidParsedS3URLTest, parseS3URLErrors)
@@ -86,11 +92,114 @@ TEST(InvalidParsedS3URLTest, parseS3URLErrors)
         testing::HasSubstrIgnoreANSIMatcher("error: URI has a missing or invalid bucket name"));
 
     /* Empty bucket (authority) */
-    ASSERT_THAT([]() { ParsedS3URL::parse("s3:///key"); }, invalidBucketMatcher);
+    ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3:///key")); }, invalidBucketMatcher);
     /* Invalid bucket name */
-    ASSERT_THAT([]() { ParsedS3URL::parse("s3://127.0.0.1"); }, invalidBucketMatcher);
+    ASSERT_THAT([]() { ParsedS3URL::parse(parseURL("s3://127.0.0.1")); }, invalidBucketMatcher);
 }
 
+// Parameterized test for s3ToHttpsUrl conversion
+struct S3ToHttpsConversionTestCase
+{
+    ParsedS3URL input;
+    ParsedURL expected;
+    std::string expectedRendered;
+    std::string description;
+};
+
+class S3ToHttpsConversionTest : public ::testing::WithParamInterface<S3ToHttpsConversionTestCase>,
+                                public ::testing::Test
+{};
+
+TEST_P(S3ToHttpsConversionTest, ConvertsCorrectly)
+{
+    const auto & testCase = GetParam();
+    auto result = testCase.input.toHttpsUrl();
+    EXPECT_EQ(result, testCase.expected) << "Failed for: " << testCase.description;
+    EXPECT_EQ(result.to_string(), testCase.expectedRendered);
+}
+
+INSTANTIATE_TEST_SUITE_P(
+    S3ToHttpsConversion,
+    S3ToHttpsConversionTest,
+    ::testing::Values(
+        S3ToHttpsConversionTestCase{
+            ParsedS3URL{
+                .bucket = "my-bucket",
+                .key = {"my-key.txt"},
+            },
+            ParsedURL{
+                .scheme = "https",
+                .authority = ParsedURL::Authority{.host = "s3.us-east-1.amazonaws.com"},
+                .path = {"", "my-bucket", "my-key.txt"},
+            },
+            "https://s3.us-east-1.amazonaws.com/my-bucket/my-key.txt",
+            "basic_s3_default_region",
+        },
+        S3ToHttpsConversionTestCase{
+            ParsedS3URL{
+                .bucket = "prod-cache",
+                .key = {"nix", "store", "abc123.nar.xz"},
+                .region = "eu-west-1",
+            },
+            ParsedURL{
+                .scheme = "https",
+                .authority = ParsedURL::Authority{.host = "s3.eu-west-1.amazonaws.com"},
+                .path = {"", "prod-cache", "nix", "store", "abc123.nar.xz"},
+            },
+            "https://s3.eu-west-1.amazonaws.com/prod-cache/nix/store/abc123.nar.xz",
+            "with_eu_west_1_region",
+        },
+        S3ToHttpsConversionTestCase{
+            ParsedS3URL{
+                .bucket = "bucket",
+                .key = {"key"},
+                .scheme = "http",
+                .endpoint = ParsedURL::Authority{.host = "custom.s3.com"},
+            },
+            ParsedURL{
+                .scheme = "http",
+                .authority = ParsedURL::Authority{.host = "custom.s3.com"},
+                .path = {"", "bucket", "key"},
+            },
+            "http://custom.s3.com/bucket/key",
+            "custom_endpoint_authority",
+        },
+        S3ToHttpsConversionTestCase{
+            ParsedS3URL{
+                .bucket = "bucket",
+                .key = {"key"},
+                .endpoint =
+                    ParsedURL{
+                        .scheme = "http",
+                        .authority = ParsedURL::Authority{.host = "server", .port = 9000},
+                        .path = {""},
+                    },
+            },
+            ParsedURL{
+                .scheme = "http",
+                .authority = ParsedURL::Authority{.host = "server", .port = 9000},
+                .path = {"", "bucket", "key"},
+            },
+            "http://server:9000/bucket/key",
+            "custom_endpoint_with_port",
+        },
+        S3ToHttpsConversionTestCase{
+            ParsedS3URL{
+                .bucket = "bucket",
+                .key = {"path", "to", "file.txt"},
+                .region = "ap-southeast-2",
+                .scheme = "https",
+            },
+            ParsedURL{
+                .scheme = "https",
+                .authority = ParsedURL::Authority{.host = "s3.ap-southeast-2.amazonaws.com"},
+                .path = {"", "bucket", "path", "to", "file.txt"},
+            },
+            "https://s3.ap-southeast-2.amazonaws.com/bucket/path/to/file.txt",
+            "complex_path_and_region",
+        }),
+    [](const ::testing::TestParamInfo<S3ToHttpsConversionTestCase> & info) { return info.param.description; });
+
 } // namespace nix
 
 #endif

src/libstore/async-path-writer.cc

@@ -1,151 +0,0 @@
-#include "nix/store/async-path-writer.hh"
-#include "nix/util/archive.hh"
-
-#include <thread>
-#include <future>
-
-namespace nix {
-
-struct AsyncPathWriterImpl : AsyncPathWriter
-{
-    ref<Store> store;
-
-    struct Item
-    {
-        StorePath storePath;
-        std::string contents;
-        std::string name;
-        Hash hash;
-        StorePathSet references;
-        RepairFlag repair;
-        std::promise<void> promise;
-    };
-
-    struct State
-    {
-        std::vector<Item> items;
-        std::unordered_map<StorePath, std::shared_future<void>> futures;
-        bool quit = false;
-    };
-
-    Sync<State> state_;
-
-    std::thread workerThread;
-
-    std::condition_variable wakeupCV;
-
-    AsyncPathWriterImpl(ref<Store> store)
-        : store(store)
-    {
-        workerThread = std::thread([&]() {
-            while (true) {
-                std::vector<Item> items;
-
-                {
-                    auto state(state_.lock());
-                    while (!state->quit && state->items.empty())
-                        state.wait(wakeupCV);
-                    if (state->items.empty() && state->quit)
-                        return;
-                    std::swap(items, state->items);
-                }
-
-                try {
-                    writePaths(items);
-                    for (auto & item : items)
-                        item.promise.set_value();
-                } catch (...) {
-                    for (auto & item : items)
-                        item.promise.set_exception(std::current_exception());
-                }
-            }
-        });
-    }
-
-    ~AsyncPathWriterImpl()
-    {
-        state_.lock()->quit = true;
-        wakeupCV.notify_all();
-        workerThread.join();
-    }
-
-    StorePath
-    addPath(std::string contents, std::string name, StorePathSet references, RepairFlag repair, bool readOnly) override
-    {
-        auto hash = hashString(HashAlgorithm::SHA256, contents);
-
-        auto storePath = store->makeFixedOutputPathFromCA(
-            name,
-            TextInfo{
-                .hash = hash,
-                .references = references,
-            });
-
-        if (!readOnly) {
-            auto state(state_.lock());
-            std::promise<void> promise;
-            state->futures.insert_or_assign(storePath, promise.get_future());
-            state->items.push_back(
-                Item{
-                    .storePath = storePath,
-                    .contents = std::move(contents),
-                    .name = std::move(name),
-                    .hash = hash,
-                    .references = std::move(references),
-                    .repair = repair,
-                    .promise = std::move(promise),
-                });
-            wakeupCV.notify_all();
-        }
-
-        return storePath;
-    }
-
-    void waitForPath(const StorePath & path) override
-    {
-        auto future = ({
-            auto state = state_.lock();
-            auto i = state->futures.find(path);
-            if (i == state->futures.end())
-                return;
-            i->second;
-        });
-        future.get();
-    }
-
-    void waitForAllPaths() override
-    {
-        auto futures = ({
-            auto state(state_.lock());
-            std::move(state->futures);
-        });
-        for (auto & future : futures)
-            future.second.get();
-    }
-
-    void writePaths(const std::vector<Item> & items)
-    {
-        // FIXME: use addMultipeToStore() once it doesn't require a
-        // NAR hash from the client for CA objects.
-
-        for (auto & item : items) {
-            StringSource source(item.contents);
-            auto storePath = store->addToStoreFromDump(
-                source,
-                item.storePath.name(),
-                FileSerialisationMethod::Flat,
-                ContentAddressMethod::Raw::Text,
-                HashAlgorithm::SHA256,
-                item.references,
-                item.repair);
-            assert(storePath == item.storePath);
-        }
-    }
-};
-
-ref<AsyncPathWriter> AsyncPathWriter::make(ref<Store> store)
-{
-    return make_ref<AsyncPathWriterImpl>(store);
-}
-
-} // namespace nix

src/libstore/build/derivation-building-goal.cc

@@ -1,4 +1,5 @@
 #include "nix/store/build/derivation-building-goal.hh"
+#include "nix/store/build/derivation-env-desugar.hh"
 #include "nix/store/build/derivation-trampoline-goal.hh"
 #ifndef _WIN32 // TODO enable build hook on Windows
 #  include "nix/store/build/hook-instance.hh"
@@ -12,6 +13,7 @@
 #include "nix/store/common-protocol.hh"
 #include "nix/store/common-protocol-impl.hh"
 #include "nix/store/local-store.hh" // TODO remove, along with remaining downcasts
+#include "nix/store/globals.hh"
 
 #include <fstream>
 #include <sys/types.h>
@@ -52,24 +54,9 @@ DerivationBuildingGoal::~DerivationBuildingGoal()
 {
     /* Careful: we should never ever throw an exception from a
        destructor. */
-    try {
-        killChild();
-    } catch (...) {
-        ignoreExceptionInDestructor();
-    }
 #ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
-    if (builder) {
-        try {
-            builder->stopDaemon();
-        } catch (...) {
-            ignoreExceptionInDestructor();
-        }
-        try {
-            builder->deleteTmpDir(false);
-        } catch (...) {
-            ignoreExceptionInDestructor();
-        }
-    }
+    if (builder)
+        builder.reset();
 #endif
     try {
         closeLogFile();
@@ -93,22 +80,8 @@ void DerivationBuildingGoal::killChild()
     hook.reset();
 #endif
 #ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
-    if (builder && builder->pid != -1) {
+    if (builder && builder->killChild())
         worker.childTerminated(this);
-
-        // FIXME: move this into DerivationBuilder.
-
-        /* If we're using a build user, then there is a tricky race
-           condition: if we kill the build user before the child has
-           done its setuid() to the build user uid, then it won't be
-           killed, and we'll potentially lock up in pid.wait().  So
-           also send a conventional kill to the child. */
-        ::kill(-builder->pid, SIGKILL); /* ignore the result */
-
-        builder->killSandbox(true);
-
-        builder->pid.wait();
-    }
 #endif
 }
 
@@ -117,7 +90,7 @@ void DerivationBuildingGoal::timedOut(Error && ex)
     killChild();
     // We're not inside a coroutine, hence we can't use co_return here.
     // Thus we ignore the return value.
-    [[maybe_unused]] Done _ = done(BuildResult::TimedOut, {}, std::move(ex));
+    [[maybe_unused]] Done _ = doneFailure({BuildResult::TimedOut, std::move(ex)});
 }
 
 /**
@@ -147,6 +120,9 @@ std::string showKnownOutputs(const StoreDirConfig & store, const Derivation & dr
     return msg;
 }
 
+static void runPostBuildHook(
+    const StoreDirConfig & store, Logger & logger, const StorePath & drvPath, const StorePathSet & outputPaths);
+
 /* At least one of the output paths could not be
    produced using a substitute.  So we have to build instead. */
 Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution()
@@ -254,7 +230,7 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution()
                 nrFailed,
                 nrFailed == 1 ? "dependency" : "dependencies");
         msg += showKnownOutputs(worker.store, *drv);
-        co_return done(BuildResult::DependencyFailed, {}, Error(msg));
+        co_return doneFailure(BuildError(BuildResult::DependencyFailed, msg));
     }
 
     /* Gather information necessary for computing the closure and/or
@@ -355,9 +331,9 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution()
 
             auto resolvedResult = resolvedDrvGoal->buildResult;
 
-            SingleDrvOutputs builtOutputs;
-
             if (resolvedResult.success()) {
+                SingleDrvOutputs builtOutputs;
+
                 auto resolvedHashes = staticOutputHashes(worker.store, drvResolved);
 
                 StorePathSet outputPaths;
@@ -407,13 +383,19 @@ Goal::Co DerivationBuildingGoal::gaveUpOnSubstitution()
                 }
 
                 runPostBuildHook(worker.store, *logger, drvPath, outputPaths);
+
+                auto status = resolvedResult.status;
+                if (status == BuildResult::AlreadyValid)
+                    status = BuildResult::ResolvesToAlreadyValid;
+
+                co_return doneSuccess(status, std::move(builtOutputs));
+            } else {
+                co_return doneFailure({
+                    BuildResult::DependencyFailed,
+                    "build of resolved derivation '%s' failed",
+                    worker.store.printStorePath(pathResolved),
+                });
             }
-
-            auto status = resolvedResult.status;
-            if (status == BuildResult::AlreadyValid)
-                status = BuildResult::ResolvesToAlreadyValid;
-
-            co_return done(status, std::move(builtOutputs));
         }
 
         /* If we get this far, we know no dynamic drvs inputs */
@@ -538,7 +520,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
         debug("skipping build of derivation '%s', someone beat us to it", worker.store.printStorePath(drvPath));
         outputLocks.setDeletion(true);
         outputLocks.unlock();
-        co_return done(BuildResult::AlreadyValid, std::move(validOutputs));
+        co_return doneSuccess(BuildResult::AlreadyValid, std::move(validOutputs));
     }
 
     /* If any of the outputs already exist but are not valid, delete
@@ -642,21 +624,6 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
                     goal.worker.childTerminated(&goal);
                 }
 
-                void noteHashMismatch() override
-                {
-                    goal.worker.hashMismatch = true;
-                }
-
-                void noteCheckMismatch() override
-                {
-                    goal.worker.checkMismatch = true;
-                }
-
-                void markContentsGood(const StorePath & path) override
-                {
-                    goal.worker.markContentsGood(path);
-                }
-
                 Path openLogFile() override
                 {
                     return goal.openLogFile();
@@ -666,29 +633,54 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
                 {
                     goal.closeLogFile();
                 }
-
-                void appendLogTailErrorMsg(std::string & msg) override
-                {
-                    goal.appendLogTailErrorMsg(msg);
-                }
             };
 
             auto * localStoreP = dynamic_cast<LocalStore *>(&worker.store);
             assert(localStoreP);
 
+            decltype(DerivationBuilderParams::defaultPathsInChroot) defaultPathsInChroot = settings.sandboxPaths.get();
+            DesugaredEnv desugaredEnv;
+
+            /* Add the closure of store paths to the chroot. */
+            StorePathSet closure;
+            for (auto & i : defaultPathsInChroot)
+                try {
+                    if (worker.store.isInStore(i.second.source))
+                        worker.store.computeFSClosure(worker.store.toStorePath(i.second.source).first, closure);
+                } catch (InvalidPath & e) {
+                } catch (Error & e) {
+                    e.addTrace({}, "while processing sandbox path '%s'", i.second.source);
+                    throw;
+                }
+            for (auto & i : closure) {
+                auto p = worker.store.printStorePath(i);
+                defaultPathsInChroot.insert_or_assign(p, ChrootPath{.source = p});
+            }
+
+            try {
+                desugaredEnv = DesugaredEnv::create(worker.store, *drv, *drvOptions, inputPaths);
+            } catch (BuildError & e) {
+                outputLocks.unlock();
+                worker.permanentFailure = true;
+                co_return doneFailure(std::move(e));
+            }
+
             /* If we have to wait and retry (see below), then `builder` will
                already be created, so we don't need to create it again. */
             builder = makeDerivationBuilder(
                 *localStoreP,
                 std::make_unique<DerivationBuildingGoalCallbacks>(*this, builder),
                 DerivationBuilderParams{
-                    drvPath,
-                    buildMode,
-                    buildResult,
-                    *drv,
-                    *drvOptions,
-                    inputPaths,
-                    initialOutputs,
+                    .drvPath = drvPath,
+                    .buildResult = buildResult,
+                    .drv = *drv,
+                    .drvOptions = *drvOptions,
+                    .inputPaths = inputPaths,
+                    .initialOutputs = initialOutputs,
+                    .buildMode = buildMode,
+                    .defaultPathsInChroot = std::move(defaultPathsInChroot),
+                    .systemFeatures = worker.store.config.systemFeatures.get(),
+                    .desugaredEnv = std::move(desugaredEnv),
                 });
         }
 
@@ -717,7 +709,7 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
         builder.reset();
         outputLocks.unlock();
         worker.permanentFailure = true;
-        co_return done(BuildResult::InputRejected, {}, std::move(e));
+        co_return doneFailure(std::move(e)); // InputRejected
     }
 
     started();
@@ -725,26 +717,60 @@ Goal::Co DerivationBuildingGoal::tryToBuild()
 
     trace("build done");
 
-    auto res = builder->unprepareBuild();
-    // N.B. cannot use `std::visit` with co-routine return
-    if (auto * ste = std::get_if<0>(&res)) {
+    SingleDrvOutputs builtOutputs;
+    try {
+        builtOutputs = builder->unprepareBuild();
+    } catch (BuilderFailureError & e) {
+        builder.reset();
         outputLocks.unlock();
-        co_return done(std::move(ste->first), {}, std::move(ste->second));
-    } else if (auto * builtOutputs = std::get_if<1>(&res)) {
+        co_return doneFailure(fixupBuilderFailureErrorMessage(std::move(e)));
+    } catch (BuildError & e) {
+        builder.reset();
+        outputLocks.unlock();
+// Allow selecting a subset of enum values
+#  pragma GCC diagnostic push
+#  pragma GCC diagnostic ignored "-Wswitch-enum"
+        switch (e.status) {
+        case BuildResult::HashMismatch:
+            worker.hashMismatch = true;
+            /* See header, the protocols don't know about `HashMismatch`
+               yet, so change it to `OutputRejected`, which they expect
+               for this case (hash mismatch is a type of output
+               rejection). */
+            e.status = BuildResult::OutputRejected;
+            break;
+        case BuildResult::NotDeterministic:
+            worker.checkMismatch = true;
+            break;
+        default:
+            /* Other statuses need no adjusting */
+            break;
+        }
+#  pragma GCC diagnostic pop
+        co_return doneFailure(std::move(e));
+    }
+    {
+        builder.reset();
+        StorePathSet outputPaths;
+        for (auto & [_, output] : builtOutputs) {
+            // for sake of `bmRepair`
+            worker.markContentsGood(output.outPath);
+            outputPaths.insert(output.outPath);
+        }
+        runPostBuildHook(worker.store, *logger, drvPath, outputPaths);
+
         /* It is now safe to delete the lock files, since all future
            lockers will see that the output paths are valid; they will
            not create new lock files with the same names as the old
            (unlinked) lock files. */
         outputLocks.setDeletion(true);
         outputLocks.unlock();
-        co_return done(BuildResult::Built, std::move(*builtOutputs));
-    } else {
-        unreachable();
+        co_return doneSuccess(BuildResult::Built, std::move(builtOutputs));
     }
 #endif
 }
 
-void runPostBuildHook(
+static void runPostBuildHook(
     const StoreDirConfig & store, Logger & logger, const StorePath & drvPath, const StorePathSet & outputPaths)
 {
     auto hook = settings.postBuildHook;
@@ -810,8 +836,16 @@ void runPostBuildHook(
     });
 }
 
-void DerivationBuildingGoal::appendLogTailErrorMsg(std::string & msg)
+BuildError DerivationBuildingGoal::fixupBuilderFailureErrorMessage(BuilderFailureError e)
 {
+    auto msg =
+        fmt("Cannot build '%s'.\n"
+            "Reason: " ANSI_RED "builder %s" ANSI_NORMAL ".",
+            Magenta(worker.store.printStorePath(drvPath)),
+            statusToString(e.builderStatus));
+
+    msg += showKnownOutputs(worker.store, *drv);
+
     if (!logger->isVerbose() && !logTail.empty()) {
         msg += fmt("\nLast %d log lines:\n", logTail.size());
         for (auto & line : logTail) {
@@ -828,6 +862,10 @@ void DerivationBuildingGoal::appendLogTailErrorMsg(std::string & msg)
                 nixLogCommand,
                 worker.store.printStorePath(drvPath));
     }
+
+    msg += e.extraMsgAfter;
+
+    return BuildError{e.status, msg};
 }
 
 Goal::Co DerivationBuildingGoal::hookDone()
@@ -868,21 +906,13 @@ Goal::Co DerivationBuildingGoal::hookDone()
 
     /* Check the exit status. */
     if (!statusOk(status)) {
-        auto msg =
-            fmt("Cannot build '%s'.\n"
-                "Reason: " ANSI_RED "builder %s" ANSI_NORMAL ".",
-                Magenta(worker.store.printStorePath(drvPath)),
-                statusToString(status));
-
-        msg += showKnownOutputs(worker.store, *drv);
-
-        appendLogTailErrorMsg(msg);
+        auto e = fixupBuilderFailureErrorMessage({BuildResult::MiscFailure, status, ""});
 
         outputLocks.unlock();
 
         /* TODO (once again) support fine-grained error codes, see issue #12641. */
 
-        co_return done(BuildResult::MiscFailure, {}, BuildError(msg));
+        co_return doneFailure(std::move(e));
     }
 
     /* Compute the FS closure of the outputs and register them as
@@ -909,7 +939,7 @@ Goal::Co DerivationBuildingGoal::hookDone()
     outputLocks.setDeletion(true);
     outputLocks.unlock();
 
-    co_return done(BuildResult::Built, std::move(builtOutputs));
+    co_return doneSuccess(BuildResult::Built, std::move(builtOutputs));
 }
 
 HookReply DerivationBuildingGoal::tryBuildHook()
@@ -1091,10 +1121,11 @@ void DerivationBuildingGoal::handleChildOutput(Descriptor fd, std::string_view d
             killChild();
             // We're not inside a coroutine, hence we can't use co_return here.
             // Thus we ignore the return value.
-            [[maybe_unused]] Done _ = done(
+            [[maybe_unused]] Done _ = doneFailure(BuildError(
                 BuildResult::LogLimitExceeded,
-                {},
-                Error("%s killed after writing more than %d bytes of log output", getName(), settings.maxLogSize));
+                "%s killed after writing more than %d bytes of log output",
+                getName(),
+                settings.maxLogSize));
             return;
         }
 
@@ -1255,13 +1286,29 @@ SingleDrvOutputs DerivationBuildingGoal::assertPathValidity()
     return validOutputs;
 }
 
-Goal::Done
-DerivationBuildingGoal::done(BuildResult::Status status, SingleDrvOutputs builtOutputs, std::optional<Error> ex)
+Goal::Done DerivationBuildingGoal::doneSuccess(BuildResult::Status status, SingleDrvOutputs builtOutputs)
 {
     outputLocks.unlock();
     buildResult.status = status;
-    if (ex)
-        buildResult.errorMsg = fmt("%s", Uncolored(ex->info().msg));
+
+    assert(buildResult.success());
+
+    mcRunningBuilds.reset();
+
+    buildResult.builtOutputs = std::move(builtOutputs);
+    if (status == BuildResult::Built)
+        worker.doneBuilds++;
+
+    worker.updateProgress();
+
+    return amDone(ecSuccess, std::nullopt);
+}
+
+Goal::Done DerivationBuildingGoal::doneFailure(BuildError ex)
+{
+    outputLocks.unlock();
+    buildResult.status = ex.status;
+    buildResult.errorMsg = fmt("%s", Uncolored(ex.info().msg));
     if (buildResult.status == BuildResult::TimedOut)
         worker.timedOut = true;
     if (buildResult.status == BuildResult::PermanentFailure)
@@ -1269,25 +1316,12 @@ DerivationBuildingGoal::done(BuildResult::Status status, SingleDrvOutputs builtO
 
     mcRunningBuilds.reset();
 
-    if (buildResult.success()) {
-        buildResult.builtOutputs = std::move(builtOutputs);
-        if (status == BuildResult::Built)
-            worker.doneBuilds++;
-    } else {
-        if (status != BuildResult::DependencyFailed)
-            worker.failedBuilds++;
-    }
+    if (ex.status != BuildResult::DependencyFailed)
+        worker.failedBuilds++;
 
     worker.updateProgress();
 
-    auto traceBuiltOutputsFile = getEnv("_NIX_TRACE_BUILT_OUTPUTS").value_or("");
-    if (traceBuiltOutputsFile != "") {
-        std::fstream fs;
-        fs.open(traceBuiltOutputsFile, std::fstream::out);
-        fs << worker.store.printStorePath(drvPath) << "\t" << buildResult.toString() << std::endl;
-    }
-
-    return amDone(buildResult.success() ? ecSuccess : ecFailed, std::move(ex));
+    return amDone(ecFailed, {std::move(ex)});
 }
 
 } // namespace nix

src/libstore/build/derivation-check.cc

@@ -0,0 +1,190 @@
+#include <queue>
+
+#include "nix/store/store-api.hh"
+#include "nix/store/build-result.hh"
+
+#include "derivation-check.hh"
+
+namespace nix {
+
+void checkOutputs(
+    Store & store,
+    const StorePath & drvPath,
+    const decltype(Derivation::outputs) & drvOutputs,
+    const decltype(DerivationOptions::outputChecks) & outputChecks,
+    const std::map<std::string, ValidPathInfo> & outputs)
+{
+    std::map<Path, const ValidPathInfo &> outputsByPath;
+    for (auto & output : outputs)
+        outputsByPath.emplace(store.printStorePath(output.second.path), output.second);
+
+    for (auto & [outputName, info] : outputs) {
+
+        auto * outputSpec = get(drvOutputs, outputName);
+        assert(outputSpec);
+
+        if (const auto * dof = std::get_if<DerivationOutput::CAFixed>(&outputSpec->raw)) {
+            auto & wanted = dof->ca.hash;
+
+            /* Check wanted hash */
+            assert(info.ca);
+            auto & got = info.ca->hash;
+            if (wanted != got) {
+                /* Throw an error after registering the path as
+                   valid. */
+                throw BuildError(
+                    BuildResult::HashMismatch,
+                    "hash mismatch in fixed-output derivation '%s':\n  specified: %s\n     got:    %s",
+                    store.printStorePath(drvPath),
+                    wanted.to_string(HashFormat::SRI, true),
+                    got.to_string(HashFormat::SRI, true));
+            }
+            if (!info.references.empty()) {
+                auto numViolations = info.references.size();
+                throw BuildError(
+                    BuildResult::HashMismatch,
+                    "fixed-output derivations must not reference store paths: '%s' references %d distinct paths, e.g. '%s'",
+                    store.printStorePath(drvPath),
+                    numViolations,
+                    store.printStorePath(*info.references.begin()));
+            }
+        }
+
+        /* Compute the closure and closure size of some output. This
+           is slightly tricky because some of its references (namely
+           other outputs) may not be valid yet. */
+        auto getClosure = [&](const StorePath & path) {
+            uint64_t closureSize = 0;
+            StorePathSet pathsDone;
+            std::queue<StorePath> pathsLeft;
+            pathsLeft.push(path);
+
+            while (!pathsLeft.empty()) {
+                auto path = pathsLeft.front();
+                pathsLeft.pop();
+                if (!pathsDone.insert(path).second)
+                    continue;
+
+                auto i = outputsByPath.find(store.printStorePath(path));
+                if (i != outputsByPath.end()) {
+                    closureSize += i->second.narSize;
+                    for (auto & ref : i->second.references)
+                        pathsLeft.push(ref);
+                } else {
+                    auto info = store.queryPathInfo(path);
+                    closureSize += info->narSize;
+                    for (auto & ref : info->references)
+                        pathsLeft.push(ref);
+                }
+            }
+
+            return std::make_pair(std::move(pathsDone), closureSize);
+        };
+
+        auto applyChecks = [&](const DerivationOptions::OutputChecks & checks) {
+            if (checks.maxSize && info.narSize > *checks.maxSize)
+                throw BuildError(
+                    BuildResult::OutputRejected,
+                    "path '%s' is too large at %d bytes; limit is %d bytes",
+                    store.printStorePath(info.path),
+                    info.narSize,
+                    *checks.maxSize);
+
+            if (checks.maxClosureSize) {
+                uint64_t closureSize = getClosure(info.path).second;
+                if (closureSize > *checks.maxClosureSize)
+                    throw BuildError(
+                        BuildResult::OutputRejected,
+                        "closure of path '%s' is too large at %d bytes; limit is %d bytes",
+                        store.printStorePath(info.path),
+                        closureSize,
+                        *checks.maxClosureSize);
+            }
+
+            auto checkRefs = [&](const StringSet & value, bool allowed, bool recursive) {
+                /* Parse a list of reference specifiers.  Each element must
+                   either be a store path, or the symbolic name of the output
+                   of the derivation (such as `out'). */
+                StorePathSet spec;
+                for (auto & i : value) {
+                    if (store.isStorePath(i))
+                        spec.insert(store.parseStorePath(i));
+                    else if (auto output = get(outputs, i))
+                        spec.insert(output->path);
+                    else {
+                        std::string outputsListing =
+                            concatMapStringsSep(", ", outputs, [](auto & o) { return o.first; });
+                        throw BuildError(
+                            BuildResult::OutputRejected,
+                            "derivation '%s' output check for '%s' contains an illegal reference specifier '%s',"
+                            " expected store path or output name (one of [%s])",
+                            store.printStorePath(drvPath),
+                            outputName,
+                            i,
+                            outputsListing);
+                    }
+                }
+
+                auto used = recursive ? getClosure(info.path).first : info.references;
+
+                if (recursive && checks.ignoreSelfRefs)
+                    used.erase(info.path);
+
+                StorePathSet badPaths;
+
+                for (auto & i : used)
+                    if (allowed) {
+                        if (!spec.count(i))
+                            badPaths.insert(i);
+                    } else {
+                        if (spec.count(i))
+                            badPaths.insert(i);
+                    }
+
+                if (!badPaths.empty()) {
+                    std::string badPathsStr;
+                    for (auto & i : badPaths) {
+                        badPathsStr += "\n  ";
+                        badPathsStr += store.printStorePath(i);
+                    }
+                    throw BuildError(
+                        BuildResult::OutputRejected,
+                        "output '%s' is not allowed to refer to the following paths:%s",
+                        store.printStorePath(info.path),
+                        badPathsStr);
+                }
+            };
+
+            /* Mandatory check: absent whitelist, and present but empty
+               whitelist mean very different things. */
+            if (auto & refs = checks.allowedReferences) {
+                checkRefs(*refs, true, false);
+            }
+            if (auto & refs = checks.allowedRequisites) {
+                checkRefs(*refs, true, true);
+            }
+
+            /* Optimization: don't need to do anything when
+               disallowed and empty set. */
+            if (!checks.disallowedReferences.empty()) {
+                checkRefs(checks.disallowedReferences, false, false);
+            }
+            if (!checks.disallowedRequisites.empty()) {
+                checkRefs(checks.disallowedRequisites, false, true);
+            }
+        };
+
+        std::visit(
+            overloaded{
+                [&](const DerivationOptions::OutputChecks & checks) { applyChecks(checks); },
+                [&](const std::map<std::string, DerivationOptions::OutputChecks> & checksPerOutput) {
+                    if (auto outputChecks = get(checksPerOutput, outputName))
+
+                        applyChecks(*outputChecks);
+                },
+            },
+            outputChecks);
+    }
+}
+
+} // namespace nix

src/libstore/build/derivation-check.hh

@@ -0,0 +1,27 @@
+#pragma once
+///@file
+
+#include "nix/store/derivations.hh"
+#include "nix/store/derivation-options.hh"
+#include "nix/store/path-info.hh"
+
+namespace nix {
+
+/**
+ * Check that outputs meets the requirements specified by the
+ * 'outputChecks' attribute (or the legacy
+ * '{allowed,disallowed}{References,Requisites}' attributes).
+ *
+ * The outputs may not be valid yet, hence outputs needs to contain all
+ * needed info like the NAR size. However, the external (not other
+ * output) references of the output must be valid, so we can compute the
+ * closure size.
+ */
+void checkOutputs(
+    Store & store,
+    const StorePath & drvPath,
+    const decltype(Derivation::outputs) & drvOutputs,
+    const decltype(DerivationOptions::outputChecks) & drvOptions,
+    const std::map<std::string, ValidPathInfo> & outputs);
+
+} // namespace nix

src/libstore/build/derivation-env-desugar.cc

@@ -0,0 +1,59 @@
+#include "nix/store/build/derivation-env-desugar.hh"
+#include "nix/store/store-api.hh"
+#include "nix/store/derivations.hh"
+#include "nix/store/derivation-options.hh"
+
+namespace nix {
+
+std::string & DesugaredEnv::atFileEnvPair(std::string_view name, std::string fileName)
+{
+    auto & ret = extraFiles[fileName];
+    variables.insert_or_assign(
+        std::string{name},
+        EnvEntry{
+            .prependBuildDirectory = true,
+            .value = std::move(fileName),
+        });
+    return ret;
+}
+
+DesugaredEnv DesugaredEnv::create(
+    Store & store, const Derivation & drv, const DerivationOptions & drvOptions, const StorePathSet & inputPaths)
+{
+    DesugaredEnv res;
+
+    if (drv.structuredAttrs) {
+        auto json = drv.structuredAttrs->prepareStructuredAttrs(store, drvOptions, inputPaths, drv.outputs);
+        res.atFileEnvPair("NIX_ATTRS_SH_FILE", ".attrs.sh") = StructuredAttrs::writeShell(json);
+        res.atFileEnvPair("NIX_ATTRS_JSON_FILE", ".attrs.json") = json.dump();
+    } else {
+        /* In non-structured mode, set all bindings either directory in the
+           environment or via a file, as specified by
+           `DerivationOptions::passAsFile`. */
+        for (auto & [envName, envValue] : drv.env) {
+            if (!drvOptions.passAsFile.contains(envName)) {
+                res.variables.insert_or_assign(
+                    envName,
+                    EnvEntry{
+                        .value = envValue,
+                    });
+            } else {
+                res.atFileEnvPair(
+                    envName + "Path",
+                    ".attr-" + hashString(HashAlgorithm::SHA256, envName).to_string(HashFormat::Nix32, false)) =
+                    envValue;
+            }
+        }
+
+        /* Handle exportReferencesGraph(), if set. */
+        for (auto & [fileName, storePaths] : drvOptions.getParsedExportReferencesGraph(store)) {
+            /* Write closure info to <fileName>. */
+            res.extraFiles.insert_or_assign(
+                fileName, store.makeValidityRegistration(store.exportReferences(storePaths, inputPaths), false, false));
+        }
+    }
+
+    return res;
+}
+
+} // namespace nix

src/libstore/build/derivation-goal.cc

@@ -11,7 +11,7 @@
 #include "nix/util/compression.hh"
 #include "nix/store/common-protocol.hh"
 #include "nix/store/common-protocol-impl.hh" // Don't remove is actually needed
-#include "nix/store/local-store.hh"          // TODO remove, along with remaining downcasts
+#include "nix/store/globals.hh"
 
 #include <fstream>
 #include <sys/types.h>
@@ -94,7 +94,7 @@ Goal::Co DerivationGoal::haveDerivation()
 
         /* If they are all valid, then we're done. */
         if (checkResult && checkResult->second == PathStatus::Valid && buildMode == bmNormal) {
-            co_return done(BuildResult::AlreadyValid, checkResult->first);
+            co_return doneSuccess(BuildResult::AlreadyValid, checkResult->first);
         }
 
         Goals waitees;
@@ -122,12 +122,10 @@ Goal::Co DerivationGoal::haveDerivation()
         assert(!drv->type().isImpure());
 
         if (nrFailed > 0 && nrFailed > nrNoSubstituters && !settings.tryFallback) {
-            co_return done(
+            co_return doneFailure(BuildError(
                 BuildResult::TransientFailure,
-                {},
-                Error(
-                    "some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ",
-                    worker.store.printStorePath(drvPath)));
+                "some substitutes for the outputs of derivation '%s' failed (usually happens due to networking issues); try '--fallback' to build derivation from source ",
+                worker.store.printStorePath(drvPath)));
         }
 
         nrFailed = nrNoSubstituters = 0;
@@ -137,7 +135,7 @@ Goal::Co DerivationGoal::haveDerivation()
         bool allValid = checkResult && checkResult->second == PathStatus::Valid;
 
         if (buildMode == bmNormal && allValid) {
-            co_return done(BuildResult::Substituted, checkResult->first);
+            co_return doneSuccess(BuildResult::Substituted, checkResult->first);
         }
         if (buildMode == bmRepair && allValid) {
             co_return repairClosure();
@@ -281,7 +279,7 @@ Goal::Co DerivationGoal::repairClosure()
                 "some paths in the output closure of derivation '%s' could not be repaired",
                 worker.store.printStorePath(drvPath));
     }
-    co_return done(BuildResult::AlreadyValid, assertPathValidity());
+    co_return doneSuccess(BuildResult::AlreadyValid, assertPathValidity());
 }
 
 std::optional<std::pair<Realisation, PathStatus>> DerivationGoal::checkPathValidity()
@@ -339,12 +337,27 @@ Realisation DerivationGoal::assertPathValidity()
     return checkResult->first;
 }
 
-Goal::Done
-DerivationGoal::done(BuildResult::Status status, std::optional<Realisation> builtOutput, std::optional<Error> ex)
+Goal::Done DerivationGoal::doneSuccess(BuildResult::Status status, Realisation builtOutput)
 {
     buildResult.status = status;
-    if (ex)
-        buildResult.errorMsg = fmt("%s", Uncolored(ex->info().msg));
+
+    assert(buildResult.success());
+
+    mcExpectedBuilds.reset();
+
+    buildResult.builtOutputs = {{wantedOutput, std::move(builtOutput)}};
+    if (status == BuildResult::Built)
+        worker.doneBuilds++;
+
+    worker.updateProgress();
+
+    return amDone(ecSuccess, std::nullopt);
+}
+
+Goal::Done DerivationGoal::doneFailure(BuildError ex)
+{
+    buildResult.status = ex.status;
+    buildResult.errorMsg = fmt("%s", Uncolored(ex.info().msg));
     if (buildResult.status == BuildResult::TimedOut)
         worker.timedOut = true;
     if (buildResult.status == BuildResult::PermanentFailure)
@@ -352,26 +365,12 @@ DerivationGoal::done(BuildResult::Status status, std::optional<Realisation> buil
 
     mcExpectedBuilds.reset();
 
-    if (buildResult.success()) {
-        assert(builtOutput);
-        buildResult.builtOutputs = {{wantedOutput, std::move(*builtOutput)}};
-        if (status == BuildResult::Built)
-            worker.doneBuilds++;
-    } else {
-        if (status != BuildResult::DependencyFailed)
-            worker.failedBuilds++;
-    }
+    if (ex.status != BuildResult::DependencyFailed)
+        worker.failedBuilds++;
 
     worker.updateProgress();
 
-    auto traceBuiltOutputsFile = getEnv("_NIX_TRACE_BUILT_OUTPUTS").value_or("");
-    if (traceBuiltOutputsFile != "") {
-        std::fstream fs;
-        fs.open(traceBuiltOutputsFile, std::fstream::out);
-        fs << worker.store.printStorePath(drvPath) << "\t" << buildResult.toString() << std::endl;
-    }
-
-    return amDone(buildResult.success() ? ecSuccess : ecFailed, std::move(ex));
+    return amDone(ecFailed, {std::move(ex)});
 }
 
 } // namespace nix

src/libstore/build/drv-output-substitution-goal.cc

@@ -4,6 +4,7 @@
 #include "nix/store/build/substitution-goal.hh"
 #include "nix/util/callback.hh"
 #include "nix/store/store-open.hh"
+#include "nix/store/globals.hh"
 
 namespace nix {
 

src/libstore/build/goal.cc

@@ -1,5 +1,6 @@
 #include "nix/store/build/goal.hh"
 #include "nix/store/build/worker.hh"
+#include "nix/store/globals.hh"
 
 namespace nix {
 

src/libstore/build/substitution-goal.cc

@@ -4,6 +4,8 @@
 #include "nix/store/nar-info.hh"
 #include "nix/util/finally.hh"
 #include "nix/util/signals.hh"
+#include "nix/store/globals.hh"
+
 #include <coroutine>
 
 namespace nix {

src/libstore/build/worker.cc

@@ -10,6 +10,7 @@
 #  include "nix/store/build/hook-instance.hh"
 #endif
 #include "nix/util/signals.hh"
+#include "nix/store/globals.hh"
 
 namespace nix {
 

src/libstore/builtins/fetchurl.cc

@@ -1,6 +1,7 @@
 #include "nix/store/builtins.hh"
 #include "nix/store/filetransfer.hh"
 #include "nix/store/store-api.hh"
+#include "nix/store/globals.hh"
 #include "nix/util/archive.hh"
 #include "nix/util/compression.hh"
 
@@ -36,7 +37,7 @@ static void builtinFetchurl(const BuiltinBuilderContext & ctx)
 
     auto fetch = [&](const std::string & url) {
         auto source = sinkToSource([&](Sink & sink) {
-            FileTransferRequest request(url);
+            FileTransferRequest request(ValidURL{url});
             request.decompress = false;
 
             auto decompressor = makeDecompressionSink(unpack && hasSuffix(mainUrl, ".xz") ? "xz" : "none", sink);

src/libstore/daemon.cc

@@ -16,6 +16,7 @@
 #include "nix/util/args.hh"
 #include "nix/util/git.hh"
 #include "nix/util/logging.hh"
+#include "nix/store/globals.hh"
 
 #ifndef _WIN32 // TODO need graceful async exit support on Windows?
 #  include "nix/util/monitor-fd.hh"

src/libstore/derivation-options.cc

@@ -5,6 +5,7 @@
 #include "nix/store/store-api.hh"
 #include "nix/util/types.hh"
 #include "nix/util/util.hh"
+#include "nix/store/globals.hh"
 
 #include <optional>
 #include <string>
@@ -255,6 +256,25 @@ DerivationOptions::fromStructuredAttrs(const StringMap & env, const StructuredAt
     };
 }
 
+std::map<std::string, StorePathSet>
+DerivationOptions::getParsedExportReferencesGraph(const StoreDirConfig & store) const
+{
+    std::map<std::string, StorePathSet> res;
+
+    for (auto & [fileName, ss] : exportReferencesGraph) {
+        StorePathSet storePaths;
+        for (auto & storePathS : ss) {
+            if (!store.isInStore(storePathS))
+                throw BuildError(
+                    BuildResult::InputRejected, "'exportReferencesGraph' contains a non-store path '%1%'", storePathS);
+            storePaths.insert(store.toStorePath(storePathS).first);
+        }
+        res.insert_or_assign(fileName, storePaths);
+    }
+
+    return res;
+}
+
 StringSet DerivationOptions::getRequiredSystemFeatures(const BasicDerivation & drv) const
 {
     // FIXME: cache this?

src/libstore/derivations.cc

@@ -9,7 +9,6 @@
 #include "nix/store/common-protocol-impl.hh"
 #include "nix/util/strings-inline.hh"
 #include "nix/util/json-utils.hh"
-#include "nix/store/async-path-writer.hh"
 
 #include <boost/container/small_vector.hpp>
 #include <nlohmann/json.hpp>
@@ -134,20 +133,6 @@ StorePath writeDerivation(Store & store, const Derivation & drv, RepairFlag repa
                                                });
 }
 
-StorePath writeDerivation(
-    Store & store, AsyncPathWriter & asyncPathWriter, const Derivation & drv, RepairFlag repair, bool readOnly)
-{
-    auto references = drv.inputSrcs;
-    for (auto & i : drv.inputDrvs.map)
-        references.insert(i.first);
-    return asyncPathWriter.addPath(
-        drv.unparse(store, false),
-        std::string(drv.name) + drvExtension,
-        references,
-        repair,
-        readOnly || settings.readOnlyMode);
-}
-
 namespace {
 /**
  * This mimics std::istream to some extent. We use this much smaller implementation
@@ -513,28 +498,33 @@ Derivation parseDerivation(
  */
 static void printString(std::string & res, std::string_view s)
 {
-    boost::container::small_vector<char, 64 * 1024> buffer;
-    buffer.reserve(s.size() * 2 + 2);
-    char * buf = buffer.data();
-    char * p = buf;
-    *p++ = '"';
-    for (auto c : s)
-        if (c == '\"' || c == '\\') {
-            *p++ = '\\';
-            *p++ = c;
-        } else if (c == '\n') {
-            *p++ = '\\';
-            *p++ = 'n';
-        } else if (c == '\r') {
-            *p++ = '\\';
-            *p++ = 'r';
-        } else if (c == '\t') {
-            *p++ = '\\';
-            *p++ = 't';
-        } else
-            *p++ = c;
-    *p++ = '"';
-    res.append(buf, p - buf);
+    res.reserve(res.size() + s.size() * 2 + 2);
+    res += '"';
+    static constexpr auto chunkSize = 1024;
+    std::array<char, 2 * chunkSize + 2> buffer;
+    while (!s.empty()) {
+        auto chunk = s.substr(0, /*n=*/chunkSize);
+        s.remove_prefix(chunk.size());
+        char * buf = buffer.data();
+        char * p = buf;
+        for (auto c : chunk)
+            if (c == '\"' || c == '\\') {
+                *p++ = '\\';
+                *p++ = c;
+            } else if (c == '\n') {
+                *p++ = '\\';
+                *p++ = 'n';
+            } else if (c == '\r') {
+                *p++ = '\\';
+                *p++ = 'r';
+            } else if (c == '\t') {
+                *p++ = '\\';
+                *p++ = 't';
+            } else
+                *p++ = c;
+        res.append(buf, p - buf);
+    }
+    res += '"';
 }
 
 static void printUnquotedString(std::string & res, std::string_view s)

src/libstore/filetransfer.cc

@@ -100,7 +100,7 @@ struct curlFileTransfer : public FileTransfer
                   lvlTalkative,
                   actFileTransfer,
                   fmt("%sing '%s'", request.verb(), request.uri),
-                  {request.uri},
+                  {request.uri.to_string()},
                   request.parentAct)
             , callback(std::move(callback))
             , finalSink([this](std::string_view data) {
@@ -121,7 +121,7 @@ struct curlFileTransfer : public FileTransfer
                     this->result.data.append(data);
             })
         {
-            result.urls.push_back(request.uri);
+            result.urls.push_back(request.uri.to_string());
 
             requestHeaders = curl_slist_append(requestHeaders, "Accept-Encoding: zstd, br, gzip, deflate, bzip2, xz");
             if (!request.expectedETag.empty())
@@ -350,7 +350,7 @@ struct curlFileTransfer : public FileTransfer
                 curl_easy_setopt(req, CURLOPT_DEBUGFUNCTION, TransferItem::debugCallback);
             }
 
-            curl_easy_setopt(req, CURLOPT_URL, request.uri.c_str());
+            curl_easy_setopt(req, CURLOPT_URL, request.uri.to_string().c_str());
             curl_easy_setopt(req, CURLOPT_FOLLOWLOCATION, 1L);
             curl_easy_setopt(req, CURLOPT_MAXREDIRS, 10);
             curl_easy_setopt(req, CURLOPT_NOSIGNAL, 1);
@@ -784,8 +784,8 @@ struct curlFileTransfer : public FileTransfer
 
     void enqueueItem(std::shared_ptr<TransferItem> item)
     {
-        if (item->request.data && !hasPrefix(item->request.uri, "http://") && !hasPrefix(item->request.uri, "https://"))
-            throw nix::Error("uploading to '%s' is not supported", item->request.uri);
+        if (item->request.data && item->request.uri.scheme() != "http" && item->request.uri.scheme() != "https")
+            throw nix::Error("uploading to '%s' is not supported", item->request.uri.to_string());
 
         {
             auto state(state_.lock());
@@ -801,11 +801,11 @@ struct curlFileTransfer : public FileTransfer
     void enqueueFileTransfer(const FileTransferRequest & request, Callback<FileTransferResult> callback) override
     {
         /* Ugly hack to support s3:// URIs. */
-        if (hasPrefix(request.uri, "s3://")) {
+        if (request.uri.scheme() == "s3") {
             // FIXME: do this on a worker thread
             try {
 #if NIX_WITH_S3_SUPPORT
-                auto parsed = ParsedS3URL::parse(request.uri);
+                auto parsed = ParsedS3URL::parse(request.uri.parsed());
 
                 std::string profile = parsed.profile.value_or("");
                 std::string region = parsed.region.value_or(Aws::Region::US_EAST_1);
@@ -815,15 +815,16 @@ struct curlFileTransfer : public FileTransfer
                 S3Helper s3Helper(profile, region, scheme, endpoint);
 
                 // FIXME: implement ETag
-                auto s3Res = s3Helper.getObject(parsed.bucket, parsed.key);
+                auto s3Res = s3Helper.getObject(parsed.bucket, encodeUrlPath(parsed.key));
                 FileTransferResult res;
                 if (!s3Res.data)
                     throw FileTransferError(NotFound, {}, "S3 object '%s' does not exist", request.uri);
                 res.data = std::move(*s3Res.data);
-                res.urls.push_back(request.uri);
+                res.urls.push_back(request.uri.to_string());
                 callback(std::move(res));
 #else
-                throw nix::Error("cannot download '%s' because Nix is not built with S3 support", request.uri);
+                throw nix::Error(
+                    "cannot download '%s' because Nix is not built with S3 support", request.uri.to_string());
 #endif
             } catch (...) {
                 callback.rethrow();

src/libstore/globals.cc

@@ -86,13 +86,22 @@ Settings::Settings()
     }
 
 #if (defined(__linux__) || defined(__FreeBSD__)) && defined(SANDBOX_SHELL)
-    sandboxPaths = tokenizeString<StringSet>("/bin/sh=" SANDBOX_SHELL);
+    sandboxPaths = {{"/bin/sh", {.source = SANDBOX_SHELL}}};
 #endif
 
     /* chroot-like behavior from Apple's sandbox */
 #ifdef __APPLE__
-    sandboxPaths = tokenizeString<StringSet>(
-        "/System/Library/Frameworks /System/Library/PrivateFrameworks /bin/sh /bin/bash /private/tmp /private/var/tmp /usr/lib");
+    for (PathView p : {
+             "/System/Library/Frameworks",
+             "/System/Library/PrivateFrameworks",
+             "/bin/sh",
+             "/bin/bash",
+             "/private/tmp",
+             "/private/var/tmp",
+             "/usr/lib",
+         }) {
+        sandboxPaths.get().insert_or_assign(std::string{p}, ChrootPath{.source = std::string{p}});
+    }
     allowedImpureHostPrefixes = tokenizeString<StringSet>("/System/Library /usr/lib /dev /bin/sh");
 #endif
 }
@@ -317,6 +326,42 @@ void BaseSetting<SandboxMode>::convertToArg(Args & args, const std::string & cat
     });
 }
 
+NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(ChrootPath, source, optional)
+
+template<>
+PathsInChroot BaseSetting<PathsInChroot>::parse(const std::string & str) const
+{
+    PathsInChroot pathsInChroot;
+    for (auto i : tokenizeString<StringSet>(str)) {
+        if (i.empty())
+            continue;
+        bool optional = false;
+        if (i[i.size() - 1] == '?') {
+            optional = true;
+            i.pop_back();
+        }
+        size_t p = i.find('=');
+        if (p == std::string::npos)
+            pathsInChroot[i] = {.source = i, .optional = optional};
+        else
+            pathsInChroot[i.substr(0, p)] = {.source = i.substr(p + 1), .optional = optional};
+    }
+    return pathsInChroot;
+}
+
+template<>
+std::string BaseSetting<PathsInChroot>::to_string() const
+{
+    std::vector<std::string> accum;
+    for (auto & [name, cp] : value) {
+        std::string s = name == cp.source ? name : name + "=" + cp.source;
+        if (cp.optional)
+            s += "?";
+        accum.push_back(std::move(s));
+    }
+    return concatStringsSep(" ", accum);
+}
+
 unsigned int MaxBuildJobsSetting::parse(const std::string & str) const
 {
     if (str == "auto")
@@ -329,6 +374,14 @@ unsigned int MaxBuildJobsSetting::parse(const std::string & str) const
     }
 }
 
+template<>
+void BaseSetting<PathsInChroot>::appendOrSet(PathsInChroot newValue, bool append)
+{
+    if (!append)
+        value.clear();
+    value.insert(std::make_move_iterator(newValue.begin()), std::make_move_iterator(newValue.end()));
+}
+
 static void preloadNSS()
 {
     /* builtin:fetchurl can trigger a DNS lookup, which with glibc can trigger a dynamic library load of

src/libstore/http-binary-cache-store.cc

@@ -27,7 +27,7 @@ HttpBinaryCacheStoreConfig::HttpBinaryCacheStoreConfig(
           + (!_cacheUri.empty() ? _cacheUri
                                 : throw UsageError("`%s` Store requires a non-empty authority in Store URL", scheme))))
 {
-    while (!cacheUri.path.empty() && cacheUri.path.back() == '/')
+    while (!cacheUri.path.empty() && cacheUri.path.back() == "")
         cacheUri.path.pop_back();
 }
 
@@ -37,7 +37,7 @@ StoreReference HttpBinaryCacheStoreConfig::getReference() const
         .variant =
             StoreReference::Specified{
                 .scheme = cacheUri.scheme,
-                .authority = (cacheUri.authority ? cacheUri.authority->to_string() : "") + cacheUri.path,
+                .authority = cacheUri.renderAuthorityAndPath(),
             },
         .params = cacheUri.query,
     };
@@ -154,22 +154,17 @@ protected:
 
     FileTransferRequest makeRequest(const std::string & path)
     {
-        /* FIXME path is not a path, but a full relative or absolute
+        /* Otherwise the last path fragment will get discarded. */
+        auto cacheUriWithTrailingSlash = config->cacheUri;
+        if (!cacheUriWithTrailingSlash.path.empty())
+            cacheUriWithTrailingSlash.path.push_back("");
+
+        /* path is not a path, but a full relative or absolute
            URL, e.g. we've seen in the wild NARINFO files have a URL
            field which is
            `nar/15f99rdaf26k39knmzry4xd0d97wp6yfpnfk1z9avakis7ipb9yg.nar?hash=zphkqn2wg8mnvbkixnl2aadkbn0rcnfj`
-           (note the query param) and that gets passed here.
-
-           What should actually happen is that we have two parsed URLs
-           (if we support relative URLs), and then we combined them with
-           a URL `operator/` which would be like
-           `std::filesystem::path`'s equivalent operator, which properly
-           combines the the URLs, whether the right is relative or
-           absolute. */
-        return FileTransferRequest(
-            hasPrefix(path, "https://") || hasPrefix(path, "http://") || hasPrefix(path, "file://")
-                ? path
-                : config->cacheUri.to_string() + "/" + path);
+           (note the query param) and that gets passed here. */
+        return FileTransferRequest(parseURLRelative(path, cacheUriWithTrailingSlash));
     }
 
     void getFile(const std::string & path, Sink & sink) override

src/libstore/include/nix/store/async-path-writer.hh

@@ -1,19 +0,0 @@
-#pragma once
-
-#include "nix/store/store-api.hh"
-
-namespace nix {
-
-struct AsyncPathWriter
-{
-    virtual StorePath addPath(
-        std::string contents, std::string name, StorePathSet references, RepairFlag repair, bool readOnly = false) = 0;
-
-    virtual void waitForPath(const StorePath & path) = 0;
-
-    virtual void waitForAllPaths() = 0;
-
-    static ref<AsyncPathWriter> make(ref<Store> store);
-};
-
-} // namespace nix

src/libstore/include/nix/store/build-result.hh

@@ -1,13 +1,13 @@
 #pragma once
 ///@file
 
-#include "nix/store/realisation.hh"
-#include "nix/store/derived-path.hh"
-
 #include <string>
 #include <chrono>
 #include <optional>
 
+#include "nix/store/derived-path.hh"
+#include "nix/store/realisation.hh"
+
 namespace nix {
 
 struct BuildResult
@@ -36,6 +36,10 @@ struct BuildResult
         NotDeterministic,
         ResolvesToAlreadyValid,
         NoSubstituters,
+        /// A certain type of `OutputRejected`. The protocols do not yet
+        /// know about this one, so change it back to `OutputRejected`
+        /// before serialization.
+        HashMismatch,
     } status = MiscFailure;
 
     /**
@@ -46,47 +50,6 @@ struct BuildResult
      */
     std::string errorMsg;
 
-    std::string toString() const
-    {
-        auto strStatus = [&]() {
-            switch (status) {
-            case Built:
-                return "Built";
-            case Substituted:
-                return "Substituted";
-            case AlreadyValid:
-                return "AlreadyValid";
-            case PermanentFailure:
-                return "PermanentFailure";
-            case InputRejected:
-                return "InputRejected";
-            case OutputRejected:
-                return "OutputRejected";
-            case TransientFailure:
-                return "TransientFailure";
-            case CachedFailure:
-                return "CachedFailure";
-            case TimedOut:
-                return "TimedOut";
-            case MiscFailure:
-                return "MiscFailure";
-            case DependencyFailed:
-                return "DependencyFailed";
-            case LogLimitExceeded:
-                return "LogLimitExceeded";
-            case NotDeterministic:
-                return "NotDeterministic";
-            case ResolvesToAlreadyValid:
-                return "ResolvesToAlreadyValid";
-            case NoSubstituters:
-                return "NoSubstituters";
-            default:
-                return "Unknown";
-            };
-        }();
-        return strStatus + ((errorMsg == "") ? "" : " : " + errorMsg);
-    }
-
     /**
      * How many times this build was performed.
      */
@@ -131,6 +94,26 @@ struct BuildResult
     }
 };
 
+/**
+ * denotes a permanent build failure
+ */
+struct BuildError : public Error
+{
+    BuildResult::Status status;
+
+    BuildError(BuildResult::Status status, BuildError && error)
+        : Error{std::move(error)}
+        , status{status}
+    {
+    }
+
+    BuildError(BuildResult::Status status, auto &&... args)
+        : Error{args...}
+        , status{status}
+    {
+    }
+};
+
 /**
  * A `BuildResult` together with its "primary key".
  */

src/libstore/include/nix/store/build/derivation-builder.hh

@@ -8,10 +8,44 @@
 #include "nix/store/parsed-derivations.hh"
 #include "nix/util/processes.hh"
 #include "nix/store/restricted-store.hh"
-#include "nix/store/user-lock.hh"
+#include "nix/store/build/derivation-env-desugar.hh"
 
 namespace nix {
 
+/**
+ * Denotes a build failure that stemmed from the builder exiting with a
+ * failing exist status.
+ */
+struct BuilderFailureError : BuildError
+{
+    int builderStatus;
+
+    std::string extraMsgAfter;
+
+    BuilderFailureError(BuildResult::Status status, int builderStatus, std::string extraMsgAfter)
+        : BuildError{
+            status,
+              /* No message for now, because the caller will make for
+                 us, with extra context */
+              "",
+          }
+        , builderStatus{std::move(builderStatus)}
+        , extraMsgAfter{std::move(extraMsgAfter)}
+    {
+    }
+};
+
+/**
+ * Stuff we need to pass to initChild().
+ */
+struct ChrootPath
+{
+    Path source;
+    bool optional = false;
+};
+
+typedef std::map<Path, ChrootPath> PathsInChroot; // maps target path to source path
+
 /**
  * Parameters by (mostly) `const` reference for `DerivationBuilder`.
  */
@@ -49,25 +83,21 @@ struct DerivationBuilderParams
 
     const BuildMode & buildMode;
 
-    DerivationBuilderParams(
-        const StorePath & drvPath,
-        const BuildMode & buildMode,
-        BuildResult & buildResult,
-        const Derivation & drv,
-        const DerivationOptions & drvOptions,
-        const StorePathSet & inputPaths,
-        std::map<std::string, InitialOutput> & initialOutputs)
-        : drvPath{drvPath}
-        , buildResult{buildResult}
-        , drv{drv}
-        , drvOptions{drvOptions}
-        , inputPaths{inputPaths}
-        , initialOutputs{initialOutputs}
-        , buildMode{buildMode}
-    {
-    }
+    /**
+     * Extra paths we want to be in the chroot, regardless of the
+     * derivation we are building.
+     */
+    PathsInChroot defaultPathsInChroot;
 
-    DerivationBuilderParams(DerivationBuilderParams &&) = default;
+    /**
+     * May be used to control various platform-specific functionality.
+     *
+     * For example, on Linux, the `kvm` system feature controls whether
+     * `/dev/kvm` should be exposed to the builder within the sandbox.
+     */
+    StringSet systemFeatures;
+
+    DesugaredEnv desugaredEnv;
 };
 
 /**
@@ -87,8 +117,6 @@ struct DerivationBuilderCallbacks
      */
     virtual void closeLogFile() = 0;
 
-    virtual void appendLogTailErrorMsg(std::string & msg) = 0;
-
     /**
      * Hook up `builderOut` to some mechanism to ingest the log
      *
@@ -100,11 +128,6 @@ struct DerivationBuilderCallbacks
      * @todo this should be reworked
      */
     virtual void childTerminated() = 0;
-
-    virtual void noteHashMismatch(void) = 0;
-    virtual void noteCheckMismatch(void) = 0;
-
-    virtual void markContentsGood(const StorePath & path) = 0;
 };
 
 /**
@@ -120,11 +143,6 @@ struct DerivationBuilderCallbacks
  */
 struct DerivationBuilder : RestrictionContext
 {
-    /**
-     * The process ID of the builder.
-     */
-    Pid pid;
-
     DerivationBuilder() = default;
     virtual ~DerivationBuilder() = default;
 
@@ -157,28 +175,23 @@ struct DerivationBuilder : RestrictionContext
      * processing. A status code and exception are returned, providing
      * more information. The second case indicates success, and
      * realisations for each output of the derivation are returned.
+     *
+     * @throws BuildError
      */
-    virtual std::variant<std::pair<BuildResult::Status, Error>, SingleDrvOutputs> unprepareBuild() = 0;
+    virtual SingleDrvOutputs unprepareBuild() = 0;
 
     /**
-     * Stop the in-process nix daemon thread.
-     * @see startDaemon
+     * Forcibly kill the child process, if any.
+     *
+     * @returns whether the child was still alive and needed to be
+     * killed.
      */
-    virtual void stopDaemon() = 0;
-
-    /**
-     * Delete the temporary directory, if we have one.
-     */
-    virtual void deleteTmpDir(bool force) = 0;
-
-    /**
-     * Kill any processes running under the build user UID or in the
-     * cgroup of the build.
-     */
-    virtual void killSandbox(bool getStats) = 0;
+    virtual bool killChild() = 0;
 };
 
+#ifndef _WIN32 // TODO enable `DerivationBuilder` on Windows
 std::unique_ptr<DerivationBuilder> makeDerivationBuilder(
     LocalStore & store, std::unique_ptr<DerivationBuilderCallbacks> miscMethods, DerivationBuilderParams params);
+#endif
 
 } // namespace nix

src/libstore/include/nix/store/build/derivation-building-goal.hh

@@ -14,6 +14,7 @@ namespace nix {
 
 using std::map;
 
+struct BuilderFailureError;
 #ifndef _WIN32 // TODO enable build hook on Windows
 struct HookInstance;
 struct DerivationBuilder;
@@ -170,9 +171,11 @@ struct DerivationBuildingGoal : public Goal
 
     void started();
 
-    Done done(BuildResult::Status status, SingleDrvOutputs builtOutputs = {}, std::optional<Error> ex = {});
+    Done doneSuccess(BuildResult::Status status, SingleDrvOutputs builtOutputs);
 
-    void appendLogTailErrorMsg(std::string & msg);
+    Done doneFailure(BuildError ex);
+
+    BuildError fixupBuilderFailureErrorMessage(BuilderFailureError msg);
 
     JobCategory jobCategory() const override
     {

src/libstore/include/nix/store/build/derivation-building-misc.hh

@@ -49,9 +49,6 @@ struct InitialOutput
     std::optional<InitialOutputStatus> known;
 };
 
-void runPostBuildHook(
-    const StoreDirConfig & store, Logger & logger, const StorePath & drvPath, const StorePathSet & outputPaths);
-
 /**
  * Format the known outputs of a derivation for use in error messages.
  */

src/libstore/include/nix/store/build/derivation-env-desugar.hh

@@ -0,0 +1,83 @@
+#pragma once
+///@file
+
+#include "nix/util/types.hh"
+#include "nix/store/path.hh"
+
+namespace nix {
+
+class Store;
+struct Derivation;
+struct DerivationOptions;
+
+/**
+ * Derivations claim to "just" specify their environment variables, but
+ * actually do a number of different features, such as "structured
+ * attrs", "pass as file", and "export references graph", things are
+ * more complicated then they appear.
+ *
+ * The good news is that we can simplify all that to the following view,
+ * where environment variables and extra files are specified exactly,
+ * with no special cases.
+ *
+ * Because we have `DesugaredEnv`, `DerivationBuilder` doesn't need to
+ * know about any of those above features, and their special case.
+ */
+struct DesugaredEnv
+{
+    struct EnvEntry
+    {
+        /**
+         * Whether to prepend the (inside via) path to the sandbox build
+         * directory to `value`. This is useful for when the env var
+         * should point to a file visible to the builder.
+         */
+        bool prependBuildDirectory = false;
+
+        /**
+         * String value of env var, or contents of the file.
+         */
+        std::string value;
+    };
+
+    /**
+     * The final environment variables to set.
+     */
+    std::map<std::string, EnvEntry, std::less<>> variables;
+
+    /**
+     * Extra file to be placed in the build directory.
+     *
+     * @note `EnvEntry::prependBuildDirectory` can be used to refer to
+     * those files without knowing what the build directory is.
+     */
+    StringMap extraFiles;
+
+    /**
+     * A common case is to define an environment variable that points to
+     * a file, which contains some contents.
+     *
+     * In base:
+     * ```
+     * export VAR=FILE_NAME
+     * echo CONTENTS >FILE_NAME
+     * ```
+     *
+     * This function assists in doing both parts, so the file name is
+     * kept in sync.
+     */
+    std::string & atFileEnvPair(std::string_view name, std::string fileName);
+
+    /**
+     * Given a (resolved) derivation, its options, and the closure of
+     * its inputs (which we can get since the derivation is resolved),
+     * desugar the environment to create a `DesguaredEnv`.
+     *
+     * @todo drvOptions will go away as a separate argument when it is
+     * just part of `Derivation`.
+     */
+    static DesugaredEnv create(
+        Store & store, const Derivation & drv, const DerivationOptions & drvOptions, const StorePathSet & inputPaths);
+};
+
+} // namespace nix

src/libstore/include/nix/store/build/derivation-goal.hh

@@ -14,9 +14,6 @@ namespace nix {
 
 using std::map;
 
-/** Used internally */
-void runPostBuildHook(Store & store, Logger & logger, const StorePath & drvPath, const StorePathSet & outputPaths);
-
 /**
  * A goal for realising a single output of a derivation. Various sorts of
  * fetching (which will be done by other goal types) is tried, and if none of
@@ -102,13 +99,9 @@ private:
 
     Co repairClosure();
 
-    /**
-     * @param builtOutput Must be set if `status` is successful.
-     */
-    Done done(
-        BuildResult::Status status,
-        std::optional<Realisation> builtOutput = std::nullopt,
-        std::optional<Error> ex = {});
+    Done doneSuccess(BuildResult::Status status, Realisation builtOutput);
+
+    Done doneFailure(BuildError ex);
 };
 
 } // namespace nix

src/libstore/include/nix/store/derivation-options.hh

@@ -8,10 +8,12 @@
 
 #include "nix/util/types.hh"
 #include "nix/util/json-impls.hh"
+#include "nix/store/path.hh"
 
 namespace nix {
 
 class Store;
+struct StoreDirConfig;
 struct BasicDerivation;
 struct StructuredAttrs;
 
@@ -116,6 +118,22 @@ struct DerivationOptions
      */
     std::map<std::string, StringSet> exportReferencesGraph;
 
+    /**
+     * Once a derivations is resolved, the strings in in
+     * `exportReferencesGraph` should all be store paths (with possible
+     * suffix paths, but those are discarded).
+     *
+     * @return The parsed path set for for each key in the map.
+     *
+     * @todo Ideally, `exportReferencesGraph` would just store
+     * `StorePath`s for this, but we can't just do that, because for CA
+     * derivations they is actually in general `DerivedPath`s (via
+     * placeholder strings) until the derivation is resolved and exact
+     * inputs store paths are known. We can use better types for that
+     * too, but that is a longer project.
+     */
+    std::map<std::string, StorePathSet> getParsedExportReferencesGraph(const StoreDirConfig & store) const;
+
     /**
      * env: __sandboxProfile
      *

src/libstore/include/nix/store/derivations.hh

@@ -17,7 +17,6 @@
 namespace nix {
 
 struct StoreDirConfig;
-struct AsyncPathWriter;
 
 /* Abstract syntax of derivations. */
 
@@ -413,16 +412,6 @@ class Store;
  */
 StorePath writeDerivation(Store & store, const Derivation & drv, RepairFlag repair = NoRepair, bool readOnly = false);
 
-/**
- * Asynchronously write a derivation to the Nix store, and return its path.
- */
-StorePath writeDerivation(
-    Store & store,
-    AsyncPathWriter & asyncPathWriter,
-    const Derivation & drv,
-    RepairFlag repair = NoRepair,
-    bool readOnly = false);
-
 /**
  * Read a derivation from a file.
  */

src/libstore/include/nix/store/filetransfer.hh

@@ -9,6 +9,7 @@
 #include "nix/util/ref.hh"
 #include "nix/util/configuration.hh"
 #include "nix/util/serialise.hh"
+#include "nix/util/url.hh"
 
 namespace nix {
 
@@ -70,7 +71,7 @@ extern const unsigned int RETRY_TIME_MS_DEFAULT;
 
 struct FileTransferRequest
 {
-    std::string uri;
+    ValidURL uri;
     Headers headers;
     std::string expectedETag;
     bool verifyTLS = true;
@@ -84,8 +85,8 @@ struct FileTransferRequest
     std::string mimeType;
     std::function<void(std::string_view data)> dataCallback;
 
-    FileTransferRequest(std::string_view uri)
-        : uri(uri)
+    FileTransferRequest(ValidURL uri)
+        : uri(std::move(uri))
         , parentAct(getCurActivity())
     {
     }
@@ -111,6 +112,9 @@ struct FileTransferResult
 
     /**
      * All URLs visited in the redirect chain.
+     *
+     * @note Intentionally strings and not `ParsedURL`s so we faithfully
+     * return what cURL gave us.
      */
     std::vector<std::string> urls;
 

src/libstore/include/nix/store/globals.hh

@@ -11,6 +11,7 @@
 #include "nix/util/environment-variables.hh"
 #include "nix/util/experimental-features.hh"
 #include "nix/util/users.hh"
+#include "nix/store/build/derivation-builder.hh"
 
 #include "nix/store/config.hh"
 
@@ -23,6 +24,20 @@ SandboxMode BaseSetting<SandboxMode>::parse(const std::string & str) const;
 template<>
 std::string BaseSetting<SandboxMode>::to_string() const;
 
+template<>
+PathsInChroot BaseSetting<PathsInChroot>::parse(const std::string & str) const;
+template<>
+std::string BaseSetting<PathsInChroot>::to_string() const;
+
+template<>
+struct BaseSetting<PathsInChroot>::trait
+{
+    static constexpr bool appendable = true;
+};
+
+template<>
+void BaseSetting<PathsInChroot>::appendOrSet(PathsInChroot newValue, bool append);
+
 struct MaxBuildJobsSetting : public BaseSetting<unsigned int>
 {
     MaxBuildJobsSetting(
@@ -697,7 +712,7 @@ public:
         )",
         {"build-use-chroot", "build-use-sandbox"}};
 
-    Setting<PathSet> sandboxPaths{
+    Setting<PathsInChroot> sandboxPaths{
         this,
         {},
         "sandbox-paths",

src/libstore/include/nix/store/http-binary-cache-store.hh

@@ -1,3 +1,6 @@
+#pragma once
+///@file
+
 #include "nix/util/url.hh"
 #include "nix/store/binary-cache-store.hh"
 

src/libstore/include/nix/store/local-fs-store.hh

@@ -22,15 +22,31 @@ struct LocalFSStoreConfig : virtual StoreConfig
 
     OptionalPathSetting rootDir{this, std::nullopt, "root", "Directory prefixed to all other paths."};
 
+private:
+
+    /**
+     * An indirection so that we don't need to refer to global settings
+     * in headers.
+     */
+    static Path getDefaultStateDir();
+
+    /**
+     * An indirection so that we don't need to refer to global settings
+     * in headers.
+     */
+    static Path getDefaultLogDir();
+
+public:
+
     PathSetting stateDir{
         this,
-        rootDir.get() ? *rootDir.get() + "/nix/var/nix" : settings.nixStateDir,
+        rootDir.get() ? *rootDir.get() + "/nix/var/nix" : getDefaultStateDir(),
         "state",
         "Directory where Nix stores state."};
 
     PathSetting logDir{
         this,
-        rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : settings.nixLogDir,
+        rootDir.get() ? *rootDir.get() + "/nix/var/log/nix" : getDefaultLogDir(),
         "log",
         "directory where Nix stores log files."};
 

src/libstore/include/nix/store/local-store.hh

@@ -74,9 +74,19 @@ struct LocalStoreConfig : std::enable_shared_from_this<LocalStoreConfig>,
 
     LocalStoreConfig(std::string_view scheme, std::string_view authority, const Params & params);
 
+private:
+
+    /**
+     * An indirection so that we don't need to refer to global settings
+     * in headers.
+     */
+    bool getDefaultRequireSigs();
+
+public:
+
     Setting<bool> requireSigs{
         this,
-        settings.requireSigs,
+        getDefaultRequireSigs(),
         "require-sigs",
         "Whether store paths copied into this store should have a trusted signature."};
 

src/libstore/include/nix/store/meson.build

@@ -10,11 +10,12 @@ config_pub_h = configure_file(
 )
 
 headers = [ config_pub_h ] + files(
-  'async-path-writer.hh',
   'binary-cache-store.hh',
   'build-result.hh',
+  'build/derivation-builder.hh',
   'build/derivation-building-goal.hh',
   'build/derivation-building-misc.hh',
+  'build/derivation-env-desugar.hh',
   'build/derivation-goal.hh',
   'build/derivation-trampoline-goal.hh',
   'build/drv-output-substitution-goal.hh',

src/libstore/include/nix/store/restricted-store.hh

@@ -1,10 +1,13 @@
 #pragma once
 ///@file
 
-#include "nix/store/local-store.hh"
+#include "nix/store/store-api.hh"
 
 namespace nix {
 
+class LocalStore;
+struct LocalStoreConfig;
+
 /**
  * A restricted store has a pointer to one of these, which manages the
  * restrictions that are in place.
@@ -55,6 +58,6 @@ struct RestrictionContext
 /**
  * Create a shared pointer to a restricted store.
  */
-ref<Store> makeRestrictedStore(ref<LocalStore::Config> config, ref<LocalStore> next, RestrictionContext & context);
+ref<Store> makeRestrictedStore(ref<LocalStoreConfig> config, ref<LocalStore> next, RestrictionContext & context);
 
 } // namespace nix

src/libstore/include/nix/store/s3.hh

@@ -54,7 +54,12 @@ struct S3Helper
 struct ParsedS3URL
 {
     std::string bucket;
-    std::string key;
+    /**
+     * @see ParsedURL::path. This is a vector for the same reason.
+     * Unlike ParsedURL::path this doesn't include the leading empty segment,
+     * since the bucket name is necessary.
+     */
+    std::vector<std::string> key;
     std::optional<std::string> profile;
     std::optional<std::string> region;
     std::optional<std::string> scheme;
@@ -74,7 +79,13 @@ struct ParsedS3URL
             endpoint);
     }
 
-    static ParsedS3URL parse(std::string_view uri);
+    static ParsedS3URL parse(const ParsedURL & uri);
+
+    /**
+     * Convert this ParsedS3URL to HTTPS ParsedURL for use with curl's AWS SigV4 authentication
+     */
+    ParsedURL toHttpsUrl() const;
+
     auto operator<=>(const ParsedS3URL & other) const = default;
 };
 

src/libstore/include/nix/store/sqlite.hh

@@ -1,6 +1,7 @@
 #pragma once
 ///@file
 
+#include <filesystem>
 #include <functional>
 #include <string>
 
@@ -41,7 +42,7 @@ struct SQLite
 
     SQLite() {}
 
-    SQLite(const Path & path, SQLiteOpenMode mode = SQLiteOpenMode::Normal);
+    SQLite(const std::filesystem::path & path, SQLiteOpenMode mode = SQLiteOpenMode::Normal);
     SQLite(const SQLite & from) = delete;
     SQLite & operator=(const SQLite & from) = delete;
 

src/libstore/include/nix/store/store-api.hh

@@ -8,7 +8,6 @@
 #include "nix/util/serialise.hh"
 #include "nix/util/lru-cache.hh"
 #include "nix/util/sync.hh"
-#include "nix/store/globals.hh"
 #include "nix/util/configuration.hh"
 #include "nix/store/path-info.hh"
 #include "nix/util/repair-flag.hh"
@@ -25,11 +24,6 @@
 
 namespace nix {
 
-MakeError(SubstError, Error);
-/**
- * denotes a permanent build failure
- */
-MakeError(BuildError, Error);
 MakeError(InvalidPath, Error);
 MakeError(Unsupported, Error);
 MakeError(SubstituteGone, Error);
@@ -89,9 +83,19 @@ struct StoreConfigBase : Config
 {
     using Config::Config;
 
+private:
+
+    /**
+     * An indirection so that we don't need to refer to global settings
+     * in headers.
+     */
+    static Path getDefaultNixStoreDir();
+
+public:
+
     const PathSetting storeDir_{
         this,
-        settings.nixStore,
+        getDefaultNixStoreDir(),
         "store",
         R"(
           Logical location of the Nix store, usually

src/libstore/include/nix/store/store-dir-config.hh

@@ -3,7 +3,6 @@
 #include "nix/store/path.hh"
 #include "nix/util/hash.hh"
 #include "nix/store/content-address.hh"
-#include "nix/store/globals.hh"
 #include "nix/util/configuration.hh"
 
 #include <map>

src/libstore/include/nix/store/store-open.hh

@@ -30,9 +30,12 @@ ref<Store> openStore(StoreReference && storeURI);
  * Opens the store at `uri`, where `uri` is in the format expected by
  * `StoreReference::parse`
  */
-ref<Store> openStore(
-    const std::string & uri = settings.storeUri.get(),
-    const StoreReference::Params & extraParams = StoreReference::Params());
+ref<Store> openStore(const std::string & uri, const StoreReference::Params & extraParams = StoreReference::Params());
+
+/**
+ * Short-hand which opens the default store, according to global settings
+ */
+ref<Store> openStore();
 
 /**
  * @return the default substituter stores, defined by the

src/libstore/include/nix/store/store-reference.hh

@@ -77,12 +77,22 @@ struct StoreReference
      */
     std::string render(bool withParams = true) const;
 
+    std::string to_string() const
+    {
+        return render();
+    }
+
     /**
      * Parse a URI into a store reference.
      */
     static StoreReference parse(const std::string & uri, const Params & extraParams = Params{});
 };
 
+static inline std::ostream & operator<<(std::ostream & os, const StoreReference & ref)
+{
+    return os << ref.render();
+}
+
 /**
  * Split URI into protocol+hierarchy part and its parameter set.
  */