Forward the experimental features to the nix repl subprocesses

Pass the current experimental features using `NIX_CONFIG` to the various Nix subprocesses that `nix repl` invokes. This is quite a hack, but having `nix repl` call Nix with a subprocess is a hack already, so I guess that’s fine.
Merge pull request #2667 from matthewbauer/cross-jobs
2021-06-29 14:52:46 +02:00 · 2021-06-29 11:39:49 +02:00 · 2021-06-28 23:02:53 +02:00 · 2021-06-28 15:08:17 -05:00 · 2021-06-28 19:33:55 +02:00 · 2021-06-28 18:42:44 +02:00
53 changed files with 979 additions and 284 deletions
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -20,8 +20,7 @@ jobs:
        name: '${{ env.CACHIX_NAME }}'
        signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
        authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
-    #- run: nix flake check
-    - run: nix-build -A checks.$(if [[ `uname` = Linux ]]; then echo x86_64-linux; else echo x86_64-darwin; fi)
+    - run: nix-build -A checks.$(nix-instantiate --eval -E '(builtins.currentSystem)')
  check_cachix:
    name: Cachix secret present for installer tests
    runs-on: ubuntu-latest
--- a/configure.ac
+++ b/configure.ac
@@ -65,7 +65,7 @@ AC_SYS_LARGEFILE
 AC_STRUCT_DIRENT_D_TYPE
 if test "$sys_name" = sunos; then
    # Solaris requires -lsocket -lnsl for network functions
-    LIBS="-lsocket -lnsl $LIBS"
+    LDFLAGS="-lsocket -lnsl $LDFLAGS"
 fi


@@ -150,7 +150,7 @@ int main() {
 }]])], GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC=no, GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC=yes)
 AC_MSG_RESULT($GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC)
 if test "x$GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC" = xyes; then
-    LIBS="-latomic $LIBS"
+    LDFLAGS="-latomic $LDFLAGS"
 fi

 PKG_PROG_PKG_CONFIG
@@ -231,7 +231,7 @@ AC_SUBST(HAVE_SECCOMP, [$have_seccomp])
 # Look for aws-cpp-sdk-s3.
 AC_LANG_PUSH(C++)
 AC_CHECK_HEADERS([aws/s3/S3Client.h],
-  [AC_DEFINE([ENABLE_S3], [1], [Whether to enable S3 support via aws-sdk-cpp.]) enable_s3=1], 
+  [AC_DEFINE([ENABLE_S3], [1], [Whether to enable S3 support via aws-sdk-cpp.]) enable_s3=1],
  [AC_DEFINE([ENABLE_S3], [0], [Whether to enable S3 support via aws-sdk-cpp.]) enable_s3=])
 AC_SUBST(ENABLE_S3, [$enable_s3])
 AC_LANG_POP(C++)
--- a/doc/manual/src/command-ref/env-common.md
+++ b/doc/manual/src/command-ref/env-common.md
@@ -10,35 +10,39 @@ Most Nix commands interpret the following environment variables:
    A colon-separated list of directories used to look up Nix
    expressions enclosed in angle brackets (i.e., `<path>`). For
    instance, the value
-    
+
        /home/eelco/Dev:/etc/nixos
-    
+
    will cause Nix to look for paths relative to `/home/eelco/Dev` and
    `/etc/nixos`, in this order. It is also possible to match paths
    against a prefix. For example, the value
-    
+
        nixpkgs=/home/eelco/Dev/nixpkgs-branch:/etc/nixos
-    
+
    will cause Nix to search for `<nixpkgs/path>` in
    `/home/eelco/Dev/nixpkgs-branch/path` and `/etc/nixos/nixpkgs/path`.
-    
+
    If a path in the Nix search path starts with `http://` or
    `https://`, it is interpreted as the URL of a tarball that will be
    downloaded and unpacked to a temporary location. The tarball must
    consist of a single top-level directory. For example, setting
    `NIX_PATH` to
-    
-        nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-15.09.tar.gz
-    
-    tells Nix to download the latest revision in the Nixpkgs/NixOS 15.09
-    channel.
-    
-    A following shorthand can be used to refer to the official channels:
-    
-        nixpkgs=channel:nixos-15.09
-    
-    The search path can be extended using the `-I` option, which takes
-    precedence over `NIX_PATH`.
+
+        nixpkgs=https://github.com/NixOS/nixpkgs/archive/master.tar.gz
+
+    tells Nix to download and use the current contents of the
+    `master` branch in the `nixpkgs` repository.
+
+    The URLs of the tarballs from the official nixos.org channels (see
+    [the manual for `nix-channel`](nix-channel.md)) can be abbreviated
+    as `channel:<channel-name>`.  For instance, the following two
+    values of `NIX_PATH` are equivalent:
+
+        nixpkgs=channel:nixos-21.05
+        nixpkgs=https://nixos.org/channels/nixos-21.05/nixexprs.tar.xz
+
+    The Nix search path can also be extended using the `-I` option to
+    many Nix commands, which takes precedence over `NIX_PATH`.

  - `NIX_IGNORE_SYMLINK_STORE`\
    Normally, the Nix store directory (typically `/nix/store`) is not
@@ -50,7 +54,7 @@ Most Nix commands interpret the following environment variables:
    builds are deployed to machines where `/nix/store` resolves
    differently. If you are sure that you’re not going to do that, you
    can set `NIX_IGNORE_SYMLINK_STORE` to `1`.
-    
+
    Note that if you’re symlinking the Nix store so that you can put it
    on another file system than the root file system, on Linux you’re
    better off using `bind` mount points, e.g.,
@@ -59,7 +63,7 @@ Most Nix commands interpret the following environment variables:
    $ mkdir /nix
    $ mount -o bind /mnt/otherdisk/nix /nix
    ```
-    
+
    Consult the mount 8 manual page for details.

  - `NIX_STORE_DIR`\
--- a/flake.lock
+++ b/flake.lock
@@ -19,11 +19,11 @@
    },
    "nixpkgs": {
      "locked": {
-        "lastModified": 1622593737,
-        "narHash": "sha256-9loxFJg85AbzJrSkU4pE/divZ1+zOxDy2FSjlrufCB8=",
+        "lastModified": 1624862269,
+        "narHash": "sha256-JFcsh2+7QtfKdJFoPibLFPLgIW6Ycnv8Bts9a7RYme0=",
        "owner": "NixOS",
        "repo": "nixpkgs",
-        "rev": "bb8a5e54845012ed1375ffd5f317d2fdf434b20e",
+        "rev": "f77036342e2b690c61c97202bf48f2ce13acc022",
        "type": "github"
      },
      "original": {
--- a/flake.nix
+++ b/flake.nix
@@ -20,6 +20,8 @@
      linuxSystems = linux64BitSystems ++ [ "i686-linux" ];
      systems = linuxSystems ++ [ "x86_64-darwin" "aarch64-darwin" ];

+      crossSystems = [ "armv6l-linux" "armv7l-linux" ];
+
      forAllSystems = f: nixpkgs.lib.genAttrs systems (system: f system);

      # Memoize nixpkgs for different platforms for efficiency.
@@ -79,7 +81,7 @@
            buildPackages.mercurial
            buildPackages.jq
          ]
-          ++ lib.optionals stdenv.isLinux [(pkgs.util-linuxMinimal or pkgs.utillinuxMinimal)];
+          ++ lib.optionals stdenv.hostPlatform.isLinux [(buildPackages.util-linuxMinimal or buildPackages.utillinuxMinimal)];

        buildDeps =
          [ curl
@@ -93,7 +95,7 @@
          ]
          ++ lib.optionals stdenv.isLinux [libseccomp]
          ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium
-          ++ lib.optional stdenv.isx86_64 libcpuid;
+          ++ lib.optional stdenv.hostPlatform.isx86_64 libcpuid;

        awsDeps = lib.optional (stdenv.isLinux || stdenv.isDarwin)
          (aws-sdk-cpp.override {
@@ -133,10 +135,11 @@

            substitute ${./scripts/install.in} $out/install \
              ${pkgs.lib.concatMapStrings
-                (system:
-                  '' \
-                  --replace '@tarballHash_${system}@' $(nix --experimental-features nix-command hash-file --base16 --type sha256 ${self.hydraJobs.binaryTarball.${system}}/*.tar.xz) \
-                  --replace '@tarballPath_${system}@' $(tarballPath ${self.hydraJobs.binaryTarball.${system}}/*.tar.xz) \
+                (system: let
+                    tarball = if builtins.elem system crossSystems then self.hydraJobs.binaryTarballCross.x86_64-linux.${system} else self.hydraJobs.binaryTarball.${system};
+                  in '' \
+                  --replace '@tarballHash_${system}@' $(nix --experimental-features nix-command hash-file --base16 --type sha256 ${tarball}/*.tar.xz) \
+                  --replace '@tarballPath_${system}@' $(tarballPath ${tarball}/*.tar.xz) \
                  ''
                )
                systems
@@ -174,6 +177,77 @@

      };

+      binaryTarball = buildPackages: nix: pkgs: let
+        inherit (pkgs) cacert;
+        installerClosureInfo = buildPackages.closureInfo { rootPaths = [ nix cacert ]; };
+      in
+
+      buildPackages.runCommand "nix-binary-tarball-${version}"
+        { #nativeBuildInputs = lib.optional (system != "aarch64-linux") shellcheck;
+          meta.description = "Distribution-independent Nix bootstrap binaries for ${pkgs.system}";
+        }
+        ''
+          cp ${installerClosureInfo}/registration $TMPDIR/reginfo
+          cp ${./scripts/create-darwin-volume.sh} $TMPDIR/create-darwin-volume.sh
+          substitute ${./scripts/install-nix-from-closure.sh} $TMPDIR/install \
+            --subst-var-by nix ${nix} \
+            --subst-var-by cacert ${cacert}
+
+          substitute ${./scripts/install-darwin-multi-user.sh} $TMPDIR/install-darwin-multi-user.sh \
+            --subst-var-by nix ${nix} \
+            --subst-var-by cacert ${cacert}
+          substitute ${./scripts/install-systemd-multi-user.sh} $TMPDIR/install-systemd-multi-user.sh \
+            --subst-var-by nix ${nix} \
+            --subst-var-by cacert ${cacert}
+          substitute ${./scripts/install-multi-user.sh} $TMPDIR/install-multi-user \
+            --subst-var-by nix ${nix} \
+            --subst-var-by cacert ${cacert}
+
+          if type -p shellcheck; then
+            # SC1090: Don't worry about not being able to find
+            #         $nix/etc/profile.d/nix.sh
+            shellcheck --exclude SC1090 $TMPDIR/install
+            shellcheck $TMPDIR/create-darwin-volume.sh
+            shellcheck $TMPDIR/install-darwin-multi-user.sh
+            shellcheck $TMPDIR/install-systemd-multi-user.sh
+
+            # SC1091: Don't panic about not being able to source
+            #         /etc/profile
+            # SC2002: Ignore "useless cat" "error", when loading
+            #         .reginfo, as the cat is a much cleaner
+            #         implementation, even though it is "useless"
+            # SC2116: Allow ROOT_HOME=$(echo ~root) for resolving
+            #         root's home directory
+            shellcheck --external-sources \
+              --exclude SC1091,SC2002,SC2116 $TMPDIR/install-multi-user
+          fi
+
+          chmod +x $TMPDIR/install
+          chmod +x $TMPDIR/create-darwin-volume.sh
+          chmod +x $TMPDIR/install-darwin-multi-user.sh
+          chmod +x $TMPDIR/install-systemd-multi-user.sh
+          chmod +x $TMPDIR/install-multi-user
+          dir=nix-${version}-${pkgs.system}
+          fn=$out/$dir.tar.xz
+          mkdir -p $out/nix-support
+          echo "file binary-dist $fn" >> $out/nix-support/hydra-build-products
+          tar cvfJ $fn \
+            --owner=0 --group=0 --mode=u+rw,uga+r \
+            --absolute-names \
+            --hard-dereference \
+            --transform "s,$TMPDIR/install,$dir/install," \
+            --transform "s,$TMPDIR/create-darwin-volume.sh,$dir/create-darwin-volume.sh," \
+            --transform "s,$TMPDIR/reginfo,$dir/.reginfo," \
+            --transform "s,$NIX_STORE,$dir/store,S" \
+            $TMPDIR/install \
+            $TMPDIR/create-darwin-volume.sh \
+            $TMPDIR/install-darwin-multi-user.sh \
+            $TMPDIR/install-systemd-multi-user.sh \
+            $TMPDIR/install-multi-user \
+            $TMPDIR/reginfo \
+            $(cat ${installerClosureInfo}/store-paths)
+        '';
+
    in {

      # A Nixpkgs overlay that overrides the 'nix' and
@@ -256,7 +330,8 @@
                boost
                nlohmann_json
              ]
-              ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium;
+              ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium
+              ++ lib.optional stdenv.isDarwin darwin.apple_sdk.frameworks.Security;

            configureFlags = ''
              --with-dbi=${perlPackages.DBI}/${pkgs.perl.libPrefix}
@@ -284,7 +359,7 @@

          outputs = [ "out" "bin" "dev" ];

-          nativeBuildInputs = [ which ];
+          nativeBuildInputs = [ buildPackages.which ];

          configurePhase = ''
              ${if (stdenv.isDarwin && stdenv.isAarch64) then "echo \"HAVE_SANDBOX_INIT=false\" > configure.local" else ""}
@@ -303,92 +378,33 @@

        buildStatic = nixpkgs.lib.genAttrs linux64BitSystems (system: self.packages.${system}.nix-static);

+        buildCross = nixpkgs.lib.genAttrs crossSystems (crossSystem:
+          nixpkgs.lib.genAttrs linux64BitSystems (system: self.packages.${system}."nix-${crossSystem}"));
+
        # Perl bindings for various platforms.
        perlBindings = nixpkgs.lib.genAttrs systems (system: self.packages.${system}.nix.perl-bindings);

        # Binary tarball for various platforms, containing a Nix store
        # with the closure of 'nix' package, and the second half of
        # the installation script.
-        binaryTarball = nixpkgs.lib.genAttrs systems (system:
+        binaryTarball = nixpkgs.lib.genAttrs systems (system: binaryTarball nixpkgsFor.${system} nixpkgsFor.${system}.nix nixpkgsFor.${system});

-          with nixpkgsFor.${system};
-
-          let
-            installerClosureInfo = closureInfo { rootPaths = [ nix cacert ]; };
-          in
-
-          runCommand "nix-binary-tarball-${version}"
-            { #nativeBuildInputs = lib.optional (system != "aarch64-linux") shellcheck;
-              meta.description = "Distribution-independent Nix bootstrap binaries for ${system}";
-            }
-            ''
-              cp ${installerClosureInfo}/registration $TMPDIR/reginfo
-              cp ${./scripts/create-darwin-volume.sh} $TMPDIR/create-darwin-volume.sh
-              substitute ${./scripts/install-nix-from-closure.sh} $TMPDIR/install \
-                --subst-var-by nix ${nix} \
-                --subst-var-by cacert ${cacert}
-
-              substitute ${./scripts/install-darwin-multi-user.sh} $TMPDIR/install-darwin-multi-user.sh \
-                --subst-var-by nix ${nix} \
-                --subst-var-by cacert ${cacert}
-              substitute ${./scripts/install-systemd-multi-user.sh} $TMPDIR/install-systemd-multi-user.sh \
-                --subst-var-by nix ${nix} \
-                --subst-var-by cacert ${cacert}
-              substitute ${./scripts/install-multi-user.sh} $TMPDIR/install-multi-user \
-                --subst-var-by nix ${nix} \
-                --subst-var-by cacert ${cacert}
-
-              if type -p shellcheck; then
-                # SC1090: Don't worry about not being able to find
-                #         $nix/etc/profile.d/nix.sh
-                shellcheck --exclude SC1090 $TMPDIR/install
-                shellcheck $TMPDIR/create-darwin-volume.sh
-                shellcheck $TMPDIR/install-darwin-multi-user.sh
-                shellcheck $TMPDIR/install-systemd-multi-user.sh
-
-                # SC1091: Don't panic about not being able to source
-                #         /etc/profile
-                # SC2002: Ignore "useless cat" "error", when loading
-                #         .reginfo, as the cat is a much cleaner
-                #         implementation, even though it is "useless"
-                # SC2116: Allow ROOT_HOME=$(echo ~root) for resolving
-                #         root's home directory
-                shellcheck --external-sources \
-                  --exclude SC1091,SC2002,SC2116 $TMPDIR/install-multi-user
-              fi
-
-              chmod +x $TMPDIR/install
-              chmod +x $TMPDIR/create-darwin-volume.sh
-              chmod +x $TMPDIR/install-darwin-multi-user.sh
-              chmod +x $TMPDIR/install-systemd-multi-user.sh
-              chmod +x $TMPDIR/install-multi-user
-              dir=nix-${version}-${system}
-              fn=$out/$dir.tar.xz
-              mkdir -p $out/nix-support
-              echo "file binary-dist $fn" >> $out/nix-support/hydra-build-products
-              tar cvfJ $fn \
-                --owner=0 --group=0 --mode=u+rw,uga+r \
-                --absolute-names \
-                --hard-dereference \
-                --transform "s,$TMPDIR/install,$dir/install," \
-                --transform "s,$TMPDIR/create-darwin-volume.sh,$dir/create-darwin-volume.sh," \
-                --transform "s,$TMPDIR/reginfo,$dir/.reginfo," \
-                --transform "s,$NIX_STORE,$dir/store,S" \
-                $TMPDIR/install \
-                $TMPDIR/create-darwin-volume.sh \
-                $TMPDIR/install-darwin-multi-user.sh \
-                $TMPDIR/install-systemd-multi-user.sh \
-                $TMPDIR/install-multi-user \
-                $TMPDIR/reginfo \
-                $(cat ${installerClosureInfo}/store-paths)
-            '');
+        binaryTarballCross = nixpkgs.lib.genAttrs systems (system: builtins.listToAttrs (map (crossSystem: {
+          name = crossSystem;
+          value = let
+            nixpkgsCross = import nixpkgs {
+              inherit system crossSystem;
+              overlays = [ self.overlay ];
+            };
+          in binaryTarball nixpkgsFor.${system} self.packages.${system}."nix-${crossSystem}" nixpkgsCross;
+        }) crossSystems));

        # The first half of the installation script. This is uploaded
        # to https://nixos.org/nix/install. It downloads the binary
        # tarball for the user's system and calls the second half of the
        # installation script.
-        installerScript = installScriptFor [ "x86_64-linux" "i686-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" ];
-        installerScriptForGHA = installScriptFor [ "x86_64-linux" "x86_64-darwin" ];
+        installerScript = installScriptFor [ "x86_64-linux" "i686-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" "armv6l-linux" "armv7l-linux" ];
+        installerScriptForGHA = installScriptFor [ "x86_64-linux" "x86_64-darwin" "armv6l-linux" "armv7l-linux"];

        # Line coverage analysis.
        coverage =
@@ -482,11 +498,14 @@
            # `NIX_DAEMON_SOCKET_PATH` which is required for the tests to work
            # againstLatestStable = testNixVersions pkgs pkgs.nix pkgs.nixStable;
          } "touch $out";
-      });
+      } // (if system == "x86_64-linux" then (builtins.listToAttrs (map (crossSystem: {
+        name = "binaryTarball-${crossSystem}";
+        value = self.hydraJobs.binaryTarballCross.${system}.${crossSystem};
+      }) crossSystems)) else {}));

      packages = forAllSystems (system: {
        inherit (nixpkgsFor.${system}) nix;
-      } // nixpkgs.lib.optionalAttrs (builtins.elem system linux64BitSystems) {
+      } // (nixpkgs.lib.optionalAttrs (builtins.elem system linux64BitSystems) {
        nix-static = let
          nixpkgs = nixpkgsFor.${system}.pkgsStatic;
        in with commonDeps nixpkgs; nixpkgs.stdenv.mkDerivation {
@@ -525,7 +544,46 @@

          strictDeps = true;
        };
-      });
+      } // builtins.listToAttrs (map (crossSystem: {
+        name = "nix-${crossSystem}";
+        value = let
+          nixpkgsCross = import nixpkgs {
+            inherit system crossSystem;
+            overlays = [ self.overlay ];
+          };
+        in with commonDeps nixpkgsCross; nixpkgsCross.stdenv.mkDerivation {
+          name = "nix-${version}";
+
+          src = self;
+
+          VERSION_SUFFIX = versionSuffix;
+
+          outputs = [ "out" "dev" "doc" ];
+
+          nativeBuildInputs = nativeBuildDeps;
+          buildInputs = buildDeps ++ propagatedDeps;
+
+          configureFlags = [ "--sysconfdir=/etc" "--disable-doc-gen" ];
+
+          enableParallelBuilding = true;
+
+          makeFlags = "profiledir=$(out)/etc/profile.d";
+
+          doCheck = true;
+
+          installFlags = "sysconfdir=$(out)/etc";
+
+          postInstall = ''
+            mkdir -p $doc/nix-support
+            echo "doc manual $doc/share/doc/nix/manual" >> $doc/nix-support/hydra-build-products
+            mkdir -p $out/nix-support
+            echo "file binary-dist $out/bin/nix" >> $out/nix-support/hydra-build-products
+          '';
+
+          doInstallCheck = true;
+          installCheckFlags = "sysconfdir=$(out)/etc";
+        };
+      }) crossSystems)));

      defaultPackage = forAllSystems (system: self.packages.${system}.nix);

--- a/maintainers/upload-release.pl
+++ b/maintainers/upload-release.pl
@@ -110,6 +110,8 @@ downloadFile("binaryTarball.i686-linux", "1");
 downloadFile("binaryTarball.x86_64-linux", "1");
 downloadFile("binaryTarball.aarch64-linux", "1");
 downloadFile("binaryTarball.x86_64-darwin", "1");
+downloadFile("binaryTarballCross.x86_64-linux.armv6l-linux", "1");
+downloadFile("binaryTarballCross.x86_64-linux.armv7l-linux", "1");
 downloadFile("installerScript", "1");

 for my $fn (glob "$tmpDir/*") {
--- a/misc/zsh/completion.zsh
+++ b/misc/zsh/completion.zsh
@@ -1,3 +1,5 @@
+#compdef nix
+
 function _nix() {
  local ifs_bk="$IFS"
  local input=("${(Q)words[@]}")
@@ -18,4 +20,4 @@ function _nix() {
  _describe 'nix' suggestions
 }

-compdef _nix nix
+_nix "$@"
--- a/scripts/install.in
+++ b/scripts/install.in
@@ -40,6 +40,16 @@ case "$(uname -s).$(uname -m)" in
        path=@tarballPath_aarch64-linux@
        system=aarch64-linux
        ;;
+    Linux.armv6l_linux)
+        hash=@tarballHash_armv6l-linux@
+        path=@tarballPath_armv6l-linux@
+        system=armv6l-linux
+        ;;
+    Linux.armv7l_linux)
+        hash=@tarballHash_armv7l-linux@
+        path=@tarballPath_armv7l-linux@
+        system=armv7l-linux
+        ;;
    Darwin.x86_64)
        hash=@tarballHash_x86_64-darwin@
        path=@tarballPath_x86_64-darwin@
--- a/src/build-remote/build-remote.cc
+++ b/src/build-remote/build-remote.cc
@@ -277,7 +277,16 @@ connected:

        auto drv = store->readDerivation(*drvPath);
        auto outputHashes = staticOutputHashes(*store, drv);
-        drv.inputSrcs = store->parseStorePathSet(inputs);
+
+        // Hijack the inputs paths of the derivation to include all the paths
+        // that come from the `inputDrvs` set.
+        // We don’t do that for the derivations whose `inputDrvs` is empty
+        // because
+        // 1. It’s not needed
+        // 2. Changing the `inputSrcs` set changes the associated output ids,
+        //  which break CA derivations
+        if (!drv.inputDrvs.empty())
+            drv.inputSrcs = store->parseStorePathSet(inputs);

        auto result = sshStore->buildDerivation(*drvPath, drv);

--- a/src/libstore/build/derivation-goal.cc
+++ b/src/libstore/build/derivation-goal.cc
@@ -739,6 +739,63 @@ void DerivationGoal::cleanupPostOutputsRegisteredModeNonCheck()
 {
 }

+void runPostBuildHook(
+    Store & store,
+    Logger & logger,
+    const StorePath & drvPath,
+    StorePathSet outputPaths
+)
+{
+    auto hook = settings.postBuildHook;
+    if (hook == "")
+        return;
+
+    Activity act(logger, lvlInfo, actPostBuildHook,
+            fmt("running post-build-hook '%s'", settings.postBuildHook),
+            Logger::Fields{store.printStorePath(drvPath)});
+    PushActivity pact(act.id);
+    std::map<std::string, std::string> hookEnvironment = getEnv();
+
+    hookEnvironment.emplace("DRV_PATH", store.printStorePath(drvPath));
+    hookEnvironment.emplace("OUT_PATHS", chomp(concatStringsSep(" ", store.printStorePathSet(outputPaths))));
+
+    RunOptions opts(settings.postBuildHook, {});
+    opts.environment = hookEnvironment;
+
+    struct LogSink : Sink {
+        Activity & act;
+        std::string currentLine;
+
+        LogSink(Activity & act) : act(act) { }
+
+        void operator() (std::string_view data) override {
+            for (auto c : data) {
+                if (c == '\n') {
+                    flushLine();
+                } else {
+                    currentLine += c;
+                }
+            }
+        }
+
+        void flushLine() {
+            act.result(resPostBuildLogLine, currentLine);
+            currentLine.clear();
+        }
+
+        ~LogSink() {
+            if (currentLine != "") {
+                currentLine += '\n';
+                flushLine();
+            }
+        }
+    };
+    LogSink sink(act);
+
+    opts.standardOut = &sink;
+    opts.mergeStderrToStdout = true;
+    runProgram2(opts);
+}

 void DerivationGoal::buildDone()
 {
@@ -804,57 +861,15 @@ void DerivationGoal::buildDone()
           being valid. */
        registerOutputs();

-        if (settings.postBuildHook != "") {
-            Activity act(*logger, lvlInfo, actPostBuildHook,
-                fmt("running post-build-hook '%s'", settings.postBuildHook),
-                Logger::Fields{worker.store.printStorePath(drvPath)});
-            PushActivity pact(act.id);
-            StorePathSet outputPaths;
-            for (auto i : drv->outputs) {
-                outputPaths.insert(finalOutputs.at(i.first));
-            }
-            std::map<std::string, std::string> hookEnvironment = getEnv();
-
-            hookEnvironment.emplace("DRV_PATH", worker.store.printStorePath(drvPath));
-            hookEnvironment.emplace("OUT_PATHS", chomp(concatStringsSep(" ", worker.store.printStorePathSet(outputPaths))));
-
-            RunOptions opts(settings.postBuildHook, {});
-            opts.environment = hookEnvironment;
-
-            struct LogSink : Sink {
-                Activity & act;
-                std::string currentLine;
-
-                LogSink(Activity & act) : act(act) { }
-
-                void operator() (std::string_view data) override {
-                    for (auto c : data) {
-                        if (c == '\n') {
-                            flushLine();
-                        } else {
-                            currentLine += c;
-                        }
-                    }
-                }
-
-                void flushLine() {
-                    act.result(resPostBuildLogLine, currentLine);
-                    currentLine.clear();
-                }
-
-                ~LogSink() {
-                    if (currentLine != "") {
-                        currentLine += '\n';
-                        flushLine();
-                    }
-                }
-            };
-            LogSink sink(act);
-
-            opts.standardOut = &sink;
-            opts.mergeStderrToStdout = true;
-            runProgram2(opts);
-        }
+        StorePathSet outputPaths;
+        for (auto & [_, path] : finalOutputs)
+            outputPaths.insert(path);
+        runPostBuildHook(
+            worker.store,
+            *logger,
+            drvPath,
+            outputPaths
+        );

        if (buildMode == bmCheck) {
            cleanupPostOutputsRegisteredModeCheck();
@@ -910,6 +925,8 @@ void DerivationGoal::resolvedFinished() {

    auto resolvedHashes = staticOutputHashes(worker.store, *resolvedDrv);

+    StorePathSet outputPaths;
+
    // `wantedOutputs` might be empty, which means “all the outputs”
    auto realWantedOutputs = wantedOutputs;
    if (realWantedOutputs.empty())
@@ -927,8 +944,10 @@ void DerivationGoal::resolvedFinished() {
            auto newRealisation = *realisation;
            newRealisation.id = DrvOutput{initialOutputs.at(wantedOutput).outputHash, wantedOutput};
            newRealisation.signatures.clear();
+            newRealisation.dependentRealisations = drvOutputReferences(worker.store, *drv, realisation->outPath);
            signRealisation(newRealisation);
            worker.store.registerDrvOutput(newRealisation);
+            outputPaths.insert(realisation->outPath);
        } else {
            // If we don't have a realisation, then it must mean that something
            // failed when building the resolved drv
@@ -936,6 +955,13 @@ void DerivationGoal::resolvedFinished() {
        }
    }

+    runPostBuildHook(
+        worker.store,
+        *logger,
+        drvPath,
+        outputPaths
+    );
+
    // This is potentially a bit fishy in terms of error reporting. Not sure
    // how to do it in a cleaner way
    amDone(nrFailed == 0 ? ecSuccess : ecFailed, ex);
--- a/src/libstore/build/drv-output-substitution-goal.cc
+++ b/src/libstore/build/drv-output-substitution-goal.cc
@@ -17,6 +17,13 @@ DrvOutputSubstitutionGoal::DrvOutputSubstitutionGoal(const DrvOutput& id, Worker
 void DrvOutputSubstitutionGoal::init()
 {
    trace("init");
+
+    /* If the derivation already exists, we’re done */
+    if (worker.store.queryRealisation(id)) {
+        amDone(ecSuccess);
+        return;
+    }
+
    subs = settings.useSubstitutes ? getDefaultSubstituters() : std::list<ref<Store>>();
    tryNext();
 }
@@ -53,6 +60,26 @@ void DrvOutputSubstitutionGoal::tryNext()
        return;
    }

+    for (const auto & [depId, depPath] : outputInfo->dependentRealisations) {
+        if (depId != id) {
+            if (auto localOutputInfo = worker.store.queryRealisation(depId);
+                localOutputInfo && localOutputInfo->outPath != depPath) {
+                warn(
+                    "substituter '%s' has an incompatible realisation for '%s', ignoring.\n"
+                    "Local:  %s\n"
+                    "Remote: %s",
+                    sub->getUri(),
+                    depId.to_string(),
+                    worker.store.printStorePath(localOutputInfo->outPath),
+                    worker.store.printStorePath(depPath)
+                );
+                tryNext();
+                return;
+            }
+            addWaitee(worker.makeDrvOutputSubstitutionGoal(depId));
+        }
+    }
+
    addWaitee(worker.makePathSubstitutionGoal(outputInfo->outPath));

    if (waitees.empty()) outPathValid();
--- a/src/libstore/build/local-derivation-goal.cc
+++ b/src/libstore/build/local-derivation-goal.cc
@@ -292,7 +292,7 @@ bool LocalDerivationGoal::cleanupDecideWhetherDiskFull()
        auto & localStore = getLocalStore();
        uint64_t required = 8ULL * 1024 * 1024; // FIXME: make configurable
        struct statvfs st;
-        if (statvfs(localStore.realStoreDir.c_str(), &st) == 0 &&
+        if (statvfs(localStore.realStoreDir.get().c_str(), &st) == 0 &&
            (uint64_t) st.f_bavail * st.f_bsize < required)
            diskFull = true;
        if (statvfs(tmpDir.c_str(), &st) == 0 &&
@@ -417,7 +417,7 @@ void LocalDerivationGoal::startBuilder()
    }

    auto & localStore = getLocalStore();
-    if (localStore.storeDir != localStore.realStoreDir) {
+    if (localStore.storeDir != localStore.realStoreDir.get()) {
        #if __linux__
            useChroot = true;
        #else
@@ -1333,13 +1333,18 @@ struct RestrictedStore : public virtual RestrictedStoreConfig, public virtual Lo
    std::optional<const Realisation> queryRealisation(const DrvOutput & id) override
    // XXX: This should probably be allowed if the realisation corresponds to
    // an allowed derivation
-    { throw Error("queryRealisation"); }
+    {
+        if (!goal.isAllowed(id))
+            throw InvalidPath("cannot query an unknown output id '%s' in recursive Nix", id.to_string());
+        return next->queryRealisation(id);
+    }

    void buildPaths(const std::vector<DerivedPath> & paths, BuildMode buildMode) override
    {
        if (buildMode != bmNormal) throw Error("unsupported build mode");

        StorePathSet newPaths;
+        std::set<Realisation> newRealisations;

        for (auto & req : paths) {
            if (!goal.isAllowed(req))
@@ -1352,16 +1357,28 @@ struct RestrictedStore : public virtual RestrictedStoreConfig, public virtual Lo
            auto p =  std::get_if<DerivedPath::Built>(&path);
            if (!p) continue;
            auto & bfd = *p;
+            auto drv = readDerivation(bfd.drvPath);
+            auto drvHashes = staticOutputHashes(*this, drv);
            auto outputs = next->queryDerivationOutputMap(bfd.drvPath);
            for (auto & [outputName, outputPath] : outputs)
-                if (wantOutput(outputName, bfd.outputs))
+                if (wantOutput(outputName, bfd.outputs)) {
                    newPaths.insert(outputPath);
+                    if (settings.isExperimentalFeatureEnabled("ca-derivations")) {
+                        auto thisRealisation = next->queryRealisation(
+                            DrvOutput{drvHashes.at(outputName), outputName}
+                        );
+                        assert(thisRealisation);
+                        newRealisations.insert(*thisRealisation);
+                    }
+                }
        }

        StorePathSet closure;
        next->computeFSClosure(newPaths, closure);
        for (auto & path : closure)
            goal.addDependency(path);
+        for (auto & real : Realisation::closure(*next, newRealisations))
+            goal.addedDrvOutputs.insert(real.id);
    }

    BuildResult buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
@@ -2464,6 +2481,7 @@ void LocalDerivationGoal::registerOutputs()
           floating CA derivations and hash-mismatching fixed-output
           derivations. */
        PathLocks dynamicOutputLock;
+        dynamicOutputLock.setDeletion(true);
        auto optFixedPath = output.path(worker.store, drv->name, outputName);
        if (!optFixedPath ||
            worker.store.printStorePath(*optFixedPath) != finalDestPath)
@@ -2487,6 +2505,7 @@ void LocalDerivationGoal::registerOutputs()
                assert(newInfo.ca);
            } else {
                auto destPath = worker.store.toRealPath(finalDestPath);
+                deletePath(destPath);
                movePath(actualPath, destPath);
                actualPath = destPath;
            }
--- a/src/libstore/build/local-derivation-goal.hh
+++ b/src/libstore/build/local-derivation-goal.hh
@@ -108,6 +108,9 @@ struct LocalDerivationGoal : public DerivationGoal
    /* Paths that were added via recursive Nix calls. */
    StorePathSet addedPaths;

+    /* Realisations that were added via recursive Nix calls. */
+    std::set<DrvOutput> addedDrvOutputs;
+
    /* Recursive Nix calls are only allowed to build or realize paths
       in the original input closure or added via a recursive Nix call
       (so e.g. you can't do 'nix-store -r /nix/store/<bla>' where
@@ -116,6 +119,11 @@ struct LocalDerivationGoal : public DerivationGoal
    {
        return inputPaths.count(path) || addedPaths.count(path);
    }
+    bool isAllowed(const DrvOutput & id)
+    {
+        return addedDrvOutputs.count(id);
+    }
+
    bool isAllowed(const DerivedPath & req);

    friend struct RestrictedStore;
--- a/src/libstore/ca-specific-schema.sql
+++ b/src/libstore/ca-specific-schema.sql
@@ -3,10 +3,19 @@
 -- is enabled

 create table if not exists Realisations (
+    id integer primary key autoincrement not null,
    drvPath text not null,
    outputName text not null, -- symbolic output id, usually "out"
    outputPath integer not null,
    signatures text, -- space-separated list
-    primary key (drvPath, outputName),
    foreign key (outputPath) references ValidPaths(id) on delete cascade
 );
+
+create index if not exists IndexRealisations on Realisations(drvPath, outputName);
+
+create table if not exists RealisationsRefs (
+    referrer integer not null,
+    realisationReference integer,
+    foreign key (referrer) references Realisations(id) on delete cascade,
+    foreign key (realisationReference) references Realisations(id) on delete restrict
+);
--- a/src/libstore/daemon.cc
+++ b/src/libstore/daemon.cc
@@ -885,10 +885,15 @@ static void performOp(TunnelLogger * logger, ref<Store> store,

    case wopRegisterDrvOutput: {
        logger->startWork();
-        auto outputId = DrvOutput::parse(readString(from));
-        auto outputPath = StorePath(readString(from));
-        store->registerDrvOutput(Realisation{
-            .id = outputId, .outPath = outputPath});
+        if (GET_PROTOCOL_MINOR(clientVersion) < 31) {
+            auto outputId = DrvOutput::parse(readString(from));
+            auto outputPath = StorePath(readString(from));
+            store->registerDrvOutput(Realisation{
+                .id = outputId, .outPath = outputPath});
+        } else {
+            auto realisation = worker_proto::read(*store, from, Phantom<Realisation>());
+            store->registerDrvOutput(realisation);
+        }
        logger->stopWork();
        break;
    }
@@ -898,9 +903,15 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
        auto outputId = DrvOutput::parse(readString(from));
        auto info = store->queryRealisation(outputId);
        logger->stopWork();
-        std::set<StorePath> outPaths;
-        if (info) outPaths.insert(info->outPath);
-        worker_proto::write(*store, to, outPaths);
+        if (GET_PROTOCOL_MINOR(clientVersion) < 31) {
+            std::set<StorePath> outPaths;
+            if (info) outPaths.insert(info->outPath);
+            worker_proto::write(*store, to, outPaths);
+        } else {
+            std::set<Realisation> realisations;
+            if (info) realisations.insert(*info);
+            worker_proto::write(*store, to, realisations);
+        }
        break;
    }

--- a/src/libstore/gc.cc
+++ b/src/libstore/gc.cc
@@ -775,7 +775,7 @@ void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)

        try {

-            AutoCloseDir dir(opendir(realStoreDir.c_str()));
+            AutoCloseDir dir(opendir(realStoreDir.get().c_str()));
            if (!dir) throw SysError("opening directory '%1%'", realStoreDir);

            /* Read the store and immediately delete all paths that
@@ -856,7 +856,7 @@ void LocalStore::autoGC(bool sync)
            return std::stoll(readFile(*fakeFreeSpaceFile));

        struct statvfs st;
-        if (statvfs(realStoreDir.c_str(), &st))
+        if (statvfs(realStoreDir.get().c_str(), &st))
            throw SysError("getting filesystem info about '%s'", realStoreDir);

        return (uint64_t) st.f_bavail * st.f_frsize;
--- a/src/libstore/local-fs-store.hh
+++ b/src/libstore/local-fs-store.hh
@@ -18,6 +18,9 @@ struct LocalFSStoreConfig : virtual StoreConfig
    const PathSetting logDir{(StoreConfig*) this, false,
        rootDir != "" ? rootDir + "/nix/var/log/nix" : settings.nixLogDir,
        "log", "directory where Nix will store state"};
+    const PathSetting realStoreDir{(StoreConfig*) this, false,
+        rootDir != "" ? rootDir + "/nix/store" : storeDir, "real",
+        "physical path to the Nix store"};
 };

 class LocalFSStore : public virtual LocalFSStoreConfig, public virtual Store
@@ -34,7 +37,7 @@ public:
    /* Register a permanent GC root. */
    Path addPermRoot(const StorePath & storePath, const Path & gcRoot);

-    virtual Path getRealStoreDir() { return storeDir; }
+    virtual Path getRealStoreDir() { return realStoreDir; }

    Path toRealPath(const Path & storePath) override
    {
--- a/src/libstore/local-store.cc
+++ b/src/libstore/local-store.cc
@@ -53,12 +53,15 @@ struct LocalStore::State::Stmts {
    SQLiteStmt InvalidatePath;
    SQLiteStmt AddDerivationOutput;
    SQLiteStmt RegisterRealisedOutput;
+    SQLiteStmt UpdateRealisedOutput;
    SQLiteStmt QueryValidDerivers;
    SQLiteStmt QueryDerivationOutputs;
    SQLiteStmt QueryRealisedOutput;
    SQLiteStmt QueryAllRealisedOutputs;
    SQLiteStmt QueryPathFromHashPart;
    SQLiteStmt QueryValidPaths;
+    SQLiteStmt QueryRealisationReferences;
+    SQLiteStmt AddRealisationReference;
 };

 int getSchema(Path schemaPath)
@@ -76,7 +79,7 @@ int getSchema(Path schemaPath)

 void migrateCASchema(SQLite& db, Path schemaPath, AutoCloseFD& lockFd)
 {
-    const int nixCASchemaVersion = 1;
+    const int nixCASchemaVersion = 2;
    int curCASchema = getSchema(schemaPath);
    if (curCASchema != nixCASchemaVersion) {
        if (curCASchema > nixCASchemaVersion) {
@@ -94,7 +97,39 @@ void migrateCASchema(SQLite& db, Path schemaPath, AutoCloseFD& lockFd)
              #include "ca-specific-schema.sql.gen.hh"
                ;
            db.exec(schema);
+            curCASchema = nixCASchemaVersion;
        }
+
+        if (curCASchema < 2) {
+            SQLiteTxn txn(db);
+            // Ugly little sql dance to add a new `id` column and make it the primary key
+            db.exec(R"(
+                create table Realisations2 (
+                    id integer primary key autoincrement not null,
+                    drvPath text not null,
+                    outputName text not null, -- symbolic output id, usually "out"
+                    outputPath integer not null,
+                    signatures text, -- space-separated list
+                    foreign key (outputPath) references ValidPaths(id) on delete cascade
+                );
+                insert into Realisations2 (drvPath, outputName, outputPath, signatures)
+                    select drvPath, outputName, outputPath, signatures from Realisations;
+                drop table Realisations;
+                alter table Realisations2 rename to Realisations;
+            )");
+            db.exec(R"(
+                create index if not exists IndexRealisations on Realisations(drvPath, outputName);
+
+                create table if not exists RealisationsRefs (
+                    referrer integer not null,
+                    realisationReference integer,
+                    foreign key (referrer) references Realisations(id) on delete cascade,
+                    foreign key (realisationReference) references Realisations(id) on delete restrict
+                );
+            )");
+            txn.commit();
+        }
+
        writeFile(schemaPath, fmt("%d", nixCASchemaVersion));
        lockFile(lockFd.get(), ltRead, true);
    }
@@ -106,9 +141,6 @@ LocalStore::LocalStore(const Params & params)
    , LocalStoreConfig(params)
    , Store(params)
    , LocalFSStore(params)
-    , realStoreDir_{this, false, rootDir != "" ? rootDir + "/nix/store" : storeDir, "real",
-        "physical path to the Nix store"}
-    , realStoreDir(realStoreDir_)
    , dbDir(stateDir + "/db")
    , linksDir(realStoreDir + "/.links")
    , reservedPath(dbDir + "/reserved")
@@ -153,13 +185,13 @@ LocalStore::LocalStore(const Params & params)
            printError("warning: the group '%1%' specified in 'build-users-group' does not exist", settings.buildUsersGroup);
        else {
            struct stat st;
-            if (stat(realStoreDir.c_str(), &st))
+            if (stat(realStoreDir.get().c_str(), &st))
                throw SysError("getting attributes of path '%1%'", realStoreDir);

            if (st.st_uid != 0 || st.st_gid != gr->gr_gid || (st.st_mode & ~S_IFMT) != perm) {
-                if (chown(realStoreDir.c_str(), 0, gr->gr_gid) == -1)
+                if (chown(realStoreDir.get().c_str(), 0, gr->gr_gid) == -1)
                    throw SysError("changing ownership of path '%1%'", realStoreDir);
-                if (chmod(realStoreDir.c_str(), perm) == -1)
+                if (chmod(realStoreDir.get().c_str(), perm) == -1)
                    throw SysError("changing permissions on path '%1%'", realStoreDir);
            }
        }
@@ -314,9 +346,18 @@ LocalStore::LocalStore(const Params & params)
                values (?, ?, (select id from ValidPaths where path = ?), ?)
                ;
            )");
+        state->stmts->UpdateRealisedOutput.create(state->db,
+            R"(
+                update Realisations
+                    set signatures = ?
+                where
+                    drvPath = ? and
+                    outputName = ?
+                ;
+            )");
        state->stmts->QueryRealisedOutput.create(state->db,
            R"(
-                select Output.path, Realisations.signatures from Realisations
+                select Realisations.id, Output.path, Realisations.signatures from Realisations
                    inner join ValidPaths as Output on Output.id = Realisations.outputPath
                    where drvPath = ? and outputName = ?
                    ;
@@ -328,6 +369,19 @@ LocalStore::LocalStore(const Params & params)
                    where drvPath = ?
                    ;
            )");
+        state->stmts->QueryRealisationReferences.create(state->db,
+            R"(
+                select drvPath, outputName from Realisations
+                    join RealisationsRefs on realisationReference = Realisations.id
+                    where referrer = ?;
+            )");
+        state->stmts->AddRealisationReference.create(state->db,
+            R"(
+                insert or replace into RealisationsRefs (referrer, realisationReference)
+                values (
+                    ?,
+                    (select id from Realisations where drvPath = ? and outputName = ?));
+            )");
    }
 }

@@ -437,14 +491,14 @@ void LocalStore::makeStoreWritable()
    if (getuid() != 0) return;
    /* Check if /nix/store is on a read-only mount. */
    struct statvfs stat;
-    if (statvfs(realStoreDir.c_str(), &stat) != 0)
+    if (statvfs(realStoreDir.get().c_str(), &stat) != 0)
        throw SysError("getting info about the Nix store mount point");

    if (stat.f_flag & ST_RDONLY) {
        if (unshare(CLONE_NEWNS) == -1)
            throw SysError("setting up a private mount namespace");

-        if (mount(0, realStoreDir.c_str(), "none", MS_REMOUNT | MS_BIND, 0) == -1)
+        if (mount(0, realStoreDir.get().c_str(), "none", MS_REMOUNT | MS_BIND, 0) == -1)
            throw SysError("remounting %1% writable", realStoreDir);
    }
 #endif
@@ -664,14 +718,54 @@ void LocalStore::registerDrvOutput(const Realisation & info, CheckSigsFlag check
 void LocalStore::registerDrvOutput(const Realisation & info)
 {
    settings.requireExperimentalFeature("ca-derivations");
-    auto state(_state.lock());
    retrySQLite<void>([&]() {
-        state->stmts->RegisterRealisedOutput.use()
-            (info.id.strHash())
-            (info.id.outputName)
-            (printStorePath(info.outPath))
-            (concatStringsSep(" ", info.signatures))
-            .exec();
+        auto state(_state.lock());
+        if (auto oldR = queryRealisation_(*state, info.id)) {
+            if (info.isCompatibleWith(*oldR)) {
+                auto combinedSignatures = oldR->signatures;
+                combinedSignatures.insert(info.signatures.begin(),
+                    info.signatures.end());
+                state->stmts->UpdateRealisedOutput.use()
+                    (concatStringsSep(" ", combinedSignatures))
+                    (info.id.strHash())
+                    (info.id.outputName)
+                    .exec();
+            } else {
+                throw Error("Trying to register a realisation of '%s', but we already "
+                            "have another one locally.\n"
+                            "Local:  %s\n"
+                            "Remote: %s",
+                    info.id.to_string(),
+                    printStorePath(oldR->outPath),
+                    printStorePath(info.outPath)
+                );
+            }
+        } else {
+            state->stmts->RegisterRealisedOutput.use()
+                (info.id.strHash())
+                (info.id.outputName)
+                (printStorePath(info.outPath))
+                (concatStringsSep(" ", info.signatures))
+                .exec();
+        }
+        uint64_t myId = state->db.getLastInsertedRowId();
+        for (auto & [outputId, depPath] : info.dependentRealisations) {
+            auto localRealisation = queryRealisationCore_(*state, outputId);
+            if (!localRealisation)
+                throw Error("unable to register the derivation '%s' as it "
+                            "depends on the non existent '%s'",
+                    info.id.to_string(), outputId.to_string());
+            if (localRealisation->second.outPath != depPath)
+                throw Error("unable to register the derivation '%s' as it "
+                            "depends on a realisation of '%s' that doesn’t"
+                            "match what we have locally",
+                    info.id.to_string(), outputId.to_string());
+            state->stmts->AddRealisationReference.use()
+                (myId)
+                (outputId.strHash())
+                (outputId.outputName)
+                .exec();
+        }
    });
 }

@@ -1682,22 +1776,68 @@ void LocalStore::createUser(const std::string & userName, uid_t userId)
    }
 }

-std::optional<const Realisation> LocalStore::queryRealisation(
-    const DrvOutput& id) {
-    typedef std::optional<const Realisation> Ret;
-    return retrySQLite<Ret>([&]() -> Ret {
-        auto state(_state.lock());
-        auto use(state->stmts->QueryRealisedOutput.use()(id.strHash())(
-            id.outputName));
-        if (!use.next())
-            return std::nullopt;
-        auto outputPath = parseStorePath(use.getStr(0));
-        auto signatures = tokenizeString<StringSet>(use.getStr(1));
-        return Ret{Realisation{
-            .id = id, .outPath = outputPath, .signatures = signatures}};
-    });
+std::optional<std::pair<int64_t, Realisation>> LocalStore::queryRealisationCore_(
+        LocalStore::State & state,
+        const DrvOutput & id)
+{
+    auto useQueryRealisedOutput(
+            state.stmts->QueryRealisedOutput.use()
+                (id.strHash())
+                (id.outputName));
+    if (!useQueryRealisedOutput.next())
+        return std::nullopt;
+    auto realisationDbId = useQueryRealisedOutput.getInt(0);
+    auto outputPath = parseStorePath(useQueryRealisedOutput.getStr(1));
+    auto signatures =
+        tokenizeString<StringSet>(useQueryRealisedOutput.getStr(2));
+
+    return {{
+        realisationDbId,
+        Realisation{
+            .id = id,
+            .outPath = outputPath,
+            .signatures = signatures,
+        }
+    }};
 }

+std::optional<const Realisation> LocalStore::queryRealisation_(
+            LocalStore::State & state,
+            const DrvOutput & id)
+{
+    auto maybeCore = queryRealisationCore_(state, id);
+    if (!maybeCore)
+        return std::nullopt;
+    auto [realisationDbId, res] = *maybeCore;
+
+    std::map<DrvOutput, StorePath> dependentRealisations;
+    auto useRealisationRefs(
+        state.stmts->QueryRealisationReferences.use()
+            (realisationDbId));
+    while (useRealisationRefs.next()) {
+        auto depId = DrvOutput {
+            Hash::parseAnyPrefixed(useRealisationRefs.getStr(0)),
+            useRealisationRefs.getStr(1),
+        };
+        auto dependentRealisation = queryRealisationCore_(state, depId);
+        assert(dependentRealisation); // Enforced by the db schema
+        auto outputPath = dependentRealisation->second.outPath;
+        dependentRealisations.insert({depId, outputPath});
+    }
+
+    res.dependentRealisations = dependentRealisations;
+
+    return { res };
+}
+
+std::optional<const Realisation>
+LocalStore::queryRealisation(const DrvOutput & id)
+{
+    return retrySQLite<std::optional<const Realisation>>([&]() {
+        auto state(_state.lock());
+        return queryRealisation_(*state, id);
+    });
+}

 FixedOutputHash LocalStore::hashCAPath(
    const FileIngestionMethod & method, const HashType & hashType,
--- a/src/libstore/local-store.hh
+++ b/src/libstore/local-store.hh
@@ -83,9 +83,6 @@ private:

 public:

-    PathSetting realStoreDir_;
-
-    const Path realStoreDir;
    const Path dbDir;
    const Path linksDir;
    const Path reservedPath;
@@ -206,6 +203,8 @@ public:
    void registerDrvOutput(const Realisation & info, CheckSigsFlag checkSigs) override;
    void cacheDrvOutputMapping(State & state, const uint64_t deriver, const string & outputName, const StorePath & output);

+    std::optional<const Realisation> queryRealisation_(State & state, const DrvOutput & id);
+    std::optional<std::pair<int64_t, Realisation>> queryRealisationCore_(State & state, const DrvOutput & id);
    std::optional<const Realisation> queryRealisation(const DrvOutput&) override;

 private:
@@ -279,8 +278,6 @@ private:
    void signPathInfo(ValidPathInfo & info);
    void signRealisation(Realisation &);

-    Path getRealStoreDir() override { return realStoreDir; }
-
    void createUser(const std::string & userName, uid_t userId) override;

    // XXX: Make a generic `Store` method
--- a/src/libstore/misc.cc
+++ b/src/libstore/misc.cc
@@ -29,9 +29,9 @@ void Store::computeFSClosure(const StorePathSet & startPaths,
                    res.insert(i);

            if (includeDerivers && path.isDerivation())
-                for (auto& i : queryDerivationOutputs(path))
-                    if (isValidPath(i) && queryPathInfo(i)->deriver == path)
-                        res.insert(i);
+                for (auto& [_, maybeOutPath] : queryPartialDerivationOutputMap(path))
+                    if (maybeOutPath && isValidPath(*maybeOutPath))
+                        res.insert(*maybeOutPath);
            return res;
        };
    else
@@ -44,9 +44,9 @@ void Store::computeFSClosure(const StorePathSet & startPaths,
                    res.insert(ref);

            if (includeOutputs && path.isDerivation())
-                for (auto& i : queryDerivationOutputs(path))
-                    if (isValidPath(i))
-                        res.insert(i);
+                for (auto& [_, maybeOutPath] : queryPartialDerivationOutputMap(path))
+                    if (maybeOutPath && isValidPath(*maybeOutPath))
+                        res.insert(*maybeOutPath);

            if (includeDerivers && info->deriver && isValidPath(*info->deriver))
                res.insert(*info->deriver);
@@ -254,5 +254,44 @@ StorePaths Store::topoSortPaths(const StorePathSet & paths)
        }});
 }

+std::map<DrvOutput, StorePath> drvOutputReferences(
+    const std::set<Realisation> & inputRealisations,
+    const StorePathSet & pathReferences)
+{
+    std::map<DrvOutput, StorePath> res;

+    for (const auto & input : inputRealisations) {
+        if (pathReferences.count(input.outPath)) {
+            res.insert({input.id, input.outPath});
+        }
+    }
+
+    return res;
+}
+
+std::map<DrvOutput, StorePath> drvOutputReferences(
+    Store & store,
+    const Derivation & drv,
+    const StorePath & outputPath)
+{
+    std::set<Realisation> inputRealisations;
+
+    for (const auto& [inputDrv, outputNames] : drv.inputDrvs) {
+        auto outputHashes =
+            staticOutputHashes(store, store.readDerivation(inputDrv));
+        for (const auto& outputName : outputNames) {
+            auto thisRealisation = store.queryRealisation(
+                DrvOutput{outputHashes.at(outputName), outputName});
+            if (!thisRealisation)
+                throw Error(
+                    "output '%s' of derivation '%s' isn’t built", outputName,
+                    store.printStorePath(inputDrv));
+            inputRealisations.insert(*thisRealisation);
+        }
+    }
+
+    auto info = store.queryPathInfo(outputPath);
+
+    return drvOutputReferences(Realisation::closure(store, inputRealisations), info->references);
+}
 }
--- a/src/libstore/optimise-store.cc
+++ b/src/libstore/optimise-store.cc
@@ -198,7 +198,7 @@ void LocalStore::optimisePath_(Activity * act, OptimiseStats & stats,
    /* Make the containing directory writable, but only if it's not
       the store itself (we don't want or need to mess with its
       permissions). */
-    bool mustToggle = dirOf(path) != realStoreDir;
+    bool mustToggle = dirOf(path) != realStoreDir.get();
    if (mustToggle) makeWritable(dirOf(path));

    /* When we're done, make the directory read-only again and reset
--- a/src/libstore/parsed-derivations.cc
+++ b/src/libstore/parsed-derivations.cc
@@ -91,6 +91,8 @@ StringSet ParsedDerivation::getRequiredSystemFeatures() const
    StringSet res;
    for (auto & i : getStringsAttr("requiredSystemFeatures").value_or(Strings()))
        res.insert(i);
+    if (!derivationHasKnownOutputPaths(drv.type()))
+        res.insert("ca-derivations");
    return res;
 }

--- a/src/libstore/realisation.cc
+++ b/src/libstore/realisation.cc
@@ -1,5 +1,6 @@
 #include "realisation.hh"
 #include "store-api.hh"
+#include "closure.hh"
 #include <nlohmann/json.hpp>

 namespace nix {
@@ -21,11 +22,52 @@ std::string DrvOutput::to_string() const {
    return strHash() + "!" + outputName;
 }

+std::set<Realisation> Realisation::closure(Store & store, const std::set<Realisation> & startOutputs)
+{
+    std::set<Realisation> res;
+    Realisation::closure(store, startOutputs, res);
+    return res;
+}
+
+void Realisation::closure(Store & store, const std::set<Realisation> & startOutputs, std::set<Realisation> & res)
+{
+    auto getDeps = [&](const Realisation& current) -> std::set<Realisation> {
+        std::set<Realisation> res;
+        for (auto& [currentDep, _] : current.dependentRealisations) {
+            if (auto currentRealisation = store.queryRealisation(currentDep))
+                res.insert(*currentRealisation);
+            else
+                throw Error(
+                    "Unrealised derivation '%s'", currentDep.to_string());
+        }
+        return res;
+    };
+
+    computeClosure<Realisation>(
+        startOutputs, res,
+        [&](const Realisation& current,
+            std::function<void(std::promise<std::set<Realisation>>&)>
+                processEdges) {
+            std::promise<std::set<Realisation>> promise;
+            try {
+                auto res = getDeps(current);
+                promise.set_value(res);
+            } catch (...) {
+                promise.set_exception(std::current_exception());
+            }
+            return processEdges(promise);
+        });
+}
+
 nlohmann::json Realisation::toJSON() const {
+    auto jsonDependentRealisations = nlohmann::json::object();
+    for (auto & [depId, depOutPath] : dependentRealisations)
+        jsonDependentRealisations.emplace(depId.to_string(), depOutPath.to_string());
    return nlohmann::json{
        {"id", id.to_string()},
        {"outPath", outPath.to_string()},
        {"signatures", signatures},
+        {"dependentRealisations", jsonDependentRealisations},
    };
 }

@@ -51,10 +93,16 @@ Realisation Realisation::fromJSON(
    if (auto signaturesIterator = json.find("signatures"); signaturesIterator != json.end())
        signatures.insert(signaturesIterator->begin(), signaturesIterator->end());

+    std::map <DrvOutput, StorePath> dependentRealisations;
+    if (auto jsonDependencies = json.find("dependentRealisations"); jsonDependencies != json.end())
+        for (auto & [jsonDepId, jsonDepOutPath] : jsonDependencies->get<std::map<std::string, std::string>>())
+            dependentRealisations.insert({DrvOutput::parse(jsonDepId), StorePath(jsonDepOutPath)});
+
    return Realisation{
        .id = DrvOutput::parse(getField("id")),
        .outPath = StorePath(getField("outPath")),
        .signatures = signatures,
+        .dependentRealisations = dependentRealisations,
    };
 }

@@ -92,6 +140,16 @@ StorePath RealisedPath::path() const {
    return std::visit([](auto && arg) { return arg.getPath(); }, raw);
 }

+bool Realisation::isCompatibleWith(const Realisation & other) const
+{
+    assert (id == other.id);
+    if (outPath == other.outPath) {
+        assert(dependentRealisations == other.dependentRealisations);
+        return true;
+    }
+    return false;
+}
+
 void RealisedPath::closure(
    Store& store,
    const RealisedPath::Set& startPaths,
--- a/src/libstore/realisation.hh
+++ b/src/libstore/realisation.hh
@@ -28,6 +28,14 @@ struct Realisation {

    StringSet signatures;

+    /**
+     * The realisations that are required for the current one to be valid.
+     *
+     * When importing this realisation, the store will first check that all its
+     * dependencies exist, and map to the correct output path
+     */
+    std::map<DrvOutput, StorePath> dependentRealisations;
+
    nlohmann::json toJSON() const;
    static Realisation fromJSON(const nlohmann::json& json, const std::string& whence);

@@ -36,6 +44,11 @@ struct Realisation {
    bool checkSignature(const PublicKeys & publicKeys, const std::string & sig) const;
    size_t checkSignatures(const PublicKeys & publicKeys) const;

+    static std::set<Realisation> closure(Store &, const std::set<Realisation> &);
+    static void closure(Store &, const std::set<Realisation> &, std::set<Realisation>& res);
+
+    bool isCompatibleWith(const Realisation & other) const;
+
    StorePath getPath() const { return outPath; }

    GENERATE_CMP(Realisation, me->id, me->outPath);
--- a/src/libstore/remote-store.cc
+++ b/src/libstore/remote-store.cc
@@ -653,8 +653,12 @@ void RemoteStore::registerDrvOutput(const Realisation & info)
 {
    auto conn(getConnection());
    conn->to << wopRegisterDrvOutput;
-    conn->to << info.id.to_string();
-    conn->to << std::string(info.outPath.to_string());
+    if (GET_PROTOCOL_MINOR(conn->daemonVersion) < 31) {
+        conn->to << info.id.to_string();
+        conn->to << std::string(info.outPath.to_string());
+    } else {
+        worker_proto::write(*this, conn->to, info);
+    }
    conn.processStderr();
 }

@@ -664,10 +668,17 @@ std::optional<const Realisation> RemoteStore::queryRealisation(const DrvOutput &
    conn->to << wopQueryRealisation;
    conn->to << id.to_string();
    conn.processStderr();
-    auto outPaths = worker_proto::read(*this, conn->from, Phantom<std::set<StorePath>>{});
-    if (outPaths.empty())
-        return std::nullopt;
-    return {Realisation{.id = id, .outPath = *outPaths.begin()}};
+    if (GET_PROTOCOL_MINOR(conn->daemonVersion) < 31) {
+        auto outPaths = worker_proto::read(*this, conn->from, Phantom<std::set<StorePath>>{});
+        if (outPaths.empty())
+            return std::nullopt;
+        return {Realisation{.id = id, .outPath = *outPaths.begin()}};
+    } else {
+        auto realisations = worker_proto::read(*this, conn->from, Phantom<std::set<Realisation>>{});
+        if (realisations.empty())
+            return std::nullopt;
+        return *realisations.begin();
+    }
 }

 static void writeDerivedPaths(RemoteStore & store, ConnectionHandle & conn, const std::vector<DerivedPath> & reqs)
--- a/src/libstore/store-api.cc
+++ b/src/libstore/store-api.cc
@@ -337,6 +337,13 @@ ValidPathInfo Store::addToStoreSlow(std::string_view name, const Path & srcPath,
    return info;
 }

+StringSet StoreConfig::getDefaultSystemFeatures()
+{
+    auto res = settings.systemFeatures.get();
+    if (settings.isExperimentalFeatureEnabled("ca-derivations"))
+        res.insert("ca-derivations");
+    return res;
+}

 Store::Store(const Params & params)
    : StoreConfig(params)
@@ -780,20 +787,39 @@ std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStor
    RepairFlag repair, CheckSigsFlag checkSigs, SubstituteFlag substitute)
 {
    StorePathSet storePaths;
-    std::set<Realisation> realisations;
+    std::set<Realisation> toplevelRealisations;
    for (auto & path : paths) {
        storePaths.insert(path.path());
        if (auto realisation = std::get_if<Realisation>(&path.raw)) {
            settings.requireExperimentalFeature("ca-derivations");
-            realisations.insert(*realisation);
+            toplevelRealisations.insert(*realisation);
        }
    }
    auto pathsMap = copyPaths(srcStore, dstStore, storePaths, repair, checkSigs, substitute);
+
+    ThreadPool pool;
+
    try {
-        for (auto & realisation : realisations) {
-            dstStore->registerDrvOutput(realisation, checkSigs);
-        }
-    } catch (MissingExperimentalFeature & e) {
+        // Copy the realisation closure
+        processGraph<Realisation>(
+            pool, Realisation::closure(*srcStore, toplevelRealisations),
+            [&](const Realisation& current) -> std::set<Realisation> {
+                std::set<Realisation> children;
+                for (const auto& [drvOutput, _] : current.dependentRealisations) {
+                    auto currentChild = srcStore->queryRealisation(drvOutput);
+                    if (!currentChild)
+                        throw Error(
+                            "Incomplete realisation closure: '%s' is a "
+                            "dependency of '%s' but isn’t registered",
+                            drvOutput.to_string(), current.id.to_string());
+                    children.insert(*currentChild);
+                }
+                return children;
+            },
+            [&](const Realisation& current) -> void {
+                dstStore->registerDrvOutput(current, checkSigs);
+            });
+    } catch (MissingExperimentalFeature& e) {
        // Don't fail if the remote doesn't support CA derivations is it might
        // not be within our control to change that, and we might still want
        // to at least copy the output paths.
--- a/src/libstore/store-api.hh
+++ b/src/libstore/store-api.hh
@@ -180,6 +180,8 @@ struct StoreConfig : public Config

    StoreConfig() = delete;

+    StringSet getDefaultSystemFeatures();
+
    virtual ~StoreConfig() { }

    virtual const std::string name() = 0;
@@ -196,7 +198,7 @@ struct StoreConfig : public Config

    Setting<bool> wantMassQuery{this, false, "want-mass-query", "whether this substituter can be queried efficiently for path validity"};

-    Setting<StringSet> systemFeatures{this, settings.systemFeatures,
+    Setting<StringSet> systemFeatures{this, getDefaultSystemFeatures(),
        "system-features",
        "Optional features that the system this store builds on implements (like \"kvm\")."};

@@ -864,4 +866,9 @@ std::pair<std::string, Store::Params> splitUriAndParams(const std::string & uri)

 std::optional<ContentAddress> getDerivationCA(const BasicDerivation & drv);

+std::map<DrvOutput, StorePath> drvOutputReferences(
+    Store & store,
+    const Derivation & drv,
+    const StorePath & outputPath);
+
 }
--- a/src/libstore/worker-protocol.hh
+++ b/src/libstore/worker-protocol.hh
@@ -9,7 +9,7 @@ namespace nix {
 #define WORKER_MAGIC_1 0x6e697863
 #define WORKER_MAGIC_2 0x6478696f

-#define PROTOCOL_VERSION (1 << 8 | 30)
+#define PROTOCOL_VERSION (1 << 8 | 31)
 #define GET_PROTOCOL_MAJOR(x) ((x) & 0xff00)
 #define GET_PROTOCOL_MINOR(x) ((x) & 0x00ff)

--- a/src/libutil/comparator.hh
+++ b/src/libutil/comparator.hh
@@ -25,6 +25,8 @@
    }
 #define GENERATE_EQUAL(args...) GENERATE_ONE_CMP(==, args)
 #define GENERATE_LEQ(args...) GENERATE_ONE_CMP(<, args)
+#define GENERATE_NEQ(args...) GENERATE_ONE_CMP(!=, args)
 #define GENERATE_CMP(args...) \
    GENERATE_EQUAL(args) \
-    GENERATE_LEQ(args)
+    GENERATE_LEQ(args) \
+    GENERATE_NEQ(args)
--- a/src/nix-build/nix-build.cc
+++ b/src/nix-build/nix-build.cc
@@ -387,6 +387,12 @@ static void main_nix_build(int argc, char * * argv)

        if (dryRun) return;

+        if (settings.isExperimentalFeatureEnabled("ca-derivations")) {
+            auto resolvedDrv = drv.tryResolve(*store);
+            assert(resolvedDrv && "Successfully resolved the derivation");
+            drv = *resolvedDrv;
+        }
+
        // Set the environment.
        auto env = getEnv();

--- a/src/nix/build.cc
+++ b/src/nix/build.cc
@@ -69,8 +69,7 @@ struct CmdBuild : InstallablesCommand, MixDryRun, MixJSON, MixProfile
                            store2->addPermRoot(bo.path, absPath(symlink));
                        },
                        [&](BuiltPath::Built bfd) {
-                            auto builtOutputs = store->queryDerivationOutputMap(bfd.drvPath);
-                            for (auto & output : builtOutputs) {
+                            for (auto & output : bfd.outputs) {
                                std::string symlink = outLink;
                                if (i) symlink += fmt("-%d", i);
                                if (output.first != "out") symlink += fmt("-%s", output.first);
--- a/src/nix/develop.cc
+++ b/src/nix/develop.cc
@@ -144,17 +144,26 @@ StorePath getDerivationEnvironment(ref<Store> store, const StorePath & drvPath)
    /* Rehash and write the derivation. FIXME: would be nice to use
       'buildDerivation', but that's privileged. */
    drv.name += "-env";
-    for (auto & output : drv.outputs) {
-        output.second = { .output = DerivationOutputInputAddressed { .path = StorePath::dummy } };
-        drv.env[output.first] = "";
-    }
    drv.inputSrcs.insert(std::move(getEnvShPath));
-    Hash h = std::get<0>(hashDerivationModulo(*store, drv, true));
+    if (settings.isExperimentalFeatureEnabled("ca-derivations")) {
+        for (auto & output : drv.outputs) {
+            output.second = {
+                .output = DerivationOutputDeferred{},
+            };
+            drv.env[output.first] = hashPlaceholder(output.first);
+        }
+    } else {
+        for (auto & output : drv.outputs) {
+            output.second = { .output = DerivationOutputInputAddressed { .path = StorePath::dummy } };
+            drv.env[output.first] = "";
+        }
+        Hash h = std::get<0>(hashDerivationModulo(*store, drv, true));

-    for (auto & output : drv.outputs) {
-        auto outPath = store->makeOutputPath(output.first, h, drv.name);
-        output.second = { .output = DerivationOutputInputAddressed { .path = outPath } };
-        drv.env[output.first] = store->printStorePath(outPath);
+        for (auto & output : drv.outputs) {
+            auto outPath = store->makeOutputPath(output.first, h, drv.name);
+            output.second = { .output = DerivationOutputInputAddressed { .path = outPath } };
+            drv.env[output.first] = store->printStorePath(outPath);
+        }
    }

    auto shellDrvPath = writeDerivation(*store, drv);
@@ -162,8 +171,7 @@ StorePath getDerivationEnvironment(ref<Store> store, const StorePath & drvPath)
    /* Build the derivation. */
    store->buildPaths({DerivedPath::Built{shellDrvPath}});

-    for (auto & [_0, outputAndOptPath] : drv.outputsAndOptPaths(*store)) {
-        auto & [_1, optPath] = outputAndOptPath;
+    for (auto & [_0, optPath] : store->queryPartialDerivationOutputMap(shellDrvPath)) {
        assert(optPath);
        auto & outPath = *optPath;
        assert(store->isValidPath(outPath));
@@ -185,6 +193,7 @@ struct Common : InstallableCommand, MixProfile
        "NIX_BUILD_TOP",
        "NIX_ENFORCE_PURITY",
        "NIX_LOG_FD",
+        "NIX_REMOTE",
        "PPID",
        "PWD",
        "SHELLOPTS",
--- a/src/nix/profile-remove.md
+++ b/src/nix/profile-remove.md
@@ -15,6 +15,7 @@ R""(
  ```

 * Remove all packages:
+
  ```console
  # nix profile remove '.*'
  ```
--- a/src/nix/repl.cc
+++ b/src/nix/repl.cc
@@ -104,6 +104,26 @@ NixRepl::~NixRepl()
    write_history(historyFile.c_str());
 }

+string runNix(Path program, const Strings & args,
+    const std::optional<std::string> & input = {})
+{
+    auto experimentalFeatures = concatStringsSep(" ", settings.experimentalFeatures.get());
+    auto nixConf = getEnv("NIX_CONFIG").value_or("");
+    nixConf.append("\nexperimental-features = " + experimentalFeatures);
+    auto subprocessEnv = getEnv();
+    subprocessEnv["NIX_CONFIG"] = nixConf;
+    RunOptions opts(settings.nixBinDir+ "/" + program, args);
+    opts.input = input;
+    opts.environment = subprocessEnv;
+
+    auto res = runProgram(opts);
+
+    if (!statusOk(res.first))
+        throw ExecError(res.first, fmt("program '%1%' %2%", program, statusToString(res.first)));
+
+    return res.second;
+}
+
 static NixRepl * curRepl; // ugly

 static char * completionCallback(char * s, int *match) {
@@ -463,7 +483,7 @@ bool NixRepl::processLine(string line)
        state->callFunction(f, v, result, Pos());

        StorePath drvPath = getDerivationPath(result);
-        runProgram(settings.nixBinDir + "/nix-shell", true, {state->store->printStorePath(drvPath)});
+        runNix("nix-shell", {state->store->printStorePath(drvPath)});
    }

    else if (command == ":b" || command == ":i" || command == ":s") {
@@ -477,7 +497,7 @@ bool NixRepl::processLine(string line)
               but doing it in a child makes it easier to recover from
               problems / SIGINT. */
            try {
-                runProgram(settings.nixBinDir + "/nix", true, {"build", "--no-link", drvPathRaw});
+                runNix("nix", {"build", "--no-link", drvPathRaw});
                auto drv = state->store->readDerivation(drvPath);
                std::cout << std::endl << "this derivation produced the following outputs:" << std::endl;
                for (auto & i : drv.outputsAndOptPaths(*state->store))
@@ -485,9 +505,9 @@ bool NixRepl::processLine(string line)
            } catch (ExecError &) {
            }
        } else if (command == ":i") {
-            runProgram(settings.nixBinDir + "/nix-env", true, {"-i", drvPathRaw});
+            runNix("nix-env", {"-i", drvPathRaw});
        } else {
-            runProgram(settings.nixBinDir + "/nix-shell", true, {drvPathRaw});
+            runNix("nix-shell", {drvPathRaw});
        }
    }

--- a/src/nix/run.cc
+++ b/src/nix/run.cc
@@ -43,8 +43,8 @@ struct RunCommon : virtual Command
           helper program (chrootHelper() below) to do the work. */
        auto store2 = store.dynamic_pointer_cast<LocalStore>();

-        if (store2 && store->storeDir != store2->realStoreDir) {
-            Strings helperArgs = { chrootHelperName, store->storeDir, store2->realStoreDir, program };
+        if (store2 && store->storeDir != store2->getRealStoreDir()) {
+            Strings helperArgs = { chrootHelperName, store->storeDir, store2->getRealStoreDir(), program };
            for (auto & arg : args) helperArgs.push_back(arg);

            execv(readLink("/proc/self/exe").c_str(), stringsToCharPtrs(helperArgs).data());
--- a/tests/build-remote-content-addressed-floating.sh
+++ b/tests/build-remote-content-addressed-floating.sh
@@ -4,4 +4,6 @@ file=build-hook-ca-floating.nix

 sed -i 's/experimental-features .*/& ca-derivations/' "$NIX_CONF_DIR"/nix.conf

+CONTENT_ADDRESSED=true
+
 source build-remote.sh
--- a/tests/build-remote.sh
+++ b/tests/build-remote.sh
@@ -6,12 +6,17 @@ unset NIX_STATE_DIR

 function join_by { local d=$1; shift; echo -n "$1"; shift; printf "%s" "${@/#/$d}"; }

+EXTRA_SYSTEM_FEATURES=()
+if [[ -n "$CONTENT_ADDRESSED" ]]; then
+    EXTRA_SYSTEM_FEATURES=("ca-derivations")
+fi
+
 builders=(
  # system-features will automatically be added to the outer URL, but not inner
  # remote-store URL.
-  "ssh://localhost?remote-store=$TEST_ROOT/machine1?system-features=foo - - 1 1 foo"
-  "$TEST_ROOT/machine2 - - 1 1 bar"
-  "ssh-ng://localhost?remote-store=$TEST_ROOT/machine3?system-features=baz - - 1 1 baz"
+  "ssh://localhost?remote-store=$TEST_ROOT/machine1?system-features=$(join_by "%20" foo ${EXTRA_SYSTEM_FEATURES[@]}) - - 1 1 $(join_by "," foo ${EXTRA_SYSTEM_FEATURES[@]})"
+  "$TEST_ROOT/machine2 - - 1 1 $(join_by "," bar ${EXTRA_SYSTEM_FEATURES[@]})"
+  "ssh-ng://localhost?remote-store=$TEST_ROOT/machine3?system-features=$(join_by "%20" baz ${EXTRA_SYSTEM_FEATURES[@]}) - - 1 1 $(join_by "," baz ${EXTRA_SYSTEM_FEATURES[@]})"
 )

 chmod -R +w $TEST_ROOT/machine* || true
--- a/tests/build.sh
+++ b/tests/build.sh
@@ -1,7 +1,7 @@
 source common.sh

 expectedJSONRegex='\[\{"drvPath":".*multiple-outputs-a.drv","outputs":\{"first":".*multiple-outputs-a-first","second":".*multiple-outputs-a-second"}},\{"drvPath":".*multiple-outputs-b.drv","outputs":\{"out":".*multiple-outputs-b"}}]'
-nix build -f multiple-outputs.nix --json a.all b.all | jq --exit-status '
+nix build -f multiple-outputs.nix --json a.all b.all --no-link | jq --exit-status '
  (.[0] |
    (.drvPath | match(".*multiple-outputs-a.drv")) and
    (.outputs.first | match(".*multiple-outputs-a-first")) and
@@ -10,10 +10,10 @@ nix build -f multiple-outputs.nix --json a.all b.all | jq --exit-status '
    (.drvPath | match(".*multiple-outputs-b.drv")) and
    (.outputs.out | match(".*multiple-outputs-b")))
 '
-
 testNormalization () {
    clearStore
-    outPath=$(nix-build ./simple.nix)
+    outPath=$(nix-build ./simple.nix --no-out-link)
    test "$(stat -c %Y $outPath)" -eq 1
 }
+
 testNormalization
--- a/tests/ca/build-with-garbage-path.sh
+++ b/tests/ca/build-with-garbage-path.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+# Regression test for https://github.com/NixOS/nix/issues/4858
+
+source common.sh
+sed -i 's/experimental-features .*/& ca-derivations ca-references/' "$NIX_CONF_DIR"/nix.conf
+
+# Get the output path of `rootCA`, and put some garbage instead
+outPath="$(nix-build ./content-addressed.nix -A rootCA --no-out-link)"
+nix-store --delete "$outPath"
+touch "$outPath"
+
+# The build should correctly remove the garbage and put the expected path instead
+nix-build ./content-addressed.nix -A rootCA --no-out-link
+
+# Rebuild it. This shouldn’t overwrite the existing path
+oldInode=$(stat -c '%i' "$outPath")
+nix-build ./content-addressed.nix -A rootCA --no-out-link --arg seed 2
+newInode=$(stat -c '%i' "$outPath")
+[[ "$oldInode" == "$newInode" ]]
--- a/tests/ca/duplicate-realisation-in-closure.sh
+++ b/tests/ca/duplicate-realisation-in-closure.sh
@@ -0,0 +1,26 @@
+source ./common.sh
+
+sed -i 's/experimental-features .*/& ca-derivations ca-references/' "$NIX_CONF_DIR"/nix.conf
+
+export REMOTE_STORE_DIR="$TEST_ROOT/remote_store"
+export REMOTE_STORE="file://$REMOTE_STORE_DIR"
+
+rm -rf $REMOTE_STORE_DIR
+clearStore
+
+# Build dep1 and push that to the binary cache.
+# This entails building (and pushing) current-time.
+nix copy --to "$REMOTE_STORE" -f nondeterministic.nix dep1
+clearStore
+sleep 2 # To make sure that `$(date)` will be different
+# Build dep2.
+# As we’ve cleared the cache, we’ll have to rebuild current-time. And because
+# the current time isn’t the same as before, this will yield a new (different)
+# realisation
+nix build -f nondeterministic.nix dep2 --no-link
+
+# Build something that depends both on dep1 and dep2.
+# If everything goes right, we should rebuild dep2 rather than fetch it from
+# the cache (because that would mean duplicating `current-time` in the closure),
+# and have `dep1 == dep2`.
+nix build --substituters "$REMOTE_STORE" -f nondeterministic.nix toplevel --no-require-sigs --no-link
--- a/tests/ca/gc.sh
+++ b/tests/ca/gc.sh
@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+
+# Ensure that garbage collection works properly with ca derivations
+
+source common.sh
+
+sed -i 's/experimental-features .*/& ca-derivations ca-references/' "$NIX_CONF_DIR"/nix.conf
+
+export NIX_TESTS_CA_BY_DEFAULT=1
+
+cd ..
+source gc.sh
--- a/tests/ca/nix-shell.sh
+++ b/tests/ca/nix-shell.sh
@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+source common.sh
+
+sed -i 's/experimental-features .*/& ca-derivations ca-references nix-command flakes/' "$NIX_CONF_DIR"/nix.conf
+
+CONTENT_ADDRESSED=true
+cd ..
+source ./nix-shell.sh
+
--- a/tests/ca/nondeterministic.nix
+++ b/tests/ca/nondeterministic.nix
@@ -0,0 +1,35 @@
+with import ./config.nix;
+
+let mkCADerivation = args: mkDerivation ({
+    __contentAddressed = true;
+    outputHashMode = "recursive";
+    outputHashAlgo = "sha256";
+} // args);
+in
+
+rec {
+  currentTime = mkCADerivation {
+    name = "current-time";
+    buildCommand = ''
+      mkdir $out
+      echo $(date) > $out/current-time
+    '';
+  };
+  dep = seed: mkCADerivation {
+    name = "dep";
+    inherit seed;
+    buildCommand = ''
+      echo ${currentTime} > $out
+    '';
+  };
+  dep1 = dep 1;
+  dep2 = dep 2;
+  toplevel = mkCADerivation {
+    name = "toplevel";
+    buildCommand = ''
+      test ${dep1} == ${dep2}
+      touch $out
+    '';
+  };
+}
+
--- a/tests/ca/post-hook.sh
+++ b/tests/ca/post-hook.sh
@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+
+source common.sh
+
+sed -i 's/experimental-features .*/& ca-derivations ca-references nix-command flakes/' "$NIX_CONF_DIR"/nix.conf
+
+export NIX_TESTS_CA_BY_DEFAULT=1
+cd ..
+source ./post-hook.sh
+
+
--- a/tests/ca/recursive.sh
+++ b/tests/ca/recursive.sh
@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+
+source common.sh
+
+sed -i 's/experimental-features .*/& ca-derivations ca-references nix-command flakes/' "$NIX_CONF_DIR"/nix.conf
+
+export NIX_TESTS_CA_BY_DEFAULT=1
+cd ..
+source ./recursive.sh
+
+
--- a/tests/ca/substitute.sh
+++ b/tests/ca/substitute.sh
@@ -17,11 +17,15 @@ buildDrvs () {

 # Populate the remote cache
 clearStore
-buildDrvs --post-build-hook ../push-to-store.sh
+nix copy --to $REMOTE_STORE --file ./content-addressed.nix

 # Restart the build on an empty store, ensuring that we don't build
 clearStore
-buildDrvs --substitute --substituters $REMOTE_STORE --no-require-sigs -j0
+buildDrvs --substitute --substituters $REMOTE_STORE --no-require-sigs -j0 transitivelyDependentCA
+# Check that the thing we’ve just substituted has its realisation stored
+nix realisation info --file ./content-addressed.nix transitivelyDependentCA
+# Check that its dependencies have it too
+nix realisation info --file ./content-addressed.nix dependentCA rootCA

 # Same thing, but
 # 1. With non-ca derivations
--- a/tests/config.nix.in
+++ b/tests/config.nix.in
@@ -1,3 +1,12 @@
+let
+  contentAddressedByDefault = builtins.getEnv "NIX_TESTS_CA_BY_DEFAULT" == "1";
+  caArgs = if contentAddressedByDefault then {
+    __contentAddressed = true;
+    outputHashMode = "recursive";
+    outputHashAlgo = "sha256";
+  } else {};
+in
+
 rec {
  shell = "@bash@";

@@ -13,6 +22,6 @@ rec {
      builder = shell;
      args = ["-e" args.builder or (builtins.toFile "builder-${args.name}.sh" "if [ -e .attrs.sh ]; then source .attrs.sh; fi; eval \"$buildCommand\"")];
      PATH = path;
-    } // removeAttrs args ["builder" "meta"])
+    } // caArgs // removeAttrs args ["builder" "meta"])
    // { meta = args.meta or {}; };
 }
--- a/tests/gc.sh
+++ b/tests/gc.sh
@@ -12,7 +12,7 @@ ln -sf $outPath "$NIX_STATE_DIR"/gcroots/foo
 nix-store --gc --print-roots | grep $outPath
 nix-store --gc --print-live | grep $outPath
 nix-store --gc --print-dead | grep $drvPath
-if nix-store --gc --print-dead | grep $outPath; then false; fi
+if nix-store --gc --print-dead | grep -E $outPath$; then false; fi

 nix-store --gc --print-dead

--- a/tests/local.mk
+++ b/tests/local.mk
@@ -2,6 +2,7 @@ nix_tests = \
  hash.sh lang.sh add.sh simple.sh dependencies.sh \
  config.sh \
  gc.sh \
+  ca/gc.sh \
  gc-concurrent.sh \
  gc-auto.sh \
  referrers.sh user-envs.sh logging.sh nix-build.sh misc.sh fixed.sh \
@@ -38,6 +39,7 @@ nix_tests = \
  search.sh \
  nix-copy-ssh.sh \
  post-hook.sh \
+  ca/post-hook.sh \
  function-trace.sh \
  recursive.sh \
  describe-stores.sh \
@@ -45,9 +47,13 @@ nix_tests = \
  build.sh \
  compute-levels.sh \
  ca/build.sh \
+  ca/build-with-garbage-path.sh \
+  ca/duplicate-realisation-in-closure.sh \
  ca/substitute.sh \
  ca/signatures.sh \
+  ca/nix-shell.sh \
  ca/nix-run.sh \
+  ca/recursive.sh \
  ca/nix-copy.sh
  # parallel.sh

--- a/tests/nix-shell.sh
+++ b/tests/nix-shell.sh
@@ -2,6 +2,20 @@ source common.sh

 clearStore

+if [[ -n ${CONTENT_ADDRESSED:-} ]]; then
+    nix-shell () {
+        command nix-shell --arg contentAddressed true "$@"
+    }
+
+    nix_develop() {
+        nix develop --arg contentAddressed true "$@"
+    }
+else
+    nix_develop() {
+        nix develop "$@"
+    }
+fi
+
 # Test nix-shell -A
 export IMPURE_VAR=foo
 export SELECTED_IMPURE_VAR=baz
@@ -41,7 +55,7 @@ output=$(NIX_PATH=nixpkgs=shell.nix nix-shell --pure -p foo bar --run 'echo "$(f
 [ "$output" = "foo bar" ]

 # Test nix-shell shebang mode
-sed -e "s|@ENV_PROG@|$(type -p env)|" shell.shebang.sh > $TEST_ROOT/shell.shebang.sh
+sed -e "s|@ENV_PROG@|$(type -P env)|" shell.shebang.sh > $TEST_ROOT/shell.shebang.sh
 chmod a+rx $TEST_ROOT/shell.shebang.sh

 output=$($TEST_ROOT/shell.shebang.sh abc def)
@@ -49,7 +63,7 @@ output=$($TEST_ROOT/shell.shebang.sh abc def)

 # Test nix-shell shebang mode again with metacharacters in the filename.
 # First word of filename is chosen to not match any file in the test root.
-sed -e "s|@ENV_PROG@|$(type -p env)|" shell.shebang.sh > $TEST_ROOT/spaced\ \\\'\"shell.shebang.sh
+sed -e "s|@ENV_PROG@|$(type -P env)|" shell.shebang.sh > $TEST_ROOT/spaced\ \\\'\"shell.shebang.sh
 chmod a+rx $TEST_ROOT/spaced\ \\\'\"shell.shebang.sh

 output=$($TEST_ROOT/spaced\ \\\'\"shell.shebang.sh abc def)
@@ -58,7 +72,7 @@ output=$($TEST_ROOT/spaced\ \\\'\"shell.shebang.sh abc def)
 # Test nix-shell shebang mode for ruby
 # This uses a fake interpreter that returns the arguments passed
 # This, in turn, verifies the `rc` script is valid and the `load()` script (given using `-e`) is as expected.
-sed -e "s|@SHELL_PROG@|$(type -p nix-shell)|" shell.shebang.rb > $TEST_ROOT/shell.shebang.rb
+sed -e "s|@SHELL_PROG@|$(type -P nix-shell)|" shell.shebang.rb > $TEST_ROOT/shell.shebang.rb
 chmod a+rx $TEST_ROOT/shell.shebang.rb

 output=$($TEST_ROOT/shell.shebang.rb abc ruby)
@@ -66,20 +80,20 @@ output=$($TEST_ROOT/shell.shebang.rb abc ruby)

 # Test nix-shell shebang mode for ruby again with metacharacters in the filename.
 # Note: fake interpreter only space-separates args without adding escapes to its output.
-sed -e "s|@SHELL_PROG@|$(type -p nix-shell)|" shell.shebang.rb > $TEST_ROOT/spaced\ \\\'\"shell.shebang.rb
+sed -e "s|@SHELL_PROG@|$(type -P nix-shell)|" shell.shebang.rb > $TEST_ROOT/spaced\ \\\'\"shell.shebang.rb
 chmod a+rx $TEST_ROOT/spaced\ \\\'\"shell.shebang.rb

 output=$($TEST_ROOT/spaced\ \\\'\"shell.shebang.rb abc ruby)
 [ "$output" = '-e load(ARGV.shift) -- '"$TEST_ROOT"'/spaced \'\''"shell.shebang.rb abc ruby' ]

 # Test 'nix develop'.
-nix develop -f shell.nix shellDrv -c bash -c '[[ -n $stdenv ]]'
+nix_develop -f shell.nix shellDrv -c bash -c '[[ -n $stdenv ]]'

 # Ensure `nix develop -c` preserves stdin
 echo foo | nix develop -f shell.nix shellDrv -c cat | grep -q foo

 # Ensure `nix develop -c` actually executes the command if stdout isn't a terminal
-nix develop -f shell.nix shellDrv -c echo foo |& grep -q foo
+nix_develop -f shell.nix shellDrv -c echo foo |& grep -q foo

 # Test 'nix print-dev-env'.
 source <(nix print-dev-env -f shell.nix shellDrv)
--- a/tests/post-hook.sh
+++ b/tests/post-hook.sh
@@ -4,7 +4,7 @@ clearStore

 rm -f $TEST_ROOT/result

-export REMOTE_STORE=$TEST_ROOT/remote_store
+export REMOTE_STORE=file:$TEST_ROOT/remote_store

 # Build the dependencies and push them to the remote store
 nix-build -o $TEST_ROOT/result dependencies.nix --post-build-hook $PWD/push-to-store.sh
--- a/tests/recursive.sh
+++ b/tests/recursive.sh
@@ -9,9 +9,9 @@ rm -f $TEST_ROOT/result

 export unreachable=$(nix store add-path ./recursive.sh)

-NIX_BIN_DIR=$(dirname $(type -p nix)) nix --experimental-features 'nix-command recursive-nix' build -o $TEST_ROOT/result -L --impure --expr '
+NIX_BIN_DIR=$(dirname $(type -p nix)) nix --extra-experimental-features 'nix-command recursive-nix' build -o $TEST_ROOT/result -L --impure --expr '
  with import ./config.nix;
-  mkDerivation {
+  mkDerivation rec {
    name = "recursive";
    dummy = builtins.toFile "dummy" "bla bla";
    SHELL = shell;
@@ -19,11 +19,13 @@ NIX_BIN_DIR=$(dirname $(type -p nix)) nix --experimental-features 'nix-command r
    # Note: this is a string without context.
    unreachable = builtins.getEnv "unreachable";

+    NIX_TESTS_CA_BY_DEFAULT = builtins.getEnv "NIX_TESTS_CA_BY_DEFAULT";
+
    requiredSystemFeatures = [ "recursive-nix" ];

    buildCommand = '\'\''
      mkdir $out
-      opts="--experimental-features nix-command"
+      opts="--experimental-features nix-command ${if (NIX_TESTS_CA_BY_DEFAULT == "1") then "--extra-experimental-features ca-derivations" else ""}"

      PATH=${builtins.getEnv "NIX_BIN_DIR"}:$PATH

@@ -46,16 +48,15 @@ NIX_BIN_DIR=$(dirname $(type -p nix)) nix --experimental-features 'nix-command r
      # Add it to our closure.
      ln -s $foobar $out/foobar

-      [[ $(nix $opts path-info --all | wc -l) -eq 3 ]]
+      [[ $(nix $opts path-info --all | wc -l) -eq 4 ]]

      # Build a derivation.
      nix $opts build -L --impure --expr '\''
-        derivation {
+        with import ${./config.nix};
+        mkDerivation {
          name = "inner1";
-          builder = builtins.getEnv "SHELL";
-          system = builtins.getEnv "system";
+          buildCommand = "echo $fnord blaat > $out";
          fnord = builtins.toFile "fnord" "fnord";
-          args = [ "-c" "echo $fnord blaat > $out" ];
        }
      '\''

--- a/tests/shell.nix
+++ b/tests/shell.nix
@@ -1,6 +1,18 @@
-{ inNixShell ? false }:
+{ inNixShell ? false, contentAddressed ? false }:

-with import ./config.nix;
+let cfg = import ./config.nix; in
+with cfg;
+
+let
+  mkDerivation =
+    if contentAddressed then
+      args: cfg.mkDerivation ({
+        __contentAddressed = true;
+        outputHashMode = "recursive";
+        outputHashAlgo = "sha256";
+      } // args)
+    else cfg.mkDerivation;
+in

 let pkgs = rec {
  setupSh = builtins.toFile "setup" ''
Author	SHA1	Message	Date
regnat	7daf0c6ef1	Forward the experimental features to the nix repl subprocesses Pass the current experimental features using `NIX_CONFIG` to the various Nix subprocesses that `nix repl` invokes. This is quite a hack, but having `nix repl` call Nix with a subprocess is a hack already, so I guess that’s fine.	2021-06-29 14:52:46 +02:00
Domen Kožar	81535022dc	Merge pull request #2667 from matthewbauer/cross-jobs Add armv6l-linux & armv7l-linux as cross jobs	2021-06-29 11:39:49 +02:00
Domen Kožar	9feca5cdf6	github actions: simplify getting the system logic	2021-06-28 23:02:53 +02:00
Matthew Bauer	6c13a3f735	Support binaryTarballCross in gha	2021-06-28 15:08:17 -05:00
Domen Kožar	4d058f49e7	Merge pull request #4954 from domenkozar/bump-nixpkgs flake.lock: Update	2021-06-28 19:33:55 +02:00
Domen Kožar	777c688a98	flake.lock: Update Flake input changes: * Updated 'nixpkgs': 'github:NixOS/nixpkgs/bb8a5e54845012ed1375ffd5f317d2fdf434b20e' -> 'github:NixOS/nixpkgs/f77036342e2b690c61c97202bf48f2ce13acc022'	2021-06-28 18:42:44 +02:00
Eelco Dolstra	c189d80b4a	Merge pull request #4952 from FlorianFranzen/patch-1 man: fix formatting of nix3-profile-remove	2021-06-28 17:05:22 +02:00
Florian Franzen	decc14d4b7	man: fix formatting of nix3-profile-remove	2021-06-28 16:27:03 +02:00
Eelco Dolstra	6182ae6898	Merge pull request #4942 from NixOS/ca/remove-lock-files Eventually delete the CA paths lock files	2021-06-28 16:12:19 +02:00
Eelco Dolstra	f5320299dd	Merge pull request #4937 from NixOS/ca/make-the-tests-useful Make the CA tests actually test something	2021-06-28 16:06:49 +02:00
Matthew Bauer	580583e0b3	Build cross-compilation in gha	2021-06-26 00:14:54 -05:00
Matthew Bauer	c906d6530d	Support cross-compiling binaryTarball	2021-06-26 00:12:03 -05:00
Matthew Bauer	c3a929349f	Merge remote-tracking branch 'origin/master' into cross-jobs	2021-06-25 15:51:02 -05:00
Eelco Dolstra	bf68c693dc	tests: Get rid of some result symlinks Fixes error: cannot create symlink '/home/eelco/Dev/nix/tests/result'; already exists	2021-06-25 11:17:19 +02:00
regnat	8b6fba2b63	Eventually delete the CA paths lock files Mark the lockfiles as having to eventually be deleted so that they don’t stay laying around in the store at the end of the build Fix #4936	2021-06-24 15:45:05 +02:00
Eelco Dolstra	08270af7fe	Merge pull request #4939 from NixOS/ca/recursive-nix Make CA derivations compatible with recursive Nix	2021-06-24 15:02:12 +02:00
regnat	7746cb13dc	Make CA derivations compatible with recursive Nix Add an access-control list to the realisations in recursive-nix (similar to the already existing one for store paths), so that we can build content-addressed derivations in the restricted store. Fix #4353	2021-06-24 14:53:10 +02:00
Eelco Dolstra	8eaf03bcb4	Merge pull request #4940 from NixOS/ca/unresolved-post-build-hook Make the post-build-hook also run for unresolved CA derivations	2021-06-24 12:16:52 +02:00
regnat	be7a4a6a13	Make the post-build-hook also run for unresolved CA derivations Fix #4837	2021-06-24 11:41:57 +02:00
regnat	01a3f4d7ec	Fix the CA gc test Broken by https://github.com/NixOS/nix/issues/4936	2021-06-23 17:37:29 +02:00
regnat	fd3f5e9085	Make the CA tests actuall test CA derivations Fix a mistake in config.nix that was preventing `NIX_TESTS_CA_BY_DEFAULT` from having any meaningful effect	2021-06-23 17:36:50 +02:00
Eelco Dolstra	323e5450a1	Merge branch 'update-nix_path-url' of https://github.com/kini/nix	2021-06-23 15:34:06 +02:00
Eelco Dolstra	0a535dd5ac	Merge pull request #4839 from NixOS/ca/gracefully-handle-duplicate-realisations Gracefully handle duplicate realisations	2021-06-23 11:50:18 +02:00
Eelco Dolstra	f9f773b332	Merge pull request #4908 from NixOS/ca/fix-nix-develop Make `nix develop` work with CA derivations	2021-06-23 11:27:58 +02:00
regnat	c878cee895	Assert that compatible realisations have the same dependencies Should always hold, but that’s not necessarily obvious, so better enforce it	2021-06-23 11:27:16 +02:00
regnat	16fb7d8d95	Display the diverging paths in case of a realisation mismatch	2021-06-23 11:27:16 +02:00
regnat	40f925b2da	Fix indentation	2021-06-23 11:27:16 +02:00
regnat	d32cf0c17a	Gracefully ignore a substituter if it holds an incompatible realisation	2021-06-23 11:27:16 +02:00
regnat	b8f7177a7b	Properly fail when trying to register an incoherent realisation	2021-06-23 11:27:14 +02:00
Eelco Dolstra	7945055c63	Merge pull request #4842 from NixOS/ca/fix-nix-shell Make `nix-shell` support content-addressed derivations	2021-06-23 11:26:22 +02:00
regnat	a5df669bc6	Add a test for the “two glibc” issue	2021-06-23 11:18:31 +02:00
Eelco Dolstra	4a5aa1dbf6	Merge pull request #4838 from NixOS/ca/recursively-substitute-realisations Recursively substitute the realisations	2021-06-23 10:33:25 +02:00
Eelco Dolstra	2ab7c821f3	Merge pull request #4911 from matthewbauer/fix-zsh-completion Fix zsh completion script	2021-06-23 10:07:36 +02:00
Eelco Dolstra	26d2c62225	Merge pull request #4906 from NixOS/collect-garbage-ca Make `computeFSClosure` ca-aware	2021-06-23 10:07:08 +02:00
Eelco Dolstra	d9a43d3137	Merge pull request #4905 from NixOS/ca-derivations-machine-feature Add a ca-derivations required machine feature	2021-06-23 10:05:53 +02:00
Eelco Dolstra	3e4126b67c	Merge pull request #4926 from NixOS/ca/build-remote-signal-6 Fix the remote build of CA derivations	2021-06-23 10:04:35 +02:00
Eelco Dolstra	4f9508c3b5	Merge pull request #4836 from NixOS/ca/track-drvoutput-dependencies-2-le-retour Track the dependencies of CA realisations	2021-06-23 10:03:31 +02:00
regnat	ed0e21a88d	Fix indentation	2021-06-23 08:16:34 +02:00
regnat	7c96a76dd7	Reformat the sql statements	2021-06-23 08:16:34 +02:00
Théophane Hufschmitt	8d09a4f9a0	Remove a useless string split Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2021-06-23 08:16:34 +02:00
Théophane Hufschmitt	c13d7d0b97	Pass more values by reference Rather than copying them around everywhere Co-authored-by: Eelco Dolstra <edolstra@gmail.com>	2021-06-23 08:16:34 +02:00
regnat	3b58dbb356	nix-shell: Replace resolving failure error by an assertion This shouldn’t happen in practice, so better make it explicit	2021-06-22 11:29:55 +02:00
Domen Kožar	4b23bf797a	Merge pull request #4930 from domenkozar/perl-binding-aarch64-darwin perlBindings: fix build on aarch64-darwin	2021-06-22 10:43:00 +02:00
Domen Kožar	9676c9f6a3	perlBindings: fix build on aarch64-darwin	2021-06-21 19:40:51 +02:00
regnat	dcabb46124	Shorten a stupidly long sql query name	2021-06-21 16:28:06 +02:00
Eelco Dolstra	610baf359a	Merge pull request #4833 from NixOS/ca/json-realisations-in-worker-protocol Always send the realisations as JSON	2021-06-21 16:12:17 +02:00
Eelco Dolstra	24e7353232	Merge pull request #4928 from NixOS/ca/remove-existing-invalid-store-path Remove a possible existing store path when building CA derivations	2021-06-21 16:09:21 +02:00
Eelco Dolstra	55b4623d21	Merge pull request #4929 from NixOS/only-symlink-wanted-outputs Only symlink the requested outputs in `nix build`	2021-06-21 16:07:33 +02:00
Théophane Hufschmitt	ce674cb2cf	Properly set the output env variables Co-authored-by: John Ericson <git@JohnEricson.me>	2021-06-21 15:52:01 +02:00
regnat	608434722b	Only symlink the requested outputs in `nix build` Fix #4925	2021-06-21 15:47:47 +02:00
regnat	3784c66a46	Remove a possible existing store path when building CA derivations In case a previous interrupted build left a garbage path laying around, remove it before trying to move the path to its final location. Fix #4858	2021-06-21 15:29:15 +02:00
regnat	498677cbed	Fix the remote build of CA derivations Make sure that the derivation we send to the remote builder is exactly the one that we want to build locally so that the output ids are exactly the same Fix #4845	2021-06-21 14:18:33 +02:00
Eelco Dolstra	db3de0727e	Merge pull request #4923 from edolstra/uds-store-root-param UDSRemoteStore: Support the 'root' store parameter	2021-06-21 10:19:57 +02:00
Eelco Dolstra	4202a3bc4e	UDSRemoteStore: Support the 'root' store parameter Useful when we're using a daemon with a chroot store, e.g. $ NIX_DAEMON_SOCKET_PATH=/tmp/chroot/nix/var/nix/daemon-socket/socket nix-daemon --store /tmp/chroot Then the client can now connect with $ nix build --store unix:///tmp/chroot/nix/var/nix/daemon-socket/socket?root=/tmp/chroot nixpkgs#hello	2021-06-18 17:04:11 +02:00
regnat	a3ce88725b	Add a test for the gc with CA derivations Also add a small architecture to easily run CA-enabled tests	2021-06-15 12:11:31 +02:00
Eelco Dolstra	e6150de90d	nix develop: Filter out NIX_REMOTE When recursive Nix is enabled, NIX_REMOTE is set to unix:///build/.nix-socket, which doesn't work outside of the sandbox.	2021-06-15 12:06:01 +02:00
Matthew Bauer	79674c6cdb	Fix zsh completion script Installed site-functions need to be run directly, not via compdef.	2021-06-12 23:50:26 -05:00
regnat	96d7170e12	Don’t check the `deriver` field on computeFSClosure That doesn’t really make sense with CA derivations (and wasn’t even really correct before because of FO derivations, though that probably didn’t matter much in practice)	2021-06-12 12:24:53 +02:00
regnat	2cf591a134	Make `nix develop` work with CA derivations Fix #4823	2021-06-11 13:35:13 +02:00
regnat	56605b4688	Make `nix-shell` support content-addressed derivations Resolve the derivation before trying to load its environment − essentially reproducing what the build loop does − so that we can effectively access our dependencies (and not just their placeholders). Fix #4821	2021-06-11 13:32:49 +02:00
regnat	7ac038fa4b	Make `computeFSClosure` ca-aware Fix #4820 by preventing nix-collect garbage from crashing if `keep-outputs` or `keep-derivations` is true	2021-06-11 09:26:49 +02:00
regnat	7c077d2a0f	Add a ca-derivations required machine feature Make ca-derivations require a `ca-derivations` machine feature, and ca-aware builders expose it. That way, a network of builders can mix ca-aware and non-ca-aware machines, and the scheduler will send them in the right place.	2021-06-11 09:12:53 +02:00
Eelco Dolstra	8e6ee1b9e9	Merge pull request #4889 from puckipedia/fix-atomic-sunos configure.ac: fix use of unread LIBS variable	2021-06-04 14:18:35 +02:00
Puck Meerburg	196b77b686	configure.ac: fix use of unread LIBS variable This fixes both the SunOS/Solaris check, and the libatomic check, which reference $LIBS, which has not been used since automake was stripped out of the code.	2021-06-04 11:25:36 +00:00
Keshav Kini	f35f9af787	Improve explanation of NIX_PATH prefix syntax The previous wording seemed to imply that the "channel:" syntax would resolve to a github archive URL, which is not the case.	2021-06-03 17:59:39 -07:00
regnat	a22755721b	Recursively substitute the realisations Make sure that whenever we substitute a realisation, we also substitute its entire closure	2021-05-26 18:44:17 +02:00
regnat	1f3ff0d193	Aso track the output path of the realisation dependencies	2021-05-26 17:09:21 +02:00
regnat	ce1a6c6b13	Merge branch 'ca/track-drvoutput-dependencies-2-le-retour' into ca/recursively-substitute-realisations	2021-05-26 17:09:21 +02:00
regnat	cb46d70794	Add a db migration script	2021-05-26 16:59:09 +02:00
regnat	63ebfc73c5	Make `copyPaths` copy the whole realisations closure Otherwise registering the realisations on the remote side might fail as it now expects a complete closure	2021-05-26 16:59:09 +02:00
regnat	8c30acc3e8	Properly track the drvoutput references when building	2021-05-26 16:59:09 +02:00
regnat	af3afd25ea	Add a method to compute the closure of a realisation Only considers the closure in term of `Realisation`, ignores all the opaque inputs. Dunno whether that’s the nicest solution, need to think it through a bit	2021-05-26 16:59:09 +02:00
regnat	eca6ff06d6	Store the realisation deps on the local store	2021-05-26 16:59:09 +02:00
regnat	7ce0441d80	Add a dependencies field to DrvOutputInfo Currently never used, nor set but will be useful shortly	2021-05-26 16:59:09 +02:00
regnat	4077d55775	Merge branch 'ca/json-realisations-in-worker-protocol'	2021-05-26 16:59:09 +02:00
regnat	7616268812	Always send the realisations as JSON Align all the worker protocol with `buildDerivation` which inlines the realisations as one opaque json blob. That way we don’t have to bother changing the remote store protocol when the definition of `Realisation` changes, as long as we keep the json backwards-compatible	2021-05-26 16:59:09 +02:00
regnat	a8416866cf	Always send the realisations as JSON Align all the worker protocol with `buildDerivation` which inlines the realisations as one opaque json blob. That way we don’t have to bother changing the remote store protocol when the definition of `Realisation` changes, as long as we keep the json backwards-compatible	2021-05-19 11:45:16 +02:00
Matthew Bauer	d5fd0f4745	Merge branch 'master' into cross-jobs	2021-03-09 11:40:16 -06:00
Matthew Bauer	4fefe26717	Re-enable armv6l support This fixes the libatomic detection.	2021-02-05 18:22:34 -06:00
Matthew Bauer	c6d878609d	Merge remote-tracking branch 'origin/master' into cross-jobs	2021-02-05 18:07:48 -06:00
Matthew Bauer	f6ea56dfac	Get shellcheck from buildPackages	2019-02-06 23:04:40 -05:00
Matthew Bauer	e9072ded97	Use nativeBuildInputs	2019-02-06 22:43:28 -05:00
Matthew Bauer	1996af425a	Use buildPackages for native dependencies Unfortunately, releaseTools.nixBuild does not separate native and non-native build inputs. As an alternative, we can just use buildPackages to get the native version of some packages like: - pkgconfig - git - curl - utillinux	2019-02-06 21:43:47 -05:00
Matthew Bauer	15833516a4	Add armv6l-linux & armv7l-linux as cross jobs This is a cheap way to get 32-bit ARM working. We don’t support it officially but lots of people have raspberry pis and similar hardware they want to install the Nix package manager on.	2019-02-05 16:42:45 -05:00