downloader: when retry but can't resume, retry the whole file

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,27 @@
+<!--
+
+# Filing a Nix issue
+
+*WAIT* Are you sure you're filing your issue in the right repository?
+
+We appreciate you taking the time to tell us about issues you encounter, but routing the issue to the right place will get you help sooner and save everyone time.
+
+This is the Nix repository, and issues here should be about Nix the build and package management *_tool_*.
+
+If you have a problem with a specific package on NixOS or when using Nix, you probably want to file an issue with _nixpkgs_, whose issue tracker is over at https://github.com/NixOS/nixpkgs/issues.
+
+Examples of _Nix_ issues:
+
+- Nix segfaults when I run `nix-build -A blahblah`
+- The Nix language needs a new builtin: `builtins.foobar`
+- Regression in the behavior of `nix-env` in Nix 2.0
+
+Examples of _nixpkgs_ issues:
+
+- glibc is b0rked on aarch64
+- chromium in NixOS doesn't support U2F but google-chrome does!
+- The OpenJDK package on macOS is missing a key component
+
+Chances are if you're a newcomer to the Nix world, you'll probably want the [nixpkgs tracker](https://github.com/NixOS/nixpkgs/issues). It also gets a lot more eyeball traffic so you'll probably get a response a lot more quickly.
+
+-->

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,32 +0,0 @@
----
-name: Bug report
-about: Create a report to help us improve
-title: ''
-labels: bug
-assignees: ''
-
----
-
-**Describe the bug**
-
-A clear and concise description of what the bug is.
-
-If you have a problem with a specific package or NixOS,
-you probably want to file an issue at https://github.com/NixOS/nixpkgs/issues. 
-
-**Steps To Reproduce**
-
-1. Go to '...'
-2. Click on '....'
-3. Scroll down to '....'
-4. See error
-
-**Expected behavior**
-
-A clear and concise description of what you expected to happen.
-
-**`nix-env --version` output**
-
-**Additional context**
-
-Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,20 +0,0 @@
----
-name: Feature request
-about: Suggest an idea for this project
-title: ''
-labels: improvement
-assignees: ''
-
----
-
-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
-
-**Describe the solution you'd like**
-A clear and concise description of what you want to happen.
-
-**Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
-
-**Additional context**
-Add any other context or screenshots about the feature request here.

.github/workflows/test.yml

@@ -10,8 +10,5 @@ jobs:
     runs-on: ${{ matrix.os }}
     steps:
     - uses: actions/checkout@v2
-      with:
-        fetch-depth: 0
     - uses: cachix/install-nix-action@v8
-    #- run: nix flake check
-    - run: nix-build -A checks.$(if [[ `uname` = Linux ]]; then echo x86_64-linux; else echo x86_64-darwin; fi)
+    - run: nix-build release.nix --arg nix '{ outPath = ./.; revCount = 123; shortRev = "abcdefgh"; }' --arg systems '[ builtins.currentSystem ]' -A installerScript -A perlBindings

.gitignore

@@ -75,8 +75,6 @@ perl/Makefile.config
 
 /src/nix-copy-closure/nix-copy-closure
 
-/src/error-demo/error-demo
-
 /src/build-remote/build-remote
 
 # /tests/

.travis.yml

@@ -0,0 +1,8 @@
+matrix:
+  include:
+    - language: osx
+      script: ./tests/install-darwin.sh
+    - language: nix
+      script: nix-build release.nix -A build.x86_64-linux
+notifications:
+  email: false

Makefile

@@ -4,14 +4,12 @@ makefiles = \
   nix-rust/local.mk \
   src/libutil/local.mk \
   src/libstore/local.mk \
-  src/libfetchers/local.mk \
   src/libmain/local.mk \
   src/libexpr/local.mk \
   src/nix/local.mk \
   src/resolve-system-dependencies/local.mk \
   scripts/local.mk \
   corepkgs/local.mk \
-  misc/bash/local.mk \
   misc/systemd/local.mk \
   misc/launchd/local.mk \
   misc/upstart/local.mk \

README.md

@@ -1,54 +1,21 @@
-# Nix
-
 [![Open Collective supporters](https://opencollective.com/nixos/tiers/supporter/badge.svg?label=Supporters&color=brightgreen)](https://opencollective.com/nixos)
-[![Test](https://github.com/NixOS/nix/workflows/Test/badge.svg)](https://github.com/NixOS/nix/actions)
 
-Nix is a powerful package manager for Linux and other Unix systems that makes package
-management reliable and reproducible. Please refer to the [Nix manual](https://nixos.org/nix/manual)
-for more details.
+Nix, the purely functional package manager
+------------------------------------------
 
-## Installation
+Nix is a new take on package management that is fairly unique. Because of its
+purity aspects, a lot of issues found in traditional package managers don't
+appear with Nix.
 
-On Linux and macOS the easiest way to Install Nix is to run the following shell command
-(as a user other than root):
+To find out more about the tool, usage and installation instructions, please
+read the manual, which is available on the Nix website at
+<https://nixos.org/nix/manual>.
 
-```
-$ curl -L https://nixos.org/nix/install | sh
-```
+## Contributing
 
-Information on additional installation methods is available on the [Nix download page](https://nixos.org/download.html).
-
-## Building And Developing
-
-### Building Nix
-
-You can build Nix using one of the targets provided by [release.nix](./release.nix):
-
-```
-$ nix-build ./release.nix -A build.aarch64-linux
-$ nix-build ./release.nix -A build.x86_64-darwin
-$ nix-build ./release.nix -A build.i686-linux
-$ nix-build ./release.nix -A build.x86_64-linux
-```
-
-### Development Environment
-
-You can use the provided `shell.nix` to get a working development environment:
-
-```
-$ nix-shell
-$ ./bootstrap.sh
-$ ./configure
-$ make
-```
-
-## Additional Resources
-
-- [Nix manual](https://nixos.org/nix/manual)
-- [Nix jobsets on hydra.nixos.org](https://hydra.nixos.org/project/nix)
-- [NixOS Discourse](https://discourse.nixos.org/)
-- [IRC - #nixos on freenode.net](irc://irc.freenode.net/#nixos)
+Take a look at the [Hacking Section](https://nixos.org/nix/manual/#chap-hacking)
+of the manual. It helps you to get started with building Nix from source.
 
 ## License
 
-Nix is released under the [LGPL v2.1](./COPYING).
+Nix is released under the LGPL v2.1

configure.ac

@@ -123,7 +123,6 @@ AC_PATH_PROG(flex, flex, false)
 AC_PATH_PROG(bison, bison, false)
 AC_PATH_PROG(dot, dot)
 AC_PATH_PROG(lsof, lsof, lsof)
-NEED_PROG(jq, jq)
 
 
 AC_SUBST(coreutils, [$(dirname $(type -p cat))])

default.nix

@@ -1,3 +0,0 @@
-(import (fetchTarball https://github.com/edolstra/flake-compat/archive/master.tar.gz) {
-  src = builtins.fetchGit ./.;
-}).defaultNix

doc/manual/command-ref/conf-file.xml

@@ -19,30 +19,26 @@
 
 <refsection><title>Description</title>
 
-<para>By default Nix reads settings from the following places:</para>
+<para>Nix reads settings from two configuration files:</para>
 
-<para>The system-wide configuration file
-<filename><replaceable>sysconfdir</replaceable>/nix/nix.conf</filename>
-(i.e. <filename>/etc/nix/nix.conf</filename> on most systems), or
-<filename>$NIX_CONF_DIR/nix.conf</filename> if
-<envar>NIX_CONF_DIR</envar> is set. Values loaded in this file are not forwarded to the Nix daemon. The
-client assumes that the daemon has already loaded them.
-</para>
+<itemizedlist>
 
-<para>User-specific configuration files:</para>
+  <listitem>
+    <para>The system-wide configuration file
+    <filename><replaceable>sysconfdir</replaceable>/nix/nix.conf</filename>
+    (i.e. <filename>/etc/nix/nix.conf</filename> on most systems), or
+    <filename>$NIX_CONF_DIR/nix.conf</filename> if
+    <envar>NIX_CONF_DIR</envar> is set.</para>
+  </listitem>
 
-<para>
-  If <envar>NIX_USER_CONF_FILES</envar> is set, then each path separated by
-  <literal>:</literal> will be loaded in reverse order.
-</para>
+  <listitem>
+    <para>The user configuration file
+    <filename>$XDG_CONFIG_HOME/nix/nix.conf</filename>, or
+    <filename>~/.config/nix/nix.conf</filename> if
+    <envar>XDG_CONFIG_HOME</envar> is not set.</para>
+  </listitem>
 
-<para>
-  Otherwise it will look for <filename>nix/nix.conf</filename> files in
-  <envar>XDG_CONFIG_DIRS</envar> and <envar>XDG_CONFIG_HOME</envar>.
-
-  The default location is <filename>$HOME/.config/nix.conf</filename> if
-  those environment variables are unset.
-</para>
+</itemizedlist>
 
 <para>The configuration files consist of
 <literal><replaceable>name</replaceable> =

doc/manual/command-ref/env-common.xml

@@ -33,7 +33,7 @@
 
     will cause Nix to look for paths relative to
     <filename>/home/eelco/Dev</filename> and
-    <filename>/etc/nixos</filename>, in this order.  It is also
+    <filename>/etc/nixos</filename>, in that order.  It is also
     possible to match paths against a prefix.  For example, the value
 
     <screen>
@@ -59,7 +59,7 @@ nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/nixos-15.09.tar.gz</sc
     15.09 channel.</para>
 
     <para>A following shorthand can be used to refer to the official channels:
-
+    
     <screen>nixpkgs=channel:nixos-15.09</screen>
     </para>
 
@@ -137,19 +137,12 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
 
 <varlistentry><term><envar>NIX_CONF_DIR</envar></term>
 
-  <listitem><para>Overrides the location of the system Nix configuration
+  <listitem><para>Overrides the location of the Nix configuration
   directory (default
   <filename><replaceable>prefix</replaceable>/etc/nix</filename>).</para></listitem>
 
 </varlistentry>
 
-<varlistentry><term><envar>NIX_USER_CONF_FILES</envar></term>
-
-  <listitem><para>Overrides the location of the user Nix configuration files
-  to load from (defaults to the XDG spec locations). The variable is treated
-  as a list separated by the <literal>:</literal> token.</para></listitem>
-
-</varlistentry>
 
 <varlistentry><term><envar>TMPDIR</envar></term>
 

doc/manual/expressions/builtins.xml

@@ -422,16 +422,6 @@ stdenv.mkDerivation { … }
             </para>
           </listitem>
         </varlistentry>
-        <varlistentry>
-          <term>submodules</term>
-          <listitem>
-            <para>
-              A Boolean parameter that specifies whether submodules
-              should be checked out. Defaults to
-              <literal>false</literal>.
-            </para>
-          </listitem>
-        </varlistentry>
       </variablelist>
 
       <example>

flake.lock

@@ -1,28 +0,0 @@
-{
-  "nodes": {
-    "nixpkgs": {
-      "info": {
-        "lastModified": 1585405475,
-        "narHash": "sha256-bESW0n4KgPmZ0luxvwJ+UyATrC6iIltVCsGdLiphVeE="
-      },
-      "locked": {
-        "owner": "NixOS",
-        "repo": "nixpkgs",
-        "rev": "b88ff468e9850410070d4e0ccd68c7011f15b2be",
-        "type": "github"
-      },
-      "original": {
-        "id": "nixpkgs",
-        "ref": "nixos-20.03-small",
-        "type": "indirect"
-      }
-    },
-    "root": {
-      "inputs": {
-        "nixpkgs": "nixpkgs"
-      }
-    }
-  },
-  "root": "root",
-  "version": 5
-}

flake.nix

@@ -1,510 +0,0 @@
-{
-  description = "The purely functional package manager";
-
-  edition = 201909; # FIXME: remove
-
-  inputs.nixpkgs.url = "nixpkgs/nixos-20.03-small";
-
-  outputs = { self, nixpkgs }:
-
-    let
-
-      version = builtins.readFile ./.version + versionSuffix;
-      versionSuffix =
-        if officialRelease
-        then ""
-        else "pre${builtins.substring 0 8 (self.lastModifiedDate or self.lastModified)}_${self.shortRev or "dirty"}";
-
-      officialRelease = false;
-
-      systems = [ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ];
-
-      forAllSystems = f: nixpkgs.lib.genAttrs systems (system: f system);
-
-      # Memoize nixpkgs for different platforms for efficiency.
-      nixpkgsFor = forAllSystems (system:
-        import nixpkgs {
-          inherit system;
-          overlays = [ self.overlay ];
-        }
-      );
-
-      commonDeps = pkgs: with pkgs; rec {
-        # Use "busybox-sandbox-shell" if present,
-        # if not (legacy) fallback and hope it's sufficient.
-        sh = pkgs.busybox-sandbox-shell or (busybox.override {
-          useMusl = true;
-          enableStatic = true;
-          enableMinimal = true;
-          extraConfig = ''
-            CONFIG_FEATURE_FANCY_ECHO y
-            CONFIG_FEATURE_SH_MATH y
-            CONFIG_FEATURE_SH_MATH_64 y
-
-            CONFIG_ASH y
-            CONFIG_ASH_OPTIMIZE_FOR_SIZE y
-
-            CONFIG_ASH_ALIAS y
-            CONFIG_ASH_BASH_COMPAT y
-            CONFIG_ASH_CMDCMD y
-            CONFIG_ASH_ECHO y
-            CONFIG_ASH_GETOPTS y
-            CONFIG_ASH_INTERNAL_GLOB y
-            CONFIG_ASH_JOB_CONTROL y
-            CONFIG_ASH_PRINTF y
-            CONFIG_ASH_TEST y
-          '';
-        });
-
-        configureFlags =
-          lib.optionals stdenv.isLinux [
-            "--with-sandbox-shell=${sh}/bin/busybox"
-          ];
-
-        buildDeps =
-          [ bison
-            flex
-            libxml2
-            libxslt
-            docbook5
-            docbook_xsl_ns
-            autoconf-archive
-            autoreconfHook
-
-            curl
-            bzip2 xz brotli zlib editline
-            openssl pkgconfig sqlite
-            libarchive
-            boost
-            (if lib.versionAtLeast lib.version "20.03pre"
-             then nlohmann_json
-             else nlohmann_json.override { multipleHeaders = true; })
-            nlohmann_json
-            rustc cargo
-
-            # Tests
-            git
-            mercurial
-            jq
-          ]
-          ++ lib.optionals stdenv.isLinux [libseccomp utillinuxMinimal]
-          ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium
-          ++ lib.optional (stdenv.isLinux || stdenv.isDarwin)
-            (aws-sdk-cpp.override {
-              apis = ["s3" "transfer"];
-              customMemoryManagement = false;
-            });
-
-        propagatedDeps =
-          [ (boehmgc.override { enableLargeConfig = true; })
-          ];
-
-        perlDeps =
-          [ perl
-            perlPackages.DBDSQLite
-          ];
-      };
-
-    in {
-
-      # A Nixpkgs overlay that overrides the 'nix' and
-      # 'nix.perl-bindings' packages.
-      overlay = final: prev: {
-
-        nix = with final; with commonDeps pkgs; (stdenv.mkDerivation {
-          name = "nix-${version}";
-
-          src = self;
-
-          VERSION_SUFFIX = versionSuffix;
-
-          outputs = [ "out" "dev" "doc" ];
-
-          buildInputs = buildDeps;
-
-          propagatedBuildInputs = propagatedDeps;
-
-          preConfigure =
-            ''
-              # Copy libboost_context so we don't get all of Boost in our closure.
-              # https://github.com/NixOS/nixpkgs/issues/45462
-              mkdir -p $out/lib
-              cp -pd ${boost}/lib/{libboost_context*,libboost_thread*,libboost_system*} $out/lib
-              rm -f $out/lib/*.a
-              ${lib.optionalString stdenv.isLinux ''
-                chmod u+w $out/lib/*.so.*
-                patchelf --set-rpath $out/lib:${stdenv.cc.cc.lib}/lib $out/lib/libboost_thread.so.*
-              ''}
-
-              ln -sfn ${final.nixVendoredCrates}/vendor/ nix-rust/vendor
-            '';
-
-          configureFlags = configureFlags ++
-            [ "--sysconfdir=/etc" ];
-
-          enableParallelBuilding = true;
-
-          makeFlags = "profiledir=$(out)/etc/profile.d";
-
-          doCheck = true;
-
-          installFlags = "sysconfdir=$(out)/etc";
-
-          doInstallCheck = true;
-          installCheckFlags = "sysconfdir=$(out)/etc";
-
-          separateDebugInfo = true;
-
-          preDist = ''
-            mkdir -p $doc/nix-support
-            echo "doc manual $doc/share/doc/nix/manual" >> $doc/nix-support/hydra-build-products
-          '';
-        }) // {
-
-          perl-bindings = with final; stdenv.mkDerivation {
-            name = "nix-perl-${version}";
-
-            src = self;
-
-            buildInputs =
-              [ autoconf-archive
-                autoreconfHook
-                nix
-                curl
-                bzip2
-                xz
-                pkgconfig
-                pkgs.perl
-                boost
-              ]
-              ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium;
-
-            configureFlags = ''
-              --with-dbi=${perlPackages.DBI}/${pkgs.perl.libPrefix}
-              --with-dbd-sqlite=${perlPackages.DBDSQLite}/${pkgs.perl.libPrefix}
-            '';
-
-            enableParallelBuilding = true;
-
-            postUnpack = "sourceRoot=$sourceRoot/perl";
-          };
-
-        };
-
-        # Create a "vendor" directory that contains the crates listed in
-        # Cargo.lock, and include it in the Nix tarball. This allows Nix
-        # to be built without network access.
-        nixVendoredCrates =
-          let
-            lockFile = builtins.fromTOML (builtins.readFile nix-rust/Cargo.lock);
-
-            files = map (pkg: import <nix/fetchurl.nix> {
-              url = "https://crates.io/api/v1/crates/${pkg.name}/${pkg.version}/download";
-              sha256 = lockFile.metadata."checksum ${pkg.name} ${pkg.version} (registry+https://github.com/rust-lang/crates.io-index)";
-            }) (builtins.filter (pkg: pkg.source or "" == "registry+https://github.com/rust-lang/crates.io-index") lockFile.package);
-
-          in final.runCommand "cargo-vendor-dir" {}
-            ''
-              mkdir -p $out/vendor
-
-              cat > $out/vendor/config <<EOF
-              [source.crates-io]
-              replace-with = "vendored-sources"
-
-              [source.vendored-sources]
-              directory = "vendor"
-              EOF
-
-              ${toString (builtins.map (file: ''
-                mkdir $out/vendor/tmp
-                tar xvf ${file} -C $out/vendor/tmp
-                dir=$(echo $out/vendor/tmp/*)
-
-                # Add just enough metadata to keep Cargo happy.
-                printf '{"files":{},"package":"${file.outputHash}"}' > "$dir/.cargo-checksum.json"
-
-                # Clean up some cruft from the winapi crates. FIXME: find
-                # a way to remove winapi* from our dependencies.
-                if [[ $dir =~ /winapi ]]; then
-                  find $dir -name "*.a" -print0 | xargs -0 rm -f --
-                fi
-
-                mv "$dir" $out/vendor/
-
-                rm -rf $out/vendor/tmp
-              '') files)}
-            '';
-
-      };
-
-      hydraJobs = {
-
-        vendoredCrates =
-          with nixpkgsFor.x86_64-linux;
-
-          runCommand "vendored-crates" {}
-            ''
-              mkdir -p $out/nix-support
-              name=nix-vendored-crates-${version}
-              fn=$out/$name.tar.xz
-              tar cvfJ $fn -C ${nixVendoredCrates} vendor \
-                --owner=0 --group=0 --mode=u+rw,uga+r \
-                --transform "s,vendor,$name,"
-              echo "file crates-tarball $fn" >> $out/nix-support/hydra-build-products
-            '';
-
-        # Binary package for various platforms.
-        build = nixpkgs.lib.genAttrs systems (system: nixpkgsFor.${system}.nix);
-
-        # Perl bindings for various platforms.
-        perlBindings = nixpkgs.lib.genAttrs systems (system: nixpkgsFor.${system}.nix.perl-bindings);
-
-        # Binary tarball for various platforms, containing a Nix store
-        # with the closure of 'nix' package, and the second half of
-        # the installation script.
-        binaryTarball = nixpkgs.lib.genAttrs systems (system:
-
-          with nixpkgsFor.${system};
-
-          let
-            installerClosureInfo = closureInfo { rootPaths = [ nix cacert ]; };
-          in
-
-          runCommand "nix-binary-tarball-${version}"
-            { #nativeBuildInputs = lib.optional (system != "aarch64-linux") shellcheck;
-              meta.description = "Distribution-independent Nix bootstrap binaries for ${system}";
-            }
-            ''
-              cp ${installerClosureInfo}/registration $TMPDIR/reginfo
-              substitute ${./scripts/install-nix-from-closure.sh} $TMPDIR/install \
-                --subst-var-by nix ${nix} \
-                --subst-var-by cacert ${cacert}
-
-              substitute ${./scripts/install-darwin-multi-user.sh} $TMPDIR/install-darwin-multi-user.sh \
-                --subst-var-by nix ${nix} \
-                --subst-var-by cacert ${cacert}
-              substitute ${./scripts/install-systemd-multi-user.sh} $TMPDIR/install-systemd-multi-user.sh \
-                --subst-var-by nix ${nix} \
-                --subst-var-by cacert ${cacert}
-              substitute ${./scripts/install-multi-user.sh} $TMPDIR/install-multi-user \
-                --subst-var-by nix ${nix} \
-                --subst-var-by cacert ${cacert}
-
-              if type -p shellcheck; then
-                # SC1090: Don't worry about not being able to find
-                #         $nix/etc/profile.d/nix.sh
-                shellcheck --exclude SC1090 $TMPDIR/install
-                shellcheck $TMPDIR/install-darwin-multi-user.sh
-                shellcheck $TMPDIR/install-systemd-multi-user.sh
-
-                # SC1091: Don't panic about not being able to source
-                #         /etc/profile
-                # SC2002: Ignore "useless cat" "error", when loading
-                #         .reginfo, as the cat is a much cleaner
-                #         implementation, even though it is "useless"
-                # SC2116: Allow ROOT_HOME=$(echo ~root) for resolving
-                #         root's home directory
-                shellcheck --external-sources \
-                  --exclude SC1091,SC2002,SC2116 $TMPDIR/install-multi-user
-              fi
-
-              chmod +x $TMPDIR/install
-              chmod +x $TMPDIR/install-darwin-multi-user.sh
-              chmod +x $TMPDIR/install-systemd-multi-user.sh
-              chmod +x $TMPDIR/install-multi-user
-              dir=nix-${version}-${system}
-              fn=$out/$dir.tar.xz
-              mkdir -p $out/nix-support
-              echo "file binary-dist $fn" >> $out/nix-support/hydra-build-products
-              tar cvfJ $fn \
-                --owner=0 --group=0 --mode=u+rw,uga+r \
-                --absolute-names \
-                --hard-dereference \
-                --transform "s,$TMPDIR/install,$dir/install," \
-                --transform "s,$TMPDIR/reginfo,$dir/.reginfo," \
-                --transform "s,$NIX_STORE,$dir/store,S" \
-                $TMPDIR/install $TMPDIR/install-darwin-multi-user.sh \
-                $TMPDIR/install-systemd-multi-user.sh \
-                $TMPDIR/install-multi-user $TMPDIR/reginfo \
-                $(cat ${installerClosureInfo}/store-paths)
-            '');
-
-        # The first half of the installation script. This is uploaded
-        # to https://nixos.org/nix/install. It downloads the binary
-        # tarball for the user's system and calls the second half of the
-        # installation script.
-        installerScript =
-          with nixpkgsFor.x86_64-linux;
-          runCommand "installer-script"
-            { buildInputs = [ nix ];
-            }
-            ''
-              mkdir -p $out/nix-support
-
-              substitute ${./scripts/install.in} $out/install \
-                ${pkgs.lib.concatMapStrings
-                  (system: "--replace '@binaryTarball_${system}@' $(nix --experimental-features nix-command hash-file --base16 --type sha256 ${self.hydraJobs.binaryTarball.${system}}/*.tar.xz) ")
-                  [ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ]
-                } \
-                --replace '@nixVersion@' ${version}
-
-              echo "file installer $out/install" >> $out/nix-support/hydra-build-products
-            '';
-
-        # Line coverage analysis.
-        coverage =
-          with nixpkgsFor.x86_64-linux;
-          with commonDeps pkgs;
-
-          releaseTools.coverageAnalysis {
-            name = "nix-coverage-${version}";
-
-            src = self;
-
-            preConfigure =
-              ''
-                ln -sfn ${nixVendoredCrates}/vendor/ nix-rust/vendor
-              '';
-
-            enableParallelBuilding = true;
-
-            buildInputs = buildDeps ++ propagatedDeps;
-
-            dontInstall = false;
-
-            doInstallCheck = true;
-
-            lcovFilter = [ "*/boost/*" "*-tab.*" ];
-
-            # We call `dot', and even though we just use it to
-            # syntax-check generated dot files, it still requires some
-            # fonts.  So provide those.
-            FONTCONFIG_FILE = texFunctions.fontsConf;
-
-            # To test building without precompiled headers.
-            makeFlagsArray = [ "PRECOMPILE_HEADERS=0" ];
-          };
-
-        # System tests.
-        tests.remoteBuilds = import ./tests/remote-builds.nix {
-          system = "x86_64-linux";
-          inherit nixpkgs;
-          inherit (self) overlay;
-        };
-
-        tests.nix-copy-closure = import ./tests/nix-copy-closure.nix {
-          system = "x86_64-linux";
-          inherit nixpkgs;
-          inherit (self) overlay;
-        };
-
-        tests.githubFlakes = (import ./tests/github-flakes.nix rec {
-          system = "x86_64-linux";
-          inherit nixpkgs;
-          inherit (self) overlay;
-        });
-
-        tests.setuid = nixpkgs.lib.genAttrs
-          ["i686-linux" "x86_64-linux"]
-          (system:
-            import ./tests/setuid.nix rec {
-              inherit nixpkgs system;
-              inherit (self) overlay;
-            });
-
-        # Test whether the binary tarball works in an Ubuntu system.
-        tests.binaryTarball =
-          with nixpkgsFor.x86_64-linux;
-          vmTools.runInLinuxImage (runCommand "nix-binary-tarball-test"
-            { diskImage = vmTools.diskImages.ubuntu1204x86_64;
-            }
-            ''
-              set -x
-              useradd -m alice
-              su - alice -c 'tar xf ${self.hydraJobs.binaryTarball.x86_64-linux}/*.tar.*'
-              mkdir /dest-nix
-              mount -o bind /dest-nix /nix # Provide a writable /nix.
-              chown alice /nix
-              su - alice -c '_NIX_INSTALLER_TEST=1 ./nix-*/install'
-              su - alice -c 'nix-store --verify'
-              su - alice -c 'PAGER= nix-store -qR ${self.hydraJobs.build.x86_64-linux}'
-
-              # Check whether 'nix upgrade-nix' works.
-              cat > /tmp/paths.nix <<EOF
-              {
-                x86_64-linux = "${self.hydraJobs.build.x86_64-linux}";
-              }
-              EOF
-              su - alice -c 'nix --experimental-features nix-command upgrade-nix -vvv --nix-store-paths-url file:///tmp/paths.nix'
-              (! [ -L /home/alice/.profile-1-link ])
-              su - alice -c 'PAGER= nix-store -qR ${self.hydraJobs.build.x86_64-linux}'
-
-              mkdir -p $out/nix-support
-              touch $out/nix-support/hydra-build-products
-              umount /nix
-            '');
-
-        /*
-        # Check whether we can still evaluate all of Nixpkgs.
-        tests.evalNixpkgs =
-          import (nixpkgs + "/pkgs/top-level/make-tarball.nix") {
-            # FIXME: fix pkgs/top-level/make-tarball.nix in NixOS to not require a revCount.
-            inherit nixpkgs;
-            pkgs = nixpkgsFor.x86_64-linux;
-            officialRelease = false;
-          };
-
-        # Check whether we can still evaluate NixOS.
-        tests.evalNixOS =
-          with nixpkgsFor.x86_64-linux;
-          runCommand "eval-nixos" { buildInputs = [ nix ]; }
-            ''
-              export NIX_STATE_DIR=$TMPDIR
-
-              nix-instantiate ${nixpkgs}/nixos/release-combined.nix -A tested --dry-run \
-                --arg nixpkgs '{ outPath = ${nixpkgs}; revCount = 123; shortRev = "abcdefgh"; }'
-
-              touch $out
-            '';
-        */
-
-      };
-
-      checks = forAllSystems (system: {
-        binaryTarball = self.hydraJobs.binaryTarball.${system};
-        perlBindings = self.hydraJobs.perlBindings.${system};
-      });
-
-      packages = forAllSystems (system: {
-        inherit (nixpkgsFor.${system}) nix;
-      });
-
-      defaultPackage = forAllSystems (system: self.packages.${system}.nix);
-
-      devShell = forAllSystems (system:
-        with nixpkgsFor.${system};
-        with commonDeps pkgs;
-
-        stdenv.mkDerivation {
-          name = "nix";
-
-          buildInputs = buildDeps ++ propagatedDeps ++ perlDeps ++ [ pkgs.rustfmt ];
-
-          inherit configureFlags;
-
-          enableParallelBuilding = true;
-
-          installFlags = "sysconfdir=$(out)/etc";
-
-          shellHook =
-            ''
-              export prefix=$(pwd)/inst
-              configureFlags+=" --prefix=$prefix"
-              PKG_CONFIG_PATH=$prefix/lib/pkgconfig:$PKG_CONFIG_PATH
-              PATH=$prefix/bin:$PATH
-              unset PYTHONPATH
-            '';
-        });
-
-  };
-}

local.mk

@@ -8,7 +8,7 @@ clean-files += Makefile.config
 
 GLOBAL_CXXFLAGS += -Wno-deprecated-declarations
 
-$(foreach i, config.h $(wildcard src/lib*/*.hh), \
+$(foreach i, config.h $(call rwildcard, src/lib*, *.hh), \
   $(eval $(call install-file-in, $(i), $(includedir)/nix, 0644)))
 
 $(GCH) $(PCH): src/libutil/util.hh config.h

maintainers/upload-release.pl

@@ -1,5 +1,5 @@
 #! /usr/bin/env nix-shell
-#! nix-shell -i perl -p perl perlPackages.LWPUserAgent perlPackages.LWPProtocolHttps perlPackages.FileSlurp perlPackages.NetAmazonS3 gnupg1
+#! nix-shell -i perl -p perl perlPackages.LWPUserAgent perlPackages.LWPProtocolHttps perlPackages.FileSlurp gnupg1
 
 use strict;
 use Data::Dumper;
@@ -9,16 +9,12 @@ use File::Slurp;
 use File::Copy;
 use JSON::PP;
 use LWP::UserAgent;
-use Net::Amazon::S3;
 
 my $evalId = $ARGV[0] or die "Usage: $0 EVAL-ID\n";
 
-my $releasesBucketName = "nix-releases";
-my $channelsBucketName = "nix-channels";
+my $releasesDir = "/home/eelco/mnt/releases";
 my $nixpkgsDir = "/home/eelco/Dev/nixpkgs-pristine";
 
-my $TMPDIR = $ENV{'TMPDIR'} // "/tmp";
-
 # FIXME: cut&paste from nixos-channel-scripts.
 sub fetch {
     my ($url, $type) = @_;
@@ -46,31 +42,13 @@ my $version = $1;
 
 print STDERR "Nix revision is $nixRev, version is $version\n";
 
-my $releaseDir = "nix/$releaseName";
+File::Path::make_path($releasesDir);
+if (system("mountpoint -q $releasesDir") != 0) {
+    system("sshfs hydra-mirror\@nixos.org:/releases $releasesDir") == 0 or die;
+}
 
-my $tmpDir = "$TMPDIR/nix-release/$releaseName";
-File::Path::make_path($tmpDir);
-
-# S3 setup.
-my $aws_access_key_id = $ENV{'AWS_ACCESS_KEY_ID'} or die "No AWS_ACCESS_KEY_ID given.";
-my $aws_secret_access_key = $ENV{'AWS_SECRET_ACCESS_KEY'} or die "No AWS_SECRET_ACCESS_KEY given.";
-
-my $s3 = Net::Amazon::S3->new(
-    { aws_access_key_id     => $aws_access_key_id,
-      aws_secret_access_key => $aws_secret_access_key,
-      retry                 => 1,
-      host                  => "s3-eu-west-1.amazonaws.com",
-    });
-
-my $releasesBucket = $s3->bucket($releasesBucketName) or die;
-
-my $s3_us = Net::Amazon::S3->new(
-    { aws_access_key_id     => $aws_access_key_id,
-      aws_secret_access_key => $aws_secret_access_key,
-      retry                 => 1,
-    });
-
-my $channelsBucket = $s3_us->bucket($channelsBucketName) or die;
+my $releaseDir = "$releasesDir/nix/$releaseName";
+File::Path::make_path($releaseDir);
 
 sub downloadFile {
     my ($jobName, $productNr, $dstName) = @_;
@@ -79,49 +57,40 @@ sub downloadFile {
 
     my $srcFile = $buildInfo->{buildproducts}->{$productNr}->{path} or die "job '$jobName' lacks product $productNr\n";
     $dstName //= basename($srcFile);
-    my $tmpFile = "$tmpDir/$dstName";
+    my $dstFile = "$releaseDir/" . $dstName;
 
-    if (!-e $tmpFile) {
-        print STDERR "downloading $srcFile to $tmpFile...\n";
-        system("NIX_REMOTE=https://cache.nixos.org/ nix cat-store '$srcFile' > '$tmpFile'") == 0
+    if (! -e $dstFile) {
+        print STDERR "downloading $srcFile to $dstFile...\n";
+        system("NIX_REMOTE=https://cache.nixos.org/ nix cat-store '$srcFile' > '$dstFile.tmp'") == 0
             or die "unable to fetch $srcFile\n";
+        rename("$dstFile.tmp", $dstFile) or die;
     }
 
     my $sha256_expected = $buildInfo->{buildproducts}->{$productNr}->{sha256hash} or die;
-    my $sha256_actual = `nix hash-file --base16 --type sha256 '$tmpFile'`;
+    my $sha256_actual = `nix hash-file --base16 --type sha256 '$dstFile'`;
     chomp $sha256_actual;
     if ($sha256_expected ne $sha256_actual) {
-        print STDERR "file $tmpFile is corrupt, got $sha256_actual, expected $sha256_expected\n";
+        print STDERR "file $dstFile is corrupt, got $sha256_actual, expected $sha256_expected\n";
         exit 1;
     }
 
-    write_file("$tmpFile.sha256", $sha256_expected);
+    write_file("$dstFile.sha256", $sha256_expected);
 
-    if (! -e "$tmpFile.asc") {
-        system("gpg2 --detach-sign --armor $tmpFile") == 0 or die "unable to sign $tmpFile\n";
+    if (! -e "$dstFile.asc") {
+        system("gpg2 --detach-sign --armor $dstFile") == 0 or die "unable to sign $dstFile\n";
     }
 
-    return $sha256_expected;
+    return ($dstFile, $sha256_expected);
 }
 
 downloadFile("tarball", "2"); # .tar.bz2
-my $tarballHash = downloadFile("tarball", "3"); # .tar.xz
+my ($tarball, $tarballHash) = downloadFile("tarball", "3"); # .tar.xz
 downloadFile("binaryTarball.i686-linux", "1");
 downloadFile("binaryTarball.x86_64-linux", "1");
 downloadFile("binaryTarball.aarch64-linux", "1");
 downloadFile("binaryTarball.x86_64-darwin", "1");
 downloadFile("installerScript", "1");
 
-for my $fn (glob "$tmpDir/*") {
-    my $name = basename($fn);
-    my $dstKey = "$releaseDir/" . $name;
-    unless (defined $releasesBucket->head_key($dstKey)) {
-        print STDERR "uploading $fn to s3://$releasesBucketName/$dstKey...\n";
-        $releasesBucket->add_key_filename($dstKey, $fn)
-            or die $releasesBucket->err . ": " . $releasesBucket->errstr;
-    }
-}
-
 exit if $version =~ /pre/;
 
 # Update Nixpkgs in a very hacky way.
@@ -156,11 +125,18 @@ write_file("$nixpkgsDir/nixos/modules/installer/tools/nix-fallback-paths.nix",
 
 system("cd $nixpkgsDir && git commit -a -m 'nix: $oldName -> $version'") == 0 or die;
 
+# Extract the HTML manual.
+File::Path::make_path("$releaseDir/manual");
+
+system("tar xvf $tarball --strip-components=3 -C $releaseDir/manual --wildcards '*/doc/manual/*.html' '*/doc/manual/*.css' '*/doc/manual/*.gif' '*/doc/manual/*.png'") == 0 or die;
+
+if (! -e "$releaseDir/manual/index.html") {
+    symlink("manual.html", "$releaseDir/manual/index.html") or die;
+}
+
 # Update the "latest" symlink.
-$channelsBucket->add_key(
-    "nix-latest/install", "",
-    { "x-amz-website-redirect-location" => "https://releases.nixos.org/$releaseDir/install" })
-    or die $channelsBucket->err . ": " . $channelsBucket->errstr;
+symlink("$releaseName", "$releasesDir/nix/latest-tmp") or die;
+rename("$releasesDir/nix/latest-tmp", "$releasesDir/nix/latest") or die;
 
 # Tag the release in Git.
 chdir("/home/eelco/Dev/nix-pristine") or die;

misc/bash/completion.sh

@@ -1,19 +0,0 @@
-function _complete_nix {
-    local -a words
-    local cword cur
-    _get_comp_words_by_ref -n ':=&' words cword cur
-    local have_type
-    while IFS= read -r line; do
-        if [[ -z $have_type ]]; then
-            have_type=1
-            if [[ $line = filenames ]]; then
-                compopt -o filenames
-            fi
-        else
-            COMPREPLY+=("$line")
-        fi
-    done < <(NIX_GET_COMPLETIONS=$cword "${words[@]}")
-    __ltrim_colon_completions "$cur"
-}
-
-complete -F _complete_nix nix

misc/bash/local.mk

@@ -1 +0,0 @@
-$(eval $(call install-file-as, $(d)/completion.sh, $(datarootdir)/bash-completion/completions/_nix3, 0644))

precompiled-headers.h

@@ -56,3 +56,6 @@
 #include <sys/wait.h>
 #include <termios.h>
 #include <unistd.h>
+
+#include "util.hh"
+#include "args.hh"

release-common.nix

@@ -0,0 +1,82 @@
+{ pkgs }:
+
+with pkgs;
+
+rec {
+  # Use "busybox-sandbox-shell" if present,
+  # if not (legacy) fallback and hope it's sufficient.
+  sh = pkgs.busybox-sandbox-shell or (busybox.override {
+    useMusl = true;
+    enableStatic = true;
+    enableMinimal = true;
+    extraConfig = ''
+      CONFIG_FEATURE_FANCY_ECHO y
+      CONFIG_FEATURE_SH_MATH y
+      CONFIG_FEATURE_SH_MATH_64 y
+
+      CONFIG_ASH y
+      CONFIG_ASH_OPTIMIZE_FOR_SIZE y
+
+      CONFIG_ASH_ALIAS y
+      CONFIG_ASH_BASH_COMPAT y
+      CONFIG_ASH_CMDCMD y
+      CONFIG_ASH_ECHO y
+      CONFIG_ASH_GETOPTS y
+      CONFIG_ASH_INTERNAL_GLOB y
+      CONFIG_ASH_JOB_CONTROL y
+      CONFIG_ASH_PRINTF y
+      CONFIG_ASH_TEST y
+    '';
+  });
+
+  configureFlags =
+    lib.optionals stdenv.isLinux [
+      "--with-sandbox-shell=${sh}/bin/busybox"
+    ];
+
+  buildDeps =
+    [ bison
+      flex
+      libxml2
+      libxslt
+      docbook5
+      docbook_xsl_ns
+      autoconf-archive
+      autoreconfHook
+
+      curl
+      bzip2 xz brotli zlib editline
+      openssl pkgconfig sqlite
+      libarchive
+      boost
+      nlohmann_json
+      rustc cargo
+
+      # Tests
+      git
+      mercurial
+    ]
+    ++ lib.optionals stdenv.isLinux [libseccomp utillinuxMinimal]
+    ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium
+    ++ lib.optional (stdenv.isLinux || stdenv.isDarwin)
+      ((aws-sdk-cpp.override {
+        apis = ["s3" "transfer"];
+        customMemoryManagement = false;
+      }).overrideDerivation (args: {
+        /*
+        patches = args.patches or [] ++ [ (fetchpatch {
+          url = https://github.com/edolstra/aws-sdk-cpp/commit/3e07e1f1aae41b4c8b340735ff9e8c735f0c063f.patch;
+          sha256 = "1pij0v449p166f9l29x7ppzk8j7g9k9mp15ilh5qxp29c7fnvxy2";
+        }) ];
+        */
+      }));
+
+  propagatedDeps =
+    [ (boehmgc.override { enableLargeConfig = true; })
+    ];
+
+  perlDeps =
+    [ perl
+      perlPackages.DBDSQLite
+    ];
+}

release.nix

@@ -0,0 +1,343 @@
+{ nix ? builtins.fetchGit ./.
+, nixpkgs ? builtins.fetchTarball https://github.com/NixOS/nixpkgs/archive/nixos-20.03-small.tar.gz
+, officialRelease ? false
+, systems ? [ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ]
+}:
+
+let
+
+  pkgs = import nixpkgs { system = builtins.currentSystem or "x86_64-linux"; };
+
+  version =
+    builtins.readFile ./.version
+    + (if officialRelease then "" else "pre${toString nix.revCount}_${nix.shortRev}");
+
+  jobs = rec {
+
+    # Create a "vendor" directory that contains the crates listed in
+    # Cargo.lock. This allows Nix to be built without network access.
+    vendoredCrates =
+      let
+        lockFile = builtins.fromTOML (builtins.readFile nix-rust/Cargo.lock);
+
+        files = map (pkg: import <nix/fetchurl.nix> {
+          url = "https://crates.io/api/v1/crates/${pkg.name}/${pkg.version}/download";
+          sha256 = lockFile.metadata."checksum ${pkg.name} ${pkg.version} (registry+https://github.com/rust-lang/crates.io-index)";
+        }) (builtins.filter (pkg: pkg.source or "" == "registry+https://github.com/rust-lang/crates.io-index") lockFile.package);
+
+      in pkgs.runCommand "cargo-vendor-dir" {}
+        ''
+          mkdir -p $out/vendor
+
+          cat > $out/vendor/config <<EOF
+          [source.crates-io]
+          replace-with = "vendored-sources"
+
+          [source.vendored-sources]
+          directory = "vendor"
+          EOF
+
+          ${toString (builtins.map (file: ''
+            mkdir $out/vendor/tmp
+            tar xvf ${file} -C $out/vendor/tmp
+            dir=$(echo $out/vendor/tmp/*)
+
+            # Add just enough metadata to keep Cargo happy.
+            printf '{"files":{},"package":"${file.outputHash}"}' > "$dir/.cargo-checksum.json"
+
+            # Clean up some cruft from the winapi crates. FIXME: find
+            # a way to remove winapi* from our dependencies.
+            if [[ $dir =~ /winapi ]]; then
+              find $dir -name "*.a" -print0 | xargs -0 rm -f --
+            fi
+
+            mv "$dir" $out/vendor/
+
+            rm -rf $out/vendor/tmp
+          '') files)}
+        '';
+
+
+    build = pkgs.lib.genAttrs systems (system:
+
+      let pkgs = import nixpkgs { inherit system; }; in
+
+      with pkgs;
+
+      with import ./release-common.nix { inherit pkgs; };
+
+      stdenv.mkDerivation {
+        name = "nix-${version}";
+
+        src = nix;
+
+        outputs = [ "out" "dev" "doc" ];
+
+        buildInputs = buildDeps;
+
+        propagatedBuildInputs = propagatedDeps;
+
+        preConfigure =
+          ''
+            # Copy libboost_context so we don't get all of Boost in our closure.
+            # https://github.com/NixOS/nixpkgs/issues/45462
+            mkdir -p $out/lib
+            cp -pd ${boost}/lib/{libboost_context*,libboost_thread*,libboost_system*} $out/lib
+            rm -f $out/lib/*.a
+            ${lib.optionalString stdenv.isLinux ''
+              chmod u+w $out/lib/*.so.*
+              patchelf --set-rpath $out/lib:${stdenv.cc.cc.lib}/lib $out/lib/libboost_thread.so.*
+            ''}
+
+            ln -sfn ${vendoredCrates}/vendor/ nix-rust/vendor
+
+            (cd perl; autoreconf --install --force --verbose)
+          '';
+
+        configureFlags = configureFlags ++
+          [ "--sysconfdir=/etc" ];
+
+        enableParallelBuilding = true;
+
+        makeFlags = "profiledir=$(out)/etc/profile.d";
+
+        installFlags = "sysconfdir=$(out)/etc";
+
+        doCheck = true;
+
+        doInstallCheck = true;
+        installCheckFlags = "sysconfdir=$(out)/etc";
+
+        separateDebugInfo = true;
+
+        preDist = ''
+          mkdir -p $doc/nix-support
+          echo "doc manual $doc/share/doc/nix/manual" >> $doc/nix-support/hydra-build-products
+        '';
+      });
+
+
+    perlBindings = pkgs.lib.genAttrs systems (system:
+
+      let pkgs = import nixpkgs { inherit system; }; in with pkgs;
+
+      releaseTools.nixBuild {
+        name = "nix-perl-${version}";
+
+        src = nix;
+
+        buildInputs =
+          [ autoconf-archive
+            autoreconfHook
+            jobs.build.${system}
+            curl
+            bzip2
+            xz
+            pkgconfig
+            pkgs.perl
+            boost
+          ]
+          ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium;
+
+        configureFlags = ''
+          --with-dbi=${perlPackages.DBI}/${pkgs.perl.libPrefix}
+          --with-dbd-sqlite=${perlPackages.DBDSQLite}/${pkgs.perl.libPrefix}
+        '';
+
+        enableParallelBuilding = true;
+
+        postUnpack = "sourceRoot=$sourceRoot/perl";
+      });
+
+
+    binaryTarball = pkgs.lib.genAttrs systems (system:
+
+      with import nixpkgs { inherit system; };
+
+      let
+        toplevel = builtins.getAttr system jobs.build;
+        installerClosureInfo = closureInfo { rootPaths = [ toplevel cacert ]; };
+      in
+
+      runCommand "nix-binary-tarball-${version}"
+        { #nativeBuildInputs = lib.optional (system != "aarch64-linux") shellcheck;
+          meta.description = "Distribution-independent Nix bootstrap binaries for ${system}";
+        }
+        ''
+          cp ${installerClosureInfo}/registration $TMPDIR/reginfo
+          substitute ${./scripts/install-nix-from-closure.sh} $TMPDIR/install \
+            --subst-var-by nix ${toplevel} \
+            --subst-var-by cacert ${cacert}
+
+          substitute ${./scripts/install-darwin-multi-user.sh} $TMPDIR/install-darwin-multi-user.sh \
+            --subst-var-by nix ${toplevel} \
+            --subst-var-by cacert ${cacert}
+          substitute ${./scripts/install-systemd-multi-user.sh} $TMPDIR/install-systemd-multi-user.sh \
+            --subst-var-by nix ${toplevel} \
+            --subst-var-by cacert ${cacert}
+          substitute ${./scripts/install-multi-user.sh} $TMPDIR/install-multi-user \
+            --subst-var-by nix ${toplevel} \
+            --subst-var-by cacert ${cacert}
+
+          if type -p shellcheck; then
+            # SC1090: Don't worry about not being able to find
+            #         $nix/etc/profile.d/nix.sh
+            shellcheck --exclude SC1090 $TMPDIR/install
+            shellcheck $TMPDIR/install-darwin-multi-user.sh
+            shellcheck $TMPDIR/install-systemd-multi-user.sh
+
+            # SC1091: Don't panic about not being able to source
+            #         /etc/profile
+            # SC2002: Ignore "useless cat" "error", when loading
+            #         .reginfo, as the cat is a much cleaner
+            #         implementation, even though it is "useless"
+            # SC2116: Allow ROOT_HOME=$(echo ~root) for resolving
+            #         root's home directory
+            shellcheck --external-sources \
+              --exclude SC1091,SC2002,SC2116 $TMPDIR/install-multi-user
+          fi
+
+          chmod +x $TMPDIR/install
+          chmod +x $TMPDIR/install-darwin-multi-user.sh
+          chmod +x $TMPDIR/install-systemd-multi-user.sh
+          chmod +x $TMPDIR/install-multi-user
+          dir=nix-${version}-${system}
+          fn=$out/$dir.tar.xz
+          mkdir -p $out/nix-support
+          echo "file binary-dist $fn" >> $out/nix-support/hydra-build-products
+          tar cvfJ $fn \
+            --owner=0 --group=0 --mode=u+rw,uga+r \
+            --absolute-names \
+            --hard-dereference \
+            --transform "s,$TMPDIR/install,$dir/install," \
+            --transform "s,$TMPDIR/reginfo,$dir/.reginfo," \
+            --transform "s,$NIX_STORE,$dir/store,S" \
+            $TMPDIR/install $TMPDIR/install-darwin-multi-user.sh \
+            $TMPDIR/install-systemd-multi-user.sh \
+            $TMPDIR/install-multi-user $TMPDIR/reginfo \
+            $(cat ${installerClosureInfo}/store-paths)
+        '');
+
+
+    coverage =
+      with pkgs;
+
+      with import ./release-common.nix { inherit pkgs; };
+
+      releaseTools.coverageAnalysis {
+        name = "nix-coverage-${version}";
+
+        src = nix;
+
+        enableParallelBuilding = true;
+
+        buildInputs = buildDeps ++ propagatedDeps;
+
+        dontInstall = false;
+
+        doInstallCheck = true;
+
+        lcovFilter = [ "*/boost/*" "*-tab.*" ];
+
+        # We call `dot', and even though we just use it to
+        # syntax-check generated dot files, it still requires some
+        # fonts.  So provide those.
+        FONTCONFIG_FILE = texFunctions.fontsConf;
+
+        # To test building without precompiled headers.
+        makeFlagsArray = [ "PRECOMPILE_HEADERS=0" ];
+      };
+
+
+    # System tests.
+    tests.remoteBuilds = (import ./tests/remote-builds.nix rec {
+      inherit nixpkgs;
+      nix = build.x86_64-linux; system = "x86_64-linux";
+    });
+
+    tests.nix-copy-closure = (import ./tests/nix-copy-closure.nix rec {
+      inherit nixpkgs;
+      nix = build.x86_64-linux; system = "x86_64-linux";
+    });
+
+    tests.setuid = pkgs.lib.genAttrs
+      ["i686-linux" "x86_64-linux"]
+      (system:
+        import ./tests/setuid.nix rec {
+          inherit nixpkgs;
+          nix = build.${system}; inherit system;
+        });
+
+    tests.binaryTarball =
+      with import nixpkgs { system = "x86_64-linux"; };
+      vmTools.runInLinuxImage (runCommand "nix-binary-tarball-test"
+        { diskImage = vmTools.diskImages.ubuntu1204x86_64;
+        }
+        ''
+          set -x
+          useradd -m alice
+          su - alice -c 'tar xf ${binaryTarball.x86_64-linux}/*.tar.*'
+          mkdir /dest-nix
+          mount -o bind /dest-nix /nix # Provide a writable /nix.
+          chown alice /nix
+          su - alice -c '_NIX_INSTALLER_TEST=1 ./nix-*/install'
+          su - alice -c 'nix-store --verify'
+          su - alice -c 'PAGER= nix-store -qR ${build.x86_64-linux}'
+
+          # Check whether 'nix upgrade-nix' works.
+          cat > /tmp/paths.nix <<EOF
+          {
+            x86_64-linux = "${build.x86_64-linux}";
+          }
+          EOF
+          su - alice -c 'nix --experimental-features nix-command upgrade-nix -vvv --nix-store-paths-url file:///tmp/paths.nix'
+          (! [ -L /home/alice/.profile-1-link ])
+          su - alice -c 'PAGER= nix-store -qR ${build.x86_64-linux}'
+
+          mkdir -p $out/nix-support
+          touch $out/nix-support/hydra-build-products
+          umount /nix
+        ''); # */
+
+    /*
+    tests.evalNixpkgs =
+      import (nixpkgs + "/pkgs/top-level/make-tarball.nix") {
+        inherit nixpkgs;
+        inherit pkgs;
+        nix = build.x86_64-linux;
+        officialRelease = false;
+      };
+
+    tests.evalNixOS =
+      pkgs.runCommand "eval-nixos" { buildInputs = [ build.x86_64-linux ]; }
+        ''
+          export NIX_STATE_DIR=$TMPDIR
+
+          nix-instantiate ${nixpkgs}/nixos/release-combined.nix -A tested --dry-run \
+            --arg nixpkgs '{ outPath = ${nixpkgs}; revCount = 123; shortRev = "abcdefgh"; }'
+
+          touch $out
+        '';
+    */
+
+
+    installerScript =
+      pkgs.runCommand "installer-script"
+        { buildInputs = [ build.${builtins.currentSystem or "x86_64-linux"} ]; }
+        ''
+          mkdir -p $out/nix-support
+
+          substitute ${./scripts/install.in} $out/install \
+            ${pkgs.lib.concatMapStrings
+              (system: "--replace '@binaryTarball_${system}@' $(nix --experimental-features nix-command hash-file --base16 --type sha256 ${binaryTarball.${system}}/*.tar.xz) ")
+              systems
+            } \
+            --replace '@nixVersion@' ${version}
+
+          echo "file installer $out/install" >> $out/nix-support/hydra-build-products
+        '';
+
+  };
+
+
+in jobs

shell.nix

@@ -1,3 +1,25 @@
-(import (fetchTarball https://github.com/edolstra/flake-compat/archive/master.tar.gz) {
-  src = builtins.fetchGit ./.;
-}).shellNix
+{ useClang ? false }:
+
+with import (builtins.fetchTarball https://github.com/NixOS/nixpkgs/archive/nixos-20.03-small.tar.gz) {};
+
+with import ./release-common.nix { inherit pkgs; };
+
+(if useClang then clangStdenv else stdenv).mkDerivation {
+  name = "nix";
+
+  buildInputs = buildDeps ++ propagatedDeps ++ perlDeps ++ [ pkgs.rustfmt ];
+
+  inherit configureFlags;
+
+  enableParallelBuilding = true;
+
+  installFlags = "sysconfdir=$(out)/etc";
+
+  shellHook =
+    ''
+      export prefix=$(pwd)/inst
+      configureFlags+=" --prefix=$prefix"
+      PKG_CONFIG_PATH=$prefix/lib/pkgconfig:$PKG_CONFIG_PATH
+      PATH=$prefix/bin:$PATH
+    '';
+}

src/error-demo/error-demo.cc

@@ -1,66 +0,0 @@
-#include "error.hh"
-#include "nixexpr.hh"
-
-#include <iostream>
-#include <optional>
-
-int main()
-{
-    using namespace nix;
-
-    // In each program where errors occur, this has to be set.
-    ErrorInfo::programName = std::optional("error-demo");
-
-    // Error in a program; no hint and no nix code.
-    printErrorInfo(
-        ErrorInfo { .level = elError,
-                    .name = "name",
-                    .description = "error description",
-                  });
-
-    // Warning with name, description, and hint.
-    // The hintfmt function makes all the substituted text yellow.
-    printErrorInfo(
-        ErrorInfo { .level = elWarning,
-                    .name = "name",
-                    .description = "error description",
-                    .hint =  std::optional(
-                                 hintfmt("there was a %1%", "warning")),
-                  });
-
-
-    // Warning with nix file, line number, column, and the lines of
-    // code where a warning occurred.
-    SymbolTable testTable;
-    auto problem_file = testTable.create("myfile.nix");
-
-    printErrorInfo(
-        ErrorInfo{
-            .level = elWarning,
-            .name = "warning name",
-            .description = "warning description",
-            .hint = hintfmt("this hint has %1% templated %2%!!", "yellow", "values"),
-            .nixCode = NixCode {
-                .errPos = Pos(problem_file, 40, 13),
-                .prevLineOfCode = std::nullopt,
-                .errLineOfCode = "this is the problem line of code",
-                .nextLineOfCode = std::nullopt
-            }});
-
-    // Error with previous and next lines of code.
-    printErrorInfo(
-        ErrorInfo{
-            .level = elError,
-            .name = "error name",
-            .description = "error description",
-            .hint = hintfmt("this hint has %1% templated %2%!!", "yellow", "values"),
-            .nixCode = NixCode {
-                .errPos = Pos(problem_file, 40, 13),
-                .prevLineOfCode = std::optional("previous line of code"),
-                .errLineOfCode = "this is the problem line of code",
-                .nextLineOfCode = std::optional("next line of code"),
-            }});
-
-
-    return 0;
-}

src/error-demo/local.mk

@@ -1,12 +0,0 @@
-programs += error-demo
-
-error-demo_DIR := $(d)
-
-error-demo_SOURCES := \
-  $(wildcard $(d)/*.cc) \
-
-error-demo_CXXFLAGS += -I src/libutil -I src/libexpr
-
-error-demo_LIBS = libutil libexpr
-
-error-demo_LDFLAGS = -pthread $(SODIUM_LIBS) $(EDITLINE_LIBS) $(BOOST_LDFLAGS) -lboost_context -lboost_thread -lboost_system

src/libexpr/attr-path.cc

@@ -6,11 +6,11 @@
 namespace nix {
 
 
-static Strings parseAttrPath(std::string_view s)
+static Strings parseAttrPath(const string & s)
 {
     Strings res;
     string cur;
-    auto i = s.begin();
+    string::const_iterator i = s.begin();
     while (i != s.end()) {
         if (*i == '.') {
             res.push_back(cur);
@@ -32,20 +32,14 @@ static Strings parseAttrPath(std::string_view s)
 }
 
 
-std::vector<Symbol> parseAttrPath(EvalState & state, std::string_view s)
-{
-    std::vector<Symbol> res;
-    for (auto & a : parseAttrPath(s))
-        res.push_back(state.symbols.create(a));
-    return res;
-}
-
-
 std::pair<Value *, Pos> findAlongAttrPath(EvalState & state, const string & attrPath,
     Bindings & autoArgs, Value & vIn)
 {
     Strings tokens = parseAttrPath(attrPath);
 
+    Error attrError =
+        Error(format("attribute selection path '%1%' does not match expression") % attrPath);
+
     Value * v = &vIn;
     Pos pos = noPos;
 

src/libexpr/attr-path.hh

@@ -16,6 +16,4 @@ std::pair<Value *, Pos> findAlongAttrPath(EvalState & state, const string & attr
 /* Heuristic to find the filename and lineno or a nix value. */
 Pos findDerivationFilename(EvalState & state, Value & v, std::string what);
 
-std::vector<Symbol> parseAttrPath(EvalState & state, std::string_view s);
-
 }

src/libexpr/common-eval-args.cc

@@ -1,59 +1,31 @@
 #include "common-eval-args.hh"
 #include "shared.hh"
-#include "filetransfer.hh"
+#include "download.hh"
 #include "util.hh"
 #include "eval.hh"
-#include "fetchers.hh"
-#include "registry.hh"
-#include "flake/flakeref.hh"
-#include "store-api.hh"
 
 namespace nix {
 
 MixEvalArgs::MixEvalArgs()
 {
-    addFlag({
-        .longName = "arg",
-        .description = "argument to be passed to Nix functions",
-        .labels = {"name", "expr"},
-        .handler = {[&](std::string name, std::string expr) { autoArgs[name] = 'E' + expr; }}
-    });
+    mkFlag()
+        .longName("arg")
+        .description("argument to be passed to Nix functions")
+        .labels({"name", "expr"})
+        .handler([&](std::vector<std::string> ss) { autoArgs[ss[0]] = 'E' + ss[1]; });
 
-    addFlag({
-        .longName = "argstr",
-        .description = "string-valued argument to be passed to Nix functions",
-        .labels = {"name", "string"},
-        .handler = {[&](std::string name, std::string s) { autoArgs[name] = 'S' + s; }},
-    });
+    mkFlag()
+        .longName("argstr")
+        .description("string-valued argument to be passed to Nix functions")
+        .labels({"name", "string"})
+        .handler([&](std::vector<std::string> ss) { autoArgs[ss[0]] = 'S' + ss[1]; });
 
-    addFlag({
-        .longName = "include",
-        .shortName = 'I',
-        .description = "add a path to the list of locations used to look up <...> file names",
-        .labels = {"path"},
-        .handler = {[&](std::string s) { searchPath.push_back(s); }}
-    });
-
-    addFlag({
-        .longName = "impure",
-        .description = "allow access to mutable paths and repositories",
-        .handler = {[&]() {
-            evalSettings.pureEval = false;
-        }},
-    });
-
-    addFlag({
-        .longName = "override-flake",
-        .description = "override a flake registry value",
-        .labels = {"original-ref", "resolved-ref"},
-        .handler = {[&](std::string _from, std::string _to) {
-            auto from = parseFlakeRef(_from, absPath("."));
-            auto to = parseFlakeRef(_to, absPath("."));
-            fetchers::Attrs extraAttrs;
-            if (to.subdir != "") extraAttrs["dir"] = to.subdir;
-            fetchers::overrideRegistry(from.input, to.input, extraAttrs);
-        }}
-    });
+    mkFlag()
+        .shortName('I')
+        .longName("include")
+        .description("add a path to the list of locations used to look up <...> file names")
+        .label("path")
+        .handler([&](std::string s) { searchPath.push_back(s); });
 }
 
 Bindings * MixEvalArgs::getAutoArgs(EvalState & state)
@@ -74,9 +46,9 @@ Bindings * MixEvalArgs::getAutoArgs(EvalState & state)
 Path lookupFileArg(EvalState & state, string s)
 {
     if (isUri(s)) {
-        return state.store->toRealPath(
-            fetchers::downloadTarball(
-                state.store, resolveUri(s), "source", false).storePath);
+        CachedDownloadRequest request(s);
+        request.unpack = true;
+        return getDownloader()->downloadCached(state.store, request).path;
     } else if (s.size() > 2 && s.at(0) == '<' && s.at(s.size() - 1) == '>') {
         Path p = s.substr(1, s.size() - 2);
         return state.findFile(p);

src/libexpr/eval-cache.cc

@@ -1,503 +0,0 @@
-#include "eval-cache.hh"
-#include "sqlite.hh"
-#include "eval.hh"
-#include "eval-inline.hh"
-
-namespace nix::eval_cache {
-
-static const char * schema = R"sql(
-create table if not exists Attributes (
-    parent      integer not null,
-    name        text,
-    type        integer not null,
-    value       text,
-    primary key (parent, name)
-);
-)sql";
-
-struct AttrDb
-{
-    struct State
-    {
-        SQLite db;
-        SQLiteStmt insertAttribute;
-        SQLiteStmt queryAttribute;
-        SQLiteStmt queryAttributes;
-        std::unique_ptr<SQLiteTxn> txn;
-    };
-
-    std::unique_ptr<Sync<State>> _state;
-
-    AttrDb(const Hash & fingerprint)
-        : _state(std::make_unique<Sync<State>>())
-    {
-        auto state(_state->lock());
-
-        Path cacheDir = getCacheDir() + "/nix/eval-cache-v1";
-        createDirs(cacheDir);
-
-        Path dbPath = cacheDir + "/" + fingerprint.to_string(Base16, false) + ".sqlite";
-
-        state->db = SQLite(dbPath);
-        state->db.isCache();
-        state->db.exec(schema);
-
-        state->insertAttribute.create(state->db,
-            "insert or replace into Attributes(parent, name, type, value) values (?, ?, ?, ?)");
-
-        state->queryAttribute.create(state->db,
-            "select rowid, type, value from Attributes where parent = ? and name = ?");
-
-        state->queryAttributes.create(state->db,
-            "select name from Attributes where parent = ?");
-
-        state->txn = std::make_unique<SQLiteTxn>(state->db);
-    }
-
-    ~AttrDb()
-    {
-        try {
-            auto state(_state->lock());
-            state->txn->commit();
-            state->txn.reset();
-        } catch (...) {
-            ignoreException();
-        }
-    }
-
-    AttrId setAttrs(
-        AttrKey key,
-        const std::vector<Symbol> & attrs)
-    {
-        auto state(_state->lock());
-
-        state->insertAttribute.use()
-            (key.first)
-            (key.second)
-            (AttrType::FullAttrs)
-            (0, false).exec();
-
-        AttrId rowId = state->db.getLastInsertedRowId();
-        assert(rowId);
-
-        for (auto & attr : attrs)
-            state->insertAttribute.use()
-                (rowId)
-                (attr)
-                (AttrType::Placeholder)
-                (0, false).exec();
-
-        return rowId;
-    }
-
-    AttrId setString(
-        AttrKey key,
-        std::string_view s)
-    {
-        auto state(_state->lock());
-
-        state->insertAttribute.use()
-            (key.first)
-            (key.second)
-            (AttrType::String)
-            (s).exec();
-
-        return state->db.getLastInsertedRowId();
-    }
-
-    AttrId setBool(
-        AttrKey key,
-        bool b)
-    {
-        auto state(_state->lock());
-
-        state->insertAttribute.use()
-            (key.first)
-            (key.second)
-            (AttrType::Bool)
-            (b ? 1 : 0).exec();
-
-        return state->db.getLastInsertedRowId();
-    }
-
-    AttrId setPlaceholder(AttrKey key)
-    {
-        auto state(_state->lock());
-
-        state->insertAttribute.use()
-            (key.first)
-            (key.second)
-            (AttrType::Placeholder)
-            (0, false).exec();
-
-        return state->db.getLastInsertedRowId();
-    }
-
-    AttrId setMissing(AttrKey key)
-    {
-        auto state(_state->lock());
-
-        state->insertAttribute.use()
-            (key.first)
-            (key.second)
-            (AttrType::Missing)
-            (0, false).exec();
-
-        return state->db.getLastInsertedRowId();
-    }
-
-    AttrId setMisc(AttrKey key)
-    {
-        auto state(_state->lock());
-
-        state->insertAttribute.use()
-            (key.first)
-            (key.second)
-            (AttrType::Misc)
-            (0, false).exec();
-
-        return state->db.getLastInsertedRowId();
-    }
-
-    AttrId setFailed(AttrKey key)
-    {
-        auto state(_state->lock());
-
-        state->insertAttribute.use()
-            (key.first)
-            (key.second)
-            (AttrType::Failed)
-            (0, false).exec();
-
-        return state->db.getLastInsertedRowId();
-    }
-
-    std::optional<std::pair<AttrId, AttrValue>> getAttr(
-        AttrKey key,
-        SymbolTable & symbols)
-    {
-        auto state(_state->lock());
-
-        auto queryAttribute(state->queryAttribute.use()(key.first)(key.second));
-        if (!queryAttribute.next()) return {};
-
-        auto rowId = (AttrType) queryAttribute.getInt(0);
-        auto type = (AttrType) queryAttribute.getInt(1);
-
-        switch (type) {
-            case AttrType::Placeholder:
-                return {{rowId, placeholder_t()}};
-            case AttrType::FullAttrs: {
-                // FIXME: expensive, should separate this out.
-                std::vector<Symbol> attrs;
-                auto queryAttributes(state->queryAttributes.use()(rowId));
-                while (queryAttributes.next())
-                    attrs.push_back(symbols.create(queryAttributes.getStr(0)));
-                return {{rowId, attrs}};
-            }
-            case AttrType::String:
-                return {{rowId, queryAttribute.getStr(2)}};
-            case AttrType::Bool:
-                return {{rowId, queryAttribute.getInt(2) != 0}};
-            case AttrType::Missing:
-                return {{rowId, missing_t()}};
-            case AttrType::Misc:
-                return {{rowId, misc_t()}};
-            case AttrType::Failed:
-                return {{rowId, failed_t()}};
-            default:
-                throw Error("unexpected type in evaluation cache");
-        }
-    }
-};
-
-EvalCache::EvalCache(
-    bool useCache,
-    const Hash & fingerprint,
-    EvalState & state,
-    RootLoader rootLoader)
-    : db(useCache ? std::make_shared<AttrDb>(fingerprint) : nullptr)
-    , state(state)
-    , rootLoader(rootLoader)
-{
-}
-
-Value * EvalCache::getRootValue()
-{
-    if (!value) {
-        debug("getting root value");
-        value = allocRootValue(rootLoader());
-    }
-    return *value;
-}
-
-std::shared_ptr<AttrCursor> EvalCache::getRoot()
-{
-    return std::make_shared<AttrCursor>(ref(shared_from_this()), std::nullopt);
-}
-
-AttrCursor::AttrCursor(
-    ref<EvalCache> root,
-    Parent parent,
-    Value * value,
-    std::optional<std::pair<AttrId, AttrValue>> && cachedValue)
-    : root(root), parent(parent), cachedValue(std::move(cachedValue))
-{
-    if (value)
-        _value = allocRootValue(value);
-}
-
-AttrKey AttrCursor::getKey()
-{
-    if (!parent)
-        return {0, root->state.sEpsilon};
-    if (!parent->first->cachedValue) {
-        parent->first->cachedValue = root->db->getAttr(
-            parent->first->getKey(), root->state.symbols);
-        assert(parent->first->cachedValue);
-    }
-    return {parent->first->cachedValue->first, parent->second};
-}
-
-Value & AttrCursor::getValue()
-{
-    if (!_value) {
-        if (parent) {
-            auto & vParent = parent->first->getValue();
-            root->state.forceAttrs(vParent);
-            auto attr = vParent.attrs->get(parent->second);
-            if (!attr)
-                throw Error("attribute '%s' is unexpectedly missing", getAttrPathStr());
-            _value = allocRootValue(attr->value);
-        } else
-            _value = allocRootValue(root->getRootValue());
-    }
-    return **_value;
-}
-
-std::vector<Symbol> AttrCursor::getAttrPath() const
-{
-    if (parent) {
-        auto attrPath = parent->first->getAttrPath();
-        attrPath.push_back(parent->second);
-        return attrPath;
-    } else
-        return {};
-}
-
-std::vector<Symbol> AttrCursor::getAttrPath(Symbol name) const
-{
-    auto attrPath = getAttrPath();
-    attrPath.push_back(name);
-    return attrPath;
-}
-
-std::string AttrCursor::getAttrPathStr() const
-{
-    return concatStringsSep(".", getAttrPath());
-}
-
-std::string AttrCursor::getAttrPathStr(Symbol name) const
-{
-    return concatStringsSep(".", getAttrPath(name));
-}
-
-Value & AttrCursor::forceValue()
-{
-    debug("evaluating uncached attribute %s", getAttrPathStr());
-
-    auto & v = getValue();
-
-    try {
-        root->state.forceValue(v);
-    } catch (EvalError &) {
-        debug("setting '%s' to failed", getAttrPathStr());
-        if (root->db)
-            cachedValue = {root->db->setFailed(getKey()), failed_t()};
-        throw;
-    }
-
-    if (root->db && (!cachedValue || std::get_if<placeholder_t>(&cachedValue->second))) {
-        if (v.type == tString)
-            cachedValue = {root->db->setString(getKey(), v.string.s), v.string.s};
-        else if (v.type == tBool)
-            cachedValue = {root->db->setBool(getKey(), v.boolean), v.boolean};
-        else if (v.type == tAttrs)
-            ; // FIXME: do something?
-        else
-            cachedValue = {root->db->setMisc(getKey()), misc_t()};
-    }
-
-    return v;
-}
-
-std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(Symbol name)
-{
-    if (root->db) {
-        if (!cachedValue)
-            cachedValue = root->db->getAttr(getKey(), root->state.symbols);
-
-        if (cachedValue) {
-            if (auto attrs = std::get_if<std::vector<Symbol>>(&cachedValue->second)) {
-                for (auto & attr : *attrs)
-                    if (attr == name)
-                        return std::make_shared<AttrCursor>(root, std::make_pair(shared_from_this(), name));
-                return nullptr;
-            } else if (std::get_if<placeholder_t>(&cachedValue->second)) {
-                auto attr = root->db->getAttr({cachedValue->first, name}, root->state.symbols);
-                if (attr) {
-                    if (std::get_if<missing_t>(&attr->second))
-                        return nullptr;
-                    else if (std::get_if<failed_t>(&attr->second))
-                        throw EvalError("cached failure of attribute '%s'", getAttrPathStr(name));
-                    else
-                        return std::make_shared<AttrCursor>(root,
-                            std::make_pair(shared_from_this(), name), nullptr, std::move(attr));
-                }
-                // Incomplete attrset, so need to fall thru and
-                // evaluate to see whether 'name' exists
-            } else
-                return nullptr;
-                //throw TypeError("'%s' is not an attribute set", getAttrPathStr());
-        }
-    }
-
-    auto & v = forceValue();
-
-    if (v.type != tAttrs)
-        return nullptr;
-        //throw TypeError("'%s' is not an attribute set", getAttrPathStr());
-
-    auto attr = v.attrs->get(name);
-
-    if (!attr) {
-        if (root->db) {
-            if (!cachedValue)
-                cachedValue = {root->db->setPlaceholder(getKey()), placeholder_t()};
-            root->db->setMissing({cachedValue->first, name});
-        }
-        return nullptr;
-    }
-
-    std::optional<std::pair<AttrId, AttrValue>> cachedValue2;
-    if (root->db) {
-        if (!cachedValue)
-            cachedValue = {root->db->setPlaceholder(getKey()), placeholder_t()};
-        cachedValue2 = {root->db->setPlaceholder({cachedValue->first, name}), placeholder_t()};
-    }
-
-    return std::make_shared<AttrCursor>(
-        root, std::make_pair(shared_from_this(), name), attr->value, std::move(cachedValue2));
-}
-
-std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(std::string_view name)
-{
-    return maybeGetAttr(root->state.symbols.create(name));
-}
-
-std::shared_ptr<AttrCursor> AttrCursor::getAttr(Symbol name)
-{
-    auto p = maybeGetAttr(name);
-    if (!p)
-        throw Error("attribute '%s' does not exist", getAttrPathStr(name));
-    return p;
-}
-
-std::shared_ptr<AttrCursor> AttrCursor::getAttr(std::string_view name)
-{
-    return getAttr(root->state.symbols.create(name));
-}
-
-std::shared_ptr<AttrCursor> AttrCursor::findAlongAttrPath(const std::vector<Symbol> & attrPath)
-{
-    auto res = shared_from_this();
-    for (auto & attr : attrPath) {
-        res = res->maybeGetAttr(attr);
-        if (!res) return {};
-    }
-    return res;
-}
-
-std::string AttrCursor::getString()
-{
-    if (root->db) {
-        if (!cachedValue)
-            cachedValue = root->db->getAttr(getKey(), root->state.symbols);
-        if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
-            if (auto s = std::get_if<std::string>(&cachedValue->second)) {
-                debug("using cached string attribute '%s'", getAttrPathStr());
-                return *s;
-            } else
-                throw TypeError("'%s' is not a string", getAttrPathStr());
-        }
-    }
-
-    auto & v = forceValue();
-
-    if (v.type != tString)
-        throw TypeError("'%s' is not a string", getAttrPathStr());
-
-    return v.string.s;
-}
-
-bool AttrCursor::getBool()
-{
-    if (root->db) {
-        if (!cachedValue)
-            cachedValue = root->db->getAttr(getKey(), root->state.symbols);
-        if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
-            if (auto b = std::get_if<bool>(&cachedValue->second)) {
-                debug("using cached Boolean attribute '%s'", getAttrPathStr());
-                return *b;
-            } else
-                throw TypeError("'%s' is not a Boolean", getAttrPathStr());
-        }
-    }
-
-    auto & v = forceValue();
-
-    if (v.type != tBool)
-        throw TypeError("'%s' is not a Boolean", getAttrPathStr());
-
-    return v.boolean;
-}
-
-std::vector<Symbol> AttrCursor::getAttrs()
-{
-    if (root->db) {
-        if (!cachedValue)
-            cachedValue = root->db->getAttr(getKey(), root->state.symbols);
-        if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
-            if (auto attrs = std::get_if<std::vector<Symbol>>(&cachedValue->second)) {
-                debug("using cached attrset attribute '%s'", getAttrPathStr());
-                return *attrs;
-            } else
-                throw TypeError("'%s' is not an attribute set", getAttrPathStr());
-        }
-    }
-
-    auto & v = forceValue();
-
-    if (v.type != tAttrs)
-        throw TypeError("'%s' is not an attribute set", getAttrPathStr());
-
-    std::vector<Symbol> attrs;
-    for (auto & attr : *getValue().attrs)
-        attrs.push_back(attr.name);
-    std::sort(attrs.begin(), attrs.end(), [](const Symbol & a, const Symbol & b) {
-        return (const string &) a < (const string &) b;
-    });
-
-    if (root->db)
-        cachedValue = {root->db->setAttrs(getKey(), attrs), attrs};
-
-    return attrs;
-}
-
-bool AttrCursor::isDerivation()
-{
-    auto aType = maybeGetAttr("type");
-    return aType && aType->getString() == "derivation";
-}
-
-}

src/libexpr/eval-cache.hh

@@ -1,106 +0,0 @@
-#pragma once
-
-#include "sync.hh"
-#include "hash.hh"
-#include "eval.hh"
-
-#include <variant>
-
-namespace nix::eval_cache {
-
-class AttrDb;
-class AttrCursor;
-
-class EvalCache : public std::enable_shared_from_this<EvalCache>
-{
-    friend class AttrCursor;
-
-    std::shared_ptr<AttrDb> db;
-    EvalState & state;
-    typedef std::function<Value *()> RootLoader;
-    RootLoader rootLoader;
-    RootValue value;
-
-    Value * getRootValue();
-
-public:
-
-    EvalCache(
-        bool useCache,
-        const Hash & fingerprint,
-        EvalState & state,
-        RootLoader rootLoader);
-
-    std::shared_ptr<AttrCursor> getRoot();
-};
-
-enum AttrType {
-    Placeholder = 0,
-    FullAttrs = 1,
-    String = 2,
-    Missing = 3,
-    Misc = 4,
-    Failed = 5,
-    Bool = 6,
-};
-
-struct placeholder_t {};
-struct missing_t {};
-struct misc_t {};
-struct failed_t {};
-typedef uint64_t AttrId;
-typedef std::pair<AttrId, Symbol> AttrKey;
-typedef std::variant<std::vector<Symbol>, std::string, placeholder_t, missing_t, misc_t, failed_t, bool> AttrValue;
-
-class AttrCursor : public std::enable_shared_from_this<AttrCursor>
-{
-    friend class EvalCache;
-
-    ref<EvalCache> root;
-    typedef std::optional<std::pair<std::shared_ptr<AttrCursor>, Symbol>> Parent;
-    Parent parent;
-    RootValue _value;
-    std::optional<std::pair<AttrId, AttrValue>> cachedValue;
-
-    AttrKey getKey();
-
-    Value & getValue();
-
-public:
-
-    AttrCursor(
-        ref<EvalCache> root,
-        Parent parent,
-        Value * value = nullptr,
-        std::optional<std::pair<AttrId, AttrValue>> && cachedValue = {});
-
-    std::vector<Symbol> getAttrPath() const;
-
-    std::vector<Symbol> getAttrPath(Symbol name) const;
-
-    std::string getAttrPathStr() const;
-
-    std::string getAttrPathStr(Symbol name) const;
-
-    std::shared_ptr<AttrCursor> maybeGetAttr(Symbol name);
-
-    std::shared_ptr<AttrCursor> maybeGetAttr(std::string_view name);
-
-    std::shared_ptr<AttrCursor> getAttr(Symbol name);
-
-    std::shared_ptr<AttrCursor> getAttr(std::string_view name);
-
-    std::shared_ptr<AttrCursor> findAlongAttrPath(const std::vector<Symbol> & attrPath);
-
-    std::string getString();
-
-    bool getBool();
-
-    std::vector<Symbol> getAttrs();
-
-    bool isDerivation();
-
-    Value & forceValue();
-};
-
-}

src/libexpr/eval-inline.hh

@@ -57,7 +57,7 @@ inline void EvalState::forceAttrs(Value & v)
 
 inline void EvalState::forceAttrs(Value & v, const Pos & pos)
 {
-    forceValue(v, pos);
+    forceValue(v);
     if (v.type != tAttrs)
         throwTypeError("value is %1% while a set was expected, at %2%", v, pos);
 }
@@ -73,7 +73,7 @@ inline void EvalState::forceList(Value & v)
 
 inline void EvalState::forceList(Value & v, const Pos & pos)
 {
-    forceValue(v, pos);
+    forceValue(v);
     if (!v.isList())
         throwTypeError("value is %1% while a list was expected, at %2%", v, pos);
 }

src/libexpr/eval.cc

@@ -5,10 +5,9 @@
 #include "derivations.hh"
 #include "globals.hh"
 #include "eval-inline.hh"
-#include "filetransfer.hh"
+#include "download.hh"
 #include "json.hh"
 #include "function-trace.hh"
-#include "flake/flake.hh"
 
 #include <algorithm>
 #include <chrono>
@@ -23,8 +22,6 @@
 
 #if HAVE_BOEHMGC
 
-#define GC_INCLUDE_NEW
-
 #include <gc/gc.h>
 #include <gc/gc_cpp.h>
 
@@ -59,12 +56,6 @@ static char * dupStringWithLen(const char * s, size_t size)
 }
 
 
-RootValue allocRootValue(Value * v)
-{
-    return std::allocate_shared<Value *>(traceable_allocator<Value *>(), v);
-}
-
-
 static void printValue(std::ostream & str, std::set<const Value *> & active, const Value & v)
 {
     checkInterrupt();
@@ -162,12 +153,12 @@ const Value *getPrimOp(const Value &v) {
 }
 
 
-string showType(ValueType type)
+string showType(const Value & v)
 {
-    switch (type) {
+    switch (v.type) {
         case tInt: return "an integer";
-        case tBool: return "a Boolean";
-        case tString: return "a string";
+        case tBool: return "a boolean";
+        case tString: return v.string.context ? "a string with context" : "a string";
         case tPath: return "a path";
         case tNull: return "null";
         case tAttrs: return "a set";
@@ -176,39 +167,14 @@ string showType(ValueType type)
         case tApp: return "a function application";
         case tLambda: return "a function";
         case tBlackhole: return "a black hole";
-        case tPrimOp: return "a built-in function";
-        case tPrimOpApp: return "a partially applied built-in function";
-        case tExternal: return "an external value";
-        case tFloat: return "a float";
-    }
-    abort();
-}
-
-
-string showType(const Value & v)
-{
-    switch (v.type) {
-        case tString: return v.string.context ? "a string with context" : "a string";
         case tPrimOp:
             return fmt("the built-in function '%s'", string(v.primOp->name));
         case tPrimOpApp:
             return fmt("the partially applied built-in function '%s'", string(getPrimOp(v)->primOp->name));
         case tExternal: return v.external->showType();
-    default:
-        return showType(v.type);
+        case tFloat: return "a float";
     }
-}
-
-
-bool Value::isTrivial() const
-{
-    return
-        type != tApp
-        && type != tPrimOpApp
-        && (type != tThunk
-            || (dynamic_cast<ExprAttrs *>(thunk.expr)
-                && ((ExprAttrs *) thunk.expr)->dynamicAttrs.empty())
-            || dynamic_cast<ExprLambda *>(thunk.expr));
+    abort();
 }
 
 
@@ -349,10 +315,6 @@ EvalState::EvalState(const Strings & _searchPath, ref<Store> store)
     , sOutputHash(symbols.create("outputHash"))
     , sOutputHashAlgo(symbols.create("outputHashAlgo"))
     , sOutputHashMode(symbols.create("outputHashMode"))
-    , sDescription(symbols.create("description"))
-    , sSelf(symbols.create("self"))
-    , sEpsilon(symbols.create(""))
-    , sRecurseForDerivations(symbols.create("recurseForDerivations"))
     , repair(NoRepair)
     , store(store)
     , baseEnv(allocEnv(128))
@@ -501,21 +463,14 @@ Value * EvalState::addConstant(const string & name, Value & v)
 Value * EvalState::addPrimOp(const string & name,
     size_t arity, PrimOpFun primOp)
 {
-    auto name2 = string(name, 0, 2) == "__" ? string(name, 2) : name;
-    Symbol sym = symbols.create(name2);
-
-    /* Hack to make constants lazy: turn them into a application of
-       the primop to a dummy value. */
     if (arity == 0) {
-        auto vPrimOp = allocValue();
-        vPrimOp->type = tPrimOp;
-        vPrimOp->primOp = new PrimOp(primOp, 1, sym);
         Value v;
-        mkApp(v, *vPrimOp, *vPrimOp);
+        primOp(*this, noPos, nullptr, v);
         return addConstant(name, v);
     }
-
     Value * v = allocValue();
+    string name2 = string(name, 0, 2) == "__" ? string(name, 2) : name;
+    Symbol sym = symbols.create(name2);
     v->type = tPrimOp;
     v->primOp = new PrimOp(primOp, arity, sym);
     staticBaseEnv.vars[symbols.create(name)] = baseEnvDispl;
@@ -781,7 +736,7 @@ Value * ExprPath::maybeThunk(EvalState & state, Env & env)
 }
 
 
-void EvalState::evalFile(const Path & path_, Value & v, bool mustBeTrivial)
+void EvalState::evalFile(const Path & path_, Value & v)
 {
     auto path = checkSourcePath(path_);
 
@@ -810,11 +765,6 @@ void EvalState::evalFile(const Path & path_, Value & v, bool mustBeTrivial)
     fileParseCache[path2] = e;
 
     try {
-        // Enforce that 'flake.nix' is a direct attrset, not a
-        // computation.
-        if (mustBeTrivial &&
-            !(dynamic_cast<ExprAttrs *>(e)))
-            throw Error("file '%s' must be an attribute set", path);
         eval(e, v);
     } catch (Error & e) {
         addErrorPrefix(e, "while evaluating the file '%1%':\n", path2);
@@ -1306,7 +1256,7 @@ void ExprWith::eval(EvalState & state, Env & env, Value & v)
 
 void ExprIf::eval(EvalState & state, Env & env, Value & v)
 {
-    (state.evalBool(env, cond, pos) ? then : else_)->eval(state, env, v);
+    (state.evalBool(env, cond) ? then : else_)->eval(state, env, v);
 }
 
 
@@ -1552,7 +1502,7 @@ NixFloat EvalState::forceFloat(Value & v, const Pos & pos)
 
 bool EvalState::forceBool(Value & v, const Pos & pos)
 {
-    forceValue(v, pos);
+    forceValue(v);
     if (v.type != tBool)
         throwTypeError("value is %1% while a Boolean was expected, at %2%", v, pos);
     return v.boolean;
@@ -1567,7 +1517,7 @@ bool EvalState::isFunctor(Value & fun)
 
 void EvalState::forceFunction(Value & v, const Pos & pos)
 {
-    forceValue(v, pos);
+    forceValue(v);
     if (v.type != tLambda && v.type != tPrimOp && v.type != tPrimOpApp && !isFunctor(v))
         throwTypeError("value is %1% while a function was expected, at %2%", v, pos);
 }
@@ -1644,7 +1594,7 @@ std::optional<string> EvalState::tryAttrsToString(const Pos & pos, Value & v,
 string EvalState::coerceToString(const Pos & pos, Value & v, PathSet & context,
     bool coerceMore, bool copyToStore)
 {
-    forceValue(v, pos);
+    forceValue(v);
 
     string s;
 

src/libexpr/eval.hh

@@ -4,13 +4,13 @@
 #include "value.hh"
 #include "nixexpr.hh"
 #include "symbol-table.hh"
+#include "hash.hh"
 #include "config.hh"
 
 #include <regex>
 #include <map>
 #include <optional>
 #include <unordered_map>
-#include <mutex>
 
 
 namespace nix {
@@ -74,8 +74,7 @@ public:
         sSystem, sOverrides, sOutputs, sOutputName, sIgnoreNulls,
         sFile, sLine, sColumn, sFunctor, sToString,
         sRight, sWrong, sStructuredAttrs, sBuilder, sArgs,
-        sOutputHash, sOutputHashAlgo, sOutputHashMode,
-        sDescription, sSelf, sEpsilon, sRecurseForDerivations;
+        sOutputHash, sOutputHashAlgo, sOutputHashMode;
     Symbol sDerivationNix;
 
     /* If set, force copying files to the Nix store even if they
@@ -90,7 +89,6 @@ public:
 
     const ref<Store> store;
 
-
 private:
     SrcToStore srcToStore;
 
@@ -153,9 +151,8 @@ public:
     Expr * parseStdin();
 
     /* Evaluate an expression read from the given file to normal
-       form. Optionally enforce that the top-level expression is
-       trivial (i.e. doesn't require arbitrary computation). */
-    void evalFile(const Path & path, Value & v, bool mustBeTrivial = false);
+       form. */
+    void evalFile(const Path & path, Value & v);
 
     void resetFileCache();
 
@@ -327,7 +324,6 @@ private:
 
 
 /* Return a string representing the type of the value `v'. */
-string showType(ValueType type);
 string showType(const Value & v);
 
 /* Decode a context string ‘!<name>!<path>’ into a pair <path,

src/libexpr/flake/call-flake.nix

@@ -1,29 +0,0 @@
-lockFileStr: rootSrc: rootSubdir:
-
-let
-
-  lockFile = builtins.fromJSON lockFileStr;
-
-  allNodes =
-    builtins.mapAttrs
-      (key: node:
-        let
-          sourceInfo =
-            if key == lockFile.root
-            then rootSrc
-            else fetchTree ({ inherit (node.info) narHash; } // removeAttrs node.locked ["dir"]);
-          subdir = if key == lockFile.root then rootSubdir else node.locked.dir or "";
-          flake = import (sourceInfo + (if subdir != "" then "/" else "") + subdir + "/flake.nix");
-          inputs = builtins.mapAttrs (inputName: key: allNodes.${key}) (node.inputs or {});
-          outputs = flake.outputs (inputs // { self = result; });
-          result = outputs // sourceInfo // { inherit inputs; inherit outputs; inherit sourceInfo; };
-        in
-          if node.flake or true then
-            assert builtins.isFunction flake.outputs;
-            result
-          else
-            sourceInfo
-      )
-      lockFile.nodes;
-
-in allNodes.${lockFile.root}

src/libexpr/flake/flake.cc

@@ -1,660 +0,0 @@
-#include "flake.hh"
-#include "lockfile.hh"
-#include "primops.hh"
-#include "eval-inline.hh"
-#include "store-api.hh"
-#include "fetchers.hh"
-#include "finally.hh"
-
-namespace nix {
-
-using namespace flake;
-
-namespace flake {
-
-/* If 'allowLookup' is true, then resolve 'flakeRef' using the
-   registries. */
-static FlakeRef maybeLookupFlake(
-    ref<Store> store,
-    const FlakeRef & flakeRef,
-    bool allowLookup)
-{
-    if (!flakeRef.input->isDirect()) {
-        if (allowLookup)
-            return flakeRef.resolve(store);
-        else
-            throw Error("'%s' is an indirect flake reference, but registry lookups are not allowed", flakeRef);
-    } else
-        return flakeRef;
-}
-
-typedef std::vector<std::pair<FlakeRef, FlakeRef>> FlakeCache;
-
-static FlakeRef lookupInFlakeCache(
-    const FlakeCache & flakeCache,
-    const FlakeRef & flakeRef)
-{
-    // FIXME: inefficient.
-    for (auto & i : flakeCache) {
-        if (flakeRef == i.first) {
-            debug("mapping '%s' to previously seen input '%s' -> '%s",
-                flakeRef, i.first, i.second);
-            return i.second;
-        }
-    }
-
-    return flakeRef;
-}
-
-static std::tuple<fetchers::Tree, FlakeRef, FlakeRef> fetchOrSubstituteTree(
-    EvalState & state,
-    const FlakeRef & originalRef,
-    std::optional<TreeInfo> treeInfo,
-    bool allowLookup,
-    FlakeCache & flakeCache)
-{
-    /* The tree may already be in the Nix store, or it could be
-       substituted (which is often faster than fetching from the
-       original source). So check that. */
-    if (treeInfo && originalRef.input->isDirect() && originalRef.input->isImmutable()) {
-        try {
-            auto storePath = treeInfo->computeStorePath(*state.store);
-
-            state.store->ensurePath(storePath);
-
-            debug("using substituted/cached input '%s' in '%s'",
-                originalRef, state.store->printStorePath(storePath));
-
-            auto actualPath = state.store->toRealPath(storePath);
-
-            if (state.allowedPaths)
-                state.allowedPaths->insert(actualPath);
-
-            return {
-                Tree {
-                    .actualPath = actualPath,
-                    .storePath = std::move(storePath),
-                    .info = *treeInfo,
-                },
-                originalRef,
-                originalRef
-            };
-        } catch (Error & e) {
-            debug("substitution of input '%s' failed: %s", originalRef, e.what());
-        }
-    }
-
-    auto resolvedRef = lookupInFlakeCache(flakeCache,
-        maybeLookupFlake(state.store,
-            lookupInFlakeCache(flakeCache, originalRef), allowLookup));
-
-    auto [tree, lockedRef] = resolvedRef.fetchTree(state.store);
-
-    debug("got tree '%s' from '%s'",
-        state.store->printStorePath(tree.storePath), lockedRef);
-
-    flakeCache.push_back({originalRef, lockedRef});
-    flakeCache.push_back({resolvedRef, lockedRef});
-
-    if (state.allowedPaths)
-        state.allowedPaths->insert(tree.actualPath);
-
-    if (treeInfo)
-        assert(tree.storePath == treeInfo->computeStorePath(*state.store));
-
-    return {std::move(tree), resolvedRef, lockedRef};
-}
-
-static void expectType(EvalState & state, ValueType type,
-    Value & value, const Pos & pos)
-{
-    if (value.type == tThunk && value.isTrivial())
-        state.forceValue(value, pos);
-    if (value.type != type)
-        throw Error("expected %s but got %s at %s",
-            showType(type), showType(value.type), pos);
-}
-
-static std::map<FlakeId, FlakeInput> parseFlakeInputs(
-    EvalState & state, Value * value, const Pos & pos);
-
-static FlakeInput parseFlakeInput(EvalState & state,
-    const std::string & inputName, Value * value, const Pos & pos)
-{
-    expectType(state, tAttrs, *value, pos);
-
-    FlakeInput input {
-        .ref = FlakeRef::fromAttrs({{"type", "indirect"}, {"id", inputName}})
-    };
-
-    auto sInputs = state.symbols.create("inputs");
-    auto sUrl = state.symbols.create("url");
-    auto sFlake = state.symbols.create("flake");
-    auto sFollows = state.symbols.create("follows");
-
-    fetchers::Attrs attrs;
-    std::optional<std::string> url;
-
-    for (nix::Attr attr : *(value->attrs)) {
-        try {
-            if (attr.name == sUrl) {
-                expectType(state, tString, *attr.value, *attr.pos);
-                url = attr.value->string.s;
-                attrs.emplace("url", *url);
-            } else if (attr.name == sFlake) {
-                expectType(state, tBool, *attr.value, *attr.pos);
-                input.isFlake = attr.value->boolean;
-            } else if (attr.name == sInputs) {
-                input.overrides = parseFlakeInputs(state, attr.value, *attr.pos);
-            } else if (attr.name == sFollows) {
-                expectType(state, tString, *attr.value, *attr.pos);
-                input.follows = parseInputPath(attr.value->string.s);
-            } else {
-                state.forceValue(*attr.value);
-                if (attr.value->type == tString)
-                    attrs.emplace(attr.name, attr.value->string.s);
-                else
-                    throw TypeError("flake input attribute '%s' is %s while a string is expected",
-                        attr.name, showType(*attr.value));
-            }
-        } catch (Error & e) {
-            e.addPrefix(fmt("in flake attribute '%s' at '%s':\n", attr.name, *attr.pos));
-            throw;
-        }
-    }
-
-    if (attrs.count("type"))
-        try {
-            input.ref = FlakeRef::fromAttrs(attrs);
-        } catch (Error & e) {
-            e.addPrefix(fmt("in flake input at '%s':\n", pos));
-            throw;
-        }
-    else {
-        attrs.erase("url");
-        if (!attrs.empty())
-            throw Error("unexpected flake input attribute '%s', at %s", attrs.begin()->first, pos);
-        if (url)
-            input.ref = parseFlakeRef(*url, {}, true);
-    }
-
-    return input;
-}
-
-static std::map<FlakeId, FlakeInput> parseFlakeInputs(
-    EvalState & state, Value * value, const Pos & pos)
-{
-    std::map<FlakeId, FlakeInput> inputs;
-
-    expectType(state, tAttrs, *value, pos);
-
-    for (nix::Attr & inputAttr : *(*value).attrs) {
-        inputs.emplace(inputAttr.name,
-            parseFlakeInput(state,
-                inputAttr.name,
-                inputAttr.value,
-                *inputAttr.pos));
-    }
-
-    return inputs;
-}
-
-static Flake getFlake(
-    EvalState & state,
-    const FlakeRef & originalRef,
-    std::optional<TreeInfo> treeInfo,
-    bool allowLookup,
-    FlakeCache & flakeCache)
-{
-    auto [sourceInfo, resolvedRef, lockedRef] = fetchOrSubstituteTree(
-        state, originalRef, treeInfo, allowLookup, flakeCache);
-
-    // Guard against symlink attacks.
-    auto flakeFile = canonPath(sourceInfo.actualPath + "/" + lockedRef.subdir + "/flake.nix");
-    if (!isInDir(flakeFile, sourceInfo.actualPath))
-        throw Error("'flake.nix' file of flake '%s' escapes from '%s'",
-            lockedRef, state.store->printStorePath(sourceInfo.storePath));
-
-    Flake flake {
-        .originalRef = originalRef,
-        .resolvedRef = resolvedRef,
-        .lockedRef = lockedRef,
-        .sourceInfo = std::make_shared<fetchers::Tree>(std::move(sourceInfo))
-    };
-
-    if (!pathExists(flakeFile))
-        throw Error("source tree referenced by '%s' does not contain a '%s/flake.nix' file", lockedRef, lockedRef.subdir);
-
-    Value vInfo;
-    state.evalFile(flakeFile, vInfo, true); // FIXME: symlink attack
-
-    expectType(state, tAttrs, vInfo, Pos(state.symbols.create(flakeFile), 0, 0));
-
-    auto sEdition = state.symbols.create("edition"); // FIXME: remove soon
-
-    if (vInfo.attrs->get(sEdition))
-        warn("flake '%s' has deprecated attribute 'edition'", lockedRef);
-
-    if (auto description = vInfo.attrs->get(state.sDescription)) {
-        expectType(state, tString, *description->value, *description->pos);
-        flake.description = description->value->string.s;
-    }
-
-    auto sInputs = state.symbols.create("inputs");
-
-    if (auto inputs = vInfo.attrs->get(sInputs))
-        flake.inputs = parseFlakeInputs(state, inputs->value, *inputs->pos);
-
-    auto sOutputs = state.symbols.create("outputs");
-
-    if (auto outputs = vInfo.attrs->get(sOutputs)) {
-        expectType(state, tLambda, *outputs->value, *outputs->pos);
-        flake.vOutputs = allocRootValue(outputs->value);
-
-        if ((*flake.vOutputs)->lambda.fun->matchAttrs) {
-            for (auto & formal : (*flake.vOutputs)->lambda.fun->formals->formals) {
-                if (formal.name != state.sSelf)
-                    flake.inputs.emplace(formal.name, FlakeInput {
-                        .ref = parseFlakeRef(formal.name)
-                    });
-            }
-        }
-
-    } else
-        throw Error("flake '%s' lacks attribute 'outputs'", lockedRef);
-
-    for (auto & attr : *vInfo.attrs) {
-        if (attr.name != sEdition &&
-            attr.name != state.sDescription &&
-            attr.name != sInputs &&
-            attr.name != sOutputs)
-            throw Error("flake '%s' has an unsupported attribute '%s', at %s",
-                lockedRef, attr.name, *attr.pos);
-    }
-
-    return flake;
-}
-
-Flake getFlake(EvalState & state, const FlakeRef & originalRef, bool allowLookup)
-{
-    FlakeCache flakeCache;
-    return getFlake(state, originalRef, {}, allowLookup, flakeCache);
-}
-
-/* Compute an in-memory lock file for the specified top-level flake,
-   and optionally write it to file, it the flake is writable. */
-LockedFlake lockFlake(
-    EvalState & state,
-    const FlakeRef & topRef,
-    const LockFlags & lockFlags)
-{
-    settings.requireExperimentalFeature("flakes");
-
-    FlakeCache flakeCache;
-
-    auto flake = getFlake(state, topRef, {}, lockFlags.useRegistries, flakeCache);
-
-    // FIXME: symlink attack
-    auto oldLockFile = LockFile::read(
-        flake.sourceInfo->actualPath + "/" + flake.lockedRef.subdir + "/flake.lock");
-
-    debug("old lock file: %s", oldLockFile);
-
-    // FIXME: check whether all overrides are used.
-    std::map<InputPath, FlakeInput> overrides;
-    std::set<InputPath> overridesUsed, updatesUsed;
-
-    for (auto & i : lockFlags.inputOverrides)
-        overrides.insert_or_assign(i.first, FlakeInput { .ref = i.second });
-
-    LockFile newLockFile;
-
-    std::vector<FlakeRef> parents;
-    std::map<InputPath, InputPath> follows;
-
-    std::function<void(
-        const FlakeInputs & flakeInputs,
-        std::shared_ptr<Node> node,
-        const InputPath & inputPathPrefix,
-        std::shared_ptr<const Node> oldNode)>
-        computeLocks;
-
-    computeLocks = [&](
-        const FlakeInputs & flakeInputs,
-        std::shared_ptr<Node> node,
-        const InputPath & inputPathPrefix,
-        std::shared_ptr<const Node> oldNode)
-    {
-        debug("computing lock file node '%s'", concatStringsSep("/", inputPathPrefix));
-
-        /* Get the overrides (i.e. attributes of the form
-           'inputs.nixops.inputs.nixpkgs.url = ...'). */
-        // FIXME: check this
-        for (auto & [id, input] : flake.inputs) {
-            for (auto & [idOverride, inputOverride] : input.overrides) {
-                auto inputPath(inputPathPrefix);
-                inputPath.push_back(id);
-                inputPath.push_back(idOverride);
-                overrides.insert_or_assign(inputPath, inputOverride);
-            }
-        }
-
-        /* Go over the flake inputs, resolve/fetch them if
-           necessary (i.e. if they're new or the flakeref changed
-           from what's in the lock file). */
-        for (auto & [id, input2] : flakeInputs) {
-            auto inputPath(inputPathPrefix);
-            inputPath.push_back(id);
-            auto inputPathS = concatStringsSep("/", inputPath);
-            debug("computing input '%s'", concatStringsSep("/", inputPath));
-
-            /* Do we have an override for this input from one of the
-               ancestors? */
-            auto i = overrides.find(inputPath);
-            bool hasOverride = i != overrides.end();
-            if (hasOverride) overridesUsed.insert(inputPath);
-            auto & input = hasOverride ? i->second : input2;
-
-            /* Resolve 'follows' later (since it may refer to an input
-               path we haven't processed yet. */
-            if (input.follows) {
-                if (hasOverride)
-                    /* 'follows' from an override is relative to the
-                       root of the graph. */
-                    follows.insert_or_assign(inputPath, *input.follows);
-                else {
-                    /* Otherwise, it's relative to the current flake. */
-                    InputPath path(inputPathPrefix);
-                    for (auto & i : *input.follows) path.push_back(i);
-                    follows.insert_or_assign(inputPath, path);
-                }
-                continue;
-            }
-
-            /* Do we have an entry in the existing lock file? And we
-               don't have a --update-input flag for this input? */
-            std::shared_ptr<const LockedNode> oldLock;
-
-            updatesUsed.insert(inputPath);
-
-            if (oldNode && !lockFlags.inputUpdates.count(inputPath)) {
-                auto oldLockIt = oldNode->inputs.find(id);
-                if (oldLockIt != oldNode->inputs.end())
-                    oldLock = std::dynamic_pointer_cast<const LockedNode>(oldLockIt->second);
-            }
-
-            if (oldLock
-                && oldLock->originalRef == input.ref
-                && !hasOverride)
-            {
-                debug("keeping existing input '%s'", inputPathS);
-
-                /* Copy the input from the old lock since its flakeref
-                   didn't change and there is no override from a
-                   higher level flake. */
-                auto childNode = std::make_shared<LockedNode>(
-                    oldLock->lockedRef, oldLock->originalRef, oldLock->info, oldLock->isFlake);
-
-                node->inputs.insert_or_assign(id, childNode);
-
-                /* If we have an --update-input flag for an input
-                   of this input, then we must fetch the flake to
-                   to update it. */
-                auto lb = lockFlags.inputUpdates.lower_bound(inputPath);
-
-                auto hasChildUpdate =
-                    lb != lockFlags.inputUpdates.end()
-                    && lb->size() > inputPath.size()
-                    && std::equal(inputPath.begin(), inputPath.end(), lb->begin());
-
-                if (hasChildUpdate) {
-                    auto inputFlake = getFlake(
-                        state, oldLock->lockedRef, oldLock->info, false, flakeCache);
-                    computeLocks(inputFlake.inputs, childNode, inputPath, oldLock);
-                } else {
-                    /* No need to fetch this flake, we can be
-                       lazy. However there may be new overrides on the
-                       inputs of this flake, so we need to check
-                       those. */
-                    FlakeInputs fakeInputs;
-
-                    for (auto & i : oldLock->inputs) {
-                        auto lockedNode = std::dynamic_pointer_cast<LockedNode>(i.second);
-                        // Note: this node is not locked in case
-                        // of a circular reference back to the root.
-                        if (lockedNode)
-                            fakeInputs.emplace(i.first, FlakeInput {
-                                .ref = lockedNode->originalRef
-                            });
-                        else {
-                            InputPath path(inputPath);
-                            path.push_back(i.first);
-                            follows.insert_or_assign(path, InputPath());
-                        }
-                    }
-
-                    computeLocks(fakeInputs, childNode, inputPath, oldLock);
-                }
-
-            } else {
-                /* We need to create a new lock file entry. So fetch
-                   this input. */
-
-                if (!lockFlags.allowMutable && !input.ref.input->isImmutable())
-                    throw Error("cannot update flake input '%s' in pure mode", inputPathS);
-
-                if (input.isFlake) {
-                    auto inputFlake = getFlake(state, input.ref, {}, lockFlags.useRegistries, flakeCache);
-
-                    /* Note: in case of an --override-input, we use
-                       the *original* ref (input2.ref) for the
-                       "original" field, rather than the
-                       override. This ensures that the override isn't
-                       nuked the next time we update the lock
-                       file. That is, overrides are sticky unless you
-                       use --no-write-lock-file. */
-                    auto childNode = std::make_shared<LockedNode>(
-                        inputFlake.lockedRef, input2.ref, inputFlake.sourceInfo->info);
-
-                    node->inputs.insert_or_assign(id, childNode);
-
-                    /* Guard against circular flake imports. */
-                    for (auto & parent : parents)
-                        if (parent == input.ref)
-                            throw Error("found circular import of flake '%s'", parent);
-                    parents.push_back(input.ref);
-                    Finally cleanup([&]() { parents.pop_back(); });
-
-                    /* Recursively process the inputs of this
-                       flake. Also, unless we already have this flake
-                       in the top-level lock file, use this flake's
-                       own lock file. */
-                    computeLocks(
-                        inputFlake.inputs, childNode, inputPath,
-                        oldLock
-                        ? std::dynamic_pointer_cast<const Node>(oldLock)
-                        : LockFile::read(
-                            inputFlake.sourceInfo->actualPath + "/" + inputFlake.lockedRef.subdir + "/flake.lock").root);
-                }
-
-                else {
-                    auto [sourceInfo, resolvedRef, lockedRef] = fetchOrSubstituteTree(
-                        state, input.ref, {}, lockFlags.useRegistries, flakeCache);
-                    node->inputs.insert_or_assign(id,
-                        std::make_shared<LockedNode>(lockedRef, input.ref, sourceInfo.info, false));
-                }
-            }
-        }
-    };
-
-    computeLocks(
-        flake.inputs, newLockFile.root, {},
-        lockFlags.recreateLockFile ? nullptr : oldLockFile.root);
-
-    /* Insert edges for 'follows' overrides. */
-    for (auto & [from, to] : follows) {
-        debug("adding 'follows' node from '%s' to '%s'",
-            concatStringsSep("/", from),
-            concatStringsSep("/", to));
-
-        assert(!from.empty());
-
-        InputPath fromParent(from);
-        fromParent.pop_back();
-
-        auto fromParentNode = newLockFile.root->findInput(fromParent);
-        assert(fromParentNode);
-
-        auto toNode = newLockFile.root->findInput(to);
-        if (!toNode)
-            throw Error("flake input '%s' follows non-existent flake input '%s'",
-                concatStringsSep("/", from),
-                concatStringsSep("/", to));
-
-        fromParentNode->inputs.insert_or_assign(from.back(), toNode);
-    }
-
-    for (auto & i : lockFlags.inputOverrides)
-        if (!overridesUsed.count(i.first))
-            warn("the flag '--override-input %s %s' does not match any input",
-                concatStringsSep("/", i.first), i.second);
-
-    for (auto & i : lockFlags.inputUpdates)
-        if (!updatesUsed.count(i))
-            warn("the flag '--update-input %s' does not match any input", concatStringsSep("/", i));
-
-    debug("new lock file: %s", newLockFile);
-
-    /* Check whether we need to / can write the new lock file. */
-    if (!(newLockFile == oldLockFile)) {
-
-        auto diff = diffLockFiles(oldLockFile, newLockFile);
-
-        if (!(oldLockFile == LockFile()))
-            printInfo("inputs of flake '%s' changed:\n%s", topRef, chomp(diff));
-
-        if (lockFlags.writeLockFile) {
-            if (auto sourcePath = topRef.input->getSourcePath()) {
-                if (!newLockFile.isImmutable()) {
-                    if (settings.warnDirty)
-                        warn("will not write lock file of flake '%s' because it has a mutable input", topRef);
-                } else {
-                    if (!lockFlags.updateLockFile)
-                        throw Error("flake '%s' requires lock file changes but they're not allowed due to '--no-update-lock-file'", topRef);
-
-                    auto relPath = (topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock";
-
-                    auto path = *sourcePath + "/" + relPath;
-
-                    bool lockFileExists = pathExists(path);
-
-                    if (lockFileExists)
-                        warn("updating lock file '%s'", path);
-                    else
-                        warn("creating lock file '%s'", path);
-
-                    newLockFile.write(path);
-
-                    topRef.input->markChangedFile(
-                        (topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock",
-                        lockFlags.commitLockFile
-                        ? std::optional<std::string>(fmt("%s: %s\n\nFlake input changes:\n\n%s",
-                                relPath, lockFileExists ? "Update" : "Add", diff))
-                        : std::nullopt);
-
-                    /* Rewriting the lockfile changed the top-level
-                       repo, so we should re-read it. FIXME: we could
-                       also just clear the 'rev' field... */
-                    auto prevLockedRef = flake.lockedRef;
-                    FlakeCache dummyCache;
-                    flake = getFlake(state, topRef, {}, lockFlags.useRegistries, dummyCache);
-
-                    if (lockFlags.commitLockFile &&
-                        flake.lockedRef.input->getRev() &&
-                        prevLockedRef.input->getRev() != flake.lockedRef.input->getRev())
-                        warn("committed new revision '%s'", flake.lockedRef.input->getRev()->gitRev());
-
-                    /* Make sure that we picked up the change,
-                       i.e. the tree should usually be dirty
-                       now. Corner case: we could have reverted from a
-                       dirty to a clean tree! */
-                    if (flake.lockedRef.input == prevLockedRef.input
-                        && !flake.lockedRef.input->isImmutable())
-                        throw Error("'%s' did not change after I updated its 'flake.lock' file; is 'flake.lock' under version control?", flake.originalRef);
-                }
-            } else
-                throw Error("cannot write modified lock file of flake '%s' (use '--no-write-lock-file' to ignore)", topRef);
-        } else
-            warn("not writing modified lock file of flake '%s'", topRef);
-    }
-
-    return LockedFlake { .flake = std::move(flake), .lockFile = std::move(newLockFile) };
-}
-
-void callFlake(EvalState & state,
-    const LockedFlake & lockedFlake,
-    Value & vRes)
-{
-    auto vLocks = state.allocValue();
-    auto vRootSrc = state.allocValue();
-    auto vRootSubdir = state.allocValue();
-    auto vTmp1 = state.allocValue();
-    auto vTmp2 = state.allocValue();
-
-    mkString(*vLocks, lockedFlake.lockFile.to_string());
-
-    emitTreeAttrs(state, *lockedFlake.flake.sourceInfo, lockedFlake.flake.lockedRef.input, *vRootSrc);
-
-    mkString(*vRootSubdir, lockedFlake.flake.lockedRef.subdir);
-
-    static RootValue vCallFlake = nullptr;
-
-    if (!vCallFlake) {
-        vCallFlake = allocRootValue(state.allocValue());
-        state.eval(state.parseExprFromString(
-            #include "call-flake.nix.gen.hh"
-            , "/"), **vCallFlake);
-    }
-
-    state.callFunction(**vCallFlake, *vLocks, *vTmp1, noPos);
-    state.callFunction(*vTmp1, *vRootSrc, *vTmp2, noPos);
-    state.callFunction(*vTmp2, *vRootSubdir, vRes, noPos);
-}
-
-static void prim_getFlake(EvalState & state, const Pos & pos, Value * * args, Value & v)
-{
-    auto flakeRefS = state.forceStringNoCtx(*args[0], pos);
-    auto flakeRef = parseFlakeRef(flakeRefS, {}, true);
-    if (evalSettings.pureEval && !flakeRef.input->isImmutable())
-        throw Error("cannot call 'getFlake' on mutable flake reference '%s', at %s (use --impure to override)", flakeRefS, pos);
-
-    callFlake(state,
-        lockFlake(state, flakeRef,
-            LockFlags {
-                .updateLockFile = false,
-                .useRegistries = !evalSettings.pureEval,
-                .allowMutable  = !evalSettings.pureEval,
-            }),
-        v);
-}
-
-static RegisterPrimOp r2("getFlake", 1, prim_getFlake);
-
-}
-
-Fingerprint LockedFlake::getFingerprint() const
-{
-    // FIXME: as an optimization, if the flake contains a lock file
-    // and we haven't changed it, then it's sufficient to use
-    // flake.sourceInfo.storePath for the fingerprint.
-    return hashString(htSHA256,
-        fmt("%s;%d;%d;%s",
-            flake.sourceInfo->storePath.to_string(),
-            flake.sourceInfo->info.revCount.value_or(0),
-            flake.sourceInfo->info.lastModified.value_or(0),
-            lockFile));
-}
-
-Flake::~Flake() { }
-
-}

src/libexpr/flake/flake.hh

@@ -1,110 +0,0 @@
-#pragma once
-
-#include "types.hh"
-#include "flakeref.hh"
-#include "lockfile.hh"
-#include "value.hh"
-
-namespace nix {
-
-class EvalState;
-
-namespace fetchers { struct Tree; }
-
-namespace flake {
-
-struct FlakeInput;
-
-typedef std::map<FlakeId, FlakeInput> FlakeInputs;
-
-struct FlakeInput
-{
-    FlakeRef ref;
-    bool isFlake = true;
-    std::optional<InputPath> follows;
-    FlakeInputs overrides;
-};
-
-struct Flake
-{
-    FlakeRef originalRef;
-    FlakeRef resolvedRef;
-    FlakeRef lockedRef;
-    std::optional<std::string> description;
-    std::shared_ptr<const fetchers::Tree> sourceInfo;
-    FlakeInputs inputs;
-    RootValue vOutputs;
-    ~Flake();
-};
-
-Flake getFlake(EvalState & state, const FlakeRef & flakeRef, bool allowLookup);
-
-/* Fingerprint of a locked flake; used as a cache key. */
-typedef Hash Fingerprint;
-
-struct LockedFlake
-{
-    Flake flake;
-    LockFile lockFile;
-
-    Fingerprint getFingerprint() const;
-};
-
-struct LockFlags
-{
-    /* Whether to ignore the existing lock file, creating a new one
-       from scratch. */
-    bool recreateLockFile = false;
-
-    /* Whether to update the lock file at all. If set to false, if any
-       change to the lock file is needed (e.g. when an input has been
-       added to flake.nix), you get a fatal error. */
-    bool updateLockFile = true;
-
-    /* Whether to write the lock file to disk. If set to true, if the
-       any changes to the lock file are needed and the flake is not
-       writable (i.e. is not a local Git working tree or similar), you
-       get a fatal error. If set to false, Nix will use the modified
-       lock file in memory only, without writing it to disk. */
-    bool writeLockFile = true;
-
-    /* Whether to use the registries to lookup indirect flake
-       references like 'nixpkgs'. */
-    bool useRegistries = true;
-
-    /* Whether mutable flake references (i.e. those without a Git
-       revision or similar) without a corresponding lock are
-       allowed. Mutable flake references with a lock are always
-       allowed. */
-    bool allowMutable = true;
-
-    /* Whether to commit changes to flake.lock. */
-    bool commitLockFile = false;
-
-    /* Flake inputs to be overriden. */
-    std::map<InputPath, FlakeRef> inputOverrides;
-
-    /* Flake inputs to be updated. This means that any existing lock
-       for those inputs will be ignored. */
-    std::set<InputPath> inputUpdates;
-};
-
-LockedFlake lockFlake(
-    EvalState & state,
-    const FlakeRef & flakeRef,
-    const LockFlags & lockFlags);
-
-void callFlake(
-    EvalState & state,
-    const LockedFlake & lockedFlake,
-    Value & v);
-
-}
-
-void emitTreeAttrs(
-    EvalState & state,
-    const fetchers::Tree & tree,
-    std::shared_ptr<const fetchers::Input> input,
-    Value & v);
-
-}

src/libexpr/flake/flakeref.cc

@@ -1,196 +0,0 @@
-#include "flakeref.hh"
-#include "store-api.hh"
-#include "url.hh"
-#include "fetchers.hh"
-#include "registry.hh"
-
-namespace nix {
-
-#if 0
-// 'dir' path elements cannot start with a '.'. We also reject
-// potentially dangerous characters like ';'.
-const static std::string subDirElemRegex = "(?:[a-zA-Z0-9_-]+[a-zA-Z0-9._-]*)";
-const static std::string subDirRegex = subDirElemRegex + "(?:/" + subDirElemRegex + ")*";
-#endif
-
-std::string FlakeRef::to_string() const
-{
-    auto url = input->toURL();
-    if (subdir != "")
-        url.query.insert_or_assign("dir", subdir);
-    return url.to_string();
-}
-
-fetchers::Attrs FlakeRef::toAttrs() const
-{
-    auto attrs = input->toAttrs();
-    if (subdir != "")
-        attrs.emplace("dir", subdir);
-    return attrs;
-}
-
-std::ostream & operator << (std::ostream & str, const FlakeRef & flakeRef)
-{
-    str << flakeRef.to_string();
-    return str;
-}
-
-bool FlakeRef::operator ==(const FlakeRef & other) const
-{
-    return *input == *other.input && subdir == other.subdir;
-}
-
-FlakeRef FlakeRef::resolve(ref<Store> store) const
-{
-    auto [input2, extraAttrs] = lookupInRegistries(store, input);
-    return FlakeRef(input2, fetchers::maybeGetStrAttr(extraAttrs, "dir").value_or(subdir));
-}
-
-FlakeRef parseFlakeRef(
-    const std::string & url, const std::optional<Path> & baseDir, bool allowMissing)
-{
-    auto [flakeRef, fragment] = parseFlakeRefWithFragment(url, baseDir, allowMissing);
-    if (fragment != "")
-        throw Error("unexpected fragment '%s' in flake reference '%s'", fragment, url);
-    return flakeRef;
-}
-
-std::optional<FlakeRef> maybeParseFlakeRef(
-    const std::string & url, const std::optional<Path> & baseDir)
-{
-    try {
-        return parseFlakeRef(url, baseDir);
-    } catch (Error &) {
-        return {};
-    }
-}
-
-std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
-    const std::string & url, const std::optional<Path> & baseDir, bool allowMissing)
-{
-    using namespace fetchers;
-
-    static std::string fnRegex = "[0-9a-zA-Z-._~!$&'\"()*+,;=]+";
-
-    static std::regex pathUrlRegex(
-        "(/?" + fnRegex + "(?:/" + fnRegex + ")*/?)"
-        + "(?:\\?(" + queryRegex + "))?"
-        + "(?:#(" + queryRegex + "))?",
-        std::regex::ECMAScript);
-
-    static std::regex flakeRegex(
-        "((" + flakeIdRegexS + ")(?:/(?:" + refAndOrRevRegex + "))?)"
-        + "(?:#(" + queryRegex + "))?",
-        std::regex::ECMAScript);
-
-    std::smatch match;
-
-    /* Check if 'url' is a flake ID. This is an abbreviated syntax for
-       'flake:<flake-id>?ref=<ref>&rev=<rev>'. */
-
-    if (std::regex_match(url, match, flakeRegex)) {
-        auto parsedURL = ParsedURL{
-            .url = url,
-            .base = "flake:" + std::string(match[1]),
-            .scheme = "flake",
-            .authority = "",
-            .path = match[1],
-        };
-
-        return std::make_pair(
-            FlakeRef(inputFromURL(parsedURL), ""),
-            percentDecode(std::string(match[6])));
-    }
-
-    /* Check if 'url' is a path (either absolute or relative to
-       'baseDir'). If so, search upward to the root of the repo
-       (i.e. the directory containing .git). */
-
-    else if (std::regex_match(url, match, pathUrlRegex)) {
-        std::string path = match[1];
-        if (!baseDir && !hasPrefix(path, "/"))
-            throw BadURL("flake reference '%s' is not an absolute path", url);
-        path = absPath(path, baseDir, true);
-
-        if (!S_ISDIR(lstat(path).st_mode))
-            throw BadURL("path '%s' is not a flake (because it's not a directory)", path);
-
-        if (!allowMissing && !pathExists(path + "/flake.nix"))
-            throw BadURL("path '%s' is not a flake (because it doesn't contain a 'flake.nix' file)", path);
-
-        auto fragment = percentDecode(std::string(match[3]));
-
-        auto flakeRoot = path;
-        std::string subdir;
-
-        while (flakeRoot != "/") {
-            if (pathExists(flakeRoot + "/.git")) {
-                auto base = std::string("git+file://") + flakeRoot;
-
-                auto parsedURL = ParsedURL{
-                    .url = base, // FIXME
-                    .base = base,
-                    .scheme = "git+file",
-                    .authority = "",
-                    .path = flakeRoot,
-                    .query = decodeQuery(match[2]),
-                };
-
-                if (subdir != "") {
-                    if (parsedURL.query.count("dir"))
-                        throw Error("flake URL '%s' has an inconsistent 'dir' parameter", url);
-                    parsedURL.query.insert_or_assign("dir", subdir);
-                }
-
-                return std::make_pair(
-                    FlakeRef(inputFromURL(parsedURL), get(parsedURL.query, "dir").value_or("")),
-                    fragment);
-            }
-
-            subdir = std::string(baseNameOf(flakeRoot)) + (subdir.empty() ? "" : "/" + subdir);
-            flakeRoot = dirOf(flakeRoot);
-        }
-
-        fetchers::Attrs attrs;
-        attrs.insert_or_assign("type", "path");
-        attrs.insert_or_assign("path", path);
-
-        return std::make_pair(FlakeRef(inputFromAttrs(attrs), ""), fragment);
-    }
-
-    else {
-        auto parsedURL = parseURL(url);
-        std::string fragment;
-        std::swap(fragment, parsedURL.fragment);
-        return std::make_pair(
-            FlakeRef(inputFromURL(parsedURL), get(parsedURL.query, "dir").value_or("")),
-            fragment);
-    }
-}
-
-std::optional<std::pair<FlakeRef, std::string>> maybeParseFlakeRefWithFragment(
-    const std::string & url, const std::optional<Path> & baseDir)
-{
-    try {
-        return parseFlakeRefWithFragment(url, baseDir);
-    } catch (Error & e) {
-        return {};
-    }
-}
-
-FlakeRef FlakeRef::fromAttrs(const fetchers::Attrs & attrs)
-{
-    auto attrs2(attrs);
-    attrs2.erase("dir");
-    return FlakeRef(
-        fetchers::inputFromAttrs(attrs2),
-        fetchers::maybeGetStrAttr(attrs, "dir").value_or(""));
-}
-
-std::pair<fetchers::Tree, FlakeRef> FlakeRef::fetchTree(ref<Store> store) const
-{
-    auto [tree, lockedInput] = input->fetchTree(store);
-    return {std::move(tree), FlakeRef(lockedInput, subdir)};
-}
-
-}

src/libexpr/flake/flakeref.hh

@@ -1,55 +0,0 @@
-#pragma once
-
-#include "types.hh"
-#include "hash.hh"
-#include "fetchers.hh"
-
-#include <variant>
-
-namespace nix {
-
-class Store;
-
-typedef std::string FlakeId;
-
-struct FlakeRef
-{
-    std::shared_ptr<const fetchers::Input> input;
-
-    Path subdir;
-
-    bool operator==(const FlakeRef & other) const;
-
-    FlakeRef(const std::shared_ptr<const fetchers::Input> & input, const Path & subdir)
-        : input(input), subdir(subdir)
-    {
-        assert(input);
-    }
-
-    // FIXME: change to operator <<.
-    std::string to_string() const;
-
-    fetchers::Attrs toAttrs() const;
-
-    FlakeRef resolve(ref<Store> store) const;
-
-    static FlakeRef fromAttrs(const fetchers::Attrs & attrs);
-
-    std::pair<fetchers::Tree, FlakeRef> fetchTree(ref<Store> store) const;
-};
-
-std::ostream & operator << (std::ostream & str, const FlakeRef & flakeRef);
-
-FlakeRef parseFlakeRef(
-    const std::string & url, const std::optional<Path> & baseDir = {}, bool allowMissing = false);
-
-std::optional<FlakeRef> maybeParseFlake(
-    const std::string & url, const std::optional<Path> & baseDir = {});
-
-std::pair<FlakeRef, std::string> parseFlakeRefWithFragment(
-    const std::string & url, const std::optional<Path> & baseDir = {}, bool allowMissing = false);
-
-std::optional<std::pair<FlakeRef, std::string>> maybeParseFlakeRefWithFragment(
-    const std::string & url, const std::optional<Path> & baseDir = {});
-
-}

src/libexpr/flake/lockfile.cc

@@ -1,256 +0,0 @@
-#include "lockfile.hh"
-#include "store-api.hh"
-
-#include <nlohmann/json.hpp>
-
-namespace nix::flake {
-
-FlakeRef flakeRefFromJson(const nlohmann::json & json)
-{
-    return FlakeRef::fromAttrs(jsonToAttrs(json));
-}
-
-FlakeRef getFlakeRef(
-    const nlohmann::json & json,
-    const char * attr)
-{
-    auto i = json.find(attr);
-    if (i != json.end())
-        return flakeRefFromJson(*i);
-
-    throw Error("attribute '%s' missing in lock file", attr);
-}
-
-LockedNode::LockedNode(const nlohmann::json & json)
-    : lockedRef(getFlakeRef(json, "locked"))
-    , originalRef(getFlakeRef(json, "original"))
-    , info(TreeInfo::fromJson(json))
-    , isFlake(json.find("flake") != json.end() ? (bool) json["flake"] : true)
-{
-    if (!lockedRef.input->isImmutable())
-        throw Error("lockfile contains mutable flakeref '%s'", lockedRef);
-}
-
-StorePath LockedNode::computeStorePath(Store & store) const
-{
-    return info.computeStorePath(store);
-}
-
-std::shared_ptr<Node> Node::findInput(const InputPath & path)
-{
-    auto pos = shared_from_this();
-
-    for (auto & elem : path) {
-        auto i = pos->inputs.find(elem);
-        if (i == pos->inputs.end())
-            return {};
-        pos = i->second;
-    }
-
-    return pos;
-}
-
-LockFile::LockFile(const nlohmann::json & json, const Path & path)
-{
-    auto version = json.value("version", 0);
-    if (version != 5)
-        throw Error("lock file '%s' has unsupported version %d", path, version);
-
-    std::unordered_map<std::string, std::shared_ptr<Node>> nodeMap;
-
-    std::function<void(Node & node, const nlohmann::json & jsonNode)> getInputs;
-
-    getInputs = [&](Node & node, const nlohmann::json & jsonNode)
-    {
-        if (jsonNode.find("inputs") == jsonNode.end()) return;
-        for (auto & i : jsonNode["inputs"].items()) {
-            std::string inputKey = i.value();
-            auto k = nodeMap.find(inputKey);
-            if (k == nodeMap.end()) {
-                auto jsonNode2 = json["nodes"][inputKey];
-                auto input = std::make_shared<LockedNode>(jsonNode2);
-                k = nodeMap.insert_or_assign(inputKey, input).first;
-                getInputs(*input, jsonNode2);
-            }
-            node.inputs.insert_or_assign(i.key(), k->second);
-        }
-    };
-
-    std::string rootKey = json["root"];
-    nodeMap.insert_or_assign(rootKey, root);
-    getInputs(*root, json["nodes"][rootKey]);
-}
-
-nlohmann::json LockFile::toJson() const
-{
-    nlohmann::json nodes;
-    std::unordered_map<std::shared_ptr<const Node>, std::string> nodeKeys;
-    std::unordered_set<std::string> keys;
-
-    std::function<std::string(const std::string & key, std::shared_ptr<const Node> node)> dumpNode;
-
-    dumpNode = [&](std::string key, std::shared_ptr<const Node> node) -> std::string
-    {
-        auto k = nodeKeys.find(node);
-        if (k != nodeKeys.end())
-            return k->second;
-
-        if (!keys.insert(key).second) {
-            for (int n = 2; ; ++n) {
-                auto k = fmt("%s_%d", key, n);
-                if (keys.insert(k).second) {
-                    key = k;
-                    break;
-                }
-            }
-        }
-
-        nodeKeys.insert_or_assign(node, key);
-
-        auto n = nlohmann::json::object();
-
-        if (!node->inputs.empty()) {
-            auto inputs = nlohmann::json::object();
-            for (auto & i : node->inputs)
-                inputs[i.first] = dumpNode(i.first, i.second);
-            n["inputs"] = std::move(inputs);
-        }
-
-        if (auto lockedNode = std::dynamic_pointer_cast<const LockedNode>(node)) {
-            n["original"] = fetchers::attrsToJson(lockedNode->originalRef.toAttrs());
-            n["locked"] = fetchers::attrsToJson(lockedNode->lockedRef.toAttrs());
-            n["info"] = lockedNode->info.toJson();
-            if (!lockedNode->isFlake) n["flake"] = false;
-        }
-
-        nodes[key] = std::move(n);
-
-        return key;
-    };
-
-    nlohmann::json json;
-    json["version"] = 5;
-    json["root"] = dumpNode("root", root);
-    json["nodes"] = std::move(nodes);
-
-    return json;
-}
-
-std::string LockFile::to_string() const
-{
-    return toJson().dump(2);
-}
-
-LockFile LockFile::read(const Path & path)
-{
-    if (!pathExists(path)) return LockFile();
-    return LockFile(nlohmann::json::parse(readFile(path)), path);
-}
-
-std::ostream & operator <<(std::ostream & stream, const LockFile & lockFile)
-{
-    stream << lockFile.toJson().dump(2);
-    return stream;
-}
-
-void LockFile::write(const Path & path) const
-{
-    createDirs(dirOf(path));
-    writeFile(path, fmt("%s\n", *this));
-}
-
-bool LockFile::isImmutable() const
-{
-    std::unordered_set<std::shared_ptr<const Node>> nodes;
-
-    std::function<void(std::shared_ptr<const Node> node)> visit;
-
-    visit = [&](std::shared_ptr<const Node> node)
-    {
-        if (!nodes.insert(node).second) return;
-        for (auto & i : node->inputs) visit(i.second);
-    };
-
-    visit(root);
-
-    for (auto & i : nodes) {
-        if (i == root) continue;
-        auto lockedNode = std::dynamic_pointer_cast<const LockedNode>(i);
-        if (lockedNode && !lockedNode->lockedRef.input->isImmutable()) return false;
-    }
-
-    return true;
-}
-
-bool LockFile::operator ==(const LockFile & other) const
-{
-    // FIXME: slow
-    return toJson() == other.toJson();
-}
-
-InputPath parseInputPath(std::string_view s)
-{
-    InputPath path;
-
-    for (auto & elem : tokenizeString<std::vector<std::string>>(s, "/")) {
-        if (!std::regex_match(elem, flakeIdRegex))
-            throw Error("invalid flake input path element '%s'", elem);
-        path.push_back(elem);
-    }
-
-    return path;
-}
-
-static void flattenLockFile(
-    std::shared_ptr<const Node> node,
-    const InputPath & prefix,
-    std::unordered_set<std::shared_ptr<const Node>> & done,
-    std::map<InputPath, std::shared_ptr<const LockedNode>> & res)
-{
-    if (!done.insert(node).second) return;
-
-    for (auto &[id, input] : node->inputs) {
-        auto inputPath(prefix);
-        inputPath.push_back(id);
-        if (auto lockedInput = std::dynamic_pointer_cast<const LockedNode>(input))
-            res.emplace(inputPath, lockedInput);
-        flattenLockFile(input, inputPath, done, res);
-    }
-}
-
-std::string diffLockFiles(const LockFile & oldLocks, const LockFile & newLocks)
-{
-    std::unordered_set<std::shared_ptr<const Node>> done;
-    std::map<InputPath, std::shared_ptr<const LockedNode>> oldFlat, newFlat;
-    flattenLockFile(oldLocks.root, {}, done, oldFlat);
-    done.clear();
-    flattenLockFile(newLocks.root, {}, done, newFlat);
-
-    auto i = oldFlat.begin();
-    auto j = newFlat.begin();
-    std::string res;
-
-    while (i != oldFlat.end() || j != newFlat.end()) {
-        if (j != newFlat.end() && (i == oldFlat.end() || i->first > j->first)) {
-            res += fmt("* Added '%s': '%s'\n", concatStringsSep("/", j->first), j->second->lockedRef);
-            ++j;
-        } else if (i != oldFlat.end() && (j == newFlat.end() || i->first < j->first)) {
-            res += fmt("* Removed '%s'\n", concatStringsSep("/", i->first));
-            ++i;
-        } else {
-            if (!(i->second->lockedRef == j->second->lockedRef)) {
-                assert(i->second->lockedRef.to_string() != j->second->lockedRef.to_string());
-                res += fmt("* Updated '%s': '%s' -> '%s'\n",
-                    concatStringsSep("/", i->first),
-                    i->second->lockedRef,
-                    j->second->lockedRef);
-            }
-            ++i;
-            ++j;
-        }
-    }
-
-    return res;
-}
-
-}

src/libexpr/flake/lockfile.hh

@@ -1,77 +0,0 @@
-#pragma once
-
-#include "flakeref.hh"
-
-#include <nlohmann/json_fwd.hpp>
-
-namespace nix {
-class Store;
-struct StorePath;
-}
-
-namespace nix::flake {
-
-using namespace fetchers;
-
-typedef std::vector<FlakeId> InputPath;
-
-/* A node in the lock file. It has outgoing edges to other nodes (its
-   inputs). Only the root node has this type; all other nodes have
-   type LockedNode. */
-struct Node : std::enable_shared_from_this<Node>
-{
-    std::map<FlakeId, std::shared_ptr<Node>> inputs;
-
-    virtual ~Node() { }
-
-    std::shared_ptr<Node> findInput(const InputPath & path);
-};
-
-/* A non-root node in the lock file. */
-struct LockedNode : Node
-{
-    FlakeRef lockedRef, originalRef;
-    TreeInfo info;
-    bool isFlake = true;
-
-    LockedNode(
-        const FlakeRef & lockedRef,
-        const FlakeRef & originalRef,
-        const TreeInfo & info,
-        bool isFlake = true)
-        : lockedRef(lockedRef), originalRef(originalRef), info(info), isFlake(isFlake)
-    { }
-
-    LockedNode(const nlohmann::json & json);
-
-    StorePath computeStorePath(Store & store) const;
-};
-
-struct LockFile
-{
-    std::shared_ptr<Node> root = std::make_shared<Node>();
-
-    LockFile() {};
-    LockFile(const nlohmann::json & json, const Path & path);
-
-    nlohmann::json toJson() const;
-
-    std::string to_string() const;
-
-    static LockFile read(const Path & path);
-
-    void write(const Path & path) const;
-
-    bool isImmutable() const;
-
-    bool operator ==(const LockFile & other) const;
-};
-
-std::ostream & operator <<(std::ostream & stream, const LockFile & lockFile);
-
-InputPath parseInputPath(std::string_view s);
-
-std::string diffLockFiles(const LockFile & oldLocks, const LockFile & newLocks);
-
-}
-

src/libexpr/get-drvs.cc

@@ -348,7 +348,7 @@ static void getDerivations(EvalState & state, Value & vIn,
                    should we recurse into it?  => Only if it has a
                    `recurseForDerivations = true' attribute. */
                 if (i->value->type == tAttrs) {
-                    Bindings::iterator j = i->value->attrs->find(state.sRecurseForDerivations);
+                    Bindings::iterator j = i->value->attrs->find(state.symbols.create("recurseForDerivations"));
                     if (j != i->value->attrs->end() && state.forceBool(*j->value, *j->pos))
                         getDerivations(state, *i->value, pathPrefix2, autoArgs, drvs, done, ignoreAssertionFailures);
                 }

src/libexpr/json-to-value.cc

@@ -4,6 +4,7 @@
 #include <nlohmann/json.hpp>
 
 using json = nlohmann::json;
+using std::unique_ptr;
 
 namespace nix {
 
@@ -12,69 +13,69 @@ namespace nix {
 class JSONSax : nlohmann::json_sax<json> {
     class JSONState {
     protected:
-        std::unique_ptr<JSONState> parent;
-        RootValue v;
+        unique_ptr<JSONState> parent;
+        Value * v;
     public:
-        virtual std::unique_ptr<JSONState> resolve(EvalState &)
+        virtual unique_ptr<JSONState> resolve(EvalState &)
         {
             throw std::logic_error("tried to close toplevel json parser state");
-        }
-        explicit JSONState(std::unique_ptr<JSONState> && p) : parent(std::move(p)) {}
-        explicit JSONState(Value * v) : v(allocRootValue(v)) {}
-        JSONState(JSONState & p) = delete;
-        Value & value(EvalState & state)
+        };
+        explicit JSONState(unique_ptr<JSONState>&& p) : parent(std::move(p)), v(nullptr) {};
+        explicit JSONState(Value* v) : v(v) {};
+        JSONState(JSONState& p) = delete;
+        Value& value(EvalState & state)
         {
-            if (!v)
-                v = allocRootValue(state.allocValue());
-            return **v;
-        }
-        virtual ~JSONState() {}
-        virtual void add() {}
+            if (v == nullptr)
+                v = state.allocValue();
+            return *v;
+        };
+        virtual ~JSONState() {};
+        virtual void add() {};
     };
 
     class JSONObjectState : public JSONState {
         using JSONState::JSONState;
-        ValueMap attrs;
-        std::unique_ptr<JSONState> resolve(EvalState & state) override
+        ValueMap attrs = ValueMap();
+        virtual unique_ptr<JSONState> resolve(EvalState & state) override
         {
-            Value & v = parent->value(state);
+            Value& v = parent->value(state);
             state.mkAttrs(v, attrs.size());
             for (auto & i : attrs)
                 v.attrs->push_back(Attr(i.first, i.second));
             return std::move(parent);
         }
-        void add() override { v = nullptr; }
+        virtual void add() override { v = nullptr; };
     public:
-        void key(string_t & name, EvalState & state)
+        void key(string_t& name, EvalState & state)
         {
-            attrs.insert_or_assign(state.symbols.create(name), &value(state));
+            attrs[state.symbols.create(name)] = &value(state);
         }
     };
 
     class JSONListState : public JSONState {
-        ValueVector values;
-        std::unique_ptr<JSONState> resolve(EvalState & state) override
+        ValueVector values = ValueVector();
+        virtual unique_ptr<JSONState> resolve(EvalState & state) override
         {
-            Value & v = parent->value(state);
+            Value& v = parent->value(state);
             state.mkList(v, values.size());
             for (size_t n = 0; n < values.size(); ++n) {
                 v.listElems()[n] = values[n];
             }
             return std::move(parent);
         }
-        void add() override {
-            values.push_back(*v);
+        virtual void add() override {
+            values.push_back(v);
             v = nullptr;
-        }
+        };
     public:
-        JSONListState(std::unique_ptr<JSONState> && p, std::size_t reserve) : JSONState(std::move(p))
+        JSONListState(unique_ptr<JSONState>&& p, std::size_t reserve) : JSONState(std::move(p))
         {
             values.reserve(reserve);
         }
     };
 
     EvalState & state;
-    std::unique_ptr<JSONState> rs;
+    unique_ptr<JSONState> rs;
 
     template<typename T, typename... Args> inline bool handle_value(T f, Args... args)
     {
@@ -106,12 +107,12 @@ public:
         return handle_value(mkInt, val);
     }
 
-    bool number_float(number_float_t val, const string_t & s)
+    bool number_float(number_float_t val, const string_t& s)
     {
         return handle_value(mkFloat, val);
     }
 
-    bool string(string_t & val)
+    bool string(string_t& val)
     {
         return handle_value<void(Value&, const char*)>(mkString, val.c_str());
     }
@@ -122,7 +123,7 @@ public:
         return true;
     }
 
-    bool key(string_t & name)
+    bool key(string_t& name)
     {
         dynamic_cast<JSONObjectState*>(rs.get())->key(name, state);
         return true;

src/libexpr/local.mk

@@ -4,16 +4,11 @@ libexpr_NAME = libnixexpr
 
 libexpr_DIR := $(d)
 
-libexpr_SOURCES := \
-  $(wildcard $(d)/*.cc) \
-  $(wildcard $(d)/primops/*.cc) \
-  $(wildcard $(d)/flake/*.cc) \
-  $(d)/lexer-tab.cc \
-  $(d)/parser-tab.cc
+libexpr_SOURCES := $(wildcard $(d)/*.cc) $(wildcard $(d)/primops/*.cc) $(d)/lexer-tab.cc $(d)/parser-tab.cc
 
-libexpr_CXXFLAGS += -I src/libutil -I src/libstore -I src/libfetchers -I src/libmain -I src/libexpr
+libexpr_CXXFLAGS += -I src/libutil -I src/libstore -I src/libmain -I src/libexpr
 
-libexpr_LIBS = libutil libstore libfetchers libnixrust
+libexpr_LIBS = libutil libstore libnixrust
 
 libexpr_LDFLAGS =
 ifneq ($(OS), FreeBSD)
@@ -39,9 +34,4 @@ dist-files += $(d)/parser-tab.cc $(d)/parser-tab.hh $(d)/lexer-tab.cc $(d)/lexer
 
 $(eval $(call install-file-in, $(d)/nix-expr.pc, $(prefix)/lib/pkgconfig, 0644))
 
-$(foreach i, $(wildcard src/libexpr/flake/*.hh), \
-  $(eval $(call install-file-in, $(i), $(includedir)/nix/flake, 0644)))
-
 $(d)/primops.cc: $(d)/imported-drv-to-derivation.nix.gen.hh
-
-$(d)/flake/flake.cc: $(d)/flake/call-flake.nix.gen.hh

src/libexpr/nixexpr.hh

@@ -209,10 +209,9 @@ struct ExprList : Expr
 
 struct Formal
 {
-    Pos pos;
     Symbol name;
     Expr * def;
-    Formal(const Pos & pos, const Symbol & name, Expr * def) : pos(pos), name(name), def(def) { };
+    Formal(const Symbol & name, Expr * def) : name(name), def(def) { };
 };
 
 struct Formals
@@ -262,9 +261,8 @@ struct ExprWith : Expr
 
 struct ExprIf : Expr
 {
-    Pos pos;
     Expr * cond, * then, * else_;
-    ExprIf(const Pos & pos, Expr * cond, Expr * then, Expr * else_) : pos(pos), cond(cond), then(then), else_(else_) { };
+    ExprIf(Expr * cond, Expr * then, Expr * else_) : cond(cond), then(then), else_(else_) { };
     COMMON_METHODS
 };
 

src/libexpr/parser.y

@@ -335,7 +335,7 @@ expr_function
   ;
 
 expr_if
-  : IF expr THEN expr ELSE expr { $$ = new ExprIf(CUR_POS, $2, $4, $6); }
+  : IF expr THEN expr ELSE expr { $$ = new ExprIf($2, $4, $6); }
   | expr_op
   ;
 
@@ -531,8 +531,8 @@ formals
   ;
 
 formal
-  : ID { $$ = new Formal(CUR_POS, data->symbols.create($1), 0); }
-  | ID '?' expr { $$ = new Formal(CUR_POS, data->symbols.create($1), $3); }
+  : ID { $$ = new Formal(data->symbols.create($1), 0); }
+  | ID '?' expr { $$ = new Formal(data->symbols.create($1), $3); }
   ;
 
 %%
@@ -544,8 +544,7 @@ formal
 #include <unistd.h>
 
 #include "eval.hh"
-#include "filetransfer.hh"
-#include "fetchers.hh"
+#include "download.hh"
 #include "store-api.hh"
 
 
@@ -688,9 +687,10 @@ std::pair<bool, std::string> EvalState::resolveSearchPathElem(const SearchPathEl
 
     if (isUri(elem.second)) {
         try {
-            res = { true, store->toRealPath(fetchers::downloadTarball(
-                        store, resolveUri(elem.second), "source", false).storePath) };
-        } catch (FileTransferError & e) {
+            CachedDownloadRequest request(elem.second);
+            request.unpack = true;
+            res = { true, getDownloader()->downloadCached(store, request).path };
+        } catch (DownloadError & e) {
             printError(format("warning: Nix search path entry '%1%' cannot be downloaded, ignoring") % elem.second);
             res = { false, "" };
         }

src/libexpr/primops.cc

@@ -1,5 +1,6 @@
 #include "archive.hh"
 #include "derivations.hh"
+#include "download.hh"
 #include "eval-inline.hh"
 #include "eval.hh"
 #include "globals.hh"
@@ -50,20 +51,20 @@ void EvalState::realiseContext(const PathSet & context)
     std::vector<StorePathWithOutputs> drvs;
 
     for (auto & i : context) {
-        auto [ctxS, outputName] = decodeContext(i);
-        auto ctx = store->parseStorePath(ctxS);
+        std::pair<string, string> decoded = decodeContext(i);
+        auto ctx = store->parseStorePath(decoded.first);
         if (!store->isValidPath(ctx))
             throw InvalidPathError(store->printStorePath(ctx));
-        if (!outputName.empty() && ctx.isDerivation()) {
-            drvs.push_back(StorePathWithOutputs{ctx.clone(), {outputName}});
+        if (!decoded.second.empty() && ctx.isDerivation()) {
+            drvs.push_back(StorePathWithOutputs{ctx.clone(), {decoded.second}});
 
             /* Add the output of this derivation to the allowed
                paths. */
             if (allowedPaths) {
-                auto drv = store->derivationFromPath(ctx);
-                DerivationOutputs::iterator i = drv.outputs.find(outputName);
+                auto drv = store->derivationFromPath(store->parseStorePath(decoded.first));
+                DerivationOutputs::iterator i = drv.outputs.find(decoded.second);
                 if (i == drv.outputs.end())
-                    throw Error("derivation '%s' does not have an output named '%s'", ctxS, outputName);
+                    throw Error("derivation '%s' does not have an output named '%s'", decoded.first, decoded.second);
                 allowedPaths->insert(store->printStorePath(i->second.path));
             }
         }
@@ -79,7 +80,6 @@ void EvalState::realiseContext(const PathSet & context)
     StorePathSet willBuild, willSubstitute, unknown;
     unsigned long long downloadSize, narSize;
     store->queryMissing(drvs, willBuild, willSubstitute, unknown, downloadSize, narSize);
-
     store->buildPaths(drvs);
 }
 
@@ -122,16 +122,16 @@ static void prim_scopedImport(EvalState & state, const Pos & pos, Value * * args
         }
         w.attrs->sort();
 
-        static RootValue fun;
+        static Value * fun = nullptr;
         if (!fun) {
-            fun = allocRootValue(state.allocValue());
+            fun = state.allocValue();
             state.eval(state.parseExprFromString(
                 #include "imported-drv-to-derivation.nix.gen.hh"
-                , "/"), **fun);
+                , "/"), *fun);
         }
 
-        state.forceFunction(**fun, pos);
-        mkApp(v, **fun, w);
+        state.forceFunction(*fun, pos);
+        mkApp(v, *fun, w);
         state.forceAttrs(v, pos);
     } else {
         state.forceAttrs(*args[0]);
@@ -242,7 +242,7 @@ void prim_exec(EvalState & state, const Pos & pos, Value * * args, Value & v)
 /* Return a string representing the type of the expression. */
 static void prim_typeOf(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     string t;
     switch (args[0]->type) {
         case tInt: t = "int"; break;
@@ -270,7 +270,7 @@ static void prim_typeOf(EvalState & state, const Pos & pos, Value * * args, Valu
 /* Determine whether the argument is the null value. */
 static void prim_isNull(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     mkBool(v, args[0]->type == tNull);
 }
 
@@ -278,7 +278,7 @@ static void prim_isNull(EvalState & state, const Pos & pos, Value * * args, Valu
 /* Determine whether the argument is a function. */
 static void prim_isFunction(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     bool res;
     switch (args[0]->type) {
         case tLambda:
@@ -297,21 +297,21 @@ static void prim_isFunction(EvalState & state, const Pos & pos, Value * * args,
 /* Determine whether the argument is an integer. */
 static void prim_isInt(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     mkBool(v, args[0]->type == tInt);
 }
 
 /* Determine whether the argument is a float. */
 static void prim_isFloat(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     mkBool(v, args[0]->type == tFloat);
 }
 
 /* Determine whether the argument is a string. */
 static void prim_isString(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     mkBool(v, args[0]->type == tString);
 }
 
@@ -319,14 +319,14 @@ static void prim_isString(EvalState & state, const Pos & pos, Value * * args, Va
 /* Determine whether the argument is a Boolean. */
 static void prim_isBool(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     mkBool(v, args[0]->type == tBool);
 }
 
 /* Determine whether the argument is a path. */
 static void prim_isPath(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     mkBool(v, args[0]->type == tPath);
 }
 
@@ -383,7 +383,7 @@ static void prim_genericClosure(EvalState & state, const Pos & pos, Value * * ar
         args[0]->attrs->find(state.symbols.create("operator"));
     if (op == args[0]->attrs->end())
         throw EvalError(format("attribute 'operator' required, at %1%") % pos);
-    state.forceValue(*op->value, pos);
+    state.forceValue(*op->value);
 
     /* Construct the closure by applying the operator to element of
        `workSet', adding the result to `workSet', continuing until
@@ -402,7 +402,7 @@ static void prim_genericClosure(EvalState & state, const Pos & pos, Value * * ar
             e->attrs->find(state.symbols.create("key"));
         if (key == e->attrs->end())
             throw EvalError(format("attribute 'key' required, at %1%") % pos);
-        state.forceValue(*key->value, pos);
+        state.forceValue(*key->value);
 
         if (!doneKeys.insert(key->value).second) continue;
         res.push_back(e);
@@ -414,7 +414,7 @@ static void prim_genericClosure(EvalState & state, const Pos & pos, Value * * ar
 
         /* Add the values returned by the operator to the work set. */
         for (unsigned int n = 0; n < call.listSize(); ++n) {
-            state.forceValue(*call.listElems()[n], pos);
+            state.forceValue(*call.listElems()[n]);
             workSet.push_back(call.listElems()[n]);
         }
     }
@@ -446,7 +446,7 @@ static void prim_throw(EvalState & state, const Pos & pos, Value * * args, Value
 static void prim_addErrorContext(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     try {
-        state.forceValue(*args[1], pos);
+        state.forceValue(*args[1]);
         v = *args[1];
     } catch (Error & e) {
         PathSet context;
@@ -462,7 +462,7 @@ static void prim_tryEval(EvalState & state, const Pos & pos, Value * * args, Val
 {
     state.mkAttrs(v, 2);
     try {
-        state.forceValue(*args[0], pos);
+        state.forceValue(*args[0]);
         v.attrs->push_back(Attr(state.sValue, args[0]));
         mkBool(*state.allocAttr(v, state.symbols.create("success")), true);
     } catch (AssertionError & e) {
@@ -484,8 +484,8 @@ static void prim_getEnv(EvalState & state, const Pos & pos, Value * * args, Valu
 /* Evaluate the first argument, then return the second argument. */
 static void prim_seq(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
-    state.forceValue(*args[1], pos);
+    state.forceValue(*args[0]);
+    state.forceValue(*args[1]);
     v = *args[1];
 }
 
@@ -495,7 +495,7 @@ static void prim_seq(EvalState & state, const Pos & pos, Value * * args, Value &
 static void prim_deepSeq(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValueDeep(*args[0]);
-    state.forceValue(*args[1], pos);
+    state.forceValue(*args[1]);
     v = *args[1];
 }
 
@@ -504,12 +504,12 @@ static void prim_deepSeq(EvalState & state, const Pos & pos, Value * * args, Val
    return the second expression.  Useful for debugging. */
 static void prim_trace(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     if (args[0]->type == tString)
         printError(format("trace: %1%") % args[0]->string.s);
     else
         printError(format("trace: %1%") % *args[0]);
-    state.forceValue(*args[1], pos);
+    state.forceValue(*args[1]);
     v = *args[1];
 }
 
@@ -600,7 +600,7 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
         try {
 
             if (ignoreNulls) {
-                state.forceValue(*i->value, pos);
+                state.forceValue(*i->value);
                 if (i->value->type == tNull) continue;
             }
 
@@ -1021,9 +1021,7 @@ static void prim_toFile(EvalState & state, const Pos & pos, Value * * args, Valu
 
     for (auto path : context) {
         if (path.at(0) != '/')
-            throw EvalError(format(
-                "in 'toFile': the file named '%1%' must not contain a reference "
-                "to a derivation but contains (%2%), at %3%") % name % path % pos);
+            throw EvalError(format("in 'toFile': the file '%1%' cannot refer to derivation outputs, at %2%") % name % pos);
         refs.insert(state.store->parseStorePath(path));
     }
 
@@ -1093,7 +1091,7 @@ static void prim_filterSource(EvalState & state, const Pos & pos, Value * * args
     if (!context.empty())
         throw EvalError(format("string '%1%' cannot refer to other paths, at %2%") % path % pos);
 
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     if (args[0]->type != tLambda)
         throw TypeError(format("first argument in call to 'filterSource' is not a function but %1%, at %2%") % showType(*args[0]) % pos);
 
@@ -1119,7 +1117,7 @@ static void prim_path(EvalState & state, const Pos & pos, Value * * args, Value
         } else if (attr.name == state.sName)
             name = state.forceStringNoCtx(*attr.value, *attr.pos);
         else if (n == "filter") {
-            state.forceValue(*attr.value, pos);
+            state.forceValue(*attr.value);
             filterFun = attr.value;
         } else if (n == "recursive")
             recursive = state.forceBool(*attr.value, *attr.pos);
@@ -1190,7 +1188,7 @@ void prim_getAttr(EvalState & state, const Pos & pos, Value * * args, Value & v)
         throw EvalError(format("attribute '%1%' missing, at %2%") % attr % pos);
     // !!! add to stack trace?
     if (state.countCalls && i->pos) state.attrSelects[*i->pos]++;
-    state.forceValue(*i->value, pos);
+    state.forceValue(*i->value);
     v = *i->value;
 }
 
@@ -1220,7 +1218,7 @@ static void prim_hasAttr(EvalState & state, const Pos & pos, Value * * args, Val
 /* Determine whether the argument is a set. */
 static void prim_isAttrs(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     mkBool(v, args[0]->type == tAttrs);
 }
 
@@ -1346,7 +1344,7 @@ static void prim_catAttrs(EvalState & state, const Pos & pos, Value * * args, Va
 */
 static void prim_functionArgs(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     if (args[0]->type != tLambda)
         throw TypeError(format("'functionArgs' requires a function, at %1%") % pos);
 
@@ -1356,12 +1354,9 @@ static void prim_functionArgs(EvalState & state, const Pos & pos, Value * * args
     }
 
     state.mkAttrs(v, args[0]->lambda.fun->formals->formals.size());
-    for (auto & i : args[0]->lambda.fun->formals->formals) {
+    for (auto & i : args[0]->lambda.fun->formals->formals)
         // !!! should optimise booleans (allocate only once)
-        Value * value = state.allocValue();
-        v.attrs->push_back(Attr(i.name, value, &i.pos));
-        mkBool(*value, i.def);
-    }
+        mkBool(*state.allocAttr(v, i.name), i.def);
     v.attrs->sort();
 }
 
@@ -1392,7 +1387,7 @@ static void prim_mapAttrs(EvalState & state, const Pos & pos, Value * * args, Va
 /* Determine whether the argument is a list. */
 static void prim_isList(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
+    state.forceValue(*args[0]);
     mkBool(v, args[0]->isList());
 }
 
@@ -1402,7 +1397,7 @@ static void elemAt(EvalState & state, const Pos & pos, Value & list, int n, Valu
     state.forceList(list, pos);
     if (n < 0 || (unsigned int) n >= list.listSize())
         throw Error(format("list index %1% is out of bounds, at %2%") % n % pos);
-    state.forceValue(*list.listElems()[n], pos);
+    state.forceValue(*list.listElems()[n]);
     v = *list.listElems()[n];
 }
 
@@ -1525,9 +1520,9 @@ static void prim_foldlStrict(EvalState & state, const Pos & pos, Value * * args,
             vCur = n == args[2]->listSize() - 1 ? &v : state.allocValue();
             state.callFunction(vTmp, *args[2]->listElems()[n], *vCur, pos);
         }
-        state.forceValue(v, pos);
+        state.forceValue(v);
     } else {
-        state.forceValue(*args[1], pos);
+        state.forceValue(*args[1]);
         v = *args[1];
     }
 }
@@ -1592,7 +1587,7 @@ static void prim_sort(EvalState & state, const Pos & pos, Value * * args, Value
     auto len = args[1]->listSize();
     state.mkList(v, len);
     for (unsigned int n = 0; n < len; ++n) {
-        state.forceValue(*args[1]->listElems()[n], pos);
+        state.forceValue(*args[1]->listElems()[n]);
         v.listElems()[n] = args[1]->listElems()[n];
     }
 
@@ -1627,7 +1622,7 @@ static void prim_partition(EvalState & state, const Pos & pos, Value * * args, V
 
     for (unsigned int n = 0; n < len; ++n) {
         auto vElem = args[1]->listElems()[n];
-        state.forceValue(*vElem, pos);
+        state.forceValue(*vElem);
         Value res;
         state.callFunction(*args[0], *vElem, res, pos);
         if (state.forceBool(res, pos))
@@ -1758,8 +1753,8 @@ static void prim_bitXor(EvalState & state, const Pos & pos, Value * * args, Valu
 
 static void prim_lessThan(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
-    state.forceValue(*args[0], pos);
-    state.forceValue(*args[1], pos);
+    state.forceValue(*args[0]);
+    state.forceValue(*args[1]);
     CompareValues comp;
     mkBool(v, comp(args[0], args[1]));
 }
@@ -2050,6 +2045,68 @@ static void prim_splitVersion(EvalState & state, const Pos & pos, Value * * args
 }
 
 
+/*************************************************************
+ * Networking
+ *************************************************************/
+
+
+void fetch(EvalState & state, const Pos & pos, Value * * args, Value & v,
+    const string & who, bool unpack, const std::string & defaultName)
+{
+    CachedDownloadRequest request("");
+    request.unpack = unpack;
+    request.name = defaultName;
+
+    state.forceValue(*args[0]);
+
+    if (args[0]->type == tAttrs) {
+
+        state.forceAttrs(*args[0], pos);
+
+        for (auto & attr : *args[0]->attrs) {
+            string n(attr.name);
+            if (n == "url")
+                request.uri = state.forceStringNoCtx(*attr.value, *attr.pos);
+            else if (n == "sha256")
+                request.expectedHash = Hash(state.forceStringNoCtx(*attr.value, *attr.pos), htSHA256);
+            else if (n == "name")
+                request.name = state.forceStringNoCtx(*attr.value, *attr.pos);
+            else
+                throw EvalError(format("unsupported argument '%1%' to '%2%', at %3%") % attr.name % who % attr.pos);
+        }
+
+        if (request.uri.empty())
+            throw EvalError(format("'url' argument required, at %1%") % pos);
+
+    } else
+        request.uri = state.forceStringNoCtx(*args[0], pos);
+
+    state.checkURI(request.uri);
+
+    if (evalSettings.pureEval && !request.expectedHash)
+        throw Error("in pure evaluation mode, '%s' requires a 'sha256' argument", who);
+
+    auto res = getDownloader()->downloadCached(state.store, request);
+
+    if (state.allowedPaths)
+        state.allowedPaths->insert(res.path);
+
+    mkString(v, res.storePath, PathSet({res.storePath}));
+}
+
+
+static void prim_fetchurl(EvalState & state, const Pos & pos, Value * * args, Value & v)
+{
+    fetch(state, pos, args, v, "fetchurl", false, "");
+}
+
+
+static void prim_fetchTarball(EvalState & state, const Pos & pos, Value * * args, Value & v)
+{
+    fetch(state, pos, args, v, "fetchTarball", true, "source");
+}
+
+
 /*************************************************************
  * Primop registration
  *************************************************************/
@@ -2232,6 +2289,10 @@ void EvalState::createBaseEnv()
     addPrimOp("derivationStrict", 1, prim_derivationStrict);
     addPrimOp("placeholder", 1, prim_placeholder);
 
+    // Networking
+    addPrimOp("__fetchurl", 1, prim_fetchurl);
+    addPrimOp("fetchTarball", 1, prim_fetchTarball);
+
     /* Add a wrapper around the derivation primop that computes the
        `drvPath' and `outPath' attributes lazily. */
     string path = canonPath(settings.nixDataDir + "/nix/corepkgs/derivation.nix", true);

src/libexpr/primops.hh

@@ -20,7 +20,6 @@ struct RegisterPrimOp
    them. */
 /* Load a ValueInitializer from a DSO and return whatever it initializes */
 void prim_importNative(EvalState & state, const Pos & pos, Value * * args, Value & v);
-
 /* Execute a program and parse its output */
 void prim_exec(EvalState & state, const Pos & pos, Value * * args, Value & v);
 

src/libexpr/primops/fetchGit.cc

@@ -1,19 +1,203 @@
 #include "primops.hh"
 #include "eval-inline.hh"
+#include "download.hh"
 #include "store-api.hh"
+#include "pathlocks.hh"
 #include "hash.hh"
-#include "fetchers.hh"
-#include "url.hh"
+#include "tarfile.hh"
+
+#include <sys/time.h>
+
+#include <regex>
+
+#include <nlohmann/json.hpp>
+
+using namespace std::string_literals;
 
 namespace nix {
 
+struct GitInfo
+{
+    Path storePath;
+    std::string rev;
+    std::string shortRev;
+    uint64_t revCount = 0;
+};
+
+std::regex revRegex("^[0-9a-fA-F]{40}$");
+
+GitInfo exportGit(ref<Store> store, const std::string & uri,
+    std::optional<std::string> ref, std::string rev,
+    const std::string & name)
+{
+    if (evalSettings.pureEval && rev == "")
+        throw Error("in pure evaluation mode, 'fetchGit' requires a Git revision");
+
+    if (!ref && rev == "" && hasPrefix(uri, "/") && pathExists(uri + "/.git")) {
+
+        bool clean = true;
+
+        try {
+            runProgram("git", true, { "-C", uri, "diff-index", "--quiet", "HEAD", "--" });
+        } catch (ExecError & e) {
+            if (!WIFEXITED(e.status) || WEXITSTATUS(e.status) != 1) throw;
+            clean = false;
+        }
+
+        if (!clean) {
+
+            /* This is an unclean working tree. So copy all tracked files. */
+            GitInfo gitInfo;
+            gitInfo.rev = "0000000000000000000000000000000000000000";
+            gitInfo.shortRev = std::string(gitInfo.rev, 0, 7);
+
+            auto files = tokenizeString<std::set<std::string>>(
+                runProgram("git", true, { "-C", uri, "ls-files", "-z" }), "\0"s);
+
+            PathFilter filter = [&](const Path & p) -> bool {
+                assert(hasPrefix(p, uri));
+                std::string file(p, uri.size() + 1);
+
+                auto st = lstat(p);
+
+                if (S_ISDIR(st.st_mode)) {
+                    auto prefix = file + "/";
+                    auto i = files.lower_bound(prefix);
+                    return i != files.end() && hasPrefix(*i, prefix);
+                }
+
+                return files.count(file);
+            };
+
+            gitInfo.storePath = store->printStorePath(store->addToStore("source", uri, true, htSHA256, filter));
+
+            return gitInfo;
+        }
+
+        // clean working tree, but no ref or rev specified.  Use 'HEAD'.
+        rev = chomp(runProgram("git", true, { "-C", uri, "rev-parse", "HEAD" }));
+        ref = "HEAD"s;
+    }
+
+    if (!ref) ref = "HEAD"s;
+
+    if (rev != "" && !std::regex_match(rev, revRegex))
+        throw Error("invalid Git revision '%s'", rev);
+
+    deletePath(getCacheDir() + "/nix/git");
+
+    Path cacheDir = getCacheDir() + "/nix/gitv2/" + hashString(htSHA256, uri).to_string(Base32, false);
+
+    if (!pathExists(cacheDir)) {
+        createDirs(dirOf(cacheDir));
+        runProgram("git", true, { "init", "--bare", cacheDir });
+    }
+
+    Path localRefFile;
+    if (ref->compare(0, 5, "refs/") == 0)
+        localRefFile = cacheDir + "/" + *ref;
+    else
+        localRefFile = cacheDir + "/refs/heads/" + *ref;
+
+    bool doFetch;
+    time_t now = time(0);
+    /* If a rev was specified, we need to fetch if it's not in the
+       repo. */
+    if (rev != "") {
+        try {
+            runProgram("git", true, { "-C", cacheDir, "cat-file", "-e", rev });
+            doFetch = false;
+        } catch (ExecError & e) {
+            if (WIFEXITED(e.status)) {
+                doFetch = true;
+            } else {
+                throw;
+            }
+        }
+    } else {
+        /* If the local ref is older than ‘tarball-ttl’ seconds, do a
+           git fetch to update the local ref to the remote ref. */
+        struct stat st;
+        doFetch = stat(localRefFile.c_str(), &st) != 0 ||
+            (uint64_t) st.st_mtime + settings.tarballTtl <= (uint64_t) now;
+    }
+    if (doFetch)
+    {
+        Activity act(*logger, lvlTalkative, actUnknown, fmt("fetching Git repository '%s'", uri));
+
+        // FIXME: git stderr messes up our progress indicator, so
+        // we're using --quiet for now. Should process its stderr.
+        runProgram("git", true, { "-C", cacheDir, "fetch", "--quiet", "--force", "--", uri, fmt("%s:%s", *ref, *ref) });
+
+        struct timeval times[2];
+        times[0].tv_sec = now;
+        times[0].tv_usec = 0;
+        times[1].tv_sec = now;
+        times[1].tv_usec = 0;
+
+        utimes(localRefFile.c_str(), times);
+    }
+
+    // FIXME: check whether rev is an ancestor of ref.
+    GitInfo gitInfo;
+    gitInfo.rev = rev != "" ? rev : chomp(readFile(localRefFile));
+    gitInfo.shortRev = std::string(gitInfo.rev, 0, 7);
+
+    printTalkative("using revision %s of repo '%s'", gitInfo.rev, uri);
+
+    std::string storeLinkName = hashString(htSHA512, name + std::string("\0"s) + gitInfo.rev).to_string(Base32, false);
+    Path storeLink = cacheDir + "/" + storeLinkName + ".link";
+    PathLocks storeLinkLock({storeLink}, fmt("waiting for lock on '%1%'...", storeLink)); // FIXME: broken
+
+    try {
+        auto json = nlohmann::json::parse(readFile(storeLink));
+
+        assert(json["name"] == name && json["rev"] == gitInfo.rev);
+
+        gitInfo.storePath = json["storePath"];
+
+        if (store->isValidPath(store->parseStorePath(gitInfo.storePath))) {
+            gitInfo.revCount = json["revCount"];
+            return gitInfo;
+        }
+
+    } catch (SysError & e) {
+        if (e.errNo != ENOENT) throw;
+    }
+
+    auto source = sinkToSource([&](Sink & sink) {
+        RunOptions gitOptions("git", { "-C", cacheDir, "archive", gitInfo.rev });
+        gitOptions.standardOut = &sink;
+        runProgram2(gitOptions);
+    });
+
+    Path tmpDir = createTempDir();
+    AutoDelete delTmpDir(tmpDir, true);
+
+    unpackTarfile(*source, tmpDir);
+
+    gitInfo.storePath = store->printStorePath(store->addToStore(name, tmpDir));
+
+    gitInfo.revCount = std::stoull(runProgram("git", true, { "-C", cacheDir, "rev-list", "--count", gitInfo.rev }));
+
+    nlohmann::json json;
+    json["storePath"] = gitInfo.storePath;
+    json["uri"] = uri;
+    json["name"] = name;
+    json["rev"] = gitInfo.rev;
+    json["revCount"] = gitInfo.revCount;
+
+    writeFile(storeLink, json.dump());
+
+    return gitInfo;
+}
+
 static void prim_fetchGit(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     std::string url;
     std::optional<std::string> ref;
-    std::optional<Hash> rev;
+    std::string rev;
     std::string name = "source";
-    bool fetchSubmodules = false;
     PathSet context;
 
     state.forceValue(*args[0]);
@@ -29,11 +213,9 @@ static void prim_fetchGit(EvalState & state, const Pos & pos, Value * * args, Va
             else if (n == "ref")
                 ref = state.forceStringNoCtx(*attr.value, *attr.pos);
             else if (n == "rev")
-                rev = Hash(state.forceStringNoCtx(*attr.value, *attr.pos), htSHA1);
+                rev = state.forceStringNoCtx(*attr.value, *attr.pos);
             else if (n == "name")
                 name = state.forceStringNoCtx(*attr.value, *attr.pos);
-            else if (n == "submodules")
-                fetchSubmodules = state.forceBool(*attr.value, *attr.pos);
             else
                 throw EvalError("unsupported argument '%s' to 'fetchGit', at %s", attr.name, *attr.pos);
         }
@@ -48,36 +230,17 @@ static void prim_fetchGit(EvalState & state, const Pos & pos, Value * * args, Va
     // whitelist. Ah well.
     state.checkURI(url);
 
-    if (evalSettings.pureEval && !rev)
-        throw Error("in pure evaluation mode, 'fetchGit' requires a Git revision");
-
-    fetchers::Attrs attrs;
-    attrs.insert_or_assign("type", "git");
-    attrs.insert_or_assign("url", url.find("://") != std::string::npos ? url : "file://" + url);
-    if (ref) attrs.insert_or_assign("ref", *ref);
-    if (rev) attrs.insert_or_assign("rev", rev->gitRev());
-    if (fetchSubmodules) attrs.insert_or_assign("submodules", true);
-    auto input = fetchers::inputFromAttrs(attrs);
-
-    // FIXME: use name?
-    auto [tree, input2] = input->fetchTree(state.store);
+    auto gitInfo = exportGit(state.store, url, ref, rev, name);
 
     state.mkAttrs(v, 8);
-    auto storePath = state.store->printStorePath(tree.storePath);
-    mkString(*state.allocAttr(v, state.sOutPath), storePath, PathSet({storePath}));
-    // Backward compatibility: set 'rev' to
-    // 0000000000000000000000000000000000000000 for a dirty tree.
-    auto rev2 = input2->getRev().value_or(Hash(htSHA1));
-    mkString(*state.allocAttr(v, state.symbols.create("rev")), rev2.gitRev());
-    mkString(*state.allocAttr(v, state.symbols.create("shortRev")), rev2.gitShortRev());
-    // Backward compatibility: set 'revCount' to 0 for a dirty tree.
-    mkInt(*state.allocAttr(v, state.symbols.create("revCount")),
-        tree.info.revCount.value_or(0));
-    mkBool(*state.allocAttr(v, state.symbols.create("submodules")), fetchSubmodules);
+    mkString(*state.allocAttr(v, state.sOutPath), gitInfo.storePath, PathSet({gitInfo.storePath}));
+    mkString(*state.allocAttr(v, state.symbols.create("rev")), gitInfo.rev);
+    mkString(*state.allocAttr(v, state.symbols.create("shortRev")), gitInfo.shortRev);
+    mkInt(*state.allocAttr(v, state.symbols.create("revCount")), gitInfo.revCount);
     v.attrs->sort();
 
     if (state.allowedPaths)
-        state.allowedPaths->insert(tree.actualPath);
+        state.allowedPaths->insert(state.store->toRealPath(gitInfo.storePath));
 }
 
 static RegisterPrimOp r("fetchGit", 1, prim_fetchGit);

src/libexpr/primops/fetchMercurial.cc

@@ -1,18 +1,174 @@
 #include "primops.hh"
 #include "eval-inline.hh"
+#include "download.hh"
 #include "store-api.hh"
-#include "fetchers.hh"
-#include "url.hh"
+#include "pathlocks.hh"
+
+#include <sys/time.h>
 
 #include <regex>
 
+#include <nlohmann/json.hpp>
+
+using namespace std::string_literals;
+
 namespace nix {
 
+struct HgInfo
+{
+    Path storePath;
+    std::string branch;
+    std::string rev;
+    uint64_t revCount = 0;
+};
+
+std::regex commitHashRegex("^[0-9a-fA-F]{40}$");
+
+HgInfo exportMercurial(ref<Store> store, const std::string & uri,
+    std::string rev, const std::string & name)
+{
+    if (evalSettings.pureEval && rev == "")
+        throw Error("in pure evaluation mode, 'fetchMercurial' requires a Mercurial revision");
+
+    if (rev == "" && hasPrefix(uri, "/") && pathExists(uri + "/.hg")) {
+
+        bool clean = runProgram("hg", true, { "status", "-R", uri, "--modified", "--added", "--removed" }) == "";
+
+        if (!clean) {
+
+            /* This is an unclean working tree. So copy all tracked
+               files. */
+
+            printTalkative("copying unclean Mercurial working tree '%s'", uri);
+
+            HgInfo hgInfo;
+            hgInfo.rev = "0000000000000000000000000000000000000000";
+            hgInfo.branch = chomp(runProgram("hg", true, { "branch", "-R", uri }));
+
+            auto files = tokenizeString<std::set<std::string>>(
+                runProgram("hg", true, { "status", "-R", uri, "--clean", "--modified", "--added", "--no-status", "--print0" }), "\0"s);
+
+            PathFilter filter = [&](const Path & p) -> bool {
+                assert(hasPrefix(p, uri));
+                std::string file(p, uri.size() + 1);
+
+                auto st = lstat(p);
+
+                if (S_ISDIR(st.st_mode)) {
+                    auto prefix = file + "/";
+                    auto i = files.lower_bound(prefix);
+                    return i != files.end() && hasPrefix(*i, prefix);
+                }
+
+                return files.count(file);
+            };
+
+            hgInfo.storePath = store->printStorePath(store->addToStore("source", uri, true, htSHA256, filter));
+
+            return hgInfo;
+        }
+    }
+
+    if (rev == "") rev = "default";
+
+    Path cacheDir = fmt("%s/nix/hg/%s", getCacheDir(), hashString(htSHA256, uri).to_string(Base32, false));
+
+    Path stampFile = fmt("%s/.hg/%s.stamp", cacheDir, hashString(htSHA512, rev).to_string(Base32, false));
+
+    /* If we haven't pulled this repo less than ‘tarball-ttl’ seconds,
+       do so now. */
+    time_t now = time(0);
+    struct stat st;
+    if (stat(stampFile.c_str(), &st) != 0 ||
+        (uint64_t) st.st_mtime + settings.tarballTtl <= (uint64_t) now)
+    {
+        /* Except that if this is a commit hash that we already have,
+           we don't have to pull again. */
+        if (!(std::regex_match(rev, commitHashRegex)
+                && pathExists(cacheDir)
+                && runProgram(
+                    RunOptions("hg", { "log", "-R", cacheDir, "-r", rev, "--template", "1" })
+                    .killStderr(true)).second == "1"))
+        {
+            Activity act(*logger, lvlTalkative, actUnknown, fmt("fetching Mercurial repository '%s'", uri));
+
+            if (pathExists(cacheDir)) {
+                try {
+                    runProgram("hg", true, { "pull", "-R", cacheDir, "--", uri });
+                }
+                catch (ExecError & e) {
+                    string transJournal = cacheDir + "/.hg/store/journal";
+                    /* hg throws "abandoned transaction" error only if this file exists */
+                    if (pathExists(transJournal)) {
+                        runProgram("hg", true, { "recover", "-R", cacheDir });
+                        runProgram("hg", true, { "pull", "-R", cacheDir, "--", uri });
+                    } else {
+                        throw ExecError(e.status, fmt("'hg pull' %s", statusToString(e.status)));
+                    }
+                }
+            } else {
+                createDirs(dirOf(cacheDir));
+                runProgram("hg", true, { "clone", "--noupdate", "--", uri, cacheDir });
+            }
+        }
+
+        writeFile(stampFile, "");
+    }
+
+    auto tokens = tokenizeString<std::vector<std::string>>(
+        runProgram("hg", true, { "log", "-R", cacheDir, "-r", rev, "--template", "{node} {rev} {branch}" }));
+    assert(tokens.size() == 3);
+
+    HgInfo hgInfo;
+    hgInfo.rev = tokens[0];
+    hgInfo.revCount = std::stoull(tokens[1]);
+    hgInfo.branch = tokens[2];
+
+    std::string storeLinkName = hashString(htSHA512, name + std::string("\0"s) + hgInfo.rev).to_string(Base32, false);
+    Path storeLink = fmt("%s/.hg/%s.link", cacheDir, storeLinkName);
+
+    try {
+        auto json = nlohmann::json::parse(readFile(storeLink));
+
+        assert(json["name"] == name && json["rev"] == hgInfo.rev);
+
+        hgInfo.storePath = json["storePath"];
+
+        if (store->isValidPath(store->parseStorePath(hgInfo.storePath))) {
+            printTalkative("using cached Mercurial store path '%s'", hgInfo.storePath);
+            return hgInfo;
+        }
+
+    } catch (SysError & e) {
+        if (e.errNo != ENOENT) throw;
+    }
+
+    Path tmpDir = createTempDir();
+    AutoDelete delTmpDir(tmpDir, true);
+
+    runProgram("hg", true, { "archive", "-R", cacheDir, "-r", rev, tmpDir });
+
+    deletePath(tmpDir + "/.hg_archival.txt");
+
+    hgInfo.storePath = store->printStorePath(store->addToStore(name, tmpDir));
+
+    nlohmann::json json;
+    json["storePath"] = hgInfo.storePath;
+    json["uri"] = uri;
+    json["name"] = name;
+    json["branch"] = hgInfo.branch;
+    json["rev"] = hgInfo.rev;
+    json["revCount"] = hgInfo.revCount;
+
+    writeFile(storeLink, json.dump());
+
+    return hgInfo;
+}
+
 static void prim_fetchMercurial(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     std::string url;
-    std::optional<Hash> rev;
-    std::optional<std::string> ref;
+    std::string rev;
     std::string name = "source";
     PathSet context;
 
@@ -26,15 +182,8 @@ static void prim_fetchMercurial(EvalState & state, const Pos & pos, Value * * ar
             string n(attr.name);
             if (n == "url")
                 url = state.coerceToString(*attr.pos, *attr.value, context, false, false);
-            else if (n == "rev") {
-                // Ugly: unlike fetchGit, here the "rev" attribute can
-                // be both a revision or a branch/tag name.
-                auto value = state.forceStringNoCtx(*attr.value, *attr.pos);
-                if (std::regex_match(value, revRegex))
-                    rev = Hash(value, htSHA1);
-                else
-                    ref = value;
-            }
+            else if (n == "rev")
+                rev = state.forceStringNoCtx(*attr.value, *attr.pos);
             else if (n == "name")
                 name = state.forceStringNoCtx(*attr.value, *attr.pos);
             else
@@ -51,35 +200,18 @@ static void prim_fetchMercurial(EvalState & state, const Pos & pos, Value * * ar
     // whitelist. Ah well.
     state.checkURI(url);
 
-    if (evalSettings.pureEval && !rev)
-        throw Error("in pure evaluation mode, 'fetchMercurial' requires a Mercurial revision");
-
-    fetchers::Attrs attrs;
-    attrs.insert_or_assign("type", "hg");
-    attrs.insert_or_assign("url", url.find("://") != std::string::npos ? url : "file://" + url);
-    if (ref) attrs.insert_or_assign("ref", *ref);
-    if (rev) attrs.insert_or_assign("rev", rev->gitRev());
-    auto input = fetchers::inputFromAttrs(attrs);
-
-    // FIXME: use name
-    auto [tree, input2] = input->fetchTree(state.store);
+    auto hgInfo = exportMercurial(state.store, url, rev, name);
 
     state.mkAttrs(v, 8);
-    auto storePath = state.store->printStorePath(tree.storePath);
-    mkString(*state.allocAttr(v, state.sOutPath), storePath, PathSet({storePath}));
-    if (input2->getRef())
-        mkString(*state.allocAttr(v, state.symbols.create("branch")), *input2->getRef());
-    // Backward compatibility: set 'rev' to
-    // 0000000000000000000000000000000000000000 for a dirty tree.
-    auto rev2 = input2->getRev().value_or(Hash(htSHA1));
-    mkString(*state.allocAttr(v, state.symbols.create("rev")), rev2.gitRev());
-    mkString(*state.allocAttr(v, state.symbols.create("shortRev")), std::string(rev2.gitRev(), 0, 12));
-    if (tree.info.revCount)
-        mkInt(*state.allocAttr(v, state.symbols.create("revCount")), *tree.info.revCount);
+    mkString(*state.allocAttr(v, state.sOutPath), hgInfo.storePath, PathSet({hgInfo.storePath}));
+    mkString(*state.allocAttr(v, state.symbols.create("branch")), hgInfo.branch);
+    mkString(*state.allocAttr(v, state.symbols.create("rev")), hgInfo.rev);
+    mkString(*state.allocAttr(v, state.symbols.create("shortRev")), std::string(hgInfo.rev, 0, 12));
+    mkInt(*state.allocAttr(v, state.symbols.create("revCount")), hgInfo.revCount);
     v.attrs->sort();
 
     if (state.allowedPaths)
-        state.allowedPaths->insert(tree.actualPath);
+        state.allowedPaths->insert(state.store->toRealPath(hgInfo.storePath));
 }
 
 static RegisterPrimOp r("fetchMercurial", 1, prim_fetchMercurial);

src/libexpr/primops/fetchTree.cc

@@ -1,173 +0,0 @@
-#include "primops.hh"
-#include "eval-inline.hh"
-#include "store-api.hh"
-#include "fetchers.hh"
-#include "filetransfer.hh"
-#include "registry.hh"
-
-#include <ctime>
-#include <iomanip>
-
-namespace nix {
-
-void emitTreeAttrs(
-    EvalState & state,
-    const fetchers::Tree & tree,
-    std::shared_ptr<const fetchers::Input> input,
-    Value & v)
-{
-    state.mkAttrs(v, 8);
-
-    auto storePath = state.store->printStorePath(tree.storePath);
-
-    mkString(*state.allocAttr(v, state.sOutPath), storePath, PathSet({storePath}));
-
-    assert(tree.info.narHash);
-    mkString(*state.allocAttr(v, state.symbols.create("narHash")),
-        tree.info.narHash.to_string(SRI));
-
-    if (input->getRev()) {
-        mkString(*state.allocAttr(v, state.symbols.create("rev")), input->getRev()->gitRev());
-        mkString(*state.allocAttr(v, state.symbols.create("shortRev")), input->getRev()->gitShortRev());
-    }
-
-    if (tree.info.revCount)
-        mkInt(*state.allocAttr(v, state.symbols.create("revCount")), *tree.info.revCount);
-
-    if (tree.info.lastModified) {
-        mkInt(*state.allocAttr(v, state.symbols.create("lastModified")), *tree.info.lastModified);
-        mkString(*state.allocAttr(v, state.symbols.create("lastModifiedDate")),
-            fmt("%s", std::put_time(std::gmtime(&*tree.info.lastModified), "%Y%m%d%H%M%S")));
-    }
-
-    v.attrs->sort();
-}
-
-static void prim_fetchTree(EvalState & state, const Pos & pos, Value * * args, Value & v)
-{
-    settings.requireExperimentalFeature("flakes");
-
-    std::shared_ptr<const fetchers::Input> input;
-    PathSet context;
-
-    state.forceValue(*args[0]);
-
-    if (args[0]->type == tAttrs) {
-        state.forceAttrs(*args[0], pos);
-
-        fetchers::Attrs attrs;
-
-        for (auto & attr : *args[0]->attrs) {
-            state.forceValue(*attr.value);
-            if (attr.value->type == tString)
-                attrs.emplace(attr.name, attr.value->string.s);
-            else if (attr.value->type == tBool)
-                attrs.emplace(attr.name, attr.value->boolean);
-            else if (attr.value->type == tInt)
-                attrs.emplace(attr.name, attr.value->integer);
-            else
-                throw TypeError("fetchTree argument '%s' is %s while a string, Boolean or integer is expected",
-                    attr.name, showType(*attr.value));
-        }
-
-        if (!attrs.count("type"))
-            throw Error("attribute 'type' is missing in call to 'fetchTree', at %s", pos);
-
-        input = fetchers::inputFromAttrs(attrs);
-    } else
-        input = fetchers::inputFromURL(state.coerceToString(pos, *args[0], context, false, false));
-
-    if (!evalSettings.pureEval && !input->isDirect())
-        input = lookupInRegistries(state.store, input).first;
-
-    if (evalSettings.pureEval && !input->isImmutable())
-        throw Error("in pure evaluation mode, 'fetchTree' requires an immutable input, at %s", pos);
-
-    // FIXME: use fetchOrSubstituteTree
-    auto [tree, input2] = input->fetchTree(state.store);
-
-    if (state.allowedPaths)
-        state.allowedPaths->insert(tree.actualPath);
-
-    emitTreeAttrs(state, tree, input2, v);
-}
-
-static RegisterPrimOp r("fetchTree", 1, prim_fetchTree);
-
-static void fetch(EvalState & state, const Pos & pos, Value * * args, Value & v,
-    const string & who, bool unpack, std::string name)
-{
-    std::optional<std::string> url;
-    std::optional<Hash> expectedHash;
-
-    state.forceValue(*args[0]);
-
-    if (args[0]->type == tAttrs) {
-
-        state.forceAttrs(*args[0], pos);
-
-        for (auto & attr : *args[0]->attrs) {
-            string n(attr.name);
-            if (n == "url")
-                url = state.forceStringNoCtx(*attr.value, *attr.pos);
-            else if (n == "sha256")
-                expectedHash = Hash(state.forceStringNoCtx(*attr.value, *attr.pos), htSHA256);
-            else if (n == "name")
-                name = state.forceStringNoCtx(*attr.value, *attr.pos);
-            else
-                throw EvalError("unsupported argument '%s' to '%s', at %s",
-                    attr.name, who, *attr.pos);
-        }
-
-        if (!url)
-            throw EvalError("'url' argument required, at %s", pos);
-
-    } else
-        url = state.forceStringNoCtx(*args[0], pos);
-
-    url = resolveUri(*url);
-
-    state.checkURI(*url);
-
-    if (name == "")
-        name = baseNameOf(*url);
-
-    if (evalSettings.pureEval && !expectedHash)
-        throw Error("in pure evaluation mode, '%s' requires a 'sha256' argument", who);
-
-    auto storePath =
-        unpack
-        ? fetchers::downloadTarball(state.store, *url, name, (bool) expectedHash).storePath
-        : fetchers::downloadFile(state.store, *url, name, (bool) expectedHash).storePath;
-
-    auto path = state.store->toRealPath(storePath);
-
-    if (expectedHash) {
-        auto hash = unpack
-            ? state.store->queryPathInfo(storePath)->narHash
-            : hashFile(htSHA256, path);
-        if (hash != *expectedHash)
-            throw Error((unsigned int) 102, "hash mismatch in file downloaded from '%s':\n  wanted: %s\n  got:    %s",
-                *url, expectedHash->to_string(), hash.to_string());
-    }
-
-    if (state.allowedPaths)
-        state.allowedPaths->insert(path);
-
-    mkString(v, path, PathSet({path}));
-}
-
-static void prim_fetchurl(EvalState & state, const Pos & pos, Value * * args, Value & v)
-{
-    fetch(state, pos, args, v, "fetchurl", false, "");
-}
-
-static void prim_fetchTarball(EvalState & state, const Pos & pos, Value * * args, Value & v)
-{
-    fetch(state, pos, args, v, "fetchTarball", true, "source");
-}
-
-static RegisterPrimOp r2("__fetchurl", 1, prim_fetchurl);
-static RegisterPrimOp r3("fetchTarball", 1, prim_fetchTarball);
-
-}

src/libexpr/symbol-table.hh

@@ -28,12 +28,6 @@ public:
         return s == s2.s;
     }
 
-    // FIXME: remove
-    bool operator == (std::string_view s2) const
-    {
-        return s->compare(s2) == 0;
-    }
-
     bool operator != (const Symbol & s2) const
     {
         return s != s2.s;
@@ -74,10 +68,9 @@ private:
     Symbols symbols;
 
 public:
-    Symbol create(std::string_view s)
+    Symbol create(const string & s)
     {
-        // FIXME: avoid allocation if 's' already exists in the symbol table.
-        std::pair<Symbols::iterator, bool> res = symbols.emplace(std::string(s));
+        std::pair<Symbols::iterator, bool> res = symbols.insert(s);
         return Symbol(&*res.first);
     }
 

src/libexpr/value.hh

@@ -166,11 +166,6 @@ struct Value
     {
         return type == tList1 ? 1 : type == tList2 ? 2 : bigList.size;
     }
-
-    /* Check whether forcing this value requires a trivial amount of
-       computation. In particular, function applications are
-       non-trivial. */
-    bool isTrivial() const;
 };
 
 
@@ -258,17 +253,12 @@ void mkPath(Value & v, const char * s);
 
 
 #if HAVE_BOEHMGC
-typedef std::vector<Value *, traceable_allocator<Value *> > ValueVector;
-typedef std::map<Symbol, Value *, std::less<Symbol>, traceable_allocator<std::pair<const Symbol, Value *> > > ValueMap;
+typedef std::vector<Value *, gc_allocator<Value *> > ValueVector;
+typedef std::map<Symbol, Value *, std::less<Symbol>, gc_allocator<std::pair<const Symbol, Value *> > > ValueMap;
 #else
 typedef std::vector<Value *> ValueVector;
 typedef std::map<Symbol, Value *> ValueMap;
 #endif
 
 
-/* A value allocated in traceable memory. */
-typedef std::shared_ptr<Value *> RootValue;
-
-RootValue allocRootValue(Value * v);
-
 }

src/libfetchers/attrs.cc

@@ -1,107 +0,0 @@
-#include "attrs.hh"
-#include "fetchers.hh"
-
-#include <nlohmann/json.hpp>
-
-namespace nix::fetchers {
-
-Attrs jsonToAttrs(const nlohmann::json & json)
-{
-    Attrs attrs;
-
-    for (auto & i : json.items()) {
-        if (i.value().is_number())
-            attrs.emplace(i.key(), i.value().get<int64_t>());
-        else if (i.value().is_string())
-            attrs.emplace(i.key(), i.value().get<std::string>());
-        else if (i.value().is_boolean())
-            attrs.emplace(i.key(), i.value().get<bool>());
-        else
-            throw Error("unsupported input attribute type in lock file");
-    }
-
-    return attrs;
-}
-
-nlohmann::json attrsToJson(const Attrs & attrs)
-{
-    nlohmann::json json;
-    for (auto & attr : attrs) {
-        if (auto v = std::get_if<int64_t>(&attr.second)) {
-            json[attr.first] = *v;
-        } else if (auto v = std::get_if<std::string>(&attr.second)) {
-            json[attr.first] = *v;
-        } else if (auto v = std::get_if<Explicit<bool>>(&attr.second)) {
-            json[attr.first] = v->t;
-        } else abort();
-    }
-    return json;
-}
-
-std::optional<std::string> maybeGetStrAttr(const Attrs & attrs, const std::string & name)
-{
-    auto i = attrs.find(name);
-    if (i == attrs.end()) return {};
-    if (auto v = std::get_if<std::string>(&i->second))
-        return *v;
-    throw Error("input attribute '%s' is not a string %s", name, attrsToJson(attrs).dump());
-}
-
-std::string getStrAttr(const Attrs & attrs, const std::string & name)
-{
-    auto s = maybeGetStrAttr(attrs, name);
-    if (!s)
-        throw Error("input attribute '%s' is missing", name);
-    return *s;
-}
-
-std::optional<int64_t> maybeGetIntAttr(const Attrs & attrs, const std::string & name)
-{
-    auto i = attrs.find(name);
-    if (i == attrs.end()) return {};
-    if (auto v = std::get_if<int64_t>(&i->second))
-        return *v;
-    throw Error("input attribute '%s' is not an integer", name);
-}
-
-int64_t getIntAttr(const Attrs & attrs, const std::string & name)
-{
-    auto s = maybeGetIntAttr(attrs, name);
-    if (!s)
-        throw Error("input attribute '%s' is missing", name);
-    return *s;
-}
-
-std::optional<bool> maybeGetBoolAttr(const Attrs & attrs, const std::string & name)
-{
-    auto i = attrs.find(name);
-    if (i == attrs.end()) return {};
-    if (auto v = std::get_if<int64_t>(&i->second))
-        return *v;
-    throw Error("input attribute '%s' is not a Boolean", name);
-}
-
-bool getBoolAttr(const Attrs & attrs, const std::string & name)
-{
-    auto s = maybeGetBoolAttr(attrs, name);
-    if (!s)
-        throw Error("input attribute '%s' is missing", name);
-    return *s;
-}
-
-std::map<std::string, std::string> attrsToQuery(const Attrs & attrs)
-{
-    std::map<std::string, std::string> query;
-    for (auto & attr : attrs) {
-        if (auto v = std::get_if<int64_t>(&attr.second)) {
-            query.insert_or_assign(attr.first, fmt("%d", *v));
-        } else if (auto v = std::get_if<std::string>(&attr.second)) {
-            query.insert_or_assign(attr.first, *v);
-        } else if (auto v = std::get_if<Explicit<bool>>(&attr.second)) {
-            query.insert_or_assign(attr.first, v->t ? "1" : "0");
-        } else abort();
-    }
-    return query;
-}
-
-}

src/libfetchers/attrs.hh

@@ -1,39 +0,0 @@
-#pragma once
-
-#include "types.hh"
-
-#include <variant>
-
-#include <nlohmann/json_fwd.hpp>
-
-namespace nix::fetchers {
-
-/* Wrap bools to prevent string literals (i.e. 'char *') from being
-   cast to a bool in Attr. */
-template<typename T>
-struct Explicit {
-    T t;
-};
-
-typedef std::variant<std::string, int64_t, Explicit<bool>> Attr;
-typedef std::map<std::string, Attr> Attrs;
-
-Attrs jsonToAttrs(const nlohmann::json & json);
-
-nlohmann::json attrsToJson(const Attrs & attrs);
-
-std::optional<std::string> maybeGetStrAttr(const Attrs & attrs, const std::string & name);
-
-std::string getStrAttr(const Attrs & attrs, const std::string & name);
-
-std::optional<int64_t> maybeGetIntAttr(const Attrs & attrs, const std::string & name);
-
-int64_t getIntAttr(const Attrs & attrs, const std::string & name);
-
-std::optional<bool> maybeGetBoolAttr(const Attrs & attrs, const std::string & name);
-
-bool getBoolAttr(const Attrs & attrs, const std::string & name);
-
-std::map<std::string, std::string> attrsToQuery(const Attrs & attrs);
-
-}

src/libfetchers/cache.cc

@@ -1,121 +0,0 @@
-#include "cache.hh"
-#include "sqlite.hh"
-#include "sync.hh"
-#include "store-api.hh"
-
-#include <nlohmann/json.hpp>
-
-namespace nix::fetchers {
-
-static const char * schema = R"sql(
-
-create table if not exists Cache (
-    input     text not null,
-    info      text not null,
-    path      text not null,
-    immutable integer not null,
-    timestamp integer not null,
-    primary key (input)
-);
-)sql";
-
-struct CacheImpl : Cache
-{
-    struct State
-    {
-        SQLite db;
-        SQLiteStmt add, lookup;
-    };
-
-    Sync<State> _state;
-
-    CacheImpl()
-    {
-        auto state(_state.lock());
-
-        auto dbPath = getCacheDir() + "/nix/fetcher-cache-v1.sqlite";
-        createDirs(dirOf(dbPath));
-
-        state->db = SQLite(dbPath);
-        state->db.isCache();
-        state->db.exec(schema);
-
-        state->add.create(state->db,
-            "insert or replace into Cache(input, info, path, immutable, timestamp) values (?, ?, ?, ?, ?)");
-
-        state->lookup.create(state->db,
-            "select info, path, immutable, timestamp from Cache where input = ?");
-    }
-
-    void add(
-        ref<Store> store,
-        const Attrs & inAttrs,
-        const Attrs & infoAttrs,
-        const StorePath & storePath,
-        bool immutable) override
-    {
-        _state.lock()->add.use()
-            (attrsToJson(inAttrs).dump())
-            (attrsToJson(infoAttrs).dump())
-            (store->printStorePath(storePath))
-            (immutable)
-            (time(0)).exec();
-    }
-
-    std::optional<std::pair<Attrs, StorePath>> lookup(
-        ref<Store> store,
-        const Attrs & inAttrs) override
-    {
-        if (auto res = lookupExpired(store, inAttrs)) {
-            if (!res->expired)
-                return std::make_pair(std::move(res->infoAttrs), std::move(res->storePath));
-            debug("ignoring expired cache entry '%s'",
-                attrsToJson(inAttrs).dump());
-        }
-        return {};
-    }
-
-    std::optional<Result> lookupExpired(
-        ref<Store> store,
-        const Attrs & inAttrs) override
-    {
-        auto state(_state.lock());
-
-        auto inAttrsJson = attrsToJson(inAttrs).dump();
-
-        auto stmt(state->lookup.use()(inAttrsJson));
-        if (!stmt.next()) {
-            debug("did not find cache entry for '%s'", inAttrsJson);
-            return {};
-        }
-
-        auto infoJson = stmt.getStr(0);
-        auto storePath = store->parseStorePath(stmt.getStr(1));
-        auto immutable = stmt.getInt(2) != 0;
-        auto timestamp = stmt.getInt(3);
-
-        store->addTempRoot(storePath);
-        if (!store->isValidPath(storePath)) {
-            // FIXME: we could try to substitute 'storePath'.
-            debug("ignoring disappeared cache entry '%s'", inAttrsJson);
-            return {};
-        }
-
-        debug("using cache entry '%s' -> '%s', '%s'",
-            inAttrsJson, infoJson, store->printStorePath(storePath));
-
-        return Result {
-            .expired = !immutable && (settings.tarballTtl.get() == 0 || timestamp + settings.tarballTtl < time(0)),
-            .infoAttrs = jsonToAttrs(nlohmann::json::parse(infoJson)),
-            .storePath = std::move(storePath)
-        };
-    }
-};
-
-ref<Cache> getCache()
-{
-    static auto cache = std::make_shared<CacheImpl>();
-    return ref<Cache>(cache);
-}
-
-}

src/libfetchers/cache.hh

@@ -1,34 +0,0 @@
-#pragma once
-
-#include "fetchers.hh"
-
-namespace nix::fetchers {
-
-struct Cache
-{
-    virtual void add(
-        ref<Store> store,
-        const Attrs & inAttrs,
-        const Attrs & infoAttrs,
-        const StorePath & storePath,
-        bool immutable) = 0;
-
-    virtual std::optional<std::pair<Attrs, StorePath>> lookup(
-        ref<Store> store,
-        const Attrs & inAttrs) = 0;
-
-    struct Result
-    {
-        bool expired = false;
-        Attrs infoAttrs;
-        StorePath storePath;
-    };
-
-    virtual std::optional<Result> lookupExpired(
-        ref<Store> store,
-        const Attrs & inAttrs) = 0;
-};
-
-ref<Cache> getCache();
-
-}

src/libfetchers/fetchers.cc

@@ -1,86 +0,0 @@
-#include "fetchers.hh"
-#include "store-api.hh"
-
-#include <nlohmann/json.hpp>
-
-namespace nix::fetchers {
-
-std::unique_ptr<std::vector<std::unique_ptr<InputScheme>>> inputSchemes = nullptr;
-
-void registerInputScheme(std::unique_ptr<InputScheme> && inputScheme)
-{
-    if (!inputSchemes) inputSchemes = std::make_unique<std::vector<std::unique_ptr<InputScheme>>>();
-    inputSchemes->push_back(std::move(inputScheme));
-}
-
-std::unique_ptr<Input> inputFromURL(const ParsedURL & url)
-{
-    for (auto & inputScheme : *inputSchemes) {
-        auto res = inputScheme->inputFromURL(url);
-        if (res) return res;
-    }
-    throw Error("input '%s' is unsupported", url.url);
-}
-
-std::unique_ptr<Input> inputFromURL(const std::string & url)
-{
-    return inputFromURL(parseURL(url));
-}
-
-std::unique_ptr<Input> inputFromAttrs(const Attrs & attrs)
-{
-    auto attrs2(attrs);
-    attrs2.erase("narHash");
-    for (auto & inputScheme : *inputSchemes) {
-        auto res = inputScheme->inputFromAttrs(attrs2);
-        if (res) {
-            if (auto narHash = maybeGetStrAttr(attrs, "narHash"))
-                // FIXME: require SRI hash.
-                res->narHash = Hash(*narHash);
-            return res;
-        }
-    }
-    throw Error("input '%s' is unsupported", attrsToJson(attrs));
-}
-
-Attrs Input::toAttrs() const
-{
-    auto attrs = toAttrsInternal();
-    if (narHash)
-        attrs.emplace("narHash", narHash->to_string(SRI));
-    attrs.emplace("type", type());
-    return attrs;
-}
-
-std::pair<Tree, std::shared_ptr<const Input>> Input::fetchTree(ref<Store> store) const
-{
-    auto [tree, input] = fetchTreeInternal(store);
-
-    if (tree.actualPath == "")
-        tree.actualPath = store->toRealPath(tree.storePath);
-
-    if (!tree.info.narHash)
-        tree.info.narHash = store->queryPathInfo(tree.storePath)->narHash;
-
-    if (input->narHash)
-        assert(input->narHash == tree.info.narHash);
-
-    if (narHash && narHash != input->narHash)
-        throw Error("NAR hash mismatch in input '%s' (%s), expected '%s', got '%s'",
-            to_string(), tree.actualPath, narHash->to_string(SRI), input->narHash->to_string(SRI));
-
-    return {std::move(tree), input};
-}
-
-std::shared_ptr<const Input> Input::applyOverrides(
-    std::optional<std::string> ref,
-    std::optional<Hash> rev) const
-{
-    if (ref)
-        throw Error("don't know how to apply '%s' to '%s'", *ref, to_string());
-    if (rev)
-        throw Error("don't know how to apply '%s' to '%s'", rev->to_string(Base16, false), to_string());
-    return shared_from_this();
-}
-
-}

src/libfetchers/fetchers.hh

@@ -1,119 +0,0 @@
-#pragma once
-
-#include "types.hh"
-#include "hash.hh"
-#include "path.hh"
-#include "tree-info.hh"
-#include "attrs.hh"
-#include "url.hh"
-
-#include <memory>
-
-namespace nix { class Store; }
-
-namespace nix::fetchers {
-
-struct Input;
-
-struct Tree
-{
-    Path actualPath;
-    StorePath storePath;
-    TreeInfo info;
-};
-
-struct Input : std::enable_shared_from_this<Input>
-{
-    std::optional<Hash> narHash; // FIXME: implement
-
-    virtual std::string type() const = 0;
-
-    virtual ~Input() { }
-
-    virtual bool operator ==(const Input & other) const { return false; }
-
-    /* Check whether this is a "direct" input, that is, not
-       one that goes through a registry. */
-    virtual bool isDirect() const { return true; }
-
-    /* Check whether this is an "immutable" input, that is,
-       one that contains a commit hash or content hash. */
-    virtual bool isImmutable() const { return (bool) narHash; }
-
-    virtual bool contains(const Input & other) const { return false; }
-
-    virtual std::optional<std::string> getRef() const { return {}; }
-
-    virtual std::optional<Hash> getRev() const { return {}; }
-
-    virtual ParsedURL toURL() const = 0;
-
-    std::string to_string() const
-    {
-        return toURL().to_string();
-    }
-
-    Attrs toAttrs() const;
-
-    std::pair<Tree, std::shared_ptr<const Input>> fetchTree(ref<Store> store) const;
-
-    virtual std::shared_ptr<const Input> applyOverrides(
-        std::optional<std::string> ref,
-        std::optional<Hash> rev) const;
-
-    virtual std::optional<Path> getSourcePath() const { return {}; }
-
-    virtual void markChangedFile(
-        std::string_view file,
-        std::optional<std::string> commitMsg) const
-    { assert(false); }
-
-    virtual void clone(const Path & destDir) const
-    {
-        throw Error("do not know how to clone input '%s'", to_string());
-    }
-
-private:
-
-    virtual std::pair<Tree, std::shared_ptr<const Input>> fetchTreeInternal(ref<Store> store) const = 0;
-
-    virtual Attrs toAttrsInternal() const = 0;
-};
-
-struct InputScheme
-{
-    virtual ~InputScheme() { }
-
-    virtual std::unique_ptr<Input> inputFromURL(const ParsedURL & url) = 0;
-
-    virtual std::unique_ptr<Input> inputFromAttrs(const Attrs & attrs) = 0;
-};
-
-std::unique_ptr<Input> inputFromURL(const ParsedURL & url);
-
-std::unique_ptr<Input> inputFromURL(const std::string & url);
-
-std::unique_ptr<Input> inputFromAttrs(const Attrs & attrs);
-
-void registerInputScheme(std::unique_ptr<InputScheme> && fetcher);
-
-struct DownloadFileResult
-{
-    StorePath storePath;
-    std::string etag;
-    std::string effectiveUrl;
-};
-
-DownloadFileResult downloadFile(
-    ref<Store> store,
-    const std::string & url,
-    const std::string & name,
-    bool immutable);
-
-Tree downloadTarball(
-    ref<Store> store,
-    const std::string & url,
-    const std::string & name,
-    bool immutable);
-
-}

src/libfetchers/git.cc

@@ -1,495 +0,0 @@
-#include "fetchers.hh"
-#include "cache.hh"
-#include "globals.hh"
-#include "tarfile.hh"
-#include "store-api.hh"
-
-#include <sys/time.h>
-
-using namespace std::string_literals;
-
-namespace nix::fetchers {
-
-static std::string readHead(const Path & path)
-{
-    return chomp(runProgram("git", true, { "-C", path, "rev-parse", "--abbrev-ref", "HEAD" }));
-}
-
-static bool isNotDotGitDirectory(const Path & path)
-{
-    static const std::regex gitDirRegex("^(?:.*/)?\\.git$");
-
-    return not std::regex_match(path, gitDirRegex);
-}
-
-struct GitInput : Input
-{
-    ParsedURL url;
-    std::optional<std::string> ref;
-    std::optional<Hash> rev;
-    bool shallow = false;
-    bool submodules = false;
-
-    GitInput(const ParsedURL & url) : url(url)
-    { }
-
-    std::string type() const override { return "git"; }
-
-    bool operator ==(const Input & other) const override
-    {
-        auto other2 = dynamic_cast<const GitInput *>(&other);
-        return
-            other2
-            && url == other2->url
-            && rev == other2->rev
-            && ref == other2->ref;
-    }
-
-    bool isImmutable() const override
-    {
-        return (bool) rev || narHash;
-    }
-
-    std::optional<std::string> getRef() const override { return ref; }
-
-    std::optional<Hash> getRev() const override { return rev; }
-
-    ParsedURL toURL() const override
-    {
-        ParsedURL url2(url);
-        if (url2.scheme != "git") url2.scheme = "git+" + url2.scheme;
-        if (rev) url2.query.insert_or_assign("rev", rev->gitRev());
-        if (ref) url2.query.insert_or_assign("ref", *ref);
-        if (shallow) url2.query.insert_or_assign("shallow", "1");
-        return url2;
-    }
-
-    Attrs toAttrsInternal() const override
-    {
-        Attrs attrs;
-        attrs.emplace("url", url.to_string());
-        if (ref)
-            attrs.emplace("ref", *ref);
-        if (rev)
-            attrs.emplace("rev", rev->gitRev());
-        if (shallow)
-            attrs.emplace("shallow", true);
-        if (submodules)
-            attrs.emplace("submodules", true);
-        return attrs;
-    }
-
-    void clone(const Path & destDir) const override
-    {
-        auto [isLocal, actualUrl] = getActualUrl();
-
-        Strings args = {"clone"};
-
-        args.push_back(actualUrl);
-
-        if (ref) {
-            args.push_back("--branch");
-            args.push_back(*ref);
-        }
-
-        if (rev) throw Error("cloning a specific revision is not implemented");
-
-        args.push_back(destDir);
-
-        runProgram("git", true, args);
-    }
-
-    std::shared_ptr<const Input> applyOverrides(
-        std::optional<std::string> ref,
-        std::optional<Hash> rev) const override
-    {
-        if (!ref && !rev) return shared_from_this();
-
-        auto res = std::make_shared<GitInput>(*this);
-
-        if (ref) res->ref = ref;
-        if (rev) res->rev = rev;
-
-        if (!res->ref && res->rev)
-            throw Error("Git input '%s' has a commit hash but no branch/tag name", res->to_string());
-
-        return res;
-    }
-
-    std::optional<Path> getSourcePath() const override
-    {
-        if (url.scheme == "file" && !ref && !rev)
-            return url.path;
-        return {};
-    }
-
-    void markChangedFile(std::string_view file, std::optional<std::string> commitMsg) const override
-    {
-        auto sourcePath = getSourcePath();
-        assert(sourcePath);
-
-        runProgram("git", true,
-            { "-C", *sourcePath, "add", "--force", "--intent-to-add", std::string(file) });
-
-        if (commitMsg)
-            runProgram("git", true,
-                { "-C", *sourcePath, "commit", std::string(file), "-m", *commitMsg });
-    }
-
-    std::pair<bool, std::string> getActualUrl() const
-    {
-        // Don't clone file:// URIs (but otherwise treat them the
-        // same as remote URIs, i.e. don't use the working tree or
-        // HEAD).
-        static bool forceHttp = getEnv("_NIX_FORCE_HTTP") == "1"; // for testing
-        bool isLocal = url.scheme == "file" && !forceHttp;
-        return {isLocal, isLocal ? url.path : url.base};
-    }
-
-    std::pair<Tree, std::shared_ptr<const Input>> fetchTreeInternal(nix::ref<Store> store) const override
-    {
-        auto name = "source";
-
-        auto input = std::make_shared<GitInput>(*this);
-
-        assert(!rev || rev->type == htSHA1);
-
-        std::string cacheType = "git";
-        if (shallow) cacheType += "-shallow";
-        if (submodules) cacheType += "-submodules";
-
-        auto getImmutableAttrs = [&]()
-        {
-            return Attrs({
-                {"type", cacheType},
-                {"name", name},
-                {"rev", input->rev->gitRev()},
-            });
-        };
-
-        auto makeResult = [&](const Attrs & infoAttrs, StorePath && storePath)
-            -> std::pair<Tree, std::shared_ptr<const Input>>
-        {
-            assert(input->rev);
-            assert(!rev || rev == input->rev);
-            return {
-                Tree {
-                    .actualPath = store->toRealPath(storePath),
-                    .storePath = std::move(storePath),
-                    .info = TreeInfo {
-                        .revCount = shallow ? std::nullopt : std::optional(getIntAttr(infoAttrs, "revCount")),
-                        .lastModified = getIntAttr(infoAttrs, "lastModified"),
-                    },
-                },
-                input
-            };
-        };
-
-        if (rev) {
-            if (auto res = getCache()->lookup(store, getImmutableAttrs()))
-                return makeResult(res->first, std::move(res->second));
-        }
-
-        auto [isLocal, actualUrl_] = getActualUrl();
-        auto actualUrl = actualUrl_; // work around clang bug
-
-        // If this is a local directory and no ref or revision is
-        // given, then allow the use of an unclean working tree.
-        if (!input->ref && !input->rev && isLocal) {
-            bool clean = false;
-
-            /* Check whether this repo has any commits. There are
-               probably better ways to do this. */
-            auto gitDir = actualUrl + "/.git";
-            auto commonGitDir = chomp(runProgram(
-                "git",
-                true,
-                { "-C", actualUrl, "rev-parse", "--git-common-dir" }
-            ));
-            if (commonGitDir != ".git")
-                    gitDir = commonGitDir;
-
-            bool haveCommits = !readDirectory(gitDir + "/refs/heads").empty();
-
-            try {
-                if (haveCommits) {
-                    runProgram("git", true, { "-C", actualUrl, "diff-index", "--quiet", "HEAD", "--" });
-                    clean = true;
-                }
-            } catch (ExecError & e) {
-                if (!WIFEXITED(e.status) || WEXITSTATUS(e.status) != 1) throw;
-            }
-
-            if (!clean) {
-
-                /* This is an unclean working tree. So copy all tracked files. */
-
-                if (!settings.allowDirty)
-                    throw Error("Git tree '%s' is dirty", actualUrl);
-
-                if (settings.warnDirty)
-                    warn("Git tree '%s' is dirty", actualUrl);
-
-                auto gitOpts = Strings({ "-C", actualUrl, "ls-files", "-z" });
-                if (submodules)
-                    gitOpts.emplace_back("--recurse-submodules");
-
-                auto files = tokenizeString<std::set<std::string>>(
-                    runProgram("git", true, gitOpts), "\0"s);
-
-                PathFilter filter = [&](const Path & p) -> bool {
-                    assert(hasPrefix(p, actualUrl));
-                    std::string file(p, actualUrl.size() + 1);
-
-                    auto st = lstat(p);
-
-                    if (S_ISDIR(st.st_mode)) {
-                        auto prefix = file + "/";
-                        auto i = files.lower_bound(prefix);
-                        return i != files.end() && hasPrefix(*i, prefix);
-                    }
-
-                    return files.count(file);
-                };
-
-                auto storePath = store->addToStore("source", actualUrl, true, htSHA256, filter);
-
-                auto tree = Tree {
-                    .actualPath = store->printStorePath(storePath),
-                    .storePath = std::move(storePath),
-                    .info = TreeInfo {
-                        // FIXME: maybe we should use the timestamp of the last
-                        // modified dirty file?
-                        .lastModified = haveCommits ? std::stoull(runProgram("git", true, { "-C", actualUrl, "log", "-1", "--format=%ct", "HEAD" })) : 0,
-                    }
-                };
-
-                return {std::move(tree), input};
-            }
-        }
-
-        if (!input->ref) input->ref = isLocal ? readHead(actualUrl) : "master";
-
-        Attrs mutableAttrs({
-            {"type", cacheType},
-            {"name", name},
-            {"url", actualUrl},
-            {"ref", *input->ref},
-        });
-
-        Path repoDir;
-
-        if (isLocal) {
-
-            if (!input->rev)
-                input->rev = Hash(chomp(runProgram("git", true, { "-C", actualUrl, "rev-parse", *input->ref })), htSHA1);
-
-            repoDir = actualUrl;
-
-        } else {
-
-            if (auto res = getCache()->lookup(store, mutableAttrs)) {
-                auto rev2 = Hash(getStrAttr(res->first, "rev"), htSHA1);
-                if (!rev || rev == rev2) {
-                    input->rev = rev2;
-                    return makeResult(res->first, std::move(res->second));
-                }
-            }
-
-            Path cacheDir = getCacheDir() + "/nix/gitv3/" + hashString(htSHA256, actualUrl).to_string(Base32, false);
-            repoDir = cacheDir;
-
-            if (!pathExists(cacheDir)) {
-                createDirs(dirOf(cacheDir));
-                runProgram("git", true, { "init", "--bare", repoDir });
-            }
-
-            Path localRefFile =
-                input->ref->compare(0, 5, "refs/") == 0
-                ? cacheDir + "/" + *input->ref
-                : cacheDir + "/refs/heads/" + *input->ref;
-
-            bool doFetch;
-            time_t now = time(0);
-
-            /* If a rev was specified, we need to fetch if it's not in the
-               repo. */
-            if (input->rev) {
-                try {
-                    runProgram("git", true, { "-C", repoDir, "cat-file", "-e", input->rev->gitRev() });
-                    doFetch = false;
-                } catch (ExecError & e) {
-                    if (WIFEXITED(e.status)) {
-                        doFetch = true;
-                    } else {
-                        throw;
-                    }
-                }
-            } else {
-                /* If the local ref is older than ‘tarball-ttl’ seconds, do a
-                   git fetch to update the local ref to the remote ref. */
-                struct stat st;
-                doFetch = stat(localRefFile.c_str(), &st) != 0 ||
-                    (uint64_t) st.st_mtime + settings.tarballTtl <= (uint64_t) now;
-            }
-
-            if (doFetch) {
-                Activity act(*logger, lvlTalkative, actUnknown, fmt("fetching Git repository '%s'", actualUrl));
-
-                // FIXME: git stderr messes up our progress indicator, so
-                // we're using --quiet for now. Should process its stderr.
-                try {
-                    runProgram("git", true, { "-C", repoDir, "fetch", "--quiet", "--force", "--", actualUrl, fmt("%s:%s", *input->ref, *input->ref) });
-                } catch (Error & e) {
-                    if (!pathExists(localRefFile)) throw;
-                    warn("could not update local clone of Git repository '%s'; continuing with the most recent version", actualUrl);
-                }
-
-                struct timeval times[2];
-                times[0].tv_sec = now;
-                times[0].tv_usec = 0;
-                times[1].tv_sec = now;
-                times[1].tv_usec = 0;
-
-                utimes(localRefFile.c_str(), times);
-            }
-
-            if (!input->rev)
-                input->rev = Hash(chomp(readFile(localRefFile)), htSHA1);
-        }
-
-        bool isShallow = chomp(runProgram("git", true, { "-C", repoDir, "rev-parse", "--is-shallow-repository" })) == "true";
-
-        if (isShallow && !shallow)
-            throw Error("'%s' is a shallow Git repository, but a non-shallow repository is needed", actualUrl);
-
-        // FIXME: check whether rev is an ancestor of ref.
-
-        printTalkative("using revision %s of repo '%s'", input->rev->gitRev(), actualUrl);
-
-        /* Now that we know the ref, check again whether we have it in
-           the store. */
-        if (auto res = getCache()->lookup(store, getImmutableAttrs()))
-            return makeResult(res->first, std::move(res->second));
-
-        Path tmpDir = createTempDir();
-        AutoDelete delTmpDir(tmpDir, true);
-        PathFilter filter = defaultPathFilter;
-
-        if (submodules) {
-            Path tmpGitDir = createTempDir();
-            AutoDelete delTmpGitDir(tmpGitDir, true);
-
-            runProgram("git", true, { "init", tmpDir, "--separate-git-dir", tmpGitDir });
-            // TODO: repoDir might lack the ref (it only checks if rev
-            // exists, see FIXME above) so use a big hammer and fetch
-            // everything to ensure we get the rev.
-            runProgram("git", true, { "-C", tmpDir, "fetch", "--quiet", "--force",
-                                      "--update-head-ok", "--", repoDir, "refs/*:refs/*" });
-
-            runProgram("git", true, { "-C", tmpDir, "checkout", "--quiet", input->rev->gitRev() });
-            runProgram("git", true, { "-C", tmpDir, "remote", "add", "origin", actualUrl });
-            runProgram("git", true, { "-C", tmpDir, "submodule", "--quiet", "update", "--init", "--recursive" });
-
-            filter = isNotDotGitDirectory;
-        } else {
-            // FIXME: should pipe this, or find some better way to extract a
-            // revision.
-            auto source = sinkToSource([&](Sink & sink) {
-                RunOptions gitOptions("git", { "-C", repoDir, "archive", input->rev->gitRev() });
-                gitOptions.standardOut = &sink;
-                runProgram2(gitOptions);
-            });
-
-            unpackTarfile(*source, tmpDir);
-        }
-
-        auto storePath = store->addToStore(name, tmpDir, true, htSHA256, filter);
-
-        auto lastModified = std::stoull(runProgram("git", true, { "-C", repoDir, "log", "-1", "--format=%ct", input->rev->gitRev() }));
-
-        Attrs infoAttrs({
-            {"rev", input->rev->gitRev()},
-            {"lastModified", lastModified},
-        });
-
-        if (!shallow)
-            infoAttrs.insert_or_assign("revCount",
-                std::stoull(runProgram("git", true, { "-C", repoDir, "rev-list", "--count", input->rev->gitRev() })));
-
-        if (!this->rev)
-            getCache()->add(
-                store,
-                mutableAttrs,
-                infoAttrs,
-                storePath,
-                false);
-
-        getCache()->add(
-            store,
-            getImmutableAttrs(),
-            infoAttrs,
-            storePath,
-            true);
-
-        return makeResult(infoAttrs, std::move(storePath));
-    }
-};
-
-struct GitInputScheme : InputScheme
-{
-    std::unique_ptr<Input> inputFromURL(const ParsedURL & url) override
-    {
-        if (url.scheme != "git" &&
-            url.scheme != "git+http" &&
-            url.scheme != "git+https" &&
-            url.scheme != "git+ssh" &&
-            url.scheme != "git+file") return nullptr;
-
-        auto url2(url);
-        if (hasPrefix(url2.scheme, "git+")) url2.scheme = std::string(url2.scheme, 4);
-        url2.query.clear();
-
-        Attrs attrs;
-        attrs.emplace("type", "git");
-
-        for (auto &[name, value] : url.query) {
-            if (name == "rev" || name == "ref")
-                attrs.emplace(name, value);
-            else
-                url2.query.emplace(name, value);
-        }
-
-        attrs.emplace("url", url2.to_string());
-
-        return inputFromAttrs(attrs);
-    }
-
-    std::unique_ptr<Input> inputFromAttrs(const Attrs & attrs) override
-    {
-        if (maybeGetStrAttr(attrs, "type") != "git") return {};
-
-        for (auto & [name, value] : attrs)
-            if (name != "type" && name != "url" && name != "ref" && name != "rev" && name != "shallow" && name != "submodules")
-                throw Error("unsupported Git input attribute '%s'", name);
-
-        auto input = std::make_unique<GitInput>(parseURL(getStrAttr(attrs, "url")));
-        if (auto ref = maybeGetStrAttr(attrs, "ref")) {
-            if (!std::regex_match(*ref, refRegex))
-                throw BadURL("invalid Git branch/tag name '%s'", *ref);
-            input->ref = *ref;
-        }
-        if (auto rev = maybeGetStrAttr(attrs, "rev"))
-            input->rev = Hash(*rev, htSHA1);
-
-        input->shallow = maybeGetBoolAttr(attrs, "shallow").value_or(false);
-
-        input->submodules = maybeGetBoolAttr(attrs, "submodules").value_or(false);
-
-        return input;
-    }
-};
-
-static auto r1 = OnStartup([] { registerInputScheme(std::make_unique<GitInputScheme>()); });
-
-}

src/libfetchers/github.cc

@@ -1,216 +0,0 @@
-#include "filetransfer.hh"
-#include "cache.hh"
-#include "fetchers.hh"
-#include "globals.hh"
-#include "store-api.hh"
-
-#include <nlohmann/json.hpp>
-
-namespace nix::fetchers {
-
-std::regex ownerRegex("[a-zA-Z][a-zA-Z0-9_-]*", std::regex::ECMAScript);
-std::regex repoRegex("[a-zA-Z][a-zA-Z0-9_-]*", std::regex::ECMAScript);
-
-struct GitHubInput : Input
-{
-    std::string owner;
-    std::string repo;
-    std::optional<std::string> ref;
-    std::optional<Hash> rev;
-
-    std::string type() const override { return "github"; }
-
-    bool operator ==(const Input & other) const override
-    {
-        auto other2 = dynamic_cast<const GitHubInput *>(&other);
-        return
-            other2
-            && owner == other2->owner
-            && repo == other2->repo
-            && rev == other2->rev
-            && ref == other2->ref;
-    }
-
-    bool isImmutable() const override
-    {
-        return (bool) rev || narHash;
-    }
-
-    std::optional<std::string> getRef() const override { return ref; }
-
-    std::optional<Hash> getRev() const override { return rev; }
-
-    ParsedURL toURL() const override
-    {
-        auto path = owner + "/" + repo;
-        assert(!(ref && rev));
-        if (ref) path += "/" + *ref;
-        if (rev) path += "/" + rev->to_string(Base16, false);
-        return ParsedURL {
-            .scheme = "github",
-            .path = path,
-        };
-    }
-
-    Attrs toAttrsInternal() const override
-    {
-        Attrs attrs;
-        attrs.emplace("owner", owner);
-        attrs.emplace("repo", repo);
-        if (ref)
-            attrs.emplace("ref", *ref);
-        if (rev)
-            attrs.emplace("rev", rev->gitRev());
-        return attrs;
-    }
-
-    void clone(const Path & destDir) const override
-    {
-        std::shared_ptr<const Input> input = inputFromURL(fmt("git+ssh://git@github.com/%s/%s.git", owner, repo));
-        input = input->applyOverrides(ref.value_or("master"), rev);
-        input->clone(destDir);
-    }
-
-    std::pair<Tree, std::shared_ptr<const Input>> fetchTreeInternal(nix::ref<Store> store) const override
-    {
-        auto rev = this->rev;
-        auto ref = this->ref.value_or("master");
-
-        if (!rev) {
-            auto url = fmt("https://api.github.com/repos/%s/%s/commits/%s",
-                owner, repo, ref);
-            auto json = nlohmann::json::parse(
-                readFile(
-                    store->toRealPath(
-                        downloadFile(store, url, "source", false).storePath)));
-            rev = Hash(json["sha"], htSHA1);
-            debug("HEAD revision for '%s' is %s", url, rev->gitRev());
-        }
-
-        auto input = std::make_shared<GitHubInput>(*this);
-        input->ref = {};
-        input->rev = *rev;
-
-        Attrs immutableAttrs({
-            {"type", "git-tarball"},
-            {"rev", rev->gitRev()},
-        });
-
-        if (auto res = getCache()->lookup(store, immutableAttrs)) {
-            return {
-                Tree{
-                    .actualPath = store->toRealPath(res->second),
-                    .storePath = std::move(res->second),
-                    .info = TreeInfo {
-                        .lastModified = getIntAttr(res->first, "lastModified"),
-                    },
-                },
-                input
-            };
-        }
-
-        // FIXME: use regular /archive URLs instead? api.github.com
-        // might have stricter rate limits.
-
-        auto url = fmt("https://api.github.com/repos/%s/%s/tarball/%s",
-            owner, repo, rev->to_string(Base16, false));
-
-        std::string accessToken = settings.githubAccessToken.get();
-        if (accessToken != "")
-            url += "?access_token=" + accessToken;
-
-        auto tree = downloadTarball(store, url, "source", true);
-
-        getCache()->add(
-            store,
-            immutableAttrs,
-            {
-                {"rev", rev->gitRev()},
-                {"lastModified", *tree.info.lastModified}
-            },
-            tree.storePath,
-            true);
-
-        return {std::move(tree), input};
-    }
-
-    std::shared_ptr<const Input> applyOverrides(
-        std::optional<std::string> ref,
-        std::optional<Hash> rev) const override
-    {
-        if (!ref && !rev) return shared_from_this();
-
-        auto res = std::make_shared<GitHubInput>(*this);
-
-        if (ref) res->ref = ref;
-        if (rev) res->rev = rev;
-
-        return res;
-    }
-};
-
-struct GitHubInputScheme : InputScheme
-{
-    std::unique_ptr<Input> inputFromURL(const ParsedURL & url) override
-    {
-        if (url.scheme != "github") return nullptr;
-
-        auto path = tokenizeString<std::vector<std::string>>(url.path, "/");
-        auto input = std::make_unique<GitHubInput>();
-
-        if (path.size() == 2) {
-        } else if (path.size() == 3) {
-            if (std::regex_match(path[2], revRegex))
-                input->rev = Hash(path[2], htSHA1);
-            else if (std::regex_match(path[2], refRegex))
-                input->ref = path[2];
-            else
-                throw BadURL("in GitHub URL '%s', '%s' is not a commit hash or branch/tag name", url.url, path[2]);
-        } else
-            throw BadURL("GitHub URL '%s' is invalid", url.url);
-
-        for (auto &[name, value] : url.query) {
-            if (name == "rev") {
-                if (input->rev)
-                    throw BadURL("GitHub URL '%s' contains multiple commit hashes", url.url);
-                input->rev = Hash(value, htSHA1);
-            }
-            else if (name == "ref") {
-                if (!std::regex_match(value, refRegex))
-                    throw BadURL("GitHub URL '%s' contains an invalid branch/tag name", url.url);
-                if (input->ref)
-                    throw BadURL("GitHub URL '%s' contains multiple branch/tag names", url.url);
-                input->ref = value;
-            }
-        }
-
-        if (input->ref && input->rev)
-            throw BadURL("GitHub URL '%s' contains both a commit hash and a branch/tag name", url.url);
-
-        input->owner = path[0];
-        input->repo = path[1];
-
-        return input;
-    }
-
-    std::unique_ptr<Input> inputFromAttrs(const Attrs & attrs) override
-    {
-        if (maybeGetStrAttr(attrs, "type") != "github") return {};
-
-        for (auto & [name, value] : attrs)
-            if (name != "type" && name != "owner" && name != "repo" && name != "ref" && name != "rev")
-                throw Error("unsupported GitHub input attribute '%s'", name);
-
-        auto input = std::make_unique<GitHubInput>();
-        input->owner = getStrAttr(attrs, "owner");
-        input->repo = getStrAttr(attrs, "repo");
-        input->ref = maybeGetStrAttr(attrs, "ref");
-        if (auto rev = maybeGetStrAttr(attrs, "rev"))
-            input->rev = Hash(*rev, htSHA1);
-        return input;
-    }
-};
-
-static auto r1 = OnStartup([] { registerInputScheme(std::make_unique<GitHubInputScheme>()); });
-
-}

src/libfetchers/indirect.cc

@@ -1,140 +0,0 @@
-#include "fetchers.hh"
-
-namespace nix::fetchers {
-
-std::regex flakeRegex("[a-zA-Z][a-zA-Z0-9_-]*", std::regex::ECMAScript);
-
-struct IndirectInput : Input
-{
-    std::string id;
-    std::optional<Hash> rev;
-    std::optional<std::string> ref;
-
-    std::string type() const override { return "indirect"; }
-
-    bool operator ==(const Input & other) const override
-    {
-        auto other2 = dynamic_cast<const IndirectInput *>(&other);
-        return
-            other2
-            && id == other2->id
-            && rev == other2->rev
-            && ref == other2->ref;
-    }
-
-    bool isDirect() const override
-    {
-        return false;
-    }
-
-    std::optional<std::string> getRef() const override { return ref; }
-
-    std::optional<Hash> getRev() const override { return rev; }
-
-    bool contains(const Input & other) const override
-    {
-        auto other2 = dynamic_cast<const IndirectInput *>(&other);
-        return
-            other2
-            && id == other2->id
-            && (!ref || ref == other2->ref)
-            && (!rev || rev == other2->rev);
-    }
-
-    ParsedURL toURL() const override
-    {
-        ParsedURL url;
-        url.scheme = "flake";
-        url.path = id;
-        if (ref) { url.path += '/'; url.path += *ref; };
-        if (rev) { url.path += '/'; url.path += rev->gitRev(); };
-        return url;
-    }
-
-    Attrs toAttrsInternal() const override
-    {
-        Attrs attrs;
-        attrs.emplace("id", id);
-        if (ref)
-            attrs.emplace("ref", *ref);
-        if (rev)
-            attrs.emplace("rev", rev->gitRev());
-        return attrs;
-    }
-
-    std::shared_ptr<const Input> applyOverrides(
-        std::optional<std::string> ref,
-        std::optional<Hash> rev) const override
-    {
-        if (!ref && !rev) return shared_from_this();
-
-        auto res = std::make_shared<IndirectInput>(*this);
-
-        if (ref) res->ref = ref;
-        if (rev) res->rev = rev;
-
-        return res;
-    }
-
-    std::pair<Tree, std::shared_ptr<const Input>> fetchTreeInternal(nix::ref<Store> store) const override
-    {
-        throw Error("indirect input '%s' cannot be fetched directly", to_string());
-    }
-};
-
-struct IndirectInputScheme : InputScheme
-{
-    std::unique_ptr<Input> inputFromURL(const ParsedURL & url) override
-    {
-        if (url.scheme != "flake") return nullptr;
-
-        auto path = tokenizeString<std::vector<std::string>>(url.path, "/");
-        auto input = std::make_unique<IndirectInput>();
-
-        if (path.size() == 1) {
-        } else if (path.size() == 2) {
-            if (std::regex_match(path[1], revRegex))
-                input->rev = Hash(path[1], htSHA1);
-            else if (std::regex_match(path[1], refRegex))
-                input->ref = path[1];
-            else
-                throw BadURL("in flake URL '%s', '%s' is not a commit hash or branch/tag name", url.url, path[1]);
-        } else if (path.size() == 3) {
-            if (!std::regex_match(path[1], refRegex))
-                throw BadURL("in flake URL '%s', '%s' is not a branch/tag name", url.url, path[1]);
-            input->ref = path[1];
-            if (!std::regex_match(path[2], revRegex))
-                throw BadURL("in flake URL '%s', '%s' is not a commit hash", url.url, path[2]);
-            input->rev = Hash(path[2], htSHA1);
-        } else
-            throw BadURL("GitHub URL '%s' is invalid", url.url);
-
-        // FIXME: forbid query params?
-
-        input->id = path[0];
-        if (!std::regex_match(input->id, flakeRegex))
-            throw BadURL("'%s' is not a valid flake ID", input->id);
-
-        return input;
-    }
-
-    std::unique_ptr<Input> inputFromAttrs(const Attrs & attrs) override
-    {
-        if (maybeGetStrAttr(attrs, "type") != "indirect") return {};
-
-        for (auto & [name, value] : attrs)
-            if (name != "type" && name != "id" && name != "ref" && name != "rev")
-                throw Error("unsupported indirect input attribute '%s'", name);
-
-        auto input = std::make_unique<IndirectInput>();
-        input->id = getStrAttr(attrs, "id");
-        input->ref = maybeGetStrAttr(attrs, "ref");
-        if (auto rev = maybeGetStrAttr(attrs, "rev"))
-            input->rev = Hash(*rev, htSHA1);
-        return input;
-    }
-};
-
-static auto r1 = OnStartup([] { registerInputScheme(std::make_unique<IndirectInputScheme>()); });
-
-}

src/libfetchers/local.mk

@@ -1,11 +0,0 @@
-libraries += libfetchers
-
-libfetchers_NAME = libnixfetchers
-
-libfetchers_DIR := $(d)
-
-libfetchers_SOURCES := $(wildcard $(d)/*.cc)
-
-libfetchers_CXXFLAGS += -I src/libutil -I src/libstore
-
-libfetchers_LIBS = libutil libstore libnixrust

src/libfetchers/mercurial.cc

@@ -1,338 +0,0 @@
-#include "fetchers.hh"
-#include "cache.hh"
-#include "globals.hh"
-#include "tarfile.hh"
-#include "store-api.hh"
-
-#include <sys/time.h>
-
-using namespace std::string_literals;
-
-namespace nix::fetchers {
-
-struct MercurialInput : Input
-{
-    ParsedURL url;
-    std::optional<std::string> ref;
-    std::optional<Hash> rev;
-
-    MercurialInput(const ParsedURL & url) : url(url)
-    { }
-
-    std::string type() const override { return "hg"; }
-
-    bool operator ==(const Input & other) const override
-    {
-        auto other2 = dynamic_cast<const MercurialInput *>(&other);
-        return
-            other2
-            && url == other2->url
-            && rev == other2->rev
-            && ref == other2->ref;
-    }
-
-    bool isImmutable() const override
-    {
-        return (bool) rev || narHash;
-    }
-
-    std::optional<std::string> getRef() const override { return ref; }
-
-    std::optional<Hash> getRev() const override { return rev; }
-
-    ParsedURL toURL() const override
-    {
-        ParsedURL url2(url);
-        url2.scheme = "hg+" + url2.scheme;
-        if (rev) url2.query.insert_or_assign("rev", rev->gitRev());
-        if (ref) url2.query.insert_or_assign("ref", *ref);
-        return url;
-    }
-
-    Attrs toAttrsInternal() const override
-    {
-        Attrs attrs;
-        attrs.emplace("url", url.to_string());
-        if (ref)
-            attrs.emplace("ref", *ref);
-        if (rev)
-            attrs.emplace("rev", rev->gitRev());
-        return attrs;
-    }
-
-    std::shared_ptr<const Input> applyOverrides(
-        std::optional<std::string> ref,
-        std::optional<Hash> rev) const override
-    {
-        if (!ref && !rev) return shared_from_this();
-
-        auto res = std::make_shared<MercurialInput>(*this);
-
-        if (ref) res->ref = ref;
-        if (rev) res->rev = rev;
-
-        return res;
-    }
-
-    std::optional<Path> getSourcePath() const
-    {
-        if (url.scheme == "file" && !ref && !rev)
-            return url.path;
-        return {};
-    }
-
-    void markChangedFile(std::string_view file, std::optional<std::string> commitMsg) const override
-    {
-        auto sourcePath = getSourcePath();
-        assert(sourcePath);
-
-        // FIXME: shut up if file is already tracked.
-        runProgram("hg", true,
-            { "add", *sourcePath + "/" + std::string(file) });
-
-        if (commitMsg)
-            runProgram("hg", true,
-                { "commit", *sourcePath + "/" + std::string(file), "-m", *commitMsg });
-    }
-
-    std::pair<bool, std::string> getActualUrl() const
-    {
-        bool isLocal = url.scheme == "file";
-        return {isLocal, isLocal ? url.path : url.base};
-    }
-
-    std::pair<Tree, std::shared_ptr<const Input>> fetchTreeInternal(nix::ref<Store> store) const override
-    {
-        auto name = "source";
-
-        auto input = std::make_shared<MercurialInput>(*this);
-
-        auto [isLocal, actualUrl_] = getActualUrl();
-        auto actualUrl = actualUrl_; // work around clang bug
-
-        // FIXME: return lastModified.
-
-        // FIXME: don't clone local repositories.
-
-        if (!input->ref && !input->rev && isLocal && pathExists(actualUrl + "/.hg")) {
-
-            bool clean = runProgram("hg", true, { "status", "-R", actualUrl, "--modified", "--added", "--removed" }) == "";
-
-            if (!clean) {
-
-                /* This is an unclean working tree. So copy all tracked
-                   files. */
-
-                if (!settings.allowDirty)
-                    throw Error("Mercurial tree '%s' is unclean", actualUrl);
-
-                if (settings.warnDirty)
-                    warn("Mercurial tree '%s' is unclean", actualUrl);
-
-                input->ref = chomp(runProgram("hg", true, { "branch", "-R", actualUrl }));
-
-                auto files = tokenizeString<std::set<std::string>>(
-                    runProgram("hg", true, { "status", "-R", actualUrl, "--clean", "--modified", "--added", "--no-status", "--print0" }), "\0"s);
-
-                PathFilter filter = [&](const Path & p) -> bool {
-                    assert(hasPrefix(p, actualUrl));
-                    std::string file(p, actualUrl.size() + 1);
-
-                    auto st = lstat(p);
-
-                    if (S_ISDIR(st.st_mode)) {
-                        auto prefix = file + "/";
-                        auto i = files.lower_bound(prefix);
-                        return i != files.end() && hasPrefix(*i, prefix);
-                    }
-
-                    return files.count(file);
-                };
-
-                auto storePath = store->addToStore("source", actualUrl, true, htSHA256, filter);
-
-                return {Tree {
-                    .actualPath = store->printStorePath(storePath),
-                    .storePath = std::move(storePath),
-                }, input};
-            }
-        }
-
-        if (!input->ref) input->ref = "default";
-
-        auto getImmutableAttrs = [&]()
-        {
-            return Attrs({
-                {"type", "hg"},
-                {"name", name},
-                {"rev", input->rev->gitRev()},
-            });
-        };
-
-        auto makeResult = [&](const Attrs & infoAttrs, StorePath && storePath)
-            -> std::pair<Tree, std::shared_ptr<const Input>>
-        {
-            assert(input->rev);
-            assert(!rev || rev == input->rev);
-            return {
-                Tree{
-                    .actualPath = store->toRealPath(storePath),
-                    .storePath = std::move(storePath),
-                    .info = TreeInfo {
-                        .revCount = getIntAttr(infoAttrs, "revCount"),
-                    },
-                },
-                input
-            };
-        };
-
-        if (input->rev) {
-            if (auto res = getCache()->lookup(store, getImmutableAttrs()))
-                return makeResult(res->first, std::move(res->second));
-        }
-
-        assert(input->rev || input->ref);
-        auto revOrRef = input->rev ? input->rev->gitRev() : *input->ref;
-
-        Attrs mutableAttrs({
-            {"type", "hg"},
-            {"name", name},
-            {"url", actualUrl},
-            {"ref", *input->ref},
-        });
-
-        if (auto res = getCache()->lookup(store, mutableAttrs)) {
-            auto rev2 = Hash(getStrAttr(res->first, "rev"), htSHA1);
-            if (!rev || rev == rev2) {
-                input->rev = rev2;
-                return makeResult(res->first, std::move(res->second));
-            }
-        }
-
-        Path cacheDir = fmt("%s/nix/hg/%s", getCacheDir(), hashString(htSHA256, actualUrl).to_string(Base32, false));
-
-        /* If this is a commit hash that we already have, we don't
-           have to pull again. */
-        if (!(input->rev
-                && pathExists(cacheDir)
-                && runProgram(
-                    RunOptions("hg", { "log", "-R", cacheDir, "-r", input->rev->gitRev(), "--template", "1" })
-                    .killStderr(true)).second == "1"))
-        {
-            Activity act(*logger, lvlTalkative, actUnknown, fmt("fetching Mercurial repository '%s'", actualUrl));
-
-            if (pathExists(cacheDir)) {
-                try {
-                    runProgram("hg", true, { "pull", "-R", cacheDir, "--", actualUrl });
-                }
-                catch (ExecError & e) {
-                    string transJournal = cacheDir + "/.hg/store/journal";
-                    /* hg throws "abandoned transaction" error only if this file exists */
-                    if (pathExists(transJournal)) {
-                        runProgram("hg", true, { "recover", "-R", cacheDir });
-                        runProgram("hg", true, { "pull", "-R", cacheDir, "--", actualUrl });
-                    } else {
-                        throw ExecError(e.status, fmt("'hg pull' %s", statusToString(e.status)));
-                    }
-                }
-            } else {
-                createDirs(dirOf(cacheDir));
-                runProgram("hg", true, { "clone", "--noupdate", "--", actualUrl, cacheDir });
-            }
-        }
-
-        auto tokens = tokenizeString<std::vector<std::string>>(
-            runProgram("hg", true, { "log", "-R", cacheDir, "-r", revOrRef, "--template", "{node} {rev} {branch}" }));
-        assert(tokens.size() == 3);
-
-        input->rev = Hash(tokens[0], htSHA1);
-        auto revCount = std::stoull(tokens[1]);
-        input->ref = tokens[2];
-
-        if (auto res = getCache()->lookup(store, getImmutableAttrs()))
-            return makeResult(res->first, std::move(res->second));
-
-        Path tmpDir = createTempDir();
-        AutoDelete delTmpDir(tmpDir, true);
-
-        runProgram("hg", true, { "archive", "-R", cacheDir, "-r", input->rev->gitRev(), tmpDir });
-
-        deletePath(tmpDir + "/.hg_archival.txt");
-
-        auto storePath = store->addToStore(name, tmpDir);
-
-        Attrs infoAttrs({
-            {"rev", input->rev->gitRev()},
-            {"revCount", (int64_t) revCount},
-        });
-
-        if (!this->rev)
-            getCache()->add(
-                store,
-                mutableAttrs,
-                infoAttrs,
-                storePath,
-                false);
-
-        getCache()->add(
-            store,
-            getImmutableAttrs(),
-            infoAttrs,
-            storePath,
-            true);
-
-        return makeResult(infoAttrs, std::move(storePath));
-    }
-};
-
-struct MercurialInputScheme : InputScheme
-{
-    std::unique_ptr<Input> inputFromURL(const ParsedURL & url) override
-    {
-        if (url.scheme != "hg+http" &&
-            url.scheme != "hg+https" &&
-            url.scheme != "hg+ssh" &&
-            url.scheme != "hg+file") return nullptr;
-
-        auto url2(url);
-        url2.scheme = std::string(url2.scheme, 3);
-        url2.query.clear();
-
-        Attrs attrs;
-        attrs.emplace("type", "hg");
-
-        for (auto &[name, value] : url.query) {
-            if (name == "rev" || name == "ref")
-                attrs.emplace(name, value);
-            else
-                url2.query.emplace(name, value);
-        }
-
-        attrs.emplace("url", url2.to_string());
-
-        return inputFromAttrs(attrs);
-    }
-
-    std::unique_ptr<Input> inputFromAttrs(const Attrs & attrs) override
-    {
-        if (maybeGetStrAttr(attrs, "type") != "hg") return {};
-
-        for (auto & [name, value] : attrs)
-            if (name != "type" && name != "url" && name != "ref" && name != "rev")
-                throw Error("unsupported Mercurial input attribute '%s'", name);
-
-        auto input = std::make_unique<MercurialInput>(parseURL(getStrAttr(attrs, "url")));
-        if (auto ref = maybeGetStrAttr(attrs, "ref")) {
-            if (!std::regex_match(*ref, refRegex))
-                throw BadURL("invalid Mercurial branch/tag name '%s'", *ref);
-            input->ref = *ref;
-        }
-        if (auto rev = maybeGetStrAttr(attrs, "rev"))
-            input->rev = Hash(*rev, htSHA1);
-        return input;
-    }
-};
-
-static auto r1 = OnStartup([] { registerInputScheme(std::make_unique<MercurialInputScheme>()); });
-
-}

src/libfetchers/path.cc

@@ -1,170 +0,0 @@
-#include "fetchers.hh"
-#include "store-api.hh"
-
-namespace nix::fetchers {
-
-struct PathInput : Input
-{
-    Path path;
-
-    /* Allow the user to pass in "fake" tree info attributes. This is
-       useful for making a pinned tree work the same as the repository
-       from which is exported
-       (e.g. path:/nix/store/...-source?lastModified=1585388205&rev=b0c285...). */
-    std::optional<Hash> rev;
-    std::optional<uint64_t> revCount;
-    std::optional<time_t> lastModified;
-
-    std::string type() const override { return "path"; }
-
-    std::optional<Hash> getRev() const override { return rev; }
-
-    bool operator ==(const Input & other) const override
-    {
-        auto other2 = dynamic_cast<const PathInput *>(&other);
-        return
-            other2
-            && path == other2->path
-            && rev == other2->rev
-            && revCount == other2->revCount
-            && lastModified == other2->lastModified;
-    }
-
-    bool isImmutable() const override
-    {
-        return narHash || rev;
-    }
-
-    ParsedURL toURL() const override
-    {
-        auto query = attrsToQuery(toAttrsInternal());
-        query.erase("path");
-        return ParsedURL {
-            .scheme = "path",
-            .path = path,
-            .query = query,
-        };
-    }
-
-    Attrs toAttrsInternal() const override
-    {
-        Attrs attrs;
-        attrs.emplace("path", path);
-        if (rev)
-            attrs.emplace("rev", rev->gitRev());
-        if (revCount)
-            attrs.emplace("revCount", *revCount);
-        if (lastModified)
-            attrs.emplace("lastModified", *lastModified);
-        if (!rev && narHash)
-            attrs.emplace("narHash", narHash->to_string(SRI));
-        return attrs;
-    }
-
-    std::optional<Path> getSourcePath() const override
-    {
-        return path;
-    }
-
-    void markChangedFile(std::string_view file, std::optional<std::string> commitMsg) const override
-    {
-    }
-
-    std::pair<Tree, std::shared_ptr<const Input>> fetchTreeInternal(nix::ref<Store> store) const override
-    {
-        auto input = std::make_shared<PathInput>(*this);
-
-        // FIXME: check whether access to 'path' is allowed.
-
-        auto storePath = store->maybeParseStorePath(path);
-
-        if (storePath)
-            store->addTempRoot(*storePath);
-
-        if (!storePath || storePath->name() != "source" || !store->isValidPath(*storePath))
-            // FIXME: try to substitute storePath.
-            storePath = store->addToStore("source", path);
-
-        input->narHash = store->queryPathInfo(*storePath)->narHash;
-
-        return
-            {
-                Tree {
-                    .actualPath = store->toRealPath(*storePath),
-                    .storePath = std::move(*storePath),
-                    .info = TreeInfo {
-                        .revCount = revCount,
-                        .lastModified = lastModified
-                    }
-                },
-                input
-            };
-    }
-
-};
-
-struct PathInputScheme : InputScheme
-{
-    std::unique_ptr<Input> inputFromURL(const ParsedURL & url) override
-    {
-        if (url.scheme != "path") return nullptr;
-
-        auto input = std::make_unique<PathInput>();
-        input->path = url.path;
-
-        if (url.authority && *url.authority != "")
-            throw Error("path URL '%s' should not have an authority ('%s')", url.url, *url.authority);
-
-        for (auto & [name, value] : url.query)
-            if (name == "rev")
-                input->rev = Hash(value, htSHA1);
-            else if (name == "revCount") {
-                uint64_t revCount;
-                if (!string2Int(value, revCount))
-                    throw Error("path URL '%s' has invalid parameter '%s'", url.to_string(), name);
-                input->revCount = revCount;
-            }
-            else if (name == "lastModified") {
-                time_t lastModified;
-                if (!string2Int(value, lastModified))
-                    throw Error("path URL '%s' has invalid parameter '%s'", url.to_string(), name);
-                input->lastModified = lastModified;
-            }
-            else if (name == "narHash")
-                // FIXME: require SRI hash.
-                input->narHash = Hash(value);
-            else
-                throw Error("path URL '%s' has unsupported parameter '%s'", url.to_string(), name);
-
-        return input;
-    }
-
-    std::unique_ptr<Input> inputFromAttrs(const Attrs & attrs) override
-    {
-        if (maybeGetStrAttr(attrs, "type") != "path") return {};
-
-        auto input = std::make_unique<PathInput>();
-        input->path = getStrAttr(attrs, "path");
-
-        for (auto & [name, value] : attrs)
-            if (name == "rev")
-                input->rev = Hash(getStrAttr(attrs, "rev"), htSHA1);
-            else if (name == "revCount")
-                input->revCount = getIntAttr(attrs, "revCount");
-            else if (name == "lastModified")
-                input->lastModified = getIntAttr(attrs, "lastModified");
-            else if (name == "narHash")
-                // FIXME: require SRI hash.
-                input->narHash = Hash(getStrAttr(attrs, "narHash"));
-            else if (name == "type" || name == "path")
-                ;
-            else
-                throw Error("unsupported path input attribute '%s'", name);
-
-        return input;
-    }
-};
-
-static auto r1 = OnStartup([] { registerInputScheme(std::make_unique<PathInputScheme>()); });
-
-}

src/libfetchers/registry.cc

@@ -1,222 +0,0 @@
-#include "registry.hh"
-#include "fetchers.hh"
-#include "util.hh"
-#include "globals.hh"
-#include "store-api.hh"
-
-#include <nlohmann/json.hpp>
-
-namespace nix::fetchers {
-
-std::shared_ptr<Registry> Registry::read(
-    const Path & path, RegistryType type)
-{
-    auto registry = std::make_shared<Registry>(type);
-
-    if (!pathExists(path))
-        return std::make_shared<Registry>(type);
-
-    try {
-
-        auto json = nlohmann::json::parse(readFile(path));
-
-        auto version = json.value("version", 0);
-
-        // FIXME: remove soon
-        if (version == 1) {
-            auto flakes = json["flakes"];
-            for (auto i = flakes.begin(); i != flakes.end(); ++i) {
-                auto url = i->value("url", i->value("uri", ""));
-                if (url.empty())
-                    throw Error("flake registry '%s' lacks a 'url' attribute for entry '%s'",
-                        path, i.key());
-                registry->entries.push_back(
-                    {inputFromURL(i.key()), inputFromURL(url), {}});
-            }
-        }
-
-        else if (version == 2) {
-            for (auto & i : json["flakes"]) {
-                auto toAttrs = jsonToAttrs(i["to"]);
-                Attrs extraAttrs;
-                auto j = toAttrs.find("dir");
-                if (j != toAttrs.end()) {
-                    extraAttrs.insert(*j);
-                    toAttrs.erase(j);
-                }
-                auto exact = i.find("exact");
-                registry->entries.push_back(
-                    Entry {
-                        .from = inputFromAttrs(jsonToAttrs(i["from"])),
-                        .to = inputFromAttrs(toAttrs),
-                        .extraAttrs = extraAttrs,
-                        .exact = exact != i.end() && exact.value()
-                    });
-            }
-        }
-
-        else
-            throw Error("flake registry '%s' has unsupported version %d", path, version);
-
-    } catch (nlohmann::json::exception & e) {
-        warn("cannot parse flake registry '%s': %s", path, e.what());
-    } catch (Error & e) {
-        warn("cannot read flake registry '%s': %s", path, e.what());
-    }
-
-    return registry;
-}
-
-void Registry::write(const Path & path)
-{
-    nlohmann::json arr;
-    for (auto & entry : entries) {
-        nlohmann::json obj;
-        obj["from"] = attrsToJson(entry.from->toAttrs());
-        obj["to"] = attrsToJson(entry.to->toAttrs());
-        if (!entry.extraAttrs.empty())
-            obj["to"].update(attrsToJson(entry.extraAttrs));
-        if (entry.exact)
-            obj["exact"] = true;
-        arr.emplace_back(std::move(obj));
-    }
-
-    nlohmann::json json;
-    json["version"] = 2;
-    json["flakes"] = std::move(arr);
-
-    createDirs(dirOf(path));
-    writeFile(path, json.dump(2));
-}
-
-void Registry::add(
-    const std::shared_ptr<const Input> & from,
-    const std::shared_ptr<const Input> & to,
-    const Attrs & extraAttrs)
-{
-    entries.emplace_back(
-        Entry {
-            .from = from,
-            .to = to,
-            .extraAttrs = extraAttrs
-        });
-}
-
-void Registry::remove(const std::shared_ptr<const Input> & input)
-{
-    // FIXME: use C++20 std::erase.
-    for (auto i = entries.begin(); i != entries.end(); )
-        if (*i->from == *input)
-            i = entries.erase(i);
-        else
-            ++i;
-}
-
-static Path getSystemRegistryPath()
-{
-    return settings.nixConfDir + "/registry.json";
-}
-
-static std::shared_ptr<Registry> getSystemRegistry()
-{
-    static auto systemRegistry =
-        Registry::read(getSystemRegistryPath(), Registry::System);
-    return systemRegistry;
-}
-
-Path getUserRegistryPath()
-{
-    return getHome() + "/.config/nix/registry.json";
-}
-
-std::shared_ptr<Registry> getUserRegistry()
-{
-    static auto userRegistry =
-        Registry::read(getUserRegistryPath(), Registry::User);
-    return userRegistry;
-}
-
-static std::shared_ptr<Registry> flagRegistry =
-    std::make_shared<Registry>(Registry::Flag);
-
-std::shared_ptr<Registry> getFlagRegistry()
-{
-    return flagRegistry;
-}
-
-void overrideRegistry(
-    const std::shared_ptr<const Input> & from,
-    const std::shared_ptr<const Input> & to,
-    const Attrs & extraAttrs)
-{
-    flagRegistry->add(from, to, extraAttrs);
-}
-
-static std::shared_ptr<Registry> getGlobalRegistry(ref<Store> store)
-{
-    static auto reg = [&]() {
-        auto path = settings.flakeRegistry.get();
-
-        if (!hasPrefix(path, "/")) {
-            auto storePath = downloadFile(store, path, "flake-registry.json", false).storePath;
-            if (auto store2 = store.dynamic_pointer_cast<LocalFSStore>())
-                store2->addPermRoot(storePath, getCacheDir() + "/nix/flake-registry.json", true);
-            path = store->toRealPath(storePath);
-        }
-
-        return Registry::read(path, Registry::Global);
-    }();
-
-    return reg;
-}
-
-Registries getRegistries(ref<Store> store)
-{
-    Registries registries;
-    registries.push_back(getFlagRegistry());
-    registries.push_back(getUserRegistry());
-    registries.push_back(getSystemRegistry());
-    registries.push_back(getGlobalRegistry(store));
-    return registries;
-}
-
-std::pair<std::shared_ptr<const Input>, Attrs> lookupInRegistries(
-    ref<Store> store,
-    std::shared_ptr<const Input> input)
-{
-    Attrs extraAttrs;
-    int n = 0;
-
- restart:
-
-    n++;
-    if (n > 100) throw Error("cycle detected in flake registr for '%s'", input);
-
-    for (auto & registry : getRegistries(store)) {
-        // FIXME: O(n)
-        for (auto & entry : registry->entries) {
-            if (entry.exact) {
-                if (*entry.from == *input) {
-                    input = entry.to;
-                    extraAttrs = entry.extraAttrs;
-                    goto restart;
-                }
-            } else {
-                if (entry.from->contains(*input)) {
-                    input = entry.to->applyOverrides(
-                        !entry.from->getRef() && input->getRef() ? input->getRef() : std::optional<std::string>(),
-                        !entry.from->getRev() && input->getRev() ? input->getRev() : std::optional<Hash>());
-                    extraAttrs = entry.extraAttrs;
-                    goto restart;
-                }
-            }
-        }
-    }
-
-    if (!input->isDirect())
-        throw Error("cannot find flake '%s' in the flake registries", input->to_string());
-
-    return {input, extraAttrs};
-}
-
-}

src/libfetchers/registry.hh

@@ -1,65 +0,0 @@
-#pragma once
-
-#include "types.hh"
-#include "fetchers.hh"
-
-namespace nix { class Store; }
-
-namespace nix::fetchers {
-
-struct Registry
-{
-    enum RegistryType {
-        Flag = 0,
-        User = 1,
-        System = 2,
-        Global = 3,
-    };
-
-    RegistryType type;
-
-    struct Entry
-    {
-        std::shared_ptr<const Input> from;
-        std::shared_ptr<const Input> to;
-        Attrs extraAttrs;
-        bool exact = false;
-    };
-
-    std::vector<Entry> entries;
-
-    Registry(RegistryType type)
-        : type(type)
-    { }
-
-    static std::shared_ptr<Registry> read(
-        const Path & path, RegistryType type);
-
-    void write(const Path & path);
-
-    void add(
-        const std::shared_ptr<const Input> & from,
-        const std::shared_ptr<const Input> & to,
-        const Attrs & extraAttrs);
-
-    void remove(const std::shared_ptr<const Input> & input);
-};
-
-typedef std::vector<std::shared_ptr<Registry>> Registries;
-
-std::shared_ptr<Registry> getUserRegistry();
-
-Path getUserRegistryPath();
-
-Registries getRegistries(ref<Store> store);
-
-void overrideRegistry(
-    const std::shared_ptr<const Input> & from,
-    const std::shared_ptr<const Input> & to,
-    const Attrs & extraAttrs);
-
-std::pair<std::shared_ptr<const Input>, Attrs> lookupInRegistries(
-    ref<Store> store,
-    std::shared_ptr<const Input> input);
-
-}

src/libfetchers/tarball.cc

@@ -1,275 +0,0 @@
-#include "fetchers.hh"
-#include "cache.hh"
-#include "filetransfer.hh"
-#include "globals.hh"
-#include "store-api.hh"
-#include "archive.hh"
-#include "tarfile.hh"
-
-namespace nix::fetchers {
-
-DownloadFileResult downloadFile(
-    ref<Store> store,
-    const std::string & url,
-    const std::string & name,
-    bool immutable)
-{
-    // FIXME: check store
-
-    Attrs inAttrs({
-        {"type", "file"},
-        {"url", url},
-        {"name", name},
-    });
-
-    auto cached = getCache()->lookupExpired(store, inAttrs);
-
-    auto useCached = [&]() -> DownloadFileResult
-    {
-        return {
-            .storePath = std::move(cached->storePath),
-            .etag = getStrAttr(cached->infoAttrs, "etag"),
-            .effectiveUrl = getStrAttr(cached->infoAttrs, "url")
-        };
-    };
-
-    if (cached && !cached->expired)
-        return useCached();
-
-    FileTransferRequest request(url);
-    if (cached)
-        request.expectedETag = getStrAttr(cached->infoAttrs, "etag");
-    FileTransferResult res;
-    try {
-        res = getFileTransfer()->download(request);
-    } catch (FileTransferError & e) {
-        if (cached) {
-            warn("%s; using cached version", e.msg());
-            return useCached();
-        } else
-            throw;
-    }
-
-    // FIXME: write to temporary file.
-
-    Attrs infoAttrs({
-        {"etag", res.etag},
-        {"url", res.effectiveUri},
-    });
-
-    std::optional<StorePath> storePath;
-
-    if (res.cached) {
-        assert(cached);
-        assert(request.expectedETag == res.etag);
-        storePath = std::move(cached->storePath);
-    } else {
-        StringSink sink;
-        dumpString(*res.data, sink);
-        auto hash = hashString(htSHA256, *res.data);
-        ValidPathInfo info(store->makeFixedOutputPath(false, hash, name));
-        info.narHash = hashString(htSHA256, *sink.s);
-        info.narSize = sink.s->size();
-        info.ca = makeFixedOutputCA(false, hash);
-        store->addToStore(info, sink.s, NoRepair, NoCheckSigs);
-        storePath = std::move(info.path);
-    }
-
-    getCache()->add(
-        store,
-        inAttrs,
-        infoAttrs,
-        *storePath,
-        immutable);
-
-    if (url != res.effectiveUri)
-        getCache()->add(
-            store,
-            {
-                {"type", "file"},
-                {"url", res.effectiveUri},
-                {"name", name},
-            },
-            infoAttrs,
-            *storePath,
-            immutable);
-
-    return {
-        .storePath = std::move(*storePath),
-        .etag = res.etag,
-        .effectiveUrl = res.effectiveUri,
-    };
-}
-
-Tree downloadTarball(
-    ref<Store> store,
-    const std::string & url,
-    const std::string & name,
-    bool immutable)
-{
-    Attrs inAttrs({
-        {"type", "tarball"},
-        {"url", url},
-        {"name", name},
-    });
-
-    auto cached = getCache()->lookupExpired(store, inAttrs);
-
-    if (cached && !cached->expired)
-        return Tree {
-            .actualPath = store->toRealPath(cached->storePath),
-            .storePath = std::move(cached->storePath),
-            .info = TreeInfo {
-                .lastModified = getIntAttr(cached->infoAttrs, "lastModified"),
-            },
-        };
-
-    auto res = downloadFile(store, url, name, immutable);
-
-    std::optional<StorePath> unpackedStorePath;
-    time_t lastModified;
-
-    if (cached && res.etag != "" && getStrAttr(cached->infoAttrs, "etag") == res.etag) {
-        unpackedStorePath = std::move(cached->storePath);
-        lastModified = getIntAttr(cached->infoAttrs, "lastModified");
-    } else {
-        Path tmpDir = createTempDir();
-        AutoDelete autoDelete(tmpDir, true);
-        unpackTarfile(store->toRealPath(res.storePath), tmpDir);
-        auto members = readDirectory(tmpDir);
-        if (members.size() != 1)
-            throw nix::Error("tarball '%s' contains an unexpected number of top-level files", url);
-        auto topDir = tmpDir + "/" + members.begin()->name;
-        lastModified = lstat(topDir).st_mtime;
-        unpackedStorePath = store->addToStore(name, topDir, true, htSHA256, defaultPathFilter, NoRepair);
-    }
-
-    Attrs infoAttrs({
-        {"lastModified", lastModified},
-        {"etag", res.etag},
-    });
-
-    getCache()->add(
-        store,
-        inAttrs,
-        infoAttrs,
-        *unpackedStorePath,
-        immutable);
-
-    return Tree {
-        .actualPath = store->toRealPath(*unpackedStorePath),
-        .storePath = std::move(*unpackedStorePath),
-        .info = TreeInfo {
-            .lastModified = lastModified,
-        },
-    };
-}
-
-struct TarballInput : Input
-{
-    ParsedURL url;
-    std::optional<Hash> hash;
-
-    TarballInput(const ParsedURL & url) : url(url)
-    { }
-
-    std::string type() const override { return "tarball"; }
-
-    bool operator ==(const Input & other) const override
-    {
-        auto other2 = dynamic_cast<const TarballInput *>(&other);
-        return
-            other2
-            && to_string() == other2->to_string()
-            && hash == other2->hash;
-    }
-
-    bool isImmutable() const override
-    {
-        return hash || narHash;
-    }
-
-    ParsedURL toURL() const override
-    {
-        auto url2(url);
-        // NAR hashes are preferred over file hashes since tar/zip files
-        // don't have a canonical representation.
-        if (narHash)
-            url2.query.insert_or_assign("narHash", narHash->to_string(SRI));
-        else if (hash)
-            url2.query.insert_or_assign("hash", hash->to_string(SRI));
-        return url2;
-    }
-
-    Attrs toAttrsInternal() const override
-    {
-        Attrs attrs;
-        attrs.emplace("url", url.to_string());
-        if (hash)
-            attrs.emplace("hash", hash->to_string(SRI));
-        return attrs;
-    }
-
-    std::pair<Tree, std::shared_ptr<const Input>> fetchTreeInternal(nix::ref<Store> store) const override
-    {
-        auto tree = downloadTarball(store, url.to_string(), "source", false);
-
-        auto input = std::make_shared<TarballInput>(*this);
-        input->narHash = store->queryPathInfo(tree.storePath)->narHash;
-
-        return {std::move(tree), input};
-    }
-};
-
-struct TarballInputScheme : InputScheme
-{
-    std::unique_ptr<Input> inputFromURL(const ParsedURL & url) override
-    {
-        if (url.scheme != "file" && url.scheme != "http" && url.scheme != "https") return nullptr;
-
-        if (!hasSuffix(url.path, ".zip")
-            && !hasSuffix(url.path, ".tar")
-            && !hasSuffix(url.path, ".tar.gz")
-            && !hasSuffix(url.path, ".tar.xz")
-            && !hasSuffix(url.path, ".tar.bz2"))
-            return nullptr;
-
-        auto input = std::make_unique<TarballInput>(url);
-
-        auto hash = input->url.query.find("hash");
-        if (hash != input->url.query.end()) {
-            // FIXME: require SRI hash.
-            input->hash = Hash(hash->second);
-            input->url.query.erase(hash);
-        }
-
-        auto narHash = input->url.query.find("narHash");
-        if (narHash != input->url.query.end()) {
-            // FIXME: require SRI hash.
-            input->narHash = Hash(narHash->second);
-            input->url.query.erase(narHash);
-        }
-
-        return input;
-    }
-
-    std::unique_ptr<Input> inputFromAttrs(const Attrs & attrs) override
-    {
-        if (maybeGetStrAttr(attrs, "type") != "tarball") return {};
-
-        for (auto & [name, value] : attrs)
-            if (name != "type" && name != "url" && name != "hash")
-                throw Error("unsupported tarball input attribute '%s'", name);
-
-        auto input = std::make_unique<TarballInput>(parseURL(getStrAttr(attrs, "url")));
-        if (auto hash = maybeGetStrAttr(attrs, "hash"))
-            // FIXME: require SRI hash.
-            input->hash = Hash(*hash);
-
-        return input;
-    }
-};
-
-static auto r1 = OnStartup([] { registerInputScheme(std::make_unique<TarballInputScheme>()); });
-
-}

src/libfetchers/tree-info.cc

@@ -1,60 +0,0 @@
-#include "tree-info.hh"
-#include "store-api.hh"
-
-#include <nlohmann/json.hpp>
-
-namespace nix::fetchers {
-
-StorePath TreeInfo::computeStorePath(Store & store) const
-{
-    assert(narHash);
-    return store.makeFixedOutputPath(true, narHash, "source");
-}
-
-TreeInfo TreeInfo::fromJson(const nlohmann::json & json)
-{
-    TreeInfo info;
-
-    auto i = json.find("info");
-    if (i != json.end()) {
-        const nlohmann::json & i2(*i);
-
-        auto j = i2.find("narHash");
-        if (j != i2.end())
-            info.narHash = Hash((std::string) *j);
-        else
-            throw Error("attribute 'narHash' missing in lock file");
-
-        j = i2.find("revCount");
-        if (j != i2.end())
-            info.revCount = *j;
-
-        j = i2.find("lastModified");
-        if (j != i2.end())
-            info.lastModified = *j;
-
-        return info;
-    }
-
-    i = json.find("narHash");
-    if (i != json.end()) {
-        info.narHash = Hash((std::string) *i);
-        return info;
-    }
-
-    throw Error("attribute 'info' missing in lock file");
-}
-
-nlohmann::json TreeInfo::toJson() const
-{
-    nlohmann::json json;
-    assert(narHash);
-    json["narHash"] = narHash.to_string(SRI);
-    if (revCount)
-        json["revCount"] = *revCount;
-    if (lastModified)
-        json["lastModified"] = *lastModified;
-    return json;
-}
-
-}

src/libfetchers/tree-info.hh

@@ -1,33 +0,0 @@
-#pragma once
-
-#include "path.hh"
-#include "hash.hh"
-
-#include <nlohmann/json_fwd.hpp>
-
-namespace nix { class Store; }
-
-namespace nix::fetchers {
-
-struct TreeInfo
-{
-    Hash narHash;
-    std::optional<uint64_t> revCount;
-    std::optional<time_t> lastModified;
-
-    bool operator ==(const TreeInfo & other) const
-    {
-        return
-            narHash == other.narHash
-            && revCount == other.revCount
-            && lastModified == other.lastModified;
-    }
-
-    StorePath computeStorePath(Store & store) const;
-
-    static TreeInfo fromJson(const nlohmann::json & json);
-
-    nlohmann::json toJson() const;
-};
-
-}

src/libmain/common-args.cc

@@ -6,57 +6,43 @@ namespace nix {
 MixCommonArgs::MixCommonArgs(const string & programName)
     : programName(programName)
 {
-    addFlag({
-        .longName = "verbose",
-        .shortName = 'v',
-        .description = "increase verbosity level",
-        .handler = {[]() { verbosity = (Verbosity) (verbosity + 1); }},
-    });
+    mkFlag()
+        .longName("verbose")
+        .shortName('v')
+        .description("increase verbosity level")
+        .handler([]() { verbosity = (Verbosity) (verbosity + 1); });
 
-    addFlag({
-        .longName = "quiet",
-        .description = "decrease verbosity level",
-        .handler = {[]() { verbosity = verbosity > lvlError ? (Verbosity) (verbosity - 1) : lvlError; }},
-    });
+    mkFlag()
+        .longName("quiet")
+        .description("decrease verbosity level")
+        .handler([]() { verbosity = verbosity > lvlError ? (Verbosity) (verbosity - 1) : lvlError; });
 
-    addFlag({
-        .longName = "debug",
-        .description = "enable debug output",
-        .handler = {[]() { verbosity = lvlDebug; }},
-    });
+    mkFlag()
+        .longName("debug")
+        .description("enable debug output")
+        .handler([]() { verbosity = lvlDebug; });
 
-    addFlag({
-        .longName = "option",
-        .description = "set a Nix configuration option (overriding nix.conf)",
-        .labels = {"name", "value"},
-        .handler = {[](std::string name, std::string value) {
+    mkFlag()
+        .longName("option")
+        .labels({"name", "value"})
+        .description("set a Nix configuration option (overriding nix.conf)")
+        .arity(2)
+        .handler([](std::vector<std::string> ss) {
             try {
-                globalConfig.set(name, value);
+                globalConfig.set(ss[0], ss[1]);
             } catch (UsageError & e) {
-                if (!completions)
-                    warn(e.what());
+                warn(e.what());
             }
-        }},
-        .completer = [](size_t index, std::string_view prefix) {
-            if (index == 0) {
-                std::map<std::string, Config::SettingInfo> settings;
-                globalConfig.getSettings(settings);
-                for (auto & s : settings)
-                    if (hasPrefix(s.first, prefix))
-                        completions->insert(s.first);
-            }
-        }
-    });
+        });
 
-    addFlag({
-        .longName = "max-jobs",
-        .shortName = 'j',
-        .description = "maximum number of parallel builds",
-        .labels = Strings{"jobs"},
-        .handler = {[=](std::string s) {
+    mkFlag()
+        .longName("max-jobs")
+        .shortName('j')
+        .label("jobs")
+        .description("maximum number of parallel builds")
+        .handler([=](std::string s) {
             settings.set("max-jobs", s);
-        }}
-    });
+        });
 
     std::string cat = "config";
     globalConfig.convertToArgs(*this, cat);

src/libmain/shared.cc

@@ -155,7 +155,7 @@ void initNix()
        sshd). This breaks build users because they don't have access
        to the TMPDIR, in particular in ‘nix-store --serve’. */
 #if __APPLE__
-    if (hasPrefix(getEnv("TMPDIR").value_or("/tmp"), "/var/folders/"))
+    if (getuid() == 0 && hasPrefix(getEnv("TMPDIR").value_or("/tmp"), "/var/folders/"))
         unsetenv("TMPDIR");
 #endif
 }
@@ -165,32 +165,28 @@ LegacyArgs::LegacyArgs(const std::string & programName,
     std::function<bool(Strings::iterator & arg, const Strings::iterator & end)> parseArg)
     : MixCommonArgs(programName), parseArg(parseArg)
 {
-    addFlag({
-        .longName = "no-build-output",
-        .shortName = 'Q',
-        .description = "do not show build output",
-        .handler = {&settings.verboseBuild, false},
-    });
+    mkFlag()
+        .longName("no-build-output")
+        .shortName('Q')
+        .description("do not show build output")
+        .set(&settings.verboseBuild, false);
 
-    addFlag({
-        .longName = "keep-failed",
-        .shortName ='K',
-        .description = "keep temporary directories of failed builds",
-        .handler = {&(bool&) settings.keepFailed, true},
-    });
+    mkFlag()
+        .longName("keep-failed")
+        .shortName('K')
+        .description("keep temporary directories of failed builds")
+        .set(&(bool&) settings.keepFailed, true);
 
-    addFlag({
-        .longName = "keep-going",
-        .shortName ='k',
-        .description = "keep going after a build fails",
-        .handler = {&(bool&) settings.keepGoing, true},
-    });
+    mkFlag()
+        .longName("keep-going")
+        .shortName('k')
+        .description("keep going after a build fails")
+        .set(&(bool&) settings.keepGoing, true);
 
-    addFlag({
-        .longName = "fallback",
-        .description = "build from source if substitution fails",
-        .handler = {&(bool&) settings.tryFallback, true},
-    });
+    mkFlag()
+        .longName("fallback")
+        .description("build from source if substitution fails")
+        .set(&(bool&) settings.tryFallback, true);
 
     auto intSettingAlias = [&](char shortName, const std::string & longName,
         const std::string & description, const std::string & dest) {
@@ -209,12 +205,11 @@ LegacyArgs::LegacyArgs(const std::string & programName,
     mkFlag(0, "no-gc-warning", "disable warning about not using '--add-root'",
         &gcWarning, false);
 
-    addFlag({
-        .longName = "store",
-        .description = "URI of the Nix store to use",
-        .labels = {"store-uri"},
-        .handler = {&(std::string&) settings.storeUri},
-    });
+    mkFlag()
+        .longName("store")
+        .label("store-uri")
+        .description("URI of the Nix store to use")
+        .dest(&(std::string&) settings.storeUri);
 }
 
 
@@ -265,10 +260,7 @@ void printVersion(const string & programName)
         cfg.push_back("signed-caches");
 #endif
         std::cout << "Features: " << concatStringsSep(", ", cfg) << "\n";
-        std::cout << "System configuration file: " << settings.nixConfDir + "/nix.conf" << "\n";
-        std::cout << "User configuration files: " <<
-            concatStringsSep(":", settings.nixUserConfFiles)
-            << "\n";
+        std::cout << "Configuration file: " << settings.nixConfDir + "/nix.conf" << "\n";
         std::cout << "Store directory: " << settings.nixStore << "\n";
         std::cout << "State directory: " << settings.nixStateDir << "\n";
     }

src/libstore/build.cc

@@ -7,7 +7,7 @@
 #include "affinity.hh"
 #include "builtins.hh"
 #include "builtins/buildenv.hh"
-#include "filetransfer.hh"
+#include "download.hh"
 #include "finally.hh"
 #include "compression.hh"
 #include "json.hh"
@@ -33,6 +33,7 @@
 #include <sys/types.h>
 #include <sys/stat.h>
 #include <sys/utsname.h>
+#include <sys/select.h>
 #include <sys/resource.h>
 #include <sys/socket.h>
 #include <sys/un.h>
@@ -42,7 +43,6 @@
 #include <errno.h>
 #include <cstring>
 #include <termios.h>
-#include <poll.h>
 
 #include <pwd.h>
 #include <grp.h>
@@ -361,7 +361,7 @@ public:
     {
         actDerivations.progress(doneBuilds, expectedBuilds + doneBuilds, runningBuilds, failedBuilds);
         actSubstitutions.progress(doneSubstitutions, expectedSubstitutions + doneSubstitutions, runningSubstitutions, failedSubstitutions);
-        act.setExpected(actFileTransfer, expectedDownloadSize + doneDownloadSize);
+        act.setExpected(actDownload, expectedDownloadSize + doneDownloadSize);
         act.setExpected(actCopyPath, expectedNarSize + doneNarSize);
     }
 };
@@ -1680,7 +1680,6 @@ void DerivationGoal::buildDone()
         }
 
         if (buildMode == bmCheck) {
-            deleteTmpDir(true);
             done(BuildResult::Built);
             return;
         }
@@ -2162,7 +2161,7 @@ void DerivationGoal::startBuilder()
     if (needsHashRewrite()) {
 
         if (pathExists(homeDir))
-            throw Error(format("home directory '%1%' exists; please remove it to assure purity of builds without sandboxing") % homeDir);
+            throw Error(format("directory '%1%' exists; please remove it") % homeDir);
 
         /* We're not doing a chroot build, but we have some valid
            output paths.  Since we can't just overwrite or delete
@@ -2250,13 +2249,10 @@ void DerivationGoal::startBuilder()
 
         if (chown(slaveName.c_str(), buildUser->getUID(), 0))
             throw SysError("changing owner of pseudoterminal slave");
-    }
-#if __APPLE__
-    else {
+    } else {
         if (grantpt(builderOut.readSide.get()))
             throw SysError("granting access to pseudoterminal slave");
     }
-#endif
 
     #if 0
     // Mount the pt in the sandbox so that the "tty" command works.
@@ -2469,7 +2465,7 @@ void DerivationGoal::initTmpDir() {
                 auto hash = hashString(htSHA256, i.first);
                 string fn = ".attr-" + hash.to_string(Base32, false);
                 Path p = tmpDir + "/" + fn;
-                writeFile(p, rewriteStrings(i.second, inputRewrites));
+                writeFile(p, i.second);
                 chownToBuilder(p);
                 env[i.first + "Path"] = tmpDirInSandbox + "/" + fn;
             }
@@ -3540,29 +3536,6 @@ StorePathSet parseReferenceSpecifiers(Store & store, const BasicDerivation & drv
 }
 
 
-static void moveCheckToStore(const Path & src, const Path & dst)
-{
-    /* For the rename of directory to succeed, we must be running as root or
-       the directory must be made temporarily writable (to update the
-       directory's parent link ".."). */
-    struct stat st;
-    if (lstat(src.c_str(), &st) == -1) {
-        throw SysError(format("getting attributes of path '%1%'") % src);
-    }
-
-    bool changePerm = (geteuid() && S_ISDIR(st.st_mode) && !(st.st_mode & S_IWUSR));
-
-    if (changePerm)
-        chmod_(src, st.st_mode | S_IWUSR);
-
-    if (rename(src.c_str(), dst.c_str()))
-        throw SysError(format("renaming '%1%' to '%2%'") % src % dst);
-
-    if (changePerm)
-        chmod_(dst, st.st_mode);
-}
-
-
 void DerivationGoal::registerOutputs()
 {
     /* When using a build hook, the build hook can register the output
@@ -3741,7 +3714,8 @@ void DerivationGoal::registerOutputs()
                 if (settings.runDiffHook || settings.keepFailed) {
                     Path dst = worker.store.toRealPath(path + checkSuffix);
                     deletePath(dst);
-                    moveCheckToStore(actualPath, dst);
+                    if (rename(actualPath.c_str(), dst.c_str()))
+                        throw SysError(format("renaming '%1%' to '%2%'") % actualPath % dst);
 
                     handleDiffHook(
                         buildUser ? buildUser->getUID() : getuid(),
@@ -3749,10 +3723,10 @@ void DerivationGoal::registerOutputs()
                         path, dst, worker.store.printStorePath(drvPath), tmpDir);
 
                     throw NotDeterministic("derivation '%s' may not be deterministic: output '%s' differs from '%s'",
-                        worker.store.printStorePath(drvPath), worker.store.toRealPath(path), dst);
+                        worker.store.printStorePath(drvPath), path, dst);
                 } else
                     throw NotDeterministic("derivation '%s' may not be deterministic: output '%s' differs",
-                        worker.store.printStorePath(drvPath), worker.store.toRealPath(path));
+                        worker.store.printStorePath(drvPath), path);
             }
 
             /* Since we verified the build, it's now ultimately trusted. */
@@ -4792,7 +4766,8 @@ void Worker::waitForInput()
        terminated. */
 
     bool useTimeout = false;
-    long timeout = 0;
+    struct timeval timeout;
+    timeout.tv_usec = 0;
     auto before = steady_time_point::clock::now();
 
     /* If we're monitoring for silence on stdout/stderr, or if there
@@ -4810,7 +4785,7 @@ void Worker::waitForInput()
             nearest = std::min(nearest, i.timeStarted + std::chrono::seconds(settings.buildTimeout));
     }
     if (nearest != steady_time_point::max()) {
-        timeout = std::max(1L, (long) std::chrono::duration_cast<std::chrono::seconds>(nearest - before).count());
+        timeout.tv_sec = std::max(1L, (long) std::chrono::duration_cast<std::chrono::seconds>(nearest - before).count());
         useTimeout = true;
     }
 
@@ -4821,28 +4796,30 @@ void Worker::waitForInput()
         if (lastWokenUp == steady_time_point::min())
             printError("waiting for locks or build slots...");
         if (lastWokenUp == steady_time_point::min() || lastWokenUp > before) lastWokenUp = before;
-        timeout = std::max(1L,
+        timeout.tv_sec = std::max(1L,
             (long) std::chrono::duration_cast<std::chrono::seconds>(
                 lastWokenUp + std::chrono::seconds(settings.pollInterval) - before).count());
     } else lastWokenUp = steady_time_point::min();
 
     if (useTimeout)
-        vomit("sleeping %d seconds", timeout);
+        vomit("sleeping %d seconds", timeout.tv_sec);
 
     /* Use select() to wait for the input side of any logger pipe to
        become `available'.  Note that `available' (i.e., non-blocking)
        includes EOF. */
-    std::vector<struct pollfd> pollStatus;
-    std::map <int, int> fdToPollStatus;
+    fd_set fds;
+    FD_ZERO(&fds);
+    int fdMax = 0;
     for (auto & i : children) {
         for (auto & j : i.fds) {
-            pollStatus.push_back((struct pollfd) { .fd = j, .events = POLLIN });
-            fdToPollStatus[j] = pollStatus.size() - 1;
+            if (j >= FD_SETSIZE)
+                throw Error("reached FD_SETSIZE limit");
+            FD_SET(j, &fds);
+            if (j >= fdMax) fdMax = j + 1;
         }
     }
 
-    if (poll(pollStatus.data(), pollStatus.size(),
-            useTimeout ? timeout * 1000 : -1) == -1) {
+    if (select(fdMax, &fds, 0, 0, useTimeout ? &timeout : 0) == -1) {
         if (errno == EINTR) return;
         throw SysError("waiting for input");
     }
@@ -4863,7 +4840,7 @@ void Worker::waitForInput()
         set<int> fds2(j->fds);
         std::vector<unsigned char> buffer(4096);
         for (auto & k : fds2) {
-            if (pollStatus.at(fdToPollStatus.at(k)).revents) {
+            if (FD_ISSET(k, &fds)) {
                 ssize_t rd = read(k, buffer.data(), buffer.size());
                 // FIXME: is there a cleaner way to handle pt close
                 // than EIO? Is this even standard?

src/libstore/builtins/buildenv.hh

@@ -9,7 +9,7 @@ struct Package {
     Path path;
     bool active;
     int priority;
-    Package(const Path & path, bool active, int priority) : path{path}, active{active}, priority{priority} {}
+    Package(Path path, bool active, int priority) : path{path}, active{active}, priority{priority} {}
 };
 
 typedef std::vector<Package> Packages;

src/libstore/builtins/fetchurl.cc

@@ -1,5 +1,5 @@
 #include "builtins.hh"
-#include "filetransfer.hh"
+#include "download.hh"
 #include "store-api.hh"
 #include "archive.hh"
 #include "compression.hh"
@@ -26,9 +26,9 @@ void builtinFetchurl(const BasicDerivation & drv, const std::string & netrcData)
     auto mainUrl = getAttr("url");
     bool unpack = get(drv.env, "unpack").value_or("") == "1";
 
-    /* Note: have to use a fresh fileTransfer here because we're in
+    /* Note: have to use a fresh downloader here because we're in
        a forked process. */
-    auto fileTransfer = makeFileTransfer();
+    auto downloader = makeDownloader();
 
     auto fetch = [&](const std::string & url) {
 
@@ -36,13 +36,13 @@ void builtinFetchurl(const BasicDerivation & drv, const std::string & netrcData)
 
             /* No need to do TLS verification, because we check the hash of
                the result anyway. */
-            FileTransferRequest request(url);
+            DownloadRequest request(url);
             request.verifyTLS = false;
             request.decompress = false;
 
             auto decompressor = makeDecompressionSink(
                 unpack && hasSuffix(mainUrl, ".xz") ? "xz" : "none", sink);
-            fileTransfer->download(std::move(request), *decompressor);
+            downloader->download(std::move(request), *decompressor);
             decompressor->finish();
         });
 

src/libstore/derivations.cc

@@ -378,7 +378,7 @@ Hash hashDerivationModulo(Store & store, const Derivation & drv, bool maskOutput
         if (h == drvHashes.end()) {
             assert(store.isValidPath(i.first));
             h = drvHashes.insert_or_assign(i.first.clone(), hashDerivationModulo(store,
-                readDerivation(store, store.toRealPath(i.first)), false)).first;
+                readDerivation(store, store.toRealPath(store.printStorePath(i.first))), false)).first;
         }
         inputs2.insert_or_assign(h->second.to_string(Base16, false), i.second);
     }

src/libstore/download.cc

@@ -1,10 +1,14 @@
-#include "filetransfer.hh"
+#include "download.hh"
 #include "util.hh"
 #include "globals.hh"
+#include "hash.hh"
 #include "store-api.hh"
+#include "archive.hh"
 #include "s3.hh"
 #include "compression.hh"
+#include "pathlocks.hh"
 #include "finally.hh"
+#include "tarfile.hh"
 
 #ifdef ENABLE_S3
 #include <aws/core/client/ClientConfiguration.h>
@@ -27,9 +31,13 @@ using namespace std::string_literals;
 
 namespace nix {
 
-FileTransferSettings fileTransferSettings;
+DownloadSettings downloadSettings;
 
-static GlobalConfig::Register r1(&fileTransferSettings);
+static GlobalConfig::Register r1(&downloadSettings);
+
+CachedDownloadRequest::CachedDownloadRequest(const std::string & uri)
+    : uri(uri), ttl(settings.tarballTtl)
+{ }
 
 std::string resolveUri(const std::string & uri)
 {
@@ -39,21 +47,21 @@ std::string resolveUri(const std::string & uri)
         return uri;
 }
 
-struct curlFileTransfer : public FileTransfer
+struct CurlDownloader : public Downloader
 {
     CURLM * curlm = 0;
 
     std::random_device rd;
     std::mt19937 mt19937;
 
-    struct TransferItem : public std::enable_shared_from_this<TransferItem>
+    struct DownloadItem : public std::enable_shared_from_this<DownloadItem>
     {
-        curlFileTransfer & fileTransfer;
-        FileTransferRequest request;
-        FileTransferResult result;
+        CurlDownloader & downloader;
+        DownloadRequest request;
+        DownloadResult result;
         Activity act;
         bool done = false; // whether either the success or failure function has been called
-        Callback<FileTransferResult> callback;
+        Callback<DownloadResult> callback;
         CURL * req = 0;
         bool active = false; // whether the handle has been added to the multi object
         std::string status;
@@ -72,26 +80,19 @@ struct curlFileTransfer : public FileTransfer
 
         curl_off_t writtenToSink = 0;
 
-        TransferItem(curlFileTransfer & fileTransfer,
-            const FileTransferRequest & request,
-            Callback<FileTransferResult> && callback)
-            : fileTransfer(fileTransfer)
+        DownloadItem(CurlDownloader & downloader,
+            const DownloadRequest & request,
+            Callback<DownloadResult> && callback)
+            : downloader(downloader)
             , request(request)
-            , act(*logger, lvlTalkative, actFileTransfer,
+            , act(*logger, lvlTalkative, actDownload,
                 fmt(request.data ? "uploading '%s'" : "downloading '%s'", request.uri),
                 {request.uri}, request.parentAct)
             , callback(std::move(callback))
             , finalSink([this](const unsigned char * data, size_t len) {
                 if (this->request.dataCallback) {
-                    long httpStatus = 0;
-                    curl_easy_getinfo(req, CURLINFO_RESPONSE_CODE, &httpStatus);
-
-                    /* Only write data to the sink if this is a
-                       successful response. */
-                    if (httpStatus == 0 || httpStatus == 200 || httpStatus == 201 || httpStatus == 206) {
-                        writtenToSink += len;
-                        this->request.dataCallback((char *) data, len);
-                    }
+                    writtenToSink += len;
+                    this->request.dataCallback((char *) data, len);
                 } else
                     this->result.data->append((char *) data, len);
               })
@@ -102,17 +103,17 @@ struct curlFileTransfer : public FileTransfer
                 requestHeaders = curl_slist_append(requestHeaders, ("Content-Type: " + request.mimeType).c_str());
         }
 
-        ~TransferItem()
+        ~DownloadItem()
         {
             if (req) {
                 if (active)
-                    curl_multi_remove_handle(fileTransfer.curlm, req);
+                    curl_multi_remove_handle(downloader.curlm, req);
                 curl_easy_cleanup(req);
             }
             if (requestHeaders) curl_slist_free_all(requestHeaders);
             try {
                 if (!done)
-                    fail(FileTransferError(Interrupted, format("download of '%s' was interrupted") % request.uri));
+                    fail(DownloadError(Interrupted, format("download of '%s' was interrupted") % request.uri));
             } catch (...) {
                 ignoreException();
             }
@@ -156,7 +157,7 @@ struct curlFileTransfer : public FileTransfer
 
         static size_t writeCallbackWrapper(void * contents, size_t size, size_t nmemb, void * userp)
         {
-            return ((TransferItem *) userp)->writeCallback(contents, size, nmemb);
+            return ((DownloadItem *) userp)->writeCallback(contents, size, nmemb);
         }
 
         size_t headerCallback(void * contents, size_t size, size_t nmemb)
@@ -198,7 +199,7 @@ struct curlFileTransfer : public FileTransfer
 
         static size_t headerCallbackWrapper(void * contents, size_t size, size_t nmemb, void * userp)
         {
-            return ((TransferItem *) userp)->headerCallback(contents, size, nmemb);
+            return ((DownloadItem *) userp)->headerCallback(contents, size, nmemb);
         }
 
         int progressCallback(double dltotal, double dlnow)
@@ -213,7 +214,7 @@ struct curlFileTransfer : public FileTransfer
 
         static int progressCallbackWrapper(void * userp, double dltotal, double dlnow, double ultotal, double ulnow)
         {
-            return ((TransferItem *) userp)->progressCallback(dltotal, dlnow);
+            return ((DownloadItem *) userp)->progressCallback(dltotal, dlnow);
         }
 
         static int debugCallback(CURL * handle, curl_infotype type, char * data, size_t size, void * userptr)
@@ -237,7 +238,7 @@ struct curlFileTransfer : public FileTransfer
 
         static size_t readCallbackWrapper(char *buffer, size_t size, size_t nitems, void * userp)
         {
-            return ((TransferItem *) userp)->readCallback(buffer, size, nitems);
+            return ((DownloadItem *) userp)->readCallback(buffer, size, nitems);
         }
 
         void init()
@@ -248,7 +249,7 @@ struct curlFileTransfer : public FileTransfer
 
             if (verbosity >= lvlVomit) {
                 curl_easy_setopt(req, CURLOPT_VERBOSE, 1);
-                curl_easy_setopt(req, CURLOPT_DEBUGFUNCTION, TransferItem::debugCallback);
+                curl_easy_setopt(req, CURLOPT_DEBUGFUNCTION, DownloadItem::debugCallback);
             }
 
             curl_easy_setopt(req, CURLOPT_URL, request.uri.c_str());
@@ -257,19 +258,19 @@ struct curlFileTransfer : public FileTransfer
             curl_easy_setopt(req, CURLOPT_NOSIGNAL, 1);
             curl_easy_setopt(req, CURLOPT_USERAGENT,
                 ("curl/" LIBCURL_VERSION " Nix/" + nixVersion +
-                    (fileTransferSettings.userAgentSuffix != "" ? " " + fileTransferSettings.userAgentSuffix.get() : "")).c_str());
+                    (downloadSettings.userAgentSuffix != "" ? " " + downloadSettings.userAgentSuffix.get() : "")).c_str());
             #if LIBCURL_VERSION_NUM >= 0x072b00
             curl_easy_setopt(req, CURLOPT_PIPEWAIT, 1);
             #endif
             #if LIBCURL_VERSION_NUM >= 0x072f00
-            if (fileTransferSettings.enableHttp2)
+            if (downloadSettings.enableHttp2)
                 curl_easy_setopt(req, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2TLS);
             else
                 curl_easy_setopt(req, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
             #endif
-            curl_easy_setopt(req, CURLOPT_WRITEFUNCTION, TransferItem::writeCallbackWrapper);
+            curl_easy_setopt(req, CURLOPT_WRITEFUNCTION, DownloadItem::writeCallbackWrapper);
             curl_easy_setopt(req, CURLOPT_WRITEDATA, this);
-            curl_easy_setopt(req, CURLOPT_HEADERFUNCTION, TransferItem::headerCallbackWrapper);
+            curl_easy_setopt(req, CURLOPT_HEADERFUNCTION, DownloadItem::headerCallbackWrapper);
             curl_easy_setopt(req, CURLOPT_HEADERDATA, this);
 
             curl_easy_setopt(req, CURLOPT_PROGRESSFUNCTION, progressCallbackWrapper);
@@ -297,17 +298,17 @@ struct curlFileTransfer : public FileTransfer
                 curl_easy_setopt(req, CURLOPT_SSL_VERIFYHOST, 0);
             }
 
-            curl_easy_setopt(req, CURLOPT_CONNECTTIMEOUT, fileTransferSettings.connectTimeout.get());
+            curl_easy_setopt(req, CURLOPT_CONNECTTIMEOUT, downloadSettings.connectTimeout.get());
 
             curl_easy_setopt(req, CURLOPT_LOW_SPEED_LIMIT, 1L);
-            curl_easy_setopt(req, CURLOPT_LOW_SPEED_TIME, fileTransferSettings.stalledDownloadTimeout.get());
+            curl_easy_setopt(req, CURLOPT_LOW_SPEED_TIME, downloadSettings.stalledDownloadTimeout.get());
 
             /* If no file exist in the specified path, curl continues to work
                anyway as if netrc support was disabled. */
             curl_easy_setopt(req, CURLOPT_NETRC_FILE, settings.netrcFile.get().c_str());
             curl_easy_setopt(req, CURLOPT_NETRC, CURL_NETRC_OPTIONAL);
 
-            if (writtenToSink)
+            if (writtenToSink && acceptRanges && encoding.empty())
                 curl_easy_setopt(req, CURLOPT_RESUME_FROM_LARGE, writtenToSink);
 
             result.data = std::make_shared<std::string>();
@@ -389,7 +390,6 @@ struct curlFileTransfer : public FileTransfer
                         case CURLE_SSL_CACERT_BADFILE:
                         case CURLE_TOO_MANY_REDIRECTS:
                         case CURLE_WRITE_ERROR:
-                        case CURLE_UNSUPPORTED_PROTOCOL:
                             err = Misc;
                             break;
                         default: // Shut up warnings
@@ -401,34 +401,32 @@ struct curlFileTransfer : public FileTransfer
 
                 auto exc =
                     code == CURLE_ABORTED_BY_CALLBACK && _isInterrupted
-                    ? FileTransferError(Interrupted, fmt("%s of '%s' was interrupted", request.verb(), request.uri))
+                    ? DownloadError(Interrupted, fmt("%s of '%s' was interrupted", request.verb(), request.uri))
                     : httpStatus != 0
-                    ? FileTransferError(err,
+                    ? DownloadError(err,
                         fmt("unable to %s '%s': HTTP error %d",
                             request.verb(), request.uri, httpStatus)
                         + (code == CURLE_OK ? "" : fmt(" (curl error: %s)", curl_easy_strerror(code)))
                         )
-                    : FileTransferError(err,
+                    : DownloadError(err,
                         fmt("unable to %s '%s': %s (%d)",
                             request.verb(), request.uri, curl_easy_strerror(code), code));
 
                 /* If this is a transient error, then maybe retry the
                    download after a while. If we're writing to a
-                   sink, we can only retry if the server supports
-                   ranged requests. */
-                if (err == Transient
-                    && attempt < request.tries
-                    && (!this->request.dataCallback
-                        || writtenToSink == 0
-                        || (acceptRanges && encoding.empty())))
+                   sink, we can only resume if the server supports
+                   ranged requests, otherwise fallback to a plain
+                   retry.
+                */
+                if (err == Transient && attempt < request.tries)
                 {
-                    int ms = request.baseRetryTimeMs * std::pow(2.0f, attempt - 1 + std::uniform_real_distribution<>(0.0, 0.5)(fileTransfer.mt19937));
-                    if (writtenToSink)
+                    int ms = request.baseRetryTimeMs * std::pow(2.0f, attempt - 1 + std::uniform_real_distribution<>(0.0, 0.5)(downloader.mt19937));
+                    if (writtenToSink && acceptRanges && encoding.empty())
                         warn("%s; retrying from offset %d in %d ms", exc.what(), writtenToSink, ms);
                     else
                         warn("%s; retrying in %d ms", exc.what(), ms);
                     embargo = std::chrono::steady_clock::now() + std::chrono::milliseconds(ms);
-                    fileTransfer.enqueueItem(shared_from_this());
+                    downloader.enqueueItem(shared_from_this());
                 }
                 else
                     fail(exc);
@@ -439,12 +437,12 @@ struct curlFileTransfer : public FileTransfer
     struct State
     {
         struct EmbargoComparator {
-            bool operator() (const std::shared_ptr<TransferItem> & i1, const std::shared_ptr<TransferItem> & i2) {
+            bool operator() (const std::shared_ptr<DownloadItem> & i1, const std::shared_ptr<DownloadItem> & i2) {
                 return i1->embargo > i2->embargo;
             }
         };
         bool quit = false;
-        std::priority_queue<std::shared_ptr<TransferItem>, std::vector<std::shared_ptr<TransferItem>>, EmbargoComparator> incoming;
+        std::priority_queue<std::shared_ptr<DownloadItem>, std::vector<std::shared_ptr<DownloadItem>>, EmbargoComparator> incoming;
     };
 
     Sync<State> state_;
@@ -456,7 +454,7 @@ struct curlFileTransfer : public FileTransfer
 
     std::thread workerThread;
 
-    curlFileTransfer()
+    CurlDownloader()
         : mt19937(rd())
     {
         static std::once_flag globalInit;
@@ -469,7 +467,7 @@ struct curlFileTransfer : public FileTransfer
         #endif
         #if LIBCURL_VERSION_NUM >= 0x071e00 // Max connections requires >= 7.30.0
         curl_multi_setopt(curlm, CURLMOPT_MAX_TOTAL_CONNECTIONS,
-            fileTransferSettings.httpConnections.get());
+            downloadSettings.httpConnections.get());
         #endif
 
         wakeupPipe.create();
@@ -478,7 +476,7 @@ struct curlFileTransfer : public FileTransfer
         workerThread = std::thread([&]() { workerThreadEntry(); });
     }
 
-    ~curlFileTransfer()
+    ~CurlDownloader()
     {
         stopWorkerThread();
 
@@ -504,7 +502,7 @@ struct curlFileTransfer : public FileTransfer
             stopWorkerThread();
         });
 
-        std::map<CURL *, std::shared_ptr<TransferItem>> items;
+        std::map<CURL *, std::shared_ptr<DownloadItem>> items;
 
         bool quit = false;
 
@@ -561,7 +559,7 @@ struct curlFileTransfer : public FileTransfer
                     throw SysError("reading curl wakeup socket");
             }
 
-            std::vector<std::shared_ptr<TransferItem>> incoming;
+            std::vector<std::shared_ptr<DownloadItem>> incoming;
             auto now = std::chrono::steady_clock::now();
 
             {
@@ -609,7 +607,7 @@ struct curlFileTransfer : public FileTransfer
         }
     }
 
-    void enqueueItem(std::shared_ptr<TransferItem> item)
+    void enqueueItem(std::shared_ptr<DownloadItem> item)
     {
         if (item->request.data
             && !hasPrefix(item->request.uri, "http://")
@@ -641,8 +639,8 @@ struct curlFileTransfer : public FileTransfer
     }
 #endif
 
-    void enqueueFileTransfer(const FileTransferRequest & request,
-        Callback<FileTransferResult> callback) override
+    void enqueueDownload(const DownloadRequest & request,
+        Callback<DownloadResult> callback) override
     {
         /* Ugly hack to support s3:// URIs. */
         if (hasPrefix(request.uri, "s3://")) {
@@ -660,9 +658,9 @@ struct curlFileTransfer : public FileTransfer
 
                 // FIXME: implement ETag
                 auto s3Res = s3Helper.getObject(bucketName, key);
-                FileTransferResult res;
+                DownloadResult res;
                 if (!s3Res.data)
-                    throw FileTransferError(NotFound, fmt("S3 object '%s' does not exist", request.uri));
+                    throw DownloadError(NotFound, fmt("S3 object '%s' does not exist", request.uri));
                 res.data = s3Res.data;
                 callback(std::move(res));
 #else
@@ -672,26 +670,26 @@ struct curlFileTransfer : public FileTransfer
             return;
         }
 
-        enqueueItem(std::make_shared<TransferItem>(*this, request, std::move(callback)));
+        enqueueItem(std::make_shared<DownloadItem>(*this, request, std::move(callback)));
     }
 };
 
-ref<FileTransfer> getFileTransfer()
+ref<Downloader> getDownloader()
 {
-    static ref<FileTransfer> fileTransfer = makeFileTransfer();
-    return fileTransfer;
+    static ref<Downloader> downloader = makeDownloader();
+    return downloader;
 }
 
-ref<FileTransfer> makeFileTransfer()
+ref<Downloader> makeDownloader()
 {
-    return make_ref<curlFileTransfer>();
+    return make_ref<CurlDownloader>();
 }
 
-std::future<FileTransferResult> FileTransfer::enqueueFileTransfer(const FileTransferRequest & request)
+std::future<DownloadResult> Downloader::enqueueDownload(const DownloadRequest & request)
 {
-    auto promise = std::make_shared<std::promise<FileTransferResult>>();
-    enqueueFileTransfer(request,
-        {[promise](std::future<FileTransferResult> fut) {
+    auto promise = std::make_shared<std::promise<DownloadResult>>();
+    enqueueDownload(request,
+        {[promise](std::future<DownloadResult> fut) {
             try {
                 promise->set_value(fut.get());
             } catch (...) {
@@ -701,21 +699,15 @@ std::future<FileTransferResult> FileTransfer::enqueueFileTransfer(const FileTran
     return promise->get_future();
 }
 
-FileTransferResult FileTransfer::download(const FileTransferRequest & request)
+DownloadResult Downloader::download(const DownloadRequest & request)
 {
-    return enqueueFileTransfer(request).get();
+    return enqueueDownload(request).get();
 }
 
-FileTransferResult FileTransfer::upload(const FileTransferRequest & request)
-{
-    /* Note: this method is the same as download, but helps in readability */
-    return enqueueFileTransfer(request).get();
-}
-
-void FileTransfer::download(FileTransferRequest && request, Sink & sink)
+void Downloader::download(DownloadRequest && request, Sink & sink)
 {
     /* Note: we can't call 'sink' via request.dataCallback, because
-       that would cause the sink to execute on the fileTransfer
+       that would cause the sink to execute on the downloader
        thread. If 'sink' is a coroutine, this will fail. Also, if the
        sink is expensive (e.g. one that does decompression and writing
        to the Nix store), it would stall the download thread too much.
@@ -761,8 +753,8 @@ void FileTransfer::download(FileTransferRequest && request, Sink & sink)
         state->avail.notify_one();
     };
 
-    enqueueFileTransfer(request,
-        {[_state](std::future<FileTransferResult> fut) {
+    enqueueDownload(request,
+        {[_state](std::future<DownloadResult> fut) {
             auto state(_state->lock());
             state->quit = true;
             try {
@@ -807,6 +799,140 @@ void FileTransfer::download(FileTransferRequest && request, Sink & sink)
     }
 }
 
+CachedDownloadResult Downloader::downloadCached(
+    ref<Store> store, const CachedDownloadRequest & request)
+{
+    auto url = resolveUri(request.uri);
+
+    auto name = request.name;
+    if (name == "") {
+        auto p = url.rfind('/');
+        if (p != string::npos) name = string(url, p + 1);
+    }
+
+    std::optional<StorePath> expectedStorePath;
+    if (request.expectedHash) {
+        expectedStorePath = store->makeFixedOutputPath(request.unpack, request.expectedHash, name);
+        if (store->isValidPath(*expectedStorePath)) {
+            CachedDownloadResult result;
+            result.storePath = store->printStorePath(*expectedStorePath);
+            result.path = store->toRealPath(result.storePath);
+            return result;
+        }
+    }
+
+    Path cacheDir = getCacheDir() + "/nix/tarballs";
+    createDirs(cacheDir);
+
+    string urlHash = hashString(htSHA256, name + std::string("\0"s) + url).to_string(Base32, false);
+
+    Path dataFile = cacheDir + "/" + urlHash + ".info";
+    Path fileLink = cacheDir + "/" + urlHash + "-file";
+
+    PathLocks lock({fileLink}, fmt("waiting for lock on '%1%'...", fileLink));
+
+    std::optional<StorePath> storePath;
+
+    string expectedETag;
+
+    bool skip = false;
+
+    CachedDownloadResult result;
+
+    if (pathExists(fileLink) && pathExists(dataFile)) {
+        storePath = store->parseStorePath(readLink(fileLink));
+        // FIXME
+        store->addTempRoot(*storePath);
+        if (store->isValidPath(*storePath)) {
+            auto ss = tokenizeString<vector<string>>(readFile(dataFile), "\n");
+            if (ss.size() >= 3 && ss[0] == url) {
+                time_t lastChecked;
+                if (string2Int(ss[2], lastChecked) && (uint64_t) lastChecked + request.ttl >= (uint64_t) time(0)) {
+                    skip = true;
+                    result.effectiveUri = request.uri;
+                    result.etag = ss[1];
+                } else if (!ss[1].empty()) {
+                    debug(format("verifying previous ETag '%1%'") % ss[1]);
+                    expectedETag = ss[1];
+                }
+            }
+        } else
+            storePath.reset();
+    }
+
+    if (!skip) {
+
+        try {
+            DownloadRequest request2(url);
+            request2.expectedETag = expectedETag;
+            auto res = download(request2);
+            result.effectiveUri = res.effectiveUri;
+            result.etag = res.etag;
+
+            if (!res.cached) {
+                StringSink sink;
+                dumpString(*res.data, sink);
+                Hash hash = hashString(request.expectedHash ? request.expectedHash.type : htSHA256, *res.data);
+                ValidPathInfo info(store->makeFixedOutputPath(false, hash, name));
+                info.narHash = hashString(htSHA256, *sink.s);
+                info.narSize = sink.s->size();
+                info.ca = makeFixedOutputCA(false, hash);
+                store->addToStore(info, sink.s, NoRepair, NoCheckSigs);
+                storePath = info.path.clone();
+            }
+
+            assert(storePath);
+            replaceSymlink(store->printStorePath(*storePath), fileLink);
+
+            writeFile(dataFile, url + "\n" + res.etag + "\n" + std::to_string(time(0)) + "\n");
+        } catch (DownloadError & e) {
+            if (!storePath) throw;
+            warn("warning: %s; using cached result", e.msg());
+            result.etag = expectedETag;
+        }
+    }
+
+    if (request.unpack) {
+        Path unpackedLink = cacheDir + "/" + ((std::string) storePath->to_string()) + "-unpacked";
+        PathLocks lock2({unpackedLink}, fmt("waiting for lock on '%1%'...", unpackedLink));
+        std::optional<StorePath> unpackedStorePath;
+        if (pathExists(unpackedLink)) {
+            unpackedStorePath = store->parseStorePath(readLink(unpackedLink));
+            // FIXME
+            store->addTempRoot(*unpackedStorePath);
+            if (!store->isValidPath(*unpackedStorePath))
+                unpackedStorePath.reset();
+        }
+        if (!unpackedStorePath) {
+            printInfo("unpacking '%s'...", url);
+            Path tmpDir = createTempDir();
+            AutoDelete autoDelete(tmpDir, true);
+            unpackTarfile(store->toRealPath(store->printStorePath(*storePath)), tmpDir);
+            auto members = readDirectory(tmpDir);
+            if (members.size() != 1)
+                throw nix::Error("tarball '%s' contains an unexpected number of top-level files", url);
+            auto topDir = tmpDir + "/" + members.begin()->name;
+            unpackedStorePath = store->addToStore(name, topDir, true, htSHA256, defaultPathFilter, NoRepair);
+        }
+        replaceSymlink(store->printStorePath(*unpackedStorePath), unpackedLink);
+        storePath = std::move(*unpackedStorePath);
+    }
+
+    if (expectedStorePath && *storePath != *expectedStorePath) {
+        unsigned int statusCode = 102;
+        Hash gotHash = request.unpack
+            ? hashPath(request.expectedHash.type, store->toRealPath(store->printStorePath(*storePath))).first
+            : hashFile(request.expectedHash.type, store->toRealPath(store->printStorePath(*storePath)));
+        throw nix::Error(statusCode, "hash mismatch in file downloaded from '%s':\n  wanted: %s\n  got:    %s",
+            url, request.expectedHash.to_string(), gotHash.to_string());
+    }
+
+    result.storePath = store->printStorePath(*storePath);
+    result.path = store->toRealPath(result.storePath);
+    return result;
+}
+
+
 bool isUri(const string & s)
 {
     if (s.compare(0, 8, "channel:") == 0) return true;

src/libstore/download.hh

@@ -9,7 +9,7 @@
 
 namespace nix {
 
-struct FileTransferSettings : Config
+struct DownloadSettings : Config
 {
     Setting<bool> enableHttp2{this, true, "http2",
         "Whether to enable HTTP/2 support."};
@@ -31,15 +31,15 @@ struct FileTransferSettings : Config
         "How often Nix will attempt to download a file before giving up."};
 };
 
-extern FileTransferSettings fileTransferSettings;
+extern DownloadSettings downloadSettings;
 
-struct FileTransferRequest
+struct DownloadRequest
 {
     std::string uri;
     std::string expectedETag;
     bool verifyTLS = true;
     bool head = false;
-    size_t tries = fileTransferSettings.tries;
+    size_t tries = downloadSettings.tries;
     unsigned int baseRetryTimeMs = 250;
     ActivityId parentAct;
     bool decompress = true;
@@ -47,7 +47,7 @@ struct FileTransferRequest
     std::string mimeType;
     std::function<void(char *, size_t)> dataCallback;
 
-    FileTransferRequest(const std::string & uri)
+    DownloadRequest(const std::string & uri)
         : uri(uri), parentAct(getCurActivity()) { }
 
     std::string verb()
@@ -56,7 +56,7 @@ struct FileTransferRequest
     }
 };
 
-struct FileTransferResult
+struct DownloadResult
 {
     bool cached = false;
     std::string etag;
@@ -65,52 +65,74 @@ struct FileTransferResult
     uint64_t bodySize = 0;
 };
 
+struct CachedDownloadRequest
+{
+    std::string uri;
+    bool unpack = false;
+    std::string name;
+    Hash expectedHash;
+    unsigned int ttl;
+
+    CachedDownloadRequest(const std::string & uri);
+    CachedDownloadRequest() = delete;
+};
+
+struct CachedDownloadResult
+{
+    // Note: 'storePath' may be different from 'path' when using a
+    // chroot store.
+    Path storePath;
+    Path path;
+    std::optional<std::string> etag;
+    std::string effectiveUri;
+};
+
 class Store;
 
-struct FileTransfer
+struct Downloader
 {
-    virtual ~FileTransfer() { }
+    virtual ~Downloader() { }
 
-    /* Enqueue a data transfer request, returning a future to the result of
-       the download. The future may throw a FileTransferError
+    /* Enqueue a download request, returning a future to the result of
+       the download. The future may throw a DownloadError
        exception. */
-    virtual void enqueueFileTransfer(const FileTransferRequest & request,
-        Callback<FileTransferResult> callback) = 0;
+    virtual void enqueueDownload(const DownloadRequest & request,
+        Callback<DownloadResult> callback) = 0;
 
-    std::future<FileTransferResult> enqueueFileTransfer(const FileTransferRequest & request);
+    std::future<DownloadResult> enqueueDownload(const DownloadRequest & request);
 
     /* Synchronously download a file. */
-    FileTransferResult download(const FileTransferRequest & request);
-
-    /* Synchronously upload a file. */
-    FileTransferResult upload(const FileTransferRequest & request);
+    DownloadResult download(const DownloadRequest & request);
 
     /* Download a file, writing its data to a sink. The sink will be
        invoked on the thread of the caller. */
-    void download(FileTransferRequest && request, Sink & sink);
+    void download(DownloadRequest && request, Sink & sink);
+
+    /* Check if the specified file is already in ~/.cache/nix/tarballs
+       and is more recent than ‘tarball-ttl’ seconds. Otherwise,
+       use the recorded ETag to verify if the server has a more
+       recent version, and if so, download it to the Nix store. */
+    CachedDownloadResult downloadCached(ref<Store> store, const CachedDownloadRequest & request);
 
     enum Error { NotFound, Forbidden, Misc, Transient, Interrupted };
 };
 
-/* Return a shared FileTransfer object. Using this object is preferred
+/* Return a shared Downloader object. Using this object is preferred
    because it enables connection reuse and HTTP/2 multiplexing. */
-ref<FileTransfer> getFileTransfer();
+ref<Downloader> getDownloader();
 
-/* Return a new FileTransfer object. */
-ref<FileTransfer> makeFileTransfer();
+/* Return a new Downloader object. */
+ref<Downloader> makeDownloader();
 
-class FileTransferError : public Error
+class DownloadError : public Error
 {
 public:
-    FileTransfer::Error error;
-    FileTransferError(FileTransfer::Error error, const FormatOrString & fs)
+    Downloader::Error error;
+    DownloadError(Downloader::Error error, const FormatOrString & fs)
         : Error(fs), error(error)
     { }
 };
 
 bool isUri(const string & s);
 
-/* Resolve deprecated 'channel:<foo>' URLs. */
-std::string resolveUri(const std::string & uri);
-
 }

src/libstore/gc.cc

@@ -202,11 +202,6 @@ void LocalStore::findTempRoots(FDs & fds, Roots & tempRoots, bool censor)
     /* Read the `temproots' directory for per-process temporary root
        files. */
     for (auto & i : readDirectory(tempRootsDir)) {
-        if (i.name[0] == '.') {
-            // Ignore hidden files. Some package managers (notably portage) create
-            // those to keep the directory alive.
-            continue;
-        }
         Path path = tempRootsDir + "/" + i.name;
 
         pid_t pid = std::stoi(i.name);
@@ -419,7 +414,7 @@ void LocalStore::findRuntimeRoots(Roots & roots, bool censor)
 
                 try {
                     auto mapFile = fmt("/proc/%s/maps", ent->d_name);
-                    auto mapLines = tokenizeString<std::vector<string>>(readFile(mapFile), "\n");
+                    auto mapLines = tokenizeString<std::vector<string>>(readFile(mapFile, true), "\n");
                     for (const auto & line : mapLines) {
                         auto match = std::smatch{};
                         if (std::regex_match(line, match, mapRegex))
@@ -427,7 +422,7 @@ void LocalStore::findRuntimeRoots(Roots & roots, bool censor)
                     }
 
                     auto envFile = fmt("/proc/%s/environ", ent->d_name);
-                    auto envString = readFile(envFile);
+                    auto envString = readFile(envFile, true);
                     auto env_end = std::sregex_iterator{};
                     for (auto i = std::sregex_iterator{envString.begin(), envString.end(), storePathRegex}; i != env_end; ++i)
                         unchecked[i->str()].emplace(envFile);
@@ -889,7 +884,7 @@ void LocalStore::autoGC(bool sync)
         if (statvfs(realStoreDir.c_str(), &st))
             throw SysError("getting filesystem info about '%s'", realStoreDir);
 
-        return (uint64_t) st.f_bavail * st.f_frsize;
+        return (uint64_t) st.f_bavail * st.f_bsize;
     };
 
     std::shared_future<void> future;

src/libstore/globals.cc

@@ -31,7 +31,6 @@ Settings::Settings()
     , nixLogDir(canonPath(getEnv("NIX_LOG_DIR").value_or(NIX_LOG_DIR)))
     , nixStateDir(canonPath(getEnv("NIX_STATE_DIR").value_or(NIX_STATE_DIR)))
     , nixConfDir(canonPath(getEnv("NIX_CONF_DIR").value_or(NIX_CONF_DIR)))
-    , nixUserConfFiles(getUserConfigFiles())
     , nixLibexecDir(canonPath(getEnv("NIX_LIBEXEC_DIR").value_or(NIX_LIBEXEC_DIR)))
     , nixBinDir(canonPath(getEnv("NIX_BIN_DIR").value_or(NIX_BIN_DIR)))
     , nixManDir(canonPath(NIX_MAN_DIR))
@@ -78,27 +77,11 @@ void loadConfFile()
        ~/.nix/nix.conf or the command line. */
     globalConfig.resetOverriden();
 
-    auto files = settings.nixUserConfFiles;
-    for (auto file = files.rbegin(); file != files.rend(); file++) {
-        globalConfig.applyConfigFile(*file);
-    }
-}
-
-std::vector<Path> getUserConfigFiles()
-{
-    // Use the paths specified in NIX_USER_CONF_FILES if it has been defined
-    auto nixConfFiles = getEnv("NIX_USER_CONF_FILES");
-    if (nixConfFiles.has_value()) {
-        return tokenizeString<std::vector<string>>(nixConfFiles.value(), ":");
-    }
-
-    // Use the paths specified by the XDG spec
-    std::vector<Path> files;
     auto dirs = getConfigDirs();
-    for (auto & dir : dirs) {
-        files.insert(files.end(), dir + "/nix/nix.conf");
+    // Iterate over them in reverse so that the ones appearing first in the path take priority
+    for (auto dir = dirs.rbegin(); dir != dirs.rend(); dir++) {
+        globalConfig.applyConfigFile(*dir + "/nix/nix.conf");
     }
-    return files;
 }
 
 unsigned int Settings::getDefaultCores()
@@ -167,24 +150,21 @@ template<> void BaseSetting<SandboxMode>::toJSON(JSONPlaceholder & out)
 
 template<> void BaseSetting<SandboxMode>::convertToArg(Args & args, const std::string & category)
 {
-    args.addFlag({
-        .longName = name,
-        .description = "Enable sandboxing.",
-        .category = category,
-        .handler = {[=]() { override(smEnabled); }}
-    });
-    args.addFlag({
-        .longName = "no-" + name,
-        .description = "Disable sandboxing.",
-        .category = category,
-        .handler = {[=]() { override(smDisabled); }}
-    });
-    args.addFlag({
-        .longName = "relaxed-" + name,
-        .description = "Enable sandboxing, but allow builds to disable it.",
-        .category = category,
-        .handler = {[=]() { override(smRelaxed); }}
-    });
+    args.mkFlag()
+        .longName(name)
+        .description("Enable sandboxing.")
+        .handler([=](std::vector<std::string> ss) { override(smEnabled); })
+        .category(category);
+    args.mkFlag()
+        .longName("no-" + name)
+        .description("Disable sandboxing.")
+        .handler([=](std::vector<std::string> ss) { override(smDisabled); })
+        .category(category);
+    args.mkFlag()
+        .longName("relaxed-" + name)
+        .description("Enable sandboxing, but allow builds to disable it.")
+        .handler([=](std::vector<std::string> ss) { override(smRelaxed); })
+        .category(category);
 }
 
 void MaxBuildJobsSetting::set(const std::string & str)

src/libstore/globals.hh

@@ -53,12 +53,9 @@ public:
     /* The directory where state is stored. */
     Path nixStateDir;
 
-    /* The directory where system configuration files are stored. */
+    /* The directory where configuration files are stored. */
     Path nixConfDir;
 
-    /* A list of user configuration files to load. */
-    std::vector<Path> nixUserConfFiles;
-
     /* The directory where internal helper programs are stored. */
     Path nixLibexecDir;
 
@@ -354,24 +351,12 @@ public:
     Setting<Paths> pluginFiles{this, {}, "plugin-files",
         "Plugins to dynamically load at nix initialization time."};
 
-    Setting<std::string> githubAccessToken{this, "", "github-access-token",
-        "GitHub access token to get access to GitHub data through the GitHub API for github:<..> flakes."};
-
     Setting<Strings> experimentalFeatures{this, {}, "experimental-features",
         "Experimental Nix features to enable."};
 
     bool isExperimentalFeatureEnabled(const std::string & name);
 
     void requireExperimentalFeature(const std::string & name);
-
-    Setting<bool> allowDirty{this, true, "allow-dirty",
-        "Whether to allow dirty Git/Mercurial trees."};
-
-    Setting<bool> warnDirty{this, true, "warn-dirty",
-        "Whether to warn about dirty Git/Mercurial trees."};
-
-    Setting<std::string> flakeRegistry{this, "https://github.com/NixOS/flake-registry/raw/master/flake-registry.json", "flake-registry",
-        "Path or URI of the global flake registry."};
 };
 
 
@@ -384,9 +369,6 @@ void initPlugins();
 
 void loadConfFile();
 
-// Used by the Settings constructor
-std::vector<Path> getUserConfigFiles();
-
 extern const string nixVersion;
 
 }

src/libstore/http-binary-cache-store.cc

@@ -1,5 +1,5 @@
 #include "binary-cache-store.hh"
-#include "filetransfer.hh"
+#include "download.hh"
 #include "globals.hh"
 #include "nar-info-disk-cache.hh"
 
@@ -85,14 +85,14 @@ protected:
         checkEnabled();
 
         try {
-            FileTransferRequest request(cacheUri + "/" + path);
+            DownloadRequest request(cacheUri + "/" + path);
             request.head = true;
-            getFileTransfer()->download(request);
+            getDownloader()->download(request);
             return true;
-        } catch (FileTransferError & e) {
+        } catch (DownloadError & e) {
             /* S3 buckets return 403 if a file doesn't exist and the
                bucket is unlistable, so treat 403 as 404. */
-            if (e.error == FileTransfer::NotFound || e.error == FileTransfer::Forbidden)
+            if (e.error == Downloader::NotFound || e.error == Downloader::Forbidden)
                 return false;
             maybeDisable();
             throw;
@@ -103,19 +103,19 @@ protected:
         const std::string & data,
         const std::string & mimeType) override
     {
-        auto req = FileTransferRequest(cacheUri + "/" + path);
+        auto req = DownloadRequest(cacheUri + "/" + path);
         req.data = std::make_shared<string>(data); // FIXME: inefficient
         req.mimeType = mimeType;
         try {
-            getFileTransfer()->upload(req);
-        } catch (FileTransferError & e) {
+            getDownloader()->download(req);
+        } catch (DownloadError & e) {
             throw UploadToHTTP("while uploading to HTTP binary cache at '%s': %s", cacheUri, e.msg());
         }
     }
 
-    FileTransferRequest makeRequest(const std::string & path)
+    DownloadRequest makeRequest(const std::string & path)
     {
-        FileTransferRequest request(cacheUri + "/" + path);
+        DownloadRequest request(cacheUri + "/" + path);
         return request;
     }
 
@@ -124,9 +124,9 @@ protected:
         checkEnabled();
         auto request(makeRequest(path));
         try {
-            getFileTransfer()->download(std::move(request), sink);
-        } catch (FileTransferError & e) {
-            if (e.error == FileTransfer::NotFound || e.error == FileTransfer::Forbidden)
+            getDownloader()->download(std::move(request), sink);
+        } catch (DownloadError & e) {
+            if (e.error == Downloader::NotFound || e.error == Downloader::Forbidden)
                 throw NoSuchBinaryCacheFile("file '%s' does not exist in binary cache '%s'", path, getUri());
             maybeDisable();
             throw;
@@ -142,12 +142,12 @@ protected:
 
         auto callbackPtr = std::make_shared<decltype(callback)>(std::move(callback));
 
-        getFileTransfer()->enqueueFileTransfer(request,
-            {[callbackPtr, this](std::future<FileTransferResult> result) {
+        getDownloader()->enqueueDownload(request,
+            {[callbackPtr, this](std::future<DownloadResult> result) {
                 try {
                     (*callbackPtr)(result.get().data);
-                } catch (FileTransferError & e) {
-                    if (e.error == FileTransfer::NotFound || e.error == FileTransfer::Forbidden)
+                } catch (DownloadError & e) {
+                    if (e.error == Downloader::NotFound || e.error == Downloader::Forbidden)
                         return (*callbackPtr)(std::shared_ptr<std::string>());
                     maybeDisable();
                     callbackPtr->rethrow();
@@ -174,3 +174,4 @@ static RegisterStoreImplementation regStore([](
 });
 
 }
+

src/libstore/local-store.cc

@@ -588,7 +588,7 @@ uint64_t LocalStore::addValidPath(State & state,
         (concatStringsSep(" ", info.sigs), !info.sigs.empty())
         (info.ca, !info.ca.empty())
         .exec();
-    uint64_t id = state.db.getLastInsertedRowId();
+    uint64_t id = sqlite3_last_insert_rowid(state.db);
 
     /* If this is a derivation, then store the derivation outputs in
        the database.  This is useful for the garbage collector: it can

src/libstore/local.mk

@@ -61,6 +61,3 @@ $(d)/build.cc:
 clean-files += $(d)/schema.sql.gen.hh
 
 $(eval $(call install-file-in, $(d)/nix-store.pc, $(prefix)/lib/pkgconfig, 0644))
-
-$(foreach i, $(wildcard src/libstore/builtins/*.hh), \
-  $(eval $(call install-file-in, $(i), $(includedir)/nix/builtins, 0644)))

src/libstore/s3-binary-cache-store.cc

@@ -6,7 +6,7 @@
 #include "nar-info-disk-cache.hh"
 #include "globals.hh"
 #include "compression.hh"
-#include "filetransfer.hh"
+#include "download.hh"
 #include "istringstream_nocopy.hh"
 
 #include <aws/core/Aws.h>
@@ -132,7 +132,7 @@ ref<Aws::Client::ClientConfiguration> S3Helper::makeConfig(const string & region
     return res;
 }
 
-S3Helper::FileTransferResult S3Helper::getObject(
+S3Helper::DownloadResult S3Helper::getObject(
     const std::string & bucketName, const std::string & key)
 {
     debug("fetching 's3://%s/%s'...", bucketName, key);
@@ -146,7 +146,7 @@ S3Helper::FileTransferResult S3Helper::getObject(
         return Aws::New<std::stringstream>("STRINGSTREAM");
     });
 
-    FileTransferResult res;
+    DownloadResult res;
 
     auto now1 = std::chrono::steady_clock::now();
 

src/libstore/s3.hh

@@ -18,13 +18,13 @@ struct S3Helper
 
     ref<Aws::Client::ClientConfiguration> makeConfig(const std::string & region, const std::string & scheme, const std::string & endpoint);
 
-    struct FileTransferResult
+    struct DownloadResult
     {
         std::shared_ptr<std::string> data;
         unsigned int durationMs;
     };
 
-    FileTransferResult getObject(
+    DownloadResult getObject(
         const std::string & bucketName, const std::string & key);
 };
 

src/libstore/sqlite.cc

@@ -61,11 +61,6 @@ void SQLite::exec(const std::string & stmt)
     });
 }
 
-uint64_t SQLite::getLastInsertedRowId()
-{
-    return sqlite3_last_insert_rowid(db);
-}
-
 void SQLiteStmt::create(sqlite3 * db, const string & sql)
 {
     checkInterrupt();
@@ -100,10 +95,10 @@ SQLiteStmt::Use::~Use()
     sqlite3_reset(stmt);
 }
 
-SQLiteStmt::Use & SQLiteStmt::Use::operator () (std::string_view value, bool notNull)
+SQLiteStmt::Use & SQLiteStmt::Use::operator () (const std::string & value, bool notNull)
 {
     if (notNull) {
-        if (sqlite3_bind_text(stmt, curArg++, value.data(), -1, SQLITE_TRANSIENT) != SQLITE_OK)
+        if (sqlite3_bind_text(stmt, curArg++, value.c_str(), -1, SQLITE_TRANSIENT) != SQLITE_OK)
             throwSQLiteError(stmt.db, "binding argument");
     } else
         bind();

src/libstore/sqlite.hh

@@ -26,8 +26,6 @@ struct SQLite
     void isCache();
 
     void exec(const std::string & stmt);
-
-    uint64_t getLastInsertedRowId();
 };
 
 /* RAII wrapper to create and destroy SQLite prepared statements. */
@@ -56,7 +54,7 @@ struct SQLiteStmt
         ~Use();
 
         /* Bind the next parameter. */
-        Use & operator () (std::string_view value, bool notNull = true);
+        Use & operator () (const std::string & value, bool notNull = true);
         Use & operator () (const unsigned char * data, size_t len, bool notNull = true);
         Use & operator () (int64_t value, bool notNull = true);
         Use & bind(); // null

src/libstore/store-api.cc

@@ -6,7 +6,6 @@
 #include "thread-pool.hh"
 #include "json.hh"
 #include "derivations.hh"
-#include "url.hh"
 
 #include <future>
 
@@ -41,7 +40,7 @@ Path Store::followLinksToStore(std::string_view _path) const
         path = absPath(target, dirOf(path));
     }
     if (!isInStore(path))
-        throw NotInStore("path '%1%' is not in the Nix store", path);
+        throw Error(format("path '%1%' is not in the Nix store") % path);
     return path;
 }
 
@@ -867,7 +866,27 @@ std::pair<std::string, Store::Params> splitUriAndParams(const std::string & uri_
     Store::Params params;
     auto q = uri.find('?');
     if (q != std::string::npos) {
-        params = decodeQuery(uri.substr(q + 1));
+        for (auto s : tokenizeString<Strings>(uri.substr(q + 1), "&")) {
+            auto e = s.find('=');
+            if (e != std::string::npos) {
+                auto value = s.substr(e + 1);
+                std::string decoded;
+                for (size_t i = 0; i < value.size(); ) {
+                    if (value[i] == '%') {
+                        if (i + 2 >= value.size())
+                            throw Error("invalid URI parameter '%s'", value);
+                        try {
+                            decoded += std::stoul(std::string(value, i + 1, 2), 0, 16);
+                            i += 3;
+                        } catch (...) {
+                            throw Error("invalid URI parameter '%s'", value);
+                        }
+                    } else
+                        decoded += value[i++];
+                }
+                params[s.substr(0, e)] = decoded;
+            }
+        }
         uri = uri_.substr(0, q);
     }
     return {uri, params};

src/libstore/store-api.hh

@@ -28,7 +28,6 @@ MakeError(InvalidPath, Error);
 MakeError(Unsupported, Error);
 MakeError(SubstituteGone, Error);
 MakeError(SubstituterDisabled, Error);
-MakeError(NotInStore, Error);
 
 
 struct BasicDerivation;

src/libutil/ansicolor.hh

@@ -1,15 +0,0 @@
-#pragma once
-
-namespace nix {
-
-/* Some ANSI escape sequences. */
-#define ANSI_NORMAL "\e[0m"
-#define ANSI_BOLD "\e[1m"
-#define ANSI_FAINT "\e[2m"
-#define ANSI_ITALIC "\e[3m"
-#define ANSI_RED "\e[31;1m"
-#define ANSI_GREEN "\e[32;1m"
-#define ANSI_YELLOW "\e[33;1m"
-#define ANSI_BLUE "\e[34;1m"
-
-}

src/libutil/args.cc

@@ -1,32 +1,18 @@
 #include "args.hh"
 #include "hash.hh"
 
-#include <glob.h>
-
 namespace nix {
 
-void Args::addFlag(Flag && flag_)
+Args::FlagMaker Args::mkFlag()
 {
-    auto flag = std::make_shared<Flag>(std::move(flag_));
-    if (flag->handler.arity != ArityAny)
-        assert(flag->handler.arity == flag->labels.size());
-    assert(flag->longName != "");
-    longFlags[flag->longName] = flag;
-    if (flag->shortName) shortFlags[flag->shortName] = flag;
+    return FlagMaker(*this);
 }
 
-bool pathCompletions = false;
-std::shared_ptr<std::set<std::string>> completions;
-
-std::string completionMarker = "___COMPLETE___";
-
-std::optional<std::string> needsCompletion(std::string_view s)
+Args::FlagMaker::~FlagMaker()
 {
-    if (!completions) return {};
-    auto i = s.find(completionMarker);
-    if (i != std::string::npos)
-        return std::string(s.begin(), i);
-    return {};
+    assert(flag->longName != "");
+    args.longFlags[flag->longName] = flag;
+    if (flag->shortName) args.shortFlags[flag->shortName] = flag;
 }
 
 void Args::parseCmdline(const Strings & _cmdline)
@@ -36,14 +22,6 @@ void Args::parseCmdline(const Strings & _cmdline)
 
     Strings cmdline(_cmdline);
 
-    if (auto s = getEnv("NIX_GET_COMPLETIONS")) {
-        size_t n = std::stoi(*s);
-        assert(n > 0 && n <= cmdline.size());
-        *std::next(cmdline.begin(), n - 1) += completionMarker;
-        completions = std::make_shared<decltype(completions)::element_type>();
-        verbosity = lvlError;
-    }
-
     for (auto pos = cmdline.begin(); pos != cmdline.end(); ) {
 
         auto arg = *pos;
@@ -83,22 +61,22 @@ void Args::parseCmdline(const Strings & _cmdline)
 
 void Args::printHelp(const string & programName, std::ostream & out)
 {
-    std::cout << fmt(ANSI_BOLD "Usage:" ANSI_NORMAL " %s " ANSI_ITALIC "FLAGS..." ANSI_NORMAL, programName);
+    std::cout << "Usage: " << programName << " <FLAGS>...";
     for (auto & exp : expectedArgs) {
         std::cout << renderLabels({exp.label});
         // FIXME: handle arity > 1
-        if (exp.handler.arity == ArityAny) std::cout << "...";
+        if (exp.arity == 0) std::cout << "...";
         if (exp.optional) std::cout << "?";
     }
     std::cout << "\n";
 
     auto s = description();
     if (s != "")
-        std::cout << "\n" ANSI_BOLD "Summary:" ANSI_NORMAL " " << s << ".\n";
+        std::cout << "\nSummary: " << s << ".\n";
 
     if (longFlags.size()) {
         std::cout << "\n";
-        std::cout << ANSI_BOLD "Flags:" ANSI_NORMAL "\n";
+        std::cout << "Flags:\n";
         printFlags(out);
     }
 }
@@ -123,28 +101,19 @@ bool Args::processFlag(Strings::iterator & pos, Strings::iterator end)
     auto process = [&](const std::string & name, const Flag & flag) -> bool {
         ++pos;
         std::vector<std::string> args;
-        for (size_t n = 0 ; n < flag.handler.arity; ++n) {
+        for (size_t n = 0 ; n < flag.arity; ++n) {
             if (pos == end) {
-                if (flag.handler.arity == ArityAny) break;
-                throw UsageError("flag '%s' requires %d argument(s)", name, flag.handler.arity);
+                if (flag.arity == ArityAny) break;
+                throw UsageError(format("flag '%1%' requires %2% argument(s)")
+                    % name % flag.arity);
             }
-            if (flag.completer)
-                if (auto prefix = needsCompletion(*pos))
-                    flag.completer(n, *prefix);
             args.push_back(*pos++);
         }
-        flag.handler.fun(std::move(args));
+        flag.handler(std::move(args));
         return true;
     };
 
     if (string(*pos, 0, 2) == "--") {
-        if (auto prefix = needsCompletion(*pos)) {
-            for (auto & [name, flag] : longFlags) {
-                if (!hiddenCategories.count(flag->category)
-                    && hasPrefix(name, std::string(*prefix, 2)))
-                    completions->insert("--" + name);
-            }
-        }
         auto i = longFlags.find(string(*pos, 2));
         if (i == longFlags.end()) return false;
         return process("--" + i->first, *i->second);
@@ -157,14 +126,6 @@ bool Args::processFlag(Strings::iterator & pos, Strings::iterator end)
         return process(std::string("-") + c, *i->second);
     }
 
-    if (auto prefix = needsCompletion(*pos)) {
-        if (prefix == "-") {
-            completions->insert("--");
-            for (auto & [flag, _] : shortFlags)
-                completions->insert(std::string("-") + flag);
-        }
-    }
-
     return false;
 }
 
@@ -180,17 +141,12 @@ bool Args::processArgs(const Strings & args, bool finish)
 
     bool res = false;
 
-    if ((exp.handler.arity == ArityAny && finish) ||
-        (exp.handler.arity != ArityAny && args.size() == exp.handler.arity))
+    if ((exp.arity == 0 && finish) ||
+        (exp.arity > 0 && args.size() == exp.arity))
     {
         std::vector<std::string> ss;
-        for (const auto &[n, s] : enumerate(args)) {
-            ss.push_back(s);
-            if (exp.completer)
-                if (auto prefix = needsCompletion(s))
-                    exp.completer(n, *prefix);
-        }
-        exp.handler.fun(ss);
+        for (auto & s : args) ss.push_back(s);
+        exp.handler(std::move(ss));
         expectedArgs.pop_front();
         res = true;
     }
@@ -201,54 +157,17 @@ bool Args::processArgs(const Strings & args, bool finish)
     return res;
 }
 
-Args::Flag Args::Flag::mkHashTypeFlag(std::string && longName, HashType * ht)
+Args::FlagMaker & Args::FlagMaker::mkHashTypeFlag(HashType * ht)
 {
-    return Flag {
-        .longName = std::move(longName),
-        .description = "hash algorithm ('md5', 'sha1', 'sha256', or 'sha512')",
-        .labels = {"hash-algo"},
-        .handler = {[ht](std::string s) {
-            *ht = parseHashType(s);
-            if (*ht == htUnknown)
-                throw UsageError("unknown hash type '%1%'", s);
-        }},
-        .completer = [](size_t index, std::string_view prefix) {
-            for (auto & type : hashTypes)
-                if (hasPrefix(type, prefix))
-                    completions->insert(type);
-        }
-    };
-}
-
-static void completePath(std::string_view prefix, bool onlyDirs)
-{
-    pathCompletions = true;
-    glob_t globbuf;
-    int flags = GLOB_NOESCAPE | GLOB_TILDE;
-    #ifdef GLOB_ONLYDIR
-    if (onlyDirs)
-        flags |= GLOB_ONLYDIR;
-    #endif
-    if (glob((std::string(prefix) + "*").c_str(), flags, nullptr, &globbuf) == 0) {
-        for (size_t i = 0; i < globbuf.gl_pathc; ++i) {
-            if (onlyDirs) {
-                auto st = lstat(globbuf.gl_pathv[i]);
-                if (!S_ISDIR(st.st_mode)) continue;
-            }
-            completions->insert(globbuf.gl_pathv[i]);
-        }
-        globfree(&globbuf);
-    }
-}
-
-void completePath(size_t, std::string_view prefix)
-{
-    completePath(prefix, false);
-}
-
-void completeDir(size_t, std::string_view prefix)
-{
-    completePath(prefix, true);
+    arity(1);
+    label("type");
+    description("hash algorithm ('md5', 'sha1', 'sha256', or 'sha512')");
+    handler([ht](std::string s) {
+        *ht = parseHashType(s);
+        if (*ht == htUnknown)
+            throw UsageError("unknown hash type '%1%'", s);
+    });
+    return *this;
 }
 
 Strings argvToStrings(int argc, char * * argv)
@@ -264,7 +183,7 @@ std::string renderLabels(const Strings & labels)
     std::string res;
     for (auto label : labels) {
         for (auto & c : label) c = std::toupper(c);
-        res += " " ANSI_ITALIC + label + ANSI_NORMAL;
+        res += " <" + label + ">";
     }
     return res;
 }
@@ -273,10 +192,10 @@ void printTable(std::ostream & out, const Table2 & table)
 {
     size_t max = 0;
     for (auto & row : table)
-        max = std::max(max, filterANSIEscapes(row.first, true).size());
+        max = std::max(max, row.first.size());
     for (auto & row : table) {
         out << "  " << row.first
-            << std::string(max - filterANSIEscapes(row.first, true).size() + 2, ' ')
+            << std::string(max - row.first.size() + 2, ' ')
             << row.second << "\n";
     }
 }
@@ -287,7 +206,8 @@ void Command::printHelp(const string & programName, std::ostream & out)
 
     auto exs = examples();
     if (!exs.empty()) {
-        out << "\n" ANSI_BOLD "Examples:" ANSI_NORMAL "\n";
+        out << "\n";
+        out << "Examples:\n";
         for (auto & ex : exs)
             out << "\n"
                 << "  " << ex.description << "\n" // FIXME: wrap
@@ -298,69 +218,54 @@ void Command::printHelp(const string & programName, std::ostream & out)
 MultiCommand::MultiCommand(const Commands & commands)
     : commands(commands)
 {
-    expectArgs({
-        .label = "command",
-        .optional = true,
-        .handler = {[=](std::string s) {
-            assert(!command);
-            if (auto prefix = needsCompletion(s)) {
-                for (auto & [name, command] : commands)
-                    if (hasPrefix(name, *prefix))
-                        completions->insert(name);
-            }
-            auto i = commands.find(s);
-            if (i == commands.end())
-                throw UsageError("'%s' is not a recognised command", s);
-            command = {s, i->second()};
-        }}
-    });
-
-    categories[Command::catDefault] = "Available commands";
+    expectedArgs.push_back(ExpectedArg{"command", 1, true, [=](std::vector<std::string> ss) {
+        assert(!command);
+        auto i = commands.find(ss[0]);
+        if (i == commands.end())
+            throw UsageError("'%s' is not a recognised command", ss[0]);
+        command = i->second();
+        command->_name = ss[0];
+    }});
 }
 
 void MultiCommand::printHelp(const string & programName, std::ostream & out)
 {
     if (command) {
-        command->second->printHelp(programName + " " + command->first, out);
+        command->printHelp(programName + " " + command->name(), out);
         return;
     }
 
-    out << fmt(ANSI_BOLD "Usage:" ANSI_NORMAL " %s " ANSI_ITALIC "COMMAND FLAGS... ARGS..." ANSI_NORMAL "\n", programName);
+    out << "Usage: " << programName << " <COMMAND> <FLAGS>... <ARGS>...\n";
 
-    out << "\n" ANSI_BOLD "Common flags:" ANSI_NORMAL "\n";
+    out << "\n";
+    out << "Common flags:\n";
     printFlags(out);
 
-    std::map<Command::Category, std::map<std::string, ref<Command>>> commandsByCategory;
+    out << "\n";
+    out << "Available commands:\n";
 
-    for (auto & [name, commandFun] : commands) {
-        auto command = commandFun();
-        commandsByCategory[command->category()].insert_or_assign(name, command);
-    }
-
-    for (auto & [category, commands] : commandsByCategory) {
-        out << fmt("\n" ANSI_BOLD "%s:" ANSI_NORMAL "\n", categories[category]);
-
-        Table2 table;
-        for (auto & [name, command] : commands) {
-            auto descr = command->description();
-            if (!descr.empty())
-                table.push_back(std::make_pair(name, descr));
-        }
-        printTable(out, table);
+    Table2 table;
+    for (auto & i : commands) {
+        auto command = i.second();
+        command->_name = i.first;
+        auto descr = command->description();
+        if (!descr.empty())
+            table.push_back(std::make_pair(command->name(), descr));
     }
+    printTable(out, table);
 }
 
 bool MultiCommand::processFlag(Strings::iterator & pos, Strings::iterator end)
 {
     if (Args::processFlag(pos, end)) return true;
-    if (command && command->second->processFlag(pos, end)) return true;
+    if (command && command->processFlag(pos, end)) return true;
     return false;
 }
 
 bool MultiCommand::processArgs(const Strings & args, bool finish)
 {
     if (command)
-        return command->second->processArgs(args, finish);
+        return command->processArgs(args, finish);
     else
         return Args::processArgs(args, finish);
 }

src/libutil/args.hh

@@ -28,69 +28,17 @@ protected:
 
     static const size_t ArityAny = std::numeric_limits<size_t>::max();
 
-    struct Handler
-    {
-        std::function<void(std::vector<std::string>)> fun;
-        size_t arity;
-
-        Handler() {}
-
-        Handler(std::function<void(std::vector<std::string>)> && fun)
-            : fun(std::move(fun))
-            , arity(ArityAny)
-        { }
-
-        Handler(std::function<void()> && handler)
-            : fun([handler{std::move(handler)}](std::vector<std::string>) { handler(); })
-            , arity(0)
-        { }
-
-        Handler(std::function<void(std::string)> && handler)
-            : fun([handler{std::move(handler)}](std::vector<std::string> ss) {
-                handler(std::move(ss[0]));
-              })
-            , arity(1)
-        { }
-
-        Handler(std::function<void(std::string, std::string)> && handler)
-            : fun([handler{std::move(handler)}](std::vector<std::string> ss) {
-                handler(std::move(ss[0]), std::move(ss[1]));
-              })
-            , arity(2)
-        { }
-
-        Handler(std::vector<std::string> * dest)
-            : fun([=](std::vector<std::string> ss) { *dest = ss; })
-            , arity(ArityAny)
-        { }
-
-        template<class T>
-        Handler(T * dest)
-            : fun([=](std::vector<std::string> ss) { *dest = ss[0]; })
-            , arity(1)
-        { }
-
-        template<class T>
-        Handler(T * dest, const T & val)
-            : fun([=](std::vector<std::string> ss) { *dest = val; })
-            , arity(0)
-        { }
-    };
-
     /* Flags. */
     struct Flag
     {
         typedef std::shared_ptr<Flag> ptr;
-
         std::string longName;
         char shortName = 0;
         std::string description;
-        std::string category;
         Strings labels;
-        Handler handler;
-        std::function<void(size_t, std::string_view)> completer;
-
-        static Flag mkHashTypeFlag(std::string && longName, HashType * ht);
+        size_t arity = 0;
+        std::function<void(std::vector<std::string>)> handler;
+        std::string category;
     };
 
     std::map<std::string, Flag::ptr> longFlags;
@@ -104,9 +52,9 @@ protected:
     struct ExpectedArg
     {
         std::string label;
-        bool optional = false;
-        Handler handler;
-        std::function<void(size_t, std::string_view)> completer;
+        size_t arity; // 0 = any
+        bool optional;
+        std::function<void(std::vector<std::string>)> handler;
     };
 
     std::list<ExpectedArg> expectedArgs;
@@ -117,7 +65,49 @@ protected:
 
 public:
 
-    void addFlag(Flag && flag);
+    class FlagMaker
+    {
+        Args & args;
+        Flag::ptr flag;
+        friend class Args;
+        FlagMaker(Args & args) : args(args), flag(std::make_shared<Flag>()) { }
+    public:
+        ~FlagMaker();
+        FlagMaker & longName(const std::string & s) { flag->longName = s; return *this; }
+        FlagMaker & shortName(char s) { flag->shortName = s; return *this; }
+        FlagMaker & description(const std::string & s) { flag->description = s; return *this; }
+        FlagMaker & label(const std::string & l) { flag->arity = 1; flag->labels = {l}; return *this; }
+        FlagMaker & labels(const Strings & ls) { flag->arity = ls.size(); flag->labels = ls; return *this; }
+        FlagMaker & arity(size_t arity) { flag->arity = arity; return *this; }
+        FlagMaker & handler(std::function<void(std::vector<std::string>)> handler) { flag->handler = handler; return *this; }
+        FlagMaker & handler(std::function<void()> handler) { flag->handler = [handler](std::vector<std::string>) { handler(); }; return *this; }
+        FlagMaker & handler(std::function<void(std::string)> handler) {
+            flag->arity = 1;
+            flag->handler = [handler](std::vector<std::string> ss) { handler(std::move(ss[0])); };
+            return *this;
+        }
+        FlagMaker & category(const std::string & s) { flag->category = s; return *this; }
+
+        template<class T>
+        FlagMaker & dest(T * dest)
+        {
+            flag->arity = 1;
+            flag->handler = [=](std::vector<std::string> ss) { *dest = ss[0]; };
+            return *this;
+        }
+
+        template<class T>
+        FlagMaker & set(T * dest, const T & val)
+        {
+            flag->arity = 0;
+            flag->handler = [=](std::vector<std::string> ss) { *dest = val; };
+            return *this;
+        }
+
+        FlagMaker & mkHashTypeFlag(HashType * ht);
+    };
+
+    FlagMaker mkFlag();
 
     /* Helper functions for constructing flags / positional
        arguments. */
@@ -126,13 +116,13 @@ public:
         const std::string & label, const std::string & description,
         std::function<void(std::string)> fun)
     {
-        addFlag({
-            .longName = longName,
-            .shortName = shortName,
-            .description = description,
-            .labels = {label},
-            .handler = {[=](std::string s) { fun(s); }}
-        });
+        mkFlag()
+            .shortName(shortName)
+            .longName(longName)
+            .labels({label})
+            .description(description)
+            .arity(1)
+            .handler([=](std::vector<std::string> ss) { fun(ss[0]); });
     }
 
     void mkFlag(char shortName, const std::string & name,
@@ -145,12 +135,11 @@ public:
     void mkFlag(char shortName, const std::string & longName, const std::string & description,
         T * dest, const T & value)
     {
-        addFlag({
-            .longName = longName,
-            .shortName = shortName,
-            .description = description,
-            .handler = {[=]() { *dest = value; }}
-        });
+        mkFlag()
+            .shortName(shortName)
+            .longName(longName)
+            .description(description)
+            .handler([=](std::vector<std::string> ss) { *dest = value; });
     }
 
     template<class I>
@@ -166,42 +155,34 @@ public:
     void mkFlag(char shortName, const std::string & longName,
         const std::string & description, std::function<void(I)> fun)
     {
-        addFlag({
-            .longName = longName,
-            .shortName = shortName,
-            .description = description,
-            .labels = {"N"},
-            .handler = {[=](std::string s) {
+        mkFlag()
+            .shortName(shortName)
+            .longName(longName)
+            .labels({"N"})
+            .description(description)
+            .arity(1)
+            .handler([=](std::vector<std::string> ss) {
                 I n;
-                if (!string2Int(s, n))
+                if (!string2Int(ss[0], n))
                     throw UsageError("flag '--%s' requires a integer argument", longName);
                 fun(n);
-            }}
-        });
-    }
-
-    void expectArgs(ExpectedArg && arg)
-    {
-        expectedArgs.emplace_back(std::move(arg));
+            });
     }
 
     /* Expect a string argument. */
     void expectArg(const std::string & label, string * dest, bool optional = false)
     {
-        expectArgs({
-            .label = label,
-            .optional = true,
-            .handler = {dest}
-        });
+        expectedArgs.push_back(ExpectedArg{label, 1, optional, [=](std::vector<std::string> ss) {
+            *dest = ss[0];
+        }});
     }
 
     /* Expect 0 or more arguments. */
     void expectArgs(const std::string & label, std::vector<std::string> * dest)
     {
-        expectArgs({
-            .label = label,
-            .handler = {dest}
-        });
+        expectedArgs.push_back(ExpectedArg{label, 0, false, [=](std::vector<std::string> ss) {
+            *dest = std::move(ss);
+        }});
     }
 
     friend class MultiCommand;
@@ -211,10 +192,17 @@ public:
    run() method. */
 struct Command : virtual Args
 {
+private:
+    std::string _name;
+
     friend class MultiCommand;
 
+public:
+
     virtual ~Command() { }
 
+    std::string name() { return _name; }
+
     virtual void prepare() { };
     virtual void run() = 0;
 
@@ -228,12 +216,6 @@ struct Command : virtual Args
 
     virtual Examples examples() { return Examples(); }
 
-    typedef int Category;
-
-    static constexpr Category catDefault = 0;
-
-    virtual Category category() { return catDefault; }
-
     void printHelp(const string & programName, std::ostream & out) override;
 };
 
@@ -246,10 +228,7 @@ class MultiCommand : virtual Args
 public:
     Commands commands;
 
-    std::map<Command::Category, std::string> categories;
-
-    // Selected command, if any.
-    std::optional<std::pair<std::string, ref<Command>>> command;
+    std::shared_ptr<Command> command;
 
     MultiCommand(const Commands & commands);
 
@@ -270,13 +249,4 @@ typedef std::vector<std::pair<std::string, std::string>> Table2;
 
 void printTable(std::ostream & out, const Table2 & table);
 
-extern std::shared_ptr<std::set<std::string>> completions;
-extern bool pathCompletions;
-
-std::optional<std::string> needsCompletion(std::string_view s);
-
-void completePath(size_t, std::string_view prefix);
-
-void completeDir(size_t, std::string_view prefix);
-
 }

src/libutil/config.cc

@@ -177,13 +177,12 @@ void BaseSetting<T>::toJSON(JSONPlaceholder & out)
 template<typename T>
 void BaseSetting<T>::convertToArg(Args & args, const std::string & category)
 {
-    args.addFlag({
-        .longName = name,
-        .description = description,
-        .category = category,
-        .labels = {"value"},
-        .handler = {[=](std::string s) { overriden = true; set(s); }},
-    });
+    args.mkFlag()
+        .longName(name)
+        .description(description)
+        .arity(1)
+        .handler([=](std::vector<std::string> ss) { overriden = true; set(ss[0]); })
+        .category(category);
 }
 
 template<> void BaseSetting<std::string>::set(const std::string & str)
@@ -228,18 +227,16 @@ template<> std::string BaseSetting<bool>::to_string() const
 
 template<> void BaseSetting<bool>::convertToArg(Args & args, const std::string & category)
 {
-    args.addFlag({
-        .longName = name,
-        .description = description,
-        .category = category,
-        .handler = {[=]() { override(true); }}
-    });
-    args.addFlag({
-        .longName = "no-" + name,
-        .description = description,
-        .category = category,
-        .handler = {[=]() { override(false); }}
-    });
+    args.mkFlag()
+        .longName(name)
+        .description(description)
+        .handler([=](std::vector<std::string> ss) { override(true); })
+        .category(category);
+    args.mkFlag()
+        .longName("no-" + name)
+        .description(description)
+        .handler([=](std::vector<std::string> ss) { override(false); })
+        .category(category);
 }
 
 template<> void BaseSetting<Strings>::set(const std::string & str)

src/libutil/error.cc

@@ -1,146 +0,0 @@
-#include "error.hh"
-
-#include <iostream>
-#include <optional>
-
-namespace nix
-{
-
-std::optional<string> ErrorInfo::programName = std::nullopt;
-
-std::ostream& operator<<(std::ostream &os, const hintformat &hf)
-{
-    return os << hf.str();
-}
-
-string showErrPos(const ErrPos &errPos)
-{
-    if (errPos.column > 0) {
-        return fmt("(%1%:%2%)", errPos.lineNumber, errPos.column);
-    } else {
-        return fmt("(%1%)", errPos.lineNumber);
-    };
-}
-
-void printCodeLines(const string &prefix, const NixCode &nixCode)
-{
-    // previous line of code.
-    if (nixCode.prevLineOfCode.has_value()) {
-        std::cout << fmt("%1% %|2$5d|| %3%",
-                         prefix,
-                         (nixCode.errPos.lineNumber - 1),
-                         *nixCode.prevLineOfCode)
-                  << std::endl;
-    }
-
-    // line of code containing the error.%2$+5d%
-    std::cout << fmt("%1% %|2$5d|| %3%",
-                     prefix,
-                     (nixCode.errPos.lineNumber),
-                     nixCode.errLineOfCode)
-              << std::endl;
-
-    // error arrows for the column range.
-    if (nixCode.errPos.column > 0) {
-        int start = nixCode.errPos.column;
-        std::string spaces;
-        for (int i = 0; i < start; ++i) {
-            spaces.append(" ");
-        }
-
-        std::string arrows("^");
-
-        std::cout << fmt("%1%      |%2%" ANSI_RED "%3%" ANSI_NORMAL,
-                         prefix,
-                         spaces,
-                         arrows) << std::endl;
-    }
-
-    // next line of code.
-    if (nixCode.nextLineOfCode.has_value()) {
-        std::cout << fmt("%1% %|2$5d|| %3%",
-                         prefix,
-                         (nixCode.errPos.lineNumber + 1),
-                         *nixCode.nextLineOfCode)
-                  << std::endl;
-    }
-}
-
-void printErrorInfo(const ErrorInfo &einfo)
-{
-    int errwidth = 80;
-    string prefix = "    ";
-
-    string levelString;
-    switch (einfo.level) {
-    case ErrLevel::elError: {
-        levelString = ANSI_RED;
-        levelString += "error:";
-        levelString += ANSI_NORMAL;
-        break;
-    }
-    case ErrLevel::elWarning: {
-        levelString = ANSI_YELLOW;
-        levelString += "warning:";
-        levelString += ANSI_NORMAL;
-        break;
-    }
-    default: {
-        levelString = fmt("invalid error level: %1%", einfo.level);
-        break;
-    }
-    }
-
-    int ndl = prefix.length() + levelString.length() + 3 + einfo.name.length() + einfo.programName.value_or("").length();
-    int dashwidth = ndl > (errwidth - 3) ? 3 : errwidth - ndl;
-
-    string dashes;
-    for (int i = 0; i < dashwidth; ++i)
-        dashes.append("-");
-
-    // divider.
-    std::cout << fmt("%1%%2%" ANSI_BLUE " %3% %4% %5% %6%" ANSI_NORMAL,
-                     prefix,
-                     levelString,
-                     "---",
-                     einfo.name,
-                     dashes,
-                     einfo.programName.value_or(""))
-              << std::endl;
-
-    // filename.
-    if (einfo.nixCode.has_value()) {
-        if (einfo.nixCode->errPos.nixFile != "") {
-            string eline = einfo.nixCode->errLineOfCode != ""
-                           ? string(" ") + showErrPos(einfo.nixCode->errPos)
-                           : "";
-
-            std::cout << fmt("%1%in file: " ANSI_BLUE "%2%%3%" ANSI_NORMAL,
-                             prefix, 
-                             einfo.nixCode->errPos.nixFile,
-                             eline) << std::endl;
-            std::cout << prefix << std::endl;
-        } else {
-            std::cout << fmt("%1%from command line argument", prefix) << std::endl;
-            std::cout << prefix << std::endl;
-        }
-    }
-
-    // description
-    std::cout << prefix << einfo.description << std::endl;
-    std::cout << prefix << std::endl;
-
-    // lines of code.
-    if (einfo.nixCode->errLineOfCode != "") {
-        printCodeLines(prefix, *einfo.nixCode);
-        std::cout << prefix << std::endl;
-    }
-
-    // hint
-    if (einfo.hint.has_value()) {
-        std::cout << prefix << *einfo.hint << std::endl;
-        std::cout << prefix << std::endl;
-    }
-}
-
-}

src/libutil/error.hh

@@ -1,121 +0,0 @@
-#ifndef error_hh
-#define error_hh
-
-#include "ansicolor.hh"
-#include <string>
-#include <optional>
-#include <iostream>
-#include "types.hh"
-
-namespace nix
-{
-
-typedef enum {
-    elWarning,
-    elError
-} ErrLevel;
-
-struct ErrPos
-{
-    int lineNumber;
-    int column;
-    string nixFile;
-
-    template <class P>
-    ErrPos& operator=(const P &pos)
-    {
-        lineNumber = pos.line;
-        column = pos.column;
-        nixFile = pos.file;
-        return *this;
-    }
-
-    template <class P>
-    ErrPos(const P &p)
-    {
-      *this = p;
-    }
-};
-
-struct NixCode
-{
-    ErrPos errPos;
-    std::optional<string> prevLineOfCode;
-    string errLineOfCode;
-    std::optional<string> nextLineOfCode;
-};
-
-// ----------------------------------------------------------------
-// format function for hints.  same as fmt, except templated values
-// are always in yellow.
-
-template <class T>
-struct yellowify
-{
-    yellowify(T &s) : value(s) {}
-    T &value;
-};
-
-template <class T>
-std::ostream& operator<<(std::ostream &out, const yellowify<T> &y)
-{
-    return out << ANSI_YELLOW << y.value << ANSI_NORMAL;
-}
-
-class hintformat
-{
-public:
-    hintformat(string format) :fmt(format)
-    {
-        fmt.exceptions(boost::io::all_error_bits ^ boost::io::too_many_args_bit);
-    }
-    template<class T>
-    hintformat& operator%(const T &value)
-    {
-        fmt % yellowify(value);
-        return *this;
-    }
-
-    std::string str() const
-    {
-        return fmt.str();
-    }
-
-    template <typename U>
-    friend class AddHint;
-private:
-    format fmt;
-};
-
-std::ostream& operator<<(std::ostream &os, const hintformat &hf);
-
-template<typename... Args>
-inline hintformat hintfmt(const std::string & fs, const Args & ... args)
-{
-    hintformat f(fs);
-    formatHelper(f, args...);
-    return f;
-}
-
-// -------------------------------------------------
-// ErrorInfo.
-struct ErrorInfo
-{
-    ErrLevel level;
-    string name;
-    string description;
-    std::optional<hintformat> hint;
-    std::optional<NixCode> nixCode;
-
-    static std::optional<string> programName;
-};
-
-// --------------------------------------------------------
-// error printing
-
-// just to cout for now.
-void printErrorInfo(const ErrorInfo &einfo);
-
-}
-
-#endif

src/libutil/hash.cc

@@ -16,9 +16,6 @@
 namespace nix {
 
 
-std::set<std::string> hashTypes = { "md5", "sha1", "sha256", "sha512" };
-
-
 void Hash::init()
 {
     if (type == htMD5) hashSize = md5HashSize;

src/libutil/hash.hh

@@ -18,8 +18,6 @@ const int sha1HashSize = 20;
 const int sha256HashSize = 32;
 const int sha512HashSize = 64;
 
-extern std::set<std::string> hashTypes;
-
 extern const string base32Chars;
 
 enum Base : int { Base64, Base32, Base16, SRI };

src/libutil/logging.cc

@@ -3,7 +3,6 @@
 
 #include <atomic>
 #include <nlohmann/json.hpp>
-#include <iostream>
 
 namespace nix {
 
@@ -25,11 +24,6 @@ void Logger::warn(const std::string & msg)
     log(lvlWarn, ANSI_YELLOW "warning:" ANSI_NORMAL " " + msg);
 }
 
-void Logger::writeToStdout(std::string_view s)
-{
-    std::cout << s << "\n";
-}
-
 class SimpleLogger : public Logger
 {
 public:
@@ -204,7 +198,7 @@ bool handleJSONLogMessage(const std::string & msg,
 
         if (action == "start") {
             auto type = (ActivityType) json["type"];
-            if (trusted || type == actFileTransfer)
+            if (trusted || type == actDownload)
                 activities.emplace(std::piecewise_construct,
                     std::forward_as_tuple(json["id"]),
                     std::forward_as_tuple(*logger, (Verbosity) json["level"], type,

src/libutil/logging.hh

@@ -17,7 +17,7 @@ typedef enum {
 typedef enum {
     actUnknown = 0,
     actCopyPath = 100,
-    actFileTransfer = 101,
+    actDownload = 101,
     actRealise = 102,
     actCopyPaths = 103,
     actBuilds = 104,
@@ -78,16 +78,6 @@ public:
     virtual void stopActivity(ActivityId act) { };
 
     virtual void result(ActivityId act, ResultType type, const Fields & fields) { };
-
-    virtual void writeToStdout(std::string_view s);
-
-    template<typename... Args>
-    inline void stdout(const std::string & fs, const Args & ... args)
-    {
-        boost::format f(fs);
-        formatHelper(f, args...);
-        writeToStdout(f.str());
-    }
 };
 
 ActivityId getCurActivity();

src/libutil/serialise.hh

@@ -148,9 +148,6 @@ struct StringSink : Sink
 {
     ref<std::string> s;
     StringSink() : s(make_ref<std::string>()) { };
-    explicit StringSink(const size_t reservedSize) : s(make_ref<std::string>()) {
-      s->reserve(reservedSize);
-    };
     StringSink(ref<std::string> s) : s(s) { };
     void operator () (const unsigned char * data, size_t len) override;
 };