Fix nix flake show --legacy when the evaluation fails

`nix flake show --legacy` was aborting the evaluation a bit too eagerly
in case of error, so that running it on nixpkgs was only showing empty
packages sets.
Catch the error earlier to allow displaying the full set of packages

.github/STALE-BOT.md

@@ -0,0 +1,35 @@
+# Stale bot information
+
+- Thanks for your contribution!
+- To remove the stale label, just leave a new comment.
+- _How to find the right people to ping?_ → [`git blame`](https://git-scm.com/docs/git-blame) to the rescue! (or GitHub's history and blame buttons.)
+- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on the [#nixos IRC channel](https://webchat.freenode.net/#nixos).
+
+## Suggestions for PRs
+
+1. GitHub sometimes doesn't notify people who commented / reviewed a PR previously, when you (force) push commits. If you have addressed the reviews you can [officially ask for a review](https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from those who commented to you or anyone else.
+2. If it is unfinished but you plan to finish it, please mark it as a draft.
+3. If you don't expect to work on it any time soon, closing it with a short comment may encourage someone else to pick up your work.
+4. To get things rolling again, rebase the PR against the target branch and address valid comments.
+5. If you need a review to move forward, ask in [the Discourse thread for PRs that need help](https://discourse.nixos.org/t/prs-in-distress/3604).
+6. If all you need is a merge, check the git history to find and [request reviews](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from people who usually merge related contributions.
+
+## Suggestions for issues
+
+1. If it is resolved (either for you personally, or in general), please consider closing it.
+2. If this might still be an issue, but you are not interested in promoting its resolution, please consider closing it while encouraging others to take over and reopen an issue if they care enough.
+3. If you still have interest in resolving it, try to ping somebody who you believe might have an interest in the topic. Consider discussing the problem in [our Discourse Forum](https://discourse.nixos.org/).
+4. As with all open source projects, your best option is to submit a Pull Request that addresses this issue. We :heart: this attitude!
+
+**Memorandum on closing issues**
+
+Don't be afraid to close an issue that holds valuable information. Closed issues stay in the system for people to search, read, cross-reference, or even reopen--nothing is lost! Closing obsolete issues is an important way to help maintainers focus their time and effort.
+
+## Useful GitHub search queries
+
+- [Open PRs with any stale-bot interaction](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+)
+- [Open PRs with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22)
+- [Open PRs with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)
+- [Open Issues with any stale-bot interaction](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+)
+- [Open Issues with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22+)
+- [Open Issues with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)

.github/stale.yml

@@ -0,0 +1,10 @@
+# Configuration for probot-stale - https://github.com/probot/stale
+daysUntilStale: 180
+daysUntilClose: 365
+exemptLabels:
+  - "critical"
+staleLabel: "stale"
+markComment: |
+  I marked this as stale due to inactivity. → [More info](https://github.com/NixOS/nix/blob/master/.github/STALE-BOT.md)
+closeComment: |
+  I closed this issue due to inactivity. → [More info](https://github.com/NixOS/nix/blob/master/.github/STALE-BOT.md)

.github/workflows/test.yml

@@ -12,6 +12,6 @@ jobs:
     - uses: actions/checkout@v2
       with:
         fetch-depth: 0
-    - uses: cachix/install-nix-action@v11
+    - uses: cachix/install-nix-action@v12
     #- run: nix flake check
     - run: nix-build -A checks.$(if [[ `uname` = Linux ]]; then echo x86_64-linux; else echo x86_64-darwin; fi)

.gitignore

@@ -18,13 +18,13 @@ perl/Makefile.config
 /doc/manual/nix.json
 /doc/manual/conf-file.json
 /doc/manual/builtins.json
-/doc/manual/src/command-ref/nix.md
+/doc/manual/src/SUMMARY.md
+/doc/manual/src/command-ref/new-cli
 /doc/manual/src/command-ref/conf-file.md
 /doc/manual/src/expressions/builtins.md
 
 # /scripts/
 /scripts/nix-profile.sh
-/scripts/nix-copy-closure
 /scripts/nix-reduce-build
 /scripts/nix-http-export.cgi
 /scripts/nix-profile-daemon.sh

.version

@@ -1 +1 @@
-3.0
+2.4

Makefile

@@ -7,10 +7,10 @@ makefiles = \
   src/libfetchers/local.mk \
   src/libmain/local.mk \
   src/libexpr/local.mk \
+  src/libcmd/local.mk \
   src/nix/local.mk \
   src/resolve-system-dependencies/local.mk \
   scripts/local.mk \
-  corepkgs/local.mk \
   misc/bash/local.mk \
   misc/systemd/local.mk \
   misc/launchd/local.mk \

Makefile.config.in

@@ -10,7 +10,6 @@ EDITLINE_LIBS = @EDITLINE_LIBS@
 ENABLE_S3 = @ENABLE_S3@
 GTEST_LIBS = @GTEST_LIBS@
 HAVE_SECCOMP = @HAVE_SECCOMP@
-HAVE_SODIUM = @HAVE_SODIUM@
 LDFLAGS = @LDFLAGS@
 LIBARCHIVE_LIBS = @LIBARCHIVE_LIBS@
 LIBBROTLI_LIBS = @LIBBROTLI_LIBS@

README.md

@@ -20,7 +20,7 @@ Information on additional installation methods is available on the [Nix download
 
 ## Building And Developing
 
-See our [Hacking guide](https://hydra.nixos.org/job/nix/master/build.x86_64-linux/latest/download-by-type/doc/manual/hacking.html) in our manual for instruction on how to
+See our [Hacking guide](https://hydra.nixos.org/job/nix/master/build.x86_64-linux/latest/download-by-type/doc/manual/contributing/hacking.html) in our manual for instruction on how to
 build nix from source with nix-build or how to get a development environment.
 
 ## Additional Resources

config/config.guess

@@ -1,8 +1,8 @@
 #! /bin/sh
 # Attempt to guess a canonical system name.
-#   Copyright 1992-2018 Free Software Foundation, Inc.
+#   Copyright 1992-2020 Free Software Foundation, Inc.
 
-timestamp='2018-08-02'
+timestamp='2020-11-19'
 
 # This file is free software; you can redistribute it and/or modify it
 # under the terms of the GNU General Public License as published by
@@ -27,12 +27,12 @@ timestamp='2018-08-02'
 # Originally written by Per Bothner; maintained since 2000 by Ben Elliston.
 #
 # You can get the latest version of this script from:
-# https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess
+# https://git.savannah.gnu.org/cgit/config.git/plain/config.guess
 #
 # Please send patches to <config-patches@gnu.org>.
 
 
-me=`echo "$0" | sed -e 's,.*/,,'`
+me=$(echo "$0" | sed -e 's,.*/,,')
 
 usage="\
 Usage: $0 [OPTION]
@@ -50,7 +50,7 @@ version="\
 GNU config.guess ($timestamp)
 
 Originally written by Per Bothner.
-Copyright 1992-2018 Free Software Foundation, Inc.
+Copyright 1992-2020 Free Software Foundation, Inc.
 
 This is free software; see the source for copying conditions.  There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
@@ -96,13 +96,14 @@ fi
 
 tmp=
 # shellcheck disable=SC2172
-trap 'test -z "$tmp" || rm -fr "$tmp"' 1 2 13 15
-trap 'exitcode=$?; test -z "$tmp" || rm -fr "$tmp"; exit $exitcode' 0
+trap 'test -z "$tmp" || rm -fr "$tmp"' 0 1 2 13 15
 
 set_cc_for_build() {
+    # prevent multiple calls if $tmp is already set
+    test "$tmp" && return 0
     : "${TMPDIR=/tmp}"
     # shellcheck disable=SC2039
-    { tmp=`(umask 077 && mktemp -d "$TMPDIR/cgXXXXXX") 2>/dev/null` && test -n "$tmp" && test -d "$tmp" ; } ||
+    { tmp=$( (umask 077 && mktemp -d "$TMPDIR/cgXXXXXX") 2>/dev/null) && test -n "$tmp" && test -d "$tmp" ; } ||
 	{ test -n "$RANDOM" && tmp=$TMPDIR/cg$$-$RANDOM && (umask 077 && mkdir "$tmp" 2>/dev/null) ; } ||
 	{ tmp=$TMPDIR/cg-$$ && (umask 077 && mkdir "$tmp" 2>/dev/null) && echo "Warning: creating insecure temp directory" >&2 ; } ||
 	{ echo "$me: cannot create a temporary directory in $TMPDIR" >&2 ; exit 1 ; }
@@ -130,16 +131,14 @@ if test -f /.attbin/uname ; then
 	PATH=$PATH:/.attbin ; export PATH
 fi
 
-UNAME_MACHINE=`(uname -m) 2>/dev/null` || UNAME_MACHINE=unknown
-UNAME_RELEASE=`(uname -r) 2>/dev/null` || UNAME_RELEASE=unknown
-UNAME_SYSTEM=`(uname -s) 2>/dev/null`  || UNAME_SYSTEM=unknown
-UNAME_VERSION=`(uname -v) 2>/dev/null` || UNAME_VERSION=unknown
+UNAME_MACHINE=$( (uname -m) 2>/dev/null) || UNAME_MACHINE=unknown
+UNAME_RELEASE=$( (uname -r) 2>/dev/null) || UNAME_RELEASE=unknown
+UNAME_SYSTEM=$( (uname -s) 2>/dev/null) || UNAME_SYSTEM=unknown
+UNAME_VERSION=$( (uname -v) 2>/dev/null) || UNAME_VERSION=unknown
 
 case "$UNAME_SYSTEM" in
 Linux|GNU|GNU/*)
-	# If the system lacks a compiler, then just pick glibc.
-	# We could probably try harder.
-	LIBC=gnu
+	LIBC=unknown
 
 	set_cc_for_build
 	cat <<-EOF > "$dummy.c"
@@ -148,17 +147,29 @@ Linux|GNU|GNU/*)
 	LIBC=uclibc
 	#elif defined(__dietlibc__)
 	LIBC=dietlibc
-	#else
+	#elif defined(__GLIBC__)
 	LIBC=gnu
+	#else
+	#include <stdarg.h>
+	/* First heuristic to detect musl libc.  */
+	#ifdef __DEFINED_va_list
+	LIBC=musl
+	#endif
 	#endif
 	EOF
-	eval "`$CC_FOR_BUILD -E "$dummy.c" 2>/dev/null | grep '^LIBC' | sed 's, ,,g'`"
+	eval "$($CC_FOR_BUILD -E "$dummy.c" 2>/dev/null | grep '^LIBC' | sed 's, ,,g')"
 
-	# If ldd exists, use it to detect musl libc.
-	if command -v ldd >/dev/null && \
-		ldd --version 2>&1 | grep -q ^musl
-	then
-	    LIBC=musl
+	# Second heuristic to detect musl libc.
+	if [ "$LIBC" = unknown ] &&
+	   command -v ldd >/dev/null &&
+	   ldd --version 2>&1 | grep -q ^musl; then
+		LIBC=musl
+	fi
+
+	# If the system lacks a compiler, then just pick glibc.
+	# We could probably try harder.
+	if [ "$LIBC" = unknown ]; then
+		LIBC=gnu
 	fi
 	;;
 esac
@@ -178,19 +189,20 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	# Note: NetBSD doesn't particularly care about the vendor
 	# portion of the name.  We always set it to "unknown".
 	sysctl="sysctl -n hw.machine_arch"
-	UNAME_MACHINE_ARCH=`(uname -p 2>/dev/null || \
+	UNAME_MACHINE_ARCH=$( (uname -p 2>/dev/null || \
 	    "/sbin/$sysctl" 2>/dev/null || \
 	    "/usr/sbin/$sysctl" 2>/dev/null || \
-	    echo unknown)`
+	    echo unknown))
 	case "$UNAME_MACHINE_ARCH" in
+	    aarch64eb) machine=aarch64_be-unknown ;;
 	    armeb) machine=armeb-unknown ;;
 	    arm*) machine=arm-unknown ;;
 	    sh3el) machine=shl-unknown ;;
 	    sh3eb) machine=sh-unknown ;;
 	    sh5el) machine=sh5le-unknown ;;
 	    earmv*)
-		arch=`echo "$UNAME_MACHINE_ARCH" | sed -e 's,^e\(armv[0-9]\).*$,\1,'`
-		endian=`echo "$UNAME_MACHINE_ARCH" | sed -ne 's,^.*\(eb\)$,\1,p'`
+		arch=$(echo "$UNAME_MACHINE_ARCH" | sed -e 's,^e\(armv[0-9]\).*$,\1,')
+		endian=$(echo "$UNAME_MACHINE_ARCH" | sed -ne 's,^.*\(eb\)$,\1,p')
 		machine="${arch}${endian}"-unknown
 		;;
 	    *) machine="$UNAME_MACHINE_ARCH"-unknown ;;
@@ -221,7 +233,7 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	case "$UNAME_MACHINE_ARCH" in
 	    earm*)
 		expr='s/^earmv[0-9]/-eabi/;s/eb$//'
-		abi=`echo "$UNAME_MACHINE_ARCH" | sed -e "$expr"`
+		abi=$(echo "$UNAME_MACHINE_ARCH" | sed -e "$expr")
 		;;
 	esac
 	# The OS release
@@ -234,7 +246,7 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 		release='-gnu'
 		;;
 	    *)
-		release=`echo "$UNAME_RELEASE" | sed -e 's/[-_].*//' | cut -d. -f1,2`
+		release=$(echo "$UNAME_RELEASE" | sed -e 's/[-_].*//' | cut -d. -f1,2)
 		;;
 	esac
 	# Since CPU_TYPE-MANUFACTURER-KERNEL-OPERATING_SYSTEM:
@@ -243,15 +255,15 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	echo "$machine-${os}${release}${abi-}"
 	exit ;;
     *:Bitrig:*:*)
-	UNAME_MACHINE_ARCH=`arch | sed 's/Bitrig.//'`
+	UNAME_MACHINE_ARCH=$(arch | sed 's/Bitrig.//')
 	echo "$UNAME_MACHINE_ARCH"-unknown-bitrig"$UNAME_RELEASE"
 	exit ;;
     *:OpenBSD:*:*)
-	UNAME_MACHINE_ARCH=`arch | sed 's/OpenBSD.//'`
+	UNAME_MACHINE_ARCH=$(arch | sed 's/OpenBSD.//')
 	echo "$UNAME_MACHINE_ARCH"-unknown-openbsd"$UNAME_RELEASE"
 	exit ;;
     *:LibertyBSD:*:*)
-	UNAME_MACHINE_ARCH=`arch | sed 's/^.*BSD\.//'`
+	UNAME_MACHINE_ARCH=$(arch | sed 's/^.*BSD\.//')
 	echo "$UNAME_MACHINE_ARCH"-unknown-libertybsd"$UNAME_RELEASE"
 	exit ;;
     *:MidnightBSD:*:*)
@@ -263,6 +275,9 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
     *:SolidBSD:*:*)
 	echo "$UNAME_MACHINE"-unknown-solidbsd"$UNAME_RELEASE"
 	exit ;;
+    *:OS108:*:*)
+	echo "$UNAME_MACHINE"-unknown-os108_"$UNAME_RELEASE"
+	exit ;;
     macppc:MirBSD:*:*)
 	echo powerpc-unknown-mirbsd"$UNAME_RELEASE"
 	exit ;;
@@ -272,26 +287,29 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
     *:Sortix:*:*)
 	echo "$UNAME_MACHINE"-unknown-sortix
 	exit ;;
+    *:Twizzler:*:*)
+	echo "$UNAME_MACHINE"-unknown-twizzler
+	exit ;;
     *:Redox:*:*)
 	echo "$UNAME_MACHINE"-unknown-redox
 	exit ;;
     mips:OSF1:*.*)
-        echo mips-dec-osf1
-        exit ;;
+	echo mips-dec-osf1
+	exit ;;
     alpha:OSF1:*:*)
 	case $UNAME_RELEASE in
 	*4.0)
-		UNAME_RELEASE=`/usr/sbin/sizer -v | awk '{print $3}'`
+		UNAME_RELEASE=$(/usr/sbin/sizer -v | awk '{print $3}')
 		;;
 	*5.*)
-		UNAME_RELEASE=`/usr/sbin/sizer -v | awk '{print $4}'`
+		UNAME_RELEASE=$(/usr/sbin/sizer -v | awk '{print $4}')
 		;;
 	esac
 	# According to Compaq, /usr/sbin/psrinfo has been available on
 	# OSF/1 and Tru64 systems produced since 1995.  I hope that
 	# covers most systems running today.  This code pipes the CPU
 	# types through head -n 1, so we only detect the type of CPU 0.
-	ALPHA_CPU_TYPE=`/usr/sbin/psrinfo -v | sed -n -e 's/^  The alpha \(.*\) processor.*$/\1/p' | head -n 1`
+	ALPHA_CPU_TYPE=$(/usr/sbin/psrinfo -v | sed -n -e 's/^  The alpha \(.*\) processor.*$/\1/p' | head -n 1)
 	case "$ALPHA_CPU_TYPE" in
 	    "EV4 (21064)")
 		UNAME_MACHINE=alpha ;;
@@ -329,7 +347,7 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	# A Tn.n version is a released field test version.
 	# A Xn.n version is an unreleased experimental baselevel.
 	# 1.2 uses "1.2" for uname -r.
-	echo "$UNAME_MACHINE"-dec-osf"`echo "$UNAME_RELEASE" | sed -e 's/^[PVTX]//' | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz`"
+	echo "$UNAME_MACHINE"-dec-osf"$(echo "$UNAME_RELEASE" | sed -e 's/^[PVTX]//' | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz)"
 	# Reset EXIT trap before exiting to avoid spurious non-zero exit code.
 	exitcode=$?
 	trap '' 0
@@ -363,7 +381,7 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	exit ;;
     Pyramid*:OSx*:*:* | MIS*:OSx*:*:* | MIS*:SMP_DC-OSx*:*:*)
 	# akee@wpdis03.wpafb.af.mil (Earle F. Ake) contributed MIS and NILE.
-	if test "`(/bin/universe) 2>/dev/null`" = att ; then
+	if test "$( (/bin/universe) 2>/dev/null)" = att ; then
 		echo pyramid-pyramid-sysv3
 	else
 		echo pyramid-pyramid-bsd
@@ -376,54 +394,59 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	echo sparc-icl-nx6
 	exit ;;
     DRS?6000:UNIX_SV:4.2*:7* | DRS?6000:isis:4.2*:7*)
-	case `/usr/bin/uname -p` in
+	case $(/usr/bin/uname -p) in
 	    sparc) echo sparc-icl-nx7; exit ;;
 	esac ;;
     s390x:SunOS:*:*)
-	echo "$UNAME_MACHINE"-ibm-solaris2"`echo "$UNAME_RELEASE" | sed -e 's/[^.]*//'`"
+	echo "$UNAME_MACHINE"-ibm-solaris2"$(echo "$UNAME_RELEASE" | sed -e 's/[^.]*//')"
 	exit ;;
     sun4H:SunOS:5.*:*)
-	echo sparc-hal-solaris2"`echo "$UNAME_RELEASE"|sed -e 's/[^.]*//'`"
+	echo sparc-hal-solaris2"$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*//')"
 	exit ;;
     sun4*:SunOS:5.*:* | tadpole*:SunOS:5.*:*)
-	echo sparc-sun-solaris2"`echo "$UNAME_RELEASE" | sed -e 's/[^.]*//'`"
+	echo sparc-sun-solaris2"$(echo "$UNAME_RELEASE" | sed -e 's/[^.]*//')"
 	exit ;;
     i86pc:AuroraUX:5.*:* | i86xen:AuroraUX:5.*:*)
 	echo i386-pc-auroraux"$UNAME_RELEASE"
 	exit ;;
     i86pc:SunOS:5.*:* | i86xen:SunOS:5.*:*)
-	UNAME_REL="`echo "$UNAME_RELEASE" | sed -e 's/[^.]*//'`"
-	case `isainfo -b` in
-	    32)
-		echo i386-pc-solaris2"$UNAME_REL"
-		;;
-	    64)
-		echo x86_64-pc-solaris2"$UNAME_REL"
-		;;
-	esac
+	set_cc_for_build
+	SUN_ARCH=i386
+	# If there is a compiler, see if it is configured for 64-bit objects.
+	# Note that the Sun cc does not turn __LP64__ into 1 like gcc does.
+	# This test works for both compilers.
+	if test "$CC_FOR_BUILD" != no_compiler_found; then
+	    if (echo '#ifdef __amd64'; echo IS_64BIT_ARCH; echo '#endif') | \
+		(CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
+		grep IS_64BIT_ARCH >/dev/null
+	    then
+		SUN_ARCH=x86_64
+	    fi
+	fi
+	echo "$SUN_ARCH"-pc-solaris2"$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*//')"
 	exit ;;
     sun4*:SunOS:6*:*)
 	# According to config.sub, this is the proper way to canonicalize
 	# SunOS6.  Hard to guess exactly what SunOS6 will be like, but
 	# it's likely to be more like Solaris than SunOS4.
-	echo sparc-sun-solaris3"`echo "$UNAME_RELEASE"|sed -e 's/[^.]*//'`"
+	echo sparc-sun-solaris3"$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*//')"
 	exit ;;
     sun4*:SunOS:*:*)
-	case "`/usr/bin/arch -k`" in
+	case "$(/usr/bin/arch -k)" in
 	    Series*|S4*)
-		UNAME_RELEASE=`uname -v`
+		UNAME_RELEASE=$(uname -v)
 		;;
 	esac
 	# Japanese Language versions have a version number like `4.1.3-JL'.
-	echo sparc-sun-sunos"`echo "$UNAME_RELEASE"|sed -e 's/-/_/'`"
+	echo sparc-sun-sunos"$(echo "$UNAME_RELEASE"|sed -e 's/-/_/')"
 	exit ;;
     sun3*:SunOS:*:*)
 	echo m68k-sun-sunos"$UNAME_RELEASE"
 	exit ;;
     sun*:*:4.2BSD:*)
-	UNAME_RELEASE=`(sed 1q /etc/motd | awk '{print substr($5,1,3)}') 2>/dev/null`
+	UNAME_RELEASE=$( (sed 1q /etc/motd | awk '{print substr($5,1,3)}') 2>/dev/null)
 	test "x$UNAME_RELEASE" = x && UNAME_RELEASE=3
-	case "`/bin/arch`" in
+	case "$(/bin/arch)" in
 	    sun3)
 		echo m68k-sun-sunos"$UNAME_RELEASE"
 		;;
@@ -503,8 +526,8 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	}
 EOF
 	$CC_FOR_BUILD -o "$dummy" "$dummy.c" &&
-	  dummyarg=`echo "$UNAME_RELEASE" | sed -n 's/\([0-9]*\).*/\1/p'` &&
-	  SYSTEM_NAME=`"$dummy" "$dummyarg"` &&
+	  dummyarg=$(echo "$UNAME_RELEASE" | sed -n 's/\([0-9]*\).*/\1/p') &&
+	  SYSTEM_NAME=$("$dummy" "$dummyarg") &&
 	    { echo "$SYSTEM_NAME"; exit; }
 	echo mips-mips-riscos"$UNAME_RELEASE"
 	exit ;;
@@ -531,11 +554,11 @@ EOF
 	exit ;;
     AViiON:dgux:*:*)
 	# DG/UX returns AViiON for all architectures
-	UNAME_PROCESSOR=`/usr/bin/uname -p`
-	if [ "$UNAME_PROCESSOR" = mc88100 ] || [ "$UNAME_PROCESSOR" = mc88110 ]
+	UNAME_PROCESSOR=$(/usr/bin/uname -p)
+	if test "$UNAME_PROCESSOR" = mc88100 || test "$UNAME_PROCESSOR" = mc88110
 	then
-	    if [ "$TARGET_BINARY_INTERFACE"x = m88kdguxelfx ] || \
-	       [ "$TARGET_BINARY_INTERFACE"x = x ]
+	    if test "$TARGET_BINARY_INTERFACE"x = m88kdguxelfx || \
+	       test "$TARGET_BINARY_INTERFACE"x = x
 	    then
 		echo m88k-dg-dgux"$UNAME_RELEASE"
 	    else
@@ -559,17 +582,17 @@ EOF
 	echo m68k-tektronix-bsd
 	exit ;;
     *:IRIX*:*:*)
-	echo mips-sgi-irix"`echo "$UNAME_RELEASE"|sed -e 's/-/_/g'`"
+	echo mips-sgi-irix"$(echo "$UNAME_RELEASE"|sed -e 's/-/_/g')"
 	exit ;;
     ????????:AIX?:[12].1:2)   # AIX 2.2.1 or AIX 2.1.1 is RT/PC AIX.
 	echo romp-ibm-aix     # uname -m gives an 8 hex-code CPU id
-	exit ;;               # Note that: echo "'`uname -s`'" gives 'AIX '
+	exit ;;               # Note that: echo "'$(uname -s)'" gives 'AIX '
     i*86:AIX:*:*)
 	echo i386-ibm-aix
 	exit ;;
     ia64:AIX:*:*)
-	if [ -x /usr/bin/oslevel ] ; then
-		IBM_REV=`/usr/bin/oslevel`
+	if test -x /usr/bin/oslevel ; then
+		IBM_REV=$(/usr/bin/oslevel)
 	else
 		IBM_REV="$UNAME_VERSION.$UNAME_RELEASE"
 	fi
@@ -589,7 +612,7 @@ EOF
 			exit(0);
 			}
 EOF
-		if $CC_FOR_BUILD -o "$dummy" "$dummy.c" && SYSTEM_NAME=`"$dummy"`
+		if $CC_FOR_BUILD -o "$dummy" "$dummy.c" && SYSTEM_NAME=$("$dummy")
 		then
 			echo "$SYSTEM_NAME"
 		else
@@ -602,15 +625,15 @@ EOF
 	fi
 	exit ;;
     *:AIX:*:[4567])
-	IBM_CPU_ID=`/usr/sbin/lsdev -C -c processor -S available | sed 1q | awk '{ print $1 }'`
+	IBM_CPU_ID=$(/usr/sbin/lsdev -C -c processor -S available | sed 1q | awk '{ print $1 }')
 	if /usr/sbin/lsattr -El "$IBM_CPU_ID" | grep ' POWER' >/dev/null 2>&1; then
 		IBM_ARCH=rs6000
 	else
 		IBM_ARCH=powerpc
 	fi
-	if [ -x /usr/bin/lslpp ] ; then
-		IBM_REV=`/usr/bin/lslpp -Lqc bos.rte.libc |
-			   awk -F: '{ print $3 }' | sed s/[0-9]*$/0/`
+	if test -x /usr/bin/lslpp ; then
+		IBM_REV=$(/usr/bin/lslpp -Lqc bos.rte.libc |
+			   awk -F: '{ print $3 }' | sed s/[0-9]*$/0/)
 	else
 		IBM_REV="$UNAME_VERSION.$UNAME_RELEASE"
 	fi
@@ -638,14 +661,14 @@ EOF
 	echo m68k-hp-bsd4.4
 	exit ;;
     9000/[34678]??:HP-UX:*:*)
-	HPUX_REV=`echo "$UNAME_RELEASE"|sed -e 's/[^.]*.[0B]*//'`
+	HPUX_REV=$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*.[0B]*//')
 	case "$UNAME_MACHINE" in
 	    9000/31?)            HP_ARCH=m68000 ;;
 	    9000/[34]??)         HP_ARCH=m68k ;;
 	    9000/[678][0-9][0-9])
-		if [ -x /usr/bin/getconf ]; then
-		    sc_cpu_version=`/usr/bin/getconf SC_CPU_VERSION 2>/dev/null`
-		    sc_kernel_bits=`/usr/bin/getconf SC_KERNEL_BITS 2>/dev/null`
+		if test -x /usr/bin/getconf; then
+		    sc_cpu_version=$(/usr/bin/getconf SC_CPU_VERSION 2>/dev/null)
+		    sc_kernel_bits=$(/usr/bin/getconf SC_KERNEL_BITS 2>/dev/null)
 		    case "$sc_cpu_version" in
 		      523) HP_ARCH=hppa1.0 ;; # CPU_PA_RISC1_0
 		      528) HP_ARCH=hppa1.1 ;; # CPU_PA_RISC1_1
@@ -657,7 +680,7 @@ EOF
 			esac ;;
 		    esac
 		fi
-		if [ "$HP_ARCH" = "" ]; then
+		if test "$HP_ARCH" = ""; then
 		    set_cc_for_build
 		    sed 's/^		//' << EOF > "$dummy.c"
 
@@ -692,11 +715,11 @@ EOF
 		    exit (0);
 		}
 EOF
-		    (CCOPTS="" $CC_FOR_BUILD -o "$dummy" "$dummy.c" 2>/dev/null) && HP_ARCH=`"$dummy"`
+		    (CCOPTS="" $CC_FOR_BUILD -o "$dummy" "$dummy.c" 2>/dev/null) && HP_ARCH=$("$dummy")
 		    test -z "$HP_ARCH" && HP_ARCH=hppa
 		fi ;;
 	esac
-	if [ "$HP_ARCH" = hppa2.0w ]
+	if test "$HP_ARCH" = hppa2.0w
 	then
 	    set_cc_for_build
 
@@ -720,7 +743,7 @@ EOF
 	echo "$HP_ARCH"-hp-hpux"$HPUX_REV"
 	exit ;;
     ia64:HP-UX:*:*)
-	HPUX_REV=`echo "$UNAME_RELEASE"|sed -e 's/[^.]*.[0B]*//'`
+	HPUX_REV=$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*.[0B]*//')
 	echo ia64-hp-hpux"$HPUX_REV"
 	exit ;;
     3050*:HI-UX:*:*)
@@ -750,7 +773,7 @@ EOF
 	  exit (0);
 	}
 EOF
-	$CC_FOR_BUILD -o "$dummy" "$dummy.c" && SYSTEM_NAME=`"$dummy"` &&
+	$CC_FOR_BUILD -o "$dummy" "$dummy.c" && SYSTEM_NAME=$("$dummy") &&
 		{ echo "$SYSTEM_NAME"; exit; }
 	echo unknown-hitachi-hiuxwe2
 	exit ;;
@@ -770,7 +793,7 @@ EOF
 	echo hppa1.0-hp-osf
 	exit ;;
     i*86:OSF1:*:*)
-	if [ -x /usr/sbin/sysversion ] ; then
+	if test -x /usr/sbin/sysversion ; then
 	    echo "$UNAME_MACHINE"-unknown-osf1mk
 	else
 	    echo "$UNAME_MACHINE"-unknown-osf1
@@ -819,14 +842,14 @@ EOF
 	echo craynv-cray-unicosmp"$UNAME_RELEASE" | sed -e 's/\.[^.]*$/.X/'
 	exit ;;
     F30[01]:UNIX_System_V:*:* | F700:UNIX_System_V:*:*)
-	FUJITSU_PROC=`uname -m | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz`
-	FUJITSU_SYS=`uname -p | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/\///'`
-	FUJITSU_REL=`echo "$UNAME_RELEASE" | sed -e 's/ /_/'`
+	FUJITSU_PROC=$(uname -m | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz)
+	FUJITSU_SYS=$(uname -p | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/\///')
+	FUJITSU_REL=$(echo "$UNAME_RELEASE" | sed -e 's/ /_/')
 	echo "${FUJITSU_PROC}-fujitsu-${FUJITSU_SYS}${FUJITSU_REL}"
 	exit ;;
     5000:UNIX_System_V:4.*:*)
-	FUJITSU_SYS=`uname -p | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/\///'`
-	FUJITSU_REL=`echo "$UNAME_RELEASE" | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/ /_/'`
+	FUJITSU_SYS=$(uname -p | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/\///')
+	FUJITSU_REL=$(echo "$UNAME_RELEASE" | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/ /_/')
 	echo "sparc-fujitsu-${FUJITSU_SYS}${FUJITSU_REL}"
 	exit ;;
     i*86:BSD/386:*:* | i*86:BSD/OS:*:* | *:Ascend\ Embedded/OS:*:*)
@@ -838,26 +861,26 @@ EOF
     *:BSD/OS:*:*)
 	echo "$UNAME_MACHINE"-unknown-bsdi"$UNAME_RELEASE"
 	exit ;;
-    arm*:FreeBSD:*:*)
-	UNAME_PROCESSOR=`uname -p`
+    arm:FreeBSD:*:*)
+	UNAME_PROCESSOR=$(uname -p)
 	set_cc_for_build
 	if echo __ARM_PCS_VFP | $CC_FOR_BUILD -E - 2>/dev/null \
 	    | grep -q __ARM_PCS_VFP
 	then
-	    echo "${UNAME_PROCESSOR}"-unknown-freebsd"`echo ${UNAME_RELEASE}|sed -e 's/[-(].*//'`"-gnueabi
+	    echo "${UNAME_PROCESSOR}"-unknown-freebsd"$(echo ${UNAME_RELEASE}|sed -e 's/[-(].*//')"-gnueabi
 	else
-	    echo "${UNAME_PROCESSOR}"-unknown-freebsd"`echo ${UNAME_RELEASE}|sed -e 's/[-(].*//'`"-gnueabihf
+	    echo "${UNAME_PROCESSOR}"-unknown-freebsd"$(echo ${UNAME_RELEASE}|sed -e 's/[-(].*//')"-gnueabihf
 	fi
 	exit ;;
     *:FreeBSD:*:*)
-	UNAME_PROCESSOR=`/usr/bin/uname -p`
+	UNAME_PROCESSOR=$(/usr/bin/uname -p)
 	case "$UNAME_PROCESSOR" in
 	    amd64)
 		UNAME_PROCESSOR=x86_64 ;;
 	    i386)
 		UNAME_PROCESSOR=i586 ;;
 	esac
-	echo "$UNAME_PROCESSOR"-unknown-freebsd"`echo "$UNAME_RELEASE"|sed -e 's/[-(].*//'`"
+	echo "$UNAME_PROCESSOR"-unknown-freebsd"$(echo "$UNAME_RELEASE"|sed -e 's/[-(].*//')"
 	exit ;;
     i*:CYGWIN*:*)
 	echo "$UNAME_MACHINE"-pc-cygwin
@@ -890,18 +913,18 @@ EOF
 	echo "$UNAME_MACHINE"-pc-uwin
 	exit ;;
     amd64:CYGWIN*:*:* | x86_64:CYGWIN*:*:*)
-	echo x86_64-unknown-cygwin
+	echo x86_64-pc-cygwin
 	exit ;;
     prep*:SunOS:5.*:*)
-	echo powerpcle-unknown-solaris2"`echo "$UNAME_RELEASE"|sed -e 's/[^.]*//'`"
+	echo powerpcle-unknown-solaris2"$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*//')"
 	exit ;;
     *:GNU:*:*)
 	# the GNU system
-	echo "`echo "$UNAME_MACHINE"|sed -e 's,[-/].*$,,'`-unknown-$LIBC`echo "$UNAME_RELEASE"|sed -e 's,/.*$,,'`"
+	echo "$(echo "$UNAME_MACHINE"|sed -e 's,[-/].*$,,')-unknown-$LIBC$(echo "$UNAME_RELEASE"|sed -e 's,/.*$,,')"
 	exit ;;
     *:GNU/*:*:*)
 	# other systems with GNU libc and userland
-	echo "$UNAME_MACHINE-unknown-`echo "$UNAME_SYSTEM" | sed 's,^[^/]*/,,' | tr "[:upper:]" "[:lower:]"``echo "$UNAME_RELEASE"|sed -e 's/[-(].*//'`-$LIBC"
+	echo "$UNAME_MACHINE-unknown-$(echo "$UNAME_SYSTEM" | sed 's,^[^/]*/,,' | tr "[:upper:]" "[:lower:]")$(echo "$UNAME_RELEASE"|sed -e 's/[-(].*//')-$LIBC"
 	exit ;;
     *:Minix:*:*)
 	echo "$UNAME_MACHINE"-unknown-minix
@@ -914,7 +937,7 @@ EOF
 	echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
 	exit ;;
     alpha:Linux:*:*)
-	case `sed -n '/^cpu model/s/^.*: \(.*\)/\1/p' < /proc/cpuinfo` in
+	case $(sed -n '/^cpu model/s/^.*: \(.*\)/\1/p' /proc/cpuinfo 2>/dev/null) in
 	  EV5)   UNAME_MACHINE=alphaev5 ;;
 	  EV56)  UNAME_MACHINE=alphaev56 ;;
 	  PCA56) UNAME_MACHINE=alphapca56 ;;
@@ -981,22 +1004,50 @@ EOF
 	exit ;;
     mips:Linux:*:* | mips64:Linux:*:*)
 	set_cc_for_build
+	IS_GLIBC=0
+	test x"${LIBC}" = xgnu && IS_GLIBC=1
 	sed 's/^	//' << EOF > "$dummy.c"
 	#undef CPU
-	#undef ${UNAME_MACHINE}
-	#undef ${UNAME_MACHINE}el
+	#undef mips
+	#undef mipsel
+	#undef mips64
+	#undef mips64el
+	#if ${IS_GLIBC} && defined(_ABI64)
+	LIBCABI=gnuabi64
+	#else
+	#if ${IS_GLIBC} && defined(_ABIN32)
+	LIBCABI=gnuabin32
+	#else
+	LIBCABI=${LIBC}
+	#endif
+	#endif
+
+	#if ${IS_GLIBC} && defined(__mips64) && defined(__mips_isa_rev) && __mips_isa_rev>=6
+	CPU=mipsisa64r6
+	#else
+	#if ${IS_GLIBC} && !defined(__mips64) && defined(__mips_isa_rev) && __mips_isa_rev>=6
+	CPU=mipsisa32r6
+	#else
+	#if defined(__mips64)
+	CPU=mips64
+	#else
+	CPU=mips
+	#endif
+	#endif
+	#endif
+
 	#if defined(__MIPSEL__) || defined(__MIPSEL) || defined(_MIPSEL) || defined(MIPSEL)
-	CPU=${UNAME_MACHINE}el
+	MIPS_ENDIAN=el
 	#else
 	#if defined(__MIPSEB__) || defined(__MIPSEB) || defined(_MIPSEB) || defined(MIPSEB)
-	CPU=${UNAME_MACHINE}
+	MIPS_ENDIAN=
 	#else
-	CPU=
+	MIPS_ENDIAN=
 	#endif
 	#endif
 EOF
-	eval "`$CC_FOR_BUILD -E "$dummy.c" 2>/dev/null | grep '^CPU'`"
-	test "x$CPU" != x && { echo "$CPU-unknown-linux-$LIBC"; exit; }
+	eval "$($CC_FOR_BUILD -E "$dummy.c" 2>/dev/null | grep '^CPU\|^MIPS_ENDIAN\|^LIBCABI')"
+	test "x$CPU" != x && { echo "$CPU${MIPS_ENDIAN}-unknown-linux-$LIBCABI"; exit; }
 	;;
     mips64el:Linux:*:*)
 	echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
@@ -1015,7 +1066,7 @@ EOF
 	exit ;;
     parisc:Linux:*:* | hppa:Linux:*:*)
 	# Look for CPU level
-	case `grep '^cpu[^a-z]*:' /proc/cpuinfo 2>/dev/null | cut -d' ' -f2` in
+	case $(grep '^cpu[^a-z]*:' /proc/cpuinfo 2>/dev/null | cut -d' ' -f2) in
 	  PA7*) echo hppa1.1-unknown-linux-"$LIBC" ;;
 	  PA8*) echo hppa2.0-unknown-linux-"$LIBC" ;;
 	  *)    echo hppa-unknown-linux-"$LIBC" ;;
@@ -1055,7 +1106,17 @@ EOF
 	echo "$UNAME_MACHINE"-dec-linux-"$LIBC"
 	exit ;;
     x86_64:Linux:*:*)
-	echo "$UNAME_MACHINE"-pc-linux-"$LIBC"
+	set_cc_for_build
+	LIBCABI=$LIBC
+	if test "$CC_FOR_BUILD" != no_compiler_found; then
+	    if (echo '#ifdef __ILP32__'; echo IS_X32; echo '#endif') | \
+		(CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
+		grep IS_X32 >/dev/null
+	    then
+		LIBCABI="$LIBC"x32
+	    fi
+	fi
+	echo "$UNAME_MACHINE"-pc-linux-"$LIBCABI"
 	exit ;;
     xtensa*:Linux:*:*)
 	echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
@@ -1095,7 +1156,7 @@ EOF
 	echo "$UNAME_MACHINE"-pc-msdosdjgpp
 	exit ;;
     i*86:*:4.*:*)
-	UNAME_REL=`echo "$UNAME_RELEASE" | sed 's/\/MP$//'`
+	UNAME_REL=$(echo "$UNAME_RELEASE" | sed 's/\/MP$//')
 	if grep Novell /usr/include/link.h >/dev/null 2>/dev/null; then
 		echo "$UNAME_MACHINE"-univel-sysv"$UNAME_REL"
 	else
@@ -1104,19 +1165,19 @@ EOF
 	exit ;;
     i*86:*:5:[678]*)
 	# UnixWare 7.x, OpenUNIX and OpenServer 6.
-	case `/bin/uname -X | grep "^Machine"` in
+	case $(/bin/uname -X | grep "^Machine") in
 	    *486*)	     UNAME_MACHINE=i486 ;;
 	    *Pentium)	     UNAME_MACHINE=i586 ;;
 	    *Pent*|*Celeron) UNAME_MACHINE=i686 ;;
 	esac
-	echo "$UNAME_MACHINE-unknown-sysv${UNAME_RELEASE}${UNAME_SYSTEM}{$UNAME_VERSION}"
+	echo "$UNAME_MACHINE-unknown-sysv${UNAME_RELEASE}${UNAME_SYSTEM}${UNAME_VERSION}"
 	exit ;;
     i*86:*:3.2:*)
 	if test -f /usr/options/cb.name; then
-		UNAME_REL=`sed -n 's/.*Version //p' </usr/options/cb.name`
+		UNAME_REL=$(sed -n 's/.*Version //p' </usr/options/cb.name)
 		echo "$UNAME_MACHINE"-pc-isc"$UNAME_REL"
 	elif /bin/uname -X 2>/dev/null >/dev/null ; then
-		UNAME_REL=`(/bin/uname -X|grep Release|sed -e 's/.*= //')`
+		UNAME_REL=$( (/bin/uname -X|grep Release|sed -e 's/.*= //'))
 		(/bin/uname -X|grep i80486 >/dev/null) && UNAME_MACHINE=i486
 		(/bin/uname -X|grep '^Machine.*Pentium' >/dev/null) \
 			&& UNAME_MACHINE=i586
@@ -1166,7 +1227,7 @@ EOF
     3[345]??:*:4.0:3.0 | 3[34]??A:*:4.0:3.0 | 3[34]??,*:*:4.0:3.0 | 3[34]??/*:*:4.0:3.0 | 4400:*:4.0:3.0 | 4850:*:4.0:3.0 | SKA40:*:4.0:3.0 | SDS2:*:4.0:3.0 | SHG2:*:4.0:3.0 | S7501*:*:4.0:3.0)
 	OS_REL=''
 	test -r /etc/.relid \
-	&& OS_REL=.`sed -n 's/[^ ]* [^ ]* \([0-9][0-9]\).*/\1/p' < /etc/.relid`
+	&& OS_REL=.$(sed -n 's/[^ ]* [^ ]* \([0-9][0-9]\).*/\1/p' < /etc/.relid)
 	/bin/uname -p 2>/dev/null | grep 86 >/dev/null \
 	  && { echo i486-ncr-sysv4.3"$OS_REL"; exit; }
 	/bin/uname -p 2>/dev/null | /bin/grep entium >/dev/null \
@@ -1177,7 +1238,7 @@ EOF
     NCR*:*:4.2:* | MPRAS*:*:4.2:*)
 	OS_REL='.3'
 	test -r /etc/.relid \
-	    && OS_REL=.`sed -n 's/[^ ]* [^ ]* \([0-9][0-9]\).*/\1/p' < /etc/.relid`
+	    && OS_REL=.$(sed -n 's/[^ ]* [^ ]* \([0-9][0-9]\).*/\1/p' < /etc/.relid)
 	/bin/uname -p 2>/dev/null | grep 86 >/dev/null \
 	    && { echo i486-ncr-sysv4.3"$OS_REL"; exit; }
 	/bin/uname -p 2>/dev/null | /bin/grep entium >/dev/null \
@@ -1210,7 +1271,7 @@ EOF
 	exit ;;
     *:SINIX-*:*:*)
 	if uname -p 2>/dev/null >/dev/null ; then
-		UNAME_MACHINE=`(uname -p) 2>/dev/null`
+		UNAME_MACHINE=$( (uname -p) 2>/dev/null)
 		echo "$UNAME_MACHINE"-sni-sysv4
 	else
 		echo ns32k-sni-sysv
@@ -1244,7 +1305,7 @@ EOF
 	echo mips-sony-newsos6
 	exit ;;
     R[34]000:*System_V*:*:* | R4000:UNIX_SYSV:*:* | R*000:UNIX_SV:*:*)
-	if [ -d /usr/nec ]; then
+	if test -d /usr/nec; then
 		echo mips-nec-sysv"$UNAME_RELEASE"
 	else
 		echo mips-unknown-sysv"$UNAME_RELEASE"
@@ -1292,44 +1353,48 @@ EOF
     *:Rhapsody:*:*)
 	echo "$UNAME_MACHINE"-apple-rhapsody"$UNAME_RELEASE"
 	exit ;;
+    arm64:Darwin:*:*)
+	echo aarch64-apple-darwin"$UNAME_RELEASE"
+	exit ;;
     *:Darwin:*:*)
-	UNAME_PROCESSOR=`uname -p` || UNAME_PROCESSOR=unknown
-	set_cc_for_build
-	if test "$UNAME_PROCESSOR" = unknown ; then
-	    UNAME_PROCESSOR=powerpc
+	UNAME_PROCESSOR=$(uname -p)
+	case $UNAME_PROCESSOR in
+	    unknown) UNAME_PROCESSOR=powerpc ;;
+	esac
+	if command -v xcode-select > /dev/null 2> /dev/null && \
+		! xcode-select --print-path > /dev/null 2> /dev/null ; then
+	    # Avoid executing cc if there is no toolchain installed as
+	    # cc will be a stub that puts up a graphical alert
+	    # prompting the user to install developer tools.
+	    CC_FOR_BUILD=no_compiler_found
+	else
+	    set_cc_for_build
 	fi
-	if test "`echo "$UNAME_RELEASE" | sed -e 's/\..*//'`" -le 10 ; then
-	    if [ "$CC_FOR_BUILD" != no_compiler_found ]; then
-		if (echo '#ifdef __LP64__'; echo IS_64BIT_ARCH; echo '#endif') | \
-		       (CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
-		       grep IS_64BIT_ARCH >/dev/null
-		then
-		    case $UNAME_PROCESSOR in
-			i386) UNAME_PROCESSOR=x86_64 ;;
-			powerpc) UNAME_PROCESSOR=powerpc64 ;;
-		    esac
-		fi
-		# On 10.4-10.6 one might compile for PowerPC via gcc -arch ppc
-		if (echo '#ifdef __POWERPC__'; echo IS_PPC; echo '#endif') | \
-		       (CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
-		       grep IS_PPC >/dev/null
-		then
-		    UNAME_PROCESSOR=powerpc
-		fi
+	if test "$CC_FOR_BUILD" != no_compiler_found; then
+	    if (echo '#ifdef __LP64__'; echo IS_64BIT_ARCH; echo '#endif') | \
+		   (CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
+		   grep IS_64BIT_ARCH >/dev/null
+	    then
+		case $UNAME_PROCESSOR in
+		    i386) UNAME_PROCESSOR=x86_64 ;;
+		    powerpc) UNAME_PROCESSOR=powerpc64 ;;
+		esac
+	    fi
+	    # On 10.4-10.6 one might compile for PowerPC via gcc -arch ppc
+	    if (echo '#ifdef __POWERPC__'; echo IS_PPC; echo '#endif') | \
+		   (CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
+		   grep IS_PPC >/dev/null
+	    then
+		UNAME_PROCESSOR=powerpc
 	    fi
 	elif test "$UNAME_PROCESSOR" = i386 ; then
-	    # Avoid executing cc on OS X 10.9, as it ships with a stub
-	    # that puts up a graphical alert prompting to install
-	    # developer tools.  Any system running Mac OS X 10.7 or
-	    # later (Darwin 11 and later) is required to have a 64-bit
-	    # processor. This is not true of the ARM version of Darwin
-	    # that Apple uses in portable devices.
-	    UNAME_PROCESSOR=x86_64
+	    # uname -m returns i386 or x86_64
+	    UNAME_PROCESSOR=$UNAME_MACHINE
 	fi
 	echo "$UNAME_PROCESSOR"-apple-darwin"$UNAME_RELEASE"
 	exit ;;
     *:procnto*:*:* | *:QNX:[0123456789]*:*)
-	UNAME_PROCESSOR=`uname -p`
+	UNAME_PROCESSOR=$(uname -p)
 	if test "$UNAME_PROCESSOR" = x86; then
 		UNAME_PROCESSOR=i386
 		UNAME_MACHINE=pc
@@ -1397,10 +1462,10 @@ EOF
 	echo mips-sei-seiux"$UNAME_RELEASE"
 	exit ;;
     *:DragonFly:*:*)
-	echo "$UNAME_MACHINE"-unknown-dragonfly"`echo "$UNAME_RELEASE"|sed -e 's/[-(].*//'`"
+	echo "$UNAME_MACHINE"-unknown-dragonfly"$(echo "$UNAME_RELEASE"|sed -e 's/[-(].*//')"
 	exit ;;
     *:*VMS:*:*)
-	UNAME_MACHINE=`(uname -p) 2>/dev/null`
+	UNAME_MACHINE=$( (uname -p) 2>/dev/null)
 	case "$UNAME_MACHINE" in
 	    A*) echo alpha-dec-vms ; exit ;;
 	    I*) echo ia64-dec-vms ; exit ;;
@@ -1410,7 +1475,7 @@ EOF
 	echo i386-pc-xenix
 	exit ;;
     i*86:skyos:*:*)
-	echo "$UNAME_MACHINE"-pc-skyos"`echo "$UNAME_RELEASE" | sed -e 's/ .*$//'`"
+	echo "$UNAME_MACHINE"-pc-skyos"$(echo "$UNAME_RELEASE" | sed -e 's/ .*$//')"
 	exit ;;
     i*86:rdos:*:*)
 	echo "$UNAME_MACHINE"-pc-rdos
@@ -1424,8 +1489,148 @@ EOF
     amd64:Isilon\ OneFS:*:*)
 	echo x86_64-unknown-onefs
 	exit ;;
+    *:Unleashed:*:*)
+	echo "$UNAME_MACHINE"-unknown-unleashed"$UNAME_RELEASE"
+	exit ;;
 esac
 
+# No uname command or uname output not recognized.
+set_cc_for_build
+cat > "$dummy.c" <<EOF
+#ifdef _SEQUENT_
+#include <sys/types.h>
+#include <sys/utsname.h>
+#endif
+#if defined(ultrix) || defined(_ultrix) || defined(__ultrix) || defined(__ultrix__)
+#if defined (vax) || defined (__vax) || defined (__vax__) || defined(mips) || defined(__mips) || defined(__mips__) || defined(MIPS) || defined(__MIPS__)
+#include <signal.h>
+#if defined(_SIZE_T_) || defined(SIGLOST)
+#include <sys/utsname.h>
+#endif
+#endif
+#endif
+main ()
+{
+#if defined (sony)
+#if defined (MIPSEB)
+  /* BFD wants "bsd" instead of "newsos".  Perhaps BFD should be changed,
+     I don't know....  */
+  printf ("mips-sony-bsd\n"); exit (0);
+#else
+#include <sys/param.h>
+  printf ("m68k-sony-newsos%s\n",
+#ifdef NEWSOS4
+  "4"
+#else
+  ""
+#endif
+  ); exit (0);
+#endif
+#endif
+
+#if defined (NeXT)
+#if !defined (__ARCHITECTURE__)
+#define __ARCHITECTURE__ "m68k"
+#endif
+  int version;
+  version=$( (hostinfo | sed -n 's/.*NeXT Mach \([0-9]*\).*/\1/p') 2>/dev/null);
+  if (version < 4)
+    printf ("%s-next-nextstep%d\n", __ARCHITECTURE__, version);
+  else
+    printf ("%s-next-openstep%d\n", __ARCHITECTURE__, version);
+  exit (0);
+#endif
+
+#if defined (MULTIMAX) || defined (n16)
+#if defined (UMAXV)
+  printf ("ns32k-encore-sysv\n"); exit (0);
+#else
+#if defined (CMU)
+  printf ("ns32k-encore-mach\n"); exit (0);
+#else
+  printf ("ns32k-encore-bsd\n"); exit (0);
+#endif
+#endif
+#endif
+
+#if defined (__386BSD__)
+  printf ("i386-pc-bsd\n"); exit (0);
+#endif
+
+#if defined (sequent)
+#if defined (i386)
+  printf ("i386-sequent-dynix\n"); exit (0);
+#endif
+#if defined (ns32000)
+  printf ("ns32k-sequent-dynix\n"); exit (0);
+#endif
+#endif
+
+#if defined (_SEQUENT_)
+  struct utsname un;
+
+  uname(&un);
+  if (strncmp(un.version, "V2", 2) == 0) {
+    printf ("i386-sequent-ptx2\n"); exit (0);
+  }
+  if (strncmp(un.version, "V1", 2) == 0) { /* XXX is V1 correct? */
+    printf ("i386-sequent-ptx1\n"); exit (0);
+  }
+  printf ("i386-sequent-ptx\n"); exit (0);
+#endif
+
+#if defined (vax)
+#if !defined (ultrix)
+#include <sys/param.h>
+#if defined (BSD)
+#if BSD == 43
+  printf ("vax-dec-bsd4.3\n"); exit (0);
+#else
+#if BSD == 199006
+  printf ("vax-dec-bsd4.3reno\n"); exit (0);
+#else
+  printf ("vax-dec-bsd\n"); exit (0);
+#endif
+#endif
+#else
+  printf ("vax-dec-bsd\n"); exit (0);
+#endif
+#else
+#if defined(_SIZE_T_) || defined(SIGLOST)
+  struct utsname un;
+  uname (&un);
+  printf ("vax-dec-ultrix%s\n", un.release); exit (0);
+#else
+  printf ("vax-dec-ultrix\n"); exit (0);
+#endif
+#endif
+#endif
+#if defined(ultrix) || defined(_ultrix) || defined(__ultrix) || defined(__ultrix__)
+#if defined(mips) || defined(__mips) || defined(__mips__) || defined(MIPS) || defined(__MIPS__)
+#if defined(_SIZE_T_) || defined(SIGLOST)
+  struct utsname *un;
+  uname (&un);
+  printf ("mips-dec-ultrix%s\n", un.release); exit (0);
+#else
+  printf ("mips-dec-ultrix\n"); exit (0);
+#endif
+#endif
+#endif
+
+#if defined (alliant) && defined (i860)
+  printf ("i860-alliant-bsd\n"); exit (0);
+#endif
+
+  exit (1);
+}
+EOF
+
+$CC_FOR_BUILD -o "$dummy" "$dummy.c" 2>/dev/null && SYSTEM_NAME=$($dummy) &&
+	{ echo "$SYSTEM_NAME"; exit; }
+
+# Apollos put the system type in the environment.
+test -d /usr/apollo && { echo "$ISP-apollo-$SYSTYPE"; exit; }
+
 echo "$0: unable to guess system type" >&2
 
 case "$UNAME_MACHINE:$UNAME_SYSTEM" in
@@ -1445,9 +1650,15 @@ This script (version $timestamp), has failed to recognize the
 operating system you are using. If your script is old, overwrite *all*
 copies of config.guess and config.sub with the latest versions from:
 
-  https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess
+  https://git.savannah.gnu.org/cgit/config.git/plain/config.guess
 and
-  https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub
+  https://git.savannah.gnu.org/cgit/config.git/plain/config.sub
+EOF
+
+year=$(echo $timestamp | sed 's,-.*,,')
+# shellcheck disable=SC2003
+if test "$(expr "$(date +%Y)" - "$year")" -lt 3 ; then
+   cat >&2 <<EOF
 
 If $0 has already been updated, send the following data and any
 information you think might be pertinent to config-patches@gnu.org to
@@ -1455,26 +1666,27 @@ provide the necessary information to handle your system.
 
 config.guess timestamp = $timestamp
 
-uname -m = `(uname -m) 2>/dev/null || echo unknown`
-uname -r = `(uname -r) 2>/dev/null || echo unknown`
-uname -s = `(uname -s) 2>/dev/null || echo unknown`
-uname -v = `(uname -v) 2>/dev/null || echo unknown`
+uname -m = $( (uname -m) 2>/dev/null || echo unknown)
+uname -r = $( (uname -r) 2>/dev/null || echo unknown)
+uname -s = $( (uname -s) 2>/dev/null || echo unknown)
+uname -v = $( (uname -v) 2>/dev/null || echo unknown)
 
-/usr/bin/uname -p = `(/usr/bin/uname -p) 2>/dev/null`
-/bin/uname -X     = `(/bin/uname -X) 2>/dev/null`
+/usr/bin/uname -p = $( (/usr/bin/uname -p) 2>/dev/null)
+/bin/uname -X     = $( (/bin/uname -X) 2>/dev/null)
 
-hostinfo               = `(hostinfo) 2>/dev/null`
-/bin/universe          = `(/bin/universe) 2>/dev/null`
-/usr/bin/arch -k       = `(/usr/bin/arch -k) 2>/dev/null`
-/bin/arch              = `(/bin/arch) 2>/dev/null`
-/usr/bin/oslevel       = `(/usr/bin/oslevel) 2>/dev/null`
-/usr/convex/getsysinfo = `(/usr/convex/getsysinfo) 2>/dev/null`
+hostinfo               = $( (hostinfo) 2>/dev/null)
+/bin/universe          = $( (/bin/universe) 2>/dev/null)
+/usr/bin/arch -k       = $( (/usr/bin/arch -k) 2>/dev/null)
+/bin/arch              = $( (/bin/arch) 2>/dev/null)
+/usr/bin/oslevel       = $( (/usr/bin/oslevel) 2>/dev/null)
+/usr/convex/getsysinfo = $( (/usr/convex/getsysinfo) 2>/dev/null)
 
 UNAME_MACHINE = "$UNAME_MACHINE"
 UNAME_RELEASE = "$UNAME_RELEASE"
 UNAME_SYSTEM  = "$UNAME_SYSTEM"
 UNAME_VERSION = "$UNAME_VERSION"
 EOF
+fi
 
 exit 1
 

configure.ac

@@ -174,11 +174,15 @@ PKG_CHECK_MODULES([OPENSSL], [libcrypto], [CXXFLAGS="$OPENSSL_CFLAGS $CXXFLAGS"]
 
 # Look for libbz2, a required dependency.
 AC_CHECK_LIB([bz2], [BZ2_bzWriteOpen], [true],
-  [AC_MSG_ERROR([Nix requires libbz2, which is part of bzip2.  See https://web.archive.org/web/20180624184756/http://www.bzip.org/.])])
+  [AC_MSG_ERROR([Nix requires libbz2, which is part of bzip2.  See https://sourceware.org/bzip2/.])])
 AC_CHECK_HEADERS([bzlib.h], [true],
-  [AC_MSG_ERROR([Nix requires libbz2, which is part of bzip2.  See https://web.archive.org/web/20180624184756/http://www.bzip.org/.])])
+  [AC_MSG_ERROR([Nix requires libbz2, which is part of bzip2.  See https://sourceware.org/bzip2/.])])
 # Checks for libarchive
 PKG_CHECK_MODULES([LIBARCHIVE], [libarchive >= 3.1.2], [CXXFLAGS="$LIBARCHIVE_CFLAGS $CXXFLAGS"])
+# Workaround until https://github.com/libarchive/libarchive/issues/1446 is fixed
+if test "$shared" != yes; then
+    LIBARCHIVE_LIBS+=' -lz'
+fi
 
 # Look for SQLite, a required dependency.
 PKG_CHECK_MODULES([SQLITE3], [sqlite3 >= 3.6.19], [CXXFLAGS="$SQLITE3_CFLAGS $CXXFLAGS"])
@@ -199,11 +203,7 @@ PKG_CHECK_MODULES([EDITLINE], [libeditline], [CXXFLAGS="$EDITLINE_CFLAGS $CXXFLA
 ])
 
 # Look for libsodium, an optional dependency.
-PKG_CHECK_MODULES([SODIUM], [libsodium],
-  [AC_DEFINE([HAVE_SODIUM], [1], [Whether to use libsodium for cryptography.])
-   CXXFLAGS="$SODIUM_CFLAGS $CXXFLAGS"
-   have_sodium=1], [have_sodium=])
-AC_SUBST(HAVE_SODIUM, [$have_sodium])
+PKG_CHECK_MODULES([SODIUM], [libsodium], [CXXFLAGS="$SODIUM_CFLAGS $CXXFLAGS"])
 
 # Look for liblzma, a required dependency.
 PKG_CHECK_MODULES([LIBLZMA], [liblzma], [CXXFLAGS="$LIBLZMA_CFLAGS $CXXFLAGS"])
@@ -251,6 +251,7 @@ if test -n "$enable_s3"; then
   declare -a aws_version_tokens=($(printf '#include <aws/core/VersionConfig.h>\nAWS_SDK_VERSION_STRING' | $CPP $CPPFLAGS - | grep -v '^#.*' | sed 's/"//g' | tr '.' ' '))
   AC_DEFINE_UNQUOTED([AWS_VERSION_MAJOR], ${aws_version_tokens@<:@0@:>@}, [Major version of aws-sdk-cpp.])
   AC_DEFINE_UNQUOTED([AWS_VERSION_MINOR], ${aws_version_tokens@<:@1@:>@}, [Minor version of aws-sdk-cpp.])
+  AC_DEFINE_UNQUOTED([AWS_VERSION_PATCH], ${aws_version_tokens@<:@2@:>@}, [Patch version of aws-sdk-cpp.])
 fi
 
 

corepkgs/local.mk

@@ -1,4 +0,0 @@
-corepkgs_FILES = \
-  fetchurl.nix
-
-$(foreach file,$(corepkgs_FILES),$(eval $(call install-data-in,$(d)/$(file),$(datadir)/nix/corepkgs)))

doc/manual/generate-manpage.nix

@@ -1,55 +1,92 @@
+command:
+
 with builtins;
 with import ./utils.nix;
 
 let
 
   showCommand =
-    { command, section, def }:
-    "${section} Name\n\n"
+    { command, def, filename }:
+    "# Name\n\n"
     + "`${command}` - ${def.description}\n\n"
-    + "${section} Synopsis\n\n"
+    + "# Synopsis\n\n"
     + showSynopsis { inherit command; args = def.args; }
-    + (if def ? doc
-       then "${section} Description\n\n" + def.doc + "\n\n"
-       else "")
-    + (let s = showFlags def.flags; in
-       if s != ""
-       then "${section} Flags\n\n${s}"
-       else "")
-    + (if def.examples or [] != []
+    + (if def.commands or {} != {}
        then
-         "${section} Examples\n\n"
-         + concatStrings (map ({ description, command }: "${description}\n\n```console\n${command}\n```\n\n") def.examples)
+         let
+           categories = sort (x: y: x.id < y.id) (unique (map (cmd: cmd.category) (attrValues def.commands)));
+           listCommands = cmds:
+             concatStrings (map (name:
+               "* [`${command} ${name}`](./${appendName filename name}.md) - ${cmds.${name}.description}\n")
+               (attrNames cmds));
+         in
+         "where *subcommand* is one of the following:\n\n"
+         # FIXME: group by category
+         + (if length categories > 1
+            then
+              concatStrings (map
+                (cat:
+                  "**${toString cat.description}:**\n\n"
+                  + listCommands (filterAttrs (n: v: v.category == cat) def.commands)
+                  + "\n"
+                ) categories)
+              + "\n"
+            else
+              listCommands def.commands
+              + "\n")
        else "")
-    + (if def.commands or [] != []
-       then concatStrings (
-         map (name:
-           "# Subcommand `${command} ${name}`\n\n"
-           + showCommand { command = command + " " + name; section = "##"; def = def.commands.${name}; })
-           (attrNames def.commands))
-       else "");
+    + (if def ? doc
+       then def.doc + "\n\n"
+       else "")
+    + (let s = showOptions def.flags; in
+       if s != ""
+       then "# Options\n\n${s}"
+       else "")
+  ;
 
-  showFlags = flags:
-    concatStrings
-      (map (longName:
-        let flag = flags.${longName}; in
-        if flag.category or "" != "config"
-        then
-          "  - `--${longName}`"
-          + (if flag ? shortName then " / `${flag.shortName}`" else "")
-          + (if flag ? labels then " " + (concatStringsSep " " (map (s: "*${s}*") flag.labels)) else "")
-          + "  \n"
-          + "    " + flag.description + "\n\n"
-        else "")
-        (attrNames flags));
+  appendName = filename: name: (if filename == "nix" then "nix3" else filename) + "-" + name;
+
+  showOptions = flags:
+    let
+      categories = sort builtins.lessThan (unique (map (cmd: cmd.category) (attrValues flags)));
+    in
+      concatStrings (map
+        (cat:
+          (if cat != ""
+           then "**${cat}:**\n\n"
+           else "")
+          + concatStrings
+            (map (longName:
+              let
+                flag = flags.${longName};
+              in
+                "  - `--${longName}`"
+                + (if flag ? shortName then " / `-${flag.shortName}`" else "")
+                + (if flag ? labels then " " + (concatStringsSep " " (map (s: "*${s}*") flag.labels)) else "")
+                + "  \n"
+                + "    " + flag.description + "\n\n"
+            ) (attrNames (filterAttrs (n: v: v.category == cat) flags))))
+        categories);
 
   showSynopsis =
     { command, args }:
-    "`${command}` [*flags*...] ${concatStringsSep " "
+    "`${command}` [*option*...] ${concatStringsSep " "
       (map (arg: "*${arg.label}*" + (if arg ? arity then "" else "...")) args)}\n\n";
 
+  processCommand = { command, def, filename }:
+    [ { name = filename + ".md"; value = showCommand { inherit command def filename; }; inherit command; } ]
+    ++ concatMap
+      (name: processCommand {
+        filename = appendName filename name;
+        command = command + " " + name;
+        def = def.commands.${name};
+      })
+      (attrNames def.commands or {});
+
 in
 
-command:
-
-showCommand { command = "nix"; section = "#"; def = command; }
+let
+  manpages = processCommand { filename = "nix"; command = "nix"; def = command; };
+  summary = concatStrings (map (manpage: "    - [${manpage.command}](command-ref/new-cli/${manpage.name})\n") manpages);
+in
+(listToAttrs manpages) // { "SUMMARY.md" = summary; }

doc/manual/local.mk

@@ -4,7 +4,7 @@ MANUAL_SRCS := $(call rwildcard, $(d)/src, *.md)
 
 # Generate man pages.
 man-pages := $(foreach n, \
-  nix-env.1 nix-build.1 nix-shell.1 nix-store.1 nix-instantiate.1 nix.1 \
+  nix-env.1 nix-build.1 nix-shell.1 nix-store.1 nix-instantiate.1 \
   nix-collect-garbage.1 \
   nix-prefetch-url.1 nix-channel.1 \
   nix-hash.1 nix-copy-closure.1 \
@@ -13,9 +13,14 @@ man-pages := $(foreach n, \
 
 clean-files += $(d)/*.1 $(d)/*.5 $(d)/*.8
 
-dist-files += $(man-pages)
+# Provide a dummy environment for nix, so that it will not access files outside the macOS sandbox.
+dummy-env = env -i \
+	HOME=/dummy \
+	NIX_CONF_DIR=/dummy \
+	NIX_SSL_CERT_FILE=/dummy/no-ca-bundle.crt \
+	NIX_STATE_DIR=/dummy
 
-nix-eval = $(bindir)/nix eval --experimental-features nix-command -I nix/corepkgs=corepkgs --store dummy:// --impure --raw --expr
+nix-eval = $(dummy-env) $(bindir)/nix eval --experimental-features nix-command -I nix/corepkgs=corepkgs --store dummy:// --impure --raw
 
 $(d)/%.1: $(d)/src/command-ref/%.md
 	@printf "Title: %s\n\n" "$$(basename $@ .1)" > $^.tmp
@@ -35,37 +40,51 @@ $(d)/nix.conf.5: $(d)/src/command-ref/conf-file.md
 	$(trace-gen) lowdown -sT man $^.tmp -o $@
 	@rm $^.tmp
 
-$(d)/src/command-ref/nix.md: $(d)/nix.json $(d)/generate-manpage.nix $(bindir)/nix
-	$(trace-gen) $(nix-eval) 'import doc/manual/generate-manpage.nix (builtins.fromJSON (builtins.readFile $<))' > $@.tmp
+$(d)/src/SUMMARY.md: $(d)/src/SUMMARY.md.in $(d)/src/command-ref/new-cli
+	$(trace-gen) cat doc/manual/src/SUMMARY.md.in | while IFS= read line; do if [[ $$line = @manpages@ ]]; then cat doc/manual/src/command-ref/new-cli/SUMMARY.md; else echo "$$line"; fi; done > $@.tmp
 	@mv $@.tmp $@
 
+$(d)/src/command-ref/new-cli: $(d)/nix.json $(d)/generate-manpage.nix $(bindir)/nix
+	@rm -rf $@
+	$(trace-gen) $(nix-eval) --write-to $@ --expr 'import doc/manual/generate-manpage.nix (builtins.fromJSON (builtins.readFile $<))'
+
 $(d)/src/command-ref/conf-file.md: $(d)/conf-file.json $(d)/generate-options.nix $(d)/src/command-ref/conf-file-prefix.md $(bindir)/nix
 	@cat doc/manual/src/command-ref/conf-file-prefix.md > $@.tmp
-	$(trace-gen) $(nix-eval) 'import doc/manual/generate-options.nix (builtins.fromJSON (builtins.readFile $<))' >> $@.tmp
+	$(trace-gen) $(nix-eval) --expr 'import doc/manual/generate-options.nix (builtins.fromJSON (builtins.readFile $<))' >> $@.tmp
 	@mv $@.tmp $@
 
 $(d)/nix.json: $(bindir)/nix
-	$(trace-gen) $(bindir)/nix __dump-args > $@.tmp
+	$(trace-gen) $(dummy-env) $(bindir)/nix __dump-args > $@.tmp
 	@mv $@.tmp $@
 
 $(d)/conf-file.json: $(bindir)/nix
-	$(trace-gen) env -i NIX_CONF_DIR=/dummy HOME=/dummy NIX_SSL_CERT_FILE=/dummy/no-ca-bundle.crt $(bindir)/nix show-config --json --experimental-features nix-command > $@.tmp
+	$(trace-gen) $(dummy-env) $(bindir)/nix show-config --json --experimental-features nix-command > $@.tmp
 	@mv $@.tmp $@
 
 $(d)/src/expressions/builtins.md: $(d)/builtins.json $(d)/generate-builtins.nix $(d)/src/expressions/builtins-prefix.md $(bindir)/nix
 	@cat doc/manual/src/expressions/builtins-prefix.md > $@.tmp
-	$(trace-gen) $(nix-eval) 'import doc/manual/generate-builtins.nix (builtins.fromJSON (builtins.readFile $<))' >> $@.tmp
+	$(trace-gen) $(nix-eval) --expr 'import doc/manual/generate-builtins.nix (builtins.fromJSON (builtins.readFile $<))' >> $@.tmp
 	@mv $@.tmp $@
 
 $(d)/builtins.json: $(bindir)/nix
-	$(trace-gen) NIX_PATH=nix/corepkgs=corepkgs $(bindir)/nix __dump-builtins > $@.tmp
-	mv $@.tmp $@
+	$(trace-gen) $(dummy-env) NIX_PATH=nix/corepkgs=corepkgs $(bindir)/nix __dump-builtins > $@.tmp
+	@mv $@.tmp $@
 
 # Generate the HTML manual.
 install: $(docdir)/manual/index.html
 
-$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/custom.css $(d)/src/command-ref/nix.md $(d)/src/command-ref/conf-file.md $(d)/src/expressions/builtins.md
-	$(trace-gen) mdbook build doc/manual -d $(docdir)/manual
+# Generate 'nix' manpages.
+install: $(d)/src/command-ref/new-cli
+	$(trace-gen) for i in doc/manual/src/command-ref/new-cli/*.md; do \
+	  name=$$(basename $$i .md); \
+	  if [[ $$name = SUMMARY ]]; then continue; fi; \
+	  printf "Title: %s\n\n" "$$name" > $$i.tmp; \
+	  cat $$i >> $$i.tmp; \
+	  lowdown -sT man $$i.tmp -o $(mandir)/man1/$$name.1; \
+	done
+
+$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/command-ref/conf-file.md $(d)/src/expressions/builtins.md
+	$(trace-gen) RUST_LOG=warn mdbook build doc/manual -d $(docdir)/manual
 	@cp doc/manual/highlight.pack.js $(docdir)/manual/highlight.js
 
 endif

doc/manual/src/SUMMARY.md.in

@@ -62,11 +62,13 @@
     - [nix-instantiate](command-ref/nix-instantiate.md)
     - [nix-prefetch-url](command-ref/nix-prefetch-url.md)
   - [Experimental Commands](command-ref/experimental-commands.md)
-    - [nix](command-ref/nix.md)
+@manpages@
   - [Files](command-ref/files.md)
     - [nix.conf](command-ref/conf-file.md)
 - [Glossary](glossary.md)
-- [Hacking](hacking.md)
+- [Contributing](contributing/contributing.md)
+  - [Hacking](contributing/hacking.md)
+  - [CLI guideline](contributing/cli-guideline.md)
 - [Release Notes](release-notes/release-notes.md)
   - [Release 2.3 (2019-09-04)](release-notes/rl-2.3.md)
   - [Release 2.2 (2019-01-11)](release-notes/rl-2.2.md)

doc/manual/src/advanced-topics/post-build-hook.md

@@ -53,7 +53,7 @@ set -f # disable globbing
 export IFS=' '
 
 echo "Signing paths" $OUT_PATHS
-nix sign-paths --key-file /etc/nix/key.private $OUT_PATHS
+nix store sign --key-file /etc/nix/key.private $OUT_PATHS
 echo "Uploading paths" $OUT_PATHS
 exec nix copy --to 's3://example-nix-cache' $OUT_PATHS
 ```
@@ -63,7 +63,7 @@ exec nix copy --to 's3://example-nix-cache' $OUT_PATHS
 > The `$OUT_PATHS` variable is a space-separated list of Nix store
 > paths. In this case, we expect and want the shell to perform word
 > splitting to make each output path its own argument to `nix
-> sign-paths`. Nix guarantees the paths will not contain any spaces,
+> store sign`. Nix guarantees the paths will not contain any spaces,
 > however a store path might contain glob characters. The `set -f`
 > disables globbing in the shell.
 

doc/manual/src/command-ref/conf-file-prefix.md

@@ -19,19 +19,33 @@ By default Nix reads settings from the following places:
     and `XDG_CONFIG_HOME`. If these are unset, it will look in
     `$HOME/.config/nix.conf`.
 
-The configuration files consist of `name =
-value` pairs, one per line. Other files can be included with a line like
-`include
-path`, where *path* is interpreted relative to the current conf file and
-a missing file is an error unless `!include` is used instead. Comments
+  - If `NIX_CONFIG` is set, its contents is treated as the contents of
+    a configuration file.
+
+The configuration files consist of `name = value` pairs, one per
+line. Other files can be included with a line like `include path`,
+where *path* is interpreted relative to the current conf file and a
+missing file is an error unless `!include` is used instead. Comments
 start with a `#` character. Here is an example configuration file:
 
     keep-outputs = true       # Nice for developers
     keep-derivations = true   # Idem
 
-You can override settings on the command line using the `--option` flag,
-e.g. `--option keep-outputs
-false`.
+You can override settings on the command line using the `--option`
+flag, e.g. `--option keep-outputs false`. Every configuration setting
+also has a corresponding command line flag, e.g. `--max-jobs 16`; for
+Boolean settings, there are two flags to enable or disable the setting
+(e.g. `--keep-failed` and `--no-keep-failed`).
+
+A configuration setting usually overrides any previous value. However,
+you can prefix the name of the setting by `extra-` to *append* to the
+previous value. For instance,
+
+    substituters = a b
+    extra-substituters = c d
+
+defines the `substituters` setting to be `a b c d`. This is also
+available as a command line flag (e.g. `--extra-substituters`).
 
 The following settings are currently available:
 

doc/manual/src/command-ref/env-common.md

@@ -81,6 +81,11 @@ Most Nix commands interpret the following environment variables:
     Overrides the location of the system Nix configuration directory
     (default `prefix/etc/nix`).
 
+  - `NIX_CONFIG`  
+    Applies settings from Nix configuration from the environment.
+    The content is treated as if it was read from a Nix configuration file.
+    Settings are separated by the newline character.
+
   - `NIX_USER_CONF_FILES`  
     Overrides the location of the user Nix configuration files to load
     from (defaults to the XDG spec locations). The variable is treated

doc/manual/src/command-ref/nix-hash.md

@@ -45,7 +45,7 @@ md5sum`.
 
   - `--type` *hashAlgo*  
     Use the specified cryptographic hash algorithm, which can be one of
-    `md5`, `sha1`, and `sha256`.
+    `md5`, `sha1`, `sha256`, and `sha512`.
 
   - `--to-base16`  
     Don’t hash anything, but convert the base-32 hash representation

doc/manual/src/command-ref/nix-prefetch-url.md

@@ -39,7 +39,7 @@ Nix store is also printed.
 
   - `--type` *hashAlgo*  
     Use the specified cryptographic hash algorithm, which can be one of
-    `md5`, `sha1`, and `sha256`.
+    `md5`, `sha1`, `sha256`, and `sha512`.
 
   - `--print-path`  
     Print the store path of the downloaded file on standard output.

doc/manual/src/command-ref/nix-shell.md

@@ -32,7 +32,7 @@ URL of a tarball that will be downloaded and unpacked to a temporary
 location. The tarball must include a single top-level directory
 containing at least a file named `default.nix`.
 
-If the derivation defines the variable `shellHook`, it will be evaluated
+If the derivation defines the variable `shellHook`, it will be run
 after `$stdenv/setup` has been sourced. Since this hook is not executed
 by regular Nix builds, it allows you to perform initialisation specific
 to `nix-shell`. For example, the derivation attribute
@@ -41,10 +41,12 @@ to `nix-shell`. For example, the derivation attribute
 shellHook =
   ''
     echo "Hello shell"
+    export SOME_API_TOKEN="$(cat ~/.config/some-app/api-token)"
   '';
 ```
 
-will cause `nix-shell` to print `Hello shell`.
+will cause `nix-shell` to print `Hello shell` and set the `SOME_API_TOKEN`
+environment variable to a user-configured value.
 
 # Options
 
@@ -76,8 +78,8 @@ All options not listed here are passed to `nix-store
     cleared before the interactive shell is started, so you get an
     environment that more closely corresponds to the “real” Nix build. A
     few variables, in particular `HOME`, `USER` and `DISPLAY`, are
-    retained. Note that `~/.bashrc` and (depending on your Bash
-    installation) `/etc/bashrc` are still sourced, so any variables set
+    retained. Note that (depending on your Bash
+    installation) `/etc/bashrc` is still sourced, so any variables set
     there will affect the interactive shell.
 
   - `--packages` / `-p` *packages*…  

doc/manual/src/command-ref/nix-store.md

@@ -226,7 +226,7 @@ control what gets deleted and in what order:
     or TiB units.
 
 The behaviour of the collector is also influenced by the
-`keep-outputs` and `keep-derivations` variables in the Nix
+`keep-outputs` and `keep-derivations` settings in the Nix
 configuration file.
 
 By default, the collector prints the total number of freed bytes when it

doc/manual/src/contributing/cli-guideline.md

@@ -0,0 +1,589 @@
+# CLI guideline
+
+## Goals
+
+Purpose of this document is to provide a clear direction to **help design
+delightful command line** experience. This document contain guidelines to
+follow to ensure a consistent and approachable user experience.
+
+## Overview
+
+`nix` command provides a single entry to a number of sub-commands that help
+**developers and system administrators** in the life-cycle of a software
+project. We particularly need to pay special attention to help and assist new
+users of Nix.
+
+# Naming the `COMMANDS`
+
+Words matter. Naming is an important part of the usability. Users will be
+interacting with Nix on a regular basis so we should **name things for ease of
+understanding**.
+
+We recommend following the [Principle of Least
+Astonishment](https://en.wikipedia.org/wiki/Principle_of_least_astonishment).
+This means that you should **never use acronyms or abbreviations** unless they
+are commonly used in other tools (e.g. `nix init`). And if the command name is
+too long (> 10-12 characters) then shortening it makes sense (e.g.
+“prioritization” → “priority”).
+
+Commands should **follow a noun-verb dialogue**. Although noun-verb formatting
+seems backwards from a speaking perspective (i.e. `nix store copy` vs. `nix
+copy store`) it allows us to organize commands the same way users think about
+completing an action (the group first, then the command).
+
+## Naming rules
+
+Rules are there to guide you by limiting your options. But not everything can
+fit the rules all the time. In those cases document the exceptions in [Appendix
+1: Commands naming exceptions](#appendix-1-commands-naming-exceptions) and
+provide reason. The rules want to force a Nix developer to look, not just at
+the command at hand, but also the command in a full context alongside other
+`nix` commands.
+
+```shell
+$ nix [<GROUP>] <COMMAND> [<ARGUMENTS>] [<OPTIONS>]
+```
+
+- `GROUP`, `COMMAND`, `ARGUMENTS` and `OPTIONS` should be lowercase and in a
+  singular form.
+- `GROUP` should be a  **NOUN**.
+- `COMMAND` should be a **VERB**.
+- `ARGUMENTS` and `OPTIONS` are discussed in [*Input* section](#input).
+
+## Classification
+
+Some commands are more important, some less. While we want all of our commands
+to be perfect we can only spend limited amount of time testing and improving
+them.
+
+This classification tries to separate commands in 3 categories in terms of
+their importance in regards to the new users. Users who are likely to be
+impacted the most by bad user experience.
+
+- **Main commands**
+
+  Commands used for our main use cases and most likely used by new users. We
+  expect attention to details, such as:
+
+    - Proper use of [colors](#colors), [emojis](#special-unicode-characters)
+      and [aligning of text](#text-alignment).
+    - [Autocomplete](#shell-completion) of options.
+    - Show [next possible steps](#next-steps).
+    - Showing some [“tips”](#educate-the-user) when running logs running tasks
+      (eg. building / downloading) in order to teach users interesting bits of
+      Nix ecosystem.
+    - [Help pages](#help-is-essential) to be as good as we can write them
+      pointing to external documentation and tutorials for more.
+
+  Examples of such commands: `nix init`, `nix develop`, `nix build`, `nix run`,
+  ...
+
+- **Infrequently used commands**
+
+  From infrequently used commands we expect less attention to details, but
+  still some:
+
+    - Proper use of [colors](#colors), [emojis](#special-unicode-characters)
+      and [aligning of text](#text-alignment).
+    - [Autocomplete](#shell-completion) of options.
+
+  Examples of such commands: `nix doctor`, `nix edit`, `nix eval`, ...
+
+- **Utility and scripting commands**
+
+  Commands that expose certain internal functionality of `nix`, mostly used by
+  other scripts.
+
+    - [Autocomplete](#shell-completion) of options.
+
+  Examples of such commands: `nix store copy`, `nix hash base16`, `nix store
+  ping`, ...
+
+
+# Help is essential
+
+Help should be built into your command line so that new users can gradually
+discover new features when they need them. 
+
+## Looking for help
+
+Since there is no standard way how user will look for help we rely on ways help
+is provided by commonly used tools. As a guide for this we took `git` and
+whenever in doubt look at it as a preferred direction.
+
+The rules are:
+
+- Help is shown by using `--help` or `help` command (eg `nix` `--``help` or
+  `nix help`).
+- For non-COMMANDs (eg. `nix` `--``help`  and `nix store` `--``help`) we **show
+  a summary** of most common use cases. Summary is presented on the STDOUT
+  without any use of PAGER.
+- For COMMANDs (eg. `nix init` `--``help` or `nix help init`) we display the
+  man page of that command. By default the PAGER is used (as in `git`).
+- At the end of either summary or man page there should be an URL pointing to
+  an online version of more detailed documentation.
+- The structure of summaries and man pages should be the same as in `git`.
+
+## Anticipate where help is needed
+
+Even better then requiring the user to search for help is to anticipate and
+predict when user might need it. Either because the lack of discoverability,
+typo in the input or simply taking the opportunity to teach the user of
+interesting - but less visible - details.
+
+### Shell completion
+
+This type of help is most common and almost expected by users. We need to
+**provide the best shell completion** for `bash`, `zsh` and `fish`.
+
+Completion needs to be **context aware**, this mean when a user types:
+
+```shell
+$ nix build n<TAB>
+```
+
+we need to display a list of flakes starting with `n`.
+
+### Wrong input
+
+As we all know we humans make mistakes, all the time. When a typo - intentional
+or unintentional - is made, we should prompt for closest possible options or
+point to the documentation which would educate user to not make the same
+errors. Here are few examples:
+
+In first example we prompt the user for typing wrong command name:
+
+
+```shell
+$ nix int
+------------------------------------------------------------------------
+  Error! Command `int` not found.
+------------------------------------------------------------------------
+  Did you mean:
+    |> nix init
+    |> nix input
+```
+
+Sometimes users will make mistake either because of a typo or simply because of
+lack of discoverability. Our handling of this cases needs to be context
+sensitive.
+
+
+```shell
+$ nix init --template=template#pyton
+------------------------------------------------------------------------
+  Error! Template `template#pyton` not found.
+------------------------------------------------------------------------
+Initializing Nix project at `/path/to/here`.
+      Select a template for you new project:
+          |> template#pyton
+             template#python-pip
+             template#python-poetry
+```
+
+### Next steps
+
+It can be invaluable to newcomers to show what a possible next steps and what
+is the usual development workflow with Nix. For example:
+
+
+```shell
+$ nix init --template=template#python
+Initializing project `template#python`
+          in `/home/USER/dev/new-project`
+
+  Next steps
+    |> nix develop   -- to enter development environment
+    |> nix build     -- to build your project
+```
+
+### Educate the user
+
+We should take any opportunity to **educate users**, but at the same time we
+must **be very very careful to not annoy users**. There is a thin line between
+being helpful and being annoying.
+
+An example of educating users might be to provide *Tips* in places where they
+are waiting.
+
+```shell
+$ nix build
+    Started building my-project 1.2.3
+ Downloaded python3.8-poetry 1.2.3 in 5.3 seconds
+ Downloaded python3.8-requests 1.2.3 in 5.3 seconds
+------------------------------------------------------------------------
+      Press `v` to increase logs verbosity
+         |> `?` to see other options
+------------------------------------------------------------------------
+      Learn something new with every build...
+         |> See last logs of a build with `nix log --last` command.
+------------------------------------------------------------------------
+  Evaluated my-project 1.2.3 in 14.43 seconds
+Downloading [12 / 200]
+         |> firefox 1.2.3 [#########>       ] 10Mb/s | 2min left
+   Building [2 / 20]
+         |> glibc 1.2.3 -> buildPhase: <last log line>
+------------------------------------------------------------------------
+```
+
+Now **Learn** part of the output is where you educate users. You should only
+show it when you know that a build will take some time and not annoy users of
+the builds that take only few seconds.
+
+Every feature like this should go though a intensive review and testing to
+collect as much a feedback as possible and to fine tune every little detail. If
+done right this can be an awesome features beginners and advance users will
+love, but if not done perfectly it will annoy users and leave bad impression.
+
+# Input
+
+Input to a command is provided via `ARGUMENTS` and `OPTIONS`. 
+
+`ARGUMENTS` represent a required input for a function. When choosing to use
+`ARGUMENT` over function please be aware of the downsides that come with it:
+
+- User will need to remember the order of `ARGUMENTS`. This is not a problem if
+  there is only one `ARGUMENT`.
+- With `OPTIONS` it is possible to provide much better auto completion.
+- With `OPTIONS` it is possible to provide much better error message.
+- Using `OPTIONS` it will mean there is a little bit more typing.
+
+We don’t discourage the use of `ARGUMENTS`, but simply want to make every
+developer consider the downsides and choose wisely.
+
+## Naming the `OPTIONS`
+
+Then only naming convention - apart from the ones mentioned in Naming the
+`COMMANDS` section is how flags are named.
+
+Flags are a type of `OPTION` that represent an option that can be turned ON of
+OFF. We can say **flags are boolean type of** `**OPTION**`.
+
+Here are few examples of flag `OPTIONS`:
+
+- `--colors` vs. `--no-colors` (showing colors in the output)
+- `--emojis` vs. `--no-emojis` (showing emojis in the output)
+
+## Prompt when input not provided
+
+For *main commands* (as [per classification](#classification)) we want command
+to improve the discoverability of possible input. A new user will most likely
+not know which `ARGUMENTS` and `OPTIONS` are required or which values are
+possible for those options.
+
+In cases, the user might not provide the input or they provide wrong input,
+rather then show the error, prompt a user with an option to find and select
+correct input (see examples).
+
+Prompting is of course not required when TTY is not attached to STDIN. This
+would mean that scripts wont need to handle prompt, but rather handle errors.
+
+A place to use prompt and provide user with interactive select
+
+
+```shell
+$ nix init
+Initializing Nix project at `/path/to/here`.
+      Select a template for you new project:
+          |> py
+             template#python-pip
+             template#python-poetry
+             [ Showing 2 templates from 1345 templates ]
+```
+
+Another great place to add prompts are **confirmation dialogues for dangerous
+actions**. For example when adding new substitutor via `OPTIONS` or via
+`flake.nix` we should prompt - for the first time - and let user review what is
+going to happen.
+
+
+```shell
+$ nix build --option substitutors https://cache.example.org
+------------------------------------------------------------------------
+  Warning! A security related question need to be answered.
+------------------------------------------------------------------------
+  The following substitutors will be used to in `my-project`: 
+    - https://cache.example.org
+
+  Do you allow `my-project` to use above mentioned substitutors?
+    [y/N] |> y
+```
+
+# Output
+
+Terminal output can be quite limiting in many ways. Which should forces us to
+think about the experience even more. As with every design the output is a
+compromise between being terse and being verbose, between showing help to
+beginners and annoying advance users. For this it is important that we know
+what are the priorities.
+
+Nix command line should be first and foremost written with beginners in mind.
+But users wont stay beginners for long and what was once useful might quickly
+become annoying. There is no golden rule that we can give in this guideline
+that would make it easier how to draw a line and find best compromise.
+
+What we would encourage is to **build prototypes**, do some **user testing**
+and collect **feedback**. Then repeat the cycle few times.
+
+First design the *happy path* and only after your iron it out, continue to work
+on **edge cases** (handling and displaying errors, changes of the output by
+certain `OPTIONS`, etc…)
+
+## Follow best practices
+
+Needless to say we Nix must be a good citizen and follow best practices in
+command line.
+
+In short: **STDOUT is for output, STDERR is for (human) messaging.**
+
+STDOUT and STDERR provide a way for you to output messages to the user while
+also allowing them to redirect content to a file. For example:
+
+```shell
+$ nix build > build.txt
+------------------------------------------------------------------------
+  Error! Atrribute `bin` missing at (1:94) from string.
+------------------------------------------------------------------------
+
+  1| with import <nixpkgs> { }; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (surge.bin) ]; } ""
+```
+
+Because this warning is on STDERR, it doesn’t end up in the file.
+
+But not everything on STDERR is an error though. For example, you can run `nix
+build` and collect logs in a file while still seeing the progress.
+
+```
+$ nix build > build.txt
+  Evaluated 1234 files in 1.2 seconds
+ Downloaded python3.8-poetry 1.2.3 in 5.3 seconds
+ Downloaded python3.8-requests 1.2.3 in 5.3 seconds
+------------------------------------------------------------------------
+      Press `v` to increase logs verbosity
+         |> `?` to see other options
+------------------------------------------------------------------------
+      Learn something new with every build...
+         |> See last logs of a build with `nix log --last` command.
+------------------------------------------------------------------------
+  Evaluated my-project 1.2.3 in 14.43 seconds
+Downloading [12 / 200]
+         |> firefox 1.2.3 [#########>       ] 10Mb/s | 2min left
+   Building [2 / 20]
+         |> glibc 1.2.3 -> buildPhase: <last log line>
+------------------------------------------------------------------------
+```
+
+## Errors (WIP)
+
+**TODO**: Once we have implementation for the *happy path* then we will think
+how to present errors.
+
+## Not only for humans
+
+Terse, machine-readable output formats can also be useful but shouldn’t get in
+the way of making beautiful CLI output. When needed, commands should offer a
+`--json` flag to allow users to easily parse and script the CLI.
+
+When TTY is not detected on STDOUT we should remove all design elements (no
+colors, no emojis and using ASCII instead of Unicode symbols). The same should
+happen when TTY is not detected on STDERR. We should not display progress /
+status section, but only print warnings and errors.
+
+## Dialog with the user
+
+CLIs don't always make it clear when an action has taken place. For every
+action a user performs, your CLI should provide an equal and appropriate
+reaction, clearly highlighting the what just happened. For example:
+
+```shell
+$ nix build
+ Downloaded python3.8-poetry 1.2.3 in 5.3 seconds
+ Downloaded python3.8-requests 1.2.3 in 5.3 seconds
+...
+   Success! You have successfully built my-project.
+$
+```
+
+Above command clearly states that command successfully completed. And in case
+of `nix build`, which is a command that might take some time to complete, it is
+equally important to also show that a command started.
+
+## Text alignment 
+
+Text alignment is the number one design element that will present all of the
+Nix commands as a family and not as separate tools glued together.
+
+The format we should follow is:
+
+```shell
+$ nix COMMAND
+   VERB_1 NOUN and other words
+  VERB__1 NOUN and other words
+       |> Some details 
+```
+
+Few rules that we can extract from above example:
+
+- Each line should start at least with one space.
+- First word should be a VERB and must be aligned to the right.
+- Second word should be a NOUN and must be aligned to the left.
+- If you can not find a good VERB / NOUN pair, don’t worry make it as
+  understandable to the user as possible.
+- More details of each line can be provided by `|>` character which is serving
+  as the first word when aligning the text
+
+Don’t forget you should also test your terminal output with colors and emojis
+off (`--no-colors --no-emojis`).
+
+## Dim / Bright
+
+After comparing few terminals with different color schemes we would **recommend
+to avoid using dimmed text**. The difference from the rest of the text is very
+little in many terminal and color scheme combinations. Sometimes the difference
+is not even notable, therefore relying on it wouldn’t make much sense.
+
+**The bright text is much better supported** across terminals and color
+schemes. Most of the time the difference is perceived as if the bright text
+would be bold. 
+
+## Colors
+
+Humans are already conditioned by society to attach certain meaning to certain
+colors. While the meaning is not universal, a simple collection of colors is
+used to represent basic emotions. 
+
+Colors that can be used in output
+
+- Red = error, danger, stop
+- Green = success, good
+- Yellow/Orange = proceed with caution, warning, in progress
+- Blue/Magenta = stability, calm
+
+While colors are nice, when command line is used by machines (in automation
+scripts) you want to remove the colors. There should be a global `--no-colors`
+option that would remove the colors.
+
+## Special (Unicode) characters
+
+Most of the terminal have good support for Unicode characters and you should
+use them in your output by default. But always have a backup solution that is
+implemented only with ASCII characters and will be used when `--ascii` option
+is going to be passed in. Please make sure that you test your output also
+without Unicode characters
+
+More they showing all the different Unicode characters it is important to
+**establish common set of characters** that we use for certain situations.
+
+## Emojis
+
+Emojis help channel emotions even better than text, colors and special
+characters.
+
+We recommend **keeping the set of emojis to a minimum**. This will enable each
+emoji to stand out more.
+
+As not everybody is happy about emojis we should provide an `--no-emojis`
+option to disable them. Please make sure that you test your output also without
+emojis.
+
+## Tables
+
+All commands that are listing certain data can be implemented in some sort of a
+table. It’s important that each row of your output is a single ‘entry’ of data.
+Never output table borders. It’s noisy and a huge pain for parsing using other
+tools such as `grep`.
+
+Be mindful of the screen width. Only show a few columns by default with the
+table header, for more the table can be manipulated by the following options:
+
+- `--no-headers`: Show column headers by default but allow to hide them.
+- `--columns`: Comma-separated list of column names to add.
+- `--sort`: Allow sorting by column. Allow inverse and multi-column sort as well.
+
+## Interactive output
+
+Interactive output was selected to be able to strike the balance between
+beginners and advance users. While the default output will target beginners it
+can, with a few key strokes, be changed into and advance introspection tool.
+
+### Progress
+
+For longer running commands we should provide and overview of the progress.
+This is shown best in `nix build` example:
+
+```shell
+$ nix build
+    Started building my-project 1.2.3
+ Downloaded python3.8-poetry 1.2.3 in 5.3 seconds
+ Downloaded python3.8-requests 1.2.3 in 5.3 seconds
+------------------------------------------------------------------------
+      Press `v` to increase logs verbosity
+         |> `?` to see other options
+------------------------------------------------------------------------
+      Learn something new with every build...
+         |> See last logs of a build with `nix log --last` command.
+------------------------------------------------------------------------
+  Evaluated my-project 1.2.3 in 14.43 seconds
+Downloading [12 / 200]
+         |> firefox 1.2.3 [#########>       ] 10Mb/s | 2min left
+   Building [2 / 20]
+         |> glibc 1.2.3 -> buildPhase: <last log line>
+------------------------------------------------------------------------
+```
+
+### Search
+
+Use a `fzf` like fuzzy search when there are multiple options to choose from.
+
+```shell
+$ nix init
+Initializing Nix project at `/path/to/here`.
+      Select a template for you new project:
+          |> py
+             template#python-pip
+             template#python-poetry
+             [ Showing 2 templates from 1345 templates ]
+```
+
+### Prompt
+
+In some situations we need to prompt the user and inform the user about what is
+going to happen.
+
+```shell
+$ nix build --option substitutors https://cache.example.org
+------------------------------------------------------------------------
+  Warning! A security related question need to be answered.
+------------------------------------------------------------------------
+  The following substitutors will be used to in `my-project`: 
+    - https://cache.example.org
+
+  Do you allow `my-project` to use above mentioned substitutors?
+    [y/N] |> y
+```
+
+## Verbosity
+
+There are many ways that you can control verbosity.
+
+Verbosity levels are: 
+
+- `ERROR` (level 0)
+- `WARN` (level 1)
+- `NOTICE` (level 2)
+- `INFO` (level 3)
+- `TALKATIVE` (level 4)
+- `CHATTY` (level 5)
+- `DEBUG` (level 6)
+- `VOMIT` (level 7)
+
+The default level that the command starts is `ERROR`. The simplest way to
+increase the verbosity by stacking `-v` option (eg: `-vvv == level 3 == INFO`).
+There are also two shortcuts, `--debug` to run in `DEBUG` verbosity level and
+`--quiet` to run in `ERROR` verbosity level.
+
+----------
+
+# Appendix 1: Commands naming exceptions
+
+`nix init` and `nix repl` are well established 

doc/manual/src/contributing/contributing.md

@@ -0,0 +1 @@
+# Contributing

doc/manual/src/expressions/language-operators.md

@@ -25,5 +25,4 @@ order of precedence (from strongest to weakest binding).
 | Inequality               | *e1* `!=` *e2*                      | none          | Inequality.                                                                                                                                                                                                                   | 11         |
 | Logical AND              | *e1* `&&` *e2*                      | left          | Logical AND.                                                                                                                                                                                                                  | 12         |
 | Logical OR               | *e1* `\|\|` *e2*                    | left          | Logical OR.                                                                                                                                                                                                                   | 13         |
-| Logical Implication      | *e1* `->` *e2*                      | none          | Logical implication (equivalent to `!e1 \|\|
-        e2`).                                                                                                                                                                    | 14         |
+| Logical Implication      | *e1* `->` *e2*                      | none          | Logical implication (equivalent to `!e1 \|\| e2`).                                                                                                                                                                            | 14         |

doc/manual/src/installation/installing-binary.md

@@ -195,7 +195,7 @@ If you are comfortable navigating these tradeoffs, you can encrypt the
 volume with something along the lines of:
 
 ```console
-alice$ diskutil apfs enableFileVault /nix -user disk
+$ diskutil apfs enableFileVault /nix -user disk
 ```
 
 ## Symlink the Nix store to a custom location
@@ -234,13 +234,13 @@ as a helpful reference if you run into trouble.
     without a reboot:
     
     ```console
-    alice$ /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -B
+    $ /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -B
     ```
 
 3.  Create the new APFS volume with diskutil:
     
     ```console
-    alice$ sudo diskutil apfs addVolume diskX APFS 'Nix Store' -mountpoint /nix
+    $ sudo diskutil apfs addVolume diskX APFS 'Nix Store' -mountpoint /nix
     ```
 
 4.  Using `vifs`, add the new mount to `/etc/fstab`. If it doesn't
@@ -280,10 +280,10 @@ it somewhere (e.g. in `/tmp`), and then run the script named `install`
 inside the binary tarball:
 
 ```console
-alice$ cd /tmp
-alice$ tar xfj nix-1.8-x86_64-darwin.tar.bz2
-alice$ cd nix-1.8-x86_64-darwin
-alice$ ./install
+$ cd /tmp
+$ tar xfj nix-1.8-x86_64-darwin.tar.bz2
+$ cd nix-1.8-x86_64-darwin
+$ ./install
 ```
 
 If you need to edit the multi-user installation script to use different

doc/manual/src/installation/prerequisites-source.md

@@ -30,7 +30,7 @@
     have bzip2 installed, including development headers and libraries.
     If your distribution does not provide these, you can obtain bzip2
     from
-    <https://web.archive.org/web/20180624184756/http://www.bzip.org/>.
+    <https://sourceware.org/bzip2/>.
 
   - `liblzma`, which is provided by XZ Utils. If your distribution does
     not provide this, you can get it from <https://tukaani.org/xz/>.

doc/manual/src/introduction.md

@@ -165,10 +165,10 @@ You’re then dropped into a shell where you can edit, build and test
 the package:
 
 ```console
-[nix-shell]$ tar xf $src
+[nix-shell]$ unpackPhase
 [nix-shell]$ cd pan-*
-[nix-shell]$ ./configure
-[nix-shell]$ make
+[nix-shell]$ configurePhase
+[nix-shell]$ buildPhase
 [nix-shell]$ ./pan/gui/pan
 ```
 

doc/manual/utils.nix

@@ -1,7 +1,15 @@
 with builtins;
 
-{
+rec {
   splitLines = s: filter (x: !isList x) (split "\n" s);
 
   concatStrings = concatStringsSep "";
+
+  # FIXME: O(n^2)
+  unique = foldl' (acc: e: if elem e acc then acc else acc ++ [ e ]) [];
+
+  nameValuePair = name: value: { inherit name value; };
+
+  filterAttrs = pred: set:
+    listToAttrs (concatMap (name: let v = set.${name}; in if pred name v then [(nameValuePair name v)] else []) (attrNames set));
 }

flake.lock

@@ -1,39 +1,22 @@
 {
   "nodes": {
-    "lowdown-src": {
-      "flake": false,
-      "locked": {
-        "lastModified": 1598695561,
-        "narHash": "sha256-gyH/5j+h/nWw0W8AcR2WKvNBUsiQ7QuxqSJNXAwV+8E=",
-        "owner": "kristapsdz",
-        "repo": "lowdown",
-        "rev": "1705b4a26fbf065d9574dce47a94e8c7c79e052f",
-        "type": "github"
-      },
-      "original": {
-        "owner": "kristapsdz",
-        "repo": "lowdown",
-        "type": "github"
-      }
-    },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1591633336,
-        "narHash": "sha256-oVXv4xAnDJB03LvZGbC72vSVlIbbJr8tpjEW5o/Fdek=",
+        "lastModified": 1602702596,
+        "narHash": "sha256-fqJ4UgOb4ZUnCDIapDb4gCrtAah5Rnr2/At3IzMitig=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "70717a337f7ae4e486ba71a500367cad697e5f09",
+        "rev": "ad0d20345219790533ebe06571f82ed6b034db31",
         "type": "github"
       },
       "original": {
         "id": "nixpkgs",
-        "ref": "nixos-20.03-small",
+        "ref": "nixos-20.09-small",
         "type": "indirect"
       }
     },
     "root": {
       "inputs": {
-        "lowdown-src": "lowdown-src",
         "nixpkgs": "nixpkgs"
       }
     }

flake.nix

@@ -1,10 +1,10 @@
 {
   description = "The purely functional package manager";
 
-  inputs.nixpkgs.url = "nixpkgs/nixos-20.03-small";
-  inputs.lowdown-src = { url = "github:kristapsdz/lowdown"; flake = false; };
+  inputs.nixpkgs.url = "nixpkgs/nixos-20.09-small";
+  #inputs.lowdown-src = { url = "github:kristapsdz/lowdown"; flake = false; };
 
-  outputs = { self, nixpkgs, lowdown-src }:
+  outputs = { self, nixpkgs }:
 
     let
 
@@ -12,11 +12,13 @@
       versionSuffix =
         if officialRelease
         then ""
-        else "pre${builtins.substring 0 8 (self.lastModifiedDate or self.lastModified)}_${self.shortRev or "dirty"}";
+        else "pre${builtins.substring 0 8 (self.lastModifiedDate or self.lastModified or "19700101")}_${self.shortRev or "dirty"}";
 
       officialRelease = false;
 
-      systems = [ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ];
+      linux64BitSystems = [ "x86_64-linux" "aarch64-linux" ];
+      linuxSystems = linux64BitSystems ++ [ "i686-linux" ];
+      systems = linuxSystems ++ [ "x86_64-darwin" ];
 
       forAllSystems = f: nixpkgs.lib.genAttrs systems (system: f system);
 
@@ -61,34 +63,41 @@
             "LDFLAGS=-fuse-ld=gold"
           ];
 
-        buildDeps =
-          [ bison
-            flex
-            mdbook
-            lowdown
-            autoconf-archive
-            autoreconfHook
 
-            curl
+        nativeBuildDeps =
+          [
+            buildPackages.bison
+            buildPackages.flex
+            (lib.getBin buildPackages.lowdown)
+            buildPackages.mdbook
+            buildPackages.autoconf-archive
+            buildPackages.autoreconfHook
+            buildPackages.pkgconfig
+
+            # Tests
+            buildPackages.git
+            buildPackages.mercurial
+            buildPackages.jq
+          ];
+
+        buildDeps =
+          [ curl
             bzip2 xz brotli zlib editline
-            openssl pkgconfig sqlite
+            openssl sqlite
             libarchive
             boost
             nlohmann_json
-
-            # Tests
-            git
-            mercurial
-            jq
+            lowdown
             gmock
           ]
           ++ lib.optionals stdenv.isLinux [libseccomp utillinuxMinimal]
-          ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium
-          ++ lib.optional (stdenv.isLinux || stdenv.isDarwin)
-            (aws-sdk-cpp.override {
-              apis = ["s3" "transfer"];
-              customMemoryManagement = false;
-            });
+          ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium;
+
+        awsDeps = lib.optional (stdenv.isLinux || stdenv.isDarwin)
+          (aws-sdk-cpp.override {
+            apis = ["s3" "transfer"];
+            customMemoryManagement = false;
+          });
 
         propagatedDeps =
           [ (boehmgc.override { enableLargeConfig = true; })
@@ -106,8 +115,9 @@
       # 'nix.perl-bindings' packages.
       overlay = final: prev: {
 
-        nix = with final; with commonDeps pkgs; (stdenv.mkDerivation {
+        nix = with final; with commonDeps pkgs; stdenv.mkDerivation {
           name = "nix-${version}";
+          inherit version;
 
           src = self;
 
@@ -115,7 +125,8 @@
 
           outputs = [ "out" "dev" "doc" ];
 
-          buildInputs = buildDeps;
+          nativeBuildInputs = nativeBuildDeps;
+          buildInputs = buildDeps ++ awsDeps;
 
           propagatedBuildInputs = propagatedDeps;
 
@@ -152,21 +163,23 @@
           installCheckFlags = "sysconfdir=$(out)/etc";
 
           separateDebugInfo = true;
-        }) // {
 
-          perl-bindings = with final; stdenv.mkDerivation {
+          passthru.perl-bindings = with final; stdenv.mkDerivation {
             name = "nix-perl-${version}";
 
             src = self;
 
+            nativeBuildInputs =
+              [ buildPackages.autoconf-archive
+                buildPackages.autoreconfHook
+                buildPackages.pkgconfig
+              ];
+
             buildInputs =
-              [ autoconf-archive
-                autoreconfHook
-                nix
+              [ nix
                 curl
                 bzip2
                 xz
-                pkgconfig
                 pkgs.perl
                 boost
                 nlohmann_json
@@ -185,27 +198,25 @@
 
         };
 
-        lowdown = with final; stdenv.mkDerivation {
-          name = "lowdown-0.7.1";
+        lowdown = with final; stdenv.mkDerivation rec {
+          name = "lowdown-0.8.0";
 
-          /*
           src = fetchurl {
-            url = https://kristaps.bsd.lv/lowdown/snapshots/lowdown-0.7.1.tar.gz;
-            hash = "sha512-1daoAQfYD0LdhK6aFhrSQvadjc5GsSPBZw0fJDb+BEHYMBLjqiUl2A7H8N+l0W4YfGKqbsPYSrCy4vct+7U6FQ==";
+            url = "https://kristaps.bsd.lv/lowdown/snapshots/${name}.tar.gz";
+            hash = "sha512-U9WeGoInT9vrawwa57t6u9dEdRge4/P+0wLxmQyOL9nhzOEUU2FRz2Be9H0dCjYE7p2v3vCXIYk40M+jjULATw==";
           };
-          */
 
-          src = lowdown-src;
+          #src = lowdown-src;
 
-          outputs = [ "out" "dev" ];
+          outputs = [ "out" "bin" "dev" ];
 
-          buildInputs = [ which ];
+          nativeBuildInputs = [ which ];
 
           configurePhase =
             ''
               ./configure \
                 PREFIX=${placeholder "dev"} \
-                BINDIR=${placeholder "out"}/bin
+                BINDIR=${placeholder "bin"}/bin
             '';
         };
 
@@ -214,10 +225,12 @@
       hydraJobs = {
 
         # Binary package for various platforms.
-        build = nixpkgs.lib.genAttrs systems (system: nixpkgsFor.${system}.nix);
+        build = nixpkgs.lib.genAttrs systems (system: self.packages.${system}.nix);
+
+        buildStatic = nixpkgs.lib.genAttrs linux64BitSystems (system: self.packages.${system}.nix-static);
 
         # Perl bindings for various platforms.
-        perlBindings = nixpkgs.lib.genAttrs systems (system: nixpkgsFor.${system}.nix.perl-bindings);
+        perlBindings = nixpkgs.lib.genAttrs systems (system: self.packages.${system}.nix.perl-bindings);
 
         # Binary tarball for various platforms, containing a Nix store
         # with the closure of 'nix' package, and the second half of
@@ -236,6 +249,7 @@
             }
             ''
               cp ${installerClosureInfo}/registration $TMPDIR/reginfo
+              cp ${./scripts/create-darwin-volume.sh} $TMPDIR/create-darwin-volume.sh
               substitute ${./scripts/install-nix-from-closure.sh} $TMPDIR/install \
                 --subst-var-by nix ${nix} \
                 --subst-var-by cacert ${cacert}
@@ -254,6 +268,7 @@
                 # SC1090: Don't worry about not being able to find
                 #         $nix/etc/profile.d/nix.sh
                 shellcheck --exclude SC1090 $TMPDIR/install
+                shellcheck $TMPDIR/create-darwin-volume.sh
                 shellcheck $TMPDIR/install-darwin-multi-user.sh
                 shellcheck $TMPDIR/install-systemd-multi-user.sh
 
@@ -269,6 +284,7 @@
               fi
 
               chmod +x $TMPDIR/install
+              chmod +x $TMPDIR/create-darwin-volume.sh
               chmod +x $TMPDIR/install-darwin-multi-user.sh
               chmod +x $TMPDIR/install-systemd-multi-user.sh
               chmod +x $TMPDIR/install-multi-user
@@ -281,11 +297,15 @@
                 --absolute-names \
                 --hard-dereference \
                 --transform "s,$TMPDIR/install,$dir/install," \
+                --transform "s,$TMPDIR/create-darwin-volume.sh,$dir/create-darwin-volume.sh," \
                 --transform "s,$TMPDIR/reginfo,$dir/.reginfo," \
                 --transform "s,$NIX_STORE,$dir/store,S" \
-                $TMPDIR/install $TMPDIR/install-darwin-multi-user.sh \
+                $TMPDIR/install \
+                $TMPDIR/create-darwin-volume.sh \
+                $TMPDIR/install-darwin-multi-user.sh \
                 $TMPDIR/install-systemd-multi-user.sh \
-                $TMPDIR/install-multi-user $TMPDIR/reginfo \
+                $TMPDIR/install-multi-user \
+                $TMPDIR/reginfo \
                 $(cat ${installerClosureInfo}/store-paths)
             '');
 
@@ -301,9 +321,26 @@
             ''
               mkdir -p $out/nix-support
 
+              # Converts /nix/store/50p3qk8kka9dl6wyq40vydq945k0j3kv-nix-2.4pre20201102_550e11f/bin/nix
+              # To 50p3qk8kka9dl6wyq40vydq945k0j3kv/bin/nix
+              tarballPath() {
+                # Remove the store prefix
+                local path=''${1#${builtins.storeDir}/}
+                # Get the path relative to the derivation root
+                local rest=''${path#*/}
+                # Get the derivation hash
+                local drvHash=''${path%%-*}
+                echo "$drvHash/$rest"
+              }
+
               substitute ${./scripts/install.in} $out/install \
                 ${pkgs.lib.concatMapStrings
-                  (system: "--replace '@binaryTarball_${system}@' $(nix --experimental-features nix-command hash-file --base16 --type sha256 ${self.hydraJobs.binaryTarball.${system}}/*.tar.xz) ")
+                  (system:
+                    '' \
+                    --replace '@tarballHash_${system}@' $(nix --experimental-features nix-command hash-file --base16 --type sha256 ${self.hydraJobs.binaryTarball.${system}}/*.tar.xz) \
+                    --replace '@tarballPath_${system}@' $(tarballPath ${self.hydraJobs.binaryTarball.${system}}/*.tar.xz) \
+                    ''
+                  )
                   [ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ]
                 } \
                 --replace '@nixVersion@' ${version}
@@ -323,7 +360,8 @@
 
             enableParallelBuilding = true;
 
-            buildInputs = buildDeps ++ propagatedDeps;
+            nativeBuildInputs = nativeBuildDeps;
+            buildInputs = buildDeps ++ propagatedDeps ++ awsDeps;
 
             dontInstall = false;
 
@@ -364,38 +402,6 @@
               inherit (self) overlay;
             });
 
-        # Test whether the binary tarball works in an Ubuntu system.
-        tests.binaryTarball =
-          with nixpkgsFor.x86_64-linux;
-          vmTools.runInLinuxImage (runCommand "nix-binary-tarball-test"
-            { diskImage = vmTools.diskImages.ubuntu1204x86_64;
-            }
-            ''
-              set -x
-              useradd -m alice
-              su - alice -c 'tar xf ${self.hydraJobs.binaryTarball.x86_64-linux}/*.tar.*'
-              mkdir /dest-nix
-              mount -o bind /dest-nix /nix # Provide a writable /nix.
-              chown alice /nix
-              su - alice -c '_NIX_INSTALLER_TEST=1 ./nix-*/install'
-              su - alice -c 'nix-store --verify'
-              su - alice -c 'PAGER= nix-store -qR ${self.hydraJobs.build.x86_64-linux}'
-
-              # Check whether 'nix upgrade-nix' works.
-              cat > /tmp/paths.nix <<EOF
-              {
-                x86_64-linux = "${self.hydraJobs.build.x86_64-linux}";
-              }
-              EOF
-              su - alice -c 'nix --experimental-features nix-command upgrade-nix -vvv --nix-store-paths-url file:///tmp/paths.nix'
-              (! [ -L /home/alice/.profile-1-link ])
-              su - alice -c 'PAGER= nix-store -qR ${self.hydraJobs.build.x86_64-linux}'
-
-              mkdir -p $out/nix-support
-              touch $out/nix-support/hydra-build-products
-              umount /nix
-            '');
-
         /*
         # Check whether we can still evaluate all of Nixpkgs.
         tests.evalNixpkgs =
@@ -429,6 +435,43 @@
 
       packages = forAllSystems (system: {
         inherit (nixpkgsFor.${system}) nix;
+      } // nixpkgs.lib.optionalAttrs (builtins.elem system linux64BitSystems) {
+        nix-static = let
+          nixpkgs = nixpkgsFor.${system}.pkgsStatic;
+        in with commonDeps nixpkgs; nixpkgs.stdenv.mkDerivation {
+          name = "nix-${version}";
+
+          src = self;
+
+          VERSION_SUFFIX = versionSuffix;
+
+          outputs = [ "out" "dev" "doc" ];
+
+          nativeBuildInputs = nativeBuildDeps;
+          buildInputs = buildDeps ++ propagatedDeps;
+
+          configureFlags = [ "--sysconfdir=/etc" ];
+
+          enableParallelBuilding = true;
+
+          makeFlags = "profiledir=$(out)/etc/profile.d";
+
+          doCheck = true;
+
+          installFlags = "sysconfdir=$(out)/etc";
+
+          postInstall = ''
+            mkdir -p $doc/nix-support
+            echo "doc manual $doc/share/doc/nix/manual" >> $doc/nix-support/hydra-build-products
+            mkdir -p $out/nix-support
+            echo "file binary-dist $out/bin/nix" >> $out/nix-support/hydra-build-products
+          '';
+
+          doInstallCheck = true;
+          installCheckFlags = "sysconfdir=$(out)/etc";
+
+          stripAllList = ["bin"];
+        };
       });
 
       defaultPackage = forAllSystems (system: self.packages.${system}.nix);
@@ -442,7 +485,8 @@
 
           outputs = [ "out" "dev" "doc" ];
 
-          buildInputs = buildDeps ++ propagatedDeps ++ perlDeps;
+          nativeBuildInputs = nativeBuildDeps;
+          buildInputs = buildDeps ++ propagatedDeps ++ awsDeps ++ perlDeps;
 
           inherit configureFlags;
 

local.mk

@@ -1,9 +1,3 @@
-ifeq ($(MAKECMDGOALS), dist)
-  dist-files += $(shell cat .dist-files)
-endif
-
-dist-files += configure config.h.in perl/configure
-
 clean-files += Makefile.config
 
 GLOBAL_CXXFLAGS += -Wno-deprecated-declarations

misc/systemd/local.mk

@@ -2,4 +2,6 @@ ifeq ($(OS), Linux)
 
   $(foreach n, nix-daemon.socket nix-daemon.service, $(eval $(call install-file-in, $(d)/$(n), $(prefix)/lib/systemd/system, 0644)))
 
+  clean-files += $(d)/nix-daemon.socket $(d)/nix-daemon.service
+
 endif

misc/upstart/local.mk

@@ -2,4 +2,6 @@ ifeq ($(OS), Linux)
 
   $(foreach n, nix-daemon.conf, $(eval $(call install-file-in, $(d)/$(n), $(sysconfdir)/init, 0644)))
 
+  clean-files += $(d)/nix-daemon.conf
+
 endif

mk/dist.mk

@@ -1,17 +0,0 @@
-ifdef PACKAGE_NAME
-
-dist-name = $(PACKAGE_NAME)-$(PACKAGE_VERSION)
-
-dist: $(dist-name).tar.bz2 $(dist-name).tar.xz
-
-$(dist-name).tar.bz2: $(dist-files)
-	$(trace-gen) tar cfj $@ $(sort $(dist-files)) --transform 's,^,$(dist-name)/,'
-
-$(dist-name).tar.xz: $(dist-files)
-	$(trace-gen) tar cfJ $@ $(sort $(dist-files)) --transform 's,^,$(dist-name)/,'
-
-clean-files += $(dist-name).tar.bz2 $(dist-name).tar.xz
-
-print-top-help += echo "  dist: Generate a source distribution";
-
-endif

mk/lib.mk

@@ -10,7 +10,6 @@ bin-scripts :=
 noinst-scripts :=
 man-pages :=
 install-tests :=
-dist-files :=
 OS = $(shell uname -s)
 
 
@@ -112,9 +111,6 @@ $(foreach test, $(install-tests), $(eval $(call run-install-test,$(test))))
 $(foreach file, $(man-pages), $(eval $(call install-data-in, $(file), $(mandir)/man$(patsubst .%,%,$(suffix $(file))))))
 
 
-include mk/dist.mk
-
-
 .PHONY: default all man help
 
 all: $(programs-list) $(libs-list) $(jars-list) $(man-pages)

mk/libraries.mk

@@ -159,5 +159,4 @@ define build-library
   libs-list += $$($(1)_PATH)
   endif
   clean-files += $$(_d)/*.a $$(_d)/*.$(SO_EXT) $$(_d)/*.o $$(_d)/.*.dep $$($(1)_DEPS) $$($(1)_OBJS)
-  dist-files += $$(_srcs)
 endef

mk/programs.mk

@@ -79,7 +79,6 @@ define build-program
 
   programs-list += $$($(1)_PATH)
   clean-files += $$($(1)_PATH) $$(_d)/*.o $$(_d)/.*.dep $$($(1)_DEPS) $$($(1)_OBJS)
-  dist-files += $$(_srcs)
 
   # Phony target to run this program (typically as a dependency of 'check').
   .PHONY: $(1)_RUN

nix-rust/local.mk

@@ -30,8 +30,6 @@ ifeq ($(OS), Darwin)
 	install_name_tool -id $@ $@
 endif
 
-dist-files += $(d)/vendor
-
 clean: clean-rust
 
 clean-rust:

perl/Makefile.config.in

@@ -2,7 +2,6 @@ CC = @CC@
 CFLAGS = @CFLAGS@
 CXX = @CXX@
 CXXFLAGS = @CXXFLAGS@
-HAVE_SODIUM = @HAVE_SODIUM@
 PACKAGE_NAME = @PACKAGE_NAME@
 PACKAGE_VERSION = @PACKAGE_VERSION@
 SODIUM_LIBS = @SODIUM_LIBS@

perl/configure.ac

@@ -40,11 +40,7 @@ AC_SUBST(perllibdir, [${libdir}/perl5/site_perl/$perlversion/$perlarchname])
 AC_MSG_RESULT($perllibdir)
 
 # Look for libsodium, an optional dependency.
-PKG_CHECK_MODULES([SODIUM], [libsodium],
-  [AC_DEFINE([HAVE_SODIUM], [1], [Whether to use libsodium for cryptography.])
-   CXXFLAGS="$SODIUM_CFLAGS $CXXFLAGS"
-   have_sodium=1], [have_sodium=])
-AC_SUBST(HAVE_SODIUM, [$have_sodium])
+PKG_CHECK_MODULES([SODIUM], [libsodium], [CXXFLAGS="$SODIUM_CFLAGS $CXXFLAGS"])
 
 # Check for the required Perl dependencies (DBI and DBD::SQLite).
 perlFlags="-I$perllibdir"

perl/lib/Nix/Store.xs

@@ -14,9 +14,7 @@
 #include "util.hh"
 #include "crypto.hh"
 
-#if HAVE_SODIUM
 #include <sodium.h>
-#endif
 
 
 using namespace nix;
@@ -110,10 +108,14 @@ SV * queryPathInfo(char * path, int base32)
             XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
             mXPUSHi(info->registrationTime);
             mXPUSHi(info->narSize);
-            AV * arr = newAV();
+            AV * refs = newAV();
             for (auto & i : info->references)
-                av_push(arr, newSVpv(store()->printStorePath(i).c_str(), 0));
-            XPUSHs(sv_2mortal(newRV((SV *) arr)));
+                av_push(refs, newSVpv(store()->printStorePath(i).c_str(), 0));
+            XPUSHs(sv_2mortal(newRV((SV *) refs)));
+            AV * sigs = newAV();
+            for (auto & i : info->sigs)
+                av_push(sigs, newSVpv(i.c_str(), 0));
+            XPUSHs(sv_2mortal(newRV((SV *) sigs)));
         } catch (Error & e) {
             croak("%s", e.what());
         }
@@ -235,12 +237,8 @@ SV * convertHash(char * algo, char * s, int toBase32)
 SV * signString(char * secretKey_, char * msg)
     PPCODE:
         try {
-#if HAVE_SODIUM
             auto sig = SecretKey(secretKey_).signDetached(msg);
             XPUSHs(sv_2mortal(newSVpv(sig.c_str(), sig.size())));
-#else
-            throw Error("Nix was not compiled with libsodium, required for signed binary cache support");
-#endif
         } catch (Error & e) {
             croak("%s", e.what());
         }
@@ -249,7 +247,6 @@ SV * signString(char * secretKey_, char * msg)
 int checkSignature(SV * publicKey_, SV * sig_, char * msg)
     CODE:
         try {
-#if HAVE_SODIUM
             STRLEN publicKeyLen;
             unsigned char * publicKey = (unsigned char *) SvPV(publicKey_, publicKeyLen);
             if (publicKeyLen != crypto_sign_PUBLICKEYBYTES)
@@ -261,9 +258,6 @@ int checkSignature(SV * publicKey_, SV * sig_, char * msg)
                 throw Error("signature is not valid");
 
             RETVAL = crypto_sign_verify_detached(sig, (unsigned char *) msg, strlen(msg), publicKey) == 0;
-#else
-            throw Error("Nix was not compiled with libsodium, required for signed binary cache support");
-#endif
         } catch (Error & e) {
             croak("%s", e.what());
         }

scripts/create-darwin-volume.sh

@@ -5,42 +5,13 @@ root_disk() {
     diskutil info -plist /
 }
 
-apfs_volumes_for() {
-    disk=$1
-    diskutil apfs list -plist "$disk"
-}
-
-disk_identifier() {
-    xpath "/plist/dict/key[text()='ParentWholeDisk']/following-sibling::string[1]/text()" 2>/dev/null
-}
-
-volume_list_true() {
-    key=$1
-    xpath "/plist/dict/array/dict/key[text()='Volumes']/following-sibling::array/dict/key[text()='$key']/following-sibling::true[1]" 2> /dev/null
-}
-
-volume_get_string() {
-    key=$1 i=$2
-    xpath "/plist/dict/array/dict/key[text()='Volumes']/following-sibling::array/dict[$i]/key[text()='$key']/following-sibling::string[1]/text()" 2> /dev/null
+# i.e., "disk1"
+root_disk_identifier() {
+    diskutil info -plist / | xmllint --xpath "/plist/dict/key[text()='ParentWholeDisk']/following-sibling::string[1]/text()" -
 }
 
 find_nix_volume() {
-    disk=$1
-    i=1
-    volumes=$(apfs_volumes_for "$disk")
-    while true; do
-        name=$(echo "$volumes" | volume_get_string "Name" "$i")
-        if [ -z "$name" ]; then
-            break
-        fi
-        case "$name" in
-            [Nn]ix*)
-                echo "$name"
-                break
-                ;;
-        esac
-        i=$((i+1))
-    done
+    diskutil apfs list -plist "$1" | xmllint --xpath "(/plist/dict/array/dict/key[text()='Volumes']/following-sibling::array/dict/key[text()='Name']/following-sibling::string[starts-with(translate(text(),'N','n'),'nix')]/text())[1]" - 2>/dev/null || true
 }
 
 test_fstab() {
@@ -55,6 +26,20 @@ test_synthetic_conf() {
     grep -q "^nix$" /etc/synthetic.conf 2>/dev/null
 }
 
+# Create the paths defined in synthetic.conf, saving us a reboot.
+create_synthetic_objects(){
+    # Big Sur takes away the -B flag we were using and replaces it
+    # with a -t flag that appears to do the same thing (but they
+    # don't behave exactly the same way in terms of return values).
+    # This feels a little dirty, but as far as I can tell the
+    # simplest way to get the right one is to just throw away stderr
+    # and call both... :]
+    {
+        /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -t || true # Big Sur
+        /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -B || true # Catalina
+    } >/dev/null 2>&1
+}
+
 test_nix() {
     test -d "/nix"
 }
@@ -89,9 +74,7 @@ test_t2_chip_present(){
 }
 
 test_filevault_in_use() {
-    disk=$1
-    #      list vols on disk | get value of Filevault key | value is true
-    apfs_volumes_for "$disk" | volume_list_true FileVault | grep -q true
+    fdesetup isactive >/dev/null
 }
 
 # use after error msg for conditions we don't understand
@@ -132,7 +115,7 @@ main() {
 
     if ! test_nix; then
         echo "Creating mountpoint for /nix..." >&2
-        /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -B || true
+        create_synthetic_objects # the ones we defined in synthetic.conf
         if ! test_nix; then
             sudo mkdir -p /nix 2>/dev/null || true
         fi
@@ -143,12 +126,12 @@ main() {
         fi
     fi
 
-    disk=$(root_disk | disk_identifier)
+    disk="$(root_disk_identifier)"
     volume=$(find_nix_volume "$disk")
     if [ -z "$volume" ]; then
         echo "Creating a Nix Store volume..." >&2
 
-        if test_filevault_in_use "$disk"; then
+        if test_filevault_in_use; then
             # TODO: Not sure if it's in-scope now, but `diskutil apfs list`
             # shows both filevault and encrypted at rest status, and it
             # may be the more semantic way to test for this? It'll show
@@ -178,6 +161,7 @@ main() {
     if ! test_fstab; then
         echo "Configuring /etc/fstab..." >&2
         label=$(echo "$volume" | sed 's/ /\\040/g')
+        # shellcheck disable=SC2209
         printf "\$a\nLABEL=%s /nix apfs rw,nobrowse\n.\nwq\n" "$label" | EDITOR=ed sudo vifs
     fi
 }

scripts/install-darwin-multi-user.sh

@@ -37,6 +37,13 @@ poly_service_setup_note() {
 EOF
 }
 
+poly_extra_try_me_commands(){
+  :
+}
+poly_extra_setup_instructions(){
+  :
+}
+
 poly_configure_nix_daemon_service() {
     _sudo "to set up the nix-daemon as a LaunchDaemon" \
           cp -f "/nix/var/nix/profiles/default$PLIST_DEST" "$PLIST_DEST"

scripts/install-multi-user.sh

@@ -71,11 +71,9 @@ uninstall_directions() {
     subheader "Uninstalling nix:"
     local step=0
 
-    if [ -e /run/systemd/system ] && poly_service_installed_check; then
+    if poly_service_installed_check; then
         step=$((step + 1))
         poly_service_uninstall_directions "$step"
-    else
-        step=$((step + 1))
     fi
 
     for profile_target in "${PROFILE_TARGETS[@]}"; do
@@ -255,40 +253,20 @@ function finish_success {
         echo "To try again later, run \"sudo -i nix-channel --update nixpkgs\"."
     fi
 
-    if [ -e /run/systemd/system ]; then
-        cat <<EOF
+    cat <<EOF
 
 Before Nix will work in your existing shells, you'll need to close
 them and open them again. Other than that, you should be ready to go.
 
 Try it! Open a new terminal, and type:
-
+$(poly_extra_try_me_commands)
   $ nix-shell -p nix-info --run "nix-info -m"
-
+$(poly_extra_setup_instructions)
 Thank you for using this installer. If you have any feedback, don't
 hesitate:
 
 $(contactme)
 EOF
-    else
-        cat <<EOF
-
-Before Nix will work in your existing shells, you'll need to close
-them and open them again. Other than that, you should be ready to go.
-
-Try it! Open a new terminal, and type:
-
-  $ sudo nix-daemon
-  $ nix-shell -p nix-info --run "nix-info -m"
-
-Additionally, you may want to add nix-daemon to your init-system.
-
-Thank you for using this installer. If you have any feedback, don't
-hesitate:
-
-$(contactme)
-EOF
-   fi
 
 }
 
@@ -630,24 +608,20 @@ EOF
 }
 
 configure_shell_profile() {
-    # If there is an /etc/profile.d directory, we want to ensure there
-    # is a nix.sh within it, so we can use the following loop to add
-    # the source lines to it. Note that I'm _not_ adding the source
-    # lines here, because we want to be using the regular machinery.
-    #
-    # If we go around that machinery, it becomes more complicated and
-    # adds complications to the uninstall instruction generator and
-    # old instruction sniffer as well.
-    if [ -d /etc/profile.d ]; then
-        _sudo "create a stub /etc/profile.d/nix.sh which will be updated" \
-              touch /etc/profile.d/nix.sh
-    fi
-
     for profile_target in "${PROFILE_TARGETS[@]}"; do
         if [ -e "$profile_target" ]; then
             _sudo "to back up your current $profile_target to $profile_target$PROFILE_BACKUP_SUFFIX" \
                   cp "$profile_target" "$profile_target$PROFILE_BACKUP_SUFFIX"
+        else
+            # try to create the file if its directory exists
+            target_dir="$(dirname "$profile_target")"
+            if [ -d "$target_dir" ]; then
+                _sudo "to create a stub $profile_target which will be updated" \
+                    touch "$profile_target"
+            fi
+        fi
 
+        if [ -e "$profile_target" ]; then
             shell_source_lines \
                 | _sudo "extend your $profile_target with nix-daemon settings" \
                         tee -a "$profile_target"
@@ -725,9 +699,7 @@ main() {
     setup_default_profile
     place_nix_configuration
 
-    if [ -e /run/systemd/system ]; then
-        poly_configure_nix_daemon_service
-    fi
+    poly_configure_nix_daemon_service
 
     trap finish_success EXIT
 }

scripts/install-nix-from-closure.sh

@@ -26,9 +26,20 @@ fi
 
 # macOS support for 10.12.6 or higher
 if [ "$(uname -s)" = "Darwin" ]; then
-    macos_major=$(sw_vers -productVersion | cut -d '.' -f 2)
-    macos_minor=$(sw_vers -productVersion | cut -d '.' -f 3)
-    if [ "$macos_major" -lt 12 ] || { [ "$macos_major" -eq 12 ] && [ "$macos_minor" -lt 6 ]; }; then
+    IFS='.' read macos_major macos_minor macos_patch << EOF
+$(sw_vers -productVersion)
+EOF
+    # TODO: this is a temporary speed-bump to keep people from naively installing Nix
+    # on macOS Big Sur (11.0+, 10.16+) until nixpkgs updates are ready for them.
+    # *Ideally* this is gone before next Nix release. If you're intentionally working on
+    # Nix + Big Sur, just comment out this block and be on your way :)
+    if [ "$macos_major" -gt 10 ] || { [ "$macos_major" -eq 10 ] && [ "$macos_minor" -gt 15 ]; }; then
+        echo "$0: nixpkgs isn't quite ready to support macOS $(sw_vers -productVersion) yet"
+        exit 1
+    fi
+
+    if [ "$macos_major" -lt 10 ] || { [ "$macos_major" -eq 10 ] && [ "$macos_minor" -lt 12 ]; } || { [ "$macos_minor" -eq 12 ] && [ "$macos_patch" -lt 6 ]; }; then
+        # patch may not be present; command substitution for simplicity
         echo "$0: macOS $(sw_vers -productVersion) is not supported, upgrade to 10.12.6 or higher"
         exit 1
     fi
@@ -87,10 +98,13 @@ while [ $# -gt 0 ]; do
                 echo ""
                 echo " --nix-extra-conf-file: Path to nix.conf to prepend when installing /etc/nix.conf"
                 echo ""
+                if [ -n "${INVOKED_FROM_INSTALL_IN:-}" ]; then
+                    echo " --tarball-url-prefix URL: Base URL to download the Nix tarball from."
+                fi
             ) >&2
 
             # darwin and Catalina+
-            if [ "$(uname -s)" = "Darwin" ] && [ "$macos_major" -gt 14 ]; then
+            if [ "$(uname -s)" = "Darwin" ] && { [ "$macos_major" -gt 10 ] || { [ "$macos_major" -eq 10 ] && [ "$macos_minor" -gt 14 ]; }; }; then
                 (
                     echo " --darwin-use-unencrypted-nix-store-volume: Create an APFS volume for the Nix"
                     echo "              store and mount it at /nix. This is the recommended way to create"
@@ -110,8 +124,8 @@ if [ "$(uname -s)" = "Darwin" ]; then
         "$self/create-darwin-volume.sh"
     fi
 
-    info=$(diskutil info -plist / | xpath "/plist/dict/key[text()='Writable']/following-sibling::true[1]" 2> /dev/null)
-    if ! [ -e $dest ] && [ -n "$info" ] && [ "$macos_major" -gt 14 ]; then
+    writable="$(diskutil info -plist / | xmllint --xpath "name(/plist/dict/key[text()='Writable']/following-sibling::*[1])" -)"
+    if ! [ -e $dest ] && [ "$writable" = "false" ]; then
         (
             echo ""
             echo "Installing on macOS >=10.15 requires relocating the store to an apfs volume."
@@ -152,9 +166,15 @@ fi
 mkdir -p $dest/store
 
 printf "copying Nix to %s..." "${dest}/store" >&2
+# Insert a newline if no progress is shown.
+if [ ! -t 0 ]; then
+  echo ""
+fi
 
 for i in $(cd "$self/store" >/dev/null && echo ./*); do
-    printf "." >&2
+    if [ -t 0 ]; then
+      printf "." >&2
+    fi
     i_tmp="$dest/store/$i.$$"
     if [ -e "$i_tmp" ]; then
         rm -rf "$i_tmp"

scripts/install-systemd-multi-user.sh

@@ -72,24 +72,45 @@ poly_service_setup_note() {
 EOF
 }
 
+poly_extra_try_me_commands(){
+    if [ -e /run/systemd/system ]; then
+        :
+    else
+        cat <<EOF
+  $ sudo nix-daemon
+EOF
+    fi
+}
+poly_extra_setup_instructions(){
+    if [ -e /run/systemd/system ]; then
+        :
+    else
+        cat <<EOF
+Additionally, you may want to add nix-daemon to your init-system.
+
+EOF
+    fi
+}
+
 poly_configure_nix_daemon_service() {
-    _sudo "to set up the nix-daemon service" \
-          systemctl link "/nix/var/nix/profiles/default$SERVICE_SRC"
+    if [ -e /run/systemd/system ]; then
+        _sudo "to set up the nix-daemon service" \
+              systemctl link "/nix/var/nix/profiles/default$SERVICE_SRC"
 
-    _sudo "to set up the nix-daemon socket service" \
-          systemctl enable "/nix/var/nix/profiles/default$SOCKET_SRC"
+        _sudo "to set up the nix-daemon socket service" \
+              systemctl enable "/nix/var/nix/profiles/default$SOCKET_SRC"
 
-    handle_network_proxy
+        handle_network_proxy
 
-    _sudo "to load the systemd unit for nix-daemon" \
-          systemctl daemon-reload
+        _sudo "to load the systemd unit for nix-daemon" \
+              systemctl daemon-reload
 
-    _sudo "to start the nix-daemon.socket" \
-          systemctl start nix-daemon.socket
-
-    _sudo "to start the nix-daemon.service" \
-          systemctl restart nix-daemon.service
+        _sudo "to start the nix-daemon.socket" \
+              systemctl start nix-daemon.socket
 
+        _sudo "to start the nix-daemon.service" \
+              systemctl restart nix-daemon.service
+    fi
 }
 
 poly_group_exists() {

scripts/install.in

@@ -25,16 +25,52 @@ require_util() {
 }
 
 case "$(uname -s).$(uname -m)" in
-    Linux.x86_64) system=x86_64-linux; hash=@binaryTarball_x86_64-linux@;;
-    Linux.i?86) system=i686-linux; hash=@binaryTarball_i686-linux@;;
-    Linux.aarch64) system=aarch64-linux; hash=@binaryTarball_aarch64-linux@;;
-    Darwin.x86_64) system=x86_64-darwin; hash=@binaryTarball_x86_64-darwin@;;
+    Linux.x86_64)
+        hash=@tarballHash_x86_64-linux@
+        path=@tarballPath_x86_64-linux@
+        system=x86_64-linux
+        ;;
+    Linux.i?86)
+        hash=@tarballHash_i686-linux@
+        path=@tarballPath_i686-linux@
+        system=i686-linux
+        ;;
+    Linux.aarch64)
+        hash=@tarballHash_aarch64-linux@
+        path=@tarballPath_aarch64-linux@
+        system=aarch64-linux
+        ;;
+    Darwin.x86_64)
+        hash=@tarballHash_x86_64-darwin@
+        path=@tarballPath_x86_64-darwin@
+        system=x86_64-darwin
+        ;;
+    Darwin.arm64|Darwin.aarch64)
+        # check for Rosetta 2 support
+        if ! [ -f /Library/Apple/System/Library/LaunchDaemons/com.apple.oahd.plist ]; then
+          oops "Rosetta 2 is not installed on this ARM64 macOS machine. Run softwareupdate --install-rosetta then restart installation"
+        fi
+
+        hash=@binaryTarball_x86_64-darwin@
+        path=@tarballPath_x86_64-darwin@
+        # eventually maybe: aarch64-darwin
+        system=x86_64-darwin
+        ;;
     *) oops "sorry, there is no binary distribution of Nix for your platform";;
 esac
 
-url="https://releases.nixos.org/nix/nix-@nixVersion@/nix-@nixVersion@-$system.tar.xz"
+# Use this command-line option to fetch the tarballs using nar-serve or Cachix
+if "${1:---tarball-url-prefix}"; then
+    if [ -z "${2:-}" ]; then
+        oops "missing argument for --tarball-url-prefix"
+    fi
+    url=${2}/${path}
+    shift 2
+else
+    url=https://releases.nixos.org/nix/nix-@nixVersion@/nix-@nixVersion@-$system.tar.xz
+fi
 
-tarball="$tmpDir/$(basename "$tmpDir/nix-@nixVersion@-$system.tar.xz")"
+tarball=$tmpDir/nix-@nixVersion@-$system.tar.xz
 
 require_util curl "download the binary tarball"
 require_util tar "unpack the binary tarball"
@@ -66,6 +102,7 @@ tar -xJf "$tarball" -C "$unpack" || oops "failed to unpack '$url'"
 script=$(echo "$unpack"/*/install)
 
 [ -e "$script" ] || oops "installation script is missing from the binary tarball!"
+export INVOKED_FROM_INSTALL_IN=1
 "$script" "$@"
 
 } # End of wrapping

scripts/nix-profile-daemon.sh.in

@@ -17,11 +17,21 @@ elif [ -e /etc/pki/tls/certs/ca-bundle.crt ]; then # Fedora, CentOS
     export NIX_SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt
 else
   # Fall back to what is in the nix profiles, favouring whatever is defined last.
-  for i in $NIX_PROFILES; do
-    if [ -e $i/etc/ssl/certs/ca-bundle.crt ]; then
-      export NIX_SSL_CERT_FILE=$i/etc/ssl/certs/ca-bundle.crt
+  check_nix_profiles() {
+    if [ "$ZSH_VERSION" ]; then
+      # Zsh by default doesn't split words in unquoted parameter expansion.
+      # Set local_options for these options to be reverted at the end of the function
+      # and shwordsplit to force splitting words in $NIX_PROFILES below.
+      setopt local_options shwordsplit
     fi
-  done
+    for i in $NIX_PROFILES; do
+      if [ -e $i/etc/ssl/certs/ca-bundle.crt ]; then
+        export NIX_SSL_CERT_FILE=$i/etc/ssl/certs/ca-bundle.crt
+      fi
+    done
+  }
+  check_nix_profiles
+  unset -f check_nix_profiles
 fi
 
 export PATH="$HOME/.nix-profile/bin:@localstatedir@/nix/profiles/default/bin:$PATH"

src/build-remote/build-remote.cc

@@ -17,7 +17,7 @@
 #include "store-api.hh"
 #include "derivations.hh"
 #include "local-store.hh"
-#include "../nix/legacy.hh"
+#include "legacy.hh"
 
 using namespace nix;
 using std::cin;
@@ -71,11 +71,15 @@ static int main_build_remote(int argc, char * * argv)
 
         initPlugins();
 
-        auto store = openStore().cast<LocalStore>();
+        auto store = openStore();
 
         /* It would be more appropriate to use $XDG_RUNTIME_DIR, since
            that gets cleared on reboot, but it wouldn't work on macOS. */
-        currentLoad = store->stateDir + "/current-load";
+        auto currentLoadName = "/current-load";
+        if (auto localStore = store.dynamic_pointer_cast<LocalFSStore>())
+            currentLoad = std::string { localStore->stateDir } + currentLoadName;
+        else
+            currentLoad = settings.nixStateDir + currentLoadName;
 
         std::shared_ptr<Store> sshStore;
         AutoCloseFD bestSlotLock;
@@ -172,13 +176,14 @@ static int main_build_remote(int argc, char * * argv)
                     else
                     {
                         // build the hint template.
-                        string hintstring =  "derivation: %s\nrequired (system, features): (%s, %s)";
-                        hintstring += "\n%s available machines:";
-                        hintstring += "\n(systems, maxjobs, supportedFeatures, mandatoryFeatures)";
+                        string errorText =
+                            "Failed to find a machine for remote build!\n"
+                            "derivation: %s\nrequired (system, features): (%s, %s)";
+                        errorText += "\n%s available machines:";
+                        errorText += "\n(systems, maxjobs, supportedFeatures, mandatoryFeatures)";
 
-                        for (unsigned int i = 0; i < machines.size(); ++i) {
-                          hintstring += "\n(%s, %s, %s, %s)";
-                        }
+                        for (unsigned int i = 0; i < machines.size(); ++i)
+                            errorText += "\n(%s, %s, %s, %s)";
 
                         // add the template values.
                         string drvstr;
@@ -187,25 +192,21 @@ static int main_build_remote(int argc, char * * argv)
                         else
                             drvstr = "<unknown>";
 
-                        auto hint = hintformat(hintstring);
-                        hint
-                          % drvstr
-                          % neededSystem
-                          % concatStringsSep<StringSet>(", ", requiredFeatures)
-                          % machines.size();
+                        auto error = hintformat(errorText);
+                        error
+                            % drvstr
+                            % neededSystem
+                            % concatStringsSep<StringSet>(", ", requiredFeatures)
+                            % machines.size();
 
-                        for (auto & m : machines) {
-                          hint % concatStringsSep<vector<string>>(", ", m.systemTypes)
-                            % m.maxJobs
-                            % concatStringsSep<StringSet>(", ", m.supportedFeatures)
-                            % concatStringsSep<StringSet>(", ", m.mandatoryFeatures);
-                        }
+                        for (auto & m : machines)
+                            error
+                                % concatStringsSep<vector<string>>(", ", m.systemTypes)
+                                % m.maxJobs
+                                % concatStringsSep<StringSet>(", ", m.supportedFeatures)
+                                % concatStringsSep<StringSet>(", ", m.mandatoryFeatures);
 
-                        logErrorInfo(lvlInfo, {
-                              .name = "Remote build",
-                              .description = "Failed to find a machine for remote build!",
-                              .hint = hint
-                        });
+                        printMsg(canBuildLocally ? lvlChatty : lvlWarn, error);
 
                         std::cerr << "# decline\n";
                     }
@@ -230,12 +231,9 @@ static int main_build_remote(int argc, char * * argv)
 
                 } catch (std::exception & e) {
                     auto msg = chomp(drainFD(5, false));
-                    logError({
-                        .name = "Remote build",
-                        .hint = hintfmt("cannot build on '%s': %s%s",
-                            bestMachine->storeUri, e.what(),
-                            (msg.empty() ? "" : ": " + msg))
-                    });
+                    printError("cannot build on '%s': %s%s",
+                        bestMachine->storeUri, e.what(),
+                        msg.empty() ? "" : ": " + msg);
                     bestMachine->enabled = false;
                     continue;
                 }
@@ -288,8 +286,9 @@ connected:
 
         if (!missing.empty()) {
             Activity act(*logger, lvlTalkative, actUnknown, fmt("copying outputs from '%s'", storeUri));
-            for (auto & i : missing)
-                store->locksHeld.insert(store->printStorePath(i)); /* FIXME: ugly */
+            if (auto localStore = store.dynamic_pointer_cast<LocalStore>())
+                for (auto & i : missing)
+                    localStore->locksHeld.insert(store->printStorePath(i)); /* FIXME: ugly */
             copyPaths(ref<Store>(sshStore), store, missing, NoRepair, NoCheckSigs, NoSubstitute);
         }
 

src/libcmd/command.cc

@@ -11,11 +11,20 @@ extern char * * environ __attribute__((weak));
 
 namespace nix {
 
-Commands * RegisterCommand::commands = nullptr;
+RegisterCommand::Commands * RegisterCommand::commands = nullptr;
 
-void NixMultiCommand::printHelp(const string & programName, std::ostream & out)
+nix::Commands RegisterCommand::getCommandsFor(const std::vector<std::string> & prefix)
 {
-    MultiCommand::printHelp(programName, out);
+    nix::Commands res;
+    for (auto & [name, command] : *RegisterCommand::commands)
+        if (name.size() == prefix.size() + 1) {
+            bool equal = true;
+            for (size_t i = 0; i < prefix.size(); ++i)
+                if (name[i] != prefix[i]) equal = false;
+            if (equal)
+                res.insert_or_assign(name[prefix.size()], command);
+        }
+    return res;
 }
 
 nlohmann::json NixMultiCommand::toJSON()
@@ -45,50 +54,67 @@ void StoreCommand::run()
     run(getStore());
 }
 
-StorePathsCommand::StorePathsCommand(bool recursive)
+RealisedPathsCommand::RealisedPathsCommand(bool recursive)
     : recursive(recursive)
 {
     if (recursive)
         addFlag({
             .longName = "no-recursive",
-            .description = "apply operation to specified paths only",
+            .description = "Apply operation to specified paths only.",
+            .category = installablesCategory,
             .handler = {&this->recursive, false},
         });
     else
         addFlag({
             .longName = "recursive",
             .shortName = 'r',
-            .description = "apply operation to closure of the specified paths",
+            .description = "Apply operation to closure of the specified paths.",
+            .category = installablesCategory,
             .handler = {&this->recursive, true},
         });
 
-    mkFlag(0, "all", "apply operation to the entire store", &all);
+    addFlag({
+        .longName = "all",
+        .description = "Apply the operation to every store path.",
+        .category = installablesCategory,
+        .handler = {&all, true},
+    });
 }
 
-void StorePathsCommand::run(ref<Store> store)
+void RealisedPathsCommand::run(ref<Store> store)
 {
-    StorePaths storePaths;
-
+    std::vector<RealisedPath> paths;
     if (all) {
         if (installables.size())
             throw UsageError("'--all' does not expect arguments");
+        // XXX: Only uses opaque paths, ignores all the realisations
         for (auto & p : store->queryAllValidPaths())
-            storePaths.push_back(p);
-    }
-
-    else {
-        for (auto & p : toStorePaths(store, realiseMode, operateOn, installables))
-            storePaths.push_back(p);
-
+            paths.push_back(p);
+    } else {
+        auto pathSet = toRealisedPaths(store, realiseMode, operateOn, installables);
         if (recursive) {
-            StorePathSet closure;
-            store->computeFSClosure(StorePathSet(storePaths.begin(), storePaths.end()), closure, false, false);
-            storePaths.clear();
-            for (auto & p : closure)
-                storePaths.push_back(p);
+            auto roots = std::move(pathSet);
+            pathSet = {};
+            RealisedPath::closure(*store, roots, pathSet);
         }
+        for (auto & path : pathSet)
+            paths.push_back(path);
     }
 
+    run(store, std::move(paths));
+}
+
+StorePathsCommand::StorePathsCommand(bool recursive)
+    : RealisedPathsCommand(recursive)
+{
+}
+
+void StorePathsCommand::run(ref<Store> store, std::vector<RealisedPath> paths)
+{
+    StorePaths storePaths;
+    for (auto & p : paths)
+        storePaths.push_back(p.path());
+
     run(store, std::move(storePaths));
 }
 
@@ -119,7 +145,7 @@ MixProfile::MixProfile()
 {
     addFlag({
         .longName = "profile",
-        .description = "profile to update",
+        .description = "The profile to update.",
         .labels = {"path"},
         .handler = {&profile},
         .completer = completePath
@@ -176,14 +202,14 @@ MixEnvironment::MixEnvironment() : ignoreEnvironment(false)
     addFlag({
         .longName = "ignore-environment",
         .shortName = 'i',
-        .description = "clear the entire environment (except those specified with --keep)",
+        .description = "Clear the entire environment (except those specified with `--keep`).",
         .handler = {&ignoreEnvironment, true},
     });
 
     addFlag({
         .longName = "keep",
         .shortName = 'k',
-        .description = "keep specified environment variable",
+        .description = "Keep the environment variable *name*.",
         .labels = {"name"},
         .handler = {[&](std::string s) { keep.insert(s); }},
     });
@@ -191,7 +217,7 @@ MixEnvironment::MixEnvironment() : ignoreEnvironment(false)
     addFlag({
         .longName = "unset",
         .shortName = 'u',
-        .description = "unset specified environment variable",
+        .description = "Unset the environment variable *name*.",
         .labels = {"name"},
         .handler = {[&](std::string s) { unset.insert(s); }},
     });

src/libcmd/command.hh

@@ -13,6 +13,8 @@ namespace nix {
 
 extern std::string programPath;
 
+extern char * * savedArgv;
+
 class EvalState;
 struct Pos;
 class Store;
@@ -21,10 +23,10 @@ static constexpr Command::Category catSecondary = 100;
 static constexpr Command::Category catUtility = 101;
 static constexpr Command::Category catNixInstallation = 102;
 
+static constexpr auto installablesCategory = "Options that change the interpretation of installables";
+
 struct NixMultiCommand : virtual MultiCommand, virtual Command
 {
-    void printHelp(const string & programName, std::ostream & out) override;
-
     nlohmann::json toJSON() override;
 };
 
@@ -46,6 +48,8 @@ struct EvalCommand : virtual StoreCommand, MixEvalArgs
     ref<EvalState> getEvalState();
 
     std::shared_ptr<EvalState> evalState;
+
+    ~EvalCommand();
 };
 
 struct MixFlakeOptions : virtual Args, EvalCommand
@@ -139,7 +143,7 @@ private:
 };
 
 /* A command that operates on zero or more store paths. */
-struct StorePathsCommand : public InstallablesCommand
+struct RealisedPathsCommand : public InstallablesCommand
 {
 private:
 
@@ -152,17 +156,28 @@ protected:
 
 public:
 
-    StorePathsCommand(bool recursive = false);
+    RealisedPathsCommand(bool recursive = false);
 
     using StoreCommand::run;
 
-    virtual void run(ref<Store> store, std::vector<StorePath> storePaths) = 0;
+    virtual void run(ref<Store> store, std::vector<RealisedPath> paths) = 0;
 
     void run(ref<Store> store) override;
 
     bool useDefaultInstallables() override { return !all; }
 };
 
+struct StorePathsCommand : public RealisedPathsCommand
+{
+    StorePathsCommand(bool recursive = false);
+
+    using RealisedPathsCommand::run;
+
+    virtual void run(ref<Store> store, std::vector<StorePath> storePaths) = 0;
+
+    void run(ref<Store> store, std::vector<RealisedPath> paths) override;
+};
+
 /* A command that operates on exactly one store path. */
 struct StorePathCommand : public InstallablesCommand
 {
@@ -176,20 +191,29 @@ struct StorePathCommand : public InstallablesCommand
 /* A helper class for registering commands globally. */
 struct RegisterCommand
 {
+    typedef std::map<std::vector<std::string>, std::function<ref<Command>()>> Commands;
     static Commands * commands;
 
-    RegisterCommand(const std::string & name,
+    RegisterCommand(std::vector<std::string> && name,
         std::function<ref<Command>()> command)
     {
         if (!commands) commands = new Commands;
         commands->emplace(name, command);
     }
+
+    static nix::Commands getCommandsFor(const std::vector<std::string> & prefix);
 };
 
 template<class T>
 static RegisterCommand registerCommand(const std::string & name)
 {
-    return RegisterCommand(name, [](){ return make_ref<T>(); });
+    return RegisterCommand({name}, [](){ return make_ref<T>(); });
+}
+
+template<class T>
+static RegisterCommand registerCommand2(std::vector<std::string> && name)
+{
+    return RegisterCommand(std::move(name), [](){ return make_ref<T>(); });
 }
 
 Buildables build(ref<Store> store, Realise mode,
@@ -207,6 +231,12 @@ std::set<StorePath> toDerivations(ref<Store> store,
     std::vector<std::shared_ptr<Installable>> installables,
     bool useDeriver = false);
 
+std::set<RealisedPath> toRealisedPaths(
+    ref<Store> store,
+    Realise mode,
+    OperateOn operateOn,
+    std::vector<std::shared_ptr<Installable>> installables);
+
 /* Helper function to generate args that invoke $EDITOR on
    filename:lineno. */
 Strings editorFor(const Pos & pos);
@@ -252,6 +282,8 @@ void completeFlakeRefWithFragment(
     const Strings & defaultFlakeAttrPaths,
     std::string_view prefix);
 
+std::string showVersions(const std::set<std::string> & versions);
+
 void printClosureDiff(
     ref<Store> store,
     const StorePath & beforePath,

src/libcmd/installables.cc

@@ -9,15 +9,41 @@
 #include "store-api.hh"
 #include "shared.hh"
 #include "flake/flake.hh"
-#include "eval-cache.hh"
 #include "url.hh"
 #include "registry.hh"
 
 #include <regex>
 #include <queue>
 
+#include <nlohmann/json.hpp>
+
 namespace nix {
 
+nlohmann::json BuildableOpaque::toJSON(ref<Store> store) const {
+    nlohmann::json res;
+    res["path"] = store->printStorePath(path);
+    return res;
+}
+
+nlohmann::json BuildableFromDrv::toJSON(ref<Store> store) const {
+    nlohmann::json res;
+    res["drvPath"] = store->printStorePath(drvPath);
+    for (const auto& [output, path] : outputs) {
+        res["outputs"][output] = path ? store->printStorePath(*path) : "";
+    }
+    return res;
+}
+
+nlohmann::json buildablesToJSON(const Buildables & buildables, ref<Store> store) {
+    auto res = nlohmann::json::array();
+    for (const Buildable & buildable : buildables) {
+        std::visit([&res, store](const auto & buildable) {
+            res.push_back(buildable.toJSON(store));
+        }, buildable);
+    }
+    return res;
+}
+
 void completeFlakeInputPath(
     ref<EvalState> evalState,
     const FlakeRef & flakeRef,
@@ -31,39 +57,47 @@ void completeFlakeInputPath(
 
 MixFlakeOptions::MixFlakeOptions()
 {
+    auto category = "Common flake-related options";
+
     addFlag({
         .longName = "recreate-lock-file",
-        .description = "recreate lock file from scratch",
+        .description = "Recreate the flake's lock file from scratch.",
+        .category = category,
         .handler = {&lockFlags.recreateLockFile, true}
     });
 
     addFlag({
         .longName = "no-update-lock-file",
-        .description = "do not allow any updates to the lock file",
+        .description = "Do not allow any updates to the flake's lock file.",
+        .category = category,
         .handler = {&lockFlags.updateLockFile, false}
     });
 
     addFlag({
         .longName = "no-write-lock-file",
-        .description = "do not write the newly generated lock file",
+        .description = "Do not write the flake's newly generated lock file.",
+        .category = category,
         .handler = {&lockFlags.writeLockFile, false}
     });
 
     addFlag({
         .longName = "no-registries",
-        .description = "don't use flake registries",
+        .description = "Don't allow lookups in the flake registries.",
+        .category = category,
         .handler = {&lockFlags.useRegistries, false}
     });
 
     addFlag({
         .longName = "commit-lock-file",
-        .description = "commit changes to the lock file",
+        .description = "Commit changes to the flake's lock file.",
+        .category = category,
         .handler = {&lockFlags.commitLockFile, true}
     });
 
     addFlag({
         .longName = "update-input",
-        .description = "update a specific flake input",
+        .description = "Update a specific flake input (ignoring its previous entry in the lock file).",
+        .category = category,
         .labels = {"input-path"},
         .handler = {[&](std::string s) {
             lockFlags.inputUpdates.insert(flake::parseInputPath(s));
@@ -76,7 +110,8 @@ MixFlakeOptions::MixFlakeOptions()
 
     addFlag({
         .longName = "override-input",
-        .description = "override a specific flake input (e.g. `dwarffs/nixpkgs`)",
+        .description = "Override a specific flake input (e.g. `dwarffs/nixpkgs`).",
+        .category = category,
         .labels = {"input-path", "flake-url"},
         .handler = {[&](std::string inputPath, std::string flakeRef) {
             lockFlags.inputOverrides.insert_or_assign(
@@ -87,7 +122,8 @@ MixFlakeOptions::MixFlakeOptions()
 
     addFlag({
         .longName = "inputs-from",
-        .description = "use the inputs of the specified flake as registry entries",
+        .description = "Use the inputs of the specified flake as registry entries.",
+        .category = category,
         .labels = {"flake-url"},
         .handler = {[&](std::string flakeRef) {
             auto evalState = getEvalState();
@@ -116,22 +152,25 @@ SourceExprCommand::SourceExprCommand()
     addFlag({
         .longName = "file",
         .shortName = 'f',
-        .description = "evaluate *file* rather than the default",
+        .description = "Interpret installables as attribute paths relative to the Nix expression stored in *file*.",
+        .category = installablesCategory,
         .labels = {"file"},
         .handler = {&file},
         .completer = completePath
     });
 
     addFlag({
-        .longName ="expr",
-        .description = "evaluate attributes from *expr*",
+        .longName = "expr",
+        .description = "Interpret installables as attribute paths relative to the Nix expression *expr*.",
+        .category = installablesCategory,
         .labels = {"expr"},
         .handler = {&expr}
     });
 
     addFlag({
-        .longName ="derivation",
-        .description = "operate on the store derivation rather than its outputs",
+        .longName = "derivation",
+        .description = "Operate on the store derivation rather than its outputs.",
+        .category = installablesCategory,
         .handler = {&operateOn, OperateOn::Derivation},
     });
 }
@@ -182,10 +221,8 @@ void completeFlakeRefWithFragment(
             // FIXME: do tilde expansion.
             auto flakeRef = parseFlakeRef(flakeRefS, absPath("."));
 
-            auto evalCache = openEvalCache(*evalState,
-                std::make_shared<flake::LockedFlake>(lockFlake(*evalState, flakeRef, lockFlags)));
-
-            auto root = evalCache->getRoot();
+            auto lockedFlake = lockFlake(*evalState, flakeRef, lockFlags);
+            auto rootValue = getFlakeValue(*evalState, lockedFlake);
 
             /* Complete 'fragment' relative to all the
                attrpath prefixes as well as the root of the
@@ -203,12 +240,18 @@ void completeFlakeRefWithFragment(
                     attrPath.pop_back();
                 }
 
-                auto attr = root->findAlongAttrPath(attrPath);
-                if (!attr) continue;
+                auto cachedFieldNames = rootValue->getCache().listChildrenAtPath(evalState->symbols, attrPath);
 
-                for (auto & attr2 : attr->getAttrs()) {
-                    if (hasPrefix(attr2, lastAttr)) {
-                        auto attrPath2 = attr->getAttrPath(attr2);
+                if (!cachedFieldNames) {
+                    auto accessResult = evalState->getOptionalAttrField(*rootValue, attrPath, noPos);
+                    if (accessResult.error) continue;
+                    cachedFieldNames = evalState->listAttrFields(*accessResult.value, *accessResult.pos);
+                }
+
+                for (auto & lastFieldName : *cachedFieldNames) {
+                    if (hasPrefix(lastFieldName, lastAttr)) {
+                        auto attrPath2 = attrPath;
+                        attrPath2.push_back(lastFieldName);
                         /* Strip the attrpath prefix. */
                         attrPath2.erase(attrPath2.begin(), attrPath2.begin() + attrPathPrefix.size());
                         completions->add(flakeRefS + "#" + concatStringsSep(".", attrPath2));
@@ -220,8 +263,8 @@ void completeFlakeRefWithFragment(
                attrpaths. */
             if (fragment.empty()) {
                 for (auto & attrPath : defaultFlakeAttrPaths) {
-                    auto attr = root->findAlongAttrPath(parseAttrPath(*evalState, attrPath));
-                    if (!attr) continue;
+                    auto accessResult = evalState->getOptionalAttrField(*rootValue, {evalState->symbols.create(attrPath)}, noPos);
+                    if (accessResult.error) continue;
                     completions->add(flakeRefS + "#");
                 }
             }
@@ -240,6 +283,12 @@ ref<EvalState> EvalCommand::getEvalState()
     return ref<EvalState>(evalState);
 }
 
+EvalCommand::~EvalCommand()
+{
+    if (evalState)
+        evalState->printStats();
+}
+
 void completeFlakeRef(ref<Store> store, std::string_view prefix)
 {
     if (prefix == "")
@@ -271,24 +320,6 @@ Buildable Installable::toBuildable()
     return std::move(buildables[0]);
 }
 
-std::vector<std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>>
-Installable::getCursors(EvalState & state)
-{
-    auto evalCache =
-        std::make_shared<nix::eval_cache::EvalCache>(std::nullopt, state,
-            [&]() { return toValue(state).first; });
-    return {{evalCache->getRoot(), ""}};
-}
-
-std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>
-Installable::getCursor(EvalState & state)
-{
-    auto cursors = getCursors(state);
-    if (cursors.empty())
-        throw Error("cannot find flake attribute '%s'", what());
-    return cursors[0];
-}
-
 struct InstallableStorePath : Installable
 {
     ref<Store> store;
@@ -359,11 +390,11 @@ struct InstallableAttrPath : InstallableValue
 
     std::string what() override { return attrPath; }
 
-    std::pair<Value *, Pos> toValue(EvalState & state) override
+    std::vector<Installable::ValueInfo> toValues(EvalState & state) override
     {
         auto [vRes, pos] = findAlongAttrPath(state, attrPath, *cmd.getAutoArgs(state), **v);
         state.forceValue(*vRes);
-        return {vRes, pos};
+        return {{vRes, pos, attrPath}};
     }
 
     virtual std::vector<InstallableValue::DerivationInfo> toDerivations() override;
@@ -371,7 +402,7 @@ struct InstallableAttrPath : InstallableValue
 
 std::vector<InstallableValue::DerivationInfo> InstallableAttrPath::toDerivations()
 {
-    auto v = toValue(*state).first;
+    auto v = toValue(*state).value;
 
     Bindings & autoArgs = *cmd.getAutoArgs(*state);
 
@@ -382,7 +413,7 @@ std::vector<InstallableValue::DerivationInfo> InstallableAttrPath::toDerivations
     for (auto & drvInfo : drvInfos) {
         res.push_back({
             state->store->parseStorePath(drvInfo.queryDrvPath()),
-            state->store->parseStorePath(drvInfo.queryOutPath()),
+            state->store->maybeParseStorePath(drvInfo.queryOutPath()),
             drvInfo.queryOutputName()
         });
     }
@@ -405,45 +436,7 @@ std::vector<std::string> InstallableFlake::getActualAttrPaths()
 
 Value * InstallableFlake::getFlakeOutputs(EvalState & state, const flake::LockedFlake & lockedFlake)
 {
-    auto vFlake = state.allocValue();
-
-    callFlake(state, lockedFlake, *vFlake);
-
-    auto aOutputs = vFlake->attrs->get(state.symbols.create("outputs"));
-    assert(aOutputs);
-
-    state.forceValue(*aOutputs->value);
-
-    return aOutputs->value;
-}
-
-ref<eval_cache::EvalCache> openEvalCache(
-    EvalState & state,
-    std::shared_ptr<flake::LockedFlake> lockedFlake)
-{
-    auto fingerprint = lockedFlake->getFingerprint();
-    return make_ref<nix::eval_cache::EvalCache>(
-        evalSettings.useEvalCache && evalSettings.pureEval
-            ? std::optional { std::cref(fingerprint) }
-            : std::nullopt,
-        state,
-        [&state, lockedFlake]()
-        {
-            /* For testing whether the evaluation cache is
-               complete. */
-            if (getEnv("NIX_ALLOW_EVAL").value_or("1") == "0")
-                throw Error("not everything is cached, but evaluation is not allowed");
-
-            auto vFlake = state.allocValue();
-            flake::callFlake(state, *lockedFlake, *vFlake);
-
-            state.forceAttrs(*vFlake);
-
-            auto aOutputs = vFlake->attrs->get(state.symbols.create("outputs"));
-            assert(aOutputs);
-
-            return aOutputs->value;
-        });
+    return getFlakeValue(state, lockedFlake);
 }
 
 static std::string showAttrPaths(const std::vector<std::string> & paths)
@@ -460,29 +453,20 @@ std::tuple<std::string, FlakeRef, InstallableValue::DerivationInfo> InstallableF
 {
     auto lockedFlake = getLockedFlake();
 
-    auto cache = openEvalCache(*state, lockedFlake);
-    auto root = cache->getRoot();
+    auto flakeValue = toValue(*state);
 
-    for (auto & attrPath : getActualAttrPaths()) {
-        auto attr = root->findAlongAttrPath(parseAttrPath(*state, attrPath));
-        if (!attr) continue;
+    auto rawDrvInfo = getDerivation(*state, *flakeValue.value, false);
+    if (!rawDrvInfo)
+        throw Error("flake output attribute '%s' is not a derivation", flakeValue.positionInfo);
 
-        if (!attr->isDerivation())
-            throw Error("flake output attribute '%s' is not a derivation", attrPath);
+    auto drvInfo = InstallableValue::DerivationInfo
+    {
+        state->store->parseStorePath(rawDrvInfo->queryDrvPath()),
+            state->store->maybeParseStorePath(rawDrvInfo->queryOutPath()),
+            rawDrvInfo->queryOutputName()
+    };
 
-        auto drvPath = attr->forceDerivation();
-
-        auto drvInfo = DerivationInfo{
-            std::move(drvPath),
-            state->store->parseStorePath(attr->getAttr(state->sOutPath)->getString()),
-            attr->getAttr(state->sOutputName)->getString()
-        };
-
-        return {attrPath, lockedFlake->flake.lockedRef, std::move(drvInfo)};
-    }
-
-    throw Error("flake '%s' does not provide attribute %s",
-        flakeRef, showAttrPaths(getActualAttrPaths()));
+    return {flakeValue.positionInfo, lockedFlake->flake.lockedRef, std::move(drvInfo)};
 }
 
 std::vector<InstallableValue::DerivationInfo> InstallableFlake::toDerivations()
@@ -492,8 +476,20 @@ std::vector<InstallableValue::DerivationInfo> InstallableFlake::toDerivations()
     return res;
 }
 
-std::pair<Value *, Pos> InstallableFlake::toValue(EvalState & state)
+Installable::ValueInfo InstallableFlake::toValue(EvalState & state)
 {
+    auto values = toValues(state);
+    if (values.empty())
+        throw Error("flake '%s' does not provide attribute %s",
+            flakeRef, showAttrPaths(getActualAttrPaths()));
+    return values[0];
+
+}
+
+std::vector<Installable::ValueInfo>
+InstallableFlake::toValues(EvalState & state)
+{
+    std::vector<Installable::ValueInfo> res;
     auto lockedFlake = getLockedFlake();
 
     auto vOutputs = getFlakeOutputs(state, *lockedFlake);
@@ -504,37 +500,21 @@ std::pair<Value *, Pos> InstallableFlake::toValue(EvalState & state)
         try {
             auto [v, pos] = findAlongAttrPath(state, attrPath, *emptyArgs, *vOutputs);
             state.forceValue(*v);
-            return {v, pos};
+            res.push_back({v, pos, attrPath});
         } catch (AttrPathNotFound & e) {
         }
     }
 
-    throw Error("flake '%s' does not provide attribute %s",
-        flakeRef, showAttrPaths(getActualAttrPaths()));
-}
-
-std::vector<std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>>
-InstallableFlake::getCursors(EvalState & state)
-{
-    auto evalCache = openEvalCache(state,
-        std::make_shared<flake::LockedFlake>(lockFlake(state, flakeRef, lockFlags)));
-
-    auto root = evalCache->getRoot();
-
-    std::vector<std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>> res;
-
-    for (auto & attrPath : getActualAttrPaths()) {
-        auto attr = root->findAlongAttrPath(parseAttrPath(state, attrPath));
-        if (attr) res.push_back({attr, attrPath});
-    }
-
     return res;
 }
 
 std::shared_ptr<flake::LockedFlake> InstallableFlake::getLockedFlake() const
 {
-    if (!_lockedFlake)
+    if (!_lockedFlake) {
         _lockedFlake = std::make_shared<flake::LockedFlake>(lockFlake(*state, flakeRef, lockFlags));
+        _lockedFlake->flake.config.apply();
+        // FIXME: send new config to the daemon.
+    }
     return _lockedFlake;
 }
 
@@ -661,23 +641,42 @@ Buildables build(ref<Store> store, Realise mode,
     return buildables;
 }
 
-StorePathSet toStorePaths(ref<Store> store,
-    Realise mode, OperateOn operateOn,
+std::set<RealisedPath> toRealisedPaths(
+    ref<Store> store,
+    Realise mode,
+    OperateOn operateOn,
     std::vector<std::shared_ptr<Installable>> installables)
 {
-    StorePathSet outPaths;
-
+    std::set<RealisedPath> res;
     if (operateOn == OperateOn::Output) {
         for (auto & b : build(store, mode, installables))
             std::visit(overloaded {
                 [&](BuildableOpaque bo) {
-                    outPaths.insert(bo.path);
+                    res.insert(bo.path);
                 },
                 [&](BuildableFromDrv bfd) {
+                    auto drv = store->readDerivation(bfd.drvPath);
+                    auto outputHashes = staticOutputHashes(*store, drv);
                     for (auto & output : bfd.outputs) {
-                        if (!output.second)
-                            throw Error("Cannot operate on output of unbuilt CA drv");
-                        outPaths.insert(*output.second);
+                        if (settings.isExperimentalFeatureEnabled("ca-derivations")) {
+                            if (!outputHashes.count(output.first))
+                                throw Error(
+                                    "the derivation '%s' doesn't have an output named '%s'",
+                                    store->printStorePath(bfd.drvPath),
+                                    output.first);
+                            auto outputId = DrvOutput{outputHashes.at(output.first), output.first};
+                            auto realisation = store->queryRealisation(outputId);
+                            if (!realisation)
+                                throw Error("cannot operate on an output of unbuilt content-addresed derivation '%s'", outputId.to_string());
+                            res.insert(RealisedPath{*realisation});
+                        }
+                        else {
+                            // If ca-derivations isn't enabled, behave as if
+                            // all the paths are opaque to keep the default
+                            // behavior
+                            assert(output.second);
+                            res.insert(*output.second);
+                        }
                     }
                 },
             }, b);
@@ -688,9 +687,19 @@ StorePathSet toStorePaths(ref<Store> store,
         for (auto & i : installables)
             for (auto & b : i->toBuildables())
                 if (auto bfd = std::get_if<BuildableFromDrv>(&b))
-                    outPaths.insert(bfd->drvPath);
+                    res.insert(bfd->drvPath);
     }
 
+    return res;
+}
+
+StorePathSet toStorePaths(ref<Store> store,
+    Realise mode, OperateOn operateOn,
+    std::vector<std::shared_ptr<Installable>> installables)
+{
+    StorePathSet outPaths;
+    for (auto & path : toRealisedPaths(store, mode, operateOn, installables))
+            outPaths.insert(path.path());
     return outPaths;
 }
 

src/libcmd/installables.hh

@@ -7,6 +7,8 @@
 
 #include <optional>
 
+#include <nlohmann/json_fwd.hpp>
+
 namespace nix {
 
 struct DrvInfo;
@@ -16,11 +18,13 @@ namespace eval_cache { class EvalCache; class AttrCursor; }
 
 struct BuildableOpaque {
     StorePath path;
+    nlohmann::json toJSON(ref<Store> store) const;
 };
 
 struct BuildableFromDrv {
     StorePath drvPath;
     std::map<std::string, std::optional<StorePath>> outputs;
+    nlohmann::json toJSON(ref<Store> store) const;
 };
 
 typedef std::variant<
@@ -29,6 +33,7 @@ typedef std::variant<
 > Buildable;
 
 typedef std::vector<Buildable> Buildables;
+nlohmann::json buildablesToJSON(const Buildables & buildables, ref<Store> store);
 
 struct App
 {
@@ -49,7 +54,23 @@ struct Installable
 
     App toApp(EvalState & state);
 
-    virtual std::pair<Value *, Pos> toValue(EvalState & state)
+    struct ValueInfo {
+        Value * value;
+        Pos pos;
+        string positionInfo;
+    };
+
+    virtual ValueInfo toValue(EvalState & state)
+    {
+        auto values = toValues(state);
+        if (values.empty())
+            throw Error("Installable '%s' does not provide a default value",
+                    what());
+        return values[0];
+    }
+
+    virtual std::vector<ValueInfo>
+    toValues(EvalState & state)
     {
         throw Error("argument '%s' cannot be evaluated", what());
     }
@@ -61,11 +82,6 @@ struct Installable
         return {};
     }
 
-    virtual std::vector<std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>>
-    getCursors(EvalState & state);
-
-    std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>
-    getCursor(EvalState & state);
 
     virtual FlakeRef nixpkgsFlakeRef() const
     {
@@ -115,18 +131,14 @@ struct InstallableFlake : InstallableValue
 
     std::vector<DerivationInfo> toDerivations() override;
 
-    std::pair<Value *, Pos> toValue(EvalState & state) override;
+    ValueInfo toValue(EvalState & state) override;
 
-    std::vector<std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>>
-    getCursors(EvalState & state) override;
+    std::vector<ValueInfo>
+    toValues(EvalState & state) override;
 
     std::shared_ptr<flake::LockedFlake> getLockedFlake() const;
 
     FlakeRef nixpkgsFlakeRef() const override;
 };
 
-ref<eval_cache::EvalCache> openEvalCache(
-    EvalState & state,
-    std::shared_ptr<flake::LockedFlake> lockedFlake);
-
 }

src/libcmd/local.mk

@@ -0,0 +1,15 @@
+libraries += libcmd
+
+libcmd_NAME = libnixcmd
+
+libcmd_DIR := $(d)
+
+libcmd_SOURCES := $(wildcard $(d)/*.cc)
+
+libcmd_CXXFLAGS += -I src/libutil -I src/libstore -I src/libexpr -I src/libmain -I src/libfetchers
+
+libcmd_LDFLAGS = -llowdown
+
+libcmd_LIBS = libstore libutil libexpr libmain libfetchers
+
+$(eval $(call install-file-in, $(d)/nix-cmd.pc, $(prefix)/lib/pkgconfig, 0644))

src/libcmd/markdown.cc

@@ -3,9 +3,7 @@
 #include "finally.hh"
 
 #include <sys/queue.h>
-extern "C" {
 #include <lowdown.h>
-}
 
 namespace nix {
 
@@ -42,7 +40,9 @@ std::string renderMarkdownToTerminal(std::string_view markdown)
         throw Error("cannot allocate Markdown output buffer");
     Finally freeBuffer([&]() { lowdown_buf_free(buf); });
 
-    lowdown_term_rndr(buf, nullptr, renderer, node);
+    int rndr_res = lowdown_term_rndr(buf, nullptr, renderer, node);
+    if (!rndr_res)
+        throw Error("allocation error while rendering Markdown");
 
     return std::string(buf->data, buf->size);
 }

src/libcmd/nix-cmd.pc.in

@@ -0,0 +1,9 @@
+prefix=@prefix@
+libdir=@libdir@
+includedir=@includedir@
+
+Name: Nix
+Description: Nix Package Manager
+Version: @PACKAGE_VERSION@
+Libs: -L${libdir} -lnixcmd
+Cflags: -I${includedir}/nix -std=c++17

src/libexpr/attr-path.cc

@@ -52,9 +52,7 @@ std::pair<Value *, Pos> findAlongAttrPath(EvalState & state, const string & attr
     for (auto & attr : tokens) {
 
         /* Is i an index (integer) or a normal attribute name? */
-        enum { apAttr, apIndex } apType = apAttr;
-        unsigned int attrIndex;
-        if (string2Int(attr, attrIndex)) apType = apIndex;
+        auto attrIndex = string2Int<unsigned int>(attr);
 
         /* Evaluate the expression. */
         Value * vNew = state.allocValue();
@@ -65,9 +63,9 @@ std::pair<Value *, Pos> findAlongAttrPath(EvalState & state, const string & attr
         /* It should evaluate to either a set or an expression,
            according to what is specified in the attrPath. */
 
-        if (apType == apAttr) {
+        if (!attrIndex) {
 
-            if (v->type != tAttrs)
+            if (v->type() != nAttrs)
                 throw TypeError(
                     "the expression selected by the selection path '%1%' should be a set but is %2%",
                     attrPath,
@@ -75,24 +73,25 @@ std::pair<Value *, Pos> findAlongAttrPath(EvalState & state, const string & attr
             if (attr.empty())
                 throw Error("empty attribute name in selection path '%1%'", attrPath);
 
-            Bindings::iterator a = v->attrs->find(state.symbols.create(attr));
-            if (a == v->attrs->end())
+            Value * vRes = state.allocValue();
+            auto getResult = state.getOptionalAttrField(*v, {state.symbols.create(attr)}, pos, *vRes);
+            if (getResult.error)
                 throw AttrPathNotFound("attribute '%1%' in selection path '%2%' not found", attr, attrPath);
-            v = &*a->value;
-            pos = *a->pos;
+            v = vRes;
+            pos = *getResult.pos;
         }
 
-        else if (apType == apIndex) {
+        else {
 
             if (!v->isList())
                 throw TypeError(
                     "the expression selected by the selection path '%1%' should be a list but is %2%",
                     attrPath,
                     showType(*v));
-            if (attrIndex >= v->listSize())
-                throw AttrPathNotFound("list index %1% in selection path '%2%' is out of range", attrIndex, attrPath);
+            if (*attrIndex >= v->listSize())
+                throw AttrPathNotFound("list index %1% in selection path '%2%' is out of range", *attrIndex, attrPath);
 
-            v = v->listElems()[attrIndex];
+            v = v->listElems()[*attrIndex];
             pos = noPos;
         }
 

src/libexpr/attr-set.cc

@@ -24,9 +24,7 @@ void EvalState::mkAttrs(Value & v, size_t capacity)
         v = vEmptySet;
         return;
     }
-    clearValue(v);
-    v.type = tAttrs;
-    v.attrs = allocBindings(capacity);
+    v.mkAttrs(allocBindings(capacity));
     nrAttrsets++;
     nrAttrsInAttrsets += capacity;
 }

src/libexpr/attr-set.hh

@@ -38,6 +38,10 @@ public:
 
 private:
     size_t size_, capacity_;
+
+public:
+
+private:
     Attr attrs[0];
 
     Bindings(size_t capacity) : size_(0), capacity_(capacity) { }
@@ -77,7 +81,7 @@ public:
         auto a = get(name);
         if (!a)
             throw Error({
-                .hint = hintfmt("attribute '%s' missing", name),
+                .msg = hintfmt("attribute '%s' missing", name),
                 .errPos = pos
             });
 

src/libexpr/common-eval-args.cc

@@ -12,16 +12,20 @@ namespace nix {
 
 MixEvalArgs::MixEvalArgs()
 {
+    auto category = "Common evaluation options";
+
     addFlag({
         .longName = "arg",
-        .description = "argument to be passed to Nix functions",
+        .description = "Pass the value *expr* as the argument *name* to Nix functions.",
+        .category = category,
         .labels = {"name", "expr"},
         .handler = {[&](std::string name, std::string expr) { autoArgs[name] = 'E' + expr; }}
     });
 
     addFlag({
         .longName = "argstr",
-        .description = "string-valued argument to be passed to Nix functions",
+        .description = "Pass the string *string* as the argument *name* to Nix functions.",
+        .category = category,
         .labels = {"name", "string"},
         .handler = {[&](std::string name, std::string s) { autoArgs[name] = 'S' + s; }},
     });
@@ -29,14 +33,16 @@ MixEvalArgs::MixEvalArgs()
     addFlag({
         .longName = "include",
         .shortName = 'I',
-        .description = "add a path to the list of locations used to look up `<...>` file names",
+        .description = "Add *path* to the list of locations used to look up `<...>` file names.",
+        .category = category,
         .labels = {"path"},
         .handler = {[&](std::string s) { searchPath.push_back(s); }}
     });
 
     addFlag({
         .longName = "impure",
-        .description = "allow access to mutable paths and repositories",
+        .description = "Allow access to mutable paths and repositories.",
+        .category = category,
         .handler = {[&]() {
             evalSettings.pureEval = false;
         }},
@@ -44,7 +50,8 @@ MixEvalArgs::MixEvalArgs()
 
     addFlag({
         .longName = "override-flake",
-        .description = "override a flake registry value",
+        .description = "Override the flake registries, redirecting *original-ref* to *resolved-ref*.",
+        .category = category,
         .labels = {"original-ref", "resolved-ref"},
         .handler = {[&](std::string _from, std::string _to) {
             auto from = parseFlakeRef(_from, absPath("."));

src/libexpr/context.cc

@@ -0,0 +1,21 @@
+#include "context.hh"
+
+namespace nix {
+
+/* Decode a context string ‘!<name>!<path>’ into a pair <path,
+   name>. */
+std::pair<string, string> decodeContext(std::string_view s)
+{
+    if (s.at(0) == '!') {
+        size_t index = s.find("!", 1);
+        return {std::string(s.substr(index + 1)), std::string(s.substr(1, index - 1))};
+    } else
+        return {s.at(0) == '/' ? std::string(s) : std::string(s.substr(1)), ""};
+}
+
+std::string encodeContext(std::string_view name, std::string_view path)
+{
+    return "!" + std::string(name) + "!" + std::string(path);
+}
+
+}

src/libexpr/context.hh

@@ -0,0 +1,11 @@
+#include "util.hh"
+
+namespace nix {
+
+/* Decode a context string ‘!<name>!<path>’ into a pair <path,
+   name>. */
+std::pair<string, string> decodeContext(std::string_view s);
+
+std::string encodeContext(std::string_view name, std::string_view path);
+
+}

src/libexpr/eval-cache.cc

@@ -1,620 +0,0 @@
-#include "eval-cache.hh"
-#include "sqlite.hh"
-#include "eval.hh"
-#include "eval-inline.hh"
-#include "store-api.hh"
-
-namespace nix::eval_cache {
-
-static const char * schema = R"sql(
-create table if not exists Attributes (
-    parent      integer not null,
-    name        text,
-    type        integer not null,
-    value       text,
-    context     text,
-    primary key (parent, name)
-);
-)sql";
-
-struct AttrDb
-{
-    std::atomic_bool failed{false};
-
-    struct State
-    {
-        SQLite db;
-        SQLiteStmt insertAttribute;
-        SQLiteStmt insertAttributeWithContext;
-        SQLiteStmt queryAttribute;
-        SQLiteStmt queryAttributes;
-        std::unique_ptr<SQLiteTxn> txn;
-    };
-
-    std::unique_ptr<Sync<State>> _state;
-
-    AttrDb(const Hash & fingerprint)
-        : _state(std::make_unique<Sync<State>>())
-    {
-        auto state(_state->lock());
-
-        Path cacheDir = getCacheDir() + "/nix/eval-cache-v2";
-        createDirs(cacheDir);
-
-        Path dbPath = cacheDir + "/" + fingerprint.to_string(Base16, false) + ".sqlite";
-
-        state->db = SQLite(dbPath);
-        state->db.isCache();
-        state->db.exec(schema);
-
-        state->insertAttribute.create(state->db,
-            "insert or replace into Attributes(parent, name, type, value) values (?, ?, ?, ?)");
-
-        state->insertAttributeWithContext.create(state->db,
-            "insert or replace into Attributes(parent, name, type, value, context) values (?, ?, ?, ?, ?)");
-
-        state->queryAttribute.create(state->db,
-            "select rowid, type, value, context from Attributes where parent = ? and name = ?");
-
-        state->queryAttributes.create(state->db,
-            "select name from Attributes where parent = ?");
-
-        state->txn = std::make_unique<SQLiteTxn>(state->db);
-    }
-
-    ~AttrDb()
-    {
-        try {
-            auto state(_state->lock());
-            if (!failed)
-                state->txn->commit();
-            state->txn.reset();
-        } catch (...) {
-            ignoreException();
-        }
-    }
-
-    template<typename F>
-    AttrId doSQLite(F && fun)
-    {
-        if (failed) return 0;
-        try {
-            return fun();
-        } catch (SQLiteError &) {
-            ignoreException();
-            failed = true;
-            return 0;
-        }
-    }
-
-    AttrId setAttrs(
-        AttrKey key,
-        const std::vector<Symbol> & attrs)
-    {
-        return doSQLite([&]()
-        {
-            auto state(_state->lock());
-
-            state->insertAttribute.use()
-                (key.first)
-                (key.second)
-                (AttrType::FullAttrs)
-                (0, false).exec();
-
-            AttrId rowId = state->db.getLastInsertedRowId();
-            assert(rowId);
-
-            for (auto & attr : attrs)
-                state->insertAttribute.use()
-                    (rowId)
-                    (attr)
-                    (AttrType::Placeholder)
-                    (0, false).exec();
-
-            return rowId;
-        });
-    }
-
-    AttrId setString(
-        AttrKey key,
-        std::string_view s,
-        const char * * context = nullptr)
-    {
-        return doSQLite([&]()
-        {
-            auto state(_state->lock());
-
-            if (context) {
-                std::string ctx;
-                for (const char * * p = context; *p; ++p) {
-                    if (p != context) ctx.push_back(' ');
-                    ctx.append(*p);
-                }
-                state->insertAttributeWithContext.use()
-                    (key.first)
-                    (key.second)
-                    (AttrType::String)
-                    (s)
-                    (ctx).exec();
-            } else {
-                state->insertAttribute.use()
-                    (key.first)
-                    (key.second)
-                    (AttrType::String)
-                (s).exec();
-            }
-
-            return state->db.getLastInsertedRowId();
-        });
-    }
-
-    AttrId setBool(
-        AttrKey key,
-        bool b)
-    {
-        return doSQLite([&]()
-        {
-            auto state(_state->lock());
-
-            state->insertAttribute.use()
-                (key.first)
-                (key.second)
-                (AttrType::Bool)
-                (b ? 1 : 0).exec();
-
-            return state->db.getLastInsertedRowId();
-        });
-    }
-
-    AttrId setPlaceholder(AttrKey key)
-    {
-        return doSQLite([&]()
-        {
-            auto state(_state->lock());
-
-            state->insertAttribute.use()
-                (key.first)
-                (key.second)
-                (AttrType::Placeholder)
-                (0, false).exec();
-
-            return state->db.getLastInsertedRowId();
-        });
-    }
-
-    AttrId setMissing(AttrKey key)
-    {
-        return doSQLite([&]()
-        {
-            auto state(_state->lock());
-
-            state->insertAttribute.use()
-                (key.first)
-                (key.second)
-                (AttrType::Missing)
-                (0, false).exec();
-
-            return state->db.getLastInsertedRowId();
-        });
-    }
-
-    AttrId setMisc(AttrKey key)
-    {
-        return doSQLite([&]()
-        {
-            auto state(_state->lock());
-
-            state->insertAttribute.use()
-                (key.first)
-                (key.second)
-                (AttrType::Misc)
-                (0, false).exec();
-
-            return state->db.getLastInsertedRowId();
-        });
-    }
-
-    AttrId setFailed(AttrKey key)
-    {
-        return doSQLite([&]()
-        {
-            auto state(_state->lock());
-
-            state->insertAttribute.use()
-                (key.first)
-                (key.second)
-                (AttrType::Failed)
-                (0, false).exec();
-
-            return state->db.getLastInsertedRowId();
-        });
-    }
-
-    std::optional<std::pair<AttrId, AttrValue>> getAttr(
-        AttrKey key,
-        SymbolTable & symbols)
-    {
-        auto state(_state->lock());
-
-        auto queryAttribute(state->queryAttribute.use()(key.first)(key.second));
-        if (!queryAttribute.next()) return {};
-
-        auto rowId = (AttrType) queryAttribute.getInt(0);
-        auto type = (AttrType) queryAttribute.getInt(1);
-
-        switch (type) {
-            case AttrType::Placeholder:
-                return {{rowId, placeholder_t()}};
-            case AttrType::FullAttrs: {
-                // FIXME: expensive, should separate this out.
-                std::vector<Symbol> attrs;
-                auto queryAttributes(state->queryAttributes.use()(rowId));
-                while (queryAttributes.next())
-                    attrs.push_back(symbols.create(queryAttributes.getStr(0)));
-                return {{rowId, attrs}};
-            }
-            case AttrType::String: {
-                std::vector<std::pair<Path, std::string>> context;
-                if (!queryAttribute.isNull(3))
-                    for (auto & s : tokenizeString<std::vector<std::string>>(queryAttribute.getStr(3), ";"))
-                        context.push_back(decodeContext(s));
-                return {{rowId, string_t{queryAttribute.getStr(2), context}}};
-            }
-            case AttrType::Bool:
-                return {{rowId, queryAttribute.getInt(2) != 0}};
-            case AttrType::Missing:
-                return {{rowId, missing_t()}};
-            case AttrType::Misc:
-                return {{rowId, misc_t()}};
-            case AttrType::Failed:
-                return {{rowId, failed_t()}};
-            default:
-                throw Error("unexpected type in evaluation cache");
-        }
-    }
-};
-
-static std::shared_ptr<AttrDb> makeAttrDb(const Hash & fingerprint)
-{
-    try {
-        return std::make_shared<AttrDb>(fingerprint);
-    } catch (SQLiteError &) {
-        ignoreException();
-        return nullptr;
-    }
-}
-
-EvalCache::EvalCache(
-    std::optional<std::reference_wrapper<const Hash>> useCache,
-    EvalState & state,
-    RootLoader rootLoader)
-    : db(useCache ? makeAttrDb(*useCache) : nullptr)
-    , state(state)
-    , rootLoader(rootLoader)
-{
-}
-
-Value * EvalCache::getRootValue()
-{
-    if (!value) {
-        debug("getting root value");
-        value = allocRootValue(rootLoader());
-    }
-    return *value;
-}
-
-std::shared_ptr<AttrCursor> EvalCache::getRoot()
-{
-    return std::make_shared<AttrCursor>(ref(shared_from_this()), std::nullopt);
-}
-
-AttrCursor::AttrCursor(
-    ref<EvalCache> root,
-    Parent parent,
-    Value * value,
-    std::optional<std::pair<AttrId, AttrValue>> && cachedValue)
-    : root(root), parent(parent), cachedValue(std::move(cachedValue))
-{
-    if (value)
-        _value = allocRootValue(value);
-}
-
-AttrKey AttrCursor::getKey()
-{
-    if (!parent)
-        return {0, root->state.sEpsilon};
-    if (!parent->first->cachedValue) {
-        parent->first->cachedValue = root->db->getAttr(
-            parent->first->getKey(), root->state.symbols);
-        assert(parent->first->cachedValue);
-    }
-    return {parent->first->cachedValue->first, parent->second};
-}
-
-Value & AttrCursor::getValue()
-{
-    if (!_value) {
-        if (parent) {
-            auto & vParent = parent->first->getValue();
-            root->state.forceAttrs(vParent);
-            auto attr = vParent.attrs->get(parent->second);
-            if (!attr)
-                throw Error("attribute '%s' is unexpectedly missing", getAttrPathStr());
-            _value = allocRootValue(attr->value);
-        } else
-            _value = allocRootValue(root->getRootValue());
-    }
-    return **_value;
-}
-
-std::vector<Symbol> AttrCursor::getAttrPath() const
-{
-    if (parent) {
-        auto attrPath = parent->first->getAttrPath();
-        attrPath.push_back(parent->second);
-        return attrPath;
-    } else
-        return {};
-}
-
-std::vector<Symbol> AttrCursor::getAttrPath(Symbol name) const
-{
-    auto attrPath = getAttrPath();
-    attrPath.push_back(name);
-    return attrPath;
-}
-
-std::string AttrCursor::getAttrPathStr() const
-{
-    return concatStringsSep(".", getAttrPath());
-}
-
-std::string AttrCursor::getAttrPathStr(Symbol name) const
-{
-    return concatStringsSep(".", getAttrPath(name));
-}
-
-Value & AttrCursor::forceValue()
-{
-    debug("evaluating uncached attribute %s", getAttrPathStr());
-
-    auto & v = getValue();
-
-    try {
-        root->state.forceValue(v);
-    } catch (EvalError &) {
-        debug("setting '%s' to failed", getAttrPathStr());
-        if (root->db)
-            cachedValue = {root->db->setFailed(getKey()), failed_t()};
-        throw;
-    }
-
-    if (root->db && (!cachedValue || std::get_if<placeholder_t>(&cachedValue->second))) {
-        if (v.type == tString)
-            cachedValue = {root->db->setString(getKey(), v.string.s, v.string.context),
-                           string_t{v.string.s, {}}};
-        else if (v.type == tPath)
-            cachedValue = {root->db->setString(getKey(), v.path), v.path};
-        else if (v.type == tBool)
-            cachedValue = {root->db->setBool(getKey(), v.boolean), v.boolean};
-        else if (v.type == tAttrs)
-            ; // FIXME: do something?
-        else
-            cachedValue = {root->db->setMisc(getKey()), misc_t()};
-    }
-
-    return v;
-}
-
-std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(Symbol name, bool forceErrors)
-{
-    if (root->db) {
-        if (!cachedValue)
-            cachedValue = root->db->getAttr(getKey(), root->state.symbols);
-
-        if (cachedValue) {
-            if (auto attrs = std::get_if<std::vector<Symbol>>(&cachedValue->second)) {
-                for (auto & attr : *attrs)
-                    if (attr == name)
-                        return std::make_shared<AttrCursor>(root, std::make_pair(shared_from_this(), name));
-                return nullptr;
-            } else if (std::get_if<placeholder_t>(&cachedValue->second)) {
-                auto attr = root->db->getAttr({cachedValue->first, name}, root->state.symbols);
-                if (attr) {
-                    if (std::get_if<missing_t>(&attr->second))
-                        return nullptr;
-                    else if (std::get_if<failed_t>(&attr->second)) {
-                        if (forceErrors)
-                            debug("reevaluating failed cached attribute '%s'");
-                        else
-                            throw CachedEvalError("cached failure of attribute '%s'", getAttrPathStr(name));
-                    } else
-                        return std::make_shared<AttrCursor>(root,
-                            std::make_pair(shared_from_this(), name), nullptr, std::move(attr));
-                }
-                // Incomplete attrset, so need to fall thru and
-                // evaluate to see whether 'name' exists
-            } else
-                return nullptr;
-                //throw TypeError("'%s' is not an attribute set", getAttrPathStr());
-        }
-    }
-
-    auto & v = forceValue();
-
-    if (v.type != tAttrs)
-        return nullptr;
-        //throw TypeError("'%s' is not an attribute set", getAttrPathStr());
-
-    auto attr = v.attrs->get(name);
-
-    if (!attr) {
-        if (root->db) {
-            if (!cachedValue)
-                cachedValue = {root->db->setPlaceholder(getKey()), placeholder_t()};
-            root->db->setMissing({cachedValue->first, name});
-        }
-        return nullptr;
-    }
-
-    std::optional<std::pair<AttrId, AttrValue>> cachedValue2;
-    if (root->db) {
-        if (!cachedValue)
-            cachedValue = {root->db->setPlaceholder(getKey()), placeholder_t()};
-        cachedValue2 = {root->db->setPlaceholder({cachedValue->first, name}), placeholder_t()};
-    }
-
-    return std::make_shared<AttrCursor>(
-        root, std::make_pair(shared_from_this(), name), attr->value, std::move(cachedValue2));
-}
-
-std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(std::string_view name)
-{
-    return maybeGetAttr(root->state.symbols.create(name));
-}
-
-std::shared_ptr<AttrCursor> AttrCursor::getAttr(Symbol name, bool forceErrors)
-{
-    auto p = maybeGetAttr(name, forceErrors);
-    if (!p)
-        throw Error("attribute '%s' does not exist", getAttrPathStr(name));
-    return p;
-}
-
-std::shared_ptr<AttrCursor> AttrCursor::getAttr(std::string_view name)
-{
-    return getAttr(root->state.symbols.create(name));
-}
-
-std::shared_ptr<AttrCursor> AttrCursor::findAlongAttrPath(const std::vector<Symbol> & attrPath)
-{
-    auto res = shared_from_this();
-    for (auto & attr : attrPath) {
-        res = res->maybeGetAttr(attr);
-        if (!res) return {};
-    }
-    return res;
-}
-
-std::string AttrCursor::getString()
-{
-    if (root->db) {
-        if (!cachedValue)
-            cachedValue = root->db->getAttr(getKey(), root->state.symbols);
-        if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
-            if (auto s = std::get_if<string_t>(&cachedValue->second)) {
-                debug("using cached string attribute '%s'", getAttrPathStr());
-                return s->first;
-            } else
-                throw TypeError("'%s' is not a string", getAttrPathStr());
-        }
-    }
-
-    auto & v = forceValue();
-
-    if (v.type != tString && v.type != tPath)
-        throw TypeError("'%s' is not a string but %s", getAttrPathStr(), showType(v.type));
-
-    return v.type == tString ? v.string.s : v.path;
-}
-
-string_t AttrCursor::getStringWithContext()
-{
-    if (root->db) {
-        if (!cachedValue)
-            cachedValue = root->db->getAttr(getKey(), root->state.symbols);
-        if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
-            if (auto s = std::get_if<string_t>(&cachedValue->second)) {
-                debug("using cached string attribute '%s'", getAttrPathStr());
-                return *s;
-            } else
-                throw TypeError("'%s' is not a string", getAttrPathStr());
-        }
-    }
-
-    auto & v = forceValue();
-
-    if (v.type == tString)
-        return {v.string.s, v.getContext()};
-    else if (v.type == tPath)
-        return {v.path, {}};
-    else
-        throw TypeError("'%s' is not a string but %s", getAttrPathStr(), showType(v.type));
-}
-
-bool AttrCursor::getBool()
-{
-    if (root->db) {
-        if (!cachedValue)
-            cachedValue = root->db->getAttr(getKey(), root->state.symbols);
-        if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
-            if (auto b = std::get_if<bool>(&cachedValue->second)) {
-                debug("using cached Boolean attribute '%s'", getAttrPathStr());
-                return *b;
-            } else
-                throw TypeError("'%s' is not a Boolean", getAttrPathStr());
-        }
-    }
-
-    auto & v = forceValue();
-
-    if (v.type != tBool)
-        throw TypeError("'%s' is not a Boolean", getAttrPathStr());
-
-    return v.boolean;
-}
-
-std::vector<Symbol> AttrCursor::getAttrs()
-{
-    if (root->db) {
-        if (!cachedValue)
-            cachedValue = root->db->getAttr(getKey(), root->state.symbols);
-        if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
-            if (auto attrs = std::get_if<std::vector<Symbol>>(&cachedValue->second)) {
-                debug("using cached attrset attribute '%s'", getAttrPathStr());
-                return *attrs;
-            } else
-                throw TypeError("'%s' is not an attribute set", getAttrPathStr());
-        }
-    }
-
-    auto & v = forceValue();
-
-    if (v.type != tAttrs)
-        throw TypeError("'%s' is not an attribute set", getAttrPathStr());
-
-    std::vector<Symbol> attrs;
-    for (auto & attr : *getValue().attrs)
-        attrs.push_back(attr.name);
-    std::sort(attrs.begin(), attrs.end(), [](const Symbol & a, const Symbol & b) {
-        return (const string &) a < (const string &) b;
-    });
-
-    if (root->db)
-        cachedValue = {root->db->setAttrs(getKey(), attrs), attrs};
-
-    return attrs;
-}
-
-bool AttrCursor::isDerivation()
-{
-    auto aType = maybeGetAttr("type");
-    return aType && aType->getString() == "derivation";
-}
-
-StorePath AttrCursor::forceDerivation()
-{
-    auto aDrvPath = getAttr(root->state.sDrvPath, true);
-    auto drvPath = root->state.store->parseStorePath(aDrvPath->getString());
-    if (!root->state.store->isValidPath(drvPath) && !settings.readOnlyMode) {
-        /* The eval cache contains 'drvPath', but the actual path has
-           been garbage-collected. So force it to be regenerated. */
-        aDrvPath->forceValue();
-        if (!root->state.store->isValidPath(drvPath))
-            throw Error("don't know how to recreate store derivation '%s'!",
-                root->state.store->printStorePath(drvPath));
-    }
-    return drvPath;
-}
-
-}

src/libexpr/eval-cache.hh

@@ -1,123 +0,0 @@
-#pragma once
-
-#include "sync.hh"
-#include "hash.hh"
-#include "eval.hh"
-
-#include <functional>
-#include <variant>
-
-namespace nix::eval_cache {
-
-MakeError(CachedEvalError, EvalError);
-
-struct AttrDb;
-class AttrCursor;
-
-class EvalCache : public std::enable_shared_from_this<EvalCache>
-{
-    friend class AttrCursor;
-
-    std::shared_ptr<AttrDb> db;
-    EvalState & state;
-    typedef std::function<Value *()> RootLoader;
-    RootLoader rootLoader;
-    RootValue value;
-
-    Value * getRootValue();
-
-public:
-
-    EvalCache(
-        std::optional<std::reference_wrapper<const Hash>> useCache,
-        EvalState & state,
-        RootLoader rootLoader);
-
-    std::shared_ptr<AttrCursor> getRoot();
-};
-
-enum AttrType {
-    Placeholder = 0,
-    FullAttrs = 1,
-    String = 2,
-    Missing = 3,
-    Misc = 4,
-    Failed = 5,
-    Bool = 6,
-};
-
-struct placeholder_t {};
-struct missing_t {};
-struct misc_t {};
-struct failed_t {};
-typedef uint64_t AttrId;
-typedef std::pair<AttrId, Symbol> AttrKey;
-typedef std::pair<std::string, std::vector<std::pair<Path, std::string>>> string_t;
-
-typedef std::variant<
-    std::vector<Symbol>,
-    string_t,
-    placeholder_t,
-    missing_t,
-    misc_t,
-    failed_t,
-    bool
-    > AttrValue;
-
-class AttrCursor : public std::enable_shared_from_this<AttrCursor>
-{
-    friend class EvalCache;
-
-    ref<EvalCache> root;
-    typedef std::optional<std::pair<std::shared_ptr<AttrCursor>, Symbol>> Parent;
-    Parent parent;
-    RootValue _value;
-    std::optional<std::pair<AttrId, AttrValue>> cachedValue;
-
-    AttrKey getKey();
-
-    Value & getValue();
-
-public:
-
-    AttrCursor(
-        ref<EvalCache> root,
-        Parent parent,
-        Value * value = nullptr,
-        std::optional<std::pair<AttrId, AttrValue>> && cachedValue = {});
-
-    std::vector<Symbol> getAttrPath() const;
-
-    std::vector<Symbol> getAttrPath(Symbol name) const;
-
-    std::string getAttrPathStr() const;
-
-    std::string getAttrPathStr(Symbol name) const;
-
-    std::shared_ptr<AttrCursor> maybeGetAttr(Symbol name, bool forceErrors = false);
-
-    std::shared_ptr<AttrCursor> maybeGetAttr(std::string_view name);
-
-    std::shared_ptr<AttrCursor> getAttr(Symbol name, bool forceErrors = false);
-
-    std::shared_ptr<AttrCursor> getAttr(std::string_view name);
-
-    std::shared_ptr<AttrCursor> findAlongAttrPath(const std::vector<Symbol> & attrPath);
-
-    std::string getString();
-
-    string_t getStringWithContext();
-
-    bool getBool();
-
-    std::vector<Symbol> getAttrs();
-
-    bool isDerivation();
-
-    Value & forceValue();
-
-    /* Force creation of the .drv file in the Nix store. */
-    StorePath forceDerivation();
-};
-
-}

src/libexpr/eval-inline.hh

@@ -10,7 +10,7 @@ namespace nix {
 LocalNoInlineNoReturn(void throwEvalError(const Pos & pos, const char * s))
 {
     throw EvalError({
-        .hint = hintfmt(s),
+        .msg = hintfmt(s),
         .errPos = pos
     });
 }
@@ -24,7 +24,7 @@ LocalNoInlineNoReturn(void throwTypeError(const char * s, const Value & v))
 LocalNoInlineNoReturn(void throwTypeError(const Pos & pos, const char * s, const Value & v))
 {
     throw TypeError({
-        .hint = hintfmt(s, showType(v)),
+        .msg = hintfmt(s, showType(v)),
         .errPos = pos
     });
 }
@@ -32,23 +32,21 @@ LocalNoInlineNoReturn(void throwTypeError(const Pos & pos, const char * s, const
 
 void EvalState::forceValue(Value & v, const Pos & pos)
 {
-    if (v.type == tThunk) {
+    if (v.isThunk()) {
         Env * env = v.thunk.env;
         Expr * expr = v.thunk.expr;
         try {
-            v.type = tBlackhole;
+            v.mkBlackhole();
             //checkInterrupt();
             expr->eval(*this, *env, v);
         } catch (...) {
-            v.type = tThunk;
-            v.thunk.env = env;
-            v.thunk.expr = expr;
+            v.mkThunk(env, expr);
             throw;
         }
     }
-    else if (v.type == tApp)
+    else if (v.isApp())
         callFunction(*v.app.left, *v.app.right, v, noPos);
-    else if (v.type == tBlackhole)
+    else if (v.isBlackhole())
         throwEvalError(pos, "infinite recursion encountered");
 }
 
@@ -56,7 +54,7 @@ void EvalState::forceValue(Value & v, const Pos & pos)
 inline void EvalState::forceAttrs(Value & v)
 {
     forceValue(v);
-    if (v.type != tAttrs)
+    if (v.type() != nAttrs)
         throwTypeError("value is %1% while a set was expected", v);
 }
 
@@ -64,7 +62,7 @@ inline void EvalState::forceAttrs(Value & v)
 inline void EvalState::forceAttrs(Value & v, const Pos & pos)
 {
     forceValue(v, pos);
-    if (v.type != tAttrs)
+    if (v.type() != nAttrs)
         throwTypeError(pos, "value is %1% while a set was expected", v);
 }
 

src/libexpr/eval.hh

@@ -1,10 +1,14 @@
 #pragma once
 
 #include "attr-set.hh"
+#include "context.hh"
 #include "value.hh"
 #include "nixexpr.hh"
 #include "symbol-table.hh"
 #include "config.hh"
+#include "hash.hh"
+#include "sqlite.hh"
+#include "tree-cache.hh"
 
 #include <map>
 #include <optional>
@@ -14,6 +18,9 @@
 
 namespace nix {
 
+namespace eval_cache {
+    class EvalCache;
+}
 
 class Store;
 class EvalState;
@@ -116,6 +123,8 @@ private:
 #endif
     FileEvalCache fileEvalCache;
 
+    std::map<Hash, std::shared_ptr<tree_cache::Cache>> evalCache;
+
     SearchPath searchPath;
 
     std::map<std::string, std::pair<bool, std::string>> searchPathResolved;
@@ -131,6 +140,9 @@ public:
     EvalState(const Strings & _searchPath, ref<Store> store);
     ~EvalState();
 
+    std::shared_ptr<eval_cache::EvalCache> openCache(Hash, std::function<Value *()> rootLoader);
+    std::shared_ptr<tree_cache::Cache> openTreeCache(Hash);
+
     void addToSearchPath(const string & s);
 
     SearchPath getSearchPath() { return searchPath; }
@@ -303,6 +315,26 @@ public:
     void mkThunk_(Value & v, Expr * expr);
     void mkPos(Value & v, Pos * pos);
 
+    struct AttrAccessError {
+        const Symbol attrName;
+
+        // To distinguish between a missing field in an attribute set
+        // and an access to something that's not an attribute set
+        std::optional<Value* > illTypedValue;
+    };
+    struct AttrAccesResult {
+        std::optional<AttrAccessError> error;
+        const Pos * pos;
+        Value* value;
+    };
+    AttrAccesResult getOptionalAttrField(Value & attrs, const std::vector<Symbol> & selector, const Pos & pos, Value & dest);
+    AttrAccesResult getOptionalAttrField(Value & attrs, const std::vector<Symbol> & selector, const Pos & pos);
+    AttrAccesResult lazyGetOptionalAttrField(Value & attrs, const std::vector<Symbol> & selector, const Pos & pos);
+    void getAttrField(Value & attrs, const std::vector<Symbol> & selector, const Pos & pos, Value & dest);
+    Value* getAttrField(Value & attrs, const std::vector<Symbol> & selector, const Pos & pos);
+
+    std::vector<Symbol> listAttrFields(Value & attrs, const Pos & pos);
+
     void concatLists(Value & v, size_t nrLists, Value * * lists, const Pos & pos);
 
     /* Print statistics. */
@@ -310,6 +342,8 @@ public:
 
     void realiseContext(const PathSet & context);
 
+    void updateCacheStats(ValueCache::CacheResult);
+
 private:
 
     unsigned long nrEnvs = 0;
@@ -323,6 +357,10 @@ private:
     unsigned long nrListConcats = 0;
     unsigned long nrPrimOpCalls = 0;
     unsigned long nrFunctionCalls = 0;
+    unsigned long nrCacheHits = 0;
+    unsigned long nrCacheMisses = 0;
+    unsigned long nrUncacheable = 0;
+    unsigned long nrUncached = 0;
 
     bool countCalls;
 
@@ -432,4 +470,6 @@ struct EvalSettings : Config
 
 extern EvalSettings evalSettings;
 
+static const std::string corepkgsPrefix{"/__corepkgs__/"};
+
 }

src/libexpr/fetchurl.nix

@@ -1,6 +1,6 @@
 { system ? "" # obsolete
 , url
-, hash ? "" # an SRI ash
+, hash ? "" # an SRI hash
 
 # Legacy hash specification
 , md5 ? "", sha1 ? "", sha256 ? "", sha512 ? ""

src/libexpr/flake/config.cc

@@ -0,0 +1,81 @@
+#include "flake.hh"
+
+#include <nlohmann/json.hpp>
+
+namespace nix::flake {
+
+// setting name -> setting value -> allow or ignore.
+typedef std::map<std::string, std::map<std::string, bool>> TrustedList;
+
+Path trustedListPath()
+{
+    return getDataDir() + "/nix/trusted-settings.json";
+}
+
+static TrustedList readTrustedList()
+{
+    auto path = trustedListPath();
+    if (!pathExists(path)) return {};
+    auto json = nlohmann::json::parse(readFile(path));
+    return json;
+}
+
+static void writeTrustedList(const TrustedList & trustedList)
+{
+    writeFile(trustedListPath(), nlohmann::json(trustedList).dump());
+}
+
+void ConfigFile::apply()
+{
+    std::set<std::string> whitelist{"bash-prompt", "bash-prompt-suffix"};
+
+    for (auto & [name, value] : settings) {
+
+        auto baseName = hasPrefix(name, "extra-") ? std::string(name, 6) : name;
+
+        // FIXME: Move into libutil/config.cc.
+        std::string valueS;
+        if (auto s = std::get_if<std::string>(&value))
+            valueS = *s;
+        else if (auto n = std::get_if<int64_t>(&value))
+            valueS = fmt("%d", n);
+        else if (auto b = std::get_if<Explicit<bool>>(&value))
+            valueS = b->t ? "true" : "false";
+        else if (auto ss = std::get_if<std::vector<std::string>>(&value))
+            valueS = concatStringsSep(" ", *ss); // FIXME: evil
+        else
+            assert(false);
+
+        if (!whitelist.count(baseName)) {
+            auto trustedList = readTrustedList();
+
+            bool trusted = false;
+
+            if (auto saved = get(get(trustedList, name).value_or(std::map<std::string, bool>()), valueS)) {
+                trusted = *saved;
+            } else {
+                // FIXME: filter ANSI escapes, newlines, \r, etc.
+                if (std::tolower(logger->ask(fmt("do you want to allow configuration setting '%s' to be set to '" ANSI_RED "%s" ANSI_NORMAL "' (y/N)?", name, valueS)).value_or('n')) != 'y') {
+                    if (std::tolower(logger->ask("do you want to permanently mark this value as untrusted (y/N)?").value_or('n')) == 'y') {
+                        trustedList[name][valueS] = false;
+                        writeTrustedList(trustedList);
+                    }
+                } else {
+                    if (std::tolower(logger->ask("do you want to permanently mark this value as trusted (y/N)?").value_or('n')) == 'y') {
+                        trustedList[name][valueS] = trusted = true;
+                        writeTrustedList(trustedList);
+                    }
+                }
+            }
+
+            if (!trusted) {
+                warn("ignoring untrusted flake configuration setting '%s'", name);
+                continue;
+            }
+        }
+
+        globalConfig.set(name, valueS);
+    }
+}
+
+}

src/libexpr/flake/flake.cc

@@ -71,14 +71,20 @@ static std::tuple<fetchers::Tree, FlakeRef, FlakeRef> fetchOrSubstituteTree(
     return {std::move(tree), resolvedRef, lockedRef};
 }
 
+static void forceTrivialValue(EvalState & state, Value & value, const Pos & pos)
+{
+    if (value.isThunk() && value.isTrivial())
+        state.forceValue(value, pos);
+}
+
+
 static void expectType(EvalState & state, ValueType type,
     Value & value, const Pos & pos)
 {
-    if (value.type == tThunk && value.isTrivial())
-        state.forceValue(value, pos);
-    if (value.type != type)
+    forceTrivialValue(state, value, pos);
+    if (value.type() != type)
         throw Error("expected %s but got %s at %s",
-            showType(type), showType(value.type), pos);
+            showType(type), showType(value.type()), pos);
 }
 
 static std::map<FlakeId, FlakeInput> parseFlakeInputs(
@@ -87,7 +93,7 @@ static std::map<FlakeId, FlakeInput> parseFlakeInputs(
 static FlakeInput parseFlakeInput(EvalState & state,
     const std::string & inputName, Value * value, const Pos & pos)
 {
-    expectType(state, tAttrs, *value, pos);
+    expectType(state, nAttrs, *value, pos);
 
     FlakeInput input;
 
@@ -102,24 +108,32 @@ static FlakeInput parseFlakeInput(EvalState & state,
     for (nix::Attr attr : *(value->attrs)) {
         try {
             if (attr.name == sUrl) {
-                expectType(state, tString, *attr.value, *attr.pos);
+                expectType(state, nString, *attr.value, *attr.pos);
                 url = attr.value->string.s;
                 attrs.emplace("url", *url);
             } else if (attr.name == sFlake) {
-                expectType(state, tBool, *attr.value, *attr.pos);
+                expectType(state, nBool, *attr.value, *attr.pos);
                 input.isFlake = attr.value->boolean;
             } else if (attr.name == sInputs) {
                 input.overrides = parseFlakeInputs(state, attr.value, *attr.pos);
             } else if (attr.name == sFollows) {
-                expectType(state, tString, *attr.value, *attr.pos);
+                expectType(state, nString, *attr.value, *attr.pos);
                 input.follows = parseInputPath(attr.value->string.s);
             } else {
-                state.forceValue(*attr.value);
-                if (attr.value->type == tString)
-                    attrs.emplace(attr.name, attr.value->string.s);
-                else
-                    throw TypeError("flake input attribute '%s' is %s while a string is expected",
-                        attr.name, showType(*attr.value));
+                switch (attr.value->type()) {
+                    case nString:
+                        attrs.emplace(attr.name, attr.value->string.s);
+                        break;
+                    case nBool:
+                        attrs.emplace(attr.name, Explicit<bool> { attr.value->boolean });
+                        break;
+                    case nInt:
+                        attrs.emplace(attr.name, (long unsigned int)attr.value->integer);
+                        break;
+                    default:
+                        throw TypeError("flake input attribute '%s' is %s while a string, Boolean, or integer is expected",
+                            attr.name, showType(*attr.value));
+                }
             }
         } catch (Error & e) {
             e.addTrace(*attr.pos, hintfmt("in flake attribute '%s'", attr.name));
@@ -153,7 +167,7 @@ static std::map<FlakeId, FlakeInput> parseFlakeInputs(
 {
     std::map<FlakeId, FlakeInput> inputs;
 
-    expectType(state, tAttrs, *value, pos);
+    expectType(state, nAttrs, *value, pos);
 
     for (nix::Attr & inputAttr : *(*value).attrs) {
         inputs.emplace(inputAttr.name,
@@ -194,15 +208,10 @@ static Flake getFlake(
     Value vInfo;
     state.evalFile(flakeFile, vInfo, true); // FIXME: symlink attack
 
-    expectType(state, tAttrs, vInfo, Pos(foFile, state.symbols.create(flakeFile), 0, 0));
-
-    auto sEdition = state.symbols.create("edition"); // FIXME: remove soon
-
-    if (vInfo.attrs->get(sEdition))
-        warn("flake '%s' has deprecated attribute 'edition'", lockedRef);
+    expectType(state, nAttrs, vInfo, Pos(foFile, state.symbols.create(flakeFile), 0, 0));
 
     if (auto description = vInfo.attrs->get(state.sDescription)) {
-        expectType(state, tString, *description->value, *description->pos);
+        expectType(state, nString, *description->value, *description->pos);
         flake.description = description->value->string.s;
     }
 
@@ -214,9 +223,9 @@ static Flake getFlake(
     auto sOutputs = state.symbols.create("outputs");
 
     if (auto outputs = vInfo.attrs->get(sOutputs)) {
-        expectType(state, tLambda, *outputs->value, *outputs->pos);
+        expectType(state, nFunction, *outputs->value, *outputs->pos);
 
-        if (outputs->value->lambda.fun->matchAttrs) {
+        if (outputs->value->isLambda() && outputs->value->lambda.fun->matchAttrs) {
             for (auto & formal : outputs->value->lambda.fun->formals->formals) {
                 if (formal.name != state.sSelf)
                     flake.inputs.emplace(formal.name, FlakeInput {
@@ -228,11 +237,41 @@ static Flake getFlake(
     } else
         throw Error("flake '%s' lacks attribute 'outputs'", lockedRef);
 
+    auto sNixConfig = state.symbols.create("nixConfig");
+
+    if (auto nixConfig = vInfo.attrs->get(sNixConfig)) {
+        expectType(state, nAttrs, *nixConfig->value, *nixConfig->pos);
+
+        for (auto & setting : *nixConfig->value->attrs) {
+            forceTrivialValue(state, *setting.value, *setting.pos);
+            if (setting.value->type() == nString)
+                flake.config.settings.insert({setting.name, state.forceStringNoCtx(*setting.value, *setting.pos)});
+            else if (setting.value->type() == nInt)
+                flake.config.settings.insert({setting.name, state.forceInt(*setting.value, *setting.pos)});
+            else if (setting.value->type() == nBool)
+                flake.config.settings.insert({setting.name, state.forceBool(*setting.value, *setting.pos)});
+            else if (setting.value->type() == nList) {
+                std::vector<std::string> ss;
+                for (unsigned int n = 0; n < setting.value->listSize(); ++n) {
+                    auto elem = setting.value->listElems()[n];
+                    if (elem->type() != nString)
+                        throw TypeError("list element in flake configuration setting '%s' is %s while a string is expected",
+                            setting.name, showType(*setting.value));
+                    ss.push_back(state.forceStringNoCtx(*elem, *setting.pos));
+                }
+                flake.config.settings.insert({setting.name, ss});
+            }
+            else
+                throw TypeError("flake configuration setting '%s' is %s",
+                    setting.name, showType(*setting.value));
+        }
+    }
+
     for (auto & attr : *vInfo.attrs) {
-        if (attr.name != sEdition &&
-            attr.name != state.sDescription &&
+        if (attr.name != state.sDescription &&
             attr.name != sInputs &&
-            attr.name != sOutputs)
+            attr.name != sOutputs &&
+            attr.name != sNixConfig)
             throw Error("flake '%s' has an unsupported attribute '%s', at %s",
                 lockedRef, attr.name, *attr.pos);
     }
@@ -259,284 +298,298 @@ LockedFlake lockFlake(
 
     auto flake = getFlake(state, topRef, lockFlags.useRegistries, flakeCache);
 
-    // FIXME: symlink attack
-    auto oldLockFile = LockFile::read(
-        flake.sourceInfo->actualPath + "/" + flake.lockedRef.subdir + "/flake.lock");
+    try {
 
-    debug("old lock file: %s", oldLockFile);
+        // FIXME: symlink attack
+        auto oldLockFile = LockFile::read(
+            flake.sourceInfo->actualPath + "/" + flake.lockedRef.subdir + "/flake.lock");
 
-    // FIXME: check whether all overrides are used.
-    std::map<InputPath, FlakeInput> overrides;
-    std::set<InputPath> overridesUsed, updatesUsed;
+        debug("old lock file: %s", oldLockFile);
 
-    for (auto & i : lockFlags.inputOverrides)
-        overrides.insert_or_assign(i.first, FlakeInput { .ref = i.second });
+        // FIXME: check whether all overrides are used.
+        std::map<InputPath, FlakeInput> overrides;
+        std::set<InputPath> overridesUsed, updatesUsed;
 
-    LockFile newLockFile;
+        for (auto & i : lockFlags.inputOverrides)
+            overrides.insert_or_assign(i.first, FlakeInput { .ref = i.second });
 
-    std::vector<FlakeRef> parents;
+        LockFile newLockFile;
 
-    std::function<void(
-        const FlakeInputs & flakeInputs,
-        std::shared_ptr<Node> node,
-        const InputPath & inputPathPrefix,
-        std::shared_ptr<const Node> oldNode)>
-        computeLocks;
+        std::vector<FlakeRef> parents;
 
-    computeLocks = [&](
-        const FlakeInputs & flakeInputs,
-        std::shared_ptr<Node> node,
-        const InputPath & inputPathPrefix,
-        std::shared_ptr<const Node> oldNode)
-    {
-        debug("computing lock file node '%s'", printInputPath(inputPathPrefix));
+        std::function<void(
+            const FlakeInputs & flakeInputs,
+            std::shared_ptr<Node> node,
+            const InputPath & inputPathPrefix,
+            std::shared_ptr<const Node> oldNode)>
+            computeLocks;
 
-        /* Get the overrides (i.e. attributes of the form
-           'inputs.nixops.inputs.nixpkgs.url = ...'). */
-        // FIXME: check this
-        for (auto & [id, input] : flake.inputs) {
-            for (auto & [idOverride, inputOverride] : input.overrides) {
+        computeLocks = [&](
+            const FlakeInputs & flakeInputs,
+            std::shared_ptr<Node> node,
+            const InputPath & inputPathPrefix,
+            std::shared_ptr<const Node> oldNode)
+        {
+            debug("computing lock file node '%s'", printInputPath(inputPathPrefix));
+
+            /* Get the overrides (i.e. attributes of the form
+               'inputs.nixops.inputs.nixpkgs.url = ...'). */
+            // FIXME: check this
+            for (auto & [id, input] : flake.inputs) {
+                for (auto & [idOverride, inputOverride] : input.overrides) {
+                    auto inputPath(inputPathPrefix);
+                    inputPath.push_back(id);
+                    inputPath.push_back(idOverride);
+                    overrides.insert_or_assign(inputPath, inputOverride);
+                }
+            }
+
+            /* Go over the flake inputs, resolve/fetch them if
+               necessary (i.e. if they're new or the flakeref changed
+               from what's in the lock file). */
+            for (auto & [id, input2] : flakeInputs) {
                 auto inputPath(inputPathPrefix);
                 inputPath.push_back(id);
-                inputPath.push_back(idOverride);
-                overrides.insert_or_assign(inputPath, inputOverride);
-            }
-        }
+                auto inputPathS = printInputPath(inputPath);
+                debug("computing input '%s'", inputPathS);
 
-        /* Go over the flake inputs, resolve/fetch them if
-           necessary (i.e. if they're new or the flakeref changed
-           from what's in the lock file). */
-        for (auto & [id, input2] : flakeInputs) {
-            auto inputPath(inputPathPrefix);
-            inputPath.push_back(id);
-            auto inputPathS = printInputPath(inputPath);
-            debug("computing input '%s'", inputPathS);
+                try {
 
-            /* Do we have an override for this input from one of the
-               ancestors? */
-            auto i = overrides.find(inputPath);
-            bool hasOverride = i != overrides.end();
-            if (hasOverride) overridesUsed.insert(inputPath);
-            auto & input = hasOverride ? i->second : input2;
+                    /* Do we have an override for this input from one of the
+                       ancestors? */
+                    auto i = overrides.find(inputPath);
+                    bool hasOverride = i != overrides.end();
+                    if (hasOverride) overridesUsed.insert(inputPath);
+                    auto & input = hasOverride ? i->second : input2;
 
-            /* Resolve 'follows' later (since it may refer to an input
-               path we haven't processed yet. */
-            if (input.follows) {
-                InputPath target;
-                if (hasOverride || input.absolute)
-                    /* 'follows' from an override is relative to the
-                       root of the graph. */
-                    target = *input.follows;
-                else {
-                    /* Otherwise, it's relative to the current flake. */
-                    target = inputPathPrefix;
-                    for (auto & i : *input.follows) target.push_back(i);
-                }
-                debug("input '%s' follows '%s'", inputPathS, printInputPath(target));
-                node->inputs.insert_or_assign(id, target);
-                continue;
-            }
+                    /* Resolve 'follows' later (since it may refer to an input
+                       path we haven't processed yet. */
+                    if (input.follows) {
+                        InputPath target;
+                        if (hasOverride || input.absolute)
+                            /* 'follows' from an override is relative to the
+                               root of the graph. */
+                            target = *input.follows;
+                        else {
+                            /* Otherwise, it's relative to the current flake. */
+                            target = inputPathPrefix;
+                            for (auto & i : *input.follows) target.push_back(i);
+                        }
+                        debug("input '%s' follows '%s'", inputPathS, printInputPath(target));
+                        node->inputs.insert_or_assign(id, target);
+                        continue;
+                    }
 
-            assert(input.ref);
+                    assert(input.ref);
 
-            /* Do we have an entry in the existing lock file? And we
-               don't have a --update-input flag for this input? */
-            std::shared_ptr<LockedNode> oldLock;
+                    /* Do we have an entry in the existing lock file? And we
+                       don't have a --update-input flag for this input? */
+                    std::shared_ptr<LockedNode> oldLock;
 
-            updatesUsed.insert(inputPath);
+                    updatesUsed.insert(inputPath);
 
-            if (oldNode && !lockFlags.inputUpdates.count(inputPath))
-                if (auto oldLock2 = get(oldNode->inputs, id))
-                    if (auto oldLock3 = std::get_if<0>(&*oldLock2))
-                        oldLock = *oldLock3;
+                    if (oldNode && !lockFlags.inputUpdates.count(inputPath))
+                        if (auto oldLock2 = get(oldNode->inputs, id))
+                            if (auto oldLock3 = std::get_if<0>(&*oldLock2))
+                                oldLock = *oldLock3;
 
-            if (oldLock
-                && oldLock->originalRef == *input.ref
-                && !hasOverride)
-            {
-                debug("keeping existing input '%s'", inputPathS);
+                    if (oldLock
+                        && oldLock->originalRef == *input.ref
+                        && !hasOverride)
+                    {
+                        debug("keeping existing input '%s'", inputPathS);
 
-                /* Copy the input from the old lock since its flakeref
-                   didn't change and there is no override from a
-                   higher level flake. */
-                auto childNode = std::make_shared<LockedNode>(
-                    oldLock->lockedRef, oldLock->originalRef, oldLock->isFlake);
+                        /* Copy the input from the old lock since its flakeref
+                           didn't change and there is no override from a
+                           higher level flake. */
+                        auto childNode = std::make_shared<LockedNode>(
+                            oldLock->lockedRef, oldLock->originalRef, oldLock->isFlake);
 
-                node->inputs.insert_or_assign(id, childNode);
+                        node->inputs.insert_or_assign(id, childNode);
 
-                /* If we have an --update-input flag for an input
-                   of this input, then we must fetch the flake to
-                   update it. */
-                auto lb = lockFlags.inputUpdates.lower_bound(inputPath);
+                        /* If we have an --update-input flag for an input
+                           of this input, then we must fetch the flake to
+                           update it. */
+                        auto lb = lockFlags.inputUpdates.lower_bound(inputPath);
 
-                auto hasChildUpdate =
-                    lb != lockFlags.inputUpdates.end()
-                    && lb->size() > inputPath.size()
-                    && std::equal(inputPath.begin(), inputPath.end(), lb->begin());
+                        auto hasChildUpdate =
+                            lb != lockFlags.inputUpdates.end()
+                            && lb->size() > inputPath.size()
+                            && std::equal(inputPath.begin(), inputPath.end(), lb->begin());
 
-                if (hasChildUpdate) {
-                    auto inputFlake = getFlake(
-                        state, oldLock->lockedRef, false, flakeCache);
-                    computeLocks(inputFlake.inputs, childNode, inputPath, oldLock);
-                } else {
-                    /* No need to fetch this flake, we can be
-                       lazy. However there may be new overrides on the
-                       inputs of this flake, so we need to check
-                       those. */
-                    FlakeInputs fakeInputs;
+                        if (hasChildUpdate) {
+                            auto inputFlake = getFlake(
+                                state, oldLock->lockedRef, false, flakeCache);
+                            computeLocks(inputFlake.inputs, childNode, inputPath, oldLock);
+                        } else {
+                            /* No need to fetch this flake, we can be
+                               lazy. However there may be new overrides on the
+                               inputs of this flake, so we need to check
+                               those. */
+                            FlakeInputs fakeInputs;
 
-                    for (auto & i : oldLock->inputs) {
-                        if (auto lockedNode = std::get_if<0>(&i.second)) {
-                            fakeInputs.emplace(i.first, FlakeInput {
-                                .ref = (*lockedNode)->originalRef,
-                                .isFlake = (*lockedNode)->isFlake,
-                            });
-                        } else if (auto follows = std::get_if<1>(&i.second)) {
-                            fakeInputs.emplace(i.first, FlakeInput {
-                                .follows = *follows,
-                                .absolute = true
-                            });
+                            for (auto & i : oldLock->inputs) {
+                                if (auto lockedNode = std::get_if<0>(&i.second)) {
+                                    fakeInputs.emplace(i.first, FlakeInput {
+                                        .ref = (*lockedNode)->originalRef,
+                                        .isFlake = (*lockedNode)->isFlake,
+                                    });
+                                } else if (auto follows = std::get_if<1>(&i.second)) {
+                                    fakeInputs.emplace(i.first, FlakeInput {
+                                        .follows = *follows,
+                                        .absolute = true
+                                    });
+                                }
+                            }
+
+                            computeLocks(fakeInputs, childNode, inputPath, oldLock);
+                        }
+
+                    } else {
+                        /* We need to create a new lock file entry. So fetch
+                           this input. */
+                        debug("creating new input '%s'", inputPathS);
+
+                        if (!lockFlags.allowMutable && !input.ref->input.isImmutable())
+                            throw Error("cannot update flake input '%s' in pure mode", inputPathS);
+
+                        if (input.isFlake) {
+                            auto inputFlake = getFlake(state, *input.ref, lockFlags.useRegistries, flakeCache);
+
+                            /* Note: in case of an --override-input, we use
+                               the *original* ref (input2.ref) for the
+                               "original" field, rather than the
+                               override. This ensures that the override isn't
+                               nuked the next time we update the lock
+                               file. That is, overrides are sticky unless you
+                               use --no-write-lock-file. */
+                            auto childNode = std::make_shared<LockedNode>(
+                                inputFlake.lockedRef, input2.ref ? *input2.ref : *input.ref);
+
+                            node->inputs.insert_or_assign(id, childNode);
+
+                            /* Guard against circular flake imports. */
+                            for (auto & parent : parents)
+                                if (parent == *input.ref)
+                                    throw Error("found circular import of flake '%s'", parent);
+                            parents.push_back(*input.ref);
+                            Finally cleanup([&]() { parents.pop_back(); });
+
+                            /* Recursively process the inputs of this
+                               flake. Also, unless we already have this flake
+                               in the top-level lock file, use this flake's
+                               own lock file. */
+                            computeLocks(
+                                inputFlake.inputs, childNode, inputPath,
+                                oldLock
+                                ? std::dynamic_pointer_cast<const Node>(oldLock)
+                                : LockFile::read(
+                                    inputFlake.sourceInfo->actualPath + "/" + inputFlake.lockedRef.subdir + "/flake.lock").root);
+                        }
+
+                        else {
+                            auto [sourceInfo, resolvedRef, lockedRef] = fetchOrSubstituteTree(
+                                state, *input.ref, lockFlags.useRegistries, flakeCache);
+                            node->inputs.insert_or_assign(id,
+                                std::make_shared<LockedNode>(lockedRef, *input.ref, false));
                         }
                     }
 
-                    computeLocks(fakeInputs, childNode, inputPath, oldLock);
-                }
-
-            } else {
-                /* We need to create a new lock file entry. So fetch
-                   this input. */
-                debug("creating new input '%s'", inputPathS);
-
-                if (!lockFlags.allowMutable && !input.ref->input.isImmutable())
-                    throw Error("cannot update flake input '%s' in pure mode", inputPathS);
-
-                if (input.isFlake) {
-                    auto inputFlake = getFlake(state, *input.ref, lockFlags.useRegistries, flakeCache);
-
-                    /* Note: in case of an --override-input, we use
-                       the *original* ref (input2.ref) for the
-                       "original" field, rather than the
-                       override. This ensures that the override isn't
-                       nuked the next time we update the lock
-                       file. That is, overrides are sticky unless you
-                       use --no-write-lock-file. */
-                    auto childNode = std::make_shared<LockedNode>(
-                        inputFlake.lockedRef, input2.ref ? *input2.ref : *input.ref);
-
-                    node->inputs.insert_or_assign(id, childNode);
-
-                    /* Guard against circular flake imports. */
-                    for (auto & parent : parents)
-                        if (parent == *input.ref)
-                            throw Error("found circular import of flake '%s'", parent);
-                    parents.push_back(*input.ref);
-                    Finally cleanup([&]() { parents.pop_back(); });
-
-                    /* Recursively process the inputs of this
-                       flake. Also, unless we already have this flake
-                       in the top-level lock file, use this flake's
-                       own lock file. */
-                    computeLocks(
-                        inputFlake.inputs, childNode, inputPath,
-                        oldLock
-                        ? std::dynamic_pointer_cast<const Node>(oldLock)
-                        : LockFile::read(
-                            inputFlake.sourceInfo->actualPath + "/" + inputFlake.lockedRef.subdir + "/flake.lock").root);
-                }
-
-                else {
-                    auto [sourceInfo, resolvedRef, lockedRef] = fetchOrSubstituteTree(
-                        state, *input.ref, lockFlags.useRegistries, flakeCache);
-                    node->inputs.insert_or_assign(id,
-                        std::make_shared<LockedNode>(lockedRef, *input.ref, false));
+                } catch (Error & e) {
+                    e.addTrace({}, "while updating the flake input '%s'", inputPathS);
+                    throw;
                 }
             }
-        }
-    };
+        };
 
-    computeLocks(
-        flake.inputs, newLockFile.root, {},
-        lockFlags.recreateLockFile ? nullptr : oldLockFile.root);
+        computeLocks(
+            flake.inputs, newLockFile.root, {},
+            lockFlags.recreateLockFile ? nullptr : oldLockFile.root);
 
-    for (auto & i : lockFlags.inputOverrides)
-        if (!overridesUsed.count(i.first))
-            warn("the flag '--override-input %s %s' does not match any input",
-                printInputPath(i.first), i.second);
+        for (auto & i : lockFlags.inputOverrides)
+            if (!overridesUsed.count(i.first))
+                warn("the flag '--override-input %s %s' does not match any input",
+                    printInputPath(i.first), i.second);
 
-    for (auto & i : lockFlags.inputUpdates)
-        if (!updatesUsed.count(i))
-            warn("the flag '--update-input %s' does not match any input", printInputPath(i));
+        for (auto & i : lockFlags.inputUpdates)
+            if (!updatesUsed.count(i))
+                warn("the flag '--update-input %s' does not match any input", printInputPath(i));
 
-    /* Check 'follows' inputs. */
-    newLockFile.check();
+        /* Check 'follows' inputs. */
+        newLockFile.check();
 
-    debug("new lock file: %s", newLockFile);
+        debug("new lock file: %s", newLockFile);
 
-    /* Check whether we need to / can write the new lock file. */
-    if (!(newLockFile == oldLockFile)) {
+        /* Check whether we need to / can write the new lock file. */
+        if (!(newLockFile == oldLockFile)) {
 
-        auto diff = LockFile::diff(oldLockFile, newLockFile);
+            auto diff = LockFile::diff(oldLockFile, newLockFile);
 
-        if (lockFlags.writeLockFile) {
-            if (auto sourcePath = topRef.input.getSourcePath()) {
-                if (!newLockFile.isImmutable()) {
-                    if (settings.warnDirty)
-                        warn("will not write lock file of flake '%s' because it has a mutable input", topRef);
-                } else {
-                    if (!lockFlags.updateLockFile)
-                        throw Error("flake '%s' requires lock file changes but they're not allowed due to '--no-update-lock-file'", topRef);
+            if (lockFlags.writeLockFile) {
+                if (auto sourcePath = topRef.input.getSourcePath()) {
+                    if (!newLockFile.isImmutable()) {
+                        if (settings.warnDirty)
+                            warn("will not write lock file of flake '%s' because it has a mutable input", topRef);
+                    } else {
+                        if (!lockFlags.updateLockFile)
+                            throw Error("flake '%s' requires lock file changes but they're not allowed due to '--no-update-lock-file'", topRef);
 
-                    auto relPath = (topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock";
+                        auto relPath = (topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock";
 
-                    auto path = *sourcePath + "/" + relPath;
+                        auto path = *sourcePath + "/" + relPath;
 
-                    bool lockFileExists = pathExists(path);
+                        bool lockFileExists = pathExists(path);
 
-                    if (lockFileExists) {
-                        auto s = chomp(diff);
-                        if (s.empty())
-                            warn("updating lock file '%s'", path);
-                        else
-                            warn("updating lock file '%s':\n%s", path, s);
-                    } else
-                        warn("creating lock file '%s'", path);
+                        if (lockFileExists) {
+                            auto s = chomp(diff);
+                            if (s.empty())
+                                warn("updating lock file '%s'", path);
+                            else
+                                warn("updating lock file '%s':\n%s", path, s);
+                        } else
+                            warn("creating lock file '%s'", path);
 
-                    newLockFile.write(path);
+                        newLockFile.write(path);
 
-                    topRef.input.markChangedFile(
-                        (topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock",
-                        lockFlags.commitLockFile
-                        ? std::optional<std::string>(fmt("%s: %s\n\nFlake input changes:\n\n%s",
-                                relPath, lockFileExists ? "Update" : "Add", diff))
-                        : std::nullopt);
+                        topRef.input.markChangedFile(
+                            (topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock",
+                            lockFlags.commitLockFile
+                            ? std::optional<std::string>(fmt("%s: %s\n\nFlake input changes:\n\n%s",
+                                    relPath, lockFileExists ? "Update" : "Add", diff))
+                            : std::nullopt);
 
-                    /* Rewriting the lockfile changed the top-level
-                       repo, so we should re-read it. FIXME: we could
-                       also just clear the 'rev' field... */
-                    auto prevLockedRef = flake.lockedRef;
-                    FlakeCache dummyCache;
-                    flake = getFlake(state, topRef, lockFlags.useRegistries, dummyCache);
+                        /* Rewriting the lockfile changed the top-level
+                           repo, so we should re-read it. FIXME: we could
+                           also just clear the 'rev' field... */
+                        auto prevLockedRef = flake.lockedRef;
+                        FlakeCache dummyCache;
+                        flake = getFlake(state, topRef, lockFlags.useRegistries, dummyCache);
 
-                    if (lockFlags.commitLockFile &&
-                        flake.lockedRef.input.getRev() &&
-                        prevLockedRef.input.getRev() != flake.lockedRef.input.getRev())
-                        warn("committed new revision '%s'", flake.lockedRef.input.getRev()->gitRev());
+                        if (lockFlags.commitLockFile &&
+                            flake.lockedRef.input.getRev() &&
+                            prevLockedRef.input.getRev() != flake.lockedRef.input.getRev())
+                            warn("committed new revision '%s'", flake.lockedRef.input.getRev()->gitRev());
 
-                    /* Make sure that we picked up the change,
-                       i.e. the tree should usually be dirty
-                       now. Corner case: we could have reverted from a
-                       dirty to a clean tree! */
-                    if (flake.lockedRef.input == prevLockedRef.input
-                        && !flake.lockedRef.input.isImmutable())
-                        throw Error("'%s' did not change after I updated its 'flake.lock' file; is 'flake.lock' under version control?", flake.originalRef);
-                }
+                        /* Make sure that we picked up the change,
+                           i.e. the tree should usually be dirty
+                           now. Corner case: we could have reverted from a
+                           dirty to a clean tree! */
+                        if (flake.lockedRef.input == prevLockedRef.input
+                            && !flake.lockedRef.input.isImmutable())
+                            throw Error("'%s' did not change after I updated its 'flake.lock' file; is 'flake.lock' under version control?", flake.originalRef);
+                    }
+                } else
+                    throw Error("cannot write modified lock file of flake '%s' (use '--no-write-lock-file' to ignore)", topRef);
             } else
-                throw Error("cannot write modified lock file of flake '%s' (use '--no-write-lock-file' to ignore)", topRef);
-        } else
-            warn("not writing modified lock file of flake '%s':\n%s", topRef, chomp(diff));
-    }
+                warn("not writing modified lock file of flake '%s':\n%s", topRef, chomp(diff));
+        }
 
-    return LockedFlake { .flake = std::move(flake), .lockFile = std::move(newLockFile) };
+        return LockedFlake { .flake = std::move(flake), .lockFile = std::move(newLockFile) };
+
+    } catch (Error & e) {
+        e.addTrace({}, "while updating the lock file of flake '%s'", flake.lockedRef.to_string());
+        throw;
+    }
 }
 
 void callFlake(EvalState & state,
@@ -564,9 +617,13 @@ void callFlake(EvalState & state,
             , "/"), **vCallFlake);
     }
 
-    state.callFunction(**vCallFlake, *vLocks, *vTmp1, noPos);
-    state.callFunction(*vTmp1, *vRootSrc, *vTmp2, noPos);
-    state.callFunction(*vTmp2, *vRootSubdir, vRes, noPos);
+    vTmp1->mkApp(*vCallFlake, vLocks);
+    vTmp2->mkApp(vTmp1, vRootSrc);
+    vRes.mkApp(vTmp2, vRootSubdir);
+    auto fingerprint = lockedFlake.getFingerprint();
+    auto evalCache = state.openTreeCache(fingerprint);
+    auto cacheRoot = evalCache ? evalCache->getRoot() : nullptr;
+    vRes.setCache(ValueCache(cacheRoot));
 }
 
 static void prim_getFlake(EvalState & state, const Pos & pos, Value * * args, Value & v)
@@ -605,4 +662,19 @@ Fingerprint LockedFlake::getFingerprint() const
 
 Flake::~Flake() { }
 
+Value* getFlakeValue(EvalState & state, const flake::LockedFlake lockedFlake)
+{
+    /* For testing whether the evaluation cache is
+        complete. */
+    if (getEnv("NIX_ALLOW_EVAL").value_or("1") == "0")
+        throw Error("not everything is cached, but evaluation is not allowed");
+
+    auto vFlake = state.allocValue();
+    flake::callFlake(state, lockedFlake, *vFlake);
+
+    auto aOutputs = state.getAttrField(*vFlake, {state.symbols.create("outputs")}, noPos);
+
+    return aOutputs;
+}
+
 }

src/libexpr/flake/flake.hh

@@ -6,7 +6,6 @@
 #include "value.hh"
 
 namespace nix {
-
 class EvalState;
 
 namespace fetchers { struct Tree; }
@@ -17,23 +16,55 @@ struct FlakeInput;
 
 typedef std::map<FlakeId, FlakeInput> FlakeInputs;
 
+/* FlakeInput is the 'Flake'-level parsed form of the "input" entries
+ * in the flake file.
+ *
+ * A FlakeInput is normally constructed by the 'parseFlakeInput'
+ * function which parses the input specification in the '.flake' file
+ * to create a 'FlakeRef' (a fetcher, the fetcher-specific
+ * representation of the input specification, and possibly the fetched
+ * local store path result) and then creating this FlakeInput to hold
+ * that FlakeRef, along with anything that might override that
+ * FlakeRef (like command-line overrides or "follows" specifications).
+ *
+ * A FlakeInput is also sometimes constructed directly from a FlakeRef
+ * instead of starting at the flake-file input specification
+ * (e.g. overrides, follows, and implicit inputs).
+ *
+ * A FlakeInput will usually have one of either "ref" or "follows"
+ * set.  If not otherwise specified, a "ref" will be generated to a
+ * 'type="indirect"' flake, which is treated as simply the name of a
+ * flake to be resolved in the registry.
+ */
+
 struct FlakeInput
 {
     std::optional<FlakeRef> ref;
-    bool isFlake = true;
+    bool isFlake = true;  // true = process flake to get outputs, false = (fetched) static source path
     std::optional<InputPath> follows;
     bool absolute = false; // whether 'follows' is relative to the flake root
     FlakeInputs overrides;
 };
 
+struct ConfigFile
+{
+    using ConfigValue = std::variant<std::string, int64_t, Explicit<bool>, std::vector<std::string>>;
+
+    std::map<std::string, ConfigValue> settings;
+
+    void apply();
+};
+
+/* The contents of a flake.nix file. */
 struct Flake
 {
-    FlakeRef originalRef;
-    FlakeRef resolvedRef;
-    FlakeRef lockedRef;
+    FlakeRef originalRef;   // the original flake specification (by the user)
+    FlakeRef resolvedRef;   // registry references and caching resolved to the specific underlying flake
+    FlakeRef lockedRef;     // the specific local store result of invoking the fetcher
     std::optional<std::string> description;
     std::shared_ptr<const fetchers::Tree> sourceInfo;
     FlakeInputs inputs;
+    ConfigFile config; // 'nixConfig' attribute
     ~Flake();
 };
 
@@ -107,4 +138,5 @@ void emitTreeAttrs(
     const fetchers::Input & input,
     Value & v, bool emptyRevFallback = false);
 
+Value* getFlakeValue(EvalState & state, const flake::LockedFlake lockedFlake);
 }

src/libexpr/flake/flakeref.hh

@@ -12,10 +12,33 @@ class Store;
 
 typedef std::string FlakeId;
 
+/* A flake reference specifies how to fetch a flake or raw source
+ * (e.g. from a Git repository).  It is created from a URL-like syntax
+ * (e.g. 'github:NixOS/patchelf'), an attrset representation (e.g. '{
+ * type="github"; owner = "NixOS"; repo = "patchelf"; }'), or a local
+ * path.
+ *
+ * Each flake will have a number of FlakeRef objects: one for each
+ * input to the flake.
+ *
+ * The normal method of constructing a FlakeRef is by starting with an
+ * input description (usually the attrs or a url from the flake file),
+ * locating a fetcher for that input, and then capturing the Input
+ * object that fetcher generates (usually via
+ * FlakeRef::fromAttrs(attrs) or parseFlakeRef(url) calls).
+ *
+ * The actual fetch not have been performed yet (i.e. a FlakeRef may
+ * be lazy), but the fetcher can be invoked at any time via the
+ * FlakeRef to ensure the store is populated with this input.
+ */
+
 struct FlakeRef
 {
+    /* fetcher-specific representation of the input, sufficient to
+       perform the fetch operation. */
     fetchers::Input input;
 
+    /* sub-path within the fetched input that represents this input */
     Path subdir;
 
     bool operator==(const FlakeRef & other) const;

src/libexpr/flake/lockfile.cc

@@ -34,7 +34,8 @@ LockedNode::LockedNode(const nlohmann::json & json)
     , isFlake(json.find("flake") != json.end() ? (bool) json["flake"] : true)
 {
     if (!lockedRef.input.isImmutable())
-        throw Error("lockfile contains mutable lock '%s'", attrsToJson(lockedRef.input.toAttrs()));
+        throw Error("lockfile contains mutable lock '%s'",
+            fetchers::attrsToJSON(lockedRef.input.toAttrs()));
 }
 
 StorePath LockedNode::computeStorePath(Store & store) const
@@ -77,7 +78,7 @@ LockFile::LockFile(const nlohmann::json & json, const Path & path)
     {
         if (jsonNode.find("inputs") == jsonNode.end()) return;
         for (auto & i : jsonNode["inputs"].items()) {
-            if (i.value().is_array()) {
+            if (i.value().is_array()) { // FIXME: remove, obsolete
                 InputPath path;
                 for (auto & j : i.value())
                     path.push_back(j);
@@ -86,10 +87,13 @@ LockFile::LockFile(const nlohmann::json & json, const Path & path)
                 std::string inputKey = i.value();
                 auto k = nodeMap.find(inputKey);
                 if (k == nodeMap.end()) {
-                    auto jsonNode2 = json["nodes"][inputKey];
-                    auto input = std::make_shared<LockedNode>(jsonNode2);
+                    auto nodes = json["nodes"];
+                    auto jsonNode2 = nodes.find(inputKey);
+                    if (jsonNode2 == nodes.end())
+                        throw Error("lock file references missing node '%s'", inputKey);
+                    auto input = std::make_shared<LockedNode>(*jsonNode2);
                     k = nodeMap.insert_or_assign(inputKey, input).first;
-                    getInputs(*input, jsonNode2);
+                    getInputs(*input, *jsonNode2);
                 }
                 if (auto child = std::dynamic_pointer_cast<LockedNode>(k->second))
                     node.inputs.insert_or_assign(i.key(), child);
@@ -110,7 +114,7 @@ LockFile::LockFile(const nlohmann::json & json, const Path & path)
     // a bit since we don't need to worry about cycles.
 }
 
-nlohmann::json LockFile::toJson() const
+nlohmann::json LockFile::toJSON() const
 {
     nlohmann::json nodes;
     std::unordered_map<std::shared_ptr<const Node>, std::string> nodeKeys;
@@ -154,8 +158,8 @@ nlohmann::json LockFile::toJson() const
         }
 
         if (auto lockedNode = std::dynamic_pointer_cast<const LockedNode>(node)) {
-            n["original"] = fetchers::attrsToJson(lockedNode->originalRef.toAttrs());
-            n["locked"] = fetchers::attrsToJson(lockedNode->lockedRef.toAttrs());
+            n["original"] = fetchers::attrsToJSON(lockedNode->originalRef.toAttrs());
+            n["locked"] = fetchers::attrsToJSON(lockedNode->lockedRef.toAttrs());
             if (!lockedNode->isFlake) n["flake"] = false;
         }
 
@@ -174,7 +178,7 @@ nlohmann::json LockFile::toJson() const
 
 std::string LockFile::to_string() const
 {
-    return toJson().dump(2);
+    return toJSON().dump(2);
 }
 
 LockFile LockFile::read(const Path & path)
@@ -185,7 +189,7 @@ LockFile LockFile::read(const Path & path)
 
 std::ostream & operator <<(std::ostream & stream, const LockFile & lockFile)
 {
-    stream << lockFile.toJson().dump(2);
+    stream << lockFile.toJSON().dump(2);
     return stream;
 }
 
@@ -223,7 +227,7 @@ bool LockFile::isImmutable() const
 bool LockFile::operator ==(const LockFile & other) const
 {
     // FIXME: slow
-    return toJson() == other.toJson();
+    return toJSON() == other.toJSON();
 }
 
 InputPath parseInputPath(std::string_view s)

src/libexpr/flake/lockfile.hh

@@ -52,7 +52,7 @@ struct LockFile
     LockFile() {};
     LockFile(const nlohmann::json & json, const Path & path);
 
-    nlohmann::json toJson() const;
+    nlohmann::json toJSON() const;
 
     std::string to_string() const;
 

src/libexpr/get-drvs.cc

@@ -10,14 +10,14 @@
 namespace nix {
 
 
-DrvInfo::DrvInfo(EvalState & state, const string & attrPath, Bindings * attrs)
+DrvInfo::DrvInfo(EvalState & state, const string & attrPath, std::optional<Value> attrs)
     : state(&state), attrs(attrs), attrPath(attrPath)
 {
 }
 
 
 DrvInfo::DrvInfo(EvalState & state, ref<Store> store, const std::string & drvPathWithOutputs)
-    : state(&state), attrs(nullptr), attrPath("")
+    : state(&state), attrs(std::nullopt), attrPath("")
 {
     auto [drvPath, selectedOutputs] = store->parsePathWithOutputs(drvPathWithOutputs);
 
@@ -46,12 +46,28 @@ DrvInfo::DrvInfo(EvalState & state, ref<Store> store, const std::string & drvPat
 }
 
 
+std::optional<Value> DrvInfo::queryOptionalAttr(const Symbol attrName) const
+{
+    // Erk
+    Value * mutableAttrs = const_cast<Value*>(&attrs.value());
+    auto accessResult = state->getOptionalAttrField(*mutableAttrs, {attrName}, noPos);
+    return accessResult.error ? std::nullopt : std::optional{*accessResult.value};
+}
+
+Value DrvInfo::queryAttr(const Symbol attrName) const
+{
+    Value res;
+    // Erk
+    Value * mutableAttrs = const_cast<Value*>(&attrs.value());
+    state->getAttrField(*mutableAttrs, {attrName}, noPos, res);
+    return res;
+}
+
 string DrvInfo::queryName() const
 {
     if (name == "" && attrs) {
-        auto i = attrs->find(state->sName);
-        if (i == attrs->end()) throw TypeError("derivation name missing");
-        name = state->forceStringNoCtx(*i->value);
+        Value nameVal = queryAttr(state->sName);
+        name = state->forceStringNoCtx(nameVal);
     }
     return name;
 }
@@ -60,8 +76,8 @@ string DrvInfo::queryName() const
 string DrvInfo::querySystem() const
 {
     if (system == "" && attrs) {
-        auto i = attrs->find(state->sSystem);
-        system = i == attrs->end() ? "unknown" : state->forceStringNoCtx(*i->value, *i->pos);
+        auto maybeSystemVal = queryOptionalAttr(state->sSystem);
+        system = maybeSystemVal ? state->forceStringNoCtx(*maybeSystemVal) : "unknown";
     }
     return system;
 }
@@ -70,9 +86,10 @@ string DrvInfo::querySystem() const
 string DrvInfo::queryDrvPath() const
 {
     if (drvPath == "" && attrs) {
-        Bindings::iterator i = attrs->find(state->sDrvPath);
+        auto drvPathVal = queryOptionalAttr(state->sDrvPath);
         PathSet context;
-        drvPath = i != attrs->end() ? state->coerceToPath(*i->pos, *i->value, context) : "";
+        if (drvPathVal)
+            drvPath = state->coerceToPath(noPos, *drvPathVal, context);
     }
     return drvPath;
 }
@@ -81,10 +98,10 @@ string DrvInfo::queryDrvPath() const
 string DrvInfo::queryOutPath() const
 {
     if (!outPath && attrs) {
-        Bindings::iterator i = attrs->find(state->sOutPath);
+        auto val = queryOptionalAttr(state->sOutPath);
         PathSet context;
-        if (i != attrs->end())
-            outPath = state->coerceToPath(*i->pos, *i->value, context);
+        if (val)
+            outPath = state->coerceToPath(noPos, *val, context);
     }
     if (!outPath)
         throw UnimplementedError("CA derivations are not yet supported");
@@ -96,23 +113,23 @@ DrvInfo::Outputs DrvInfo::queryOutputs(bool onlyOutputsToInstall)
 {
     if (outputs.empty()) {
         /* Get the ‘outputs’ list. */
-        Bindings::iterator i;
-        if (attrs && (i = attrs->find(state->sOutputs)) != attrs->end()) {
-            state->forceList(*i->value, *i->pos);
+        std::optional<Value> outputsList;
+        if (attrs && (outputsList = queryOptionalAttr(state->sOutputs))) {
+            state->forceList(*outputsList);
 
             /* For each output... */
-            for (unsigned int j = 0; j < i->value->listSize(); ++j) {
+            for (unsigned int j = 0; j < outputsList->listSize(); ++j) {
                 /* Evaluate the corresponding set. */
-                string name = state->forceStringNoCtx(*i->value->listElems()[j], *i->pos);
-                Bindings::iterator out = attrs->find(state->symbols.create(name));
-                if (out == attrs->end()) continue; // FIXME: throw error?
-                state->forceAttrs(*out->value);
+                string name = state->forceStringNoCtx(*outputsList->listElems()[j]);
+                auto out = queryOptionalAttr(state->symbols.create(name));
+                if (!out) continue;
+                state->forceAttrs(*out);
 
                 /* And evaluate its ‘outPath’ attribute. */
-                Bindings::iterator outPath = out->value->attrs->find(state->sOutPath);
-                if (outPath == out->value->attrs->end()) continue; // FIXME: throw error?
+                Value outPath;
+                state->getAttrField(*out, {state->sOutPath}, noPos, outPath);
                 PathSet context;
-                outputs[name] = state->coerceToPath(*outPath->pos, *outPath->value, context);
+                outputs[name] = state->coerceToPath(noPos, outPath, context);
             }
         } else
             outputs["out"] = queryOutPath();
@@ -128,7 +145,7 @@ DrvInfo::Outputs DrvInfo::queryOutputs(bool onlyOutputsToInstall)
     if (!outTI->isList()) throw errMsg;
     Outputs result;
     for (auto i = outTI->listElems(); i != outTI->listElems() + outTI->listSize(); ++i) {
-        if ((*i)->type != tString) throw errMsg;
+        if ((*i)->type() != nString) throw errMsg;
         auto out = outputs.find((*i)->string.s);
         if (out == outputs.end()) throw errMsg;
         result.insert(*out);
@@ -140,8 +157,8 @@ DrvInfo::Outputs DrvInfo::queryOutputs(bool onlyOutputsToInstall)
 string DrvInfo::queryOutputName() const
 {
     if (outputName == "" && attrs) {
-        Bindings::iterator i = attrs->find(state->sOutputName);
-        outputName = i != attrs->end() ? state->forceStringNoCtx(*i->value) : "";
+        auto val = queryOptionalAttr(state->sOutputName);
+        outputName = val ? state->forceStringNoCtx(*val) : "";
     }
     return outputName;
 }
@@ -151,10 +168,10 @@ Bindings * DrvInfo::getMeta()
 {
     if (meta) return meta;
     if (!attrs) return 0;
-    Bindings::iterator a = attrs->find(state->sMeta);
-    if (a == attrs->end()) return 0;
-    state->forceAttrs(*a->value, *a->pos);
-    meta = a->value->attrs;
+    auto val = queryOptionalAttr(state->sMeta);
+    if (!val) return 0;
+    state->forceAttrs(*val);
+    meta = val->attrs;
     return meta;
 }
 
@@ -172,20 +189,20 @@ StringSet DrvInfo::queryMetaNames()
 bool DrvInfo::checkMeta(Value & v)
 {
     state->forceValue(v);
-    if (v.isList()) {
+    if (v.type() == nList) {
         for (unsigned int n = 0; n < v.listSize(); ++n)
             if (!checkMeta(*v.listElems()[n])) return false;
         return true;
     }
-    else if (v.type == tAttrs) {
+    else if (v.type() == nAttrs) {
         Bindings::iterator i = v.attrs->find(state->sOutPath);
         if (i != v.attrs->end()) return false;
         for (auto & i : *v.attrs)
             if (!checkMeta(*i.value)) return false;
         return true;
     }
-    else return v.type == tInt || v.type == tBool || v.type == tString ||
-                v.type == tFloat;
+    else return v.type() == nInt || v.type() == nBool || v.type() == nString ||
+                v.type() == nFloat;
 }
 
 
@@ -201,7 +218,7 @@ Value * DrvInfo::queryMeta(const string & name)
 string DrvInfo::queryMetaString(const string & name)
 {
     Value * v = queryMeta(name);
-    if (!v || v->type != tString) return "";
+    if (!v || v->type() != nString) return "";
     return v->string.s;
 }
 
@@ -210,12 +227,12 @@ NixInt DrvInfo::queryMetaInt(const string & name, NixInt def)
 {
     Value * v = queryMeta(name);
     if (!v) return def;
-    if (v->type == tInt) return v->integer;
-    if (v->type == tString) {
+    if (v->type() == nInt) return v->integer;
+    if (v->type() == nString) {
         /* Backwards compatibility with before we had support for
            integer meta fields. */
-        NixInt n;
-        if (string2Int(v->string.s, n)) return n;
+        if (auto n = string2Int<NixInt>(v->string.s))
+            return *n;
     }
     return def;
 }
@@ -224,12 +241,12 @@ NixFloat DrvInfo::queryMetaFloat(const string & name, NixFloat def)
 {
     Value * v = queryMeta(name);
     if (!v) return def;
-    if (v->type == tFloat) return v->fpoint;
-    if (v->type == tString) {
+    if (v->type() == nFloat) return v->fpoint;
+    if (v->type() == nString) {
         /* Backwards compatibility with before we had support for
            float meta fields. */
-        NixFloat n;
-        if (string2Float(v->string.s, n)) return n;
+        if (auto n = string2Float<NixFloat>(v->string.s))
+            return *n;
     }
     return def;
 }
@@ -239,8 +256,8 @@ bool DrvInfo::queryMetaBool(const string & name, bool def)
 {
     Value * v = queryMeta(name);
     if (!v) return def;
-    if (v->type == tBool) return v->boolean;
-    if (v->type == tString) {
+    if (v->type() == nBool) return v->boolean;
+    if (v->type() == nString) {
         /* Backwards compatibility with before we had support for
            Boolean meta fields. */
         if (strcmp(v->string.s, "true") == 0) return true;
@@ -285,7 +302,7 @@ static bool getDerivation(EvalState & state, Value & v,
            derivation {...}; y = x;}'. */
         if (!done.insert(v.attrs).second) return false;
 
-        DrvInfo drv(state, attrPath, v.attrs);
+        DrvInfo drv(state, attrPath, v);
 
         drv.queryName();
 
@@ -331,7 +348,7 @@ static void getDerivations(EvalState & state, Value & vIn,
     /* Process the expression. */
     if (!getDerivation(state, v, pathPrefix, drvs, done, ignoreAssertionFailures)) ;
 
-    else if (v.type == tAttrs) {
+    else if (v.type() == nAttrs) {
 
         /* !!! undocumented hackery to support combining channels in
            nix-env.cc. */
@@ -353,7 +370,7 @@ static void getDerivations(EvalState & state, Value & vIn,
                 /* If the value of this attribute is itself a set,
                    should we recurse into it?  => Only if it has a
                    `recurseForDerivations = true' attribute. */
-                if (i->value->type == tAttrs) {
+                if (i->value->type() == nAttrs) {
                     Bindings::iterator j = i->value->attrs->find(state.sRecurseForDerivations);
                     if (j != i->value->attrs->end() && state.forceBool(*j->value, *j->pos))
                         getDerivations(state, *i->value, pathPrefix2, autoArgs, drvs, done, ignoreAssertionFailures);
@@ -362,7 +379,7 @@ static void getDerivations(EvalState & state, Value & vIn,
         }
     }
 
-    else if (v.isList()) {
+    else if (v.type() == nList) {
         for (unsigned int n = 0; n < v.listSize(); ++n) {
             string pathPrefix2 = addToPath(pathPrefix, (format("%1%") % n).str());
             if (getDerivation(state, *v.listElems()[n], pathPrefix2, drvs, done, ignoreAssertionFailures))

src/libexpr/get-drvs.hh

@@ -26,7 +26,8 @@ private:
 
     bool failed = false; // set if we get an AssertionError
 
-    Bindings * attrs = nullptr, * meta = nullptr;
+    std::optional<Value> attrs;
+    Bindings * meta = nullptr;
 
     Bindings * getMeta();
 
@@ -36,9 +37,12 @@ public:
     string attrPath; /* path towards the derivation */
 
     DrvInfo(EvalState & state) : state(&state) { };
-    DrvInfo(EvalState & state, const string & attrPath, Bindings * attrs);
+    DrvInfo(EvalState & state, const string & attrPath, std::optional<Value> attrs);
     DrvInfo(EvalState & state, ref<Store> store, const std::string & drvPathWithOutputs);
 
+    Value queryAttr(const Symbol attrName) const;
+    std::optional<Value> queryOptionalAttr(const Symbol attrName) const;
+
     string queryName() const;
     string querySystem() const;
     string queryDrvPath() const;

src/libexpr/lexer.l

@@ -12,6 +12,10 @@
 
 
 %{
+#ifdef __clang__
+#pragma clang diagnostic ignored "-Wunneeded-internal-declaration"
+#endif
+
 #include <boost/lexical_cast.hpp>
 
 #include "nixexpr.hh"

src/libexpr/local.mk

@@ -15,7 +15,7 @@ libexpr_CXXFLAGS += -I src/libutil -I src/libstore -I src/libfetchers -I src/lib
 
 libexpr_LIBS = libutil libstore libfetchers
 
-libexpr_LDFLAGS =
+libexpr_LDFLAGS = -lboost_context
 ifneq ($(OS), FreeBSD)
  libexpr_LDFLAGS += -ldl
 endif
@@ -35,13 +35,11 @@ $(d)/lexer-tab.cc $(d)/lexer-tab.hh: $(d)/lexer.l
 
 clean-files += $(d)/parser-tab.cc $(d)/parser-tab.hh $(d)/lexer-tab.cc $(d)/lexer-tab.hh
 
-dist-files += $(d)/parser-tab.cc $(d)/parser-tab.hh $(d)/lexer-tab.cc $(d)/lexer-tab.hh
-
 $(eval $(call install-file-in, $(d)/nix-expr.pc, $(prefix)/lib/pkgconfig, 0644))
 
 $(foreach i, $(wildcard src/libexpr/flake/*.hh), \
   $(eval $(call install-file-in, $(i), $(includedir)/nix/flake, 0644)))
 
-$(d)/primops.cc: $(d)/imported-drv-to-derivation.nix.gen.hh $(d)/primops/derivation.nix.gen.hh
+$(d)/primops.cc: $(d)/imported-drv-to-derivation.nix.gen.hh $(d)/primops/derivation.nix.gen.hh $(d)/fetchurl.nix.gen.hh
 
 $(d)/flake/flake.cc: $(d)/flake/call-flake.nix.gen.hh

src/libexpr/nixexpr.cc

@@ -284,7 +284,7 @@ void ExprVar::bindVars(const StaticEnv & env)
        "undefined variable" error now. */
     if (withLevel == -1)
         throw UndefinedVarError({
-            .hint = hintfmt("undefined variable '%1%'", name),
+            .msg = hintfmt("undefined variable '%1%'", name),
             .errPos = pos
         });
     fromWith = true;

src/libexpr/nixexpr.hh

@@ -17,6 +17,7 @@ MakeError(ThrownError, AssertionError);
 MakeError(Abort, EvalError);
 MakeError(TypeError, EvalError);
 MakeError(UndefinedVarError, Error);
+MakeError(MissingArgumentError, Error);
 MakeError(RestrictedPathError, Error);
 
 
@@ -129,7 +130,7 @@ struct ExprPath : Expr
 {
     string s;
     Value v;
-    ExprPath(const string & s) : s(s) { mkPathNoCopy(v, this->s.c_str()); };
+    ExprPath(const string & s) : s(s) { v.mkPath(this->s.c_str()); };
     COMMON_METHODS
     Value * maybeThunk(EvalState & state, Env & env);
 };
@@ -165,6 +166,10 @@ struct ExprSelect : Expr
     AttrPath attrPath;
     ExprSelect(const Pos & pos, Expr * e, const AttrPath & attrPath, Expr * def) : pos(pos), e(e), def(def), attrPath(attrPath) { };
     ExprSelect(const Pos & pos, Expr * e, const Symbol & name) : pos(pos), e(e), def(0) { attrPath.push_back(AttrName(name)); };
+    ExprSelect(const Pos & pos, Expr * e, const std::vector<Symbol> & names) : pos(pos), e(e), def(0) {
+        for (auto & name : names)
+            attrPath.push_back(AttrName(name));
+    };
     COMMON_METHODS
 };
 
@@ -238,7 +243,7 @@ struct ExprLambda : Expr
     {
         if (!arg.empty() && formals && formals->argNames.find(arg) != formals->argNames.end())
             throw ParseError({
-                .hint = hintfmt("duplicate formal function argument '%1%'", arg),
+                .msg = hintfmt("duplicate formal function argument '%1%'", arg),
                 .errPos = pos
             });
     };

src/libexpr/parser.y

@@ -32,7 +32,7 @@ namespace nix {
         Path basePath;
         Symbol file;
         FileOrigin origin;
-        ErrorInfo error;
+        std::optional<ErrorInfo> error;
         Symbol sLetBody;
         ParseData(EvalState & state)
             : state(state)
@@ -66,8 +66,8 @@ namespace nix {
 static void dupAttr(const AttrPath & attrPath, const Pos & pos, const Pos & prevPos)
 {
     throw ParseError({
-         .hint = hintfmt("attribute '%1%' already defined at %2%",
-            showAttrPath(attrPath), prevPos),
+         .msg = hintfmt("attribute '%1%' already defined at %2%",
+             showAttrPath(attrPath), prevPos),
          .errPos = pos
     });
 }
@@ -75,7 +75,7 @@ static void dupAttr(const AttrPath & attrPath, const Pos & pos, const Pos & prev
 static void dupAttr(Symbol attr, const Pos & pos, const Pos & prevPos)
 {
     throw ParseError({
-        .hint = hintfmt("attribute '%1%' already defined at %2%", attr, prevPos),
+        .msg = hintfmt("attribute '%1%' already defined at %2%", attr, prevPos),
         .errPos = pos
     });
 }
@@ -146,7 +146,7 @@ static void addFormal(const Pos & pos, Formals * formals, const Formal & formal)
 {
     if (!formals->argNames.insert(formal.name).second)
         throw ParseError({
-            .hint = hintfmt("duplicate formal function argument '%1%'",
+            .msg = hintfmt("duplicate formal function argument '%1%'",
                 formal.name),
             .errPos = pos
         });
@@ -258,7 +258,7 @@ static inline Pos makeCurPos(const YYLTYPE & loc, ParseData * data)
 void yyerror(YYLTYPE * loc, yyscan_t scanner, ParseData * data, const char * error)
 {
     data->error = {
-        .hint = hintfmt(error),
+        .msg = hintfmt(error),
         .errPos = makeCurPos(*loc, data)
     };
 }
@@ -338,7 +338,7 @@ expr_function
   | LET binds IN expr_function
     { if (!$2->dynamicAttrs.empty())
         throw ParseError({
-            .hint = hintfmt("dynamic attributes not allowed in let"),
+            .msg = hintfmt("dynamic attributes not allowed in let"),
             .errPos = CUR_POS
         });
       $$ = new ExprLet($2, $4);
@@ -418,7 +418,7 @@ expr_simple
       static bool noURLLiterals = settings.isExperimentalFeatureEnabled("no-url-literals");
       if (noURLLiterals)
           throw ParseError({
-              .hint = hintfmt("URL literals are disabled"),
+              .msg = hintfmt("URL literals are disabled"),
               .errPos = CUR_POS
           });
       $$ = new ExprString(data->symbols.create($1));
@@ -491,7 +491,7 @@ attrs
           delete str;
       } else
           throw ParseError({
-              .hint = hintfmt("dynamic attributes not allowed in inherit"),
+              .msg = hintfmt("dynamic attributes not allowed in inherit"),
               .errPos = makeCurPos(@2, data)
           });
     }
@@ -576,7 +576,7 @@ Expr * EvalState::parse(const char * text, FileOrigin origin,
     ParseData data(*this);
     data.origin = origin;
     switch (origin) {
-        case foFile: 
+        case foFile:
             data.file = data.symbols.create(path);
             break;
         case foStdin:
@@ -593,7 +593,7 @@ Expr * EvalState::parse(const char * text, FileOrigin origin,
     int res = yyparse(scanner, &data);
     yylex_destroy(scanner);
 
-    if (res) throw ParseError(data.error);
+    if (res) throw ParseError(data.error.value());
 
     data.result->bindVars(staticEnv);
 
@@ -698,8 +698,12 @@ Path EvalState::findFile(SearchPath & searchPath, const string & path, const Pos
         Path res = r.second + suffix;
         if (pathExists(res)) return canonPath(res);
     }
+
+    if (hasPrefix(path, "nix/"))
+        return corepkgsPrefix + path.substr(4);
+
     throw ThrownError({
-        .hint = hintfmt(evalSettings.pureEval
+        .msg = hintfmt(evalSettings.pureEval
             ? "cannot look up '<%s>' in pure evaluation mode (use '--impure' to override)"
             : "file '%s' was not found in the Nix search path (add it using $NIX_PATH or -I)",
             path),
@@ -721,8 +725,7 @@ std::pair<bool, std::string> EvalState::resolveSearchPathElem(const SearchPathEl
                         store, resolveUri(elem.second), "source", false).first.storePath) };
         } catch (FileTransferError & e) {
             logWarning({
-                .name = "Entry download",
-                .hint = hintfmt("Nix search path entry '%1%' cannot be downloaded, ignoring", elem.second)
+                .msg = hintfmt("Nix search path entry '%1%' cannot be downloaded, ignoring", elem.second)
             });
             res = { false, "" };
         }
@@ -732,8 +735,7 @@ std::pair<bool, std::string> EvalState::resolveSearchPathElem(const SearchPathEl
             res = { true, path };
         else {
             logWarning({
-                .name = "Entry not found",
-                .hint = hintfmt("warning: Nix search path entry '%1%' does not exist, ignoring", elem.second)
+                .msg = hintfmt("warning: Nix search path entry '%1%' does not exist, ignoring", elem.second)
             });
             res = { false, "" };
         }

src/libexpr/primops.cc

@@ -115,9 +115,12 @@ static void import(EvalState & state, const Pos & pos, Value & vPath, Value * vS
         state.realiseContext(context);
     } catch (InvalidPathError & e) {
         throw EvalError({
-            .hint = hintfmt("cannot import '%1%', since path '%2%' is not valid", path, e.path),
+            .msg = hintfmt("cannot import '%1%', since path '%2%' is not valid", path, e.path),
             .errPos = pos
         });
+    } catch (Error & e) {
+        e.addTrace(pos, "while importing '%s'", path);
+        throw e;
     }
 
     Path realPath = state.checkSourcePath(state.toRealPath(path, context));
@@ -164,7 +167,15 @@ static void import(EvalState & state, const Pos & pos, Value & vPath, Value * vS
         state.forceFunction(**fun, pos);
         mkApp(v, **fun, w);
         state.forceAttrs(v, pos);
-    } else {
+    }
+
+    else if (path == corepkgsPrefix + "fetchurl.nix") {
+        state.eval(state.parseExprFromString(
+            #include "fetchurl.nix.gen.hh"
+            , "/"), v);
+    }
+
+    else {
         if (!vScope)
             state.evalFile(realPath, v);
         else {
@@ -274,7 +285,7 @@ void prim_importNative(EvalState & state, const Pos & pos, Value * * args, Value
         state.realiseContext(context);
     } catch (InvalidPathError & e) {
         throw EvalError({
-            .hint = hintfmt(
+            .msg = hintfmt(
                 "cannot import '%1%', since path '%2%' is not valid",
                 path, e.path),
             .errPos = pos
@@ -314,7 +325,7 @@ void prim_exec(EvalState & state, const Pos & pos, Value * * args, Value & v)
     auto count = args[0]->listSize();
     if (count == 0) {
         throw EvalError({
-            .hint = hintfmt("at least one argument to 'exec' required"),
+            .msg = hintfmt("at least one argument to 'exec' required"),
             .errPos = pos
         });
     }
@@ -328,7 +339,7 @@ void prim_exec(EvalState & state, const Pos & pos, Value * * args, Value & v)
         state.realiseContext(context);
     } catch (InvalidPathError & e) {
         throw EvalError({
-            .hint = hintfmt("cannot execute '%1%', since path '%2%' is not valid",
+            .msg = hintfmt("cannot execute '%1%', since path '%2%' is not valid",
                 program, e.path),
             .errPos = pos
         });
@@ -356,24 +367,20 @@ static void prim_typeOf(EvalState & state, const Pos & pos, Value * * args, Valu
 {
     state.forceValue(*args[0], pos);
     string t;
-    switch (args[0]->type) {
-        case tInt: t = "int"; break;
-        case tBool: t = "bool"; break;
-        case tString: t = "string"; break;
-        case tPath: t = "path"; break;
-        case tNull: t = "null"; break;
-        case tAttrs: t = "set"; break;
-        case tList1: case tList2: case tListN: t = "list"; break;
-        case tLambda:
-        case tPrimOp:
-        case tPrimOpApp:
-            t = "lambda";
-            break;
-        case tExternal:
+    switch (args[0]->type()) {
+        case nInt: t = "int"; break;
+        case nBool: t = "bool"; break;
+        case nString: t = "string"; break;
+        case nPath: t = "path"; break;
+        case nNull: t = "null"; break;
+        case nAttrs: t = "set"; break;
+        case nList: t = "list"; break;
+        case nFunction: t = "lambda"; break;
+        case nExternal:
             t = args[0]->external->typeOf();
             break;
-        case tFloat: t = "float"; break;
-        default: abort();
+        case nFloat: t = "float"; break;
+        case nThunk: abort();
     }
     mkString(v, state.symbols.create(t));
 }
@@ -393,7 +400,7 @@ static RegisterPrimOp primop_typeOf({
 static void prim_isNull(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValue(*args[0], pos);
-    mkBool(v, args[0]->type == tNull);
+    mkBool(v, args[0]->type() == nNull);
 }
 
 static RegisterPrimOp primop_isNull({
@@ -403,7 +410,7 @@ static RegisterPrimOp primop_isNull({
       Return `true` if *e* evaluates to `null`, and `false` otherwise.
 
       > **Warning**
-      > 
+      >
       > This function is *deprecated*; just write `e == null` instead.
     )",
     .fun = prim_isNull,
@@ -413,18 +420,7 @@ static RegisterPrimOp primop_isNull({
 static void prim_isFunction(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValue(*args[0], pos);
-    bool res;
-    switch (args[0]->type) {
-        case tLambda:
-        case tPrimOp:
-        case tPrimOpApp:
-            res = true;
-            break;
-        default:
-            res = false;
-            break;
-    }
-    mkBool(v, res);
+    mkBool(v, args[0]->type() == nFunction);
 }
 
 static RegisterPrimOp primop_isFunction({
@@ -440,7 +436,7 @@ static RegisterPrimOp primop_isFunction({
 static void prim_isInt(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValue(*args[0], pos);
-    mkBool(v, args[0]->type == tInt);
+    mkBool(v, args[0]->type() == nInt);
 }
 
 static RegisterPrimOp primop_isInt({
@@ -456,7 +452,7 @@ static RegisterPrimOp primop_isInt({
 static void prim_isFloat(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValue(*args[0], pos);
-    mkBool(v, args[0]->type == tFloat);
+    mkBool(v, args[0]->type() == nFloat);
 }
 
 static RegisterPrimOp primop_isFloat({
@@ -472,7 +468,7 @@ static RegisterPrimOp primop_isFloat({
 static void prim_isString(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValue(*args[0], pos);
-    mkBool(v, args[0]->type == tString);
+    mkBool(v, args[0]->type() == nString);
 }
 
 static RegisterPrimOp primop_isString({
@@ -488,7 +484,7 @@ static RegisterPrimOp primop_isString({
 static void prim_isBool(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValue(*args[0], pos);
-    mkBool(v, args[0]->type == tBool);
+    mkBool(v, args[0]->type() == nBool);
 }
 
 static RegisterPrimOp primop_isBool({
@@ -504,7 +500,7 @@ static RegisterPrimOp primop_isBool({
 static void prim_isPath(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValue(*args[0], pos);
-    mkBool(v, args[0]->type == tPath);
+    mkBool(v, args[0]->type() == nPath);
 }
 
 static RegisterPrimOp primop_isPath({
@@ -520,20 +516,20 @@ struct CompareValues
 {
     bool operator () (const Value * v1, const Value * v2) const
     {
-        if (v1->type == tFloat && v2->type == tInt)
+        if (v1->type() == nFloat && v2->type() == nInt)
             return v1->fpoint < v2->integer;
-        if (v1->type == tInt && v2->type == tFloat)
+        if (v1->type() == nInt && v2->type() == nFloat)
             return v1->integer < v2->fpoint;
-        if (v1->type != v2->type)
+        if (v1->type() != v2->type())
             throw EvalError("cannot compare %1% with %2%", showType(*v1), showType(*v2));
-        switch (v1->type) {
-            case tInt:
+        switch (v1->type()) {
+            case nInt:
                 return v1->integer < v2->integer;
-            case tFloat:
+            case nFloat:
                 return v1->fpoint < v2->fpoint;
-            case tString:
+            case nString:
                 return strcmp(v1->string.s, v2->string.s) < 0;
-            case tPath:
+            case nPath:
                 return strcmp(v1->path, v2->path) < 0;
             default:
                 throw EvalError("cannot compare %1% with %2%", showType(*v1), showType(*v2));
@@ -558,7 +554,7 @@ static void prim_genericClosure(EvalState & state, const Pos & pos, Value * * ar
         args[0]->attrs->find(state.symbols.create("startSet"));
     if (startSet == args[0]->attrs->end())
         throw EvalError({
-            .hint = hintfmt("attribute 'startSet' required"),
+            .msg = hintfmt("attribute 'startSet' required"),
             .errPos = pos
         });
     state.forceList(*startSet->value, pos);
@@ -572,7 +568,7 @@ static void prim_genericClosure(EvalState & state, const Pos & pos, Value * * ar
         args[0]->attrs->find(state.symbols.create("operator"));
     if (op == args[0]->attrs->end())
         throw EvalError({
-            .hint = hintfmt("attribute 'operator' required"),
+            .msg = hintfmt("attribute 'operator' required"),
             .errPos = pos
         });
     state.forceValue(*op->value, pos);
@@ -594,7 +590,7 @@ static void prim_genericClosure(EvalState & state, const Pos & pos, Value * * ar
             e->attrs->find(state.symbols.create("key"));
         if (key == e->attrs->end())
             throw EvalError({
-                .hint = hintfmt("attribute 'key' required"),
+                .msg = hintfmt("attribute 'key' required"),
                 .errPos = pos
             });
         state.forceValue(*key->value, pos);
@@ -700,10 +696,14 @@ static RegisterPrimOp primop_tryEval({
       Try to shallowly evaluate *e*. Return a set containing the
       attributes `success` (`true` if *e* evaluated successfully,
       `false` if an error was thrown) and `value`, equalling *e* if
-      successful and `false` otherwise. Note that this doesn't evaluate
-      *e* deeply, so ` let e = { x = throw ""; }; in (builtins.tryEval
-      e).success ` will be `true`. Using ` builtins.deepSeq ` one can
-      get the expected result: `let e = { x = throw ""; }; in
+      successful and `false` otherwise. `tryEval` will only prevent
+      errors created by `throw` or `assert` from being thrown.
+      Errors `tryEval` will not catch are for example those created
+      by `abort` and type errors generated by builtins. Also note that
+      this doesn't evaluate *e* deeply, so `let e = { x = throw ""; };
+      in (builtins.tryEval e).success` will be `true`. Using
+      `builtins.deepSeq` one can get the expected result:
+      `let e = { x = throw ""; }; in
       (builtins.tryEval (builtins.deepSeq e e)).success` will be
       `false`.
     )",
@@ -777,7 +777,7 @@ static RegisterPrimOp primop_deepSeq({
 static void prim_trace(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValue(*args[0], pos);
-    if (args[0]->type == tString)
+    if (args[0]->type() == nString)
         printError("trace: %1%", args[0]->string.s);
     else
         printError("trace: %1%", *args[0]);
@@ -813,17 +813,19 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
 {
     state.forceAttrs(*args[0], pos);
 
+    Value & arg = *(args[0]);
+
+    auto getOptionalAttr= [&](Symbol attrName) -> std::optional<Value> {
+        auto accessResult = state.getOptionalAttrField(arg, {attrName}, pos);
+        return accessResult.error ? std::nullopt : std::optional{*accessResult.value};
+    };
+
     /* Figure out the name first (for stack backtraces). */
-    Bindings::iterator attr = args[0]->attrs->find(state.sName);
-    if (attr == args[0]->attrs->end())
-        throw EvalError({
-            .hint = hintfmt("required attribute 'name' missing"),
-            .errPos = pos
-        });
+    state.getAttrField(arg, {state.sName}, pos, v);
     string drvName;
-    Pos & posDrvName(*attr->pos);
+    auto posDrvName = pos; // FIXME: Should be tho position of the `name` attribute
     try {
-        drvName = state.forceStringNoCtx(*attr->value, pos);
+        drvName = state.forceStringNoCtx(v, pos);
     } catch (Error & e) {
         e.addTrace(posDrvName, "while evaluating the derivation attribute 'name'");
         throw;
@@ -832,15 +834,15 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
     /* Check whether attributes should be passed as a JSON file. */
     std::ostringstream jsonBuf;
     std::unique_ptr<JSONObject> jsonObject;
-    attr = args[0]->attrs->find(state.sStructuredAttrs);
-    if (attr != args[0]->attrs->end() && state.forceBool(*attr->value, pos))
+    if (auto maybeRawObj = getOptionalAttr(state.sStructuredAttrs)) {
+        if (state.forceBool(*maybeRawObj, pos))
         jsonObject = std::make_unique<JSONObject>(jsonBuf);
+    }
 
     /* Check whether null attributes should be ignored. */
     bool ignoreNulls = false;
-    attr = args[0]->attrs->find(state.sIgnoreNulls);
-    if (attr != args[0]->attrs->end())
-        ignoreNulls = state.forceBool(*attr->value, pos);
+    if (auto maybeRawIgnoreNull = getOptionalAttr(state.sIgnoreNulls))
+        ignoreNulls = state.forceBool(*maybeRawIgnoreNull, pos);
 
     /* Build the derivation expression by processing the attributes. */
     Derivation drv;
@@ -856,9 +858,9 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
     StringSet outputs;
     outputs.insert("out");
 
-    for (auto & i : args[0]->attrs->lexicographicOrder()) {
-        if (i->name == state.sIgnoreNulls) continue;
-        const string & key = i->name;
+    for (auto & currentArg : args[0]->attrs->lexicographicOrder()) {
+        if (currentArg->name == state.sIgnoreNulls) continue;
+        const string & key = currentArg->name;
         vomit("processing attribute '%1%'", key);
 
         auto handleHashMode = [&](const std::string & s) {
@@ -866,7 +868,7 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
             else if (s == "flat") ingestionMethod = FileIngestionMethod::Flat;
             else
                 throw EvalError({
-                    .hint = hintfmt("invalid value '%s' for 'outputHashMode' attribute", s),
+                    .msg = hintfmt("invalid value '%s' for 'outputHashMode' attribute", s),
                     .errPos = posDrvName
                 });
         };
@@ -876,7 +878,7 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
             for (auto & j : ss) {
                 if (outputs.find(j) != outputs.end())
                     throw EvalError({
-                        .hint = hintfmt("duplicate derivation output '%1%'", j),
+                        .msg = hintfmt("duplicate derivation output '%1%'", j),
                         .errPos = posDrvName
                     });
                 /* !!! Check whether j is a valid attribute
@@ -886,36 +888,40 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
                    the resulting set. */
                 if (j == "drv")
                     throw EvalError({
-                        .hint = hintfmt("invalid derivation output name 'drv'" ),
+                        .msg = hintfmt("invalid derivation output name 'drv'" ),
                         .errPos = posDrvName
                     });
                 outputs.insert(j);
             }
             if (outputs.empty())
                 throw EvalError({
-                    .hint = hintfmt("derivation cannot have an empty set of outputs"),
+                    .msg = hintfmt("derivation cannot have an empty set of outputs"),
                     .errPos = posDrvName
                 });
         };
 
         try {
 
+            Value argValue;
+            state.getAttrField(arg, {currentArg->name}, pos, argValue);
+
+
             if (ignoreNulls) {
-                state.forceValue(*i->value, pos);
-                if (i->value->type == tNull) continue;
+                state.forceValue(argValue, pos);
+                if (argValue.type() == nNull) continue;
             }
 
-            if (i->name == state.sContentAddressed) {
+            if (currentArg->name == state.sContentAddressed) {
                 settings.requireExperimentalFeature("ca-derivations");
-                contentAddressed = state.forceBool(*i->value, pos);
+                contentAddressed = state.forceBool(argValue, pos);
             }
 
             /* The `args' attribute is special: it supplies the
                command-line arguments to the builder. */
-            else if (i->name == state.sArgs) {
-                state.forceList(*i->value, pos);
-                for (unsigned int n = 0; n < i->value->listSize(); ++n) {
-                    string s = state.coerceToString(posDrvName, *i->value->listElems()[n], context, true);
+            else if (currentArg->name == state.sArgs) {
+                state.forceList(argValue, pos);
+                for (unsigned int n = 0; n < argValue.listSize(); ++n) {
+                    string s = state.coerceToString(posDrvName, *argValue.listElems()[n], context, true);
                     drv.args.push_back(s);
                 }
             }
@@ -926,39 +932,39 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
 
                 if (jsonObject) {
 
-                    if (i->name == state.sStructuredAttrs) continue;
+                    if (currentArg->name == state.sStructuredAttrs) continue;
 
                     auto placeholder(jsonObject->placeholder(key));
-                    printValueAsJSON(state, true, *i->value, placeholder, context);
+                    printValueAsJSON(state, true, argValue, placeholder, context);
 
-                    if (i->name == state.sBuilder)
-                        drv.builder = state.forceString(*i->value, context, posDrvName);
-                    else if (i->name == state.sSystem)
-                        drv.platform = state.forceStringNoCtx(*i->value, posDrvName);
-                    else if (i->name == state.sOutputHash)
-                        outputHash = state.forceStringNoCtx(*i->value, posDrvName);
-                    else if (i->name == state.sOutputHashAlgo)
-                        outputHashAlgo = state.forceStringNoCtx(*i->value, posDrvName);
-                    else if (i->name == state.sOutputHashMode)
-                        handleHashMode(state.forceStringNoCtx(*i->value, posDrvName));
-                    else if (i->name == state.sOutputs) {
+                    if (currentArg->name == state.sBuilder)
+                        drv.builder = state.forceString(argValue, context, posDrvName);
+                    else if (currentArg->name == state.sSystem)
+                        drv.platform = state.forceStringNoCtx(argValue, posDrvName);
+                    else if (currentArg->name == state.sOutputHash)
+                        outputHash = state.forceStringNoCtx(argValue, posDrvName);
+                    else if (currentArg->name == state.sOutputHashAlgo)
+                        outputHashAlgo = state.forceStringNoCtx(argValue, posDrvName);
+                    else if (currentArg->name == state.sOutputHashMode)
+                        handleHashMode(state.forceStringNoCtx(argValue, posDrvName));
+                    else if (currentArg->name == state.sOutputs) {
                         /* Require ‘outputs’ to be a list of strings. */
-                        state.forceList(*i->value, posDrvName);
+                        state.forceList(argValue, posDrvName);
                         Strings ss;
-                        for (unsigned int n = 0; n < i->value->listSize(); ++n)
-                            ss.emplace_back(state.forceStringNoCtx(*i->value->listElems()[n], posDrvName));
+                        for (unsigned int n = 0; n < argValue.listSize(); ++n)
+                            ss.emplace_back(state.forceStringNoCtx(*argValue.listElems()[n], posDrvName));
                         handleOutputs(ss);
                     }
 
                 } else {
-                    auto s = state.coerceToString(posDrvName, *i->value, context, true);
+                    auto s = state.coerceToString(posDrvName, argValue, context, true);
                     drv.env.emplace(key, s);
-                    if (i->name == state.sBuilder) drv.builder = s;
-                    else if (i->name == state.sSystem) drv.platform = s;
-                    else if (i->name == state.sOutputHash) outputHash = s;
-                    else if (i->name == state.sOutputHashAlgo) outputHashAlgo = s;
-                    else if (i->name == state.sOutputHashMode) handleHashMode(s);
-                    else if (i->name == state.sOutputs)
+                    if (currentArg->name == state.sBuilder) drv.builder = s;
+                    else if (currentArg->name == state.sSystem) drv.platform = s;
+                    else if (currentArg->name == state.sOutputHash) outputHash = s;
+                    else if (currentArg->name == state.sOutputHashAlgo) outputHashAlgo = s;
+                    else if (currentArg->name == state.sOutputHashMode) handleHashMode(s);
+                    else if (currentArg->name == state.sOutputs)
                         handleOutputs(tokenizeString<Strings>(s));
                 }
 
@@ -1014,20 +1020,20 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
     /* Do we have all required attributes? */
     if (drv.builder == "")
         throw EvalError({
-            .hint = hintfmt("required attribute 'builder' missing"),
+            .msg = hintfmt("required attribute 'builder' missing"),
             .errPos = posDrvName
         });
 
     if (drv.platform == "")
         throw EvalError({
-            .hint = hintfmt("required attribute 'system' missing"),
+            .msg = hintfmt("required attribute 'system' missing"),
             .errPos = posDrvName
         });
 
     /* Check whether the derivation name is valid. */
     if (isDerivation(drvName))
         throw EvalError({
-            .hint = hintfmt("derivation names are not allowed to end in '%s'", drvExtension),
+            .msg = hintfmt("derivation names are not allowed to end in '%s'", drvExtension),
             .errPos = posDrvName
         });
 
@@ -1038,7 +1044,7 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
            already content addressed. */
         if (outputs.size() != 1 || *(outputs.begin()) != "out")
             throw Error({
-                .hint = hintfmt("multiple outputs are not supported in fixed-output derivations"),
+                .msg = hintfmt("multiple outputs are not supported in fixed-output derivations"),
                 .errPos = posDrvName
             });
 
@@ -1089,18 +1095,35 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
 
         // Regular, non-CA derivation should always return a single hash and not
         // hash per output.
-        Hash h = std::get<0>(hashDerivationModulo(*state.store, Derivation(drv), true));
+        auto hashModulo = hashDerivationModulo(*state.store, Derivation(drv), true);
+        std::visit(overloaded {
+            [&](Hash h) {
+                for (auto & i : outputs) {
+                    auto outPath = state.store->makeOutputPath(i, h, drvName);
+                    drv.env[i] = state.store->printStorePath(outPath);
+                    drv.outputs.insert_or_assign(i,
+                        DerivationOutput {
+                            .output = DerivationOutputInputAddressed {
+                                .path = std::move(outPath),
+                            },
+                        });
+                }
+            },
+            [&](CaOutputHashes) {
+                // Shouldn't happen as the toplevel derivation is not CA.
+                assert(false);
+            },
+            [&](DeferredHash _) {
+                for (auto & i : outputs) {
+                    drv.outputs.insert_or_assign(i,
+                        DerivationOutput {
+                            .output = DerivationOutputDeferred{},
+                        });
+                }
+            },
+        },
+        hashModulo);
 
-        for (auto & i : outputs) {
-            auto outPath = state.store->makeOutputPath(i, h, drvName);
-            drv.env[i] = state.store->printStorePath(outPath);
-            drv.outputs.insert_or_assign(i,
-                DerivationOutput {
-                    .output = DerivationOutputInputAddressed {
-                        .path = std::move(outPath),
-                    },
-                });
-        }
     }
 
     /* Write the resulting term into the Nix store directory. */
@@ -1115,9 +1138,10 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
 
        However, we don't bother doing this for floating CA derivations because
        their "hash modulo" is indeterminate until built. */
-    if (drv.type() != DerivationType::CAFloating)
-        drvHashes.insert_or_assign(drvPath,
-            hashDerivationModulo(*state.store, Derivation(drv), false));
+    if (drv.type() != DerivationType::CAFloating) {
+        auto h = hashDerivationModulo(*state.store, Derivation(drv), false);
+        drvHashes.lock()->insert_or_assign(drvPath, h);
+    }
 
     state.mkAttrs(v, 1 + drv.outputs.size());
     mkString(*state.allocAttr(v, state.sDrvPath), drvPathS, {"=" + drvPathS});
@@ -1200,7 +1224,7 @@ static void prim_storePath(EvalState & state, const Pos & pos, Value * * args, V
     if (!state.store->isStorePath(path)) path = canonPath(path, true);
     if (!state.store->isInStore(path))
         throw EvalError({
-            .hint = hintfmt("path '%1%' is not in the Nix store", path),
+            .msg = hintfmt("path '%1%' is not in the Nix store", path),
             .errPos = pos
         });
     auto path2 = state.store->toStorePath(path).first;
@@ -1236,7 +1260,7 @@ static void prim_pathExists(EvalState & state, const Pos & pos, Value * * args,
         state.realiseContext(context);
     } catch (InvalidPathError & e) {
         throw EvalError({
-            .hint = hintfmt(
+            .msg = hintfmt(
                 "cannot check the existence of '%1%', since path '%2%' is not valid",
                 path, e.path),
             .errPos = pos
@@ -1290,7 +1314,7 @@ static void prim_dirOf(EvalState & state, const Pos & pos, Value * * args, Value
 {
     PathSet context;
     Path dir = dirOf(state.coerceToString(pos, *args[0], context, false, false));
-    if (args[0]->type == tPath) mkPath(v, dir.c_str()); else mkString(v, dir, context);
+    if (args[0]->type() == nPath) mkPath(v, dir.c_str()); else mkString(v, dir, context);
 }
 
 static RegisterPrimOp primop_dirOf({
@@ -1313,7 +1337,7 @@ static void prim_readFile(EvalState & state, const Pos & pos, Value * * args, Va
         state.realiseContext(context);
     } catch (InvalidPathError & e) {
         throw EvalError({
-            .hint = hintfmt("cannot read '%1%', since path '%2%' is not valid", path, e.path),
+            .msg = hintfmt("cannot read '%1%', since path '%2%' is not valid", path, e.path),
             .errPos = pos
         });
     }
@@ -1352,7 +1376,7 @@ static void prim_findFile(EvalState & state, const Pos & pos, Value * * args, Va
         i = v2.attrs->find(state.symbols.create("path"));
         if (i == v2.attrs->end())
             throw EvalError({
-                .hint = hintfmt("attribute 'path' missing"),
+                .msg = hintfmt("attribute 'path' missing"),
                 .errPos = pos
             });
 
@@ -1363,7 +1387,7 @@ static void prim_findFile(EvalState & state, const Pos & pos, Value * * args, Va
             state.realiseContext(context);
         } catch (InvalidPathError & e) {
             throw EvalError({
-                .hint = hintfmt("cannot find '%1%', since path '%2%' is not valid", path, e.path),
+                .msg = hintfmt("cannot find '%1%', since path '%2%' is not valid", path, e.path),
                 .errPos = pos
             });
         }
@@ -1389,7 +1413,7 @@ static void prim_hashFile(EvalState & state, const Pos & pos, Value * * args, Va
     std::optional<HashType> ht = parseHashType(type);
     if (!ht)
       throw Error({
-          .hint = hintfmt("unknown hash type '%1%'", type),
+          .msg = hintfmt("unknown hash type '%1%'", type),
           .errPos = pos
       });
 
@@ -1419,7 +1443,7 @@ static void prim_readDir(EvalState & state, const Pos & pos, Value * * args, Val
         state.realiseContext(ctx);
     } catch (InvalidPathError & e) {
         throw EvalError({
-            .hint = hintfmt("cannot read '%1%', since path '%2%' is not valid", path, e.path),
+            .msg = hintfmt("cannot read '%1%', since path '%2%' is not valid", path, e.path),
             .errPos = pos
         });
     }
@@ -1431,7 +1455,7 @@ static void prim_readDir(EvalState & state, const Pos & pos, Value * * args, Val
         Value * ent_val = state.allocAttr(v, state.symbols.create(ent.name));
         if (ent.type == DT_UNKNOWN)
             ent.type = getFileType(path + "/" + ent.name);
-        mkStringNoCopy(*ent_val,
+        ent_val->mkString(
             ent.type == DT_REG ? "regular" :
             ent.type == DT_DIR ? "directory" :
             ent.type == DT_LNK ? "symlink" :
@@ -1603,7 +1627,12 @@ static RegisterPrimOp primop_toJSON({
 static void prim_fromJSON(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     string s = state.forceStringNoCtx(*args[0], pos);
-    parseJSON(state, s, v);
+    try {
+        parseJSON(state, s, v);
+    } catch (JSONParseError &e) {
+        e.addTrace(pos, "while decoding a JSON string");
+        throw e;
+    }
 }
 
 static RegisterPrimOp primop_fromJSON({
@@ -1634,7 +1663,7 @@ static void prim_toFile(EvalState & state, const Pos & pos, Value * * args, Valu
     for (auto path : context) {
         if (path.at(0) != '/')
             throw EvalError( {
-                .hint = hintfmt(
+                .msg = hintfmt(
                     "in 'toFile': the file named '%1%' must not contain a reference "
                     "to a derivation but contains (%2%)",
                     name, path),
@@ -1785,14 +1814,14 @@ static void prim_filterSource(EvalState & state, const Pos & pos, Value * * args
     Path path = state.coerceToPath(pos, *args[1], context);
     if (!context.empty())
         throw EvalError({
-            .hint = hintfmt("string '%1%' cannot refer to other paths", path),
+            .msg = hintfmt("string '%1%' cannot refer to other paths", path),
             .errPos = pos
         });
 
     state.forceValue(*args[0], pos);
-    if (args[0]->type != tLambda)
+    if (args[0]->type() != nFunction)
         throw TypeError({
-            .hint = hintfmt(
+            .msg = hintfmt(
                 "first argument in call to 'filterSource' is not a function but %1%",
                 showType(*args[0])),
             .errPos = pos
@@ -1859,7 +1888,7 @@ static void prim_path(EvalState & state, const Pos & pos, Value * * args, Value
             path = state.coerceToPath(*attr.pos, *attr.value, context);
             if (!context.empty())
                 throw EvalError({
-                    .hint = hintfmt("string '%1%' cannot refer to other paths", path),
+                    .msg = hintfmt("string '%1%' cannot refer to other paths", path),
                     .errPos = *attr.pos
                 });
         } else if (attr.name == state.sName)
@@ -1873,13 +1902,13 @@ static void prim_path(EvalState & state, const Pos & pos, Value * * args, Value
             expectedHash = newHashAllowEmpty(state.forceStringNoCtx(*attr.value, *attr.pos), htSHA256);
         else
             throw EvalError({
-                .hint = hintfmt("unsupported argument '%1%' to 'addPath'", attr.name),
+                .msg = hintfmt("unsupported argument '%1%' to 'addPath'", attr.name),
                 .errPos = *attr.pos
             });
     }
     if (path.empty())
         throw EvalError({
-            .hint = hintfmt("'path' required"),
+            .msg = hintfmt("'path' required"),
             .errPos = pos
         });
     if (name.empty())
@@ -1895,26 +1924,26 @@ static RegisterPrimOp primop_path({
       An enrichment of the built-in path type, based on the attributes
       present in *args*. All are optional except `path`:
 
-        - path  
+        - path
           The underlying path.
 
-        - name  
+        - name
           The name of the path when added to the store. This can used to
           reference paths that have nix-illegal characters in their names,
           like `@`.
 
-        - filter  
+        - filter
           A function of the type expected by `builtins.filterSource`,
           with the same semantics.
 
-        - recursive  
+        - recursive
           When `false`, when `path` is added to the store it is with a
           flat hash, rather than a hash of the NAR serialization of the
           file. Thus, `path` must refer to a regular file, not a
           directory. This allows similar behavior to `fetchurl`. Defaults
           to `true`.
 
-        - sha256  
+        - sha256
           When provided, this is the expected hash of the file at the
           path. Evaluation will fail if the hash is incorrect, and
           providing a hash allows `builtins.path` to be used even when the
@@ -1933,13 +1962,14 @@ static RegisterPrimOp primop_path({
    strings. */
 static void prim_attrNames(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
+    auto attrNames = state.listAttrFields(*args[0], pos);
     state.forceAttrs(*args[0], pos);
 
-    state.mkList(v, args[0]->attrs->size());
+    state.mkList(v, attrNames.size());
 
     size_t n = 0;
-    for (auto & i : *args[0]->attrs)
-        mkString(*(v.listElems()[n++] = state.allocValue()), i.name);
+    for (auto & name : attrNames)
+        mkString(*(v.listElems()[n++] = state.allocValue()), name);
 
     std::sort(v.listElems(), v.listElems() + n,
               [](Value * v1, Value * v2) { return strcmp(v1->string.s, v2->string.s) < 0; });
@@ -1990,17 +2020,8 @@ void prim_getAttr(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     string attr = state.forceStringNoCtx(*args[0], pos);
     state.forceAttrs(*args[1], pos);
+    state.getAttrField(*args[1], {state.symbols.create(attr)}, pos, v);
     // !!! Should we create a symbol here or just do a lookup?
-    Bindings::iterator i = args[1]->attrs->find(state.symbols.create(attr));
-    if (i == args[1]->attrs->end())
-        throw EvalError({
-            .hint = hintfmt("attribute '%1%' missing", attr),
-            .errPos = pos
-        });
-    // !!! add to stack trace?
-    if (state.countCalls && i->pos) state.attrSelects[*i->pos]++;
-    state.forceValue(*i->value, pos);
-    v = *i->value;
 }
 
 static RegisterPrimOp primop_getAttr({
@@ -2056,7 +2077,7 @@ static RegisterPrimOp primop_hasAttr({
 static void prim_isAttrs(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValue(*args[0], pos);
-    mkBool(v, args[0]->type == tAttrs);
+    mkBool(v, args[0]->type() == nAttrs);
 }
 
 static RegisterPrimOp primop_isAttrs({
@@ -2126,7 +2147,7 @@ static void prim_listToAttrs(EvalState & state, const Pos & pos, Value * * args,
         Bindings::iterator j = v2.attrs->find(state.sName);
         if (j == v2.attrs->end())
             throw TypeError({
-                .hint = hintfmt("'name' attribute missing in a call to 'listToAttrs'"),
+                .msg = hintfmt("'name' attribute missing in a call to 'listToAttrs'"),
                 .errPos = pos
             });
         string name = state.forceStringNoCtx(*j->value, pos);
@@ -2136,7 +2157,7 @@ static void prim_listToAttrs(EvalState & state, const Pos & pos, Value * * args,
             Bindings::iterator j2 = v2.attrs->find(state.symbols.create(state.sValue));
             if (j2 == v2.attrs->end())
                 throw TypeError({
-                    .hint = hintfmt("'value' attribute missing in a call to 'listToAttrs'"),
+                    .msg = hintfmt("'value' attribute missing in a call to 'listToAttrs'"),
                     .errPos = pos
                 });
             v.attrs->push_back(Attr(sym, j2->value, j2->pos));
@@ -2236,13 +2257,13 @@ static RegisterPrimOp primop_catAttrs({
 static void prim_functionArgs(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValue(*args[0], pos);
-    if (args[0]->type == tPrimOpApp || args[0]->type == tPrimOp) {
+    if (args[0]->isPrimOpApp() || args[0]->isPrimOp()) {
         state.mkAttrs(v, 0);
         return;
     }
-    if (args[0]->type != tLambda)
+    if (!args[0]->isLambda())
         throw TypeError({
-            .hint = hintfmt("'functionArgs' requires a function"),
+            .msg = hintfmt("'functionArgs' requires a function"),
             .errPos = pos
         });
 
@@ -2319,7 +2340,7 @@ static RegisterPrimOp primop_mapAttrs({
 static void prim_isList(EvalState & state, const Pos & pos, Value * * args, Value & v)
 {
     state.forceValue(*args[0], pos);
-    mkBool(v, args[0]->isList());
+    mkBool(v, args[0]->type() == nList);
 }
 
 static RegisterPrimOp primop_isList({
@@ -2333,10 +2354,16 @@ static RegisterPrimOp primop_isList({
 
 static void elemAt(EvalState & state, const Pos & pos, Value & list, int n, Value & v)
 {
+    auto evalCache = list.getCache();
+    auto atSymbol = state.symbols.create(std::to_string(n));
+    auto cacheResult = evalCache.getValue(*state.store, {atSymbol}, v);
+    if (cacheResult.returnCode == ValueCache::CacheHit) return;
+
     state.forceList(list, pos);
+    evalCache.addListChilds(state.symbols, list.listElems(), list.listSize());
     if (n < 0 || (unsigned int) n >= list.listSize())
         throw Error({
-            .hint = hintfmt("list index %1% is out of bounds", n),
+            .msg = hintfmt("list index %1% is out of bounds", n),
             .errPos = pos
         });
     state.forceValue(*list.listElems()[n], pos);
@@ -2384,7 +2411,7 @@ static void prim_tail(EvalState & state, const Pos & pos, Value * * args, Value
     state.forceList(*args[0], pos);
     if (args[0]->listSize() == 0)
         throw Error({
-            .hint = hintfmt("'tail' called on an empty list"),
+            .msg = hintfmt("'tail' called on an empty list"),
             .errPos = pos
         });
 
@@ -2401,7 +2428,7 @@ static RegisterPrimOp primop_tail({
       the argument isn’t a list or is an empty list.
 
       > **Warning**
-      > 
+      >
       > This function should generally be avoided since it's inefficient:
       > unlike Haskell's `tail`, it takes O(n) time, so recursing over a
       > list by repeatedly calling `tail` takes O(n^2) time.
@@ -2623,7 +2650,7 @@ static void prim_genList(EvalState & state, const Pos & pos, Value * * args, Val
 
     if (len < 0)
         throw EvalError({
-            .hint = hintfmt("cannot create list of size %1%", len),
+            .msg = hintfmt("cannot create list of size %1%", len),
             .errPos = pos
         });
 
@@ -2671,7 +2698,7 @@ static void prim_sort(EvalState & state, const Pos & pos, Value * * args, Value
     auto comparator = [&](Value * a, Value * b) {
         /* Optimization: if the comparator is lessThan, bypass
            callFunction. */
-        if (args[0]->type == tPrimOp && args[0]->primOp->fun == prim_lessThan)
+        if (args[0]->isPrimOp() && args[0]->primOp->fun == prim_lessThan)
             return CompareValues()(a, b);
 
         Value vTmp1, vTmp2;
@@ -2813,7 +2840,7 @@ static void prim_add(EvalState & state, const Pos & pos, Value * * args, Value &
 {
     state.forceValue(*args[0], pos);
     state.forceValue(*args[1], pos);
-    if (args[0]->type == tFloat || args[1]->type == tFloat)
+    if (args[0]->type() == nFloat || args[1]->type() == nFloat)
         mkFloat(v, state.forceFloat(*args[0], pos) + state.forceFloat(*args[1], pos));
     else
         mkInt(v, state.forceInt(*args[0], pos) + state.forceInt(*args[1], pos));
@@ -2832,7 +2859,7 @@ static void prim_sub(EvalState & state, const Pos & pos, Value * * args, Value &
 {
     state.forceValue(*args[0], pos);
     state.forceValue(*args[1], pos);
-    if (args[0]->type == tFloat || args[1]->type == tFloat)
+    if (args[0]->type() == nFloat || args[1]->type() == nFloat)
         mkFloat(v, state.forceFloat(*args[0], pos) - state.forceFloat(*args[1], pos));
     else
         mkInt(v, state.forceInt(*args[0], pos) - state.forceInt(*args[1], pos));
@@ -2851,7 +2878,7 @@ static void prim_mul(EvalState & state, const Pos & pos, Value * * args, Value &
 {
     state.forceValue(*args[0], pos);
     state.forceValue(*args[1], pos);
-    if (args[0]->type == tFloat || args[1]->type == tFloat)
+    if (args[0]->type() == nFloat || args[1]->type() == nFloat)
         mkFloat(v, state.forceFloat(*args[0], pos) * state.forceFloat(*args[1], pos));
     else
         mkInt(v, state.forceInt(*args[0], pos) * state.forceInt(*args[1], pos));
@@ -2874,11 +2901,11 @@ static void prim_div(EvalState & state, const Pos & pos, Value * * args, Value &
     NixFloat f2 = state.forceFloat(*args[1], pos);
     if (f2 == 0)
         throw EvalError({
-            .hint = hintfmt("division by zero"),
+            .msg = hintfmt("division by zero"),
             .errPos = pos
         });
 
-    if (args[0]->type == tFloat || args[1]->type == tFloat) {
+    if (args[0]->type() == nFloat || args[1]->type() == nFloat) {
         mkFloat(v, state.forceFloat(*args[0], pos) / state.forceFloat(*args[1], pos));
     } else {
         NixInt i1 = state.forceInt(*args[0], pos);
@@ -2886,7 +2913,7 @@ static void prim_div(EvalState & state, const Pos & pos, Value * * args, Value &
         /* Avoid division overflow as it might raise SIGFPE. */
         if (i1 == std::numeric_limits<NixInt>::min() && i2 == -1)
             throw EvalError({
-                .hint = hintfmt("overflow in integer division"),
+                .msg = hintfmt("overflow in integer division"),
                 .errPos = pos
             });
 
@@ -3017,7 +3044,7 @@ static void prim_substring(EvalState & state, const Pos & pos, Value * * args, V
 
     if (start < 0)
         throw EvalError({
-            .hint = hintfmt("negative start position in 'substring'"),
+            .msg = hintfmt("negative start position in 'substring'"),
             .errPos = pos
         });
 
@@ -3068,7 +3095,7 @@ static void prim_hashString(EvalState & state, const Pos & pos, Value * * args,
     std::optional<HashType> ht = parseHashType(type);
     if (!ht)
         throw Error({
-            .hint = hintfmt("unknown hash type '%1%'", type),
+            .msg = hintfmt("unknown hash type '%1%'", type),
             .errPos = pos
         });
 
@@ -3132,12 +3159,12 @@ void prim_match(EvalState & state, const Pos & pos, Value * * args, Value & v)
         if (e.code() == std::regex_constants::error_space) {
             // limit is _GLIBCXX_REGEX_STATE_LIMIT for libstdc++
             throw EvalError({
-                .hint = hintfmt("memory limit exceeded by regular expression '%s'", re),
+                .msg = hintfmt("memory limit exceeded by regular expression '%s'", re),
                 .errPos = pos
             });
         } else {
             throw EvalError({
-                .hint = hintfmt("invalid regular expression '%s'", re),
+                .msg = hintfmt("invalid regular expression '%s'", re),
                 .errPos = pos
             });
         }
@@ -3240,12 +3267,12 @@ static void prim_split(EvalState & state, const Pos & pos, Value * * args, Value
         if (e.code() == std::regex_constants::error_space) {
             // limit is _GLIBCXX_REGEX_STATE_LIMIT for libstdc++
             throw EvalError({
-                .hint = hintfmt("memory limit exceeded by regular expression '%s'", re),
+                .msg = hintfmt("memory limit exceeded by regular expression '%s'", re),
                 .errPos = pos
             });
         } else {
             throw EvalError({
-                .hint = hintfmt("invalid regular expression '%s'", re),
+                .msg = hintfmt("invalid regular expression '%s'", re),
                 .errPos = pos
             });
         }
@@ -3325,7 +3352,7 @@ static void prim_replaceStrings(EvalState & state, const Pos & pos, Value * * ar
     state.forceList(*args[1], pos);
     if (args[0]->listSize() != args[1]->listSize())
         throw EvalError({
-            .hint = hintfmt("'from' and 'to' arguments to 'replaceStrings' have different lengths"),
+            .msg = hintfmt("'from' and 'to' arguments to 'replaceStrings' have different lengths"),
             .errPos = pos
         });
 

src/libexpr/primops/context.cc

@@ -147,7 +147,7 @@ static void prim_appendContext(EvalState & state, const Pos & pos, Value * * arg
     for (auto & i : *args[1]->attrs) {
         if (!state.store->isStorePath(i.name))
             throw EvalError({
-                .hint = hintfmt("Context key '%s' is not a store path", i.name),
+                .msg = hintfmt("Context key '%s' is not a store path", i.name),
                 .errPos = *i.pos
             });
         if (!settings.readOnlyMode)
@@ -164,7 +164,7 @@ static void prim_appendContext(EvalState & state, const Pos & pos, Value * * arg
             if (state.forceBool(*iter->value, *iter->pos)) {
                 if (!isDerivation(i.name)) {
                     throw EvalError({
-                        .hint = hintfmt("Tried to add all-outputs context of %s, which is not a derivation, to a string", i.name),
+                        .msg = hintfmt("Tried to add all-outputs context of %s, which is not a derivation, to a string", i.name),
                         .errPos = *i.pos
                     });
                 }
@@ -177,7 +177,7 @@ static void prim_appendContext(EvalState & state, const Pos & pos, Value * * arg
             state.forceList(*iter->value, *iter->pos);
             if (iter->value->listSize() && !isDerivation(i.name)) {
                 throw EvalError({
-                    .hint = hintfmt("Tried to add derivation output context of %s, which is not a derivation, to a string", i.name),
+                    .msg = hintfmt("Tried to add derivation output context of %s, which is not a derivation, to a string", i.name),
                     .errPos = *i.pos
                 });
             }

src/libexpr/primops/fetchMercurial.cc

@@ -17,7 +17,7 @@ static void prim_fetchMercurial(EvalState & state, const Pos & pos, Value * * ar
 
     state.forceValue(*args[0]);
 
-    if (args[0]->type == tAttrs) {
+    if (args[0]->type() == nAttrs) {
 
         state.forceAttrs(*args[0], pos);
 
@@ -38,14 +38,14 @@ static void prim_fetchMercurial(EvalState & state, const Pos & pos, Value * * ar
                 name = state.forceStringNoCtx(*attr.value, *attr.pos);
             else
                 throw EvalError({
-                    .hint = hintfmt("unsupported argument '%s' to 'fetchMercurial'", attr.name),
+                    .msg = hintfmt("unsupported argument '%s' to 'fetchMercurial'", attr.name),
                     .errPos = *attr.pos
                 });
         }
 
         if (url.empty())
             throw EvalError({
-                .hint = hintfmt("'url' argument required"),
+                .msg = hintfmt("'url' argument required"),
                 .errPos = pos
             });
 

src/libexpr/primops/fetchTree.cc

@@ -39,11 +39,12 @@ void emitTreeAttrs(
         // Backwards compat for `builtins.fetchGit`: dirty repos return an empty sha1 as rev
         auto emptyHash = Hash(htSHA1);
         mkString(*state.allocAttr(v, state.symbols.create("rev")), emptyHash.gitRev());
-        mkString(*state.allocAttr(v, state.symbols.create("shortRev")), emptyHash.gitRev());
+        mkString(*state.allocAttr(v, state.symbols.create("shortRev")), emptyHash.gitShortRev());
     }
 
     if (input.getType() == "git")
-        mkBool(*state.allocAttr(v, state.symbols.create("submodules")), maybeGetBoolAttr(input.attrs, "submodules").value_or(false));
+        mkBool(*state.allocAttr(v, state.symbols.create("submodules")),
+            fetchers::maybeGetBoolAttr(input.attrs, "submodules").value_or(false));
 
     if (auto revCount = input.getRevCount())
         mkInt(*state.allocAttr(v, state.symbols.create("revCount")), *revCount);
@@ -84,26 +85,26 @@ static void fetchTree(
 
     state.forceValue(*args[0]);
 
-    if (args[0]->type == tAttrs) {
+    if (args[0]->type() == nAttrs) {
         state.forceAttrs(*args[0], pos);
 
         fetchers::Attrs attrs;
 
         for (auto & attr : *args[0]->attrs) {
             state.forceValue(*attr.value);
-            if (attr.value->type == tPath || attr.value->type == tString)
+            if (attr.value->type() == nPath || attr.value->type() == nString)
                 addURI(
                     state,
                     attrs,
                     attr.name,
                     state.coerceToString(*attr.pos, *attr.value, context, false, false)
                 );
-            else if (attr.value->type == tString)
+            else if (attr.value->type() == nString)
                 addURI(state, attrs, attr.name, attr.value->string.s);
-            else if (attr.value->type == tBool)
-                attrs.emplace(attr.name, fetchers::Explicit<bool>{attr.value->boolean});
-            else if (attr.value->type == tInt)
-                attrs.emplace(attr.name, attr.value->integer);
+            else if (attr.value->type() == nBool)
+                attrs.emplace(attr.name, Explicit<bool>{attr.value->boolean});
+            else if (attr.value->type() == nInt)
+                attrs.emplace(attr.name, uint64_t(attr.value->integer));
             else
                 throw TypeError("fetchTree argument '%s' is %s while a string, Boolean or integer is expected",
                     attr.name, showType(*attr.value));
@@ -114,7 +115,7 @@ static void fetchTree(
 
         if (!attrs.count("type"))
             throw Error({
-                .hint = hintfmt("attribute 'type' is missing in call to 'fetchTree'"),
+                .msg = hintfmt("attribute 'type' is missing in call to 'fetchTree'"),
                 .errPos = pos
             });
 
@@ -152,6 +153,7 @@ static void prim_fetchTree(EvalState & state, const Pos & pos, Value * * args, V
     fetchTree(state, pos, args, v, std::nullopt);
 }
 
+// FIXME: document
 static RegisterPrimOp primop_fetchTree("fetchTree", 1, prim_fetchTree);
 
 static void fetch(EvalState & state, const Pos & pos, Value * * args, Value & v,
@@ -162,7 +164,7 @@ static void fetch(EvalState & state, const Pos & pos, Value * * args, Value & v,
 
     state.forceValue(*args[0]);
 
-    if (args[0]->type == tAttrs) {
+    if (args[0]->type() == nAttrs) {
 
         state.forceAttrs(*args[0], pos);
 
@@ -176,14 +178,14 @@ static void fetch(EvalState & state, const Pos & pos, Value * * args, Value & v,
                 name = state.forceStringNoCtx(*attr.value, *attr.pos);
             else
                 throw EvalError({
-                    .hint = hintfmt("unsupported argument '%s' to '%s'", attr.name, who),
+                    .msg = hintfmt("unsupported argument '%s' to '%s'", attr.name, who),
                     .errPos = *attr.pos
                 });
             }
 
         if (!url)
             throw EvalError({
-                .hint = hintfmt("'url' argument required"),
+                .msg = hintfmt("'url' argument required"),
                 .errPos = pos
             });
     } else
@@ -211,7 +213,7 @@ static void fetch(EvalState & state, const Pos & pos, Value * * args, Value & v,
             ? state.store->queryPathInfo(storePath)->narHash
             : hashFile(htSHA256, path);
         if (hash != *expectedHash)
-            throw Error((unsigned int) 102, "hash mismatch in file downloaded from '%s':\n  wanted: %s\n  got:    %s",
+            throw Error((unsigned int) 102, "hash mismatch in file downloaded from '%s':\n  specified: %s\n  got:       %s",
                 *url, expectedHash->to_string(Base32, true), hash.to_string(Base32, true));
     }
 
@@ -323,6 +325,11 @@ static RegisterPrimOp primop_fetchGit({
           A Boolean parameter that specifies whether submodules should be
           checked out. Defaults to `false`.
 
+        - allRefs  
+          Whether to fetch all refs of the repository. With this argument being
+          true, it's possible to load a `rev` from *any* `ref` (by default only
+          `rev`s from the specified `ref` are supported).
+
       Here are some examples of how to use `fetchGit`.
 
         - To fetch a private repository over SSH:

src/libexpr/primops/fromTOML.cc

@@ -82,7 +82,7 @@ static void prim_fromTOML(EvalState & state, const Pos & pos, Value * * args, Va
         visit(v, parser(tomlStream).parse());
     } catch (std::runtime_error & e) {
         throw EvalError({
-            .hint = hintfmt("while parsing a TOML string: %s", e.what()),
+            .msg = hintfmt("while parsing a TOML string: %s", e.what()),
             .errPos = pos
         });
     }

src/libexpr/tree-cache.cc

@@ -0,0 +1,390 @@
+#include "tree-cache.hh"
+#include "sqlite.hh"
+#include "store-api.hh"
+#include "context.hh"
+
+namespace nix::tree_cache {
+
+static const char * schema = R"sql(
+create table if not exists Attributes (
+    id          integer primary key autoincrement not null,
+    parent      integer not null,
+    name        text,
+    type        integer not null,
+    value       text,
+    context     text,
+    unique      (parent, name)
+);
+
+create index if not exists IndexByParent on Attributes(parent, name);
+)sql";
+
+struct AttrDb
+{
+    std::atomic_bool failed{false};
+
+    struct State
+    {
+        SQLite db;
+        SQLiteStmt insertAttribute;
+        SQLiteStmt updateAttribute;
+        SQLiteStmt insertAttributeWithContext;
+        SQLiteStmt queryAttribute;
+        SQLiteStmt queryAttributes;
+        std::unique_ptr<SQLiteTxn> txn;
+    };
+
+    std::unique_ptr<Sync<State>> _state;
+
+    AttrDb(const Hash & fingerprint)
+        : _state(std::make_unique<Sync<State>>())
+    {
+        auto state(_state->lock());
+
+        Path cacheDir = getCacheDir() + "/nix/eval-cache-v3";
+        createDirs(cacheDir);
+
+        Path dbPath = cacheDir + "/" + fingerprint.to_string(Base16, false) + ".sqlite";
+
+        state->db = SQLite(dbPath);
+        state->db.isCache();
+        state->db.exec(schema);
+
+        state->insertAttribute.create(state->db,
+            "insert into Attributes(parent, name, type, value) values (?, ?, ?, ?)");
+
+        state->updateAttribute.create(state->db,
+            "update Attributes set type = ?, value = ?, context = ? where id = ?");
+
+        state->insertAttributeWithContext.create(state->db,
+            "insert into Attributes(parent, name, type, value, context) values (?, ?, ?, ?, ?)");
+
+        state->queryAttribute.create(state->db,
+            "select id, type, value, context from Attributes where parent = ? and name = ?");
+
+        state->queryAttributes.create(state->db,
+            "select name from Attributes where parent = ?");
+
+        state->txn = std::make_unique<SQLiteTxn>(state->db);
+    }
+
+    ~AttrDb()
+    {
+        try {
+            auto state(_state->lock());
+            if (!failed)
+                state->txn->commit();
+            state->txn.reset();
+        } catch (...) {
+            ignoreException();
+        }
+    }
+
+    template<typename F>
+    AttrId doSQLite(F && fun)
+    {
+        if (failed) return 0;
+        try {
+            return fun();
+        } catch (SQLiteError &) {
+            ignoreException();
+            failed = true;
+            return 0;
+        }
+    }
+
+    /**
+     * Store a leaf of the tree in the db
+     */
+    AttrId addEntry(
+        const AttrKey & key,
+        const AttrValue & value)
+    {
+        return doSQLite([&]()
+        {
+            auto state(_state->lock());
+            auto rawValue = RawValue::fromVariant(value);
+
+            state->insertAttributeWithContext.use()
+                (key.first)
+                (key.second)
+                (rawValue.type)
+                (rawValue.value.value_or(""), rawValue.value.has_value())
+                (rawValue.serializeContext())
+                .exec();
+            AttrId rowId = state->db.getLastInsertedRowId();
+            assert(rowId);
+            return rowId;
+        });
+    }
+
+    std::optional<AttrId> getId(const AttrKey& key)
+    {
+        auto state(_state->lock());
+
+        auto queryAttribute(state->queryAttribute.use()(key.first)(key.second));
+        if (!queryAttribute.next()) return std::nullopt;
+
+        return (AttrType) queryAttribute.getInt(0);
+    }
+
+    AttrId setOrUpdate(const AttrKey& key, const AttrValue& value)
+    {
+        debug("cache: miss for the attribute %s", key.second);
+        if (auto existingId = getId(key)) {
+            setValue(*existingId, value);
+            return *existingId;
+        }
+        return addEntry(key, value);
+    }
+
+    void setValue(const AttrId & id, const AttrValue & value)
+    {
+        auto state(_state->lock());
+            auto rawValue = RawValue::fromVariant(value);
+
+            state->updateAttribute.use()
+                (rawValue.type)
+                (rawValue.value.value_or(""), rawValue.value.has_value())
+                (rawValue.serializeContext())
+                (id)
+                .exec();
+    }
+
+    std::optional<std::pair<AttrId, AttrValue>> getValue(AttrKey key)
+    {
+        auto state(_state->lock());
+
+        auto queryAttribute(state->queryAttribute.use()(key.first)(key.second));
+        if (!queryAttribute.next()) return {};
+
+        auto rowId = (AttrType) queryAttribute.getInt(0);
+        auto type = (AttrType) queryAttribute.getInt(1);
+
+        switch (type) {
+            case AttrType::Attrs: {
+                return {{rowId, attributeSet_t()}};
+            }
+            case AttrType::String: {
+                std::vector<std::pair<Path, std::string>> context;
+                if (!queryAttribute.isNull(3))
+                    for (auto & s : tokenizeString<std::vector<std::string>>(queryAttribute.getStr(3), ";"))
+                        context.push_back(decodeContext(s));
+                return {{rowId, string_t{queryAttribute.getStr(2), context}}};
+            }
+            case AttrType::Bool:
+                return {{rowId, wrapped_basetype<bool>{queryAttribute.getInt(2) != 0}}};
+            case AttrType::Int:
+                return {{rowId, wrapped_basetype<int64_t>{queryAttribute.getInt(2)}}};
+            case AttrType::Double:
+                return {{rowId, wrapped_basetype<double>{(double)queryAttribute.getInt(2)}}};
+            case AttrType::Unknown:
+                return {{rowId, unknown_t{}}};
+            case AttrType::Thunk:
+                return {{rowId, thunk_t{}}};
+            case AttrType::Missing:
+                return {{rowId, missing_t{key.second}}};
+            case AttrType::Failed:
+                return {{rowId, failed_t{queryAttribute.getStr(2)}}};
+            default:
+                throw Error("unexpected type in evaluation cache");
+        }
+    }
+
+    std::vector<std::string> getChildren(AttrId parentId)
+    {
+        std::vector<std::string> res;
+        auto state(_state->lock());
+
+        auto queryAttributes(state->queryAttributes.use()(parentId));
+
+        while (queryAttributes.next())
+            res.push_back(queryAttributes.getStr(0));
+
+        return res;
+    }
+};
+
+Cache::Cache(const Hash & useCache,
+        SymbolTable & symbols)
+    : db(std::make_shared<AttrDb>(useCache))
+    , symbols(symbols)
+    , rootSymbol(symbols.create(""))
+{
+}
+
+std::shared_ptr<Cache> Cache::tryCreate(const Hash & useCache, SymbolTable & symbols)
+{
+    try {
+        return std::make_shared<Cache>(useCache, symbols);
+    } catch (SQLiteError &) {
+        ignoreException();
+        return nullptr;
+    }
+}
+
+void Cache::commit()
+{
+    if (db) {
+        debug("Saving the cache");
+        auto state(db->_state->lock());
+        if (state->txn->active) {
+            state->txn->commit();
+            state->txn.reset();
+            state->txn = std::make_unique<SQLiteTxn>(state->db);
+        }
+    }
+}
+
+Cursor::Ref Cache::getRoot()
+{
+    return new Cursor(ref(shared_from_this()), std::nullopt, thunk_t{});
+}
+
+Cursor::Cursor(
+    ref<Cache> root,
+    const Parent & parent,
+    const AttrValue& value
+    )
+    : root(root)
+    , parentId(parent ? std::optional{parent->first.cachedValue.first} : std::nullopt)
+    , label(parent ? parent->second : root->rootSymbol)
+    , cachedValue({root->db->setOrUpdate(getKey(), value), value})
+{
+}
+
+Cursor::Cursor(
+    ref<Cache> root,
+    const Parent & parent,
+    const AttrId & id,
+    const AttrValue & value
+    )
+    : root(root)
+    , parentId(parent ? std::optional{parent->first.cachedValue.first} : std::nullopt)
+    , label(parent ? parent->second : root->rootSymbol)
+    , cachedValue({id, value})
+{
+}
+
+
+AttrKey Cursor::getKey()
+{
+    if (!parentId)
+        return {0, root->rootSymbol};
+    return {*parentId, label};
+}
+
+AttrValue Cursor::getCachedValue()
+{
+    return cachedValue.second;
+}
+
+void Cursor::setValue(const AttrValue & v)
+{
+    root->db->setValue(cachedValue.first, v);
+    cachedValue.second = v;
+}
+
+Cursor::Ref Cursor::addChild(const Symbol & attrPath, const AttrValue & v)
+{
+    Parent parent = {{*this, attrPath}};
+    auto childCursor = new Cursor(
+        root,
+        parent,
+        v
+    );
+    return childCursor;
+}
+
+std::vector<std::string> Cursor::getChildren()
+{
+    return root->db->getChildren(cachedValue.first);
+}
+
+std::optional<std::vector<std::string>> Cursor::getChildrenAtPath(const std::vector<Symbol> & attrPath)
+{
+    auto cursorAtPath = findAlongAttrPath(attrPath);
+    if (cursorAtPath)
+        return cursorAtPath->getChildren();
+    return std::nullopt;
+}
+
+Cursor::Ref Cursor::maybeGetAttr(const Symbol & name)
+{
+    auto rawAttr = root->db->getValue({cachedValue.first, name});
+    if (rawAttr) {
+        Parent parent = {{*this, name}};
+        debug("cache: hit for the attribute %s", cachedValue.first);
+        return new Cursor (
+            root, parent, rawAttr->first,
+            rawAttr->second);
+    }
+    if (std::holds_alternative<attributeSet_t>(cachedValue.second)) {
+        // If the parent is an attribute set but we're not present in the db,
+        // then we're not a member of this attribute set. So mark as missing
+        return addChild(name, missing_t{name});
+    }
+    return nullptr;
+}
+
+Cursor::Ref Cursor::findAlongAttrPath(const std::vector<Symbol> & attrPath)
+{
+    auto currentCursor = this;
+    for (auto & currentAccessor : attrPath) {
+        currentCursor = currentCursor->maybeGetAttr(currentAccessor);
+        if (!currentCursor)
+            break;
+        if (std::holds_alternative<missing_t>(currentCursor->cachedValue.second))
+            break;
+        if (std::holds_alternative<failed_t>(currentCursor->cachedValue.second))
+            break;
+    }
+    return currentCursor;
+}
+
+const RawValue RawValue::fromVariant(const AttrValue & value)
+{
+    RawValue res;
+    std::visit(overloaded{
+      [&](attributeSet_t x) { res.type = AttrType::Attrs; },
+      [&](string_t x) {
+        res.type = AttrType::String;
+        res.value = x.first;
+        res.context = x.second;
+      },
+      [&](wrapped_basetype<bool> x) {
+        res.type = AttrType::Bool;
+        res.value = x.value ? "1" : "0";
+      },
+      [&](wrapped_basetype<int64_t> x) {
+        res.type = AttrType::Int;
+        res.value = std::to_string(x.value);
+      },
+      [&](wrapped_basetype<double> x) {
+        res.type = AttrType::Double;
+        res.value = std::to_string(x.value);
+      },
+      [&](unknown_t x) { res.type = AttrType::Unknown; },
+      [&](missing_t x) { res.type = AttrType::Missing; },
+      [&](thunk_t x) { res.type = AttrType::Thunk; },
+      [&](failed_t x) {
+          res.type = AttrType::Failed;
+          res.value = x.error;
+      }
+    }, value);
+    return res;
+}
+
+std::string RawValue::serializeContext() const
+{
+    std::string res;
+    for (auto & elt : context) {
+        res.append(encodeContext(elt.second, elt.first));
+        res.push_back(' ');
+    }
+    if (!res.empty())
+        res.pop_back(); // Remove the trailing space
+    return res;
+}
+
+}

src/libexpr/tree-cache.hh

@@ -0,0 +1,156 @@
+/**
+ * caching for a tree-like data structure (like Nix values)
+ *
+ * The cache is an sqlite db whose rows are the nodes of the tree, with a
+ * pointer to their parent (except for the root of course)
+ */
+
+#pragma once
+
+#include "sync.hh"
+#include "hash.hh"
+#include "symbol-table.hh"
+
+#include <functional>
+#include <variant>
+
+namespace nix::tree_cache {
+
+struct AttrDb;
+class Cursor;
+
+class Cache : public std::enable_shared_from_this<Cache>
+{
+private:
+    friend class Cursor;
+
+    /**
+     * The database holding the cache
+     */
+    std::shared_ptr<AttrDb> db;
+
+    SymbolTable & symbols;
+
+    /**
+     * Distinguished symbol indicating the root of the tree
+     */
+    const Symbol rootSymbol;
+
+public:
+
+    Cache(
+        const Hash & useCache,
+        SymbolTable & symbols
+    );
+
+    static std::shared_ptr<Cache> tryCreate(const Hash & useCache, SymbolTable & symbols);
+
+    Cursor * getRoot();
+
+    /**
+     * Flush the cache to disk
+     */
+    void commit();
+};
+
+enum AttrType {
+    Unknown = 0,
+    Attrs = 1,
+    String = 2,
+    Bool = 3,
+    Int = 4,
+    Double = 5,
+    Thunk = 6,
+    Missing = 7, // Missing fields of attribute sets
+    Failed = 8,
+};
+
+struct attributeSet_t {};
+struct unknown_t {};
+struct thunk_t {};
+struct failed_t { string error; };
+struct missing_t { Symbol attrName; };
+
+// Putting several different primitive types in an `std::variant` partially
+// breaks the `std::visit(overloaded{...` hackery because of the implicit cast
+// from one to another which breaks the exhaustiveness check.
+// So we wrap them in a trivial class just to force the disambiguation
+template<typename T>
+struct wrapped_basetype{ T value; };
+
+typedef uint64_t AttrId;
+
+typedef std::pair<AttrId, Symbol> AttrKey;
+typedef std::pair<std::string, std::vector<std::pair<Path, std::string>>> string_t;
+
+typedef std::variant<
+    attributeSet_t,
+    string_t,
+    unknown_t,
+    thunk_t,
+    missing_t,
+    failed_t,
+    wrapped_basetype<bool>,
+    wrapped_basetype<int64_t>,
+    wrapped_basetype<double>
+> AttrValue;
+
+struct RawValue {
+    AttrType type;
+    std::optional<std::string> value;
+    std::vector<std::pair<Path, std::string>> context;
+
+    std::string serializeContext() const;
+
+    static const RawValue fromVariant(const AttrValue&);
+    AttrValue toVariant() const;
+};
+
+/**
+ * View inside the cache.
+ *
+ * A `Cursor` represents a node in the cached tree (be it a leaf or not)
+ */
+class Cursor : public std::enable_shared_from_this<Cursor>
+{
+    /**
+     * The overall cache of which this cursor is a view
+     */
+    ref<Cache> root;
+
+    typedef std::optional<std::pair<Cursor&, Symbol>> Parent;
+
+    std::optional<AttrId> parentId;
+    Symbol label;
+
+    std::pair<AttrId, AttrValue> cachedValue;
+
+    /**
+     * Get the identifier for this node in the database
+     */
+    AttrKey getKey();
+
+public:
+
+    using Ref = Cursor*;
+
+    // Create a new cache entry
+    Cursor(ref<Cache> root, const Parent & parent, const AttrValue&);
+    // Build a cursor from an existing cache entry
+    Cursor(ref<Cache> root, const Parent & parent, const AttrId& id, const AttrValue&);
+
+    AttrValue getCachedValue();
+
+    void setValue(const AttrValue & v);
+
+    Ref addChild(const Symbol & attrPath, const AttrValue & v);
+
+    Ref findAlongAttrPath(const std::vector<Symbol> & attrPath);
+    Ref maybeGetAttr(const Symbol & attrPath);
+
+
+    std::vector<std::string> getChildren();
+    std::optional<std::vector<std::string>> getChildrenAtPath(const std::vector<Symbol> & attrPath);
+};
+
+}

src/libexpr/value-to-json.cc

@@ -16,30 +16,30 @@ void printValueAsJSON(EvalState & state, bool strict,
 
     if (strict) state.forceValue(v);
 
-    switch (v.type) {
+    switch (v.type()) {
 
-        case tInt:
+        case nInt:
             out.write(v.integer);
             break;
 
-        case tBool:
+        case nBool:
             out.write(v.boolean);
             break;
 
-        case tString:
+        case nString:
             copyContext(v, context);
             out.write(v.string.s);
             break;
 
-        case tPath:
+        case nPath:
             out.write(state.copyPathToStore(context, v.path));
             break;
 
-        case tNull:
+        case nNull:
             out.write(nullptr);
             break;
 
-        case tAttrs: {
+        case nAttrs: {
             auto maybeString = state.tryAttrsToString(noPos, v, context, false, false);
             if (maybeString) {
                 out.write(*maybeString);
@@ -61,7 +61,7 @@ void printValueAsJSON(EvalState & state, bool strict,
             break;
         }
 
-        case tList1: case tList2: case tListN: {
+        case nList: {
             auto list(out.list());
             for (unsigned int n = 0; n < v.listSize(); ++n) {
                 auto placeholder(list.placeholder());
@@ -70,15 +70,18 @@ void printValueAsJSON(EvalState & state, bool strict,
             break;
         }
 
-        case tExternal:
+        case nExternal:
             v.external->printValueAsJSON(state, strict, out, context);
             break;
 
-        case tFloat:
+        case nFloat:
             out.write(v.fpoint);
             break;
 
-        default:
+        case nThunk:
+            throw TypeError("cannot convert %1% to JSON", showType(v));
+
+        case nFunction:
             throw TypeError("cannot convert %1% to JSON", showType(v));
     }
 }

src/libexpr/value-to-xml.cc

@@ -58,31 +58,31 @@ static void printValueAsXML(EvalState & state, bool strict, bool location,
 
     if (strict) state.forceValue(v);
 
-    switch (v.type) {
+    switch (v.type()) {
 
-        case tInt:
+        case nInt:
             doc.writeEmptyElement("int", singletonAttrs("value", (format("%1%") % v.integer).str()));
             break;
 
-        case tBool:
+        case nBool:
             doc.writeEmptyElement("bool", singletonAttrs("value", v.boolean ? "true" : "false"));
             break;
 
-        case tString:
+        case nString:
             /* !!! show the context? */
             copyContext(v, context);
             doc.writeEmptyElement("string", singletonAttrs("value", v.string.s));
             break;
 
-        case tPath:
+        case nPath:
             doc.writeEmptyElement("path", singletonAttrs("value", v.path));
             break;
 
-        case tNull:
+        case nNull:
             doc.writeEmptyElement("null");
             break;
 
-        case tAttrs:
+        case nAttrs:
             if (state.isDerivation(v)) {
                 XMLAttrs xmlAttrs;
 
@@ -92,14 +92,14 @@ static void printValueAsXML(EvalState & state, bool strict, bool location,
                 a = v.attrs->find(state.sDrvPath);
                 if (a != v.attrs->end()) {
                     if (strict) state.forceValue(*a->value);
-                    if (a->value->type == tString)
+                    if (a->value->type() == nString)
                         xmlAttrs["drvPath"] = drvPath = a->value->string.s;
                 }
 
                 a = v.attrs->find(state.sOutPath);
                 if (a != v.attrs->end()) {
                     if (strict) state.forceValue(*a->value);
-                    if (a->value->type == tString)
+                    if (a->value->type() == nString)
                         xmlAttrs["outPath"] = a->value->string.s;
                 }
 
@@ -118,14 +118,19 @@ static void printValueAsXML(EvalState & state, bool strict, bool location,
 
             break;
 
-        case tList1: case tList2: case tListN: {
+        case nList: {
             XMLOpenElement _(doc, "list");
             for (unsigned int n = 0; n < v.listSize(); ++n)
                 printValueAsXML(state, strict, location, *v.listElems()[n], doc, context, drvsSeen);
             break;
         }
 
-        case tLambda: {
+        case nFunction: {
+            if (!v.isLambda()) {
+                // FIXME: Serialize primops and primopapps
+                doc.writeEmptyElement("unevaluated");
+                break;
+            }
             XMLAttrs xmlAttrs;
             if (location) posToXML(xmlAttrs, v.lambda.fun->pos);
             XMLOpenElement _(doc, "function", xmlAttrs);
@@ -143,15 +148,15 @@ static void printValueAsXML(EvalState & state, bool strict, bool location,
             break;
         }
 
-        case tExternal:
+        case nExternal:
             v.external->printValueAsXML(state, strict, location, doc, context, drvsSeen);
             break;
 
-        case tFloat:
+        case nFloat:
             doc.writeEmptyElement("float", singletonAttrs("value", (format("%1%") % v.fpoint).str()));
             break;
 
-        default:
+        case nThunk:
             doc.writeEmptyElement("unevaluated");
     }
 }

src/libexpr/value.hh

@@ -1,6 +1,7 @@
 #pragma once
 
 #include "symbol-table.hh"
+#include "tree-cache.hh"
 
 #if HAVE_BOEHMGC
 #include <gc/gc_allocator.h>
@@ -27,8 +28,24 @@ typedef enum {
     tPrimOpApp,
     tExternal,
     tFloat
-} ValueType;
+} InternalType;
 
+// This type abstracts over all actual value types in the language,
+// grouping together implementation details like tList*, different function
+// types, and types in non-normal form (so thunks and co.)
+typedef enum {
+    nThunk,
+    nInt,
+    nFloat,
+    nBool,
+    nString,
+    nPath,
+    nNull,
+    nAttrs,
+    nList,
+    nFunction,
+    nExternal
+} ValueType;
 
 class Bindings;
 struct Env;
@@ -40,8 +57,51 @@ struct Pos;
 class EvalState;
 class XMLWriter;
 class JSONPlaceholder;
+class Store;
 
 
+struct Value;
+class ValueCache {
+    tree_cache::Cursor::Ref rawCache;
+
+public:
+
+    ValueCache(tree_cache::Cursor::Ref rawCache) : rawCache(rawCache) {}
+
+    const static ValueCache empty;
+
+    enum ReturnCode {
+        // The cache result was an attribute set, so we forward it later in the
+        // chain
+        Forward,
+        CacheMiss,
+        CacheHit,
+        UnCacheable,
+        NoCacheKey,
+    };
+
+    struct CacheResult {
+        ReturnCode returnCode;
+
+        // In case the query returns a `missing_t`, the symbol that's missing
+        std::optional<Symbol> lastQueriedSymbolIfMissing;
+    };
+    CacheResult getValue(Store & store, const std::vector<Symbol> & selector, Value & dest);
+
+    ValueCache addChild(const Symbol & attrName, const Value & value);
+    ValueCache addFailedChild(const Symbol & attrName, const Error & error);
+    ValueCache addNumChild(SymbolTable & symbols, int idx, const Value & value);
+    void addAttrSetChilds(Bindings & children);
+    void addListChilds(SymbolTable & symbols, Value** elems, int listSize);
+
+    std::optional<std::vector<Symbol>> listChildren(SymbolTable&);
+    std::optional<std::vector<Symbol>> listChildrenAtPath(SymbolTable&, const std::vector<Symbol> & attrPath);
+
+    std::optional<tree_cache::AttrValue> getRawValue();
+
+    ValueCache() : rawCache(nullptr) {}
+};
+
 typedef int64_t NixInt;
 typedef double NixFloat;
 
@@ -90,7 +150,34 @@ std::ostream & operator << (std::ostream & str, const ExternalValueBase & v);
 
 struct Value
 {
-    ValueType type;
+private:
+    InternalType internalType;
+
+friend std::string showType(const Value & v);
+friend void printValue(std::ostream & str, std::set<const Value *> & active, const Value & v);
+
+public:
+    /*
+     * An optional evaluation cache (for flakes in particular).
+     * If this is set, then trying to get a value from this attrset will first
+     * try to get it from the cache
+     */
+    ValueCache evalCache;
+
+    // Functions needed to distinguish the type
+    // These should be removed eventually, by putting the functionality that's
+    // needed by callers into methods of this type
+
+    // type() == nThunk
+    inline bool isThunk() const { return internalType == tThunk; };
+    inline bool isApp() const { return internalType == tApp; };
+    inline bool isBlackhole() const { return internalType == tBlackhole; };
+
+    // type() == nFunction
+    inline bool isLambda() const { return internalType == tLambda; };
+    inline bool isPrimOp() const { return internalType == tPrimOp; };
+    inline bool isPrimOpApp() const { return internalType == tPrimOpApp; };
+
     union
     {
         NixInt integer;
@@ -147,24 +234,161 @@ struct Value
         NixFloat fpoint;
     };
 
+    // Returns the normal type of a Value. This only returns nThunk if the
+    // Value hasn't been forceValue'd
+    inline ValueType type() const
+    {
+        switch (internalType) {
+            case tInt: return nInt;
+            case tBool: return nBool;
+            case tString: return nString;
+            case tPath: return nPath;
+            case tNull: return nNull;
+            case tAttrs: return nAttrs;
+            case tList1: case tList2: case tListN: return nList;
+            case tLambda: case tPrimOp: case tPrimOpApp: return nFunction;
+            case tExternal: return nExternal;
+            case tFloat: return nFloat;
+            case tThunk: case tApp: case tBlackhole: return nThunk;
+        }
+        abort();
+    }
+
+    /* After overwriting an app node, be sure to clear pointers in the
+       Value to ensure that the target isn't kept alive unnecessarily. */
+    inline void clearValue()
+    {
+        app.left = app.right = 0;
+    }
+
+    inline void mkInt(NixInt n)
+    {
+        clearValue();
+        internalType = tInt;
+        integer = n;
+    }
+
+    inline void mkBool(bool b)
+    {
+        clearValue();
+        internalType = tBool;
+        boolean = b;
+    }
+
+    inline void mkString(const char * s, const char * * context = 0)
+    {
+        internalType = tString;
+        string.s = s;
+        string.context = context;
+    }
+
+    inline void mkPath(const char * s)
+    {
+        clearValue();
+        internalType = tPath;
+        path = s;
+    }
+
+    inline void mkNull()
+    {
+        clearValue();
+        internalType = tNull;
+    }
+
+    inline void mkAttrs(Bindings * a)
+    {
+        clearValue();
+        internalType = tAttrs;
+        attrs = a;
+    }
+
+    inline void mkList(size_t size)
+    {
+        clearValue();
+        if (size == 1)
+            internalType = tList1;
+        else if (size == 2)
+            internalType = tList2;
+        else {
+            internalType = tListN;
+            bigList.size = size;
+        }
+    }
+
+    inline void mkThunk(Env * e, Expr * ex)
+    {
+        internalType = tThunk;
+        thunk.env = e;
+        thunk.expr = ex;
+    }
+
+    inline void mkApp(Value * l, Value * r)
+    {
+        internalType = tApp;
+        app.left = l;
+        app.right = r;
+    }
+
+    inline void mkLambda(Env * e, ExprLambda * f)
+    {
+        internalType = tLambda;
+        lambda.env = e;
+        lambda.fun = f;
+    }
+
+    inline void mkBlackhole()
+    {
+        internalType = tBlackhole;
+        // Value will be overridden anyways
+    }
+
+    inline void mkPrimOp(PrimOp * p)
+    {
+        clearValue();
+        internalType = tPrimOp;
+        primOp = p;
+    }
+
+
+    inline void mkPrimOpApp(Value * l, Value * r)
+    {
+        internalType = tPrimOpApp;
+        app.left = l;
+        app.right = r;
+    }
+
+    inline void mkExternal(ExternalValueBase * e)
+    {
+        clearValue();
+        internalType = tExternal;
+        external = e;
+    }
+
+    inline void mkFloat(NixFloat n)
+    {
+        clearValue();
+        internalType = tFloat;
+        fpoint = n;
+    }
+
     bool isList() const
     {
-        return type == tList1 || type == tList2 || type == tListN;
+        return internalType == tList1 || internalType == tList2 || internalType == tListN;
     }
 
     Value * * listElems()
     {
-        return type == tList1 || type == tList2 ? smallList : bigList.elems;
+        return internalType == tList1 || internalType == tList2 ? smallList : bigList.elems;
     }
 
     const Value * const * listElems() const
     {
-        return type == tList1 || type == tList2 ? smallList : bigList.elems;
+        return internalType == tList1 || internalType == tList2 ? smallList : bigList.elems;
     }
 
     size_t listSize() const
     {
-        return type == tList1 ? 1 : type == tList2 ? 2 : bigList.size;
+        return internalType == tList1 ? 1 : internalType == tList2 ? 2 : bigList.size;
     }
 
     /* Check whether forcing this value requires a trivial amount of
@@ -172,90 +396,58 @@ struct Value
        non-trivial. */
     bool isTrivial() const;
 
-    std::vector<std::pair<Path, std::string>> getContext();
+    std::vector<std::pair<Path, std::string>> getContext() const;
+
+    /*
+     * Set the associated cache view for this value.
+     * This cache will be used to speed-up some operations like accessing
+     * attribute sets elements.
+     */
+    void setCache(ValueCache);
+
+    /*
+     * Get the cache associated with this value, if any.
+     */
+    ValueCache getCache() const;
 };
 
 
-/* After overwriting an app node, be sure to clear pointers in the
-   Value to ensure that the target isn't kept alive unnecessarily. */
-static inline void clearValue(Value & v)
-{
-    v.app.left = v.app.right = 0;
-}
-
 
+// TODO: Remove these static functions, replace call sites with v.mk* instead
 static inline void mkInt(Value & v, NixInt n)
 {
-    clearValue(v);
-    v.type = tInt;
-    v.integer = n;
+    v.mkInt(n);
 }
 
-
 static inline void mkFloat(Value & v, NixFloat n)
 {
-    clearValue(v);
-    v.type = tFloat;
-    v.fpoint = n;
+    v.mkFloat(n);
 }
 
-
 static inline void mkBool(Value & v, bool b)
 {
-    clearValue(v);
-    v.type = tBool;
-    v.boolean = b;
+    v.mkBool(b);
 }
 
-
 static inline void mkNull(Value & v)
 {
-    clearValue(v);
-    v.type = tNull;
+    v.mkNull();
 }
 
-
 static inline void mkApp(Value & v, Value & left, Value & right)
 {
-    v.type = tApp;
-    v.app.left = &left;
-    v.app.right = &right;
+    v.mkApp(&left, &right);
 }
 
-
-static inline void mkPrimOpApp(Value & v, Value & left, Value & right)
-{
-    v.type = tPrimOpApp;
-    v.app.left = &left;
-    v.app.right = &right;
-}
-
-
-static inline void mkStringNoCopy(Value & v, const char * s)
-{
-    v.type = tString;
-    v.string.s = s;
-    v.string.context = 0;
-}
-
-
 static inline void mkString(Value & v, const Symbol & s)
 {
-    mkStringNoCopy(v, ((const string &) s).c_str());
+    v.mkString(((const string &) s).c_str());
 }
 
 
 void mkString(Value & v, const char * s);
 
 
-static inline void mkPathNoCopy(Value & v, const char * s)
-{
-    clearValue(v);
-    v.type = tPath;
-    v.path = s;
-}
-
-
 void mkPath(Value & v, const char * s);
 
 

src/libfetchers/attrs.cc

@@ -11,11 +11,11 @@ Attrs jsonToAttrs(const nlohmann::json & json)
 
     for (auto & i : json.items()) {
         if (i.value().is_number())
-            attrs.emplace(i.key(), i.value().get<int64_t>());
+            attrs.emplace(i.key(), i.value().get<uint64_t>());
         else if (i.value().is_string())
             attrs.emplace(i.key(), i.value().get<std::string>());
         else if (i.value().is_boolean())
-            attrs.emplace(i.key(), i.value().get<bool>());
+            attrs.emplace(i.key(), Explicit<bool> { i.value().get<bool>() });
         else
             throw Error("unsupported input attribute type in lock file");
     }
@@ -23,7 +23,7 @@ Attrs jsonToAttrs(const nlohmann::json & json)
     return attrs;
 }
 
-nlohmann::json attrsToJson(const Attrs & attrs)
+nlohmann::json attrsToJSON(const Attrs & attrs)
 {
     nlohmann::json json;
     for (auto & attr : attrs) {
@@ -44,7 +44,7 @@ std::optional<std::string> maybeGetStrAttr(const Attrs & attrs, const std::strin
     if (i == attrs.end()) return {};
     if (auto v = std::get_if<std::string>(&i->second))
         return *v;
-    throw Error("input attribute '%s' is not a string %s", name, attrsToJson(attrs).dump());
+    throw Error("input attribute '%s' is not a string %s", name, attrsToJSON(attrs).dump());
 }
 
 std::string getStrAttr(const Attrs & attrs, const std::string & name)

src/libfetchers/attrs.hh

@@ -8,24 +8,12 @@
 
 namespace nix::fetchers {
 
-/* Wrap bools to prevent string literals (i.e. 'char *') from being
-   cast to a bool in Attr. */
-template<typename T>
-struct Explicit {
-    T t;
-
-    bool operator ==(const Explicit<T> & other) const
-    {
-        return t == other.t;
-    }
-};
-
 typedef std::variant<std::string, uint64_t, Explicit<bool>> Attr;
 typedef std::map<std::string, Attr> Attrs;
 
 Attrs jsonToAttrs(const nlohmann::json & json);
 
-nlohmann::json attrsToJson(const Attrs & attrs);
+nlohmann::json attrsToJSON(const Attrs & attrs);
 
 std::optional<std::string> maybeGetStrAttr(const Attrs & attrs, const std::string & name);
 

src/libfetchers/cache.cc

@@ -55,8 +55,8 @@ struct CacheImpl : Cache
         bool immutable) override
     {
         _state.lock()->add.use()
-            (attrsToJson(inAttrs).dump())
-            (attrsToJson(infoAttrs).dump())
+            (attrsToJSON(inAttrs).dump())
+            (attrsToJSON(infoAttrs).dump())
             (store->printStorePath(storePath))
             (immutable)
             (time(0)).exec();
@@ -70,7 +70,7 @@ struct CacheImpl : Cache
             if (!res->expired)
                 return std::make_pair(std::move(res->infoAttrs), std::move(res->storePath));
             debug("ignoring expired cache entry '%s'",
-                attrsToJson(inAttrs).dump());
+                attrsToJSON(inAttrs).dump());
         }
         return {};
     }
@@ -81,15 +81,15 @@ struct CacheImpl : Cache
     {
         auto state(_state.lock());
 
-        auto inAttrsJson = attrsToJson(inAttrs).dump();
+        auto inAttrsJSON = attrsToJSON(inAttrs).dump();
 
-        auto stmt(state->lookup.use()(inAttrsJson));
+        auto stmt(state->lookup.use()(inAttrsJSON));
         if (!stmt.next()) {
-            debug("did not find cache entry for '%s'", inAttrsJson);
+            debug("did not find cache entry for '%s'", inAttrsJSON);
             return {};
         }
 
-        auto infoJson = stmt.getStr(0);
+        auto infoJSON = stmt.getStr(0);
         auto storePath = store->parseStorePath(stmt.getStr(1));
         auto immutable = stmt.getInt(2) != 0;
         auto timestamp = stmt.getInt(3);
@@ -97,16 +97,16 @@ struct CacheImpl : Cache
         store->addTempRoot(storePath);
         if (!store->isValidPath(storePath)) {
             // FIXME: we could try to substitute 'storePath'.
-            debug("ignoring disappeared cache entry '%s'", inAttrsJson);
+            debug("ignoring disappeared cache entry '%s'", inAttrsJSON);
             return {};
         }
 
         debug("using cache entry '%s' -> '%s', '%s'",
-            inAttrsJson, infoJson, store->printStorePath(storePath));
+            inAttrsJSON, infoJSON, store->printStorePath(storePath));
 
         return Result {
             .expired = !immutable && (settings.tarballTtl.get() == 0 || timestamp + settings.tarballTtl < time(0)),
-            .infoAttrs = jsonToAttrs(nlohmann::json::parse(infoJson)),
+            .infoAttrs = jsonToAttrs(nlohmann::json::parse(infoJSON)),
             .storePath = std::move(storePath)
         };
     }