set up builder pipe

flake.lock: Update

.github/STALE-BOT.md

@@ -0,0 +1,35 @@
+# Stale bot information
+
+- Thanks for your contribution!
+- To remove the stale label, just leave a new comment.
+- _How to find the right people to ping?_ → [`git blame`](https://git-scm.com/docs/git-blame) to the rescue! (or GitHub's history and blame buttons.)
+- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on [Matrix - #nix:nixos.org](https://matrix.to/#/#nix:nixos.org).
+
+## Suggestions for PRs
+
+1. GitHub sometimes doesn't notify people who commented / reviewed a PR previously, when you (force) push commits. If you have addressed the reviews you can [officially ask for a review](https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from those who commented to you or anyone else.
+2. If it is unfinished but you plan to finish it, please mark it as a draft.
+3. If you don't expect to work on it any time soon, closing it with a short comment may encourage someone else to pick up your work.
+4. To get things rolling again, rebase the PR against the target branch and address valid comments.
+5. If you need a review to move forward, ask in [the Discourse thread for PRs that need help](https://discourse.nixos.org/t/prs-in-distress/3604).
+6. If all you need is a merge, check the git history to find and [request reviews](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from people who usually merge related contributions.
+
+## Suggestions for issues
+
+1. If it is resolved (either for you personally, or in general), please consider closing it.
+2. If this might still be an issue, but you are not interested in promoting its resolution, please consider closing it while encouraging others to take over and reopen an issue if they care enough.
+3. If you still have interest in resolving it, try to ping somebody who you believe might have an interest in the topic. Consider discussing the problem in [our Discourse Forum](https://discourse.nixos.org/).
+4. As with all open source projects, your best option is to submit a Pull Request that addresses this issue. We :heart: this attitude!
+
+**Memorandum on closing issues**
+
+Don't be afraid to close an issue that holds valuable information. Closed issues stay in the system for people to search, read, cross-reference, or even reopen--nothing is lost! Closing obsolete issues is an important way to help maintainers focus their time and effort.
+
+## Useful GitHub search queries
+
+- [Open PRs with any stale-bot interaction](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+)
+- [Open PRs with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22)
+- [Open PRs with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)
+- [Open Issues with any stale-bot interaction](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+)
+- [Open Issues with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22+)
+- [Open Issues with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)

.github/stale.yml

@@ -0,0 +1,10 @@
+# Configuration for probot-stale - https://github.com/probot/stale
+daysUntilStale: 180
+daysUntilClose: 365
+exemptLabels:
+  - "critical"
+staleLabel: "stale"
+markComment: |
+  I marked this as stale due to inactivity. → [More info](https://github.com/NixOS/nix/blob/master/.github/STALE-BOT.md)
+closeComment: |
+  I closed this issue due to inactivity. → [More info](https://github.com/NixOS/nix/blob/master/.github/STALE-BOT.md)

.github/workflows/test.yml

@@ -8,10 +8,61 @@ jobs:
       matrix:
         os: [ubuntu-latest, macos-latest]
     runs-on: ${{ matrix.os }}
+
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v2.3.4
       with:
         fetch-depth: 0
-    - uses: cachix/install-nix-action@v11
-    #- run: nix flake check
-    - run: nix-build -A checks.$(if [[ `uname` = Linux ]]; then echo x86_64-linux; else echo x86_64-darwin; fi)
+    - uses: cachix/install-nix-action@v13
+    - run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
+    - uses: cachix/cachix-action@v10
+      with:
+        name: '${{ env.CACHIX_NAME }}'
+        signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
+        authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
+    - run: nix-build -A checks.$(nix-instantiate --eval -E '(builtins.currentSystem)')
+  check_cachix:
+    name: Cachix secret present for installer tests
+    runs-on: ubuntu-latest
+    outputs:
+      secret: ${{ steps.secret.outputs.secret }}
+    steps:
+      - name: Check for Cachix secret
+        id: secret
+        env:
+          _CACHIX_SECRETS: ${{ secrets.CACHIX_SIGNING_KEY }}${{ secrets.CACHIX_AUTH_TOKEN }}
+        run: echo "::set-output name=secret::${{ env._CACHIX_SECRETS != '' }}"
+  installer:
+    needs: [tests, check_cachix]
+    if: github.event_name == 'push' && needs.check_cachix.outputs.secret == 'true'
+    runs-on: ubuntu-latest
+    outputs:
+      installerURL: ${{ steps.prepare-installer.outputs.installerURL }}
+    steps:
+    - uses: actions/checkout@v2.3.4
+      with:
+        fetch-depth: 0
+    - run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
+    - uses: cachix/install-nix-action@v13
+    - uses: cachix/cachix-action@v10
+      with:
+        name: '${{ env.CACHIX_NAME }}'
+        signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
+        authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
+    - id: prepare-installer
+      run: scripts/prepare-installer-for-github-actions
+  installer_test:
+    needs: [installer, check_cachix]
+    if: github.event_name == 'push' && needs.check_cachix.outputs.secret == 'true'
+    strategy:
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+    - uses: actions/checkout@v2.3.4
+    - run: echo CACHIX_NAME="$(echo $GITHUB_REPOSITORY-install-tests | tr "[A-Z]/" "[a-z]-")" >> $GITHUB_ENV
+    - uses: cachix/install-nix-action@v13
+      with:
+        install_url: '${{needs.installer.outputs.installerURL}}'
+        install_options: "--tarball-url-prefix https://${{ env.CACHIX_NAME }}.cachix.org/serve"
+    - run: nix-instantiate -E 'builtins.currentTime' --eval

.gitignore

@@ -18,13 +18,13 @@ perl/Makefile.config
 /doc/manual/nix.json
 /doc/manual/conf-file.json
 /doc/manual/builtins.json
-/doc/manual/src/command-ref/nix.md
+/doc/manual/src/SUMMARY.md
+/doc/manual/src/command-ref/new-cli
 /doc/manual/src/command-ref/conf-file.md
 /doc/manual/src/expressions/builtins.md
 
 # /scripts/
 /scripts/nix-profile.sh
-/scripts/nix-copy-closure
 /scripts/nix-reduce-build
 /scripts/nix-http-export.cgi
 /scripts/nix-profile-daemon.sh
@@ -82,6 +82,7 @@ perl/Makefile.config
 /tests/shell
 /tests/shell.drv
 /tests/config.nix
+/tests/ca/config.nix
 
 # /tests/lang/
 /tests/lang/*.out

.version

@@ -1 +1 @@
-3.0
+2.4

Makefile

@@ -7,11 +7,12 @@ makefiles = \
   src/libfetchers/local.mk \
   src/libmain/local.mk \
   src/libexpr/local.mk \
+  src/libcmd/local.mk \
   src/nix/local.mk \
   src/resolve-system-dependencies/local.mk \
   scripts/local.mk \
-  corepkgs/local.mk \
   misc/bash/local.mk \
+  misc/zsh/local.mk \
   misc/systemd/local.mk \
   misc/launchd/local.mk \
   misc/upstart/local.mk \

Makefile.config.in

@@ -9,14 +9,14 @@ CXXFLAGS = @CXXFLAGS@
 EDITLINE_LIBS = @EDITLINE_LIBS@
 ENABLE_S3 = @ENABLE_S3@
 GTEST_LIBS = @GTEST_LIBS@
+HAVE_LIBCPUID = @HAVE_LIBCPUID@
 HAVE_SECCOMP = @HAVE_SECCOMP@
-HAVE_SODIUM = @HAVE_SODIUM@
 LDFLAGS = @LDFLAGS@
 LIBARCHIVE_LIBS = @LIBARCHIVE_LIBS@
 LIBBROTLI_LIBS = @LIBBROTLI_LIBS@
 LIBCURL_LIBS = @LIBCURL_LIBS@
-LIBLZMA_LIBS = @LIBLZMA_LIBS@
 OPENSSL_LIBS = @OPENSSL_LIBS@
+LIBSECCOMP_LIBS = @LIBSECCOMP_LIBS@
 PACKAGE_NAME = @PACKAGE_NAME@
 PACKAGE_VERSION = @PACKAGE_VERSION@
 SHELL = @bash@

README.md

@@ -20,7 +20,7 @@ Information on additional installation methods is available on the [Nix download
 
 ## Building And Developing
 
-See our [Hacking guide](https://hydra.nixos.org/job/nix/master/build.x86_64-linux/latest/download-by-type/doc/manual/hacking.html) in our manual for instruction on how to
+See our [Hacking guide](https://hydra.nixos.org/job/nix/master/build.x86_64-linux/latest/download-by-type/doc/manual/contributing/hacking.html) in our manual for instruction on how to
 build nix from source with nix-build or how to get a development environment.
 
 ## Additional Resources
@@ -28,7 +28,8 @@ build nix from source with nix-build or how to get a development environment.
 - [Nix manual](https://nixos.org/nix/manual)
 - [Nix jobsets on hydra.nixos.org](https://hydra.nixos.org/project/nix)
 - [NixOS Discourse](https://discourse.nixos.org/)
-- [IRC - #nixos on freenode.net](irc://irc.freenode.net/#nixos)
+- [Matrix - #nix:nixos.org](https://matrix.to/#/#nix:nixos.org)
+- [IRC - #nixos on libera.chat](irc://irc.libera.chat/#nixos)
 
 ## License
 

config/config.guess

@@ -1,8 +1,8 @@
 #! /bin/sh
 # Attempt to guess a canonical system name.
-#   Copyright 1992-2018 Free Software Foundation, Inc.
+#   Copyright 1992-2021 Free Software Foundation, Inc.
 
-timestamp='2018-08-02'
+timestamp='2021-01-25'
 
 # This file is free software; you can redistribute it and/or modify it
 # under the terms of the GNU General Public License as published by
@@ -27,12 +27,12 @@ timestamp='2018-08-02'
 # Originally written by Per Bothner; maintained since 2000 by Ben Elliston.
 #
 # You can get the latest version of this script from:
-# https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess
+# https://git.savannah.gnu.org/cgit/config.git/plain/config.guess
 #
 # Please send patches to <config-patches@gnu.org>.
 
 
-me=`echo "$0" | sed -e 's,.*/,,'`
+me=$(echo "$0" | sed -e 's,.*/,,')
 
 usage="\
 Usage: $0 [OPTION]
@@ -50,7 +50,7 @@ version="\
 GNU config.guess ($timestamp)
 
 Originally written by Per Bothner.
-Copyright 1992-2018 Free Software Foundation, Inc.
+Copyright 1992-2021 Free Software Foundation, Inc.
 
 This is free software; see the source for copying conditions.  There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
@@ -96,13 +96,14 @@ fi
 
 tmp=
 # shellcheck disable=SC2172
-trap 'test -z "$tmp" || rm -fr "$tmp"' 1 2 13 15
-trap 'exitcode=$?; test -z "$tmp" || rm -fr "$tmp"; exit $exitcode' 0
+trap 'test -z "$tmp" || rm -fr "$tmp"' 0 1 2 13 15
 
 set_cc_for_build() {
+    # prevent multiple calls if $tmp is already set
+    test "$tmp" && return 0
     : "${TMPDIR=/tmp}"
     # shellcheck disable=SC2039
-    { tmp=`(umask 077 && mktemp -d "$TMPDIR/cgXXXXXX") 2>/dev/null` && test -n "$tmp" && test -d "$tmp" ; } ||
+    { tmp=$( (umask 077 && mktemp -d "$TMPDIR/cgXXXXXX") 2>/dev/null) && test -n "$tmp" && test -d "$tmp" ; } ||
 	{ test -n "$RANDOM" && tmp=$TMPDIR/cg$$-$RANDOM && (umask 077 && mkdir "$tmp" 2>/dev/null) ; } ||
 	{ tmp=$TMPDIR/cg-$$ && (umask 077 && mkdir "$tmp" 2>/dev/null) && echo "Warning: creating insecure temp directory" >&2 ; } ||
 	{ echo "$me: cannot create a temporary directory in $TMPDIR" >&2 ; exit 1 ; }
@@ -130,16 +131,14 @@ if test -f /.attbin/uname ; then
 	PATH=$PATH:/.attbin ; export PATH
 fi
 
-UNAME_MACHINE=`(uname -m) 2>/dev/null` || UNAME_MACHINE=unknown
-UNAME_RELEASE=`(uname -r) 2>/dev/null` || UNAME_RELEASE=unknown
-UNAME_SYSTEM=`(uname -s) 2>/dev/null`  || UNAME_SYSTEM=unknown
-UNAME_VERSION=`(uname -v) 2>/dev/null` || UNAME_VERSION=unknown
+UNAME_MACHINE=$( (uname -m) 2>/dev/null) || UNAME_MACHINE=unknown
+UNAME_RELEASE=$( (uname -r) 2>/dev/null) || UNAME_RELEASE=unknown
+UNAME_SYSTEM=$( (uname -s) 2>/dev/null) || UNAME_SYSTEM=unknown
+UNAME_VERSION=$( (uname -v) 2>/dev/null) || UNAME_VERSION=unknown
 
 case "$UNAME_SYSTEM" in
 Linux|GNU|GNU/*)
-	# If the system lacks a compiler, then just pick glibc.
-	# We could probably try harder.
-	LIBC=gnu
+	LIBC=unknown
 
 	set_cc_for_build
 	cat <<-EOF > "$dummy.c"
@@ -148,17 +147,29 @@ Linux|GNU|GNU/*)
 	LIBC=uclibc
 	#elif defined(__dietlibc__)
 	LIBC=dietlibc
-	#else
+	#elif defined(__GLIBC__)
 	LIBC=gnu
+	#else
+	#include <stdarg.h>
+	/* First heuristic to detect musl libc.  */
+	#ifdef __DEFINED_va_list
+	LIBC=musl
+	#endif
 	#endif
 	EOF
-	eval "`$CC_FOR_BUILD -E "$dummy.c" 2>/dev/null | grep '^LIBC' | sed 's, ,,g'`"
+	eval "$($CC_FOR_BUILD -E "$dummy.c" 2>/dev/null | grep '^LIBC' | sed 's, ,,g')"
 
-	# If ldd exists, use it to detect musl libc.
-	if command -v ldd >/dev/null && \
-		ldd --version 2>&1 | grep -q ^musl
-	then
-	    LIBC=musl
+	# Second heuristic to detect musl libc.
+	if [ "$LIBC" = unknown ] &&
+	   command -v ldd >/dev/null &&
+	   ldd --version 2>&1 | grep -q ^musl; then
+		LIBC=musl
+	fi
+
+	# If the system lacks a compiler, then just pick glibc.
+	# We could probably try harder.
+	if [ "$LIBC" = unknown ]; then
+		LIBC=gnu
 	fi
 	;;
 esac
@@ -177,20 +188,20 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	#
 	# Note: NetBSD doesn't particularly care about the vendor
 	# portion of the name.  We always set it to "unknown".
-	sysctl="sysctl -n hw.machine_arch"
-	UNAME_MACHINE_ARCH=`(uname -p 2>/dev/null || \
-	    "/sbin/$sysctl" 2>/dev/null || \
-	    "/usr/sbin/$sysctl" 2>/dev/null || \
-	    echo unknown)`
+	UNAME_MACHINE_ARCH=$( (uname -p 2>/dev/null || \
+	    /sbin/sysctl -n hw.machine_arch 2>/dev/null || \
+	    /usr/sbin/sysctl -n hw.machine_arch 2>/dev/null || \
+	    echo unknown))
 	case "$UNAME_MACHINE_ARCH" in
+	    aarch64eb) machine=aarch64_be-unknown ;;
 	    armeb) machine=armeb-unknown ;;
 	    arm*) machine=arm-unknown ;;
 	    sh3el) machine=shl-unknown ;;
 	    sh3eb) machine=sh-unknown ;;
 	    sh5el) machine=sh5le-unknown ;;
 	    earmv*)
-		arch=`echo "$UNAME_MACHINE_ARCH" | sed -e 's,^e\(armv[0-9]\).*$,\1,'`
-		endian=`echo "$UNAME_MACHINE_ARCH" | sed -ne 's,^.*\(eb\)$,\1,p'`
+		arch=$(echo "$UNAME_MACHINE_ARCH" | sed -e 's,^e\(armv[0-9]\).*$,\1,')
+		endian=$(echo "$UNAME_MACHINE_ARCH" | sed -ne 's,^.*\(eb\)$,\1,p')
 		machine="${arch}${endian}"-unknown
 		;;
 	    *) machine="$UNAME_MACHINE_ARCH"-unknown ;;
@@ -221,7 +232,7 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	case "$UNAME_MACHINE_ARCH" in
 	    earm*)
 		expr='s/^earmv[0-9]/-eabi/;s/eb$//'
-		abi=`echo "$UNAME_MACHINE_ARCH" | sed -e "$expr"`
+		abi=$(echo "$UNAME_MACHINE_ARCH" | sed -e "$expr")
 		;;
 	esac
 	# The OS release
@@ -234,7 +245,7 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 		release='-gnu'
 		;;
 	    *)
-		release=`echo "$UNAME_RELEASE" | sed -e 's/[-_].*//' | cut -d. -f1,2`
+		release=$(echo "$UNAME_RELEASE" | sed -e 's/[-_].*//' | cut -d. -f1,2)
 		;;
 	esac
 	# Since CPU_TYPE-MANUFACTURER-KERNEL-OPERATING_SYSTEM:
@@ -243,15 +254,15 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	echo "$machine-${os}${release}${abi-}"
 	exit ;;
     *:Bitrig:*:*)
-	UNAME_MACHINE_ARCH=`arch | sed 's/Bitrig.//'`
+	UNAME_MACHINE_ARCH=$(arch | sed 's/Bitrig.//')
 	echo "$UNAME_MACHINE_ARCH"-unknown-bitrig"$UNAME_RELEASE"
 	exit ;;
     *:OpenBSD:*:*)
-	UNAME_MACHINE_ARCH=`arch | sed 's/OpenBSD.//'`
+	UNAME_MACHINE_ARCH=$(arch | sed 's/OpenBSD.//')
 	echo "$UNAME_MACHINE_ARCH"-unknown-openbsd"$UNAME_RELEASE"
 	exit ;;
     *:LibertyBSD:*:*)
-	UNAME_MACHINE_ARCH=`arch | sed 's/^.*BSD\.//'`
+	UNAME_MACHINE_ARCH=$(arch | sed 's/^.*BSD\.//')
 	echo "$UNAME_MACHINE_ARCH"-unknown-libertybsd"$UNAME_RELEASE"
 	exit ;;
     *:MidnightBSD:*:*)
@@ -263,6 +274,9 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
     *:SolidBSD:*:*)
 	echo "$UNAME_MACHINE"-unknown-solidbsd"$UNAME_RELEASE"
 	exit ;;
+    *:OS108:*:*)
+	echo "$UNAME_MACHINE"-unknown-os108_"$UNAME_RELEASE"
+	exit ;;
     macppc:MirBSD:*:*)
 	echo powerpc-unknown-mirbsd"$UNAME_RELEASE"
 	exit ;;
@@ -272,26 +286,29 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
     *:Sortix:*:*)
 	echo "$UNAME_MACHINE"-unknown-sortix
 	exit ;;
+    *:Twizzler:*:*)
+	echo "$UNAME_MACHINE"-unknown-twizzler
+	exit ;;
     *:Redox:*:*)
 	echo "$UNAME_MACHINE"-unknown-redox
 	exit ;;
     mips:OSF1:*.*)
-        echo mips-dec-osf1
-        exit ;;
+	echo mips-dec-osf1
+	exit ;;
     alpha:OSF1:*:*)
 	case $UNAME_RELEASE in
 	*4.0)
-		UNAME_RELEASE=`/usr/sbin/sizer -v | awk '{print $3}'`
+		UNAME_RELEASE=$(/usr/sbin/sizer -v | awk '{print $3}')
 		;;
 	*5.*)
-		UNAME_RELEASE=`/usr/sbin/sizer -v | awk '{print $4}'`
+		UNAME_RELEASE=$(/usr/sbin/sizer -v | awk '{print $4}')
 		;;
 	esac
 	# According to Compaq, /usr/sbin/psrinfo has been available on
 	# OSF/1 and Tru64 systems produced since 1995.  I hope that
 	# covers most systems running today.  This code pipes the CPU
 	# types through head -n 1, so we only detect the type of CPU 0.
-	ALPHA_CPU_TYPE=`/usr/sbin/psrinfo -v | sed -n -e 's/^  The alpha \(.*\) processor.*$/\1/p' | head -n 1`
+	ALPHA_CPU_TYPE=$(/usr/sbin/psrinfo -v | sed -n -e 's/^  The alpha \(.*\) processor.*$/\1/p' | head -n 1)
 	case "$ALPHA_CPU_TYPE" in
 	    "EV4 (21064)")
 		UNAME_MACHINE=alpha ;;
@@ -329,7 +346,7 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	# A Tn.n version is a released field test version.
 	# A Xn.n version is an unreleased experimental baselevel.
 	# 1.2 uses "1.2" for uname -r.
-	echo "$UNAME_MACHINE"-dec-osf"`echo "$UNAME_RELEASE" | sed -e 's/^[PVTX]//' | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz`"
+	echo "$UNAME_MACHINE"-dec-osf"$(echo "$UNAME_RELEASE" | sed -e 's/^[PVTX]//' | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz)"
 	# Reset EXIT trap before exiting to avoid spurious non-zero exit code.
 	exitcode=$?
 	trap '' 0
@@ -363,7 +380,7 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	exit ;;
     Pyramid*:OSx*:*:* | MIS*:OSx*:*:* | MIS*:SMP_DC-OSx*:*:*)
 	# akee@wpdis03.wpafb.af.mil (Earle F. Ake) contributed MIS and NILE.
-	if test "`(/bin/universe) 2>/dev/null`" = att ; then
+	if test "$( (/bin/universe) 2>/dev/null)" = att ; then
 		echo pyramid-pyramid-sysv3
 	else
 		echo pyramid-pyramid-bsd
@@ -376,54 +393,59 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	echo sparc-icl-nx6
 	exit ;;
     DRS?6000:UNIX_SV:4.2*:7* | DRS?6000:isis:4.2*:7*)
-	case `/usr/bin/uname -p` in
+	case $(/usr/bin/uname -p) in
 	    sparc) echo sparc-icl-nx7; exit ;;
 	esac ;;
     s390x:SunOS:*:*)
-	echo "$UNAME_MACHINE"-ibm-solaris2"`echo "$UNAME_RELEASE" | sed -e 's/[^.]*//'`"
+	echo "$UNAME_MACHINE"-ibm-solaris2"$(echo "$UNAME_RELEASE" | sed -e 's/[^.]*//')"
 	exit ;;
     sun4H:SunOS:5.*:*)
-	echo sparc-hal-solaris2"`echo "$UNAME_RELEASE"|sed -e 's/[^.]*//'`"
+	echo sparc-hal-solaris2"$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*//')"
 	exit ;;
     sun4*:SunOS:5.*:* | tadpole*:SunOS:5.*:*)
-	echo sparc-sun-solaris2"`echo "$UNAME_RELEASE" | sed -e 's/[^.]*//'`"
+	echo sparc-sun-solaris2"$(echo "$UNAME_RELEASE" | sed -e 's/[^.]*//')"
 	exit ;;
     i86pc:AuroraUX:5.*:* | i86xen:AuroraUX:5.*:*)
 	echo i386-pc-auroraux"$UNAME_RELEASE"
 	exit ;;
     i86pc:SunOS:5.*:* | i86xen:SunOS:5.*:*)
-	UNAME_REL="`echo "$UNAME_RELEASE" | sed -e 's/[^.]*//'`"
-	case `isainfo -b` in
-	    32)
-		echo i386-pc-solaris2"$UNAME_REL"
-		;;
-	    64)
-		echo x86_64-pc-solaris2"$UNAME_REL"
-		;;
-	esac
+	set_cc_for_build
+	SUN_ARCH=i386
+	# If there is a compiler, see if it is configured for 64-bit objects.
+	# Note that the Sun cc does not turn __LP64__ into 1 like gcc does.
+	# This test works for both compilers.
+	if test "$CC_FOR_BUILD" != no_compiler_found; then
+	    if (echo '#ifdef __amd64'; echo IS_64BIT_ARCH; echo '#endif') | \
+		(CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
+		grep IS_64BIT_ARCH >/dev/null
+	    then
+		SUN_ARCH=x86_64
+	    fi
+	fi
+	echo "$SUN_ARCH"-pc-solaris2"$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*//')"
 	exit ;;
     sun4*:SunOS:6*:*)
 	# According to config.sub, this is the proper way to canonicalize
 	# SunOS6.  Hard to guess exactly what SunOS6 will be like, but
 	# it's likely to be more like Solaris than SunOS4.
-	echo sparc-sun-solaris3"`echo "$UNAME_RELEASE"|sed -e 's/[^.]*//'`"
+	echo sparc-sun-solaris3"$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*//')"
 	exit ;;
     sun4*:SunOS:*:*)
-	case "`/usr/bin/arch -k`" in
+	case "$(/usr/bin/arch -k)" in
 	    Series*|S4*)
-		UNAME_RELEASE=`uname -v`
+		UNAME_RELEASE=$(uname -v)
 		;;
 	esac
 	# Japanese Language versions have a version number like `4.1.3-JL'.
-	echo sparc-sun-sunos"`echo "$UNAME_RELEASE"|sed -e 's/-/_/'`"
+	echo sparc-sun-sunos"$(echo "$UNAME_RELEASE"|sed -e 's/-/_/')"
 	exit ;;
     sun3*:SunOS:*:*)
 	echo m68k-sun-sunos"$UNAME_RELEASE"
 	exit ;;
     sun*:*:4.2BSD:*)
-	UNAME_RELEASE=`(sed 1q /etc/motd | awk '{print substr($5,1,3)}') 2>/dev/null`
+	UNAME_RELEASE=$( (sed 1q /etc/motd | awk '{print substr($5,1,3)}') 2>/dev/null)
 	test "x$UNAME_RELEASE" = x && UNAME_RELEASE=3
-	case "`/bin/arch`" in
+	case "$(/bin/arch)" in
 	    sun3)
 		echo m68k-sun-sunos"$UNAME_RELEASE"
 		;;
@@ -503,8 +525,8 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
 	}
 EOF
 	$CC_FOR_BUILD -o "$dummy" "$dummy.c" &&
-	  dummyarg=`echo "$UNAME_RELEASE" | sed -n 's/\([0-9]*\).*/\1/p'` &&
-	  SYSTEM_NAME=`"$dummy" "$dummyarg"` &&
+	  dummyarg=$(echo "$UNAME_RELEASE" | sed -n 's/\([0-9]*\).*/\1/p') &&
+	  SYSTEM_NAME=$("$dummy" "$dummyarg") &&
 	    { echo "$SYSTEM_NAME"; exit; }
 	echo mips-mips-riscos"$UNAME_RELEASE"
 	exit ;;
@@ -531,11 +553,11 @@ EOF
 	exit ;;
     AViiON:dgux:*:*)
 	# DG/UX returns AViiON for all architectures
-	UNAME_PROCESSOR=`/usr/bin/uname -p`
-	if [ "$UNAME_PROCESSOR" = mc88100 ] || [ "$UNAME_PROCESSOR" = mc88110 ]
+	UNAME_PROCESSOR=$(/usr/bin/uname -p)
+	if test "$UNAME_PROCESSOR" = mc88100 || test "$UNAME_PROCESSOR" = mc88110
 	then
-	    if [ "$TARGET_BINARY_INTERFACE"x = m88kdguxelfx ] || \
-	       [ "$TARGET_BINARY_INTERFACE"x = x ]
+	    if test "$TARGET_BINARY_INTERFACE"x = m88kdguxelfx || \
+	       test "$TARGET_BINARY_INTERFACE"x = x
 	    then
 		echo m88k-dg-dgux"$UNAME_RELEASE"
 	    else
@@ -559,17 +581,17 @@ EOF
 	echo m68k-tektronix-bsd
 	exit ;;
     *:IRIX*:*:*)
-	echo mips-sgi-irix"`echo "$UNAME_RELEASE"|sed -e 's/-/_/g'`"
+	echo mips-sgi-irix"$(echo "$UNAME_RELEASE"|sed -e 's/-/_/g')"
 	exit ;;
     ????????:AIX?:[12].1:2)   # AIX 2.2.1 or AIX 2.1.1 is RT/PC AIX.
 	echo romp-ibm-aix     # uname -m gives an 8 hex-code CPU id
-	exit ;;               # Note that: echo "'`uname -s`'" gives 'AIX '
+	exit ;;               # Note that: echo "'$(uname -s)'" gives 'AIX '
     i*86:AIX:*:*)
 	echo i386-ibm-aix
 	exit ;;
     ia64:AIX:*:*)
-	if [ -x /usr/bin/oslevel ] ; then
-		IBM_REV=`/usr/bin/oslevel`
+	if test -x /usr/bin/oslevel ; then
+		IBM_REV=$(/usr/bin/oslevel)
 	else
 		IBM_REV="$UNAME_VERSION.$UNAME_RELEASE"
 	fi
@@ -589,7 +611,7 @@ EOF
 			exit(0);
 			}
 EOF
-		if $CC_FOR_BUILD -o "$dummy" "$dummy.c" && SYSTEM_NAME=`"$dummy"`
+		if $CC_FOR_BUILD -o "$dummy" "$dummy.c" && SYSTEM_NAME=$("$dummy")
 		then
 			echo "$SYSTEM_NAME"
 		else
@@ -602,15 +624,15 @@ EOF
 	fi
 	exit ;;
     *:AIX:*:[4567])
-	IBM_CPU_ID=`/usr/sbin/lsdev -C -c processor -S available | sed 1q | awk '{ print $1 }'`
+	IBM_CPU_ID=$(/usr/sbin/lsdev -C -c processor -S available | sed 1q | awk '{ print $1 }')
 	if /usr/sbin/lsattr -El "$IBM_CPU_ID" | grep ' POWER' >/dev/null 2>&1; then
 		IBM_ARCH=rs6000
 	else
 		IBM_ARCH=powerpc
 	fi
-	if [ -x /usr/bin/lslpp ] ; then
-		IBM_REV=`/usr/bin/lslpp -Lqc bos.rte.libc |
-			   awk -F: '{ print $3 }' | sed s/[0-9]*$/0/`
+	if test -x /usr/bin/lslpp ; then
+		IBM_REV=$(/usr/bin/lslpp -Lqc bos.rte.libc |
+			   awk -F: '{ print $3 }' | sed s/[0-9]*$/0/)
 	else
 		IBM_REV="$UNAME_VERSION.$UNAME_RELEASE"
 	fi
@@ -638,14 +660,14 @@ EOF
 	echo m68k-hp-bsd4.4
 	exit ;;
     9000/[34678]??:HP-UX:*:*)
-	HPUX_REV=`echo "$UNAME_RELEASE"|sed -e 's/[^.]*.[0B]*//'`
+	HPUX_REV=$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*.[0B]*//')
 	case "$UNAME_MACHINE" in
 	    9000/31?)            HP_ARCH=m68000 ;;
 	    9000/[34]??)         HP_ARCH=m68k ;;
 	    9000/[678][0-9][0-9])
-		if [ -x /usr/bin/getconf ]; then
-		    sc_cpu_version=`/usr/bin/getconf SC_CPU_VERSION 2>/dev/null`
-		    sc_kernel_bits=`/usr/bin/getconf SC_KERNEL_BITS 2>/dev/null`
+		if test -x /usr/bin/getconf; then
+		    sc_cpu_version=$(/usr/bin/getconf SC_CPU_VERSION 2>/dev/null)
+		    sc_kernel_bits=$(/usr/bin/getconf SC_KERNEL_BITS 2>/dev/null)
 		    case "$sc_cpu_version" in
 		      523) HP_ARCH=hppa1.0 ;; # CPU_PA_RISC1_0
 		      528) HP_ARCH=hppa1.1 ;; # CPU_PA_RISC1_1
@@ -657,7 +679,7 @@ EOF
 			esac ;;
 		    esac
 		fi
-		if [ "$HP_ARCH" = "" ]; then
+		if test "$HP_ARCH" = ""; then
 		    set_cc_for_build
 		    sed 's/^		//' << EOF > "$dummy.c"
 
@@ -692,11 +714,11 @@ EOF
 		    exit (0);
 		}
 EOF
-		    (CCOPTS="" $CC_FOR_BUILD -o "$dummy" "$dummy.c" 2>/dev/null) && HP_ARCH=`"$dummy"`
+		    (CCOPTS="" $CC_FOR_BUILD -o "$dummy" "$dummy.c" 2>/dev/null) && HP_ARCH=$("$dummy")
 		    test -z "$HP_ARCH" && HP_ARCH=hppa
 		fi ;;
 	esac
-	if [ "$HP_ARCH" = hppa2.0w ]
+	if test "$HP_ARCH" = hppa2.0w
 	then
 	    set_cc_for_build
 
@@ -720,7 +742,7 @@ EOF
 	echo "$HP_ARCH"-hp-hpux"$HPUX_REV"
 	exit ;;
     ia64:HP-UX:*:*)
-	HPUX_REV=`echo "$UNAME_RELEASE"|sed -e 's/[^.]*.[0B]*//'`
+	HPUX_REV=$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*.[0B]*//')
 	echo ia64-hp-hpux"$HPUX_REV"
 	exit ;;
     3050*:HI-UX:*:*)
@@ -750,7 +772,7 @@ EOF
 	  exit (0);
 	}
 EOF
-	$CC_FOR_BUILD -o "$dummy" "$dummy.c" && SYSTEM_NAME=`"$dummy"` &&
+	$CC_FOR_BUILD -o "$dummy" "$dummy.c" && SYSTEM_NAME=$("$dummy") &&
 		{ echo "$SYSTEM_NAME"; exit; }
 	echo unknown-hitachi-hiuxwe2
 	exit ;;
@@ -770,7 +792,7 @@ EOF
 	echo hppa1.0-hp-osf
 	exit ;;
     i*86:OSF1:*:*)
-	if [ -x /usr/sbin/sysversion ] ; then
+	if test -x /usr/sbin/sysversion ; then
 	    echo "$UNAME_MACHINE"-unknown-osf1mk
 	else
 	    echo "$UNAME_MACHINE"-unknown-osf1
@@ -819,14 +841,14 @@ EOF
 	echo craynv-cray-unicosmp"$UNAME_RELEASE" | sed -e 's/\.[^.]*$/.X/'
 	exit ;;
     F30[01]:UNIX_System_V:*:* | F700:UNIX_System_V:*:*)
-	FUJITSU_PROC=`uname -m | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz`
-	FUJITSU_SYS=`uname -p | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/\///'`
-	FUJITSU_REL=`echo "$UNAME_RELEASE" | sed -e 's/ /_/'`
+	FUJITSU_PROC=$(uname -m | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz)
+	FUJITSU_SYS=$(uname -p | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/\///')
+	FUJITSU_REL=$(echo "$UNAME_RELEASE" | sed -e 's/ /_/')
 	echo "${FUJITSU_PROC}-fujitsu-${FUJITSU_SYS}${FUJITSU_REL}"
 	exit ;;
     5000:UNIX_System_V:4.*:*)
-	FUJITSU_SYS=`uname -p | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/\///'`
-	FUJITSU_REL=`echo "$UNAME_RELEASE" | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/ /_/'`
+	FUJITSU_SYS=$(uname -p | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/\///')
+	FUJITSU_REL=$(echo "$UNAME_RELEASE" | tr ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz | sed -e 's/ /_/')
 	echo "sparc-fujitsu-${FUJITSU_SYS}${FUJITSU_REL}"
 	exit ;;
     i*86:BSD/386:*:* | i*86:BSD/OS:*:* | *:Ascend\ Embedded/OS:*:*)
@@ -838,26 +860,26 @@ EOF
     *:BSD/OS:*:*)
 	echo "$UNAME_MACHINE"-unknown-bsdi"$UNAME_RELEASE"
 	exit ;;
-    arm*:FreeBSD:*:*)
-	UNAME_PROCESSOR=`uname -p`
+    arm:FreeBSD:*:*)
+	UNAME_PROCESSOR=$(uname -p)
 	set_cc_for_build
 	if echo __ARM_PCS_VFP | $CC_FOR_BUILD -E - 2>/dev/null \
 	    | grep -q __ARM_PCS_VFP
 	then
-	    echo "${UNAME_PROCESSOR}"-unknown-freebsd"`echo ${UNAME_RELEASE}|sed -e 's/[-(].*//'`"-gnueabi
+	    echo "${UNAME_PROCESSOR}"-unknown-freebsd"$(echo ${UNAME_RELEASE}|sed -e 's/[-(].*//')"-gnueabi
 	else
-	    echo "${UNAME_PROCESSOR}"-unknown-freebsd"`echo ${UNAME_RELEASE}|sed -e 's/[-(].*//'`"-gnueabihf
+	    echo "${UNAME_PROCESSOR}"-unknown-freebsd"$(echo ${UNAME_RELEASE}|sed -e 's/[-(].*//')"-gnueabihf
 	fi
 	exit ;;
     *:FreeBSD:*:*)
-	UNAME_PROCESSOR=`/usr/bin/uname -p`
+	UNAME_PROCESSOR=$(/usr/bin/uname -p)
 	case "$UNAME_PROCESSOR" in
 	    amd64)
 		UNAME_PROCESSOR=x86_64 ;;
 	    i386)
 		UNAME_PROCESSOR=i586 ;;
 	esac
-	echo "$UNAME_PROCESSOR"-unknown-freebsd"`echo "$UNAME_RELEASE"|sed -e 's/[-(].*//'`"
+	echo "$UNAME_PROCESSOR"-unknown-freebsd"$(echo "$UNAME_RELEASE"|sed -e 's/[-(].*//')"
 	exit ;;
     i*:CYGWIN*:*)
 	echo "$UNAME_MACHINE"-pc-cygwin
@@ -890,18 +912,18 @@ EOF
 	echo "$UNAME_MACHINE"-pc-uwin
 	exit ;;
     amd64:CYGWIN*:*:* | x86_64:CYGWIN*:*:*)
-	echo x86_64-unknown-cygwin
+	echo x86_64-pc-cygwin
 	exit ;;
     prep*:SunOS:5.*:*)
-	echo powerpcle-unknown-solaris2"`echo "$UNAME_RELEASE"|sed -e 's/[^.]*//'`"
+	echo powerpcle-unknown-solaris2"$(echo "$UNAME_RELEASE"|sed -e 's/[^.]*//')"
 	exit ;;
     *:GNU:*:*)
 	# the GNU system
-	echo "`echo "$UNAME_MACHINE"|sed -e 's,[-/].*$,,'`-unknown-$LIBC`echo "$UNAME_RELEASE"|sed -e 's,/.*$,,'`"
+	echo "$(echo "$UNAME_MACHINE"|sed -e 's,[-/].*$,,')-unknown-$LIBC$(echo "$UNAME_RELEASE"|sed -e 's,/.*$,,')"
 	exit ;;
     *:GNU/*:*:*)
 	# other systems with GNU libc and userland
-	echo "$UNAME_MACHINE-unknown-`echo "$UNAME_SYSTEM" | sed 's,^[^/]*/,,' | tr "[:upper:]" "[:lower:]"``echo "$UNAME_RELEASE"|sed -e 's/[-(].*//'`-$LIBC"
+	echo "$UNAME_MACHINE-unknown-$(echo "$UNAME_SYSTEM" | sed 's,^[^/]*/,,' | tr "[:upper:]" "[:lower:]")$(echo "$UNAME_RELEASE"|sed -e 's/[-(].*//')-$LIBC"
 	exit ;;
     *:Minix:*:*)
 	echo "$UNAME_MACHINE"-unknown-minix
@@ -914,7 +936,7 @@ EOF
 	echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
 	exit ;;
     alpha:Linux:*:*)
-	case `sed -n '/^cpu model/s/^.*: \(.*\)/\1/p' < /proc/cpuinfo` in
+	case $(sed -n '/^cpu model/s/^.*: \(.*\)/\1/p' /proc/cpuinfo 2>/dev/null) in
 	  EV5)   UNAME_MACHINE=alphaev5 ;;
 	  EV56)  UNAME_MACHINE=alphaev56 ;;
 	  PCA56) UNAME_MACHINE=alphapca56 ;;
@@ -973,6 +995,9 @@ EOF
     k1om:Linux:*:*)
 	echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
 	exit ;;
+    loongarch32:Linux:*:* | loongarch64:Linux:*:* | loongarchx32:Linux:*:*)
+	echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
+	exit ;;
     m32r*:Linux:*:*)
 	echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
 	exit ;;
@@ -981,22 +1006,50 @@ EOF
 	exit ;;
     mips:Linux:*:* | mips64:Linux:*:*)
 	set_cc_for_build
+	IS_GLIBC=0
+	test x"${LIBC}" = xgnu && IS_GLIBC=1
 	sed 's/^	//' << EOF > "$dummy.c"
 	#undef CPU
-	#undef ${UNAME_MACHINE}
-	#undef ${UNAME_MACHINE}el
+	#undef mips
+	#undef mipsel
+	#undef mips64
+	#undef mips64el
+	#if ${IS_GLIBC} && defined(_ABI64)
+	LIBCABI=gnuabi64
+	#else
+	#if ${IS_GLIBC} && defined(_ABIN32)
+	LIBCABI=gnuabin32
+	#else
+	LIBCABI=${LIBC}
+	#endif
+	#endif
+
+	#if ${IS_GLIBC} && defined(__mips64) && defined(__mips_isa_rev) && __mips_isa_rev>=6
+	CPU=mipsisa64r6
+	#else
+	#if ${IS_GLIBC} && !defined(__mips64) && defined(__mips_isa_rev) && __mips_isa_rev>=6
+	CPU=mipsisa32r6
+	#else
+	#if defined(__mips64)
+	CPU=mips64
+	#else
+	CPU=mips
+	#endif
+	#endif
+	#endif
+
 	#if defined(__MIPSEL__) || defined(__MIPSEL) || defined(_MIPSEL) || defined(MIPSEL)
-	CPU=${UNAME_MACHINE}el
+	MIPS_ENDIAN=el
 	#else
 	#if defined(__MIPSEB__) || defined(__MIPSEB) || defined(_MIPSEB) || defined(MIPSEB)
-	CPU=${UNAME_MACHINE}
+	MIPS_ENDIAN=
 	#else
-	CPU=
+	MIPS_ENDIAN=
 	#endif
 	#endif
 EOF
-	eval "`$CC_FOR_BUILD -E "$dummy.c" 2>/dev/null | grep '^CPU'`"
-	test "x$CPU" != x && { echo "$CPU-unknown-linux-$LIBC"; exit; }
+	eval "$($CC_FOR_BUILD -E "$dummy.c" 2>/dev/null | grep '^CPU\|^MIPS_ENDIAN\|^LIBCABI')"
+	test "x$CPU" != x && { echo "$CPU${MIPS_ENDIAN}-unknown-linux-$LIBCABI"; exit; }
 	;;
     mips64el:Linux:*:*)
 	echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
@@ -1015,7 +1068,7 @@ EOF
 	exit ;;
     parisc:Linux:*:* | hppa:Linux:*:*)
 	# Look for CPU level
-	case `grep '^cpu[^a-z]*:' /proc/cpuinfo 2>/dev/null | cut -d' ' -f2` in
+	case $(grep '^cpu[^a-z]*:' /proc/cpuinfo 2>/dev/null | cut -d' ' -f2) in
 	  PA7*) echo hppa1.1-unknown-linux-"$LIBC" ;;
 	  PA8*) echo hppa2.0-unknown-linux-"$LIBC" ;;
 	  *)    echo hppa-unknown-linux-"$LIBC" ;;
@@ -1033,7 +1086,7 @@ EOF
     ppcle:Linux:*:*)
 	echo powerpcle-unknown-linux-"$LIBC"
 	exit ;;
-    riscv32:Linux:*:* | riscv64:Linux:*:*)
+    riscv32:Linux:*:* | riscv32be:Linux:*:* | riscv64:Linux:*:* | riscv64be:Linux:*:*)
 	echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
 	exit ;;
     s390:Linux:*:* | s390x:Linux:*:*)
@@ -1055,7 +1108,17 @@ EOF
 	echo "$UNAME_MACHINE"-dec-linux-"$LIBC"
 	exit ;;
     x86_64:Linux:*:*)
-	echo "$UNAME_MACHINE"-pc-linux-"$LIBC"
+	set_cc_for_build
+	LIBCABI=$LIBC
+	if test "$CC_FOR_BUILD" != no_compiler_found; then
+	    if (echo '#ifdef __ILP32__'; echo IS_X32; echo '#endif') | \
+		(CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
+		grep IS_X32 >/dev/null
+	    then
+		LIBCABI="$LIBC"x32
+	    fi
+	fi
+	echo "$UNAME_MACHINE"-pc-linux-"$LIBCABI"
 	exit ;;
     xtensa*:Linux:*:*)
 	echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
@@ -1095,7 +1158,7 @@ EOF
 	echo "$UNAME_MACHINE"-pc-msdosdjgpp
 	exit ;;
     i*86:*:4.*:*)
-	UNAME_REL=`echo "$UNAME_RELEASE" | sed 's/\/MP$//'`
+	UNAME_REL=$(echo "$UNAME_RELEASE" | sed 's/\/MP$//')
 	if grep Novell /usr/include/link.h >/dev/null 2>/dev/null; then
 		echo "$UNAME_MACHINE"-univel-sysv"$UNAME_REL"
 	else
@@ -1104,19 +1167,19 @@ EOF
 	exit ;;
     i*86:*:5:[678]*)
 	# UnixWare 7.x, OpenUNIX and OpenServer 6.
-	case `/bin/uname -X | grep "^Machine"` in
+	case $(/bin/uname -X | grep "^Machine") in
 	    *486*)	     UNAME_MACHINE=i486 ;;
 	    *Pentium)	     UNAME_MACHINE=i586 ;;
 	    *Pent*|*Celeron) UNAME_MACHINE=i686 ;;
 	esac
-	echo "$UNAME_MACHINE-unknown-sysv${UNAME_RELEASE}${UNAME_SYSTEM}{$UNAME_VERSION}"
+	echo "$UNAME_MACHINE-unknown-sysv${UNAME_RELEASE}${UNAME_SYSTEM}${UNAME_VERSION}"
 	exit ;;
     i*86:*:3.2:*)
 	if test -f /usr/options/cb.name; then
-		UNAME_REL=`sed -n 's/.*Version //p' </usr/options/cb.name`
+		UNAME_REL=$(sed -n 's/.*Version //p' </usr/options/cb.name)
 		echo "$UNAME_MACHINE"-pc-isc"$UNAME_REL"
 	elif /bin/uname -X 2>/dev/null >/dev/null ; then
-		UNAME_REL=`(/bin/uname -X|grep Release|sed -e 's/.*= //')`
+		UNAME_REL=$( (/bin/uname -X|grep Release|sed -e 's/.*= //'))
 		(/bin/uname -X|grep i80486 >/dev/null) && UNAME_MACHINE=i486
 		(/bin/uname -X|grep '^Machine.*Pentium' >/dev/null) \
 			&& UNAME_MACHINE=i586
@@ -1166,7 +1229,7 @@ EOF
     3[345]??:*:4.0:3.0 | 3[34]??A:*:4.0:3.0 | 3[34]??,*:*:4.0:3.0 | 3[34]??/*:*:4.0:3.0 | 4400:*:4.0:3.0 | 4850:*:4.0:3.0 | SKA40:*:4.0:3.0 | SDS2:*:4.0:3.0 | SHG2:*:4.0:3.0 | S7501*:*:4.0:3.0)
 	OS_REL=''
 	test -r /etc/.relid \
-	&& OS_REL=.`sed -n 's/[^ ]* [^ ]* \([0-9][0-9]\).*/\1/p' < /etc/.relid`
+	&& OS_REL=.$(sed -n 's/[^ ]* [^ ]* \([0-9][0-9]\).*/\1/p' < /etc/.relid)
 	/bin/uname -p 2>/dev/null | grep 86 >/dev/null \
 	  && { echo i486-ncr-sysv4.3"$OS_REL"; exit; }
 	/bin/uname -p 2>/dev/null | /bin/grep entium >/dev/null \
@@ -1177,7 +1240,7 @@ EOF
     NCR*:*:4.2:* | MPRAS*:*:4.2:*)
 	OS_REL='.3'
 	test -r /etc/.relid \
-	    && OS_REL=.`sed -n 's/[^ ]* [^ ]* \([0-9][0-9]\).*/\1/p' < /etc/.relid`
+	    && OS_REL=.$(sed -n 's/[^ ]* [^ ]* \([0-9][0-9]\).*/\1/p' < /etc/.relid)
 	/bin/uname -p 2>/dev/null | grep 86 >/dev/null \
 	    && { echo i486-ncr-sysv4.3"$OS_REL"; exit; }
 	/bin/uname -p 2>/dev/null | /bin/grep entium >/dev/null \
@@ -1210,7 +1273,7 @@ EOF
 	exit ;;
     *:SINIX-*:*:*)
 	if uname -p 2>/dev/null >/dev/null ; then
-		UNAME_MACHINE=`(uname -p) 2>/dev/null`
+		UNAME_MACHINE=$( (uname -p) 2>/dev/null)
 		echo "$UNAME_MACHINE"-sni-sysv4
 	else
 		echo ns32k-sni-sysv
@@ -1244,7 +1307,7 @@ EOF
 	echo mips-sony-newsos6
 	exit ;;
     R[34]000:*System_V*:*:* | R4000:UNIX_SYSV:*:* | R*000:UNIX_SV:*:*)
-	if [ -d /usr/nec ]; then
+	if test -d /usr/nec; then
 		echo mips-nec-sysv"$UNAME_RELEASE"
 	else
 		echo mips-unknown-sysv"$UNAME_RELEASE"
@@ -1292,44 +1355,48 @@ EOF
     *:Rhapsody:*:*)
 	echo "$UNAME_MACHINE"-apple-rhapsody"$UNAME_RELEASE"
 	exit ;;
+    arm64:Darwin:*:*)
+	echo aarch64-apple-darwin"$UNAME_RELEASE"
+	exit ;;
     *:Darwin:*:*)
-	UNAME_PROCESSOR=`uname -p` || UNAME_PROCESSOR=unknown
-	set_cc_for_build
-	if test "$UNAME_PROCESSOR" = unknown ; then
-	    UNAME_PROCESSOR=powerpc
+	UNAME_PROCESSOR=$(uname -p)
+	case $UNAME_PROCESSOR in
+	    unknown) UNAME_PROCESSOR=powerpc ;;
+	esac
+	if command -v xcode-select > /dev/null 2> /dev/null && \
+		! xcode-select --print-path > /dev/null 2> /dev/null ; then
+	    # Avoid executing cc if there is no toolchain installed as
+	    # cc will be a stub that puts up a graphical alert
+	    # prompting the user to install developer tools.
+	    CC_FOR_BUILD=no_compiler_found
+	else
+	    set_cc_for_build
 	fi
-	if test "`echo "$UNAME_RELEASE" | sed -e 's/\..*//'`" -le 10 ; then
-	    if [ "$CC_FOR_BUILD" != no_compiler_found ]; then
-		if (echo '#ifdef __LP64__'; echo IS_64BIT_ARCH; echo '#endif') | \
-		       (CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
-		       grep IS_64BIT_ARCH >/dev/null
-		then
-		    case $UNAME_PROCESSOR in
-			i386) UNAME_PROCESSOR=x86_64 ;;
-			powerpc) UNAME_PROCESSOR=powerpc64 ;;
-		    esac
-		fi
-		# On 10.4-10.6 one might compile for PowerPC via gcc -arch ppc
-		if (echo '#ifdef __POWERPC__'; echo IS_PPC; echo '#endif') | \
-		       (CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
-		       grep IS_PPC >/dev/null
-		then
-		    UNAME_PROCESSOR=powerpc
-		fi
+	if test "$CC_FOR_BUILD" != no_compiler_found; then
+	    if (echo '#ifdef __LP64__'; echo IS_64BIT_ARCH; echo '#endif') | \
+		   (CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
+		   grep IS_64BIT_ARCH >/dev/null
+	    then
+		case $UNAME_PROCESSOR in
+		    i386) UNAME_PROCESSOR=x86_64 ;;
+		    powerpc) UNAME_PROCESSOR=powerpc64 ;;
+		esac
+	    fi
+	    # On 10.4-10.6 one might compile for PowerPC via gcc -arch ppc
+	    if (echo '#ifdef __POWERPC__'; echo IS_PPC; echo '#endif') | \
+		   (CCOPTS="" $CC_FOR_BUILD -E - 2>/dev/null) | \
+		   grep IS_PPC >/dev/null
+	    then
+		UNAME_PROCESSOR=powerpc
 	    fi
 	elif test "$UNAME_PROCESSOR" = i386 ; then
-	    # Avoid executing cc on OS X 10.9, as it ships with a stub
-	    # that puts up a graphical alert prompting to install
-	    # developer tools.  Any system running Mac OS X 10.7 or
-	    # later (Darwin 11 and later) is required to have a 64-bit
-	    # processor. This is not true of the ARM version of Darwin
-	    # that Apple uses in portable devices.
-	    UNAME_PROCESSOR=x86_64
+	    # uname -m returns i386 or x86_64
+	    UNAME_PROCESSOR=$UNAME_MACHINE
 	fi
 	echo "$UNAME_PROCESSOR"-apple-darwin"$UNAME_RELEASE"
 	exit ;;
     *:procnto*:*:* | *:QNX:[0123456789]*:*)
-	UNAME_PROCESSOR=`uname -p`
+	UNAME_PROCESSOR=$(uname -p)
 	if test "$UNAME_PROCESSOR" = x86; then
 		UNAME_PROCESSOR=i386
 		UNAME_MACHINE=pc
@@ -1397,10 +1464,10 @@ EOF
 	echo mips-sei-seiux"$UNAME_RELEASE"
 	exit ;;
     *:DragonFly:*:*)
-	echo "$UNAME_MACHINE"-unknown-dragonfly"`echo "$UNAME_RELEASE"|sed -e 's/[-(].*//'`"
+	echo "$UNAME_MACHINE"-unknown-dragonfly"$(echo "$UNAME_RELEASE"|sed -e 's/[-(].*//')"
 	exit ;;
     *:*VMS:*:*)
-	UNAME_MACHINE=`(uname -p) 2>/dev/null`
+	UNAME_MACHINE=$( (uname -p) 2>/dev/null)
 	case "$UNAME_MACHINE" in
 	    A*) echo alpha-dec-vms ; exit ;;
 	    I*) echo ia64-dec-vms ; exit ;;
@@ -1410,13 +1477,13 @@ EOF
 	echo i386-pc-xenix
 	exit ;;
     i*86:skyos:*:*)
-	echo "$UNAME_MACHINE"-pc-skyos"`echo "$UNAME_RELEASE" | sed -e 's/ .*$//'`"
+	echo "$UNAME_MACHINE"-pc-skyos"$(echo "$UNAME_RELEASE" | sed -e 's/ .*$//')"
 	exit ;;
     i*86:rdos:*:*)
 	echo "$UNAME_MACHINE"-pc-rdos
 	exit ;;
-    i*86:AROS:*:*)
-	echo "$UNAME_MACHINE"-pc-aros
+    *:AROS:*:*)
+	echo "$UNAME_MACHINE"-unknown-aros
 	exit ;;
     x86_64:VMkernel:*:*)
 	echo "$UNAME_MACHINE"-unknown-esx
@@ -1424,8 +1491,148 @@ EOF
     amd64:Isilon\ OneFS:*:*)
 	echo x86_64-unknown-onefs
 	exit ;;
+    *:Unleashed:*:*)
+	echo "$UNAME_MACHINE"-unknown-unleashed"$UNAME_RELEASE"
+	exit ;;
 esac
 
+# No uname command or uname output not recognized.
+set_cc_for_build
+cat > "$dummy.c" <<EOF
+#ifdef _SEQUENT_
+#include <sys/types.h>
+#include <sys/utsname.h>
+#endif
+#if defined(ultrix) || defined(_ultrix) || defined(__ultrix) || defined(__ultrix__)
+#if defined (vax) || defined (__vax) || defined (__vax__) || defined(mips) || defined(__mips) || defined(__mips__) || defined(MIPS) || defined(__MIPS__)
+#include <signal.h>
+#if defined(_SIZE_T_) || defined(SIGLOST)
+#include <sys/utsname.h>
+#endif
+#endif
+#endif
+main ()
+{
+#if defined (sony)
+#if defined (MIPSEB)
+  /* BFD wants "bsd" instead of "newsos".  Perhaps BFD should be changed,
+     I don't know....  */
+  printf ("mips-sony-bsd\n"); exit (0);
+#else
+#include <sys/param.h>
+  printf ("m68k-sony-newsos%s\n",
+#ifdef NEWSOS4
+  "4"
+#else
+  ""
+#endif
+  ); exit (0);
+#endif
+#endif
+
+#if defined (NeXT)
+#if !defined (__ARCHITECTURE__)
+#define __ARCHITECTURE__ "m68k"
+#endif
+  int version;
+  version=$( (hostinfo | sed -n 's/.*NeXT Mach \([0-9]*\).*/\1/p') 2>/dev/null);
+  if (version < 4)
+    printf ("%s-next-nextstep%d\n", __ARCHITECTURE__, version);
+  else
+    printf ("%s-next-openstep%d\n", __ARCHITECTURE__, version);
+  exit (0);
+#endif
+
+#if defined (MULTIMAX) || defined (n16)
+#if defined (UMAXV)
+  printf ("ns32k-encore-sysv\n"); exit (0);
+#else
+#if defined (CMU)
+  printf ("ns32k-encore-mach\n"); exit (0);
+#else
+  printf ("ns32k-encore-bsd\n"); exit (0);
+#endif
+#endif
+#endif
+
+#if defined (__386BSD__)
+  printf ("i386-pc-bsd\n"); exit (0);
+#endif
+
+#if defined (sequent)
+#if defined (i386)
+  printf ("i386-sequent-dynix\n"); exit (0);
+#endif
+#if defined (ns32000)
+  printf ("ns32k-sequent-dynix\n"); exit (0);
+#endif
+#endif
+
+#if defined (_SEQUENT_)
+  struct utsname un;
+
+  uname(&un);
+  if (strncmp(un.version, "V2", 2) == 0) {
+    printf ("i386-sequent-ptx2\n"); exit (0);
+  }
+  if (strncmp(un.version, "V1", 2) == 0) { /* XXX is V1 correct? */
+    printf ("i386-sequent-ptx1\n"); exit (0);
+  }
+  printf ("i386-sequent-ptx\n"); exit (0);
+#endif
+
+#if defined (vax)
+#if !defined (ultrix)
+#include <sys/param.h>
+#if defined (BSD)
+#if BSD == 43
+  printf ("vax-dec-bsd4.3\n"); exit (0);
+#else
+#if BSD == 199006
+  printf ("vax-dec-bsd4.3reno\n"); exit (0);
+#else
+  printf ("vax-dec-bsd\n"); exit (0);
+#endif
+#endif
+#else
+  printf ("vax-dec-bsd\n"); exit (0);
+#endif
+#else
+#if defined(_SIZE_T_) || defined(SIGLOST)
+  struct utsname un;
+  uname (&un);
+  printf ("vax-dec-ultrix%s\n", un.release); exit (0);
+#else
+  printf ("vax-dec-ultrix\n"); exit (0);
+#endif
+#endif
+#endif
+#if defined(ultrix) || defined(_ultrix) || defined(__ultrix) || defined(__ultrix__)
+#if defined(mips) || defined(__mips) || defined(__mips__) || defined(MIPS) || defined(__MIPS__)
+#if defined(_SIZE_T_) || defined(SIGLOST)
+  struct utsname *un;
+  uname (&un);
+  printf ("mips-dec-ultrix%s\n", un.release); exit (0);
+#else
+  printf ("mips-dec-ultrix\n"); exit (0);
+#endif
+#endif
+#endif
+
+#if defined (alliant) && defined (i860)
+  printf ("i860-alliant-bsd\n"); exit (0);
+#endif
+
+  exit (1);
+}
+EOF
+
+$CC_FOR_BUILD -o "$dummy" "$dummy.c" 2>/dev/null && SYSTEM_NAME=$($dummy) &&
+	{ echo "$SYSTEM_NAME"; exit; }
+
+# Apollos put the system type in the environment.
+test -d /usr/apollo && { echo "$ISP-apollo-$SYSTYPE"; exit; }
+
 echo "$0: unable to guess system type" >&2
 
 case "$UNAME_MACHINE:$UNAME_SYSTEM" in
@@ -1445,9 +1652,15 @@ This script (version $timestamp), has failed to recognize the
 operating system you are using. If your script is old, overwrite *all*
 copies of config.guess and config.sub with the latest versions from:
 
-  https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess
+  https://git.savannah.gnu.org/cgit/config.git/plain/config.guess
 and
-  https://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub
+  https://git.savannah.gnu.org/cgit/config.git/plain/config.sub
+EOF
+
+year=$(echo $timestamp | sed 's,-.*,,')
+# shellcheck disable=SC2003
+if test "$(expr "$(date +%Y)" - "$year")" -lt 3 ; then
+   cat >&2 <<EOF
 
 If $0 has already been updated, send the following data and any
 information you think might be pertinent to config-patches@gnu.org to
@@ -1455,26 +1668,27 @@ provide the necessary information to handle your system.
 
 config.guess timestamp = $timestamp
 
-uname -m = `(uname -m) 2>/dev/null || echo unknown`
-uname -r = `(uname -r) 2>/dev/null || echo unknown`
-uname -s = `(uname -s) 2>/dev/null || echo unknown`
-uname -v = `(uname -v) 2>/dev/null || echo unknown`
+uname -m = $( (uname -m) 2>/dev/null || echo unknown)
+uname -r = $( (uname -r) 2>/dev/null || echo unknown)
+uname -s = $( (uname -s) 2>/dev/null || echo unknown)
+uname -v = $( (uname -v) 2>/dev/null || echo unknown)
 
-/usr/bin/uname -p = `(/usr/bin/uname -p) 2>/dev/null`
-/bin/uname -X     = `(/bin/uname -X) 2>/dev/null`
+/usr/bin/uname -p = $( (/usr/bin/uname -p) 2>/dev/null)
+/bin/uname -X     = $( (/bin/uname -X) 2>/dev/null)
 
-hostinfo               = `(hostinfo) 2>/dev/null`
-/bin/universe          = `(/bin/universe) 2>/dev/null`
-/usr/bin/arch -k       = `(/usr/bin/arch -k) 2>/dev/null`
-/bin/arch              = `(/bin/arch) 2>/dev/null`
-/usr/bin/oslevel       = `(/usr/bin/oslevel) 2>/dev/null`
-/usr/convex/getsysinfo = `(/usr/convex/getsysinfo) 2>/dev/null`
+hostinfo               = $( (hostinfo) 2>/dev/null)
+/bin/universe          = $( (/bin/universe) 2>/dev/null)
+/usr/bin/arch -k       = $( (/usr/bin/arch -k) 2>/dev/null)
+/bin/arch              = $( (/bin/arch) 2>/dev/null)
+/usr/bin/oslevel       = $( (/usr/bin/oslevel) 2>/dev/null)
+/usr/convex/getsysinfo = $( (/usr/convex/getsysinfo) 2>/dev/null)
 
 UNAME_MACHINE = "$UNAME_MACHINE"
 UNAME_RELEASE = "$UNAME_RELEASE"
 UNAME_SYSTEM  = "$UNAME_SYSTEM"
 UNAME_VERSION = "$UNAME_VERSION"
 EOF
+fi
 
 exit 1
 

configure.ac

@@ -1,4 +1,4 @@
-AC_INIT(nix, m4_esyscmd([bash -c "echo -n $(cat ./.version)$VERSION_SUFFIX"]))
+AC_INIT([nix],[m4_esyscmd(bash -c "echo -n $(cat ./.version)$VERSION_SUFFIX")])
 AC_CONFIG_MACRO_DIRS([m4])
 AC_CONFIG_SRCDIR(README.md)
 AC_CONFIG_AUX_DIR(config)
@@ -9,8 +9,7 @@ AC_PROG_SED
 AC_CANONICAL_HOST
 AC_MSG_CHECKING([for the canonical Nix system name])
 
-AC_ARG_WITH(system, AC_HELP_STRING([--with-system=SYSTEM],
-  [Platform identifier (e.g., `i686-linux').]),
+AC_ARG_WITH(system, AS_HELP_STRING([--with-system=SYSTEM],[Platform identifier (e.g., `i686-linux').]),
   [system=$withval],
   [case "$host_cpu" in
      i*86)
@@ -66,7 +65,7 @@ AC_SYS_LARGEFILE
 AC_STRUCT_DIRENT_D_TYPE
 if test "$sys_name" = sunos; then
     # Solaris requires -lsocket -lnsl for network functions
-    LIBS="-lsocket -lnsl $LIBS"
+    LDFLAGS="-lsocket -lnsl $LDFLAGS"
 fi
 
 
@@ -127,8 +126,7 @@ NEED_PROG(jq, jq)
 AC_SUBST(coreutils, [$(dirname $(type -p cat))])
 
 
-AC_ARG_WITH(store-dir, AC_HELP_STRING([--with-store-dir=PATH],
-  [path of the Nix store (defaults to /nix/store)]),
+AC_ARG_WITH(store-dir, AS_HELP_STRING([--with-store-dir=PATH],[path of the Nix store (defaults to /nix/store)]),
   storedir=$withval, storedir='/nix/store')
 AC_SUBST(storedir)
 
@@ -152,13 +150,12 @@ int main() {
 }]])], GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC=no, GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC=yes)
 AC_MSG_RESULT($GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC)
 if test "x$GCC_ATOMIC_BUILTINS_NEED_LIBATOMIC" = xyes; then
-    LIBS="-latomic $LIBS"
+    LDFLAGS="-latomic $LDFLAGS"
 fi
 
 PKG_PROG_PKG_CONFIG
 
-AC_ARG_ENABLE(shared, AC_HELP_STRING([--enable-shared],
-  [Build shared libraries for Nix [default=yes]]),
+AC_ARG_ENABLE(shared, AS_HELP_STRING([--enable-shared],[Build shared libraries for Nix [default=yes]]),
   shared=$enableval, shared=yes)
 if test "$shared" = yes; then
   AC_SUBST(BUILD_SHARED_LIBS, 1, [Whether to build shared libraries.])
@@ -172,13 +169,12 @@ fi
 PKG_CHECK_MODULES([OPENSSL], [libcrypto], [CXXFLAGS="$OPENSSL_CFLAGS $CXXFLAGS"])
 
 
-# Look for libbz2, a required dependency.
-AC_CHECK_LIB([bz2], [BZ2_bzWriteOpen], [true],
-  [AC_MSG_ERROR([Nix requires libbz2, which is part of bzip2.  See https://web.archive.org/web/20180624184756/http://www.bzip.org/.])])
-AC_CHECK_HEADERS([bzlib.h], [true],
-  [AC_MSG_ERROR([Nix requires libbz2, which is part of bzip2.  See https://web.archive.org/web/20180624184756/http://www.bzip.org/.])])
 # Checks for libarchive
 PKG_CHECK_MODULES([LIBARCHIVE], [libarchive >= 3.1.2], [CXXFLAGS="$LIBARCHIVE_CFLAGS $CXXFLAGS"])
+# Workaround until https://github.com/libarchive/libarchive/issues/1446 is fixed
+if test "$shared" != yes; then
+    LIBARCHIVE_LIBS+=' -lz'
+fi
 
 # Look for SQLite, a required dependency.
 PKG_CHECK_MODULES([SQLITE3], [sqlite3 >= 3.6.19], [CXXFLAGS="$SQLITE3_CFLAGS $CXXFLAGS"])
@@ -199,32 +195,25 @@ PKG_CHECK_MODULES([EDITLINE], [libeditline], [CXXFLAGS="$EDITLINE_CFLAGS $CXXFLA
 ])
 
 # Look for libsodium, an optional dependency.
-PKG_CHECK_MODULES([SODIUM], [libsodium],
-  [AC_DEFINE([HAVE_SODIUM], [1], [Whether to use libsodium for cryptography.])
-   CXXFLAGS="$SODIUM_CFLAGS $CXXFLAGS"
-   have_sodium=1], [have_sodium=])
-AC_SUBST(HAVE_SODIUM, [$have_sodium])
-
-# Look for liblzma, a required dependency.
-PKG_CHECK_MODULES([LIBLZMA], [liblzma], [CXXFLAGS="$LIBLZMA_CFLAGS $CXXFLAGS"])
-AC_CHECK_LIB([lzma], [lzma_stream_encoder_mt],
-  [AC_DEFINE([HAVE_LZMA_MT], [1], [xz multithreaded compression support])])
-
-# Look for zlib, a required dependency.
-PKG_CHECK_MODULES([ZLIB], [zlib], [CXXFLAGS="$ZLIB_CFLAGS $CXXFLAGS"])
-AC_CHECK_HEADER([zlib.h],[:],[AC_MSG_ERROR([could not find the zlib.h header])])
-LDFLAGS="-lz $LDFLAGS"
+PKG_CHECK_MODULES([SODIUM], [libsodium], [CXXFLAGS="$SODIUM_CFLAGS $CXXFLAGS"])
 
 # Look for libbrotli{enc,dec}.
 PKG_CHECK_MODULES([LIBBROTLI], [libbrotlienc libbrotlidec], [CXXFLAGS="$LIBBROTLI_CFLAGS $CXXFLAGS"])
 
+# Look for libcpuid.
+if test "$machine_name" = "x86_64"; then
+  PKG_CHECK_MODULES([LIBCPUID], [libcpuid], [CXXFLAGS="$LIBCPUID_CFLAGS $CXXFLAGS"])
+  have_libcpuid=1
+  AC_DEFINE([HAVE_LIBCPUID], [1], [Use libcpuid])
+fi
+AC_SUBST(HAVE_LIBCPUID, [$have_libcpuid])
+
 
 # Look for libseccomp, required for Linux sandboxing.
 if test "$sys_name" = linux; then
   AC_ARG_ENABLE([seccomp-sandboxing],
-                AC_HELP_STRING([--disable-seccomp-sandboxing],
-                               [Don't build support for seccomp sandboxing (only recommended if your arch doesn't support libseccomp yet!)]
-                              ))
+                AS_HELP_STRING([--disable-seccomp-sandboxing],[Don't build support for seccomp sandboxing (only recommended if your arch doesn't support libseccomp yet!)
+                              ]))
   if test "x$enable_seccomp_sandboxing" != "xno"; then
     PKG_CHECK_MODULES([LIBSECCOMP], [libseccomp],
                       [CXXFLAGS="$LIBSECCOMP_CFLAGS $CXXFLAGS"])
@@ -242,8 +231,8 @@ AC_SUBST(HAVE_SECCOMP, [$have_seccomp])
 # Look for aws-cpp-sdk-s3.
 AC_LANG_PUSH(C++)
 AC_CHECK_HEADERS([aws/s3/S3Client.h],
-  [AC_DEFINE([ENABLE_S3], [1], [Whether to enable S3 support via aws-sdk-cpp.])
-  enable_s3=1], [enable_s3=])
+  [AC_DEFINE([ENABLE_S3], [1], [Whether to enable S3 support via aws-sdk-cpp.]) enable_s3=1], 
+  [AC_DEFINE([ENABLE_S3], [0], [Whether to enable S3 support via aws-sdk-cpp.]) enable_s3=])
 AC_SUBST(ENABLE_S3, [$enable_s3])
 AC_LANG_POP(C++)
 
@@ -251,12 +240,12 @@ if test -n "$enable_s3"; then
   declare -a aws_version_tokens=($(printf '#include <aws/core/VersionConfig.h>\nAWS_SDK_VERSION_STRING' | $CPP $CPPFLAGS - | grep -v '^#.*' | sed 's/"//g' | tr '.' ' '))
   AC_DEFINE_UNQUOTED([AWS_VERSION_MAJOR], ${aws_version_tokens@<:@0@:>@}, [Major version of aws-sdk-cpp.])
   AC_DEFINE_UNQUOTED([AWS_VERSION_MINOR], ${aws_version_tokens@<:@1@:>@}, [Minor version of aws-sdk-cpp.])
+  AC_DEFINE_UNQUOTED([AWS_VERSION_PATCH], ${aws_version_tokens@<:@2@:>@}, [Patch version of aws-sdk-cpp.])
 fi
 
 
 # Whether to use the Boehm garbage collector.
-AC_ARG_ENABLE(gc, AC_HELP_STRING([--enable-gc],
-  [enable garbage collection in the Nix expression evaluator (requires Boehm GC) [default=yes]]),
+AC_ARG_ENABLE(gc, AS_HELP_STRING([--enable-gc],[enable garbage collection in the Nix expression evaluator (requires Boehm GC) [default=yes]]),
   gc=$enableval, gc=yes)
 if test "$gc" = yes; then
   PKG_CHECK_MODULES([BDW_GC], [bdw-gc])
@@ -270,8 +259,7 @@ PKG_CHECK_MODULES([GTEST], [gtest_main])
 
 
 # documentation generation switch
-AC_ARG_ENABLE(doc-gen, AC_HELP_STRING([--disable-doc-gen],
-  [disable documentation generation]),
+AC_ARG_ENABLE(doc-gen, AS_HELP_STRING([--disable-doc-gen],[disable documentation generation]),
   doc_generate=$enableval, doc_generate=yes)
 AC_SUBST(doc_generate)
 
@@ -291,19 +279,7 @@ if test "$(uname)" = "Darwin"; then
 fi
 
 
-# Do we have GNU tar?
-AC_MSG_CHECKING([if you have a recent GNU tar])
-if $tar --version 2> /dev/null | grep -q GNU && tar cvf /dev/null --warning=no-timestamp ./config.log > /dev/null; then
-    AC_MSG_RESULT(yes)
-    tarFlags="--warning=no-timestamp"
-else
-    AC_MSG_RESULT(no)
-fi
-AC_SUBST(tarFlags)
-
-
-AC_ARG_WITH(sandbox-shell, AC_HELP_STRING([--with-sandbox-shell=PATH],
-  [path of a statically-linked shell to use as /bin/sh in sandboxes]),
+AC_ARG_WITH(sandbox-shell, AS_HELP_STRING([--with-sandbox-shell=PATH],[path of a statically-linked shell to use as /bin/sh in sandboxes]),
   sandbox_shell=$withval)
 AC_SUBST(sandbox_shell)
 
@@ -318,6 +294,6 @@ done
 
 rm -f Makefile.config
 
-AC_CONFIG_HEADER([config.h])
+AC_CONFIG_HEADERS([config.h])
 AC_CONFIG_FILES([])
 AC_OUTPUT

corepkgs/local.mk

@@ -1,4 +0,0 @@
-corepkgs_FILES = \
-  fetchurl.nix
-
-$(foreach file,$(corepkgs_FILES),$(eval $(call install-data-in,$(d)/$(file),$(datadir)/nix/corepkgs)))

doc/manual/generate-manpage.nix

@@ -1,55 +1,95 @@
+command:
+
 with builtins;
 with import ./utils.nix;
 
 let
 
   showCommand =
-    { command, section, def }:
-    "${section} Name\n\n"
+    { command, def, filename }:
+    ''
+      **Warning**: This program is **experimental** and its interface is subject to change.
+    ''
+    + "# Name\n\n"
     + "`${command}` - ${def.description}\n\n"
-    + "${section} Synopsis\n\n"
+    + "# Synopsis\n\n"
     + showSynopsis { inherit command; args = def.args; }
-    + (if def ? doc
-       then "${section} Description\n\n" + def.doc + "\n\n"
-       else "")
-    + (let s = showFlags def.flags; in
-       if s != ""
-       then "${section} Flags\n\n${s}"
-       else "")
-    + (if def.examples or [] != []
+    + (if def.commands or {} != {}
        then
-         "${section} Examples\n\n"
-         + concatStrings (map ({ description, command }: "${description}\n\n```console\n${command}\n```\n\n") def.examples)
+         let
+           categories = sort (x: y: x.id < y.id) (unique (map (cmd: cmd.category) (attrValues def.commands)));
+           listCommands = cmds:
+             concatStrings (map (name:
+               "* [`${command} ${name}`](./${appendName filename name}.md) - ${cmds.${name}.description}\n")
+               (attrNames cmds));
+         in
+         "where *subcommand* is one of the following:\n\n"
+         # FIXME: group by category
+         + (if length categories > 1
+            then
+              concatStrings (map
+                (cat:
+                  "**${toString cat.description}:**\n\n"
+                  + listCommands (filterAttrs (n: v: v.category == cat) def.commands)
+                  + "\n"
+                ) categories)
+              + "\n"
+            else
+              listCommands def.commands
+              + "\n")
        else "")
-    + (if def.commands or [] != []
-       then concatStrings (
-         map (name:
-           "# Subcommand `${command} ${name}`\n\n"
-           + showCommand { command = command + " " + name; section = "##"; def = def.commands.${name}; })
-           (attrNames def.commands))
-       else "");
+    + (if def ? doc
+       then def.doc + "\n\n"
+       else "")
+    + (let s = showOptions def.flags; in
+       if s != ""
+       then "# Options\n\n${s}"
+       else "")
+  ;
 
-  showFlags = flags:
-    concatStrings
-      (map (longName:
-        let flag = flags.${longName}; in
-        if flag.category or "" != "config"
-        then
-          "  - `--${longName}`"
-          + (if flag ? shortName then " / `${flag.shortName}`" else "")
-          + (if flag ? labels then " " + (concatStringsSep " " (map (s: "*${s}*") flag.labels)) else "")
-          + "  \n"
-          + "    " + flag.description + "\n\n"
-        else "")
-        (attrNames flags));
+  appendName = filename: name: (if filename == "nix" then "nix3" else filename) + "-" + name;
+
+  showOptions = flags:
+    let
+      categories = sort builtins.lessThan (unique (map (cmd: cmd.category) (attrValues flags)));
+    in
+      concatStrings (map
+        (cat:
+          (if cat != ""
+           then "**${cat}:**\n\n"
+           else "")
+          + concatStrings
+            (map (longName:
+              let
+                flag = flags.${longName};
+              in
+                "  - `--${longName}`"
+                + (if flag ? shortName then " / `-${flag.shortName}`" else "")
+                + (if flag ? labels then " " + (concatStringsSep " " (map (s: "*${s}*") flag.labels)) else "")
+                + "  \n"
+                + "    " + flag.description + "\n\n"
+            ) (attrNames (filterAttrs (n: v: v.category == cat) flags))))
+        categories);
 
   showSynopsis =
     { command, args }:
-    "`${command}` [*flags*...] ${concatStringsSep " "
+    "`${command}` [*option*...] ${concatStringsSep " "
       (map (arg: "*${arg.label}*" + (if arg ? arity then "" else "...")) args)}\n\n";
 
+  processCommand = { command, def, filename }:
+    [ { name = filename + ".md"; value = showCommand { inherit command def filename; }; inherit command; } ]
+    ++ concatMap
+      (name: processCommand {
+        filename = appendName filename name;
+        command = command + " " + name;
+        def = def.commands.${name};
+      })
+      (attrNames def.commands or {});
+
 in
 
-command:
-
-showCommand { command = "nix"; section = "#"; def = command; }
+let
+  manpages = processCommand { filename = "nix"; command = "nix"; def = command; };
+  summary = concatStrings (map (manpage: "    - [${manpage.command}](command-ref/new-cli/${manpage.name})\n") manpages);
+in
+(listToAttrs manpages) // { "SUMMARY.md" = summary; }

doc/manual/local.mk

@@ -4,7 +4,7 @@ MANUAL_SRCS := $(call rwildcard, $(d)/src, *.md)
 
 # Generate man pages.
 man-pages := $(foreach n, \
-  nix-env.1 nix-build.1 nix-shell.1 nix-store.1 nix-instantiate.1 nix.1 \
+  nix-env.1 nix-build.1 nix-shell.1 nix-store.1 nix-instantiate.1 \
   nix-collect-garbage.1 \
   nix-prefetch-url.1 nix-channel.1 \
   nix-hash.1 nix-copy-closure.1 \
@@ -13,59 +13,78 @@ man-pages := $(foreach n, \
 
 clean-files += $(d)/*.1 $(d)/*.5 $(d)/*.8
 
-dist-files += $(man-pages)
+# Provide a dummy environment for nix, so that it will not access files outside the macOS sandbox.
+dummy-env = env -i \
+	HOME=/dummy \
+	NIX_CONF_DIR=/dummy \
+	NIX_SSL_CERT_FILE=/dummy/no-ca-bundle.crt \
+	NIX_STATE_DIR=/dummy
 
-nix-eval = $(bindir)/nix eval --experimental-features nix-command -I nix/corepkgs=corepkgs --store dummy:// --impure --raw --expr
+nix-eval = $(dummy-env) $(bindir)/nix eval --experimental-features nix-command -I nix/corepkgs=corepkgs --store dummy:// --impure --raw
 
 $(d)/%.1: $(d)/src/command-ref/%.md
 	@printf "Title: %s\n\n" "$$(basename $@ .1)" > $^.tmp
 	@cat $^ >> $^.tmp
-	$(trace-gen) lowdown -sT man $^.tmp -o $@
+	$(trace-gen) lowdown -sT man -M section=1 $^.tmp -o $@
 	@rm $^.tmp
 
 $(d)/%.8: $(d)/src/command-ref/%.md
 	@printf "Title: %s\n\n" "$$(basename $@ .8)" > $^.tmp
 	@cat $^ >> $^.tmp
-	$(trace-gen) lowdown -sT man $^.tmp -o $@
+	$(trace-gen) lowdown -sT man -M section=8 $^.tmp -o $@
 	@rm $^.tmp
 
 $(d)/nix.conf.5: $(d)/src/command-ref/conf-file.md
 	@printf "Title: %s\n\n" "$$(basename $@ .5)" > $^.tmp
 	@cat $^ >> $^.tmp
-	$(trace-gen) lowdown -sT man $^.tmp -o $@
+	$(trace-gen) lowdown -sT man -M section=5 $^.tmp -o $@
 	@rm $^.tmp
 
-$(d)/src/command-ref/nix.md: $(d)/nix.json $(d)/generate-manpage.nix $(bindir)/nix
-	$(trace-gen) $(nix-eval) 'import doc/manual/generate-manpage.nix (builtins.fromJSON (builtins.readFile $<))' > $@.tmp
+$(d)/src/SUMMARY.md: $(d)/src/SUMMARY.md.in $(d)/src/command-ref/new-cli
+	$(trace-gen) cat doc/manual/src/SUMMARY.md.in | while IFS= read line; do if [[ $$line = @manpages@ ]]; then cat doc/manual/src/command-ref/new-cli/SUMMARY.md; else echo "$$line"; fi; done > $@.tmp
 	@mv $@.tmp $@
 
+$(d)/src/command-ref/new-cli: $(d)/nix.json $(d)/generate-manpage.nix $(bindir)/nix
+	@rm -rf $@
+	$(trace-gen) $(nix-eval) --write-to $@ --expr 'import doc/manual/generate-manpage.nix (builtins.fromJSON (builtins.readFile $<))'
+
 $(d)/src/command-ref/conf-file.md: $(d)/conf-file.json $(d)/generate-options.nix $(d)/src/command-ref/conf-file-prefix.md $(bindir)/nix
 	@cat doc/manual/src/command-ref/conf-file-prefix.md > $@.tmp
-	$(trace-gen) $(nix-eval) 'import doc/manual/generate-options.nix (builtins.fromJSON (builtins.readFile $<))' >> $@.tmp
+	$(trace-gen) $(nix-eval) --expr 'import doc/manual/generate-options.nix (builtins.fromJSON (builtins.readFile $<))' >> $@.tmp
 	@mv $@.tmp $@
 
 $(d)/nix.json: $(bindir)/nix
-	$(trace-gen) $(bindir)/nix __dump-args > $@.tmp
+	$(trace-gen) $(dummy-env) $(bindir)/nix __dump-args > $@.tmp
 	@mv $@.tmp $@
 
 $(d)/conf-file.json: $(bindir)/nix
-	$(trace-gen) env -i NIX_CONF_DIR=/dummy HOME=/dummy NIX_SSL_CERT_FILE=/dummy/no-ca-bundle.crt $(bindir)/nix show-config --json --experimental-features nix-command > $@.tmp
+	$(trace-gen) $(dummy-env) $(bindir)/nix show-config --json --experimental-features nix-command > $@.tmp
 	@mv $@.tmp $@
 
 $(d)/src/expressions/builtins.md: $(d)/builtins.json $(d)/generate-builtins.nix $(d)/src/expressions/builtins-prefix.md $(bindir)/nix
 	@cat doc/manual/src/expressions/builtins-prefix.md > $@.tmp
-	$(trace-gen) $(nix-eval) 'import doc/manual/generate-builtins.nix (builtins.fromJSON (builtins.readFile $<))' >> $@.tmp
+	$(trace-gen) $(nix-eval) --expr 'import doc/manual/generate-builtins.nix (builtins.fromJSON (builtins.readFile $<))' >> $@.tmp
 	@mv $@.tmp $@
 
 $(d)/builtins.json: $(bindir)/nix
-	$(trace-gen) NIX_PATH=nix/corepkgs=corepkgs $(bindir)/nix __dump-builtins > $@.tmp
-	mv $@.tmp $@
+	$(trace-gen) $(dummy-env) NIX_PATH=nix/corepkgs=corepkgs $(bindir)/nix __dump-builtins > $@.tmp
+	@mv $@.tmp $@
 
 # Generate the HTML manual.
 install: $(docdir)/manual/index.html
 
-$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/custom.css $(d)/src/command-ref/nix.md $(d)/src/command-ref/conf-file.md $(d)/src/expressions/builtins.md
-	$(trace-gen) mdbook build doc/manual -d $(docdir)/manual
+# Generate 'nix' manpages.
+install: $(d)/src/command-ref/new-cli
+	$(trace-gen) for i in doc/manual/src/command-ref/new-cli/*.md; do \
+	  name=$$(basename $$i .md); \
+	  if [[ $$name = SUMMARY ]]; then continue; fi; \
+	  printf "Title: %s\n\n" "$$name" > $$i.tmp; \
+	  cat $$i >> $$i.tmp; \
+	  lowdown -sT man -M section=1 $$i.tmp -o $(mandir)/man1/$$name.1; \
+	done
+
+$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/command-ref/conf-file.md $(d)/src/expressions/builtins.md
+	$(trace-gen) RUST_LOG=warn mdbook build doc/manual -d $(docdir)/manual
 	@cp doc/manual/highlight.pack.js $(docdir)/manual/highlight.js
 
 endif

doc/manual/src/SUMMARY.md.in

@@ -62,11 +62,13 @@
     - [nix-instantiate](command-ref/nix-instantiate.md)
     - [nix-prefetch-url](command-ref/nix-prefetch-url.md)
   - [Experimental Commands](command-ref/experimental-commands.md)
-    - [nix](command-ref/nix.md)
+@manpages@
   - [Files](command-ref/files.md)
     - [nix.conf](command-ref/conf-file.md)
 - [Glossary](glossary.md)
-- [Hacking](hacking.md)
+- [Contributing](contributing/contributing.md)
+  - [Hacking](contributing/hacking.md)
+  - [CLI guideline](contributing/cli-guideline.md)
 - [Release Notes](release-notes/release-notes.md)
   - [Release 2.3 (2019-09-04)](release-notes/rl-2.3.md)
   - [Release 2.2 (2019-01-11)](release-notes/rl-2.2.md)

doc/manual/src/advanced-topics/cores-vs-jobs.md

@@ -4,13 +4,13 @@ Nix has two relevant settings with regards to how your CPU cores will
 be utilized: `cores` and `max-jobs`. This chapter will talk about what
 they are, how they interact, and their configuration trade-offs.
 
-  - `max-jobs`  
+  - `max-jobs`\
     Dictates how many separate derivations will be built at the same
     time. If you set this to zero, the local machine will do no
     builds.  Nix will still substitute from binary caches, and build
     remotely if remote builders are configured.
 
-  - `cores`  
+  - `cores`\
     Suggests how many cores each derivation should use. Similar to
     `make -j`.
 

doc/manual/src/advanced-topics/distributed-builds.md

@@ -37,7 +37,7 @@ then you need to ensure that the `PATH` of non-interactive login shells
 contains Nix.
 
 > **Warning**
-> 
+>
 > If you are building via the Nix daemon, it is the Nix daemon user
 > account (that is, `root`) that should have SSH access to the remote
 > machine. If you can’t or don’t want to configure `root` to be able to
@@ -52,7 +52,7 @@ example, the following command allows you to build a derivation for
 ```console
 $ uname
 Linux
-    
+
 $ nix build \
   '(with import <nixpkgs> { system = "x86_64-darwin"; }; runCommand "foo" {} "uname > $out")' \
   --builders 'ssh://mac x86_64-darwin'
@@ -103,7 +103,7 @@ default, set it to `-`.
     ```nix
     requiredSystemFeatures = [ "kvm" ];
     ```
-    
+
     will cause the build to be performed on a machine that has the `kvm`
     feature.
 
@@ -112,6 +112,10 @@ default, set it to `-`.
     features appear in the derivation’s `requiredSystemFeatures`
     attribute..
 
+8.  The (base64-encoded) public host key of the remote machine. If omitted, SSH
+    will use its regular known-hosts file. Specifically, the field is calculated
+    via `base64 -w0 /etc/ssh/ssh_host_ed25519_key.pub`.
+
 For example, the machine specification
 
     nix@scratchy.labs.cs.uu.nl  i686-linux      /home/nix/.ssh/id_scratchy_auto        8 1 kvm

doc/manual/src/advanced-topics/post-build-hook.md

@@ -53,7 +53,7 @@ set -f # disable globbing
 export IFS=' '
 
 echo "Signing paths" $OUT_PATHS
-nix sign-paths --key-file /etc/nix/key.private $OUT_PATHS
+nix store sign --key-file /etc/nix/key.private $OUT_PATHS
 echo "Uploading paths" $OUT_PATHS
 exec nix copy --to 's3://example-nix-cache' $OUT_PATHS
 ```
@@ -63,7 +63,7 @@ exec nix copy --to 's3://example-nix-cache' $OUT_PATHS
 > The `$OUT_PATHS` variable is a space-separated list of Nix store
 > paths. In this case, we expect and want the shell to perform word
 > splitting to make each output path its own argument to `nix
-> sign-paths`. Nix guarantees the paths will not contain any spaces,
+> store sign`. Nix guarantees the paths will not contain any spaces,
 > however a store path might contain glob characters. The `set -f`
 > disables globbing in the shell.
 

doc/manual/src/command-ref/conf-file-prefix.md

@@ -19,19 +19,33 @@ By default Nix reads settings from the following places:
     and `XDG_CONFIG_HOME`. If these are unset, it will look in
     `$HOME/.config/nix.conf`.
 
-The configuration files consist of `name =
-value` pairs, one per line. Other files can be included with a line like
-`include
-path`, where *path* is interpreted relative to the current conf file and
-a missing file is an error unless `!include` is used instead. Comments
+  - If `NIX_CONFIG` is set, its contents is treated as the contents of
+    a configuration file.
+
+The configuration files consist of `name = value` pairs, one per
+line. Other files can be included with a line like `include path`,
+where *path* is interpreted relative to the current conf file and a
+missing file is an error unless `!include` is used instead. Comments
 start with a `#` character. Here is an example configuration file:
 
     keep-outputs = true       # Nice for developers
     keep-derivations = true   # Idem
 
-You can override settings on the command line using the `--option` flag,
-e.g. `--option keep-outputs
-false`.
+You can override settings on the command line using the `--option`
+flag, e.g. `--option keep-outputs false`. Every configuration setting
+also has a corresponding command line flag, e.g. `--max-jobs 16`; for
+Boolean settings, there are two flags to enable or disable the setting
+(e.g. `--keep-failed` and `--no-keep-failed`).
+
+A configuration setting usually overrides any previous value. However,
+you can prefix the name of the setting by `extra-` to *append* to the
+previous value. For instance,
+
+    substituters = a b
+    extra-substituters = c d
+
+defines the `substituters` setting to be `a b c d`. This is also
+available as a command line flag (e.g. `--extra-substituters`).
 
 The following settings are currently available:
 

doc/manual/src/command-ref/env-common.md

@@ -2,45 +2,49 @@
 
 Most Nix commands interpret the following environment variables:
 
-  - `IN_NIX_SHELL`  
+  - `IN_NIX_SHELL`\
     Indicator that tells if the current environment was set up by
     `nix-shell`. Since Nix 2.0 the values are `"pure"` and `"impure"`
 
-  - `NIX_PATH`  
+  - `NIX_PATH`\
     A colon-separated list of directories used to look up Nix
     expressions enclosed in angle brackets (i.e., `<path>`). For
     instance, the value
-    
+
         /home/eelco/Dev:/etc/nixos
-    
+
     will cause Nix to look for paths relative to `/home/eelco/Dev` and
     `/etc/nixos`, in this order. It is also possible to match paths
     against a prefix. For example, the value
-    
+
         nixpkgs=/home/eelco/Dev/nixpkgs-branch:/etc/nixos
-    
+
     will cause Nix to search for `<nixpkgs/path>` in
     `/home/eelco/Dev/nixpkgs-branch/path` and `/etc/nixos/nixpkgs/path`.
-    
+
     If a path in the Nix search path starts with `http://` or
     `https://`, it is interpreted as the URL of a tarball that will be
     downloaded and unpacked to a temporary location. The tarball must
     consist of a single top-level directory. For example, setting
     `NIX_PATH` to
-    
-        nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-15.09.tar.gz
-    
-    tells Nix to download the latest revision in the Nixpkgs/NixOS 15.09
-    channel.
-    
-    A following shorthand can be used to refer to the official channels:
-    
-        nixpkgs=channel:nixos-15.09
-    
-    The search path can be extended using the `-I` option, which takes
-    precedence over `NIX_PATH`.
 
-  - `NIX_IGNORE_SYMLINK_STORE`  
+        nixpkgs=https://github.com/NixOS/nixpkgs/archive/master.tar.gz
+
+    tells Nix to download and use the current contents of the
+    `master` branch in the `nixpkgs` repository.
+
+    The URLs of the tarballs from the official nixos.org channels (see
+    [the manual for `nix-channel`](nix-channel.md)) can be abbreviated
+    as `channel:<channel-name>`.  For instance, the following two
+    values of `NIX_PATH` are equivalent:
+
+        nixpkgs=channel:nixos-21.05
+        nixpkgs=https://nixos.org/channels/nixos-21.05/nixexprs.tar.xz
+
+    The Nix search path can also be extended using the `-I` option to
+    many Nix commands, which takes precedence over `NIX_PATH`.
+
+  - `NIX_IGNORE_SYMLINK_STORE`\
     Normally, the Nix store directory (typically `/nix/store`) is not
     allowed to contain any symlink components. This is to prevent
     “impure” builds. Builders sometimes “canonicalise” paths by
@@ -50,7 +54,7 @@ Most Nix commands interpret the following environment variables:
     builds are deployed to machines where `/nix/store` resolves
     differently. If you are sure that you’re not going to do that, you
     can set `NIX_IGNORE_SYMLINK_STORE` to `1`.
-    
+
     Note that if you’re symlinking the Nix store so that you can put it
     on another file system than the root file system, on Linux you’re
     better off using `bind` mount points, e.g.,
@@ -59,39 +63,44 @@ Most Nix commands interpret the following environment variables:
     $ mkdir /nix
     $ mount -o bind /mnt/otherdisk/nix /nix
     ```
-    
+
     Consult the mount 8 manual page for details.
 
-  - `NIX_STORE_DIR`  
+  - `NIX_STORE_DIR`\
     Overrides the location of the Nix store (default `prefix/store`).
 
-  - `NIX_DATA_DIR`  
+  - `NIX_DATA_DIR`\
     Overrides the location of the Nix static data directory (default
     `prefix/share`).
 
-  - `NIX_LOG_DIR`  
+  - `NIX_LOG_DIR`\
     Overrides the location of the Nix log directory (default
     `prefix/var/log/nix`).
 
-  - `NIX_STATE_DIR`  
+  - `NIX_STATE_DIR`\
     Overrides the location of the Nix state directory (default
     `prefix/var/nix`).
 
-  - `NIX_CONF_DIR`  
+  - `NIX_CONF_DIR`\
     Overrides the location of the system Nix configuration directory
     (default `prefix/etc/nix`).
 
-  - `NIX_USER_CONF_FILES`  
+  - `NIX_CONFIG`\
+    Applies settings from Nix configuration from the environment.
+    The content is treated as if it was read from a Nix configuration file.
+    Settings are separated by the newline character.
+
+  - `NIX_USER_CONF_FILES`\
     Overrides the location of the user Nix configuration files to load
     from (defaults to the XDG spec locations). The variable is treated
     as a list separated by the `:` token.
 
-  - `TMPDIR`  
+  - `TMPDIR`\
     Use the specified directory to store temporary files. In particular,
     this includes temporary build directories; these can take up
     substantial amounts of disk space. The default is `/tmp`.
 
-  - `NIX_REMOTE`  
+  - `NIX_REMOTE`\
     This variable should be set to `daemon` if you want to use the Nix
     daemon to execute Nix operations. This is necessary in [multi-user
     Nix installations](../installation/multi-user.md). If the Nix
@@ -99,16 +108,16 @@ Most Nix commands interpret the following environment variables:
     should be set to `unix://path/to/socket`. Otherwise, it should be
     left unset.
 
-  - `NIX_SHOW_STATS`  
+  - `NIX_SHOW_STATS`\
     If set to `1`, Nix will print some evaluation statistics, such as
     the number of values allocated.
 
-  - `NIX_COUNT_CALLS`  
+  - `NIX_COUNT_CALLS`\
     If set to `1`, Nix will print how often functions were called during
     Nix expression evaluation. This is useful for profiling your Nix
     expressions.
 
-  - `GC_INITIAL_HEAP_SIZE`  
+  - `GC_INITIAL_HEAP_SIZE`\
     If Nix has been configured to use the Boehm garbage collector, this
     variable sets the initial size of the heap in bytes. It defaults to
     384 MiB. Setting it to a low value reduces memory consumption, but

doc/manual/src/command-ref/nix-build.md

@@ -47,16 +47,16 @@ All options not listed here are passed to `nix-store
 --realise`, except for `--arg` and `--attr` / `-A` which are passed to
 `nix-instantiate`.
 
-  - `--no-out-link`  
+  - `--no-out-link`\
     Do not create a symlink to the output path. Note that as a result
     the output does not become a root of the garbage collector, and so
     might be deleted by `nix-store
                     --gc`.
 
-  - `--dry-run`  
+  - `--dry-run`\
     Show what store paths would be built or downloaded.
 
-  - `--out-link` / `-o` *outlink*  
+  - `--out-link` / `-o` *outlink*\
     Change the name of the symlink to the output path created from
     `result` to *outlink*.
 

doc/manual/src/command-ref/nix-channel.md

@@ -17,26 +17,26 @@ To see the list of official NixOS channels, visit
 
 This command has the following operations:
 
-  - `--add` *url* \[*name*\]  
+  - `--add` *url* \[*name*\]\
     Adds a channel named *name* with URL *url* to the list of subscribed
     channels. If *name* is omitted, it defaults to the last component of
     *url*, with the suffixes `-stable` or `-unstable` removed.
 
-  - `--remove` *name*  
+  - `--remove` *name*\
     Removes the channel named *name* from the list of subscribed
     channels.
 
-  - `--list`  
+  - `--list`\
     Prints the names and URLs of all subscribed channels on standard
     output.
 
-  - `--update` \[*names*…\]  
+  - `--update` \[*names*…\]\
     Downloads the Nix expressions of all subscribed channels (or only
     those included in *names* if specified) and makes them the default
     for `nix-env` operations (by symlinking them from the directory
     `~/.nix-defexpr`).
 
-  - `--rollback` \[*generation*\]  
+  - `--rollback` \[*generation*\]\
     Reverts the previous call to `nix-channel
                     --update`. Optionally, you can specify a specific channel generation
     number to restore.
@@ -70,14 +70,14 @@ $ nix-instantiate --eval -E '(import <nixpkgs> {}).lib.version'
 
 # Files
 
-  - `/nix/var/nix/profiles/per-user/username/channels`  
+  - `/nix/var/nix/profiles/per-user/username/channels`\
     `nix-channel` uses a `nix-env` profile to keep track of previous
     versions of the subscribed channels. Every time you run `nix-channel
     --update`, a new channel generation (that is, a symlink to the
     channel Nix expressions in the Nix store) is created. This enables
     `nix-channel --rollback` to revert to previous versions.
 
-  - `~/.nix-defexpr/channels`  
+  - `~/.nix-defexpr/channels`\
     This is a symlink to
     `/nix/var/nix/profiles/per-user/username/channels`. It ensures that
     `nix-env` can find your channels. In a multi-user installation, you
@@ -89,7 +89,7 @@ $ nix-instantiate --eval -E '(import <nixpkgs> {}).lib.version'
 A channel URL should point to a directory containing the following
 files:
 
-  - `nixexprs.tar.xz`  
+  - `nixexprs.tar.xz`\
     A tarball containing Nix expressions and files referenced by them
     (such as build scripts and patches). At the top level, the tarball
     should contain a single directory. That directory must contain a

doc/manual/src/command-ref/nix-copy-closure.md

@@ -35,21 +35,21 @@ and second to send the dump of those paths.  If this bothers you, use
 
 # Options
 
-  - `--to`  
+  - `--to`\
     Copy the closure of _paths_ from the local Nix store to the Nix
     store on _machine_. This is the default.
 
-  - `--from`  
+  - `--from`\
     Copy the closure of _paths_ from the Nix store on _machine_ to the
     local Nix store.
 
-  - `--gzip`  
+  - `--gzip`\
     Enable compression of the SSH connection.
 
-  - `--include-outputs`  
+  - `--include-outputs`\
     Also copy the outputs of store derivations included in the closure.
 
-  - `--use-substitutes` / `-s`  
+  - `--use-substitutes` / `-s`\
     Attempt to download missing paths on the target machine using Nix’s
     substitute mechanism.  Any paths that cannot be substituted on the
     target are still copied normally from the source.  This is useful,
@@ -58,12 +58,12 @@ and second to send the dump of those paths.  If this bothers you, use
     `nixos.org` (the default binary cache server) is
     fast.
 
-  - `-v`  
+  - `-v`\
     Show verbose output.
 
 # Environment variables
 
-  - `NIX_SSHOPTS`  
+  - `NIX_SSHOPTS`\
     Additional options to be passed to `ssh` on the command
     line.
 

doc/manual/src/command-ref/nix-env.md

@@ -36,27 +36,27 @@ case-sensitive. The regular expression can optionally be followed by a
 dash and a version number; if omitted, any version of the package will
 match. Here are some examples:
 
-  - `firefox`  
+  - `firefox`\
     Matches the package name `firefox` and any version.
 
-  - `firefox-32.0`  
+  - `firefox-32.0`\
     Matches the package name `firefox` and version `32.0`.
 
-  - `gtk\\+`  
+  - `gtk\\+`\
     Matches the package name `gtk+`. The `+` character must be escaped
     using a backslash to prevent it from being interpreted as a
     quantifier, and the backslash must be escaped in turn with another
     backslash to ensure that the shell passes it on.
 
-  - `.\*`  
+  - `.\*`\
     Matches any package name. This is the default for most commands.
 
-  - `'.*zip.*'`  
+  - `'.*zip.*'`\
     Matches any package name containing the string `zip`. Note the dots:
     `'*zip*'` does not work, because in a regular expression, the
     character `*` is interpreted as a quantifier.
 
-  - `'.*(firefox|chromium).*'`  
+  - `'.*(firefox|chromium).*'`\
     Matches any package name containing the strings `firefox` or
     `chromium`.
 
@@ -66,7 +66,7 @@ This section lists the options that are common to all operations. These
 options are allowed for every subcommand, though they may not always
 have an effect.
 
-  - `--file` / `-f` *path*  
+  - `--file` / `-f` *path*\
     Specifies the Nix expression (designated below as the *active Nix
     expression*) used by the `--install`, `--upgrade`, and `--query
     --available` operations to obtain derivations. The default is
@@ -77,13 +77,13 @@ have an effect.
     unpacked to a temporary location. The tarball must include a single
     top-level directory containing at least a file named `default.nix`.
 
-  - `--profile` / `-p` *path*  
+  - `--profile` / `-p` *path*\
     Specifies the profile to be used by those operations that operate on
     a profile (designated below as the *active profile*). A profile is a
     sequence of user environments called *generations*, one of which is
     the *current generation*.
 
-  - `--dry-run`  
+  - `--dry-run`\
     For the `--install`, `--upgrade`, `--uninstall`,
     `--switch-generation`, `--delete-generations` and `--rollback`
     operations, this flag will cause `nix-env` to print what *would* be
@@ -93,7 +93,7 @@ have an effect.
     [substituted](../glossary.md) (i.e., downloaded) and which paths
     will be built from source (because no substitute is available).
 
-  - `--system-filter` *system*  
+  - `--system-filter` *system*\
     By default, operations such as `--query
                     --available` show derivations matching any platform. This option
     allows you to use derivations for the specified platform *system*.
@@ -102,7 +102,7 @@ have an effect.
 
 # Files
 
-  - `~/.nix-defexpr`  
+  - `~/.nix-defexpr`\
     The source for the default Nix expressions used by the
     `--install`, `--upgrade`, and `--query --available` operations to
     obtain derivations. The `--file` option may be used to override
@@ -140,7 +140,7 @@ have an effect.
     The command `nix-channel` places symlinks to the downloaded Nix
     expressions from each subscribed channel in this directory.
 
-  - `~/.nix-profile`  
+  - `~/.nix-profile`\
     A symbolic link to the user's current profile. By default, this
     symlink points to `prefix/var/nix/profiles/default`. The `PATH`
     environment variable should include `~/.nix-profile/bin` for the
@@ -217,13 +217,13 @@ a number of possible ways:
 
 ## Flags
 
-  - `--prebuilt-only` / `-b`  
+  - `--prebuilt-only` / `-b`\
     Use only derivations for which a substitute is registered, i.e.,
     there is a pre-built binary available that can be downloaded in lieu
     of building the derivation. Thus, no packages will be built from
     source.
 
-  - `--preserve-installed`; `-P`  
+  - `--preserve-installed`; `-P`\
     Do not remove derivations with a name matching one of the
     derivations being installed. Usually, trying to have two versions of
     the same package installed in the same generation of a profile will
@@ -231,7 +231,7 @@ a number of possible ways:
     clashes between the two versions. However, this is not the case for
     all packages.
 
-  - `--remove-all`; `-r`  
+  - `--remove-all`; `-r`\
     Remove all previously installed packages first. This is equivalent
     to running `nix-env -e '.*'` first, except that everything happens
     in a single transaction.
@@ -346,24 +346,24 @@ version is installed.
 
 ## Flags
 
-  - `--lt`  
+  - `--lt`\
     Only upgrade a derivation to newer versions. This is the default.
 
-  - `--leq`  
+  - `--leq`\
     In addition to upgrading to newer versions, also “upgrade” to
     derivations that have the same version. Version are not a unique
     identification of a derivation, so there may be many derivations
     that have the same version. This flag may be useful to force
     “synchronisation” between the installed and available derivations.
 
-  - `--eq`  
+  - `--eq`\
     *Only* “upgrade” to derivations that have the same version. This may
     not seem very useful, but it actually is, e.g., when there is a new
     release of Nixpkgs and you want to replace installed applications
     with the same versions built against newer dependencies (to reduce
     the number of dependencies floating around on your system).
 
-  - `--always`  
+  - `--always`\
     In addition to upgrading to newer versions, also “upgrade” to
     derivations that have the same or a lower version. I.e., derivations
     may actually be downgraded depending on what is available in the
@@ -578,11 +578,11 @@ The derivations are sorted by their `name` attributes.
 The following flags specify the set of things on which the query
 operates.
 
-  - `--installed`  
+  - `--installed`\
     The query operates on the store paths that are installed in the
     current generation of the active profile. This is the default.
 
-  - `--available`; `-a`  
+  - `--available`; `-a`\
     The query operates on the derivations that are available in the
     active Nix expression.
 
@@ -593,24 +593,24 @@ selected derivations. Multiple flags may be specified, in which case the
 information is shown in the order given here. Note that the name of the
 derivation is shown unless `--no-name` is specified.
 
-  - `--xml`  
+  - `--xml`\
     Print the result in an XML representation suitable for automatic
     processing by other tools. The root element is called `items`, which
     contains a `item` element for each available or installed
     derivation. The fields discussed below are all stored in attributes
     of the `item` elements.
 
-  - `--json`  
+  - `--json`\
     Print the result in a JSON representation suitable for automatic
     processing by other tools.
 
-  - `--prebuilt-only` / `-b`  
+  - `--prebuilt-only` / `-b`\
     Show only derivations for which a substitute is registered, i.e.,
     there is a pre-built binary available that can be downloaded in lieu
     of building the derivation. Thus, this shows all packages that
     probably can be installed quickly.
 
-  - `--status`; `-s`  
+  - `--status`; `-s`\
     Print the *status* of the derivation. The status consists of three
     characters. The first is `I` or `-`, indicating whether the
     derivation is currently installed in the current generation of the
@@ -621,49 +621,49 @@ derivation is shown unless `--no-name` is specified.
     derivation to be built. The third is `S` or `-`, indicating whether
     a substitute is available for the derivation.
 
-  - `--attr-path`; `-P`  
+  - `--attr-path`; `-P`\
     Print the *attribute path* of the derivation, which can be used to
     unambiguously select it using the `--attr` option available in
     commands that install derivations like `nix-env --install`. This
     option only works together with `--available`
 
-  - `--no-name`  
+  - `--no-name`\
     Suppress printing of the `name` attribute of each derivation.
 
-  - `--compare-versions` / `-c`  
+  - `--compare-versions` / `-c`\
     Compare installed versions to available versions, or vice versa (if
     `--available` is given). This is useful for quickly seeing whether
     upgrades for installed packages are available in a Nix expression. A
     column is added with the following meaning:
 
-      - `<` *version*  
+      - `<` *version*\
         A newer version of the package is available or installed.
 
-      - `=` *version*  
+      - `=` *version*\
         At most the same version of the package is available or
         installed.
 
-      - `>` *version*  
+      - `>` *version*\
         Only older versions of the package are available or installed.
 
-      - `- ?`  
+      - `- ?`\
         No version of the package is available or installed.
 
-  - `--system`  
+  - `--system`\
     Print the `system` attribute of the derivation.
 
-  - `--drv-path`  
+  - `--drv-path`\
     Print the path of the store derivation.
 
-  - `--out-path`  
+  - `--out-path`\
     Print the output path of the derivation.
 
-  - `--description`  
+  - `--description`\
     Print a short (one-line) description of the derivation, if
     available. The description is taken from the `meta.description`
     attribute of the derivation.
 
-  - `--meta`  
+  - `--meta`\
     Print all of the meta-attributes of the derivation. This option is
     only available with `--xml` or `--json`.
 
@@ -874,7 +874,7 @@ error: no generation older than the current (91) exists
 
 # Environment variables
 
-  - `NIX_PROFILE`  
+  - `NIX_PROFILE`\
     Location of the Nix profile. Defaults to the target of the symlink
     `~/.nix-profile`, if it exists, or `/nix/var/nix/profiles/default`
     otherwise.

doc/manual/src/command-ref/nix-hash.md

@@ -29,29 +29,29 @@ md5sum`.
 
 # Options
 
-  - `--flat`  
+  - `--flat`\
     Print the cryptographic hash of the contents of each regular file
     *path*. That is, do not compute the hash over the dump of *path*.
     The result is identical to that produced by the GNU commands
     `md5sum` and `sha1sum`.
 
-  - `--base32`  
+  - `--base32`\
     Print the hash in a base-32 representation rather than hexadecimal.
     This base-32 representation is more compact and can be used in Nix
     expressions (such as in calls to `fetchurl`).
 
-  - `--truncate`  
+  - `--truncate`\
     Truncate hashes longer than 160 bits (such as SHA-256) to 160 bits.
 
-  - `--type` *hashAlgo*  
+  - `--type` *hashAlgo*\
     Use the specified cryptographic hash algorithm, which can be one of
-    `md5`, `sha1`, and `sha256`.
+    `md5`, `sha1`, `sha256`, and `sha512`.
 
-  - `--to-base16`  
+  - `--to-base16`\
     Don’t hash anything, but convert the base-32 hash representation
     *hash* to hexadecimal.
 
-  - `--to-base32`  
+  - `--to-base32`\
     Don’t hash anything, but convert the hexadecimal hash representation
     *hash* to base-32.
 

doc/manual/src/command-ref/nix-instantiate.md

@@ -29,26 +29,26 @@ standard input.
 
 # Options
 
-  - `--add-root` *path*  
+  - `--add-root` *path*\
     See the [corresponding option](nix-store.md) in `nix-store`.
 
-  - `--parse`  
+  - `--parse`\
     Just parse the input files, and print their abstract syntax trees on
     standard output in ATerm format.
 
-  - `--eval`  
+  - `--eval`\
     Just parse and evaluate the input files, and print the resulting
     values on standard output. No instantiation of store derivations
     takes place.
 
-  - `--find-file`  
+  - `--find-file`\
     Look up the given files in Nix’s search path (as specified by the
     `NIX_PATH` environment variable). If found, print the corresponding
     absolute paths on standard output. For instance, if `NIX_PATH` is
     `nixpkgs=/home/alice/nixpkgs`, then `nix-instantiate --find-file
     nixpkgs/default.nix` will print `/home/alice/nixpkgs/default.nix`.
 
-  - `--strict`  
+  - `--strict`\
     When used with `--eval`, recursively evaluate list elements and
     attributes. Normally, such sub-expressions are left unevaluated
     (since the Nix expression language is lazy).
@@ -58,17 +58,17 @@ standard input.
     > This option can cause non-termination, because lazy data
     > structures can be infinitely large.
 
-  - `--json`  
+  - `--json`\
     When used with `--eval`, print the resulting value as an JSON
     representation of the abstract syntax tree rather than as an ATerm.
 
-  - `--xml`  
+  - `--xml`\
     When used with `--eval`, print the resulting value as an XML
     representation of the abstract syntax tree rather than as an ATerm.
     The schema is the same as that used by the [`toXML`
     built-in](../expressions/builtins.md).
 
-  - `--read-write-mode`  
+  - `--read-write-mode`\
     When used with `--eval`, perform evaluation in read/write mode so
     nix language features that require it will still work (at the cost
     of needing to do instantiation of every evaluated derivation). If

doc/manual/src/command-ref/nix-prefetch-url.md

@@ -37,22 +37,22 @@ Nix store is also printed.
 
 # Options
 
-  - `--type` *hashAlgo*  
+  - `--type` *hashAlgo*\
     Use the specified cryptographic hash algorithm, which can be one of
-    `md5`, `sha1`, and `sha256`.
+    `md5`, `sha1`, `sha256`, and `sha512`.
 
-  - `--print-path`  
+  - `--print-path`\
     Print the store path of the downloaded file on standard output.
 
-  - `--unpack`  
+  - `--unpack`\
     Unpack the archive (which must be a tarball or zip file) and add the
     result to the Nix store. The resulting hash can be used with
     functions such as Nixpkgs’s `fetchzip` or `fetchFromGitHub`.
 
-  - `--executable`  
+  - `--executable`\
     Set the executable bit on the downloaded file.
 
-  - `--name` *name*  
+  - `--name` *name*\
     Override the name of the file in the Nix store. By default, this is
     `hash-basename`, where *basename* is the last component of *url*.
     Overriding the name is necessary when *basename* contains characters

doc/manual/src/command-ref/nix-shell.md

@@ -32,7 +32,7 @@ URL of a tarball that will be downloaded and unpacked to a temporary
 location. The tarball must include a single top-level directory
 containing at least a file named `default.nix`.
 
-If the derivation defines the variable `shellHook`, it will be evaluated
+If the derivation defines the variable `shellHook`, it will be run
 after `$stdenv/setup` has been sourced. Since this hook is not executed
 by regular Nix builds, it allows you to perform initialisation specific
 to `nix-shell`. For example, the derivation attribute
@@ -41,10 +41,12 @@ to `nix-shell`. For example, the derivation attribute
 shellHook =
   ''
     echo "Hello shell"
+    export SOME_API_TOKEN="$(cat ~/.config/some-app/api-token)"
   '';
 ```
 
-will cause `nix-shell` to print `Hello shell`.
+will cause `nix-shell` to print `Hello shell` and set the `SOME_API_TOKEN`
+environment variable to a user-configured value.
 
 # Options
 
@@ -52,7 +54,7 @@ All options not listed here are passed to `nix-store
 --realise`, except for `--arg` and `--attr` / `-A` which are passed to
 `nix-instantiate`.
 
-  - `--command` *cmd*  
+  - `--command` *cmd*\
     In the environment of the derivation, run the shell command *cmd*.
     This command is executed in an interactive shell. (Use `--run` to
     use a non-interactive shell instead.) However, a call to `exit` is
@@ -62,36 +64,34 @@ All options not listed here are passed to `nix-store
     drop you into the interactive shell. This can be useful for doing
     any additional initialisation.
 
-  - `--run` *cmd*  
+  - `--run` *cmd*\
     Like `--command`, but executes the command in a non-interactive
     shell. This means (among other things) that if you hit Ctrl-C while
     the command is running, the shell exits.
 
-  - `--exclude` *regexp*  
+  - `--exclude` *regexp*\
     Do not build any dependencies whose store path matches the regular
     expression *regexp*. This option may be specified multiple times.
 
-  - `--pure`  
+  - `--pure`\
     If this flag is specified, the environment is almost entirely
     cleared before the interactive shell is started, so you get an
     environment that more closely corresponds to the “real” Nix build. A
     few variables, in particular `HOME`, `USER` and `DISPLAY`, are
-    retained. Note that `~/.bashrc` and (depending on your Bash
-    installation) `/etc/bashrc` are still sourced, so any variables set
-    there will affect the interactive shell.
+    retained.
 
-  - `--packages` / `-p` *packages*…  
+  - `--packages` / `-p` *packages*…\
     Set up an environment in which the specified packages are present.
     The command line arguments are interpreted as attribute names inside
     the Nix Packages collection. Thus, `nix-shell -p libjpeg openjdk`
     will start a shell in which the packages denoted by the attribute
     names `libjpeg` and `openjdk` are present.
 
-  - `-i` *interpreter*  
+  - `-i` *interpreter*\
     The chained script interpreter to be invoked by `nix-shell`. Only
     applicable in `#!`-scripts (described below).
 
-  - `--keep` *name*  
+  - `--keep` *name*\
     When a `--pure` shell is started, keep the listed environment
     variables.
 
@@ -99,7 +99,7 @@ The following common options are supported:
 
 # Environment variables
 
-  - `NIX_BUILD_SHELL`  
+  - `NIX_BUILD_SHELL`\
     Shell used to start the interactive environment. Defaults to the
     `bash` found in `PATH`.
 
@@ -230,22 +230,23 @@ terraform apply
 > in a nix-shell shebang.
 
 Finally, using the merging of multiple nix-shell shebangs the following
-Haskell script uses a specific branch of Nixpkgs/NixOS (the 18.03 stable
+Haskell script uses a specific branch of Nixpkgs/NixOS (the 20.03 stable
 branch):
 
 ```haskell
 #! /usr/bin/env nix-shell
-#! nix-shell -i runghc -p "haskellPackages.ghcWithPackages (ps: [ps.HTTP ps.tagsoup])"
-#! nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-18.03.tar.gz
+#! nix-shell -i runghc -p "haskellPackages.ghcWithPackages (ps: [ps.download-curl ps.tagsoup])"
+#! nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/nixos-20.03.tar.gz
 
-import Network.HTTP
+import Network.Curl.Download
 import Text.HTML.TagSoup
+import Data.Either
+import Data.ByteString.Char8 (unpack)
 
 -- Fetch nixos.org and print all hrefs.
 main = do
-  resp <- Network.HTTP.simpleHTTP (getRequest "http://nixos.org/")
-  body <- getResponseBody resp
-  let tags = filter (isTagOpenName "a") $ parseTags body
+  resp <- openURI "https://nixos.org/"
+  let tags = filter (isTagOpenName "a") $ parseTags $ unpack $ fromRight undefined resp
   let tags' = map (fromAttrib "href") tags
   mapM_ putStrLn $ filter (/= "") tags'
 ```

doc/manual/src/command-ref/nix-store.md

@@ -22,7 +22,7 @@ This section lists the options that are common to all operations. These
 options are allowed for every subcommand, though they may not always
 have an effect.
 
-  - `--add-root` *path*  
+  - `--add-root` *path*\
     Causes the result of a realisation (`--realise` and
     `--force-realise`) to be registered as a root of the garbage
     collector. *path* will be created as a symlink to the resulting
@@ -79,22 +79,22 @@ paths. Realisation is a somewhat overloaded term:
     system). If the path is already valid, we are done immediately.
     Otherwise, the path and any missing paths in its closure may be
     produced through substitutes. If there are no (successful)
-    subsitutes, realisation fails.
+    substitutes, realisation fails.
 
 The output path of each derivation is printed on standard output. (For
 non-derivations argument, the argument itself is printed.)
 
 The following flags are available:
 
-  - `--dry-run`  
+  - `--dry-run`\
     Print on standard error a description of what packages would be
     built or downloaded, without actually performing the operation.
 
-  - `--ignore-unknown`  
+  - `--ignore-unknown`\
     If a non-derivation path does not have a substitute, then silently
     ignore it.
 
-  - `--check`  
+  - `--check`\
     This option allows you to check whether a derivation is
     deterministic. It rebuilds the specified derivation and checks
     whether the result is bitwise-identical with the existing outputs,
@@ -110,20 +110,20 @@ The following flags are available:
 
 Special exit codes:
 
-  - `100`  
+  - `100`\
     Generic build failure, the builder process returned with a non-zero
     exit code.
 
-  - `101`  
+  - `101`\
     Build timeout, the build was aborted because it did not complete
     within the specified `timeout`.
 
-  - `102`  
+  - `102`\
     Hash mismatch, the build output was rejected because it does not
     match the [`outputHash` attribute of the
     derivation](../expressions/advanced-attributes.md).
 
-  - `104`  
+  - `104`\
     Not deterministic, the build succeeded in check mode but the
     resulting output is not binary reproducable.
 
@@ -170,7 +170,7 @@ access to a restricted ssh user.
 
 The following flags are available:
 
-  - `--write`  
+  - `--write`\
     Allow the connected client to request the realization of
     derivations. In effect, this can be used to make the host act as a
     remote builder.
@@ -200,18 +200,18 @@ reachable via file system references from a set of “roots”, are deleted.
 
 The following suboperations may be specified:
 
-  - `--print-roots`  
+  - `--print-roots`\
     This operation prints on standard output the set of roots used by
     the garbage collector.
 
-  - `--print-live`  
+  - `--print-live`\
     This operation prints on standard output the set of “live” store
     paths, which are all the store paths reachable from the roots. Live
     paths should never be deleted, since that would break consistency —
     it would become possible that applications are installed that
     reference things that are no longer present in the store.
 
-  - `--print-dead`  
+  - `--print-dead`\
     This operation prints out on standard output the set of “dead” store
     paths, which is just the opposite of the set of live paths: any path
     in the store that is not live (with respect to the roots) is dead.
@@ -219,14 +219,14 @@ The following suboperations may be specified:
 By default, all unreachable paths are deleted. The following options
 control what gets deleted and in what order:
 
-  - `--max-freed` *bytes*  
+  - `--max-freed` *bytes*\
     Keep deleting paths until at least *bytes* bytes have been deleted,
     then stop. The argument *bytes* can be followed by the
     multiplicative suffix `K`, `M`, `G` or `T`, denoting KiB, MiB, GiB
     or TiB units.
 
 The behaviour of the collector is also influenced by the
-`keep-outputs` and `keep-derivations` variables in the Nix
+`keep-outputs` and `keep-derivations` settings in the Nix
 configuration file.
 
 By default, the collector prints the total number of freed bytes when it
@@ -300,22 +300,22 @@ symlink.
 
 ## Common query options
 
-  - `--use-output`; `-u`  
+  - `--use-output`; `-u`\
     For each argument to the query that is a store derivation, apply the
     query to the output path of the derivation instead.
 
-  - `--force-realise`; `-f`  
+  - `--force-realise`; `-f`\
     Realise each argument to the query first (see [`nix-store
     --realise`](#operation---realise)).
 
 ## Queries
 
-  - `--outputs`  
+  - `--outputs`\
     Prints out the [output paths](../glossary.md) of the store
     derivations *paths*. These are the paths that will be produced when
     the derivation is built.
 
-  - `--requisites`; `-R`  
+  - `--requisites`; `-R`\
     Prints out the [closure](../glossary.md) of the store path *paths*.
 
     This query has one option:
@@ -332,31 +332,31 @@ symlink.
     dependencies) is obtained by distributing the closure of a store
     derivation and specifying the option `--include-outputs`.
 
-  - `--references`  
+  - `--references`\
     Prints the set of [references](../glossary.md) of the store paths
     *paths*, that is, their immediate dependencies. (For *all*
     dependencies, use `--requisites`.)
 
-  - `--referrers`  
+  - `--referrers`\
     Prints the set of *referrers* of the store paths *paths*, that is,
     the store paths currently existing in the Nix store that refer to
     one of *paths*. Note that contrary to the references, the set of
     referrers is not constant; it can change as store paths are added or
     removed.
 
-  - `--referrers-closure`  
+  - `--referrers-closure`\
     Prints the closure of the set of store paths *paths* under the
     referrers relation; that is, all store paths that directly or
     indirectly refer to one of *paths*. These are all the path currently
     in the Nix store that are dependent on *paths*.
 
-  - `--deriver`; `-d`  
+  - `--deriver`; `-d`\
     Prints the [deriver](../glossary.md) of the store paths *paths*. If
     the path has no deriver (e.g., if it is a source file), or if the
     deriver is not known (e.g., in the case of a binary-only
     deployment), the string `unknown-deriver` is printed.
 
-  - `--graph`  
+  - `--graph`\
     Prints the references graph of the store paths *paths* in the format
     of the `dot` tool of AT\&T's [Graphviz
     package](http://www.graphviz.org/). This can be used to visualise
@@ -364,39 +364,39 @@ symlink.
     this to a store derivation. To obtain a runtime dependency graph,
     apply it to an output path.
 
-  - `--tree`  
+  - `--tree`\
     Prints the references graph of the store paths *paths* as a nested
     ASCII tree. References are ordered by descending closure size; this
     tends to flatten the tree, making it more readable. The query only
     recurses into a store path when it is first encountered; this
     prevents a blowup of the tree representation of the graph.
 
-  - `--graphml`  
+  - `--graphml`\
     Prints the references graph of the store paths *paths* in the
     [GraphML](http://graphml.graphdrawing.org/) file format. This can be
     used to visualise dependency graphs. To obtain a build-time
     dependency graph, apply this to a store derivation. To obtain a
     runtime dependency graph, apply it to an output path.
 
-  - `--binding` *name*; `-b` *name*  
+  - `--binding` *name*; `-b` *name*\
     Prints the value of the attribute *name* (i.e., environment
     variable) of the store derivations *paths*. It is an error for a
     derivation to not have the specified attribute.
 
-  - `--hash`  
+  - `--hash`\
     Prints the SHA-256 hash of the contents of the store paths *paths*
     (that is, the hash of the output of `nix-store --dump` on the given
     paths). Since the hash is stored in the Nix database, this is a fast
     operation.
 
-  - `--size`  
+  - `--size`\
     Prints the size in bytes of the contents of the store paths *paths*
     — to be precise, the size of the output of `nix-store --dump` on
     the given paths. Note that the actual disk space required by the
     store paths may be higher, especially on filesystems with large
     cluster sizes.
 
-  - `--roots`  
+  - `--roots`\
     Prints the garbage collector roots that point, directly or
     indirectly, at the store paths *paths*.
 
@@ -513,7 +513,7 @@ public url or broke since the download expression was written.
 
 This operation has the following options:
 
-  - `--recursive`  
+  - `--recursive`\
     Use recursive instead of flat hashing mode, used when adding
     directories to the store.
 
@@ -540,14 +540,14 @@ being modified by non-Nix tools, or of bugs in Nix itself.
 
 This operation has the following options:
 
-  - `--check-contents`  
+  - `--check-contents`\
     Checks that the contents of every valid store path has not been
     altered by computing a SHA-256 hash of the contents and comparing it
     with the hash stored in the Nix database at build time. Paths that
     have been modified are printed out. For large stores,
     `--check-contents` is obviously quite slow.
 
-  - `--repair`  
+  - `--repair`\
     If any valid path is missing from the store, or (if
     `--check-contents` is given) the contents of a valid path has been
     modified, then try to repair the path by redownloading it. See

doc/manual/src/command-ref/opt-common.md

@@ -2,56 +2,56 @@
 
 Most Nix commands accept the following command-line options:
 
-  - `--help`  
+  - `--help`\
     Prints out a summary of the command syntax and exits.
 
-  - `--version`  
+  - `--version`\
     Prints out the Nix version number on standard output and exits.
 
-  - `--verbose` / `-v`  
+  - `--verbose` / `-v`\
     Increases the level of verbosity of diagnostic messages printed on
     standard error. For each Nix operation, the information printed on
     standard output is well-defined; any diagnostic information is
     printed on standard error, never on standard output.
-    
+
     This option may be specified repeatedly. Currently, the following
     verbosity levels exist:
-    
-      - 0  
+
+      - 0\
         “Errors only”: only print messages explaining why the Nix
         invocation failed.
-    
-      - 1  
+
+      - 1\
         “Informational”: print *useful* messages about what Nix is
         doing. This is the default.
-    
-      - 2  
+
+      - 2\
         “Talkative”: print more informational messages.
-    
-      - 3  
+
+      - 3\
         “Chatty”: print even more informational messages.
-    
-      - 4  
+
+      - 4\
         “Debug”: print debug information.
-    
-      - 5  
+
+      - 5\
         “Vomit”: print vast amounts of debug information.
 
-  - `--quiet`  
+  - `--quiet`\
     Decreases the level of verbosity of diagnostic messages printed on
     standard error. This is the inverse option to `-v` / `--verbose`.
-    
+
     This option may be specified repeatedly. See the previous verbosity
     levels list.
 
-  - `--log-format` *format*  
+  - `--log-format` *format*\
     This option can be used to change the output of the log format, with
     *format* being one of:
-    
-      - raw  
+
+      - raw\
         This is the raw format, as outputted by nix-build.
-    
-      - internal-json  
+
+      - internal-json\
         Outputs the logs in a structured manner.
 
         > **Warning**
@@ -60,30 +60,30 @@ Most Nix commands accept the following command-line options:
         > the error-messages (namely of the `msg`-field) can change
         > between releases.
 
-      - bar  
+      - bar\
         Only display a progress bar during the builds.
-    
-      - bar-with-logs  
+
+      - bar-with-logs\
         Display the raw logs, with the progress bar at the bottom.
 
-  - `--no-build-output` / `-Q`  
+  - `--no-build-output` / `-Q`\
     By default, output written by builders to standard output and
     standard error is echoed to the Nix command's standard error. This
     option suppresses this behaviour. Note that the builder's standard
     output and error are always written to a log file in
     `prefix/nix/var/log/nix`.
 
-  - `--max-jobs` / `-j` *number*  
+  - `--max-jobs` / `-j` *number*\
     Sets the maximum number of build jobs that Nix will perform in
     parallel to the specified number. Specify `auto` to use the number
     of CPUs in the system. The default is specified by the `max-jobs`
     configuration setting, which itself defaults to `1`. A higher
     value is useful on SMP systems or to exploit I/O latency.
-    
+
     Setting it to `0` disallows building on the local machine, which is
     useful when you want builds to happen only on remote builders.
 
-  - `--cores`  
+  - `--cores`\
     Sets the value of the `NIX_BUILD_CORES` environment variable in
     the invocation of builders. Builders can use this variable at
     their discretion to control the maximum amount of parallelism. For
@@ -94,18 +94,18 @@ Most Nix commands accept the following command-line options:
     means that the builder should use all available CPU cores in the
     system.
 
-  - `--max-silent-time`  
+  - `--max-silent-time`\
     Sets the maximum number of seconds that a builder can go without
     producing any data on standard output or standard error. The
     default is specified by the `max-silent-time` configuration
     setting. `0` means no time-out.
 
-  - `--timeout`  
+  - `--timeout`\
     Sets the maximum number of seconds that a builder can run. The
     default is specified by the `timeout` configuration setting. `0`
     means no timeout.
 
-  - `--keep-going` / `-k`  
+  - `--keep-going` / `-k`\
     Keep going in case of failed builds, to the greatest extent
     possible. That is, if building an input of some derivation fails,
     Nix will still build the other inputs, but not the derivation
@@ -113,17 +113,17 @@ Most Nix commands accept the following command-line options:
     for builds of substitutes), possibly killing builds in progress (in
     case of parallel or distributed builds).
 
-  - `--keep-failed` / `-K`  
+  - `--keep-failed` / `-K`\
     Specifies that in case of a build failure, the temporary directory
     (usually in `/tmp`) in which the build takes place should not be
     deleted. The path of the build directory is printed as an
     informational message.
 
-  - `--fallback`  
+  - `--fallback`\
     Whenever Nix attempts to build a derivation for which substitutes
     are known for each output path, but realising the output paths
     through the substitutes fails, fall back on building the derivation.
-    
+
     The most common scenario in which this is useful is when we have
     registered substitutes in order to perform binary distribution from,
     say, a network repository. If the repository is down, the
@@ -134,21 +134,12 @@ Most Nix commands accept the following command-line options:
     failure in obtaining the substitutes to lead to a full build from
     source (with the related consumption of resources).
 
-  - `--no-build-hook`  
-    Disables the build hook mechanism. This allows to ignore remote
-    builders if they are setup on the machine.
-    
-    It's useful in cases where the bandwidth between the client and the
-    remote builder is too low. In that case it can take more time to
-    upload the sources to the remote builder and fetch back the result
-    than to do the computation locally.
-
-  - `--readonly-mode`  
+  - `--readonly-mode`\
     When this option is used, no attempt is made to open the Nix
     database. Most Nix operations do need database access, so those
     operations will fail.
 
-  - `--arg` *name* *value*  
+  - `--arg` *name* *value*\
     This option is accepted by `nix-env`, `nix-instantiate`,
     `nix-shell` and `nix-build`. When evaluating Nix expressions, the
     expression evaluator will automatically try to call functions that
@@ -160,7 +151,7 @@ Most Nix commands accept the following command-line options:
     override a default value). That is, if the evaluator encounters a
     function with an argument named *name*, it will call it with value
     *value*.
-    
+
     For instance, the top-level `default.nix` in Nixpkgs is actually a
     function:
 
@@ -170,7 +161,7 @@ Most Nix commands accept the following command-line options:
       ...
     }: ...
     ```
-    
+
     So if you call this Nix expression (e.g., when you do `nix-env -i
     pkgname`), the function will be called automatically using the
     value [`builtins.currentSystem`](../expressions/builtins.md) for
@@ -179,13 +170,13 @@ Most Nix commands accept the following command-line options:
     since the argument is a Nix string literal, you have to escape the
     quotes.)
 
-  - `--argstr` *name* *value*  
+  - `--argstr` *name* *value*\
     This option is like `--arg`, only the value is not a Nix
     expression but a string. So instead of `--arg system
     \"i686-linux\"` (the outer quotes are to keep the shell happy) you
     can say `--argstr system i686-linux`.
 
-  - `--attr` / `-A` *attrPath*  
+  - `--attr` / `-A` *attrPath*\
     Select an attribute from the top-level Nix expression being
     evaluated. (`nix-env`, `nix-instantiate`, `nix-build` and
     `nix-shell` only.) The *attribute path* *attrPath* is a sequence
@@ -194,34 +185,34 @@ Most Nix commands accept the following command-line options:
     would cause the expression `e.xorg.xorgserver` to be used. See
     [`nix-env --install`](nix-env.md#operation---install) for some
     concrete examples.
-    
+
     In addition to attribute names, you can also specify array indices.
     For instance, the attribute path `foo.3.bar` selects the `bar`
     attribute of the fourth element of the array in the `foo` attribute
     of the top-level expression.
 
-  - `--expr` / `-E`  
+  - `--expr` / `-E`\
     Interpret the command line arguments as a list of Nix expressions to
     be parsed and evaluated, rather than as a list of file names of Nix
     expressions. (`nix-instantiate`, `nix-build` and `nix-shell` only.)
-    
+
     For `nix-shell`, this option is commonly used to give you a shell in
     which you can build the packages returned by the expression. If you
     want to get a shell which contain the *built* packages ready for
     use, give your expression to the `nix-shell -p` convenience flag
     instead.
 
-  - `-I` *path*  
+  - `-I` *path*\
     Add a path to the Nix expression search path. This option may be
     given multiple times. See the `NIX_PATH` environment variable for
     information on the semantics of the Nix search path. Paths added
     through `-I` take precedence over `NIX_PATH`.
 
-  - `--option` *name* *value*  
+  - `--option` *name* *value*\
     Set the Nix configuration option *name* to *value*. This overrides
     settings in the Nix configuration file (see nix.conf5).
 
-  - `--repair`  
+  - `--repair`\
     Fix corrupted or missing store paths by redownloading or rebuilding
     them. Note that this is slow because it requires computing a
     cryptographic hash of the contents of every path in the closure of

doc/manual/src/contributing/cli-guideline.md

@@ -0,0 +1,589 @@
+# CLI guideline
+
+## Goals
+
+Purpose of this document is to provide a clear direction to **help design
+delightful command line** experience. This document contain guidelines to
+follow to ensure a consistent and approachable user experience.
+
+## Overview
+
+`nix` command provides a single entry to a number of sub-commands that help
+**developers and system administrators** in the life-cycle of a software
+project. We particularly need to pay special attention to help and assist new
+users of Nix.
+
+# Naming the `COMMANDS`
+
+Words matter. Naming is an important part of the usability. Users will be
+interacting with Nix on a regular basis so we should **name things for ease of
+understanding**.
+
+We recommend following the [Principle of Least
+Astonishment](https://en.wikipedia.org/wiki/Principle_of_least_astonishment).
+This means that you should **never use acronyms or abbreviations** unless they
+are commonly used in other tools (e.g. `nix init`). And if the command name is
+too long (> 10-12 characters) then shortening it makes sense (e.g.
+“prioritization” → “priority”).
+
+Commands should **follow a noun-verb dialogue**. Although noun-verb formatting
+seems backwards from a speaking perspective (i.e. `nix store copy` vs. `nix
+copy store`) it allows us to organize commands the same way users think about
+completing an action (the group first, then the command).
+
+## Naming rules
+
+Rules are there to guide you by limiting your options. But not everything can
+fit the rules all the time. In those cases document the exceptions in [Appendix
+1: Commands naming exceptions](#appendix-1-commands-naming-exceptions) and
+provide reason. The rules want to force a Nix developer to look, not just at
+the command at hand, but also the command in a full context alongside other
+`nix` commands.
+
+```shell
+$ nix [<GROUP>] <COMMAND> [<ARGUMENTS>] [<OPTIONS>]
+```
+
+- `GROUP`, `COMMAND`, `ARGUMENTS` and `OPTIONS` should be lowercase and in a
+  singular form.
+- `GROUP` should be a  **NOUN**.
+- `COMMAND` should be a **VERB**.
+- `ARGUMENTS` and `OPTIONS` are discussed in [*Input* section](#input).
+
+## Classification
+
+Some commands are more important, some less. While we want all of our commands
+to be perfect we can only spend limited amount of time testing and improving
+them.
+
+This classification tries to separate commands in 3 categories in terms of
+their importance in regards to the new users. Users who are likely to be
+impacted the most by bad user experience.
+
+- **Main commands**
+
+  Commands used for our main use cases and most likely used by new users. We
+  expect attention to details, such as:
+
+    - Proper use of [colors](#colors), [emojis](#special-unicode-characters)
+      and [aligning of text](#text-alignment).
+    - [Autocomplete](#shell-completion) of options.
+    - Show [next possible steps](#next-steps).
+    - Showing some [“tips”](#educate-the-user) when running logs running tasks
+      (eg. building / downloading) in order to teach users interesting bits of
+      Nix ecosystem.
+    - [Help pages](#help-is-essential) to be as good as we can write them
+      pointing to external documentation and tutorials for more.
+
+  Examples of such commands: `nix init`, `nix develop`, `nix build`, `nix run`,
+  ...
+
+- **Infrequently used commands**
+
+  From infrequently used commands we expect less attention to details, but
+  still some:
+
+    - Proper use of [colors](#colors), [emojis](#special-unicode-characters)
+      and [aligning of text](#text-alignment).
+    - [Autocomplete](#shell-completion) of options.
+
+  Examples of such commands: `nix doctor`, `nix edit`, `nix eval`, ...
+
+- **Utility and scripting commands**
+
+  Commands that expose certain internal functionality of `nix`, mostly used by
+  other scripts.
+
+    - [Autocomplete](#shell-completion) of options.
+
+  Examples of such commands: `nix store copy`, `nix hash base16`, `nix store
+  ping`, ...
+
+
+# Help is essential
+
+Help should be built into your command line so that new users can gradually
+discover new features when they need them. 
+
+## Looking for help
+
+Since there is no standard way how user will look for help we rely on ways help
+is provided by commonly used tools. As a guide for this we took `git` and
+whenever in doubt look at it as a preferred direction.
+
+The rules are:
+
+- Help is shown by using `--help` or `help` command (eg `nix` `--``help` or
+  `nix help`).
+- For non-COMMANDs (eg. `nix` `--``help`  and `nix store` `--``help`) we **show
+  a summary** of most common use cases. Summary is presented on the STDOUT
+  without any use of PAGER.
+- For COMMANDs (eg. `nix init` `--``help` or `nix help init`) we display the
+  man page of that command. By default the PAGER is used (as in `git`).
+- At the end of either summary or man page there should be an URL pointing to
+  an online version of more detailed documentation.
+- The structure of summaries and man pages should be the same as in `git`.
+
+## Anticipate where help is needed
+
+Even better then requiring the user to search for help is to anticipate and
+predict when user might need it. Either because the lack of discoverability,
+typo in the input or simply taking the opportunity to teach the user of
+interesting - but less visible - details.
+
+### Shell completion
+
+This type of help is most common and almost expected by users. We need to
+**provide the best shell completion** for `bash`, `zsh` and `fish`.
+
+Completion needs to be **context aware**, this mean when a user types:
+
+```shell
+$ nix build n<TAB>
+```
+
+we need to display a list of flakes starting with `n`.
+
+### Wrong input
+
+As we all know we humans make mistakes, all the time. When a typo - intentional
+or unintentional - is made, we should prompt for closest possible options or
+point to the documentation which would educate user to not make the same
+errors. Here are few examples:
+
+In first example we prompt the user for typing wrong command name:
+
+
+```shell
+$ nix int
+------------------------------------------------------------------------
+  Error! Command `int` not found.
+------------------------------------------------------------------------
+  Did you mean:
+    |> nix init
+    |> nix input
+```
+
+Sometimes users will make mistake either because of a typo or simply because of
+lack of discoverability. Our handling of this cases needs to be context
+sensitive.
+
+
+```shell
+$ nix init --template=template#pyton
+------------------------------------------------------------------------
+  Error! Template `template#pyton` not found.
+------------------------------------------------------------------------
+Initializing Nix project at `/path/to/here`.
+      Select a template for you new project:
+          |> template#pyton
+             template#python-pip
+             template#python-poetry
+```
+
+### Next steps
+
+It can be invaluable to newcomers to show what a possible next steps and what
+is the usual development workflow with Nix. For example:
+
+
+```shell
+$ nix init --template=template#python
+Initializing project `template#python`
+          in `/home/USER/dev/new-project`
+
+  Next steps
+    |> nix develop   -- to enter development environment
+    |> nix build     -- to build your project
+```
+
+### Educate the user
+
+We should take any opportunity to **educate users**, but at the same time we
+must **be very very careful to not annoy users**. There is a thin line between
+being helpful and being annoying.
+
+An example of educating users might be to provide *Tips* in places where they
+are waiting.
+
+```shell
+$ nix build
+    Started building my-project 1.2.3
+ Downloaded python3.8-poetry 1.2.3 in 5.3 seconds
+ Downloaded python3.8-requests 1.2.3 in 5.3 seconds
+------------------------------------------------------------------------
+      Press `v` to increase logs verbosity
+         |> `?` to see other options
+------------------------------------------------------------------------
+      Learn something new with every build...
+         |> See last logs of a build with `nix log --last` command.
+------------------------------------------------------------------------
+  Evaluated my-project 1.2.3 in 14.43 seconds
+Downloading [12 / 200]
+         |> firefox 1.2.3 [#########>       ] 10Mb/s | 2min left
+   Building [2 / 20]
+         |> glibc 1.2.3 -> buildPhase: <last log line>
+------------------------------------------------------------------------
+```
+
+Now **Learn** part of the output is where you educate users. You should only
+show it when you know that a build will take some time and not annoy users of
+the builds that take only few seconds.
+
+Every feature like this should go though a intensive review and testing to
+collect as much a feedback as possible and to fine tune every little detail. If
+done right this can be an awesome features beginners and advance users will
+love, but if not done perfectly it will annoy users and leave bad impression.
+
+# Input
+
+Input to a command is provided via `ARGUMENTS` and `OPTIONS`. 
+
+`ARGUMENTS` represent a required input for a function. When choosing to use
+`ARGUMENT` over function please be aware of the downsides that come with it:
+
+- User will need to remember the order of `ARGUMENTS`. This is not a problem if
+  there is only one `ARGUMENT`.
+- With `OPTIONS` it is possible to provide much better auto completion.
+- With `OPTIONS` it is possible to provide much better error message.
+- Using `OPTIONS` it will mean there is a little bit more typing.
+
+We don’t discourage the use of `ARGUMENTS`, but simply want to make every
+developer consider the downsides and choose wisely.
+
+## Naming the `OPTIONS`
+
+Then only naming convention - apart from the ones mentioned in Naming the
+`COMMANDS` section is how flags are named.
+
+Flags are a type of `OPTION` that represent an option that can be turned ON of
+OFF. We can say **flags are boolean type of** `**OPTION**`.
+
+Here are few examples of flag `OPTIONS`:
+
+- `--colors` vs. `--no-colors` (showing colors in the output)
+- `--emojis` vs. `--no-emojis` (showing emojis in the output)
+
+## Prompt when input not provided
+
+For *main commands* (as [per classification](#classification)) we want command
+to improve the discoverability of possible input. A new user will most likely
+not know which `ARGUMENTS` and `OPTIONS` are required or which values are
+possible for those options.
+
+In cases, the user might not provide the input or they provide wrong input,
+rather then show the error, prompt a user with an option to find and select
+correct input (see examples).
+
+Prompting is of course not required when TTY is not attached to STDIN. This
+would mean that scripts wont need to handle prompt, but rather handle errors.
+
+A place to use prompt and provide user with interactive select
+
+
+```shell
+$ nix init
+Initializing Nix project at `/path/to/here`.
+      Select a template for you new project:
+          |> py
+             template#python-pip
+             template#python-poetry
+             [ Showing 2 templates from 1345 templates ]
+```
+
+Another great place to add prompts are **confirmation dialogues for dangerous
+actions**. For example when adding new substitutor via `OPTIONS` or via
+`flake.nix` we should prompt - for the first time - and let user review what is
+going to happen.
+
+
+```shell
+$ nix build --option substitutors https://cache.example.org
+------------------------------------------------------------------------
+  Warning! A security related question need to be answered.
+------------------------------------------------------------------------
+  The following substitutors will be used to in `my-project`: 
+    - https://cache.example.org
+
+  Do you allow `my-project` to use above mentioned substitutors?
+    [y/N] |> y
+```
+
+# Output
+
+Terminal output can be quite limiting in many ways. Which should forces us to
+think about the experience even more. As with every design the output is a
+compromise between being terse and being verbose, between showing help to
+beginners and annoying advance users. For this it is important that we know
+what are the priorities.
+
+Nix command line should be first and foremost written with beginners in mind.
+But users wont stay beginners for long and what was once useful might quickly
+become annoying. There is no golden rule that we can give in this guideline
+that would make it easier how to draw a line and find best compromise.
+
+What we would encourage is to **build prototypes**, do some **user testing**
+and collect **feedback**. Then repeat the cycle few times.
+
+First design the *happy path* and only after your iron it out, continue to work
+on **edge cases** (handling and displaying errors, changes of the output by
+certain `OPTIONS`, etc…)
+
+## Follow best practices
+
+Needless to say we Nix must be a good citizen and follow best practices in
+command line.
+
+In short: **STDOUT is for output, STDERR is for (human) messaging.**
+
+STDOUT and STDERR provide a way for you to output messages to the user while
+also allowing them to redirect content to a file. For example:
+
+```shell
+$ nix build > build.txt
+------------------------------------------------------------------------
+  Error! Atrribute `bin` missing at (1:94) from string.
+------------------------------------------------------------------------
+
+  1| with import <nixpkgs> { }; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (surge.bin) ]; } ""
+```
+
+Because this warning is on STDERR, it doesn’t end up in the file.
+
+But not everything on STDERR is an error though. For example, you can run `nix
+build` and collect logs in a file while still seeing the progress.
+
+```
+$ nix build > build.txt
+  Evaluated 1234 files in 1.2 seconds
+ Downloaded python3.8-poetry 1.2.3 in 5.3 seconds
+ Downloaded python3.8-requests 1.2.3 in 5.3 seconds
+------------------------------------------------------------------------
+      Press `v` to increase logs verbosity
+         |> `?` to see other options
+------------------------------------------------------------------------
+      Learn something new with every build...
+         |> See last logs of a build with `nix log --last` command.
+------------------------------------------------------------------------
+  Evaluated my-project 1.2.3 in 14.43 seconds
+Downloading [12 / 200]
+         |> firefox 1.2.3 [#########>       ] 10Mb/s | 2min left
+   Building [2 / 20]
+         |> glibc 1.2.3 -> buildPhase: <last log line>
+------------------------------------------------------------------------
+```
+
+## Errors (WIP)
+
+**TODO**: Once we have implementation for the *happy path* then we will think
+how to present errors.
+
+## Not only for humans
+
+Terse, machine-readable output formats can also be useful but shouldn’t get in
+the way of making beautiful CLI output. When needed, commands should offer a
+`--json` flag to allow users to easily parse and script the CLI.
+
+When TTY is not detected on STDOUT we should remove all design elements (no
+colors, no emojis and using ASCII instead of Unicode symbols). The same should
+happen when TTY is not detected on STDERR. We should not display progress /
+status section, but only print warnings and errors.
+
+## Dialog with the user
+
+CLIs don't always make it clear when an action has taken place. For every
+action a user performs, your CLI should provide an equal and appropriate
+reaction, clearly highlighting the what just happened. For example:
+
+```shell
+$ nix build
+ Downloaded python3.8-poetry 1.2.3 in 5.3 seconds
+ Downloaded python3.8-requests 1.2.3 in 5.3 seconds
+...
+   Success! You have successfully built my-project.
+$
+```
+
+Above command clearly states that command successfully completed. And in case
+of `nix build`, which is a command that might take some time to complete, it is
+equally important to also show that a command started.
+
+## Text alignment 
+
+Text alignment is the number one design element that will present all of the
+Nix commands as a family and not as separate tools glued together.
+
+The format we should follow is:
+
+```shell
+$ nix COMMAND
+   VERB_1 NOUN and other words
+  VERB__1 NOUN and other words
+       |> Some details 
+```
+
+Few rules that we can extract from above example:
+
+- Each line should start at least with one space.
+- First word should be a VERB and must be aligned to the right.
+- Second word should be a NOUN and must be aligned to the left.
+- If you can not find a good VERB / NOUN pair, don’t worry make it as
+  understandable to the user as possible.
+- More details of each line can be provided by `|>` character which is serving
+  as the first word when aligning the text
+
+Don’t forget you should also test your terminal output with colors and emojis
+off (`--no-colors --no-emojis`).
+
+## Dim / Bright
+
+After comparing few terminals with different color schemes we would **recommend
+to avoid using dimmed text**. The difference from the rest of the text is very
+little in many terminal and color scheme combinations. Sometimes the difference
+is not even notable, therefore relying on it wouldn’t make much sense.
+
+**The bright text is much better supported** across terminals and color
+schemes. Most of the time the difference is perceived as if the bright text
+would be bold. 
+
+## Colors
+
+Humans are already conditioned by society to attach certain meaning to certain
+colors. While the meaning is not universal, a simple collection of colors is
+used to represent basic emotions. 
+
+Colors that can be used in output
+
+- Red = error, danger, stop
+- Green = success, good
+- Yellow/Orange = proceed with caution, warning, in progress
+- Blue/Magenta = stability, calm
+
+While colors are nice, when command line is used by machines (in automation
+scripts) you want to remove the colors. There should be a global `--no-colors`
+option that would remove the colors.
+
+## Special (Unicode) characters
+
+Most of the terminal have good support for Unicode characters and you should
+use them in your output by default. But always have a backup solution that is
+implemented only with ASCII characters and will be used when `--ascii` option
+is going to be passed in. Please make sure that you test your output also
+without Unicode characters
+
+More they showing all the different Unicode characters it is important to
+**establish common set of characters** that we use for certain situations.
+
+## Emojis
+
+Emojis help channel emotions even better than text, colors and special
+characters.
+
+We recommend **keeping the set of emojis to a minimum**. This will enable each
+emoji to stand out more.
+
+As not everybody is happy about emojis we should provide an `--no-emojis`
+option to disable them. Please make sure that you test your output also without
+emojis.
+
+## Tables
+
+All commands that are listing certain data can be implemented in some sort of a
+table. It’s important that each row of your output is a single ‘entry’ of data.
+Never output table borders. It’s noisy and a huge pain for parsing using other
+tools such as `grep`.
+
+Be mindful of the screen width. Only show a few columns by default with the
+table header, for more the table can be manipulated by the following options:
+
+- `--no-headers`: Show column headers by default but allow to hide them.
+- `--columns`: Comma-separated list of column names to add.
+- `--sort`: Allow sorting by column. Allow inverse and multi-column sort as well.
+
+## Interactive output
+
+Interactive output was selected to be able to strike the balance between
+beginners and advance users. While the default output will target beginners it
+can, with a few key strokes, be changed into and advance introspection tool.
+
+### Progress
+
+For longer running commands we should provide and overview of the progress.
+This is shown best in `nix build` example:
+
+```shell
+$ nix build
+    Started building my-project 1.2.3
+ Downloaded python3.8-poetry 1.2.3 in 5.3 seconds
+ Downloaded python3.8-requests 1.2.3 in 5.3 seconds
+------------------------------------------------------------------------
+      Press `v` to increase logs verbosity
+         |> `?` to see other options
+------------------------------------------------------------------------
+      Learn something new with every build...
+         |> See last logs of a build with `nix log --last` command.
+------------------------------------------------------------------------
+  Evaluated my-project 1.2.3 in 14.43 seconds
+Downloading [12 / 200]
+         |> firefox 1.2.3 [#########>       ] 10Mb/s | 2min left
+   Building [2 / 20]
+         |> glibc 1.2.3 -> buildPhase: <last log line>
+------------------------------------------------------------------------
+```
+
+### Search
+
+Use a `fzf` like fuzzy search when there are multiple options to choose from.
+
+```shell
+$ nix init
+Initializing Nix project at `/path/to/here`.
+      Select a template for you new project:
+          |> py
+             template#python-pip
+             template#python-poetry
+             [ Showing 2 templates from 1345 templates ]
+```
+
+### Prompt
+
+In some situations we need to prompt the user and inform the user about what is
+going to happen.
+
+```shell
+$ nix build --option substitutors https://cache.example.org
+------------------------------------------------------------------------
+  Warning! A security related question need to be answered.
+------------------------------------------------------------------------
+  The following substitutors will be used to in `my-project`: 
+    - https://cache.example.org
+
+  Do you allow `my-project` to use above mentioned substitutors?
+    [y/N] |> y
+```
+
+## Verbosity
+
+There are many ways that you can control verbosity.
+
+Verbosity levels are: 
+
+- `ERROR` (level 0)
+- `WARN` (level 1)
+- `NOTICE` (level 2)
+- `INFO` (level 3)
+- `TALKATIVE` (level 4)
+- `CHATTY` (level 5)
+- `DEBUG` (level 6)
+- `VOMIT` (level 7)
+
+The default level that the command starts is `ERROR`. The simplest way to
+increase the verbosity by stacking `-v` option (eg: `-vvv == level 3 == INFO`).
+There are also two shortcuts, `--debug` to run in `DEBUG` verbosity level and
+`--quiet` to run in `ERROR` verbosity level.
+
+----------
+
+# Appendix 1: Commands naming exceptions
+
+`nix init` and `nix repl` are well established 

doc/manual/src/contributing/contributing.md

@@ -0,0 +1 @@
+# Contributing

doc/manual/src/expressions/advanced-attributes.md

@@ -2,14 +2,14 @@
 
 Derivations can declare some infrequently used optional attributes.
 
-  - `allowedReferences`  
+  - `allowedReferences`\
     The optional attribute `allowedReferences` specifies a list of legal
     references (dependencies) of the output of the builder. For example,
-    
+
     ```nix
     allowedReferences = [];
     ```
-    
+
     enforces that the output of a derivation cannot have any runtime
     dependencies on its inputs. To allow an output to have a runtime
     dependency on itself, use `"out"` as a list item. This is used in
@@ -17,45 +17,45 @@ Derivations can declare some infrequently used optional attributes.
     booting Linux don’t have accidental dependencies on other paths in
     the Nix store.
 
-  - `allowedRequisites`  
+  - `allowedRequisites`\
     This attribute is similar to `allowedReferences`, but it specifies
     the legal requisites of the whole closure, so all the dependencies
     recursively. For example,
-    
+
     ```nix
     allowedRequisites = [ foobar ];
     ```
-    
+
     enforces that the output of a derivation cannot have any other
     runtime dependency than `foobar`, and in addition it enforces that
     `foobar` itself doesn't introduce any other dependency itself.
 
-  - `disallowedReferences`  
+  - `disallowedReferences`\
     The optional attribute `disallowedReferences` specifies a list of
     illegal references (dependencies) of the output of the builder. For
     example,
-    
+
     ```nix
     disallowedReferences = [ foo ];
     ```
-    
+
     enforces that the output of a derivation cannot have a direct
     runtime dependencies on the derivation `foo`.
 
-  - `disallowedRequisites`  
+  - `disallowedRequisites`\
     This attribute is similar to `disallowedReferences`, but it
     specifies illegal requisites for the whole closure, so all the
     dependencies recursively. For example,
-    
+
     ```nix
     disallowedRequisites = [ foobar ];
     ```
-    
+
     enforces that the output of a derivation cannot have any runtime
     dependency on `foobar` or any other derivation depending recursively
     on `foobar`.
 
-  - `exportReferencesGraph`  
+  - `exportReferencesGraph`\
     This attribute allows builders access to the references graph of
     their inputs. The attribute is a list of inputs in the Nix store
     whose references graph the builder needs to know. The value of
@@ -65,17 +65,17 @@ Derivations can declare some infrequently used optional attributes.
     files have the format used by `nix-store --register-validity`
     (with the deriver fields left empty). For example, when the
     following derivation is built:
-    
+
     ```nix
     derivation {
       ...
       exportReferencesGraph = [ "libfoo-graph" libfoo ];
     };
     ```
-    
+
     the references graph of `libfoo` is placed in the file
     `libfoo-graph` in the temporary build directory.
-    
+
     `exportReferencesGraph` is useful for builders that want to do
     something with the closure of a store path. Examples include the
     builders in NixOS that generate the initial ramdisk for booting
@@ -84,66 +84,66 @@ Derivations can declare some infrequently used optional attributes.
     with a Nix store containing the closure of a bootable NixOS
     configuration).
 
-  - `impureEnvVars`  
+  - `impureEnvVars`\
     This attribute allows you to specify a list of environment variables
     that should be passed from the environment of the calling user to
     the builder. Usually, the environment is cleared completely when the
     builder is executed, but with this attribute you can allow specific
     environment variables to be passed unmodified. For example,
     `fetchurl` in Nixpkgs has the line
-    
+
     ```nix
     impureEnvVars = [ "http_proxy" "https_proxy" ... ];
     ```
-    
+
     to make it use the proxy server configuration specified by the user
     in the environment variables `http_proxy` and friends.
-    
+
     This attribute is only allowed in *fixed-output derivations* (see
     below), where impurities such as these are okay since (the hash
     of) the output is known in advance. It is ignored for all other
     derivations.
-    
+
     > **Warning**
-    > 
+    >
     > `impureEnvVars` implementation takes environment variables from
     > the current builder process. When a daemon is building its
     > environmental variables are used. Without the daemon, the
     > environmental variables come from the environment of the
     > `nix-build`.
 
-  - `outputHash`; `outputHashAlgo`; `outputHashMode`  
+  - `outputHash`; `outputHashAlgo`; `outputHashMode`\
     These attributes declare that the derivation is a so-called
     *fixed-output derivation*, which means that a cryptographic hash of
     the output is already known in advance. When the build of a
     fixed-output derivation finishes, Nix computes the cryptographic
     hash of the output and compares it to the hash declared with these
     attributes. If there is a mismatch, the build fails.
-    
+
     The rationale for fixed-output derivations is derivations such as
     those produced by the `fetchurl` function. This function downloads a
     file from a given URL. To ensure that the downloaded file has not
     been modified, the caller must also specify a cryptographic hash of
     the file. For example,
-    
+
     ```nix
     fetchurl {
       url = "http://ftp.gnu.org/pub/gnu/hello/hello-2.1.1.tar.gz";
       sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
     }
     ```
-    
+
     It sometimes happens that the URL of the file changes, e.g., because
     servers are reorganised or no longer available. We then must update
     the call to `fetchurl`, e.g.,
-    
+
     ```nix
     fetchurl {
       url = "ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz";
       sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
     }
     ```
-    
+
     If a `fetchurl` derivation was treated like a normal derivation, the
     output paths of the derivation and *all derivations depending on it*
     would change. For instance, if we were to change the URL of the
@@ -151,16 +151,16 @@ Derivations can declare some infrequently used optional attributes.
     other packages depend) massive rebuilds would be needed. This is
     unfortunate for a change which we know cannot have a real effect as
     it propagates upwards through the dependency graph.
-    
+
     For fixed-output derivations, on the other hand, the name of the
     output path only depends on the `outputHash*` and `name` attributes,
     while all other attributes are ignored for the purpose of computing
     the output path. (The `name` attribute is included because it is
     part of the path.)
-    
+
     As an example, here is the (simplified) Nix expression for
     `fetchurl`:
-    
+
     ```nix
     { stdenv, curl }: # The curl program is used for downloading.
 
@@ -180,43 +180,51 @@ Derivations can declare some infrequently used optional attributes.
       inherit url;
     }
     ```
-    
+
     The `outputHashAlgo` attribute specifies the hash algorithm used to
     compute the hash. It can currently be `"sha1"`, `"sha256"` or
     `"sha512"`.
-    
+
     The `outputHashMode` attribute determines how the hash is computed.
     It must be one of the following two values:
-    
-      - `"flat"`  
+
+      - `"flat"`\
         The output must be a non-executable regular file. If it isn’t,
         the build fails. The hash is simply computed over the contents
         of that file (so it’s equal to what Unix commands like
         `sha256sum` or `sha1sum` produce).
-        
+
         This is the default.
-    
-      - `"recursive"`  
+
+      - `"recursive"`\
         The hash is computed over the NAR archive dump of the output
         (i.e., the result of [`nix-store
         --dump`](../command-ref/nix-store.md#operation---dump)). In
         this case, the output can be anything, including a directory
         tree.
-    
+
     The `outputHash` attribute, finally, must be a string containing
     the hash in either hexadecimal or base-32 notation. (See the
     [`nix-hash` command](../command-ref/nix-hash.md) for information
     about converting to and from base-32 notation.)
+    
+  - `__contentAddressed`
+    If this **experimental** attribute is set to true, then the derivation
+    outputs will be stored in a content-addressed location rather than the
+    traditional input-addressed one.
+    This only has an effect if the `ca-derivation` experimental feature is enabled.
+    
+    Setting this attribute also requires setting `outputHashMode` and `outputHashAlgo` like for *fixed-output derivations* (see above).
 
-  - `passAsFile`  
+  - `passAsFile`\
     A list of names of attributes that should be passed via files rather
     than environment variables. For example, if you have
-    
+
     ```nix
     passAsFile = ["big"];
     big = "a very long string";
     ```
-    
+
     then when the builder runs, the environment variable `bigPath`
     will contain the absolute path to a temporary file containing `a
     very long string`. That is, for any attribute *x* listed in
@@ -226,7 +234,7 @@ Derivations can declare some infrequently used optional attributes.
     builder, since most operating systems impose a limit on the size
     of the environment (typically, a few hundred kilobyte).
 
-  - `preferLocalBuild`  
+  - `preferLocalBuild`\
     If this attribute is set to `true` and [distributed building is
     enabled](../advanced-topics/distributed-builds.md), then, if
     possible, the derivaton will be built locally instead of forwarded
@@ -234,14 +242,14 @@ Derivations can declare some infrequently used optional attributes.
     where the cost of doing a download or remote build would exceed
     the cost of building locally.
 
-  - `allowSubstitutes`  
+  - `allowSubstitutes`\
     If this attribute is set to `false`, then Nix will always build this
     derivation; it will not try to substitute its outputs. This is
     useful for very trivial derivations (such as `writeText` in Nixpkgs)
     that are cheaper to build than to substitute from a binary cache.
-    
+
     > **Note**
-    > 
+    >
     > You need to have a builder configured which satisfies the
     > derivation’s `system` attribute, since the derivation cannot be
     > substituted. Thus it is usually a good idea to align `system` with

doc/manual/src/expressions/builtin-constants.md

@@ -2,7 +2,7 @@
 
 Here are the constants built into the Nix expression evaluator:
 
-  - `builtins`  
+  - `builtins`\
     The set `builtins` contains all the built-in functions and values.
     You can use `builtins` to test for the availability of features in
     the Nix installation, e.g.,
@@ -14,7 +14,7 @@ Here are the constants built into the Nix expression evaluator:
     This allows a Nix expression to fall back gracefully on older Nix
     installations that don’t have the desired built-in function.
 
-  - `builtins.currentSystem`  
+  - `builtins.currentSystem`\
     The built-in value `currentSystem` evaluates to the Nix platform
     identifier for the Nix installation on which the expression is being
     evaluated, such as `"i686-linux"` or `"x86_64-darwin"`.

doc/manual/src/expressions/builtins-prefix.md

@@ -9,7 +9,7 @@ scope. Instead, you can access them through the `builtins` built-in
 value, which is a set that contains all built-in functions and values.
 For instance, `derivation` is also available as `builtins.derivation`.
 
-  - `derivation` *attrs*; `builtins.derivation` *attrs*  
+  - `derivation` *attrs*; `builtins.derivation` *attrs*\
 
     `derivation` is described in [its own section](derivations.md).
 

doc/manual/src/expressions/language-operators.md

@@ -25,5 +25,4 @@ order of precedence (from strongest to weakest binding).
 | Inequality               | *e1* `!=` *e2*                      | none          | Inequality.                                                                                                                                                                                                                   | 11         |
 | Logical AND              | *e1* `&&` *e2*                      | left          | Logical AND.                                                                                                                                                                                                                  | 12         |
 | Logical OR               | *e1* `\|\|` *e2*                    | left          | Logical OR.                                                                                                                                                                                                                   | 13         |
-| Logical Implication      | *e1* `->` *e2*                      | none          | Logical implication (equivalent to `!e1 \|\|
-        e2`).                                                                                                                                                                    | 14         |
+| Logical Implication      | *e1* `->` *e2*                      | none          | Logical implication (equivalent to `!e1 \|\| e2`).                                                                                                                                                                            | 14         |

doc/manual/src/glossary.md

@@ -1,48 +1,48 @@
 # Glossary
 
-  - derivation  
+  - derivation\
     A description of a build action. The result of a derivation is a
     store object. Derivations are typically specified in Nix expressions
     using the [`derivation` primitive](expressions/derivations.md). These are
     translated into low-level *store derivations* (implicitly by
     `nix-env` and `nix-build`, or explicitly by `nix-instantiate`).
 
-  - store  
+  - store\
     The location in the file system where store objects live. Typically
     `/nix/store`.
 
-  - store path  
+  - store path\
     The location in the file system of a store object, i.e., an
     immediate child of the Nix store directory.
 
-  - store object  
+  - store object\
     A file that is an immediate child of the Nix store directory. These
     can be regular files, but also entire directory trees. Store objects
     can be sources (objects copied from outside of the store),
     derivation outputs (objects produced by running a build action), or
     derivations (files describing a build action).
 
-  - substitute  
+  - substitute\
     A substitute is a command invocation stored in the Nix database that
     describes how to build a store object, bypassing the normal build
     mechanism (i.e., derivations). Typically, the substitute builds the
     store object by downloading a pre-built version of the store object
     from some server.
 
-  - purity  
+  - purity\
     The assumption that equal Nix derivations when run always produce
     the same output. This cannot be guaranteed in general (e.g., a
     builder can rely on external inputs such as the network or the
     system time) but the Nix model assumes it.
 
-  - Nix expression  
+  - Nix expression\
     A high-level description of software packages and compositions
     thereof. Deploying software using Nix entails writing Nix
     expressions for your packages. Nix expressions are translated to
     derivations that are stored in the Nix store. These derivations can
     then be built.
 
-  - reference  
+  - reference\
     A store path `P` is said to have a reference to a store path `Q` if
     the store object at `P` contains the path `Q` somewhere. The
     *references* of a store path are the set of store paths to which it
@@ -52,11 +52,11 @@
     output paths), whereas an output path only references other output
     paths.
 
-  - reachable  
+  - reachable\
     A store path `Q` is reachable from another store path `P` if `Q`
     is in the *closure* of the *references* relation.
 
-  - closure  
+  - closure\
     The closure of a store path is the set of store paths that are
     directly or indirectly “reachable” from that store path; that is,
     it’s the closure of the path under the *references* relation. For
@@ -71,29 +71,29 @@
     to path `Q`, then `Q` is in the closure of `P`. Further, if `Q`
     references `R` then `R` is also in the closure of `P`.
 
-  - output path  
+  - output path\
     A store path produced by a derivation.
 
-  - deriver  
+  - deriver\
     The deriver of an *output path* is the store
     derivation that built it.
 
-  - validity  
+  - validity\
     A store path is considered *valid* if it exists in the file system,
     is listed in the Nix database as being valid, and if all paths in
     its closure are also valid.
 
-  - user environment  
+  - user environment\
     An automatically generated store object that consists of a set of
     symlinks to “active” applications, i.e., other store paths. These
     are generated automatically by
     [`nix-env`](command-ref/nix-env.md). See *profiles*.
 
-  - profile  
+  - profile\
     A symlink to the current *user environment* of a user, e.g.,
     `/nix/var/nix/profiles/default`.
 
-  - NAR  
+  - NAR\
     A *N*ix *AR*chive. This is a serialisation of a path in the Nix
     store. It can contain regular files, directories and symbolic
     links.  NARs are generated and unpacked using `nix-store --dump`

doc/manual/src/installation/installing-binary.md

@@ -1,18 +1,26 @@
 # Installing a Binary Distribution
 
-If you are using Linux or macOS versions up to 10.14 (Mojave), the
-easiest way to install Nix is to run the following command:
+The easiest way to install Nix is to run the following command:
 
 ```console
 $ sh <(curl -L https://nixos.org/nix/install)
 ```
 
-If you're using macOS 10.15 (Catalina) or newer, consult [the macOS
-installation instructions](#macos-installation) before installing.
+This will run the installer interactively (causing it to explain what
+it is doing more explicitly), and perform the default "type" of install
+for your platform:
+- single-user on Linux
+- multi-user on macOS
 
-As of Nix 2.1.0, the Nix installer will always default to creating a
-single-user installation, however opting in to the multi-user
-installation is highly recommended.
+  > **Notes on read-only filesystem root in macOS 10.15 Catalina +**
+  > 
+  > - It took some time to support this cleanly. You may see posts,
+  >   examples, and tutorials using obsolete workarounds.
+  > - Supporting it cleanly made macOS installs too complex to qualify
+  >   as single-user, so this type is no longer supported on macOS.
+
+We recommend the multi-user install if it supports your platform and
+you can authenticate with `sudo`.
 
 # Single User Installation
 
@@ -50,9 +58,9 @@ $ rm -rf /nix
 The multi-user Nix installation creates system users, and a system
 service for the Nix daemon.
 
-  - Linux running systemd, with SELinux disabled
-
-  - macOS
+**Supported Systems**
+- Linux running systemd, with SELinux disabled
+- macOS
 
 You can instruct the installer to perform a multi-user installation on
 your system:
@@ -96,165 +104,28 @@ sudo rm /Library/LaunchDaemons/org.nixos.nix-daemon.plist
 There may also be references to Nix in `/etc/profile`, `/etc/bashrc`,
 and `/etc/zshrc` which you may remove.
 
-# macOS Installation
+# macOS Installation <a name="sect-macos-installation-change-store-prefix"></a><a name="sect-macos-installation-encrypted-volume"></a><a name="sect-macos-installation-symlink"></a><a name="sect-macos-installation-recommended-notes"></a>
+<!-- Note: anchors above to catch permalinks to old explanations -->
 
-Starting with macOS 10.15 (Catalina), the root filesystem is read-only.
-This means `/nix` can no longer live on your system volume, and that
-you'll need a workaround to install Nix.
+We believe we have ironed out how to cleanly support the read-only root
+on modern macOS. New installs will do this automatically, and you can
+also re-run a new installer to convert your existing setup.
 
-The recommended approach, which creates an unencrypted APFS volume for
-your Nix store and a "synthetic" empty directory to mount it over at
-`/nix`, is least likely to impair Nix or your system.
+This section previously detailed the situation, options, and trade-offs,
+but it now only outlines what the installer does. You don't need to know
+this to run the installer, but it may help if you run into trouble:
 
-> **Note**
-> 
-> With all separate-volume approaches, it's possible something on your
-> system (particularly daemons/services and restored apps) may need
-> access to your Nix store before the volume is mounted. Adding
-> additional encryption makes this more likely.
-
-If you're using a recent Mac with a [T2
-chip](https://www.apple.com/euro/mac/shared/docs/Apple_T2_Security_Chip_Overview.pdf),
-your drive will still be encrypted at rest (in which case "unencrypted"
-is a bit of a misnomer). To use this approach, just install Nix with:
-
-```console
-$ sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume
-```
-
-If you don't like the sound of this, you'll want to weigh the other
-approaches and tradeoffs detailed in this section.
-
-> **Note**
-> 
-> All of the known workarounds have drawbacks, but we hope better
-> solutions will be available in the future. Some that we have our eye
-> on are:
-> 
-> 1.  A true firmlink would enable the Nix store to live on the primary
->     data volume without the build problems caused by the symlink
->     approach. End users cannot currently create true firmlinks.
-> 
-> 2.  If the Nix store volume shared FileVault encryption with the
->     primary data volume (probably by using the same volume group and
->     role), FileVault encryption could be easily supported by the
->     installer without requiring manual setup by each user.
-
-## Change the Nix store path prefix
-
-Changing the default prefix for the Nix store is a simple approach which
-enables you to leave it on your root volume, where it can take full
-advantage of FileVault encryption if enabled. Unfortunately, this
-approach also opts your device out of some benefits that are enabled by
-using the same prefix across systems:
-
-  - Your system won't be able to take advantage of the binary cache
-    (unless someone is able to stand up and support duplicate caching
-    infrastructure), which means you'll spend more time waiting for
-    builds.
-
-  - It's harder to build and deploy packages to Linux systems.
-
-It would also possible (and often requested) to just apply this change
-ecosystem-wide, but it's an intrusive process that has side effects we
-want to avoid for now.
-
-## Use a separate encrypted volume
-
-If you like, you can also add encryption to the recommended approach
-taken by the installer. You can do this by pre-creating an encrypted
-volume before you run the installer--or you can run the installer and
-encrypt the volume it creates later.
-
-In either case, adding encryption to a second volume isn't quite as
-simple as enabling FileVault for your boot volume. Before you dive in,
-there are a few things to weigh:
-
-1.  The additional volume won't be encrypted with your existing
-    FileVault key, so you'll need another mechanism to decrypt the
-    volume.
-
-2.  You can store the password in Keychain to automatically decrypt the
-    volume on boot--but it'll have to wait on Keychain and may not mount
-    before your GUI apps restore. If any of your launchd agents or apps
-    depend on Nix-installed software (for example, if you use a
-    Nix-installed login shell), the restore may fail or break.
-    
-    On a case-by-case basis, you may be able to work around this problem
-    by using `wait4path` to block execution until your executable is
-    available.
-    
-    It's also possible to decrypt and mount the volume earlier with a
-    login hook--but this mechanism appears to be deprecated and its
-    future is unclear.
-
-3.  You can hard-code the password in the clear, so that your store
-    volume can be decrypted before Keychain is available.
-
-If you are comfortable navigating these tradeoffs, you can encrypt the
-volume with something along the lines of:
-
-```console
-alice$ diskutil apfs enableFileVault /nix -user disk
-```
-
-## Symlink the Nix store to a custom location
-
-Another simple approach is using `/etc/synthetic.conf` to symlink the
-Nix store to the data volume. This option also enables your store to
-share any configured FileVault encryption. Unfortunately, builds that
-resolve the symlink may leak the canonical path or even fail.
-
-Because of these downsides, we can't recommend this approach.
-
-## Notes on the recommended approach
-
-This section goes into a little more detail on the recommended approach.
-You don't need to understand it to run the installer, but it can serve
-as a helpful reference if you run into trouble.
-
-1.  In order to compose user-writable locations into the new read-only
-    system root, Apple introduced a new concept called `firmlinks`,
-    which it describes as a "bi-directional wormhole" between two
-    filesystems. You can see the current firmlinks in
-    `/usr/share/firmlinks`. Unfortunately, firmlinks aren't (currently?)
-    user-configurable.
-    
-    For special cases like NFS mount points or package manager roots,
-    [synthetic.conf(5)](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man5/synthetic.conf.5.html)
-    supports limited user-controlled file-creation (of symlinks, and
-    synthetic empty directories) at `/`. To create a synthetic empty
-    directory for mounting at `/nix`, add the following line to
-    `/etc/synthetic.conf` (create it if necessary):
-    
-        nix
-
-2.  This configuration is applied at boot time, but you can use
-    `apfs.util` to trigger creation (not deletion) of new entries
-    without a reboot:
-    
-    ```console
-    alice$ /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -B
-    ```
-
-3.  Create the new APFS volume with diskutil:
-    
-    ```console
-    alice$ sudo diskutil apfs addVolume diskX APFS 'Nix Store' -mountpoint /nix
-    ```
-
-4.  Using `vifs`, add the new mount to `/etc/fstab`. If it doesn't
-    already have other entries, it should look something like:
-    
-        #
-        # Warning - this file should only be modified with vifs(8)
-        #
-        # Failure to do so is unsupported and may be destructive.
-        #
-        LABEL=Nix\040Store /nix apfs rw,nobrowse
-    
-    The nobrowse setting will keep Spotlight from indexing this volume,
-    and keep it from showing up on your desktop.
+- create a new APFS volume for your Nix store
+- update `/etc/synthetic.conf` to direct macOS to create a "synthetic"
+  empty root directory to mount your volume
+- specify mount options for the volume in `/etc/fstab`
+- if you have FileVault enabled
+    - generate an encryption password
+    - put it in your system Keychain
+    - use it to encrypt the volume
+- create a system LaunchDaemon to mount this volume early enough in the
+  boot process to avoid problems loading or restoring any programs that
+  need access to your Nix store
 
 # Installing a pinned Nix version from a URL
 
@@ -280,10 +151,10 @@ it somewhere (e.g. in `/tmp`), and then run the script named `install`
 inside the binary tarball:
 
 ```console
-alice$ cd /tmp
-alice$ tar xfj nix-1.8-x86_64-darwin.tar.bz2
-alice$ cd nix-1.8-x86_64-darwin
-alice$ ./install
+$ cd /tmp
+$ tar xfj nix-1.8-x86_64-darwin.tar.bz2
+$ cd nix-1.8-x86_64-darwin
+$ ./install
 ```
 
 If you need to edit the multi-user installation script to use different

doc/manual/src/installation/prerequisites-source.md

@@ -30,7 +30,7 @@
     have bzip2 installed, including development headers and libraries.
     If your distribution does not provide these, you can obtain bzip2
     from
-    <https://web.archive.org/web/20180624184756/http://www.bzip.org/>.
+    <https://sourceware.org/bzip2/>.
 
   - `liblzma`, which is provided by XZ Utils. If your distribution does
     not provide this, you can get it from <https://tukaani.org/xz/>.

doc/manual/src/introduction.md

@@ -165,10 +165,10 @@ You’re then dropped into a shell where you can edit, build and test
 the package:
 
 ```console
-[nix-shell]$ tar xf $src
+[nix-shell]$ unpackPhase
 [nix-shell]$ cd pan-*
-[nix-shell]$ ./configure
-[nix-shell]$ make
+[nix-shell]$ configurePhase
+[nix-shell]$ buildPhase
 [nix-shell]$ ./pan/gui/pan
 ```
 

doc/manual/src/package-management/s3-substituter.md

@@ -7,17 +7,17 @@ cache mechanism that Nix usually uses to fetch prebuilt binaries from
 
 The following options can be specified as URL parameters to the S3 URL:
 
-  - `profile`  
+  - `profile`\
     The name of the AWS configuration profile to use. By default Nix
     will use the `default` profile.
 
-  - `region`  
+  - `region`\
     The region of the S3 bucket. `us–east-1` by default.
     
     If your bucket is not in `us–east-1`, you should always explicitly
     specify the region parameter.
 
-  - `endpoint`  
+  - `endpoint`\
     The URL to your S3-compatible service, for when not using Amazon S3.
     Do not specify this value if you're using Amazon S3.
     
@@ -26,7 +26,7 @@ The following options can be specified as URL parameters to the S3 URL:
     > This endpoint must support HTTPS and will use path-based
     > addressing instead of virtual host based addressing.
 
-  - `scheme`  
+  - `scheme`\
     The scheme used for S3 requests, `https` (default) or `http`. This
     option allows you to disable HTTPS for binary caches which don't
     support it.

doc/manual/src/release-notes/rl-2.4.md

@@ -0,0 +1,8 @@
+# Release 2.4 (202X-XX-XX)
+
+  - It is now an error to modify the `plugin-files` setting via a
+    command-line flag that appears after the first non-flag argument
+    to any command, including a subcommand to `nix`. For example,
+    `nix-instantiate default.nix --plugin-files ""` must now become
+    `nix-instantiate --plugin-files "" default.nix`.
+  - Plugins that add new `nix` subcommands are now actually respected.

doc/manual/utils.nix

@@ -1,7 +1,15 @@
 with builtins;
 
-{
+rec {
   splitLines = s: filter (x: !isList x) (split "\n" s);
 
   concatStrings = concatStringsSep "";
+
+  # FIXME: O(n^2)
+  unique = foldl' (acc: e: if elem e acc then acc else acc ++ [ e ]) [];
+
+  nameValuePair = name: value: { inherit name value; };
+
+  filterAttrs = pred: set:
+    listToAttrs (concatMap (name: let v = set.${name}; in if pred name v then [(nameValuePair name v)] else []) (attrNames set));
 }

flake.lock

@@ -3,31 +3,32 @@
     "lowdown-src": {
       "flake": false,
       "locked": {
-        "lastModified": 1598695561,
-        "narHash": "sha256-gyH/5j+h/nWw0W8AcR2WKvNBUsiQ7QuxqSJNXAwV+8E=",
+        "lastModified": 1617481909,
+        "narHash": "sha256-SqnfOFuLuVRRNeVJr1yeEPJue/qWoCp5N6o5Kr///p4=",
         "owner": "kristapsdz",
         "repo": "lowdown",
-        "rev": "1705b4a26fbf065d9574dce47a94e8c7c79e052f",
+        "rev": "148f9b2f586c41b7e36e73009db43ea68c7a1a4d",
         "type": "github"
       },
       "original": {
         "owner": "kristapsdz",
+        "ref": "VERSION_0_8_4",
         "repo": "lowdown",
         "type": "github"
       }
     },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1591633336,
-        "narHash": "sha256-oVXv4xAnDJB03LvZGbC72vSVlIbbJr8tpjEW5o/Fdek=",
+        "lastModified": 1624862269,
+        "narHash": "sha256-JFcsh2+7QtfKdJFoPibLFPLgIW6Ycnv8Bts9a7RYme0=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "70717a337f7ae4e486ba71a500367cad697e5f09",
+        "rev": "f77036342e2b690c61c97202bf48f2ce13acc022",
         "type": "github"
       },
       "original": {
         "id": "nixpkgs",
-        "ref": "nixos-20.03-small",
+        "ref": "nixos-21.05-small",
         "type": "indirect"
       }
     },

flake.nix

@@ -1,8 +1,8 @@
 {
   description = "The purely functional package manager";
 
-  inputs.nixpkgs.url = "nixpkgs/nixos-20.03-small";
-  inputs.lowdown-src = { url = "github:kristapsdz/lowdown"; flake = false; };
+  inputs.nixpkgs.url = "nixpkgs/nixos-21.05-small";
+  inputs.lowdown-src = { url = "github:kristapsdz/lowdown/VERSION_0_8_4"; flake = false; };
 
   outputs = { self, nixpkgs, lowdown-src }:
 
@@ -12,11 +12,13 @@
       versionSuffix =
         if officialRelease
         then ""
-        else "pre${builtins.substring 0 8 (self.lastModifiedDate or self.lastModified)}_${self.shortRev or "dirty"}";
+        else "pre${builtins.substring 0 8 (self.lastModifiedDate or self.lastModified or "19700101")}_${self.shortRev or "dirty"}";
 
       officialRelease = false;
 
-      systems = [ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ];
+      linux64BitSystems = [ "x86_64-linux" "aarch64-linux" ];
+      linuxSystems = linux64BitSystems ++ [ "i686-linux" ];
+      systems = linuxSystems ++ [ "x86_64-darwin" "aarch64-darwin" ];
 
       forAllSystems = f: nixpkgs.lib.genAttrs systems (system: f system);
 
@@ -61,34 +63,43 @@
             "LDFLAGS=-fuse-ld=gold"
           ];
 
-        buildDeps =
-          [ bison
-            flex
-            mdbook
-            lowdown
-            autoconf-archive
-            autoreconfHook
 
-            curl
-            bzip2 xz brotli zlib editline
-            openssl pkgconfig sqlite
+        nativeBuildDeps =
+          [
+            buildPackages.bison
+            buildPackages.flex
+            (lib.getBin buildPackages.lowdown)
+            buildPackages.mdbook
+            buildPackages.autoconf-archive
+            buildPackages.autoreconfHook
+            buildPackages.pkgconfig
+
+            # Tests
+            buildPackages.git
+            buildPackages.mercurial
+            buildPackages.jq
+          ]
+          ++ lib.optionals stdenv.isLinux [(pkgs.util-linuxMinimal or pkgs.utillinuxMinimal)];
+
+        buildDeps =
+          [ curl
+            bzip2 xz brotli editline
+            openssl sqlite
             libarchive
             boost
             nlohmann_json
-
-            # Tests
-            git
-            mercurial
-            jq
+            lowdown
             gmock
           ]
-          ++ lib.optionals stdenv.isLinux [libseccomp utillinuxMinimal]
+          ++ lib.optionals stdenv.isLinux [libseccomp]
           ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium
-          ++ lib.optional (stdenv.isLinux || stdenv.isDarwin)
-            (aws-sdk-cpp.override {
-              apis = ["s3" "transfer"];
-              customMemoryManagement = false;
-            });
+          ++ lib.optional stdenv.isx86_64 libcpuid;
+
+        awsDeps = lib.optional (stdenv.isLinux || stdenv.isDarwin)
+          (aws-sdk-cpp.override {
+            apis = ["s3" "transfer"];
+            customMemoryManagement = false;
+          });
 
         propagatedDeps =
           [ (boehmgc.override { enableLargeConfig = true; })
@@ -100,14 +111,83 @@
           ];
       };
 
+    installScriptFor = systems:
+      with nixpkgsFor.x86_64-linux;
+        runCommand "installer-script"
+          { buildInputs = [ nix ];
+          }
+          ''
+            mkdir -p $out/nix-support
+
+            # Converts /nix/store/50p3qk8kka9dl6wyq40vydq945k0j3kv-nix-2.4pre20201102_550e11f/bin/nix
+            # To 50p3qk8kka9dl6wyq40vydq945k0j3kv/bin/nix
+            tarballPath() {
+              # Remove the store prefix
+              local path=''${1#${builtins.storeDir}/}
+              # Get the path relative to the derivation root
+              local rest=''${path#*/}
+              # Get the derivation hash
+              local drvHash=''${path%%-*}
+              echo "$drvHash/$rest"
+            }
+
+            substitute ${./scripts/install.in} $out/install \
+              ${pkgs.lib.concatMapStrings
+                (system:
+                  '' \
+                  --replace '@tarballHash_${system}@' $(nix --experimental-features nix-command hash-file --base16 --type sha256 ${self.hydraJobs.binaryTarball.${system}}/*.tar.xz) \
+                  --replace '@tarballPath_${system}@' $(tarballPath ${self.hydraJobs.binaryTarball.${system}}/*.tar.xz) \
+                  ''
+                )
+                systems
+              } --replace '@nixVersion@' ${version}
+
+            echo "file installer $out/install" >> $out/nix-support/hydra-build-products
+          '';
+
+      testNixVersions = pkgs: client: daemon: with commonDeps pkgs; pkgs.stdenv.mkDerivation {
+        NIX_DAEMON_PACKAGE = daemon;
+        NIX_CLIENT_PACKAGE = client;
+        # Must keep this name short as OSX has a rather strict limit on the
+        # socket path length, and this name appears in the path of the
+        # nix-daemon socket used in the tests
+        name = "nix-tests";
+        inherit version;
+
+        src = self;
+
+        VERSION_SUFFIX = versionSuffix;
+
+        nativeBuildInputs = nativeBuildDeps;
+        buildInputs = buildDeps ++ awsDeps;
+        propagatedBuildInputs = propagatedDeps;
+
+        enableParallelBuilding = true;
+
+        dontBuild = true;
+        doInstallCheck = true;
+
+        installPhase = ''
+          mkdir -p $out
+        '';
+        installCheckPhase = "make installcheck";
+
+      };
+
     in {
 
       # A Nixpkgs overlay that overrides the 'nix' and
       # 'nix.perl-bindings' packages.
       overlay = final: prev: {
 
-        nix = with final; with commonDeps pkgs; (stdenv.mkDerivation {
+        # An older version of Nix to test against when using the daemon.
+        # Currently using `nixUnstable` as the stable one doesn't respect
+        # `NIX_DAEMON_SOCKET_PATH` which is needed for the tests.
+        nixStable = prev.nix;
+
+        nix = with final; with commonDeps pkgs; stdenv.mkDerivation {
           name = "nix-${version}";
+          inherit version;
 
           src = self;
 
@@ -115,7 +195,8 @@
 
           outputs = [ "out" "dev" "doc" ];
 
-          buildInputs = buildDeps;
+          nativeBuildInputs = nativeBuildDeps;
+          buildInputs = buildDeps ++ awsDeps;
 
           propagatedBuildInputs = propagatedDeps;
 
@@ -152,26 +233,31 @@
           installCheckFlags = "sysconfdir=$(out)/etc";
 
           separateDebugInfo = true;
-        }) // {
 
-          perl-bindings = with final; stdenv.mkDerivation {
+          strictDeps = true;
+
+          passthru.perl-bindings = with final; stdenv.mkDerivation {
             name = "nix-perl-${version}";
 
             src = self;
 
+            nativeBuildInputs =
+              [ buildPackages.autoconf-archive
+                buildPackages.autoreconfHook
+                buildPackages.pkgconfig
+              ];
+
             buildInputs =
-              [ autoconf-archive
-                autoreconfHook
-                nix
+              [ nix
                 curl
                 bzip2
                 xz
-                pkgconfig
                 pkgs.perl
                 boost
                 nlohmann_json
               ]
-              ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium;
+              ++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium
+              ++ lib.optional stdenv.isDarwin darwin.apple_sdk.frameworks.Security;
 
             configureFlags = ''
               --with-dbi=${perlPackages.DBI}/${pkgs.perl.libPrefix}
@@ -185,27 +271,27 @@
 
         };
 
-        lowdown = with final; stdenv.mkDerivation {
-          name = "lowdown-0.7.1";
+        lowdown = with final; stdenv.mkDerivation rec {
+          name = "lowdown-0.8.4";
 
           /*
           src = fetchurl {
-            url = https://kristaps.bsd.lv/lowdown/snapshots/lowdown-0.7.1.tar.gz;
-            hash = "sha512-1daoAQfYD0LdhK6aFhrSQvadjc5GsSPBZw0fJDb+BEHYMBLjqiUl2A7H8N+l0W4YfGKqbsPYSrCy4vct+7U6FQ==";
+            url = "https://kristaps.bsd.lv/lowdown/snapshots/${name}.tar.gz";
+            hash = "sha512-U9WeGoInT9vrawwa57t6u9dEdRge4/P+0wLxmQyOL9nhzOEUU2FRz2Be9H0dCjYE7p2v3vCXIYk40M+jjULATw==";
           };
           */
 
           src = lowdown-src;
 
-          outputs = [ "out" "dev" ];
+          outputs = [ "out" "bin" "dev" ];
 
-          buildInputs = [ which ];
+          nativeBuildInputs = [ which ];
 
-          configurePhase =
-            ''
+          configurePhase = ''
+              ${if (stdenv.isDarwin && stdenv.isAarch64) then "echo \"HAVE_SANDBOX_INIT=false\" > configure.local" else ""}
               ./configure \
                 PREFIX=${placeholder "dev"} \
-                BINDIR=${placeholder "out"}/bin
+                BINDIR=${placeholder "bin"}/bin
             '';
         };
 
@@ -214,10 +300,12 @@
       hydraJobs = {
 
         # Binary package for various platforms.
-        build = nixpkgs.lib.genAttrs systems (system: nixpkgsFor.${system}.nix);
+        build = nixpkgs.lib.genAttrs systems (system: self.packages.${system}.nix);
+
+        buildStatic = nixpkgs.lib.genAttrs linux64BitSystems (system: self.packages.${system}.nix-static);
 
         # Perl bindings for various platforms.
-        perlBindings = nixpkgs.lib.genAttrs systems (system: nixpkgsFor.${system}.nix.perl-bindings);
+        perlBindings = nixpkgs.lib.genAttrs systems (system: self.packages.${system}.nix.perl-bindings);
 
         # Binary tarball for various platforms, containing a Nix store
         # with the closure of 'nix' package, and the second half of
@@ -236,6 +324,7 @@
             }
             ''
               cp ${installerClosureInfo}/registration $TMPDIR/reginfo
+              cp ${./scripts/create-darwin-volume.sh} $TMPDIR/create-darwin-volume.sh
               substitute ${./scripts/install-nix-from-closure.sh} $TMPDIR/install \
                 --subst-var-by nix ${nix} \
                 --subst-var-by cacert ${cacert}
@@ -254,6 +343,7 @@
                 # SC1090: Don't worry about not being able to find
                 #         $nix/etc/profile.d/nix.sh
                 shellcheck --exclude SC1090 $TMPDIR/install
+                shellcheck $TMPDIR/create-darwin-volume.sh
                 shellcheck $TMPDIR/install-darwin-multi-user.sh
                 shellcheck $TMPDIR/install-systemd-multi-user.sh
 
@@ -269,6 +359,7 @@
               fi
 
               chmod +x $TMPDIR/install
+              chmod +x $TMPDIR/create-darwin-volume.sh
               chmod +x $TMPDIR/install-darwin-multi-user.sh
               chmod +x $TMPDIR/install-systemd-multi-user.sh
               chmod +x $TMPDIR/install-multi-user
@@ -281,11 +372,15 @@
                 --absolute-names \
                 --hard-dereference \
                 --transform "s,$TMPDIR/install,$dir/install," \
+                --transform "s,$TMPDIR/create-darwin-volume.sh,$dir/create-darwin-volume.sh," \
                 --transform "s,$TMPDIR/reginfo,$dir/.reginfo," \
                 --transform "s,$NIX_STORE,$dir/store,S" \
-                $TMPDIR/install $TMPDIR/install-darwin-multi-user.sh \
+                $TMPDIR/install \
+                $TMPDIR/create-darwin-volume.sh \
+                $TMPDIR/install-darwin-multi-user.sh \
                 $TMPDIR/install-systemd-multi-user.sh \
-                $TMPDIR/install-multi-user $TMPDIR/reginfo \
+                $TMPDIR/install-multi-user \
+                $TMPDIR/reginfo \
                 $(cat ${installerClosureInfo}/store-paths)
             '');
 
@@ -293,23 +388,8 @@
         # to https://nixos.org/nix/install. It downloads the binary
         # tarball for the user's system and calls the second half of the
         # installation script.
-        installerScript =
-          with nixpkgsFor.x86_64-linux;
-          runCommand "installer-script"
-            { buildInputs = [ nix ];
-            }
-            ''
-              mkdir -p $out/nix-support
-
-              substitute ${./scripts/install.in} $out/install \
-                ${pkgs.lib.concatMapStrings
-                  (system: "--replace '@binaryTarball_${system}@' $(nix --experimental-features nix-command hash-file --base16 --type sha256 ${self.hydraJobs.binaryTarball.${system}}/*.tar.xz) ")
-                  [ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ]
-                } \
-                --replace '@nixVersion@' ${version}
-
-              echo "file installer $out/install" >> $out/nix-support/hydra-build-products
-            '';
+        installerScript = installScriptFor [ "x86_64-linux" "i686-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" ];
+        installerScriptForGHA = installScriptFor [ "x86_64-linux" "x86_64-darwin" ];
 
         # Line coverage analysis.
         coverage =
@@ -323,7 +403,8 @@
 
             enableParallelBuilding = true;
 
-            buildInputs = buildDeps ++ propagatedDeps;
+            nativeBuildInputs = nativeBuildDeps;
+            buildInputs = buildDeps ++ propagatedDeps ++ awsDeps;
 
             dontInstall = false;
 
@@ -364,38 +445,6 @@
               inherit (self) overlay;
             });
 
-        # Test whether the binary tarball works in an Ubuntu system.
-        tests.binaryTarball =
-          with nixpkgsFor.x86_64-linux;
-          vmTools.runInLinuxImage (runCommand "nix-binary-tarball-test"
-            { diskImage = vmTools.diskImages.ubuntu1204x86_64;
-            }
-            ''
-              set -x
-              useradd -m alice
-              su - alice -c 'tar xf ${self.hydraJobs.binaryTarball.x86_64-linux}/*.tar.*'
-              mkdir /dest-nix
-              mount -o bind /dest-nix /nix # Provide a writable /nix.
-              chown alice /nix
-              su - alice -c '_NIX_INSTALLER_TEST=1 ./nix-*/install'
-              su - alice -c 'nix-store --verify'
-              su - alice -c 'PAGER= nix-store -qR ${self.hydraJobs.build.x86_64-linux}'
-
-              # Check whether 'nix upgrade-nix' works.
-              cat > /tmp/paths.nix <<EOF
-              {
-                x86_64-linux = "${self.hydraJobs.build.x86_64-linux}";
-              }
-              EOF
-              su - alice -c 'nix --experimental-features nix-command upgrade-nix -vvv --nix-store-paths-url file:///tmp/paths.nix'
-              (! [ -L /home/alice/.profile-1-link ])
-              su - alice -c 'PAGER= nix-store -qR ${self.hydraJobs.build.x86_64-linux}'
-
-              mkdir -p $out/nix-support
-              touch $out/nix-support/hydra-build-products
-              umount /nix
-            '');
-
         /*
         # Check whether we can still evaluate all of Nixpkgs.
         tests.evalNixpkgs =
@@ -425,10 +474,58 @@
       checks = forAllSystems (system: {
         binaryTarball = self.hydraJobs.binaryTarball.${system};
         perlBindings = self.hydraJobs.perlBindings.${system};
+        installTests =
+          let pkgs = nixpkgsFor.${system}; in
+          pkgs.runCommand "install-tests" {
+            againstSelf = testNixVersions pkgs pkgs.nix pkgs.pkgs.nix;
+            againstCurrentUnstable = testNixVersions pkgs pkgs.nix pkgs.nixUnstable;
+            # Disabled because the latest stable version doesn't handle
+            # `NIX_DAEMON_SOCKET_PATH` which is required for the tests to work
+            # againstLatestStable = testNixVersions pkgs pkgs.nix pkgs.nixStable;
+          } "touch $out";
       });
 
       packages = forAllSystems (system: {
         inherit (nixpkgsFor.${system}) nix;
+      } // nixpkgs.lib.optionalAttrs (builtins.elem system linux64BitSystems) {
+        nix-static = let
+          nixpkgs = nixpkgsFor.${system}.pkgsStatic;
+        in with commonDeps nixpkgs; nixpkgs.stdenv.mkDerivation {
+          name = "nix-${version}";
+
+          src = self;
+
+          VERSION_SUFFIX = versionSuffix;
+
+          outputs = [ "out" "dev" "doc" ];
+
+          nativeBuildInputs = nativeBuildDeps;
+          buildInputs = buildDeps ++ propagatedDeps;
+
+          configureFlags = [ "--sysconfdir=/etc" ];
+
+          enableParallelBuilding = true;
+
+          makeFlags = "profiledir=$(out)/etc/profile.d";
+
+          doCheck = true;
+
+          installFlags = "sysconfdir=$(out)/etc";
+
+          postInstall = ''
+            mkdir -p $doc/nix-support
+            echo "doc manual $doc/share/doc/nix/manual" >> $doc/nix-support/hydra-build-products
+            mkdir -p $out/nix-support
+            echo "file binary-dist $out/bin/nix" >> $out/nix-support/hydra-build-products
+          '';
+
+          doInstallCheck = true;
+          installCheckFlags = "sysconfdir=$(out)/etc";
+
+          stripAllList = ["bin"];
+
+          strictDeps = true;
+        };
       });
 
       defaultPackage = forAllSystems (system: self.packages.${system}.nix);
@@ -442,7 +539,8 @@
 
           outputs = [ "out" "dev" "doc" ];
 
-          buildInputs = buildDeps ++ propagatedDeps ++ perlDeps;
+          nativeBuildInputs = nativeBuildDeps;
+          buildInputs = buildDeps ++ propagatedDeps ++ awsDeps ++ perlDeps;
 
           inherit configureFlags;
 

local.mk

@@ -1,9 +1,3 @@
-ifeq ($(MAKECMDGOALS), dist)
-  dist-files += $(shell cat .dist-files)
-endif
-
-dist-files += configure config.h.in perl/configure
-
 clean-files += Makefile.config
 
 GLOBAL_CXXFLAGS += -Wno-deprecated-declarations

maintainers/upload-release.pl

@@ -133,20 +133,8 @@ for my $fn (glob "$tmpDir/*") {
 
 exit if $version =~ /pre/;
 
-# Update Nixpkgs in a very hacky way.
+# Update nix-fallback-paths.nix.
 system("cd $nixpkgsDir && git pull") == 0 or die;
-my $oldName = `nix-instantiate --eval $nixpkgsDir -A nix.name`; chomp $oldName;
-my $oldHash = `nix-instantiate --eval $nixpkgsDir -A nix.src.outputHash`; chomp $oldHash;
-print STDERR "old stable version in Nixpkgs = $oldName / $oldHash\n";
-
-my $fn = "$nixpkgsDir/pkgs/tools/package-management/nix/default.nix";
-my $oldFile = read_file($fn);
-$oldFile =~ s/$oldName/"$releaseName"/g;
-$oldFile =~ s/$oldHash/"$tarballHash"/g;
-write_file($fn, $oldFile);
-
-$oldName =~ s/nix-//g;
-$oldName =~ s/"//g;
 
 sub getStorePath {
     my ($jobName) = @_;
@@ -167,7 +155,7 @@ write_file("$nixpkgsDir/nixos/modules/installer/tools/nix-fallback-paths.nix",
            "  x86_64-darwin = \"" . getStorePath("build.x86_64-darwin") . "\";\n" .
            "}\n");
 
-system("cd $nixpkgsDir && git commit -a -m 'nix: $oldName -> $version'") == 0 or die;
+system("cd $nixpkgsDir && git commit -a -m 'nix-fallback-paths.nix: Update to $version'") == 0 or die;
 
 # Update the "latest" symlink.
 $channelsBucket->add_key(

misc/launchd/org.nixos.nix-daemon.plist.in

@@ -19,7 +19,7 @@
     <array>
       <string>/bin/sh</string>
       <string>-c</string>
-      <string>/bin/wait4path /nix/var/nix/profiles/default/bin/nix-daemon &amp;&amp; /nix/var/nix/profiles/default/bin/nix-daemon</string>
+      <string>/bin/wait4path /nix/var/nix/profiles/default/bin/nix-daemon &amp;&amp; exec /nix/var/nix/profiles/default/bin/nix-daemon</string>
     </array>
     <key>StandardErrorPath</key>
     <string>/var/log/nix-daemon.log</string>

misc/systemd/local.mk

@@ -2,4 +2,6 @@ ifeq ($(OS), Linux)
 
   $(foreach n, nix-daemon.socket nix-daemon.service, $(eval $(call install-file-in, $(d)/$(n), $(prefix)/lib/systemd/system, 0644)))
 
+  clean-files += $(d)/nix-daemon.socket $(d)/nix-daemon.service
+
 endif

misc/upstart/local.mk

@@ -2,4 +2,6 @@ ifeq ($(OS), Linux)
 
   $(foreach n, nix-daemon.conf, $(eval $(call install-file-in, $(d)/$(n), $(sysconfdir)/init, 0644)))
 
+  clean-files += $(d)/nix-daemon.conf
+
 endif

misc/zsh/completion.zsh

@@ -1,3 +1,5 @@
+#compdef nix
+
 function _nix() {
   local ifs_bk="$IFS"
   local input=("${(Q)words[@]}")
@@ -18,4 +20,4 @@ function _nix() {
   _describe 'nix' suggestions
 }
 
-compdef _nix nix
+_nix "$@"

misc/zsh/local.mk

@@ -0,0 +1 @@
+$(eval $(call install-file-as, $(d)/completion.zsh, $(datarootdir)/zsh/site-functions/_nix, 0644))

mk/dist.mk

@@ -1,17 +0,0 @@
-ifdef PACKAGE_NAME
-
-dist-name = $(PACKAGE_NAME)-$(PACKAGE_VERSION)
-
-dist: $(dist-name).tar.bz2 $(dist-name).tar.xz
-
-$(dist-name).tar.bz2: $(dist-files)
-	$(trace-gen) tar cfj $@ $(sort $(dist-files)) --transform 's,^,$(dist-name)/,'
-
-$(dist-name).tar.xz: $(dist-files)
-	$(trace-gen) tar cfJ $@ $(sort $(dist-files)) --transform 's,^,$(dist-name)/,'
-
-clean-files += $(dist-name).tar.bz2 $(dist-name).tar.xz
-
-print-top-help += echo "  dist: Generate a source distribution";
-
-endif

mk/jars.mk

@@ -1,36 +0,0 @@
-define build-jar
-
-  $(1)_NAME ?= $(1)
-
-  _d := $$(strip $$($(1)_DIR))
-
-  $(1)_PATH := $$(_d)/$$($(1)_NAME).jar
-
-  $(1)_TMPDIR := $$(_d)/.$$($(1)_NAME).jar.tmp
-
-  _jars := $$(foreach jar, $$($(1)_JARS), $$($$(jar)_PATH))
-
-  $$($(1)_PATH): $$($(1)_SOURCES) $$(_jars) $$($(1)_EXTRA_DEPS)| $$($(1)_ORDER_AFTER)
-	@rm -rf $$($(1)_TMPDIR)
-	@mkdir -p $$($(1)_TMPDIR)
-	$$(trace-javac) javac $(GLOBAL_JAVACFLAGS) $$($(1)_JAVACFLAGS) -d $$($(1)_TMPDIR) \
-	  $$(foreach fn, $$($(1)_SOURCES), '$$(fn)') \
-	  -cp "$$(subst $$(space),,$$(foreach jar,$$($(1)_JARS),$$($$(jar)_PATH):))$$$$CLASSPATH"
-	@echo -e '$$(subst $$(newline),\n,$$($(1)_MANIFEST))' > $$($(1)_PATH).manifest
-	$$(trace-jar) jar cfm $$($(1)_PATH) $$($(1)_PATH).manifest -C $$($(1)_TMPDIR) .
-	@rm $$($(1)_PATH).manifest
-	@rm -rf $$($(1)_TMPDIR)
-
-  $(1)_INSTALL_DIR ?= $$(jardir)
-
-  $(1)_INSTALL_PATH := $$($(1)_INSTALL_DIR)/$$($(1)_NAME).jar
-
-  $$(eval $$(call install-file-as, $$($(1)_PATH), $$($(1)_INSTALL_PATH), 0644))
-
-  install: $$($(1)_INSTALL_PATH)
-
-  jars-list += $$($(1)_PATH)
-
-  clean-files += $$($(1)_PATH)
-
-endef

mk/lib.mk

@@ -10,7 +10,6 @@ bin-scripts :=
 noinst-scripts :=
 man-pages :=
 install-tests :=
-dist-files :=
 OS = $(shell uname -s)
 
 
@@ -32,7 +31,6 @@ libdir ?= $(prefix)/lib
 bindir ?= $(prefix)/bin
 libexecdir ?= $(prefix)/libexec
 datadir ?= $(prefix)/share
-jardir ?= $(datadir)/java
 localstatedir ?= $(prefix)/var
 sysconfdir ?= $(prefix)/etc
 mandir ?= $(prefix)/share/man
@@ -75,7 +73,6 @@ BUILD_DEBUG ?= 1
 ifeq ($(BUILD_DEBUG), 1)
   GLOBAL_CFLAGS += -g
   GLOBAL_CXXFLAGS += -g
-  GLOBAL_JAVACFLAGS += -g
 endif
 
 
@@ -85,7 +82,6 @@ include mk/clean.mk
 include mk/install.mk
 include mk/libraries.mk
 include mk/programs.mk
-include mk/jars.mk
 include mk/patterns.mk
 include mk/templates.mk
 include mk/tests.mk
@@ -103,7 +99,6 @@ $(foreach mf, $(makefiles), $(eval $(call include-sub-makefile, $(mf))))
 # Instantiate stuff.
 $(foreach lib, $(libraries), $(eval $(call build-library,$(lib))))
 $(foreach prog, $(programs), $(eval $(call build-program,$(prog))))
-$(foreach jar, $(jars), $(eval $(call build-jar,$(jar))))
 $(foreach script, $(bin-scripts), $(eval $(call install-program-in,$(script),$(bindir))))
 $(foreach script, $(bin-scripts), $(eval programs-list += $(script)))
 $(foreach script, $(noinst-scripts), $(eval programs-list += $(script)))
@@ -112,12 +107,9 @@ $(foreach test, $(install-tests), $(eval $(call run-install-test,$(test))))
 $(foreach file, $(man-pages), $(eval $(call install-data-in, $(file), $(mandir)/man$(patsubst .%,%,$(suffix $(file))))))
 
 
-include mk/dist.mk
-
-
 .PHONY: default all man help
 
-all: $(programs-list) $(libs-list) $(jars-list) $(man-pages)
+all: $(programs-list) $(libs-list) $(man-pages)
 
 man: $(man-pages)
 
@@ -141,12 +133,6 @@ ifdef libs-list
 	@echo "The following libraries can be built:"
 	@echo ""
 	@for i in $(libs-list); do echo "  $$i"; done
-endif
-ifdef jars-list
-	@echo ""
-	@echo "The following JARs can be built:"
-	@echo ""
-	@for i in $(jars-list); do echo "  $$i"; done
 endif
 	@echo ""
 	@echo "The following variables control the build:"
@@ -157,4 +143,5 @@ endif
 	@echo "  CFLAGS: Flags for the C compiler"
 	@echo "  CXX ($(CXX)): C++ compiler to be used"
 	@echo "  CXXFLAGS: Flags for the C++ compiler"
+	@echo "  CPPFLAGS: C preprocessor flags, used for both CC and CXX"
 	@$(print-var-help)

mk/libraries.mk

@@ -159,5 +159,4 @@ define build-library
   libs-list += $$($(1)_PATH)
   endif
   clean-files += $$(_d)/*.a $$(_d)/*.$(SO_EXT) $$(_d)/*.o $$(_d)/.*.dep $$($(1)_DEPS) $$($(1)_OBJS)
-  dist-files += $$(_srcs)
 endef

mk/patterns.mk

@@ -1,11 +1,11 @@
 $(buildprefix)%.o: %.cc
 	@mkdir -p "$(dir $@)"
-	$(trace-cxx) $(CXX) -o $@ -c $< $(GLOBAL_CXXFLAGS_PCH) $(GLOBAL_CXXFLAGS) $(CXXFLAGS) $($@_CXXFLAGS) -MMD -MF $(call filename-to-dep, $@) -MP
+	$(trace-cxx) $(CXX) -o $@ -c $< $(CPPFLAGS) $(GLOBAL_CXXFLAGS_PCH) $(GLOBAL_CXXFLAGS) $(CXXFLAGS) $($@_CXXFLAGS) -MMD -MF $(call filename-to-dep, $@) -MP
 
 $(buildprefix)%.o: %.cpp
 	@mkdir -p "$(dir $@)"
-	$(trace-cxx) $(CXX) -o $@ -c $< $(GLOBAL_CXXFLAGS_PCH) $(GLOBAL_CXXFLAGS) $(CXXFLAGS) $($@_CXXFLAGS) -MMD -MF $(call filename-to-dep, $@) -MP
+	$(trace-cxx) $(CXX) -o $@ -c $< $(CPPFLAGS) $(GLOBAL_CXXFLAGS_PCH) $(GLOBAL_CXXFLAGS) $(CXXFLAGS) $($@_CXXFLAGS) -MMD -MF $(call filename-to-dep, $@) -MP
 
 $(buildprefix)%.o: %.c
 	@mkdir -p "$(dir $@)"
-	$(trace-cc) $(CC) -o $@ -c $< $(GLOBAL_CFLAGS) $(CFLAGS) $($@_CFLAGS) -MMD -MF $(call filename-to-dep, $@) -MP
+	$(trace-cc) $(CC) -o $@ -c $< $(CPPFLAGS) $(GLOBAL_CFLAGS) $(CFLAGS) $($@_CFLAGS) -MMD -MF $(call filename-to-dep, $@) -MP

mk/programs.mk

@@ -79,7 +79,6 @@ define build-program
 
   programs-list += $$($(1)_PATH)
   clean-files += $$($(1)_PATH) $$(_d)/*.o $$(_d)/.*.dep $$($(1)_DEPS) $$($(1)_OBJS)
-  dist-files += $$(_srcs)
 
   # Phony target to run this program (typically as a dependency of 'check').
   .PHONY: $(1)_RUN

mk/run_test.sh

@@ -14,7 +14,7 @@ if [ -t 1 ]; then
     yellow=""
     normal=""
 fi
-(cd $(dirname $1) && env ${TESTS_ENVIRONMENT} init.sh 2>/dev/null > /dev/null)
+(cd tests && env ${TESTS_ENVIRONMENT} init.sh 2>/dev/null > /dev/null)
 log="$(cd $(dirname $1) && env ${TESTS_ENVIRONMENT} $(basename $1) 2>&1)"
 status=$?
 if [ $status -eq 0 ]; then

mk/tests.mk

@@ -8,7 +8,7 @@ define run-install-test
 
   .PHONY: $1.test
   $1.test: $1 $(test-deps)
-	@env TEST_NAME=$(notdir $(basename $1)) TESTS_ENVIRONMENT="$(tests-environment)" mk/run_test.sh $1 < /dev/null
+	@env TEST_NAME=$(basename $1) TESTS_ENVIRONMENT="$(tests-environment)" mk/run_test.sh $1 < /dev/null
 
 endef
 

mk/tracing.mk

@@ -8,8 +8,6 @@ ifeq ($(V), 0)
   trace-ld      = @echo "  LD    " $@;
   trace-ar      = @echo "  AR    " $@;
   trace-install = @echo "  INST  " $@;
-  trace-javac   = @echo "  JAVAC " $@;
-  trace-jar     = @echo "  JAR   " $@;
   trace-mkdir   = @echo "  MKDIR " $@;
   trace-test    = @echo "  TEST  " $@;
 

nix-rust/local.mk

@@ -8,8 +8,13 @@ endif
 
 libnixrust_PATH := $(d)/target/$(RUST_DIR)/libnixrust.$(SO_EXT)
 libnixrust_INSTALL_PATH := $(libdir)/libnixrust.$(SO_EXT)
-libnixrust_LDFLAGS_USE := -L$(d)/target/$(RUST_DIR) -lnixrust -ldl
-libnixrust_LDFLAGS_USE_INSTALLED := -L$(libdir) -lnixrust -ldl
+libnixrust_LDFLAGS_USE := -L$(d)/target/$(RUST_DIR) -lnixrust
+libnixrust_LDFLAGS_USE_INSTALLED := -L$(libdir) -lnixrust
+
+ifeq ($(OS), Linux)
+libnixrust_LDFLAGS_USE += -ldl
+libnixrust_LDFLAGS_USE_INSTALLED += -ldl
+endif
 
 ifeq ($(OS), Darwin)
 libnixrust_BUILD_FLAGS = NIX_LDFLAGS="-undefined dynamic_lookup"
@@ -30,8 +35,6 @@ ifeq ($(OS), Darwin)
 	install_name_tool -id $@ $@
 endif
 
-dist-files += $(d)/vendor
-
 clean: clean-rust
 
 clean-rust:

perl/Makefile.config.in

@@ -2,7 +2,6 @@ CC = @CC@
 CFLAGS = @CFLAGS@
 CXX = @CXX@
 CXXFLAGS = @CXXFLAGS@
-HAVE_SODIUM = @HAVE_SODIUM@
 PACKAGE_NAME = @PACKAGE_NAME@
 PACKAGE_VERSION = @PACKAGE_VERSION@
 SODIUM_LIBS = @SODIUM_LIBS@

perl/configure.ac

@@ -40,11 +40,7 @@ AC_SUBST(perllibdir, [${libdir}/perl5/site_perl/$perlversion/$perlarchname])
 AC_MSG_RESULT($perllibdir)
 
 # Look for libsodium, an optional dependency.
-PKG_CHECK_MODULES([SODIUM], [libsodium],
-  [AC_DEFINE([HAVE_SODIUM], [1], [Whether to use libsodium for cryptography.])
-   CXXFLAGS="$SODIUM_CFLAGS $CXXFLAGS"
-   have_sodium=1], [have_sodium=])
-AC_SUBST(HAVE_SODIUM, [$have_sodium])
+PKG_CHECK_MODULES([SODIUM], [libsodium], [CXXFLAGS="$SODIUM_CFLAGS $CXXFLAGS"])
 
 # Check for the required Perl dependencies (DBI and DBD::SQLite).
 perlFlags="-I$perllibdir"

perl/lib/Nix/Store.xs

@@ -14,9 +14,7 @@
 #include "util.hh"
 #include "crypto.hh"
 
-#if HAVE_SODIUM
 #include <sodium.h>
-#endif
 
 
 using namespace nix;
@@ -110,10 +108,14 @@ SV * queryPathInfo(char * path, int base32)
             XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
             mXPUSHi(info->registrationTime);
             mXPUSHi(info->narSize);
-            AV * arr = newAV();
+            AV * refs = newAV();
             for (auto & i : info->references)
-                av_push(arr, newSVpv(store()->printStorePath(i).c_str(), 0));
-            XPUSHs(sv_2mortal(newRV((SV *) arr)));
+                av_push(refs, newSVpv(store()->printStorePath(i).c_str(), 0));
+            XPUSHs(sv_2mortal(newRV((SV *) refs)));
+            AV * sigs = newAV();
+            for (auto & i : info->sigs)
+                av_push(sigs, newSVpv(i.c_str(), 0));
+            XPUSHs(sv_2mortal(newRV((SV *) sigs)));
         } catch (Error & e) {
             croak("%s", e.what());
         }
@@ -235,12 +237,8 @@ SV * convertHash(char * algo, char * s, int toBase32)
 SV * signString(char * secretKey_, char * msg)
     PPCODE:
         try {
-#if HAVE_SODIUM
             auto sig = SecretKey(secretKey_).signDetached(msg);
             XPUSHs(sv_2mortal(newSVpv(sig.c_str(), sig.size())));
-#else
-            throw Error("Nix was not compiled with libsodium, required for signed binary cache support");
-#endif
         } catch (Error & e) {
             croak("%s", e.what());
         }
@@ -249,7 +247,6 @@ SV * signString(char * secretKey_, char * msg)
 int checkSignature(SV * publicKey_, SV * sig_, char * msg)
     CODE:
         try {
-#if HAVE_SODIUM
             STRLEN publicKeyLen;
             unsigned char * publicKey = (unsigned char *) SvPV(publicKey_, publicKeyLen);
             if (publicKeyLen != crypto_sign_PUBLICKEYBYTES)
@@ -261,9 +258,6 @@ int checkSignature(SV * publicKey_, SV * sig_, char * msg)
                 throw Error("signature is not valid");
 
             RETVAL = crypto_sign_verify_detached(sig, (unsigned char *) msg, strlen(msg), publicKey) == 0;
-#else
-            throw Error("Nix was not compiled with libsodium, required for signed binary cache support");
-#endif
         } catch (Error & e) {
             croak("%s", e.what());
         }

scripts/bigsur-nixbld-user-migration.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+
+((NEW_NIX_FIRST_BUILD_UID=301))
+
+id_available(){
+	dscl . list /Users UniqueID | grep -E '\b'$1'\b' >/dev/null
+}
+
+change_nixbld_names_and_ids(){
+	local name uid next_id
+	((next_id=NEW_NIX_FIRST_BUILD_UID))
+	echo "Attempting to migrate nixbld users."
+	echo "Each user should change from nixbld# to _nixbld#"
+	echo "and their IDs relocated to $next_id+"
+	while read -r name uid; do
+		echo "   Checking $name (uid: $uid)"
+		# iterate for a clean ID
+		while id_available "$next_id"; do
+			((next_id++))
+			if ((next_id >= 400)); then
+				echo "We've hit UID 400 without placing all of your users :("
+				echo "You should use the commands in this script as a starting"
+				echo "point to review your UID-space and manually move the"
+				echo "remaining users (or delete them, if you don't need them)."
+				exit 1
+			fi
+		done
+
+		if [[ $name == _* ]]; then
+			echo "      It looks like $name has already been renamed--skipping."
+		else
+			# first 3 are cleanup, it's OK if they aren't here
+			sudo dscl . delete /Users/$name dsAttrTypeNative:_writers_passwd &>/dev/null || true
+			sudo dscl . change /Users/$name NFSHomeDirectory "/private/var/empty 1" "/var/empty" &>/dev/null || true
+			# remove existing user from group
+			sudo dseditgroup -o edit -t user -d $name nixbld || true
+			sudo dscl . change /Users/$name UniqueID $uid $next_id
+			sudo dscl . change /Users/$name RecordName $name _$name
+			# add renamed user to group
+			sudo dseditgroup -o edit -t user -a _$name nixbld
+			echo "      $name migrated to _$name (uid: $next_id)"
+		fi
+	done < <(dscl . list /Users UniqueID | grep nixbld | sort -n -k2)
+}
+
+change_nixbld_names_and_ids

scripts/create-darwin-volume.sh

@@ -1,185 +1,839 @@
-#!/bin/sh
-set -e
+#!/usr/bin/env bash
+set -eu
+set -o pipefail
 
-root_disk() {
-    diskutil info -plist /
-}
+# I'm a little agnostic on the choices, but supporting a wide
+# slate of uses for now, including:
+# - import-only: `. create-darwin-volume.sh no-main[ ...]`
+# - legacy: `./create-darwin-volume.sh` or `. create-darwin-volume.sh`
+#   (both will run main())
+# - external alt-routine: `./create-darwin-volume.sh no-main func[ ...]`
+if [ "${1-}" = "no-main" ]; then
+    shift
+    readonly _CREATE_VOLUME_NO_MAIN=1
+else
+    readonly _CREATE_VOLUME_NO_MAIN=0
+    # declare some things we expect to inherit from install-multi-user
+    # I don't love this (because it's a bit of a kludge).
+    #
+    # CAUTION: (Dec 19 2020)
+    # This is a stopgap. It doesn't cover the full slate of
+    # identifiers we inherit--just those necessary to:
+    # - avoid breaking direct invocations of this script (here/now)
+    # - avoid hard-to-reverse structural changes before the call to rm
+    #   single-user support is verified
+    #
+    # In the near-mid term, I (personally) think we should:
+    # - decide to deprecate the direct call and add a notice
+    # - fold all of this into install-darwin-multi-user.sh
+    # - intentionally remove the old direct-invocation form (kill the
+    #   routine, replace this script w/ deprecation notice and a note
+    #   on the remove-after date)
+    #
+    readonly NIX_ROOT="${NIX_ROOT:-/nix}"
 
-apfs_volumes_for() {
-    disk=$1
-    diskutil apfs list -plist "$disk"
-}
-
-disk_identifier() {
-    xpath "/plist/dict/key[text()='ParentWholeDisk']/following-sibling::string[1]/text()" 2>/dev/null
-}
-
-volume_list_true() {
-    key=$1
-    xpath "/plist/dict/array/dict/key[text()='Volumes']/following-sibling::array/dict/key[text()='$key']/following-sibling::true[1]" 2> /dev/null
-}
-
-volume_get_string() {
-    key=$1 i=$2
-    xpath "/plist/dict/array/dict/key[text()='Volumes']/following-sibling::array/dict[$i]/key[text()='$key']/following-sibling::string[1]/text()" 2> /dev/null
-}
-
-find_nix_volume() {
-    disk=$1
-    i=1
-    volumes=$(apfs_volumes_for "$disk")
-    while true; do
-        name=$(echo "$volumes" | volume_get_string "Name" "$i")
-        if [ -z "$name" ]; then
-            break
+    _sudo() {
+        shift # throw away the 'explanation'
+        /usr/bin/sudo "$@"
+    }
+    failure() {
+        if [ "$*" = "" ]; then
+            cat
+        else
+            echo "$@"
+        fi
+        exit 1
+    }
+    task() {
+        echo "$@"
+    }
+fi
+
+# usually "disk1"
+root_disk_identifier() {
+    # For performance (~10ms vs 280ms) I'm parsing 'diskX' from stat output
+    # (~diskXsY)--but I'm retaining the more-semantic approach since
+    # it documents intent better.
+    # /usr/sbin/diskutil info -plist / | xmllint --xpath "/plist/dict/key[text()='ParentWholeDisk']/following-sibling::string[1]/text()" -
+    #
+    local special_device
+    special_device="$(/usr/bin/stat -f "%Sd" /)"
+    echo "${special_device%s[0-9]*}"
+}
+
+# make it easy to play w/ 'Case-sensitive APFS'
+readonly NIX_VOLUME_FS="${NIX_VOLUME_FS:-APFS}"
+readonly NIX_VOLUME_LABEL="${NIX_VOLUME_LABEL:-Nix Store}"
+# Strongly assuming we'll make a volume on the device / is on
+# But you can override NIX_VOLUME_USE_DISK to create it on some other device
+readonly NIX_VOLUME_USE_DISK="${NIX_VOLUME_USE_DISK:-$(root_disk_identifier)}"
+NIX_VOLUME_USE_SPECIAL="${NIX_VOLUME_USE_SPECIAL:-}"
+NIX_VOLUME_USE_UUID="${NIX_VOLUME_USE_UUID:-}"
+readonly NIX_VOLUME_MOUNTD_DEST="${NIX_VOLUME_MOUNTD_DEST:-/Library/LaunchDaemons/org.nixos.darwin-store.plist}"
+
+if /usr/bin/fdesetup isactive >/dev/null; then
+    test_filevault_in_use() { return 0; }
+    # no readonly; we may modify if user refuses from cure_volume
+    NIX_VOLUME_DO_ENCRYPT="${NIX_VOLUME_DO_ENCRYPT:-1}"
+else
+    test_filevault_in_use() { return 1; }
+    NIX_VOLUME_DO_ENCRYPT="${NIX_VOLUME_DO_ENCRYPT:-0}"
+fi
+
+should_encrypt_volume() {
+    test_filevault_in_use && (( NIX_VOLUME_DO_ENCRYPT == 1 ))
+}
+
+substep() {
+    printf "   %s\n" "" "- $1" "" "${@:2}"
+}
+
+
+volumes_labeled() {
+    local label="$1"
+    xsltproc --novalid --stringparam label "$label" - <(/usr/sbin/ioreg -ra -c "AppleAPFSVolume") <<'EOF'
+<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
+  <xsl:output method="text"/>
+  <xsl:template match="/">
+    <xsl:apply-templates select="/plist/array/dict/key[text()='IORegistryEntryName']/following-sibling::*[1][text()=$label]/.."/>
+  </xsl:template>
+  <xsl:template match="dict">
+    <xsl:apply-templates match="string" select="key[text()='BSD Name']/following-sibling::*[1]"/>
+    <xsl:text>=</xsl:text>
+    <xsl:apply-templates match="string" select="key[text()='UUID']/following-sibling::*[1]"/>
+    <xsl:text>&#xA;</xsl:text>
+  </xsl:template>
+</xsl:stylesheet>
+EOF
+    # I cut label out of the extracted values, but here it is for reference:
+    # <xsl:apply-templates match="string" select="key[text()='IORegistryEntryName']/following-sibling::*[1]"/>
+    # <xsl:text>=</xsl:text>
+}
+
+right_disk() {
+    local volume_special="$1" # (i.e., disk1s7)
+    [[ "$volume_special" == "$NIX_VOLUME_USE_DISK"s* ]]
+}
+
+right_volume() {
+    local volume_special="$1" # (i.e., disk1s7)
+    # if set, it must match; otherwise ensure it's on the right disk
+    if [ -z "$NIX_VOLUME_USE_SPECIAL" ]; then
+        if right_disk "$volume_special"; then
+            NIX_VOLUME_USE_SPECIAL="$volume_special" # latch on
+            return 0
+        else
+            return 1
+        fi
+    else
+        [ "$volume_special" = "$NIX_VOLUME_USE_SPECIAL" ]
+    fi
+}
+
+right_uuid() {
+    local volume_uuid="$1"
+    # if set, it must match; otherwise allow
+    if [ -z "$NIX_VOLUME_USE_UUID" ]; then
+        NIX_VOLUME_USE_UUID="$volume_uuid" # latch on
+        return 0
+    else
+        [ "$volume_uuid" = "$NIX_VOLUME_USE_UUID" ]
+    fi
+}
+
+cure_volumes() {
+    local found volume special uuid
+    # loop just in case they have more than one volume
+    # (nothing stops you from doing this)
+    for volume in $(volumes_labeled "$NIX_VOLUME_LABEL"); do
+        # CAUTION: this could (maybe) be a more normal read
+        # loop like:
+        #   while IFS== read -r special uuid; do
+        #       # ...
+        #   done <<<"$(volumes_labeled "$NIX_VOLUME_LABEL")"
+        #
+        # I did it with for to skirt a problem with the obvious
+        # pattern replacing stdin and causing user prompts
+        # inside (which also use read and access stdin) to skip
+        #
+        # If there's an existing encrypted volume we can't find
+        # in keychain, the user never gets prompted to delete
+        # the volume, and the install fails.
+        #
+        # If you change this, a human needs to test a very
+        # specific scenario: you already have an encrypted
+        # Nix Store volume, and have deleted its credential
+        # from keychain. Ensure the script asks you if it can
+        # delete the volume, and then prompts for your sudo
+        # password to confirm.
+        #
+        # shellcheck disable=SC1097
+        IFS== read -r special uuid <<< "$volume"
+        # take the first one that's on the right disk
+        if [ -z "${found:-}" ]; then
+            if right_volume "$special" && right_uuid "$uuid"; then
+                cure_volume "$special" "$uuid"
+                found="${special} (${uuid})"
+            else
+                warning <<EOF
+Ignoring ${special} (${uuid}) because I am looking for:
+disk=${NIX_VOLUME_USE_DISK} special=${NIX_VOLUME_USE_SPECIAL:-${NIX_VOLUME_USE_DISK}sX} uuid=${NIX_VOLUME_USE_UUID:-any}
+EOF
+                # TODO: give chance to delete if ! headless?
+            fi
+        else
+            warning <<EOF
+Ignoring ${special} (${uuid}), already found target: $found
+EOF
+            # TODO reminder? I feel like I want one
+            # idiom that reminds some warnings, or warns
+            # some reminders?
+            # TODO: if ! headless, chance to delete?
         fi
-        case "$name" in
-            [Nn]ix*)
-                echo "$name"
-                break
-                ;;
-        esac
-        i=$((i+1))
     done
+    if [ -z "${found:-}" ]; then
+        readonly NIX_VOLUME_USE_SPECIAL NIX_VOLUME_USE_UUID
+    fi
+}
+
+volume_encrypted() {
+    local volume_special="$1" # (i.e., disk1s7)
+    # Trying to match the first line of output; known first lines:
+    # No cryptographic users for <special>
+    # Cryptographic user for <special> (1 found)
+    # Cryptographic users for <special> (2 found)
+    /usr/sbin/diskutil apfs listCryptoUsers -plist "$volume_special" | /usr/bin/grep -q APFSCryptoUserUUID
 }
 
 test_fstab() {
-    grep -q "/nix apfs rw" /etc/fstab 2>/dev/null
+    /usr/bin/grep -q "$NIX_ROOT apfs rw" /etc/fstab 2>/dev/null
 }
 
-test_nix_symlink() {
-    [ -L "/nix" ] || grep -q "^nix." /etc/synthetic.conf 2>/dev/null
+test_nix_root_is_symlink() {
+    [ -L "$NIX_ROOT" ]
 }
 
-test_synthetic_conf() {
-    grep -q "^nix$" /etc/synthetic.conf 2>/dev/null
+test_synthetic_conf_either(){
+    /usr/bin/grep -qE "^${NIX_ROOT:1}($|\t.{3,}$)" /etc/synthetic.conf 2>/dev/null
+}
+
+test_synthetic_conf_mountable() {
+    /usr/bin/grep -q "^${NIX_ROOT:1}$" /etc/synthetic.conf 2>/dev/null
+}
+
+test_synthetic_conf_symlinked() {
+    /usr/bin/grep -qE "^${NIX_ROOT:1}\t.{3,}$" /etc/synthetic.conf 2>/dev/null
+}
+
+test_nix_volume_mountd_installed() {
+    test -e "$NIX_VOLUME_MOUNTD_DEST"
+}
+
+# current volume password
+test_keychain_by_uuid() {
+    local volume_uuid="$1"
+    # Note: doesn't need sudo just to check; doesn't output pw
+    security find-generic-password -s "$volume_uuid" &>/dev/null
+}
+
+get_volume_pass() {
+    local volume_uuid="$1"
+    _sudo \
+        "to confirm keychain has a password that unlocks this volume" \
+        security find-generic-password -s "$volume_uuid" -w
+}
+
+verify_volume_pass() {
+    local volume_special="$1" # (i.e., disk1s7)
+    local volume_uuid="$2"
+    /usr/sbin/diskutil apfs unlockVolume "$volume_special" -verify -stdinpassphrase -user "$volume_uuid"
+}
+
+volume_pass_works() {
+    local volume_special="$1" # (i.e., disk1s7)
+    local volume_uuid="$2"
+    get_volume_pass "$volume_uuid" | verify_volume_pass "$volume_special" "$volume_uuid"
+}
+
+# Create the paths defined in synthetic.conf, saving us a reboot.
+create_synthetic_objects() {
+    # Big Sur takes away the -B flag we were using and replaces it
+    # with a -t flag that appears to do the same thing (but they
+    # don't behave exactly the same way in terms of return values).
+    # This feels a little dirty, but as far as I can tell the
+    # simplest way to get the right one is to just throw away stderr
+    # and call both... :]
+    {
+        /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -t || true # Big Sur
+        /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -B || true # Catalina
+    } >/dev/null 2>&1
 }
 
 test_nix() {
-    test -d "/nix"
+    test -d "$NIX_ROOT"
 }
 
-test_t2_chip_present(){
-    # Use xartutil to see if system has a t2 chip.
-    #
-    # This isn't well-documented on its own; until it is,
-    # let's keep track of knowledge/assumptions.
-    #
-    # Warnings:
-    # - Don't search "xart" if porn will cause you trouble :)
-    # - Other xartutil flags do dangerous things. Don't run them
-    #   naively. If you must, search "xartutil" first.
-    #
-    # Assumptions:
-    # - the "xART session seeds recovery utility"
-    #   appears to interact with xartstorageremoted
-    # - `sudo xartutil --list` lists xART sessions
-    #   and their seeds and exits 0 if successful. If
-    #   not, it exits 1 and prints an error such as:
-    #   xartutil: ERROR: No supported link to the SEP present
-    # - xART sessions/seeds are present when a T2 chip is
-    #   (and not, otherwise)
-    # - the presence of a T2 chip means a newly-created
-    #   volume on the primary drive will be
-    #   encrypted at rest
-    # - all together: `sudo xartutil --list`
-    #   should exit 0 if a new Nix Store volume will
-    #   be encrypted at rest, and exit 1 if not.
-    sudo xartutil --list >/dev/null 2>/dev/null
+test_voldaemon() {
+    test -f "$NIX_VOLUME_MOUNTD_DEST"
 }
 
-test_filevault_in_use() {
-    disk=$1
-    #      list vols on disk | get value of Filevault key | value is true
-    apfs_volumes_for "$disk" | volume_list_true FileVault | grep -q true
+generate_mount_command() {
+    local cmd_type="$1" # encrypted|unencrypted
+    local volume_uuid mountpoint cmd=()
+    printf -v volume_uuid "%q" "$2"
+    printf -v mountpoint "%q" "$NIX_ROOT"
+
+    case "$cmd_type" in
+        encrypted)
+            cmd=(/bin/sh -c "/usr/bin/security find-generic-password -s '$volume_uuid' -w | /usr/sbin/diskutil apfs unlockVolume '$volume_uuid' -mountpoint '$mountpoint' -stdinpassphrase");;
+        unencrypted)
+            cmd=(/usr/sbin/diskutil mount -mountPoint "$mountpoint" "$volume_uuid");;
+        *)
+            failure "Invalid first arg $cmd_type to generate_mount_command";;
+    esac
+
+    printf "    <string>%s</string>\n" "${cmd[@]}"
 }
 
-# use after error msg for conditions we don't understand
-suggest_report_error(){
-    # ex "error: something sad happened :(" >&2
-    echo "       please report this @ https://github.com/nixos/nix/issues" >&2
+generate_mount_daemon() {
+    local cmd_type="$1" # encrypted|unencrypted
+    local volume_uuid="$2"
+    cat <<EOF
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+  <key>RunAtLoad</key>
+  <true/>
+  <key>Label</key>
+  <string>org.nixos.darwin-store</string>
+  <key>ProgramArguments</key>
+  <array>
+$(generate_mount_command "$cmd_type" "$volume_uuid")
+  </array>
+</dict>
+</plist>
+EOF
 }
 
-main() {
-    (
-        echo ""
-        echo "     ------------------------------------------------------------------ "
-        echo "    | This installer will create a volume for the nix store and        |"
-        echo "    | configure it to mount at /nix.  Follow these steps to uninstall. |"
-        echo "     ------------------------------------------------------------------ "
-        echo ""
-        echo "  1. Remove the entry from fstab using 'sudo vifs'"
-        echo "  2. Destroy the data volume using 'diskutil apfs deleteVolume'"
-        echo "  3. Remove the 'nix' line from /etc/synthetic.conf or the file"
-        echo ""
-    ) >&2
+_eat_bootout_err() {
+    /usr/bin/grep -v "Boot-out failed: 36: Operation now in progress"
+}
 
-    if test_nix_symlink; then
-        echo "error: /nix is a symlink, please remove it and make sure it's not in synthetic.conf (in which case a reboot is required)" >&2
-        echo "  /nix -> $(readlink "/nix")" >&2
-        exit 2
+# TODO: remove with --uninstall?
+uninstall_launch_daemon_directions() {
+    local daemon_label="$1" # i.e., org.nixos.blah-blah
+    local daemon_plist="$2" # abspath
+    substep "Uninstall LaunchDaemon $daemon_label" \
+      "  sudo launchctl bootout system/$daemon_label" \
+      "  sudo rm $daemon_plist"
+}
+
+uninstall_launch_daemon_prompt() {
+    local daemon_label="$1" # i.e., org.nixos.blah-blah
+    local daemon_plist="$2" # abspath
+    local reason_for_daemon="$3"
+    cat <<EOF
+
+The installer adds a LaunchDaemon to $reason_for_daemon: $daemon_label
+EOF
+    if ui_confirm "Can I remove it?"; then
+        _sudo "to terminate the daemon" \
+            launchctl bootout "system/$daemon_label" 2> >(_eat_bootout_err >&2) || true
+            # this can "fail" with a message like:
+            # Boot-out failed: 36: Operation now in progress
+        _sudo "to remove the daemon definition" rm "$daemon_plist"
+    fi
+}
+
+nix_volume_mountd_uninstall_directions() {
+    uninstall_launch_daemon_directions "org.nixos.darwin-store" \
+        "$NIX_VOLUME_MOUNTD_DEST"
+}
+
+nix_volume_mountd_uninstall_prompt() {
+    uninstall_launch_daemon_prompt "org.nixos.darwin-store" \
+        "$NIX_VOLUME_MOUNTD_DEST" \
+        "mount your Nix volume"
+}
+
+# TODO: move nix_daemon to install-darwin-multi-user if/when uninstall_launch_daemon_prompt moves up to install-multi-user
+nix_daemon_uninstall_prompt() {
+    uninstall_launch_daemon_prompt "org.nixos.nix-daemon" \
+        "$NIX_DAEMON_DEST" \
+        "run the nix-daemon"
+}
+
+# TODO: remove with --uninstall?
+nix_daemon_uninstall_directions() {
+    uninstall_launch_daemon_directions "org.nixos.nix-daemon" \
+        "$NIX_DAEMON_DEST"
+}
+
+
+# TODO: remove with --uninstall?
+synthetic_conf_uninstall_directions() {
+    # :1 to strip leading slash
+    substep "Remove ${NIX_ROOT:1} from /etc/synthetic.conf" \
+        "  If nix is the only entry: sudo rm /etc/synthetic.conf" \
+        "  Otherwise: sudo /usr/bin/sed -i '' -e '/^${NIX_ROOT:1}$/d' /etc/synthetic.conf"
+}
+
+synthetic_conf_uninstall_prompt() {
+    cat <<EOF
+
+During install, I add '${NIX_ROOT:1}' to /etc/synthetic.conf, which instructs
+macOS to create an empty root directory for mounting the Nix volume.
+EOF
+    # make the edit to a copy
+    /usr/bin/grep -vE "^${NIX_ROOT:1}($|\t.{3,}$)" /etc/synthetic.conf > "$SCRATCH/synthetic.conf.edit"
+
+    if test_synthetic_conf_symlinked; then
+        warning <<EOF
+
+/etc/synthetic.conf already contains a line instructing your system
+to make '${NIX_ROOT}' as a symlink:
+    $(/usr/bin/grep -nE "^${NIX_ROOT:1}\t.{3,}$" /etc/synthetic.conf)
+
+This may mean your system has/had a non-standard Nix install.
+
+The volume-creation process in this installer is *not* compatible
+with a symlinked store, so I'll have to remove this instruction to
+continue.
+
+If you want/need to keep this instruction, answer 'n' to abort.
+
+EOF
     fi
 
-    if ! test_synthetic_conf; then
-        echo "Configuring /etc/synthetic.conf..." >&2
-        echo nix | sudo tee -a /etc/synthetic.conf
-        if ! test_synthetic_conf; then
-            echo "error: failed to configure synthetic.conf;" >&2
-            suggest_report_error
-            exit 1
+    # ask to rm if this left the file empty aside from comments, else edit
+    if /usr/bin/diff -q <(:) <(/usr/bin/grep -v "^#" "$SCRATCH/synthetic.conf.edit") &>/dev/null; then
+        if confirm_rm "/etc/synthetic.conf"; then
+            if test_nix_root_is_symlink; then
+                failure >&2 <<EOF
+I removed /etc/synthetic.conf, but $NIX_ROOT is already a symlink
+(-> $(readlink "$NIX_ROOT")). The system should remove it when you reboot.
+Once you've rebooted, run the installer again.
+EOF
+            fi
+            return 0
+        fi
+    else
+        if confirm_edit "$SCRATCH/synthetic.conf.edit" "/etc/synthetic.conf"; then
+            if test_nix_root_is_symlink; then
+                failure >&2 <<EOF
+I edited Nix out of /etc/synthetic.conf, but $NIX_ROOT is already a symlink
+(-> $(readlink "$NIX_ROOT")). The system should remove it when you reboot.
+Once you've rebooted, run the installer again.
+EOF
+            fi
+            return 0
         fi
     fi
+    # fallback instructions
+    echo "Manually remove nix from /etc/synthetic.conf"
+    return 1
+}
 
-    if ! test_nix; then
-        echo "Creating mountpoint for /nix..." >&2
-        /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -B || true
-        if ! test_nix; then
-            sudo mkdir -p /nix 2>/dev/null || true
+add_nix_vol_fstab_line() {
+    local uuid="$1"
+    # shellcheck disable=SC1003,SC2026
+    local escaped_mountpoint="${NIX_ROOT/ /'\\\'040}"
+    shift
+    EDITOR="/usr/bin/ex" _sudo "to add nix to fstab" "$@" <<EOF
+:a
+UUID=$uuid $escaped_mountpoint apfs rw,noauto,nobrowse,suid,owners
+.
+:x
+EOF
+    # TODO: preserving my notes on suid,owners above until resolved
+    # There *may* be some issue regarding volume ownership, see nix#3156
+    #
+    # It seems like the cheapest fix is adding "suid,owners" to fstab, but:
+    # - We don't have much info on this condition yet
+    # - I'm not certain if these cause other problems?
+    # - There's a "chown" component some people claim to need to fix this
+    #   that I don't understand yet
+    #   (Note however that I've had to add a chown step to handle
+    #   single->multi-user reinstalls, which may cover this)
+    #
+    # I'm not sure if it's safe to approach this way?
+    #
+    # I think I think the most-proper way to test for it is:
+    # diskutil info -plist "$NIX_VOLUME_LABEL" | xmllint --xpath "(/plist/dict/key[text()='GlobalPermissionsEnabled'])/following-sibling::*[1][name()='true']" -; echo $?
+    #
+    # There's also `sudo /usr/sbin/vsdbutil -c /path` (which is much faster, but is also
+    # deprecated and needs minor parsing).
+    #
+    # If no one finds a problem with doing so, I think the simplest approach
+    # is to just eagerly set this. I found a few imperative approaches:
+    # (diskutil enableOwnership, ~100ms), a cheap one (/usr/sbin/vsdbutil -a, ~40-50ms),
+    # a very cheap one (append the internal format to /var/db/volinfo.database).
+    #
+    # But vsdbutil's deprecation notice suggests using fstab, so I want to
+    # give that a whirl first.
+    #
+    # TODO: when this is workable, poke infinisil about reproducing the issue
+    # and confirming this fix?
+}
+
+delete_nix_vol_fstab_line() {
+    # TODO: I'm scaffolding this to handle the new nix volumes
+    # but it might be nice to generalize a smidge further to
+    # go ahead and set up a pattern for curing "old" things
+    # we no longer do?
+    EDITOR="/usr/bin/patch" _sudo "to cut nix from fstab" "$@" < <(/usr/bin/diff /etc/fstab <(/usr/bin/grep -v "$NIX_ROOT apfs rw" /etc/fstab))
+    # leaving some parts out of the grep; people may fiddle this a little?
+}
+
+# TODO: hope to remove with --uninstall
+fstab_uninstall_directions() {
+    substep "Remove ${NIX_ROOT} from /etc/fstab" \
+      "  If nix is the only entry: sudo rm /etc/fstab" \
+      "  Otherwise, run 'sudo /usr/sbin/vifs' to remove the nix line"
+}
+
+fstab_uninstall_prompt() {
+    cat <<EOF
+During install, I add '${NIX_ROOT}' to /etc/fstab so that macOS knows what
+mount options to use for the Nix volume.
+EOF
+    cp /etc/fstab "$SCRATCH/fstab.edit"
+    # technically doesn't need the _sudo path, but throwing away the
+    # output is probably better than mostly-duplicating the code...
+    delete_nix_vol_fstab_line patch "$SCRATCH/fstab.edit" &>/dev/null
+
+    # if the patch test edit, minus comment lines, is equal to empty (:)
+    if /usr/bin/diff -q <(:) <(/usr/bin/grep -v "^#" "$SCRATCH/fstab.edit") &>/dev/null; then
+        # this edit would leave it empty; propose deleting it
+        if confirm_rm "/etc/fstab"; then
+            return 0
+        else
+            echo "Remove nix from /etc/fstab (or remove the file)"
         fi
-        if ! test_nix; then
-            echo "error: failed to bootstrap /nix; if a reboot doesn't help," >&2
-            suggest_report_error
-            exit 1
+    else
+        echo "I might be able to help you make this edit. Here's the diff:"
+        if ! _diff "/etc/fstab" "$SCRATCH/fstab.edit" && ui_confirm "Does the change above look right?"; then
+            delete_nix_vol_fstab_line /usr/sbin/vifs
+        else
+            echo "Remove nix from /etc/fstab (or remove the file)"
         fi
     fi
+}
 
-    disk=$(root_disk | disk_identifier)
-    volume=$(find_nix_volume "$disk")
-    if [ -z "$volume" ]; then
-        echo "Creating a Nix Store volume..." >&2
+remove_volume() {
+    local volume_special="$1" # (i.e., disk1s7)
+    _sudo "to unmount the Nix volume" \
+        /usr/sbin/diskutil unmount force "$volume_special" || true # might not be mounted
+    _sudo "to delete the Nix volume" \
+        /usr/sbin/diskutil apfs deleteVolume "$volume_special"
+}
 
-        if test_filevault_in_use "$disk"; then
-            # TODO: Not sure if it's in-scope now, but `diskutil apfs list`
-            # shows both filevault and encrypted at rest status, and it
-            # may be the more semantic way to test for this? It'll show
-            # `FileVault:                 No (Encrypted at rest)`
-            # `FileVault:                 No`
-            # `FileVault:                 Yes (Unlocked)`
-            # and so on.
-            if test_t2_chip_present; then
-                echo "warning: boot volume is FileVault-encrypted, but the Nix store volume" >&2
-                echo "         is only encrypted at rest." >&2
-                echo "         See https://nixos.org/nix/manual/#sect-macos-installation" >&2
+# aspiration: robust enough to both fix problems
+# *and* update older darwin volumes
+cure_volume() {
+    local volume_special="$1" # (i.e., disk1s7)
+    local volume_uuid="$2"
+    header "Found existing Nix volume"
+    row "  special" "$volume_special"
+    row "     uuid" "$volume_uuid"
+
+    if volume_encrypted "$volume_special"; then
+        row "encrypted" "yes"
+        if volume_pass_works "$volume_special" "$volume_uuid"; then
+            NIX_VOLUME_DO_ENCRYPT=0
+            ok "Found a working decryption password in keychain :)"
+            echo ""
+        else
+            # - this is a volume we made, and
+            #   - the user encrypted it on their own
+            #   - something deleted the credential
+            # - this is an old or BYO volume and the pw
+            #   just isn't somewhere we can find it.
+            #
+            # We're going to explain why we're freaking out
+            # and prompt them to either delete the volume
+            # (requiring a sudo auth), or abort to fix
+            warning <<EOF
+
+This volume is encrypted, but I don't see a password to decrypt it.
+The quick fix is to let me delete this volume and make you a new one.
+If that's okay, enter your (sudo) password to continue. If not, you
+can ensure the decryption password is in your system keychain with a
+"Where" (service) field set to this volume's UUID:
+  $volume_uuid
+EOF
+            if password_confirm "delete this volume"; then
+                remove_volume "$volume_special"
             else
-                echo "error: refusing to create Nix store volume because the boot volume is" >&2
-                echo "       FileVault encrypted, but encryption-at-rest is not available." >&2
-                echo "       Manually create a volume for the store and re-run this script." >&2
-                echo "       See https://nixos.org/nix/manual/#sect-macos-installation" >&2
-                exit 1
+                # TODO: this is a good design case for a warn-and
+                # remind idiom...
+                failure <<EOF
+Your Nix volume is encrypted, but I couldn't find its password. Either:
+- Delete or rename the volume out of the way
+- Ensure its decryption password is in the system keychain with a
+  "Where" (service) field set to this volume's UUID:
+    $volume_uuid
+EOF
+            fi
+        fi
+    elif test_filevault_in_use; then
+        row "encrypted" "no"
+        warning <<EOF
+FileVault is on, but your $NIX_VOLUME_LABEL volume isn't encrypted.
+EOF
+        # if we're interactive, give them a chance to
+        # encrypt the volume. If not, /shrug
+        if ! headless && (( NIX_VOLUME_DO_ENCRYPT == 1 )); then
+            if ui_confirm "Should I encrypt it and add the decryption key to your keychain?"; then
+                encrypt_volume "$volume_uuid" "$NIX_VOLUME_LABEL"
+                NIX_VOLUME_DO_ENCRYPT=0
+            else
+                NIX_VOLUME_DO_ENCRYPT=0
+                reminder "FileVault is on, but your $NIX_VOLUME_LABEL volume isn't encrypted."
             fi
         fi
-
-        sudo diskutil apfs addVolume "$disk" APFS 'Nix Store' -mountpoint /nix
-        volume="Nix Store"
     else
-        echo "Using existing '$volume' volume" >&2
-    fi
-
-    if ! test_fstab; then
-        echo "Configuring /etc/fstab..." >&2
-        label=$(echo "$volume" | sed 's/ /\\040/g')
-        printf "\$a\nLABEL=%s /nix apfs rw,nobrowse\n.\nwq\n" "$label" | EDITOR=ed sudo vifs
+        row "encrypted" "no"
     fi
 }
 
-main "$@"
+remove_volume_artifacts() {
+    if test_synthetic_conf_either; then
+        # NIX_ROOT is in synthetic.conf
+        if synthetic_conf_uninstall_prompt; then
+            # TODO: moot until we tackle uninstall, but when we're
+            # actually uninstalling, we should issue:
+            # reminder "macOS will clean up the empty mount-point directory at $NIX_ROOT on reboot."
+            :
+        fi
+    fi
+    if test_fstab; then
+        fstab_uninstall_prompt
+    fi
+
+    if test_nix_volume_mountd_installed; then
+        nix_volume_mountd_uninstall_prompt
+    fi
+}
+
+setup_synthetic_conf() {
+    if test_nix_root_is_symlink; then
+        if ! test_synthetic_conf_symlinked; then
+            failure >&2 <<EOF
+error: $NIX_ROOT is a symlink (-> $(readlink "$NIX_ROOT")).
+Please remove it. If nix is in /etc/synthetic.conf, remove it and reboot.
+EOF
+        fi
+    fi
+    if ! test_synthetic_conf_mountable; then
+        task "Configuring /etc/synthetic.conf to make a mount-point at $NIX_ROOT" >&2
+        # technically /etc/synthetic.d/nix is supported in Big Sur+
+        # but handling both takes even more code...
+        _sudo "to add Nix to /etc/synthetic.conf" \
+            /usr/bin/ex /etc/synthetic.conf <<EOF
+:a
+${NIX_ROOT:1}
+.
+:x
+EOF
+        if ! test_synthetic_conf_mountable; then
+            failure "error: failed to configure synthetic.conf" >&2
+        fi
+        create_synthetic_objects
+        if ! test_nix; then
+            failure >&2 <<EOF
+error: failed to bootstrap $NIX_ROOT
+If you enabled FileVault after booting, this is likely a known issue
+with macOS that you'll have to reboot to fix. If you didn't enable FV,
+though, please open an issue describing how the system that you see
+this error on was set up.
+EOF
+        fi
+    fi
+}
+
+setup_fstab() {
+    local volume_uuid="$1"
+    # fstab used to be responsible for mounting the volume. Now the last
+    # step adds a LaunchDaemon responsible for mounting. This is technically
+    # redundant for mounting, but diskutil appears to pick up mount options
+    # from fstab (and diskutil's support for specifying them directly is not
+    # consistent across versions/subcommands).
+    if ! test_fstab; then
+        task "Configuring /etc/fstab to specify volume mount options" >&2
+        add_nix_vol_fstab_line "$volume_uuid" /usr/sbin/vifs
+    fi
+}
+
+encrypt_volume() {
+    local volume_uuid="$1"
+    local volume_label="$2"
+    local password
+    # Note: mount/unmount are late additions to support the right order
+    # of operations for creating the volume and then baking its uuid into
+    # other artifacts; not as well-trod wrt to potential errors, race
+    # conditions, etc.
+
+    /usr/sbin/diskutil mount "$volume_label"
+
+    password="$(/usr/bin/xxd -l 32 -p -c 256 /dev/random)"
+    _sudo "to add your Nix volume's password to Keychain" \
+        /usr/bin/security -i <<EOF
+add-generic-password -a "$volume_label" -s "$volume_uuid" -l "$volume_label encryption password" -D "Encrypted volume password" -j "Added automatically by the Nix installer for use by $NIX_VOLUME_MOUNTD_DEST" -w "$password" -T /System/Library/CoreServices/APFSUserAgent -T /System/Library/CoreServices/CSUserAgent -T /usr/bin/security "/Library/Keychains/System.keychain"
+EOF
+    builtin printf "%s" "$password" | _sudo "to encrypt your Nix volume" \
+        /usr/sbin/diskutil apfs encryptVolume "$volume_label" -user disk -stdinpassphrase
+
+    /usr/sbin/diskutil unmount force "$volume_label"
+}
+
+create_volume() {
+    # Notes:
+    # 1) using `-nomount` instead of `-mountpoint "$NIX_ROOT"` to get
+    # its UUID and set mount opts in fstab before first mount
+    #
+    # 2) system is in some sense less secure than user keychain... (it's
+    # possible to read the password for decrypting the keychain) but
+    # the user keychain appears to be available too late. As far as I
+    # can tell, the file with this password (/var/db/SystemKey) is
+    # inside the FileVault envelope. If that isn't true, it may make
+    # sense to store the password inside the envelope?
+    #
+    # 3) At some point it would be ideal to have a small binary to serve
+    # as the daemon itself, and for it to replace /usr/bin/security here.
+    #
+    # 4) *UserAgent exemptions should let the system seamlessly supply the
+    # password if noauto is removed from fstab entry. This is intentional;
+    # the user will hopefully look for help if the volume stops mounting,
+    # rather than failing over into subtle race-condition problems.
+    #
+    # 5) If we ever get users griping about not having space to do
+    # anything useful with Nix, it is possibly to specify
+    # `-reserve 10g` or something, which will fail w/o that much
+    #
+    # 6) getting special w/ awk may be fragile, but doing it to:
+    #    - save time over running slow diskutil commands
+    #    - skirt risk we grab wrong volume if multiple match
+    /usr/sbin/diskutil apfs addVolume "$NIX_VOLUME_USE_DISK" "$NIX_VOLUME_FS" "$NIX_VOLUME_LABEL" -nomount | /usr/bin/awk '/Created new APFS Volume/ {print $5}'
+}
+
+volume_uuid_from_special() {
+    local volume_special="$1" # (i.e., disk1s7)
+    # For reasons I won't pretend to fathom, this returns 253 when it works
+    /System/Library/Filesystems/apfs.fs/Contents/Resources/apfs.util -k "$volume_special" || true
+}
+
+# this sometimes clears immediately, and AFAIK clears
+# within about 1s. diskutil info on an unmounted path
+# fails in around 50-100ms and a match takes about
+# 250-300ms. I suspect it's usually ~250-750ms
+await_volume() {
+    # caution: this could, in theory, get stuck
+    until /usr/sbin/diskutil info "$NIX_ROOT" &>/dev/null; do
+        :
+    done
+}
+
+setup_volume() {
+    local use_special use_uuid profile_packages
+    task "Creating a Nix volume" >&2
+    # DOING: I'm tempted to wrap this call in a grep to get the new disk special without doing anything too complex, but this sudo wrapper *is* a little complex, so it'll be a PITA unless maybe we can skip sudo on this. Let's just try it without.
+
+    use_special="${NIX_VOLUME_USE_SPECIAL:-$(create_volume)}"
+
+    use_uuid=${NIX_VOLUME_USE_UUID:-$(volume_uuid_from_special "$use_special")}
+
+    setup_fstab "$use_uuid"
+
+    if should_encrypt_volume; then
+        encrypt_volume "$use_uuid" "$NIX_VOLUME_LABEL"
+        setup_volume_daemon "encrypted" "$use_uuid"
+    # TODO: might be able to save ~60ms by caching or setting
+    # this somewhere rather than re-checking here.
+    elif volume_encrypted "$use_special"; then
+        setup_volume_daemon "encrypted" "$use_uuid"
+    else
+        setup_volume_daemon "unencrypted" "$use_uuid"
+    fi
+
+    await_volume
+
+    # TODO: below is a vague kludge for now; I just don't know
+    # what if any safe action there is to take here. Also, the
+    # reminder isn't very helpful.
+    # I'm less sure where this belongs, but it also wants mounted, pre-install
+    if type -p nix-env; then
+        profile_packages="$(nix-env --query --installed)"
+        # TODO: can probably do below faster w/ read
+        # intentionally unquoted string to eat whitespace in wc output
+        # shellcheck disable=SC2046,SC2059
+        if ! [ $(printf "$profile_packages" | /usr/bin/wc -l) = "0" ]; then
+            reminder <<EOF
+Nix now supports only multi-user installs on Darwin/macOS, and your user's
+Nix profile has some packages in it. These packages may obscure those in the
+default profile, including the Nix this installer will add. You should
+review these packages:
+$profile_packages
+EOF
+        fi
+    fi
+
+}
+
+setup_volume_daemon() {
+    local cmd_type="$1" # encrypted|unencrypted
+    local volume_uuid="$2"
+    if ! test_voldaemon; then
+        task "Configuring LaunchDaemon to mount '$NIX_VOLUME_LABEL'" >&2
+        _sudo "to install the Nix volume mounter" /usr/bin/ex "$NIX_VOLUME_MOUNTD_DEST" <<EOF
+:a
+$(generate_mount_daemon "$cmd_type" "$volume_uuid")
+.
+:x
+EOF
+
+        # TODO: should probably alert the user if this is disabled?
+        _sudo "to launch the Nix volume mounter" \
+            launchctl bootstrap system "$NIX_VOLUME_MOUNTD_DEST" || true
+        # TODO: confirm whether kickstart is necessesary?
+        # I feel a little superstitous, but it can guard
+        # against multiple problems (doesn't start, old
+        # version still running for some reason...)
+        _sudo "to launch the Nix volume mounter" \
+            launchctl kickstart -k system/org.nixos.darwin-store
+    fi
+}
+
+setup_darwin_volume() {
+    setup_synthetic_conf
+    setup_volume
+}
+
+if [ "$_CREATE_VOLUME_NO_MAIN" = 1 ]; then
+    if [ -n "$*" ]; then
+        "$@" # expose functions in case we want multiple routines?
+    fi
+else
+    # no reason to pay for bash to process this
+    main() {
+        {
+            echo ""
+            echo "     ------------------------------------------------------------------ "
+            echo "    | This installer will create a volume for the nix store and        |"
+            echo "    | configure it to mount at $NIX_ROOT.  Follow these steps to uninstall. |"
+            echo "     ------------------------------------------------------------------ "
+            echo ""
+            echo "  1. Remove the entry from fstab using 'sudo /usr/sbin/vifs'"
+            echo "  2. Run 'sudo launchctl bootout system/org.nixos.darwin-store'"
+            echo "  3. Remove $NIX_VOLUME_MOUNTD_DEST"
+            echo "  4. Destroy the data volume using '/usr/sbin/diskutil apfs deleteVolume'"
+            echo "  5. Remove the 'nix' line from /etc/synthetic.conf (or the file)"
+            echo ""
+        } >&2
+
+        setup_darwin_volume
+    }
+
+    main "$@"
+fi

scripts/install-darwin-multi-user.sh

@@ -3,50 +3,99 @@
 set -eu
 set -o pipefail
 
-readonly PLIST_DEST=/Library/LaunchDaemons/org.nixos.nix-daemon.plist
+readonly NIX_DAEMON_DEST=/Library/LaunchDaemons/org.nixos.nix-daemon.plist
+# create by default; set 0 to DIY, use a symlink, etc.
+readonly NIX_VOLUME_CREATE=${NIX_VOLUME_CREATE:-1} # now default
+NIX_FIRST_BUILD_UID="301"
+NIX_BUILD_USER_NAME_TEMPLATE="_nixbld%d"
+
+# caution: may update times on / if not run as normal non-root user
+read_only_root() {
+    # this touch command ~should~ always produce an error
+    # as of this change I confirmed /usr/bin/touch emits:
+    # "touch: /: Read-only file system" Catalina+ and Big Sur
+    # "touch: /: Permission denied" Mojave
+    # (not matching prefix for compat w/ coreutils touch in case using
+    # an explicit path causes problems; its prefix differs)
+    [[ "$(/usr/bin/touch / 2>&1)" = *"Read-only file system" ]]
+
+    # Avoiding the slow semantic way to get this information (~330ms vs ~8ms)
+    # unless using touch causes problems. Just in case, that approach is:
+    # diskutil info -plist / | <find the Writable or WritableVolume keys>, i.e.
+    # diskutil info -plist / | xmllint --xpath "name(/plist/dict/key[text()='Writable']/following-sibling::*[1])" -
+}
+
+if read_only_root && [ "$NIX_VOLUME_CREATE" = 1 ]; then
+    should_create_volume() { return 0; }
+else
+    should_create_volume() { return 1; }
+fi
+
+# shellcheck source=./create-darwin-volume.sh
+. "$EXTRACTED_NIX_PATH/create-darwin-volume.sh" "no-main"
 
 dsclattr() {
     /usr/bin/dscl . -read "$1" \
-        | awk "/$2/ { print \$2 }"
+        | /usr/bin/awk "/$2/ { print \$2 }"
 }
 
-poly_validate_assumptions() {
-    if [ "$(uname -s)" != "Darwin" ]; then
-        failure "This script is for use with macOS!"
+test_nix_daemon_installed() {
+  test -e "$NIX_DAEMON_DEST"
+}
+
+poly_cure_artifacts() {
+    if should_create_volume; then
+        task "Fixing any leftover Nix volume state"
+        cat <<EOF
+Before I try to install, I'll check for any existing Nix volume config
+and ask for your permission to remove it (so that the installer can
+start fresh). I'll also ask for permission to fix any issues I spot.
+EOF
+        cure_volumes
+        remove_volume_artifacts
     fi
 }
 
 poly_service_installed_check() {
-    [ -e "$PLIST_DEST" ]
+    if should_create_volume; then
+        test_nix_daemon_installed || test_nix_volume_mountd_installed
+    else
+        test_nix_daemon_installed
+    fi
 }
 
 poly_service_uninstall_directions() {
-        cat <<EOF
-$1. Delete $PLIST_DEST
-
-  sudo launchctl unload $PLIST_DEST
-  sudo rm $PLIST_DEST
-
-EOF
+    echo "$1. Remove macOS-specific components:"
+    if should_create_volume && test_nix_volume_mountd_installed; then
+        darwin_volume_uninstall_directions
+    fi
+    if test_nix_daemon_installed; then
+        nix_daemon_uninstall_directions
+    fi
 }
 
 poly_service_setup_note() {
-    cat <<EOF
- - load and start a LaunchDaemon (at $PLIST_DEST) for nix-daemon
+    if should_create_volume; then
+        echo " - create a Nix volume and a LaunchDaemon to mount it"
+    fi
+    echo " - create a LaunchDaemon (at $NIX_DAEMON_DEST) for nix-daemon"
+    echo ""
+}
 
-EOF
+poly_extra_try_me_commands() {
+    :
 }
 
 poly_configure_nix_daemon_service() {
+    task "Setting up the nix-daemon LaunchDaemon"
     _sudo "to set up the nix-daemon as a LaunchDaemon" \
-          cp -f "/nix/var/nix/profiles/default$PLIST_DEST" "$PLIST_DEST"
+          /bin/cp -f "/nix/var/nix/profiles/default$NIX_DAEMON_DEST" "$NIX_DAEMON_DEST"
 
     _sudo "to load the LaunchDaemon plist for nix-daemon" \
           launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist
 
     _sudo "to start the nix-daemon" \
-          launchctl start org.nixos.nix-daemon
-
+          launchctl kickstart -k system/org.nixos.nix-daemon
 }
 
 poly_group_exists() {
@@ -87,6 +136,8 @@ poly_user_home_get() {
 }
 
 poly_user_home_set() {
+    # This can trigger a permission prompt now:
+    # "Terminal" would like to administer your computer. Administration can include modifying passwords, networking, and system settings.
     _sudo "in order to give $1 a safe home directory" \
           /usr/bin/dscl . -create "/Users/$1" "NFSHomeDirectory" "$2"
 }
@@ -112,7 +163,7 @@ poly_user_shell_set() {
 poly_user_in_group_check() {
     username=$1
     group=$2
-    dseditgroup -o checkmember -m "$username" "$group" > /dev/null 2>&1
+    /usr/sbin/dseditgroup -o checkmember -m "$username" "$group" > /dev/null 2>&1
 }
 
 poly_user_in_group_set() {
@@ -142,3 +193,17 @@ poly_create_build_user() {
           /usr/bin/dscl . create "/Users/$username" \
           UniqueID "${uid}"
 }
+
+poly_prepare_to_install() {
+    if should_create_volume; then
+        header "Preparing a Nix volume"
+        # intentional indent below to match task indent
+        cat <<EOF
+    Nix traditionally stores its data in the root directory $NIX_ROOT, but
+    macOS now (starting in 10.15 Catalina) has a read-only root directory.
+    To support Nix, I will create a volume and configure macOS to mount it
+    at $NIX_ROOT.
+EOF
+        setup_darwin_volume
+    fi
+}

scripts/install-multi-user.sh

@@ -25,13 +25,15 @@ readonly RED='\033[31m'
 readonly NIX_USER_COUNT=${NIX_USER_COUNT:-32}
 readonly NIX_BUILD_GROUP_ID="30000"
 readonly NIX_BUILD_GROUP_NAME="nixbld"
-readonly NIX_FIRST_BUILD_UID="30001"
+# darwin installer needs to override these
+NIX_FIRST_BUILD_UID="30001"
+NIX_BUILD_USER_NAME_TEMPLATE="nixbld%d"
 # Please don't change this. We don't support it, because the
 # default shell profile that comes with Nix doesn't support it.
 readonly NIX_ROOT="/nix"
 readonly NIX_EXTRA_CONF=${NIX_EXTRA_CONF:-}
 
-readonly PROFILE_TARGETS=("/etc/bashrc" "/etc/profile.d/nix.sh" "/etc/zshenv")
+readonly PROFILE_TARGETS=("/etc/bashrc" "/etc/profile.d/nix.sh" "/etc/zshenv" "/etc/bash.bashrc" "/etc/zsh/zshenv")
 readonly PROFILE_BACKUP_SUFFIX=".backup-before-nix"
 readonly PROFILE_NIX_FILE="$NIX_ROOT/var/nix/profiles/default/etc/profile.d/nix-daemon.sh"
 
@@ -41,7 +43,7 @@ readonly NIX_INSTALLED_CACERT="@cacert@"
 #readonly NIX_INSTALLED_CACERT="/nix/store/7dxhzymvy330i28ii676fl1pqwcahv2f-nss-cacert-3.49.2"
 readonly EXTRACTED_NIX_PATH="$(dirname "$0")"
 
-readonly ROOT_HOME=$(echo ~root)
+readonly ROOT_HOME=~root
 
 if [ -t 0 ]; then
     readonly IS_HEADLESS='no'
@@ -57,25 +59,28 @@ headless() {
     fi
 }
 
-contactme() {
+contact_us() {
+    echo "You can open an issue at https://github.com/nixos/nix/issues"
+    echo ""
+    echo "Or feel free to contact the team:"
+    echo " - Matrix: #nix:nixos.org"
+    echo " - IRC: in #nixos on irc.libera.chat"
+    echo " - twitter: @nixos_org"
+    echo " - forum: https://discourse.nixos.org"
+}
+get_help() {
     echo "We'd love to help if you need it."
     echo ""
-    echo "If you can, open an issue at https://github.com/nixos/nix/issues"
-    echo ""
-    echo "Or feel free to contact the team,"
-    echo " - on IRC #nixos on irc.freenode.net"
-    echo " - on twitter @nixos_org"
+    contact_us
 }
 
 uninstall_directions() {
     subheader "Uninstalling nix:"
     local step=0
 
-    if [ -e /run/systemd/system ] && poly_service_installed_check; then
+    if poly_service_installed_check; then
         step=$((step + 1))
         poly_service_uninstall_directions "$step"
-    else
-        step=$((step + 1))
     fi
 
     for profile_target in "${PROFILE_TARGETS[@]}"; do
@@ -102,11 +107,10 @@ $step. Delete the files Nix added to your system:
 and that is it.
 
 EOF
-
 }
 
 nix_user_for_core() {
-    printf "nixbld%d" "$1"
+    printf "$NIX_BUILD_USER_NAME_TEMPLATE" "$1"
 }
 
 nix_uid_for_core() {
@@ -170,7 +174,7 @@ failure() {
     header "oh no!"
     _textout "$RED" "$@"
     echo ""
-    _textout "$RED" "$(contactme)"
+    _textout "$RED" "$(get_help)"
     trap finish_cleanup EXIT
     exit 1
 }
@@ -201,6 +205,95 @@ ui_confirm() {
     return 1
 }
 
+printf -v _UNCHANGED_GRP_FMT "%b" $'\033[2m%='"$ESC" # "dim"
+# bold+invert+red and bold+invert+green just for the +/- below
+# red/green foreground for rest of the line
+printf -v _OLD_LINE_FMT "%b" $'\033[1;7;31m-'"$ESC ${RED}%L${ESC}"
+printf -v _NEW_LINE_FMT "%b" $'\033[1;7;32m+'"$ESC ${GREEN}%L${ESC}"
+
+_diff() {
+    # simple colorized diff comatible w/ pre `--color` versions
+    diff --unchanged-group-format="$_UNCHANGED_GRP_FMT" --old-line-format="$_OLD_LINE_FMT" --new-line-format="$_NEW_LINE_FMT" --unchanged-line-format="  %L" "$@"
+}
+
+confirm_rm() {
+    local path="$1"
+    if ui_confirm "Can I remove $path?"; then
+        _sudo "to remove $path" rm "$path"
+    fi
+}
+
+confirm_edit() {
+    local path="$1"
+    local edit_path="$2"
+    cat <<EOF
+
+Nix isn't the only thing in $path,
+but I think I know how to edit it out.
+Here's the diff:
+EOF
+
+    # could technically test the diff, but caller should do it
+    _diff "$path" "$edit_path"
+    if ui_confirm "Does the change above look right?"; then
+        _sudo "remove nix from $path" cp "$edit_path" "$path"
+    fi
+}
+
+_SERIOUS_BUSINESS="${RED}%s:${ESC} "
+password_confirm() {
+    local do_something_consequential="$1"
+    if ui_confirm "Can I $do_something_consequential?"; then
+        # shellcheck disable=SC2059
+        sudo -kv --prompt="$(printf "${_SERIOUS_BUSINESS}" "Enter your password to $do_something_consequential")"
+    else
+        return 1
+    fi
+}
+
+# Support accumulating reminders over the course of a run and showing
+# them at the end. An example where this helps: the installer changes
+# something, but it won't work without a reboot. If you tell the user
+# when you do it, they may miss it in the stream. The value of the
+# setting isn't enough to decide whether to message because you only
+# need to message if you *changed* it.
+
+# reminders stored in array delimited by empty entry; if ! headless,
+# user is asked to confirm after each delimiter.
+_reminders=()
+((_remind_num=1))
+
+remind() {
+    # (( arithmetic expression ))
+    if (( _remind_num > 1 )); then
+        header "Reminders"
+        for line in "${_reminders[@]}"; do
+            echo "$line"
+            if ! headless && [ "${#line}" = 0 ]; then
+                if read -r -p "Press enter/return to acknowledge."; then
+                    printf $'\033[A\33[2K\r'
+                fi
+            fi
+        done
+    fi
+}
+
+reminder() {
+    printf -v label "${BLUE}[ %d ]${ESC}" "$_remind_num"
+    _reminders+=("$label")
+    if [[ "$*" = "" ]]; then
+        while read -r line; do
+            _reminders+=("$line")
+        done
+    else
+        # this expands each arg to an array entry (and each entry will
+        # ultimately be a separate line in the output)
+        _reminders+=("$@")
+    fi
+    _reminders+=("")
+    ((_remind_num++))
+}
+
 __sudo() {
     local expl="$1"
     local cmd="$2"
@@ -221,18 +314,18 @@ _sudo() {
     local expl="$1"
     shift
     if ! headless; then
-        __sudo "$expl" "$*"
+        __sudo "$expl" "$*" >&2
     fi
     sudo "$@"
 }
 
 
-readonly SCRATCH=$(mktemp -d -t tmp.XXXXXXXXXX)
-function finish_cleanup {
+readonly SCRATCH=$(mktemp -d "${TMPDIR:-/tmp/}tmp.XXXXXXXXXX")
+finish_cleanup() {
     rm -rf "$SCRATCH"
 }
 
-function finish_fail {
+finish_fail() {
     finish_cleanup
 
     failure <<EOF
@@ -244,65 +337,46 @@ EOF
 }
 trap finish_fail EXIT
 
-channel_update_failed=0
-function finish_success {
-    finish_cleanup
-
+finish_success() {
     ok "Alright! We're done!"
-    if [ "x$channel_update_failed" = x1 ]; then
-        echo ""
-        echo "But fetching the nixpkgs channel failed. (Are you offline?)"
-        echo "To try again later, run \"sudo -i nix-channel --update nixpkgs\"."
-    fi
-
-    if [ -e /run/systemd/system ]; then
-        cat <<EOF
-
-Before Nix will work in your existing shells, you'll need to close
-them and open them again. Other than that, you should be ready to go.
 
+    cat <<EOF
 Try it! Open a new terminal, and type:
-
+$(poly_extra_try_me_commands)
   $ nix-shell -p nix-info --run "nix-info -m"
 
-Thank you for using this installer. If you have any feedback, don't
-hesitate:
+Thank you for using this installer. If you have any feedback or need
+help, don't hesitate:
 
-$(contactme)
+$(contact_us)
 EOF
-    else
-        cat <<EOF
-
-Before Nix will work in your existing shells, you'll need to close
-them and open them again. Other than that, you should be ready to go.
-
-Try it! Open a new terminal, and type:
-
-  $ sudo nix-daemon
-  $ nix-shell -p nix-info --run "nix-info -m"
-
-Additionally, you may want to add nix-daemon to your init-system.
-
-Thank you for using this installer. If you have any feedback, don't
-hesitate:
-
-$(contactme)
-EOF
-   fi
-
+    remind
+    finish_cleanup
 }
 
+finish_uninstall_success() {
+    ok "Alright! Nix should be removed!"
+
+    cat <<EOF
+If you spot anything this uninstaller missed or have feedback,
+don't hesitate:
+
+$(contact_us)
+EOF
+    remind
+    finish_cleanup
+}
+
+remove_nix_artifacts() {
+    failure "Not implemented yet"
+}
+
+cure_artifacts() {
+    poly_cure_artifacts
+    # remove_nix_artifacts (LATER)
+}
 
 validate_starting_assumptions() {
-    poly_validate_assumptions
-
-    if [ $EUID -eq 0 ]; then
-        failure <<EOF
-Please do not run this script with root privileges. We will call sudo
-when we need to.
-EOF
-    fi
-
     if type nix-env 2> /dev/null >&2; then
         warning <<EOF
 Nix already appears to be installed. This installer may run into issues.
@@ -464,18 +538,46 @@ create_build_users() {
 
 create_directories() {
     # FIXME: remove all of this because it duplicates LocalStore::LocalStore().
+    task "Setting up the basic directory structure"
+    if [ -d "$NIX_ROOT" ]; then
+        # if /nix already exists, take ownership
+        #
+        # Caution: notes below are macOS-y
+        # This is a bit of a goldilocks zone for taking ownership
+        # if there are already files on the volume; the volume is
+        # now mounted, but we haven't added a bunch of new files
 
+        # this is probably a bit slow; I've been seeing 3.3-4s even
+        # when promptly installed over a fresh single-user install.
+        # In case anyone's aware of a shortcut.
+        # `|| true`: .Trashes errors w/o full disk perm
+
+        # rumor per #4488 that macOS 11.2 may not have
+        # sbin on path, and that's where chown is, but
+        # since this bit is cross-platform:
+        # - first try with `command -vp` to try and find
+        #   chown in the usual places
+        # - fall back on `command -v` which would find
+        #   any chown on path
+        # if we don't find one, the command is already
+        # hiding behind || true, and the general state
+        # should be one the user can repair once they
+        # figure out where chown is...
+        local get_chr_own="$(command -vp chown)"
+        if [[ -z "$get_chr_own" ]]; then
+            get_chr_own="$(command -v chown)"
+        fi
+        _sudo "to take root ownership of existing Nix store files" \
+              "$get_chr_own" -R "root:$NIX_BUILD_GROUP_NAME" "$NIX_ROOT" || true
+    fi
     _sudo "to make the basic directory structure of Nix (part 1)" \
-          mkdir -pv -m 0755 /nix /nix/var /nix/var/log /nix/var/log/nix /nix/var/log/nix/drvs /nix/var/nix{,/db,/gcroots,/profiles,/temproots,/userpool} /nix/var/nix/{gcroots,profiles}/per-user
+          install -dv -m 0755 /nix /nix/var /nix/var/log /nix/var/log/nix /nix/var/log/nix/drvs /nix/var/nix{,/db,/gcroots,/profiles,/temproots,/userpool} /nix/var/nix/{gcroots,profiles}/per-user
 
     _sudo "to make the basic directory structure of Nix (part 2)" \
-          mkdir -pv -m 1775 /nix/store
-
-    _sudo "to make the basic directory structure of Nix (part 3)" \
-          chgrp "$NIX_BUILD_GROUP_NAME" /nix/store
+          install -dv -g "$NIX_BUILD_GROUP_NAME" -m 1775 /nix/store
 
     _sudo "to place the default nix daemon configuration (part 1)" \
-          mkdir -pv -m 0555 /etc/nix
+          install -dv -m 0555 /etc/nix
 }
 
 place_channel_configuration() {
@@ -495,7 +597,7 @@ This installation tool will set up your computer with the Nix package
 manager. This will happen in a few stages:
 
 1. Make sure your computer doesn't already have Nix. If it does, I
-   will show you instructions on how to clean up your old one.
+   will show you instructions on how to clean up your old install.
 
 2. Show you what we are going to install and where. Then we will ask
    if you are ready to continue.
@@ -594,6 +696,7 @@ EOF
 }
 
 install_from_extracted_nix() {
+    task "Installing Nix"
     (
         cd "$EXTRACTED_NIX_PATH"
 
@@ -609,9 +712,8 @@ $NIX_INSTALLED_NIX.
 EOF
         fi
 
-        cat ./.reginfo \
-            | _sudo "to load data for the first time in to the Nix Database" \
-                   "$NIX_INSTALLED_NIX/bin/nix-store" --load-db
+        _sudo "to load data for the first time in to the Nix Database" \
+              "$NIX_INSTALLED_NIX/bin/nix-store" --load-db < ./.reginfo
 
         echo "      Just finished getting the nix database ready."
     )
@@ -630,37 +732,47 @@ EOF
 }
 
 configure_shell_profile() {
-    # If there is an /etc/profile.d directory, we want to ensure there
-    # is a nix.sh within it, so we can use the following loop to add
-    # the source lines to it. Note that I'm _not_ adding the source
-    # lines here, because we want to be using the regular machinery.
-    #
-    # If we go around that machinery, it becomes more complicated and
-    # adds complications to the uninstall instruction generator and
-    # old instruction sniffer as well.
-    if [ -d /etc/profile.d ]; then
-        _sudo "create a stub /etc/profile.d/nix.sh which will be updated" \
-              touch /etc/profile.d/nix.sh
-    fi
-
+    task "Setting up shell profiles: ${PROFILE_TARGETS[*]}"
     for profile_target in "${PROFILE_TARGETS[@]}"; do
         if [ -e "$profile_target" ]; then
             _sudo "to back up your current $profile_target to $profile_target$PROFILE_BACKUP_SUFFIX" \
                   cp "$profile_target" "$profile_target$PROFILE_BACKUP_SUFFIX"
+        else
+            # try to create the file if its directory exists
+            target_dir="$(dirname "$profile_target")"
+            if [ -d "$target_dir" ]; then
+                _sudo "to create a stub $profile_target which will be updated" \
+                    touch "$profile_target"
+            fi
+        fi
 
+        if [ -e "$profile_target" ]; then
             shell_source_lines \
                 | _sudo "extend your $profile_target with nix-daemon settings" \
                         tee -a "$profile_target"
         fi
     done
+    # TODO: should we suggest '. $PROFILE_NIX_FILE'? It would get them on
+    # their way less disruptively, but a counter-argument is that they won't
+    # immediately notice if something didn't get set up right?
+    reminder "Nix won't work in active shell sessions until you restart them."
+}
+
+cert_in_store() {
+    # in a subshell
+    # - change into the cert-file dir
+    # - get the phyiscal pwd
+    # and test if this path is in the Nix store
+    [[ "$(cd -- "$(dirname "$NIX_SSL_CERT_FILE")" && exec pwd -P)" == "$NIX_ROOT/store/"* ]]
 }
 
 setup_default_profile() {
-    _sudo "to installing a bootstrapping Nix in to the default Profile" \
+    task "Setting up the default profile"
+    _sudo "to install a bootstrapping Nix in to the default profile" \
           HOME="$ROOT_HOME" "$NIX_INSTALLED_NIX/bin/nix-env" -i "$NIX_INSTALLED_NIX"
 
-    if [ -z "${NIX_SSL_CERT_FILE:-}" ] || ! [ -f "${NIX_SSL_CERT_FILE:-}" ]; then
-        _sudo "to installing a bootstrapping SSL certificate just for Nix in to the default Profile" \
+    if [ -z "${NIX_SSL_CERT_FILE:-}" ] || ! [ -f "${NIX_SSL_CERT_FILE:-}" ] || cert_in_store; then
+        _sudo "to install a bootstrapping SSL certificate just for Nix in to the default profile" \
               HOME="$ROOT_HOME" "$NIX_INSTALLED_NIX/bin/nix-env" -i "$NIX_INSTALLED_CACERT"
         export NIX_SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt
     fi
@@ -669,9 +781,13 @@ setup_default_profile() {
         # Have to explicitly pass NIX_SSL_CERT_FILE as part of the sudo call,
         # otherwise it will be lost in environments where sudo doesn't pass
         # all the environment variables by default.
-        _sudo "to update the default channel in the default profile" \
-            HOME="$ROOT_HOME" NIX_SSL_CERT_FILE="$NIX_SSL_CERT_FILE" "$NIX_INSTALLED_NIX/bin/nix-channel" --update nixpkgs \
-            || channel_update_failed=1
+        if ! _sudo "to update the default channel in the default profile" \
+            HOME="$ROOT_HOME" NIX_SSL_CERT_FILE="$NIX_SSL_CERT_FILE" "$NIX_INSTALLED_NIX/bin/nix-channel" --update nixpkgs; then
+            reminder <<EOF
+I had trouble fetching the nixpkgs channel (are you offline?)
+To try again later, run: sudo -i nix-channel --update nixpkgs
+EOF
+        fi
     fi
 }
 
@@ -686,6 +802,17 @@ EOF
 }
 
 main() {
+    # TODO: I've moved this out of validate_starting_assumptions so we
+    # can fail faster in this case. Sourcing install-darwin... now runs
+    # `touch /` to detect Read-only root, but it could update times on
+    # pre-Catalina macOS if run as root user.
+    if [ $EUID -eq 0 ]; then
+        failure <<EOF
+Please do not run this script with root privileges. We will call sudo
+when we need to.
+EOF
+    fi
+
     if [ "$(uname -s)" = "Darwin" ]; then
         # shellcheck source=./install-darwin-multi-user.sh
         . "$EXTRACTED_NIX_PATH/install-darwin-multi-user.sh"
@@ -699,17 +826,24 @@ main() {
     welcome_to_nix
     chat_about_sudo
 
+    cure_artifacts
+    # TODO: there's a tension between cure and validate. I moved the
+    # the sudo/root check out of validate to the head of this func.
+    # Cure is *intended* to subsume the validate-and-abort approach,
+    # so it may eventually obsolete it.
     validate_starting_assumptions
 
     setup_report
 
     if ! ui_confirm "Ready to continue?"; then
         ok "Alright, no changes have been made :)"
-        contactme
+        get_help
         trap finish_cleanup EXIT
         exit 1
     fi
 
+    poly_prepare_to_install
+
     create_build_group
     create_build_users
     create_directories
@@ -719,18 +853,32 @@ main() {
     configure_shell_profile
 
     set +eu
+    # shellcheck disable=SC1091
     . /etc/profile
     set -eu
 
     setup_default_profile
     place_nix_configuration
 
-    if [ -e /run/systemd/system ]; then
-        poly_configure_nix_daemon_service
-    fi
+    poly_configure_nix_daemon_service
 
     trap finish_success EXIT
 }
 
+# set an empty initial arg for bare invocations in case we need to
+# disambiguate someone directly invoking this later.
+if [ "${#@}" = 0 ]; then
+    set ""
+fi
 
-main
+# ACTION for override
+case "${1-}" in
+    # uninstall)
+    #     shift
+    #     uninstall "$@";;
+    # install == same as the no-arg condition for now (but, explicit)
+    ""|install)
+        main;;
+    *) # holding space for future options (like uninstall + install?)
+        failure "install-multi-user: invalid argument";;
+esac

scripts/install-nix-from-closure.sh

@@ -26,30 +26,51 @@ fi
 
 # macOS support for 10.12.6 or higher
 if [ "$(uname -s)" = "Darwin" ]; then
-    macos_major=$(sw_vers -productVersion | cut -d '.' -f 2)
-    macos_minor=$(sw_vers -productVersion | cut -d '.' -f 3)
-    if [ "$macos_major" -lt 12 ] || { [ "$macos_major" -eq 12 ] && [ "$macos_minor" -lt 6 ]; }; then
+    IFS='.' read -r macos_major macos_minor macos_patch << EOF
+$(sw_vers -productVersion)
+EOF
+    if [ "$macos_major" -lt 10 ] || { [ "$macos_major" -eq 10 ] && [ "$macos_minor" -lt 12 ]; } || { [ "$macos_minor" -eq 12 ] && [ "$macos_patch" -lt 6 ]; }; then
+        # patch may not be present; command substitution for simplicity
         echo "$0: macOS $(sw_vers -productVersion) is not supported, upgrade to 10.12.6 or higher"
         exit 1
     fi
 fi
 
 # Determine if we could use the multi-user installer or not
-if [ "$(uname -s)" = "Darwin" ]; then
-    echo "Note: a multi-user installation is possible. See https://nixos.org/nix/manual/#sect-multi-user-installation" >&2
-elif [ "$(uname -s)" = "Linux" ]; then
+if [ "$(uname -s)" = "Linux" ]; then
     echo "Note: a multi-user installation is possible. See https://nixos.org/nix/manual/#sect-multi-user-installation" >&2
 fi
 
-INSTALL_MODE=no-daemon
-CREATE_DARWIN_VOLUME=0
+case "$(uname -s)" in
+    "Darwin")
+        INSTALL_MODE=daemon;;
+    *)
+        INSTALL_MODE=no-daemon;;
+esac
+
+# space-separated string
+ACTIONS=
+
 # handle the command line flags
 while [ $# -gt 0 ]; do
     case $1 in
         --daemon)
-            INSTALL_MODE=daemon;;
+            INSTALL_MODE=daemon
+            ACTIONS="${ACTIONS}install "
+            ;;
         --no-daemon)
-            INSTALL_MODE=no-daemon;;
+            if [ "$(uname -s)" = "Darwin" ]; then
+                printf '\e[1;31mError: --no-daemon installs are no-longer supported on Darwin/macOS!\e[0m\n' >&2
+                exit 1
+            fi
+            INSTALL_MODE=no-daemon
+            # intentional tail space
+            ACTIONS="${ACTIONS}install "
+            ;;
+        # --uninstall)
+        #     # intentional tail space
+        #     ACTIONS="${ACTIONS}uninstall "
+        #     ;;
         --no-channel-add)
             export NIX_INSTALLER_NO_CHANNEL_ADD=1;;
         --daemon-user-count)
@@ -58,13 +79,18 @@ while [ $# -gt 0 ]; do
         --no-modify-profile)
             NIX_INSTALLER_NO_MODIFY_PROFILE=1;;
         --darwin-use-unencrypted-nix-store-volume)
-            CREATE_DARWIN_VOLUME=1;;
+            {
+                echo "Warning: the flag --darwin-use-unencrypted-nix-store-volume"
+                echo "         is no longer needed and will be removed in the future."
+                echo ""
+            } >&2;;
         --nix-extra-conf-file)
-            export NIX_EXTRA_CONF="$(cat $2)"
+            # shellcheck disable=SC2155
+            export NIX_EXTRA_CONF="$(cat "$2")"
             shift;;
         *)
-            (
-                echo "Nix Installer [--daemon|--no-daemon] [--daemon-user-count INT] [--no-channel-add] [--no-modify-profile] [--darwin-use-unencrypted-nix-store-volume] [--nix-extra-conf-file FILE]"
+            {
+                echo "Nix Installer [--daemon|--no-daemon] [--daemon-user-count INT] [--no-channel-add] [--no-modify-profile] [--nix-extra-conf-file FILE]"
 
                 echo "Choose installation method."
                 echo ""
@@ -87,45 +113,19 @@ while [ $# -gt 0 ]; do
                 echo ""
                 echo " --nix-extra-conf-file: Path to nix.conf to prepend when installing /etc/nix.conf"
                 echo ""
-            ) >&2
+                if [ -n "${INVOKED_FROM_INSTALL_IN:-}" ]; then
+                    echo " --tarball-url-prefix URL: Base URL to download the Nix tarball from."
+                fi
+            } >&2
 
-            # darwin and Catalina+
-            if [ "$(uname -s)" = "Darwin" ] && [ "$macos_major" -gt 14 ]; then
-                (
-                    echo " --darwin-use-unencrypted-nix-store-volume: Create an APFS volume for the Nix"
-                    echo "              store and mount it at /nix. This is the recommended way to create"
-                    echo "              /nix with a read-only / on macOS >=10.15."
-                    echo "              See: https://nixos.org/nix/manual/#sect-macos-installation"
-                    echo ""
-                ) >&2
-            fi
             exit;;
     esac
     shift
 done
 
-if [ "$(uname -s)" = "Darwin" ]; then
-    if [ "$CREATE_DARWIN_VOLUME" = 1 ]; then
-        printf '\e[1;31mCreating volume and mountpoint /nix.\e[0m\n'
-        "$self/create-darwin-volume.sh"
-    fi
-
-    info=$(diskutil info -plist / | xpath "/plist/dict/key[text()='Writable']/following-sibling::true[1]" 2> /dev/null)
-    if ! [ -e $dest ] && [ -n "$info" ] && [ "$macos_major" -gt 14 ]; then
-        (
-            echo ""
-            echo "Installing on macOS >=10.15 requires relocating the store to an apfs volume."
-            echo "Use sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume or run the preparation steps manually."
-            echo "See https://nixos.org/nix/manual/#sect-macos-installation"
-            echo ""
-        ) >&2
-        exit 1
-    fi
-fi
-
 if [ "$INSTALL_MODE" = "daemon" ]; then
     printf '\e[1;31mSwitching to the Multi-user Installer\e[0m\n'
-    exec "$self/install-multi-user"
+    exec "$self/install-multi-user" $ACTIONS # let ACTIONS split
     exit 0
 fi
 
@@ -152,9 +152,15 @@ fi
 mkdir -p $dest/store
 
 printf "copying Nix to %s..." "${dest}/store" >&2
+# Insert a newline if no progress is shown.
+if [ ! -t 0 ]; then
+  echo ""
+fi
 
 for i in $(cd "$self/store" >/dev/null && echo ./*); do
-    printf "." >&2
+    if [ -t 0 ]; then
+      printf "." >&2
+    fi
     i_tmp="$dest/store/$i.$$"
     if [ -e "$i_tmp" ]; then
         rm -rf "$i_tmp"
@@ -174,6 +180,7 @@ if ! "$nix/bin/nix-store" --load-db < "$self/.reginfo"; then
     exit 1
 fi
 
+# shellcheck source=./nix-profile.sh.in
 . "$nix/etc/profile.d/nix.sh"
 
 if ! "$nix/bin/nix-env" -i "$nix"; then

scripts/install-systemd-multi-user.sh

@@ -41,10 +41,8 @@ handle_network_proxy() {
     fi
 }
 
-poly_validate_assumptions() {
-    if [ "$(uname -s)" != "Linux" ]; then
-        failure "This script is for use with Linux!"
-    fi
+poly_cure_artifacts() {
+    :
 }
 
 poly_service_installed_check() {
@@ -72,24 +70,38 @@ poly_service_setup_note() {
 EOF
 }
 
+poly_extra_try_me_commands() {
+    if [ -e /run/systemd/system ]; then
+        :
+    else
+        cat <<EOF
+  $ sudo nix-daemon
+EOF
+    fi
+}
+
 poly_configure_nix_daemon_service() {
-    _sudo "to set up the nix-daemon service" \
-          systemctl link "/nix/var/nix/profiles/default$SERVICE_SRC"
+    if [ -e /run/systemd/system ]; then
+        task "Setting up the nix-daemon systemd service"
+        _sudo "to set up the nix-daemon service" \
+              systemctl link "/nix/var/nix/profiles/default$SERVICE_SRC"
 
-    _sudo "to set up the nix-daemon socket service" \
-          systemctl enable "/nix/var/nix/profiles/default$SOCKET_SRC"
+        _sudo "to set up the nix-daemon socket service" \
+              systemctl enable "/nix/var/nix/profiles/default$SOCKET_SRC"
 
-    handle_network_proxy
+        handle_network_proxy
 
-    _sudo "to load the systemd unit for nix-daemon" \
-          systemctl daemon-reload
+        _sudo "to load the systemd unit for nix-daemon" \
+              systemctl daemon-reload
 
-    _sudo "to start the nix-daemon.socket" \
-          systemctl start nix-daemon.socket
-
-    _sudo "to start the nix-daemon.service" \
-          systemctl restart nix-daemon.service
+        _sudo "to start the nix-daemon.socket" \
+              systemctl start nix-daemon.socket
 
+        _sudo "to start the nix-daemon.service" \
+              systemctl restart nix-daemon.service
+    else
+        reminder "I don't support your init system yet; you may want to add nix-daemon manually."
+    fi
 }
 
 poly_group_exists() {
@@ -186,3 +198,7 @@ poly_create_build_user() {
           --password "!" \
           "$username"
 }
+
+poly_prepare_to_install() {
+    :
+}

scripts/install.in

@@ -25,16 +25,46 @@ require_util() {
 }
 
 case "$(uname -s).$(uname -m)" in
-    Linux.x86_64) system=x86_64-linux; hash=@binaryTarball_x86_64-linux@;;
-    Linux.i?86) system=i686-linux; hash=@binaryTarball_i686-linux@;;
-    Linux.aarch64) system=aarch64-linux; hash=@binaryTarball_aarch64-linux@;;
-    Darwin.x86_64) system=x86_64-darwin; hash=@binaryTarball_x86_64-darwin@;;
+    Linux.x86_64)
+        hash=@tarballHash_x86_64-linux@
+        path=@tarballPath_x86_64-linux@
+        system=x86_64-linux
+        ;;
+    Linux.i?86)
+        hash=@tarballHash_i686-linux@
+        path=@tarballPath_i686-linux@
+        system=i686-linux
+        ;;
+    Linux.aarch64)
+        hash=@tarballHash_aarch64-linux@
+        path=@tarballPath_aarch64-linux@
+        system=aarch64-linux
+        ;;
+    Darwin.x86_64)
+        hash=@tarballHash_x86_64-darwin@
+        path=@tarballPath_x86_64-darwin@
+        system=x86_64-darwin
+        ;;
+    Darwin.arm64|Darwin.aarch64)
+        hash=@binaryTarball_aarch64-darwin@
+        path=@tarballPath_aarch64-darwin@
+        system=aarch64-darwin
+        ;;
     *) oops "sorry, there is no binary distribution of Nix for your platform";;
 esac
 
-url="https://releases.nixos.org/nix/nix-@nixVersion@/nix-@nixVersion@-$system.tar.xz"
+# Use this command-line option to fetch the tarballs using nar-serve or Cachix
+if [ "${1:-}" = "--tarball-url-prefix" ]; then
+    if [ -z "${2:-}" ]; then
+        oops "missing argument for --tarball-url-prefix"
+    fi
+    url=${2}/${path}
+    shift 2
+else
+    url=https://releases.nixos.org/nix/nix-@nixVersion@/nix-@nixVersion@-$system.tar.xz
+fi
 
-tarball="$tmpDir/$(basename "$tmpDir/nix-@nixVersion@-$system.tar.xz")"
+tarball=$tmpDir/nix-@nixVersion@-$system.tar.xz
 
 require_util curl "download the binary tarball"
 require_util tar "unpack the binary tarball"
@@ -66,6 +96,7 @@ tar -xJf "$tarball" -C "$unpack" || oops "failed to unpack '$url'"
 script=$(echo "$unpack"/*/install)
 
 [ -e "$script" ] || oops "installation script is missing from the binary tarball!"
+export INVOKED_FROM_INSTALL_IN=1
 "$script" "$@"
 
 } # End of wrapping

scripts/nix-profile-daemon.sh.in

@@ -17,11 +17,21 @@ elif [ -e /etc/pki/tls/certs/ca-bundle.crt ]; then # Fedora, CentOS
     export NIX_SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt
 else
   # Fall back to what is in the nix profiles, favouring whatever is defined last.
-  for i in $NIX_PROFILES; do
-    if [ -e $i/etc/ssl/certs/ca-bundle.crt ]; then
-      export NIX_SSL_CERT_FILE=$i/etc/ssl/certs/ca-bundle.crt
+  check_nix_profiles() {
+    if [ "$ZSH_VERSION" ]; then
+      # Zsh by default doesn't split words in unquoted parameter expansion.
+      # Set local_options for these options to be reverted at the end of the function
+      # and shwordsplit to force splitting words in $NIX_PROFILES below.
+      setopt local_options shwordsplit
     fi
-  done
+    for i in $NIX_PROFILES; do
+      if [ -e $i/etc/ssl/certs/ca-bundle.crt ]; then
+        export NIX_SSL_CERT_FILE=$i/etc/ssl/certs/ca-bundle.crt
+      fi
+    done
+  }
+  check_nix_profiles
+  unset -f check_nix_profiles
 fi
 
 export PATH="$HOME/.nix-profile/bin:@localstatedir@/nix/profiles/default/bin:$PATH"

scripts/prepare-installer-for-github-actions

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+
+set -e
+
+script=$(nix-build -A outputs.hydraJobs.installerScriptForGHA --no-out-link)
+installerHash=$(echo $script | cut -b12-43 -)
+
+installerURL=https://$CACHIX_NAME.cachix.org/serve/$installerHash/install
+
+echo "::set-output name=installerURL::$installerURL"

src/build-remote/build-remote.cc

@@ -17,7 +17,7 @@
 #include "store-api.hh"
 #include "derivations.hh"
 #include "local-store.hh"
-#include "../nix/legacy.hh"
+#include "legacy.hh"
 
 using namespace nix;
 using std::cin;
@@ -53,6 +53,9 @@ static int main_build_remote(int argc, char * * argv)
         unsetenv("DISPLAY");
         unsetenv("SSH_ASKPASS");
 
+        /* If we ever use the common args framework, make sure to
+           remove initPlugins below and initialize settings first.
+        */
         if (argc != 2)
             throw UsageError("called without required arguments");
 
@@ -71,11 +74,15 @@ static int main_build_remote(int argc, char * * argv)
 
         initPlugins();
 
-        auto store = openStore().cast<LocalStore>();
+        auto store = openStore();
 
         /* It would be more appropriate to use $XDG_RUNTIME_DIR, since
            that gets cleared on reboot, but it wouldn't work on macOS. */
-        currentLoad = store->stateDir + "/current-load";
+        auto currentLoadName = "/current-load";
+        if (auto localStore = store.dynamic_pointer_cast<LocalFSStore>())
+            currentLoad = std::string { localStore->stateDir } + currentLoadName;
+        else
+            currentLoad = settings.nixStateDir + currentLoadName;
 
         std::shared_ptr<Store> sshStore;
         AutoCloseFD bestSlotLock;
@@ -172,13 +179,14 @@ static int main_build_remote(int argc, char * * argv)
                     else
                     {
                         // build the hint template.
-                        string hintstring =  "derivation: %s\nrequired (system, features): (%s, %s)";
-                        hintstring += "\n%s available machines:";
-                        hintstring += "\n(systems, maxjobs, supportedFeatures, mandatoryFeatures)";
+                        string errorText =
+                            "Failed to find a machine for remote build!\n"
+                            "derivation: %s\nrequired (system, features): (%s, %s)";
+                        errorText += "\n%s available machines:";
+                        errorText += "\n(systems, maxjobs, supportedFeatures, mandatoryFeatures)";
 
-                        for (unsigned int i = 0; i < machines.size(); ++i) {
-                          hintstring += "\n(%s, %s, %s, %s)";
-                        }
+                        for (unsigned int i = 0; i < machines.size(); ++i)
+                            errorText += "\n(%s, %s, %s, %s)";
 
                         // add the template values.
                         string drvstr;
@@ -187,25 +195,21 @@ static int main_build_remote(int argc, char * * argv)
                         else
                             drvstr = "<unknown>";
 
-                        auto hint = hintformat(hintstring);
-                        hint
-                          % drvstr
-                          % neededSystem
-                          % concatStringsSep<StringSet>(", ", requiredFeatures)
-                          % machines.size();
+                        auto error = hintformat(errorText);
+                        error
+                            % drvstr
+                            % neededSystem
+                            % concatStringsSep<StringSet>(", ", requiredFeatures)
+                            % machines.size();
 
-                        for (auto & m : machines) {
-                          hint % concatStringsSep<vector<string>>(", ", m.systemTypes)
-                            % m.maxJobs
-                            % concatStringsSep<StringSet>(", ", m.supportedFeatures)
-                            % concatStringsSep<StringSet>(", ", m.mandatoryFeatures);
-                        }
+                        for (auto & m : machines)
+                            error
+                                % concatStringsSep<vector<string>>(", ", m.systemTypes)
+                                % m.maxJobs
+                                % concatStringsSep<StringSet>(", ", m.supportedFeatures)
+                                % concatStringsSep<StringSet>(", ", m.mandatoryFeatures);
 
-                        logErrorInfo(lvlInfo, {
-                              .name = "Remote build",
-                              .description = "Failed to find a machine for remote build!",
-                              .hint = hint
-                        });
+                        printMsg(canBuildLocally ? lvlChatty : lvlWarn, error);
 
                         std::cerr << "# decline\n";
                     }
@@ -230,12 +234,9 @@ static int main_build_remote(int argc, char * * argv)
 
                 } catch (std::exception & e) {
                     auto msg = chomp(drainFD(5, false));
-                    logError({
-                        .name = "Remote build",
-                        .hint = hintfmt("cannot build on '%s': %s%s",
-                            bestMachine->storeUri, e.what(),
-                            (msg.empty() ? "" : ": " + msg))
-                    });
+                    printError("cannot build on '%s': %s%s",
+                        bestMachine->storeUri, e.what(),
+                        msg.empty() ? "" : ": " + msg);
                     bestMachine->enabled = false;
                     continue;
                 }
@@ -250,7 +251,7 @@ connected:
         std::cerr << "# accept\n" << storeUri << "\n";
 
         auto inputs = readStrings<PathSet>(source);
-        auto outputs = readStrings<PathSet>(source);
+        auto wantedOutputs = readStrings<StringSet>(source);
 
         AutoCloseFD uploadLock = openLockFile(currentLoad + "/" + escapeUri(storeUri) + ".upload-lock", true);
 
@@ -275,22 +276,59 @@ connected:
         uploadLock = -1;
 
         auto drv = store->readDerivation(*drvPath);
-        drv.inputSrcs = store->parseStorePathSet(inputs);
+        auto outputHashes = staticOutputHashes(*store, drv);
+
+        // Hijack the inputs paths of the derivation to include all the paths
+        // that come from the `inputDrvs` set.
+        // We don’t do that for the derivations whose `inputDrvs` is empty
+        // because
+        // 1. It’s not needed
+        // 2. Changing the `inputSrcs` set changes the associated output ids,
+        //  which break CA derivations
+        if (!drv.inputDrvs.empty())
+            drv.inputSrcs = store->parseStorePathSet(inputs);
 
         auto result = sshStore->buildDerivation(*drvPath, drv);
 
         if (!result.success())
             throw Error("build of '%s' on '%s' failed: %s", store->printStorePath(*drvPath), storeUri, result.errorMsg);
 
-        StorePathSet missing;
-        for (auto & path : outputs)
-            if (!store->isValidPath(store->parseStorePath(path))) missing.insert(store->parseStorePath(path));
+        std::set<Realisation> missingRealisations;
+        StorePathSet missingPaths;
+        if (settings.isExperimentalFeatureEnabled("ca-derivations") && !derivationHasKnownOutputPaths(drv.type())) {
+            for (auto & outputName : wantedOutputs) {
+                auto thisOutputHash = outputHashes.at(outputName);
+                auto thisOutputId = DrvOutput{ thisOutputHash, outputName };
+                if (!store->queryRealisation(thisOutputId)) {
+                    debug("missing output %s", outputName);
+                    assert(result.builtOutputs.count(thisOutputId));
+                    auto newRealisation = result.builtOutputs.at(thisOutputId);
+                    missingRealisations.insert(newRealisation);
+                    missingPaths.insert(newRealisation.outPath);
+                }
+            }
+        } else {
+            auto outputPaths = drv.outputsAndOptPaths(*store);
+            for (auto & [outputName, hopefullyOutputPath] : outputPaths) {
+                assert(hopefullyOutputPath.second);
+                if (!store->isValidPath(*hopefullyOutputPath.second))
+                    missingPaths.insert(*hopefullyOutputPath.second);
+            }
+        }
 
-        if (!missing.empty()) {
+        if (!missingPaths.empty()) {
             Activity act(*logger, lvlTalkative, actUnknown, fmt("copying outputs from '%s'", storeUri));
-            for (auto & i : missing)
-                store->locksHeld.insert(store->printStorePath(i)); /* FIXME: ugly */
-            copyPaths(ref<Store>(sshStore), store, missing, NoRepair, NoCheckSigs, NoSubstitute);
+            if (auto localStore = store.dynamic_pointer_cast<LocalStore>())
+                for (auto & path : missingPaths)
+                    localStore->locksHeld.insert(store->printStorePath(path)); /* FIXME: ugly */
+            copyPaths(ref<Store>(sshStore), store, missingPaths, NoRepair, NoCheckSigs, NoSubstitute);
+        }
+        // XXX: Should be done as part of `copyPaths`
+        for (auto & realisation : missingRealisations) {
+            // Should hold, because if the feature isn't enabled the set
+            // of missing realisations should be empty
+            settings.requireExperimentalFeature("ca-derivations");
+            store->registerDrvOutput(realisation);
         }
 
         return 0;

src/libcmd/command.cc

@@ -11,11 +11,20 @@ extern char * * environ __attribute__((weak));
 
 namespace nix {
 
-Commands * RegisterCommand::commands = nullptr;
+RegisterCommand::Commands * RegisterCommand::commands = nullptr;
 
-void NixMultiCommand::printHelp(const string & programName, std::ostream & out)
+nix::Commands RegisterCommand::getCommandsFor(const std::vector<std::string> & prefix)
 {
-    MultiCommand::printHelp(programName, out);
+    nix::Commands res;
+    for (auto & [name, command] : *RegisterCommand::commands)
+        if (name.size() == prefix.size() + 1) {
+            bool equal = true;
+            for (size_t i = 0; i < prefix.size(); ++i)
+                if (name[i] != prefix[i]) equal = false;
+            if (equal)
+                res.insert_or_assign(name[prefix.size()], command);
+        }
+    return res;
 }
 
 nlohmann::json NixMultiCommand::toJSON()
@@ -45,57 +54,78 @@ void StoreCommand::run()
     run(getStore());
 }
 
-StorePathsCommand::StorePathsCommand(bool recursive)
+BuiltPathsCommand::BuiltPathsCommand(bool recursive)
     : recursive(recursive)
 {
     if (recursive)
         addFlag({
             .longName = "no-recursive",
-            .description = "apply operation to specified paths only",
+            .description = "Apply operation to specified paths only.",
+            .category = installablesCategory,
             .handler = {&this->recursive, false},
         });
     else
         addFlag({
             .longName = "recursive",
             .shortName = 'r',
-            .description = "apply operation to closure of the specified paths",
+            .description = "Apply operation to closure of the specified paths.",
+            .category = installablesCategory,
             .handler = {&this->recursive, true},
         });
 
-    mkFlag(0, "all", "apply operation to the entire store", &all);
+    addFlag({
+        .longName = "all",
+        .description = "Apply the operation to every store path.",
+        .category = installablesCategory,
+        .handler = {&all, true},
+    });
 }
 
-void StorePathsCommand::run(ref<Store> store)
+void BuiltPathsCommand::run(ref<Store> store)
 {
-    StorePaths storePaths;
-
+    BuiltPaths paths;
     if (all) {
         if (installables.size())
             throw UsageError("'--all' does not expect arguments");
+        // XXX: Only uses opaque paths, ignores all the realisations
         for (auto & p : store->queryAllValidPaths())
-            storePaths.push_back(p);
-    }
-
-    else {
-        for (auto & p : toStorePaths(store, realiseMode, operateOn, installables))
-            storePaths.push_back(p);
-
+            paths.push_back(BuiltPath::Opaque{p});
+    } else {
+        paths = toBuiltPaths(store, realiseMode, operateOn, installables);
         if (recursive) {
-            StorePathSet closure;
-            store->computeFSClosure(StorePathSet(storePaths.begin(), storePaths.end()), closure, false, false);
-            storePaths.clear();
-            for (auto & p : closure)
-                storePaths.push_back(p);
+            // XXX: This only computes the store path closure, ignoring
+            // intermediate realisations
+            StorePathSet pathsRoots, pathsClosure;
+            for (auto & root: paths) {
+                auto rootFromThis = root.outPaths();
+                pathsRoots.insert(rootFromThis.begin(), rootFromThis.end());
+            }
+            store->computeFSClosure(pathsRoots, pathsClosure);
+            for (auto & path : pathsClosure)
+                paths.push_back(BuiltPath::Opaque{path});
         }
     }
 
+    run(store, std::move(paths));
+}
+
+StorePathsCommand::StorePathsCommand(bool recursive)
+    : BuiltPathsCommand(recursive)
+{
+}
+
+void StorePathsCommand::run(ref<Store> store, BuiltPaths paths)
+{
+    StorePaths storePaths;
+    for (auto& builtPath : paths)
+        for (auto& p : builtPath.outPaths())
+            storePaths.push_back(p);
+
     run(store, std::move(storePaths));
 }
 
-void StorePathCommand::run(ref<Store> store)
+void StorePathCommand::run(ref<Store> store, std::vector<StorePath> storePaths)
 {
-    auto storePaths = toStorePaths(store, Realise::Nothing, operateOn, installables);
-
     if (storePaths.size() != 1)
         throw UsageError("this command requires exactly one store path");
 
@@ -119,7 +149,7 @@ MixProfile::MixProfile()
 {
     addFlag({
         .longName = "profile",
-        .description = "profile to update",
+        .description = "The profile to update.",
         .labels = {"path"},
         .handler = {&profile},
         .completer = completePath
@@ -138,7 +168,7 @@ void MixProfile::updateProfile(const StorePath & storePath)
             profile2, storePath));
 }
 
-void MixProfile::updateProfile(const Buildables & buildables)
+void MixProfile::updateProfile(const BuiltPaths & buildables)
 {
     if (!profile) return;
 
@@ -146,18 +176,15 @@ void MixProfile::updateProfile(const Buildables & buildables)
 
     for (auto & buildable : buildables) {
         std::visit(overloaded {
-            [&](BuildableOpaque bo) {
+            [&](BuiltPath::Opaque bo) {
                 result.push_back(bo.path);
             },
-            [&](BuildableFromDrv bfd) {
+            [&](BuiltPath::Built bfd) {
                 for (auto & output : bfd.outputs) {
-                    /* Output path should be known because we just tried to
-                       build it. */
-                    assert(output.second);
-                    result.push_back(*output.second);
+                    result.push_back(output.second);
                 }
             },
-        }, buildable);
+        }, buildable.raw());
     }
 
     if (result.size() != 1)
@@ -176,14 +203,14 @@ MixEnvironment::MixEnvironment() : ignoreEnvironment(false)
     addFlag({
         .longName = "ignore-environment",
         .shortName = 'i',
-        .description = "clear the entire environment (except those specified with --keep)",
+        .description = "Clear the entire environment (except those specified with `--keep`).",
         .handler = {&ignoreEnvironment, true},
     });
 
     addFlag({
         .longName = "keep",
         .shortName = 'k',
-        .description = "keep specified environment variable",
+        .description = "Keep the environment variable *name*.",
         .labels = {"name"},
         .handler = {[&](std::string s) { keep.insert(s); }},
     });
@@ -191,7 +218,7 @@ MixEnvironment::MixEnvironment() : ignoreEnvironment(false)
     addFlag({
         .longName = "unset",
         .shortName = 'u',
-        .description = "unset specified environment variable",
+        .description = "Unset the environment variable *name*.",
         .labels = {"name"},
         .handler = {[&](std::string s) { unset.insert(s); }},
     });

src/libcmd/command.hh

@@ -13,6 +13,8 @@ namespace nix {
 
 extern std::string programPath;
 
+extern char * * savedArgv;
+
 class EvalState;
 struct Pos;
 class Store;
@@ -21,10 +23,10 @@ static constexpr Command::Category catSecondary = 100;
 static constexpr Command::Category catUtility = 101;
 static constexpr Command::Category catNixInstallation = 102;
 
+static constexpr auto installablesCategory = "Options that change the interpretation of installables";
+
 struct NixMultiCommand : virtual MultiCommand, virtual Command
 {
-    void printHelp(const string & programName, std::ostream & out) override;
-
     nlohmann::json toJSON() override;
 };
 
@@ -46,6 +48,8 @@ struct EvalCommand : virtual StoreCommand, MixEvalArgs
     ref<EvalState> getEvalState();
 
     std::shared_ptr<EvalState> evalState;
+
+    ~EvalCommand();
 };
 
 struct MixFlakeOptions : virtual Args, EvalCommand
@@ -139,7 +143,7 @@ private:
 };
 
 /* A command that operates on zero or more store paths. */
-struct StorePathsCommand : public InstallablesCommand
+struct BuiltPathsCommand : public InstallablesCommand
 {
 private:
 
@@ -152,47 +156,67 @@ protected:
 
 public:
 
-    StorePathsCommand(bool recursive = false);
+    BuiltPathsCommand(bool recursive = false);
 
     using StoreCommand::run;
 
-    virtual void run(ref<Store> store, std::vector<StorePath> storePaths) = 0;
+    virtual void run(ref<Store> store, BuiltPaths paths) = 0;
 
     void run(ref<Store> store) override;
 
     bool useDefaultInstallables() override { return !all; }
 };
 
-/* A command that operates on exactly one store path. */
-struct StorePathCommand : public InstallablesCommand
+struct StorePathsCommand : public BuiltPathsCommand
 {
-    using StoreCommand::run;
+    StorePathsCommand(bool recursive = false);
+
+    using BuiltPathsCommand::run;
+
+    virtual void run(ref<Store> store, std::vector<StorePath> storePaths) = 0;
+
+    void run(ref<Store> store, BuiltPaths paths) override;
+};
+
+/* A command that operates on exactly one store path. */
+struct StorePathCommand : public StorePathsCommand
+{
+    using StorePathsCommand::run;
 
     virtual void run(ref<Store> store, const StorePath & storePath) = 0;
 
-    void run(ref<Store> store) override;
+    void run(ref<Store> store, std::vector<StorePath> storePaths) override;
 };
 
 /* A helper class for registering commands globally. */
 struct RegisterCommand
 {
+    typedef std::map<std::vector<std::string>, std::function<ref<Command>()>> Commands;
     static Commands * commands;
 
-    RegisterCommand(const std::string & name,
+    RegisterCommand(std::vector<std::string> && name,
         std::function<ref<Command>()> command)
     {
         if (!commands) commands = new Commands;
         commands->emplace(name, command);
     }
+
+    static nix::Commands getCommandsFor(const std::vector<std::string> & prefix);
 };
 
 template<class T>
 static RegisterCommand registerCommand(const std::string & name)
 {
-    return RegisterCommand(name, [](){ return make_ref<T>(); });
+    return RegisterCommand({name}, [](){ return make_ref<T>(); });
 }
 
-Buildables build(ref<Store> store, Realise mode,
+template<class T>
+static RegisterCommand registerCommand2(std::vector<std::string> && name)
+{
+    return RegisterCommand(std::move(name), [](){ return make_ref<T>(); });
+}
+
+BuiltPaths build(ref<Store> store, Realise mode,
     std::vector<std::shared_ptr<Installable>> installables, BuildMode bMode = bmNormal);
 
 std::set<StorePath> toStorePaths(ref<Store> store,
@@ -207,6 +231,12 @@ std::set<StorePath> toDerivations(ref<Store> store,
     std::vector<std::shared_ptr<Installable>> installables,
     bool useDeriver = false);
 
+BuiltPaths toBuiltPaths(
+    ref<Store> store,
+    Realise mode,
+    OperateOn operateOn,
+    std::vector<std::shared_ptr<Installable>> installables);
+
 /* Helper function to generate args that invoke $EDITOR on
    filename:lineno. */
 Strings editorFor(const Pos & pos);
@@ -222,7 +252,7 @@ struct MixProfile : virtual StoreCommand
 
     /* If 'profile' is set, make it point at the store path produced
        by 'buildables'. */
-    void updateProfile(const Buildables & buildables);
+    void updateProfile(const BuiltPaths & buildables);
 };
 
 struct MixDefaultProfile : MixProfile
@@ -252,6 +282,8 @@ void completeFlakeRefWithFragment(
     const Strings & defaultFlakeAttrPaths,
     std::string_view prefix);
 
+std::string showVersions(const std::set<std::string> & versions);
+
 void printClosureDiff(
     ref<Store> store,
     const StorePath & beforePath,

src/libcmd/installables.cc

@@ -16,6 +16,8 @@
 #include <regex>
 #include <queue>
 
+#include <nlohmann/json.hpp>
+
 namespace nix {
 
 void completeFlakeInputPath(
@@ -31,39 +33,47 @@ void completeFlakeInputPath(
 
 MixFlakeOptions::MixFlakeOptions()
 {
+    auto category = "Common flake-related options";
+
     addFlag({
         .longName = "recreate-lock-file",
-        .description = "recreate lock file from scratch",
+        .description = "Recreate the flake's lock file from scratch.",
+        .category = category,
         .handler = {&lockFlags.recreateLockFile, true}
     });
 
     addFlag({
         .longName = "no-update-lock-file",
-        .description = "do not allow any updates to the lock file",
+        .description = "Do not allow any updates to the flake's lock file.",
+        .category = category,
         .handler = {&lockFlags.updateLockFile, false}
     });
 
     addFlag({
         .longName = "no-write-lock-file",
-        .description = "do not write the newly generated lock file",
+        .description = "Do not write the flake's newly generated lock file.",
+        .category = category,
         .handler = {&lockFlags.writeLockFile, false}
     });
 
     addFlag({
         .longName = "no-registries",
-        .description = "don't use flake registries",
+        .description = "Don't allow lookups in the flake registries.",
+        .category = category,
         .handler = {&lockFlags.useRegistries, false}
     });
 
     addFlag({
         .longName = "commit-lock-file",
-        .description = "commit changes to the lock file",
+        .description = "Commit changes to the flake's lock file.",
+        .category = category,
         .handler = {&lockFlags.commitLockFile, true}
     });
 
     addFlag({
         .longName = "update-input",
-        .description = "update a specific flake input",
+        .description = "Update a specific flake input (ignoring its previous entry in the lock file).",
+        .category = category,
         .labels = {"input-path"},
         .handler = {[&](std::string s) {
             lockFlags.inputUpdates.insert(flake::parseInputPath(s));
@@ -76,9 +86,11 @@ MixFlakeOptions::MixFlakeOptions()
 
     addFlag({
         .longName = "override-input",
-        .description = "override a specific flake input (e.g. `dwarffs/nixpkgs`)",
+        .description = "Override a specific flake input (e.g. `dwarffs/nixpkgs`). This implies `--no-write-lock-file`.",
+        .category = category,
         .labels = {"input-path", "flake-url"},
         .handler = {[&](std::string inputPath, std::string flakeRef) {
+            lockFlags.writeLockFile = false;
             lockFlags.inputOverrides.insert_or_assign(
                 flake::parseInputPath(inputPath),
                 parseFlakeRef(flakeRef, absPath(".")));
@@ -87,7 +99,8 @@ MixFlakeOptions::MixFlakeOptions()
 
     addFlag({
         .longName = "inputs-from",
-        .description = "use the inputs of the specified flake as registry entries",
+        .description = "Use the inputs of the specified flake as registry entries.",
+        .category = category,
         .labels = {"flake-url"},
         .handler = {[&](std::string flakeRef) {
             auto evalState = getEvalState();
@@ -116,22 +129,25 @@ SourceExprCommand::SourceExprCommand()
     addFlag({
         .longName = "file",
         .shortName = 'f',
-        .description = "evaluate *file* rather than the default",
+        .description = "Interpret installables as attribute paths relative to the Nix expression stored in *file*.",
+        .category = installablesCategory,
         .labels = {"file"},
         .handler = {&file},
         .completer = completePath
     });
 
     addFlag({
-        .longName ="expr",
-        .description = "evaluate attributes from *expr*",
+        .longName = "expr",
+        .description = "Interpret installables as attribute paths relative to the Nix expression *expr*.",
+        .category = installablesCategory,
         .labels = {"expr"},
         .handler = {&expr}
     });
 
     addFlag({
-        .longName ="derivation",
-        .description = "operate on the store derivation rather than its outputs",
+        .longName = "derivation",
+        .description = "Operate on the store derivation rather than its outputs.",
+        .category = installablesCategory,
         .handler = {&operateOn, OperateOn::Derivation},
     });
 }
@@ -240,6 +256,12 @@ ref<EvalState> EvalCommand::getEvalState()
     return ref<EvalState>(evalState);
 }
 
+EvalCommand::~EvalCommand()
+{
+    if (evalState)
+        evalState->printStats();
+}
+
 void completeFlakeRef(ref<Store> store, std::string_view prefix)
 {
     if (prefix == "")
@@ -263,9 +285,9 @@ void completeFlakeRef(ref<Store> store, std::string_view prefix)
     }
 }
 
-Buildable Installable::toBuildable()
+DerivedPath Installable::toDerivedPath()
 {
-    auto buildables = toBuildables();
+    auto buildables = toDerivedPaths();
     if (buildables.size() != 1)
         throw Error("installable '%s' evaluates to %d derivations, where only one is expected", what(), buildables.size());
     return std::move(buildables[0]);
@@ -299,22 +321,19 @@ struct InstallableStorePath : Installable
 
     std::string what() override { return store->printStorePath(storePath); }
 
-    Buildables toBuildables() override
+    DerivedPaths toDerivedPaths() override
     {
         if (storePath.isDerivation()) {
-            std::map<std::string, std::optional<StorePath>> outputs;
             auto drv = store->readDerivation(storePath);
-            for (auto & [name, output] : drv.outputsAndOptPaths(*store))
-                outputs.emplace(name, output.second);
             return {
-                BuildableFromDrv {
+                DerivedPath::Built {
                     .drvPath = storePath,
-                    .outputs = std::move(outputs)
+                    .outputs = drv.outputNames(),
                 }
             };
         } else {
             return {
-                BuildableOpaque {
+                DerivedPath::Opaque {
                     .path = storePath,
                 }
             };
@@ -327,22 +346,22 @@ struct InstallableStorePath : Installable
     }
 };
 
-Buildables InstallableValue::toBuildables()
+DerivedPaths InstallableValue::toDerivedPaths()
 {
-    Buildables res;
+    DerivedPaths res;
 
-    std::map<StorePath, std::map<std::string, std::optional<StorePath>>> drvsToOutputs;
+    std::map<StorePath, std::set<std::string>> drvsToOutputs;
 
     // Group by derivation, helps with .all in particular
     for (auto & drv : toDerivations()) {
         auto outputName = drv.outputName;
         if (outputName == "")
             throw Error("derivation '%s' lacks an 'outputName' attribute", state->store->printStorePath(drv.drvPath));
-        drvsToOutputs[drv.drvPath].insert_or_assign(outputName, drv.outPath);
+        drvsToOutputs[drv.drvPath].insert(outputName);
     }
 
     for (auto & i : drvsToOutputs)
-        res.push_back(BuildableFromDrv { i.first, i.second });
+        res.push_back(DerivedPath::Built { i.first, i.second });
 
     return res;
 }
@@ -382,7 +401,7 @@ std::vector<InstallableValue::DerivationInfo> InstallableAttrPath::toDerivations
     for (auto & drvInfo : drvInfos) {
         res.push_back({
             state->store->parseStorePath(drvInfo.queryDrvPath()),
-            state->store->parseStorePath(drvInfo.queryOutPath()),
+            state->store->maybeParseStorePath(drvInfo.queryOutPath()),
             drvInfo.queryOutputName()
         });
     }
@@ -456,6 +475,23 @@ static std::string showAttrPaths(const std::vector<std::string> & paths)
     return s;
 }
 
+InstallableFlake::InstallableFlake(
+    SourceExprCommand * cmd,
+    ref<EvalState> state,
+    FlakeRef && flakeRef,
+    Strings && attrPaths,
+    Strings && prefixes,
+    const flake::LockFlags & lockFlags)
+    : InstallableValue(state),
+      flakeRef(flakeRef),
+      attrPaths(attrPaths),
+      prefixes(prefixes),
+      lockFlags(lockFlags)
+{
+    if (cmd && cmd->getAutoArgs(*state)->size())
+        throw UsageError("'--arg' and '--argstr' are incompatible with flakes");
+}
+
 std::tuple<std::string, FlakeRef, InstallableValue::DerivationInfo> InstallableFlake::toDerivation()
 {
     auto lockedFlake = getLockedFlake();
@@ -464,7 +500,11 @@ std::tuple<std::string, FlakeRef, InstallableValue::DerivationInfo> InstallableF
     auto root = cache->getRoot();
 
     for (auto & attrPath : getActualAttrPaths()) {
-        auto attr = root->findAlongAttrPath(parseAttrPath(*state, attrPath));
+        auto attr = root->findAlongAttrPath(
+            parseAttrPath(*state, attrPath),
+            true
+        );
+
         if (!attr) continue;
 
         if (!attr->isDerivation())
@@ -474,7 +514,7 @@ std::tuple<std::string, FlakeRef, InstallableValue::DerivationInfo> InstallableF
 
         auto drvInfo = DerivationInfo{
             std::move(drvPath),
-            state->store->parseStorePath(attr->getAttr(state->sOutPath)->getString()),
+            state->store->maybeParseStorePath(attr->getAttr(state->sOutPath)->getString()),
             attr->getAttr(state->sOutputName)->getString()
         };
 
@@ -533,8 +573,11 @@ InstallableFlake::getCursors(EvalState & state)
 
 std::shared_ptr<flake::LockedFlake> InstallableFlake::getLockedFlake() const
 {
-    if (!_lockedFlake)
+    if (!_lockedFlake) {
         _lockedFlake = std::make_shared<flake::LockedFlake>(lockFlake(*state, flakeRef, lockFlags));
+        _lockedFlake->flake.config.apply();
+        // FIXME: send new config to the daemon.
+    }
     return _lockedFlake;
 }
 
@@ -585,9 +628,12 @@ std::vector<std::shared_ptr<Installable>> SourceExprCommand::parseInstallables(
             try {
                 auto [flakeRef, fragment] = parseFlakeRefWithFragment(s, absPath("."));
                 result.push_back(std::make_shared<InstallableFlake>(
-                        getEvalState(), std::move(flakeRef),
+                        this,
+                        getEvalState(),
+                        std::move(flakeRef),
                         fragment == "" ? getDefaultFlakeAttrPaths() : Strings{fragment},
-                        getDefaultFlakeAttrPathPrefixes(), lockFlags));
+                        getDefaultFlakeAttrPathPrefixes(),
+                        lockFlags));
                 continue;
             } catch (...) {
                 ex = std::current_exception();
@@ -626,31 +672,67 @@ std::shared_ptr<Installable> SourceExprCommand::parseInstallable(
     return installables.front();
 }
 
-Buildables build(ref<Store> store, Realise mode,
+BuiltPaths getBuiltPaths(ref<Store> store, DerivedPaths hopefullyBuiltPaths)
+{
+    BuiltPaths res;
+    for (auto& b : hopefullyBuiltPaths)
+        std::visit(
+            overloaded{
+                [&](DerivedPath::Opaque bo) {
+                    res.push_back(BuiltPath::Opaque{bo.path});
+                },
+                [&](DerivedPath::Built bfd) {
+                    OutputPathMap outputs;
+                    auto drv = store->readDerivation(bfd.drvPath);
+                    auto outputHashes = staticOutputHashes(*store, drv);
+                    auto drvOutputs = drv.outputsAndOptPaths(*store);
+                    for (auto& output : bfd.outputs) {
+                        if (!outputHashes.count(output))
+                            throw Error(
+                                "the derivation '%s' doesn't have an output "
+                                "named '%s'",
+                                store->printStorePath(bfd.drvPath), output);
+                        if (settings.isExperimentalFeatureEnabled(
+                                "ca-derivations")) {
+                            auto outputId =
+                                DrvOutput{outputHashes.at(output), output};
+                            auto realisation =
+                                store->queryRealisation(outputId);
+                            if (!realisation)
+                                throw Error(
+                                    "cannot operate on an output of unbuilt "
+                                    "content-addresed derivation '%s'",
+                                    outputId.to_string());
+                            outputs.insert_or_assign(
+                                output, realisation->outPath);
+                        } else {
+                            // If ca-derivations isn't enabled, assume that
+                            // the output path is statically known.
+                            assert(drvOutputs.count(output));
+                            assert(drvOutputs.at(output).second);
+                            outputs.insert_or_assign(
+                                output, *drvOutputs.at(output).second);
+                        }
+                    }
+                    res.push_back(BuiltPath::Built{bfd.drvPath, outputs});
+                },
+            },
+            b.raw());
+
+    return res;
+}
+
+BuiltPaths build(ref<Store> store, Realise mode,
     std::vector<std::shared_ptr<Installable>> installables, BuildMode bMode)
 {
     if (mode == Realise::Nothing)
         settings.readOnlyMode = true;
 
-    Buildables buildables;
-
-    std::vector<StorePathWithOutputs> pathsToBuild;
+    std::vector<DerivedPath> pathsToBuild;
 
     for (auto & i : installables) {
-        for (auto & b : i->toBuildables()) {
-            std::visit(overloaded {
-                [&](BuildableOpaque bo) {
-                    pathsToBuild.push_back({bo.path});
-                },
-                [&](BuildableFromDrv bfd) {
-                    StringSet outputNames;
-                    for (auto & output : bfd.outputs)
-                        outputNames.insert(output.first);
-                    pathsToBuild.push_back({bfd.drvPath, outputNames});
-                },
-            }, b);
-            buildables.push_back(std::move(b));
-        }
+        auto b = i->toDerivedPaths();
+        pathsToBuild.insert(pathsToBuild.end(), b.begin(), b.end());
     }
 
     if (mode == Realise::Nothing)
@@ -658,7 +740,26 @@ Buildables build(ref<Store> store, Realise mode,
     else if (mode == Realise::Outputs)
         store->buildPaths(pathsToBuild, bMode);
 
-    return buildables;
+    return getBuiltPaths(store, pathsToBuild);
+}
+
+BuiltPaths toBuiltPaths(
+    ref<Store> store,
+    Realise mode,
+    OperateOn operateOn,
+    std::vector<std::shared_ptr<Installable>> installables)
+{
+    if (operateOn == OperateOn::Output) {
+        return build(store, mode, installables);
+    } else {
+        if (mode == Realise::Nothing)
+            settings.readOnlyMode = true;
+
+        BuiltPaths res;
+        for (auto & drvPath : toDerivations(store, installables, true))
+            res.push_back(BuiltPath::Opaque{drvPath});
+        return res;
+    }
 }
 
 StorePathSet toStorePaths(ref<Store> store,
@@ -666,31 +767,10 @@ StorePathSet toStorePaths(ref<Store> store,
     std::vector<std::shared_ptr<Installable>> installables)
 {
     StorePathSet outPaths;
-
-    if (operateOn == OperateOn::Output) {
-        for (auto & b : build(store, mode, installables))
-            std::visit(overloaded {
-                [&](BuildableOpaque bo) {
-                    outPaths.insert(bo.path);
-                },
-                [&](BuildableFromDrv bfd) {
-                    for (auto & output : bfd.outputs) {
-                        if (!output.second)
-                            throw Error("Cannot operate on output of unbuilt CA drv");
-                        outPaths.insert(*output.second);
-                    }
-                },
-            }, b);
-    } else {
-        if (mode == Realise::Nothing)
-            settings.readOnlyMode = true;
-
-        for (auto & i : installables)
-            for (auto & b : i->toBuildables())
-                if (auto bfd = std::get_if<BuildableFromDrv>(&b))
-                    outPaths.insert(bfd->drvPath);
+    for (auto & path : toBuiltPaths(store, mode, operateOn, installables)) {
+        auto thisOutPaths = path.outPaths();
+        outPaths.insert(thisOutPaths.begin(), thisOutPaths.end());
     }
-
     return outPaths;
 }
 
@@ -712,9 +792,9 @@ StorePathSet toDerivations(ref<Store> store,
     StorePathSet drvPaths;
 
     for (auto & i : installables)
-        for (auto & b : i->toBuildables())
+        for (auto & b : i->toDerivedPaths())
             std::visit(overloaded {
-                [&](BuildableOpaque bo) {
+                [&](DerivedPath::Opaque bo) {
                     if (!useDeriver)
                         throw Error("argument '%s' did not evaluate to a derivation", i->what());
                     auto derivers = store->queryValidDerivers(bo.path);
@@ -723,10 +803,10 @@ StorePathSet toDerivations(ref<Store> store,
                     // FIXME: use all derivers?
                     drvPaths.insert(*derivers.begin());
                 },
-                [&](BuildableFromDrv bfd) {
+                [&](DerivedPath::Built bfd) {
                     drvPaths.insert(bfd.drvPath);
                 },
-            }, b);
+            }, b.raw());
 
     return drvPaths;
 }

src/libcmd/installables.hh

@@ -2,6 +2,8 @@
 
 #include "util.hh"
 #include "path.hh"
+#include "path-with-outputs.hh"
+#include "derived-path.hh"
 #include "eval.hh"
 #include "flake/flake.hh"
 
@@ -14,22 +16,6 @@ struct SourceExprCommand;
 
 namespace eval_cache { class EvalCache; class AttrCursor; }
 
-struct BuildableOpaque {
-    StorePath path;
-};
-
-struct BuildableFromDrv {
-    StorePath drvPath;
-    std::map<std::string, std::optional<StorePath>> outputs;
-};
-
-typedef std::variant<
-    BuildableOpaque,
-    BuildableFromDrv
-> Buildable;
-
-typedef std::vector<Buildable> Buildables;
-
 struct App
 {
     std::vector<StorePathWithOutputs> context;
@@ -37,17 +23,23 @@ struct App
     // FIXME: add args, sandbox settings, metadata, ...
 };
 
+struct UnresolvedApp
+{
+    App unresolved;
+    App resolve(ref<Store>);
+};
+
 struct Installable
 {
     virtual ~Installable() { }
 
     virtual std::string what() = 0;
 
-    virtual Buildables toBuildables() = 0;
+    virtual DerivedPaths toDerivedPaths() = 0;
 
-    Buildable toBuildable();
+    DerivedPath toDerivedPath();
 
-    App toApp(EvalState & state);
+    UnresolvedApp toApp(EvalState & state);
 
     virtual std::pair<Value *, Pos> toValue(EvalState & state)
     {
@@ -88,7 +80,7 @@ struct InstallableValue : Installable
 
     virtual std::vector<DerivationInfo> toDerivations() = 0;
 
-    Buildables toBuildables() override;
+    DerivedPaths toDerivedPaths() override;
 };
 
 struct InstallableFlake : InstallableValue
@@ -99,11 +91,13 @@ struct InstallableFlake : InstallableValue
     const flake::LockFlags & lockFlags;
     mutable std::shared_ptr<flake::LockedFlake> _lockedFlake;
 
-    InstallableFlake(ref<EvalState> state, FlakeRef && flakeRef,
-        Strings && attrPaths, Strings && prefixes, const flake::LockFlags & lockFlags)
-        : InstallableValue(state), flakeRef(flakeRef), attrPaths(attrPaths),
-          prefixes(prefixes), lockFlags(lockFlags)
-    { }
+    InstallableFlake(
+        SourceExprCommand * cmd,
+        ref<EvalState> state,
+        FlakeRef && flakeRef,
+        Strings && attrPaths,
+        Strings && prefixes,
+        const flake::LockFlags & lockFlags);
 
     std::string what() override { return flakeRef.to_string() + "#" + *attrPaths.begin(); }
 

src/libcmd/local.mk

@@ -0,0 +1,15 @@
+libraries += libcmd
+
+libcmd_NAME = libnixcmd
+
+libcmd_DIR := $(d)
+
+libcmd_SOURCES := $(wildcard $(d)/*.cc)
+
+libcmd_CXXFLAGS += -I src/libutil -I src/libstore -I src/libexpr -I src/libmain -I src/libfetchers
+
+libcmd_LDFLAGS = -llowdown
+
+libcmd_LIBS = libstore libutil libexpr libmain libfetchers
+
+$(eval $(call install-file-in, $(d)/nix-cmd.pc, $(prefix)/lib/pkgconfig, 0644))

src/libcmd/markdown.cc

@@ -3,9 +3,7 @@
 #include "finally.hh"
 
 #include <sys/queue.h>
-extern "C" {
 #include <lowdown.h>
-}
 
 namespace nix {
 
@@ -42,7 +40,9 @@ std::string renderMarkdownToTerminal(std::string_view markdown)
         throw Error("cannot allocate Markdown output buffer");
     Finally freeBuffer([&]() { lowdown_buf_free(buf); });
 
-    lowdown_term_rndr(buf, nullptr, renderer, node);
+    int rndr_res = lowdown_term_rndr(buf, nullptr, renderer, node);
+    if (!rndr_res)
+        throw Error("allocation error while rendering Markdown");
 
     return std::string(buf->data, buf->size);
 }

src/libcmd/nix-cmd.pc.in

@@ -0,0 +1,9 @@
+prefix=@prefix@
+libdir=@libdir@
+includedir=@includedir@
+
+Name: Nix
+Description: Nix Package Manager
+Version: @PACKAGE_VERSION@
+Libs: -L${libdir} -lnixcmd
+Cflags: -I${includedir}/nix -std=c++17

src/libexpr/attr-path.cc

@@ -52,9 +52,7 @@ std::pair<Value *, Pos> findAlongAttrPath(EvalState & state, const string & attr
     for (auto & attr : tokens) {
 
         /* Is i an index (integer) or a normal attribute name? */
-        enum { apAttr, apIndex } apType = apAttr;
-        unsigned int attrIndex;
-        if (string2Int(attr, attrIndex)) apType = apIndex;
+        auto attrIndex = string2Int<unsigned int>(attr);
 
         /* Evaluate the expression. */
         Value * vNew = state.allocValue();
@@ -65,9 +63,9 @@ std::pair<Value *, Pos> findAlongAttrPath(EvalState & state, const string & attr
         /* It should evaluate to either a set or an expression,
            according to what is specified in the attrPath. */
 
-        if (apType == apAttr) {
+        if (!attrIndex) {
 
-            if (v->type != tAttrs)
+            if (v->type() != nAttrs)
                 throw TypeError(
                     "the expression selected by the selection path '%1%' should be a set but is %2%",
                     attrPath,
@@ -82,17 +80,17 @@ std::pair<Value *, Pos> findAlongAttrPath(EvalState & state, const string & attr
             pos = *a->pos;
         }
 
-        else if (apType == apIndex) {
+        else {
 
             if (!v->isList())
                 throw TypeError(
                     "the expression selected by the selection path '%1%' should be a list but is %2%",
                     attrPath,
                     showType(*v));
-            if (attrIndex >= v->listSize())
-                throw AttrPathNotFound("list index %1% in selection path '%2%' is out of range", attrIndex, attrPath);
+            if (*attrIndex >= v->listSize())
+                throw AttrPathNotFound("list index %1% in selection path '%2%' is out of range", *attrIndex, attrPath);
 
-            v = v->listElems()[attrIndex];
+            v = v->listElems()[*attrIndex];
             pos = noPos;
         }
 

src/libexpr/attr-set.cc

@@ -24,9 +24,7 @@ void EvalState::mkAttrs(Value & v, size_t capacity)
         v = vEmptySet;
         return;
     }
-    clearValue(v);
-    v.type = tAttrs;
-    v.attrs = allocBindings(capacity);
+    v.mkAttrs(allocBindings(capacity));
     nrAttrsets++;
     nrAttrsInAttrsets += capacity;
 }

src/libexpr/attr-set.hh

@@ -35,6 +35,7 @@ class Bindings
 {
 public:
     typedef uint32_t size_t;
+    Pos *pos;
 
 private:
     size_t size_, capacity_;
@@ -77,7 +78,7 @@ public:
         auto a = get(name);
         if (!a)
             throw Error({
-                .hint = hintfmt("attribute '%s' missing", name),
+                .msg = hintfmt("attribute '%s' missing", name),
                 .errPos = pos
             });
 

src/libexpr/common-eval-args.cc

@@ -12,16 +12,20 @@ namespace nix {
 
 MixEvalArgs::MixEvalArgs()
 {
+    auto category = "Common evaluation options";
+
     addFlag({
         .longName = "arg",
-        .description = "argument to be passed to Nix functions",
+        .description = "Pass the value *expr* as the argument *name* to Nix functions.",
+        .category = category,
         .labels = {"name", "expr"},
         .handler = {[&](std::string name, std::string expr) { autoArgs[name] = 'E' + expr; }}
     });
 
     addFlag({
         .longName = "argstr",
-        .description = "string-valued argument to be passed to Nix functions",
+        .description = "Pass the string *string* as the argument *name* to Nix functions.",
+        .category = category,
         .labels = {"name", "string"},
         .handler = {[&](std::string name, std::string s) { autoArgs[name] = 'S' + s; }},
     });
@@ -29,14 +33,16 @@ MixEvalArgs::MixEvalArgs()
     addFlag({
         .longName = "include",
         .shortName = 'I',
-        .description = "add a path to the list of locations used to look up `<...>` file names",
+        .description = "Add *path* to the list of locations used to look up `<...>` file names.",
+        .category = category,
         .labels = {"path"},
         .handler = {[&](std::string s) { searchPath.push_back(s); }}
     });
 
     addFlag({
         .longName = "impure",
-        .description = "allow access to mutable paths and repositories",
+        .description = "Allow access to mutable paths and repositories.",
+        .category = category,
         .handler = {[&]() {
             evalSettings.pureEval = false;
         }},
@@ -44,7 +50,8 @@ MixEvalArgs::MixEvalArgs()
 
     addFlag({
         .longName = "override-flake",
-        .description = "override a flake registry value",
+        .description = "Override the flake registries, redirecting *original-ref* to *resolved-ref*.",
+        .category = category,
         .labels = {"original-ref", "resolved-ref"},
         .handler = {[&](std::string _from, std::string _to) {
             auto from = parseFlakeRef(_from, absPath("."));

src/libexpr/eval-cache.cc

@@ -390,14 +390,14 @@ Value & AttrCursor::forceValue()
     }
 
     if (root->db && (!cachedValue || std::get_if<placeholder_t>(&cachedValue->second))) {
-        if (v.type == tString)
+        if (v.type() == nString)
             cachedValue = {root->db->setString(getKey(), v.string.s, v.string.context),
                            string_t{v.string.s, {}}};
-        else if (v.type == tPath)
-            cachedValue = {root->db->setString(getKey(), v.path), v.path};
-        else if (v.type == tBool)
+        else if (v.type() == nPath)
+            cachedValue = {root->db->setString(getKey(), v.path), string_t{v.path, {}}};
+        else if (v.type() == nBool)
             cachedValue = {root->db->setBool(getKey(), v.boolean), v.boolean};
-        else if (v.type == tAttrs)
+        else if (v.type() == nAttrs)
             ; // FIXME: do something?
         else
             cachedValue = {root->db->setMisc(getKey()), misc_t()};
@@ -442,7 +442,7 @@ std::shared_ptr<AttrCursor> AttrCursor::maybeGetAttr(Symbol name, bool forceErro
 
     auto & v = forceValue();
 
-    if (v.type != tAttrs)
+    if (v.type() != nAttrs)
         return nullptr;
         //throw TypeError("'%s' is not an attribute set", getAttrPathStr());
 
@@ -486,11 +486,11 @@ std::shared_ptr<AttrCursor> AttrCursor::getAttr(std::string_view name)
     return getAttr(root->state.symbols.create(name));
 }
 
-std::shared_ptr<AttrCursor> AttrCursor::findAlongAttrPath(const std::vector<Symbol> & attrPath)
+std::shared_ptr<AttrCursor> AttrCursor::findAlongAttrPath(const std::vector<Symbol> & attrPath, bool force)
 {
     auto res = shared_from_this();
     for (auto & attr : attrPath) {
-        res = res->maybeGetAttr(attr);
+        res = res->maybeGetAttr(attr, force);
         if (!res) return {};
     }
     return res;
@@ -512,10 +512,10 @@ std::string AttrCursor::getString()
 
     auto & v = forceValue();
 
-    if (v.type != tString && v.type != tPath)
-        throw TypeError("'%s' is not a string but %s", getAttrPathStr(), showType(v.type));
+    if (v.type() != nString && v.type() != nPath)
+        throw TypeError("'%s' is not a string but %s", getAttrPathStr(), showType(v.type()));
 
-    return v.type == tString ? v.string.s : v.path;
+    return v.type() == nString ? v.string.s : v.path;
 }
 
 string_t AttrCursor::getStringWithContext()
@@ -525,8 +525,17 @@ string_t AttrCursor::getStringWithContext()
             cachedValue = root->db->getAttr(getKey(), root->state.symbols);
         if (cachedValue && !std::get_if<placeholder_t>(&cachedValue->second)) {
             if (auto s = std::get_if<string_t>(&cachedValue->second)) {
-                debug("using cached string attribute '%s'", getAttrPathStr());
-                return *s;
+                bool valid = true;
+                for (auto & c : s->second) {
+                    if (!root->state.store->isValidPath(root->state.store->parseStorePath(c.first))) {
+                        valid = false;
+                        break;
+                    }
+                }
+                if (valid) {
+                    debug("using cached string attribute '%s'", getAttrPathStr());
+                    return *s;
+                }
             } else
                 throw TypeError("'%s' is not a string", getAttrPathStr());
         }
@@ -534,12 +543,12 @@ string_t AttrCursor::getStringWithContext()
 
     auto & v = forceValue();
 
-    if (v.type == tString)
+    if (v.type() == nString)
         return {v.string.s, v.getContext()};
-    else if (v.type == tPath)
+    else if (v.type() == nPath)
         return {v.path, {}};
     else
-        throw TypeError("'%s' is not a string but %s", getAttrPathStr(), showType(v.type));
+        throw TypeError("'%s' is not a string but %s", getAttrPathStr(), showType(v.type()));
 }
 
 bool AttrCursor::getBool()
@@ -558,7 +567,7 @@ bool AttrCursor::getBool()
 
     auto & v = forceValue();
 
-    if (v.type != tBool)
+    if (v.type() != nBool)
         throw TypeError("'%s' is not a Boolean", getAttrPathStr());
 
     return v.boolean;
@@ -580,7 +589,7 @@ std::vector<Symbol> AttrCursor::getAttrs()
 
     auto & v = forceValue();
 
-    if (v.type != tAttrs)
+    if (v.type() != nAttrs)
         throw TypeError("'%s' is not an attribute set", getAttrPathStr());
 
     std::vector<Symbol> attrs;

src/libexpr/eval-cache.hh

@@ -102,7 +102,7 @@ public:
 
     std::shared_ptr<AttrCursor> getAttr(std::string_view name);
 
-    std::shared_ptr<AttrCursor> findAlongAttrPath(const std::vector<Symbol> & attrPath);
+    std::shared_ptr<AttrCursor> findAlongAttrPath(const std::vector<Symbol> & attrPath, bool force = false);
 
     std::string getString();
 

src/libexpr/eval-inline.hh

@@ -10,7 +10,7 @@ namespace nix {
 LocalNoInlineNoReturn(void throwEvalError(const Pos & pos, const char * s))
 {
     throw EvalError({
-        .hint = hintfmt(s),
+        .msg = hintfmt(s),
         .errPos = pos
     });
 }
@@ -24,7 +24,7 @@ LocalNoInlineNoReturn(void throwTypeError(const char * s, const Value & v))
 LocalNoInlineNoReturn(void throwTypeError(const Pos & pos, const char * s, const Value & v))
 {
     throw TypeError({
-        .hint = hintfmt(s, showType(v)),
+        .msg = hintfmt(s, showType(v)),
         .errPos = pos
     });
 }
@@ -32,23 +32,21 @@ LocalNoInlineNoReturn(void throwTypeError(const Pos & pos, const char * s, const
 
 void EvalState::forceValue(Value & v, const Pos & pos)
 {
-    if (v.type == tThunk) {
+    if (v.isThunk()) {
         Env * env = v.thunk.env;
         Expr * expr = v.thunk.expr;
         try {
-            v.type = tBlackhole;
+            v.mkBlackhole();
             //checkInterrupt();
             expr->eval(*this, *env, v);
         } catch (...) {
-            v.type = tThunk;
-            v.thunk.env = env;
-            v.thunk.expr = expr;
+            v.mkThunk(env, expr);
             throw;
         }
     }
-    else if (v.type == tApp)
+    else if (v.isApp())
         callFunction(*v.app.left, *v.app.right, v, noPos);
-    else if (v.type == tBlackhole)
+    else if (v.isBlackhole())
         throwEvalError(pos, "infinite recursion encountered");
 }
 
@@ -56,7 +54,7 @@ void EvalState::forceValue(Value & v, const Pos & pos)
 inline void EvalState::forceAttrs(Value & v)
 {
     forceValue(v);
-    if (v.type != tAttrs)
+    if (v.type() != nAttrs)
         throwTypeError("value is %1% while a set was expected", v);
 }
 
@@ -64,7 +62,7 @@ inline void EvalState::forceAttrs(Value & v)
 inline void EvalState::forceAttrs(Value & v, const Pos & pos)
 {
     forceValue(v, pos);
-    if (v.type != tAttrs)
+    if (v.type() != nAttrs)
         throwTypeError(pos, "value is %1% while a set was expected", v);
 }
 

src/libexpr/eval.cc

@@ -27,6 +27,10 @@
 #include <gc/gc.h>
 #include <gc/gc_cpp.h>
 
+#include <boost/coroutine2/coroutine.hpp>
+#include <boost/coroutine2/protected_fixedsize_stack.hpp>
+#include <boost/context/stack_context.hpp>
+
 #endif
 
 namespace nix {
@@ -64,7 +68,7 @@ RootValue allocRootValue(Value * v)
 }
 
 
-static void printValue(std::ostream & str, std::set<const Value *> & active, const Value & v)
+void printValue(std::ostream & str, std::set<const Value *> & active, const Value & v)
 {
     checkInterrupt();
 
@@ -73,7 +77,7 @@ static void printValue(std::ostream & str, std::set<const Value *> & active, con
         return;
     }
 
-    switch (v.type) {
+    switch (v.internalType) {
     case tInt:
         str << v.integer;
         break;
@@ -154,32 +158,27 @@ std::ostream & operator << (std::ostream & str, const Value & v)
 
 const Value *getPrimOp(const Value &v) {
     const Value * primOp = &v;
-    while (primOp->type == tPrimOpApp) {
+    while (primOp->isPrimOpApp()) {
         primOp = primOp->primOpApp.left;
     }
-    assert(primOp->type == tPrimOp);
+    assert(primOp->isPrimOp());
     return primOp;
 }
 
-
 string showType(ValueType type)
 {
     switch (type) {
-        case tInt: return "an integer";
-        case tBool: return "a Boolean";
-        case tString: return "a string";
-        case tPath: return "a path";
-        case tNull: return "null";
-        case tAttrs: return "a set";
-        case tList1: case tList2: case tListN: return "a list";
-        case tThunk: return "a thunk";
-        case tApp: return "a function application";
-        case tLambda: return "a function";
-        case tBlackhole: return "a black hole";
-        case tPrimOp: return "a built-in function";
-        case tPrimOpApp: return "a partially applied built-in function";
-        case tExternal: return "an external value";
-        case tFloat: return "a float";
+        case nInt: return "an integer";
+        case nBool: return "a Boolean";
+        case nString: return "a string";
+        case nPath: return "a path";
+        case nNull: return "null";
+        case nAttrs: return "a set";
+        case nList: return "a list";
+        case nFunction: return "a function";
+        case nExternal: return "an external value";
+        case nFloat: return "a float";
+        case nThunk: return "a thunk";
     }
     abort();
 }
@@ -187,28 +186,41 @@ string showType(ValueType type)
 
 string showType(const Value & v)
 {
-    switch (v.type) {
+    switch (v.internalType) {
         case tString: return v.string.context ? "a string with context" : "a string";
         case tPrimOp:
             return fmt("the built-in function '%s'", string(v.primOp->name));
         case tPrimOpApp:
             return fmt("the partially applied built-in function '%s'", string(getPrimOp(v)->primOp->name));
         case tExternal: return v.external->showType();
+        case tThunk: return "a thunk";
+        case tApp: return "a function application";
+        case tBlackhole: return "a black hole";
     default:
-        return showType(v.type);
+        return showType(v.type());
     }
 }
 
+Pos Value::determinePos(const Pos &pos) const
+{
+    switch (internalType) {
+        case tAttrs: return *attrs->pos;
+        case tLambda: return lambda.fun->pos;
+        case tApp: return app.left->determinePos(pos);
+        default: return pos;
+    }
+}
 
 bool Value::isTrivial() const
 {
     return
-        type != tApp
-        && type != tPrimOpApp
-        && (type != tThunk
+        internalType != tApp
+        && internalType != tPrimOpApp
+        && (internalType != tThunk
             || (dynamic_cast<ExprAttrs *>(thunk.expr)
                 && ((ExprAttrs *) thunk.expr)->dynamicAttrs.empty())
-            || dynamic_cast<ExprLambda *>(thunk.expr));
+            || dynamic_cast<ExprLambda *>(thunk.expr)
+            || dynamic_cast<ExprList *>(thunk.expr));
 }
 
 
@@ -219,6 +231,31 @@ static void * oomHandler(size_t requested)
     /* Convert this to a proper C++ exception. */
     throw std::bad_alloc();
 }
+
+class BoehmGCStackAllocator : public StackAllocator {
+  boost::coroutines2::protected_fixedsize_stack stack {
+    // We allocate 8 MB, the default max stack size on NixOS.
+    // A smaller stack might be quicker to allocate but reduces the stack
+    // depth available for source filter expressions etc.
+    std::max(boost::context::stack_traits::default_size(), static_cast<std::size_t>(8 * 1024 * 1024))
+    };
+
+  public:
+    boost::context::stack_context allocate() override {
+        auto sctx = stack.allocate();
+        GC_add_roots(static_cast<char *>(sctx.sp) - sctx.size, sctx.sp);
+        return sctx;
+    }
+
+    void deallocate(boost::context::stack_context sctx) override {
+        GC_remove_roots(static_cast<char *>(sctx.sp) - sctx.size, sctx.sp);
+        stack.deallocate(sctx);
+    }
+
+};
+
+static BoehmGCStackAllocator boehmGCStackAllocator;
+
 #endif
 
 
@@ -256,6 +293,8 @@ void initGC()
 
     GC_set_oom_fn(oomHandler);
 
+    StackAllocator::defaultAllocator = &boehmGCStackAllocator;
+
     /* Set the initial heap size to something fairly big (25% of
        physical RAM, up to a maximum of 384 MiB) so that in most cases
        we don't need to garbage collect at all.  (Collection has a
@@ -372,11 +411,6 @@ EvalState::EvalState(const Strings & _searchPath, ref<Store> store)
         for (auto & i : evalSettings.nixPath.get()) addToSearchPath(i);
     }
 
-    try {
-        addToSearchPath("nix=" + canonPath(settings.nixDataDir + "/nix/corepkgs", true));
-    } catch (Error &) {
-    }
-
     if (evalSettings.restrictEval || evalSettings.pureEval) {
         allowedPaths = PathSet();
 
@@ -400,9 +434,7 @@ EvalState::EvalState(const Strings & _searchPath, ref<Store> store)
         }
     }
 
-    clearValue(vEmptySet);
-    vEmptySet.type = tAttrs;
-    vEmptySet.attrs = allocBindings(0);
+    vEmptySet.mkAttrs(allocBindings(0));
 
     createBaseEnv();
 }
@@ -429,6 +461,8 @@ Path EvalState::checkSourcePath(const Path & path_)
      */
     Path abspath = canonPath(path_);
 
+    if (hasPrefix(abspath, corepkgsPrefix)) return abspath;
+
     for (auto & i : *allowedPaths) {
         if (isDirOrInDir(abspath, i)) {
             found = true;
@@ -518,16 +552,14 @@ Value * EvalState::addPrimOp(const string & name,
        the primop to a dummy value. */
     if (arity == 0) {
         auto vPrimOp = allocValue();
-        vPrimOp->type = tPrimOp;
-        vPrimOp->primOp = new PrimOp { .fun = primOp, .arity = 1, .name = sym };
+        vPrimOp->mkPrimOp(new PrimOp { .fun = primOp, .arity = 1, .name = sym });
         Value v;
         mkApp(v, *vPrimOp, *vPrimOp);
         return addConstant(name, v);
     }
 
     Value * v = allocValue();
-    v->type = tPrimOp;
-    v->primOp = new PrimOp { .fun = primOp, .arity = arity, .name = sym };
+    v->mkPrimOp(new PrimOp { .fun = primOp, .arity = arity, .name = sym });
     staticBaseEnv.vars[symbols.create(name)] = baseEnvDispl;
     baseEnv.values[baseEnvDispl++] = v;
     baseEnv.values[0]->attrs->push_back(Attr(sym, v));
@@ -542,8 +574,7 @@ Value * EvalState::addPrimOp(PrimOp && primOp)
     if (primOp.arity == 0) {
         primOp.arity = 1;
         auto vPrimOp = allocValue();
-        vPrimOp->type = tPrimOp;
-        vPrimOp->primOp = new PrimOp(std::move(primOp));
+        vPrimOp->mkPrimOp(new PrimOp(std::move(primOp)));
         Value v;
         mkApp(v, *vPrimOp, *vPrimOp);
         return addConstant(primOp.name, v);
@@ -554,8 +585,7 @@ Value * EvalState::addPrimOp(PrimOp && primOp)
         primOp.name = symbols.create(std::string(primOp.name, 2));
 
     Value * v = allocValue();
-    v->type = tPrimOp;
-    v->primOp = new PrimOp(std::move(primOp));
+    v->mkPrimOp(new PrimOp(std::move(primOp)));
     staticBaseEnv.vars[envName] = baseEnvDispl;
     baseEnv.values[baseEnvDispl++] = v;
     baseEnv.values[0]->attrs->push_back(Attr(primOp.name, v));
@@ -571,10 +601,8 @@ Value & EvalState::getBuiltin(const string & name)
 
 std::optional<EvalState::Doc> EvalState::getDoc(Value & v)
 {
-    if (v.type == tPrimOp || v.type == tPrimOpApp) {
+    if (v.isPrimOp()) {
         auto v2 = &v;
-        while (v2->type == tPrimOpApp)
-            v2 = v2->primOpApp.left;
         if (v2->primOp->doc)
             return Doc {
                 .pos = noPos,
@@ -601,7 +629,7 @@ LocalNoInlineNoReturn(void throwEvalError(const char * s, const string & s2))
 LocalNoInlineNoReturn(void throwEvalError(const Pos & pos, const char * s, const string & s2))
 {
     throw EvalError({
-        .hint = hintfmt(s, s2),
+        .msg = hintfmt(s, s2),
         .errPos = pos
     });
 }
@@ -614,7 +642,7 @@ LocalNoInlineNoReturn(void throwEvalError(const char * s, const string & s2, con
 LocalNoInlineNoReturn(void throwEvalError(const Pos & pos, const char * s, const string & s2, const string & s3))
 {
     throw EvalError({
-        .hint = hintfmt(s, s2, s3),
+        .msg = hintfmt(s, s2, s3),
         .errPos = pos
     });
 }
@@ -623,7 +651,7 @@ LocalNoInlineNoReturn(void throwEvalError(const Pos & p1, const char * s, const
 {
     // p1 is where the error occurred; p2 is a position mentioned in the message.
     throw EvalError({
-        .hint = hintfmt(s, sym, p2),
+        .msg = hintfmt(s, sym, p2),
         .errPos = p1
     });
 }
@@ -631,20 +659,15 @@ LocalNoInlineNoReturn(void throwEvalError(const Pos & p1, const char * s, const
 LocalNoInlineNoReturn(void throwTypeError(const Pos & pos, const char * s))
 {
     throw TypeError({
-        .hint = hintfmt(s),
+        .msg = hintfmt(s),
         .errPos = pos
     });
 }
 
-LocalNoInlineNoReturn(void throwTypeError(const char * s, const string & s1))
-{
-    throw TypeError(s, s1);
-}
-
 LocalNoInlineNoReturn(void throwTypeError(const Pos & pos, const char * s, const ExprLambda & fun, const Symbol & s2))
 {
     throw TypeError({
-        .hint = hintfmt(s, fun.showNamePos(), s2),
+        .msg = hintfmt(s, fun.showNamePos(), s2),
         .errPos = pos
     });
 }
@@ -652,7 +675,7 @@ LocalNoInlineNoReturn(void throwTypeError(const Pos & pos, const char * s, const
 LocalNoInlineNoReturn(void throwAssertionError(const Pos & pos, const char * s, const string & s1))
 {
     throw AssertionError({
-        .hint = hintfmt(s, s1),
+        .msg = hintfmt(s, s1),
         .errPos = pos
     });
 }
@@ -660,7 +683,15 @@ LocalNoInlineNoReturn(void throwAssertionError(const Pos & pos, const char * s,
 LocalNoInlineNoReturn(void throwUndefinedVarError(const Pos & pos, const char * s, const string & s1))
 {
     throw UndefinedVarError({
-        .hint = hintfmt(s, s1),
+        .msg = hintfmt(s, s1),
+        .errPos = pos
+    });
+}
+
+LocalNoInlineNoReturn(void throwMissingArgumentError(const Pos & pos, const char * s, const string & s1))
+{
+    throw MissingArgumentError({
+        .msg = hintfmt(s, s1),
         .errPos = pos
     });
 }
@@ -678,15 +709,13 @@ LocalNoInline(void addErrorTrace(Error & e, const Pos & pos, const char * s, con
 
 void mkString(Value & v, const char * s)
 {
-    mkStringNoCopy(v, dupString(s));
+    v.mkString(dupString(s));
 }
 
 
 Value & mkString(Value & v, std::string_view s, const PathSet & context)
 {
-    v.type = tString;
-    v.string.s = dupStringWithLen(s.data(), s.size());
-    v.string.context = 0;
+    v.mkString(dupStringWithLen(s.data(), s.size()));
     if (!context.empty()) {
         size_t n = 0;
         v.string.context = (const char * *)
@@ -701,7 +730,7 @@ Value & mkString(Value & v, std::string_view s, const PathSet & context)
 
 void mkPath(Value & v, const char * s)
 {
-    mkPathNoCopy(v, dupString(s));
+    v.mkPath(dupString(s));
 }
 
 
@@ -762,16 +791,9 @@ Env & EvalState::allocEnv(size_t size)
 
 void EvalState::mkList(Value & v, size_t size)
 {
-    clearValue(v);
-    if (size == 1)
-        v.type = tList1;
-    else if (size == 2)
-        v.type = tList2;
-    else {
-        v.type = tListN;
-        v.bigList.size = size;
-        v.bigList.elems = size ? (Value * *) allocBytes(size * sizeof(Value *)) : 0;
-    }
+    v.mkList(size);
+    if (size > 2)
+        v.bigList.elems = (Value * *) allocBytes(size * sizeof(Value *));
     nrListElems += size;
 }
 
@@ -780,9 +802,7 @@ unsigned long nrThunks = 0;
 
 static inline void mkThunk(Value & v, Env & env, Expr * expr)
 {
-    v.type = tThunk;
-    v.thunk.env = &env;
-    v.thunk.expr = expr;
+    v.mkThunk(&env, expr);
     nrThunks++;
 }
 
@@ -917,7 +937,7 @@ inline bool EvalState::evalBool(Env & env, Expr * e)
 {
     Value v;
     e->eval(*this, env, v);
-    if (v.type != tBool)
+    if (v.type() != nBool)
         throwTypeError("value is %1% while a Boolean was expected", v);
     return v.boolean;
 }
@@ -927,7 +947,7 @@ inline bool EvalState::evalBool(Env & env, Expr * e, const Pos & pos)
 {
     Value v;
     e->eval(*this, env, v);
-    if (v.type != tBool)
+    if (v.type() != nBool)
         throwTypeError(pos, "value is %1% while a Boolean was expected", v);
     return v.boolean;
 }
@@ -936,7 +956,7 @@ inline bool EvalState::evalBool(Env & env, Expr * e, const Pos & pos)
 inline void EvalState::evalAttrs(Env & env, Expr * e, Value & v)
 {
     e->eval(*this, env, v);
-    if (v.type != tAttrs)
+    if (v.type() != nAttrs)
         throwTypeError("value is %1% while a set was expected", v);
 }
 
@@ -1036,7 +1056,7 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
         Value nameVal;
         i.nameExpr->eval(state, *dynamicEnv, nameVal);
         state.forceValue(nameVal, i.pos);
-        if (nameVal.type == tNull)
+        if (nameVal.type() == nNull)
             continue;
         state.forceStringNoCtx(nameVal);
         Symbol nameSym = state.symbols.create(nameVal.string.s);
@@ -1049,6 +1069,8 @@ void ExprAttrs::eval(EvalState & state, Env & env, Value & v)
         v.attrs->push_back(Attr(nameSym, i.valueExpr->maybeThunk(state, *dynamicEnv), &i.pos));
         v.attrs->sort(); // FIXME: inefficient
     }
+
+    v.attrs->pos = &pos;
 }
 
 
@@ -1121,7 +1143,7 @@ void ExprSelect::eval(EvalState & state, Env & env, Value & v)
             Symbol name = getName(i, state, env);
             if (def) {
                 state.forceValue(*vAttrs, pos);
-                if (vAttrs->type != tAttrs ||
+                if (vAttrs->type() != nAttrs ||
                     (j = vAttrs->attrs->find(name)) == vAttrs->attrs->end())
                 {
                     def->eval(state, env, v);
@@ -1161,7 +1183,7 @@ void ExprOpHasAttr::eval(EvalState & state, Env & env, Value & v)
         state.forceValue(*vAttrs);
         Bindings::iterator j;
         Symbol name = getName(i, state, env);
-        if (vAttrs->type != tAttrs ||
+        if (vAttrs->type() != nAttrs ||
             (j = vAttrs->attrs->find(name)) == vAttrs->attrs->end())
         {
             mkBool(v, false);
@@ -1177,9 +1199,7 @@ void ExprOpHasAttr::eval(EvalState & state, Env & env, Value & v)
 
 void ExprLambda::eval(EvalState & state, Env & env, Value & v)
 {
-    v.type = tLambda;
-    v.lambda.env = &env;
-    v.lambda.fun = this;
+    v.mkLambda(&env, this);
 }
 
 
@@ -1197,11 +1217,11 @@ void EvalState::callPrimOp(Value & fun, Value & arg, Value & v, const Pos & pos)
     /* Figure out the number of arguments still needed. */
     size_t argsDone = 0;
     Value * primOp = &fun;
-    while (primOp->type == tPrimOpApp) {
+    while (primOp->isPrimOpApp()) {
         argsDone++;
         primOp = primOp->primOpApp.left;
     }
-    assert(primOp->type == tPrimOp);
+    assert(primOp->isPrimOp());
     auto arity = primOp->primOp->arity;
     auto argsLeft = arity - argsDone;
 
@@ -1212,7 +1232,7 @@ void EvalState::callPrimOp(Value & fun, Value & arg, Value & v, const Pos & pos)
         Value * vArgs[arity];
         auto n = arity - 1;
         vArgs[n--] = &arg;
-        for (Value * arg = &fun; arg->type == tPrimOpApp; arg = arg->primOpApp.left)
+        for (Value * arg = &fun; arg->isPrimOpApp(); arg = arg->primOpApp.left)
             vArgs[n--] = arg->primOpApp.right;
 
         /* And call the primop. */
@@ -1222,9 +1242,7 @@ void EvalState::callPrimOp(Value & fun, Value & arg, Value & v, const Pos & pos)
     } else {
         Value * fun2 = allocValue();
         *fun2 = fun;
-        v.type = tPrimOpApp;
-        v.primOpApp.left = fun2;
-        v.primOpApp.right = &arg;
+        v.mkPrimOpApp(fun2, &arg);
     }
 }
 
@@ -1234,12 +1252,12 @@ void EvalState::callFunction(Value & fun, Value & arg, Value & v, const Pos & po
 
     forceValue(fun, pos);
 
-    if (fun.type == tPrimOp || fun.type == tPrimOpApp) {
+    if (fun.isPrimOp() || fun.isPrimOpApp()) {
         callPrimOp(fun, arg, v, pos);
         return;
     }
 
-    if (fun.type == tAttrs) {
+    if (fun.type() == nAttrs) {
       auto found = fun.attrs->find(sFunctor);
       if (found != fun.attrs->end()) {
         /* fun may be allocated on the stack of the calling function,
@@ -1255,7 +1273,7 @@ void EvalState::callFunction(Value & fun, Value & arg, Value & v, const Pos & po
       }
     }
 
-    if (fun.type != tLambda)
+    if (!fun.isLambda())
         throwTypeError(pos, "attempt to call something which is not a function but %1%", fun);
 
     ExprLambda & lambda(*fun.lambda.fun);
@@ -1338,7 +1356,7 @@ void EvalState::autoCallFunction(Bindings & args, Value & fun, Value & res)
 {
     forceValue(fun);
 
-    if (fun.type == tAttrs) {
+    if (fun.type() == nAttrs) {
         auto found = fun.attrs->find(sFunctor);
         if (found != fun.attrs->end()) {
             Value * v = allocValue();
@@ -1348,7 +1366,7 @@ void EvalState::autoCallFunction(Bindings & args, Value & fun, Value & res)
         }
     }
 
-    if (fun.type != tLambda || !fun.lambda.fun->matchAttrs) {
+    if (!fun.isLambda() || !fun.lambda.fun->matchAttrs) {
         res = fun;
         return;
     }
@@ -1370,7 +1388,13 @@ void EvalState::autoCallFunction(Bindings & args, Value & fun, Value & res)
             if (j != args.end()) {
                 actualArgs->attrs->push_back(*j);
             } else if (!i.def) {
-                throwTypeError("cannot auto-call a function that has an argument without a default value ('%1%')", i.name);
+                throwMissingArgumentError(i.pos, R"(cannot evaluate a function that has an argument without a value ('%1%')
+
+Nix attempted to evaluate a function as a top level expression; in
+this case it must have its arguments supplied either by default
+values, or passed explicitly with '--arg' or '--argstr'. See
+https://nixos.org/manual/nix/stable/#ss-functions.)", i.name);
+
             }
         }
     }
@@ -1404,7 +1428,7 @@ void ExprAssert::eval(EvalState & state, Env & env, Value & v)
     if (!state.evalBool(env, cond, pos)) {
         std::ostringstream out;
         cond->show(out);
-        throwAssertionError(pos, "assertion '%1%' failed at %2%", out.str());
+        throwAssertionError(pos, "assertion '%1%' failed", out.str());
     }
     body->eval(state, env, v);
 }
@@ -1532,7 +1556,7 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
     NixFloat nf = 0;
 
     bool first = !forceString;
-    ValueType firstType = tString;
+    ValueType firstType = nString;
 
     for (auto & i : *es) {
         Value vTmp;
@@ -1543,36 +1567,36 @@ void ExprConcatStrings::eval(EvalState & state, Env & env, Value & v)
            since paths are copied when they are used in a derivation),
            and none of the strings are allowed to have contexts. */
         if (first) {
-            firstType = vTmp.type;
+            firstType = vTmp.type();
             first = false;
         }
 
-        if (firstType == tInt) {
-            if (vTmp.type == tInt) {
+        if (firstType == nInt) {
+            if (vTmp.type() == nInt) {
                 n += vTmp.integer;
-            } else if (vTmp.type == tFloat) {
+            } else if (vTmp.type() == nFloat) {
                 // Upgrade the type from int to float;
-                firstType = tFloat;
+                firstType = nFloat;
                 nf = n;
                 nf += vTmp.fpoint;
             } else
                 throwEvalError(pos, "cannot add %1% to an integer", showType(vTmp));
-        } else if (firstType == tFloat) {
-            if (vTmp.type == tInt) {
+        } else if (firstType == nFloat) {
+            if (vTmp.type() == nInt) {
                 nf += vTmp.integer;
-            } else if (vTmp.type == tFloat) {
+            } else if (vTmp.type() == nFloat) {
                 nf += vTmp.fpoint;
             } else
                 throwEvalError(pos, "cannot add %1% to a float", showType(vTmp));
         } else
-            s << state.coerceToString(pos, vTmp, context, false, firstType == tString);
+            s << state.coerceToString(pos, vTmp, context, false, firstType == nString);
     }
 
-    if (firstType == tInt)
+    if (firstType == nInt)
         mkInt(v, n);
-    else if (firstType == tFloat)
+    else if (firstType == nFloat)
         mkFloat(v, nf);
-    else if (firstType == tPath) {
+    else if (firstType == nPath) {
         if (!context.empty())
             throwEvalError(pos, "a string that refers to a store path cannot be appended to a path");
         auto path = canonPath(s.str());
@@ -1599,7 +1623,7 @@ void EvalState::forceValueDeep(Value & v)
 
         forceValue(v);
 
-        if (v.type == tAttrs) {
+        if (v.type() == nAttrs) {
             for (auto & i : *v.attrs)
                 try {
                     recurse(*i.value);
@@ -1622,7 +1646,7 @@ void EvalState::forceValueDeep(Value & v)
 NixInt EvalState::forceInt(Value & v, const Pos & pos)
 {
     forceValue(v, pos);
-    if (v.type != tInt)
+    if (v.type() != nInt)
         throwTypeError(pos, "value is %1% while an integer was expected", v);
     return v.integer;
 }
@@ -1631,9 +1655,9 @@ NixInt EvalState::forceInt(Value & v, const Pos & pos)
 NixFloat EvalState::forceFloat(Value & v, const Pos & pos)
 {
     forceValue(v, pos);
-    if (v.type == tInt)
+    if (v.type() == nInt)
         return v.integer;
-    else if (v.type != tFloat)
+    else if (v.type() != nFloat)
         throwTypeError(pos, "value is %1% while a float was expected", v);
     return v.fpoint;
 }
@@ -1642,7 +1666,7 @@ NixFloat EvalState::forceFloat(Value & v, const Pos & pos)
 bool EvalState::forceBool(Value & v, const Pos & pos)
 {
     forceValue(v, pos);
-    if (v.type != tBool)
+    if (v.type() != nBool)
         throwTypeError(pos, "value is %1% while a Boolean was expected", v);
     return v.boolean;
 }
@@ -1650,14 +1674,14 @@ bool EvalState::forceBool(Value & v, const Pos & pos)
 
 bool EvalState::isFunctor(Value & fun)
 {
-    return fun.type == tAttrs && fun.attrs->find(sFunctor) != fun.attrs->end();
+    return fun.type() == nAttrs && fun.attrs->find(sFunctor) != fun.attrs->end();
 }
 
 
 void EvalState::forceFunction(Value & v, const Pos & pos)
 {
     forceValue(v, pos);
-    if (v.type != tLambda && v.type != tPrimOp && v.type != tPrimOpApp && !isFunctor(v))
+    if (v.type() != nFunction && !isFunctor(v))
         throwTypeError(pos, "value is %1% while a function was expected", v);
 }
 
@@ -1665,7 +1689,7 @@ void EvalState::forceFunction(Value & v, const Pos & pos)
 string EvalState::forceString(Value & v, const Pos & pos)
 {
     forceValue(v, pos);
-    if (v.type != tString) {
+    if (v.type() != nString) {
         if (pos)
             throwTypeError(pos, "value is %1% while a string was expected", v);
         else
@@ -1698,7 +1722,7 @@ void copyContext(const Value & v, PathSet & context)
 std::vector<std::pair<Path, std::string>> Value::getContext()
 {
     std::vector<std::pair<Path, std::string>> res;
-    assert(type == tString);
+    assert(internalType == tString);
     if (string.context)
         for (const char * * p = string.context; *p; ++p)
             res.push_back(decodeContext(*p));
@@ -1731,11 +1755,11 @@ string EvalState::forceStringNoCtx(Value & v, const Pos & pos)
 
 bool EvalState::isDerivation(Value & v)
 {
-    if (v.type != tAttrs) return false;
+    if (v.type() != nAttrs) return false;
     Bindings::iterator i = v.attrs->find(sType);
     if (i == v.attrs->end()) return false;
     forceValue(*i->value);
-    if (i->value->type != tString) return false;
+    if (i->value->type() != nString) return false;
     return strcmp(i->value->string.s, "derivation") == 0;
 }
 
@@ -1760,17 +1784,17 @@ string EvalState::coerceToString(const Pos & pos, Value & v, PathSet & context,
 
     string s;
 
-    if (v.type == tString) {
+    if (v.type() == nString) {
         copyContext(v, context);
         return v.string.s;
     }
 
-    if (v.type == tPath) {
+    if (v.type() == nPath) {
         Path path(canonPath(v.path));
         return copyToStore ? copyPathToStore(context, path) : path;
     }
 
-    if (v.type == tAttrs) {
+    if (v.type() == nAttrs) {
         auto maybeString = tryAttrsToString(pos, v, context, coerceMore, copyToStore);
         if (maybeString) {
             return *maybeString;
@@ -1780,18 +1804,18 @@ string EvalState::coerceToString(const Pos & pos, Value & v, PathSet & context,
         return coerceToString(pos, *i->value, context, coerceMore, copyToStore);
     }
 
-    if (v.type == tExternal)
+    if (v.type() == nExternal)
         return v.external->coerceToString(pos, context, coerceMore, copyToStore);
 
     if (coerceMore) {
 
         /* Note that `false' is represented as an empty string for
            shell scripting convenience, just like `null'. */
-        if (v.type == tBool && v.boolean) return "1";
-        if (v.type == tBool && !v.boolean) return "";
-        if (v.type == tInt) return std::to_string(v.integer);
-        if (v.type == tFloat) return std::to_string(v.fpoint);
-        if (v.type == tNull) return "";
+        if (v.type() == nBool && v.boolean) return "1";
+        if (v.type() == nBool && !v.boolean) return "";
+        if (v.type() == nInt) return std::to_string(v.integer);
+        if (v.type() == nFloat) return std::to_string(v.fpoint);
+        if (v.type() == nNull) return "";
 
         if (v.isList()) {
             string result;
@@ -1854,40 +1878,38 @@ bool EvalState::eqValues(Value & v1, Value & v2)
     if (&v1 == &v2) return true;
 
     // Special case type-compatibility between float and int
-    if (v1.type == tInt && v2.type == tFloat)
+    if (v1.type() == nInt && v2.type() == nFloat)
         return v1.integer == v2.fpoint;
-    if (v1.type == tFloat && v2.type == tInt)
+    if (v1.type() == nFloat && v2.type() == nInt)
         return v1.fpoint == v2.integer;
 
     // All other types are not compatible with each other.
-    if (v1.type != v2.type) return false;
+    if (v1.type() != v2.type()) return false;
 
-    switch (v1.type) {
+    switch (v1.type()) {
 
-        case tInt:
+        case nInt:
             return v1.integer == v2.integer;
 
-        case tBool:
+        case nBool:
             return v1.boolean == v2.boolean;
 
-        case tString:
+        case nString:
             return strcmp(v1.string.s, v2.string.s) == 0;
 
-        case tPath:
+        case nPath:
             return strcmp(v1.path, v2.path) == 0;
 
-        case tNull:
+        case nNull:
             return true;
 
-        case tList1:
-        case tList2:
-        case tListN:
+        case nList:
             if (v1.listSize() != v2.listSize()) return false;
             for (size_t n = 0; n < v1.listSize(); ++n)
                 if (!eqValues(*v1.listElems()[n], *v2.listElems()[n])) return false;
             return true;
 
-        case tAttrs: {
+        case nAttrs: {
             /* If both sets denote a derivation (type = "derivation"),
                then compare their outPaths. */
             if (isDerivation(v1) && isDerivation(v2)) {
@@ -1909,15 +1931,13 @@ bool EvalState::eqValues(Value & v1, Value & v2)
         }
 
         /* Functions are incomparable. */
-        case tLambda:
-        case tPrimOp:
-        case tPrimOpApp:
+        case nFunction:
             return false;
 
-        case tExternal:
+        case nExternal:
             return *v1.external == *v2.external;
 
-        case tFloat:
+        case nFloat:
             return v1.fpoint == v2.fpoint;
 
         default:
@@ -2046,7 +2066,7 @@ void EvalState::printStats()
 string ExternalValueBase::coerceToString(const Pos & pos, PathSet & context, bool copyMore, bool copyToStore) const
 {
     throw TypeError({
-        .hint = hintfmt("cannot coerce %1% to a string", showType()),
+        .msg = hintfmt("cannot coerce %1% to a string", showType()),
         .errPos = pos
     });
 }
@@ -2072,10 +2092,22 @@ EvalSettings::EvalSettings()
 Strings EvalSettings::getDefaultNixPath()
 {
     Strings res;
-    auto add = [&](const Path & p) { if (pathExists(p)) { res.push_back(p); } };
-    add(getHome() + "/.nix-defexpr/channels");
-    add("nixpkgs=" + settings.nixStateDir + "/nix/profiles/per-user/root/channels/nixpkgs");
-    add(settings.nixStateDir + "/nix/profiles/per-user/root/channels");
+    auto add = [&](const Path & p, const std::string & s = std::string()) {
+        if (pathExists(p)) {
+            if (s.empty()) {
+                res.push_back(p);
+            } else {
+                res.push_back(s + "=" + p);
+            }
+        }
+    };
+
+    if (!evalSettings.restrictEval && !evalSettings.pureEval) {
+        add(getHome() + "/.nix-defexpr/channels");
+        add(settings.nixStateDir + "/profiles/per-user/root/channels/nixpkgs", "nixpkgs");
+        add(settings.nixStateDir + "/profiles/per-user/root/channels");
+    }
+
     return res;
 }
 

src/libexpr/eval.hh

@@ -432,4 +432,6 @@ struct EvalSettings : Config
 
 extern EvalSettings evalSettings;
 
+static const std::string corepkgsPrefix{"/__corepkgs__/"};
+
 }

src/libexpr/fetchurl.nix

@@ -1,6 +1,6 @@
 { system ? "" # obsolete
 , url
-, hash ? "" # an SRI ash
+, hash ? "" # an SRI hash
 
 # Legacy hash specification
 , md5 ? "", sha1 ? "", sha256 ? "", sha512 ? ""

src/libexpr/flake/config.cc

@@ -0,0 +1,83 @@
+#include "flake.hh"
+
+#include <nlohmann/json.hpp>
+
+namespace nix::flake {
+
+// setting name -> setting value -> allow or ignore.
+typedef std::map<std::string, std::map<std::string, bool>> TrustedList;
+
+Path trustedListPath()
+{
+    return getDataDir() + "/nix/trusted-settings.json";
+}
+
+static TrustedList readTrustedList()
+{
+    auto path = trustedListPath();
+    if (!pathExists(path)) return {};
+    auto json = nlohmann::json::parse(readFile(path));
+    return json;
+}
+
+static void writeTrustedList(const TrustedList & trustedList)
+{
+    auto path = trustedListPath();
+    createDirs(dirOf(path));
+    writeFile(path, nlohmann::json(trustedList).dump());
+}
+
+void ConfigFile::apply()
+{
+    std::set<std::string> whitelist{"bash-prompt", "bash-prompt-suffix"};
+
+    for (auto & [name, value] : settings) {
+
+        auto baseName = hasPrefix(name, "extra-") ? std::string(name, 6) : name;
+
+        // FIXME: Move into libutil/config.cc.
+        std::string valueS;
+        if (auto s = std::get_if<std::string>(&value))
+            valueS = *s;
+        else if (auto n = std::get_if<int64_t>(&value))
+            valueS = fmt("%d", n);
+        else if (auto b = std::get_if<Explicit<bool>>(&value))
+            valueS = b->t ? "true" : "false";
+        else if (auto ss = std::get_if<std::vector<std::string>>(&value))
+            valueS = concatStringsSep(" ", *ss); // FIXME: evil
+        else
+            assert(false);
+
+        if (!whitelist.count(baseName)) {
+            auto trustedList = readTrustedList();
+
+            bool trusted = false;
+
+            if (auto saved = get(get(trustedList, name).value_or(std::map<std::string, bool>()), valueS)) {
+                trusted = *saved;
+            } else {
+                // FIXME: filter ANSI escapes, newlines, \r, etc.
+                if (std::tolower(logger->ask(fmt("do you want to allow configuration setting '%s' to be set to '" ANSI_RED "%s" ANSI_NORMAL "' (y/N)?", name, valueS)).value_or('n')) != 'y') {
+                    if (std::tolower(logger->ask("do you want to permanently mark this value as untrusted (y/N)?").value_or('n')) == 'y') {
+                        trustedList[name][valueS] = false;
+                        writeTrustedList(trustedList);
+                    }
+                } else {
+                    if (std::tolower(logger->ask("do you want to permanently mark this value as trusted (y/N)?").value_or('n')) == 'y') {
+                        trustedList[name][valueS] = trusted = true;
+                        writeTrustedList(trustedList);
+                    }
+                }
+            }
+
+            if (!trusted) {
+                warn("ignoring untrusted flake configuration setting '%s'", name);
+                continue;
+            }
+        }
+
+        globalConfig.set(name, valueS);
+    }
+}
+
+}

src/libexpr/flake/flake.cc

@@ -71,14 +71,20 @@ static std::tuple<fetchers::Tree, FlakeRef, FlakeRef> fetchOrSubstituteTree(
     return {std::move(tree), resolvedRef, lockedRef};
 }
 
+static void forceTrivialValue(EvalState & state, Value & value, const Pos & pos)
+{
+    if (value.isThunk() && value.isTrivial())
+        state.forceValue(value, pos);
+}
+
+
 static void expectType(EvalState & state, ValueType type,
     Value & value, const Pos & pos)
 {
-    if (value.type == tThunk && value.isTrivial())
-        state.forceValue(value, pos);
-    if (value.type != type)
+    forceTrivialValue(state, value, pos);
+    if (value.type() != type)
         throw Error("expected %s but got %s at %s",
-            showType(type), showType(value.type), pos);
+            showType(type), showType(value.type()), pos);
 }
 
 static std::map<FlakeId, FlakeInput> parseFlakeInputs(
@@ -87,7 +93,7 @@ static std::map<FlakeId, FlakeInput> parseFlakeInputs(
 static FlakeInput parseFlakeInput(EvalState & state,
     const std::string & inputName, Value * value, const Pos & pos)
 {
-    expectType(state, tAttrs, *value, pos);
+    expectType(state, nAttrs, *value, pos);
 
     FlakeInput input;
 
@@ -102,24 +108,32 @@ static FlakeInput parseFlakeInput(EvalState & state,
     for (nix::Attr attr : *(value->attrs)) {
         try {
             if (attr.name == sUrl) {
-                expectType(state, tString, *attr.value, *attr.pos);
+                expectType(state, nString, *attr.value, *attr.pos);
                 url = attr.value->string.s;
                 attrs.emplace("url", *url);
             } else if (attr.name == sFlake) {
-                expectType(state, tBool, *attr.value, *attr.pos);
+                expectType(state, nBool, *attr.value, *attr.pos);
                 input.isFlake = attr.value->boolean;
             } else if (attr.name == sInputs) {
                 input.overrides = parseFlakeInputs(state, attr.value, *attr.pos);
             } else if (attr.name == sFollows) {
-                expectType(state, tString, *attr.value, *attr.pos);
+                expectType(state, nString, *attr.value, *attr.pos);
                 input.follows = parseInputPath(attr.value->string.s);
             } else {
-                state.forceValue(*attr.value);
-                if (attr.value->type == tString)
-                    attrs.emplace(attr.name, attr.value->string.s);
-                else
-                    throw TypeError("flake input attribute '%s' is %s while a string is expected",
-                        attr.name, showType(*attr.value));
+                switch (attr.value->type()) {
+                    case nString:
+                        attrs.emplace(attr.name, attr.value->string.s);
+                        break;
+                    case nBool:
+                        attrs.emplace(attr.name, Explicit<bool> { attr.value->boolean });
+                        break;
+                    case nInt:
+                        attrs.emplace(attr.name, (long unsigned int)attr.value->integer);
+                        break;
+                    default:
+                        throw TypeError("flake input attribute '%s' is %s while a string, Boolean, or integer is expected",
+                            attr.name, showType(*attr.value));
+                }
             }
         } catch (Error & e) {
             e.addTrace(*attr.pos, hintfmt("in flake attribute '%s'", attr.name));
@@ -153,7 +167,7 @@ static std::map<FlakeId, FlakeInput> parseFlakeInputs(
 {
     std::map<FlakeId, FlakeInput> inputs;
 
-    expectType(state, tAttrs, *value, pos);
+    expectType(state, nAttrs, *value, pos);
 
     for (nix::Attr & inputAttr : *(*value).attrs) {
         inputs.emplace(inputAttr.name,
@@ -194,15 +208,10 @@ static Flake getFlake(
     Value vInfo;
     state.evalFile(flakeFile, vInfo, true); // FIXME: symlink attack
 
-    expectType(state, tAttrs, vInfo, Pos(foFile, state.symbols.create(flakeFile), 0, 0));
-
-    auto sEdition = state.symbols.create("edition"); // FIXME: remove soon
-
-    if (vInfo.attrs->get(sEdition))
-        warn("flake '%s' has deprecated attribute 'edition'", lockedRef);
+    expectType(state, nAttrs, vInfo, Pos(foFile, state.symbols.create(flakeFile), 0, 0));
 
     if (auto description = vInfo.attrs->get(state.sDescription)) {
-        expectType(state, tString, *description->value, *description->pos);
+        expectType(state, nString, *description->value, *description->pos);
         flake.description = description->value->string.s;
     }
 
@@ -214,9 +223,9 @@ static Flake getFlake(
     auto sOutputs = state.symbols.create("outputs");
 
     if (auto outputs = vInfo.attrs->get(sOutputs)) {
-        expectType(state, tLambda, *outputs->value, *outputs->pos);
+        expectType(state, nFunction, *outputs->value, *outputs->pos);
 
-        if (outputs->value->lambda.fun->matchAttrs) {
+        if (outputs->value->isLambda() && outputs->value->lambda.fun->matchAttrs) {
             for (auto & formal : outputs->value->lambda.fun->formals->formals) {
                 if (formal.name != state.sSelf)
                     flake.inputs.emplace(formal.name, FlakeInput {
@@ -228,11 +237,41 @@ static Flake getFlake(
     } else
         throw Error("flake '%s' lacks attribute 'outputs'", lockedRef);
 
+    auto sNixConfig = state.symbols.create("nixConfig");
+
+    if (auto nixConfig = vInfo.attrs->get(sNixConfig)) {
+        expectType(state, nAttrs, *nixConfig->value, *nixConfig->pos);
+
+        for (auto & setting : *nixConfig->value->attrs) {
+            forceTrivialValue(state, *setting.value, *setting.pos);
+            if (setting.value->type() == nString)
+                flake.config.settings.insert({setting.name, state.forceStringNoCtx(*setting.value, *setting.pos)});
+            else if (setting.value->type() == nInt)
+                flake.config.settings.insert({setting.name, state.forceInt(*setting.value, *setting.pos)});
+            else if (setting.value->type() == nBool)
+                flake.config.settings.insert({setting.name, state.forceBool(*setting.value, *setting.pos)});
+            else if (setting.value->type() == nList) {
+                std::vector<std::string> ss;
+                for (unsigned int n = 0; n < setting.value->listSize(); ++n) {
+                    auto elem = setting.value->listElems()[n];
+                    if (elem->type() != nString)
+                        throw TypeError("list element in flake configuration setting '%s' is %s while a string is expected",
+                            setting.name, showType(*setting.value));
+                    ss.push_back(state.forceStringNoCtx(*elem, *setting.pos));
+                }
+                flake.config.settings.insert({setting.name, ss});
+            }
+            else
+                throw TypeError("flake configuration setting '%s' is %s",
+                    setting.name, showType(*setting.value));
+        }
+    }
+
     for (auto & attr : *vInfo.attrs) {
-        if (attr.name != sEdition &&
-            attr.name != state.sDescription &&
+        if (attr.name != state.sDescription &&
             attr.name != sInputs &&
-            attr.name != sOutputs)
+            attr.name != sOutputs &&
+            attr.name != sNixConfig)
             throw Error("flake '%s' has an unsupported attribute '%s', at %s",
                 lockedRef, attr.name, *attr.pos);
     }
@@ -259,284 +298,298 @@ LockedFlake lockFlake(
 
     auto flake = getFlake(state, topRef, lockFlags.useRegistries, flakeCache);
 
-    // FIXME: symlink attack
-    auto oldLockFile = LockFile::read(
-        flake.sourceInfo->actualPath + "/" + flake.lockedRef.subdir + "/flake.lock");
+    try {
 
-    debug("old lock file: %s", oldLockFile);
+        // FIXME: symlink attack
+        auto oldLockFile = LockFile::read(
+            flake.sourceInfo->actualPath + "/" + flake.lockedRef.subdir + "/flake.lock");
 
-    // FIXME: check whether all overrides are used.
-    std::map<InputPath, FlakeInput> overrides;
-    std::set<InputPath> overridesUsed, updatesUsed;
+        debug("old lock file: %s", oldLockFile);
 
-    for (auto & i : lockFlags.inputOverrides)
-        overrides.insert_or_assign(i.first, FlakeInput { .ref = i.second });
+        // FIXME: check whether all overrides are used.
+        std::map<InputPath, FlakeInput> overrides;
+        std::set<InputPath> overridesUsed, updatesUsed;
 
-    LockFile newLockFile;
+        for (auto & i : lockFlags.inputOverrides)
+            overrides.insert_or_assign(i.first, FlakeInput { .ref = i.second });
 
-    std::vector<FlakeRef> parents;
+        LockFile newLockFile;
 
-    std::function<void(
-        const FlakeInputs & flakeInputs,
-        std::shared_ptr<Node> node,
-        const InputPath & inputPathPrefix,
-        std::shared_ptr<const Node> oldNode)>
-        computeLocks;
+        std::vector<FlakeRef> parents;
 
-    computeLocks = [&](
-        const FlakeInputs & flakeInputs,
-        std::shared_ptr<Node> node,
-        const InputPath & inputPathPrefix,
-        std::shared_ptr<const Node> oldNode)
-    {
-        debug("computing lock file node '%s'", printInputPath(inputPathPrefix));
+        std::function<void(
+            const FlakeInputs & flakeInputs,
+            std::shared_ptr<Node> node,
+            const InputPath & inputPathPrefix,
+            std::shared_ptr<const Node> oldNode)>
+            computeLocks;
 
-        /* Get the overrides (i.e. attributes of the form
-           'inputs.nixops.inputs.nixpkgs.url = ...'). */
-        // FIXME: check this
-        for (auto & [id, input] : flake.inputs) {
-            for (auto & [idOverride, inputOverride] : input.overrides) {
+        computeLocks = [&](
+            const FlakeInputs & flakeInputs,
+            std::shared_ptr<Node> node,
+            const InputPath & inputPathPrefix,
+            std::shared_ptr<const Node> oldNode)
+        {
+            debug("computing lock file node '%s'", printInputPath(inputPathPrefix));
+
+            /* Get the overrides (i.e. attributes of the form
+               'inputs.nixops.inputs.nixpkgs.url = ...'). */
+            // FIXME: check this
+            for (auto & [id, input] : flake.inputs) {
+                for (auto & [idOverride, inputOverride] : input.overrides) {
+                    auto inputPath(inputPathPrefix);
+                    inputPath.push_back(id);
+                    inputPath.push_back(idOverride);
+                    overrides.insert_or_assign(inputPath, inputOverride);
+                }
+            }
+
+            /* Go over the flake inputs, resolve/fetch them if
+               necessary (i.e. if they're new or the flakeref changed
+               from what's in the lock file). */
+            for (auto & [id, input2] : flakeInputs) {
                 auto inputPath(inputPathPrefix);
                 inputPath.push_back(id);
-                inputPath.push_back(idOverride);
-                overrides.insert_or_assign(inputPath, inputOverride);
-            }
-        }
+                auto inputPathS = printInputPath(inputPath);
+                debug("computing input '%s'", inputPathS);
 
-        /* Go over the flake inputs, resolve/fetch them if
-           necessary (i.e. if they're new or the flakeref changed
-           from what's in the lock file). */
-        for (auto & [id, input2] : flakeInputs) {
-            auto inputPath(inputPathPrefix);
-            inputPath.push_back(id);
-            auto inputPathS = printInputPath(inputPath);
-            debug("computing input '%s'", inputPathS);
+                try {
 
-            /* Do we have an override for this input from one of the
-               ancestors? */
-            auto i = overrides.find(inputPath);
-            bool hasOverride = i != overrides.end();
-            if (hasOverride) overridesUsed.insert(inputPath);
-            auto & input = hasOverride ? i->second : input2;
+                    /* Do we have an override for this input from one of the
+                       ancestors? */
+                    auto i = overrides.find(inputPath);
+                    bool hasOverride = i != overrides.end();
+                    if (hasOverride) overridesUsed.insert(inputPath);
+                    auto & input = hasOverride ? i->second : input2;
 
-            /* Resolve 'follows' later (since it may refer to an input
-               path we haven't processed yet. */
-            if (input.follows) {
-                InputPath target;
-                if (hasOverride || input.absolute)
-                    /* 'follows' from an override is relative to the
-                       root of the graph. */
-                    target = *input.follows;
-                else {
-                    /* Otherwise, it's relative to the current flake. */
-                    target = inputPathPrefix;
-                    for (auto & i : *input.follows) target.push_back(i);
-                }
-                debug("input '%s' follows '%s'", inputPathS, printInputPath(target));
-                node->inputs.insert_or_assign(id, target);
-                continue;
-            }
+                    /* Resolve 'follows' later (since it may refer to an input
+                       path we haven't processed yet. */
+                    if (input.follows) {
+                        InputPath target;
+                        if (hasOverride || input.absolute)
+                            /* 'follows' from an override is relative to the
+                               root of the graph. */
+                            target = *input.follows;
+                        else {
+                            /* Otherwise, it's relative to the current flake. */
+                            target = inputPathPrefix;
+                            for (auto & i : *input.follows) target.push_back(i);
+                        }
+                        debug("input '%s' follows '%s'", inputPathS, printInputPath(target));
+                        node->inputs.insert_or_assign(id, target);
+                        continue;
+                    }
 
-            assert(input.ref);
+                    assert(input.ref);
 
-            /* Do we have an entry in the existing lock file? And we
-               don't have a --update-input flag for this input? */
-            std::shared_ptr<LockedNode> oldLock;
+                    /* Do we have an entry in the existing lock file? And we
+                       don't have a --update-input flag for this input? */
+                    std::shared_ptr<LockedNode> oldLock;
 
-            updatesUsed.insert(inputPath);
+                    updatesUsed.insert(inputPath);
 
-            if (oldNode && !lockFlags.inputUpdates.count(inputPath))
-                if (auto oldLock2 = get(oldNode->inputs, id))
-                    if (auto oldLock3 = std::get_if<0>(&*oldLock2))
-                        oldLock = *oldLock3;
+                    if (oldNode && !lockFlags.inputUpdates.count(inputPath))
+                        if (auto oldLock2 = get(oldNode->inputs, id))
+                            if (auto oldLock3 = std::get_if<0>(&*oldLock2))
+                                oldLock = *oldLock3;
 
-            if (oldLock
-                && oldLock->originalRef == *input.ref
-                && !hasOverride)
-            {
-                debug("keeping existing input '%s'", inputPathS);
+                    if (oldLock
+                        && oldLock->originalRef == *input.ref
+                        && !hasOverride)
+                    {
+                        debug("keeping existing input '%s'", inputPathS);
 
-                /* Copy the input from the old lock since its flakeref
-                   didn't change and there is no override from a
-                   higher level flake. */
-                auto childNode = std::make_shared<LockedNode>(
-                    oldLock->lockedRef, oldLock->originalRef, oldLock->isFlake);
+                        /* Copy the input from the old lock since its flakeref
+                           didn't change and there is no override from a
+                           higher level flake. */
+                        auto childNode = std::make_shared<LockedNode>(
+                            oldLock->lockedRef, oldLock->originalRef, oldLock->isFlake);
 
-                node->inputs.insert_or_assign(id, childNode);
+                        node->inputs.insert_or_assign(id, childNode);
 
-                /* If we have an --update-input flag for an input
-                   of this input, then we must fetch the flake to
-                   update it. */
-                auto lb = lockFlags.inputUpdates.lower_bound(inputPath);
+                        /* If we have an --update-input flag for an input
+                           of this input, then we must fetch the flake to
+                           update it. */
+                        auto lb = lockFlags.inputUpdates.lower_bound(inputPath);
 
-                auto hasChildUpdate =
-                    lb != lockFlags.inputUpdates.end()
-                    && lb->size() > inputPath.size()
-                    && std::equal(inputPath.begin(), inputPath.end(), lb->begin());
+                        auto hasChildUpdate =
+                            lb != lockFlags.inputUpdates.end()
+                            && lb->size() > inputPath.size()
+                            && std::equal(inputPath.begin(), inputPath.end(), lb->begin());
 
-                if (hasChildUpdate) {
-                    auto inputFlake = getFlake(
-                        state, oldLock->lockedRef, false, flakeCache);
-                    computeLocks(inputFlake.inputs, childNode, inputPath, oldLock);
-                } else {
-                    /* No need to fetch this flake, we can be
-                       lazy. However there may be new overrides on the
-                       inputs of this flake, so we need to check
-                       those. */
-                    FlakeInputs fakeInputs;
+                        if (hasChildUpdate) {
+                            auto inputFlake = getFlake(
+                                state, oldLock->lockedRef, false, flakeCache);
+                            computeLocks(inputFlake.inputs, childNode, inputPath, oldLock);
+                        } else {
+                            /* No need to fetch this flake, we can be
+                               lazy. However there may be new overrides on the
+                               inputs of this flake, so we need to check
+                               those. */
+                            FlakeInputs fakeInputs;
 
-                    for (auto & i : oldLock->inputs) {
-                        if (auto lockedNode = std::get_if<0>(&i.second)) {
-                            fakeInputs.emplace(i.first, FlakeInput {
-                                .ref = (*lockedNode)->originalRef,
-                                .isFlake = (*lockedNode)->isFlake,
-                            });
-                        } else if (auto follows = std::get_if<1>(&i.second)) {
-                            fakeInputs.emplace(i.first, FlakeInput {
-                                .follows = *follows,
-                                .absolute = true
-                            });
+                            for (auto & i : oldLock->inputs) {
+                                if (auto lockedNode = std::get_if<0>(&i.second)) {
+                                    fakeInputs.emplace(i.first, FlakeInput {
+                                        .ref = (*lockedNode)->originalRef,
+                                        .isFlake = (*lockedNode)->isFlake,
+                                    });
+                                } else if (auto follows = std::get_if<1>(&i.second)) {
+                                    fakeInputs.emplace(i.first, FlakeInput {
+                                        .follows = *follows,
+                                        .absolute = true
+                                    });
+                                }
+                            }
+
+                            computeLocks(fakeInputs, childNode, inputPath, oldLock);
+                        }
+
+                    } else {
+                        /* We need to create a new lock file entry. So fetch
+                           this input. */
+                        debug("creating new input '%s'", inputPathS);
+
+                        if (!lockFlags.allowMutable && !input.ref->input.isImmutable())
+                            throw Error("cannot update flake input '%s' in pure mode", inputPathS);
+
+                        if (input.isFlake) {
+                            auto inputFlake = getFlake(state, *input.ref, lockFlags.useRegistries, flakeCache);
+
+                            /* Note: in case of an --override-input, we use
+                               the *original* ref (input2.ref) for the
+                               "original" field, rather than the
+                               override. This ensures that the override isn't
+                               nuked the next time we update the lock
+                               file. That is, overrides are sticky unless you
+                               use --no-write-lock-file. */
+                            auto childNode = std::make_shared<LockedNode>(
+                                inputFlake.lockedRef, input2.ref ? *input2.ref : *input.ref);
+
+                            node->inputs.insert_or_assign(id, childNode);
+
+                            /* Guard against circular flake imports. */
+                            for (auto & parent : parents)
+                                if (parent == *input.ref)
+                                    throw Error("found circular import of flake '%s'", parent);
+                            parents.push_back(*input.ref);
+                            Finally cleanup([&]() { parents.pop_back(); });
+
+                            /* Recursively process the inputs of this
+                               flake. Also, unless we already have this flake
+                               in the top-level lock file, use this flake's
+                               own lock file. */
+                            computeLocks(
+                                inputFlake.inputs, childNode, inputPath,
+                                oldLock
+                                ? std::dynamic_pointer_cast<const Node>(oldLock)
+                                : LockFile::read(
+                                    inputFlake.sourceInfo->actualPath + "/" + inputFlake.lockedRef.subdir + "/flake.lock").root);
+                        }
+
+                        else {
+                            auto [sourceInfo, resolvedRef, lockedRef] = fetchOrSubstituteTree(
+                                state, *input.ref, lockFlags.useRegistries, flakeCache);
+                            node->inputs.insert_or_assign(id,
+                                std::make_shared<LockedNode>(lockedRef, *input.ref, false));
                         }
                     }
 
-                    computeLocks(fakeInputs, childNode, inputPath, oldLock);
-                }
-
-            } else {
-                /* We need to create a new lock file entry. So fetch
-                   this input. */
-                debug("creating new input '%s'", inputPathS);
-
-                if (!lockFlags.allowMutable && !input.ref->input.isImmutable())
-                    throw Error("cannot update flake input '%s' in pure mode", inputPathS);
-
-                if (input.isFlake) {
-                    auto inputFlake = getFlake(state, *input.ref, lockFlags.useRegistries, flakeCache);
-
-                    /* Note: in case of an --override-input, we use
-                       the *original* ref (input2.ref) for the
-                       "original" field, rather than the
-                       override. This ensures that the override isn't
-                       nuked the next time we update the lock
-                       file. That is, overrides are sticky unless you
-                       use --no-write-lock-file. */
-                    auto childNode = std::make_shared<LockedNode>(
-                        inputFlake.lockedRef, input2.ref ? *input2.ref : *input.ref);
-
-                    node->inputs.insert_or_assign(id, childNode);
-
-                    /* Guard against circular flake imports. */
-                    for (auto & parent : parents)
-                        if (parent == *input.ref)
-                            throw Error("found circular import of flake '%s'", parent);
-                    parents.push_back(*input.ref);
-                    Finally cleanup([&]() { parents.pop_back(); });
-
-                    /* Recursively process the inputs of this
-                       flake. Also, unless we already have this flake
-                       in the top-level lock file, use this flake's
-                       own lock file. */
-                    computeLocks(
-                        inputFlake.inputs, childNode, inputPath,
-                        oldLock
-                        ? std::dynamic_pointer_cast<const Node>(oldLock)
-                        : LockFile::read(
-                            inputFlake.sourceInfo->actualPath + "/" + inputFlake.lockedRef.subdir + "/flake.lock").root);
-                }
-
-                else {
-                    auto [sourceInfo, resolvedRef, lockedRef] = fetchOrSubstituteTree(
-                        state, *input.ref, lockFlags.useRegistries, flakeCache);
-                    node->inputs.insert_or_assign(id,
-                        std::make_shared<LockedNode>(lockedRef, *input.ref, false));
+                } catch (Error & e) {
+                    e.addTrace({}, "while updating the flake input '%s'", inputPathS);
+                    throw;
                 }
             }
-        }
-    };
+        };
 
-    computeLocks(
-        flake.inputs, newLockFile.root, {},
-        lockFlags.recreateLockFile ? nullptr : oldLockFile.root);
+        computeLocks(
+            flake.inputs, newLockFile.root, {},
+            lockFlags.recreateLockFile ? nullptr : oldLockFile.root);
 
-    for (auto & i : lockFlags.inputOverrides)
-        if (!overridesUsed.count(i.first))
-            warn("the flag '--override-input %s %s' does not match any input",
-                printInputPath(i.first), i.second);
+        for (auto & i : lockFlags.inputOverrides)
+            if (!overridesUsed.count(i.first))
+                warn("the flag '--override-input %s %s' does not match any input",
+                    printInputPath(i.first), i.second);
 
-    for (auto & i : lockFlags.inputUpdates)
-        if (!updatesUsed.count(i))
-            warn("the flag '--update-input %s' does not match any input", printInputPath(i));
+        for (auto & i : lockFlags.inputUpdates)
+            if (!updatesUsed.count(i))
+                warn("the flag '--update-input %s' does not match any input", printInputPath(i));
 
-    /* Check 'follows' inputs. */
-    newLockFile.check();
+        /* Check 'follows' inputs. */
+        newLockFile.check();
 
-    debug("new lock file: %s", newLockFile);
+        debug("new lock file: %s", newLockFile);
 
-    /* Check whether we need to / can write the new lock file. */
-    if (!(newLockFile == oldLockFile)) {
+        /* Check whether we need to / can write the new lock file. */
+        if (!(newLockFile == oldLockFile)) {
 
-        auto diff = LockFile::diff(oldLockFile, newLockFile);
+            auto diff = LockFile::diff(oldLockFile, newLockFile);
 
-        if (lockFlags.writeLockFile) {
-            if (auto sourcePath = topRef.input.getSourcePath()) {
-                if (!newLockFile.isImmutable()) {
-                    if (settings.warnDirty)
-                        warn("will not write lock file of flake '%s' because it has a mutable input", topRef);
-                } else {
-                    if (!lockFlags.updateLockFile)
-                        throw Error("flake '%s' requires lock file changes but they're not allowed due to '--no-update-lock-file'", topRef);
+            if (lockFlags.writeLockFile) {
+                if (auto sourcePath = topRef.input.getSourcePath()) {
+                    if (!newLockFile.isImmutable()) {
+                        if (settings.warnDirty)
+                            warn("will not write lock file of flake '%s' because it has a mutable input", topRef);
+                    } else {
+                        if (!lockFlags.updateLockFile)
+                            throw Error("flake '%s' requires lock file changes but they're not allowed due to '--no-update-lock-file'", topRef);
 
-                    auto relPath = (topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock";
+                        auto relPath = (topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock";
 
-                    auto path = *sourcePath + "/" + relPath;
+                        auto path = *sourcePath + "/" + relPath;
 
-                    bool lockFileExists = pathExists(path);
+                        bool lockFileExists = pathExists(path);
 
-                    if (lockFileExists) {
-                        auto s = chomp(diff);
-                        if (s.empty())
-                            warn("updating lock file '%s'", path);
-                        else
-                            warn("updating lock file '%s':\n%s", path, s);
-                    } else
-                        warn("creating lock file '%s'", path);
+                        if (lockFileExists) {
+                            auto s = chomp(diff);
+                            if (s.empty())
+                                warn("updating lock file '%s'", path);
+                            else
+                                warn("updating lock file '%s':\n%s", path, s);
+                        } else
+                            warn("creating lock file '%s'", path);
 
-                    newLockFile.write(path);
+                        newLockFile.write(path);
 
-                    topRef.input.markChangedFile(
-                        (topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock",
-                        lockFlags.commitLockFile
-                        ? std::optional<std::string>(fmt("%s: %s\n\nFlake input changes:\n\n%s",
-                                relPath, lockFileExists ? "Update" : "Add", diff))
-                        : std::nullopt);
+                        topRef.input.markChangedFile(
+                            (topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock",
+                            lockFlags.commitLockFile
+                            ? std::optional<std::string>(fmt("%s: %s\n\nFlake input changes:\n\n%s",
+                                    relPath, lockFileExists ? "Update" : "Add", diff))
+                            : std::nullopt);
 
-                    /* Rewriting the lockfile changed the top-level
-                       repo, so we should re-read it. FIXME: we could
-                       also just clear the 'rev' field... */
-                    auto prevLockedRef = flake.lockedRef;
-                    FlakeCache dummyCache;
-                    flake = getFlake(state, topRef, lockFlags.useRegistries, dummyCache);
+                        /* Rewriting the lockfile changed the top-level
+                           repo, so we should re-read it. FIXME: we could
+                           also just clear the 'rev' field... */
+                        auto prevLockedRef = flake.lockedRef;
+                        FlakeCache dummyCache;
+                        flake = getFlake(state, topRef, lockFlags.useRegistries, dummyCache);
 
-                    if (lockFlags.commitLockFile &&
-                        flake.lockedRef.input.getRev() &&
-                        prevLockedRef.input.getRev() != flake.lockedRef.input.getRev())
-                        warn("committed new revision '%s'", flake.lockedRef.input.getRev()->gitRev());
+                        if (lockFlags.commitLockFile &&
+                            flake.lockedRef.input.getRev() &&
+                            prevLockedRef.input.getRev() != flake.lockedRef.input.getRev())
+                            warn("committed new revision '%s'", flake.lockedRef.input.getRev()->gitRev());
 
-                    /* Make sure that we picked up the change,
-                       i.e. the tree should usually be dirty
-                       now. Corner case: we could have reverted from a
-                       dirty to a clean tree! */
-                    if (flake.lockedRef.input == prevLockedRef.input
-                        && !flake.lockedRef.input.isImmutable())
-                        throw Error("'%s' did not change after I updated its 'flake.lock' file; is 'flake.lock' under version control?", flake.originalRef);
-                }
+                        /* Make sure that we picked up the change,
+                           i.e. the tree should usually be dirty
+                           now. Corner case: we could have reverted from a
+                           dirty to a clean tree! */
+                        if (flake.lockedRef.input == prevLockedRef.input
+                            && !flake.lockedRef.input.isImmutable())
+                            throw Error("'%s' did not change after I updated its 'flake.lock' file; is 'flake.lock' under version control?", flake.originalRef);
+                    }
+                } else
+                    throw Error("cannot write modified lock file of flake '%s' (use '--no-write-lock-file' to ignore)", topRef);
             } else
-                throw Error("cannot write modified lock file of flake '%s' (use '--no-write-lock-file' to ignore)", topRef);
-        } else
-            warn("not writing modified lock file of flake '%s':\n%s", topRef, chomp(diff));
-    }
+                warn("not writing modified lock file of flake '%s':\n%s", topRef, chomp(diff));
+        }
 
-    return LockedFlake { .flake = std::move(flake), .lockFile = std::move(newLockFile) };
+        return LockedFlake { .flake = std::move(flake), .lockFile = std::move(newLockFile) };
+
+    } catch (Error & e) {
+        e.addTrace({}, "while updating the lock file of flake '%s'", flake.lockedRef.to_string());
+        throw;
+    }
 }
 
 void callFlake(EvalState & state,

src/libexpr/flake/flake.hh

@@ -17,23 +17,55 @@ struct FlakeInput;
 
 typedef std::map<FlakeId, FlakeInput> FlakeInputs;
 
+/* FlakeInput is the 'Flake'-level parsed form of the "input" entries
+ * in the flake file.
+ *
+ * A FlakeInput is normally constructed by the 'parseFlakeInput'
+ * function which parses the input specification in the '.flake' file
+ * to create a 'FlakeRef' (a fetcher, the fetcher-specific
+ * representation of the input specification, and possibly the fetched
+ * local store path result) and then creating this FlakeInput to hold
+ * that FlakeRef, along with anything that might override that
+ * FlakeRef (like command-line overrides or "follows" specifications).
+ *
+ * A FlakeInput is also sometimes constructed directly from a FlakeRef
+ * instead of starting at the flake-file input specification
+ * (e.g. overrides, follows, and implicit inputs).
+ *
+ * A FlakeInput will usually have one of either "ref" or "follows"
+ * set.  If not otherwise specified, a "ref" will be generated to a
+ * 'type="indirect"' flake, which is treated as simply the name of a
+ * flake to be resolved in the registry.
+ */
+
 struct FlakeInput
 {
     std::optional<FlakeRef> ref;
-    bool isFlake = true;
+    bool isFlake = true;  // true = process flake to get outputs, false = (fetched) static source path
     std::optional<InputPath> follows;
     bool absolute = false; // whether 'follows' is relative to the flake root
     FlakeInputs overrides;
 };
 
+struct ConfigFile
+{
+    using ConfigValue = std::variant<std::string, int64_t, Explicit<bool>, std::vector<std::string>>;
+
+    std::map<std::string, ConfigValue> settings;
+
+    void apply();
+};
+
+/* The contents of a flake.nix file. */
 struct Flake
 {
-    FlakeRef originalRef;
-    FlakeRef resolvedRef;
-    FlakeRef lockedRef;
+    FlakeRef originalRef;   // the original flake specification (by the user)
+    FlakeRef resolvedRef;   // registry references and caching resolved to the specific underlying flake
+    FlakeRef lockedRef;     // the specific local store result of invoking the fetcher
     std::optional<std::string> description;
     std::shared_ptr<const fetchers::Tree> sourceInfo;
     FlakeInputs inputs;
+    ConfigFile config; // 'nixConfig' attribute
     ~Flake();
 };
 
@@ -81,7 +113,7 @@ struct LockFlags
     /* Whether to commit changes to flake.lock. */
     bool commitLockFile = false;
 
-    /* Flake inputs to be overriden. */
+    /* Flake inputs to be overridden. */
     std::map<InputPath, FlakeRef> inputOverrides;
 
     /* Flake inputs to be updated. This means that any existing lock