Compare commits

...

69 Commits

Author SHA1 Message Date
regnat
ea792bcdc8 Make nix search lazier
Only 4 evals left \o/
2021-06-10 12:26:32 +02:00
regnat
71650c83c6 Make the error cleaner when getFields is misused 2021-06-10 12:26:32 +02:00
regnat
d51deeac1c Remove the obsolete CmdSearch::visit function 2021-06-10 12:26:32 +02:00
regnat
1f8541258c Make getFields just return the name of the fields
Makes it much easier to deal with non-evaluating stuff
2021-06-10 12:26:32 +02:00
regnat
3a9753132e Make the derivation check more lazy 2021-06-10 08:09:25 +02:00
regnat
2324ee4891 Add a ugly hack to delay errors in getFields 2021-06-10 08:09:25 +02:00
regnat
28c1f8800b Don’t list known absent attributes in listChildren 2021-06-10 08:09:25 +02:00
regnat
af775bdcf9 Make lazyGetAttrField return something more informative 2021-06-10 08:09:25 +02:00
regnat
8d2be51d19 Cache the evaluation errors
Doesn’t seem to make much of a difference on `nix search`, but we’ll
need it at some point
2021-06-10 08:09:25 +02:00
regnat
9102508f33 Use the cache in nix search
Not optimal atm, possibly because we don’t cache the evaluation failures
2021-06-10 08:09:25 +02:00
regnat
a6aaf81103 [TMP]: Disable the tests
To allow the benchmarks to run even when something’s broken
2021-06-10 08:09:25 +02:00
regnat
c7a232e200 Be even lazier for the build evaluation
Now we can not evaluate anything \o/
2021-06-08 16:05:51 +02:00
regnat
30d14b772f Evaluate more lazily in findAlongAttrPath 2021-06-04 16:53:59 +02:00
regnat
fbaee9b8fb Add a “cached thunk” value type 2021-06-04 16:45:22 +02:00
regnat
753730c410 Optimize the nix build caching 2021-06-04 15:21:03 +02:00
regnat
c116e6e837 Another attempt at caching nix build
A different tradeof set probably. I guess we could unify both
2021-06-04 15:16:28 +02:00
regnat
3e261410bc First attempt at caching the evaluation in nix build 2021-06-04 11:39:29 +02:00
regnat
ffec547ebc Add some benchmarks for the actual caching 2021-06-03 16:41:39 +02:00
regnat
512afd8b7a Use the cache in the outer shell
Make sure that we don’t discard it before entering the evaluator proper
2021-06-03 15:29:43 +02:00
regnat
69505c84e1 Commit the cache at the end of the evaluation
Otherwise the db is never properly filled
2021-06-03 15:29:12 +02:00
regnat
af5c323e93 Make the root symbol more telling in the DB
I spent a few minutes trying to understand why I had a field with an
empty symbol, until I realise that it’s because that was the symbol for
the root element.
Now that shouldn’t happen anymore
2021-06-03 15:28:04 +02:00
regnat
9053ac0693 use the setEvalCache function at the flake root
Probably doesn’t change much, but much cleaner (and robust)
2021-06-03 15:27:33 +02:00
regnat
b39ab10749 Properly fill the cache in case of a miss/forward 2021-06-03 15:27:03 +02:00
regnat
8787218c7c (maybe) Actually cache things 2021-06-03 13:06:01 +02:00
regnat
89951cf7fb Extract a Value method to set the eval cache 2021-06-03 12:26:22 +02:00
regnat
45a28ed36f Extract a Value method to get the eval cache 2021-06-03 12:26:02 +02:00
regnat
6ec852e7f0 Simplify the forcing of nested records 2021-06-03 12:11:51 +02:00
regnat
7c718646cb Add some stats for the evaluation caching 2021-06-03 12:11:51 +02:00
regnat
8d95e1f299 Set the cache when opening a flake 2021-06-03 12:11:51 +02:00
regnat
6396416dfa Query the eval cache
(It’s still always empty)
2021-06-03 12:11:51 +02:00
regnat
44f390ed48 Re-split the function, but without the exception 2021-06-03 12:11:51 +02:00
regnat
4ca1a0b864 Try removing the exception 2021-06-03 12:11:51 +02:00
regnat
2021b1d8d1 [TMP] Try inlining the getAttrField function
See what’s making things slow
2021-06-03 12:11:51 +02:00
regnat
b012852cb2 Split a standalone function for accessing an attribute set field 2021-06-03 12:11:51 +02:00
regnat
074e0678bd Add an (always empty) eval cache to the attr sets 2021-06-03 12:11:51 +02:00
regnat
891390d76f [TMP]: Add some benchmarking tools
Not intended to be merged (at least definitely not as it is)
2021-06-03 12:11:51 +02:00
Eelco Dolstra
bb06640971 Merge pull request #4871 from chuahou/master
Install zsh completion script
2021-06-02 13:32:15 +02:00
Eelco Dolstra
f1b604f603 Merge pull request #4879 from keke-cute/fix-optional-namespace
Fix error: 'optional' in namespace 'std' does not name a template type
2021-06-02 13:30:54 +02:00
keke
50dc88a56c fix error: 'optional' in namespace 'std' does not name a template type 2021-06-02 18:09:03 +08:00
Domen Kožar
7c3cb8506f flake.lock: Update
Flake input changes:

* Updated 'nixpkgs': 'github:NixOS/nixpkgs/3a2e0c36e79cecaf196cbea23e75e74710140ea4' -> 'github:NixOS/nixpkgs/bb8a5e54845012ed1375ffd5f317d2fdf434b20e'
2021-06-02 11:58:46 +02:00
Eelco Dolstra
cc9aa8d4b1 Merge pull request #4878 from NixOS/flake-check-keep-going
Let nix flake check keep going when keep-going is set
2021-06-02 11:31:19 +02:00
regnat
838f862f4f doc: Wrap at 80 characters 2021-06-02 11:26:04 +02:00
Théophane Hufschmitt
7565308d04 Fix a documentation typo
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-06-02 11:25:47 +02:00
regnat
7d651f5c3f throw_ -> reportError
This function might or might not throw depending on the value of
`keepGoing`, so naming it `throw_` was a bit confusing
2021-06-02 11:24:31 +02:00
regnat
d12b12a15b Let nix flake check keep going when keep-going is set
When the `keep-going` option is set to `true`, make `nix flake check`
continue as much as it can before failing.

The UI isn’t perfect as-it-is as all the lines currently start with a
mostly useless `error (ignored): error:` prefix, but I’m not sure what
the best output would be, so I’ll leave it as-it-is for the time being

(This is a bit hijacking the `keep-going` flag as it’s supposed to be a
build-time only thing. But I think it’s faire to reuse it here).

Fix https://github.com/NixOS/nix/issues/4450
2021-06-02 11:13:12 +02:00
Chua Hou
aedb5c7301 Install zsh completion script 2021-06-02 00:44:03 +08:00
regnat
5985b8b527 Check the CA hash when importing stuff in the local store
When adding a path to the local store (via `LocalStore::addToStore`),
ensure that the `ca` field of the provided `ValidPathInfo` does indeed
correspond to the content of the path.
Otherwise any untrusted user (or any binary cache) can add arbitrary
content-addressed paths to the store (as content-addressed paths don’t
need a signature).
2021-06-01 15:09:24 +02:00
Eelco Dolstra
48396d940e Merge pull request #4866 from alyssais/libdl
Only link with libdl on Linux
2021-06-01 11:50:50 +02:00
Eelco Dolstra
b8fbfc80fd Merge pull request #4864 from jeremyschlatter/fix-typo
Fix typo in documentation
2021-06-01 11:49:46 +02:00
Eelco Dolstra
f357cea40e Run autoupdate 2021-06-01 11:42:38 +02:00
Eelco Dolstra
83f694762e Merge branch 'aarch64-darwin' of https://github.com/Kloenk/nixos-nix 2021-06-01 11:35:13 +02:00
Eelco Dolstra
caef6f4314 Merge pull request #4734 from p01arst0rm/fix-s3-ifdef
unified macro style for ENABLE_S3
2021-06-01 11:30:21 +02:00
Alyssa Ross
c57ab17687 Only link with libdl on Linux
Linux is (as far as I know) the only mainstream operating system that
requires linking with libdl for dlopen.  On BSD, libdl doesn't exist,
so on non-FreeBSD BSDs linking will currently fail.  On macOS, it's
apparently just a symlink to libSystem (macOS libc), presumably
present for compatibility with things that assume Linux.

So the right thing to do here is to only add -ldl on Linux, not to add
it for everything that isn't FreeBSD.
2021-06-01 08:05:21 +00:00
Finn Behrens
9f1a7f9d37 Include aarch64-darwin in installer
Co-authored-by: Matthew Bauer <mjbauer95@gmail.com>
2021-06-01 09:48:35 +02:00
Finn Behrens
1fefe808f6 enable aarch64-darwin build
disable lowdown sandbox on aarch64-darwin
2021-06-01 09:45:14 +02:00
Jeremy Schlatter
f674f7f434 Fix typo in documentation 2021-06-01 00:05:34 -07:00
Eelco Dolstra
e8f585be70 Merge pull request #4855 from timothyklim/master
Add .tar.zst support for TarballInputScheme
2021-05-31 10:38:04 +02:00
Domen Kožar
1f390922d0 Build for aarch64-darwin 2021-05-29 19:40:56 +02:00
Timothy Klim
4da9ec772c Add .tar.zst support for TarballInputScheme 2021-05-29 16:03:26 +07:00
Eelco Dolstra
b10256af51 Merge pull request #4849 from NixOS/ca/fix-nix-store--export
Make the Nar hash non modulo
2021-05-28 10:48:36 +02:00
Eelco Dolstra
cf1d4299a8 Merge pull request #4853 from Synthetica9/no-freenode
Throw freenode down the memory hole
2021-05-28 10:47:15 +02:00
Patrick Hilhorst
822e338e5c throw freenode down the memory hole 2021-05-27 21:48:39 +02:00
regnat
129384bcf3 Remove the remaining occurenceses of a NarHash modulo 2021-05-26 09:39:29 +02:00
regnat
79ae9e4558 Make the Nar hash non modulo
It makes much more sense to have the Nar hash be a plain straight hash
rather than a hash modulo
2021-05-25 10:58:43 +02:00
Eelco Dolstra
19396f2a8a Merge pull request #4846 from michaeladler/fix-doc
fix doc: nix profile info -> nix profile list
2021-05-25 09:51:39 +02:00
Michael Adler
9e9a8456d7 fix doc: nix profile info -> nix profile list
Signed-off-by: Michael Adler <therisen06@gmail.com>
2021-05-24 12:36:08 +02:00
Eelco Dolstra
af4ff644d5 Merge pull request #4834 from NixOS/generic-closure-function
Extract a generic `computeClosure` function
2021-05-19 13:31:30 +02:00
regnat
184558834a Extract a generic computeClosure function
Move the `closure` logic of `computeFSClosure` to its own (templated) function.

This doesn’t bring much by itself (except for the ability to properly
test the “closure” functionality independently from the rest), but it
allows reusing it (in particular for the realisations which will require
a very similar closure computation)
2021-05-19 11:44:58 +02:00
p01arst0rm
45473d02c9 unified macro style for ENABLE_S3 2021-04-23 07:30:05 +01:00
51 changed files with 1757 additions and 371 deletions

View File

@@ -3,7 +3,7 @@
- Thanks for your contribution!
- To remove the stale label, just leave a new comment.
- _How to find the right people to ping?_ &rarr; [`git blame`](https://git-scm.com/docs/git-blame) to the rescue! (or GitHub's history and blame buttons.)
- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on the [#nixos IRC channel](https://webchat.freenode.net/#nixos).
- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on [Matrix - #nix:nixos.org](https://matrix.to/#/#nix:nixos.org).
## Suggestions for PRs

View File

@@ -12,6 +12,7 @@ makefiles = \
src/resolve-system-dependencies/local.mk \
scripts/local.mk \
misc/bash/local.mk \
misc/zsh/local.mk \
misc/systemd/local.mk \
misc/launchd/local.mk \
misc/upstart/local.mk \

View File

@@ -28,7 +28,8 @@ build nix from source with nix-build or how to get a development environment.
- [Nix manual](https://nixos.org/nix/manual)
- [Nix jobsets on hydra.nixos.org](https://hydra.nixos.org/project/nix)
- [NixOS Discourse](https://discourse.nixos.org/)
- [IRC - #nixos on freenode.net](irc://irc.freenode.net/#nixos)
- [Matrix - #nix:nixos.org](https://matrix.to/#/#nix:nixos.org)
- [IRC - #nixos on libera.chat](irc://irc.libera.chat/#nixos)
## License

53
benchmark.sh Executable file
View File

@@ -0,0 +1,53 @@
#!/bin/sh
set -euo pipefail
set -x
callNix () {
nix \
--experimental-features "nix-command flakes" \
--store /tmp/nix \
"$@"
}
callBuild () {
callNix \
eval --impure --file "$THINGTOBENCH" drvPath \
"$@"
}
getCompletions () {
NIX_GET_COMPLETIONS=6 callNix build "github:NixOS/nixpkgs?rev=ad0d20345219790533ebe06571f82ed6b034db31#firef" "$@"
}
runSearch () {
callNix search "github:NixOS/nixpkgs?rev=ad0d20345219790533ebe06571f82ed6b034db31" firefox "$@"
}
noCache () {
"$@" --option eval-cache false
}
coldCache () {
if [[ -e ~/.cache/nix/eval-cache-v2 || -e ~/.cache/nix/eval-cache-v3 ]]; then
echo "Error: The cache should be clean"
exit 1
fi
"$@"
}
run_all () {
mkdir -p "$out"
NIX_SHOW_STATS=1 NIX_SHOW_STATS_PATH=$out/eval-stats.json bash $0 noCache callBuild
hyperfine \
--warmup 2 \
--export-csv "$out/result.csv" \
--export-json "$out/result.json" \
--export-markdown "$out/result.md" \
--style basic \
--prepare '' "bash $0 noCache callBuild" \
--prepare 'rm -rf ~/.cache/nix/' "bash $0 coldCache callBuild" \
--prepare '' "bash $0 callBuild"
}
"$@"

20
config/config.guess vendored
View File

@@ -1,8 +1,8 @@
#! /bin/sh
# Attempt to guess a canonical system name.
# Copyright 1992-2020 Free Software Foundation, Inc.
# Copyright 1992-2021 Free Software Foundation, Inc.
timestamp='2020-11-19'
timestamp='2021-01-25'
# This file is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
@@ -50,7 +50,7 @@ version="\
GNU config.guess ($timestamp)
Originally written by Per Bothner.
Copyright 1992-2020 Free Software Foundation, Inc.
Copyright 1992-2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
@@ -188,10 +188,9 @@ case "$UNAME_MACHINE:$UNAME_SYSTEM:$UNAME_RELEASE:$UNAME_VERSION" in
#
# Note: NetBSD doesn't particularly care about the vendor
# portion of the name. We always set it to "unknown".
sysctl="sysctl -n hw.machine_arch"
UNAME_MACHINE_ARCH=$( (uname -p 2>/dev/null || \
"/sbin/$sysctl" 2>/dev/null || \
"/usr/sbin/$sysctl" 2>/dev/null || \
/sbin/sysctl -n hw.machine_arch 2>/dev/null || \
/usr/sbin/sysctl -n hw.machine_arch 2>/dev/null || \
echo unknown))
case "$UNAME_MACHINE_ARCH" in
aarch64eb) machine=aarch64_be-unknown ;;
@@ -996,6 +995,9 @@ EOF
k1om:Linux:*:*)
echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
exit ;;
loongarch32:Linux:*:* | loongarch64:Linux:*:* | loongarchx32:Linux:*:*)
echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
exit ;;
m32r*:Linux:*:*)
echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
exit ;;
@@ -1084,7 +1086,7 @@ EOF
ppcle:Linux:*:*)
echo powerpcle-unknown-linux-"$LIBC"
exit ;;
riscv32:Linux:*:* | riscv64:Linux:*:*)
riscv32:Linux:*:* | riscv32be:Linux:*:* | riscv64:Linux:*:* | riscv64be:Linux:*:*)
echo "$UNAME_MACHINE"-unknown-linux-"$LIBC"
exit ;;
s390:Linux:*:* | s390x:Linux:*:*)
@@ -1480,8 +1482,8 @@ EOF
i*86:rdos:*:*)
echo "$UNAME_MACHINE"-pc-rdos
exit ;;
i*86:AROS:*:*)
echo "$UNAME_MACHINE"-pc-aros
*:AROS:*:*)
echo "$UNAME_MACHINE"-unknown-aros
exit ;;
x86_64:VMkernel:*:*)
echo "$UNAME_MACHINE"-unknown-esx

20
config/config.sub vendored
View File

@@ -1,8 +1,8 @@
#! /bin/sh
# Configuration validation subroutine script.
# Copyright 1992-2020 Free Software Foundation, Inc.
# Copyright 1992-2021 Free Software Foundation, Inc.
timestamp='2020-12-02'
timestamp='2021-01-08'
# This file is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
@@ -67,7 +67,7 @@ Report bugs and patches to <config-patches@gnu.org>."
version="\
GNU config.sub ($timestamp)
Copyright 1992-2020 Free Software Foundation, Inc.
Copyright 1992-2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
@@ -1185,6 +1185,7 @@ case $cpu-$vendor in
| k1om \
| le32 | le64 \
| lm32 \
| loongarch32 | loongarch64 | loongarchx32 \
| m32c | m32r | m32rle \
| m5200 | m68000 | m680[012346]0 | m68360 | m683?2 | m68k \
| m6811 | m68hc11 | m6812 | m68hc12 | m68hcs12x \
@@ -1229,7 +1230,7 @@ case $cpu-$vendor in
| powerpc | powerpc64 | powerpc64le | powerpcle | powerpcspe \
| pru \
| pyramid \
| riscv | riscv32 | riscv64 \
| riscv | riscv32 | riscv32be | riscv64 | riscv64be \
| rl78 | romp | rs6000 | rx \
| s390 | s390x \
| score \
@@ -1682,11 +1683,14 @@ fi
# Now, validate our (potentially fixed-up) OS.
case $os in
# Sometimes we do "kernel-abi", so those need to count as OSes.
# Sometimes we do "kernel-libc", so those need to count as OSes.
musl* | newlib* | uclibc*)
;;
# Likewise for "kernel-libc"
eabi | eabihf | gnueabi | gnueabihf)
# Likewise for "kernel-abi"
eabi* | gnueabi*)
;;
# VxWorks passes extra cpu info in the 4th filed.
simlinux | simwindows | spe)
;;
# Now accept the basic system types.
# The portable systems comes first.
@@ -1750,6 +1754,8 @@ case $kernel-$os in
;;
kfreebsd*-gnu* | kopensolaris*-gnu*)
;;
vxworks-simlinux | vxworks-simwindows | vxworks-spe)
;;
nto-qnx*)
;;
os2-emx)

View File

@@ -1,4 +1,4 @@
AC_INIT(nix, m4_esyscmd([bash -c "echo -n $(cat ./.version)$VERSION_SUFFIX"]))
AC_INIT([nix],[m4_esyscmd(bash -c "echo -n $(cat ./.version)$VERSION_SUFFIX")])
AC_CONFIG_MACRO_DIRS([m4])
AC_CONFIG_SRCDIR(README.md)
AC_CONFIG_AUX_DIR(config)
@@ -9,8 +9,7 @@ AC_PROG_SED
AC_CANONICAL_HOST
AC_MSG_CHECKING([for the canonical Nix system name])
AC_ARG_WITH(system, AC_HELP_STRING([--with-system=SYSTEM],
[Platform identifier (e.g., `i686-linux').]),
AC_ARG_WITH(system, AS_HELP_STRING([--with-system=SYSTEM],[Platform identifier (e.g., `i686-linux').]),
[system=$withval],
[case "$host_cpu" in
i*86)
@@ -127,8 +126,7 @@ NEED_PROG(jq, jq)
AC_SUBST(coreutils, [$(dirname $(type -p cat))])
AC_ARG_WITH(store-dir, AC_HELP_STRING([--with-store-dir=PATH],
[path of the Nix store (defaults to /nix/store)]),
AC_ARG_WITH(store-dir, AS_HELP_STRING([--with-store-dir=PATH],[path of the Nix store (defaults to /nix/store)]),
storedir=$withval, storedir='/nix/store')
AC_SUBST(storedir)
@@ -157,8 +155,7 @@ fi
PKG_PROG_PKG_CONFIG
AC_ARG_ENABLE(shared, AC_HELP_STRING([--enable-shared],
[Build shared libraries for Nix [default=yes]]),
AC_ARG_ENABLE(shared, AS_HELP_STRING([--enable-shared],[Build shared libraries for Nix [default=yes]]),
shared=$enableval, shared=yes)
if test "$shared" = yes; then
AC_SUBST(BUILD_SHARED_LIBS, 1, [Whether to build shared libraries.])
@@ -215,9 +212,8 @@ AC_SUBST(HAVE_LIBCPUID, [$have_libcpuid])
# Look for libseccomp, required for Linux sandboxing.
if test "$sys_name" = linux; then
AC_ARG_ENABLE([seccomp-sandboxing],
AC_HELP_STRING([--disable-seccomp-sandboxing],
[Don't build support for seccomp sandboxing (only recommended if your arch doesn't support libseccomp yet!)]
))
AS_HELP_STRING([--disable-seccomp-sandboxing],[Don't build support for seccomp sandboxing (only recommended if your arch doesn't support libseccomp yet!)
]))
if test "x$enable_seccomp_sandboxing" != "xno"; then
PKG_CHECK_MODULES([LIBSECCOMP], [libseccomp],
[CXXFLAGS="$LIBSECCOMP_CFLAGS $CXXFLAGS"])
@@ -235,8 +231,8 @@ AC_SUBST(HAVE_SECCOMP, [$have_seccomp])
# Look for aws-cpp-sdk-s3.
AC_LANG_PUSH(C++)
AC_CHECK_HEADERS([aws/s3/S3Client.h],
[AC_DEFINE([ENABLE_S3], [1], [Whether to enable S3 support via aws-sdk-cpp.])
enable_s3=1], [enable_s3=])
[AC_DEFINE([ENABLE_S3], [1], [Whether to enable S3 support via aws-sdk-cpp.]) enable_s3=1],
[AC_DEFINE([ENABLE_S3], [0], [Whether to enable S3 support via aws-sdk-cpp.]) enable_s3=])
AC_SUBST(ENABLE_S3, [$enable_s3])
AC_LANG_POP(C++)
@@ -249,8 +245,7 @@ fi
# Whether to use the Boehm garbage collector.
AC_ARG_ENABLE(gc, AC_HELP_STRING([--enable-gc],
[enable garbage collection in the Nix expression evaluator (requires Boehm GC) [default=yes]]),
AC_ARG_ENABLE(gc, AS_HELP_STRING([--enable-gc],[enable garbage collection in the Nix expression evaluator (requires Boehm GC) [default=yes]]),
gc=$enableval, gc=yes)
if test "$gc" = yes; then
PKG_CHECK_MODULES([BDW_GC], [bdw-gc])
@@ -264,8 +259,7 @@ PKG_CHECK_MODULES([GTEST], [gtest_main])
# documentation generation switch
AC_ARG_ENABLE(doc-gen, AC_HELP_STRING([--disable-doc-gen],
[disable documentation generation]),
AC_ARG_ENABLE(doc-gen, AS_HELP_STRING([--disable-doc-gen],[disable documentation generation]),
doc_generate=$enableval, doc_generate=yes)
AC_SUBST(doc_generate)
@@ -285,8 +279,7 @@ if test "$(uname)" = "Darwin"; then
fi
AC_ARG_WITH(sandbox-shell, AC_HELP_STRING([--with-sandbox-shell=PATH],
[path of a statically-linked shell to use as /bin/sh in sandboxes]),
AC_ARG_WITH(sandbox-shell, AS_HELP_STRING([--with-sandbox-shell=PATH],[path of a statically-linked shell to use as /bin/sh in sandboxes]),
sandbox_shell=$withval)
AC_SUBST(sandbox_shell)
@@ -301,6 +294,6 @@ done
rm -f Makefile.config
AC_CONFIG_HEADER([config.h])
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([])
AC_OUTPUT

8
flake.lock generated
View File

@@ -19,16 +19,16 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1614309161,
"narHash": "sha256-93kRxDPyEW9QIpxU71kCaV1r+hgOgP6/aVgC7vvO8IU=",
"lastModified": 1622593737,
"narHash": "sha256-9loxFJg85AbzJrSkU4pE/divZ1+zOxDy2FSjlrufCB8=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "0e499fde7af3c28d63e9b13636716b86c3162b93",
"rev": "bb8a5e54845012ed1375ffd5f317d2fdf434b20e",
"type": "github"
},
"original": {
"id": "nixpkgs",
"ref": "nixos-20.09-small",
"ref": "nixos-21.05-small",
"type": "indirect"
}
},

View File

@@ -1,7 +1,7 @@
{
description = "The purely functional package manager";
inputs.nixpkgs.url = "nixpkgs/nixos-20.09-small";
inputs.nixpkgs.url = "nixpkgs/nixos-21.05-small";
inputs.lowdown-src = { url = "github:kristapsdz/lowdown/VERSION_0_8_4"; flake = false; };
outputs = { self, nixpkgs, lowdown-src }:
@@ -18,7 +18,7 @@
linux64BitSystems = [ "x86_64-linux" "aarch64-linux" ];
linuxSystems = linux64BitSystems ++ [ "i686-linux" ];
systems = linuxSystems ++ [ "x86_64-darwin" ];
systems = linuxSystems ++ [ "x86_64-darwin" "aarch64-darwin" ];
forAllSystems = f: nixpkgs.lib.genAttrs systems (system: f system);
@@ -229,7 +229,7 @@
echo "doc manual $doc/share/doc/nix/manual" >> $doc/nix-support/hydra-build-products
'';
doInstallCheck = true;
doInstallCheck = false;
installCheckFlags = "sysconfdir=$(out)/etc";
separateDebugInfo = true;
@@ -286,14 +286,21 @@
nativeBuildInputs = [ which ];
configurePhase =
''
configurePhase = ''
${if (stdenv.isDarwin && stdenv.isAarch64) then "echo \"HAVE_SANDBOX_INIT=false\" > configure.local" else ""}
./configure \
PREFIX=${placeholder "dev"} \
BINDIR=${placeholder "bin"}/bin
'';
};
nix-benchmarks = prev.runCommandNoCC "nix-benchmarks" {
buildInputs = [ final.nix prev.hyperfine ];
THINGTOBENCH = ./thingToBench.nix;
NIX_PATH="nixpkgs=${prev.path}";
} ''
bash ${./benchmark.sh} run_all
'';
};
hydraJobs = {
@@ -387,7 +394,7 @@
# to https://nixos.org/nix/install. It downloads the binary
# tarball for the user's system and calls the second half of the
# installation script.
installerScript = installScriptFor [ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ];
installerScript = installScriptFor [ "x86_64-linux" "i686-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" ];
installerScriptForGHA = installScriptFor [ "x86_64-linux" "x86_64-darwin" ];
# Line coverage analysis.
@@ -485,7 +492,7 @@
});
packages = forAllSystems (system: {
inherit (nixpkgsFor.${system}) nix;
inherit (nixpkgsFor.${system}) nix nix-benchmarks;
} // nixpkgs.lib.optionalAttrs (builtins.elem system linux64BitSystems) {
nix-static = let
nixpkgs = nixpkgsFor.${system}.pkgsStatic;

1
misc/zsh/local.mk Normal file
View File

@@ -0,0 +1 @@
$(eval $(call install-file-as, $(d)/completion.zsh, $(datarootdir)/zsh/site-functions/_nix, 0644))

View File

@@ -8,8 +8,13 @@ endif
libnixrust_PATH := $(d)/target/$(RUST_DIR)/libnixrust.$(SO_EXT)
libnixrust_INSTALL_PATH := $(libdir)/libnixrust.$(SO_EXT)
libnixrust_LDFLAGS_USE := -L$(d)/target/$(RUST_DIR) -lnixrust -ldl
libnixrust_LDFLAGS_USE_INSTALLED := -L$(libdir) -lnixrust -ldl
libnixrust_LDFLAGS_USE := -L$(d)/target/$(RUST_DIR) -lnixrust
libnixrust_LDFLAGS_USE_INSTALLED := -L$(libdir) -lnixrust
ifeq ($(OS), Linux)
libnixrust_LDFLAGS_USE += -ldl
libnixrust_LDFLAGS_USE_INSTALLED += -ldl
endif
ifeq ($(OS), Darwin)
libnixrust_BUILD_FLAGS = NIX_LDFLAGS="-undefined dynamic_lookup"

View File

@@ -63,7 +63,8 @@ contact_us() {
echo "You can open an issue at https://github.com/nixos/nix/issues"
echo ""
echo "Or feel free to contact the team:"
echo " - IRC: in #nixos on irc.freenode.net"
echo " - Matrix: #nix:nixos.org"
echo " - IRC: in #nixos on irc.libera.chat"
echo " - twitter: @nixos_org"
echo " - forum: https://discourse.nixos.org"
}

View File

@@ -46,15 +46,9 @@ case "$(uname -s).$(uname -m)" in
system=x86_64-darwin
;;
Darwin.arm64|Darwin.aarch64)
# check for Rosetta 2 support
if ! [ -f /Library/Apple/System/Library/LaunchDaemons/com.apple.oahd.plist ]; then
oops "Rosetta 2 is not installed on this ARM64 macOS machine. Run softwareupdate --install-rosetta then restart installation"
fi
hash=@tarballHash_x86_64-darwin@
path=@tarballPath_x86_64-darwin@
# eventually maybe: aarch64-darwin
system=x86_64-darwin
hash=@binaryTarball_aarch64-darwin@
path=@tarballPath_aarch64-darwin@
system=aarch64-darwin
;;
*) oops "sorry, there is no binary distribution of Nix for your platform";;
esac

View File

@@ -302,6 +302,14 @@ Installable::getCursors(EvalState & state)
return {{evalCache->getRoot(), ""}};
}
std::pair<Value*, Pos> Installable::toValue(EvalState & state)
{
auto values = toValues(state);
if (values.empty())
throw Error("cannot find flake attribute '%s'", what());
return {std::get<0>(values[0]), std::get<1>(values[0])};
}
std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>
Installable::getCursor(EvalState & state)
{
@@ -378,11 +386,11 @@ struct InstallableAttrPath : InstallableValue
std::string what() override { return attrPath; }
std::pair<Value *, Pos> toValue(EvalState & state) override
std::vector<std::tuple<Value *, Pos, std::string>> toValues(EvalState & state) override
{
auto [vRes, pos] = findAlongAttrPath(state, attrPath, *cmd.getAutoArgs(state), **v);
state.forceValue(*vRes);
return {vRes, pos};
return {{vRes, pos, attrPath}};
}
virtual std::vector<InstallableValue::DerivationInfo> toDerivations() override;
@@ -428,12 +436,10 @@ Value * InstallableFlake::getFlakeOutputs(EvalState & state, const flake::Locked
callFlake(state, lockedFlake, *vFlake);
auto aOutputs = vFlake->attrs->get(state.symbols.create("outputs"));
assert(aOutputs);
state.forceValue(*aOutputs->value);
return aOutputs->value;
auto vRes = state.allocValue();
auto gotField = state.lazyGetAttrField(*vFlake, {state.symbols.create("outputs")}, noPos, *vRes);
assert(gotField != EvalState::LazyValueType::Missing);
return vRes;
}
ref<eval_cache::EvalCache> openEvalCache(
@@ -500,25 +506,34 @@ std::tuple<std::string, FlakeRef, InstallableValue::DerivationInfo> InstallableF
auto root = cache->getRoot();
for (auto & attrPath : getActualAttrPaths()) {
auto attr = root->findAlongAttrPath(
parseAttrPath(*state, attrPath),
true
);
auto emptyArgs = state->allocBindings(0);
try {
auto [drvValue, pos] = findAlongAttrPath(
*state,
attrPath,
*emptyArgs,
*getFlakeOutputs(*state, *lockedFlake)
);
Value * v = state->allocValue();
if (!state->getAttrField(*drvValue, {state->sDrvPath}, pos, *v))
break;
auto drvPath = state->forceString(*v);
if (!state->getAttrField(*drvValue, {state->sOutPath}, pos, *v))
break;
auto outPath = state->forceString(*v);
if (!state->getAttrField(*drvValue, {state->sOutputName}, pos, *v))
break;
auto outputName = state->forceString(*v);
if (!attr) continue;
auto drvInfo = DerivationInfo{
state->store->parseStorePath(drvPath),
state->store->maybeParseStorePath(outPath),
outputName
};
if (!attr->isDerivation())
throw Error("flake output attribute '%s' is not a derivation", attrPath);
auto drvPath = attr->forceDerivation();
auto drvInfo = DerivationInfo{
std::move(drvPath),
state->store->maybeParseStorePath(attr->getAttr(state->sOutPath)->getString()),
attr->getAttr(state->sOutputName)->getString()
};
return {attrPath, lockedFlake->flake.lockedRef, std::move(drvInfo)};
return {attrPath, lockedFlake->flake.lockedRef, std::move(drvInfo)};
} catch (AttrPathNotFound & e) {
}
}
throw Error("flake '%s' does not provide attribute %s",
@@ -532,25 +547,22 @@ std::vector<InstallableValue::DerivationInfo> InstallableFlake::toDerivations()
return res;
}
std::pair<Value *, Pos> InstallableFlake::toValue(EvalState & state)
std::vector<std::tuple<Value *, Pos, std::string>>
InstallableFlake::toValues(EvalState & state)
{
auto lockedFlake = getLockedFlake();
auto vOutputs = getFlakeOutputs(state, *lockedFlake);
auto vOutputs = getFlakeOutputs(state, *getLockedFlake());
auto emptyArgs = state.allocBindings(0);
std::vector<std::tuple<Value *, Pos, std::string>> res;
for (auto & attrPath : getActualAttrPaths()) {
try {
auto [v, pos] = findAlongAttrPath(state, attrPath, *emptyArgs, *vOutputs);
state.forceValue(*v);
return {v, pos};
} catch (AttrPathNotFound & e) {
}
res.push_back({v, pos, attrPath});
} catch (Error &) {}
}
throw Error("flake '%s' does not provide attribute %s",
flakeRef, showAttrPaths(getActualAttrPaths()));
return res;
}
std::vector<std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>>

View File

@@ -41,10 +41,11 @@ struct Installable
UnresolvedApp toApp(EvalState & state);
virtual std::pair<Value *, Pos> toValue(EvalState & state)
virtual std::vector<std::tuple<Value *, Pos, std::string>> toValues(EvalState & state)
{
throw Error("argument '%s' cannot be evaluated", what());
}
std::pair<Value *, Pos> toValue(EvalState & state);
/* Return a value only if this installable is a store path or a
symlink to it. */
@@ -109,7 +110,7 @@ struct InstallableFlake : InstallableValue
std::vector<DerivationInfo> toDerivations() override;
std::pair<Value *, Pos> toValue(EvalState & state) override;
std::vector<std::tuple<Value *, Pos, std::string>> toValues(EvalState & state) override;
std::vector<std::pair<std::shared_ptr<eval_cache::AttrCursor>, std::string>>
getCursors(EvalState & state) override;

View File

@@ -58,26 +58,23 @@ std::pair<Value *, Pos> findAlongAttrPath(EvalState & state, const string & attr
Value * vNew = state.allocValue();
state.autoCallFunction(autoArgs, *v, *vNew);
v = vNew;
state.forceValue(*v);
/* It should evaluate to either a set or an expression,
according to what is specified in the attrPath. */
if (!attrIndex) {
if (v->type() != nAttrs)
throw TypeError(
"the expression selected by the selection path '%1%' should be a set but is %2%",
attrPath,
showType(*v));
if (attr.empty())
throw Error("empty attribute name in selection path '%1%'", attrPath);
Bindings::iterator a = v->attrs->find(state.symbols.create(attr));
if (a == v->attrs->end())
auto v2 = state.allocValue();
auto gotField = state.lazyGetAttrField(*v, {state.symbols.create(attr)}, pos, *v2) != EvalState::LazyValueType::Missing;
if (!gotField)
throw AttrPathNotFound("attribute '%1%' in selection path '%2%' not found", attr, attrPath);
v = &*a->value;
pos = *a->pos;
v = v2;
/* Bindings::iterator a = v->attrs->find(state.symbols.create(attr)); */
/* v = &*a->value; */
/* pos = *a->pos; */
}
else {

View File

@@ -2,6 +2,7 @@
#include "nixexpr.hh"
#include "symbol-table.hh"
#include "value-cache.hh"
#include <algorithm>
#include <optional>
@@ -36,6 +37,7 @@ class Bindings
public:
typedef uint32_t size_t;
Pos *pos;
ValueCache eval_cache;
private:
size_t size_, capacity_;

21
src/libexpr/context.cc Normal file
View File

@@ -0,0 +1,21 @@
#include "context.hh"
namespace nix {
/* Decode a context string !<name>!<path> into a pair <path,
name>. */
std::pair<string, string> decodeContext(std::string_view s)
{
if (s.at(0) == '!') {
size_t index = s.find("!", 1);
return {std::string(s.substr(index + 1)), std::string(s.substr(1, index - 1))};
} else
return {s.at(0) == '/' ? std::string(s) : std::string(s.substr(1)), ""};
}
std::string encodeContext(std::string_view name, std::string_view path)
{
return "!" + std::string(name) + "!" + std::string(path);
}
}

11
src/libexpr/context.hh Normal file
View File

@@ -0,0 +1,11 @@
#include "util.hh"
namespace nix {
/* Decode a context string !<name>!<path> into a pair <path,
name>. */
std::pair<string, string> decodeContext(std::string_view s);
std::string encodeContext(std::string_view name, std::string_view path);
}

View File

@@ -46,6 +46,12 @@ void EvalState::forceValue(Value & v, const Pos & pos)
}
else if (v.isApp())
callFunction(*v.app.left, *v.app.right, v, noPos);
else if (v.isCachedThunk()) {
auto evalCache = v.getEvalCache();
v.mkThunk(v.cachedThunk.thunk->env, v.cachedThunk.thunk->expr);
forceValue(v, pos);
v.setEvalCache(evalCache);
}
else if (v.isBlackhole())
throwEvalError(pos, "infinite recursion encountered");
}

View File

@@ -1,4 +1,5 @@
#include "eval.hh"
#include "value-cache.hh"
#include "hash.hh"
#include "util.hh"
#include "store-api.hh"
@@ -8,6 +9,7 @@
#include "filetransfer.hh"
#include "json.hh"
#include "function-trace.hh"
#include "value-cache.hh"
#include <algorithm>
#include <chrono>
@@ -122,6 +124,7 @@ void printValue(std::ostream & str, std::set<const Value *> & active, const Valu
str << "]";
break;
case tThunk:
case tCachedThunk:
case tApp:
str << "<CODE>";
break;
@@ -193,7 +196,7 @@ string showType(const Value & v)
case tPrimOpApp:
return fmt("the partially applied built-in function '%s'", string(getPrimOp(v)->primOp->name));
case tExternal: return v.external->showType();
case tThunk: return "a thunk";
case tThunk: case tCachedThunk: return "a thunk";
case tApp: return "a function application";
case tBlackhole: return "a black hole";
default:
@@ -216,7 +219,7 @@ bool Value::isTrivial() const
return
internalType != tApp
&& internalType != tPrimOpApp
&& (internalType != tThunk
&& ((internalType != tThunk && internalType != tCachedThunk)
|| (dynamic_cast<ExprAttrs *>(thunk.expr)
&& ((ExprAttrs *) thunk.expr)->dynamicAttrs.empty())
|| dynamic_cast<ExprLambda *>(thunk.expr)
@@ -442,6 +445,9 @@ EvalState::EvalState(const Strings & _searchPath, ref<Store> store)
EvalState::~EvalState()
{
for (auto [_, cache] : evalCache) {
cache->commit();
}
}
@@ -1107,68 +1113,235 @@ void ExprVar::eval(EvalState & state, Env & env, Value & v)
v = *v2;
}
unsigned long nrLookups = 0;
static string showAttrPath(EvalState & state, Env & env, const AttrPath & attrPath)
std::pair<ValueCache::CacheResult, ValueCache> ValueCache::getValue(EvalState & state, const std::vector<Symbol> & selector, Value & dest)
{
std::ostringstream out;
bool first = true;
for (auto & i : attrPath) {
if (!first) out << '.'; else first = false;
try {
out << getName(i, state, env);
} catch (Error & e) {
assert(!i.symbol.set());
out << "\"${" << *i.expr << "}\"";
}
}
return out.str();
if (!rawCache)
return { {NoCacheKey}, ValueCache(nullptr) };
auto resultingCursor = rawCache->findAlongAttrPath(selector);
if (!resultingCursor)
return { {CacheMiss}, ValueCache(nullptr) };
auto cachedValue = resultingCursor->getCachedValue();
auto cacheResult = std::visit(
overloaded{
[&](tree_cache::attributeSet_t) { return ValueCache::CacheResult{ Forward }; },
[&](tree_cache::unknown_t) { return ValueCache::CacheResult{ UnCacheable }; },
[&](tree_cache::thunk_t) { return ValueCache::CacheResult{ CacheMiss }; },
[&](tree_cache::failed_t x) -> ValueCache::CacheResult {throw EvalError(x.error); },
[&](tree_cache::missing_t x) {
return ValueCache::CacheResult{
.returnCode = CacheHit,
.lastQueriedSymbolIfMissing = x.attrName
};
},
[&](tree_cache::string_t s) {
PathSet context;
for (auto& [pathName, outputName] : s.second) {
// If the cached value depends on some non-existent
// path, we need to discard it and force the evaluation
// to bring back the context in the store
if (!state.store->isValidPath(
state.store->parseStorePath(pathName)))
return ValueCache::CacheResult{UnCacheable};
context.insert("!" + outputName + "!" + pathName);
}
mkString(dest, s.first, context);
return ValueCache::CacheResult{CacheHit};
},
[&](tree_cache::wrapped_basetype<bool> b) {
dest.mkBool(b.value);
return ValueCache::CacheResult{CacheHit};
},
[&](tree_cache::wrapped_basetype<int64_t> i) {
dest.mkInt(i.value);
return ValueCache::CacheResult{CacheHit};
},
[&](tree_cache::wrapped_basetype<double> d) {
dest.mkFloat(d.value);
return ValueCache::CacheResult{CacheHit};
},
},
cachedValue);
return { cacheResult, ValueCache(resultingCursor) };
}
void EvalState::updateCacheStats(ValueCache::CacheResult cacheResult)
{
switch (cacheResult.returnCode) {
case ValueCache::CacheHit:
nrCacheHits++;
break;
case ValueCache::CacheMiss:
nrCacheMisses++;
break;
case ValueCache::UnCacheable:
nrUncacheable++;
break;
case ValueCache::NoCacheKey:
nrUncached++;
break;
case ValueCache::Forward:
nrCacheHits++;
break;
};
}
unsigned long nrLookups = 0;
EvalState::LazyValueType EvalState::lazyGetAttrField(Value & attrs, const std::vector<Symbol> & selector, const Pos & pos, Value & dest)
{
auto eval_cache = attrs.getEvalCache();
if (eval_cache.isEmpty()) {
forceValue(attrs, pos);
eval_cache = attrs.getEvalCache();
}
auto [ cacheResult, resultingCursor ] = eval_cache.getValue(*this, selector, dest);
updateCacheStats(cacheResult);
auto delayValue = [&](ValueCache & cursor) {
auto recordAsVar = new ExprCastedVar(&attrs);
auto accessExpr = new ExprSelect(pos, recordAsVar, selector);
auto thunk = (Thunk*)allocBytes(sizeof(Thunk));
thunk->expr = accessExpr;
thunk->env = &baseEnv;
dest.mkCachedThunk(
thunk,
new ValueCache(cursor)
);
};
switch (cacheResult.returnCode) {
case ValueCache::CacheHit:
if (cacheResult.lastQueriedSymbolIfMissing)
return LazyValueType::Missing;
return LazyValueType::PlainValue;
case ValueCache::UnCacheable:
delayValue(resultingCursor);
return LazyValueType::DelayedUncacheable;
case ValueCache::Forward:
delayValue(resultingCursor);
return LazyValueType::DelayedAttr;
default:
if(getAttrField(attrs, selector, pos, dest))
return LazyValueType::PlainValue;
else
return LazyValueType::Missing;
}
}
bool EvalState::getAttrField(Value & attrs, const std::vector<Symbol> & selector, const Pos & pos, Value & dest)
{
Pos * pos2 = 0;
auto eval_cache = attrs.getEvalCache();
if (eval_cache.isEmpty()) {
forceValue(attrs, pos);
eval_cache = attrs.getEvalCache();
}
auto [ cacheResult, resultingCursor ] = eval_cache.getValue(*this, selector, dest);
updateCacheStats(cacheResult);
switch (cacheResult.returnCode) {
case ValueCache::CacheHit:
if (cacheResult.lastQueriedSymbolIfMissing)
return false;
return true;
case ValueCache::CacheMiss:
resultingCursor = eval_cache;
break;
case ValueCache::Forward: // FIXME: Handle properly
case ValueCache::NoCacheKey:
case ValueCache::UnCacheable:
;
}
forceValue(attrs, pos);
Value * vAttrs = &attrs;
try {
for (auto & name : selector) {
nrLookups++;
Bindings::iterator j;
if (vAttrs->type() != nAttrs ||
(j = vAttrs->attrs->find(name)) == vAttrs->attrs->end()) {
return false;
}
vAttrs = j->value;
pos2 = j->pos;
try {
forceValue(*vAttrs, pos2 != NULL ? *pos2 : pos );
} catch (EvalError & e) {
resultingCursor.addFailedChild(name, e);
throw;
}
if (cacheResult.returnCode == ValueCache::CacheMiss) {
resultingCursor = resultingCursor.addChild(name, *vAttrs);
vAttrs->setEvalCache(resultingCursor);
}
if (countCalls && pos2) attrSelects[*pos2]++;
}
} catch (Error & e) {
if (pos2 && pos2->file != sDerivationNix) {
vector<string> strSelector;
addErrorTrace(e, *pos2, "while evaluating the attribute '%1%'", concatStringsSep(".", selector));
}
throw;
}
if (cacheResult.returnCode == ValueCache::Forward)
vAttrs->setEvalCache(resultingCursor);
dest = *vAttrs;
return true;
}
void EvalState::getAttrFieldThrow(Value & attrs, const std::vector<Symbol> & selector, const Pos & pos, Value & dest)
{
if (!getAttrField(attrs, selector, pos, dest))
throw Error("Missing attribute path '%s'", "ImTooLazyToImplementThisRightNow");
}
std::vector<Symbol> EvalState::getFields(Value & attrs, const Pos & pos)
{
auto eval_cache = attrs.getEvalCache();
if (eval_cache.isEmpty()) {
forceValue(attrs, pos);
eval_cache = attrs.getEvalCache();
}
if (auto attrNames = eval_cache.listChildren(symbols)) {
return *attrNames;
}
forceAttrs(attrs);
std::vector<Symbol> res;
for (auto & attr : *attrs.attrs)
res.push_back(attr.name);
return res;
}
void ExprSelect::eval(EvalState & state, Env & env, Value & v)
{
Value vTmp;
Pos * pos2 = 0;
Value * vAttrs = &vTmp;
e->eval(state, env, vTmp);
try {
std::vector<Symbol> selector;
for (auto & i : attrPath)
selector.push_back(getName(i, state, env));
for (auto & i : attrPath) {
nrLookups++;
Bindings::iterator j;
Symbol name = getName(i, state, env);
if (def) {
state.forceValue(*vAttrs, pos);
if (vAttrs->type() != nAttrs ||
(j = vAttrs->attrs->find(name)) == vAttrs->attrs->end())
{
def->eval(state, env, v);
return;
}
} else {
state.forceAttrs(*vAttrs, pos);
if ((j = vAttrs->attrs->find(name)) == vAttrs->attrs->end())
throwEvalError(pos, "attribute '%1%' missing", name);
}
vAttrs = j->value;
pos2 = j->pos;
if (state.countCalls && pos2) state.attrSelects[*pos2]++;
bool gotField = state.getAttrField(vTmp, selector, pos, v);
if (!gotField) {
if (def) {
def->eval(state, env, v);
return;
} else {
throwEvalError(pos, "Missing field");
}
state.forceValue(*vAttrs, ( pos2 != NULL ? *pos2 : this->pos ) );
} catch (Error & e) {
if (pos2 && pos2->file != state.sDerivationNix)
addErrorTrace(e, *pos2, "while evaluating the attribute '%1%'",
showAttrPath(state, env, attrPath));
throw;
}
v = *vAttrs;
}
@@ -1351,9 +1524,38 @@ void EvalState::incrFunctionCall(ExprLambda * fun)
functionCalls[fun]++;
}
std::optional<tree_cache::AttrValue> ValueCache::getRawValue()
{
if (!rawCache)
return std::nullopt;
return rawCache->getCachedValue();
}
std::optional<std::vector<Symbol>> ValueCache::listChildren(SymbolTable& symbols)
{
auto ret = std::vector<Symbol>();
if (rawCache) {
auto cachedValue = rawCache->getCachedValue();
if (std::get_if<tree_cache::attributeSet_t>(&cachedValue)) {
for (auto & fieldStr : rawCache->getChildren())
ret.push_back(symbols.create(fieldStr));
}
return ret;
}
return std::nullopt;
}
void EvalState::autoCallFunction(Bindings & args, Value & fun, Value & res)
{
if (auto evalCache = fun.getEvalCache(); !evalCache.isEmpty()) {
if (auto cacheValue = evalCache.getRawValue()) {
if (std::holds_alternative<tree_cache::attributeSet_t>(*cacheValue)) {
res = fun;
return;
}
}
}
forceValue(fun);
if (fun.type() == nAttrs) {
@@ -1699,18 +1901,6 @@ string EvalState::forceString(Value & v, const Pos & pos)
}
/* Decode a context string !<name>!<path> into a pair <path,
name>. */
std::pair<string, string> decodeContext(std::string_view s)
{
if (s.at(0) == '!') {
size_t index = s.find("!", 1);
return {std::string(s.substr(index + 1)), std::string(s.substr(1, index - 1))};
} else
return {s.at(0) == '/' ? std::string(s) : std::string(s.substr(1)), ""};
}
void copyContext(const Value & v, PathSet & context)
{
if (v.string.context)
@@ -1729,6 +1919,102 @@ std::vector<std::pair<Path, std::string>> Value::getContext()
return res;
}
ValueCache & Value::getEvalCache()
{
if (internalType == tAttrs) {
return attrs->eval_cache;
} else if (internalType == tCachedThunk) {
return *cachedThunk.cache;
} else {
return ValueCache::empty;
}
}
ValueCache ValueCache::empty = ValueCache(nullptr);
tree_cache::AttrValue cachedValueFor(Value& v)
{
tree_cache::AttrValue valueToCache;
switch (v.type()) {
case nThunk:
valueToCache = tree_cache::thunk_t{};
break;
case nNull:
case nList:
case nFunction:
case nExternal:
valueToCache = tree_cache::unknown_t{};
break;
case nBool:
valueToCache = tree_cache::wrapped_basetype<bool>{v.boolean};
break;
case nString:
valueToCache = tree_cache::string_t{
v.string.s,
v.getContext()
};
break;
case nPath:
valueToCache = tree_cache::string_t{
v.path,
{}
};
break;
case nAttrs:
valueToCache = tree_cache::attributeSet_t{};
break;
case nInt:
valueToCache = tree_cache::wrapped_basetype<int64_t>{v.integer};
break;
case nFloat:
valueToCache = tree_cache::wrapped_basetype<double>{v.fpoint};
break;
};
return valueToCache;
}
ValueCache ValueCache::addChild(const Symbol& name, Value& value)
{
if (!rawCache)
return ValueCache::empty;
auto cachedValue = cachedValueFor(value);
auto ret = ValueCache(rawCache->addChild(name, cachedValue));
if (value.type() == nAttrs)
ret.addAttrSetChilds(*value.attrs);
return ret;
}
ValueCache ValueCache::addFailedChild(const Symbol& name, const Error & error)
{
if (!rawCache) return ValueCache::empty;
return ValueCache(rawCache->addChild(name, tree_cache::failed_t{ .error = error.msg() }));
}
void ValueCache::addAttrSetChilds(Bindings & children)
{
if (!rawCache) return;
for (auto & attr : children) {
// We could in theory directly store the value, but that would cause
// an infinite recursion in case of a cyclic attrset.
// So in a first time, just store a thunk
if (attr.value->type() == nAttrs)
rawCache->addChild(attr.name, tree_cache::thunk_t{});
else
addChild(attr.name, *attr.value);
}
}
void Value::setEvalCache(ValueCache & newCache)
{
if (internalType == tAttrs) {
attrs->eval_cache = newCache;
} else if (internalType == tCachedThunk) {
cachedThunk.cache = &newCache;
}
}
string EvalState::forceString(Value & v, PathSet & context, const Pos & pos)
{
@@ -2011,6 +2297,13 @@ void EvalState::printStats()
topObj.attr("nrLookups", nrLookups);
topObj.attr("nrPrimOpCalls", nrPrimOpCalls);
topObj.attr("nrFunctionCalls", nrFunctionCalls);
{
auto cache = topObj.object("evalCache");
cache.attr("nrCacheMisses", nrCacheMisses);
cache.attr("nrCacheHits", nrCacheHits);
cache.attr("nrUncached", nrUncached);
cache.attr("nrUncacheable", nrUncacheable);
}
#if HAVE_BOEHMGC
{
auto gc = topObj.object("gc");
@@ -2111,6 +2404,21 @@ Strings EvalSettings::getDefaultNixPath()
return res;
}
std::shared_ptr<tree_cache::Cache> EvalState::openTreeCache(Hash cacheKey)
{
if (auto iter = evalCache.find(cacheKey); iter != evalCache.end())
return iter->second;
if (!(evalSettings.useEvalCache && evalSettings.pureEval))
return nullptr;
auto thisCache = tree_cache::Cache::tryCreate(
cacheKey,
symbols
);
evalCache.insert({cacheKey, thisCache});
return thisCache;
}
EvalSettings evalSettings;
static GlobalConfig::Register rEvalSettings(&evalSettings);

View File

@@ -1,6 +1,7 @@
#pragma once
#include "attr-set.hh"
#include "context.hh"
#include "value.hh"
#include "nixexpr.hh"
#include "symbol-table.hh"
@@ -115,6 +116,7 @@ private:
typedef std::map<Path, Value> FileEvalCache;
#endif
FileEvalCache fileEvalCache;
std::map<Hash, std::shared_ptr<tree_cache::Cache>> evalCache;
SearchPath searchPath;
@@ -131,6 +133,8 @@ public:
EvalState(const Strings & _searchPath, ref<Store> store);
~EvalState();
std::shared_ptr<tree_cache::Cache> openTreeCache(Hash);
void addToSearchPath(const string & s);
SearchPath getSearchPath() { return searchPath; }
@@ -310,6 +314,8 @@ public:
void realiseContext(const PathSet & context);
void updateCacheStats(ValueCache::CacheResult);
private:
unsigned long nrEnvs = 0;
@@ -323,6 +329,10 @@ private:
unsigned long nrListConcats = 0;
unsigned long nrPrimOpCalls = 0;
unsigned long nrFunctionCalls = 0;
unsigned long nrCacheHits = 0;
unsigned long nrCacheMisses = 0;
unsigned long nrUncacheable = 0;
unsigned long nrUncached = 0;
bool countCalls;
@@ -342,6 +352,27 @@ private:
friend struct ExprSelect;
friend void prim_getAttr(EvalState & state, const Pos & pos, Value * * args, Value & v);
friend void prim_match(EvalState & state, const Pos & pos, Value * * args, Value & v);
public:
bool getAttrField(Value & attrs, const std::vector<Symbol> & selector, const Pos & pos, Value & dest);
enum struct LazyValueType {
PlainValue,
DelayedUncacheable,
DelayedAttr,
Missing,
};
// Similar to `getAttrField`, but if the cache says that the result is an
// attribute set, just return a thunk to it rather than forcing it.
LazyValueType lazyGetAttrField(Value & attrs, const std::vector<Symbol> & selector, const Pos & pos, Value & dest);
// Similar to `getAttrField`, but throws an `Error` if the field cant be
// found
void getAttrFieldThrow(Value & attrs, const std::vector<Symbol> & selector, const Pos & pos, Value & dest);
std::vector<Symbol> getFields(Value & attrs, const Pos & pos);
};
@@ -349,10 +380,6 @@ private:
string showType(ValueType type);
string showType(const Value & v);
/* Decode a context string !<name>!<path> into a pair <path,
name>. */
std::pair<string, string> decodeContext(std::string_view s);
/* If `path' refers to a directory, then append "/default.nix". */
Path resolveExprPath(Path path);

View File

@@ -596,30 +596,44 @@ void callFlake(EvalState & state,
const LockedFlake & lockedFlake,
Value & vRes)
{
auto vLocks = state.allocValue();
auto lockExpr = new ExprString(
state.symbols.create(lockedFlake.lockFile.to_string())
);
auto subdirExpr = new ExprString(
state.symbols.create(lockedFlake.flake.lockedRef.subdir)
);
auto vRootSrc = state.allocValue();
auto vRootSubdir = state.allocValue();
auto vTmp1 = state.allocValue();
auto vTmp2 = state.allocValue();
mkString(*vLocks, lockedFlake.lockFile.to_string());
emitTreeAttrs(state, *lockedFlake.flake.sourceInfo, lockedFlake.flake.lockedRef.input, *vRootSrc);
mkString(*vRootSubdir, lockedFlake.flake.lockedRef.subdir);
static RootValue vCallFlake = nullptr;
if (!vCallFlake) {
vCallFlake = allocRootValue(state.allocValue());
state.eval(state.parseExprFromString(
static Expr * callFlakeExpr = nullptr;
if (!callFlakeExpr) {
callFlakeExpr = state.parseExprFromString(
#include "call-flake.nix.gen.hh"
, "/"), **vCallFlake);
, "/");
}
state.callFunction(**vCallFlake, *vLocks, *vTmp1, noPos);
state.callFunction(*vTmp1, *vRootSrc, *vTmp2, noPos);
state.callFunction(*vTmp2, *vRootSubdir, vRes, noPos);
auto resExpr = new ExprApp(
new ExprApp(
new ExprApp(
callFlakeExpr,
lockExpr
),
new ExprCastedVar(vRootSrc)
),
subdirExpr
);
auto thunk = (Thunk*)allocBytes(sizeof(Thunk));
thunk->expr = resExpr;
thunk->env = &state.baseEnv;
auto fingerprint = lockedFlake.getFingerprint();
auto treeCache = state.openTreeCache(fingerprint);
auto cacheRoot = treeCache ? treeCache->getRoot() : nullptr;
auto evalCache = new ValueCache(cacheRoot);
vRes.mkCachedThunk(
thunk,
evalCache
);
}
static void prim_getFlake(EvalState & state, const Pos & pos, Value * * args, Value & v)

View File

@@ -16,7 +16,7 @@ libexpr_CXXFLAGS += -I src/libutil -I src/libstore -I src/libfetchers -I src/lib
libexpr_LIBS = libutil libstore libfetchers
libexpr_LDFLAGS = -lboost_context
ifneq ($(OS), FreeBSD)
ifeq ($(OS), Linux)
libexpr_LDFLAGS += -ldl
endif

View File

@@ -440,6 +440,18 @@ string ExprLambda::showNamePos() const
return (format("%1% at %2%") % (name.set() ? "'" + (string) name + "'" : "anonymous function") % pos).str();
}
void ExprCastedVar::show(std::ostream & str) const {
std::set<const Value*> active;
printValue(str, active, *v);
}
void ExprCastedVar::bindVars(const StaticEnv & env) {}
void ExprCastedVar::eval(EvalState & state, Env & env, Value & v) {
v = std::move(*this->v);
}
Value * ExprCastedVar::maybeThunk(EvalState & state, Env & env) {
return v;
}
/* Symbol table. */

View File

@@ -165,7 +165,12 @@ struct ExprSelect : Expr
Expr * e, * def;
AttrPath attrPath;
ExprSelect(const Pos & pos, Expr * e, const AttrPath & attrPath, Expr * def) : pos(pos), e(e), def(def), attrPath(attrPath) { };
ExprSelect(const Pos & pos, Expr * e, const Symbol & name) : pos(pos), e(e), def(0) { attrPath.push_back(AttrName(name)); };
ExprSelect(const Pos & pos, Expr * e, const Symbol & name)
: ExprSelect(pos, e, std::vector{name}) {};
ExprSelect(const Pos & pos, Expr * e, const std::vector<Symbol> & symbolicAttrPath) : pos(pos), e(e), def(0) {
for (auto & name : symbolicAttrPath)
attrPath.push_back(AttrName(name));
};
COMMON_METHODS
};
@@ -334,6 +339,14 @@ struct ExprPos : Expr
COMMON_METHODS
};
struct ExprCastedVar : Expr
{
Value * v;
ExprCastedVar(Value * v) : v(v) {};
Value * maybeThunk(EvalState & state, Env & env);
COMMON_METHODS
};
/* Static environments are used to map variable names onto (level,
displacement) pairs used to obtain the value of the variable at

392
src/libexpr/tree-cache.cc Normal file
View File

@@ -0,0 +1,392 @@
#include "tree-cache.hh"
#include "sqlite.hh"
#include "store-api.hh"
#include "context.hh"
namespace nix::tree_cache {
static const char * schema = R"sql(
create table if not exists Attributes (
id integer primary key autoincrement not null,
parent integer not null,
name text,
type integer not null,
value text,
context text,
unique (parent, name)
);
create index if not exists IndexByParent on Attributes(parent, name);
)sql";
struct AttrDb
{
std::atomic_bool failed{false};
struct State
{
SQLite db;
SQLiteStmt insertAttribute;
SQLiteStmt updateAttribute;
SQLiteStmt insertAttributeWithContext;
SQLiteStmt queryAttribute;
SQLiteStmt queryAttributes;
std::unique_ptr<SQLiteTxn> txn;
};
std::unique_ptr<Sync<State>> _state;
AttrDb(const Hash & fingerprint)
: _state(std::make_unique<Sync<State>>())
{
auto state(_state->lock());
Path cacheDir = getCacheDir() + "/nix/eval-cache-v3";
createDirs(cacheDir);
Path dbPath = cacheDir + "/" + fingerprint.to_string(Base16, false) + ".sqlite";
state->db = SQLite(dbPath);
state->db.isCache();
state->db.exec(schema);
state->insertAttribute.create(state->db,
"insert into Attributes(parent, name, type, value) values (?, ?, ?, ?)");
state->updateAttribute.create(state->db,
"update Attributes set type = ?, value = ?, context = ? where id = ?");
state->insertAttributeWithContext.create(state->db,
"insert into Attributes(parent, name, type, value, context) values (?, ?, ?, ?, ?)");
state->queryAttribute.create(state->db,
"select id, type, value, context from Attributes where parent = ? and name = ?");
state->queryAttributes.create(state->db,
"select name, type from Attributes where parent = ?");
state->txn = std::make_unique<SQLiteTxn>(state->db);
}
~AttrDb()
{
try {
auto state(_state->lock());
if (!failed)
state->txn->commit();
state->txn.reset();
} catch (...) {
ignoreException();
}
}
template<typename F>
AttrId doSQLite(F && fun)
{
if (failed) return 0;
try {
return fun();
} catch (SQLiteError &) {
ignoreException();
failed = true;
return 0;
}
}
/**
* Store a leaf of the tree in the db
*/
AttrId addEntry(
const AttrKey & key,
const AttrValue & value)
{
return doSQLite([&]()
{
auto state(_state->lock());
auto rawValue = RawValue::fromVariant(value);
state->insertAttributeWithContext.use()
(key.first)
(key.second)
(rawValue.type)
(rawValue.value.value_or(""), rawValue.value.has_value())
(rawValue.serializeContext())
.exec();
AttrId rowId = state->db.getLastInsertedRowId();
assert(rowId);
return rowId;
});
}
std::optional<AttrId> getId(const AttrKey& key)
{
auto state(_state->lock());
auto queryAttribute(state->queryAttribute.use()(key.first)(key.second));
if (!queryAttribute.next()) return std::nullopt;
return (AttrType) queryAttribute.getInt(0);
}
AttrId setOrUpdate(const AttrKey& key, const AttrValue& value)
{
debug("cache: miss for the attribute %s", key.second);
if (auto existingId = getId(key)) {
setValue(*existingId, value);
return *existingId;
}
return addEntry(key, value);
}
void setValue(const AttrId & id, const AttrValue & value)
{
auto state(_state->lock());
auto rawValue = RawValue::fromVariant(value);
state->updateAttribute.use()
(rawValue.type)
(rawValue.value.value_or(""), rawValue.value.has_value())
(rawValue.serializeContext())
(id)
.exec();
}
std::optional<std::pair<AttrId, AttrValue>> getValue(AttrKey key)
{
auto state(_state->lock());
auto queryAttribute(state->queryAttribute.use()(key.first)(key.second));
if (!queryAttribute.next()) return {};
auto rowId = (AttrType) queryAttribute.getInt(0);
auto type = (AttrType) queryAttribute.getInt(1);
switch (type) {
case AttrType::Attrs: {
return {{rowId, attributeSet_t()}};
}
case AttrType::String: {
std::vector<std::pair<Path, std::string>> context;
if (!queryAttribute.isNull(3))
for (auto & s : tokenizeString<std::vector<std::string>>(queryAttribute.getStr(3), ";"))
context.push_back(decodeContext(s));
return {{rowId, string_t{queryAttribute.getStr(2), context}}};
}
case AttrType::Bool:
return {{rowId, wrapped_basetype<bool>{queryAttribute.getInt(2) != 0}}};
case AttrType::Int:
return {{rowId, wrapped_basetype<int64_t>{queryAttribute.getInt(2)}}};
case AttrType::Double:
return {{rowId, wrapped_basetype<double>{(double)queryAttribute.getInt(2)}}};
case AttrType::Unknown:
return {{rowId, unknown_t{}}};
case AttrType::Thunk:
return {{rowId, thunk_t{}}};
case AttrType::Missing:
return {{rowId, missing_t{key.second}}};
case AttrType::Failed:
return {{rowId, failed_t{queryAttribute.getStr(2)}}};
default:
throw Error("unexpected type in evaluation cache");
}
}
std::vector<std::string> getChildren(AttrId parentId)
{
std::vector<std::string> res;
auto state(_state->lock());
auto queryAttributes(state->queryAttributes.use()(parentId));
while (queryAttributes.next()) {
if (queryAttributes.getInt(1) != AttrType::Missing)
res.push_back(queryAttributes.getStr(0));
}
return res;
}
};
Cache::Cache(const Hash & useCache,
SymbolTable & symbols)
: db(std::make_shared<AttrDb>(useCache))
, symbols(symbols)
, rootSymbol(symbols.create("%root%"))
{
}
std::shared_ptr<Cache> Cache::tryCreate(const Hash & useCache, SymbolTable & symbols)
{
try {
return std::make_shared<Cache>(useCache, symbols);
} catch (SQLiteError &) {
ignoreException();
return nullptr;
}
}
void Cache::commit()
{
if (db) {
debug("Saving the cache");
auto state(db->_state->lock());
if (state->txn->active) {
state->txn->commit();
state->txn.reset();
state->txn = std::make_unique<SQLiteTxn>(state->db);
}
}
}
Cursor::Ref Cache::getRoot()
{
return new Cursor(ref(shared_from_this()), std::nullopt, thunk_t{});
}
Cursor::Cursor(
ref<Cache> root,
const Parent & parent,
const AttrValue& value
)
: root(root)
, parentId(parent ? std::optional{parent->first.cachedValue.first} : std::nullopt)
, label(parent ? parent->second : root->rootSymbol)
, cachedValue({root->db->setOrUpdate(getKey(), value), value})
{
}
Cursor::Cursor(
ref<Cache> root,
const Parent & parent,
const AttrId & id,
const AttrValue & value
)
: root(root)
, parentId(parent ? std::optional{parent->first.cachedValue.first} : std::nullopt)
, label(parent ? parent->second : root->rootSymbol)
, cachedValue({id, value})
{
}
AttrKey Cursor::getKey()
{
if (!parentId)
return {0, root->rootSymbol};
return {*parentId, label};
}
AttrValue Cursor::getCachedValue()
{
return cachedValue.second;
}
void Cursor::setValue(const AttrValue & v)
{
root->db->setValue(cachedValue.first, v);
cachedValue.second = v;
}
Cursor::Ref Cursor::addChild(const Symbol & attrPath, const AttrValue & v)
{
Parent parent = {{*this, attrPath}};
auto childCursor = new Cursor(
root,
parent,
v
);
return childCursor;
}
std::vector<std::string> Cursor::getChildren()
{
return root->db->getChildren(cachedValue.first);
}
std::optional<std::vector<std::string>> Cursor::getChildrenAtPath(const std::vector<Symbol> & attrPath)
{
auto cursorAtPath = findAlongAttrPath(attrPath);
if (cursorAtPath)
return cursorAtPath->getChildren();
return std::nullopt;
}
Cursor::Ref Cursor::maybeGetAttr(const Symbol & name)
{
auto rawAttr = root->db->getValue({cachedValue.first, name});
if (rawAttr) {
Parent parent = {{*this, name}};
debug("cache: hit for the attribute %s", cachedValue.first);
return new Cursor (
root, parent, rawAttr->first,
rawAttr->second);
}
if (std::holds_alternative<attributeSet_t>(cachedValue.second)) {
// If the parent is an attribute set but we're not present in the db,
// then we're not a member of this attribute set. So mark as missing
return addChild(name, missing_t{name});
}
return nullptr;
}
Cursor::Ref Cursor::findAlongAttrPath(const std::vector<Symbol> & attrPath)
{
auto currentCursor = this;
for (auto & currentAccessor : attrPath) {
currentCursor = currentCursor->maybeGetAttr(currentAccessor);
if (!currentCursor)
break;
if (std::holds_alternative<missing_t>(currentCursor->cachedValue.second))
break;
if (std::holds_alternative<failed_t>(currentCursor->cachedValue.second))
break;
}
return currentCursor;
}
const RawValue RawValue::fromVariant(const AttrValue & value)
{
RawValue res;
std::visit(overloaded{
[&](attributeSet_t x) { res.type = AttrType::Attrs; },
[&](string_t x) {
res.type = AttrType::String;
res.value = x.first;
res.context = x.second;
},
[&](wrapped_basetype<bool> x) {
res.type = AttrType::Bool;
res.value = x.value ? "1" : "0";
},
[&](wrapped_basetype<int64_t> x) {
res.type = AttrType::Int;
res.value = std::to_string(x.value);
},
[&](wrapped_basetype<double> x) {
res.type = AttrType::Double;
res.value = std::to_string(x.value);
},
[&](unknown_t x) { res.type = AttrType::Unknown; },
[&](missing_t x) { res.type = AttrType::Missing; },
[&](thunk_t x) { res.type = AttrType::Thunk; },
[&](failed_t x) {
res.type = AttrType::Failed;
res.value = x.error;
}
}, value);
return res;
}
std::string RawValue::serializeContext() const
{
std::string res;
for (auto & elt : context) {
res.append(encodeContext(elt.second, elt.first));
res.push_back(' ');
}
if (!res.empty())
res.pop_back(); // Remove the trailing space
return res;
}
}

156
src/libexpr/tree-cache.hh Normal file
View File

@@ -0,0 +1,156 @@
/**
* caching for a tree-like data structure (like Nix values)
*
* The cache is an sqlite db whose rows are the nodes of the tree, with a
* pointer to their parent (except for the root of course)
*/
#pragma once
#include "sync.hh"
#include "hash.hh"
#include "symbol-table.hh"
#include <functional>
#include <variant>
namespace nix::tree_cache {
struct AttrDb;
class Cursor;
class Cache : public std::enable_shared_from_this<Cache>
{
private:
friend class Cursor;
/**
* The database holding the cache
*/
std::shared_ptr<AttrDb> db;
SymbolTable & symbols;
/**
* Distinguished symbol indicating the root of the tree
*/
const Symbol rootSymbol;
public:
Cache(
const Hash & useCache,
SymbolTable & symbols
);
static std::shared_ptr<Cache> tryCreate(const Hash & useCache, SymbolTable & symbols);
Cursor * getRoot();
/**
* Flush the cache to disk
*/
void commit();
};
enum AttrType {
Unknown = 0,
Attrs = 1,
String = 2,
Bool = 3,
Int = 4,
Double = 5,
Thunk = 6,
Missing = 7, // Missing fields of attribute sets
Failed = 8,
};
struct attributeSet_t {};
struct unknown_t {};
struct thunk_t {};
struct failed_t { string error; };
struct missing_t { Symbol attrName; };
// Putting several different primitive types in an `std::variant` partially
// breaks the `std::visit(overloaded{...` hackery because of the implicit cast
// from one to another which breaks the exhaustiveness check.
// So we wrap them in a trivial class just to force the disambiguation
template<typename T>
struct wrapped_basetype{ T value; };
typedef uint64_t AttrId;
typedef std::pair<AttrId, Symbol> AttrKey;
typedef std::pair<std::string, std::vector<std::pair<Path, std::string>>> string_t;
typedef std::variant<
attributeSet_t,
string_t,
unknown_t,
thunk_t,
missing_t,
failed_t,
wrapped_basetype<bool>,
wrapped_basetype<int64_t>,
wrapped_basetype<double>
> AttrValue;
struct RawValue {
AttrType type;
std::optional<std::string> value;
std::vector<std::pair<Path, std::string>> context;
std::string serializeContext() const;
static const RawValue fromVariant(const AttrValue&);
AttrValue toVariant() const;
};
/**
* View inside the cache.
*
* A `Cursor` represents a node in the cached tree (be it a leaf or not)
*/
class Cursor : public std::enable_shared_from_this<Cursor>
{
/**
* The overall cache of which this cursor is a view
*/
ref<Cache> root;
typedef std::optional<std::pair<Cursor&, Symbol>> Parent;
std::optional<AttrId> parentId;
Symbol label;
std::pair<AttrId, AttrValue> cachedValue;
/**
* Get the identifier for this node in the database
*/
AttrKey getKey();
public:
using Ref = Cursor*;
// Create a new cache entry
Cursor(ref<Cache> root, const Parent & parent, const AttrValue&);
// Build a cursor from an existing cache entry
Cursor(ref<Cache> root, const Parent & parent, const AttrId& id, const AttrValue&);
AttrValue getCachedValue();
void setValue(const AttrValue & v);
Ref addChild(const Symbol & attrPath, const AttrValue & v);
Ref findAlongAttrPath(const std::vector<Symbol> & attrPath);
Ref maybeGetAttr(const Symbol & attrPath);
std::vector<std::string> getChildren();
std::optional<std::vector<std::string>> getChildrenAtPath(const std::vector<Symbol> & attrPath);
};
}

View File

@@ -0,0 +1,51 @@
#pragma once
#include "tree-cache.hh"
namespace nix {
struct Value;
class EvalState;
class Bindings;
class ValueCache {
tree_cache::Cursor::Ref rawCache;
public:
ValueCache(tree_cache::Cursor::Ref rawCache) : rawCache(rawCache) {}
static ValueCache empty;
bool isEmpty () { return rawCache == nullptr; }
enum ReturnCode {
// The cache result was an attribute set, so we forward it later in the
// chain
Forward,
CacheMiss,
CacheHit,
UnCacheable,
NoCacheKey,
};
struct CacheResult {
ReturnCode returnCode;
// In case the query returns a `missing_t`, the symbol that's missing
std::optional<Symbol> lastQueriedSymbolIfMissing;
};
std::pair<CacheResult, ValueCache> getValue(EvalState & state, const std::vector<Symbol> & selector, Value & dest);
ValueCache addChild(const Symbol & attrName, Value & value);
ValueCache addFailedChild(const Symbol & attrName, const Error & error);
ValueCache addNumChild(SymbolTable & symbols, int idx, const Value & value);
void addAttrSetChilds(Bindings & children);
void addListChilds(SymbolTable & symbols, Value** elems, int listSize);
std::optional<std::vector<Symbol>> listChildren(SymbolTable&);
std::optional<std::vector<Symbol>> listChildrenAtPath(SymbolTable&, const std::vector<Symbol> & attrPath);
std::optional<tree_cache::AttrValue> getRawValue();
ValueCache() : rawCache(nullptr) {}
};
}

View File

@@ -23,6 +23,7 @@ typedef enum {
tApp,
tLambda,
tBlackhole,
tCachedThunk,
tPrimOp,
tPrimOpApp,
tExternal,
@@ -56,6 +57,7 @@ struct Pos;
class EvalState;
class XMLWriter;
class JSONPlaceholder;
class ValueCache;
typedef int64_t NixInt;
@@ -103,6 +105,10 @@ class ExternalValueBase
std::ostream & operator << (std::ostream & str, const ExternalValueBase & v);
struct Thunk {
Env * env;
Expr * expr;
};
struct Value
{
@@ -122,6 +128,7 @@ public:
inline bool isThunk() const { return internalType == tThunk; };
inline bool isApp() const { return internalType == tApp; };
inline bool isBlackhole() const { return internalType == tBlackhole; };
inline bool isCachedThunk() const { return internalType == tCachedThunk; };
// type() == nFunction
inline bool isLambda() const { return internalType == tLambda; };
@@ -165,10 +172,13 @@ public:
Value * * elems;
} bigList;
Value * smallList[2];
Thunk thunk;
struct {
Env * env;
Expr * expr;
} thunk;
Thunk * thunk;
// This is a pointer only to prevent a recursive import as
// `EvalCache` is already a pointer so would fit very nicely here.
ValueCache * cache;
} cachedThunk;
struct {
Value * left, * right;
} app;
@@ -199,7 +209,7 @@ public:
case tLambda: case tPrimOp: case tPrimOpApp: return nFunction;
case tExternal: return nExternal;
case tFloat: return nFloat;
case tThunk: case tApp: case tBlackhole: return nThunk;
case tThunk: case tApp: case tBlackhole: case tCachedThunk: return nThunk;
}
abort();
}
@@ -272,6 +282,13 @@ public:
thunk.expr = ex;
}
inline void mkCachedThunk(Thunk * t, ValueCache * cache)
{
internalType = tCachedThunk;
cachedThunk.thunk = t;
cachedThunk.cache = cache;
}
inline void mkApp(Value * l, Value * r)
{
internalType = tApp;
@@ -349,6 +366,9 @@ public:
bool isTrivial() const;
std::vector<std::pair<Path, std::string>> getContext();
ValueCache & getEvalCache();
void setEvalCache(ValueCache &);
};

View File

@@ -6,6 +6,8 @@
#include <nlohmann/json_fwd.hpp>
#include <optional>
namespace nix::fetchers {
typedef std::variant<std::string, uint64_t, Explicit<bool>> Attr;

View File

@@ -178,7 +178,8 @@ struct TarballInputScheme : InputScheme
&& !hasSuffix(url.path, ".tar")
&& !hasSuffix(url.path, ".tar.gz")
&& !hasSuffix(url.path, ".tar.xz")
&& !hasSuffix(url.path, ".tar.bz2"))
&& !hasSuffix(url.path, ".tar.bz2")
&& !hasSuffix(url.path, ".tar.zst"))
return {};
Input input;

View File

@@ -2354,32 +2354,19 @@ void LocalDerivationGoal::registerOutputs()
}
auto got = caSink.finish().first;
auto refs = rewriteRefs();
HashModuloSink narSink { htSHA256, oldHashPart };
dumpPath(actualPath, narSink);
auto narHashAndSize = narSink.finish();
ValidPathInfo newInfo0 {
worker.store.makeFixedOutputPath(
auto finalPath = worker.store.makeFixedOutputPath(
outputHash.method,
got,
outputPathName(drv->name, outputName),
refs.second,
refs.first),
narHashAndSize.first,
};
newInfo0.narSize = narHashAndSize.second;
newInfo0.ca = FixedOutputHash {
.method = outputHash.method,
.hash = got,
};
newInfo0.references = refs.second;
if (refs.first)
newInfo0.references.insert(newInfo0.path);
if (scratchPath != newInfo0.path) {
refs.first);
if (scratchPath != finalPath) {
// Also rewrite the output path
auto source = sinkToSource([&](Sink & nextSink) {
StringSink sink;
dumpPath(actualPath, sink);
RewritingSink rsink2(oldHashPart, std::string(newInfo0.path.hashPart()), nextSink);
RewritingSink rsink2(oldHashPart, std::string(finalPath.hashPart()), nextSink);
rsink2(*sink.s);
rsink2.flush();
});
@@ -2389,6 +2376,21 @@ void LocalDerivationGoal::registerOutputs()
movePath(tmpPath, actualPath);
}
HashResult narHashAndSize = hashPath(htSHA256, actualPath);
ValidPathInfo newInfo0 {
finalPath,
narHashAndSize.first,
};
newInfo0.narSize = narHashAndSize.second;
newInfo0.ca = FixedOutputHash {
.method = outputHash.method,
.hash = got,
};
newInfo0.references = refs.second;
if (refs.first)
newInfo0.references.insert(newInfo0.path);
assert(newInfo0.ca);
return newInfo0;
};

View File

@@ -7,7 +7,7 @@
#include "finally.hh"
#include "callback.hh"
#ifdef ENABLE_S3
#if ENABLE_S3
#include <aws/core/client/ClientConfiguration.h>
#endif
@@ -665,7 +665,7 @@ struct curlFileTransfer : public FileTransfer
writeFull(wakeupPipe.writeSide.get(), " ");
}
#ifdef ENABLE_S3
#if ENABLE_S3
std::tuple<std::string, std::string, Store::Params> parseS3Uri(std::string uri)
{
auto [path, params] = splitUriAndParams(uri);
@@ -688,7 +688,7 @@ struct curlFileTransfer : public FileTransfer
if (hasPrefix(request.uri, "s3://")) {
// FIXME: do this on a worker thread
try {
#ifdef ENABLE_S3
#if ENABLE_S3
auto [bucketName, key, params] = parseS3Uri(request.uri);
std::string profile = get(params, "profile").value_or("");

View File

@@ -1152,17 +1152,13 @@ void LocalStore::addToStore(const ValidPathInfo & info, Source & source,
/* While restoring the path from the NAR, compute the hash
of the NAR. */
std::unique_ptr<AbstractHashSink> hashSink;
if (!info.ca.has_value() || !info.references.count(info.path))
hashSink = std::make_unique<HashSink>(htSHA256);
else
hashSink = std::make_unique<HashModuloSink>(htSHA256, std::string(info.path.hashPart()));
HashSink hashSink(htSHA256);
TeeSource wrapperSource { source, *hashSink };
TeeSource wrapperSource { source, hashSink };
restorePath(realPath, wrapperSource);
auto hashResult = hashSink->finish();
auto hashResult = hashSink.finish();
if (hashResult.first != info.narHash)
throw Error("hash mismatch importing path '%s';\n specified: %s\n got: %s",
@@ -1172,6 +1168,31 @@ void LocalStore::addToStore(const ValidPathInfo & info, Source & source,
throw Error("size mismatch importing path '%s';\n specified: %s\n got: %s",
printStorePath(info.path), info.narSize, hashResult.second);
if (info.ca) {
if (auto foHash = std::get_if<FixedOutputHash>(&*info.ca)) {
auto actualFoHash = hashCAPath(
foHash->method,
foHash->hash.type,
info.path
);
if (foHash->hash != actualFoHash.hash) {
throw Error("ca hash mismatch importing path '%s';\n specified: %s\n got: %s",
printStorePath(info.path),
foHash->hash.to_string(Base32, true),
actualFoHash.hash.to_string(Base32, true));
}
}
if (auto textHash = std::get_if<TextHash>(&*info.ca)) {
auto actualTextHash = hashString(htSHA256, readFile(realPath));
if (textHash->hash != actualTextHash) {
throw Error("ca hash mismatch importing path '%s';\n specified: %s\n got: %s",
printStorePath(info.path),
textHash->hash.to_string(Base32, true),
actualTextHash.to_string(Base32, true));
}
}
}
autoGC();
canonicalisePathMetaData(realPath, -1);
@@ -1440,14 +1461,10 @@ bool LocalStore::verifyStore(bool checkContents, RepairFlag repair)
/* Check the content hash (optionally - slow). */
printMsg(lvlTalkative, "checking contents of '%s'", printStorePath(i));
std::unique_ptr<AbstractHashSink> hashSink;
if (!info->ca || !info->references.count(info->path))
hashSink = std::make_unique<HashSink>(info->narHash.type);
else
hashSink = std::make_unique<HashModuloSink>(info->narHash.type, std::string(info->path.hashPart()));
auto hashSink = HashSink(info->narHash.type);
dumpPath(Store::toRealPath(i), *hashSink);
auto current = hashSink->finish();
dumpPath(Store::toRealPath(i), hashSink);
auto current = hashSink.finish();
if (info->narHash != nullHash && info->narHash != current.first) {
printError("path '%s' was modified! expected hash '%s', got '%s'",
@@ -1680,4 +1697,36 @@ std::optional<const Realisation> LocalStore::queryRealisation(
.id = id, .outPath = outputPath, .signatures = signatures}};
});
}
FixedOutputHash LocalStore::hashCAPath(
const FileIngestionMethod & method, const HashType & hashType,
const StorePath & path)
{
return hashCAPath(method, hashType, Store::toRealPath(path), path.hashPart());
}
FixedOutputHash LocalStore::hashCAPath(
const FileIngestionMethod & method,
const HashType & hashType,
const Path & path,
const std::string_view pathHash
)
{
HashModuloSink caSink ( hashType, std::string(pathHash) );
switch (method) {
case FileIngestionMethod::Recursive:
dumpPath(path, caSink);
break;
case FileIngestionMethod::Flat:
readFile(path, caSink);
break;
}
auto hash = caSink.finish().first;
return FixedOutputHash{
.method = method,
.hash = hash,
};
}
} // namespace nix

View File

@@ -283,6 +283,19 @@ private:
void createUser(const std::string & userName, uid_t userId) override;
// XXX: Make a generic `Store` method
FixedOutputHash hashCAPath(
const FileIngestionMethod & method,
const HashType & hashType,
const StorePath & path);
FixedOutputHash hashCAPath(
const FileIngestionMethod & method,
const HashType & hashType,
const Path & path,
const std::string_view pathHash
);
friend struct LocalDerivationGoal;
friend struct PathSubstitutionGoal;
friend struct SubstitutionGoal;

View File

@@ -9,7 +9,7 @@ libstore_SOURCES := $(wildcard $(d)/*.cc $(d)/builtins/*.cc $(d)/build/*.cc)
libstore_LIBS = libutil
libstore_LDFLAGS = $(SQLITE3_LIBS) -lbz2 $(LIBCURL_LIBS) $(SODIUM_LIBS) -pthread
ifneq ($(OS), FreeBSD)
ifeq ($(OS), Linux)
libstore_LDFLAGS += -ldl
endif

View File

@@ -6,98 +6,73 @@
#include "thread-pool.hh"
#include "topo-sort.hh"
#include "callback.hh"
#include "closure.hh"
namespace nix {
void Store::computeFSClosure(const StorePathSet & startPaths,
StorePathSet & paths_, bool flipDirection, bool includeOutputs, bool includeDerivers)
{
struct State
{
size_t pending;
StorePathSet & paths;
std::exception_ptr exc;
};
std::function<std::set<StorePath>(const StorePath & path, std::future<ref<const ValidPathInfo>> &)> queryDeps;
if (flipDirection)
queryDeps = [&](const StorePath& path,
std::future<ref<const ValidPathInfo>> & fut) {
StorePathSet res;
StorePathSet referrers;
queryReferrers(path, referrers);
for (auto& ref : referrers)
if (ref != path)
res.insert(ref);
Sync<State> state_(State{0, paths_, 0});
if (includeOutputs)
for (auto& i : queryValidDerivers(path))
res.insert(i);
std::function<void(const StorePath &)> enqueue;
if (includeDerivers && path.isDerivation())
for (auto& i : queryDerivationOutputs(path))
if (isValidPath(i) && queryPathInfo(i)->deriver == path)
res.insert(i);
return res;
};
else
queryDeps = [&](const StorePath& path,
std::future<ref<const ValidPathInfo>> & fut) {
StorePathSet res;
auto info = fut.get();
for (auto& ref : info->references)
if (ref != path)
res.insert(ref);
std::condition_variable done;
if (includeOutputs && path.isDerivation())
for (auto& i : queryDerivationOutputs(path))
if (isValidPath(i))
res.insert(i);
enqueue = [&](const StorePath & path) -> void {
{
auto state(state_.lock());
if (state->exc) return;
if (!state->paths.insert(path).second) return;
state->pending++;
}
if (includeDerivers && info->deriver && isValidPath(*info->deriver))
res.insert(*info->deriver);
return res;
};
queryPathInfo(path, {[&](std::future<ref<const ValidPathInfo>> fut) {
// FIXME: calls to isValidPath() should be async
try {
auto info = fut.get();
if (flipDirection) {
StorePathSet referrers;
queryReferrers(path, referrers);
for (auto & ref : referrers)
if (ref != path)
enqueue(ref);
if (includeOutputs)
for (auto & i : queryValidDerivers(path))
enqueue(i);
if (includeDerivers && path.isDerivation())
for (auto & i : queryDerivationOutputs(path))
if (isValidPath(i) && queryPathInfo(i)->deriver == path)
enqueue(i);
} else {
for (auto & ref : info->references)
if (ref != path)
enqueue(ref);
if (includeOutputs && path.isDerivation())
for (auto & i : queryDerivationOutputs(path))
if (isValidPath(i)) enqueue(i);
if (includeDerivers && info->deriver && isValidPath(*info->deriver))
enqueue(*info->deriver);
}
{
auto state(state_.lock());
assert(state->pending);
if (!--state->pending) done.notify_one();
}
} catch (...) {
auto state(state_.lock());
if (!state->exc) state->exc = std::current_exception();
assert(state->pending);
if (!--state->pending) done.notify_one();
};
}});
};
for (auto & startPath : startPaths)
enqueue(startPath);
{
auto state(state_.lock());
while (state->pending) state.wait(done);
if (state->exc) std::rethrow_exception(state->exc);
}
computeClosure<StorePath>(
startPaths, paths_,
[&](const StorePath& path,
std::function<void(std::promise<std::set<StorePath>>&)>
processEdges) {
std::promise<std::set<StorePath>> promise;
std::function<void(std::future<ref<const ValidPathInfo>>)>
getDependencies =
[&](std::future<ref<const ValidPathInfo>> fut) {
try {
promise.set_value(queryDeps(path, fut));
} catch (...) {
promise.set_exception(std::current_exception());
}
};
queryPathInfo(path, getDependencies);
processEdges(promise);
});
}
void Store::computeFSClosure(const StorePath & startPath,
StorePathSet & paths_, bool flipDirection, bool includeOutputs, bool includeDerivers)
{

69
src/libutil/closure.hh Normal file
View File

@@ -0,0 +1,69 @@
#include <set>
#include <future>
#include "sync.hh"
using std::set;
namespace nix {
template<typename T>
using GetEdgesAsync = std::function<void(const T &, std::function<void(std::promise<set<T>> &)>)>;
template<typename T>
void computeClosure(
const set<T> startElts,
set<T> & res,
GetEdgesAsync<T> getEdgesAsync
)
{
struct State
{
size_t pending;
set<T> & res;
std::exception_ptr exc;
};
Sync<State> state_(State{0, res, 0});
std::function<void(const T &)> enqueue;
std::condition_variable done;
enqueue = [&](const T & current) -> void {
{
auto state(state_.lock());
if (state->exc) return;
if (!state->res.insert(current).second) return;
state->pending++;
}
getEdgesAsync(current, [&](std::promise<set<T>> & prom) {
try {
auto children = prom.get_future().get();
for (auto & child : children)
enqueue(child);
{
auto state(state_.lock());
assert(state->pending);
if (!--state->pending) done.notify_one();
}
} catch (...) {
auto state(state_.lock());
if (!state->exc) state->exc = std::current_exception();
assert(state->pending);
if (!--state->pending) done.notify_one();
};
});
};
for (auto & startElt : startElts)
enqueue(startElt);
{
auto state(state_.lock());
while (state->pending) state.wait(done);
if (state->exc) std::rethrow_exception(state->exc);
}
}
}

View File

@@ -0,0 +1,70 @@
#include "closure.hh"
#include <gtest/gtest.h>
namespace nix {
using namespace std;
map<string, set<string>> testGraph = {
{ "A", { "B", "C", "G" } },
{ "B", { "A" } }, // Loops back to A
{ "C", { "F" } }, // Indirect reference
{ "D", { "A" } }, // Not reachable, but has backreferences
{ "E", {} }, // Just not reachable
{ "F", {} },
{ "G", { "G" } }, // Self reference
};
TEST(closure, correctClosure) {
set<string> aClosure;
set<string> expectedClosure = {"A", "B", "C", "F", "G"};
computeClosure<string>(
{"A"},
aClosure,
[&](const string currentNode, function<void(promise<set<string>> &)> processEdges) {
promise<set<string>> promisedNodes;
promisedNodes.set_value(testGraph[currentNode]);
processEdges(promisedNodes);
}
);
ASSERT_EQ(aClosure, expectedClosure);
}
TEST(closure, properlyHandlesDirectExceptions) {
struct TestExn {};
set<string> aClosure;
EXPECT_THROW(
computeClosure<string>(
{"A"},
aClosure,
[&](const string currentNode, function<void(promise<set<string>> &)> processEdges) {
throw TestExn();
}
),
TestExn
);
}
TEST(closure, properlyHandlesExceptionsInPromise) {
struct TestExn {};
set<string> aClosure;
EXPECT_THROW(
computeClosure<string>(
{"A"},
aClosure,
[&](const string currentNode, function<void(promise<set<string>> &)> processEdges) {
promise<set<string>> promise;
try {
throw TestExn();
} catch (...) {
promise.set_exception(std::current_exception());
}
processEdges(promise);
}
),
TestExn
);
}
}

View File

@@ -22,9 +22,13 @@ This command verifies that the flake specified by flake reference
that the derivations specified by the flake's `checks` output can be
built successfully.
If the `keep-going` option is set to `true`, Nix will keep evaluating as much
as it can and report the errors as it encounters them. Otherwise it will stop
at the first error.
# Evaluation checks
This following flake output attributes must be derivations:
The following flake output attributes must be derivations:
* `checks.`*system*`.`*name*
* `defaultPackage.`*system*`

View File

@@ -272,25 +272,40 @@ struct CmdFlakeCheck : FlakeCommand
auto state = getEvalState();
auto flake = lockFlake();
bool hasErrors = false;
auto reportError = [&](const Error & e) {
try {
throw e;
} catch (Error & e) {
if (settings.keepGoing) {
ignoreException();
hasErrors = true;
}
else
throw;
}
};
// FIXME: rewrite to use EvalCache.
auto checkSystemName = [&](const std::string & system, const Pos & pos) {
// FIXME: what's the format of "system"?
if (system.find('-') == std::string::npos)
throw Error("'%s' is not a valid system type, at %s", system, pos);
reportError(Error("'%s' is not a valid system type, at %s", system, pos));
};
auto checkDerivation = [&](const std::string & attrPath, Value & v, const Pos & pos) {
auto checkDerivation = [&](const std::string & attrPath, Value & v, const Pos & pos) -> std::optional<StorePath> {
try {
auto drvInfo = getDerivation(*state, v, false);
if (!drvInfo)
throw Error("flake attribute '%s' is not a derivation", attrPath);
// FIXME: check meta attributes
return store->parseStorePath(drvInfo->queryDrvPath());
return std::make_optional(store->parseStorePath(drvInfo->queryDrvPath()));
} catch (Error & e) {
e.addTrace(pos, hintfmt("while checking the derivation '%s'", attrPath));
throw;
reportError(e);
}
return std::nullopt;
};
std::vector<DerivedPath> drvPaths;
@@ -307,7 +322,7 @@ struct CmdFlakeCheck : FlakeCommand
#endif
} catch (Error & e) {
e.addTrace(pos, hintfmt("while checking the app definition '%s'", attrPath));
throw;
reportError(e);
}
};
@@ -323,7 +338,7 @@ struct CmdFlakeCheck : FlakeCommand
// evaluate the overlay.
} catch (Error & e) {
e.addTrace(pos, hintfmt("while checking the overlay '%s'", attrPath));
throw;
reportError(e);
}
};
@@ -347,7 +362,7 @@ struct CmdFlakeCheck : FlakeCommand
// check the module.
} catch (Error & e) {
e.addTrace(pos, hintfmt("while checking the NixOS module '%s'", attrPath));
throw;
reportError(e);
}
};
@@ -369,7 +384,7 @@ struct CmdFlakeCheck : FlakeCommand
} catch (Error & e) {
e.addTrace(pos, hintfmt("while checking the Hydra jobset '%s'", attrPath));
throw;
reportError(e);
}
};
@@ -384,7 +399,7 @@ struct CmdFlakeCheck : FlakeCommand
throw Error("attribute 'config.system.build.toplevel' is not a derivation");
} catch (Error & e) {
e.addTrace(pos, hintfmt("while checking the NixOS configuration '%s'", attrPath));
throw;
reportError(e);
}
};
@@ -418,7 +433,7 @@ struct CmdFlakeCheck : FlakeCommand
}
} catch (Error & e) {
e.addTrace(pos, hintfmt("while checking the template '%s'", attrPath));
throw;
reportError(e);
}
};
@@ -433,7 +448,7 @@ struct CmdFlakeCheck : FlakeCommand
throw Error("bundler must take formal arguments 'program' and 'system'");
} catch (Error & e) {
e.addTrace(pos, hintfmt("while checking the template '%s'", attrPath));
throw;
reportError(e);
}
};
@@ -461,8 +476,8 @@ struct CmdFlakeCheck : FlakeCommand
auto drvPath = checkDerivation(
fmt("%s.%s.%s", name, attr.name, attr2.name),
*attr2.value, *attr2.pos);
if ((std::string) attr.name == settings.thisSystem.get())
drvPaths.push_back(DerivedPath::Built{drvPath});
if (drvPath && (std::string) attr.name == settings.thisSystem.get())
drvPaths.push_back(DerivedPath::Built{*drvPath});
}
}
}
@@ -574,7 +589,7 @@ struct CmdFlakeCheck : FlakeCommand
} catch (Error & e) {
e.addTrace(pos, hintfmt("while checking flake output '%s'", name));
throw;
reportError(e);
}
});
}
@@ -583,6 +598,8 @@ struct CmdFlakeCheck : FlakeCommand
Activity act(*logger, lvlInfo, actUnknown, "running flake checks");
store->buildPaths(drvPaths);
}
if (hasErrors)
throw Error("Some errors were encountered during the evaluation");
}
};

View File

@@ -186,8 +186,8 @@ Currently the `type` attribute can be one of the following:
attribute `url`.
In URL form, the schema must be `http://`, `https://` or `file://`
URLs and the extension must be `.zip`, `.tar`, `.tar.gz`, `.tar.xz`
or `.tar.bz2`.
URLs and the extension must be `.zip`, `.tar`, `.tar.gz`, `.tar.xz`,
`.tar.bz2` or `.tar.zst`.
* `github`: A more efficient way to fetch repositories from
GitHub. The following attributes are required:

View File

@@ -18,7 +18,7 @@ R""(
* Upgrade a specific profile element by number:
```console
# nix profile info
# nix profile list
0 flake:nixpkgs#legacyPackages.x86_64-linux.spotify …
# nix profile upgrade 0

View File

@@ -9,6 +9,7 @@
#include "shared.hh"
#include "eval-cache.hh"
#include "attr-path.hh"
#include "value.hh"
#include <regex>
#include <fstream>
@@ -81,31 +82,46 @@ struct CmdSearch : InstallableCommand, MixJSON
uint64_t results = 0;
std::function<void(eval_cache::AttrCursor & cursor, const std::vector<Symbol> & attrPath, bool initialRecurse)> visit;
std::function<void(Value & current, const std::vector<Symbol> & attrPath, bool initialRecurse)> visit;
visit = [&](eval_cache::AttrCursor & cursor, const std::vector<Symbol> & attrPath, bool initialRecurse)
Value * vTmp = state->allocValue();
visit = [&](Value & current, const std::vector<Symbol> & attrPath, bool initialRecurse)
{
Activity act(*logger, lvlInfo, actUnknown,
fmt("evaluating '%s'", concatStringsSep(".", attrPath)));
try {
auto recurse = [&]()
{
for (const auto & attr : cursor.getAttrs()) {
auto cursor2 = cursor.getAttr(attr);
auto attrPath2(attrPath);
attrPath2.push_back(attr);
visit(*cursor2, attrPath2, false);
}
};
auto recurse = [&]()
{
for (auto & attrName : state->getFields(current, noPos)) {
try {
auto attrPath2(attrPath);
attrPath2.push_back(attrName);
auto attrValue = state->allocValue();
auto value_ = allocRootValue(attrValue);
auto childField = state->lazyGetAttrField(current, {attrName}, noPos,
*attrValue);
if (childField != EvalState::LazyValueType::DelayedUncacheable)
visit(*attrValue, attrPath2, false);
} catch (EvalError &e) {
if (!(attrPath.size() > 0 &&
attrPath[0] == "legacyPackages"))
throw;
}
}
};
if (cursor.isDerivation()) {
try {
auto maybeTypeField = state->lazyGetAttrField(current, {state->sType}, noPos, *vTmp);
if (maybeTypeField == EvalState::LazyValueType::PlainValue
&& vTmp->type() == nix::nString
&& strcmp(vTmp->string.s, "derivation") == 0) {
size_t found = 0;
DrvName name(cursor.getAttr("name")->getString());
state->getAttrFieldThrow(current, {state->sName}, noPos, *vTmp);
DrvName name(state->forceString(*vTmp));
auto aMeta = cursor.maybeGetAttr("meta");
auto aDescription = aMeta ? aMeta->maybeGetAttr("description") : nullptr;
auto description = aDescription ? aDescription->getString() : "";
auto hasDescription = state->getAttrField(current, {state->sMeta, state->sDescription}, noPos, *vTmp);
auto description = hasDescription ? state->forceString(*vTmp) : "";
std::replace(description.begin(), description.end(), '\n', ' ');
auto attrPath2 = concatStringsSep(".", attrPath);
@@ -118,8 +134,8 @@ struct CmdSearch : InstallableCommand, MixJSON
std::regex_search(name.name, nameMatch, regex);
std::regex_search(description, descriptionMatch, regex);
if (!attrPathMatch.empty()
|| !nameMatch.empty()
|| !descriptionMatch.empty())
|| !nameMatch.empty()
|| !descriptionMatch.empty())
found++;
}
@@ -134,39 +150,37 @@ struct CmdSearch : InstallableCommand, MixJSON
auto name2 = hilite(name.name, nameMatch, "\e[0;2m");
if (results > 1) logger->cout("");
logger->cout(
"* %s%s",
wrap("\e[0;1m", hilite(attrPath2, attrPathMatch, "\e[0;1m")),
name.version != "" ? " (" + name.version + ")" : "");
"* %s%s",
wrap("\e[0;1m", hilite(attrPath2, attrPathMatch, "\e[0;1m")),
name.version != "" ? " (" + name.version + ")" : "");
if (description != "")
logger->cout(
" %s", hilite(description, descriptionMatch, ANSI_NORMAL));
" %s", hilite(description, descriptionMatch, ANSI_NORMAL));
}
}
}
else if (
attrPath.size() == 0
|| (attrPath[0] == "legacyPackages" && attrPath.size() <= 2)
|| (attrPath[0] == "packages" && attrPath.size() <= 2))
} else if (
attrPath.size() == 0
|| (attrPath[0] == "legacyPackages" && attrPath.size() <= 2)
|| (attrPath[0] == "packages" && attrPath.size() <= 2))
recurse();
else if (initialRecurse)
recurse();
else if (attrPath[0] == "legacyPackages" && attrPath.size() > 2) {
auto attr = cursor.maybeGetAttr(state->sRecurseForDerivations);
if (attr && attr->getBool())
auto recurseFieldInfo = state->lazyGetAttrField(current, {state->sRecurseForDerivations}, noPos, *vTmp);
auto hasRecurse = recurseFieldInfo == EvalState::LazyValueType::PlainValue;
if (hasRecurse && state->forceBool(*vTmp, noPos))
recurse();
}
} catch (EvalError & e) {
if (!(attrPath.size() > 0 && attrPath[0] == "legacyPackages"))
throw;
}
};
for (auto & [cursor, prefix] : installable->getCursors(*state))
visit(*cursor, parseAttrPath(*state, prefix), true);
for (auto & [value, pos, prefix] : installable->toValues(*state))
visit(*value, parseAttrPath(*state, prefix), true);
if (!json && !results)
throw Error("no results for the given search term(s)!");

View File

@@ -97,15 +97,11 @@ struct CmdVerify : StorePathsCommand
if (!noContents) {
std::unique_ptr<AbstractHashSink> hashSink;
if (!info->ca)
hashSink = std::make_unique<HashSink>(info->narHash.type);
else
hashSink = std::make_unique<HashModuloSink>(info->narHash.type, std::string(info->path.hashPart()));
auto hashSink = HashSink(info->narHash.type);
store->narFromPath(info->path, *hashSink);
store->narFromPath(info->path, hashSink);
auto hash = hashSink->finish();
auto hash = hashSink.finish();
if (hash.first != info->narHash) {
corrupted++;

View File

@@ -535,6 +535,21 @@ EOF
(! nix flake check $flake3Dir)
cat > $flake3Dir/flake.nix <<EOF
{
outputs = { flake1, self }: {
defaultPackage = {
system-1 = "foo";
system-2 = "bar";
};
};
}
EOF
checkRes=$(nix flake check --keep-going $flake3Dir 2>&1 && fail "nix flake check should have failed" || true)
echo "$checkRes" | grep -q "defaultPackage.system-1"
echo "$checkRes" | grep -q "defaultPackage.system-2"
# Test 'follows' inputs.
cat > $flake3Dir/flake.nix <<EOF
{

View File

@@ -3,5 +3,6 @@
http://www2.mplayerhq.hu/MPlayer/releases/fonts/font-arial-iso-8859-1.tar.bz2
http://losser.st-lab.cs.uu.nl/~armijn/.nix/gcc-3.3.4-static-nix.tar.gz
http://fpdownload.macromedia.com/get/shockwave/flash/english/linux/7.0r25/install_flash_player_7_linux.tar.gz
https://ftp5.gwdg.de/pub/linux/archlinux/extra/os/x86_64/unzip-6.0-14-x86_64.pkg.tar.zst
ftp://ftp.gtk.org/pub/gtk/v1.2/gtk+-1.2.10.tar.gz
]

View File

@@ -11,6 +11,7 @@ nix_tests = \
timeout.sh secure-drv-outputs.sh nix-channel.sh \
multiple-outputs.sh import-derivation.sh fetchurl.sh optimise-store.sh \
binary-cache.sh \
substitute-with-invalid-ca.sh \
binary-cache-build-remote.sh \
nix-profile.sh repair.sh dump-db.sh case-hack.sh \
check-reqs.sh pass-as-file.sh tarball.sh restricted.sh \

View File

@@ -0,0 +1,38 @@
source common.sh
BINARY_CACHE=file://$cacheDir
getHash() {
basename "$1" | cut -d '-' -f 1
}
getRemoteNarInfo () {
echo "$cacheDir/$(getHash "$1").narinfo"
}
cat <<EOF > $TEST_HOME/good.txt
Im a good path
EOF
cat <<EOF > $TEST_HOME/bad.txt
Im a bad path
EOF
good=$(nix-store --add $TEST_HOME/good.txt)
bad=$(nix-store --add $TEST_HOME/bad.txt)
nix copy --to "$BINARY_CACHE" "$good"
nix copy --to "$BINARY_CACHE" "$bad"
nix-collect-garbage >/dev/null 2>&1
# Falsifying the narinfo file for '$good'
goodPathNarInfo=$(getRemoteNarInfo "$good")
badPathNarInfo=$(getRemoteNarInfo "$bad")
for fieldName in URL FileHash FileSize NarHash NarSize; do
sed -i "/^$fieldName/d" "$goodPathNarInfo"
grep -E "^$fieldName" "$badPathNarInfo" >> "$goodPathNarInfo"
done
# Copying back '$good' from the binary cache. This should fail as it is
# corrupted
if nix copy --from "$BINARY_CACHE" "$good"; then
fail "Importing a path with a wrong CA field should fail"
fi

12
thingToBench.nix Normal file
View File

@@ -0,0 +1,12 @@
let
# pkgs = (builtins.getFlake "github:NixOS/nixpkgs?rev=ad0d20345219790533ebe06571f82ed6b034db31").legacyPackages.x86_64-linux;
pkgs = import <nixpkgs> {};
in
pkgs.runCommandNoCC "foo" {
buildInputs = [
pkgs.firefox
pkgs.pandoc
# pkgs.nixosTests.xfce
];
}
"echo bar > $out"