Compare commits

..
Author SHA1 Message Date
Domen Kožar 475fc109e7 Remove rsync usage in the installer
It's not commonly installed on systems like debian,
so avoid the bootstrapping problem by using cp and
chmod.
2021-08-18 16:50:36 +02:00
Eelco Dolstra 2cd1a5b8f3 Merge pull request #5134 from pszubiak/fix-pkgconfig-install-path
Install pkg-config files in the correct location
2021-08-16 11:56:49 +02:00
Eelco Dolstra 902b050511 Merge pull request #5135 from Mic92/ca-typo
fix typo in ca-error message
2021-08-16 11:55:48 +02:00
Jörg Thalheim 21d0334e49 fix typo in ca-error message 2021-08-14 00:18:45 +02:00
Piotr Szubiakowski 9f13cb31e8 Install pkg-config files in the correct location
Use `$(libdir)` while installing .pc files looks like a more generic
solution. For example, it will work for distributions like RHEL or
Fedora where .pc files are installed in `/usr/lib64/pkgconfig`.
2021-08-13 21:08:58 +00:00
Eelco Dolstra d581129ef9 Merge pull request #5130 from alyssais/sys_name
configure.ac: remove another uname check
2021-08-13 12:53:37 +02:00
Alyssa Ross cff8fd69b6 configure.ac: remove another uname check
uname checks are not cross-safe.

The normalization for Cygwin doesn't need any equivalent for host_os
because nothing actually checked whether sys_name was cygwin any more.
2021-08-13 08:19:43 +00:00
Eelco Dolstra 43856b0d6d Merge pull request #5124 from edolstra/lock-file-diff
Improve flake lock file diffs
2021-08-11 23:07:14 +02:00
Eelco Dolstra 3af1c28ebb flake.lock: Update
Flake lock file changes:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/f77036342e2b690c61c97202bf48f2ce13acc022' (2021-06-28)
  → 'github:NixOS/nixpkgs/f6551e1efa261568c82b76c3a582b2c2ceb1f53f' (2021-08-11)
2021-08-11 19:36:34 +02:00
Eelco Dolstra 6b8069b823 In flake lock file diffs, show the last-modified date of inputs if available
This is a bit more informative than just the hash.

Also, format the diffs a bit nicer.
2021-08-11 19:35:28 +02:00
Eelco Dolstra 467a6fcdc2 Merge pull request #5123 from Pamplemousse/clean
15f4d4f follow up
2021-08-11 12:15:51 +02:00
Pamplemousse 0da416f820 15f4d4f follow up
* libstore: `bz2` should not be linked
  * libutil: `zlib.h` should not be included

Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-10 16:04:53 -07:00
Domen Kožar a6ba313a0a Merge pull request #5115 from r-burns/darwin-host-os
Fix host OS detection for darwin-specific linker flag
2021-08-10 11:01:57 +02:00
Ryan Burns d86d43c34c Fix host OS detection for darwin-specific linker flag 2021-08-10 00:01:39 -07:00
Eelco Dolstra 8943e3176d Merge pull request #5111 from Pamplemousse/clean
Minor maintenance cleaning
2021-08-09 20:05:03 +02:00
Pamplemousse 2de7a1fe67 libexpr: Squash similar conditions
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-09 10:10:11 -07:00
Eelco Dolstra 27444d40cf Merge pull request #5101 from fzakaria/faridzakaria/fix-nix-shell
nix-shell --pure: Let it work for any derivation
2021-08-09 14:52:11 +02:00
Eelco Dolstra 4c5ce2a345 Merge pull request #5104 from andir/refscan-race
Fix potential race-condition in reference scanning code
2021-08-09 14:49:24 +02:00
Eelco Dolstra 873df3ab03 Merge pull request #5106 from andir/libstore-waiter-set
libstore: use set instead of list for waiter list
2021-08-09 14:46:49 +02:00
Eelco Dolstra 6c50dc1faf Merge pull request #5102 from serokell/balsoft/whitelist-flake-registry
Whitelist nixConfig.flake-registry
2021-08-09 14:45:12 +02:00
Andreas Rammhold a9cb1ca32c libstore: use set instead of list for waiter list
This replaces the O(n) search complexity in our insert code with a
lookup of O(log n). It also makes removing waitees easier as we can use
the extract method provided by the set class.
2021-08-08 14:05:38 +02:00
Andreas Rammhold b2d3976163 Fix potential race-condition in reference scanning code
Previously the code ensures that the isBase32 array would only be
initialised once in a single-threaded context. If two threads happen to
call the function before the initialisation was completed both of them
would have completed the initialization step. This allowed for a
race-condition where one thread might be done with the initialization
but the other thread sets all the fields to false again. For a brief
moment the base32 detection would then produce false-negatives.
2021-08-07 19:10:25 +02:00
Alexander Bantyev 54c580b71f Whitelist nixConfig.flake-registry
flake-registry should be safe to set to an aribtrary value, since it
is identical to just setting `inputs`.
2021-08-07 14:42:59 +03:00
Farid Zakaria fbf42c55ac nix-shell --pure: Let it work for any derivation
`nix-shell --pure` when applied to a non stdenv derivation doesn't seem
to clear the PATH. It expects the stdenv/setup file to do so.

This adds an explicit `unset PATH` by nix-build.cc (nix-shell) itself so
that it's not reliant on stdenv/setup anymore.

This does not break impure nix-shell since the PATH is persisted as the
variable `p` prior in the bash rcfile

fixes #5092
2021-08-06 15:30:49 -07:00
Eelco Dolstra e277c0c479 Merge pull request #5075 from andir/libutil-base64-init
libutil: initialize the base64 decode array only once
2021-08-06 17:41:11 +02:00
Eelco Dolstra d64f9671fc Merge pull request #5094 from Pamplemousse/simpler_doc
doc/manual: don't need to copy `highlight.js` manually
2021-08-05 15:13:07 +02:00
Pamplemousse de39cfb9f3 doc/manual: don't need to copy highlight.js manually
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-08-04 19:47:05 -07:00
Eelco Dolstra 47e96bb533 Merge pull request #5082 from manveru/fix-git-init-race
fix git init race condition
2021-08-02 16:13:53 +02:00
Michael Fellinger 5c99ec374c fix git init race condition 2021-08-02 13:47:59 +02:00
Eelco Dolstra 94ec9e4703 Merge pull request #5077 from ldesgoui/nix-shell-args
nix-shell -p: pass `--arg`s as nixpkgs parameters
2021-08-01 21:57:14 +02:00
ldesgoui 6eeb6f9c84 nix-shell -p: pass --arg values as nixpkgs params 2021-07-31 17:25:10 +02:00
Andreas Rammhold b9c9c25766 libutil: initialize the base64 decode array only once
Previously, despite having a boolean that tracked initialization, the
decode characters have been "calculated" every single time a base64
string was being decoded.

With this change we only initialize the decode array once in a
thread-safe manner.
2021-07-30 21:07:32 +02:00
Pamplemousse d64cb33e90 Remove unused struct in libstore
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-30 10:34:28 -07:00
Eelco Dolstra 2b67cb7b8c Merge pull request #5072 from NixOS/ca/queryRealisation-perl
Expose a perl method to query a derivation
2021-07-30 16:14:03 +02:00
regnat c15e121e32 Expose a perl method to query a derivation
Just doing a very stupid thing taking as argument a serialised drv
output and returning a serialised realisation.

This is needed for `nix-serve` to handle ca derivations
2021-07-30 11:55:14 +02:00
Eelco Dolstra 48e35585a6 upload-release.pl: Fix deprecated 'nix' calls 2021-07-28 22:58:25 +02:00
Eelco Dolstra c5ade241f0 Merge remote-tracking branch 'origin/test-daemon-everywhere' 2021-07-28 21:57:50 +02:00
Eelco Dolstra 7816ef6c51 Merge pull request #5042 from alyssais/pthread
Enable pthreads for new libraries
2021-07-28 12:31:01 +02:00
Eelco Dolstra 64a07d3d18 Merge pull request #5057 from edolstra/nix-develop-chroot
nix develop: Support chroot stores
2021-07-28 12:30:38 +02:00
Pamplemousse 686ee74f09 Remove unecessary stuff in .gitignore
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-07-27 16:53:45 -07:00
regnat c2c0dba792 Remove the right socket before starting the daemon
For some reason, an old socket occasionally stays here on OSX, causing
the subsequent tests to fail
2021-07-27 17:06:11 +02:00
regnat addacfce4a Allow running all the tests with the daemon
When `NIX_DAEMON_PACKAGE` is set, make all the tests use the Nix daemon.
That way we can test every piece of Nix functionality both with and
without the daemon.

Tests for which using the daemon isn’t possible or doesn’t make sens can
selectively be disabled with `needLocalStore`
2021-07-27 17:06:11 +02:00
Eelco Dolstra 706777a4a8 Merge pull request #5055 from NixOS/fix-manpages-install
Fix the generation of nix3 manpages
2021-07-27 14:53:16 +02:00
Eelco Dolstra ed5ad59dc1 nix develop: Support chroot stores
Fixes #5024.
2021-07-27 14:24:03 +02:00
Eelco Dolstra c000cec27f Merge pull request #5048 from tweag/flox-eval-store
--eval-store and faster closure copying
2021-07-27 12:20:32 +02:00
Eelco Dolstra 29e4913f79 Add --eval-store test 2021-07-27 11:17:56 +02:00
Eelco Dolstra 47002108d1 nix-instantiate: Fix --eval-store 2021-07-27 11:16:47 +02:00
Eelco Dolstra 72c5bac39d Revert no longer necessary change 2021-07-26 13:50:18 +02:00
Eelco Dolstra fe1f34fa60 Low-latency closure copy
This adds a new store operation 'addMultipleToStore' that reads a
number of NARs and ValidPathInfos from a Source, allowing any number
of store paths to be copied in a single call. This is much faster on
high-latency links when copying a lot of small files, like .drv
closures.

For example, on a connection with an 50 ms delay:

Before:

  $ nix copy --to 'unix:///tmp/proxy-socket?root=/tmp/dest-chroot' \
    /nix/store/90jjw94xiyg5drj70whm9yll6xjj0ca9-hello-2.10.drv \
    --derivation --no-check-sigs
  real    0m57.868s
  user    0m0.103s
  sys     0m0.056s

After:

  real    0m0.690s
  user    0m0.017s
  sys     0m0.011s
2021-07-26 13:31:09 +02:00
Alyssa Ross ae0c026fe9 Enable pthreads for new libraries
Otherwise the lack of pthread causes linking to fail for NetBSD.
2021-07-24 09:15:01 +00:00
Eelco Dolstra 9957315ce0 RemoteStore::buildPaths(): Handle evalStore 2021-07-22 22:50:48 +02:00
Eelco Dolstra a7b7fcfb16 Remove redundant RealisedPath::closure() call 2021-07-22 22:43:18 +02:00
Eelco Dolstra eb6db4fd38 buildPaths(): Add an evalStore argument
With this, we don't have to copy the entire .drv closure to the
destination store ahead of time (or at all). Instead, buildPaths()
reads .drv files from the eval store and copies inputSrcs to the
destination store if it needs to build a derivation.

Issue #5025.
2021-07-22 09:59:51 +02:00
Eelco Dolstra 668abd3e57 copyPaths: Pass store by reference 2021-07-22 09:59:51 +02:00
Eelco Dolstra 8d9f7048cd Use eval-store in more places
In particular, this now works:

  $ nix path-info --eval-store auto --store https://cache.nixos.org nixpkgs#hello

Previously this would fail as it would try to upload the hello .drv to
cache.nixos.org. Now the .drv is instantiated in the local store, and
then we check for the existence of the outputs in cache.nixos.org.
2021-07-22 09:59:51 +02:00
Eelco Dolstra 95e915a993 DummyStore: Remove redundant method 2021-07-22 09:59:51 +02:00
Eelco Dolstra 7321657746 nix-shell: Handle --eval-store correctly 2021-07-22 09:59:51 +02:00
Eelco Dolstra e9848beca7 nix-build: Copy drv closure between eval store and build store 2021-07-22 09:59:51 +02:00
Eelco Dolstra 2ff3035cf4 Support --eval-store in nix-instantiate and nix-build 2021-07-22 09:59:51 +02:00
Eelco Dolstra 3d9de41a5b Hacky fast closure copying mechanism 2021-07-22 09:59:51 +02:00
Eelco Dolstra bef40c2949 Add --eval-store option 2021-07-22 09:59:51 +02:00
94 changed files with 739 additions and 356 deletions

No files matched your search

-3
View File
@@ -57,9 +57,6 @@ perl/Makefile.config
/src/nix-prefetch-url/nix-prefetch-url
# /src/nix-daemon/
/src/nix-daemon/nix-daemon
/src/nix-collect-garbage/nix-collect-garbage
# /src/nix-channel/
+25 -26
View File
@@ -32,14 +32,6 @@ AC_ARG_WITH(system, AS_HELP_STRING([--with-system=SYSTEM],[Platform identifier (
system="$machine_name-`echo $host_os | "$SED" -e's/@<:@0-9.@:>@*$//g'`";;
esac])
sys_name=$(uname -s | tr 'A-Z ' 'a-z_')
case $sys_name in
cygwin*)
sys_name=cygwin
;;
esac
AC_MSG_RESULT($system)
AC_SUBST(system)
AC_DEFINE_UNQUOTED(SYSTEM, ["$system"], [platform identifier ('cpu-os')])
@@ -63,10 +55,12 @@ AC_SYS_LARGEFILE
# Solaris-specific stuff.
AC_STRUCT_DIRENT_D_TYPE
if test "$sys_name" = sunos; then
case "$host_os" in
solaris*)
# Solaris requires -lsocket -lnsl for network functions
LDFLAGS="-lsocket -lnsl $LDFLAGS"
fi
;;
esac
# Check for pubsetbuf.
@@ -210,21 +204,24 @@ AC_SUBST(HAVE_LIBCPUID, [$have_libcpuid])
# Look for libseccomp, required for Linux sandboxing.
if test "$sys_name" = linux; then
AC_ARG_ENABLE([seccomp-sandboxing],
AS_HELP_STRING([--disable-seccomp-sandboxing],[Don't build support for seccomp sandboxing (only recommended if your arch doesn't support libseccomp yet!)
]))
if test "x$enable_seccomp_sandboxing" != "xno"; then
PKG_CHECK_MODULES([LIBSECCOMP], [libseccomp],
[CXXFLAGS="$LIBSECCOMP_CFLAGS $CXXFLAGS"])
have_seccomp=1
AC_DEFINE([HAVE_SECCOMP], [1], [Whether seccomp is available and should be used for sandboxing.])
else
case "$host_os" in
linux*)
AC_ARG_ENABLE([seccomp-sandboxing],
AS_HELP_STRING([--disable-seccomp-sandboxing],[Don't build support for seccomp sandboxing (only recommended if your arch doesn't support libseccomp yet!)
]))
if test "x$enable_seccomp_sandboxing" != "xno"; then
PKG_CHECK_MODULES([LIBSECCOMP], [libseccomp],
[CXXFLAGS="$LIBSECCOMP_CFLAGS $CXXFLAGS"])
have_seccomp=1
AC_DEFINE([HAVE_SECCOMP], [1], [Whether seccomp is available and should be used for sandboxing.])
else
have_seccomp=
fi
;;
*)
have_seccomp=
fi
else
have_seccomp=
fi
;;
esac
AC_SUBST(HAVE_SECCOMP, [$have_seccomp])
@@ -274,9 +271,11 @@ AC_CHECK_FUNCS([strsignal posix_fallocate sysconf])
# This is needed if bzip2 is a static library, and the Nix libraries
# are dynamic.
if test "$(uname)" = "Darwin"; then
case "${host_os}" in
darwin*)
LDFLAGS="-all_load $LDFLAGS"
fi
;;
esac
AC_ARG_WITH(sandbox-shell, AS_HELP_STRING([--with-sandbox-shell=PATH],[path of a statically-linked shell to use as /bin/sh in sandboxes]),
-1
View File
@@ -96,6 +96,5 @@ doc/manual/generated/man1/nix3-manpages: $(d)/src/command-ref/new-cli
$(docdir)/manual/index.html: $(MANUAL_SRCS) $(d)/book.toml $(d)/custom.css $(d)/src/SUMMARY.md $(d)/src/command-ref/new-cli $(d)/src/command-ref/conf-file.md $(d)/src/expressions/builtins.md
$(trace-gen) RUST_LOG=warn mdbook build doc/manual -d $(docdir)/manual
@cp doc/manual/highlight.pack.js $(docdir)/manual/highlight.js
endif
File renamed without changes.
Generated
+3 -3
View File
@@ -19,11 +19,11 @@
},
"nixpkgs": {
"locked": {
"lastModified": 1624862269,
"narHash": "sha256-JFcsh2+7QtfKdJFoPibLFPLgIW6Ycnv8Bts9a7RYme0=",
"lastModified": 1628689438,
"narHash": "sha256-YMINW6YmubHZVdliGsAJpnnMYXRrvppv59LgwtnyYhs=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "f77036342e2b690c61c97202bf48f2ce13acc022",
"rev": "f6551e1efa261568c82b76c3a582b2c2ceb1f53f",
"type": "github"
},
"original": {
+2 -2
View File
@@ -83,12 +83,12 @@ sub downloadFile {
if (!-e $tmpFile) {
print STDERR "downloading $srcFile to $tmpFile...\n";
system("NIX_REMOTE=https://cache.nixos.org/ nix cat-store '$srcFile' > '$tmpFile'") == 0
system("NIX_REMOTE=https://cache.nixos.org/ nix store cat '$srcFile' > '$tmpFile'") == 0
or die "unable to fetch $srcFile\n";
}
my $sha256_expected = $buildInfo->{buildproducts}->{$productNr}->{sha256hash} or die;
my $sha256_actual = `nix hash-file --base16 --type sha256 '$tmpFile'`;
my $sha256_actual = `nix hash file --base16 --type sha256 '$tmpFile'`;
chomp $sha256_actual;
if ($sha256_expected ne $sha256_actual) {
print STDERR "file $tmpFile is corrupt, got $sha256_actual, expected $sha256_expected\n";
+1
View File
@@ -22,6 +22,7 @@ our @EXPORT = qw(
derivationFromPath
addTempRoot
getBinDir getStoreDir
queryRawRealisation
);
our $VERSION = '0.15';
+13
View File
@@ -15,6 +15,7 @@
#include "crypto.hh"
#include <sodium.h>
#include <nlohmann/json.hpp>
using namespace nix;
@@ -120,6 +121,18 @@ SV * queryPathInfo(char * path, int base32)
croak("%s", e.what());
}
SV * queryRawRealisation(char * outputId)
PPCODE:
try {
auto realisation = store()->queryRealisation(DrvOutput::parse(outputId));
if (realisation)
XPUSHs(sv_2mortal(newSVpv(realisation->toJSON().dump().c_str(), 0)));
else
XPUSHs(sv_2mortal(newSVpv("", 0)));
} catch (Error & e) {
croak("%s", e.what());
}
SV * queryPathFromHashPart(char * hashPart)
PPCODE:
+4 -1
View File
@@ -701,7 +701,10 @@ install_from_extracted_nix() {
cd "$EXTRACTED_NIX_PATH"
_sudo "to copy the basic Nix files to the new store at $NIX_ROOT/store" \
rsync -rlpt --chmod=-w ./store/* "$NIX_ROOT/store/"
cp -RLp ./store/* "$NIX_ROOT/store/"
_sudo "to make the new store non-writable at $NIX_ROOT/store" \
chmod -R ugo-w "$NIX_ROOT/store/"
if [ -d "$NIX_INSTALLED_NIX" ]; then
echo " Alright! We have our first nix at $NIX_INSTALLED_NIX"
+2 -2
View File
@@ -270,7 +270,7 @@ connected:
{
Activity act(*logger, lvlTalkative, actUnknown, fmt("copying dependencies to '%s'", storeUri));
copyPaths(store, ref<Store>(sshStore), store->parseStorePathSet(inputs), NoRepair, NoCheckSigs, substitute);
copyPaths(*store, *sshStore, store->parseStorePathSet(inputs), NoRepair, NoCheckSigs, substitute);
}
uploadLock = -1;
@@ -321,7 +321,7 @@ connected:
if (auto localStore = store.dynamic_pointer_cast<LocalStore>())
for (auto & path : missingPaths)
localStore->locksHeld.insert(store->printStorePath(path)); /* FIXME: ugly */
copyPaths(ref<Store>(sshStore), store, missingPaths, NoRepair, NoCheckSigs, NoSubstitute);
copyPaths(*sshStore, *store, missingPaths, NoRepair, NoCheckSigs, NoSubstitute);
}
// XXX: Should be done as part of `copyPaths`
for (auto & realisation : missingRealisations) {
+25 -1
View File
@@ -54,6 +54,30 @@ void StoreCommand::run()
run(getStore());
}
EvalCommand::EvalCommand()
{
}
EvalCommand::~EvalCommand()
{
if (evalState)
evalState->printStats();
}
ref<Store> EvalCommand::getEvalStore()
{
if (!evalStore)
evalStore = evalStoreUrl ? openStore(*evalStoreUrl) : getStore();
return ref<Store>(evalStore);
}
ref<EvalState> EvalCommand::getEvalState()
{
if (!evalState)
evalState = std::make_shared<EvalState>(searchPath, getEvalStore(), getStore());
return ref<EvalState>(evalState);
}
BuiltPathsCommand::BuiltPathsCommand(bool recursive)
: recursive(recursive)
{
@@ -91,7 +115,7 @@ void BuiltPathsCommand::run(ref<Store> store)
for (auto & p : store->queryAllValidPaths())
paths.push_back(BuiltPath::Opaque{p});
} else {
paths = toBuiltPaths(store, realiseMode, operateOn, installables);
paths = toBuiltPaths(getEvalStore(), store, realiseMode, operateOn, installables);
if (recursive) {
// XXX: This only computes the store path closure, ignoring
// intermediate realisations
+22 -8
View File
@@ -45,11 +45,18 @@ private:
struct EvalCommand : virtual StoreCommand, MixEvalArgs
{
ref<EvalState> getEvalState();
std::shared_ptr<EvalState> evalState;
EvalCommand();
~EvalCommand();
ref<Store> getEvalStore();
ref<EvalState> getEvalState();
private:
std::shared_ptr<Store> evalStore;
std::shared_ptr<EvalState> evalState;
};
struct MixFlakeOptions : virtual Args, EvalCommand
@@ -216,15 +223,21 @@ static RegisterCommand registerCommand2(std::vector<std::string> && name)
return RegisterCommand(std::move(name), [](){ return make_ref<T>(); });
}
BuiltPaths build(ref<Store> store, Realise mode,
BuiltPaths build(ref<Store> evalStore, ref<Store> store, Realise mode,
std::vector<std::shared_ptr<Installable>> installables, BuildMode bMode = bmNormal);
std::set<StorePath> toStorePaths(ref<Store> store,
Realise mode, OperateOn operateOn,
std::set<StorePath> toStorePaths(
ref<Store> evalStore,
ref<Store> store,
Realise mode,
OperateOn operateOn,
std::vector<std::shared_ptr<Installable>> installables);
StorePath toStorePath(ref<Store> store,
Realise mode, OperateOn operateOn,
StorePath toStorePath(
ref<Store> evalStore,
ref<Store> store,
Realise mode,
OperateOn operateOn,
std::shared_ptr<Installable> installable);
std::set<StorePath> toDerivations(ref<Store> store,
@@ -232,6 +245,7 @@ std::set<StorePath> toDerivations(ref<Store> store,
bool useDeriver = false);
BuiltPaths toBuiltPaths(
ref<Store> evalStore,
ref<Store> store,
Realise mode,
OperateOn operateOn,
+24 -31
View File
@@ -289,19 +289,6 @@ void completeFlakeRefWithFragment(
completeFlakeRef(evalState->store, prefix);
}
ref<EvalState> EvalCommand::getEvalState()
{
if (!evalState)
evalState = std::make_shared<EvalState>(searchPath, getStore());
return ref<EvalState>(evalState);
}
EvalCommand::~EvalCommand()
{
if (evalState)
evalState->printStats();
}
void completeFlakeRef(ref<Store> store, std::string_view prefix)
{
if (prefix == "")
@@ -391,6 +378,7 @@ DerivedPaths InstallableValue::toDerivedPaths()
DerivedPaths res;
std::map<StorePath, std::set<std::string>> drvsToOutputs;
RealisedPath::Set drvsToCopy;
// Group by derivation, helps with .all in particular
for (auto & drv : toDerivations()) {
@@ -398,6 +386,7 @@ DerivedPaths InstallableValue::toDerivedPaths()
if (outputName == "")
throw Error("derivation '%s' lacks an 'outputName' attribute", state->store->printStorePath(drv.drvPath));
drvsToOutputs[drv.drvPath].insert(outputName);
drvsToCopy.insert(drv.drvPath);
}
for (auto & i : drvsToOutputs)
@@ -712,10 +701,10 @@ std::shared_ptr<Installable> SourceExprCommand::parseInstallable(
return installables.front();
}
BuiltPaths getBuiltPaths(ref<Store> store, DerivedPaths hopefullyBuiltPaths)
BuiltPaths getBuiltPaths(ref<Store> evalStore, ref<Store> store, const DerivedPaths & hopefullyBuiltPaths)
{
BuiltPaths res;
for (auto& b : hopefullyBuiltPaths)
for (auto & b : hopefullyBuiltPaths)
std::visit(
overloaded{
[&](DerivedPath::Opaque bo) {
@@ -723,14 +712,13 @@ BuiltPaths getBuiltPaths(ref<Store> store, DerivedPaths hopefullyBuiltPaths)
},
[&](DerivedPath::Built bfd) {
OutputPathMap outputs;
auto drv = store->readDerivation(bfd.drvPath);
auto outputHashes = staticOutputHashes(*store, drv);
auto drv = evalStore->readDerivation(bfd.drvPath);
auto outputHashes = staticOutputHashes(*evalStore, drv); // FIXME: expensive
auto drvOutputs = drv.outputsAndOptPaths(*store);
for (auto& output : bfd.outputs) {
for (auto & output : bfd.outputs) {
if (!outputHashes.count(output))
throw Error(
"the derivation '%s' doesn't have an output "
"named '%s'",
"the derivation '%s' doesn't have an output named '%s'",
store->printStorePath(bfd.drvPath), output);
if (settings.isExperimentalFeatureEnabled(
"ca-derivations")) {
@@ -741,7 +729,7 @@ BuiltPaths getBuiltPaths(ref<Store> store, DerivedPaths hopefullyBuiltPaths)
if (!realisation)
throw Error(
"cannot operate on an output of unbuilt "
"content-addresed derivation '%s'",
"content-addressed derivation '%s'",
outputId.to_string());
outputs.insert_or_assign(
output, realisation->outPath);
@@ -762,7 +750,7 @@ BuiltPaths getBuiltPaths(ref<Store> store, DerivedPaths hopefullyBuiltPaths)
return res;
}
BuiltPaths build(ref<Store> store, Realise mode,
BuiltPaths build(ref<Store> evalStore, ref<Store> store, Realise mode,
std::vector<std::shared_ptr<Installable>> installables, BuildMode bMode)
{
if (mode == Realise::Nothing)
@@ -778,20 +766,21 @@ BuiltPaths build(ref<Store> store, Realise mode,
if (mode == Realise::Nothing)
printMissing(store, pathsToBuild, lvlError);
else if (mode == Realise::Outputs)
store->buildPaths(pathsToBuild, bMode);
store->buildPaths(pathsToBuild, bMode, evalStore);
return getBuiltPaths(store, pathsToBuild);
return getBuiltPaths(evalStore, store, pathsToBuild);
}
BuiltPaths toBuiltPaths(
ref<Store> evalStore,
ref<Store> store,
Realise mode,
OperateOn operateOn,
std::vector<std::shared_ptr<Installable>> installables)
{
if (operateOn == OperateOn::Output) {
return build(store, mode, installables);
} else {
if (operateOn == OperateOn::Output)
return build(evalStore, store, mode, installables);
else {
if (mode == Realise::Nothing)
settings.readOnlyMode = true;
@@ -802,23 +791,27 @@ BuiltPaths toBuiltPaths(
}
}
StorePathSet toStorePaths(ref<Store> store,
StorePathSet toStorePaths(
ref<Store> evalStore,
ref<Store> store,
Realise mode, OperateOn operateOn,
std::vector<std::shared_ptr<Installable>> installables)
{
StorePathSet outPaths;
for (auto & path : toBuiltPaths(store, mode, operateOn, installables)) {
for (auto & path : toBuiltPaths(evalStore, store, mode, operateOn, installables)) {
auto thisOutPaths = path.outPaths();
outPaths.insert(thisOutPaths.begin(), thisOutPaths.end());
}
return outPaths;
}
StorePath toStorePath(ref<Store> store,
StorePath toStorePath(
ref<Store> evalStore,
ref<Store> store,
Realise mode, OperateOn operateOn,
std::shared_ptr<Installable> installable)
{
auto paths = toStorePaths(store, mode, operateOn, {installable});
auto paths = toStorePaths(evalStore, store, mode, operateOn, {installable});
if (paths.size() != 1)
throw Error("argument '%s' should evaluate to one store path", installable->what());
+1 -1
View File
@@ -26,7 +26,7 @@ struct App
struct UnresolvedApp
{
App unresolved;
App resolve(ref<Store>);
App resolve(ref<Store> evalStore, ref<Store> store);
};
struct Installable
+2 -2
View File
@@ -8,8 +8,8 @@ libcmd_SOURCES := $(wildcard $(d)/*.cc)
libcmd_CXXFLAGS += -I src/libutil -I src/libstore -I src/libexpr -I src/libmain -I src/libfetchers
libcmd_LDFLAGS = -llowdown
libcmd_LDFLAGS = -llowdown -pthread
libcmd_LIBS = libstore libutil libexpr libmain libfetchers
$(eval $(call install-file-in, $(d)/nix-cmd.pc, $(prefix)/lib/pkgconfig, 0644))
$(eval $(call install-file-in, $(d)/nix-cmd.pc, $(libdir)/pkgconfig, 0644))
+8
View File
@@ -61,6 +61,14 @@ MixEvalArgs::MixEvalArgs()
fetchers::overrideRegistry(from.input, to.input, extraAttrs);
}}
});
addFlag({
.longName = "eval-store",
.description = "The Nix store to use for evaluations.",
.category = category,
.labels = {"store-url"},
.handler = {&evalStoreUrl},
});
}
Bindings * MixEvalArgs::getAutoArgs(EvalState & state)
+2 -1
View File
@@ -16,8 +16,9 @@ struct MixEvalArgs : virtual Args
Strings searchPath;
private:
std::optional<std::string> evalStoreUrl;
private:
std::map<std::string, std::string> autoArgs;
};
+5 -1
View File
@@ -378,7 +378,10 @@ static Strings parseNixPath(const string & s)
}
EvalState::EvalState(const Strings & _searchPath, ref<Store> store)
EvalState::EvalState(
const Strings & _searchPath,
ref<Store> store,
std::shared_ptr<Store> buildStore)
: sWith(symbols.create("<with>"))
, sOutPath(symbols.create("outPath"))
, sDrvPath(symbols.create("drvPath"))
@@ -411,6 +414,7 @@ EvalState::EvalState(const Strings & _searchPath, ref<Store> store)
, sEpsilon(symbols.create(""))
, repair(NoRepair)
, store(store)
, buildStore(buildStore ? buildStore : store)
, regexCache(makeRegexCache())
, baseEnv(allocEnv(128))
, staticBaseEnv(false, 0)
+8 -1
View File
@@ -94,8 +94,12 @@ public:
Value vEmptySet;
/* Store used to materialise .drv files. */
const ref<Store> store;
/* Store used to build stuff. */
const ref<Store> buildStore;
private:
SrcToStore srcToStore;
@@ -128,7 +132,10 @@ private:
public:
EvalState(const Strings & _searchPath, ref<Store> store);
EvalState(
const Strings & _searchPath,
ref<Store> store,
std::shared_ptr<Store> buildStore = nullptr);
~EvalState();
void addToSearchPath(const string & s);
+1 -1
View File
@@ -29,7 +29,7 @@ static void writeTrustedList(const TrustedList & trustedList)
void ConfigFile::apply()
{
std::set<std::string> whitelist{"bash-prompt", "bash-prompt-suffix"};
std::set<std::string> whitelist{"bash-prompt", "bash-prompt-suffix", "flake-registry"};
for (auto & [name, value] : settings) {
+2 -2
View File
@@ -567,8 +567,8 @@ LockedFlake lockFlake(
topRef.input.markChangedFile(
(topRef.subdir == "" ? "" : topRef.subdir + "/") + "flake.lock",
lockFlags.commitLockFile
? std::optional<std::string>(fmt("%s: %s\n\nFlake input changes:\n\n%s",
relPath, lockFileExists ? "Update" : "Add", diff))
? std::optional<std::string>(fmt("%s: %s\n\nFlake lock file changes:\n\n%s",
relPath, lockFileExists ? "Update" : "Add", filterANSIEscapes(diff, true)))
: std::nullopt);
/* Rewriting the lockfile changed the top-level
+17 -4
View File
@@ -2,6 +2,8 @@
#include "store-api.hh"
#include "url-parts.hh"
#include <iomanip>
#include <nlohmann/json.hpp>
namespace nix::flake {
@@ -268,10 +270,20 @@ std::map<InputPath, Node::Edge> LockFile::getAllInputs() const
return res;
}
static std::string describe(const FlakeRef & flakeRef)
{
auto s = fmt("'%s'", flakeRef.to_string());
if (auto lastModified = flakeRef.input.getLastModified())
s += fmt(" (%s)", std::put_time(std::gmtime(&*lastModified), "%Y-%m-%d"));
return s;
}
std::ostream & operator <<(std::ostream & stream, const Node::Edge & edge)
{
if (auto node = std::get_if<0>(&edge))
stream << "'" << (*node)->lockedRef << "'";
stream << describe((*node)->lockedRef);
else if (auto follows = std::get_if<1>(&edge))
stream << fmt("follows '%s'", printInputPath(*follows));
return stream;
@@ -299,14 +311,15 @@ std::string LockFile::diff(const LockFile & oldLocks, const LockFile & newLocks)
while (i != oldFlat.end() || j != newFlat.end()) {
if (j != newFlat.end() && (i == oldFlat.end() || i->first > j->first)) {
res += fmt("* Added '%s': %s\n", printInputPath(j->first), j->second);
res += fmt("• " ANSI_GREEN "Added input '%s':" ANSI_NORMAL "\n %s\n",
printInputPath(j->first), j->second);
++j;
} else if (i != oldFlat.end() && (j == newFlat.end() || i->first < j->first)) {
res += fmt("* Removed '%s'\n", printInputPath(i->first));
res += fmt("• " ANSI_RED "Removed input '%s'" ANSI_NORMAL "\n", printInputPath(i->first));
++i;
} else {
if (!equals(i->second, j->second)) {
res += fmt("* Updated '%s': %s -> %s\n",
res += fmt("• " ANSI_BOLD "Updated input '%s':" ANSI_NORMAL "\n %s\n → %s\n",
printInputPath(i->first),
i->second,
j->second);
+2 -2
View File
@@ -15,7 +15,7 @@ libexpr_CXXFLAGS += -I src/libutil -I src/libstore -I src/libfetchers -I src/lib
libexpr_LIBS = libutil libstore libfetchers
libexpr_LDFLAGS = -lboost_context
libexpr_LDFLAGS = -lboost_context -pthread
ifdef HOST_LINUX
libexpr_LDFLAGS += -ldl
endif
@@ -35,7 +35,7 @@ $(d)/lexer-tab.cc $(d)/lexer-tab.hh: $(d)/lexer.l
clean-files += $(d)/parser-tab.cc $(d)/parser-tab.hh $(d)/lexer-tab.cc $(d)/lexer-tab.hh
$(eval $(call install-file-in, $(d)/nix-expr.pc, $(prefix)/lib/pkgconfig, 0644))
$(eval $(call install-file-in, $(d)/nix-expr.pc, $(libdir)/pkgconfig, 0644))
$(foreach i, $(wildcard src/libexpr/flake/*.hh), \
$(eval $(call install-file-in, $(i), $(includedir)/nix/flake, 0644)))
-2
View File
@@ -3645,9 +3645,7 @@ void EvalState::createBaseEnv()
if (!evalSettings.pureEval) {
mkInt(v, time(0));
addConstant("__currentTime", v);
}
if (!evalSettings.pureEval) {
mkString(v, settings.thisSystem.get());
addConstant("__currentSystem", v);
}
+8 -1
View File
@@ -4,6 +4,7 @@
#include "tarfile.hh"
#include "store-api.hh"
#include "url-parts.hh"
#include "pathlocks.hh"
#include <sys/time.h>
#include <sys/wait.h>
@@ -317,11 +318,17 @@ struct GitInputScheme : InputScheme
Path cacheDir = getCacheDir() + "/nix/gitv3/" + hashString(htSHA256, actualUrl).to_string(Base32, false);
repoDir = cacheDir;
Path cacheDirLock = cacheDir + ".lock";
createDirs(dirOf(cacheDir));
AutoCloseFD lock = openLockFile(cacheDirLock, true);
lockFile(lock.get(), ltWrite, true);
if (!pathExists(cacheDir)) {
createDirs(dirOf(cacheDir));
runProgram("git", true, { "init", "--bare", repoDir });
}
deleteLockFile(cacheDirLock, lock.get());
Path localRefFile =
input.getRef()->compare(0, 5, "refs/") == 0
? cacheDir + "/" + *input.getRef()
+2
View File
@@ -8,4 +8,6 @@ libfetchers_SOURCES := $(wildcard $(d)/*.cc)
libfetchers_CXXFLAGS += -I src/libutil -I src/libstore
libfetchers_LDFLAGS = -pthread
libfetchers_LIBS = libutil libstore
+1 -1
View File
@@ -14,4 +14,4 @@ libmain_LIBS = libstore libutil
libmain_ALLOW_UNDEFINED = 1
$(eval $(call install-file-in, $(d)/nix-main.pc, $(prefix)/lib/pkgconfig, 0644))
$(eval $(call install-file-in, $(d)/nix-main.pc, $(libdir)/pkgconfig, 0644))
-11
View File
@@ -130,17 +130,6 @@ AutoCloseFD openFile(const Path & path)
return fd;
}
struct FileSource : FdSource
{
AutoCloseFD fd2;
FileSource(const Path & path)
: fd2(openFile(path))
{
fd = fd2.get();
}
};
ref<const ValidPathInfo> BinaryCacheStore::addToStoreCommon(
Source & narSource, RepairFlag repair, CheckSigsFlag checkSigs,
std::function<ValidPathInfo(HashResult)> mkInfo)
+17 -8
View File
@@ -165,7 +165,7 @@ void DerivationGoal::getDerivation()
/* The first thing to do is to make sure that the derivation
exists. If it doesn't, it may be created through a
substitute. */
if (buildMode == bmNormal && worker.store.isValidPath(drvPath)) {
if (buildMode == bmNormal && worker.evalStore.isValidPath(drvPath)) {
loadDerivation();
return;
}
@@ -188,12 +188,12 @@ void DerivationGoal::loadDerivation()
/* `drvPath' should already be a root, but let's be on the safe
side: if the user forgot to make it a root, we wouldn't want
things being garbage collected while we're busy. */
worker.store.addTempRoot(drvPath);
worker.evalStore.addTempRoot(drvPath);
assert(worker.store.isValidPath(drvPath));
assert(worker.evalStore.isValidPath(drvPath));
/* Get the derivation. */
drv = std::make_unique<Derivation>(worker.store.derivationFromPath(drvPath));
drv = std::make_unique<Derivation>(worker.evalStore.derivationFromPath(drvPath));
haveDerivation();
}
@@ -212,8 +212,8 @@ void DerivationGoal::haveDerivation()
if (i.second.second)
worker.store.addTempRoot(*i.second.second);
auto outputHashes = staticOutputHashes(worker.store, *drv);
for (auto &[outputName, outputHash] : outputHashes)
auto outputHashes = staticOutputHashes(worker.evalStore, *drv);
for (auto & [outputName, outputHash] : outputHashes)
initialOutputs.insert({
outputName,
InitialOutput{
@@ -337,6 +337,15 @@ void DerivationGoal::gaveUpOnSubstitution()
for (auto & i : dynamic_cast<Derivation *>(drv.get())->inputDrvs)
addWaitee(worker.makeDerivationGoal(i.first, i.second, buildMode == bmRepair ? bmRepair : bmNormal));
/* Copy the input sources from the eval store to the build
store. */
if (&worker.evalStore != &worker.store) {
RealisedPath::Set inputSrcs;
for (auto & i : drv->inputSrcs)
inputSrcs.insert(i);
copyClosure(worker.evalStore, worker.store, inputSrcs);
}
for (auto & i : drv->inputSrcs) {
if (worker.store.isValidPath(i)) continue;
if (!settings.useSubstitutes)
@@ -478,8 +487,8 @@ void DerivationGoal::inputsRealised()
/* Add the relevant output closures of the input derivation
`i' as input paths. Only add the closures of output paths
that are specified as inputs. */
assert(worker.store.isValidPath(drvPath));
auto outputs = worker.store.queryPartialDerivationOutputMap(depDrvPath);
assert(worker.evalStore.isValidPath(drvPath));
auto outputs = worker.evalStore.queryPartialDerivationOutputMap(depDrvPath);
for (auto & j : wantedDepOutputs) {
if (outputs.count(j) > 0) {
auto optRealizedInput = outputs.at(j);
+5 -5
View File
@@ -6,9 +6,9 @@
namespace nix {
void Store::buildPaths(const std::vector<DerivedPath> & reqs, BuildMode buildMode)
void Store::buildPaths(const std::vector<DerivedPath> & reqs, BuildMode buildMode, std::shared_ptr<Store> evalStore)
{
Worker worker(*this);
Worker worker(*this, evalStore ? *evalStore : *this);
Goals goals;
for (auto & br : reqs) {
@@ -51,7 +51,7 @@ void Store::buildPaths(const std::vector<DerivedPath> & reqs, BuildMode buildMod
BuildResult Store::buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
BuildMode buildMode)
{
Worker worker(*this);
Worker worker(*this, *this);
auto goal = worker.makeBasicDerivationGoal(drvPath, drv, {}, buildMode);
BuildResult result;
@@ -93,7 +93,7 @@ void Store::ensurePath(const StorePath & path)
/* If the path is already valid, we're done. */
if (isValidPath(path)) return;
Worker worker(*this);
Worker worker(*this, *this);
GoalPtr goal = worker.makePathSubstitutionGoal(path);
Goals goals = {goal};
@@ -111,7 +111,7 @@ void Store::ensurePath(const StorePath & path)
void LocalStore::repairPath(const StorePath & path)
{
Worker worker(*this);
Worker worker(*this, *this);
GoalPtr goal = worker.makePathSubstitutionGoal(path, Repair);
Goals goals = {goal};
+4 -9
View File
@@ -13,11 +13,9 @@ bool CompareGoalPtrs::operator() (const GoalPtr & a, const GoalPtr & b) const {
void addToWeakGoals(WeakGoals & goals, GoalPtr p)
{
// FIXME: necessary?
// FIXME: O(n)
for (auto & i : goals)
if (i.lock() == p) return;
goals.push_back(p);
if (goals.find(p) != goals.end())
return;
goals.insert(p);
}
@@ -46,10 +44,7 @@ void Goal::waiteeDone(GoalPtr waitee, ExitCode result)
/* If we failed and keepGoing is not set, we remove all
remaining waitees. */
for (auto & goal : waitees) {
WeakGoals waiters2;
for (auto & j : goal->waiters)
if (j.lock() != shared_from_this()) waiters2.push_back(j);
goal->waiters = waiters2;
goal->waiters.extract(shared_from_this());
}
waitees.clear();
+1 -1
View File
@@ -19,7 +19,7 @@ struct CompareGoalPtrs {
/* Set of goals. */
typedef set<GoalPtr, CompareGoalPtrs> Goals;
typedef list<WeakGoalPtr> WeakGoals;
typedef set<WeakGoalPtr, std::owner_less<WeakGoalPtr>> WeakGoals;
/* A map of paths to goals (and the other way around). */
typedef std::map<StorePath, WeakGoalPtr> WeakGoalMap;
+3 -1
View File
@@ -1254,8 +1254,10 @@ struct RestrictedStore : public virtual RestrictedStoreConfig, public virtual Lo
return next->queryRealisation(id);
}
void buildPaths(const std::vector<DerivedPath> & paths, BuildMode buildMode) override
void buildPaths(const std::vector<DerivedPath> & paths, BuildMode buildMode, std::shared_ptr<Store> evalStore) override
{
assert(!evalStore);
if (buildMode != bmNormal) throw Error("unsupported build mode");
StorePathSet newPaths;
+1 -1
View File
@@ -204,7 +204,7 @@ void PathSubstitutionGoal::tryToRun()
Activity act(*logger, actSubstitute, Logger::Fields{worker.store.printStorePath(storePath), sub->getUri()});
PushActivity pact(act.id);
copyStorePath(ref<Store>(sub), ref<Store>(worker.store.shared_from_this()),
copyStorePath(*sub, worker.store,
subPath ? *subPath : storePath, repair, sub->isTrusted ? NoCheckSigs : CheckSigs);
promise.set_value();
+2 -1
View File
@@ -9,11 +9,12 @@
namespace nix {
Worker::Worker(Store & store)
Worker::Worker(Store & store, Store & evalStore)
: act(*logger, actRealise)
, actDerivations(*logger, actBuilds)
, actSubstitutions(*logger, actCopyPaths)
, store(store)
, evalStore(evalStore)
{
/* Debugging: prevent recursive workers. */
nrLocalBuilds = 0;
+2 -1
View File
@@ -110,6 +110,7 @@ public:
bool checkMismatch;
Store & store;
Store & evalStore;
std::unique_ptr<HookInstance> hook;
@@ -131,7 +132,7 @@ public:
it answers with "decline-permanently", we don't try again. */
bool tryBuildHook = true;
Worker(Store & store);
Worker(Store & store, Store & evalStore);
~Worker();
/* Make a goal (with caching). */
+17 -21
View File
@@ -243,23 +243,6 @@ struct ClientSettings
}
};
static void writeValidPathInfo(
ref<Store> store,
unsigned int clientVersion,
Sink & to,
std::shared_ptr<const ValidPathInfo> info)
{
to << (info->deriver ? store->printStorePath(*info->deriver) : "")
<< info->narHash.to_string(Base16, false);
worker_proto::write(*store, to, info->references);
to << info->registrationTime << info->narSize;
if (GET_PROTOCOL_MINOR(clientVersion) >= 16) {
to << info->ultimate
<< info->sigs
<< renderContentAddress(info->ca);
}
}
static std::vector<DerivedPath> readDerivedPaths(Store & store, unsigned int clientVersion, Source & from)
{
std::vector<DerivedPath> reqs;
@@ -422,9 +405,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
}();
logger->stopWork();
to << store->printStorePath(pathInfo->path);
writeValidPathInfo(store, clientVersion, to, pathInfo);
pathInfo->write(to, *store, GET_PROTOCOL_MINOR(clientVersion));
} else {
HashType hashAlgo;
std::string baseName;
@@ -471,6 +452,21 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
break;
}
case wopAddMultipleToStore: {
bool repair, dontCheckSigs;
from >> repair >> dontCheckSigs;
if (!trusted && dontCheckSigs)
dontCheckSigs = false;
logger->startWork();
FramedSource source(from);
store->addMultipleToStore(source,
RepairFlag{repair},
dontCheckSigs ? NoCheckSigs : CheckSigs);
logger->stopWork();
break;
}
case wopAddTextToStore: {
string suffix = readString(from);
string s = readString(from);
@@ -770,7 +766,7 @@ static void performOp(TunnelLogger * logger, ref<Store> store,
if (info) {
if (GET_PROTOCOL_MINOR(clientVersion) >= 17)
to << 1;
writeValidPathInfo(store, clientVersion, to, info);
info->write(to, *store, GET_PROTOCOL_MINOR(clientVersion), false);
} else {
assert(GET_PROTOCOL_MINOR(clientVersion) >= 17);
to << 0;
+1 -1
View File
@@ -568,7 +568,7 @@ DrvHashModulo hashDerivationModulo(Store & store, const Derivation & drv, bool m
}
std::map<std::string, Hash> staticOutputHashes(Store& store, const Derivation& drv)
std::map<std::string, Hash> staticOutputHashes(Store & store, const Derivation & drv)
{
std::map<std::string, Hash> res;
std::visit(overloaded {
-5
View File
@@ -43,11 +43,6 @@ struct DummyStore : public virtual DummyStoreConfig, public virtual Store
RepairFlag repair, CheckSigsFlag checkSigs) override
{ unsupported("addToStore"); }
StorePath addToStore(const string & name, const Path & srcPath,
FileIngestionMethod method, HashType hashAlgo,
PathFilter & filter, RepairFlag repair) override
{ unsupported("addToStore"); }
StorePath addTextToStore(const string & name, const string & s,
const StorePathSet & references, RepairFlag repair) override
{ unsupported("addTextToStore"); }
+4 -1
View File
@@ -267,8 +267,11 @@ public:
return status;
}
void buildPaths(const std::vector<DerivedPath> & drvPaths, BuildMode buildMode) override
void buildPaths(const std::vector<DerivedPath> & drvPaths, BuildMode buildMode, std::shared_ptr<Store> evalStore) override
{
if (evalStore && evalStore.get() != this)
throw Error("building on an SSH store is incompatible with '--eval-store'");
auto conn(connections->get());
conn->to << cmdBuildPaths;
+2 -2
View File
@@ -8,7 +8,7 @@ libstore_SOURCES := $(wildcard $(d)/*.cc $(d)/builtins/*.cc $(d)/build/*.cc)
libstore_LIBS = libutil
libstore_LDFLAGS = $(SQLITE3_LIBS) -lbz2 $(LIBCURL_LIBS) $(SODIUM_LIBS) -pthread
libstore_LDFLAGS = $(SQLITE3_LIBS) $(LIBCURL_LIBS) $(SODIUM_LIBS) -pthread
ifdef HOST_LINUX
libstore_LDFLAGS += -ldl
endif
@@ -60,7 +60,7 @@ $(d)/build.cc:
clean-files += $(d)/schema.sql.gen.hh $(d)/ca-specific-schema.sql.gen.hh
$(eval $(call install-file-in, $(d)/nix-store.pc, $(prefix)/lib/pkgconfig, 0644))
$(eval $(call install-file-in, $(d)/nix-store.pc, $(libdir)/pkgconfig, 0644))
$(foreach i, $(wildcard src/libstore/builtins/*.hh), \
$(eval $(call install-file-in, $(i), $(includedir)/nix/builtins, 0644)))
+46
View File
@@ -0,0 +1,46 @@
#include "path-info.hh"
#include "worker-protocol.hh"
namespace nix {
ValidPathInfo ValidPathInfo::read(Source & source, const Store & store, unsigned int format)
{
return read(source, store, format, store.parseStorePath(readString(source)));
}
ValidPathInfo ValidPathInfo::read(Source & source, const Store & store, unsigned int format, StorePath && path)
{
auto deriver = readString(source);
auto narHash = Hash::parseAny(readString(source), htSHA256);
ValidPathInfo info(path, narHash);
if (deriver != "") info.deriver = store.parseStorePath(deriver);
info.references = worker_proto::read(store, source, Phantom<StorePathSet> {});
source >> info.registrationTime >> info.narSize;
if (format >= 16) {
source >> info.ultimate;
info.sigs = readStrings<StringSet>(source);
info.ca = parseContentAddressOpt(readString(source));
}
return info;
}
void ValidPathInfo::write(
Sink & sink,
const Store & store,
unsigned int format,
bool includePath) const
{
if (includePath)
sink << store.printStorePath(path);
sink << (deriver ? store.printStorePath(*deriver) : "")
<< narHash.to_string(Base16, false);
worker_proto::write(store, sink, references);
sink << registrationTime << narSize;
if (format >= 16) {
sink << ultimate
<< sigs
<< renderContentAddress(ca);
}
}
}
+5
View File
@@ -105,6 +105,11 @@ struct ValidPathInfo
ValidPathInfo(const StorePath & path, Hash narHash) : path(path), narHash(narHash) { };
virtual ~ValidPathInfo() { }
static ValidPathInfo read(Source & source, const Store & store, unsigned int format);
static ValidPathInfo read(Source & source, const Store & store, unsigned int format, StorePath && path);
void write(Sink & sink, const Store & store, unsigned int format, bool includePath = true) const;
};
typedef std::map<StorePath, ValidPathInfo> ValidPathInfos;
+1 -1
View File
@@ -45,7 +45,7 @@ struct Realisation {
size_t checkSignatures(const PublicKeys & publicKeys) const;
static std::set<Realisation> closure(Store &, const std::set<Realisation> &);
static void closure(Store &, const std::set<Realisation> &, std::set<Realisation>& res);
static void closure(Store &, const std::set<Realisation> &, std::set<Realisation> & res);
bool isCompatibleWith(const Realisation & other) const;
+4 -4
View File
@@ -5,6 +5,7 @@
#include <map>
#include <cstdlib>
#include <mutex>
namespace nix {
@@ -16,14 +17,13 @@ static unsigned int refLength = 32; /* characters */
static void search(const unsigned char * s, size_t len,
StringSet & hashes, StringSet & seen)
{
static bool initialised = false;
static std::once_flag initialised;
static bool isBase32[256];
if (!initialised) {
std::call_once(initialised, [](){
for (unsigned int i = 0; i < 256; ++i) isBase32[i] = false;
for (unsigned int i = 0; i < base32Chars.size(); ++i)
isBase32[(unsigned char) base32Chars[i]] = true;
initialised = true;
}
});
for (size_t i = 0; i + refLength <= len; ) {
int j;
+37 -25
View File
@@ -386,23 +386,6 @@ void RemoteStore::querySubstitutablePathInfos(const StorePathCAMap & pathsMap, S
}
ref<const ValidPathInfo> RemoteStore::readValidPathInfo(ConnectionHandle & conn, const StorePath & path)
{
auto deriver = readString(conn->from);
auto narHash = Hash::parseAny(readString(conn->from), htSHA256);
auto info = make_ref<ValidPathInfo>(path, narHash);
if (deriver != "") info->deriver = parseStorePath(deriver);
info->references = worker_proto::read(*this, conn->from, Phantom<StorePathSet> {});
conn->from >> info->registrationTime >> info->narSize;
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 16) {
conn->from >> info->ultimate;
info->sigs = readStrings<StringSet>(conn->from);
info->ca = parseContentAddressOpt(readString(conn->from));
}
return info;
}
void RemoteStore::queryPathInfoUncached(const StorePath & path,
Callback<std::shared_ptr<const ValidPathInfo>> callback) noexcept
{
@@ -423,7 +406,8 @@ void RemoteStore::queryPathInfoUncached(const StorePath & path,
bool valid; conn->from >> valid;
if (!valid) throw InvalidPath("path '%s' is not valid", printStorePath(path));
}
info = readValidPathInfo(conn, path);
info = std::make_shared<ValidPathInfo>(
ValidPathInfo::read(conn->from, *this, GET_PROTOCOL_MINOR(conn->daemonVersion), StorePath{path}));
}
callback(std::move(info));
} catch (...) { callback.rethrow(); }
@@ -525,8 +509,8 @@ ref<const ValidPathInfo> RemoteStore::addCAToStore(
});
}
auto path = parseStorePath(readString(conn->from));
return readValidPathInfo(conn, path);
return make_ref<ValidPathInfo>(
ValidPathInfo::read(conn->from, *this, GET_PROTOCOL_MINOR(conn->daemonVersion)));
}
else {
if (repair) throw Error("repairing is not supported when building through the Nix daemon protocol < 1.25");
@@ -642,6 +626,25 @@ void RemoteStore::addToStore(const ValidPathInfo & info, Source & source,
}
void RemoteStore::addMultipleToStore(
Source & source,
RepairFlag repair,
CheckSigsFlag checkSigs)
{
if (GET_PROTOCOL_MINOR(getConnection()->daemonVersion) >= 32) {
auto conn(getConnection());
conn->to
<< wopAddMultipleToStore
<< repair
<< !checkSigs;
conn.withFramedSink([&](Sink & sink) {
source.drainInto(sink);
});
} else
Store::addMultipleToStore(source, repair, checkSigs);
}
StorePath RemoteStore::addTextToStore(const string & name, const string & s,
const StorePathSet & references, RepairFlag repair)
{
@@ -705,8 +708,18 @@ static void writeDerivedPaths(RemoteStore & store, ConnectionHandle & conn, cons
}
}
void RemoteStore::buildPaths(const std::vector<DerivedPath> & drvPaths, BuildMode buildMode)
void RemoteStore::buildPaths(const std::vector<DerivedPath> & drvPaths, BuildMode buildMode, std::shared_ptr<Store> evalStore)
{
if (evalStore && evalStore.get() != this) {
/* The remote doesn't have a way to access evalStore, so copy
the .drvs. */
RealisedPath::Set drvPaths2;
for (auto & i : drvPaths)
if (auto p = std::get_if<DerivedPath::Built>(&i))
drvPaths2.insert(p->drvPath);
copyClosure(*evalStore, *this, drvPaths2);
}
auto conn(getConnection());
conn->to << wopBuildPaths;
assert(GET_PROTOCOL_MINOR(conn->daemonVersion) >= 13);
@@ -1001,14 +1014,14 @@ std::exception_ptr RemoteStore::Connection::processStderr(Sink * sink, Source *
return nullptr;
}
void ConnectionHandle::withFramedSink(std::function<void(Sink &sink)> fun)
void ConnectionHandle::withFramedSink(std::function<void(Sink & sink)> fun)
{
(*this)->to.flush();
std::exception_ptr ex;
/* Handle log messages / exceptions from the remote on a
separate thread. */
/* Handle log messages / exceptions from the remote on a separate
thread. */
std::thread stderrThread([&]()
{
try {
@@ -1041,7 +1054,6 @@ void ConnectionHandle::withFramedSink(std::function<void(Sink &sink)> fun)
stderrThread.join();
if (ex)
std::rethrow_exception(ex);
}
}
+6 -3
View File
@@ -78,6 +78,11 @@ public:
void addToStore(const ValidPathInfo & info, Source & nar,
RepairFlag repair, CheckSigsFlag checkSigs) override;
void addMultipleToStore(
Source & source,
RepairFlag repair,
CheckSigsFlag checkSigs) override;
StorePath addTextToStore(const string & name, const string & s,
const StorePathSet & references, RepairFlag repair) override;
@@ -85,7 +90,7 @@ public:
std::optional<const Realisation> queryRealisation(const DrvOutput &) override;
void buildPaths(const std::vector<DerivedPath> & paths, BuildMode buildMode) override;
void buildPaths(const std::vector<DerivedPath> & paths, BuildMode buildMode, std::shared_ptr<Store> evalStore) override;
BuildResult buildDerivation(const StorePath & drvPath, const BasicDerivation & drv,
BuildMode buildMode) override;
@@ -151,8 +156,6 @@ protected:
virtual void narFromPath(const StorePath & path, Sink & sink) override;
ref<const ValidPathInfo> readValidPathInfo(ConnectionHandle & conn, const StorePath & path);
private:
std::atomic_bool failed{false};
+121 -37
View File
@@ -9,6 +9,7 @@
#include "url.hh"
#include "archive.hh"
#include "callback.hh"
#include "remote-store.hh"
#include <regex>
@@ -249,6 +250,20 @@ StorePath Store::addToStore(const string & name, const Path & _srcPath,
}
void Store::addMultipleToStore(
Source & source,
RepairFlag repair,
CheckSigsFlag checkSigs)
{
auto expected = readNum<uint64_t>(source);
for (uint64_t i = 0; i < expected; ++i) {
auto info = ValidPathInfo::read(source, *this, 16);
info.ultimate = false;
addToStore(info, source, repair, checkSigs);
}
}
/*
The aim of this function is to compute in one pass the correct ValidPathInfo for
the files that we are trying to add to the store. To accomplish that in one
@@ -770,30 +785,43 @@ const Store::Stats & Store::getStats()
}
void copyStorePath(ref<Store> srcStore, ref<Store> dstStore,
const StorePath & storePath, RepairFlag repair, CheckSigsFlag checkSigs)
static std::string makeCopyPathMessage(
std::string_view srcUri,
std::string_view dstUri,
std::string_view storePath)
{
auto srcUri = srcStore->getUri();
auto dstUri = dstStore->getUri();
return srcUri == "local" || srcUri == "daemon"
? fmt("copying path '%s' to '%s'", storePath, dstUri)
: dstUri == "local" || dstUri == "daemon"
? fmt("copying path '%s' from '%s'", storePath, srcUri)
: fmt("copying path '%s' from '%s' to '%s'", storePath, srcUri, dstUri);
}
void copyStorePath(
Store & srcStore,
Store & dstStore,
const StorePath & storePath,
RepairFlag repair,
CheckSigsFlag checkSigs)
{
auto srcUri = srcStore.getUri();
auto dstUri = dstStore.getUri();
auto storePathS = srcStore.printStorePath(storePath);
Activity act(*logger, lvlInfo, actCopyPath,
srcUri == "local" || srcUri == "daemon"
? fmt("copying path '%s' to '%s'", srcStore->printStorePath(storePath), dstUri)
: dstUri == "local" || dstUri == "daemon"
? fmt("copying path '%s' from '%s'", srcStore->printStorePath(storePath), srcUri)
: fmt("copying path '%s' from '%s' to '%s'", srcStore->printStorePath(storePath), srcUri, dstUri),
{srcStore->printStorePath(storePath), srcUri, dstUri});
makeCopyPathMessage(srcUri, dstUri, storePathS),
{storePathS, srcUri, dstUri});
PushActivity pact(act.id);
auto info = srcStore->queryPathInfo(storePath);
auto info = srcStore.queryPathInfo(storePath);
uint64_t total = 0;
// recompute store path on the chance dstStore does it differently
if (info->ca && info->references.empty()) {
auto info2 = make_ref<ValidPathInfo>(*info);
info2->path = dstStore->makeFixedOutputPathFromCA(info->path.name(), *info->ca);
if (dstStore->storeDir == srcStore->storeDir)
info2->path = dstStore.makeFixedOutputPathFromCA(info->path.name(), *info->ca);
if (dstStore.storeDir == srcStore.storeDir)
assert(info->path == info2->path);
info = info2;
}
@@ -810,17 +838,22 @@ void copyStorePath(ref<Store> srcStore, ref<Store> dstStore,
act.progress(total, info->narSize);
});
TeeSink tee { sink, progressSink };
srcStore->narFromPath(storePath, tee);
srcStore.narFromPath(storePath, tee);
}, [&]() {
throw EndOfFile("NAR for '%s' fetched from '%s' is incomplete", srcStore->printStorePath(storePath), srcStore->getUri());
throw EndOfFile("NAR for '%s' fetched from '%s' is incomplete", srcStore.printStorePath(storePath), srcStore.getUri());
});
dstStore->addToStore(*info, *source, repair, checkSigs);
dstStore.addToStore(*info, *source, repair, checkSigs);
}
std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStore, const RealisedPath::Set & paths,
RepairFlag repair, CheckSigsFlag checkSigs, SubstituteFlag substitute)
std::map<StorePath, StorePath> copyPaths(
Store & srcStore,
Store & dstStore,
const RealisedPath::Set & paths,
RepairFlag repair,
CheckSigsFlag checkSigs,
SubstituteFlag substitute)
{
StorePathSet storePaths;
std::set<Realisation> toplevelRealisations;
@@ -838,11 +871,11 @@ std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStor
try {
// Copy the realisation closure
processGraph<Realisation>(
pool, Realisation::closure(*srcStore, toplevelRealisations),
pool, Realisation::closure(srcStore, toplevelRealisations),
[&](const Realisation & current) -> std::set<Realisation> {
std::set<Realisation> children;
for (const auto & [drvOutput, _] : current.dependentRealisations) {
auto currentChild = srcStore->queryRealisation(drvOutput);
auto currentChild = srcStore.queryRealisation(drvOutput);
if (!currentChild)
throw Error(
"incomplete realisation closure: '%s' is a "
@@ -853,9 +886,9 @@ std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStor
return children;
},
[&](const Realisation& current) -> void {
dstStore->registerDrvOutput(current, checkSigs);
dstStore.registerDrvOutput(current, checkSigs);
});
} catch (MissingExperimentalFeature& e) {
} catch (MissingExperimentalFeature & e) {
// Don't fail if the remote doesn't support CA derivations is it might
// not be within our control to change that, and we might still want
// to at least copy the output paths.
@@ -868,10 +901,15 @@ std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStor
return pathsMap;
}
std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStore, const StorePathSet & storePaths,
RepairFlag repair, CheckSigsFlag checkSigs, SubstituteFlag substitute)
std::map<StorePath, StorePath> copyPaths(
Store & srcStore,
Store & dstStore,
const StorePathSet & storePaths,
RepairFlag repair,
CheckSigsFlag checkSigs,
SubstituteFlag substitute)
{
auto valid = dstStore->queryValidPaths(storePaths, substitute);
auto valid = dstStore.queryValidPaths(storePaths, substitute);
StorePathSet missing;
for (auto & path : storePaths)
@@ -881,9 +919,31 @@ std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStor
for (auto & path : storePaths)
pathsMap.insert_or_assign(path, path);
Activity act(*logger, lvlInfo, actCopyPaths, fmt("copying %d paths", missing.size()));
auto sorted = srcStore.topoSortPaths(missing);
std::reverse(sorted.begin(), sorted.end());
auto source = sinkToSource([&](Sink & sink) {
sink << sorted.size();
for (auto & storePath : sorted) {
auto srcUri = srcStore.getUri();
auto dstUri = dstStore.getUri();
auto storePathS = srcStore.printStorePath(storePath);
Activity act(*logger, lvlInfo, actCopyPath,
makeCopyPathMessage(srcUri, dstUri, storePathS),
{storePathS, srcUri, dstUri});
PushActivity pact(act.id);
auto info = srcStore.queryPathInfo(storePath);
info->write(sink, srcStore, 16);
srcStore.narFromPath(storePath, sink);
}
});
dstStore.addMultipleToStore(*source, repair, checkSigs);
#if 0
std::atomic<size_t> nrDone{0};
std::atomic<size_t> nrFailed{0};
std::atomic<uint64_t> bytesExpected{0};
@@ -899,18 +959,21 @@ std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStor
StorePathSet(missing.begin(), missing.end()),
[&](const StorePath & storePath) {
auto info = srcStore->queryPathInfo(storePath);
auto info = srcStore.queryPathInfo(storePath);
auto storePathForDst = storePath;
if (info->ca && info->references.empty()) {
storePathForDst = dstStore->makeFixedOutputPathFromCA(storePath.name(), *info->ca);
if (dstStore->storeDir == srcStore->storeDir)
storePathForDst = dstStore.makeFixedOutputPathFromCA(storePath.name(), *info->ca);
if (dstStore.storeDir == srcStore.storeDir)
assert(storePathForDst == storePath);
if (storePathForDst != storePath)
debug("replaced path '%s' to '%s' for substituter '%s'", srcStore->printStorePath(storePath), dstStore->printStorePath(storePathForDst), dstStore->getUri());
debug("replaced path '%s' to '%s' for substituter '%s'",
srcStore.printStorePath(storePath),
dstStore.printStorePath(storePathForDst),
dstStore.getUri());
}
pathsMap.insert_or_assign(storePath, storePathForDst);
if (dstStore->isValidPath(storePath)) {
if (dstStore.isValidPath(storePath)) {
nrDone++;
showProgress();
return StorePathSet();
@@ -925,19 +988,22 @@ std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStor
[&](const StorePath & storePath) {
checkInterrupt();
auto info = srcStore->queryPathInfo(storePath);
auto info = srcStore.queryPathInfo(storePath);
auto storePathForDst = storePath;
if (info->ca && info->references.empty()) {
storePathForDst = dstStore->makeFixedOutputPathFromCA(storePath.name(), *info->ca);
if (dstStore->storeDir == srcStore->storeDir)
storePathForDst = dstStore.makeFixedOutputPathFromCA(storePath.name(), *info->ca);
if (dstStore.storeDir == srcStore.storeDir)
assert(storePathForDst == storePath);
if (storePathForDst != storePath)
debug("replaced path '%s' to '%s' for substituter '%s'", srcStore->printStorePath(storePath), dstStore->printStorePath(storePathForDst), dstStore->getUri());
debug("replaced path '%s' to '%s' for substituter '%s'",
srcStore.printStorePath(storePath),
dstStore.printStorePath(storePathForDst),
dstStore.getUri());
}
pathsMap.insert_or_assign(storePath, storePathForDst);
if (!dstStore->isValidPath(storePathForDst)) {
if (!dstStore.isValidPath(storePathForDst)) {
MaintainCount<decltype(nrRunning)> mc(nrRunning);
showProgress();
try {
@@ -946,7 +1012,7 @@ std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStor
nrFailed++;
if (!settings.keepGoing)
throw e;
logger->log(lvlError, fmt("could not copy %s: %s", dstStore->printStorePath(storePath), e.what()));
logger->log(lvlError, fmt("could not copy %s: %s", dstStore.printStorePath(storePath), e.what()));
showProgress();
return;
}
@@ -955,9 +1021,27 @@ std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStor
nrDone++;
showProgress();
});
#endif
return pathsMap;
}
void copyClosure(
Store & srcStore,
Store & dstStore,
const RealisedPath::Set & paths,
RepairFlag repair,
CheckSigsFlag checkSigs,
SubstituteFlag substitute)
{
if (&srcStore == &dstStore) return;
RealisedPath::Set closure;
RealisedPath::closure(srcStore, paths, closure);
copyPaths(srcStore, dstStore, closure, repair, checkSigs, substitute);
}
std::optional<ValidPathInfo> decodeValidPathInfo(const Store & store, std::istream & str, std::optional<HashResult> hashGiven)
{
std::string path;
+26 -5
View File
@@ -440,6 +440,12 @@ public:
virtual void addToStore(const ValidPathInfo & info, Source & narSource,
RepairFlag repair = NoRepair, CheckSigsFlag checkSigs = CheckSigs) = 0;
/* Import multiple paths into the store. */
virtual void addMultipleToStore(
Source & source,
RepairFlag repair = NoRepair,
CheckSigsFlag checkSigs = CheckSigs);
/* Copy the contents of a path to the store and register the
validity the resulting path. The resulting path is returned.
The function object `filter' can be used to exclude files (see
@@ -497,7 +503,8 @@ public:
not derivations, substitute them. */
virtual void buildPaths(
const std::vector<DerivedPath> & paths,
BuildMode buildMode = bmNormal);
BuildMode buildMode = bmNormal,
std::shared_ptr<Store> evalStore = nullptr);
/* Build a single non-materialized derivation (i.e. not from an
on-disk .drv file).
@@ -751,8 +758,12 @@ protected:
/* Copy a path from one store to another. */
void copyStorePath(ref<Store> srcStore, ref<Store> dstStore,
const StorePath & storePath, RepairFlag repair = NoRepair, CheckSigsFlag checkSigs = CheckSigs);
void copyStorePath(
Store & srcStore,
Store & dstStore,
const StorePath & storePath,
RepairFlag repair = NoRepair,
CheckSigsFlag checkSigs = CheckSigs);
/* Copy store paths from one store to another. The paths may be copied
@@ -761,17 +772,27 @@ void copyStorePath(ref<Store> srcStore, ref<Store> dstStore,
of store paths is not automatically closed; use copyClosure() for
that. Returns a map of what each path was copied to the dstStore
as. */
std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStore,
std::map<StorePath, StorePath> copyPaths(
Store & srcStore, Store & dstStore,
const RealisedPath::Set &,
RepairFlag repair = NoRepair,
CheckSigsFlag checkSigs = CheckSigs,
SubstituteFlag substitute = NoSubstitute);
std::map<StorePath, StorePath> copyPaths(ref<Store> srcStore, ref<Store> dstStore,
std::map<StorePath, StorePath> copyPaths(
Store & srcStore, Store & dstStore,
const StorePathSet & paths,
RepairFlag repair = NoRepair,
CheckSigsFlag checkSigs = CheckSigs,
SubstituteFlag substitute = NoSubstitute);
/* Copy the closure of `paths` from `srcStore` to `dstStore`. */
void copyClosure(
Store & srcStore, Store & dstStore,
const RealisedPath::Set & paths,
RepairFlag repair = NoRepair,
CheckSigsFlag checkSigs = CheckSigs,
SubstituteFlag substitute = NoSubstitute);
/* Remove the temporary roots file for this process. Any temporary
root becomes garbage after this point unless it has been registered
+2 -1
View File
@@ -9,7 +9,7 @@ namespace nix {
#define WORKER_MAGIC_1 0x6e697863
#define WORKER_MAGIC_2 0x6478696f
#define PROTOCOL_VERSION (1 << 8 | 31)
#define PROTOCOL_VERSION (1 << 8 | 32)
#define GET_PROTOCOL_MAJOR(x) ((x) & 0xff00)
#define GET_PROTOCOL_MINOR(x) ((x) & 0x00ff)
@@ -55,6 +55,7 @@ typedef enum {
wopQueryDerivationOutputMap = 41,
wopRegisterDrvOutput = 42,
wopQueryRealisation = 43,
wopAddMultipleToStore = 44,
} WorkerOp;
-2
View File
@@ -12,8 +12,6 @@
#include <brotli/decode.h>
#include <brotli/encode.h>
#include <zlib.h>
#include <iostream>
namespace nix {
+10 -12
View File
@@ -6,14 +6,15 @@
#include <cctype>
#include <cerrno>
#include <climits>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <climits>
#include <future>
#include <iostream>
#include <mutex>
#include <sstream>
#include <thread>
#include <future>
#include <fcntl.h>
#include <grp.h>
@@ -1447,7 +1448,7 @@ std::string filterANSIEscapes(const std::string & s, bool filterAll, unsigned in
static char base64Chars[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
static std::array<char, 256> base64DecodeChars;
string base64Encode(std::string_view s)
{
@@ -1472,15 +1473,12 @@ string base64Encode(std::string_view s)
string base64Decode(std::string_view s)
{
bool init = false;
char decode[256];
if (!init) {
// FIXME: not thread-safe.
memset(decode, -1, sizeof(decode));
static std::once_flag flag;
std::call_once(flag, [](){
base64DecodeChars = { (char)-1 };
for (int i = 0; i < 64; i++)
decode[(int) base64Chars[i]] = i;
init = true;
}
base64DecodeChars[(int) base64Chars[i]] = i;
});
string res;
unsigned int d = 0, bits = 0;
@@ -1489,7 +1487,7 @@ string base64Decode(std::string_view s)
if (c == '=') break;
if (c == '\n') continue;
char digit = decode[(unsigned char) c];
char digit = base64DecodeChars[(unsigned char) c];
if (digit == -1)
throw Error("invalid character in Base64 string: '%c'", c);
+25 -12
View File
@@ -250,8 +250,9 @@ static void main_nix_build(int argc, char * * argv)
throw UsageError("'-p' and '-E' are mutually exclusive");
auto store = openStore();
auto evalStore = myArgs.evalStoreUrl ? openStore(*myArgs.evalStoreUrl) : store;
auto state = std::make_unique<EvalState>(myArgs.searchPath, store);
auto state = std::make_unique<EvalState>(myArgs.searchPath, evalStore, store);
state->repair = repair;
auto autoArgs = myArgs.getAutoArgs(*state);
@@ -268,7 +269,7 @@ static void main_nix_build(int argc, char * * argv)
if (packages) {
std::ostringstream joined;
joined << "with import <nixpkgs> { }; (pkgs.runCommandCC or pkgs.runCommand) \"shell\" { buildInputs = [ ";
joined << "{...}@args: with import <nixpkgs> args; (pkgs.runCommandCC or pkgs.runCommand) \"shell\" { buildInputs = [ ";
for (const auto & i : left)
joined << '(' << i << ") ";
joined << "]; } \"\"";
@@ -301,8 +302,8 @@ static void main_nix_build(int argc, char * * argv)
absolute = canonPath(absPath(i), true);
} catch (Error & e) {};
auto [path, outputNames] = parsePathWithOutputs(absolute);
if (store->isStorePath(path) && hasSuffix(path, ".drv"))
drvs.push_back(DrvInfo(*state, store, absolute));
if (evalStore->isStorePath(path) && hasSuffix(path, ".drv"))
drvs.push_back(DrvInfo(*state, evalStore, absolute));
else
/* If we're in a #! script, interpret filenames
relative to the script. */
@@ -340,7 +341,7 @@ static void main_nix_build(int argc, char * * argv)
printMissing(ref<Store>(store), willBuild, willSubstitute, unknown, downloadSize, narSize);
if (!dryRun)
store->buildPaths(paths, buildMode);
store->buildPaths(paths, buildMode, evalStore);
};
if (runEnv) {
@@ -348,9 +349,10 @@ static void main_nix_build(int argc, char * * argv)
throw UsageError("nix-shell requires a single derivation");
auto & drvInfo = drvs.front();
auto drv = store->derivationFromPath(store->parseStorePath(drvInfo.queryDrvPath()));
auto drv = evalStore->derivationFromPath(evalStore->parseStorePath(drvInfo.queryDrvPath()));
std::vector<StorePathWithOutputs> pathsToBuild;
RealisedPath::Set pathsToCopy;
/* Figure out what bash shell to use. If $NIX_BUILD_SHELL
is not set, then build bashInteractive from
@@ -369,7 +371,9 @@ static void main_nix_build(int argc, char * * argv)
if (!drv)
throw Error("the 'bashInteractive' attribute in <nixpkgs> did not evaluate to a derivation");
pathsToBuild.push_back({store->parseStorePath(drv->queryDrvPath())});
auto bashDrv = store->parseStorePath(drv->queryDrvPath());
pathsToBuild.push_back({bashDrv});
pathsToCopy.insert(bashDrv);
shell = drv->queryOutPath() + "/bin/bash";
@@ -384,9 +388,14 @@ static void main_nix_build(int argc, char * * argv)
for (const auto & input : drv.inputDrvs)
if (std::all_of(envExclude.cbegin(), envExclude.cend(),
[&](const string & exclude) { return !std::regex_search(store->printStorePath(input.first), std::regex(exclude)); }))
{
pathsToBuild.push_back({input.first, input.second});
for (const auto & src : drv.inputSrcs)
pathsToCopy.insert(input.first);
}
for (const auto & src : drv.inputSrcs) {
pathsToBuild.push_back({src});
pathsToCopy.insert(src);
}
buildPaths(pathsToBuild);
@@ -438,7 +447,7 @@ static void main_nix_build(int argc, char * * argv)
if (env.count("__json")) {
StorePathSet inputs;
for (auto & [depDrvPath, wantedDepOutputs] : drv.inputDrvs) {
auto outputs = store->queryPartialDerivationOutputMap(depDrvPath);
auto outputs = evalStore->queryPartialDerivationOutputMap(depDrvPath);
for (auto & i : wantedDepOutputs) {
auto o = outputs.at(i);
store->computeFSClosure(*o, inputs);
@@ -480,6 +489,9 @@ static void main_nix_build(int argc, char * * argv)
"_nix_shell_clean_tmpdir; ") +
(pure ? "" : "[ -n \"$PS1\" ] && [ -e ~/.bashrc ] && source ~/.bashrc;") +
"%2%"
// always clear PATH.
// when nix-shell is run impure, we rehydrate it with the `p=$PATH` above
"unset PATH;"
"dontAddDisableDepTrack=1;\n"
+ structuredAttrsRC +
"\n[ -e $stdenv/setup ] && source $stdenv/setup; "
@@ -531,6 +543,7 @@ static void main_nix_build(int argc, char * * argv)
std::vector<StorePathWithOutputs> pathsToBuild;
std::vector<std::pair<StorePath, std::string>> pathsToBuildOrdered;
RealisedPath::Set drvsToCopy;
std::map<StorePath, std::pair<size_t, StringSet>> drvMap;
@@ -543,13 +556,13 @@ static void main_nix_build(int argc, char * * argv)
pathsToBuild.push_back({drvPath, {outputName}});
pathsToBuildOrdered.push_back({drvPath, {outputName}});
drvsToCopy.insert(drvPath);
auto i = drvMap.find(drvPath);
if (i != drvMap.end())
i->second.second.insert(outputName);
else {
else
drvMap[drvPath] = {drvMap.size(), {outputName}};
}
}
buildPaths(pathsToBuild);
@@ -564,7 +577,7 @@ static void main_nix_build(int argc, char * * argv)
if (counter)
drvPrefix += fmt("-%d", counter + 1);
auto builtOutputs = store->queryPartialDerivationOutputMap(drvPath);
auto builtOutputs = evalStore->queryPartialDerivationOutputMap(drvPath);
auto maybeOutputPath = builtOutputs.at(outputName);
assert(maybeOutputPath);
+1 -4
View File
@@ -54,10 +54,7 @@ static int main_nix_copy_closure(int argc, char ** argv)
for (auto & path : storePaths)
storePaths2.insert(from->followLinksToStorePath(path));
RealisedPath::Set closure;
RealisedPath::closure(*from, storePaths2, closure);
copyPaths(from, to, closure, NoRepair, NoCheckSigs, useSubstitutes);
copyClosure(*from, *to, storePaths2, NoRepair, NoCheckSigs, useSubstitutes);
return 0;
}
+2 -1
View File
@@ -153,8 +153,9 @@ static int main_nix_instantiate(int argc, char * * argv)
settings.readOnlyMode = true;
auto store = openStore();
auto evalStore = myArgs.evalStoreUrl ? openStore(*myArgs.evalStoreUrl) : store;
auto state = std::make_unique<EvalState>(myArgs.searchPath, store);
auto state = std::make_unique<EvalState>(myArgs.searchPath, evalStore, store);
state->repair = repair;
Bindings & autoArgs = *myArgs.getAutoArgs(*state);
+3 -2
View File
@@ -100,7 +100,8 @@ UnresolvedApp Installable::toApp(EvalState & state)
throw Error("attribute '%s' has unsupported type '%s'", attrPath, type);
}
App UnresolvedApp::resolve(ref<Store> store)
// FIXME: move to libcmd
App UnresolvedApp::resolve(ref<Store> evalStore, ref<Store> store)
{
auto res = unresolved;
@@ -110,7 +111,7 @@ App UnresolvedApp::resolve(ref<Store> store)
installableContext.push_back(
std::make_shared<InstallableDerivedPath>(store, ctxElt.toDerivedPath()));
auto builtContext = build(store, Realise::Outputs, installableContext);
auto builtContext = build(evalStore, store, Realise::Outputs, installableContext);
res.program = resolveString(*store, unresolved.program, builtContext);
if (!store->isInStore(res.program))
throw Error("app program '%s' is not in the Nix store", res.program);
+4 -1
View File
@@ -52,7 +52,10 @@ struct CmdBuild : InstallablesCommand, MixDryRun, MixJSON, MixProfile
void run(ref<Store> store) override
{
auto buildables = build(store, dryRun ? Realise::Nothing : Realise::Outputs, installables, buildMode);
auto buildables = build(
getEvalStore(), store,
dryRun ? Realise::Nothing : Realise::Outputs,
installables, buildMode);
if (json) logger->cout("%s", derivedPathsWithHintsToJSON(buildables, store).dump());
+1 -1
View File
@@ -69,7 +69,7 @@ struct CmdBundle : InstallableCommand
{
auto evalState = getEvalState();
auto app = installable->toApp(*evalState).resolve(store);
auto app = installable->toApp(*evalState).resolve(getEvalStore(), store);
auto [bundlerFlakeRef, bundlerName] = parseFlakeRefWithFragment(bundler, absPath("."));
const flake::LockFlags lockFlags{ .writeLockFile = false };
+1 -1
View File
@@ -90,7 +90,7 @@ struct CmdCopy : BuiltPathsCommand
}
copyPaths(
srcStore, dstStore, stuffToCopy, NoRepair, checkSigs, substitute);
*srcStore, *dstStore, stuffToCopy, NoRepair, checkSigs, substitute);
}
};
+14 -19
View File
@@ -7,6 +7,7 @@
#include "derivations.hh"
#include "affinity.hh"
#include "progress-bar.hh"
#include "run.hh"
#include <nlohmann/json.hpp>
@@ -171,15 +172,15 @@ const static std::string getEnvSh =
modified derivation with the same dependencies and nearly the same
initial environment variables, that just writes the resulting
environment to a file and exits. */
StorePath getDerivationEnvironment(ref<Store> store, const StorePath & drvPath)
static StorePath getDerivationEnvironment(ref<Store> store, ref<Store> evalStore, const StorePath & drvPath)
{
auto drv = store->derivationFromPath(drvPath);
auto drv = evalStore->derivationFromPath(drvPath);
auto builder = baseNameOf(drv.builder);
if (builder != "bash")
throw Error("'nix develop' only works on derivations that use 'bash' as their builder");
auto getEnvShPath = store->addTextToStore("get-env.sh", getEnvSh, {});
auto getEnvShPath = evalStore->addTextToStore("get-env.sh", getEnvSh, {});
drv.args = {store->printStorePath(getEnvShPath)};
@@ -205,7 +206,7 @@ StorePath getDerivationEnvironment(ref<Store> store, const StorePath & drvPath)
output.second = { .output = DerivationOutputInputAddressed { .path = StorePath::dummy } };
drv.env[output.first] = "";
}
Hash h = std::get<0>(hashDerivationModulo(*store, drv, true));
Hash h = std::get<0>(hashDerivationModulo(*evalStore, drv, true));
for (auto & output : drv.outputs) {
auto outPath = store->makeOutputPath(output.first, h, drv.name);
@@ -214,12 +215,12 @@ StorePath getDerivationEnvironment(ref<Store> store, const StorePath & drvPath)
}
}
auto shellDrvPath = writeDerivation(*store, drv);
auto shellDrvPath = writeDerivation(*evalStore, drv);
/* Build the derivation. */
store->buildPaths({DerivedPath::Built{shellDrvPath}});
store->buildPaths({DerivedPath::Built{shellDrvPath}}, bmNormal, evalStore);
for (auto & [_0, optPath] : store->queryPartialDerivationOutputMap(shellDrvPath)) {
for (auto & [_0, optPath] : evalStore->queryPartialDerivationOutputMap(shellDrvPath)) {
assert(optPath);
auto & outPath = *optPath;
assert(store->isValidPath(outPath));
@@ -307,7 +308,7 @@ struct Common : InstallableCommand, MixProfile
auto dir = absPath(dir_);
auto installable = parseInstallable(store, installable_);
auto builtPaths = toStorePaths(
store, Realise::Nothing, OperateOn::Output, {installable});
getEvalStore(), store, Realise::Nothing, OperateOn::Output, {installable});
for (auto & path: builtPaths) {
auto from = store->printStorePath(path);
if (script.find(from) == std::string::npos)
@@ -347,7 +348,7 @@ struct Common : InstallableCommand, MixProfile
auto & drvPath = *drvs.begin();
return getDerivationEnvironment(store, drvPath);
return getDerivationEnvironment(store, getEvalStore(), drvPath);
}
}
@@ -361,7 +362,7 @@ struct Common : InstallableCommand, MixProfile
debug("reading environment file '%s'", strPath);
return {BuildEnvironment::fromJSON(readFile(strPath)), strPath};
return {BuildEnvironment::fromJSON(readFile(store->toRealPath(shellOutPath))), strPath};
}
};
@@ -472,8 +473,6 @@ struct CmdDevelop : Common, MixEnvironment
writeFull(rcFileFd.get(), script);
stopProgressBar();
setEnviron();
// prevent garbage collection until shell exits
setenv("NIX_GCROOT", gcroot.data(), 1);
@@ -495,8 +494,8 @@ struct CmdDevelop : Common, MixEnvironment
Strings{"legacyPackages." + settings.thisSystem.get() + "."},
nixpkgsLockFlags);
shell = state->store->printStorePath(
toStorePath(state->store, Realise::Outputs, OperateOn::Output, bashInstallable)) + "/bin/bash";
shell = store->printStorePath(
toStorePath(getEvalStore(), store, Realise::Outputs, OperateOn::Output, bashInstallable)) + "/bin/bash";
} catch (Error &) {
ignoreException();
}
@@ -506,11 +505,7 @@ struct CmdDevelop : Common, MixEnvironment
auto args = phase || !command.empty() ? Strings{std::string(baseNameOf(shell)), rcFilePath}
: Strings{std::string(baseNameOf(shell)), "--rcfile", rcFilePath};
restoreProcessContext();
execvp(shell.c_str(), stringsToCharPtrs(args).data());
throw SysError("executing shell '%s'", shell);
runProgramInStore(store, shell, args);
}
};
+2 -2
View File
@@ -131,9 +131,9 @@ struct CmdDiffClosures : SourceExprCommand
void run(ref<Store> store) override
{
auto before = parseInstallable(store, _before);
auto beforePath = toStorePath(store, Realise::Outputs, operateOn, before);
auto beforePath = toStorePath(getEvalStore(), store, Realise::Outputs, operateOn, before);
auto after = parseInstallable(store, _after);
auto afterPath = toStorePath(store, Realise::Outputs, operateOn, after);
auto afterPath = toStorePath(getEvalStore(), store, Realise::Outputs, operateOn, after);
printClosureDiff(store, beforePath, afterPath, "");
}
};
+1 -1
View File
@@ -841,7 +841,7 @@ struct CmdFlakeArchive : FlakeCommand, MixJSON, MixDryRun
if (!dryRun && !dstUri.empty()) {
ref<Store> dstStore = dstUri.empty() ? openStore() : openStore(dstUri);
copyPaths(store, dstStore, sources);
copyPaths(*store, *dstStore, sources);
}
}
};
+1 -1
View File
@@ -253,7 +253,7 @@ struct CmdProfileInstall : InstallablesCommand, MixDefaultProfile
manifest.elements.emplace_back(std::move(element));
} else {
auto buildables = build(store, Realise::Outputs, {installable}, bmNormal);
auto buildables = build(getEvalStore(), store, Realise::Outputs, {installable}, bmNormal);
for (auto & buildable : buildables) {
ProfileElement element;
+35 -36
View File
@@ -1,3 +1,4 @@
#include "run.hh"
#include "command.hh"
#include "common-args.hh"
#include "shared.hh"
@@ -20,45 +21,43 @@ using namespace nix;
std::string chrootHelperName = "__run_in_chroot";
struct RunCommon : virtual Command
namespace nix {
void runProgramInStore(ref<Store> store,
const std::string & program,
const Strings & args)
{
stopProgressBar();
using Command::run;
restoreProcessContext();
void runProgram(ref<Store> store,
const std::string & program,
const Strings & args)
{
stopProgressBar();
/* If this is a diverted store (i.e. its "logical" location
(typically /nix/store) differs from its "physical" location
(e.g. /home/eelco/nix/store), then run the command in a
chroot. For non-root users, this requires running it in new
mount and user namespaces. Unfortunately,
unshare(CLONE_NEWUSER) doesn't work in a multithreaded program
(which "nix" is), so we exec() a single-threaded helper program
(chrootHelper() below) to do the work. */
auto store2 = store.dynamic_pointer_cast<LocalStore>();
restoreProcessContext();
if (store2 && store->storeDir != store2->getRealStoreDir()) {
Strings helperArgs = { chrootHelperName, store->storeDir, store2->getRealStoreDir(), program };
for (auto & arg : args) helperArgs.push_back(arg);
/* If this is a diverted store (i.e. its "logical" location
(typically /nix/store) differs from its "physical" location
(e.g. /home/eelco/nix/store), then run the command in a
chroot. For non-root users, this requires running it in new
mount and user namespaces. Unfortunately,
unshare(CLONE_NEWUSER) doesn't work in a multithreaded
program (which "nix" is), so we exec() a single-threaded
helper program (chrootHelper() below) to do the work. */
auto store2 = store.dynamic_pointer_cast<LocalStore>();
execv(readLink("/proc/self/exe").c_str(), stringsToCharPtrs(helperArgs).data());
if (store2 && store->storeDir != store2->getRealStoreDir()) {
Strings helperArgs = { chrootHelperName, store->storeDir, store2->getRealStoreDir(), program };
for (auto & arg : args) helperArgs.push_back(arg);
execv(readLink("/proc/self/exe").c_str(), stringsToCharPtrs(helperArgs).data());
throw SysError("could not execute chroot helper");
}
execvp(program.c_str(), stringsToCharPtrs(args).data());
throw SysError("unable to execute '%s'", program);
throw SysError("could not execute chroot helper");
}
};
struct CmdShell : InstallablesCommand, RunCommon, MixEnvironment
execvp(program.c_str(), stringsToCharPtrs(args).data());
throw SysError("unable to execute '%s'", program);
}
}
struct CmdShell : InstallablesCommand, MixEnvironment
{
using InstallablesCommand::run;
@@ -93,7 +92,7 @@ struct CmdShell : InstallablesCommand, RunCommon, MixEnvironment
void run(ref<Store> store) override
{
auto outPaths = toStorePaths(store, Realise::Outputs, OperateOn::Output, installables);
auto outPaths = toStorePaths(getEvalStore(), store, Realise::Outputs, OperateOn::Output, installables);
auto accessor = store->getFSAccessor();
@@ -125,13 +124,13 @@ struct CmdShell : InstallablesCommand, RunCommon, MixEnvironment
Strings args;
for (auto & arg : command) args.push_back(arg);
runProgram(store, *command.begin(), args);
runProgramInStore(store, *command.begin(), args);
}
};
static auto rCmdShell = registerCommand<CmdShell>("shell");
struct CmdRun : InstallableCommand, RunCommon
struct CmdRun : InstallableCommand
{
using InstallableCommand::run;
@@ -178,12 +177,12 @@ struct CmdRun : InstallableCommand, RunCommon
{
auto state = getEvalState();
auto app = installable->toApp(*state).resolve(store);
auto app = installable->toApp(*state).resolve(getEvalStore(), store);
Strings allArgs{app.program};
for (auto & i : args) allArgs.push_back(i);
runProgram(store, app.program, allArgs);
runProgramInStore(store, app.program, allArgs);
}
};
+11
View File
@@ -0,0 +1,11 @@
#pragma once
#include "store-api.hh"
namespace nix {
void runProgramInStore(ref<Store> store,
const std::string & program,
const Strings & args);
}
+2 -2
View File
@@ -62,9 +62,9 @@ struct CmdWhyDepends : SourceExprCommand
void run(ref<Store> store) override
{
auto package = parseInstallable(store, _package);
auto packagePath = toStorePath(store, Realise::Outputs, operateOn, package);
auto packagePath = toStorePath(getEvalStore(), store, Realise::Outputs, operateOn, package);
auto dependency = parseInstallable(store, _dependency);
auto dependencyPath = toStorePath(store, Realise::Derivation, operateOn, dependency);
auto dependencyPath = toStorePath(getEvalStore(), store, Realise::Derivation, operateOn, dependency);
auto dependencyPathHash = dependencyPath.hashPart();
StorePathSet closure;
+2
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "“--no-require-sigs” can’t be used with the daemon"
# We can produce drvs directly into the binary cache
clearStore
clearCacheCache
+2 -1
View File
@@ -3,7 +3,8 @@
# Regression test for https://github.com/NixOS/nix/issues/4858
source common.sh
sed -i 's/experimental-features .*/& ca-derivations ca-references/' "$NIX_CONF_DIR"/nix.conf
requireDaemonNewerThan "2.4pre20210621"
# Get the output path of `rootCA`, and put some garbage instead
outPath="$(nix-build ./content-addressed.nix -A rootCA --no-out-link)"
+4
View File
@@ -1 +1,5 @@
source ../common.sh
sed -i 's/experimental-features .*/& ca-derivations ca-references/' "$NIX_CONF_DIR"/nix.conf
restartDaemon
+1 -1
View File
@@ -5,7 +5,7 @@
source common.sh
sed -i 's/experimental-features .*/& ca-derivations ca-references/' "$NIX_CONF_DIR"/nix.conf
buggyNeedLocalStore "For some reason, this deadlocks with the daemon"
export NIX_TESTS_CA_BY_DEFAULT=1
@@ -1,5 +1,7 @@
source ./common.sh
requireDaemonNewerThan "2.4pre20210625"
sed -i 's/experimental-features .*/& ca-derivations ca-references/' "$NIX_CONF_DIR"/nix.conf
export REMOTE_STORE_DIR="$TEST_ROOT/remote_store"
-2
View File
@@ -4,8 +4,6 @@
source common.sh
sed -i 's/experimental-features .*/& ca-derivations ca-references/' "$NIX_CONF_DIR"/nix.conf
export NIX_TESTS_CA_BY_DEFAULT=1
cd ..
+2
View File
@@ -2,6 +2,8 @@
source common.sh
requireDaemonNewerThan "2.4pre20210626"
sed -i 's/experimental-features .*/& ca-derivations ca-references nix-command flakes/' "$NIX_CONF_DIR"/nix.conf
export NIX_TESTS_CA_BY_DEFAULT=1
+2
View File
@@ -2,6 +2,8 @@
source common.sh
requireDaemonNewerThan "2.4pre20210623"
sed -i 's/experimental-features .*/& ca-derivations ca-references nix-command flakes/' "$NIX_CONF_DIR"/nix.conf
export NIX_TESTS_CA_BY_DEFAULT=1
+1 -1
View File
@@ -4,7 +4,7 @@
source common.sh
sed -i 's/experimental-features .*/& ca-derivations ca-references/' "$NIX_CONF_DIR"/nix.conf
needLocalStore "“--no-require-sigs” can’t be used with the daemon"
rm -rf $TEST_ROOT/binary_cache
+3
View File
@@ -1,5 +1,8 @@
source common.sh
# XXX: This shouldn’t be, but #4813 cause this test to fail
buggyNeedLocalStore "see #4813"
checkBuildTempDirRemoved ()
{
buildDir=$(sed -n 's/CHECK_TMPDIR=//p' $1 | head -1)
+57 -4
View File
@@ -1,5 +1,9 @@
set -e
if [[ -z "$COMMON_SH_SOURCED" ]]; then
COMMON_SH_SOURCED=1
export TEST_ROOT=$(realpath ${TMPDIR:-/tmp}/nix-test)/${TEST_NAME:-default}
export NIX_STORE_DIR
if ! NIX_STORE_DIR=$(readlink -f $TEST_ROOT/store 2> /dev/null); then
@@ -45,6 +49,9 @@ export busybox="@sandbox_shell@"
export version=@PACKAGE_VERSION@
export system=@system@
export IMPURE_VAR1=foo
export IMPURE_VAR2=bar
cacheDir=$TEST_ROOT/binary-cache
readLink() {
@@ -75,29 +82,57 @@ clearCacheCache() {
}
startDaemon() {
# Don’t start the daemon twice, as this would just make it loop indefinitely
if [[ "$NIX_REMOTE" == daemon ]]; then
return
fi
# Start the daemon, wait for the socket to appear. !!!
# ‘nix-daemon’ should have an option to fork into the background.
rm -f $NIX_STATE_DIR/daemon-socket/socket
rm -f $NIX_DAEMON_SOCKET_PATH
${NIX_DAEMON_COMMAND:-nix daemon} &
for ((i = 0; i < 30; i++)); do
if [ -e $NIX_DAEMON_SOCKET_PATH ]; then break; fi
if [[ -S $NIX_DAEMON_SOCKET_PATH ]]; then break; fi
sleep 1
done
pidDaemon=$!
trap "kill -9 $pidDaemon" EXIT
trap "killDaemon" EXIT
export NIX_REMOTE=daemon
}
killDaemon() {
kill -9 $pidDaemon
kill $pidDaemon
for i in {0.10}; do
kill -0 $pidDaemon || break
sleep 1
done
kill -9 $pidDaemon || true
wait $pidDaemon || true
trap "" EXIT
}
restartDaemon() {
[[ -z "${pidDaemon:-}" ]] && return 0
killDaemon
unset NIX_REMOTE
startDaemon
}
if [[ $(uname) == Linux ]] && [[ -L /proc/self/ns/user ]] && unshare --user true; then
_canUseSandbox=1
fi
isDaemonNewer () {
[[ -n "${NIX_DAEMON_PACKAGE:-}" ]] || return 0
local requiredVersion="$1"
local daemonVersion=$($NIX_DAEMON_PACKAGE/bin/nix-daemon --version | cut -d' ' -f3)
return [[ $(nix eval --expr "builtins.compareVersions ''$daemonVersion'' ''2.4''") -ge 0 ]]
}
requireDaemonNewerThan () {
isDaemonNewer "$1" || exit 99
}
canUseSandbox() {
if [[ ! $_canUseSandbox ]]; then
echo "Sandboxing not supported, skipping this test..."
@@ -123,4 +158,22 @@ expect() {
[[ $res -eq $expected ]]
}
needLocalStore() {
if [[ "$NIX_REMOTE" == "daemon" ]]; then
echo "Can’t run through the daemon ($1), skipping this test..."
return 99
fi
}
# Just to make it easy to find which tests should be fixed
buggyNeedLocalStore () {
needLocalStore
}
set -x
if [[ -n "${NIX_DAEMON_PACKAGE:-}" ]]; then
startDaemon
fi
fi # COMMON_SH_SOURCED
+4 -1
View File
@@ -3,11 +3,14 @@
# Only run this if we have an older Nix available
# XXX: This assumes that the `daemon` package is older than the `client` one
if [[ -z "$NIX_DAEMON_PACKAGE" ]]; then
exit 0
exit 99
fi
source common.sh
killDaemon
unset NIX_REMOTE
# Fill the db using the older Nix
PATH_WITH_NEW_NIX="$PATH"
export PATH="$NIX_DAEMON_PACKAGE/bin:$PATH"
+2
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "--dump-db requires a local store"
clearStore
path=$(nix-build dependencies.nix -o $TEST_ROOT/result)
+28
View File
@@ -0,0 +1,28 @@
source common.sh
requireDaemonNewerThan "2.4pre20210727"
eval_store=$TEST_ROOT/eval-store
clearStore
rm -rf "$eval_store"
nix build -f dependencies.nix --eval-store "$eval_store" -o "$TEST_ROOT/result"
[[ -e $TEST_ROOT/result/foobar ]]
(! ls $NIX_STORE_DIR/*.drv)
ls $eval_store/nix/store/*.drv
clearStore
rm -rf "$eval_store"
nix-instantiate dependencies.nix --eval-store "$eval_store"
(! ls $NIX_STORE_DIR/*.drv)
ls $eval_store/nix/store/*.drv
clearStore
rm -rf "$eval_store"
nix-build dependencies.nix --eval-store "$eval_store" -o "$TEST_ROOT/result"
[[ -e $TEST_ROOT/result/foobar ]]
(! ls $NIX_STORE_DIR/*.drv)
ls $eval_store/nix/store/*.drv
-3
View File
@@ -2,9 +2,6 @@ source common.sh
clearStore
export IMPURE_VAR1=foo
export IMPURE_VAR2=bar
path=$(nix-store -q $(nix-instantiate fixed.nix -A good.0))
echo 'testing bad...'
+2
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "“min-free” and “max-free” are daemon options"
clearStore
garbage1=$(nix store add-path --name garbage1 ./nar-access.sh)
+1
View File
@@ -23,6 +23,7 @@ substituters =
flake-registry = $TEST_ROOT/registry.json
show-trace = true
include nix.conf.extra
trusted-users = $(whoami)
EOF
cat > "$NIX_CONF_DIR"/nix.conf.extra <<EOF
+2
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "the sandbox only runs on the builder side, so it makes no sense to test it with the daemon"
clearStore
if ! canUseSandbox; then exit 99; fi
+2 -1
View File
@@ -56,7 +56,8 @@ nix_tests = \
ca/nix-run.sh \
ca/recursive.sh \
ca/concurrent-builds.sh \
ca/nix-copy.sh
ca/nix-copy.sh \
eval-store.sh
# parallel.sh
install-tests += $(foreach x, $(nix_tests), tests/$(x))
+1 -1
View File
@@ -58,7 +58,7 @@ outPath2=$(nix-build $(nix-instantiate multiple-outputs.nix -A a.second) --no-ou
# Delete one of the outputs and rebuild it. This will cause a hash
# rewrite.
nix store delete $TEST_ROOT/result-second --ignore-liveness
env -u NIX_REMOTE nix store delete $TEST_ROOT/result-second --ignore-liveness
nix-build multiple-outputs.nix -A a.all -o $TEST_ROOT/result
[ "$(cat $TEST_ROOT/result-second/file)" = "second" ]
[ "$(cat $TEST_ROOT/result-second/link/file)" = "first" ]
+4
View File
@@ -54,6 +54,10 @@ nix-instantiate shell.nix -A shellDrv --add-root $TEST_ROOT/shell
output=$(NIX_PATH=nixpkgs=shell.nix nix-shell --pure -p foo bar --run 'echo "$(foo) $(bar)"')
[ "$output" = "foo bar" ]
# Test nix-shell -p --arg x y
output=$(NIX_PATH=nixpkgs=shell.nix nix-shell --pure -p foo --argstr fooContents baz --run 'echo "$(foo)"')
[ "$output" = "baz" ]
# Test nix-shell shebang mode
sed -e "s|@ENV_PROG@|$(type -P env)|" shell.shebang.sh > $TEST_ROOT/shell.shebang.sh
chmod a+rx $TEST_ROOT/shell.shebang.sh
+2 -1
View File
@@ -26,7 +26,8 @@ if [ "$inode1" = "$inode3" ]; then
exit 1
fi
nix-store --optimise
# XXX: This should work through the daemon too
NIX_REMOTE="" nix-store --optimise
inode1="$(stat --format=%i $outPath1/foo)"
inode3="$(stat --format=%i $outPath3/foo)"
+3
View File
@@ -5,6 +5,9 @@ clearStore
rm -f $TEST_ROOT/result
export REMOTE_STORE=file:$TEST_ROOT/remote_store
echo 'require-sigs = false' >> $NIX_CONF_DIR/nix.conf
restartDaemon
# Build the dependencies and push them to the remote store
nix-build -o $TEST_ROOT/result dependencies.nix --post-build-hook $PWD/push-to-store.sh
+2
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "uses some low-level store manipulations that aren’t available through the daemon"
clearStore
max=500
+2
View File
@@ -1,5 +1,7 @@
source common.sh
needLocalStore "--repair needs a local store"
clearStore
path=$(nix-build dependencies.nix -o $TEST_ROOT/result)
+2 -2
View File
@@ -1,4 +1,4 @@
{ inNixShell ? false, contentAddressed ? false }:
{ inNixShell ? false, contentAddressed ? false, fooContents ? "foo" }:
let cfg = import ./config.nix; in
with cfg;
@@ -62,7 +62,7 @@ let pkgs = rec {
foo = runCommand "foo" {} ''
mkdir -p $out/bin
echo 'echo foo' > $out/bin/foo
echo 'echo ${fooContents}' > $out/bin/foo
chmod a+rx $out/bin/foo
ln -s ${shell} $out/bin/bash
'';
+4
View File
@@ -1,5 +1,9 @@
source common.sh
# 27ce722638 required some incompatible changes to the nix file, so skip this
# tests for the older versions
requireDaemonNewerThan "2.4pre20210622"
clearStore
rm -f $TEST_ROOT/result
+2
View File
@@ -2,6 +2,8 @@
source common.sh
# XXX: This shouldn’t be, but #4813 cause this test to fail
needLocalStore "see #4813"
set +e
messages=$(nix-build -Q timeout.nix -A infiniteLoop --timeout 2 2>&1)