Compare commits

..
Author SHA1 Message Date
Domen Kožar e7763e4576 release.nix: add tarballs to release critical 2017-07-13 14:58:15 +02:00
Domen Kožar 9f29ddd07c release.nix: Use only one version of tarball 2017-07-13 14:56:59 +02:00
Domen Kožar af10cd93a5 release.nix: respect top-level nixpkgs input 2017-07-12 16:37:17 +02:00
Domen Kožar 555f283064 Build tarball per system
This catches any build failures that otherwise users might encounter.

For example libseccomp package doesn't build on darwin.
2017-07-12 16:31:31 +02:00
Eelco Dolstra 1888f7889b macOS: Ugly hack to make the tests succeed
Sandboxes cannot be nested, so if Nix's build runs inside a sandbox,
it cannot use a sandbox itself. I don't see a clean way to detect
whether we're in a sandbox, so use a test-specific hack.

https://github.com/NixOS/nix/issues/1413
2017-06-19 14:26:05 +02:00
Eelco Dolstra b5bdfdef73 macOS: Remove flags
In particular, UF_IMMUTABLE (uchg) needs to be cleared to allow the
path to be garbage-collected or optimised.

See https://github.com/NixOS/nixpkgs/issues/25819.
+       the file from being garbage-collected.
2017-06-19 14:19:21 +02:00
Eelco Dolstra a10951de08 OS X -> macOS
(cherry picked from commit c20641ce56)
2017-06-14 23:28:06 +02:00
Eelco Dolstra 1dcadadf74 Add 1.11.10 release notes
(cherry picked from commit 0fb60e4e0f)
2017-06-14 23:27:52 +02:00
Eelco Dolstra 38b7d55af1 Remove redundant debug line 2017-06-14 13:45:38 +02:00
Eelco Dolstra 88b291ffc4 canonicalisePathMetaData(): Ignore security.selinux attribute
Untested, hopefully fixes #1406.
2017-06-14 11:41:03 +02:00
Eelco Dolstra 177f3996e2 Suppress spurious "killing process N: Operation not permitted" on macOS 2017-06-12 18:34:48 +02:00
Eelco Dolstra 25230a17a9 On macOS, don't use /var/folders for TMPDIR
This broke "nix-store --serve".
2017-06-12 17:43:19 +02:00
Eelco Dolstra 847f19a5f7 Provide a builtin default for $NIX_SSL_CERT_FILE
This is mostly to ensure that when Nix is started on macOS via a
launchd service or sshd (for a remote build), it gets a certificate
bundle.
2017-06-12 16:44:43 +02:00
Eelco Dolstra 7f5b750b40 Don't run pre-build-hook if we don't have a derivation
This fixes a build failure on OS X when using Hydra or Nix 1.12's
build-remote (since they don't copy the derivation to the build
machine).
2017-06-12 16:07:34 +02:00
Eelco Dolstra 186571965d Don't show flags from config settings in "nix --help" 2017-06-07 18:41:20 +02:00
Eelco Dolstra aa952d5f0b nix: Add --help-config flag 2017-06-07 16:49:54 +02:00
Eelco Dolstra b8283773bd nix: Make all options available as flags
Thus, instead of ‘--option <name> <value>’, you can write ‘--<name>
<value>’. So

  --option http-connections 100

becomes

  --http-connections 100

Apart from brevity, the difference is that it's not an error to set a
non-existent option via --option, but unrecognized arguments are
fatal.

Boolean options have special treatment: they're mapped to the
argument-less flags ‘--<name>’ and ‘--no-<name>’. E.g.

  --option auto-optimise-store false

becomes

  --no-auto-optimise-store
2017-06-07 16:17:17 +02:00
Eelco Dolstra c8cc50d46e Disable the build user mechanism on all platforms except Linux and OS X 2017-06-06 18:52:15 +02:00
Eelco Dolstra 85e93d7b87 Always use the Darwin sandbox
Even with "build-use-sandbox = false", we now use sandboxing with a
permissive profile that allows everything except the creation of
setuid/setgid binaries.
2017-06-06 18:44:49 +02:00
Eelco Dolstra d3f780996c Merge pull request #1399 from Mic92/master
Add .editorconfig
2017-06-06 14:13:36 +02:00
Jörg Thalheim 88acb64610 Add .editorconfig
- Automatically adjust editor to nix coding style
-> less nitpiks/styling issues in pull requests
-> profit(!)

see also nixpkgs' editorconfig: https://github.com/NixOS/nixpkgs/blob/master/.editorconfig
2017-06-05 22:57:28 +01:00
Eelco Dolstra b4b1f4525f Fix coverage job 2017-06-01 14:43:15 +02:00
Eelco Dolstra ab5834f7a1 RPM, Deb: Add dependency on libseccomp 2017-06-01 14:28:21 +02:00
Eelco Dolstra 52fec8dde8 Remove listxattr assertion
It appears that sometimes, listxattr() returns a different value for
the query case (i.e. when the buffer size is 0).
2017-05-31 20:43:47 +02:00
Eelco Dolstra c96e8cd097 OS X sandbox: Improve builtin sandbox profile
Also, add rules to allow fixed-output derivations to access the
network.

These rules are sufficient to build stdenvDarwin without any
__sandboxProfile magic.
2017-05-31 17:25:51 +02:00
Eelco Dolstra 5ea8161b55 resolve-system-dependencies: Misc fixes
This fixes

  Could not find any mach64 blobs in file ‘/usr/lib/libSystem.B.dylib’, continuing...
2017-05-31 16:10:10 +02:00
Eelco Dolstra c368e079ca resolve-system-dependencies: Simplify 2017-05-31 15:34:03 +02:00
Eelco Dolstra 44f3f8048f OS X sandbox: Don't use a deterministic $TMPDIR
This doesn't work because the OS X sandbox cannot bind-mount
path to a different location.
2017-05-31 14:09:00 +02:00
Eelco Dolstra c740c3ce50 OS X sandbox: Store .sb file in $TMPDIR rather than the Nix store
The filename used was not unique and owned by the build user, so
builds could fail with

error: while setting up the build environment: cannot unlink ‘/nix/store/99i210ihnsjacajaw8r33fmgjvzpg6nr-bison-3.0.4.drv.sb’: Permission denied
2017-05-31 14:09:00 +02:00
Eelco Dolstra 683a499ebb resolve-system-dependencies: Fix another segfault
runResolver() was barfing on directories like
/System/Library/Frameworks/Security.framework/Versions/Current/PlugIns. It
should probably do something sophisticated for frameworks, but let's
ignore them for now.
2017-05-30 20:39:40 +02:00
Eelco Dolstra acc889c821 Darwin sandbox: Use sandbox-defaults.sb
Issue #759.

Also, remove nix.conf from the sandbox since I don't really see a
legitimate reason for builders to access the Nix configuration.
2017-05-30 17:40:12 +02:00
Eelco Dolstra 53a1644187 Darwin sandbox: Disallow creating setuid/setgid binaries
Suggested by Daiderd Jordan.
2017-05-30 17:17:17 +02:00
Eelco Dolstra 83eec5a997 resolve-system-dependencies: Several fixes
This fixes

  error: getting attributes of path ‘Versions/Current/CoreFoundation’: No such file or directory

when /System/Library/Frameworks/CoreFoundation.framework/CoreFoundation is a symlink.

Also fixes a segfault when encounting a file that is not a MACH binary (such
as /dev/null, which is included in __impureHostDeps in Nixpkgs).

Possibly fixes #786.
2017-05-30 16:03:37 +02:00
Eelco Dolstra 9bdb88ea6e Only pass --with-sandbox-shell on Linux 2017-05-30 15:56:15 +02:00
Eelco Dolstra fe08d17934 Fix seccomp build failure on clang
Fixes

  src/libstore/build.cc:2321:45: error: non-constant-expression cannot be narrowed from type 'int' to 'scmp_datum_t' (aka 'unsigned long') in initializer list [-Wc++11-narrowing]
2017-05-30 14:37:24 +02:00
Eelco Dolstra d552d38758 Shut up some clang warnings 2017-05-30 14:35:50 +02:00
Eelco Dolstra 2ac99a32da Add a seccomp rule to disallow setxattr() 2017-05-30 13:59:24 +02:00
Eelco Dolstra d798349ede canonicalisePathMetaData(): Remove extended attributes / ACLs
EAs/ACLs are not part of the NAR canonicalisation. Worse, setting an
ACL allows a builder to create writable files in the Nix store. So get
rid of them.

Closes #185.
2017-05-30 13:47:41 +02:00
Eelco Dolstra ff6becafa8 Require seccomp only in multi-user setups 2017-05-30 12:37:04 +02:00
Eelco Dolstra 1d9ab273ba Add test for setuid seccomp filter 2017-05-29 16:14:10 +02:00
Eelco Dolstra cf93397d3f Fix seccomp initialisation on i686-linux 2017-05-29 16:14:10 +02:00
Eelco Dolstra 6cc6c15a2d Add a seccomp filter to prevent creating setuid/setgid binaries
This prevents builders from setting the S_ISUID or S_ISGID bits,
preventing users from using a nixbld* user to create a setuid/setgid
binary to interfere with subsequent builds under the same nixbld* uid.

This is based on aszlig's seccomp code
(47f587700d).

Reported by Linus Heckemann.
2017-05-29 16:14:10 +02:00
Eelco Dolstra 6e01ecd112 Fix nix-copy-closure test
Fixes

  client# error: size mismatch importing path ‘/nix/store/ywf5fihjlxwijm6ygh6s0a353b5yvq4d-libidn2-0.16’; expected 0, got 120264

This is mostly an artifact of the NixOS VM test environment, where the
Nix database doesn't contain hashes/sizes.

http://hydra.nixos.org/build/53537471
2017-05-29 16:08:56 +02:00
Eelco Dolstra 588dad4084 Fix build failure on Debian/Ubuntu
http://hydra.nixos.org/build/53537463
2017-05-29 15:59:18 +02:00
Eelco Dolstra 63145be2a5 Fix typo 2017-05-29 15:52:36 +02:00
Eelco Dolstra d6064dd19b Merge pull request #1393 from pyrtsa/patch-1
Fix variable name typo in derivations doc
2017-05-29 11:32:06 +02:00
Eelco Dolstra 2740a22d2c Merge pull request #1394 from pyrtsa/patch-2
Remove stray `>` in builtins doc
2017-05-29 11:31:40 +02:00
Pyry Jahkola 370428f86d Remove stray > in builtins doc 2017-05-28 20:47:35 +03:00
Pyry Jahkola 86ea7d1566 Fix variable name typo in derivations doc 2017-05-28 15:48:57 +03:00
Eelco Dolstra a7e55151a8 Fix #1314
Also, make nix-shell respect --option. (Previously it only passed it
along to nix-instantiate and nix-build.)
2017-05-24 11:33:42 +02:00
Eelco Dolstra fbe9fe0e75 Merge branch 'topic/cores-master' of https://github.com/neilmayhew/nix 2017-05-24 11:28:36 +02:00
Eelco Dolstra 9e43a4a041 Merge pull request #1376 from Mic92/patch-1
nix-profile.sh: remove sbin from PATH
2017-05-24 11:25:32 +02:00
Eelco Dolstra 9711524188 Fix #1380
It lacked a backslash. Use a raw string and single quotes around PS1
to simplify this.
2017-05-24 11:23:28 +02:00
Eelco Dolstra 01200d07d2 Merge branch 'prompt-terminator' of https://github.com/lheckemann/nix 2017-05-24 11:22:48 +02:00
Eelco Dolstra b303ad012c Merge pull request #1382 from FRidh/patch-1
Document fetchTarball can take a sha256
2017-05-24 11:07:36 +02:00
Eelco Dolstra edbb105e98 Merge branch 'nar-accessor-tree' of https://github.com/bennofs/nix 2017-05-24 11:04:43 +02:00
Eelco Dolstra f134fc4cbe Document that builtins.match takes a POSIX extended RE 2017-05-17 12:18:18 +02:00
Eelco Dolstra e46090edb1 builtins.match: Improve error message for bad regular expression
Issue #1331.
2017-05-17 11:58:01 +02:00
Eelco Dolstra b01d62285c Improve progress indicator 2017-05-16 16:09:57 +02:00
Benno Fünfstück a1f428b13b nar-accessor.cc: remove unused member NarIndexer::currentName 2017-05-15 19:41:59 +02:00
Benno Fünfstück 5ee06e612a nar-accessor: non-recursive NarMember::find
This avoids a possible stack overflow if directories are very deeply nested.
2017-05-15 19:34:18 +02:00
Eelco Dolstra e80257f122 Simplify fixed-output check 2017-05-15 18:50:54 +02:00
Eelco Dolstra c05d9ae7a5 Disallow outputHash being null or an empty string
Fixes #1384.
2017-05-15 18:47:13 +02:00
Eelco Dolstra a2d92bb20e Add --with-sandbox-shell configure flag
And add a 116 KiB ash shell from busybox to the release build. This
helps to make sandbox builds work out of the box on non-NixOS systems
and with diverted stores.
2017-05-15 17:36:32 +02:00
Eelco Dolstra b30f5784d0 Linux sandbox: Don't barf on invalid paths
This is useful when we're using a diverted store (e.g. "--store
local?root=/tmp/nix") in conjunction with a statically-linked sh from
the host store (e.g. "sandbox-paths =/bin/sh=/nix/store/.../bin/busybox").
2017-05-15 17:36:32 +02:00
Eelco Dolstra 2b761d5f50 Make fmt() non-recursive 2017-05-15 17:36:32 +02:00
Benno Fünfstück 4412f7c083 nar-archive.cc: add tests for the nar index 2017-05-15 12:23:21 +02:00
Eelco Dolstra 0124d118ef Merge pull request #1387 from bennofs/nix-ls-slash
nix ls: support '/' for the root directory
2017-05-15 11:02:14 +02:00
Benno Fünfstück 06880d7ed8 nix ls: support '/' for the root directory 2017-05-15 10:25:55 +02:00
Benno Fünfstück 75a1d9849d nar-accessor: use tree, fixes readDirectory missing children
Previously, if a directory `foo` existed and a file `foo-` (where `-` is any character that is sorted before `/`), then  `readDirectory` would return an empty list.

To fix this, we now use a tree where we can just access the children of the node, and do not need to rely on sorting behavior to list the contents of a directory.
2017-05-15 10:23:16 +02:00
Eelco Dolstra 510bc1735b Add an option for extending the user agent header
This is useful e.g. for distinguishing traffic to a binary cache
(e.g. certain machines can use a different tag in the user agent).
2017-05-11 15:10:22 +02:00
Eelco Dolstra 62d476c7ee Fix typo 2017-05-11 14:02:03 +02:00
Eelco Dolstra ea65ae0f9c Tweak error message 2017-05-11 13:59:47 +02:00
Eelco Dolstra 1a8e15053a Don't allow untrusted users to set info.ultimate
Note that a trusted signature was still required in this case so it
was not a huge deal.
2017-05-11 13:58:09 +02:00
Frederik Rietdijk 2b2de5ef6a Document fetchTarball can take a sha256
Note that I refer to `nix-prefetch-url`.
2017-05-11 13:38:13 +02:00
Eelco Dolstra 6f245bf24a Change the meaning of info.ultimate
It now means "paths that were built locally". It no longer includes
paths that were added locally. For those we don't need info.ultimate,
since we have the content-addressability assertion (info.ca).
2017-05-11 13:31:23 +02:00
Eelco Dolstra 45d7b1a9e9 LocalStore::addToStore(): Check info.narSize
It allowed the client to specify bogus narSize values. In particular,
Downloader::downloadCached wasn't setting narSize at all.
2017-05-11 13:26:03 +02:00
Eelco Dolstra 1fd59447d5 Typo 2017-05-10 18:38:17 +02:00
Eelco Dolstra c5f23f10a8 Replace readline by linenoise
Using linenoise avoids a license compatibility issue (#1356), is a lot
smaller and doesn't pull in ncurses.
2017-05-10 18:37:42 +02:00
Linus Heckemann d48edcc3a5 nix-shell: use appropriate prompt terminator
If running nix-shell as root, the terminator should be # and not $.
2017-05-10 12:03:45 +01:00
Domen Kožar 82a9c93c7f doc: builtins.attrNames returns alphabetically sorted list 2017-05-10 11:23:12 +02:00
Eelco Dolstra 03ae5e6459 Add "nix edit" command
This is a little convenience command that opens the Nix expression of
the specified package. For example,

  nix edit nixpkgs.perlPackages.Moose

opens <nixpkgs/pkgs/top-level/perl-packages.nix> in $EDITOR (at the
right line number for some editors).

This requires the package to have a meta.position attribute.
2017-05-08 18:42:30 +02:00
Eelco Dolstra 7689181e4f Minor cleanup 2017-05-08 15:56:52 +02:00
Eelco Dolstra 00b286275c Linux sandbox: Fix compatibility with older kernels 2017-05-08 15:42:59 +02:00
Eelco Dolstra ebfceeb333 build-remote: Check remote build status 2017-05-08 14:27:12 +02:00
Eelco Dolstra 0a97eb6bd7 Remove superfluous #ifdef 2017-05-08 11:27:20 +02:00
Jörg Thalheim 542fe0d8f3 nix-profile.sh: remove sbin from PATH
sbin is a symlink to bin. 
profiles only contains packages, which have this symlink. 
It is a subset of bin.

related to https://github.com/NixOS/nixpkgs/pull/25550
2017-05-07 07:41:19 +01:00
Eelco Dolstra bb50c89319 Make the location of the build directory in the sandbox configurable
This is mostly for use in the sandbox tests, since if the Nix store is
under /build, then we can't use /build as the build directory.
2017-05-05 17:45:22 +02:00
Eelco Dolstra 465cb68244 Figure out the user's home directory if $HOME is not set 2017-05-05 17:08:23 +02:00
Eelco Dolstra eba840c8a1 Linux sandbox: Use /build instead of /tmp as $TMPDIR
There is a security issue when a build accidentally stores its $TMPDIR
in some critical place, such as an RPATH. If
TMPDIR=/tmp/nix-build-..., then any user on the system can recreate
that directory and inject libraries into the RPATH of programs
executed by other users. Since /build probably doesn't exist (or isn't
world-writable), this mitigates the issue.
2017-05-04 16:57:03 +02:00
Eelco Dolstra 2da6a42448 nix dump-path: Add
This is primarily useful for extracting NARs from other stores (like
binary caches), which "nix-store --dump" cannot do.
2017-05-04 14:21:22 +02:00
Eelco Dolstra 44309c5067 Fix Ubuntu 16.10 build
http://hydra.nixos.org/build/52420073
2017-05-03 18:30:47 +02:00
Eelco Dolstra 493d4bd949 perl-bindings: Remove unused --with-store-dir flag 2017-05-03 17:43:52 +02:00
Eelco Dolstra 72fb2a7edc Fix build on gcc 4.9
http://hydra.nixos.org/build/52408843
2017-05-03 16:08:48 +02:00
Eelco Dolstra 08355643ab nix-shell: Implement passAsFile 2017-05-03 15:01:15 +02:00
Eelco Dolstra 782c0bff45 nix eval: Add a --raw flag
Similar to "jq -r", this prints the evaluation result (which must be a
string value) unquoted.
2017-05-03 14:08:18 +02:00
Eelco Dolstra d3dcdfa006 Fix perlBindings.x86_64-darwin
http://hydra.nixos.org/build/52401151
2017-05-03 11:30:22 +02:00
Eelco Dolstra d3449b286b Merge pull request #1371 from regnat/doc_--xml_fix
fix the description of --xml and --json
2017-05-03 11:06:12 +02:00
regnat a786d26dc2 doc: fix the description of --xml and --json
Those options seem to only apply with --eval and not with --parse.
2017-05-03 10:49:34 +02:00
Eelco Dolstra 16535552ad build-remote: Add a basic test
This only runs on Linux because it requires a diverted store (which
uses mount/user namespaces).
2017-05-02 15:46:10 +02:00
Eelco Dolstra cef8c169b1 Fix "nix ... --all"
When "--all" is used, we should not fill in a default installable.
2017-05-02 15:46:10 +02:00
Eelco Dolstra 7dedd3fa24 Add a test for diverted stores 2017-05-02 15:46:09 +02:00
Eelco Dolstra c5bea16611 LocalStoreAccessor: Fix handling of diverted stores 2017-05-02 15:46:09 +02:00
Eelco Dolstra 7f6837a0f6 Replace $NIX_REMOTE_SYSTEMS with an option "builder-files"
Also, to unify with hydra-queue-runner, allow it to be a list of
files.
2017-05-02 15:46:09 +02:00
Eelco Dolstra cd4d2705ec build-remote: Fix fallback to other machines when connecting fails
Opening an SSHStore or LegacySSHStore does not actually establish a
connection, so the try/catch block here did nothing. Added a
Store::connect() method to test whether a connection can be
established.
2017-05-02 15:46:09 +02:00
Eelco Dolstra 1a68710d4d Add an option for specifying remote builders
This is useful for one-off situations where you want to specify a
builder on the command line instead of having to mess with
nix.machines. E.g.

  $ nix-build -A hello --argstr system x86_64-darwin \
    --option builders 'root@macstadium1 x86_64-darwin'

will perform the specified build on "macstadium1".

It also removes the need for a separate nix.machines file since you
can specify builders in nix.conf directly. (In fact nix.machines is
yet another hack that predates the general nix.conf configuration
file, IIRC.)

Note: this option is supported by the daemon for trusted users. The
fact that this allows trusted users to specify paths to SSH keys to
which they don't normally have access is maybe a bit too much trust...
2017-05-02 15:42:43 +02:00
Eelco Dolstra ebc9f36a81 Factor out machines.conf parsing
This allows hydra-queue-runner to use it.
2017-05-02 13:17:37 +02:00
Eelco Dolstra 174b68a2a2 build-hook: If there are no machines defined, quit permanently 2017-05-02 12:16:29 +02:00
Eelco Dolstra 70581b6363 Fix build hook test 2017-05-02 12:07:56 +02:00
Eelco Dolstra feefcb3a98 build-remote: Ugly hackery to get build logs to work
The build hook mechanism expects build log output to go to file
descriptor 4, so do that.
2017-05-02 12:02:23 +02:00
Eelco Dolstra 3a5f04f48c build-remote: Don't require signatures
This restores the old behaviour.
2017-05-01 20:03:25 +02:00
Eelco Dolstra 031d70e500 Support arbitrary store URIs in nix.machines
For backwards compatibility, if the URI is just a hostname, ssh://
(i.e. LegacySSHStore) is prepended automatically.

Also, all fields except the URI are now optional. For example, this is
a valid nix.machines file:

  local?root=/tmp/nix

This is useful for testing the remote build machinery since you don't
have to mess around with ssh.
2017-05-01 17:35:30 +02:00
Eelco Dolstra 3e4bdfedee Minor cleanup 2017-05-01 17:30:17 +02:00
Eelco Dolstra deac171925 Implement LegacySSHStore::buildDerivation()
This makes LegacySSHStore usable by build-remote and
hydra-queue-runner.
2017-05-01 17:30:16 +02:00
Eelco Dolstra 3f5b98e65a Chomp log output from the build hook 2017-05-01 17:30:16 +02:00
Eelco Dolstra d7653dfc6d Remove $NIX_BUILD_HOOK and $NIX_CURRENT_LOAD
This is to simplify remote build configuration. These environment
variables predate nix.conf.

The build hook now has a sensible default (namely build-remote).

The current load is kept in the Nix state directory now.
2017-05-01 17:30:16 +02:00
Eelco Dolstra ca9f589a93 build-remote: Don't copy the .drv closure
Since build-remote uses buildDerivation() now, we don't need to copy
the .drv file anymore. This greatly reduces the set of input paths
copied to the remote side (e.g. from 392 to 51 store paths for GNU
hello on x86_64-darwin).
2017-05-01 17:30:16 +02:00
Eelco Dolstra b986c7f8b1 Pass verbosity level to build hook 2017-05-01 14:43:14 +02:00
Eelco Dolstra 227a48f86f Reduce severity of EMLINK warnings
Fixes #1357.
2017-05-01 14:36:56 +02:00
Eelco Dolstra 0dddcf867a Add a dummy Store::buildPaths() method
This default implementation of buildPaths() does nothing if all
requested paths are already valid, and throws an "unsupported
operation" error otherwise. This fixes a regression introduced by
c30330df6f in binary cache and legacy
SSH stores.
2017-05-01 13:43:34 +02:00
Eelco Dolstra b0b81b7500 Merge pull request #1366 from Mic92/fix-nix-daemon-service
nix-daemon.service: fix startup
2017-05-01 11:16:56 +02:00
Eelco Dolstra 8b039ba74f Merge branch 'remove-catchall' of https://github.com/layus/nix 2017-05-01 11:16:04 +02:00
Guillaume Maudoux a143014d73 lexer: remove catch-all rules hiding real errors
With catch-all rules, we hide potential errors.
It turns out that a4744254 made one cath-all useless. Flex detected that
is was impossible to reach.
The other is more subtle, as it can only trigger on unfinished escapes
in unfinished strings, which only occurs at EOF.
2017-05-01 01:18:06 +02:00
Guillaume Maudoux a474425425 Fix lexer to support $' in multiline strings. 2017-05-01 01:15:40 +02:00
Jörg Thalheim b3f55fdf62 nix-daemon.service: set XDG_CONFIG_HOME
Otherwise starting nix-daemon fails

● nix-daemon.service - Nix Daemon
   Loaded: loaded
(/nix/store/mnf00a6gc55xl47smk0b32gmi7xpvlfp-nix-1.12pre5308_2f21d522/lib/systemd/system/nix-daemon.service;
enabled; vendor preset: enabled)
  Drop-In:
/nix/store/m2rgjp71n4kyp8j5fxgbrlv13scd5vvv-system-units/nix-daemon.service.d
           └─overrides.conf
   Active: failed (Result: exit-code) since Sat 2017-04-29 11:29:21
CEST; 9s ago
  Process: 7299 ExecStart=nix-daemon --daemon (code=exited, status=1/FAILURE)
 Main PID: 7299 (code=exited, status=1/FAILURE)
      CPU: 19ms

... systemd[1]: Started Nix Daemon.
... nix-daemon[7299]: error: $XDG_CONFIG_HOME and $HOME are not set
... systemd[1]: nix-daemon.service: Main process exited, code=exited, status=1/FAILURE
... systemd[1]: nix-daemon.service: Unit entered failed state.
... systemd[1]: nix-daemon.service: Failed with result 'exit-code'.
... systemd[1]: nix-daemon.service: Start request repeated too quickly.
... systemd[1]: Failed to start Nix Daemon.
... systemd[1]: nix-daemon.service: Failed with result 'exit-code'.
2017-04-29 11:33:55 +02:00
Eelco Dolstra 2f21d522c2 Hopefully fix the Darwin build
http://hydra.nixos.org/build/52080911
2017-04-28 17:13:55 +02:00
Eelco Dolstra 895f00c372 Suppress warning about ssh-auth-sock 2017-04-28 16:55:52 +02:00
Eelco Dolstra 73bba12d8b Check for libreadline 2017-04-28 16:53:56 +02:00
Eelco Dolstra a1a5e63e14 Fix brainfart 2017-04-28 16:21:54 +02:00
Eelco Dolstra 41c4558afe Fix hash computation when importing NARs greater than 4 GiB
This caused "nix-store --import" to compute an incorrect hash on NARs
that don't fit in an unsigned int. The import would succeed, but
"nix-store --verify-path" or subsequent exports would detect an
incorrect hash.

A deeper issue is that the export/import format does not contain a
hash, so we can't detect such issues early.

Also, I learned that -Wall does not warn about this.
2017-04-28 15:24:05 +02:00
Eelco Dolstra 39b08f4c0c Merge pull request #1358 from shlevy/store-nesting
Add Store nesting to fix import-from-derivation within filterSource
2017-04-26 20:28:49 +02:00
Shea Levy 4bc00760f9 Add Store nesting to fix import-from-derivation within filterSource 2017-04-26 14:15:47 -04:00
Eelco Dolstra 45ce2c7413 Doh 2017-04-26 17:58:09 +02:00
Eelco Dolstra 98a2adb135 Simplify building nix-perl in nix-shell 2017-04-26 17:04:45 +02:00
Eelco Dolstra 6734c18c99 nix repl: Fix Ctrl-C 2017-04-25 19:19:48 +02:00
Eelco Dolstra 23aa1619da Minor cleanup 2017-04-25 19:10:47 +02:00
Eelco Dolstra 4c95ef3768 Fix nix-shell test 2017-04-25 18:59:18 +02:00
Eelco Dolstra 536f061765 "using namespace std" considered harmful 2017-04-25 18:58:02 +02:00
Eelco Dolstra 5bd8795e1f nix repl: Use $XDG_DATA_HOME for the readline history 2017-04-25 18:56:29 +02:00
Eelco Dolstra 921a2aeb05 Make "nix repl" build 2017-04-25 18:48:40 +02:00
Eelco Dolstra c31000bc93 Merge nix-repl repository 2017-04-25 18:14:13 +02:00
Eelco Dolstra 40daf0d800 Cleanup in preparation of merging nix-repl repo into nix repo 2017-04-25 18:13:23 +02:00
Eelco Dolstra fee93541a4 Merge branch 'master' of https://github.com/olejorgenb/nix-repl 2017-04-25 16:57:07 +02:00
Eelco Dolstra 2dff9556a4 Fix build 2017-04-25 16:55:03 +02:00
Eelco Dolstra c30330df6f StorePathCommands: Build installables
So for instance "nix copy --to ... nixpkgs.hello" will build
nixpkgs.hello first. It's debatable whether this is a good idea. It
seems desirable for commands like "nix copy" but maybe not for
commands like "nix path-info".
2017-04-25 16:19:22 +02:00
Eelco Dolstra d48c973ece Set default installable
Thus

  $ nix build -f foo.nix

will build foo.nix.

And

  $ nix build

will build default.nix. However, this may not be a good idea because
it's kind of inconsistent, given that "nix build foo" will build the
"foo" attribute from the default installation source (i.e. the
synthesis of $NIX_PATH), rather than ./default.nix. So I may revert
this.
2017-04-25 15:18:05 +02:00
Eelco Dolstra 0b6220fbd6 Interpret any installable containing a slash as a path
So "nix path-info ./result" now works.
2017-04-25 14:09:01 +02:00
Eelco Dolstra 7ee81f3887 Make StorePathsCommand a subclass of InstallablesCommand
This allows commands like 'nix path-info', 'nix copy', 'nix verify'
etc. to work on arbitrary installables. E.g. to copy geeqie to a
binary cache:

  $ nix copy -r --to file:///tmp/binary-cache nixpkgs.geeqie

Or to get the closure size of thunderbird:

  $ nix path-info -S nixpkgs.thunderbird
2017-04-25 13:20:26 +02:00
Eelco Dolstra c769841bc4 Move code around 2017-04-25 12:07:31 +02:00
Eelco Dolstra 6267d74889 Add "nix eval" command
This replaces "nix-instantiate --eval". The result is evaluated
strictly since this seems more useful.
2017-04-25 11:23:47 +02:00
Eelco Dolstra bcecc99007 Restructure installables handling in the "nix" command 2017-04-25 11:20:37 +02:00
Eelco Dolstra 1bb87c0487 Remove debug statement 2017-04-24 15:01:28 +02:00
Eelco Dolstra 66577a1c64 Factor out --json 2017-04-24 14:21:36 +02:00
Eelco Dolstra 9b63bb88c8 nix-shell -p: Use runCommandCC
This restores pre-17.03 behaviour by making gcc available.
2017-04-24 12:04:01 +02:00
Eelco Dolstra 1196470e92 Merge pull request #1347 from kennyballou/sm-grammar-fix
Fix small grammar issue about page
2017-04-24 10:49:46 +02:00
Eelco Dolstra 240399e059 Merge pull request #1351 from earldouglas/ellipses
Drop misleading ellipses
2017-04-24 10:49:09 +02:00
Eelco Dolstra 8d6af08530 Merge pull request #1348 from armijnhemel/nix-env
better document --meta option for nix-env
2017-04-24 10:18:23 +02:00
Eelco Dolstra 169edf9407 Merge pull request #1352 from corngood/cygwin-fixes
Cygwin fixes
2017-04-21 17:39:33 +02:00
David McFarland 804ac52489 add helper function to set 'interruptThrown'
this fixes a linker failure on cygwin 64 due to some bad
interaction between tls and shared libraries.

see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64697
2017-04-21 11:28:14 -03:00
David McFarland d35231ec60 set _GNU_SOURCE on cygwin
this is needed for pipe2()
2017-04-21 11:27:27 -03:00
James Earl Douglas 111d347237 Drop misleading ellipses
This portion of the quick start guide may lead to confusion for
newcomers to Nix.  This change clarifies the example to one that can be
copied in its entirety.
2017-04-20 18:07:23 -06:00
Eelco Dolstra 749696e71c Detect lsof
Also, don't use lsof on Linux since it's not needed.

Fixes #1328.
2017-04-20 19:11:45 +02:00
Eelco Dolstra efa4bdbfcd Improve nix show-config --json
In particular, show descriptions. This could be used for manpage
generation etc.
2017-04-20 17:34:47 +02:00
Eelco Dolstra 4410e9d995 Setting: Remove "Tag" template argument 2017-04-20 16:52:53 +02:00
Eelco Dolstra f05d5f89ff Read per-user settings from ~/.config/nix/nix.conf 2017-04-20 14:58:16 +02:00
Eelco Dolstra 562585e901 binary-caches-parallel-connections -> http-connections 2017-04-20 14:04:00 +02:00
Eelco Dolstra 4222402219 nix.conf man page: binary-caches -> substituters 2017-04-20 13:41:29 +02:00
Eelco Dolstra 76cb3c702c Reimplement trusted-substituters (aka trusted-binary-caches) 2017-04-20 13:41:29 +02:00
Armijn Hemel 1559c596f6 document option 2017-04-19 19:10:12 +02:00
kballou 00b6c6d0c3 Fix small grammar issue about page
Fix subject-verb agreement issue in introduction/about.
2017-04-19 09:27:28 -06:00
Eelco Dolstra 9cc8047f44 Reimplement connect-timeout
Fixes #1339.
2017-04-19 14:54:52 +02:00
Eelco Dolstra b0cb117226 getDerivations(): Filter out packages with bad derivation names
In particular, this disallows attribute names containing dots or
starting with dots. Hydra already disallowed these. This affects the
following packages in Nixpkgs master:

  2048-in-terminal
  2bwm
  389-ds-base
  90secondportraits
  lispPackages.3bmd
  lispPackages.hu.dwim.asdf
  lispPackages.hu.dwim.def

Closes #1342.
2017-04-19 14:18:26 +02:00
Eelco Dolstra 62a07992bd Merge pull request #1337 from lheckemann/doc-tryEval
Manual: document tryEval
2017-04-18 16:54:37 +02:00
Eelco Dolstra 67fe3e07b2 Merge pull request #1321 from shlevy/channel-direct-tarball-error
nix-channel: error out if direct tarball unpack fails.
2017-04-18 16:51:46 +02:00
Linus Heckemann 9622d00afa Manual: document tryEval 2017-04-17 10:33:53 +01:00
Eelco Dolstra f8a2e8a552 Shut up some warnings 2017-04-14 14:42:20 +02:00
Eelco Dolstra dd3714f6ef Doh 2017-04-14 14:42:08 +02:00
Eelco Dolstra da76c72bc9 Build on aarch64-linux 2017-04-14 14:02:43 +02:00
Eelco Dolstra 6520b757c5 Fix 32-bit build
http://hydra.nixos.org/build/51569816
2017-04-14 13:59:39 +02:00
Eelco Dolstra 3b36c64b15 Fix Perl bindings 2017-04-14 13:47:10 +02:00
Eelco Dolstra 3872371f25 Minor cleanup 2017-04-14 13:42:22 +02:00
Eelco Dolstra 01dcdfcf33 nix-daemon: Don't set untrusted-* settings
These are no longer used anywhere.
2017-04-14 11:59:51 +02:00
Eelco Dolstra 1673c373c9 nix-daemon: Don't die if the user sends an unknown setting 2017-04-14 11:57:02 +02:00
Eelco Dolstra 872ba75d8b Add "nix show-config" command
This dumps the entire Nix configuration, including all options that
have default values.
2017-04-13 20:59:38 +02:00
Eelco Dolstra ba9ad29fdb Convert Settings to the new config system
This makes all config options self-documenting.

Unknown or unparseable config settings and --option flags now cause a
warning.
2017-04-13 20:53:23 +02:00
Eelco Dolstra 6bd9576aeb Support arbitrary numeric types for settings 2017-04-13 17:54:05 +02:00
Eelco Dolstra 0bf34de43b Validate Boolean settings better 2017-04-13 16:31:28 +02:00
Eelco Dolstra 1860070548 Merge branch 'rework-options' of https://github.com/copumpkin/nix 2017-04-13 16:15:51 +02:00
Eelco Dolstra 2040240e23 Add a Config class to simplify adding configuration settings
The typical use is to inherit Config and add Setting<T> members:

  class MyClass : private Config
  {
    Setting<int> foo{this, 123, "foo", "the number of foos to use"};
    Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"};

    MyClass() : Config(readConfigFile("/etc/my-app.conf"))
    {
      std::cout << foo << "\n"; // will print 123 unless overriden
    }
  };

Currently, this is used by Store and its subclasses for store
parameters. You now get a warning if you specify a non-existant store
parameter in a store URI.
2017-04-13 16:03:31 +02:00
Eelco Dolstra 568a099c88 canonPath(): Check against empty paths 2017-04-13 16:03:31 +02:00
Eelco Dolstra 6d97d81656 Add warn function 2017-04-13 16:03:31 +02:00
Eelco Dolstra 31cc9366fc Initialise logger 2017-04-13 16:03:31 +02:00
Eelco Dolstra 16c4856027 Merge pull request #1325 from benley/float-rl-1.12
Move note about float support out of the wrong release notes
2017-04-13 16:00:37 +02:00
Benjamin Staffin d267db0d75 Move note about float support out of the wrong release notes
Looks like this snuck into the 1.11 release notes post-release, but
float support isn't actually present until 1.12.
2017-04-12 18:01:43 -04:00
Eelco Dolstra 23304f527a Merge pull request #1302 from dtzWill/fix/nix-options
Process nix.conf options in "new" commands, add test
2017-04-12 11:03:19 +02:00
Eelco Dolstra b134c2d052 Drop WWW::Curl dependency
Somehow this came back after d1da6967b8.
2017-04-11 15:41:50 +02:00
Shea Levy 503cc4431b nix-channel: error out if direct tarball unpack fails.
It's very unlikely a path ending in .tar.gz is a directory

Fixes #1318
2017-04-10 18:16:46 -04:00
Eelco Dolstra 915f62fa19 shell.nix: Remove more dependencies
Thanks @copumpkin.
2017-04-10 17:23:15 +02:00
Eelco Dolstra 1fe1976e0d Merge pull request #1316 from copumpkin/nix-retries-default
Default to 5 download retries
2017-04-10 15:58:57 +02:00
Eelco Dolstra 8945a0ea2c Merge pull request #1317 from copumpkin/nix-retry-write-error
Add CURLE_WRITE_ERROR as a transient error condition
2017-04-10 15:58:17 +02:00
Dan Peebles d1fdade755 Add CURLE_WRITE_ERROR as a transient error condition
We've observed it failing downloads in the wild and retrying the same URL
a few moments later seemed to fix it.
2017-04-10 09:28:44 -04:00
Dan Peebles e43e8be8e7 Default to 5 download retries
This should help certain downloaders that don't request anything special
for the number of retries, like nix-channel.
2017-04-10 09:22:24 -04:00
Eelco Dolstra 53edb55588 shell.nix: Remove obsolete flags 2017-04-10 11:50:01 +02:00
Eelco Dolstra 105f8ffc98 Minor cleanup
Also, possible fix for #1310 on 32-bit systems.
2017-04-10 11:27:33 +02:00
Eelco Dolstra 95295482ea Allow "auto" as a store URI
Using the empty string is likely to be ambiguous in some contexts.
2017-04-10 11:27:29 +02:00
Neil Mayhew f12a048a05 Propagate NIX_BUILD_CORES to nix-shell environments 2017-04-09 08:21:52 -06:00
Will Dietz a0c56197fc tests/timeout: create output so tests don't trivially pass
Timeout tests rely on failed build to determine success,
so make sure these derivations (silent in particular)
don't fail regardless of timeout behavior.
2017-04-08 12:59:42 -05:00
Will Dietz 30f89e0d65 Process nix.conf options in "new" nix commands, add test.
Without this (minor) change, the options set using "--option"
or read from nix.conf were parsed but not used.
2017-04-08 12:59:42 -05:00
Eelco Dolstra d8ff3aaae7 Merge pull request #1306 from copumpkin/retry-on-ssl-error
Retry downloads on transient SSL errors too
2017-04-07 13:53:38 +02:00
Dan Peebles 98283915f5 Retry downloads on transient SSL errors too 2017-04-06 18:18:43 +00:00
Eelco Dolstra ba20730b3f Implement RemoteStore::queryMissing()
This provides a significant speedup, e.g. 64 s -> 12 s for

  nix-build --dry-run -I nixpkgs=channel:nixos-16.03 '<nixpkgs/nixos/tests/misc.nix>' -A test

on a cold local and CloudFront cache.

The alternative is to use lots of concurrent daemon connections but
that seems wasteful.
2017-04-06 18:40:19 +02:00
Eelco Dolstra 963f2bf12b Fix bogus "unexpected Nix daemon error: interrupted by the user" 2017-04-06 17:19:32 +02:00
Eelco Dolstra 6b5e271163 Add a method to allow hydra-queue-runner to flush the path info cache 2017-04-06 15:22:37 +02:00
Eelco Dolstra 256940fc48 nix-daemon: Disable path info cache
This is useless because the client also caches path info, and can
cause problems for long-running clients like hydra-queue-runner
(i.e. it may return cached info about paths that have been
garbage-collected).
2017-04-06 14:30:31 +02:00
Eelco Dolstra 8decb07c31 Allow default sandbox paths to be overriden
E.g. you can now redirect /etc/resolv.conf to a different file.
2017-04-04 17:54:16 +02:00
Eelco Dolstra 488792a87d Make /var/run/nscd/socket optional
Not every distribution uses nscd.
2017-04-04 17:40:50 +02:00
Eelco Dolstra dfcd78d851 Really fix the RPM build 2017-04-03 16:25:20 +02:00
Eelco Dolstra 3b8946e09a Maybe fix the RPM build
http://hydra.nixos.org/build/51095532
2017-03-31 18:30:50 +02:00
Eelco Dolstra 29d35805c6 Sandbox: Fix /dev/ptmx on recent kernels
This fixes "No such file or directory" when opening /dev/ptmx
(e.g. http://hydra.nixos.org/build/51094249).

The reason appears to be some changes to /dev/ptmx / /dev/pts handling
between Linux 4.4 and 4.9. See
https://patchwork.kernel.org/patch/7832531/.

The fix is to go back to mounting a proper /dev/pts instance inside
the sandbox. Happily, this now works inside user namespaces, even for
unprivileged users. So

  NIX_REMOTE=local?root=/tmp/nix nix-build \
    '<nixpkgs/nixos/tests/misc.nix>' -A test

works for non-root users.

The downside is that the fix breaks sandbox builds on older kernels
(probably pre-4.6), since mounting a devpts fails inside user
namespaces for some reason I've never been able to figure out. Builds
on those systems will fail with

  error: while setting up the build environment: mounting /dev/pts: Invalid argument

Ah well.
2017-03-31 18:20:19 +02:00
Shea Levy 3ecb09a40a builtins.exec: Make the argument just a list 2017-03-31 11:58:41 -04:00
Shea Levy d299bd710a Merge branch 'builtins.exec' 2017-03-31 11:22:39 -04:00
Eelco Dolstra b9b8b8a63b Fix evaluation error 2017-03-31 15:54:15 +02:00
Eelco Dolstra e1509adbbb Retry curl error 16 2017-03-31 15:50:11 +02:00
Eelco Dolstra d087700347 Fix perl build 2017-03-31 15:31:34 +02:00
Eelco Dolstra c0745a2531 Merge branch 'remove-perl' of https://github.com/shlevy/nix 2017-03-31 14:13:32 +02:00
Shea Levy a75475ca61 Remove tabs 2017-03-30 16:51:50 -04:00
Shea Levy 0bb8db257d Add exec primop behind allow-unsafe-native-code-during-evaluation.
Execute a given program with the (optional) given arguments as the
user running the evaluation, parsing stdout as an expression to be
evaluated.

There are many use cases for nix that would benefit from being able to
run arbitrary code during evaluation, including but not limited to:

* Automatic git fetching to get a sha256 from a git revision
* git rev-parse HEAD
* Automatic extraction of information from build specifications from
  other tools, particularly language-specific package managers like
  cabal or npm
* Secrets decryption (e.g. with nixops)
* Private repository fetching

Ideally, we would add this functionality in a more principled way to
nix, but in the mean time 'builtins.exec' can be used to get these
tasks done.

The primop is only available when the
'allow-unsafe-native-code-during-evaluation' nix option is true. That
flag also enables the 'importNative' primop, which is strictly more
powerful but less convenient (since it requires compiling a plugin
against the running version of nix).
2017-03-30 08:04:21 -04:00
Eelco Dolstra c60715e937 Ignore broken "Deriver: unknown-deriver" fields in .narinfo
These were generated by a legacy tool.
2017-03-28 13:08:13 +02:00
Daiderd Jordan 023217f07c use std::tuple for ValueMap allocator 2017-03-24 23:05:49 +01:00
Eelco Dolstra 165786dbc0 Merge pull request #1287 from mattaudesse/readme-grammar-its-typo
Fix minor grammatical nitpick ("it's" vs. "its") in `README.md`.
2017-03-22 15:41:46 +01:00
Matt Audesse 8edf107177 Fix minor grammatical nitpick ("it's" vs. "its") in README.md.
See: http://data.grammarbook.com/blog/pronouns/1-grammar-error/
2017-03-22 10:11:23 -04:00
Eelco Dolstra 0a7ca24c26 Fix xz decompression
Fixes #1285.
2017-03-22 11:53:33 +01:00
Eelco Dolstra 895a74a814 LocalFSStore::getBuildLog(): Handle corrupted logs 2017-03-21 19:23:35 +01:00
Eelco Dolstra ed5c0f69f2 Don't hang in decompression if bzip2 data ends prematurely 2017-03-21 19:23:35 +01:00
Eelco Dolstra aa23bba27f Fix tests to reflect the signed-binary-caches default change 2017-03-21 18:06:13 +01:00
Eelco Dolstra e1e49c58e1 Only use cache.nixos.org when the store is /nix/store
This is consistent with the behaviour of the old
download-from-binary-cache substituter.
2017-03-21 17:59:18 +01:00
Eelco Dolstra ecbc3fedd3 Require signatures by default
This corresponds to the NixOS default.
2017-03-21 15:10:48 +01:00
Eelco Dolstra 4bb38591e5 Restore cache.nixos.org as the default substituter
Fixes #1283.
2017-03-21 15:06:46 +01:00
Eelco Dolstra 7347daba8c Don't make brotli a hard dependency 2017-03-21 14:47:35 +01:00
Eelco Dolstra f8b84a3b8c Move istringstream_nocopy to a separate file 2017-03-21 14:43:03 +01:00
Eelco Dolstra 3229f85585 Honor $NIX_SSHOPTS again
NixOps needs this.
2017-03-21 14:35:50 +01:00
Eelco Dolstra 20df50d8e1 Merge branch 'darwin-s3-binary-cache-store' of https://github.com/shlevy/nix 2017-03-20 17:46:35 +01:00
Eelco Dolstra dc931fe1cd Merge pull request #1275 from steveeJ/patch-1
nix-shell/pure: keep environment variable SHLVL
2017-03-20 17:38:40 +01:00
Eelco Dolstra 558eda0115 nix copy: Make -r option use the "from" store
Previously, we tried to compute the closure in the local store, which
obviously doesn't work.
2017-03-16 14:25:54 +01:00
Eelco Dolstra 287084d688 ssh:// -> ssh-ng://, legacy-ssh:// -> ssh:// 2017-03-16 14:19:32 +01:00
Eelco Dolstra c5b83d8913 copyPaths(): Use queryValidPaths() to reduce SSH latency 2017-03-16 13:50:01 +01:00
Eelco Dolstra 91d67692cf copyPaths(): Don't query path info for a path the target already has
For example, this cuts "nix-copy-closure --from" on a NixOS system
closure from 15.9s to 0.5s.
2017-03-16 12:05:51 +01:00
Eelco Dolstra ea7fa88131 LegacySSHStore: Provide a faster implementation of computeFSClosure()
This avoids the latency of the standard implementation, which can make
a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16 11:44:52 +01:00
Eelco Dolstra 7a716ef2a5 Fix nix-copy-closure --to 2017-03-16 10:58:48 +01:00
Eelco Dolstra 0ec7f47b00 Remove "killing process <pid>" messages
They convey no useful information.
2017-03-16 10:52:28 +01:00
Eelco Dolstra 43f158bb08 nix-copy-closure: Fix assertion failure
$ ./inst/bin/nix-copy-closure --to bla $(type -p firefox)
  nix-copy-closure: src/libstore/store-api.cc:80: std::__cxx11::string nix::storePathToHash(const Path&): Assertion `base.size() >= storePathHashLen' failed.
2017-03-16 10:45:45 +01:00
Eelco Dolstra 3f35612c04 More precise compression settings 2017-03-15 17:20:19 +01:00
Eelco Dolstra 25dff2b7db runProgram(): Distinguish between empty input and no input
For example, if we call brotli with an empty input, it shouldn't read
from the caller's stdin.
2017-03-15 16:50:19 +01:00
Eelco Dolstra 042975ea8e Compress NAR listings using the "text-compression" method
So if "text-compression=br", the .ls file in S3 will get a
Content-Encoding of "br". Brotli appears to compress better than xz
for this kind of file and is natively supported by browsers.
2017-03-15 16:50:19 +01:00
Eelco Dolstra 45c70382ac S3BinaryCacheStore: Set Content-Type
This is necessary for serving log files to browsers.
2017-03-15 16:50:19 +01:00
Eelco Dolstra 8b1d65bebe S3BinaryCacheStore: Support compression of narinfo and log files
You can now set the store parameter "text-compression=br" to compress
textual files in the binary cache (i.e. narinfo and logs) using
Brotli. This sets the Content-Encoding header; the extension of
compressed files is unchanged.

You can separately specify the compression of log files using
"log-compression=br". This is useful when you don't want to compress
narinfo files for backward compatibility.
2017-03-15 16:49:28 +01:00
Eelco Dolstra 2691498b5c Fix assertion failure on SIGINT
nix: src/libutil/compression.cc:142: virtual nix::XzSink::~XzSink(): Assertion `finished' failed.
2017-03-15 16:49:06 +01:00
Eelco Dolstra fbbc4d8dda Fix deadlock in runProgram() when input is larger than the pipe buffer size 2017-03-15 16:49:06 +01:00
Eelco Dolstra e8186085e0 Add support for brotli compression
Build logs on cache.nixos.org are compressed using Brotli (since this
allows them to be decompressed automatically by Chrome and Firefox),
so it's handy if "nix log" can decompress them.
2017-03-15 16:49:06 +01:00
Eelco Dolstra 73d7a51ee6 Remove dependency on "curl" binary 2017-03-15 16:48:29 +01:00
Eelco Dolstra 532d73d5d8 BinaryCacheStore: Implement getBuildLog()
We assume that build logs are stored under log/<drv>, e.g.

  /nix/store/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv

maps to

  https://cache.nixos.org/log/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv
2017-03-15 16:48:29 +01:00
Eelco Dolstra 5b86451f02 Add a "nix log" command
This replaces "nix-store --read-log". It checks the local store and
any configured substituters for the requested logs.
2017-03-15 16:48:29 +01:00
Eelco Dolstra 0afeb7f51e Store: Add a method for getting build logs
This allows various Store implementations to provide different ways to
get build logs. For example, BinaryCacheStore can get the build logs
from the binary cache.

Also, remove the log-servers option since we can use substituters for
this.
2017-03-15 16:48:29 +01:00
Shea Levy 96443e94a1 Merge branch 'configurable-aws-region' 2017-03-12 10:01:11 -04:00
Stefan Junker f628ca2a1f nix-shell/pure: keep environment variable SHLVL 2017-03-12 01:04:21 +01:00
Domen Kožar ae568847f5 Dockerfile: 1.11.2 -> 1.11.7 2017-03-11 19:39:13 +01:00
Eelco Dolstra 19643a781e nix -> Nix 2017-03-09 13:36:56 +01:00
Eelco Dolstra 86227390c5 Merge branch 'allow-import-from-derivation' of https://github.com/shlevy/nix 2017-03-09 13:36:30 +01:00
Eelco Dolstra 6a308c7da4 Merge pull request #1267 from adevress/master
Add missing header <sstream>
2017-03-09 13:35:19 +01:00
Adrien Devresse d853877ce9 Add missing header <sstream> 2017-03-08 22:24:10 +01:00
Shea Levy 5f831c1057 Add docs for allow-import-from-derivation 2017-03-08 09:12:03 -05:00
Shea Levy 93f863be96 Add option to disable import-from-derivation completely, even if the drv is already realized 2017-03-08 08:46:12 -05:00
Eelco Dolstra 121a407eec Merge pull request #1264 from shlevy/s3-ca-file
Properly set the caFile for aws-sdk-cpp s3
2017-03-06 20:54:52 +01:00
Shea Levy 3cc18d3753 Properly set the caFile for aws-sdk-cpp s3 2017-03-06 14:30:35 -05:00
Shea Levy 4fc30922cf istringstream_nocopy: Implement in a standards-compliant way.
Fixes the problem mentioned in e6a61b8da7

See #1135
2017-03-06 13:03:02 -05:00
Eelco Dolstra 1cf4801108 Revert fa125b9b28
This causes quadratic performance.
2017-03-06 13:56:20 +01:00
Shea Levy 689b825627 nix-daemon.plist: Set XDG_CACHE_HOME 2017-03-05 18:15:44 -05:00
Shea Levy b667abc699 Add signing and s3 support on darwin 2017-03-05 07:39:10 -05:00
Shea Levy 5789eaa3f4 Add aws-region param to S3 store URLs 2017-03-03 16:12:17 -05:00
Eelco Dolstra d1158bb816 Cache connection failures 2017-03-03 19:36:01 +01:00
Eelco Dolstra 8490ee37a6 SSHMaster: Make thread-safe 2017-03-03 19:28:27 +01:00
Eelco Dolstra d3eb1cf3bb build-remote: Don't use a SSH master
This is unnecessary because we make only one connection.
2017-03-03 19:23:20 +01:00
Eelco Dolstra d140c75530 Pool: Don't hang if creating a connection fails 2017-03-03 19:21:43 +01:00
Eelco Dolstra 577ebeaefb Improve SSH handling
* Unify SSH code in SSHStore and LegacySSHStore.

* Fix a race starting the SSH master. We now wait synchronously for
  the SSH master to finish starting. This prevents the SSH clients
  from starting their own connections.

* Don't use a master if max-connections == 1.

* Add a "max-connections" store parameter.

* Add a "compress" store parameter.
2017-03-03 19:05:50 +01:00
Eelco Dolstra 7f62be1bcd build-remote: Fix passing SSH key 2017-03-03 16:33:18 +01:00
Eelco Dolstra 5a1fb03b8f build-remote: Misc cleanup 2017-03-03 16:18:49 +01:00
Eelco Dolstra 2f992692e2 Fix fatal "broken pipe" error when $NIX_BUILD_HOOK is missing 2017-03-03 15:40:06 +01:00
Eelco Dolstra ffcf9d24a6 builtins.fetchgit: Support paths 2017-03-02 11:46:28 +01:00
Eelco Dolstra fbf17f1ad7 builtins.fetchgit: Fix bad format string 2017-03-02 11:40:11 +01:00
Eelco Dolstra fa125b9b28 TeeSink: Pre-reserve string space
When receiving a very large file, this can prevent the string from
having tobe copied, which temporarily doubles memory consumption.
2017-03-01 16:16:04 +01:00
Eelco Dolstra f61f67ddee RemoteStore::addToStore(): Send NAR rather than string containing NAR
This allows the NAR to be streamed in the future (though we're not
doing that yet).
2017-03-01 16:07:15 +01:00
Eelco Dolstra 374908726b readString(): Read directly into std::string
When reading a huge string, this halves memory consumption.

(Strictly speaking, this appears only valid in C++17, but who cares...)
2017-03-01 14:54:11 +01:00
Eelco Dolstra 07a0b8ca67 Tweak message 2017-03-01 14:52:16 +01:00
Eelco Dolstra 56e19d970d nix-store --import: Fix importing unsigned paths 2017-03-01 14:47:52 +01:00
Eelco Dolstra e321551d54 Fix assertion failure in nix-store --export
Fixes #1173.
2017-03-01 14:41:54 +01:00
Eelco Dolstra c4a40949d9 Handle importing NARs containing files greater than 4 GiB
Also templatize readInt() to work for various integer types.
2017-03-01 13:52:54 +01:00
Eelco Dolstra 0780805246 Fix building against older curl versions
http://hydra.nixos.org/build/49490928
2017-02-28 14:03:43 +01:00
Eelco Dolstra fd86dd93dd Improve SQLite busy handling 2017-02-28 13:59:11 +01:00
Eelco Dolstra 34b12bad59 NarInfoDiskCache: Handle SQLite busy errors 2017-02-28 13:44:11 +01:00
Eelco Dolstra 80027144ae In SQLite errors, include the database path
This is necessary because we have multiple SQLite databases (e.g. the
binary cache cache).
2017-02-28 13:20:12 +01:00
Eelco Dolstra 7251d048fa Support auto-configuration of build-max-jobs
"build-max-jobs" and the "-j" option can now be set to "auto" to use
the number of CPUs in the system. (Unlike build-cores, it doesn't use
0 to imply auto-configuration, because a) magic values are a bad idea
in general; b) 0 is a legitimate value used to disable local
building.)

Fixes #1198.
2017-02-28 12:54:50 +01:00
Eelco Dolstra 3fab1f04a7 _SC_NPROCESSORS_ONLN -> std::thread::hardware_concurrency() 2017-02-27 16:01:54 +01:00
Eelco Dolstra 4f3fb34844 Fix 1.8 release note about build-max-jobs
We set build-cores automatically, not build-max-jobs. (The commit
message for de4cdd0d47 also got this
wrong.)
2017-02-27 15:58:24 +01:00
Eelco Dolstra 3cb0387d3f Retry downloads on HTTP/2 stream errors
Issue #1254.
2017-02-27 14:36:09 +01:00
Domen Kožar f0bbd153c6 Merge pull request #1252 from bjornfor/channel-rename-fix
unpack-channel.nix: fix 'mv' corner case
2017-02-27 13:12:35 +01:00
Bjørn Forsman f72206b736 unpack-channel.nix: fix 'mv' corner case
unpack-channel.nix fails if the tarball contains a directory named the
same as the channel:

  mv: cannot move 'nixpkgs' to a subdirectory of itself, '.../nixpkgs'

This commit fixes that by not moving the directory if it already has the
correct name.
2017-02-27 10:12:58 +01:00
Dan Peebles e7cb2847ab Explicitly model all settings and fail on unrecognized ones
Previously, the Settings class allowed other code to query for string
properties, which led to a proliferation of code all over the place making
up new options without any sort of central registry of valid options. This
commit pulls all those options back into the central Settings class and
removes the public get() methods, to discourage future abuses like that.

Furthermore, because we know the full set of options ahead of time, we
now fail loudly if someone enters an unrecognized option, thus preventing
subtle typos. With some template fun, we could probably also dump the full
set of options (with documentation, defaults, etc.) to the command line,
but I'm not doing that yet here.
2017-02-22 20:19:25 -05:00
Janus Troelsen 2b4c24f46a Remove Perl dependency listing in doc 2017-02-07 15:56:32 -05:00
Shea Levy f7b7df8d1f Add nix-perl package for the perl bindings 2017-02-07 15:56:32 -05:00
Shea Levy 418a837897 Remove perl dependency.
Fixes #341
2017-02-07 15:56:32 -05:00
Eelco Dolstra 0e49f94120 Merge pull request #35 from orbekk/master
Update <nixpkgs/nixos> path in documentation.
2017-01-02 11:51:21 +01:00
Kjetil Orbekk 5476e987d5 Update <nixpkgs/nixos> path in documentation. 2017-01-01 16:13:11 -05:00
Eelco Dolstra 8b60529231 Strip ANSI escapes from file names
Also, use bright blue for lambdas, otherwise my eyes hurt.
2016-07-21 11:21:59 +02:00
Eelco Dolstra af5ff6c918 Merge branch 'lambda-position' of https://github.com/fkz/nix-repl 2016-07-21 11:16:22 +02:00
Eelco Dolstra df32610dfa Merge branch 'show-trace' of https://github.com/fkz/nix-repl 2016-07-21 11:14:37 +02:00
Eelco Dolstra 00ee039c27 Merge branch 'colorize' of https://github.com/ehmry/nix-repl 2016-07-21 11:12:45 +02:00
Emery Hemingway 34ec98176e fixup "Colorize"
Do not quote escape sequence macro
2016-06-25 13:40:50 +02:00
Emery Hemingway eaabcba1c3 Colorize
Berlin NixOS meetup
2016-06-25 13:28:27 +02:00
Fabian Schmitthenner 828cf7b058 show trace of errors when using --show-trace 2016-03-19 13:55:10 +00:00
Eelco Dolstra e37bca136e Merge pull request #19 from tsion/x-command
Add :u command which works like `nix-shell -p`.
2016-03-07 15:52:02 +01:00
Fabian Schmitthenner 8bec2c07a1 When showing a lambda, also show the position of the definition 2016-03-05 16:48:04 +00:00
Scott Olson e2ff27da07 Rename :x to :u, for 'use'. 2016-03-01 04:03:18 -06:00
Scott Olson 86e93b9f61 Add :x command which works like nix-shell -p. 2016-03-01 04:03:18 -06:00
Eelco Dolstra 79b02dffcb Merge pull request #20 from tsion/more-help
Expand the help message printed from --help.
2016-02-29 12:48:04 +01:00
Eelco Dolstra 2bfb00c66e Merge pull request #17 from tsion/install-cmd
Add :i command to install a derivation to the current profile.
2016-02-29 12:47:00 +01:00
Eelco Dolstra b1cc845413 Merge pull request #25 from fkz/fix-store
fix nix-repl compilation after we don't have a global store variable anymore
2016-02-29 12:46:15 +01:00
Fabian Schmitthenner ff8d0698c7 fix nix-repl after we don't have a global store variable anymore (cf nix@c10c61449f954702ae6d8092120321744acd82ff) 2016-02-28 22:44:08 +00:00
Eelco Dolstra a1ea85e92b Merge pull request #24 from tsion/whitespace
Fix handling of whitespace and ignore blank inputs.
2016-02-24 12:02:06 +01:00
Scott Olson 38816759fc Ignore blank inputs.
Previously, nix-repl would consider this an incomplete parse and wait for the
next line as if it was a multiline input.

Blank lines in the middle of a multiline input will continue to work.
2016-02-23 18:30:21 -06:00
Scott Olson 87e6649fc3 Fix handling of whitespace.
Whitespace will no longer be removed from input lines, which fixes pasting
multiline strings containing end-of-line or beginning-of-line whitespace.
2016-02-23 18:29:56 -06:00
Ole Jørgen Brønner 103c46abc2 Preserve readline history across sessions. Add rl_readline_name. 2016-02-23 23:19:49 +01:00
Eelco Dolstra 5599665a27 Merge pull request #22 from tsion/quote-invalid-vars
Print syntactially invalid attribute names as strings.
2016-02-23 11:07:55 +01:00
Scott Olson 97da6d62f2 Print syntactially invalid attribute names as strings. 2016-02-20 01:14:22 -06:00
Eelco Dolstra a52fd0dbd0 Merge branch 'issue-13' of https://github.com/tsion/nix-repl 2016-02-18 19:40:27 +01:00
Eelco Dolstra 02daf2ec0b Merge branch 'multiline' of https://github.com/tsion/nix-repl 2016-02-18 19:33:46 +01:00
Scott Olson 56c7f0e8c5 Fix typo in comment. 2016-02-18 07:04:55 -06:00
Scott Olson 60ba98242f Fix recognition of REPL commands. 2016-02-18 06:59:51 -06:00
Scott Olson 64080d26fe Cancel multiline input on Ctrl-C. 2016-02-18 06:50:52 -06:00
Scott Olson 2d729e4f6f Support multiline input by detecting "unfinished" parse errors.
Fixes #4,
2016-02-18 06:27:39 -06:00
Scott Olson 287dfee35e Expand the help message printed from --help.
Fixes #10. I consider this a temporary measure, however, until nix-repl has a
manpage (see #14). Then it can just open its manpage on --help like the other
nix tools do.

Much of the text in this commit was copied from nix-build's manpage.
2016-02-18 04:05:11 -06:00
Scott Olson cfc874ee52 Open the store before constructing EvalState.
EvalState requires the `store` global to be initialized before it is constructed
in some cases, e.g. when it needs to download a tarball for something in
NIX_PATH. Hence, this fixes #13.
2016-02-16 18:40:47 -06:00
Scott Olson 82aca33899 Add :i command to install a package to the current profile.
It works by running `nix-env -i <derivation path>`.

Fixes #15.
2016-02-16 00:36:50 -06:00
Scott Olson 30a7bfbebe Fix grammar. 2016-02-15 23:11:26 -06:00
Scott Olson 3cfb8d1584 Remove unused global variable. 2016-02-15 19:16:24 -06:00
Eelco Dolstra dc8b51754b Merge pull request #16 from tsion/dashed-assign
Improve variable name parsing for assignments.
2016-02-15 10:25:45 +01:00
Scott Olson f30fd9c47b Don't consider empty strings or strings beginning with numbers as variable names. 2016-02-14 01:50:47 -06:00
Scott Olson 2111098a3a Don't consider strings starting with - or ' as variable names. 2016-02-14 01:29:48 -06:00
Scott Olson f7980b4712 Parse foo-bar = expr as an assignment. 2016-02-14 01:16:30 -06:00
Eelco Dolstra 8a2f5f0607 Fix building against Nix 1.10
Fixes #12.
2015-09-07 13:05:58 +02:00
Susan Potter 57aeef0b6a Fix nix-repl does not support '--help'
According to popular practice and convention `nix-repl` now supports `--help`
like a good POSIX citizen[1].

[1] https://www.gnu.org/prep/standards/html_node/Command_002dLine-Interfaces.html
2015-07-06 15:53:47 +02:00
Eelco Dolstra 45c6405a30 Fix building against latest Nix
Fixes #8.
Fixes #9.
2015-06-02 13:24:24 +02:00
Eelco Dolstra f92408136e Fix building against current Nix master 2014-12-01 10:07:10 +01:00
Eelco Dolstra 71d61508f2 Support -I flag 2014-08-26 20:05:08 +02:00
Eelco Dolstra 89f9c0d41b Fix building against current Nix master 2014-08-26 20:03:12 +02:00
Eelco Dolstra 02b66e97ba Fix building against current Nix master 2014-07-24 17:53:32 +02:00
Eelco Dolstra 2cf0e67761 Handle non-numeric version strings
Fixes #2.
2014-07-24 17:46:58 +02:00
Eelco Dolstra 66b2d18243 Don't parse 'var == expr' as an assignment 2014-06-16 10:05:09 -04:00
Eelco Dolstra 1734e8a149 Fix crash in tab completion
Fixes #1. Patch by Maxdamantus.
2014-04-11 12:51:15 +02:00
Eelco Dolstra 22a47ab03c Fix building against Nix 1.7 2014-04-11 12:50:46 +02:00
Eelco Dolstra 6a4a8208be Fix building against current Nix 2014-01-28 10:42:23 +01:00
Eelco Dolstra 3beb6f6e76 Show derivations more concisely 2014-01-28 10:42:23 +01:00
Eelco Dolstra 81d658fe4a Fix readme 2013-09-09 18:11:47 +02:00
Eelco Dolstra 853d2e0aa4 Fix markdown 2013-09-09 18:10:57 +02:00
Eelco Dolstra ae50a5e7be Add a README 2013-09-09 18:09:46 +02:00
Eelco Dolstra dc670a173a Make tab-completion work on builtins 2013-09-09 17:22:42 +02:00
Eelco Dolstra e91160021f On reload, wipe the environment 2013-09-09 17:06:14 +02:00
Eelco Dolstra 498f8b0485 Add license 2013-09-09 16:17:45 +02:00
Eelco Dolstra adde4f0c8d Add :reload command 2013-09-09 16:02:46 +02:00
Eelco Dolstra ddd22c37c5 Build on Mac OS X 2013-09-09 16:02:35 +02:00
Eelco Dolstra 03ef6b69be Add system argument 2013-09-09 15:17:08 +02:00
Eelco Dolstra 3567bdb514 Add :quit command 2013-09-09 15:02:56 +02:00
Eelco Dolstra a5dffb3d3d Temporary hack to parameterize nixpkgs 2013-09-09 14:03:28 +02:00
Eelco Dolstra 3c67df928f Add sugar for defining a variable
‘x = <expr>’ is short for ‘:a { x = <expr>; }’.  Note that the
right-hand side refers to the original scope, so you get:

  nix-repl> x = 1
  nix-repl> x = x + 1
  nix-repl> x
  2

rather than an infinite recursion.
2013-09-09 13:56:53 +02:00
Eelco Dolstra 4b33c2dd4c Add help (:?) 2013-09-09 13:22:33 +02:00
Eelco Dolstra e133e91410 Support tab-completion on attribute sets
Example:

$ nix-repl '<nixos>'

> config.services.xserver.desktop<TAB>

comletes to

> config.services.xserver.desktopManager

You also get suggestions if there are multiple matches:

> config.services.xserver.desktopManager.kde4
config.services.xserver.desktopManager.kde4.enable
config.services.xserver.desktopManager.kde4.phononBackends
2013-09-09 12:00:33 +02:00
Eelco Dolstra 8e765b8876 printValue: Show assertion errors inline 2013-09-09 11:37:05 +02:00
Eelco Dolstra e587aec123 printValue: Don't show lists/attribute sets twice 2013-09-09 11:14:43 +02:00
Eelco Dolstra 7e3625f924 Improved value display
By default, we don't recurse into attribute sets or lists when
printing a value.  However, the new :p command does recurse.
2013-09-07 00:35:54 +02:00
Eelco Dolstra c6f2b89c0e Restore affinity 2013-09-06 21:00:36 +02:00
Eelco Dolstra 0abdf4beaa Add basic variable name completion 2013-09-06 19:51:59 +02:00
Eelco Dolstra cf4c29d90a Load files specified on the command line
For example:

  $ nix-repl '<nixpkgs>' '<nixos>'
  Welcome to Nix version 1.6pre3215_2c1ecf8. Type :? for help.

  Loading ‘<nixpkgs>’...
  Added 3337 variables.

  Loading ‘<nixos>’...
  Added 7 variables.

  nix-repl>
2013-09-06 15:20:06 +02:00
Eelco Dolstra b5944ac4ff Add a command :s to start a nix-shell for a derivation 2013-09-06 15:05:18 +02:00
Eelco Dolstra c6712a007f Add a command :b to build a derivation 2013-09-06 14:58:53 +02:00
Eelco Dolstra ad0dd359b4 Don't exit on SIGINT during evaluation
However, this may leave thunks in black-holed state, so it's not
really safe.
2013-09-06 13:20:35 +02:00
Eelco Dolstra 0b419c048b Use readline history file 2013-09-06 13:14:28 +02:00
Eelco Dolstra 4fb82d3d80 Handle SIGINT to cancel the current line 2013-09-06 13:01:02 +02:00
Eelco Dolstra 504563ea44 Fix build 2013-09-06 11:54:49 +02:00
Eelco Dolstra 0f6279d874 Add a command :l for loading a file into scope
Example:

nix-repl> :l <nixpkgs>

nix-repl> lib.range 0 10
[ 0 1 2 3 4 5 6 7 8 9 10 ]

nix-repl> :l <nixos>

nix-repl> config.boot.kernelModules
[ "vboxdrv" "vboxnetadp" ... ]
2013-09-02 16:48:41 +00:00
Eelco Dolstra 3202206d1d Add a command :t for showing the type of an expression 2013-09-02 16:00:48 +00:00
Eelco Dolstra 287c88ca59 Support adding variables to the scope
The command ":a <expr>" evaluates <expr> and adds the attributes in
the resulting attribute set to the interpreter scope.  For instance:

nix-repl> :a import <nixpkgs> {}

nix-repl> lib.range 0 10
[ 0 1 2 3 4 5 6 7 8 9 10 ]
2013-09-02 17:53:58 +02:00
Eelco Dolstra e90569905e Initial version of nix-repl
This program interactively reads a Nix expression from the user,
evaluates it, and prints the result.
2013-09-02 15:18:15 +02:00
173 changed files with 7062 additions and 2675 deletions

No files matched your search

+26
View File
@@ -0,0 +1,26 @@
# EditorConfig configuration for nix
# http://EditorConfig.org
# Top-most EditorConfig file
root = true
# Unix-style newlines with a newline ending every file, utf-8 charset
[*]
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
charset = utf-8
# Match nix files, set indent to spaces with width of two
[*.nix]
indent_style = space
indent_size = 2
# Match c++/shell/perl, set indent to spaces with width of four
[*.{hpp,cc,hh,sh,pl}]
indent_style = space
indent_size = 4
# Match diffs, avoid to trim trailing whitespace
[*.{diff,patch}]
trim_trailing_whitespace = false
+2 -2
View File
@@ -1,4 +1,5 @@
Makefile.config
perl/Makefile.config
# /
/aclocal.m4
@@ -47,8 +48,7 @@ Makefile.config
/src/libexpr/nix.tbl
# /src/libstore/
/src/libstore/schema.sql.hh
/src/libstore/sandbox-defaults.sb
/src/libstore/*.gen.hh
/src/nix/nix
-1
View File
@@ -18,7 +18,6 @@ makefiles = \
src/nix-channel/local.mk \
src/nix-build/local.mk \
src/build-remote/local.mk \
perl/local.mk \
scripts/local.mk \
corepkgs/local.mk \
misc/systemd/local.mk \
+4 -4
View File
@@ -5,6 +5,7 @@ CXX = @CXX@
CXXFLAGS = @CXXFLAGS@
ENABLE_S3 = @ENABLE_S3@
HAVE_SODIUM = @HAVE_SODIUM@
HAVE_READLINE = @HAVE_READLINE@
LIBCURL_LIBS = @LIBCURL_LIBS@
OPENSSL_LIBS = @OPENSSL_LIBS@
PACKAGE_NAME = @PACKAGE_NAME@
@@ -14,7 +15,8 @@ LIBLZMA_LIBS = @LIBLZMA_LIBS@
SQLITE3_LIBS = @SQLITE3_LIBS@
bash = @bash@
bindir = @bindir@
curl = @curl@
bro = @bro@
lsof = @lsof@
datadir = @datadir@
datarootdir = @datarootdir@
docdir = @docdir@
@@ -24,11 +26,9 @@ libdir = @libdir@
libexecdir = @libexecdir@
localstatedir = @localstatedir@
mandir = @mandir@
perl = @perl@
perlbindings = @perlbindings@
perllibdir = @perllibdir@
pkglibdir = $(libdir)/$(PACKAGE_NAME)
prefix = @prefix@
sandbox_shell = @sandbox_shell@
storedir = @storedir@
sysconfdir = @sysconfdir@
doc_generate = @doc_generate@
+1 -1
View File
@@ -1,7 +1,7 @@
Nix, the purely functional package manager
------------------------------------------
Nix is a new take on package management that is fairly unique. Because of it's
Nix is a new take on package management that is fairly unique. Because of its
purity aspects, a lot of issues found in traditional package managers don't
appear with Nix.
+16 -54
View File
@@ -114,14 +114,12 @@ if test -z "$$1"; then
fi
])
NEED_PROG(curl, curl)
NEED_PROG(bash, bash)
NEED_PROG(patch, patch)
AC_PATH_PROG(xmllint, xmllint, false)
AC_PATH_PROG(xsltproc, xsltproc, false)
AC_PATH_PROG(flex, flex, false)
AC_PATH_PROG(bison, bison, false)
NEED_PROG(perl, perl)
NEED_PROG(sed, sed)
NEED_PROG(tar, tar)
NEED_PROG(bzip2, bzip2)
@@ -129,23 +127,8 @@ NEED_PROG(gzip, gzip)
NEED_PROG(xz, xz)
AC_PATH_PROG(dot, dot)
AC_PATH_PROG(pv, pv, pv)
# Test that Perl has the open/fork feature (Perl 5.8.0 and beyond).
AC_MSG_CHECKING([whether Perl is recent enough])
if ! $perl -e 'open(FOO, "-|", "true"); while (<FOO>) { print; }; close FOO or die;'; then
AC_MSG_RESULT(no)
AC_MSG_ERROR([Your Perl version is too old. Nix requires Perl 5.8.0 or newer.])
fi
AC_MSG_RESULT(yes)
# Figure out where to install Perl modules.
AC_MSG_CHECKING([for the Perl installation prefix])
perlversion=$($perl -e 'use Config; print $Config{version};')
perlarchname=$($perl -e 'use Config; print $Config{archname};')
AC_SUBST(perllibdir, [${libdir}/perl5/site_perl/$perlversion/$perlarchname])
AC_MSG_RESULT($perllibdir)
AC_PATH_PROG(bro, bro, bro)
AC_PATH_PROG(lsof, lsof, lsof)
NEED_PROG(cat, cat)
@@ -193,6 +176,13 @@ AC_SUBST(HAVE_SODIUM, [$have_sodium])
PKG_CHECK_MODULES([LIBLZMA], [liblzma], [CXXFLAGS="$LIBLZMA_CFLAGS $CXXFLAGS"])
# Look for libseccomp, required for Linux sandboxing.
if test "$sys_name" = linux; then
PKG_CHECK_MODULES([LIBSECCOMP], [libseccomp],
[CXXFLAGS="$LIBSECCOMP_CFLAGS $CXXFLAGS"])
fi
# Look for aws-cpp-sdk-s3.
AC_LANG_PUSH(C++)
AC_CHECK_HEADERS([aws/s3/S3Client.h],
@@ -213,40 +203,6 @@ if test "$gc" = yes; then
fi
# Check for the required Perl dependencies (DBI, DBD::SQLite).
perlFlags="-I$perllibdir"
AC_ARG_WITH(dbi, AC_HELP_STRING([--with-dbi=PATH],
[prefix of the Perl DBI library]),
perlFlags="$perlFlags -I$withval")
AC_ARG_WITH(dbd-sqlite, AC_HELP_STRING([--with-dbd-sqlite=PATH],
[prefix of the Perl DBD::SQLite library]),
perlFlags="$perlFlags -I$withval")
AC_MSG_CHECKING([whether DBD::SQLite works])
if ! $perl $perlFlags -e 'use DBI; use DBD::SQLite;' 2>&5; then
AC_MSG_RESULT(no)
AC_MSG_FAILURE([The Perl modules DBI and/or DBD::SQLite are missing.])
fi
AC_MSG_RESULT(yes)
AC_SUBST(perlFlags)
# Whether to build the Perl bindings
AC_MSG_CHECKING([whether to build the Perl bindings])
AC_ARG_ENABLE(perl-bindings, AC_HELP_STRING([--enable-perl-bindings],
[whether to build the Perl bindings (recommended) [default=yes]]),
perlbindings=$enableval, perlbindings=yes)
if test "$enable_shared" = no; then
# Perl bindings require shared libraries.
perlbindings=no
fi
AC_SUBST(perlbindings)
AC_MSG_RESULT($perlbindings)
AC_ARG_ENABLE(init-state, AC_HELP_STRING([--disable-init-state],
[do not initialise DB etc. in `make install']),
init_state=$enableval, init_state=yes)
@@ -265,7 +221,7 @@ AC_CHECK_FUNCS([setresuid setreuid lchown])
# Nice to have, but not essential.
AC_CHECK_FUNCS([strsignal posix_fallocate nanosleep sysconf])
AC_CHECK_FUNCS([strsignal posix_fallocate sysconf])
# This is needed if bzip2 is a static library, and the Nix libraries
@@ -291,6 +247,12 @@ fi
AC_SUBST(tarFlags)
AC_ARG_WITH(sandbox-shell, AC_HELP_STRING([--with-sandbox-shell=PATH],
[path of a statically-linked shell to use as /bin/sh in sandboxes]),
sandbox_shell=$withval)
AC_SUBST(sandbox_shell)
# Expand all variables in config.status.
test "$prefix" = NONE && prefix=$ac_default_prefix
test "$exec_prefix" = NONE && exec_prefix='${prefix}'
+3
View File
@@ -14,6 +14,9 @@ in rec {
nixBinDir = fromEnv "NIX_BIN_DIR" "@bindir@";
nixPrefix = "@prefix@";
nixLibexecDir = fromEnv "NIX_LIBEXEC_DIR" "@libexecdir@";
nixLocalstateDir = "@localstatedir@";
nixSysconfDir = "@sysconfdir@";
nixStoreDir = fromEnv "NIX_STORE_DIR" "@storedir@";
# If Nix is installed in the Nix store, then automatically add it as
# a dependency to the core packages. This ensures that they work
+3 -1
View File
@@ -15,7 +15,9 @@ let
else
${bzip2} -d < $src | ${tar} xf - ${tarFlags}
fi
mv * $out/$channelName
if [ * != $channelName ]; then
mv * $out/$channelName
fi
if [ -n "$binaryCacheURL" ]; then
mkdir $out/binary-caches
echo -n "$binaryCacheURL" > $out/binary-caches/$channelName
@@ -22,10 +22,7 @@ will call whenever it wants to build a derivation. The build hook
will perform it in the usual way if possible, or it can accept it, in
which case it is responsible for somehow getting the inputs of the
build to another machine, doing the build there, and getting the
results back. The details of the build hook protocol are described in
the documentation of the <link
linkend="envar-build-hook"><envar>NIX_BUILD_HOOK</envar>
variable</link>.</para>
results back.</para>
<example xml:id='ex-remote-systems'><title>Remote machine configuration:
<filename>remote-systems.conf</filename></title>
@@ -42,7 +39,7 @@ purposes. It uses <command>ssh</command> and
<command>nix-copy-closure</command> to copy the build inputs and
outputs and perform the remote build. To use it, you should set
<envar>NIX_BUILD_HOOK</envar> to
<filename><replaceable>prefix</replaceable>/libexec/nix/build-remote.pl</filename>.
<filename><replaceable>prefix</replaceable>/libexec/nix/build-remote</filename>.
You should also define a list of available build machines and point
the environment variable <envar>NIX_REMOTE_SYSTEMS</envar> to
it. <envar>NIX_REMOTE_SYSTEMS</envar> must be an absolute path. An
@@ -68,7 +65,7 @@ bits of information:
should not have a passphrase!</para></listitem>
<listitem><para>The maximum number of builds that
<filename>build-remote.pl</filename> will execute in parallel on the
<filename>build-remote</filename> will execute in parallel on the
machine. Typically this should be equal to the number of CPU cores.
For instance, the machine <literal>itchy</literal> in the example
will execute up to 8 builds in parallel.</para></listitem>
@@ -80,7 +77,7 @@ bits of information:
<listitem><para>A comma-separated list of <emphasis>supported
features</emphasis>. If a derivation has the
<varname>requiredSystemFeatures</varname> attribute, then
<filename>build-remote.pl</filename> will only perform the
<filename>build-remote</filename> will only perform the
derivation on a machine that has the specified features. For
instance, the attribute
@@ -103,14 +100,6 @@ requiredSystemFeatures = [ "kvm" ];
</orderedlist>
You should also set up the environment variable
<envar>NIX_CURRENT_LOAD</envar> to point at a directory (e.g.,
<filename>/var/run/nix/current-load</filename>) that
<filename>build-remote.pl</filename> uses to remember how many builds
it is currently executing remotely. It doesn't look at the actual
load on the remote machine, so if you have multiple instances of Nix
running, they should use the same <envar>NIX_CURRENT_LOAD</envar>
file. Maybe in the future <filename>build-remote.pl</filename> will
look at the actual remote load.</para>
</para>
</chapter>
+59 -60
View File
@@ -17,13 +17,32 @@
<refsection><title>Description</title>
<para>A number of persistent settings of Nix are stored in the file
<filename><replaceable>sysconfdir</replaceable>/nix/nix.conf</filename> or
<filename>$NIX_CONF_DIR/nix.conf</filename> if <envar>NIX_CONF_DIR</envar> is set.
This file is a list of <literal><replaceable>name</replaceable> =
<para>Nix reads settings from two configuration files:</para>
<itemizedlist>
<listitem>
<para>The system-wide configuration file
<filename><replaceable>sysconfdir</replaceable>/nix/nix.conf</filename>
(i.e. <filename>/etc/nix/nix.conf</filename> on most systems), or
<filename>$NIX_CONF_DIR/nix.conf</filename> if
<envar>NIX_CONF_DIR</envar> is set.</para>
</listitem>
<listitem>
<para>The user configuration file
<filename>$XDG_CONFIG_HOME/nix/nix.conf</filename>, or
<filename>~/.config/nix/nix.conf</filename> if
<envar>XDG_CONFIG_HOME</envar> is not set.</para>
</listitem>
</itemizedlist>
<para>The configuration files consist of
<literal><replaceable>name</replaceable> =
<replaceable>value</replaceable></literal> pairs, one per line.
Comments start with a <literal>#</literal> character. Here is an example
configuration file:</para>
Comments start with a <literal>#</literal> character. Here is an
example configuration file:</para>
<programlisting>
gc-keep-outputs = true # Nice for developers
@@ -31,8 +50,9 @@ gc-keep-derivations = true # Idem
env-keep-derivations = false
</programlisting>
<para>You can override settings using the <option>--option</option>
flag, e.g. <literal>--option gc-keep-outputs false</literal>.</para>
<para>You can override settings on the command line using the
<option>--option</option> flag, e.g. <literal>--option gc-keep-outputs
false</literal>.</para>
<para>The following settings are currently available:
@@ -101,9 +121,9 @@ flag, e.g. <literal>--option gc-keep-outputs false</literal>.</para>
<listitem><para>This option defines the maximum number of jobs
that Nix will try to build in parallel. The default is
<literal>1</literal>. You should generally set it to the number
of CPUs in your system (e.g., <literal>2</literal> on an Athlon 64
X2). It can be overridden using the <option
<literal>1</literal>. The special value <literal>auto</literal>
causes Nix to use the number of CPUs in your system. It can be
overridden using the <option
linkend='opt-max-jobs'>--max-jobs</option> (<option>-j</option>)
command line switch.</para></listitem>
@@ -334,25 +354,16 @@ flag, e.g. <literal>--option gc-keep-outputs false</literal>.</para>
</varlistentry>
<varlistentry><term><literal>use-binary-caches</literal></term>
<varlistentry><term><literal>substituters</literal></term>
<listitem><para>If set to <literal>true</literal> (the default),
Nix will check the binary caches specified by
<option>binary-caches</option> and related options to obtain
binary substitutes.</para></listitem>
</varlistentry>
<varlistentry><term><literal>binary-caches</literal></term>
<listitem><para>A list of URLs of binary caches, separated by
<listitem><para>A list of URLs of substituters, separated by
whitespace. The default is
<literal>https://cache.nixos.org</literal>.</para></listitem>
</varlistentry>
<!--
<varlistentry><term><literal>binary-caches-files</literal></term>
<listitem><para>A list of names of files that will be read to
@@ -365,28 +376,28 @@ flag, e.g. <literal>--option gc-keep-outputs false</literal>.</para>
option to read files created by untrusted users!</para></listitem>
</varlistentry>
-->
<varlistentry><term><literal>trusted-binary-caches</literal></term>
<varlistentry><term><literal>trusted-substituters</literal></term>
<listitem><para>A list of URLs of binary caches, separated by
<listitem><para>A list of URLs of substituters, separated by
whitespace. These are not used by default, but can be enabled by
users of the Nix daemon by specifying <literal>--option
binary-caches <replaceable>urls</replaceable></literal> on the
substituters <replaceable>urls</replaceable></literal> on the
command line. Unprivileged users are only allowed to pass a
subset of the URLs listed in <literal>binary-caches</literal> and
<literal>trusted-binary-caches</literal>.</para></listitem>
subset of the URLs listed in <literal>substituters</literal> and
<literal>trusted-substituters</literal>.</para></listitem>
</varlistentry>
<varlistentry><term><literal>extra-binary-caches</literal></term>
<varlistentry><term><literal>extra-substituters</literal></term>
<listitem><para>Additional binary caches appended to those
specified in <option>binary-caches</option> and
<option>binary-caches-files</option>. When used by unprivileged
users, untrusted binary caches (i.e. those not listed in
<option>trusted-binary-caches</option>) are silently
specified in <option>substituters</option>. When used by
unprivileged users, untrusted substituters (i.e. those not listed
in <option>trusted-substituters</option>) are silently
ignored.</para></listitem>
</varlistentry>
@@ -394,9 +405,10 @@ flag, e.g. <literal>--option gc-keep-outputs false</literal>.</para>
<varlistentry><term><literal>signed-binary-caches</literal></term>
<listitem><para>If set to <literal>*</literal>, Nix will only
download binaries if they are signed using one of the keys listed
in <option>binary-cache-public-keys</option>.</para></listitem>
<listitem><para>If set to <literal>*</literal> (the default), Nix
will only download binaries if they are signed using one of the
keys listed in <option>binary-cache-public-keys</option>. Set to
the empty string to disable signature checking.</para></listitem>
</varlistentry>
@@ -412,7 +424,7 @@ flag, e.g. <literal>--option gc-keep-outputs false</literal>.</para>
</varlistentry>
<varlistentry><term><literal>binary-caches-parallel-connections</literal></term>
<varlistentry><term><literal>http-connections</literal></term>
<listitem><para>The maximum number of parallel TCP connections
used to fetch files from binary caches and by other downloads. It
@@ -421,15 +433,6 @@ flag, e.g. <literal>--option gc-keep-outputs false</literal>.</para>
</varlistentry>
<varlistentry><term><literal>verify-https-binary-caches</literal></term>
<listitem><para>Whether HTTPS binary caches are required to have a
certificate that can be verified. Defaults to
<literal>true</literal>.</para></listitem>
</varlistentry>
<varlistentry><term><literal>netrc-file</literal></term>
<listitem><para>If set to an absolute path to a <filename>netrc</filename>
@@ -512,20 +515,6 @@ password <replaceable>my-password</replaceable>
</varlistentry>
<varlistentry xml:id="conf-log-servers"><term><literal>log-servers</literal></term>
<listitem>
<para>A list of URL prefixes (such as
<literal>http://hydra.nixos.org/log</literal>) from which
<command>nix-store -l</command> will try to fetch build logs if
they’re not available locally.</para>
</listitem>
</varlistentry>
<varlistentry xml:id="conf-trusted-users"><term><literal>trusted-users</literal></term>
<listitem>
@@ -644,6 +633,16 @@ password <replaceable>my-password</replaceable>
</varlistentry>
<varlistentry xml:id="conf-allow-import-from-derivation"><term><literal>allow-import-from-derivation</literal></term>
<listitem><para>By default, Nix allows you to <function>import</function> from a derivation,
allowing building at evaluation time. With this option set to false, Nix will throw an error
when evaluating an expression that uses this feature, allowing users to ensure their evaluation
will not require any builds to take place.</para></listitem>
</varlistentry>
</variablelist>
</para>
-133
View File
@@ -148,139 +148,6 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
</varlistentry>
<varlistentry xml:id="envar-build-hook"><term><envar>NIX_BUILD_HOOK</envar></term>
<listitem>
<para>Specifies the location of the <emphasis>build hook</emphasis>,
which is a program (typically some script) that Nix will call
whenever it wants to build a derivation. This is used to implement
distributed builds<phrase condition="manual"> (see <xref
linkend="chap-distributed-builds" />)</phrase>.</para>
<!--
The protocol by
which the calling Nix process and the build hook communicate is as
follows.
<para>The build hook is called with the following command-line
arguments:
<orderedlist>
<listitem><para>A boolean value <literal>0</literal> or
<literal>1</literal> specifying whether Nix can locally execute
more builds, as per the <link
linkend="opt-max-jobs"><option>- -max-jobs</option> option</link>.
The purpose of this argument is to allow the hook to not have to
maintain bookkeeping for the local machine.</para></listitem>
<listitem><para>The Nix platform identifier for the local machine
(e.g., <literal>i686-linux</literal>).</para></listitem>
<listitem><para>The Nix platform identifier for the derivation,
i.e., its <link linkend="attr-system"><varname>system</varname>
attribute</link>.</para></listitem>
<listitem><para>The store path of the derivation.</para></listitem>
</orderedlist>
</para>
<para>On the basis of this information, and whatever persistent
state the build hook keeps about other machines and their current
load, it has to decide what to do with the build. It should print
out on standard error one of the following responses (terminated by
a newline, <literal>"\n"</literal>):
<variablelist>
<varlistentry><term><literal># decline</literal></term>
<listitem><para>The build hook is not willing or able to perform
the build; the calling Nix process should do the build itself,
if possible.</para></listitem>
</varlistentry>
<varlistentry><term><literal># postpone</literal></term>
<listitem><para>The build hook cannot perform the build now, but
can do so in the future (e.g., because all available build slots
on remote machines are in use). The calling Nix process should
postpone this build until at least one currently running build
has terminated.</para></listitem>
</varlistentry>
<varlistentry><term><literal># accept</literal></term>
<listitem><para>The build hook has accepted the
build.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>After sending <literal># accept</literal>, the hook should
read one line from standard input, which will be the string
<literal>okay</literal>. It can then proceed with the build.
Before sending <literal>okay</literal>, Nix will store in the hook’s
current directory a number of text files that contain information
about the derivation:
<variablelist>
<varlistentry><term><filename>inputs</filename></term>
<listitem><para>The set of store paths that are inputs to the
build process (one per line). These have to be copied
<emphasis>to</emphasis> the remote machine (in addition to the
store derivation itself).</para></listitem>
</varlistentry>
<varlistentry><term><filename>outputs</filename></term>
<listitem><para>The set of store paths that are outputs of the
derivation (one per line). These have to be copied
<emphasis>from</emphasis> the remote machine if the build
succeeds.</para></listitem>
</varlistentry>
<varlistentry><term><filename>references</filename></term>
<listitem><para>The reference graph of the inputs, in the format
accepted by the command <command>nix-store
- -register-validity</command>. It is necessary to run this
command on the remote machine after copying the inputs to inform
Nix on the remote machine that the inputs are valid
paths.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>The hook should copy the inputs to the remote machine,
register the validity of the inputs, perform the remote build, and
copy the outputs back to the local machine. An exit code other than
<literal>0</literal> indicates that the hook has failed. An exit
code equal to 100 means that the remote build failed (as opposed to,
e.g., a network error).</para>
-->
</listitem>
</varlistentry>
<varlistentry xml:id="envar-remote"><term><envar>NIX_REMOTE</envar></term>
<listitem><para>This variable should be set to
+1 -1
View File
@@ -1136,7 +1136,7 @@ user environment elements, etc. -->
<listitem><para>Print all of the meta-attributes of the
derivation. This option is only available with
<option>--xml</option>.</para></listitem>
<option>--xml</option> or <option>--json</option>.</para></listitem>
</varlistentry>
+8 -10
View File
@@ -117,21 +117,19 @@ input.</para>
<varlistentry><term><option>--xml</option></term>
<listitem><para>When used with <option>--parse</option> and
<option>--eval</option>, print the resulting expression as an
XML representation of the abstract syntax tree rather than as an
ATerm. The schema is the same as that used by the <link
linkend="builtin-toXML"><function>toXML</function>
built-in</link>.</para></listitem>
<listitem><para>When used with <option>--eval</option>, print the resulting
value as an XML representation of the abstract syntax tree rather than as
an ATerm. The schema is the same as that used by the <link
linkend="builtin-toXML"><function>toXML</function> built-in</link>.
</para></listitem>
</varlistentry>
<varlistentry><term><option>--json</option></term>
<listitem><para>When used with <option>--parse</option> and
<option>--eval</option>, print the resulting expression as an
JSON representation of the abstract syntax tree rather than as an
ATerm.</para></listitem>
<listitem><para>When used with <option>--eval</option>, print the resulting
value as an JSON representation of the abstract syntax tree rather
than as an ATerm.</para></listitem>
</varlistentry>
+1 -6
View File
@@ -1236,12 +1236,7 @@ the store path is used.</para>
<filename>/nix/var/log/nix/drvs</filename>. However, there is no
guarantee that a build log is available for any particular store path.
For instance, if the path was downloaded as a pre-built binary through
a substitute, then the log is unavailable. If the log is not available
locally, then <command>nix-store</command> will try to download the
log from the servers specified in the Nix option
<option>log-servers</option>. For example, if it’s set to
<literal>http://hydra.nixos.org/log</literal>, then Nix will check
<literal>http://hydra.nixos.org/log/<replaceable>base-name</replaceable></literal>.</para>
a substitute, then the log is unavailable.</para>
</refsection>
+3 -2
View File
@@ -93,8 +93,9 @@
<term><option>-j</option></term>
<listitem><para>Sets the maximum number of build jobs that Nix will
perform in parallel to the specified number. The default is
specified by the <link
perform in parallel to the specified number. Specify
<literal>auto</literal> to use the number of CPUs in the system.
The default is specified by the <link
linkend='conf-build-max-jobs'><literal>build-max-jobs</literal></link>
configuration setting, which itself defaults to
<literal>1</literal>. A higher value is useful on SMP systems or to
+43 -6
View File
@@ -65,7 +65,7 @@ available as <function>builtins.derivation</function>.</para>
<replaceable>set</replaceable></term>
<listitem><para>Return the names of the attributes in the set
<replaceable>set</replaceable> in a sorted list. For instance,
<replaceable>set</replaceable> in an alphabetically sorted list. For instance,
<literal>builtins.attrNames { y = 1; x = "foo"; }</literal>
evaluates to <literal>[ "x" "y" ]</literal>.</para></listitem>
@@ -213,10 +213,11 @@ if builtins ? getEnv then builtins.getEnv "PATH" else ""</programlisting>
<varlistentry><term><function>builtins.match</function>
<replaceable>regex</replaceable> <replaceable>str</replaceable></term>
<listitem><para>Returns a list if
<replaceable>regex</replaceable> matches
<replaceable>str</replaceable> precisely, otherwise returns <literal>null</literal>.
Each item in the list is a regex group.
<listitem><para>Returns a list if the <link
xlink:href="http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_04">extended
POSIX regular expression</link> <replaceable>regex</replaceable>
matches <replaceable>str</replaceable> precisely, otherwise returns
<literal>null</literal>. Each item in the list is a regex group.
<programlisting>
builtins.match "ab" "abc"
@@ -236,6 +237,11 @@ builtins.match "a(b)(c)" "abc"
Evaluates to <literal>[ "b" "c" ]</literal>.
<programlisting>
builtins.match "[[:space:]]+([[:upper:]]+)[[:space:]]+" " FOO "
</programlisting>
Evaluates to <literal>[ "foo" ]</literal>.
</para></listitem>
</varlistentry>
@@ -292,6 +298,24 @@ with import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/nixo
stdenv.mkDerivation { … }
</programlisting>
Note that when obtaining the hash with <varname>nix-prefetch-url
</varname> the option <varname>--unpack</varname> is required.
</para>
<para>This function can also verify the contents against a hash.
In that case, the function takes a set instead of a URL. The set
requires the attribute <varname>url</varname> and the attribute
<varname>sha256</varname>, e.g.
<programlisting>
with import (fetchTarball {
url = https://github.com/NixOS/nixpkgs-channels/archive/nixos-14.12.tar.gz;
sha256 = "1jppksrfvbk5ypiqdz4cddxdl8z6zyzdb2srq8fcffr327ld5jj2";
}) {};
stdenv.mkDerivation { … }
</programlisting>
</para>
<para>This function is not available if <link
@@ -416,7 +440,7 @@ builtins.fromJSON ''{"x": [1, 2, 3], "y": null}''
<listitem><para>Generate list of size
<replaceable>length</replaceable>, with each element
<replaceable>i></replaceable> equal to the value returned by
<replaceable>i</replaceable> equal to the value returned by
<replaceable>generator</replaceable> <literal>i</literal>. For
example,
@@ -1163,6 +1187,19 @@ stdenv.mkDerivation (rec {
</varlistentry>
<varlistentry><term><function>builtins.tryEval</function>
<replaceable>e</replaceable></term>
<listitem><para>Try to evaluate <replaceable>e</replaceable>.
Return a set containing the attributes <literal>success</literal>
(<literal>true</literal> if <replaceable>e</replaceable> evaluated
successfully, <literal>false</literal> if an error was thrown) and
<literal>value</literal>, equalling <replaceable>e</replaceable>
if successful and <literal>false</literal> otherwise.
</para></listitem>
</varlistentry>
<varlistentry><term><function>builtins.typeOf</function>
<replaceable>e</replaceable></term>
+1 -1
View File
@@ -100,7 +100,7 @@ outputs = [ "lib" "headers" "doc" ];
<programlisting>
buildInputs = [ pkg.lib pkg.headers ];
</programlisting>
The first element of <varname>output</varname> determines the
The first element of <varname>outputs</varname> determines the
<emphasis>default output</emphasis>. Thus, you could also write
<programlisting>
buildInputs = [ pkg pkg.headers ];
@@ -12,8 +12,6 @@
<listitem><para>A version of GCC or Clang that supports C++11.</para></listitem>
<listitem><para>Perl 5.8 or higher.</para></listitem>
<listitem><para><command>pkg-config</command> to locate
dependencies. If your distribution does not provide it, you can get
it from <link
@@ -34,11 +32,6 @@
or higher. If your distribution does not provide it, please install
it from <link xlink:href="http://www.sqlite.org/" />.</para></listitem>
<listitem><para>The Perl DBI and DBD::SQLite libraries, which are
available from <link
xlink:href="http://search.cpan.org/">CPAN</link> if your
distribution does not provide them.</para></listitem>
<listitem><para>The <link
xlink:href="http://www.hboehm.info/gc/">Boehm
garbage collector</link> to reduce the evaluator’s memory
+7 -1
View File
@@ -93,7 +93,7 @@ time window in which the package has some files from the old version
and some files from the new version — which would be bad because a
program might well crash if it’s started during that period.</para>
<para>And since package aren’t overwritten, the old versions are still
<para>And since packages aren’t overwritten, the old versions are still
there after an upgrade. This means that you can <emphasis>roll
back</emphasis> to the old version:</para>
@@ -261,6 +261,12 @@ xlink:href="http://nixos.org/">NixOS homepage</link>.</para>
xlink:href="http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html">GNU
LGPLv2.1 or (at your option) any later version</link>.</para>
<para>Nix uses the <link
xlink:href="https://github.com/antirez/linenoise">linenoise
library</link>, which has the following license:</para>
<programlisting><xi:include href="../../../src/linenoise/LICENSE" parse="text" /></programlisting>
</simplesect>
+1 -1
View File
@@ -40,7 +40,7 @@ libxslt-1.1.28
<step><para>Install some packages from the channel:
<screen>
$ nix-env -i hello <replaceable>...</replaceable> </screen>
$ nix-env -i hello</screen>
This should download pre-built packages; it should not build them
locally (if it does, something went wrong).</para></step>
@@ -13,6 +13,7 @@
-->
<xi:include href="rl-1.12.xml" />
<xi:include href="rl-1.11.10.xml" />
<xi:include href="rl-1.11.xml" />
<xi:include href="rl-1.10.xml" />
<xi:include href="rl-1.9.xml" />
+31
View File
@@ -0,0 +1,31 @@
<section xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="ssec-relnotes-1.11.10">
<title>Release 1.11.10 (2017-06-12)</title>
<para>This release fixes a security bug in Nix’s “build user” build
isolation mechanism. Previously, Nix builders had the ability to
create setuid binaries owned by a <literal>nixbld</literal>
user. Such a binary could then be used by an attacker to assume a
<literal>nixbld</literal> identity and interfere with subsequent
builds running under the same UID.</para>
<para>To prevent this issue, Nix now disallows builders to create
setuid and setgid binaries. On Linux, this is done using a seccomp BPF
filter. Note that this imposes a small performance penalty (e.g. 1%
when building GNU Hello). Using seccomp, we now also prevent the
creation of extended attributes and POSIX ACLs since these cannot be
represented in the NAR format and (in the case of POSIX ACLs) allow
bypassing regular Nix store permissions. On macOS, the restriction is
implemented using the existing sandbox mechanism, which now uses a
minimal “allow all except the creation of setuid/setgid binaries”
profile when regular sandboxing is disabled. On other platforms, the
“build user” mechanism is now disabled.</para>
<para>Thanks go to Linus Heckemann for discovering and reporting this
bug.</para>
</section>
-7
View File
@@ -121,13 +121,6 @@ $ diffoscope /nix/store/11a27shh6n2i…-zlib-1.2.8 /nix/store/11a27shh6n2i…-zl
also improves performance.</para>
</listitem>
<listitem>
<para>The Nix language now supports floating point numbers. They are
based on regular C++ <literal>float</literal> and compatible with
existing integers and number-related operations. Export and import to and
from JSON and XML works, too.
</para>
</listitem>
<listitem>
<para>All "chroot"-containing strings got renamed to "sandbox".
In particular, some Nix options got renamed, but the old names
+7
View File
@@ -17,6 +17,13 @@
have write access to the Nix database.</para>
</listitem>
<listitem>
<para>The Nix language now supports floating point numbers. They are
based on regular C++ <literal>float</literal> and compatible with
existing integers and number-related operations. Export and import to and
from JSON and XML works, too.
</para>
</listitem>
</itemizedlist>
<para>This release has contributions from TBD.</para>
+2 -2
View File
@@ -83,8 +83,8 @@ $ nix-store -l $(which xterm)
caches).</para></listitem>
<listitem><para>The configuration option
<option>build-max-jobs</option> now defaults to the number of
available CPU cores.</para></listitem>
<option>build-cores</option> now defaults to the number of available
CPU cores.</para></listitem>
<listitem><para>Build users are now used by default when Nix is
invoked as root. This prevents builds from accidentally running as
+2 -3
View File
@@ -3,12 +3,11 @@ ifeq ($(MAKECMDGOALS), dist)
dist-files += $(shell git --git-dir=.git ls-files || find * -type f)
endif
dist-files += configure config.h.in nix.spec
dist-files += configure config.h.in nix.spec perl/configure
clean-files += Makefile.config
GLOBAL_CXXFLAGS += -I . -I src -I src/libutil -I src/libstore -I src/libmain -I src/libexpr \
-Wno-unneeded-internal-declaration
GLOBAL_CXXFLAGS += -I . -I src -I src/libutil -I src/libstore -I src/libmain -I src/libexpr
$(foreach i, config.h $(call rwildcard, src/lib*, *.hh), \
$(eval $(call install-file-in, $(i), $(includedir)/nix, 0644)))
+1 -1
View File
@@ -1,6 +1,6 @@
FROM alpine
RUN wget -O- http://nixos.org/releases/nix/nix-1.11.2/nix-1.11.2-x86_64-linux.tar.bz2 | bzcat - | tar xf - \
RUN wget -O- http://nixos.org/releases/nix/nix-1.11.7/nix-1.11.7-x86_64-linux.tar.bz2 | bzcat - | tar xf - \
&& echo "nixbld:x:30000:nixbld1,nixbld2,nixbld3,nixbld4,nixbld5,nixbld6,nixbld7,nixbld8,nixbld9,nixbld10,nixbld11,nixbld12,nixbld13,nixbld14,nixbld15,nixbld16,nixbld17,nixbld18,nixbld19,nixbld20,nixbld21,nixbld22,nixbld23,nixbld24,nixbld25,nixbld26,nixbld27,nixbld28,nixbld29,nixbld30" >> /etc/group \
&& for i in $(seq 1 30); do echo "nixbld$i:x:$((30000 + $i)):30000:::" >> /etc/passwd; done \
&& mkdir -m 0755 /nix && USER=root sh nix-*-x86_64-linux/install \
@@ -12,10 +12,5 @@
<string>/var/log/nix-daemon.log</string>
<key>StandardOutPath</key>
<string>/dev/null</string>
<key>EnvironmentVariables</key>
<dict>
<key>NIX_SSL_CERT_FILE</key>
<string>/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt</string>
</dict>
</dict>
</plist>
-1
View File
@@ -7,4 +7,3 @@ ConditionPathIsReadWrite=@localstatedir@/nix/daemon-socket
[Service]
ExecStart=@@bindir@/nix-daemon nix-daemon --daemon
KillMode=process
Environment=XDG_CACHE_HOME=/root/.cache
+2 -2
View File
@@ -53,8 +53,8 @@ BUILD_SHARED_LIBS ?= 1
ifeq ($(BUILD_SHARED_LIBS), 1)
ifeq (CYGWIN,$(findstring CYGWIN,$(OS)))
GLOBAL_CFLAGS += -U__STRICT_ANSI__
GLOBAL_CXXFLAGS += -U__STRICT_ANSI__
GLOBAL_CFLAGS += -U__STRICT_ANSI__ -D_GNU_SOURCE
GLOBAL_CXXFLAGS += -U__STRICT_ANSI__ -D_GNU_SOURCE
else
GLOBAL_CFLAGS += -fPIC
GLOBAL_CXXFLAGS += -fPIC
+2 -12
View File
@@ -16,18 +16,15 @@ Source0: %{name}-%{version}.tar.bz2
%if 0%{?el5}
BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)
%endif
BuildRequires: perl(DBD::SQLite)
BuildRequires: perl(DBI)
BuildRequires: perl(ExtUtils::ParseXS)
Requires: /usr/bin/perl
Requires: curl
Requires: perl-DBD-SQLite
Requires: bzip2
Requires: gzip
Requires: xz
Requires: libseccomp
BuildRequires: bzip2-devel
BuildRequires: sqlite-devel
BuildRequires: libcurl-devel
BuildRequires: libseccomp-devel
# Hack to make that shitty RPM scanning hack shut up.
Provides: perl(Nix::SSH)
@@ -92,11 +89,6 @@ the emacs-%{name} package to edit Nix expressions with GNU Emacs.
%prep
%setup -q
# Install Perl modules to vendor_perl
# configure.ac need to be changed to make this global; however, this will
# also affect NixOS. Use discretion.
%{__sed} -i 's|perl5/site_perl/$perlversion/$perlarchname|perl5/vendor_perl|' \
configure
%build
@@ -169,8 +161,6 @@ systemctl start nix-daemon.socket
%files
%{_bindir}/nix*
%{_libdir}/*.so
%{perl_vendorarch}/*
%exclude %dir %{perl_vendorarch}/auto/
%{_prefix}/libexec/*
%if ! 0%{?rhel} || 0%{?rhel} >= 7
%{_prefix}/lib/systemd/system/nix-daemon.socket
+14
View File
@@ -0,0 +1,14 @@
makefiles = local.mk
GLOBAL_CXXFLAGS += -std=c++14 -g -Wall
-include Makefile.config
OPTIMIZE = 1
ifeq ($(OPTIMIZE), 1)
GLOBAL_CFLAGS += -O3
GLOBAL_CXXFLAGS += -O3
endif
include mk/lib.mk
+18
View File
@@ -0,0 +1,18 @@
CC = @CC@
CFLAGS = @CFLAGS@
CXX = @CXX@
CXXFLAGS = @CXXFLAGS@
HAVE_SODIUM = @HAVE_SODIUM@
PACKAGE_NAME = @PACKAGE_NAME@
PACKAGE_VERSION = @PACKAGE_VERSION@
SODIUM_LIBS = @SODIUM_LIBS@
NIX_CFLAGS = @NIX_CFLAGS@
NIX_LIBS = @NIX_LIBS@
nixbindir = @nixbindir@
curl = @curl@
nixlibexecdir = @nixlibexecdir@
nixlocalstatedir = @nixlocalstatedir@
perl = @perl@
perllibdir = @perllibdir@
nixstoredir = @nixstoredir@
nixsysconfdir = @nixsysconfdir@
+99
View File
@@ -0,0 +1,99 @@
AC_INIT(nix-perl, m4_esyscmd([bash -c "echo -n $(cat ../version)$VERSION_SUFFIX"]))
AC_CONFIG_SRCDIR(MANIFEST)
AC_CONFIG_AUX_DIR(../config)
CFLAGS=
CXXFLAGS=
AC_PROG_CC
AC_PROG_CXX
AX_CXX_COMPILE_STDCXX_11
# Use 64-bit file system calls so that we can support files > 2 GiB.
AC_SYS_LARGEFILE
AC_DEFUN([NEED_PROG],
[
AC_PATH_PROG($1, $2)
if test -z "$$1"; then
AC_MSG_ERROR([$2 is required])
fi
])
NEED_PROG(perl, perl)
NEED_PROG(curl, curl)
NEED_PROG(bzip2, bzip2)
NEED_PROG(xz, xz)
# Test that Perl has the open/fork feature (Perl 5.8.0 and beyond).
AC_MSG_CHECKING([whether Perl is recent enough])
if ! $perl -e 'open(FOO, "-|", "true"); while (<FOO>) { print; }; close FOO or die;'; then
AC_MSG_RESULT(no)
AC_MSG_ERROR([Your Perl version is too old. Nix requires Perl 5.8.0 or newer.])
fi
AC_MSG_RESULT(yes)
# Figure out where to install Perl modules.
AC_MSG_CHECKING([for the Perl installation prefix])
perlversion=$($perl -e 'use Config; print $Config{version};')
perlarchname=$($perl -e 'use Config; print $Config{archname};')
AC_SUBST(perllibdir, [${libdir}/perl5/site_perl/$perlversion/$perlarchname])
AC_MSG_RESULT($perllibdir)
# Look for libsodium, an optional dependency.
PKG_CHECK_MODULES([SODIUM], [libsodium],
[AC_DEFINE([HAVE_SODIUM], [1], [Whether to use libsodium for cryptography.])
CXXFLAGS="$SODIUM_CFLAGS $CXXFLAGS"
have_sodium=1], [have_sodium=])
AC_SUBST(HAVE_SODIUM, [$have_sodium])
# Check for the required Perl dependencies (DBI and DBD::SQLite).
perlFlags="-I$perllibdir"
AC_ARG_WITH(dbi, AC_HELP_STRING([--with-dbi=PATH],
[prefix of the Perl DBI library]),
perlFlags="$perlFlags -I$withval")
AC_ARG_WITH(dbd-sqlite, AC_HELP_STRING([--with-dbd-sqlite=PATH],
[prefix of the Perl DBD::SQLite library]),
perlFlags="$perlFlags -I$withval")
AC_MSG_CHECKING([whether DBD::SQLite works])
if ! $perl $perlFlags -e 'use DBI; use DBD::SQLite;' 2>&5; then
AC_MSG_RESULT(no)
AC_MSG_FAILURE([The Perl modules DBI and/or DBD::SQLite are missing.])
fi
AC_MSG_RESULT(yes)
AC_SUBST(perlFlags)
PKG_CHECK_MODULES([NIX], [nix-store])
NEED_PROG([NIX_INSTANTIATE_PROGRAM], [nix-instantiate])
# Get nix configure values
nixbindir=$("$NIX_INSTANTIATE_PROGRAM" --eval '<nix/config.nix>' -A nixBinDir | tr -d \")
nixlibexecdir=$("$NIX_INSTANTIATE_PROGRAM" --eval '<nix/config.nix>' -A nixLibexecDir | tr -d \")
nixlocalstatedir=$("$NIX_INSTANTIATE_PROGRAM" --eval '<nix/config.nix>' -A nixLocalstateDir | tr -d \")
nixsysconfdir=$("$NIX_INSTANTIATE_PROGRAM" --eval '<nix/config.nix>' -A nixSysconfDir | tr -d \")
nixstoredir=$("$NIX_INSTANTIATE_PROGRAM" --eval '<nix/config.nix>' -A nixStoreDir | tr -d \")
AC_SUBST(nixbindir)
AC_SUBST(nixlibexecdir)
AC_SUBST(nixlocalstatedir)
AC_SUBST(nixsysconfdir)
AC_SUBST(nixstoredir)
# Expand all variables in config.status.
test "$prefix" = NONE && prefix=$ac_default_prefix
test "$exec_prefix" = NONE && exec_prefix='${prefix}'
for name in $ac_subst_vars; do
declare $name="$(eval echo "${!name}")"
declare $name="$(eval echo "${!name}")"
declare $name="$(eval echo "${!name}")"
done
rm -f Makefile.config
ln -sfn ../mk mk
AC_CONFIG_FILES([])
AC_OUTPUT
+14 -31
View File
@@ -4,48 +4,31 @@ use MIME::Base64;
$version = "@PACKAGE_VERSION@";
$binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
$libexecDir = $ENV{"NIX_LIBEXEC_DIR"} || "@libexecdir@";
$stateDir = $ENV{"NIX_STATE_DIR"} || "@localstatedir@/nix";
$logDir = $ENV{"NIX_LOG_DIR"} || "@localstatedir@/log/nix";
$confDir = $ENV{"NIX_CONF_DIR"} || "@sysconfdir@/nix";
$storeDir = $ENV{"NIX_STORE_DIR"} || "@storedir@";
$binDir = $ENV{"NIX_BIN_DIR"} || "@nixbindir@";
$libexecDir = $ENV{"NIX_LIBEXEC_DIR"} || "@nixlibexecdir@";
$stateDir = $ENV{"NIX_STATE_DIR"} || "@nixlocalstatedir@/nix";
$logDir = $ENV{"NIX_LOG_DIR"} || "@nixlocalstatedir@/log/nix";
$confDir = $ENV{"NIX_CONF_DIR"} || "@nixsysconfdir@/nix";
$storeDir = $ENV{"NIX_STORE_DIR"} || "@nixstoredir@";
$bzip2 = "@bzip2@";
$xz = "@xz@";
$curl = "@curl@";
$useBindings = "@perlbindings@" eq "yes";
$useBindings = 1;
%config = ();
%binaryCachePublicKeys = ();
$defaultPublicKeys = "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=";
sub readConfig {
if (defined $ENV{'_NIX_OPTIONS'}) {
foreach my $s (split '\n', $ENV{'_NIX_OPTIONS'}) {
my ($n, $v) = split '=', $s, 2;
$config{$n} = $v;
}
} else {
my $config = "$confDir/nix.conf";
return unless -f $config;
my $config = "$confDir/nix.conf";
return unless -f $config;
open CONFIG, "<$config" or die "cannot open ‘$config’";
while (<CONFIG>) {
/^\s*([\w\-\.]+)\s*=\s*(.*)$/ or next;
$config{$1} = $2;
}
close CONFIG;
}
foreach my $s (split(/ /, $config{"binary-cache-public-keys"} // $defaultPublicKeys)) {
my ($keyName, $publicKey) = split ":", $s;
next unless defined $keyName && defined $publicKey;
$binaryCachePublicKeys{$keyName} = decode_base64($publicKey);
open CONFIG, "<$config" or die "cannot open ‘$config’";
while (<CONFIG>) {
/^\s*([\w\-\.]+)\s*=\s*(.*)$/ or next;
$config{$1} = $2;
}
close CONFIG;
}
return 1;
+2 -2
View File
@@ -1,3 +1,5 @@
#include "config.h"
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
@@ -25,9 +27,7 @@ static ref<Store> store()
static std::shared_ptr<Store> _store;
if (!_store) {
try {
logger = makeDefaultLogger();
settings.loadConfFile();
settings.update();
settings.lockCPU = false;
_store = openStore();
} catch (Error & e) {
+28 -33
View File
@@ -1,48 +1,43 @@
nix_perl_sources := \
$(d)/lib/Nix/Store.pm \
$(d)/lib/Nix/Manifest.pm \
$(d)/lib/Nix/SSH.pm \
$(d)/lib/Nix/CopyClosure.pm \
$(d)/lib/Nix/Config.pm.in \
$(d)/lib/Nix/Utils.pm
lib/Nix/Store.pm \
lib/Nix/Manifest.pm \
lib/Nix/SSH.pm \
lib/Nix/CopyClosure.pm \
lib/Nix/Config.pm.in \
lib/Nix/Utils.pm
nix_perl_modules := $(nix_perl_sources:.in=)
$(foreach x, $(nix_perl_modules), $(eval $(call install-data-in, $(x), $(perllibdir)/Nix)))
ifeq ($(perlbindings), yes)
$(d)/lib/Nix/Store.cc: $(d)/lib/Nix/Store.xs
lib/Nix/Store.cc: lib/Nix/Store.xs
$(trace-gen) xsubpp $^ -output $@
libraries += Store
libraries += Store
Store_DIR := $(d)/lib/Nix
Store_DIR := lib/Nix
Store_SOURCES := $(Store_DIR)/Store.cc
Store_SOURCES := $(Store_DIR)/Store.cc
Store_CXXFLAGS = \
-I$(shell $(perl) -e 'use Config; print $$Config{archlibexp};')/CORE \
-D_FILE_OFFSET_BITS=64 \
-Wno-unknown-warning-option -Wno-unused-variable -Wno-literal-suffix \
-Wno-reserved-user-defined-literal -Wno-duplicate-decl-specifier -Wno-pointer-bool-conversion
Store_CXXFLAGS = \
$(NIX_CFLAGS) \
-I$(shell perl -e 'use Config; print $$Config{archlibexp};')/CORE \
-D_FILE_OFFSET_BITS=64 \
-Wno-unknown-warning-option -Wno-unused-variable -Wno-literal-suffix \
-Wno-reserved-user-defined-literal -Wno-duplicate-decl-specifier -Wno-pointer-bool-conversion
Store_LIBS = libstore libutil
Store_LDFLAGS := $(SODIUM_LIBS)
ifeq (CYGWIN,$(findstring CYGWIN,$(OS)))
archlib = $(shell perl -E 'use Config; print $$Config{archlib};')
libperl = $(shell perl -E 'use Config; print $$Config{libperl};')
Store_LDFLAGS += $(shell find ${archlib} -name ${libperl})
endif
Store_ALLOW_UNDEFINED = 1
Store_FORCE_INSTALL = 1
Store_INSTALL_DIR = $(perllibdir)/auto/Nix/Store
Store_LDFLAGS := $(SODIUM_LIBS) $(NIX_LIBS)
ifeq (CYGWIN,$(findstring CYGWIN,$(OS)))
archlib = $(shell perl -E 'use Config; print $$Config{archlib};')
libperl = $(shell perl -E 'use Config; print $$Config{libperl};')
Store_LDFLAGS += $(shell find ${archlib} -name ${libperl})
endif
clean-files += $(d)/lib/Nix/Config.pm $(d)/lib/Nix/Store.cc
Store_ALLOW_UNDEFINED = 1
Store_FORCE_INSTALL = 1
Store_INSTALL_DIR = $(perllibdir)/auto/Nix/Store
clean-files += lib/Nix/Config.pm lib/Nix/Store.cc Makefile.config
+22
View File
@@ -0,0 +1,22 @@
{ pkgs }:
rec {
sh = pkgs.busybox.override {
useMusl = true;
enableStatic = true;
enableMinimal = true;
extraConfig = ''
CONFIG_ASH y
CONFIG_ASH_BUILTIN_ECHO y
CONFIG_ASH_BUILTIN_TEST y
CONFIG_ASH_OPTIMIZE_FOR_SIZE y
'';
};
configureFlags =
[ "--disable-init-state"
"--enable-gc"
] ++ pkgs.lib.optionals pkgs.stdenv.isLinux [
"--with-sandbox-shell=${sh}/bin/busybox"
];
}
+71 -40
View File
@@ -5,16 +5,16 @@
let
pkgs = import <nixpkgs> {};
pkgs = import nixpkgs {};
systems = [ "x86_64-linux" "i686-linux" "x86_64-darwin" ];
systems = [ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ];
jobs = rec {
tarball =
with pkgs;
tarball = pkgs.lib.genAttrs systems (system:
with import nixpkgs { inherit system; };
releaseTools.sourceTarball {
name = "nix-tarball";
@@ -24,18 +24,15 @@ let
inherit officialRelease;
buildInputs =
[ curl bison flex perl libxml2 libxslt bzip2 xz
[ curl bison flex libxml2 libxslt
bzip2 xz brotli
pkgconfig sqlite libsodium boehmgc
docbook5 docbook5_xsl
autoconf-archive
git
];
] ++ lib.optional stdenv.isLinux libseccomp;
configureFlags = ''
--with-dbi=${perlPackages.DBI}/${perl.libPrefix}
--with-dbd-sqlite=${perlPackages.DBDSQLite}/${perl.libPrefix}
--enable-gc
'';
configureFlags = "--enable-gc";
postUnpack = ''
# Clean up when building from a working tree.
@@ -45,6 +42,7 @@ let
'';
preConfigure = ''
(cd perl ; autoreconf --install --force --verbose)
# TeX needs a writable font cache.
export VARTEXFONTS=$TMPDIR/texfonts
'';
@@ -61,33 +59,36 @@ let
make install docdir=$out/share/doc/nix makefiles=doc/manual/local.mk
echo "doc manual $out/share/doc/nix/manual" >> $out/nix-support/hydra-build-products
'';
};
}
);
build = pkgs.lib.genAttrs systems (system:
with import <nixpkgs> { inherit system; };
with import nixpkgs { inherit system; };
with import ./release-common.nix { inherit pkgs; };
releaseTools.nixBuild {
name = "nix";
src = tarball;
src = tarball."x86_64-linux";
buildInputs =
[ curl perl bzip2 xz openssl pkgconfig sqlite boehmgc ]
++ lib.optional stdenv.isLinux libsodium
++ lib.optional stdenv.isLinux
[ curl
bzip2 xz brotli
openssl pkgconfig sqlite boehmgc
]
++ lib.optional stdenv.isLinux libseccomp
++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium
++ lib.optional (stdenv.isLinux || stdenv.isDarwin)
(aws-sdk-cpp.override {
apis = ["s3"];
customMemoryManagement = false;
});
configureFlags = ''
--disable-init-state
--with-dbi=${perlPackages.DBI}/${perl.libPrefix}
--with-dbd-sqlite=${perlPackages.DBDSQLite}/${perl.libPrefix}
--enable-gc
--sysconfdir=/etc
'';
configureFlags = configureFlags ++
[ "--sysconfdir=/etc" ];
enableParallelBuilding = true;
@@ -102,10 +103,35 @@ let
});
perlBindings = pkgs.lib.genAttrs systems (system:
let pkgs = import nixpkgs { inherit system; }; in with pkgs;
releaseTools.nixBuild {
name = "nix-perl";
src = tarball."x86_64-linux";
buildInputs =
[ (builtins.getAttr system jobs.build) curl bzip2 xz pkgconfig pkgs.perl ]
++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium;
configureFlags = ''
--with-dbi=${perlPackages.DBI}/${pkgs.perl.libPrefix}
--with-dbd-sqlite=${perlPackages.DBDSQLite}/${pkgs.perl.libPrefix}
'';
enableParallelBuilding = true;
postUnpack = "sourceRoot=$sourceRoot/perl";
preBuild = "unset NIX_INDENT_MAKE";
});
binaryTarball = pkgs.lib.genAttrs systems (system:
# FIXME: temporarily use a different branch for the Darwin build.
with import <nixpkgs> { inherit system; };
with import nixpkgs { inherit system; };
let
toplevel = builtins.getAttr system jobs.build;
@@ -140,22 +166,20 @@ let
coverage =
with import <nixpkgs> { system = "x86_64-linux"; };
with import nixpkgs { system = "x86_64-linux"; };
releaseTools.coverageAnalysis {
name = "nix-build";
src = tarball;
src = tarball."x86_64-linux";
buildInputs =
[ curl perl bzip2 openssl pkgconfig sqlite xz libsodium
[ curl bzip2 openssl pkgconfig sqlite xz libsodium libseccomp
# These are for "make check" only:
graphviz libxml2 libxslt
];
configureFlags = ''
--disable-init-state
--with-dbi=${perlPackages.DBI}/${perl.libPrefix}
--with-dbd-sqlite=${perlPackages.DBDSQLite}/${perl.libPrefix}
'';
dontInstall = false;
@@ -195,8 +219,13 @@ let
nix = build.x86_64-linux; system = "x86_64-linux";
});
tests.setuid = pkgs.lib.genAttrs (pkgs.lib.filter (pkgs.lib.hasSuffix "-linux") systems) (system:
import ./tests/setuid.nix rec {
nix = build.${system}; inherit system;
});
tests.binaryTarball =
with import <nixpkgs> { system = "x86_64-linux"; };
with import nixpkgs { system = "x86_64-linux"; };
vmTools.runInLinuxImage (runCommand "nix-binary-tarball-test"
{ diskImage = vmTools.diskImages.ubuntu1204x86_64;
}
@@ -236,10 +265,12 @@ let
# Aggregate job containing the release-critical jobs.
release = pkgs.releaseTools.aggregate {
name = "nix-${tarball.version}";
name = "nix-${tarball."x86_64-linux".version}";
meta.description = "Release-critical builds";
constituents =
[ tarball
[ tarball.x86_64-linux
tarball.x86_64-darwin
tarball.i686-linux
build.i686-linux
build.x86_64-darwin
build.x86_64-linux
@@ -269,14 +300,14 @@ let
makeRPM =
system: diskImageFun: extraPackages:
with import <nixpkgs> { inherit system; };
with import nixpkgs { inherit system; };
releaseTools.rpmBuild rec {
name = "nix-rpm";
src = jobs.tarball;
src = jobs.tarball."x86_64-linux";
diskImage = (diskImageFun vmTools.diskImageFuns)
{ extraPackages =
[ "perl-DBD-SQLite" "perl-devel" "sqlite" "sqlite-devel" "bzip2-devel" "emacs" "libcurl-devel" "openssl-devel" "xz-devel" ]
[ "sqlite" "sqlite-devel" "bzip2-devel" "emacs" "libcurl-devel" "openssl-devel" "xz-devel" "libseccomp-devel" ]
++ extraPackages; };
memSize = 1024;
meta.schedulingPriority = 50;
@@ -291,21 +322,21 @@ let
makeDeb =
system: diskImageFun: extraPackages: extraDebPackages:
with import <nixpkgs> { inherit system; };
with import nixpkgs { inherit system; };
releaseTools.debBuild {
name = "nix-deb";
src = jobs.tarball;
src = jobs.tarball."x86_64-linux";
diskImage = (diskImageFun vmTools.diskImageFuns)
{ extraPackages =
[ "libdbd-sqlite3-perl" "libsqlite3-dev" "libbz2-dev" "libwww-curl-perl" "libcurl-dev" "libcurl3-nss" "libssl-dev" "liblzma-dev" ]
[ "libsqlite3-dev" "libbz2-dev" "libcurl-dev" "libcurl3-nss" "libssl-dev" "liblzma-dev" "libseccomp-dev" ]
++ extraPackages; };
memSize = 1024;
meta.schedulingPriority = 50;
postInstall = "make installcheck";
configureFlags = "--sysconfdir=/etc";
debRequires =
[ "curl" "libdbd-sqlite3-perl" "libsqlite3-0" "libbz2-1.0" "bzip2" "xz-utils" "libwww-curl-perl" "libssl1.0.0" "liblzma5" ]
[ "curl" "libsqlite3-0" "libbz2-1.0" "bzip2" "xz-utils" "libssl1.0.0" "liblzma5" "libseccomp2" ]
++ extraDebPackages;
debMaintainer = "Eelco Dolstra <eelco.dolstra@logicblox.com>";
doInstallCheck = true;
+1 -1
View File
@@ -85,6 +85,6 @@ if [ -n "$HOME" ] && [ -n "$USER" ]; then
export MANPATH="$NIX_LINK/share/man:$MANPATH"
fi
export PATH="$NIX_LINK/bin:$NIX_LINK/sbin:$__savedpath"
export PATH="$NIX_LINK/bin:$__savedpath"
unset __savedpath NIX_LINK NIX_USER_PROFILE_DIR NIX_PROFILES
fi
+15 -10
View File
@@ -2,11 +2,14 @@
with import <nixpkgs> {};
with import ./release-common.nix { inherit pkgs; };
(if useClang then clangStdenv else stdenv).mkDerivation {
name = "nix";
buildInputs =
[ curl bison flex perl libxml2 libxslt bzip2 xz
[ curl bison flex libxml2 libxslt
bzip2 xz brotli
pkgconfig sqlite libsodium boehmgc
docbook5 docbook5_xsl
autoconf-archive
@@ -15,15 +18,14 @@ with import <nixpkgs> {};
customMemoryManagement = false;
})
autoreconfHook
perlPackages.DBDSQLite
];
configureFlags =
[ "--disable-init-state"
"--enable-gc"
"--with-dbi=${perlPackages.DBI}/${perl.libPrefix}"
"--with-dbd-sqlite=${perlPackages.DBDSQLite}/${perl.libPrefix}"
];
# For nix-perl
perl
perlPackages.DBDSQLite
]
++ lib.optional stdenv.isLinux libseccomp;
inherit configureFlags;
enableParallelBuilding = true;
@@ -31,6 +33,9 @@ with import <nixpkgs> {};
shellHook =
''
configureFlags+=" --prefix=$(pwd)/inst"
export prefix=$(pwd)/inst
configureFlags+=" --prefix=$prefix"
PKG_CONFIG_PATH=$prefix/lib/pkgconfig:$PKG_CONFIG_PATH
PATH=$prefix/bin:$PATH
'';
}
+7
View File
@@ -37,6 +37,13 @@ namespace {
os.str(emptyStr);
}
void do_pad( std::string & s,
std::streamsize w,
const char c,
std::ios::fmtflags f,
bool center)
__attribute__ ((unused));
void do_pad( std::string & s,
std::streamsize w,
const char c,
+80 -137
View File
@@ -9,6 +9,7 @@
#include <sys/time.h>
#endif
#include "machines.hh"
#include "shared.hh"
#include "pathlocks.hh"
#include "globals.hh"
@@ -17,145 +18,66 @@
#include "derivations.hh"
using namespace nix;
using std::cerr;
using std::cin;
static void handle_alarm(int sig) {
static void handleAlarm(int sig) {
}
class machine {
const std::set<string> supportedFeatures;
const std::set<string> mandatoryFeatures;
public:
const string hostName;
const std::vector<string> systemTypes;
const string sshKey;
const unsigned long long maxJobs;
const unsigned long long speedFactor;
bool enabled;
bool allSupported(const std::set<string> & features) const {
return std::all_of(features.begin(), features.end(),
[&](const string & feature) {
return supportedFeatures.count(feature) ||
mandatoryFeatures.count(feature);
});
}
bool mandatoryMet(const std::set<string> & features) const {
return std::all_of(mandatoryFeatures.begin(), mandatoryFeatures.end(),
[&](const string & feature) {
return features.count(feature);
});
}
machine(decltype(hostName) hostName,
decltype(systemTypes) systemTypes,
decltype(sshKey) sshKey,
decltype(maxJobs) maxJobs,
decltype(speedFactor) speedFactor,
decltype(supportedFeatures) supportedFeatures,
decltype(mandatoryFeatures) mandatoryFeatures) :
supportedFeatures{std::move(supportedFeatures)},
mandatoryFeatures{std::move(mandatoryFeatures)},
hostName{std::move(hostName)},
systemTypes{std::move(systemTypes)},
sshKey{std::move(sshKey)},
maxJobs{std::move(maxJobs)},
speedFactor{speedFactor == 0 ? 1 : std::move(speedFactor)},
enabled{true} {};
};;
static std::vector<machine> read_conf()
std::string escapeUri(std::string uri)
{
auto conf = getEnv("NIX_REMOTE_SYSTEMS", SYSCONFDIR "/nix/machines");
auto machines = std::vector<machine>{};
auto lines = std::vector<string>{};
try {
lines = tokenizeString<std::vector<string>>(readFile(conf), "\n");
} catch (const SysError & e) {
if (e.errNo != ENOENT)
throw;
}
for (auto line : lines) {
chomp(line);
line.erase(std::find(line.begin(), line.end(), '#'), line.end());
if (line.empty()) {
continue;
}
auto tokens = tokenizeString<std::vector<string>>(line);
auto sz = tokens.size();
if (sz < 4) {
throw new FormatError(format("Bad machines.conf file %1%")
% conf);
}
machines.emplace_back(tokens[0],
tokenizeString<std::vector<string>>(tokens[1], ","),
tokens[2],
stoull(tokens[3]),
sz >= 5 ? stoull(tokens[4]) : 1LL,
sz >= 6 ?
tokenizeString<std::set<string>>(tokens[5], ",") :
std::set<string>{},
sz >= 7 ?
tokenizeString<std::set<string>>(tokens[6], ",") :
std::set<string>{});
}
return machines;
std::replace(uri.begin(), uri.end(), '/', '_');
return uri;
}
static string currentLoad;
static AutoCloseFD openSlotLock(const machine & m, unsigned long long slot)
static AutoCloseFD openSlotLock(const Machine & m, unsigned long long slot)
{
std::ostringstream fn_stream(currentLoad, std::ios_base::ate | std::ios_base::out);
fn_stream << "/";
for (auto t : m.systemTypes) {
fn_stream << t << "-";
}
fn_stream << m.hostName << "-" << slot;
return openLockFile(fn_stream.str(), true);
return openLockFile(fmt("%s/%s-%d", currentLoad, escapeUri(m.storeUri), slot), true);
}
static char display_env[] = "DISPLAY=";
static char ssh_env[] = "SSH_ASKPASS=";
int main (int argc, char * * argv)
{
return handleExceptions(argv[0], [&]() {
initNix();
/* Ensure we don't get any SSH passphrase or host key popups. */
if (putenv(display_env) == -1 ||
putenv(ssh_env) == -1) {
throw SysError("Setting SSH env vars");
}
if (argc != 4) {
/* Ensure we don't get any SSH passphrase or host key popups. */
unsetenv("DISPLAY");
unsetenv("SSH_ASKPASS");
if (argc != 6)
throw UsageError("called without required arguments");
}
auto store = openStore();
auto localSystem = argv[1];
settings.maxSilentTime = stoull(string(argv[2]));
settings.buildTimeout = stoull(string(argv[3]));
settings.maxSilentTime = std::stoll(argv[2]);
settings.buildTimeout = std::stoll(argv[3]);
verbosity = (Verbosity) std::stoll(argv[4]);
settings.builders = argv[5];
currentLoad = getEnv("NIX_CURRENT_LOAD", "/run/nix/current-load");
/* It would be more appropriate to use $XDG_RUNTIME_DIR, since
that gets cleared on reboot, but it wouldn't work on OS X. */
currentLoad = settings.nixStateDir + "/current-load";
std::shared_ptr<Store> sshStore;
AutoCloseFD bestSlotLock;
auto machines = read_conf();
auto machines = getMachines();
debug("got %d remote builders", machines.size());
if (machines.empty()) {
std::cerr << "# decline-permanently\n";
return;
}
string drvPath;
string hostName;
string storeUri;
for (string line; getline(cin, line);) {
auto tokens = tokenizeString<std::vector<string>>(line);
auto sz = tokens.size();
if (sz != 3 && sz != 4) {
throw Error(format("invalid build hook line %1%") % line);
}
if (sz != 3 && sz != 4)
throw Error("invalid build hook line ‘%1%’", line);
auto amWilling = tokens[0] == "1";
auto neededSystem = tokens[1];
drvPath = tokens[2];
@@ -174,9 +96,11 @@ int main (int argc, char * * argv)
bool rightType = false;
machine * bestMachine = nullptr;
Machine * bestMachine = nullptr;
unsigned long long bestLoad = 0;
for (auto & m : machines) {
debug("considering building on ‘%s’", m.storeUri);
if (m.enabled && std::find(m.systemTypes.begin(),
m.systemTypes.end(),
neededSystem) != m.systemTypes.end() &&
@@ -221,11 +145,10 @@ int main (int argc, char * * argv)
}
if (!bestSlotLock) {
if (rightType && !canBuildLocally) {
cerr << "# postpone\n";
} else {
cerr << "# decline\n";
}
if (rightType && !canBuildLocally)
std::cerr << "# postpone\n";
else
std::cerr << "# decline\n";
break;
}
@@ -238,47 +161,67 @@ int main (int argc, char * * argv)
lock = -1;
try {
sshStore = openStore("ssh://" + bestMachine->hostName + "?key=" + bestMachine->sshKey);
hostName = bestMachine->hostName;
Store::Params storeParams{{"max-connections", "1"}, {"log-fd", "4"}};
if (bestMachine->sshKey != "")
storeParams["ssh-key"] = bestMachine->sshKey;
sshStore = openStore(bestMachine->storeUri, storeParams);
sshStore->connect();
storeUri = bestMachine->storeUri;
} catch (std::exception & e) {
cerr << e.what() << '\n';
cerr << "unable to open SSH connection to ‘" << bestMachine->hostName << "’, trying other available machines...\n";
printError("unable to open SSH connection to ‘%s’: %s; trying other available machines...",
bestMachine->storeUri, e.what());
bestMachine->enabled = false;
continue;
}
goto connected;
}
}
connected:
cerr << "# accept\n";
std::cerr << "# accept\n";
string line;
if (!getline(cin, line)) {
if (!getline(cin, line))
throw Error("hook caller didn't send inputs");
}
auto inputs = tokenizeString<std::list<string>>(line);
if (!getline(cin, line)) {
auto inputs = tokenizeString<PathSet>(line);
if (!getline(cin, line))
throw Error("hook caller didn't send outputs");
}
auto outputs = tokenizeString<Strings>(line);
AutoCloseFD uploadLock = openLockFile(currentLoad + "/" + hostName + ".upload-lock", true);
auto old = signal(SIGALRM, handle_alarm);
auto outputs = tokenizeString<PathSet>(line);
AutoCloseFD uploadLock = openLockFile(currentLoad + "/" + escapeUri(storeUri) + ".upload-lock", true);
auto old = signal(SIGALRM, handleAlarm);
alarm(15 * 60);
if (!lockFile(uploadLock.get(), ltWrite, true)) {
cerr << "somebody is hogging the upload lock for " << hostName << ", continuing...\n";
}
if (!lockFile(uploadLock.get(), ltWrite, true))
printError("somebody is hogging the upload lock for ‘%s’, continuing...");
alarm(0);
signal(SIGALRM, old);
copyPaths(store, ref<Store>(sshStore), inputs);
copyPaths(store, ref<Store>(sshStore), inputs, false, true);
uploadLock = -1;
cerr << "building ‘" << drvPath << "’ on ‘" << hostName << "’\n";
sshStore->buildDerivation(drvPath, readDerivation(drvPath));
BasicDerivation drv(readDerivation(drvPath));
drv.inputSrcs = inputs;
std::remove_if(outputs.begin(), outputs.end(), [=](const Path & path) { return store->isValidPath(path); });
if (!outputs.empty()) {
setenv("NIX_HELD_LOCKS", concatStringsSep(" ", outputs).c_str(), 1); /* FIXME: ugly */
copyPaths(ref<Store>(sshStore), store, outputs);
printError("building ‘%s’ on ‘%s’", drvPath, storeUri);
auto result = sshStore->buildDerivation(drvPath, drv);
if (!result.success())
throw Error("build of ‘%s’ on ‘%s’ failed: %s", drvPath, storeUri, result.errorMsg);
PathSet missing;
for (auto & path : outputs)
if (!store->isValidPath(path)) missing.insert(path);
if (!missing.empty()) {
setenv("NIX_HELD_LOCKS", concatStringsSep(" ", missing).c_str(), 1); /* FIXME: ugly */
copyPaths(ref<Store>(sshStore), store, missing, false, true);
}
return;
});
}
-2
View File
@@ -7,5 +7,3 @@ build-remote_INSTALL_DIR := $(libexecdir)/nix
build-remote_LIBS = libmain libutil libformat libstore
build-remote_SOURCES := $(d)/build-remote.cc
build-remote_CXXFLAGS = -DSYSCONFDIR="\"$(sysconfdir)\""
+3 -3
View File
@@ -202,7 +202,7 @@ void initGC()
GC_INIT();
GC_oom_fn = oomHandler;
GC_set_oom_fn(oomHandler);
/* Set the initial heap size to something fairly big (25% of
physical RAM, up to a maximum of 384 MiB) so that in most cases
@@ -299,7 +299,7 @@ EvalState::EvalState(const Strings & _searchPath, ref<Store> store)
{
countCalls = getEnv("NIX_COUNT_CALLS", "0") != "0";
restricted = settings.get("restrict-eval", false);
restricted = settings.restrictEval;
assert(gcInitialised);
@@ -642,7 +642,7 @@ void EvalState::evalFile(const Path & path, Value & v)
return;
}
Activity act(*logger, lvlTalkative, format("evaluating file ‘%1%’") % path2);
printTalkative("evaluating file ‘%1%’", path2);
Expr * e = parseExprFromFile(checkSourcePath(path2));
try {
eval(e, v);
+7 -2
View File
@@ -3,6 +3,7 @@
#include "eval-inline.hh"
#include <cstring>
#include <regex>
namespace nix {
@@ -262,6 +263,9 @@ static string addToPath(const string & s1, const string & s2)
}
static std::regex attrRegex("[A-Za-z_][A-Za-z0-9-_+]*");
static void getDerivations(EvalState & state, Value & vIn,
const string & pathPrefix, Bindings & autoArgs,
DrvInfos & drvs, Done & done,
@@ -285,7 +289,9 @@ static void getDerivations(EvalState & state, Value & vIn,
bound to the attribute with the "lower" name should take
precedence). */
for (auto & i : v.attrs->lexicographicOrder()) {
Activity act(*logger, lvlDebug, format("evaluating attribute ‘%1%’") % i->name);
debug("evaluating attribute ‘%1%’", i->name);
if (!std::regex_match(std::string(i->name), attrRegex))
continue;
string pathPrefix2 = addToPath(pathPrefix, i->name);
if (combineChannels)
getDerivations(state, *i->value, pathPrefix2, autoArgs, drvs, done, ignoreAssertionFailures);
@@ -304,7 +310,6 @@ static void getDerivations(EvalState & state, Value & vIn,
else if (v.isList()) {
for (unsigned int n = 0; n < v.listSize(); ++n) {
Activity act(*logger, lvlDebug, "evaluating list element");
string pathPrefix2 = addToPath(pathPrefix, (format("%1%") % n).str());
if (getDerivation(state, *v.listElems()[n], pathPrefix2, drvs, done, ignoreAssertionFailures))
getDerivations(state, *v.listElems()[n], pathPrefix2, autoArgs, drvs, done, ignoreAssertionFailures);
+19 -11
View File
@@ -142,25 +142,34 @@ or { return OR_KW; }
\{ { return '{'; }
<INSIDE_DOLLAR_CURLY>\{ { PUSH_STATE(INSIDE_DOLLAR_CURLY); return '{'; }
<INITIAL,INSIDE_DOLLAR_CURLY>\" { PUSH_STATE(STRING); return '"'; }
<INITIAL,INSIDE_DOLLAR_CURLY>\" {
PUSH_STATE(STRING); return '"';
}
<STRING>([^\$\"\\]|\$[^\{\"\\]|\\.|\$\\.)*\$/\" |
<STRING>([^\$\"\\]|\$[^\{\"\\]|\\.|\$\\.)+ {
/* It is impossible to match strings ending with '$' with one
regex because trailing contexts are only valid at the end
of a rule. (A sane but undocumented limitation.) */
yylval->e = unescapeStr(data->symbols, yytext);
return STR;
}
/* It is impossible to match strings ending with '$' with one
regex because trailing contexts are only valid at the end
of a rule. (A sane but undocumented limitation.) */
yylval->e = unescapeStr(data->symbols, yytext);
return STR;
}
<STRING>\$\{ { PUSH_STATE(INSIDE_DOLLAR_CURLY); return DOLLAR_CURLY; }
<STRING>\" { POP_STATE(); return '"'; }
<STRING>. return yytext[0]; /* just in case: shouldn't be reached */
<STRING>\" { POP_STATE(); return '"'; }
<STRING>\$|\\|\$\\ {
/* This can only occur when we reach EOF, otherwise the above
(...|\$[^\{\"\\]|\\.|\$\\.)+ would have triggered.
This is technically invalid, but we leave the problem to the
parser who fails with exact location. */
return STR;
}
<INITIAL,INSIDE_DOLLAR_CURLY>\'\'(\ *\n)? { PUSH_STATE(IND_STRING); return IND_STRING_OPEN; }
<IND_STRING>([^\$\']|\$[^\{\']|\'[^\'\$])+ {
yylval->e = new ExprIndStr(yytext);
return IND_STR;
}
<IND_STRING>\'\'\$ {
<IND_STRING>\'\'\$ |
<IND_STRING>\$ {
yylval->e = new ExprIndStr("$");
return IND_STR;
}
@@ -178,7 +187,6 @@ or { return OR_KW; }
yylval->e = new ExprIndStr("'");
return IND_STR;
}
<IND_STRING>. return yytext[0]; /* just in case: shouldn't be reached */
<INITIAL,INSIDE_DOLLAR_CURLY>{
-2
View File
@@ -6,8 +6,6 @@ libexpr_DIR := $(d)
libexpr_SOURCES := $(wildcard $(d)/*.cc) $(wildcard $(d)/primops/*.cc) $(d)/lexer-tab.cc $(d)/parser-tab.cc
libexpr_CXXFLAGS := -Wno-deprecated-register
libexpr_LIBS = libutil libstore libformat
libexpr_LDFLAGS =
+1 -1
View File
@@ -376,7 +376,7 @@ expr_simple
$$ = stripIndentation(CUR_POS, data->symbols, *$2);
}
| PATH { $$ = new ExprPath(absPath($1, data->basePath)); }
| HPATH { $$ = new ExprPath(getEnv("HOME", "") + string{$1 + 1}); }
| HPATH { $$ = new ExprPath(getHome() + string{$1 + 1}); }
| SPATH {
string path($1 + 1, strlen($1) - 2);
$$ = new ExprApp(CUR_POS,
+74 -27
View File
@@ -59,6 +59,8 @@ void EvalState::realiseContext(const PathSet & context)
drvs.insert(decoded.first + "!" + decoded.second);
}
if (!drvs.empty()) {
if (!settings.enableImportFromDerivation)
throw EvalError(format("attempted to realize ‘%1%’ during evaluation but 'allow-import-from-derivation' is false") % *(drvs.begin()));
/* For performance, prefetch all substitute info. */
PathSet willBuild, willSubstitute, unknown;
unsigned long long downloadSize, narSize;
@@ -125,7 +127,7 @@ static void prim_scopedImport(EvalState & state, const Pos & pos, Value * * args
env->values[displ++] = attr.value;
}
Activity act(*logger, lvlTalkative, format("evaluating file ‘%1%’") % path);
printTalkative("evaluating file ‘%1%’", path);
Expr * e = state.parseExprFromFile(resolveExprPath(path), staticEnv);
e->eval(state, *env, v);
@@ -176,6 +178,45 @@ static void prim_importNative(EvalState & state, const Pos & pos, Value * * args
}
/* Execute a program and parse its output */
static void prim_exec(EvalState & state, const Pos & pos, Value * * args, Value & v)
{
state.forceList(*args[0], pos);
auto elems = args[0]->listElems();
auto count = args[0]->listSize();
if (count == 0) {
throw EvalError(format("at least one argument to 'exec' required, at %1%") % pos);
}
PathSet context;
auto program = state.coerceToString(pos, *elems[0], context, false, false);
Strings commandArgs;
for (unsigned int i = 1; i < args[0]->listSize(); ++i) {
commandArgs.emplace_back(state.coerceToString(pos, *elems[i], context, false, false));
}
try {
state.realiseContext(context);
} catch (InvalidPathError & e) {
throw EvalError(format("cannot execute ‘%1%’, since path ‘%2%’ is not valid, at %3%")
% program % e.path % pos);
}
auto output = runProgram(program, true, commandArgs);
Expr * parsed;
try {
parsed = state.parseExprFromString(output, pos.file);
} catch (Error & e) {
e.addPrefix(format("While parsing the output from ‘%1%’, at %2%\n") % program % pos);
throw;
}
try {
state.eval(parsed, v);
} catch (Error & e) {
e.addPrefix(format("While evaluating the output from ‘%1%’, at %2%\n") % program % pos);
throw;
}
}
/* Return a string representing the type of the expression. */
static void prim_typeOf(EvalState & state, const Pos & pos, Value * * args, Value & v)
{
@@ -285,8 +326,6 @@ typedef list<Value *> ValueList;
static void prim_genericClosure(EvalState & state, const Pos & pos, Value * * args, Value & v)
{
Activity act(*logger, lvlDebug, "finding dependencies");
state.forceAttrs(*args[0], pos);
/* Get the start set. */
@@ -458,8 +497,6 @@ void prim_valueSize(EvalState & state, const Pos & pos, Value * * args, Value &
derivation. */
static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * * args, Value & v)
{
Activity act(*logger, lvlVomit, "evaluating derivation");
state.forceAttrs(*args[0], pos);
/* Figure out the name first (for stack backtraces). */
@@ -493,7 +530,8 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
PathSet context;
string outputHash, outputHashAlgo;
std::experimental::optional<std::string> outputHash;
std::string outputHashAlgo;
bool outputHashRecursive = false;
StringSet outputs;
@@ -502,7 +540,7 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
for (auto & i : args[0]->attrs->lexicographicOrder()) {
if (i->name == state.sIgnoreNulls) continue;
string key = i->name;
Activity act(*logger, lvlVomit, format("processing attribute ‘%1%’") % key);
vomit("processing attribute ‘%1%’", key);
auto handleHashMode = [&](const std::string & s) {
if (s == "recursive") outputHashRecursive = true;
@@ -662,7 +700,7 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
throw EvalError(format("derivation names are not allowed to end in ‘%1%’, at %2%")
% drvExtension % posDrvName);
if (outputHash != "") {
if (outputHash) {
/* Handle fixed-output derivations. */
if (outputs.size() != 1 || *(outputs.begin()) != "out")
throw Error(format("multiple outputs are not supported in fixed-output derivations, at %1%") % posDrvName);
@@ -670,13 +708,13 @@ static void prim_derivationStrict(EvalState & state, const Pos & pos, Value * *
HashType ht = parseHashType(outputHashAlgo);
if (ht == htUnknown)
throw EvalError(format("unknown hash algorithm ‘%1%’, at %2%") % outputHashAlgo % posDrvName);
Hash h = parseHash16or32(ht, outputHash);
Hash h = parseHash16or32(ht, *outputHash);
outputHash = printHash(h);
if (outputHashRecursive) outputHashAlgo = "r:" + outputHashAlgo;
Path outPath = state.store->makeFixedOutputPath(outputHashRecursive, h, drvName);
drv.env["out"] = outPath;
drv.outputs["out"] = DerivationOutput(outPath, outputHashAlgo, outputHash);
drv.outputs["out"] = DerivationOutput(outPath, outputHashAlgo, *outputHash);
}
else {
@@ -1671,26 +1709,33 @@ static void prim_hashString(EvalState & state, const Pos & pos, Value * * args,
‘null’ or a list containing substring matches. */
static void prim_match(EvalState & state, const Pos & pos, Value * * args, Value & v)
{
std::regex regex(state.forceStringNoCtx(*args[0], pos), std::regex::extended);
auto re = state.forceStringNoCtx(*args[0], pos);
PathSet context;
const std::string str = state.forceString(*args[1], context, pos);
try {
std::regex regex(re, std::regex::extended);
std::smatch match;
if (!std::regex_match(str, match, regex)) {
mkNull(v);
return;
}
PathSet context;
const std::string str = state.forceString(*args[1], context, pos);
// the first match is the whole string
const size_t len = match.size() - 1;
state.mkList(v, len);
for (size_t i = 0; i < len; ++i) {
if (!match[i+1].matched)
mkNull(*(v.listElems()[i] = state.allocValue()));
else
mkString(*(v.listElems()[i] = state.allocValue()), match[i + 1].str().c_str());
std::smatch match;
if (!std::regex_match(str, match, regex)) {
mkNull(v);
return;
}
// the first match is the whole string
const size_t len = match.size() - 1;
state.mkList(v, len);
for (size_t i = 0; i < len; ++i) {
if (!match[i+1].matched)
mkNull(*(v.listElems()[i] = state.allocValue()));
else
mkString(*(v.listElems()[i] = state.allocValue()), match[i + 1].str().c_str());
}
} catch (std::regex_error &) {
throw EvalError("invalid regular expression ‘%s’, at %s", re, pos);
}
}
@@ -1901,8 +1946,10 @@ void EvalState::createBaseEnv()
mkApp(v, *baseEnv.values[baseEnvDispl - 1], *v2);
forceValue(v);
addConstant("import", v);
if (settings.enableImportNative)
if (settings.enableNativeCode) {
addPrimOp("__importNative", 2, prim_importNative);
addPrimOp("__exec", 1, prim_exec);
}
addPrimOp("__typeOf", 1, prim_typeOf);
addPrimOp("isNull", 1, prim_isNull);
addPrimOp("__isFunction", 1, prim_isFunction);
+7 -5
View File
@@ -17,7 +17,7 @@ Path exportGit(ref<Store> store, const std::string & uri, const std::string & re
runProgram("git", true, { "init", "--bare", cacheDir });
}
Activity act(*logger, lvlInfo, format("fetching Git repository ‘%s’") % uri);
//Activity act(*logger, lvlInfo, format("fetching Git repository ‘%s’") % uri);
std::string localRef = "pid-" + std::to_string(getpid());
Path localRefFile = cacheDir + "/refs/heads/" + localRef;
@@ -58,12 +58,14 @@ static void prim_fetchgit(EvalState & state, const Pos & pos, Value * * args, Va
for (auto & attr : *args[0]->attrs) {
string name(attr.name);
if (name == "url")
url = state.forceStringNoCtx(*attr.value, *attr.pos);
else if (name == "rev")
if (name == "url") {
PathSet context;
url = state.coerceToString(*attr.pos, *attr.value, context, false, false);
if (hasPrefix(url, "/")) url = "file://" + url;
} else if (name == "rev")
rev = state.forceStringNoCtx(*attr.value, *attr.pos);
else
throw EvalError(format("unsupported argument ‘%1%’ to ‘fetchgit’, at %3%") % attr.name % attr.pos);
throw EvalError("unsupported argument ‘%s’ to ‘fetchgit’, at %s", attr.name, *attr.pos);
}
if (url.empty())
+9 -1
View File
@@ -220,6 +220,14 @@ static inline void mkApp(Value & v, Value & left, Value & right)
}
static inline void mkPrimOpApp(Value & v, Value & left, Value & right)
{
v.type = tPrimOpApp;
v.app.left = &left;
v.app.right = &right;
}
static inline void mkStringNoCopy(Value & v, const char * s)
{
v.type = tString;
@@ -256,7 +264,7 @@ size_t valueSize(Value & v);
#if HAVE_BOEHMGC
typedef std::vector<Value *, gc_allocator<Value *> > ValueVector;
typedef std::map<Symbol, Value *, std::less<Symbol>, gc_allocator<Value *> > ValueMap;
typedef std::map<Symbol, Value *, std::less<Symbol>, gc_allocator<std::pair<const Symbol, Value *> > > ValueMap;
#else
typedef std::vector<Value *> ValueVector;
typedef std::map<Symbol, Value *> ValueMap;
+11 -3
View File
@@ -18,11 +18,19 @@ MixCommonArgs::MixCommonArgs(const string & programName)
verbosity = lvlDebug;
});
mkFlag(0, "option", {"name", "value"}, "set a Nix configuration option (overriding nix.conf)", 2,
[](Strings ss) {
mkFlag()
.longName("option")
.labels({"name", "value"})
.description("set a Nix configuration option (overriding nix.conf)")
.arity(2)
.handler([](Strings ss) {
auto name = ss.front(); ss.pop_front();
auto value = ss.front();
settings.set(name, value);
try {
settings.set(name, value);
} catch (UsageError & e) {
warn(e.what());
}
});
}
+11 -1
View File
@@ -12,7 +12,7 @@ struct MixCommonArgs : virtual Args
struct MixDryRun : virtual Args
{
bool dryRun;
bool dryRun = false;
MixDryRun()
{
@@ -20,4 +20,14 @@ struct MixDryRun : virtual Args
}
};
struct MixJSON : virtual Args
{
bool json = false;
MixJSON()
{
mkFlag(0, "json", "produce JSON output", &json);
}
};
}
+17 -16
View File
@@ -106,8 +106,6 @@ void initNix()
std::cerr.rdbuf()->pubsetbuf(buf, sizeof(buf));
#endif
logger = makeDefaultLogger();
/* Initialise OpenSSL locking. */
opensslLocks = std::vector<std::mutex>(CRYPTO_num_locks());
CRYPTO_set_locking_callback(opensslLockCallback);
@@ -141,8 +139,13 @@ void initNix()
gettimeofday(&tv, 0);
srandom(tv.tv_usec);
if (char *pack = getenv("_NIX_OPTIONS"))
settings.unpack(pack);
/* On macOS, don't use the per-session TMPDIR (as set e.g. by
sshd). This breaks build users because they don't have access
to the TMPDIR, in particular in ‘nix-store --serve’. */
#if __APPLE__
if (getuid() == 0 && hasPrefix(getEnv("TMPDIR"), "/var/folders/"))
unsetenv("TMPDIR");
#endif
}
@@ -158,13 +161,17 @@ struct LegacyArgs : public MixCommonArgs
&settings.verboseBuild, false);
mkFlag('K', "keep-failed", "keep temporary directories of failed builds",
&settings.keepFailed);
&(bool&) settings.keepFailed);
mkFlag('k', "keep-going", "keep going after a build fails",
&settings.keepGoing);
&(bool&) settings.keepGoing);
mkFlag(0, "fallback", "build from source if substitution fails", []() {
settings.set("build-fallback", "true");
settings.tryFallback = true;
});
mkFlag1('j', "max-jobs", "jobs", "maximum number of parallel builds", [=](std::string s) {
settings.set("build-max-jobs", s);
});
auto intSettingAlias = [&](char shortName, const std::string & longName,
@@ -174,7 +181,6 @@ struct LegacyArgs : public MixCommonArgs
});
};
intSettingAlias('j', "max-jobs", "maximum number of parallel builds", "build-max-jobs");
intSettingAlias(0, "cores", "maximum number of CPU cores to use inside a build", "build-cores");
intSettingAlias(0, "max-silent-time", "number of seconds of silence before a build is killed", "build-max-silent-time");
intSettingAlias(0, "timeout", "number of seconds before a build is killed", "build-timeout");
@@ -183,7 +189,7 @@ struct LegacyArgs : public MixCommonArgs
&settings.readOnlyMode);
mkFlag(0, "no-build-hook", "disable use of the build hook mechanism",
&settings.useBuildHook, false);
&(bool&) settings.useBuildHook, false);
mkFlag(0, "show-trace", "show Nix expression stack trace in evaluation errors",
&settings.showTrace);
@@ -217,7 +223,6 @@ void parseCmdLine(int argc, char * * argv,
std::function<bool(Strings::iterator & arg, const Strings::iterator & end)> parseArg)
{
LegacyArgs(baseNameOf(argv[0]), parseArg).parseCmdline(argvToStrings(argc, argv));
settings.update();
}
@@ -262,7 +267,7 @@ int handleExceptions(const string & programName, std::function<void()> fun)
condition is discharged before we reach printMsg()
below, since otherwise it will throw an (uncaught)
exception. */
interruptThrown = true;
setInterruptThrown();
throw;
}
} catch (Exit & e) {
@@ -329,11 +334,7 @@ RunPager::~RunPager()
pid.wait();
}
} catch (...) {
try {
pid.kill(true);
} catch (...) {
ignoreException();
}
ignoreException();
}
}
+29 -12
View File
@@ -79,10 +79,7 @@ struct BinaryCacheStoreAccessor : public FSAccessor
BinaryCacheStore::BinaryCacheStore(const Params & params)
: Store(params)
, compression(get(params, "compression", "xz"))
, writeNARListing(get(params, "write-nar-listing", "0") == "1")
{
auto secretKeyFile = get(params, "secret-key", "");
if (secretKeyFile != "")
secretKey = std::unique_ptr<SecretKey>(new SecretKey(readFile(secretKeyFile)));
@@ -97,7 +94,7 @@ void BinaryCacheStore::init()
auto cacheInfo = getFile(cacheInfoFile);
if (!cacheInfo) {
upsertFile(cacheInfoFile, "StoreDir: " + storeDir + "\n");
upsertFile(cacheInfoFile, "StoreDir: " + storeDir + "\n", "text/x-nix-cache-info");
} else {
for (auto & line : tokenizeString<Strings>(*cacheInfo, "\n")) {
size_t colon = line.find(':');
@@ -117,11 +114,6 @@ void BinaryCacheStore::init()
}
}
void BinaryCacheStore::notImpl()
{
throw Error("operation not implemented for binary cache stores");
}
std::shared_ptr<std::string> BinaryCacheStore::getFile(const std::string & path)
{
std::promise<std::shared_ptr<std::string>> promise;
@@ -224,7 +216,7 @@ void BinaryCacheStore::addToStore(const ValidPathInfo & info, const ref<std::str
}
}
upsertFile(storePathToHash(info.path) + ".ls.xz", *compress("xz", jsonOut.str()));
upsertFile(storePathToHash(info.path) + ".ls", jsonOut.str(), "application/json");
}
else {
@@ -250,10 +242,11 @@ void BinaryCacheStore::addToStore(const ValidPathInfo & info, const ref<std::str
narInfo->url = "nar/" + printHash32(narInfo->fileHash) + ".nar"
+ (compression == "xz" ? ".xz" :
compression == "bzip2" ? ".bz2" :
compression == "br" ? ".br" :
"");
if (repair || !fileExists(narInfo->url)) {
stats.narWrite++;
upsertFile(narInfo->url, *narCompressed);
upsertFile(narInfo->url, *narCompressed, "application/x-nix-nar");
} else
stats.narWriteAverted++;
@@ -264,7 +257,7 @@ void BinaryCacheStore::addToStore(const ValidPathInfo & info, const ref<std::str
/* Atomically write the NAR info file.*/
if (secretKey) narInfo->sign(*secretKey);
upsertFile(narInfoFile, narInfo->to_string());
upsertFile(narInfoFile, narInfo->to_string(), "text/x-nix-narinfo");
auto hashPart = storePathToHash(narInfo->path);
@@ -382,4 +375,28 @@ ref<FSAccessor> BinaryCacheStore::getFSAccessor()
return make_ref<RemoteFSAccessor>(ref<Store>(shared_from_this()));
}
std::shared_ptr<std::string> BinaryCacheStore::getBuildLog(const Path & path)
{
Path drvPath;
if (isDerivation(path))
drvPath = path;
else {
try {
auto info = queryPathInfo(path);
// FIXME: add a "Log" field to .narinfo
if (info->deriver == "") return nullptr;
drvPath = info->deriver;
} catch (InvalidPath &) {
return nullptr;
}
}
auto logPath = "log/" + baseNameOf(drvPath);
debug("fetching build log from binary cache ‘%s/%s’", getUri(), logPath);
return getFile(logPath);
}
}
+23 -22
View File
@@ -13,25 +13,27 @@ struct NarInfo;
class BinaryCacheStore : public Store
{
public:
const Setting<std::string> compression{this, "xz", "compression", "NAR compression method ('xz', 'bzip2', or 'none')"};
const Setting<bool> writeNARListing{this, false, "write-nar-listing", "whether to write a JSON file listing the files in each NAR"};
const Setting<Path> secretKeyFile{this, "", "secret-key", "path to secret key used to sign the binary cache"};
private:
std::unique_ptr<SecretKey> secretKey;
std::string compression;
bool writeNARListing;
protected:
BinaryCacheStore(const Params & params);
[[noreturn]] void notImpl();
public:
virtual bool fileExists(const std::string & path) = 0;
virtual void upsertFile(const std::string & path, const std::string & data) = 0;
virtual void upsertFile(const std::string & path,
const std::string & data,
const std::string & mimeType) = 0;
/* Return the contents of the specified file, or null if it
doesn't exist. */
@@ -61,7 +63,7 @@ public:
bool isValidPathUncached(const Path & path) override;
PathSet queryAllValidPaths() override
{ notImpl(); }
{ unsupported(); }
void queryPathInfoUncached(const Path & path,
std::function<void(std::shared_ptr<ValidPathInfo>)> success,
@@ -69,16 +71,16 @@ public:
void queryReferrers(const Path & path,
PathSet & referrers) override
{ notImpl(); }
{ unsupported(); }
PathSet queryDerivationOutputs(const Path & path) override
{ notImpl(); }
{ unsupported(); }
StringSet queryDerivationOutputNames(const Path & path) override
{ notImpl(); }
{ unsupported(); }
Path queryPathFromHashPart(const string & hashPart) override
{ notImpl(); }
{ unsupported(); }
bool wantMassQuery() override { return wantMassQuery_; }
@@ -95,32 +97,31 @@ public:
void narFromPath(const Path & path, Sink & sink) override;
void buildPaths(const PathSet & paths, BuildMode buildMode) override
{ notImpl(); }
BuildResult buildDerivation(const Path & drvPath, const BasicDerivation & drv,
BuildMode buildMode) override
{ notImpl(); }
{ unsupported(); }
void ensurePath(const Path & path) override
{ notImpl(); }
{ unsupported(); }
void addTempRoot(const Path & path) override
{ notImpl(); }
{ unsupported(); }
void addIndirectRoot(const Path & path) override
{ notImpl(); }
{ unsupported(); }
Roots findRoots() override
{ notImpl(); }
{ unsupported(); }
void collectGarbage(const GCOptions & options, GCResults & results) override
{ notImpl(); }
{ unsupported(); }
ref<FSAccessor> getFSAccessor() override;
void addSignatures(const Path & storePath, const StringSet & sigs) override
{ notImpl(); }
{ unsupported(); }
std::shared_ptr<std::string> getBuildLog(const Path & path) override;
};
+340 -325
View File
File diff suppressed because it is too large. Load diff
-3
View File
@@ -28,9 +28,6 @@ void builtinFetchurl(const BasicDerivation & drv, const std::string & netrcData)
DownloadRequest request(url);
request.verifyTLS = false;
/* Show a progress indicator, even though stderr is not a tty. */
request.showProgress = DownloadRequest::yes;
/* Note: have to use a fresh downloader here because we're in
a forked process. */
auto data = makeDownloader()->download(request);
+2 -2
View File
@@ -105,12 +105,12 @@ PublicKeys getDefaultPublicKeys()
// FIXME: filter duplicates
for (auto s : settings.get("binary-cache-public-keys", Strings())) {
for (auto s : settings.binaryCachePublicKeys.get()) {
PublicKey key(s);
publicKeys.emplace(key.name, key);
}
for (auto secretKeyFile : settings.get("secret-key-files", Strings())) {
for (auto secretKeyFile : settings.secretKeyFiles.get()) {
try {
SecretKey secretKey(readFile(secretKeyFile));
publicKeys.emplace(secretKey.name, secretKey.toPublicKey());
+6 -15
View File
@@ -4,7 +4,7 @@
#include "util.hh"
#include "worker-protocol.hh"
#include "fs-accessor.hh"
#include "istringstream_nocopy.hh"
namespace nix {
@@ -152,7 +152,7 @@ static StringSet parseStrings(std::istream & str, bool arePaths)
static Derivation parseDerivation(const string & s)
{
Derivation drv;
std::istringstream str(s);
istringstream_nocopy str(s);
expect(str, "Derive([");
/* Parse the list of outputs. */
@@ -309,15 +309,6 @@ bool BasicDerivation::isFixedOutput() const
}
bool BasicDerivation::isImpure() const
{
// FIXME: drop single output restriction
return outputs.size() == 1 &&
outputs.begin()->first == "out" &&
get(env, "__impure", "") == "1";
}
DrvHashes drvHashes;
@@ -406,8 +397,8 @@ PathSet BasicDerivation::outputPaths() const
Source & readDerivation(Source & in, Store & store, BasicDerivation & drv)
{
drv.outputs.clear();
auto nr = readInt(in);
for (unsigned int n = 0; n < nr; n++) {
auto nr = readNum<size_t>(in);
for (size_t n = 0; n < nr; n++) {
auto name = readString(in);
DerivationOutput o;
in >> o.path >> o.hashAlgo >> o.hash;
@@ -419,8 +410,8 @@ Source & readDerivation(Source & in, Store & store, BasicDerivation & drv)
in >> drv.platform >> drv.builder;
drv.args = readStrings<Strings>(in);
nr = readInt(in);
for (unsigned int n = 0; n < nr; n++) {
nr = readNum<size_t>(in);
for (size_t n = 0; n < nr; n++) {
auto key = readString(in);
auto value = readString(in);
drv.env[key] = value;
-3
View File
@@ -66,9 +66,6 @@ struct BasicDerivation
/* Return true iff this is a fixed-output derivation. */
bool isFixedOutput() const;
/* Return true iff this is an impure derivation. */
bool isImpure() const;
/* Return the output paths of a derivation. */
PathSet outputPaths() const;
+65 -47
View File
@@ -5,6 +5,11 @@
#include "store-api.hh"
#include "archive.hh"
#include "s3.hh"
#include "compression.hh"
#ifdef ENABLE_S3
#include <aws/core/client/ClientConfiguration.h>
#endif
#include <unistd.h>
#include <fcntl.h>
@@ -34,6 +39,16 @@ std::string resolveUri(const std::string & uri)
return uri;
}
ref<std::string> decodeContent(const std::string & encoding, ref<std::string> data)
{
if (encoding == "")
return data;
else if (encoding == "br")
return decompress(encoding, *data);
else
throw Error("unsupported Content-Encoding ‘%s’", encoding);
}
struct CurlDownloader : public Downloader
{
CURLM * curlm = 0;
@@ -48,6 +63,7 @@ struct CurlDownloader : public Downloader
CurlDownloader & downloader;
DownloadRequest request;
DownloadResult result;
Activity act;
bool done = false; // whether either the success or failure function has been called
std::function<void(const DownloadResult &)> success;
std::function<void(std::exception_ptr exc)> failure;
@@ -55,10 +71,6 @@ struct CurlDownloader : public Downloader
bool active = false; // whether the handle has been added to the multi object
std::string status;
bool showProgress = false;
double prevProgressTime{0}, startTime{0};
unsigned int moveBack{1};
unsigned int attempt = 0;
/* Don't start this download until the specified time point
@@ -67,15 +79,15 @@ struct CurlDownloader : public Downloader
struct curl_slist * requestHeaders = 0;
std::string encoding;
DownloadItem(CurlDownloader & downloader, const DownloadRequest & request)
: downloader(downloader), request(request)
{
showProgress =
request.showProgress == DownloadRequest::yes ||
(request.showProgress == DownloadRequest::automatic && isatty(STDERR_FILENO));
if (!request.expectedETag.empty())
requestHeaders = curl_slist_append(requestHeaders, ("If-None-Match: " + request.expectedETag).c_str());
logger->event(evDownloadCreated, act, request.uri);
}
~DownloadItem()
@@ -92,6 +104,7 @@ struct CurlDownloader : public Downloader
} catch (...) {
ignoreException();
}
logger->event(evDownloadDestroyed, act);
}
template<class T>
@@ -124,6 +137,7 @@ struct CurlDownloader : public Downloader
auto ss = tokenizeString<vector<string>>(line, " ");
status = ss.size() >= 2 ? ss[1] : "";
result.data = std::make_shared<std::string>();
encoding = "";
} else {
auto i = line.find(':');
if (i != string::npos) {
@@ -139,7 +153,8 @@ struct CurlDownloader : public Downloader
debug(format("shutting down on 200 HTTP response with expected ETag"));
return 0;
}
}
} else if (name == "content-encoding")
encoding = trim(string(line, i + 1));;
}
}
return realSize;
@@ -152,19 +167,7 @@ struct CurlDownloader : public Downloader
int progressCallback(double dltotal, double dlnow)
{
if (showProgress) {
double now = getTime();
if (prevProgressTime <= now - 1) {
string s = (format(" [%1$.0f/%2$.0f KiB, %3$.1f KiB/s]")
% (dlnow / 1024.0)
% (dltotal / 1024.0)
% (now == startTime ? 0 : dlnow / 1024.0 / (now - startTime))).str();
std::cerr << "\e[" << moveBack << "D" << s;
moveBack = s.size();
std::cerr.flush();
prevProgressTime = now;
}
}
logger->event(evDownloadProgress, act, dltotal, dlnow);
return _isInterrupted;
}
@@ -182,13 +185,6 @@ struct CurlDownloader : public Downloader
void init()
{
// FIXME: handle parallel downloads.
if (showProgress) {
std::cerr << (format("downloading ‘%1%’... ") % request.uri);
std::cerr.flush();
startTime = getTime();
}
if (!req) req = curl_easy_init();
curl_easy_reset(req);
@@ -201,7 +197,9 @@ struct CurlDownloader : public Downloader
curl_easy_setopt(req, CURLOPT_URL, request.uri.c_str());
curl_easy_setopt(req, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(req, CURLOPT_NOSIGNAL, 1);
curl_easy_setopt(req, CURLOPT_USERAGENT, ("curl/" LIBCURL_VERSION " Nix/" + nixVersion).c_str());
curl_easy_setopt(req, CURLOPT_USERAGENT,
("curl/" LIBCURL_VERSION " Nix/" + nixVersion +
(settings.userAgentSuffix != "" ? " " + settings.userAgentSuffix.get() : "")).c_str());
#if LIBCURL_VERSION_NUM >= 0x072b00
curl_easy_setopt(req, CURLOPT_PIPEWAIT, 1);
#endif
@@ -223,17 +221,19 @@ struct CurlDownloader : public Downloader
if (request.head)
curl_easy_setopt(req, CURLOPT_NOBODY, 1);
if (request.verifyTLS)
curl_easy_setopt(req, CURLOPT_CAINFO,
getEnv("NIX_SSL_CERT_FILE", getEnv("SSL_CERT_FILE", "/etc/ssl/certs/ca-certificates.crt")).c_str());
else {
if (request.verifyTLS) {
if (settings.caFile != "")
curl_easy_setopt(req, CURLOPT_CAINFO, settings.caFile.c_str());
} else {
curl_easy_setopt(req, CURLOPT_SSL_VERIFYPEER, 0);
curl_easy_setopt(req, CURLOPT_SSL_VERIFYHOST, 0);
}
curl_easy_setopt(req, CURLOPT_CONNECTTIMEOUT, settings.connectTimeout.get());
/* If no file exist in the specified path, curl continues to work
anyway as if netrc support was disabled. */
curl_easy_setopt(req, CURLOPT_NETRC_FILE, settings.netrcFile.c_str());
curl_easy_setopt(req, CURLOPT_NETRC_FILE, settings.netrcFile.get().c_str());
curl_easy_setopt(req, CURLOPT_NETRC, CURL_NETRC_OPTIONAL);
result.data = std::make_shared<std::string>();
@@ -241,10 +241,6 @@ struct CurlDownloader : public Downloader
void finish(CURLcode code)
{
if (showProgress)
//std::cerr << "\e[" << moveBack << "D\e[K\n";
std::cerr << "\n";
long httpStatus = 0;
curl_easy_getinfo(req, CURLINFO_RESPONSE_CODE, &httpStatus);
@@ -266,14 +262,35 @@ struct CurlDownloader : public Downloader
{
result.cached = httpStatus == 304;
done = true;
callSuccess(success, failure, const_cast<const DownloadResult &>(result));
try {
result.data = decodeContent(encoding, ref<std::string>(result.data));
callSuccess(success, failure, const_cast<const DownloadResult &>(result));
logger->event(evDownloadSucceeded, act, result.data->size());
} catch (...) {
done = true;
callFailure(failure, std::current_exception());
}
} else {
Error err =
(httpStatus == 404 || code == CURLE_FILE_COULDNT_READ_FILE) ? NotFound :
httpStatus == 403 ? Forbidden :
(httpStatus == 408 || httpStatus == 500 || httpStatus == 503
|| httpStatus == 504 || httpStatus == 522 || httpStatus == 524
|| code == CURLE_COULDNT_RESOLVE_HOST || code == CURLE_RECV_ERROR) ? Transient :
|| code == CURLE_COULDNT_RESOLVE_HOST
|| code == CURLE_RECV_ERROR
// this seems to occur occasionally for retriable reasons, and shows up in an error like this:
// curl: (23) Failed writing body (315 != 16366)
|| code == CURLE_WRITE_ERROR
// this is a generic SSL failure that in some cases (e.g., certificate error) is permanent but also appears in transient cases, so we consider it retryable
|| code == CURLE_SSL_CONNECT_ERROR
#if LIBCURL_VERSION_NUM >= 0x073200
|| code == CURLE_HTTP2
|| code == CURLE_HTTP2_STREAM
#endif
) ? Transient :
Misc;
attempt++;
@@ -331,9 +348,9 @@ struct CurlDownloader : public Downloader
curl_multi_setopt(curlm, CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
#endif
curl_multi_setopt(curlm, CURLMOPT_MAX_TOTAL_CONNECTIONS,
settings.get("binary-caches-parallel-connections", 25));
settings.binaryCachesParallelConnections.get());
enableHttp2 = settings.get("enable-http2", true);
enableHttp2 = settings.enableHttp2;
wakeupPipe.create();
fcntl(wakeupPipe.readSide.get(), F_SETFL, O_NONBLOCK);
@@ -491,7 +508,7 @@ struct CurlDownloader : public Downloader
// FIXME: do this on a worker thread
sync2async<DownloadResult>(success, failure, [&]() -> DownloadResult {
#ifdef ENABLE_S3
S3Helper s3Helper;
S3Helper s3Helper(Aws::Region::US_EAST_1); // FIXME: make configurable
auto slash = request.uri.find('/', 5);
if (slash == std::string::npos)
throw nix::Error("bad S3 URI ‘%s’", request.uri);
@@ -573,7 +590,7 @@ Path Downloader::downloadCached(ref<Store> store, const string & url_, bool unpa
string expectedETag;
int ttl = settings.get("tarball-ttl", 60 * 60);
int ttl = settings.tarballTtl;
bool skip = false;
if (pathExists(fileLink) && pathExists(dataFile)) {
@@ -612,6 +629,7 @@ Path Downloader::downloadCached(ref<Store> store, const string & url_, bool unpa
Hash hash = hashString(expectedHash ? expectedHash.type : htSHA256, *res.data);
info.path = store->makeFixedOutputPath(false, hash, name);
info.narHash = hashString(htSHA256, *sink.s);
info.narSize = sink.s->size();
info.ca = makeFixedOutputCA(false, hash);
store->addToStore(info, sink.s, false, true);
storePath = info.path;
@@ -641,7 +659,7 @@ Path Downloader::downloadCached(ref<Store> store, const string & url_, bool unpa
Path tmpDir = createTempDir();
AutoDelete autoDelete(tmpDir, true);
// FIXME: this requires GNU tar for decompression.
runProgram("tar", true, {"xf", storePath, "-C", tmpDir, "--strip-components", "1"}, "");
runProgram("tar", true, {"xf", storePath, "-C", tmpDir, "--strip-components", "1"});
unpackedStorePath = store->addToStore(name, tmpDir, true, htSHA256, defaultPathFilter, false);
}
replaceSymlink(unpackedStorePath, unpackedLink);
@@ -649,7 +667,7 @@ Path Downloader::downloadCached(ref<Store> store, const string & url_, bool unpa
}
if (expectedStorePath != "" && storePath != expectedStorePath)
throw nix::Error(format("hash mismatch in file downloaded from ‘%s’") % url);
throw nix::Error("store path mismatch in file downloaded from ‘%s’", url);
return storePath;
}
+4 -2
View File
@@ -13,9 +13,8 @@ struct DownloadRequest
std::string uri;
std::string expectedETag;
bool verifyTLS = true;
enum { yes, no, automatic } showProgress = yes;
bool head = false;
size_t tries = 1;
size_t tries = 5;
unsigned int baseRetryTimeMs = 250;
DownloadRequest(const std::string & uri) : uri(uri) { }
@@ -73,4 +72,7 @@ public:
bool isUri(const string & s);
/* Decode data according to the Content-Encoding header. */
ref<std::string> decodeContent(const std::string & encoding, ref<std::string> data);
}
+10 -32
View File
@@ -30,13 +30,13 @@ void Store::exportPaths(const Paths & paths, Sink & sink)
std::reverse(sorted.begin(), sorted.end());
std::string doneLabel("paths exported");
logger->incExpected(doneLabel, sorted.size());
//logger->incExpected(doneLabel, sorted.size());
for (auto & path : sorted) {
Activity act(*logger, lvlInfo, format("exporting path ‘%s’") % path);
//Activity act(*logger, lvlInfo, format("exporting path ‘%s’") % path);
sink << 1;
exportPath(path, sink);
logger->incProgress(doneLabel);
//logger->incProgress(doneLabel);
}
sink << 0;
@@ -61,39 +61,17 @@ void Store::exportPath(const Path & path, Sink & sink)
hashAndWriteSink << exportMagic << path << info->references << info->deriver << 0;
}
struct TeeSource : Source
{
Source & readSource;
ref<std::string> data;
TeeSource(Source & readSource)
: readSource(readSource)
, data(make_ref<std::string>())
{
}
size_t read(unsigned char * data, size_t len)
{
size_t n = readSource.read(data, len);
this->data->append((char *) data, n);
return n;
}
};
struct NopSink : ParseSink
{
};
Paths Store::importPaths(Source & source, std::shared_ptr<FSAccessor> accessor, bool dontCheckSigs)
{
Paths res;
while (true) {
unsigned long long n = readLongLong(source);
auto n = readNum<uint64_t>(source);
if (n == 0) break;
if (n != 1) throw Error("input doesn't look like something created by ‘nix-store --export’");
/* Extract the NAR from the source. */
TeeSource tee(source);
NopSink sink;
parseDump(sink, tee);
TeeSink tee(source);
parseDump(tee, tee.source);
uint32_t magic = readInt(source);
if (magic != exportMagic)
@@ -103,21 +81,21 @@ Paths Store::importPaths(Source & source, std::shared_ptr<FSAccessor> accessor,
info.path = readStorePath(*this, source);
Activity act(*logger, lvlInfo, format("importing path ‘%s’") % info.path);
//Activity act(*logger, lvlInfo, format("importing path ‘%s’") % info.path);
info.references = readStorePaths<PathSet>(*this, source);
info.deriver = readString(source);
if (info.deriver != "") assertStorePath(info.deriver);
info.narHash = hashString(htSHA256, *tee.data);
info.narSize = tee.data->size();
info.narHash = hashString(htSHA256, *tee.source.data);
info.narSize = tee.source.data->size();
// Ignore optional legacy signature.
if (readInt(source) == 1)
readString(source);
addToStore(info, tee.data, false, dontCheckSigs, accessor);
addToStore(info, tee.source.data, false, dontCheckSigs, accessor);
res.push_back(info.path);
}
+9 -5
View File
@@ -426,22 +426,26 @@ void LocalStore::findRuntimeRoots(PathSet & roots)
throw SysError("iterating /proc");
}
#if !defined(__linux__)
try {
auto lsofRegex = std::regex(R"(^n(/.*)$)");
std::regex lsofRegex(R"(^n(/.*)$)");
auto lsofLines =
tokenizeString<std::vector<string>>(runProgram("lsof", true, { "-n", "-w", "-F", "n" }), "\n");
tokenizeString<std::vector<string>>(runProgram(LSOF, true, { "-n", "-w", "-F", "n" }), "\n");
for (const auto & line : lsofLines) {
auto match = std::smatch{};
std::smatch match;
if (std::regex_match(line, match, lsofRegex))
paths.emplace(match[1]);
}
} catch (ExecError & e) {
/* lsof not installed, lsof failed */
}
#endif
#if defined(__linux__)
readFileRoots("/proc/sys/kernel/modprobe", paths);
readFileRoots("/proc/sys/kernel/fbsplash", paths);
readFileRoots("/proc/sys/kernel/poweroff_cmd", paths);
#endif
for (auto & i : paths)
if (isInStore(i)) {
@@ -611,7 +615,7 @@ void LocalStore::tryToDelete(GCState & state, const Path & path)
auto realPath = realStoreDir + "/" + baseNameOf(path);
if (realPath == linksDir || realPath == trashDir) return;
Activity act(*logger, lvlDebug, format("considering whether to delete ‘%1%’") % path);
//Activity act(*logger, lvlDebug, format("considering whether to delete ‘%1%’") % path);
if (!isStorePath(path) || !isValidPath(path)) {
/* A lock file belonging to a path that we're building right
@@ -679,7 +683,7 @@ void LocalStore::removeUnusedLinks(const GCState & state)
if (unlink(path.c_str()) == -1)
throw SysError(format("deleting ‘%1%’") % path);
state.results.bytesFreed += st.st_blocks * 512;
state.results.bytesFreed += st.st_blocks * 512ULL;
}
struct stat st;
+92 -223
View File
@@ -1,10 +1,11 @@
#include "globals.hh"
#include "util.hh"
#include "archive.hh"
#include "args.hh"
#include <map>
#include <algorithm>
#include <unistd.h>
#include <map>
#include <thread>
namespace nix {
@@ -17,250 +18,118 @@ namespace nix {
must be deleted and recreated on startup.) */
#define DEFAULT_SOCKET_PATH "/daemon-socket/socket"
/* chroot-like behavior from Apple's sandbox */
#if __APPLE__
#define DEFAULT_ALLOWED_IMPURE_PREFIXES "/System/Library /usr/lib /dev /bin/sh"
#else
#define DEFAULT_ALLOWED_IMPURE_PREFIXES ""
#endif
Settings settings;
Settings::Settings()
: Config({})
, nixPrefix(NIX_PREFIX)
, nixStore(canonPath(getEnv("NIX_STORE_DIR", getEnv("NIX_STORE", NIX_STORE_DIR))))
, nixDataDir(canonPath(getEnv("NIX_DATA_DIR", NIX_DATA_DIR)))
, nixLogDir(canonPath(getEnv("NIX_LOG_DIR", NIX_LOG_DIR)))
, nixStateDir(canonPath(getEnv("NIX_STATE_DIR", NIX_STATE_DIR)))
, nixConfDir(canonPath(getEnv("NIX_CONF_DIR", NIX_CONF_DIR)))
, nixLibexecDir(canonPath(getEnv("NIX_LIBEXEC_DIR", NIX_LIBEXEC_DIR)))
, nixBinDir(canonPath(getEnv("NIX_BIN_DIR", NIX_BIN_DIR)))
, nixDaemonSocketFile(canonPath(nixStateDir + DEFAULT_SOCKET_PATH))
{
nixPrefix = NIX_PREFIX;
nixStore = canonPath(getEnv("NIX_STORE_DIR", getEnv("NIX_STORE", NIX_STORE_DIR)));
nixDataDir = canonPath(getEnv("NIX_DATA_DIR", NIX_DATA_DIR));
nixLogDir = canonPath(getEnv("NIX_LOG_DIR", NIX_LOG_DIR));
nixStateDir = canonPath(getEnv("NIX_STATE_DIR", NIX_STATE_DIR));
nixConfDir = canonPath(getEnv("NIX_CONF_DIR", NIX_CONF_DIR));
nixLibexecDir = canonPath(getEnv("NIX_LIBEXEC_DIR", NIX_LIBEXEC_DIR));
nixBinDir = canonPath(getEnv("NIX_BIN_DIR", NIX_BIN_DIR));
nixDaemonSocketFile = canonPath(nixStateDir + DEFAULT_SOCKET_PATH);
// should be set with the other config options, but depends on nixLibexecDir
#ifdef __APPLE__
preBuildHook = nixLibexecDir + "/nix/resolve-system-dependencies";
#endif
keepFailed = false;
keepGoing = false;
tryFallback = false;
maxBuildJobs = 1;
buildCores = 1;
#ifdef _SC_NPROCESSORS_ONLN
long res = sysconf(_SC_NPROCESSORS_ONLN);
if (res > 0) buildCores = res;
#endif
readOnlyMode = false;
thisSystem = SYSTEM;
maxSilentTime = 0;
buildTimeout = 0;
useBuildHook = true;
reservedSize = 8 * 1024 * 1024;
fsyncMetadata = true;
useSQLiteWAL = true;
syncBeforeRegistering = false;
useSubstitutes = true;
buildUsersGroup = getuid() == 0 ? "nixbld" : "";
useSshSubstituter = true;
impersonateLinux26 = false;
keepLog = true;
compressLog = true;
maxLogSize = 0;
pollInterval = 5;
checkRootReachability = false;
gcKeepOutputs = false;
gcKeepDerivations = true;
autoOptimiseStore = false;
envKeepDerivations = false;
lockCPU = getEnv("NIX_AFFINITY_HACK", "1") == "1";
showTrace = false;
enableImportNative = false;
netrcFile = fmt("%s/%s", nixConfDir, "netrc");
}
caFile = getEnv("NIX_SSL_CERT_FILE", getEnv("SSL_CERT_FILE", ""));
if (caFile == "") {
for (auto & fn : {"/etc/ssl/certs/ca-certificates.crt", "/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"})
if (pathExists(fn)) {
caFile = fn;
break;
}
}
/* Backwards compatibility. */
auto s = getEnv("NIX_REMOTE_SYSTEMS");
if (s != "") builderFiles = tokenizeString<Strings>(s, ":");
#if defined(__linux__) && defined(SANDBOX_SHELL)
sandboxPaths = tokenizeString<StringSet>("/bin/sh=" SANDBOX_SHELL);
#endif
allowedImpureHostPrefixes = tokenizeString<StringSet>(DEFAULT_ALLOWED_IMPURE_PREFIXES);
}
void Settings::loadConfFile()
{
Path settingsFile = (format("%1%/%2%") % nixConfDir % "nix.conf").str();
if (!pathExists(settingsFile)) return;
string contents = readFile(settingsFile);
applyConfigFile(nixConfDir + "/nix.conf");
unsigned int pos = 0;
/* We only want to send overrides to the daemon, i.e. stuff from
~/.nix/nix.conf or the command line. */
resetOverriden();
while (pos < contents.size()) {
string line;
while (pos < contents.size() && contents[pos] != '\n')
line += contents[pos++];
pos++;
string::size_type hash = line.find('#');
if (hash != string::npos)
line = string(line, 0, hash);
vector<string> tokens = tokenizeString<vector<string> >(line);
if (tokens.empty()) continue;
if (tokens.size() < 2 || tokens[1] != "=")
throw Error(format("illegal configuration line ‘%1%’ in ‘%2%’") % line % settingsFile);
string name = tokens[0];
vector<string>::iterator i = tokens.begin();
advance(i, 2);
settings[name] = concatStringsSep(" ", Strings(i, tokens.end())); // FIXME: slow
};
applyConfigFile(getConfigDir() + "/nix/nix.conf");
}
void Settings::set(const string & name, const string & value)
{
settings[name] = value;
overrides[name] = value;
Config::set(name, value);
}
string Settings::get(const string & name, const string & def)
unsigned int Settings::getDefaultCores()
{
auto i = settings.find(name);
if (i == settings.end()) return def;
return i->second;
return std::max(1U, std::thread::hardware_concurrency());
}
Strings Settings::get(const string & name, const Strings & def)
{
auto i = settings.find(name);
if (i == settings.end()) return def;
return tokenizeString<Strings>(i->second);
}
bool Settings::get(const string & name, bool def)
{
bool res = def;
_get(res, name);
return res;
}
int Settings::get(const string & name, int def)
{
int res = def;
_get(res, name);
return res;
}
void Settings::update()
{
_get(tryFallback, "build-fallback");
_get(maxBuildJobs, "build-max-jobs");
_get(buildCores, "build-cores");
_get(thisSystem, "system");
_get(maxSilentTime, "build-max-silent-time");
_get(buildTimeout, "build-timeout");
_get(reservedSize, "gc-reserved-space");
_get(fsyncMetadata, "fsync-metadata");
_get(useSQLiteWAL, "use-sqlite-wal");
_get(syncBeforeRegistering, "sync-before-registering");
_get(useSubstitutes, "build-use-substitutes");
_get(buildUsersGroup, "build-users-group");
_get(impersonateLinux26, "build-impersonate-linux-26");
_get(keepLog, "build-keep-log");
_get(compressLog, "build-compress-log");
_get(maxLogSize, "build-max-log-size");
_get(pollInterval, "build-poll-interval");
_get(checkRootReachability, "gc-check-reachability");
_get(gcKeepOutputs, "gc-keep-outputs");
_get(gcKeepDerivations, "gc-keep-derivations");
_get(autoOptimiseStore, "auto-optimise-store");
_get(envKeepDerivations, "env-keep-derivations");
_get(sshSubstituterHosts, "ssh-substituter-hosts");
_get(useSshSubstituter, "use-ssh-substituter");
_get(logServers, "log-servers");
_get(enableImportNative, "allow-unsafe-native-code-during-evaluation");
_get(useCaseHack, "use-case-hack");
_get(preBuildHook, "pre-build-hook");
_get(keepGoing, "keep-going");
_get(keepFailed, "keep-failed");
_get(netrcFile, "netrc-file");
}
void Settings::_get(string & res, const string & name)
{
SettingsMap::iterator i = settings.find(name);
if (i == settings.end()) return;
res = i->second;
}
void Settings::_get(bool & res, const string & name)
{
SettingsMap::iterator i = settings.find(name);
if (i == settings.end()) return;
if (i->second == "true") res = true;
else if (i->second == "false") res = false;
else throw Error(format("configuration option ‘%1%’ should be either ‘true’ or ‘false’, not ‘%2%’")
% name % i->second);
}
void Settings::_get(StringSet & res, const string & name)
{
SettingsMap::iterator i = settings.find(name);
if (i == settings.end()) return;
res.clear();
Strings ss = tokenizeString<Strings>(i->second);
res.insert(ss.begin(), ss.end());
}
void Settings::_get(Strings & res, const string & name)
{
SettingsMap::iterator i = settings.find(name);
if (i == settings.end()) return;
res = tokenizeString<Strings>(i->second);
}
template<class N> void Settings::_get(N & res, const string & name)
{
SettingsMap::iterator i = settings.find(name);
if (i == settings.end()) return;
if (!string2Int(i->second, res))
throw Error(format("configuration setting ‘%1%’ should have an integer value") % name);
}
string Settings::pack()
{
string s;
for (auto & i : settings) {
if (i.first.find('\n') != string::npos ||
i.first.find('=') != string::npos ||
i.second.find('\n') != string::npos)
throw Error("illegal option name/value");
s += i.first; s += '='; s += i.second; s += '\n';
}
return s;
}
void Settings::unpack(const string & pack) {
Strings lines = tokenizeString<Strings>(pack, "\n");
for (auto & i : lines) {
string::size_type eq = i.find('=');
if (eq == string::npos)
throw Error("illegal option name/value");
set(i.substr(0, eq), i.substr(eq + 1));
}
}
Settings::SettingsMap Settings::getOverrides()
{
return overrides;
}
const string nixVersion = PACKAGE_VERSION;
template<> void BaseSetting<SandboxMode>::set(const std::string & str)
{
if (str == "true") value = smEnabled;
else if (str == "relaxed") value = smRelaxed;
else if (str == "false") value = smDisabled;
else throw UsageError("option '%s' has invalid value '%s'", name, str);
}
template<> std::string BaseSetting<SandboxMode>::to_string()
{
if (value == smEnabled) return "true";
else if (value == smRelaxed) return "relaxed";
else if (value == smDisabled) return "false";
else abort();
}
template<> void BaseSetting<SandboxMode>::toJSON(JSONPlaceholder & out)
{
AbstractSetting::toJSON(out);
}
template<> void BaseSetting<SandboxMode>::convertToArg(Args & args, const std::string & category)
{
args.mkFlag()
.longName(name)
.description("Enable sandboxing.")
.handler([=](Strings ss) { value = smEnabled; })
.category(category);
args.mkFlag()
.longName("no-" + name)
.description("Disable sandboxing.")
.handler([=](Strings ss) { value = smDisabled; })
.category(category);
args.mkFlag()
.longName("relaxed-" + name)
.description("Enable sandboxing, but allow builds to disable it.")
.handler([=](Strings ss) { value = smRelaxed; })
.category(category);
}
void MaxBuildJobsSetting::set(const std::string & str)
{
if (str == "auto") value = std::max(1U, std::thread::hardware_concurrency());
else if (!string2Int(str, value))
throw UsageError("configuration setting ‘%s’ should be ‘auto’ or an integer", name);
}
}
+228 -110
View File
@@ -1,7 +1,7 @@
#pragma once
#include "types.hh"
#include "logging.hh"
#include "config.hh"
#include <map>
#include <sys/types.h>
@@ -9,10 +9,48 @@
namespace nix {
typedef enum { smEnabled, smRelaxed, smDisabled } SandboxMode;
struct Settings {
extern bool useCaseHack; // FIXME
typedef std::map<string, string> SettingsMap;
struct CaseHackSetting : public BaseSetting<bool>
{
CaseHackSetting(Config * options,
const std::string & name,
const std::string & description,
const std::set<std::string> & aliases = {})
: BaseSetting<bool>(useCaseHack, name, description, aliases)
{
options->addSetting(this);
}
void set(const std::string & str) override
{
BaseSetting<bool>::set(str);
nix::useCaseHack = true;
}
};
struct MaxBuildJobsSetting : public BaseSetting<unsigned int>
{
MaxBuildJobsSetting(Config * options,
unsigned int def,
const std::string & name,
const std::string & description,
const std::set<std::string> & aliases = {})
: BaseSetting<unsigned int>(def, name, description, aliases)
{
options->addSetting(this);
}
void set(const std::string & str) override;
};
class Settings : public Config {
unsigned int getDefaultCores();
public:
Settings();
@@ -20,29 +58,13 @@ struct Settings {
void set(const string & name, const string & value);
string get(const string & name, const string & def);
Strings get(const string & name, const Strings & def);
bool get(const string & name, bool def);
int get(const string & name, int def);
void update();
string pack();
void unpack(const string & pack);
SettingsMap getOverrides();
Path nixPrefix;
/* The directory where we store sources and derived files. */
Path nixStore;
Path nixDataDir; /* !!! fix */
Path nixPrefix;
/* The directory where we log various operations. */
Path nixLogDir;
@@ -61,17 +83,14 @@ struct Settings {
/* File name of the socket the daemon listens to. */
Path nixDaemonSocketFile;
/* Whether to keep temporary directories of failed builds. */
bool keepFailed;
Setting<bool> keepFailed{this, false, "keep-failed",
"Whether to keep temporary directories of failed builds."};
/* Whether to keep building subgoals when a sibling (another
subgoal of the same goal) fails. */
bool keepGoing;
Setting<bool> keepGoing{this, false, "keep-going",
"Whether to keep building derivations when another build fails."};
/* Whether, if we cannot realise the known closure corresponding
to a derivation, we should try to normalise the derivation
instead. */
bool tryFallback;
Setting<bool> tryFallback{this, false, "build-fallback",
"Whether to fall back to building when substitution fails."};
/* Whether to show build log output in real time. */
bool verboseBuild = true;
@@ -80,129 +99,228 @@ struct Settings {
the log to show if a build fails. */
size_t logLines = 10;
/* Maximum number of parallel build jobs. 0 means unlimited. */
unsigned int maxBuildJobs;
MaxBuildJobsSetting maxBuildJobs{this, 1, "build-max-jobs",
"Maximum number of parallel build jobs. \"auto\" means use number of cores."};
/* Number of CPU cores to utilize in parallel within a build,
i.e. by passing this number to Make via '-j'. 0 means that the
number of actual CPU cores on the local host ought to be
auto-detected. */
unsigned int buildCores;
Setting<unsigned int> buildCores{this, getDefaultCores(), "build-cores",
"Number of CPU cores to utilize in parallel within a build, "
"i.e. by passing this number to Make via '-j'. 0 means that the "
"number of actual CPU cores on the local host ought to be "
"auto-detected."};
/* Read-only mode. Don't copy stuff to the store, don't change
the database. */
bool readOnlyMode;
bool readOnlyMode = false;
/* The canonical system name, as returned by config.guess. */
string thisSystem;
Setting<std::string> thisSystem{this, SYSTEM, "system",
"The canonical Nix system name."};
/* The maximum time in seconds that a builer can go without
producing any output on stdout/stderr before it is killed. 0
means infinity. */
time_t maxSilentTime;
Setting<time_t> maxSilentTime{this, 0, "build-max-silent-time",
"The maximum time in seconds that a builer can go without "
"producing any output on stdout/stderr before it is killed. "
"0 means infinity."};
/* The maximum duration in seconds that a builder can run. 0
means infinity. */
time_t buildTimeout;
Setting<time_t> buildTimeout{this, 0, "build-timeout",
"The maximum duration in seconds that a builder can run. "
"0 means infinity."};
/* Whether to use build hooks (for distributed builds). Sometimes
users want to disable this from the command-line. */
bool useBuildHook;
Setting<bool> useBuildHook{this, true, "remote-builds",
"Whether to use build hooks (for distributed builds)."};
/* Amount of reserved space for the garbage collector
(/nix/var/nix/db/reserved). */
off_t reservedSize;
PathSetting buildHook{this, true, nixLibexecDir + "/nix/build-remote", "build-hook",
"The path of the helper program that executes builds to remote machines."};
/* Whether SQLite should use fsync. */
bool fsyncMetadata;
Setting<std::string> builders{this, "", "builders",
"A semicolon-separated list of build machines, in the format of nix.machines."};
/* Whether SQLite should use WAL mode. */
bool useSQLiteWAL;
Setting<Strings> builderFiles{this,
{nixConfDir + "/machines"}, "builder-files",
"A list of files specifying build machines."};
/* Whether to call sync() before registering a path as valid. */
bool syncBeforeRegistering;
Setting<off_t> reservedSize{this, 8 * 1024 * 1024, "gc-reserved-space",
"Amount of reserved disk space for the garbage collector."};
/* Whether to use substitutes. */
bool useSubstitutes;
Setting<bool> fsyncMetadata{this, true, "fsync-metadata",
"Whether SQLite should use fsync()."};
/* The Unix group that contains the build users. */
string buildUsersGroup;
Setting<bool> useSQLiteWAL{this, true, "use-sqlite-wal",
"Whether SQLite should use WAL mode."};
/* Set of ssh connection strings for the ssh substituter */
Strings sshSubstituterHosts;
Setting<bool> syncBeforeRegistering{this, false, "sync-before-registering",
"Whether to call sync() before registering a path as valid."};
/* Whether to use the ssh substituter at all */
bool useSshSubstituter;
Setting<bool> useSubstitutes{this, true, "build-use-substitutes",
"Whether to use substitutes."};
/* Whether to impersonate a Linux 2.6 machine on newer kernels. */
bool impersonateLinux26;
Setting<std::string> buildUsersGroup{this, "", "build-users-group",
"The Unix group that contains the build users."};
/* Whether to store build logs. */
bool keepLog;
Setting<bool> impersonateLinux26{this, false, "build-impersonate-linux-26",
"Whether to impersonate a Linux 2.6 machine on newer kernels."};
/* Whether to compress logs. */
bool compressLog;
Setting<bool> keepLog{this, true, "build-keep-log",
"Whether to store build logs."};
/* Maximum number of bytes a builder can write to stdout/stderr
before being killed (0 means no limit). */
unsigned long maxLogSize;
Setting<bool> compressLog{this, true, "build-compress-log",
"Whether to compress logs."};
Setting<unsigned long> maxLogSize{this, 0, "build-max-log-size",
"Maximum number of bytes a builder can write to stdout/stderr "
"before being killed (0 means no limit)."};
/* When build-repeat > 0 and verboseBuild == true, whether to
print repeated builds (i.e. builds other than the first one) to
stderr. Hack to prevent Hydra logs from being polluted. */
bool printRepeatedBuilds = true;
/* How often (in seconds) to poll for locks. */
unsigned int pollInterval;
Setting<unsigned int> pollInterval{this, 5, "build-poll-interval",
"How often (in seconds) to poll for locks."};
/* Whether to check if new GC roots can in fact be found by the
garbage collector. */
bool checkRootReachability;
Setting<bool> checkRootReachability{this, false, "gc-check-reachability",
"Whether to check if new GC roots can in fact be found by the "
"garbage collector."};
/* Whether the garbage collector should keep outputs of live
derivations. */
bool gcKeepOutputs;
Setting<bool> gcKeepOutputs{this, false, "gc-keep-outputs",
"Whether the garbage collector should keep outputs of live derivations."};
/* Whether the garbage collector should keep derivers of live
paths. */
bool gcKeepDerivations;
Setting<bool> gcKeepDerivations{this, true, "gc-keep-derivations",
"Whether the garbage collector should keep derivers of live paths."};
/* Whether to automatically replace files with identical contents
with hard links. */
bool autoOptimiseStore;
Setting<bool> autoOptimiseStore{this, false, "auto-optimise-store",
"Whether to automatically replace files with identical contents with hard links."};
/* Whether to add derivations as a dependency of user environments
(to prevent them from being GCed). */
bool envKeepDerivations;
Setting<bool> envKeepDerivations{this, false, "env-keep-derivations",
"Whether to add derivations as a dependency of user environments "
"(to prevent them from being GCed)."};
/* Whether to lock the Nix client and worker to the same CPU. */
bool lockCPU;
/* Whether to show a stack trace if Nix evaluation fails. */
bool showTrace;
bool showTrace = false;
/* A list of URL prefixes that can return Nix build logs. */
Strings logServers;
Setting<bool> enableNativeCode{this, false, "allow-unsafe-native-code-during-evaluation",
"Whether builtin functions that allow executing native code should be enabled."};
/* Whether the importNative primop should be enabled */
bool enableImportNative;
Setting<SandboxMode> sandboxMode{this, smDisabled, "build-use-sandbox",
"Whether to enable sandboxed builds. Can be \"true\", \"false\" or \"relaxed\".",
{"build-use-chroot"}};
/* The hook to run just before a build to set derivation-specific
build settings */
Path preBuildHook;
Setting<PathSet> sandboxPaths{this, {}, "build-sandbox-paths",
"The paths to make available inside the build sandbox.",
{"build-chroot-dirs"}};
/* Path to the netrc file used to obtain usernames/passwords for
downloads. */
Path netrcFile;
Setting<PathSet> extraSandboxPaths{this, {}, "build-extra-sandbox-paths",
"Additional paths to make available inside the build sandbox.",
{"build-extra-chroot-dirs"}};
private:
SettingsMap settings, overrides;
Setting<bool> restrictEval{this, false, "restrict-eval",
"Whether to restrict file system access to paths in $NIX_PATH, "
"and to disallow fetching files from the network."};
void _get(string & res, const string & name);
void _get(bool & res, const string & name);
void _get(StringSet & res, const string & name);
void _get(Strings & res, const string & name);
template<class N> void _get(N & res, const string & name);
Setting<size_t> buildRepeat{this, 0, "build-repeat",
"The number of times to repeat a build in order to verify determinism."};
#if __linux__
Setting<std::string> sandboxShmSize{this, "50%", "sandbox-dev-shm-size",
"The size of /dev/shm in the build sandbox."};
Setting<Path> sandboxBuildDir{this, "/build", "sandbox-build-dir",
"The build directory inside the sandbox."};
#endif
Setting<PathSet> allowedImpureHostPrefixes{this, {}, "allowed-impure-host-deps",
"Which prefixes to allow derivations to ask for access to (primarily for Darwin)."};
#if __APPLE__
Setting<bool> darwinLogSandboxViolations{this, false, "darwin-log-sandbox-violations",
"Whether to log Darwin sandbox access violations to the system log."};
#endif
Setting<bool> runDiffHook{this, false, "run-diff-hook",
"Whether to run the program specified by the diff-hook setting "
"repeated builds produce a different result. Typically used to "
"plug in diffoscope."};
PathSetting diffHook{this, true, "", "diff-hook",
"A program that prints out the differences between the two paths "
"specified on its command line."};
Setting<bool> enforceDeterminism{this, true, "enforce-determinism",
"Whether to fail if repeated builds produce different output."};
Setting<Strings> binaryCachePublicKeys{this,
{"cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="},
"binary-cache-public-keys",
"Trusted public keys for secure substitution."};
Setting<Strings> secretKeyFiles{this, {}, "secret-key-files",
"Secret keys with which to sign local builds."};
Setting<size_t> binaryCachesParallelConnections{this, 25, "http-connections",
"Number of parallel HTTP connections.",
{"binary-caches-parallel-connections"}};
Setting<bool> enableHttp2{this, true, "enable-http2",
"Whether to enable HTTP/2 support."};
Setting<unsigned int> tarballTtl{this, 60 * 60, "tarball-ttl",
"How soon to expire files fetched by builtins.fetchTarball and builtins.fetchurl."};
Setting<std::string> signedBinaryCaches{this, "*", "signed-binary-caches",
"Obsolete."};
Setting<Strings> substituters{this,
nixStore == "/nix/store" ? Strings{"https://cache.nixos.org/"} : Strings(),
"substituters",
"The URIs of substituters (such as https://cache.nixos.org/).",
{"binary-caches"}};
// FIXME: provide a way to add to option values.
Setting<Strings> extraSubstituters{this, {}, "extra-substituters",
"Additional URIs of substituters.",
{"extra-binary-caches"}};
Setting<StringSet> trustedSubstituters{this, {}, "trusted-substituters",
"Disabled substituters that may be enabled via the substituters option by untrusted users.",
{"trusted-binary-caches"}};
Setting<Strings> trustedUsers{this, {"root"}, "trusted-users",
"Which users or groups are trusted to ask the daemon to do unsafe things."};
/* ?Who we trust to use the daemon in safe ways */
Setting<Strings> allowedUsers{this, {"*"}, "allowed-users",
"Which users or groups are allowed to connect to the daemon."};
Setting<bool> printMissing{this, true, "print-missing",
"Whether to print what paths need to be built or downloaded."};
Setting<std::string> preBuildHook{this,
#if __APPLE__
nixLibexecDir + "/nix/resolve-system-dependencies",
#else
"",
#endif
"pre-build-hook",
"A program to run just before a build to set derivation-specific build settings."};
Setting<std::string> netrcFile{this, fmt("%s/%s", nixConfDir, "netrc"), "netrc-file",
"Path to the netrc file used to obtain usernames/passwords for downloads."};
/* Path to the SSL CA file used */
Path caFile;
Setting<bool> enableImportFromDerivation{this, true, "allow-import-from-derivation",
"Whether the evaluator allows importing the result of a derivation."};
CaseHackSetting useCaseHack{this, "use-case-hack",
"Whether to enable a Darwin-specific hack for dealing with file name collisions."};
Setting<unsigned long> connectTimeout{this, 0, "connect-timeout",
"Timeout for connecting to servers during downloads. 0 means use curl's builtin default."};
Setting<std::string> userAgentSuffix{this, "", "user-agent-suffix",
"String appended to the user agent in HTTP requests."};
};
+3 -3
View File
@@ -50,7 +50,6 @@ protected:
{
try {
DownloadRequest request(cacheUri + "/" + path);
request.showProgress = DownloadRequest::no;
request.head = true;
request.tries = 5;
getDownloader()->download(request);
@@ -64,7 +63,9 @@ protected:
}
}
void upsertFile(const std::string & path, const std::string & data) override
void upsertFile(const std::string & path,
const std::string & data,
const std::string & mimeType) override
{
throw UploadToHTTP("uploading to an HTTP binary cache is not supported");
}
@@ -74,7 +75,6 @@ protected:
std::function<void(std::exception_ptr exc)> failure) override
{
DownloadRequest request(cacheUri + "/" + path);
request.showProgress = DownloadRequest::no;
request.tries = 8;
getDownloader()->enqueueDownload(request,
+96 -63
View File
@@ -4,82 +4,60 @@
#include "serve-protocol.hh"
#include "store-api.hh"
#include "worker-protocol.hh"
#include "ssh.hh"
#include "derivations.hh"
namespace nix {
static std::string uriScheme = "legacy-ssh://";
static std::string uriScheme = "ssh://";
struct LegacySSHStore : public Store
{
string host;
const Setting<int> maxConnections{this, 1, "max-connections", "maximum number of concurrent SSH connections"};
const Setting<Path> sshKey{this, "", "ssh-key", "path to an SSH private key"};
const Setting<bool> compress{this, false, "compress", "whether to compress the connection"};
// Hack for getting remote build log output.
const Setting<int> logFD{this, -1, "log-fd", "file descriptor to which SSH's stderr is connected"};
struct Connection
{
Pid sshPid;
AutoCloseFD out;
AutoCloseFD in;
std::unique_ptr<SSHMaster::Connection> sshConn;
FdSink to;
FdSource from;
int remoteVersion;
};
AutoDelete tmpDir;
Path socketPath;
Pid sshMaster;
std::string host;
ref<Pool<Connection>> connections;
Path key;
SSHMaster master;
LegacySSHStore(const string & host, const Params & params,
size_t maxConnections = std::numeric_limits<size_t>::max())
LegacySSHStore(const string & host, const Params & params)
: Store(params)
, host(host)
, tmpDir(createTempDir("", "nix", true, true, 0700))
, socketPath((Path) tmpDir + "/ssh.sock")
, connections(make_ref<Pool<Connection>>(
maxConnections,
std::max(1, (int) maxConnections),
[this]() { return openConnection(); },
[](const ref<Connection> & r) { return true; }
))
, key(get(params, "ssh-key", ""))
, master(
host,
sshKey,
// Use SSH master only if using more than 1 connection.
connections->capacity() > 1,
compress,
logFD)
{
}
ref<Connection> openConnection()
{
if ((pid_t) sshMaster == -1) {
sshMaster = startProcess([&]() {
restoreSignals();
Strings args{ "ssh", "-M", "-S", socketPath, "-N", "-x", "-a", host };
if (!key.empty())
args.insert(args.end(), {"-i", key});
execvp("ssh", stringsToCharPtrs(args).data());
throw SysError("starting SSH master connection to host ‘%s’", host);
});
}
auto conn = make_ref<Connection>();
Pipe in, out;
in.create();
out.create();
conn->sshPid = startProcess([&]() {
if (dup2(in.readSide.get(), STDIN_FILENO) == -1)
throw SysError("duping over STDIN");
if (dup2(out.writeSide.get(), STDOUT_FILENO) == -1)
throw SysError("duping over STDOUT");
execlp("ssh", "ssh", "-S", socketPath.c_str(), host.c_str(), "nix-store", "--serve", "--write", nullptr);
throw SysError("executing ‘nix-store --serve’ on remote host ‘%s’", host);
});
in.readSide = -1;
out.writeSide = -1;
conn->out = std::move(out.readSide);
conn->in = std::move(in.writeSide);
conn->to = FdSink(conn->in.get());
conn->from = FdSource(conn->out.get());
int remoteVersion;
conn->sshConn = master.startCommand("nix-store --serve --write");
conn->to = FdSink(conn->sshConn->in.get());
conn->from = FdSource(conn->sshConn->out.get());
try {
conn->to << SERVE_MAGIC_1 << SERVE_PROTOCOL_VERSION;
@@ -88,8 +66,8 @@ struct LegacySSHStore : public Store
unsigned int magic = readInt(conn->from);
if (magic != SERVE_MAGIC_2)
throw Error("protocol mismatch with ‘nix-store --serve’ on ‘%s’", host);
remoteVersion = readInt(conn->from);
if (GET_PROTOCOL_MAJOR(remoteVersion) != 0x200)
conn->remoteVersion = readInt(conn->from);
if (GET_PROTOCOL_MAJOR(conn->remoteVersion) != 0x200)
throw Error("unsupported ‘nix-store --serve’ protocol version on ‘%s’", host);
} catch (EndOfFile & e) {
@@ -169,15 +147,9 @@ struct LegacySSHStore : public Store
/* FIXME: inefficient. */
ParseSink parseSink; /* null sink; just parse the NAR */
SavingSourceAdapter savedNAR(conn->from);
TeeSource savedNAR(conn->from);
parseDump(parseSink, savedNAR);
sink(savedNAR.s);
}
/* Unsupported methods. */
[[noreturn]] void unsupported()
{
throw Error("operation not supported on SSH stores");
sink(*savedNAR.data);
}
PathSet queryAllValidPaths() override { unsupported(); }
@@ -203,12 +175,36 @@ struct LegacySSHStore : public Store
const PathSet & references, bool repair) override
{ unsupported(); }
void buildPaths(const PathSet & paths, BuildMode buildMode) override
{ unsupported(); }
BuildResult buildDerivation(const Path & drvPath, const BasicDerivation & drv,
BuildMode buildMode) override
{ unsupported(); }
{
auto conn(connections->get());
conn->to
<< cmdBuildDerivation
<< drvPath
<< drv
<< settings.maxSilentTime
<< settings.buildTimeout;
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 2)
conn->to
<< settings.maxLogSize;
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 3)
conn->to
<< settings.buildRepeat
<< settings.enforceDeterminism;
conn->to.flush();
BuildResult status;
status.status = (BuildResult::Status) readInt(conn->from);
conn->from >> status.errorMsg;
if (GET_PROTOCOL_MINOR(conn->remoteVersion) >= 3)
conn->from >> status.timesBuilt >> status.isNonDeterministic >> status.startTime >> status.stopTime;
return status;
}
void ensurePath(const Path & path) override
{ unsupported(); }
@@ -231,9 +227,46 @@ struct LegacySSHStore : public Store
void addSignatures(const Path & storePath, const StringSet & sigs) override
{ unsupported(); }
bool isTrusted() override
{ return true; }
void computeFSClosure(const PathSet & paths,
PathSet & out, bool flipDirection = false,
bool includeOutputs = false, bool includeDerivers = false) override
{
if (flipDirection || includeDerivers) {
Store::computeFSClosure(paths, out, flipDirection, includeOutputs, includeDerivers);
return;
}
auto conn(connections->get());
conn->to
<< cmdQueryClosure
<< includeOutputs
<< paths;
conn->to.flush();
auto res = readStorePaths<PathSet>(*this, conn->from);
out.insert(res.begin(), res.end());
}
PathSet queryValidPaths(const PathSet & paths, bool maybeSubstitute = false) override
{
auto conn(connections->get());
conn->to
<< cmdQueryValidPaths
<< false // lock
<< maybeSubstitute
<< paths;
conn->to.flush();
return readStorePaths<PathSet>(*this, conn->from);
}
void connect() override
{
auto conn(connections->get());
}
};
static RegisterStoreImplementation regStore([](
+6 -2
View File
@@ -30,7 +30,9 @@ protected:
bool fileExists(const std::string & path) override;
void upsertFile(const std::string & path, const std::string & data) override;
void upsertFile(const std::string & path,
const std::string & data,
const std::string & mimeType) override;
void getFile(const std::string & path,
std::function<void(std::shared_ptr<std::string>)> success,
@@ -83,7 +85,9 @@ bool LocalBinaryCacheStore::fileExists(const std::string & path)
return pathExists(binaryCacheDir + "/" + path);
}
void LocalBinaryCacheStore::upsertFile(const std::string & path, const std::string & data)
void LocalBinaryCacheStore::upsertFile(const std::string & path,
const std::string & data,
const std::string & mimeType)
{
atomicWrite(binaryCacheDir + "/" + path, data);
}
+50 -6
View File
@@ -2,14 +2,13 @@
#include "fs-accessor.hh"
#include "store-api.hh"
#include "globals.hh"
#include "compression.hh"
#include "derivations.hh"
namespace nix {
LocalFSStore::LocalFSStore(const Params & params)
: Store(params)
, rootDir(get(params, "root"))
, stateDir(canonPath(get(params, "state", rootDir != "" ? rootDir + "/nix/var/nix" : settings.nixStateDir)))
, logDir(canonPath(get(params, "log", rootDir != "" ? rootDir + "/nix/var/log/nix" : settings.nixLogDir)))
{
}
@@ -32,7 +31,7 @@ struct LocalStoreAccessor : public FSAccessor
auto realPath = toRealPath(path);
struct stat st;
if (lstat(path.c_str(), &st)) {
if (lstat(realPath.c_str(), &st)) {
if (errno == ENOENT || errno == ENOTDIR) return {Type::tMissing, 0, false};
throw SysError(format("getting status of ‘%1%’") % path);
}
@@ -52,7 +51,7 @@ struct LocalStoreAccessor : public FSAccessor
{
auto realPath = toRealPath(path);
auto entries = nix::readDirectory(path);
auto entries = nix::readDirectory(realPath);
StringSet res;
for (auto & entry : entries)
@@ -74,7 +73,8 @@ struct LocalStoreAccessor : public FSAccessor
ref<FSAccessor> LocalFSStore::getFSAccessor()
{
return make_ref<LocalStoreAccessor>(ref<LocalFSStore>(std::dynamic_pointer_cast<LocalFSStore>(shared_from_this())));
return make_ref<LocalStoreAccessor>(ref<LocalFSStore>(
std::dynamic_pointer_cast<LocalFSStore>(shared_from_this())));
}
void LocalFSStore::narFromPath(const Path & path, Sink & sink)
@@ -84,4 +84,48 @@ void LocalFSStore::narFromPath(const Path & path, Sink & sink)
dumpPath(getRealStoreDir() + std::string(path, storeDir.size()), sink);
}
const string LocalFSStore::drvsLogDir = "drvs";
std::shared_ptr<std::string> LocalFSStore::getBuildLog(const Path & path_)
{
auto path(path_);
assertStorePath(path);
if (!isDerivation(path)) {
try {
path = queryPathInfo(path)->deriver;
} catch (InvalidPath &) {
return nullptr;
}
if (path == "") return nullptr;
}
string baseName = baseNameOf(path);
for (int j = 0; j < 2; j++) {
Path logPath =
j == 0
? fmt("%s/%s/%s/%s", logDir, drvsLogDir, string(baseName, 0, 2), string(baseName, 2))
: fmt("%s/%s/%s", logDir, drvsLogDir, baseName);
Path logBz2Path = logPath + ".bz2";
if (pathExists(logPath))
return std::make_shared<std::string>(readFile(logPath));
else if (pathExists(logBz2Path)) {
try {
return decompress("bzip2", readFile(logBz2Path));
} catch (Error &) { }
}
}
return nullptr;
}
}
+51 -12
View File
@@ -27,6 +27,7 @@
#include <sys/statvfs.h>
#include <sys/mount.h>
#include <sys/ioctl.h>
#include <sys/xattr.h>
#endif
#include <sqlite3.h>
@@ -38,13 +39,14 @@ namespace nix {
LocalStore::LocalStore(const Params & params)
: Store(params)
, LocalFSStore(params)
, realStoreDir(get(params, "real", rootDir != "" ? rootDir + "/nix/store" : storeDir))
, realStoreDir_{this, false, rootDir != "" ? rootDir + "/nix/store" : storeDir, "real",
"physical path to the Nix store"}
, realStoreDir(realStoreDir_)
, dbDir(stateDir + "/db")
, linksDir(realStoreDir + "/.links")
, reservedPath(dbDir + "/reserved")
, schemaPath(dbDir + "/schema")
, trashDir(realStoreDir + "/trash")
, requireSigs(trim(settings.get("signed-binary-caches", std::string(""))) != "") // FIXME: rename option
, publicKeys(getDefaultPublicKeys())
{
auto state(_state.lock());
@@ -74,7 +76,7 @@ LocalStore::LocalStore(const Params & params)
mode_t perm = 01775;
struct group * gr = getgrnam(settings.buildUsersGroup.c_str());
struct group * gr = getgrnam(settings.buildUsersGroup.get().c_str());
if (!gr)
printError(format("warning: the group ‘%1%’ specified in ‘build-users-group’ does not exist")
% settings.buildUsersGroup);
@@ -318,7 +320,7 @@ void LocalStore::openDB(State & state, bool create)
/* Initialise the database schema, if necessary. */
if (create) {
const char * schema =
#include "schema.sql.hh"
#include "schema.sql.gen.hh"
;
db.exec(schema);
}
@@ -398,6 +400,16 @@ static void canonicalisePathMetaData_(const Path & path, uid_t fromUid, InodesSe
{
checkInterrupt();
#if __APPLE__
/* Remove flags, in particular UF_IMMUTABLE which would prevent
the file from being garbage-collected. FIXME: Use
setattrlist() to remove other attributes as well. */
if (lchflags(path.c_str(), 0)) {
if (errno != ENOTSUP)
throw SysError(format("clearing flags of path ‘%1%’") % path);
}
#endif
struct stat st;
if (lstat(path.c_str(), &st))
throw SysError(format("getting attributes of path ‘%1%’") % path);
@@ -406,6 +418,29 @@ static void canonicalisePathMetaData_(const Path & path, uid_t fromUid, InodesSe
if (!(S_ISREG(st.st_mode) || S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode)))
throw Error(format("file ‘%1%’ has an unsupported type") % path);
#if __linux__
/* Remove extended attributes / ACLs. */
ssize_t eaSize = llistxattr(path.c_str(), nullptr, 0);
if (eaSize < 0) {
if (errno != ENOTSUP)
throw SysError("querying extended attributes of ‘%s’", path);
} else if (eaSize > 0) {
std::vector<char> eaBuf(eaSize);
if ((eaSize = llistxattr(path.c_str(), eaBuf.data(), eaBuf.size())) < 0)
throw SysError("querying extended attributes of ‘%s’", path);
for (auto & eaName: tokenizeString<Strings>(std::string(eaBuf.data(), eaSize), std::string("\000", 1))) {
/* Ignore SELinux security labels since these cannot be
removed even by root. */
if (eaName == "security.selinux") continue;
if (lremovexattr(path.c_str(), eaName.c_str()) == -1)
throw SysError("removing extended attribute ‘%s’ from ‘%s’", eaName, path);
}
}
#endif
/* Fail if the file is not owned by the build user. This prevents
us from messing up the ownership/permissions of files
hard-linked into the output (e.g. "ln /etc/shadow $out/foo").
@@ -669,7 +704,7 @@ bool LocalStore::isValidPathUncached(const Path & path)
}
PathSet LocalStore::queryValidPaths(const PathSet & paths)
PathSet LocalStore::queryValidPaths(const PathSet & paths, bool maybeSubstitute)
{
PathSet res;
for (auto & i : paths)
@@ -914,13 +949,19 @@ void LocalStore::invalidatePath(State & state, const Path & path)
void LocalStore::addToStore(const ValidPathInfo & info, const ref<std::string> & nar,
bool repair, bool dontCheckSigs, std::shared_ptr<FSAccessor> accessor)
{
assert(info.narHash);
Hash h = hashString(htSHA256, *nar);
if (h != info.narHash)
throw Error(format("hash mismatch importing path ‘%s’; expected hash ‘%s’, got ‘%s’") %
info.path % info.narHash.to_string() % h.to_string());
throw Error("hash mismatch importing path ‘%s’; expected hash ‘%s’, got ‘%s’",
info.path, info.narHash.to_string(), h.to_string());
if (nar->size() != info.narSize)
throw Error("size mismatch importing path ‘%s’; expected %s, got %s",
info.path, info.narSize, nar->size());
if (requireSigs && !dontCheckSigs && !info.checkSignatures(*this, publicKeys))
throw Error(format("cannot import path ‘%s’ because it lacks a valid signature") % info.path);
throw Error("cannot add path ‘%s’ because it lacks a valid signature", info.path);
addTempRoot(info.path);
@@ -1003,7 +1044,6 @@ Path LocalStore::addToStoreFromDump(const string & dump, const string & name,
info.path = dstPath;
info.narHash = hash.first;
info.narSize = hash.second;
info.ultimate = true;
info.ca = makeFixedOutputCA(recursive, h);
registerValidPath(info);
}
@@ -1066,7 +1106,6 @@ Path LocalStore::addTextToStore(const string & name, const string & s,
info.narHash = narHash;
info.narSize = sink.s->size();
info.references = references;
info.ultimate = true;
info.ca = "text:" + hash.to_string();
registerValidPath(info);
}
@@ -1332,9 +1371,9 @@ void LocalStore::signPathInfo(ValidPathInfo & info)
{
// FIXME: keep secret keys in memory.
auto secretKeyFiles = settings.get("secret-key-files", Strings());
auto secretKeyFiles = settings.secretKeyFiles;
for (auto & secretKeyFile : secretKeyFiles) {
for (auto & secretKeyFile : secretKeyFiles.get()) {
SecretKey secretKey(readFile(secretKeyFile));
info.sign(secretKey);
}
+9 -13
View File
@@ -21,22 +21,14 @@ namespace nix {
const int nixSchemaVersion = 10;
extern string drvsLogDir;
struct Derivation;
struct OptimiseStats
{
unsigned long filesLinked;
unsigned long long bytesFreed;
unsigned long long blocksFreed;
OptimiseStats()
{
filesLinked = 0;
bytesFreed = blocksFreed = 0;
}
unsigned long filesLinked = 0;
unsigned long long bytesFreed = 0;
unsigned long long blocksFreed = 0;
};
@@ -75,6 +67,8 @@ private:
public:
PathSetting realStoreDir_;
const Path realStoreDir;
const Path dbDir;
const Path linksDir;
@@ -84,7 +78,9 @@ public:
private:
bool requireSigs;
Setting<bool> requireSigs{(Store*) this,
settings.signedBinaryCaches != "", // FIXME
"require-sigs", "whether store paths should have a trusted signature on import"};
PublicKeys publicKeys;
@@ -102,7 +98,7 @@ public:
bool isValidPathUncached(const Path & path) override;
PathSet queryValidPaths(const PathSet & paths) override;
PathSet queryValidPaths(const PathSet & paths, bool maybeSubstitute = false) override;
PathSet queryAllValidPaths() override;
+17 -6
View File
@@ -18,6 +18,10 @@ ifeq ($(OS), SunOS)
libstore_LDFLAGS += -lsocket
endif
ifeq ($(OS), Linux)
libstore_LDFLAGS += -lseccomp
endif
libstore_CXXFLAGS = \
-DNIX_PREFIX=\"$(prefix)\" \
-DNIX_STORE_DIR=\"$(storedir)\" \
@@ -27,14 +31,21 @@ libstore_CXXFLAGS = \
-DNIX_CONF_DIR=\"$(sysconfdir)/nix\" \
-DNIX_LIBEXEC_DIR=\"$(libexecdir)\" \
-DNIX_BIN_DIR=\"$(bindir)\" \
-DBASH_PATH="\"$(bash)\""
-DSANDBOX_SHELL="\"$(sandbox_shell)\"" \
-DLSOF=\"$(lsof)\"
$(d)/local-store.cc: $(d)/schema.sql.hh
$(d)/local-store.cc: $(d)/schema.sql.gen.hh
%.sql.hh: %.sql
$(trace-gen) sed -e 's/"/\\"/g' -e 's/\(.*\)/"\1\\n"/' < $< > $@ || (rm $@ && exit 1)
sandbox-headers = $(d)/sandbox-defaults.sb.gen.hh $(d)/sandbox-network.sb.gen.hh $(d)/sandbox-minimal.sb.gen.hh
clean-files += $(d)/schema.sql.hh
$(d)/build.cc: $(sandbox-headers)
%.gen.hh: %
@echo 'R"foo(' >> $@.tmp
$(trace-gen) cat $< >> $@.tmp
@echo ')foo"' >> $@.tmp
@mv $@.tmp $@
clean-files += $(d)/schema.sql.gen.hh $(sandbox-headers)
$(eval $(call install-file-in, $(d)/nix-store.pc, $(prefix)/lib/pkgconfig, 0644))
$(eval $(call install-file-in, $(d)/sandbox-defaults.sb, $(datadir)/nix, 0644))
+91
View File
@@ -0,0 +1,91 @@
#include "machines.hh"
#include "util.hh"
#include "globals.hh"
#include <algorithm>
namespace nix {
Machine::Machine(decltype(storeUri) storeUri,
decltype(systemTypes) systemTypes,
decltype(sshKey) sshKey,
decltype(maxJobs) maxJobs,
decltype(speedFactor) speedFactor,
decltype(supportedFeatures) supportedFeatures,
decltype(mandatoryFeatures) mandatoryFeatures,
decltype(sshPublicHostKey) sshPublicHostKey) :
storeUri(
// Backwards compatibility: if the URI is a hostname,
// prepend ssh://.
storeUri.find("://") != std::string::npos || hasPrefix(storeUri, "local") || hasPrefix(storeUri, "remote") || hasPrefix(storeUri, "auto")
? storeUri
: "ssh://" + storeUri),
systemTypes(systemTypes),
sshKey(sshKey),
maxJobs(maxJobs),
speedFactor(std::max(1U, speedFactor)),
supportedFeatures(supportedFeatures),
mandatoryFeatures(mandatoryFeatures),
sshPublicHostKey(sshPublicHostKey)
{}
bool Machine::allSupported(const std::set<string> & features) const {
return std::all_of(features.begin(), features.end(),
[&](const string & feature) {
return supportedFeatures.count(feature) ||
mandatoryFeatures.count(feature);
});
}
bool Machine::mandatoryMet(const std::set<string> & features) const {
return std::all_of(mandatoryFeatures.begin(), mandatoryFeatures.end(),
[&](const string & feature) {
return features.count(feature);
});
}
void parseMachines(const std::string & s, Machines & machines)
{
for (auto line : tokenizeString<std::vector<string>>(s, "\n;")) {
chomp(line);
line.erase(std::find(line.begin(), line.end(), '#'), line.end());
if (line.empty()) continue;
auto tokens = tokenizeString<std::vector<string>>(line);
auto sz = tokens.size();
if (sz < 1)
throw FormatError("bad machine specification ‘%s’", line);
auto isSet = [&](size_t n) {
return tokens.size() > n && tokens[n] != "" && tokens[n] != "-";
};
machines.emplace_back(tokens[0],
isSet(1) ? tokenizeString<std::vector<string>>(tokens[1], ",") : std::vector<string>{settings.thisSystem},
isSet(2) ? tokens[2] : "",
isSet(3) ? std::stoull(tokens[3]) : 1LL,
isSet(4) ? std::stoull(tokens[4]) : 1LL,
isSet(5) ? tokenizeString<std::set<string>>(tokens[5], ",") : std::set<string>{},
isSet(6) ? tokenizeString<std::set<string>>(tokens[6], ",") : std::set<string>{},
isSet(7) ? tokens[7] : "");
}
}
Machines getMachines()
{
Machines machines;
for (auto & file : settings.builderFiles.get()) {
try {
parseMachines(readFile(file), machines);
} catch (const SysError & e) {
if (e.errNo != ENOENT)
throw;
}
}
parseMachines(settings.builders, machines);
return machines;
}
}
+39
View File
@@ -0,0 +1,39 @@
#pragma once
#include "types.hh"
namespace nix {
struct Machine {
const string storeUri;
const std::vector<string> systemTypes;
const string sshKey;
const unsigned int maxJobs;
const unsigned int speedFactor;
const std::set<string> supportedFeatures;
const std::set<string> mandatoryFeatures;
const std::string sshPublicHostKey;
bool enabled = true;
bool allSupported(const std::set<string> & features) const;
bool mandatoryMet(const std::set<string> & features) const;
Machine(decltype(storeUri) storeUri,
decltype(systemTypes) systemTypes,
decltype(sshKey) sshKey,
decltype(maxJobs) maxJobs,
decltype(speedFactor) speedFactor,
decltype(supportedFeatures) supportedFeatures,
decltype(mandatoryFeatures) mandatoryFeatures,
decltype(sshPublicHostKey) sshPublicHostKey);
};
typedef std::vector<Machine> Machines;
void parseMachines(const std::string & s, Machines & machines);
Machines getMachines();
}
+75 -33
View File
@@ -2,6 +2,8 @@
#include "archive.hh"
#include <map>
#include <stack>
#include <algorithm>
namespace nix {
@@ -16,16 +18,16 @@ struct NarMember
size_t start, size;
std::string target;
/* If this is a directory, all the children of the directory. */
std::map<std::string, NarMember> children;
};
struct NarIndexer : ParseSink, StringSource
{
// FIXME: should store this as a tree. Now we're vulnerable to
// O(nm) memory consumption (e.g. for x_0/.../x_n/{y_0..y_m}).
typedef std::map<Path, NarMember> Members;
Members members;
NarMember root;
std::stack<NarMember*> parents;
Path currentPath;
std::string currentStart;
bool isExec = false;
@@ -33,28 +35,45 @@ struct NarIndexer : ParseSink, StringSource
{
}
void createMember(const Path & path, NarMember member) {
size_t level = std::count(path.begin(), path.end(), '/');
while(parents.size() > level) {
parents.pop();
}
if(parents.empty()) {
root = std::move(member);
parents.push(&root);
} else {
if(parents.top()->type != FSAccessor::Type::tDirectory) {
throw Error(format("NAR file missing parent directory of path ‘%1%’") % path);
}
auto result = parents.top()->children.emplace(baseNameOf(path), std::move(member));
parents.push(&result.first->second);
}
}
void createDirectory(const Path & path) override
{
members.emplace(path,
NarMember{FSAccessor::Type::tDirectory, false, 0, 0});
createMember(path, {FSAccessor::Type::tDirectory, false, 0, 0 });
}
void createRegularFile(const Path & path) override
{
currentPath = path;
createMember(path, {FSAccessor::Type::tRegular, false, 0, 0 });
}
void isExecutable() override
{
isExec = true;
parents.top()->isExecutable = true;
}
void preallocateContents(unsigned long long size) override
{
currentStart = string(s, pos, 16);
assert(size <= std::numeric_limits<size_t>::max());
members.emplace(currentPath,
NarMember{FSAccessor::Type::tRegular, isExec, pos, (size_t) size});
parents.top()->size = (size_t)size;
parents.top()->start = pos;
}
void receiveContents(unsigned char * data, unsigned int len) override
@@ -68,16 +87,42 @@ struct NarIndexer : ParseSink, StringSource
void createSymlink(const Path & path, const string & target) override
{
members.emplace(path,
createMember(path,
NarMember{FSAccessor::Type::tSymlink, false, 0, 0, target});
}
Members::iterator find(const Path & path)
NarMember* find(const Path & path)
{
auto i = members.find(path);
if (i == members.end())
Path canon = path == "" ? "" : canonPath(path);
NarMember* current = &root;
auto end = path.end();
for(auto it = path.begin(); it != end; ) {
// because it != end, the remaining component is non-empty so we need
// a directory
if(current->type != FSAccessor::Type::tDirectory) return nullptr;
// skip slash (canonPath above ensures that this is always a slash)
assert(*it == '/');
it += 1;
// lookup current component
auto next = std::find(it, end, '/');
auto child = current->children.find(std::string(it, next));
if(child == current->children.end()) return nullptr;
current = &child->second;
it = next;
}
return current;
}
NarMember& at(const Path & path) {
auto result = find(path);
if(result == nullptr) {
throw Error(format("NAR file does not contain path ‘%1%’") % path);
return i;
}
return *result;
}
};
@@ -93,44 +138,41 @@ struct NarAccessor : public FSAccessor
Stat stat(const Path & path) override
{
auto i = indexer.members.find(path);
if (i == indexer.members.end())
auto i = indexer.find(path);
if (i == nullptr)
return {FSAccessor::Type::tMissing, 0, false};
return {i->second.type, i->second.size, i->second.isExecutable};
return {i->type, i->size, i->isExecutable};
}
StringSet readDirectory(const Path & path) override
{
auto i = indexer.find(path);
auto i = indexer.at(path);
if (i->second.type != FSAccessor::Type::tDirectory)
if (i.type != FSAccessor::Type::tDirectory)
throw Error(format("path ‘%1%’ inside NAR file is not a directory") % path);
++i;
StringSet res;
while (i != indexer.members.end() && isInDir(i->first, path)) {
// FIXME: really bad performance.
if (i->first.find('/', path.size() + 1) == std::string::npos)
res.insert(std::string(i->first, path.size() + 1));
++i;
for(auto&& child : i.children) {
res.insert(child.first);
}
return res;
}
std::string readFile(const Path & path) override
{
auto i = indexer.find(path);
if (i->second.type != FSAccessor::Type::tRegular)
auto i = indexer.at(path);
if (i.type != FSAccessor::Type::tRegular)
throw Error(format("path ‘%1%’ inside NAR file is not a regular file") % path);
return std::string(*nar, i->second.start, i->second.size);
return std::string(*nar, i.start, i.size);
}
std::string readLink(const Path & path) override
{
auto i = indexer.find(path);
if (i->second.type != FSAccessor::Type::tSymlink)
auto i = indexer.at(path);
if (i.type != FSAccessor::Type::tSymlink)
throw Error(format("path ‘%1%’ inside NAR file is not a symlink") % path);
return i->second.target;
return i.target;
}
};
+98 -87
View File
@@ -106,25 +106,27 @@ public:
"select * from NARs where cache = ? and hashPart = ? and ((present = 0 and timestamp > ?) or (present = 1 and timestamp > ?))");
/* Periodically purge expired entries from the database. */
auto now = time(0);
retrySQLite<void>([&]() {
auto now = time(0);
SQLiteStmt queryLastPurge(state->db, "select value from LastPurge");
auto queryLastPurge_(queryLastPurge.use());
SQLiteStmt queryLastPurge(state->db, "select value from LastPurge");
auto queryLastPurge_(queryLastPurge.use());
if (!queryLastPurge_.next() || queryLastPurge_.getInt(0) < now - purgeInterval) {
SQLiteStmt(state->db,
"delete from NARs where ((present = 0 and timestamp < ?) or (present = 1 and timestamp < ?))")
.use()
(now - ttlNegative)
(now - ttlPositive)
.exec();
if (!queryLastPurge_.next() || queryLastPurge_.getInt(0) < now - purgeInterval) {
SQLiteStmt(state->db,
"delete from NARs where ((present = 0 and timestamp < ?) or (present = 1 and timestamp < ?))")
.use()
(now - ttlNegative)
(now - ttlPositive)
.exec();
debug("deleted %d entries from the NAR info disk cache", sqlite3_changes(state->db));
debug("deleted %d entries from the NAR info disk cache", sqlite3_changes(state->db));
SQLiteStmt(state->db,
"insert or replace into LastPurge(dummy, value) values ('', ?)")
.use()(now).exec();
}
SQLiteStmt(state->db,
"insert or replace into LastPurge(dummy, value) values ('', ?)")
.use()(now).exec();
}
});
}
Cache & getCache(State & state, const std::string & uri)
@@ -136,114 +138,123 @@ public:
void createCache(const std::string & uri, const Path & storeDir, bool wantMassQuery, int priority) override
{
auto state(_state.lock());
retrySQLite<void>([&]() {
auto state(_state.lock());
// FIXME: race
// FIXME: race
state->insertCache.use()(uri)(time(0))(storeDir)(wantMassQuery)(priority).exec();
assert(sqlite3_changes(state->db) == 1);
state->caches[uri] = Cache{(int) sqlite3_last_insert_rowid(state->db), storeDir, wantMassQuery, priority};
state->insertCache.use()(uri)(time(0))(storeDir)(wantMassQuery)(priority).exec();
assert(sqlite3_changes(state->db) == 1);
state->caches[uri] = Cache{(int) sqlite3_last_insert_rowid(state->db), storeDir, wantMassQuery, priority};
});
}
bool cacheExists(const std::string & uri,
bool & wantMassQuery, int & priority) override
{
auto state(_state.lock());
return retrySQLite<bool>([&]() {
auto state(_state.lock());
auto i = state->caches.find(uri);
if (i == state->caches.end()) {
auto queryCache(state->queryCache.use()(uri));
if (!queryCache.next()) return false;
state->caches.emplace(uri,
Cache{(int) queryCache.getInt(0), queryCache.getStr(1), queryCache.getInt(2) != 0, (int) queryCache.getInt(3)});
}
auto i = state->caches.find(uri);
if (i == state->caches.end()) {
auto queryCache(state->queryCache.use()(uri));
if (!queryCache.next()) return false;
state->caches.emplace(uri,
Cache{(int) queryCache.getInt(0), queryCache.getStr(1), queryCache.getInt(2) != 0, (int) queryCache.getInt(3)});
}
auto & cache(getCache(*state, uri));
auto & cache(getCache(*state, uri));
wantMassQuery = cache.wantMassQuery;
priority = cache.priority;
wantMassQuery = cache.wantMassQuery;
priority = cache.priority;
return true;
return true;
});
}
std::pair<Outcome, std::shared_ptr<NarInfo>> lookupNarInfo(
const std::string & uri, const std::string & hashPart) override
{
auto state(_state.lock());
return retrySQLite<std::pair<Outcome, std::shared_ptr<NarInfo>>>(
[&]() -> std::pair<Outcome, std::shared_ptr<NarInfo>> {
auto state(_state.lock());
auto & cache(getCache(*state, uri));
auto & cache(getCache(*state, uri));
auto now = time(0);
auto now = time(0);
auto queryNAR(state->queryNAR.use()
(cache.id)
(hashPart)
(now - ttlNegative)
(now - ttlPositive));
auto queryNAR(state->queryNAR.use()
(cache.id)
(hashPart)
(now - ttlNegative)
(now - ttlPositive));
if (!queryNAR.next())
return {oUnknown, 0};
if (!queryNAR.next())
return {oUnknown, 0};
if (!queryNAR.getInt(13))
return {oInvalid, 0};
if (!queryNAR.getInt(13))
return {oInvalid, 0};
auto narInfo = make_ref<NarInfo>();
auto narInfo = make_ref<NarInfo>();
auto namePart = queryNAR.getStr(2);
narInfo->path = cache.storeDir + "/" +
hashPart + (namePart.empty() ? "" : "-" + namePart);
narInfo->url = queryNAR.getStr(3);
narInfo->compression = queryNAR.getStr(4);
if (!queryNAR.isNull(5))
narInfo->fileHash = parseHash(queryNAR.getStr(5));
narInfo->fileSize = queryNAR.getInt(6);
narInfo->narHash = parseHash(queryNAR.getStr(7));
narInfo->narSize = queryNAR.getInt(8);
for (auto & r : tokenizeString<Strings>(queryNAR.getStr(9), " "))
narInfo->references.insert(cache.storeDir + "/" + r);
if (!queryNAR.isNull(10))
narInfo->deriver = cache.storeDir + "/" + queryNAR.getStr(10);
for (auto & sig : tokenizeString<Strings>(queryNAR.getStr(11), " "))
narInfo->sigs.insert(sig);
auto namePart = queryNAR.getStr(2);
narInfo->path = cache.storeDir + "/" +
hashPart + (namePart.empty() ? "" : "-" + namePart);
narInfo->url = queryNAR.getStr(3);
narInfo->compression = queryNAR.getStr(4);
if (!queryNAR.isNull(5))
narInfo->fileHash = parseHash(queryNAR.getStr(5));
narInfo->fileSize = queryNAR.getInt(6);
narInfo->narHash = parseHash(queryNAR.getStr(7));
narInfo->narSize = queryNAR.getInt(8);
for (auto & r : tokenizeString<Strings>(queryNAR.getStr(9), " "))
narInfo->references.insert(cache.storeDir + "/" + r);
if (!queryNAR.isNull(10))
narInfo->deriver = cache.storeDir + "/" + queryNAR.getStr(10);
for (auto & sig : tokenizeString<Strings>(queryNAR.getStr(11), " "))
narInfo->sigs.insert(sig);
return {oValid, narInfo};
return {oValid, narInfo};
});
}
void upsertNarInfo(
const std::string & uri, const std::string & hashPart,
std::shared_ptr<ValidPathInfo> info) override
{
auto state(_state.lock());
retrySQLite<void>([&]() {
auto state(_state.lock());
auto & cache(getCache(*state, uri));
auto & cache(getCache(*state, uri));
if (info) {
if (info) {
auto narInfo = std::dynamic_pointer_cast<NarInfo>(info);
auto narInfo = std::dynamic_pointer_cast<NarInfo>(info);
assert(hashPart == storePathToHash(info->path));
assert(hashPart == storePathToHash(info->path));
state->insertNAR.use()
(cache.id)
(hashPart)
(storePathToName(info->path))
(narInfo ? narInfo->url : "", narInfo != 0)
(narInfo ? narInfo->compression : "", narInfo != 0)
(narInfo && narInfo->fileHash ? narInfo->fileHash.to_string() : "", narInfo && narInfo->fileHash)
(narInfo ? narInfo->fileSize : 0, narInfo != 0 && narInfo->fileSize)
(info->narHash.to_string())
(info->narSize)
(concatStringsSep(" ", info->shortRefs()))
(info->deriver != "" ? baseNameOf(info->deriver) : "", info->deriver != "")
(concatStringsSep(" ", info->sigs))
(time(0)).exec();
state->insertNAR.use()
(cache.id)
(hashPart)
(storePathToName(info->path))
(narInfo ? narInfo->url : "", narInfo != 0)
(narInfo ? narInfo->compression : "", narInfo != 0)
(narInfo && narInfo->fileHash ? narInfo->fileHash.to_string() : "", narInfo && narInfo->fileHash)
(narInfo ? narInfo->fileSize : 0, narInfo != 0 && narInfo->fileSize)
(info->narHash.to_string())
(info->narSize)
(concatStringsSep(" ", info->shortRefs()))
(info->deriver != "" ? baseNameOf(info->deriver) : "", info->deriver != "")
(concatStringsSep(" ", info->sigs))
(time(0)).exec();
} else {
state->insertMissingNAR.use()
(cache.id)
(hashPart)
(time(0)).exec();
}
} else {
state->insertMissingNAR.use()
(cache.id)
(hashPart)
(time(0)).exec();
}
});
}
};
+5 -3
View File
@@ -59,9 +59,11 @@ NarInfo::NarInfo(const Store & store, const std::string & s, const std::string &
}
}
else if (name == "Deriver") {
auto p = store.storeDir + "/" + value;
if (!store.isStorePath(p)) corrupt();
deriver = p;
if (value != "unknown-deriver") {
auto p = store.storeDir + "/" + value;
if (!store.isStorePath(p)) corrupt();
deriver = p;
}
}
else if (name == "System")
system = value;
+2 -3
View File
@@ -220,8 +220,7 @@ void LocalStore::optimisePath_(OptimiseStats & stats, const Path & path, InodeHa
rather than on the original link. (Probably it
temporarily increases the st_nlink field before
decreasing it again.) */
if (st.st_size)
printInfo(format("‘%1%’ has maximum number of links") % linkPath);
debug("‘%s’ has reached maximum number of links", linkPath);
return;
}
throw SysError(format("cannot rename ‘%1%’ to ‘%2%’") % tempLink % path);
@@ -241,7 +240,7 @@ void LocalStore::optimiseStore(OptimiseStats & stats)
for (auto & i : paths) {
addTempRoot(i);
if (!isValidPath(i)) continue; /* path was GC'ed, probably */
Activity act(*logger, lvlChatty, format("hashing files in ‘%1%’") % i);
//Activity act(*logger, lvlChatty, format("hashing files in ‘%1%’") % i);
optimisePath_(stats, realStoreDir + "/" + baseNameOf(i), inodeHash);
}
}
+69 -26
View File
@@ -40,21 +40,34 @@ template PathSet readStorePaths(Store & store, Source & from);
template Paths readStorePaths(Store & store, Source & from);
/* TODO: Separate these store impls into different files, give them better names */
RemoteStore::RemoteStore(const Params & params, size_t maxConnections)
RemoteStore::RemoteStore(const Params & params)
: Store(params)
, connections(make_ref<Pool<Connection>>(
maxConnections,
[this]() { return openConnection(); },
std::max(1, (int) maxConnections),
[this]() { return openConnectionWrapper(); },
[](const ref<Connection> & r) { return r->to.good() && r->from.good(); }
))
{
}
UDSRemoteStore::UDSRemoteStore(const Params & params, size_t maxConnections)
ref<RemoteStore::Connection> RemoteStore::openConnectionWrapper()
{
if (failed)
throw Error("opening a connection to remote store ‘%s’ previously failed", getUri());
try {
return openConnection();
} catch (...) {
failed = true;
throw;
}
}
UDSRemoteStore::UDSRemoteStore(const Params & params)
: Store(params)
, LocalFSStore(params)
, RemoteStore(params, maxConnections)
, RemoteStore(params)
{
}
@@ -87,7 +100,7 @@ ref<RemoteStore::Connection> UDSRemoteStore::openConnection()
throw Error(format("socket path ‘%1%’ is too long") % socketPath);
strcpy(addr.sun_path, socketPath.c_str());
if (connect(conn->fd.get(), (struct sockaddr *) &addr, sizeof(addr)) == -1)
if (::connect(conn->fd.get(), (struct sockaddr *) &addr, sizeof(addr)) == -1)
throw SysError(format("cannot connect to daemon at ‘%1%’") % socketPath);
conn->from.fd = conn->fd.get();
@@ -108,7 +121,7 @@ void RemoteStore::initConnection(Connection & conn)
unsigned int magic = readInt(conn.from);
if (magic != WORKER_MAGIC_2) throw Error("protocol mismatch");
conn.daemonVersion = readInt(conn.from);
conn.from >> conn.daemonVersion;
if (GET_PROTOCOL_MAJOR(conn.daemonVersion) != GET_PROTOCOL_MAJOR(PROTOCOL_VERSION))
throw Error("Nix daemon protocol version not supported");
if (GET_PROTOCOL_MINOR(conn.daemonVersion) < 10)
@@ -129,7 +142,7 @@ void RemoteStore::initConnection(Connection & conn)
conn.processStderr();
}
catch (Error & e) {
throw Error(format("cannot start daemon worker: %1%") % e.msg());
throw Error("cannot open connection to remote store ‘%s’: %s", getUri(), e.what());
}
setOptions(conn);
@@ -153,9 +166,7 @@ void RemoteStore::setOptions(Connection & conn)
<< settings.useSubstitutes;
if (GET_PROTOCOL_MINOR(conn.daemonVersion) >= 12) {
Settings::SettingsMap overrides = settings.getOverrides();
if (overrides["ssh-auth-sock"] == "")
overrides["ssh-auth-sock"] = getEnv("SSH_AUTH_SOCK");
auto overrides = settings.getSettings(true);
conn.to << overrides.size();
for (auto & i : overrides)
conn.to << i.first << i.second;
@@ -170,12 +181,11 @@ bool RemoteStore::isValidPathUncached(const Path & path)
auto conn(connections->get());
conn->to << wopIsValidPath << path;
conn->processStderr();
unsigned int reply = readInt(conn->from);
return reply != 0;
return readInt(conn->from);
}
PathSet RemoteStore::queryValidPaths(const PathSet & paths)
PathSet RemoteStore::queryValidPaths(const PathSet & paths, bool maybeSubstitute)
{
auto conn(connections->get());
if (GET_PROTOCOL_MINOR(conn->daemonVersion) < 12) {
@@ -246,8 +256,8 @@ void RemoteStore::querySubstitutablePathInfos(const PathSet & paths,
conn->to << wopQuerySubstitutablePathInfos << paths;
conn->processStderr();
unsigned int count = readInt(conn->from);
for (unsigned int n = 0; n < count; n++) {
size_t count = readNum<size_t>(conn->from);
for (size_t n = 0; n < count; n++) {
Path path = readStorePath(*this, conn->from);
SubstitutablePathInfo & info(infos[path]);
info.deriver = readString(conn->from);
@@ -277,7 +287,7 @@ void RemoteStore::queryPathInfoUncached(const Path & path,
throw;
}
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 17) {
bool valid = readInt(conn->from) != 0;
bool valid; conn->from >> valid;
if (!valid) throw InvalidPath(format("path ‘%s’ is not valid") % path);
}
auto info = std::make_shared<ValidPathInfo>();
@@ -286,12 +296,11 @@ void RemoteStore::queryPathInfoUncached(const Path & path,
if (info->deriver != "") assertStorePath(info->deriver);
info->narHash = parseHash(htSHA256, readString(conn->from));
info->references = readStorePaths<PathSet>(*this, conn->from);
info->registrationTime = readInt(conn->from);
info->narSize = readLongLong(conn->from);
conn->from >> info->registrationTime >> info->narSize;
if (GET_PROTOCOL_MINOR(conn->daemonVersion) >= 16) {
info->ultimate = readInt(conn->from) != 0;
conn->from >> info->ultimate;
info->sigs = readStrings<StringSet>(conn->from);
info->ca = readString(conn->from);
conn->from >> info->ca;
}
return info;
});
@@ -380,8 +389,9 @@ void RemoteStore::addToStore(const ValidPathInfo & info, const ref<std::string>
conn->to << wopAddToStoreNar
<< info.path << info.deriver << printHash(info.narHash)
<< info.references << info.registrationTime << info.narSize
<< info.ultimate << info.sigs << info.ca << *nar << repair << dontCheckSigs;
// FIXME: don't send nar as a string
<< info.ultimate << info.sigs << info.ca
<< repair << dontCheckSigs;
conn->to(*nar);
conn->processStderr();
}
}
@@ -404,7 +414,9 @@ Path RemoteStore::addToStore(const string & name, const Path & _srcPath,
try {
conn->to.written = 0;
conn->to.warn = true;
connections->incCapacity();
dumpPath(srcPath, conn->to, filter);
connections->decCapacity();
conn->to.warn = false;
conn->processStderr();
} catch (SysError & e) {
@@ -515,7 +527,7 @@ Roots RemoteStore::findRoots()
auto conn(connections->get());
conn->to << wopFindRoots;
conn->processStderr();
unsigned int count = readInt(conn->from);
size_t count = readNum<size_t>(conn->from);
Roots result;
while (count--) {
Path link = readString(conn->from);
@@ -563,7 +575,7 @@ bool RemoteStore::verifyStore(bool checkContents, bool repair)
auto conn(connections->get());
conn->to << wopVerifyStore << checkContents << repair;
conn->processStderr();
return readInt(conn->from) != 0;
return readInt(conn->from);
}
@@ -576,6 +588,37 @@ void RemoteStore::addSignatures(const Path & storePath, const StringSet & sigs)
}
void RemoteStore::queryMissing(const PathSet & targets,
PathSet & willBuild, PathSet & willSubstitute, PathSet & unknown,
unsigned long long & downloadSize, unsigned long long & narSize)
{
{
auto conn(connections->get());
if (GET_PROTOCOL_MINOR(conn->daemonVersion) < 19)
// Don't hold the connection handle in the fallback case
// to prevent a deadlock.
goto fallback;
conn->to << wopQueryMissing << targets;
conn->processStderr();
willBuild = readStorePaths<PathSet>(*this, conn->from);
willSubstitute = readStorePaths<PathSet>(*this, conn->from);
unknown = readStorePaths<PathSet>(*this, conn->from);
conn->from >> downloadSize >> narSize;
return;
}
fallback:
return Store::queryMissing(targets, willBuild, willSubstitute,
unknown, downloadSize, narSize);
}
void RemoteStore::connect()
{
auto conn(connections->get());
}
RemoteStore::Connection::~Connection()
{
try {
@@ -599,7 +642,7 @@ void RemoteStore::Connection::processStderr(Sink * sink, Source * source)
}
else if (msg == STDERR_READ) {
if (!source) throw Error("no source");
size_t len = readInt(from);
size_t len = readNum<size_t>(from);
auto buf = std::make_unique<unsigned char[]>(len);
writeString(buf.get(), source->read(buf.get(), len), to);
to.flush();
+16 -3
View File
@@ -22,13 +22,16 @@ class RemoteStore : public virtual Store
{
public:
RemoteStore(const Params & params, size_t maxConnections = std::numeric_limits<size_t>::max());
const Setting<int> maxConnections{(Store*) this, 1,
"max-connections", "maximum number of concurrent connections to the Nix daemon"};
RemoteStore(const Params & params);
/* Implementations of abstract store API methods. */
bool isValidPathUncached(const Path & path) override;
PathSet queryValidPaths(const PathSet & paths) override;
PathSet queryValidPaths(const PathSet & paths, bool maybeSubstitute = false) override;
PathSet queryAllValidPaths() override;
@@ -85,6 +88,12 @@ public:
void addSignatures(const Path & storePath, const StringSet & sigs) override;
void queryMissing(const PathSet & targets,
PathSet & willBuild, PathSet & willSubstitute, PathSet & unknown,
unsigned long long & downloadSize, unsigned long long & narSize) override;
void connect() override;
protected:
struct Connection
@@ -98,6 +107,8 @@ protected:
void processStderr(Sink * sink = 0, Source * source = 0);
};
ref<Connection> openConnectionWrapper();
virtual ref<Connection> openConnection() = 0;
void initConnection(Connection & conn);
@@ -106,6 +117,8 @@ protected:
private:
std::atomic_bool failed{false};
void setOptions(Connection & conn);
};
@@ -113,7 +126,7 @@ class UDSRemoteStore : public LocalFSStore, public RemoteStore
{
public:
UDSRemoteStore(const Params & params, size_t maxConnections = std::numeric_limits<size_t>::max());
UDSRemoteStore(const Params & params);
std::string getUri() override;
+39 -22
View File
@@ -5,6 +5,9 @@
#include "nar-info.hh"
#include "nar-info-disk-cache.hh"
#include "globals.hh"
#include "compression.hh"
#include "download.hh"
#include "istringstream_nocopy.hh"
#include <aws/core/Aws.h>
#include <aws/core/client/ClientConfiguration.h>
@@ -52,8 +55,8 @@ static void initAWS()
});
}
S3Helper::S3Helper()
: config(makeConfig())
S3Helper::S3Helper(const string & region)
: config(makeConfig(region))
, client(make_ref<Aws::S3::S3Client>(*config))
{
}
@@ -70,13 +73,14 @@ class RetryStrategy : public Aws::Client::DefaultRetryStrategy
}
};
ref<Aws::Client::ClientConfiguration> S3Helper::makeConfig()
ref<Aws::Client::ClientConfiguration> S3Helper::makeConfig(const string & region)
{
initAWS();
auto res = make_ref<Aws::Client::ClientConfiguration>();
res->region = Aws::Region::US_EAST_1; // FIXME: make configurable
res->region = region;
res->requestTimeoutMs = 600 * 1000;
res->retryStrategy = std::make_shared<RetryStrategy>();
res->caFile = settings.caFile;
return res;
}
@@ -103,8 +107,10 @@ S3Helper::DownloadResult S3Helper::getObject(
auto result = checkAws(fmt("AWS error fetching ‘%s’", key),
client->GetObject(request));
res.data = std::make_shared<std::string>(
dynamic_cast<std::stringstream &>(result.GetBody()).str());
res.data = decodeContent(
result.GetContentEncoding(),
make_ref<std::string>(
dynamic_cast<std::stringstream &>(result.GetBody()).str()));
} catch (S3Error & e) {
if (e.err != Aws::S3::S3Errors::NO_SUCH_KEY) throw;
@@ -117,19 +123,13 @@ S3Helper::DownloadResult S3Helper::getObject(
return res;
}
#if __linux__
struct istringstream_nocopy : public std::stringstream
{
istringstream_nocopy(const std::string & s)
{
rdbuf()->pubsetbuf(
(char *) s.data(), s.size());
}
};
struct S3BinaryCacheStoreImpl : public S3BinaryCacheStore
{
const Setting<std::string> region{this, Aws::Region::US_EAST_1, "region", {"aws-region"}};
const Setting<std::string> narinfoCompression{this, "", "narinfo-compression", "compression method for .narinfo files"};
const Setting<std::string> lsCompression{this, "", "ls-compression", "compression method for .ls files"};
const Setting<std::string> logCompression{this, "", "log-compression", "compression method for log/* files"};
std::string bucketName;
Stats stats;
@@ -140,6 +140,7 @@ struct S3BinaryCacheStoreImpl : public S3BinaryCacheStore
const Params & params, const std::string & bucketName)
: S3BinaryCacheStore(params)
, bucketName(bucketName)
, s3Helper(region)
{
diskCache = getNarInfoDiskCache();
}
@@ -218,13 +219,20 @@ struct S3BinaryCacheStoreImpl : public S3BinaryCacheStore
return true;
}
void upsertFile(const std::string & path, const std::string & data) override
void uploadFile(const std::string & path, const std::string & data,
const std::string & mimeType,
const std::string & contentEncoding)
{
auto request =
Aws::S3::Model::PutObjectRequest()
.WithBucket(bucketName)
.WithKey(path);
request.SetContentType(mimeType);
if (contentEncoding != "")
request.SetContentEncoding(contentEncoding);
auto stream = std::make_shared<istringstream_nocopy>(data);
request.SetBody(stream);
@@ -247,13 +255,24 @@ struct S3BinaryCacheStoreImpl : public S3BinaryCacheStore
stats.putTimeMs += duration;
}
void upsertFile(const std::string & path, const std::string & data,
const std::string & mimeType) override
{
if (narinfoCompression != "" && hasSuffix(path, ".narinfo"))
uploadFile(path, *compress(narinfoCompression, data), mimeType, narinfoCompression);
else if (lsCompression != "" && hasSuffix(path, ".ls"))
uploadFile(path, *compress(lsCompression, data), mimeType, lsCompression);
else if (logCompression != "" && hasPrefix(path, "log/"))
uploadFile(path, *compress(logCompression, data), mimeType, logCompression);
else
uploadFile(path, data, mimeType, "");
}
void getFile(const std::string & path,
std::function<void(std::shared_ptr<std::string>)> success,
std::function<void(std::exception_ptr exc)> failure) override
{
sync2async<std::shared_ptr<std::string>>(success, failure, [&]() {
debug(format("fetching ‘s3://%1%/%2%’...") % bucketName % path);
stats.get++;
auto res = s3Helper.getObject(bucketName, path);
@@ -313,8 +332,6 @@ static RegisterStoreImplementation regStore([](
return store;
});
#endif
}
#endif
+2 -2
View File
@@ -14,9 +14,9 @@ struct S3Helper
ref<Aws::Client::ClientConfiguration> config;
ref<Aws::S3::S3Client> client;
S3Helper();
S3Helper(const std::string & region);
ref<Aws::Client::ClientConfiguration> makeConfig();
ref<Aws::Client::ClientConfiguration> makeConfig(const std::string & region);
struct DownloadResult
{
+58
View File
@@ -0,0 +1,58 @@
(define TMPDIR (param "_GLOBAL_TMP_DIR"))
(deny default)
; Disallow creating setuid/setgid binaries, since that
; would allow breaking build user isolation.
(deny file-write-setugid)
; Allow forking.
(allow process-fork)
; Allow reading system information like #CPUs, etc.
(allow sysctl-read)
; Allow POSIX semaphores and shared memory.
(allow ipc-posix*)
; Allow socket creation.
(allow system-socket)
; Allow sending signals within the sandbox.
(allow signal (target same-sandbox))
; Access to /tmp.
(allow file* process-exec (literal "/tmp") (subpath TMPDIR))
; Some packages like to read the system version.
(allow file-read* (literal "/System/Library/CoreServices/SystemVersion.plist"))
; Without this line clang cannot write to /dev/null, breaking some configure tests.
(allow file-read-metadata (literal "/dev"))
; Standard devices.
(allow file*
(literal "/dev/null")
(literal "/dev/random")
(literal "/dev/stdin")
(literal "/dev/stdout")
(literal "/dev/tty")
(literal "/dev/urandom")
(literal "/dev/zero")
(subpath "/dev/fd"))
; Does nothing, but reduces build noise.
(allow file* (literal "/dev/dtracehelper"))
; Allow access to zoneinfo since libSystem needs it.
(allow file-read* (subpath "/usr/share/zoneinfo"))
(allow file-read* (subpath "/usr/share/locale"))
; This is mostly to get more specific log messages when builds try to
; access something in /etc or /var.
(allow file-read-metadata
(literal "/etc")
(literal "/var")
(literal "/private/var/tmp")
)
-63
View File
@@ -1,63 +0,0 @@
(allow file-read* file-write-data (literal "/dev/null"))
(allow ipc-posix*)
(allow mach-lookup (global-name "com.apple.SecurityServer"))
(allow file-read*
(literal "/dev/dtracehelper")
(literal "/dev/tty")
(literal "/dev/autofs_nowait")
(literal "/System/Library/CoreServices/SystemVersion.plist")
(literal "/private/var/run/systemkeychaincheck.done")
(literal "/private/etc/protocols")
(literal "/private/var/tmp")
(literal "/private/var/db")
(subpath "/private/var/db/mds"))
(allow file-read*
(subpath "/usr/share/icu")
(subpath "/usr/share/locale")
(subpath "/usr/share/zoneinfo"))
(allow file-write*
(literal "/dev/tty")
(literal "/dev/dtracehelper")
(literal "/mds"))
(allow file-ioctl (literal "/dev/dtracehelper"))
(allow file-read-metadata
(literal "/var")
(literal "/tmp")
; symlinks
(literal "@sysconfdir@")
(literal "@sysconfdir@/nix")
(literal "@sysconfdir@/nix/nix.conf")
(literal "/etc/resolv.conf")
(literal "/private/etc/resolv.conf"))
(allow file-read*
(literal "/private@sysconfdir@/nix/nix.conf")
(literal "/private/var/run/resolv.conf"))
; some builders use filehandles other than stdin/stdout
(allow file*
(subpath "/dev/fd")
(literal "/dev/ptmx")
(regex #"^/dev/[pt]ty.*$"))
; allow everything inside TMP
(allow file* process-exec
(subpath (param "_GLOBAL_TMP_DIR"))
(subpath "/private/tmp"))
(allow process-fork)
(allow sysctl-read)
(allow signal (target same-sandbox))
; allow getpwuid (for git and other packages)
(allow mach-lookup
(global-name "com.apple.system.notification_center")
(global-name "com.apple.system.opendirectoryd.libinfo"))
; allow local networking
(allow network* (local ip) (remote unix-socket))
+5
View File
@@ -0,0 +1,5 @@
(allow default)
; Disallow creating setuid/setgid binaries, since that
; would allow breaking build user isolation.
(deny file-write-setugid)
+16
View File
@@ -0,0 +1,16 @@
; Allow local and remote network traffic.
(allow network* (local ip) (remote ip))
; Allow access to /etc/resolv.conf (which is a symlink to
; /private/var/run/resolv.conf).
(allow file-read-metadata
(literal "/var")
(literal "/etc")
(literal "/etc/resolv.conf")
(literal "/private/etc/resolv.conf"))
(allow file-read*
(literal "/private/var/run/resolv.conf"))
; Allow DNS lookups.
(allow network-outbound (remote unix-socket (path-literal "/private/var/run/mDNSResponder")))
+42 -30
View File
@@ -3,36 +3,25 @@
#include <sqlite3.h>
#include <atomic>
namespace nix {
[[noreturn]] void throwSQLiteError(sqlite3 * db, const format & f)
{
int err = sqlite3_errcode(db);
auto path = sqlite3_db_filename(db, nullptr);
if (!path) path = "(in-memory)";
if (err == SQLITE_BUSY || err == SQLITE_PROTOCOL) {
if (err == SQLITE_PROTOCOL)
printError("warning: SQLite database is busy (SQLITE_PROTOCOL)");
else {
static bool warned = false;
if (!warned) {
printError("warning: SQLite database is busy");
warned = true;
}
}
/* Sleep for a while since retrying the transaction right away
is likely to fail again. */
checkInterrupt();
#if HAVE_NANOSLEEP
struct timespec t;
t.tv_sec = 0;
t.tv_nsec = (random() % 100) * 1000 * 1000; /* <= 0.1s */
nanosleep(&t, 0);
#else
sleep(1);
#endif
throw SQLiteBusy(format("%1%: %2%") % f.str() % sqlite3_errmsg(db));
throw SQLiteBusy(
err == SQLITE_PROTOCOL
? fmt("SQLite database ‘%s’ is busy (SQLITE_PROTOCOL)", path)
: fmt("SQLite database ‘%s’ is busy", path));
}
else
throw SQLiteError(format("%1%: %2%") % f.str() % sqlite3_errmsg(db));
throw SQLiteError("%s: %s (in ‘%s’)", f.str(), sqlite3_errstr(err), path);
}
SQLite::SQLite(const Path & path)
@@ -54,24 +43,27 @@ SQLite::~SQLite()
void SQLite::exec(const std::string & stmt)
{
if (sqlite3_exec(db, stmt.c_str(), 0, 0, 0) != SQLITE_OK)
throwSQLiteError(db, format("executing SQLite statement ‘%s’") % stmt);
retrySQLite<void>([&]() {
if (sqlite3_exec(db, stmt.c_str(), 0, 0, 0) != SQLITE_OK)
throwSQLiteError(db, format("executing SQLite statement ‘%s’") % stmt);
});
}
void SQLiteStmt::create(sqlite3 * db, const string & s)
void SQLiteStmt::create(sqlite3 * db, const string & sql)
{
checkInterrupt();
assert(!stmt);
if (sqlite3_prepare_v2(db, s.c_str(), -1, &stmt, 0) != SQLITE_OK)
throwSQLiteError(db, "creating statement");
if (sqlite3_prepare_v2(db, sql.c_str(), -1, &stmt, 0) != SQLITE_OK)
throwSQLiteError(db, fmt("creating statement ‘%s’", sql));
this->db = db;
this->sql = sql;
}
SQLiteStmt::~SQLiteStmt()
{
try {
if (stmt && sqlite3_finalize(stmt) != SQLITE_OK)
throwSQLiteError(db, "finalizing statement");
throwSQLiteError(db, fmt("finalizing statement ‘%s’", sql));
} catch (...) {
ignoreException();
}
@@ -128,14 +120,14 @@ void SQLiteStmt::Use::exec()
int r = step();
assert(r != SQLITE_ROW);
if (r != SQLITE_DONE)
throwSQLiteError(stmt.db, "executing SQLite statement");
throwSQLiteError(stmt.db, fmt("executing SQLite statement ‘%s’", stmt.sql));
}
bool SQLiteStmt::Use::next()
{
int r = step();
if (r != SQLITE_DONE && r != SQLITE_ROW)
throwSQLiteError(stmt.db, "executing SQLite query");
throwSQLiteError(stmt.db, fmt("executing SQLite query ‘%s’", stmt.sql));
return r == SQLITE_ROW;
}
@@ -182,4 +174,24 @@ SQLiteTxn::~SQLiteTxn()
}
}
void handleSQLiteBusy(const SQLiteBusy & e)
{
static std::atomic<time_t> lastWarned{0};
time_t now = time(0);
if (now > lastWarned + 10) {
lastWarned = now;
printError("warning: %s", e.what());
}
/* Sleep for a while since retrying the transaction right away
is likely to fail again. */
checkInterrupt();
struct timespec t;
t.tv_sec = 0;
t.tv_nsec = (random() % 100) * 1000 * 1000; /* <= 0.1s */
nanosleep(&t, 0);
}
}
+5 -1
View File
@@ -30,8 +30,9 @@ struct SQLiteStmt
{
sqlite3 * db = 0;
sqlite3_stmt * stmt = 0;
std::string sql;
SQLiteStmt() { }
SQLiteStmt(sqlite3 * db, const std::string & s) { create(db, s); }
SQLiteStmt(sqlite3 * db, const std::string & sql) { create(db, sql); }
void create(sqlite3 * db, const std::string & s);
~SQLiteStmt();
operator sqlite3_stmt * () { return stmt; }
@@ -94,6 +95,8 @@ MakeError(SQLiteBusy, SQLiteError);
[[noreturn]] void throwSQLiteError(sqlite3 * db, const format & f);
void handleSQLiteBusy(const SQLiteBusy & e);
/* Convenience function for retrying a SQLite transaction when the
database is busy. */
template<typename T>
@@ -103,6 +106,7 @@ T retrySQLite(std::function<T()> fun)
try {
return fun();
} catch (SQLiteBusy & e) {
handleSQLiteBusy(e);
}
}
}
+27 -61
View File
@@ -4,18 +4,36 @@
#include "archive.hh"
#include "worker-protocol.hh"
#include "pool.hh"
#include "ssh.hh"
namespace nix {
static std::string uriScheme = "ssh://";
static std::string uriScheme = "ssh-ng://";
class SSHStore : public RemoteStore
{
public:
SSHStore(string host, const Params & params, size_t maxConnections = std::numeric_limits<size_t>::max());
const Setting<Path> sshKey{(Store*) this, "", "ssh-key", "path to an SSH private key"};
const Setting<bool> compress{(Store*) this, false, "compress", "whether to compress the connection"};
std::string getUri() override;
SSHStore(const std::string & host, const Params & params)
: Store(params)
, RemoteStore(params)
, host(host)
, master(
host,
sshKey,
// Use SSH master only if using more than 1 connection.
connections->capacity() > 1,
compress)
{
}
std::string getUri() override
{
return uriScheme + host;
}
void narFromPath(const Path & path, Sink & sink) override;
@@ -25,43 +43,16 @@ private:
struct Connection : RemoteStore::Connection
{
Pid sshPid;
AutoCloseFD out;
AutoCloseFD in;
std::unique_ptr<SSHMaster::Connection> sshConn;
};
ref<RemoteStore::Connection> openConnection() override;
AutoDelete tmpDir;
std::string host;
Path socketPath;
Pid sshMaster;
string host;
Path key;
bool compress;
SSHMaster master;
};
SSHStore::SSHStore(string host, const Params & params, size_t maxConnections)
: Store(params)
, RemoteStore(params, maxConnections)
, tmpDir(createTempDir("", "nix", true, true, 0700))
, socketPath((Path) tmpDir + "/ssh.sock")
, host(std::move(host))
, key(get(params, "ssh-key", ""))
, compress(get(params, "compress", "") == "true")
{
/* open a connection and perform the handshake to verify all is well */
connections->get();
}
string SSHStore::getUri()
{
return uriScheme + host;
}
class ForwardSource : public Source
{
@@ -94,35 +85,10 @@ ref<FSAccessor> SSHStore::getFSAccessor()
ref<RemoteStore::Connection> SSHStore::openConnection()
{
if ((pid_t) sshMaster == -1) {
sshMaster = startProcess([&]() {
restoreSignals();
if (key.empty())
execlp("ssh", "ssh", "-N", "-M", "-S", socketPath.c_str(), host.c_str(), NULL);
else
execlp("ssh", "ssh", "-N", "-M", "-S", socketPath.c_str(), "-i", key.c_str(), host.c_str(), NULL);
throw SysError("starting ssh master");
});
}
auto conn = make_ref<Connection>();
Pipe in, out;
in.create();
out.create();
conn->sshPid = startProcess([&]() {
if (dup2(in.readSide.get(), STDIN_FILENO) == -1)
throw SysError("duping over STDIN");
if (dup2(out.writeSide.get(), STDOUT_FILENO) == -1)
throw SysError("duping over STDOUT");
execlp("ssh", "ssh", "-S", socketPath.c_str(), host.c_str(), "nix-daemon", "--stdio", NULL);
throw SysError("executing nix-daemon --stdio over ssh");
});
in.readSide = -1;
out.writeSide = -1;
conn->out = std::move(out.readSide);
conn->in = std::move(in.writeSide);
conn->to = FdSink(conn->in.get());
conn->from = FdSource(conn->out.get());
conn->sshConn = master.startCommand("nix-daemon --stdio");
conn->to = FdSink(conn->sshConn->in.get());
conn->from = FdSource(conn->sshConn->out.get());
initConnection(*conn);
return conn;
}
+104
View File
@@ -0,0 +1,104 @@
#include "ssh.hh"
namespace nix {
void SSHMaster::addCommonSSHOpts(Strings & args)
{
for (auto & i : tokenizeString<Strings>(getEnv("NIX_SSHOPTS")))
args.push_back(i);
if (!keyFile.empty())
args.insert(args.end(), {"-i", keyFile});
if (compress)
args.push_back("-C");
}
std::unique_ptr<SSHMaster::Connection> SSHMaster::startCommand(const std::string & command)
{
Path socketPath = startMaster();
Pipe in, out;
in.create();
out.create();
auto conn = std::make_unique<Connection>();
conn->sshPid = startProcess([&]() {
restoreSignals();
close(in.writeSide.get());
close(out.readSide.get());
if (dup2(in.readSide.get(), STDIN_FILENO) == -1)
throw SysError("duping over stdin");
if (dup2(out.writeSide.get(), STDOUT_FILENO) == -1)
throw SysError("duping over stdout");
if (logFD != -1 && dup2(logFD, STDERR_FILENO) == -1)
throw SysError("duping over stderr");
Strings args = { "ssh", host.c_str(), "-x", "-a" };
addCommonSSHOpts(args);
if (socketPath != "")
args.insert(args.end(), {"-S", socketPath});
args.push_back(command);
execvp(args.begin()->c_str(), stringsToCharPtrs(args).data());
throw SysError("executing ‘%s’ on ‘%s’", command, host);
});
in.readSide = -1;
out.writeSide = -1;
conn->out = std::move(out.readSide);
conn->in = std::move(in.writeSide);
return conn;
}
Path SSHMaster::startMaster()
{
if (!useMaster) return "";
auto state(state_.lock());
if (state->sshMaster != -1) return state->socketPath;
state->tmpDir = std::make_unique<AutoDelete>(createTempDir("", "nix", true, true, 0700));
state->socketPath = (Path) *state->tmpDir + "/ssh.sock";
Pipe out;
out.create();
state->sshMaster = startProcess([&]() {
restoreSignals();
close(out.readSide.get());
if (dup2(out.writeSide.get(), STDOUT_FILENO) == -1)
throw SysError("duping over stdout");
Strings args =
{ "ssh", host.c_str(), "-M", "-N", "-S", state->socketPath
, "-o", "LocalCommand=echo started"
, "-o", "PermitLocalCommand=yes"
};
addCommonSSHOpts(args);
execvp(args.begin()->c_str(), stringsToCharPtrs(args).data());
throw SysError("starting SSH master");
});
out.writeSide = -1;
std::string reply;
try {
reply = readLine(out.readSide.get());
} catch (EndOfFile & e) { }
if (reply != "started")
throw Error("failed to start SSH master connection to ‘%s’", host);
return state->socketPath;
}
}
+51
View File
@@ -0,0 +1,51 @@
#pragma once
#include "util.hh"
#include "sync.hh"
namespace nix {
class SSHMaster
{
private:
const std::string host;
const std::string keyFile;
const bool useMaster;
const bool compress;
const int logFD;
struct State
{
Pid sshMaster;
std::unique_ptr<AutoDelete> tmpDir;
Path socketPath;
};
Sync<State> state_;
void addCommonSSHOpts(Strings & args);
public:
SSHMaster(const std::string & host, const std::string & keyFile, bool useMaster, bool compress, int logFD = -1)
: host(host)
, keyFile(keyFile)
, useMaster(useMaster)
, compress(compress)
, logFD(logFD)
{
}
struct Connection
{
Pid sshPid;
AutoCloseFD out, in;
};
std::unique_ptr<Connection> startCommand(const std::string & command);
Path startMaster();
};
}
+67 -52
View File
@@ -167,7 +167,7 @@ void checkStoreName(const string & name)
collisions (for security). For instance, it shouldn't be feasible
to come up with a derivation whose output path collides with the
path for a copied source. The former would have a <s> starting with
"output:out:", while the latter would have a <2> starting with
"output:out:", while the latter would have a <s> starting with
"source:".
*/
@@ -241,7 +241,8 @@ Path Store::computeStorePathForText(const string & name, const string & s,
Store::Store(const Params & params)
: storeDir(get(params, "store", settings.nixStore))
: Config(params)
, state({(size_t) pathInfoCacheSize})
{
}
@@ -377,7 +378,7 @@ void Store::queryPathInfo(const Path & storePath,
}
PathSet Store::queryValidPaths(const PathSet & paths)
PathSet Store::queryValidPaths(const PathSet & paths, bool maybeSubstitute)
{
struct State
{
@@ -481,21 +482,23 @@ void Store::pathInfoToJSON(JSONPlaceholder & jsonOut, const PathSet & storePaths
if (showClosureSize)
jsonPath.attr("closureSize", getClosureSize(storePath));
if (!includeImpureInfo) continue;
if (includeImpureInfo) {
if (info->deriver != "")
jsonPath.attr("deriver", info->deriver);
if (info->deriver != "")
jsonPath.attr("deriver", info->deriver);
if (info->registrationTime)
jsonPath.attr("registrationTime", info->registrationTime);
if (info->registrationTime)
jsonPath.attr("registrationTime", info->registrationTime);
if (info->ultimate)
jsonPath.attr("ultimate", info->ultimate);
if (info->ultimate)
jsonPath.attr("ultimate", info->ultimate);
if (!info->sigs.empty()) {
auto jsonSigs = jsonPath.list("signatures");
for (auto & sig : info->sigs)
jsonSigs.elem(sig);
}
if (!info->sigs.empty()) {
auto jsonSigs = jsonPath.list("signatures");
for (auto & sig : info->sigs)
jsonSigs.elem(sig);
}
}
}
@@ -522,6 +525,17 @@ const Store::Stats & Store::getStats()
}
void Store::buildPaths(const PathSet & paths, BuildMode buildMode)
{
for (auto & path : paths)
if (isDerivation(path))
unsupported();
if (queryValidPaths(paths).size() != paths.size())
unsupported();
}
void copyStorePath(ref<Store> srcStore, ref<Store> dstStore,
const Path & storePath, bool repair, bool dontCheckSigs)
{
@@ -530,15 +544,23 @@ void copyStorePath(ref<Store> srcStore, ref<Store> dstStore,
StringSink sink;
srcStore->narFromPath({storePath}, sink);
if (srcStore->isTrusted())
dontCheckSigs = true;
if (!info->narHash && dontCheckSigs) {
auto info2 = make_ref<ValidPathInfo>(*info);
info2->narHash = hashString(htSHA256, *sink.s);
if (!info->narSize) info2->narSize = sink.s->size();
info = info2;
}
assert(info->narHash);
if (info->ultimate) {
auto info2 = make_ref<ValidPathInfo>(*info);
info2->ultimate = false;
info = info2;
}
assert(info->narHash);
dstStore->addToStore(*info, sink.s, repair, dontCheckSigs);
}
@@ -550,6 +572,8 @@ void copyClosure(ref<Store> srcStore, ref<Store> dstStore,
for (auto & path : storePaths)
srcStore->computeFSClosure(path, closure);
// FIXME: use copyStorePaths()
PathSet valid = dstStore->queryValidPaths(closure);
if (valid.size() == closure.size()) return;
@@ -695,10 +719,11 @@ namespace nix {
RegisterStoreImplementation::Implementations * RegisterStoreImplementation::implementations = 0;
ref<Store> openStore(const std::string & uri_)
ref<Store> openStore(const std::string & uri_,
const Store::Params & extraParams)
{
auto uri(uri_);
Store::Params params;
Store::Params params(extraParams);
auto q = uri.find('?');
if (q != std::string::npos) {
for (auto s : tokenizeString<Strings>(uri.substr(q + 1), "&")) {
@@ -711,10 +736,13 @@ ref<Store> openStore(const std::string & uri_)
for (auto fun : *RegisterStoreImplementation::implementations) {
auto store = fun(uri, params);
if (store) return ref<Store>(store);
if (store) {
store->warnUnknownSettings();
return ref<Store>(store);
}
}
throw Error(format("don't know how to open Nix store ‘%s’") % uri);
throw Error("don't know how to open Nix store ‘%s’", uri);
}
@@ -724,7 +752,7 @@ StoreType getStoreType(const std::string & uri, const std::string & stateDir)
return tDaemon;
} else if (uri == "local") {
return tLocal;
} else if (uri == "") {
} else if (uri == "" || uri == "auto") {
if (access(stateDir.c_str(), R_OK | W_OK) == 0)
return tLocal;
else if (pathExists(settings.nixDaemonSocketFile))
@@ -772,13 +800,10 @@ std::list<ref<Store>> getDefaultSubstituters()
state->stores.push_back(openStore(uri));
};
for (auto uri : settings.get("substituters", Strings()))
for (auto uri : settings.substituters.get())
addStore(uri);
for (auto uri : settings.get("binary-caches", Strings()))
addStore(uri);
for (auto uri : settings.get("extra-binary-caches", Strings()))
for (auto uri : settings.extraSubstituters.get())
addStore(uri);
state->done = true;
@@ -787,37 +812,26 @@ std::list<ref<Store>> getDefaultSubstituters()
}
void copyPaths(ref<Store> from, ref<Store> to, const Paths & storePaths, bool substitute)
void copyPaths(ref<Store> from, ref<Store> to, const PathSet & storePaths,
bool substitute, bool dontCheckSigs)
{
if (substitute) {
/* Filter out .drv files (we don't want to build anything). */
PathSet paths2;
for (auto & path : storePaths)
if (!isDerivation(path)) paths2.insert(path);
unsigned long long downloadSize, narSize;
PathSet willBuild, willSubstitute, unknown;
to->queryMissing(PathSet(paths2.begin(), paths2.end()),
willBuild, willSubstitute, unknown, downloadSize, narSize);
/* FIXME: should use ensurePath(), but it only
does one path at a time. */
if (!willSubstitute.empty())
try {
to->buildPaths(willSubstitute);
} catch (Error & e) {
printMsg(lvlError, format("warning: %1%") % e.msg());
}
}
PathSet valid = to->queryValidPaths(storePaths, substitute);
PathSet missing;
for (auto & path : storePaths)
if (!valid.count(path)) missing.insert(path);
std::string copiedLabel = "copied";
logger->setExpected(copiedLabel, storePaths.size());
//logger->setExpected(copiedLabel, missing.size());
ThreadPool pool;
processGraph<Path>(pool,
PathSet(storePaths.begin(), storePaths.end()),
PathSet(missing.begin(), missing.end()),
[&](const Path & storePath) {
if (to->isValidPath(storePath)) return PathSet();
return from->queryPathInfo(storePath)->references;
},
@@ -825,13 +839,14 @@ void copyPaths(ref<Store> from, ref<Store> to, const Paths & storePaths, bool su
checkInterrupt();
if (!to->isValidPath(storePath)) {
Activity act(*logger, lvlInfo, format("copying ‘%s’...") % storePath);
//Activity act(*logger, lvlInfo, format("copying ‘%s’...") % storePath);
copyStorePath(from, to, storePath);
copyStorePath(from, to, storePath, false, dontCheckSigs);
logger->incProgress(copiedLabel);
//logger->incProgress(copiedLabel);
} else
logger->incExpected(copiedLabel, -1);
;
//logger->incExpected(copiedLabel, -1);
});
pool.process();
+81 -35
View File
@@ -6,6 +6,7 @@
#include "lru-cache.hh"
#include "sync.hh"
#include "globals.hh"
#include "config.hh"
#include <atomic>
#include <limits>
@@ -17,6 +18,12 @@
namespace nix {
MakeError(SubstError, Error)
MakeError(BuildError, Error) /* denotes a permanent build failure */
MakeError(InvalidPath, Error)
MakeError(Unsupported, Error)
struct BasicDerivation;
struct Derivation;
class FSAccessor;
@@ -81,12 +88,7 @@ struct GCResults
/* For `gcReturnDead', `gcDeleteDead' and `gcDeleteSpecific', the
number of bytes that would be or was freed. */
unsigned long long bytesFreed;
GCResults()
{
bytesFreed = 0;
}
unsigned long long bytesFreed = 0;
};
@@ -111,9 +113,8 @@ struct ValidPathInfo
uint64_t narSize = 0; // 0 = unknown
uint64_t id; // internal use only
/* Whether the path is ultimately trusted, that is, it was built
locally or is content-addressable (e.g. added via addToStore()
or the result of a fixed-output derivation). */
/* Whether the path is ultimately trusted, that is, it's a
derivation output that was built locally. */
bool ultimate = false;
StringSet sigs; // note: not necessarily verified
@@ -229,19 +230,23 @@ struct BuildResult
};
class Store : public std::enable_shared_from_this<Store>
class Store : public std::enable_shared_from_this<Store>, public Config
{
public:
typedef std::map<std::string, std::string> Params;
const Path storeDir;
const PathSetting storeDir_{this, false, settings.nixStore,
"store", "path to the Nix store"};
const Path storeDir = storeDir_;
const Setting<int> pathInfoCacheSize{this, 65536, "path-info-cache-size", "size of the in-memory store path information cache"};
protected:
struct State
{
LRUCache<std::string, std::shared_ptr<ValidPathInfo>> pathInfoCache{64 * 1024};
LRUCache<std::string, std::shared_ptr<ValidPathInfo>> pathInfoCache;
};
Sync<State> state;
@@ -324,8 +329,10 @@ protected:
public:
/* Query which of the given paths is valid. */
virtual PathSet queryValidPaths(const PathSet & paths);
/* Query which of the given paths is valid. Optionally, try to
substitute missing paths. */
virtual PathSet queryValidPaths(const PathSet & paths,
bool maybeSubstitute = false);
/* Query the set of all valid paths. Note that for some store
backends, the name part of store paths may be omitted
@@ -412,7 +419,7 @@ public:
output paths can be created by running the builder, after
recursively building any sub-derivations. For inputs that are
not derivations, substitute them. */
virtual void buildPaths(const PathSet & paths, BuildMode buildMode = bmNormal) = 0;
virtual void buildPaths(const PathSet & paths, BuildMode buildMode = bmNormal);
/* Build a single non-materialized derivation (i.e. not from an
on-disk .drv file). Note that ‘drvPath’ is only used for
@@ -511,7 +518,7 @@ public:
`storePath' is returned; that is, the closures under the
`referrers' relation instead of the `references' relation is
returned. */
void computeFSClosure(const PathSet & paths,
virtual void computeFSClosure(const PathSet & paths,
PathSet & out, bool flipDirection = false,
bool includeOutputs = false, bool includeDerivers = false);
@@ -522,7 +529,7 @@ public:
/* Given a set of paths that are to be built, return the set of
derivations that will be built, and the set of output paths
that will be substituted. */
void queryMissing(const PathSet & targets,
virtual void queryMissing(const PathSet & targets,
PathSet & willBuild, PathSet & willSubstitute, PathSet & unknown,
unsigned long long & downloadSize, unsigned long long & narSize);
@@ -562,23 +569,52 @@ public:
const Stats & getStats();
/* Whether this store paths from this store can be imported even
if they lack a signature. */
virtual bool isTrusted() { return false; }
/* Return the build log of the specified store path, if available,
or null otherwise. */
virtual std::shared_ptr<std::string> getBuildLog(const Path & path)
{ return nullptr; }
/* Hack to allow long-running processes like hydra-queue-runner to
occasionally flush their path info cache. */
void clearPathInfoCache()
{
state.lock()->pathInfoCache.clear();
}
/* Establish a connection to the store, for store types that have
a notion of connection. Otherwise this is a no-op. */
virtual void connect() { };
protected:
Stats stats;
/* Unsupported methods. */
[[noreturn]] void unsupported()
{
throw Unsupported("requested operation is not supported by store ‘%s’", getUri());
}
};
class LocalFSStore : public virtual Store
{
public:
const Path rootDir;
const Path stateDir;
const Path logDir;
// FIXME: the (Store*) cast works around a bug in gcc that causes
// it to emit the call to the Option constructor. Clang works fine
// either way.
const PathSetting rootDir{(Store*) this, true, "",
"root", "directory prefixed to all other paths"};
const PathSetting stateDir{(Store*) this, false,
rootDir != "" ? rootDir + "/nix/var/nix" : settings.nixStateDir,
"state", "directory where Nix will store state"};
const PathSetting logDir{(Store*) this, false,
rootDir != "" ? rootDir + "/nix/var/log/nix" : settings.nixLogDir,
"log", "directory where Nix will store state"};
const static string drvsLogDir;
LocalFSStore(const Params & params);
@@ -595,6 +631,8 @@ public:
{
return getRealStoreDir() + "/" + baseNameOf(storePath);
}
std::shared_ptr<std::string> getBuildLog(const Path & path) override;
};
@@ -629,21 +667,35 @@ void removeTempRoots();
/* Return a Store object to access the Nix store denoted by
‘uri’ (slight misnomer...). Supported values are:
* ‘direct’: The Nix store in /nix/store and database in
* ‘local’: The Nix store in /nix/store and database in
/nix/var/nix/db, accessed directly.
* ‘daemon’: The Nix store accessed via a Unix domain socket
connection to nix-daemon.
* ‘auto’ or ‘’: Equivalent to ‘local’ or ‘daemon’ depending on
whether the user has write access to the local Nix
store/database.
* ‘file://<path>’: A binary cache stored in <path>.
If ‘uri’ is empty, it defaults to ‘direct’ or ‘daemon’ depending on
whether the user has write access to the local Nix store/database.
set to true *unless* you're going to collect garbage. */
ref<Store> openStore(const std::string & uri = getEnv("NIX_REMOTE"));
* ‘https://<path>’: A binary cache accessed via HTTP.
* ‘s3://<path>’: A writable binary cache stored on Amazon's Simple
Storage Service.
* ‘ssh://[user@]<host>’: A remote Nix store accessed by running
‘nix-store --serve’ via SSH.
You can pass parameters to the store implementation by appending
‘?key=value&key=value&...’ to the URI.
*/
ref<Store> openStore(const std::string & uri = getEnv("NIX_REMOTE"),
const Store::Params & extraParams = Store::Params());
void copyPaths(ref<Store> from, ref<Store> to, const Paths & storePaths, bool substitute = false);
void copyPaths(ref<Store> from, ref<Store> to, const PathSet & storePaths,
bool substitute = false, bool dontCheckSigs = false);
enum StoreType {
tDaemon,
@@ -691,10 +743,4 @@ ValidPathInfo decodeValidPathInfo(std::istream & str,
for paths created by makeFixedOutputPath() / addToStore(). */
std::string makeFixedOutputCA(bool recursive, const Hash & hash);
MakeError(SubstError, Error)
MakeError(BuildError, Error) /* denotes a permanent build failure */
MakeError(InvalidPath, Error)
}
+3 -2
View File
@@ -6,7 +6,7 @@ namespace nix {
#define WORKER_MAGIC_1 0x6e697863
#define WORKER_MAGIC_2 0x6478696f
#define PROTOCOL_VERSION 0x112
#define PROTOCOL_VERSION 0x113
#define GET_PROTOCOL_MAJOR(x) ((x) & 0xff00)
#define GET_PROTOCOL_MINOR(x) ((x) & 0x00ff)
@@ -47,7 +47,8 @@ typedef enum {
wopBuildDerivation = 36,
wopAddSignatures = 37,
wopNarFromPath = 38,
wopAddToStoreNar = 39
wopAddToStoreNar = 39,
wopQueryMissing = 40,
} WorkerOp;
+7
View File
@@ -70,6 +70,13 @@ struct ParseSink
virtual void createSymlink(const Path & path, const string & target) { };
};
struct TeeSink : ParseSink
{
TeeSource source;
TeeSink(Source & source) : source(source) { }
};
void parseDump(ParseSink & sink, Source & source);
void restorePath(const Path & path, Source & source);
+20 -6
View File
@@ -3,6 +3,18 @@
namespace nix {
Args::FlagMaker Args::mkFlag()
{
return FlagMaker(*this);
}
Args::FlagMaker::~FlagMaker()
{
assert(flag->longName != "");
args.longFlags[flag->longName] = flag;
if (flag->shortName) args.shortFlags[flag->shortName] = flag;
}
void Args::parseCmdline(const Strings & _cmdline)
{
Strings pendingArgs;
@@ -71,11 +83,13 @@ void Args::printHelp(const string & programName, std::ostream & out)
void Args::printFlags(std::ostream & out)
{
Table2 table;
for (auto & flag : longFlags)
for (auto & flag : longFlags) {
if (hiddenCategories.count(flag.second->category)) continue;
table.push_back(std::make_pair(
(flag.second.shortName ? std::string("-") + flag.second.shortName + ", " : " ")
+ "--" + flag.first + renderLabels(flag.second.labels),
flag.second.description));
(flag.second->shortName ? std::string("-") + flag.second->shortName + ", " : " ")
+ "--" + flag.first + renderLabels(flag.second->labels),
flag.second->description));
}
printTable(out, table);
}
@@ -99,14 +113,14 @@ bool Args::processFlag(Strings::iterator & pos, Strings::iterator end)
if (string(*pos, 0, 2) == "--") {
auto i = longFlags.find(string(*pos, 2));
if (i == longFlags.end()) return false;
return process("--" + i->first, i->second);
return process("--" + i->first, *i->second);
}
if (string(*pos, 0, 1) == "-" && pos->size() == 2) {
auto c = (*pos)[1];
auto i = shortFlags.find(c);
if (i == shortFlags.end()) return false;
return process(std::string("-") + c, i->second);
return process(std::string("-") + c, *i->second);
}
return false;
+58 -27
View File
@@ -29,19 +29,22 @@ protected:
/* Flags. */
struct Flag
{
char shortName;
typedef std::shared_ptr<Flag> ptr;
std::string longName;
char shortName = 0;
std::string description;
Strings labels;
size_t arity;
size_t arity = 0;
std::function<void(Strings)> handler;
std::string category;
};
std::map<std::string, Flag> longFlags;
std::map<char, Flag> shortFlags;
std::map<std::string, Flag::ptr> longFlags;
std::map<char, Flag::ptr> shortFlags;
virtual bool processFlag(Strings::iterator & pos, Strings::iterator end);
void printFlags(std::ostream & out);
virtual void printFlags(std::ostream & out);
/* Positional arguments. */
struct ExpectedArg
@@ -55,33 +58,53 @@ protected:
virtual bool processArgs(const Strings & args, bool finish);
std::set<std::string> hiddenCategories;
public:
class FlagMaker
{
Args & args;
Flag::ptr flag;
friend class Args;
FlagMaker(Args & args) : args(args), flag(std::make_shared<Flag>()) { };
public:
~FlagMaker();
FlagMaker & longName(const std::string & s) { flag->longName = s; return *this; };
FlagMaker & shortName(char s) { flag->shortName = s; return *this; };
FlagMaker & description(const std::string & s) { flag->description = s; return *this; };
FlagMaker & labels(const Strings & ls) { flag->labels = ls; return *this; };
FlagMaker & arity(size_t arity) { flag->arity = arity; return *this; };
FlagMaker & handler(std::function<void(Strings)> handler) { flag->handler = handler; return *this; };
FlagMaker & category(const std::string & s) { flag->category = s; return *this; };
};
FlagMaker mkFlag();
/* Helper functions for constructing flags / positional
arguments. */
void mkFlag(char shortName, const std::string & longName,
const Strings & labels, const std::string & description,
size_t arity, std::function<void(Strings)> handler)
{
auto flag = Flag{shortName, description, labels, arity, handler};
if (shortName) shortFlags[shortName] = flag;
longFlags[longName] = flag;
}
void mkFlag(char shortName, const std::string & longName,
const std::string & description, std::function<void()> fun)
{
mkFlag(shortName, longName, {}, description, 0, std::bind(fun));
mkFlag()
.shortName(shortName)
.longName(longName)
.description(description)
.handler(std::bind(fun));
}
void mkFlag1(char shortName, const std::string & longName,
const std::string & label, const std::string & description,
std::function<void(std::string)> fun)
{
mkFlag(shortName, longName, {label}, description, 1, [=](Strings ss) {
fun(ss.front());
});
mkFlag()
.shortName(shortName)
.longName(longName)
.labels({label})
.description(description)
.arity(1)
.handler([=](Strings ss) { fun(ss.front()); });
}
void mkFlag(char shortName, const std::string & name,
@@ -105,9 +128,11 @@ public:
void mkFlag(char shortName, const std::string & longName, const std::string & description,
T * dest, const T & value)
{
mkFlag(shortName, longName, {}, description, 0, [=](Strings ss) {
*dest = value;
});
mkFlag()
.shortName(shortName)
.longName(longName)
.description(description)
.handler([=](Strings ss) { *dest = value; });
}
template<class I>
@@ -123,12 +148,18 @@ public:
void mkFlag(char shortName, const std::string & longName,
const std::string & description, std::function<void(I)> fun)
{
mkFlag(shortName, longName, {"N"}, description, 1, [=](Strings ss) {
I n;
if (!string2Int(ss.front(), n))
throw UsageError(format("flag ‘--%1%’ requires a integer argument") % longName);
fun(n);
});
mkFlag()
.shortName(shortName)
.longName(longName)
.labels({"N"})
.description(description)
.arity(1)
.handler([=](Strings ss) {
I n;
if (!string2Int(ss.front(), n))
throw UsageError(format("flag ‘--%1%’ requires a integer argument") % longName);
fun(n);
});
}
/* Expect a string argument. */
Loaded 100 of 173 files, more files were not shown because too many files have changed in this diff. Show more