Compare commits

..
Author SHA1 Message Date
Eelco Dolstra 33d58a1904 Bump version 2017-12-12 17:48:02 +01:00
Eelco Dolstra c6e15c4322 Disable shellcheck
This breaks eval on aarch64-linux due to the dependency on GHC.
2017-12-12 15:33:53 +01:00
Eelco Dolstra 7487debaf5 Merge pull request #1700 from grahamc/fixup-darwin-hs
darwin installer: fix on High Sierra
2017-12-12 11:51:14 +01:00
Eelco Dolstra a18c085af8 Merge pull request #1716 from grahamc/build-1.11-on-aarch
Build 1.11 on aarch64-linux
2017-12-08 14:00:21 +01:00
Graham Christensen 6b887844d6 replace lolcat with nix-info
(cherry picked from commit 1db034364a)
2017-12-08 07:30:51 -05:00
Graham Christensen c32bdf0548 darwin installer: fix on High Sierra
(cherry picked from commit be79d1f189)
2017-12-08 07:30:44 -05:00
Eelco Dolstra e462fa746b Build on aarch64-linux
(cherry picked from commit da76c72bc9)
2017-12-07 17:22:28 -05:00
Eelco Dolstra 3454356001 Merge pull request #1648 from danbst/patch-1
nix-collect-garbage: little doc fix
2017-11-03 10:51:16 +01:00
Danylo Hlynskyi c7c806ec21 nix-collect-garbage: little doc fix
This removes confusing documentation. It's better to remove doc than add implementation, because Nix 1.12 will surely have new GC interface anyway.

Fixes https://github.com/NixOS/nix/issues/641
2017-10-31 05:12:08 +02:00
Eelco Dolstra 76431e0a26 nix-shell -p: Use runCommandCC
This restores pre-17.03 behaviour by making gcc available.

(Ported from 9b63bb88c8)
2017-10-13 17:28:04 +02:00
Shea Levy 3c52567b3e darwin installer: Fix on systems where sudo -i is disabled.
(cherry picked from commit fb98e29067)
2017-10-12 13:08:33 +02:00
Anthony Cowley b03066835c install-darwin-multi-user: relax assumption check
The installer will error out if a user's shell configuration includes any mention of ~nix-profile~, even if this is in a comment. This change is designed to do the bare minimum to ignore lines beginning with a `#`.

(cherry picked from commit 92f9d18aa0)
2017-10-12 13:06:46 +02:00
Eelco Dolstra aa57c32318 Bump version 2017-09-15 16:40:20 +02:00
Eelco Dolstra f217acd9e7 Merge pull request #1566 from FPtje/1.11-maintenance
scripts/nix-copy-closure.in: Automatically flush STDERR handle
2017-09-10 17:51:31 +02:00
Falco Peijnenburg b3ad7c5052 scripts/nix-copy-closure.in: Automatically flush STDERR handle
Fixes https://github.com/NixOS/nixops/issues/455
2017-09-10 17:38:18 +02:00
Eelco Dolstra e8235c4f3e Bump version 2017-09-01 11:13:16 +02:00
Sergei Trofimovich 41352d50db src/libmain/stack.cc: fix 'ucontext' usage on glibc-2.26
Build fails as:

$ make
  CXX    src/libmain/stack.o
src/libmain/stack.cc: In function 'void nix::sigsegvHandler(int, siginfo_t*, void*)':
src/libmain/stack.cc:21:21: error: 'ucontext' was not declared in this scope
     sp = (char *) ((ucontext *) ctx)->uc_mcontext.gregs[REG_RSP];
                     ^~~~~~~~
src/libmain/stack.cc:21:21: note: suggested alternative: 'ucontext_t'
     sp = (char *) ((ucontext *) ctx)->uc_mcontext.gregs[REG_RSP];
                     ^~~~~~~~
                     ucontext_t

It's caused by upstream rename:
https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=251287734e89a52da3db682a8241eb6bccc050c9

which basically changes
    typedef struct ucontext {} ucontext_t;
to
    typedef struct ucontext_t {} ucontext_t;

The change uses ucontext_t.

Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
(cherry picked from commit c9857ef262)
2017-09-01 11:12:49 +02:00
Eelco Dolstra b3a616e860 Bump version 2017-08-30 20:03:33 +02:00
Eelco Dolstra 617dab1515 Merge pull request #1543 from matthewbauer/1.11-maintenance
OS X sandbox: Store .sb file in $TMPDIR rather than the Nix store
2017-08-30 19:50:40 +02:00
Eelco Dolstra 35f541a7a0 OS X sandbox: Store .sb file in $TMPDIR rather than the Nix store
The filename used was not unique and owned by the build user, so
builds could fail with

error: while setting up the build environment: cannot unlink ‘/nix/store/99i210ihnsjacajaw8r33fmgjvzpg6nr-bison-3.0.4.drv.sb’: Permission denied
2017-08-30 12:06:31 -05:00
Eelco Dolstra 23795d47fb Merge pull request #1540 from grahamc/test-o
1.11: Replace stat / uid comparison with test -O
2017-08-30 15:43:47 +02:00
Graham Christensen d07697de5f Replace stat / uid comparison with test -O 2017-08-29 18:56:32 -04:00
Domen Kožar 3afb744185 Merge pull request #1313 from neilmayhew/topic/cores
Propagate NIX_BUILD_CORES to nix-shell environments
2017-08-02 15:02:25 +02:00
Eelco Dolstra 04532ee2c8 Fix test failure
(cherry picked from commit 46e36f9b73)
Signed-off-by: Domen Kožar <domen@dev.si>
2017-07-25 11:50:31 +02:00
Eelco Dolstra 24883645a6 Merge pull request #1486 from grahamc/detect-already-running
Test to see if nix-daemon is running already
2017-07-25 09:24:06 +02:00
Graham Christensen 9a04bc6f5e Test to see if nix-daemon is running already
A user had an install, uninstalled it via the instructions, and found their nix broken.

The issue was based around nix-daemon already running.

I tried to be very smart around searching their launchctl files:

  $ grep -ri nix-daemon  ~/Library/LaunchAgents /Library/LaunchAgents /Library/LaunchDaemons /System/Library/LaunchAgents /System/Library/LaunchDaemons

but then couldn't figure out how to write out (nicely) commands for uninstalling
or unloading them based on their name, and also couching the instrunctions in a
bit of doubt so we don't piss off some poor mac admin because their users just
deleted some important launchd files.
2017-07-24 22:37:22 -04:00
Eelco Dolstra 0ec723375b Merge pull request #1469 from grahamc/domen-feedback-nix
Show users how to source the profile, to get a working Nix.
2017-07-17 15:05:17 +02:00
Graham Christensen 4b45f94b5c Show users how to source the profile, to get a working Nix. 2017-07-17 08:39:14 -04:00
Eelco Dolstra 16daed49dc Merge pull request #1463 from grahamc/fix-stdin-errors
Fix stdin errors
2017-07-14 15:07:57 +02:00
Graham Christensen 41f0a08a31 bump version to 1.11.13 2017-07-14 06:28:47 -04:00
Graham Christensen e4831a4412 Rename PINCH_ME_IM_SILLY to ALLOW_PREEXISTING_INSTALLATION 2017-07-14 06:28:24 -04:00
Graham Christensen 6ba624f25a If there is no TTY, also skip verbose sudo messages 2017-07-14 06:23:51 -04:00
Graham Christensen 6b6d4b69c1 Ensure PINCH_ME_IM_SILLY allows a /nix/store to stick around between builds
Also output in the status report that the user is very silly
2017-07-13 19:45:00 -04:00
Graham Christensen b01eaa6114 Assume yes if we have no TTY
Starve the TTY of input to ensure this works, but provide yes to the
current installer to handle the current broken case.
2017-07-13 18:03:53 -04:00
Eelco Dolstra 04e071a5e4 Bump version 2017-07-13 15:10:09 +02:00
Domen Kožar 44f51a5dd2 Fix rpm build 2017-07-13 14:11:21 +02:00
Eelco Dolstra e135db704f Merge pull request #1453 from grahamc/multi-user-darwin
Multi user darwin installer
2017-07-13 13:03:28 +02:00
Graham Christensen c82126790d Cleanup and more specificity around set -e 2017-07-12 20:43:57 -04:00
Graham Christensen 85acfcd6bd Only clean if the file exists 2017-07-12 20:31:33 -04:00
Graham Christensen b2917c8246 Clean up nix hints from the old insstaller 2017-07-12 20:29:58 -04:00
Graham Christensen a31347d6ec release: don't build libseccomp if we're on darwin 2017-07-12 19:23:42 -04:00
Graham Christensen c4f349d572 Run nix-build inside a fresh bash login 2017-07-12 17:10:14 -04:00
Graham Christensen 163d93125e chmod 2017-07-12 12:58:37 -04:00
Graham Christensen 302e820660 Test the installer 2017-07-12 11:45:38 -04:00
Graham Christensen 2442c4684d Address feedback around printf & exec 2017-07-12 11:45:35 -04:00
Graham Christensen d4f128352e Don't install a second nix after the initial installation, and the rsync change fixes a bug hidden by the nix replacement where the store files were being owned by the installing user due to rsync's -a implying -og. 2017-07-12 11:45:32 -04:00
Graham Christensen 0c13077d83 nix: build with libsodium on macOS 2017-07-12 11:45:28 -04:00
Graham Christensen 661daed683 Clean up issues around uninstall directions, and only show
relevant directions
2017-07-12 11:45:25 -04:00
Graham Christensen 262a08c0e2 Prompt for sudo before validating assumptions, and check ourselves for root-owned files instead of making a scary warning. 2017-07-12 11:45:22 -04:00
Graham Christensen 3ebd25a644 multi-user install: move the profile in to the nix etc/profiles.d output 2017-07-12 11:45:19 -04:00
Graham Christensen 218978154a Switch to a fancy multi-user installer on Darwin 2017-07-12 11:45:13 -04:00
Graham Christensen 799f5adf79 Shellcheck the existing installer 2017-07-12 11:44:28 -04:00
Eelco Dolstra 84d10d248e Fix build 2017-07-12 13:00:41 +02:00
Robert Vollmert 758a3044f1 Fix nix-instantiate manpage indentation
The second command variant is now its own cmdsynopsis, which ensures
it's not indented as was the case using sbrk.

(cherry picked from commit 60da5d2b8f)
2017-07-12 11:31:24 +02:00
Matthew Bauer 5a7d00ced8 Don’t hardlink disallowed paths in OS X.
Fixes #1443

(cherry picked from commit 72e80c59b5)
2017-07-12 11:31:17 +02:00
Eelco Dolstra 9943f98c35 Add X32 to the seccomp filter
Fixes #1432.

(cherry picked from commit a3dc1e65ab)
2017-07-12 11:30:28 +02:00
Eelco Dolstra b59788fc48 fetchTarball: Prevent concurrent downloads of the same file
Fixes #849.

(cherry picked from commit 8e8caf7f3e)
2017-07-12 11:28:59 +02:00
Eelco Dolstra 7577d35895 replaceSymlink(): Handle the case where the temporary file already exists
Not really necessary anymore for #849, but still nice to have.

(cherry picked from commit 2965d40612)
2017-07-12 11:28:05 +02:00
Shea Levy 2a0112a370 Merge branch 'add-nix-profile-daemon' of git://github.com/grahamc/nix into 1.11-maintenance 2017-07-09 16:03:15 -04:00
Graham Christensen 3e0a503bf7 Create a profile suitable for multi-user installs 2017-07-09 14:52:33 -04:00
Eelco Dolstra 026f4f9ae8 macOS: Remove flags
In particular, UF_IMMUTABLE (uchg) needs to be cleared to allow the
path to be garbage-collected or optimised.

See https://github.com/NixOS/nixpkgs/issues/25819.
+       the file from being garbage-collected.

(cherry picked from commit b5bdfdef73)
2017-06-19 14:32:38 +02:00
Eelco Dolstra 11dd08f02e macOS: Ugly hack to make the tests succeed
Sandboxes cannot be nested, so if Nix's build runs inside a sandbox,
it cannot use a sandbox itself. I don't see a clean way to detect
whether we're in a sandbox, so use a test-specific hack.

https://github.com/NixOS/nix/issues/1413
(cherry picked from commit 1888f7889b)
2017-06-19 14:28:04 +02:00
Shea Levy 3e574c3691 Merge branch '1.11-maintenance' of git://github.com/cyraxjoe/nix into 1.11-maintenance 2017-06-14 07:52:36 -04:00
Eelco Dolstra 5ac7088726 Bump version 2017-06-14 11:43:51 +02:00
Eelco Dolstra 35ea3d62dc canonicalisePathMetaData(): Ignore security.selinux attribute
Untested, hopefully fixes #1406.

(cherry picked from commit 88b291ffc4)
2017-06-14 11:43:33 +02:00
Joel Rivera 7917494c45 Add support for the curl netrc file in nix-channel/nix-pull.
Based on the implementation in download-from-binary-cache.pl.in.
2017-06-13 18:18:20 -05:00
Eelco Dolstra 1e4885e316 Grmbl 2017-06-12 18:52:26 +02:00
Eelco Dolstra 36f363b8f0 On macOS, don't use /var/folders for TMPDIR
This broke "nix-store --serve".

(cherry picked from commit 25230a17a9)
2017-06-12 18:39:34 +02:00
Eelco Dolstra c33854513a Remove Ubuntu 13.10 build
Seccomp is too old there.
2017-06-12 16:47:16 +02:00
Eelco Dolstra 0be5b949d3 Don't run pre-build-hook if we don't have a derivation
This fixes a build failure on OS X when using Hydra or Nix 1.12's
build-remote (since they don't copy the derivation to the build
machine).

(cherry picked from commit 7f5b750b40)
2017-06-12 16:46:31 +02:00
Eelco Dolstra c20641ce56 OS X -> macOS 2017-06-12 14:04:52 +02:00
Eelco Dolstra 0fb60e4e0f Add 1.11.10 release notes 2017-06-12 13:56:38 +02:00
Eelco Dolstra 3414f3804c Fix build 2017-06-12 13:55:59 +02:00
Eelco Dolstra 8e298e8ad9 Always use the Darwin sandbox
Even with "build-use-sandbox = false", we now use sandboxing with a
permissive profile that allows everything except the creation of
setuid/setgid binaries.

Based on 85e93d7b87.
2017-06-06 20:35:55 +02:00
Eelco Dolstra f534627929 Fix bad cherrypick 2017-06-06 19:52:40 +02:00
Eelco Dolstra 0ca9502264 Disable the build user mechanism on all platforms except Linux and OS X
(cherry picked from commit c8cc50d46e)
2017-06-06 19:52:24 +02:00
Eelco Dolstra bcc21744df Bump version 2017-06-01 16:53:10 +02:00
Eelco Dolstra 833aae4509 Fix coverage job
(cherry picked from commit b4b1f4525f)
2017-06-01 16:51:32 +02:00
Eelco Dolstra aabe20bf78 RPM, Deb: Add dependency on libseccomp
(cherry picked from commit ab5834f7a1)
2017-06-01 16:51:03 +02:00
Eelco Dolstra c48697d617 Remove listxattr assertion
It appears that sometimes, listxattr() returns a different value for
the query case (i.e. when the buffer size is 0).

(cherry picked from commit 52fec8dde8)
2017-06-01 16:50:23 +02:00
Eelco Dolstra 4be5a65b39 Fix seccomp build failure on clang
Fixes

  src/libstore/build.cc:2321:45: error: non-constant-expression cannot be narrowed from type 'int' to 'scmp_datum_t' (aka 'unsigned long') in initializer list [-Wc++11-narrowing]

(cherry picked from commit fe08d17934)
2017-06-01 16:50:08 +02:00
Eelco Dolstra 634d117ede Add a seccomp rule to disallow setxattr()
(cherry picked from commit 2ac99a32da)
2017-06-01 16:50:04 +02:00
Eelco Dolstra 66618dbad5 canonicalisePathMetaData(): Remove extended attributes / ACLs
EAs/ACLs are not part of the NAR canonicalisation. Worse, setting an
ACL allows a builder to create writable files in the Nix store. So get
rid of them.

Closes #185.

(cherry picked from commit d798349ede)
2017-06-01 16:50:00 +02:00
Eelco Dolstra 1e0f1dab1e Require seccomp only in multi-user setups
(cherry picked from commit ff6becafa8)
2017-06-01 16:49:52 +02:00
Eelco Dolstra 17da82e04d Add test for setuid seccomp filter
(cherry picked from commit 1d9ab273ba)
2017-06-01 16:49:47 +02:00
Eelco Dolstra a2cf0f1018 Fix seccomp initialisation on i686-linux
(cherry picked from commit cf93397d3f)
2017-06-01 16:49:42 +02:00
Eelco Dolstra e296b8884e Add a seccomp filter to prevent creating setuid/setgid binaries
This prevents builders from setting the S_ISUID or S_ISGID bits,
preventing users from using a nixbld* user to create a setuid/setgid
binary to interfere with subsequent builds under the same nixbld* uid.

This is based on aszlig's seccomp code
(47f587700d).

Reported by Linus Heckemann.

(cherry picked from commit 6cc6c15a2d)
2017-06-01 16:48:57 +02:00
Guillaume Maudoux a8d13e66ee lexer: remove catch-all rules hiding real errors
With catch-all rules, we hide potential errors.
It turns out that a4744254 made one cath-all useless. Flex detected that
is was impossible to reach.
The other is more subtle, as it can only trigger on unfinished escapes
in unfinished strings, which only occurs at EOF.

(cherry picked from commit a143014d73)
2017-05-01 11:26:44 +02:00
Guillaume Maudoux a10bd3355a Fix lexer to support $' in multiline strings.
(cherry picked from commit a474425425)
2017-05-01 11:26:41 +02:00
Eelco Dolstra 26f7e8305b Fix hash computation when importing NARs greater than 4 GiB
This caused "nix-store --import" to compute an incorrect hash on NARs
that don't fit in an unsigned int. The import would succeed, but
"nix-store --verify-path" or subsequent exports would detect an
incorrect hash.

A deeper issue is that the export/import format does not contain a
hash, so we can't detect such issues early.

Also, I learned that -Wall does not warn about this.

(cherry picked from commit 41c4558afe)
2017-05-01 11:26:27 +02:00
Eelco Dolstra 5d59ec86d4 Bump version to 1.11.9 2017-04-25 16:58:00 +02:00
Eelco Dolstra 1ab5cc0f5a Merge pull request #1333 from copumpkin/fix-schema-version
Ensure that curSchema is set before opening the DB
2017-04-14 20:58:06 +02:00
Dan Peebles ad9e6037a4 Ensure that curSchema is set before opening the DB
Without this, it's possible to get `curSchema = 0` which then causes us
not to trigger the branch that maintains forward compatibility with the
1.12 schema.

Fixes #1332
2017-04-14 14:44:28 -04:00
Neil Mayhew de4e706776 Propagate NIX_BUILD_CORES to nix-shell environments 2017-04-08 21:31:44 -06:00
Shea Levy eb6f3d5159 builtins.exec: Make the argument just a list
(cherry picked from commit 3ecb09a40a)
2017-03-31 11:58:51 -04:00
Shea Levy 967f231981 Add exec primop behind allow-unsafe-native-code-during-evaluation.
Execute a given program with the (optional) given arguments as the
user running the evaluation, parsing stdout as an expression to be
evaluated.

There are many use cases for nix that would benefit from being able to
run arbitrary code during evaluation, including but not limited to:

* Automatic git fetching to get a sha256 from a git revision
* git rev-parse HEAD
* Automatic extraction of information from build specifications from
  other tools, particularly language-specific package managers like
  cabal or npm
* Secrets decryption (e.g. with nixops)
* Private repository fetching

Ideally, we would add this functionality in a more principled way to
nix, but in the mean time 'builtins.exec' can be used to get these
tasks done.

The primop is only available when the
'allow-unsafe-native-code-during-evaluation' nix option is true. That
flag also enables the 'importNative' primop, which is strictly more
powerful but less convenient (since it requires compiling a plugin
against the running version of nix).

(cherry picked from commit 0bb8db257d)
2017-03-30 10:08:38 -04:00
Daiderd Jordan 773313591f use std::tuple for ValueMap allocator
(cherry picked from commit 023217f07c)
2017-03-24 23:31:47 +01:00
Eelco Dolstra 206b61b074 useChroot -> useSandbox
(cherry picked from commit 8d7c6644c5)
2017-03-21 13:50:47 +01:00
Eelco Dolstra 201ad43d1a Bump version to 1.11.8 2017-03-21 11:34:29 +01:00
Vladimír Čunát 9504bcf03c nix-env: respect meta.outputsToInstall
Discussed on https://github.com/NixOS/nixpkgs/pull/12653#discussion_r51601849

(cherry picked from commit 03cbb9ad59)
2017-03-21 11:33:21 +01:00
Eelco Dolstra d8638ae6b4 Merge pull request #1259 from shlevy/1.11-netrc
Backport netrc-file option to 1.11
2017-03-20 17:37:54 +01:00
Shea Levy a437b8c5d2 download-from-binary-cache.pl: Respect netrc-file 2017-03-05 17:40:17 -05:00
Shea Levy 37bdb9d7f2 Backport netrc-file option to 1.11 2017-03-04 09:55:49 -05:00
Domen Kožar 13fe83dc8e bail out if macOS 10.9 or lower is used during installer
(cherry picked from commit 48d4a23aa0)
Signed-off-by: Domen Kožar <domen@dev.si>

cc @edolstra
2017-02-23 15:12:35 +01:00
Eelco Dolstra 697b5755e4 Merge pull request #1247 from veprbl/ssl_fix2
use --cacert instead of --capath
2017-02-22 20:14:47 +01:00
Dmitry Kalinkin 06b46f646d use --cacert instead of --capath
This forces curl to use nix bundled crt instead of picking one up from
system.

Fixes: 142c77711 ('Propagate path of CA bundle to curl child processes')
2017-02-22 14:04:47 -05:00
Eelco Dolstra 2936a8d1ca Bump version 2017-02-22 12:43:54 +01:00
Eelco Dolstra 142c777112 Propagate path of CA bundle to curl child processes 2017-02-22 12:43:13 +01:00
Eelco Dolstra 41230dd463 SSL_CERT_FILE -> NIX_SSL_CERT_FILE
This prevents collisions with the "native" OpenSSL, in particular on
OS X.

Fixes #921.

(cherry picked from commit fb2dd32100)
2017-02-22 12:30:45 +01:00
Eelco Dolstra 8ca944e009 Add forward compatibility with the Nix 1.12 database schema 2017-02-22 12:19:42 +01:00
Eelco Dolstra 1fa2c86db5 Bump version to 1.11.6 2017-01-13 11:02:55 +01:00
Eelco Dolstra 0085fc6d3e Merge pull request #1178 from dezgeg/bash-env-breakage-stable
nix-shell: Fix 'nix-shell --command' doing nothing without TTY (stable backport)
2017-01-09 16:47:14 +01:00
Tuomas Tynkkynen a4ad1ffa25 nix-shell: Fix 'nix-shell --command' doing nothing without TTY
Regression from 8bddc3d4 ("Fix early removal of rc-file for nix-shell").
The removal of BASH_ENV causes nothing to be executed by bash if it
detects itself in a non-interactive context. Instead, just
use the same condition used by bash to launch bash differently.

According to bash sources, the condition (stdin and stder both
must be TTYs) is specified by POSIX so this should be pretty
safe to rely on.

Fixes #1171 on 1.11.
2017-01-07 19:36:45 +02:00
Scott R. Parish 0c67498be9 fix "nix-build" examples
The existing "nix-build" examples were failing:

  error: cannot auto-call a function that has an argument without a default value (‘system’)

Thanks to @groxxda on irc for pointing out the fix!

Updated to completely remove unneeded path argument, suggested by @joachifm

Updated to remove other occurences of `all-packages.nix` from files as
suggested by @domenkozar

(cherry picked from commit 96d3534a9e)
2017-01-03 16:38:48 +01:00
Eelco Dolstra d39f51fa34 Bump version number 2017-01-02 15:18:50 +01:00
Dmitry Kalinkin 5bf9689e0c override rx directory permissions in deletePath()
This fixes instantiation of pythonPackages.pytest that produces a
directory with less permissions during one of it's tests that leads to
a nix error like:

error: opening directory ‘/tmp/nix-build-python2.7-pytest-2.9.2.drv-0/pytest-of-user/pytest-0/testdir/test_cache_failure_warns0/.cache’: Permission denied
(cherry picked from commit f91748ba73)
2016-12-19 14:28:57 +01:00
Shea Levy 612c77a399 Update darwin build for optional sandbox paths
Fixes #1132

(cherry picked from commit 8bf378e999)
2016-11-22 10:46:26 +01:00
Eelco Dolstra 2eb840eefa Support optional sandbox paths
For example, you can now set

  build-sandbox-paths = /dev/nvidiactl?

to specify that /dev/nvidiactl should only be mounted in the sandbox
if it exists in the host filesystem. This is useful e.g. for EC2
images that should support both CUDA and non-CUDA instances.

(cherry picked from commit 18b7363a69)
2016-11-22 10:46:26 +01:00
Shea Levy fd9fc15c0c Add nix.conf options for -k and -K
Fixes #1084
2016-10-27 12:51:27 -04:00
Shea Levy 8bddc3d406 Fix early removal of rc-file for nix-shell
BASH_ENV causes all non-interactive shells called via eg. /etc/bashrc to
remove the rc-file before the main shell gets to run it. Completion
scripts will often do this. Fixes #976.

Adapted from and fixes #1034.
2016-09-08 09:10:50 -04:00
Eelco Dolstra fb577a431f Fix Debian 8 build
http://hydra.nixos.org/build/36462150
(cherry picked from commit 88b79cd55c)
2016-09-06 13:12:53 +02:00
Eelco Dolstra 8c353ea698 Fix incomplete cherry-pick in 0eb8bbb31e 2016-09-06 13:12:04 +02:00
Eelco Dolstra a566927003 Fix OOM in the installer test
http://hydra.nixos.org/build/36462209
(cherry picked from commit 1b5b654fe2)
2016-09-06 13:07:54 +02:00
Dan Peebles f2495212b1 Kill the temporary darwin-specific channel
The issues have been resolved upstream in the main nixpkgs channel now
(cherry picked from commit c89783b6a7)
2016-09-06 13:06:51 +02:00
Eelco Dolstra 7ead75ca85 Drop all distros that are not down with C++11
(cherry picked from commit 7251a81bde)
2016-09-06 13:06:40 +02:00
Eelco Dolstra f668fdb026 Fix nix-copy-closure test on 16.03
(cherry picked from commit b86555aa2b)
2016-09-06 12:59:48 +02:00
aszlig 16e88f0b5a Don't hardcode docbook XSL namespace URL
Docbook XSL got updated to version 1.79.1 in NixOS/nixpkgs@fb893a8 and
we're still referring to the hardcoded previous version.

So instead of just updating this to 1.79.1 we're going to use "current"
in the hope that this won't happen again.

I have tested this by building the manual under Nix(OS) but I haven't
tested this in a non-Nix environment, so I'm not sure whether this could
have implications.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @edolstra
(cherry picked from commit 72fb2ccfa1)
2016-09-06 12:52:49 +02:00
Eelco Dolstra 2179dd3e5b Bump version 2016-09-02 14:58:47 +02:00
Eelco Dolstra efa2e451fb Make the search path lazier with non-fatal errors
Thus, -I / $NIX_PATH entries are now downloaded only when they are
needed for evaluation. An error to download an entry is a non-fatal
warning (just like non-existant paths).

This does change the semantics of builtins.nixPath, which now returns
the original, rather than resulting path. E.g., before we had

  [ { path = "/nix/store/hgm3yxf1lrrwa3z14zpqaj5p9vs0qklk-nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

but now

  [ { path = "https://nixos.org/channels/nixos-16.03/nixexprs.tar.xz"; prefix = "nixpkgs"; } ... ]

Fixes #792.

(cherry picked from commit 363f37d084)
2016-09-02 11:55:14 +02:00
Shea Levy dea9de79b7 callFunction: Copy functors to the heap
Normally it's impossible to take a reference to the function passed to
callFunction, so some callers (e.g. ExprApp::eval) allocate that value
on the stack. For functors, a reference to the functor itself may be
kept, so we need to have it on the heap.

Fixes #1045

(cherry picked from commit 9fa21765e7)
2016-08-29 07:40:03 -04:00
Shea Levy 2ac966a464 Bump version to 1.11.3 2016-08-22 10:20:41 -04:00
Eelco Dolstra 6a493a7c44 launchd: Set $SSL_CERT_FILE
Otherwise in particular https://cache.nixos.org won't work in the
daemon.

(cherry picked from commit eff80419c7)
2016-08-19 13:11:03 +02:00
Shea Levy 7bb4d028a8 builtins.fetch{url,tarball}: Allow name attribute
(cherry picked from commit d52d391164)
2016-08-15 07:42:51 -04:00
Shea Levy 66151dc154 Respect --keep-going when a substituter fails.
Fixes #977

(cherry picked from commit 18b0808475)
2016-07-23 13:22:52 -04:00
Domen Kožar 0eb8bbb31e add Ubuntu 16.03 .deb builds
(cherry picked from commit bf386de9f2)
Signed-off-by: Domen Kožar <domen@dev.si>
2016-05-26 15:55:51 +01:00
Eelco Dolstra 437d3cdc7a Bump version to 1.11.2 2016-02-10 12:10:03 +01:00
Alex Cruice 0322c92560 Check shell profile is writeable before modifying
The `set -e` at the top of the script causes the installation to fail to
complete if the shell profile is not writeable. Checking file existence
only is not enough.

(cherry picked from commit ad0dc41899)
2016-02-10 12:04:47 +01:00
Brian McKenna 1852f7dbf3 Fix broken call to shellwords
nix-shell shebangs were broken by 9018deab

(cherry picked from commit 3baf8be1d1)
2016-02-10 12:04:18 +01:00
Eelco Dolstra 08500066ea Revert "Revert "next try for "don't abort when given unmatched '}' with 'start-condition stack underflow'. This fixes #751"""
This reverts commit b669d3d2e8.

(cherry picked from commit 5d8b7eb3e1)
2016-02-10 12:04:08 +01:00
342 changed files with 14453 additions and 20838 deletions

No files matched your search

-16
View File
@@ -1,16 +0,0 @@
((c++-mode . (
(c-file-style . "k&r")
(c-basic-offset . 4)
(indent-tabs-mode . nil)
(tab-width . 4)
(show-trailing-whitespace . t)
(indicate-empty-lines . t)
(eval . (c-set-offset 'innamespace 0))
(eval . (c-set-offset 'defun-open 0))
(eval . (c-set-offset 'inline-open 0))
(eval . (c-set-offset 'arglist-intro '+))
(eval . (c-set-offset 'arglist-cont 0))
(eval . (c-set-offset 'arglist-cont-nonempty '+))
(eval . (c-set-offset 'substatement-open 0))
(eval . (c-set-offset 'access-label '-))
)))
-26
View File
@@ -1,26 +0,0 @@
# EditorConfig configuration for nix
# http://EditorConfig.org
# Top-most EditorConfig file
root = true
# Unix-style newlines with a newline ending every file, utf-8 charset
[*]
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
charset = utf-8
# Match nix files, set indent to spaces with width of two
[*.nix]
indent_style = space
indent_size = 2
# Match c++/shell/perl, set indent to spaces with width of four
[*.{hpp,cc,hh,sh,pl}]
indent_style = space
indent_size = 4
# Match diffs, avoid to trim trailing whitespace
[*.{diff,patch}]
trim_trailing_whitespace = false
+31 -22
View File
@@ -1,5 +1,4 @@
Makefile.config
perl/Makefile.config
# /
/aclocal.m4
@@ -35,10 +34,32 @@ perl/Makefile.config
# /scripts/
/scripts/nix-profile.sh
/scripts/nix-profile-daemon.sh
/scripts/nix-pull
/scripts/nix-push
/scripts/nix-switch
/scripts/nix-collect-garbage
/scripts/nix-prefetch-url
/scripts/nix-install-package
/scripts/nix-channel
/scripts/nix-build
/scripts/nix-copy-closure
/scripts/nix-generate-patches
/scripts/NixConfig.pm
/scripts/NixManifest.pm
/scripts/GeneratePatches.pm
/scripts/download-using-manifests.pl
/scripts/copy-from-other-stores.pl
/scripts/download-from-binary-cache.pl
/scripts/find-runtime-roots.pl
/scripts/build-remote.pl
/scripts/nix-reduce-build
/scripts/nix-http-export.cgi
# /src/bsdiff-4.3/
/src/bsdiff-4.3/bsdiff
/src/bsdiff-4.3/bspatch
# /src/libexpr/
/src/libexpr/lexer-tab.cc
/src/libexpr/lexer-tab.hh
@@ -48,38 +69,28 @@ perl/Makefile.config
/src/libexpr/nix.tbl
# /src/libstore/
/src/libstore/*.gen.hh
/src/nix/nix
/src/libstore/schema.sql.hh
# /src/nix-env/
/src/nix-env/nix-env
# /src/nix-hash/
/src/nix-hash/nix-hash
# /src/nix-instantiate/
/src/nix-instantiate/nix-instantiate
# /src/nix-log2xml/
/src/nix-log2xml/nix-log2xml
# /src/nix-store/
/src/nix-store/nix-store
/src/nix-prefetch-url/nix-prefetch-url
# /src/nix-daemon/
/src/nix-daemon/nix-daemon
/src/nix-collect-garbage/nix-collect-garbage
# /src/nix-channel/
/src/nix-channel/nix-channel
# /src/buildenv/
/src/buildenv/buildenv
# /src/nix-build/
/src/nix-build/nix-build
/src/nix-copy-closure/nix-copy-closure
/src/build-remote/build-remote
# /src/download-via-ssh/
/src/download-via-ssh/download-via-ssh
# /tests/
/tests/test-tmp
@@ -99,8 +110,6 @@ perl/Makefile.config
/misc/systemd/nix-daemon.socket
/misc/upstart/nix-daemon.conf
inst/
*.a
*.o
*.so
+2
View File
@@ -0,0 +1,2 @@
os: osx
script: ./tests/install-darwin.sh
+229
View File
@@ -0,0 +1,229 @@
Copyright 1994, 1995, 1996, 1999, 2000, 2001, 2002 Free Software
Foundation, Inc.
This file is free documentation; the Free Software Foundation gives
unlimited permission to copy, distribute and modify it.
Basic Installation
==================
These are generic installation instructions.
The `configure' shell script attempts to guess correct values for
various system-dependent variables used during compilation. It uses
those values to create a `Makefile' in each directory of the package.
It may also create one or more `.h' files containing system-dependent
definitions. Finally, it creates a shell script `config.status' that
you can run in the future to recreate the current configuration, and a
file `config.log' containing compiler output (useful mainly for
debugging `configure').
It can also use an optional file (typically called `config.cache'
and enabled with `--cache-file=config.cache' or simply `-C') that saves
the results of its tests to speed up reconfiguring. (Caching is
disabled by default to prevent problems with accidental use of stale
cache files.)
If you need to do unusual things to compile the package, please try
to figure out how `configure' could check whether to do them, and mail
diffs or instructions to the address given in the `README' so they can
be considered for the next release. If you are using the cache, and at
some point `config.cache' contains results you don't want to keep, you
may remove or edit it.
The file `configure.ac' (or `configure.in') is used to create
`configure' by a program called `autoconf'. You only need
`configure.ac' if you want to change it or regenerate `configure' using
a newer version of `autoconf'.
The simplest way to compile this package is:
1. `cd' to the directory containing the package's source code and type
`./configure' to configure the package for your system. If you're
using `csh' on an old version of System V, you might need to type
`sh ./configure' instead to prevent `csh' from trying to execute
`configure' itself.
Running `configure' takes awhile. While running, it prints some
messages telling which features it is checking for.
2. Type `make' to compile the package.
3. Optionally, type `make check' to run any self-tests that come with
the package.
4. Type `make install' to install the programs and any data files and
documentation.
5. You can remove the program binaries and object files from the
source code directory by typing `make clean'. To also remove the
files that `configure' created (so you can compile the package for
a different kind of computer), type `make distclean'. There is
also a `make maintainer-clean' target, but that is intended mainly
for the package's developers. If you use it, you may have to get
all sorts of other programs in order to regenerate files that came
with the distribution.
Compilers and Options
=====================
Some systems require unusual options for compilation or linking that
the `configure' script does not know about. Run `./configure --help'
for details on some of the pertinent environment variables.
You can give `configure' initial values for configuration parameters
by setting variables in the command line or in the environment. Here
is an example:
./configure CC=c89 CFLAGS=-O2 LIBS=-lposix
*Note Defining Variables::, for more details.
Compiling For Multiple Architectures
====================================
You can compile the package for more than one kind of computer at the
same time, by placing the object files for each architecture in their
own directory. To do this, you must use a version of `make' that
supports the `VPATH' variable, such as GNU `make'. `cd' to the
directory where you want the object files and executables to go and run
the `configure' script. `configure' automatically checks for the
source code in the directory that `configure' is in and in `..'.
If you have to use a `make' that does not support the `VPATH'
variable, you have to compile the package for one architecture at a
time in the source code directory. After you have installed the
package for one architecture, use `make distclean' before reconfiguring
for another architecture.
Installation Names
==================
By default, `make install' will install the package's files in
`/usr/local/bin', `/usr/local/man', etc. You can specify an
installation prefix other than `/usr/local' by giving `configure' the
option `--prefix=PATH'.
You can specify separate installation prefixes for
architecture-specific files and architecture-independent files. If you
give `configure' the option `--exec-prefix=PATH', the package will use
PATH as the prefix for installing programs and libraries.
Documentation and other data files will still use the regular prefix.
In addition, if you use an unusual directory layout you can give
options like `--bindir=PATH' to specify different values for particular
kinds of files. Run `configure --help' for a list of the directories
you can set and what kinds of files go in them.
If the package supports it, you can cause programs to be installed
with an extra prefix or suffix on their names by giving `configure' the
option `--program-prefix=PREFIX' or `--program-suffix=SUFFIX'.
Optional Features
=================
Some packages pay attention to `--enable-FEATURE' options to
`configure', where FEATURE indicates an optional part of the package.
They may also pay attention to `--with-PACKAGE' options, where PACKAGE
is something like `gnu-as' or `x' (for the X Window System). The
`README' should mention any `--enable-' and `--with-' options that the
package recognizes.
For packages that use the X Window System, `configure' can usually
find the X include and library files automatically, but if it doesn't,
you can use the `configure' options `--x-includes=DIR' and
`--x-libraries=DIR' to specify their locations.
Specifying the System Type
==========================
There may be some features `configure' cannot figure out
automatically, but needs to determine by the type of machine the package
will run on. Usually, assuming the package is built to be run on the
_same_ architectures, `configure' can figure that out, but if it prints
a message saying it cannot guess the machine type, give it the
`--build=TYPE' option. TYPE can either be a short name for the system
type, such as `sun4', or a canonical name which has the form:
CPU-COMPANY-SYSTEM
where SYSTEM can have one of these forms:
OS KERNEL-OS
See the file `config.sub' for the possible values of each field. If
`config.sub' isn't included in this package, then this package doesn't
need to know the machine type.
If you are _building_ compiler tools for cross-compiling, you should
use the `--target=TYPE' option to select the type of system they will
produce code for.
If you want to _use_ a cross compiler, that generates code for a
platform different from the build platform, you should specify the
"host" platform (i.e., that on which the generated programs will
eventually be run) with `--host=TYPE'.
Sharing Defaults
================
If you want to set default values for `configure' scripts to share,
you can create a site shell script called `config.site' that gives
default values for variables like `CC', `cache_file', and `prefix'.
`configure' looks for `PREFIX/share/config.site' if it exists, then
`PREFIX/etc/config.site' if it exists. Or, you can set the
`CONFIG_SITE' environment variable to the location of the site script.
A warning: not all `configure' scripts look for a site script.
Defining Variables
==================
Variables not defined in a site shell script can be set in the
environment passed to `configure'. However, some packages may run
configure again during the build, and the customized values of these
variables may be lost. In order to avoid this problem, you should set
them in the `configure' command line, using `VAR=value'. For example:
./configure CC=/usr/local2/bin/gcc
will cause the specified gcc to be used as the C compiler (unless it is
overridden in the site shell script).
`configure' Invocation
======================
`configure' recognizes the following options to control how it
operates.
`--help'
`-h'
Print a summary of the options to `configure', and exit.
`--version'
`-V'
Print the version of Autoconf used to generate the `configure'
script, and exit.
`--cache-file=FILE'
Enable the cache: use and save the results of the tests in FILE,
traditionally `config.cache'. FILE defaults to `/dev/null' to
disable caching.
`--config-cache'
`-C'
Alias for `--cache-file=config.cache'.
`--quiet'
`--silent'
`-q'
Do not print messages saying which checks are being made. To
suppress all normal output, redirect it to `/dev/null' (any error
messages will still be shown).
`--srcdir=DIR'
Look for the package's source code in directory DIR. Usually
`configure' can determine that directory automatically.
`configure' also accepts some other, not widely useful, options. Run
`configure --help' for more details.
+6 -8
View File
@@ -5,19 +5,17 @@ makefiles = \
src/libstore/local.mk \
src/libmain/local.mk \
src/libexpr/local.mk \
src/nix/local.mk \
src/nix-hash/local.mk \
src/nix-store/local.mk \
src/nix-instantiate/local.mk \
src/nix-env/local.mk \
src/nix-daemon/local.mk \
src/nix-collect-garbage/local.mk \
src/nix-copy-closure/local.mk \
src/download-via-ssh/local.mk \
src/nix-log2xml/local.mk \
src/nix-prefetch-url/local.mk \
src/buildenv/local.mk \
src/resolve-system-dependencies/local.mk \
src/nix-channel/local.mk \
src/nix-build/local.mk \
src/build-remote/local.mk \
src/bsdiff-4.3/local.mk \
perl/local.mk \
scripts/local.mk \
corepkgs/local.mk \
misc/systemd/local.mk \
@@ -27,7 +25,7 @@ makefiles = \
doc/manual/local.mk \
tests/local.mk
GLOBAL_CXXFLAGS += -std=c++14 -g -Wall -include config.h
GLOBAL_CXXFLAGS += -std=c++0x -g -Wall
-include Makefile.config
+6 -7
View File
@@ -3,22 +3,20 @@ CC = @CC@
CFLAGS = @CFLAGS@
CXX = @CXX@
CXXFLAGS = @CXXFLAGS@
ENABLE_S3 = @ENABLE_S3@
HAVE_SODIUM = @HAVE_SODIUM@
HAVE_READLINE = @HAVE_READLINE@
LIBCURL_LIBS = @LIBCURL_LIBS@
OPENSSL_LIBS = @OPENSSL_LIBS@
PACKAGE_NAME = @PACKAGE_NAME@
PACKAGE_VERSION = @PACKAGE_VERSION@
SODIUM_LIBS = @SODIUM_LIBS@
LIBLZMA_LIBS = @LIBLZMA_LIBS@
SQLITE3_LIBS = @SQLITE3_LIBS@
bash = @bash@
bindir = @bindir@
bro = @bro@
lsof = @lsof@
bsddiff_compat_include = @bsddiff_compat_include@
curl = @curl@
datadir = @datadir@
datarootdir = @datarootdir@
dblatex = @dblatex@
docdir = @docdir@
exec_prefix = @exec_prefix@
includedir = @includedir@
@@ -26,11 +24,12 @@ libdir = @libdir@
libexecdir = @libexecdir@
localstatedir = @localstatedir@
mandir = @mandir@
perl = @perl@
perlbindings = @perlbindings@
perllibdir = @perllibdir@
pkglibdir = $(libdir)/$(PACKAGE_NAME)
prefix = @prefix@
sandbox_shell = @sandbox_shell@
storedir = @storedir@
sysconfdir = @sysconfdir@
doc_generate = @doc_generate@
xmllint = @xmllint@
xsltproc = @xsltproc@
+10
View File
@@ -0,0 +1,10 @@
Nix is a purely functional package manager. For installation and
usage instructions, please read the manual, which can be found in
`docs/manual/manual.html', and additionally at the Nix website at
<http://nixos.org/>.
Acknowledgments
This product includes software developed by the OpenSSL Project for
use in the OpenSSL Toolkit (http://www.OpenSSL.org/).
-22
View File
@@ -1,22 +0,0 @@
Nix, the purely functional package manager
------------------------------------------
Nix is a new take on package management that is fairly unique. Because of its
purity aspects, a lot of issues found in traditional package managers don't
appear with Nix.
To find out more about the tool, usage and installation instructions, please
read the manual, which is available on the Nix website at
<http://nixos.org/nix/manual>.
## Contributing
Take a look at the [Hacking Section](http://nixos.org/nix/manual/#chap-hacking)
of the manual. It helps you to get started with building Nix from source.
## License
Nix is released under the LGPL v2.1
This product includes software developed by the OpenSSL Project for
use in the [OpenSSL Toolkit](http://www.OpenSSL.org/).
+95 -31
View File
@@ -1,5 +1,5 @@
AC_INIT(nix, m4_esyscmd([bash -c "echo -n $(cat ./version)$VERSION_SUFFIX"]))
AC_CONFIG_SRCDIR(README.md)
AC_CONFIG_SRCDIR(README)
AC_CONFIG_AUX_DIR(config)
AC_PROG_SED
@@ -16,14 +16,12 @@ AC_ARG_WITH(system, AC_HELP_STRING([--with-system=SYSTEM],
machine_name="i686";;
amd64)
machine_name="x86_64";;
armv6|armv7)
machine_name="${host_cpu}l";;
*)
machine_name="$host_cpu";;
esac
case "$host_os" in
linux-gnu*|linux-musl*)
linux-gnu*)
# For backward compatibility, strip the `-gnu' part.
system="$machine_name-linux";;
*)
@@ -61,7 +59,6 @@ CFLAGS=
CXXFLAGS=
AC_PROG_CC
AC_PROG_CXX
AX_CXX_COMPILE_STDCXX_11
# Use 64-bit file system calls so that we can support files > 2 GiB.
@@ -80,7 +77,7 @@ static char buf[1024];]],
AC_LANG_POP(C++)
AC_CHECK_FUNCS([statvfs pipe2])
AC_CHECK_FUNCS([statvfs])
# Check for lutimes, optionally used for changing the mtime of
@@ -106,6 +103,11 @@ AC_CHECK_HEADERS([locale])
AC_LANG_POP(C++)
# Check for <err.h>.
AC_CHECK_HEADER([err.h], [], [bsddiff_compat_include="-Icompat-include"])
AC_SUBST([bsddiff_compat_include])
AC_DEFUN([NEED_PROG],
[
AC_PATH_PROG($1, $2)
@@ -114,21 +116,39 @@ if test -z "$$1"; then
fi
])
NEED_PROG(curl, curl)
NEED_PROG(bash, bash)
NEED_PROG(patch, patch)
AC_PATH_PROG(xmllint, xmllint, false)
AC_PATH_PROG(xsltproc, xsltproc, false)
AC_PATH_PROG(flex, flex, false)
AC_PATH_PROG(bison, bison, false)
NEED_PROG(perl, perl)
NEED_PROG(sed, sed)
NEED_PROG(tar, tar)
NEED_PROG(bzip2, bzip2)
NEED_PROG(gzip, gzip)
NEED_PROG(xz, xz)
AC_PATH_PROG(dot, dot)
AC_PATH_PROG(dblatex, dblatex)
AC_PATH_PROG(pv, pv, pv)
AC_PATH_PROG(bro, bro, bro)
AC_PATH_PROG(lsof, lsof, lsof)
# Test that Perl has the open/fork feature (Perl 5.8.0 and beyond).
AC_MSG_CHECKING([whether Perl is recent enough])
if ! $perl -e 'open(FOO, "-|", "true"); while (<FOO>) { print; }; close FOO or die;'; then
AC_MSG_RESULT(no)
AC_MSG_ERROR([Your Perl version is too old. Nix requires Perl 5.8.0 or newer.])
fi
AC_MSG_RESULT(yes)
# Figure out where to install Perl modules.
AC_MSG_CHECKING([for the Perl installation prefix])
perlversion=$($perl -e 'use Config; print $Config{version};')
perlarchname=$($perl -e 'use Config; print $Config{archname};')
AC_SUBST(perllibdir, [${libdir}/perl5/site_perl/$perlversion/$perlarchname])
AC_MSG_RESULT($perllibdir)
NEED_PROG(cat, cat)
@@ -146,6 +166,10 @@ AC_SUBST(storedir)
# Look for OpenSSL, a required dependency.
AC_PATH_PROG(openssl, openssl, openssl) # if not found, call openssl in $PATH
AC_SUBST(openssl)
AC_DEFINE_UNQUOTED(OPENSSL_PATH, ["$openssl"], [Path of the OpenSSL binary])
PKG_CHECK_MODULES([OPENSSL], [libcrypto], [CXXFLAGS="$OPENSSL_CFLAGS $CXXFLAGS"])
@@ -183,15 +207,6 @@ if test "$sys_name" = linux; then
fi
# Look for aws-cpp-sdk-s3.
AC_LANG_PUSH(C++)
AC_CHECK_HEADERS([aws/s3/S3Client.h],
[AC_DEFINE([ENABLE_S3], [1], [Whether to enable S3 support via aws-cpp-sdk-s3.])
enable_s3=1], [enable_s3=])
AC_SUBST(ENABLE_S3, [$enable_s3])
AC_LANG_POP(C++)
# Whether to use the Boehm garbage collector.
AC_ARG_ENABLE(gc, AC_HELP_STRING([--enable-gc],
[enable garbage collection in the Nix expression evaluator (requires Boehm GC) [default=no]]),
@@ -203,25 +218,80 @@ if test "$gc" = yes; then
fi
# Check for the required Perl dependencies (DBI, DBD::SQLite and WWW::Curl).
perlFlags="-I$perllibdir"
AC_ARG_WITH(dbi, AC_HELP_STRING([--with-dbi=PATH],
[prefix of the Perl DBI library]),
perlFlags="$perlFlags -I$withval")
AC_ARG_WITH(dbd-sqlite, AC_HELP_STRING([--with-dbd-sqlite=PATH],
[prefix of the Perl DBD::SQLite library]),
perlFlags="$perlFlags -I$withval")
AC_ARG_WITH(www-curl, AC_HELP_STRING([--with-www-curl=PATH],
[prefix of the Perl WWW::Curl library]),
perlFlags="$perlFlags -I$withval")
AC_MSG_CHECKING([whether DBD::SQLite works])
if ! $perl $perlFlags -e 'use DBI; use DBD::SQLite;' 2>&5; then
AC_MSG_RESULT(no)
AC_MSG_FAILURE([The Perl modules DBI and/or DBD::SQLite are missing.])
fi
AC_MSG_RESULT(yes)
AC_MSG_CHECKING([whether WWW::Curl works])
if ! $perl $perlFlags -e 'use WWW::Curl;' 2>&5; then
AC_MSG_RESULT(no)
AC_MSG_FAILURE([The Perl module WWW::Curl is missing.])
fi
AC_MSG_RESULT(yes)
AC_SUBST(perlFlags)
# Check for otool, an optional dependency on Darwin.
AC_PATH_PROG(otool, otool)
AC_MSG_CHECKING([that otool works])
case $host_os in
darwin*)
if test -z "$otool" || ! $otool --version 2>/dev/null; then
AC_MSG_RESULT(no)
AC_MSG_ERROR([Can't get version from otool; do you need to install developer tools?])
fi
AC_MSG_RESULT(yes)
;;
*)
AC_MSG_RESULT(not needed)
;;
esac
# Whether to build the Perl bindings
AC_MSG_CHECKING([whether to build the Perl bindings])
AC_ARG_ENABLE(perl-bindings, AC_HELP_STRING([--enable-perl-bindings],
[whether to build the Perl bindings (recommended) [default=yes]]),
perlbindings=$enableval, perlbindings=yes)
if test "$enable_shared" = no; then
# Perl bindings require shared libraries.
perlbindings=no
fi
AC_SUBST(perlbindings)
AC_MSG_RESULT($perlbindings)
AC_ARG_ENABLE(init-state, AC_HELP_STRING([--disable-init-state],
[do not initialise DB etc. in `make install']),
init_state=$enableval, init_state=yes)
#AM_CONDITIONAL(INIT_STATE, test "$init_state" = "yes")
# documentation generation switch
AC_ARG_ENABLE(doc-gen, AC_HELP_STRING([--disable-doc-gen],
[disable documentation generation]),
doc_generate=$enableval, doc_generate=yes)
AC_SUBST(doc_generate)
# Setuid installations.
AC_CHECK_FUNCS([setresuid setreuid lchown])
# Nice to have, but not essential.
AC_CHECK_FUNCS([strsignal posix_fallocate sysconf])
AC_CHECK_FUNCS([strsignal posix_fallocate nanosleep sysconf])
# This is needed if bzip2 is a static library, and the Nix libraries
@@ -247,12 +317,6 @@ fi
AC_SUBST(tarFlags)
AC_ARG_WITH(sandbox-shell, AC_HELP_STRING([--with-sandbox-shell=PATH],
[path of a statically-linked shell to use as /bin/sh in sandboxes]),
sandbox_shell=$withval)
AC_SUBST(sandbox_shell)
# Expand all variables in config.status.
test "$prefix" = NONE && prefix=$ac_default_prefix
test "$exec_prefix" = NONE && exec_prefix='${prefix}'
+2 -1
View File
@@ -5,7 +5,8 @@ with import <nix/config.nix>;
derivation {
name = "user-environment";
system = builtins.currentSystem;
builder = nixLibexecDir + "/nix/buildenv";
builder = perl;
args = [ "-w" ./buildenv.pl ];
inherit manifest;
+168
View File
@@ -0,0 +1,168 @@
use strict;
use Cwd;
use IO::Handle;
use utf8;
STDOUT->autoflush(1);
my $out = $ENV{"out"};
mkdir "$out", 0755 || die "error creating $out";
my $symlinks = 0;
my %priorities;
# For each activated package, create symlinks.
sub createLinks {
my $srcDir = shift;
my $dstDir = shift;
my $priority = shift;
my @srcFiles = glob("$srcDir/*");
foreach my $srcFile (@srcFiles) {
my $baseName = $srcFile;
$baseName =~ s/^.*\///g; # strip directory
my $dstFile = "$dstDir/$baseName";
# The files below are special-cased so that they don't show up
# in user profiles, either because they are useless, or
# because they would cause pointless collisions (e.g., each
# Python package brings its own
# `$out/lib/pythonX.Y/site-packages/easy-install.pth'.)
# Urgh, hacky...
if ($srcFile =~ /\/propagated-build-inputs$/ ||
$srcFile =~ /\/nix-support$/ ||
$srcFile =~ /\/perllocal.pod$/ ||
$srcFile =~ /\/info\/dir$/ ||
$srcFile =~ /\/log$/)
{
# Do nothing.
}
elsif (-d $srcFile) {
lstat $dstFile;
if (-d _) {
createLinks($srcFile, $dstFile, $priority);
}
elsif (-l _) {
my $target = readlink $dstFile or die;
if (!-d $target) {
die "collision between directory ‘$srcFile’ and non-directory ‘$target’";
}
unlink $dstFile or die "error unlinking ‘$dstFile’: $!";
mkdir $dstFile, 0755 ||
die "error creating directory ‘$dstFile’: $!";
createLinks($target, $dstFile, $priorities{$dstFile});
createLinks($srcFile, $dstFile, $priority);
}
else {
symlink($srcFile, $dstFile) ||
die "error creating link ‘$dstFile’: $!";
$priorities{$dstFile} = $priority;
$symlinks++;
}
}
else {
if (-l $dstFile) {
my $target = readlink $dstFile;
my $prevPriority = $priorities{$dstFile};
die("collision between ‘$srcFile’ and ‘$target’; " .
"use ‘nix-env --set-flag priority NUMBER PKGNAME’ " .
"to change the priority of one of the conflicting packages\n")
if $prevPriority == $priority;
next if $prevPriority < $priority;
unlink $dstFile or die;
}
symlink($srcFile, $dstFile) ||
die "error creating link ‘$dstFile’: $!";
$priorities{$dstFile} = $priority;
$symlinks++;
}
}
}
my %done;
my %postponed;
sub addPkg;
sub addPkg {
my $pkgDir = shift;
my $priority = shift;
return if (defined $done{$pkgDir});
$done{$pkgDir} = 1;
# print "symlinking $pkgDir\n";
createLinks("$pkgDir", "$out", $priority);
my $propagatedFN = "$pkgDir/nix-support/propagated-user-env-packages";
if (-e $propagatedFN) {
open PROP, "<$propagatedFN" or die;
my $propagated = <PROP>;
close PROP;
my @propagated = split ' ', $propagated;
foreach my $p (@propagated) {
$postponed{$p} = 1 unless defined $done{$p};
}
}
}
# Convert the stuff we get from the environment back into a coherent
# data type.
my @pkgs;
my @derivations = split ' ', $ENV{"derivations"};
while (scalar @derivations) {
my $active = shift @derivations;
my $priority = shift @derivations;
my $outputs = shift @derivations;
for (my $n = 0; $n < $outputs; $n++) {
my $path = shift @derivations;
push @pkgs,
{ path => $path
, active => $active ne "false"
, priority => int($priority) };
}
}
# Symlink to the packages that have been installed explicitly by the
# user. Process in priority order to reduce unnecessary
# symlink/unlink steps.
@pkgs = sort { $a->{priority} <=> $b->{priority} || $a->{path} cmp $b->{path} } @pkgs;
foreach my $pkg (@pkgs) {
#print $pkg, " ", $pkgs{$pkg}->{priority}, "\n";
addPkg($pkg->{path}, $pkg->{priority}) if $pkg->{active};
}
# Symlink to the packages that have been "propagated" by packages
# installed by the user (i.e., package X declares that it wants Y
# installed as well). We do these later because they have a lower
# priority in case of collisions.
my $priorityCounter = 1000; # don't care about collisions
while (scalar(keys %postponed) > 0) {
my @pkgDirs = keys %postponed;
%postponed = ();
foreach my $pkgDir (sort @pkgDirs) {
addPkg($pkgDir, $priorityCounter++);
}
}
print STDERR "created $symlinks symlinks in user environment\n";
symlink($ENV{"manifest"}, "$out/manifest.nix") or die "cannot create manifest";
+1 -4
View File
@@ -3,6 +3,7 @@ let
let val = builtins.getEnv var; in
if val != "" then val else def;
in rec {
perl = "@perl@";
shell = "@bash@";
coreutils = "@coreutils@";
bzip2 = "@bzip2@";
@@ -13,10 +14,6 @@ in rec {
tr = "@tr@";
nixBinDir = fromEnv "NIX_BIN_DIR" "@bindir@";
nixPrefix = "@prefix@";
nixLibexecDir = fromEnv "NIX_LIBEXEC_DIR" "@libexecdir@";
nixLocalstateDir = "@localstatedir@";
nixSysconfDir = "@sysconfdir@";
nixStoreDir = fromEnv "NIX_STORE_DIR" "@storedir@";
# If Nix is installed in the Nix store, then automatically add it as
# a dependency to the core packages. This ensures that they work
+11 -8
View File
@@ -1,20 +1,26 @@
with import <nix/config.nix>;
{ system ? builtins.currentSystem
, url
, outputHash ? ""
, outputHashAlgo ? ""
, md5 ? "", sha1 ? "", sha256 ? ""
, outputHash ?
if sha1 != "" then sha1 else if md5 != "" then md5 else sha256
, outputHashAlgo ?
if sha1 != "" then "sha1" else if md5 != "" then "md5" else "sha256"
, executable ? false
, unpack ? false
, name ? baseNameOf (toString url)
}:
assert (outputHash != "" && outputHashAlgo != "")
|| md5 != "" || sha1 != "" || sha256 != "";
derivation {
builder = "builtin:fetchurl";
# New-style output content requirements.
inherit outputHashAlgo outputHash;
outputHashAlgo = if outputHashAlgo != "" then outputHashAlgo else
if sha256 != "" then "sha256" else if sha1 != "" then "sha1" else "md5";
outputHash = if outputHash != "" then outputHash else
if sha256 != "" then sha256 else if sha1 != "" then sha1 else md5;
outputHashMode = if unpack || executable then "recursive" else "flat";
inherit name system url executable unpack;
@@ -29,7 +35,4 @@ derivation {
# by definition pure.
"http_proxy" "https_proxy" "ftp_proxy" "all_proxy" "no_proxy"
];
# To make "nix-prefetch-url" work.
urls = [ url ];
}
+1 -1
View File
@@ -1,4 +1,4 @@
corepkgs_FILES = buildenv.nix unpack-channel.nix derivation.nix fetchurl.nix imported-drv-to-derivation.nix
corepkgs_FILES = nar.nix buildenv.nix buildenv.pl unpack-channel.nix derivation.nix fetchurl.nix imported-drv-to-derivation.nix
$(foreach file,config.nix $(corepkgs_FILES),$(eval $(call install-data-in,$(d)/$(file),$(datadir)/nix/corepkgs)))
+48
View File
@@ -0,0 +1,48 @@
with import <nix/config.nix>;
let
builder = builtins.toFile "nar.sh"
''
export PATH=${nixBinDir}:${coreutils}
if [ $compressionType = xz ]; then
ext=.xz
compressor="| ${xz} -7"
elif [ $compressionType = bzip2 ]; then
ext=.bz2
compressor="| ${bzip2}"
else
ext=
compressor=
fi
echo "packing ‘$storePath’..."
mkdir $out
dst=$out/tmp.nar$ext
set -o pipefail
eval "nix-store --dump \"$storePath\" $compressor > $dst"
hash=$(nix-hash --flat --type $hashAlgo --base32 $dst)
echo -n $hash > $out/nar-compressed-hash
mv $dst $out/$hash.nar$ext
'';
in
{ storePath, hashAlgo, compressionType }:
derivation {
name = "nar";
system = builtins.currentSystem;
builder = shell;
args = [ "-e" builder ];
inherit storePath hashAlgo compressionType;
# Remote machines may not have ${nixBinDir} or ${coreutils} in the same prefixes
preferLocalBuild = true;
inherit chrootDeps;
}
+1 -3
View File
@@ -15,9 +15,7 @@ let
else
${bzip2} -d < $src | ${tar} xf - ${tarFlags}
fi
if [ * != $channelName ]; then
mv * $out/$channelName
fi
mv * $out/$channelName
if [ -n "$binaryCacheURL" ]; then
mkdir $out/binary-caches
echo -n "$binaryCacheURL" > $out/binary-caches/$channelName
Executable
+18
View File
@@ -0,0 +1,18 @@
#!/usr/bin/env bash
if [ -e tests/test-tmp ]; then
chmod -R u+w tests/test-tmp
rm -rf tests/test-tmp
fi
s=$(type -p nix-shell)
exec $s release.nix -A tarball --command "
unset http_proxy
export NIX_REMOTE=$NIX_REMOTE
export NIX_PATH='$NIX_PATH'
export NIX_BUILD_SHELL=$(type -p bash)
export c=\$configureFlags
exec $s release.nix -A build.$(if [ $(uname -s) = Darwin ]; then echo x86_64-darwin; else echo x86_64-linux; fi) --exclude tarball --command '
configureFlags+=\" \$c --prefix=$(pwd)/inst --sysconfdir=$(pwd)/inst/etc\"
return
'" \
"$@"
@@ -11,9 +11,9 @@ forward Nix builds to other machines over the network. This allows
multiple builds to be performed in parallel (thus improving
performance) and allows Nix to perform multi-platform builds in a
semi-transparent way. For instance, if you perform a build for a
<literal>x86_64-darwin</literal> on an <literal>i686-linux</literal>
<literal>powerpc-darwin</literal> on an <literal>i686-linux</literal>
machine, Nix can automatically forward the build to a
<literal>x86_64-darwin</literal> machine, if available.</para>
<literal>powerpc-darwin</literal> machine, if available.</para>
<para>You can enable distributed builds by setting the environment
variable <envar>NIX_BUILD_HOOK</envar> to point to a program that Nix
@@ -22,12 +22,15 @@ will call whenever it wants to build a derivation. The build hook
will perform it in the usual way if possible, or it can accept it, in
which case it is responsible for somehow getting the inputs of the
build to another machine, doing the build there, and getting the
results back.</para>
results back. The details of the build hook protocol are described in
the documentation of the <link
linkend="envar-build-hook"><envar>NIX_BUILD_HOOK</envar>
variable</link>.</para>
<example xml:id='ex-remote-systems'><title>Remote machine configuration:
<filename>remote-systems.conf</filename></title>
<programlisting>
nix@mcflurry.labs.cs.uu.nl x86_64-darwin /home/nix/.ssh/id_quarterpounder_auto 2
nix@mcflurry.labs.cs.uu.nl powerpc-darwin /home/nix/.ssh/id_quarterpounder_auto 2
nix@scratchy.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy_auto 8 1 kvm
nix@itchy.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy_auto 8 2
nix@poochie.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy_auto 8 2 kvm perf
@@ -39,10 +42,9 @@ purposes. It uses <command>ssh</command> and
<command>nix-copy-closure</command> to copy the build inputs and
outputs and perform the remote build. To use it, you should set
<envar>NIX_BUILD_HOOK</envar> to
<filename><replaceable>prefix</replaceable>/libexec/nix/build-remote</filename>.
<filename><replaceable>prefix</replaceable>/libexec/nix/build-remote.pl</filename>.
You should also define a list of available build machines and point
the environment variable <envar>NIX_REMOTE_SYSTEMS</envar> to
it. <envar>NIX_REMOTE_SYSTEMS</envar> must be an absolute path. An
the environment variable <envar>NIX_REMOTE_SYSTEMS</envar> to it. An
example configuration is shown in <xref linkend='ex-remote-systems'
/>. Each line in the file specifies a machine, with the following
bits of information:
@@ -56,7 +58,7 @@ bits of information:
<filename>~/.ssh/config</filename>.</para></listitem>
<listitem><para>A comma-separated list of Nix platform type
identifiers, such as <literal>x86_64-darwin</literal>. It is
identifiers, such as <literal>powerpc-darwin</literal>. It is
possible for a machine to support multiple platform types, e.g.,
<literal>i686-linux,x86_64-linux</literal>.</para></listitem>
@@ -65,7 +67,7 @@ bits of information:
should not have a passphrase!</para></listitem>
<listitem><para>The maximum number of builds that
<filename>build-remote</filename> will execute in parallel on the
<filename>build-remote.pl</filename> will execute in parallel on the
machine. Typically this should be equal to the number of CPU cores.
For instance, the machine <literal>itchy</literal> in the example
will execute up to 8 builds in parallel.</para></listitem>
@@ -77,7 +79,7 @@ bits of information:
<listitem><para>A comma-separated list of <emphasis>supported
features</emphasis>. If a derivation has the
<varname>requiredSystemFeatures</varname> attribute, then
<filename>build-remote</filename> will only perform the
<filename>build-remote.pl</filename> will only perform the
derivation on a machine that has the specified features. For
instance, the attribute
@@ -100,6 +102,14 @@ requiredSystemFeatures = [ "kvm" ];
</orderedlist>
</para>
You should also set up the environment variable
<envar>NIX_CURRENT_LOAD</envar> to point at a directory (e.g.,
<filename>/var/run/nix/current-load</filename>) that
<filename>build-remote.pl</filename> uses to remember how many builds
it is currently executing remotely. It doesn't look at the actual
load on the remote machine, so if you have multiple instances of Nix
running, they should use the same <envar>NIX_CURRENT_LOAD</envar>
file. Maybe in the future <filename>build-remote.pl</filename> will
look at the actual remote load.</para>
</chapter>
+93 -78
View File
@@ -17,32 +17,12 @@
<refsection><title>Description</title>
<para>Nix reads settings from two configuration files:</para>
<itemizedlist>
<listitem>
<para>The system-wide configuration file
<filename><replaceable>sysconfdir</replaceable>/nix/nix.conf</filename>
(i.e. <filename>/etc/nix/nix.conf</filename> on most systems), or
<filename>$NIX_CONF_DIR/nix.conf</filename> if
<envar>NIX_CONF_DIR</envar> is set.</para>
</listitem>
<listitem>
<para>The user configuration file
<filename>$XDG_CONFIG_HOME/nix/nix.conf</filename>, or
<filename>~/.config/nix/nix.conf</filename> if
<envar>XDG_CONFIG_HOME</envar> is not set.</para>
</listitem>
</itemizedlist>
<para>The configuration files consist of
<literal><replaceable>name</replaceable> =
<para>A number of persistent settings of Nix are stored in the file
<filename><replaceable>sysconfdir</replaceable>/nix/nix.conf</filename>.
This file is a list of <literal><replaceable>name</replaceable> =
<replaceable>value</replaceable></literal> pairs, one per line.
Comments start with a <literal>#</literal> character. Here is an
example configuration file:</para>
Comments start with a <literal>#</literal> character. Here is an example
configuration file:</para>
<programlisting>
gc-keep-outputs = true # Nice for developers
@@ -50,9 +30,8 @@ gc-keep-derivations = true # Idem
env-keep-derivations = false
</programlisting>
<para>You can override settings on the command line using the
<option>--option</option> flag, e.g. <literal>--option gc-keep-outputs
false</literal>.</para>
<para>You can override settings using the <option>--option</option>
flag, e.g. <literal>--option gc-keep-outputs false</literal>.</para>
<para>The following settings are currently available:
@@ -121,9 +100,9 @@ false</literal>.</para>
<listitem><para>This option defines the maximum number of jobs
that Nix will try to build in parallel. The default is
<literal>1</literal>. The special value <literal>auto</literal>
causes Nix to use the number of CPUs in your system. It can be
overridden using the <option
<literal>1</literal>. You should generally set it to the number
of CPUs in your system (e.g., <literal>2</literal> on an Athlon 64
X2). It can be overridden using the <option
linkend='opt-max-jobs'>--max-jobs</option> (<option>-j</option>)
command line switch.</para></listitem>
@@ -332,6 +311,21 @@ false</literal>.</para>
</varlistentry>
<varlistentry><term><literal>build-cache-failure</literal></term>
<listitem><para>If set to <literal>true</literal>, Nix will
“cache” build failures, meaning that it will remember (in its
database) that a derivation previously failed. If you then try to
build the derivation again, Nix will immediately fail rather than
perform the build again. Failures in fixed-output derivations
(such as <function>fetchurl</function> calls) are never cached.
The “failed” status of a derivation can be cleared using
<command>nix-store --clear-failed-paths</command>. By default,
failure caching is disabled.</para></listitem>
</varlistentry>
<varlistentry><term><literal>build-keep-log</literal></term>
<listitem><para>If set to <literal>true</literal> (the default),
@@ -354,16 +348,25 @@ false</literal>.</para>
</varlistentry>
<varlistentry><term><literal>substituters</literal></term>
<varlistentry><term><literal>use-binary-caches</literal></term>
<listitem><para>A list of URLs of substituters, separated by
<listitem><para>If set to <literal>true</literal> (the default),
Nix will check the binary caches specified by
<option>binary-caches</option> and related options to obtain
binary substitutes.</para></listitem>
</varlistentry>
<varlistentry><term><literal>binary-caches</literal></term>
<listitem><para>A list of URLs of binary caches, separated by
whitespace. The default is
<literal>https://cache.nixos.org</literal>.</para></listitem>
</varlistentry>
<!--
<varlistentry><term><literal>binary-caches-files</literal></term>
<listitem><para>A list of names of files that will be read to
@@ -376,28 +379,28 @@ false</literal>.</para>
option to read files created by untrusted users!</para></listitem>
</varlistentry>
-->
<varlistentry><term><literal>trusted-substituters</literal></term>
<varlistentry><term><literal>trusted-binary-caches</literal></term>
<listitem><para>A list of URLs of substituters, separated by
<listitem><para>A list of URLs of binary caches, separated by
whitespace. These are not used by default, but can be enabled by
users of the Nix daemon by specifying <literal>--option
substituters <replaceable>urls</replaceable></literal> on the
binary-caches <replaceable>urls</replaceable></literal> on the
command line. Unprivileged users are only allowed to pass a
subset of the URLs listed in <literal>substituters</literal> and
<literal>trusted-substituters</literal>.</para></listitem>
subset of the URLs listed in <literal>binary-caches</literal> and
<literal>trusted-binary-caches</literal>.</para></listitem>
</varlistentry>
<varlistentry><term><literal>extra-substituters</literal></term>
<varlistentry><term><literal>extra-binary-caches</literal></term>
<listitem><para>Additional binary caches appended to those
specified in <option>substituters</option>. When used by
unprivileged users, untrusted substituters (i.e. those not listed
in <option>trusted-substituters</option>) are silently
specified in <option>binary-caches</option> and
<option>binary-caches-files</option>. When used by unprivileged
users, untrusted binary caches (i.e. those not listed in
<option>trusted-binary-caches</option>) are silently
ignored.</para></listitem>
</varlistentry>
@@ -405,10 +408,9 @@ false</literal>.</para>
<varlistentry><term><literal>signed-binary-caches</literal></term>
<listitem><para>If set to <literal>*</literal> (the default), Nix
will only download binaries if they are signed using one of the
keys listed in <option>binary-cache-public-keys</option>. Set to
the empty string to disable signature checking.</para></listitem>
<listitem><para>If set to <literal>*</literal>, Nix will only
download binaries if they are signed using one of the keys listed
in <option>binary-cache-public-keys</option>.</para></listitem>
</varlistentry>
@@ -424,11 +426,33 @@ false</literal>.</para>
</varlistentry>
<varlistentry><term><literal>http-connections</literal></term>
<varlistentry><term><literal>binary-caches-parallel-connections</literal></term>
<listitem><para>The maximum number of parallel TCP connections
used to fetch files from binary caches and by other downloads. It
defaults to 25. 0 means no limit.</para></listitem>
<listitem><para>The maximum number of parallel HTTP connections
used by the binary cache substituter to get NAR info files. This
number should be high to minimise latency. It defaults to
25.</para></listitem>
</varlistentry>
<varlistentry><term><literal>verify-https-binary-caches</literal></term>
<listitem><para>Whether HTTPS binary caches are required to have a
certificate that can be verified. Defaults to
<literal>true</literal>.</para></listitem>
</varlistentry>
<varlistentry><term><literal>force-manifest</literal></term>
<listitem><para>If this option is set to <literal>false</literal>
(default) and a Nix channel provides both a manifest and a binary
cache, only the binary cache will be used. If set to
<literal>true</literal>, the manifest will be fetched as well.
This is useful if you want to use binary patches (which are
currently not supported by binary caches).</para></listitem>
</varlistentry>
@@ -461,15 +485,15 @@ password <replaceable>my-password</replaceable>
<listitem><para>This option specifies the canonical Nix system
name of the current installation, such as
<literal>i686-linux</literal> or
<literal>x86_64-darwin</literal>. Nix can only build derivations
<literal>powerpc-darwin</literal>. Nix can only build derivations
whose <literal>system</literal> attribute equals the value
specified here. In general, it never makes sense to modify this
value from its default, since you can use it to ‘lie’ about the
platform you are building on (e.g., perform a Mac OS build on a
Linux machine; the result would obviously be wrong). It only
makes sense if the Nix binaries can run on multiple platforms,
e.g., ‘universal binaries’ that run on <literal>x86_64-linux</literal> and
<literal>i686-linux</literal>.</para>
e.g., ‘universal binaries’ that run on <literal>powerpc-darwin</literal> and
<literal>i686-darwin</literal>.</para>
<para>It defaults to the canonical Nix system name detected by
<filename>configure</filename> at build time.</para></listitem>
@@ -515,6 +539,20 @@ password <replaceable>my-password</replaceable>
</varlistentry>
<varlistentry xml:id="conf-log-servers"><term><literal>log-servers</literal></term>
<listitem>
<para>A list of URL prefixes (such as
<literal>http://hydra.nixos.org/log</literal>) from which
<command>nix-store -l</command> will try to fetch build logs if
they’re not available locally.</para>
</listitem>
</varlistentry>
<varlistentry xml:id="conf-trusted-users"><term><literal>trusted-users</literal></term>
<listitem>
@@ -620,29 +658,6 @@ password <replaceable>my-password</replaceable>
</varlistentry>
<varlistentry xml:id="conf-sandbox-dev-shm-size"><term><literal>sandbox-dev-shm-size</literal></term>
<listitem><para>This option determines the maximum size of the
<literal>tmpfs</literal> filesystem mounted on
<filename>/dev/shm</filename> in Linux sandboxes. For the format,
see the description of the <option>size</option> option of
<literal>tmpfs</literal> in
<citerefentry><refentrytitle>mount</refentrytitle><manvolnum>8</manvolnum></citerefentry>. The
default is <literal>50%</literal>.</para></listitem>
</varlistentry>
<varlistentry xml:id="conf-allow-import-from-derivation"><term><literal>allow-import-from-derivation</literal></term>
<listitem><para>By default, Nix allows you to <function>import</function> from a derivation,
allowing building at evaluation time. With this option set to false, Nix will throw an error
when evaluating an expression that uses this feature, allowing users to ensure their evaluation
will not require any builds to take place.</para></listitem>
</varlistentry>
</variablelist>
</para>
+142 -6
View File
@@ -11,12 +11,6 @@
<variablelist xml:id="env-common">
<varlistentry><term><envar>IN_NIX_SHELL</envar></term>
<listitem><para>Indicator that tells if the current environment was set up by
<command>nix-shell</command>.</para></listitem>
</varlistentry>
<varlistentry xml:id="env-NIX_PATH"><term><envar>NIX_PATH</envar></term>
@@ -129,6 +123,15 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
</varlistentry>
<varlistentry><term><envar>NIX_DB_DIR</envar></term>
<listitem><para>Overrides the location of the Nix database (default
<filename><replaceable>$NIX_STATE_DIR</replaceable>/db</filename>, i.e.,
<filename><replaceable>prefix</replaceable>/var/nix/db</filename>).</para></listitem>
</varlistentry>
<varlistentry><term><envar>NIX_CONF_DIR</envar></term>
<listitem><para>Overrides the location of the Nix configuration
@@ -148,6 +151,139 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
</varlistentry>
<varlistentry xml:id="envar-build-hook"><term><envar>NIX_BUILD_HOOK</envar></term>
<listitem>
<para>Specifies the location of the <emphasis>build hook</emphasis>,
which is a program (typically some script) that Nix will call
whenever it wants to build a derivation. This is used to implement
distributed builds<phrase condition="manual"> (see <xref
linkend="chap-distributed-builds" />)</phrase>.</para>
<!--
The protocol by
which the calling Nix process and the build hook communicate is as
follows.
<para>The build hook is called with the following command-line
arguments:
<orderedlist>
<listitem><para>A boolean value <literal>0</literal> or
<literal>1</literal> specifying whether Nix can locally execute
more builds, as per the <link
linkend="opt-max-jobs"><option>- -max-jobs</option> option</link>.
The purpose of this argument is to allow the hook to not have to
maintain bookkeeping for the local machine.</para></listitem>
<listitem><para>The Nix platform identifier for the local machine
(e.g., <literal>i686-linux</literal>).</para></listitem>
<listitem><para>The Nix platform identifier for the derivation,
i.e., its <link linkend="attr-system"><varname>system</varname>
attribute</link>.</para></listitem>
<listitem><para>The store path of the derivation.</para></listitem>
</orderedlist>
</para>
<para>On the basis of this information, and whatever persistent
state the build hook keeps about other machines and their current
load, it has to decide what to do with the build. It should print
out on standard error one of the following responses (terminated by
a newline, <literal>"\n"</literal>):
<variablelist>
<varlistentry><term><literal># decline</literal></term>
<listitem><para>The build hook is not willing or able to perform
the build; the calling Nix process should do the build itself,
if possible.</para></listitem>
</varlistentry>
<varlistentry><term><literal># postpone</literal></term>
<listitem><para>The build hook cannot perform the build now, but
can do so in the future (e.g., because all available build slots
on remote machines are in use). The calling Nix process should
postpone this build until at least one currently running build
has terminated.</para></listitem>
</varlistentry>
<varlistentry><term><literal># accept</literal></term>
<listitem><para>The build hook has accepted the
build.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>After sending <literal># accept</literal>, the hook should
read one line from standard input, which will be the string
<literal>okay</literal>. It can then proceed with the build.
Before sending <literal>okay</literal>, Nix will store in the hook’s
current directory a number of text files that contain information
about the derivation:
<variablelist>
<varlistentry><term><filename>inputs</filename></term>
<listitem><para>The set of store paths that are inputs to the
build process (one per line). These have to be copied
<emphasis>to</emphasis> the remote machine (in addition to the
store derivation itself).</para></listitem>
</varlistentry>
<varlistentry><term><filename>outputs</filename></term>
<listitem><para>The set of store paths that are outputs of the
derivation (one per line). These have to be copied
<emphasis>from</emphasis> the remote machine if the build
succeeds.</para></listitem>
</varlistentry>
<varlistentry><term><filename>references</filename></term>
<listitem><para>The reference graph of the inputs, in the format
accepted by the command <command>nix-store
- -register-validity</command>. It is necessary to run this
command on the remote machine after copying the inputs to inform
Nix on the remote machine that the inputs are valid
paths.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>The hook should copy the inputs to the remote machine,
register the validity of the inputs, perform the remote build, and
copy the outputs back to the local machine. An exit code other than
<literal>0</literal> indicates that the hook has failed. An exit
code equal to 100 means that the remote build failed (as opposed to,
e.g., a network error).</para>
-->
</listitem>
</varlistentry>
<varlistentry xml:id="envar-remote"><term><envar>NIX_REMOTE</envar></term>
<listitem><para>This variable should be set to
+17 -4
View File
@@ -73,10 +73,11 @@ condition="manual">See also <xref linkend="sec-channels"
<listitem><para>Downloads the Nix expressions of all subscribed
channels (or only those included in
<replaceable>names</replaceable> if specified) and makes them the
<replaceable>names</replaceable> if specified), makes them the
default for <command>nix-env</command> operations (by symlinking
them from the directory
<filename>~/.nix-defexpr</filename>).</para></listitem>
them from the directory <filename>~/.nix-defexpr</filename>), and
performs a <command>nix-pull</command> on the manifests of all
channels to make pre-built binaries available.</para></listitem>
</varlistentry>
@@ -180,7 +181,19 @@ following files:</para>
sufficient rights to add binary caches. For instance, in a
multi-user Nix setup, the binary caches provided by the channels
of the root user are used automatically, but caches corresponding
to the channels of non-root users are ignored.</para></listitem>
to the channels of non-root users are ignored. Binary caches can
be created and maintained using
<command>nix-push</command>.</para></listitem>
</varlistentry>
<varlistentry><term><filename>MANIFEST.bz2</filename></term>
<listitem><para>(Deprecated in favour of binary caches.) A
manifest as created by <command>nix-push</command>. Only used if
<filename>binary-cache-url</filename> is not present or if the
<filename>nix.conf</filename> option
<option>force-manifest</option> is set.</para></listitem>
</varlistentry>
@@ -22,12 +22,6 @@
<arg><option>--delete-old</option></arg>
<arg><option>-d</option></arg>
<arg><option>--delete-older-than</option> <replaceable>period</replaceable></arg>
<group choice='opt'>
<arg choice='plain'><option>--print-roots</option></arg>
<arg choice='plain'><option>--print-live</option></arg>
<arg choice='plain'><option>--print-dead</option></arg>
<arg choice='plain'><option>--delete</option></arg>
</group>
<arg><option>--max-freed</option> <replaceable>bytes</replaceable></arg>
<arg><option>--dry-run</option></arg>
</cmdsynopsis>
@@ -22,6 +22,7 @@
<arg choice='plain'><option>--to</option></arg>
<arg choice='plain'><option>--from</option></arg>
</group>
<arg><option>--sign</option></arg>
<arg><option>--gzip</option></arg>
<!--
<arg><option>- -show-progress</option></arg>
@@ -86,6 +87,23 @@ those paths. If this bothers you, use
</varlistentry>
<varlistentry><term><option>--sign</option></term>
<listitem><para>Let the sending machine cryptographically sign the
dump of each path with the key in
<filename><replaceable>sysconfdir</replaceable>/nix/signing-key.sec</filename>.
If the user on the target machine does not have direct access to
the Nix store (i.e., if the target machine has a multi-user Nix
installation), then the target machine will check the dump against
<filename><replaceable>sysconfdir</replaceable>/nix/signing-key.pub</filename>
before unpacking it in its Nix store. This allows secure sharing
of store paths between untrusted users on two machines, provided
that there is a trust relation between the Nix installations on
both machines (namely, they have matching public/secret
keys).</para></listitem>
</varlistentry>
<varlistentry><term><option>--gzip</option></term>
<listitem><para>Enable compression of the SSH
+11 -5
View File
@@ -493,11 +493,17 @@ set returned by calling the function defined in
source:
<screen>
$ nix-env -f '&lt;nixpkgs>' -iA hello --dry-run
$ nix-env -f pkgs/top-level/all-packages.nix -i f-spot --dry-run
(dry run; not doing anything)
installing ‘hello-2.10’
these paths will be fetched (0.04 MiB download, 0.19 MiB unpacked):
/nix/store/wkhdf9jinag5750mqlax6z2zbwhqb76n-hello-2.10
installing `f-spot-0.0.10'
the following derivations will be built:
/nix/store/0g63jv9aagwbgci4nnzs2dkxqz84kdja-libgnomeprintui-2.12.1.tar.bz2.drv
/nix/store/0gfarvxq6sannsdw8a1ir40j1ys2mqb4-ORBit2-2.14.2.tar.bz2.drv
/nix/store/0i9gs5zc04668qiy60ga2rc16abkj7g8-sqlite-2.8.17.drv
<replaceable>...</replaceable>
the following paths will be substituted:
/nix/store/8zbipvm4gp9jfqh9nnk1n3bary1a37gs-perl-XML-Parser-2.34
/nix/store/b8a2bg7gnyvvvjjibp4axg9x1hzkw36c-mono-1.1.4
<replaceable>...</replaceable></screen>
</para>
@@ -1136,7 +1142,7 @@ user environment elements, etc. -->
<listitem><para>Print all of the meta-attributes of the
derivation. This option is only available with
<option>--xml</option> or <option>--json</option>.</para></listitem>
<option>--xml</option>.</para></listitem>
</varlistentry>
@@ -0,0 +1,44 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="sec-nix-generate-patches">
<refmeta>
<refentrytitle>nix-generate-patches</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo class="source">Nix</refmiscinfo>
<refmiscinfo class="version"><xi:include href="../version.txt" parse="text"/></refmiscinfo>
</refmeta>
<refnamediv>
<refname>nix-generate-patches</refname>
<refpurpose>generates binary patches between NAR files</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>nix-generate-patches</command>
<arg choice='plain'><replaceable>NAR-DIR</replaceable></arg>
<arg choice='plain'><replaceable>PATCH-DIR</replaceable></arg>
<arg choice='plain'><replaceable>PATCH-URI</replaceable></arg>
<arg choice='plain'><replaceable>OLD-MANIFEST</replaceable></arg>
<arg choice='plain'><replaceable>NEW-MANIFEST</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>
<refsection><title>Description</title>
<para>The command <command>nix-generate-patches</command> generates
binary patches between NAR files listed in OLD-MANIFEST and NEW-MANIFEST.
The patches are written to the directory PATCH-DIR, and the prefix
PATCH-URI is used to generate URIs for the patches. The patches are
added to NEW-MANIFEST. All NARs are required to exist in NAR-DIR.
Patches are generated between succeeding versions of packages with
the same name.</para>
</refsection>
</refentry>
@@ -0,0 +1,210 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="sec-nix-install-package">
<refmeta>
<refentrytitle>nix-install-package</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo class="source">Nix</refmiscinfo>
<refmiscinfo class="version"><xi:include href="../version.txt" parse="text"/></refmiscinfo>
</refmeta>
<refnamediv>
<refname>nix-install-package</refname>
<refpurpose>install a Nix Package file</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>nix-install-package</command>
<arg><option>--non-interactive</option></arg>
<arg>
<group choice='req'>
<arg choice='plain'><option>--profile</option></arg>
<arg choice='plain'><option>-p</option></arg>
</group>
<replaceable>path</replaceable>
</arg>
<arg><option>--set</option></arg>
<sbr />
<group choice='req'>
<arg choice='req'>
<option>--url</option>
<arg choice='plain'><replaceable>url</replaceable></arg>
</arg>
<arg choice='req'>
<arg choice='plain'><replaceable>file</replaceable></arg>
</arg>
</group>
</cmdsynopsis>
</refsynopsisdiv>
<refsection><title>Description</title>
<para>The command <command>nix-install-package</command> interactively
installs a Nix Package file (<filename>*.nixpkg</filename>), which is
a small file that contains a store path to be installed along with the
URL of a binary cache. The Nix Package file is either
<replaceable>file</replaceable>, or automatically downloaded from
<replaceable>url</replaceable> if the <option>--url</option> switch is
used.</para>
<para><command>nix-install-package</command> is used in <link
linkend="sec-one-click">one-click installs</link> to download and
install pre-built binary packages with all necessary dependencies.
<command>nix-install-package</command> is intended to be associated
with the MIME type <literal>application/nix-package</literal> in a web
browser so that it is invoked automatically when you click on
<filename>*.nixpkg</filename> files. When invoked, it restarts itself
in a terminal window (since otherwise it would be invisible when run
from a browser), asks the user to confirm whether to install the
package, and if so downloads and installs the package into the user’s
current profile.</para>
<para>To obtain a window, <command>nix-install-package</command> tries
to restart itself with <command>xterm</command>,
<command>konsole</command> and
<command>gnome-terminal</command>.</para>
</refsection>
<refsection><title>Options</title>
<variablelist>
<varlistentry><term><option>--non-interactive</option></term>
<listitem><para>Do not open a new terminal window and do not ask
for confirmation.</para></listitem>
</varlistentry>
<varlistentry><term><option>--profile</option></term>
<term><option>-p</option></term>
<listitem><para>Install the package into the specified profile
rather than the user’s current profile.</para></listitem>
</varlistentry>
<varlistentry><term><option>--set</option></term>
<listitem><para>Install the package as the profile so that the
profile contains exactly the contents of the package.</para></listitem>
</varlistentry>
</variablelist>
</refsection>
<refsection><title>Examples</title>
<para>To install <filename>subversion-1.4.0.nixpkg</filename> into the
user’s current profile, without any prompting:
<screen>
$ nix-install-package --non-interactive subversion-1.4.0.nixpkg</screen>
</para>
<para>To install the same package from some URL into a different
profile:
<screen>
$ nix-install-package --non-interactive -p /nix/var/nix/profiles/eelco \
--url http://nix.cs.uu.nl/dist/nix/nixpkgs-0.10pre6622/pkgs/subversion-1.4.0-i686-linux.nixpkg</screen>
</para>
</refsection>
<refsection><title>Format of <literal>nixpkg</literal> files</title>
<para>A Nix Package file consists of a single line with the following
format:
<screen>
NIXPKG1 <replaceable>manifestURL</replaceable> <replaceable>name</replaceable> <replaceable>system</replaceable> <replaceable>drvPath</replaceable> <replaceable>outPath</replaceable></screen>
The elements are as follows:
<variablelist>
<varlistentry><term><literal>NIXPKG1</literal></term>
<listitem><para>The version of the Nix Package
file.</para></listitem>
</varlistentry>
<varlistentry><term><replaceable>manifestURL</replaceable></term>
<listitem><para>The manifest to be pulled by
<command>nix-pull</command>. The manifest must contain
<replaceable>outPath</replaceable>.</para></listitem>
</varlistentry>
<varlistentry><term><replaceable>name</replaceable></term>
<listitem><para>The symbolic name and version of the
package.</para></listitem>
</varlistentry>
<varlistentry><term><replaceable>system</replaceable></term>
<listitem><para>The platform identifier of the platform for which
this binary package is intended.</para></listitem>
</varlistentry>
<varlistentry><term><replaceable>drvPath</replaceable></term>
<listitem><para>The path in the Nix store of the derivation from
which <replaceable>outPath</replaceable> was built. Not currently
used.</para></listitem>
</varlistentry>
<varlistentry><term><replaceable>outPath</replaceable></term>
<listitem><para>The path in the Nix store of the
package.</para></listitem>
</varlistentry>
<varlistentry><term><replaceable>binaryCacheURL</replaceable></term>
<listitem><para>The URL of a binary cache containing the closure
of <replaceable>outPath</replaceable>.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>An example follows:
<screen>
NIXPKG1 http://.../nixpkgs-0.10pre6622/MANIFEST subversion-1.4.0 i686-darwin \
/nix/store/4kh60jkp...-subversion-1.4.0.drv \
/nix/store/nkw7wpgb...-subversion-1.4.0</screen>
(The line breaks (<literal>\</literal>) are for presentation purposes
and not part of the actual file.)
</para>
</refsection>
</refentry>
+12 -9
View File
@@ -43,7 +43,8 @@
<arg choice='plain'><option>-E</option></arg>
</group>
<arg choice='plain' rep='repeat'><replaceable>files</replaceable></arg>
<sbr/>
</cmdsynopsis>
<cmdsynopsis>
<command>nix-instantiate</command>
<arg choice='plain'><option>--find-file</option></arg>
<arg choice='plain' rep='repeat'><replaceable>files</replaceable></arg>
@@ -117,19 +118,21 @@ input.</para>
<varlistentry><term><option>--xml</option></term>
<listitem><para>When used with <option>--eval</option>, print the resulting
value as an XML representation of the abstract syntax tree rather than as
an ATerm. The schema is the same as that used by the <link
linkend="builtin-toXML"><function>toXML</function> built-in</link>.
</para></listitem>
<listitem><para>When used with <option>--parse</option> and
<option>--eval</option>, print the resulting expression as an
XML representation of the abstract syntax tree rather than as an
ATerm. The schema is the same as that used by the <link
linkend="builtin-toXML"><function>toXML</function>
built-in</link>.</para></listitem>
</varlistentry>
<varlistentry><term><option>--json</option></term>
<listitem><para>When used with <option>--eval</option>, print the resulting
value as an JSON representation of the abstract syntax tree rather
than as an ATerm.</para></listitem>
<listitem><para>When used with <option>--parse</option> and
<option>--eval</option>, print the resulting expression as an
JSON representation of the abstract syntax tree rather than as an
ATerm.</para></listitem>
</varlistentry>
+54
View File
@@ -0,0 +1,54 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="sec-nix-pull">
<refmeta>
<refentrytitle>nix-pull</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo class="source">Nix</refmiscinfo>
<refmiscinfo class="version"><xi:include href="../version.txt" parse="text"/></refmiscinfo>
</refmeta>
<refnamediv>
<refname>nix-pull</refname>
<refpurpose>register availability of pre-built binaries (deprecated)</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>nix-pull</command>
<arg choice='plain'><replaceable>url</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>
<refsection><title>Description</title>
<note><para>This command and the use of manifests is deprecated. It is
better to use binary caches.</para></note>
<para>The command <command>nix-pull</command> obtains a list of
pre-built store paths from the URL <replaceable>url</replaceable>, and
for each of these store paths, registers a substitute derivation that
downloads and unpacks it into the Nix store. This is used to speed up
installations: if you attempt to install something that has already
been built and stored into the network cache, Nix can transparently
re-use the pre-built store paths.</para>
<para>The file at <replaceable>url</replaceable> must be compatible
with the files created by <replaceable>nix-push</replaceable>.</para>
</refsection>
<refsection><title>Examples</title>
<screen>
$ nix-pull https://nixos.org/releases/nixpkgs/nixpkgs-15.05pre54468.69858d7/MANIFEST</screen>
</refsection>
</refentry>
+464
View File
@@ -0,0 +1,464 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="sec-nix-push">
<refmeta>
<refentrytitle>nix-push</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo class="source">Nix</refmiscinfo>
<refmiscinfo class="version"><xi:include href="../version.txt" parse="text"/></refmiscinfo>
</refmeta>
<refnamediv>
<refname>nix-push</refname>
<refpurpose>generate a binary cache</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>nix-push</command>
<arg choice='plain'><option>--dest</option> <replaceable>dest-dir</replaceable></arg>
<arg><option>--bzip2</option></arg>
<arg><option>--none</option></arg>
<arg><option>--force</option></arg>
<arg><option>--link</option></arg>
<arg><option>--manifest</option></arg>
<arg><option>--manifest-path</option> <replaceable>filename</replaceable></arg>
<arg><option>--url-prefix</option> <replaceable>url</replaceable></arg>
<arg><option>--key-file</option> <replaceable>path</replaceable></arg>
<arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>
<refsection><title>Description</title>
<para>The command <command>nix-push</command> produces a
<emphasis>binary cache</emphasis>, a directory containing compressed
Nix archives (NARs) plus some metadata of the closure of the specified
store paths. This directory can then be made available through a web
server to other Nix installations, allowing them to skip building from
source and instead download binaries from the cache
automatically.</para>
<para><command>nix-push</command> performs the following actions.
<orderedlist>
<listitem><para>Each path in <replaceable>paths</replaceable> is
built (using <link
linkend='rsec-nix-store-realise'><command>nix-store
--realise</command></link>).</para></listitem>
<listitem><para>All paths in the closure of
<replaceable>paths</replaceable> are determined (using
<command>nix-store --query --requisites
--include-outputs</command>). Note that since the
<option>--include-outputs</option> flag is used, if
<replaceable>paths</replaceable> includes a store derivation, you
get a combined source/binary distribution (e.g., source tarballs
will be included).</para></listitem>
<listitem><para>All store paths determined in the previous step are
packaged into a NAR (using <command>nix-store --dump</command>) and
compressed using <command>xz</command> or <command>bzip2</command>.
The resulting files have the extension <filename>.nar.xz</filename>
or <filename>.nar.bz2</filename>. Also for each store path, Nix
generates a file with extension <filename>.narinfo</filename>
containing metadata such as the references, cryptographic hash and
size of each path.</para></listitem>
<listitem><para>Optionally, a single <emphasis>manifest</emphasis>
file is created that contains the same metadata as the
<filename>.narinfo</filename> files. This is for compatibility with
Nix versions prior to 1.2 (see <command>nix-pull</command> for
details).</para></listitem>
<listitem><para>A file named <option>nix-cache-info</option> is
placed in the destination directory. The existence of this file
marks the directory as a binary cache.</para></listitem>
</orderedlist>
</para>
</refsection>
<refsection><title>Options</title>
<variablelist>
<varlistentry><term><option>--dest</option> <replaceable>dest-dir</replaceable></term>
<listitem><para>Set the destination directory to
<replaceable>dir</replaceable>, which is created if it does not
exist. This flag is required.</para></listitem>
</varlistentry>
<varlistentry><term><option>--bzip2</option></term>
<listitem><para>Compress NARs using <command>bzip2</command>
instead of <command>xz</command>. The latter compresses about 30%
better on typical archives, decompresses about twice as fast, but
compresses a lot slower and is not supported by Nix prior to
version 1.2.</para></listitem>
</varlistentry>
<varlistentry><term><option>--none</option></term>
<listitem><para>Do not compress NARs.</para></listitem>
</varlistentry>
<varlistentry><term><option>--force</option></term>
<listitem><para>Overwrite <filename>.narinfo</filename> files if
they already exist.</para></listitem>
</varlistentry>
<varlistentry><term><option>--link</option></term>
<listitem><para>By default, NARs are generated in the Nix store
and then copied to <replaceable>dest-dir</replaceable>. If this
option is given, hard links are used instead. This only works if
<replaceable>dest-dir</replaceable> is on the same filesystem as
the Nix store.</para></listitem>
</varlistentry>
<varlistentry><term><option>--manifest</option></term>
<listitem><para>Force the generation of a manifest suitable for
use by <command>nix-pull</command>. The manifest is stored as
<filename><replaceable>dest-dir</replaceable>/MANIFEST</filename>.</para></listitem>
</varlistentry>
<varlistentry><term><option>--manifest-path</option> <replaceable>filename</replaceable></term>
<listitem><para>Like <option>--manifest</option>, but store the
manifest in <replaceable>filename</replaceable>.</para></listitem>
</varlistentry>
<varlistentry><term><option>--url-prefix</option> <replaceable>url</replaceable></term>
<listitem><para>Manifests are expected to contain the absolute
URLs of NARs. For generating these URLs, the prefix
<replaceable>url</replaceable> is used. It defaults to
<uri>file://<replaceable>dest-dir</replaceable></uri>.</para></listitem>
</varlistentry>
<varlistentry><term><option>--key-file</option> <replaceable>path</replaceable></term>
<listitem><para>Sign the binary cache using the secret key stored
in <replaceable>path</replaceable>. This secret key must have been
created using <command
linkend="rsec-nix-store-generate-binary-cache-key">nix-store
--generate-binary-cache-key</command>. Users of this binary cache
should add the corresponding public key to the option
<option>binary-cache-public-keys</option> in
<filename>nix.conf</filename>.</para></listitem>
</varlistentry>
</variablelist>
</refsection>
<refsection><title>Examples</title>
<para>To add the closure of Thunderbird to a binary cache:
<screen>
$ nix-push --dest /tmp/cache $(nix-build -A thunderbird)
</screen>
Assuming that <filename>/tmp/cache</filename> is exported by a web
server as <uri>http://example.org/cache</uri>, you can then use this
cache on another machine to speed up the installation of Thunderbird:
<screen>
$ nix-build -A thunderbird --option binary-caches http://example.org/cache
</screen>
Alternatively, you could add <literal>binary-caches =
http://example.org/cache</literal> to
<filename>nix.conf</filename>.</para>
<para>To also include build-time dependencies (such as source
tarballs):
<screen>
$ nix-push --dest /tmp/cache $(nix-instantiate -A thunderbird)
</screen>
</para>
<para>To generate a manifest suitable for <command>nix-pull</command>:
<screen>
$ nix-push --dest /tmp/cache $(nix-build -A thunderbird) --manifest
</screen>
On another machine you can then do:
<screen>
$ nix-pull http://example.org/cache
</screen>
to cause the binaries to be used by subsequent Nix operations.</para>
<para>To generate a signed binary cache, you must first generate a key
pair, in this example called <literal>cache.example.org-1</literal>,
storing the secret key in <filename>./sk</filename> and the public key
in <filename>./pk</filename>:
<screen>
$ nix-store --generate-binary-cache-key cache.example.org-1 sk pk
$ cat sk
cache.example.org-1:jcMRQYFo8pQKzTtimpQLIPeHkMYZjfhB24hGfwF+u9PuX8H8FO7q564+X3G/JDlqqIqGar3OXRRwS9N3Wh3vbw==
$ cat pk
cache.example.org-1:7l/B/BTu6ueuPl9xvyQ5aqiKhmq9zl0UcEvTd1od728=
</screen>
You can then generate a binary cache signed with the secret key:
<screen>
$ nix-push --dest /tmp/cache --key-file ./sk $(type -p firefox)
</screen>
Users who wish to verify the integrity of binaries downloaded from
your cache would add the following to their
<filename>nix.conf</filename>:
<programlisting>
binary-caches = http://cache.example.org
signed-binary-caches = *
binary-cache-public-keys = cache.example.org-1:7l/B/BTu6ueuPl9xvyQ5aqiKhmq9zl0UcEvTd1od728=
</programlisting>
Nix will then ignore any binary that has a missing, incorrect or
unrecognised signature.</para>
</refsection>
<refsection><title>Binary cache format and operation</title>
<para>A binary cache with URL <replaceable>url</replaceable> only
denotes a valid binary cache if the file
<uri><replaceable>url</replaceable>/nix-cache-info</uri> exists. If
this file does not exist (or cannot be downloaded), the cache is
ignored. If it does exist, it must be a text file containing cache
properties. Here’s an example:
<screen>
StoreDir: /nix/store
WantMassQuery: 1
Priority: 10
</screen>
The properties that are currently supported are:
<variablelist>
<varlistentry><term><literal>StoreDir</literal></term>
<listitem><para>The path of the Nix store to which this binary
cache applies. Binaries are not relocatable — a binary built for
<filename>/nix/store</filename> won’t generally work in
<filename>/home/alice/store</filename> — so to prevent binaries
from being used in a wrong store, a binary cache is only used if
its <literal>StoreDir</literal> matches the local Nix
configuration. The default is
<filename>/nix/store</filename>.</para></listitem>
</varlistentry>
<varlistentry><term><literal>WantMassQuery</literal></term>
<listitem><para>Query operations such as <command>nix-env
-qas</command> can cause thousands of cache queries, and thus
thousands of HTTP requests, to determine which packages are
available in binary form. While these requests are small, not
every server may appreciate a potential onslaught of queries. If
<literal>WantMassQuery</literal> is set to <literal>0</literal>
(default), “mass queries” such as <command>nix-env -qas</command>
will skip this cache. Thus a package may appear not to have a
binary substitute. However, the binary will still be used when
you actually install the package. If
<literal>WantMassQuery</literal> is set to <literal>1</literal>,
mass queries will use this cache.</para></listitem>
</varlistentry>
<varlistentry><term><literal>Priority</literal></term>
<listitem><para>Each binary cache has a priority (defaulting to
50). Binary caches are checked for binaries in order of ascending
priority; thus a higher number denotes a lower priority. The
binary cache <uri>https://cache.nixos.org</uri> has priority
40.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>Every time Nix needs to build some store path
<replaceable>p</replaceable>, it will check each configured binary
cache to see if it has a NAR file for <replaceable>p</replaceable>,
until it finds one. If no cache has a NAR, Nix will fall back to
building the path from source (if applicable). To see if a cache with
URL <replaceable>url</replaceable> has a binary for
<replaceable>p</replaceable>, Nix fetches
<replaceable>url/h</replaceable>, where <replaceable>h</replaceable>
is the hash part of <replaceable>p</replaceable>. Thus, if we have a
cache <uri>https://cache.nixos.org</uri> and we want to obtain the
store path
<screen>
/nix/store/a8922c0h87iilxzzvwn2hmv8x210aqb9-glibc-2.7
</screen>
then Nix will attempt to fetch
<screen>
https://cache.nixos.org/a8922c0h87iilxzzvwn2hmv8x210aqb9.narinfo
</screen>
(Commands such as <command>nix-env -qas</command> will issue an HTTP
HEAD request, since it only needs to know if the
<filename>.narinfo</filename> file exists.) The
<filename>.narinfo</filename> file is a simple text file that looks
like this:
<screen>
StorePath: /nix/store/a8922c0h87iilxzzvwn2hmv8x210aqb9-glibc-2.7
URL: nar/0zzjpdz46mdn74v09m053yczlz4am038g8r74iy8w43gx8801h70.nar.bz2
Compression: bzip2
FileHash: sha256:0zzjpdz46mdn74v09m053yczlz4am038g8r74iy8w43gx8801h70
FileSize: 24473768
NarHash: sha256:0s491y1h9hxj5ghiizlxk7ax6jwbha00zwn7lpyd5xg5bhf60vzg
NarSize: 109521136
References: 2ma2k0ys8knh4an48n28vigcmc2z8773-linux-headers-2.6.23.16 ...
Deriver: 7akyyc87ka32xwmqza9dvyg5pwx3j212-glibc-2.7.drv
Sig: cache.example.org-1:WepnSp2UT0odDpR3NRjPVhJBHmdBgSBSTbHpdh4SCz92nGXwFY82bkPEmISoC0hGqBXDXEmB6y3Ohgna3mMgDg==
</screen>
The fields are as follows:
<variablelist>
<varlistentry><term><literal>StorePath</literal></term>
<listitem><para>The full store path, including the name part
(e.g., <literal>glibc-2.7</literal>). It must match the
requested store path.</para></listitem>
</varlistentry>
<varlistentry><term><literal>URL</literal></term>
<listitem><para>The URL of the NAR, relative to the binary cache
URL.</para></listitem>
</varlistentry>
<varlistentry><term><literal>Compression</literal></term>
<listitem><para>The compression method; either
<literal>xz</literal> or
<literal>bzip2</literal>.</para></listitem>
</varlistentry>
<varlistentry><term><literal>FileHash</literal></term>
<listitem><para>The SHA-256 hash of the compressed
NAR.</para></listitem>
</varlistentry>
<varlistentry><term><literal>FileSize</literal></term>
<listitem><para>The size of the compressed NAR.</para></listitem>
</varlistentry>
<varlistentry><term><literal>NarHash</literal></term>
<listitem><para>The SHA-256 hash of the uncompressed NAR. This is
equal to the hash of the store path as returned by
<command>nix-store -q --hash
<replaceable>p</replaceable></command>.</para></listitem>
</varlistentry>
<varlistentry><term><literal>NarSize</literal></term>
<listitem><para>The size of the uncompressed NAR.</para></listitem>
</varlistentry>
<varlistentry><term><literal>References</literal></term>
<listitem><para>The references of the store path, without the Nix
store prefix.</para></listitem>
</varlistentry>
<varlistentry><term><literal>Deriver</literal></term>
<listitem><para>The deriver of the store path, without the Nix
store prefix. This field is optional.</para></listitem>
</varlistentry>
<varlistentry><term><literal>System</literal></term>
<listitem><para>The Nix platform type of this binary, if known.
This field is optional.</para></listitem>
</varlistentry>
<varlistentry><term><literal>Sig</literal></term>
<listitem><para>A signature of the the form
<literal><replaceable>key-name</replaceable>:<replaceable>sig</replaceable></literal>,
where <replaceable>key-name</replaceable> is the symbolic name of
the key pair used to sign and verify the cache
(e.g. <literal>cache.example.org-1</literal>), and
<replaceable>sig</replaceable> is the actual signature, computed
over the <varname>StorePath</varname>, <varname>NarHash</varname>,
<varname>NarSize</varname> and <varname>References</varname>
fields using the <link
xlink:href="http://ed25519.cr.yp.to/">Ed25519 public-key signature
system</link>.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>Thus, in our example, after recursively ensuring that the
references exist (e.g.,
<filename>/nix/store/2ma2k0ys8knh4an48n28vigcmc2z8773-linux-headers-2.6.23.16</filename>),
Nix will fetch <screen>
https://cache.nixos.org/nar/0zzjpdz46mdn74v09m053yczlz4am038g8r74iy8w43gx8801h70.nar.bz2
</screen> and decompress and unpack it to
<filename>/nix/store/a8922c0h87iilxzzvwn2hmv8x210aqb9-glibc-2.7</filename>.</para>
</refsection>
</refentry>
+1 -1
View File
@@ -267,7 +267,7 @@ dependencies in Nixpkgs.</para>
<para>The lines starting with <literal>#! nix-shell</literal> specify
<command>nix-shell</command> options (see above). Note that you cannot
write <literal>#! /usr/bin/env nix-shell -i ...</literal> because
write <literal>#1 /usr/bin/env nix-shell -i ...</literal> because
many operating systems only allow one argument in
<literal>#!</literal> lines.</para>
+90 -67
View File
@@ -246,65 +246,6 @@ $ nix-build -r '&lt;nixpkgs>' -A hello --check -K
<!--######################################################################-->
<refsection xml:id='rsec-nix-store-serve'><title>Operation <option>--serve</option></title>
<refsection><title>Synopsis</title>
<cmdsynopsis>
<command>nix-store</command>
<arg choice='plain'><option>--serve</option></arg>
<arg><option>--write</option></arg>
</cmdsynopsis>
</refsection>
<refsection><title>Description</title>
<para>The operation <option>--serve</option> provides access to
the Nix store over stdin and stdout, and is intended to be used
as a means of providing Nix store access to a restricted ssh user.
</para>
<para>The following flags are available:</para>
<variablelist>
<varlistentry><term><option>--write</option></term>
<listitem><para>Allow the connected client to request the realization
of derivations. In effect, this can be used to make the host act
as a build slave.</para></listitem>
</varlistentry>
</variablelist>
</refsection>
<refsection><title>Examples</title>
<para>To turn a host into a build server, the
<filename>authorized_keys</filename> file can be used to provide build
access to a given SSH public key:
<screen>
$ cat &lt;&lt;EOF >>/root/.ssh/authorized_keys
command="nice -n20 nix-store --serve --write" ssh-rsa AAAAB3NzaC1yc2EAAAA...
EOF
</screen>
</para>
</refsection>
</refsection>
<!--######################################################################-->
<refsection xml:id='rsec-nix-store-gc'><title>Operation <option>--gc</option></title>
@@ -1115,18 +1056,18 @@ path).</para>
<para>This command does not produce a <emphasis>closure</emphasis> of
the specified paths, so if a store path references other store paths
that are missing in the target Nix store, the import will fail. To
copy a whole closure, do something like:
copy a whole closure, do something like
<screen>
$ nix-store --export $(nix-store -qR <replaceable>paths</replaceable>) > out</screen>
To import the whole closure again, run:
<screen>
$ nix-store --import &lt; out</screen>
</para>
<para>For an example of how <option>--export</option> and
<option>--import</option> can be used, see the source of the <command
linkend="sec-nix-copy-closure">nix-copy-closure</command>
command.</para>
</refsection>
@@ -1236,7 +1177,12 @@ the store path is used.</para>
<filename>/nix/var/log/nix/drvs</filename>. However, there is no
guarantee that a build log is available for any particular store path.
For instance, if the path was downloaded as a pre-built binary through
a substitute, then the log is unavailable.</para>
a substitute, then the log is unavailable. If the log is not available
locally, then <command>nix-store</command> will try to download the
log from the servers specified in the Nix option
<option>log-servers</option>. For example, if it’s set to
<literal>http://hydra.nixos.org/log</literal>, then Nix will check
<literal>http://hydra.nixos.org/log/<replaceable>base-name</replaceable></literal>.</para>
</refsection>
@@ -1343,6 +1289,82 @@ export _args; _args='-e /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25c-default-buil
</refsection>
<!--######################################################################-->
<refsection><title>Operation <option>--query-failed-paths</option></title>
<refsection>
<title>Synopsis</title>
<cmdsynopsis>
<command>nix-store</command>
<arg choice='plain'><option>--query-failed-paths</option></arg>
</cmdsynopsis>
</refsection>
<refsection><title>Description</title>
<para>If build failure caching is enabled through the
<literal>build-cache-failure</literal> configuration option, the
operation <option>--query-failed-paths</option> will print out all
store paths that have failed to build.</para>
</refsection>
<refsection><title>Example</title>
<screen>
$ nix-store --query-failed-paths
/nix/store/000zi5dcla86l92jn1g997jb06sidm7x-perl-PerlMagick-6.59
/nix/store/0011iy7sfwbc1qj5a1f6ifjnbcdail8a-haskell-gitit-ghc7.0.4-0.8.1
/nix/store/001c0yn1hkh86gprvrb46cxnz3pki7q3-gamin-0.1.10
<replaceable>…</replaceable>
</screen>
</refsection>
</refsection>
<!--######################################################################-->
<refsection><title>Operation <option>--clear-failed-paths</option></title>
<refsection>
<title>Synopsis</title>
<cmdsynopsis>
<command>nix-store</command>
<arg choice='plain'><option>--clear-failed-paths</option></arg>
<arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg>
</cmdsynopsis>
</refsection>
<refsection><title>Description</title>
<para>If build failure caching is enabled through the
<literal>build-cache-failure</literal> configuration option, the
operation <option>--clear-failed-paths</option> clears the “failed”
state of the given store paths, allowing them to be built again. This
is useful if the failure was actually transient (e.g. because the disk
was full).</para>
<para>If a path denotes a derivation, its output paths are cleared.
You can provide the argument <literal>*</literal> to clear all store
paths.</para>
</refsection>
<refsection><title>Example</title>
<screen>
$ nix-store --clear-failed-paths /nix/store/000zi5dcla86l92jn1g997jb06sidm7x-perl-PerlMagick-6.59
$ nix-store --clear-failed-paths *
</screen>
</refsection>
</refsection>
<!--######################################################################-->
<refsection xml:id='rsec-nix-store-generate-binary-cache-key'><title>Operation <option>--generate-binary-cache-key</option></title>
@@ -1385,7 +1407,8 @@ parameters:
</orderedlist>
</para>
For an example, see the manual page for <command
linkend="sec-nix-push">nix-push</command>.</para>
</refsection>
@@ -31,6 +31,7 @@
<arg><option>-K</option></arg>
<arg><option>--fallback</option></arg>
<arg><option>--readonly-mode</option></arg>
<arg><option>--log-type</option> <replaceable>type</replaceable></arg>
<arg><option>--show-trace</option></arg>
<arg>
<option>-I</option>
+60 -22
View File
@@ -93,9 +93,8 @@
<term><option>-j</option></term>
<listitem><para>Sets the maximum number of build jobs that Nix will
perform in parallel to the specified number. Specify
<literal>auto</literal> to use the number of CPUs in the system.
The default is specified by the <link
perform in parallel to the specified number. The default is
specified by the <link
linkend='conf-build-max-jobs'><literal>build-max-jobs</literal></link>
configuration setting, which itself defaults to
<literal>1</literal>. A higher value is useful on SMP systems or to
@@ -192,23 +191,6 @@
</varlistentry>
<varlistentry><term><option>--no-build-hook</option></term>
<listitem>
<para>Disables the build hook mechanism. This allows to ignore remote
builders if they are setup on the machine.</para>
<para>It's useful in cases where the bandwidth between the client and the
remote builder is too low. In that case it can take more time to upload the
sources to the remote builder and fetch back the result than to do the
computation locally.</para>
</listitem>
</varlistentry>
<varlistentry><term><option>--readonly-mode</option></term>
@@ -219,6 +201,61 @@
</varlistentry>
<varlistentry xml:id="opt-log-type"><term><option>--log-type</option>
<replaceable>type</replaceable></term>
<listitem>
<para>This option determines how the output written to standard
error is formatted. Nix’s diagnostic messages are typically
<emphasis>nested</emphasis>. For instance, when tracing Nix
expression evaluation (<command>nix-env -vvvvv</command>, messages
from subexpressions are nested inside their parent expressions. Nix
builder output is also often nested. For instance, the Nix Packages
generic builder nests the various build tasks (unpack, configure,
compile, etc.), and the GNU Make in <literal>stdenv-linux</literal>
has been patched to provide nesting for recursive Make
invocations.</para>
<para><replaceable>type</replaceable> can be one of the
following:
<variablelist>
<varlistentry><term><literal>pretty</literal></term>
<listitem><para>Pretty-print the output, indicating different
nesting levels using spaces. This is the
default.</para></listitem>
</varlistentry>
<varlistentry><term><literal>escapes</literal></term>
<listitem><para>Indicate nesting using escape codes that can be
interpreted by the <command>nix-log2xml</command> tool in the
Nix source distribution. The resulting XML file can be fed into
the <command>log2html.xsl</command> stylesheet to create an HTML
file that can be browsed interactively, using JavaScript to
expand and collapse parts of the output.</para></listitem>
</varlistentry>
<varlistentry><term><literal>flat</literal></term>
<listitem><para>Remove all nesting.</para></listitem>
</varlistentry>
</variablelist>
</para>
</listitem>
</varlistentry>
<varlistentry><term><option>--arg</option> <replaceable>name</replaceable> <replaceable>value</replaceable></term>
<listitem><para>This option is accepted by
@@ -236,8 +273,9 @@
named <replaceable>name</replaceable>, it will call it with value
<replaceable>value</replaceable>.</para>
<para>For instance, the top-level <literal>default.nix</literal> in
Nixpkgs is actually a function:
<para>For instance, the file
<literal>pkgs/top-level/all-packages.nix</literal> in Nixpkgs is
actually a function:
<programlisting>
{ # The system (e.g., `i686-linux') for which to build the packages.
+6
View File
@@ -13,8 +13,14 @@ work with Nix.</para>
<xi:include href="nix-collect-garbage.xml" />
<xi:include href="nix-copy-closure.xml" />
<xi:include href="nix-daemon.xml" />
<!--
<xi:include href="nix-generate-patches.xml" />
-->
<xi:include href="nix-hash.xml" />
<xi:include href="nix-install-package.xml" />
<xi:include href="nix-instantiate.xml" />
<xi:include href="nix-prefetch-url.xml" />
<xi:include href="nix-pull.xml" />
<xi:include href="nix-push.xml" />
</chapter>
+16 -96
View File
@@ -32,7 +32,7 @@ available as <function>builtins.derivation</function>.</para>
<varlistentry><term><function>builtins.add</function>
<replaceable>e1</replaceable> <replaceable>e2</replaceable></term>
<listitem><para>Return the sum of the numbers
<listitem><para>Return the sum of the integers
<replaceable>e1</replaceable> and
<replaceable>e2</replaceable>.</para></listitem>
@@ -65,7 +65,7 @@ available as <function>builtins.derivation</function>.</para>
<replaceable>set</replaceable></term>
<listitem><para>Return the names of the attributes in the set
<replaceable>set</replaceable> in an alphabetically sorted list. For instance,
<replaceable>set</replaceable> in a sorted list. For instance,
<literal>builtins.attrNames { y = 1; x = "foo"; }</literal>
evaluates to <literal>[ "x" "y" ]</literal>.</para></listitem>
@@ -142,7 +142,7 @@ if builtins ? getEnv then builtins.getEnv "PATH" else ""</programlisting>
evaluates to the Nix platform identifier for the Nix installation
on which the expression is being evaluated, such as
<literal>"i686-linux"</literal> or
<literal>"x86_64-darwin"</literal>.</para></listitem>
<literal>"powerpc-darwin"</literal>.</para></listitem>
</varlistentry>
@@ -204,47 +204,12 @@ if builtins ? getEnv then builtins.getEnv "PATH" else ""</programlisting>
<varlistentry><term><function>builtins.div</function>
<replaceable>e1</replaceable> <replaceable>e2</replaceable></term>
<listitem><para>Return the quotient of the numbers
<listitem><para>Return the quotient of the integers
<replaceable>e1</replaceable> and
<replaceable>e2</replaceable>.</para></listitem>
</varlistentry>
<varlistentry><term><function>builtins.match</function>
<replaceable>regex</replaceable> <replaceable>str</replaceable></term>
<listitem><para>Returns a list if the <link
xlink:href="http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_04">extended
POSIX regular expression</link> <replaceable>regex</replaceable>
matches <replaceable>str</replaceable> precisely, otherwise returns
<literal>null</literal>. Each item in the list is a regex group.
<programlisting>
builtins.match "ab" "abc"
</programlisting>
Evaluates to <literal>null</literal>.
<programlisting>
builtins.match "abc" "abc"
</programlisting>
Evaluates to <literal>[ ]</literal>.
<programlisting>
builtins.match "a(b)(c)" "abc"
</programlisting>
Evaluates to <literal>[ "b" "c" ]</literal>.
<programlisting>
builtins.match "[[:space:]]+([[:upper:]]+)[[:space:]]+" " FOO "
</programlisting>
Evaluates to <literal>[ "foo" ]</literal>.
</para></listitem>
</varlistentry>
<varlistentry><term><function>builtins.elem</function>
<replaceable>x</replaceable> <replaceable>xs</replaceable></term>
@@ -298,24 +263,6 @@ with import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/nixo
stdenv.mkDerivation { … }
</programlisting>
Note that when obtaining the hash with <varname>nix-prefetch-url
</varname> the option <varname>--unpack</varname> is required.
</para>
<para>This function can also verify the contents against a hash.
In that case, the function takes a set instead of a URL. The set
requires the attribute <varname>url</varname> and the attribute
<varname>sha256</varname>, e.g.
<programlisting>
with import (fetchTarball {
url = https://github.com/NixOS/nixpkgs-channels/archive/nixos-14.12.tar.gz;
sha256 = "1jppksrfvbk5ypiqdz4cddxdl8z6zyzdb2srq8fcffr327ld5jj2";
}) {};
stdenv.mkDerivation { … }
</programlisting>
</para>
<para>This function is not available if <link
@@ -440,7 +387,7 @@ builtins.fromJSON ''{"x": [1, 2, 3], "y": null}''
<listitem><para>Generate list of size
<replaceable>length</replaceable>, with each element
<replaceable>i</replaceable> equal to the value returned by
<replaceable>i></replaceable> equal to the value returned by
<replaceable>generator</replaceable> <literal>i</literal>. For
example,
@@ -673,12 +620,12 @@ x: x + 456</programlisting>
<varlistentry><term><function>builtins.lessThan</function>
<replaceable>e1</replaceable> <replaceable>e2</replaceable></term>
<listitem><para>Return <literal>true</literal> if the number
<replaceable>e1</replaceable> is less than the number
<listitem><para>Return <literal>true</literal> if the integer
<replaceable>e1</replaceable> is less than the integer
<replaceable>e2</replaceable>, and <literal>false</literal>
otherwise. Evaluation aborts if either
<replaceable>e1</replaceable> or <replaceable>e2</replaceable>
does not evaluate to a number.</para></listitem>
does not evaluate to an integer.</para></listitem>
</varlistentry>
@@ -729,7 +676,7 @@ map (x: "foo" + x) [ "bar" "bla" "abc" ]</programlisting>
<varlistentry><term><function>builtins.mul</function>
<replaceable>e1</replaceable> <replaceable>e2</replaceable></term>
<listitem><para>Return the product of the numbers
<listitem><para>Return the product of the integers
<replaceable>e1</replaceable> and
<replaceable>e2</replaceable>.</para></listitem>
@@ -886,7 +833,7 @@ builtins.sort builtins.lessThan [ 483 249 526 147 42 77 ]
<varlistentry><term><function>builtins.sub</function>
<replaceable>e1</replaceable> <replaceable>e2</replaceable></term>
<listitem><para>Return the difference between the numbers
<listitem><para>Return the difference between the integers
<replaceable>e1</replaceable> and
<replaceable>e2</replaceable>.</para></listitem>
@@ -906,14 +853,7 @@ builtins.sort builtins.lessThan [ 483 249 526 147 42 77 ]
len</replaceable> lies beyond the end of the string, only the
substring up to the end of the string is returned.
<replaceable>start</replaceable> must be
non-negative. For example,
<programlisting>
builtins.substring 0 3 "nixos"
</programlisting>
evaluates to <literal>"nix"</literal>.
</para></listitem>
non-negative.</para></listitem>
</varlistentry>
@@ -1020,7 +960,7 @@ in foo</programlisting>
<varlistentry><term><function>builtins.toJSON</function> <replaceable>e</replaceable></term>
<listitem><para>Return a string containing a JSON representation
of <replaceable>e</replaceable>. Strings, integers, floats, booleans,
of <replaceable>e</replaceable>. Strings, integers, booleans,
nulls and lists are mapped to their JSON equivalents. Sets
(except derivations) are represented as objects. Derivations are
translated to a JSON string containing the derivation’s output
@@ -1047,17 +987,10 @@ in foo</programlisting>
<listitem><para>Convert the expression
<replaceable>e</replaceable> to a string.
<replaceable>e</replaceable> can be:</para>
<itemizedlist>
<listitem><para>A string (in which case the string is returned unmodified).</para></listitem>
<listitem><para>A path (e.g., <literal>toString /foo/bar</literal> yields <literal>"/foo/bar"</literal>.</para></listitem>
<listitem><para>A set containing <literal>{ __toString = self: ...; }</literal>.</para></listitem>
<listitem><para>An integer.</para></listitem>
<listitem><para>A list, in which case the string representations of its elements are joined with spaces.</para></listitem>
<listitem><para>A Boolean (<literal>false</literal> yields <literal>""</literal>, <literal>true</literal> yields <literal>"1"</literal>.</para></listitem>
<listitem><para><literal>null</literal>, which yields the empty string.</para></listitem>
</itemizedlist>
</listitem>
<replaceable>e</replaceable> can be a string (in which case
<function>toString</function> is a no-op) or a path (e.g.,
<literal>toString /foo/bar</literal> yields
<literal>"/foo/bar"</literal>.</para></listitem>
</varlistentry>
@@ -1187,19 +1120,6 @@ stdenv.mkDerivation (rec {
</varlistentry>
<varlistentry><term><function>builtins.tryEval</function>
<replaceable>e</replaceable></term>
<listitem><para>Try to evaluate <replaceable>e</replaceable>.
Return a set containing the attributes <literal>success</literal>
(<literal>true</literal> if <replaceable>e</replaceable> evaluated
successfully, <literal>false</literal> if an error was thrown) and
<literal>value</literal>, equalling <replaceable>e</replaceable>
if successful and <literal>false</literal> otherwise.
</para></listitem>
</varlistentry>
<varlistentry><term><function>builtins.typeOf</function>
<replaceable>e</replaceable></term>
+3 -3
View File
@@ -16,7 +16,7 @@ of which specify the inputs of the build.</para>
<listitem xml:id="attr-system"><para>There must be an attribute named
<varname>system</varname> whose value must be a string specifying a
Nix platform identifier, such as <literal>"i686-linux"</literal> or
<literal>"x86_64-darwin"</literal><footnote><para>To figure out
<literal>"powerpc-darwin"</literal><footnote><para>To figure out
your platform identifier, look at the line <quote>Checking for the
canonical Nix system name</quote> in the output of Nix's
<filename>configure</filename> script.</para></footnote> The build
@@ -43,7 +43,7 @@ of which specify the inputs of the build.</para>
<itemizedlist>
<listitem><para>Strings and numbers are just passed
<listitem><para>Strings and integers are just passed
verbatim.</para></listitem>
<listitem><para>A <emphasis>path</emphasis> (e.g.,
@@ -100,7 +100,7 @@ outputs = [ "lib" "headers" "doc" ];
<programlisting>
buildInputs = [ pkg.lib pkg.headers ];
</programlisting>
The first element of <varname>outputs</varname> determines the
The first element of <varname>output</varname> determines the
<emphasis>default output</emphasis>. Thus, you could also write
<programlisting>
buildInputs = [ pkg pkg.headers ];
+23 -11
View File
@@ -177,19 +177,13 @@ map (concat "foo") [ "bar" "bla" "abc" ]</programlisting>
<listitem><para>An <literal>@</literal>-pattern provides a means of referring
to the whole value being matched:
<programlisting> args@{ x, y, z, ... }: z + y + x + args.a</programlisting>
but can also be written as:
<programlisting> { x, y, z, ... } @ args: z + y + x + args.a</programlisting>
<programlisting>
args@{ x, y, z, ... }: z + y + x + args.a</programlisting>
Here <varname>args</varname> is bound to the entire argument, which
is further matched against the pattern <literal>{ x, y, z,
... }</literal>. <literal>@</literal>-pattern makes mainly sense with an
ellipsis(<literal>...</literal>) as you can access attribute names as
<literal>a</literal>, using <literal>args.a</literal>, which was given as an
additional attribute to the function.
</para></listitem>
... }</literal>.</para></listitem>
</itemizedlist>
@@ -202,6 +196,24 @@ in concat { x = "foo"; y = "bar"; }</programlisting>
</para>
<para>A set that has a <literal>__functor</literal> attribute whose value
is callable (i.e. is itself a function or a set with a
<literal>__functor</literal> attribute whose value is callable) can be
applied as if it were a function, with the set itself passed in first
, e.g.,
<programlisting>
let add = { __functor = self: x: x + self.x; };
inc = add // { x = 1; };
in inc 1
</programlisting>
evaluates to <literal>2</literal>. This can be used to attach metadata to a
function without the caller needing to treat it specially, or to implement
a form of object-oriented programming, for example.
</para>
</simplesect>
@@ -347,4 +359,4 @@ character, or inline/multi-line, enclosed within <literal>/*
</simplesect>
</section>
</section>
+3 -34
View File
@@ -140,13 +140,8 @@ stdenv.mkDerivation {
</listitem>
<listitem><para>Numbers, which can be <emphasis>integers</emphasis> (like
<literal>123</literal>) or <emphasis>floating point</emphasis> (like
<literal>123.43</literal> or <literal>.27e13</literal>).</para>
<para>Numbers are type-compatible: pure integer operations will always
return integers, whereas any operation involving at least one floating point
number will have a floating point number as a result.</para></listitem>
<listitem><para><emphasis>Integers</emphasis>, e.g.,
<literal>123</literal>.</para></listitem>
<listitem><para><emphasis>Paths</emphasis>, e.g.,
<filename>/bin/sh</filename> or <filename>./builder.sh</filename>.
@@ -167,16 +162,7 @@ stdenv.mkDerivation {
user's home directory. e.g. <filename>~/foo</filename> would be
equivalent to <filename>/home/edolstra/foo</filename> for a user
whose home directory is <filename>/home/edolstra</filename>.
</para>
<para>Paths can also be specified between angle brackets, e.g.
<literal>&lt;nixpkgs&gt;</literal>. This means that the directories
listed in the environment variable
<envar linkend="env-NIX_PATH">NIX_PATH</envar> will be searched
for the given file or directory name.
</para>
</listitem>
</para></listitem>
<listitem><para><emphasis>Booleans</emphasis> with values
<literal>true</literal> and
@@ -285,23 +271,6 @@ added to the set:
This will evaluate to <literal>{}</literal> if <literal>foo</literal>
evaluates to <literal>false</literal>.</para>
<para>A set that has a <literal>__functor</literal> attribute whose value
is callable (i.e. is itself a function or a set with a
<literal>__functor</literal> attribute whose value is callable) can be
applied as if it were a function, with the set itself passed in first
, e.g.,
<programlisting>
let add = { __functor = self: x: x + self.x; };
inc = add // { x = 1; };
in inc 1
</programlisting>
evaluates to <literal>2</literal>. This can be used to attach metadata to a
function without the caller needing to treat it specially, or to implement
a form of object-oriented programming, for example.
</para>
</simplesect>
+2 -2
View File
@@ -22,7 +22,7 @@ $ nix-build release.nix -A build.x86_64-linux
environment variables are set up so that those dependencies can be
found:
<screen>
$ nix-shell
$ ./dev-shell
</screen>
To build Nix itself in this shell:
<screen>
@@ -30,7 +30,7 @@ To build Nix itself in this shell:
[nix-shell]$ configurePhase
[nix-shell]$ make
</screen>
To install it in <literal>$(pwd)/nix</literal> and test it:
To test it:
<screen>
[nix-shell]$ make install
[nix-shell]$ make installcheck
@@ -26,14 +26,6 @@ $ mkdir /nix
$ chown alice /nix
</screen>
The install script will modify the first writable file from amongst
<filename>.bash_profile</filename>, <filename>.bash_login</filename>
and <filename>.profile</filename> to source
<filename>~/.nix-profile/etc/profile.d/nix.sh</filename>. You can set
the <command>NIX_INSTALLER_NO_MODIFY_PROFILE</command> environment
variable before executing the install script to disable this
behaviour.
</para>
<!--
@@ -61,7 +53,6 @@ install it like this:
$ dpkg -i nix_1.8-1_amd64.deb</screen>
</para>
-->
<para>You can also download a binary tarball that contains Nix and all
its dependencies. (This is what the install script at
@@ -88,6 +79,7 @@ other auxiliary data, if desired:
$ rm -rf /nix</screen>
</para>
-->
<para>You can uninstall Nix simply by running:
@@ -12,15 +12,13 @@
<listitem><para>A version of GCC or Clang that supports C++11.</para></listitem>
<listitem><para>Perl 5.8 or higher.</para></listitem>
<listitem><para><command>pkg-config</command> to locate
dependencies. If your distribution does not provide it, you can get
it from <link
xlink:href="http://www.freedesktop.org/wiki/Software/pkg-config"
/>.</para></listitem>
<listitem><para>The OpenSSL library to calculate cryptographic hashes.
If your distribution does not provide it, you can get it from <link
xlink:href="https://www.openssl.org"/>.</para></listitem>
<listitem><para>The bzip2 compressor program and the
<literal>libbz2</literal> library. Thus you must have bzip2
@@ -32,6 +30,11 @@
or higher. If your distribution does not provide it, please install
it from <link xlink:href="http://www.sqlite.org/" />.</para></listitem>
<listitem><para>The Perl DBI, DBD::SQLite, and WWW::Curl libraries, which are
available from <link
xlink:href="http://search.cpan.org/">CPAN</link> if your
distribution does not provide them.</para></listitem>
<listitem><para>The <link
xlink:href="http://www.hboehm.info/gc/">Boehm
garbage collector</link> to reduce the evaluator’s memory
+4 -11
View File
@@ -62,10 +62,9 @@ directories such as
so if a package builds correctly on your system, this is because you
specified the dependency explicitly.</para>
<para>Once a package is built, runtime dependencies are found by
scanning binaries for the hash parts of Nix store paths (such as
<literal>r8vvq9kq…</literal>). This sounds risky, but it works
extremely well.</para>
<para>Runtime dependencies are found by scanning binaries for the hash
parts of Nix store paths (such as <literal>r8vvq9kq…</literal>). This
sounds risky, but it works extremely well.</para>
</simplesect>
@@ -93,7 +92,7 @@ time window in which the package has some files from the old version
and some files from the new version — which would be bad because a
program might well crash if it’s started during that period.</para>
<para>And since packages aren’t overwritten, the old versions are still
<para>And since package aren’t overwritten, the old versions are still
there after an upgrade. This means that you can <emphasis>roll
back</emphasis> to the old version:</para>
@@ -261,12 +260,6 @@ xlink:href="http://nixos.org/">NixOS homepage</link>.</para>
xlink:href="http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html">GNU
LGPLv2.1 or (at your option) any later version</link>.</para>
<para>Nix uses the <link
xlink:href="https://github.com/antirez/linenoise">linenoise
library</link>, which has the following license:</para>
<programlisting><xi:include href="../../../src/linenoise/LICENSE" parse="text" /></programlisting>
</simplesect>
+13 -1
View File
@@ -40,7 +40,7 @@ libxslt-1.1.28
<step><para>Install some packages from the channel:
<screen>
$ nix-env -i hello</screen>
$ nix-env -i hello <replaceable>...</replaceable> </screen>
This should download pre-built packages; it should not build them
locally (if it does, something went wrong).</para></step>
@@ -95,6 +95,18 @@ The latter command will upgrade each installed package for which there
is a “newer” version (as determined by comparing the version
numbers).</para></step>
<!--
<step><para>You can also install specific packages directly from
your web browser. For instance, you can go to <link
xlink:href="http://hydra.nixos.org/jobset/nixpkgs/trunk/channel/latest"
/> and click on any link for the individual packages for your
platform. Associate <literal>application/nix-package</literal> with
the program <command>nix-install-package</command>. A window should
appear asking you whether it’s okay to install the package. Say
<literal>Y</literal>. The package and all its dependencies will be
installed.</para></step>
-->
<step><para>If you're unhappy with the result of a
<command>nix-env</command> action (e.g., an upgraded package turned
out not to work properly), you can go back:
+14 -9
View File
@@ -1,6 +1,3 @@
ifeq ($(doc_generate),yes)
XSLTPROC = $(xsltproc) --nonet $(xmlflags) \
--param section.autolabel 1 \
--param section.label.includes.component.label 1 \
@@ -42,9 +39,9 @@ dist-files += $(d)/manual.xmli $(d)/version.txt $(d)/manual.is-valid
# Generate man pages.
man-pages := $(foreach n, \
nix-env.1 nix-build.1 nix-shell.1 nix-store.1 nix-instantiate.1 \
nix-collect-garbage.1 \
nix-collect-garbage.1 nix-push.1 nix-pull.1 \
nix-prefetch-url.1 nix-channel.1 \
nix-hash.1 nix-copy-closure.1 \
nix-install-package.1 nix-hash.1 nix-copy-closure.1 \
nix.conf.5 nix-daemon.8, \
$(d)/$(n))
@@ -74,14 +71,22 @@ $(foreach file, $(wildcard $(d)/images/callouts/*.gif), $(eval $(call install-da
$(eval $(call install-symlink, manual.html, $(docdir)/manual/index.html))
all: $(d)/manual.html
clean-files += $(d)/manual.html
dist-files += $(d)/manual.html
endif
# Generate the PDF manual.
$(d)/manual.pdf: $(d)/manual.xml $(MANUAL_SRCS) $(d)/manual.is-valid
$(trace-gen) if test "$(dblatex)" != ""; then \
cd doc/manual && $(XSLTPROC) --xinclude --stringparam profile.condition manual \
$(docbookxsl)/profiling/profile.xsl manual.xml | \
$(dblatex) -o $(notdir $@) $(dblatex_opts) -; \
else \
echo "Please install dblatex and rerun configure."; \
exit 1; \
fi
clean-files += $(d)/manual.pdf
+37
View File
@@ -0,0 +1,37 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="sec-one-click">
<title>One-Click Installation</title>
<para>Often, when you want to install a specific package (e.g., from
the <link
xlink:href="http://nixos.org/nixpkgs/">Nix
Packages collection</link>), subscribing to a channel is a bit
cumbersome. And channels don’t help you at all if you want to install
an older version of a package than the one provided by the current
contents of the channel, or a package that has been removed from the
channel. That’s when <emphasis>one-click installs</emphasis> come in
handy: you can just go to the web page that contains the package,
click on it, and it will be installed with all the necessary
dependencies.</para>
<para>For instance, you can go to <link
xlink:href="http://hydra.nixos.org/jobset/nixpkgs/trunk/channel/latest"
/> and click on any link for the individual packages for your
platform. The first time you do this, your browser will ask what to
do with <literal>application/nix-package</literal> files. You should
open them with <filename>/nix/bin/nix-install-package</filename>.
This will open a window that asks you to confirm that you want to
install the package. When you answer <literal>Y</literal>, the
package and all its dependencies will be installed. This is a binary
deployment mechanism — you get packages pre-compiled for the selected
platform type.</para>
<para>You can also install <literal>application/nix-package</literal>
files from the command line directly. See <xref
linkend='sec-nix-install-package' /> for details.</para>
</chapter>
@@ -18,6 +18,7 @@ who want to <emphasis>create</emphasis> packages should consult
<xi:include href="profiles.xml" />
<xi:include href="garbage-collection.xml" />
<xi:include href="channels.xml" />
<xi:include href="one-click.xml" />
<xi:include href="sharing-packages.xml" />
</part>
@@ -12,7 +12,6 @@
</partintro>
-->
<xi:include href="rl-1.12.xml" />
<xi:include href="rl-1.11.10.xml" />
<xi:include href="rl-1.11.xml" />
<xi:include href="rl-1.10.xml" />
-31
View File
@@ -1,31 +0,0 @@
<section xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="ssec-relnotes-1.12">
<title>Release 1.12 (TBA)</title>
<para>This release has the following new features:</para>
<itemizedlist>
<listitem>
<para>It is no longer necessary to set the
<envar>NIX_REMOTE</envar> environment variable if you need to use
the Nix daemon. Nix will use the daemon automatically if you don’t
have write access to the Nix database.</para>
</listitem>
<listitem>
<para>The Nix language now supports floating point numbers. They are
based on regular C++ <literal>float</literal> and compatible with
existing integers and number-related operations. Export and import to and
from JSON and XML works, too.
</para>
</listitem>
</itemizedlist>
<para>This release has contributions from TBD.</para>
</section>
+2 -2
View File
@@ -83,8 +83,8 @@ $ nix-store -l $(which xterm)
caches).</para></listitem>
<listitem><para>The configuration option
<option>build-cores</option> now defaults to the number of available
CPU cores.</para></listitem>
<option>build-max-jobs</option> now defaults to the number of
available CPU cores.</para></listitem>
<listitem><para>Build users are now used by default when Nix is
invoked as root. This prevents builds from accidentally running as
+24
View File
@@ -0,0 +1,24 @@
Generate a private key:
$ (umask 277 && openssl genrsa -out /etc/nix/signing-key.sec 2048)
The private key should be kept secret (only readable to the Nix daemon
user).
Generate the corresponding public key:
$ openssl rsa -in /etc/nix/signing-key.sec -pubout > /etc/nix/signing-key.pub
The public key should be copied to all machines to which you want to
export store paths.
Signing:
$ nix-hash --type sha256 --flat svn.nar | openssl rsautl -sign -inkey mykey.sec > svn.nar.sign
Verifying a signature:
$ test "$(nix-hash --type sha256 --flat svn.nar)" = "$(openssl rsautl -verify -inkey mykey.pub -pubin -in svn.nar.sign)"
+4 -3
View File
@@ -3,13 +3,14 @@ ifeq ($(MAKECMDGOALS), dist)
dist-files += $(shell git --git-dir=.git ls-files || find * -type f)
endif
dist-files += configure config.h.in nix.spec perl/configure
dist-files += configure config.h.in nix.spec
clean-files += Makefile.config
GLOBAL_CXXFLAGS += -I . -I src -I src/libutil -I src/libstore -I src/libmain -I src/libexpr
GLOBAL_CXXFLAGS += -I . -I src -I src/libutil -I src/libstore -I src/libmain -I src/libexpr \
-Wno-unneeded-internal-declaration
$(foreach i, config.h $(call rwildcard, src/lib*, *.hh), \
$(foreach i, config.h $(call rwildcard, src/lib*, *.hh) src/nix-store/serve-protocol.hh, \
$(eval $(call install-file-in, $(i), $(includedir)/nix, 0644)))
$(foreach i, $(call rwildcard, src/boost, *.hpp), $(eval $(call install-file-in, $(i), $(includedir)/nix/$(patsubst src/%/,%,$(dir $(i))), 0644)))
-152
View File
@@ -1,152 +0,0 @@
#! /usr/bin/env nix-shell
#! nix-shell -i perl -p perl perlPackages.LWPUserAgent perlPackages.LWPProtocolHttps perlPackages.FileSlurp gnupg1
use strict;
use Data::Dumper;
use File::Basename;
use File::Path;
use File::Slurp;
use JSON::PP;
use LWP::UserAgent;
my $evalId = $ARGV[0] or die "Usage: $0 EVAL-ID\n";
my $releasesDir = "/home/eelco/mnt/releases";
my $nixpkgsDir = "/home/eelco/Dev/nixpkgs-pristine";
# FIXME: cut&paste from nixos-channel-scripts.
sub fetch {
my ($url, $type) = @_;
my $ua = LWP::UserAgent->new;
$ua->default_header('Accept', $type) if defined $type;
my $response = $ua->get($url);
die "could not download $url: ", $response->status_line, "\n" unless $response->is_success;
return $response->decoded_content;
}
my $evalUrl = "https://hydra.nixos.org/eval/$evalId";
my $evalInfo = decode_json(fetch($evalUrl, 'application/json'));
#print Dumper($evalInfo);
my $nixRev = $evalInfo->{jobsetevalinputs}->{nix}->{revision} or die;
my $tarballInfo = decode_json(fetch("$evalUrl/job/tarball", 'application/json'));
my $releaseName = $tarballInfo->{releasename};
$releaseName =~ /nix-(.*)$/ or die;
my $version = $1;
print STDERR "Nix revision is $nixRev, version is $version\n";
File::Path::make_path($releasesDir);
if (system("mountpoint -q $releasesDir") != 0) {
system("sshfs hydra-mirror:/releases $releasesDir") == 0 or die;
}
my $releaseDir = "$releasesDir/nix/$releaseName";
File::Path::make_path($releaseDir);
sub downloadFile {
my ($jobName, $productNr, $dstName) = @_;
my $buildInfo = decode_json(fetch("$evalUrl/job/$jobName", 'application/json'));
my $srcFile = $buildInfo->{buildproducts}->{$productNr}->{path} or die;
$dstName //= basename($srcFile);
my $dstFile = "$releaseDir/" . $dstName;
if (! -e $dstFile) {
print STDERR "downloading $srcFile to $dstFile...\n";
system("NIX_REMOTE=https://cache.nixos.org/ nix cat-store '$srcFile' > '$dstFile.tmp'") == 0
or die "unable to fetch $srcFile\n";
rename("$dstFile.tmp", $dstFile) or die;
}
my $sha256_expected = $buildInfo->{buildproducts}->{$productNr}->{sha256hash} or die;
my $sha256_actual = `nix hash-file --type sha256 '$dstFile'`;
chomp $sha256_actual;
if ($sha256_expected ne $sha256_actual) {
print STDERR "file $dstFile is corrupt\n";
exit 1;
}
write_file("$dstFile.sha256", $sha256_expected);
return ($dstFile, $sha256_expected);
}
downloadFile("tarball", "2"); # PDF
downloadFile("tarball", "3"); # .tar.bz2
my ($tarball, $tarballHash) = downloadFile("tarball", "4"); # .tar.xz
my ($tarball_i686_linux, $tarball_i686_linux_hash) = downloadFile("binaryTarball.i686-linux", "1");
my ($tarball_x86_64_linux, $tarball_x86_64_linux_hash) = downloadFile("binaryTarball.x86_64-linux", "1");
my ($tarball_x86_64_darwin, $tarball_x86_64_darwin_hash) = downloadFile("binaryTarball.x86_64-darwin", "1");
# Update Nixpkgs in a very hacky way.
system("cd $nixpkgsDir && git pull") == 0 or die;
my $oldName = `nix-instantiate --eval $nixpkgsDir -A nix.name`; chomp $oldName;
my $oldHash = `nix-instantiate --eval $nixpkgsDir -A nix.src.outputHash`; chomp $oldHash;
print STDERR "old stable version in Nixpkgs = $oldName / $oldHash\n";
my $fn = "$nixpkgsDir/pkgs/tools/package-management/nix/default.nix";
my $oldFile = read_file($fn);
$oldFile =~ s/$oldName/"$releaseName"/g;
$oldFile =~ s/$oldHash/"$tarballHash"/g;
write_file($fn, $oldFile);
$oldName =~ s/nix-//g;
$oldName =~ s/"//g;
sub getStorePath {
my ($jobName) = @_;
my $buildInfo = decode_json(fetch("$evalUrl/job/$jobName", 'application/json'));
die unless $buildInfo->{buildproducts}->{1}->{type} eq "nix-build";
return $buildInfo->{buildproducts}->{1}->{path};
}
write_file("$nixpkgsDir/nixos/modules/installer/tools/nix-fallback-paths.nix",
"{\n" .
" x86_64-linux = \"" . getStorePath("build.x86_64-linux") . "\";\n" .
" i686-linux = \"" . getStorePath("build.i686-linux") . "\";\n" .
" x86_64-darwin = \"" . getStorePath("build.x86_64-darwin") . "\";\n" .
"}\n");
system("cd $nixpkgsDir && git commit -a -m 'nix: $oldName -> $version'") == 0 or die;
# Extract the HTML manual.
File::Path::make_path("$releaseDir/manual");
system("tar xvf $tarball --strip-components=3 -C $releaseDir/manual --wildcards '*/doc/manual/*.html' '*/doc/manual/*.css' '*/doc/manual/*.gif' '*/doc/manual/*.png'") == 0 or die;
if (! -e "$releaseDir/manual/index.html") {
symlink("manual.html", "$releaseDir/manual/index.html") or die;
}
# Update the "latest" symlink.
symlink("$releaseName", "$releasesDir/nix/latest-tmp") or die;
rename("$releasesDir/nix/latest-tmp", "$releasesDir/nix/latest") or die;
# Tag the release in Git.
chdir("/home/eelco/Dev/nix-pristine") or die;
system("git remote update origin") == 0 or die;
system("git tag --force --sign $version $nixRev -m 'Tagging release $version'") == 0 or die;
# Update the website.
my $siteDir = "/home/eelco/Dev/nixos-homepage-pristine";
system("cd $siteDir && git pull") == 0 or die;
write_file("$siteDir/nix-release.tt",
"[%-\n" .
"latestNixVersion = \"$version\"\n" .
"nix_hash_i686_linux = \"$tarball_i686_linux_hash\"\n" .
"nix_hash_x86_64_linux = \"$tarball_x86_64_linux_hash\"\n" .
"nix_hash_x86_64_darwin = \"$tarball_x86_64_darwin_hash\"\n" .
"-%]\n");
system("cd $siteDir && nix-shell --run 'make nix/install nix/install.sig'") == 0 or die;
system("cd $siteDir && git commit -a -m 'Nix $version released'") == 0 or die;
+10 -10
View File
@@ -1,13 +1,14 @@
FROM alpine
FROM busybox
RUN wget -O- http://nixos.org/releases/nix/nix-1.11.7/nix-1.11.7-x86_64-linux.tar.bz2 | bzcat - | tar xf - \
&& echo "nixbld:x:30000:nixbld1,nixbld2,nixbld3,nixbld4,nixbld5,nixbld6,nixbld7,nixbld8,nixbld9,nixbld10,nixbld11,nixbld12,nixbld13,nixbld14,nixbld15,nixbld16,nixbld17,nixbld18,nixbld19,nixbld20,nixbld21,nixbld22,nixbld23,nixbld24,nixbld25,nixbld26,nixbld27,nixbld28,nixbld29,nixbld30" >> /etc/group \
&& for i in $(seq 1 30); do echo "nixbld$i:x:$((30000 + $i)):30000:::" >> /etc/passwd; done \
&& mkdir -m 0755 /nix && USER=root sh nix-*-x86_64-linux/install \
RUN set -x \
&& wget -O- http://nixos.org/releases/nix/nix-1.9/nix-1.9-x86_64-linux.tar.bz2 | \
bzcat - | tar xf - \
&& echo "nixbld:x:30000:nixbld1,nixbld10,nixbld2,nixbld3,nixbld4,nixbld5,nixbld6,nixbld7,nixbld8,nixbld9" >> /etc/group \
&& for i in $(seq 1 9); do echo "nixbld$i:x:3000$i:30000:::" >> /etc/passwd; done \
&& sed -i 's/\$HOME\/\.nix-profile\/etc\/ssl\/certs\/ca-bundle\.crt/\$HOME\/\.nix-profile\/etc\/ca-bundle\.crt/g' nix-1.9-x86_64-linux/install \
&& mkdir -m 0755 /nix && USER=root sh nix-1.9-x86_64-linux/install \
&& echo ". /root/.nix-profile/etc/profile.d/nix.sh" >> /etc/profile \
&& rm -r /nix-*-x86_64-linux \
&& apk --update add bash tar \
&& rm -rf /var/cache/apk/*
&& rm -r /nix-1.9-x86_64-linux
ONBUILD ENV \
ENV=/etc/profile \
@@ -19,5 +20,4 @@ ENV \
ENV=/etc/profile \
PATH=/root/.nix-profile/bin:/root/.nix-profile/sbin:/bin:/sbin:/usr/bin:/usr/sbin \
GIT_SSL_CAINFO=/root/.nix-profile/etc/ssl/certs/ca-bundle.crt \
NIX_SSL_CERT_FILE=/root/.nix-profile/etc/ssl/certs/ca-bundle.crt \
NIX_PATH=/nix/var/nix/profiles/per-user/root/channels/
NIX_SSL_CERT_FILE=/root/.nix-profile/etc/ssl/certs/ca-bundle.crt
+2 -84
View File
@@ -8,20 +8,6 @@
;;; Code:
(defun nix-syntax-match-antiquote (limit)
(let ((pos (next-single-char-property-change (point) 'nix-syntax-antiquote
nil limit)))
(when (and pos (> pos (point)))
(goto-char pos)
(let ((char (char-after pos)))
(pcase char
(`?$
(forward-char 2))
(`?}
(forward-char 1)))
(set-match-data (list pos (point)))
t))))
(defconst nix-font-lock-keywords
'("\\_<if\\_>" "\\_<then\\_>" "\\_<else\\_>" "\\_<assert\\_>" "\\_<with\\_>"
"\\_<let\\_>" "\\_<in\\_>" "\\_<rec\\_>" "\\_<inherit\\_>" "\\_<or\\_>"
@@ -40,8 +26,7 @@
("<[a-zA-Z0-9._\\+-]+\\(/[a-zA-Z0-9._\\+-]+\\)*>"
. font-lock-constant-face)
("[a-zA-Z0-9._\\+-]*\\(/[a-zA-Z0-9._\\+-]+\\)+"
. font-lock-constant-face)
(nix-syntax-match-antiquote 0 font-lock-preprocessor-face t))
. font-lock-constant-face))
"Font lock keywords for nix.")
(defvar nix-mode-syntax-table
@@ -53,67 +38,6 @@
table)
"Syntax table for Nix mode.")
(defun nix-syntax-propertize-escaped-antiquote ()
"Set syntax properies for escaped antiquote marks."
nil)
(defun nix-syntax-propertize-multiline-string ()
"Set syntax properies for multiline string delimiters."
(let* ((start (match-beginning 0))
(end (match-end 0))
(context (save-excursion (save-match-data (syntax-ppss start))))
(string-type (nth 3 context)))
(pcase string-type
(`t
;; inside a multiline string
;; ending multi-line string delimiter
(put-text-property (1- end) end
'syntax-table (string-to-syntax "|")))
(`nil
;; beginning multi-line string delimiter
(put-text-property start (1+ start)
'syntax-table (string-to-syntax "|"))))))
(defun nix-syntax-propertize-antiquote ()
"Set syntax properties for antiquote marks."
(let* ((start (match-beginning 0)))
(put-text-property start (1+ start)
'syntax-table (string-to-syntax "|"))
(put-text-property start (+ start 2)
'nix-syntax-antiquote t)))
(defun nix-syntax-propertize-close-brace ()
"Set syntax properties for close braces.
If a close brace `}' ends an antiquote, the next character begins a string."
(let* ((start (match-beginning 0))
(end (match-end 0))
(context (save-excursion (save-match-data (syntax-ppss start))))
(open (nth 1 context)))
(when open ;; a corresponding open-brace was found
(let* ((antiquote (get-text-property open 'nix-syntax-antiquote)))
(when antiquote
(put-text-property (+ start 1) (+ start 2)
'syntax-table (string-to-syntax "|"))
(put-text-property start (1+ start)
'nix-syntax-antiquote t))))))
(defun nix-syntax-propertize (start end)
"Special syntax properties for Nix."
;; search for multi-line string delimiters
(goto-char start)
(remove-text-properties start end '(syntax-table nil nix-syntax-antiquote nil))
(funcall
(syntax-propertize-rules
("''\\${"
(0 (ignore (nix-syntax-propertize-escaped-antiquote))))
("''"
(0 (ignore (nix-syntax-propertize-multiline-string))))
("\\${"
(0 (ignore (nix-syntax-propertize-antiquote))))
("}"
(0 (ignore (nix-syntax-propertize-close-brace)))))
start end))
(defun nix-indent-line ()
"Indent current line in a Nix expression."
(interactive)
@@ -145,13 +69,7 @@ The hook `nix-mode-hook' is run when Nix mode is started.
(set-syntax-table nix-mode-syntax-table)
;; Font lock support.
(setq-local font-lock-defaults '(nix-font-lock-keywords nil nil nil nil))
;; Special syntax properties for Nix
(setq-local syntax-propertize-function 'nix-syntax-propertize)
;; Look at text properties when parsing
(setq-local parse-sexp-lookup-properties t)
(setq font-lock-defaults '(nix-font-lock-keywords nil nil nil nil))
;; Automatic indentation [C-j].
(set (make-local-variable 'indent-line-function) 'nix-indent-line)
@@ -12,5 +12,10 @@
<string>/var/log/nix-daemon.log</string>
<key>StandardOutPath</key>
<string>/dev/null</string>
<key>EnvironmentVariables</key>
<dict>
<key>NIX_SSL_CERT_FILE</key>
<string>/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt</string>
</dict>
</dict>
</plist>
-113
View File
@@ -1,113 +0,0 @@
#!/bin/sh
#
# nix-daemon: Starts the nix package manager daemon
#
# chkconfig: 345 24 02
# description: This is a daemon which enable the multi-user mode
# of the nix package manager.
# processname: nix-daemon
# pidfile: /var/run/nix/nix-daemon.pid
### BEGIN INIT INFO
# Required-Start:
# Required-Stop:
# Should-Start:
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Short-Description: Starts the nix daemon
# Description: This is a daemon which enable the multi-user mode
# of the nix package manager.
### END INIT INFO
NIX_DAEMON_BIN=/usr/bin/nix-daemon
#NIX_DAEMON_USER="root"
NIX_DAEMON_USER="nix-daemon"
NIX_DAEMON_OPTS="--daemon"
umask 0022
if [ "$1" = 'status' ]; then
test -x $NIX_DAEMON_BIN || exit 4
else
test -x $NIX_DAEMON_BIN || exit 5
fi
# Source function library.
. /etc/init.d/functions
LOCKFILE=/var/lock/subsys/nix-daemon
RUNDIR=/var/run/nix
PIDFILE=${RUNDIR}/nix-daemon.pid
RETVAL=0
base=${0##*/}
start() {
mkdir -p ${RUNDIR}
chown ${NIX_DAEMON_USER}:${NIX_DAEMON_USER} ${RUNDIR}
echo -n $"Starting nix daemon... "
daemonize -u $NIX_DAEMON_USER -p ${PIDFILE} $NIX_DAEMON_BIN $NIX_DAEMON_OPTS
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch ${LOCKFILE}
return $RETVAL
}
stop() {
echo -n $"Shutting down nix daemon: "
killproc -p ${PIDFILE} $NIX_DAEMON_BIN
RETVAL=$?
[ $RETVAL -eq 0 ] && rm -f ${LOCKFILE} ${PIDFILE}
echo
return $RETVAL
}
reload() {
echo -n $"Reloading nix daemon... "
killproc -p ${PIDFILE} $NIX_DAEMON_BIN -HUP
RETVAL=$?
echo
return $RETVAL
}
restart() {
stop
start
}
RETVAL=0
# caller switch
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p ${PIDFILE} $NIX_DAEMON_BIN
RETVAL=$?
;;
restart)
restart
;;
reload)
reload
;;
condrestart)
if [ -f $LOCKFILE ]; then
restart
fi
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart}"
exit 2
;;
esac
exit $RETVAL
+2 -2
View File
@@ -53,8 +53,8 @@ BUILD_SHARED_LIBS ?= 1
ifeq ($(BUILD_SHARED_LIBS), 1)
ifeq (CYGWIN,$(findstring CYGWIN,$(OS)))
GLOBAL_CFLAGS += -U__STRICT_ANSI__ -D_GNU_SOURCE
GLOBAL_CXXFLAGS += -U__STRICT_ANSI__ -D_GNU_SOURCE
GLOBAL_CFLAGS += -U__STRICT_ANSI__
GLOBAL_CXXFLAGS += -U__STRICT_ANSI__
else
GLOBAL_CFLAGS += -fPIC
GLOBAL_CXXFLAGS += -fPIC
+25 -16
View File
@@ -1,14 +1,12 @@
%undefine _hardened_build
%global nixbld_user "nix-builder-"
%global nixbld_group "nixbld"
%global nixbld_group "nix-builders"
Summary: The Nix software deployment system
Name: nix
Version: @PACKAGE_VERSION@
Release: 2%{?dist}
License: LGPLv2+
%if 0%{?rhel} && 0%{?rhel} < 7
%if 0%{?rhel}
Group: Applications/System
%endif
URL: http://nixos.org/
@@ -16,7 +14,13 @@ Source0: %{name}-%{version}.tar.bz2
%if 0%{?el5}
BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)
%endif
BuildRequires: perl(DBD::SQLite)
BuildRequires: perl(DBI)
BuildRequires: perl(WWW::Curl)
BuildRequires: perl(ExtUtils::ParseXS)
Requires: /usr/bin/perl
Requires: curl
Requires: perl-DBD-SQLite
Requires: bzip2
Requires: gzip
Requires: xz
@@ -39,7 +43,7 @@ it can be used equally well under other Unix systems.
%package devel
Summary: Development files for %{name}
%if 0%{?rhel} && 0%{?rhel} < 7
%if 0%{?rhel}
Group: Development/Libraries
%endif
Requires: %{name}%{?_isa} = %{version}-%{release}
@@ -51,7 +55,7 @@ developing applications that use %{name}.
%package doc
Summary: Documentation files for %{name}
%if 0%{?rhel} && 0%{?rhel} < 7
%if 0%{?rhel}
Group: Documentation
%endif
BuildArch: noarch
@@ -63,7 +67,7 @@ The %{name}-doc package contains documentation files for %{name}.
%package -n emacs-%{name}
Summary: Nix mode for Emacs
%if 0%{?rhel} && 0%{?rhel} < 7
%if 0%{?rhel}
Group: Applications/Editors
%endif
BuildArch: noarch
@@ -75,7 +79,7 @@ This package provides a major mode for editing Nix expressions.
%package -n emacs-%{name}-el
Summary: Elisp source files for emacs-%{name}
%if 0%{?rhel} && 0%{?rhel} < 7
%if 0%{?rhel}
Group: Applications/Editors
%endif
BuildArch: noarch
@@ -89,6 +93,11 @@ the emacs-%{name} package to edit Nix expressions with GNU Emacs.
%prep
%setup -q
# Install Perl modules to vendor_perl
# configure.ac need to be changed to make this global; however, this will
# also affect NixOS. Use discretion.
%{__sed} -i 's|perl5/site_perl/$perlversion/$perlarchname|perl5/vendor_perl|' \
configure
%build
@@ -99,7 +108,7 @@ extraFlags=
%configure --localstatedir=/nix/var \
--docdir=%{_defaultdocdir}/%{name}-doc-%{version} \
$extraFlags
make -j$NIX_BUILD_CORES -l$NIX_BUILD_CORES
make %{?_smp_flags}
%{_emacs_bytecompile} misc/emacs/nix-mode.el
@@ -111,10 +120,6 @@ make DESTDIR=$RPM_BUILD_ROOT install
find $RPM_BUILD_ROOT -name '*.la' -exec rm -f {} ';'
# make the store
mkdir -p $RPM_BUILD_ROOT/nix/store
chmod 1775 $RPM_BUILD_ROOT/nix/store
# make per-user directories
for d in profiles gcroots;
do
@@ -152,17 +157,20 @@ done
%post
chgrp %{nixbld_group} /nix/store
%if ! 0%{?rhel} || 0%{?rhel} >= 7
chmod 1775 /nix/store
%if ! 0%{?rhel}
# Enable and start Nix worker
systemctl enable nix-daemon.socket nix-daemon.service
systemctl start nix-daemon.socket
%endif
%files
%{_bindir}/nix*
%{_bindir}/nix-*
%{_libdir}/*.so
%{perl_vendorarch}/*
%exclude %dir %{perl_vendorarch}/auto/
%{_prefix}/libexec/*
%if ! 0%{?rhel} || 0%{?rhel} >= 7
%if ! 0%{?rhel}
%{_prefix}/lib/systemd/system/nix-daemon.socket
%{_prefix}/lib/systemd/system/nix-daemon.service
%endif
@@ -172,6 +180,7 @@ systemctl start nix-daemon.socket
%{_mandir}/man5/*.5*
%{_mandir}/man8/*.8*
%config(noreplace) %{_sysconfdir}/profile.d/nix.sh
%config(noreplace) %{_sysconfdir}/profile.d/nix-daemon.sh
/nix
%files devel
-14
View File
@@ -1,14 +0,0 @@
makefiles = local.mk
GLOBAL_CXXFLAGS += -std=c++14 -g -Wall
-include Makefile.config
OPTIMIZE = 1
ifeq ($(OPTIMIZE), 1)
GLOBAL_CFLAGS += -O3
GLOBAL_CXXFLAGS += -O3
endif
include mk/lib.mk
-18
View File
@@ -1,18 +0,0 @@
CC = @CC@
CFLAGS = @CFLAGS@
CXX = @CXX@
CXXFLAGS = @CXXFLAGS@
HAVE_SODIUM = @HAVE_SODIUM@
PACKAGE_NAME = @PACKAGE_NAME@
PACKAGE_VERSION = @PACKAGE_VERSION@
SODIUM_LIBS = @SODIUM_LIBS@
NIX_CFLAGS = @NIX_CFLAGS@
NIX_LIBS = @NIX_LIBS@
nixbindir = @nixbindir@
curl = @curl@
nixlibexecdir = @nixlibexecdir@
nixlocalstatedir = @nixlocalstatedir@
perl = @perl@
perllibdir = @perllibdir@
nixstoredir = @nixstoredir@
nixsysconfdir = @nixsysconfdir@
-99
View File
@@ -1,99 +0,0 @@
AC_INIT(nix-perl, m4_esyscmd([bash -c "echo -n $(cat ../version)$VERSION_SUFFIX"]))
AC_CONFIG_SRCDIR(MANIFEST)
AC_CONFIG_AUX_DIR(../config)
CFLAGS=
CXXFLAGS=
AC_PROG_CC
AC_PROG_CXX
AX_CXX_COMPILE_STDCXX_11
# Use 64-bit file system calls so that we can support files > 2 GiB.
AC_SYS_LARGEFILE
AC_DEFUN([NEED_PROG],
[
AC_PATH_PROG($1, $2)
if test -z "$$1"; then
AC_MSG_ERROR([$2 is required])
fi
])
NEED_PROG(perl, perl)
NEED_PROG(curl, curl)
NEED_PROG(bzip2, bzip2)
NEED_PROG(xz, xz)
# Test that Perl has the open/fork feature (Perl 5.8.0 and beyond).
AC_MSG_CHECKING([whether Perl is recent enough])
if ! $perl -e 'open(FOO, "-|", "true"); while (<FOO>) { print; }; close FOO or die;'; then
AC_MSG_RESULT(no)
AC_MSG_ERROR([Your Perl version is too old. Nix requires Perl 5.8.0 or newer.])
fi
AC_MSG_RESULT(yes)
# Figure out where to install Perl modules.
AC_MSG_CHECKING([for the Perl installation prefix])
perlversion=$($perl -e 'use Config; print $Config{version};')
perlarchname=$($perl -e 'use Config; print $Config{archname};')
AC_SUBST(perllibdir, [${libdir}/perl5/site_perl/$perlversion/$perlarchname])
AC_MSG_RESULT($perllibdir)
# Look for libsodium, an optional dependency.
PKG_CHECK_MODULES([SODIUM], [libsodium],
[AC_DEFINE([HAVE_SODIUM], [1], [Whether to use libsodium for cryptography.])
CXXFLAGS="$SODIUM_CFLAGS $CXXFLAGS"
have_sodium=1], [have_sodium=])
AC_SUBST(HAVE_SODIUM, [$have_sodium])
# Check for the required Perl dependencies (DBI and DBD::SQLite).
perlFlags="-I$perllibdir"
AC_ARG_WITH(dbi, AC_HELP_STRING([--with-dbi=PATH],
[prefix of the Perl DBI library]),
perlFlags="$perlFlags -I$withval")
AC_ARG_WITH(dbd-sqlite, AC_HELP_STRING([--with-dbd-sqlite=PATH],
[prefix of the Perl DBD::SQLite library]),
perlFlags="$perlFlags -I$withval")
AC_MSG_CHECKING([whether DBD::SQLite works])
if ! $perl $perlFlags -e 'use DBI; use DBD::SQLite;' 2>&5; then
AC_MSG_RESULT(no)
AC_MSG_FAILURE([The Perl modules DBI and/or DBD::SQLite are missing.])
fi
AC_MSG_RESULT(yes)
AC_SUBST(perlFlags)
PKG_CHECK_MODULES([NIX], [nix-store])
NEED_PROG([NIX_INSTANTIATE_PROGRAM], [nix-instantiate])
# Get nix configure values
nixbindir=$("$NIX_INSTANTIATE_PROGRAM" --eval '<nix/config.nix>' -A nixBinDir | tr -d \")
nixlibexecdir=$("$NIX_INSTANTIATE_PROGRAM" --eval '<nix/config.nix>' -A nixLibexecDir | tr -d \")
nixlocalstatedir=$("$NIX_INSTANTIATE_PROGRAM" --eval '<nix/config.nix>' -A nixLocalstateDir | tr -d \")
nixsysconfdir=$("$NIX_INSTANTIATE_PROGRAM" --eval '<nix/config.nix>' -A nixSysconfDir | tr -d \")
nixstoredir=$("$NIX_INSTANTIATE_PROGRAM" --eval '<nix/config.nix>' -A nixStoreDir | tr -d \")
AC_SUBST(nixbindir)
AC_SUBST(nixlibexecdir)
AC_SUBST(nixlocalstatedir)
AC_SUBST(nixsysconfdir)
AC_SUBST(nixstoredir)
# Expand all variables in config.status.
test "$prefix" = NONE && prefix=$ac_default_prefix
test "$exec_prefix" = NONE && exec_prefix='${prefix}'
for name in $ac_subst_vars; do
declare $name="$(eval echo "${!name}")"
declare $name="$(eval echo "${!name}")"
declare $name="$(eval echo "${!name}")"
done
rm -f Makefile.config
ln -sfn ../mk mk
AC_CONFIG_FILES([])
AC_OUTPUT
+40 -15
View File
@@ -4,31 +4,56 @@ use MIME::Base64;
$version = "@PACKAGE_VERSION@";
$binDir = $ENV{"NIX_BIN_DIR"} || "@nixbindir@";
$libexecDir = $ENV{"NIX_LIBEXEC_DIR"} || "@nixlibexecdir@";
$stateDir = $ENV{"NIX_STATE_DIR"} || "@nixlocalstatedir@/nix";
$logDir = $ENV{"NIX_LOG_DIR"} || "@nixlocalstatedir@/log/nix";
$confDir = $ENV{"NIX_CONF_DIR"} || "@nixsysconfdir@/nix";
$storeDir = $ENV{"NIX_STORE_DIR"} || "@nixstoredir@";
$binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
$libexecDir = $ENV{"NIX_LIBEXEC_DIR"} || "@libexecdir@";
$stateDir = $ENV{"NIX_STATE_DIR"} || "@localstatedir@/nix";
$manifestDir = $ENV{"NIX_MANIFESTS_DIR"} || "@localstatedir@/nix/manifests";
$logDir = $ENV{"NIX_LOG_DIR"} || "@localstatedir@/log/nix";
$confDir = $ENV{"NIX_CONF_DIR"} || "@sysconfdir@/nix";
$storeDir = $ENV{"NIX_STORE_DIR"} || "@storedir@";
$caBundle = $ENV{"NIX_SSL_CERT_FILE"} // $ENV{"SSL_CERT_FILE"} // $ENV{"CURL_CA_BUNDLE"} // $ENV{"OPENSSL_X509_CERT_FILE"};
$caBundle = "/etc/ssl/certs/ca-bundle.crt" if !$caBundle && -f "/etc/ssl/certs/ca-bundle.crt";
$caBundle = "/etc/ssl/certs/ca-certificates.crt" if !$caBundle && -f "/etc/ssl/certs/ca-certificates.crt";
$curlCaFlag = defined $caBundle ? "--cacert $caBundle" : "";
$bzip2 = "@bzip2@";
$xz = "@xz@";
$curl = "@curl@";
$openssl = "@openssl@";
$useBindings = 1;
$useBindings = "@perlbindings@" eq "yes";
%config = ();
sub readConfig {
my $config = "$confDir/nix.conf";
return unless -f $config;
%binaryCachePublicKeys = ();
open CONFIG, "<$config" or die "cannot open ‘$config’";
while (<CONFIG>) {
/^\s*([\w\-\.]+)\s*=\s*(.*)$/ or next;
$config{$1} = $2;
$defaultPublicKeys = "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=";
sub readConfig {
if (defined $ENV{'_NIX_OPTIONS'}) {
foreach my $s (split '\n', $ENV{'_NIX_OPTIONS'}) {
my ($n, $v) = split '=', $s, 2;
$config{$n} = $v;
}
} else {
my $config = "$confDir/nix.conf";
return unless -f $config;
open CONFIG, "<$config" or die "cannot open ‘$config’";
while (<CONFIG>) {
/^\s*([\w\-\.]+)\s*=\s*(.*)$/ or next;
$config{$1} = $2;
}
close CONFIG;
}
foreach my $s (split(/ /, $config{"binary-cache-public-keys"} // $defaultPublicKeys)) {
my ($keyName, $publicKey) = split ":", $s;
next unless defined $keyName && defined $publicKey;
$binaryCachePublicKeys{$keyName} = decode_base64($publicKey);
}
close CONFIG;
}
return 1;
+59 -5
View File
@@ -10,7 +10,7 @@ use IPC::Open2;
sub copyToOpen {
my ($from, $to, $sshHost, $storePaths, $includeOutputs, $dryRun, $useSubstitutes) = @_;
my ($from, $to, $sshHost, $storePaths, $includeOutputs, $dryRun, $sign, $useSubstitutes) = @_;
$useSubstitutes = 0 if $dryRun || !defined $useSubstitutes;
@@ -41,21 +41,75 @@ sub copyToOpen {
# Send the "import paths" command.
syswrite($to, pack("L<x4", 4)) or die;
exportPaths(fileno($to), @missing);
exportPaths(fileno($to), $sign, @missing);
readInt($from) == 1 or die "remote machine ‘$sshHost’ failed to import closure\n";
}
sub copyTo {
my ($sshHost, $storePaths, $includeOutputs, $dryRun, $useSubstitutes) = @_;
my ($sshHost, $storePaths, $includeOutputs, $dryRun, $sign, $useSubstitutes) = @_;
# Connect to the remote host.
my ($from, $to) = connectToRemoteNix($sshHost, []);
my ($from, $to);
eval {
($from, $to) = connectToRemoteNix($sshHost, []);
};
if ($@) {
chomp $@;
warn "$@; falling back to old closure copying method\n";
$@ = "";
return oldCopyTo(@_);
}
copyToOpen($from, $to, $sshHost, $storePaths, $includeOutputs, $dryRun, $useSubstitutes);
copyToOpen($from, $to, $sshHost, $storePaths, $includeOutputs, $dryRun, $sign, $useSubstitutes);
close $to;
}
# For backwards compatibility with Nix <= 1.7. Will be removed
# eventually.
sub oldCopyTo {
my ($sshHost, $storePaths, $includeOutputs, $dryRun, $sign, $useSubstitutes) = @_;
# Get the closure of this path.
my @closure = reverse(topoSortPaths(computeFSClosure(0, $includeOutputs,
map { followLinksToStorePath $_ } @{$storePaths})));
# Optionally use substitutes on the remote host.
if (!$dryRun && $useSubstitutes) {
system "ssh $sshHost @globalSshOpts nix-store -r --ignore-unknown @closure";
# Ignore exit status because this is just an optimisation.
}
# Ask the remote host which paths are invalid. Because of limits
# to the command line length, do this in chunks. Eventually,
# we'll want to use ‘--from-stdin’, but we can't rely on the
# target having this option yet.
my @missing;
my $missingSize = 0;
while (scalar(@closure) > 0) {
my @ps = splice(@closure, 0, 1500);
open(READ, "set -f; ssh $sshHost @globalSshOpts nix-store --check-validity --print-invalid @ps|");
while (<READ>) {
chomp;
push @missing, $_;
my ($deriver, $narHash, $time, $narSize, $refs) = queryPathInfo($_, 1);
$missingSize += $narSize;
}
close READ or die;
}
# Export the store paths and import them on the remote machine.
if (scalar @missing > 0) {
print STDERR "copying ", scalar @missing, " missing paths to ‘$sshHost’...\n";
unless ($dryRun) {
open SSH, "| ssh $sshHost @globalSshOpts 'nix-store --import' > /dev/null" or die;
exportPaths(fileno(SSH), $sign, @missing);
close SSH or die "copying store paths to remote machine ‘$sshHost’ failed: $?";
}
}
}
1;
+340
View File
@@ -0,0 +1,340 @@
package Nix::GeneratePatches;
use strict;
use File::Temp qw(tempdir);
use File::stat;
use Nix::Config;
use Nix::Manifest;
our @ISA = qw(Exporter);
our @EXPORT = qw(generatePatches propagatePatches copyPatches);
# Some patch generations options.
# Max size of NAR archives to generate patches for.
my $maxNarSize = $ENV{"NIX_MAX_NAR_SIZE"};
$maxNarSize = 160 * 1024 * 1024 if !defined $maxNarSize;
# If patch is bigger than this fraction of full archive, reject.
my $maxPatchFraction = $ENV{"NIX_PATCH_FRACTION"};
$maxPatchFraction = 0.60 if !defined $maxPatchFraction;
my $timeLimit = $ENV{"NIX_BSDIFF_TIME_LIMIT"};
$timeLimit = 180 if !defined $timeLimit;
my $hashAlgo = "sha256";
sub findOutputPaths {
my $narFiles = shift;
my %outPaths;
foreach my $p (keys %{$narFiles}) {
# Ignore derivations.
next if ($p =~ /\.drv$/);
# Ignore builders (too much ambiguity -- they're all called
# `builder.sh').
next if ($p =~ /\.sh$/);
next if ($p =~ /\.patch$/);
# Don't bother including tar files etc.
next if ($p =~ /\.tar$/ || $p =~ /\.tar\.(gz|bz2|Z|lzma|xz)$/ || $p =~ /\.zip$/ || $p =~ /\.bin$/ || $p =~ /\.tgz$/ || $p =~ /\.rpm$/ || $p =~ /cvs-export$/ || $p =~ /fetchhg$/);
$outPaths{$p} = 1;
}
return %outPaths;
}
sub getNameVersion {
my $p = shift;
$p =~ /\/[0-9a-z]+((?:-[a-zA-Z][^\/-]*)+)([^\/]*)$/;
my $name = $1;
my $version = $2;
return undef unless defined $name && defined $version;
$name =~ s/^-//;
$version =~ s/^-//;
return ($name, $version);
}
# A quick hack to get a measure of the `distance' between two
# versions: it's just the position of the first character that differs
# (or 999 if they are the same).
sub versionDiff {
my $s = shift;
my $t = shift;
my $i;
return 999 if $s eq $t;
for ($i = 0; $i < length $s; $i++) {
return $i if $i >= length $t or
substr($s, $i, 1) ne substr($t, $i, 1);
}
return $i;
}
sub getNarBz2 {
my $narPath = shift;
my $narFiles = shift;
my $storePath = shift;
my $narFileList = $$narFiles{$storePath};
die "missing path $storePath" unless defined $narFileList;
my $narFile = @{$narFileList}[0];
die unless defined $narFile;
$narFile->{url} =~ /\/([^\/]+)$/;
die unless defined $1;
return "$narPath/$1";
}
sub containsPatch {
my $patches = shift;
my $storePath = shift;
my $basePath = shift;
my $patchList = $$patches{$storePath};
return 0 if !defined $patchList;
my $found = 0;
foreach my $patch (@{$patchList}) {
# !!! baseHash might differ
return 1 if $patch->{basePath} eq $basePath;
}
return 0;
}
sub generatePatches {
my ($srcNarFiles, $dstNarFiles, $srcPatches, $dstPatches, $narPath, $patchesPath, $patchesURL, $tmpDir) = @_;
my %srcOutPaths = findOutputPaths $srcNarFiles;
my %dstOutPaths = findOutputPaths $dstNarFiles;
# For each output path in the destination, see if we need to / can
# create a patch.
print STDERR "creating patches...\n";
foreach my $p (keys %dstOutPaths) {
# If exactly the same path already exists in the source, skip it.
next if defined $srcOutPaths{$p};
print " $p\n";
# If not, then we should find the paths in the source that are
# `most' likely to be present on a system that wants to
# install this path.
(my $name, my $version) = getNameVersion $p;
next unless defined $name && defined $version;
my @closest = ();
my $closestVersion;
my $minDist = -1; # actually, larger means closer
# Find all source paths with the same name.
foreach my $q (keys %srcOutPaths) {
(my $name2, my $version2) = getNameVersion $q;
next unless defined $name2 && defined $version2;
if ($name eq $name2) {
my $srcSystem = @{$$dstNarFiles{$p}}[0]->{system};
my $dstSystem = @{$$srcNarFiles{$q}}[0]->{system};
if (defined $srcSystem && defined $dstSystem && $srcSystem ne $dstSystem) {
print " SKIPPING $q due to different systems ($srcSystem vs. $dstSystem)\n";
next;
}
# If the sizes differ too much, then skip. This
# disambiguates between, e.g., a real component and a
# wrapper component (cf. Firefox in Nixpkgs).
my $srcSize = @{$$srcNarFiles{$q}}[0]->{size};
my $dstSize = @{$$dstNarFiles{$p}}[0]->{size};
my $ratio = $srcSize / $dstSize;
$ratio = 1 / $ratio if $ratio < 1;
# print " SIZE $srcSize $dstSize $ratio $q\n";
if ($ratio >= 3) {
print " SKIPPING $q due to size ratio $ratio ($srcSize vs. $dstSize)\n";
next;
}
# If there are multiple matching names, include the
# ones with the closest version numbers.
my $dist = versionDiff $version, $version2;
if ($dist > $minDist) {
$minDist = $dist;
@closest = ($q);
$closestVersion = $version2;
} elsif ($dist == $minDist) {
push @closest, $q;
}
}
}
if (scalar(@closest) == 0) {
print " NO BASE: $p\n";
next;
}
foreach my $closest (@closest) {
# Generate a patch between $closest and $p.
print STDERR " $p <- $closest\n";
# If the patch already exists, skip it.
if (containsPatch($srcPatches, $p, $closest) ||
containsPatch($dstPatches, $p, $closest))
{
print " skipping, already exists\n";
next;
}
my $srcNarBz2 = getNarBz2 $narPath, $srcNarFiles, $closest;
my $dstNarBz2 = getNarBz2 $narPath, $dstNarFiles, $p;
if (! -f $srcNarBz2) {
warn "patch source archive $srcNarBz2 is missing\n";
next;
}
system("$Nix::Config::bzip2 -d < $srcNarBz2 > $tmpDir/A") == 0
or die "cannot unpack $srcNarBz2";
if (stat("$tmpDir/A")->size >= $maxNarSize) {
print " skipping, source is too large\n";
next;
}
system("$Nix::Config::bzip2 -d < $dstNarBz2 > $tmpDir/B") == 0
or die "cannot unpack $dstNarBz2";
if (stat("$tmpDir/B")->size >= $maxNarSize) {
print " skipping, destination is too large\n";
next;
}
my $time1 = time();
my $res = system("ulimit -t $timeLimit; $Nix::Config::libexecDir/nix/bsdiff $tmpDir/A $tmpDir/B $tmpDir/DIFF");
my $time2 = time();
if ($res) {
warn "binary diff computation aborted after ", $time2 - $time1, " seconds\n";
next;
}
my $baseHash = `$Nix::Config::binDir/nix-hash --flat --type $hashAlgo --base32 $tmpDir/A` or die;
chomp $baseHash;
my $narHash = `$Nix::Config::binDir/nix-hash --flat --type $hashAlgo --base32 $tmpDir/B` or die;
chomp $narHash;
my $narDiffHash = `$Nix::Config::binDir/nix-hash --flat --type $hashAlgo --base32 $tmpDir/DIFF` or die;
chomp $narDiffHash;
my $narDiffSize = stat("$tmpDir/DIFF")->size;
my $dstNarBz2Size = stat($dstNarBz2)->size;
print " size $narDiffSize; full size $dstNarBz2Size; ", $time2 - $time1, " seconds\n";
if ($narDiffSize >= $dstNarBz2Size) {
print " rejecting; patch bigger than full archive\n";
next;
}
if ($narDiffSize / $dstNarBz2Size >= $maxPatchFraction) {
print " rejecting; patch too large relative to full archive\n";
next;
}
my $finalName = "$narDiffHash.nar-bsdiff";
if (-e "$patchesPath/$finalName") {
print " not copying, already exists\n";
}
else {
system("cp '$tmpDir/DIFF' '$patchesPath/$finalName.tmp'") == 0
or die "cannot copy diff";
rename("$patchesPath/$finalName.tmp", "$patchesPath/$finalName")
or die "cannot rename $patchesPath/$finalName.tmp";
}
# Add the patch to the manifest.
addPatch $dstPatches, $p,
{ url => "$patchesURL/$finalName", hash => "$hashAlgo:$narDiffHash"
, size => $narDiffSize, basePath => $closest, baseHash => "$hashAlgo:$baseHash"
, narHash => "$hashAlgo:$narHash", patchType => "nar-bsdiff"
};
}
}
}
# Propagate useful patches from $srcPatches to $dstPatches. A patch
# is useful if it produces either paths in the $dstNarFiles or paths
# that can be used as the base for other useful patches.
sub propagatePatches {
my ($srcPatches, $dstNarFiles, $dstPatches) = @_;
print STDERR "propagating patches...\n";
my $changed;
do {
# !!! we repeat this to reach the transitive closure; inefficient
$changed = 0;
print STDERR "loop\n";
my %dstBasePaths;
foreach my $q (keys %{$dstPatches}) {
foreach my $patch (@{$$dstPatches{$q}}) {
$dstBasePaths{$patch->{basePath}} = 1;
}
}
foreach my $p (keys %{$srcPatches}) {
my $patchList = $$srcPatches{$p};
my $include = 0;
# Is path $p included in the destination? If so, include
# patches that produce it.
$include = 1 if defined $$dstNarFiles{$p};
# Is path $p a path that serves as a base for paths in the
# destination? If so, include patches that produce it.
# !!! check baseHash
$include = 1 if defined $dstBasePaths{$p};
if ($include) {
foreach my $patch (@{$patchList}) {
$changed = 1 if addPatch $dstPatches, $p, $patch;
}
}
}
} while $changed;
}
# Add all new patches in $srcPatches to $dstPatches.
sub copyPatches {
my ($srcPatches, $dstPatches) = @_;
foreach my $p (keys %{$srcPatches}) {
addPatch $dstPatches, $p, $_ foreach @{$$srcPatches{$p}};
}
}
return 1;
+167 -1
View File
@@ -13,7 +13,7 @@ use Nix::Config;
use Nix::Store;
our @ISA = qw(Exporter);
our @EXPORT = qw(readManifest writeManifest addPatch parseNARInfo fingerprintPath);
our @EXPORT = qw(readManifest writeManifest updateManifestDB addPatch deleteOldManifests parseNARInfo fingerprintPath);
sub addNAR {
@@ -228,6 +228,172 @@ sub writeManifest {
}
sub updateManifestDB {
my $manifestDir = $Nix::Config::manifestDir;
my @manifests = glob "$manifestDir/*.nixmanifest";
return undef if scalar @manifests == 0;
mkpath($manifestDir);
unlink "$manifestDir/cache.sqlite"; # remove obsolete cache
my $dbPath = "$manifestDir/cache-v2.sqlite";
# Open/create the database.
our $dbh = DBI->connect("dbi:SQLite:dbname=$dbPath", "", "")
or die "cannot open database ‘$dbPath’";
$dbh->{RaiseError} = 1;
$dbh->{PrintError} = 0;
$dbh->do("pragma foreign_keys = on");
$dbh->do("pragma synchronous = off"); # we can always reproduce the cache
$dbh->do("pragma journal_mode = truncate");
# Initialise the database schema, if necessary.
$dbh->do(<<EOF);
create table if not exists Manifests (
id integer primary key autoincrement not null,
path text unique not null,
timestamp integer not null
);
EOF
$dbh->do(<<EOF);
create table if not exists NARs (
id integer primary key autoincrement not null,
manifest integer not null,
storePath text not null,
url text not null,
compressionType text not null,
hash text,
size integer,
narHash text,
narSize integer,
refs text,
deriver text,
system text,
foreign key (manifest) references Manifests(id) on delete cascade
);
EOF
$dbh->do("create index if not exists NARs_storePath on NARs(storePath)");
$dbh->do(<<EOF);
create table if not exists Patches (
id integer primary key autoincrement not null,
manifest integer not null,
storePath text not null,
basePath text not null,
baseHash text not null,
url text not null,
hash text,
size integer,
narHash text,
narSize integer,
patchType text not null,
foreign key (manifest) references Manifests(id) on delete cascade
);
EOF
$dbh->do("create index if not exists Patches_storePath on Patches(storePath)");
# Acquire an exclusive lock to ensure that only one process
# updates the DB at the same time. This isn't really necessary,
# but it prevents work duplication and lock contention in SQLite.
my $lockFile = "$manifestDir/cache.lock";
open MAINLOCK, ">>$lockFile" or die "unable to acquire lock ‘$lockFile’: $!\n";
flock(MAINLOCK, LOCK_EX) or die;
our $insertNAR = $dbh->prepare(
"insert into NARs(manifest, storePath, url, compressionType, hash, size, narHash, " .
"narSize, refs, deriver, system) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)") or die;
our $insertPatch = $dbh->prepare(
"insert into Patches(manifest, storePath, basePath, baseHash, url, hash, " .
"size, narHash, narSize, patchType) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
$dbh->begin_work;
# Read each manifest in $manifestDir and add it to the database,
# unless we've already done so on a previous run.
my %seen;
for my $manifestLink (@manifests) {
my $manifest = Cwd::abs_path($manifestLink);
next unless -f $manifest;
my $timestamp = lstat($manifest)->mtime;
$seen{$manifest} = 1;
next if scalar @{$dbh->selectcol_arrayref(
"select 1 from Manifests where path = ? and timestamp = ?",
{}, $manifest, $timestamp)} == 1;
print STDERR "caching $manifest...\n";
$dbh->do("delete from Manifests where path = ?", {}, $manifest);
$dbh->do("insert into Manifests(path, timestamp) values (?, ?)",
{}, $manifest, $timestamp);
our $id = $dbh->last_insert_id("", "", "", "");
sub addNARToDB {
my ($storePath, $narFile) = @_;
$insertNAR->execute(
$id, $storePath, $narFile->{url}, $narFile->{compressionType}, $narFile->{hash},
$narFile->{size}, $narFile->{narHash}, $narFile->{narSize}, $narFile->{references},
$narFile->{deriver}, $narFile->{system});
};
sub addPatchToDB {
my ($storePath, $patch) = @_;
$insertPatch->execute(
$id, $storePath, $patch->{basePath}, $patch->{baseHash}, $patch->{url},
$patch->{hash}, $patch->{size}, $patch->{narHash}, $patch->{narSize},
$patch->{patchType});
};
my $version = readManifest_($manifest, \&addNARToDB, \&addPatchToDB);
if ($version < 3) {
die "you have an old-style or corrupt manifest ‘$manifestLink’; please delete it\n";
}
if ($version >= 10) {
die "manifest ‘$manifestLink’ is too new; please delete it or upgrade Nix\n";
}
}
# Removed cached information for removed manifests from the DB.
foreach my $manifest (@{$dbh->selectcol_arrayref("select path from Manifests")}) {
next if defined $seen{$manifest};
$dbh->do("delete from Manifests where path = ?", {}, $manifest);
}
$dbh->commit;
close MAINLOCK;
return $dbh;
}
# Delete all old manifests downloaded from a given URL.
sub deleteOldManifests {
my ($url, $curUrlFile) = @_;
for my $urlFile (glob "$Nix::Config::manifestDir/*.url") {
next if defined $curUrlFile && $urlFile eq $curUrlFile;
open URL, "<$urlFile" or die;
my $url2 = <URL>;
chomp $url2;
close URL;
next unless $url eq $url2;
my $base = $urlFile; $base =~ s/.url$//;
unlink "${base}.url";
unlink "${base}.nixmanifest";
}
}
# Return a fingerprint of a store path to be used in binary cache
# signatures. It contains the store path, the base-32 SHA-256 hash of
# the contents of the path, and the references.
+61 -41
View File
@@ -1,5 +1,3 @@
#include "config.h"
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
@@ -8,11 +6,10 @@
#undef do_open
#undef do_close
#include "derivations.hh"
#include "globals.hh"
#include "store-api.hh"
#include "util.hh"
#include "crypto.hh"
#include <store-api.hh>
#include <globals.hh>
#include <misc.hh>
#include <util.hh>
#if HAVE_SODIUM
#include <sodium.h>
@@ -22,19 +19,19 @@
using namespace nix;
static ref<Store> store()
void doInit()
{
static std::shared_ptr<Store> _store;
if (!_store) {
if (!store) {
try {
settings.processEnvironment();
settings.loadConfFile();
settings.update();
settings.lockCPU = false;
_store = openStore();
store = openStore();
} catch (Error & e) {
croak("%s", e.what());
}
}
return ref<Store>(_store);
}
@@ -48,7 +45,7 @@ PROTOTYPES: ENABLE
void init()
CODE:
store();
doInit();
void setVerbosity(int level)
@@ -59,7 +56,8 @@ void setVerbosity(int level)
int isValidPath(char * path)
CODE:
try {
RETVAL = store()->isValidPath(path);
doInit();
RETVAL = store->isValidPath(path);
} catch (Error & e) {
croak("%s", e.what());
}
@@ -70,7 +68,9 @@ int isValidPath(char * path)
SV * queryReferences(char * path)
PPCODE:
try {
PathSet paths = store()->queryPathInfo(path)->references;
doInit();
PathSet paths;
store->queryReferences(path, paths);
for (PathSet::iterator i = paths.begin(); i != paths.end(); ++i)
XPUSHs(sv_2mortal(newSVpv(i->c_str(), 0)));
} catch (Error & e) {
@@ -81,7 +81,8 @@ SV * queryReferences(char * path)
SV * queryPathHash(char * path)
PPCODE:
try {
auto hash = store()->queryPathInfo(path)->narHash;
doInit();
Hash hash = store->queryPathHash(path);
string s = "sha256:" + printHash32(hash);
XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
} catch (Error & e) {
@@ -92,7 +93,8 @@ SV * queryPathHash(char * path)
SV * queryDeriver(char * path)
PPCODE:
try {
auto deriver = store()->queryPathInfo(path)->deriver;
doInit();
Path deriver = store->queryDeriver(path);
if (deriver == "") XSRETURN_UNDEF;
XPUSHs(sv_2mortal(newSVpv(deriver.c_str(), 0)));
} catch (Error & e) {
@@ -103,17 +105,18 @@ SV * queryDeriver(char * path)
SV * queryPathInfo(char * path, int base32)
PPCODE:
try {
auto info = store()->queryPathInfo(path);
if (info->deriver == "")
doInit();
ValidPathInfo info = store->queryPathInfo(path);
if (info.deriver == "")
XPUSHs(&PL_sv_undef);
else
XPUSHs(sv_2mortal(newSVpv(info->deriver.c_str(), 0)));
string s = "sha256:" + (base32 ? printHash32(info->narHash) : printHash(info->narHash));
XPUSHs(sv_2mortal(newSVpv(info.deriver.c_str(), 0)));
string s = "sha256:" + (base32 ? printHash32(info.hash) : printHash(info.hash));
XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
mXPUSHi(info->registrationTime);
mXPUSHi(info->narSize);
mXPUSHi(info.registrationTime);
mXPUSHi(info.narSize);
AV * arr = newAV();
for (PathSet::iterator i = info->references.begin(); i != info->references.end(); ++i)
for (PathSet::iterator i = info.references.begin(); i != info.references.end(); ++i)
av_push(arr, newSVpv(i->c_str(), 0));
XPUSHs(sv_2mortal(newRV((SV *) arr)));
} catch (Error & e) {
@@ -124,7 +127,8 @@ SV * queryPathInfo(char * path, int base32)
SV * queryPathFromHashPart(char * hashPart)
PPCODE:
try {
Path path = store()->queryPathFromHashPart(hashPart);
doInit();
Path path = store->queryPathFromHashPart(hashPart);
XPUSHs(sv_2mortal(newSVpv(path.c_str(), 0)));
} catch (Error & e) {
croak("%s", e.what());
@@ -134,9 +138,10 @@ SV * queryPathFromHashPart(char * hashPart)
SV * computeFSClosure(int flipDirection, int includeOutputs, ...)
PPCODE:
try {
doInit();
PathSet paths;
for (int n = 2; n < items; ++n)
store()->computeFSClosure(SvPV_nolen(ST(n)), paths, flipDirection, includeOutputs);
computeFSClosure(*store, SvPV_nolen(ST(n)), paths, flipDirection, includeOutputs);
for (PathSet::iterator i = paths.begin(); i != paths.end(); ++i)
XPUSHs(sv_2mortal(newSVpv(i->c_str(), 0)));
} catch (Error & e) {
@@ -147,9 +152,10 @@ SV * computeFSClosure(int flipDirection, int includeOutputs, ...)
SV * topoSortPaths(...)
PPCODE:
try {
doInit();
PathSet paths;
for (int n = 0; n < items; ++n) paths.insert(SvPV_nolen(ST(n)));
Paths sorted = store()->topoSortPaths(paths);
Paths sorted = topoSortPaths(*store, paths);
for (Paths::iterator i = sorted.begin(); i != sorted.end(); ++i)
XPUSHs(sv_2mortal(newSVpv(i->c_str(), 0)));
} catch (Error & e) {
@@ -160,7 +166,8 @@ SV * topoSortPaths(...)
SV * followLinksToStorePath(char * path)
CODE:
try {
RETVAL = newSVpv(store()->followLinksToStorePath(path).c_str(), 0);
doInit();
RETVAL = newSVpv(followLinksToStorePath(path).c_str(), 0);
} catch (Error & e) {
croak("%s", e.what());
}
@@ -168,23 +175,25 @@ SV * followLinksToStorePath(char * path)
RETVAL
void exportPaths(int fd, ...)
void exportPaths(int fd, int sign, ...)
PPCODE:
try {
doInit();
Paths paths;
for (int n = 1; n < items; ++n) paths.push_back(SvPV_nolen(ST(n)));
for (int n = 2; n < items; ++n) paths.push_back(SvPV_nolen(ST(n)));
FdSink sink(fd);
store()->exportPaths(paths, sink);
exportPaths(*store, paths, sign, sink);
} catch (Error & e) {
croak("%s", e.what());
}
void importPaths(int fd, int dontCheckSigs)
void importPaths(int fd)
PPCODE:
try {
doInit();
FdSource source(fd);
store()->importPaths(source, 0, dontCheckSigs);
store->importPaths(false, source);
} catch (Error & e) {
croak("%s", e.what());
}
@@ -234,12 +243,19 @@ SV * convertHash(char * algo, char * s, int toBase32)
}
SV * signString(char * secretKey_, char * msg)
SV * signString(SV * secretKey_, char * msg)
PPCODE:
try {
#if HAVE_SODIUM
auto sig = SecretKey(secretKey_).signDetached(msg);
XPUSHs(sv_2mortal(newSVpv(sig.c_str(), sig.size())));
STRLEN secretKeyLen;
unsigned char * secretKey = (unsigned char *) SvPV(secretKey_, secretKeyLen);
if (secretKeyLen != crypto_sign_SECRETKEYBYTES)
throw Error("secret key is not valid");
unsigned char sig[crypto_sign_BYTES];
unsigned long long sigLen;
crypto_sign_detached(sig, &sigLen, (unsigned char *) msg, strlen(msg), secretKey);
XPUSHs(sv_2mortal(newSVpv((char *) sig, sigLen)));
#else
throw Error("Nix was not compiled with libsodium, required for signed binary cache support");
#endif
@@ -276,7 +292,8 @@ int checkSignature(SV * publicKey_, SV * sig_, char * msg)
SV * addToStore(char * srcPath, int recursive, char * algo)
PPCODE:
try {
Path path = store()->addToStore(baseNameOf(srcPath), srcPath, recursive, parseHashType(algo));
doInit();
Path path = store->addToStore(baseNameOf(srcPath), srcPath, recursive, parseHashType(algo));
XPUSHs(sv_2mortal(newSVpv(path.c_str(), 0)));
} catch (Error & e) {
croak("%s", e.what());
@@ -286,9 +303,10 @@ SV * addToStore(char * srcPath, int recursive, char * algo)
SV * makeFixedOutputPath(int recursive, char * algo, char * hash, char * name)
PPCODE:
try {
doInit();
HashType ht = parseHashType(algo);
Hash h = parseHash16or32(ht, hash);
Path path = store()->makeFixedOutputPath(recursive, h, name);
Path path = makeFixedOutputPath(recursive, ht,
parseHash16or32(ht, hash), name);
XPUSHs(sv_2mortal(newSVpv(path.c_str(), 0)));
} catch (Error & e) {
croak("%s", e.what());
@@ -300,7 +318,8 @@ SV * derivationFromPath(char * drvPath)
HV *hash;
CODE:
try {
Derivation drv = store()->derivationFromPath(drvPath);
doInit();
Derivation drv = derivationFromPath(*store, drvPath);
hash = newHV();
HV * outputs = newHV();
@@ -342,7 +361,8 @@ SV * derivationFromPath(char * drvPath)
void addTempRoot(char * storePath)
PPCODE:
try {
store()->addTempRoot(storePath);
doInit();
store->addTempRoot(storePath);
} catch (Error & e) {
croak("%s", e.what());
}
+34 -28
View File
@@ -1,43 +1,49 @@
nix_perl_sources := \
lib/Nix/Store.pm \
lib/Nix/Manifest.pm \
lib/Nix/SSH.pm \
lib/Nix/CopyClosure.pm \
lib/Nix/Config.pm.in \
lib/Nix/Utils.pm
$(d)/lib/Nix/Store.pm \
$(d)/lib/Nix/Manifest.pm \
$(d)/lib/Nix/GeneratePatches.pm \
$(d)/lib/Nix/SSH.pm \
$(d)/lib/Nix/CopyClosure.pm \
$(d)/lib/Nix/Config.pm.in \
$(d)/lib/Nix/Utils.pm
nix_perl_modules := $(nix_perl_sources:.in=)
$(foreach x, $(nix_perl_modules), $(eval $(call install-data-in, $(x), $(perllibdir)/Nix)))
lib/Nix/Store.cc: lib/Nix/Store.xs
ifeq ($(perlbindings), yes)
$(d)/lib/Nix/Store.cc: $(d)/lib/Nix/Store.xs
$(trace-gen) xsubpp $^ -output $@
libraries += Store
libraries += Store
Store_DIR := lib/Nix
Store_DIR := $(d)/lib/Nix
Store_SOURCES := $(Store_DIR)/Store.cc
Store_SOURCES := $(Store_DIR)/Store.cc
Store_CXXFLAGS = \
$(NIX_CFLAGS) \
-I$(shell perl -e 'use Config; print $$Config{archlibexp};')/CORE \
-D_FILE_OFFSET_BITS=64 \
-Wno-unknown-warning-option -Wno-unused-variable -Wno-literal-suffix \
-Wno-reserved-user-defined-literal -Wno-duplicate-decl-specifier -Wno-pointer-bool-conversion
Store_CXXFLAGS = \
-I$(shell $(perl) -e 'use Config; print $$Config{archlibexp};')/CORE \
-D_FILE_OFFSET_BITS=64 \
-Wno-unknown-warning-option -Wno-unused-variable -Wno-literal-suffix \
-Wno-reserved-user-defined-literal -Wno-duplicate-decl-specifier -Wno-pointer-bool-conversion
Store_LDFLAGS := $(SODIUM_LIBS) $(NIX_LIBS)
Store_LIBS = libstore libutil
Store_LDFLAGS := $(SODIUM_LIBS)
ifeq (CYGWIN,$(findstring CYGWIN,$(OS)))
archlib = $(shell perl -E 'use Config; print $$Config{archlib};')
libperl = $(shell perl -E 'use Config; print $$Config{libperl};')
Store_LDFLAGS += $(shell find ${archlib} -name ${libperl})
endif
Store_ALLOW_UNDEFINED = 1
Store_FORCE_INSTALL = 1
Store_INSTALL_DIR = $(perllibdir)/auto/Nix/Store
ifeq (CYGWIN,$(findstring CYGWIN,$(OS)))
archlib = $(shell perl -E 'use Config; print $$Config{archlib};')
libperl = $(shell perl -E 'use Config; print $$Config{libperl};')
Store_LDFLAGS += $(shell find ${archlib} -name ${libperl})
endif
Store_ALLOW_UNDEFINED = 1
Store_FORCE_INSTALL = 1
Store_INSTALL_DIR = $(perllibdir)/auto/Nix/Store
clean-files += lib/Nix/Config.pm lib/Nix/Store.cc Makefile.config
clean-files += $(d)/lib/Nix/Config.pm $(d)/lib/Nix/Store.cc
-22
View File
@@ -1,22 +0,0 @@
{ pkgs }:
rec {
sh = pkgs.busybox.override {
useMusl = true;
enableStatic = true;
enableMinimal = true;
extraConfig = ''
CONFIG_ASH y
CONFIG_ASH_BUILTIN_ECHO y
CONFIG_ASH_BUILTIN_TEST y
CONFIG_ASH_OPTIMIZE_FOR_SIZE y
'';
};
configureFlags =
[ "--disable-init-state"
"--enable-gc"
] ++ pkgs.lib.optionals pkgs.stdenv.isLinux [
"--with-sandbox-shell=${sh}/bin/busybox"
];
}
+92 -85
View File
@@ -5,34 +5,36 @@
let
pkgs = import nixpkgs {};
pkgs = import <nixpkgs> {};
systems = [ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ];
systems = [ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" /* "x86_64-freebsd" "i686-freebsd" */ ];
jobs = rec {
tarball = pkgs.lib.genAttrs systems (system:
with import nixpkgs { inherit system; };
tarball =
with pkgs;
releaseTools.sourceTarball {
name = "nix-tarball";
version = builtins.readFile ./version;
versionSuffix = if officialRelease then "" else "pre${toString nix.revCount}_${nix.shortRev}";
src = nix;
src = if lib.inNixShell then null else nix;
inherit officialRelease;
buildInputs =
[ curl bison flex libxml2 libxslt
bzip2 xz brotli
pkgconfig sqlite libsodium boehmgc
[ curl bison flex perl libxml2 libxslt bzip2 xz
dblatex (dblatex.tex or tetex) nukeReferences pkgconfig sqlite libsodium
docbook5 docbook5_xsl
autoconf-archive
git
] ++ lib.optional stdenv.isLinux libseccomp;
] ++ lib.optional stdenv.isLinux libseccomp
++ lib.optional (!lib.inNixShell) git;
configureFlags = "--enable-gc";
configureFlags = ''
--with-dbi=${perlPackages.DBI}/${perl.libPrefix}
--with-dbd-sqlite=${perlPackages.DBDSQLite}/${perl.libPrefix}
--with-www-curl=${perlPackages.WWWCurl}/${perl.libPrefix}
'';
postUnpack = ''
# Clean up when building from a working tree.
@@ -42,7 +44,6 @@ let
'';
preConfigure = ''
(cd perl ; autoreconf --install --force --verbose)
# TeX needs a writable font cache.
export VARTEXFONTS=$TMPDIR/texfonts
'';
@@ -57,38 +58,45 @@ let
preDist = ''
make install docdir=$out/share/doc/nix makefiles=doc/manual/local.mk
make doc/manual/manual.pdf
cp doc/manual/manual.pdf $out/manual.pdf
# The PDF containes filenames of included graphics (see
# http://www.tug.org/pipermail/pdftex/2007-August/007290.html).
# This causes a retained dependency on dblatex, which Hydra
# doesn't like (the output of the tarball job is distributed
# to Windows and Macs, so there should be no Linux binaries
# in the closure).
nuke-refs $out/manual.pdf
echo "doc manual $out/share/doc/nix/manual" >> $out/nix-support/hydra-build-products
echo "doc-pdf manual $out/manual.pdf" >> $out/nix-support/hydra-build-products
'';
}
);
};
build = pkgs.lib.genAttrs systems (system:
with import nixpkgs { inherit system; };
with import ./release-common.nix { inherit pkgs; };
with import <nixpkgs> { inherit system; };
releaseTools.nixBuild {
name = "nix";
src = tarball."x86_64-linux";
src = tarball;
buildInputs =
[ curl
bzip2 xz brotli
openssl pkgconfig sqlite boehmgc
]
[ curl perl bzip2 xz openssl pkgconfig sqlite boehmgc ]
++ lib.optional stdenv.isLinux libseccomp
++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium
++ lib.optional (stdenv.isLinux || stdenv.isDarwin)
(aws-sdk-cpp.override {
apis = ["s3"];
customMemoryManagement = false;
});
++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium;
configureFlags = configureFlags ++
[ "--sysconfdir=/etc" ];
configureFlags = ''
--disable-init-state
--with-dbi=${perlPackages.DBI}/${perl.libPrefix}
--with-dbd-sqlite=${perlPackages.DBDSQLite}/${perl.libPrefix}
--with-www-curl=${perlPackages.WWWCurl}/${perl.libPrefix}
--enable-gc
--sysconfdir=/etc
'';
enableParallelBuilding = true;
@@ -103,35 +111,10 @@ let
});
perlBindings = pkgs.lib.genAttrs systems (system:
let pkgs = import nixpkgs { inherit system; }; in with pkgs;
releaseTools.nixBuild {
name = "nix-perl";
src = tarball."x86_64-linux";
buildInputs =
[ (builtins.getAttr system jobs.build) curl bzip2 xz pkgconfig pkgs.perl ]
++ lib.optional (stdenv.isLinux || stdenv.isDarwin) libsodium;
configureFlags = ''
--with-dbi=${perlPackages.DBI}/${pkgs.perl.libPrefix}
--with-dbd-sqlite=${perlPackages.DBDSQLite}/${pkgs.perl.libPrefix}
'';
enableParallelBuilding = true;
postUnpack = "sourceRoot=$sourceRoot/perl";
preBuild = "unset NIX_INDENT_MAKE";
});
binaryTarball = pkgs.lib.genAttrs systems (system:
# FIXME: temporarily use a different branch for the Darwin build.
with import nixpkgs { inherit system; };
with import <nixpkgs> { inherit system; };
let
toplevel = builtins.getAttr system jobs.build;
@@ -140,7 +123,7 @@ let
runCommand "nix-binary-tarball-${version}"
{ exportReferencesGraph = [ "closure1" toplevel "closure2" cacert ];
buildInputs = [ perl ];
buildInputs = [ perl /* shellcheck */ ];
meta.description = "Distribution-independent Nix bootstrap binaries for ${system}";
}
''
@@ -149,7 +132,15 @@ let
substitute ${./scripts/install-nix-from-closure.sh} $TMPDIR/install \
--subst-var-by nix ${toplevel} \
--subst-var-by cacert ${cacert}
substitute ${./scripts/install-darwin-multi-user.sh} $TMPDIR/install-darwin-multi-user \
--subst-var-by nix ${toplevel} \
--subst-var-by cacert ${cacert}
#shellcheck -e SC1090 $TMPDIR/install
#shellcheck -e SC1091,SC2002 $TMPDIR/install-darwin-multi-user
chmod +x $TMPDIR/install
chmod +x $TMPDIR/install-darwin-multi-user
dir=nix-${version}-${system}
fn=$out/$dir.tar.bz2
mkdir -p $out/nix-support
@@ -161,25 +152,28 @@ let
--transform "s,$TMPDIR/install,$dir/install," \
--transform "s,$TMPDIR/reginfo,$dir/.reginfo," \
--transform "s,$NIX_STORE,$dir/store,S" \
$TMPDIR/install $TMPDIR/reginfo $storePaths
$TMPDIR/install $TMPDIR/install-darwin-multi-user $TMPDIR/reginfo $storePaths
'');
coverage =
with import nixpkgs { system = "x86_64-linux"; };
with import <nixpkgs> { system = "x86_64-linux"; };
releaseTools.coverageAnalysis {
name = "nix-build";
src = tarball."x86_64-linux";
src = tarball;
buildInputs =
[ curl bzip2 openssl pkgconfig sqlite xz libsodium libseccomp
[ curl perl bzip2 openssl pkgconfig sqlite xz libsodium libseccomp
# These are for "make check" only:
graphviz libxml2 libxslt
];
configureFlags = ''
--disable-init-state
--with-dbi=${perlPackages.DBI}/${perl.libPrefix}
--with-dbd-sqlite=${perlPackages.DBDSQLite}/${perl.libPrefix}
--with-www-curl=${perlPackages.WWWCurl}/${perl.libPrefix}
'';
dontInstall = false;
@@ -195,19 +189,27 @@ let
};
rpm_fedora25i386 = makeRPM_i686 (diskImageFuns: diskImageFuns.fedora25i386) [ "libsodium-devel" ];
rpm_fedora25x86_64 = makeRPM_x86_64 (diskImageFunsFun: diskImageFunsFun.fedora25x86_64) [ "libsodium-devel" ];
rpm_fedora19i386 = makeRPM_i686 (diskImageFuns: diskImageFuns.fedora19i386) [];
rpm_fedora19x86_64 = makeRPM_x86_64 (diskImageFunsFun: diskImageFunsFun.fedora19x86_64) [];
rpm_fedora20i386 = makeRPM_i686 (diskImageFuns: diskImageFuns.fedora20i386) [];
rpm_fedora20x86_64 = makeRPM_x86_64 (diskImageFunsFun: diskImageFunsFun.fedora20x86_64) [];
rpm_fedora21i386 = makeRPM_i686 (diskImageFuns: diskImageFuns.fedora21i386) [ "libsodium-devel" ];
rpm_fedora21x86_64 = makeRPM_x86_64 (diskImageFunsFun: diskImageFunsFun.fedora21x86_64) [ "libsodium-devel" ];
deb_debian8i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.debian8i386) [ "libsodium-dev" ] [ "libsodium13" ];
deb_debian8x86_64 = makeDeb_x86_64 (diskImageFunsFun: diskImageFunsFun.debian8x86_64) [ "libsodium-dev" ] [ "libsodium13" ];
deb_ubuntu1404i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.ubuntu1404i386) [] [];
deb_ubuntu1404x86_64 = makeDeb_x86_64 (diskImageFuns: diskImageFuns.ubuntu1404x86_64) [] [];
deb_ubuntu1410i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.ubuntu1410i386) [] [];
deb_ubuntu1410x86_64 = makeDeb_x86_64 (diskImageFuns: diskImageFuns.ubuntu1410x86_64) [] [];
deb_ubuntu1504i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.ubuntu1504i386) [ "libsodium-dev" ] [ "libsodium13" ];
deb_ubuntu1504x86_64 = makeDeb_x86_64 (diskImageFuns: diskImageFuns.ubuntu1504x86_64) [ "libsodium-dev" ] [ "libsodium13" ];
deb_ubuntu1510i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.ubuntu1510i386) [ "libsodium-dev" ] [ "libsodium13"];
deb_ubuntu1510x86_64 = makeDeb_x86_64 (diskImageFuns: diskImageFuns.ubuntu1510x86_64) [ "libsodium-dev" ] [ "libsodium13" ];
deb_ubuntu1604i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.ubuntu1604i386) [ "libsodium-dev" ] [ "libsodium18" ];
deb_ubuntu1604x86_64 = makeDeb_x86_64 (diskImageFuns: diskImageFuns.ubuntu1604x86_64) [ "libsodium-dev" ] [ "libsodium18" ];
deb_ubuntu1610i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.ubuntu1610i386) [ "libsodium-dev" ] [ "libsodium18" ];
deb_ubuntu1610x86_64 = makeDeb_x86_64 (diskImageFuns: diskImageFuns.ubuntu1610x86_64) [ "libsodium-dev" ] [ "libsodium18" ];
# System tests.
@@ -225,7 +227,7 @@ let
});
tests.binaryTarball =
with import nixpkgs { system = "x86_64-linux"; };
with import <nixpkgs> { system = "x86_64-linux"; };
vmTools.runInLinuxImage (runCommand "nix-binary-tarball-test"
{ diskImage = vmTools.diskImages.ubuntu1204x86_64;
}
@@ -254,6 +256,7 @@ let
tests.evalNixOS =
pkgs.runCommand "eval-nixos" { buildInputs = [ build.x86_64-linux ]; }
''
export NIX_DB_DIR=$TMPDIR
export NIX_STATE_DIR=$TMPDIR
nix-store --init
@@ -265,24 +268,30 @@ let
# Aggregate job containing the release-critical jobs.
release = pkgs.releaseTools.aggregate {
name = "nix-${tarball."x86_64-linux".version}";
name = "nix-${tarball.version}";
meta.description = "Release-critical builds";
constituents =
[ tarball.x86_64-linux
tarball.x86_64-darwin
tarball.i686-linux
[ tarball
#build.i686-freebsd
build.i686-linux
build.x86_64-darwin
#build.x86_64-freebsd
build.x86_64-linux
#binaryTarball.i686-freebsd
binaryTarball.i686-linux
binaryTarball.x86_64-darwin
#binaryTarball.x86_64-freebsd
binaryTarball.x86_64-linux
deb_debian8i386
deb_debian8x86_64
deb_ubuntu1604i386
deb_ubuntu1604x86_64
rpm_fedora25i386
rpm_fedora25x86_64
deb_ubuntu1404i386 # LTS
deb_ubuntu1404x86_64 # LTS
deb_ubuntu1504i386
deb_ubuntu1504x86_64
rpm_fedora20i386
rpm_fedora20x86_64
rpm_fedora21i386
rpm_fedora21x86_64
tests.remoteBuilds
tests.nix-copy-closure
tests.binaryTarball
@@ -300,19 +309,18 @@ let
makeRPM =
system: diskImageFun: extraPackages:
with import nixpkgs { inherit system; };
with import <nixpkgs> { inherit system; };
releaseTools.rpmBuild rec {
name = "nix-rpm";
src = jobs.tarball."x86_64-linux";
src = jobs.tarball;
diskImage = (diskImageFun vmTools.diskImageFuns)
{ extraPackages =
[ "sqlite" "sqlite-devel" "bzip2-devel" "emacs" "libcurl-devel" "openssl-devel" "xz-devel" "libseccomp-devel" ]
[ "perl-DBD-SQLite" "perl-devel" "sqlite" "sqlite-devel" "bzip2-devel" "emacs" "perl-WWW-Curl" "libcurl-devel" "openssl-devel" "xz-devel" "libseccomp-devel" ]
++ extraPackages; };
memSize = 1024;
meta.schedulingPriority = 50;
postRPMInstall = "cd /tmp/rpmout/BUILD/nix-* && make installcheck";
#enableParallelBuilding = true;
};
@@ -322,25 +330,24 @@ let
makeDeb =
system: diskImageFun: extraPackages: extraDebPackages:
with import nixpkgs { inherit system; };
with import <nixpkgs> { inherit system; };
releaseTools.debBuild {
name = "nix-deb";
src = jobs.tarball."x86_64-linux";
src = jobs.tarball;
diskImage = (diskImageFun vmTools.diskImageFuns)
{ extraPackages =
[ "libsqlite3-dev" "libbz2-dev" "libcurl-dev" "libcurl3-nss" "libssl-dev" "liblzma-dev" "libseccomp-dev" ]
[ "libdbd-sqlite3-perl" "libsqlite3-dev" "libbz2-dev" "libwww-curl-perl" "libcurl-dev" "libcurl3-nss" "libssl-dev" "liblzma-dev" "libseccomp-dev" ]
++ extraPackages; };
memSize = 1024;
meta.schedulingPriority = 50;
postInstall = "make installcheck";
configureFlags = "--sysconfdir=/etc";
debRequires =
[ "curl" "libsqlite3-0" "libbz2-1.0" "bzip2" "xz-utils" "libssl1.0.0" "liblzma5" "libseccomp2" ]
[ "curl" "libdbd-sqlite3-perl" "libsqlite3-0" "libbz2-1.0" "bzip2" "xz-utils" "libwww-curl-perl" "libssl1.0.0" "liblzma5" "libseccomp2" ]
++ extraDebPackages;
debMaintainer = "Eelco Dolstra <eelco.dolstra@logicblox.com>";
doInstallCheck = true;
#enableParallelBuilding = true;
};
+284
View File
@@ -0,0 +1,284 @@
#! @perl@ -w @perlFlags@
use utf8;
use Fcntl qw(:DEFAULT :flock);
use English '-no_match_vars';
use IO::Handle;
use Nix::Config;
use Nix::SSH;
use Nix::CopyClosure;
use Nix::Store;
use Encode;
no warnings('once');
STDERR->autoflush(1);
binmode STDERR, ":encoding(utf8)";
my $debug = defined $ENV{NIX_DEBUG_HOOK};
# General operation:
#
# Try to find a free machine of type $neededSystem. We do this as
# follows:
# - We acquire an exclusive lock on $currentLoad/main-lock.
# - For each machine $machine of type $neededSystem and for each $slot
# less than the maximum load for that machine, we try to get an
# exclusive lock on $currentLoad/$machine-$slot (without blocking).
# If we get such a lock, we send "accept" to the caller. Otherwise,
# we send "postpone" and exit.
# - We release the exclusive lock on $currentLoad/main-lock.
# - We perform the build on $neededSystem.
# - We release the exclusive lock on $currentLoad/$machine-$slot.
#
# The nice thing about this scheme is that if we die prematurely, the
# locks are released automatically.
# Make sure that we don't get any SSH passphrase or host key popups -
# if there is any problem it should fail, not do something
# interactive.
$ENV{"DISPLAY"} = "";
$ENV{"SSH_ASKPASS"} = "";
sub sendReply {
my $reply = shift;
print STDERR "# $reply\n";
}
sub all { $_ || return 0 for @_; 1 }
# Initialisation.
my $loadIncreased = 0;
my ($localSystem, $maxSilentTime, $printBuildTrace, $buildTimeout) = @ARGV;
my $currentLoad = $ENV{"NIX_CURRENT_LOAD"} // "/run/nix/current-load";
my $conf = $ENV{"NIX_REMOTE_SYSTEMS"} // "@sysconfdir@/nix/machines";
sub openSlotLock {
my ($machine, $slot) = @_;
my $slotLockFn = "$currentLoad/" . (join '+', @{$machine->{systemTypes}}) . "-" . $machine->{hostName} . "-$slot";
my $slotLock = new IO::Handle;
sysopen $slotLock, "$slotLockFn", O_RDWR|O_CREAT, 0600 or die;
return $slotLock;
}
# Read the list of machines.
my @machines;
if (defined $conf && -e $conf) {
open CONF, "<$conf" or die;
while (<CONF>) {
chomp;
s/\#.*$//g;
next if /^\s*$/;
my @tokens = split /\s/, $_;
my @supportedFeatures = split(/,/, $tokens[5] || "");
my @mandatoryFeatures = split(/,/, $tokens[6] || "");
push @machines,
{ hostName => $tokens[0]
, systemTypes => [ split(/,/, $tokens[1]) ]
, sshKey => $tokens[2]
, maxJobs => int($tokens[3])
, speedFactor => 1.0 * (defined $tokens[4] ? int($tokens[4]) : 1)
, supportedFeatures => [ @supportedFeatures, @mandatoryFeatures ]
, mandatoryFeatures => [ @mandatoryFeatures ]
, enabled => 1
};
}
close CONF;
}
# Wait for the calling process to ask us whether we can build some derivation.
my ($drvPath, $hostName, $slotLock);
my ($from, $to);
REQ: while (1) {
$_ = <STDIN> || exit 0;
(my $amWilling, my $neededSystem, $drvPath, my $requiredFeatures) = split;
my @requiredFeatures = split /,/, $requiredFeatures;
my $canBuildLocally = $amWilling && ($localSystem eq $neededSystem);
if (!defined $currentLoad) {
sendReply "decline";
next;
}
# Acquire the exclusive lock on $currentLoad/main-lock.
mkdir $currentLoad, 0777 or die unless -d $currentLoad;
my $mainLock = "$currentLoad/main-lock";
sysopen MAINLOCK, "$mainLock", O_RDWR|O_CREAT, 0600 or die;
flock(MAINLOCK, LOCK_EX) or die;
while (1) {
# Find all machine that can execute this build, i.e., that
# support builds for the given platform and features, and are
# not at their job limit.
my $rightType = 0;
my @available = ();
LOOP: foreach my $cur (@machines) {
if ($cur->{enabled}
&& (grep { $neededSystem eq $_ } @{$cur->{systemTypes}})
&& all(map { my $f = $_; 0 != grep { $f eq $_ } @{$cur->{supportedFeatures}} } (@requiredFeatures, @mandatoryFeatures))
&& all(map { my $f = $_; 0 != grep { $f eq $_ } @requiredFeatures } @{$cur->{mandatoryFeatures}})
)
{
$rightType = 1;
# We have a machine of the right type. Determine the load on
# the machine.
my $slot = 0;
my $load = 0;
my $free;
while ($slot < $cur->{maxJobs}) {
my $slotLock = openSlotLock($cur, $slot);
if (flock($slotLock, LOCK_EX | LOCK_NB)) {
$free = $slot unless defined $free;
flock($slotLock, LOCK_UN) or die;
} else {
$load++;
}
close $slotLock;
$slot++;
}
push @available, { machine => $cur, load => $load, free => $free }
if $load < $cur->{maxJobs};
}
}
if ($debug) {
print STDERR "load on " . $_->{machine}->{hostName} . " = " . $_->{load} . "\n"
foreach @available;
}
# Didn't find any available machine? Then decline or postpone.
if (scalar @available == 0) {
# Postpone if we have a machine of the right type, except
# if the local system can and wants to do the build.
if ($rightType && !$canBuildLocally) {
sendReply "postpone";
} else {
sendReply "decline";
}
close MAINLOCK;
next REQ;
}
# Prioritise the available machines as follows:
# - First by load divided by speed factor, rounded to the nearest
# integer. This causes fast machines to be preferred over slow
# machines with similar loads.
# - Then by speed factor.
# - Finally by load.
sub lf { my $x = shift; return int($x->{load} / $x->{machine}->{speedFactor} + 0.4999); }
@available = sort
{ lf($a) <=> lf($b)
|| $b->{machine}->{speedFactor} <=> $a->{machine}->{speedFactor}
|| $a->{load} <=> $b->{load}
} @available;
# Select the best available machine and lock a free slot.
my $selected = $available[0];
my $machine = $selected->{machine};
$slotLock = openSlotLock($machine, $selected->{free});
flock($slotLock, LOCK_EX | LOCK_NB) or die;
utime undef, undef, $slotLock;
close MAINLOCK;
# Connect to the selected machine.
my @sshOpts = ("-i", $machine->{sshKey});
$hostName = $machine->{hostName};
eval {
($from, $to) = connectToRemoteNix($hostName, \@sshOpts, "2>&4");
# FIXME: check if builds are inhibited.
};
last REQ unless $@;
print STDERR "$@";
warn "unable to open SSH connection to ‘$hostName’, trying other available machines...\n";
$from = undef;
$to = undef;
$machine->{enabled} = 0;
}
}
# Tell Nix we've accepted the build.
sendReply "accept";
my @inputs = split /\s/, readline(STDIN);
my @outputs = split /\s/, readline(STDIN);
print STDERR "@ build-remote $drvPath $hostName\n" if $printBuildTrace;
my $maybeSign = "";
$maybeSign = "--sign" if -e "$Nix::Config::confDir/signing-key.sec";
# Copy the derivation and its dependencies to the build machine. This
# is guarded by an exclusive lock per machine to prevent multiple
# build-remote instances from copying to a machine simultaneously.
# That's undesirable because we may end up with N instances uploading
# the same missing path simultaneously, causing the effective network
# bandwidth and target disk speed to be divided by N.
my $uploadLock = "$currentLoad/$hostName.upload-lock";
sysopen UPLOADLOCK, "$uploadLock", O_RDWR|O_CREAT, 0600 or die;
eval {
local $SIG{ALRM} = sub { die "alarm\n" };
# Don't wait forever, so that a process that gets stuck while
# holding the lock doesn't block everybody else indefinitely.
# It's safe to continue after a timeout, just (potentially)
# inefficient.
alarm 15 * 60;
flock(UPLOADLOCK, LOCK_EX);
alarm 0;
};
if ($@) {
die unless $@ eq "alarm\n";
print STDERR "somebody is hogging $uploadLock, continuing...\n";
unlink $uploadLock;
}
Nix::CopyClosure::copyToOpen($from, $to, $hostName, [ $drvPath, @inputs ], 0, 0, $maybeSign ne "");
close UPLOADLOCK;
# Perform the build.
print STDERR "building ‘$drvPath’ on ‘$hostName’\n";
print STDERR "@ build-remote-start $drvPath $hostName\n" if $printBuildTrace;
writeInt(6, $to) or die; # == cmdBuildPaths
writeStrings([$drvPath], $to);
writeInt($maxSilentTime, $to);
writeInt($buildTimeout, $to);
my $res = readInt($from);
print STDERR "@ build-remote-done $drvPath $hostName\n" if $printBuildTrace;
if ($res != 0) {
my $msg = decode("utf-8", readString($from));
print STDERR "error: $msg on ‘$hostName’\n";
exit $res;
}
# Copy the output from the build machine.
my @outputs2 = grep { !isValidPath($_) } @outputs;
if (scalar @outputs2 > 0) {
writeInt(5, $to); # == cmdExportPaths
writeInt(0, $to); # don't sign
writeStrings(\@outputs2, $to);
$ENV{'NIX_HELD_LOCKS'} = "@outputs2"; # FIXME: ugly
importPaths(fileno($from));
}
+103
View File
@@ -0,0 +1,103 @@
#! @perl@ -w @perlFlags@
use utf8;
use strict;
use File::Basename;
use IO::Handle;
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
STDOUT->autoflush(1);
binmode STDERR, ":encoding(utf8)";
my @remoteStoresAll = split ':', ($ENV{"NIX_OTHER_STORES"} or "");
my @remoteStores;
foreach my $dir (@remoteStoresAll) {
push @remoteStores, glob($dir);
}
exit if scalar @remoteStores == 0;
print "\n";
$ENV{"NIX_REMOTE"} = "";
sub findStorePath {
my $storePath = shift;
foreach my $store (@remoteStores) {
my $sourcePath = "$store/store/" . basename $storePath;
next unless -e $sourcePath || -l $sourcePath;
$ENV{"NIX_DB_DIR"} = "$store/var/nix/db";
return ($store, $sourcePath) if
system("$binDir/nix-store --check-validity $storePath") == 0;
}
return undef;
}
if ($ARGV[0] eq "--query") {
while (<STDIN>) {
chomp;
my ($cmd, @args) = split " ", $_;
if ($cmd eq "have") {
foreach my $storePath (@args) {
print "$storePath\n" if defined findStorePath($storePath);
}
print "\n";
}
elsif ($cmd eq "info") {
foreach my $storePath (@args) {
my ($store, $sourcePath) = findStorePath($storePath);
next unless defined $store;
$ENV{"NIX_DB_DIR"} = "$store/var/nix/db";
my $deriver = `$binDir/nix-store --query --deriver $storePath`;
die "cannot query deriver of ‘$storePath’" if $? != 0;
chomp $deriver;
$deriver = "" if $deriver eq "unknown-deriver";
my @references = split "\n",
`$binDir/nix-store --query --references $storePath`;
die "cannot query references of ‘$storePath’" if $? != 0;
my $narSize = `$binDir/nix-store --query --size $storePath`;
die "cannot query size of ‘$storePath’" if $? != 0;
chomp $narSize;
print "$storePath\n";
print "$deriver\n";
print scalar @references, "\n";
print "$_\n" foreach @references;
print "0\n";
print "$narSize\n";
}
print "\n";
}
else { die "unknown command ‘$cmd’"; }
}
}
elsif ($ARGV[0] eq "--substitute") {
die unless scalar @ARGV == 3;
my $storePath = $ARGV[1];
my $destPath = $ARGV[2];
my ($store, $sourcePath) = findStorePath $storePath;
die unless $store;
print STDERR "\n*** Copying ‘$storePath’ from ‘$sourcePath’\n\n";
system("@coreutils@/cp", "-rpd", $sourcePath, $destPath) == 0
or die "cannot copy ‘$sourcePath’ to ‘$storePath’";
print "\n"; # no hash to verify
}
else { die; }
+633
View File
@@ -0,0 +1,633 @@
#! @perl@ -w @perlFlags@
use utf8;
use DBI;
use DBD::SQLite;
use File::Basename;
use IO::Select;
use Nix::Config;
use Nix::Store;
use Nix::Utils;
use Nix::Manifest;
use WWW::Curl::Easy;
use WWW::Curl::Multi;
use strict;
STDERR->autoflush(1);
binmode STDERR, ":encoding(utf8)";
Nix::Config::readConfig;
my @caches;
my $gotCaches = 0;
my $maxParallelRequests = int($Nix::Config::config{"binary-caches-parallel-connections"} // 25);
$maxParallelRequests = 1 if $maxParallelRequests < 1;
my $ttlNegative = 24 * 3600; # when to purge negative lookups from the database
my $ttlNegativeUse = 3600; # how long negative lookups are valid for non-"have" lookups
my $didExpiration = 0;
my $showAfter = 5; # show that we're waiting for a request after this many seconds
my $debug = ($Nix::Config::config{"debug-subst"} // "") eq 1 || ($Nix::Config::config{"untrusted-debug-subst"} // "") eq 1;
my $cacheFileURLs = ($ENV{"_NIX_CACHE_FILE_URLS"} // "") eq 1; # for testing
my ($dbh, $queryCache, $insertNAR, $queryNAR, $insertNARExistence, $queryNARExistence, $expireNARExistence);
my $curlm = WWW::Curl::Multi->new;
my $activeRequests = 0;
my $curlIdCount = 1;
my %requests;
my %scheduled;
my $userName = getpwuid($<) || $ENV{"USER"} or die "cannot figure out user name";
my $userAgent = "Nix/$Nix::Config::version";
sub isTrue {
my ($x) = @_;
return $x eq "true" || $x eq "1";
}
# FIXME: this should be cache URLs required to have valid signatures,
# or "*" to require signatures on all binary caches.
# FIXME: should binary caches using a key in
# ‘binary-cache-public-keys’ be trusted by default?
my $requireSignedBinaryCaches = ($Nix::Config::config{"signed-binary-caches"} // "0") ne "0";
my $curlConnectTimeout = int(
$Nix::Config::config{"untrusted-connect-timeout"} //
$Nix::Config::config{"connect-timeout"} //
$ENV{"NIX_CONNECT_TIMEOUT"} // 0);
my $netrcFile = $Nix::Config::config{"netrc-file"} //
"$Nix::Config::confDir/netrc";
sub addRequest {
my ($storePath, $url, $head) = @_;
my $curl = WWW::Curl::Easy->new;
my $curlId = $curlIdCount++;
$requests{$curlId} = { storePath => $storePath, url => $url, handle => $curl, content => "", type => $head ? "HEAD" : "GET"
, shown => 0, started => time() };
$curl->setopt(CURLOPT_PRIVATE, $curlId);
$curl->setopt(CURLOPT_URL, $url);
open (my $fh, ">", \$requests{$curlId}->{content});
$curl->setopt(CURLOPT_WRITEDATA, $fh);
$curl->setopt(CURLOPT_FOLLOWLOCATION, 1);
$curl->setopt(CURLOPT_CAINFO, $Nix::Config::caBundle) if defined $Nix::Config::caBundle;
unless (isTrue($Nix::Config::config{"verify-https-binary-caches"} // "1")) {
$curl->setopt(CURLOPT_SSL_VERIFYPEER, 0);
$curl->setopt(CURLOPT_SSL_VERIFYHOST, 0);
}
$curl->setopt(CURLOPT_USERAGENT, $userAgent);
$curl->setopt(CURLOPT_NOBODY, 1) if $head;
$curl->setopt(CURLOPT_FAILONERROR, 1);
$curl->setopt(CURLOPT_CONNECTTIMEOUT, $curlConnectTimeout);
$curl->setopt(CURLOPT_TIMEOUT, 20 * 60);
$curl->setopt(CURLOPT_NETRC_FILE, $netrcFile);
$curl->setopt(CURLOPT_NETRC, CURL_NETRC_OPTIONAL);
if ($activeRequests >= $maxParallelRequests) {
$scheduled{$curlId} = 1;
} else {
$curlm->add_handle($curl);
$activeRequests++;
}
return $requests{$curlId};
}
sub processRequests {
while ($activeRequests) {
my ($rfds, $wfds, $efds) = $curlm->fdset();
#print STDERR "R = @{$rfds}, W = @{$wfds}, E = @{$efds}\n";
# Sleep until we can read or write some data.
if (scalar @{$rfds} + scalar @{$wfds} + scalar @{$efds} > 0) {
IO::Select->select(IO::Select->new(@{$rfds}), IO::Select->new(@{$wfds}), IO::Select->new(@{$efds}), 1.0);
}
if ($curlm->perform() != $activeRequests) {
while (my ($id, $result) = $curlm->info_read) {
if ($id) {
my $request = $requests{$id} or die;
my $handle = $request->{handle};
$request->{result} = $result;
$request->{httpStatus} = $handle->getinfo(CURLINFO_RESPONSE_CODE);
print STDERR "$request->{type} on $request->{url} [$request->{result}, $request->{httpStatus}]\n" if $debug;
$activeRequests--;
delete $request->{handle};
if (scalar(keys %scheduled) > 0) {
my $id2 = (keys %scheduled)[0];
$curlm->add_handle($requests{$id2}->{handle});
$activeRequests++;
delete $scheduled{$id2};
}
}
}
}
my $time = time();
while (my ($key, $request) = each %requests) {
next unless defined $request->{handle};
next if $request->{shown};
if ($time > $request->{started} + $showAfter) {
print STDERR "still waiting for ‘$request->{url}’ after $showAfter seconds...\n";
$request->{shown} = 1;
}
}
}
}
sub initCache {
my $dbPath = "$Nix::Config::stateDir/binary-cache-v3.sqlite";
unlink "$Nix::Config::stateDir/binary-cache-v1.sqlite";
unlink "$Nix::Config::stateDir/binary-cache-v2.sqlite";
# Open/create the database.
$dbh = DBI->connect("dbi:SQLite:dbname=$dbPath", "", "")
or die "cannot open database ‘$dbPath’";
$dbh->{RaiseError} = 1;
$dbh->{PrintError} = 0;
$dbh->sqlite_busy_timeout(60 * 60 * 1000);
$dbh->do("pragma synchronous = off"); # we can always reproduce the cache
$dbh->do("pragma journal_mode = truncate");
# Initialise the database schema, if necessary.
$dbh->do(<<EOF);
create table if not exists BinaryCaches (
id integer primary key autoincrement not null,
url text unique not null,
timestamp integer not null,
storeDir text not null,
wantMassQuery integer not null,
priority integer not null
);
EOF
$dbh->do(<<EOF);
create table if not exists NARs (
cache integer not null,
storePath text not null,
url text not null,
compression text not null,
fileHash text,
fileSize integer,
narHash text,
narSize integer,
refs text,
deriver text,
signedBy text,
timestamp integer not null,
primary key (cache, storePath),
foreign key (cache) references BinaryCaches(id) on delete cascade
);
EOF
$dbh->do(<<EOF);
create table if not exists NARExistence (
cache integer not null,
storePath text not null,
exist integer not null,
timestamp integer not null,
primary key (cache, storePath),
foreign key (cache) references BinaryCaches(id) on delete cascade
);
EOF
$dbh->do("create index if not exists NARExistenceByExistTimestamp on NARExistence (exist, timestamp)");
$queryCache = $dbh->prepare("select id, storeDir, wantMassQuery, priority from BinaryCaches where url = ?") or die;
$insertNAR = $dbh->prepare(
"insert or replace into NARs(cache, storePath, url, compression, fileHash, fileSize, narHash, " .
"narSize, refs, deriver, signedBy, timestamp) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)") or die;
$queryNAR = $dbh->prepare("select * from NARs where cache = ? and storePath = ?") or die;
$insertNARExistence = $dbh->prepare(
"insert or replace into NARExistence(cache, storePath, exist, timestamp) values (?, ?, ?, ?)") or die;
$queryNARExistence = $dbh->prepare("select exist, timestamp from NARExistence where cache = ? and storePath = ?") or die;
$expireNARExistence = $dbh->prepare("delete from NARExistence where exist = ? and timestamp < ?") or die;
}
sub getAvailableCaches {
return if $gotCaches;
$gotCaches = 1;
sub strToList {
my ($s) = @_;
return map { s/\/+$//; $_ } split(/ /, $s);
}
my @urls = strToList($Nix::Config::config{"binary-caches"} //
($Nix::Config::storeDir eq "/nix/store" ? "https://cache.nixos.org" : ""));
my $urlsFiles = $Nix::Config::config{"binary-cache-files"}
// "$Nix::Config::stateDir/profiles/per-user/$userName/channels/binary-caches/*";
foreach my $urlFile (glob $urlsFiles) {
next unless -f $urlFile;
open FILE, "<$urlFile" or die "cannot open ‘$urlFile’\n";
my $url = <FILE>; chomp $url;
close FILE;
push @urls, strToList($url);
}
push @urls, strToList($Nix::Config::config{"extra-binary-caches"} // "");
# Allow Nix daemon users to override the binary caches to a subset
# of those listed in the config file. Note that ‘untrusted-*’
# denotes options passed by the client.
my @trustedUrls = uniq(@urls, strToList($Nix::Config::config{"trusted-binary-caches"} // ""));
if (defined $Nix::Config::config{"untrusted-binary-caches"}) {
my @untrustedUrls = strToList $Nix::Config::config{"untrusted-binary-caches"};
@urls = ();
foreach my $url (@untrustedUrls) {
die "binary cache ‘$url’ is not trusted (please add it to ‘trusted-binary-caches’ in $Nix::Config::confDir/nix.conf)\n"
unless scalar(grep { $url eq $_ } @trustedUrls) > 0;
push @urls, $url;
}
}
my @untrustedUrls = strToList $Nix::Config::config{"untrusted-extra-binary-caches"} // "";
foreach my $url (@untrustedUrls) {
unless (scalar(grep { $url eq $_ } @trustedUrls) > 0) {
warn "binary cache ‘$url’ is not trusted (please add it to ‘trusted-binary-caches’ in $Nix::Config::confDir/nix.conf)\n";
next;
}
push @urls, $url;
}
foreach my $url (uniq @urls) {
# FIXME: not atomic.
$queryCache->execute($url);
my $res = $queryCache->fetchrow_hashref();
if (defined $res) {
next if $res->{storeDir} ne $Nix::Config::storeDir;
push @caches, { id => $res->{id}, url => $url, wantMassQuery => $res->{wantMassQuery}, priority => $res->{priority} };
next;
}
# Get the cache info file.
my $request = addRequest(undef, $url . "/nix-cache-info");
processRequests;
if ($request->{result} != 0) {
print STDERR "could not download ‘$request->{url}’ (" .
($request->{result} != 0 ? "Curl error $request->{result}" : "HTTP status $request->{httpStatus}") . ")\n";
next;
}
my $storeDir = "/nix/store";
my $wantMassQuery = 0;
my $priority = 50;
foreach my $line (split "\n", $request->{content}) {
unless ($line =~ /^(.*): (.*)$/) {
print STDERR "bad cache info file ‘$request->{url}’\n";
return undef;
}
if ($1 eq "StoreDir") { $storeDir = $2; }
elsif ($1 eq "WantMassQuery") { $wantMassQuery = int($2); }
elsif ($1 eq "Priority") { $priority = int($2); }
}
$dbh->do("insert or replace into BinaryCaches(url, timestamp, storeDir, wantMassQuery, priority) values (?, ?, ?, ?, ?)",
{}, $url, time(), $storeDir, $wantMassQuery, $priority);
$queryCache->execute($url);
$res = $queryCache->fetchrow_hashref() or die;
next if $storeDir ne $Nix::Config::storeDir;
push @caches, { id => $res->{id}, url => $url, wantMassQuery => $wantMassQuery, priority => $priority };
}
@caches = sort { $a->{priority} <=> $b->{priority} } @caches;
expireNegative();
}
sub shouldCache {
my ($url) = @_;
return $cacheFileURLs || $url !~ /^file:/;
}
sub processNARInfo {
my ($storePath, $cache, $request) = @_;
if ($request->{result} != 0) {
if ($request->{result} != 37 && $request->{httpStatus} != 404 && $request->{httpStatus} != 403) {
print STDERR "could not download ‘$request->{url}’ (" .
($request->{result} != 0 ? "Curl error $request->{result}" : "HTTP status $request->{httpStatus}") . ")\n";
} else {
$insertNARExistence->execute($cache->{id}, basename($storePath), 0, time())
if shouldCache $request->{url};
}
return undef;
}
my $narInfo = parseNARInfo($storePath, $request->{content}, $requireSignedBinaryCaches, $request->{url});
return undef unless defined $narInfo;
die if $requireSignedBinaryCaches && !defined $narInfo->{signedBy};
# Cache the result.
$insertNAR->execute(
$cache->{id}, basename($storePath), $narInfo->{url}, $narInfo->{compression},
$narInfo->{fileHash}, $narInfo->{fileSize}, $narInfo->{narHash}, $narInfo->{narSize},
join(" ", @{$narInfo->{refs}}), $narInfo->{deriver}, $narInfo->{signedBy}, time())
if shouldCache $request->{url};
return $narInfo;
}
sub getCachedInfoFrom {
my ($storePath, $cache) = @_;
$queryNAR->execute($cache->{id}, basename($storePath));
my $res = $queryNAR->fetchrow_hashref();
return undef unless defined $res;
# We may previously have cached this info when signature checking
# was disabled. In that case, ignore the cached info.
return undef if $requireSignedBinaryCaches && !defined $res->{signedBy};
return
{ url => $res->{url}
, compression => $res->{compression}
, fileHash => $res->{fileHash}
, fileSize => $res->{fileSize}
, narHash => $res->{narHash}
, narSize => $res->{narSize}
, refs => [ split " ", $res->{refs} ]
, deriver => $res->{deriver}
, signedBy => $res->{signedBy}
} if defined $res;
}
sub negativeHit {
my ($storePath, $cache) = @_;
$queryNARExistence->execute($cache->{id}, basename($storePath));
my $res = $queryNARExistence->fetchrow_hashref();
return defined $res && $res->{exist} == 0 && time() - $res->{timestamp} < $ttlNegativeUse;
}
sub positiveHit {
my ($storePath, $cache) = @_;
return 1 if defined getCachedInfoFrom($storePath, $cache);
$queryNARExistence->execute($cache->{id}, basename($storePath));
my $res = $queryNARExistence->fetchrow_hashref();
return defined $res && $res->{exist} == 1;
}
sub expireNegative {
return if $didExpiration;
$didExpiration = 1;
my $time = time();
# Round up to the next multiple of the TTL to ensure that we do
# expiration only once per time interval. E.g. if $ttlNegative ==
# 3600, we expire entries at most once per hour. This is
# presumably faster than expiring a few entries per request (and
# thus doing a transaction).
my $limit = (int($time / $ttlNegative) - 1) * $ttlNegative;
$expireNARExistence->execute($limit, 0);
print STDERR "expired ", $expireNARExistence->rows, " negative entries\n" if $debug;
}
sub printInfo {
my ($storePath, $info) = @_;
print "$storePath\n";
print $info->{deriver} ? "$Nix::Config::storeDir/$info->{deriver}" : "", "\n";
print scalar @{$info->{refs}}, "\n";
print "$Nix::Config::storeDir/$_\n" foreach @{$info->{refs}};
print $info->{fileSize} || 0, "\n";
print $info->{narSize} || 0, "\n";
}
sub infoUrl {
my ($binaryCacheUrl, $storePath) = @_;
my $pathHash = substr(basename($storePath), 0, 32);
my $infoUrl = "$binaryCacheUrl/$pathHash.narinfo";
}
sub printInfoParallel {
my @paths = @_;
# First print all paths for which we have cached info.
my @left;
foreach my $storePath (@paths) {
my $found = 0;
foreach my $cache (@caches) {
my $info = getCachedInfoFrom($storePath, $cache);
if (defined $info) {
printInfo($storePath, $info);
$found = 1;
last;
}
}
push @left, $storePath if !$found;
}
return if scalar @left == 0;
foreach my $cache (@caches) {
my @left2;
%requests = ();
foreach my $storePath (@left) {
if (negativeHit($storePath, $cache)) {
push @left2, $storePath;
next;
}
addRequest($storePath, infoUrl($cache->{url}, $storePath));
}
processRequests;
foreach my $request (values %requests) {
my $info = processNARInfo($request->{storePath}, $cache, $request);
if (defined $info) {
printInfo($request->{storePath}, $info);
} else {
push @left2, $request->{storePath};
}
}
@left = @left2;
}
}
sub printSubstitutablePaths {
my @paths = @_;
# First look for paths that have cached info.
my @left;
foreach my $storePath (@paths) {
my $found = 0;
foreach my $cache (@caches) {
next unless $cache->{wantMassQuery};
if (positiveHit($storePath, $cache)) {
print "$storePath\n";
$found = 1;
last;
}
}
push @left, $storePath if !$found;
}
return if scalar @left == 0;
# For remaining paths, do HEAD requests.
foreach my $cache (@caches) {
next unless $cache->{wantMassQuery};
my @left2;
%requests = ();
foreach my $storePath (@left) {
if (negativeHit($storePath, $cache)) {
push @left2, $storePath;
next;
}
addRequest($storePath, infoUrl($cache->{url}, $storePath), 1);
}
processRequests;
foreach my $request (values %requests) {
if ($request->{result} != 0) {
if ($request->{result} != 37 && $request->{httpStatus} != 404 && $request->{httpStatus} != 403) {
print STDERR "could not check ‘$request->{url}’ (" .
($request->{result} != 0 ? "Curl error $request->{result}" : "HTTP status $request->{httpStatus}") . ")\n";
} else {
$insertNARExistence->execute($cache->{id}, basename($request->{storePath}), 0, time())
if shouldCache $request->{url};
}
push @left2, $request->{storePath};
} else {
$insertNARExistence->execute($cache->{id}, basename($request->{storePath}), 1, time())
if shouldCache $request->{url};
print "$request->{storePath}\n";
}
}
@left = @left2;
}
}
sub downloadBinary {
my ($storePath, $destPath) = @_;
foreach my $cache (@caches) {
my $info = getCachedInfoFrom($storePath, $cache);
unless (defined $info) {
next if negativeHit($storePath, $cache);
my $request = addRequest($storePath, infoUrl($cache->{url}, $storePath));
processRequests;
$info = processNARInfo($storePath, $cache, $request);
}
next unless defined $info;
my $decompressor;
if ($info->{compression} eq "bzip2") { $decompressor = "| $Nix::Config::bzip2 -d"; }
elsif ($info->{compression} eq "xz") { $decompressor = "| $Nix::Config::xz -d"; }
elsif ($info->{compression} eq "none") { $decompressor = ""; }
else {
print STDERR "unknown compression method ‘$info->{compression}’\n";
next;
}
my $url = "$cache->{url}/$info->{url}"; # FIXME: handle non-relative URLs
die if $requireSignedBinaryCaches && !defined $info->{signedBy};
print STDERR "\n*** Downloading ‘$url’ ", ($requireSignedBinaryCaches ? "(signed by ‘$info->{signedBy}’) " : ""), "to ‘$storePath’...\n";
checkURL $url;
if (system("$Nix::Config::curl --fail --location --netrc-file $netrcFile --netrc-optional --connect-timeout $curlConnectTimeout -A '$userAgent' $Nix::Config::curlCaFlag '$url' $decompressor | $Nix::Config::binDir/nix-store --restore $destPath") != 0) {
warn "download of ‘$url’ failed" . ($! ? ": $!" : "") . "\n";
next;
}
# Tell Nix about the expected hash so it can verify it.
die unless defined $info->{narHash} && $info->{narHash} ne "";
print "$info->{narHash}\n";
print STDERR "\n";
return;
}
print STDERR "could not download ‘$storePath’ from any binary cache\n";
exit 1;
}
# Bail out right away if binary caches are disabled.
exit 0 if
($Nix::Config::config{"use-binary-caches"} // "true") eq "false" ||
($Nix::Config::config{"untrusted-use-binary-caches"} // "true") eq "false";
print "\n";
flush STDOUT;
initCache();
if ($ARGV[0] eq "--query") {
while (<STDIN>) {
getAvailableCaches;
chomp;
my ($cmd, @args) = split " ", $_;
if ($cmd eq "have") {
print STDERR "checking binary caches for existence of @args\n" if $debug;
printSubstitutablePaths(@args);
print "\n";
}
elsif ($cmd eq "info") {
print STDERR "checking binary caches for info on @args\n" if $debug;
printInfoParallel(@args);
print "\n";
}
else { die "unknown command ‘$cmd’"; }
flush STDOUT;
}
}
elsif ($ARGV[0] eq "--substitute") {
my $storePath = $ARGV[1] or die;
my $destPath = $ARGV[2] or die;
getAvailableCaches;
downloadBinary($storePath, $destPath);
}
else {
die;
}
+376
View File
@@ -0,0 +1,376 @@
#! @perl@ -w @perlFlags@
use utf8;
use strict;
use Nix::Config;
use Nix::Manifest;
use Nix::Store;
use Nix::Utils;
use POSIX qw(strftime);
STDOUT->autoflush(1);
binmode STDERR, ":encoding(utf8)";
my $logFile = "$Nix::Config::logDir/downloads";
# For queries, skip expensive calls to nix-hash etc. We're just
# estimating the expected download size.
my $fast = 1;
my $curl = "$Nix::Config::curl $Nix::Config::curlCaFlag --fail --location";
# Open the manifest cache and update it if necessary.
my $dbh = updateManifestDB();
exit 0 unless defined $dbh; # exit if there are no manifests
print "\n";
# $hashCache->{$algo}->{$path} yields the $algo-hash of $path.
my $hashCache;
sub parseHash {
my $hash = shift;
if ($hash =~ /^(.+):(.+)$/) {
return ($1, $2);
} else {
return ("md5", $hash);
}
}
# Compute the most efficient sequence of downloads to produce the
# given path.
sub computeSmallestDownload {
my $targetPath = shift;
# Build a graph of all store paths that might contribute to the
# construction of $targetPath, and the special node "start". The
# edges are either patch operations, or downloads of full NAR
# files. The latter edges only occur between "start" and a store
# path.
my %graph;
$graph{"start"} = {d => 0, pred => undef, edges => []};
my @queue = ();
my $queueFront = 0;
my %done;
sub addNode {
my $graph = shift;
my $u = shift;
$$graph{$u} = {d => 999999999999, pred => undef, edges => []}
unless defined $$graph{$u};
}
sub addEdge {
my $graph = shift;
my $u = shift;
my $v = shift;
my $w = shift;
my $type = shift;
my $info = shift;
addNode $graph, $u;
push @{$$graph{$u}->{edges}},
{weight => $w, start => $u, end => $v, type => $type, info => $info};
my $n = scalar @{$$graph{$u}->{edges}};
}
push @queue, $targetPath;
while ($queueFront < scalar @queue) {
my $u = $queue[$queueFront++];
next if defined $done{$u};
$done{$u} = 1;
addNode \%graph, $u;
# If the path already exists, it has distance 0 from the
# "start" node.
if (isValidPath($u)) {
addEdge \%graph, "start", $u, 0, "present", undef;
}
else {
# Add patch edges.
my $patchList = $dbh->selectall_arrayref(
"select * from Patches where storePath = ?",
{ Slice => {} }, $u);
foreach my $patch (@{$patchList}) {
if (isValidPath($patch->{basePath})) {
my ($baseHashAlgo, $baseHash) = parseHash $patch->{baseHash};
my $hash = $hashCache->{$baseHashAlgo}->{$patch->{basePath}};
if (!defined $hash) {
$hash = $fast && $baseHashAlgo eq "sha256"
? queryPathHash($patch->{basePath})
: hashPath($baseHashAlgo, $baseHashAlgo ne "md5", $patch->{basePath});
$hash =~ s/.*://;
$hashCache->{$baseHashAlgo}->{$patch->{basePath}} = $hash;
}
next if $hash ne $baseHash;
}
push @queue, $patch->{basePath};
addEdge \%graph, $patch->{basePath}, $u, $patch->{size}, "patch", $patch;
}
# Add NAR file edges to the start node.
my $narFileList = $dbh->selectall_arrayref(
"select * from NARs where storePath = ?",
{ Slice => {} }, $u);
foreach my $narFile (@{$narFileList}) {
# !!! how to handle files whose size is not known in advance?
# For now, assume some arbitrary size (1 GB).
# This has the side-effect of preferring non-Hydra downloads.
addEdge \%graph, "start", $u, ($narFile->{size} || 1000000000), "narfile", $narFile;
}
}
}
# Run Dijkstra's shortest path algorithm to determine the shortest
# sequence of download and/or patch actions that will produce
# $targetPath.
my @todo = keys %graph;
while (scalar @todo > 0) {
# Remove the closest element from the todo list.
# !!! inefficient, use a priority queue
@todo = sort { -($graph{$a}->{d} <=> $graph{$b}->{d}) } @todo;
my $u = pop @todo;
my $u_ = $graph{$u};
foreach my $edge (@{$u_->{edges}}) {
my $v_ = $graph{$edge->{end}};
if ($v_->{d} > $u_->{d} + $edge->{weight}) {
$v_->{d} = $u_->{d} + $edge->{weight};
# Store the edge; to edge->start is actually the
# predecessor.
$v_->{pred} = $edge;
}
}
}
# Retrieve the shortest path from "start" to $targetPath.
my @path = ();
my $cur = $targetPath;
return () unless defined $graph{$targetPath}->{pred};
while ($cur ne "start") {
push @path, $graph{$cur}->{pred};
$cur = $graph{$cur}->{pred}->{start};
}
return @path;
}
# Parse the arguments.
if ($ARGV[0] eq "--query") {
while (<STDIN>) {
chomp;
my ($cmd, @args) = split " ", $_;
if ($cmd eq "have") {
foreach my $storePath (@args) {
print "$storePath\n" if scalar @{$dbh->selectcol_arrayref("select 1 from NARs where storePath = ?", {}, $storePath)} > 0;
}
print "\n";
}
elsif ($cmd eq "info") {
foreach my $storePath (@args) {
my $infos = $dbh->selectall_arrayref(
"select * from NARs where storePath = ?",
{ Slice => {} }, $storePath);
next unless scalar @{$infos} > 0;
my $info = @{$infos}[0];
print "$storePath\n";
print "$info->{deriver}\n";
my @references = split " ", $info->{refs};
print scalar @references, "\n";
print "$_\n" foreach @references;
my @path = computeSmallestDownload $storePath;
my $downloadSize = 0;
while (scalar @path > 0) {
my $edge = pop @path;
my $u = $edge->{start};
my $v = $edge->{end};
if ($edge->{type} eq "patch") {
$downloadSize += $edge->{info}->{size} || 0;
}
elsif ($edge->{type} eq "narfile") {
$downloadSize += $edge->{info}->{size} || 0;
}
}
print "$downloadSize\n";
my $narSize = $info->{narSize} || 0;
print "$narSize\n";
}
print "\n";
}
else { die "unknown command ‘$cmd’"; }
}
exit 0;
}
elsif ($ARGV[0] ne "--substitute") {
die;
}
die unless scalar @ARGV == 3;
my $targetPath = $ARGV[1];
my $destPath = $ARGV[2];
$fast = 0;
# Create a temporary directory.
my $tmpDir = mkTempDir("nix-download");
my $tmpNar = "$tmpDir/nar";
my $tmpNar2 = "$tmpDir/nar2";
open LOGFILE, ">>$logFile" or die "cannot open log file $logFile";
my $date = strftime ("%F %H:%M:%S UTC", gmtime (time));
print LOGFILE "$$ get $targetPath $date\n";
print STDERR "\n*** Trying to download/patch ‘$targetPath’\n";
# Compute the shortest path.
my @path = computeSmallestDownload $targetPath;
die "don't know how to produce $targetPath\n" if scalar @path == 0;
# We don't need the manifest anymore, so close it as an optimisation:
# if we still have SQLite locks blocking other processes (we
# shouldn't), this gets rid of them.
$dbh->disconnect;
# Traverse the shortest path, perform the actions described by the
# edges.
my $curStep = 1;
my $maxStep = scalar @path;
my $finalNarHash;
while (scalar @path > 0) {
my $edge = pop @path;
my $u = $edge->{start};
my $v = $edge->{end};
print STDERR "\n*** Step $curStep/$maxStep: ";
if ($edge->{type} eq "present") {
print STDERR "using already present path ‘$v’\n";
print LOGFILE "$$ present $v\n";
if ($curStep < $maxStep) {
# Since this is not the last step, the path will be used
# as a base to one or more patches. So turn the base path
# into a NAR archive, to which we can apply the patch.
print STDERR " packing base path...\n";
system("$Nix::Config::binDir/nix-store --dump $v > $tmpNar") == 0
or die "cannot dump ‘$v’";
}
}
elsif ($edge->{type} eq "patch") {
my $patch = $edge->{info};
print STDERR "applying patch ‘$patch->{url}’ to ‘$u’ to create ‘$v’\n";
print LOGFILE "$$ patch $patch->{url} $patch->{size} $patch->{baseHash} $u $v\n";
# Download the patch.
print STDERR " downloading patch...\n";
my $patchPath = "$tmpDir/patch";
checkURL $patch->{url};
system("$curl '$patch->{url}' -o $patchPath") == 0
or die "cannot download patch ‘$patch->{url}’\n";
# Apply the patch to the NAR archive produced in step 1 (for
# the already present path) or a later step (for patch sequences).
print STDERR " applying patch...\n";
system("$Nix::Config::libexecDir/nix/bspatch $tmpNar $tmpNar2 $patchPath") == 0
or die "cannot apply patch ‘$patchPath’ to $tmpNar\n";
if ($curStep < $maxStep) {
# The archive will be used as the base of the next patch.
rename "$tmpNar2", "$tmpNar" or die "cannot rename NAR archive: $!";
} else {
# This was the last patch. Unpack the final NAR archive
# into the target path.
print STDERR " unpacking patched archive...\n";
system("$Nix::Config::binDir/nix-store --restore $destPath < $tmpNar2") == 0
or die "cannot unpack $tmpNar2 to ‘$v’\n";
}
$finalNarHash = $patch->{narHash};
}
elsif ($edge->{type} eq "narfile") {
my $narFile = $edge->{info};
print STDERR "downloading ‘$narFile->{url}’ to ‘$v’\n";
my $size = $narFile->{size} || -1;
print LOGFILE "$$ narfile $narFile->{url} $size $v\n";
checkURL $narFile->{url};
my $decompressor =
$narFile->{compressionType} eq "bzip2" ? "| $Nix::Config::bzip2 -d" :
$narFile->{compressionType} eq "xz" ? "| $Nix::Config::xz -d" :
$narFile->{compressionType} eq "none" ? "" :
die "unknown compression type ‘$narFile->{compressionType}’";
if ($curStep < $maxStep) {
# The archive will be used a base to a patch.
system("$curl '$narFile->{url}' $decompressor > $tmpNar") == 0
or die "cannot download and unpack ‘$narFile->{url}’ to ‘$v’\n";
} else {
# Unpack the archive to the target path.
system("$curl '$narFile->{url}' $decompressor | $Nix::Config::binDir/nix-store --restore '$destPath'") == 0
or die "cannot download and unpack ‘$narFile->{url}’ to ‘$v’\n";
}
$finalNarHash = $narFile->{narHash};
}
$curStep++;
}
# Tell Nix about the expected hash so it can verify it.
die "cannot check integrity of the downloaded path since its hash is not known\n"
unless defined $finalNarHash;
print "$finalNarHash\n";
print STDERR "\n";
print LOGFILE "$$ success\n";
close LOGFILE;
+79
View File
@@ -0,0 +1,79 @@
#! @perl@ -w @perlFlags@
use strict;
use Nix::Utils;
use Nix::Config;
sub readProc {
return unless -d "/proc";
opendir DIR, "/proc" or return;
foreach my $name (readdir DIR) {
next unless $name =~ /^\d+$/;
my $process = "/proc/$name";
#print STDERR "=== $process\n";
my $target;
print "$target\n" if $target = readlink "$process/exe";
print "$target\n" if $target = readlink "$process/cwd";
if (opendir FDS, "$process/fd") {
foreach my $name (readdir FDS) {
$target = readlink "$process/fd/$name";
print "$target\n" if $target && substr($target, 0, 1) eq "/";
}
closedir FDS;
}
if (open MAP, "<$process/maps") {
while (<MAP>) {
next unless /^ \s* \S+ \s+ \S+ \s+ \S+ \s+ \S+ \s+ \S+ \s+ (\/\S+) \s* $/x;
print "$1\n";
}
close MAP;
}
# Get all store paths that appear in the environment of this process.
eval {
my $env = Nix::Utils::readFile "$process/environ";
my @matches = $env =~ /\Q$Nix::Config::storeDir\E\/[0-9a-z]+[0-9a-zA-Z\+\-\._\?=]*/g;
print "$_\n" foreach @matches;
}
}
closedir DIR;
}
sub lsof {
return unless open LSOF, "lsof -n -w -F n 2> /dev/null |";
while (<LSOF>) {
next unless /^n (\/ .*)$/x;
print $1, "\n";
}
close LSOF;
}
readProc;
lsof;
sub printFile {
my ($fn) = @_;
if (-e $fn) {
print Nix::Utils::readFile($fn), "\n";
}
}
# This is rather NixOS-specific, so it probably shouldn't be here.
printFile "/proc/sys/kernel/modprobe";
printFile "/proc/sys/kernel/fbsplash";
printFile "/proc/sys/kernel/poweroff_cmd";
+845
View File
@@ -0,0 +1,845 @@
#!/bin/bash
set -eu
set -o pipefail
# Sourced from:
# - https://github.com/LnL7/nix-darwin/blob/8c29d0985d74b4a990238497c47a2542a5616b3c/bootstrap.sh
# - https://gist.github.com/expipiplus1/e571ce88c608a1e83547c918591b149f/ac504c6c1b96e65505fbda437a28ce563408ecb0
# - https://github.com/NixOS/nixos-org-configurations/blob/a122f418797713d519aadf02e677fce0dc1cb446/delft/scripts/nix-mac-installer.sh
# - https://github.com/matthewbauer/macNixOS/blob/f6045394f9153edea417be90c216788e754feaba/install-macNixOS.sh
# - https://gist.github.com/LnL7/9717bd6cdcb30b086fd7f2093e5f8494/86b26f852ce563e973acd30f796a9a416248c34a
#
# however tracking which bits came from which would be impossible.
readonly ESC='\033[0m'
readonly BOLD='\033[38;1m'
readonly BLUE='\033[38;34m'
readonly BLUE_UL='\033[38;4;34m'
readonly GREEN='\033[38;32m'
readonly GREEN_UL='\033[38;4;32m'
readonly RED='\033[38;31m'
readonly RED_UL='\033[38;4;31m'
readonly YELLOW='\033[38;33m'
readonly YELLOW_UL='\033[38;4;33m'
readonly CORES=$(sysctl -n hw.ncpu)
readonly NIX_USER_COUNT="$CORES"
readonly NIX_BUILD_GROUP_ID="30000"
readonly NIX_BUILD_GROUP_NAME="nixbld"
readonly NIX_FIRST_BUILD_UID="30001"
# Please don't change this. We don't support it, because the
# default shell profile that comes with Nix doesn't support it.
readonly NIX_ROOT="/nix"
readonly PLIST_DEST=/Library/LaunchDaemons/org.nixos.nix-daemon.plist
readonly PROFILE_TARGETS=("/etc/profile" "/etc/bashrc" "/etc/zshrc")
readonly PROFILE_BACKUP_SUFFIX=".backup-before-nix"
readonly PROFILE_NIX_FILE="$NIX_ROOT/var/nix/profiles/default/etc/profile.d/nix-daemon.sh"
readonly NIX_INSTALLED_NIX="@nix@"
readonly NIX_INSTALLED_CACERT="@cacert@"
readonly EXTRACTED_NIX_PATH="$(dirname "$0")"
readonly ROOT_HOME="/var/root"
if [ -t 0 ]; then
readonly IS_HEADLESS='no'
else
readonly IS_HEADLESS='yes'
fi
headless() {
if [ "$IS_HEADLESS" = "yes" ]; then
return 0
else
return 1
fi
}
contactme() {
echo "We'd love to help if you need it."
echo ""
echo "If you can, open an issue at https://github.com/nixos/nix/issues"
echo ""
echo "Or feel free to contact the team,"
echo " - on IRC #nixos on irc.freenode.net"
echo " - on twitter @nixos_org"
}
uninstall_directions() {
subheader "Uninstalling nix:"
local step=0
if [ -e "$PLIST_DEST" ]; then
step=$((step + 1))
cat <<EOF
$step. Delete $PLIST_DEST
sudo launchctl unload $PLIST_DEST
sudo rm $PLIST_DEST
EOF
fi
for profile_target in "${PROFILE_TARGETS[@]}"; do
if [ -e "$profile_target" ] && [ -e "$profile_target$PROFILE_BACKUP_SUFFIX" ]; then
step=$((step + 1))
cat <<EOF
$step. Restore $profile_target$PROFILE_BACKUP_SUFFIX back to $profile_target
sudo mv $profile_target$PROFILE_BACKUP_SUFFIX $profile_target
(after this one, you may need to re-open any terminals that were
opened while it existed.)
EOF
fi
done
step=$((step + 1))
cat <<EOF
$step. Delete the files Nix added to your system:
sudo rm -rf /etc/nix $NIX_ROOT $ROOT_HOME/.nix-profile $ROOT_HOME/.nix-defexpr $ROOT_HOME/.nix-channels $HOME/.nix-profile $HOME/.nix-defexpr $HOME/.nix-channels
and that is it.
EOF
}
nix_user_for_core() {
printf "nixbld%d" "$1"
}
nix_uid_for_core() {
echo $((NIX_FIRST_BUILD_UID + $1 - 1))
}
dsclattr() {
/usr/bin/dscl . -read "$1" \
| awk "/$2/ { print \$2 }"
}
_textout() {
echo -en "$1"
shift
if [ "$*" = "" ]; then
cat
else
echo "$@"
fi
echo -en "$ESC"
}
header() {
follow="---------------------------------------------------------"
header=$(echo "---- $* $follow$follow$follow" | head -c 80)
echo ""
_textout "$BLUE" "$header"
}
warningheader() {
follow="---------------------------------------------------------"
header=$(echo "---- $* $follow$follow$follow" | head -c 80)
echo ""
_textout "$RED" "$header"
}
subheader() {
echo ""
_textout "$BLUE_UL" "$*"
}
row() {
printf "$BOLD%s$ESC:\t%s\n" "$1" "$2"
}
task() {
echo ""
ok "~~> $1"
}
bold() {
echo "$BOLD$*$ESC"
}
ok() {
_textout "$GREEN" "$@"
}
warning() {
warningheader "warning!"
cat
echo ""
}
failure() {
header "oh no!"
_textout "$RED" "$@"
echo ""
_textout "$RED" "$(contactme)"
trap finish_cleanup EXIT
exit 1
}
ui_confirm() {
_textout "$GREEN$GREEN_UL" "$1"
if headless; then
echo "No TTY, assuming you would say yes :)"
return 0
fi
local prompt="[y/n] "
echo -n "$prompt"
while read -r y; do
if [ "$y" = "y" ]; then
echo ""
return 0
elif [ "$y" = "n" ]; then
echo ""
return 1
else
_textout "$RED" "Sorry, I didn't understand. I can only understand answers of y or n"
echo -n "$prompt"
fi
done
echo ""
return 1
}
__sudo() {
local expl="$1"
local cmd="$2"
shift
header "sudo execution"
echo "I am executing:"
echo ""
printf " $ sudo %s\n" "$cmd"
echo ""
echo "$expl"
echo ""
return 0
}
_sudo() {
local expl="$1"
shift
if ! headless; then
__sudo "$expl" "$*"
fi
sudo "$@"
}
readonly SCRATCH=$(mktemp -d -t tmp.XXXXXXXXXX)
function finish_cleanup {
rm -rf "$SCRATCH"
}
function finish_fail {
finish_cleanup
failure <<EOF
Jeeze, something went wrong. If you can take all the output and open
an issue, we'd love to fix the problem so nobody else has this issue.
:(
EOF
}
trap finish_fail EXIT
function finish_success {
finish_cleanup
ok "Alright! We're done!"
cat <<EOF
Before Nix will work in your existing shells, you'll need to either
run:
$ source $PROFILE_NIX_FILE
or close them and open them again. Other than that, you should be
ready to go.
Try it! Open a new terminal, and type:
$ nix-shell -p nix-info --run "nix-info -m"
Thank you for using this installer. If you have any feedback, don't
hesitate:
$(contactme)
EOF
}
validate_starting_assumptions() {
if [ "$(uname -s)" != "Darwin" ]; then
failure "This script is for use with macOS!"
fi
if [ $EUID -eq 0 ]; then
failure <<EOF
Please do not run this script with root privileges. We will call sudo
when we need to.
EOF
fi
if type nix-env 2> /dev/null >&2; then
failure <<EOF
Nix already appears to be installed, and this tool assumes it is
_not_ yet installed.
$(uninstall_directions)
EOF
fi
if pgrep nix-daemon 2> /dev/null >&2; then
failure <<EOF
Nix seems to be partially installed, because the nix-daemon is
currently running. It may be in a launchd service. Please stop the
the old nix-daemon, and try again.
If you have an existing launchd plist for nix-daemon, please delete
it.
$(uninstall_directions)
EOF
fi
if [ "${NIX_REMOTE:-}" != "" ]; then
failure <<EOF
For some reason, \$NIX_REMOTE is set. It really should not be set
before this installer runs, and it hints that Nix is currently
installed. Please delete the old Nix installation and start again.
Note: You might need to close your shell window and open a new shell
to clear the variable.
EOF
fi
if echo "${SSL_CERT_FILE:-}" | grep -qE "(nix/var/nix|nix-profile)"; then
failure <<EOF
It looks like \$SSL_CERT_FILE is set to a path that used to be part of
the old Nix installation. Please unset that variable and try again:
$ unset SSL_CERT_FILE
EOF
fi
for file in ~/.bash_profile ~/.bash_login ~/.profile ~/.zshenv ~/.zprofile ~/.zshrc ~/.zlogin; do
if [ -f "$file" ]; then
if grep -l "^[^#].*.nix-profile" "$file"; then
failure <<EOF
I found a reference to a ".nix-profile" in $file.
This has a high chance of breaking a new nix installation. It was most
likely put there by a previous Nix installer.
Please remove this reference and try running this again. You should
also look for similar references in:
- ~/.bash_profile
- ~/.bash_login
- ~/.profile
or other shell init files that you may have.
$(uninstall_directions)
EOF
fi
fi
done
if [ -d /nix ]; then
failure <<EOF
There are some relics of a previous installation of Nix at /nix, and
this scripts assumes Nix is _not_ yet installed. Please delete the old
Nix installation and start again.
$(uninstall_directions)
EOF
fi
if [ -d /etc/nix ]; then
failure <<EOF
There are some relics of a previous installation of Nix at /etc/nix, and
this scripts assumes Nix is _not_ yet installed. Please delete the old
Nix installation and start again.
$(uninstall_directions)
EOF
fi
for profile_target in "${PROFILE_TARGETS[@]}"; do
if [ -e "$profile_target$PROFILE_BACKUP_SUFFIX" ]; then
failure <<EOF
When this script runs, it backs up the current $profile_target to
$profile_target$PROFILE_BACKUP_SUFFIX. This backup file already exists, though.
Please follow these instructions to clean up the old backup file:
1. Copy $profile_target and $profile_target$PROFILE_BACKUP_SUFFIX to another place, just
in case.
2. Take care to make sure that $profile_target$PROFILE_BACKUP_SUFFIX doesn't look like
it has anything nix-related in it. If it does, something is probably
quite wrong. Please open an issue or get in touch immediately.
3. Take care to make sure that $profile_target doesn't look like it has
anything nix-related in it. If it does, and $profile_target _did not_,
run:
$ /usr/bin/sudo /bin/mv $profile_target$PROFILE_BACKUP_SUFFIX $profile_target
and try again.
EOF
fi
if grep -qi "nix" "$profile_target"; then
failure <<EOF
It looks like $profile_target already has some Nix configuration in
there. There should be no reason to run this again. If you're having
trouble, please open an issue.
EOF
fi
done
danger_paths=("$ROOT_HOME/.nix-defexpr" "$ROOT_HOME/.nix-channels" "$ROOT_HOME/.nix-profile")
for danger_path in "${danger_paths[@]}"; do
if _sudo "making sure that $danger_path doesn't exist" \
test -e "$danger_path"; then
failure <<EOF
I found a file at $danger_path, which is a relic of a previous
installation. You must first delete this file before continuing.
$(uninstall_directions)
EOF
fi
done
}
setup_report() {
header "hardware report"
row " Cores" "$CORES"
header "Nix config report"
row " Temp Dir" "$SCRATCH"
row " Nix Root" "$NIX_ROOT"
row " Build Users" "$NIX_USER_COUNT"
row " Build Group ID" "$NIX_BUILD_GROUP_ID"
row "Build Group Name" "$NIX_BUILD_GROUP_NAME"
if [ "${ALLOW_PREEXISTING_INSTALLATION:-}" != "" ]; then
row "Preexisting Install" "Allowed"
fi
subheader "build users:"
row " Username" "UID"
for i in $(seq 1 "$NIX_USER_COUNT"); do
row " $(nix_user_for_core "$i")" "$(nix_uid_for_core "$i")"
done
echo ""
}
create_build_group() {
local primary_group_id
task "Setting up the build group $NIX_BUILD_GROUP_NAME"
if ! /usr/bin/dscl . -read "/Groups/$NIX_BUILD_GROUP_NAME" > /dev/null 2>&1; then
_sudo "Create the Nix build group, $NIX_BUILD_GROUP_NAME" \
/usr/sbin/dseditgroup -o create \
-r "Nix build group for nix-daemon" \
-i "$NIX_BUILD_GROUP_ID" \
"$NIX_BUILD_GROUP_NAME" >&2
row " Created" "Yes"
else
primary_group_id=$(dsclattr "/Groups/$NIX_BUILD_GROUP_NAME" "PrimaryGroupID")
if [ "$primary_group_id" -ne "$NIX_BUILD_GROUP_ID" ]; then
failure <<EOF
It seems the build group $NIX_BUILD_GROUP_NAME already exists, but
with the UID $primary_group_id. This script can't really handle
that right now, so I'm going to give up.
You can fix this by editing this script and changing the
NIX_BUILD_GROUP_ID variable near the top to from $NIX_BUILD_GROUP_ID
to $primary_group_id and re-run.
EOF
else
row " Exists" "Yes"
fi
fi
}
create_build_user_for_core() {
local coreid
local username
local uid
coreid="$1"
username=$(nix_user_for_core "$coreid")
uid=$(nix_uid_for_core "$coreid")
dsclpath="/Users/$username"
task "Setting up the build user $username"
if ! /usr/bin/dscl . -read "$dsclpath" > /dev/null 2>&1; then
_sudo "Creating the Nix build user, $username" \
/usr/bin/dscl . create "$dsclpath" \
UniqueID "${uid}"
row " Created" "Yes"
else
actual_uid=$(dsclattr "$dsclpath" "UniqueID")
if [ "$actual_uid" -ne "$uid" ]; then
failure <<EOF
It seems the build user $username already exists, but with the UID
with the UID $actual_uid. This script can't really handle that right
now, so I'm going to give up.
If you already created the users and you know they start from
$actual_uid and go up from there, you can edit this script and change
NIX_FIRST_BUILD_UID near the top of the file to $actual_uid and try
again.
EOF
else
row " Exists" "Yes"
fi
fi
if [ "$(dsclattr "$dsclpath" "IsHidden")" = "1" ]; then
row " IsHidden" "Yes"
else
_sudo "in order to make $username a hidden user" \
/usr/bin/dscl . -create "$dsclpath" "IsHidden" "1"
row " IsHidden" "Yes"
fi
if [ "$(dsclattr "$dsclpath" "NFSHomeDirectory")" = "/var/empty" ]; then
row " NFSHomeDirectory" "/var/empty"
else
_sudo "in order to give $username a safe home directory" \
/usr/bin/dscl . -create "$dsclpath" "NFSHomeDirectory" "/var/empty"
row " NFSHomeDirectory" "/var/empty"
fi
if [ "$(dsclattr "$dsclpath" "RealName")" = "Nix build user $coreid" ]; then
row " RealName" "Nix build user $coreid"
else
_sudo "in order to give $username a useful name" \
/usr/bin/dscl . -create "$dsclpath" "RealName" "Nix build user $coreid"
row " RealName" "Nix build user $coreid"
fi
if [ "$(dsclattr "$dsclpath" "UserShell")" = "/sbin/nologin" ]; then
row " Logins Disabled" "Yes"
else
_sudo "in order to prevent $username from logging in" \
/usr/bin/dscl . -create "$dsclpath" "UserShell" "/sbin/nologin"
row " Logins Disabled" "Yes"
fi
if dseditgroup -o checkmember -m "$username" "$NIX_BUILD_GROUP_NAME" > /dev/null 2>&1 ; then
row " Member of $NIX_BUILD_GROUP_NAME" "Yes"
else
_sudo "Add $username to the $NIX_BUILD_GROUP_NAME group"\
/usr/sbin/dseditgroup -o edit -t user \
-a "$username" "$NIX_BUILD_GROUP_NAME"
row " Member of $NIX_BUILD_GROUP_NAME" "Yes"
fi
if [ "$(dsclattr "$dsclpath" "PrimaryGroupID")" = "$NIX_BUILD_GROUP_ID" ]; then
row " PrimaryGroupID" "$NIX_BUILD_GROUP_ID"
else
_sudo "to let the nix daemon use this user for builds (this might seem redundant, but there are two concepts of group membership)" \
/usr/bin/dscl . -create "$dsclpath" "PrimaryGroupID" "$NIX_BUILD_GROUP_ID"
row " PrimaryGroupID" "$NIX_BUILD_GROUP_ID"
fi
}
create_build_users() {
for i in $(seq 1 "$NIX_USER_COUNT"); do
create_build_user_for_core "$i"
done
}
create_directories() {
_sudo "to make the basic directory structure of Nix (part 1)" \
mkdir -pv -m 0755 /nix /nix/var /nix/var/log /nix/var/log/nix /nix/var/log/nix/drvs /nix/var/nix{,/db,/gcroots,/profiles,/temproots,/userpool}
_sudo "to make the basic directory structure of Nix (part 2)" \
mkdir -pv -m 1777 /nix/var/nix/{gcroots,profiles}/per-user
_sudo "to make the basic directory structure of Nix (part 3)" \
mkdir -pv -m 1775 /nix/store
_sudo "to make the basic directory structure of Nix (part 4)" \
chgrp "$NIX_BUILD_GROUP_NAME" /nix/store
_sudo "to set up the root user's profile (part 1)" \
mkdir -pv -m 0755 /nix/var/nix/profiles/per-user/root
_sudo "to set up the root user's profile (part 2)" \
mkdir -pv -m 0700 "$ROOT_HOME/.nix-defexpr"
_sudo "to place the default nix daemon configuration (part 1)" \
mkdir -pv -m 0555 /etc/nix
}
place_channel_configuration() {
echo "https://nixos.org/channels/nixpkgs-unstable nixpkgs" > "$SCRATCH/.nix-channels"
_sudo "to set up the default system channel (part 1)" \
install -m 0664 "$SCRATCH/.nix-channels" "$ROOT_HOME/.nix-channels"
}
welcome_to_nix() {
ok "Welcome to the Multi-User Nix Installation"
cat <<EOF
This installation tool will set up your computer with the Nix package
manager. This will happen in a few stages:
1. Make sure your computer doesn't already have Nix. If it does, I
will show you instructions on how to clean up your old one.
2. Show you what we are going to install and where. Then we will ask
if you are ready to continue.
3. Create the system users and groups that the Nix daemon uses to run
builds.
4. Perform the basic installation of the Nix files daemon.
5. Configure your shell to import special Nix Profile files, so you
can use Nix.
6. Start the Nix daemon.
EOF
if ui_confirm "Would you like to see a more detailed list of what we will do?"; then
cat <<EOF
We will:
- make sure your computer doesn't already have Nix files
(if it does, I will tell you how to clean them up.)
- create local users (see the list above for the users we'll make)
- create a local group ($NIX_BUILD_GROUP_NAME)
- install Nix in to $NIX_ROOT
- create a configuration file in /etc/nix
- set up the "default profile" by creating some Nix-related files in
$ROOT_HOME
EOF
for profile_target in "${PROFILE_TARGETS[@]}"; do
if [ -e "$profile_target" ]; then
cat <<EOF
- back up $profile_target to $profile_target$PROFILE_BACKUP_SUFFIX
- update $profile_target to include some Nix configuration
EOF
fi
done
cat <<EOF
- load and start a LaunchDaemon (at $PLIST_DEST) for nix-daemon
EOF
if ! ui_confirm "Ready to continue?"; then
failure <<EOF
Okay, maybe you would like to talk to the team.
EOF
fi
fi
}
chat_about_sudo() {
header "let's talk about sudo"
if headless; then
cat <<EOF
This script is going to call sudo a lot. Normally, it would show you
exactly what commands it is running and why. However, the script is
run in a headless fashion, like this:
$ curl https://nixos.org/nix/install | sh
or maybe in a CI pipeline. Because of that, we're going to skip the
verbose output in the interest of brevity.
If you would like to
see the output, try like this:
$ curl -o install-nix https://nixos.org/nix/install
$ sh ./install-nix
EOF
return 0
fi
cat <<EOF
This script is going to call sudo a lot. Every time we do, it'll
output exactly what it'll do, and why.
Just like this:
EOF
__sudo "to demonstrate how our sudo prompts look" \
echo "this is a sudo prompt"
cat <<EOF
This might look scary, but everything can be undone by running just a
few commands. We used to ask you to confirm each time sudo ran, but it
was too many times. Instead, I'll just ask you this one time:
EOF
if ui_confirm "Can we use sudo?"; then
ok "Yay! Thanks! Let's get going!"
else
failure <<EOF
That is okay, but we can't install.
EOF
fi
}
install_from_extracted_nix() {
(
cd "$EXTRACTED_NIX_PATH"
_sudo "to copy the basic Nix files to the new store at $NIX_ROOT/store" \
rsync -rlpt "$(pwd)/store/" "$NIX_ROOT/store/"
if [ -d "$NIX_INSTALLED_NIX" ]; then
echo " Alright! We have our first nix at $NIX_INSTALLED_NIX"
else
failure <<EOF
Something went wrong, and I didn't find Nix installed at
$NIX_INSTALLED_NIX.
EOF
fi
_sudo "to initialize the Nix Database" \
$NIX_INSTALLED_NIX/bin/nix-store --init
cat ./.reginfo \
| _sudo "to load data for the first time in to the Nix Database" \
"$NIX_INSTALLED_NIX/bin/nix-store" --load-db
echo " Just finished getting the nix database ready."
)
}
shell_source_lines() {
cat <<EOF
# Nix
if [ -e '$PROFILE_NIX_FILE' ]; then
. '$PROFILE_NIX_FILE'
fi
# End Nix
EOF
}
configure_shell_profile() {
for profile_target in "${PROFILE_TARGETS[@]}"; do
if [ -e "$profile_target" ]; then
_sudo "to back up your current $profile_target to $profile_target$PROFILE_BACKUP_SUFFIX" \
cp "$profile_target" "$profile_target$PROFILE_BACKUP_SUFFIX"
shell_source_lines \
| _sudo "extend your $profile_target with nix-daemon settings" \
tee -a "$profile_target"
fi
done
}
setup_default_profile() {
_sudo "to installing a bootstrapping Nix in to the default Profile" \
HOME=$ROOT_HOME "$NIX_INSTALLED_NIX/bin/nix-env" -i "$NIX_INSTALLED_NIX"
_sudo "to installing a bootstrapping SSL certificate just for Nix in to the default Profile" \
HOME=$ROOT_HOME "$NIX_INSTALLED_NIX/bin/nix-env" -i "$NIX_INSTALLED_CACERT"
_sudo "to update the default channel in the default profile" \
HOME=$ROOT_HOME NIX_SSL_CERT_FILE=/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt "$NIX_INSTALLED_NIX/bin/nix-channel" --update nixpkgs
}
place_nix_configuration() {
cat <<EOF > "$SCRATCH/nix.conf"
build-users-group = $NIX_BUILD_GROUP_NAME
build-max-jobs = $NIX_USER_COUNT
build-cores = 1
build-use-sandbox = false
binary-caches = https://cache.nixos.org/
trusted-binary-caches =
binary-cache-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
signed-binary-caches = *
trusted-users = root
allowed-users = *
EOF
_sudo "to place the default nix daemon configuration (part 2)" \
install -m 0664 "$SCRATCH/nix.conf" /etc/nix/nix.conf
}
configure_nix_daemon_plist() {
_sudo "to set up the nix-daemon as a LaunchDaemon" \
ln -sfn "/nix/var/nix/profiles/default$PLIST_DEST" "$PLIST_DEST"
_sudo "to load the LaunchDaemon plist for nix-daemon" \
launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist
_sudo "to start the nix-daemon" \
launchctl start org.nixos.nix-daemon
}
main() {
welcome_to_nix
chat_about_sudo
if [ "${ALLOW_PREEXISTING_INSTALLATION:-}" = "" ]; then
validate_starting_assumptions
fi
setup_report
if ! ui_confirm "Ready to continue?"; then
ok "Alright, no changes have been made :)"
contactme
trap finish_cleanup EXIT
exit 1
fi
create_build_group
create_build_users
create_directories
place_channel_configuration
install_from_extracted_nix
configure_shell_profile
set +eu
. /etc/profile
set -eu
setup_default_profile
place_nix_configuration
configure_nix_daemon_plist
trap finish_success EXIT
}
main
+41 -38
View File
@@ -1,4 +1,4 @@
#!/bin/sh
#! /usr/bin/env bash
set -e
@@ -7,13 +7,6 @@ self="$(dirname "$0")"
nix="@nix@"
cacert="@cacert@"
# macOS support for 10.10 or higher
if [[ "$(uname -s)" = "Darwin" && $(($(sw_vers -productVersion | cut -d '.' -f 2))) -lt 10 ]]; then
echo "$0: macOS $(sw_vers -productVersion) is not supported, upgrade to 10.10 or higher"
exit 1
fi
if ! [ -e "$self/.reginfo" ]; then
echo "$0: incomplete installer (.reginfo is missing)" >&2
exit 1
@@ -24,6 +17,23 @@ if [ -z "$USER" ]; then
exit 1
fi
if [ -z "$HOME" ]; then
echo "$0: \$HOME is not set" >&2
exit 1
fi
# macOS support for 10.10 or higher
if [[ "$(uname -s)" = "Darwin" ]]; then
if [[ $(($(sw_vers -productVersion | cut -d '.' -f 2))) -lt 10 ]]; then
echo "$0: macOS $(sw_vers -productVersion) is not supported, upgrade to 10.10 or higher"
exit 1
fi
printf '\e[1;31mSwitching to the Multi-User Darwin Installer\e[0m\n'
exec "$self/install-darwin-multi-user"
exit 0
fi
if [ "$(id -u)" -eq 0 ]; then
printf '\e[1;31mwarning: installing Nix as root is not supported by this script!\e[0m\n'
fi
@@ -32,7 +42,7 @@ echo "performing a single-user installation of Nix..." >&2
if ! [ -e $dest ]; then
cmd="mkdir -m 0755 $dest && chown $USER $dest"
echo "directory $dest does not exist; creating it by running '$cmd' using sudo" >&2
echo "directory $dest does not exist; creating it by running ‘$cmd’ using sudo" >&2
if ! sudo sh -c "$cmd"; then
echo "$0: please manually run ‘$cmd’ as root to create $dest" >&2
exit 1
@@ -40,32 +50,29 @@ if ! [ -e $dest ]; then
fi
if ! [ -w $dest ]; then
echo "$0: directory $dest exists, but is not writable by you. This could indicate that another user has already performed a single-user installation of Nix on this system. If you wish to enable multi-user support see http://nixos.org/nix/manual/#ssec-multi-user. If you wish to continue with a single-user install for $USER please run ‘chown -R $USER $dest’ as root." >&2
echo "$0: directory $dest exists, but is not writable by you; please run ‘chown -R $USER $dest’ as root" >&2
exit 1
fi
mkdir -p $dest/store
printf "copying Nix to %s..." "${dest}/store" >&2
echo -n "copying Nix to $dest/store..." >&2
for i in $(cd "$self/store" >/dev/null && echo ./*); do
printf "." >&2
echo -n "." >&2
i_tmp="$dest/store/$i.$$"
if [ -e "$i_tmp" ]; then
rm -rf "$i_tmp"
fi
if ! [ -e "$dest/store/$i" ]; then
cp -Rp "$self/store/$i" "$i_tmp"
chmod -R a-w "$i_tmp"
chmod +w "$i_tmp"
mv "$i_tmp" "$dest/store/$i"
chmod -w "$dest/store/$i"
fi
done
echo "" >&2
echo "initialising Nix database..." >&2
if ! $nix/bin/nix-store --init; then
if ! "$nix/bin/nix-store" --init; then
echo "$0: failed to initialize the Nix database" >&2
exit 1
fi
@@ -83,38 +90,34 @@ if ! "$nix/bin/nix-env" -i "$nix"; then
fi
# Install an SSL certificate bundle.
if [ -z "$NIX_SSL_CERT_FILE" ] || ! [ -f "$NIX_SSL_CERT_FILE" ]; then
$nix/bin/nix-env -i "$cacert"
if [ -z "$NIX_SSL_CERT_FILE" ] || [ ! -f "$NIX_SSL_CERT_FILE" ]; then
"$nix/bin/nix-env" -i "$cacert"
export NIX_SSL_CERT_FILE="$HOME/.nix-profile/etc/ssl/certs/ca-bundle.crt"
fi
# Subscribe the user to the Nixpkgs channel and fetch it.
if ! $nix/bin/nix-channel --list | grep -q "^nixpkgs "; then
$nix/bin/nix-channel --add https://nixos.org/channels/nixpkgs-unstable
if ! "$nix/bin/nix-channel" --list | grep -q "^nixpkgs "; then
"$nix/bin/nix-channel" --add https://nixos.org/channels/nixpkgs-unstable
fi
if [ -z "$_NIX_INSTALLER_TEST" ]; then
$nix/bin/nix-channel --update nixpkgs
"$nix/bin/nix-channel" --update nixpkgs
fi
# Make the shell source nix.sh during login.
p="$NIX_LINK/etc/profile.d/nix.sh"
added=
if [ -z "$NIX_INSTALLER_NO_MODIFY_PROFILE" ]; then
# Make the shell source nix.sh during login.
p=$HOME/.nix-profile/etc/profile.d/nix.sh
for i in .bash_profile .bash_login .profile; do
fn="$HOME/$i"
if [ -w "$fn" ]; then
if ! grep -q "$p" "$fn"; then
echo "modifying $fn..." >&2
echo "if [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn"
fi
added=1
break
for i in .bash_profile .bash_login .profile; do
fn="$HOME/$i"
if [ -w "$fn" ]; then
if ! grep -q "$p" "$fn"; then
echo "modifying $fn..." >&2
echo "if [ -e $p ]; then . $p; fi # added by Nix installer" >> "$fn"
fi
done
fi
added=1
break
fi
done
if [ -z "$added" ]; then
cat >&2 <<EOF
+29 -2
View File
@@ -1,12 +1,39 @@
nix_bin_scripts := \
$(d)/nix-build \
$(d)/nix-channel \
$(d)/nix-copy-closure \
$(d)/nix-generate-patches \
$(d)/nix-install-package \
$(d)/nix-pull \
$(d)/nix-push
bin-scripts += $(nix_bin_scripts)
nix_substituters := \
$(d)/copy-from-other-stores.pl \
$(d)/download-from-binary-cache.pl \
$(d)/download-using-manifests.pl
nix_noinst_scripts := \
$(d)/build-remote.pl \
$(d)/find-runtime-roots.pl \
$(d)/resolve-system-dependencies.pl \
$(d)/nix-http-export.cgi \
$(d)/nix-profile.sh \
$(d)/nix-reduce-build
$(d)/nix-profile-daemon.sh \
$(d)/nix-reduce-build \
$(nix_substituters)
noinst-scripts += $(nix_noinst_scripts)
profiledir = $(sysconfdir)/profile.d
$(eval $(call install-file-as, $(d)/nix-profile.sh, $(profiledir)/nix.sh, 0644))
$(eval $(call install-file-as, $(d)/nix-profile-daemon.sh, $(profiledir)/nix-daemon.sh, 0644))
$(eval $(call install-program-in, $(d)/find-runtime-roots.pl, $(libexecdir)/nix))
$(eval $(call install-program-in, $(d)/build-remote.pl, $(libexecdir)/nix))
$(eval $(call install-program-in, $(d)/resolve-system-dependencies.pl, $(libexecdir)/nix))
$(foreach prog, $(nix_substituters), $(eval $(call install-program-in, $(prog), $(libexecdir)/nix/substituters)))
$(eval $(call install-symlink, nix-build, $(bindir)/nix-shell))
clean-files += $(nix_noinst_scripts)
clean-files += $(nix_bin_scripts) $(nix_noinst_scripts)
+356
View File
@@ -0,0 +1,356 @@
#! @perl@ -w @perlFlags@
use utf8;
use strict;
use Nix::Config;
use Nix::Store;
use Nix::Utils;
use File::Basename;
use Text::ParseWords;
use Cwd;
binmode STDERR, ":encoding(utf8)";
Nix::Config::readConfig;
my $dryRun = 0;
my $verbose = 0;
my $runEnv = $0 =~ /nix-shell$/;
my $pure = 0;
my $fromArgs = 0;
my $packages = 0;
# Same condition as bash uses for interactive shells
my $interactive = -t STDIN && -t STDERR;
my @instArgs = ();
my @buildArgs = ();
my @exprs = ();
my $shell = $ENV{SHELL} || "/bin/sh";
my $envCommand = ""; # interactive shell
my @envExclude = ();
my $myName = $runEnv ? "nix-shell" : "nix-build";
my $inShebang = 0;
my $script;
my @savedArgs;
my $tmpDir = mkTempDir($myName);
my $outLink = "./result";
my $drvLink = "$tmpDir/derivation";
# Ensure that the $tmpDir is deleted.
$SIG{'INT'} = sub { exit 1 };
# Heuristic to see if we're invoked as a shebang script, namely, if we
# have a single argument, it's the name of an executable file, and it
# starts with "#!".
if ($runEnv && defined $ARGV[0] && $ARGV[0] !~ /nix-shell/) {
$script = $ARGV[0];
if (-f $script && -x $script) {
open SCRIPT, "<$script" or die "$0: cannot open ‘$script’: $!\n";
my $first = <SCRIPT>;
if ($first =~ /^\#\!/) {
$inShebang = 1;
@savedArgs = @ARGV; shift @savedArgs;
@ARGV = ();
while (<SCRIPT>) {
chomp;
if (/^\#\!\s*nix-shell (.*)$/) {
push @ARGV, shellwords($1);
}
}
}
close SCRIPT;
}
}
for (my $n = 0; $n < scalar @ARGV; $n++) {
my $arg = $ARGV[$n];
if ($arg eq "--help") {
exec "man $myName" or die;
}
elsif ($arg eq "--version") {
print "$myName (Nix) $Nix::Config::version\n";
exit 0;
}
elsif ($arg eq "--add-drv-link") {
$drvLink = "./derivation";
}
elsif ($arg eq "--no-out-link" || $arg eq "--no-link") {
$outLink = "$tmpDir/result";
}
elsif ($arg eq "--drv-link") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
$drvLink = $ARGV[$n];
}
elsif ($arg eq "--out-link" || $arg eq "-o") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
$outLink = $ARGV[$n];
}
elsif ($arg eq "--attr" || $arg eq "-A" || $arg eq "-I") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
push @instArgs, ($arg, $ARGV[$n]);
}
elsif ($arg eq "--arg" || $arg eq "--argstr") {
die "$0: ‘$arg’ requires two arguments\n" unless $n + 2 < scalar @ARGV;
push @instArgs, ($arg, $ARGV[$n + 1], $ARGV[$n + 2]);
$n += 2;
}
elsif ($arg eq "--log-type") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
push @instArgs, ($arg, $ARGV[$n]);
push @buildArgs, ($arg, $ARGV[$n]);
}
elsif ($arg eq "--option") {
die "$0: ‘$arg’ requires two arguments\n" unless $n + 2 < scalar @ARGV;
push @instArgs, ($arg, $ARGV[$n + 1], $ARGV[$n + 2]);
push @buildArgs, ($arg, $ARGV[$n + 1], $ARGV[$n + 2]);
$n += 2;
}
elsif ($arg eq "--max-jobs" || $arg eq "-j" || $arg eq "--max-silent-time" || $arg eq "--log-type" || $arg eq "--cores" || $arg eq "--timeout" || $arg eq '--add-root') {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
push @buildArgs, ($arg, $ARGV[$n]);
}
elsif ($arg eq "--dry-run") {
push @buildArgs, "--dry-run";
$dryRun = 1;
}
elsif ($arg eq "--show-trace") {
push @instArgs, $arg;
}
elsif ($arg eq "-") {
@exprs = ("-");
}
elsif ($arg eq "--verbose" || substr($arg, 0, 2) eq "-v") {
push @buildArgs, $arg;
push @instArgs, $arg;
$verbose = 1;
}
elsif ($arg eq "--quiet" || $arg eq "--repair") {
push @buildArgs, $arg;
push @instArgs, $arg;
}
elsif ($arg eq "--check") {
push @buildArgs, $arg;
}
elsif ($arg eq "--run-env") { # obsolete
$runEnv = 1;
}
elsif ($arg eq "--command" || $arg eq "--run") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
$envCommand = "$ARGV[$n]\nexit";
$interactive = 0 if $arg eq "--run";
}
elsif ($arg eq "--exclude") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
push @envExclude, $ARGV[$n];
}
elsif ($arg eq "--pure") { $pure = 1; }
elsif ($arg eq "--impure") { $pure = 0; }
elsif ($arg eq "--expr" || $arg eq "-E") {
$fromArgs = 1;
push @instArgs, "--expr";
}
elsif ($arg eq "--packages" || $arg eq "-p") {
$packages = 1;
}
elsif ($inShebang && $arg eq "-i") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
my $interpreter = $ARGV[$n];
# Überhack to support Perl. Perl examines the shebang and
# executes it unless it contains the string "perl" or "indir",
# or (undocumented) argv[0] does not contain "perl". Exploit
# the latter by doing "exec -a".
my $execArgs = $interpreter =~ /perl/ ? "-a PERL" : "";
sub shellEscape {
my $s = $_;
$s =~ s/'/'\\''/g;
return "'" . $s . "'";
}
$envCommand = "exec $execArgs $interpreter $script ${\(join ' ', (map shellEscape, @savedArgs))}";
}
elsif (substr($arg, 0, 1) eq "-") {
push @buildArgs, $arg;
}
elsif ($arg eq "-Q" || $arg eq "--no-build-output") {
push @buildArgs, $arg;
push @instArgs, $arg;
}
else {
push @exprs, $arg;
}
}
die "$0: ‘-p’ and ‘-E’ are mutually exclusive\n" if $packages && $fromArgs;
if ($packages) {
push @instArgs, "--expr";
@exprs = (
'with import <nixpkgs> { }; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ '
. (join " ", map { "($_)" } @exprs) . ']; } ""');
} elsif (!$fromArgs) {
@exprs = ("shell.nix") if scalar @exprs == 0 && $runEnv && -e "shell.nix";
@exprs = ("default.nix") if scalar @exprs == 0;
}
$ENV{'IN_NIX_SHELL'} = 1 if $runEnv;
foreach my $expr (@exprs) {
# Instantiate.
my @drvPaths;
if ($expr !~ /^\/.*\.drv$/) {
# If we're in a #! script, interpret filenames relative to the
# script.
$expr = dirname(Cwd::abs_path($script)) . "/" . $expr
if $inShebang && !$packages && $expr !~ /^\//;
# !!! would prefer the perl 5.8.0 pipe open feature here.
my $pid = open(DRVPATHS, "-|") || exec "$Nix::Config::binDir/nix-instantiate", "--add-root", $drvLink, "--indirect", @instArgs, $expr;
while (<DRVPATHS>) {chomp; push @drvPaths, $_;}
if (!close DRVPATHS) {
die "nix-instantiate killed by signal " . ($? & 127) . "\n" if ($? & 127);
exit 1;
}
} else {
push @drvPaths, $expr;
}
if ($runEnv) {
die "$0: a single derivation is required\n" if scalar @drvPaths != 1;
my $drvPath = $drvPaths[0];
$drvPath = (split '!',$drvPath)[0];
$drvPath = readlink $drvPath or die "cannot read symlink ‘$drvPath’" if -l $drvPath;
my $drv = derivationFromPath($drvPath);
# Build or fetch all dependencies of the derivation.
my @inputDrvs = grep { my $x = $_; (grep { $x =~ $_ } @envExclude) == 0 } @{$drv->{inputDrvs}};
system("$Nix::Config::binDir/nix-store", "-r", "--no-output", "--no-gc-warning", @buildArgs, @inputDrvs, @{$drv->{inputSrcs}}) == 0
or die "$0: failed to build all dependencies\n";
# Set the environment.
my $tmp = $ENV{"TMPDIR"} // $ENV{"XDG_RUNTIME_DIR"} // "/tmp";
if ($pure) {
foreach my $name (keys %ENV) {
next if grep { $_ eq $name } ("HOME", "USER", "LOGNAME", "DISPLAY", "PATH", "TERM", "IN_NIX_SHELL", "TZ", "PAGER", "NIX_BUILD_SHELL");
delete $ENV{$name};
}
# NixOS hack: prevent /etc/bashrc from sourcing /etc/profile.
$ENV{'__ETC_PROFILE_SOURCED'} = 1;
}
$ENV{'NIX_BUILD_TOP'} = $ENV{'TMPDIR'} = $ENV{'TEMPDIR'} = $ENV{'TMP'} = $ENV{'TEMP'} = $tmp;
$ENV{'NIX_STORE'} = $Nix::Config::storeDir;
if (defined $Nix::Config::config{"build-cores"}) {
$ENV{'NIX_BUILD_CORES'} = $Nix::Config::config{"build-cores"};
}
$ENV{$_} = $drv->{env}->{$_} foreach keys %{$drv->{env}};
# Run a shell using the derivation's environment. For
# convenience, source $stdenv/setup to setup additional
# environment variables and shell functions. Also don't lose
# the current $PATH directories.
my $rcfile = "$tmpDir/rc";
writeFile(
$rcfile,
"rm -rf '$tmpDir'; " .
'[ -n "$PS1" ] && [ -e ~/.bashrc ] && source ~/.bashrc; ' .
($pure ? '' : 'p=$PATH; ' ) .
'dontAddDisableDepTrack=1; ' .
'[ -e $stdenv/setup ] && source $stdenv/setup; ' .
($pure ? '' : 'PATH=$PATH:$p; unset p; ') .
'set +e; ' .
'[ -n "$PS1" ] && PS1="\n\[\033[1;32m\][nix-shell:\w]$\[\033[0m\] "; ' .
'if [ "$(type -t runHook)" = function ]; then runHook shellHook; fi; ' .
'unset NIX_ENFORCE_PURITY; ' .
'unset NIX_INDENT_MAKE; ' .
'shopt -u nullglob; ' .
'unset TZ; ' . (defined $ENV{'TZ'} ? "export TZ='${ENV{'TZ'}}'; " : '') .
$envCommand);
my @args = ($ENV{NIX_BUILD_SHELL} // "bash");
push @args, "--rcfile" if $interactive;
push @args, $rcfile;
exec @args;
die;
}
# Ugly hackery to make "nix-build -A foo.all" produce symlinks
# ./result, ./result-dev, and so on, rather than ./result,
# ./result-2-dev, and so on. This combines multiple derivation
# paths into one "/nix/store/drv-path!out1,out2,..." argument.
my $prevDrvPath = "";
my @drvPaths2;
foreach my $drvPath (@drvPaths) {
my $p = $drvPath; my $output = "out";
if ($drvPath =~ /(.*)!(.*)/) {
$p = $1; $output = $2;
} else {
$p = $drvPath;
}
my $target = readlink $p or die "cannot read symlink ‘$p’";
print STDERR "derivation is $target\n" if $verbose;
if ($target eq $prevDrvPath) {
push @drvPaths2, (pop @drvPaths2) . "," . $output;
} else {
push @drvPaths2, $target . "!" . $output;
$prevDrvPath = $target;
}
}
# Build.
my @outPaths;
my $pid = open(OUTPATHS, "-|") || exec "$Nix::Config::binDir/nix-store", "--add-root", $outLink, "--indirect", "-r",
@buildArgs, @drvPaths2;
while (<OUTPATHS>) {chomp; push @outPaths, $_;}
if (!close OUTPATHS) {
die "nix-store killed by signal " . ($? & 127) . "\n" if ($? & 127);
exit ($? >> 8 || 1);
}
next if $dryRun;
foreach my $outPath (@outPaths) {
my $target = readlink $outPath or die "cannot read symlink ‘$outPath’";
print "$target\n";
}
}
+250
View File
@@ -0,0 +1,250 @@
#! @perl@ -w @perlFlags@
use utf8;
use strict;
use File::Basename;
use File::Path qw(mkpath);
use Nix::Config;
use Nix::Manifest;
use File::Temp qw(tempdir);
binmode STDERR, ":encoding(utf8)";
Nix::Config::readConfig;
my $manifestDir = $Nix::Config::manifestDir;
# Turn on caching in nix-prefetch-url.
my $channelCache = "$Nix::Config::stateDir/channel-cache";
mkdir $channelCache, 0755 unless -e $channelCache;
$ENV{'NIX_DOWNLOAD_CACHE'} = $channelCache if -W $channelCache;
# Figure out the name of the `.nix-channels' file to use.
my $home = $ENV{"HOME"} or die '$HOME not set\n';
my $channelsList = "$home/.nix-channels";
my $nixDefExpr = "$home/.nix-defexpr";
# Figure out the name of the channels profile.
my $userName = getpwuid($<) || $ENV{"USER"} or die "cannot figure out user name";
my $profile = "$Nix::Config::stateDir/profiles/per-user/$userName/channels";
mkpath(dirname $profile, 0, 0755);
my %channels;
my $netrcFile = $Nix::Config::config{"netrc-file"} //
"$Nix::Config::confDir/netrc";
# Reads the list of channels.
sub readChannels {
return if (!-f $channelsList);
open CHANNELS, "<$channelsList" or die "cannot open ‘$channelsList’: $!";
while (<CHANNELS>) {
chomp;
next if /^\s*\#/;
my ($url, $name) = split ' ', $_;
$url =~ s/\/*$//; # remove trailing slashes
$name = basename $url unless defined $name;
$channels{$name} = $url;
}
close CHANNELS;
}
# Writes the list of channels.
sub writeChannels {
open CHANNELS, ">$channelsList" or die "cannot open ‘$channelsList’: $!";
foreach my $name (keys %channels) {
print CHANNELS "$channels{$name} $name\n";
}
close CHANNELS;
}
# Adds a channel.
sub addChannel {
my ($url, $name) = @_;
die "invalid channel URL ‘$url’" unless $url =~ /^(file|http|https):\/\//;
die "invalid channel identifier ‘$name’" unless $name =~ /^[a-zA-Z0-9_][a-zA-Z0-9_\-\.]*$/;
readChannels;
$channels{$name} = $url;
writeChannels;
}
# Remove a channel.
sub removeChannel {
my ($name) = @_;
readChannels;
my $url = $channels{$name};
deleteOldManifests($url . "/MANIFEST", undef) if defined $url;
delete $channels{$name};
writeChannels;
system("$Nix::Config::binDir/nix-env --profile '$profile' -e '$name'") == 0
or die "cannot remove channel ‘$name’\n";
}
# Fetch Nix expressions and pull manifests from the subscribed
# channels.
sub update {
my @channelNames = @_;
readChannels;
# Download each channel.
my $exprs = "";
foreach my $name (keys %channels) {
next if scalar @channelNames > 0 && ! grep { $_ eq $name } @{channelNames};
my $url = $channels{$name};
my $origUrl = "$url/MANIFEST";
# We want to download the url to a file to see if it's a tarball while also checking if we
# got redirected in the process, so that we can grab the various parts of a nix channel
# definition from a consistent location if the redirect changes mid-download.
my $tmpdir = tempdir( CLEANUP => 1 );
my $filename;
($url, $filename) = `cd $tmpdir && $Nix::Config::curl $Nix::Config::curlCaFlag --netrc-file $netrcFile --netrc-optional --silent --write-out '%{url_effective}\n%{filename_effective}' -L '$url' -O`;
chomp $url;
die "$0: unable to check ‘$url’\n" if $? != 0;
# If the URL contains a version number, append it to the name
# attribute (so that "nix-env -q" on the channels profile
# shows something useful).
my $cname = $name;
$cname .= $1 if basename($url) =~ /(-\d.*)$/;
my $path;
my $ret = -1;
if (-e "$tmpdir/$filename" && $filename =~ /\.tar\.(gz|bz2|xz)$/) {
# Get our temporary download into the store.
(my $hash, $path) = `PRINT_PATH=1 QUIET=1 $Nix::Config::binDir/nix-prefetch-url 'file://$tmpdir/$filename'`;
chomp $path;
# Try unpacking the expressions to see if they'll be valid for us to process later.
# Like anything in nix, this will cache the result so we don't do it again outside of the loop below.
$ret = system("$Nix::Config::binDir/nix-build --no-out-link -E 'import <nix/unpack-channel.nix> " .
"{ name = \"$cname\"; channelName = \"$name\"; src = builtins.storePath \"$path\"; }'");
}
# The URL doesn't unpack directly, so let's try treating it like a full channel folder with files in it
my $extraAttrs = "";
if ($ret != 0) {
# Check if the channel advertises a binary cache.
my $binaryCacheURL = `$Nix::Config::curl $Nix::Config::curlCaFlag --netrc-file $netrcFile --netrc-optional --silent '$url'/binary-cache-url`;
my $getManifest = ($Nix::Config::config{"force-manifest"} // "false") eq "true";
if ($? == 0 && $binaryCacheURL ne "") {
$extraAttrs .= "binaryCacheURL = \"$binaryCacheURL\"; ";
deleteOldManifests($origUrl, undef);
} else {
$getManifest = 1;
}
if ($getManifest) {
# No binary cache, so pull the channel manifest.
mkdir $manifestDir, 0755 unless -e $manifestDir;
die "$0: you do not have write permission to ‘$manifestDir’!\n" unless -W $manifestDir;
$ENV{'NIX_ORIG_URL'} = $origUrl;
system("$Nix::Config::binDir/nix-pull", "--skip-wrong-store", "$url/MANIFEST") == 0
or die "cannot pull manifest from ‘$url’\n";
}
# Download the channel tarball.
my $fullURL = "$url/nixexprs.tar.xz";
system("$Nix::Config::curl $Nix::Config::curlCaFlag --netrc-file $netrcFile --netrc-optional --fail --silent --head '$fullURL' > /dev/null") == 0 or
$fullURL = "$url/nixexprs.tar.bz2";
print STDERR "downloading Nix expressions from ‘$fullURL’...\n";
(my $hash, $path) = `PRINT_PATH=1 QUIET=1 $Nix::Config::binDir/nix-prefetch-url '$fullURL'`;
die "cannot fetch ‘$fullURL’\n" if $? != 0;
chomp $path;
}
# Regardless of where it came from, add the expression representing this channel to accumulated expression
$exprs .= "'f: f { name = \"$cname\"; channelName = \"$name\"; src = builtins.storePath \"$path\"; $extraAttrs }' ";
}
# Unpack the channel tarballs into the Nix store and install them
# into the channels profile.
print STDERR "unpacking channels...\n";
system("$Nix::Config::binDir/nix-env --profile '$profile' " .
"-f '<nix/unpack-channel.nix>' -i -E $exprs --quiet") == 0
or die "cannot unpack the channels";
# Make the channels appear in nix-env.
unlink $nixDefExpr if -l $nixDefExpr; # old-skool ~/.nix-defexpr
mkdir $nixDefExpr or die "cannot create directory ‘$nixDefExpr’" if !-e $nixDefExpr;
my $channelLink = "$nixDefExpr/channels";
unlink $channelLink; # !!! not atomic
symlink($profile, $channelLink) or die "cannot symlink ‘$channelLink’ to ‘$profile’";
}
die "$0: argument expected\n" if scalar @ARGV == 0;
while (scalar @ARGV) {
my $arg = shift @ARGV;
if ($arg eq "--add") {
die "$0: ‘--add’ requires one or two arguments\n" if scalar @ARGV < 1 || scalar @ARGV > 2;
my $url = shift @ARGV;
my $name = shift @ARGV;
unless (defined $name) {
$name = basename $url;
$name =~ s/-unstable//;
$name =~ s/-stable//;
}
addChannel($url, $name);
last;
}
if ($arg eq "--remove") {
die "$0: ‘--remove’ requires one argument\n" if scalar @ARGV != 1;
removeChannel(shift @ARGV);
last;
}
if ($arg eq "--list") {
die "$0: ‘--list’ requires one argument\n" if scalar @ARGV != 0;
readChannels;
foreach my $name (keys %channels) {
print "$name $channels{$name}\n";
}
last;
}
elsif ($arg eq "--update") {
update(@ARGV);
last;
}
elsif ($arg eq "--rollback") {
die "$0: ‘--rollback’ has at most one argument\n" if scalar @ARGV > 1;
my $generation = shift @ARGV;
my @args = ("$Nix::Config::binDir/nix-env", "--profile", $profile);
if (defined $generation) {
die "invalid channel generation number ‘$generation’" unless $generation =~ /^[0-9]+$/;
push @args, "--switch-generation", $generation;
} else {
push @args, "--rollback";
}
system(@args) == 0 or exit 1;
last;
}
elsif ($arg eq "--help") {
exec "man nix-channel" or die;
}
elsif ($arg eq "--version") {
print "nix-channel (Nix) $Nix::Config::version\n";
exit 0;
}
else {
die "unknown argument ‘$arg’; try ‘--help’\n";
}
}
+109
View File
@@ -0,0 +1,109 @@
#! @perl@ -w @perlFlags@
use utf8;
use strict;
use Nix::SSH;
use Nix::Config;
use Nix::Store;
use Nix::CopyClosure;
use List::Util qw(sum);
use IO::Handle;
binmode STDERR, ":encoding(utf8)";
STDERR->autoflush(1);
if (scalar @ARGV < 1) {
print STDERR <<EOF
Usage: nix-copy-closure [--from | --to] HOSTNAME [--sign] [--gzip] [--bzip2] [--xz] PATHS...
EOF
;
exit 1;
}
# Get the target host.
my $sshHost;
my $sign = 0;
my $toMode = 1;
my $includeOutputs = 0;
my $dryRun = 0;
my $useSubstitutes = 0;
my $verbosity = 1;
# !!! Copied from nix-pack-closure, should put this in a module.
my @storePaths = ();
while (@ARGV) {
my $arg = shift @ARGV;
if ($arg eq "--help") {
exec "man nix-copy-closure" or die;
}
elsif ($arg eq "--sign") {
$sign = 1;
}
elsif ($arg eq "--gzip" || $arg eq "--bzip2" || $arg eq "--xz") {
warn "$0: ‘$arg’ is not implemented\n" if $arg ne "--gzip";
push @globalSshOpts, "-C";
}
elsif ($arg eq "--from") {
$toMode = 0;
}
elsif ($arg eq "--to") {
$toMode = 1;
}
elsif ($arg eq "--include-outputs") {
$includeOutputs = 1;
}
elsif ($arg eq "--show-progress") {
warn "$0: ‘$arg’ is not implemented\n";
}
elsif ($arg eq "--dry-run") {
$dryRun = 1;
}
elsif ($arg eq "--use-substitutes" || $arg eq "-s") {
$useSubstitutes = 1;
}
elsif ($arg eq "-v") {
$verbosity++;
setVerbosity($verbosity);
}
elsif (!defined $sshHost) {
$sshHost = $arg;
}
else {
push @storePaths, $arg;
}
}
die "$0: you did not specify a host name\n" unless defined $sshHost;
if ($toMode) { # Copy TO the remote machine.
Nix::CopyClosure::copyTo(
$sshHost, [ @storePaths ],
$includeOutputs, $dryRun, $sign, $useSubstitutes);
}
else { # Copy FROM the remote machine.
my ($from, $to) = connectToRemoteNix($sshHost, []);
# Query the closure of the given store paths on the remote
# machine. Paths are assumed to be store paths; there is no
# resolution (following of symlinks).
syswrite($to, pack("L<x4L<x4", 7, $includeOutputs ? 1 : 0)) or die;
writeStrings(\@storePaths, $to);
my @missing = grep { !isValidPath($_) } readStrings($from);
# Export the store paths on the remote machine and import them locally.
if (scalar @missing > 0) {
print STDERR "copying ", scalar @missing, " missing paths from ‘$sshHost’...\n";
writeInt(5, $to); # == cmdExportPaths
writeInt($sign ? 1 : 0, $to);
writeStrings(\@missing, $to);
importPaths(fileno($from));
}
}
+51
View File
@@ -0,0 +1,51 @@
#! @perl@ -w @perlFlags@
use strict;
use Nix::Manifest;
use Nix::GeneratePatches;
use Nix::Utils;
if (scalar @ARGV != 5) {
print STDERR <<EOF;
Usage: nix-generate-patches NAR-DIR PATCH-DIR PATCH-URI OLD-MANIFEST NEW-MANIFEST
This command generates binary patches between NAR files listed in
OLD-MANIFEST and NEW-MANIFEST. The patches are written to the
directory PATCH-DIR, and the prefix PATCH-URI is used to generate URIs
for the patches. The patches are added to NEW-MANIFEST. All NARs are
required to exist in NAR-DIR. Patches are generated between
succeeding versions of packages with the same name.
EOF
exit 1;
}
my $narPath = $ARGV[0];
my $patchesPath = $ARGV[1];
my $patchesURL = $ARGV[2];
my $srcManifest = $ARGV[3];
my $dstManifest = $ARGV[4];
my (%srcNarFiles, %srcLocalPaths, %srcPatches);
readManifest $srcManifest, \%srcNarFiles, \%srcPatches;
my (%dstNarFiles, %dstLocalPaths, %dstPatches);
readManifest $dstManifest, \%dstNarFiles, \%dstPatches;
my $tmpDir = mkTempDir("nix-generate-patches");
generatePatches \%srcNarFiles, \%dstNarFiles, \%srcPatches, \%dstPatches,
$narPath, $patchesPath, $patchesURL, $tmpDir;
propagatePatches \%srcPatches, \%dstNarFiles, \%dstPatches;
# Optionally add all new patches to the manifest in $NIX_ALL_PATCHES.
my $allPatchesFile = $ENV{"NIX_ALL_PATCHES"};
if (defined $allPatchesFile) {
my (%dummy, %allPatches);
readManifest("$patchesPath/all-patches", \%dummy, \%allPatches)
if -f $allPatchesFile;
copyPatches \%dstPatches, \%allPatches;
writeManifest($allPatchesFile, {}, \%allPatches, 0);
}
writeManifest $dstManifest, \%dstNarFiles, \%dstPatches;
+143
View File
@@ -0,0 +1,143 @@
#! @perl@ -w @perlFlags@
use utf8;
use strict;
use Nix::Config;
use Nix::Utils;
binmode STDERR, ":encoding(utf8)";
# Parse the command line arguments.
my @args = @ARGV;
my $source;
my $fromURL = 0;
my @extraNixEnvArgs = ();
my $interactive = 1;
my $op = "--install";
while (scalar @args) {
my $arg = shift @args;
if ($arg eq "--help") {
exec "man nix-install-package" or die;
}
elsif ($arg eq "--url") {
$fromURL = 1;
}
elsif ($arg eq "--profile" || $arg eq "-p") {
my $profile = shift @args;
die "$0: ‘--profile’ requires an argument\n" if !defined $profile;
push @extraNixEnvArgs, "-p", $profile;
}
elsif ($arg eq "--set") {
$op = "--set";
}
elsif ($arg eq "--non-interactive") {
$interactive = 0;
}
else {
$source = $arg;
}
}
die "$0: please specify a .nixpkg file or URL\n" unless defined $source;
# Re-execute in a terminal, if necessary, so that if we're executed
# from a web browser, the user gets to see us.
if ($interactive && !defined $ENV{"NIX_HAVE_TERMINAL"}) {
$ENV{"NIX_HAVE_TERMINAL"} = "1";
$ENV{"LD_LIBRARY_PATH"} = "";
foreach my $term ("xterm", "konsole", "gnome-terminal", "xterm") {
exec($term, "-e", "$Nix::Config::binDir/nix-install-package", @ARGV);
}
die "cannot execute ‘xterm’";
}
my $tmpDir = mkTempDir("nix-install-package");
sub barf {
my $msg = shift;
print "\nInstallation failed: $msg\n";
<STDIN> if $interactive;
exit 1;
}
# Download the package description, if necessary.
my $pkgFile = $source;
if ($fromURL) {
$pkgFile = "$tmpDir/tmp.nixpkg";
system("@curl@", "-L", "--silent", $source, "-o", $pkgFile) == 0
or barf "curl failed: $?";
}
# Read and parse the package file.
open PKGFILE, "<$pkgFile" or barf "cannot open ‘$pkgFile’: $!";
my $contents = <PKGFILE>;
close PKGFILE;
my $nameRE = "(?: [A-Za-z0-9\+\-\.\_\?\=]+ )"; # see checkStoreName()
my $systemRE = "(?: [A-Za-z0-9\+\-\_]+ )";
my $pathRE = "(?: \/ [\/A-Za-z0-9\+\-\.\_\?\=]* )";
# Note: $pathRE doesn't check that whether we're looking at a valid
# store path. We'll let nix-env do that.
$contents =~
/ ^ \s* (\S+) \s+ ($Nix::Utils::urlRE) \s+ ($nameRE) \s+ ($systemRE) \s+ ($pathRE) \s+ ($pathRE) ( \s+ ($Nix::Utils::urlRE) )? /x
or barf "invalid package contents";
my $version = $1;
my $manifestURL = $2;
my $drvName = $3;
my $system = $4;
my $drvPath = $5;
my $outPath = $6;
my $binaryCacheURL = $8;
barf "invalid package version ‘$version’" unless $version eq "NIXPKG1";
if ($interactive) {
# Ask confirmation.
print "Do you want to install ‘$drvName’ (Y/N)? ";
my $reply = <STDIN>;
chomp $reply;
exit if $reply ne "y" && $reply ne "Y";
}
if (defined $binaryCacheURL) {
push @extraNixEnvArgs, "--option", "extra-binary-caches", $binaryCacheURL;
} else {
# Store the manifest in the temporary directory so that we don't
# pollute /nix/var/nix/manifests. This also requires that we
# don't use the Nix daemon (because otherwise
# download-using-manifests won't see our NIX_MANIFESTS_DIRS
# environment variable).
$ENV{NIX_MANIFESTS_DIR} = $tmpDir;
$ENV{NIX_REMOTE} = "";
print "\nPulling manifests...\n";
system("$Nix::Config::binDir/nix-pull", $manifestURL) == 0
or barf "nix-pull failed: $?";
}
print "\nInstalling package...\n";
system("$Nix::Config::binDir/nix-env", $op, $outPath, "--force-name", $drvName, @extraNixEnvArgs) == 0
or barf "nix-env failed: $?";
if ($interactive) {
print "\nInstallation succeeded! Press Enter to continue.\n";
<STDIN>;
}
+54
View File
@@ -0,0 +1,54 @@
# Only execute this file once per shell.
if [ -n "$__ETC_PROFILE_NIX_SOURCED" ]; then return; fi
__ETC_PROFILE_NIX_SOURCED=1
# Set up secure multi-user builds: non-root users build through the
# Nix daemon.
if [ "$USER" != root -o ! -w @localstatedir@/nix/db ]; then
export NIX_REMOTE=daemon
fi
export NIX_USER_PROFILE_DIR="@localstatedir@/nix/profiles/per-user/$USER"
export NIX_PROFILES="@localstatedir@/nix/profiles/default $HOME/.nix-profile"
# Set up the per-user profile.
mkdir -m 0755 -p $NIX_USER_PROFILE_DIR
if ! test -O "$NIX_USER_PROFILE_DIR"; then
echo "WARNING: bad ownership on $NIX_USER_PROFILE_DIR" >&2
fi
if test -w $HOME; then
if ! test -L $HOME/.nix-profile; then
if test "$USER" != root; then
ln -s $NIX_USER_PROFILE_DIR/profile $HOME/.nix-profile
else
# Root installs in the system-wide profile by default.
ln -s @localstatedir@/nix/profiles/default $HOME/.nix-profile
fi
fi
# Subscribe the root user to the NixOS channel by default.
if [ "$USER" = root -a ! -e $HOME/.nix-channels ]; then
echo "https://nixos.org/channels/nixpkgs-unstable nixpkgs" > $HOME/.nix-channels
fi
# Create the per-user garbage collector roots directory.
NIX_USER_GCROOTS_DIR=@localstatedir@/nix/gcroots/per-user/$USER
mkdir -m 0755 -p $NIX_USER_GCROOTS_DIR
if ! test -O "$NIX_USER_GCROOTS_DIR"; then
echo "WARNING: bad ownership on $NIX_USER_GCROOTS_DIR" >&2
fi
# Set up a default Nix expression from which to install stuff.
if [ ! -e $HOME/.nix-defexpr -o -L $HOME/.nix-defexpr ]; then
rm -f $HOME/.nix-defexpr
mkdir -p $HOME/.nix-defexpr
if [ "$USER" != root ]; then
ln -s @localstatedir@/nix/profiles/per-user/root/channels $HOME/.nix-defexpr/channels_root
fi
fi
fi
export NIX_SSL_CERT_FILE="@localstatedir@/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
export NIX_PATH="@localstatedir@/nix/profiles/per-user/root/channels"
export PATH="$HOME/.nix-profile/bin:$HOME/.nix-profile/sbin:$HOME/.nix-profile/lib/kde4/libexec:@localstatedir@/nix/profiles/default/bin:@localstatedir@/nix/profiles/default/sbin:@localstatedir@/nix/profiles/default/lib/kde4/libexec:$PATH"
+12 -65
View File
@@ -1,70 +1,24 @@
if [ -n "$HOME" ] && [ -n "$USER" ]; then
__savedpath="$PATH"
export PATH=@coreutils@
if [ -n "$HOME" ]; then
NIX_LINK="$HOME/.nix-profile"
# Set up the per-user profile.
# This part should be kept in sync with nixpkgs:nixos/modules/programs/shell.nix
NIX_LINK=$HOME/.nix-profile
NIX_USER_PROFILE_DIR=@localstatedir@/nix/profiles/per-user/$USER
mkdir -m 0755 -p "$NIX_USER_PROFILE_DIR"
if [ "$(stat --printf '%u' "$NIX_USER_PROFILE_DIR")" != "$(id -u)" ]; then
echo "Nix: WARNING: bad ownership on "$NIX_USER_PROFILE_DIR", should be $(id -u)" >&2
# Set the default profile.
if ! [ -L "$NIX_LINK" ]; then
echo "creating $NIX_LINK" >&2
_NIX_DEF_LINK=@localstatedir@/nix/profiles/default
@coreutils@/ln -s "$_NIX_DEF_LINK" "$NIX_LINK"
fi
if [ -w "$HOME" ]; then
if ! [ -L "$NIX_LINK" ]; then
echo "Nix: creating $NIX_LINK" >&2
if [ "$USER" != root ]; then
if ! ln -s "$NIX_USER_PROFILE_DIR"/profile "$NIX_LINK"; then
echo "Nix: WARNING: could not create $NIX_LINK -> $NIX_USER_PROFILE_DIR/profile" >&2
fi
else
# Root installs in the system-wide profile by default.
ln -s @localstatedir@/nix/profiles/default "$NIX_LINK"
fi
fi
export PATH=$NIX_LINK/bin:$NIX_LINK/sbin:$PATH
# Subscribe the user to the unstable Nixpkgs channel by default.
if [ ! -e "$HOME/.nix-channels" ]; then
echo "https://nixos.org/channels/nixpkgs-unstable nixpkgs" > "$HOME/.nix-channels"
fi
# Create the per-user garbage collector roots directory.
__user_gcroots=@localstatedir@/nix/gcroots/per-user/"$USER"
mkdir -m 0755 -p "$__user_gcroots"
if [ "$(stat --printf '%u' "$__user_gcroots")" != "$(id -u)" ]; then
echo "Nix: WARNING: bad ownership on $__user_gcroots, should be $(id -u)" >&2
fi
unset __user_gcroots
# Set up a default Nix expression from which to install stuff.
__nix_defexpr="$HOME"/.nix-defexpr
[ -L "$__nix_defexpr" ] && rm -f "$__nix_defexpr"
mkdir -m 0755 -p "$__nix_defexpr"
if [ "$USER" != root ] && [ ! -L "$__nix_defexpr"/channels_root ]; then
ln -s @localstatedir@/nix/profiles/per-user/root/channels "$__nix_defexpr"/channels_root
fi
unset __nix_defexpr
# Subscribe the user to the Nixpkgs channel by default.
if [ ! -e $HOME/.nix-channels ]; then
echo "https://nixos.org/channels/nixpkgs-unstable nixpkgs" > $HOME/.nix-channels
fi
# Append ~/.nix-defexpr/channels/nixpkgs to $NIX_PATH so that
# <nixpkgs> paths work when the user has fetched the Nixpkgs
# channel.
export NIX_PATH="${NIX_PATH:+$NIX_PATH:}nixpkgs=$HOME/.nix-defexpr/channels/nixpkgs"
# Set up environment.
# This part should be kept in sync with nixpkgs:nixos/modules/programs/environment.nix
NIX_PROFILES="@localstatedir@/nix/profiles/default $NIX_USER_PROFILE_DIR"
for i in $NIX_PROFILES; do
if [ -d "$i/lib/aspell" ]; then
export ASPELL_CONF="dict-dir $i/lib/aspell"
fi
done
export NIX_PATH=${NIX_PATH:+$NIX_PATH:}nixpkgs=$HOME/.nix-defexpr/channels/nixpkgs
# Set $NIX_SSL_CERT_FILE so that Nixpkgs applications like curl work.
if [ -e /etc/ssl/certs/ca-certificates.crt ]; then # NixOS, Ubuntu, Debian, Gentoo, Arch
@@ -80,11 +34,4 @@ if [ -n "$HOME" ] && [ -n "$USER" ]; then
elif [ -e "$NIX_LINK/etc/ca-bundle.crt" ]; then # old cacert in Nix profile
export NIX_SSL_CERT_FILE="$NIX_LINK/etc/ca-bundle.crt"
fi
if [ -n ${MANPATH} ]; then
export MANPATH="$NIX_LINK/share/man:$MANPATH"
fi
export PATH="$NIX_LINK/bin:$__savedpath"
unset __savedpath NIX_LINK NIX_USER_PROFILE_DIR NIX_PROFILES
fi
+103
View File
@@ -0,0 +1,103 @@
#! @perl@ -w @perlFlags@
use utf8;
use strict;
use Nix::Config;
use Nix::Manifest;
binmode STDERR, ":encoding(utf8)";
my $manifestDir = $Nix::Config::manifestDir;
my $netrcFile = $Nix::Config::config{"netrc-file"} //
"$Nix::Config::confDir/netrc";
# Prevent access problems in shared-stored installations.
umask 0022;
# Create the manifests directory if it doesn't exist.
if (! -e $manifestDir) {
mkdir $manifestDir, 0755 or die "cannot create directory ‘$manifestDir’";
}
# Make sure that the manifests directory is scanned for GC roots.
my $gcRootsDir = "$Nix::Config::stateDir/gcroots";
my $manifestDirLink = "$gcRootsDir/manifests";
if (! -l $manifestDirLink) {
symlink($manifestDir, $manifestDirLink) or die "cannot create symlink ‘$manifestDirLink’";
}
# Process the URLs specified on the command line.
sub downloadFile {
my $url = shift;
$ENV{"PRINT_PATH"} = 1;
$ENV{"QUIET"} = 1;
my ($dummy, $path) = `$Nix::Config::binDir/nix-prefetch-url '$url'`;
die "cannot fetch ‘$url’" if $? != 0;
die "nix-prefetch-url did not return a path" unless defined $path;
chomp $path;
return $path;
}
sub processURL {
my $url = shift;
$url =~ s/\/$//;
my $manifest;
my $origUrl = $ENV{'NIX_ORIG_URL'} || $url;
# First see if a bzipped manifest is available.
if (system("$Nix::Config::curl $Nix::Config::curlCaFlag --netrc-file $netrcFile --netrc-optional --fail --silent --location --head '$url'.bz2 > /dev/null") == 0) {
print "fetching list of Nix archives at ‘$url.bz2’...\n";
$manifest = downloadFile "$url.bz2";
}
# Otherwise, just get the uncompressed manifest.
else {
print "fetching list of Nix archives at ‘$url’...\n";
$manifest = downloadFile $url;
}
my $baseName = "unnamed";
if ($url =~ /\/([^\/]+)\/[^\/]+$/) { # get the forelast component
$baseName = $1;
}
my $hash = `$Nix::Config::binDir/nix-hash --flat '$manifest'`
or die "cannot hash ‘$manifest’";
chomp $hash;
my $urlFile = "$manifestDir/$baseName-$hash.url";
open URL, ">$urlFile" or die "cannot create ‘$urlFile’";
print URL $origUrl;
close URL;
my $finalPath = "$manifestDir/$baseName-$hash.nixmanifest";
unlink $finalPath if -e $finalPath;
symlink("$manifest", "$finalPath")
or die "cannot link ‘$finalPath’ to ‘$manifest’";
deleteOldManifests($origUrl, $urlFile);
}
while (@ARGV) {
my $url = shift @ARGV;
if ($url eq "--help") {
exec "man nix-pull" or die;
} elsif ($url eq "--skip-wrong-store") {
# No-op, no longer supported.
} else {
processURL $url;
}
}
# Update the cache.
updateManifestDB();
+299
View File
@@ -0,0 +1,299 @@
#! @perl@ -w @perlFlags@
use utf8;
use strict;
use File::Basename;
use File::Path qw(mkpath);
use File::stat;
use File::Copy;
use MIME::Base64;
use Nix::Config;
use Nix::Store;
use Nix::Manifest;
use Nix::Utils;
binmode STDERR, ":encoding(utf8)";
my $tmpDir = mkTempDir("nix-push");
my $nixExpr = "$tmpDir/create-nars.nix";
# Parse the command line.
my $compressionType = "xz";
my $force = 0;
my $destDir;
my $writeManifest = 0;
my $manifestPath;
my $archivesURL;
my $link = 0;
my $secretKeyFile;
my @roots;
my @buildArgs;
for (my $n = 0; $n < scalar @ARGV; $n++) {
my $arg = $ARGV[$n];
if ($arg eq "--help") {
exec "man nix-push" or die;
} elsif ($arg eq "--bzip2") {
$compressionType = "bzip2";
} elsif ($arg eq "--none") {
$compressionType = "none";
} elsif ($arg eq "--force") {
$force = 1;
} elsif ($arg eq "--dest") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
$destDir = $ARGV[$n];
mkpath($destDir, 0, 0755);
} elsif ($arg eq "--manifest") {
$writeManifest = 1;
} elsif ($arg eq "--manifest-path") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
$manifestPath = $ARGV[$n];
$writeManifest = 1;
mkpath(dirname($manifestPath), 0, 0755);
} elsif ($arg eq "--url-prefix") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
$archivesURL = $ARGV[$n];
} elsif ($arg eq "--link") {
$link = 1;
} elsif ($arg eq "--key-file") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
$secretKeyFile = $ARGV[$n];
} elsif ($arg eq "--max-jobs" || $arg eq "-j") {
$n++;
die "$0: ‘$arg’ requires an argument\n" unless $n < scalar @ARGV;
push @buildArgs, ($arg, $ARGV[$n]);
} elsif (substr($arg, 0, 1) eq "-") {
die "$0: unknown flag ‘$arg’\n";
} else {
push @roots, $arg;
}
}
die "$0: please specify a destination directory\n" if !defined $destDir;
$archivesURL = "file://$destDir" unless defined $archivesURL;
# From the given store paths, determine the set of requisite store
# paths, i.e, the paths required to realise them.
my %storePaths;
foreach my $path (@roots) {
# Get all paths referenced by the normalisation of the given
# Nix expression.
my $pid = open(READ,
"$Nix::Config::binDir/nix-store --query --requisites --force-realise " .
"--include-outputs '$path'|") or die;
while (<READ>) {
chomp;
die "bad: $_" unless /^\//;
$storePaths{$_} = "";
}
close READ or die "nix-store failed: $?";
}
my @storePaths = keys %storePaths;
# Don't create archives for files that are already in the binary cache.
my @storePaths2;
my %narFiles;
foreach my $storePath (@storePaths) {
my $pathHash = substr(basename($storePath), 0, 32);
my $narInfoFile = "$destDir/$pathHash.narinfo";
if (!$force && -e $narInfoFile) {
my $narInfo = parseNARInfo($storePath, readFile($narInfoFile), 0, $narInfoFile) or die "cannot read ‘$narInfoFile’\n";
my $narFile = "$destDir/$narInfo->{url}";
if (-e $narFile) {
print STDERR "skipping existing $storePath\n";
# Add the NAR info to $narFiles if we're writing a
# manifest.
$narFiles{$storePath} = [
{ url => ("$archivesURL/" . basename $narInfo->{url})
, hash => $narInfo->{fileHash}
, size => $narInfo->{fileSize}
, compressionType => $narInfo->{compression}
, narHash => $narInfo->{narHash}
, narSize => $narInfo->{narSize}
, references => join(" ", map { "$Nix::Config::storeDir/$_" } @{$narInfo->{refs}})
, deriver => $narInfo->{deriver} ? "$Nix::Config::storeDir/$narInfo->{deriver}" : undef
}
] if $writeManifest;
next;
}
}
push @storePaths2, $storePath;
}
# Create a list of Nix derivations that turn each path into a Nix
# archive.
open NIX, ">$nixExpr";
print NIX "[";
foreach my $storePath (@storePaths2) {
die unless ($storePath =~ /\/[0-9a-z]{32}[^\"\\\$]*$/);
# Construct a Nix expression that creates a Nix archive.
my $nixexpr =
"(import <nix/nar.nix> " .
"{ storePath = builtins.storePath \"$storePath\"; hashAlgo = \"sha256\"; compressionType = \"$compressionType\"; }) ";
print NIX $nixexpr;
}
print NIX "]";
close NIX;
# Build the Nix expression.
print STDERR "building compressed archives...\n";
my @narPaths;
my $pid = open(READ, "-|", "$Nix::Config::binDir/nix-build", $nixExpr, "-o", "$tmpDir/result", @buildArgs)
or die "cannot run nix-build";
while (<READ>) {
chomp;
die unless /^\//;
push @narPaths, $_;
}
close READ or die "nix-build failed: $?";
# Write the cache info file.
my $cacheInfoFile = "$destDir/nix-cache-info";
if (! -e $cacheInfoFile) {
open FILE, ">$cacheInfoFile" or die "cannot create $cacheInfoFile: $!";
print FILE "StoreDir: $Nix::Config::storeDir\n";
print FILE "WantMassQuery: 0\n"; # by default, don't hit this cache for "nix-env -qas"
close FILE;
}
# Copy the archives and the corresponding NAR info files.
print STDERR "copying archives...\n";
my $totalNarSize = 0;
my $totalCompressedSize = 0;
for (my $n = 0; $n < scalar @storePaths2; $n++) {
my $storePath = $storePaths2[$n];
my $narDir = $narPaths[$n];
my $baseName = basename $storePath;
# Get info about the store path.
my ($deriver, $narHash, $time, $narSize, $refs) = queryPathInfo($storePath, 1);
# In some exceptional cases (such as VM tests that use the Nix
# store of the host), the database doesn't contain the hash. So
# compute it.
if ($narHash =~ /^sha256:0*$/) {
my $nar = "$tmpDir/nar";
system("$Nix::Config::binDir/nix-store --dump $storePath > $nar") == 0
or die "cannot dump $storePath\n";
$narHash = `$Nix::Config::binDir/nix-hash --type sha256 --base32 --flat $nar`;
die "cannot hash ‘$nar’" if $? != 0;
chomp $narHash;
$narHash = "sha256:$narHash";
$narSize = stat("$nar")->size;
unlink $nar or die;
}
$totalNarSize += $narSize;
# Get info about the compressed NAR.
open HASH, "$narDir/nar-compressed-hash" or die "cannot open nar-compressed-hash";
my $compressedHash = <HASH>;
chomp $compressedHash;
$compressedHash =~ /^[0-9a-z]+$/ or die "invalid hash";
close HASH;
my $narName = "$compressedHash.nar" . ($compressionType eq "xz" ? ".xz" : $compressionType eq "bzip2" ? ".bz2" : "");
my $narFile = "$narDir/$narName";
(-f $narFile) or die "NAR file for $storePath not found";
my $compressedSize = stat($narFile)->size;
$totalCompressedSize += $compressedSize;
printf STDERR "%s [%.2f MiB, %.1f%%]\n", $storePath,
$compressedSize / (1024 * 1024), $compressedSize / $narSize * 100;
# Copy the compressed NAR.
my $dst = "$destDir/$narName";
if (! -f $dst) {
my $tmp = "$destDir/.tmp.$$.$narName";
if ($link) {
link($narFile, $tmp) or die "cannot link $tmp to $narFile: $!\n";
} else {
copy($narFile, $tmp) or die "cannot copy $narFile to $tmp: $!\n";
}
rename($tmp, $dst) or die "cannot rename $tmp to $dst: $!\n";
}
# Write the info file.
my $info;
$info .= "StorePath: $storePath\n";
$info .= "URL: $narName\n";
$info .= "Compression: $compressionType\n";
$info .= "FileHash: sha256:$compressedHash\n";
$info .= "FileSize: $compressedSize\n";
$info .= "NarHash: $narHash\n";
$info .= "NarSize: $narSize\n";
$info .= "References: " . join(" ", map { basename $_ } @{$refs}) . "\n";
if (defined $deriver) {
$info .= "Deriver: " . basename $deriver . "\n";
if (isValidPath($deriver)) {
my $drv = derivationFromPath($deriver);
$info .= "System: $drv->{platform}\n";
}
}
if (defined $secretKeyFile) {
my $s = readFile $secretKeyFile;
chomp $s;
my ($keyName, $secretKey) = split ":", $s;
die "invalid secret key file ‘$secretKeyFile’\n" unless defined $keyName && defined $secretKey;
my $fingerprint = fingerprintPath($storePath, $narHash, $narSize, $refs);
my $sig = encode_base64(signString(decode_base64($secretKey), $fingerprint), "");
$info .= "Sig: $keyName:$sig\n";
}
my $pathHash = substr(basename($storePath), 0, 32);
$dst = "$destDir/$pathHash.narinfo";
if ($force || ! -f $dst) {
my $tmp = "$destDir/.tmp.$$.$pathHash.narinfo";
open INFO, ">$tmp" or die;
print INFO "$info" or die;
close INFO or die;
rename($tmp, $dst) or die "cannot rename $tmp to $dst: $!\n";
}
$narFiles{$storePath} = [
{ url => "$archivesURL/$narName"
, hash => "sha256:$compressedHash"
, size => $compressedSize
, compressionType => $compressionType
, narHash => "$narHash"
, narSize => $narSize
, references => join(" ", @{$refs})
, deriver => $deriver
}
] if $writeManifest;
}
printf STDERR "total compressed size %.2f MiB, %.1f%%\n",
$totalCompressedSize / (1024 * 1024), $totalCompressedSize / ($totalNarSize || 1) * 100;
# Optionally write a manifest.
writeManifest($manifestPath // "$destDir/MANIFEST", \%narFiles, \()) if $writeManifest;
+122
View File
@@ -0,0 +1,122 @@
#! @perl@ -w @perlFlags@
use utf8;
use strict;
use warnings;
use Cwd qw(realpath);
use Errno;
use File::Basename qw(dirname);
use File::Path qw(make_path);
use File::Spec::Functions qw(catfile);
use List::Util qw(reduce);
use IPC::Open3;
use Nix::Config;
use Nix::Store qw(derivationFromPath);
use POSIX qw(uname);
use Storable qw(lock_retrieve lock_store);
my ($sysname, undef, $version, undef, $machine) = uname;
$sysname =~ /Darwin/ or die "This tool is only meant to be used on Darwin systems.";
my $cache = "$Nix::Config::stateDir/dependency-maps/$machine-$sysname-$version.map";
make_path dirname($cache);
our $DEPS;
eval {
$DEPS = lock_retrieve($cache);
};
if($!{ENOENT}) {
lock_store {}, $cache;
$DEPS = {};
} elsif($@) {
die "Unable to obtain a lock on dependency-map file $cache: $@";
}
sub mkset(@) {
my %set;
@set{@_} = ();
\%set
}
sub union($$) {
my ($set1, $set2) = @_;
my %new = (%$set1, %$set2);
\%new
}
sub cache_filepath($) {
my $fp = shift;
$fp =~ s/-/--/g;
$fp =~ s/\//-/g;
$fp =~ s/^-//g;
catfile $cache, $fp
}
sub resolve_tree {
sub resolve_tree_inner {
my ($lib, $TREE) = @_;
return if (defined $TREE->{$lib});
$TREE->{$lib} = mkset(@{cache_get($lib)});
foreach my $dep (keys %{$TREE->{$lib}}) {
resolve_tree_inner($dep, $TREE);
}
values %$TREE
}
reduce { union($a, $b) } {}, resolve_tree_inner(@_)
}
sub cache_get {
my $key = shift;
if (defined $DEPS->{$key}) {
$DEPS->{$key}
} else {
cache_insert($key);
cache_get($key)
}
}
sub cache_insert($) {
my $key = shift;
print STDERR "Finding dependencies for $key...\n";
my @deps = find_deps($key);
$DEPS->{$key} = \@deps;
}
sub find_deps($) {
my $lib = shift;
my($chld_in, $chld_out, $chld_err);
my $pid = open3($chld_in, $chld_out, $chld_err, "@otool@", "-L", "-arch", "x86_64", $lib);
waitpid($pid, 0);
my $line = readline $chld_out;
if($? == 0 and $line !~ /not an object file/) {
my @libs;
while(<$chld_out>) {
my $dep = (split /\s+/)[1];
push @libs, $dep unless $dep eq $lib or $dep =~ /\@rpath/;
}
@libs
} elsif (-l $lib) {
(realpath($lib))
} else {
()
}
}
if (defined $ARGV[0]) {
my $deps = derivationFromPath($ARGV[0])->{"env"}->{"__impureHostDeps"};
if (defined $deps) {
my @files = split(/\s+/, $deps);
my $depcache = {};
my $depset = reduce { union($a, $b) } (map { resolve_tree($_, $depcache) } @files);
print "extra-chroot-dirs\n";
print join("\n", keys %$depset);
print "\n";
}
lock_store($DEPS, $cache);
} else {
print STDERR "Usage: $0 path/to/derivation.drv\n";
exit 1
}
+73
View File
@@ -0,0 +1,73 @@
#! /usr/bin/perl -w
if (scalar @ARGV != 1) {
print "syntax: show-duplication.pl PATH\n";
exit 1;
}
my $root = $ARGV[0];
my $nameRE = "(?:(?:[A-Za-z0-9\+\_]|(?:-[^0-9]))+)";
my $versionRE = "(?:[A-Za-z0-9\.\-]+)";
my %pkgInstances;
my $pid = open(PATHS, "-|") || exec "nix-store", "-qR", $root;
while (<PATHS>) {
chomp;
/^.*\/[0-9a-z]*-(.*)$/;
my $nameVersion = $1;
$nameVersion =~ /^($nameRE)(-($versionRE))?$/;
$name = $1;
$version = $3;
$version = "(unnumbered)" unless defined $version;
# print "$nameVersion $name $version\n";
push @{$pkgInstances{$name}}, {version => $version, path => $_};
}
close PATHS or exit 1;
sub pathSize {
my $path = shift;
my @st = lstat $path or die;
my $size = $st[7];
if (-d $path) {
opendir DIR, $path or die;
foreach my $name (readdir DIR) {
next if $name eq "." || $name eq "..";
$size += pathSize("$path/$name");
}
}
return $size;
}
my $totalPaths = 0;
my $totalSize = 0, $totalWaste = 0;
foreach my $name (sort {scalar @{$pkgInstances{$b}} <=> scalar @{$pkgInstances{$a}}} (keys %pkgInstances)) {
print "$name ", scalar @{$pkgInstances{$name}}, "\n";
my $allSize = 0;
foreach my $x (sort {$a->{version} cmp $b->{version}} @{$pkgInstances{$name}}) {
$totalPaths++;
my $size = pathSize $x->{path};
$allSize += $size;
print " $x->{version} $size\n";
}
my $avgSize = int($allSize / scalar @{$pkgInstances{$name}});
my $waste = $allSize - $avgSize;
$totalSize += $allSize;
$totalWaste += $waste;
print " average $avgSize, waste $waste\n";
}
my $avgDupl = $totalPaths / scalar (keys %pkgInstances);
my $wasteFactor = ($totalWaste / $totalSize) * 100;
print "average package duplication $avgDupl, total size $totalSize, total waste $totalWaste, $wasteFactor% wasted\n";
-41
View File
@@ -1,41 +0,0 @@
{ useClang ? false }:
with import <nixpkgs> {};
with import ./release-common.nix { inherit pkgs; };
(if useClang then clangStdenv else stdenv).mkDerivation {
name = "nix";
buildInputs =
[ curl bison flex libxml2 libxslt
bzip2 xz brotli
pkgconfig sqlite libsodium boehmgc
docbook5 docbook5_xsl
autoconf-archive
(aws-sdk-cpp.override {
apis = ["s3"];
customMemoryManagement = false;
})
autoreconfHook
# For nix-perl
perl
perlPackages.DBDSQLite
]
++ lib.optional stdenv.isLinux libseccomp;
inherit configureFlags;
enableParallelBuilding = true;
installFlags = "sysconfdir=$(out)/etc";
shellHook =
''
export prefix=$(pwd)/inst
configureFlags+=" --prefix=$prefix"
PKG_CONFIG_PATH=$prefix/lib/pkgconfig:$PKG_CONFIG_PATH
PATH=$prefix/bin:$PATH
'';
}
+5 -5
View File
@@ -33,7 +33,7 @@ namespace io {
class format_error : public std::exception
{
public:
format_error() { abort(); }
format_error() {}
virtual const char *what() const throw()
{
return "boost::format_error: "
@@ -44,7 +44,7 @@ public:
class bad_format_string : public format_error
{
public:
bad_format_string() { abort(); }
bad_format_string() {}
virtual const char *what() const throw()
{
return "boost::bad_format_string: "
@@ -55,7 +55,7 @@ public:
class too_few_args : public format_error
{
public:
too_few_args() { abort(); }
too_few_args() {}
virtual const char *what() const throw()
{
return "boost::too_few_args: "
@@ -66,7 +66,7 @@ public:
class too_many_args : public format_error
{
public:
too_many_args() { abort(); }
too_many_args() {}
virtual const char *what() const throw()
{
return "boost::too_many_args: "
@@ -78,7 +78,7 @@ public:
class out_of_range : public format_error
{
public:
out_of_range() { abort(); }
out_of_range() {}
virtual const char *what() const throw()
{
return "boost::out_of_range: "
-7
View File
@@ -37,13 +37,6 @@ namespace {
os.str(emptyStr);
}
void do_pad( std::string & s,
std::streamsize w,
const char c,
std::ios::fmtflags f,
bool center)
__attribute__ ((unused));
void do_pad( std::string & s,
std::streamsize w,
const char c,
+63
View File
@@ -0,0 +1,63 @@
.\"-
.\" Copyright 2003-2005 Colin Percival
.\" All rights reserved
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted providing that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
.\" STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
.\" IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
.\" POSSIBILITY OF SUCH DAMAGE.
.\"
.\" $FreeBSD: src/usr.bin/bsdiff/bsdiff/bsdiff.1,v 1.1 2005/08/06 01:59:05 cperciva Exp $
.\"
.Dd May 18, 2003
.Dt BSDIFF 1
.Os FreeBSD
.Sh NAME
.Nm bsdiff
.Nd generate a patch between two binary files
.Sh SYNOPSIS
.Nm
.Ao Ar oldfile Ac Ao Ar newfile Ac Ao Ar patchfile Ac
.Sh DESCRIPTION
.Nm
compares
.Ao Ar oldfile Ac
to
.Ao Ar newfile Ac
and writes to
.Ao Ar patchfile Ac
a binary patch suitable for use by bspatch(1).
When
.Ao Ar oldfile Ac
and
.Ao Ar newfile Ac
are two versions of an executable program, the
patches produced are on average a factor of five smaller
than those produced by any other binary patch tool known
to the author.
.Pp
.Nm
uses memory equal to 17 times the size of
.Ao Ar oldfile Ac ,
and requires
an absolute minimum working set size of 8 times the size of oldfile.
.Sh SEE ALSO
.Xr bspatch 1
.Sh AUTHORS
.An Colin Percival Aq cperciva@freebsd.org
+405
View File
@@ -0,0 +1,405 @@
/*-
* Copyright 2003-2005 Colin Percival
* All rights reserved
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted providing that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
* STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
* IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#if 0
__FBSDID("$FreeBSD: src/usr.bin/bsdiff/bsdiff/bsdiff.c,v 1.1 2005/08/06 01:59:05 cperciva Exp $");
#endif
#include <sys/types.h>
#include <bzlib.h>
#include <err.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#define MIN(x,y) (((x)<(y)) ? (x) : (y))
static void split(off_t *I,off_t *V,off_t start,off_t len,off_t h)
{
off_t i,j,k,x,tmp,jj,kk;
if(len<16) {
for(k=start;k<start+len;k+=j) {
j=1;x=V[I[k]+h];
for(i=1;k+i<start+len;i++) {
if(V[I[k+i]+h]<x) {
x=V[I[k+i]+h];
j=0;
};
if(V[I[k+i]+h]==x) {
tmp=I[k+j];I[k+j]=I[k+i];I[k+i]=tmp;
j++;
};
};
for(i=0;i<j;i++) V[I[k+i]]=k+j-1;
if(j==1) I[k]=-1;
};
return;
};
x=V[I[start+len/2]+h];
jj=0;kk=0;
for(i=start;i<start+len;i++) {
if(V[I[i]+h]<x) jj++;
if(V[I[i]+h]==x) kk++;
};
jj+=start;kk+=jj;
i=start;j=0;k=0;
while(i<jj) {
if(V[I[i]+h]<x) {
i++;
} else if(V[I[i]+h]==x) {
tmp=I[i];I[i]=I[jj+j];I[jj+j]=tmp;
j++;
} else {
tmp=I[i];I[i]=I[kk+k];I[kk+k]=tmp;
k++;
};
};
while(jj+j<kk) {
if(V[I[jj+j]+h]==x) {
j++;
} else {
tmp=I[jj+j];I[jj+j]=I[kk+k];I[kk+k]=tmp;
k++;
};
};
if(jj>start) split(I,V,start,jj-start,h);
for(i=0;i<kk-jj;i++) V[I[jj+i]]=kk-1;
if(jj==kk-1) I[jj]=-1;
if(start+len>kk) split(I,V,kk,start+len-kk,h);
}
static void qsufsort(off_t *I,off_t *V,u_char *old,off_t oldsize)
{
off_t buckets[256];
off_t i,h,len;
for(i=0;i<256;i++) buckets[i]=0;
for(i=0;i<oldsize;i++) buckets[old[i]]++;
for(i=1;i<256;i++) buckets[i]+=buckets[i-1];
for(i=255;i>0;i--) buckets[i]=buckets[i-1];
buckets[0]=0;
for(i=0;i<oldsize;i++) I[++buckets[old[i]]]=i;
I[0]=oldsize;
for(i=0;i<oldsize;i++) V[i]=buckets[old[i]];
V[oldsize]=0;
for(i=1;i<256;i++) if(buckets[i]==buckets[i-1]+1) I[buckets[i]]=-1;
I[0]=-1;
for(h=1;I[0]!=-(oldsize+1);h+=h) {
len=0;
for(i=0;i<oldsize+1;) {
if(I[i]<0) {
len-=I[i];
i-=I[i];
} else {
if(len) I[i-len]=-len;
len=V[I[i]]+1-i;
split(I,V,i,len,h);
i+=len;
len=0;
};
};
if(len) I[i-len]=-len;
};
for(i=0;i<oldsize+1;i++) I[V[i]]=i;
}
static off_t matchlen(u_char *old,off_t oldsize,u_char *new,off_t newsize)
{
off_t i;
for(i=0;(i<oldsize)&&(i<newsize);i++)
if(old[i]!=new[i]) break;
return i;
}
static off_t search(off_t *I,u_char *old,off_t oldsize,
u_char *new,off_t newsize,off_t st,off_t en,off_t *pos)
{
off_t x,y;
if(en-st<2) {
x=matchlen(old+I[st],oldsize-I[st],new,newsize);
y=matchlen(old+I[en],oldsize-I[en],new,newsize);
if(x>y) {
*pos=I[st];
return x;
} else {
*pos=I[en];
return y;
}
};
x=st+(en-st)/2;
if(memcmp(old+I[x],new,MIN(oldsize-I[x],newsize))<0) {
return search(I,old,oldsize,new,newsize,x,en,pos);
} else {
return search(I,old,oldsize,new,newsize,st,x,pos);
};
}
static void offtout(off_t x,u_char *buf)
{
off_t y;
if(x<0) y=-x; else y=x;
buf[0]=y%256;y-=buf[0];
y=y/256;buf[1]=y%256;y-=buf[1];
y=y/256;buf[2]=y%256;y-=buf[2];
y=y/256;buf[3]=y%256;y-=buf[3];
y=y/256;buf[4]=y%256;y-=buf[4];
y=y/256;buf[5]=y%256;y-=buf[5];
y=y/256;buf[6]=y%256;y-=buf[6];
y=y/256;buf[7]=y%256;
if(x<0) buf[7]|=0x80;
}
int main(int argc,char *argv[])
{
int fd;
u_char *old,*new;
off_t oldsize,newsize;
off_t *I,*V;
off_t scan,pos,len;
off_t lastscan,lastpos,lastoffset;
off_t oldscore,scsc;
off_t s,Sf,lenf,Sb,lenb;
off_t overlap,Ss,lens;
off_t i;
off_t dblen,eblen;
u_char *db,*eb;
u_char buf[8];
u_char header[32];
FILE * pf;
BZFILE * pfbz2;
int bz2err;
if(argc!=4) errx(1,"usage: %s oldfile newfile patchfile\n",argv[0]);
/* Allocate oldsize+1 bytes instead of oldsize bytes to ensure
that we never try to malloc(0) and get a NULL pointer */
if(((fd=open(argv[1],O_RDONLY,0))<0) ||
((oldsize=lseek(fd,0,SEEK_END))==-1) ||
((old=malloc(oldsize+1))==NULL) ||
(lseek(fd,0,SEEK_SET)!=0) ||
(read(fd,old,oldsize)!=oldsize) ||
(close(fd)==-1)) err(1,"%s",argv[1]);
if(((I=malloc((oldsize+1)*sizeof(off_t)))==NULL) ||
((V=malloc((oldsize+1)*sizeof(off_t)))==NULL)) err(1,NULL);
qsufsort(I,V,old,oldsize);
free(V);
/* Allocate newsize+1 bytes instead of newsize bytes to ensure
that we never try to malloc(0) and get a NULL pointer */
if(((fd=open(argv[2],O_RDONLY,0))<0) ||
((newsize=lseek(fd,0,SEEK_END))==-1) ||
((new=malloc(newsize+1))==NULL) ||
(lseek(fd,0,SEEK_SET)!=0) ||
(read(fd,new,newsize)!=newsize) ||
(close(fd)==-1)) err(1,"%s",argv[2]);
if(((db=malloc(newsize+1))==NULL) ||
((eb=malloc(newsize+1))==NULL)) err(1,NULL);
dblen=0;
eblen=0;
/* Create the patch file */
if ((pf = fopen(argv[3], "w")) == NULL)
err(1, "%s", argv[3]);
/* Header is
0 8 "BSDIFF40"
8 8 length of bzip2ed ctrl block
16 8 length of bzip2ed diff block
24 8 length of new file */
/* File is
0 32 Header
32 ?? Bzip2ed ctrl block
?? ?? Bzip2ed diff block
?? ?? Bzip2ed extra block */
memcpy(header,"BSDIFF40",8);
offtout(0, header + 8);
offtout(0, header + 16);
offtout(newsize, header + 24);
if (fwrite(header, 32, 1, pf) != 1)
err(1, "fwrite(%s)", argv[3]);
/* Compute the differences, writing ctrl as we go */
if ((pfbz2 = BZ2_bzWriteOpen(&bz2err, pf, 9, 0, 0)) == NULL)
errx(1, "BZ2_bzWriteOpen, bz2err = %d", bz2err);
scan=0;len=0;
lastscan=0;lastpos=0;lastoffset=0;
while(scan<newsize) {
oldscore=0;
for(scsc=scan+=len;scan<newsize;scan++) {
len=search(I,old,oldsize,new+scan,newsize-scan,
0,oldsize,&pos);
if (len > 64 * 1024) break;
for(;scsc<scan+len;scsc++)
if((scsc+lastoffset<oldsize) &&
(old[scsc+lastoffset] == new[scsc]))
oldscore++;
if(((len==oldscore) && (len!=0)) ||
(len>oldscore+8)) break;
if((scan+lastoffset<oldsize) &&
(old[scan+lastoffset] == new[scan]))
oldscore--;
};
if((len!=oldscore) || (scan==newsize)) {
s=0;Sf=0;lenf=0;
for(i=0;(lastscan+i<scan)&&(lastpos+i<oldsize);) {
if(old[lastpos+i]==new[lastscan+i]) s++;
i++;
if(s*2-i>Sf*2-lenf) { Sf=s; lenf=i; };
};
lenb=0;
if(scan<newsize) {
s=0;Sb=0;
for(i=1;(scan>=lastscan+i)&&(pos>=i);i++) {
if(old[pos-i]==new[scan-i]) s++;
if(s*2-i>Sb*2-lenb) { Sb=s; lenb=i; };
};
};
if(lastscan+lenf>scan-lenb) {
overlap=(lastscan+lenf)-(scan-lenb);
s=0;Ss=0;lens=0;
for(i=0;i<overlap;i++) {
if(new[lastscan+lenf-overlap+i]==
old[lastpos+lenf-overlap+i]) s++;
if(new[scan-lenb+i]==
old[pos-lenb+i]) s--;
if(s>Ss) { Ss=s; lens=i+1; };
};
lenf+=lens-overlap;
lenb-=lens;
};
for(i=0;i<lenf;i++)
db[dblen+i]=new[lastscan+i]-old[lastpos+i];
for(i=0;i<(scan-lenb)-(lastscan+lenf);i++)
eb[eblen+i]=new[lastscan+lenf+i];
dblen+=lenf;
eblen+=(scan-lenb)-(lastscan+lenf);
offtout(lenf,buf);
BZ2_bzWrite(&bz2err, pfbz2, buf, 8);
if (bz2err != BZ_OK)
errx(1, "BZ2_bzWrite, bz2err = %d", bz2err);
offtout((scan-lenb)-(lastscan+lenf),buf);
BZ2_bzWrite(&bz2err, pfbz2, buf, 8);
if (bz2err != BZ_OK)
errx(1, "BZ2_bzWrite, bz2err = %d", bz2err);
offtout((pos-lenb)-(lastpos+lenf),buf);
BZ2_bzWrite(&bz2err, pfbz2, buf, 8);
if (bz2err != BZ_OK)
errx(1, "BZ2_bzWrite, bz2err = %d", bz2err);
lastscan=scan-lenb;
lastpos=pos-lenb;
lastoffset=pos-scan;
};
};
BZ2_bzWriteClose(&bz2err, pfbz2, 0, NULL, NULL);
if (bz2err != BZ_OK)
errx(1, "BZ2_bzWriteClose, bz2err = %d", bz2err);
/* Compute size of compressed ctrl data */
if ((len = ftello(pf)) == -1)
err(1, "ftello");
offtout(len-32, header + 8);
/* Write compressed diff data */
if ((pfbz2 = BZ2_bzWriteOpen(&bz2err, pf, 9, 0, 0)) == NULL)
errx(1, "BZ2_bzWriteOpen, bz2err = %d", bz2err);
BZ2_bzWrite(&bz2err, pfbz2, db, dblen);
if (bz2err != BZ_OK)
errx(1, "BZ2_bzWrite, bz2err = %d", bz2err);
BZ2_bzWriteClose(&bz2err, pfbz2, 0, NULL, NULL);
if (bz2err != BZ_OK)
errx(1, "BZ2_bzWriteClose, bz2err = %d", bz2err);
/* Compute size of compressed diff data */
if ((newsize = ftello(pf)) == -1)
err(1, "ftello");
offtout(newsize - len, header + 16);
/* Write compressed extra data */
if ((pfbz2 = BZ2_bzWriteOpen(&bz2err, pf, 9, 0, 0)) == NULL)
errx(1, "BZ2_bzWriteOpen, bz2err = %d", bz2err);
BZ2_bzWrite(&bz2err, pfbz2, eb, eblen);
if (bz2err != BZ_OK)
errx(1, "BZ2_bzWrite, bz2err = %d", bz2err);
BZ2_bzWriteClose(&bz2err, pfbz2, 0, NULL, NULL);
if (bz2err != BZ_OK)
errx(1, "BZ2_bzWriteClose, bz2err = %d", bz2err);
/* Seek to the beginning, write the header, and close the file */
if (fseeko(pf, 0, SEEK_SET))
err(1, "fseeko");
if (fwrite(header, 32, 1, pf) != 1)
err(1, "fwrite(%s)", argv[3]);
if (fclose(pf))
err(1, "fclose");
/* Free the memory we used */
free(db);
free(eb);
free(I);
free(old);
free(new);
return 0;
}
+59
View File
@@ -0,0 +1,59 @@
.\"-
.\" Copyright 2003-2005 Colin Percival
.\" All rights reserved
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted providing that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
.\" STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
.\" IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
.\" POSSIBILITY OF SUCH DAMAGE.
.\"
.\" $FreeBSD: src/usr.bin/bsdiff/bspatch/bspatch.1,v 1.1 2005/08/06 01:59:06 cperciva Exp $
.\"
.Dd May 18, 2003
.Dt BSPATCH 1
.Os FreeBSD
.Sh NAME
.Nm bspatch
.Nd apply a patch built with bsdiff(1)
.Sh SYNOPSIS
.Nm
.Ao Ar oldfile Ac Ao Ar newfile Ac Ao Ar patchfile Ac
.Sh DESCRIPTION
.Nm
generates
.Ao Ar newfile Ac
from
.Ao Ar oldfile Ac
and
.Ao Ar patchfile Ac
where
.Ao Ar patchfile Ac
is a binary patch built by bsdiff(1).
.Pp
.Nm
uses memory equal to the size of
.Ao Ar oldfile Ac
plus the size of
.Ao Ar newfile Ac ,
but can tolerate a very small working set without a dramatic loss
of performance.
.Sh SEE ALSO
.Xr bsdiff 1
.Sh AUTHORS
.An Colin Percival Aq cperciva@freebsd.org
+224
View File
@@ -0,0 +1,224 @@
/*-
* Copyright 2003-2005 Colin Percival
* All rights reserved
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted providing that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
* STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
* IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#if 0
__FBSDID("$FreeBSD: src/usr.bin/bsdiff/bspatch/bspatch.c,v 1.1 2005/08/06 01:59:06 cperciva Exp $");
#endif
#include <bzlib.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <err.h>
#include <errno.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/types.h>
static off_t offtin(u_char *buf)
{
off_t y;
y=buf[7]&0x7F;
y=y*256;y+=buf[6];
y=y*256;y+=buf[5];
y=y*256;y+=buf[4];
y=y*256;y+=buf[3];
y=y*256;y+=buf[2];
y=y*256;y+=buf[1];
y=y*256;y+=buf[0];
if(buf[7]&0x80) y=-y;
return y;
}
void writeFull(const char * name, int fd,
const unsigned char * buf, size_t count)
{
while (count) {
ssize_t res = write(fd, (char *) buf, count);
if (res == -1) {
if (errno == EINTR) continue;
err(1,"writing to %s",name);
}
count -= res;
buf += res;
}
}
int main(int argc,char * argv[])
{
FILE * f, * cpf, * dpf, * epf;
BZFILE * cpfbz2, * dpfbz2, * epfbz2;
int cbz2err, dbz2err, ebz2err;
int fd;
ssize_t oldsize,newsize;
ssize_t bzctrllen,bzdatalen;
u_char header[32],buf[8];
u_char *old, *new;
off_t oldpos,newpos;
off_t ctrl[3];
off_t lenread;
off_t i;
if(argc!=4) errx(1,"usage: %s oldfile newfile patchfile\n",argv[0]);
/* Open patch file */
if ((f = fopen(argv[3], "r")) == NULL)
err(1, "fopen(%s)", argv[3]);
/*
File format:
0 8 "BSDIFF40"
8 8 X
16 8 Y
24 8 sizeof(newfile)
32 X bzip2(control block)
32+X Y bzip2(diff block)
32+X+Y ??? bzip2(extra block)
with control block a set of triples (x,y,z) meaning "add x bytes
from oldfile to x bytes from the diff block; copy y bytes from the
extra block; seek forwards in oldfile by z bytes".
*/
/* Read header */
if (fread(header, 1, 32, f) < 32) {
if (feof(f))
errx(1, "Corrupt patch\n");
err(1, "fread(%s)", argv[3]);
}
/* Check for appropriate magic */
if (memcmp(header, "BSDIFF40", 8) != 0)
errx(1, "Corrupt patch\n");
/* Read lengths from header */
bzctrllen=offtin(header+8);
bzdatalen=offtin(header+16);
newsize=offtin(header+24);
if((bzctrllen<0) || (bzdatalen<0) || (newsize<0))
errx(1,"Corrupt patch\n");
/* Close patch file and re-open it via libbzip2 at the right places */
if (fclose(f))
err(1, "fclose(%s)", argv[3]);
if ((cpf = fopen(argv[3], "r")) == NULL)
err(1, "fopen(%s)", argv[3]);
if (fseeko(cpf, 32, SEEK_SET))
err(1, "fseeko(%s, %lld)", argv[3],
(long long)32);
if ((cpfbz2 = BZ2_bzReadOpen(&cbz2err, cpf, 0, 0, NULL, 0)) == NULL)
errx(1, "BZ2_bzReadOpen, bz2err = %d", cbz2err);
if ((dpf = fopen(argv[3], "r")) == NULL)
err(1, "fopen(%s)", argv[3]);
if (fseeko(dpf, 32 + bzctrllen, SEEK_SET))
err(1, "fseeko(%s, %lld)", argv[3],
(long long)(32 + bzctrllen));
if ((dpfbz2 = BZ2_bzReadOpen(&dbz2err, dpf, 0, 0, NULL, 0)) == NULL)
errx(1, "BZ2_bzReadOpen, bz2err = %d", dbz2err);
if ((epf = fopen(argv[3], "r")) == NULL)
err(1, "fopen(%s)", argv[3]);
if (fseeko(epf, 32 + bzctrllen + bzdatalen, SEEK_SET))
err(1, "fseeko(%s, %lld)", argv[3],
(long long)(32 + bzctrllen + bzdatalen));
if ((epfbz2 = BZ2_bzReadOpen(&ebz2err, epf, 0, 0, NULL, 0)) == NULL)
errx(1, "BZ2_bzReadOpen, bz2err = %d", ebz2err);
if(((fd=open(argv[1],O_RDONLY,0))<0) ||
((oldsize=lseek(fd,0,SEEK_END))==-1) ||
((old=malloc(oldsize+1))==NULL) ||
(lseek(fd,0,SEEK_SET)!=0) ||
(read(fd,old,oldsize)!=oldsize) ||
(close(fd)==-1)) err(1,"%s",argv[1]);
if((new=malloc(newsize+1))==NULL) err(1,NULL);
oldpos=0;newpos=0;
while(newpos<newsize) {
/* Read control data */
for(i=0;i<=2;i++) {
lenread = BZ2_bzRead(&cbz2err, cpfbz2, buf, 8);
if ((lenread < 8) || ((cbz2err != BZ_OK) &&
(cbz2err != BZ_STREAM_END)))
errx(1, "Corrupt patch\n");
ctrl[i]=offtin(buf);
};
/* Sanity-check */
if(newpos+ctrl[0]>newsize)
errx(1,"Corrupt patch\n");
/* Read diff string */
lenread = BZ2_bzRead(&dbz2err, dpfbz2, new + newpos, ctrl[0]);
if ((lenread < ctrl[0]) ||
((dbz2err != BZ_OK) && (dbz2err != BZ_STREAM_END)))
errx(1, "Corrupt patch\n");
/* Add old data to diff string */
for(i=0;i<ctrl[0];i++)
if((oldpos+i>=0) && (oldpos+i<oldsize))
new[newpos+i]+=old[oldpos+i];
/* Adjust pointers */
newpos+=ctrl[0];
oldpos+=ctrl[0];
/* Sanity-check */
if(newpos+ctrl[1]>newsize)
errx(1,"Corrupt patch\n");
/* Read extra string */
lenread = BZ2_bzRead(&ebz2err, epfbz2, new + newpos, ctrl[1]);
if ((lenread < ctrl[1]) ||
((ebz2err != BZ_OK) && (ebz2err != BZ_STREAM_END)))
errx(1, "Corrupt patch\n");
/* Adjust pointers */
newpos+=ctrl[1];
oldpos+=ctrl[2];
};
/* Clean up the bzip2 reads */
BZ2_bzReadClose(&cbz2err, cpfbz2);
BZ2_bzReadClose(&dbz2err, dpfbz2);
BZ2_bzReadClose(&ebz2err, epfbz2);
if (fclose(cpf) || fclose(dpf) || fclose(epf))
err(1, "fclose(%s)", argv[3]);
/* Write the new file */
if((fd=open(argv[2],O_CREAT|O_TRUNC|O_WRONLY,0666))<0)
err(1,"%s",argv[2]);
writeFull(argv[2], fd, new, newsize);
if(close(fd)==-1)
err(1,"%s",argv[2]);
free(new);
free(old);
return 0;
}
+12
View File
@@ -0,0 +1,12 @@
/* Simulate BSD's <err.h> functionality. */
#ifndef COMPAT_ERR_H_INCLUDED
#define COMPAT_ERR_H_INCLUDED 1
#include <stdio.h>
#include <stdlib.h>
#define err(rc,...) do { fprintf(stderr,__VA_ARGS__); exit(rc); } while(0)
#define errx(rc,...) do { fprintf(stderr,__VA_ARGS__); exit(rc); } while(0)
#endif
+11
View File
@@ -0,0 +1,11 @@
programs += bsdiff bspatch
bsdiff_DIR := $(d)
bsdiff_SOURCES := $(d)/bsdiff.c
bsdiff_LDFLAGS = -lbz2 $(bsddiff_compat_include)
bsdiff_INSTALL_DIR = $(libexecdir)/nix
bspatch_DIR := $(d)
bspatch_SOURCES := $(d)/bspatch.c
bspatch_LDFLAGS = -lbz2 $(bsddiff_compat_include)
bspatch_INSTALL_DIR = $(libexecdir)/nix
-227
View File
@@ -1,227 +0,0 @@
#include <cstdlib>
#include <cstring>
#include <algorithm>
#include <set>
#include <memory>
#include <tuple>
#include <iomanip>
#if __APPLE__
#include <sys/time.h>
#endif
#include "machines.hh"
#include "shared.hh"
#include "pathlocks.hh"
#include "globals.hh"
#include "serialise.hh"
#include "store-api.hh"
#include "derivations.hh"
using namespace nix;
using std::cin;
static void handleAlarm(int sig) {
}
std::string escapeUri(std::string uri)
{
std::replace(uri.begin(), uri.end(), '/', '_');
return uri;
}
static string currentLoad;
static AutoCloseFD openSlotLock(const Machine & m, unsigned long long slot)
{
return openLockFile(fmt("%s/%s-%d", currentLoad, escapeUri(m.storeUri), slot), true);
}
int main (int argc, char * * argv)
{
return handleExceptions(argv[0], [&]() {
initNix();
/* Ensure we don't get any SSH passphrase or host key popups. */
unsetenv("DISPLAY");
unsetenv("SSH_ASKPASS");
if (argc != 6)
throw UsageError("called without required arguments");
auto store = openStore();
auto localSystem = argv[1];
settings.maxSilentTime = std::stoll(argv[2]);
settings.buildTimeout = std::stoll(argv[3]);
verbosity = (Verbosity) std::stoll(argv[4]);
settings.builders = argv[5];
/* It would be more appropriate to use $XDG_RUNTIME_DIR, since
that gets cleared on reboot, but it wouldn't work on OS X. */
currentLoad = settings.nixStateDir + "/current-load";
std::shared_ptr<Store> sshStore;
AutoCloseFD bestSlotLock;
auto machines = getMachines();
debug("got %d remote builders", machines.size());
if (machines.empty()) {
std::cerr << "# decline-permanently\n";
return;
}
string drvPath;
string storeUri;
for (string line; getline(cin, line);) {
auto tokens = tokenizeString<std::vector<string>>(line);
auto sz = tokens.size();
if (sz != 3 && sz != 4)
throw Error("invalid build hook line ‘%1%’", line);
auto amWilling = tokens[0] == "1";
auto neededSystem = tokens[1];
drvPath = tokens[2];
auto requiredFeatures = sz == 3 ?
std::set<string>{} :
tokenizeString<std::set<string>>(tokens[3], ",");
auto canBuildLocally = amWilling && (neededSystem == localSystem);
/* Error ignored here, will be caught later */
mkdir(currentLoad.c_str(), 0777);
while (true) {
bestSlotLock = -1;
AutoCloseFD lock = openLockFile(currentLoad + "/main-lock", true);
lockFile(lock.get(), ltWrite, true);
bool rightType = false;
Machine * bestMachine = nullptr;
unsigned long long bestLoad = 0;
for (auto & m : machines) {
debug("considering building on ‘%s’", m.storeUri);
if (m.enabled && std::find(m.systemTypes.begin(),
m.systemTypes.end(),
neededSystem) != m.systemTypes.end() &&
m.allSupported(requiredFeatures) &&
m.mandatoryMet(requiredFeatures)) {
rightType = true;
AutoCloseFD free;
unsigned long long load = 0;
for (unsigned long long slot = 0; slot < m.maxJobs; ++slot) {
auto slotLock = openSlotLock(m, slot);
if (lockFile(slotLock.get(), ltWrite, false)) {
if (!free) {
free = std::move(slotLock);
}
} else {
++load;
}
}
if (!free) {
continue;
}
bool best = false;
if (!bestSlotLock) {
best = true;
} else if (load / m.speedFactor < bestLoad / bestMachine->speedFactor) {
best = true;
} else if (load / m.speedFactor == bestLoad / bestMachine->speedFactor) {
if (m.speedFactor > bestMachine->speedFactor) {
best = true;
} else if (m.speedFactor == bestMachine->speedFactor) {
if (load < bestLoad) {
best = true;
}
}
}
if (best) {
bestLoad = load;
bestSlotLock = std::move(free);
bestMachine = &m;
}
}
}
if (!bestSlotLock) {
if (rightType && !canBuildLocally)
std::cerr << "# postpone\n";
else
std::cerr << "# decline\n";
break;
}
#if __APPLE__
futimes(bestSlotLock.get(), NULL);
#else
futimens(bestSlotLock.get(), NULL);
#endif
lock = -1;
try {
Store::Params storeParams{{"max-connections", "1"}, {"log-fd", "4"}};
if (bestMachine->sshKey != "")
storeParams["ssh-key"] = bestMachine->sshKey;
sshStore = openStore(bestMachine->storeUri, storeParams);
sshStore->connect();
storeUri = bestMachine->storeUri;
} catch (std::exception & e) {
printError("unable to open SSH connection to ‘%s’: %s; trying other available machines...",
bestMachine->storeUri, e.what());
bestMachine->enabled = false;
continue;
}
goto connected;
}
}
connected:
std::cerr << "# accept\n";
string line;
if (!getline(cin, line))
throw Error("hook caller didn't send inputs");
auto inputs = tokenizeString<PathSet>(line);
if (!getline(cin, line))
throw Error("hook caller didn't send outputs");
auto outputs = tokenizeString<PathSet>(line);
AutoCloseFD uploadLock = openLockFile(currentLoad + "/" + escapeUri(storeUri) + ".upload-lock", true);
auto old = signal(SIGALRM, handleAlarm);
alarm(15 * 60);
if (!lockFile(uploadLock.get(), ltWrite, true))
printError("somebody is hogging the upload lock for ‘%s’, continuing...");
alarm(0);
signal(SIGALRM, old);
copyPaths(store, ref<Store>(sshStore), inputs, false, true);
uploadLock = -1;
BasicDerivation drv(readDerivation(drvPath));
drv.inputSrcs = inputs;
printError("building ‘%s’ on ‘%s’", drvPath, storeUri);
auto result = sshStore->buildDerivation(drvPath, drv);
if (!result.success())
throw Error("build of ‘%s’ on ‘%s’ failed: %s", drvPath, storeUri, result.errorMsg);
PathSet missing;
for (auto & path : outputs)
if (!store->isValidPath(path)) missing.insert(path);
if (!missing.empty()) {
setenv("NIX_HELD_LOCKS", concatStringsSep(" ", missing).c_str(), 1); /* FIXME: ugly */
copyPaths(ref<Store>(sshStore), store, missing, false, true);
}
return;
});
}
-9
View File
@@ -1,9 +0,0 @@
programs += build-remote
build-remote_DIR := $(d)
build-remote_INSTALL_DIR := $(libexecdir)/nix
build-remote_LIBS = libmain libutil libformat libstore
build-remote_SOURCES := $(d)/build-remote.cc
Loaded 100 of 342 files, more files were not shown because too many files have changed in this diff. Show more