Compare commits

...
Author SHA1 Message Date
Eelco Dolstra 9936da6b54 * Check whether the outputName attribute works. 2012-01-04 12:45:53 +00:00
Eelco Dolstra 9d43a02382 * Let --disable-gc work. 2012-01-04 12:45:40 +00:00
Eelco Dolstra a0477a458f * currentOutput -> outputName. "current" implies some temporal
aspect.
2012-01-04 12:00:39 +00:00
Eelco Dolstra b79b85ad76 * Export the original input attributes of the derivation in
‘drvAttrs’.  This will simplify the implementation of functions such
  as ‘overrideDerivation’ in Nixpkgs, which need to filter out any
  added attributes such as outPath.
2012-01-04 11:56:15 +00:00
Eelco Dolstra 83647f4ef1 * Simplify the implementation of "derivation" a bit: lift out the
common attribution so that they're evaluated only once, etc.  Note
  that the default output is now the first element of the "outputs"
  attribute, rather than the first element of the sorted list of
  outputs.  This seems more user-friendly.
2012-01-04 11:04:19 +00:00
Eelco Dolstra 71f3c46cf6 * Drop the inefficient "Path" suffix in output attribute names. 2012-01-03 15:27:18 +00:00
Eelco Dolstra 921111d197 * Move the implementation of the ‘derivation’ primop into a separate
file.
2012-01-03 14:01:47 +00:00
Eelco Dolstra 6c31232e14 * Sync with the trunk. 2012-01-03 12:59:31 +00:00
Eelco Dolstra 502d94048a * Ignore missing manifest symlinks. 2012-01-03 12:59:00 +00:00
Eelco Dolstra 63227d434c * FreeBSD tar defaults to the tape device instead of stdout. 2012-01-03 12:18:41 +00:00
Eelco Dolstra 9b7df1ef00 * Forgot to add. 2012-01-03 09:19:40 +00:00
Eelco Dolstra 39d45a6b09 * Add a test for nix-channel.
* Refactor the nix-channel unpacker a bit.
2012-01-03 01:51:38 +00:00
Eelco Dolstra dadbb51d96 * Use Nix::Config. 2012-01-03 00:47:27 +00:00
Eelco Dolstra 48cea0d01e * Refactoring: Get rid of a few subdirectories in corepkgs/, and some
other simplifications.
* Use <nix/...> to locate the corepkgs.  This allows them to be
  overriden through $NIX_PATH.
* Use bash's pipefail option in the NAR builder so that we don't need
  to create a temporary file.
2012-01-03 00:16:29 +00:00
Eelco Dolstra 93e71e6ab6 * Follow our own coding conventions. 2011-12-30 17:39:03 +00:00
Eelco Dolstra f2d65c9c80 * Remove a redundant dot (Nicolas Pierron). 2011-12-30 17:31:58 +00:00
Eelco Dolstra 93b56acb2d * Support multiple outputs in nix-store (specifically the ‘--query’
and ‘--realise’ actions).
2011-12-30 17:25:19 +00:00
Eelco Dolstra a71d02440b * Oops. 2011-12-30 17:13:25 +00:00
Eelco Dolstra 6f5e3326ce * Move topoSortPaths() out of gc.cc. 2011-12-30 15:02:50 +00:00
Eelco Dolstra b1004f40f7 * Reject a build if there is a cycle among the outputs. This is
necessary because existing code assumes that the references graph is
  acyclic.
2011-12-30 14:47:14 +00:00
Eelco Dolstra ed11b17b2e * Fix doc error. 2011-12-30 13:43:29 +00:00
Eelco Dolstra 254b3399ba * Sync with the trunk. 2011-12-30 13:08:14 +00:00
Eelco Dolstra 8c42a8c8ff * Make sure that lock files are cleaned up properly when building
through the build hook.
2011-12-25 16:38:37 +00:00
Eelco Dolstra 524fa8a4f1 * Oops. 2011-12-22 16:27:03 +00:00
Eelco Dolstra b33da599c5 * In the garbage collector, delete invalid paths before deleting
unreachable paths.  This matters when using --max-freed etc.:
  unreachable paths could become reachable again, so it's nicer to
  keep them if there is "real" garbage to be deleted.  Also, don't use
  readDirectory() but read the Nix store and delete invalid paths in
  parallel.  This reduces GC latency on very large Nix stores.
2011-12-22 15:55:53 +00:00
Eelco Dolstra 58d974336c * Drop unnecessary call to canonPath() (nixStore is already canonical). 2011-12-22 14:33:34 +00:00
Eelco Dolstra 66c99b0cf5 * ‘--disable-shared’ is no longer supported. Fortunately it's not
needed for the coverage analysis.
2011-12-22 10:58:27 +00:00
Eelco Dolstra 2aac7cd021 * Another case of lock file permissions being too liberal. 2011-12-21 19:17:45 +00:00
Eelco Dolstra 4d728bc3e6 * Security: make sure the lock files used by build-remote.pl are not
readable to other users.  Otherwise, any user can open the lock file
  for reading and lock it, thus DoSing the remote build mechanism.
2011-12-21 19:11:58 +00:00
Eelco Dolstra 69d6f0936a * Use a lock to ensure that only one build-remote instance can copy a
closure to a given machine at the same time.  This prevents the case
  where multiple instances try to copy the same missing store path to
  the target machine, which is very wasteful.
2011-12-21 18:59:25 +00:00
Eelco Dolstra 5679041132 * The ‘foo.drvPath’ feature was already broken in read-only mode.
Since it's rarely used and fixing it is too much work right now,
  just document it.
2011-12-21 18:19:05 +00:00
Shea Levy f8e54b7874 Make the reference cycle in the cyclic outputs test indirect 2011-12-21 17:34:44 +00:00
Shea Levy b4cee3f816 Revert previous commit
It doesn't detect indirect references
2011-12-21 17:31:34 +00:00
Shea Levy f3c88f297d Detect and reject mutually-referential outputs
There is probably a more efficient way to do this.
2011-12-21 17:14:28 +00:00
Eelco Dolstra b19a0f63db * Simplify the context handling logic. 2011-12-21 15:33:30 +00:00
Eelco Dolstra 4be5a2c096 * Add a test for unsafeDiscardOutputDependency. Not really related to
multiple outputs, but good to have anyway.
2011-12-21 14:42:06 +00:00
Eelco Dolstra 179409b911 * Add a test for referring to another derivation's ‘drvPath’. This
currently fails in read-only mode.
2011-12-21 13:47:21 +00:00
Eelco Dolstra 3877619888 * Add some accidentally committed files. 2011-12-21 11:47:52 +00:00
Eelco Dolstra edd9359beb * Doing a GC after building a derivation with cyclic outputs currently
segfaults.
2011-12-20 17:10:39 +00:00
Eelco Dolstra 1f3b0ede7d * Add a (currently failing) test that checks whether mutually
recursive outputs are properly rejected.
* Add a (also failing) test for "nix-build -A <output-name>".
2011-12-20 17:08:43 +00:00
Eelco Dolstra 46e42c92c1 * Refactor a bit so that more tests can be added. 2011-12-20 17:01:02 +00:00
Eelco Dolstra b5363810bb * Fix the build. 2011-12-20 16:37:01 +00:00
Eelco Dolstra 194d21f9f6 * Sync with the trunk. 2011-12-16 23:33:01 +00:00
Eelco Dolstra 273b288a7e * importPath() -> importPaths(). Because of buffering of the input
stream it's now necessary for the daemon to process the entire
  sequence of exported paths, rather than letting the client do it.
2011-12-16 22:31:25 +00:00
Eelco Dolstra 8d3dfa2c17 * Avoid expensive conversions from char arrays to STL strings. 2011-12-16 21:29:46 +00:00
Eelco Dolstra e0bd307802 * Make the import operation through the daemon much more efficient
(way fewer roundtrips) by allowing the client to send data in bigger
  chunks.
* Some refactoring.
2011-12-16 19:44:13 +00:00
Eelco Dolstra 78598d06f0 * Clean up exception handling. 2011-12-16 15:45:42 +00:00
Eelco Dolstra 5a1b9ed0aa * Refactoring: move sink/source buffering into separate classes.
* Buffer the HashSink.  This speeds up hashing a bit because it
  prevents lots of calls to the hash update functions (e.g. nix-hash
  went from 9.3s to 8.7s of user time on the closure of my
  /var/run/current-system).
2011-12-15 16:19:53 +00:00
Eelco Dolstra a67b8ae224 * Typo. 2011-12-15 14:04:35 +00:00
Eelco Dolstra a3e0656cbb * Buffer reads in FdSource. Together with write buffering, this
significantly cuts down the number of syscalls (e.g., for "nix-store
  -qR /var/run/current-system" via the daemon, it reduced the number
  of syscalls in the client from 29134 to 4766 and in the daemon from
  44266 to 20666).
2011-12-15 12:32:08 +00:00
Eelco Dolstra 3a48282b06 * Buffer writes in FdSink. This significantly reduces the number of
system calls / context switches when dumping a NAR and in the worker
  protocol.
2011-12-14 23:30:06 +00:00
Eelco Dolstra 893cac1402 * Remove the terminate handler, which was only really needed because
of Berkeley DB (see r8632).
2011-12-14 22:41:10 +00:00
Eelco Dolstra 23bf700196 * Oops, the daemon test wasn't actually using the daemon. 2011-12-14 22:31:31 +00:00
Eelco Dolstra c8c0380744 * Remove unnecessary quotes. showPaths() already adds quotes. 2011-12-05 21:04:20 +00:00
Eelco Dolstra 000160f5b9 * In ‘nix-store --verify --check-contents’, repair missing hashes
rather than complain about them.
2011-12-02 17:52:18 +00:00
Eelco Dolstra 3964d95abf * nix-prefetch-url: rewritten in Perl. 2011-12-02 12:09:50 +00:00
Eelco Dolstra 92d6a5ed73 * Add some more functions to the Perl bindings. 2011-12-02 12:09:24 +00:00
Eelco Dolstra 49f59dceca * Move parseHash16or32 into libutil, and use in nix-hash. 2011-12-02 11:47:06 +00:00
Eelco Dolstra b12b21825c * Allow '<nixexpr>' syntax to be used in nix-instantiate, nix-build
and nix-env, e.g.,

  $ nix-env -f '<nixpkgs>' -i patchelf

  or

  $ nix-build '<nixos/tests>' -A login.test
2011-12-01 16:41:43 +00:00
Eelco Dolstra 23c38a04cc * Slight improvement. 2011-12-01 13:51:05 +00:00
Eelco Dolstra 24f863d86b * When doing "nix-store --add-fixed" without "--recursive" via the Nix
daemon (which is an error), print a nicer error message than
  "Connection reset by peer" or "broken pipe".
* In the daemon, log errors that occur during request parameter
  processing.
2011-12-01 13:48:48 +00:00
Peter Simons be9be4c147 doc/manual/nix-env.xml: fixed "nix-env -qaA" typo 2011-12-01 08:03:30 +00:00
Peter Simons 3c7ec8fc1b doc/manual/nix-env.xml: stripped trailing whitespace 2011-12-01 08:02:37 +00:00
Eelco Dolstra f35c4351e5 * Don't require a specific Perl version. 2011-11-29 22:15:07 +00:00
Eelco Dolstra 4d0407ba08 * Fix make check. 2011-11-29 13:38:52 +00:00
Eelco Dolstra 1749a7b0ae * download-using-manifests: use the Perl bindings. 2011-11-29 13:01:24 +00:00
Eelco Dolstra b1eb8f4249 * Get rid of some superfluous error messages if a substituter fails.
* Say "fetch" instead of "substitute".
2011-11-29 13:00:41 +00:00
Eelco Dolstra 216440b3ff * For consistency with "nix-store -q --hash", produce hashes in
base-32.  (This affects Hydra manifests.)
2011-11-29 12:32:55 +00:00
Eelco Dolstra 1df120cb05 * Get rid of the shell in ssh calls. 2011-11-29 10:51:22 +00:00
Eelco Dolstra 784083176a * Fix race. 2011-11-25 17:04:26 +00:00
Eelco Dolstra f3bc98b001 2011-11-23 15:39:54 +00:00
Eelco Dolstra 10e2b2b79e * Document the --include-outputs option. 2011-11-23 15:39:02 +00:00
Eelco Dolstra d5ac78e0d6 * Add bzip2 and xz support to nix-copy-closure. 2011-11-23 15:29:58 +00:00
Eelco Dolstra 5bbd693cae * Add an API function exportPaths() that provides the functionality of
‘nix-store --export’.
* Add a Perl module that provides the functionality of
  ‘nix-copy-closure --to’.  This is used by build-remote.pl so it no
  longer needs to start a separate nix-copy-closure process.  Also, it
  uses the Perl API to do the export, so it doesn't need to start a
  separate nix-store process either.  As a result, nix-copy-closure
  and build-remote.pl should no longer fail on very large closures due
  to an "Argument list too long" error.  (Note that having very many
  dependencies in a single derivation can still fail because the
  environment can become too large.  Can't be helped though.)
2011-11-23 15:13:37 +00:00
Eelco Dolstra ab20af3e6f * build-remote.pl: drop a hard-coded reference to /nix/etc/nix. 2011-11-23 12:21:35 +00:00
Eelco Dolstra 993fa94fb4 * Move initialisation of variables like nixConfDir from libmain to
libstore so that the Perl bindings can use it as well.  It's vital
  that the Perl bindings use the configuration file, because otherwise
  nix-copy-closure will fail with a ‘database locked’ message if the
  value of ‘use-sqlite-wal’ is changed from the default.
2011-11-22 17:28:41 +00:00
Rob Vermaas 4e1ea17052 nix: add /etc/hosts with localhost entry to chroot builds. 2011-11-21 15:19:51 +00:00
Eelco Dolstra 4de3e2a0db * Doh. 2011-11-21 13:22:34 +00:00
Eelco Dolstra 23e933b3b3 * Put back the "sys_name" variable which got removed somewhere. This
broke building on Cygwin and Solaris.
2011-11-21 12:23:48 +00:00
Eelco Dolstra a6abade8e8 * Escape the [ and ] characters in the sed call, otherwise autoconf
will eat them.
2011-11-21 12:18:26 +00:00
Eelco Dolstra 964399c079 * "sed" on FreeBSD doesn't know the "+" operator. 2011-11-20 19:23:50 +00:00
Eelco Dolstra b92f76374f * In the platform, canonicalise "amd64" to "x86_64". FreeBSD 8.2's
uname reports amd64.
* Drop the FreeBSD version number, e.g. "i686-freebsd" instead of
  "i686-freebsd8.2".
2011-11-18 17:25:40 +00:00
Ludovic Courtès 42164d6de4 configure: Change i*86 to i686 as has always been done. 2011-11-16 20:58:21 +00:00
Ludovic Courtès 45ec69cbdf configure: Rely on `AC_CANONICAL_HOST' to determine the Nix system name.
This should be more robust and also plays better with
cross-compilation---it uses the host name, instead of using the build
name.
2011-11-16 20:39:59 +00:00
Eelco Dolstra f8e609c341 * nix-pull: update the Nix manifest cache if necessary. Also, don't
read the manifest just to check the version and print the number of
  paths.  This makes nix-pull very fast for the cached cache (speeding
  up nixos-rebuild without the ‘--no-pull’ or ‘--fast’ options).
2011-11-16 16:41:48 +00:00
Eelco Dolstra d7d7910ba4 * Don't decompress the manifests in /nix/var/nix/manifest. This saves
disk space, and, since they're typically only decompressed once (to
  fill the manifest cache), doesn't make things slower.
2011-11-16 16:25:38 +00:00
Eelco Dolstra 63ee5e4d2a * Remove obsolete line. 2011-11-16 11:56:19 +00:00
Eelco Dolstra a5952405d2 * Re-use prepared statements across insertions into the manifest cache
DB.  This speeds up creating the cache from 16.1s to 7.9s on my
  system.
2011-11-16 11:37:40 +00:00
Eelco Dolstra c0b706213d * Boldly make SQLite WAL mode the default again. Hopefully the
intermittent problems are gone by now.  WAL mode is preferrable
  because it does way fewer fsyncs.
2011-11-07 21:11:59 +00:00
Shea Levy 3c3107da86 There's no need to mess with drvPath at all 2011-11-06 07:18:19 +00:00
Shea Levy 2ab29be70c Fix faulty reversion of my changes to unsafeDiscardOutputDependency 2011-11-06 07:03:14 +00:00
Shea Levy ca0d47a70c Respect all outputs passed to the derivation, not just the last one 2011-11-06 06:54:05 +00:00
Shea Levy 24b65937e1 Remove the unused sCurrentOutput symbol 2011-11-06 06:28:34 +00:00
Shea Levy 3522730316 Embed output name into the context of the *OutPath attributes and extract it for input derivations
Multiple outputs test passes!
2011-11-06 06:28:30 +00:00
Shea Levy 46876ff203 Fix stupid typo in multiple outputs test 2011-11-06 06:28:25 +00:00
Shea Levy af2e53fd48 Include all outputs of derivations in the closure of explicitly-passed derivation paths
This required adding a queryOutputDerivationNames function in the store API
2011-11-06 06:28:20 +00:00
Shea Levy 981edeab7b The 'insert output between = signs' approach was not helpful 2011-11-06 06:28:14 +00:00
Shea Levy f883afa1a1 The nixinstantiate and nixstore env vars are no longer set in common.sh 2011-11-06 06:28:08 +00:00
Shea Levy 2721e9f56f Merge from trunk 2011-11-06 00:13:09 +00:00
Eelco Dolstra a6a3f3a8c2 * Fix race condition in the test. 2011-11-05 21:23:01 +00:00
Eelco Dolstra fa69ff5726 * Fix the broken reference to bunzip2 in the channel unpack script. 2011-11-05 21:06:24 +00:00
Eelco Dolstra daed9aeac5 2011-11-03 19:22:24 +00:00
Eelco Dolstra d7b87bebe3 * The Nix configuration file is usually /etc/nix/nix.conf. 2011-11-03 18:47:10 +00:00
Eelco Dolstra 325b5a8aee * Fix permission on /nix/store in the manual for multi-user installs
(reported by Silvio Frischknecht).
2011-11-02 19:14:54 +00:00
Eelco Dolstra a12095d3be * In printValueAsXML, handle the case where a "type" attribute is not
a string.  This happens in the NixOS option system.
* Remove a bogus comparison of a unsigned integer with -1.
2011-10-27 19:06:23 +00:00
Eelco Dolstra 00b41e46ed * Print a consistent message. 2011-10-19 21:34:13 +00:00
Eelco Dolstra f186a9141e * nix-copy-closure: support ‘--dry-run’ and ‘--include-outputs’. 2011-10-18 21:21:22 +00:00
Eelco Dolstra 6761757428 * Use the Store API bindings in nix-copy-closure. 2011-10-11 15:41:13 +00:00
Eelco Dolstra d43a148204 * Add a test for nix-copy-closure. 2011-10-11 13:58:47 +00:00
Eelco Dolstra 2492914fbc * Move the remote building test from the NixOS tree to the Nix tree. 2011-10-11 13:06:59 +00:00
Eelco Dolstra c362e4d718 * Move SSH.pm. 2011-10-11 11:45:36 +00:00
Eelco Dolstra 7d314b8c95 * Work around a race condition starting the Nix daemon. 2011-10-11 11:14:30 +00:00
Eelco Dolstra 5193db048e * Set svn:ignore. 2011-10-11 09:32:34 +00:00
Eelco Dolstra a2a317eb0b * Distribute GeneratePatches.pm. 2011-10-11 09:31:55 +00:00
Eelco Dolstra 5090c34ee1 * Set the executable bit on scripts. 2011-10-10 22:40:17 +00:00
Eelco Dolstra 8af7d766f0 * Refactoring: remove unnecessary variables from the tests. 2011-10-10 21:32:34 +00:00
Eelco Dolstra cd6d02c366 2011-10-10 21:30:59 +00:00
Eelco Dolstra 6fcdbcac20 * Install NixManifest.pm, NixConfig.pm and GeneratePatches.pm under
the Nix:: namespace.
2011-10-10 21:11:08 +00:00
Eelco Dolstra 659c427caa * Hopefully perl-devel contains the required headers (untested). 2011-10-10 18:58:49 +00:00
Eelco Dolstra 73fe6871c4 * Include the Nix Perl bindings in Nix itself. This will allow the
bindings to be used in Nix's own Perl scripts.

  The only downside is that Perl XS and Automake/libtool don't really
  like each other, so building is a bit tricky.
2011-10-10 18:12:40 +00:00
Shea Levy bffe35aced Update gitignore 2011-09-19 04:36:49 +00:00
Shea Levy 6c38cc9025 Ignore everything created during build 2011-09-19 04:15:26 +00:00
Eelco Dolstra 55481c44d4 * Don't assume that we want a shared Nix store. 2011-09-17 09:53:31 +00:00
Shea Levy e81c09edbf Remove the current output metadata from the string for unsaveDiscardOutputDependency 2011-09-16 11:31:00 +00:00
Shea Levy bf50d6ad32 Add information about which output is active to drvPath's context
This will break things that depend on being able to just strip away an equals sign, so those have to be updated next
2011-09-16 11:30:52 +00:00
Shea Levy ffa038f66d Add an sCurrentOutput member to EvalState 2011-09-16 11:30:44 +00:00
Shea Levy f3e410d4bf Add a currentOutput attribute to derivations keep track of which output is active 2011-09-16 11:30:03 +00:00
Shea Levy 8f28a3ba25 Add a test for multiple outputs
This currently fails. Yay test-driven development!
2011-09-14 05:59:29 +00:00
Shea Levy c172d16b00 First attempt at the output-as-derivation semantics
For each output, this adds a corresponding attribute to the derivation that is
the same as the derivation except for outPath, which is set to the path specific
to that output. Additionally, an "all" attribute is added that is a list of all
of the output derivations. This has to be done outside of derivationStrict as
each output is itself a derivation that contains itself (and all other outputs)
as an attribute. The derivation itself is equivalent to the first output in the
outputs list (or "out" if that list isn't set).
2011-09-14 05:59:17 +00:00
Shea Levy 0b34e57eb8 Create a branch for me to play around with finishing off the multiple outputs implementation 2011-09-14 00:41:02 +00:00
Eelco Dolstra 13114daa3e * Ouch. A store upgrade could cause a substituter to be triggered,
causing a deadlock.
2011-09-12 09:07:43 +00:00
Ludovic Courtès 281e3ed059 bootstrap: Simplify & make more robust. 2011-09-06 12:11:05 +00:00
Eelco Dolstra e6cb3d0a0d * Added a command ‘nix-store --verify-paths PATHS’ to check whether
the contents of any of the given store paths have been modified.
  E.g.

    $ nix-store --verify-path $(nix-store -qR /var/run/current-system)
    path `/nix/store/m2smyiwbxidlprfxfz4rjlvz2c3mg58y-etc' was modified! expected hash `fc87e271c5fdf179b47939b08ad13440493805584b35e3014109d04d8436e7b8', got `20f1a47281b3c0cbe299ce47ad5ca7340b20ab34246426915fce0ee9116483aa'

  All paths are checked; the exit code is 1 if any path has been
  modified, 0 otherwise.
2011-09-06 12:06:30 +00:00
Eelco Dolstra 82710f96f7 * Add some -f flags, never hurts. 2011-09-06 12:00:11 +00:00
Eelco Dolstra 93227ff65c * Eliminate all uses of the global variable ‘store’ from libstore.
This should also fix:

    nix-instantiate: ./../boost/shared_ptr.hpp:254: T* boost::shared_ptr<T>::operator->() const [with T = nix::StoreAPI]: Assertion `px != 0' failed.

  which was caused by hashDerivationModulo() calling the ‘store’
  object (during store upgrades) before openStore() assigned it.
2011-08-31 21:11:50 +00:00
Eelco Dolstra 5bcdc7e351 * Update the cleanup script. 2011-08-27 16:59:32 +00:00
Eelco Dolstra a95ba4cdd9 * Use last_insert_id instead of sqlite_last_insert_rowid, which you're
not really supposed to use according to the DBD::SQLite docs, and
  fails on some systems (e.g. http://hydra.nixos.org/build/1246662).
2011-08-17 14:17:19 +00:00
Eelco Dolstra da18b11b05 * On FreeBSD, ‘touch’ is not in the test $PATH, so don't use it. 2011-08-08 14:08:38 +00:00
Eelco Dolstra 76e0029f7a * Add perl-DBD-SQLite as a dependency of the RPM builds. 2011-08-08 14:06:10 +00:00
Eelco Dolstra d374be551d * Add DBD-SQLite as a dependency to the Debian/Ubuntu builds.
* Drop some old Fedora/Debian/Ubuntu releases.
2011-08-08 13:19:00 +00:00
Eelco Dolstra 2d663b502d * Cache the result of file evaluation (i.e, memoize evalFile()). This
prevents files from being evaluated and stored as values multiple
  times.  For instance, evaluation of the ‘system’ attribute in NixOS
  causes ‘nixpkgs/pkgs/lib/lists.nix’ to be evaluated 2019 times.

  Caching gives a modest speedup and a decent memory footprint
  reduction (e.g., from 1.44s to 1.28s, and from 81 MiB to 59 MiB with
  GC_INITIAL_HEAP_SIZE=100000 on my system).
2011-08-06 19:45:43 +00:00
Eelco Dolstra 510033e783 * Handle <path> syntax. 2011-08-06 18:54:29 +00:00
Eelco Dolstra 9d091ee99a * Handle the case where the search path element is a regular file. 2011-08-06 18:45:28 +00:00
Eelco Dolstra 00a724ebc6 * Remove a debug statement. 2011-08-06 18:24:43 +00:00
Eelco Dolstra 07340b8be7 * Add the Nix corepkgs to the end of the search path. This makes it
possible for other Nix expressions to use corepkgs (mostly useful
  for the buildenv function).
2011-08-06 18:23:38 +00:00
Eelco Dolstra c7101dac0b * Allow redirections in search path entries. E.g. if you have a
directory

    /home/eelco/src/stdenv-updates

  that you want to use as the directory for import such as

    with (import <nixpkgs> { });

  then you can say

    $ nix-build -I nixpkgs=/home/eelco/src/stdenv-updates
2011-08-06 17:48:57 +00:00
Eelco Dolstra 1578b2261d * Add lang/dir* to the distribution. 2011-08-06 16:18:54 +00:00
Eelco Dolstra 1ecc97b6bd * Add a Nix expression search path feature. Paths between angle
brackets, e.g.

    import <nixpkgs/pkgs/lib>

  are resolved by looking them up relative to the elements listed in
  the search path.  This allows us to get rid of hacks like

    import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib"

  The search path can be specified through the ‘-I’ command-line flag
  and through the colon-separated ‘NIX_PATH’ environment variable,
  e.g.,

    $ nix-build -I /etc/nixos ...

  If a file is not found in the search path, an error message is
  lazily thrown.
2011-08-06 16:05:24 +00:00
Eelco Dolstra 54945a2950 * Refactoring: move parseExprFromFile() and parseExprFromString() into
the EvalState class.
2011-08-06 13:02:55 +00:00
Eelco Dolstra c8606664ab * Don't allow derivations with fixed and non-fixed outputs. 2011-07-20 18:26:00 +00:00
Eelco Dolstra b2027f70d9 * Fix a huuuuge security hole in the Nix daemon. It didn't check that
derivations added to the store by clients have "correct" output
  paths (meaning that the output paths are computed by hashing the
  derivation according to a certain algorithm).  This means that a
  malicious user could craft a special .drv file to build *any*
  desired path in the store with any desired contents (so long as the
  path doesn't already exist).  Then the attacker just needs to wait
  for a victim to come along and install the compromised path.

  For instance, if Alice (the attacker) knows that the latest Firefox
  derivation in Nixpkgs produces the path

    /nix/store/1a5nyfd4ajxbyy97r1fslhgrv70gj8a7-firefox-5.0.1

  then (provided this path doesn't already exist) she can craft a .drv
  file that creates that path (i.e., has it as one of its outputs),
  add it to the store using "nix-store --add", and build it with
  "nix-store -r".  So the fake .drv could write a Trojan to the
  Firefox path.  Then, if user Bob (the victim) comes along and does

    $ nix-env -i firefox
    $ firefox

  he executes the Trojan injected by Alice.

  The fix is to have the Nix daemon verify that derivation outputs are
  correct (in addValidPath()).  This required some refactoring to move
  the hash computation code to libstore.
2011-07-20 18:10:47 +00:00
Eelco Dolstra d2bfe1b071 * Added a test that make sure that users cannot register
specially-crafted derivations that produce output paths belonging to
  other derivations.  This could be used to inject malware into the
  store.
2011-07-20 12:15:40 +00:00
Eelco Dolstra 4bdb51e621 * Refactoring. 2011-07-20 11:50:13 +00:00
Eelco Dolstra 0243eea4b9 * Create a symlink to /nix/var/nix/manifests in /nix/var/nix/gcroots
if it doesn't exist.
2011-07-20 11:47:00 +00:00
Eelco Dolstra d329c3ea9d * Support multiple outputs. A derivation can declare multiple outputs
by setting the ‘outputs’ attribute.  For example:

    stdenv.mkDerivation {
      name = "aterm-2.5";

      src = ...;

      outputs = [ "out" "tools" "dev" ];

      configureFlags = "--bindir=$(tools)/bin --includedir=$(dev)/include";
    }

  This derivation creates three outputs, named like this:

    /nix/store/gcnqgllbh01p3d448q8q6pzn2nc2gpyl-aterm-2.5
    /nix/store/gjf1sgirwfnrlr0bdxyrwzpw2r304j02-aterm-2.5-tools
    /nix/store/hp6108bqfgxvza25nnxfs7kj88xi2vdx-aterm-2.5-dev

  That is, the symbolic name of the output is suffixed to the store
  path (except for the ‘out’ output).  Each path is passed to the
  builder through the corresponding environment variable, e.g.,
  ${tools}.

  The main reason for multiple outputs is to allow parts of a package
  to be distributed and garbage-collected separately.  For instance,
  most packages depend on Glibc for its libraries, but don't need its
  header files.  If these are separated into different store paths,
  then a package that depends on the Glibc libraries only causes the
  libraries and not the headers to be downloaded.

  The main problem with multiple outputs is that if one output exists
  while the others have been garbage-collected (or never downloaded in
  the first place), and we want to rebuild the other outputs, then
  this isn't possible because we can't clobber a valid output (it
  might be in active use).  This currently gives an error message
  like:

    error: derivation `/nix/store/1s9zw4c8qydpjyrayxamx2z7zzp5pcgh-aterm-2.5.drv' is blocked by its output paths

  There are two solutions: 1) Do the build in a chroot.  Then we don't
  need to overwrite the existing path.  2) Use hash rewriting (see the
  ASE-2005 paper).  Scary but it should work.

  This is not finished yet.  There is not yet an easy way to refer to
  non-default outputs in Nix expressions.  Also, mutually recursive
  outputs aren't detected yet and cause the garbage collector to
  crash.
2011-07-18 23:31:03 +00:00
Eelco Dolstra d9a5959139 * Show the default for --with-store-dir (Nix/211). 2011-07-13 15:57:44 +00:00
Eelco Dolstra d5d4dcd4c9 * Allow attribute names to be strings. Based on the
allow-arbitrary-strinsg-in-names patch by Marc Weber.
2011-07-13 15:53:24 +00:00
Eelco Dolstra e649f3168b * Fix concurrency issues in download-using-manifests' handling of the
SQLite manifest cache.  The DBI AutoCommit feature caused every
  process to have an active transaction at all times, which could
  indefinitely block processes wanting to update the manifest cache.

* Disable fsync() in the manifest cache because we don't need
  integrity (the cache can always be recreated if it gets corrupted).
2011-07-13 14:05:54 +00:00
Eelco Dolstra 0a623a10c7 * Allow a default value in attribute selection by writing
x.y.z or default

  (as originally proposed in
  https://mail.cs.uu.nl/pipermail/nix-dev/2009-September/002989.html).

  For instance, an expression like

    stdenv.lib.attrByPath ["features" "ckSched"] false args

  can now be written as

    args.features.ckSched or false
2011-07-13 12:19:57 +00:00
Eelco Dolstra 2b9e29b1c8 * Change the right-hand side of the ‘.’ operator from an attribute to
an attribute path.  This is a refactoring to support default values.
2011-07-06 12:28:57 +00:00
Eelco Dolstra 5580f3817c * Test case. 2011-07-06 10:58:53 +00:00
Eelco Dolstra 5637037802 * In the ‘?’ operator, allow attribute paths. For instance, you can
write ‘attrs ? a.b’ to test whether ‘attrs’ has an attribute ‘a’
  containing an attribute ‘b’.  This is more convenient than ‘attrs ?
  a && attrs.a ? b’.

  Slight change in the semantics: it's no longer an error if the
  left-hand side of ‘?’ is not an attribute set.  In that case it just
  returns false.  So, ‘null ? foo’ no longer throws an error.
2011-07-06 10:58:17 +00:00
Eelco Dolstra 34f4b91820 2011-07-04 14:44:53 +00:00
Ludovic Courtès 1ea6e06445 doc: Fix typo. 2011-06-30 15:48:40 +00:00
Ludovic Courtès 5c9e9f732d Add support for the build-timeout' and --timeout' options. 2011-06-30 15:19:13 +00:00
Eelco Dolstra 9c99aa2620 2011-06-27 09:01:09 +00:00
Eelco Dolstra e4d8148585 2011-06-27 08:04:54 +00:00
Eelco Dolstra 4891b21f34 * Use SQLite 3.7.6.2. 2011-05-03 16:17:24 +00:00
Eelco Dolstra d94cb02bfe * nix-install-package: unset NIX_REMOTE because $NIX_MANIFESTS_DIR
doesn't work when building through the Nix daemon.  This also
  ensures an error message when the user doesn't have sufficient
  privileges to do nix-pull.
2011-04-19 10:52:35 +00:00
Eelco Dolstra 8c69dac8a1 * Handle error messages from the Nix worker containing the `%'
character.  (Nix/216)
2011-04-19 10:44:44 +00:00
Eelco Dolstra 4ba6afaf48 * nix-env -ub' (--prebuilt-only') didn't really work because it
checked too soon whether substitutes are available.  That is, it did
  so for every available package, rather than those matching installed
  packages.  This was very slow and subject to assertion failures.  So
  do the check much later.  Idem for `nix-env -qab' and `nix-env -ib'.
2011-04-11 16:27:05 +00:00
Eelco Dolstra 412914d004 * Read manifests directly into the database, rather than first reading
them into memory.  This brings memory use down to (more or less)
  O(1).  For instance, on my test case, the maximum resident size of
  download-using-manifests while filling the DB went from 142 MiB to
  11 MiB.
2011-04-11 13:16:54 +00:00
Eelco Dolstra 08c8971498 * Lock the database during updates. 2011-04-11 12:40:13 +00:00
Eelco Dolstra b2c11b9ed0 2011-04-11 10:23:15 +00:00
Eelco Dolstra 3cbf680f5d * configure: detect whether DBD::SQLite is present. If necessary the
location to DBI and DBD::SQLite can be passed with --with-dbi and
  --with-dbd-sqlite.
2011-04-11 10:13:53 +00:00
Eelco Dolstra 83252b4ca9 * Subtle bug: if you import File::stat in one module, it affects other
modules as well.  So use symbolic field names everywhere (which is
  nicer anyway).
2011-04-11 08:21:30 +00:00
Eelco Dolstra b1882c3ef7 * Create $manifestDir if it doesn't exist. 2011-04-11 08:06:14 +00:00
Eelco Dolstra 5591fcc529 * Cache the manifests in /nix/var/nix/manifests in a SQLite database.
This significantly speeds up the download-using-manifests
  substituter, especially if manifests are very large.  For instance,
  one "nix-build -A geeqie" operation that updated four packages using
  binary patches went from 18.5s to 1.6s.  It also significantly
  reduces memory use.

  The cache is kept in /nix/var/nix/manifests/cache.sqlite.  It's
  updated automatically when manifests are added to or removed from
  /nix/var/nix/manifests.  It might be interesting to have nix-pull
  store manifests directly in the DB, rather than storing them as
  separate flat files, but then we would need a command line interface
  to delete manifests from the DB.
2011-04-10 23:22:46 +00:00
Eelco Dolstra 1e7e4f21ba * Remove the localPaths feature in manifests since it's no longer used
and redundant anyway.
2011-04-06 09:16:22 +00:00
Eelco Dolstra 0423d0692a * Print a better error message. 2011-03-16 15:55:57 +00:00
Eelco Dolstra 48bdbbf070 * Fix a bug in the documentation (reported by Olexiy Buyanskyy,
Nix/215).
2011-03-15 13:12:59 +00:00
Eelco Dolstra 802c2651c8 2011-02-17 15:08:33 +00:00
Eelco Dolstra e945b52a31 * nix-push: handle the case where the hash is not set in the DB. 2011-02-17 13:26:33 +00:00
Eelco Dolstra c3cee5395b * Build for Ubuntu 10.04. 2011-02-14 14:56:02 +00:00
Florian Friesdorf 8fcaf3e9c6 make nix-mode provide 'nix-mode
this enables (require 'nix-mode)
2011-02-14 03:11:56 +00:00
Eelco Dolstra 538b7caab0 * Don't allocate a big initial GC address space on machines with
little RAM.  Even if the memory isn't actually used, it can cause
  problems with the overcommit heuristics in the kernel.  So use a VM
  space of 25% of RAM, up to 384 MB.
2011-02-10 14:31:04 +00:00
Eelco Dolstra 5a6b039802 * Don't call GC_expand_hp unless we're actually using the garbage
collector.
2011-02-09 22:59:50 +00:00
Ludovic Courtès f1462c208b Use $BDW_GC_LIBS instead of a custom variable. 2011-02-09 21:23:13 +00:00
Eelco Dolstra 1876ab764f * A better fix. $boehmgc isn't set anywhere, we should use the flags
returned by pkg-config.
2011-02-09 14:13:09 +00:00
Eelco Dolstra dfc4117e90 * The GC library can't be found on Solaris
(http://hydra.nixos.org/build/890714), so don't build with GC
  support for now.
2011-02-09 14:08:32 +00:00
Eelco Dolstra d159ea1b7e * Urgh, FreeBSD doesn't have a "seq" command. 2011-02-09 14:03:16 +00:00
Eelco Dolstra 3087b3f751 * Obsolete. 2011-02-09 12:48:54 +00:00
Eelco Dolstra eb94581d39 * Remove obsolete directory. 2011-02-09 12:43:44 +00:00
Eelco Dolstra 7bba67c401 * Remove obsolete file. 2011-02-09 12:42:15 +00:00
Eelco Dolstra d0eda1f3e9 * Merged the SQLite branch. 2011-02-09 12:41:54 +00:00
Eelco Dolstra 543988572e * Sync with the trunk. 2011-02-09 12:25:53 +00:00
Eelco Dolstra c0340eec5a 2011-02-08 12:30:28 +00:00
Eelco Dolstra 0304fda3cf * Propagate the CC setting. 2011-02-05 16:40:15 +00:00
Eelco Dolstra aeae0beba4 * Clang compatibility fix. Clang actually checks template definitions
when they are defined
2011-02-05 16:29:10 +00:00
Ludovic Courtès 3854fc9b42 Remove useless <config.h> inclusion from public header. 2011-01-14 13:55:58 +00:00
Eelco Dolstra 9db190eb31 * builtins.substring: if "start" is beyond the end of the string,
return the empty string.
2011-01-14 12:47:10 +00:00
Eelco Dolstra d6c8b995c5 * In Hydra manifests the Size field is missing, so don't rely on it.
This caused a lot of "Use of uninitialized value" warnings from
  Perl.
2010-12-22 10:32:32 +00:00
Eelco Dolstra c931a7aec5 * Do a short sleep after SQLITE_BUSY. 2010-12-17 17:23:15 +00:00
Eelco Dolstra b1eb252172 * Propagate the "100" exit status for failed builds through the Nix
daemon.
2010-12-17 11:28:26 +00:00
Eelco Dolstra eac93d6efe * Use the right `make'. 2010-12-17 09:28:51 +00:00
Eelco Dolstra a0be433fec * Disable X11 forwarding, it's not needed. 2010-12-15 14:25:54 +00:00
Eelco Dolstra f1a6b97639 * nix-copy-closure: make sure that the shell doesn't do globbing of
`+' and `?' in filenames.  This is very slow if /nix/store is very
  large.  (This is a quick hack - a cleaner solution would be to
  bypass the shell entirely.)
2010-12-15 08:39:37 +00:00
Eelco Dolstra 3dd02580e3 * I forgot to catch SQLiteBusy in registerValidPaths(). So
registerValidPaths() now handles busy errors and registerValidPath()
  is simply a wrapper around it.
2010-12-14 13:25:20 +00:00
Eelco Dolstra d787285af9 * nix-instantiate: return exit status 100 to denote a permanent build
failure.  The build hook can use this to distinguish between
  transient and permanent failures on the remote side.
2010-12-13 16:53:23 +00:00
Eelco Dolstra 5833243c92 * Create /nix/var/nix/db if it's missing. 2010-12-13 13:42:34 +00:00
Eelco Dolstra 100becf8d1 * createDirs(path): if path already exists, make sure it's a
directory.
* Provide a C++ wrapper around lstat().
2010-12-13 13:32:58 +00:00
Eelco Dolstra d7ca6f44eb * Update some comments. 2010-12-13 13:19:46 +00:00
Eelco Dolstra 542fc69062 * When doing a query (e.g. `nix-store -r --dry-run'), don't make a lot
of expensive calls to `nix-store --check-validity'.
2010-12-13 08:39:10 +00:00
Eelco Dolstra 4d57776813 * Use SQLite 3.7.4. 2010-12-10 11:45:56 +00:00
Eelco Dolstra e4720b1a79 * Ignore the result of sqlite3_reset(). 2010-12-08 18:19:15 +00:00
Eelco Dolstra 7d0444e244 * Bad things happen when a filehandle has the same name as a module. 2010-12-07 12:33:42 +00:00
Eelco Dolstra 8062d3af30 * `nix-store --verify --check-contents': don't hold the global GC lock
while checking the contents, since this operation can take a very
  long time to finish.  Also, fill in missing narSize fields in the DB
  while doing this.
2010-12-06 15:29:38 +00:00
Eelco Dolstra de79d23f76 * Retry a transaction if SQLite returns SQLITE_BUSY. This can happen
even with a very long busy timeout, because SQLITE_BUSY is also
  returned to resolve deadlocks.  This should get rid of random
  "database is locked" errors.  This is kind of hard to test though.
* Fix a horrible bug in deleteFromStore(): deletePathWrapped() should
  be called after committing the transaction, not before, because the
  commit might not succeed.
2010-12-05 18:23:19 +00:00
Eelco Dolstra 365f3028dd * Use CamelCase for the Perl modules. 2010-12-05 17:50:29 +00:00
Eelco Dolstra f42a505ab7 * Add a script `nix-generate-patches'.
* Fix the binary patching test.
2010-12-05 17:36:02 +00:00
Eelco Dolstra 77f7a6d591 * Quick hack around one of the pathological cases in bsdiff
(namely, when there is a long region of 0s).  On one 86 MiB
  case, this cut patch generation time from 44m to 2m.
2010-11-29 17:07:04 +00:00
Eelco Dolstra d3bba0c2d8 * Move the patch generator into a module. 2010-11-29 15:26:28 +00:00
Eelco Dolstra 9737a7eba0 * Don't generate patches if the system type differs. 2010-11-29 09:26:05 +00:00
Eelco Dolstra bd48fd97f6 * Impose a configurable time limit on patch generation. This is
necessary because bsdiff has some pathological worst-case behaviour,
  e.g. O(n^2) on files consisting only of 0s.
2010-11-29 09:15:45 +00:00
Eelco Dolstra 703e5a2ce2 * Sync with the trunk. 2010-11-25 14:09:13 +00:00
Eelco Dolstra 812fae424e * DrvInfo contains pointers to expressions, so DrvInfos should be
traced by the garbage collector.  Otherwise "nix-env -u" can crash
  randomly.
2010-11-25 13:47:34 +00:00
Eelco Dolstra d92ccbf1ac * Test whether sequences of patches work. 2010-11-17 18:01:37 +00:00
Eelco Dolstra 1a211d812f * Oops. 2010-11-17 17:54:49 +00:00
Eelco Dolstra a4f0365b2d * When checking whether a patch is applicable, for efficiency, use
`nix-store -q --hash' to get the hash of the base path rather than
  `nix-hash'.  However, only do this for estimating the size of a
  download, not for the actual substitution, because sometimes the
  contents of store paths are modified (which they shouldn't, of
  course).
2010-11-17 17:41:59 +00:00
Eelco Dolstra 3d38a49840 * In the download size indication, take binary patches into account.
Hopefully this doesn't slow things down too much.
2010-11-17 17:32:25 +00:00
Eelco Dolstra f69626ed3e 2010-11-17 15:45:09 +00:00
Eelco Dolstra a07c68f05e * Finally, a test for the binary patch functionality. 2010-11-17 15:30:07 +00:00
Eelco Dolstra bf658f016f * Test that download sizes are shown correctly. 2010-11-17 14:39:02 +00:00
Eelco Dolstra 1e24cbaba3 * Fix the test. 2010-11-17 14:35:14 +00:00
Eelco Dolstra bdf089f463 * Before a build, show the disk space that the downloaded store paths
will approximately require.
2010-11-17 14:31:42 +00:00
Eelco Dolstra 06699d4219 * Store the NAR size in the manifest. 2010-11-17 12:57:52 +00:00
Eelco Dolstra 5693b8a7e2 * nix-push: no need to compute the NAR hash, since the Nix database
already has it (`nix-store -q --hash').
2010-11-17 12:51:54 +00:00
Eelco Dolstra e60c962fb8 * Add an operation `nix-store -q --size'. 2010-11-17 12:40:52 +00:00
Eelco Dolstra 1db6259076 * Implement RemoteStore::queryPathInfo(). 2010-11-17 12:08:01 +00:00
Eelco Dolstra a3883cbd28 * Store the size of a store path in the database (to be precise, the
size of the NAR serialisation of the path, i.e., `nix-store --dump
  PATH').  This is useful for Hydra.
2010-11-16 17:11:46 +00:00
Eelco Dolstra fb9368b5a0 * Sync with the trunk. 2010-11-16 12:49:47 +00:00
Eelco Dolstra 4aced7f8d0 * Merge the GC branch. 2010-10-29 15:04:39 +00:00
Eelco Dolstra 26def5392f * Document Boehm GC support. 2010-10-29 14:44:02 +00:00
Eelco Dolstra 3d71c8013e * Use pkgconfig to locate the Boehm GC (as suggested by Ludo), if
--enable-gc is given.
2010-10-29 14:00:47 +00:00
Eelco Dolstra 14fbf85380 * Set libgc's initial heap size to 384 MiB to prevent garbage
collection in most cases (and therefore its performance overhead).
2010-10-29 13:11:50 +00:00
Eelco Dolstra 0c4828ea05 * new(UseGC) is inexplicably slower than GC_MALLOC, so prefer the
latter.
2010-10-28 12:50:01 +00:00
Eelco Dolstra e11e6fb1c6 * Handle out of memory condition. 2010-10-28 12:29:40 +00:00
Eelco Dolstra 8a788e38ac * Install config.h. 2010-10-26 10:47:02 +00:00
Eelco Dolstra 11ccd44e95 * We need Bison 2.4 now. 2010-10-24 21:48:59 +00:00
Eelco Dolstra 43535499f3 * When allocating an attribute set, reserve enough space for all
elements.  This prevents the vector from having to resize itself.
2010-10-24 20:09:37 +00:00
Eelco Dolstra e0b7fb8f27 * Keep attribute sets in sorted order to speed up attribute lookups.
* Simplify the representation of attributes in the AST.
* Change the behaviour of listToAttrs() in case of duplicate names.
2010-10-24 19:52:33 +00:00
Eelco Dolstra 2dc6d50941 * Don't create thunks for variable lookups (if possible). This
significantly reduces the number of values allocated (e.g. from 8.7m
  to 4.9m for the Bittorrent test).
2010-10-24 14:20:02 +00:00
Eelco Dolstra 0b305c534f * Store attribute sets as a vector instead of a map (i.e. a red-black
tree).  This saves a lot of memory.  The vector should be sorted so
  that names can be looked up using binary search, but this is not the
  case yet.  (Surprisingly, looking up attributes using linear search
  doesn't have a big impact on performance.)

  Memory consumption for

    $ nix-instantiate /etc/nixos/nixos/tests -A bittorrent.test --readonly-mode

  on x86_64-linux with GC enabled is now 185 MiB (compared to 946
  MiB on the trunk).
2010-10-24 00:41:29 +00:00
Eelco Dolstra a247d20604 * Fix compiling without Boehm.
* Fix the stats.
2010-10-23 22:58:24 +00:00
Eelco Dolstra 02934b1200 * Regression test for listToAttr's behaviour if an attribute name
occurs multiple times.
2010-10-23 22:55:30 +00:00
Eelco Dolstra b2ba62170c * Optimise string constants by putting them in the symbol table. 2010-10-23 21:11:59 +00:00
Eelco Dolstra 8ac06726b9 * Make Value smaller by not storing redundant PrimOp info.
* Clear pointers in Values after overwriting them to make sure that no
  objects are kept alive unnecessarily.
2010-10-23 20:07:47 +00:00
Eelco Dolstra 3f66cfb96b * Remove allocValues(). 2010-10-23 18:18:07 +00:00
Eelco Dolstra 4dee289550 * In environments, store pointers to values rather than values. This
improves GC effectiveness a bit more (because a live value doesn't
  keep other values in the environment plus the parent environments
  alive), and removes the need for copy nodes.
2010-10-22 15:51:52 +00:00
Eelco Dolstra cf7e645a48 * Regression test for __overrides. 2010-10-22 15:15:12 +00:00
Eelco Dolstra 41c45a9b31 * Store Value nodes outside of attribute sets. I.e., Attr now stores
a pointer to a Value, rather than the Value directly.  This improves
  the effectiveness of garbage collection a lot: if the Value is
  stored inside the set directly, then any live pointer to the Value
  causes all other attributes in the set to be live as well.
2010-10-22 14:47:42 +00:00
Eelco Dolstra 64c3325b0b * Make building against the Boehm GC a configure option. 2010-10-22 13:39:15 +00:00
Eelco Dolstra 76feaf016a * Keep some more stats. 2010-10-20 15:48:00 +00:00
Eelco Dolstra e879a0371b * Use the Boehm garbage collector to reclaim unused memory in the Nix
expression evaluator.
2010-10-20 11:38:30 +00:00
Eelco Dolstra b0c11cda7e * Evaluator garbage collection branch. 2010-10-20 09:08:39 +00:00
Eelco Dolstra 64fd29855a * Wrap deleteFromStore() in a transaction. Otherwise there might be a
race with other processes that add new referrers to a path,
  resulting in the garbage collector crashing with "foreign key
  constraint failed".  (Nix/4)
* Make --gc --print-dead etc. interruptible.
2010-10-14 15:55:51 +00:00
Rob Vermaas 8dadcede65 nix manual: fix 'install' -> 'uninstall' in garbage collection section of introduction 2010-10-06 19:04:04 +00:00
Eelco Dolstra 7119d38287 * In the referrers test, lower the nesting depth from 2500 to 1000 to
prevent hitting a stack overflow bug in the garbage collector.
2010-10-04 22:26:38 +00:00
Eelco Dolstra 705868a8a9 * Make sure that config.h is included before the system headers,
because it defines _FILE_OFFSET_BITS.  Without this, on 
  OpenSolaris the system headers define it to be 32, and then 
  the 32-bit stat() ends up being called with a 64-bit "struct 
  stat", or vice versa.

  This also ensures that we get 64-bit file sizes everywhere.

* Remove the redundant call to stat() in parseExprFromFile().
  The file cannot be a symlink because that's the exit condition
  of the loop before.
2010-10-04 17:55:38 +00:00
Eelco Dolstra 95f4f2cf61 * If std::tr1::unordered_set is unavailable, use std::set. 2010-10-04 16:16:19 +00:00
Eelco Dolstra 36a23e86b6 * "type -P" isn't portable. 2010-10-04 15:50:08 +00:00
Eelco Dolstra bfa6ee7d91 * Don't use SSH's `-tt' flag because it doesn't seem to work
on OpenSolaris when using connection sharing.  Instead have
  the remote side check for disconnection and kill the process 
  group when that happens.
2010-10-04 12:30:46 +00:00
Eelco Dolstra 71dfe4b90b * Sync with the trunk. 2010-10-04 11:44:47 +00:00
Eelco Dolstra 450837bcc8 * In printMsg(), ignore failing writes to stderr if we're in an
exception handler, otherwise throw an exception.  We need to ignore
  write errors in exception handlers to ensure that cleanup code runs
  to completion if the other side of stderr has been closed
  unexpectedly.
2010-10-04 11:23:07 +00:00
Eelco Dolstra 4aa9245083 * Hack needed for GCC 4.3.2 on OpenSolaris. 2010-10-04 10:51:16 +00:00
Eelco Dolstra 923736df38 * Doh. Remove debug message. 2010-09-14 12:47:19 +00:00
Eelco Dolstra e4907411c2 * Only do "pragma journal_mode = ..." if the current journal mode
differs from the desired mode.  There is an open SQLite ticket
  `Executing "PRAGMA journal_mode" may delete journal file while it is
  in use.'
2010-09-01 11:36:22 +00:00
Eelco Dolstra bf0dde9597 * Always print hook output on stderr, even if --no-build-output is
set.
* In the build hook, print a trace message to allow Hydra to pick up
  the name of the remote machine used for the build.
2010-08-31 12:36:24 +00:00
Eelco Dolstra e2e168f7c2 `nix-store --verify' improvements:
* If a path has disappeared, check its referrers first, and don't try
  to invalidate paths that have valid referrers.  Otherwise we get a
  foreign key constraint violation.
* Read the whole Nix store directory instead of statting each valid
  path, which is slower.
* Acquire the global GC lock.
2010-08-31 11:47:31 +00:00
Eelco Dolstra 80e722278c * When using the build hook, distinguish between the stderr of the
hook script proper, and the stdout/stderr of the builder.  Only the
  latter should be saved in /nix/var/log/nix/drvs.
* Allow the verbosity to be set through an option.
* Added a flag --quiet to lower the verbosity level.
2010-08-30 14:53:03 +00:00
Eelco Dolstra 20acd43c25 * Disable the GC reachability check for now (when creating new roots),
as it's hopelessly inefficient.
2010-08-30 14:11:57 +00:00
Eelco Dolstra 766f708418 * Experimental feature: allow a derivation to tell the build hook that
it requires a certain feature on the build machine, e.g.

    requiredSystemFeatures = [ "kvm" ];

  We need this in Hydra to make sure that builds that require KVM
  support are forwarded to machines that have KVM support.  Probably
  this should also be enforced for local builds.
2010-08-27 13:18:13 +00:00
Eelco Dolstra df50916e46 * Oops - "null" was displayed as "true". 2010-08-27 12:10:56 +00:00
Eelco Dolstra e41ecbf730 2010-08-27 11:09:04 +00:00
Eelco Dolstra e437b08250 * Made the build hook mechanism more efficient. Rather than starting
the hook every time we want to ask whether we can run a remote build
  (which can be very often), we now reuse a hook process for answering
  those queries until it accepts a build.  So if there are N
  derivations to be built, at most N hooks will be started.
2010-08-25 20:44:28 +00:00
Eelco Dolstra 1a396f3789 * Don't call "cat". 2010-08-25 12:19:30 +00:00
Eelco Dolstra 95deba581d * In the build hook, temporarily register the derivation and its
output as GC roots.  This prevents a race if the garbage collector
  is running during the build.
2010-08-25 11:54:11 +00:00
Eelco Dolstra 1e5f5ea2e9 * Correctly handle SSH failing to establish a connection. Because
this didn't work right, the build hook wouldn't fall back to using
  other machines of the desired type.
2010-08-24 14:27:07 +00:00
Eelco Dolstra 034f608e00 * Don't complain if the stored hash of a path is zero (unknown). 2010-08-24 14:25:33 +00:00
Eelco Dolstra f58f51f380 * Handle the unlikely case where a derivation has no dependencies at
all.
2010-08-24 11:45:44 +00:00
Eelco Dolstra 955d11aae7 * Sync with the trunk. 2010-08-24 09:56:28 +00:00
Eelco Dolstra c67eccc26d * nix-build: pass --cores. 2010-08-24 09:21:59 +00:00
Eelco Dolstra 2c8e070e5d * Bump the version number. 2010-08-17 15:39:35 +00:00
Eelco Dolstra ed133e6e64 * rpmBuild already includes the disk image in the output name. 2010-08-17 14:08:44 +00:00
Eelco Dolstra 2de17f4edc * Update date. 2010-08-17 10:06:26 +00:00
Eelco Dolstra 86f65edf4e * Document --cores in the manual. 2010-08-17 07:22:05 +00:00
Eelco Dolstra b75e1043a3 * Typo. 2010-08-16 13:23:45 +00:00
Eelco Dolstra 8ec6594d6d * Remove the "tarball" jobset argument. 2010-08-16 13:01:31 +00:00
Eelco Dolstra 12721a3a9a * Nix 0.16 release notes. 2010-08-16 12:38:32 +00:00
Eelco Dolstra 5fb824e896 * Urgh, this was supposed to go in the trunk... 2010-08-12 13:36:56 +00:00
Eelco Dolstra 5c5ab2bc12 * Don't link against pthreads. This was added way back in r211
because Berkeley DB needed it on some platforms, but we don't use
  BDB anymore.

  On FreeBSD, if you link against pthreads, then the main thread gets
  a 2 MB stack which cannot be overriden (it ignores "ulimit -s"):

    http://www.mail-archive.com/freebsd-hackers@freebsd.org/msg62445.html
  
  This is not enough for Nix.  For instance, the garbage collector can
  fail if there is a pathologically deep chain of references
  (http://hydra.nixos.org/build/556199).  2 MB is also not enough for
  many Nix expressions.
  
  Arguably the garbage collector shouldn't use recursion, because in
  NixOS unprivileged users can DOS the garbage collector by creating a
  sufficiently deeply nested chain of references.  But getting rid of
  recursion is a bit harder.
2010-08-12 13:34:34 +00:00
Eelco Dolstra 6846ed8b44 * Make --cores work when building through the Nix daemon. 2010-08-12 09:21:50 +00:00
Eelco Dolstra 5f9aad44ca * nix-build: recognise --cores. 2010-08-11 15:28:02 +00:00
Eelco Dolstra d7875d1648 * Use SQLite 3.7.0.1. 2010-08-06 07:51:27 +00:00
Eelco Dolstra 587dc8aa00 * Sync with the trunk. 2010-08-04 17:48:29 +00:00
Eelco Dolstra fd9c77dfc7 * Use SQLite 3.7.0's write-ahead logging (WAL mode). This is a lot
faster than the old mode when fsyncs are enabled, because it only
  performs an fsync() when doing a checkpoint, rather than at every
  commit.  Some timings for doing a "nix-instantiate /etc/nixos/nixos
  -A system" after modifying the stdenv setup script:

    42.5s - SQLite 3.6.23 with truncate mode and fsync
     3.4s - SQLite 3.6.23 with truncate mode and no fsync
    32.1s - SQLite 3.7.0 with truncate mode and fsync
    16.8s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
            every 1000 pages
     8.3s - SQLite 3.7.0 with WAL mode and fsync, auto-checkpoint
            every 8192 pages
     1.7s - SQLite 3.7.0 with WAL mode and no fsync

  The default is now to use WAL mode with fsyncs.  Because WAL doesn't
  work on remote filesystems such as NFS (as it uses shared memory),
  truncate mode can be re-enabled by setting the "use-sqlite-wal"
  option to false.
2010-08-04 17:35:59 +00:00
Eelco Dolstra 750be19ae8 * Remove "auto" and "guess" as synonyms for 0 in the handling of
build-cores and --cores.  They're superfluous and just complicate
  the parsing.
2010-08-04 12:23:59 +00:00
Eelco Dolstra 7f893b7a43 * Allow derivations to hint that they should not be built remotely
using the build hook mechanism, by setting the derivation attribute
  "preferLocalBuild" to true.  This has a few use cases:

  - The user environment builder.  Since it just creates a bunch of
    symlinks without much computation, there is no reason to do it
    remotely.  In fact, doing it remotely requires the entire closure
    of the user environment to be copied to the remote machine, which
    is extremely wasteful.

  - `fetchurl'.  Performing the download on a remote machine and then
    copying it to the local machine involves twice as much network
    traffic as performing the download locally, and doesn't save any
    CPU cycles on the local machine.
2010-08-04 12:13:58 +00:00
Eelco Dolstra 315d8fbd75 * Set the default system filter to "*". This ensures that (for
instance) "nix-env -i wine" works on x86_64-linux, even though Wine
  is built on i686-linux.  In the event that there are multiple
  matching derivations, prefer those built for the current system.
2010-08-04 09:32:42 +00:00
Eelco Dolstra 6d6200f37a * Optimisation in the // operator: if one of the sets is empty, return
the other set.
2010-08-02 16:31:05 +00:00
Eelco Dolstra 7af6a2fd71 * intersectAttrs: optimise for the case where the second set is larger
than the first set.  (That's usually the case with callPackage.)
2010-08-02 11:54:44 +00:00
Eelco Dolstra 532d766c27 * Don't barf if the source NAR for a patch has disappeared. 2010-07-21 11:30:23 +00:00
Peter Simons 7e043d28a6 src/bsdiff-4.3/Makefile.am: include the 'compat-include' directory in distribution tarballs 2010-07-15 14:35:20 +00:00
Peter Simons 60b632b173 tests/build-hook.hook.sh: prefer more portable ... syntax over $(...) for running sub-shells
The /bin/sh interpreter on Solaris doesn't understand $(...) syntax for running
sub-shells. Consequently, this test fails on Solaris. To remedy the situation,
the script either needs to be run by /bin/bash -- which is non-standard --, or
it needs to use the ancient but portable `...` syntax.
2010-06-25 14:05:37 +00:00
Peter Simons a0d29040f7 Revert "configure.ac: make flex and bison required programs"
This reverts commit 22405. Apparently, these programs aren't necessarily
required when building from a release archive.
2010-06-24 22:22:24 +00:00
Peter Simons af09fe12dd Fix build of bsdiff-4.3 on machines that don't have <err.h>, such as Solaris. 2010-06-24 17:51:31 +00:00
Peter Simons d63375d529 configure.ac: Incredibly enough, tr(1) on Solaris doesn't understand A-Z syntax for ranges. 2010-06-24 17:51:24 +00:00
Peter Simons 4c21c016c5 configure.ac: make flex and bison required programs
The build fails if these tools aren't available.
2010-06-24 17:51:19 +00:00
Peter Simons bcec46057c src/libutil/util.cc: include <limit.h> to ensure that PATH_MAX is defined 2010-06-24 17:51:13 +00:00
Peter Simons a17071fef1 Include <cstring> to ensure that strcpy(), strlen(), and memset() are declared.
An "using namespace std" was added locally in those functions that refer to
names from <cstring>. That is not pretty, but it's a very portable solution,
because strcpy() and friends will be found in both the 'std' and in the global
namespace.
2010-06-24 17:51:04 +00:00
Eelco Dolstra 560ab22f7d * Ignore packages that don't have a version.
* Work on a manifest instead of a channel directory.
2010-06-23 21:11:33 +00:00
Peter Simons 8b7f8b56f1 Added support for passing an (impure) NIX_BUILD_CORES variable to build expressions.
This patch adds the configuration file variable "build-cores" and the
command line argument "--cores". These settings specify the number of
CPU cores to utilize for parallel building within a job, i.e. by passing
an appropriate "-j" flag to GNU Make. The default value is 1, which
means that parallel building is *disabled*. If the number of build cores
is specified as 0 (synonymously: "guess" or "auto"), then the actual
value is supposed to be auto-detected by builders at run-time, i.e by
calling the nproc(1) utility from coreutils.

The environment variable $NIX_BUILD_CORES is available to builders, but
the contents of that variable does *not* influence the hash that goes
into the $out store path, i.e. the number of build cores to be utilized
can be changed at will without requiring any re-builds.
2010-06-23 14:34:08 +00:00
Eelco Dolstra 87ef5907e9 * Sync. 2010-06-22 14:42:34 +00:00
Eelco Dolstra 819548d92f * Pass `--fallback' to the remote build to ignore failing
substituters.
2010-06-22 14:41:22 +00:00
Eelco Dolstra 3e5e0faf9c * Okay, putting a lock on the temporary directory used by importPath()
doesn't work because the garbage collector doesn't actually look at
  locks.  So r22253 was stupid.  Use addTempRoot() instead.  Also,
  locking the temporary directory in exportPath() was silly because it
  isn't even in the store.
2010-06-21 11:08:09 +00:00
Eelco Dolstra bf87cc44b4 * Sync with the trunk. 2010-06-21 07:55:38 +00:00
Eelco Dolstra b57189174f * In importPath() and exportPath(), lock the temporary directory to
prevent it from being deleted by the garbage collector.
2010-06-14 08:34:48 +00:00
Eelco Dolstra f16fe2af8d * builtins.toXML: propagate the string context. This is a regression
from the old ATerm-based evaluator.
2010-06-10 10:29:50 +00:00
Eelco Dolstra d1f6c0cbe3 * Replacing ValidPath rows doesn't work because it causes a constraint
violation of the Refs table.  So don't do that.
2010-06-08 13:38:28 +00:00
Eelco Dolstra 07ca66cf24 * Applied a patch from David Brown to prevent `nix-store --optimise'
from failing on rename() on BtrFS.
2010-06-04 13:56:11 +00:00
Eelco Dolstra 1ab67cf437 2010-06-02 09:43:04 +00:00
Eelco Dolstra 89865da76d * Turn build errors during evaluation into EvalErrors. 2010-06-01 11:19:32 +00:00
Eelco Dolstra a443c7573b * Hack to allow derivations to disable chroot builds by setting the
attribute "__noChroot = true" (requested by Rob).
2010-06-01 10:01:14 +00:00
Ludovic Courtès 8bcdd36f10 Add XML output to `nix-store'.
* src/nix-store/Makefile.am (nix_store_SOURCES): Add `xmlgraph.cc' and
  `xmlgraph.hh'.

* src/nix-store/help.txt (Operations): Document `--xml'.

* src/nix-store/nix-store.cc (opQuery): Handle `--xml'.

* src/nix-store/xmlgraph.cc, src/nix-store/xmlgraph.hh: New files.
2010-05-31 16:36:24 +00:00
Ludovic Courtès da52f8bea0 Comment out dead code in `nix-store'.
* src/nix-store/dotgraph.cc (pathLabel): Move within #if 0 section.
2010-05-31 16:36:20 +00:00
Eelco Dolstra 7343e6c8ae * Remove an accidentally committed debug statement. 2010-05-30 20:29:56 +00:00
Eelco Dolstra b92a2e5cc2 * Sync with the trunk. 2010-05-20 12:36:03 +00:00
Eelco Dolstra 93cd5a4a13 * The << operator on values should be const. 2010-05-18 10:36:37 +00:00
Eelco Dolstra 32539e41d5 * Sync with trunk. 2010-05-16 17:16:00 +00:00
Eelco Dolstra b2235d81d1 * Restore the __overrides feature that was lost somewhere in the
fast-eval branch.
2010-05-15 08:10:12 +00:00
Eelco Dolstra aa45027818 * Sync with the trunk. 2010-05-12 22:13:09 +00:00
Eelco Dolstra 8032f26ca0 * Merged the `fast-eval' branch. 2010-05-12 13:59:36 +00:00
Eelco Dolstra bd25ac2260 * Print attributes in sorted order. 2010-05-12 12:15:49 +00:00
Eelco Dolstra 81a4b4e49b * Implemented tryEval, the last missing primop in the fast-eval
branch.  Also added a test for tryEval.
2010-05-12 11:23:44 +00:00
Eelco Dolstra 1a8eb6e3ec 2010-05-07 15:26:33 +00:00
Eelco Dolstra 83dfa89870 * Sync with the trunk. 2010-05-07 14:46:47 +00:00
Eelco Dolstra 01e58adce0 * Store position info for inherited attributes. 2010-05-07 12:43:57 +00:00
Eelco Dolstra 83d7b89660 * Updated addErrorContext. 2010-05-07 12:33:14 +00:00
Eelco Dolstra e2d5e40f4f * Keep track of the source positions of attributes. 2010-05-07 12:11:05 +00:00
Eelco Dolstra 84ce7ac76f * Store attribute positions in the AST and report duplicate attribute
errors with position info.
* For all positions, use the position of the first character of the
  first token, rather than the last character of the first token plus
  one.
2010-05-06 16:46:48 +00:00
Ludovic Courtès 4750065ada buildenv: Special-case Python's site.py' and site.pyc'.
* corepkgs/buildenv/builder.pl.in (createLinks): Skip `site.py' and
  `site.pyc' files.
2010-05-05 20:46:41 +00:00
Eelco Dolstra a0e3b84fac * Revert r15436. This was a workaround for a bug in btrfs which seems
to have been fixed now.
2010-05-04 12:42:58 +00:00
Eelco Dolstra f92c9a0ac5 * Allow unprivileged users to do `nix-store --clear-failed-paths' and
`nix-store --query-failed-paths'.
2010-05-04 10:45:10 +00:00
Ludovic Courtès 4bab25a28d buildenv: Special-case Python's `easy-install.pth' files.
* corepkgs/buildenv/builder.pl.in (createLinks): Skip `easy-install.pth'
  files.  Comment the hack.
2010-05-03 13:13:38 +00:00
Eelco Dolstra 7fa338f4ba * Don't use smart quotes where normal ASCII quotes are intended.
Actually, don't use quotes at all.  (Reported by Howard B. Golden.)
2010-05-02 21:27:50 +00:00
Eelco Dolstra c82782f9a5 2010-04-27 09:05:11 +00:00
Eelco Dolstra c778ed1768 * Fix the copy-from-other-stores substituter. 2010-04-26 13:39:55 +00:00
Eelco Dolstra ef337f7089 2010-04-26 12:58:12 +00:00
Eelco Dolstra 6199f9b93e * Added a command `nix-store --clear-failed-paths <PATHS>' to clear
the "failed" status of the given store paths.  The special value `*'
  clears all failed paths.
2010-04-26 12:56:42 +00:00
Eelco Dolstra 2398af13c5 * Add an command `nix-store --query-failed-paths' to list the cached
failed paths (when using the `build-cache-failure' option).
2010-04-26 12:43:42 +00:00
Lluís Batlle i Rossell d77331d32f Fixing a typo in the nix-store manual, that could confuse a bit too much (--delete/--gc) 2010-04-25 20:52:49 +00:00
Eelco Dolstra 2be6118f4c * Don't need the test program anymore. 2010-04-23 09:09:32 +00:00
Eelco Dolstra 0bc468f195 * Simplify the implementation of `with'. This gives a 7% speedup in
evaluating the NixOS system configuration.
2010-04-22 15:08:09 +00:00
Eelco Dolstra ee0384fb96 2010-04-22 11:34:15 +00:00
Eelco Dolstra ebade9ff8b * Check for duplicate attribute names / function arguments. `make
check' now succeeds :-)
* An attribute set such as `{ foo = { enable = true; };
  foo.port = 23; }' now parses.  It was previously rejected, but I'm
  too lazy to implement the check.  (The only reason to reject it is
  that the reverse, `{ foo.port = 23; foo = { enable = true; }; }', is
  rejected, which is kind of ugly.)
2010-04-22 11:02:24 +00:00
Eelco Dolstra 2d7636529f * String equality tests should take the context into account. All the
evaluation test cases now succeed.
2010-04-22 09:54:11 +00:00
Eelco Dolstra 6bbfe95e30 * Don't use an ostringstream in unparseDerivation(), because it's
slow.  A `nix-env -qa --drv-path \*' on Nixpkgs was spending 40% of
  its time in unparseDerivation() because of this (now 11%).
2010-04-21 19:25:50 +00:00
Eelco Dolstra 7148df7971 * Update the expected test output (no longer an ATerm). 2010-04-21 16:22:03 +00:00
Eelco Dolstra 6f0f16497a * Fix the interpretation of ''\<character> in indented strings. 2010-04-21 16:18:27 +00:00
Eelco Dolstra cae4efdca3 * Because --parse-only no longer produces an ATerm, don't check the
output.  Whether it parses at all should be enough.
2010-04-21 16:02:12 +00:00
Eelco Dolstra 0777448ca6 * Fixed builtins.genericClosure. 2010-04-21 15:57:11 +00:00
Eelco Dolstra fe2d869e04 * Store user environment manifests as a Nix expression in
$out/manifest.nix rather than as an ATerm.

  (Hm, I thought I committed this two days ago...)
2010-04-21 15:08:58 +00:00
Eelco Dolstra d66ea83a76 * Sync with the trunk. 2010-04-20 09:20:29 +00:00
Eelco Dolstra f3b8833a48 * Drop the dependency on the ATerm library. 2010-04-19 14:51:58 +00:00
Eelco Dolstra efc7a579e8 * Don't use the ATerm library for parsing/printing .drv files. 2010-04-19 13:46:58 +00:00
Eelco Dolstra 55b5ddd3ca * Added parsing of manifests in ATerm format. 2010-04-19 12:10:04 +00:00
Eelco Dolstra b7ff69eb7c * Refactoring: move the user environment stuff into its own module. 2010-04-19 10:47:56 +00:00
Eelco Dolstra 5c31995bb8 * Updated some more primops. 2010-04-16 15:13:47 +00:00
Eelco Dolstra 8bb0210fea * _combineChannels shouldn't be an integer. 2010-04-16 14:07:52 +00:00
Eelco Dolstra 8ca4a001cb * Improve sharing a bit. 2010-04-16 14:03:26 +00:00
Eelco Dolstra 497e4ad126 * Remove some redundant tests. 2010-04-16 13:51:01 +00:00
Eelco Dolstra 02c1dac909 * In an nested with' where the inner with is a variable (with ...;
with someVar; ...'), the contents of the variable would be
  clobbered.  (The attributes in the outer `with' were added to the
  variable.)
2010-04-16 13:44:02 +00:00
Eelco Dolstra 04c4bd3624 * Store lists as lists of pointers to values rather than as lists of
values.  This improves sharing and gives another speed up.
  Evaluation of the NixOS system attribute is now almost 7 times
  faster than the old evaluator.
2010-04-15 00:37:36 +00:00
Eelco Dolstra e41b5828db * Better stats. 2010-04-14 23:48:46 +00:00
Eelco Dolstra d39d3c6264 * Implemented inherit. 2010-04-14 23:25:05 +00:00
Eelco Dolstra 267dc693d2 * Fix builtins. 2010-04-14 22:59:39 +00:00
Eelco Dolstra 81de12bc8f * Refactoring: move variable uses to a separate class. 2010-04-14 15:14:23 +00:00
Eelco Dolstra 110d155778 * Implemented withs. 2010-04-14 15:01:04 +00:00
Eelco Dolstra 9985230c00 * After parsing, compute level/displacement pairs for each variable
use site, allowing environments to be stores as vectors of values
  rather than maps.  This should speed up evaluation and reduce the
  number of allocations.
2010-04-14 14:42:32 +00:00
Eelco Dolstra 816dd3f061 * Remove more obsolete code. 2010-04-14 12:49:05 +00:00
Eelco Dolstra 011b5da0f4 * Get nix-env to compile again. 2010-04-14 09:39:06 +00:00
Eelco Dolstra 85d13c8f93 * Change the semantics of "with" so that inner "withs" take
precedence, i.e. `with {x=1;}; with {x=2;}; x' evaluates to 2'.
  This has a simpler implementation and seems more natural.  There
  doesn't seem to be any code in Nixpkgs or NixOS that relies on the
  old behaviour.
2010-04-14 08:37:08 +00:00
Eelco Dolstra 816f9c0f6f * Use std::tr1::unordered_set instead of std::set for the symbol
table.  This gives a 10% speed increase on `nix-instantiate
  /etc/nixos/nixos -A system --readonly-mode'.
2010-04-13 14:34:11 +00:00
Eelco Dolstra 7d47498b5e * Evaluate lets directly (i.e. without desugaring to `rec { attrs...;
<let-body> = e; }.<let-body>).  This prevents the unnecessary
  allocation of an attribute set.
2010-04-13 13:42:25 +00:00
Eelco Dolstra ac1e8f40d4 * Use a symbol table to represent identifiers and attribute names
efficiently.  The symbol table ensures that there is only one copy
  of each symbol, thus allowing symbols to be compared efficiently
  using a pointer equality test.
2010-04-13 12:25:42 +00:00
Eelco Dolstra 10e8b1fd15 * Finished the ATerm-less parser. 2010-04-12 23:33:23 +00:00
Eelco Dolstra 0d272fca79 * Remove some obsolete functions. 2010-04-12 23:31:47 +00:00
Eelco Dolstra d4f0b0fc6c * Indented strings. 2010-04-12 22:03:27 +00:00
Eelco Dolstra a60317f20f * More missing constructs. 2010-04-12 21:21:24 +00:00
Eelco Dolstra 4d6ad5be17 * Don't use ATerms for the abstract syntax trees anymore. Not
finished yet.
2010-04-12 18:30:11 +00:00
Eelco Dolstra ed711f73bc * Don't use ATerms to represent integers in the lexer. 2010-04-12 10:38:18 +00:00
Eelco Dolstra db90b88e65 * Hack to support builderDefs expressions. 2010-04-12 09:50:20 +00:00
Eelco Dolstra 4e49002576 * Doh. 2010-04-12 09:45:00 +00:00
Eelco Dolstra c3f228f296 2010-04-12 09:14:27 +00:00
Ludovic Courtès aac5fcfbb5 Re-add drvPath' and outPath' attributes to <derivation> XML nodes.
This fixes a regression introduced in r20882 ("Add source location
information to the XML output.").

* src/libexpr/expr-to-xml.cc (nix::printTermAsXML): Dereference the
  attribute RHS from "drvPath" and "outPath".
2010-04-09 21:30:55 +00:00
Eelco Dolstra f3dc7ab877 * Keep more statistics about stack space usage.
* Reduce stack space usage.
2010-04-09 12:00:49 +00:00
Eelco Dolstra b7b3dd55f9 * Remove a lot of dead code. 2010-04-08 11:41:19 +00:00
Eelco Dolstra 7e048eddf5 * Fix blackholing. If evaluation fails due to an assertion failure,
then the blackhole has to be removed to ensure that repeated
  evaluation of the same value gives an assertion failure again rather
  than an "infinite recursion" error.
2010-04-08 11:25:14 +00:00
Eelco Dolstra af2a372bb0 * Update autoCallFunction() and findAlongAttrPath(). 2010-04-07 15:47:06 +00:00
Eelco Dolstra 9a64454faa * expr-to-xml -> value-to-xml. 2010-04-07 13:59:45 +00:00
Eelco Dolstra fc92244ba8 * Implemented the primops necessary for generating the NixOS manual. 2010-04-07 13:55:46 +00:00
Eelco Dolstra a353aef0b1 * In eval(), don't use the target value `v' as a temporary.
Overwriting `v' breaks when the expression evaluation to an
  assertion failure or throw.
2010-04-06 14:15:29 +00:00
Eelco Dolstra a5ece7d016 * Removed the `~' operator. 2010-04-01 16:59:07 +00:00
Eelco Dolstra c172274e17 * Quick hack to make coerceToString work more or less correctly on
nested lists.  `nix-instantiate' can now evaluate the NixOS system
  derivation attribute correctly (in 2.1s on my laptop vs. 6.2s for
  the trunk).
2010-04-01 14:35:03 +00:00
Eelco Dolstra 7b851915bf * Improve sharing. 2010-04-01 12:04:57 +00:00
Eelco Dolstra 95cc417d76 * Functions are incomparable. 2010-04-01 10:55:36 +00:00
Eelco Dolstra 71f026292b * Make `derivation' lazy again for performance. It also turns out
that there are some places in Nixpkgs (php_configurable /
  composableDerivation, it seems) that call `derivation' with
  incorrect arguments (namely, the `name' attribute missing) but get
  away with it because of laziness.
2010-04-01 09:55:57 +00:00
Eelco Dolstra dc31305b38 * Fixed the trace primop and path comparison.
* Removed exprToString and stringToExpr because there is no ATerm
  representation to work on anymore (and exposing the internals of the
  evaluator like this is not a good idea anyway).
2010-03-31 20:09:20 +00:00
Eelco Dolstra 979f163615 * Handle string contexts. `nix-instantiate' can now correctly compute
the `firefoxWrapper' attribute in Nixpkgs, and it's about 3 times
  faster than the trunk :-)
2010-03-31 19:52:29 +00:00
Eelco Dolstra d8cd3115d8 * Get nix-env to compile. 2010-03-31 19:12:08 +00:00
Eelco Dolstra 55e207b2dc * Cache parse trees to prevent repeated parsing of imported Nix
expressions.
2010-03-31 16:14:32 +00:00
Eelco Dolstra 3d94be61ea * Implemented derivations. 2010-03-31 15:38:03 +00:00
Eelco Dolstra 5187678913 2010-03-31 15:14:23 +00:00
Eelco Dolstra f061086a93 * Fix the broken test for listToAttrs. 2010-03-31 13:35:29 +00:00
Ludovic Courtès 09381cccff Make source location info in the XML output optional.
* src/libexpr/expr-to-xml.cc (nix::showAttrs): Add `location'
  parameter.  Provide location XML attributes when it's true.  Update
  callers.
  (nix::printTermAsXML): Likewise.

* src/libexpr/expr-to-xml.hh (nix::printTermAsXML): Update prototype;
  have `location' default to `false'.

* src/nix-instantiate/nix-instantiate.cc (printResult, processExpr): Add
  `location' parameter; update callers.
  (run): Add support for `--no-location'.

* src/nix-instantiate/help.txt: Update accordingly.

* tests/lang.sh: Invoke `nix-instantiate' with `--no-location' for the
  XML tests.

* tests/lang/eval-okay-toxml.exp, tests/lang/eval-okay-to-xml.nix: New
  files.
2010-03-31 12:38:31 +00:00
Eelco Dolstra 13c2adc897 * Implemented `rec { inherit ...; }'. 2010-03-31 11:05:39 +00:00
Eelco Dolstra 4c53ca2692 * Compare nulls. 2010-03-31 09:54:12 +00:00
Ludovic Courtès 471419d1fa Add source location information to the XML output.
* src/libexpr/expr-to-xml.cc (nix::showAttrs): Dereference the attribute
  RHS.  Add "path", "line", and "column" XML attributes to the node when
  source location information is available.
  (nix::printTermAsXML): Likewise for functions.
2010-03-31 08:29:05 +00:00
Ludovic Courtès eb07a4f1ee Escape `>' signs in the XML output.
* src/libutil/xml-writer.cc (nix::XMLWriter::writeAttrs): Escape `>'.
2010-03-31 08:29:01 +00:00
Eelco Dolstra 7f19e03c65 * More primops. 2010-03-30 22:39:48 +00:00
Eelco Dolstra 47df476daa * More operators / primops. 2010-03-30 18:05:54 +00:00
Eelco Dolstra c9170be2bd * More primops. 2010-03-30 15:18:20 +00:00
Eelco Dolstra c3aa615a5f * More primops. 2010-03-30 14:39:27 +00:00
Eelco Dolstra 5b72d8a749 * Implemented `map'. 2010-03-30 13:47:59 +00:00
Eelco Dolstra d78a05ab40 * Make `import' work. 2010-03-30 09:22:33 +00:00
Eelco Dolstra 31428c3a06 * Started integrating the new evaluator. 2010-03-29 14:37:56 +00:00
Eelco Dolstra 52090d2418 2010-03-29 10:13:51 +00:00
Eelco Dolstra e3f32ac5af 2010-03-29 09:43:55 +00:00
Eelco Dolstra 807a67bc74 2010-03-29 09:43:39 +00:00
Eelco Dolstra 392811eb8f * Strings. 2010-03-28 18:27:07 +00:00
Eelco Dolstra d96cdcea6b 2010-03-28 16:57:16 +00:00
Eelco Dolstra 3d2b835f30 * Implemented multi-argument primops. 2010-03-28 16:37:39 +00:00
Eelco Dolstra 45d822f29c * Primops (not yet finished). 2010-03-26 15:45:53 +00:00
Eelco Dolstra cad8726b2c * Implemented the ==' and !=' operators. These now use a deep
equality test, so they also work for (finite) attribute sets and
  lists.
2010-03-26 13:27:26 +00:00
Eelco Dolstra 8da118e4d0 * Measure stack usage. 2010-03-25 16:35:24 +00:00
Eelco Dolstra c2ba4313fb * Implemented lists. 2010-03-25 15:38:37 +00:00
Eelco Dolstra 25eedf085d * Quick and dirty implementation of with'. with e1; e2' is
basically desugared to `let <with> = e1; e2', and `lookupVar' looks
  in each <with> in the environment chain for an attribute with the
  specified name.
2010-03-25 14:51:04 +00:00
Eelco Dolstra 3c9f8fc9b6 * Don't convert variable names to strings. 2010-03-25 13:10:04 +00:00
Eelco Dolstra f450384ded * Implement blackholing. 2010-03-25 12:51:14 +00:00
Eelco Dolstra ef8bd919fc * Implement `...' and default function arguments. 2010-03-25 12:45:23 +00:00
Eelco Dolstra 8a10360c91 * Simplify @-patterns: only {attrs}@name' or name@{attrs}' are now
allowed.  So `name1@name2', `{attrs1}@{attrs2}' and so on are now no
  longer legal.  This is no big loss because they were not useful
  anyway.

  This also changes the output of builtins.toXML for @-patterns
  slightly.
2010-03-25 12:19:41 +00:00
Eelco Dolstra 7482349fe8 * Implemented attribute set pattern matches. 2010-03-24 23:40:00 +00:00
Eelco Dolstra 0fd3648d34 * Store values in environments. 2010-03-24 12:41:08 +00:00
Eelco Dolstra b70bd8fe56 * Reduce the number of value allocations in eval() by moving
responsibility for allocation of the result to the caller.
2010-03-24 12:11:38 +00:00
Eelco Dolstra d31c59eb17 * Plain lambdas. 2010-03-24 11:06:05 +00:00
Eelco Dolstra e8f7978274 2010-03-23 19:19:52 +00:00
Eelco Dolstra 0910ae9568 * Start of an evaluator that uses call-by-need (with thunk updating)
instead of (memoised) call-by-name.
2010-03-23 17:30:50 +00:00
Eelco Dolstra 90039e0863 * Branch for experimenting with faster / less memory-hungry
Nix expression evaluation.
2010-03-23 15:09:29 +00:00
Eelco Dolstra 71be50cc25 * Doh. 2010-03-23 14:51:32 +00:00
Eelco Dolstra 3bfd3a4e95 * Test "with as; with bs;" since nobody knows what its semantics is. 2010-03-23 14:26:27 +00:00
Eelco Dolstra f71ea9c911 * Use SQLite 3.6.23. 2010-03-23 10:38:59 +00:00
Eelco Dolstra 141294ff38 * Clean up error messages in killUser(). 2010-03-19 11:36:34 +00:00
Eelco Dolstra 74299c1cfb * Bump version number. 2010-03-17 12:12:45 +00:00
Eelco Dolstra e020d80e4e * Sync with the trunk. 2010-03-11 15:45:05 +00:00
Eelco Dolstra 070057c1b9 * Sync with the trunk. 2010-03-11 10:52:52 +00:00
Eelco Dolstra 03afc34805 * Simplify. 2010-03-10 13:07:37 +00:00
Eelco Dolstra 1a65142ec4 * Remove a debug statement. 2010-03-10 12:46:25 +00:00
Eelco Dolstra 4c356acd04 * In `nix-store --export', abort if the contents of a path has
changed.  This prevents corrupt paths from spreading to other
  machines.  Note that checking the hash is cheap because we're
  hashing anyway (because of the --sign feature).
2010-03-09 14:32:03 +00:00
Eelco Dolstra 44f6e6de77 * Set gc-keep-outputs' or gc-keep-derivations' to false with
`--delete --ignore-liveness'.
2010-03-08 21:31:42 +00:00
Eelco Dolstra 2e4ef03aa3 * Increase the sqlite timeout. 2010-03-08 10:35:45 +00:00
Eelco Dolstra 04791840f4 * Emit warning='1' or error='1' attributes for lines marked as
warnings or errors with \e[w or \e[e.
2010-03-05 12:54:58 +00:00
Eelco Dolstra bc6f7fc139 * Remove some unused functions. 2010-03-04 13:03:26 +00:00
Eelco Dolstra fb6e223ddc * Synced with the trunk. 2010-03-04 12:15:00 +00:00
Eelco Dolstra 5ff87c982e 2010-03-03 16:17:09 +00:00
Eelco Dolstra e14e2399ed * Prevent a potential memory corruption problem if an ATerm garbage
collection happens during fixAttrs().
2010-03-02 20:23:42 +00:00
Eelco Dolstra 158aa89317 * Ensure that make install' works without a prior make'. 2010-03-02 20:22:46 +00:00
Eelco Dolstra 56af8e86e3 * Protect the true', false' and `__overrides' constants. Without
an ATprotect call, these could be garbage collected, leading to
  weird crashes or wrong results.
2010-03-02 20:09:12 +00:00
Eelco Dolstra 3f9e647ae8 * checkInterrupt() shouldn't be called from a destructor. 2010-03-02 19:04:17 +00:00
Eelco Dolstra d8c5745c41 * Support static builds. 2010-03-02 17:12:47 +00:00
Eelco Dolstra e07d7284a2 * Fix DESTDIR builds. 2010-03-02 16:55:07 +00:00
Eelco Dolstra 5414b3b2db * Remove the --enable-static-nix flag. 2010-03-02 16:40:40 +00:00
Eelco Dolstra 594eaddd11 * When using the included sqlite/aterm libraries, build with
--enable-shared.
* In libutil/libstore/libexpr etc., link against sqlite and aterm.
* Some more header file hygiene.
2010-03-02 15:58:13 +00:00
Eelco Dolstra 966ffb29a7 * Update the Valgrind suppressions for ATerm 2.5. 2010-02-26 13:10:57 +00:00
Eelco Dolstra 24035b98b1 * Implement RemoteStore::queryValidPaths(). 2010-02-26 12:05:01 +00:00
Eelco Dolstra e42401ee7b * Implement RemoteStore::queryDerivationOutputs(). 2010-02-25 15:52:22 +00:00
Eelco Dolstra af565c348a * Support read-only access to the database. 2010-02-24 16:44:43 +00:00
Eelco Dolstra e33f67ff0b * Refactor the upgrade / database initialisation logic a bit. 2010-02-24 16:30:20 +00:00
Eelco Dolstra cfe742cfc5 * A function to query just the database id of a valid path. 2010-02-24 15:07:23 +00:00
Eelco Dolstra 6baa2a2f5e 2010-02-24 14:39:52 +00:00
Eelco Dolstra 9fd85c94de * Use `truncate' journal mode, which should be a bit faster. 2010-02-24 14:22:34 +00:00
Eelco Dolstra fefd467539 * `helpText' is now zero-terminated. 2010-02-24 13:24:27 +00:00
Eelco Dolstra 21b134b4e5 2010-02-24 13:13:39 +00:00
Eelco Dolstra a3c63d0d6c * Disable fsync() in SQLite if the fsync-metadata option is set to
false.
* Change the default for `fsync-metadata' to true.
* Disable `fsync-metadata' in `make check'.
2010-02-24 13:12:57 +00:00
Eelco Dolstra 90b6352d0a * Do registerValidPaths() in one transaction, which is much faster.
E.g. it cuts the runtime of the referrers test from 50s to 23s.
2010-02-24 12:48:00 +00:00
Eelco Dolstra fae0427324 * ADDITIONAL_NETWORK_LIBS -> LIBS. 2010-02-24 12:25:48 +00:00
Eelco Dolstra fa6a4fcb11 * Add ${sqlite_lib} everywhere. Just adding it in `libstore' doesn't
work on x86_64 when sqlite is compiled statically.
2010-02-24 12:18:48 +00:00
Eelco Dolstra 5954eadf67 * Remove the fdatasync check since it's no longer needed. 2010-02-24 12:16:50 +00:00
Eelco Dolstra bb82310dba * Set the path to sqlite3 properly. 2010-02-24 11:06:17 +00:00
Eelco Dolstra 69d9df7fe6 * Don't fork so much. 2010-02-24 11:04:03 +00:00
Eelco Dolstra 462bd50aef * Use normal (rather than full) synchronous mode, which I gather from
the description at http://www.sqlite.org/atomiccommit.html should be
  safe enough.
2010-02-24 10:57:57 +00:00
Eelco Dolstra 8520de4720 * Include the sqlite tarball in the Nix tarball. 2010-02-24 10:48:57 +00:00
Eelco Dolstra dc6d1ec67e 2010-02-23 23:24:49 +00:00
Eelco Dolstra 63b09c5e41 2010-02-23 22:31:38 +00:00
Eelco Dolstra 0efc986ba1 * Sync with the trunk. 2010-02-23 22:14:33 +00:00
Eelco Dolstra b4e6d98fc3 * configure: flag --with-sqlite. 2010-02-23 22:12:46 +00:00
Eelco Dolstra 2b20318b0e 2010-02-22 14:24:37 +00:00
Eelco Dolstra 9cda616949 * The database needs a trigger to get rid of self-references to
prevent a foreign key constraint violation on the Refs table when
  deleting a path.
2010-02-22 14:18:55 +00:00
Eelco Dolstra c4d388add4 * Get derivation outputs from the database instead of the .drv file,
which requires more I/O.
2010-02-22 12:44:36 +00:00
Eelco Dolstra 103cfee056 * Revert r19650 (implement gc-keep-outputs by looking for derivations
with the same name as the output) and instead use the
  DerivationOutputs table in the database, which is the correct way to
  to do things.
2010-02-22 11:44:17 +00:00
Eelco Dolstra 299ff64812 * Put the derivation outputs in the database. This is useful for the
garbage collector.
2010-02-22 11:15:50 +00:00
Eelco Dolstra 1930570ad9 * Foreign key support in SQLite is not a persistent setting, so enable
it at startup.
* Implement negative caching.  Now `make check' passes.
2010-02-19 17:15:22 +00:00
Eelco Dolstra 9c9a88e9e2 * Implement more stuff. 2010-02-19 16:43:25 +00:00
Eelco Dolstra 762cee72cc * Implement registerValidPath(). 2010-02-19 16:04:51 +00:00
Eelco Dolstra 268f9aaf28 * Implemented queryValidPaths() and verifyStore(). 2010-02-18 16:51:27 +00:00
Eelco Dolstra 836e5b6f57 * Implemented queryReferrers(). 2010-02-18 16:21:59 +00:00
Eelco Dolstra 77cb9e3fb1 * Implement queryPathInfo(). 2010-02-18 15:52:57 +00:00
Eelco Dolstra 885e22b16e * Implement isValidPath(). 2010-02-18 15:11:08 +00:00
Eelco Dolstra cfb09e0fad * Automatically abort transactions if they go out of scope without
committing.
2010-02-18 14:40:07 +00:00
Eelco Dolstra e0305bb7a8 * Some wrapper objects to ensure that SQLite objects are properly
destroyed.
2010-02-18 14:30:42 +00:00
Eelco Dolstra a053d2d8e5 * Add the deriver to the ValidPaths table. In principle we could now
store all the derivers of a path efficiently.  But that opens a big
  can of worms with respect to garbage collector semantics.
2010-02-18 13:48:18 +00:00
Eelco Dolstra dbddac0fe9 * Assign an integer id to every row in the ValidPaths table in order
to make the Refs table more space-efficient.  For instance, this
  reduces the size of the database on my laptop from 93 MiB to 18
  MiB.  (It was 72 MiB with the old schema on an ext3 disk with a 1
  KiB block size.)
2010-02-18 13:40:46 +00:00
Eelco Dolstra c1a07f9445 * Convert the Nix database to SQLite. 2010-02-18 13:16:59 +00:00
Eelco Dolstra eaaa13ce47 * Experimental branch for using SQLite to store the Nix database. 2010-02-18 13:14:51 +00:00
310 changed files with 10110 additions and 8530 deletions

No files matched your search

+10 -17
View File
@@ -61,23 +61,11 @@
# /externals/
/externals/Makefile
/externals/Makefile.in
/externals/aterm-*
/externals/have-aterm
/externals/build-aterm
/externals/inst-aterm
/externals/bzip2-*
/externals/have-bzip2
/externals/build-bzip2
/externals/inst-bzip2
# /make/examples/aterm/
/make/examples/aterm/result*
# /make/examples/aterm/aterm/
/make/examples/aterm/aterm/*
# /make/examples/aterm/test/
/make/examples/aterm/test/*
/externals/sqlite-*
/externals/build-sqlite
# /misc/
/misc/Makefile.in
@@ -100,13 +88,16 @@
/scripts/nix-channel
/scripts/nix-build
/scripts/nix-copy-closure
/scripts/readmanifest.pm
/scripts/readconfig.pm
/scripts/nix-generate-patches
/scripts/NixConfig.pm
/scripts/NixManifest.pm
/scripts/GeneratePatches.pm
/scripts/download-using-manifests.pl
/scripts/copy-from-other-stores.pl
/scripts/generate-patches.pl
/scripts/find-runtime-roots.pl
/scripts/build-remote.pl
/scripts/nix-reduce-build
/scripts/nix-http-export.cgi
# /src/
/src/Makefile
@@ -168,6 +159,7 @@
/src/libstore/derivations-ast.cc
/src/libstore/derivations-ast.hh
/src/libstore/.libs
/src/libstore/schema.sql.hh
# /src/libutil/
/src/libutil/Makefile
@@ -242,6 +234,7 @@
/tests/config.nix
/tests/common.sh
/tests/dummy
/tests/result*
# /tests/lang/
/tests/lang/*.out
+4 -2
View File
@@ -1,7 +1,9 @@
SUBDIRS = externals src scripts corepkgs doc misc tests
SUBDIRS = externals src perl scripts corepkgs doc misc tests
EXTRA_DIST = substitute.mk nix.spec nix.spec.in bootstrap.sh \
nix.conf.example NEWS version
pkginclude_HEADERS = config.h
include ./substitute.mk
nix.spec: nix.spec.in
@@ -31,7 +33,7 @@ init-state:
$(INSTALL) $(INIT_FLAGS) -d $(DESTDIR)$(localstatedir)/nix/temproots
ln -sfn $(localstatedir)/nix/profiles $(DESTDIR)$(localstatedir)/nix/gcroots/profiles
$(INSTALL) $(INIT_FLAGS) -d $(DESTDIR)$(localstatedir)/nix/userpool
-$(INSTALL) $(INIT_FLAGS) -m 1777 -d $(DESTDIR)$(storedir)
-$(INSTALL) $(INIT_FLAGS) -d $(DESTDIR)$(storedir)
$(INSTALL) $(INIT_FLAGS) $(GROUP_WRITABLE) -d $(DESTDIR)$(localstatedir)/nix/manifests
ln -sfn $(localstatedir)/nix/manifests $(DESTDIR)$(localstatedir)/nix/gcroots/manifests
-117
View File
@@ -1,117 +0,0 @@
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:*
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value8
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:*
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value8
fun:*
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Addr4
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Addr8
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value8
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Addr4
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Addr8
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:*
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value8
fun:*
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:*
fun:*
fun:AT_collect
}
-252
View File
@@ -1,252 +0,0 @@
#! /usr/bin/perl -w -I /home/eelco/.nix-profile/lib/site_perl
use strict;
use XML::LibXML;
#use XML::Simple;
my $blacklistFN = shift @ARGV;
die unless defined $blacklistFN;
my $userEnv = shift @ARGV;
die unless defined $userEnv;
# Read the blacklist.
my $parser = XML::LibXML->new();
my $blacklist = $parser->parse_file($blacklistFN)->getDocumentElement;
#print $blacklist->toString() , "\n";
# Get all the elements of the user environment.
my $userEnvElems = `nix-store --query --references '$userEnv'`;
die "cannot query user environment elements" if $? != 0;
my @userEnvElems = split ' ', $userEnvElems;
my %storePathHashes;
sub getElemNodes {
my $node = shift;
my @elems = ();
foreach my $node ($node->getChildNodes) {
push @elems, $node if $node->nodeType == XML_ELEMENT_NODE;
}
return @elems;
}
my %referencesCache;
sub getReferences {
my $path = shift;
return $referencesCache{$path} if defined $referencesCache{$path};
my $references = `nix-store --query --references '$path'`;
die "cannot query references" if $? != 0;
$referencesCache{$path} = [split ' ', $references];
return $referencesCache{$path};
}
my %attrsCache;
sub getAttr {
my $path = shift;
my $name = shift;
my $key = "$path/$name";
return $referencesCache{$key} if defined $referencesCache{$key};
my $value = `nix-store --query --binding '$name' '$path' 2> /dev/null`;
$value = "" if $? != 0; # !!!
chomp $value;
$referencesCache{$key} = $value;
return $value;
}
sub evalCondition;
sub traverse {
my $done = shift;
my $set = shift;
my $path = shift;
my $stopCondition = shift;
return if defined $done->{$path};
$done->{$path} = 1;
$set->{$path} = 1;
# print " in $path\n";
if (!evalCondition({$path => 1}, $stopCondition)) {
# print " STOPPING in $path\n";
return;
}
# Get the requisites of the deriver.
foreach my $reference (@{getReferences $path}) {
traverse($done, $set, $reference, $stopCondition);
}
}
sub evalSet {
my $inSet = shift;
my $expr = shift;
my $name = $expr->getName;
if ($name eq "traverse") {
my $stopCondition = (getElemNodes $expr)[0];
my $done = { };
my $set = { };
foreach my $path (keys %{$inSet}) {
traverse($done, $set, $path, $stopCondition);
}
return $set;
}
else {
die "unknown element `$name'";
}
}
# Function for evaluating conditions.
sub evalCondition {
my $storePaths = shift;
my $condition = shift;
my $elemName = $condition->getName;
if ($elemName eq "containsSource") {
my $hash = $condition->attributes->getNamedItem("hash")->getValue;
foreach my $path (keys %{$storePathHashes{$hash}}) {
return 1 if defined $storePaths->{$path};
}
return 0;
}
elsif ($elemName eq "hasName") {
my $nameRE = $condition->attributes->getNamedItem("name")->getValue;
foreach my $path (keys %{$storePaths}) {
return 1 if $path =~ /$nameRE/;
}
return 0;
}
elsif ($elemName eq "hasAttr") {
my $name = $condition->attributes->getNamedItem("name")->getValue;
my $valueRE = $condition->attributes->getNamedItem("value")->getValue;
foreach my $path (keys %{$storePaths}) {
if ($path =~ /\.drv$/) {
my $value = getAttr($path, $name);
# print " $path $name $value\n";
return 1 if $value =~ /$valueRE/;
}
}
return 0;
}
elsif ($elemName eq "and") {
my $result = 1;
foreach my $node (getElemNodes $condition) {
$result &= evalCondition($storePaths, $node);
}
return $result;
}
elsif ($elemName eq "not") {
return !evalCondition($storePaths, (getElemNodes $condition)[0]);
}
elsif ($elemName eq "within") {
my @elems = getElemNodes $condition;
my $set = evalSet($storePaths, $elems[0]);
return evalCondition($set, $elems[1]);
}
elsif ($elemName eq "true") {
return 1;
}
elsif ($elemName eq "false") {
return 0;
}
else {
die "unknown element `$elemName'";
}
}
sub evalOr {
my $storePaths = shift;
my $nodes = shift;
my $result = 0;
foreach my $node (@{$nodes}) {
$result |= evalCondition($storePaths, $node);
}
return $result;
}
# Iterate over all elements, check them.
foreach my $userEnvElem (@userEnvElems) {
# Get the deriver of this path.
my $deriver = `nix-store --query --deriver '$userEnvElem'`;
die "cannot query deriver" if $? != 0;
chomp $deriver;
if ($deriver eq "unknown-deriver") {
# print " deriver unknown, cannot check sources\n";
next;
}
print "CHECKING $userEnvElem\n";
# Get the requisites of the deriver.
# my $requisites = `nix-store --query --requisites --include-outputs '$deriver'`;
# die "cannot query requisites" if $? != 0;
# my @requisites = split ' ', $requisites;
# Get the hashes of the requisites.
# my $hashes = `nix-store --query --hash @requisites`;
# die "cannot query hashes" if $? != 0;
# my @hashes = split ' ', $hashes;
# for (my $i = 0; $i < scalar @requisites; $i++) {
# die unless $i < scalar @hashes;
# my $hash = $hashes[$i];
# $storePathHashes{$hash} = {} unless defined $storePathHashes{$hash};
# my $r = $storePathHashes{$hash}; # !!! fix
# $$r{$requisites[$i]} = 1;
# }
# Evaluate each blacklist item.
foreach my $item ($blacklist->getChildrenByTagName("item")) {
my $itemId = $item->getAttributeNode("id")->getValue;
# print " CHECKING FOR $itemId\n";
my $condition = ($item->getChildrenByTagName("condition"))[0];
die unless $condition;
# Evaluate the condition.
my @elems = getElemNodes $condition;
if (evalOr({$deriver => 1}, \@elems)) {
# Oops, condition triggered.
my $reason = ($item->getChildrenByTagName("reason"))[0]->getChildNodes->to_literal;
$reason =~ s/\s+/ /g;
$reason =~ s/^\s+//g;
print " VULNERABLE TO `$itemId': $reason\n";
}
}
}
+2 -5
View File
@@ -1,7 +1,4 @@
#! /bin/sh -e
rm -f aclocal.m4
mkdir -p config
libtoolize --copy
aclocal
autoheader
automake --add-missing --copy
autoconf
exec autoreconf -vfi
+113 -80
View File
@@ -5,30 +5,33 @@ AM_INIT_AUTOMAKE([dist-bzip2 foreign])
AC_DEFINE_UNQUOTED(NIX_VERSION, ["$VERSION"], [Nix version.])
AC_CANONICAL_HOST
AC_PROG_SED
# Construct a Nix system name (like "i686-linux").
AC_CANONICAL_HOST
AC_MSG_CHECKING([for the canonical Nix system name])
cpu_name=$(uname -p | tr 'A-Z ' 'a-z_')
machine_name=$(uname -m | tr 'A-Z ' 'a-z_')
case $machine_name in
i*86)
machine_name=i686
;;
x86_64)
machine_name=x86_64
;;
ppc)
machine_name=powerpc
;;
*)
if test "$cpu_name" != "unknown"; then
machine_name=$cpu_name
fi
;;
esac
AC_ARG_WITH(system, AC_HELP_STRING([--with-system=SYSTEM],
[Platform identifier (e.g., `i686-linux').]),
[system=$withval],
[case "$host_cpu" in
i*86)
machine_name="i686";;
amd64)
machine_name="x86_64";;
*)
machine_name="$host_cpu";;
esac
case "$host_os" in
linux-gnu*)
# For backward compatibility, strip the `-gnu' part.
system="$machine_name-linux";;
*)
# Strip the version number from names such as `gnu0.3',
# `darwin10.2.0', etc.
system="$machine_name-`echo $host_os | "$SED" -e's/@<:@0-9.@:>@*$//g'`";;
esac])
sys_name=$(uname -s | tr 'A-Z ' 'a-z_')
@@ -37,10 +40,7 @@ case $sys_name in
sys_name=cygwin
;;
esac
AC_ARG_WITH(system, AC_HELP_STRING([--with-system=SYSTEM],
[Platform identifier (e.g., `i686-linux').]),
system=$withval, system="${machine_name}-${sys_name}")
AC_MSG_RESULT($system)
AC_SUBST(system)
AC_DEFINE_UNQUOTED(SYSTEM, ["$system"], [platform identifier (`cpu-os')])
@@ -50,38 +50,23 @@ AC_DEFINE_UNQUOTED(SYSTEM, ["$system"], [platform identifier (`cpu-os')])
test "$localstatedir" = '${prefix}/var' && localstatedir=/nix/var
# Whether to produce a statically linked binary. On Cygwin, this is
# the default: dynamically linking against the ATerm DLL does work,
# except that it requires the ATerm "lib" directory to be in $PATH, as
# Windows doesn't have anything like an RPATH embedded in executable.
# Since this is kind of annoying, we use static libraries for now.
AC_ARG_ENABLE(static-nix, AC_HELP_STRING([--enable-static-nix],
[produce statically linked binaries]),
static_nix=$enableval, static_nix=no)
if test "$sys_name" = cygwin; then
static_nix=yes
fi
if test "$static_nix" = yes; then
# Windows-specific stuff. On Cygwin, dynamically linking against the
# ATerm DLL works, except that it requires the ATerm "lib" directory
# to be in $PATH, as Windows doesn't have anything like an RPATH
# embedded in executable. Since this is kind of annoying, we use
# static libraries for now.
if test "$sys_name" = "cygwin"; then
AC_DISABLE_SHARED
AC_ENABLE_STATIC
fi
# Windows-specific stuff.
if test "$sys_name" = "cygwin"; then
# We cannot delete open files.
AC_DEFINE(CANNOT_DELETE_OPEN_FILES, 1, [Whether it is impossible to delete open files.])
# Solaris-specific stuff.
if test "$sys_name" = sunos; then
# Solaris requires -lsocket -lnsl for network functions
LIBS="-lsocket -lnsl $LIBS"
fi
# Solaris-specific stuff.
if test "$sys_name" = "sunos"; then
# Solaris requires -lsocket -lnsl for network functions
ADDITIONAL_NETWORK_LIBS="-lsocket -lnsl"
AC_SUBST(ADDITIONAL_NETWORK_LIBS)
fi
AC_PROG_CC
AC_PROG_CXX
@@ -101,6 +86,13 @@ AC_DISABLE_STATIC
AC_ENABLE_SHARED
AC_PROG_LIBTOOL
if test "$enable_shared" = yes; then
SUB_CONFIGURE_FLAGS="--enable-shared --disable-static"
else
SUB_CONFIGURE_FLAGS="--enable-static --disable-shared"
fi
AC_SUBST(SUB_CONFIGURE_FLAGS)
# Use 64-bit file system calls so that we can support files > 2 GiB.
AC_SYS_LARGEFILE
@@ -136,11 +128,22 @@ AC_CHECK_HEADERS([locale], [], [], [])
AC_LANG_POP(C++)
# Check for <err.h>.
AC_CHECK_HEADER([err.h], [], [bsddiff_compat_include="-Icompat-include"])
AC_SUBST([bsddiff_compat_include])
# Check whether we have the personality() syscall, which allows us to
# do i686-linux builds on x86_64-linux machines.
AC_CHECK_HEADERS([sys/personality.h])
# Check for tr1/unordered_set.
AC_LANG_PUSH(C++)
AC_CHECK_HEADERS([tr1/unordered_set], [], [], [])
AC_LANG_POP(C++)
AC_DEFUN([NEED_PROG],
[
AC_PATH_PROG($1, $2)
@@ -168,6 +171,7 @@ AC_PATH_PROG(openssl_prog, openssl, openssl) # if not found, call openssl in $PA
AC_SUBST(openssl_prog)
AC_DEFINE_UNQUOTED(OPENSSL_PATH, ["$openssl_prog"], [Path of the OpenSSL binary])
# Test that Perl has the open/fork feature (Perl 5.8.0 and beyond).
AC_MSG_CHECKING([whether Perl is recent enough])
if ! $perl -e 'open(FOO, "-|", "true"); while (<FOO>) { print; }; close FOO or die;'; then
@@ -176,6 +180,7 @@ if ! $perl -e 'open(FOO, "-|", "true"); while (<FOO>) { print; }; close FOO or d
fi
AC_MSG_RESULT(yes)
NEED_PROG(cat, cat)
NEED_PROG(tr, tr)
AC_ARG_WITH(coreutils-bin, AC_HELP_STRING([--with-coreutils-bin=PATH],
@@ -183,6 +188,7 @@ AC_ARG_WITH(coreutils-bin, AC_HELP_STRING([--with-coreutils-bin=PATH],
coreutils=$withval, coreutils=$(dirname $cat))
AC_SUBST(coreutils)
AC_ARG_WITH(docbook-rng, AC_HELP_STRING([--with-docbook-rng=PATH],
[path of the DocBook RelaxNG schema]),
docbookrng=$withval, docbookrng=/docbook-rng-missing)
@@ -193,32 +199,18 @@ AC_ARG_WITH(docbook-xsl, AC_HELP_STRING([--with-docbook-xsl=PATH],
docbookxsl=$withval, docbookxsl=/docbook-xsl-missing)
AC_SUBST(docbookxsl)
AC_ARG_WITH(xml-flags, AC_HELP_STRING([--with-xml-flags=FLAGS],
[extra flags to be passed to xmllint and xsltproc]),
xmlflags=$withval, xmlflags=)
AC_SUBST(xmlflags)
AC_ARG_WITH(store-dir, AC_HELP_STRING([--with-store-dir=PATH],
[path of the Nix store]),
[path of the Nix store (defaults to /nix/store)]),
storedir=$withval, storedir='/nix/store')
AC_SUBST(storedir)
AC_ARG_WITH(aterm, AC_HELP_STRING([--with-aterm=PATH],
[prefix of CWI ATerm library]),
aterm=$withval, aterm=)
AM_CONDITIONAL(HAVE_ATERM, test -n "$aterm")
if test -z "$aterm"; then
aterm_lib='-L${top_builddir}/externals/inst-aterm/lib -lATerm'
aterm_include='-I${top_builddir}/externals/inst-aterm/include'
aterm_bin='${top_builddir}/externals/inst-aterm/bin'
else
aterm_lib="-L$aterm/lib -lATerm"
aterm_include="-I$aterm/include"
aterm_bin="$aterm/bin"
fi
AC_SUBST(aterm_lib)
AC_SUBST(aterm_include)
AC_SUBST(aterm_bin)
AC_ARG_WITH(openssl, AC_HELP_STRING([--with-openssl=PATH],
[prefix of the OpenSSL library]),
@@ -231,10 +223,13 @@ if test -n "$openssl"; then
AC_DEFINE(HAVE_OPENSSL, 1, [Whether to use OpenSSL.])
fi
AC_ARG_WITH(bzip2, AC_HELP_STRING([--with-bzip2=PATH],
[prefix of bzip2]),
bzip2=$withval, bzip2=)
AM_CONDITIONAL(HAVE_BZIP2, test -n "$bzip2")
ATERM_VERSION=2.5
AC_SUBST(ATERM_VERSION)
if test -z "$bzip2"; then
# Headers and libraries will be used from the temporary installation
# in externals/inst-bzip2.
@@ -256,7 +251,56 @@ AC_SUBST(bzip2_bin)
AC_SUBST(bzip2_bin_test)
AC_CHECK_LIB(pthread, pthread_mutex_init)
AC_ARG_WITH(sqlite, AC_HELP_STRING([--with-sqlite=PATH],
[prefix of SQLite]),
sqlite=$withval, sqlite=)
AM_CONDITIONAL(HAVE_SQLITE, test -n "$sqlite")
SQLITE_VERSION=3070900
AC_SUBST(SQLITE_VERSION)
if test -z "$sqlite"; then
sqlite_lib='${top_builddir}/externals/sqlite-autoconf-$(SQLITE_VERSION)/libsqlite3.la'
sqlite_include='-I${top_builddir}/externals/sqlite-autoconf-$(SQLITE_VERSION)'
sqlite_bin='${top_builddir}/externals/sqlite-autoconf-$(SQLITE_VERSION)'
else
sqlite_lib="-L$sqlite/lib -lsqlite3"
sqlite_include="-I$sqlite/include"
sqlite_bin="$sqlite/bin"
fi
AC_SUBST(sqlite_lib)
AC_SUBST(sqlite_include)
AC_SUBST(sqlite_bin)
# Whether to use the Boehm garbage collector.
AC_ARG_ENABLE(gc, AC_HELP_STRING([--enable-gc],
[enable garbage collection in the Nix expression evaluator (requires Boehm GC)]),
gc=$enableval, gc=no)
if test "$gc" = yes; then
PKG_CHECK_MODULES([BDW_GC], [bdw-gc])
CXXFLAGS="$BDW_GC_CFLAGS $CXXFLAGS"
AC_DEFINE(HAVE_BOEHMGC, 1, [Whether to use the Boehm garbage collector.])
fi
# Check for the required Perl dependencies (DBI and DBD::SQLite).
perlFlags="-I${libdir}/perl5/site_perl"
AC_ARG_WITH(dbi, AC_HELP_STRING([--with-dbi=PATH],
[prefix of the Perl DBI library]),
perlFlags="$perlFlags -I$withval")
AC_ARG_WITH(dbd-sqlite, AC_HELP_STRING([--with-dbd-sqlite=PATH],
[prefix of the Perl DBD::SQLite library]),
perlFlags="$perlFlags -I$withval")
AC_MSG_CHECKING([whether DBD::SQLite works])
if ! $perl $perlFlags -e 'use DBI; use DBD::SQLite;' 2>&5; then
AC_MSG_RESULT(no)
AC_MSG_FAILURE([The Perl modules DBI and/or DBD::SQLite are missing.])
fi
AC_MSG_RESULT(yes)
AC_SUBST(perlFlags)
AC_ARG_ENABLE(init-state, AC_HELP_STRING([--disable-init-state],
@@ -270,8 +314,7 @@ AC_CHECK_FUNCS([setresuid setreuid lchown])
# Nice to have, but not essential.
AC_CHECK_FUNCS([strsignal])
AC_CHECK_FUNCS([posix_fallocate])
AC_CHECK_FUNCS([strsignal posix_fallocate nanosleep sysconf])
# This is needed if ATerm or bzip2 are static libraries,
@@ -281,14 +324,6 @@ if test "$(uname)" = "Darwin"; then
fi
if test "$static_nix" = yes; then
# `-all-static' has to be added at the end of configure, because
# the C compiler doesn't know about -all-static (it's filtered out
# by libtool, but configure doesn't use libtool).
LDFLAGS="-all-static $LDFLAGS"
fi
AM_CONFIG_HEADER([config.h])
AC_CONFIG_FILES([Makefile
externals/Makefile
@@ -308,11 +343,9 @@ AC_CONFIG_FILES([Makefile
src/nix-setuid-helper/Makefile
src/nix-log2xml/Makefile
src/bsdiff-4.3/Makefile
perl/Makefile
scripts/Makefile
corepkgs/Makefile
corepkgs/nar/Makefile
corepkgs/buildenv/Makefile
corepkgs/channels/Makefile
doc/Makefile
doc/manual/Makefile
misc/Makefile
+11 -1
View File
@@ -1 +1,11 @@
SUBDIRS = nar buildenv channels
all-local: config.nix
files = nar.nix buildenv.nix buildenv.pl unpack-channel.nix unpack-channel.sh derivation.nix
install-exec-local:
$(INSTALL) -d $(DESTDIR)$(datadir)/nix/corepkgs
$(INSTALL_DATA) config.nix $(files) $(DESTDIR)$(datadir)/nix/corepkgs
include ../substitute.mk
EXTRA_DIST = config.nix.in $(files)
@@ -1,9 +1,12 @@
{system, derivations, manifest}:
with import <nix/config.nix>;
{ system, derivations, manifest }:
derivation {
name = "user-environment";
system = system;
builder = ./builder.pl;
builder = perl;
args = [ "-w" ./buildenv.pl ];
manifest = manifest;
@@ -11,4 +14,8 @@ derivation {
paths = derivations;
active = map (x: if x ? meta && x.meta ? active then x.meta.active else "true") derivations;
priority = map (x: if x ? meta && x.meta ? priority then x.meta.priority else "5") derivations;
# Building user environments remotely just causes huge amounts of
# network traffic, so don't do that.
preferLocalBuild = true;
}
+10 -4
View File
@@ -1,5 +1,3 @@
#! @perl@ -w
use strict;
use Cwd;
use IO::Handle;
@@ -29,10 +27,18 @@ sub createLinks {
$baseName =~ s/^.*\///g; # strip directory
my $dstFile = "$dstDir/$baseName";
# The files below are special-cased so that they don't show up
# in user profiles, either because they are useless, or
# because they would cause pointless collisions (e.g., each
# Python package brings its own
# `$out/lib/pythonX.Y/site-packages/easy-install.pth'.)
# Urgh, hacky...
if ($srcFile =~ /\/propagated-build-inputs$/ ||
if ($srcFile =~ /\/propagated-build-inputs$/ ||
$srcFile =~ /\/nix-support$/ ||
$srcFile =~ /\/perllocal.pod$/ ||
$srcFile =~ /\/easy-install.pth$/ ||
$srcFile =~ /\/site.py$/ ||
$srcFile =~ /\/site.pyc$/ ||
$srcFile =~ /\/info\/dir$/ ||
$srcFile =~ /\/log$/)
{
@@ -160,4 +166,4 @@ while (scalar(keys %postponed) > 0) {
print STDERR "created $symlinks symlinks in user environment\n";
symlink($ENV{"manifest"}, "$out/manifest") or die "cannot create manifest";
symlink($ENV{"manifest"}, "$out/manifest.nix") or die "cannot create manifest";
-11
View File
@@ -1,11 +0,0 @@
all-local: builder.pl
install-exec-local:
$(INSTALL) -d $(DESTDIR)$(datadir)/nix/corepkgs
$(INSTALL) -d $(DESTDIR)$(datadir)/nix/corepkgs/buildenv
$(INSTALL_DATA) $(srcdir)/default.nix $(DESTDIR)$(datadir)/nix/corepkgs/buildenv
$(INSTALL_PROGRAM) builder.pl $(DESTDIR)$(datadir)/nix/corepkgs/buildenv
include ../../substitute.mk
EXTRA_DIST = default.nix builder.pl.in
-11
View File
@@ -1,11 +0,0 @@
all-local: unpack.sh
install-exec-local:
$(INSTALL) -d $(DESTDIR)$(datadir)/nix/corepkgs
$(INSTALL) -d $(DESTDIR)$(datadir)/nix/corepkgs/channels
$(INSTALL_DATA) $(srcdir)/unpack.nix $(DESTDIR)$(datadir)/nix/corepkgs/channels
$(INSTALL_PROGRAM) unpack.sh $(DESTDIR)$(datadir)/nix/corepkgs/channels
include ../../substitute.mk
EXTRA_DIST = unpack.nix unpack.sh.in
-7
View File
@@ -1,7 +0,0 @@
{system, inputs}:
derivation {
name = "channels";
builder = ./unpack.sh;
inherit system inputs;
}
-35
View File
@@ -1,35 +0,0 @@
#! @shell@ -e
# Cygwin compatibility hack: bunzip2 expects cygwin.dll in $PATH.
export PATH=@coreutils@
@coreutils@/mkdir $out
@coreutils@/mkdir $out/tmp
cd $out/tmp
inputs=($inputs)
for ((n = 0; n < ${#inputs[*]}; n += 2)); do
channelName=${inputs[n]}
channelTarball=${inputs[n+1]}
echo "unpacking channel $channelName"
@bunzip2@ < $channelTarball | @tar@ xf -
if test -e */channel-name; then
channelName="$(@coreutils@/cat */channel-name)"
fi
nr=1
attrName=$(echo $channelName | @tr@ -- '- ' '__')
dirName=$attrName
while test -e ../$dirName; do
nr=$((nr+1))
dirName=$attrName-$nr
done
@coreutils@/mv * ../$dirName # !!! hacky
done
cd ..
@coreutils@/rmdir tmp
+13
View File
@@ -0,0 +1,13 @@
let
fromEnv = var: def:
let val = builtins.getEnv var; in
if val != "" then val else def;
in {
perl = "@perl@";
shell = "@shell@";
coreutils = "@coreutils@";
bzip2 = fromEnv "NIX_BZIP2" "@bzip2@";
tar = "@tar@";
tr = "@tr@";
nixBinDir = fromEnv "NIX_BIN_DIR" "@bindir@";
}
+27
View File
@@ -0,0 +1,27 @@
/* This is the implementation of the ‘derivation’ builtin function.
It's actually a wrapper around the ‘derivationStrict’ primop. */
drvAttrs @ { outputs ? [ "out" ], ... }:
let
strict = derivationStrict drvAttrs;
commonAttrs = drvAttrs // (builtins.listToAttrs outputsList) //
{ all = map (x: x.value) outputsList;
inherit drvAttrs;
};
outputToAttrListElement = outputName:
{ name = outputName;
value = commonAttrs // {
outPath = builtins.getAttr outputName strict;
drvPath = strict.drvPath;
type = "derivation";
inherit outputName;
};
};
outputsList = map outputToAttrListElement outputs;
in (builtins.head outputsList).value
+30
View File
@@ -0,0 +1,30 @@
with import <nix/config.nix>;
let
builder = builtins.toFile "nar.sh"
''
export PATH=${nixBinDir}:${coreutils}
echo "packing ‘$storePath’..."
mkdir $out
dst=$out/tmp.nar.bz2
set -o pipefail
nix-store --dump "$storePath" | ${bzip2} > $dst
nix-hash --flat --type $hashAlgo --base32 $dst > $out/narbz2-hash
mv $out/tmp.nar.bz2 $out/$(cat $out/narbz2-hash).nar.bz2
'';
in
{ system, storePath, hashAlgo }:
derivation {
name = "nar";
builder = shell;
args = [ "-e" builder ];
inherit system storePath hashAlgo;
}
-11
View File
@@ -1,11 +0,0 @@
all-local: nar.sh
install-exec-local:
$(INSTALL) -d $(DESTDIR)$(datadir)/nix/corepkgs
$(INSTALL) -d $(DESTDIR)$(datadir)/nix/corepkgs/nar
$(INSTALL_DATA) $(srcdir)/nar.nix $(DESTDIR)$(datadir)/nix/corepkgs/nar
$(INSTALL_PROGRAM) nar.sh $(DESTDIR)$(datadir)/nix/corepkgs/nar
include ../../substitute.mk
EXTRA_DIST = nar.nix nar.sh.in
-7
View File
@@ -1,7 +0,0 @@
{system, storePath, hashAlgo}:
derivation {
name = "nar";
builder = ./nar.sh;
inherit system storePath hashAlgo;
}
-14
View File
@@ -1,14 +0,0 @@
#! @shell@ -e
echo "packing $storePath into $out..."
@coreutils@/mkdir $out
dst=$out/tmp.nar.bz2
@bindir@/nix-store --dump "$storePath" > tmp
@bzip2@ < tmp > $dst
@bindir@/nix-hash -vvvvv --flat --type $hashAlgo --base32 tmp > $out/nar-hash
@bindir@/nix-hash --flat --type $hashAlgo --base32 $dst > $out/narbz2-hash
@coreutils@/mv $out/tmp.nar.bz2 $out/$(@coreutils@/cat $out/narbz2-hash).nar.bz2
+11
View File
@@ -0,0 +1,11 @@
with import <nix/config.nix>;
{ system, inputs }:
derivation {
name = "channels";
builder = shell;
args = [ "-e" ./unpack-channel.sh ];
inherit system inputs bzip2 tar tr;
PATH = "${nixBinDir}:${coreutils}";
}
+30
View File
@@ -0,0 +1,30 @@
mkdir $out
mkdir $out/tmp
cd $out/tmp
inputs=($inputs)
for ((n = 0; n < ${#inputs[*]}; n += 2)); do
channelName=${inputs[n]}
channelTarball=${inputs[n+1]}
echo "unpacking channel $channelName"
$bzip2 -d < $channelTarball | $tar xf -
if test -e */channel-name; then
channelName="$(cat */channel-name)"
fi
nr=1
attrName=$(echo $channelName | $tr -- '- ' '__')
dirName=$attrName
while test -e ../$dirName; do
nr=$((nr+1))
dirName=$attrName-$nr
done
mv * ../$dirName # !!! hacky
done
cd ..
rmdir tmp
+19 -19
View File
@@ -43,10 +43,10 @@ is also available as <function>builtins.derivation</function>.</para>
<listitem><para>Return the names of the attributes in the
attribute set <replaceable>attrs</replaceable> in a sorted list.
For instance, <literal>builtins.attrNames {y = 1; x =
"foo";}</literal> evaluates to <literal>["x" "y"]</literal>.
There is no built-in function <function>attrValues</function>, but
you can easily define it yourself:
For instance, <literal>builtins.attrNames { y = 1; x = "foo";
}</literal> evaluates to <literal>[ "x" "y" ]</literal>. There is
no built-in function <function>attrValues</function>, but you can
easily define it yourself:
<programlisting>
attrValues = attrs: map (name: builtins.getAttr name attrs) (builtins.attrNames attrs);</programlisting>
@@ -442,10 +442,10 @@ x: x + 456</programlisting>
Example:
<programlisting>
builtins.listToAttrs [
{name = "foo"; value = 123;}
{name = "bar"; value = 456;}
]
builtins.listToAttrs
[ { name = "foo"; value = 123; }
{ name = "bar"; value = 456; }
]
</programlisting>
evaluates to
@@ -466,10 +466,10 @@ builtins.listToAttrs [
example,
<programlisting>
map (x: "foo" + x) ["bar" "bla" "abc"]</programlisting>
map (x: "foo" + x) [ "bar" "bla" "abc" ]</programlisting>
evaluates to <literal>["foobar" "foobla"
"fooabc"]</literal>.</para></listitem>
evaluates to <literal>[ "foobar" "foobla" "fooabc"
]</literal>.</para></listitem>
</varlistentry>
@@ -491,10 +491,10 @@ map (x: "foo" + x) ["bar" "bla" "abc"]</programlisting>
a package name and version. The package name is everything up to
but not including the first dash followed by a digit, and the
version is everything following that dash. The result is returned
in an attribute set <literal>{name, version}</literal>. Thus,
in an attribute set <literal>{ name, version }</literal>. Thus,
<literal>builtins.parseDrvName "nix-0.12pre12876"</literal>
returns <literal>{name = "nix"; version =
"0.12pre12876";}</literal>.</para></listitem>
returns <literal>{ name = "nix"; version = "0.12pre12876";
}</literal>.</para></listitem>
</varlistentry>
@@ -550,9 +550,9 @@ in config.someSetting</programlisting>
exist in <replaceable>attrs</replaceable>. For instance,
<screen>
removeAttrs { x = 1; y = 2; z = 3; } ["a" "x" "z"]</screen>
removeAttrs { x = 1; y = 2; z = 3; } [ "a" "x" "z" ]</screen>
evaluates to <literal>{y = 2;}</literal>.</para></listitem>
evaluates to <literal>{ y = 2; }</literal>.</para></listitem>
</varlistentry>
@@ -632,7 +632,7 @@ removeAttrs { x = 1; y = 2; z = 3; } ["a" "x" "z"]</screen>
linkend='ex-hello-builder' /> into one file:
<programlisting>
{stdenv, fetchurl, perl}:
{ stdenv, fetchurl, perl }:
stdenv.mkDerivation {
name = "hello-2.1.1";
@@ -765,12 +765,12 @@ in foo</programlisting>
using <function>toXML</function></title>
<programlisting><![CDATA[
{stdenv, fetchurl, libxslt, jira, uberwiki}:
{ stdenv, fetchurl, libxslt, jira, uberwiki }:
stdenv.mkDerivation (rec {
name = "web-server";
buildInputs = [libxslt];
buildInputs = [ libxslt ];
builder = builtins.toFile "builder.sh" "
source $stdenv/setup
+40 -2
View File
@@ -6,7 +6,7 @@
<para>A number of persistent settings of Nix are stored in the file
<filename><replaceable>prefix</replaceable>/etc/nix/nix.conf</filename>.
<filename><replaceable>sysconfdir</replaceable>/nix/nix.conf</filename>.
This file is a list of <literal><replaceable>name</replaceable> =
<replaceable>value</replaceable></literal> pairs, one per line.
Comments start with a <literal>#</literal> character. An example
@@ -97,6 +97,25 @@ env-keep-derivations = false
</varlistentry>
<varlistentry xml:id="conf-build-cores"><term><literal>build-cores</literal></term>
<listitem><para>Sets the value of the
<envar>NIX_BUILD_CORES</envar> environment variable in the
invocation of builders. Builders can use this variable at their
discretion to control the maximum amount of parallelism. For
instance, in Nixpkgs, if the derivation attribute
<varname>enableParallelBuilding</varname> is set to
<literal>true</literal>, the builder passes the
<option>-j<replaceable>N</replaceable></option> flag to GNU Make.
It can be overriden using the <option
linkend='opt-cores'>--cores</option> command line switch and
defaults to <literal>1</literal>. The value <literal>0</literal>
means that the builder should use all available CPU cores in the
system.</para></listitem>
</varlistentry>
<varlistentry xml:id="conf-build-max-silent-time"><term><literal>build-max-silent-time</literal></term>
<listitem>
@@ -117,6 +136,25 @@ env-keep-derivations = false
</varlistentry>
<varlistentry xml:id="conf-build-timeout"><term><literal>build-timeout</literal></term>
<listitem>
<para>This option defines the maximum number of seconds that a
builder can run. This is useful (for instance in a automated
build system) to catch builds that are stuck in an infinite loop
but keep writing to their standard output or standard error. It
can be overriden using the <option
linkend="opt-timeout">--timeout</option> command line
switch.</para>
<para>The value <literal>0</literal> means that there is no
timeout. This is also the default.</para>
</listitem>
</varlistentry>
<varlistentry xml:id="conf-build-users-group"><term><literal>build-users-group</literal></term>
@@ -241,7 +279,7 @@ build-use-chroot = /dev /proc /bin</programlisting>
Nix store metadata (in <filename>/nix/var/nix/db</filename>) are
synchronously flushed to disk. This improves robustness in case
of system crashes, but reduces performance. The default is
<literal>false</literal>.</para></listitem>
<literal>true</literal>.</para></listitem>
</varlistentry>
+11
View File
@@ -271,6 +271,17 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
</varlistentry>
<varlistentry><term><envar>GC_INITIAL_HEAP_SIZE</envar></term>
<listitem><para>If Nix has been configured to use the Boehm garbage
collector, this variable sets the initial size of the heap in bytes.
It defaults to 384 MiB. Setting it to a low value reduces memory
consumption, but will increase runtime due to the overhead of
garbage collection.</para></listitem>
</varlistentry>
</variablelist>
+16 -11
View File
@@ -96,17 +96,22 @@ ubiquitous 2.5.4a won't. Note that these are only required if you
modify the parser or when you are building from the Subversion
repository.</para>
<para>Nix uses CWI's ATerm library and the bzip2 compressor (including
the bzip2 library). These are included in the Nix source
distribution. If you build from the Subversion repository, you must
download them yourself and place them in the
<filename>externals/</filename> directory. See
<para>Nix uses the bzip2 compressor (including the bzip2 library). It
is included in the Nix source distribution. If you build from the
Subversion repository, you must download it yourself and place it in
the <filename>externals/</filename> directory. See
<filename>externals/Makefile.am</filename> for the precise URLs of
these packages. Alternatively, if you already have them installed,
you can use <command>configure</command>'s
<option>--with-aterm</option> and <option>--with-bzip2</option>
this packages. Alternatively, if you already have it installed, you
can use <command>configure</command>'s <option>--with-bzip2</option>
options to point to their respective locations.</para>
<para>Nix can optionally use the <link
xlink:href="http://www.hpl.hp.com/personal/Hans_Boehm/gc/">Boehm
garbage collector</link> to reduce the evaluator’s memory consumption.
To enable it, install <literal>pkgconfig</literal> and the Boehm
garbage collector, and pass the flag <option>--enable-gc</option> to
<command>configure</command>.</para>
</section>
@@ -309,7 +314,7 @@ bit turned on (like <filename>/tmp</filename>):
<screen>
$ chgrp nixbld /nix/store
$ chmod 1777 /nix/store
$ chmod 1775 /nix/store
</screen>
</para>
@@ -318,7 +323,7 @@ $ chmod 1777 /nix/store
specifying the build users group in the <link
linkend="conf-build-users-group"><literal>build-users-group</literal>
option</link> in the <link linkend="sec-conf-file">Nix configuration
file</link> (<literal>/nix/etc/nix/nix.conf</literal>):
file</link> (usually <literal>/etc/nix/nix.conf</literal>):
<programlisting>
build-users-group = nixbld
@@ -375,7 +380,7 @@ is a special account called <literal>nix</literal>, but it can be
named anything. It should own the Nix store and database:
<screen>
$ chown -R root /nix/store /nix/var/nix</screen>
$ chown -R nix /nix/store /nix/var/nix</screen>
and of course <command>nix-worker --daemon</command> should be started
under that user, e.g.,
+1 -1
View File
@@ -113,7 +113,7 @@ $ nix-env --rollback
<simplesect><title>Garbage collection</title>
<para>When you install a package like this…
<para>When you uninstall a package like this…
<screen>
$ nix-env --uninstall firefox
+2 -1
View File
@@ -27,10 +27,11 @@
<year>2007</year>
<year>2008</year>
<year>2009</year>
<year>2010</year>
<holder>Eelco Dolstra</holder>
</copyright>
<date>September 2009</date>
<date>August 2010</date>
</info>
+27 -13
View File
@@ -24,8 +24,11 @@
</group>
<arg><option>--sign</option></arg>
<arg><option>--gzip</option></arg>
<arg><option>--bzip2</option></arg>
<arg><option>--xz</option></arg>
<arg><option>--include-outputs</option></arg>
<arg choice='plain'>
<arg><replaceable>user@</replaceable></arg><replaceable>machine</replaceable>
<replaceable>user@</replaceable><replaceable>machine</replaceable>
</arg>
<arg choice='plain'><replaceable>paths</replaceable></arg>
</cmdsynopsis>
@@ -84,25 +87,36 @@ those paths. If this bothers you, use
<listitem><para>Let the sending machine cryptographically sign the
dump of each path with the key in
<filename>/nix/etc/nix/signing-key.sec</filename>. If the user on
the target machine does not have direct access to the Nix store
(i.e., if the target machine has a multi-user Nix installation),
then the target machine will check the dump against
<filename>/nix/etc/nix/signing-key.pub</filename> before unpacking
it in its Nix store. This allows secure sharing of store paths
between untrusted users on two machines, provided that there is a
trust relation between the Nix installations on both machines
(namely, they have matching public/secret keys).</para></listitem>
<filename><replaceable>sysconfdir</replaceable>/nix/signing-key.sec</filename>.
If the user on the target machine does not have direct access to
the Nix store (i.e., if the target machine has a multi-user Nix
installation), then the target machine will check the dump against
<filename><replaceable>sysconfdir</replaceable>/nix/signing-key.pub</filename>
before unpacking it in its Nix store. This allows secure sharing
of store paths between untrusted users on two machines, provided
that there is a trust relation between the Nix installations on
both machines (namely, they have matching public/secret
keys).</para></listitem>
</varlistentry>
<varlistentry><term><option>--gzip</option></term>
<varlistentry><term><option>--gzip</option> / <option>--bzip2</option> / <option>--xz</option></term>
<listitem><para>Compress the dump of each path with
<command>gzip</command> before sending it.</para></listitem>
<listitem><para>Compress the dump of each path with respectively
<command>gzip</command>, <command>bzip2</command> or
<command>xz</command> before sending it. The corresponding
decompression program must be installed on the target
machine.</para></listitem>
</varlistentry>
<varlistentry><term><option>--include-outputs</option></term>
<listitem><para>Also copy the outputs of store derivations included
in the closure.</para></listitem>
</varlistentry>
</variablelist>
</refsection>
+101 -101
View File
@@ -2,7 +2,7 @@
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-env">
<refmeta>
<refentrytitle>nix-env</refentrytitle>
<manvolnum>1</manvolnum>
@@ -60,7 +60,7 @@ environments.</para>
<para><command>nix-env</command> takes exactly one
<emphasis>operation</emphasis> flag which indicates the subcommand to
be performed. These are documented below.</para>
</refsection>
@@ -78,19 +78,19 @@ linkend="sec-common-options" />.</para>
<varlistentry><term><option>--file</option></term>
<term><option>-f</option></term>
<listitem><para>Specifies the Nix expression (designated below as
the <emphasis>active Nix expression</emphasis>) used by the
<option>--install</option>, <option>--upgrade</option>, and
<option>--query --available</option> operations to obtain
derivations. The default is
<filename>~/.nix-defexpr</filename>.</para></listitem>
</varlistentry>
<varlistentry><term><option>--profile</option></term>
<term><option>-p</option></term>
<listitem><para>Specifies the profile to be used by those
operations that operate on a profile (designated below as the
<emphasis>active profile</emphasis>). A profile is sequence of
@@ -98,11 +98,11 @@ linkend="sec-common-options" />.</para>
which is the <emphasis>current generation</emphasis>. The default
profile is the target of the symbolic link
<filename>~/.nix-profile</filename> (see below).</para></listitem>
</varlistentry>
<varlistentry><term><option>--dry-run</option></term>
<listitem><para>For the <option>--install</option>,
<option>--upgrade</option>, <option>--uninstall</option>,
<option>--switch-generation</option> and
@@ -117,23 +117,23 @@ linkend="sec-common-options" />.</para>
substitute is available).</para></listitem>
</varlistentry>
<varlistentry><term><option>--system-filter</option> <replaceable>system</replaceable></term>
<listitem><para>By default, operations such as <option>--query
--available</option> only include derivations matching the current
platform. This option allows you to use derivations for the
specified platform <replaceable>system</replaceable>. The special
value <literal>*</literal> causes derivations for any platform to
be included.</para></listitem>
</varlistentry>
</variablelist>
</refsection>
<!--######################################################################-->
@@ -170,25 +170,25 @@ linkend="sec-common-options" />.</para>
this directory.</para>
</listitem>
</varlistentry>
<varlistentry><term><filename>~/.nix-profile</filename></term>
<listitem><para>A symbolic link to the user's current profile. By
default, this symlink points to
<filename><replaceable>prefix</replaceable>/var/nix/profiles/default</filename>.
The <envar>PATH</envar> environment variable should include
<filename>~/.nix-profile/bin</filename> for the user environment
to be visible to the user.</para></listitem>
</varlistentry>
</variablelist>
</refsection>
<!--######################################################################-->
@@ -214,7 +214,7 @@ linkend="sec-common-options" />.</para>
<refsection><title>Description</title>
<para>The install operation creates a new user environment, based on
the current generation of the active profile, to which a set of store
paths described by <replaceable>args</replaceable> is added. The
@@ -256,7 +256,7 @@ number of possible ways:
<emphasis>attribute paths</emphasis> that select attributes from the
top-level Nix expression. This is faster than using derivation
names and unambiguous. To find out the attribute paths of available
packages, use <literal>nix-env -qaA '*'</literal>.</para></listitem>
packages, use <literal>nix-env -qaP '*'</literal>.</para></listitem>
<listitem><para>If <option>--from-profile</option>
<replaceable>path</replaceable> is given,
@@ -306,20 +306,20 @@ number of possible ways:
<varlistentry><term><option>--preserve-installed</option></term>
<term><option>-P</option></term>
<listitem><para>Do not remove derivations with a name matching one
of the derivations being installed. Usually, trying to have two
versions of the same package installed in the same generation of a
profile will lead to an error in building the generation, due to
file name clashes between the two versions. However, this is not
the case for all packages.</para></listitem>
</varlistentry>
</variablelist>
</refsection>
<refsection xml:id='refsec-nix-env-install-examples'><title>Examples</title>
@@ -327,7 +327,7 @@ number of possible ways:
active Nix expression:
<screen>
$ nix-env --install gcc-3.3.2
$ nix-env --install gcc-3.3.2
installing `gcc-3.3.2'
uninstalling `gcc-3.1'</screen>
@@ -408,15 +408,15 @@ the following paths will be substituted:
/nix/store/8zbipvm4gp9jfqh9nnk1n3bary1a37gs-perl-XML-Parser-2.34
/nix/store/b8a2bg7gnyvvvjjibp4axg9x1hzkw36c-mono-1.1.4
<replaceable>...</replaceable></screen>
</para>
</refsection>
</refsection>
<!--######################################################################-->
<refsection xml:id="rsec-nix-env-upgrade"><title>Operation <option>--upgrade</option></title>
@@ -442,7 +442,7 @@ the following paths will be substituted:
</refsection>
<refsection><title>Description</title>
<para>The upgrade operation creates a new user environment, based on
the current generation of the active profile, in which all store paths
are replaced for which there are newer versions in the set of paths
@@ -459,47 +459,47 @@ the same symbolic name, only the one with the highest version is
installed.</para>
</refsection>
<refsection><title>Flags</title>
<variablelist>
<varlistentry><term><option>--lt</option></term>
<listitem><para>Only upgrade a derivation to newer versions. This
is the default.</para></listitem>
</varlistentry>
<varlistentry><term><option>--leq</option></term>
<listitem><para>In addition to upgrading to newer versions, also
“upgrade” to derivations that have the same version. Version are
not a unique identification of a derivation, so there may be many
derivations that have the same version. This flag may be useful
to force “synchronisation” between the installed and available
derivations.</para></listitem>
</varlistentry>
<varlistentry><term><option>--eq</option></term>
<listitem><para><emphasis>Only</emphasis> “upgrade” to derivations
that have the same version. This may not seem very useful, but it
actually is, e.g., when there is a new release of Nixpkgs and you
want to replace installed applications with the same versions
built against newer dependencies (to reduce the number of
dependencies floating around on your system).</para></listitem>
</varlistentry>
<varlistentry><term><option>--always</option></term>
<listitem><para>In addition to upgrading to newer versions, also
“upgrade” to derivations that have the same or a lower version.
I.e., derivations may actually be downgraded depending on what is
available in the active Nix expression.</para></listitem>
</varlistentry>
</variablelist>
@@ -523,10 +523,10 @@ $ nix-env --upgrade pan
$ nix-env -u '*' <lineannotation>(try to upgrade everything)</lineannotation>
upgrading `hello-2.1.2' to `hello-2.1.3'
upgrading `mozilla-1.2' to `mozilla-1.4'</screen>
upgrading `mozilla-1.2' to `mozilla-1.4'</screen>
</refsection>
<refsection xml:id="ssec-version-comparisons"><title>Versions</title>
<para>The upgrade operation determines whether a derivation
@@ -570,14 +570,14 @@ lexicographically (i.e., using case-sensitive string comparison).</para>
2.3a &lt; 2.3c
2.3pre1 &lt; 2.3c
2.3pre1 &lt; 2.3q</screen>
</para>
</refsection>
</refsection>
<!--######################################################################-->
@@ -596,14 +596,14 @@ lexicographically (i.e., using case-sensitive string comparison).</para>
</refsection>
<refsection><title>Description</title>
<para>The uninstall operation creates a new user environment, based on
the current generation of the active profile, from which the store
paths designated by the symbolic names
<replaceable>names</replaceable> are removed.</para>
</refsection>
<refsection><title>Examples</title>
<screen>
@@ -611,11 +611,11 @@ $ nix-env --uninstall gcc
$ nix-env -e '*' <lineannotation>(remove everything)</lineannotation></screen>
</refsection>
</refsection>
<!--######################################################################-->
<refsection xml:id="rsec-nix-env-set-flag"><title>Operation <option>--set-flag</option></title>
@@ -632,7 +632,7 @@ $ nix-env -e '*' <lineannotation>(remove everything)</lineannotation></screen>
</refsection>
<refsection><title>Description</title>
<para>The <option>--set-flag</option> operation allows meta attributes
of installed packages to be modified. There are several attributes
that can be usefully modified, because they affect the behaviour of
@@ -670,7 +670,7 @@ script:
</para>
</refsection>
<refsection><title>Examples</title>
<para>To prevent the currently installed Firefox from being upgraded:
@@ -716,13 +716,13 @@ $ nix-env --set-flag priority 10 gcc</screen>
</para>
</refsection>
</refsection>
<!--######################################################################-->
<refsection><title>Operation <option>--query</option></title>
<refsection><title>Synopsis</title>
@@ -738,9 +738,9 @@ $ nix-env --set-flag priority 10 gcc</screen>
<arg choice='plain'><option>--available</option></arg>
<arg choice='plain'><option>-a</option></arg>
</group>
<sbr />
<arg>
<group choice='req'>
<arg choice='plain'><option>--status</option></arg>
@@ -785,7 +785,7 @@ $ nix-env --set-flag priority 10 gcc</screen>
</arg>
<sbr />
<arg choice='plain' rep='repeat'><replaceable>names</replaceable></arg>
</cmdsynopsis>
@@ -793,7 +793,7 @@ $ nix-env --set-flag priority 10 gcc</screen>
<refsection><title>Description</title>
<para>The query operation displays information about either the store
paths that are installed in the current generation of the active
profile (<option>--installed</option>), or the derivations that are
@@ -817,23 +817,23 @@ operates.</para>
<variablelist>
<varlistentry><term><option>--installed</option></term>
<listitem><para>The query operates on the store paths that are
installed in the current generation of the active profile. This
is the default.</para></listitem>
</varlistentry>
<varlistentry><term><option>--available</option></term>
<term><option>-a</option></term>
<listitem><para>The query operates on the derivations that are
available in the active Nix expression.</para></listitem>
</varlistentry>
</variablelist>
</refsection>
@@ -874,7 +874,7 @@ user environment elements, etc. -->
<varlistentry><term><option>--status</option></term>
<term><option>-s</option></term>
<listitem><para>Print the <emphasis>status</emphasis> of the
derivation. The status consists of three characters. The first
is <literal>I</literal> or <literal>-</literal>, indicating
@@ -888,7 +888,7 @@ user environment elements, etc. -->
third is <literal>S</literal> or <literal>-</literal>, indicating
whether a substitute is available for the
derivation.</para></listitem>
</varlistentry>
<varlistentry><term><option>--attr-path</option></term>
@@ -901,17 +901,17 @@ user environment elements, etc. -->
<literal>nix-env --install</literal>.</para></listitem>
</varlistentry>
<varlistentry><term><option>--no-name</option></term>
<listitem><para>Suppress printing of the <literal>name</literal>
attribute of each derivation.</para></listitem>
</varlistentry>
<varlistentry><term><option>--compare-versions</option> /
<option>-c</option></term>
<listitem><para>Compare installed versions to available versions,
or vice versa (if <option>--available</option> is given). This is
useful for quickly seeing whether upgrades for installed
@@ -926,21 +926,21 @@ user environment elements, etc. -->
or installed.</para></listitem>
</varlistentry>
<varlistentry><term><literal>=</literal> <replaceable>version</replaceable></term>
<listitem><para>At most the same version of the package is
available or installed.</para></listitem>
</varlistentry>
<varlistentry><term><literal>></literal> <replaceable>version</replaceable></term>
<listitem><para>Only older versions of the package are
available or installed.</para></listitem>
</varlistentry>
<varlistentry><term><literal>- ?</literal></term>
<listitem><para>No version of the package is available or
@@ -951,45 +951,45 @@ user environment elements, etc. -->
</variablelist>
</para></listitem>
</varlistentry>
<varlistentry><term><option>--system</option></term>
<listitem><para>Print the <literal>system</literal> attribute of
the derivation.</para></listitem>
</varlistentry>
<varlistentry><term><option>--drv-path</option></term>
<listitem><para>Print the path of the store
derivation.</para></listitem>
</varlistentry>
<varlistentry><term><option>--out-path</option></term>
<listitem><para>Print the output path of the
derivation.</para></listitem>
</varlistentry>
<varlistentry><term><option>--description</option></term>
<listitem><para>Print a short (one-line) description of the
derivation, if available. The description is taken from the
<literal>meta.description</literal> attribute of the
derivation.</para></listitem>
</varlistentry>
<varlistentry><term><option>--meta</option></term>
<listitem><para>Print all of the meta-attributes of the
derivation. This option is only available with
<option>--xml</option>.</para></listitem>
</varlistentry>
</variablelist>
@@ -1023,7 +1023,7 @@ IP- ORBit2-2.8.3 <lineannotation>(installed and by definition present)</linea
...
<lineannotation>(show available derivations in the Nix expression <!-- !!! <filename>-->foo.nix<!-- </filename> -->)</lineannotation>
$ nix-env -f ./foo.nix -qa '*'
$ nix-env -f ./foo.nix -qa '*'
foo-1.2.3
$ nix-env -qc '*' <lineannotation>(compare installed versions to what’s available)</lineannotation>
@@ -1034,7 +1034,7 @@ firefox-1.0.4 &lt; 1.0.7 <lineannotation>(a more recent version is availab
<replaceable>...</replaceable>
<lineannotation>(show info about a specific package, in XML)</lineannotation>
$ nix-env -qa --xml --description firefox
$ nix-env -qa --xml --description firefox
<![CDATA[<?xml version='1.0' encoding='utf-8'?>
<items>
<item attrPath="0.0.firefoxWrapper"
@@ -1045,8 +1045,8 @@ $ nix-env -qa --xml --description firefox
</refsection>
</refsection>
<!--######################################################################-->
@@ -1067,25 +1067,25 @@ $ nix-env -qa --xml --description firefox
<refsection><title>Description</title>
<para>This operation makes <replaceable>path</replaceable> the current
profile for the user. That is, the symlink
<filename>~/.nix-profile</filename> is made to point to
<replaceable>path</replaceable>.</para>
</refsection>
<refsection><title>Examples</title>
<screen>
$ nix-env -S ~/my-profile</screen>
</refsection>
</refsection>
<!--######################################################################-->
<refsection><title>Operation <option>--list-generations</option></title>
@@ -1101,7 +1101,7 @@ $ nix-env -S ~/my-profile</screen>
<refsection><title>Description</title>
<para>This operation print a list of all the currently existing
generations for the active profile. These may be switched to using
the <option>--switch-generation</option> operation. It also prints
@@ -1121,11 +1121,11 @@ $ nix-env --list-generations
98 2004-02-06 16:24:33 (current)</screen>
</refsection>
</refsection>
<!--######################################################################-->
<refsection><title>Operation <option>--delete-generations</option></title>
@@ -1142,7 +1142,7 @@ $ nix-env --list-generations
<refsection><title>Description</title>
<para>This operation deletes the specified generations of the current
profile. The generations can be a list of generation numbers, or the
special value <literal>old</literal> to delete all non-current
@@ -1150,7 +1150,7 @@ generations. Periodically deleting old generations is important to
make garbage collection effective.</para>
</refsection>
<refsection><title>Examples</title>
<screen>
@@ -1159,11 +1159,11 @@ $ nix-env --delete-generations 3 4 8
$ nix-env -p other_profile --delete-generations old</screen>
</refsection>
</refsection>
<!--######################################################################-->
<refsection><title>Operation <option>--switch-generation</option></title>
@@ -1183,7 +1183,7 @@ $ nix-env -p other_profile --delete-generations old</screen>
<refsection><title>Description</title>
<para>This operation makes generation number
<replaceable>generation</replaceable> the current generation of the
active profile. That is, if the
@@ -1207,11 +1207,11 @@ $ nix-env -G 42
switching from generation 50 to 42</screen>
</refsection>
</refsection>
<!--######################################################################-->
<refsection><title>Operation <option>--rollback</option></title>
@@ -1226,7 +1226,7 @@ switching from generation 50 to 42</screen>
</refsection>
<refsection><title>Description</title>
<para>This operation switches to the “previous” generation of the
active profile, that is, the highest numbered generation lower than
the current generation, if it exists. It is just a convenience
@@ -1246,9 +1246,9 @@ $ nix-env --rolback
error: no generation older than the current (91) exists</screen>
</refsection>
</refsection>
</refentry>
+1 -1
View File
@@ -12,7 +12,7 @@
<refnamediv>
<refname>nix-prefetch-url</refname>
<refpurpose>copy a file from a URL into the store and print its MD5 hash</refpurpose>
<refpurpose>copy a file from a URL into the store and print its hash</refpurpose>
</refnamediv>
<refsynopsisdiv>
+1 -1
View File
@@ -120,7 +120,7 @@ dependencies used in the build, such as compilers).</para>
dependencies, we can do:
<screen>
$ nix-push <replaceable>urls</replaceable> $(nix-instantiate $(nix-store -r foo.nix))</screen>
$ nix-push <replaceable>urls</replaceable> $(nix-store -r $(nix-instantiate foo.nix))</screen>
</para>
+17 -4
View File
@@ -342,7 +342,7 @@ $ nix-store --gc --max-freed $((100 * 1024 * 1024))</screen>
<cmdsynopsis>
<command>nix-store</command>
<arg choice='plain'><option>--gc</option></arg>
<arg choice='plain'><option>--delete</option></arg>
<arg><option>--ignore-liveness</option></arg>
<arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg>
</cmdsynopsis>
@@ -404,6 +404,7 @@ error: cannot delete path `/nix/store/zq0h41l75vlb4z45kzgjjmsjxvcv1qk7-mesa-6.4'
<arg choice='plain'><option>--tree</option></arg>
<arg choice='plain'><option>--binding</option> <replaceable>name</replaceable></arg>
<arg choice='plain'><option>--hash</option></arg>
<arg choice='plain'><option>--size</option></arg>
<arg choice='plain'><option>--roots</option></arg>
</group>
<arg><option>--use-output</option></arg>
@@ -587,9 +588,21 @@ query is applied to the target of the symlink.</para>
<varlistentry><term><option>--hash</option></term>
<listitem><para>Prints the SHA-256 hash of the contents of the
store paths <replaceable>paths</replaceable>. Since the hash is
stored in the Nix database, this is a fast
operation.</para></listitem>
store paths <replaceable>paths</replaceable> (that is, the hash of
the output of <command>nix-store --dump</command> on the given
paths). Since the hash is stored in the Nix database, this is a
fast operation.</para></listitem>
</varlistentry>
<varlistentry><term><option>--size</option></term>
<listitem><para>Prints the size in bytes of the contents of the
store paths <replaceable>paths</replaceable> — to be precise, the
size of the output of <command>nix-store --dump</command> on the
given paths. Note that the actual disk space required by the
store paths may be higher, especially on filesystems with large
cluster sizes.</para></listitem>
</varlistentry>
+8
View File
@@ -13,10 +13,18 @@
</group>
<replaceable>number</replaceable>
</arg>
<arg>
<arg><option>--cores</option></arg>
<replaceable>number</replaceable>
</arg>
<arg>
<arg><option>--max-silent-time</option></arg>
<replaceable>number</replaceable>
</arg>
<arg>
<arg><option>--timeout</option></arg>
<replaceable>number</replaceable>
</arg>
<arg><option>--keep-going</option></arg>
<arg><option>-k</option></arg>
<arg><option>--keep-failed</option></arg>
+31 -3
View File
@@ -98,7 +98,25 @@
linkend='conf-build-max-jobs'><literal>build-max-jobs</literal></link>
configuration setting, which itself defaults to
<literal>1</literal>. A higher value is useful on SMP systems or to
exploit I/O latency. </para></listitem>
exploit I/O latency.</para></listitem>
</varlistentry>
<varlistentry xml:id="opt-cores"><term><option>--cores</option></term>
<listitem><para>Sets the value of the <envar>NIX_BUILD_CORES</envar>
environment variable in the invocation of builders. Builders can
use this variable at their discretion to control the maximum amount
of parallelism. For instance, in Nixpkgs, if the derivation
attribute <varname>enableParallelBuilding</varname> is set to
<literal>true</literal>, the builder passes the
<option>-j<replaceable>N</replaceable></option> flag to GNU Make.
It defaults to the value of the <link
linkend='conf-build-cores'><literal>build-cores</literal></link>
configuration setting, if set, or <literal>1</literal> otherwise.
The value <literal>0</literal> means that the builder should use all
available CPU cores in the system.</para></listitem>
</varlistentry>
@@ -114,6 +132,16 @@
</varlistentry>
<varlistentry xml:id="opt-timeout"><term><option>--timeout</option></term>
<listitem><para>Sets the maximum number of seconds that a builder
can run. The default is specified by the <link
linkend='conf-build-timeout'><literal>build-timeout</literal></link>
configuration setting. <literal>0</literal> means no
timeout.</para></listitem>
</varlistentry>
<varlistentry><term><option>--keep-going</option></term>
<term><option>-k</option></term>
@@ -236,8 +264,8 @@
expression evaluator will automatically try to call functions that
it encounters. It can automatically call functions for which every
argument has a <link linkend='ss-functions'>default value</link>
(e.g., <literal>{<replaceable>argName</replaceable> ?
<replaceable>defaultValue</replaceable>}:
(e.g., <literal>{ <replaceable>argName</replaceable> ?
<replaceable>defaultValue</replaceable> }:
<replaceable>...</replaceable></literal>). With
<option>--arg</option>, you can also call functions that have
arguments without a default value (or override a default value).
+1 -1
View File
@@ -60,7 +60,7 @@ available remotely.</para></listitem>
in the channel:
<screen>
$ nix-env -qa ’*’ <lineannotation>(mind the quotes!)</lineannotation>
$ nix-env -qa \*
docbook-xml-4.2
firefox-1.0pre-PR-0.10.1
hello-2.1.1
+96
View File
@@ -6,6 +6,102 @@
<!--==================================================================-->
<section xml:id="ssec-relnotes-1.0"><title>Release 1.0 (TBA)</title>
<para>This release has the following improvements:</para>
<itemizedlist>
<listitem>
<para>Nix can now optionally use the Boehm garbage collector.
This significantly reduces the Nix evaluator’s memory footprint,
especially when evaluating large NixOS system configurations. It
can be enabled using the <option>--enable-gc</option> configure
option.</para>
</listitem>
<listitem>
<para>The option <option>--timeout</option> (corresponding to the
configuration setting <literal>build-timeout</literal>) allows you
to set an absolute timeout on builds — if a build runs for more than
the given number of seconds, it is terminated. This is useful for
recovering automatically from builds that are stuck in an infinite
loop but keep producing output, and for which
<literal>--max-silent-time</literal> is ineffective.</para>
</listitem>
<listitem>
<para>TODO: “or” keyword.</para>
</listitem>
<listitem>
<para>TODO: Nix expression search path (<literal>import &lt;foo/bar.nix></literal>).</para>
</listitem>
<listitem>
<para>TODO: <command>nix-store --verify-path</command> command.</para>
</listitem>
</itemizedlist>
</section>
<!--==================================================================-->
<section xml:id="ssec-relnotes-0.16"><title>Release 0.16 (August 17, 2010)</title>
<para>This release has the following improvements:</para>
<itemizedlist>
<listitem>
<para>The Nix expression evaluator is now much faster in most
cases: typically, <link
xlink:href="http://www.mail-archive.com/nix-dev@cs.uu.nl/msg04113.html">3
to 8 times compared to the old implementation</link>. It also
uses less memory. It no longer depends on the ATerm
library.</para>
</listitem>
<listitem>
<para>
Support for configurable parallelism inside builders. Build
scripts have always had the ability to perform multiple build
actions in parallel (for instance, by running <command>make -j
2</command>), but this was not desirable because the number of
actions to be performed in parallel was not configurable. Nix
now has an option <option>--cores
<replaceable>N</replaceable></option> as well as a configuration
setting <varname>build-cores =
<replaceable>N</replaceable></varname> that causes the
environment variable <envar>NIX_BUILD_CORES</envar> to be set to
<replaceable>N</replaceable> when the builder is invoked. The
builder can use this at its discretion to perform a parallel
build, e.g., by calling <command>make -j
<replaceable>N</replaceable></command>. In Nixpkgs, this can be
enabled on a per-package basis by setting the derivation
attribute <varname>enableParallelBuilding</varname> to
<literal>true</literal>.
</para>
</listitem>
<listitem>
<para><command>nix-store -q</command> now supports XML output
through the <option>--xml</option> flag.</para>
</listitem>
<listitem>
<para>Several bug fixes.</para>
</listitem>
</itemizedlist>
</section>
<!--==================================================================-->
<section xml:id="ssec-relnotes-0.15"><title>Release 0.15 (March 17, 2010)</title>
+39 -43
View File
@@ -52,7 +52,7 @@ need to do three things:
<example xml:id='ex-hello-nix'><title>Nix expression for GNU Hello
(<filename>default.nix</filename>)</title>
<programlisting>
{stdenv, fetchurl, perl}: <co xml:id='ex-hello-nix-co-1' />
{ stdenv, fetchurl, perl }: <co xml:id='ex-hello-nix-co-1' />
stdenv.mkDerivation { <co xml:id='ex-hello-nix-co-2' />
name = "hello-2.1.1"; <co xml:id='ex-hello-nix-co-3' />
@@ -92,8 +92,8 @@ the single Nix expression in that directory
function that downloads files. <varname>perl</varname> is the
Perl interpreter.</para>
<para>Nix functions generally have the form <literal>{x, y, ...,
z}: e</literal> where <varname>x</varname>, <varname>y</varname>,
<para>Nix functions generally have the form <literal>{ x, y, ...,
z }: e</literal> where <varname>x</varname>, <varname>y</varname>,
etc. are the names of the expected arguments, and where
<replaceable>e</replaceable> is the body of the function. So
here, the entire remainder of the file is the body of the
@@ -114,10 +114,10 @@ the single Nix expression in that directory
<emphasis>attributes</emphasis>. An attribute set is just a list
of key/value pairs where each value is an arbitrary Nix
expression. They take the general form
<literal>{<replaceable>name1</replaceable> =
<literal>{ <replaceable>name1</replaceable> =
<replaceable>expr1</replaceable>; <replaceable>...</replaceable>
<replaceable>nameN</replaceable> =
<replaceable>exprN</replaceable>;}</literal>.</para>
<replaceable>exprN</replaceable>; }</literal>.</para>
</callout>
@@ -564,7 +564,7 @@ genericBuild <co xml:id='ex-hello-builder2-co-3' /></programlisting>
expression, like this:
<programlisting>
buildInputs = [perl];</programlisting>
buildInputs = [ perl ];</programlisting>
The <varname>perl</varname> attribute can then be removed, and the
builder becomes even shorter:
@@ -771,14 +771,14 @@ stdenv.mkDerivation {
values between square brackets. For example,
<programlisting>
[ 123 ./foo.nix "abc" (f {x=y;}) ]</programlisting>
[ 123 ./foo.nix "abc" (f { x = y; }) ]</programlisting>
defines a list of four elements, the last being the result of a call
to the function <varname>f</varname>. Note that function calls have
to be enclosed in parentheses. If they had been omitted, e.g.,
<programlisting>
[ 123 ./foo.nix "abc" f {x=y;} ]</programlisting>
[ 123 ./foo.nix "abc" f { x = y; } ]</programlisting>
the result would be a list of five elements, the fourth one being a
function and the fifth being an attribute set.</para>
@@ -891,15 +891,12 @@ propagate attributes). This can be shortened using the
<literal>inherit</literal> keyword. For instance,
<programlisting>
let
x = 123;
in
{
inherit x;
y = 456;
}</programlisting>
let x = 123; in
{ inherit x;
y = 456;
}</programlisting>
evaluates to <literal>{x = 123; y = 456;}</literal>. (Note that this
evaluates to <literal>{ x = 123; y = 456; }</literal>. (Note that this
works because <varname>x</varname> is added to the lexical scope by
the <literal>let</literal> construct.) It is also possible to inherit
attributes from another attribute set. For instance, in this fragment
@@ -960,20 +957,20 @@ in if negate true then concat "foo" "bar" else ""</programlisting>
arguments of a function); e.g.,
<programlisting>
map (concat "foo") ["bar" "bla" "abc"]</programlisting>
map (concat "foo") [ "bar" "bla" "abc" ]</programlisting>
evaluates to <literal>["foobar" "foobla"
"fooabc"]</literal>.</para></listitem>
evaluates to <literal>[ "foobar" "foobla"
"fooabc" ]</literal>.</para></listitem>
<listitem><para>An <emphasis>attribute set pattern</emphasis> of the
form <literal>{name1, name2, …, nameN}</literal>
form <literal>{ name1, name2, …, nameN }</literal>
matches an attribute set containing the listed attributes, and binds
the values of those attributes to variables in the function body.
For example, the function
<programlisting>
{x, y, z}: z + y + x</programlisting>
{ x, y, z }: z + y + x</programlisting>
can only be called with a set containing exactly the attributes
<varname>x</varname>, <varname>y</varname> and
@@ -982,7 +979,7 @@ map (concat "foo") ["bar" "bla" "abc"]</programlisting>
(<literal>...</literal>):
<programlisting>
{x, y, z, ....}: z + y + x</programlisting>
{ x, y, z, ... }: z + y + x</programlisting>
This works on any set that contains at least the three named
attributes.</para>
@@ -995,7 +992,7 @@ map (concat "foo") ["bar" "bla" "abc"]</programlisting>
<replaceable>e</replaceable> is an arbitrary expression. For example,
<programlisting>
{x, y ? "foo", z ? "bar"}: z + y + x</programlisting>
{ x, y ? "foo", z ? "bar" }: z + y + x</programlisting>
specifies a function that only requires an attribute named
<varname>x</varname>, but optionally accepts <varname>y</varname>
@@ -1007,11 +1004,11 @@ map (concat "foo") ["bar" "bla" "abc"]</programlisting>
of the <literal>@</literal>-sign. For example:
<programlisting>
args@{x, y, z, ...}: z + y + x + args.a</programlisting>
args@{ x, y, z, ... }: z + y + x + args.a</programlisting>
Here <varname>args</varname> is bound to the entire argument, which
is further matches against the pattern <literal>{x, y, z,
...}</literal>.</para></listitem>
is further matches against the pattern <literal>{ x, y, z,
... }</literal>.</para></listitem>
</itemizedlist>
@@ -1020,8 +1017,8 @@ args@{x, y, z, ...}: z + y + x + args.a</programlisting>
a name, you can bind them to an attribute, e.g.,
<programlisting>
let concat = {x, y}: x + y;
in concat {x = "foo"; y = "bar";}</programlisting>
let concat = { x, y }: x + y;
in concat { x = "foo"; y = "bar"; }</programlisting>
</para>
@@ -1142,7 +1139,7 @@ lexical scope of the expression <replaceable>e2</replaceable>. For
instance,
<programlisting>
let as = {x = "foo"; y = "bar";};
let as = { x = "foo"; y = "bar"; };
in with as; x + y</programlisting>
evaluates to <literal>"foobar"</literal> since the
@@ -1480,21 +1477,20 @@ allowedReferences = [];
references graph of their inputs. The attribute is a list of
inputs in the Nix store whose references graph the builder needs
to know. The value of this attribute should be a list of pairs
<literal>[<replaceable>name1</replaceable>
<literal>[ <replaceable>name1</replaceable>
<replaceable>path1</replaceable> <replaceable>name2</replaceable>
<replaceable>path2</replaceable>
<replaceable>...</replaceable>]</literal>. The references graph
of each <replaceable>pathN</replaceable> will be stored in a text
file <replaceable>nameN</replaceable> in the temporary build
directory. The text files have the format used by
<command>nix-store --register-validity</command> (with the deriver
fields left empty). For example, when the following derivation is
built:
<replaceable>path2</replaceable> <replaceable>...</replaceable>
]</literal>. The references graph of each
<replaceable>pathN</replaceable> will be stored in a text file
<replaceable>nameN</replaceable> in the temporary build directory.
The text files have the format used by <command>nix-store
--register-validity</command> (with the deriver fields left
empty). For example, when the following derivation is built:
<programlisting>
derivation {
...
exportReferencesGraph = ["libfoo-graph" libfoo];
exportReferencesGraph = [ "libfoo-graph" libfoo ];
};
</programlisting>
@@ -1571,14 +1567,14 @@ fetchurl {
<varname>fetchurl</varname>:
<programlisting>
{stdenv, curl}: # The <command>curl</command> program is used for downloading.
{ stdenv, curl }: # The <command>curl</command> program is used for downloading.
{url, md5}:
{ url, md5 }:
stdenv.mkDerivation {
name = baseNameOf (toString url);
builder = ./builder.sh;
buildInputs = [curl];
buildInputs = [ curl ];
# This is a fixed-output derivation; the output must be a regular
# file with MD5 hash <varname>md5</varname>.
@@ -1650,7 +1646,7 @@ stdenv.mkDerivation {
Nixpkgs has the line
<programlisting>
impureEnvVars = ["http_proxy" "https_proxy" <replaceable>...</replaceable>];
impureEnvVars = [ "http_proxy" "https_proxy" <replaceable>...</replaceable> ];
</programlisting>
to make it use the proxy server configuration specified by the
+42 -51
View File
@@ -1,38 +1,3 @@
# CWI ATerm
ATERM = aterm-2.5
$(ATERM).tar.gz:
@echo "Nix requires the CWI ATerm library to build."
@echo "Please download version 2.5 from"
@echo " http://nixos.org/tarballs/aterm-2.5.tar.gz"
@echo "and place it in the externals/ directory."
false
$(ATERM): $(ATERM).tar.gz
gzip -d < $(srcdir)/$(ATERM).tar.gz | tar xvf -
patch -d $(ATERM) -p1 < ./max-long.patch
patch -d $(ATERM) -p1 < ./sizeof.patch
have-aterm:
$(MAKE) $(ATERM)
touch have-aterm
if HAVE_ATERM
build-aterm:
else
build-aterm: have-aterm
(pfx=`pwd` && \
cd $(ATERM) && \
CC="$(CC)" ./configure --prefix=$$pfx/inst-aterm \
--disable-shared --enable-static && \
$(MAKE) && \
$(MAKE) check && \
$(MAKE) install)
touch build-aterm
endif
# bzip2
BZIP2 = bzip2-1.0.5
@@ -47,30 +12,56 @@ $(BZIP2).tar.gz:
$(BZIP2): $(BZIP2).tar.gz
gunzip < $(srcdir)/$(BZIP2).tar.gz | tar xvf -
have-bzip2:
$(MAKE) $(BZIP2)
touch have-bzip2
if HAVE_BZIP2
build-bzip2:
else
build-bzip2: have-bzip2
(pfx=`pwd` && \
cd $(BZIP2) && \
$(MAKE) && \
$(MAKE) install PREFIX=$$pfx/inst-bzip2)
build-bzip2: $(BZIP2)
(cd $(BZIP2) && \
$(MAKE) CC="$(CC)" && \
$(MAKE) install PREFIX=$(abs_builddir)/inst-bzip2)
touch build-bzip2
install:
install-exec-local:: build-bzip2
mkdir -p $(DESTDIR)${bzip2_bin}
$(INSTALL_PROGRAM) $(bzip2_bin_test)/bzip2 $(bzip2_bin_test)/bunzip2 $(DESTDIR)${bzip2_bin}
$(INSTALL_PROGRAM) $(bzip2_bin_test)/bzip2 $(DESTDIR)${bzip2_bin}
endif
all: build-aterm build-bzip2
# SQLite
EXTRA_DIST = $(ATERM).tar.gz $(BZIP2).tar.gz max-long.patch sizeof.patch
SQLITE = sqlite-autoconf-$(SQLITE_VERSION)
SQLITE_TAR = sqlite-autoconf-$(SQLITE_VERSION).tar.gz
ext-clean:
$(RM) -f have-aterm build-aterm have-bzip2 build-bzip2
$(RM) -rf $(ATERM) $(BZIP2)
$(SQLITE_TAR):
@echo "Nix requires the SQLite library to build."
@echo "Please download version $(SQLITE_VERSION) from"
@echo " http://www.sqlite.org/$(SQLITE_TAR)"
@echo "and place it in the externals/ directory."
false
$(SQLITE): $(SQLITE_TAR)
gzip -d < $(srcdir)/$(SQLITE_TAR) | tar xvf -
if HAVE_SQLITE
build-sqlite:
else
build-sqlite: $(SQLITE)
(cd $(SQLITE) && \
CC="$(CC)" CFLAGS="-DSQLITE_ENABLE_COLUMN_METADATA=1" ./configure --disable-static --prefix=$(pkglibdir)/dummy --libdir=${pkglibdir} $(SUB_CONFIGURE_FLAGS) && \
$(MAKE) )
touch build-sqlite
install-exec-local:: build-sqlite
cd $(SQLITE) && $(MAKE) install
rm -rf "$(DESTDIR)/$(pkglibdir)/dummy"
endif
all: build-bzip2 build-sqlite
EXTRA_DIST = $(BZIP2).tar.gz $(SQLITE_TAR)
clean:
$(RM) -f build-bzip2 build-sqlite
$(RM) -rf $(BZIP2) $(SQLITE)
$(RM) -rf inst-bzip2
-77
View File
@@ -1,77 +0,0 @@
diff -rc aterm-2.8-orig/aterm/hash.c aterm-2.8/aterm/hash.c
*** aterm-2.8-orig/aterm/hash.c 2008-11-10 13:54:22.000000000 +0100
--- aterm-2.8/aterm/hash.c 2009-01-27 18:14:14.000000000 +0100
***************
*** 93,146 ****
}
/*}}} */
- /*{{{ static long calc_long_max() */
- static long calc_long_max()
- {
- long try_long_max;
- long long_max;
- long delta;
-
- try_long_max = 1;
- do {
- long_max = try_long_max;
- try_long_max = long_max * 2;
- } while (try_long_max > 0);
-
- delta = long_max;
- while (delta > 1) {
- while (long_max + delta < 0) {
- delta /= 2;
- }
- long_max += delta;
- }
-
- return long_max;
-
- }
- /*}}} */
/*{{{ static long calculateNewSize(sizeMinus1, nrdel, nrentries) */
static long calculateNewSize
(long sizeMinus1, long nr_deletions, long nr_entries)
{
-
- /* Hack: LONG_MAX (limits.h) is often unreliable, we need to find
- * out the maximum possible value of a signed long dynamically.
- */
- static long st_long_max = 0;
-
- /* the resulting length has the form 2^k-1 */
-
if (nr_deletions >= nr_entries/2) {
return sizeMinus1;
}
! if (st_long_max == 0) {
! st_long_max = calc_long_max();
! }
!
! if (sizeMinus1 > st_long_max / 2) {
! return st_long_max-1;
}
return (2*sizeMinus1)+1;
--- 93,109 ----
}
/*}}} */
/*{{{ static long calculateNewSize(sizeMinus1, nrdel, nrentries) */
static long calculateNewSize
(long sizeMinus1, long nr_deletions, long nr_entries)
{
if (nr_deletions >= nr_entries/2) {
return sizeMinus1;
}
! if (sizeMinus1 > LONG_MAX / 2) {
! return LONG_MAX-1;
}
return (2*sizeMinus1)+1;
-56
View File
@@ -1,56 +0,0 @@
diff -rc -x '*~' aterm-2.5-orig/aterm/aterm.c aterm-2.5/aterm/aterm.c
*** aterm-2.5-orig/aterm/aterm.c 2007-02-27 23:41:31.000000000 +0100
--- aterm-2.5/aterm/aterm.c 2010-02-23 15:10:38.000000000 +0100
***************
*** 150,155 ****
--- 150,157 ----
if (initialized)
return;
+ assert(sizeof(long) == sizeof(void *));
+
/*{{{ Handle arguments */
for (lcv=1; lcv < argc; lcv++) {
diff -rc -x '*~' aterm-2.5-orig/aterm/encoding.h aterm-2.5/aterm/encoding.h
*** aterm-2.5-orig/aterm/encoding.h 2007-02-27 23:41:31.000000000 +0100
--- aterm-2.5/aterm/encoding.h 2010-02-23 15:36:05.000000000 +0100
***************
*** 10,24 ****
{
#endif/* __cplusplus */
! #if SIZEOF_LONG > 4
! #define AT_64BIT
#endif
! #if SIZEOF_LONG != SIZEOF_VOID_P
! #error Size of long is not the same as the size of a pointer
#endif
! #if SIZEOF_INT > 4
#error Size of int is not 32 bits
#endif
--- 10,30 ----
{
#endif/* __cplusplus */
! #include <limits.h>
!
! #ifndef SIZEOF_LONG
! #if ULONG_MAX > 4294967295
! #define SIZEOF_LONG 8
! #else
! #define SIZEOF_LONG 4
! #endif
#endif
! #if SIZEOF_LONG > 4
! #define AT_64BIT
#endif
! #if UINT_MAX > 4294967295
#error Size of int is not 32 bits
#endif
-30
View File
@@ -1,30 +0,0 @@
{sharedLib ? true}:
rec {
inherit (import ../../../lib) compileC makeLibrary;
sources = [
./afun.c
./aterm.c
./bafio.c
./byteio.c
./gc.c
./hash.c
./list.c
./make.c
./md5c.c
./memory.c
./tafio.c
./version.c
];
compile = main: compileC {inherit main sharedLib;};
libATerm = makeLibrary {
libraryName = "ATerm";
objects = map compile sources;
inherit sharedLib;
};
}
-1
View File
@@ -1 +0,0 @@
import test/default.nix
-15
View File
@@ -1,15 +0,0 @@
with (import ../../../lib);
let {
inherit (import ../aterm {}) libATerm;
compileTest = main: link {
objects = [(compileC {inherit main; localIncludePath = [ ../aterm ];})];
libraries = libATerm;
};
body = [
(compileTest ./fib.c)
(compileTest ./primes.c)
];
}
-6
View File
@@ -1,6 +0,0 @@
[ (import ./trivial)
(import ./simple-header)
(import ./not-so-simple-header)
(import ./not-so-simple-header-auto)
(import ./aterm)
]
@@ -1 +0,0 @@
#define WHAT "World"
@@ -1,13 +0,0 @@
with import ../../lib;
let {
hello = link {programName = "hello"; objects = compileC {
main = ./foo/hello.c;
localIncludes = "auto";
};};
# body = findIncludes {main = ./foo/hello.c;};
body = [hello];
}
@@ -1,3 +0,0 @@
#define HELLO "Hello"
#include "../../bar/hello.h"
@@ -1,9 +0,0 @@
#include <stdio.h>
#include "fnord/indirect.h"
int main(int argc, char * * argv)
{
printf(HELLO " " WHAT "\n");
return 0;
}
@@ -1 +0,0 @@
#define WHAT "World"
@@ -1,14 +0,0 @@
let {
inherit (import ../../lib) compileC link;
hello = link {programName = "hello"; objects = compileC {
main = ./foo/hello.c;
localIncludes = [
[./foo/fnord/indirect.h "fnord/indirect.h"]
[./bar/hello.h "fnord/../../bar/hello.h"]
];
};};
body = [hello];
}
@@ -1,3 +0,0 @@
#define HELLO "Hello"
#include "../../bar/hello.h"
@@ -1,9 +0,0 @@
#include <stdio.h>
#include "fnord/indirect.h"
int main(int argc, char * * argv)
{
printf(HELLO " " WHAT "\n");
return 0;
}
-11
View File
@@ -1,11 +0,0 @@
let {
inherit (import ../../lib) compileC link;
hello = link {objects = compileC {
main = ./hello.c;
localIncludes = [ [./hello.h "hello.h"] ];
};};
body = [hello];
}
-9
View File
@@ -1,9 +0,0 @@
#include <stdio.h>
#include "hello.h"
int main(int argc, char * * argv)
{
printf("Hello " WHAT "\n");
return 0;
}
-1
View File
@@ -1 +0,0 @@
#define WHAT "World"
-8
View File
@@ -1,8 +0,0 @@
let {
inherit (import ../../lib) compileC link;
hello = link {objects = compileC {main = ./hello.c;};};
body = [hello];
}
-7
View File
@@ -1,7 +0,0 @@
#include <stdio.h>
int main(int argc, char * * argv)
{
printf("Hello World\n");
return 0;
}
-73
View File
@@ -1,73 +0,0 @@
. $stdenv/setup
mainName=$(basename $main | cut -c34-)
echo "compiling \`$mainName'..."
# Turn $localIncludes into an array.
localIncludes=($localIncludes)
# Determine how many `..' levels appear in the header file references.
# E.g., if there is some reference `../../foo.h', then we have to
# insert two extra levels in the directory structure, so that `a.c' is
# stored at `dotdot/dotdot/a.c', and a reference from it to
# `../../foo.h' resolves to `dotdot/dotdot/../../foo.h' == `foo.h'.
n=0
maxDepth=0
for ((n = 0; n < ${#localIncludes[*]}; n += 2)); do
target=${localIncludes[$((n + 1))]}
# Split the target name into path components using some IFS magic.
savedIFS="$IFS"
IFS=/
components=($target)
depth=0
for ((m = 0; m < ${#components[*]}; m++)); do
c=${components[m]}
if test "$c" = ".."; then
depth=$((depth + 1))
fi
done
IFS="$savedIFS"
if test $depth -gt $maxDepth; then
maxDepth=$depth;
fi
done
# Create the extra levels in the directory hierarchy.
prefix=
for ((n = 0; n < maxDepth; n++)); do
prefix="dotdot/$prefix"
done
# Create symlinks to the header files.
for ((n = 0; n < ${#localIncludes[*]}; n += 2)); do
source=${localIncludes[n]}
target=${localIncludes[$((n + 1))]}
# Create missing directories. We use IFS magic to split the path
# into path components.
savedIFS="$IFS"
IFS=/
components=($prefix$target)
fullPath=(.)
for ((m = 0; m < ${#components[*]} - 1; m++)); do
fullPath=("${fullPath[@]}" ${components[m]})
if ! test -d "${fullPath[*]}"; then
mkdir "${fullPath[*]}"
fi
done
IFS="$savedIFS"
ln -sf $source $prefix$target
done
# Create a symlink to the main file.
if ! test "$(readlink $prefix$mainName)" = $main; then
ln -s $main $prefix$mainName
fi
mkdir $out
test "$prefix" && cd $prefix
gcc -Wall $cFlags -c $mainName -o $out/$mainName.o
-69
View File
@@ -1,69 +0,0 @@
rec {
# Should point at your Nixpkgs installation.
pkgPath = ./pkgs;
pkgs = import (pkgPath + /system/all-packages.nix) {};
stdenv = pkgs.stdenv;
compileC =
{ main
, localIncludes ? "auto"
, localIncludePath ? []
, cFlags ? ""
, sharedLib ? false
}:
stdenv.mkDerivation {
name = "compile-c";
builder = ./compile-c.sh;
localIncludes =
if localIncludes == "auto" then
dependencyClosure {
scanner = main:
import (findIncludes {
inherit main;
});
searchPath = localIncludePath;
startSet = [main];
}
else
localIncludes;
inherit main;
cFlags = [
cFlags
(if sharedLib then ["-fpic"] else [])
(map (p: "-I" + (relativise (dirOf main) p)) localIncludePath)
];
};
findIncludes = {main}: stdenv.mkDerivation {
name = "find-includes";
realBuilder = pkgs.perl ~ "bin/perl";
args = [ ./find-includes.pl ];
inherit main;
};
link = {objects, programName ? "program", libraries ? []}: stdenv.mkDerivation {
name = "link";
builder = ./link.sh;
inherit objects programName libraries;
};
makeLibrary = {objects, libraryName ? [], sharedLib ? false}:
# assert sharedLib -> fold (obj: x: assert obj.sharedLib && x) false objects
stdenv.mkDerivation {
name = "library";
builder = ./make-library.sh;
inherit objects libraryName sharedLib;
};
}
-21
View File
@@ -1,21 +0,0 @@
use strict;
my $root = $ENV{"main"};
my $out = $ENV{"out"};
open OUT, ">$out" or die "$!";
print OUT "[\n";
open IN, "<$root" or die "$!";
while (<IN>) {
if (/^\#include\s+\"(.*)\"/) {
print OUT "\"$1\"\n";
}
if (/^\#include\s+\<(.*)\>/) {
print OUT "\"$1\"\n";
}
}
close IN;
print OUT "]\n";
close OUT;
-21
View File
@@ -1,21 +0,0 @@
. $stdenv/setup
shopt -s nullglob
objs=
for i in $objects; do
obj=$(echo $i/*.o)
objs="$objs $obj"
done
libs=
for i in $libraries; do
lib=$(echo $i/*.a; echo $i/*.so)
name=$(echo $(basename $lib) | sed -e 's/^lib//' -e 's/.a$//' -e 's/.so$//')
libs="$libs -L$(dirname $lib) -l$name"
done
echo "linking object files into \`$programName'..."
mkdir $out
gcc -o $out/$programName $objs $libs
-28
View File
@@ -1,28 +0,0 @@
. $stdenv/setup
objs=
for i in $objects; do
obj=$(echo $i/*.o)
objs="$objs $obj"
done
echo "archiving object files into library \`$libraryName'..."
ensureDir $out
if test -z "$sharedLib"; then
outPath=$out/lib${libraryName}.a
ar crs $outPath $objs
ranlib $outPath
else
outPath=$out/lib${libraryName}.so
gcc -shared -o $outPath $objs
fi
+5 -1
View File
@@ -67,7 +67,7 @@ The hook `nix-mode-hook' is run when Nix mode is started.
(defvar nix-keywords
'("\\<if\\>" "\\<then\\>" "\\<else\\>" "\\<assert\\>" "\\<with\\>"
"\\<let\\>" "\\<in\\>" "\\<rec\\>" "\\<inherit\\>"
"\\<let\\>" "\\<in\\>" "\\<rec\\>" "\\<inherit\\>" "\\<or\\>"
("\\<true\\>" . font-lock-builtin-face)
("\\<false\\>" . font-lock-builtin-face)
("\\<null\\>" . font-lock-builtin-face)
@@ -80,6 +80,8 @@ The hook `nix-mode-hook' is run when Nix mode is started.
. font-lock-constant-face)
("\\<\\([a-zA-Z_][a-zA-Z0-9_'\.]*\\)[ \t]*="
(1 font-lock-variable-name-face nil nil))
("<[a-zA-Z0-9._\\+-]+\\(/[a-zA-Z0-9._\\+-]+\\)*>"
. font-lock-constant-face)
("[a-zA-Z0-9._\\+-]*\\(/[a-zA-Z0-9._\\+-]+\\)+"
. font-lock-constant-face)
))
@@ -107,3 +109,5 @@ The hook `nix-mode-hook' is run when Nix mode is started.
(setq auto-mode-alist (cons '("\\.nix\\'" . nix-mode) auto-mode-alist))
(setq auto-mode-alist (cons '("\\.nix.in\\'" . nix-mode) auto-mode-alist))
(provide 'nix-mode)
+11
View File
@@ -59,6 +59,17 @@
#build-max-jobs = 1
### Option `build-cores'
#
# This option defines the number of CPU cores to utilize in parallel
# within a build job, i.e. by passing an appropriate `-jN' flag to GNU
# Make. The default is 1, meaning that parallel building within jobs
# is disabled. Passing the special value `0' causes Nix to try and
# auto-detect the number of available cores on the local host. This
# setting can be overridden using the `--cores' command line switch.
#build-cores = 1
### Option `build-max-silent-time'
#
# This option defines the maximum number of seconds that a builder can
+2
View File
@@ -19,6 +19,8 @@ BuildRoot: %{_tmppath}/%{name}-%{version}-buildroot
Prefix: /usr
Requires: /usr/bin/perl
Requires: curl
Requires: perl-DBD-SQLite
Requires: perl-devel
# Hack to make that shitty RPM scanning hack shut up.
Provides: perl(readmanifest)
+7
View File
@@ -0,0 +1,7 @@
Changes
Makefile.PL
MANIFEST
Nix.xs
README
t/Nix.t
lib/Nix.pm
+34
View File
@@ -0,0 +1,34 @@
perlversion := $(shell perl -e 'use Config; print $$Config{version};')
perlarchname := $(shell perl -e 'use Config; print $$Config{archname};')
perllibdir = $(libdir)/perl5/site_perl/$(perlversion)/$(perlarchname)
PERL_MODULES = lib/Nix/Store.pm lib/Nix/Manifest.pm lib/Nix/GeneratePatches.pm lib/Nix/SSH.pm lib/Nix/CopyClosure.pm lib/Nix/Config.pm.in
# Hack required by "make check".
all: $(PERL_MODULES:.in=)
mkdir -p lib/auto/Nix/Store
ln -sfn $(abs_builddir)/.libs/libNixStore.so lib/auto/Nix/Store/Store.so
install-exec-local: $(PERL_MODULES:.in=)
$(INSTALL) -d $(DESTDIR)$(perllibdir)/Nix
$(INSTALL_DATA) $(PERL_MODULES:.in=) $(DESTDIR)$(perllibdir)/Nix
$(INSTALL) -d $(DESTDIR)$(perllibdir)/auto/Nix/Store
ln -sfn $(pkglibdir)/libNixStore.so $(DESTDIR)$(perllibdir)/auto/Nix/Store/Store.so
# Awful hackery to get libtool to build Perl XS bindings.
pkglib_LTLIBRARIES = libNixStore.la
libNixStore_la_SOURCES = lib/Nix/Store.cc
libNixStore_la_LIBADD = $(top_srcdir)/src/libstore/libstore.la
AM_CXXFLAGS = \
-I$(top_srcdir)/src -I$(top_srcdir)/src/libutil -I$(top_srcdir)/src/libstore \
-I$(shell perl -e 'use Config; print $$Config{archlibexp};')/CORE
lib/Nix/Store.cc: lib/Nix/Store.xs
xsubpp $^ -output $@
EXTRA_DIST = $(PERL_MODULES) lib/Nix/Store.xs
include ../substitute.mk
+27
View File
@@ -0,0 +1,27 @@
package Nix::Config;
$binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
$libexecDir = $ENV{"NIX_LIBEXEC_DIR"} || "@libexecdir@";
$stateDir = $ENV{"NIX_STATE_DIR"} || "@localstatedir@/nix";
$manifestDir = $ENV{"NIX_MANIFESTS_DIR"} || "@localstatedir@/nix/manifests";
$logDir = $ENV{"NIX_LOG_DIR"} || "@localstatedir@/log/nix";
$confDir = $ENV{"NIX_CONF_DIR"} || "@sysconfdir@/nix";
$bzip2 = $ENV{"NIX_BZIP2"} || "@bzip2@";
$curl = "@curl@";
sub readConfig {
my %config;
my $config = "@sysconfdir@/nix/nix.conf";
return unless -f $config;
open CONFIG, "<$config" or die "cannot open `$config'";
while (<CONFIG>) {
/^\s*([\w|-]+)\s*=\s*(.*)$/ or next;
$config{$1} = $2;
print "|$1| -> |$2|\n";
}
close CONFIG;
}
return 1;
+46
View File
@@ -0,0 +1,46 @@
package Nix::CopyClosure;
use strict;
use Nix::Config;
use Nix::Store;
sub copyTo {
my ($sshHost, $sshOpts, $storePaths, $compressor, $decompressor, $includeOutputs, $dryRun, $sign) = @_;
$compressor = "$compressor |" if $compressor ne "";
$decompressor = "$decompressor |" if $decompressor ne "";
# Get the closure of this path.
my @closure = reverse(topoSortPaths(computeFSClosure(0, $includeOutputs,
map { followLinksToStorePath $_ } @{$storePaths})));
# Ask the remote host which paths are invalid. Because of limits
# to the command line length, do this in chunks. Eventually,
# we'll want to use ‘--from-stdin’, but we can't rely on the
# target having this option yet.
my @missing = ();
while (scalar(@closure) > 0) {
my @ps = splice(@closure, 0, 1500);
open(READ, "set -f; ssh $sshHost @{$sshOpts} nix-store --check-validity --print-invalid @ps|");
while (<READ>) {
chomp;
push @missing, $_;
}
close READ or die;
}
# Export the store paths and import them on the remote machine.
if (scalar @missing > 0) {
print STDERR "copying ", scalar @missing, " missing paths to ‘$sshHost’...\n";
#print STDERR " $_\n" foreach @missing;
unless ($dryRun) {
open SSH, "| $compressor ssh $sshHost @{$sshOpts} '$decompressor nix-store --import'" or die;
exportPaths(fileno(SSH), $sign, @missing);
close SSH or die "copying store paths to remote machine `$sshHost' failed: $?";
}
}
}
1;
+340
View File
@@ -0,0 +1,340 @@
package Nix::GeneratePatches;
use strict;
use File::Temp qw(tempdir);
use File::stat;
use Nix::Config;
use Nix::Manifest;
our @ISA = qw(Exporter);
our @EXPORT = qw(generatePatches propagatePatches copyPatches);
# Some patch generations options.
# Max size of NAR archives to generate patches for.
my $maxNarSize = $ENV{"NIX_MAX_NAR_SIZE"};
$maxNarSize = 160 * 1024 * 1024 if !defined $maxNarSize;
# If patch is bigger than this fraction of full archive, reject.
my $maxPatchFraction = $ENV{"NIX_PATCH_FRACTION"};
$maxPatchFraction = 0.60 if !defined $maxPatchFraction;
my $timeLimit = $ENV{"NIX_BSDIFF_TIME_LIMIT"};
$timeLimit = 180 if !defined $timeLimit;
my $hashAlgo = "sha256";
sub findOutputPaths {
my $narFiles = shift;
my %outPaths;
foreach my $p (keys %{$narFiles}) {
# Ignore derivations.
next if ($p =~ /\.drv$/);
# Ignore builders (too much ambiguity -- they're all called
# `builder.sh').
next if ($p =~ /\.sh$/);
next if ($p =~ /\.patch$/);
# Don't bother including tar files etc.
next if ($p =~ /\.tar$/ || $p =~ /\.tar\.(gz|bz2|Z|lzma|xz)$/ || $p =~ /\.zip$/ || $p =~ /\.bin$/ || $p =~ /\.tgz$/ || $p =~ /\.rpm$/ || $p =~ /cvs-export$/ || $p =~ /fetchhg$/);
$outPaths{$p} = 1;
}
return %outPaths;
}
sub getNameVersion {
my $p = shift;
$p =~ /\/[0-9a-z]+((?:-[a-zA-Z][^\/-]*)+)([^\/]*)$/;
my $name = $1;
my $version = $2;
return undef unless defined $name && defined $version;
$name =~ s/^-//;
$version =~ s/^-//;
return ($name, $version);
}
# A quick hack to get a measure of the `distance' between two
# versions: it's just the position of the first character that differs
# (or 999 if they are the same).
sub versionDiff {
my $s = shift;
my $t = shift;
my $i;
return 999 if $s eq $t;
for ($i = 0; $i < length $s; $i++) {
return $i if $i >= length $t or
substr($s, $i, 1) ne substr($t, $i, 1);
}
return $i;
}
sub getNarBz2 {
my $narPath = shift;
my $narFiles = shift;
my $storePath = shift;
my $narFileList = $$narFiles{$storePath};
die "missing path $storePath" unless defined $narFileList;
my $narFile = @{$narFileList}[0];
die unless defined $narFile;
$narFile->{url} =~ /\/([^\/]+)$/;
die unless defined $1;
return "$narPath/$1";
}
sub containsPatch {
my $patches = shift;
my $storePath = shift;
my $basePath = shift;
my $patchList = $$patches{$storePath};
return 0 if !defined $patchList;
my $found = 0;
foreach my $patch (@{$patchList}) {
# !!! baseHash might differ
return 1 if $patch->{basePath} eq $basePath;
}
return 0;
}
sub generatePatches {
my ($srcNarFiles, $dstNarFiles, $srcPatches, $dstPatches, $narPath, $patchesPath, $patchesURL, $tmpDir) = @_;
my %srcOutPaths = findOutputPaths $srcNarFiles;
my %dstOutPaths = findOutputPaths $dstNarFiles;
# For each output path in the destination, see if we need to / can
# create a patch.
print STDERR "creating patches...\n";
foreach my $p (keys %dstOutPaths) {
# If exactly the same path already exists in the source, skip it.
next if defined $srcOutPaths{$p};
print " $p\n";
# If not, then we should find the paths in the source that are
# `most' likely to be present on a system that wants to
# install this path.
(my $name, my $version) = getNameVersion $p;
next unless defined $name && defined $version;
my @closest = ();
my $closestVersion;
my $minDist = -1; # actually, larger means closer
# Find all source paths with the same name.
foreach my $q (keys %srcOutPaths) {
(my $name2, my $version2) = getNameVersion $q;
next unless defined $name2 && defined $version2;
if ($name eq $name2) {
my $srcSystem = @{$$dstNarFiles{$p}}[0]->{system};
my $dstSystem = @{$$srcNarFiles{$q}}[0]->{system};
if (defined $srcSystem && defined $dstSystem && $srcSystem ne $dstSystem) {
print " SKIPPING $q due to different systems ($srcSystem vs. $dstSystem)\n";
next;
}
# If the sizes differ too much, then skip. This
# disambiguates between, e.g., a real component and a
# wrapper component (cf. Firefox in Nixpkgs).
my $srcSize = @{$$srcNarFiles{$q}}[0]->{size};
my $dstSize = @{$$dstNarFiles{$p}}[0]->{size};
my $ratio = $srcSize / $dstSize;
$ratio = 1 / $ratio if $ratio < 1;
# print " SIZE $srcSize $dstSize $ratio $q\n";
if ($ratio >= 3) {
print " SKIPPING $q due to size ratio $ratio ($srcSize vs. $dstSize)\n";
next;
}
# If there are multiple matching names, include the
# ones with the closest version numbers.
my $dist = versionDiff $version, $version2;
if ($dist > $minDist) {
$minDist = $dist;
@closest = ($q);
$closestVersion = $version2;
} elsif ($dist == $minDist) {
push @closest, $q;
}
}
}
if (scalar(@closest) == 0) {
print " NO BASE: $p\n";
next;
}
foreach my $closest (@closest) {
# Generate a patch between $closest and $p.
print STDERR " $p <- $closest\n";
# If the patch already exists, skip it.
if (containsPatch($srcPatches, $p, $closest) ||
containsPatch($dstPatches, $p, $closest))
{
print " skipping, already exists\n";
next;
}
my $srcNarBz2 = getNarBz2 $narPath, $srcNarFiles, $closest;
my $dstNarBz2 = getNarBz2 $narPath, $dstNarFiles, $p;
if (! -f $srcNarBz2) {
warn "patch source archive $srcNarBz2 is missing\n";
next;
}
system("$Nix::Config::bzip2 -d < $srcNarBz2 > $tmpDir/A") == 0
or die "cannot unpack $srcNarBz2";
if (stat("$tmpDir/A")->size >= $maxNarSize) {
print " skipping, source is too large\n";
next;
}
system("$Nix::Config::bzip2 -d < $dstNarBz2 > $tmpDir/B") == 0
or die "cannot unpack $dstNarBz2";
if (stat("$tmpDir/B")->size >= $maxNarSize) {
print " skipping, destination is too large\n";
next;
}
my $time1 = time();
my $res = system("ulimit -t $timeLimit; $Nix::Config::libexecDir/bsdiff $tmpDir/A $tmpDir/B $tmpDir/DIFF");
my $time2 = time();
if ($res) {
warn "binary diff computation aborted after ", $time2 - $time1, " seconds\n";
next;
}
my $baseHash = `$Nix::Config::binDir/nix-hash --flat --type $hashAlgo --base32 $tmpDir/A` or die;
chomp $baseHash;
my $narHash = `$Nix::Config::binDir/nix-hash --flat --type $hashAlgo --base32 $tmpDir/B` or die;
chomp $narHash;
my $narDiffHash = `$Nix::Config::binDir/nix-hash --flat --type $hashAlgo --base32 $tmpDir/DIFF` or die;
chomp $narDiffHash;
my $narDiffSize = stat("$tmpDir/DIFF")->size;
my $dstNarBz2Size = stat($dstNarBz2)->size;
print " size $narDiffSize; full size $dstNarBz2Size; ", $time2 - $time1, " seconds\n";
if ($narDiffSize >= $dstNarBz2Size) {
print " rejecting; patch bigger than full archive\n";
next;
}
if ($narDiffSize / $dstNarBz2Size >= $maxPatchFraction) {
print " rejecting; patch too large relative to full archive\n";
next;
}
my $finalName = "$narDiffHash.nar-bsdiff";
if (-e "$patchesPath/$finalName") {
print " not copying, already exists\n";
}
else {
system("cp '$tmpDir/DIFF' '$patchesPath/$finalName.tmp'") == 0
or die "cannot copy diff";
rename("$patchesPath/$finalName.tmp", "$patchesPath/$finalName")
or die "cannot rename $patchesPath/$finalName.tmp";
}
# Add the patch to the manifest.
addPatch $dstPatches, $p,
{ url => "$patchesURL/$finalName", hash => "$hashAlgo:$narDiffHash"
, size => $narDiffSize, basePath => $closest, baseHash => "$hashAlgo:$baseHash"
, narHash => "$hashAlgo:$narHash", patchType => "nar-bsdiff"
};
}
}
}
# Propagate useful patches from $srcPatches to $dstPatches. A patch
# is useful if it produces either paths in the $dstNarFiles or paths
# that can be used as the base for other useful patches.
sub propagatePatches {
my ($srcPatches, $dstNarFiles, $dstPatches) = @_;
print STDERR "propagating patches...\n";
my $changed;
do {
# !!! we repeat this to reach the transitive closure; inefficient
$changed = 0;
print STDERR "loop\n";
my %dstBasePaths;
foreach my $q (keys %{$dstPatches}) {
foreach my $patch (@{$$dstPatches{$q}}) {
$dstBasePaths{$patch->{basePath}} = 1;
}
}
foreach my $p (keys %{$srcPatches}) {
my $patchList = $$srcPatches{$p};
my $include = 0;
# Is path $p included in the destination? If so, include
# patches that produce it.
$include = 1 if defined $$dstNarFiles{$p};
# Is path $p a path that serves as a base for paths in the
# destination? If so, include patches that produce it.
# !!! check baseHash
$include = 1 if defined $dstBasePaths{$p};
if ($include) {
foreach my $patch (@{$patchList}) {
$changed = 1 if addPatch $dstPatches, $p, $patch;
}
}
}
} while $changed;
}
# Add all new patches in $srcPatches to $dstPatches.
sub copyPatches {
my ($srcPatches, $dstPatches) = @_;
foreach my $p (keys %{$srcPatches}) {
addPatch $dstPatches, $p, $_ foreach @{$$srcPatches{$p}};
}
}
return 1;
+367
View File
@@ -0,0 +1,367 @@
package Nix::Manifest;
use strict;
use DBI;
use Cwd;
use File::stat;
use File::Path;
use Fcntl ':flock';
use Nix::Config;
our @ISA = qw(Exporter);
our @EXPORT = qw(readManifest writeManifest updateManifestDB addPatch);
sub addNAR {
my ($narFiles, $storePath, $info) = @_;
$$narFiles{$storePath} = []
unless defined $$narFiles{$storePath};
my $narFileList = $$narFiles{$storePath};
my $found = 0;
foreach my $narFile (@{$narFileList}) {
$found = 1 if $narFile->{url} eq $info->{url};
}
push @{$narFileList}, $info if !$found;
}
sub addPatch {
my ($patches, $storePath, $patch) = @_;
$$patches{$storePath} = []
unless defined $$patches{$storePath};
my $patchList = $$patches{$storePath};
my $found = 0;
foreach my $patch2 (@{$patchList}) {
$found = 1 if
$patch2->{url} eq $patch->{url} &&
$patch2->{basePath} eq $patch->{basePath};
}
push @{$patchList}, $patch if !$found;
return !$found;
}
sub readManifest_ {
my ($manifest, $addNAR, $addPatch) = @_;
# Decompress the manifest if necessary.
if ($manifest =~ /\.bz2$/) {
open MANIFEST, "$Nix::Config::bzip2 -d < $manifest |"
or die "cannot decompress `$manifest': $!";
} else {
open MANIFEST, "<$manifest"
or die "cannot open `$manifest': $!";
}
my $inside = 0;
my $type;
my $manifestVersion = 2;
my ($storePath, $url, $hash, $size, $basePath, $baseHash, $patchType);
my ($narHash, $narSize, $references, $deriver, $copyFrom, $system);
while (<MANIFEST>) {
chomp;
s/\#.*$//g;
next if (/^$/);
if (!$inside) {
if (/^\s*(\w*)\s*\{$/) {
$type = $1;
$type = "narfile" if $type eq "";
$inside = 1;
undef $storePath;
undef $url;
undef $hash;
undef $size;
undef $narHash;
undef $narSize;
undef $basePath;
undef $baseHash;
undef $patchType;
undef $system;
$references = "";
$deriver = "";
}
} else {
if (/^\}$/) {
$inside = 0;
if ($type eq "narfile") {
&$addNAR($storePath,
{ url => $url, hash => $hash, size => $size
, narHash => $narHash, narSize => $narSize
, references => $references
, deriver => $deriver
, system => $system
});
}
elsif ($type eq "patch") {
&$addPatch($storePath,
{ url => $url, hash => $hash, size => $size
, basePath => $basePath, baseHash => $baseHash
, narHash => $narHash, narSize => $narSize
, patchType => $patchType
});
}
}
elsif (/^\s*StorePath:\s*(\/\S+)\s*$/) { $storePath = $1; }
elsif (/^\s*CopyFrom:\s*(\/\S+)\s*$/) { $copyFrom = $1; }
elsif (/^\s*Hash:\s*(\S+)\s*$/) { $hash = $1; }
elsif (/^\s*URL:\s*(\S+)\s*$/) { $url = $1; }
elsif (/^\s*Size:\s*(\d+)\s*$/) { $size = $1; }
elsif (/^\s*BasePath:\s*(\/\S+)\s*$/) { $basePath = $1; }
elsif (/^\s*BaseHash:\s*(\S+)\s*$/) { $baseHash = $1; }
elsif (/^\s*Type:\s*(\S+)\s*$/) { $patchType = $1; }
elsif (/^\s*NarHash:\s*(\S+)\s*$/) { $narHash = $1; }
elsif (/^\s*NarSize:\s*(\d+)\s*$/) { $narSize = $1; }
elsif (/^\s*References:\s*(.*)\s*$/) { $references = $1; }
elsif (/^\s*Deriver:\s*(\S+)\s*$/) { $deriver = $1; }
elsif (/^\s*ManifestVersion:\s*(\d+)\s*$/) { $manifestVersion = $1; }
elsif (/^\s*System:\s*(\S+)\s*$/) { $system = $1; }
# Compatibility;
elsif (/^\s*NarURL:\s*(\S+)\s*$/) { $url = $1; }
elsif (/^\s*MD5:\s*(\S+)\s*$/) { $hash = "md5:$1"; }
}
}
close MANIFEST;
return $manifestVersion;
}
sub readManifest {
my ($manifest, $narFiles, $patches) = @_;
readManifest_($manifest,
sub { addNAR($narFiles, @_); },
sub { addPatch($patches, @_); } );
}
sub writeManifest {
my ($manifest, $narFiles, $patches, $noCompress) = @_;
open MANIFEST, ">$manifest.tmp"; # !!! check exclusive
print MANIFEST "version {\n";
print MANIFEST " ManifestVersion: 3\n";
print MANIFEST "}\n";
foreach my $storePath (sort (keys %{$narFiles})) {
my $narFileList = $$narFiles{$storePath};
foreach my $narFile (@{$narFileList}) {
print MANIFEST "{\n";
print MANIFEST " StorePath: $storePath\n";
print MANIFEST " NarURL: $narFile->{url}\n";
print MANIFEST " Hash: $narFile->{hash}\n" if defined $narFile->{hash};
print MANIFEST " Size: $narFile->{size}\n" if defined $narFile->{size};
print MANIFEST " NarHash: $narFile->{narHash}\n";
print MANIFEST " NarSize: $narFile->{narSize}\n" if $narFile->{narSize};
print MANIFEST " References: $narFile->{references}\n"
if defined $narFile->{references} && $narFile->{references} ne "";
print MANIFEST " Deriver: $narFile->{deriver}\n"
if defined $narFile->{deriver} && $narFile->{deriver} ne "";
print MANIFEST " System: $narFile->{system}\n" if defined $narFile->{system};
print MANIFEST "}\n";
}
}
foreach my $storePath (sort (keys %{$patches})) {
my $patchList = $$patches{$storePath};
foreach my $patch (@{$patchList}) {
print MANIFEST "patch {\n";
print MANIFEST " StorePath: $storePath\n";
print MANIFEST " NarURL: $patch->{url}\n";
print MANIFEST " Hash: $patch->{hash}\n";
print MANIFEST " Size: $patch->{size}\n";
print MANIFEST " NarHash: $patch->{narHash}\n";
print MANIFEST " NarSize: $patch->{narSize}\n" if $patch->{narSize};
print MANIFEST " BasePath: $patch->{basePath}\n";
print MANIFEST " BaseHash: $patch->{baseHash}\n";
print MANIFEST " Type: $patch->{patchType}\n";
print MANIFEST "}\n";
}
}
close MANIFEST;
rename("$manifest.tmp", $manifest)
or die "cannot rename $manifest.tmp: $!";
# Create a bzipped manifest.
unless (defined $noCompress) {
system("$Nix::Config::bzip2 < $manifest > $manifest.bz2.tmp") == 0
or die "cannot compress manifest";
rename("$manifest.bz2.tmp", "$manifest.bz2")
or die "cannot rename $manifest.bz2.tmp: $!";
}
}
sub updateManifestDB {
my $manifestDir = $Nix::Config::manifestDir;
mkpath($manifestDir);
my $dbPath = "$manifestDir/cache.sqlite";
# Open/create the database.
our $dbh = DBI->connect("dbi:SQLite:dbname=$dbPath", "", "")
or die "cannot open database `$dbPath'";
$dbh->{RaiseError} = 1;
$dbh->{PrintError} = 0;
$dbh->do("pragma foreign_keys = on");
$dbh->do("pragma synchronous = off"); # we can always reproduce the cache
$dbh->do("pragma journal_mode = truncate");
# Initialise the database schema, if necessary.
$dbh->do(<<EOF);
create table if not exists Manifests (
id integer primary key autoincrement not null,
path text unique not null,
timestamp integer not null
);
EOF
$dbh->do(<<EOF);
create table if not exists NARs (
id integer primary key autoincrement not null,
manifest integer not null,
storePath text not null,
url text not null,
hash text,
size integer,
narHash text,
narSize integer,
refs text,
deriver text,
system text,
foreign key (manifest) references Manifests(id) on delete cascade
);
EOF
$dbh->do("create index if not exists NARs_storePath on NARs(storePath)");
$dbh->do(<<EOF);
create table if not exists Patches (
id integer primary key autoincrement not null,
manifest integer not null,
storePath text not null,
basePath text not null,
baseHash text not null,
url text not null,
hash text,
size integer,
narHash text,
narSize integer,
patchType text not null,
foreign key (manifest) references Manifests(id) on delete cascade
);
EOF
$dbh->do("create index if not exists Patches_storePath on Patches(storePath)");
# Acquire an exclusive lock to ensure that only one process
# updates the DB at the same time. This isn't really necessary,
# but it prevents work duplication and lock contention in SQLite.
my $lockFile = "$manifestDir/cache.lock";
open MAINLOCK, ">>$lockFile" or die "unable to acquire lock ‘$lockFile’: $!\n";
flock(MAINLOCK, LOCK_EX) or die;
our $insertNAR = $dbh->prepare(
"insert into NARs(manifest, storePath, url, hash, size, narHash, " .
"narSize, refs, deriver, system) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)") or die;
our $insertPatch = $dbh->prepare(
"insert into Patches(manifest, storePath, basePath, baseHash, url, hash, " .
"size, narHash, narSize, patchType) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
$dbh->begin_work;
# Read each manifest in $manifestDir and add it to the database,
# unless we've already done so on a previous run.
my %seen;
for my $manifestLink (glob "$manifestDir/*.nixmanifest") {
my $manifest = Cwd::abs_path($manifestLink);
next unless -f $manifest;
my $timestamp = lstat($manifest)->mtime;
$seen{$manifest} = 1;
next if scalar @{$dbh->selectcol_arrayref(
"select 1 from Manifests where path = ? and timestamp = ?",
{}, $manifest, $timestamp)} == 1;
print STDERR "caching $manifest...\n";
$dbh->do("delete from Manifests where path = ?", {}, $manifest);
$dbh->do("insert into Manifests(path, timestamp) values (?, ?)",
{}, $manifest, $timestamp);
our $id = $dbh->last_insert_id("", "", "", "");
sub addNARToDB {
my ($storePath, $narFile) = @_;
$insertNAR->execute(
$id, $storePath, $narFile->{url}, $narFile->{hash}, $narFile->{size},
$narFile->{narHash}, $narFile->{narSize}, $narFile->{references},
$narFile->{deriver}, $narFile->{system});
};
sub addPatchToDB {
my ($storePath, $patch) = @_;
$insertPatch->execute(
$id, $storePath, $patch->{basePath}, $patch->{baseHash}, $patch->{url},
$patch->{hash}, $patch->{size}, $patch->{narHash}, $patch->{narSize},
$patch->{patchType});
};
my $version = readManifest_($manifest, \&addNARToDB, \&addPatchToDB);
if ($version < 3) {
die "you have an old-style or corrupt manifest `$manifestLink'; please delete it";
}
if ($version >= 10) {
die "manifest `$manifestLink' is too new; please delete it or upgrade Nix";
}
}
# Removed cached information for removed manifests from the DB.
foreach my $manifest (@{$dbh->selectcol_arrayref("select path from Manifests")}) {
next if defined $seen{$manifest};
$dbh->do("delete from Manifests where path = ?", {}, $manifest);
}
$dbh->commit;
close MAINLOCK;
return $dbh;
}
return 1;
+11 -6
View File
@@ -3,6 +3,8 @@ use File::Temp qw(tempdir);
our @sshOpts = split ' ', ($ENV{"NIX_SSHOPTS"} or "");
push @sshOpts, "-x";
my $sshStarted = 0;
my $sshHost;
@@ -24,14 +26,17 @@ sub openSSHConnection {
# child continues to run if we are killed. So instead make SSH
# print "started" when it has established the connection, and wait
# until we see that.
open SSH, "ssh $sshHost @sshOpts -M -N -o LocalCommand='echo started' -o PermitLocalCommand=yes |" or die;
while (<SSH>) {
open SSHPIPE, "ssh $sshHost @sshOpts -M -N -o LocalCommand='echo started' -o PermitLocalCommand=yes |" or die;
while (<SSHPIPE>) {
chomp;
last if /started/;
if ($_ eq "started") {
$sshStarted = 1;
return 1;
}
}
$sshStarted = 1;
return 1;
return 0;
}
# Tell the master SSH client to exit.
+27
View File
@@ -0,0 +1,27 @@
package Nix::Store;
use strict;
use warnings;
require Exporter;
our @ISA = qw(Exporter);
our %EXPORT_TAGS = ( 'all' => [ qw( ) ] );
our @EXPORT_OK = ( @{ $EXPORT_TAGS{'all'} } );
our @EXPORT = qw(
isValidPath queryReferences queryPathInfo queryDeriver queryPathHash
topoSortPaths computeFSClosure followLinksToStorePath exportPaths
hashPath hashFile hashString
addToStore makeFixedOutputPath
);
our $VERSION = '0.15';
require XSLoader;
XSLoader::load('Nix::Store', $VERSION);
1;
__END__
+218
View File
@@ -0,0 +1,218 @@
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
/* Prevent a clash between some Perl and libstdc++ macros. */
#undef do_open
#undef do_close
#include <store-api.hh>
#include <globals.hh>
#include <misc.hh>
#include <util.hh>
using namespace nix;
void doInit()
{
if (!store) {
try {
setDefaultsFromEnvironment();
store = openStore();
} catch (Error & e) {
croak(e.what());
}
}
}
MODULE = Nix::Store PACKAGE = Nix::Store
PROTOTYPES: ENABLE
void init()
CODE:
doInit();
int isValidPath(char * path)
CODE:
try {
doInit();
RETVAL = store->isValidPath(path);
} catch (Error & e) {
croak(e.what());
}
OUTPUT:
RETVAL
SV * queryReferences(char * path)
PPCODE:
try {
doInit();
PathSet paths;
store->queryReferences(path, paths);
for (PathSet::iterator i = paths.begin(); i != paths.end(); ++i)
XPUSHs(sv_2mortal(newSVpv(i->c_str(), 0)));
} catch (Error & e) {
croak(e.what());
}
SV * queryPathHash(char * path)
PPCODE:
try {
doInit();
Hash hash = store->queryPathHash(path);
string s = "sha256:" + printHash32(hash);
XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
} catch (Error & e) {
croak(e.what());
}
SV * queryDeriver(char * path)
PPCODE:
try {
doInit();
Path deriver = store->queryDeriver(path);
if (deriver == "") XSRETURN_UNDEF;
XPUSHs(sv_2mortal(newSVpv(deriver.c_str(), 0)));
} catch (Error & e) {
croak(e.what());
}
SV * queryPathInfo(char * path)
PPCODE:
try {
doInit();
ValidPathInfo info = store->queryPathInfo(path);
if (info.deriver == "")
XPUSHs(&PL_sv_undef);
else
XPUSHs(sv_2mortal(newSVpv(info.deriver.c_str(), 0)));
string s = "sha256:" + printHash(info.hash);
XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
mXPUSHi(info.registrationTime);
mXPUSHi(info.narSize);
AV * arr = newAV();
for (PathSet::iterator i = info.references.begin(); i != info.references.end(); ++i)
av_push(arr, newSVpv(i->c_str(), 0));
XPUSHs(sv_2mortal(newRV((SV *) arr)));
} catch (Error & e) {
croak(e.what());
}
SV * computeFSClosure(int flipDirection, int includeOutputs, ...)
PPCODE:
try {
doInit();
PathSet paths;
for (int n = 2; n < items; ++n)
computeFSClosure(*store, SvPV_nolen(ST(n)), paths, flipDirection, includeOutputs);
for (PathSet::iterator i = paths.begin(); i != paths.end(); ++i)
XPUSHs(sv_2mortal(newSVpv(i->c_str(), 0)));
} catch (Error & e) {
croak(e.what());
}
SV * topoSortPaths(...)
PPCODE:
try {
doInit();
PathSet paths;
for (int n = 0; n < items; ++n) paths.insert(SvPV_nolen(ST(n)));
Paths sorted = topoSortPaths(*store, paths);
for (Paths::iterator i = sorted.begin(); i != sorted.end(); ++i)
XPUSHs(sv_2mortal(newSVpv(i->c_str(), 0)));
} catch (Error & e) {
croak(e.what());
}
SV * followLinksToStorePath(char * path)
CODE:
try {
doInit();
RETVAL = newSVpv(followLinksToStorePath(path).c_str(), 0);
} catch (Error & e) {
croak(e.what());
}
OUTPUT:
RETVAL
void exportPaths(int fd, int sign, ...)
PPCODE:
try {
doInit();
Paths paths;
for (int n = 2; n < items; ++n) paths.push_back(SvPV_nolen(ST(n)));
FdSink sink(fd);
exportPaths(*store, paths, sign, sink);
} catch (Error & e) {
croak(e.what());
}
SV * hashPath(char * algo, int base32, char * path)
PPCODE:
try {
Hash h = hashPath(parseHashType(algo), path).first;
string s = base32 ? printHash32(h) : printHash(h);
XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
} catch (Error & e) {
croak(e.what());
}
SV * hashFile(char * algo, int base32, char * path)
PPCODE:
try {
Hash h = hashFile(parseHashType(algo), path);
string s = base32 ? printHash32(h) : printHash(h);
XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
} catch (Error & e) {
croak(e.what());
}
SV * hashString(char * algo, int base32, char * s)
PPCODE:
try {
Hash h = hashString(parseHashType(algo), s);
string s = base32 ? printHash32(h) : printHash(h);
XPUSHs(sv_2mortal(newSVpv(s.c_str(), 0)));
} catch (Error & e) {
croak(e.what());
}
SV * addToStore(char * srcPath, int recursive, char * algo)
PPCODE:
try {
doInit();
Path path = store->addToStore(srcPath, recursive, parseHashType(algo));
XPUSHs(sv_2mortal(newSVpv(path.c_str(), 0)));
} catch (Error & e) {
croak(e.what());
}
SV * makeFixedOutputPath(int recursive, char * algo, char * hash, char * name)
PPCODE:
try {
doInit();
HashType ht = parseHashType(algo);
Path path = makeFixedOutputPath(recursive, ht,
parseHash16or32(ht, hash), name);
XPUSHs(sv_2mortal(newSVpv(path.c_str(), 0)));
} catch (Error & e) {
croak(e.what());
}
+73 -89
View File
@@ -1,4 +1,7 @@
{ nixpkgs ? ../nixpkgs }:
{ nixpkgs ? <nixpkgs>, nixos ? <nixos>
, nix ? { outPath = ./.; rev = 1234; }
, officialRelease ? false
}:
let
@@ -6,10 +9,6 @@ let
tarball =
{ nix ? {outPath = ./.; rev = 1234;}
, officialRelease ? false
}:
with import nixpkgs {};
releaseTools.sourceTarball {
@@ -19,24 +18,26 @@ let
inherit officialRelease;
buildInputs =
[ curl bison flex2533 perl libxml2 libxslt w3m bzip2
tetex dblatex nukeReferences
[ curl bison24 flex2535 perl libxml2 libxslt w3m bzip2
tetex dblatex nukeReferences pkgconfig
];
configureFlags = ''
--with-docbook-rng=${docbook5}/xml/rng/docbook
--with-docbook-xsl=${docbook5_xsl}/xml/xsl/docbook
--with-xml-flags=--nonet
--with-dbi=${perlPackages.DBI}/lib/perl5/site_perl
--with-dbd-sqlite=${perlPackages.DBDSQLite}/lib/perl5/site_perl
'';
# Include the ATerm and Bzip2 tarballs in the distribution.
# Include the Bzip2 tarball in the distribution.
preConfigure = ''
stripHash ${aterm.src}
cp -pv ${aterm.src} externals/$strippedName
stripHash ${bzip2.src}
cp -pv ${bzip2.src} externals/$strippedName
stripHash ${sqlite.src}
cp -pv ${sqlite.src} externals/$strippedName
# TeX needs a writable font cache.
export VARTEXFONTS=$TMPDIR/texfonts
'';
@@ -63,70 +64,47 @@ let
build =
{ tarball ? jobs.tarball {}
, system ? "i686-linux"
}:
{ system ? "i686-linux" }:
with import nixpkgs {inherit system;};
with import nixpkgs { inherit system; };
releaseTools.nixBuild {
name = "nix";
src = tarball;
buildInputs = [curl perl bzip2 openssl];
buildInputs = [ curl perl bzip2 openssl pkgconfig boehmgc ];
configureFlags = ''
--disable-init-state
--with-aterm=${aterm} --with-bzip2=${bzip2}
--with-bzip2=${bzip2} --with-sqlite=${sqlite}
--with-dbi=${perlPackages.DBI}/lib/perl5/site_perl
--with-dbd-sqlite=${perlPackages.DBDSQLite}/lib/perl5/site_perl
--enable-gc
'';
};
/*
static =
{ tarball ? jobs.tarball {}
, system ? "i686-linux"
}:
with import nixpkgs {inherit system;};
releaseTools.binaryTarball {
name = "nix-static-tarball";
src = tarball;
buildInputs = [curl perl bzip2];
configureFlags = ''
--disable-init-state
--with-aterm=${aterm} --with-bzip2=${bzip2}
--enable-static-nix
'';
};
*/
coverage =
{ tarball ? jobs.tarball {}
}:
with import nixpkgs {};
with import nixpkgs { system = "x86_64-linux"; };
releaseTools.coverageAnalysis {
name = "nix-build";
src = tarball;
buildInputs = [
curl perl bzip2 openssl
# These are for "make check" only:
graphviz libxml2 libxslt
];
buildInputs =
[ curl perl bzip2 openssl
# These are for "make check" only:
graphviz libxml2 libxslt
];
configureFlags = ''
--disable-init-state --disable-shared
--with-aterm=${aterm} --with-bzip2=${bzip2}
--disable-init-state
--with-bzip2=${bzip2} --with-sqlite=${sqlite}
--with-dbi=${perlPackages.DBI}/lib/perl5/site_perl
--with-dbd-sqlite=${perlPackages.DBDSQLite}/lib/perl5/site_perl
'';
lcovFilter = ["*/boost/*" "*-tab.*"];
lcovFilter = [ "*/boost/*" "*-tab.*" ];
# We call `dot', and even though we just use it to
# syntax-check generated dot files, it still requires some
@@ -135,34 +113,42 @@ let
};
rpm_fedora5i386 = makeRPM_i686 (diskImages: diskImages.fedora5i386) 10;
rpm_fedora9i386 = makeRPM_i686 (diskImages: diskImages.fedora9i386) 20;
rpm_fedora9x86_64 = makeRPM_x86_64 (diskImages: diskImages.fedora9x86_64) 20;
rpm_fedora10i386 = makeRPM_i686 (diskImages: diskImages.fedora10i386) 30;
rpm_fedora10x86_64 = makeRPM_x86_64 (diskImages: diskImages.fedora10x86_64) 30;
rpm_fedora11i386 = makeRPM_i686 (diskImages: diskImages.fedora11i386) 40;
rpm_fedora11x86_64 = makeRPM_x86_64 (diskImages: diskImages.fedora11x86_64) 40;
rpm_fedora12i386 = makeRPM_i686 (diskImages: diskImages.fedora12i386) 50;
rpm_fedora12x86_64 = makeRPM_x86_64 (diskImages: diskImages.fedora12x86_64) 50;
rpm_opensuse103i386 = makeRPM_i686 (diskImages: diskImages.opensuse103i386) 40;
rpm_opensuse110i386 = makeRPM_i686 (diskImages: diskImages.opensuse110i386) 50;
rpm_opensuse110x86_64 = makeRPM_x86_64 (diskImages: diskImages.opensuse110x86_64) 50;
rpm_fedora10i386 = makeRPM_i686 (diskImageFuns: diskImageFuns.fedora10i386) 20;
rpm_fedora10x86_64 = makeRPM_x86_64 (diskImageFuns: diskImageFuns.fedora10x86_64) 20;
rpm_fedora11i386 = makeRPM_i686 (diskImageFuns: diskImageFuns.fedora11i386) 30;
rpm_fedora11x86_64 = makeRPM_x86_64 (diskImageFuns: diskImageFuns.fedora11x86_64) 30;
rpm_fedora12i386 = makeRPM_i686 (diskImageFuns: diskImageFuns.fedora12i386) 40;
rpm_fedora12x86_64 = makeRPM_x86_64 (diskImageFuns: diskImageFuns.fedora12x86_64) 40;
rpm_fedora13i386 = makeRPM_i686 (diskImageFuns: diskImageFuns.fedora13i386) 50;
rpm_fedora13x86_64 = makeRPM_x86_64 (diskImageFunsFun: diskImageFunsFun.fedora13x86_64) 50;
rpm_opensuse103i386 = makeRPM_i686 (diskImageFuns: diskImageFuns.opensuse103i386) 40;
rpm_opensuse110i386 = makeRPM_i686 (diskImageFuns: diskImageFuns.opensuse110i386) 50;
rpm_opensuse110x86_64 = makeRPM_x86_64 (diskImageFuns: diskImageFuns.opensuse110x86_64) 50;
deb_debian40i386 = makeDeb_i686 (diskImages: diskImages.debian40i386) 40;
deb_debian40x86_64 = makeDeb_x86_64 (diskImages: diskImages.debian40x86_64) 40;
deb_debian50i386 = makeDeb_i686 (diskImages: diskImages.debian50i386) 50;
deb_debian50x86_64 = makeDeb_x86_64 (diskImages: diskImages.debian50x86_64) 50;
deb_ubuntu804i386 = makeDeb_i686 (diskImages: diskImages.ubuntu804i386) 20;
deb_ubuntu804x86_64 = makeDeb_x86_64 (diskImages: diskImages.ubuntu804x86_64) 20;
deb_ubuntu810i386 = makeDeb_i686 (diskImages: diskImages.ubuntu810i386) 30;
deb_ubuntu810x86_64 = makeDeb_x86_64 (diskImages: diskImages.ubuntu810x86_64) 30;
deb_ubuntu904i386 = makeDeb_i686 (diskImages: diskImages.ubuntu904i386) 40;
deb_ubuntu904x86_64 = makeDeb_x86_64 (diskImages: diskImages.ubuntu904x86_64) 40;
deb_ubuntu910i386 = makeDeb_i686 (diskImages: diskImages.ubuntu910i386) 50;
deb_ubuntu910x86_64 = makeDeb_x86_64 (diskImages: diskImages.ubuntu910x86_64) 50;
deb_debian50i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.debian50i386) 50;
deb_debian50x86_64 = makeDeb_x86_64 (diskImageFuns: diskImageFuns.debian50x86_64) 50;
deb_debian60i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.debian60i386) 50;
deb_debian60x86_64 = makeDeb_x86_64 (diskImageFunsFun: diskImageFunsFun.debian60x86_64) 50;
deb_ubuntu904i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.ubuntu904i386) 40;
deb_ubuntu904x86_64 = makeDeb_x86_64 (diskImageFuns: diskImageFuns.ubuntu904x86_64) 40;
deb_ubuntu910i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.ubuntu910i386) 50;
deb_ubuntu910x86_64 = makeDeb_x86_64 (diskImageFuns: diskImageFuns.ubuntu910x86_64) 50;
deb_ubuntu1004i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.ubuntu1004i386) 50;
deb_ubuntu1004x86_64 = makeDeb_x86_64 (diskImageFuns: diskImageFuns.ubuntu1004x86_64) 50;
deb_ubuntu1010i386 = makeDeb_i686 (diskImageFuns: diskImageFuns.ubuntu1010i386) 50;
deb_ubuntu1010x86_64 = makeDeb_x86_64 (diskImageFuns: diskImageFuns.ubuntu1010x86_64) 50;
# System tests.
tests.remote_builds = (import ./tests/remote-builds.nix rec {
inherit nixpkgs nixos; nix = build { inherit system; }; system = "x86_64-linux";
}).test;
tests.nix_copy_closure = (import ./tests/nix-copy-closure.nix rec {
inherit nixpkgs nixos; nix = build { inherit system; }; system = "x86_64-linux";
}).test;
};
@@ -171,17 +157,16 @@ let
makeRPM =
system: diskImageFun: prio:
{ tarball ? jobs.tarball {}
}:
with import nixpkgs {inherit system;};
with import nixpkgs { inherit system; };
releaseTools.rpmBuild rec {
name = "nix-rpm-${diskImage.name}";
src = tarball;
diskImage = diskImageFun vmTools.diskImages;
src = jobs.tarball;
diskImage = (diskImageFun vmTools.diskImageFuns)
{ extraPackages = [ "perl-DBD-SQLite" "perl-devel" ]; };
memSize = 1024;
meta = { schedulingPriority = toString prio; };
meta.schedulingPriority = prio;
};
@@ -190,19 +175,18 @@ let
makeDeb =
system: diskImageFun: prio:
{ tarball ? jobs.tarball {}
}:
with import nixpkgs {inherit system;};
with import nixpkgs { inherit system; };
releaseTools.debBuild {
name = "nix-deb";
src = tarball;
diskImage = diskImageFun vmTools.diskImages;
src = jobs.tarball;
diskImage = (diskImageFun vmTools.diskImageFuns)
{ extraPackages = [ "libdbd-sqlite3-perl" ]; };
memSize = 1024;
meta = { schedulingPriority = toString prio; };
meta.schedulingPriority = prio;
configureFlags = "--sysconfdir=/etc";
debRequires = ["curl"];
debRequires = [ "curl" "libdbd-sqlite3-perl" ];
};
+9 -13
View File
@@ -1,23 +1,19 @@
bin_SCRIPTS = nix-collect-garbage \
nix-pull nix-push nix-prefetch-url \
nix-install-package nix-channel nix-build \
nix-copy-closure
nix-copy-closure nix-generate-patches
noinst_SCRIPTS = nix-profile.sh generate-patches.pl \
noinst_SCRIPTS = nix-profile.sh \
find-runtime-roots.pl build-remote.pl nix-reduce-build \
copy-from-other-stores.pl nix-http-export.cgi
nix-pull nix-push: readmanifest.pm readconfig.pm download-using-manifests.pl
nix-pull nix-push: download-using-manifests.pl
install-exec-local: readmanifest.pm download-using-manifests.pl copy-from-other-stores.pl find-runtime-roots.pl
install-exec-local: download-using-manifests.pl copy-from-other-stores.pl find-runtime-roots.pl
$(INSTALL) -d $(DESTDIR)$(sysconfdir)/profile.d
$(INSTALL_PROGRAM) nix-profile.sh $(DESTDIR)$(sysconfdir)/profile.d/nix.sh
$(INSTALL) -d $(DESTDIR)$(libexecdir)/nix
$(INSTALL_DATA) readmanifest.pm $(DESTDIR)$(libexecdir)/nix
$(INSTALL_DATA) readconfig.pm $(DESTDIR)$(libexecdir)/nix
$(INSTALL_DATA) ssh.pm $(DESTDIR)$(libexecdir)/nix
$(INSTALL_PROGRAM) find-runtime-roots.pl $(DESTDIR)$(libexecdir)/nix
$(INSTALL_PROGRAM) generate-patches.pl $(DESTDIR)$(libexecdir)/nix
$(INSTALL_PROGRAM) build-remote.pl $(DESTDIR)$(libexecdir)/nix
$(INSTALL) -d $(DESTDIR)$(libexecdir)/nix/substituters
$(INSTALL_PROGRAM) download-using-manifests.pl $(DESTDIR)$(libexecdir)/nix/substituters
@@ -30,15 +26,15 @@ EXTRA_DIST = nix-collect-garbage.in \
nix-pull.in nix-push.in nix-profile.sh.in \
nix-prefetch-url.in nix-install-package.in \
nix-channel.in \
readmanifest.pm.in \
readconfig.pm.in \
ssh.pm \
nix-build.in \
download-using-manifests.pl.in \
copy-from-other-stores.pl.in \
generate-patches.pl.in \
nix-copy-closure.in \
find-runtime-roots.pl.in \
build-remote.pl.in \
nix-reduce-build.in \
nix-http-export.cgi.in
nix-http-export.cgi.in \
nix-generate-patches.in
clean:
rm -f $(bin_SCRIPTS) $(noinst_SCRIPTS)
+197 -158
View File
@@ -1,9 +1,12 @@
#! @perl@ -w -I@libexecdir@/nix
#! @perl@ -w @perlFlags@
use Fcntl ':flock';
use Fcntl qw(:DEFAULT :flock);
use English '-no_match_vars';
use IO::Handle;
use ssh qw/sshOpts openSSHConnection/;
use Nix::Config;
use Nix::SSH qw/sshOpts openSSHConnection/;
use Nix::CopyClosure;
no warnings('once');
# General operation:
@@ -31,210 +34,246 @@ $ENV{"DISPLAY"} = "";
$ENV{"SSH_ASKPASS"} = "";
my $loadIncreased = 0;
my ($amWilling, $localSystem, $neededSystem, $drvPath, $maxSilentTime) = @ARGV;
$maxSilentTime = 0 unless defined $maxSilentTime;
sub sendReply {
my $reply = shift;
print STDERR "# $reply\n";
}
sub decline {
sendReply "decline";
exit 0;
}
sub all { $_ || return 0 for @_; 1 }
# Initialisation.
my $loadIncreased = 0;
my ($localSystem, $maxSilentTime, $printBuildTrace) = @ARGV;
$maxSilentTime = 0 unless defined $maxSilentTime;
my $currentLoad = $ENV{"NIX_CURRENT_LOAD"};
decline unless defined $currentLoad;
mkdir $currentLoad, 0777 or die unless -d $currentLoad;
my $conf = $ENV{"NIX_REMOTE_SYSTEMS"};
decline if !defined $conf || ! -e $conf;
my $canBuildLocally = $amWilling && ($localSystem eq $neededSystem);
# Read the list of machines.
my @machines;
open CONF, "< $conf" or die;
while (<CONF>) {
chomp;
s/\#.*$//g;
next if /^\s*$/;
/^\s*(\S+)\s+(\S+)\s+(\S+)\s+(\d+)(\s+([0-9\.]+))?\s*$/ or die;
push @machines,
{ hostName => $1
, systemTypes => [split(/,/, $2)]
, sshKeys => $3
, maxJobs => $4
, speedFactor => 1.0 * ($6 || 1)
, enabled => 1
};
}
close CONF;
# Acquire the exclusive lock on $currentLoad/main-lock.
my $mainLock = "$currentLoad/main-lock";
open MAINLOCK, ">>$mainLock" or die;
flock(MAINLOCK, LOCK_EX) or die;
sub openSlotLock {
my ($machine, $slot) = @_;
my $slotLockFn = "$currentLoad/" . (join '+', @{$machine->{systemTypes}}) . "-" . $machine->{hostName} . "-$slot";
my $slotLock = new IO::Handle;
open $slotLock, ">>$slotLockFn" or die;
sysopen $slotLock, "$slotLockFn", O_RDWR|O_CREAT, 0600 or die;
return $slotLock;
}
# Read the list of machines.
my @machines;
if (defined $conf && -e $conf) {
open CONF, "<$conf" or die;
while (<CONF>) {
chomp;
s/\#.*$//g;
next if /^\s*$/;
my @tokens = split /\s/, $_;
push @machines,
{ hostName => $tokens[0]
, systemTypes => [ split(/,/, $tokens[1]) ]
, sshKeys => $tokens[2]
, maxJobs => int($tokens[3])
, speedFactor => 1.0 * (defined $tokens[4] ? int($tokens[4]) : 1)
, features => [ split(/,/, $tokens[5] || "") ]
, enabled => 1
};
}
close CONF;
}
# Wait for the calling process to ask us whether we can build some derivation.
my ($drvPath, $hostName, $slotLock);
REQ: while (1) {
$_ = <STDIN> || exit 0;
my ($amWilling, $neededSystem);
($amWilling, $neededSystem, $drvPath, $requiredFeatures) = split;
my @requiredFeatures = split /,/, $requiredFeatures;
my $canBuildLocally = $amWilling && ($localSystem eq $neededSystem);
if (!defined $currentLoad) {
sendReply "decline";
next;
}
my $hostName;
my $slotLock;
while (1) {
# Acquire the exclusive lock on $currentLoad/main-lock.
mkdir $currentLoad, 0777 or die unless -d $currentLoad;
my $mainLock = "$currentLoad/main-lock";
sysopen MAINLOCK, "$mainLock", O_RDWR|O_CREAT, 0600 or die;
flock(MAINLOCK, LOCK_EX) or die;
# Find all machine that can execute this build, i.e., that support
# builds for the given platform and are not at their job limit.
my $rightType = 0;
my @available = ();
LOOP: foreach my $cur (@machines) {
if ($cur->{enabled} && grep { $neededSystem eq $_ } @{$cur->{systemTypes}}) {
$rightType = 1;
while (1) {
# Find all machine that can execute this build, i.e., that
# support builds for the given platform and features, and are
# not at their job limit.
my $rightType = 0;
my @available = ();
LOOP: foreach my $cur (@machines) {
if ($cur->{enabled}
&& (grep { $neededSystem eq $_ } @{$cur->{systemTypes}})
&& all(map { my $f = $_; 0 != grep { $f eq $_ } @{$cur->{features}} } @requiredFeatures))
{
$rightType = 1;
# We have a machine of the right type. Determine the load on
# the machine.
my $slot = 0;
my $load = 0;
my $free;
while ($slot < $cur->{maxJobs}) {
my $slotLock = openSlotLock($cur, $slot);
if (flock($slotLock, LOCK_EX | LOCK_NB)) {
$free = $slot unless defined $free;
flock($slotLock, LOCK_UN) or die;
} else {
$load++;
# We have a machine of the right type. Determine the load on
# the machine.
my $slot = 0;
my $load = 0;
my $free;
while ($slot < $cur->{maxJobs}) {
my $slotLock = openSlotLock($cur, $slot);
if (flock($slotLock, LOCK_EX | LOCK_NB)) {
$free = $slot unless defined $free;
flock($slotLock, LOCK_UN) or die;
} else {
$load++;
}
close $slotLock;
$slot++;
}
close $slotLock;
$slot++;
push @available, { machine => $cur, load => $load, free => $free }
if $load < $cur->{maxJobs};
}
push @available, { machine => $cur, load => $load, free => $free }
if $load < $cur->{maxJobs};
}
}
if (defined $ENV{NIX_DEBUG_HOOK}) {
print STDERR "load on " . $_->{machine}->{hostName} . " = " . $_->{load} . "\n"
foreach @available;
}
# Didn't find any available machine? Then decline or postpone.
if (scalar @available == 0) {
# Postpone if we have a machine of the right type, except if the
# local system can and wants to do the build.
if ($rightType && !$canBuildLocally) {
sendReply "postpone";
exit 0;
} else {
decline;
if (defined $ENV{NIX_DEBUG_HOOK}) {
print STDERR "load on " . $_->{machine}->{hostName} . " = " . $_->{load} . "\n"
foreach @available;
}
}
# Prioritise the available machines as follows:
# - First by load divided by speed factor, rounded to the nearest
# integer. This causes fast machines to be preferred over slow
# machines with similar loads.
# - Then by speed factor.
# - Finally by load.
sub lf { my $x = shift; return int($x->{load} / $x->{machine}->{speedFactor} + 0.4999); }
@available = sort
{ lf($a) <=> lf($b)
|| $b->{machine}->{speedFactor} <=> $a->{machine}->{speedFactor}
|| $a->{load} <=> $b->{load}
} @available;
# Didn't find any available machine? Then decline or postpone.
if (scalar @available == 0) {
# Postpone if we have a machine of the right type, except
# if the local system can and wants to do the build.
if ($rightType && !$canBuildLocally) {
sendReply "postpone";
} else {
sendReply "decline";
}
close MAINLOCK;
next REQ;
}
# Select the best available machine and lock a free slot.
my $selected = $available[0];
my $machine = $selected->{machine};
$slotLock = openSlotLock($machine, $selected->{free});
flock($slotLock, LOCK_EX | LOCK_NB) or die;
utime undef, undef, $slotLock;
close MAINLOCK;
# Prioritise the available machines as follows:
# - First by load divided by speed factor, rounded to the nearest
# integer. This causes fast machines to be preferred over slow
# machines with similar loads.
# - Then by speed factor.
# - Finally by load.
sub lf { my $x = shift; return int($x->{load} / $x->{machine}->{speedFactor} + 0.4999); }
@available = sort
{ lf($a) <=> lf($b)
|| $b->{machine}->{speedFactor} <=> $a->{machine}->{speedFactor}
|| $a->{load} <=> $b->{load}
} @available;
# Connect to the selected machine.
@sshOpts = ("-i", $machine->{sshKeys}, "-x");
$hostName = $machine->{hostName};
last if openSSHConnection $hostName;
# Select the best available machine and lock a free slot.
my $selected = $available[0];
my $machine = $selected->{machine};
$slotLock = openSlotLock($machine, $selected->{free});
flock($slotLock, LOCK_EX | LOCK_NB) or die;
utime undef, undef, $slotLock;
close MAINLOCK;
# Connect to the selected machine.
@sshOpts = ("-i", $machine->{sshKeys}, "-x");
$hostName = $machine->{hostName};
last REQ if openSSHConnection $hostName;
warn "unable to open SSH connection to $hostName, trying other available machines...\n";
$machine->{enabled} = 0;
warn "unable to open SSH connection to $hostName, trying other available machines...\n";
$machine->{enabled} = 0;
}
}
# Tell Nix we've accepted the build.
sendReply "accept";
my $x = <STDIN>;
chomp $x;
my @inputs = split /\s/, readline(STDIN);
my @outputs = split /\s/, readline(STDIN);
if ($x ne "okay") {
exit 0;
print STDERR "building `$drvPath' on `$hostName'\n";
print STDERR "@ build-remote $drvPath $hostName\n" if $printBuildTrace;
my $maybeSign = "";
$maybeSign = "--sign" if -e "$Nix::Config::confDir/signing-key.sec";
# Register the derivation as a temporary GC root. Note that $PPID is
# the PID of the remote SSH process, which, due to the use of a
# persistant SSH connection, should be the same across all remote
# command invocations for this session.
my $rootsDir = "@localstatedir@/nix/gcroots/tmp";
system("ssh $hostName @sshOpts 'mkdir -m 1777 -p $rootsDir; ln -sfn $drvPath $rootsDir/\$PPID.drv'");
sub removeRoots {
system("ssh $hostName @sshOpts 'rm -f $rootsDir/\$PPID.drv $rootsDir/\$PPID.out'");
}
# Do the actual build.
print STDERR "building `$drvPath' on `$hostName'\n";
# Copy the derivation and its dependencies to the build machine. This
# is guarded by an exclusive lock per machine to prevent multiple
# build-remote instances from copying to a machine simultaneously.
# That's undesirable because we may end up with N instances uploading
# the same missing path simultaneously, causing the effective network
# bandwidth and target disk speed to be divided by N.
my $uploadLock = "$currentLoad/$hostName.upload-lock";
sysopen MAINLOCK, "$uploadLock", O_RDWR|O_CREAT, 0600 or die;
flock(MAINLOCK, LOCK_EX) or die;
Nix::CopyClosure::copyTo($hostName, [ @sshOpts ], [ $drvPath, @inputs ], "", "", 0, 0, $maybeSign ne "");
close MAINLOCK;
my $inputs = `cat inputs`; die if ($? != 0);
$inputs =~ s/\n/ /g;
my $outputs = `cat outputs`; die if ($? != 0);
$outputs =~ s/\n/ /g;
# Perform the build.
my $buildFlags = "--max-silent-time $maxSilentTime --fallback --add-root $rootsDir/\$PPID.out --option verbosity 0";
print "copying inputs...\n";
my $maybeSign = "";
$maybeSign = "--sign" if -e "/nix/etc/nix/signing-key.sec";
system("NIX_SSHOPTS=\"@sshOpts\" @bindir@/nix-copy-closure $hostName $maybeSign $drvPath $inputs") == 0
or die "cannot copy inputs to $hostName: $?";
print "building...\n";
my $buildFlags = "--max-silent-time $maxSilentTime";
# `-tt' forces allocation of a pseudo-terminal. This is required to
# make the remote nix-store process receive a signal when the
# connection dies. Without it, the remote process might continue to
# run indefinitely (that is, until it next tries to write to
# stdout/stderr).
if (system("ssh $hostName @sshOpts -tt 'nix-store --realise $buildFlags $drvPath > /dev/null'") != 0) {
# If we couldn't run ssh or there was an ssh problem (indicated by
# exit code 255), then we return exit code 1; otherwise we assume
# that the builder failed, which we indicate to Nix using exit
# code 100. It's important to distinguish between the two because
# the first is a transient failure and the latter is permanent.
my $res = $? == -1 || ($? >> 8) == 255 ? 1 : 100;
print STDERR "build of `$drvPath' on `$hostName' failed with exit code $?\n";
# We let the remote side kill its process group when the connection is
# closed unexpectedly. This is necessary to ensure that no processes
# are left running on the remote system if the local Nix process is
# killed. (SSH itself doesn't kill child processes if the connection
# is interrupted unless the `-tt' flag is used to force a pseudo-tty,
# in which case every child receives SIGHUP; however, `-tt' doesn't
# work on some platforms when connection sharing is used.)
pipe STDIN, DUMMY; # make sure we have a readable STDIN
if (system("exec ssh $hostName @sshOpts '(read; kill -INT -\$\$) <&0 & nix-store -r $drvPath $buildFlags > /dev/null' 2>&4") != 0) {
# Note that if we get exit code 100 from `nix-store -r', it
# denotes a permanent build failure (as opposed to an SSH problem
# or a temporary Nix problem). We propagate this to the caller to
# allow it to distinguish between transient and permanent
# failures.
my $res = $? >> 8;
print STDERR "build of `$drvPath' on `$hostName' failed with exit code $res\n";
removeRoots;
exit $res;
}
print "build of `$drvPath' on `$hostName' succeeded\n";
#print "build of `$drvPath' on `$hostName' succeeded\n";
foreach my $output (split '\n', $outputs) {
# Copy the output from the build machine.
foreach my $output (@outputs) {
my $maybeSignRemote = "";
$maybeSignRemote = "--sign" if $UID != 0;
system("ssh $hostName @sshOpts 'nix-store --export $maybeSignRemote $output' | @bindir@/nix-store --import > /dev/null") == 0
system("exec ssh $hostName @sshOpts 'nix-store --export $maybeSignRemote $output'" .
"| NIX_HELD_LOCKS=$output @bindir@/nix-store --import > /dev/null") == 0
or die "cannot copy $output from $hostName: $?";
}
# Get rid of the temporary GC roots.
removeRoots;
+29 -34
View File
@@ -17,25 +17,19 @@ foreach my $dir (@remoteStoresAll) {
}
$ENV{"NIX_REMOTE"} = "";
sub findStorePath {
my $storePath = shift;
my $storePathName = basename $storePath;
foreach my $store (@remoteStores) {
# Determine whether $storePath exists by looking for the
# existence of the info file, and if so, get store path info
# from that file. This rather breaks abstraction: we should
# be using `nix-store' for that. But right now there is no
# good way to tell nix-store to access a store mounted under a
# different location (there's $NIX_STORE, but that only works
# if the remote store is mounted under its "real" location).
my $infoFile = "$store/var/nix/db/info/$storePathName";
my $storePath2 = "$store/store/$storePathName";
if (-f $infoFile && -e $storePath2) {
return ($infoFile, $storePath2);
}
my $sourcePath = "$store/store/" . basename $storePath;
next unless -e $sourcePath || -l $sourcePath;
$ENV{"NIX_DB_DIR"} = "$store/var/nix/db";
return ($store, $sourcePath) if
system("@bindir@/nix-store --check-validity $storePath") == 0;
}
return undef;
}
@@ -46,37 +40,38 @@ if ($ARGV[0] eq "--query") {
if ($cmd eq "have") {
my $storePath = <STDIN>; chomp $storePath;
(my $infoFile) = findStorePath $storePath;
print STDOUT ($infoFile ? "1\n" : "0\n");
print STDOUT (defined findStorePath($storePath) ? "1\n" : "0\n");
}
elsif ($cmd eq "info") {
my $storePath = <STDIN>; chomp $storePath;
(my $infoFile) = findStorePath $storePath;
if (!$infoFile) {
my ($store, $sourcePath) = findStorePath($storePath);
if (!defined $store) {
print "0\n";
next; # not an error
}
print "1\n";
my $deriver = "";
my @references = ();
$ENV{"NIX_DB_DIR"} = "$store/var/nix/db";
my $deriver = `@bindir@/nix-store --query --deriver $storePath`;
die "cannot query deriver of `$storePath'" if $? != 0;
chomp $deriver;
$deriver = "" if $deriver eq "unknown-deriver";
open INFO, "<$infoFile" or die "cannot read info file $infoFile\n";
while (<INFO>) {
chomp;
/^([\w-]+): (.*)$/ or die "bad info file";
my $key = $1;
my $value = $2;
if ($key eq "Deriver") { $deriver = $value; }
elsif ($key eq "References") { @references = split ' ', $value; }
}
close INFO;
my @references = split "\n",
`@bindir@/nix-store --query --references $storePath`;
die "cannot query references of `$storePath'" if $? != 0;
my $narSize = `@bindir@/nix-store --query --size $storePath`;
die "cannot query size of `$storePath'" if $? != 0;
chomp $narSize;
print "$deriver\n";
print scalar @references, "\n";
print "$_\n" foreach @references;
print "0\n"; # !!! showing size not supported (yet)
print "$narSize\n";
print "$narSize\n";
}
else { die "unknown command `$cmd'"; }
@@ -87,8 +82,8 @@ if ($ARGV[0] eq "--query") {
elsif ($ARGV[0] eq "--substitute") {
die unless scalar @ARGV == 2;
my $storePath = $ARGV[1];
(my $infoFile, my $sourcePath) = findStorePath $storePath;
die unless $infoFile;
my ($store, $sourcePath) = findStorePath $storePath;
die unless $store;
print "\n*** Copying `$storePath' from `$sourcePath'\n\n";
system("$binDir/nix-store --dump $sourcePath | $binDir/nix-store --restore $storePath") == 0
or die "cannot copy `$sourcePath' to `$storePath'";
+216 -207
View File
@@ -1,33 +1,170 @@
#! @perl@ -w -I@libexecdir@/nix
#! @perl@ -w @perlFlags@
use strict;
use readmanifest;
use Nix::Config;
use Nix::Manifest;
use Nix::Store;
use POSIX qw(strftime);
use File::Temp qw(tempdir);
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
STDOUT->autoflush(1);
my $manifestDir = ($ENV{"NIX_MANIFESTS_DIR"} or "@localstatedir@/nix/manifests");
my $logFile = "@localstatedir@/log/nix/downloads";
my $logFile = "$Nix::Config::logDir/downloads";
# For queries, skip expensive calls to nix-hash etc. We're just
# estimating the expected download size.
my $fast = 1;
# Load all manifests.
my %narFiles;
my %localPaths;
my %patches;
# Open the manifest cache and update it if necessary.
my $dbh = updateManifestDB();
for my $manifest (glob "$manifestDir/*.nixmanifest") {
my $version = readManifest($manifest, \%narFiles, \%localPaths, \%patches);
if ($version < 3) {
print STDERR "you have an old-style manifest `$manifest'; please delete it\n";
exit 1;
# $hashCache->{$algo}->{$path} yields the $algo-hash of $path.
my $hashCache;
sub parseHash {
my $hash = shift;
if ($hash =~ /^(.+):(.+)$/) {
return ($1, $2);
} else {
return ("md5", $hash);
}
if ($version >= 10) {
print STDERR "manifest `$manifest' is too new; please delete it or upgrade Nix\n";
exit 1;
}
# Compute the most efficient sequence of downloads to produce the
# given path.
sub computeSmallestDownload {
my $targetPath = shift;
# Build a graph of all store paths that might contribute to the
# construction of $targetPath, and the special node "start". The
# edges are either patch operations, or downloads of full NAR
# files. The latter edges only occur between "start" and a store
# path.
my %graph;
$graph{"start"} = {d => 0, pred => undef, edges => []};
my @queue = ();
my $queueFront = 0;
my %done;
sub addNode {
my $graph = shift;
my $u = shift;
$$graph{$u} = {d => 999999999999, pred => undef, edges => []}
unless defined $$graph{$u};
}
sub addEdge {
my $graph = shift;
my $u = shift;
my $v = shift;
my $w = shift;
my $type = shift;
my $info = shift;
addNode $graph, $u;
push @{$$graph{$u}->{edges}},
{weight => $w, start => $u, end => $v, type => $type, info => $info};
my $n = scalar @{$$graph{$u}->{edges}};
}
push @queue, $targetPath;
while ($queueFront < scalar @queue) {
my $u = $queue[$queueFront++];
next if defined $done{$u};
$done{$u} = 1;
addNode \%graph, $u;
# If the path already exists, it has distance 0 from the
# "start" node.
if (isValidPath($u)) {
addEdge \%graph, "start", $u, 0, "present", undef;
}
else {
# Add patch edges.
my $patchList = $dbh->selectall_arrayref(
"select * from Patches where storePath = ?",
{ Slice => {} }, $u);
foreach my $patch (@{$patchList}) {
if (isValidPath($patch->{basePath})) {
my ($baseHashAlgo, $baseHash) = parseHash $patch->{baseHash};
my $hash = $hashCache->{$baseHashAlgo}->{$patch->{basePath}};
if (!defined $hash) {
$hash = $fast && $baseHashAlgo eq "sha256"
? queryPathHash($patch->{basePath})
: hashPath($baseHashAlgo, $baseHashAlgo ne "md5", $patch->{basePath});
$hash =~ s/.*://;
$hashCache->{$baseHashAlgo}->{$patch->{basePath}} = $hash;
}
next if $hash ne $baseHash;
}
push @queue, $patch->{basePath};
addEdge \%graph, $patch->{basePath}, $u, $patch->{size}, "patch", $patch;
}
# Add NAR file edges to the start node.
my $narFileList = $dbh->selectall_arrayref(
"select * from NARs where storePath = ?",
{ Slice => {} }, $u);
foreach my $narFile (@{$narFileList}) {
# !!! how to handle files whose size is not known in advance?
# For now, assume some arbitrary size (1 GB).
# This has the side-effect of preferring non-Hydra downloads.
addEdge \%graph, "start", $u, ($narFile->{size} || 1000000000), "narfile", $narFile;
}
}
}
# Run Dijkstra's shortest path algorithm to determine the shortest
# sequence of download and/or patch actions that will produce
# $targetPath.
my @todo = keys %graph;
while (scalar @todo > 0) {
# Remove the closest element from the todo list.
# !!! inefficient, use a priority queue
@todo = sort { -($graph{$a}->{d} <=> $graph{$b}->{d}) } @todo;
my $u = pop @todo;
my $u_ = $graph{$u};
foreach my $edge (@{$u_->{edges}}) {
my $v_ = $graph{$edge->{end}};
if ($v_->{d} > $u_->{d} + $edge->{weight}) {
$v_->{d} = $u_->{d} + $edge->{weight};
# Store the edge; to edge->start is actually the
# predecessor.
$v_->{pred} = $edge;
}
}
}
# Retrieve the shortest path from "start" to $targetPath.
my @path = ();
my $cur = $targetPath;
return () unless defined $graph{$targetPath}->{pred};
while ($cur ne "start") {
push @path, $graph{$cur}->{pred};
$cur = $graph{$cur}->{pred}->{start};
}
return @path;
}
@@ -40,30 +177,52 @@ if ($ARGV[0] eq "--query") {
if ($cmd eq "have") {
my $storePath = <STDIN>; chomp $storePath;
print STDOUT ((defined $narFiles{$storePath} or defined $localPaths{$storePath})
print STDOUT (
scalar @{$dbh->selectcol_arrayref("select 1 from NARs where storePath = ?", {}, $storePath)} > 0
? "1\n" : "0\n");
}
elsif ($cmd eq "info") {
my $storePath = <STDIN>; chomp $storePath;
my $infos = $dbh->selectall_arrayref(
"select * from NARs where storePath = ?",
{ Slice => {} }, $storePath);
my $info;
if (defined $narFiles{$storePath}) {
$info = @{$narFiles{$storePath}}[0];
}
elsif (defined $localPaths{$storePath}) {
$info = @{$localPaths{$storePath}}[0];
if (scalar @{$infos} > 0) {
$info = @{$infos}[0];
}
else {
print "0\n";
next; # not an error
}
print "1\n";
print "$info->{deriver}\n";
my @references = split " ", $info->{references};
my @references = split " ", $info->{refs};
print scalar @references, "\n";
print "$_\n" foreach @references;
my $size = $info->{size} || 0;
print "$size\n";
my @path = computeSmallestDownload $storePath;
my $downloadSize = 0;
while (scalar @path > 0) {
my $edge = pop @path;
my $u = $edge->{start};
my $v = $edge->{end};
if ($edge->{type} eq "patch") {
$downloadSize += $edge->{info}->{size} || 0;
}
elsif ($edge->{type} eq "narfile") {
$downloadSize += $edge->{info}->{size} || 0;
}
}
print "$downloadSize\n";
my $narSize = $info->{narSize} || 0;
print "$narSize\n";
}
else { die "unknown command `$cmd'"; }
@@ -79,6 +238,7 @@ elsif ($ARGV[0] ne "--substitute") {
die unless scalar @ARGV == 2;
my $targetPath = $ARGV[1];
$fast = 0;
# Create a temporary directory.
@@ -94,164 +254,18 @@ open LOGFILE, ">>$logFile" or die "cannot open log file $logFile";
my $date = strftime ("%F %H:%M:%S UTC", gmtime (time));
print LOGFILE "$$ get $targetPath $date\n";
print "\n*** Trying to download/patch `$targetPath'\n";
print STDERR "\n*** Trying to download/patch `$targetPath'\n";
# If we can copy from a local path, do that.
my $localPathList = $localPaths{$targetPath};
foreach my $localPath (@{$localPathList}) {
my $sourcePath = $localPath->{copyFrom};
if (-e $sourcePath) {
print "\n*** Step 1/1: copying from $sourcePath\n";
system("$binDir/nix-store --dump $sourcePath | $binDir/nix-store --restore $targetPath") == 0
or die "cannot copy `$sourcePath' to `$targetPath'";
exit 0;
}
}
# Compute the shortest path.
my @path = computeSmallestDownload $targetPath;
die "don't know how to produce $targetPath\n" if scalar @path == 0;
# Build a graph of all store paths that might contribute to the
# construction of $targetPath, and the special node "start". The
# edges are either patch operations, or downloads of full NAR files.
# The latter edges only occur between "start" and a store path.
my %graph;
$graph{"start"} = {d => 0, pred => undef, edges => []};
my @queue = ();
my $queueFront = 0;
my %done;
sub addToQueue {
my $v = shift;
return if defined $done{$v};
$done{$v} = 1;
push @queue, $v;
}
sub addNode {
my $u = shift;
$graph{$u} = {d => 999999999999, pred => undef, edges => []}
unless defined $graph{$u};
}
sub addEdge {
my $u = shift;
my $v = shift;
my $w = shift;
my $type = shift;
my $info = shift;
addNode $u;
push @{$graph{$u}->{edges}},
{weight => $w, start => $u, end => $v, type => $type, info => $info};
my $n = scalar @{$graph{$u}->{edges}};
}
addToQueue $targetPath;
sub isValidPath {
my $p = shift;
return system("$binDir/nix-store --check-validity '$p' 2> /dev/null") == 0;
}
sub parseHash {
my $hash = shift;
if ($hash =~ /^(.+):(.+)$/) {
return ($1, $2);
} else {
return ("md5", $hash);
}
}
while ($queueFront < scalar @queue) {
my $u = $queue[$queueFront++];
# print "$u\n";
addNode $u;
# If the path already exists, it has distance 0 from the "start"
# node.
if (isValidPath($u)) {
addEdge "start", $u, 0, "present", undef;
}
else {
# Add patch edges.
my $patchList = $patches{$u};
foreach my $patch (@{$patchList}) {
if (isValidPath($patch->{basePath})) {
# !!! this should be cached
my ($baseHashAlgo, $baseHash) = parseHash $patch->{baseHash};
my $format = "--base32";
$format = "" if $baseHashAlgo eq "md5";
my $hash = `$binDir/nix-hash --type '$baseHashAlgo' $format "$patch->{basePath}"`;
chomp $hash;
if ($hash ne $baseHash) {
print LOGFILE "$$ rejecting $patch->{basePath}\n";
next;
}
}
addToQueue $patch->{basePath};
addEdge $patch->{basePath}, $u, $patch->{size}, "patch", $patch;
}
# Add NAR file edges to the start node.
my $narFileList = $narFiles{$u};
foreach my $narFile (@{$narFileList}) {
# !!! how to handle files whose size is not known in advance?
# For now, assume some arbitrary size (1 MB).
addEdge "start", $u, ($narFile->{size} || 1000000), "narfile", $narFile;
if ($u eq $targetPath) {
my $size = $narFile->{size} || -1;
print LOGFILE "$$ full-download-would-be $size\n";
}
}
}
}
# Run Dijkstra's shortest path algorithm to determine the shortest
# sequence of download and/or patch actions that will produce
# $targetPath.
sub byDistance { # sort by distance, reversed
return -($graph{$a}->{d} <=> $graph{$b}->{d});
}
my @todo = keys %graph;
while (scalar @todo > 0) {
# Remove the closest element from the todo list.
@todo = sort byDistance @todo;
my $u = pop @todo;
my $u_ = $graph{$u};
foreach my $edge (@{$u_->{edges}}) {
my $v_ = $graph{$edge->{end}};
if ($v_->{d} > $u_->{d} + $edge->{weight}) {
$v_->{d} = $u_->{d} + $edge->{weight};
# Store the edge; to edge->start is actually the
# predecessor.
$v_->{pred} = $edge;
}
}
}
# Retrieve the shortest path from "start" to $targetPath.
my @path = ();
my $cur = $targetPath;
die "don't know how to produce $targetPath\n"
unless defined $graph{$targetPath}->{pred};
while ($cur ne "start") {
push @path, $graph{$cur}->{pred};
$cur = $graph{$cur}->{pred}->{start};
}
# We don't need the manifest anymore, so close it as an optimisation:
# if we still have SQLite locks blocking other processes (we
# shouldn't), this gets rid of them.
$dbh->disconnect;
# Traverse the shortest path, perform the actions described by the
@@ -263,8 +277,8 @@ sub downloadFile {
my $url = shift;
$ENV{"PRINT_PATH"} = 1;
$ENV{"QUIET"} = 1;
my ($hash, $path) = `$binDir/nix-prefetch-url '$url'`;
die "download of `$url' failed" unless $? == 0;
my ($hash, $path) = `$Nix::Config::binDir/nix-prefetch-url '$url'`;
die "download of `$url' failed" . ($! ? ": $!" : "") . "\n" unless $? == 0;
chomp $path;
return $path;
}
@@ -276,36 +290,36 @@ while (scalar @path > 0) {
my $u = $edge->{start};
my $v = $edge->{end};
print "\n*** Step $curStep/$maxStep: ";
print STDERR "\n*** Step $curStep/$maxStep: ";
if ($edge->{type} eq "present") {
print "using already present path `$v'\n";
print STDERR "using already present path `$v'\n";
print LOGFILE "$$ present $v\n";
if ($curStep < $maxStep) {
# Since this is not the last step, the path will be used
# as a base to one or more patches. So turn the base path
# into a NAR archive, to which we can apply the patch.
print " packing base path...\n";
system("$binDir/nix-store --dump $v > $tmpNar") == 0
print STDERR " packing base path...\n";
system("$Nix::Config::binDir/nix-store --dump $v > $tmpNar") == 0
or die "cannot dump `$v'";
}
}
elsif ($edge->{type} eq "patch") {
my $patch = $edge->{info};
print "applying patch `$patch->{url}' to `$u' to create `$v'\n";
print STDERR "applying patch `$patch->{url}' to `$u' to create `$v'\n";
print LOGFILE "$$ patch $patch->{url} $patch->{size} $patch->{baseHash} $u $v\n";
# Download the patch.
print " downloading patch...\n";
print STDERR " downloading patch...\n";
my $patchPath = downloadFile "$patch->{url}";
# Apply the patch to the NAR archive produced in step 1 (for
# the already present path) or a later step (for patch sequences).
print " applying patch...\n";
system("@libexecdir@/bspatch $tmpNar $tmpNar2 $patchPath") == 0
print STDERR " applying patch...\n";
system("$Nix::Config::libexecDir/bspatch $tmpNar $tmpNar2 $patchPath") == 0
or die "cannot apply patch `$patchPath' to $tmpNar";
if ($curStep < $maxStep) {
@@ -314,8 +328,8 @@ while (scalar @path > 0) {
} else {
# This was the last patch. Unpack the final NAR archive
# into the target path.
print " unpacking patched archive...\n";
system("$binDir/nix-store --restore $v < $tmpNar2") == 0
print STDERR " unpacking patched archive...\n";
system("$Nix::Config::binDir/nix-store --restore $v < $tmpNar2") == 0
or die "cannot unpack $tmpNar2 into `$v'";
}
@@ -324,23 +338,23 @@ while (scalar @path > 0) {
elsif ($edge->{type} eq "narfile") {
my $narFile = $edge->{info};
print "downloading `$narFile->{url}' into `$v'\n";
print STDERR "downloading `$narFile->{url}' into `$v'\n";
my $size = $narFile->{size} || -1;
print LOGFILE "$$ narfile $narFile->{url} $size $v\n";
# Download the archive.
print " downloading archive...\n";
print STDERR " downloading archive...\n";
my $narFilePath = downloadFile "$narFile->{url}";
if ($curStep < $maxStep) {
# The archive will be used a base to a patch.
system("@bunzip2@ < '$narFilePath' > $tmpNar") == 0
system("$Nix::Config::bzip2 -d < '$narFilePath' > $tmpNar") == 0
or die "cannot unpack `$narFilePath' into `$v'";
} else {
# Unpack the archive into the target path.
print " unpacking archive...\n";
system("@bunzip2@ < '$narFilePath' | $binDir/nix-store --restore '$v'") == 0
print STDERR " unpacking archive...\n";
system("$Nix::Config::bzip2 -d < '$narFilePath' | $Nix::Config::binDir/nix-store --restore '$v'") == 0
or die "cannot unpack `$narFilePath' into `$v'";
}
@@ -359,20 +373,15 @@ if (defined $finalNarHash) {
# The hash in the manifest can be either in base-16 or base-32.
# Handle both.
my $extraFlag =
($hashAlgo eq "sha256" && length($hash) != 64)
? "--base32" : "";
my $hash2 = hashPath($hashAlgo, $hashAlgo eq "sha256" && length($hash) != 64, $targetPath);
my $hash2 = `@bindir@/nix-hash --type $hashAlgo $extraFlag $targetPath`
or die "cannot compute hash of path `$targetPath'";
chomp $hash2;
die "hash mismatch in downloaded path $targetPath; expected $hash, got $hash2"
die "hash mismatch in downloaded path $targetPath; expected $hash, got $hash2\n"
if $hash ne $hash2;
} else {
die "cannot check integrity of the downloaded path since its hash is not known";
die "cannot check integrity of the downloaded path since its hash is not known\n";
}
print STDERR "\n";
print LOGFILE "$$ success\n";
close LOGFILE;
Regular → Executable
View File
File mode changed.
-408
View File
@@ -1,408 +0,0 @@
#! @perl@ -w -I@libexecdir@/nix
use strict;
use File::Temp qw(tempdir);
use readmanifest;
# Some patch generations options.
# Max size of NAR archives to generate patches for.
my $maxNarSize = $ENV{"NIX_MAX_NAR_SIZE"};
$maxNarSize = 100 * 1024 * 1024 if !defined $maxNarSize;
# If patch is bigger than this fraction of full archive, reject.
my $maxPatchFraction = $ENV{"NIX_PATCH_FRACTION"};
$maxPatchFraction = 0.60 if !defined $maxPatchFraction;
die unless scalar @ARGV == 5;
my $hashAlgo = "sha256";
my $cacheDir = $ARGV[0];
my $patchesDir = $ARGV[1];
my $patchesURL = $ARGV[2];
my $srcDir = $ARGV[3];
my $dstDir = $ARGV[4];
my $tmpDir = tempdir("nix-generate-patches.XXXXXX", CLEANUP => 1, TMPDIR => 1)
or die "cannot create a temporary directory";
print "TEMP = $tmpDir\n";
#END { rmdir $tmpDir; }
my %srcNarFiles;
my %srcLocalPaths;
my %srcPatches;
my %dstNarFiles;
my %dstLocalPaths;
my %dstPatches;
readManifest "$srcDir/MANIFEST",
\%srcNarFiles, \%srcLocalPaths, \%srcPatches;
readManifest "$dstDir/MANIFEST",
\%dstNarFiles, \%dstLocalPaths, \%dstPatches;
sub findOutputPaths {
my $narFiles = shift;
my %outPaths;
foreach my $p (keys %{$narFiles}) {
# Ignore store expressions.
next if ($p =~ /\.store$/);
next if ($p =~ /\.drv$/);
# Ignore builders (too much ambiguity -- they're all called
# `builder.sh').
next if ($p =~ /\.sh$/);
next if ($p =~ /\.patch$/);
# Don't bother including tar files etc.
next if ($p =~ /\.tar\.(gz|bz2)$/ || $p =~ /\.zip$/ || $p =~ /\.bin$/);
$outPaths{$p} = 1;
}
return %outPaths;
}
print "finding src output paths...\n";
my %srcOutPaths = findOutputPaths \%srcNarFiles;
print "finding dst output paths...\n";
my %dstOutPaths = findOutputPaths \%dstNarFiles;
sub getNameVersion {
my $p = shift;
$p =~ /\/[0-9a-z]+((?:-[a-zA-Z][^\/-]*)+)([^\/]*)$/;
my $name = $1;
my $version = $2;
$name =~ s/^-//;
$version =~ s/^-//;
return ($name, $version);
}
# A quick hack to get a measure of the `distance' between two
# versions: it's just the position of the first character that differs
# (or 999 if they are the same).
sub versionDiff {
my $s = shift;
my $t = shift;
my $i;
return 999 if $s eq $t;
for ($i = 0; $i < length $s; $i++) {
return $i if $i >= length $t or
substr($s, $i, 1) ne substr($t, $i, 1);
}
return $i;
}
sub getNarBz2 {
my $narFiles = shift;
my $storePath = shift;
my $narFileList = $$narFiles{$storePath};
die "missing store expression $storePath" unless defined $narFileList;
my $narFile = @{$narFileList}[0];
die unless defined $narFile;
$narFile->{url} =~ /\/([^\/]+)$/;
die unless defined $1;
return "$cacheDir/$1";
}
sub containsPatch {
my $patches = shift;
my $storePath = shift;
my $basePath = shift;
my $patchList = $$patches{$storePath};
return 0 if !defined $patchList;
my $found = 0;
foreach my $patch (@{$patchList}) {
# !!! baseHash might differ
return 1 if $patch->{basePath} eq $basePath;
}
return 0;
}
# Compute the "weighted" number of uses of a path in the build graph.
sub computeUses {
my $narFiles = shift;
my $path = shift;
# Find the deriver of $path.
return 1 unless defined $$narFiles{$path};
my $deriver = @{$$narFiles{$path}}[0]->{deriver};
return 1 unless defined $deriver && $deriver ne "";
# print " DERIVER $deriver\n";
# Optimisation: build the referrers graph from the references
# graph.
my %referrers;
foreach my $q (keys %{$narFiles}) {
my @refs = split " ", @{$$narFiles{$q}}[0]->{references};
foreach my $r (@refs) {
$referrers{$r} = [] unless defined $referrers{$r};
push @{$referrers{$r}}, $q;
}
}
# Determine the shortest path from $deriver to all other reachable
# paths in the `referrers' graph.
my %dist;
$dist{$deriver} = 0;
my @queue = ($deriver);
my $pos = 0;
while ($pos < scalar @queue) {
my $p = $queue[$pos];
$pos++;
foreach my $q (@{$referrers{$p}}) {
if (!defined $dist{$q}) {
$dist{$q} = $dist{$p} + 1;
# print " $q $dist{$q}\n";
push @queue, $q;
}
}
}
my $wuse = 1.0;
foreach my $user (keys %dist) {
next if $user eq $deriver;
# print " $user $dist{$user}\n";
$wuse += 1.0 / 2.0**$dist{$user};
}
# print " XXX $path $wuse\n";
return $wuse;
}
# For each output path in the destination, see if we need to / can
# create a patch.
print "creating patches...\n";
foreach my $p (keys %dstOutPaths) {
# If exactly the same path already exists in the source, skip it.
next if defined $srcOutPaths{$p};
print " $p\n";
# If not, then we should find the paths in the source that are
# `most' likely to be present on a system that wants to install
# this path.
(my $name, my $version) = getNameVersion $p;
my @closest = ();
my $closestVersion;
my $minDist = -1; # actually, larger means closer
# Find all source paths with the same name.
foreach my $q (keys %srcOutPaths) {
(my $name2, my $version2) = getNameVersion $q;
if ($name eq $name2) {
# If the sizes differ too much, then skip. This
# disambiguates between, e.g., a real component and a
# wrapper component (cf. Firefox in Nixpkgs).
my $srcSize = @{$srcNarFiles{$q}}[0]->{size};
my $dstSize = @{$dstNarFiles{$p}}[0]->{size};
my $ratio = $srcSize / $dstSize;
$ratio = 1 / $ratio if $ratio < 1;
# print " SIZE $srcSize $dstSize $ratio $q\n";
if ($ratio >= 3) {
print " SKIPPING $q due to size ratio $ratio ($srcSize $dstSize)\n";
next;
}
# If the numbers of weighted uses differ too much, then
# skip. This disambiguates between, e.g., the bootstrap
# GCC and the final GCC in Nixpkgs.
my $srcUses = computeUses \%srcNarFiles, $q;
my $dstUses = computeUses \%dstNarFiles, $p;
$ratio = $srcUses / $dstUses;
$ratio = 1 / $ratio if $ratio < 1;
print " USE $srcUses $dstUses $ratio $q\n";
# if ($ratio >= 2) {
# print " SKIPPING $q due to use ratio $ratio ($srcUses $dstUses)\n";
# next;
# }
# If there are multiple matching names, include the ones
# with the closest version numbers.
my $dist = versionDiff $version, $version2;
if ($dist > $minDist) {
$minDist = $dist;
@closest = ($q);
$closestVersion = $version2;
} elsif ($dist == $minDist) {
push @closest, $q;
}
}
}
if (scalar(@closest) == 0) {
print " NO BASE: $p\n";
next;
}
foreach my $closest (@closest) {
# Generate a patch between $closest and $p.
print " $p <- $closest\n";
# If the patch already exists, skip it.
if (containsPatch(\%srcPatches, $p, $closest) ||
containsPatch(\%dstPatches, $p, $closest))
{
print " skipping, already exists\n";
next;
}
# next;
my $srcNarBz2 = getNarBz2 \%srcNarFiles, $closest;
my $dstNarBz2 = getNarBz2 \%dstNarFiles, $p;
system("@bunzip2@ < $srcNarBz2 > $tmpDir/A") == 0
or die "cannot unpack $srcNarBz2";
if ((stat "$tmpDir/A")[7] >= $maxNarSize) {
print " skipping, source is too large\n";
next;
}
system("@bunzip2@ < $dstNarBz2 > $tmpDir/B") == 0
or die "cannot unpack $dstNarBz2";
if ((stat "$tmpDir/B")[7] >= $maxNarSize) {
print " skipping, destination is too large\n";
next;
}
system("@libexecdir@/bsdiff $tmpDir/A $tmpDir/B $tmpDir/DIFF") == 0
or die "cannot compute binary diff";
my $baseHash = `@bindir@/nix-hash --flat --type $hashAlgo --base32 $tmpDir/A` or die;
chomp $baseHash;
my $narHash = `@bindir@/nix-hash --flat --type $hashAlgo --base32 $tmpDir/B` or die;
chomp $narHash;
my $narDiffHash = `@bindir@/nix-hash --flat --type $hashAlgo --base32 $tmpDir/DIFF` or die;
chomp $narDiffHash;
my $narDiffSize = (stat "$tmpDir/DIFF")[7];
my $dstNarBz2Size = (stat $dstNarBz2)[7];
print " size $narDiffSize; full size $dstNarBz2Size\n";
if ($narDiffSize >= $dstNarBz2Size) {
print " rejecting; patch bigger than full archive\n";
next;
}
if ($narDiffSize / $dstNarBz2Size >= $maxPatchFraction) {
print " rejecting; patch too large relative to full archive\n";
next;
}
my $finalName =
"$narDiffHash.nar-bsdiff";
if (-e "$patchesDir/$finalName") {
print " not copying, already exists\n";
}
else {
system("cp '$tmpDir/DIFF' '$patchesDir/$finalName.tmp'") == 0
or die "cannot copy diff";
rename("$patchesDir/$finalName.tmp", "$patchesDir/$finalName")
or die "cannot rename $patchesDir/$finalName.tmp";
}
# Add the patch to the manifest.
addPatch \%dstPatches, $p,
{ url => "$patchesURL/$finalName", hash => "$hashAlgo:$narDiffHash"
, size => $narDiffSize, basePath => $closest, baseHash => "$hashAlgo:$baseHash"
, narHash => "$hashAlgo:$narHash", patchType => "nar-bsdiff"
}, 0;
}
}
# Add in any potentially useful patches in the source (namely, those
# patches that produce either paths in the destination or paths that
# can be used as the base for other useful patches).
print "propagating patches...\n";
my $changed;
do {
# !!! we repeat this to reach the transitive closure; inefficient
$changed = 0;
print "loop\n";
my %dstBasePaths;
foreach my $q (keys %dstPatches) {
foreach my $patch (@{$dstPatches{$q}}) {
$dstBasePaths{$patch->{basePath}} = 1;
}
}
foreach my $p (keys %srcPatches) {
my $patchList = $srcPatches{$p};
my $include = 0;
# Is path $p included in the destination? If so, include
# patches that produce it.
$include = 1 if defined $dstNarFiles{$p};
# Is path $p a path that serves as a base for paths in the
# destination? If so, include patches that produce it.
# !!! check baseHash
$include = 1 if defined $dstBasePaths{$p};
if ($include) {
foreach my $patch (@{$patchList}) {
$changed = 1 if addPatch \%dstPatches, $p, $patch;
}
}
}
} while $changed;
# Rewrite the manifest of the destination (with the new patches).
writeManifest "$dstDir/MANIFEST",
\%dstNarFiles, \%dstPatches;
+2 -2
View File
@@ -1,2 +1,2 @@
./gc-releases.pl /data/webserver/dist/*/*/MANIFEST > dead
cat dead | xargs mv --target-directory=/data/webserver/trash/
./gc-releases.pl /data/releases/patches/all-patches $(find /data/releases -name MANIFEST) | sort > dead
cat dead | xargs -d '\n' sh -c 'find "$@" -mtime +100 -print' | xargs -d '\n' mv -v --target-directory=/data/releases/trash/
+8 -5
View File
@@ -1,18 +1,18 @@
#! /usr/bin/perl -w -I. -I..
#! /var/run/current-system/sw/bin/perl -w -I. -I..
use strict;
use readmanifest;
use NixManifest;
use readcache;
use File::Basename;
# Read the manifests.
my %narFiles;
my %localPaths;
my %patches;
foreach my $manifest (@ARGV) {
print STDERR "loading $manifest\n";
if (readManifest($manifest, \%narFiles, \%localPaths, \%patches, 1) < 3) {
if (readManifest($manifest, \%narFiles, \%patches, 1) < 3) {
# die "manifest `$manifest' is too old (i.e., for Nix <= 0.7)\n";
}
}
@@ -50,6 +50,9 @@ foreach my $patch (keys %patches) {
foreach my $archive (keys %readcache::archives) {
next if $archive eq "." || $archive eq "..";
if (!defined $usedFiles{$archive}) {
print $readcache::archives{$archive}, "\n";
my $file = $readcache::archives{$archive};
print "$file\n";
my $hashFile = dirname($file) . "/.hash." . basename($file);
print "$hashFile\n" if -e $hashFile;
}
}
+3 -3
View File
@@ -9,13 +9,13 @@ sub readDir {
opendir(DIR, "$dir") or die "cannot open `$dir': $!";
my @as = readdir DIR;
foreach my $archive (@as) {
next unless $archive =~ /^sha256_/ || $archive =~ /\.nar-bsdiff$/ || $archive =~ /\.nar\.bz2$/;
$archives{$archive} = "$dir/$archive";
}
closedir DIR;
}
readDir "/data/webserver/dist/nix-cache";
readDir "/data/webserver/dist/test-cache";
readDir "/data/webserver/dist/patches";
readDir "/data/releases/nars";
readDir "/data/releases/patches";
print STDERR scalar (keys %archives), "\n";
Regular → Executable
+11 -7
View File
@@ -1,8 +1,7 @@
#! @perl@ -w -I@libexecdir@/nix
use strict;
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
use Nix::Config;
my $addDrvLink = 0;
@@ -76,10 +75,10 @@ EOF
$outLink = $ARGV[$n];
}
elsif ($arg eq "--attr" or $arg eq "-A") {
elsif ($arg eq "--attr" or $arg eq "-A" or $arg eq "-I") {
$n++;
die "$0: `$arg' requires an argument\n" unless $n < scalar @ARGV;
push @instArgs, ("--attr", $ARGV[$n]);
push @instArgs, ($arg, $ARGV[$n]);
}
elsif ($arg eq "--arg" || $arg eq "--argstr") {
@@ -102,7 +101,7 @@ EOF
$n += 2;
}
elsif ($arg eq "--max-jobs" or $arg eq "-j" or $arg eq "--max-silent-time" or $arg eq "--log-type") {
elsif ($arg eq "--max-jobs" or $arg eq "-j" or $arg eq "--max-silent-time" or $arg eq "--log-type" or $arg eq "--cores") {
$n++;
die "$0: `$arg' requires an argument\n" unless $n < scalar @ARGV;
push @buildArgs, ($arg, $ARGV[$n]);
@@ -123,6 +122,11 @@ EOF
$verbose = 1;
}
elsif ($arg eq "--quiet") {
push @buildArgs, $arg;
push @instArgs, $arg;
}
elsif (substr($arg, 0, 1) eq "-") {
push @buildArgs, $arg;
}
@@ -151,7 +155,7 @@ foreach my $expr (@exprs) {
# Instantiate.
my @drvPaths;
# !!! would prefer the perl 5.8.0 pipe open feature here.
my $pid = open(DRVPATHS, "-|") || exec "$binDir/nix-instantiate", "--add-root", $drvLink, "--indirect", @instArgs, $expr;
my $pid = open(DRVPATHS, "-|") || exec "$Nix::Config::binDir/nix-instantiate", "--add-root", $drvLink, "--indirect", @instArgs, $expr;
while (<DRVPATHS>) {chomp; push @drvPaths, $_;}
if (!close DRVPATHS) {
die "nix-instantiate killed by signal " . ($? & 127) . "\n" if ($? & 127);
@@ -165,7 +169,7 @@ foreach my $expr (@exprs) {
# Build.
my @outPaths;
$pid = open(OUTPATHS, "-|") || exec "$binDir/nix-store", "--add-root", $outLink, "--indirect", "-rv",
$pid = open(OUTPATHS, "-|") || exec "$Nix::Config::binDir/nix-store", "--add-root", $outLink, "--indirect", "-r",
@buildArgs, @drvPaths;
while (<OUTPATHS>) {chomp; push @outPaths, $_;}
if (!close OUTPATHS) {
Regular → Executable
+10 -12
View File
@@ -1,15 +1,13 @@
#! @perl@ -w
use strict;
use Nix::Config;
my $rootsDir = "@localstatedir@/nix/gcroots";
my $stateDir = $ENV{"NIX_STATE_DIR"};
$stateDir = "@localstatedir@/nix" unless defined $stateDir;
my $manifestDir = $Nix::Config::manifestDir;
# Turn on caching in nix-prefetch-url.
my $channelCache = "$stateDir/channel-cache";
my $channelCache = "$Nix::Config::stateDir/channel-cache";
mkdir $channelCache, 0755 unless -e $channelCache;
$ENV{'NIX_DOWNLOAD_CACHE'} = $channelCache if -W $channelCache;
@@ -79,19 +77,19 @@ sub update {
readChannels;
# Create the manifests directory if it doesn't exist.
mkdir "$stateDir/manifests", 0755 unless -e "$stateDir/manifests";
mkdir $manifestDir, 0755 unless -e $manifestDir;
# Do we have write permission to the manifests directory? If not,
# then just skip pulling the manifest and just download the Nix
# expressions. If the user is a non-privileged user in a
# multi-user Nix installation, he at least gets installation from
# source.
if (-W "$stateDir/manifests") {
if (-W $manifestDir) {
# Pull cache manifests.
foreach my $url (@channels) {
#print "pulling cache manifest from `$url'\n";
system("@bindir@/nix-pull", "--skip-wrong-store", "$url/MANIFEST") == 0
system("$Nix::Config::binDir/nix-pull", "--skip-wrong-store", "$url/MANIFEST") == 0
or die "cannot pull cache manifest from `$url'";
}
@@ -110,7 +108,7 @@ sub update {
print "downloading Nix expressions from `$fullURL'...\n";
$ENV{"PRINT_PATH"} = 1;
$ENV{"QUIET"} = 1;
my ($hash, $path) = `@bindir@/nix-prefetch-url '$fullURL'`;
my ($hash, $path) = `$Nix::Config::binDir/nix-prefetch-url '$fullURL'`;
die "cannot fetch `$fullURL'" if $? != 0;
chomp $path;
$inputs .= '"' . $channelName . '"' . " " . $path . " ";
@@ -121,13 +119,13 @@ sub update {
my $userName = getpwuid($<);
die "who ARE you? go away" unless defined $userName;
my $rootFile = "$rootsDir/per-user/$userName/channels";
my $rootFile = "$Nix::Config::stateDir/gcroots/per-user/$userName/channels";
# Build the Nix expression.
print "unpacking channel Nix expressions...\n";
my $outPath = `\\
@bindir@/nix-build --out-link '$rootFile' --drv-link '$rootFile'.tmp \\
@datadir@/nix/corepkgs/channels/unpack.nix \\
$Nix::Config::binDir/nix-build --out-link '$rootFile' --drv-link '$rootFile'.tmp \\
'<nix/unpack-channel.nix>' \\
--argstr system @system@ --arg inputs '$inputs'`
or die "cannot unpack the channels";
chomp $outPath;
Regular → Executable
+3 -4
View File
@@ -1,11 +1,10 @@
#! @perl@ -w
use strict;
use Nix::Config;
my $profilesDir = "@localstatedir@/nix/profiles";
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
# Process the command line arguments.
my @args = ();
@@ -36,7 +35,7 @@ sub removeOldGenerations {
$name = $dir . "/" . $name;
if (-l $name && (readlink($name) =~ /link/)) {
print STDERR "removing old generations of profile $name\n";
system("$binDir/nix-env", "-p", $name, "--delete-generations", "old");
system("$Nix::Config::binDir/nix-env", "-p", $name, "--delete-generations", "old");
}
elsif (! -l $name && -d $name) {
removeOldGenerations $name;
@@ -50,4 +49,4 @@ removeOldGenerations $profilesDir if $removeOld;
# Run the actual garbage collector.
exec "$binDir/nix-store", "--gc", @args;
exec "$Nix::Config::binDir/nix-store", "--gc", @args;
Regular → Executable
+40 -65
View File
@@ -1,13 +1,14 @@
#! @perl@ -w -I@libexecdir@/nix
#! @perl@ -w @perlFlags@
use ssh;
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
use Nix::SSH;
use Nix::Config;
use Nix::Store;
use Nix::CopyClosure;
if (scalar @ARGV < 1) {
print STDERR <<EOF
Usage: nix-copy-closure [--from | --to] HOSTNAME [--sign] [--gzip] PATHS...
Usage: nix-copy-closure [--from | --to] HOSTNAME [--sign] [--gzip] [--bzip2] [--xz] PATHS...
EOF
;
exit 1;
@@ -24,6 +25,10 @@ my $decompressor = "";
my $toMode = 1;
my $includeOutputs = 0;
my $dryRun = 0;
# !!! Copied from nix-pack-closure, should put this in a module.
my @storePaths = ();
@@ -35,8 +40,16 @@ while (@ARGV) {
$sign = 1;
}
elsif ($arg eq "--gzip") {
$compressor = "| gzip";
$decompressor = "gunzip |";
$compressor = "gzip";
$decompressor = "gunzip";
}
elsif ($arg eq "--bzip2") {
$compressor = "bzip2";
$decompressor = "bunzip2";
}
elsif ($arg eq "--xz") {
$compressor = "xz";
$decompressor = "xz -d";
}
elsif ($arg eq "--from") {
$toMode = 0;
@@ -44,6 +57,12 @@ while (@ARGV) {
elsif ($arg eq "--to") {
$toMode = 1;
}
elsif ($arg eq "--include-outputs") {
$includeOutputs = 1;
}
elsif ($arg eq "--dry-run") {
$dryRun = 1;
}
elsif (!defined $sshHost) {
$sshHost = $arg;
}
@@ -57,81 +76,37 @@ openSSHConnection $sshHost or die "$0: unable to start SSH\n";
if ($toMode) { # Copy TO the remote machine.
my @allStorePaths;
# Get the closure of this path.
my $pid = open(READ, "$binDir/nix-store --query --requisites @storePaths|") or die;
while (<READ>) {
chomp;
die "bad: $_" unless /^\//;
push @allStorePaths, $_;
}
close READ or die "nix-store failed: $?";
# Ask the remote host which paths are invalid.
open(READ, "ssh $sshHost @sshOpts nix-store --check-validity --print-invalid @allStorePaths|");
my @missing = ();
while (<READ>) {
chomp;
push @missing, $_;
}
close READ or die;
# Export the store paths and import them on the remote machine.
if (scalar @missing > 0) {
print STDERR "copying these missing paths:\n";
print STDERR " $_\n" foreach @missing;
my $extraOpts = "";
$extraOpts .= "--sign" if $sign == 1;
system("nix-store --export $extraOpts @missing $compressor | ssh $sshHost @sshOpts '$decompressor nix-store --import'") == 0
or die "copying store paths to remote machine `$sshHost' failed: $?";
}
Nix::CopyClosure::copyTo($sshHost, [ @sshOpts ], [ @storePaths ], $compressor, $decompressor, $includeOutputs, $dryRun, $sign);
}
else { # Copy FROM the remote machine.
# Query the closure of the given store paths on the remote
# machine. Paths are assumed to be store paths; there is no
# resolution (following of symlinks).
my $extraOpts = $includeOutputs ? "--include-outputs" : "";
my $pid = open(READ,
"ssh @sshOpts $sshHost nix-store --query --requisites @storePaths|") or die;
"set -f; ssh @sshOpts $sshHost nix-store --query --requisites $extraOpts @storePaths|") or die;
my @allStorePaths;
while (<READ>) {
chomp;
die "bad: $_" unless /^\//;
push @allStorePaths, $_;
push @missing, $_ unless isValidPath($_);
}
close READ or die "nix-store on remote machine `$sshHost' failed: $?";
# What paths are already valid locally?
open(READ, "@bindir@/nix-store --check-validity --print-invalid @allStorePaths|");
my @missing = ();
while (<READ>) {
chomp;
push @missing, $_;
}
close READ or die;
# Export the store paths on the remote machine and import them on locally.
# Export the store paths on the remote machine and import them locally.
if (scalar @missing > 0) {
print STDERR "copying these missing paths:\n";
print STDERR " $_\n" foreach @missing;
my $extraOpts = "";
$extraOpts .= "--sign" if $sign == 1;
system("ssh $sshHost @sshOpts 'nix-store --export $extraOpts @missing $compressor' | $decompressor @bindir@/nix-store --import") == 0
or die "copying store paths from remote machine `$sshHost' failed: $?";
print STDERR "copying ", scalar @missing, " missing paths from ‘$sshHost’...\n";
#print STDERR " $_\n" foreach @missing;
$compressor = "| $compressor" if $compressor ne "";
$decompressor = "$decompressor |" if $decompressor ne "";
unless ($dryRun) {
my $extraOpts = $sign ? "--sign" : "";
system("set -f; ssh $sshHost @sshOpts 'nix-store --export $extraOpts @missing $compressor' | $decompressor $Nix::Config::binDir/nix-store --import") == 0
or die "copying store paths from remote machine `$sshHost' failed: $?";
}
}
}
+42
View File
@@ -0,0 +1,42 @@
#! @perl@ -w @perlFlags@
use strict;
use File::Temp qw(tempdir);
use Nix::Manifest;
use Nix::GeneratePatches;
if (scalar @ARGV != 5) {
print STDERR <<EOF;
Usage: nix-generate-patches NAR-DIR PATCH-DIR PATCH-URI OLD-MANIFEST NEW-MANIFEST
This command generates binary patches between NAR files listed in
OLD-MANIFEST and NEW-MANIFEST. The patches are written to the
directory PATCH-DIR, and the prefix PATCH-URI is used to generate URIs
for the patches. The patches are added to NEW-MANIFEST. All NARs are
required to exist in NAR-DIR. Patches are generated between
succeeding versions of packages with the same name.
EOF
exit 1;
}
my $narPath = $ARGV[0];
my $patchesPath = $ARGV[1];
my $patchesURL = $ARGV[2];
my $srcManifest = $ARGV[3];
my $dstManifest = $ARGV[4];
my (%srcNarFiles, %srcLocalPaths, %srcPatches);
readManifest $srcManifest, \%srcNarFiles, \%srcPatches;
my (%dstNarFiles, %dstLocalPaths, %dstPatches);
readManifest $dstManifest, \%dstNarFiles, \%dstPatches;
my $tmpDir = tempdir("nix-generate-patches.XXXXXX", CLEANUP => 1, TMPDIR => 1)
or die "cannot create a temporary directory";
generatePatches \%srcNarFiles, \%dstNarFiles, \%srcPatches, \%dstPatches,
$narPath, $patchesPath, $patchesURL, $tmpDir;
propagatePatches \%srcPatches, \%dstNarFiles, \%dstPatches;
writeManifest $dstManifest, \%dstNarFiles, \%dstPatches;
Regular → Executable
+8 -6
View File
@@ -2,8 +2,7 @@
use strict;
use File::Temp qw(tempdir);
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
use Nix::Config;
sub usageError {
@@ -61,7 +60,7 @@ if ($interactive && !defined $ENV{"NIX_HAVE_TERMINAL"}) {
$ENV{"NIX_HAVE_TERMINAL"} = "1";
$ENV{"LD_LIBRARY_PATH"} = "";
foreach my $term ("xterm", "konsole", "gnome-terminal", "xterm") {
exec($term, "-e", "$binDir/nix-install-package", @ARGV);
exec($term, "-e", "$Nix::Config::binDir/nix-install-package", @ARGV);
}
die "cannot execute `xterm'";
}
@@ -124,17 +123,20 @@ if ($interactive) {
# Store the manifest in the temporary directory so that we don't
# pollute /nix/var/nix/manifests.
# pollute /nix/var/nix/manifests. This also requires that we don't
# use the Nix daemon (because otherwise download-using-manifests won't
# see our NIX_MANIFESTS_DIRS environment variable).
$ENV{NIX_MANIFESTS_DIR} = $tmpDir;
$ENV{NIX_REMOTE} = "";
print "\nPulling manifests...\n";
system("$binDir/nix-pull", $manifestURL) == 0
system("$Nix::Config::binDir/nix-pull", $manifestURL) == 0
or barf "nix-pull failed: $?";
print "\nInstalling package...\n";
system("$binDir/nix-env", "--install", $outPath, "--force-name", $drvName, @extraNixEnvArgs) == 0
system("$Nix::Config::binDir/nix-env", "--install", $outPath, "--force-name", $drvName, @extraNixEnvArgs) == 0
or barf "nix-env failed: $?";
Regular → Executable
+96 -130
View File
@@ -1,162 +1,128 @@
#! @shell@ -e
#! @perl@ -w @perlFlags@
url=$1
expHash=$2
use strict;
use File::Basename;
use File::Temp qw(tempdir);
use File::stat;
use Nix::Store;
use Nix::Config;
# needed to make it work on NixOS
export PATH=$PATH:@coreutils@
my $url = shift;
my $expHash = shift;
my $hashType = $ENV{'NIX_HASH_ALGO'} || "sha256";
my $cacheDir = $ENV{'NIX_DOWNLOAD_CACHE'};
hashType=$NIX_HASH_ALGO
if test -z "$hashType"; then
hashType=sha256
fi
if (!defined $url || $url eq "") {
print STDERR <<EOF
Usage: nix-prefetch-url URL [EXPECTED-HASH]
EOF
;
exit 1;
}
hashFormat=
if test "$hashType" != "md5"; then
hashFormat=--base32
fi
sub writeFile {
my ($fn, $s) = @_;
open TMP, ">$fn" or die;
print TMP "$s" or die;
close TMP or die;
}
if test -z "$url"; then
echo "syntax: nix-prefetch-url URL [EXPECTED-HASH]" >&2
exit 1
fi
sub readFile {
local $/ = undef;
my ($fn) = @_;
open TMP, "<$fn" or die;
my $s = <TMP>;
close TMP or die;
return $s;
}
my $tmpDir = tempdir("nix-prefetch-url.XXXXXX", CLEANUP => 1, TMPDIR => 1)
or die "cannot create a temporary directory";
# Hack to support the mirror:// scheme from Nixpkgs.
if ($url =~ /^mirror:\/\//) {
system("$Nix::Config::binDir/nix-build '<nixpkgs>' -A resolveMirrorURLs --argstr url '$url' -o $tmpDir/urls > /dev/null") == 0
or die "$0: nix-build failed; maybe \$NIX_PATH is not set properly\n";
my @expanded = split ' ', readFile("$tmpDir/urls");
die "$0: cannot resolve ‘$url’" unless scalar @expanded > 0;
print STDERR "$url expands to $expanded[0]\n";
$url = $expanded[0];
}
# Handle escaped characters in the URI. `+', `=' and `?' are the only
# characters that are valid in Nix store path names but have a special
# meaning in URIs.
name=$(basename "$url" | @sed@ -e 's/%2b/+/g' -e 's/%3d/=/g' -e 's/%3f/\?/g')
if test -z "$name"; then echo "invalid url"; exit 1; fi
my $name = basename $url;
die "cannot figure out file name for ‘$url’\n" if $name eq "";
$name =~ s/%2b/+/g;
$name =~ s/%3d/=/g;
$name =~ s/%3f/?/g;
my $finalPath;
my $hash;
# If the hash was given, a file with that hash may already be in the
# store.
if test -n "$expHash"; then
finalPath=$(@bindir@/nix-store --print-fixed-path "$hashType" "$expHash" "$name")
if ! @bindir@/nix-store --check-validity "$finalPath" 2> /dev/null; then
finalPath=
fi
hash=$expHash
fi
mkTempDir() {
if test -n "$tmpPath"; then return; fi
local i=0
while true; do
if test -z "$TMPDIR"; then TMPDIR=/tmp; fi
tmpPath=$TMPDIR/nix-prefetch-url-$$-$i
if mkdir "$tmpPath"; then break; fi
# !!! to bad we can't check for ENOENT in mkdir, so this check
# is slightly racy (it bombs out if somebody just removed
# $tmpPath...).
if ! test -e "$tmpPath"; then exit 1; fi
i=$((i + 1))
done
trap removeTempDir EXIT SIGINT SIGQUIT
if (defined $expHash) {
$finalPath = makeFixedOutputPath(0, $hashType, $expHash, $name);
if (isValidPath($finalPath)) { $hash = $expHash; } else { $finalPath = undef; }
}
removeTempDir() {
if test -n "$tmpPath"; then
rm -rf "$tmpPath" || true
fi
}
doDownload() {
@curl@ $cacheFlags --fail --location --max-redirs 20 --disable-epsv \
--cookie-jar $tmpPath/cookies "$url" -o $tmpFile
}
# Hack to support the mirror:// scheme from Nixpkgs.
if test "${url:0:9}" = "mirror://"; then
if test -z "$NIXPKGS_ALL"; then
echo "Resolving mirror:// URLs requires Nixpkgs. Please point \$NIXPKGS_ALL at a Nixpkgs tree." >&2
exit 1
fi
mkTempDir
nix-build "$NIXPKGS_ALL" -A resolveMirrorURLs --argstr url "$url" -o $tmpPath/urls > /dev/null
expanded=($(cat $tmpPath/urls))
if test "${#expanded[*]}" = 0; then
echo "$0: cannot resolve $url." >&2
exit 1
fi
echo "$url expands to ${expanded[*]} (using ${expanded[0]})" >&2
url="${expanded[0]}"
fi
# If we don't know the hash or a file with that hash doesn't exist,
# download the file and add it to the store.
if test -z "$finalPath"; then
mkTempDir
tmpFile=$tmpPath/$name
if (!defined $finalPath) {
my $tmpFile = "$tmpDir/$name";
# Optionally do timestamp-based caching of the download.
# Actually, the only thing that we cache in $NIX_DOWNLOAD_CACHE is
# the hash and the timestamp of the file at $url. The caching of
# the file *contents* is done in Nix store, where it can be
# garbage-collected independently.
if test -n "$NIX_DOWNLOAD_CACHE"; then
echo -n "$url" > $tmpPath/url
urlHash=$(@bindir@/nix-hash --type sha256 --base32 --flat $tmpPath/url)
echo "$url" > "$NIX_DOWNLOAD_CACHE/$urlHash.url"
cachedHashFN="$NIX_DOWNLOAD_CACHE/$urlHash.$hashType"
cachedTimestampFN="$NIX_DOWNLOAD_CACHE/$urlHash.stamp"
cacheFlags="--remote-time"
if test -e "$cachedTimestampFN" -a -e "$cachedHashFN"; then
# Only download the file if it is newer than the cached version.
cacheFlags="$cacheFlags --time-cond $cachedTimestampFN"
fi
fi
my ($cachedTimestampFN, $cachedHashFN, @cacheFlags);
if (defined $cacheDir) {
my $urlHash = hashString("sha256", 1, $url);
writeFile "$cacheDir/$urlHash.url", $url;
$cachedHashFN = "$cacheDir/$urlHash.$hashType";
$cachedTimestampFN = "$cacheDir/$urlHash.stamp";
@cacheFlags = ("--time-cond", $cachedTimestampFN) if -f $cachedHashFN && -f $cachedTimestampFN;
}
# Perform the download.
doDownload
my @curlFlags = ("curl", $url, "-o", $tmpFile, "--fail", "--location", "--max-redirs", "20", "--disable-epsv", "--cookie-jar", "$tmpDir/cookies", "--remote-time", (split " ", ($ENV{NIX_CURL_FLAGS} || "")));
(system $Nix::Config::curl @curlFlags, @cacheFlags) == 0 or die "$0: download of ‘$url’ failed\n";
if test -n "$NIX_DOWNLOAD_CACHE" -a ! -e $tmpFile; then
if (defined $cacheDir && ! -e $tmpFile) {
# Curl didn't create $tmpFile, so apparently there's no newer
# file on the server.
hash=$(cat $cachedHashFN)
finalPath=$(@bindir@/nix-store --print-fixed-path "$hashType" "$hash" "$name")
if ! @bindir@/nix-store --check-validity "$finalPath" 2> /dev/null; then
echo "cached contents of \`$url' disappeared, redownloading..." >&2
finalPath=
cacheFlags="--remote-time"
doDownload
fi
fi
$hash = readFile $cachedHashFN or die;
$finalPath = makeFixedOutputPath(0, $hashType, $hash, $name);
unless (isValidPath $finalPath) {
print STDERR "cached contents of ‘$url’ disappeared, redownloading...\n";
$finalPath = undef;
(system $Nix::Config::curl @curlFlags) == 0 or die "$0: download of ‘$url’ failed\n";
}
}
if test -z "$finalPath"; then
# Compute the hash.
hash=$(@bindir@/nix-hash --type "$hashType" $hashFormat --flat $tmpFile)
if ! test -n "$QUIET"; then echo "hash is $hash" >&2; fi
if test -n "$NIX_DOWNLOAD_CACHE"; then
echo $hash > $cachedHashFN
touch -r $tmpFile $cachedTimestampFN
fi
# Add the downloaded file to the Nix store.
finalPath=$(@bindir@/nix-store --add-fixed "$hashType" $tmpFile)
if test -n "$expHash" -a "$expHash" != "$hash"; then
echo "hash mismatch for URL \`$url'" >&2
exit 1
fi
if (!defined $finalPath) {
fi
fi
# Compute the hash.
$hash = hashFile($hashType, $hashType ne "md5", $tmpFile);
if (defined $cacheDir) {
writeFile $cachedHashFN, $hash;
my $st = stat($tmpFile) or die;
open STAMP, ">$cachedTimestampFN" or die; close STAMP;
utime($st->atime, $st->mtime, $cachedTimestampFN) or die;
}
# Add the downloaded file to the Nix store.
$finalPath = addToStore($tmpFile, 0, $hashType);
}
if ! test -n "$QUIET"; then echo "path is $finalPath" >&2; fi
die "$0: hash mismatch for ‘$url’\n" if defined $expHash && $expHash ne $hash;
}
echo $hash
if test -n "$PRINT_PATH"; then
echo $finalPath
fi
print STDERR "path is ‘$finalPath’\n" unless $ENV{'QUIET'};
print "$hash\n";
print "$finalPath\n" if $ENV{'PRINT_PATH'};
Regular → Executable
+20 -43
View File
@@ -1,17 +1,14 @@
#! @perl@ -w -I@libexecdir@/nix
#! @perl@ -w @perlFlags@
use strict;
use File::Temp qw(tempdir);
use readmanifest;
use Nix::Config;
use Nix::Manifest;
my $tmpDir = tempdir("nix-pull.XXXXXX", CLEANUP => 1, TMPDIR => 1)
or die "cannot create a temporary directory";
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
my $libexecDir = ($ENV{"NIX_LIBEXEC_DIR"} or "@libexecdir@");
my $storeDir = ($ENV{"NIX_STORE_DIR"} or "@storedir@");
my $stateDir = ($ENV{"NIX_STATE_DIR"} or "@localstatedir@/nix");
my $manifestDir = ($ENV{"NIX_MANIFESTS_DIR"} or "$stateDir/manifests");
my $manifestDir = $Nix::Config::manifestDir;
# Prevent access problems in shared-stored installations.
@@ -24,18 +21,21 @@ if (! -e $manifestDir) {
}
# Process the URLs specified on the command line.
my %narFiles;
my %localPaths;
my %patches;
# Make sure that the manifests directory is scanned for GC roots.
my $gcRootsDir = "$Nix::Config::stateDir/gcroots";
my $manifestDirLink = "$gcRootsDir/manifests";
if (! -l $manifestDirLink) {
symlink($manifestDir, $manifestDirLink) or die "cannot create symlink `$manifestDirLink'";
}
my $skipWrongStore = 0;
# Process the URLs specified on the command line.
sub downloadFile {
my $url = shift;
$ENV{"PRINT_PATH"} = 1;
$ENV{"QUIET"} = 1;
my ($dummy, $path) = `$binDir/nix-prefetch-url '$url'`;
my ($dummy, $path) = `$Nix::Config::binDir/nix-prefetch-url '$url'`;
die "cannot fetch `$url'" if $? != 0;
die "nix-prefetch-url did not return a path" unless defined $path;
chomp $path;
@@ -50,46 +50,23 @@ sub processURL {
my $manifest;
# First see if a bzipped manifest is available.
if (system("@curl@ --fail --silent --head '$url'.bz2 > /dev/null") == 0) {
if (system("$Nix::Config::curl --fail --silent --head '$url'.bz2 > /dev/null") == 0) {
print "fetching list of Nix archives at `$url.bz2'...\n";
my $bzipped = downloadFile "$url.bz2";
$manifest = "$tmpDir/MANIFEST";
system("@bunzip2@ < $bzipped > $manifest") == 0
or die "cannot decompress manifest";
$manifest = (`$binDir/nix-store --add $manifest`
or die "cannot copy $manifest to the store");
chomp $manifest;
$manifest = downloadFile "$url.bz2";
}
# Otherwise, just get the uncompressed manifest.
else {
print "obtaining list of Nix archives at `$url'...\n";
print "fetching list of Nix archives at `$url'...\n";
$manifest = downloadFile $url;
}
my $version = readManifest($manifest, \%narFiles, \%localPaths, \%patches);
die "`$url' is not a manifest or it is too old (i.e., for Nix <= 0.7)\n" if $version < 3;
die "manifest `$url' is too new\n" if $version >= 5;
if ($skipWrongStore) {
foreach my $path (keys %narFiles) {
if (substr($path, 0, length($storeDir) + 1) ne "$storeDir/") {
print STDERR "warning: manifest `$url' assumes a Nix store at a different location than $storeDir, skipping...\n";
exit 0;
}
}
}
my $baseName = "unnamed";
if ($url =~ /\/([^\/]+)\/[^\/]+$/) { # get the forelast component
$baseName = $1;
}
my $hash = `$binDir/nix-hash --flat '$manifest'`
my $hash = `$Nix::Config::binDir/nix-hash --flat '$manifest'`
or die "cannot hash `$manifest'";
chomp $hash;
@@ -122,12 +99,12 @@ sub processURL {
while (@ARGV) {
my $url = shift @ARGV;
if ($url eq "--skip-wrong-store") {
$skipWrongStore = 1;
# No-op, no longer supported.
} else {
processURL $url;
}
}
my $size = scalar (keys %narFiles) + scalar (keys %localPaths);
print "$size store paths in manifest\n";
# Update the cache.
updateManifestDB();
Regular → Executable
+33 -23
View File
@@ -1,8 +1,10 @@
#! @perl@ -w -I@libexecdir@/nix
#! @perl@ -w @perlFlags@
use strict;
use File::Temp qw(tempdir);
use readmanifest;
use File::stat;
use Nix::Config;
use Nix::Manifest;
my $hashAlgo = "sha256";
@@ -12,15 +14,10 @@ my $tmpDir = tempdir("nix-push.XXXXXX", CLEANUP => 1, TMPDIR => 1)
my $nixExpr = "$tmpDir/create-nars.nix";
my $manifest = "$tmpDir/MANIFEST";
my $curl = "@curl@ --fail --silent";
my $curl = "$Nix::Config::curl --fail --silent";
my $extraCurlFlags = ${ENV{'CURL_FLAGS'}};
$curl = "$curl $extraCurlFlags" if defined $extraCurlFlags;
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
my $dataDir = $ENV{"NIX_DATA_DIR"};
$dataDir = "@datadir@" unless defined $dataDir;
# Parse the command line.
my $localCopy;
@@ -80,7 +77,7 @@ foreach my $path (@ARGV) {
# Get all paths referenced by the normalisation of the given
# Nix expression.
my $pid = open(READ,
"$binDir/nix-store --query --requisites --force-realise " .
"$Nix::Config::binDir/nix-store --query --requisites --force-realise " .
"--include-outputs '$path'|") or die;
while (<READ>) {
@@ -105,8 +102,8 @@ foreach my $storePath (@storePaths) {
# Construct a Nix expression that creates a Nix archive.
my $nixexpr =
"((import $dataDir/nix/corepkgs/nar/nar.nix) " .
"{storePath = builtins.storePath \"$storePath\"; system = \"@system@\"; hashAlgo = \"$hashAlgo\";}) ";
"(import <nix/nar.nix> " .
"{ storePath = builtins.storePath \"$storePath\"; system = \"@system@\"; hashAlgo = \"$hashAlgo\"; }) ";
print NIX $nixexpr;
}
@@ -118,7 +115,7 @@ close NIX;
# Instantiate store derivations from the Nix expression.
my @storeExprs;
print STDERR "instantiating store derivations...\n";
my $pid = open(READ, "$binDir/nix-instantiate $nixExpr|")
my $pid = open(READ, "$Nix::Config::binDir/nix-instantiate $nixExpr|")
or die "cannot run nix-instantiate";
while (<READ>) {
chomp;
@@ -140,7 +137,7 @@ while (scalar @tmp > 0) {
my @tmp2 = @tmp[0..$n - 1];
@tmp = @tmp[$n..scalar @tmp - 1];
my $pid = open(READ, "$binDir/nix-store --realise @tmp2|")
my $pid = open(READ, "$Nix::Config::binDir/nix-store --realise @tmp2|")
or die "cannot run nix-store";
while (<READ>) {
chomp;
@@ -172,29 +169,41 @@ for (my $n = 0; $n < scalar @storePaths; $n++) {
$narbz2Hash =~ /^[0-9a-z]+$/ or die "invalid hash";
close HASH;
open HASH, "$narDir/nar-hash" or die "cannot open nar-hash";
my $narHash = <HASH>;
chomp $narHash;
$narHash =~ /^[0-9a-z]+$/ or die "invalid hash";
close HASH;
my $narName = "$narbz2Hash.nar.bz2";
my $narFile = "$narDir/$narName";
(-f $narFile) or die "narfile for $storePath not found";
push @narArchives, $narFile;
my $narbz2Size = (stat $narFile)[7];
my $narbz2Size = stat($narFile)->size;
my $references = `$binDir/nix-store --query --references '$storePath'`;
my $references = `$Nix::Config::binDir/nix-store --query --references '$storePath'`;
die "cannot query references for `$storePath'" if $? != 0;
$references = join(" ", split(" ", $references));
my $deriver = `$binDir/nix-store --query --deriver '$storePath'`;
my $deriver = `$Nix::Config::binDir/nix-store --query --deriver '$storePath'`;
die "cannot query deriver for `$storePath'" if $? != 0;
chomp $deriver;
$deriver = "" if $deriver eq "unknown-deriver";
my $narHash = `$Nix::Config::binDir/nix-store --query --hash '$storePath'`;
die "cannot query hash for `$storePath'" if $? != 0;
chomp $narHash;
# In some exceptional cases (such as VM tests that use the Nix
# store of the host), the database doesn't contain the hash. So
# compute it.
if ($narHash =~ /^sha256:0*$/) {
$narHash = `$Nix::Config::binDir/nix-hash --type sha256 --base32 '$storePath'`;
die "cannot hash `$storePath'" if $? != 0;
chomp $narHash;
$narHash = "sha256:$narHash";
}
my $narSize = `$Nix::Config::binDir/nix-store --query --size '$storePath'`;
die "cannot query size for `$storePath'" if $? != 0;
chomp $narSize;
my $url;
if ($localCopy) {
$url = "$targetArchivesUrl/$narName";
@@ -205,7 +214,8 @@ for (my $n = 0; $n < scalar @storePaths; $n++) {
{ url => $url
, hash => "$hashAlgo:$narbz2Hash"
, size => $narbz2Size
, narHash => "$hashAlgo:$narHash"
, narHash => "$narHash"
, narSize => $narSize
, references => $references
, deriver => $deriver
}
Regular → Executable
View File
File mode changed.
-17
View File
@@ -1,17 +0,0 @@
use strict;
sub readConfig {
my %config;
my $config = "@sysconfdir@/nix/nix.conf";
return unless -f $config;
open CONFIG, "<$config" or die "cannot open `$config'";
while (<CONFIG>) {
/^\s*([\w|-]+)\s*=\s*(.*)$/ or next;
$config{$1} = $2;
print "|$1| -> |$2|\n";
}
close CONFIG;
}
return 1;
-210
View File
@@ -1,210 +0,0 @@
use strict;
sub addPatch {
my ($patches, $storePath, $patch) = @_;
$$patches{$storePath} = []
unless defined $$patches{$storePath};
my $patchList = $$patches{$storePath};
my $found = 0;
foreach my $patch2 (@{$patchList}) {
$found = 1 if
$patch2->{url} eq $patch->{url} &&
$patch2->{basePath} eq $patch->{basePath};
}
push @{$patchList}, $patch if !$found;
return !$found;
}
sub readManifest {
my ($manifest, $narFiles, $localPaths, $patches) = @_;
open MANIFEST, "<$manifest"
or die "cannot open `$manifest': $!";
my $inside = 0;
my $type;
my $manifestVersion = 2;
my $storePath;
my $url;
my $hash;
my $size;
my $basePath;
my $baseHash;
my $patchType;
my $narHash;
my $references;
my $deriver;
my $hashAlgo;
my $copyFrom;
while (<MANIFEST>) {
chomp;
s/\#.*$//g;
next if (/^$/);
if (!$inside) {
if (/^\s*(\w*)\s*\{$/) {
$type = $1;
$type = "narfile" if $type eq "";
$inside = 1;
undef $storePath;
undef $url;
undef $hash;
undef $size;
undef $narHash;
undef $basePath;
undef $baseHash;
undef $patchType;
$references = "";
$deriver = "";
$hashAlgo = "md5";
}
} else {
if (/^\}$/) {
$inside = 0;
if ($type eq "narfile") {
$$narFiles{$storePath} = []
unless defined $$narFiles{$storePath};
my $narFileList = $$narFiles{$storePath};
my $found = 0;
foreach my $narFile (@{$narFileList}) {
$found = 1 if $narFile->{url} eq $url;
}
if (!$found) {
push @{$narFileList},
{ url => $url, hash => $hash, size => $size
, narHash => $narHash, references => $references
, deriver => $deriver, hashAlgo => $hashAlgo
};
}
}
elsif ($type eq "patch") {
addPatch $patches, $storePath,
{ url => $url, hash => $hash, size => $size
, basePath => $basePath, baseHash => $baseHash
, narHash => $narHash, patchType => $patchType
, hashAlgo => $hashAlgo
};
}
elsif ($type eq "localPath") {
$$localPaths{$storePath} = []
unless defined $$localPaths{$storePath};
my $localPathsList = $$localPaths{$storePath};
# !!! remove duplicates
push @{$localPathsList},
{ copyFrom => $copyFrom, references => $references
, deriver => ""
};
}
}
elsif (/^\s*StorePath:\s*(\/\S+)\s*$/) { $storePath = $1; }
elsif (/^\s*CopyFrom:\s*(\/\S+)\s*$/) { $copyFrom = $1; }
elsif (/^\s*Hash:\s*(\S+)\s*$/) { $hash = $1; }
elsif (/^\s*URL:\s*(\S+)\s*$/) { $url = $1; }
elsif (/^\s*Size:\s*(\d+)\s*$/) { $size = $1; }
elsif (/^\s*SuccOf:\s*(\/\S+)\s*$/) { } # obsolete
elsif (/^\s*BasePath:\s*(\/\S+)\s*$/) { $basePath = $1; }
elsif (/^\s*BaseHash:\s*(\S+)\s*$/) { $baseHash = $1; }
elsif (/^\s*Type:\s*(\S+)\s*$/) { $patchType = $1; }
elsif (/^\s*NarHash:\s*(\S+)\s*$/) { $narHash = $1; }
elsif (/^\s*References:\s*(.*)\s*$/) { $references = $1; }
elsif (/^\s*Deriver:\s*(\S+)\s*$/) { $deriver = $1; }
elsif (/^\s*ManifestVersion:\s*(\d+)\s*$/) { $manifestVersion = $1; }
# Compatibility;
elsif (/^\s*NarURL:\s*(\S+)\s*$/) { $url = $1; }
elsif (/^\s*MD5:\s*(\S+)\s*$/) { $hash = "md5:$1"; }
}
}
close MANIFEST;
return $manifestVersion;
}
sub writeManifest {
my ($manifest, $narFiles, $patches) = @_;
open MANIFEST, ">$manifest.tmp"; # !!! check exclusive
print MANIFEST "version {\n";
print MANIFEST " ManifestVersion: 3\n";
print MANIFEST "}\n";
foreach my $storePath (sort (keys %{$narFiles})) {
my $narFileList = $$narFiles{$storePath};
foreach my $narFile (@{$narFileList}) {
print MANIFEST "{\n";
print MANIFEST " StorePath: $storePath\n";
print MANIFEST " NarURL: $narFile->{url}\n";
print MANIFEST " Hash: $narFile->{hash}\n" if defined $narFile->{hash};
print MANIFEST " NarHash: $narFile->{narHash}\n";
print MANIFEST " Size: $narFile->{size}\n" if defined $narFile->{size};
print MANIFEST " References: $narFile->{references}\n"
if defined $narFile->{references} && $narFile->{references} ne "";
print MANIFEST " Deriver: $narFile->{deriver}\n"
if defined $narFile->{deriver} && $narFile->{deriver} ne "";
print MANIFEST "}\n";
}
}
foreach my $storePath (sort (keys %{$patches})) {
my $patchList = $$patches{$storePath};
foreach my $patch (@{$patchList}) {
print MANIFEST "patch {\n";
print MANIFEST " StorePath: $storePath\n";
print MANIFEST " NarURL: $patch->{url}\n";
print MANIFEST " Hash: $patch->{hash}\n";
print MANIFEST " NarHash: $patch->{narHash}\n";
print MANIFEST " Size: $patch->{size}\n";
print MANIFEST " BasePath: $patch->{basePath}\n";
print MANIFEST " BaseHash: $patch->{baseHash}\n";
print MANIFEST " Type: $patch->{patchType}\n";
print MANIFEST "}\n";
}
}
close MANIFEST;
rename("$manifest.tmp", $manifest)
or die "cannot rename $manifest.tmp: $!";
# Create a bzipped manifest.
system("@bzip2@ < $manifest > $manifest.bz2.tmp") == 0
or die "cannot compress manifest";
rename("$manifest.bz2.tmp", "$manifest.bz2")
or die "cannot rename $manifest.bz2.tmp: $!";
}
return 1;
+1 -2
View File
@@ -6,10 +6,9 @@ use readmanifest;
for my $p (@ARGV) {
my %narFiles;
my %localPaths;
my %patches;
readManifest $p, \%narFiles, \%localPaths, \%patches;
readManifest $p, \%narFiles, \%patches;
%patches = ();
-53
View File
@@ -1,53 +0,0 @@
#! /usr/bin/perl -w -I.
use strict;
use readmanifest;
die unless scalar @ARGV == 2;
my $cache = $ARGV[0];
my $manifest = $ARGV[1];
my %narFiles;
my %localPaths;
my %patches;
readManifest $manifest, \%narFiles, \%localPaths, \%patches;
foreach my $storePath (keys %narFiles) {
my $narFileList = $narFiles{$storePath};
foreach my $narFile (@{$narFileList}) {
if (!defined $narFile->{size} or
!defined $narFile->{narHash})
{
$narFile->{url} =~ /\/([^\/]+)$/;
die unless defined $1;
my $fn = "$cache/$1";
my @info = stat $fn or die;
$narFile->{size} = $info[7];
my $narHash;
my $hashFile = "$fn.NARHASH";
if (-e $hashFile) {
open HASH, "<$hashFile" or die;
$narHash = <HASH>;
close HASH;
} else {
print "$fn\n";
$narHash = `bunzip2 < '$fn' | nix-hash --flat /dev/stdin` or die;
open HASH, ">$hashFile" or die;
print HASH $narHash;
close HASH;
}
chomp $narHash;
$narFile->{narHash} = $narHash;
}
}
}
if (! -e "$manifest.backup") {
system "mv --reply=no '$manifest' '$manifest.backup'";
}
writeManifest $manifest, \%narFiles, \%patches;
-2
View File
@@ -1,5 +1,3 @@
SUBDIRS = bin2c boost libutil libstore libmain nix-store nix-hash \
libexpr nix-instantiate nix-env nix-worker nix-setuid-helper \
nix-log2xml bsdiff-4.3
EXTRA_DIST = aterm-helper.pl
-179
View File
@@ -1,179 +0,0 @@
#! /usr/bin/perl -w
# This program generates C/C++ code for efficiently manipulating
# ATerms. It generates functions to build and match ATerms according
# to a set of constructor definitions defined in a file read from
# standard input. A constructor is defined by a line with the
# following format:
#
# SYM | ARGS | TYPE | FUN?
#
# where SYM is the name of the constructor, ARGS is a
# whitespace-separated list of argument types, TYPE is the type of the
# resulting ATerm (which should be `ATerm' or a type synonym for
# `ATerm'), and the optional FUN is used to construct the names of the
# build and match functions (it defaults to SYM; overriding it is
# useful if there are overloaded constructors, e.g., with different
# arities). Note that SYM may be empty.
#
# A line of the form
#
# VAR = EXPR
#
# causes a ATerm variable to be generated that is initialised to the
# value EXPR.
#
# Finally, a line of the form
#
# init NAME
#
# causes the initialisation function to be called `NAME'. This
# function must be called before any of the build/match functions or
# the generated variables are used.
die if scalar @ARGV != 2;
my $syms = "";
my $init = "";
my $initFun = "init";
open HEADER, ">$ARGV[0]";
open IMPL, ">$ARGV[1]";
print HEADER "#include <aterm2.h>\n";
print HEADER "#ifdef __cplusplus\n";
print HEADER "namespace nix {\n";
print HEADER "#endif\n\n\n";
print IMPL "namespace nix {\n";
while (<STDIN>) {
s/\#.*//;
next if (/^\s*$/);
if (/^\s*(\w*)\s*\|([^\|]*)\|\s*(\w+)\s*\|\s*(\w+)?/) {
my $const = $1;
my @types = split ' ', $2;
my $result = $3;
my $funname = $4;
$funname = $const unless defined $funname;
my $formals = "";
my $formals2 = "";
my $args = "";
my $unpack = "";
my $n = 1;
foreach my $type (@types) {
my $realType = $type;
$args .= ", ";
if ($type eq "string") {
# $args .= "(ATerm) ATmakeAppl0(ATmakeAFun((char *) e$n, 0, ATtrue))";
# $type = "const char *";
$type = "ATerm";
$args .= "e$n";
# !!! in the matcher, we should check that the
# argument is a string (i.e., a nullary application).
} elsif ($type eq "int") {
$args .= "(ATerm) ATmakeInt(e$n)";
} elsif ($type eq "ATermList" || $type eq "ATermBlob") {
$args .= "(ATerm) e$n";
} else {
$args .= "e$n";
}
$formals .= ", " if $formals ne "";
$formals .= "$type e$n";
$formals2 .= ", ";
$formals2 .= "$type & e$n";
my $m = $n - 1;
# !!! more checks here
if ($type eq "int") {
$unpack .= " e$n = ATgetInt((ATermInt) ATgetArgument(e, $m));\n";
} elsif ($type eq "ATermList") {
$unpack .= " e$n = (ATermList) ATgetArgument(e, $m);\n";
} elsif ($type eq "ATermBlob") {
$unpack .= " e$n = (ATermBlob) ATgetArgument(e, $m);\n";
} elsif ($realType eq "string") {
$unpack .= " e$n = ATgetArgument(e, $m);\n";
$unpack .= " if (ATgetType(e$n) != AT_APPL) return false;\n";
} else {
$unpack .= " e$n = ATgetArgument(e, $m);\n";
}
$n++;
}
my $arity = scalar @types;
print HEADER "extern AFun sym$funname;\n\n";
print IMPL "AFun sym$funname = 0;\n";
if ($arity == 0) {
print HEADER "extern ATerm const$funname;\n\n";
print IMPL "ATerm const$funname = 0;\n";
}
print HEADER "static inline $result make$funname($formals) __attribute__ ((pure, nothrow));\n";
print HEADER "static inline $result make$funname($formals) {\n";
if ($arity == 0) {
print HEADER " return const$funname;\n";
}
elsif ($arity <= 6) {
print HEADER " return (ATerm) ATmakeAppl$arity(sym$funname$args);\n";
} else {
$args =~ s/^,//;
print HEADER " ATerm array[$arity] = {$args};\n";
print HEADER " return (ATerm) ATmakeApplArray(sym$funname, array);\n";
}
print HEADER "}\n\n";
print HEADER "#ifdef __cplusplus\n";
print HEADER "static inline bool match$funname(ATerm e$formals2) {\n";
print HEADER " if (ATgetType(e) != AT_APPL || (AFun) ATgetAFun(e) != sym$funname) return false;\n";
print HEADER "$unpack";
print HEADER " return true;\n";
print HEADER "}\n";
print HEADER "#endif\n\n\n";
$init .= " sym$funname = ATmakeAFun(\"$const\", $arity, ATfalse);\n";
$init .= " ATprotectAFun(sym$funname);\n";
if ($arity == 0) {
$init .= " const$funname = (ATerm) ATmakeAppl0(sym$funname);\n";
$init .= " ATprotect(&const$funname);\n";
}
}
elsif (/^\s*(\w+)\s*=\s*(.*)$/) {
my $name = $1;
my $value = $2;
print HEADER "extern ATerm $name;\n";
print IMPL "ATerm $name = 0;\n";
$init .= " $name = $value;\n";
$init .= " ATprotect(&$name);\n";
}
elsif (/^\s*init\s+(\w+)\s*$/) {
$initFun = $1;
}
else {
die "bad line: `$_'";
}
}
print HEADER "void $initFun();\n\n";
print HEADER "static inline const char * aterm2String(ATerm t) {\n";
print HEADER " return (const char *) ATgetName(ATgetAFun(t));\n";
print HEADER "}\n\n";
print IMPL "\n";
print IMPL "void $initFun() {\n";
print IMPL "$init";
print IMPL "}\n";
print HEADER "#ifdef __cplusplus\n";
print HEADER "}\n";
print HEADER "#endif\n\n\n";
print IMPL "}\n";
close HEADER;
close IMPL;
Loaded 100 of 310 files, more files were not shown because too many files have changed in this diff. Show more