Compare commits

..
Author SHA1 Message Date
Eelco Dolstra 29fd60131b * Add a primop to return the context of a string. 2010-01-31 20:54:44 +00:00
Eelco Dolstra 6d1abdc6d9 * In string contexts containing derivations
(e.g. "${somePkg}/bin/foo"), store the original derivation
  expression in the context.
2010-01-31 20:08:57 +00:00
Eelco Dolstra 25fc95d56d 2010-01-31 20:06:58 +00:00
Eelco Dolstra 07ffdc2862 * Added an option "fsync-metadata" to fsync() changes to
/nix/var/nix/db.
* Removed the function writeStringToFile since it does (almost) the
  same thing as writeFile.
2010-01-29 12:22:58 +00:00
Eelco Dolstra ad529fb89f * Don't consider a store path valid if its info file exists but is
zero bytes long.  That makes Nix more robust in case of crashes
  (especially on ext4).
2010-01-29 11:53:58 +00:00
Eelco Dolstra fdcaf37361 * Made `nix-store -qR --include-outputs' much faster if there are
multiple paths specified on the command line (from O(n * m) to O(n +
  m), where n is the number of arguments and m is the size of the
  closure).
2010-01-25 17:18:44 +00:00
Eelco Dolstra 50e34891f0 * Disable gzip compression in build-remote.pl because it puts too much
load on the Hydra build farm (where it's unnecessary anyway because
  it has a fast connection to the build machines).  In any case,
  compression can be enabled by using the `-C' option to ssh.
2010-01-25 16:14:45 +00:00
Eelco Dolstra 5388944e8d * Make the garbage collector do the right thing when `gc-keep-outputs'
is enabled by not depending on the deriver.
2010-01-25 16:04:32 +00:00
Eelco Dolstra f0c0277970 * On startup, set the default SIGCHLD handler. This is so that Nix
works correctly in weird environments where the SIGCHLD handler is
  set to "ignore".
2010-01-12 12:22:38 +00:00
Eelco Dolstra ef92a14bfe * Include config.h before the C library headers, because it defines
_FILE_OFFSET_BITS=64.  Without it, functions like stat() fail on
  large file sizes.  This happened with a Nix store on squashfs:

  $ nix-store --dump /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds > /dev/null
  error: getting attributes of path `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds': Value too large for defined data type

  $ stat /tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds
  File: `/tmp/mnt/46wzqnk4cbdwh1dclhrpqnnz1icak6n7-local-net-cmds'
  Size: 0               Blocks: 36028797018963968 IO Block: 1024   regular empty file

  (This is a bug in squashfs or mksquashfs, but it shouldn't cause Nix
  to fail.)
2009-12-17 14:12:44 +00:00
Eelco Dolstra 945d8218fb * Build correctly against newer ATerm releases. Fixes "error: 'union'
tag used in naming 'struct _ATerm'".
2009-12-16 15:29:50 +00:00
Eelco Dolstra d8a5dc02fc * Build on Fedora 12. 2009-12-10 13:14:22 +00:00
Eelco Dolstra 3a78af1e24 * Release notes. 2009-12-09 21:02:24 +00:00
Eelco Dolstra 7ca9972636 * When doing a nix-pull, remove old manifests downloaded from the same
URL.  This prevents lots of old cruft accumulating in
  /nix/var/nix/manifests.
2009-12-09 19:36:54 +00:00
Eelco Dolstra c4c84d1edb * nix-build: be less verbose. 2009-12-09 18:08:28 +00:00
Eelco Dolstra bcd6cdf0d8 * Give a better error message when trying to build something and
readOnlyMode is set.
2009-12-09 17:45:22 +00:00
Eelco Dolstra 13618b191e * Grrr. 2009-11-24 13:28:46 +00:00
Eelco Dolstra aa5a768720 * GCC 4.4 is stricter about the EOF macro
(http://hydra.nixos.org/build/156340).
2009-11-24 12:56:26 +00:00
Eelco Dolstra 9b8fda796b * Templatise getIntArg / string2Int. 2009-11-24 12:26:25 +00:00
Eelco Dolstra 8022015552 * In the garbage collector, don't count files with a link count > 1 in
the "bytes/blocks freed" statistics.
2009-11-24 10:51:52 +00:00
Eelco Dolstra f9e766db98 * Randomise the order in which we delete entries to make the collector
less biased towards deleting paths that come alphabetically first
  (e.g. /nix/store/000...).  This matters when using --max-freed etc.
2009-11-24 09:53:18 +00:00
Eelco Dolstra ca50c83fbb 2009-11-23 21:21:29 +00:00
Eelco Dolstra 3d55f1eb57 * A command `nix-store --query --roots <paths>' to find the garbage
collector roots that point (directly or indirectly) to the given
  paths.
2009-11-23 18:16:25 +00:00
Eelco Dolstra ae6bf87273 * `nix-store --gc --print-roots': also print the path of the actual
root symlink, not just its target.  E.g.:

  /nix/var/nix/profiles/system-99-link -> /nix/store/76kwf88657nq7wgk1hx3l1z5q91zb9wd-system
2009-11-23 17:23:12 +00:00
Eelco Dolstra c364d5d1e3 * Made the garbage collector a lot faster. It no longer computes the
complete set of live and dead paths before starting the actual
  deletion, but determines liveness on demand.  I.e. for any path in
  the store, it first tries to delete all the referrers, and then the
  path itself.  This means that the collector can start deleting paths
  almost immediately.
2009-11-23 16:34:24 +00:00
Eelco Dolstra 4f7e5f5810 * Don't create /nix/var/nix/gcroots/{tmp,channels}, since they don't
seem to be used anymore.
2009-11-23 12:48:54 +00:00
Eelco Dolstra 8824d60fe5 * Remove the --use-atime / --max-atime garbage collector flags. Many
(Linux) machines no longer maintain the atime because it's too
  expensive, and on the machines where --use-atime is useful (like the
  buildfarm), reading the atimes on the entire Nix store takes way too
  much time to make it practical.
2009-11-20 17:12:38 +00:00
Eelco Dolstra 997db91e07 * Don't pass -K. It should really inherit the setting of the calling
Nix though.
2009-11-17 16:22:39 +00:00
Eelco Dolstra 3392d32e8b * In nix-pull/nix-channel, create the manifests directory if it
doesn't exist.  The Debian packages don't include the manifests
  directory, so nix-channel would silently skip doing a nix-pull,
  resulting in everything being built from source.  Thanks to Juan
  Pedro Bolívar Puente.
2009-11-13 10:08:31 +00:00
Eelco Dolstra 327a232c85 * Remove support for old (before Nix 0.12pre12020) databases. 2009-11-06 01:15:44 +00:00
Eelco Dolstra c60d796f04 * Version bump. 2009-11-05 22:23:38 +00:00
Eelco Dolstra e8bad77c7c 2009-11-05 15:20:19 +00:00
Eelco Dolstra 7680904839 * Build on Karmic. 2009-11-05 14:53:01 +00:00
Eelco Dolstra 58f3338bfa * The Nix .deb package depends on curl. 2009-11-05 14:40:42 +00:00
Eelco Dolstra 268d90a03e * Various updates. 2009-11-05 09:07:43 +00:00
Eelco Dolstra 1ff8758f76 * Manual updates. 2009-11-04 16:52:35 +00:00
Eelco Dolstra 8520542071 * When building in a chroot, make a copy of a file if hard-linking
fails.  This is likely to happen after a `nix-store --optimise',
  because some files may have 32000 links (NIX-111).
2009-10-22 08:28:33 +00:00
Eelco Dolstra 6b9f6b0222 * Remove a prototype for a function that no longer exists. 2009-10-22 08:12:38 +00:00
Eelco Dolstra deb342fb08 * builtins.trace: in the common case that the value is a string, then
show the string, not the ATerm, so we get `trace: bla' instead of
  `trace: Str("bla",[])'.
2009-10-22 08:10:12 +00:00
Eelco Dolstra 437077c39d * Added a primop unsafeDiscardOutputDependency needed by Disnix to
pass derivation paths to a builder without actually building them.
2009-10-21 15:05:30 +00:00
Eelco Dolstra 6f7d7bc1de * Give a useful error message when an evaluation error occurs while
trying to upgrade a package.
2009-10-13 09:30:17 +00:00
Sander van der Burg 53a4981fa2 Added optional parameter which adds -lnsl -lsocket to make the Nix package manager work on OpenSolaris 2009-10-08 14:50:37 +00:00
Peter Simons 18f0ff003d configure.ac: use AC_SYS_LARGEFILE to determine how to enable 64-bit file size support
Defining -D_FILE_OFFSET_BITS=64 works on most platforms, but not on all (i.e.
Solaris). Also, the Autoconf macro offers the user a switch to disable the
functionality in case of problems.
2009-10-06 09:14:06 +00:00
Eelco Dolstra 96f1517831 * Support platforms that don't have O_ASYNC (e.g. OpenSolaris
apparently).
2009-09-30 11:32:04 +00:00
Eelco Dolstra 1a8f8fd86f * OpenSolaris compatibility. 2009-09-30 09:54:29 +00:00
Eelco Dolstra 0f79ad47c5 2009-09-25 12:36:03 +00:00
Eelco Dolstra c7057fc1f2 * And some more. 2009-09-24 07:39:55 +00:00
Eelco Dolstra 193f59e077 * Fix a build failure on Fedora 11. rename() needs <stdio.h>. 2009-09-24 07:21:29 +00:00
Eelco Dolstra 0ae2be5692 2009-09-24 07:05:06 +00:00
Eelco Dolstra 1332dd1ed3 * tryEval shouldn't catch all exceptions of type Error, since not all
of them leave the evaluator in a continuable state.  Also, it should
  be less chatty.
2009-09-23 19:19:26 +00:00
Eelco Dolstra 63a17d4bd5 * Don't build against BDB on Cygwin, it's been broken for unknown
reasons for a while (e.g. http://hydra.nixos.org/build/79164).
2009-09-23 18:52:18 +00:00
Eelco Dolstra 676e07902e * Darwin hack. 2009-09-23 18:04:55 +00:00
Eelco Dolstra 64e89980e8 * Create some state directories automatically as a convenience. 2009-09-23 17:05:51 +00:00
Rob Vermaas 48b58617e9 * include wait.h for WEXITSTATUS 2009-09-23 12:57:15 +00:00
Eelco Dolstra 51ad64cc07 * Use xmllint (>= 2.7.4) for RelaxNG validation instead of Jing. 2009-09-18 11:45:56 +00:00
Eelco Dolstra df05a759e4 * In "make init-state", ignore errors creating /nix/store. Hack to
get the Debian VM builds to work (where /nix/store is a mount point
  containing the store of the host).
2009-09-18 11:01:30 +00:00
Eelco Dolstra d3de71efc9 2009-09-17 17:44:13 +00:00
Eelco Dolstra e1df4ef73c 2009-09-17 17:02:14 +00:00
Eelco Dolstra 86408b3f47 * build-remote.pl: Pick machines in a round-robin order, rather than
giving jobs to the first machine until it hits its job limit, then
  the second machine and so on.  This should improve utilisation of
  the Hydra build farm a lot.  Also take an optional speed factor
  into account to cause fast machines to be preferred over slower
  machines with a similar load.
2009-09-17 15:48:17 +00:00
Eelco Dolstra 57e0d73c77 * build-remote.pl: allow the system type to be a comma-separated list
of system types.  Don't treat the x86_64-linux system type
  specially.
2009-09-17 13:51:04 +00:00
Eelco Dolstra 0dbd4638e0 * Two primops: builtins.intersectAttrs and builtins.functionArgs.
intersectAttrs returns the (right-biased) intersection between two
  attribute sets, e.g. every attribute from the second set that also
  exists in the first.  functionArgs returns the set of attributes
  expected by a function.

  The main goal of these is to allow the elimination of most of
  all-packages.nix.  Most package instantiations in all-packages.nix
  have this form:

    foo = import ./foo.nix {
      inherit a b c;
    };

  With intersectAttrs and functionArgs, this can be written as:

    foo = callPackage (import ./foo.nix) { };

  where

   callPackage = f: args:
     f ((builtins.intersectAttrs (builtins.functionArgs f) pkgs) // args);

  I.e., foo.nix is called with all attributes from "pkgs" that it
  actually needs (e.g., pkgs.a, pkgs.b and pkgs.c).  (callPackage can
  do any other generic package-level stuff we might want, such as
  applying makeOverridable.)  Of course, the automatically supplied
  arguments can be overriden if needed, e.g.

    foo = callPackage (import ./foo.nix) {
      c = c_version_2;
    };

  but for the vast majority of packages, this won't be needed.

  The advantages are to reduce the amount of typing needed to add a
  dependency (from three sites to two), and to reduce the number of
  trivial commits to all-packages.nix.  For the former, there have
  been two previous attempts:

    - Use "args: with args;" in the package's function definition.
      This however obscures the actual expected arguments of a
      function, which is very bad.

    - Use "{ arg1, arg2, ... }:" in the package's function definition
      (i.e. use the ellipis "..." to allow arbitrary additional
      arguments), and then call the function with all of "pkgs" as an
      argument.  But this inhibits error detection if you call it with
      an misspelled (or obsolete) argument.
2009-09-15 13:01:46 +00:00
Michael Raskin 3bca8931e8 Adding tryEval builtin. It allows to catch presence of errors in an expression. 2009-08-25 16:06:46 +00:00
Eelco Dolstra 5e9a4e5101 2009-08-03 13:32:13 +00:00
Eelco Dolstra 9b46d1ae6f 2009-08-03 12:24:20 +00:00
Eelco Dolstra 20b6f94b65 * nix-build: pass the --show-trace flag. 2009-07-15 09:10:38 +00:00
Eelco Dolstra d413612029 * Remove the redundant <sections> around refentries. 2009-07-14 14:58:12 +00:00
Eelco Dolstra 1f169f43b3 * Leave out the collaborators / revision history page. 2009-07-10 13:42:12 +00:00
Eelco Dolstra 5e2e2f10ef 2009-07-10 11:48:49 +00:00
Eelco Dolstra c6890d6b5c * Replace newlines in table output with spaces. 2009-07-02 08:52:12 +00:00
Eelco Dolstra 749dd97a54 * Support integers and lists of strings in meta fields. This is
useful for fields like meta.maintainers, meta.priority (which can be
  a proper integer now) and even meta.license (if there are multiple
  licenses).
2009-06-30 15:53:39 +00:00
Eelco Dolstra f2c3fc5191 * Don't show trace information by default (`--show-trace' to enable).
NixOS evaluation errors in particular look intimidating and
  generally aren't very useful.  Ideally the builtins.throw messages
  should be self-contained.
2009-06-30 13:28:29 +00:00
Eelco Dolstra a2fc3a53ba * Highlight URLs containing "=" properly. 2009-06-18 10:04:14 +00:00
Eelco Dolstra d53603c928 * Attributes names in attrsets can contain dots now. 2009-06-18 09:56:50 +00:00
Eelco Dolstra 14bc3ce3d6 * Canonicalise timestamps in the Nix store to 1 (1970-01-01 00:00:01
UTC) rather than 0 (00:00:00).  1 is a better choice because some
  programs use 0 as a special value.  For instance, the Template
  Toolkit uses a timestamp of 0 to denote the non-existence of a file,
  so it barfs on files in the Nix store (see
  template-toolkit-nix-store.patch in Nixpkgs).  Similarly, Maya 2008
  fails to load script directories with a timestamp of 0 and can't be
  patched because it's closed source.

  This will also shut up those "implausibly old time stamp" GNU tar
  warnings.
2009-06-13 16:30:58 +00:00
Eelco Dolstra f24cf5d303 * nix-build: if nix-instantiate or nix-store fails due to a signal
(e.g. out of memory or a segfault), print an error message.
  Otherwise the user doesn't see anything.
2009-06-10 11:30:34 +00:00
Eelco Dolstra c6cb792d48 2009-05-26 16:32:02 +00:00
Eelco Dolstra a64bbe049e * Change the scoping of "inherit (e) ..." in recs so that the
attributes of the rec are in scope of `e'.  This is useful in
  expressions such as

    rec {
      lib = import ./lib;
      inherit (lib) concatStrings;
    }

  It does change the semantics of expressions such as

    let x = {y = 1;}; in rec { x = {y = 2;}; inherit (x) y; }.y

  This now returns 2 instead of 1.  However, no code in Nixpkgs or
  NixOS seems to rely on the old behaviour.
2009-05-15 13:46:13 +00:00
Eelco Dolstra d407d572fd * Some syntactic sugar for attribute sets: allow {x.y.z = ...;} as a
shorthand for {x = {y = {z = ...;};};}.  This is especially useful
  for NixOS configuration files, e.g.

    {
      services = {
        sshd = {
          enable = true;
          port = 2022;
        };
      };
    }

  can now be written as

    {
      services.sshd.enable = true;
      services.sshd.port = 2022;
    }

  However, it is currently not permitted to write
  
    {
      services.sshd = {enable = true;};
      services.sshd.port = 2022;
    }

  as this is considered a duplicate definition of `services.sshd'.
2009-05-15 12:35:23 +00:00
Eelco Dolstra e42975490f * Check for duplicate attributes in fixAttrs, rather than doing a
separate traversal after parsing.  Likewise, check for duplicate
  pattern variables right away.
2009-05-14 14:29:45 +00:00
Eelco Dolstra 50d11b90ca * Allow unsafe (unspecified) comparisons between attrsets unless
NIX_NO_UNSAFE_EQ is set, for now.
2009-05-12 11:06:24 +00:00
Eelco Dolstra c34e6d71bc * Disallow equality tests between attribute sets. This was always
broken, but now the evaluator checks for it to prevent Nix
  expressions from relying on undefined behaviour.  Equality tests are
  implemented using a shallow pointer equality test between ATerms.
  However, because attribute sets are lazy and contain position
  information, this can give false positives.  For instance,
  previously

    let y = {x = 1;}; in y == y

  evaluated to true, while the equivalent expression

    {x = 1;} == {x = 1;}

  evaluated to false.  So disallow these tests for now.  (Eventually
  we may want to implement deep equality tests for attribute sets,
  like lib.eqStrict.)
  
* Idem: disallow comparisons between functions.

* Implemented deep comparisons of lists.  This had the same problem as
  attribute sets - the elements in the list weren't evaluated.  For
  instance,

    ["xy"] == [("x" + "y")]

  evaluated to false.  Now it works properly.
2009-05-11 15:50:14 +00:00
Eelco Dolstra 9536ba19d4 2009-05-07 13:22:26 +00:00
Eelco Dolstra 83bd320b39 * Build a PDF version of the manual in the tarball job (it's not
included in the tarball though).
2009-05-07 13:12:46 +00:00
Eelco Dolstra f751c2966b * Don't have a bugs section in the manual. Bugs should be kept in the
bug tracker.
2009-05-07 13:11:58 +00:00
Eelco Dolstra 404996ca1f * Produce the manual in the tarball job. 2009-05-07 12:57:04 +00:00
Eelco Dolstra 7660e2a068 * Remove a right recursion that causes the parser to barf on very long
lists.  The comment about ATreverse requiring unbounded stack space
  was unfounded anyway.
2009-05-07 11:35:52 +00:00
Eelco Dolstra 52a9ba96f5 * Include the version file. 2009-05-07 11:34:08 +00:00
Eelco Dolstra 6a5599fd01 * Typo. 2009-05-07 11:33:57 +00:00
Michael Raskin 098cb9d233 Add an ftruncate call paired with fallocate to play safe with some FSes (namely, BtrFS fallocate sets file size to allocated size, i.e. multiple of block size) 2009-05-04 08:10:24 +00:00
Eelco Dolstra c710fe540e * Machines of type x86_64-linux can do i686-linux builds. 2009-04-21 12:42:47 +00:00
Eelco Dolstra d5eab2fc82 * Use foreach in a lot of places. 2009-04-21 11:52:16 +00:00
Eelco Dolstra 8f1bf28505 * nix-store --verify: don't bail out if a referenced path is missing.
(It can't fix it though.)
2009-04-21 11:06:27 +00:00
Eelco Dolstra 4e646b0ddb * Fix a few "comparison is always false/true due to limited range of
data type" warnings on 64-bit platforms.  The one in parser.y is
  likely to be a real bug.
2009-04-16 12:03:17 +00:00
Eelco Dolstra 0460ea4c39 * Cleanup. 2009-04-16 09:27:33 +00:00
Eelco Dolstra dfb863f333 * Don't cache transient build hook problems. 2009-04-15 06:25:02 +00:00
Eelco Dolstra 435a93b5d8 * Remove references to __XXX primops. 2009-04-14 13:03:27 +00:00
Eelco Dolstra 8b2a01a8c2 * Release notes. 2009-04-14 12:02:22 +00:00
Eelco Dolstra 50cb6f9782 * Debian packages: use /etc, not /usr/etc. 2009-04-02 08:16:15 +00:00
Eelco Dolstra 351bf658f9 * Do a substitution even if --max-jobs == 0. 2009-03-31 21:14:07 +00:00
Eelco Dolstra cff2b2a13a * Doh! 2009-03-31 15:50:03 +00:00
Eelco Dolstra 160a60d663 2009-03-30 19:35:55 +00:00
Eelco Dolstra 805144b705 * Make the poll interval configurable. 2009-03-30 11:34:03 +00:00
Eelco Dolstra 096affb55b * Update the mtime on the lock file to make it easy to see when a slot
was last used.
2009-03-29 18:40:44 +00:00
Eelco Dolstra cbc1f57b48 * Undocument the "system" option. No sane person would use it :-) 2009-03-29 18:08:32 +00:00
Eelco Dolstra 7377195297 * With `--max-jobs 0', print a nicer error message than "Assertion
`!awake.empty()' failed."
2009-03-29 18:06:00 +00:00
Eelco Dolstra 737423a89c * Use polling to wait for a remote build slot when using a build hook
(that is, call the build hook with a certain interval until it
  accepts the build).
* build-remote.pl was totally broken: for all system types other than
  the local system type, it would send all builds to the *first*
  machine of the appropriate type.
2009-03-29 18:00:45 +00:00
Eelco Dolstra 47706e3924 * nix-copy-closure: compute the closure in one call to nix-store,
which is much faster.
2009-03-28 21:10:29 +00:00
Eelco Dolstra 6e946c8e72 * Scan for references and compute the SHA-256 hash of the output in
one pass.  This halves the amount of I/O.
2009-03-28 20:51:33 +00:00
Eelco Dolstra c7152c8f97 * Don't use the non-standard __gnu_cxx::stdio_filebuf class. 2009-03-28 19:41:53 +00:00
Eelco Dolstra 3a2bbe7f8a * Simplify communication with the hook a bit (don't use file
descriptors 3/4, just use stdin/stderr).
2009-03-28 19:29:55 +00:00
Eelco Dolstra 7fb548aa26 * Clean up the output a bit. 2009-03-28 16:12:46 +00:00
Eelco Dolstra f54e800366 * The `fixedOutput' variable didn't get initialised when using a build
hook, causing negative caching to fail randomly.
2009-03-28 16:12:27 +00:00
Eelco Dolstra 38f98b3282 * Argh, stupid timing sensitive tests... 2009-03-27 22:40:22 +00:00
Eelco Dolstra e23a7a8b7b * Debian 5.0, Ubuntu 8.10 builds. 2009-03-27 14:20:03 +00:00
Eelco Dolstra cb85bc396e * Create /nix/store if it doesn't exist. 2009-03-27 14:19:04 +00:00
Eelco Dolstra 61686926d1 * A job for creating a fully statically linked binary tarball. 2009-03-26 14:12:42 +00:00
Eelco Dolstra 9856efc7b0 * Support doing a fully static build. Statically linked Nix binaries
will be useful as a fallback on platforms for which there are no
  RPMs/Debs available.
2009-03-26 12:22:08 +00:00
Eelco Dolstra 5dd8fb2069 * Don't use ULLONG_MAX in maxFreed - use 0 to mean "no limit".
18446744073709551615ULL breaks on GCC 3.3.6 (`integer constant is
  too large for "long" type').
2009-03-26 11:02:07 +00:00
Eelco Dolstra 92f525ecf4 * Negative caching, i.e. caching of build failures. Disabled by
default.  This is mostly useful for Hydra.
2009-03-25 21:05:42 +00:00
Eelco Dolstra 7024a1ef07 * Removed the locking.sh test; it's redundant because of the extended
parallel.sh test.  Also, don't call multiple nix-builds in parallel,
  since they can race creating .nix-build-tmp-derivation.
2009-03-25 16:43:38 +00:00
Eelco Dolstra 8146a0c731 * Use bash in the tests. 2009-03-25 16:11:04 +00:00
Eelco Dolstra a046858a22 * Doh. 2009-03-24 23:40:24 +00:00
Eelco Dolstra 12c8c64efa 2009-03-24 14:07:37 +00:00
Eelco Dolstra 7aedcf9460 * Make this test a bit more robust. It's still timing dependent
though.
2009-03-23 15:16:36 +00:00
Eelco Dolstra 3a87163b24 * Shut up a GCC warning. 2009-03-23 13:52:52 +00:00
Eelco Dolstra cacff1be88 * No longer block while waiting for a lock on a store path. Instead
poll for it (i.e. if we can't acquire the lock, then let the main
  select() loop wait for at most a few seconds and then try again).
  This improves parallelism: if two nix-store processes are both
  trying to build a path at the same time, the second one shouldn't
  block; it should first see if it can build other goals.  Also, it
  prevents the deadlocks that have been occuring in Hydra lately,
  where a process waits for a lock held by another process that's
  waiting for a lock held by the first.

  The downside is that polling isn't really elegant, but POSIX doesn't
  provide a way to wait for locks in a select() loop.  The only
  solution would be to spawn a thread for each lock to do a blocking
  fcntl() and then signal the main thread, but that would require
  pthreads.
2009-03-23 01:05:54 +00:00
Eelco Dolstra 58969fa2bf * Refactoring. 2009-03-22 23:53:05 +00:00
Eelco Dolstra d7b2d11255 * Test case (currently fails): multiple Nix builds shouldn't block
waiting on the same lock when there are other builds that can be
  done.
2009-03-22 23:16:18 +00:00
Eelco Dolstra 7a57b2920b * Better error message when nix-store --import is applied to garbage
(previously it would likely say "implementation cannot deal with >
  32-bit integers").
2009-03-22 17:51:45 +00:00
Eelco Dolstra 77d272623f * NAR archives: handle files larger than 2^32 bytes. Previously it
would just silently store only (fileSize % 2^32) bytes.
* Use posix_fallocate if available when unpacking archives.
* Provide a better error message when trying to unpack something that
  isn't a NAR archive.
2009-03-22 17:36:43 +00:00
Eelco Dolstra 7e05b8b75e * Future proofing: assume we can read manifests up to version 10
(which should therefore be backwards compatible).
2009-03-19 10:02:02 +00:00
Eelco Dolstra b88460bcbc * Disregard the Hash field in manifests. 2009-03-19 09:47:34 +00:00
Eelco Dolstra 2897286487 * Unify exportReferencesGraph and exportBuildReferencesGraph, and make
sure that it works as expected when you pass it a derivation.  That
  is, we have to make sure that all build-time dependencies are built,
  and that they are all in the input closure (otherwise remote builds
  might fail, for example).  This is ensured at instantiation time by
  adding all derivations and their sources to inputDrvs and inputSrcs.
2009-03-18 17:36:42 +00:00
Eelco Dolstra e530e0a350 * Improve the test. 2009-03-18 16:36:13 +00:00
Eelco Dolstra 9485ec31ea * Better cleanup after tests. 2009-03-18 16:35:35 +00:00
Eelco Dolstra 13df3915ef * Missing file. 2009-03-18 16:23:29 +00:00
Eelco Dolstra c183ee5c79 * Acquire the locks on the output paths before trying to run the build
hook.  This fixes a problem with log files being partially or
  completely filled with 0's because another nix-store process
  truncates the log file.  It should also be more efficient.
2009-03-18 14:48:42 +00:00
Eelco Dolstra 1dcf208f56 * Clean up some tests (use nix-build where appropriate). 2009-03-18 13:15:55 +00:00
Eelco Dolstra 93b6926054 * Regression test for exportBuildReferencesGraph. It currently fails. 2009-03-17 17:38:32 +00:00
Eelco Dolstra 51e7e32c3b * Refactoring: renamed *.nix.in to *.nix. 2009-03-17 17:11:55 +00:00
Eelco Dolstra 2d5114452d * Regression test for the `exportReferencesGraph'
derivation attribute.
2009-03-17 16:33:48 +00:00
Eelco Dolstra 33ecb42991 * Cleanup. 2009-03-17 11:42:55 +00:00
Eelco Dolstra be88248add * Make the version available to release.nix. 2009-03-09 15:05:08 +00:00
Eelco Dolstra a96cac0d18 2009-03-06 17:00:58 +00:00
Eelco Dolstra d4753c944f 2009-03-06 11:01:45 +00:00
Eelco Dolstra 0e6f604178 * Install some headers in the right location. 2009-03-05 14:57:50 +00:00
Eelco Dolstra 7f254706b0 * Allow the channel to declare a name for itself. 2009-03-03 14:47:39 +00:00
Eelco Dolstra 1273d355ac * nix-install-package: don't pollute /nix/var/nix/manifests. 2009-02-27 14:06:38 +00:00
Eelco Dolstra d4ca5c3952 * Use the regular progress bar; the hash bar isn't very useful when
the size of the download isn't known in advance.
2009-02-27 13:55:11 +00:00
Eelco Dolstra fd2e14b3c8 * nix-build: support --option. 2009-02-27 12:09:30 +00:00
Eelco Dolstra 60cb7de336 * Allow options from the Nix config file to be overriden from the
command line (e.g. "--option build-use-chroot true").
2009-02-27 11:04:41 +00:00
Eelco Dolstra 8ab6bc5a49 * nix-channel: use nix-build. 2009-02-27 11:01:03 +00:00
Eelco Dolstra f052c10eed * Check the manifest version. 2009-02-27 09:53:58 +00:00
Eelco Dolstra a7cee528c5 * Handle base-16 hashes in manifests. 2009-02-26 21:12:35 +00:00
Eelco Dolstra 041717eda3 * download-using-manifests: don't check the cryptographic hash of
downloaded files; rather, check the hash of the unpacked store
  path.

  When the server produces bzipped NAR archives on demand (like Hydra
  does), the hash of the file is not known in advance; it's streamed
  from the server.  Thus the manifest doesn't contain a hash for the
  bzipped NAR archive.  However, the server does know the hash of the
  *uncompressed* NAR archive (the "NarHash" field), since it's stored
  in the Nix database (nix-store -q --hash /nix/store/bla).  So we use
  that instead for checking the integrity of the download.
2009-02-19 23:46:37 +00:00
Michael Raskin 621093cb1c Replace wrong (w.r.t. PATH) sed call with in-shell substitution 2009-02-19 20:46:45 +00:00
Eelco Dolstra 824b154ce8 * Release output locks as soon as possible, not when the destructor of
the DerivationGoal runs.  Otherwise, if a goal is a top-level goal,
  then the lock won't be released until nix-store finishes.  With
  --keep-going and lots of top-level goals, it's possible to run out
  of file descriptors (this happened sometimes in the build farm for
  Nixpkgs).  Also, for failed derivation, it won't be possible to
  build it again  until the lock is released.
  
* Idem for locks on build users: these weren't released in a timely
  manner for failed top-level derivation goals.  So if there were more
  than (say) 10 such failed builds, you would get an error about
  having run out of build users.
2009-02-16 09:24:20 +00:00
Marc Weber 2ef579d1aa documentation for previous commit 2009-02-05 19:35:44 +00:00
Marc Weber 1407a1ec99 added primop functions __isBool, __isString, __isInt 2009-02-05 19:35:40 +00:00
Eelco Dolstra 1bb0f1e84b * Build hook: compress the transferred data. 2009-02-03 10:34:15 +00:00
Eelco Dolstra b682fae9d9 * Build hooks: use nix-store --import. This prevents a redundant
scan for runtime dependencies (i.e. the local machine shouldn't do a
  scan that the remote machine has already done).  Also pipe directly
  into `nix-store --import': don't use a temporary file.
2009-02-02 17:24:10 +00:00
Marc Weber 6f8c96d123 vim syntax: support for indented strings ('' .. '') 2009-01-28 12:14:53 +00:00
Nicolas Pierron 110606d470 Add the "addErrorContext" builtin to add more information in the stack trace. 2009-01-27 14:36:44 +00:00
Eelco Dolstra 061141e632 * Make it compile on Debian 4.0 (which doesn't define PER_LINUX32_3GB
in sys/personality.h).
2009-01-27 13:36:59 +00:00
Eelco Dolstra 019176137f * When using a build hook, distinguish between transient failures
(e.g. an SSH connection problem) and permanent failures (i.e. the
  builder failed).  This matters to Hydra (it wants to know whether it
  makes sense to retry a build).
2009-01-13 11:39:09 +00:00
Eelco Dolstra 4ce692df88 2009-01-13 10:42:13 +00:00
Eelco Dolstra c504d90c11 * Support i686-linux builds directly on x86_64-linux Nix
installations.  This is implemented using the personality() syscall,
  which causes uname to return "i686" in child processes.
2009-01-12 16:30:32 +00:00
Eelco Dolstra 8e39d9bdb3 * Make Nix build with Bison 2.4. 2009-01-12 12:51:28 +00:00
Eelco Dolstra 28355dafcf * Removed reference to losser.st-lab.cs.uu.nl, which is RIP after 9
years of loyal service :-)
2009-01-05 12:14:43 +00:00
Eelco Dolstra 8fce03e0ad * nix-store --verify: repair bad hash fields in the metadata file. 2008-12-16 13:28:18 +00:00
Eelco Dolstra 60ec75048a * Pass --use-atime / --max-atime to the daemon. 2008-12-16 12:23:35 +00:00
Eelco Dolstra 6f6bb1fdea * Delete the chroot if it already exists (e.g. left over from an
interrupted build).
2008-12-15 23:55:11 +00:00
Eelco Dolstra 92cb7c4dfe * Put chroots in the Nix store. This ensures that we can create hard
links to the inputs.
2008-12-12 17:14:57 +00:00
Eelco Dolstra 0008b0006d * Simplify deleting .lock files in /nix/store: just don't delete them
if they belong a path that's currently being built.  This gets rid
  of some Cygwin-specific code.
2008-12-12 17:03:18 +00:00
Eelco Dolstra ac36c6cd44 * Some hackery to make "make check" succeed on Cygwin. 2008-12-12 15:36:18 +00:00
Eelco Dolstra 2b70a8e7c9 * Detect whether unshare() is available. 2008-12-12 13:41:36 +00:00
Eelco Dolstra 9122dcecbb * We can't use string objects in signal handlers because they might
allocate memory, which is verboten in signal handlers.  This caused
  random failures in the test suite on Mac OS X (triggered by the spurious
  SIGPOLL signals on Mac OS X, which should also be fixed).
2008-12-12 12:59:27 +00:00
Eelco Dolstra 6776a52bb3 * Use a PathSet for the chroot directories so that we don't
accidentally bind-mount a directory twice.
2008-12-12 11:49:42 +00:00
Eelco Dolstra d86bd22d24 * Define _GNU_SOURCE. Hopefully this fixes the build on Debian 4.0
(http://hydra.nixos.org/nixlog/384/1).
2008-12-12 10:20:19 +00:00
Eelco Dolstra ac5478eb52 * Don't provide the whole Nix store in the chroot, but only the
closure of the inputs.  This really enforces that there can't be any
  undeclared dependencies on paths in the store.  This is done by
  creating a fake Nix store and creating bind-mounts or hard-links in
  the fake store for all paths in the closure.  After the build, the
  build output is moved from the fake store to the real store.  TODO:
  the chroot has to be on the same filesystem as the Nix store for
  this to work, but this isn't enforced yet.  (I.e. it only works
  currently if /tmp is on the same FS as /nix/store.)
2008-12-11 18:57:10 +00:00
Eelco Dolstra 652817046b * Revert r13150: now that we use private namespaces for the chroot, we
don't have to put the chroot in /nix/var/nix/chroots anymore.
  They're back in /tmp now.
2008-12-11 17:52:34 +00:00
Eelco Dolstra 5a569509b4 * Provide a minimal /etc/passwd in the chroot to keep some builders
happy.
2008-12-11 17:44:02 +00:00
Eelco Dolstra 7c54f1603f * Do chroot builds in a private namespace. This means that all the
bind-mounts we do are only visible to the builder process and its
  children.  So accidentally doing "rm -rf" on the chroot directory
  won't wipe out /nix/store and other bind-mounted directories
  anymore.  Also, the bind-mounts in the private namespace disappear
  automatically when the builder exits.
2008-12-11 17:00:12 +00:00
Eelco Dolstra 07cdfb09fb * Open the connection to the daemon lazily (on demand) so that
read-only operations (like nix-env -qa) work properly when the
  daemon isn't running.
2008-12-11 14:30:25 +00:00
Eelco Dolstra a0766eca27 * Build on Debian 4.0 (GCC 4.1.1). 2008-12-04 21:07:29 +00:00
Eelco Dolstra cd16d5dc3d * Doh. 2008-12-04 17:56:12 +00:00
Eelco Dolstra 5b949241a5 * Build some 64-bit RPMs/Debs. 2008-12-04 17:54:14 +00:00
Eelco Dolstra 67958f21df * Be sure to clean up the daemon if the test fails. 2008-12-04 16:55:22 +00:00
Eelco Dolstra d91dc086bb 2008-12-04 16:51:36 +00:00
Eelco Dolstra 9ac3f5df9c * Propagate --max-silent-time to remote machines. 2008-12-04 16:51:16 +00:00
Eelco Dolstra bcfe98acff * Prefer building on a remote machine over a local machine. This
makes more sense for the build farm, otherwise every nix-store
  invocation will lead to at least one local build.  Will come up with
  a better solution later...
2008-12-04 16:35:47 +00:00
Eelco Dolstra 9850262a72 * Build RPMs, Debs, coverage analysis. 2008-12-04 15:25:28 +00:00
Eelco Dolstra 909fbb9de1 * When using build hooks, for any nix-store -r build operation, it is
necessary that at least one build hook doesn't return "postpone",
  otherwise nix-store will barf ("waiting for a build slot, yet there
  are no running children").  So inform the build hook when this is
  the case, so that it can start a build even when that would exceed
  the maximum load on a machine.
2008-12-04 14:29:41 +00:00
Eelco Dolstra 5dfba0b4db * Force allocation of a pseudo-terminal to clean up the remote
nix-store process when the connection is interrupted.
2008-12-04 13:36:52 +00:00
Eelco Dolstra 9ccdb80de3 * Don't ignore errors from writing to stderr. That way, when
nix-store -r (or some other operation) is started via ssh, it will
  at least have a chance of terminating quickly when the connection is
  killed.  Right now it just runs to completion, because it never
  notices that stderr is no longer connected to anything.  Of course
  it would be better if sshd would just send a SIGHUP, but it doesn't
  (https://bugzilla.mindrot.org/show_bug.cgi?id=396).
2008-12-04 13:13:31 +00:00
Eelco Dolstra 9fd9c4c635 * Support multiple system types per remote machine, e.g. a machine
list like

    root@example.org x86_64-linux /root/.ssh/id_buildfarm 1
    root@example.org i686-darwin  /root/.ssh/id_buildfarm 1

  This is possible when the Nix installation on example.org itself has
  remote builds enabled.
2008-12-04 12:20:06 +00:00
Eelco Dolstra 63b8f09d8d 2008-12-04 10:45:47 +00:00
Eelco Dolstra f8713e1287 * Dirty hack to make nix-push work properly on derivations: the
derivation should be a source rather than a derivation dependency of
  the call to the NAR derivation.  Otherwise the derivation (and all
  its dependencies) will be built as a side-effect, which may not even
  succeed.
2008-12-04 10:40:41 +00:00
Eelco Dolstra 82ae85de27 * addToStore() in nix-worker: don't write the NAR dump received from
the client to a temporary directory, as that is highly inefficient.
2008-12-03 18:05:14 +00:00
Eelco Dolstra 5eaf644c99 * A simple API for parsing NAR archives. 2008-12-03 17:30:32 +00:00
Eelco Dolstra cdee317419 * Backwards compatibility. 2008-12-03 17:02:29 +00:00
Eelco Dolstra d95b68fde3 2008-12-03 16:15:38 +00:00
Eelco Dolstra ff762fb499 * Pass HashType values instead of strings. 2008-12-03 16:10:17 +00:00
Eelco Dolstra 1307b22223 * Made addToStore() a lot more efficient: it no longer reads the path
being copied 3 times in the worst case.  It doesn't run in constant space,
  but it didn't do that anyway.
2008-12-03 15:51:17 +00:00
Eelco Dolstra 64519cfd65 * Unify the treatment of sources copied to the store, and recursive
SHA-256 outputs of fixed-output derivations.  I.e. they now produce
  the same store path:

  $ nix-store --add x
  /nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x

  $ nix-store --add-fixed --recursive sha256 x
  /nix/store/j2fq9qxvvxgqymvpszhs773ncci45xsj-x

  the latter being the same as the path that a derivation

    derivation {
      name = "x";
      outputHashAlgo = "sha256";
      outputHashMode = "recursive";
      outputHash = "...";
      ...
    };

  produces.

  This does change the output path for such fixed-output derivations.
  Fortunately they are quite rare.  The most common use is fetchsvn
  calls with SHA-256 hashes.  (There are a handful of those is
  Nixpkgs, mostly unstable development packages.)
  
* Documented the computation of store paths (in store-api.cc).
2008-12-03 15:06:30 +00:00
Eelco Dolstra 09bc0c502c * Install the release notes. 2008-11-29 00:31:39 +00:00
Eelco Dolstra 5d4eb9dd07 * Moved the build farm job for building Nix to the Nix tree. 2008-11-26 01:13:29 +00:00
Eelco Dolstra 0c478d2f4d * Ignore carriage returns. 2008-11-25 02:30:35 +00:00
Eelco Dolstra 5024bde8f4 * Handle prematurely ended logfiles, i.e. make sure we emit enough
close tags.
2008-11-25 01:06:15 +00:00
Eelco Dolstra 2ab09a55cf * Bump the version number. 2008-11-20 21:51:58 +00:00
Eelco Dolstra 6bbff48079 2008-11-20 17:22:42 +00:00
Eelco Dolstra b8eb32f4d2 * Urgh. 2008-11-20 16:42:52 +00:00
Eelco Dolstra bba87589cc 2008-11-20 16:28:04 +00:00
Eelco Dolstra f3e2e6076a 2008-11-20 16:10:55 +00:00
Eelco Dolstra 53cca4a445 2008-11-20 16:06:13 +00:00
Eelco Dolstra 4213b8d8ec * Urgh. 2008-11-20 15:44:59 +00:00
Eelco Dolstra eb86b6f5a5 2008-11-20 15:08:39 +00:00
Eelco Dolstra 3d2035ea86 * Blindly doing a replacement of occurences of $bindir (when running
the tests) is a bad idea when $bindir = /usr and some programs (like
  perl) live there.  Fortunately it doesn't seem to be needed anymore.
2008-11-20 15:08:34 +00:00
Eelco Dolstra 285d26374a * Don't set the prefix to /nix by default, rather use the Autoconf
default of /usr/local.  However, localstatedir and storedir are set
  to /nix/var/nix and /nix/store respectively unless they're
  explicitly overriden.
2008-11-20 14:14:35 +00:00
Eelco Dolstra a55113411f * Nix daemon: reload the configuration file after forking (NIX-100). 2008-11-20 12:25:11 +00:00
Eelco Dolstra c202523e53 2008-11-20 12:01:05 +00:00
Eelco Dolstra aab530e971 * Primop builtins.storePath for declaring a store path as a
dependency.  `storePath /nix/store/bla' gives exactly the same
  result as `toPath /nix/store/bla', except that the former includes
  /nix/store/bla in the dependency context of the string.

  Useful in some generated Nix expressions like nix-push, which now
  finally does the right thing wrt distributed builds.  (Previously
  the path to be packed wasn't an explicit dependency, so it wouldn't
  be copied to the remote machine.)
2008-11-19 23:26:19 +00:00
Eelco Dolstra 60564410ef * Patterns. 2008-11-19 17:50:25 +00:00
Eelco Dolstra 2668a43388 2008-11-19 17:27:52 +00:00
Eelco Dolstra 63ccd72496 * Updated the manual. 2008-11-19 17:00:32 +00:00
Eelco Dolstra e13da525a7 * Files in the info directory starting with "." are temporary files
and don't indicate path validity.
2008-11-19 16:27:07 +00:00
Eelco Dolstra 5d250ad1ea * nix-store --dump-db: be more streamy. 2008-11-19 16:26:34 +00:00
Eelco Dolstra 7509d70f9d * Documented some of the sharing mechanisms. 2008-11-19 15:20:46 +00:00
Eelco Dolstra 2369b122d1 * Install documentation in $(docdir) (i.e. share/doc/nix). 2008-11-19 13:19:09 +00:00
Eelco Dolstra 6c2c771af7 * Removed obsolete option. 2008-11-19 13:18:46 +00:00
Eelco Dolstra 07d3a38726 * Remove references to Berkeley DB, including most of the
troubleshooting section.  W00t.
2008-11-19 11:58:33 +00:00
Eelco Dolstra f5325d292d * Release notes. 2008-11-19 10:59:36 +00:00
Eelco Dolstra fa791116a3 * Get rid of nix-pack-closure / nix-unpack-closure, they're redundant. 2008-11-18 14:43:40 +00:00
Eelco Dolstra 3f4ed681c2 * Prevent zombies. Previous the SIGCHLD handler only reaped one
zombie at a time, so if multiple children died before the handler
  got to run, some of them would not be cleaned up.
2008-11-14 16:50:01 +00:00
Eelco Dolstra 6fedb7aa0f * Restore SIGPIPE to SIG_DFL when running the builder. This prevents
subtle and often hard-to-reproduce bugs where programs in pipes
  either barf with a "Broken pipe" message or not, depending on the
  exact timing conditions.  This particularly happened in GNU M4 (and
  Bison, which uses M4).
2008-11-14 15:46:45 +00:00
Eelco Dolstra a519bb0635 * Some somewhat ad hoc mechanism to allow the build farm to monitor
build progress.
2008-11-12 11:08:27 +00:00
Eelco Dolstra 96598e7b06 * Pass the --no-build-output flag to the daemon. 2008-11-11 15:11:10 +00:00
Eelco Dolstra 4166b11a53 * Add /dev/pts to the default nix.conf. 2008-11-11 14:59:20 +00:00
Eelco Dolstra 2b7c839b4e * Typo. 2008-11-11 14:58:37 +00:00
Eelco Dolstra 709b55ee02 * Put the chroots under /nix/var/nix/chroots to reduce the risk of
disasters involving `rm -rf' on bind mounts.  Will try the
  definitive fix (per-process mounts, apparently possible via the
  CLONE_NEWNS flag in clone()) some other time.
2008-10-29 15:34:48 +00:00
Ludovic Courtès c98ea254dc libstore: Always mount `/dev/pts' individually.
This fixes problems such as Tcl's PTY handling:

  ERROR: The system has no more ptys.  Ask your system administrator to
  create more.
2008-10-16 21:04:32 +00:00
Eelco Dolstra 9d6d50269b * Bug fix for building on some old installations (contributed by Pjotr). 2008-10-16 14:16:03 +00:00
Eelco Dolstra fa61ee70ee * Fix `--from-profile'. 2008-09-18 09:08:54 +00:00
Eelco Dolstra f32fef1b07 * GC option `--max-atime' that specifies an upper limit to the last
accessed time of paths that may be deleted.  Anything more recently
  used won't be deleted.  The time is specified in time_t,
  e.g. seconds since 1970-01-01 00:00:00 UTC; use `date +%s' to
  convert to time_t from the command line. 

  Example: to delete everything that hasn't been used in the last two
  months:

  $ nix-store --gc -v --max-atime $(date +%s -d "2 months ago")
2008-09-17 14:52:35 +00:00
Eelco Dolstra 4af2fdba6d * Typo. 2008-09-17 13:00:55 +00:00
Eelco Dolstra c987061aa4 * Some refactoring. Better output with `-v' for --use-atime. 2008-09-17 12:54:07 +00:00
Eelco Dolstra 77afd97a99 * nix-store --gc / --delete: show how many store paths were deleted. 2008-09-17 12:53:33 +00:00
Eelco Dolstra 7ab68961e4 * Garbage collector: added an option `--use-atime' to delete paths in
order of ascending last access time.  This is useful in conjunction
  with --max-freed or --max-links to prefer deleting non-recently used
  garbage, which is good (especially in the build farm) since garbage
  may become live again.

  The code could easily be modified to accept other criteria for
  ordering garbage by changing the comparison operator used by the
  priority queue in collectGarbage().
2008-09-17 10:02:55 +00:00
Eelco Dolstra 2b2aa8a820 * Doh. 2008-09-08 11:02:15 +00:00
Eelco Dolstra 7933cdc6dc * When writing the user environment manifest, filter out non-string
attributes from the meta attribute.  Not doing so caused nix-env to
  barf on the "psi" package, which has a meta.function attribute,
  the textual serialisation of which causes a gigantic string to be
  produced --- so big that it causes nix-env to run out of memory.

  Note however that "meta" really only should contain strings.  
  meta.function should be passthru.function.
2008-09-02 09:21:38 +00:00
Eelco Dolstra 0f0dbe8c0c * Extend the ATerm suppressions to 64-bit. 2008-08-29 14:38:04 +00:00
Eelco Dolstra 311c222f47 2008-08-29 13:59:03 +00:00
Michael Raskin 2a01d06da6 Added nix-http-export.cgi to extra distributed scripts; so it can be installed from releases, not only from SVN. 2008-08-29 08:34:38 +00:00
Eelco Dolstra 7718b19389 * Explicitly set PWD to prevent problems with chroot builds. In
particular, dietlibc cannot figure out the cwd because the inode of
  the current directory doesn't appear in .. (because getdents returns
  the inode of the mount point).
2008-08-27 17:20:25 +00:00
Eelco Dolstra 9cc0da8453 * Create a /tmp with 1777 permission in the chroot. Some builders
need a writable /tmp (they don't respect $TMPDIR).
2008-08-27 16:03:03 +00:00
Eelco Dolstra 99dc3e613a * Require that __overrides is defined as a non-recursive attribute
(which means it can only be defined via "inherit"), otherwise we get
  scoping bugs, since __overrides can't be recursive (or at least, it
  would be hard).
2008-08-26 14:05:59 +00:00
Eelco Dolstra d06be428f6 * Disable chroot builds for fixed-output derivations so that we don't
need /etc in the chroot (in particular, /etc/resolv.conf for
  fetchurl).  Not having /etc/resolv.conf in the chroot is a good
  thing, since we don't want normal derivations to download files.
2008-08-25 15:49:22 +00:00
Eelco Dolstra abec1c0004 * Evaluate attributes in sorted order for better determinism. 2008-08-25 14:31:29 +00:00
Eelco Dolstra c4f1c2114b * Minor simplification. 2008-08-25 14:15:56 +00:00
Eelco Dolstra 49829da8b4 * Doh. 2008-08-25 13:32:27 +00:00
Eelco Dolstra b428adc267 * Strip off the `.nix' suffix from the attribute name for files in
~/.nix-defexpr,  otherwise the attribute cannot be selected with the
  `-A' option.  Useful if you want to stick a Nix expression directly
  in ~/.nix-defexpr.
2008-08-25 13:31:57 +00:00
Eelco Dolstra cc826dc03e * Simplify the Valgrind suppressions using wildcards. 2008-08-22 14:32:29 +00:00
Eelco Dolstra 51e2dda58c * Some more ATerm Valgrind suppressions. 2008-08-19 12:46:43 +00:00
Michael Raskin b7ff182b6e Fixing an obvious typo in override code. I do not know whether it works correctly after the change, but at least it ca nbe compiled now. 2008-08-14 22:01:43 +00:00
Eelco Dolstra ca07f3e370 * Another experimental feature: a way to truly override attributes in
a rec.  This will be very useful to allow end-user customisation of
  all-packages.nix, for instance globally overriding GCC or some other
  dependency.  The // operator doesn't cut it: you could replace the
  "gcc" attribute, but all other attributes would continue to
  reference the original value due to the substitution semantics of
  rec.

  The syntax is a bit hacky but this is to allow backwards
  compatibility.
2008-08-14 16:59:37 +00:00
Eelco Dolstra 9279174dde * Added an experimental feature suggested by Andres: ellipses ("...")
in attribute set pattern matches.  This allows defining a function
  that takes *at least* the listed attributes, while ignoring
  additional attributes.  For instance,

    {stdenv, fetchurl, fuse, ...}:
    
    stdenv.mkDerivation {
      ...
    };
    
  defines a function that requires an attribute set that contains the 
  specified attributes but ignores others.  The main advantage is that
  we can then write in all-packages.nix

    aefs = import ../bla/aefs pkgs;

  instead of

    aefs = import ../bla/aefs {
      inherit stdenv fetchurl fuse;
    };

  This saves a lot of typing (not to mention not having to update
  all-packages.nix with purely mechanical changes).  It saves as much
  typing as the "args: with args;" style, but has the advantage that
  the function arguments are properly declared (not implicit in what
  the body of the "with" uses).
2008-08-14 14:00:44 +00:00
Eelco Dolstra db4f4a8425 * Backward compatibility check to prevent nixos-rebuild from barfing
when upgrading Nix.
2008-08-14 13:02:19 +00:00
Eelco Dolstra 1b962fc720 * @-patterns as in Haskell. For instance, in a function definition
f = args @ {x, y, z}: ...;

  `args' refers to the argument as a whole, which is further
  pattern-matched against the attribute set pattern {x, y, z}.
2008-08-14 12:53:29 +00:00
Eelco Dolstra e818838412 * "pattern" non-terminal. 2008-08-14 10:14:34 +00:00
Eelco Dolstra efe4b690ae * Refactoring: combine functions that take an attribute set and
functions that take a single argument (plain lambdas) into one AST
  node (Function) that contains a Pattern node describing the
  arguments.  Current patterns are single lazy arguments (VarPat) and
  matching against an attribute set (AttrsPat).

  This refactoring allows other kinds of patterns to be added easily,
  such as Haskell-style @-patterns, or list pattern matching.
2008-08-14 10:04:22 +00:00
Eelco Dolstra c03b729319 * Increase the sleep periods a bit to make the test less likely to
fail on slow machines.  Of course it would be better if this test
  wasn't timing dependent...
2008-08-14 09:26:30 +00:00
Eelco Dolstra 5664b6d7ba * Removed the "valid values" feature. Nobody uses it anyway. 2008-08-11 13:36:40 +00:00
Michael Raskin b455c4c45c Updates to nix-reduce-build
Common code in local build package sources refactored out in a function; before building the real set of derivations needed is found (slightly slower for only one build strategy, but less garbage on output and better performance for multiple build strategies).

Now you have full choice of best-effort build regardless of method (substituters or actual build), using substituters, building only fixed derivations (should get you all the downloads) and local build without even trying substituters. 

Some minor fix in the help text about behavior with no package sources.
2008-08-06 19:43:53 +00:00
Eelco Dolstra 72f3ea7358 * Moved some stuff to the Nixpkgs manual.
* Updated the release notes.
2008-08-05 11:03:05 +00:00
Eelco Dolstra 98b07466fb * Better error checking of the data from the substituters. 2008-08-05 10:57:53 +00:00
Eelco Dolstra 339c142009 * Use optimistic profile locking for nix-env operations like `-i' and
`-u'.  Instead of acquiring an exclusive lock on the profile for the
  entire duration of the operation, we just perform the operation
  optimistically (without an exclusive lock), and check at the end
  whether the profile changed while we were busy (i.e., the symlink
  target changed).  If so, the operation is restarted.  Restarting is
  generally cheap, since the build results are still in the Nix store.
  Most of the time, only the user environment has to be rebuilt.
2008-08-04 16:21:45 +00:00
Eelco Dolstra a87b5256e2 * Fix the tests. 2008-08-04 16:16:49 +00:00
Eelco Dolstra 001b3f06ec * `nix-env --set': support --dry-run. 2008-08-04 14:58:50 +00:00
Eelco Dolstra 7592f48c83 * nix-build: `--dry-run' flag. 2008-08-04 13:46:01 +00:00
Eelco Dolstra a1d310b6b5 * `nix-store --realise': print what paths will be built/downloaded,
just like nix-env.
* `nix-store --realise': --dry-run option.
2008-08-04 13:44:46 +00:00
Eelco Dolstra 42043953c3 * Doh. 2008-08-04 13:15:47 +00:00
Eelco Dolstra 5adbb0aabe * build.cc: only use a substituter if it returns info for a path. 2008-08-04 13:15:35 +00:00
Eelco Dolstra 5b1052663a * Always show what paths we're going to build/download (as in
--dry-run).  Maybe there should be an option to turn this on/off?
2008-08-04 13:11:09 +00:00
Eelco Dolstra c4f98941ed * nix-env --dry-run: show the total size of the substituter
downloads.
2008-08-04 12:29:04 +00:00
Eelco Dolstra 03427e76f1 * querySubstitutablePathInfo: work properly when run via the daemon.
* --dry-run: print the paths that we don't know how to build/substitute.
2008-08-04 11:44:50 +00:00
Michael Raskin b3c26180e3 Updates to nix-reduce-build: only realize fixed derivations if user asks so, or only use substituters. Oh, and add possibility to use : for things like /etc/nixos/nixpkgs:-A:gnused 2008-08-02 16:43:25 +00:00
Eelco Dolstra 3c92ea399d * Make nix-env --dry-run print the paths to be substituted correctly
again.  (After the previous substituter mechanism refactoring I
  didn't update the code that obtains the references of substitutable
  paths.)  This required some refactoring: the substituter programs
  are now kept running and receive/respond to info requests via
  stdin/stdout.
2008-08-02 12:54:35 +00:00
Eelco Dolstra fc691e1cbd * Print a better error message when a non-derivation attribute set is
coerced to a string.
2008-07-24 14:52:25 +00:00
Eelco Dolstra 096198d11f * A quick hack to make nix-prefetch-url support mirror:// URLs. It
requires that $NIXPKGS_ALL points at a Nixpkgs tree.
2008-07-23 16:02:58 +00:00
Eelco Dolstra 660244f65f * Make sure that copy-from-other-stores.pl is built. 2008-07-23 09:38:38 +00:00
Eelco Dolstra e139d7fc68 * Fix the tests. 2008-07-18 20:03:12 +00:00
Eelco Dolstra 989176c56e * Allow read-only access to the store (e.g., non-root users on NixOS
can do operations like "nix-store -qR <path>" even without the Nix
  daemon).
2008-07-18 15:34:46 +00:00
Eelco Dolstra 8bc591a6f0 * Use the copy-from-other-stores substituter by default. Of course,
it only does something if $NIX_OTHER_STORES (not really a good
  name...) is set.
* Do globbing on the elements of $NIX_OTHER_STORES.  E.g. you could
  set it to /mnts/*/nix or something.
* Install substituters in libexec/nix/substituters.
2008-07-18 13:05:10 +00:00
Eelco Dolstra 15f39aba8c * Quick prototype of a substituter that copies paths from other Nix
stores (typically remote Nix stores mounted via e.g. NFS, or the Nix
  store on the NixOS installation CD).  Example use:

  $ sshfs foo@example.org:/ /mnt
  $ NIX_OTHER_STORES=/mnt/nix \
    NIX_SUBSTITUTERS=.../copy-from-other-stores.pl \
    nix-env -i foo

  This will be especially useful for the installation CD since it
  doesn't require a manifest for the CD contents.
2008-07-12 18:58:24 +00:00
Eelco Dolstra 7cd88b1dec * Generalised the dependencyClosure primop to builtins.genericClosure,
which is hopefully more useful.
* New primops: length, mul, div.
2008-07-11 13:29:04 +00:00
Eelco Dolstra d567baabbd * Export the nix-env derivation name parsing and version comparison
logic through the `parseDrvName' and `compareVersions' primops.
  This will allow expressions to easily check whether some dependency
  is a specific needed version or falls in some version range.  See
  tests/lang/eval-okay-versions.nix for examples.
2008-07-01 10:10:32 +00:00
Eelco Dolstra b3b0b2a29e * `make ext-clean': remove the bzip2 build. 2008-06-23 13:52:28 +00:00
Eelco Dolstra 6c8641a542 2008-06-18 19:17:05 +00:00
Eelco Dolstra 5af84139a8 * --max-freed: support values >= 4 GB. 2008-06-18 15:20:33 +00:00
Eelco Dolstra d3aa183beb * Garbage collector: option `--max-freed' to stop after at least N
bytes have been freed, `--max-links' to stop when the Nix store
  directory has fewer than N hard links (the latter being important
  for very large Nix stores on filesystems with a 32000 subdirectories
  limit).
2008-06-18 14:20:16 +00:00
Eelco Dolstra a8f3b02092 * `nix-store --optimise': handle files with >= 32000 hard links.
(There can easily be more than 32000 occurrences of the empty file.)
2008-06-18 14:13:00 +00:00
Eelco Dolstra a72709afd8 * Some refactoring: put the GC options / results in separate structs.
* The garbage collector now also prints the number of blocks freed.
2008-06-18 09:34:17 +00:00
Eelco Dolstra 934c58aa38 * Use bzip2 1.0.5. 2008-06-17 08:12:12 +00:00
Eelco Dolstra ee8f15930d * Test instrumentation. 2008-06-15 15:10:03 +00:00
Eelco Dolstra f351834f77 * nix-worker: clean up the temporary root for the worker processes
in /nix/var/nix/temproots.
2008-06-14 16:03:02 +00:00
Eelco Dolstra 94fd46fa1c * Note. 2008-06-14 16:02:31 +00:00
Michael Raskin 955b8841cd Also trying to build derivers in case we cannot get substituters 2008-06-14 08:48:40 +00:00
Michael Raskin 18e27629d3 Added local best-effort builds (i.e. one failure does not ruin all packages you would like to see built) 2008-06-14 08:30:35 +00:00
Eelco Dolstra 826b271d9a * Garbage collector: don't do a complete topological sort of the Nix
store under the reference relation, since that means that the
  garbage collector will need a long time to start deleting paths.
  Instead just delete the referrers of a path first.
2008-06-13 18:25:24 +00:00
Eelco Dolstra 30c9f909b2 * Print some progress info during the early GC stages. 2008-06-13 17:21:20 +00:00
Michael Raskin 194c66eeeb Stupid error in script 2008-06-13 14:34:19 +00:00
Michael Raskin f903d86740 OK, I will believe that fix does no worse.. 2008-06-13 13:53:14 +00:00
Michael Raskin ce85b55cf0 Updated help text 2008-06-12 17:45:38 +00:00
Michael Raskin 4532e4b90d Added verbosity for nix-reduce-build 2008-06-12 16:26:53 +00:00
Eelco Dolstra 2818b7cee6 * Updated some URLs. 2008-06-11 15:39:38 +00:00
Eelco Dolstra 997b95a4af * Fixed compatibility with old versions of "wc" that print whitespace
before the count.
2008-06-10 10:08:15 +00:00
Eelco Dolstra b0e92f6d47 * Merged the no-bdb branch (-r10900:HEAD
https://svn.nixos.org/repos/nix/nix/branches/no-bdb).
2008-06-09 13:52:45 +00:00
Eelco Dolstra 4ed01ed791 * Updated some URLs (did this a long time ago but forgot to
commit...).
2008-06-09 13:42:13 +00:00
Sander van der Burg c41a3ec3a9 First attempt to update Nix SDF grammar to match the actual bison grammar 2008-06-04 14:36:46 +00:00
Eelco Dolstra bd955e15e1 * GCC 4.3.0 (Fedora 9) compatibility fixes. Reported by Gour and
Armijn Hemel.
2008-05-21 11:17:31 +00:00
Michael Raskin 9819bb20da Added support for file:// archive (.nar.gz) repositories to nix-reduce-build. /tmp/nix-export created by nix-http-export.cgi is OK. 2008-05-11 15:54:30 +00:00
Michael Raskin b4bc8b7616 --proxy=proxy:3128 2008-05-07 14:18:28 +00:00
Michael Raskin b1e321d6ce Added http alternative transport for nix-reduce-build 2008-04-29 04:03:54 +00:00
Eelco Dolstra 658816ddc9 * Make really sure that we use bash. The line
NEED_PROG(shell, bash)

  actually uses the content of $shell if set, which often points at
  /bin/sh.
2008-04-10 09:54:23 +00:00
Eelco Dolstra 72034ab35d * sockaddr_un doesn't allow path names of more than 108 characters.
This isn't usually a problem, except that it causes tests to fail
  when performed in a directory with a very long path name.  So chdir
  to the socket directory and use a relative path name.
2008-04-09 05:57:01 +00:00
Eelco Dolstra f8985d195e * Fix for NIX-101 (should use an absolute path for call to nix-hash). 2008-03-28 17:52:33 +00:00
Eelco Dolstra 329025253d * Use /tmp/nix-build-<drvpath>-<counter> instead of
/tmp/nix-<pid>-<counter> for temporary build directories.  This
  increases purity a bit: many packages store the temporary build path
  in their output, causing (generally unimportant) binary differences.
2008-03-27 13:45:17 +00:00
Eelco Dolstra 5bb08db55b * Updated URL. 2008-03-21 14:57:16 +00:00
Eelco Dolstra 98968fbb63 * Disable the don't-run-as-root sanity check because it breaks RPM
builds (which are done as root...).
2008-03-20 18:15:20 +00:00
Eelco Dolstra 2f1e2cf632 * Note that the SDF grammar isn't used. 2008-03-20 14:59:33 +00:00
Eelco Dolstra f106868110 * Cleanup. 2008-03-20 10:16:36 +00:00
Eelco Dolstra f789ea1d09 * A transaction is unnecessary since the path cannot be
garbage-collected (it's a temporary root).
2008-03-01 21:05:33 +00:00
Eelco Dolstra d7caac3e81 * Don't install nix-reduce-build by default yet please, I first want
to understand better what it does...
2008-02-28 21:27:47 +00:00
Michael Raskin 11d512e7a8 Added nix-reduce-build. You point it to some path you want to build and it fetches whatever it can from specified computers via nix-copy-closure. NOTE: You do want to set up RSA keys or ssh-agent or something... You really do want it. It will run separate ssh instances insane number of times. 2008-02-27 21:26:47 +00:00
Eelco Dolstra 0a84137c45 * checkVarDefs: don't check in closed terms, which don't have
undefined variables by definition.  This matters for the
  implementation of "with", which does a call to checkVarDefs to see
  if the body of the with has no undefined variables.  (It can't be
  checked at parse time because you don't know which variables are in
  the "with" attribute set.)  If we check closed terms, then we check
  not just the with body but also the substituted terms, which are
  typically very large.  This is the cause of the poor nix-env
  performance on Nixpkgs lately.  It didn't happen earlier because
  "with" wasn't used very often in the past.

  This fix improves nix-env performance roughly 60x on current Nixpkgs.
  nix-env -qa is down from 29.3s to 0.5s on my laptop, and nix-env -qa
  --out-path is down from 229s to 3.39s.  Not bad for a 1-line fix :-)
2008-02-21 12:01:24 +00:00
Eelco Dolstra 0ed89c569f * Fix the parsing of
''
      '${foo}'
    ''
    
  where the antiquote should work as expected, instead of giving the
  string "'${foo}'".
2008-02-05 13:38:07 +00:00
Eelco Dolstra 4066f450c2 * Doh. 2008-02-05 13:35:49 +00:00
Eelco Dolstra e7bdde981f * Regression test. 2008-02-05 13:25:18 +00:00
Eelco Dolstra 66c51dc215 * nix-store --dump-db / --load-db to dump/load the Nix DB.
* nix-store --register-validity: option to supply the content hash of
  each path.
* Removed compatibility with Nix <= 0.7 stores.
2008-01-29 18:17:36 +00:00
Michael Raskin 5b5a3af983 Probably fixed __exprToString 2008-01-20 20:44:03 +00:00
Michael Raskin 5eb5c23447 Fixed exportBuildReferenceGraph 2008-01-15 04:32:08 +00:00
Eelco Dolstra 7d0f6aed59 * New primop `unsafeDiscardStringContext' to get rid of string
contexts.  Needed to prevent unnecessary dependencies when building
  the NixOS manual.
2008-01-04 14:22:49 +00:00
Eelco Dolstra 895c953817 * Bumped the version number to 0.12. 2007-12-31 17:53:59 +00:00
208 changed files with 7867 additions and 5757 deletions

No files matched your search

+8 -17
View File
@@ -1,23 +1,19 @@
SUBDIRS = externals src scripts corepkgs doc misc tests
EXTRA_DIST = substitute.mk nix.spec nix.spec.in bootstrap.sh \
svn-revision nix.conf.example NEWS
nix.conf.example NEWS version
include ./substitute.mk
nix.spec: nix.spec.in
rpm: nix.spec dist
rpm $(EXTRA_RPM_FLAGS) -ta $(distdir).tar.gz
relname:
echo -n $(distdir) > relname
install-data-local: init-state
$(INSTALL) -d $(DESTDIR)$(sysconfdir)/nix
$(INSTALL_DATA) $(srcdir)/nix.conf.example $(DESTDIR)$(sysconfdir)/nix
if ! test -e $(DESTDIR)$(sysconfdir)/nix/nix.conf; then \
$(INSTALL_DATA) $(srcdir)/nix.conf.example $(DESTDIR)$(sysconfdir)/nix/nix.conf; \
fi
$(INSTALL) -d $(DESTDIR)$(docdir)
$(INSTALL_DATA) README $(DESTDIR)$(docdir)/
if INIT_STATE
@@ -33,23 +29,18 @@ init-state:
$(INSTALL) $(INIT_FLAGS) -d $(DESTDIR)$(localstatedir)/nix/profiles
$(INSTALL) $(INIT_FLAGS) -d $(DESTDIR)$(localstatedir)/nix/gcroots
$(INSTALL) $(INIT_FLAGS) -d $(DESTDIR)$(localstatedir)/nix/temproots
$(INSTALL) $(INIT_FLAGS) $(GROUP_WRITABLE) -d $(DESTDIR)$(localstatedir)/nix/gcroots/tmp
$(INSTALL) $(INIT_FLAGS) $(GROUP_WRITABLE) -d $(DESTDIR)$(localstatedir)/nix/gcroots/channels
ln -sfn $(localstatedir)/nix/profiles $(DESTDIR)$(localstatedir)/nix/gcroots/profiles
$(INSTALL) $(INIT_FLAGS) -d $(DESTDIR)$(localstatedir)/nix/userpool
$(INSTALL) $(INIT_FLAGS) -m 1777 -d $(DESTDIR)$(prefix)/store
-$(INSTALL) $(INIT_FLAGS) -m 1777 -d $(DESTDIR)$(storedir)
$(INSTALL) $(INIT_FLAGS) $(GROUP_WRITABLE) -d $(DESTDIR)$(localstatedir)/nix/manifests
ln -sfn $(localstatedir)/nix/manifests $(DESTDIR)$(localstatedir)/nix/gcroots/manifests
# $(bindir)/nix-store --init
else
init-state:
endif
svn-revision:
svnversion . > svn-revision
all-local: NEWS
NEWS: doc/manual/NEWS.txt
NEWS:
$(MAKE) -C doc/manual NEWS.txt
cp $(srcdir)/doc/manual/NEWS.txt NEWS
+5 -4
View File
@@ -1,9 +1,10 @@
For installation and usage instructions, please read the manual, which
can be found in `docs/manual/manual.html', and additionally at the Nix
website at <http://www.cs.uu.nl/groups/ST/Trace/Nix>.
Nix is a purely functional package manager. For installation and
usage instructions, please read the manual, which can be found in
`docs/manual/manual.html', and additionally at the Nix website at
<http://nixos.org/>.
Acknowledgments
This product includes software developed by the OpenSSL Project for
use in the OpenSSL Toolkit (http://www.OpenSSL.org/)
use in the OpenSSL Toolkit (http://www.OpenSSL.org/).
+84 -151
View File
@@ -1,184 +1,117 @@
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:mark_memory_young
fun:mark_phase_young
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:AT_isValidSymbol
fun:mark_memory_young
fun:mark_phase_young
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:AT_isValidSymbol
fun:mark_memory_young
fun:mark_phase_young
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:mark_memory_young
fun:mark_phase_young
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:AT_isInsideValidTerm
fun:mark_memory_young
fun:mark_phase_young
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:AT_isInsideValidTerm
fun:mark_memory_young
fun:mark_phase_young
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:mark_memory_young
fun:mark_phase_young
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:AT_markTerm_young
fun:mark_memory_young
fun:mark_phase_young
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:AT_markTerm_young
fun:mark_memory_young
fun:mark_phase_young
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:mark_memory
fun:mark_phase
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:mark_memory
fun:mark_phase
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:AT_isValidSymbol
fun:mark_memory
fun:mark_phase
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:AT_isValidSymbol
fun:mark_memory
fun:mark_phase
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:AT_isInsideValidTerm
fun:mark_memory
fun:mark_phase
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:AT_isInsideValidTerm
fun:mark_memory
fun:mark_phase
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:AT_markTerm
fun:mark_memory
fun:mark_phase
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:AT_markTerm
fun:mark_memory
fun:mark_phase
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:AT_markTerm
fun:mark_memory
fun:mark_phase
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:mark_phase_young
fun:*
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:mark_phase_young
fun:*
fun:AT_collect_minor
}
{
<insert a suppression name here>
Memcheck:Cond
fun:AT_isValidSymbol
fun:mark_phase_young
ATerm library conservatively scans for GC roots
Memcheck:Value8
fun:*
fun:AT_collect_minor
}
{
<insert a suppression name here>
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:AT_isValidSymbol
fun:mark_phase_young
fun:*
fun:*
fun:AT_collect_minor
}
{
<insert a suppression name here>
Memcheck:Value4
fun:AT_isInsideValidTerm
fun:mark_phase_young
ATerm library conservatively scans for GC roots
Memcheck:Value8
fun:*
fun:*
fun:AT_collect_minor
}
{
<insert a suppression name here>
ATerm library conservatively scans for GC roots
Memcheck:Addr4
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Addr8
fun:*
fun:AT_collect_minor
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:AT_isInsideValidTerm
fun:mark_phase_young
fun:AT_collect_minor
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value8
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Addr4
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Addr8
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value4
fun:*
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Value8
fun:*
fun:*
fun:AT_collect
}
{
ATerm library conservatively scans for GC roots
Memcheck:Cond
fun:*
fun:*
fun:AT_collect
}
+60 -48
View File
@@ -1,24 +1,9 @@
AC_INIT(nix, 0.11)
AC_INIT(nix, m4_esyscmd([echo -n $(cat ./version)$VERSION_SUFFIX]))
AC_CONFIG_SRCDIR(README)
AC_CONFIG_AUX_DIR(config)
AM_INIT_AUTOMAKE([dist-bzip2 foreign])
# Change to `1' to produce a `stable' release (i.e., the `preREVISION'
# suffix is not added).
STABLE=1
# Put the revision number in the version.
if test "$STABLE" != "1"; then
if REVISION=`test -d $srcdir/.svn && svnversion -n $srcdir 2> /dev/null`; then
VERSION=${VERSION}pre${REVISION}
elif REVISION=`cat $srcdir/svn-revision 2> /dev/null`; then
VERSION=${VERSION}pre${REVISION}
fi
fi
AC_DEFINE_UNQUOTED(NIX_VERSION, ["$VERSION"], [version])
AC_PREFIX_DEFAULT(/nix)
AC_DEFINE_UNQUOTED(NIX_VERSION, ["$VERSION"], [Nix version.])
AC_CANONICAL_HOST
@@ -54,24 +39,50 @@ case $sys_name in
esac
AC_ARG_WITH(system, AC_HELP_STRING([--with-system=SYSTEM],
[platform identifier (e.g., `i686-linux')]),
[Platform identifier (e.g., `i686-linux').]),
system=$withval, system="${machine_name}-${sys_name}")
AC_MSG_RESULT($system)
AC_SUBST(system)
AC_DEFINE_UNQUOTED(SYSTEM, ["$system"], [platform identifier (`cpu-os')])
# Windows-specific stuff.
if test "$sys_name" = "cygwin"; then
# We cannot delete open files.
AC_DEFINE(CANNOT_DELETE_OPEN_FILES, 1, [Whether it is impossible to delete open files.])
# State should be stored in /nix/var, unless the user overrides it explicitly.
test "$localstatedir" = '${prefix}/var' && localstatedir=/nix/var
# Shared libraries don't work, currently.
# Whether to produce a statically linked binary. On Cygwin, this is
# the default: dynamically linking against the ATerm DLL does work,
# except that it requires the ATerm "lib" directory to be in $PATH, as
# Windows doesn't have anything like an RPATH embedded in executable.
# Since this is kind of annoying, we use static libraries for now.
AC_ARG_ENABLE(static-nix, AC_HELP_STRING([--enable-static-nix],
[produce statically linked binaries]),
static_nix=$enableval, static_nix=no)
if test "$sys_name" = cygwin; then
static_nix=yes
fi
if test "$static_nix" = yes; then
AC_DISABLE_SHARED
AC_ENABLE_STATIC
fi
# Windows-specific stuff.
if test "$sys_name" = "cygwin"; then
# We cannot delete open files.
AC_DEFINE(CANNOT_DELETE_OPEN_FILES, 1, [Whether it is impossible to delete open files.])
fi
# Solaris-specific stuff.
if test "$sys_name" = "sunos"; then
# Solaris requires -lsocket -lnsl for network functions
ADDITIONAL_NETWORK_LIBS="-lsocket -lnsl"
AC_SUBST(ADDITIONAL_NETWORK_LIBS)
fi
AC_PROG_CC
AC_PROG_CXX
@@ -83,8 +94,7 @@ AC_PROG_LIBTOOL
# Use 64-bit file system calls so that we can support files > 2 GiB.
CFLAGS="-D_FILE_OFFSET_BITS=64 $CFLAGS"
CXXFLAGS="-D_FILE_OFFSET_BITS=64 $CXXFLAGS"
AC_SYS_LARGEFILE
# Check for pubsetbuf.
@@ -94,13 +104,15 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <iostream>
using namespace std;
static char buf[1024];]],
[[cerr.rdbuf()->pubsetbuf(buf, sizeof(buf));]])],
[AC_MSG_RESULT(yes) AC_DEFINE(HAVE_PUBSETBUF, 1, [whether pubsetbuf is available])],
[AC_MSG_RESULT(yes) AC_DEFINE(HAVE_PUBSETBUF, 1, [Whether pubsetbuf is available.])],
AC_MSG_RESULT(no))
AC_LANG_POP(C++)
# Check for chroot support (requires chroot() and bind mounts).
AC_CHECK_FUNCS([chroot])
AC_CHECK_FUNCS([unshare])
AC_CHECK_HEADERS([sched.h], [], [], [])
AC_CHECK_HEADERS([sys/param.h], [], [], [])
AC_CHECK_HEADERS([sys/mount.h], [], [],
[#ifdef HAVE_SYS_PARAM_H
@@ -109,12 +121,17 @@ AC_CHECK_HEADERS([sys/mount.h], [], [],
])
# Check for <locale>
# Check for <locale>.
AC_LANG_PUSH(C++)
AC_CHECK_HEADERS([locale], [], [], [])
AC_LANG_POP(C++)
# Check whether we have the personality() syscall, which allows us to
# do i686-linux builds on x86_64-linux machines.
AC_CHECK_HEADERS([sys/personality.h])
AC_DEFUN([NEED_PROG],
[
AC_PATH_PROG($1, $2)
@@ -124,11 +141,10 @@ fi
])
NEED_PROG(curl, curl)
NEED_PROG(shell, bash)
NEED_PROG(bash, bash)
NEED_PROG(patch, patch)
AC_PATH_PROG(xmllint, xmllint, false)
AC_PATH_PROG(xsltproc, xsltproc, false)
AC_PATH_PROG(jing, jing, false) # needed because xmllint --relaxng seems broken
AC_PATH_PROG(w3m, w3m, false)
AC_PATH_PROG(flex, flex, false)
AC_PATH_PROG(bison, bison, false)
@@ -136,6 +152,7 @@ NEED_PROG(perl, perl)
NEED_PROG(tar, tar)
AC_PATH_PROG(dot, dot)
AC_PATH_PROG(dblatex, dblatex)
AC_PATH_PROG(gzip, gzip)
AC_PATH_PROG(openssl_prog, openssl, openssl) # if not found, call openssl in $PATH
AC_SUBST(openssl_prog)
@@ -173,23 +190,9 @@ AC_SUBST(xmlflags)
AC_ARG_WITH(store-dir, AC_HELP_STRING([--with-store-dir=PATH],
[path of the Nix store]),
storedir=$withval, storedir='${prefix}/store')
storedir=$withval, storedir='/nix/store')
AC_SUBST(storedir)
AC_ARG_WITH(bdb, AC_HELP_STRING([--with-bdb=PATH],
[prefix of Berkeley DB]),
bdb=$withval, bdb=)
AM_CONDITIONAL(HAVE_BDB, test -n "$bdb")
if test -z "$bdb"; then
bdb_lib='-L${top_builddir}/externals/inst-bdb/lib -ldb_cxx'
bdb_include='-I${top_builddir}/externals/inst-bdb/include'
else
bdb_lib="-L$bdb/lib -ldb_cxx"
bdb_include="-I$bdb/include"
fi
AC_SUBST(bdb_lib)
AC_SUBST(bdb_include)
AC_ARG_WITH(aterm, AC_HELP_STRING([--with-aterm=PATH],
[prefix of CWI ATerm library]),
aterm=$withval, aterm=)
@@ -215,7 +218,7 @@ if test -n "$openssl"; then
LDFLAGS="-L$openssl/lib -lcrypto $LDFLAGS"
CFLAGS="-I$openssl/include $CFLAGS"
CXXFLAGS="-I$openssl/include $CXXFLAGS"
AC_DEFINE(HAVE_OPENSSL, 1, [whether to use OpenSSL])
AC_DEFINE(HAVE_OPENSSL, 1, [Whether to use OpenSSL.])
fi
AC_ARG_WITH(bzip2, AC_HELP_STRING([--with-bzip2=PATH],
@@ -228,7 +231,7 @@ if test -z "$bzip2"; then
bzip2_lib='-L${top_builddir}/externals/inst-bzip2/lib -lbz2'
bzip2_include='-I${top_builddir}/externals/inst-bzip2/include'
# The binary will be copied to $libexecdir.
bzip2_bin='${libexecdir}'
bzip2_bin='${libexecdir}/nix'
# But for testing, we have to use the temporary copy :-(
bzip2_bin_test='${top_builddir}/externals/inst-bzip2/bin'
else
@@ -236,7 +239,7 @@ else
bzip2_include="-I$bzip2/include"
bzip2_bin="$bzip2/bin"
bzip2_bin_test="$bzip2/bin"
fi
fi
AC_SUBST(bzip2_lib)
AC_SUBST(bzip2_include)
AC_SUBST(bzip2_bin)
@@ -258,15 +261,24 @@ AC_CHECK_FUNCS([setresuid setreuid lchown])
# Nice to have, but not essential.
AC_CHECK_FUNCS([strsignal])
AC_CHECK_FUNCS([posix_fallocate])
# This is needed if ATerm, Berkeley DB or bzip2 are static libraries,
# This is needed if ATerm or bzip2 are static libraries,
# and the Nix libraries are dynamic.
if test "$(uname)" = "Darwin"; then
LDFLAGS="-all_load $LDFLAGS"
fi
if test "$static_nix" = yes; then
# `-all-static' has to be added at the end of configure, because
# the C compiler doesn't know about -all-static (it's filtered out
# by libtool, but configure doesn't use libtool).
LDFLAGS="-all-static $LDFLAGS"
fi
AM_CONFIG_HEADER([config.h])
AC_CONFIG_FILES([Makefile
externals/Makefile
+6
View File
@@ -8,9 +8,15 @@ inputs=($inputs)
for ((n = 0; n < ${#inputs[*]}; n += 2)); do
channelName=${inputs[n]}
channelTarball=${inputs[n+1]}
echo "unpacking channel $channelName"
@bunzip2@ < $channelTarball | @tar@ xf -
if test -e */channel-name; then
channelName="$(@coreutils@/cat */channel-name)"
fi
nr=1
attrName=$(echo $channelName | @tr@ -- '- ' '__')
dirName=$attrName
+22 -16
View File
@@ -6,7 +6,12 @@ XSLTPROC = $(xsltproc) $(xmlflags) \
--param xref.with.number.and.title 1 \
--param toc.section.depth 3 \
--param admon.style \'\' \
--param callout.graphics.extension \'.gif\'
--param callout.graphics.extension \'.gif\' \
--param contrib.inline.enabled 0
dblatex_opts = \
-P doc.collab.show=0 \
-P latex.output.revhistory=0
# Note: we use GIF for now, since the PNGs shipped with Docbook aren't
# transparent.
@@ -14,27 +19,24 @@ XSLTPROC = $(xsltproc) $(xmlflags) \
man1_MANS = nix-env.1 nix-build.1 nix-store.1 nix-instantiate.1 \
nix-collect-garbage.1 nix-push.1 nix-pull.1 \
nix-prefetch-url.1 nix-channel.1 \
nix-pack-closure.1 nix-unpack-closure.1 \
nix-install-package.1 nix-hash.1 nix-copy-closure.1
man8_MANS = nix-worker.8
FIGURES = figures/user-environments.png
MANUAL_SRCS = manual.xml introduction.xml installation.xml \
package-management.xml writing-nix-expressions.xml builtins.xml \
build-farm.xml \
$(man1_MANS:.1=.xml) \
$(man1_MANS:.1=.xml) $(man8_MANS:.8=.xml) \
troubleshooting.xml bugs.xml opt-common.xml opt-common-syn.xml \
env-common.xml quick-start.xml nix-lang-ref.xml glossary.xml \
conf-file.xml release-notes.xml \
style.css images
# Note: RelaxNG validation requires xmllint >= 2.7.4.
manual.is-valid: $(MANUAL_SRCS) version.txt
# $(XMLLINT) --xinclude $< | $(XMLLINT) --noout --nonet --relaxng $(docbookrng)/docbook.rng -
if test "$(jing)" != "false"; then \
$(XMLLINT) --xinclude $< | $(jing) $(docbookrng)/docbook.rng /dev/fd/0; \
else \
echo "Not validating."; \
fi
$(XMLLINT) --noout --nonet --xinclude --noxincludenode --relaxng $(docbookrng)/docbook.rng $<
touch $@
version.txt:
@@ -49,7 +51,7 @@ manual.html: $(MANUAL_SRCS) manual.is-valid images
manual.pdf: $(MANUAL_SRCS) manual.is-valid images
if test "$(dblatex)" != ""; then \
$(dblatex) manual.xml; \
$(dblatex) $(dblatex_opts) manual.xml; \
else \
echo "Please install dblatex and rerun configure."; \
exit 1; \
@@ -76,12 +78,16 @@ NEWS.txt: release-notes.xml
all-local: manual.html NEWS.html NEWS.txt
install-data-local: manual.html
$(INSTALL) -d $(DESTDIR)$(datadir)/nix/manual
$(INSTALL_DATA) manual.html $(DESTDIR)$(datadir)/nix/manual
$(INSTALL_DATA) style.css $(DESTDIR)$(datadir)/nix/manual
cp -r images $(DESTDIR)$(datadir)/nix/manual/images
$(INSTALL) -d $(DESTDIR)$(datadir)/nix/manual/figures
$(INSTALL_DATA) $(FIGURES) $(DESTDIR)$(datadir)/nix/manual/figures
$(INSTALL) -d $(DESTDIR)$(docdir)/manual
$(INSTALL_DATA) manual.html $(DESTDIR)$(docdir)/manual
ln -sf manual.html $(DESTDIR)$(docdir)/manual/index.html
$(INSTALL_DATA) style.css $(DESTDIR)$(docdir)/manual
cp -r images $(DESTDIR)$(docdir)/manual/images
$(INSTALL) -d $(DESTDIR)$(docdir)/manual/figures
$(INSTALL_DATA) $(FIGURES) $(DESTDIR)$(docdir)/manual/figures
$(INSTALL) -d $(DESTDIR)$(docdir)/release-notes
$(INSTALL_DATA) NEWS.html $(DESTDIR)$(docdir)/release-notes/index.html
$(INSTALL_DATA) style.css $(DESTDIR)$(docdir)/release-notes/
images:
mkdir images
+2 -2
View File
@@ -56,7 +56,7 @@ build farm, since:
<para>TODO</para>
<para>The sources of the Nix build farm are at <link
xlink:href='https://svn.cs.uu.nl:12443/repos/trace/release/trunk'/>.</para>
xlink:href='https://svn.nixos.org/repos/nix/release/trunk'/>.</para>
</section>
@@ -85,7 +85,7 @@ nix@scratchy.labs.cs.uu.nl i686-linux /home/nix/.ssh/id_scratchy_auto
<para>An example build hook can be found in the Nix build farm
sources: <link
xlink:href='https://svn.cs.uu.nl:12443/repos/trace/release/trunk/common/distributed/build-remote.pl'
xlink:href='https://svn.nixos.org/repos/nix/release/trunk/common/distributed/build-remote.pl'
/>. It should be suitable for most purposes, with maybe some minor
adjustments. It uses <command>ssh</command> and
<command>rsync</command> to copy the build inputs and outputs and
+101 -10
View File
@@ -77,18 +77,25 @@ attrValues = attrs: map (name: builtins.getAttr name attrs) (builtins.attrNames
if builtins ? getEnv then builtins.getEnv "PATH" else ""</programlisting>
This allows a Nix expression to fall back gracefully on older Nix
installations that don’t have the desired built-in function.
However, in that case you should not write
installations that don’t have the desired built-in
function.</para></listitem>
<programlisting>
if builtins ? getEnv then __getEnv "PATH" else ""</programlisting>
</varlistentry>
This Nix expression will trigger an “undefined variable” error on
older Nix versions since <function>__getEnv</function> doesn’t
exist. <literal>builtins.getEnv</literal>, on the other hand, is
safe since <literal>builtins</literal> always exists and attribute
selection is lazy, so it’s only performed if the test
succeeds.</para></listitem>
<varlistentry><term><function>builtins.compareVersions</function>
<replaceable>s1</replaceable> <replaceable>s2</replaceable></term>
<listitem><para>Compare two strings representing versions and
return <literal>-1</literal> if version
<replaceable>s1</replaceable> is older than version
<replaceable>s2</replaceable>, <literal>0</literal> if they are
the same, and <literal>1</literal> if
<replaceable>s1</replaceable> is newer than
<replaceable>s2</replaceable>. The version comparison algorithm
is the same as the one used by <link
linkend="ssec-version-comparisons"><command>nix-env
-u</command></link>.</para></listitem>
</varlistentry>
@@ -146,6 +153,16 @@ if builtins ? getEnv then __getEnv "PATH" else ""</programlisting>
</varlistentry>
<varlistentry><term><function>builtins.div</function>
<replaceable>e1</replaceable> <replaceable>e2</replaceable></term>
<listitem><para>Return the quotient of the integers
<replaceable>e1</replaceable> and
<replaceable>e2</replaceable>.</para></listitem>
</varlistentry>
<varlistentry><term><function>builtins.filterSource</function>
<replaceable>e1</replaceable> <replaceable>e2</replaceable></term>
@@ -307,6 +324,16 @@ x: x + 456</programlisting>
</varlistentry>
<varlistentry><term><function>builtins.intersectAttrs</function>
<replaceable>e1</replaceable> <replaceable>e2</replaceable></term>
<listitem><para>Return an attribute set consisting of the
attributes in the set <replaceable>e2</replaceable> that also
exist in the set <replaceable>e1</replaceable>.</para></listitem>
</varlistentry>
<varlistentry><term><function>builtins.isAttrs</function>
<replaceable>e</replaceable></term>
@@ -337,6 +364,36 @@ x: x + 456</programlisting>
</varlistentry>
<varlistentry><term><function>builtins.isString</function>
<replaceable>e</replaceable></term>
<listitem><para>Return <literal>true</literal> if
<replaceable>e</replaceable> evaluates to a string, and
<literal>false</literal> otherwise.</para></listitem>
</varlistentry>
<varlistentry><term><function>builtins.isInt</function>
<replaceable>e</replaceable></term>
<listitem><para>Return <literal>true</literal> if
<replaceable>e</replaceable> evaluates to a int, and
<literal>false</literal> otherwise.</para></listitem>
</varlistentry>
<varlistentry><term><function>builtins.isBool</function>
<replaceable>e</replaceable></term>
<listitem><para>Return <literal>true</literal> if
<replaceable>e</replaceable> evaluates to a bool, and
<literal>false</literal> otherwise.</para></listitem>
</varlistentry>
<varlistentry><term><function>isNull</function>
<replaceable>e</replaceable></term>
@@ -352,6 +409,15 @@ x: x + 456</programlisting>
</varlistentry>
<varlistentry><term><function>builtins.length</function>
<replaceable>e</replaceable></term>
<listitem><para>Return the length of the list
<replaceable>e</replaceable>.</para></listitem>
</varlistentry>
<varlistentry><term><function>builtins.lessThan</function>
<replaceable>e1</replaceable> <replaceable>e2</replaceable></term>
@@ -408,6 +474,31 @@ map (x: "foo" + x) ["bar" "bla" "abc"]</programlisting>
</varlistentry>
<varlistentry><term><function>builtins.mul</function>
<replaceable>e1</replaceable> <replaceable>e2</replaceable></term>
<listitem><para>Return the product of the integers
<replaceable>e1</replaceable> and
<replaceable>e2</replaceable>.</para></listitem>
</varlistentry>
<varlistentry><term><function>builtins.parseDrvName</function>
<replaceable>s</replaceable></term>
<listitem><para>Split the string <replaceable>s</replaceable> into
a package name and version. The package name is everything up to
but not including the first dash followed by a digit, and the
version is everything following that dash. The result is returned
in an attribute set <literal>{name, version}</literal>. Thus,
<literal>builtins.parseDrvName "nix-0.12pre12876"</literal>
returns <literal>{name = "nix"; version =
"0.12pre12876";}</literal>.</para></listitem>
</varlistentry>
<varlistentry><term><function>builtins.pathExists</function>
<replaceable>path</replaceable></term>
+11 -22
View File
@@ -60,27 +60,6 @@ env-keep-derivations = false
</varlistentry>
<varlistentry xml:id="conf-gc-reserved-space"><term><literal>gc-reserved-space</literal></term>
<listitem><para>This option specifies how much space should be
reserved in normal use so that the garbage collector can run
succesfully. Since the garbage collector must perform Berkeley DB
transactions, it needs some disk space for itself. However, when
the disk is full, this space is not available, so the collector
would not be able to run precisely when it is most needed.</para>
<para>For this reason, when Nix is run, it allocates a file
<filename>/nix/var/nix/db/reserved</filename> of the size
specified by this option. When the garbage collector is run, this
file is deleted before the Berkeley DB environment is opened.
This should give it enough room to proceed.</para>
<para>The default is <literal>1048576</literal> (1
MiB).</para></listitem>
</varlistentry>
<varlistentry><term><literal>env-keep-derivations</literal></term>
<listitem><para>If <literal>false</literal> (default), derivations
@@ -254,7 +233,17 @@ build-use-chroot = /dev /proc /bin</programlisting>
<filename>configure</filename> at build time.</para></listitem>
</varlistentry>
<varlistentry><term><literal>fsync-metadata</literal></term>
<listitem><para>If set to <literal>true</literal>, changes to the
Nix store metadata (in <filename>/nix/var/nix/db</filename>) are
synchronously flushed to disk. This improves robustness in case
of system crashes, but reduces performance. The default is
<literal>false</literal>.</para></listitem>
</varlistentry>
</variablelist>
+35 -37
View File
@@ -151,12 +151,12 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
<para>On the basis of this information, and whatever persistent
state the build hook keeps about other machines and their current
load, it has to decide what to do with the build. It should print
out on file descriptor 3 one of the following responses (terminated
by a newline, <literal>"\n"</literal>):
out on standard error one of the following responses (terminated by
a newline, <literal>"\n"</literal>):
<variablelist>
<varlistentry><term><literal>decline</literal></term>
<varlistentry><term><literal># decline</literal></term>
<listitem><para>The build hook is not willing or able to perform
the build; the calling Nix process should do the build itself,
@@ -164,7 +164,7 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
</varlistentry>
<varlistentry><term><literal>postpone</literal></term>
<varlistentry><term><literal># postpone</literal></term>
<listitem><para>The build hook cannot perform the build now, but
can do so in the future (e.g., because all available build slots
@@ -174,7 +174,7 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
</varlistentry>
<varlistentry><term><literal>accept</literal></term>
<varlistentry><term><literal># accept</literal></term>
<listitem><para>The build hook has accepted the
build.</para></listitem>
@@ -185,37 +185,12 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
</para>
<para>If the build hook accepts the build, it is possible that it is
no longer necessary to do the build because some other process has
performed the build in the meantime. To prevent races, the hook
must read from file descriptor 4 a single line that tells it whether
to continue:
<variablelist>
<varlistentry><term><literal>cancel</literal></term>
<listitem><para>The build has already been done, so the hook
should exit.</para></listitem>
</varlistentry>
<varlistentry><term><literal>okay</literal></term>
<listitem><para>The hook should proceed with the build. At this
point, the calling Nix process has acquired locks on the output
path, so no other Nix process will perform the
build.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>If the hook has been told to proceed, Nix will store in the
hook’s current directory a number of text files that contain
information about the derivation:
<para>After sending <literal># accept</literal>, the hook should
read one line from standard input, which will be the string
<literal>okay</literal>. It can then proceed with the build.
Before sending <literal>okay</literal>, Nix will store in the hook’s
current directory a number of text files that contain information
about the derivation:
<variablelist>
@@ -255,7 +230,9 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
<para>The hook should copy the inputs to the remote machine,
register the validity of the inputs, perform the remote build, and
copy the outputs back to the local machine. An exit code other than
<literal>0</literal> indicates that the hook has failed.</para>
<literal>0</literal> indicates that the hook has failed. An exit
code equal to 100 means that the remote build failed (as opposed to,
e.g., a network error).</para>
</listitem>
@@ -274,6 +251,27 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
</varlistentry>
<varlistentry xml:id="envar-other-stores"><term><envar>NIX_OTHER_STORES</envar></term>
<listitem><para>This variable contains the paths of remote Nix
installations from whichs paths can be copied, separated by colons.
See <xref linkend="sec-sharing-packages" /> for details. Each path
should be the <filename>/nix</filename> directory of a remote Nix
installation (i.e., not the <filename>/nix/store</filename>
directory). The paths are subject to globbing, so you can set it so
something like <literal>/var/run/nix/remote-stores/*/nix</literal>
and mount multiple remote filesystems in
<literal>/var/run/nix/remote-stores</literal>.</para>
<para>Note that if you’re building through the <link
linkend="sec-nix-worker">Nix daemon</link>, the only setting for
this variable that matters is the one that the
<command>nix-worker</command> process uses. So if you want to
change it, you have to restart the daemon.</para></listitem>
</varlistentry>
</variablelist>
+48 -43
View File
@@ -42,25 +42,22 @@ platforms as well.</para>
<section><title>Obtaining Nix</title>
<para>The easiest way to obtain Nix is to download a <link
xlink:href="http://nix.cs.uu.nl/">source distribution</link>. RPMs
xlink:href="http://nixos.org/">source distribution</link>. RPMs
for Red Hat, SuSE, and Fedora Core are also available.</para>
<para>Alternatively, the most recent sources of Nix can be obtained
from its <link
xlink:href="https://svn.cs.uu.nl:12443/repos/trace/nix/trunk">Subversion
xlink:href="https://svn.nixos.org/repos/nix/nix/trunk">Subversion
repository</link>. For example, the following command will check out
the latest revision into a directory called
<filename>nix</filename>:</para>
<screen>
$ svn checkout https://svn.cs.uu.nl:12443/repos/trace/nix/trunk nix</screen>
$ svn checkout https://svn.nixos.org/repos/nix/nix/trunk nix</screen>
<para>Likewise, specific releases can be obtained from the <link
xlink:href="https://svn.cs.uu.nl:12443/repos/trace/nix/tags">tags
directory</link> of the repository. If you don't have Subversion, you
can also download an automatically generated <link
xlink:href="https://svn.cs.uu.nl:12443/dist/trace/">compressed
tar-file</link> of the head revision of the trunk.</para>
xlink:href="https://svn.nixos.org/repos/nix/nix/tags">tags
directory</link> of the repository.</para>
</section>
@@ -99,18 +96,16 @@ ubiquitous 2.5.4a won't. Note that these are only required if you
modify the parser or when you are building from the Subversion
repository.</para>
<para>Nix uses Sleepycat's Berkeley DB, CWI's ATerm library and the
bzip2 compressor (including the bzip2 library). These are included in
the Nix source distribution. If you build from the Subversion
repository, you must download them yourself and place them in the
<para>Nix uses CWI's ATerm library and the bzip2 compressor (including
the bzip2 library). These are included in the Nix source
distribution. If you build from the Subversion repository, you must
download them yourself and place them in the
<filename>externals/</filename> directory. See
<filename>externals/Makefile.am</filename> for the precise URLs of
these packages. Alternatively, if you already have them installed,
you can use <command>configure</command>'s
<option>--with-bdb</option>, <option>--with-aterm</option> and
<option>--with-bzip2</option> options to point to their respective
locations. Note that Berkeley DB <emphasis>must</emphasis> be version
4.5; other versions may not have compatible database formats.</para>
<option>--with-aterm</option> and <option>--with-bzip2</option>
options to point to their respective locations.</para>
</section>
@@ -138,16 +133,25 @@ $ ./bootstrap</screen>
<para>The installation path can be specified by passing the
<option>--prefix=<replaceable>prefix</replaceable></option> to
<command>configure</command>. The default installation directory is
<filename>/nix</filename>. You can change this to any location you
like. You must have write permission to the
<filename>/usr/local</filename>. You can change this to any location
you like. You must have write permission to the
<replaceable>prefix</replaceable> path.</para>
<warning><para>It is best <emphasis>not</emphasis> to change the
installation prefix from its default, since doing so makes it
impossible to use pre-built binaries from the standard Nixpkgs
channels.</para></warning>
<para>Nix keeps its <emphasis>store</emphasis> (the place where
packages are stored) in <filename>/nix/store</filename> by default.
This can be changed using
<option>--with-store-dir=<replaceable>path</replaceable></option>.</para>
<para>If you want to rebuilt the documentation, pass the full path to
<warning><para>It is best <emphasis>not</emphasis> to change the Nix
store from its default, since doing so makes it impossible to use
pre-built binaries from the standard Nixpkgs channels — that is, all
packages will need to be built from source.</para></warning>
<para>Nix keeps state (such as its database and log files) in
<filename>/nix/var</filename> by default. This can be changed using
<option>--localstatedir=<replaceable>path</replaceable></option>.</para>
<para>If you want to rebuild the documentation, pass the full path to
the DocBook RELAX NG schemas and to the DocBook XSL stylesheets using
the
<option>--with-docbook-rng=<replaceable>path</replaceable></option>
@@ -158,27 +162,26 @@ options.</para>
</section>
<section><title>Installing from RPMs</title>
<section><title>Installing a binary distribution</title>
<para>RPM packages of Nix can be downloaded from <link
xlink:href="http://nix.cs.uu.nl/" />. These RPMs should work for most
fairly recent releases of SuSE and Red Hat Linux. They have been
known to work work on SuSE Linux 8.1 and 9.0, and Red Hat 9.0. In
fact, it should work on any RPM-based Linux distribution based on
<literal>glibc</literal> 2.3 or later.</para>
<para>Once downloaded, the RPMs can be installed or upgraded using
<command>rpm -U</command>. For example,
<para>RPM and Deb packages of Nix for a number of different versions
of Fedora, openSUSE, Debian and Ubuntu can be downloaded from <link
xlink:href="http://nixos.org/" />. Once downloaded, the RPMs can be
installed or upgraded using <command>rpm -U</command>. For example,
<screen>
$ rpm -U nix-0.5pre664-1.i386.rpm</screen>
$ rpm -U nix-0.13pre18104-1.i386.rpm</screen>
Likewise, for a Deb package:
<screen>
$ dpkg -i nix_0.13pre18104-1_amd64.deb</screen>
</para>
<para>The RPMs install into the directory <filename>/nix</filename>.
Nix can be uninstalled using <command>rpm -e nix</command>. After
this it will be necessary to manually remove the Nix store and other
auxiliary data:
<para>Nix can be uninstalled using <command>rpm -e nix</command> or
<command>dpkg -r nix</command>. After this you should manually remove
the Nix store and other auxiliary data, if desired:
<screen>
$ rm -rf /nix/store
@@ -189,18 +192,20 @@ $ rm -rf /nix/var</screen>
</section>
<!-- TODO: should be updated
<section><title>Upgrading Nix through Nix</title>
<para>You can install the latest stable version of Nix through Nix
itself by subscribing to the channel <link
xlink:href="http://nix.cs.uu.nl/dist/nix/channels-v3/nix-stable" />,
xlink:href="http://nixos.org/releases/nix/channels/nix-stable" />,
or the latest unstable version by subscribing to the channel <link
xlink:href="http://nix.cs.uu.nl/dist/nix/channels-v3/nix-unstable" />.
xlink:href="http://nixos.org/releases/nix/channels/nix-unstable" />.
You can also do a <link linkend="sec-one-click">one-click
installation</link> by clicking on the package links at <link
xlink:href="http://nix.cs.uu.nl/dist/nix/" />.</para>
xlink:href="http://nixos.org/releases/full-index-nix.html" />.</para>
</section>
-->
<section><title>Security</title>
@@ -334,8 +339,8 @@ $ chown -R root /nix/store /nix/var/nix</screen>
</para>
<para>The Nix daemon should be started as follows (as
<literal>root</literal>):
<para>The <link linkend="sec-nix-worker">Nix daemon</link> should be
started as follows (as <literal>root</literal>):
<screen>
$ nix-worker --daemon</screen>
+13 -12
View File
@@ -240,7 +240,7 @@ configuration (e.g., to build configuration files in
possible to easily roll back the entire configuration of the system to
an earlier state. Also, users can install software without root
privileges. For more information and downloads, see the <link
xlink:href="http://nix.cs.uu.nl/nixos/">NixOS homepage</link>.</para>
xlink:href="http://nixos.org/">NixOS homepage</link>.</para>
</simplesect>
@@ -257,14 +257,15 @@ xlink:href="http://nix.cs.uu.nl/nixos/">NixOS homepage</link>.</para>
<section><title>About us</title>
<para>Nix was developed at the <link
<para>Nix was originally developed at the <link
xlink:href="http://www.cs.uu.nl/">Department of Information and
Computing Sciences</link>, Utrecht University by the <link
xlink:href="http://www.cs.uu.nl/wiki/Trace/WebHome">TraCE
project</link>. The project is funded by the Software Engineering
Research Program <link
project</link> (2003-2008). The project was funded by the Software
Engineering Research Program <link
xlink:href="http://www.jacquard.nl/">Jacquard</link> to improve the
support for variability in software systems.</para>
support for variability in software systems. Further funding is now
provided by the NIRICT LaQuSo Build Farm project.</para>
</section>
@@ -299,27 +300,27 @@ Lesser General Public License for more details.</para>
<para>Some background information on Nix can be found in a number of
papers. The ICSE 2004 paper <citetitle
xlink:href='http://www.cs.uu.nl/~eelco/pubs/immdsd-icse2004-final.pdf'>Imposing
xlink:href='http://www.st.ewi.tudelft.nl/~dolstra/pubs/immdsd-icse2004-final.pdf'>Imposing
a Memory Management Discipline on Software Deployment</citetitle>
discusses the hashing mechanism used to ensure reliable dependency
identification and non-interference between different versions and
variants of packages. The LISA 2004 paper <citetitle
xlink:href='http://www.cs.uu.nl/~eelco/pubs/nspfssd-lisa2004-final.pdf'>Nix:
xlink:href='http://www.st.ewi.tudelft.nl/~dolstra/pubs/nspfssd-lisa2004-final.pdf'>Nix:
A Safe and Policy-Free System for Software Deployment</citetitle>
gives a more general discussion of Nix from a system-administration
perspective. The CBSE 2005 paper <citetitle
xlink:href='http://www.cs.uu.nl/~eelco/pubs/eupfcdm-cbse2005-final.pdf'>Efficient
xlink:href='http://www.st.ewi.tudelft.nl/~dolstra/pubs/eupfcdm-cbse2005-final.pdf'>Efficient
Upgrading in a Purely Functional Component Deployment Model
</citetitle> is about transparent patch deployment in Nix. The SCM-12
paper <citetitle
xlink:href='http://www.cs.uu.nl/~eelco/pubs/servicecm-scm12-final.pdf'>
xlink:href='http://www.st.ewi.tudelft.nl/~dolstra/pubs/servicecm-scm12-final.pdf'>
Service Configuration Management</citetitle> shows how services (e.g.,
web servers) can be deployed and managed through Nix. A short
overview of NixOS is given in the HotOS XI paper <citetitle
xlink:href="http://www.cs.uu.nl/~eelco/pubs/hotos-final.pdf">Purely
xlink:href="http://www.st.ewi.tudelft.nl/~dolstra/pubs/hotos-final.pdf">Purely
Functional System Configuration Management</citetitle>. The Nix
homepage has <link
xlink:href="http://nix.cs.uu.nl/docs/papers.html">an up-to-date list
xlink:href="http://nixos.org/docs/papers.html">an up-to-date list
of Nix-related papers</link>.</para>
<para>Nix is the subject of Eelco Dolstra’s PhD thesis <citetitle
@@ -328,7 +329,7 @@ Purely Functional Software Deployment Model</citetitle>, which
contains most of the papers listed above.</para>
<para>Nix has a homepage at <link
xlink:href="http://nix.cs.uu.nl/"/>.</para>
xlink:href="http://nixos.org/"/>.</para>
</section>
+21 -61
View File
@@ -14,9 +14,10 @@
<surname>Dolstra</surname>
</personname>
<affiliation>
<orgname>Utrecht University</orgname>
<orgdiv>Faculty of Science, Department of Information and Computing Sciences</orgdiv>
<orgname>Delft University of Technology</orgname>
<orgdiv>Department of Software Technology</orgdiv>
</affiliation>
<contrib>Author</contrib>
</author>
<copyright>
@@ -24,10 +25,12 @@
<year>2005</year>
<year>2006</year>
<year>2007</year>
<year>2008</year>
<year>2009</year>
<holder>Eelco Dolstra</holder>
</copyright>
<date>September 2007</date>
<date>September 2009</date>
</info>
@@ -48,72 +51,29 @@
<section>
<title>Main commands</title>
<section xml:id="sec-nix-env">
<title>nix-env</title>
<xi:include href="nix-env.xml" />
</section>
<section xml:id="sec-nix-instantiate">
<title>nix-instantiate</title>
<xi:include href="nix-instantiate.xml" />
</section>
<section xml:id="sec-nix-store">
<title>nix-store</title>
<xi:include href="nix-store.xml" />
</section>
<xi:include href="nix-env.xml" />
<xi:include href="nix-instantiate.xml" />
<xi:include href="nix-store.xml" />
</section>
<section>
<title>Utilities</title>
<section xml:id="sec-nix-build">
<title>nix-build</title>
<xi:include href="nix-build.xml" />
</section>
<section xml:id="sec-nix-channel">
<title>nix-channel</title>
<xi:include href="nix-channel.xml" />
</section>
<section xml:id="sec-nix-collect-garbage">
<title>nix-collect-garbage</title>
<xi:include href="nix-collect-garbage.xml" />
</section>
<section xml:id="sec-nix-copy-closure">
<title>nix-copy-closure</title>
<xi:include href="nix-copy-closure.xml" />
</section>
<section xml:id="sec-nix-hash">
<title>nix-hash</title>
<xi:include href="nix-hash.xml" />
</section>
<section xml:id="sec-nix-install-package">
<title>nix-install-package</title>
<xi:include href="nix-install-package.xml" />
</section>
<section xml:id="sec-nix-pack-closure">
<title>nix-pack-closure</title>
<xi:include href="nix-pack-closure.xml" />
</section>
<section xml:id="sec-nix-prefetch-url">
<title>nix-prefetch-url</title>
<xi:include href="nix-prefetch-url.xml" />
</section>
<section xml:id="sec-nix-pull">
<title>nix-pull</title>
<xi:include href="nix-pull.xml" />
</section>
<section xml:id="sec-nix-push">
<title>nix-push</title>
<xi:include href="nix-push.xml" />
</section>
<section xml:id="sec-nix-unpack-closure">
<title>nix-unpack-closure</title>
<xi:include href="nix-unpack-closure.xml" />
</section>
<xi:include href="nix-build.xml" />
<xi:include href="nix-channel.xml" />
<xi:include href="nix-collect-garbage.xml" />
<xi:include href="nix-copy-closure.xml" />
<xi:include href="nix-hash.xml" />
<xi:include href="nix-install-package.xml" />
<xi:include href="nix-prefetch-url.xml" />
<xi:include href="nix-pull.xml" />
<xi:include href="nix-push.xml" />
<xi:include href="nix-worker.xml" />
</section>
</appendix>
<xi:include href="troubleshooting.xml" />
<xi:include href="bugs.xml" />
<!-- <xi:include href="bugs.xml" /> -->
<xi:include href="glossary.xml" />
<appendix>
+2 -1
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-build">
<refmeta>
<refentrytitle>nix-build</refentrytitle>
+2 -1
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-channel">
<refmeta>
<refentrytitle>nix-channel</refentrytitle>
+2 -1
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-collect-garbage">
<refmeta>
<refentrytitle>nix-collect-garbage</refentrytitle>
+3 -2
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-copy-closure">
<refmeta>
<refentrytitle>nix-copy-closure</refentrytitle>
@@ -128,7 +129,7 @@ those paths. If this bothers you, use
<para>Copy Firefox with all its dependencies to a remote machine:
<screen>
$ nix-copy-closure alice@itchy.labs $(type -tP firefox)</screen>
$ nix-copy-closure --to alice@itchy.labs $(type -tP firefox)</screen>
</para>
+3 -2
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-env">
<refmeta>
<refentrytitle>nix-env</refentrytitle>
@@ -526,7 +527,7 @@ upgrading `mozilla-1.2' to `mozilla-1.4'</screen>
</refsection>
<refsection><title>Versions</title>
<refsection xml:id="ssec-version-comparisons"><title>Versions</title>
<para>The upgrade operation determines whether a derivation
<varname>y</varname> is an upgrade of a derivation
+2 -1
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-hash">
<refmeta>
<refentrytitle>nix-hash</refentrytitle>
+2 -1
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-install-package">
<refmeta>
<refentrytitle>nix-install-package</refentrytitle>
+2 -1
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-instantiate">
<refmeta>
<refentrytitle>nix-instantiate</refentrytitle>
-95
View File
@@ -178,100 +178,5 @@
</productionset>
</sect1>
<sect1>
<title>Semantics</title>
<sect2>
<title>Built-in functions</title>
<para>
The Nix language provides the following built-in function
(<quote>primops</quote>):
</para>
<variablelist>
<varlistentry>
<term><function>import</function>
<replaceable>e</replaceable></term>
<listitem>
<para>
Evaluates the expression <replaceable>e</replaceable>,
which must yield a path value. The Nix expression
stored at this path in the file system is then read,
parsed, and evaluated. Returns the result of the
evaluation of the Nix expression just read.
</para>
<para>
Example: <literal>import ./foo.nix</literal> evaluates
the expression stored in <filename>foo.nix</filename>
(in the directory containing the expression in which the
<function>import</function> occurs).
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><function>derivation</function>
<replaceable>e</replaceable></term>
<listitem>
<para>
Evaluates the expression <replaceable>e</replaceable>,
which must yield an attribute set. [...]
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><function>baseNameOf</function>
<replaceable>e</replaceable></term>
<listitem>
<para>
Evaluates the expression <replaceable>e</replaceable>,
which must yield a string value, and returns a string
representing its <emphasis>base name</emphasis>. This
is the substring following the last path separator
(<literal>/</literal>).
</para>
<para>
Example: <literal>baseNameOf "/foo/bar"</literal>
returns <literal>"bar"</literal>, and
<literal>baseNameOf "/foo/bar/"</literal> returns
<literal>""</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><function>toString</function>
<replaceable>e</replaceable></term>
<listitem>
<para>
Evaluates the expression <replaceable>e</replaceable>
and coerces it into a string, if possible. Only
strings, paths, and URIs can be so coerced.
</para>
<para>
Example: <literal>toString
http://www.cs.uu.nl/</literal> returns
<literal>"http://www.cs.uu.nl/"</literal>.
</para>
</listitem>
</varlistentry>
</variablelist>
</sect2>
</sect1>
</appendix>
-82
View File
@@ -1,82 +0,0 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
<refmeta>
<refentrytitle>nix-pack-closure</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo class="source">Nix</refmiscinfo>
<refmiscinfo class="version"><xi:include href="version.txt" parse="text"/></refmiscinfo>
</refmeta>
<refnamediv>
<refname>nix-pack-closure</refname>
<refpurpose>pack the closure of a store path into a single file that
can be unpacked with
<command>nix-unpack-closure</command></refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>nix-pack-closure</command>
<arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>
<refsection><title>Description</title>
<para>The command <command>nix-pack-closure</command> packs the
contents of the store paths <replaceable>paths</replaceable> and
<emphasis>all their dependencies</emphasis> into a single file, which
is written to standard output. (That is, it
<emphasis>serialises</emphasis> <replaceable>paths</replaceable>.)
The output can then be unpacked into the Nix store of another machine
using <command>nix-unpack-closure</command>.</para>
<para>Together, <command>nix-pack-closure</command> and
<command>nix-unpack-closure</command> provide a quick and easy way to
deploy a package to a different machine. However, as the output of
<command>nix-pack-closure</command> tends to be rather large (since it
contains all dependencies), it’s not very efficient.
<command>nix-push</command> and <command>nix-pull</command> are more
efficient, but are also a bit more cumbersome to use.</para>
</refsection>
<refsection><title>Examples</title>
<para>To copy some instance of Subversion with all its dependencies to
another machine:
<screen>
$ nix-pack-closure /nix/store/hj232g1r...-subversion-1.3.0 > svn.closure
<lineannotation>Copy <!-- !!! <filename> -->svn.closure to the remote machine, then on the remote machine do:</lineannotation>
$ nix-unpack-closure &lt; svn.closure</screen>
</para>
<para>Copy the program <command>azureus</command> with all its
dependencies to the machine <literal>scratchy</literal>:
<screen>
$ nix-pack-closure $(which azureus) | ssh scratchy nix-unpack-closure</screen>
</para>
<para>As a variation on the previous example, copy
<command>azureus</command>, and also install it in the user’s profile
on the target machine:
<screen>
$ nix-pack-closure $(which azureus) | ssh scratchy 'nix-env -i $(nix-unpack-closure)'</screen>
</para>
</refsection>
</refentry>
+2 -1
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-prefetch-url">
<refmeta>
<refentrytitle>nix-prefetch-url</refentrytitle>
+2 -1
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-pull">
<refmeta>
<refentrytitle>nix-pull</refentrytitle>
+2 -1
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-push">
<refmeta>
<refentrytitle>nix-push</refentrytitle>
+112 -4
View File
@@ -1,6 +1,7 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-store">
<refmeta>
<refentrytitle>nix-store</refentrytitle>
@@ -128,6 +129,7 @@ lrwxrwxrwx 1 ... 2005-03-13 21:10 /home/eelco/bla/result -> /nix/store/1r1134
<arg choice='plain'><option>-r</option></arg>
</group>
<arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg>
<arg><option>--dry-run</option></arg>
</cmdsynopsis>
</refsection>
@@ -166,6 +168,11 @@ the specified store paths. Realisation is a somewhat overloaded term:
output. (For non-derivations argument, the argument itself is
printed.)</para>
<para>If the <option>--dry-run</option> option is used, then
<command>nix-store</command> will print on standard error a
description of what packages would be built or downloaded, and then
quit.</para>
</refsection>
@@ -204,6 +211,8 @@ linkend="sec-nix-build"><command>nix-build</command></link> does.</para>
<arg choice='plain'><option>--print-dead</option></arg>
<arg choice='plain'><option>--delete</option></arg>
</group>
<arg><option>--max-freed</option> <replaceable>bytes</replaceable></arg>
<arg><option>--max-links</option> <replaceable>nrlinks</replaceable></arg>
</cmdsynopsis>
</refsection>
@@ -258,7 +267,34 @@ the Nix store not reachable via file system references from a set of
</variablelist>
<para>The behaviour of the collector is influenced by the <link
<para>By default, all unreachable paths are deleted. The following
options control what gets deleted and in what order:
<variablelist>
<varlistentry><term><option>--max-freed</option> <replaceable>bytes</replaceable></term>
<listitem><para>Keep deleting paths until at least
<replaceable>bytes</replaceable> bytes have been
deleted, then stop.</para></listitem>
</varlistentry>
<varlistentry><term><option>--max-links</option> <replaceable>nrlinks</replaceable></term>
<listitem><para>Keep deleting paths until the hard link count on
<filename>/nix/store</filename> is less than
<replaceable>nrlinks</replaceable>, then stop. This is useful for
very large Nix stores on filesystems with a 32000 subdirectories
limit (like <literal>ext3</literal>).</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>The behaviour of the collector is also influenced by the <link
linkend="conf-gc-keep-outputs"><literal>gc-keep-outputs</literal></link>
and <link
linkend="conf-gc-keep-derivations"><literal>gc-keep-derivations</literal></link>
@@ -284,6 +320,13 @@ deleting `/nix/store/kq82idx6g0nyzsp2s14gfsc38npai7lf-cairo-1.0.4.tar.gz.drv'
</para>
<para>To delete at least 100 MiBs of unreachable paths:
<screen>
$ nix-store --gc --max-freed $((100 * 1024 * 1024))</screen>
</para>
</refsection>
@@ -361,6 +404,7 @@ error: cannot delete path `/nix/store/zq0h41l75vlb4z45kzgjjmsjxvcv1qk7-mesa-6.4'
<arg choice='plain'><option>--tree</option></arg>
<arg choice='plain'><option>--binding</option> <replaceable>name</replaceable></arg>
<arg choice='plain'><option>--hash</option></arg>
<arg choice='plain'><option>--roots</option></arg>
</group>
<arg><option>--use-output</option></arg>
<arg><option>-u</option></arg>
@@ -543,12 +587,20 @@ query is applied to the target of the symlink.</para>
<varlistentry><term><option>--hash</option></term>
<listitem><para>Prints the SHA-256 hash of the contents of the
store path <replaceable>paths</replaceable>. Since the hash is
store paths <replaceable>paths</replaceable>. Since the hash is
stored in the Nix database, this is a fast
operation.</para></listitem>
</varlistentry>
<varlistentry><term><option>--roots</option></term>
<listitem><para>Prints the garbage collector roots that point,
directly or indirectly, at the store paths
<replaceable>paths</replaceable>.</para></listitem>
</varlistentry>
</variablelist>
</refsection>
@@ -626,6 +678,18 @@ $ gv graph.ps</screen>
</para>
<para>Show every garbage collector root that points to a store path
that depends on <command>svn</command>:
<screen>
$ nix-store -q --roots $(which svn)
/nix/var/nix/profiles/default-81-link
/nix/var/nix/profiles/default-82-link
/nix/var/nix/profiles/per-user/eelco/profile-97-link
</screen>
</para>
</refsection>
@@ -950,7 +1014,51 @@ ktorrent-2.2.1/NEWS
</refsection>
<!-- TODO: export, import operations -->
<!--######################################################################-->
<refsection><title>Operation <option>--dump-db</option></title>
<refsection>
<title>Synopsis</title>
<cmdsynopsis>
<command>nix-store</command>
<arg choice='plain'><option>--dump-db</option></arg>
</cmdsynopsis>
</refsection>
<refsection><title>Description</title>
<para>The operation <option>--dump-db</option> writes a dump of the
Nix database to standard output. It can be loaded into an empty Nix
store using <option>--load-db</option>. This is useful for making
backups and when migrating to different database schemas.</para>
</refsection>
</refsection>
<!--######################################################################-->
<refsection><title>Operation <option>--dump-db</option></title>
<refsection>
<title>Synopsis</title>
<cmdsynopsis>
<command>nix-store</command>
<arg choice='plain'><option>--load-db</option></arg>
</cmdsynopsis>
</refsection>
<refsection><title>Description</title>
<para>The operation <option>--load-db</option> reads a dump of the Nix
database created by <option>--dump-db</option> from standard input and
loads it into the Nix database.</para>
</refsection>
</refsection>
</refentry>
-42
View File
@@ -1,42 +0,0 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude">
<refmeta>
<refentrytitle>nix-unpack-closure</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo class="source">Nix</refmiscinfo>
<refmiscinfo class="version"><xi:include href="version.txt" parse="text"/></refmiscinfo>
</refmeta>
<refnamediv>
<refname>nix-unpack-closure</refname>
<refpurpose>unpack the closure of a store path created by <command>nix-pack-closure</command> into the Nix store</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>nix-unpack-closure</command>
</cmdsynopsis>
</refsynopsisdiv>
<refsection><title>Description</title>
<para>The command <command>nix-unpack-closure</command> unpacks the
closure of a set of store paths created by
<command>nix-pack-closure</command> into the local Nix store. The
closure is a single file read from standard input. See the
description of <command>nix-pack-closure</command> for details and
examples.</para>
<para>The top-level paths in the closure (i.e., the paths passed to
the original <command>nix-pack-closure</command> call that created the
closure) are printed on standard output. These paths can be passed,
for instance, to <literal>nix-env -i</literal> to install them into a
user environment on the target machine.</para>
</refsection>
</refentry>
+35
View File
@@ -0,0 +1,35 @@
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-nix-worker">
<refmeta>
<refentrytitle>nix-worker</refentrytitle>
<manvolnum>8</manvolnum>
<refmiscinfo class="source">Nix</refmiscinfo>
<refmiscinfo class="version"><xi:include href="version.txt" parse="text"/></refmiscinfo>
</refmeta>
<refnamediv>
<refname>nix-worker</refname>
<refpurpose>Nix multi-user support daemon</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>nix-worker</command>
<arg choice="plain"><option>--daemon</option></arg>
</cmdsynopsis>
</refsynopsisdiv>
<refsection><title>Description</title>
<para>The Nix daemon is necessary in multi-user Nix installations. It
performs build actions and other operations on the Nix store on behalf
of unprivileged users.</para>
</refsection>
</refentry>
+1
View File
@@ -24,6 +24,7 @@
<arg><option>--fallback</option></arg>
<arg><option>--readonly-mode</option></arg>
<arg><option>--log-type</option> <replaceable>type</replaceable></arg>
<arg><option>--show-trace</option></arg>
<sbr />
</nop>
+10 -2
View File
@@ -251,14 +251,14 @@
<programlisting>
{ # The system (e.g., `i686-linux') for which to build the packages.
system ? __currentSystem
system ? builtins.currentSystem
<replaceable>...</replaceable>
}: <replaceable>...</replaceable></programlisting>
So if you call this Nix expression (e.g., when you do
<literal>nix-env -i <replaceable>pkgname</replaceable></literal>),
the function will be called automatically using the value <link
linkend='builtin-currentSystem'><literal>__currentSystem</literal></link>
linkend='builtin-currentSystem'><literal>builtins.currentSystem</literal></link>
for the <literal>system</literal> argument. You can override this
using <option>--arg</option>, e.g., <literal>nix-env -i
<replaceable>pkgname</replaceable> --arg system
@@ -305,6 +305,14 @@
</varlistentry>
<varlistentry><term><option>--show-trace</option></term>
<listitem><para>Causes Nix to print out a stack trace in case of Nix
expression evaluation errors.</para></listitem>
</varlistentry>
</variablelist>
+104 -30
View File
@@ -38,7 +38,7 @@ to end-user applications like Mozilla Firefox. (Nix is however not
tied to the Nix Package collection; you could write your own Nix
expressions based on it, or completely new ones.) You can download
the latest version from <link
xlink:href='http://nix.cs.uu.nl/dist/nix' />.</para>
xlink:href='http://nixos.org/releases/full-index-nixpkgs.html' />.</para>
<para>Assuming that you have downloaded and unpacked a release of Nix
Packages, you can view the set of available packages in the release:
@@ -118,15 +118,15 @@ available somewhere. This is done using the
containing a <emphasis>manifest</emphasis> describing what binaries
are available. This URL should correspond to the Nix Packages release
that you’re using. For instance, if you obtained a release from <link
xlink:href='http://nix.cs.uu.nl/dist/nix/nixpkgs-0.6pre1554/' />, then
you should do:
xlink:href='http://nixos.org/releases/nixpkgs/nixpkgs-0.12pre11712-4lrp7j8x'
/>, then you should do:
<screen>
$ nix-pull http://nix.cs.uu.nl/dist/nix/nixpkgs-0.6pre1554/MANIFEST</screen>
$ nix-pull http://nixos.org/releases/nixpkgs/nixpkgs-0.12pre11712-4lrp7j8x/MANIFEST</screen>
If you then issue the installation command, it should start
downloading binaries from <systemitem
class='fqdomainname'>nix.cs.uu.nl</systemitem>, instead of building
class='fqdomainname'>nixos.org</systemitem>, instead of building
them from source. This might still take a while since all
dependencies must be downloaded, but on a reasonably fast connection
such as an DSL line it’s on the order of a few minutes.</para>
@@ -331,7 +331,7 @@ default profile, respectively. If the profile doesn’t exist, it will
be created automatically. You should be careful about storing a
profile in another location than the <filename>profiles</filename>
directory, since otherwise it might not be used as a root of the
garbage collector (see section <xref linkend='sec-garbage-collection'
garbage collector (see <xref linkend='sec-garbage-collection'
/>).</para>
<para>All <command>nix-env</command> operations work on the profile
@@ -458,7 +458,7 @@ URL.</para>
<command>nix-channel --add</command>, e.g.,
<screen>
$ nix-channel --add http://nix.cs.uu.nl/dist/nix/channels-v3/nixpkgs-unstable</screen>
$ nix-channel --add http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable</screen>
subscribes you to a channel that always contains that latest version
of the Nix Packages collection. (Instead of
@@ -496,31 +496,27 @@ available in the subscribed channels.</para>
<para>Often, when you want to install a specific package (e.g., from
the <link
xlink:href="http://nix.cs.uu.nl/dist/nix/nixpkgs-unstable-latest/">Nix
Packages collection</link> or from our <link
xlink:href='http://nix.cs.uu.nl/dist/'>release server</link>),
subscribing to a channel is a bit cumbersome. And channels don’t help
you at all if you want to install an older version of a package than
the one provided by the current contents of the channel, or a package
that has been removed from the channel. That’s when
<emphasis>one-click installs</emphasis> come in handy: you can just go
to the web page that contains the package, click on it, and it will be
installed with all the necessary dependencies.</para>
xlink:href="http://nixos.org/releases/nixpkgs/nixpkgs-unstable/">Nix
Packages collection</link>), subscribing to a channel is a bit
cumbersome. And channels don’t help you at all if you want to install
an older version of a package than the one provided by the current
contents of the channel, or a package that has been removed from the
channel. That’s when <emphasis>one-click installs</emphasis> come in
handy: you can just go to the web page that contains the package,
click on it, and it will be installed with all the necessary
dependencies.</para>
<para>For instance, you can go to <link
xlink:href="http://nix.cs.uu.nl/dist/nix/nixpkgs-unstable-latest/" />
— or to any older release of Nix Packages — and click on any link for
the individual packages for your platform (say, <link
xlink:href='http://nix.cs.uu.nl/dist/nix/nixpkgs-0.10pre6622/pkgs/subversion-1.4.0-i686-linux.nixpkg'><literal>subversion-1.4.0</literal>
for <literal>i686-linux</literal></link>). The first time you do
this, your browser will ask what to do with
<literal>application/nix-package</literal> files. You should open
them with <filename>/nix/bin/nix-install-package</filename>. This
will open a window that asks you to confirm that you want to install
the package. When you answer <literal>Y</literal>, the package and
all its dependencies will be installed. This is a binary deployment
mechanism — you get packages pre-compiled for the selected platform
type.</para>
xlink:href="http://hydra.nixos.org/jobset/nixpkgs/trunk/channel/latest"
/> and click on any link for the individual packages for your
platform. The first time you do this, your browser will ask what to
do with <literal>application/nix-package</literal> files. You should
open them with <filename>/nix/bin/nix-install-package</filename>.
This will open a window that asks you to confirm that you want to
install the package. When you answer <literal>Y</literal>, the
package and all its dependencies will be installed. This is a binary
deployment mechanism — you get packages pre-compiled for the selected
platform type.</para>
<para>You can also install <literal>application/nix-package</literal>
files from the command line directly. See <xref
@@ -529,4 +525,82 @@ linkend='sec-nix-install-package' /> for details.</para>
</section>
<section xml:id="sec-sharing-packages"><title>Sharing packages between machines</title>
<para>Sometimes you want to copy a package from one machine to
another. Or, you want to install some packages and you know that
another machine already has some or all of those packages or their
dependencies. In that case there are mechanisms to quickly copy
packages between machines.</para>
<para>The command <command
linkend="sec-nix-copy-closure">nix-copy-closure</command> copies a Nix
store path along with all its dependencies to or from another machine
via the SSH protocol. It doesn’t copy store paths that are already
present on the target machine. For example, the following command
copies Firefox with all its dependencies:
<screen>
$ nix-copy-closure --to alice@itchy.example.org $(type -p firefox)</screen>
See <xref linkend='sec-nix-copy-closure' /> for details.</para>
<para>With <command linkend='refsec-nix-store-export'>nix-store
--export</command> and <command
linkend='refsec-nix-store-import'>nix-store --import</command> you can
write the closure of a store path (that is, the path and all its
dependencies) to a file, and then unpack that file into another Nix
store. For example,
<screen>
$ nix-store --export $(type -p firefox) > firefox.closure</screen>
writes the closure of Firefox to a file. You can then copy this file
to another machine and install the closure:
<screen>
$ nix-store --import &lt; firefox.closure</screen>
Any store paths in the closure that are already present in the target
store are ignored. It is also possible to pipe the export into
another command, e.g. to copy and install a closure directly to/on
another machine:
<screen>
$ nix-store --export $(type -p firefox) | bzip2 | \
ssh alice@itchy.example.org "bunzip2 | nix-store --import"</screen>
But note that <command>nix-copy-closure</command> is generally more
efficient in this example because it only copies paths that are not
already present in the target Nix store.</para>
<para>Finally, if you can mount the Nix store of a remote machine in
your local filesystem, Nix can copy paths from the remote Nix store to
the local Nix store <emphasis>on demand</emphasis>. For instance,
suppose that you mount a remote machine containing a Nix store via
<command
xlink:href="http://fuse.sourceforge.net/sshfs.html">sshfs</command>:
<screen>
$ sshfs alice@itchy.example.org:/ /mnt</screen>
You should then set the <envar>NIX_OTHER_STORES</envar> environment
variable to tell Nix about this remote Nix store:
<screen>
$ export NIX_OTHER_STORES=/mnt/nix</screen>
Then if you do any Nix operation, e.g.
<screen>
$ nix-env -i firefox</screen>
and Nix has to build a path that it sees is already present in
<filename>/mnt/nix</filename>, then it will just copy from there
instead of building it from source.</para>
</section>
</chapter>
+22 -13
View File
@@ -1,5 +1,6 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink">
xmlns:xlink="http://www.w3.org/1999/xlink"
xml:id="chap-quick-start">
<title>Quick Start</title>
@@ -10,9 +11,9 @@ to the following chapters.</para>
<orderedlist>
<listitem><para>Download a source tarball or RPM from <link
xlink:href='http://nix.cs.uu.nl/'/>. Build source
distributions using the regular sequence:
<listitem><para>Download a source tarball, RPM or Deb from <link
xlink:href='http://nixos.org/'/>. Build source distributions using
the regular sequence:
<screen>
$ tar xvfj nix-<replaceable>version</replaceable>.tar.bz2
@@ -20,20 +21,28 @@ $ ./configure
$ make
$ make install <lineannotation>(as root)</lineannotation></screen>
This will install Nix in <filename>/nix</filename>. You shouldn't
change the prefix if at all possible since that will make it
impossible to use pre-built binaries from the Nixpkgs channel and
other channels. Alternatively, you could grab an RPM if you're on an
RPM-based system. You should also add
<filename>/nix/etc/profile.d/nix.sh</filename> to your
<filename>~/.bashrc</filename> (or some other login
This will install the Nix binaries in <filename>/usr/local</filename>
and keep the Nix store and other state in <filename>/nix</filename>.
You can change the former by specifying
<option>--prefix=<replaceable>path</replaceable></option>. The
location of the store can be changed using
<option>--with-store-dir=<replaceable>path</replaceable></option>.
However, you shouldn't change the store location, if at all possible,
since that will make it impossible to use pre-built binaries from the
Nixpkgs channel and other channels. The location of the state can be
changed using
<option>--localstatedir=<replaceable>path</replaceable>.</option></para></listitem>
<listitem><para>You should add
<filename><replaceable>prefix</replaceable>/etc/profile.d/nix.sh</filename>
to your <filename>~/.bashrc</filename> (or some other login
file).</para></listitem>
<listitem><para>Subscribe to the Nix Packages channel.
<screen>
$ nix-channel --add \
http://nix.cs.uu.nl/dist/nix/channels-v3/nixpkgs-unstable</screen>
http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable</screen>
</para></listitem>
@@ -99,7 +108,7 @@ numbers).</para></listitem>
<listitem><para>You can also install specific packages directly from
your web browser. For instance, you can go to <link
xlink:href="http://nix.cs.uu.nl/dist/nix/nixpkgs-unstable-latest/" />
xlink:href="http://hydra.nixos.org/jobset/nixpkgs/trunk/channel/latest" />
and click on any link for the individual packages for your platform.
Associate <literal>application/nix-package</literal> with the program
<filename>/nix/bin/nix-install-package</filename>. A window should
+319
View File
@@ -6,6 +6,325 @@
<!--==================================================================-->
<section xml:id="ssec-relnotes-0.14"><title>Release 0.14 (TBA)</title>
<para>This release has the following improvements:</para>
<itemizedlist>
<listitem>
<para>The garbage collector now starts deleting garbage much
faster than before. It no longer determines liveness of all paths
in the store, but does so on demand.</para>
</listitem>
<listitem>
<para>Added a new operation, <command>nix-store --query
--roots</command>, that shows the garbage collector roots that
directly or indirectly point to the given store paths.</para>
</listitem>
<listitem>
<para>Removed support for converting Berkeley DB-based Nix
databases to the new schema.</para>
</listitem>
<listitem>
<para>Removed the <option>--use-atime</option> and
<option>--max-atime</option> garbage collector options. They were
not very useful in practice.</para>
</listitem>
<listitem>
<para>A few bug fixes.</para>
</listitem>
</itemizedlist>
</section>
<!--==================================================================-->
<section xml:id="ssec-relnotes-0.13"><title>Release 0.13 (November 5,
2009)</title>
<para>This is primarily a bug fix release. It has some new
features:</para>
<itemizedlist>
<listitem>
<para>Syntactic sugar for writing nested attribute sets. Instead of
<programlisting>
{
foo = {
bar = 123;
xyzzy = true;
};
a = { b = { c = "d"; }; };
}
</programlisting>
you can write
<programlisting>
{
foo.bar = 123;
foo.xyzzy = true;
a.b.c = "d";
}
</programlisting>
This is useful, for instance, in NixOS configuration files.</para>
</listitem>
<listitem>
<para>Support for Nix channels generated by Hydra, the Nix-based
continuous build system. (Hydra generates NAR archives on the
fly, so the size and hash of these archives isn’t known in
advance.)</para>
</listitem>
<listitem>
<para>Support <literal>i686-linux</literal> builds directly on
<literal>x86_64-linux</literal> Nix installations. This is
implemented using the <function>personality()</function> syscall,
which causes <command>uname</command> to return
<literal>i686</literal> in child processes.</para>
</listitem>
<listitem>
<para>Various improvements to the <literal>chroot</literal>
support. Building in a <literal>chroot</literal> works quite well
now.</para>
</listitem>
<listitem>
<para>Nix no longer blocks if it tries to build a path and another
process is already building the same path. Instead it tries to
build another buildable path first. This improves
parallelism.</para>
</listitem>
<listitem>
<para>Support for large (> 4 GiB) files in NAR archives.</para>
</listitem>
<listitem>
<para>Various (performance) improvements to the remote build
mechanism.</para>
</listitem>
<listitem>
<para>New primops: <varname>builtins.addErrorContext</varname> (to
add a string to stack traces — useful for debugging),
<varname>builtins.isBool</varname>,
<varname>builtins.isString</varname>,
<varname>builtins.isInt</varname>,
<varname>builtins.intersectAttrs</varname>.</para>
</listitem>
<listitem>
<para>OpenSolaris support (Sander van der Burg).</para>
</listitem>
<listitem>
<para>Stack traces are no longer displayed unless the
<option>--show-trace</option> option is used.</para>
</listitem>
<listitem>
<para>The scoping rules for <literal>inherit
(<replaceable>e</replaceable>) ...</literal> in recursive
attribute sets have changed. The expression
<replaceable>e</replaceable> can now refer to the attributes
defined in the containing set.</para>
</listitem>
</itemizedlist>
</section>
<!--==================================================================-->
<section xml:id="ssec-relnotes-0.12"><title>Release 0.12 (November 20,
2008)</title>
<itemizedlist>
<listitem>
<para>Nix no longer uses Berkeley DB to store Nix store metadata.
The principal advantages of the new storage scheme are: it works
properly over decent implementations of NFS (allowing Nix stores
to be shared between multiple machines); no recovery is needed
when a Nix process crashes; no write access is needed for
read-only operations; no more running out of Berkeley DB locks on
certain operations.</para>
<para>You still need to compile Nix with Berkeley DB support if
you want Nix to automatically convert your old Nix store to the
new schema. If you don’t need this, you can build Nix with the
<filename>configure</filename> option
<option>--disable-old-db-compat</option>.</para>
<para>After the automatic conversion to the new schema, you can
delete the old Berkeley DB files:
<screen>
$ cd /nix/var/nix/db
$ rm __db* log.* derivers references referrers reserved validpaths DB_CONFIG</screen>
The new metadata is stored in the directories
<filename>/nix/var/nix/db/info</filename> and
<filename>/nix/var/nix/db/referrer</filename>. Though the
metadata is stored in human-readable plain-text files, they are
not intended to be human-editable, as Nix is rather strict about
the format.</para>
<para>The new storage schema may or may not require less disk
space than the Berkeley DB environment, mostly depending on the
cluster size of your file system. With 1 KiB clusters (which
seems to be the <literal>ext3</literal> default nowadays) it
usually takes up much less space.</para>
</listitem>
<listitem><para>There is a new substituter that copies paths
directly from other (remote) Nix stores mounted somewhere in the
filesystem. For instance, you can speed up an installation by
mounting some remote Nix store that already has the packages in
question via NFS or <literal>sshfs</literal>. The environment
variable <envar>NIX_OTHER_STORES</envar> specifies the locations of
the remote Nix directories,
e.g. <literal>/mnt/remote-fs/nix</literal>.</para></listitem>
<listitem><para>New <command>nix-store</command> operations
<option>--dump-db</option> and <option>--load-db</option> to dump
and reload the Nix database.</para></listitem>
<listitem><para>The garbage collector has a number of new options to
allow only some of the garbage to be deleted. The option
<option>--max-freed <replaceable>N</replaceable></option> tells the
collector to stop after at least <replaceable>N</replaceable> bytes
have been deleted. The option <option>--max-links
<replaceable>N</replaceable></option> tells it to stop after the
link count on <filename>/nix/store</filename> has dropped below
<replaceable>N</replaceable>. This is useful for very large Nix
stores on filesystems with a 32000 subdirectories limit (like
<literal>ext3</literal>). The option <option>--use-atime</option>
causes store paths to be deleted in order of ascending last access
time. This allows non-recently used stuff to be deleted. The
option <option>--max-atime <replaceable>time</replaceable></option>
specifies an upper limit to the last accessed time of paths that may
be deleted. For instance,
<screen>
$ nix-store --gc -v --max-atime $(date +%s -d "2 months ago")</screen>
deletes everything that hasn’t been accessed in two months.</para></listitem>
<listitem><para><command>nix-env</command> now uses optimistic
profile locking when performing an operation like installing or
upgrading, instead of setting an exclusive lock on the profile.
This allows multiple <command>nix-env -i / -u / -e</command>
operations on the same profile in parallel. If a
<command>nix-env</command> operation sees at the end that the profile
was changed in the meantime by another process, it will just
restart. This is generally cheap because the build results are
still in the Nix store.</para></listitem>
<listitem><para>The option <option>--dry-run</option> is now
supported by <command>nix-store -r</command> and
<command>nix-build</command>.</para></listitem>
<listitem><para>The information previously shown by
<option>--dry-run</option> (i.e., which derivations will be built
and which paths will be substituted) is now always shown by
<command>nix-env</command>, <command>nix-store -r</command> and
<command>nix-build</command>. The total download size of
substitutable paths is now also shown. For instance, a build will
show something like
<screen>
the following derivations will be built:
/nix/store/129sbxnk5n466zg6r1qmq1xjv9zymyy7-activate-configuration.sh.drv
/nix/store/7mzy971rdm8l566ch8hgxaf89x7lr7ik-upstart-jobs.drv
...
the following paths will be downloaded/copied (30.02 MiB):
/nix/store/4m8pvgy2dcjgppf5b4cj5l6wyshjhalj-samba-3.2.4
/nix/store/7h1kwcj29ip8vk26rhmx6bfjraxp0g4l-libunwind-0.98.6
...</screen>
</para></listitem>
<listitem><para>Language features:
<itemizedlist>
<listitem><para>@-patterns as in Haskell. For instance, in a
function definition
<programlisting>f = args @ {x, y, z}: <replaceable>...</replaceable>;</programlisting>
<varname>args</varname> refers to the argument as a whole, which
is further pattern-matched against the attribute set pattern
<literal>{x, y, z}</literal>.</para></listitem>
<listitem><para>“<literal>...</literal>” (ellipsis) patterns.
An attribute set pattern can now say <literal>...</literal> at
the end of the attribute name list to specify that the function
takes <emphasis>at least</emphasis> the listed attributes, while
ignoring additional attributes. For instance,
<programlisting>{stdenv, fetchurl, fuse, ...}: <replaceable>...</replaceable></programlisting>
defines a function that accepts any attribute set that includes
at least the three listed attributes.</para></listitem>
<listitem><para>New primops:
<varname>builtins.parseDrvName</varname> (split a package name
string like <literal>"nix-0.12pre12876"</literal> into its name
and version components, e.g. <literal>"nix"</literal> and
<literal>"0.12pre12876"</literal>),
<varname>builtins.compareVersions</varname> (compare two version
strings using the same algorithm that <command>nix-env</command>
uses), <varname>builtins.length</varname> (efficiently compute
the length of a list), <varname>builtins.mul</varname> (integer
multiplication), <varname>builtins.div</varname> (integer
division).
<!-- <varname>builtins.genericClosure</varname> -->
</para></listitem>
</itemizedlist>
</para></listitem>
<listitem><para><command>nix-prefetch-url</command> now supports
<literal>mirror://</literal> URLs, provided that the environment
variable <envar>NIXPKGS_ALL</envar> points at a Nixpkgs
tree.</para></listitem>
<listitem><para>Removed the commands
<command>nix-pack-closure</command> and
<command>nix-unpack-closure</command>. You can do almost the same
thing but much more efficiently by doing <literal>nix-store --export
$(nix-store -qR <replaceable>paths</replaceable>) > closure</literal> and
<literal>nix-store --import &lt;
closure</literal>.</para></listitem>
<listitem><para>Lots of bug fixes, including a big performance bug in
the handling of <literal>with</literal>-expressions.</para></listitem>
</itemizedlist>
</section>
<!--==================================================================-->
<section xml:id="ssec-relnotes-0.11"><title>Release 0.11 (December 31,
+3 -113
View File
@@ -5,121 +5,10 @@
<para>This section provides solutions for some common problems. See
the <link xlink:href="https://bugs.cs.uu.nl/browse/NIX">Nix
the <link xlink:href="http://bugs.strategoxt.org/browse/NIX">Nix
bug tracker</link> for a list of currently known issues.</para>
<section><title>Berkeley DB: <quote>Cannot allocate memory</quote></title>
<para>Symptom: Nix operations (in particular the
<command>nix-store</command> operations <option>--gc</option>,
<option>--verify</option>, and <option>--clear-substitutes</option> —
the latter being called by <command>nix-channel --update</command>)
failing:
<screen>
$ nix-store --verify
error: Db::del: Cannot allocate memory</screen>
</para>
<para>Possible solution: make sure that no Nix processes are running,
then do:
<screen>
$ cd /nix/var/nix/db
$ rm __db.00*</screen>
</para>
</section>
<section><title>Berkeley DB gives weird error messages</title>
<para>Symptom: you get error messages such as
<screen>
Berkeley DB message: Finding last valid log LSN: file: 1 offset 28
Berkeley DB error: file validpaths (meta pgno = 0) has LSN [483][34721].
Berkeley DB error: end of log is [1][28]
Berkeley DB error: /nix/var/nix/db/validpaths: unexpected file type or format</screen>
or other weird Berkeley DB errors, and they don’t go away (i.e.,
automatic recovery doesn’t work). This may be the case after a system
crash.</para>
<para>Solution: first try to run <command>db_recover</command> and
then <link linkend='refsec-nix-store-verify'><command>nix-store
--verify</command></link>:
<screen>
$ db_recover -h /nix/var/nix/db
$ nix-store --verify</screen>
(Make sure that you have the right version of
<command>db_recover</command>, namely, Berkeley DB 4.4 for Nix 0.10,
and 4.5 for Nix 0.11.)</para>
<para>If that doesn’t work, it’s time to bring out the big guns:
<screen>
$ cd /nix/var/nix
$ cp -pr db db-backup <lineannotation>(making a backup just in case)</lineannotation>
$ cd db
$ rm __db.* log* <lineannotation>(removing the Berkeley DB environment)</lineannotation>
$ mkdir tmp
$ for i in *; do db_dump $i | (cd tmp &amp;&amp; db_load $i); done
<lineannotation>(ignore error messages about non-database files like “reserved”)</lineannotation>
$ mv tmp/* .
$ nix-store --verify</screen>
</para>
</section>
<section><title>Berkeley DB out of locks</title>
<para>It is possible, especially in <command>nix-store
--verify</command> or when running the garbage collector, to run out
of Berkeley DB locks, like this:
<screen>
$ nix-store --verify
checking path existence
checking path realisability
checking the derivers table
checking the references table
Berkeley DB error: Lock table is out of available object entries
error: Db::get: Cannot allocate memory</screen>
</para>
<para>A workaround is to increase the number of locks that Berkeley DB
allocates. (The real solution would be for Nix to not use so many
locks.) This can be done by putting the following in the file
<filename>/nix/var/nix/db/<link
xlink:href="http://www.oracle.com/technology/documentation/berkeley-db/db/ref/env/db_config.html">DB_CONFIG</link></filename>:
<programlisting>
set_lk_max_locks 100000
set_lk_max_lockers 100000
set_lk_max_objects 100000
</programlisting>
(Increase these numbers if necessary.) Then make sure that there are
no running Nix processes and delete the Berkeley DB environment:
<screen>
$ rm /nix/var/nix/db/__db.*</screen>
The Berkeley DB environment is automatically recreated with the new
limits when you run any Nix command.</para>
</section>
<section><title>Collisions in <command>nix-env</command></title>
<para>Symptom: when installing or upgrading, you get an error message such as
@@ -187,7 +76,8 @@ Furthermore, the <literal>st_nlink</literal> field of the
<para>This only happens on very large Nix installations (such as build
machines).</para>
<para>Quick solution: run the garbage collector.</para>
<para>Quick solution: run the garbage collector. You may want to use
the <option>--max-links</option> option.</para>
<para>Real solution: put the Nix store on a file system that supports
more than 32,000 subdirectories per directory, such as ReiserFS.
+81 -201
View File
@@ -731,9 +731,9 @@ stdenv.mkDerivation {
xlink:href='http://www.ietf.org/rfc/rfc2396.txt'>RFC 2396</link>
can be written <emphasis>as is</emphasis>, without quotes. For
instance, the string
<literal>"https://svn.cs.uu.nl:12443/dist/trace/trace-nix-trunk.tar.bz2"</literal>
<literal>"http://example.org/foo.tar.bz2"</literal>
can also be written as
<literal>https://svn.cs.uu.nl:12443/dist/trace/trace-nix-trunk.tar.bz2</literal>.</para>
<literal>http://example.org/foo.tar.bz2</literal>.</para>
</listitem>
@@ -937,17 +937,84 @@ set.</para>
<para>Functions have the following form:
<programlisting>
{<replaceable>params</replaceable>}: <replaceable>body</replaceable></programlisting>
<replaceable>pattern</replaceable>: <replaceable>body</replaceable></programlisting>
This defines a function that must be called with an attribute set
containing the attributes listed in <replaceable>params</replaceable>,
which is a comma-separated list of attribute names. Optionally, for
each parameter a <emphasis>default value</emphasis> may be specified
by writing <literal><replaceable>param</replaceable> ?
<replaceable>e</replaceable></literal>, where
<replaceable>e</replaceable> is an arbitrary expression. If a
parameter has a default, the corresponding attribute may be omitted in
function calls.</para>
The pattern specifies what the argument of the function must look
like, and binds variables in the body to (parts of) the
argument. There are three kinds of patterns:</para>
<itemizedlist>
<listitem><para>If a pattern is a single identifier, then the
function matches any argument. Example:
<programlisting>
let negate = x: !x;
concat = x: y: x + y;
in if negate true then concat "foo" "bar" else ""</programlisting>
Note that <function>concat</function> is a function that takes one
argument and returns a function that takes another argument. This
allows partial parameterisation (i.e., only filling some of the
arguments of a function); e.g.,
<programlisting>
map (concat "foo") ["bar" "bla" "abc"]</programlisting>
evaluates to <literal>["foobar" "foobla"
"fooabc"]</literal>.</para></listitem>
<listitem><para>An <emphasis>attribute set pattern</emphasis> of the
form <literal>{name1, name2, …, nameN}</literal>
matches an attribute set containing the listed attributes, and binds
the values of those attributes to variables in the function body.
For example, the function
<programlisting>
{x, y, z}: z + y + x</programlisting>
can only be called with a set containing exactly the attributes
<varname>x</varname>, <varname>y</varname> and
<varname>z</varname>. No other attributes are allowed. If you want
to allow additional arguments, you can use an ellipsis
(<literal>...</literal>):
<programlisting>
{x, y, z, ....}: z + y + x</programlisting>
This works on any set that contains at least the three named
attributes.</para>
<para>It is possible to provide <emphasis>default values</emphasis>
for attributes, in which case they are allowed to be missing. A
default value is specified by writing
<literal><replaceable>name</replaceable> ?
<replaceable>e</replaceable></literal>, where
<replaceable>e</replaceable> is an arbitrary expression. For example,
<programlisting>
{x, y ? "foo", z ? "bar"}: z + y + x</programlisting>
specifies a function that only requires an attribute named
<varname>x</varname>, but optionally accepts <varname>y</varname>
and <varname>z</varname>.</para></listitem>
<listitem><para>An <literal>@</literal>-pattern requires that the
argument matches with the patterns on the left- and right-hand side
of the <literal>@</literal>-sign. For example:
<programlisting>
args@{x, y, z, ...}: z + y + x + args.a</programlisting>
Here <varname>args</varname> is bound to the entire argument, which
is further matches against the pattern <literal>{x, y, z,
...}</literal>.</para></listitem>
</itemizedlist>
<para>Note that functions do not have names. If you want to give them
a name, you can bind them to an attribute, e.g.,
@@ -958,31 +1025,6 @@ in concat {x = "foo"; y = "bar";}</programlisting>
</para>
<para>It is also possible to define a function that takes a single
argument and that does not need to be called with an attribute set as
argument. The syntax is
<programlisting>
<replaceable>var</replaceable>: <replaceable>body</replaceable></programlisting>
where <replaceable>var</replaceable> is the name of the argument. It
is not possible to define a default. Example:
<programlisting>
let negate = x: !x;
concat = x: y: x + y;
in if negate true then concat "foo" "bar" else ""</programlisting>
Note that <function>concat</function> is a function that takes one
arguments and returns a function that takes another argument. This
allows partial parameterisation (i.e., only filling some of the
arguments of a function); e.g.,
<programlisting>
map (concat "foo") ["bar" "bla" "abc"]</programlisting>
evaluates to <literal>["foobar" "foobla" "fooabc"]</literal>.</para>
</simplesect>
@@ -1064,7 +1106,7 @@ used in the Nix expression for Subversion.</para>
incompatibility might occur.</para>
</callout>
<callout arearefs='ex-subversion-nix-co-2'>
<callout arearefs='ex-subversion-nix-co-3'>
<para>This assertion says that in order for Subversion to have SSL
support (so that it can access <literal>https</literal> URLs), an
OpenSSL library must be passed. Additionally, it says that
@@ -1390,7 +1432,7 @@ command-line argument. See <xref linkend='sec-standard-environment'
inputs.</para></listitem>
<listitem><para>After the build, Nix sets the last-modified
timestamp on all files in the build result to 0 (00:00:00 1/1/1970
timestamp on all files in the build result to 1 (00:00:01 1/1/1970
UTC), sets the group to the default group, and sets the mode of the
file to 0444 or 0555 (i.e., read-only, with execute permission
enabled if the file was originally executable). Note that possible
@@ -1644,59 +1686,6 @@ impureEnvVars = ["http_proxy" "https_proxy" <replaceable>...</replaceable>];
<section xml:id='sec-standard-environment'><title>The standard environment</title>
<para>The standard build environment in the Nix Packages collection
provides a basic environment for building Unix packages. It consists
of the following packages:
<itemizedlist>
<listitem><para>The GNU C Compiler, configured with C and C++
support. On Linux, the compiler has been patched to provide greater
<quote>purity</quote> assurance. For instance, the compiler doesn't
search in locations such as <filename>/usr/include</filename>. In
fact, attempts to add such directories through the
<option>-I</option> flag are filtered out. Likewise, the linker
(from GNU binutils) doesn't search in standard locations such as
<filename>/usr/lib</filename>. Programs built on Linux are linked
against a GNU C Library that likewise doesn't search in the default
system locations.</para></listitem>
<listitem><para>GNU coreutils (contains a few dozen standard Unix
commands).</para></listitem>
<listitem><para>GNU findutils (contains
<command>find</command>).</para></listitem>
<listitem><para>GNU diffutils (contains <command>diff</command>,
<command>cmp</command>).</para></listitem>
<listitem><para>GNU <command>sed</command>.</para></listitem>
<listitem><para>GNU <command>grep</command>.</para></listitem>
<listitem><para>GNU <command>awk</command>.</para></listitem>
<listitem><para>GNU <command>tar</command>.</para></listitem>
<listitem><para><command>gzip</command> and
<command>bzip2</command>.</para></listitem>
<listitem><para>GNU Make. It has been patched to provide
<quote>nested</quote> output that can be fed into the
<command>nix-log2xml</command> command and
<command>log2html</command> stylesheet to create a structured,
readable output of the build steps performed by
Make.</para></listitem>
<listitem><para>Bash. This is the shell used for all builders in
the Nix Packages collection. Not using <command>/bin/sh</command>
removes a large source of portability problems.</para></listitem>
<listitem><para>Patch.</para></listitem>
</itemizedlist>
</para>
<para>The standard environment is used by passing it as an input
called <envar>stdenv</envar> to the derivation, and then doing
@@ -1765,115 +1754,6 @@ myPostInstall() {
</para>
<para>The generic builder has a number of <emphasis>phases</emphasis>,
each of which can be override in its entirety by setting the indicated
variable. The phases are:
<itemizedlist>
<listitem>
<para><function>unpackPhase</function> unpacks the source files
listed in the <envar>src</envar> environment variable to the
current directory. It supports <filename>tar</filename> files,
optionally compressed with <command>gzip</command> or
<command>bzip2</command>; Zip files (but note that the
<command>unzip</command> command is not a part of the standard
environment; you should add it as a build input yourself); and
unpacked source trees (i.e., directories; they are copied
verbatim). You can add support for other file types by setting
the <varname>findUnpacker</varname> hook. This hook should set
the variable <varname>unpackCmd</varname> to contain the command
to be executed to unpack the file.</para>
<para>After unpacking all source files,
<function>unpackPhase</function> changes the current directory to
the directory created by unpacking the sources. If there are
multiple source directories, you should set
<varname>sourceRoot</varname> to the name of the intended
directory.</para>
<para>It also calls the hook <varname>postUnpack</varname> after
unpacking.</para>
</listitem>
<listitem><para><function>patchPhase</function> calls the
<command>patch</command> command with the <option>-p1</option>
option for each patch file listed in the <envar>patches</envar>
variable.</para></listitem>
<listitem>
<para><function>configurePhase</function> runs the script called
<filename>configure</filename> in the current directory with a
<option>--prefix</option> set to the output path. You can add
additional flags through the <varname>configureFlags</varname>
variable. If <filename>configure</filename> does not exist,
nothing happens.</para>
<para>Before and after running <filename>configure</filename>, the
hooks <varname>preConfigure</varname> and
<varname>postConfigure</varname> are called, respectively.</para>
</listitem>
<listitem>
<para><function>buildPhase</function> calls
<command>make</command>. You can set flags for
<command>make</command> through the <varname>makeFlags</varname>
variable.</para>
<para>Before and after running <command>make</command>, the hooks
<varname>preBuild</varname> and <varname>postBuild</varname> are
called, respectively.</para>
</listitem>
<listitem><para><function>checkPhase</function> calls <command>make
check</command>, but only if the <varname>doCheck</varname> variable
is set to <literal>1</literal>. Additional flags can be set through
the <varname>checkFlags</varname> variable.</para></listitem>
<listitem>
<para><function>installPhase</function> calls <command>make
install</command>. Additional flags can be set through the
<varname>installFlags</varname> variable. It also strips any
static libraries in the output path of debug information unless
<varname>dontStrip</varname> is set to
<literal>1</literal>.</para>
<para>Before and after running <command>make install</command>,
the hooks <varname>preInstall</varname> and
<varname>postInstall</varname> are called, respectively.</para>
</listitem>
<listitem>
<para><function>distPhase</function> calls <command>make
dist</command>, but only if the <varname>doDist</varname> variable
is set to <literal>1</literal>. Additional flags can be set
through the <varname>distFlags</varname> variable. The resulting
tarball is copied to the <filename>/tarballs</filename>
subdirectory of the output path.</para>
<para>Before and after running <command>make dist</command>, the
hooks <varname>preDist</varname> and <varname>postDist</varname>
are called, respectively.</para>
</listitem>
</itemizedlist>
</para>
<para>You can change the order in which phases are executed, or add
new phases, by setting the <varname>phases</varname> variable. The
default is <literal>patchPhase configurePhase buildPhase checkPhase
installPhase distPhase</literal>.</para>
</section>
+8 -44
View File
@@ -1,38 +1,3 @@
# Berkeley DB
DB = db-4.5.20
$(DB).tar.gz:
@echo "Nix requires Berkeley DB to build."
@echo "Please download version 4.5.20 from"
@echo " http://download-east.oracle.com/berkeley-db/db-4.5.20.tar.gz"
@echo "and place it in the externals/ directory."
false
$(DB): $(DB).tar.gz
gunzip < $(srcdir)/$(DB).tar.gz | tar xvf -
(cd $(DB) && $(patch) -p1) < $(srcdir)/bdb-cygwin.patch
have-db:
$(MAKE) $(DB)
touch have-db
if HAVE_BDB
build-db:
else
build-db: have-db
(pfx=`pwd` && \
cd $(DB)/build_unix && \
CC="$(CC)" CXX="$(CXX)" CFLAGS="$(CFLAGS)" CXXFLAGS="$(CXXFLAGS)" \
../dist/configure --prefix=$$pfx/inst-bdb \
--enable-cxx --disable-shared --disable-cryptography \
--disable-replication --disable-verify && \
$(MAKE) && \
$(MAKE) install_include install_lib)
touch build-db
endif
# CWI ATerm
ATERM = aterm-2.4.2-fixes-r2
@@ -40,7 +5,7 @@ ATERM = aterm-2.4.2-fixes-r2
$(ATERM).tar.bz2:
@echo "Nix requires the CWI ATerm library to build."
@echo "Please download version 2.4.2-fixes-r2 from"
@echo " http://losser.st-lab.cs.uu.nl/~eelco/dist/aterm-2.4.2-fixes-r2.tar.bz2"
@echo " http://nixos.org/tarballs/aterm-2.4.2-fixes-r2.tar.bz2"
@echo "and place it in the externals/ directory."
false
@@ -67,12 +32,12 @@ endif
# bzip2
BZIP2 = bzip2-1.0.4
BZIP2 = bzip2-1.0.5
$(BZIP2).tar.gz:
@echo "Nix requires bzip2 to build."
@echo "Please download version 1.0.4 from"
@echo " http://www.bzip.org/1.0.4/bzip2-1.0.4.tar.gz"
@echo "Please download version 1.0.5 from"
@echo " http://www.bzip.org/1.0.5/bzip2-1.0.5.tar.gz"
@echo "and place it in the externals/ directory."
false
@@ -99,11 +64,10 @@ install:
endif
all: build-db build-aterm build-bzip2
all: build-aterm build-bzip2
EXTRA_DIST = $(DB).tar.gz $(ATERM).tar.bz2 $(BZIP2).tar.gz \
bdb-cygwin.patch
EXTRA_DIST = $(ATERM).tar.bz2 $(BZIP2).tar.gz
ext-clean:
$(RM) -f have-db build-db have-aterm build-aterm
$(RM) -rf $(DB) $(ATERM) $(BZIP2)
$(RM) -f have-aterm build-aterm have-bzip2 build-bzip2
$(RM) -rf $(ATERM) $(BZIP2)
-22
View File
@@ -1,22 +0,0 @@
diff -rc db-4.5.20-orig/os/os_flock.c db-4.5.20/os/os_flock.c
*** db-4.5.20-orig/os/os_flock.c 2006-10-13 12:36:12.000000000 +0200
--- db-4.5.20/os/os_flock.c 2006-10-13 12:40:11.000000000 +0200
***************
*** 30,35 ****
--- 30,44 ----
DB_ASSERT(dbenv, F_ISSET(fhp, DB_FH_OPENED) && fhp->fd != -1);
+ #ifdef __CYGWIN__
+ /*
+ * Windows file locking interferes with read/write operations, so we
+ * map the ranges to an area past the end of the file.
+ */
+ DB_ASSERT(dbenv, offset < (off_t) 1 << 62);
+ offset += (off_t) 1 << 62;
+ #endif
+
fl.l_start = offset;
fl.l_len = 1;
fl.l_type = acquire ? F_WRLCK : F_UNLCK;
Only in db-4.5.20/os: os_flock.c~
+2 -2
View File
@@ -76,10 +76,10 @@ The hook `nix-mode-hook' is run when Nix mode is started.
("\\<baseNameOf\\>" . font-lock-builtin-face)
("\\<toString\\>" . font-lock-builtin-face)
("\\<isNull\\>" . font-lock-builtin-face)
("\\<\\([a-zA-Z_][a-zA-Z0-9_']*\\)[ \t]*="
(1 font-lock-variable-name-face nil nil))
("[a-zA-Z][a-zA-Z0-9\\+-\\.]*:[a-zA-Z0-9%/\\?:@&=\\+\\$,_\\.!~\\*'-]+"
. font-lock-constant-face)
("\\<\\([a-zA-Z_][a-zA-Z0-9_'\.]*\\)[ \t]*="
(1 font-lock-variable-name-face nil nil))
("[a-zA-Z0-9._\\+-]*\\(/[a-zA-Z0-9._\\+-]+\\)+"
. font-lock-constant-face)
))
+3 -1
View File
@@ -21,12 +21,14 @@ syn match nixFuncArg "\zs\w\+\ze\s*:"
syn region nixStringParam start=+\${+ end=+}+
syn region nixMultiLineComment start=+/\*+ skip=+\\"+ end=+\*/+
syn match nixEndOfLineComment "#.*$"
syn region nixString start=+"+ skip=+\\"+ end=+"+ contains=nixStringParam
syn region nixStringIndented start=+''+ skip=+'''\|''${\|"+ end=+''+ contains=nixStringParam
syn region nixString start=+"+ skip=+\\"+ end=+"+ contains=nixStringParam
hi def link nixKeyword Keyword
hi def link nixConditional Conditional
hi def link nixBrace Special
hi def link nixString String
hi def link nixStringIndented String
hi def link nixBuiltin Special
hi def link nixStringParam Macro
hi def link nixMultiLineComment Comment
+15 -38
View File
@@ -29,25 +29,6 @@
#gc-keep-derivations = true
### Option `gc-reserved-space'
#
# This option specifies how much space should be reserved in normal
# use so that the garbage collector can run succesfully. Since the
# garbage collector must perform Berkeley DB transactions, it needs
# some disk space for itself. However, when the disk is full, this
# space is not available, so the collector would not be able to run
# precisely when it is most needed.
#
# For this reason, when Nix is run, it allocates a file
# /nix/var/nix/db/reserved of the size specified by this option. When
# the garbage collector is run, this file is deleted before the
# Berkeley DB environment is opened. This should give it enough room
# to proceed.
#
# The default is "1048576" (1 MiB).
#gc-reserved-space = 1048576
### Option `env-keep-derivations'
#
# If `false' (default), derivations are not stored in Nix user
@@ -164,30 +145,26 @@
# (using `mount --bind' on Linux) some directories from the normal
# file system hierarchy inside the chroot. These are the Nix store,
# the temporary build directory (usually /tmp/nix-<pid>-<number>) and
# the directories listed here. The default is "/dev /proc". Files
# in /dev (such as /dev/null) are needed by many builds, and some
# files in /proc may also be needed occasionally.
# the directories listed here. The default is "/dev /dev/pts /proc".
# Files in /dev (such as /dev/null) are needed by many builds, and
# some files in /proc may also be needed occasionally.
#
# Example:
# build-use-chroot = /dev /proc /bin
#build-chroot-dirs = /dev /proc
#build-chroot-dirs = /dev /dev/pts /proc
### Option `system'
### Option `build-cache-failure'
#
# This option specifies the canonical Nix system name of the current
# installation, such as `i686-linux' or `powerpc-darwin'. Nix can
# only build derivations whose `system' attribute equals the value
# specified here. In general, it never makes sense to modify this
# value from its default, since you can use it to `lie' about the
# platform you are building on (e.g., perform a Mac OS build on a
# Linux machine; the result would obviously be wrong). It only makes
# sense if the Nix binaries can run on multiple platforms, e.g.,
# `universal binaries' that run on `powerpc-darwin' and `i686-darwin'.
#
# It defaults to the canonical Nix system name detected by `configure'
# at build time.
# If this option is enabled, Nix will do negative caching; that is, it
# will remember failed builds, and won't attempt to try to build them
# again if you ask for it. Negative caching is disabled by default
# because Nix cannot distinguish between permanent build errors (e.g.,
# a syntax error in a source file) and transient build errors (e.g., a
# full disk), as they both cause the builder to return a non-zero exit
# code. You can clear the cache by doing `rm -f
# /nix/var/nix/db/failed/*'.
#
# Example:
# system = i686-darwin
#system =
# build-cache-failure = true
#build-cache-failure = false
+8 -10
View File
@@ -13,11 +13,10 @@ Version: @version@
Release: 1
License: GPL
Group: Software Deployment
URL: http://nix.cs.uu.nl/
URL: http://nixos.org/
Source0: %{name}-@version@.tar.bz2
BuildRoot: %{_tmppath}/%{name}-%{version}-buildroot
%define _prefix /nix
Prefix: %{_prefix}
Prefix: /usr
Requires: /usr/bin/perl
Requires: curl
@@ -47,14 +46,14 @@ if test -n "%{enable_setuid}"; then
extraFlags="$extraFlags --with-nix-group=%{nix_group}"
fi
fi
./configure --prefix=%{_prefix} $extraFlags
./configure --prefix=%{_prefix} --sysconfdir=/etc $extraFlags
make
make check
%install
rm -rf $RPM_BUILD_ROOT
make DESTDIR=$RPM_BUILD_ROOT install
rm $RPM_BUILD_ROOT/%{_prefix}/etc/nix/nix.conf
rm $RPM_BUILD_ROOT/etc/nix/nix.conf
strip $RPM_BUILD_ROOT/%{_prefix}/bin/* || true
%clean
@@ -76,10 +75,9 @@ fi
%{_prefix}/lib
%{_prefix}/libexec
%{_prefix}/include
%{_prefix}/var
%{_prefix}/share
%{_prefix}/store
/etc/profile.d/nix.sh
/nix/var
/nix/store
%config
%{_prefix}/etc
#%doc
#%{_prefix}/share/nix/manual
/etc/nix
+209
View File
@@ -0,0 +1,209 @@
{ nixpkgs ? ../nixpkgs }:
let
jobs = rec {
tarball =
{ nix ? {outPath = ./.; rev = 1234;}
, officialRelease ? false
}:
with import nixpkgs {};
releaseTools.sourceTarball {
name = "nix-tarball";
version = builtins.readFile ./version;
src = nix;
inherit officialRelease;
buildInputs =
[ curl bison flex2533 perl libxml2 libxslt w3m bzip2
tetex dblatex nukeReferences
];
configureFlags = ''
--with-docbook-rng=${docbook5}/xml/rng/docbook
--with-docbook-xsl=${docbook5_xsl}/xml/xsl/docbook
--with-xml-flags=--nonet
'';
# Include the ATerm and Bzip2 tarballs in the distribution.
preConfigure = ''
stripHash ${aterm242fixes.src}
cp -pv ${aterm242fixes.src} externals/$strippedName
stripHash ${bzip2.src}
cp -pv ${bzip2.src} externals/$strippedName
# TeX needs a writable font cache.
export VARTEXFONTS=$TMPDIR/texfonts
'';
preDist = ''
make -C doc/manual install prefix=$out
make -C doc/manual manual.pdf prefix=$out
cp doc/manual/manual.pdf $out/manual.pdf
# The PDF containes filenames of included graphics (see
# http://www.tug.org/pipermail/pdftex/2007-August/007290.html).
# This causes a retained dependency on dblatex, which Hydra
# doesn't like (the output of the tarball job is distributed
# to Windows and Macs, so there should be no Linux binaries
# in the closure).
nuke-refs $out/manual.pdf
echo "doc manual $out/share/doc/nix/manual" >> $out/nix-support/hydra-build-products
echo "doc-pdf manual $out/manual.pdf" >> $out/nix-support/hydra-build-products
echo "doc release-notes $out/share/doc/nix/release-notes" >> $out/nix-support/hydra-build-products
'';
};
build =
{ tarball ? jobs.tarball {}
, system ? "i686-linux"
}:
with import nixpkgs {inherit system;};
releaseTools.nixBuild {
name = "nix";
src = tarball;
buildInputs = [curl perl bzip2 openssl];
configureFlags = ''
--disable-init-state
--with-aterm=${aterm242fixes} --with-bzip2=${bzip2}
'';
};
/*
static =
{ tarball ? jobs.tarball {}
, system ? "i686-linux"
}:
with import nixpkgs {inherit system;};
releaseTools.binaryTarball {
name = "nix-static-tarball";
src = tarball;
buildInputs = [curl perl bzip2];
configureFlags = ''
--disable-init-state
--with-aterm=${aterm242fixes} --with-bzip2=${bzip2}
--enable-static-nix
'';
};
*/
coverage =
{ tarball ? jobs.tarball {}
}:
with import nixpkgs {};
releaseTools.coverageAnalysis {
name = "nix-build";
src = tarball;
buildInputs = [
curl perl bzip2 openssl
# These are for "make check" only:
graphviz libxml2 libxslt
];
configureFlags = ''
--disable-init-state --disable-shared
--with-aterm=${aterm242fixes} --with-bzip2=${bzip2}
'';
lcovFilter = ["*/boost/*" "*-tab.*"];
# We call `dot', and even though we just use it to
# syntax-check generated dot files, it still requires some
# fonts. So provide those.
FONTCONFIG_FILE = texFunctions.fontsConf;
};
rpm_fedora5i386 = makeRPM_i686 (diskImages: diskImages.fedora5i386) 10;
rpm_fedora9i386 = makeRPM_i686 (diskImages: diskImages.fedora9i386) 20;
rpm_fedora9x86_64 = makeRPM_x86_64 (diskImages: diskImages.fedora9x86_64) 20;
rpm_fedora10i386 = makeRPM_i686 (diskImages: diskImages.fedora10i386) 30;
rpm_fedora10x86_64 = makeRPM_x86_64 (diskImages: diskImages.fedora10x86_64) 30;
rpm_fedora11i386 = makeRPM_i686 (diskImages: diskImages.fedora11i386) 40;
rpm_fedora11x86_64 = makeRPM_x86_64 (diskImages: diskImages.fedora11x86_64) 40;
rpm_fedora12i386 = makeRPM_i686 (diskImages: diskImages.fedora12i386) 50;
rpm_fedora12x86_64 = makeRPM_x86_64 (diskImages: diskImages.fedora12x86_64) 50;
rpm_opensuse103i386 = makeRPM_i686 (diskImages: diskImages.opensuse103i386) 40;
rpm_opensuse110i386 = makeRPM_i686 (diskImages: diskImages.opensuse110i386) 50;
rpm_opensuse110x86_64 = makeRPM_x86_64 (diskImages: diskImages.opensuse110x86_64) 50;
deb_debian40i386 = makeDeb_i686 (diskImages: diskImages.debian40i386) 40;
deb_debian40x86_64 = makeDeb_x86_64 (diskImages: diskImages.debian40x86_64) 40;
deb_debian50i386 = makeDeb_i686 (diskImages: diskImages.debian50i386) 50;
deb_debian50x86_64 = makeDeb_x86_64 (diskImages: diskImages.debian50x86_64) 50;
deb_ubuntu804i386 = makeDeb_i686 (diskImages: diskImages.ubuntu804i386) 20;
deb_ubuntu804x86_64 = makeDeb_x86_64 (diskImages: diskImages.ubuntu804x86_64) 20;
deb_ubuntu810i386 = makeDeb_i686 (diskImages: diskImages.ubuntu810i386) 30;
deb_ubuntu810x86_64 = makeDeb_x86_64 (diskImages: diskImages.ubuntu810x86_64) 30;
deb_ubuntu904i386 = makeDeb_i686 (diskImages: diskImages.ubuntu904i386) 40;
deb_ubuntu904x86_64 = makeDeb_x86_64 (diskImages: diskImages.ubuntu904x86_64) 40;
deb_ubuntu910i386 = makeDeb_i686 (diskImages: diskImages.ubuntu910i386) 50;
deb_ubuntu910x86_64 = makeDeb_x86_64 (diskImages: diskImages.ubuntu910x86_64) 50;
};
makeRPM_i686 = makeRPM "i686-linux";
makeRPM_x86_64 = makeRPM "x86_64-linux";
makeRPM =
system: diskImageFun: prio:
{ tarball ? jobs.tarball {}
}:
with import nixpkgs {inherit system;};
releaseTools.rpmBuild rec {
name = "nix-rpm-${diskImage.name}";
src = tarball;
diskImage = diskImageFun vmTools.diskImages;
memSize = 1024;
meta = { schedulingPriority = toString prio; };
};
makeDeb_i686 = makeDeb "i686-linux";
makeDeb_x86_64 = makeDeb "x86_64-linux";
makeDeb =
system: diskImageFun: prio:
{ tarball ? jobs.tarball {}
}:
with import nixpkgs {inherit system;};
releaseTools.debBuild {
name = "nix-deb";
src = tarball;
diskImage = diskImageFun vmTools.diskImages;
memSize = 1024;
meta = { schedulingPriority = toString prio; };
configureFlags = "--sysconfdir=/etc";
debRequires = ["curl"];
};
in jobs
+11 -7
View File
@@ -1,24 +1,26 @@
bin_SCRIPTS = nix-collect-garbage \
nix-pull nix-push nix-prefetch-url \
nix-install-package nix-channel nix-build \
nix-pack-closure nix-unpack-closure \
nix-copy-closure
nix-copy-closure
noinst_SCRIPTS = nix-profile.sh generate-patches.pl \
find-runtime-roots.pl build-remote.pl
find-runtime-roots.pl build-remote.pl nix-reduce-build \
copy-from-other-stores.pl nix-http-export.cgi
nix-pull nix-push: readmanifest.pm readconfig.pm download-using-manifests.pl
install-exec-local: readmanifest.pm download-using-manifests.pl find-runtime-roots.pl
install-exec-local: readmanifest.pm download-using-manifests.pl copy-from-other-stores.pl find-runtime-roots.pl
$(INSTALL) -d $(DESTDIR)$(sysconfdir)/profile.d
$(INSTALL_PROGRAM) nix-profile.sh $(DESTDIR)$(sysconfdir)/profile.d/nix.sh
$(INSTALL) -d $(DESTDIR)$(libexecdir)/nix
$(INSTALL_DATA) readmanifest.pm $(DESTDIR)$(libexecdir)/nix
$(INSTALL_DATA) readconfig.pm $(DESTDIR)$(libexecdir)/nix
$(INSTALL_PROGRAM) download-using-manifests.pl $(DESTDIR)$(libexecdir)/nix
$(INSTALL_PROGRAM) find-runtime-roots.pl $(DESTDIR)$(libexecdir)/nix
$(INSTALL_PROGRAM) generate-patches.pl $(DESTDIR)$(libexecdir)/nix
$(INSTALL_PROGRAM) build-remote.pl $(DESTDIR)$(libexecdir)/nix
$(INSTALL) -d $(DESTDIR)$(libexecdir)/nix/substituters
$(INSTALL_PROGRAM) download-using-manifests.pl $(DESTDIR)$(libexecdir)/nix/substituters
$(INSTALL_PROGRAM) copy-from-other-stores.pl $(DESTDIR)$(libexecdir)/nix/substituters
$(INSTALL) -d $(DESTDIR)$(sysconfdir)/nix
include ../substitute.mk
@@ -31,8 +33,10 @@ EXTRA_DIST = nix-collect-garbage.in \
readconfig.pm.in \
nix-build.in \
download-using-manifests.pl.in \
copy-from-other-stores.pl.in \
generate-patches.pl.in \
nix-pack-closure.in nix-unpack-closure.in \
nix-copy-closure.in \
find-runtime-roots.pl.in \
build-remote.pl.in
build-remote.pl.in \
nix-reduce-build.in \
nix-http-export.cgi.in
+119 -71
View File
@@ -3,6 +3,7 @@
use strict;
use Fcntl ':flock';
use English '-no_match_vars';
use IO::Handle;
# General operation:
#
@@ -23,16 +24,12 @@ use English '-no_match_vars';
my $loadIncreased = 0;
my $amWilling = shift @ARGV;
my $localSystem = shift @ARGV;
my $neededSystem = shift @ARGV;
my $drvPath = shift @ARGV;
my ($amWilling, $localSystem, $neededSystem, $drvPath, $maxSilentTime) = @ARGV;
$maxSilentTime = 0 unless defined $maxSilentTime;
sub sendReply {
my $reply = shift;
open OUT, ">&3" or die;
print OUT "$reply\n";
close OUT;
print STDERR "# $reply\n";
}
sub decline {
@@ -47,30 +44,25 @@ mkdir $currentLoad, 0777 or die unless -d $currentLoad;
my $conf = $ENV{"NIX_REMOTE_SYSTEMS"};
decline if !defined $conf || ! -e $conf;
# Decline if the local system can do the build.
decline if $amWilling && ($localSystem eq $neededSystem);
# Otherwise find a willing remote machine.
my %machines;
my %systemTypes;
my %sshKeys;
my %maxJobs;
my %curJobs;
my $canBuildLocally = $amWilling && ($localSystem eq $neededSystem);
# Read the list of machines.
my @machines;
open CONF, "< $conf" or die;
while (<CONF>) {
chomp;
s/\#.*$//g;
next if /^\s*$/;
/^\s*(\S+)\s+(\S+)\s+(\S+)\s+(\d+)\s*$/ or die;
$machines{$1} = "";
$systemTypes{$1} = $2;
$sshKeys{$1} = $3;
$maxJobs{$1} = $4;
/^\s*(\S+)\s+(\S+)\s+(\S+)\s+(\d+)(\s+([0-9\.]+))?\s*$/ or die;
push @machines,
{ hostName => $1
, systemTypes => [split(/,/, $2)]
, sshKeys => $3
, maxJobs => $4
, speedFactor => 1.0 * ($6 || 1)
};
}
close CONF;
@@ -82,35 +74,56 @@ open MAINLOCK, ">>$mainLock" or die;
flock(MAINLOCK, LOCK_EX) or die;
# Find a suitable system.
sub openSlotLock {
my ($machine, $slot) = @_;
my $slotLockFn = "$currentLoad/" . (join '+', @{$machine->{systemTypes}}) . "-" . $machine->{hostName} . "-$slot";
my $slotLock = new IO::Handle;
open $slotLock, ">>$slotLockFn" or die;
return $slotLock;
}
# Find all machine that can execute this build, i.e., that support
# builds for the given platform and are not at their job limit.
my $rightType = 0;
my $machine;
LOOP: foreach my $cur (keys %machines) {
if ($neededSystem eq $systemTypes{$cur}) {
my @available = ();
LOOP: foreach my $cur (@machines) {
if (grep { $neededSystem eq $_ } @{$cur->{systemTypes}}) {
$rightType = 1;
# We have a machine of the right type. Try to get a lock on
# one of the machine's lock files.
# We have a machine of the right type. Determine the load on
# the machine.
my $slot = 0;
while ($slot < $maxJobs{$cur}) {
my $slotLock = "$currentLoad/$cur-$slot";
open SLOTLOCK, ">>$slotLock" or die;
if (flock(SLOTLOCK, LOCK_EX | LOCK_NB)) {
$machine = $cur;
last LOOP;
}
close SLOTLOCK;
my $load = 0;
my $free;
while ($slot < $cur->{maxJobs}) {
my $slotLock = openSlotLock($cur, $slot);
if (flock($slotLock, LOCK_EX | LOCK_NB)) {
$free = $slot unless defined $free;
flock($slotLock, LOCK_UN) or die;
} else {
$load++;
}
close $slotLock;
$slot++;
}
push @available, { machine => $cur, load => $load, free => $free }
if $load < $cur->{maxJobs};
}
}
close MAINLOCK;
if (defined $ENV{NIX_DEBUG_HOOK}) {
print STDERR "load on " . $_->{machine}->{hostName} . " = " . $_->{load} . "\n"
foreach @available;
}
# Didn't find one?
if (!defined $machine) {
if ($rightType) {
# Didn't find any available machine? Then decline or postpone.
if (scalar @available == 0) {
# Postpone if we have a machine of the right type, except if the
# local system can and wants to do the build.
if ($rightType && !$canBuildLocally) {
sendReply "postpone";
exit 0;
} else {
@@ -118,21 +131,52 @@ if (!defined $machine) {
}
}
# Yes we did, accept.
# Prioritise the available machines as follows:
# - First by load divided by speed factor, rounded to the nearest
# integer. This causes fast machines to be preferred over slow
# machines with similar loads.
# - Then by speed factor.
# - Finally by load.
sub lf { my $x = shift; return int($x->{load} / $x->{machine}->{speedFactor} + 0.4999); }
@available = sort
{ lf($a) <=> lf($b)
|| $b->{machine}->{speedFactor} <=> $a->{machine}->{speedFactor}
|| $a->{load} <=> $b->{load}
} @available;
# Select the best available machine and lock a free slot.
my $selected = $available[0];
my $machine = $selected->{machine};
my $slotLock = openSlotLock($machine, $selected->{free});
flock($slotLock, LOCK_EX | LOCK_NB) or die;
utime undef, undef, $slotLock;
close MAINLOCK;
# Tell Nix we've accepted the build.
sendReply "accept";
open IN, "<&4" or die;
my $x = <IN>;
if (defined $ENV{NIX_DEBUG_HOOK}) {
my $hostName = $machine->{hostName};
my $sp = $machine->{speedFactor};
print STDERR "building `$drvPath' on `$hostName' - $sp - " . $selected->{free} . "\n";
sleep 10;
exit 0;
}
my $x = <STDIN>;
chomp $x;
#print "got $x\n";
close IN;
if ($x ne "okay") {
exit 0;
}
# Do the actual job.
print "BUILDING REMOTE: $drvPath on $machine\n";
# Do the actual build.
my $hostName = $machine->{hostName};
print STDERR "building `$drvPath' on `$hostName'\n";
# Make sure that we don't get any SSH passphrase or host key popups -
# if there is any problem it should fail, not do something
@@ -141,10 +185,10 @@ $ENV{"DISPLAY"} = "";
$ENV{"SSH_PASSWORD_FILE="} = "";
$ENV{"SSH_ASKPASS="} = "";
my $sshOpts = "-i $sshKeys{$machine} -x";
my $sshOpts = "-i " . $machine->{sshKeys} . " -x";
# Hack to support Cygwin: if we login without a password, we don't
# have exactly the same right as when we do. This causes the
# have exactly the same rights as when we do. This causes the
# Microsoft C compiler to fail with certain flags:
#
# http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=99676
@@ -154,8 +198,8 @@ my $sshOpts = "-i $sshKeys{$machine} -x";
# password. (It only calls this command when there is no controlling
# terminal, but Nix ensures that is is the case. When doing this
# manually, use setsid(1).)
if ($sshKeys{$machine} =~ /^password:/) {
my $passwordFile = $sshKeys{$machine};
if ($machine->{sshKeys} =~ /^password:/) {
my $passwordFile = $machine->{sshKeys};
$passwordFile =~ s/^password://;
$sshOpts = "ssh -x";
$ENV{"SSH_PASSWORD_FILE"} = $passwordFile;
@@ -173,36 +217,40 @@ $inputs =~ s/\n/ /g;
my $outputs = `cat outputs`; die if ($? != 0);
$outputs =~ s/\n/ /g;
print "COPYING INPUTS...\n";
print "copying inputs...\n";
my $maybeSign = "";
$maybeSign = "--sign" if -e "/nix/etc/nix/signing-key.sec";
system("NIX_SSHOPTS=\"$sshOpts\" nix-copy-closure $machine $maybeSign $drvPath $inputs") == 0
or die "cannot copy inputs to $machine: $?";
system("NIX_SSHOPTS=\"$sshOpts\" @bindir@/nix-copy-closure $hostName $maybeSign $drvPath $inputs") == 0
or die "cannot copy inputs to $hostName: $?";
print "BUILDING...\n";
print "building...\n";
system("ssh $sshOpts $machine 'nix-store -rvvK $drvPath'") == 0
or die "remote build on $machine failed: $?";
my $buildFlags = "--max-silent-time $maxSilentTime";
print "REMOTE BUILD DONE: $drvPath on $machine\n";
# `-tt' forces allocation of a pseudo-terminal. This is required to
# make the remote nix-store process receive a signal when the
# connection dies. Without it, the remote process might continue to
# run indefinitely (that is, until it next tries to write to
# stdout/stderr).
if (system("ssh -tt $sshOpts $hostName 'nix-store --realise $buildFlags $drvPath > /dev/null'") != 0) {
# If we couldn't run ssh or there was an ssh problem (indicated by
# exit code 255), then we return exit code 1; otherwise we assume
# that the builder failed, which we indicated to Nix using exit
# code 100. It's important to distinguish between the two because
# the first is a transient failure and the latter is permanent.
my $res = $? == -1 || ($? >> 8) == 255 ? 1 : 100;
print STDERR "build of `$drvPath' on `$hostName' failed with exit code $?\n";
exit $res;
}
print "build of `$drvPath' on `$hostName' succeeded\n";
foreach my $output (split '\n', $outputs) {
my $maybeSignRemote = "";
$maybeSignRemote = "--sign" if $UID != 0;
system("ssh $sshOpts $machine 'nix-store --export $maybeSignRemote $output' > dump") == 0
or die "cannot copy $output from $machine: $?";
# This doesn't work yet, since the caller has a lock on the output
# path. We should move towards lock-free invocation of build
# hooks and substitutes.
#system("nix-store --import < dump") == 0
# or die "cannot import $output: $?";
# Hack: skip the first 8 bytes (the nix-store --export next
# archive marker). The archive follows.
system("(dd bs=1 count=8 of=/dev/null && cat) < dump | nix-store --restore $output") == 0
or die "cannot restore $output: $?";
system("ssh $sshOpts $hostName 'nix-store --export $maybeSignRemote $output' | @bindir@/nix-store --import > /dev/null") == 0
or die "cannot copy $output from $hostName: $?";
}
+98
View File
@@ -0,0 +1,98 @@
#! @perl@ -w
use strict;
use File::Basename;
use IO::Handle;
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
STDOUT->autoflush(1);
my @remoteStoresAll = split ':', ($ENV{"NIX_OTHER_STORES"} or "");
my @remoteStores;
foreach my $dir (@remoteStoresAll) {
push @remoteStores, glob($dir);
}
sub findStorePath {
my $storePath = shift;
my $storePathName = basename $storePath;
foreach my $store (@remoteStores) {
# Determine whether $storePath exists by looking for the
# existence of the info file, and if so, get store path info
# from that file. This rather breaks abstraction: we should
# be using `nix-store' for that. But right now there is no
# good way to tell nix-store to access a store mounted under a
# different location (there's $NIX_STORE, but that only works
# if the remote store is mounted under its "real" location).
my $infoFile = "$store/var/nix/db/info/$storePathName";
my $storePath2 = "$store/store/$storePathName";
if (-f $infoFile && -e $storePath2) {
return ($infoFile, $storePath2);
}
}
}
if ($ARGV[0] eq "--query") {
while (<STDIN>) {
my $cmd = $_; chomp $cmd;
if ($cmd eq "have") {
my $storePath = <STDIN>; chomp $storePath;
(my $infoFile) = findStorePath $storePath;
print STDOUT ($infoFile ? "1\n" : "0\n");
}
elsif ($cmd eq "info") {
my $storePath = <STDIN>; chomp $storePath;
(my $infoFile) = findStorePath $storePath;
if (!$infoFile) {
print "0\n";
next; # not an error
}
print "1\n";
my $deriver = "";
my @references = ();
open INFO, "<$infoFile" or die "cannot read info file $infoFile\n";
while (<INFO>) {
chomp;
/^([\w-]+): (.*)$/ or die "bad info file";
my $key = $1;
my $value = $2;
if ($key eq "Deriver") { $deriver = $value; }
elsif ($key eq "References") { @references = split ' ', $value; }
}
close INFO;
print "$deriver\n";
print scalar @references, "\n";
print "$_\n" foreach @references;
print "0\n"; # !!! showing size not supported (yet)
}
else { die "unknown command `$cmd'"; }
}
}
elsif ($ARGV[0] eq "--substitute") {
die unless scalar @ARGV == 2;
my $storePath = $ARGV[1];
(my $infoFile, my $sourcePath) = findStorePath $storePath;
die unless $infoFile;
print "\n*** Copying `$storePath' from `$sourcePath'\n\n";
system("$binDir/nix-store --dump $sourcePath | $binDir/nix-store --restore $storePath") == 0
or die "cannot copy `$sourcePath' to `$storePath'";
}
else { die; }
+102 -62
View File
@@ -5,20 +5,13 @@ use readmanifest;
use POSIX qw(strftime);
use File::Temp qw(tempdir);
my $manifestDir = "@localstatedir@/nix/manifests";
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
STDOUT->autoflush(1);
my $manifestDir = ($ENV{"NIX_MANIFESTS_DIR"} or "@localstatedir@/nix/manifests");
my $logFile = "@localstatedir@/log/nix/downloads";
open LOGFILE, ">>$logFile" or die "cannot open log file $logFile";
# Create a temporary directory.
my $tmpDir = tempdir("nix-download.XXXXXX", CLEANUP => 1, TMPDIR => 1)
or die "cannot create a temporary directory";
chdir $tmpDir or die "cannot change to `$tmpDir': $!";
my $tmpNar = "$tmpDir/nar";
my $tmpNar2 = "$tmpDir/nar2";
# Load all manifests.
my %narFiles;
@@ -26,44 +19,56 @@ my %localPaths;
my %patches;
for my $manifest (glob "$manifestDir/*.nixmanifest") {
# print STDERR "reading $manifest\n";
if (readManifest($manifest, \%narFiles, \%localPaths, \%patches) < 3) {
my $version = readManifest($manifest, \%narFiles, \%localPaths, \%patches);
if ($version < 3) {
print STDERR "you have an old-style manifest `$manifest'; please delete it\n";
exit 1;
}
if ($version >= 10) {
print STDERR "manifest `$manifest' is too new; please delete it or upgrade Nix\n";
exit 1;
}
}
# Parse the arguments.
if ($ARGV[0] eq "--query-paths") {
foreach my $storePath (keys %narFiles) { print "$storePath\n"; }
foreach my $storePath (keys %localPaths) { print "$storePath\n"; }
exit 0;
}
if ($ARGV[0] eq "--query") {
elsif ($ARGV[0] eq "--query-info") {
shift @ARGV;
foreach my $storePath (@ARGV) {
my $info;
if (defined $narFiles{$storePath}) {
$info = @{$narFiles{$storePath}}[0];
while (<STDIN>) {
my $cmd = $_; chomp $cmd;
if ($cmd eq "have") {
my $storePath = <STDIN>; chomp $storePath;
print STDOUT ((defined $narFiles{$storePath} or defined $localPaths{$storePath})
? "1\n" : "0\n");
}
elsif (defined $localPaths{$storePath}) {
$info = @{$localPaths{$storePath}}[0];
}
else {
next; # not an error
}
print "$storePath\n";
print "$info->{deriver}\n";
my @references = split " ", $info->{references};
my $count = scalar @references;
print "$count\n";
foreach my $reference (@references) {
print "$reference\n";
elsif ($cmd eq "info") {
my $storePath = <STDIN>; chomp $storePath;
my $info;
if (defined $narFiles{$storePath}) {
$info = @{$narFiles{$storePath}}[0];
}
elsif (defined $localPaths{$storePath}) {
$info = @{$localPaths{$storePath}}[0];
}
else {
print "0\n";
next; # not an error
}
print "1\n";
print "$info->{deriver}\n";
my @references = split " ", $info->{references};
print scalar @references, "\n";
print "$_\n" foreach @references;
my $size = $info->{size} || 0;
print "$size\n";
}
else { die "unknown command `$cmd'"; }
}
exit 0;
}
@@ -76,6 +81,18 @@ die unless scalar @ARGV == 2;
my $targetPath = $ARGV[1];
# Create a temporary directory.
my $tmpDir = tempdir("nix-download.XXXXXX", CLEANUP => 1, TMPDIR => 1)
or die "cannot create a temporary directory";
chdir $tmpDir or die "cannot change to `$tmpDir': $!";
my $tmpNar = "$tmpDir/nar";
my $tmpNar2 = "$tmpDir/nar2";
open LOGFILE, ">>$logFile" or die "cannot open log file $logFile";
my $date = strftime ("%F %H:%M:%S UTC", gmtime (time));
print LOGFILE "$$ get $targetPath $date\n";
@@ -88,7 +105,7 @@ foreach my $localPath (@{$localPathList}) {
my $sourcePath = $localPath->{copyFrom};
if (-e $sourcePath) {
print "\n*** Step 1/1: copying from $sourcePath\n";
system("@bindir@/nix-store --dump $sourcePath | @bindir@/nix-store --restore $targetPath") == 0
system("$binDir/nix-store --dump $sourcePath | $binDir/nix-store --restore $targetPath") == 0
or die "cannot copy `$sourcePath' to `$targetPath'";
exit 0;
}
@@ -137,7 +154,7 @@ addToQueue $targetPath;
sub isValidPath {
my $p = shift;
return system("@bindir@/nix-store --check-validity '$p' 2> /dev/null") == 0;
return system("$binDir/nix-store --check-validity '$p' 2> /dev/null") == 0;
}
sub parseHash {
@@ -171,15 +188,13 @@ while ($queueFront < scalar @queue) {
my ($baseHashAlgo, $baseHash) = parseHash $patch->{baseHash};
my $format = "--base32";
$format = "" if $baseHashAlgo eq "md5";
my $hash = `@bindir@/nix-hash --type '$baseHashAlgo' $format "$patch->{basePath}"`;
my $hash = `$binDir/nix-hash --type '$baseHashAlgo' $format "$patch->{basePath}"`;
chomp $hash;
# print " MY HASH is $hash\n";
if ($hash ne $baseHash) {
print LOGFILE "$$ rejecting $patch->{basePath}\n";
next;
}
}
# print " PATCH from $patch->{basePath}\n";
addToQueue $patch->{basePath};
addEdge $patch->{basePath}, $u, $patch->{size}, "patch", $patch;
}
@@ -187,10 +202,12 @@ while ($queueFront < scalar @queue) {
# Add NAR file edges to the start node.
my $narFileList = $narFiles{$u};
foreach my $narFile (@{$narFileList}) {
# print " NAR from $narFile->{url}\n";
addEdge "start", $u, $narFile->{size}, "narfile", $narFile;
# !!! how to handle files whose size is not known in advance?
# For now, assume some arbitrary size (1 MB).
addEdge "start", $u, ($narFile->{size} || 1000000), "narfile", $narFile;
if ($u eq $targetPath) {
print LOGFILE "$$ full-download-would-be $narFile->{size}\n";
my $size = $narFile->{size} || -1;
print LOGFILE "$$ full-download-would-be $size\n";
}
}
@@ -216,8 +233,6 @@ while (scalar @todo > 0) {
my $u_ = $graph{$u};
# print "IN $u $u_->{d}\n";
foreach my $edge (@{$u_->{edges}}) {
my $v_ = $graph{$edge->{end}};
if ($v_->{d} > $u_->{d} + $edge->{weight}) {
@@ -225,7 +240,6 @@ while (scalar @todo > 0) {
# Store the edge; to edge->start is actually the
# predecessor.
$v_->{pred} = $edge;
# print " RELAX $edge->{end} $v_->{d}\n";
}
}
}
@@ -247,20 +261,18 @@ while ($cur ne "start") {
my $curStep = 1;
my $maxStep = scalar @path;
sub downloadFile {
my $url = shift;
my ($hashAlgo, $hash) = parseHash(shift);
sub downloadFile {
my $url = shift;
$ENV{"PRINT_PATH"} = 1;
$ENV{"QUIET"} = 1;
$ENV{"NIX_HASH_ALGO"} = $hashAlgo;
my ($hash2, $path) = `@bindir@/nix-prefetch-url '$url' '$hash'`;
my ($hash, $path) = `$binDir/nix-prefetch-url '$url'`;
die "download of `$url' failed" unless $? == 0;
chomp $hash2;
chomp $path;
die "hash mismatch, expected $hash, got $hash2" if $hash ne $hash2;
return $path;
}
my $finalNarHash;
while (scalar @path > 0) {
my $edge = pop @path;
my $u = $edge->{start};
@@ -277,7 +289,7 @@ while (scalar @path > 0) {
# as a base to one or more patches. So turn the base path
# into a NAR archive, to which we can apply the patch.
print " packing base path...\n";
system("@bindir@/nix-store --dump $v > $tmpNar") == 0
system("$binDir/nix-store --dump $v > $tmpNar") == 0
or die "cannot dump `$v'";
}
}
@@ -290,7 +302,7 @@ while (scalar @path > 0) {
# Download the patch.
print " downloading patch...\n";
my $patchPath = downloadFile "$patch->{url}", "$patch->{hash}";
my $patchPath = downloadFile "$patch->{url}";
# Apply the patch to the NAR archive produced in step 1 (for
# the already present path) or a later step (for patch sequences).
@@ -305,20 +317,23 @@ while (scalar @path > 0) {
# This was the last patch. Unpack the final NAR archive
# into the target path.
print " unpacking patched archive...\n";
system("@bindir@/nix-store --restore $v < $tmpNar2") == 0
system("$binDir/nix-store --restore $v < $tmpNar2") == 0
or die "cannot unpack $tmpNar2 into `$v'";
}
$finalNarHash = $patch->{narHash};
}
elsif ($edge->{type} eq "narfile") {
my $narFile = $edge->{info};
print "downloading `$narFile->{url}' into `$v'\n";
print LOGFILE "$$ narfile $narFile->{url} $narFile->{size} $v\n";
my $size = $narFile->{size} || -1;
print LOGFILE "$$ narfile $narFile->{url} $size $v\n";
# Download the archive.
print " downloading archive...\n";
my $narFilePath = downloadFile "$narFile->{url}", "$narFile->{hash}";
my $narFilePath = downloadFile "$narFile->{url}";
if ($curStep < $maxStep) {
# The archive will be used a base to a patch.
@@ -327,14 +342,39 @@ while (scalar @path > 0) {
} else {
# Unpack the archive into the target path.
print " unpacking archive...\n";
system("@bunzip2@ < '$narFilePath' | @bindir@/nix-store --restore '$v'") == 0
system("@bunzip2@ < '$narFilePath' | $binDir/nix-store --restore '$v'") == 0
or die "cannot unpack `$narFilePath' into `$v'";
}
$finalNarHash = $narFile->{narHash};
}
$curStep++;
}
# Make sure that the hash declared in the manifest matches what we
# downloaded and unpacked.
if (defined $finalNarHash) {
my ($hashAlgo, $hash) = parseHash $finalNarHash;
# The hash in the manifest can be either in base-16 or base-32.
# Handle both.
my $extraFlag =
($hashAlgo eq "sha256" && length($hash) != 64)
? "--base32" : "";
my $hash2 = `@bindir@/nix-hash --type $hashAlgo $extraFlag $targetPath`
or die "cannot compute hash of path `$targetPath'";
chomp $hash2;
die "hash mismatch in downloaded path $targetPath; expected $hash, got $hash2"
if $hash ne $hash2;
} else {
die "cannot check integrity of the downloaded path since its hash is not known";
}
print LOGFILE "$$ success\n";
close LOGFILE;
+49 -7
View File
@@ -2,6 +2,8 @@
use strict;
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
my $addDrvLink = 0;
my $addOutLink = 1;
@@ -9,6 +11,9 @@ my $addOutLink = 1;
my $outLink;
my $drvLink;
my $dryRun = 0;
my $verbose = 0;
my @instArgs = ();
my @buildArgs = ();
my @exprs = ();
@@ -73,7 +78,7 @@ EOF
elsif ($arg eq "--attr" or $arg eq "-A") {
$n++;
die "$0: `--attr' requires an argument\n" unless $n < scalar @ARGV;
die "$0: `$arg' requires an argument\n" unless $n < scalar @ARGV;
push @instArgs, ("--attr", $ARGV[$n]);
}
@@ -83,12 +88,41 @@ EOF
$n += 2;
}
elsif ($arg eq "--max-jobs" or $arg eq "-j" or $arg eq "--max-silent-time") {
elsif ($arg eq "--log-type") {
$n++;
die "$0: `$arg' requires an argument\n" unless $n < scalar @ARGV;
push @instArgs, ($arg, $ARGV[$n]);
push @buildArgs, ($arg, $ARGV[$n]);
}
elsif ($arg eq "--option") {
die "$0: `$arg' requires two arguments\n" unless $n + 2 < scalar @ARGV;
push @instArgs, ($arg, $ARGV[$n + 1], $ARGV[$n + 2]);
push @buildArgs, ($arg, $ARGV[$n + 1], $ARGV[$n + 2]);
$n += 2;
}
elsif ($arg eq "--max-jobs" or $arg eq "-j" or $arg eq "--max-silent-time" or $arg eq "--log-type") {
$n++;
die "$0: `$arg' requires an argument\n" unless $n < scalar @ARGV;
push @buildArgs, ($arg, $ARGV[$n]);
}
elsif ($arg eq "--dry-run") {
push @buildArgs, "--dry-run";
$dryRun = 1;
}
elsif ($arg eq "--show-trace") {
push @instArgs, $arg;
}
elsif ($arg eq "--verbose" or substr($arg, 0, 2) eq "-v") {
push @buildArgs, $arg;
push @instArgs, $arg;
$verbose = 1;
}
elsif (substr($arg, 0, 1) eq "-") {
push @buildArgs, $arg;
}
@@ -117,21 +151,29 @@ foreach my $expr (@exprs) {
# Instantiate.
my @drvPaths;
# !!! would prefer the perl 5.8.0 pipe open feature here.
my $pid = open(DRVPATHS, "-|") || exec "@bindir@/nix-instantiate", "--add-root", $drvLink, "--indirect", @instArgs, $expr;
my $pid = open(DRVPATHS, "-|") || exec "$binDir/nix-instantiate", "--add-root", $drvLink, "--indirect", @instArgs, $expr;
while (<DRVPATHS>) {chomp; push @drvPaths, $_;}
close DRVPATHS or exit 1;
if (!close DRVPATHS) {
die "nix-instantiate killed by signal " . ($? & 127) . "\n" if ($? & 127);
exit 1;
}
foreach my $drvPath (@drvPaths) {
my $target = readlink $drvPath or die "cannot read symlink `$drvPath'";
print STDERR "store derivation is $target\n";
print STDERR "derivation is $target\n" if $verbose;
}
# Build.
my @outPaths;
$pid = open(OUTPATHS, "-|") || exec "@bindir@/nix-store", "--add-root", $outLink, "--indirect", "-rv",
$pid = open(OUTPATHS, "-|") || exec "$binDir/nix-store", "--add-root", $outLink, "--indirect", "-rv",
@buildArgs, @drvPaths;
while (<OUTPATHS>) {chomp; push @outPaths, $_;}
close OUTPATHS or exit 1;
if (!close OUTPATHS) {
die "nix-store killed by signal " . ($? & 127) . "\n" if ($? & 127);
exit 1;
}
next if $dryRun;
foreach my $outPath (@outPaths) {
my $target = readlink $outPath or die "cannot read symlink `$outPath'";
+9 -13
View File
@@ -78,6 +78,9 @@ sub removeChannel {
sub update {
readChannels;
# Create the manifests directory if it doesn't exist.
mkdir "$stateDir/manifests", 0755 unless -e "$stateDir/manifests";
# Do we have write permission to the manifests directory? If not,
# then just skip pulling the manifest and just download the Nix
# expressions. If the user is a non-privileged user in a
@@ -85,11 +88,6 @@ sub update {
# source.
if (-W "$stateDir/manifests") {
# Remove all the old manifests.
for my $manifest (glob "$stateDir/manifests/*.nixmanifest") {
unlink $manifest or die "cannot remove `$manifest': $!";
}
# Pull cache manifests.
foreach my $url (@channels) {
#print "pulling cache manifest from `$url'\n";
@@ -125,15 +123,13 @@ sub update {
my $rootFile = "$rootsDir/per-user/$userName/channels";
# Instantiate the Nix expression.
# Build the Nix expression.
print "unpacking channel Nix expressions...\n";
my $storeExpr = `@bindir@/nix-instantiate --add-root '$rootFile'.tmp @datadir@/nix/corepkgs/channels/unpack.nix --argstr system @system@ --arg inputs '$inputs'`
or die "cannot instantiate Nix expression";
chomp $storeExpr;
# Build the resulting derivation.
my $outPath = `@bindir@/nix-store --add-root '$rootFile' -r '$storeExpr'`
or die "cannot realise store expression";
my $outPath = `\\
@bindir@/nix-build --out-link '$rootFile' --drv-link '$rootFile'.tmp \\
@datadir@/nix/corepkgs/channels/unpack.nix \\
--argstr system @system@ --arg inputs '$inputs'`
or die "cannot unpack the channels";
chomp $outPath;
unlink "$rootFile.tmp";
+4 -2
View File
@@ -4,6 +4,8 @@ use strict;
my $profilesDir = "@localstatedir@/nix/profiles";
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
# Process the command line arguments.
my @args = ();
@@ -34,7 +36,7 @@ sub removeOldGenerations {
$name = $dir . "/" . $name;
if (-l $name && (readlink($name) =~ /link/)) {
print STDERR "removing old generations of profile $name\n";
system("@bindir@/nix-env", "-p", $name, "--delete-generations", "old");
system("$binDir/nix-env", "-p", $name, "--delete-generations", "old");
}
elsif (! -l $name && -d $name) {
removeOldGenerations $name;
@@ -48,4 +50,4 @@ removeOldGenerations $profilesDir if $removeOld;
# Run the actual garbage collector.
exec "@bindir@/nix-store", "--gc", @args;
exec "$binDir/nix-store", "--gc", @args;
+10 -27
View File
@@ -1,7 +1,6 @@
#! @perl@ -w
my $binDir = $ENV{"NIX_BIN_DIR"};
$binDir = "@bindir@" unless defined $binDir;
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
if (scalar @ARGV < 1) {
@@ -56,30 +55,18 @@ while (@ARGV) {
if ($toMode) { # Copy TO the remote machine.
my @allStorePaths;
my %storePathsSeen;
foreach my $storePath (@storePaths) {
# $arg might be a symlink to the store, so resolve it.
my $storePath2 = (`$binDir/nix-store --query --resolve '$storePath'`
or die "cannot resolve `$storePath'");
chomp $storePath2;
# Get the closure of this path.
my $pid = open(READ,
"$binDir/nix-store --query --requisites '$storePath2'|") or die;
# Get the closure of this path.
my $pid = open(READ, "$binDir/nix-store --query --requisites @storePaths|") or die;
while (<READ>) {
chomp;
die "bad: $_" unless /^\//;
if (!defined $storePathsSeen{$_}) {
push @allStorePaths, $_;
$storePathsSeen{$_} = 1;
}
}
close READ or die "nix-store failed: $?";
while (<READ>) {
chomp;
die "bad: $_" unless /^\//;
push @allStorePaths, $_;
}
close READ or die "nix-store failed: $?";
# Ask the remote host which paths are invalid.
open(READ, "ssh @sshOpts $sshHost nix-store --check-validity --print-invalid @allStorePaths|");
@@ -112,15 +99,11 @@ else { # Copy FROM the remote machine.
"ssh @sshOpts $sshHost nix-store --query --requisites @storePaths|") or die;
my @allStorePaths;
my %storePathsSeen;
while (<READ>) {
chomp;
die "bad: $_" unless /^\//;
if (!defined $storePathsSeen{$_}) {
push @allStorePaths, $_;
$storePathsSeen{$_} = 1;
}
push @allStorePaths, $_;
}
close READ or die "nix-store on remote machine `$sshHost' failed: $?";
+51
View File
@@ -0,0 +1,51 @@
#! /bin/sh
export HOME=/tmp
export NIX_REMOTE=daemon
TMP_DIR="${TMP_DIR:-/tmp/nix-export}"
@coreutils@/mkdir -p "$TMP_DIR" || true
@coreutils@/chmod a+r "$TMP_DIR"
needed_path="?$QUERY_STRING"
needed_path="${needed_path#*[?&]needed_path=}"
needed_path="${needed_path%%&*}"
#needed_path="$(echo $needed_path | ./unhttp)"
needed_path="${needed_path//%2B/+}"
needed_path="${needed_path//%3D/=}"
echo needed_path: "$needed_path" >&2
NIX_STORE="${NIX_STORE_DIR:-/nix/store}"
echo NIX_STORE: "${NIX_STORE}" >&2
full_path="${NIX_STORE}"/"$needed_path"
if [ "$needed_path" != "${needed_path%.drv}" ]; then
echo "Status: 403 You should create the derivation file yourself"
echo "Content-Type: text/plain"
echo
echo "Refusing to disclose derivation contents"
exit
fi
if @bindir@/nix-store --check-validity "$full_path"; then
if ! [ -e nix-export/"$needed_path".nar.gz ]; then
@bindir@/nix-store --export "$full_path" | @gzip@ > "$TMP_DIR"/"$needed_path".nar.gz
@coreutils@/ln -fs "$TMP_DIR"/"$needed_path".nar.gz nix-export/"$needed_path".nar.gz
fi;
echo "Status: 301 Moved"
echo "Location: nix-export/"$needed_path".nar.gz"
echo
else
echo "Status: 404 No such path found"
echo "Content-Type: text/plain"
echo
echo "Path not found:"
echo "$needed_path"
echo "checked:"
echo "$full_path"
fi
+10 -3
View File
@@ -3,6 +3,8 @@
use strict;
use File::Temp qw(tempdir);
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
sub usageError {
print STDERR <<EOF;
@@ -59,7 +61,7 @@ if ($interactive && !defined $ENV{"NIX_HAVE_TERMINAL"}) {
$ENV{"NIX_HAVE_TERMINAL"} = "1";
$ENV{"LD_LIBRARY_PATH"} = "";
foreach my $term ("xterm", "konsole", "gnome-terminal", "xterm") {
exec($term, "-e", "@bindir@/nix-install-package", @ARGV);
exec($term, "-e", "$binDir/nix-install-package", @ARGV);
}
die "cannot execute `xterm'";
}
@@ -121,13 +123,18 @@ if ($interactive) {
}
# Store the manifest in the temporary directory so that we don't
# pollute /nix/var/nix/manifests.
$ENV{NIX_MANIFESTS_DIR} = $tmpDir;
print "\nPulling manifests...\n";
system("@bindir@/nix-pull", $manifestURL) == 0
system("$binDir/nix-pull", $manifestURL) == 0
or barf "nix-pull failed: $?";
print "\nInstalling package...\n";
system("@bindir@/nix-env", "--install", $outPath, "--force-name", $drvName, @extraNixEnvArgs) == 0
system("$binDir/nix-env", "--install", $outPath, "--force-name", $drvName, @extraNixEnvArgs) == 0
or barf "nix-env failed: $?";
-75
View File
@@ -1,75 +0,0 @@
#! @perl@ -w
# This tool computes the closure of a path (using "nix-store --query
# --requisites") and puts the contents of each path in the closure in
# a big NAR archive that can be installed on another Nix installation
# using "nix-unpack-closure".
# TODO: make this program "streamy", i.e., don't use a temporary
# directory.
use strict;
use File::Temp qw(tempdir);
my $binDir = $ENV{"NIX_BIN_DIR"};
$binDir = "@bindir@" unless defined $binDir;
my $tmpDir = tempdir("nix-pack-closure.XXXXXX", CLEANUP => 1, TMPDIR => 1)
or die "cannot create a temporary directory";
mkdir "$tmpDir/contents", 0755 or die;
mkdir "$tmpDir/references", 0755 or die;
mkdir "$tmpDir/derivers", 0755 or die;
open TOPLEVEL, ">$tmpDir/top-level" or die;
my %storePaths;
while (@ARGV) {
my $storePath = shift @ARGV;
# $storePath might be a symlink to the store, so resolve it.
$storePath = (`$binDir/nix-store --query --resolve '$storePath'`
or die "cannot resolve `$storePath'");
chomp $storePath;
print TOPLEVEL $storePath, "\n";
# Get the closure of this path.
my $pid = open(READ,
"$binDir/nix-store --query --requisites '$storePath'|") or die;
while (<READ>) {
chomp;
die "bad: $_" unless /^\//;
$storePaths{$_} = "";
}
close READ or die "nix-store failed: $?";
}
close TOPLEVEL or die;
foreach my $storePath (sort(keys %storePaths)) {
print STDERR "packing `$storePath'...\n";
$storePath =~ /\/([^\/]+)$/;
my $name = $1;
system("$binDir/nix-store --dump '$storePath' > $tmpDir/contents/$name") == 0
or die "nix-store --dump failed on `$storePath': $?";
system("$binDir/nix-store --query --references '$storePath' > $tmpDir/references/$name") == 0
or die "nix-store --query --references failed on `$storePath': $?";
system("$binDir/nix-store --query --deriver '$storePath' > $tmpDir/derivers/$name") == 0
or die "nix-store --query --deriver failed on `$storePath': $?";
}
# Write a NAR archive of everything to standard output.
system("nix-store --dump '$tmpDir'") == 0
or die "nix-store --dump failed";
+24 -2
View File
@@ -37,6 +37,7 @@ fi
mkTempDir() {
if test -n "$tmpPath"; then return; fi
local i=0
while true; do
if test -z "$TMPDIR"; then TMPDIR=/tmp; fi
@@ -59,11 +60,32 @@ removeTempDir() {
doDownload() {
@curl@ $cacheFlags --fail -# --location --max-redirs 20 --disable-epsv \
@curl@ $cacheFlags --fail --location --max-redirs 20 --disable-epsv \
--cookie-jar $tmpPath/cookies "$url" -o $tmpFile
}
# Hack to support the mirror:// scheme from Nixpkgs.
if test "${url:0:9}" = "mirror://"; then
if test -z "$NIXPKGS_ALL"; then
echo "Resolving mirror:// URLs requires Nixpkgs. Please point \$NIXPKGS_ALL at a Nixpkgs tree." >&2
exit 1
fi
mkTempDir
nix-build "$NIXPKGS_ALL" -A resolveMirrorURLs --argstr url "$url" -o $tmpPath/urls > /dev/null
expanded=($(cat $tmpPath/urls))
if test "${#expanded[*]}" = 0; then
echo "$0: cannot resolve $url." >&2
exit 1
fi
echo "$url expands to ${expanded[*]} (using ${expanded[0]})" >&2
url="${expanded[0]}"
fi
# If we don't know the hash or a file with that hash doesn't exist,
# download the file and add it to the store.
if test -z "$finalPath"; then
@@ -78,7 +100,7 @@ if test -z "$finalPath"; then
# garbage-collected independently.
if test -n "$NIX_DOWNLOAD_CACHE"; then
echo -n "$url" > $tmpPath/url
urlHash=$(nix-hash --type sha256 --base32 --flat $tmpPath/url)
urlHash=$(@bindir@/nix-hash --type sha256 --base32 --flat $tmpPath/url)
echo "$url" > "$NIX_DOWNLOAD_CACHE/$urlHash.url"
cachedHashFN="$NIX_DOWNLOAD_CACHE/$urlHash.$hashType"
cachedTimestampFN="$NIX_DOWNLOAD_CACHE/$urlHash.stamp"
+8 -1
View File
@@ -7,5 +7,12 @@ if test -n "$HOME"; then
@coreutils@/ln -s "$_NIX_DEF_LINK" "$NIX_LINK"
fi
export PATH=$NIX_LINK/bin:@prefix@/bin:$PATH
export PATH=$NIX_LINK/bin:$PATH
fi
# This is a quick hack to make fontconfig-based packages in Nixpkgs
# work out of the box on non-NixOS systems. Of course, we should
# really fix fontconfig...
if test -z "$FONTCONFIG_FILE" -a -e /etc/fonts/fonts.conf; then
export FONTCONFIG_FILE=/etc/fonts/fonts.conf
fi
+39 -18
View File
@@ -7,23 +7,23 @@ use readmanifest;
my $tmpDir = tempdir("nix-pull.XXXXXX", CLEANUP => 1, TMPDIR => 1)
or die "cannot create a temporary directory";
my $binDir = $ENV{"NIX_BIN_DIR"};
$binDir = "@bindir@" unless defined $binDir;
my $libexecDir = $ENV{"NIX_LIBEXEC_DIR"};
$libexecDir = "@libexecdir@" unless defined $libexecDir;
my $stateDir = $ENV{"NIX_STATE_DIR"};
$stateDir = "@localstatedir@/nix" unless defined $stateDir;
my $storeDir = $ENV{"NIX_STORE_DIR"};
$storeDir = "@storedir@" unless defined $storeDir;
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
my $libexecDir = ($ENV{"NIX_LIBEXEC_DIR"} or "@libexecdir@");
my $storeDir = ($ENV{"NIX_STORE_DIR"} or "@storedir@");
my $stateDir = ($ENV{"NIX_STATE_DIR"} or "@localstatedir@/nix");
my $manifestDir = ($ENV{"NIX_MANIFESTS_DIR"} or "$stateDir/manifests");
# Prevent access problems in shared-stored installations.
umask 0022;
# Create the manifests directory if it doesn't exist.
if (! -e $manifestDir) {
mkdir $manifestDir, 0755 or die "cannot create directory `$manifestDir'";
}
# Process the URLs specified on the command line.
my %narFiles;
my %localPaths;
@@ -51,7 +51,7 @@ sub processURL {
# First see if a bzipped manifest is available.
if (system("@curl@ --fail --silent --head '$url'.bz2 > /dev/null") == 0) {
print "obtaining list of Nix archives at `$url.bz2'...\n";
print "fetching list of Nix archives at `$url.bz2'...\n";
my $bzipped = downloadFile "$url.bz2";
$manifest = "$tmpDir/MANIFEST";
@@ -69,10 +69,11 @@ sub processURL {
print "obtaining list of Nix archives at `$url'...\n";
$manifest = downloadFile $url;
}
my $version = readManifest($manifest, \%narFiles, \%localPaths, \%patches);
if (readManifest($manifest, \%narFiles, \%localPaths, \%patches) < 3) {
die "`$url' is not manifest or it is too old (i.e., for Nix <= 0.7)\n";
}
die "`$url' is not a manifest or it is too old (i.e., for Nix <= 0.7)\n" if $version < 3;
die "manifest `$url' is too new\n" if $version >= 5;
if ($skipWrongStore) {
foreach my $path (keys %narFiles) {
@@ -91,11 +92,31 @@ sub processURL {
my $hash = `$binDir/nix-hash --flat '$manifest'`
or die "cannot hash `$manifest'";
chomp $hash;
my $urlFile = "$manifestDir/$baseName-$hash.url";
open URL, ">$urlFile" or die "cannot create `$urlFile'";
print URL "$url";
close URL;
my $finalPath = "$stateDir/manifests/$baseName-$hash.nixmanifest";
system("@coreutils@/ln", "-sfn", "$manifest", "$finalPath") == 0
my $finalPath = "$manifestDir/$baseName-$hash.nixmanifest";
unlink $finalPath if -e $finalPath;
symlink("$manifest", "$finalPath")
or die "cannot link `$finalPath to `$manifest'";
# Delete all old manifests downloaded from this URL.
for my $urlFile2 (glob "$manifestDir/*.url") {
next if $urlFile eq $urlFile2;
open URL, "<$urlFile2" or die;
my $url2 = <URL>;
chomp $url2;
close URL;
next unless $url eq $url2;
my $base = $urlFile2; $base =~ s/.url$//;
unlink "${base}.url";
unlink "${base}.nixmanifest";
}
}
while (@ARGV) {
+13 -15
View File
@@ -16,8 +16,7 @@ my $curl = "@curl@ --fail --silent";
my $extraCurlFlags = ${ENV{'CURL_FLAGS'}};
$curl = "$curl $extraCurlFlags" if defined $extraCurlFlags;
my $binDir = $ENV{"NIX_BIN_DIR"};
$binDir = "@bindir@" unless defined $binDir;
my $binDir = $ENV{"NIX_BIN_DIR"} || "@bindir@";
my $dataDir = $ENV{"NIX_DATA_DIR"};
$dataDir = "@datadir@" unless defined $dataDir;
@@ -107,7 +106,7 @@ foreach my $storePath (@storePaths) {
# Construct a Nix expression that creates a Nix archive.
my $nixexpr =
"((import $dataDir/nix/corepkgs/nar/nar.nix) " .
"{storePath = builtins.toPath \"$storePath\"; system = \"@system@\"; hashAlgo = \"$hashAlgo\";}) ";
"{storePath = builtins.storePath \"$storePath\"; system = \"@system@\"; hashAlgo = \"$hashAlgo\";}) ";
print NIX $nixexpr;
}
@@ -129,7 +128,7 @@ while (<READ>) {
close READ or die "nix-instantiate failed: $?";
# Realise the store expressions.
# Build the derivations.
print STDERR "creating archives...\n";
my @narPaths;
@@ -141,12 +140,7 @@ while (scalar @tmp > 0) {
my @tmp2 = @tmp[0..$n - 1];
@tmp = @tmp[$n..scalar @tmp - 1];
# Note: we disable build hooks because of the impure path
# reference (see above). Even if that is fixed, using a hook
# probably wouldn't make that much sense; pumping lots of data
# around just to compress them won't gain that much.
$ENV{"NIX_BUILD_HOOK"} = "";
my $pid = open(READ, "$binDir/nix-store --no-build-hook --realise @tmp2|")
my $pid = open(READ, "$binDir/nix-store --realise @tmp2|")
or die "cannot run nix-store";
while (<READ>) {
chomp;
@@ -224,13 +218,14 @@ writeManifest $manifest, \%narFiles, \%patches;
sub copyFile {
my $src = shift;
my $dst = shift;
system("@coreutils@/cp", $src, "$dst.tmp") == 0 or die "cannot copy file";
rename("$dst.tmp", "$dst") or die "cannot rename file";
my $tmp = "$dst.tmp.$$";
system("@coreutils@/cp", $src, $tmp) == 0 or die "cannot copy file";
rename($tmp, $dst) or die "cannot rename file: $!";
}
# Upload the archives.
print STDERR "uploading archives...\n";
# Upload/copy the archives.
print STDERR "uploading/copying archives...\n";
sub archiveExists {
my $name = shift;
@@ -244,9 +239,12 @@ foreach my $narArchive (@narArchives) {
my $basename = $1;
if ($localCopy) {
# Since nix-push creates $dst atomically, if it exists we
# don't have to copy again.
my $dst = "$localArchivesDir/$basename";
if (! -f "$localArchivesDir/$basename") {
print STDERR " $narArchive\n";
copyFile $narArchive, "$localArchivesDir/$basename";
copyFile $narArchive, $dst;
}
}
else {
+171
View File
@@ -0,0 +1,171 @@
#! @shell@
WORKING_DIRECTORY=$(mktemp -d "${TMPDIR:-/tmp}"/nix-reduce-build-XXXXXX);
cd "$WORKING_DIRECTORY";
if test -z "$1" || test "a--help" = "a$1" ; then
echo 'nix-reduce-build (paths or Nix expressions) -- (package sources)' >&2
echo As in: >&2
echo nix-reduce-build /etc/nixos/nixos -- ssh://user@somewhere.nowhere.example.org >&2
echo nix-reduce-build /etc/nixos/nixos -- \\
echo " " \''http://somewhere.nowhere.example.org/nix/nix-http-export.cgi?needed_path='\' >&2
echo " store path name will be added into the end of the URL" >&2
echo nix-reduce-build /etc/nixos/nixos -- file://home/user/nar/ >&2
echo " that should be a directory where gzipped 'nix-store --export' ">&2
echo " files are located (they should have .nar.gz extension)" >&2
echo " Or all together: " >&2
echo -e nix-reduce-build /expr.nix /e2.nix -- \\\\\\\n\
" ssh://a@b.example.com http://n.example.com/get-nar?q= file://nar/" >&2
echo " Also supports best-effort local builds of failing expression set:" >&2
echo "nix-reduce-build /e.nix -- nix-daemon:// nix-self://" >&2
echo " nix-daemon:// builds using daemon"
echo " nix-self:// builds directly using nix-store from current installation" >&2
echo " nix-daemon-fixed:// and nix-self-fixed:// do the same, but only for" >&2;
echo "derivations with specified output hash (sha256, sha1 or md5)." >&2
echo " nix-daemon-substitute:// and nix-self-substitute:// try to substitute" >&2;
echo "maximum amount of paths" >&2;
echo " nix-daemon-build:// and nix-self-build:// try to build (not substitute)" >&2;
echo "maximum amount of paths" >&2;
echo " If no package sources are specified, required paths are listed." >&2;
exit;
fi;
while ! test "$1" = "--" || test "$1" = "" ; do
echo "$1" >> initial; >&2
shift;
done
shift;
echo Will work on $(cat initial | wc -l) targets. >&2
while read ; do
case "$REPLY" in
${NIX_STORE_DIR:-/nix/store}/*)
echo "$REPLY" >> paths; >&2
;;
*)
(
IFS=: ;
nix-instantiate $REPLY >> paths;
);
;;
esac;
done < initial;
echo Proceeding $(cat paths | wc -l) paths. >&2
while read; do
case "$REPLY" in
*.drv)
echo "$REPLY" >> derivers; >&2
;;
*)
nix-store --query --deriver "$REPLY" >>derivers;
;;
esac;
done < paths;
echo Found $(cat derivers | wc -l) derivers. >&2
cat derivers | xargs nix-store --query -R > derivers-closure;
echo Proceeding at most $(cat derivers-closure | wc -l) derivers. >&2
cat derivers-closure | egrep '[.]drv$' | xargs nix-store --query --outputs > wanted-paths;
cat derivers-closure | egrep -v '[.]drv$' >> wanted-paths;
echo Prepared $(cat wanted-paths | wc -l) paths to get. >&2
cat wanted-paths | xargs nix-store --check-validity --print-invalid > needed-paths;
echo We need $(cat needed-paths | wc -l) paths. >&2
egrep '[.]drv$' derivers-closure > critical-derivers;
if test -z "$1" ; then
cat needed-paths;
fi;
refresh_critical_derivers() {
echo "Finding needed derivers..." >&2;
cat critical-derivers | while read; do
if ! (nix-store --query --outputs "$REPLY" | xargs nix-store --check-validity &> /dev/null;); then
echo "$REPLY";
fi;
done > new-critical-derivers;
mv new-critical-derivers critical-derivers;
echo The needed paths are realized by $(cat critical-derivers | wc -l) derivers. >&2
}
build_here() {
cat critical-derivers | while read; do
echo "Realising $REPLY using nix-daemon" >&2
@bindir@/nix-store -r "${REPLY}"
done;
}
try_to_substitute(){
cat needed-paths | while read ; do
echo "Building $REPLY using nix-daemon" >&2
@bindir@/nix-store -r "${NIX_STORE_DIR:-/nix/store}/${REPLY##*/}"
done;
}
for i in "$@"; do
sshHost="${i#ssh://}";
httpHost="${i#http://}";
httpsHost="${i#https://}";
filePath="${i#file:/}";
if [ "$i" != "$sshHost" ]; then
cat needed-paths | while read; do
echo "Getting $REPLY and its closure over ssh" >&2
nix-copy-closure --from "$sshHost" --gzip "$REPLY" </dev/null || true;
done;
elif [ "$i" != "$httpHost" ] || [ "$i" != "$httpsHost" ]; then
cat needed-paths | while read; do
echo "Getting $REPLY over http/https" >&2
curl ${BAD_CERTIFICATE:+-k} -L "$i${REPLY##*/}" | gunzip | nix-store --import;
done;
elif [ "$i" != "$filePath" ] ; then
cat needed-paths | while read; do
echo "Installing $REPLY from file" >&2
gunzip < "$filePath/${REPLY##*/}".nar.gz | nix-store --import;
done;
elif [ "$i" = "nix-daemon://" ] ; then
NIX_REMOTE=daemon try_to_substitute;
refresh_critical_derivers;
NIX_REMOTE=daemon build_here;
elif [ "$i" = "nix-self://" ] ; then
NIX_REMOTE= try_to_substitute;
refresh_critical_derivers;
NIX_REMOTE= build_here;
elif [ "$i" = "nix-daemon-fixed://" ] ; then
refresh_critical_derivers;
cat critical-derivers | while read; do
if egrep '"(md5|sha1|sha256)"' "$REPLY" &>/dev/null; then
echo "Realising $REPLY using nix-daemon" >&2
NIX_REMOTE=daemon @bindir@/nix-store -r "${REPLY}"
fi;
done;
elif [ "$i" = "nix-self-fixed://" ] ; then
refresh_critical_derivers;
cat critical-derivers | while read; do
if egrep '"(md5|sha1|sha256)"' "$REPLY" &>/dev/null; then
echo "Realising $REPLY using direct Nix build" >&2
NIX_REMOTE= @bindir@/nix-store -r "${REPLY}"
fi;
done;
elif [ "$i" = "nix-daemon-substitute://" ] ; then
NIX_REMOTE=daemon try_to_substitute;
elif [ "$i" = "nix-self-substitute://" ] ; then
NIX_REMOTE= try_to_substitute;
elif [ "$i" = "nix-daemon-build://" ] ; then
refresh_critical_derivers;
NIX_REMOTE=daemon build_here;
elif [ "$i" = "nix-self-build://" ] ; then
refresh_critical_derivers;
NIX_REMOTE= build_here;
fi;
mv needed-paths wanted-paths;
cat wanted-paths | xargs nix-store --check-validity --print-invalid > needed-paths;
echo We still need $(cat needed-paths | wc -l) paths. >&2
done;
cd /
rm -r "$WORKING_DIRECTORY"
-88
View File
@@ -1,88 +0,0 @@
#! @perl@ -w
# This tool unpacks the closures created by "nix-pack-closure" and
# adds them to the Nix store.
# TODO: make this program "streamy", i.e., don't use a temporary
# directory.
use strict;
use File::Temp qw(tempdir);
my $binDir = $ENV{"NIX_BIN_DIR"};
$binDir = "@bindir@" unless defined $binDir;
my $tmpDir = tempdir("nix-unpack-closure.XXXXXX", CLEANUP => 1, TMPDIR => 1)
or die "cannot create a temporary directory";
# Unpack the NAR archive on standard input.
system("nix-store --restore '$tmpDir/unpacked'") == 0
or die "nix-store --restore failed";
open VALID, ">$tmpDir/validity" or die;
# For each path in the closure that is not yet valid, add it to the
# store. TODO: use proper locking. Or even better, let nix-store do
# this.
opendir(DIR, "$tmpDir/unpacked/contents") or die "cannot open directory: $!";
foreach my $name (sort(readdir DIR)) {
next if $name eq "." or $name eq "..";
my $storePath = "@storedir@/$name"; # !!!
# !!! this really isn't a good validity check!
system "$binDir/nix-store --check-validity '$storePath' 2> /dev/null";
if ($? != 0) {
print STDERR "unpacking `$storePath'...\n";
# !!! race
system("@coreutils@/rm -rf '$storePath'") == 0
or die "cannot remove `$storePath': $?";
system("$binDir/nix-store --restore '$storePath' < '$tmpDir/unpacked/contents/$name'") == 0
or die "nix-store --dump failed on `$storePath': $?";
print VALID "$storePath\n";
open DRV, "<$tmpDir/unpacked/derivers/$name" or die;
my $deriver = <DRV>;
chomp $deriver;
$deriver = "" if $deriver eq "unknown-deriver";
close DRV;
my @refs;
open REFS, "<$tmpDir/unpacked/references/$name" or die;
while (<REFS>) {
chomp;
push @refs, $_;
}
close REFS;
print VALID "$deriver\n";
print VALID (scalar @refs), "\n";
foreach my $ref (@refs) {
print VALID "$ref\n";
}
}
}
closedir(DIR) or die;
# Register the invalid paths as valid.
system("nix-store --register-validity <'$tmpDir/validity'") == 0
or die "nix-store --register-validity failed";
# Show the top-level paths so that something useful can be done with
# them, e.g., passing them to `nix-env -i'.
if (-e "$tmpDir/unpacked/top-level") {
open TOPLEVEL, "<$tmpDir/unpacked/top-level" or die;
while (<TOPLEVEL>) { print "$_"; }
close TOPLEVEL;
}
+11 -37
View File
@@ -2,10 +2,7 @@ use strict;
sub addPatch {
my $patches = shift;
my $storePath = shift;
my $patch = shift;
my $allowConflicts = shift;
my ($patches, $storePath, $patch) = @_;
$$patches{$storePath} = []
unless defined $$patches{$storePath};
@@ -14,15 +11,9 @@ sub addPatch {
my $found = 0;
foreach my $patch2 (@{$patchList}) {
if ($patch2->{url} eq $patch->{url}) {
if ($patch2->{hash} eq $patch->{hash}) {
$found = 1 if ($patch2->{basePath} eq $patch->{basePath});
} else {
die "conflicting hashes for URL $patch->{url}, " .
"namely $patch2->{hash} and $patch->{hash}"
unless $allowConflicts;
}
}
$found = 1 if
$patch2->{url} eq $patch->{url} &&
$patch2->{basePath} eq $patch->{basePath};
}
push @{$patchList}, $patch if !$found;
@@ -32,12 +23,7 @@ sub addPatch {
sub readManifest {
my $manifest = shift;
my $narFiles = shift;
my $localPaths = shift;
my $patches = shift;
my $allowConflicts = shift;
$allowConflicts = 0 unless defined $allowConflicts;
my ($manifest, $narFiles, $localPaths, $patches) = @_;
open MANIFEST, "<$manifest"
or die "cannot open `$manifest': $!";
@@ -98,15 +84,7 @@ sub readManifest {
my $found = 0;
foreach my $narFile (@{$narFileList}) {
if ($narFile->{url} eq $url) {
if ($narFile->{hash} eq $hash) {
$found = 1;
} else {
die "conflicting hashes for URL $url, " .
"namely $narFile->{hash} and $hash"
unless $allowConflicts;
}
}
$found = 1 if $narFile->{url} eq $url;
}
if (!$found) {
push @{$narFileList},
@@ -124,7 +102,7 @@ sub readManifest {
, basePath => $basePath, baseHash => $baseHash
, narHash => $narHash, patchType => $patchType
, hashAlgo => $hashAlgo
}, $allowConflicts;
};
}
elsif ($type eq "localPath") {
@@ -171,12 +149,8 @@ sub readManifest {
}
sub writeManifest
{
my $manifest = shift;
my $narFiles = shift;
my $patches = shift;
my $copySources = shift;
sub writeManifest {
my ($manifest, $narFiles, $patches) = @_;
open MANIFEST, ">$manifest.tmp"; # !!! check exclusive
@@ -190,9 +164,9 @@ sub writeManifest
print MANIFEST "{\n";
print MANIFEST " StorePath: $storePath\n";
print MANIFEST " NarURL: $narFile->{url}\n";
print MANIFEST " Hash: $narFile->{hash}\n";
print MANIFEST " Hash: $narFile->{hash}\n" if defined $narFile->{hash};
print MANIFEST " NarHash: $narFile->{narHash}\n";
print MANIFEST " Size: $narFile->{size}\n";
print MANIFEST " Size: $narFile->{size}\n" if defined $narFile->{size};
print MANIFEST " References: $narFile->{references}\n"
if defined $narFile->{references} && $narFile->{references} ne "";
print MANIFEST " Deriver: $narFile->{deriver}\n"
+1 -1
View File
@@ -1,6 +1,6 @@
SUBDIRS = format
pkginclude_HEADERS = assert.hpp checked_delete.hpp format.hpp \
nobase_pkginclude_HEADERS = assert.hpp checked_delete.hpp format.hpp \
shared_ptr.hpp weak_ptr.hpp throw_exception.hpp \
enable_shared_from_this.hpp \
detail/shared_count.hpp detail/workaround.hpp
+5 -3
View File
@@ -2,11 +2,13 @@ pkglib_LTLIBRARIES = libexpr.la
libexpr_la_SOURCES = \
nixexpr.cc eval.cc primops.cc lexer-tab.cc parser-tab.cc \
get-drvs.cc attr-path.cc expr-to-xml.cc common-opts.cc
get-drvs.cc attr-path.cc expr-to-xml.cc common-opts.cc \
names.cc
pkginclude_HEADERS = \
nixexpr.hh eval.hh parser.hh lexer-tab.hh parser-tab.hh \
get-drvs.hh attr-path.hh expr-to-xml.hh common-opts.hh
get-drvs.hh attr-path.hh expr-to-xml.hh common-opts.hh \
names.hh nixexpr-ast.hh
libexpr_la_LIBADD = ../libutil/libutil.la ../libstore/libstore.la \
../boost/format/libformat.la
@@ -17,7 +19,7 @@ BUILT_SOURCES = nixexpr-ast.cc nixexpr-ast.hh \
EXTRA_DIST = lexer.l parser.y nixexpr-ast.def nixexpr-ast.cc
AM_CXXFLAGS = \
-I$(srcdir)/.. ${bdb_include} ${aterm_include} \
-I$(srcdir)/.. ${aterm_include} \
-I$(srcdir)/../libutil -I$(srcdir)/../libstore
AM_CFLAGS = \
${aterm_include}
+188 -132
View File
@@ -23,6 +23,8 @@ EvalState::EvalState()
initNixExprHelpers();
addPrimOps();
allowUnsafeEquality = getEnv("NIX_NO_UNSAFE_EQ", "") == "";
}
@@ -48,6 +50,11 @@ LocalNoInlineNoReturn(void throwEvalError(const char * s, const string & s2))
throw EvalError(format(s) % s2);
}
LocalNoInlineNoReturn(void throwTypeError(const char * s))
{
throw TypeError(s);
}
LocalNoInlineNoReturn(void throwTypeError(const char * s, const string & s2))
{
throw TypeError(format(s) % s2);
@@ -69,77 +76,91 @@ LocalNoInline(void addErrorPrefix(Error & e, const char * s, const string & s2,
}
/* Pattern-match `pat' against `arg'. The result is a set of
substitutions (`subs') and a set of recursive substitutions
(`subsRecursive'). The latter can refer to the variables bound by
both `subs' and `subsRecursive'. */
static void patternMatch(EvalState & state,
Pattern pat, Expr arg, ATermMap & subs, ATermMap & subsRecursive)
{
ATerm name;
ATermList formals;
Pattern pat1, pat2;
ATermBool ellipsis;
if (matchVarPat(pat, name))
subs.set(name, arg);
else if (matchAttrsPat(pat, formals, ellipsis)) {
arg = evalExpr(state, arg);
/* Get the actual arguments. */
ATermMap attrs;
queryAllAttrs(arg, attrs);
unsigned int nrAttrs = attrs.size();
/* For each formal argument, get the actual argument. If
there is no matching actual argument but the formal
argument has a default, use the default. */
unsigned int attrsUsed = 0;
for (ATermIterator i(formals); i; ++i) {
Expr name, def;
DefaultValue def2;
if (!matchFormal(*i, name, def2)) abort(); /* can't happen */
Expr value = attrs[name];
if (value == 0) {
if (!matchDefaultValue(def2, def)) def = 0;
if (def == 0) throw TypeError(format("the argument named `%1%' required by the function is missing")
% aterm2String(name));
subsRecursive.set(name, def);
} else {
attrsUsed++;
attrs.remove(name);
subs.set(name, value);
}
}
/* Check that each actual argument is listed as a formal
argument (unless the attribute match specifies a `...'). */
if (ellipsis == eFalse && attrsUsed != nrAttrs)
throw TypeError(format("the function does not expect an argument named `%1%'")
% aterm2String(attrs.begin()->key));
}
else if (matchAtPat(pat, pat1, pat2)) {
patternMatch(state, pat1, arg, subs, subsRecursive);
patternMatch(state, pat2, arg, subs, subsRecursive);
}
else abort();
}
/* Substitute an argument set into the body of a function. */
static Expr substArgs(EvalState & state,
Expr body, ATermList formals, Expr arg)
Expr body, Pattern pat, Expr arg)
{
unsigned int nrFormals = ATgetLength(formals);
ATermMap subs(nrFormals);
/* Get the actual arguments and put them in the substitution. */
ATermMap args;
queryAllAttrs(arg, args);
for (ATermMap::const_iterator i = args.begin(); i != args.end(); ++i)
subs.set(i->key, i->value);
ATermMap subs(16), subsRecursive(16);
/* Get the formal arguments. */
ATermVector defsUsed;
ATermList recAttrs = ATempty;
for (ATermIterator i(formals); i; ++i) {
Expr name, def;
ValidValues valids2;
DefaultValue def2;
if (!matchFormal(*i, name, valids2, def2)) abort(); /* can't happen */
patternMatch(state, pat, arg, subs, subsRecursive);
Expr value = subs[name];
if (value == 0) {
if (!matchDefaultValue(def2, def)) def = 0;
if (def == 0) throw TypeError(format("the argument named `%1%' required by the function is missing")
% aterm2String(name));
value = def;
defsUsed.push_back(name);
recAttrs = ATinsert(recAttrs, makeBind(name, def, makeNoPos()));
}
ATermList valids;
if (matchValidValues(valids2, valids)) {
value = evalExpr(state, value);
bool found = false;
for (ATermIterator j(valids); j; ++j) {
Expr v = evalExpr(state, *j);
if (value == v) {
found = true;
break;
}
}
if (!found) throw TypeError(format("the argument named `%1%' has an illegal value")
% aterm2String(name));
}
}
/* Make a recursive attribute set out of the (argument-name,
value) tuples. This is so that we can support default
parameters that refer to each other, e.g. ({x, y ? x + x}: y)
{x = "foo";} evaluates to "foofoo". */
if (defsUsed.size() != 0) {
for (ATermMap::const_iterator i = args.begin(); i != args.end(); ++i)
/* If we used any default values, make a recursive attribute set
out of the (argument-name, value) tuples. This is so that we
can support default values that refer to each other, e.g. ({x,
y ? x + x}: y) {x = "foo";} evaluates to "foofoo". */
if (subsRecursive.size() != 0) {
ATermList recAttrs = ATempty;
foreach (ATermMap::const_iterator, i, subs)
recAttrs = ATinsert(recAttrs, makeBind(i->key, i->value, makeNoPos()));
foreach (ATermMap::const_iterator, i, subsRecursive)
recAttrs = ATinsert(recAttrs, makeBind(i->key, i->value, makeNoPos()));
Expr rec = makeRec(recAttrs, ATempty);
for (ATermVector::iterator i = defsUsed.begin(); i != defsUsed.end(); ++i)
subs.set(*i, makeSelect(rec, *i));
}
if (subs.size() != nrFormals) {
/* One or more actual arguments were not declared as formal
arguments. Find out which. */
for (ATermIterator i(formals); i; ++i) {
Expr name; ATerm d1, d2;
if (!matchFormal(*i, name, d1, d2)) abort();
subs.remove(name);
}
throw TypeError(format("the function does not expect an argument named `%1%'")
% aterm2String(subs.begin()->key));
foreach (ATermMap::const_iterator, i, subsRecursive)
subs.set(i->key, makeSelect(rec, i->key));
}
return substitute(Substitution(0, &subs), body);
@@ -151,11 +172,12 @@ static Expr substArgs(EvalState & state,
to attributes substituted with selection expressions on the
original set. E.g., e = `rec {x = f x y; y = x;}' becomes `{x = f
(e.x) (e.y); y = e.x;}'. */
LocalNoInline(ATerm expandRec(ATerm e, ATermList rbnds, ATermList nrbnds))
LocalNoInline(ATerm expandRec(EvalState & state, ATerm e, ATermList rbnds, ATermList nrbnds))
{
ATerm name;
Expr e2;
Pos pos;
Expr eOverrides = 0;
/* Create the substitution list. */
ATermMap subs(ATgetLength(rbnds) + ATgetLength(nrbnds));
@@ -165,9 +187,26 @@ LocalNoInline(ATerm expandRec(ATerm e, ATermList rbnds, ATermList nrbnds))
}
for (ATermIterator i(nrbnds); i; ++i) {
if (!matchBind(*i, name, e2, pos)) abort(); /* can't happen */
if (name == sOverrides) eOverrides = e2;
subs.set(name, e2);
}
/* If the rec contains an attribute called `__overrides', then
evaluate it, and add the attributes in that set to the rec.
This allows overriding of recursive attributes, which is
otherwise not possible. (You can use the // operator to
replace an attribute, but other attributes in the rec will
still reference the original value, because that value has been
substituted into the bodies of the other attributes. Hence we
need __overrides.) */
ATermMap overrides;
if (eOverrides) {
eOverrides = evalExpr(state, eOverrides);
queryAllAttrs(eOverrides, overrides, false);
foreach (ATermMap::const_iterator, i, overrides)
subs.set(i->key, i->value);
}
Substitution subs_(0, &subs);
/* Create the non-recursive set. */
@@ -177,6 +216,10 @@ LocalNoInline(ATerm expandRec(ATerm e, ATermList rbnds, ATermList nrbnds))
as.set(name, makeAttrRHS(substitute(subs_, e2), pos));
}
if (eOverrides)
foreach (ATermMap::const_iterator, i, overrides)
as.set(i->key, makeAttrRHS(i->value, makeNoPos()));
/* Copy the non-recursive bindings. !!! inefficient */
for (ATermIterator i(nrbnds); i; ++i) {
if (!matchBind(*i, name, e2, pos)) abort(); /* can't happen */
@@ -199,7 +242,7 @@ LocalNoInline(Expr updateAttrs(Expr e1, Expr e2))
}
string evalString(EvalState & state, Expr e, PathSet & context)
string evalString(EvalState & state, Expr e, ATermList & context)
{
e = evalExpr(state, e);
string s;
@@ -211,11 +254,10 @@ string evalString(EvalState & state, Expr e, PathSet & context)
string evalStringNoCtx(EvalState & state, Expr e)
{
PathSet context;
ATermList context;
string s = evalString(state, e, context);
if (!context.empty())
throw EvalError(format("the string `%1%' is not allowed to refer to a store path (such as `%2%')")
% s % *(context.begin()));
if (context != ATempty)
throw EvalError(format("the string `%1%' is not allowed to refer to a store path") % s);
return s;
}
@@ -269,7 +311,7 @@ ATermList flattenList(EvalState & state, Expr e)
}
string coerceToString(EvalState & state, Expr e, PathSet & context,
string coerceToString(EvalState & state, Expr e, Context & context,
bool coerceMore, bool copyToStore)
{
e = evalExpr(state, e);
@@ -300,14 +342,22 @@ string coerceToString(EvalState & state, Expr e, PathSet & context,
% path % dstPath);
}
context.insert(dstPath);
context.set(toATerm(dstPath), makeNull());
return dstPath;
}
ATermList es;
if (matchAttrs(e, es))
return coerceToString(state, makeSelect(e, toATerm("outPath")),
context, coerceMore, copyToStore);
if (matchAttrs(e, es)) {
Expr e2 = queryAttr(e, "outPath");
if (!e2) throwTypeError("cannot coerce an attribute set (except a derivation) to a string");
/* !!! hacky */
ATermList c;
string s = evalString(state, e2, c);
Context c2; matchContext(c, c2);
foreach (Context::const_iterator, i, c2)
context.set(i->key, e);
return s;
}
if (coerceMore) {
@@ -340,9 +390,9 @@ string coerceToString(EvalState & state, Expr e, PathSet & context,
static ATerm concatStrings(EvalState & state, ATermVector & args,
string separator = "")
{
if (args.empty()) return makeStr("", PathSet());
if (args.empty()) return makeStr("", ATempty);
PathSet context;
Context context;
std::ostringstream s;
/* If the first element is a path, then the result will also be a
@@ -358,7 +408,7 @@ static ATerm concatStrings(EvalState & state, ATermVector & args,
s << coerceToString(state, *i, context, false, !isPath);
}
if (isPath && !context.empty())
if (isPath && context.size() != 0)
throw EvalError(format("a string that refers to a store path cannot be appended to a path, in `%1%'")
% s.str());
@@ -368,7 +418,7 @@ static ATerm concatStrings(EvalState & state, ATermVector & args,
}
Path coerceToPath(EvalState & state, Expr e, PathSet & context)
Path coerceToPath(EvalState & state, Expr e, Context & context)
{
string path = coerceToString(state, e, context, false, false);
if (path == "" || path[0] != '/')
@@ -379,15 +429,18 @@ Path coerceToPath(EvalState & state, Expr e, PathSet & context)
Expr autoCallFunction(Expr e, const ATermMap & args)
{
ATermList formals;
Pattern pat;
ATerm body, pos;
ATermList formals;
ATermBool ellipsis;
if (matchFunction(e, formals, body, pos)) {
/* !!! this should be more general */
if (matchFunction(e, pat, body, pos) && matchAttrsPat(pat, formals, ellipsis)) {
ATermMap actualArgs(ATgetLength(formals));
for (ATermIterator i(formals); i; ++i) {
Expr name, def, value; ATerm values, def2;
if (!matchFormal(*i, name, values, def2)) abort();
Expr name, def, value; ATerm def2;
if (!matchFormal(*i, name, def2)) abort();
if ((value = args.get(name)))
actualArgs.set(name, makeAttrRHS(value, makeNoPos()));
else if (!matchDefaultValue(def2, def))
@@ -427,8 +480,8 @@ LocalNoInline(Expr evalVar(EvalState & state, ATerm name))
LocalNoInline(Expr evalCall(EvalState & state, Expr fun, Expr arg))
{
ATermList formals;
ATerm pos, name;
Pattern pat;
ATerm pos;
Expr body;
/* Evaluate the left-hand side. */
@@ -454,10 +507,9 @@ LocalNoInline(Expr evalCall(EvalState & state, Expr fun, Expr arg))
return makePrimOp(arity, funBlob, args);
}
else if (matchFunction(fun, formals, body, pos)) {
arg = evalExpr(state, arg);
else if (matchFunction(fun, pat, body, pos)) {
try {
return evalExpr(state, substArgs(state, body, formals, arg));
return evalExpr(state, substArgs(state, body, pat, arg));
} catch (Error & e) {
addErrorPrefix(e, "while evaluating the function at %1%:\n",
showPos(pos));
@@ -465,18 +517,6 @@ LocalNoInline(Expr evalCall(EvalState & state, Expr fun, Expr arg))
}
}
else if (matchFunction1(fun, name, body, pos)) {
try {
ATermMap subs(1);
subs.set(name, arg);
return evalExpr(state, substitute(Substitution(0, &subs), body));
} catch (Error & e) {
addErrorPrefix(e, "while evaluating the function at %1%:\n",
showPos(pos));
throw;
}
}
else throwTypeError(
"attempt to call something which is neither a function nor a primop (built-in operation) but %1%",
showType(fun));
@@ -565,7 +605,7 @@ LocalNoInline(Expr evalPlusConcat(EvalState & state, Expr e))
"concatenation of a derivation and a path is deprecated; "
"you should write `drv + \"%1%\"' instead of `drv + %1%'")
% aterm2String(p));
PathSet context;
Context context;
return makeStr(
coerceToString(state, makeSelect(e1, toATerm("outPath")), context)
+ aterm2String(p), context);
@@ -611,6 +651,48 @@ LocalNoInline(Expr evalOpConcat(EvalState & state, Expr e1, Expr e2))
}
/* Implementation of the `==' and `!=' operators. */
LocalNoInline(bool areEqual(EvalState & state, Expr e1, Expr e2))
{
e1 = evalExpr(state, e1);
e2 = evalExpr(state, e2);
/* We cannot test functions/primops for equality, and we currently
don't support testing equality between attribute sets or lists
- that would have to be a deep equality test to be sound. */
AFun sym1 = ATgetAFun(e1);
AFun sym2 = ATgetAFun(e2);
if (sym1 != sym2) return false;
/* Functions are incomparable. */
if (sym1 == symFunction || sym1 == symPrimOp) return false;
if (!state.allowUnsafeEquality && sym1 == symAttrs)
throw EvalError("comparison of attribute sets is not implemented");
/* !!! This allows comparisons of infinite data structures to
succeed, such as `let x = [x]; in x == x'. This is
undesirable, since equivalent (?) terms such as `let x = [x]; y
= [y]; in x == y' don't terminate. */
if (e1 == e2) return true;
if (sym1 == symList) {
ATermList es1; matchList(e1, es1);
ATermList es2; matchList(e2, es2);
if (ATgetLength(es1) != ATgetLength(es2)) return false;
ATermIterator i(es1), j(es2);
while (*i) {
if (!areEqual(state, *i, *j)) return false;
++i; ++j;
}
return true;
}
return false;
}
static char * deepestStack = (char *) -1; /* for measuring stack usage */
@@ -633,7 +715,6 @@ Expr evalExpr2(EvalState & state, Expr e)
sym == symInt ||
sym == symBool ||
sym == symFunction ||
sym == symFunction1 ||
sym == symAttrs ||
sym == symList ||
sym == symPrimOp)
@@ -657,7 +738,7 @@ Expr evalExpr2(EvalState & state, Expr e)
/* Mutually recursive sets. */
ATermList rbnds, nrbnds;
if (matchRec(e, rbnds, nrbnds))
return expandRec(e, rbnds, nrbnds);
return expandRec(state, e, rbnds, nrbnds);
/* Conditionals. */
if (matchIf(e, e1, e2, e3))
@@ -675,12 +756,10 @@ Expr evalExpr2(EvalState & state, Expr e)
However, we don't want to make (==) strict, because that would
make operations like `big_derivation == null' very slow (unless
we were to evaluate them side-by-side). */
if (matchOpEq(e, e1, e2))
return makeBool(evalExpr(state, e1) == evalExpr(state, e2));
if (matchOpNEq(e, e1, e2))
return makeBool(evalExpr(state, e1) != evalExpr(state, e2));
if (matchOpEq(e, e1, e2)) return makeBool(areEqual(state, e1, e2));
if (matchOpNEq(e, e1, e2)) return makeBool(!areEqual(state, e1, e2));
/* Negation. */
if (matchOpNot(e, e1))
return makeBool(!evalBool(state, e1));
@@ -793,29 +872,6 @@ static Expr strictEvalExpr_(EvalState & state, Expr e, ATermMap & nfs)
return makeList(ATreverse(es2));
}
ATermList formals;
ATerm body, pos;
if (matchFunction(e, formals, body, pos)) {
ATermList formals2 = ATempty;
for (ATermIterator i(formals); i; ++i) {
Expr name; ValidValues valids; ATerm dummy;
if (!matchFormal(*i, name, valids, dummy)) abort();
ATermList valids2;
if (matchValidValues(valids, valids2)) {
ATermList valids3 = ATempty;
for (ATermIterator j(valids2); j; ++j)
valids3 = ATinsert(valids3, strictEvalExpr(state, *j, nfs));
valids = makeValidValues(ATreverse(valids3));
}
formals2 = ATinsert(formals2, makeFormal(name, valids, dummy));
}
return makeFunction(ATreverse(formals2), body, pos);
}
return e;
}
+5 -3
View File
@@ -38,6 +38,8 @@ struct EvalState
unsigned int nrEvaluated;
unsigned int nrCached;
bool allowUnsafeEquality;
EvalState();
void addPrimOps();
@@ -59,7 +61,7 @@ Expr evalFile(EvalState & state, const Path & path);
Expr strictEvalExpr(EvalState & state, Expr e);
/* Specific results. */
string evalString(EvalState & state, Expr e, PathSet & context);
string evalString(EvalState & state, Expr e, ATermList & context);
string evalStringNoCtx(EvalState & state, Expr e);
int evalInt(EvalState & state, Expr e);
bool evalBool(EvalState & state, Expr e);
@@ -72,13 +74,13 @@ ATermList flattenList(EvalState & state, Expr e);
/* String coercion. Converts strings, paths and derivations to a
string. If `coerceMore' is set, also converts nulls, integers,
booleans and lists to a string. */
string coerceToString(EvalState & state, Expr e, PathSet & context,
string coerceToString(EvalState & state, Expr e, Context & context,
bool coerceMore = false, bool copyToStore = true);
/* Path coercion. Converts strings, paths and derivations to a path.
The result is guaranteed to be an canonicalised, absolute path.
Nothing is copied to the store. */
Path coerceToPath(EvalState & state, Expr e, PathSet & context);
Path coerceToPath(EvalState & state, Expr e, Context & context);
/* Automatically call a function for which each argument has a default
value or has a binding in the `args' map. Note: result is a call,
+35 -21
View File
@@ -4,6 +4,8 @@
#include "aterm.hh"
#include "util.hh"
#include <cstdlib>
namespace nix {
@@ -21,12 +23,12 @@ static XMLAttrs singletonAttrs(const string & name, const string & value)
typedef set<Expr> ExprSet;
static void printTermAsXML(Expr e, XMLWriter & doc, PathSet & context,
static void printTermAsXML(Expr e, XMLWriter & doc, Context & context,
ExprSet & drvsSeen);
static void showAttrs(const ATermMap & attrs, XMLWriter & doc,
PathSet & context, ExprSet & drvsSeen)
Context & context, ExprSet & drvsSeen)
{
StringSet names;
for (ATermMap::const_iterator i = attrs.begin(); i != attrs.end(); ++i)
@@ -38,15 +40,40 @@ static void showAttrs(const ATermMap & attrs, XMLWriter & doc,
}
static void printTermAsXML(Expr e, XMLWriter & doc, PathSet & context,
static void printPatternAsXML(Pattern pat, XMLWriter & doc)
{
ATerm name;
ATermList formals;
Pattern pat1, pat2;
ATermBool ellipsis;
if (matchVarPat(pat, name))
doc.writeEmptyElement("varpat", singletonAttrs("name", aterm2String(name)));
else if (matchAttrsPat(pat, formals, ellipsis)) {
XMLOpenElement _(doc, "attrspat");
for (ATermIterator i(formals); i; ++i) {
Expr name; ATerm dummy;
if (!matchFormal(*i, name, dummy)) abort();
doc.writeEmptyElement("attr", singletonAttrs("name", aterm2String(name)));
}
if (ellipsis == eTrue) doc.writeEmptyElement("ellipsis");
}
else if (matchAtPat(pat, pat1, pat2)) {
XMLOpenElement _(doc, "at");
printPatternAsXML(pat1, doc);
printPatternAsXML(pat2, doc);
}
}
static void printTermAsXML(Expr e, XMLWriter & doc, Context & context,
ExprSet & drvsSeen)
{
XMLAttrs attrs;
string s;
ATerm s2;
int i;
ATermList as, es, formals;
ATerm body, pos;
ATermList as, es;
ATerm pat, body, pos;
checkInterrupt();
@@ -107,22 +134,9 @@ static void printTermAsXML(Expr e, XMLWriter & doc, PathSet & context,
printTermAsXML(*i, doc, context, drvsSeen);
}
else if (matchFunction(e, formals, body, pos)) {
else if (matchFunction(e, pat, body, pos)) {
XMLOpenElement _(doc, "function");
for (ATermIterator i(formals); i; ++i) {
Expr name; ValidValues valids; ATerm dummy;
if (!matchFormal(*i, name, valids, dummy)) abort();
XMLOpenElement _(doc, "arg", singletonAttrs("name", aterm2String(name)));
ATermList valids2;
if (matchValidValues(valids, valids2)) {
for (ATermIterator j(valids2); j; ++j) {
XMLOpenElement _(doc, "value");
printTermAsXML(*j, doc, context, drvsSeen);
}
}
}
printPatternAsXML(pat, doc);
}
else
@@ -130,7 +144,7 @@ static void printTermAsXML(Expr e, XMLWriter & doc, PathSet & context,
}
void printTermAsXML(Expr e, std::ostream & out, PathSet & context)
void printTermAsXML(Expr e, std::ostream & out, Context & context)
{
XMLWriter doc(true, out);
XMLOpenElement root(doc, "expr");
+1 -1
View File
@@ -9,7 +9,7 @@
namespace nix {
void printTermAsXML(Expr e, std::ostream & out, PathSet & context);
void printTermAsXML(Expr e, std::ostream & out, Context & context);
}
+58 -28
View File
@@ -17,7 +17,7 @@ string DrvInfo::queryDrvPath(EvalState & state) const
if (a && matchPath(evalExpr(state, a), t))
return aterm2String(t);
PathSet context;
Context context;
(string &) drvPath = a ? coerceToPath(state, a, context) : "";
}
return drvPath;
@@ -29,7 +29,7 @@ string DrvInfo::queryOutPath(EvalState & state) const
if (outPath == "") {
Expr a = attrs->get(toATerm("outPath"));
if (!a) throw TypeError("output path missing");
PathSet context;
Context context;
(string &) outPath = coerceToPath(state, a, context);
}
return outPath;
@@ -49,32 +49,56 @@ MetaInfo DrvInfo::queryMetaInfo(EvalState & state) const
for (ATermMap::const_iterator i = attrs2.begin(); i != attrs2.end(); ++i) {
Expr e = evalExpr(state, i->value);
string s;
PathSet context;
if (matchStr(e, s, context))
meta[aterm2String(i->key)] = s;
/* For future compatibility, ignore attribute values that are
not strings. */
ATermList context;
MetaValue value;
int n;
ATermList es;
if (matchStr(e, s, context)) {
value.type = MetaValue::tpString;
value.stringValue = s;
meta[aterm2String(i->key)] = value;
} else if (matchInt(e, n)) {
value.type = MetaValue::tpInt;
value.intValue = n;
meta[aterm2String(i->key)] = value;
} else if (matchList(e, es)) {
value.type = MetaValue::tpStrings;
for (ATermIterator j(es); j; ++j)
value.stringValues.push_back(evalStringNoCtx(state, *j));
meta[aterm2String(i->key)] = value;
}
}
return meta;
}
string DrvInfo::queryMetaInfo(EvalState & state, const string & name) const
MetaValue DrvInfo::queryMetaInfo(EvalState & state, const string & name) const
{
/* !!! evaluates all meta attributes => inefficient */
MetaInfo meta = queryMetaInfo(state);
MetaInfo::iterator i = meta.find(name);
return i == meta.end() ? "" : i->second;
return queryMetaInfo(state)[name];
}
void DrvInfo::setMetaInfo(const MetaInfo & meta)
{
ATermMap metaAttrs;
for (MetaInfo::const_iterator i = meta.begin(); i != meta.end(); ++i)
metaAttrs.set(toATerm(i->first),
makeAttrRHS(makeStr(i->second), makeNoPos()));
foreach (MetaInfo::const_iterator, i, meta) {
Expr e;
switch (i->second.type) {
case MetaValue::tpInt: e = makeInt(i->second.intValue); break;
case MetaValue::tpString: e = makeStr(i->second.stringValue); break;
case MetaValue::tpStrings: {
ATermList es = ATempty;
foreach (Strings::const_iterator, j, i->second.stringValues)
es = ATinsert(es, makeStr(*j));
e = makeList(ATreverse(es));
break;
}
default: abort();
}
metaAttrs.set(toATerm(i->first), makeAttrRHS(e, makeNoPos()));
}
attrs->set(toATerm("meta"), makeAttrs(metaAttrs));
}
@@ -102,7 +126,7 @@ static bool getDerivation(EvalState & state, Expr e,
boost::shared_ptr<ATermMap> attrs(new ATermMap());
queryAllAttrs(e, *attrs, false);
Expr a = attrs->get(toATerm("type"));
if (!a || evalStringNoCtx(state, a) != "derivation") return true;
@@ -173,24 +197,30 @@ static void getDerivations(EvalState & state, Expr e,
/* !!! undocumented hackery to support combining channels in
nix-env.cc. */
Expr e2 = drvMap.get(toATerm("_combineChannels"));
bool combineChannels = e2 && evalBool(state, e2);
for (ATermMap::const_iterator i = drvMap.begin(); i != drvMap.end(); ++i) {
startNest(nest, lvlDebug,
format("evaluating attribute `%1%'") % aterm2String(i->key));
string pathPrefix2 = addToPath(pathPrefix, aterm2String(i->key));
if (combineChannels) {
if (((string) aterm2String(i->key)) != "_combineChannels")
getDerivations(state, i->value, pathPrefix2, autoArgs, drvs, doneExprs);
}
else if (getDerivation(state, i->value, pathPrefix2, drvs, doneExprs)) {
bool combineChannels = drvMap.get(toATerm("_combineChannels"));
/* Consider the attributes in sorted order to get more
deterministic behaviour in nix-env operations (e.g. when
there are names clashes between derivations, the derivation
bound to the attribute with the "lower" name should take
precedence). */
typedef std::map<string, Expr> AttrsSorted;
AttrsSorted attrsSorted;
foreach (ATermMap::const_iterator, i, drvMap)
attrsSorted[aterm2String(i->key)] = i->value;
foreach (AttrsSorted::iterator, i, attrsSorted) {
startNest(nest, lvlDebug, format("evaluating attribute `%1%'") % i->first);
string pathPrefix2 = addToPath(pathPrefix, i->first);
if (combineChannels)
getDerivations(state, i->second, pathPrefix2, autoArgs, drvs, doneExprs);
else if (getDerivation(state, i->second, pathPrefix2, drvs, doneExprs)) {
/* If the value of this attribute is itself an
attribute set, should we recurse into it? => Only
if it has a `recurseForDerivations = true'
attribute. */
ATermList es;
Expr e = evalExpr(state, i->value);
Expr e = evalExpr(state, i->second), e2;
if (matchAttrs(e, es)) {
ATermMap attrs(ATgetLength(es));
queryAllAttrs(e, attrs, false);
+11 -2
View File
@@ -12,7 +12,16 @@
namespace nix {
typedef std::map<string, string> MetaInfo;
struct MetaValue
{
enum { tpNone, tpString, tpStrings, tpInt } type;
string stringValue;
Strings stringValues;
int intValue;
};
typedef std::map<string, MetaValue> MetaInfo;
struct DrvInfo
@@ -34,7 +43,7 @@ public:
string queryDrvPath(EvalState & state) const;
string queryOutPath(EvalState & state) const;
MetaInfo queryMetaInfo(EvalState & state) const;
string queryMetaInfo(EvalState & state, const string & name) const;
MetaValue queryMetaInfo(EvalState & state, const string & name) const;
void setDrvPath(const string & s)
{
+6 -1
View File
@@ -95,6 +95,7 @@ let { return LET; }
in { return IN; }
rec { return REC; }
inherit { return INHERIT; }
\.\.\. { return ELLIPSIS; }
\=\= { return EQ; }
\!\= { return NEQ; }
@@ -124,7 +125,7 @@ inherit { return INHERIT; }
<STRING>. return yytext[0]; /* just in case: shouldn't be reached */
\'\'(\ *\n)? { BEGIN(IND_STRING); return IND_STRING_OPEN; }
<IND_STRING>([^\$\']|\$[^\{\']|\'[^\'])+ {
<IND_STRING>([^\$\']|\$[^\{\']|\'[^\'\$])+ {
yylval->t = makeIndStr(toATerm(yytext));
return IND_STR;
}
@@ -142,6 +143,10 @@ inherit { return INHERIT; }
}
<IND_STRING>\$\{ { BEGIN(INITIAL); return DOLLAR_CURLY; }
<IND_STRING>\'\' { BEGIN(INITIAL); return IND_STRING_CLOSE; }
<IND_STRING>\' {
yylval->t = makeIndStr(toATerm("'"));
return IND_STR;
}
<IND_STRING>. return yytext[0]; /* just in case: shouldn't be reached */
{PATH} { yylval->t = toATerm(yytext); return PATH; /* !!! alloc */ }
@@ -92,25 +92,6 @@ int compareVersions(const string & v1, const string & v2)
}
static void testCompareVersions()
{
#define TEST(v1, v2, n) assert( \
compareVersions(v1, v2) == n && compareVersions(v2, v1) == -n)
TEST("1.0", "2.3", -1);
TEST("2.1", "2.3", -1);
TEST("2.3", "2.3", 0);
TEST("2.5", "2.3", 1);
TEST("3.1", "2.3", 1);
TEST("2.3.1", "2.3", 1);
TEST("2.3.1", "2.3a", 1);
TEST("2.3pre1", "2.3", -1);
TEST("2.3pre3", "2.3pre12", -1);
TEST("2.3a", "2.3c", -1);
TEST("2.3pre1", "2.3c", -1);
TEST("2.3pre1", "2.3q", -1);
}
DrvNames drvNamesFromArgs(const Strings & opArgs)
{
DrvNames result;
File renamed without changes.
+7 -2
View File
@@ -1,10 +1,13 @@
%% Note: this SDF grammar is no longer used in the Nix expression
%% parser and may not be up to date.
definition
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Top level syntax.
module Nix
module Main
imports Nix-Exprs Nix-Layout
@@ -38,6 +41,7 @@ exports
"with" Expr ";" Expr -> Expr {cons("With")}
"rec" "{" Bind* "}" -> Expr {cons("Rec")}
"let" Bind* "in" Expr -> Expr {cons("Let")}
"let" "{" Bind* "}" -> Expr {cons("LetRec")}
"{" Bind* "}" -> Expr {cons("Attrs")}
@@ -102,7 +106,8 @@ exports
[0-9]+ -> Int
"\"" ~[\n\"]* "\"" -> Str
"\"" (~[\"\\] | ("\\" ~[]) )* "\"" -> Str
"''" (~[\"\\] | ("\\" ~[]) )* "''" -> Str
[a-zA-Z0-9\.\_\-\+]* ("/"[a-zA-Z0-9\.\_\-\+]+)+ -> Path
+12 -8
View File
@@ -3,8 +3,7 @@ init initNixExprHelpers
Pos | string int int | Pos |
NoPos | | Pos |
Function | ATermList Expr Pos | Expr |
Function1 | string Expr Pos | Expr |
Function | Pattern Expr Pos | Expr |
Assert | Expr Expr Pos | Expr |
With | Expr Expr Pos | Expr |
If | Expr Expr Expr | Expr |
@@ -46,6 +45,8 @@ Int | int | Expr |
Str | string ATermList | Expr |
Str | string | Expr | ObsoleteStr
ContextElem | string Expr | ContextElem |
# Internal to the parser, doesn't occur in ASTs.
IndStr | string | Expr |
@@ -67,25 +68,27 @@ PrimOp | int ATermBlob ATermList | Expr |
Attrs | ATermList | Expr |
Closed | Expr | Expr |
Rec | ATermList ATermList | Expr |
Bool | ATerm | Expr |
Bool | ATermBool | Expr |
Null | | Expr |
Bind | string Expr Pos | ATerm |
BindAttrPath | ATermList Expr Pos | ATerm | # desugared during parsing
Bind | string Expr | ATerm | ObsoleteBind
Inherit | Expr ATermList Pos | ATerm |
Scope | | Expr |
Formal | string ValidValues DefaultValue | ATerm |
VarPat | string | Pattern |
AttrsPat | ATermList ATermBool | Pattern | # bool = `...'
AtPat | Pattern Pattern | Pattern |
ValidValues | ATermList | ValidValues |
UnrestrictedValues | | ValidValues |
Formal | string DefaultValue | ATerm |
DefaultValue | Expr | DefaultValue |
NoDefaultValue | | DefaultValue |
True | | ATerm |
False | | ATerm |
True | | ATermBool |
False | | ATermBool |
PrimOpDef | int ATermBlob | ATerm |
@@ -93,3 +96,4 @@ AttrRHS | Expr Pos | ATerm |
eTrue = makeBool(makeTrue())
eFalse = makeBool(makeFalse())
sOverrides = toATerm("__overrides")
+79 -53
View File
@@ -6,6 +6,8 @@
#include "nixexpr-ast.hh"
#include "nixexpr-ast.cc"
#include <cstdlib>
namespace nix {
@@ -17,7 +19,7 @@ string showPos(ATerm pos)
if (matchNoPos(pos)) return "undefined position";
if (!matchPos(pos, path, line, column))
throw badTerm("position expected", pos);
return (format("`%1%', line %2%") % aterm2String(path) % line).str();
return (format("`%1%:%2%:%3%'") % aterm2String(path) % line % column).str();
}
@@ -108,6 +110,31 @@ Expr makeAttrs(const ATermMap & attrs)
}
static void varsBoundByPattern(ATermMap & map, Pattern pat)
{
ATerm name;
ATermList formals;
Pattern pat1, pat2;
ATermBool ellipsis;
/* Use makeRemoved() so that it can be used directly in
substitute(). */
if (matchVarPat(pat, name))
map.set(name, makeRemoved());
else if (matchAttrsPat(pat, formals, ellipsis)) {
for (ATermIterator i(formals); i; ++i) {
ATerm d1;
if (!matchFormal(*i, name, d1)) abort();
map.set(name, makeRemoved());
}
}
else if (matchAtPat(pat, pat1, pat2)) {
varsBoundByPattern(map, pat1);
varsBoundByPattern(map, pat2);
}
else abort();
}
Expr substitute(const Substitution & subs, Expr e)
{
checkInterrupt();
@@ -133,27 +160,17 @@ Expr substitute(const Substitution & subs, Expr e)
/* In case of a function, filter out all variables bound by this
function. */
ATermList formals;
Pattern pat;
ATerm body;
if (matchFunction(e, formals, body, pos)) {
ATermMap map(ATgetLength(formals));
for (ATermIterator i(formals); i; ++i) {
ATerm d1, d2;
if (!matchFormal(*i, name, d1, d2)) abort();
map.set(name, makeRemoved());
}
if (matchFunction(e, pat, body, pos)) {
ATermMap map(16);
varsBoundByPattern(map, pat);
Substitution subs2(&subs, &map);
return makeFunction(
(ATermList) substitute(subs2, (ATerm) formals),
(Pattern) substitute(subs2, (Expr) pat),
substitute(subs2, body), pos);
}
if (matchFunction1(e, name, body, pos)) {
ATermMap map(1);
map.set(name, makeRemoved());
return makeFunction1(name, substitute(Substitution(&subs, &map), body), pos);
}
/* Idem for a mutually recursive attribute set. */
ATermList rbnds, nrbnds;
if (matchRec(e, rbnds, nrbnds)) {
@@ -211,38 +228,25 @@ static void checkVarDefs2(set<Expr> & done, const ATermMap & defs, Expr e)
done.insert(e);
ATerm name, pos, value;
ATermList formals;
ATerm with, body;
ATermList rbnds, nrbnds;
Pattern pat;
if (matchVar(e, name)) {
/* Closed terms don't have free variables, so we don't have to
check by definition. */
if (matchClosed(e, value)) return;
else if (matchVar(e, name)) {
if (!defs.get(name))
throw EvalError(format("undefined variable `%1%'")
% aterm2String(name));
}
else if (matchFunction(e, formals, body, pos)) {
else if (matchFunction(e, pat, body, pos)) {
ATermMap defs2(defs);
for (ATermIterator i(formals); i; ++i) {
ATerm d1, d2;
if (!matchFormal(*i, name, d1, d2)) abort();
defs2.set(name, (ATerm) ATempty);
}
for (ATermIterator i(formals); i; ++i) {
ATerm valids, deflt;
set<Expr> done2;
if (!matchFormal(*i, name, valids, deflt)) abort();
checkVarDefs2(done, defs, valids);
checkVarDefs2(done2, defs2, deflt);
}
set<Expr> done2;
checkVarDefs2(done2, defs2, body);
}
else if (matchFunction1(e, name, body, pos)) {
ATermMap defs2(defs);
defs2.set(name, (ATerm) ATempty);
varsBoundByPattern(defs2, pat);
set<Expr> done2;
checkVarDefs2(done2, defs2, pat);
checkVarDefs2(done2, defs2, body);
}
@@ -332,41 +336,63 @@ Expr makeBool(bool b)
}
bool matchStr(Expr e, string & s, PathSet & context)
bool matchStr(Expr e, string & s, ATermList & context)
{
ATermList l;
ATerm s_;
if (!matchStr(e, s_, l)) return false;
if (!matchStr(e, s_, context)) return false;
s = aterm2String(s_);
for (ATermIterator i(l); i; ++i)
context.insert(aterm2String(*i));
return true;
}
Expr makeStr(const string & s, const PathSet & context)
bool matchStr(Expr e, string & s, Context & context)
{
return makeStr(toATerm(s), toATermList(context));
ATermList c;
if (!matchStr(e, s, c)) return false;
matchContext(c, context);
return true;
}
Expr makeStr(const string & s, ATermList context)
{
return makeStr(toATerm(s), context);
}
Expr makeStr(const string & s, const Context & context)
{
ATermList l = ATempty;
/* !!! should define a canonical ordering of context elements. */
foreach (Context::const_iterator, i, context)
l = ATinsert(l, makeContextElem(i->key, i->value));
return makeStr(s, l);
}
void matchContext(ATermList context, Context & result)
{
for (ATermIterator i(context); i; ++i) {
ATerm s, e;
if (!matchContextElem(*i, s, e)) abort();
result.set(s, e);
}
}
string showType(Expr e)
{
ATerm t1, t2, t3;
ATerm t1, t2;
ATermList l1;
ATermBlob b1;
int i1;
Pattern p1;
if (matchStr(e, t1, l1)) return "a string";
if (matchPath(e, t1)) return "a path";
if (matchNull(e)) return "null";
if (matchInt(e, i1)) return "an integer";
if (matchBool(e, t1)) return "a boolean";
if (matchFunction(e, l1, t1, t2)) return "a function";
if (matchFunction1(e, t1, t2, t3)) return "a function";
if (matchFunction(e, p1, t1, t2)) return "a function";
if (matchAttrs(e, l1)) return "an attribute set";
if (matchList(e, l1)) return "a list";
if (matchPrimOp(e, i1, b1, l1)) return "a partially applied built-in function";
@@ -376,7 +402,7 @@ string showType(Expr e)
string showValue(Expr e)
{
PathSet context;
ATermList context;
string s;
ATerm s2;
int i;
+15 -7
View File
@@ -11,6 +11,7 @@ namespace nix {
MakeError(EvalError, Error)
MakeError(ParseError, Error)
MakeError(AssertionError, EvalError)
MakeError(ThrownError, AssertionError)
MakeError(Abort, EvalError)
@@ -21,11 +22,11 @@ MakeError(TypeError, EvalError)
property of the ATerm library allows us to implement caching of
normals forms efficiently. */
typedef ATerm Expr;
typedef ATerm DefaultValue;
typedef ATerm ValidValues;
typedef ATerm Pos;
typedef ATerm Pattern;
typedef ATerm ATermBool;
typedef ATerm ContextElem;
/* A STL vector of ATerms. Should be used with great care since it's
@@ -104,11 +105,18 @@ Expr canonicaliseExpr(Expr e);
Expr makeBool(bool b);
/* Manipulation of Str() nodes. Note: matchStr() does not clear
context! */
bool matchStr(Expr e, string & s, PathSet & context);
/* Manipulation of Str() nodes. */
typedef ATermMap Context;
Expr makeStr(const string & s, const PathSet & context = PathSet());
bool matchStr(Expr e, string & s, ATermList & context);
bool matchStr(Expr e, string & s, Context & context);
Expr makeStr(const string & s, ATermList context = ATempty);
Expr makeStr(const string & s, const Context & context);
void matchContext(ATermList context, Context & result);
/* Showing types, values. */
+161 -98
View File
@@ -25,6 +25,7 @@
#include "parser-tab.hh"
#include "lexer-tab.hh"
#define YYSTYPE YYSTYPE // workaround a bug in Bison 2.4
#include "nixexpr.hh"
#include "nixexpr-ast.hh"
@@ -43,28 +44,128 @@ struct ParseData
Path path;
string error;
};
static Expr fixAttrs(int recursive, ATermList as)
static string showAttrPath(ATermList attrPath)
{
ATermList bs = ATempty, cs = ATempty;
ATermList * is = recursive ? &cs : &bs;
string s;
for (ATermIterator i(attrPath); i; ++i) {
if (!s.empty()) s += '.';
s += aterm2String(*i);
}
return s;
}
struct Tree
{
Expr leaf; ATerm pos; bool recursive;
typedef std::map<ATerm, Tree> Children;
Children children;
Tree() { leaf = 0; recursive = true; }
};
static ATermList buildAttrs(const Tree & t, ATermList & nonrec)
{
ATermList res = ATempty;
for (Tree::Children::const_reverse_iterator i = t.children.rbegin();
i != t.children.rend(); ++i)
if (!i->second.recursive)
nonrec = ATinsert(nonrec, makeBind(i->first, i->second.leaf, i->second.pos));
else
res = ATinsert(res, i->second.leaf
? makeBind(i->first, i->second.leaf, i->second.pos)
: makeBind(i->first, makeAttrs(buildAttrs(i->second, nonrec)), makeNoPos()));
return res;
}
static Expr fixAttrs(bool recursive, ATermList as)
{
Tree attrs;
for (ATermIterator i(as); i; ++i) {
ATermList names;
Expr src;
ATerm pos;
ATermList names, attrPath; Expr src, e; ATerm name, pos;
if (matchInherit(*i, src, names, pos)) {
bool fromScope = matchScope(src);
for (ATermIterator j(names); j; ++j) {
Expr rhs = fromScope ? makeVar(*j) : makeSelect(src, *j);
*is = ATinsert(*is, makeBind(*j, rhs, pos));
if (attrs.children.find(*j) != attrs.children.end())
throw ParseError(format("duplicate definition of attribute `%1%' at %2%")
% showAttrPath(ATmakeList1(*j)) % showPos(pos));
Tree & t(attrs.children[*j]);
t.leaf = fromScope ? makeVar(*j) : makeSelect(src, *j);
t.pos = pos;
if (recursive && fromScope) t.recursive = false;
}
} else bs = ATinsert(bs, *i);
}
else if (matchBindAttrPath(*i, attrPath, e, pos)) {
Tree * t(&attrs);
for (ATermIterator j(attrPath); j; ) {
name = *j; ++j;
if (t->leaf) throw ParseError(format("attribute set containing `%1%' at %2% already defined at %3%")
% showAttrPath(attrPath) % showPos(pos) % showPos (t->pos));
t = &(t->children[name]);
}
if (t->leaf)
throw ParseError(format("duplicate definition of attribute `%1%' at %2% and %3%")
% showAttrPath(attrPath) % showPos(pos) % showPos (t->pos));
if (!t->children.empty())
throw ParseError(format("duplicate definition of attribute `%1%' at %2%")
% showAttrPath(attrPath) % showPos(pos));
t->leaf = e; t->pos = pos;
}
else abort(); /* can't happen */
}
if (recursive)
return makeRec(bs, cs);
else
return makeAttrs(bs);
ATermList nonrec = ATempty;
ATermList rec = buildAttrs(attrs, nonrec);
return recursive ? makeRec(rec, nonrec) : makeAttrs(rec);
}
static void checkPatternVars(ATerm pos, ATermMap & map, Pattern pat)
{
ATerm name;
ATermList formals;
Pattern pat1, pat2;
ATermBool ellipsis;
if (matchVarPat(pat, name)) {
if (map.get(name))
throw ParseError(format("duplicate formal function argument `%1%' at %2%")
% aterm2String(name) % showPos(pos));
map.set(name, name);
}
else if (matchAttrsPat(pat, formals, ellipsis)) {
for (ATermIterator i(formals); i; ++i) {
ATerm d1;
if (!matchFormal(*i, name, d1)) abort();
if (map.get(name))
throw ParseError(format("duplicate formal function argument `%1%' at %2%")
% aterm2String(name) % showPos(pos));
map.set(name, name);
}
}
else if (matchAtPat(pat, pat1, pat2)) {
checkPatternVars(pos, map, pat1);
checkPatternVars(pos, map, pat2);
}
else abort();
}
static void checkPatternVars(ATerm pos, Pattern pat)
{
ATermMap map;
checkPatternVars(pos, map, pat);
}
@@ -146,7 +247,7 @@ static Expr stripIndentation(ATermList es)
/* Remove the last line if it is empty and consists only of
spaces. */
if (n == 1) {
unsigned int p = s2.find_last_of('\n');
string::size_type p = s2.find_last_of('\n');
if (p != string::npos && s2.find_first_not_of(' ', p + 1) == string::npos)
s2 = string(s2, 0, p + 1);
}
@@ -208,15 +309,22 @@ static void freeAndUnprotect(void * p)
%union {
ATerm t;
ATermList ts;
struct {
ATermList formals;
bool ellipsis;
} formals;
}
%type <t> start expr expr_function expr_if expr_op
%type <t> expr_app expr_select expr_simple bind inheritsrc formal
%type <ts> binds ids expr_list formals string_parts ind_string_parts
%type <t> pattern pattern2
%type <ts> binds ids attrpath expr_list string_parts ind_string_parts
%type <formals> formals
%token <t> ID INT STR IND_STR PATH URI
%token IF THEN ELSE ASSERT WITH LET IN REC INHERIT EQ NEQ AND OR IMPL
%token DOLLAR_CURLY /* == ${ */
%token IND_STRING_OPEN IND_STRING_CLOSE
%token ELLIPSIS
%nonassoc IMPL
%left OR
@@ -236,16 +344,14 @@ start: expr { data->result = $1; };
expr: expr_function;
expr_function
: '{' formals '}' ':' expr_function
{ $$ = makeFunction($2, $5, CUR_POS); }
| ID ':' expr_function
{ $$ = makeFunction1($1, $3, CUR_POS); }
: pattern ':' expr_function
{ checkPatternVars(CUR_POS, $1); $$ = makeFunction($1, $3, CUR_POS); }
| ASSERT expr ';' expr_function
{ $$ = makeAssert($2, $4, CUR_POS); }
| WITH expr ';' expr_function
{ $$ = makeWith($2, $4, CUR_POS); }
| LET binds IN expr_function
{ $$ = makeSelect(fixAttrs(1, ATinsert($2, makeBind(toATerm("<let-body>"), $4, CUR_POS))), toATerm("<let-body>")); }
{ $$ = makeSelect(fixAttrs(true, ATinsert($2, makeBindAttrPath(ATmakeList1(toATerm("<let-body>")), $4, CUR_POS))), toATerm("<let-body>")); }
| expr_if
;
@@ -300,12 +406,12 @@ expr_simple
/* Let expressions `let {..., body = ...}' are just desugared
into `(rec {..., body = ...}).body'. */
| LET '{' binds '}'
{ $$ = makeSelect(fixAttrs(1, $3), toATerm("body")); }
{ $$ = makeSelect(fixAttrs(true, $3), toATerm("body")); }
| REC '{' binds '}'
{ $$ = fixAttrs(1, $3); }
{ $$ = fixAttrs(true, $3); }
| '{' binds '}'
{ $$ = fixAttrs(0, $2); }
| '[' expr_list ']' { $$ = makeList($2); }
{ $$ = fixAttrs(false, $2); }
| '[' expr_list ']' { $$ = makeList(ATreverse($2)); }
;
string_parts
@@ -320,14 +426,24 @@ ind_string_parts
| { $$ = ATempty; }
;
pattern
: pattern2 '@' pattern { $$ = makeAtPat($1, $3); }
| pattern2
;
pattern2
: ID { $$ = makeVarPat($1); }
| '{' formals '}' { $$ = makeAttrsPat($2.formals, $2.ellipsis ? eTrue : eFalse); }
;
binds
: binds bind { $$ = ATinsert($1, $2); }
| { $$ = ATempty; }
;
bind
: ID '=' expr ';'
{ $$ = makeBind($1, $3, CUR_POS); }
: attrpath '=' expr ';'
{ $$ = makeBindAttrPath(ATreverse($1), $3, CUR_POS); }
| INHERIT inheritsrc ids ';'
{ $$ = makeInherit($2, $3, CUR_POS); }
;
@@ -339,24 +455,30 @@ inheritsrc
ids: ids ID { $$ = ATinsert($1, $2); } | { $$ = ATempty; };
attrpath
: attrpath '.' ID { $$ = ATinsert($1, $3); }
| ID { $$ = ATmakeList1($1); }
;
expr_list
: expr_select expr_list { $$ = ATinsert($2, $1); }
/* yes, this is right-recursive, but it doesn't matter since
otherwise we would need ATreverse which requires unbounded
stack space */
: expr_list expr_select { $$ = ATinsert($1, $2); }
| { $$ = ATempty; }
;
formals
: formal ',' formals { $$ = ATinsert($3, $1); } /* idem - right recursive */
| formal { $$ = ATinsert(ATempty, $1); }
| { $$ = ATempty; }
: formal ',' formals /* !!! right recursive */
{ $$.formals = ATinsert($3.formals, $1); $$.ellipsis = $3.ellipsis; }
| formal
{ $$.formals = ATinsert(ATempty, $1); $$.ellipsis = false; }
|
{ $$.formals = ATempty; $$.ellipsis = false; }
| ELLIPSIS
{ $$.formals = ATempty; $$.ellipsis = true; }
;
formal
: ID { $$ = makeFormal($1, makeUnrestrictedValues(), makeNoDefaultValue()); }
| ID ':' '[' expr_list ']' { $$ = makeFormal($1, makeValidValues($4), makeNoDefaultValue()); }
| ID '?' expr { $$ = makeFormal($1, makeUnrestrictedValues(), makeDefaultValue($3)); }
: ID { $$ = makeFormal($1, makeNoDefaultValue()); }
| ID '?' expr { $$ = makeFormal($1, makeDefaultValue($3)); }
;
%%
@@ -373,63 +495,6 @@ formal
namespace nix {
static void checkAttrs(ATermMap & names, ATermList bnds)
{
for (ATermIterator i(bnds); i; ++i) {
ATerm name;
Expr e;
ATerm pos;
if (!matchBind(*i, name, e, pos)) abort(); /* can't happen */
if (names.get(name))
throw EvalError(format("duplicate attribute `%1%' at %2%")
% aterm2String(name) % showPos(pos));
names.set(name, name);
}
}
static void checkAttrSets(ATerm e)
{
ATermList formals;
ATerm body, pos;
if (matchFunction(e, formals, body, pos)) {
ATermMap names(ATgetLength(formals));
for (ATermIterator i(formals); i; ++i) {
ATerm name;
ATerm d1, d2;
if (!matchFormal(*i, name, d1, d2)) abort();
if (names.get(name))
throw EvalError(format("duplicate formal function argument `%1%' at %2%")
% aterm2String(name) % showPos(pos));
names.set(name, name);
}
}
ATermList bnds;
if (matchAttrs(e, bnds)) {
ATermMap names(ATgetLength(bnds));
checkAttrs(names, bnds);
}
ATermList rbnds, nrbnds;
if (matchRec(e, rbnds, nrbnds)) {
ATermMap names(ATgetLength(rbnds) + ATgetLength(nrbnds));
checkAttrs(names, rbnds);
checkAttrs(names, nrbnds);
}
if (ATgetType(e) == AT_APPL) {
int arity = ATgetArity(ATgetAFun(e));
for (int i = 0; i < arity; ++i)
checkAttrSets(ATgetArgument(e, i));
}
else if (ATgetType(e) == AT_LIST)
for (ATermIterator i((ATermList) e); i; ++i)
checkAttrSets(*i);
}
static Expr parse(EvalState & state,
const char * text, const Path & path,
const Path & basePath)
@@ -444,16 +509,14 @@ static Expr parse(EvalState & state,
int res = yyparse(scanner, &data);
yylex_destroy(scanner);
if (res) throw EvalError(data.error);
if (res) throw ParseError(data.error);
try {
checkVarDefs(state.primOps, data.result);
} catch (Error & e) {
throw EvalError(format("%1%, in `%2%'") % e.msg() % path);
throw ParseError(format("%1%, in `%2%'") % e.msg() % path);
}
checkAttrSets(data.result);
return data.result;
}
+388 -201
View File
@@ -6,6 +6,8 @@
#include "archive.hh"
#include "expr-to-xml.hh"
#include "nixexpr-ast.hh"
#include "parser.hh"
#include "names.hh"
#include <sys/types.h>
#include <sys/stat.h>
@@ -91,16 +93,17 @@ static Expr prim_currentTime(EvalState & state, const ATermVector & args)
argument. */
static Expr prim_import(EvalState & state, const ATermVector & args)
{
PathSet context;
Context context;
Path path = coerceToPath(state, args[0], context);
for (PathSet::iterator i = context.begin(); i != context.end(); ++i) {
assert(isStorePath(*i));
if (!store->isValidPath(*i))
foreach (Context::const_iterator, i, context) {
Path p = aterm2String(i->key);
assert(isStorePath(p));
if (!store->isValidPath(p))
throw EvalError(format("cannot import `%1%', since path `%2%' is not valid")
% path % *i);
if (isDerivation(*i))
store->buildDerivations(singleton<PathSet>(*i));
% path % p);
if (isDerivation(p))
store->buildDerivations(singleton<PathSet>(p));
}
return evalFile(state, path);
@@ -118,67 +121,34 @@ static Expr prim_isNull(EvalState & state, const ATermVector & args)
static Expr prim_isFunction(EvalState & state, const ATermVector & args)
{
Expr e = evalExpr(state, args[0]);
ATermList formals;
ATerm name, body, pos;
return makeBool(
matchFunction(e, formals, body, pos) ||
matchFunction1(e, name, body, pos));
Pattern pat;
ATerm body, pos;
return makeBool(matchFunction(e, pat, body, pos));
}
static Path findDependency(Path dir, string dep)
/* Determine whether the argument is an Int. */
static Expr prim_isInt(EvalState & state, const ATermVector & args)
{
if (dep[0] == '/') throw EvalError(
format("illegal absolute dependency `%1%'") % dep);
Path p = canonPath(dir + "/" + dep);
if (pathExists(p))
return p;
else
return "";
int i;
return makeBool(matchInt(evalExpr(state, args[0]), i));
}
/* Make path `p' relative to directory `pivot'. E.g.,
relativise("/a/b/c", "a/b/x/y") => "../x/y". Both input paths
should be in absolute canonical form. */
static string relativise(Path pivot, Path p)
/* Determine whether the argument is an String. */
static Expr prim_isString(EvalState & state, const ATermVector & args)
{
assert(pivot.size() > 0 && pivot[0] == '/');
assert(p.size() > 0 && p[0] == '/');
if (pivot == p) return ".";
/* `p' is in `pivot'? */
Path pivot2 = pivot + "/";
if (p.substr(0, pivot2.size()) == pivot2) {
return p.substr(pivot2.size());
}
/* Otherwise, `p' is in a parent of `pivot'. Find up till which
path component `p' and `pivot' match, and add an appropriate
number of `..' components. */
string::size_type i = 1;
while (1) {
string::size_type j = pivot.find('/', i);
if (j == string::npos) break;
j++;
if (pivot.substr(0, j) != p.substr(0, j)) break;
i = j;
}
string prefix;
unsigned int slashes = count(pivot.begin() + i, pivot.end(), '/') + 1;
while (slashes--) {
prefix += "../";
}
return prefix + p.substr(i);
string s;
ATermList context;
return makeBool(matchStr(evalExpr(state, args[0]), s, context));
}
/* Determine whether the argument is an Bool. */
static Expr prim_isBool(EvalState & state, const ATermVector & args)
{
ATermBool b;
return makeBool(matchBool(evalExpr(state, args[0]), b));
}
static Expr prim_dependencyClosure(EvalState & state, const ATermVector & args)
static Expr prim_genericClosure(EvalState & state, const ATermVector & args)
{
startNest(nest, lvlDebug, "finding dependencies");
@@ -189,93 +159,46 @@ static Expr prim_dependencyClosure(EvalState & state, const ATermVector & args)
if (!startSet) throw EvalError("attribute `startSet' required");
ATermList startSet2 = evalList(state, startSet);
Path pivot;
PathSet workSet;
for (ATermIterator i(startSet2); i; ++i) {
PathSet context; /* !!! what to do? */
Path p = coerceToPath(state, *i, context);
workSet.insert(p);
pivot = dirOf(p);
}
set<Expr> workSet; // !!! gc roots
for (ATermIterator i(startSet2); i; ++i) workSet.insert(*i);
/* Get the search path. */
PathSet searchPath;
Expr e = queryAttr(attrs, "searchPath");
if (e) {
ATermList list = evalList(state, e);
for (ATermIterator i(list); i; ++i) {
PathSet context; /* !!! what to do? */
Path p = coerceToPath(state, *i, context);
searchPath.insert(p);
}
}
Expr scanner = queryAttr(attrs, "scanner");
if (!scanner) throw EvalError("attribute `scanner' required");
/* Get the operator. */
Expr op = queryAttr(attrs, "operator");
if (!op) throw EvalError("attribute `operator' required");
/* Construct the dependency closure by querying the dependency of
each path in `workSet', adding the dependencies to
`workSet'. */
PathSet doneSet;
/* Construct the closure by applying the operator to element of
`workSet', adding the result to `workSet', continuing until
no new elements are found. */
ATermList res = ATempty;
set<Expr> doneKeys; // !!! gc roots
while (!workSet.empty()) {
Path path = *(workSet.begin());
workSet.erase(path);
Expr e = *(workSet.begin());
workSet.erase(e);
if (doneSet.find(path) != doneSet.end()) continue;
doneSet.insert(path);
e = strictEvalExpr(state, e);
try {
/* Call the `scanner' function with `path' as argument. */
debug(format("finding dependencies in `%1%'") % path);
ATermList deps = evalList(state, makeCall(scanner, makeStr(path)));
Expr key = queryAttr(e, "key");
if (!key) throw EvalError("attribute `key' required");
/* Try to find the dependencies relative to the `path'. */
for (ATermIterator i(deps); i; ++i) {
string s = evalStringNoCtx(state, *i);
Path dep = findDependency(dirOf(path), s);
if (doneKeys.find(key) != doneKeys.end()) continue;
doneKeys.insert(key);
res = ATinsert(res, e);
/* Call the `operator' function with `e' as argument. */
ATermList res = evalList(state, makeCall(op, e));
if (dep == "") {
for (PathSet::iterator j = searchPath.begin();
j != searchPath.end(); ++j)
{
dep = findDependency(*j, s);
if (dep != "") break;
}
}
if (dep == "")
debug(format("did NOT find dependency `%1%'") % s);
else {
debug(format("found dependency `%1%'") % dep);
workSet.insert(dep);
}
}
} catch (Error & e) {
e.addPrefix(format("while finding dependencies in `%1%':\n")
% path);
throw;
}
/* Try to find the dependencies relative to the `path'. */
for (ATermIterator i(res); i; ++i)
workSet.insert(evalExpr(state, *i));
}
/* Return a list of the dependencies we've just found. */
ATermList deps = ATempty;
for (PathSet::iterator i = doneSet.begin(); i != doneSet.end(); ++i) {
deps = ATinsert(deps, makeStr(relativise(pivot, *i)));
deps = ATinsert(deps, makeStr(*i));
}
debug(format("dependency list is `%1%'") % makeList(deps));
return makeList(deps);
return makeList(res);
}
static Expr prim_abort(EvalState & state, const ATermVector & args)
{
PathSet context;
ATermList context;
throw Abort(format("evaluation aborted with the following error message: `%1%'") %
evalString(state, args[0], context));
}
@@ -283,12 +206,42 @@ static Expr prim_abort(EvalState & state, const ATermVector & args)
static Expr prim_throw(EvalState & state, const ATermVector & args)
{
PathSet context;
throw ThrownError(format("user-thrown exception: `%1%'") %
ATermList context;
throw ThrownError(format("user-thrown exception: %1%") %
evalString(state, args[0], context));
}
static Expr prim_addErrorContext(EvalState & state, const ATermVector & args)
{
ATermList context;
try {
return evalExpr(state, args[1]);
} catch (Error & e) {
e.addPrefix(format("%1%\n") %
evalString(state, args[0], context));
throw;
}
}
/* Try evaluating the argument. Success => {success=true; value=something;},
* else => {success=false; value=false;} */
static Expr prim_tryEval(EvalState & state, const ATermVector & args)
{
ATermMap res = ATermMap();
try {
Expr val = evalExpr(state, args[0]);
res.set(toATerm("value"), makeAttrRHS(val, makeNoPos()));
res.set(toATerm("success"), makeAttrRHS(eTrue, makeNoPos()));
} catch (AssertionError & e) {
printMsg(lvlDebug, format("tryEval caught an error: %1%: %2%") % e.prefix() % e.msg());
res.set(toATerm("value"), makeAttrRHS(eFalse, makeNoPos()));
res.set(toATerm("success"), makeAttrRHS(eFalse, makeNoPos()));
}
return makeAttrs(res);
}
/* Return an environment variable. Use with care. */
static Expr prim_getEnv(EvalState & state, const ATermVector & args)
{
@@ -304,25 +257,29 @@ static Expr prim_getEnv(EvalState & state, const ATermVector & args)
static Expr prim_trace(EvalState & state, const ATermVector & args)
{
Expr e = evalExpr(state, args[0]);
printMsg(lvlError, format("trace: %1%") % e);
string s;
ATermList context;
if (matchStr(e, s, context))
printMsg(lvlError, format("trace: %1%") % s);
else
printMsg(lvlError, format("trace: %1%") % e);
return evalExpr(state, args[1]);
}
static Expr prim_relativise(EvalState & state, const ATermVector & args)
{
PathSet context; /* !!! what to do? */
Path pivot = coerceToPath(state, args[0], context);
Path path = coerceToPath(state, args[1], context);
return makeStr(relativise(pivot, path));
}
/*************************************************************
* Derivations
*************************************************************/
static bool isFixedOutput(const Derivation & drv)
{
return drv.outputs.size() == 1 &&
drv.outputs.begin()->first == "out" &&
drv.outputs.begin()->second.hash != "";
}
/* Returns the hash of a derivation modulo fixed-output
subderivations. A fixed-output derivation is a derivation with one
output (`out') for which an expected hash and hash algorithm are
@@ -335,35 +292,29 @@ static Expr prim_relativise(EvalState & state, const ATermVector & args)
function, we do not want to rebuild everything depending on it
(after all, (the hash of) the file being downloaded is unchanged).
So the *output paths* should not change. On the other hand, the
*derivation store expression paths* should change to reflect the
new dependency graph.
*derivation paths* should change to reflect the new dependency
graph.
That's what this function does: it returns a hash which is just the
of the derivation ATerm, except that any input store expression
hash of the derivation ATerm, except that any input derivation
paths have been replaced by the result of a recursive call to this
function, and that for fixed-output derivations we return
(basically) its outputHash. */
function, and that for fixed-output derivations we return a hash of
its output path. */
static Hash hashDerivationModulo(EvalState & state, Derivation drv)
{
/* Return a fixed hash for fixed-output derivations. */
if (drv.outputs.size() == 1) {
if (isFixedOutput(drv)) {
DerivationOutputs::const_iterator i = drv.outputs.begin();
if (i->first == "out" &&
i->second.hash != "")
{
return hashString(htSHA256, "fixed:out:"
+ i->second.hashAlgo + ":"
+ i->second.hash + ":"
+ i->second.path);
}
return hashString(htSHA256, "fixed:out:"
+ i->second.hashAlgo + ":"
+ i->second.hash + ":"
+ i->second.path);
}
/* For other derivations, replace the inputs paths with recursive
calls to this function.*/
DerivationInputs inputs2;
for (DerivationInputs::iterator i = drv.inputDrvs.begin();
i != drv.inputDrvs.end(); ++i)
{
foreach (DerivationInputs::const_iterator, i, drv.inputDrvs) {
Hash h = state.drvHashes[i->first];
if (h.type == htUnknown) {
Derivation drv2 = derivationFromPath(i->first);
@@ -410,10 +361,9 @@ static Expr prim_derivationStrict(EvalState & state, const ATermVector & args)
/* Build the derivation expression by processing the attributes. */
Derivation drv;
PathSet context;
Context context;
string outputHash;
string outputHashAlgo;
string outputHash, outputHashAlgo;
bool outputHashRecursive = false;
for (ATermMap::const_iterator i = attrs.begin(); i != attrs.end(); ++i) {
@@ -472,13 +422,40 @@ static Expr prim_derivationStrict(EvalState & state, const ATermVector & args)
/* Everything in the context of the strings in the derivation
attributes should be added as dependencies of the resulting
derivation. */
for (PathSet::iterator i = context.begin(); i != context.end(); ++i) {
debug(format("derivation uses `%1%'") % *i);
assert(isStorePath(*i));
if (isDerivation(*i))
drv.inputDrvs[*i] = singleton<StringSet>("out");
foreach (Context::const_iterator, i, context) {
Path path = aterm2String(i->key);
/* Paths marked with `=' denote that the path of a derivation
is explicitly passed to the builder. Since that allows the
builder to gain access to every path in the dependency
graph of the derivation (including all outputs), all paths
in the graph must be added to this derivation's list of
inputs to ensure that they are available when the builder
runs. */
if (path.at(0) == '=') {
path = string(path, 1);
PathSet refs; computeFSClosure(path, refs);
foreach (PathSet::iterator, j, refs) {
drv.inputSrcs.insert(*j);
if (isDerivation(*j))
drv.inputDrvs[*j] = singleton<StringSet>("out");
}
}
/* See prim_unsafeDiscardOutputDependency. */
bool useDrvAsSrc = false;
if (path.at(0) == '~') {
path = string(path, 1);
useDrvAsSrc = true;
}
assert(isStorePath(path));
debug(format("derivation uses `%1%'") % path);
if (!useDrvAsSrc && isDerivation(path))
drv.inputDrvs[path] = singleton<StringSet>("out");
else
drv.inputSrcs.insert(*i);
drv.inputSrcs.insert(path);
}
/* Do we have all required attributes? */
@@ -488,6 +465,7 @@ static Expr prim_derivationStrict(EvalState & state, const ATermVector & args)
throw EvalError("required attribute `system' missing");
/* If an output hash was given, check it. */
Path outPath;
if (outputHash == "")
outputHashAlgo = "";
else {
@@ -506,6 +484,7 @@ static Expr prim_derivationStrict(EvalState & state, const ATermVector & args)
% outputHash % outputHashAlgo);
string s = outputHash;
outputHash = printHash(h);
outPath = makeFixedOutputPath(outputHashRecursive, ht, h, drvName);
if (outputHashRecursive) outputHashAlgo = "r:" + outputHashAlgo;
}
@@ -521,13 +500,12 @@ static Expr prim_derivationStrict(EvalState & state, const ATermVector & args)
have an empty value. This ensures that changes in the set of
output names do get reflected in the hash. */
drv.env["out"] = "";
drv.outputs["out"] =
DerivationOutput("", outputHashAlgo, outputHash);
drv.outputs["out"] = DerivationOutput("", outputHashAlgo, outputHash);
/* Use the masked derivation expression to compute the output
path. */
Path outPath = makeStorePath("output:out",
hashDerivationModulo(state, drv), drvName);
if (outPath == "")
outPath = makeStorePath("output:out", hashDerivationModulo(state, drv), drvName);
/* Construct the final derivation store expression. */
drv.env["out"] = outPath;
@@ -548,9 +526,9 @@ static Expr prim_derivationStrict(EvalState & state, const ATermVector & args)
/* !!! assumes a single output */
ATermMap outAttrs(2);
outAttrs.set(toATerm("outPath"),
makeAttrRHS(makeStr(outPath, singleton<PathSet>(drvPath)), makeNoPos()));
makeAttrRHS(makeStr(outPath, ATmakeList1(makeContextElem(toATerm(drvPath), makeNull()))), makeNoPos()));
outAttrs.set(toATerm("drvPath"),
makeAttrRHS(makeStr(drvPath, singleton<PathSet>(drvPath)), makeNoPos()));
makeAttrRHS(makeStr(drvPath, ATmakeList1(makeContextElem(toATerm("=" + drvPath), makeNull()))), makeNoPos()));
return makeAttrs(outAttrs);
}
@@ -584,17 +562,39 @@ static Expr prim_derivationLazy(EvalState & state, const ATermVector & args)
/* Convert the argument to a path. !!! obsolete? */
static Expr prim_toPath(EvalState & state, const ATermVector & args)
{
PathSet context;
Context context;
string path = coerceToPath(state, args[0], context);
return makeStr(canonPath(path), context);
}
/* Allow a valid store path to be used in an expression. This is
useful in some generated expressions such as in nix-push, which
generates a call to a function with an already existing store path
as argument. You don't want to use `toPath' here because it copies
the path to the Nix store, which yields a copy like
/nix/store/newhash-oldhash-oldname. In the past, `toPath' had
special case behaviour for store paths, but that created weird
corner cases. */
static Expr prim_storePath(EvalState & state, const ATermVector & args)
{
Context context;
Path path = canonPath(coerceToPath(state, args[0], context));
if (!isInStore(path))
throw EvalError(format("path `%1%' is not in the Nix store") % path);
Path path2 = toStorePath(path);
if (!store->isValidPath(path2))
throw EvalError(format("store path `%1%' is not valid") % path2);
context.set(toATerm(path2), makeNull());
return makeStr(path, context);
}
static Expr prim_pathExists(EvalState & state, const ATermVector & args)
{
PathSet context;
Context context;
Path path = coerceToPath(state, args[0], context);
if (!context.empty())
if (context.size() != 0)
throw EvalError(format("string `%1%' cannot refer to other paths") % path);
return makeBool(pathExists(path));
}
@@ -604,7 +604,7 @@ static Expr prim_pathExists(EvalState & state, const ATermVector & args)
following the last slash. */
static Expr prim_baseNameOf(EvalState & state, const ATermVector & args)
{
PathSet context;
Context context;
return makeStr(baseNameOf(coerceToString(state, args[0], context)), context);
}
@@ -614,7 +614,7 @@ static Expr prim_baseNameOf(EvalState & state, const ATermVector & args)
of the argument. */
static Expr prim_dirOf(EvalState & state, const ATermVector & args)
{
PathSet context;
Context context;
Expr e = evalExpr(state, args[0]); ATerm dummy;
bool isPath = matchPath(e, dummy);
Path dir = dirOf(coerceToPath(state, e, context));
@@ -625,9 +625,9 @@ static Expr prim_dirOf(EvalState & state, const ATermVector & args)
/* Return the contents of a file as a string. */
static Expr prim_readFile(EvalState & state, const ATermVector & args)
{
PathSet context;
Context context;
Path path = coerceToPath(state, args[0], context);
if (!context.empty())
if (context.size() != 0)
throw EvalError(format("string `%1%' cannot refer to other paths") % path);
return makeStr(readFile(path));
}
@@ -644,7 +644,7 @@ static Expr prim_readFile(EvalState & state, const ATermVector & args)
static Expr prim_toXML(EvalState & state, const ATermVector & args)
{
std::ostringstream out;
PathSet context;
Context context;
printTermAsXML(strictEvalExpr(state, args[0]), out, context);
return makeStr(out.str(), context);
}
@@ -654,16 +654,19 @@ static Expr prim_toXML(EvalState & state, const ATermVector & args)
as an input by derivations. */
static Expr prim_toFile(EvalState & state, const ATermVector & args)
{
PathSet context;
ATermList context;
string name = evalStringNoCtx(state, args[0]);
string contents = evalString(state, args[1], context);
PathSet refs;
for (PathSet::iterator i = context.begin(); i != context.end(); ++i) {
if (isDerivation(*i))
Context context2; matchContext(context, context2);
foreach (Context::const_iterator, i, context2) {
Path path = aterm2String(i->key);
if (path.at(0) == '=') path = string(path, 1);
if (isDerivation(path))
throw EvalError(format("in `toFile': the file `%1%' cannot refer to derivation outputs") % name);
refs.insert(*i);
refs.insert(path);
}
Path storePath = readOnlyMode
@@ -674,7 +677,7 @@ static Expr prim_toFile(EvalState & state, const ATermVector & args)
result, since `storePath' itself has references to the paths
used in args[1]. */
return makeStr(storePath, singleton<PathSet>(storePath));
return makeStr(storePath, ATmakeList1(makeContextElem(toATerm(storePath), makeNull())));
}
@@ -711,18 +714,18 @@ struct FilterFromExpr : PathFilter
static Expr prim_filterSource(EvalState & state, const ATermVector & args)
{
PathSet context;
Context context;
Path path = coerceToPath(state, args[1], context);
if (!context.empty())
if (context.size() != 0)
throw EvalError(format("string `%1%' cannot refer to other paths") % path);
FilterFromExpr filter(state, args[0]);
Path dstPath = readOnlyMode
? computeStorePathForPath(path, false, false, "", filter).first
: store->addToStore(path, false, false, "", filter);
? computeStorePathForPath(path, true, htSHA256, filter).first
: store->addToStore(path, true, htSHA256, filter);
return makeStr(dstPath, singleton<PathSet>(dstPath));
return makeStr(dstPath, ATmakeList1(makeContextElem(toATerm(dstPath), makeNull())));
}
@@ -745,7 +748,7 @@ static Expr prim_attrNames(EvalState & state, const ATermVector & args)
ATermList list = ATempty;
for (StringSet::const_reverse_iterator i = names.rbegin();
i != names.rend(); ++i)
list = ATinsert(list, makeStr(*i, PathSet()));
list = ATinsert(list, makeStr(*i));
return makeList(list);
}
@@ -816,7 +819,7 @@ static Expr prim_removeAttrs(EvalState & state, const ATermVector & args)
}
/* Determine whether the argument is a list. */
/* Determine whether the argument is an attribute set. */
static Expr prim_isAttrs(EvalState & state, const ATermVector & args)
{
ATermList list;
@@ -824,6 +827,70 @@ static Expr prim_isAttrs(EvalState & state, const ATermVector & args)
}
/* Return the right-biased intersection of two attribute sets as1 and
as2, i.e. a set that contains every attribute from as2 that is also
a member of as1. */
static Expr prim_intersectAttrs(EvalState & state, const ATermVector & args)
{
ATermMap as1, as2;
queryAllAttrs(evalExpr(state, args[0]), as1, true);
queryAllAttrs(evalExpr(state, args[1]), as2, true);
ATermMap res;
foreach (ATermMap::const_iterator, i, as2)
if (as1[i->key]) res.set(i->key, i->value);
return makeAttrs(res);
}
static void attrsInPattern(ATermMap & map, Pattern pat)
{
ATerm name;
ATermList formals;
Pattern pat1, pat2;
ATermBool ellipsis;
if (matchAttrsPat(pat, formals, ellipsis)) {
for (ATermIterator i(formals); i; ++i) {
ATerm def;
if (!matchFormal(*i, name, def)) abort();
map.set(name, makeAttrRHS(makeBool(def != constNoDefaultValue), makeNoPos()));
}
}
else if (matchAtPat(pat, pat1, pat2)) {
attrsInPattern(map, pat1);
attrsInPattern(map, pat2);
}
}
/* Return a set containing the names of the formal arguments expected
by the function `f'. The value of each attribute is a Boolean
denoting whether has a default value. For instance,
functionArgs ({ x, y ? 123}: ...)
=> { x = false; y = true; }
"Formal argument" here refers to the attributes pattern-matched by
the function. Plain lambdas are not included, e.g.
functionArgs (x: ...)
=> { }
*/
static Expr prim_functionArgs(EvalState & state, const ATermVector & args)
{
Expr f = evalExpr(state, args[0]);
ATerm pat, body, pos;
if (!matchFunction(f, pat, body, pos))
throw TypeError("`functionArgs' required a function");
ATermMap as;
attrsInPattern(as, pat);
return makeAttrs(as);
}
/*************************************************************
* Lists
*************************************************************/
@@ -872,6 +939,14 @@ static Expr prim_map(EvalState & state, const ATermVector & args)
}
/* Return the length of a list. This is an O(1) time operation. */
static Expr prim_length(EvalState & state, const ATermVector & args)
{
ATermList list = evalList(state, args[0]);
return makeInt(ATgetLength(list));
}
/*************************************************************
* Integer arithmetic
*************************************************************/
@@ -893,6 +968,23 @@ static Expr prim_sub(EvalState & state, const ATermVector & args)
}
static Expr prim_mul(EvalState & state, const ATermVector & args)
{
int i1 = evalInt(state, args[0]);
int i2 = evalInt(state, args[1]);
return makeInt(i1 * i2);
}
static Expr prim_div(EvalState & state, const ATermVector & args)
{
int i1 = evalInt(state, args[0]);
int i2 = evalInt(state, args[1]);
if (i2 == 0) throw EvalError("division by zero");
return makeInt(i1 / i2);
}
static Expr prim_lessThan(EvalState & state, const ATermVector & args)
{
int i1 = evalInt(state, args[0]);
@@ -911,7 +1003,7 @@ static Expr prim_lessThan(EvalState & state, const ATermVector & args)
`"/nix/store/whatever..."'. */
static Expr prim_toString(EvalState & state, const ATermVector & args)
{
PathSet context;
Context context;
string s = coerceToString(state, args[0], context, true, false);
return makeStr(s, context);
}
@@ -925,7 +1017,7 @@ static Expr prim_substring(EvalState & state, const ATermVector & args)
{
int start = evalInt(state, args[0]);
int len = evalInt(state, args[1]);
PathSet context;
Context context;
string s = coerceToString(state, args[2], context);
if (start < 0) throw EvalError("negative start position in `substring'");
@@ -936,12 +1028,88 @@ static Expr prim_substring(EvalState & state, const ATermVector & args)
static Expr prim_stringLength(EvalState & state, const ATermVector & args)
{
PathSet context;
Context context;
string s = coerceToString(state, args[0], context);
return makeInt(s.size());
}
static Expr prim_unsafeDiscardStringContext(EvalState & state, const ATermVector & args)
{
Context context;
string s = coerceToString(state, args[0], context);
return makeStr(s);
}
/* Sometimes we want to pass a derivation path (i.e. pkg.drvPath) to a
builder without causing the derivation to be built (for instance,
in the derivation that builds NARs in nix-push, when doing
source-only deployment). This primop marks the string context so
that builtins.derivation adds the path to drv.inputSrcs rather than
drv.inputDrvs. */
static Expr prim_unsafeDiscardOutputDependency(EvalState & state, const ATermVector & args)
{
Context context;
string s = coerceToString(state, args[0], context);
Context context2;
foreach (Context::const_iterator, i, context) {
Path p = aterm2String(i->key);
if (p.at(0) == '=') p = "~" + string(p, 1);
context2.set(toATerm(p), i->value);
}
return makeStr(s, context2);
}
static Expr prim_queryStringContext(EvalState & state, const ATermVector & args)
{
Context context;
string s = coerceToString(state, args[0], context);
ATermList l = ATempty;
foreach (Context::const_iterator, i, context)
l = ATinsert(l, i->value);
return makeList(l);
}
/* Expression serialization/deserialization */
static Expr prim_exprToString(EvalState & state, const ATermVector & args)
{
/* !!! this disregards context */
return makeStr(atPrint(evalExpr(state, args[0])));
}
/*************************************************************
* Versions
*************************************************************/
static Expr prim_parseDrvName(EvalState & state, const ATermVector & args)
{
string name = evalStringNoCtx(state, args[0]);
DrvName parsed(name);
ATermMap attrs(2);
attrs.set(toATerm("name"), makeAttrRHS(makeStr(parsed.name), makeNoPos()));
attrs.set(toATerm("version"), makeAttrRHS(makeStr(parsed.version), makeNoPos()));
return makeAttrs(attrs);
}
static Expr prim_compareVersions(EvalState & state, const ATermVector & args)
{
string version1 = evalStringNoCtx(state, args[0]);
string version2 = evalStringNoCtx(state, args[1]);
int d = compareVersions(version1, version2);
return makeInt(d);
}
/*************************************************************
* Primop registration
*************************************************************/
@@ -962,13 +1130,19 @@ void EvalState::addPrimOps()
addPrimOp("import", 1, prim_import);
addPrimOp("isNull", 1, prim_isNull);
addPrimOp("__isFunction", 1, prim_isFunction);
addPrimOp("dependencyClosure", 1, prim_dependencyClosure);
addPrimOp("__isString", 1, prim_isString);
addPrimOp("__isInt", 1, prim_isInt);
addPrimOp("__isBool", 1, prim_isBool);
addPrimOp("__genericClosure", 1, prim_genericClosure);
addPrimOp("abort", 1, prim_abort);
addPrimOp("throw", 1, prim_throw);
addPrimOp("__addErrorContext", 2, prim_addErrorContext);
addPrimOp("__tryEval", 1, prim_tryEval);
addPrimOp("__getEnv", 1, prim_getEnv);
addPrimOp("__trace", 2, prim_trace);
addPrimOp("relativise", 2, prim_relativise);
// Expr <-> String
addPrimOp("__exprToString", 1, prim_exprToString);
// Derivations
addPrimOp("derivation!", 1, prim_derivationStrict);
@@ -976,6 +1150,7 @@ void EvalState::addPrimOps()
// Paths
addPrimOp("__toPath", 1, prim_toPath);
addPrimOp("__storePath", 1, prim_storePath);
addPrimOp("__pathExists", 1, prim_pathExists);
addPrimOp("baseNameOf", 1, prim_baseNameOf);
addPrimOp("dirOf", 1, prim_dirOf);
@@ -993,22 +1168,34 @@ void EvalState::addPrimOps()
addPrimOp("__isAttrs", 1, prim_isAttrs);
addPrimOp("removeAttrs", 2, prim_removeAttrs);
addPrimOp("__listToAttrs", 1, prim_listToAttrs);
addPrimOp("__intersectAttrs", 2, prim_intersectAttrs);
addPrimOp("__functionArgs", 1, prim_functionArgs);
// Lists
addPrimOp("__isList", 1, prim_isList);
addPrimOp("__head", 1, prim_head);
addPrimOp("__tail", 1, prim_tail);
addPrimOp("map", 2, prim_map);
addPrimOp("__length", 1, prim_length);
// Integer arithmetic
addPrimOp("__add", 2, prim_add);
addPrimOp("__sub", 2, prim_sub);
addPrimOp("__mul", 2, prim_mul);
addPrimOp("__div", 2, prim_div);
addPrimOp("__lessThan", 2, prim_lessThan);
// String manipulation
addPrimOp("toString", 1, prim_toString);
addPrimOp("__substring", 3, prim_substring);
addPrimOp("__stringLength", 1, prim_stringLength);
addPrimOp("__unsafeDiscardStringContext", 1, prim_unsafeDiscardStringContext);
addPrimOp("__unsafeDiscardOutputDependency", 1, prim_unsafeDiscardOutputDependency);
addPrimOp("__queryStringContext", 1, prim_queryStringContext);
// Versions
addPrimOp("__parseDrvName", 1, prim_parseDrvName);
addPrimOp("__compareVersions", 2, prim_compareVersions);
}
+5 -1
View File
@@ -1,6 +1,10 @@
pkglib_LTLIBRARIES = libmain.la
libmain_la_SOURCES = shared.cc shared.hh
libmain_la_SOURCES = shared.cc
libmain_la_LIBADD = ../libstore/libstore.la
pkginclude_HEADERS = shared.hh
AM_CXXFLAGS = \
-DNIX_STORE_DIR=\"$(storedir)\" \
+71 -34
View File
@@ -4,6 +4,7 @@
#include "globals.hh"
#include "store-api.hh"
#include "util.hh"
#include "misc.hh"
#include <iostream>
#include <cctype>
@@ -49,6 +50,34 @@ void printGCWarning()
}
void printMissing(const PathSet & paths)
{
unsigned long long downloadSize;
PathSet willBuild, willSubstitute, unknown;
queryMissing(paths, willBuild, willSubstitute, unknown, downloadSize);
if (!willBuild.empty()) {
printMsg(lvlInfo, format("the following derivations will be built:"));
foreach (PathSet::iterator, i, willBuild)
printMsg(lvlInfo, format(" %1%") % *i);
}
if (!willSubstitute.empty()) {
printMsg(lvlInfo, format("the following paths will be downloaded/copied (%.2f MiB):") %
(downloadSize / (1024.0 * 1024.0)));
foreach (PathSet::iterator, i, willSubstitute)
printMsg(lvlInfo, format(" %1%") % *i);
}
if (!unknown.empty()) {
printMsg(lvlInfo, format("don't know how to build the following paths%1%:")
% (readOnlyMode ? " (may be caused by read-only store access)" : ""));
foreach (PathSet::iterator, i, unknown)
printMsg(lvlInfo, format(" %1%") % *i);
}
}
static void setLogType(string lt)
{
if (lt == "pretty") logType = ltPretty;
@@ -58,27 +87,6 @@ static void setLogType(string lt)
}
static unsigned int getIntArg(const string & opt,
Strings::iterator & i, const Strings::iterator & end)
{
++i;
if (i == end) throw UsageError(format("`%1%' requires an argument") % opt);
int n;
if (!string2Int(*i, n) || n < 0)
throw UsageError(format("`%1%' requires a non-negative integer") % opt);
return n;
}
struct RemoveTempRoots
{
~RemoveTempRoots()
{
removeTempRoots();
}
};
void initDerivationsHelpers();
@@ -99,6 +107,15 @@ static void closeStore()
}
RemoveTempRoots::~RemoveTempRoots()
{
removeTempRoots();
}
static bool showTrace = false;
/* Initialize and reorder arguments, then call the actual argument
processor. */
static void initAndRun(int argc, char * * argv)
@@ -114,9 +131,10 @@ static void initAndRun(int argc, char * * argv)
nixBinDir = canonPath(getEnv("NIX_BIN_DIR", NIX_BIN_DIR));
string subs = getEnv("NIX_SUBSTITUTERS", "default");
if (subs == "default")
substituters.push_back(nixLibexecDir + "/nix/download-using-manifests.pl");
else
if (subs == "default") {
substituters.push_back(nixLibexecDir + "/nix/substituters/copy-from-other-stores.pl");
substituters.push_back(nixLibexecDir + "/nix/substituters/download-using-manifests.pl");
} else
substituters = tokenizeString(subs, ":");
/* Get some settings from the configuration file. */
@@ -125,23 +143,29 @@ static void initAndRun(int argc, char * * argv)
maxSilentTime = queryIntSetting("build-max-silent-time", 0);
/* Catch SIGINT. */
struct sigaction act, oact;
struct sigaction act;
act.sa_handler = sigintHandler;
sigfillset(&act.sa_mask);
act.sa_flags = 0;
if (sigaction(SIGINT, &act, &oact))
if (sigaction(SIGINT, &act, 0))
throw SysError("installing handler for SIGINT");
if (sigaction(SIGTERM, &act, &oact))
if (sigaction(SIGTERM, &act, 0))
throw SysError("installing handler for SIGTERM");
if (sigaction(SIGHUP, &act, &oact))
if (sigaction(SIGHUP, &act, 0))
throw SysError("installing handler for SIGHUP");
/* Ignore SIGPIPE. */
act.sa_handler = SIG_IGN;
act.sa_flags = 0;
if (sigaction(SIGPIPE, &act, &oact))
if (sigaction(SIGPIPE, &act, 0))
throw SysError("ignoring SIGPIPE");
/* Reset SIGCHLD to its default. */
act.sa_handler = SIG_DFL;
act.sa_flags = 0;
if (sigaction(SIGCHLD, &act, 0))
throw SysError("resetting SIGCHLD");
/* There is no privacy in the Nix system ;-) At least not for
now. In particular, store objects should be readable by
everybody. This prevents nasty surprises when using a shared
@@ -165,7 +189,7 @@ static void initAndRun(int argc, char * * argv)
for (Strings::iterator i = args.begin(); i != args.end(); ++i) {
string arg = *i;
if (string(arg, 0, 4) == "-at-") ;
else if (arg.length() > 2 && arg[0] == '-' && arg[1] != '-') {
else if (arg.length() > 2 && arg[0] == '-' && arg[1] != '-' && !isdigit(arg[1])) {
for (unsigned int j = 1; j < arg.length(); j++)
if (isalpha(arg[j]))
remaining.push_back((string) "-" + arg[j]);
@@ -192,6 +216,8 @@ static void initAndRun(int argc, char * * argv)
; /* !!! obsolete - remove eventually */
else if (arg == "--no-build-output" || arg == "-Q")
buildVerbosity = lvlVomit;
else if (arg == "--print-build-trace")
printBuildTrace = true;
else if (arg == "--help") {
printHelp();
return;
@@ -207,19 +233,28 @@ static void initAndRun(int argc, char * * argv)
else if (arg == "--fallback")
tryFallback = true;
else if (arg == "--max-jobs" || arg == "-j")
maxBuildJobs = getIntArg(arg, i, args.end());
maxBuildJobs = getIntArg<unsigned int>(arg, i, args.end());
else if (arg == "--readonly-mode")
readOnlyMode = true;
else if (arg == "--max-silent-time")
maxSilentTime = getIntArg(arg, i, args.end());
maxSilentTime = getIntArg<unsigned int>(arg, i, args.end());
else if (arg == "--no-build-hook")
useBuildHook = false;
else if (arg == "--show-trace")
showTrace = true;
else if (arg == "--option") {
++i; if (i == args.end()) throw UsageError("`--option' requires two arguments");
string name = *i;
++i; if (i == args.end()) throw UsageError("`--option' requires two arguments");
string value = *i;
overrideSetting(name, tokenizeString(value));
}
else remaining.push_back(arg);
}
/* Automatically clean up the temporary roots file when we
exit. */
RemoveTempRoots removeTempRoots; /* unused variable - don't remove */
RemoveTempRoots removeTempRoots __attribute__((unused));
/* Make sure that the database gets closed properly, even if
terminate() is called (which happens sometimes due to bugs in
@@ -329,7 +364,9 @@ int main(int argc, char * * argv)
% e.what() % programId);
return 1;
} catch (BaseError & e) {
printMsg(lvlError, format("error: %1%") % e.msg());
printMsg(lvlError, format("error: %1%%2%") % (showTrace ? e.prefix() : "") % e.msg());
if (e.prefix() != "" && !showTrace)
printMsg(lvlError, "(use `--show-trace' to show detailed location information)");
return 1;
} catch (std::exception & e) {
printMsg(lvlError, format("error: %1%") % e.what());
+19 -1
View File
@@ -22,16 +22,34 @@ extern std::string programId;
namespace nix {
MakeError(UsageError, nix::Error);
/* Ugh. No better place to put this. */
Path makeRootName(const Path & gcRoot, int & counter);
void printGCWarning();
void printMissing(const PathSet & paths);
template<class N> N getIntArg(const string & opt,
Strings::iterator & i, const Strings::iterator & end)
{
++i;
if (i == end) throw UsageError(format("`%1%' requires an argument") % opt);
N n;
if (!string2Int(*i, n))
throw UsageError(format("`%1%' requires an integer argument") % opt);
return n;
}
/* Whether we're running setuid. */
extern bool setuidMode;
extern volatile ::sig_atomic_t blockInt;
MakeError(UsageError, nix::Error);
struct RemoveTempRoots
{
~RemoveTempRoots();
};
}
+8 -7
View File
@@ -1,22 +1,23 @@
pkglib_LTLIBRARIES = libstore.la
libstore_la_SOURCES = \
store-api.cc local-store.cc remote-store.cc derivations.cc build.cc misc.cc \
globals.cc db.cc references.cc pathlocks.cc gc.cc
store-api.cc local-store.cc remote-store.cc derivations.cc build.cc misc.cc \
globals.cc references.cc pathlocks.cc gc.cc \
optimise-store.cc
pkginclude_HEADERS = \
store-api.hh local-store.hh remote-store.hh derivations.hh misc.hh \
globals.hh db.hh references.hh pathlocks.hh \
worker-protocol.hh
store-api.hh local-store.hh remote-store.hh derivations.hh misc.hh \
globals.hh references.hh pathlocks.hh \
worker-protocol.hh
libstore_la_LIBADD = ../libutil/libutil.la ../boost/format/libformat.la
libstore_la_LIBADD = ../libutil/libutil.la ../boost/format/libformat.la @ADDITIONAL_NETWORK_LIBS@
BUILT_SOURCES = derivations-ast.cc derivations-ast.hh
EXTRA_DIST = derivations-ast.def derivations-ast.cc
AM_CXXFLAGS = -Wall \
-I$(srcdir)/.. ${bdb_include} ${aterm_include} -I$(srcdir)/../libutil
-I$(srcdir)/.. ${aterm_include} -I$(srcdir)/../libutil
derivations-ast.cc derivations-ast.hh: ../aterm-helper.pl derivations-ast.def
$(perl) $(srcdir)/../aterm-helper.pl derivations-ast.hh derivations-ast.cc < $(srcdir)/derivations-ast.def
+644 -786
View File
File diff suppressed because it is too large. Load diff
-468
View File
@@ -1,468 +0,0 @@
#include "db.hh"
#include "util.hh"
#include "pathlocks.hh"
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <memory>
#include <db_cxx.h>
namespace nix {
/* Wrapper class to ensure proper destruction. */
class DestroyDbc
{
Dbc * dbc;
public:
DestroyDbc(Dbc * _dbc) : dbc(_dbc) { }
~DestroyDbc() { dbc->close(); /* close() frees dbc */ }
};
class DestroyDbEnv
{
DbEnv * dbenv;
public:
DestroyDbEnv(DbEnv * _dbenv) : dbenv(_dbenv) { }
~DestroyDbEnv() {
if (dbenv) {
if (dbenv->get_DB_ENV()) dbenv->close(0);
delete dbenv;
}
}
void release() { dbenv = 0; };
};
static void rethrow(DbException & e)
{
throw Error(e.what());
}
Transaction::Transaction()
: txn(0)
{
}
Transaction::Transaction(Database & db)
: txn(0)
{
begin(db);
}
Transaction::~Transaction()
{
if (txn) abort();
}
void Transaction::begin(Database & db)
{
assert(txn == 0);
db.requireEnv();
try {
db.env->txn_begin(0, &txn, 0);
} catch (DbException e) { rethrow(e); }
}
void Transaction::commit()
{
if (!txn) throw Error("commit called on null transaction");
debug(format("committing transaction %1%") % (void *) txn);
DbTxn * txn2 = txn;
txn = 0;
try {
txn2->commit(0);
} catch (DbException e) { rethrow(e); }
}
void Transaction::abort()
{
if (!txn) throw Error("abort called on null transaction");
debug(format("aborting transaction %1%") % (void *) txn);
DbTxn * txn2 = txn;
txn = 0;
try {
txn2->abort();
} catch (DbException e) { rethrow(e); }
}
void Transaction::moveTo(Transaction & t)
{
if (t.txn) throw Error("target txn already exists");
t.txn = txn;
txn = 0;
}
void Database::requireEnv()
{
checkInterrupt();
if (!env) throw Error("database environment is not open "
"(maybe you don't have sufficient permission?)");
}
Db * Database::getDb(TableId table)
{
if (table == 0)
throw Error("database table is not open "
"(maybe you don't have sufficient permission?)");
std::map<TableId, Db *>::iterator i = tables.find(table);
if (i == tables.end())
throw Error("unknown table id");
return i->second;
}
Database::Database()
: env(0)
, nextId(1)
{
}
Database::~Database()
{
close();
}
void openEnv(DbEnv * & env, const string & path, u_int32_t flags)
{
try {
createDirs(path);
} catch (SysError & e) {
if (e.errNo == EPERM || e.errNo == EACCES)
throw DbNoPermission(format("cannot create the Nix database in `%1%'") % path);
else
throw;
}
try {
env->open(path.c_str(),
DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN |
DB_CREATE | flags,
0666);
} catch (DbException & e) {
printMsg(lvlError, format("environment open failed: %1%") % e.what());
throw;
}
}
static int my_fsync(int fd)
{
return 0;
}
static void errorPrinter(const DbEnv * env, const char * errpfx, const char * msg)
{
printMsg(lvlError, format("Berkeley DB error: %1%") % msg);
}
static void messagePrinter(const DbEnv * env, const char * msg)
{
printMsg(lvlError, format("Berkeley DB message: %1%") % msg);
}
void Database::open2(const string & path, bool removeOldEnv)
{
if (env) throw Error(format("environment already open"));
debug(format("opening database environment"));
/* Create the database environment object. */
DbEnv * env = new DbEnv(0);
DestroyDbEnv deleteEnv(env);
env->set_errcall(errorPrinter);
env->set_msgcall(messagePrinter);
if (getEnv("NIX_DEBUG_DB_REGISTER") == "1")
env->set_verbose(DB_VERB_REGISTER, 1);
env->set_verbose(DB_VERB_RECOVERY, 1);
/* Smaller log files. */
env->set_lg_bsize(32 * 1024); /* default */
env->set_lg_max(256 * 1024); /* must be > 4 * lg_bsize */
/* Write the log, but don't sync. This protects transactions
against application crashes, but if the system crashes, some
transactions may be undone. An acceptable risk, I think. */
env->set_flags(DB_TXN_WRITE_NOSYNC | DB_LOG_AUTOREMOVE, 1);
/* Increase the locking limits. If you ever get `Dbc::get: Cannot
allocate memory' or similar, especially while running
`nix-store --verify', just increase the following number, then
run db_recover on the database to remove the existing DB
environment (since changes only take effect on new
environments). */
env->set_lk_max_locks(10000);
env->set_lk_max_lockers(10000);
env->set_lk_max_objects(10000);
env->set_lk_detect(DB_LOCK_DEFAULT);
/* Dangerous, probably, but from the docs it *seems* that BDB
shouldn't sync when DB_TXN_WRITE_NOSYNC is used, but it still
fsync()s sometimes. */
db_env_set_func_fsync(my_fsync);
if (removeOldEnv) {
printMsg(lvlError, "removing old Berkeley DB database environment...");
env->remove(path.c_str(), DB_FORCE);
return;
}
openEnv(env, path, DB_REGISTER | DB_RECOVER);
deleteEnv.release();
this->env = env;
}
void Database::open(const string & path)
{
try {
open2(path, false);
} catch (DbException e) {
if (e.get_errno() == DB_VERSION_MISMATCH) {
/* Remove the environment while we are holding the global
lock. If things go wrong there, we bail out.
!!! argh, we abolished the global lock :-( */
open2(path, true);
/* Try again. */
open2(path, false);
/* Force a checkpoint, as per the BDB docs. */
env->txn_checkpoint(DB_FORCE, 0, 0);
printMsg(lvlError, "database succesfully upgraded to new version");
}
#if 0
else if (e.get_errno() == DB_RUNRECOVERY) {
/* If recovery is needed, do it. */
printMsg(lvlError, "running recovery...");
open2(path, false, true);
}
#endif
else
rethrow(e);
}
}
void Database::close()
{
if (!env) return;
/* Close the database environment. */
debug(format("closing database environment"));
try {
for (std::map<TableId, Db *>::iterator i = tables.begin();
i != tables.end(); )
{
std::map<TableId, Db *>::iterator j = i;
++j;
closeTable(i->first);
i = j;
}
/* Do a checkpoint every 128 kilobytes, or every 5 minutes. */
env->txn_checkpoint(128, 5, 0);
env->close(0);
} catch (DbException e) { rethrow(e); }
delete env;
env = 0;
}
TableId Database::openTable(const string & tableName, bool sorted)
{
requireEnv();
TableId table = nextId++;
try {
Db * db = new Db(env, 0);
try {
db->open(0, tableName.c_str(), 0,
sorted ? DB_BTREE : DB_HASH,
DB_CREATE | DB_AUTO_COMMIT, 0666);
} catch (...) {
delete db;
throw;
}
tables[table] = db;
} catch (DbException e) { rethrow(e); }
return table;
}
void Database::closeTable(TableId table)
{
try {
Db * db = getDb(table);
db->close(DB_NOSYNC);
delete db;
tables.erase(table);
} catch (DbException e) { rethrow(e); }
}
void Database::deleteTable(const string & table)
{
try {
env->dbremove(0, table.c_str(), 0, DB_AUTO_COMMIT);
} catch (DbException e) { rethrow(e); }
}
bool Database::queryString(const Transaction & txn, TableId table,
const string & key, string & data)
{
checkInterrupt();
try {
Db * db = getDb(table);
Dbt kt((void *) key.c_str(), key.length());
Dbt dt;
int err = db->get(txn.txn, &kt, &dt, 0);
if (err) return false;
if (!dt.get_data())
data = "";
else
data = string((char *) dt.get_data(), dt.get_size());
} catch (DbException e) { rethrow(e); }
return true;
}
bool Database::queryStrings(const Transaction & txn, TableId table,
const string & key, Strings & data)
{
string d;
if (!queryString(txn, table, key, d))
return false;
data = unpackStrings(d);
return true;
}
void Database::setString(const Transaction & txn, TableId table,
const string & key, const string & data)
{
checkInterrupt();
try {
Db * db = getDb(table);
Dbt kt((void *) key.c_str(), key.length());
Dbt dt((void *) data.c_str(), data.length());
db->put(txn.txn, &kt, &dt, 0);
} catch (DbException e) { rethrow(e); }
}
void Database::setStrings(const Transaction & txn, TableId table,
const string & key, const Strings & data, bool deleteEmpty)
{
if (deleteEmpty && data.size() == 0)
delPair(txn, table, key);
else
setString(txn, table, key, packStrings(data));
}
void Database::delPair(const Transaction & txn, TableId table,
const string & key)
{
checkInterrupt();
try {
Db * db = getDb(table);
Dbt kt((void *) key.c_str(), key.length());
db->del(txn.txn, &kt, 0);
/* Non-existence of a pair with the given key is not an
error. */
} catch (DbException e) { rethrow(e); }
}
void Database::enumTable(const Transaction & txn, TableId table,
Strings & keys, const string & keyPrefix)
{
try {
Db * db = getDb(table);
Dbc * dbc;
db->cursor(txn.txn, &dbc, 0);
DestroyDbc destroyDbc(dbc);
Dbt kt, dt;
u_int32_t flags = DB_NEXT;
if (!keyPrefix.empty()) {
flags = DB_SET_RANGE;
kt = Dbt((void *) keyPrefix.c_str(), keyPrefix.size());
}
while (dbc->get(&kt, &dt, flags) != DB_NOTFOUND) {
checkInterrupt();
string data((char *) kt.get_data(), kt.get_size());
if (!keyPrefix.empty() &&
string(data, 0, keyPrefix.size()) != keyPrefix)
break;
keys.push_back(data);
flags = DB_NEXT;
}
} catch (DbException e) { rethrow(e); }
}
void Database::clearTable(const Transaction & txn, TableId table)
{
try {
Db * db = getDb(table);
u_int32_t count;
db->truncate(txn.txn, &count, 0);
} catch (DbException e) { rethrow(e); }
}
}
-107
View File
@@ -1,107 +0,0 @@
#ifndef __DB_H
#define __DB_H
#include "types.hh"
#include <map>
/* Defined externally. */
class DbTxn;
class DbEnv;
class Db;
namespace nix {
class Database;
class Transaction
{
friend class Database;
private:
DbTxn * txn;
public:
Transaction();
Transaction(Database & _db);
~Transaction();
void begin(Database & db);
void abort();
void commit();
void moveTo(Transaction & t);
};
#define noTxn Transaction()
typedef unsigned int TableId; /* table handles */
class Database
{
friend class Transaction;
private:
DbEnv * env;
TableId nextId;
std::map<TableId, Db *> tables;
void requireEnv();
Db * getDb(TableId table);
void open2(const string & path, bool removeOldEnv);
public:
Database();
~Database();
void open(const string & path);
void close();
TableId openTable(const string & table, bool sorted = false);
void closeTable(TableId table);
void deleteTable(const string & table);
bool queryString(const Transaction & txn, TableId table,
const string & key, string & data);
bool queryStrings(const Transaction & txn, TableId table,
const string & key, Strings & data);
void setString(const Transaction & txn, TableId table,
const string & key, const string & data);
void setStrings(const Transaction & txn, TableId table,
const string & key, const Strings & data,
bool deleteEmpty = true);
void delPair(const Transaction & txn, TableId table,
const string & key);
void enumTable(const Transaction & txn, TableId table,
Strings & keys, const string & keyPrefix = "");
void clearTable(const Transaction & txn, TableId table);
};
class DbNoPermission : public Error
{
public:
DbNoPermission(const format & f) : Error(f) { };
};
}
#endif /* !__DB_H */
+3 -5
View File
@@ -2,6 +2,7 @@
#include "store-api.hh"
#include "aterm.hh"
#include "globals.hh"
#include "util.hh"
#include "derivations-ast.hh"
#include "derivations-ast.cc"
@@ -20,8 +21,7 @@ Path writeDerivation(const Derivation & drv, const string & name)
{
PathSet references;
references.insert(drv.inputSrcs.begin(), drv.inputSrcs.end());
for (DerivationInputs::const_iterator i = drv.inputDrvs.begin();
i != drv.inputDrvs.end(); ++i)
foreach (DerivationInputs::const_iterator, i, drv.inputDrvs)
references.insert(i->first);
/* Note that the outputs of a derivation are *not* references
(that can be missing (of course) and should not necessarily be
@@ -163,9 +163,7 @@ ATerm unparseDerivation(const Derivation & drv)
bool isDerivation(const string & fileName)
{
return
fileName.size() >= drvExtension.size() &&
string(fileName, fileName.size() - drvExtension.size()) == drvExtension;
return hasSuffix(fileName, drvExtension);
}
+1 -1
View File
@@ -1,7 +1,7 @@
#ifndef __DERIVATIONS_H
#define __DERIVATIONS_H
typedef struct _ATerm * ATerm;
#include <aterm1.h>
#include "hash.hh"
+263 -182
View File
@@ -2,11 +2,13 @@
#include "misc.hh"
#include "pathlocks.hh"
#include "local-store.hh"
#include "db.hh"
#include "util.hh"
#include <boost/shared_ptr.hpp>
#include <functional>
#include <queue>
#include <algorithm>
#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>
@@ -125,7 +127,11 @@ Path addPermRoot(const Path & _storePath, const Path & _gcRoot,
createSymlink(gcRoot, storePath, false);
}
/* Check that the root can be found by the garbage collector. */
/* Check that the root can be found by the garbage collector.
!!! This can be very slow on machines that have many roots.
Instead of reading all the roots, it would be more efficient to
check if the root is in a directory in or linked from the
gcroots directory. */
if (queryBoolSetting("gc-check-reachability", true)) {
Roots roots = store->findRoots();
if (roots.find(gcRoot) == roots.end())
@@ -242,9 +248,7 @@ static void readTempRoots(PathSet & tempRoots, FDs & fds)
Strings tempRootFiles = readDirectory(
(format("%1%/%2%") % nixStateDir % tempRootsDir).str());
for (Strings::iterator i = tempRootFiles.begin();
i != tempRootFiles.end(); ++i)
{
foreach (Strings::iterator, i, tempRootFiles) {
Path path = (format("%1%/%2%/%3%") % nixStateDir % tempRootsDir % *i).str();
debug(format("reading temporary root file `%1%'") % path);
@@ -323,7 +327,7 @@ static void findRoots(const Path & path, bool recurseSymlinks,
if (S_ISDIR(st.st_mode)) {
Strings names = readDirectory(path);
for (Strings::iterator i = names.begin(); i != names.end(); ++i)
foreach (Strings::iterator, i, names)
findRoots(path + "/" + *i, recurseSymlinks, deleteStale, roots);
}
@@ -394,11 +398,11 @@ static void addAdditionalRoots(PathSet & roots)
Strings paths = tokenizeString(result, "\n");
for (Strings::iterator i = paths.begin(); i != paths.end(); ++i) {
foreach (Strings::iterator, i, paths) {
if (isInStore(*i)) {
Path path = toStorePath(*i);
if (roots.find(path) == roots.end() && store->isValidPath(path)) {
debug(format("found additional root `%1%'") % path);
debug(format("got additional root `%1%'") % path);
roots.insert(path);
}
}
@@ -416,8 +420,7 @@ static void dfsVisit(const PathSet & paths, const Path & path,
if (store->isValidPath(path))
store->queryReferences(path, references);
for (PathSet::iterator i = references.begin();
i != references.end(); ++i)
foreach (PathSet::iterator, i, references)
/* Don't traverse into paths that don't exist. That can
happen due to substitutes for non-existent paths. */
if (*i != path && paths.find(*i) != paths.end())
@@ -431,25 +434,224 @@ Paths topoSortPaths(const PathSet & paths)
{
Paths sorted;
PathSet visited;
for (PathSet::const_iterator i = paths.begin(); i != paths.end(); ++i)
foreach (PathSet::const_iterator, i, paths)
dfsVisit(paths, *i, visited, sorted);
return sorted;
}
void LocalStore::collectGarbage(GCAction action, const PathSet & pathsToDelete,
bool ignoreLiveness, PathSet & result, unsigned long long & bytesFreed)
struct GCLimitReached { };
struct LocalStore::GCState
{
result.clear();
bytesFreed = 0;
GCOptions options;
GCResults & results;
PathSet roots;
PathSet tempRoots;
PathSet deleted;
PathSet live;
PathSet busy;
bool gcKeepOutputs;
bool gcKeepDerivations;
bool gcKeepOutputs =
queryBoolSetting("gc-keep-outputs", false);
bool gcKeepDerivations =
queryBoolSetting("gc-keep-derivations", true);
int gcKeepOutputsThreshold =
queryIntSetting ("gc-keep-outputs-threshold", defaultGcLevel);
bool drvsIndexed;
typedef std::multimap<string, Path> DrvsByName;
DrvsByName drvsByName; // derivation paths hashed by name attribute
GCState(GCResults & results_) : results(results_), drvsIndexed(false)
{
}
};
static bool doDelete(GCOptions::GCAction action)
{
return action == GCOptions::gcDeleteDead
|| action == GCOptions::gcDeleteSpecific;
}
bool LocalStore::isActiveTempFile(const GCState & state,
const Path & path, const string & suffix)
{
return hasSuffix(path, suffix)
&& state.tempRoots.find(string(path, 0, path.size() - suffix.size())) != state.tempRoots.end();
}
/* Return all the derivations in the Nix store that have `path' as an
output. This function assumes that derivations have the same name
as their outputs. */
PathSet LocalStore::findDerivers(GCState & state, const Path & path)
{
PathSet derivers;
Path deriver = queryDeriver(path);
if (deriver != "") derivers.insert(deriver);
if (!state.drvsIndexed) {
Paths entries = readDirectory(nixStore);
foreach (Paths::iterator, i, entries)
if (isDerivation(*i))
state.drvsByName.insert(std::pair<string, Path>(
getNameOfStorePath(*i), nixStore + "/" + *i));
state.drvsIndexed = true;
}
string name = getNameOfStorePath(path);
// Urgh, I should have used Haskell...
std::pair<GCState::DrvsByName::iterator, GCState::DrvsByName::iterator> range =
state.drvsByName.equal_range(name);
for (GCState::DrvsByName::iterator i = range.first; i != range.second; ++i)
if (isValidPath(i->second)) {
Derivation drv = derivationFromPath(i->second);
foreach (DerivationOutputs::iterator, j, drv.outputs)
if (j->second.path == path) derivers.insert(i->second);
}
return derivers;
}
bool LocalStore::tryToDelete(GCState & state, const Path & path)
{
if (!pathExists(path)) return true;
if (state.deleted.find(path) != state.deleted.end()) return true;
if (state.live.find(path) != state.live.end()) return false;
startNest(nest, lvlDebug, format("considering whether to delete `%1%'") % path);
if (state.roots.find(path) != state.roots.end()) {
printMsg(lvlDebug, format("cannot delete `%1%' because it's a root") % path);
goto isLive;
}
if (isValidPath(path)) {
/* Recursively try to delete the referrers of this path. If
any referrer can't be deleted, then this path can't be
deleted either. */
PathSet referrers;
queryReferrers(path, referrers);
foreach (PathSet::iterator, i, referrers)
if (*i != path && !tryToDelete(state, *i)) {
printMsg(lvlDebug, format("cannot delete `%1%' because it has live referrers") % path);
goto isLive;
}
/* If gc-keep-derivations is set and this is a derivation,
then don't delete the derivation if any of the outputs are
live. */
if (state.gcKeepDerivations && isDerivation(path)) {
Derivation drv = derivationFromPath(path);
foreach (DerivationOutputs::iterator, i, drv.outputs)
if (!tryToDelete(state, i->second.path)) {
printMsg(lvlDebug, format("cannot delete derivation `%1%' because its output is alive") % path);
goto isLive;
}
}
/* If gc-keep-derivations and gc-keep-outputs are both set,
it's possible that the path has already been deleted (due
to the recursion below), so bail out. */
if (!pathExists(path)) return true;
/* If gc-keep-outputs is set, then don't delete this path if
its deriver is not garbage. !!! Nix does not reliably
store derivers, so we have to look at all derivations to
determine which of them derive `path'. Since this makes
the garbage collector very slow to start on large Nix
stores, here we just look for all derivations that have the
same name as `path' (where the name is the part of the
filename after the hash, i.e. the `name' attribute of the
derivation). This is somewhat hacky: currently, the
deriver of a path always has the same name as the output,
but this might change in the future. */
if (state.gcKeepOutputs) {
PathSet derivers = findDerivers(state, path);
foreach (PathSet::iterator, deriver, derivers) {
/* Break an infinite recursion if gc-keep-derivations
and gc-keep-outputs are both set by tentatively
assuming that this path is garbage. This is a safe
assumption because at this point, the only thing
that can prevent it from being garbage is the
deriver. Since tryToDelete() works "upwards"
through the dependency graph, it won't encouter
this path except in the call to tryToDelete() in
the gc-keep-derivation branch. */
state.deleted.insert(path);
if (!tryToDelete(state, *deriver)) {
state.deleted.erase(path);
printMsg(lvlDebug, format("cannot delete `%1%' because its deriver `%2%' is alive") % path % *deriver);
goto isLive;
}
}
}
}
else {
/* A lock file belonging to a path that we're building right
now isn't garbage. */
if (isActiveTempFile(state, path, ".lock")) return false;
/* Don't delete .chroot directories for derivations that are
currently being built. */
if (isActiveTempFile(state, path, ".chroot")) return false;
}
/* The path is garbage, so delete it. */
if (doDelete(state.options.action)) {
printMsg(lvlInfo, format("deleting `%1%'") % path);
unsigned long long bytesFreed, blocksFreed;
deleteFromStore(path, bytesFreed, blocksFreed);
state.results.bytesFreed += bytesFreed;
state.results.blocksFreed += blocksFreed;
if (state.options.maxFreed && state.results.bytesFreed > state.options.maxFreed) {
printMsg(lvlInfo, format("deleted more than %1% bytes; stopping") % state.options.maxFreed);
throw GCLimitReached();
}
if (state.options.maxLinks) {
struct stat st;
if (stat(nixStore.c_str(), &st) == -1)
throw SysError(format("statting `%1%'") % nixStore);
if (st.st_nlink < state.options.maxLinks) {
printMsg(lvlInfo, format("link count on the store has dropped below %1%; stopping") % state.options.maxLinks);
throw GCLimitReached();
}
}
} else
printMsg(lvlTalkative, format("would delete `%1%'") % path);
state.deleted.insert(path);
if (state.options.action != GCOptions::gcReturnLive)
state.results.paths.insert(path);
return true;
isLive:
state.live.insert(path);
if (state.options.action == GCOptions::gcReturnLive)
state.results.paths.insert(path);
return false;
}
void LocalStore::collectGarbage(const GCOptions & options, GCResults & results)
{
GCState state(results);
state.options = options;
state.gcKeepOutputs = queryBoolSetting("gc-keep-outputs", false);
state.gcKeepDerivations = queryBoolSetting("gc-keep-derivations", true);
/* Acquire the global GC root. This prevents
a) New roots from being added.
b) Processes from creating new temporary root files. */
@@ -457,185 +659,64 @@ void LocalStore::collectGarbage(GCAction action, const PathSet & pathsToDelete,
/* Find the roots. Since we've grabbed the GC lock, the set of
permanent roots cannot increase now. */
Roots rootMap = ignoreLiveness ? Roots() : nix::findRoots(true);
printMsg(lvlError, format("finding garbage collector roots..."));
Roots rootMap = options.ignoreLiveness ? Roots() : nix::findRoots(true);
PathSet roots;
for (Roots::iterator i = rootMap.begin(); i != rootMap.end(); ++i)
roots.insert(i->second);
foreach (Roots::iterator, i, rootMap) state.roots.insert(i->second);
/* Add additional roots returned by the program specified by the
NIX_ROOT_FINDER environment variable. This is typically used
to add running programs to the set of roots (to prevent them
from being garbage collected). */
if (!ignoreLiveness)
addAdditionalRoots(roots);
if (action == gcReturnRoots) {
result = roots;
return;
}
/* Determine the live paths which is just the closure of the
roots under the `references' relation. */
PathSet livePaths;
for (PathSet::const_iterator i = roots.begin(); i != roots.end(); ++i)
computeFSClosure(canonPath(*i), livePaths);
if (gcKeepDerivations) {
for (PathSet::iterator i = livePaths.begin();
i != livePaths.end(); ++i)
{
/* Note that the deriver need not be valid (e.g., if we
previously ran the collector with `gcKeepDerivations'
turned off). */
Path deriver = store->queryDeriver(*i);
if (deriver != "" && store->isValidPath(deriver))
computeFSClosure(deriver, livePaths);
}
}
if (gcKeepOutputs) {
/* Hmz, identical to storePathRequisites in nix-store. */
for (PathSet::iterator i = livePaths.begin();
i != livePaths.end(); ++i)
if (isDerivation(*i)) {
Derivation drv = derivationFromPath(*i);
string gcLevelStr = drv.env["__gcLevel"];
int gcLevel;
if (!string2Int(gcLevelStr, gcLevel))
gcLevel = defaultGcLevel;
if (gcLevel >= gcKeepOutputsThreshold)
for (DerivationOutputs::iterator j = drv.outputs.begin();
j != drv.outputs.end(); ++j)
if (store->isValidPath(j->second.path))
computeFSClosure(j->second.path, livePaths);
}
}
if (action == gcReturnLive) {
result = livePaths;
return;
}
if (!options.ignoreLiveness)
addAdditionalRoots(state.roots);
/* Read the temporary roots. This acquires read locks on all
per-process temporary root files. So after this point no paths
can be added to the set of temporary roots. */
PathSet tempRoots;
FDs fds;
readTempRoots(tempRoots, fds);
readTempRoots(state.tempRoots, fds);
state.roots.insert(state.tempRoots.begin(), state.tempRoots.end());
/* Close the temporary roots. Note that we *cannot* do this in
readTempRoots(), because there we may not have all locks yet,
meaning that an invalid path can become valid (and thus add to
the references graph) after we have added it to the closure
(and computeFSClosure() assumes that the presence of a path
means that it has already been closed). */
PathSet tempRootsClosed;
for (PathSet::iterator i = tempRoots.begin(); i != tempRoots.end(); ++i)
if (store->isValidPath(*i))
computeFSClosure(*i, tempRootsClosed);
else
tempRootsClosed.insert(*i);
/* For testing - see tests/gc-concurrent.sh. */
if (getenv("NIX_DEBUG_GC_WAIT"))
sleep(2);
/* After this point the set of roots or temporary roots cannot
increase, since we hold locks on everything. So everything
that is not currently in in `livePaths' or `tempRootsClosed'
can be deleted. */
/* Read the Nix store directory to find all currently existing
paths. */
PathSet storePathSet;
if (action != gcDeleteSpecific) {
Paths entries = readDirectory(nixStore);
for (Paths::iterator i = entries.begin(); i != entries.end(); ++i)
storePathSet.insert(canonPath(nixStore + "/" + *i));
} else {
for (PathSet::iterator i = pathsToDelete.begin();
i != pathsToDelete.end(); ++i)
{
that is not reachable from `roots'. */
/* Now either delete all garbage paths, or just the specified
paths (for gcDeleteSpecific). */
if (options.action == GCOptions::gcDeleteSpecific) {
foreach (PathSet::iterator, i, options.pathsToDelete) {
assertStorePath(*i);
storePathSet.insert(*i);
}
}
/* Topologically sort them under the `referrers' relation. That
is, a < b iff a is in referrers(b). This gives us the order in
which things can be deleted safely. */
/* !!! when we have multiple output paths per derivation, this
will not work anymore because we get cycles. */
Paths storePaths = topoSortPaths(storePathSet);
/* Try to delete store paths in the topologically sorted order. */
for (Paths::iterator i = storePaths.begin(); i != storePaths.end(); ++i) {
debug(format("considering deletion of `%1%'") % *i);
if (livePaths.find(*i) != livePaths.end()) {
if (action == gcDeleteSpecific)
if (!tryToDelete(state, *i))
throw Error(format("cannot delete path `%1%' since it is still alive") % *i);
debug(format("live path `%1%'") % *i);
continue;
}
} else {
printMsg(lvlError, format("reading the Nix store..."));
Paths entries = readDirectory(nixStore);
if (tempRootsClosed.find(*i) != tempRootsClosed.end()) {
debug(format("temporary root `%1%'") % *i);
continue;
/* Randomise the order in which we delete entries to make the
collector less biased towards deleting paths that come
alphabetically first (e.g. /nix/store/000...). This
matters when using --max-freed etc. */
vector<Path> entries_(entries.begin(), entries.end());
random_shuffle(entries_.begin(), entries_.end());
if (doDelete(state.options.action))
printMsg(lvlError, format("deleting garbage..."));
else
printMsg(lvlError, format("determining live/dead paths..."));
try {
foreach (vector<Path>::iterator, i, entries_)
tryToDelete(state, canonPath(nixStore + "/" + *i));
} catch (GCLimitReached & e) {
}
debug(format("dead path `%1%'") % *i);
result.insert(*i);
/* If just returning the set of dead paths, we also return the
space that would be freed if we deleted them. */
if (action == gcReturnDead)
bytesFreed += computePathSize(*i);
if (action == gcDeleteDead || action == gcDeleteSpecific) {
#ifndef __CYGWIN__
AutoCloseFD fdLock;
/* Only delete a lock file if we can acquire a write lock
on it. That means that it's either stale, or the
process that created it hasn't locked it yet. In the
latter case the other process will detect that we
deleted the lock, and retry (see pathlocks.cc). */
if (i->size() >= 5 && string(*i, i->size() - 5) == ".lock") {
fdLock = openLockFile(*i, false);
if (fdLock != -1 && !lockFile(fdLock, ltWrite, false)) {
debug(format("skipping active lock `%1%'") % *i);
continue;
}
}
#endif
if (!pathExists(*i)) continue;
printMsg(lvlInfo, format("deleting `%1%'") % *i);
/* Okay, it's safe to delete. */
try {
unsigned long long freed;
deleteFromStore(*i, freed);
bytesFreed += freed;
} catch (PathInUse & e) {
printMsg(lvlError, format("warning: %1%") % e.msg());
}
#ifndef __CYGWIN__
if (fdLock != -1)
/* Write token to stale (deleted) lock file. */
writeFull(fdLock, (const unsigned char *) "d", 1);
#endif
}
}
}
}
}
+21 -1
View File
@@ -24,15 +24,19 @@ Verbosity buildVerbosity = lvlInfo;
unsigned int maxBuildJobs = 1;
bool readOnlyMode = false;
string thisSystem = "unset";
unsigned int maxSilentTime = 0;
time_t maxSilentTime = 0;
Paths substituters;
bool useBuildHook = true;
bool printBuildTrace = false;
static bool settingsRead = false;
static std::map<string, Strings> settings;
/* Overriden settings. */
std::map<string, Strings> settingsCmdline;
string & at(Strings & ss, unsigned int n)
{
@@ -72,6 +76,8 @@ static void readSettings()
advance(i, 2);
settings[name] = Strings(i, tokens.end());
};
settings.insert(settingsCmdline.begin(), settingsCmdline.end());
settingsRead = true;
}
@@ -116,5 +122,19 @@ unsigned int queryIntSetting(const string & name, unsigned int def)
return n;
}
void overrideSetting(const string & name, const Strings & value)
{
if (settingsRead) settings[name] = value;
settingsCmdline[name] = value;
}
void reloadSettings()
{
settingsRead = false;
settings.clear();
}
}
+21 -1
View File
@@ -65,7 +65,7 @@ extern string thisSystem;
/* The maximum time in seconds that a builer can go without producing
any output on stdout/stderr before it is killed. 0 means
infinity. */
extern unsigned int maxSilentTime;
extern time_t maxSilentTime;
/* The substituters. There are programs that can somehow realise a
store path without building, e.g., by downloading it or copying it
@@ -76,6 +76,22 @@ extern Paths substituters;
users want to disable this from the command-line. */
extern bool useBuildHook;
/* Whether buildDerivations() should print out lines on stderr in a
fixed format to allow its progress to be monitored. Each line
starts with a "@". The following are defined:
@ build-started <drvpath> <outpath> <system> <logfile>
@ build-failed <drvpath> <outpath> <exitcode> <error text>
@ build-succeeded <drvpath> <outpath>
@ substituter-started <outpath> <substituter>
@ substituter-failed <outpath> <exitcode> <error text>
@ substituter-succeeded <outpath>
Best combined with --no-build-output, otherwise stderr might
conceivably contain lines in this format printed by the builders.
*/
extern bool printBuildTrace;
Strings querySetting(const string & name, const Strings & def);
@@ -85,6 +101,10 @@ bool queryBoolSetting(const string & name, bool def);
unsigned int queryIntSetting(const string & name, unsigned int def);
void overrideSetting(const string & name, const Strings & value);
void reloadSettings();
}
File diff suppressed because it is too large. Load diff
+113 -60
View File
@@ -4,18 +4,16 @@
#include <string>
#include "store-api.hh"
#include "util.hh"
namespace nix {
class Transaction;
/* Nix store and database schema version. Version 1 (or 0) was Nix <=
0.7. Version 2 was Nix 0.8 and 0.9. Version 3 is Nix 0.10.
Version 4 is Nix 0.11. */
const int nixSchemaVersion = 4;
Version 4 is Nix 0.11. Version 5 is Nix 0.12*/
const int nixSchemaVersion = 5;
extern string drvsLogDir;
@@ -27,31 +25,36 @@ struct OptimiseStats
unsigned long sameContents;
unsigned long filesLinked;
unsigned long long bytesFreed;
unsigned long long blocksFreed;
OptimiseStats()
{
totalFiles = sameContents = filesLinked = 0;
bytesFreed = 0;
bytesFreed = blocksFreed = 0;
}
};
struct RunningSubstituter
{
Pid pid;
AutoCloseFD to, from;
};
class LocalStore : public StoreAPI
{
private:
bool substitutablePathsLoaded;
PathSet substitutablePaths;
typedef std::map<Path, RunningSubstituter> RunningSubstituters;
RunningSubstituters runningSubstituters;
public:
/* Open the database environment. If `reserveSpace' is true, make
sure that a big empty file exists in /nix/var/nix/db/reserved.
If `reserveSpace' is false, delete this file if it exists. The
idea is that on normal operation, the file exists; but when we
run the garbage collector, it is deleted. This is to ensure
that the garbage collector has a small amount of disk space
available, which is required to open the Berkeley DB
environment. */
LocalStore(bool reserveSpace);
/* Initialise the local store, upgrading the schema if
necessary. */
LocalStore();
~LocalStore();
@@ -59,6 +62,8 @@ public:
bool isValidPath(const Path & path);
PathSet queryValidPaths();
Hash queryPathHash(const Path & path);
void queryReferences(const Path & path, PathSet & references);
@@ -70,12 +75,25 @@ public:
PathSet querySubstitutablePaths();
bool hasSubstitutes(const Path & path);
bool querySubstitutablePathInfo(const Path & path,
SubstitutablePathInfo & info);
Path addToStore(const Path & srcPath, bool fixed = false,
bool recursive = false, string hashAlgo = "",
bool querySubstitutablePathInfo(const Path & substituter,
const Path & path, SubstitutablePathInfo & info);
Path addToStore(const Path & srcPath,
bool recursive = true, HashType hashAlgo = htSHA256,
PathFilter & filter = defaultPathFilter);
Path addTextToStore(const string & suffix, const string & s,
/* Like addToStore(), but the contents of the path are contained
in `dump', which is either a NAR serialisation (if recursive ==
true) or simply the contents of a regular file (if recursive ==
false). */
Path addToStoreFromDump(const string & dump, const string & name,
bool recursive = true, HashType hashAlgo = htSHA256);
Path addTextToStore(const string & name, const string & s,
const PathSet & references);
void exportPath(const Path & path, bool sign,
@@ -95,65 +113,100 @@ public:
Roots findRoots();
void collectGarbage(GCAction action, const PathSet & pathsToDelete,
bool ignoreLiveness, PathSet & result, unsigned long long & bytesFreed);
void collectGarbage(const GCOptions & options, GCResults & results);
/* Delete a path from the Nix store. */
void deleteFromStore(const Path & path, unsigned long long & bytesFreed,
unsigned long long & blocksFreed);
/* Optimise the disk space usage of the Nix store by hard-linking
files with the same contents. */
void optimiseStore(bool dryRun, OptimiseStats & stats);
/* Check the integrity of the Nix store. */
void verifyStore(bool checkContents);
/* Register the validity of a path, i.e., that `path' exists, that
the paths referenced by it exists, and in the case of an output
path of a derivation, that it has been produced by a succesful
execution of the derivation (or something equivalent). Also
register the hash of the file system contents of the path. The
hash must be a SHA-256 hash. */
void registerValidPath(const Path & path,
const Hash & hash, const PathSet & references, const Path & deriver);
void registerValidPaths(const ValidPathInfos & infos);
/* Register that the build of a derivation with output `path' has
failed. */
void registerFailedPath(const Path & path);
/* Query whether `path' previously failed to build. */
bool hasPathFailed(const Path & path);
private:
Path schemaPath;
/* Lock file used for upgrading. */
AutoCloseFD globalLock;
/* !!! The cache can grow very big. Maybe it should be pruned
every once in a while. */
std::map<Path, ValidPathInfo> pathInfoCache;
/* Store paths for which the referrers file must be purged. */
PathSet delayedUpdates;
/* Whether to do an fsync() after writing Nix metadata. */
bool doFsync;
int getSchema();
void registerValidPath(const ValidPathInfo & info, bool ignoreValidity = false);
ValidPathInfo queryPathInfo(const Path & path, bool ignoreErrors = false);
void appendReferrer(const Path & from, const Path & to, bool lock);
void rewriteReferrers(const Path & path, bool purge, PathSet referrers);
void flushDelayedUpdates();
bool queryReferrersInternal(const Path & path, PathSet & referrers);
void invalidatePath(const Path & path);
void upgradeStore12();
struct GCState;
bool tryToDelete(GCState & state, const Path & path);
PathSet findDerivers(GCState & state, const Path & path);
bool isActiveTempFile(const GCState & state,
const Path & path, const string & suffix);
void startSubstituter(const Path & substituter,
RunningSubstituter & runningSubstituter);
};
/* Get a transaction object. */
void createStoreTransaction(Transaction & txn);
/* Copy a path recursively. */
void copyPath(const Path & src, const Path & dst);
/* Register the validity of a path, i.e., that `path' exists, that the
paths referenced by it exists, and in the case of an output path of
a derivation, that it has been produced by a succesful execution of
the derivation (or something equivalent). Also register the hash
of the file system contents of the path. The hash must be a
SHA-256 hash. */
void registerValidPath(const Transaction & txn,
const Path & path, const Hash & hash, const PathSet & references,
const Path & deriver);
typedef list<ValidPathInfo> ValidPathInfos;
void registerValidPaths(const Transaction & txn,
const ValidPathInfos & infos);
/* "Fix", or canonicalise, the meta-data of the files in a store path
after it has been built. In particular:
- the last modification date on each file is set to 0 (i.e.,
00:00:00 1/1/1970 UTC)
- the last modification date on each file is set to 1 (i.e.,
00:00:01 1/1/1970 UTC)
- the permissions are set of 444 or 555 (i.e., read-only with or
without execute permission; setuid bits etc. are cleared)
- the owner and group are set to the Nix user and group, if we're
in a setuid Nix installation. */
void canonicalisePathMetaData(const Path & path);
/* Checks whether a path is valid. */
bool isValidPathTxn(const Transaction & txn, const Path & path);
/* Sets the set of outgoing FS references for a store path. Use with
care! */
void setReferences(const Transaction & txn, const Path & path,
const PathSet & references);
/* Sets the deriver of a store path. Use with care! */
void setDeriver(const Transaction & txn, const Path & path,
const Path & deriver);
/* Delete a value from the nixStore directory. */
void deleteFromStore(const Path & path, unsigned long long & bytesFreed);
void canonicalisePathMetaData(const Path & path, bool recurse);
MakeError(PathInUse, Error);
void verifyStore(bool checkContents);
/* Whether we are in build users mode. */
bool haveBuildUsers();
@@ -166,7 +219,7 @@ void getOwnership(const Path & path);
/* Like deletePath(), but changes the ownership of `path' using the
setuid wrapper if necessary (and possible). */
void deletePathWrapped(const Path & path,
unsigned long long & bytesFreed);
unsigned long long & bytesFreed, unsigned long long & blocksFreed);
void deletePathWrapped(const Path & path);
+29 -18
View File
@@ -1,6 +1,6 @@
#include "misc.hh"
#include "store-api.hh"
#include "db.hh"
#include "local-store.hh"
#include <aterm2.h>
@@ -19,7 +19,7 @@ Derivation derivationFromPath(const Path & drvPath)
void computeFSClosure(const Path & storePath,
PathSet & paths, bool flipDirection)
PathSet & paths, bool flipDirection, bool includeOutputs)
{
if (paths.find(storePath) != paths.end()) return;
paths.insert(storePath);
@@ -30,24 +30,32 @@ void computeFSClosure(const Path & storePath,
else
store->queryReferences(storePath, references);
for (PathSet::iterator i = references.begin();
i != references.end(); ++i)
computeFSClosure(*i, paths, flipDirection);
if (includeOutputs && isDerivation(storePath)) {
Derivation drv = derivationFromPath(storePath);
foreach (DerivationOutputs::iterator, i, drv.outputs)
if (store->isValidPath(i->second.path))
computeFSClosure(i->second.path, paths, flipDirection, true);
}
foreach (PathSet::iterator, i, references)
computeFSClosure(*i, paths, flipDirection, includeOutputs);
}
Path findOutput(const Derivation & drv, string id)
{
for (DerivationOutputs::const_iterator i = drv.outputs.begin();
i != drv.outputs.end(); ++i)
foreach (DerivationOutputs::const_iterator, i, drv.outputs)
if (i->first == id) return i->second.path;
throw Error(format("derivation has no output `%1%'") % id);
}
void queryMissing(const PathSet & targets,
PathSet & willBuild, PathSet & willSubstitute)
PathSet & willBuild, PathSet & willSubstitute, PathSet & unknown,
unsigned long long & downloadSize)
{
downloadSize = 0;
PathSet todo(targets.begin(), targets.end()), done;
while (!todo.empty()) {
@@ -57,33 +65,36 @@ void queryMissing(const PathSet & targets,
done.insert(p);
if (isDerivation(p)) {
if (!store->isValidPath(p)) continue;
if (!store->isValidPath(p)) {
unknown.insert(p);
continue;
}
Derivation drv = derivationFromPath(p);
bool mustBuild = false;
for (DerivationOutputs::iterator i = drv.outputs.begin();
i != drv.outputs.end(); ++i)
foreach (DerivationOutputs::iterator, i, drv.outputs)
if (!store->isValidPath(i->second.path) && !store->hasSubstitutes(i->second.path))
mustBuild = true;
if (mustBuild) {
willBuild.insert(p);
todo.insert(drv.inputSrcs.begin(), drv.inputSrcs.end());
for (DerivationInputs::iterator i = drv.inputDrvs.begin();
i != drv.inputDrvs.end(); ++i)
foreach (DerivationInputs::iterator, i, drv.inputDrvs)
todo.insert(i->first);
} else
for (DerivationOutputs::iterator i = drv.outputs.begin();
i != drv.outputs.end(); ++i)
foreach (DerivationOutputs::iterator, i, drv.outputs)
todo.insert(i->second.path);
}
else {
if (store->isValidPath(p)) continue;
if (store->hasSubstitutes(p))
SubstitutablePathInfo info;
if (store->querySubstitutablePathInfo(p, info)) {
willSubstitute.insert(p);
// XXX call the substituters
// store->queryReferences(p, todo);
downloadSize += info.downloadSize;
todo.insert(info.references.begin(), info.references.end());
} else
unknown.insert(p);
}
}
}
+4 -2
View File
@@ -19,7 +19,8 @@ Derivation derivationFromPath(const Path & drvPath);
`referrers' relation instead of the `references' relation is
returned. */
void computeFSClosure(const Path & storePath,
PathSet & paths, bool flipDirection = false);
PathSet & paths, bool flipDirection = false,
bool includeOutputs = false);
/* Return the path corresponding to the output identifier `id' in the
given derivation. */
@@ -29,7 +30,8 @@ Path findOutput(const Derivation & drv, string id);
derivations that will be built, and the set of output paths that
will be substituted. */
void queryMissing(const PathSet & targets,
PathSet & willBuild, PathSet & willSubstitute);
PathSet & willBuild, PathSet & willSubstitute, PathSet & unknown,
unsigned long long & downloadSize);
}
+156
View File
@@ -0,0 +1,156 @@
#include "util.hh"
#include "local-store.hh"
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <errno.h>
#include <stdio.h>
namespace nix {
typedef std::map<Hash, std::pair<Path, ino_t> > HashToPath;
static void makeWritable(const Path & path)
{
struct stat st;
if (lstat(path.c_str(), &st))
throw SysError(format("getting attributes of path `%1%'") % path);
if (chmod(path.c_str(), st.st_mode | S_IWUSR) == -1)
throw SysError(format("changing writability of `%1%'") % path);
}
struct MakeReadOnly
{
Path path;
MakeReadOnly(const Path & path) : path(path) { }
~MakeReadOnly()
{
try {
if (path != "") canonicalisePathMetaData(path, false);
} catch (...) {
ignoreException();
}
}
};
static void hashAndLink(bool dryRun, HashToPath & hashToPath,
OptimiseStats & stats, const Path & path)
{
struct stat st;
if (lstat(path.c_str(), &st))
throw SysError(format("getting attributes of path `%1%'") % path);
/* Sometimes SNAFUs can cause files in the Nix store to be
modified, in particular when running programs as root under
NixOS (example: $fontconfig/var/cache being modified). Skip
those files. */
if (S_ISREG(st.st_mode) && (st.st_mode & S_IWUSR)) {
printMsg(lvlError, format("skipping suspicious writable file `%1%'") % path);
return;
}
/* We can hard link regular files and symlinks. */
if (S_ISREG(st.st_mode) || S_ISLNK(st.st_mode)) {
/* Hash the file. Note that hashPath() returns the hash over
the NAR serialisation, which includes the execute bit on
the file. Thus, executable and non-executable files with
the same contents *won't* be linked (which is good because
otherwise the permissions would be screwed up).
Also note that if `path' is a symlink, then we're hashing
the contents of the symlink (i.e. the result of
readlink()), not the contents of the target (which may not
even exist). */
Hash hash = hashPath(htSHA256, path);
stats.totalFiles++;
printMsg(lvlDebug, format("`%1%' has hash `%2%'") % path % printHash(hash));
std::pair<Path, ino_t> prevPath = hashToPath[hash];
if (prevPath.first == "") {
hashToPath[hash] = std::pair<Path, ino_t>(path, st.st_ino);
return;
}
/* Yes! We've seen a file with the same contents. Replace
the current file with a hard link to that file. */
stats.sameContents++;
if (prevPath.second == st.st_ino) {
printMsg(lvlDebug, format("`%1%' is already linked to `%2%'") % path % prevPath.first);
return;
}
if (!dryRun) {
printMsg(lvlTalkative, format("linking `%1%' to `%2%'") % path % prevPath.first);
Path tempLink = (format("%1%.tmp-%2%-%3%")
% path % getpid() % rand()).str();
/* Make the containing directory writable, but only if
it's not the store itself (we don't want or need to
mess with its permissions). */
bool mustToggle = !isStorePath(path);
if (mustToggle) makeWritable(dirOf(path));
/* When we're done, make the directory read-only again and
reset its timestamp back to 0. */
MakeReadOnly makeReadOnly(mustToggle ? dirOf(path) : "");
if (link(prevPath.first.c_str(), tempLink.c_str()) == -1) {
if (errno == EMLINK) {
/* Too many links to the same file (>= 32000 on
most file systems). This is likely to happen
with empty files. Just start over, creating
links to the current file. */
printMsg(lvlInfo, format("`%1%' has maximum number of links") % prevPath.first);
hashToPath[hash] = std::pair<Path, ino_t>(path, st.st_ino);
return;
}
throw SysError(format("cannot link `%1%' to `%2%'")
% tempLink % prevPath.first);
}
/* Atomically replace the old file with the new hard link. */
if (rename(tempLink.c_str(), path.c_str()) == -1)
throw SysError(format("cannot rename `%1%' to `%2%'")
% tempLink % path);
} else
printMsg(lvlTalkative, format("would link `%1%' to `%2%'") % path % prevPath.first);
stats.filesLinked++;
stats.bytesFreed += st.st_size;
stats.blocksFreed += st.st_blocks;
}
if (S_ISDIR(st.st_mode)) {
Strings names = readDirectory(path);
foreach (Strings::iterator, i, names)
hashAndLink(dryRun, hashToPath, stats, path + "/" + *i);
}
}
void LocalStore::optimiseStore(bool dryRun, OptimiseStats & stats)
{
HashToPath hashToPath;
PathSet paths = queryValidPaths();
foreach (PathSet::iterator, i, paths) {
addTempRoot(*i);
if (!isValidPath(*i)) continue; /* path was GC'ed, probably */
startNest(nest, lvlChatty, format("hashing files in `%1%'") % *i);
hashAndLink(dryRun, hashToPath, stats, *i);
}
}
}
+24 -6
View File
@@ -2,6 +2,7 @@
#include "util.hh"
#include <cerrno>
#include <cstdlib>
#include <sys/types.h>
#include <sys/stat.h>
@@ -140,9 +141,9 @@ PathLocks::PathLocks(const PathSet & paths, const string & waitMsg)
}
void PathLocks::lockPaths(const PathSet & _paths, const string & waitMsg)
bool PathLocks::lockPaths(const PathSet & _paths,
const string & waitMsg, bool wait)
{
/* May be called only once! */
assert(fds.empty());
/* Note that `fds' is built incrementally so that the destructor
@@ -154,7 +155,7 @@ void PathLocks::lockPaths(const PathSet & _paths, const string & waitMsg)
paths.sort();
/* Acquire the lock for each path. */
for (Paths::iterator i = paths.begin(); i != paths.end(); i++) {
foreach (Paths::iterator, i, paths) {
checkInterrupt();
Path path = *i;
Path lockPath = path + ".lock";
@@ -173,8 +174,15 @@ void PathLocks::lockPaths(const PathSet & _paths, const string & waitMsg)
/* Acquire an exclusive lock. */
if (!lockFile(fd, ltWrite, false)) {
if (waitMsg != "") printMsg(lvlError, waitMsg);
lockFile(fd, ltWrite, true);
if (wait) {
if (waitMsg != "") printMsg(lvlError, waitMsg);
lockFile(fd, ltWrite, true);
} else {
/* Failed to lock this path; release all other
locks. */
unlock();
return false;
}
}
debug(format("lock acquired on `%1%'") % lockPath);
@@ -198,12 +206,20 @@ void PathLocks::lockPaths(const PathSet & _paths, const string & waitMsg)
fds.push_back(FDPair(fd.borrow(), lockPath));
lockedPaths.insert(lockPath);
}
return true;
}
PathLocks::~PathLocks()
{
for (list<FDPair>::iterator i = fds.begin(); i != fds.end(); i++) {
unlock();
}
void PathLocks::unlock()
{
foreach (list<FDPair>::iterator, i, fds) {
if (deletePaths) deleteLockFilePreClose(i->second, i->first);
lockedPaths.erase(i->second);
@@ -215,6 +231,8 @@ PathLocks::~PathLocks()
debug(format("lock released on `%1%'") % i->second);
}
fds.clear();
}
+5 -3
View File
@@ -17,7 +17,7 @@ int openLockFile(const Path & path, bool create);
void deleteLockFilePreClose(const Path & path, int fd);
void deleteLockFilePostClose(const Path & path);
typedef enum LockType { ltRead, ltWrite, ltNone };
enum LockType { ltRead, ltWrite, ltNone };
bool lockFile(int fd, LockType lockType, bool wait);
@@ -33,9 +33,11 @@ public:
PathLocks();
PathLocks(const PathSet & paths,
const string & waitMsg = "");
void lockPaths(const PathSet & _paths,
const string & waitMsg = "");
bool lockPaths(const PathSet & _paths,
const string & waitMsg = "",
bool wait = true);
~PathLocks();
void unlock();
void setDeletion(bool deletePaths);
};
+41 -75
View File
@@ -1,15 +1,10 @@
#include "references.hh"
#include "hash.hh"
#include "util.hh"
#include "archive.hh"
#include <cerrno>
#include <map>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <dirent.h>
#include <fcntl.h>
#include <cstdlib>
namespace nix {
@@ -18,8 +13,8 @@ namespace nix {
static unsigned int refLength = 32; /* characters */
static void search(size_t len, const unsigned char * s,
StringSet & ids, StringSet & seen)
static void search(const unsigned char * s, unsigned int len,
StringSet & hashes, StringSet & seen)
{
static bool initialised = false;
static bool isBase32[256];
@@ -41,93 +36,60 @@ static void search(size_t len, const unsigned char * s,
}
if (!match) continue;
string ref((const char *) s + i, refLength);
if (ids.find(ref) != ids.end()) {
if (hashes.find(ref) != hashes.end()) {
debug(format("found reference to `%1%' at offset `%2%'")
% ref % i);
seen.insert(ref);
ids.erase(ref);
hashes.erase(ref);
}
++i;
}
}
void checkPath(const string & path,
StringSet & ids, StringSet & seen)
struct RefScanSink : Sink
{
checkInterrupt();
HashSink hashSink;
StringSet hashes;
StringSet seen;
string tail;
RefScanSink() : hashSink(htSHA256) { }
debug(format("checking `%1%'") % path);
void operator () (const unsigned char * data, unsigned int len);
};
struct stat st;
if (lstat(path.c_str(), &st))
throw SysError(format("getting attributes of path `%1%'") % path);
if (S_ISDIR(st.st_mode)) {
Strings names = readDirectory(path);
for (Strings::iterator i = names.begin(); i != names.end(); i++) {
search(i->size(), (const unsigned char *) i->c_str(), ids, seen);
checkPath(path + "/" + *i, ids, seen);
}
}
void RefScanSink::operator () (const unsigned char * data, unsigned int len)
{
hashSink(data, len);
else if (S_ISREG(st.st_mode)) {
AutoCloseFD fd = open(path.c_str(), O_RDONLY);
if (fd == -1) throw SysError(format("opening file `%1%'") % path);
/* It's possible that a reference spans the previous and current
fragment, so search in the concatenation of the tail of the
previous fragment and the start of the current fragment. */
string s = tail + string((const char *) data, len > refLength ? refLength : len);
search((const unsigned char *) s.c_str(), s.size(), hashes, seen);
size_t bufSize = 1024 * 1024;
assert(refLength <= bufSize);
unsigned char * buf = new unsigned char[bufSize];
search(data, len, hashes, seen);
size_t left = st.st_size;
bool firstBlock = true;
while (left > 0) {
checkInterrupt();
size_t read = left > bufSize ? bufSize : left;
size_t copiedBytes = 0;
if (!firstBlock) {
/* Move the last (refLength - 1) bytes from the last
block to the start of the buffer to deal with
references that cross block boundaries. */
copiedBytes = refLength - 1;
if (read + copiedBytes > bufSize)
read -= copiedBytes;
memcpy(buf, buf + (bufSize - copiedBytes), copiedBytes);
}
firstBlock = false;
readFull(fd, buf + copiedBytes, read);
left -= read;
search(copiedBytes + read, buf, ids, seen);
}
delete[] buf; /* !!! autodelete */
}
else if (S_ISLNK(st.st_mode)) {
string target = readLink(path);
search(target.size(), (const unsigned char *) target.c_str(), ids, seen);
}
else throw Error(format("unknown file type: %1%") % path);
unsigned int tailLen = len <= refLength ? len : refLength;
tail =
string(tail, tail.size() < refLength - tailLen ? 0 : tail.size() - (refLength - tailLen)) +
string((const char *) data + len - tailLen, tailLen);
}
PathSet scanForReferences(const string & path, const PathSet & paths)
PathSet scanForReferences(const string & path,
const PathSet & refs, Hash & hash)
{
RefScanSink sink;
std::map<string, Path> backMap;
StringSet ids;
StringSet seen;
/* For efficiency (and a higher hit rate), just search for the
hash part of the file name. (This assumes that all references
have the form `HASH-bla'). */
for (PathSet::const_iterator i = paths.begin(); i != paths.end(); i++) {
foreach (PathSet::const_iterator, i, refs) {
string baseName = baseNameOf(*i);
string::size_type pos = baseName.find('-');
if (pos == string::npos)
@@ -136,21 +98,25 @@ PathSet scanForReferences(const string & path, const PathSet & paths)
assert(s.size() == refLength);
assert(backMap.find(s) == backMap.end());
// parseHash(htSHA256, s);
ids.insert(s);
sink.hashes.insert(s);
backMap[s] = *i;
}
checkPath(path, ids, seen);
/* Look for the hashes in the NAR dump of the path. */
dumpPath(path, sink);
/* Map the hashes found back to their store paths. */
PathSet found;
for (StringSet::iterator i = seen.begin(); i != seen.end(); i++) {
foreach (StringSet::iterator, i, sink.seen) {
std::map<string, Path>::iterator j;
if ((j = backMap.find(*i)) == backMap.end()) abort();
found.insert(j->second);
}
hash = sink.hashSink.finish();
return found;
}
}
+3 -1
View File
@@ -2,10 +2,12 @@
#define __REFERENCES_H
#include "types.hh"
#include "hash.hh"
namespace nix {
PathSet scanForReferences(const Path & path, const PathSet & refs);
PathSet scanForReferences(const Path & path, const PathSet & refs,
Hash & hash);
}
+98 -34
View File
@@ -29,14 +29,22 @@ Path readStorePath(Source & from)
PathSet readStorePaths(Source & from)
{
PathSet paths = readStringSet(from);
for (PathSet::iterator i = paths.begin(); i != paths.end(); ++i)
assertStorePath(*i);
foreach (PathSet::iterator, i, paths) assertStorePath(*i);
return paths;
}
RemoteStore::RemoteStore()
{
initialised = false;
}
void RemoteStore::openConnection()
{
if (initialised) return;
initialised = true;
string remoteMode = getEnv("NIX_REMOTE");
if (remoteMode == "slave")
@@ -64,8 +72,8 @@ RemoteStore::RemoteStore()
throw Error("Nix daemon protocol version not supported");
writeInt(PROTOCOL_VERSION, to);
processStderr();
} catch (Error & e) {
}
catch (Error & e) {
throw Error(format("cannot start worker (%1%)")
% e.msg());
}
@@ -137,14 +145,26 @@ void RemoteStore::connectToDaemon()
string socketPath = nixStateDir + DEFAULT_SOCKET_PATH;
/* Urgh, sockaddr_un allows path names of only 108 characters. So
chdir to the socket directory so that we can pass a relative
path name. !!! this is probably a bad idea in multi-threaded
applications... */
AutoCloseFD fdPrevDir = open(".", O_RDONLY);
if (fdPrevDir == -1) throw SysError("couldn't open current directory");
chdir(dirOf(socketPath).c_str());
Path socketPathRel = "./" + baseNameOf(socketPath);
struct sockaddr_un addr;
addr.sun_family = AF_UNIX;
if (socketPath.size() >= sizeof(addr.sun_path))
throw Error(format("socket path `%1%' is too long") % socketPath);
strcpy(addr.sun_path, socketPath.c_str());
if (socketPathRel.size() >= sizeof(addr.sun_path))
throw Error(format("socket path `%1%' is too long") % socketPathRel);
strcpy(addr.sun_path, socketPathRel.c_str());
if (connect(fdSocket, (struct sockaddr *) &addr, sizeof(addr)) == -1)
throw SysError(format("cannot connect to daemon at `%1%'") % socketPath);
if (fchdir(fdPrevDir) == -1)
throw SysError("couldn't change back to previous directory");
}
@@ -171,12 +191,18 @@ void RemoteStore::setOptions()
writeInt(maxSilentTime, to);
if (GET_PROTOCOL_MINOR(daemonVersion) >= 2)
writeInt(useBuildHook, to);
if (GET_PROTOCOL_MINOR(daemonVersion) >= 4) {
writeInt(buildVerbosity, to);
writeInt(logType, to);
writeInt(printBuildTrace, to);
}
processStderr();
}
bool RemoteStore::isValidPath(const Path & path)
{
openConnection();
writeInt(wopIsValidPath, to);
writeString(path, to);
processStderr();
@@ -185,8 +211,16 @@ bool RemoteStore::isValidPath(const Path & path)
}
PathSet RemoteStore::queryValidPaths()
{
openConnection();
throw Error("not implemented");
}
bool RemoteStore::hasSubstitutes(const Path & path)
{
openConnection();
writeInt(wopHasSubstitutes, to);
writeString(path, to);
processStderr();
@@ -195,8 +229,27 @@ bool RemoteStore::hasSubstitutes(const Path & path)
}
bool RemoteStore::querySubstitutablePathInfo(const Path & path,
SubstitutablePathInfo & info)
{
openConnection();
if (GET_PROTOCOL_MINOR(daemonVersion) < 3) return false;
writeInt(wopQuerySubstitutablePathInfo, to);
writeString(path, to);
processStderr();
unsigned int reply = readInt(from);
if (reply == 0) return false;
info.deriver = readString(from);
if (info.deriver != "") assertStorePath(info.deriver);
info.references = readStorePaths(from);
info.downloadSize = readLongLong(from);
return true;
}
Hash RemoteStore::queryPathHash(const Path & path)
{
openConnection();
writeInt(wopQueryPathHash, to);
writeString(path, to);
processStderr();
@@ -208,6 +261,7 @@ Hash RemoteStore::queryPathHash(const Path & path)
void RemoteStore::queryReferences(const Path & path,
PathSet & references)
{
openConnection();
writeInt(wopQueryReferences, to);
writeString(path, to);
processStderr();
@@ -219,6 +273,7 @@ void RemoteStore::queryReferences(const Path & path,
void RemoteStore::queryReferrers(const Path & path,
PathSet & referrers)
{
openConnection();
writeInt(wopQueryReferrers, to);
writeString(path, to);
processStderr();
@@ -229,6 +284,7 @@ void RemoteStore::queryReferrers(const Path & path,
Path RemoteStore::queryDeriver(const Path & path)
{
openConnection();
writeInt(wopQueryDeriver, to);
writeString(path, to);
processStderr();
@@ -238,33 +294,31 @@ Path RemoteStore::queryDeriver(const Path & path)
}
PathSet RemoteStore::querySubstitutablePaths()
{
throw Error("not implemented");
}
Path RemoteStore::addToStore(const Path & _srcPath, bool fixed,
bool recursive, string hashAlgo, PathFilter & filter)
Path RemoteStore::addToStore(const Path & _srcPath,
bool recursive, HashType hashAlgo, PathFilter & filter)
{
openConnection();
Path srcPath(absPath(_srcPath));
writeInt(wopAddToStore, to);
writeString(baseNameOf(srcPath), to);
writeInt(fixed ? 1 : 0, to);
/* backwards compatibility hack */
writeInt((hashAlgo == htSHA256 && recursive) ? 0 : 1, to);
writeInt(recursive ? 1 : 0, to);
writeString(hashAlgo, to);
writeString(printHashType(hashAlgo), to);
dumpPath(srcPath, to, filter);
processStderr();
return readStorePath(from);
}
Path RemoteStore::addTextToStore(const string & suffix, const string & s,
Path RemoteStore::addTextToStore(const string & name, const string & s,
const PathSet & references)
{
openConnection();
writeInt(wopAddTextToStore, to);
writeString(suffix, to);
writeString(name, to);
writeString(s, to);
writeStringSet(references, to);
@@ -276,6 +330,7 @@ Path RemoteStore::addTextToStore(const string & suffix, const string & s,
void RemoteStore::exportPath(const Path & path, bool sign,
Sink & sink)
{
openConnection();
writeInt(wopExportPath, to);
writeString(path, to);
writeInt(sign ? 1 : 0, to);
@@ -286,10 +341,10 @@ void RemoteStore::exportPath(const Path & path, bool sign,
Path RemoteStore::importPath(bool requireSignature, Source & source)
{
openConnection();
writeInt(wopImportPath, to);
/* We ignore requireSignature, since the worker forces it to true
anyway. */
anyway. */
processStderr(0, &source);
return readStorePath(from);
}
@@ -297,6 +352,7 @@ Path RemoteStore::importPath(bool requireSignature, Source & source)
void RemoteStore::buildDerivations(const PathSet & drvPaths)
{
openConnection();
writeInt(wopBuildDerivations, to);
writeStringSet(drvPaths, to);
processStderr();
@@ -306,6 +362,7 @@ void RemoteStore::buildDerivations(const PathSet & drvPaths)
void RemoteStore::ensurePath(const Path & path)
{
openConnection();
writeInt(wopEnsurePath, to);
writeString(path, to);
processStderr();
@@ -315,6 +372,7 @@ void RemoteStore::ensurePath(const Path & path)
void RemoteStore::addTempRoot(const Path & path)
{
openConnection();
writeInt(wopAddTempRoot, to);
writeString(path, to);
processStderr();
@@ -324,6 +382,7 @@ void RemoteStore::addTempRoot(const Path & path)
void RemoteStore::addIndirectRoot(const Path & path)
{
openConnection();
writeInt(wopAddIndirectRoot, to);
writeString(path, to);
processStderr();
@@ -333,6 +392,7 @@ void RemoteStore::addIndirectRoot(const Path & path)
void RemoteStore::syncWithGC()
{
openConnection();
writeInt(wopSyncWithGC, to);
processStderr();
readInt(from);
@@ -341,6 +401,7 @@ void RemoteStore::syncWithGC()
Roots RemoteStore::findRoots()
{
openConnection();
writeInt(wopFindRoots, to);
processStderr();
unsigned int count = readInt(from);
@@ -354,24 +415,27 @@ Roots RemoteStore::findRoots()
}
void RemoteStore::collectGarbage(GCAction action, const PathSet & pathsToDelete,
bool ignoreLiveness, PathSet & result, unsigned long long & bytesFreed)
void RemoteStore::collectGarbage(const GCOptions & options, GCResults & results)
{
result.clear();
bytesFreed = 0;
openConnection();
writeInt(wopCollectGarbage, to);
writeInt(action, to);
writeStringSet(pathsToDelete, to);
writeInt(ignoreLiveness, to);
writeInt(options.action, to);
writeStringSet(options.pathsToDelete, to);
writeInt(options.ignoreLiveness, to);
writeLongLong(options.maxFreed, to);
writeInt(options.maxLinks, to);
if (GET_PROTOCOL_MINOR(daemonVersion) >= 5) {
/* removed options */
writeInt(0, to);
writeInt(0, to);
}
processStderr();
result = readStringSet(from);
/* Ugh - NAR integers are 64 bits, but read/writeInt() aren't. */
unsigned int lo = readInt(from);
unsigned int hi = readInt(from);
bytesFreed = (((unsigned long long) hi) << 32) | lo;
results.paths = readStringSet(from);
results.bytesFreed = readLongLong(from);
results.blocksFreed = readLongLong(from);
}
Loaded 100 of 208 files, more files were not shown because too many files have changed in this diff. Show more